2024-03-23T11:05:44.397 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-23T11:05:44.397 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rishabh@teuthology --name rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234 --verbose --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} -- /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234/orig.config.yaml 2024-03-23T11:05:44.440 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T11:05:44.531 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234 branch: rishabh-yuri-RunCephCmd-base-reef-2 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} email: null first_in_suite: false job_id: '7618234' kernel: flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-yuri-RunCephCmd-base-reef-2 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - inode wrongly marked free - bad backtrace on inode - inode table repaired for inode - Scrub error on inode - Scrub error on dir - Metadata damage detected sha1: b630d6cd702af63dfefcf906294a092ddce1c609 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b630d6cd702af63dfefcf906294a092ddce1c609 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-yuri-RunCephCmd-base-reef-2 sha1: b630d6cd702af63dfefcf906294a092ddce1c609 owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1956 sha1: b630d6cd702af63dfefcf906294a092ddce1c609 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: rishabh-yuri-RunCephCmd-base-reef-2 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: b630d6cd702af63dfefcf906294a092ddce1c609 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBfVklLINf8hI4SjP+XzEWierEPkHx2H2V2ejczUNn1z4nGxBS+XfW1RinZqevtqk6b+cm2PeBcbUbgepxvTjw= smithi120.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMeZfkagDYOHSU0NTiM1iySUjN7prNazzwfIgxQoca6hVKsAjI6H2iOKJJ613zG69zBqLzNGYi5FLgQyo/nSRB4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_forward_scrub teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-23_09:38:22 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-23T11:05:44.531 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa; will attempt to use it 2024-03-23T11:05:44.532 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa/tasks 2024-03-23T11:05:44.532 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-23T11:05:44.534 INFO:teuthology.task.internal:Checking packages... 2024-03-23T11:05:44.562 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b630d6cd702af63dfefcf906294a092ddce1c609' 2024-03-23T11:05:44.562 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T11:05:44.562 INFO:teuthology.packaging:ref: None 2024-03-23T11:05:44.562 INFO:teuthology.packaging:tag: None 2024-03-23T11:05:44.562 INFO:teuthology.packaging:branch: rishabh-yuri-RunCephCmd-base-reef-2 2024-03-23T11:05:44.562 INFO:teuthology.packaging:sha1: b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:05:44.563 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-yuri-RunCephCmd-base-reef-2 2024-03-23T11:05:44.707 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-820.gb630d6cd 2024-03-23T11:05:44.708 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-23T11:05:44.715 INFO:teuthology.task.internal:no buildpackages task found 2024-03-23T11:05:44.715 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-23T11:05:44.721 INFO:teuthology.task.internal:Saving configuration 2024-03-23T11:05:44.736 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-23T11:05:44.742 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T11:05:44.763 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 11:00:41.438059', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBfVklLINf8hI4SjP+XzEWierEPkHx2H2V2ejczUNn1z4nGxBS+XfW1RinZqevtqk6b+cm2PeBcbUbgepxvTjw='} 2024-03-23T11:05:44.785 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 11:00:41.435771', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMeZfkagDYOHSU0NTiM1iySUjN7prNazzwfIgxQoca6hVKsAjI6H2iOKJJ613zG69zBqLzNGYi5FLgQyo/nSRB4='} 2024-03-23T11:05:44.785 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-23T11:05:44.791 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-23T11:05:44.791 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-23T11:05:44.791 INFO:teuthology.run_tasks:Running task console_log... 2024-03-23T11:05:44.879 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5f85866160>, signals=[15]) 2024-03-23T11:05:44.879 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-23T11:05:44.886 INFO:teuthology.task.internal:Opening connections... 2024-03-23T11:05:44.887 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:05:44.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:05:44.996 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:05:44.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:05:45.088 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-23T11:05:45.094 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-03-23T11:05:45.111 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-03-23T11:05:45.111 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="9" 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="9" 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el9" 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:LOGO="fedora-logo-icon" 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2024-03-23T11:05:45.168 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T11:05:45.169 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T11:05:45.169 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T11:05:45.169 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-03-23T11:05:45.205 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2024-03-23T11:05:45.222 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2024-03-23T11:05:45.223 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2024-03-23T11:05:45.277 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Stream" 2024-03-23T11:05:45.277 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="9" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="9" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el9" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:LOGO="fedora-logo-icon" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://centos.org/" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T11:05:45.278 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T11:05:45.279 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2024-03-23T11:05:45.304 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-23T11:05:45.313 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-23T11:05:45.317 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-23T11:05:45.317 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2024-03-23T11:05:45.320 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2024-03-23T11:05:45.335 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-23T11:05:45.341 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-23T11:05:45.341 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2024-03-23T11:05:45.377 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2024-03-23T11:05:45.397 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-23T11:05:45.451 INFO:teuthology.run_tasks:Running task kernel... 2024-03-23T11:05:45.465 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-23T11:05:45.465 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}}, timeout 300 2024-03-23T11:05:45.465 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T11:05:45.465 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T11:05:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:05:45.482 DEBUG:teuthology.task.kernel:sha1 for mon.a is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:05:45.482 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-23T11:05:45.482 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-03-23T11:05:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:05:45.483 DEBUG:teuthology.task.kernel:sha1 for mon.b is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:05:45.483 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-23T11:05:45.483 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2024-03-23T11:05:45.539 INFO:teuthology.orchestra.run.smithi120.stdout:5.14.0-361.el9.x86_64 2024-03-23T11:05:45.539 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:05:45.539 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-23T11:05:45.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:05:45.543 INFO:teuthology.orchestra.run.smithi062.stdout:5.14.0-361.el9.x86_64 2024-03-23T11:05:45.543 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:05:45.543 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-23T11:05:45.543 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:05:45.686 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum upgrade -y linux-firmware 2024-03-23T11:05:45.756 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum upgrade -y linux-firmware 2024-03-23T11:05:47.896 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-23T11:05:47.899 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-23T11:05:49.370 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-23T11:05:49.375 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-23T11:05:50.324 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-23T11:05:50.353 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-23T11:05:53.846 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 19 MB 00:02 2024-03-23T11:06:01.438 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.1 MB 00:04 2024-03-23T11:06:01.532 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 19 MB 00:10 2024-03-23T11:06:02.896 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-23T11:06:03.756 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 28 MB/s | 21 MB 00:00 2024-03-23T11:06:05.819 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.1 MB 00:00 2024-03-23T11:06:07.309 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 16 kB 00:00 2024-03-23T11:06:08.385 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-23T11:06:08.931 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-23T11:06:11.211 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-23T11:06:11.217 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-23T11:06:11.217 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-23T11:06:11.217 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-23T11:06:11.217 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrading: 2024-03-23T11:06:11.217 INFO:teuthology.orchestra.run.smithi062.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-23T11:06:11.218 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:06:11.218 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-23T11:06:11.218 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-23T11:06:11.218 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrade 1 Package 2024-03-23T11:06:11.218 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:06:11.218 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 375 M 2024-03-23T11:06:11.218 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-23T11:06:13.319 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-03-23T11:06:15.536 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-23T11:06:15.547 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-23T11:06:15.547 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-23T11:06:15.547 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-23T11:06:15.547 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2024-03-23T11:06:15.547 INFO:teuthology.orchestra.run.smithi120.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-23T11:06:15.548 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:06:15.548 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-23T11:06:15.548 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-23T11:06:15.548 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 1 Package 2024-03-23T11:06:15.548 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:06:15.548 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 375 M 2024-03-23T11:06:15.548 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-23T11:06:18.224 INFO:teuthology.orchestra.run.smithi062.stdout:linux-firmware-20240219-143.el9.noarch.rpm 54 MB/s | 375 MB 00:07 2024-03-23T11:06:18.225 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:06:18.225 INFO:teuthology.orchestra.run.smithi062.stdout:Total 54 MB/s | 375 MB 00:07 2024-03-23T11:06:18.226 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-23T11:06:18.309 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-23T11:06:18.309 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-23T11:06:20.791 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-23T11:06:20.792 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-23T11:06:22.862 INFO:teuthology.orchestra.run.smithi120.stdout:linux-firmware-20240219-143.el9.noarch.rpm 51 MB/s | 375 MB 00:07 2024-03-23T11:06:22.863 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:06:22.863 INFO:teuthology.orchestra.run.smithi120.stdout:Total 51 MB/s | 375 MB 00:07 2024-03-23T11:06:22.863 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-23T11:06:22.987 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-23T11:06:22.987 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-23T11:06:25.391 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-23T11:06:25.391 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-23T11:06:27.283 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-23T11:06:27.553 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T11:06:27.649 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T11:06:28.970 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T11:06:28.971 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T11:06:29.377 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T11:06:29.377 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:06:29.378 INFO:teuthology.orchestra.run.smithi062.stdout:Upgraded: 2024-03-23T11:06:29.378 INFO:teuthology.orchestra.run.smithi062.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-23T11:06:29.378 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:06:29.378 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-23T11:06:29.670 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.a... 2024-03-23T11:06:29.670 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:06:29.895 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-23T11:06:29.895 DEBUG:teuthology.orchestra.run.smithi062:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-23T11:06:29.898 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi062.front.sepia.ceph.com to complete... 2024-03-23T11:06:31.760 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-23T11:06:31.998 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T11:06:32.072 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T11:06:33.546 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T11:06:33.546 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T11:06:33.975 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T11:06:33.975 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:06:33.976 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2024-03-23T11:06:33.976 INFO:teuthology.orchestra.run.smithi120.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-23T11:06:33.976 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:06:33.976 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-23T11:06:34.291 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.b... 2024-03-23T11:06:34.291 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T11:06:34.519 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-23T11:06:34.519 DEBUG:teuthology.orchestra.run.smithi120:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-23T11:06:34.521 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi120.front.sepia.ceph.com to complete... 2024-03-23T11:06:42.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23 11:06:42 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-23T11:06:42.882 INFO:teuthology.orchestra.run.smithi062.stderr:FINISHED --2024-03-23 11:06:42-- 2024-03-23T11:06:42.882 INFO:teuthology.orchestra.run.smithi062.stderr:Total wall clock time: 13s 2024-03-23T11:06:42.882 INFO:teuthology.orchestra.run.smithi062.stderr:Downloaded: 1 files, 321M in 13s (25.4 MB/s) 2024-03-23T11:06:42.884 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.a... 2024-03-23T11:06:42.885 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-23T11:06:44.910 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying... ######################################## 2024-03-23T11:06:44.968 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing... ######################################## 2024-03-23T11:06:45.053 INFO:teuthology.orchestra.run.smithi062.stdout:Updating / installing... 2024-03-23T11:06:52.909 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-23T11:06:54.063 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23 11:06:54 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-23T11:06:54.063 INFO:teuthology.orchestra.run.smithi120.stderr:FINISHED --2024-03-23 11:06:54-- 2024-03-23T11:06:54.063 INFO:teuthology.orchestra.run.smithi120.stderr:Total wall clock time: 20s 2024-03-23T11:06:54.063 INFO:teuthology.orchestra.run.smithi120.stderr:Downloaded: 1 files, 321M in 19s (16.9 MB/s) 2024-03-23T11:06:54.066 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.b... 2024-03-23T11:06:54.066 DEBUG:teuthology.orchestra.run.smithi120:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-23T11:06:55.996 INFO:teuthology.orchestra.run.smithi120.stdout:Verifying... ######################################## 2024-03-23T11:06:56.121 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing... ######################################## 2024-03-23T11:06:56.200 INFO:teuthology.orchestra.run.smithi120.stdout:Updating / installing... 2024-03-23T11:07:03.683 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-23T11:07:08.721 INFO:teuthology.orchestra.run.smithi062.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T11:07:20.188 INFO:teuthology.orchestra.run.smithi120.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T11:07:31.948 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-23T11:07:31.948 DEBUG:teuthology.orchestra.run.smithi062:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-23T11:07:32.028 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-23T11:07:32.029 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-23T11:07:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-23T11:07:32.031 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-23T11:07:32.032 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-23T11:07:32.033 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-23T11:07:32.034 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-23T11:07:32.035 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-23T11:07:32.036 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-23T11:07:32.037 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-23T11:07:32.038 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-23T11:07:32.039 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-23T11:07:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-23T11:07:32.041 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-23T11:07:32.042 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-23T11:07:32.043 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-23T11:07:32.044 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-23T11:07:32.045 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-23T11:07:32.046 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-23T11:07:32.047 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-23T11:07:32.048 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-23T11:07:32.049 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-23T11:07:32.050 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-23T11:07:32.051 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-23T11:07:32.052 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-23T11:07:32.053 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-23T11:07:32.054 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-23T11:07:32.055 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-23T11:07:32.056 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-23T11:07:32.057 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-23T11:07:32.058 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-23T11:07:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-23T11:07:32.060 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-23T11:07:32.061 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-23T11:07:32.062 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-23T11:07:32.063 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-23T11:07:32.064 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-23T11:07:32.065 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-23T11:07:32.066 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-23T11:07:32.067 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-23T11:07:32.068 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-23T11:07:32.069 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-23T11:07:32.070 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-23T11:07:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-23T11:07:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-23T11:07:32.073 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-23T11:07:32.074 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-23T11:07:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-23T11:07:32.076 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-23T11:07:32.077 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-23T11:07:32.078 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-23T11:07:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-23T11:07:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-23T11:07:32.081 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-23T11:07:32.082 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-23T11:07:32.083 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-23T11:07:32.084 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-23T11:07:32.085 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-23T11:07:32.086 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-23T11:07:32.087 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-23T11:07:32.088 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-23T11:07:32.089 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-23T11:07:32.090 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-23T11:07:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-23T11:07:32.092 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-23T11:07:32.093 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-23T11:07:32.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-23T11:07:32.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-23T11:07:32.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-23T11:07:32.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-23T11:07:32.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-23T11:07:32.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-23T11:07:32.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-23T11:07:32.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-23T11:07:32.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-23T11:07:32.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-23T11:07:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-23T11:07:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-23T11:07:32.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-23T11:07:32.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-23T11:07:32.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-23T11:07:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-23T11:07:32.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-23T11:07:32.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-23T11:07:32.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-23T11:07:32.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-23T11:07:32.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-23T11:07:32.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-23T11:07:32.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-23T11:07:32.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-23T11:07:32.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-23T11:07:32.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-23T11:07:32.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-23T11:07:32.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-23T11:07:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-23T11:07:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-23T11:07:32.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-23T11:07:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-23T11:07:32.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-23T11:07:32.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-23T11:07:32.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-23T11:07:32.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-23T11:07:32.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-23T11:07:32.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-23T11:07:32.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-23T11:07:32.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-23T11:07:32.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-23T11:07:32.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-23T11:07:32.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-23T11:07:32.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-23T11:07:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-23T11:07:32.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-23T11:07:32.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-23T11:07:32.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-23T11:07:32.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-23T11:07:32.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-23T11:07:32.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-23T11:07:32.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-23T11:07:32.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-23T11:07:32.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-23T11:07:32.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-23T11:07:32.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-23T11:07:32.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-23T11:07:32.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-23T11:07:32.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-23T11:07:32.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-23T11:07:32.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-23T11:07:32.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-23T11:07:32.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-23T11:07:32.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-23T11:07:32.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-23T11:07:32.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-23T11:07:32.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-23T11:07:32.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-23T11:07:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-23T11:07:32.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-23T11:07:32.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-23T11:07:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-23T11:07:32.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-23T11:07:32.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-23T11:07:32.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-23T11:07:32.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-23T11:07:32.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-23T11:07:32.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-23T11:07:32.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-23T11:07:32.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-23T11:07:32.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-23T11:07:32.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-23T11:07:32.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-23T11:07:32.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-23T11:07:32.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-23T11:07:32.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-23T11:07:32.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-23T11:07:32.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-23T11:07:32.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-23T11:07:32.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T11:07:32.176 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:32.176 DEBUG:teuthology.orchestra.run.smithi062:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-23T11:07:32.199 INFO:teuthology.orchestra.run.smithi062.stdout:postinstall scriptlet (using /bin/sh): 2024-03-23T11:07:32.199 INFO:teuthology.orchestra.run.smithi062.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:fi 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:for file in vmlinuz System.map config; do 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout: fi 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:done 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:32.200 INFO:teuthology.orchestra.run.smithi062.stdout:fi 2024-03-23T11:07:32.201 INFO:teuthology.orchestra.run.smithi062.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-23T11:07:32.201 INFO:teuthology.orchestra.run.smithi062.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-23T11:07:32.201 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:32.201 INFO:teuthology.orchestra.run.smithi062.stdout:fi 2024-03-23T11:07:32.202 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -qi grub2-tools 2024-03-23T11:07:32.281 INFO:teuthology.orchestra.run.smithi062.stdout:Name : grub2-tools 2024-03-23T11:07:32.281 INFO:teuthology.orchestra.run.smithi062.stdout:Epoch : 1 2024-03-23T11:07:32.281 INFO:teuthology.orchestra.run.smithi062.stdout:Version : 2.06 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:Release : 68.el9 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: x86_64 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:Group : Unspecified 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:Size : 8264254 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:License : GPLv3+ 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:Packager : builder@centos.org 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:Vendor : CentOS 2024-03-23T11:07:32.282 INFO:teuthology.orchestra.run.smithi062.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T11:07:32.283 INFO:teuthology.orchestra.run.smithi062.stdout:Summary : Support tools for GRUB. 2024-03-23T11:07:32.283 INFO:teuthology.orchestra.run.smithi062.stdout:Description : 2024-03-23T11:07:32.283 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:07:32.283 INFO:teuthology.orchestra.run.smithi062.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T11:07:32.283 INFO:teuthology.orchestra.run.smithi062.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T11:07:32.283 INFO:teuthology.orchestra.run.smithi062.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T11:07:32.283 INFO:teuthology.orchestra.run.smithi062.stdout:hardware devices. 2024-03-23T11:07:32.283 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:07:32.283 INFO:teuthology.orchestra.run.smithi062.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T11:07:32.284 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T11:07:32.284 INFO:teuthology.task.kernel:Updating grub on smithi062 to boot 6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:32.284 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T11:07:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2024-03-23T11:07:34.397 INFO:teuthology.orchestra.run.smithi062.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-23T11:07:34.431 INFO:teuthology.orchestra.run.smithi062.stderr:done 2024-03-23T11:07:34.434 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-03-23T11:07:34.489 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.QktVTG8i2h 2024-03-23T11:07:34.490 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QktVTG8i2h 2024-03-23T11:07:34.572 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.QktVTG8i2h 2024-03-23T11:07:34.714 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.QktVTG8i2h is 6KB 2024-03-23T11:07:34.762 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.QktVTG8i2h 2024-03-23T11:07:34.776 DEBUG:teuthology.orchestra.run.smithi062:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T11:07:34.845 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-23T11:07:34.845 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-23T11:07:34.845 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-23T11:07:34.845 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-23T11:07:34.846 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:34.997 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2024-03-23T11:07:42.692 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi120.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-23T11:07:42.692 DEBUG:teuthology.orchestra.run.smithi120:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-23T11:07:42.763 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:42.764 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-23T11:07:42.764 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-23T11:07:42.764 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-23T11:07:42.764 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-23T11:07:42.764 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-23T11:07:42.764 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-23T11:07:42.764 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-23T11:07:42.765 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-23T11:07:42.765 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-23T11:07:42.765 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-23T11:07:42.765 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-23T11:07:42.765 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-23T11:07:42.765 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-23T11:07:42.765 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-23T11:07:42.766 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-23T11:07:42.766 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-23T11:07:42.766 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-23T11:07:42.766 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-23T11:07:42.766 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-23T11:07:42.766 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-23T11:07:42.766 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-23T11:07:42.766 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-23T11:07:42.767 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-23T11:07:42.767 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-23T11:07:42.767 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-23T11:07:42.767 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-23T11:07:42.767 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-23T11:07:42.767 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-23T11:07:42.767 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-23T11:07:42.767 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-23T11:07:42.767 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-23T11:07:42.767 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-23T11:07:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-23T11:07:42.769 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-23T11:07:42.770 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-23T11:07:42.771 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-23T11:07:42.772 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-23T11:07:42.772 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-23T11:07:42.772 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-23T11:07:42.772 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-23T11:07:42.772 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-23T11:07:42.772 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-23T11:07:42.772 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-23T11:07:42.772 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-23T11:07:42.772 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-23T11:07:42.772 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-23T11:07:42.772 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-23T11:07:42.773 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-23T11:07:42.773 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-23T11:07:42.773 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-23T11:07:42.773 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-23T11:07:42.773 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-23T11:07:42.773 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-23T11:07:42.773 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-23T11:07:42.773 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-23T11:07:42.773 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-23T11:07:42.773 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-23T11:07:42.773 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-23T11:07:42.774 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-23T11:07:42.775 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-23T11:07:42.776 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-23T11:07:42.777 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-23T11:07:42.777 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-23T11:07:42.777 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-23T11:07:42.777 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-23T11:07:42.777 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-23T11:07:42.777 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-23T11:07:42.777 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-23T11:07:42.777 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-23T11:07:42.778 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-23T11:07:42.779 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-23T11:07:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-23T11:07:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-23T11:07:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-23T11:07:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-23T11:07:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-23T11:07:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-23T11:07:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-23T11:07:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-23T11:07:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-23T11:07:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-23T11:07:42.780 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-23T11:07:42.781 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-23T11:07:42.781 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-23T11:07:42.781 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-23T11:07:42.781 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-23T11:07:42.781 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-23T11:07:42.781 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-23T11:07:42.781 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-23T11:07:42.781 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-23T11:07:42.782 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-23T11:07:42.783 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-23T11:07:42.783 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-23T11:07:42.783 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-23T11:07:42.783 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-23T11:07:42.783 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-23T11:07:42.783 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-23T11:07:42.783 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-23T11:07:42.783 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-23T11:07:42.783 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-23T11:07:42.783 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-23T11:07:42.783 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-23T11:07:42.784 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-23T11:07:42.785 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-23T11:07:42.785 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-23T11:07:42.785 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-23T11:07:42.785 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-23T11:07:42.785 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-23T11:07:42.785 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-23T11:07:42.785 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-23T11:07:42.785 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-23T11:07:42.785 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-23T11:07:42.785 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-23T11:07:42.785 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-23T11:07:42.786 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-23T11:07:42.787 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-23T11:07:42.787 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-23T11:07:42.787 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-23T11:07:42.787 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-23T11:07:42.787 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-23T11:07:42.787 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-23T11:07:42.787 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-23T11:07:42.787 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-23T11:07:42.787 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-23T11:07:42.787 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-23T11:07:42.788 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-23T11:07:42.789 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-23T11:07:42.789 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-23T11:07:42.789 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-23T11:07:42.789 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-23T11:07:42.789 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-23T11:07:42.789 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-23T11:07:42.789 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-23T11:07:42.789 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-23T11:07:42.790 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-23T11:07:42.790 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-23T11:07:42.790 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-23T11:07:42.790 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-23T11:07:42.790 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-23T11:07:42.790 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-23T11:07:42.791 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-23T11:07:42.791 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-23T11:07:42.791 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-23T11:07:42.791 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-23T11:07:42.791 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-23T11:07:42.791 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-23T11:07:42.791 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-23T11:07:42.791 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-23T11:07:42.791 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-23T11:07:42.791 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-23T11:07:42.791 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-23T11:07:42.792 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-23T11:07:42.793 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-23T11:07:42.794 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-23T11:07:42.795 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-23T11:07:42.796 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-23T11:07:42.797 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-23T11:07:42.798 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-23T11:07:42.798 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-23T11:07:42.798 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-23T11:07:42.798 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-23T11:07:42.798 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-23T11:07:42.798 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-23T11:07:42.798 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-23T11:07:42.798 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-23T11:07:42.799 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-23T11:07:42.800 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-23T11:07:42.801 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-23T11:07:42.802 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-23T11:07:42.803 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-23T11:07:42.804 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-23T11:07:42.805 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-23T11:07:42.806 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-23T11:07:42.807 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-23T11:07:42.808 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-23T11:07:42.808 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-23T11:07:42.808 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-23T11:07:42.808 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-23T11:07:42.808 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-23T11:07:42.808 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-23T11:07:42.808 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-23T11:07:42.808 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-23T11:07:42.808 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-23T11:07:42.808 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-23T11:07:42.808 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-23T11:07:42.809 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-23T11:07:42.810 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-23T11:07:42.811 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-23T11:07:42.812 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-23T11:07:42.813 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-23T11:07:42.814 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-23T11:07:42.815 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-23T11:07:42.816 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-23T11:07:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-23T11:07:42.818 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-23T11:07:42.819 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-23T11:07:42.819 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-23T11:07:42.819 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-23T11:07:42.819 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-23T11:07:42.819 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-23T11:07:42.819 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-23T11:07:42.819 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-23T11:07:42.819 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-23T11:07:42.819 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-23T11:07:42.819 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-23T11:07:42.820 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-23T11:07:42.821 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-23T11:07:42.822 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-23T11:07:42.823 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-23T11:07:42.824 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-23T11:07:42.825 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-23T11:07:42.826 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-23T11:07:42.827 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-23T11:07:42.828 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-23T11:07:42.829 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-23T11:07:42.829 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-23T11:07:42.829 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-23T11:07:42.829 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-23T11:07:42.829 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-23T11:07:42.829 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-23T11:07:42.829 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-23T11:07:42.829 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-23T11:07:42.829 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-23T11:07:42.829 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-23T11:07:42.830 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-23T11:07:42.831 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-23T11:07:42.832 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-23T11:07:42.833 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-23T11:07:42.834 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-23T11:07:42.835 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-23T11:07:42.836 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-23T11:07:42.837 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-23T11:07:42.838 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-23T11:07:42.839 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-23T11:07:42.840 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-23T11:07:42.840 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-23T11:07:42.840 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-23T11:07:42.840 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-23T11:07:42.840 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-23T11:07:42.840 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-23T11:07:42.840 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-23T11:07:42.840 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-23T11:07:42.840 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-23T11:07:42.840 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-23T11:07:42.841 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-23T11:07:42.841 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-23T11:07:42.841 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-23T11:07:42.841 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-23T11:07:42.841 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-23T11:07:42.841 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-23T11:07:42.841 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-23T11:07:42.841 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-23T11:07:42.841 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-23T11:07:42.841 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-23T11:07:42.841 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-23T11:07:42.842 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-23T11:07:42.842 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-23T11:07:42.842 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-23T11:07:42.842 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-23T11:07:42.842 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-23T11:07:42.843 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-23T11:07:42.843 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-23T11:07:42.843 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-23T11:07:42.843 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-23T11:07:42.843 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-23T11:07:42.843 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-23T11:07:42.843 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-23T11:07:42.844 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-23T11:07:42.844 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-23T11:07:42.844 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-23T11:07:42.844 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-23T11:07:42.844 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-23T11:07:42.844 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-23T11:07:42.844 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-23T11:07:42.845 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-23T11:07:42.845 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-23T11:07:42.845 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-23T11:07:42.845 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-23T11:07:42.845 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-23T11:07:42.845 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-23T11:07:42.845 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-23T11:07:42.845 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-23T11:07:42.845 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-23T11:07:42.845 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-23T11:07:42.846 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-23T11:07:42.846 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-23T11:07:42.846 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-23T11:07:42.846 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-23T11:07:42.846 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-23T11:07:42.846 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-23T11:07:42.846 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-23T11:07:42.846 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-23T11:07:42.846 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-23T11:07:42.846 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-23T11:07:42.846 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-23T11:07:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-23T11:07:42.848 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-23T11:07:42.849 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-23T11:07:42.850 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-23T11:07:42.850 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-23T11:07:42.850 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-23T11:07:42.850 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-23T11:07:42.850 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-23T11:07:42.850 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-23T11:07:42.850 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-23T11:07:42.850 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-23T11:07:42.850 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-23T11:07:42.851 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-23T11:07:42.852 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-23T11:07:42.853 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-23T11:07:42.853 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-23T11:07:42.853 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-23T11:07:42.853 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-23T11:07:42.853 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-23T11:07:42.853 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-23T11:07:42.853 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-23T11:07:42.853 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-23T11:07:42.853 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-23T11:07:42.853 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-23T11:07:42.853 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-23T11:07:42.854 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-23T11:07:42.854 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-23T11:07:42.854 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-23T11:07:42.854 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-23T11:07:42.854 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-23T11:07:42.854 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-23T11:07:42.854 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-23T11:07:42.854 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-23T11:07:42.854 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-23T11:07:42.854 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-23T11:07:42.854 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-23T11:07:42.855 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-23T11:07:42.856 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-23T11:07:42.857 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-23T11:07:42.858 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-23T11:07:42.859 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-23T11:07:42.860 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-23T11:07:42.861 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-23T11:07:42.861 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-23T11:07:42.861 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-23T11:07:42.861 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-23T11:07:42.861 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-23T11:07:42.861 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-23T11:07:42.861 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-23T11:07:42.861 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-23T11:07:42.861 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-23T11:07:42.861 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-23T11:07:42.861 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-23T11:07:42.862 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-23T11:07:42.863 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-23T11:07:42.864 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-23T11:07:42.865 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-23T11:07:42.866 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-23T11:07:42.867 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-23T11:07:42.868 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-23T11:07:42.869 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-23T11:07:42.870 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-23T11:07:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-23T11:07:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-23T11:07:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-23T11:07:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-23T11:07:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-23T11:07:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-23T11:07:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-23T11:07:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-23T11:07:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-23T11:07:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-23T11:07:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-23T11:07:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-23T11:07:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-23T11:07:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-23T11:07:42.875 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-23T11:07:42.876 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-23T11:07:42.877 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-23T11:07:42.878 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-23T11:07:42.878 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-23T11:07:42.878 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-23T11:07:42.878 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-23T11:07:42.878 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-23T11:07:42.878 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-23T11:07:42.878 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-23T11:07:42.878 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-23T11:07:42.878 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-23T11:07:42.878 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-23T11:07:42.878 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-23T11:07:42.879 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-23T11:07:42.880 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-23T11:07:42.881 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-23T11:07:42.881 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-23T11:07:42.881 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-23T11:07:42.881 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-23T11:07:42.881 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-23T11:07:42.881 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-23T11:07:42.881 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-23T11:07:42.881 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-23T11:07:42.882 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-23T11:07:42.882 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-23T11:07:42.882 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-23T11:07:42.882 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-23T11:07:42.882 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-23T11:07:42.882 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-23T11:07:42.882 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-23T11:07:42.882 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-23T11:07:42.882 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-23T11:07:42.882 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-23T11:07:42.883 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-23T11:07:42.884 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-23T11:07:42.885 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-23T11:07:42.886 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-23T11:07:42.887 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-23T11:07:42.888 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-23T11:07:42.889 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-23T11:07:42.890 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-23T11:07:42.891 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-23T11:07:42.892 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-23T11:07:42.893 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-23T11:07:42.894 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-23T11:07:42.895 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-23T11:07:42.896 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-23T11:07:42.897 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-23T11:07:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-23T11:07:42.899 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-23T11:07:42.900 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-23T11:07:42.901 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-23T11:07:42.902 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-23T11:07:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-23T11:07:42.904 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-23T11:07:42.905 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-23T11:07:42.906 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-23T11:07:42.907 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-23T11:07:42.908 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-23T11:07:42.909 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-23T11:07:42.910 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-23T11:07:42.911 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-23T11:07:42.912 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-23T11:07:42.913 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-23T11:07:42.913 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-23T11:07:42.913 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-23T11:07:42.913 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-23T11:07:42.913 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-23T11:07:42.913 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T11:07:42.916 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:42.916 DEBUG:teuthology.orchestra.run.smithi120:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-23T11:07:42.939 INFO:teuthology.orchestra.run.smithi120.stdout:postinstall scriptlet (using /bin/sh): 2024-03-23T11:07:42.939 INFO:teuthology.orchestra.run.smithi120.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-23T11:07:42.939 INFO:teuthology.orchestra.run.smithi120.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T11:07:42.939 INFO:teuthology.orchestra.run.smithi120.stdout:fi 2024-03-23T11:07:42.939 INFO:teuthology.orchestra.run.smithi120.stdout:for file in vmlinuz System.map config; do 2024-03-23T11:07:42.939 INFO:teuthology.orchestra.run.smithi120.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-23T11:07:42.939 INFO:teuthology.orchestra.run.smithi120.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout: fi 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout:done 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout:fi 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:42.940 INFO:teuthology.orchestra.run.smithi120.stdout:fi 2024-03-23T11:07:42.941 DEBUG:teuthology.orchestra.run.smithi120:> sudo rpm -qi grub2-tools 2024-03-23T11:07:43.018 INFO:teuthology.orchestra.run.smithi120.stdout:Name : grub2-tools 2024-03-23T11:07:43.018 INFO:teuthology.orchestra.run.smithi120.stdout:Epoch : 1 2024-03-23T11:07:43.018 INFO:teuthology.orchestra.run.smithi120.stdout:Version : 2.06 2024-03-23T11:07:43.018 INFO:teuthology.orchestra.run.smithi120.stdout:Release : 68.el9 2024-03-23T11:07:43.018 INFO:teuthology.orchestra.run.smithi120.stdout:Architecture: x86_64 2024-03-23T11:07:43.018 INFO:teuthology.orchestra.run.smithi120.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-23T11:07:43.018 INFO:teuthology.orchestra.run.smithi120.stdout:Group : Unspecified 2024-03-23T11:07:43.018 INFO:teuthology.orchestra.run.smithi120.stdout:Size : 8264254 2024-03-23T11:07:43.018 INFO:teuthology.orchestra.run.smithi120.stdout:License : GPLv3+ 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:Packager : builder@centos.org 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:Vendor : CentOS 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:Summary : Support tools for GRUB. 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:Description : 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T11:07:43.019 INFO:teuthology.orchestra.run.smithi120.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T11:07:43.020 INFO:teuthology.orchestra.run.smithi120.stdout:hardware devices. 2024-03-23T11:07:43.020 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:07:43.020 INFO:teuthology.orchestra.run.smithi120.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T11:07:43.021 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T11:07:43.021 INFO:teuthology.task.kernel:Updating grub on smithi120 to boot 6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:43.021 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T11:07:43.671 INFO:teuthology.orchestra.run.smithi120.stderr:Generating grub configuration file ... 2024-03-23T11:07:45.081 INFO:teuthology.orchestra.run.smithi120.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-23T11:07:45.112 INFO:teuthology.orchestra.run.smithi120.stderr:done 2024-03-23T11:07:45.114 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-03-23T11:07:45.169 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.oXWqlOwqE3 2024-03-23T11:07:45.169 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oXWqlOwqE3 2024-03-23T11:07:45.244 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.oXWqlOwqE3 2024-03-23T11:07:45.390 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.oXWqlOwqE3 is 6KB 2024-03-23T11:07:45.439 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.oXWqlOwqE3 2024-03-23T11:07:45.452 DEBUG:teuthology.orchestra.run.smithi120:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T11:07:45.517 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-23T11:07:45.517 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-23T11:07:45.517 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-23T11:07:45.517 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-23T11:07:45.518 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-23T11:07:45.687 DEBUG:teuthology.orchestra.run.smithi120:> sudo shutdown -r now 2024-03-23T11:08:05.020 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-23T11:08:05.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-23T11:08:05.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:08:15.698 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-23T11:08:15.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-23T11:08:15.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:08:23.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-03-23T11:08:32.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-23T11:08:32.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:08:34.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2024-03-23T11:08:35.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-03-23T11:08:43.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-23T11:08:43.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:08:46.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2024-03-23T11:08:47.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-23T11:08:47.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:08:58.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-23T11:08:58.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:09:19.395 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-03-23T11:09:20.030 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-23T11:09:20.030 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.8.0-g5ec3e6e88b2e"... 2024-03-23T11:09:20.030 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-03-23T11:09:20.071 INFO:teuthology.orchestra.run.smithi062.stdout:6.8.0-g5ec3e6e88b2e 2024-03-23T11:09:20.072 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-23T11:09:20.072 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T11:09:20.072 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T11:09:21.072 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-23T11:09:21.073 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-23T11:09:21.211 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2024-03-23T11:09:21.262 DEBUG:teuthology.parallel:result is None 2024-03-23T11:09:30.264 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-03-23T11:09:30.848 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-23T11:09:30.848 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.8.0-g5ec3e6e88b2e"... 2024-03-23T11:09:30.848 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2024-03-23T11:09:30.888 INFO:teuthology.orchestra.run.smithi120.stdout:6.8.0-g5ec3e6e88b2e 2024-03-23T11:09:30.888 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-23T11:09:30.888 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T11:09:30.888 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T11:09:31.889 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-23T11:09:31.889 DEBUG:teuthology.orchestra.run.smithi120:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-23T11:09:32.004 INFO:teuthology.orchestra.run.smithi120.stdout:ttyS1 2024-03-23T11:09:32.045 DEBUG:teuthology.parallel:result is None 2024-03-23T11:09:32.046 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-23T11:09:32.054 INFO:teuthology.task.internal:Creating test directory... 2024-03-23T11:09:32.054 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T11:09:32.066 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T11:09:32.144 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-23T11:09:32.151 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-23T11:09:32.158 INFO:teuthology.task.internal:Creating archive directory... 2024-03-23T11:09:32.158 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T11:09:32.167 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T11:09:32.252 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-23T11:09:32.258 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-23T11:09:32.259 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T11:09:32.271 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T11:09:32.359 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T11:09:32.394 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T11:09:32.412 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T11:09:32.473 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T11:09:32.479 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-23T11:09:32.487 INFO:teuthology.task.internal:Configuring sudo... 2024-03-23T11:09:32.487 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T11:09:32.494 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T11:09:32.615 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-23T11:09:32.625 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-23T11:09:32.625 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T11:09:32.635 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T11:09:32.699 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T11:09:32.797 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T11:09:32.937 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T11:09:33.054 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T11:09:33.194 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:09:33.194 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T11:09:33.344 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T11:09:33.415 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T11:09:33.496 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T11:09:33.571 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T11:09:33.649 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:09:33.649 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T11:09:33.716 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2024-03-23T11:09:33.720 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2024-03-23T11:09:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T11:09:33.863 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T11:09:34.360 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-23T11:09:34.368 INFO:teuthology.task.internal:Starting timer... 2024-03-23T11:09:34.368 INFO:teuthology.run_tasks:Running task pcp... 2024-03-23T11:09:34.378 INFO:teuthology.run_tasks:Running task selinux... 2024-03-23T11:09:34.386 DEBUG:teuthology.orchestra.run.smithi062:> sudo service auditd rotate 2024-03-23T11:09:34.511 INFO:teuthology.orchestra.run.smithi062.stdout:Rotating logs: 2024-03-23T11:09:34.515 DEBUG:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2024-03-23T11:09:34.654 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: 2024-03-23T11:09:34.658 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-23T11:09:34.658 DEBUG:teuthology.orchestra.run.smithi062:> /usr/sbin/getenforce 2024-03-23T11:09:34.719 INFO:teuthology.orchestra.run.smithi062.stdout:Permissive 2024-03-23T11:09:34.720 DEBUG:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2024-03-23T11:09:34.776 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2024-03-23T11:09:34.776 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi062.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi120.front.sepia.ceph.com': 'permissive'} 2024-03-23T11:09:34.776 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T11:09:34.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:09:34.879 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T11:09:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:09:34.967 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-23T11:09:34.967 DEBUG:teuthology.orchestra.run.smithi062:> sudo /usr/sbin/setenforce permissive 2024-03-23T11:09:35.068 DEBUG:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2024-03-23T11:09:35.163 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-23T11:09:35.185 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-23T11:09:35.190 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-23T11:09:35.330 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-23T11:09:35.345 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-23T11:09:35.349 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi062.front.sepia.ceph.com,smithi120.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-23T11:15:45.990 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi120.front.sepia.ceph.com')] 2024-03-23T11:15:45.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-23T11:15:45.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:15:46.143 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-03-23T11:15:46.299 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-23T11:15:46.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-23T11:15:46.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T11:15:46.436 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-03-23T11:15:46.585 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-23T11:15:46.586 INFO:teuthology.run_tasks:Running task clock... 2024-03-23T11:15:46.595 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-23T11:15:46.596 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T11:15:46.596 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T11:15:46.604 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T11:15:46.604 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T11:15:46.701 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T11:15:46.704 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T11:15:46.751 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T11:15:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T11:15:46.844 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2024-03-23T11:15:46.861 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: ntpd: command not found 2024-03-23T11:15:46.884 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2024-03-23T11:15:46.906 INFO:teuthology.orchestra.run.smithi062.stdout:506 Cannot talk to daemon 2024-03-23T11:15:46.936 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T11:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T11:15:46.986 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T11:15:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T11:15:47.121 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-03-23T11:15:47.129 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T11:15:47.129 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-03-23T11:15:47.129 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:15:47.130 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:15:47.130 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:15:47.130 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:15:47.154 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-03-23T11:15:47.164 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T11:15:47.164 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-03-23T11:15:47.165 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:15:47.165 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:15:47.165 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:15:47.165 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:15:47.166 INFO:teuthology.run_tasks:Running task install... 2024-03-23T11:15:47.175 DEBUG:teuthology.task.install:project ceph 2024-03-23T11:15:47.175 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609'}} 2024-03-23T11:15:47.175 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609'} 2024-03-23T11:15:47.176 INFO:teuthology.task.install:Using flavor: default 2024-03-23T11:15:47.181 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-23T11:15:47.182 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-03-23T11:15:47.182 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609', 'tag': None, 'wait_for_package': False} 2024-03-23T11:15:47.183 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:15:47.184 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609', 'tag': None, 'wait_for_package': False} 2024-03-23T11:15:47.185 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:15:47.326 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-yuri-RunCephCmd-base-reef-2/b630d6cd702af63dfefcf906294a092ddce1c609/centos/9/flavors/default/ 2024-03-23T11:15:47.327 INFO:teuthology.task.install.rpm:Package version is 18.2.2-820.gb630d6cd 2024-03-23T11:15:47.397 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-yuri-RunCephCmd-base-reef-2/b630d6cd702af63dfefcf906294a092ddce1c609/centos/9/flavors/default/ 2024-03-23T11:15:47.397 INFO:teuthology.task.install.rpm:Package version is 18.2.2-820.gb630d6cd 2024-03-23T11:15:47.435 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-yuri-RunCephCmd-base-reef-2/b630d6cd702af63dfefcf906294a092ddce1c609/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-yuri-RunCephCmd-base-reef-2/b630d6cd702af63dfefcf906294a092ddce1c609/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-yuri-RunCephCmd-base-reef-2/b630d6cd702af63dfefcf906294a092ddce1c609/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T11:15:47.435 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:15:47.436 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T11:15:47.539 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-23T11:15:47.539 DEBUG:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b630d6cd702af63dfefcf906294a092ddce1c609/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T11:15:47.564 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-yuri-RunCephCmd-base-reef-2/b630d6cd702af63dfefcf906294a092ddce1c609/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-yuri-RunCephCmd-base-reef-2/b630d6cd702af63dfefcf906294a092ddce1c609/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-yuri-RunCephCmd-base-reef-2/b630d6cd702af63dfefcf906294a092ddce1c609/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T11:15:47.564 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:15:47.564 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T11:15:47.671 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-23T11:15:47.671 DEBUG:teuthology.orchestra.run.smithi062:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b630d6cd702af63dfefcf906294a092ddce1c609/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T11:15:47.676 DEBUG:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T11:15:47.801 DEBUG:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T11:15:47.812 DEBUG:teuthology.orchestra.run.smithi062:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T11:15:47.885 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2024-03-23T11:15:47.890 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum clean all 2024-03-23T11:15:47.944 DEBUG:teuthology.orchestra.run.smithi062:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T11:15:48.038 INFO:teuthology.orchestra.run.smithi062.stdout:check_obsoletes = 1 2024-03-23T11:15:48.043 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum clean all 2024-03-23T11:15:48.247 INFO:teuthology.orchestra.run.smithi120.stdout:74 files removed 2024-03-23T11:15:48.285 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-23T11:15:48.403 INFO:teuthology.orchestra.run.smithi062.stdout:74 files removed 2024-03-23T11:15:48.446 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-23T11:15:49.161 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 204 kB/s | 77 kB 00:00 2024-03-23T11:15:49.433 INFO:teuthology.orchestra.run.smithi062.stdout:ceph packages for x86_64 165 kB/s | 77 kB 00:00 2024-03-23T11:15:49.669 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 24 kB/s | 9.9 kB 00:00 2024-03-23T11:15:50.009 INFO:teuthology.orchestra.run.smithi062.stdout:ceph noarch packages 21 kB/s | 9.9 kB 00:00 2024-03-23T11:15:50.211 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-03-23T11:15:50.585 INFO:teuthology.orchestra.run.smithi062.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-23T11:15:51.173 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 7.4 MB/s | 6.7 MB 00:00 2024-03-23T11:15:51.205 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-23T11:15:53.033 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-23T11:15:53.047 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-23T11:15:54.290 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.0 MB 00:00 2024-03-23T11:15:56.228 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.0 MB 00:02 2024-03-23T11:15:57.044 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-23T11:15:58.570 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-23T11:16:02.503 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-03-23T11:16:03.288 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 6.1 MB 00:00 2024-03-23T11:16:04.112 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 99 kB/s | 16 kB 00:00 2024-03-23T11:16:04.905 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-23T11:16:06.131 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-23T11:16:06.669 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-23T11:16:12.181 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-23T11:16:12.323 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-23T11:16:14.894 INFO:teuthology.orchestra.run.smithi062.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T11:16:14.896 INFO:teuthology.orchestra.run.smithi062.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T11:16:14.916 INFO:teuthology.orchestra.run.smithi062.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-23T11:16:14.917 INFO:teuthology.orchestra.run.smithi062.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-23T11:16:14.919 INFO:teuthology.orchestra.run.smithi062.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-23T11:16:14.920 INFO:teuthology.orchestra.run.smithi062.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-23T11:16:14.923 INFO:teuthology.orchestra.run.smithi062.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-23T11:16:14.927 INFO:teuthology.orchestra.run.smithi062.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-23T11:16:14.928 INFO:teuthology.orchestra.run.smithi062.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-23T11:16:14.960 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T11:16:14.962 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T11:16:14.985 INFO:teuthology.orchestra.run.smithi120.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-23T11:16:14.986 INFO:teuthology.orchestra.run.smithi120.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-23T11:16:14.988 INFO:teuthology.orchestra.run.smithi120.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-23T11:16:14.989 INFO:teuthology.orchestra.run.smithi120.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-23T11:16:14.992 INFO:teuthology.orchestra.run.smithi120.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-23T11:16:14.996 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-23T11:16:14.998 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-23T11:16:15.020 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-23T11:16:15.042 INFO:teuthology.orchestra.run.smithi062.stdout:=========================================================================================== 2024-03-23T11:16:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-23T11:16:15.042 INFO:teuthology.orchestra.run.smithi062.stdout:=========================================================================================== 2024-03-23T11:16:15.042 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-23T11:16:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-23T11:16:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: ceph x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 6.5 k 2024-03-23T11:16:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 5.2 M 2024-03-23T11:16:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 841 k 2024-03-23T11:16:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 143 k 2024-03-23T11:16:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 1.4 M 2024-03-23T11:16:15.042 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 138 k 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 1.7 M 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 7.4 M 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 49 k 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 7.7 M 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 40 M 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 264 k 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 224 k 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-mirror x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 222 k 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-top noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 18 k 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 31 k 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2 x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 659 k 2024-03-23T11:16:15.043 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 127 k 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 161 k 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 320 k 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 298 k 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 99 k 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 87 k 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 3.0 M 2024-03-23T11:16:15.044 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 172 k 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrading: 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-23T11:16:15.045 INFO:teuthology.orchestra.run.smithi062.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-23T11:16:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-23T11:16:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-23T11:16:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: librados2 x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 3.3 M 2024-03-23T11:16:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1 x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 3.0 M 2024-03-23T11:16:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-23T11:16:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-23T11:16:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-23T11:16:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-23T11:16:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-23T11:16:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-23T11:16:15.046 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 18 M 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 23 k 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 2.1 M 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 245 k 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 4.5 M 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 18 M 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 15 k 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 25 k 2024-03-23T11:16:15.047 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 166 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-23T11:16:15.048 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1 x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 474 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2 x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 4.5 M 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-23T11:16:15.049 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 45 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 127 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-23T11:16:15.050 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-23T11:16:15.051 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-23T11:16:15.052 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:16:15.053 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-23T11:16:15.054 INFO:teuthology.orchestra.run.smithi062.stdout:=========================================================================================== 2024-03-23T11:16:15.054 INFO:teuthology.orchestra.run.smithi062.stdout:Install 132 Packages 2024-03-23T11:16:15.054 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrade 24 Packages 2024-03-23T11:16:15.054 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:16:15.054 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 205 M 2024-03-23T11:16:15.054 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-23T11:16:15.084 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout:=========================================================================================== 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout:=========================================================================================== 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 6.5 k 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 5.2 M 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 841 k 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 143 k 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 1.4 M 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 138 k 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 1.7 M 2024-03-23T11:16:15.107 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 7.4 M 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 49 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 7.7 M 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 40 M 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 264 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 224 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 222 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 18 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 31 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 659 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 127 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-23T11:16:15.108 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 161 k 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 320 k 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 298 k 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 99 k 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 87 k 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 3.0 M 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 172 k 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-23T11:16:15.109 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 3.3 M 2024-03-23T11:16:15.110 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 3.0 M 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 18 M 2024-03-23T11:16:15.111 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 23 k 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 2.1 M 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 245 k 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 4.5 M 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 18 M 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts noarch 2:18.2.2-820.gb630d6cd.el9 ceph-noarch 15 k 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 25 k 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-23T11:16:15.112 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 166 k 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 474 k 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 4.5 M 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-23T11:16:15.113 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 45 k 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common x86_64 2:18.2.2-820.gb630d6cd.el9 ceph 127 k 2024-03-23T11:16:15.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-23T11:16:15.115 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-23T11:16:15.116 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-23T11:16:15.117 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout:=========================================================================================== 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout:Install 132 Packages 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 24 Packages 2024-03-23T11:16:15.118 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:16:15.119 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 205 M 2024-03-23T11:16:15.119 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-23T11:16:15.965 INFO:teuthology.orchestra.run.smithi062.stdout:(1/156): ceph-18.2.2-820.gb630d6cd.el9.x86_64.r 24 kB/s | 6.5 kB 00:00 2024-03-23T11:16:16.157 INFO:teuthology.orchestra.run.smithi062.stdout:(2/156): ceph-fuse-18.2.2-820.gb630d6cd.el9.x86 4.3 MB/s | 841 kB 00:00 2024-03-23T11:16:16.207 INFO:teuthology.orchestra.run.smithi062.stdout:(3/156): ceph-immutable-object-cache-18.2.2-820 2.9 MB/s | 143 kB 00:00 2024-03-23T11:16:16.250 INFO:teuthology.orchestra.run.smithi120.stdout:(1/156): ceph-18.2.2-820.gb630d6cd.el9.x86_64.r 25 kB/s | 6.5 kB 00:00 2024-03-23T11:16:16.574 INFO:teuthology.orchestra.run.smithi062.stdout:(4/156): ceph-mds-18.2.2-820.gb630d6cd.el9.x86_ 5.8 MB/s | 2.1 MB 00:00 2024-03-23T11:16:16.691 INFO:teuthology.orchestra.run.smithi062.stdout:(5/156): ceph-base-18.2.2-820.gb630d6cd.el9.x86 5.2 MB/s | 5.2 MB 00:00 2024-03-23T11:16:16.783 INFO:teuthology.orchestra.run.smithi062.stdout:(6/156): ceph-mgr-18.2.2-820.gb630d6cd.el9.x86_ 7.0 MB/s | 1.4 MB 00:00 2024-03-23T11:16:16.799 INFO:teuthology.orchestra.run.smithi120.stdout:(2/156): ceph-fuse-18.2.2-820.gb630d6cd.el9.x86 1.5 MB/s | 841 kB 00:00 2024-03-23T11:16:16.950 INFO:teuthology.orchestra.run.smithi120.stdout:(3/156): ceph-immutable-object-cache-18.2.2-820 952 kB/s | 143 kB 00:00 2024-03-23T11:16:17.317 INFO:teuthology.orchestra.run.smithi062.stdout:(7/156): ceph-mon-18.2.2-820.gb630d6cd.el9.x86_ 7.2 MB/s | 4.5 MB 00:00 2024-03-23T11:16:18.050 INFO:teuthology.orchestra.run.smithi120.stdout:(4/156): ceph-mds-18.2.2-820.gb630d6cd.el9.x86_ 1.9 MB/s | 2.1 MB 00:01 2024-03-23T11:16:18.184 INFO:teuthology.orchestra.run.smithi062.stdout:(8/156): ceph-common-18.2.2-820.gb630d6cd.el9.x 7.4 MB/s | 18 MB 00:02 2024-03-23T11:16:18.259 INFO:teuthology.orchestra.run.smithi062.stdout:(9/156): ceph-selinux-18.2.2-820.gb630d6cd.el9. 339 kB/s | 25 kB 00:00 2024-03-23T11:16:18.400 INFO:teuthology.orchestra.run.smithi120.stdout:(5/156): ceph-mgr-18.2.2-820.gb630d6cd.el9.x86_ 4.2 MB/s | 1.4 MB 00:00 2024-03-23T11:16:18.401 INFO:teuthology.orchestra.run.smithi062.stdout:(10/156): ceph-radosgw-18.2.2-820.gb630d6cd.el9 7.1 MB/s | 7.7 MB 00:01 2024-03-23T11:16:18.535 INFO:teuthology.orchestra.run.smithi062.stdout:(11/156): cephfs-mirror-18.2.2-820.gb630d6cd.el 1.6 MB/s | 222 kB 00:00 2024-03-23T11:16:18.635 INFO:teuthology.orchestra.run.smithi062.stdout:(12/156): libcephfs-devel-18.2.2-820.gb630d6cd. 314 kB/s | 31 kB 00:00 2024-03-23T11:16:18.844 INFO:teuthology.orchestra.run.smithi062.stdout:(13/156): libcephfs2-18.2.2-820.gb630d6cd.el9.x 3.1 MB/s | 659 kB 00:00 2024-03-23T11:16:18.944 INFO:teuthology.orchestra.run.smithi062.stdout:(14/156): libcephsqlite-18.2.2-820.gb630d6cd.el 1.6 MB/s | 166 kB 00:00 2024-03-23T11:16:19.052 INFO:teuthology.orchestra.run.smithi062.stdout:(15/156): librados-devel-18.2.2-820.gb630d6cd.e 1.2 MB/s | 127 kB 00:00 2024-03-23T11:16:19.303 INFO:teuthology.orchestra.run.smithi062.stdout:(16/156): ceph-osd-18.2.2-820.gb630d6cd.el9.x86 7.0 MB/s | 18 MB 00:02 2024-03-23T11:16:19.345 INFO:teuthology.orchestra.run.smithi062.stdout:(17/156): libradosstriper1-18.2.2-820.gb630d6cd 1.6 MB/s | 474 kB 00:00 2024-03-23T11:16:19.445 INFO:teuthology.orchestra.run.smithi062.stdout:(18/156): python3-ceph-argparse-18.2.2-820.gb63 454 kB/s | 45 kB 00:00 2024-03-23T11:16:19.562 INFO:teuthology.orchestra.run.smithi062.stdout:(19/156): python3-ceph-common-18.2.2-820.gb630d 1.1 MB/s | 127 kB 00:00 2024-03-23T11:16:19.687 INFO:teuthology.orchestra.run.smithi062.stdout:(20/156): python3-cephfs-18.2.2-820.gb630d6cd.e 1.3 MB/s | 161 kB 00:00 2024-03-23T11:16:19.825 INFO:teuthology.orchestra.run.smithi120.stdout:(6/156): ceph-base-18.2.2-820.gb630d6cd.el9.x86 1.4 MB/s | 5.2 MB 00:03 2024-03-23T11:16:19.836 INFO:teuthology.orchestra.run.smithi062.stdout:(21/156): python3-rados-18.2.2-820.gb630d6cd.el 2.1 MB/s | 320 kB 00:00 2024-03-23T11:16:19.946 INFO:teuthology.orchestra.run.smithi062.stdout:(22/156): python3-rbd-18.2.2-820.gb630d6cd.el9. 2.7 MB/s | 298 kB 00:00 2024-03-23T11:16:20.096 INFO:teuthology.orchestra.run.smithi062.stdout:(23/156): librgw2-18.2.2-820.gb630d6cd.el9.x86_ 5.6 MB/s | 4.5 MB 00:00 2024-03-23T11:16:20.130 INFO:teuthology.orchestra.run.smithi062.stdout:(24/156): python3-rgw-18.2.2-820.gb630d6cd.el9. 544 kB/s | 99 kB 00:00 2024-03-23T11:16:20.151 INFO:teuthology.orchestra.run.smithi120.stdout:(7/156): ceph-mon-18.2.2-820.gb630d6cd.el9.x86_ 2.6 MB/s | 4.5 MB 00:01 2024-03-23T11:16:20.155 INFO:teuthology.orchestra.run.smithi062.stdout:(25/156): rbd-fuse-18.2.2-820.gb630d6cd.el9.x86 1.5 MB/s | 87 kB 00:00 2024-03-23T11:16:20.264 INFO:teuthology.orchestra.run.smithi062.stdout:(26/156): rbd-nbd-18.2.2-820.gb630d6cd.el9.x86_ 1.6 MB/s | 172 kB 00:00 2024-03-23T11:16:20.356 INFO:teuthology.orchestra.run.smithi062.stdout:(27/156): ceph-grafana-dashboards-18.2.2-820.gb 258 kB/s | 23 kB 00:00 2024-03-23T11:16:20.464 INFO:teuthology.orchestra.run.smithi062.stdout:(28/156): ceph-mgr-cephadm-18.2.2-820.gb630d6cd 1.3 MB/s | 138 kB 00:00 2024-03-23T11:16:20.556 INFO:teuthology.orchestra.run.smithi062.stdout:(29/156): rbd-mirror-18.2.2-820.gb630d6cd.el9.x 7.0 MB/s | 3.0 MB 00:00 2024-03-23T11:16:20.890 INFO:teuthology.orchestra.run.smithi062.stdout:(30/156): ceph-mgr-dashboard-18.2.2-820.gb630d6 4.1 MB/s | 1.7 MB 00:00 2024-03-23T11:16:20.974 INFO:teuthology.orchestra.run.smithi062.stdout:(31/156): ceph-mgr-modules-core-18.2.2-820.gb63 2.9 MB/s | 245 kB 00:00 2024-03-23T11:16:21.033 INFO:teuthology.orchestra.run.smithi062.stdout:(32/156): ceph-mgr-rook-18.2.2-820.gb630d6cd.el 861 kB/s | 49 kB 00:00 2024-03-23T11:16:21.083 INFO:teuthology.orchestra.run.smithi062.stdout:(33/156): ceph-prometheus-alerts-18.2.2-820.gb6 303 kB/s | 15 kB 00:00 2024-03-23T11:16:21.243 INFO:teuthology.orchestra.run.smithi120.stdout:(8/156): ceph-common-18.2.2-820.gb630d6cd.el9.x 3.5 MB/s | 18 MB 00:05 2024-03-23T11:16:21.257 INFO:teuthology.orchestra.run.smithi062.stdout:(34/156): ceph-mgr-diskprediction-local-18.2.2- 11 MB/s | 7.4 MB 00:00 2024-03-23T11:16:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:(35/156): ceph-volume-18.2.2-820.gb630d6cd.el9. 1.3 MB/s | 264 kB 00:00 2024-03-23T11:16:21.309 INFO:teuthology.orchestra.run.smithi120.stdout:(9/156): ceph-selinux-18.2.2-820.gb630d6cd.el9. 384 kB/s | 25 kB 00:00 2024-03-23T11:16:21.325 INFO:teuthology.orchestra.run.smithi062.stdout:(36/156): cephadm-18.2.2-820.gb630d6cd.el9.noar 3.3 MB/s | 224 kB 00:00 2024-03-23T11:16:21.342 INFO:teuthology.orchestra.run.smithi062.stdout:(37/156): cephfs-top-18.2.2-820.gb630d6cd.el9.n 313 kB/s | 18 kB 00:00 2024-03-23T11:16:21.567 INFO:teuthology.orchestra.run.smithi062.stdout:(38/156): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 662 kB 00:00 2024-03-23T11:16:21.609 INFO:teuthology.orchestra.run.smithi062.stdout:(39/156): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 920 kB 00:00 2024-03-23T11:16:21.635 INFO:teuthology.orchestra.run.smithi062.stdout:(40/156): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-23T11:16:21.668 INFO:teuthology.orchestra.run.smithi062.stdout:(41/156): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-03-23T11:16:21.694 INFO:teuthology.orchestra.run.smithi062.stdout:(42/156): flexiblas-3.0.4-8.el9.x86_64.rpm 575 kB/s | 32 kB 00:00 2024-03-23T11:16:21.794 INFO:teuthology.orchestra.run.smithi062.stdout:(43/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-23T11:16:21.819 INFO:teuthology.orchestra.run.smithi062.stdout:(44/156): flexiblas-openblas-openmp-3.0.4-8.el9 156 kB/s | 17 kB 00:00 2024-03-23T11:16:21.844 INFO:teuthology.orchestra.run.smithi062.stdout:(45/156): libaio-devel-0.3.111-13.el9.x86_64.rp 205 kB/s | 10 kB 00:00 2024-03-23T11:16:21.887 INFO:teuthology.orchestra.run.smithi062.stdout:(46/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2024-03-23T11:16:21.928 INFO:teuthology.orchestra.run.smithi062.stdout:(47/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 543 kB/s | 44 kB 00:00 2024-03-23T11:16:21.979 INFO:teuthology.orchestra.run.smithi062.stdout:(48/156): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-03-23T11:16:22.021 INFO:teuthology.orchestra.run.smithi062.stdout:(49/156): libtool-2.4.6-45.el9.x86_64.rpm 6.6 MB/s | 578 kB 00:00 2024-03-23T11:16:22.120 INFO:teuthology.orchestra.run.smithi062.stdout:(50/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 344 kB/s | 46 kB 00:00 2024-03-23T11:16:22.238 INFO:teuthology.orchestra.run.smithi062.stdout:(51/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 282 kB 00:00 2024-03-23T11:16:22.272 INFO:teuthology.orchestra.run.smithi062.stdout:(52/156): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-23T11:16:22.276 INFO:teuthology.orchestra.run.smithi120.stdout:(10/156): ceph-radosgw-18.2.2-820.gb630d6cd.el9 3.6 MB/s | 7.7 MB 00:02 2024-03-23T11:16:22.298 INFO:teuthology.orchestra.run.smithi062.stdout:(53/156): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 34 kB 00:00 2024-03-23T11:16:22.351 INFO:teuthology.orchestra.run.smithi120.stdout:(11/156): cephfs-mirror-18.2.2-820.gb630d6cd.el 2.9 MB/s | 222 kB 00:00 2024-03-23T11:16:22.401 INFO:teuthology.orchestra.run.smithi120.stdout:(12/156): libcephfs-devel-18.2.2-820.gb630d6cd. 634 kB/s | 31 kB 00:00 2024-03-23T11:16:22.431 INFO:teuthology.orchestra.run.smithi062.stdout:(54/156): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-23T11:16:22.480 INFO:teuthology.orchestra.run.smithi062.stdout:(55/156): postgresql-13.11-1.el9.x86_64.rpm 9.2 MB/s | 1.5 MB 00:00 2024-03-23T11:16:22.523 INFO:teuthology.orchestra.run.smithi062.stdout:(56/156): postgresql-contrib-13.11-1.el9.x86_64 8.7 MB/s | 811 kB 00:00 2024-03-23T11:16:22.548 INFO:teuthology.orchestra.run.smithi062.stdout:(57/156): postgresql-private-libs-13.11-1.el9.x 2.2 MB/s | 134 kB 00:00 2024-03-23T11:16:22.577 INFO:teuthology.orchestra.run.smithi120.stdout:(13/156): libcephfs2-18.2.2-820.gb630d6cd.el9.x 3.7 MB/s | 659 kB 00:00 2024-03-23T11:16:22.644 INFO:teuthology.orchestra.run.smithi120.stdout:(14/156): libcephsqlite-18.2.2-820.gb630d6cd.el 2.5 MB/s | 166 kB 00:00 2024-03-23T11:16:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:(58/156): postgresql-server-13.11-1.el9.x86_64. 33 MB/s | 5.7 MB 00:00 2024-03-23T11:16:22.702 INFO:teuthology.orchestra.run.smithi120.stdout:(15/156): librados-devel-18.2.2-820.gb630d6cd.e 2.1 MB/s | 127 kB 00:00 2024-03-23T11:16:22.832 INFO:teuthology.orchestra.run.smithi062.stdout:(59/156): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-23T11:16:22.836 INFO:teuthology.orchestra.run.smithi120.stdout:(16/156): libradosstriper1-18.2.2-820.gb630d6cd 3.5 MB/s | 474 kB 00:00 2024-03-23T11:16:22.849 INFO:teuthology.orchestra.run.smithi062.stdout:(60/156): python3-jsonpatch-1.21-16.el9.noarch. 181 kB/s | 24 kB 00:00 2024-03-23T11:16:22.866 INFO:teuthology.orchestra.run.smithi062.stdout:(61/156): python3-jsonpointer-2.0-4.el9.noarch. 561 kB/s | 18 kB 00:00 2024-03-23T11:16:22.908 INFO:teuthology.orchestra.run.smithi062.stdout:(62/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-23T11:16:22.925 INFO:teuthology.orchestra.run.smithi062.stdout:(63/156): python3-markupsafe-1.1.1-12.el9.x86_6 555 kB/s | 32 kB 00:00 2024-03-23T11:16:23.092 INFO:teuthology.orchestra.run.smithi062.stdout:(64/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2024-03-23T11:16:23.117 INFO:teuthology.orchestra.run.smithi062.stdout:(65/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-23T11:16:23.159 INFO:teuthology.orchestra.run.smithi062.stdout:(66/156): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-03-23T11:16:23.184 INFO:teuthology.orchestra.run.smithi062.stdout:(67/156): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-23T11:16:23.218 INFO:teuthology.orchestra.run.smithi062.stdout:(68/156): python3-prettytable-0.7.2-27.el9.noar 1.3 MB/s | 41 kB 00:00 2024-03-23T11:16:23.243 INFO:teuthology.orchestra.run.smithi062.stdout:(69/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-23T11:16:23.285 INFO:teuthology.orchestra.run.smithi062.stdout:(70/156): python3-pyasn1-modules-0.4.8-6.el9.no 5.1 MB/s | 211 kB 00:00 2024-03-23T11:16:23.310 INFO:teuthology.orchestra.run.smithi062.stdout:(71/156): python3-pytz-2021.1-5.el9.noarch.rpm 723 kB/s | 47 kB 00:00 2024-03-23T11:16:23.352 INFO:teuthology.orchestra.run.smithi062.stdout:(72/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-03-23T11:16:23.562 INFO:teuthology.orchestra.run.smithi120.stdout:(17/156): librgw2-18.2.2-820.gb630d6cd.el9.x86_ 6.2 MB/s | 4.5 MB 00:00 2024-03-23T11:16:23.612 INFO:teuthology.orchestra.run.smithi120.stdout:(18/156): python3-ceph-argparse-18.2.2-820.gb63 920 kB/s | 45 kB 00:00 2024-03-23T11:16:23.670 INFO:teuthology.orchestra.run.smithi120.stdout:(19/156): python3-ceph-common-18.2.2-820.gb630d 2.2 MB/s | 127 kB 00:00 2024-03-23T11:16:23.737 INFO:teuthology.orchestra.run.smithi120.stdout:(20/156): python3-cephfs-18.2.2-820.gb630d6cd.e 2.4 MB/s | 161 kB 00:00 2024-03-23T11:16:23.829 INFO:teuthology.orchestra.run.smithi120.stdout:(21/156): python3-rados-18.2.2-820.gb630d6cd.el 3.4 MB/s | 320 kB 00:00 2024-03-23T11:16:23.921 INFO:teuthology.orchestra.run.smithi120.stdout:(22/156): python3-rbd-18.2.2-820.gb630d6cd.el9. 3.2 MB/s | 298 kB 00:00 2024-03-23T11:16:23.961 INFO:teuthology.orchestra.run.smithi062.stdout:(73/156): ceph-test-18.2.2-820.gb630d6cd.el9.x8 7.0 MB/s | 40 MB 00:05 2024-03-23T11:16:23.971 INFO:teuthology.orchestra.run.smithi120.stdout:(23/156): python3-rgw-18.2.2-820.gb630d6cd.el9. 2.0 MB/s | 99 kB 00:00 2024-03-23T11:16:24.011 INFO:teuthology.orchestra.run.smithi062.stdout:(74/156): python3-toml-0.10.2-6.el9.noarch.rpm 61 kB/s | 39 kB 00:00 2024-03-23T11:16:24.030 INFO:teuthology.orchestra.run.smithi120.stdout:(24/156): rbd-fuse-18.2.2-820.gb630d6cd.el9.x86 1.5 MB/s | 87 kB 00:00 2024-03-23T11:16:24.220 INFO:teuthology.orchestra.run.smithi062.stdout:(75/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-03-23T11:16:24.245 INFO:teuthology.orchestra.run.smithi062.stdout:(76/156): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2024-03-23T11:16:24.270 INFO:teuthology.orchestra.run.smithi062.stdout:(77/156): uuid-1.6.2-55.el9.x86_64.rpm 224 kB/s | 56 kB 00:00 2024-03-23T11:16:24.295 INFO:teuthology.orchestra.run.smithi062.stdout:(78/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-03-23T11:16:24.321 INFO:teuthology.orchestra.run.smithi062.stdout:(79/156): zlib-devel-1.2.11-40.el9.x86_64.rpm 599 kB/s | 44 kB 00:00 2024-03-23T11:16:24.346 INFO:teuthology.orchestra.run.smithi062.stdout:(80/156): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-23T11:16:24.388 INFO:teuthology.orchestra.run.smithi062.stdout:(81/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.6 MB/s | 241 kB 00:00 2024-03-23T11:16:24.430 INFO:teuthology.orchestra.run.smithi062.stdout:(82/156): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-23T11:16:24.455 INFO:teuthology.orchestra.run.smithi062.stdout:(83/156): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-23T11:16:24.489 INFO:teuthology.orchestra.run.smithi062.stdout:(84/156): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-23T11:16:24.514 INFO:teuthology.orchestra.run.smithi062.stdout:(85/156): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-23T11:16:24.564 INFO:teuthology.orchestra.run.smithi120.stdout:(25/156): ceph-osd-18.2.2-820.gb630d6cd.el9.x86 3.7 MB/s | 18 MB 00:04 2024-03-23T11:16:24.648 INFO:teuthology.orchestra.run.smithi062.stdout:(86/156): jq-1.6-16.el9.x86_64.rpm 994 kB/s | 187 kB 00:00 2024-03-23T11:16:24.656 INFO:teuthology.orchestra.run.smithi120.stdout:(26/156): rbd-mirror-18.2.2-820.gb630d6cd.el9.x 4.8 MB/s | 3.0 MB 00:00 2024-03-23T11:16:24.673 INFO:teuthology.orchestra.run.smithi062.stdout:(87/156): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-03-23T11:16:24.689 INFO:teuthology.orchestra.run.smithi120.stdout:(27/156): rbd-nbd-18.2.2-820.gb630d6cd.el9.x86_ 1.4 MB/s | 172 kB 00:00 2024-03-23T11:16:24.706 INFO:teuthology.orchestra.run.smithi062.stdout:(88/156): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-23T11:16:24.714 INFO:teuthology.orchestra.run.smithi120.stdout:(28/156): ceph-grafana-dashboards-18.2.2-820.gb 406 kB/s | 23 kB 00:00 2024-03-23T11:16:24.748 INFO:teuthology.orchestra.run.smithi062.stdout:(89/156): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2024-03-23T11:16:24.765 INFO:teuthology.orchestra.run.smithi120.stdout:(29/156): ceph-mgr-cephadm-18.2.2-820.gb630d6cd 1.8 MB/s | 138 kB 00:00 2024-03-23T11:16:24.815 INFO:teuthology.orchestra.run.smithi062.stdout:(90/156): elfutils-libelf-devel-0.190-2.el9.x86 152 kB/s | 21 kB 00:00 2024-03-23T11:16:24.857 INFO:teuthology.orchestra.run.smithi062.stdout:(91/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 170 kB/s | 25 kB 00:00 2024-03-23T11:16:24.958 INFO:teuthology.orchestra.run.smithi062.stdout:(92/156): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-23T11:16:24.991 INFO:teuthology.orchestra.run.smithi062.stdout:(93/156): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-03-23T11:16:25.024 INFO:teuthology.orchestra.run.smithi120.stdout:(30/156): ceph-mgr-dashboard-18.2.2-820.gb630d6 5.7 MB/s | 1.7 MB 00:00 2024-03-23T11:16:25.033 INFO:teuthology.orchestra.run.smithi062.stdout:(94/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.2 MB/s | 94 kB 00:00 2024-03-23T11:16:25.110 INFO:teuthology.orchestra.run.smithi120.stdout:(31/156): ceph-mgr-modules-core-18.2.2-820.gb63 2.9 MB/s | 245 kB 00:00 2024-03-23T11:16:25.166 INFO:teuthology.orchestra.run.smithi120.stdout:(32/156): ceph-mgr-rook-18.2.2-820.gb630d6cd.el 858 kB/s | 49 kB 00:00 2024-03-23T11:16:25.224 INFO:teuthology.orchestra.run.smithi120.stdout:(33/156): ceph-prometheus-alerts-18.2.2-820.gb6 259 kB/s | 15 kB 00:00 2024-03-23T11:16:25.333 INFO:teuthology.orchestra.run.smithi120.stdout:(34/156): ceph-volume-18.2.2-820.gb630d6cd.el9. 2.4 MB/s | 264 kB 00:00 2024-03-23T11:16:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:(95/156): fmt-8.1.1-5.el9.x86_64.rpm 289 kB/s | 111 kB 00:00 2024-03-23T11:16:25.416 INFO:teuthology.orchestra.run.smithi120.stdout:(35/156): cephadm-18.2.2-820.gb630d6cd.el9.noar 2.7 MB/s | 224 kB 00:00 2024-03-23T11:16:25.475 INFO:teuthology.orchestra.run.smithi120.stdout:(36/156): cephfs-top-18.2.2-820.gb630d6cd.el9.n 313 kB/s | 18 kB 00:00 2024-03-23T11:16:25.475 INFO:teuthology.orchestra.run.smithi062.stdout:(96/156): gperftools-libs-2.9.1-2.el9.x86_64.rp 700 kB/s | 309 kB 00:00 2024-03-23T11:16:25.658 INFO:teuthology.orchestra.run.smithi120.stdout:(37/156): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 662 kB 00:00 2024-03-23T11:16:25.726 INFO:teuthology.orchestra.run.smithi062.stdout:(97/156): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-23T11:16:25.734 INFO:teuthology.orchestra.run.smithi120.stdout:(38/156): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 920 kB 00:00 2024-03-23T11:16:25.751 INFO:teuthology.orchestra.run.smithi062.stdout:(98/156): libarrow-doc-9.0.0-10.el9.noarch.rpm 95 kB/s | 26 kB 00:00 2024-03-23T11:16:25.775 INFO:teuthology.orchestra.run.smithi120.stdout:(39/156): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-23T11:16:25.834 INFO:teuthology.orchestra.run.smithi120.stdout:(40/156): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-23T11:16:25.843 INFO:teuthology.orchestra.run.smithi062.stdout:(99/156): openssl-devel-3.0.7-27.el9.x86_64.rpm 3.8 MB/s | 4.1 MB 00:01 2024-03-23T11:16:25.868 INFO:teuthology.orchestra.run.smithi062.stdout:(100/156): liboath-2.6.7-2.el9.x86_64.rpm 420 kB/s | 49 kB 00:00 2024-03-23T11:16:25.893 INFO:teuthology.orchestra.run.smithi062.stdout:(101/156): libunwind-1.6.2-1.el9.x86_64.rpm 479 kB/s | 67 kB 00:00 2024-03-23T11:16:25.992 INFO:teuthology.orchestra.run.smithi062.stdout:(102/156): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.6 MB/s | 838 kB 00:00 2024-03-23T11:16:26.035 INFO:teuthology.orchestra.run.smithi062.stdout:(103/156): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-03-23T11:16:26.063 INFO:teuthology.orchestra.run.smithi062.stdout:(104/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 262 kB/s | 43 kB 00:00 2024-03-23T11:16:26.104 INFO:teuthology.orchestra.run.smithi062.stdout:(105/156): python3-cachetools-4.2.4-1.el9.noarc 309 kB/s | 32 kB 00:00 2024-03-23T11:16:26.170 INFO:teuthology.orchestra.run.smithi062.stdout:(106/156): python3-certifi-2021.10.8-2.el9.noar 139 kB/s | 15 kB 00:00 2024-03-23T11:16:26.206 INFO:teuthology.orchestra.run.smithi062.stdout:(107/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.2 MB/s | 172 kB 00:00 2024-03-23T11:16:26.237 INFO:teuthology.orchestra.run.smithi062.stdout:(108/156): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-23T11:16:26.279 INFO:teuthology.orchestra.run.smithi062.stdout:(109/156): python3-google-auth-2.28.2-1.el9.noa 1.9 MB/s | 203 kB 00:00 2024-03-23T11:16:26.296 INFO:teuthology.orchestra.run.smithi062.stdout:(110/156): python3-jaraco-8.2.1-3.el9.noarch.rp 121 kB/s | 11 kB 00:00 2024-03-23T11:16:26.301 INFO:teuthology.orchestra.run.smithi120.stdout:(41/156): ceph-test-18.2.2-820.gb630d6cd.el9.x8 7.9 MB/s | 40 MB 00:04 2024-03-23T11:16:26.326 INFO:teuthology.orchestra.run.smithi120.stdout:(42/156): flexiblas-3.0.4-8.el9.x86_64.rpm 64 kB/s | 32 kB 00:00 2024-03-23T11:16:26.329 INFO:teuthology.orchestra.run.smithi062.stdout:(111/156): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-03-23T11:16:26.380 INFO:teuthology.orchestra.run.smithi062.stdout:(112/156): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-03-23T11:16:26.405 INFO:teuthology.orchestra.run.smithi062.stdout:(113/156): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-23T11:16:26.430 INFO:teuthology.orchestra.run.smithi062.stdout:(114/156): python3-jaraco-text-3.2.0-6.el9.noar 197 kB/s | 20 kB 00:00 2024-03-23T11:16:26.468 INFO:teuthology.orchestra.run.smithi120.stdout:(43/156): ceph-mgr-diskprediction-local-18.2.2- 4.3 MB/s | 7.4 MB 00:01 2024-03-23T11:16:26.497 INFO:teuthology.orchestra.run.smithi062.stdout:(115/156): python3-kubernetes-26.1.0-2.el9.noar 8.8 MB/s | 1.0 MB 00:00 2024-03-23T11:16:26.510 INFO:teuthology.orchestra.run.smithi120.stdout:(44/156): flexiblas-openblas-openmp-3.0.4-8.el9 94 kB/s | 17 kB 00:00 2024-03-23T11:16:26.522 INFO:teuthology.orchestra.run.smithi062.stdout:(116/156): python3-logutils-0.3.5-21.el9.noarch 398 kB/s | 46 kB 00:00 2024-03-23T11:16:26.547 INFO:teuthology.orchestra.run.smithi062.stdout:(117/156): python3-more-itertools-8.12.0-2.el9. 676 kB/s | 79 kB 00:00 2024-03-23T11:16:26.573 INFO:teuthology.orchestra.run.smithi062.stdout:(118/156): python3-natsort-7.1.1-5.el9.noarch.r 784 kB/s | 58 kB 00:00 2024-03-23T11:16:26.594 INFO:teuthology.orchestra.run.smithi120.stdout:(45/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 10 MB/s | 3.0 MB 00:00 2024-03-23T11:16:26.614 INFO:teuthology.orchestra.run.smithi062.stdout:(119/156): python3-pecan-1.4.2-3.el9.noarch.rpm 4.1 MB/s | 272 kB 00:00 2024-03-23T11:16:26.640 INFO:teuthology.orchestra.run.smithi062.stdout:(120/156): python3-portend-3.1.0-2.el9.noarch.r 182 kB/s | 16 kB 00:00 2024-03-23T11:16:26.642 INFO:teuthology.orchestra.run.smithi120.stdout:(46/156): libaio-devel-0.3.111-13.el9.x86_64.rp 62 kB/s | 10 kB 00:00 2024-03-23T11:16:26.665 INFO:teuthology.orchestra.run.smithi062.stdout:(121/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 983 kB/s | 90 kB 00:00 2024-03-23T11:16:26.677 INFO:teuthology.orchestra.run.smithi120.stdout:(47/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-23T11:16:26.707 INFO:teuthology.orchestra.run.smithi062.stdout:(122/156): python3-repoze-lru-0.7-16.el9.noarch 473 kB/s | 31 kB 00:00 2024-03-23T11:16:26.711 INFO:teuthology.orchestra.run.smithi120.stdout:(48/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 669 kB/s | 44 kB 00:00 2024-03-23T11:16:26.732 INFO:teuthology.orchestra.run.smithi062.stdout:(123/156): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-03-23T11:16:26.753 INFO:teuthology.orchestra.run.smithi120.stdout:(49/156): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-23T11:16:26.757 INFO:teuthology.orchestra.run.smithi062.stdout:(124/156): python3-rsa-4.9-2.el9.noarch.rpm 646 kB/s | 59 kB 00:00 2024-03-23T11:16:26.795 INFO:teuthology.orchestra.run.smithi120.stdout:(50/156): libtool-2.4.6-45.el9.x86_64.rpm 5.1 MB/s | 578 kB 00:00 2024-03-23T11:16:26.799 INFO:teuthology.orchestra.run.smithi062.stdout:(125/156): python3-tempora-5.0.0-2.el9.noarch.r 549 kB/s | 36 kB 00:00 2024-03-23T11:16:26.820 INFO:teuthology.orchestra.run.smithi120.stdout:(51/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 755 kB/s | 46 kB 00:00 2024-03-23T11:16:26.833 INFO:teuthology.orchestra.run.smithi062.stdout:(126/156): python3-typing-extensions-4.4.0-2.el 516 kB/s | 51 kB 00:00 2024-03-23T11:16:26.845 INFO:teuthology.orchestra.run.smithi120.stdout:(52/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.2 MB/s | 282 kB 00:00 2024-03-23T11:16:26.858 INFO:teuthology.orchestra.run.smithi062.stdout:(127/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-23T11:16:26.879 INFO:teuthology.orchestra.run.smithi120.stdout:(53/156): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.6 MB/s | 217 kB 00:00 2024-03-23T11:16:26.883 INFO:teuthology.orchestra.run.smithi062.stdout:(128/156): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-23T11:16:26.904 INFO:teuthology.orchestra.run.smithi120.stdout:(54/156): openblas-0.3.21-2.el9.x86_64.rpm 608 kB/s | 34 kB 00:00 2024-03-23T11:16:26.942 INFO:teuthology.orchestra.run.smithi062.stdout:(129/156): python3-werkzeug-2.0.3-3.el9.1.noarc 5.1 MB/s | 427 kB 00:00 2024-03-23T11:16:26.967 INFO:teuthology.orchestra.run.smithi062.stdout:(130/156): python3-zc-lockfile-2.0-10.el9.noarc 187 kB/s | 20 kB 00:00 2024-03-23T11:16:27.018 INFO:teuthology.orchestra.run.smithi062.stdout:(131/156): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-23T11:16:27.080 INFO:teuthology.orchestra.run.smithi120.stdout:(55/156): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-03-23T11:16:27.135 INFO:teuthology.orchestra.run.smithi062.stdout:(132/156): thrift-0.15.0-2.el9.x86_64.rpm 9.6 MB/s | 1.6 MB 00:00 2024-03-23T11:16:27.155 INFO:teuthology.orchestra.run.smithi120.stdout:(56/156): postgresql-13.11-1.el9.x86_64.rpm 5.7 MB/s | 1.5 MB 00:00 2024-03-23T11:16:27.177 INFO:teuthology.orchestra.run.smithi062.stdout:(133/156): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-03-23T11:16:27.188 INFO:teuthology.orchestra.run.smithi120.stdout:(57/156): postgresql-contrib-13.11-1.el9.x86_64 2.9 MB/s | 811 kB 00:00 2024-03-23T11:16:27.214 INFO:teuthology.orchestra.run.smithi120.stdout:(58/156): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2024-03-23T11:16:27.385 INFO:teuthology.orchestra.run.smithi062.stdout:(134/156): NetworkManager-1.47.2-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-03-23T11:16:27.469 INFO:teuthology.orchestra.run.smithi062.stdout:(135/156): librados2-18.2.2-820.gb630d6cd.el9.x 6.6 MB/s | 3.3 MB 00:00 2024-03-23T11:16:27.472 INFO:teuthology.orchestra.run.smithi120.stdout:(59/156): postgresql-server-13.11-1.el9.x86_64. 20 MB/s | 5.7 MB 00:00 2024-03-23T11:16:27.569 INFO:teuthology.orchestra.run.smithi062.stdout:(136/156): librbd1-18.2.2-820.gb630d6cd.el9.x86 5.5 MB/s | 3.0 MB 00:00 2024-03-23T11:16:27.594 INFO:teuthology.orchestra.run.smithi062.stdout:(137/156): NetworkManager-team-1.47.2-1.el9.x86 339 kB/s | 42 kB 00:00 2024-03-23T11:16:27.606 INFO:teuthology.orchestra.run.smithi120.stdout:(60/156): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-23T11:16:27.631 INFO:teuthology.orchestra.run.smithi120.stdout:(61/156): python3-jsonpatch-1.21-16.el9.noarch. 61 kB/s | 24 kB 00:00 2024-03-23T11:16:27.653 INFO:teuthology.orchestra.run.smithi062.stdout:(138/156): NetworkManager-libnm-1.47.2-1.el9.x8 6.9 MB/s | 1.8 MB 00:00 2024-03-23T11:16:27.656 INFO:teuthology.orchestra.run.smithi120.stdout:(62/156): python3-jsonpointer-2.0-4.el9.noarch. 101 kB/s | 18 kB 00:00 2024-03-23T11:16:27.678 INFO:teuthology.orchestra.run.smithi062.stdout:(139/156): acl-2.3.1-4.el9.x86_64.rpm 862 kB/s | 71 kB 00:00 2024-03-23T11:16:27.681 INFO:teuthology.orchestra.run.smithi120.stdout:(63/156): python3-markupsafe-1.1.1-12.el9.x86_6 647 kB/s | 32 kB 00:00 2024-03-23T11:16:27.703 INFO:teuthology.orchestra.run.smithi062.stdout:(140/156): elfutils-debuginfod-client-0.190-2.e 734 kB/s | 36 kB 00:00 2024-03-23T11:16:27.732 INFO:teuthology.orchestra.run.smithi120.stdout:(64/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2024-03-23T11:16:27.737 INFO:teuthology.orchestra.run.smithi062.stdout:(141/156): NetworkManager-tui-1.47.2-1.el9.x86_ 1.5 MB/s | 248 kB 00:00 2024-03-23T11:16:27.771 INFO:teuthology.orchestra.run.smithi062.stdout:(142/156): elfutils-libelf-0.190-2.el9.x86_64.r 2.1 MB/s | 192 kB 00:00 2024-03-23T11:16:27.788 INFO:teuthology.orchestra.run.smithi062.stdout:(143/156): libacl-2.3.1-4.el9.x86_64.rpm 466 kB/s | 23 kB 00:00 2024-03-23T11:16:27.813 INFO:teuthology.orchestra.run.smithi062.stdout:(144/156): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-23T11:16:27.830 INFO:teuthology.orchestra.run.smithi062.stdout:(145/156): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-03-23T11:16:27.863 INFO:teuthology.orchestra.run.smithi062.stdout:(146/156): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-23T11:16:27.866 INFO:teuthology.orchestra.run.smithi120.stdout:(65/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-03-23T11:16:27.889 INFO:teuthology.orchestra.run.smithi062.stdout:(147/156): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-23T11:16:27.890 INFO:teuthology.orchestra.run.smithi120.stdout:(66/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-23T11:16:27.914 INFO:teuthology.orchestra.run.smithi062.stdout:(148/156): libuuid-2.37.4-18.el9.x86_64.rpm 580 kB/s | 29 kB 00:00 2024-03-23T11:16:27.924 INFO:teuthology.orchestra.run.smithi120.stdout:(67/156): python3-oauthlib-3.1.1-5.el9.noarch.r 993 kB/s | 174 kB 00:00 2024-03-23T11:16:27.939 INFO:teuthology.orchestra.run.smithi062.stdout:(149/156): libsmartcols-2.37.4-18.el9.x86_64.rp 590 kB/s | 64 kB 00:00 2024-03-23T11:16:27.949 INFO:teuthology.orchestra.run.smithi120.stdout:(68/156): python3-prettytable-0.7.2-27.el9.noar 739 kB/s | 41 kB 00:00 2024-03-23T11:16:27.974 INFO:teuthology.orchestra.run.smithi120.stdout:(69/156): python3-packaging-20.9-5.el9.noarch.r 845 kB/s | 69 kB 00:00 2024-03-23T11:16:28.000 INFO:teuthology.orchestra.run.smithi120.stdout:(70/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-23T11:16:28.031 INFO:teuthology.orchestra.run.smithi062.stdout:(150/156): openssl-3.0.7-27.el9.x86_64.rpm 8.6 MB/s | 1.2 MB 00:00 2024-03-23T11:16:28.033 INFO:teuthology.orchestra.run.smithi120.stdout:(71/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-03-23T11:16:28.058 INFO:teuthology.orchestra.run.smithi120.stdout:(72/156): python3-pytz-2021.1-5.el9.noarch.rpm 849 kB/s | 47 kB 00:00 2024-03-23T11:16:28.083 INFO:teuthology.orchestra.run.smithi120.stdout:(73/156): python3-requests-oauthlib-1.3.0-12.el 530 kB/s | 43 kB 00:00 2024-03-23T11:16:28.115 INFO:teuthology.orchestra.run.smithi062.stdout:(151/156): openssl-libs-3.0.7-27.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-03-23T11:16:28.157 INFO:teuthology.orchestra.run.smithi062.stdout:(152/156): util-linux-core-2.37.4-18.el9.x86_64 3.7 MB/s | 465 kB 00:00 2024-03-23T11:16:28.182 INFO:teuthology.orchestra.run.smithi062.stdout:(153/156): util-linux-user-2.37.4-18.el9.x86_64 472 kB/s | 31 kB 00:00 2024-03-23T11:16:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:(154/156): util-linux-2.37.4-18.el9.x86_64.rpm 7.2 MB/s | 2.3 MB 00:00 2024-03-23T11:16:28.308 INFO:teuthology.orchestra.run.smithi062.stdout:(155/156): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.1 MB/s | 1.1 MB 00:00 2024-03-23T11:16:28.392 INFO:teuthology.orchestra.run.smithi062.stdout:(156/156): libacl-devel-2.3.1-4.el9.x86_64.rpm 356 kB/s | 74 kB 00:00 2024-03-23T11:16:28.405 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:16:28.405 INFO:teuthology.orchestra.run.smithi062.stdout:Total 15 MB/s | 205 MB 00:13 2024-03-23T11:16:28.476 INFO:teuthology.orchestra.run.smithi120.stdout:(74/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-23T11:16:28.501 INFO:teuthology.orchestra.run.smithi120.stdout:(75/156): python3-toml-0.10.2-6.el9.noarch.rpm 96 kB/s | 39 kB 00:00 2024-03-23T11:16:28.620 INFO:teuthology.orchestra.run.smithi120.stdout:(76/156): socat-1.7.4.1-5.el9.x86_64.rpm 577 kB/s | 300 kB 00:00 2024-03-23T11:16:28.651 INFO:teuthology.orchestra.run.smithi120.stdout:(77/156): uuid-1.6.2-55.el9.x86_64.rpm 383 kB/s | 56 kB 00:00 2024-03-23T11:16:28.676 INFO:teuthology.orchestra.run.smithi120.stdout:(78/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 364 kB/s | 63 kB 00:00 2024-03-23T11:16:28.702 INFO:teuthology.orchestra.run.smithi120.stdout:(79/156): zlib-devel-1.2.11-40.el9.x86_64.rpm 556 kB/s | 44 kB 00:00 2024-03-23T11:16:28.727 INFO:teuthology.orchestra.run.smithi120.stdout:(80/156): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-23T11:16:28.752 INFO:teuthology.orchestra.run.smithi120.stdout:(81/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.2 MB/s | 241 kB 00:00 2024-03-23T11:16:28.811 INFO:teuthology.orchestra.run.smithi120.stdout:(82/156): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-23T11:16:28.836 INFO:teuthology.orchestra.run.smithi120.stdout:(83/156): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 124 kB 00:00 2024-03-23T11:16:28.878 INFO:teuthology.orchestra.run.smithi120.stdout:(84/156): python3-ply-3.11-14.el9.noarch.rpm 897 kB/s | 103 kB 00:00 2024-03-23T11:16:28.903 INFO:teuthology.orchestra.run.smithi120.stdout:(85/156): python3-urllib3-1.26.5-5.el9.noarch.r 2.0 MB/s | 187 kB 00:00 2024-03-23T11:16:29.078 INFO:teuthology.orchestra.run.smithi120.stdout:(86/156): jq-1.6-16.el9.x86_64.rpm 786 kB/s | 187 kB 00:00 2024-03-23T11:16:29.112 INFO:teuthology.orchestra.run.smithi120.stdout:(87/156): libquadmath-11.4.1-3.el9.x86_64.rpm 943 kB/s | 195 kB 00:00 2024-03-23T11:16:29.146 INFO:teuthology.orchestra.run.smithi120.stdout:(88/156): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-03-23T11:16:29.196 INFO:teuthology.orchestra.run.smithi120.stdout:(89/156): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-03-23T11:16:29.288 INFO:teuthology.orchestra.run.smithi120.stdout:(90/156): elfutils-libelf-devel-0.190-2.el9.x86 123 kB/s | 21 kB 00:00 2024-03-23T11:16:29.313 INFO:teuthology.orchestra.run.smithi120.stdout:(91/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 153 kB/s | 25 kB 00:00 2024-03-23T11:16:29.472 INFO:teuthology.orchestra.run.smithi120.stdout:(92/156): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-03-23T11:16:29.505 INFO:teuthology.orchestra.run.smithi120.stdout:(93/156): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-23T11:16:29.539 INFO:teuthology.orchestra.run.smithi120.stdout:(94/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-03-23T11:16:29.671 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-23T11:16:29.697 INFO:teuthology.orchestra.run.smithi120.stdout:(95/156): fmt-8.1.1-5.el9.x86_64.rpm 580 kB/s | 111 kB 00:00 2024-03-23T11:16:29.814 INFO:teuthology.orchestra.run.smithi120.stdout:(96/156): openssl-devel-3.0.7-27.el9.x86_64.rpm 6.7 MB/s | 4.1 MB 00:00 2024-03-23T11:16:29.848 INFO:teuthology.orchestra.run.smithi120.stdout:(97/156): gperftools-libs-2.9.1-2.el9.x86_64.rp 1.0 MB/s | 309 kB 00:00 2024-03-23T11:16:29.906 INFO:teuthology.orchestra.run.smithi120.stdout:(98/156): liboath-2.6.7-2.el9.x86_64.rpm 845 kB/s | 49 kB 00:00 2024-03-23T11:16:29.932 INFO:teuthology.orchestra.run.smithi120.stdout:(99/156): libarrow-doc-9.0.0-10.el9.noarch.rpm 224 kB/s | 26 kB 00:00 2024-03-23T11:16:29.957 INFO:teuthology.orchestra.run.smithi120.stdout:(100/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-23T11:16:30.061 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-23T11:16:30.061 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-23T11:16:30.066 INFO:teuthology.orchestra.run.smithi120.stdout:(101/156): python3-asyncssh-2.13.2-5.el9.noarch 5.0 MB/s | 548 kB 00:00 2024-03-23T11:16:30.147 INFO:teuthology.orchestra.run.smithi120.stdout:(102/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 539 kB/s | 43 kB 00:00 2024-03-23T11:16:30.199 INFO:teuthology.orchestra.run.smithi120.stdout:(103/156): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-03-23T11:16:30.216 INFO:teuthology.orchestra.run.smithi120.stdout:(104/156): python3-cachetools-4.2.4-1.el9.noarc 474 kB/s | 32 kB 00:00 2024-03-23T11:16:30.333 INFO:teuthology.orchestra.run.smithi120.stdout:(105/156): libarrow-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 4.4 MB 00:00 2024-03-23T11:16:30.350 INFO:teuthology.orchestra.run.smithi120.stdout:(106/156): python3-certifi-2021.10.8-2.el9.noar 100 kB/s | 15 kB 00:00 2024-03-23T11:16:30.384 INFO:teuthology.orchestra.run.smithi120.stdout:(107/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.0 MB/s | 172 kB 00:00 2024-03-23T11:16:30.417 INFO:teuthology.orchestra.run.smithi120.stdout:(108/156): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-23T11:16:30.443 INFO:teuthology.orchestra.run.smithi120.stdout:(109/156): python3-jaraco-8.2.1-3.el9.noarch.rp 185 kB/s | 11 kB 00:00 2024-03-23T11:16:30.468 INFO:teuthology.orchestra.run.smithi120.stdout:(110/156): python3-jaraco-classes-3.2.1-5.el9.n 358 kB/s | 18 kB 00:00 2024-03-23T11:16:30.493 INFO:teuthology.orchestra.run.smithi120.stdout:(111/156): python3-jaraco-collections-3.0.0-8.e 468 kB/s | 23 kB 00:00 2024-03-23T11:16:30.543 INFO:teuthology.orchestra.run.smithi120.stdout:(112/156): python3-jaraco-functools-3.5.0-2.el9 263 kB/s | 19 kB 00:00 2024-03-23T11:16:30.569 INFO:teuthology.orchestra.run.smithi120.stdout:(113/156): python3-jaraco-text-3.2.0-6.el9.noar 264 kB/s | 20 kB 00:00 2024-03-23T11:16:30.594 INFO:teuthology.orchestra.run.smithi120.stdout:(114/156): python3-google-auth-2.28.2-1.el9.noa 835 kB/s | 203 kB 00:00 2024-03-23T11:16:30.627 INFO:teuthology.orchestra.run.smithi120.stdout:(115/156): python3-logutils-0.3.5-21.el9.noarch 799 kB/s | 46 kB 00:00 2024-03-23T11:16:30.644 INFO:teuthology.orchestra.run.smithi120.stdout:(116/156): python3-more-itertools-8.12.0-2.el9. 1.6 MB/s | 79 kB 00:00 2024-03-23T11:16:30.695 INFO:teuthology.orchestra.run.smithi120.stdout:(117/156): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-03-23T11:16:30.720 INFO:teuthology.orchestra.run.smithi120.stdout:(118/156): python3-natsort-7.1.1-5.el9.noarch.r 632 kB/s | 58 kB 00:00 2024-03-23T11:16:30.754 INFO:teuthology.orchestra.run.smithi120.stdout:(119/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-23T11:16:30.779 INFO:teuthology.orchestra.run.smithi120.stdout:(120/156): python3-portend-3.1.0-2.el9.noarch.r 198 kB/s | 16 kB 00:00 2024-03-23T11:16:30.804 INFO:teuthology.orchestra.run.smithi120.stdout:(121/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-03-23T11:16:30.829 INFO:teuthology.orchestra.run.smithi120.stdout:(122/156): python3-repoze-lru-0.7-16.el9.noarch 410 kB/s | 31 kB 00:00 2024-03-23T11:16:30.863 INFO:teuthology.orchestra.run.smithi120.stdout:(123/156): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-23T11:16:30.888 INFO:teuthology.orchestra.run.smithi120.stdout:(124/156): python3-rsa-4.9-2.el9.noarch.rpm 713 kB/s | 59 kB 00:00 2024-03-23T11:16:30.938 INFO:teuthology.orchestra.run.smithi120.stdout:(125/156): python3-tempora-5.0.0-2.el9.noarch.r 331 kB/s | 36 kB 00:00 2024-03-23T11:16:30.964 INFO:teuthology.orchestra.run.smithi120.stdout:(126/156): python3-typing-extensions-4.4.0-2.el 511 kB/s | 51 kB 00:00 2024-03-23T11:16:30.997 INFO:teuthology.orchestra.run.smithi120.stdout:(127/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-23T11:16:31.023 INFO:teuthology.orchestra.run.smithi120.stdout:(128/156): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-23T11:16:31.048 INFO:teuthology.orchestra.run.smithi120.stdout:(129/156): python3-zc-lockfile-2.0-10.el9.noarc 404 kB/s | 20 kB 00:00 2024-03-23T11:16:31.081 INFO:teuthology.orchestra.run.smithi120.stdout:(130/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-23T11:16:31.115 INFO:teuthology.orchestra.run.smithi120.stdout:(131/156): re2-20211101-3.el9.x86_64.rpm 2.1 MB/s | 192 kB 00:00 2024-03-23T11:16:31.191 INFO:teuthology.orchestra.run.smithi120.stdout:(132/156): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-23T11:16:31.233 INFO:teuthology.orchestra.run.smithi120.stdout:(133/156): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-03-23T11:16:31.574 INFO:teuthology.orchestra.run.smithi120.stdout:(134/156): librados2-18.2.2-820.gb630d6cd.el9.x 6.7 MB/s | 3.3 MB 00:00 2024-03-23T11:16:31.641 INFO:teuthology.orchestra.run.smithi120.stdout:(135/156): NetworkManager-1.47.2-1.el9.x86_64.r 5.7 MB/s | 2.3 MB 00:00 2024-03-23T11:16:31.741 INFO:teuthology.orchestra.run.smithi120.stdout:(136/156): librbd1-18.2.2-820.gb630d6cd.el9.x86 4.8 MB/s | 3.0 MB 00:00 2024-03-23T11:16:31.767 INFO:teuthology.orchestra.run.smithi120.stdout:(137/156): NetworkManager-team-1.47.2-1.el9.x86 337 kB/s | 42 kB 00:00 2024-03-23T11:16:31.842 INFO:teuthology.orchestra.run.smithi120.stdout:(138/156): NetworkManager-libnm-1.47.2-1.el9.x8 6.9 MB/s | 1.8 MB 00:00 2024-03-23T11:16:31.867 INFO:teuthology.orchestra.run.smithi120.stdout:(139/156): acl-2.3.1-4.el9.x86_64.rpm 717 kB/s | 71 kB 00:00 2024-03-23T11:16:31.900 INFO:teuthology.orchestra.run.smithi120.stdout:(140/156): NetworkManager-tui-1.47.2-1.el9.x86_ 1.5 MB/s | 248 kB 00:00 2024-03-23T11:16:31.926 INFO:teuthology.orchestra.run.smithi120.stdout:(141/156): elfutils-debuginfod-client-0.190-2.e 439 kB/s | 36 kB 00:00 2024-03-23T11:16:31.960 INFO:teuthology.orchestra.run.smithi120.stdout:(142/156): elfutils-libelf-0.190-2.el9.x86_64.r 2.1 MB/s | 192 kB 00:00 2024-03-23T11:16:31.993 INFO:teuthology.orchestra.run.smithi120.stdout:(143/156): libacl-2.3.1-4.el9.x86_64.rpm 347 kB/s | 23 kB 00:00 2024-03-23T11:16:32.010 INFO:teuthology.orchestra.run.smithi120.stdout:(144/156): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-23T11:16:32.044 INFO:teuthology.orchestra.run.smithi120.stdout:(145/156): elfutils-libs-0.190-2.el9.x86_64.rpm 1.8 MB/s | 256 kB 00:00 2024-03-23T11:16:32.109 INFO:teuthology.orchestra.run.smithi120.stdout:(146/156): libmount-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 136 kB 00:00 2024-03-23T11:16:32.144 INFO:teuthology.orchestra.run.smithi120.stdout:(147/156): libfdisk-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 155 kB 00:00 2024-03-23T11:16:32.161 INFO:teuthology.orchestra.run.smithi120.stdout:(148/156): libuuid-2.37.4-18.el9.x86_64.rpm 566 kB/s | 29 kB 00:00 2024-03-23T11:16:32.178 INFO:teuthology.orchestra.run.smithi120.stdout:(149/156): libsmartcols-2.37.4-18.el9.x86_64.rp 479 kB/s | 64 kB 00:00 2024-03-23T11:16:32.253 INFO:teuthology.orchestra.run.smithi120.stdout:(150/156): openssl-3.0.7-27.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 2024-03-23T11:16:32.320 INFO:teuthology.orchestra.run.smithi120.stdout:(151/156): util-linux-core-2.37.4-18.el9.x86_64 6.9 MB/s | 465 kB 00:00 2024-03-23T11:16:32.395 INFO:teuthology.orchestra.run.smithi120.stdout:(152/156): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.3 MB 00:00 2024-03-23T11:16:32.421 INFO:teuthology.orchestra.run.smithi120.stdout:(153/156): util-linux-user-2.37.4-18.el9.x86_64 312 kB/s | 31 kB 00:00 2024-03-23T11:16:32.466 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-23T11:16:32.466 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-23T11:16:32.504 INFO:teuthology.orchestra.run.smithi120.stdout:(154/156): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.3 MB/s | 2.2 MB 00:00 2024-03-23T11:16:32.563 INFO:teuthology.orchestra.run.smithi120.stdout:(155/156): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.5 MB/s | 1.1 MB 00:00 2024-03-23T11:16:32.580 INFO:teuthology.orchestra.run.smithi120.stdout:(156/156): libacl-devel-2.3.1-4.el9.x86_64.rpm 471 kB/s | 74 kB 00:00 2024-03-23T11:16:32.585 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-23T11:16:32.585 INFO:teuthology.orchestra.run.smithi120.stdout:Total 12 MB/s | 205 MB 00:17 2024-03-23T11:16:33.792 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-23T11:16:34.161 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-23T11:16:34.161 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-23T11:16:36.367 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-23T11:16:36.368 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-23T11:16:39.288 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-23T11:16:39.435 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/180 2024-03-23T11:16:39.482 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/180 2024-03-23T11:16:39.511 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/180 2024-03-23T11:16:39.628 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/180 2024-03-23T11:16:39.669 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/180 2024-03-23T11:16:39.745 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/180 2024-03-23T11:16:39.809 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/180 2024-03-23T11:16:39.853 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/180 2024-03-23T11:16:39.949 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/180 2024-03-23T11:16:40.362 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/180 2024-03-23T11:16:40.400 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librados2-2:18.2.2-820.gb630d6cd.el9.x86_64 10/180 2024-03-23T11:16:40.585 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:18.2.2-820.gb630d6cd.el9.x86_64 10/180 2024-03-23T11:16:40.609 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs2-2:18.2.2-820.gb630d6cd.el9.x86_64 11/180 2024-03-23T11:16:40.766 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs2-2:18.2.2-820.gb630d6cd.el9.x86_64 11/180 2024-03-23T11:16:40.803 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rados-2:18.2.2-820.gb630d6cd.el9.x86_64 12/180 2024-03-23T11:16:40.864 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/180 2024-03-23T11:16:41.175 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/180 2024-03-23T11:16:41.234 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/180 2024-03-23T11:16:41.267 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephsqlite-2:18.2.2-820.gb630d6cd.el9.x86_64 16/180 2024-03-23T11:16:41.448 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephsqlite-2:18.2.2-820.gb630d6cd.el9.x86_64 16/180 2024-03-23T11:16:41.477 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libradosstriper1-2:18.2.2-820.gb630d6cd.el9.x86_ 17/180 2024-03-23T11:16:41.577 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libradosstriper1-2:18.2.2-820.gb630d6cd.el9.x86_ 17/180 2024-03-23T11:16:41.642 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/180 2024-03-23T11:16:41.693 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 19/180 2024-03-23T11:16:42.040 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 20/180 2024-03-23T11:16:42.225 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 21/180 2024-03-23T11:16:42.315 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 21/180 2024-03-23T11:16:42.510 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T11:16:42.561 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T11:16:43.032 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T11:16:43.083 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 23/180 2024-03-23T11:16:43.123 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : liboath-2.6.7-2.el9.x86_64 24/180 2024-03-23T11:16:43.200 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 25/180 2024-03-23T11:16:43.244 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-23T11:16:43.299 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 26/180 2024-03-23T11:16:43.365 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/180 2024-03-23T11:16:43.371 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/180 2024-03-23T11:16:43.419 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/180 2024-03-23T11:16:43.424 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/180 2024-03-23T11:16:43.457 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/180 2024-03-23T11:16:43.518 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/180 2024-03-23T11:16:43.571 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/180 2024-03-23T11:16:43.572 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/180 2024-03-23T11:16:43.607 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/180 2024-03-23T11:16:43.614 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/180 2024-03-23T11:16:43.697 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/180 2024-03-23T11:16:43.722 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 32/180 2024-03-23T11:16:43.761 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-common-2:18.2.2-820.gb630d6cd.el9.x 33/180 2024-03-23T11:16:43.780 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/180 2024-03-23T11:16:43.810 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-argparse-2:18.2.2-820.gb630d6cd.el9 34/180 2024-03-23T11:16:43.823 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/180 2024-03-23T11:16:43.916 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cephfs-2:18.2.2-820.gb630d6cd.el9.x86_64 35/180 2024-03-23T11:16:43.918 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/180 2024-03-23T11:16:44.094 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/180 2024-03-23T11:16:44.156 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/180 2024-03-23T11:16:44.332 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/180 2024-03-23T11:16:44.378 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:18.2.2-820.gb630d6cd.el9.x86_64 10/180 2024-03-23T11:16:44.514 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:18.2.2-820.gb630d6cd.el9.x86_64 10/180 2024-03-23T11:16:44.545 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:18.2.2-820.gb630d6cd.el9.x86_64 11/180 2024-03-23T11:16:44.702 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:18.2.2-820.gb630d6cd.el9.x86_64 11/180 2024-03-23T11:16:44.741 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:18.2.2-820.gb630d6cd.el9.x86_64 12/180 2024-03-23T11:16:44.811 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/180 2024-03-23T11:16:44.883 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/180 2024-03-23T11:16:44.983 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/180 2024-03-23T11:16:45.078 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/180 2024-03-23T11:16:45.120 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/180 2024-03-23T11:16:45.190 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/180 2024-03-23T11:16:45.217 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephsqlite-2:18.2.2-820.gb630d6cd.el9.x86_64 16/180 2024-03-23T11:16:45.379 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 41/180 2024-03-23T11:16:45.395 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephsqlite-2:18.2.2-820.gb630d6cd.el9.x86_64 16/180 2024-03-23T11:16:45.419 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 41/180 2024-03-23T11:16:45.426 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:18.2.2-820.gb630d6cd.el9.x86_ 17/180 2024-03-23T11:16:45.507 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 42/180 2024-03-23T11:16:45.524 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:18.2.2-820.gb630d6cd.el9.x86_ 17/180 2024-03-23T11:16:45.564 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librados-devel-2:18.2.2-820.gb630d6cd.el9.x86_64 43/180 2024-03-23T11:16:45.573 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/180 2024-03-23T11:16:45.613 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/180 2024-03-23T11:16:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 19/180 2024-03-23T11:16:45.947 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 20/180 2024-03-23T11:16:46.039 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 45/180 2024-03-23T11:16:46.273 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 46/180 2024-03-23T11:16:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 47/180 2024-03-23T11:16:46.392 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 21/180 2024-03-23T11:16:46.417 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 48/180 2024-03-23T11:16:46.517 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 21/180 2024-03-23T11:16:46.667 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/180 2024-03-23T11:16:46.712 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 50/180 2024-03-23T11:16:46.725 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T11:16:46.788 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T11:16:46.869 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : re2-1:20211101-3.el9.x86_64 51/180 2024-03-23T11:16:46.915 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 52/180 2024-03-23T11:16:46.997 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 53/180 2024-03-23T11:16:47.109 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/180 2024-03-23T11:16:47.152 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/180 2024-03-23T11:16:47.181 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T11:16:47.198 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/180 2024-03-23T11:16:47.226 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 23/180 2024-03-23T11:16:47.270 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/180 2024-03-23T11:16:47.287 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.7-2.el9.x86_64 24/180 2024-03-23T11:16:47.331 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/180 2024-03-23T11:16:47.353 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 25/180 2024-03-23T11:16:47.393 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 59/180 2024-03-23T11:16:47.477 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 26/180 2024-03-23T11:16:47.554 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/180 2024-03-23T11:16:47.591 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 60/180 2024-03-23T11:16:47.610 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/180 2024-03-23T11:16:47.638 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 61/180 2024-03-23T11:16:47.686 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 62/180 2024-03-23T11:16:47.731 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/180 2024-03-23T11:16:47.783 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/180 2024-03-23T11:16:47.816 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/180 2024-03-23T11:16:47.821 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 63/180 2024-03-23T11:16:47.862 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 64/180 2024-03-23T11:16:47.908 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 65/180 2024-03-23T11:16:47.928 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 32/180 2024-03-23T11:16:47.945 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 66/180 2024-03-23T11:16:47.971 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:18.2.2-820.gb630d6cd.el9.x 33/180 2024-03-23T11:16:47.983 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 67/180 2024-03-23T11:16:48.015 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:18.2.2-820.gb630d6cd.el9 34/180 2024-03-23T11:16:48.124 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:18.2.2-820.gb630d6cd.el9.x86_64 35/180 2024-03-23T11:16:48.148 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fmt-8.1.1-5.el9.x86_64 68/180 2024-03-23T11:16:48.200 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 69/180 2024-03-23T11:16:48.270 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ply-3.11-14.el9.noarch 70/180 2024-03-23T11:16:48.297 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/180 2024-03-23T11:16:48.347 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 71/180 2024-03-23T11:16:48.365 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/180 2024-03-23T11:16:48.602 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 72/180 2024-03-23T11:16:48.663 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 73/180 2024-03-23T11:16:48.849 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 74/180 2024-03-23T11:16:48.893 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 75/180 2024-03-23T11:16:48.941 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 76/180 2024-03-23T11:16:48.972 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fuse-2.9.9-15.el9.x86_64 77/180 2024-03-23T11:16:49.013 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 78/180 2024-03-23T11:16:49.059 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 79/180 2024-03-23T11:16:49.074 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/180 2024-03-23T11:16:49.112 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : uuid-1.6.2-55.el9.x86_64 80/180 2024-03-23T11:16:49.159 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 81/180 2024-03-23T11:16:49.169 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/180 2024-03-23T11:16:49.292 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 82/180 2024-03-23T11:16:49.303 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/180 2024-03-23T11:16:49.338 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 83/180 2024-03-23T11:16:49.347 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 41/180 2024-03-23T11:16:49.381 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 41/180 2024-03-23T11:16:49.398 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 84/180 2024-03-23T11:16:49.449 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 42/180 2024-03-23T11:16:49.474 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 85/180 2024-03-23T11:16:49.493 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:18.2.2-820.gb630d6cd.el9.x86_64 43/180 2024-03-23T11:16:49.537 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/180 2024-03-23T11:16:49.538 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 86/180 2024-03-23T11:16:49.700 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 87/180 2024-03-23T11:16:49.748 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 88/180 2024-03-23T11:16:49.811 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 89/180 2024-03-23T11:16:49.965 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-packaging-20.9-5.el9.noarch 90/180 2024-03-23T11:16:50.019 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 45/180 2024-03-23T11:16:50.030 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 91/180 2024-03-23T11:16:50.248 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 46/180 2024-03-23T11:16:50.372 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 47/180 2024-03-23T11:16:50.441 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 48/180 2024-03-23T11:16:50.648 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/180 2024-03-23T11:16:50.696 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 50/180 2024-03-23T11:16:50.840 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : re2-1:20211101-3.el9.x86_64 51/180 2024-03-23T11:16:50.891 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 52/180 2024-03-23T11:16:50.955 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 53/180 2024-03-23T11:16:51.028 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/180 2024-03-23T11:16:51.041 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/180 2024-03-23T11:16:51.079 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 93/180 2024-03-23T11:16:51.100 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/180 2024-03-23T11:16:51.113 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 94/180 2024-03-23T11:16:51.146 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 95/180 2024-03-23T11:16:51.160 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/180 2024-03-23T11:16:51.239 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/180 2024-03-23T11:16:51.308 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/180 2024-03-23T11:16:51.368 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 59/180 2024-03-23T11:16:51.401 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-0.3.21-2.el9.x86_64 96/180 2024-03-23T11:16:51.424 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 97/180 2024-03-23T11:16:51.527 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 98/180 2024-03-23T11:16:51.558 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 60/180 2024-03-23T11:16:51.607 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 61/180 2024-03-23T11:16:51.651 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 62/180 2024-03-23T11:16:51.795 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 63/180 2024-03-23T11:16:51.814 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 64/180 2024-03-23T11:16:51.866 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 65/180 2024-03-23T11:16:51.915 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 66/180 2024-03-23T11:16:51.960 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 67/180 2024-03-23T11:16:52.116 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fmt-8.1.1-5.el9.x86_64 68/180 2024-03-23T11:16:52.170 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 69/180 2024-03-23T11:16:52.236 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ply-3.11-14.el9.noarch 70/180 2024-03-23T11:16:52.346 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 71/180 2024-03-23T11:16:52.402 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 99/180 2024-03-23T11:16:52.600 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 72/180 2024-03-23T11:16:52.658 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 73/180 2024-03-23T11:16:52.783 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 100/180 2024-03-23T11:16:52.831 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 74/180 2024-03-23T11:16:52.889 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 75/180 2024-03-23T11:16:52.929 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 76/180 2024-03-23T11:16:52.967 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.9-15.el9.x86_64 77/180 2024-03-23T11:16:53.008 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 78/180 2024-03-23T11:16:53.055 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 79/180 2024-03-23T11:16:53.140 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : uuid-1.6.2-55.el9.x86_64 80/180 2024-03-23T11:16:53.189 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 81/180 2024-03-23T11:16:53.316 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 82/180 2024-03-23T11:16:53.358 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 83/180 2024-03-23T11:16:53.392 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 84/180 2024-03-23T11:16:53.444 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 85/180 2024-03-23T11:16:53.505 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 86/180 2024-03-23T11:16:53.657 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 87/180 2024-03-23T11:16:53.703 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 88/180 2024-03-23T11:16:53.764 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 89/180 2024-03-23T11:16:53.913 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-packaging-20.9-5.el9.noarch 90/180 2024-03-23T11:16:53.977 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 91/180 2024-03-23T11:16:54.443 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 101/180 2024-03-23T11:16:54.538 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 102/180 2024-03-23T11:16:54.575 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 103/180 2024-03-23T11:16:54.606 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : jq-1.6-16.el9.x86_64 104/180 2024-03-23T11:16:54.633 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 105/180 2024-03-23T11:16:54.937 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/180 2024-03-23T11:16:54.992 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 93/180 2024-03-23T11:16:55.001 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 106/180 2024-03-23T11:16:55.036 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librbd1-2:18.2.2-820.gb630d6cd.el9.x86_64 107/180 2024-03-23T11:16:55.044 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 94/180 2024-03-23T11:16:55.085 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 95/180 2024-03-23T11:16:55.190 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:18.2.2-820.gb630d6cd.el9.x86_64 107/180 2024-03-23T11:16:55.225 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rbd-2:18.2.2-820.gb630d6cd.el9.x86_64 108/180 2024-03-23T11:16:55.292 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 109/180 2024-03-23T11:16:55.332 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-0.3.21-2.el9.x86_64 96/180 2024-03-23T11:16:55.373 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 97/180 2024-03-23T11:16:55.522 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 98/180 2024-03-23T11:16:55.771 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 110/180 2024-03-23T11:16:55.822 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librgw2-2:18.2.2-820.gb630d6cd.el9.x86_64 111/180 2024-03-23T11:16:55.948 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librgw2-2:18.2.2-820.gb630d6cd.el9.x86_64 111/180 2024-03-23T11:16:55.974 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rgw-2:18.2.2-820.gb630d6cd.el9.x86_64 112/180 2024-03-23T11:16:56.316 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 99/180 2024-03-23T11:16:56.450 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 100/180 2024-03-23T11:16:57.970 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 101/180 2024-03-23T11:16:58.061 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 102/180 2024-03-23T11:16:58.127 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 103/180 2024-03-23T11:16:58.163 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : jq-1.6-16.el9.x86_64 104/180 2024-03-23T11:16:58.206 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 105/180 2024-03-23T11:16:58.425 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 113/180 2024-03-23T11:16:58.470 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 113/180 2024-03-23T11:16:58.575 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 106/180 2024-03-23T11:16:58.610 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:18.2.2-820.gb630d6cd.el9.x86_64 107/180 2024-03-23T11:16:58.721 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:18.2.2-820.gb630d6cd.el9.x86_64 107/180 2024-03-23T11:16:58.749 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:18.2.2-820.gb630d6cd.el9.x86_64 108/180 2024-03-23T11:16:58.803 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 109/180 2024-03-23T11:16:59.148 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 113/180 2024-03-23T11:16:59.190 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-base-2:18.2.2-820.gb630d6cd.el9.x86_64 114/180 2024-03-23T11:16:59.259 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 110/180 2024-03-23T11:16:59.297 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:18.2.2-820.gb630d6cd.el9.x86_64 111/180 2024-03-23T11:16:59.358 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-base-2:18.2.2-820.gb630d6cd.el9.x86_64 114/180 2024-03-23T11:16:59.358 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-23T11:16:59.358 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-23T11:16:59.359 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:16:59.387 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:18.2.2-820.gb630d6cd.el9.x86_64 111/180 2024-03-23T11:16:59.408 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-selinux-2:18.2.2-820.gb630d6cd.el9.x86_64 115/180 2024-03-23T11:16:59.415 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:18.2.2-820.gb630d6cd.el9.x86_64 112/180 2024-03-23T11:17:01.870 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 113/180 2024-03-23T11:17:01.912 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 113/180 2024-03-23T11:17:02.551 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 113/180 2024-03-23T11:17:02.599 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:18.2.2-820.gb630d6cd.el9.x86_64 114/180 2024-03-23T11:17:02.748 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:18.2.2-820.gb630d6cd.el9.x86_64 114/180 2024-03-23T11:17:02.748 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-23T11:17:02.748 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-23T11:17:02.748 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:02.800 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:18.2.2-820.gb630d6cd.el9.x86_64 115/180 2024-03-23T11:17:12.566 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-selinux-2:18.2.2-820.gb630d6cd.el9.x86_64 115/180 2024-03-23T11:17:12.566 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /sys 2024-03-23T11:17:12.566 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /proc 2024-03-23T11:17:12.566 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /mnt 2024-03-23T11:17:12.566 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /var/tmp 2024-03-23T11:17:12.566 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /home 2024-03-23T11:17:12.566 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /root 2024-03-23T11:17:12.566 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /tmp 2024-03-23T11:17:12.567 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:12.796 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-volume-2:18.2.2-820.gb630d6cd.el9.noarch 116/180 2024-03-23T11:17:12.851 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-volume-2:18.2.2-820.gb630d6cd.el9.noarch 116/180 2024-03-23T11:17:12.851 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:12.851 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-23T11:17:12.851 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:14.615 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-osd-2:18.2.2-820.gb630d6cd.el9.x86_64 117/180 2024-03-23T11:17:14.702 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-osd-2:18.2.2-820.gb630d6cd.el9.x86_64 117/180 2024-03-23T11:17:14.702 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:14.702 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-23T11:17:14.702 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T11:17:14.702 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T11:17:14.703 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:14.957 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mds-2:18.2.2-820.gb630d6cd.el9.x86_64 118/180 2024-03-23T11:17:15.035 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mds-2:18.2.2-820.gb630d6cd.el9.x86_64 118/180 2024-03-23T11:17:15.035 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:15.035 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-23T11:17:15.035 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T11:17:15.036 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T11:17:15.036 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:15.535 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:18.2.2-820.gb630d6cd.el9.x86_64 115/180 2024-03-23T11:17:15.535 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2024-03-23T11:17:15.535 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2024-03-23T11:17:15.535 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2024-03-23T11:17:15.535 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2024-03-23T11:17:15.535 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2024-03-23T11:17:15.535 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /root 2024-03-23T11:17:15.535 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2024-03-23T11:17:15.535 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:15.566 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mon-2:18.2.2-820.gb630d6cd.el9.x86_64 119/180 2024-03-23T11:17:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mon-2:18.2.2-820.gb630d6cd.el9.x86_64 119/180 2024-03-23T11:17:15.641 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:15.641 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-23T11:17:15.641 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T11:17:15.641 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T11:17:15.641 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:15.760 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : automake-1.16.2-8.el9.noarch 120/180 2024-03-23T11:17:15.776 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-volume-2:18.2.2-820.gb630d6cd.el9.noarch 116/180 2024-03-23T11:17:15.826 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-volume-2:18.2.2-820.gb630d6cd.el9.noarch 116/180 2024-03-23T11:17:15.826 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:15.826 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-23T11:17:15.826 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:16.390 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:18.2.2-820.gb630d6cd.el9.noarch 121/180 2024-03-23T11:17:16.428 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephadm-2:18.2.2-820.gb630d6cd.el9.noarch 121/180 2024-03-23T11:17:16.474 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-820.gb630d6cd.el 122/180 2024-03-23T11:17:16.583 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-820.gb630d6cd.e 123/180 2024-03-23T11:17:16.614 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-820.gb630d6cd.el9.noar 124/180 2024-03-23T11:17:16.951 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-820.gb630d6cd.el9.noar 124/180 2024-03-23T11:17:16.998 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-820.gb630d6cd.el9.no 125/180 2024-03-23T11:17:17.587 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:18.2.2-820.gb630d6cd.el9.x86_64 117/180 2024-03-23T11:17:17.692 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:18.2.2-820.gb630d6cd.el9.x86_64 117/180 2024-03-23T11:17:17.692 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:17.692 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-23T11:17:17.692 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T11:17:17.692 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T11:17:17.692 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:17.929 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:18.2.2-820.gb630d6cd.el9.x86_64 118/180 2024-03-23T11:17:18.023 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:18.2.2-820.gb630d6cd.el9.x86_64 118/180 2024-03-23T11:17:18.023 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:18.023 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-23T11:17:18.023 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T11:17:18.023 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T11:17:18.024 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:18.170 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-820.gb630d6cd.el9.no 125/180 2024-03-23T11:17:18.199 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-820.gb630 126/180 2024-03-23T11:17:18.347 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-820.gb630 126/180 2024-03-23T11:17:18.521 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:18.2.2-820.gb630d6cd.el9.x86_64 119/180 2024-03-23T11:17:18.522 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-820.gb630d6cd.el9 127/180 2024-03-23T11:17:18.560 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-2:18.2.2-820.gb630d6cd.el9.x86_64 128/180 2024-03-23T11:17:18.618 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:18.2.2-820.gb630d6cd.el9.x86_64 119/180 2024-03-23T11:17:18.618 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:18.618 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-23T11:17:18.619 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T11:17:18.619 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T11:17:18.619 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:18.641 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-2:18.2.2-820.gb630d6cd.el9.x86_64 128/180 2024-03-23T11:17:18.642 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:18.642 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-23T11:17:18.642 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T11:17:18.642 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T11:17:18.642 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:18.702 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-rook-2:18.2.2-820.gb630d6cd.el9.noarch 129/180 2024-03-23T11:17:18.740 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : automake-1.16.2-8.el9.noarch 120/180 2024-03-23T11:17:18.743 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-820.gb630d6cd.el9.noarch 129/180 2024-03-23T11:17:18.837 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-2:18.2.2-820.gb630d6cd.el9.x86_64 130/180 2024-03-23T11:17:19.405 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:18.2.2-820.gb630d6cd.el9.noarch 121/180 2024-03-23T11:17:19.450 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:18.2.2-820.gb630d6cd.el9.noarch 121/180 2024-03-23T11:17:19.496 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-820.gb630d6cd.el 122/180 2024-03-23T11:17:19.604 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-820.gb630d6cd.e 123/180 2024-03-23T11:17:19.637 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-820.gb630d6cd.el9.noar 124/180 2024-03-23T11:17:19.693 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libtool-2.4.6-45.el9.x86_64 131/180 2024-03-23T11:17:19.728 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-radosgw-2:18.2.2-820.gb630d6cd.el9.x86_64 132/180 2024-03-23T11:17:19.821 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-820.gb630d6cd.el9.x86_64 132/180 2024-03-23T11:17:19.822 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:19.822 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-23T11:17:19.822 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T11:17:19.822 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T11:17:19.822 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:19.871 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-820.gb630d6 133/180 2024-03-23T11:17:19.947 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-820.gb630d6cd.el9.noar 124/180 2024-03-23T11:17:19.969 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-820.gb630d6 133/180 2024-03-23T11:17:19.969 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:19.969 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-23T11:17:19.969 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:20.007 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-820.gb630d6cd.el9.no 125/180 2024-03-23T11:17:20.063 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephfs-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 134/180 2024-03-23T11:17:20.144 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 134/180 2024-03-23T11:17:20.145 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:20.145 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-23T11:17:20.145 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T11:17:20.145 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T11:17:20.145 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:20.514 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 135/180 2024-03-23T11:17:20.601 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 135/180 2024-03-23T11:17:20.601 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:20.601 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-23T11:17:20.601 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T11:17:20.601 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T11:17:20.601 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:21.156 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-820.gb630d6cd.el9.no 125/180 2024-03-23T11:17:21.188 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-820.gb630 126/180 2024-03-23T11:17:21.328 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-820.gb630 126/180 2024-03-23T11:17:21.506 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-820.gb630d6cd.el9 127/180 2024-03-23T11:17:21.566 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:18.2.2-820.gb630d6cd.el9.x86_64 128/180 2024-03-23T11:17:21.657 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:18.2.2-820.gb630d6cd.el9.x86_64 128/180 2024-03-23T11:17:21.657 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:21.657 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-23T11:17:21.657 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T11:17:21.657 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T11:17:21.657 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:21.717 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-rook-2:18.2.2-820.gb630d6cd.el9.noarch 129/180 2024-03-23T11:17:21.755 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-820.gb630d6cd.el9.noarch 129/180 2024-03-23T11:17:21.846 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:18.2.2-820.gb630d6cd.el9.x86_64 130/180 2024-03-23T11:17:22.692 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libtool-2.4.6-45.el9.x86_64 131/180 2024-03-23T11:17:22.749 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:18.2.2-820.gb630d6cd.el9.x86_64 132/180 2024-03-23T11:17:22.831 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-820.gb630d6cd.el9.x86_64 132/180 2024-03-23T11:17:22.831 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:22.831 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-23T11:17:22.831 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T11:17:22.832 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T11:17:22.832 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:22.886 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-820.gb630d6 133/180 2024-03-23T11:17:22.965 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-820.gb630d6 133/180 2024-03-23T11:17:22.965 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:22.965 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-23T11:17:22.965 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:23.037 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 134/180 2024-03-23T11:17:23.127 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 134/180 2024-03-23T11:17:23.127 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:23.127 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-23T11:17:23.127 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T11:17:23.127 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T11:17:23.127 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:23.496 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 135/180 2024-03-23T11:17:23.570 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 135/180 2024-03-23T11:17:23.570 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T11:17:23.570 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-23T11:17:23.570 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T11:17:23.570 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T11:17:23.570 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:24.830 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-test-2:18.2.2-820.gb630d6cd.el9.x86_64 136/180 2024-03-23T11:17:24.886 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 137/180 2024-03-23T11:17:24.927 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-nbd-2:18.2.2-820.gb630d6cd.el9.x86_64 138/180 2024-03-23T11:17:25.231 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 139/180 2024-03-23T11:17:25.354 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 140/180 2024-03-23T11:17:25.403 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 141/180 2024-03-23T11:17:25.459 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/180 2024-03-23T11:17:25.526 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 143/180 2024-03-23T11:17:25.597 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 144/180 2024-03-23T11:17:25.642 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs-devel-2:18.2.2-820.gb630d6cd.el9.x86_6 145/180 2024-03-23T11:17:25.694 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 146/180 2024-03-23T11:17:25.746 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 147/180 2024-03-23T11:17:25.837 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 148/180 2024-03-23T11:17:25.943 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 149/180 2024-03-23T11:17:25.976 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 150/180 2024-03-23T11:17:26.901 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T11:17:26.940 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T11:17:27.000 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T11:17:27.661 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:18.2.2-820.gb630d6cd.el9.x86_64 136/180 2024-03-23T11:17:27.705 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 137/180 2024-03-23T11:17:27.765 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:18.2.2-820.gb630d6cd.el9.x86_64 138/180 2024-03-23T11:17:28.040 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 139/180 2024-03-23T11:17:28.160 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 140/180 2024-03-23T11:17:28.209 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 141/180 2024-03-23T11:17:28.269 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/180 2024-03-23T11:17:28.326 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 143/180 2024-03-23T11:17:28.372 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 144/180 2024-03-23T11:17:28.406 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:18.2.2-820.gb630d6cd.el9.x86_6 145/180 2024-03-23T11:17:28.428 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephfs-top-2:18.2.2-820.gb630d6cd.el9.noarch 152/180 2024-03-23T11:17:28.452 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 146/180 2024-03-23T11:17:28.503 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 147/180 2024-03-23T11:17:28.538 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 153/180 2024-03-23T11:17:28.589 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/180 2024-03-23T11:17:28.640 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 148/180 2024-03-23T11:17:28.731 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/180 2024-03-23T11:17:28.747 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 149/180 2024-03-23T11:17:28.765 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/180 2024-03-23T11:17:28.786 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 150/180 2024-03-23T11:17:28.797 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 157/180 2024-03-23T11:17:28.832 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 158/180 2024-03-23T11:17:28.874 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 159/180 2024-03-23T11:17:28.874 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/180 2024-03-23T11:17:29.021 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/180 2024-03-23T11:17:29.061 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 161/180 2024-03-23T11:17:29.061 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 162/180 2024-03-23T11:17:29.183 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 162/180 2024-03-23T11:17:29.253 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 163/180 2024-03-23T11:17:29.323 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 164/180 2024-03-23T11:17:29.356 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 165/180 2024-03-23T11:17:29.389 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 166/180 2024-03-23T11:17:29.461 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 167/180 2024-03-23T11:17:29.528 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 168/180 2024-03-23T11:17:29.585 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 169/180 2024-03-23T11:17:29.618 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 170/180 2024-03-23T11:17:29.660 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 171/180 2024-03-23T11:17:29.694 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 172/180 2024-03-23T11:17:29.715 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T11:17:29.727 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 173/180 2024-03-23T11:17:29.764 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T11:17:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T11:17:29.767 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T11:17:29.824 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T11:17:30.103 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T11:17:30.103 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 175/180 2024-03-23T11:17:30.225 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 175/180 2024-03-23T11:17:30.255 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 176/180 2024-03-23T11:17:30.314 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 177/180 2024-03-23T11:17:30.350 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 178/180 2024-03-23T11:17:30.390 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 179/180 2024-03-23T11:17:30.518 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 180/180 2024-03-23T11:17:30.783 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 180/180 2024-03-23T11:17:30.783 INFO:teuthology.orchestra.run.smithi062.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-23T11:17:30.784 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:31.126 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-top-2:18.2.2-820.gb630d6cd.el9.noarch 152/180 2024-03-23T11:17:31.234 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 153/180 2024-03-23T11:17:31.306 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/180 2024-03-23T11:17:31.420 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/180 2024-03-23T11:17:31.461 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/180 2024-03-23T11:17:31.486 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 157/180 2024-03-23T11:17:31.512 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 158/180 2024-03-23T11:17:31.553 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 159/180 2024-03-23T11:17:31.553 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/180 2024-03-23T11:17:31.724 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/180 2024-03-23T11:17:31.766 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 161/180 2024-03-23T11:17:31.767 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 162/180 2024-03-23T11:17:31.905 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 162/180 2024-03-23T11:17:31.959 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 163/180 2024-03-23T11:17:32.012 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 164/180 2024-03-23T11:17:32.078 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 165/180 2024-03-23T11:17:32.111 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 166/180 2024-03-23T11:17:32.179 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 167/180 2024-03-23T11:17:32.258 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 168/180 2024-03-23T11:17:32.308 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 169/180 2024-03-23T11:17:32.349 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 170/180 2024-03-23T11:17:32.383 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 171/180 2024-03-23T11:17:32.437 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 172/180 2024-03-23T11:17:32.476 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 173/180 2024-03-23T11:17:32.510 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T11:17:32.511 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T11:17:32.850 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T11:17:32.851 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 175/180 2024-03-23T11:17:32.981 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 175/180 2024-03-23T11:17:33.012 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 176/180 2024-03-23T11:17:33.062 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 177/180 2024-03-23T11:17:33.098 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 178/180 2024-03-23T11:17:33.138 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 179/180 2024-03-23T11:17:33.219 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 180/180 2024-03-23T11:17:33.477 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 180/180 2024-03-23T11:17:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-23T11:17:33.477 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:35.380 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-2:18.2.2-820.gb630d6cd.el9.x86_64 1/180 2024-03-23T11:17:35.380 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-base-2:18.2.2-820.gb630d6cd.el9.x86_64 2/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 3/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 4/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-820.gb630d6 5/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mds-2:18.2.2-820.gb630d6cd.el9.x86_64 6/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-2:18.2.2-820.gb630d6cd.el9.x86_64 7/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mon-2:18.2.2-820.gb630d6cd.el9.x86_64 8/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-osd-2:18.2.2-820.gb630d6cd.el9.x86_64 9/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-radosgw-2:18.2.2-820.gb630d6cd.el9.x86_64 10/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-selinux-2:18.2.2-820.gb630d6cd.el9.x86_64 11/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-test-2:18.2.2-820.gb630d6cd.el9.x86_64 12/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephfs-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 13/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs-devel-2:18.2.2-820.gb630d6cd.el9.x86_6 14/180 2024-03-23T11:17:35.381 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs2-2:18.2.2-820.gb630d6cd.el9.x86_64 15/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephsqlite-2:18.2.2-820.gb630d6cd.el9.x86_64 16/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados-devel-2:18.2.2-820.gb630d6cd.el9.x86_64 17/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libradosstriper1-2:18.2.2-820.gb630d6cd.el9.x86_ 18/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librgw2-2:18.2.2-820.gb630d6cd.el9.x86_64 19/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-argparse-2:18.2.2-820.gb630d6cd.el9 20/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-common-2:18.2.2-820.gb630d6cd.el9.x 21/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cephfs-2:18.2.2-820.gb630d6cd.el9.x86_64 22/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rados-2:18.2.2-820.gb630d6cd.el9.x86_64 23/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rbd-2:18.2.2-820.gb630d6cd.el9.x86_64 24/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rgw-2:18.2.2-820.gb630d6cd.el9.x86_64 25/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 26/180 2024-03-23T11:17:35.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 27/180 2024-03-23T11:17:35.395 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-nbd-2:18.2.2-820.gb630d6cd.el9.x86_64 28/180 2024-03-23T11:17:35.395 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-820.gb630d6cd.e 29/180 2024-03-23T11:17:35.395 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-820.gb630d6cd.el9.noar 30/180 2024-03-23T11:17:35.395 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-820.gb630d6cd.el9.no 31/180 2024-03-23T11:17:35.395 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-820.gb630 32/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-820.gb630d6cd.el9 33/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-rook-2:18.2.2-820.gb630d6cd.el9.noarch 34/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-820.gb630d6cd.el 35/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-volume-2:18.2.2-820.gb630d6cd.el9.noarch 36/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephadm-2:18.2.2-820.gb630d6cd.el9.noarch 37/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephfs-top-2:18.2.2-820.gb630d6cd.el9.noarch 38/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flex-2.6.4-9.el9.x86_64 42/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 43/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 44/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 45/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 46/180 2024-03-23T11:17:35.396 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 47/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 48/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 49/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 50/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 51/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 52/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 53/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 54/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 55/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-13.11-1.el9.x86_64 56/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 57/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 58/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 59/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/180 2024-03-23T11:17:35.397 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 70/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 71/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 72/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 73/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 74/180 2024-03-23T11:17:35.398 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 75/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 76/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 77/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 78/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 79/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 80/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 81/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 82/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ply-3.11-14.el9.noarch 83/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 84/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 85/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : jq-1.6-16.el9.x86_64 86/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 87/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 88/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 89/180 2024-03-23T11:17:35.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 90/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 91/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 92/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 93/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 94/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 95/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 96/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 102/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 103/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 104/180 2024-03-23T11:17:35.400 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 105/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 106/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 107/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 108/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 109/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 110/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 111/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 112/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 113/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 114/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 115/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 116/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 117/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 118/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 119/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 120/180 2024-03-23T11:17:35.401 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 122/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 123/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 124/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 125/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 126/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 127/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 128/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 129/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 130/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : re2-1:20211101-3.el9.x86_64 131/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 132/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:18.2.2-820.gb630d6cd.el9.x86_64 133/180 2024-03-23T11:17:35.402 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 134/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:18.2.2-820.gb630d6cd.el9.x86_64 135/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 136/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 137/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 138/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 139/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 140/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 141/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 142/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 143/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 144/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 145/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 146/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : acl-2.3.1-4.el9.x86_64 147/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : acl-2.3.1-3.el9.x86_64 148/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 149/180 2024-03-23T11:17:35.403 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 150/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 151/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 152/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 153/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 154/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 155/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 156/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 157/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 158/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 159/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 160/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 161/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 162/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 163/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 164/180 2024-03-23T11:17:35.404 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 165/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 166/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 167/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 168/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 169/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 170/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 171/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 172/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 173/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 174/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 175/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 176/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 177/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 178/180 2024-03-23T11:17:35.405 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 179/180 2024-03-23T11:17:37.934 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 180/180 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout:Upgraded: 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout: acl-2.3.1-4.el9.x86_64 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-23T11:17:37.935 INFO:teuthology.orchestra.run.smithi062.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: librados2-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-23T11:17:37.936 INFO:teuthology.orchestra.run.smithi062.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout: automake-1.16.2-8.el9.noarch 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout: bison-3.7.4-5.el9.x86_64 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.937 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.938 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-top-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: flex-2.6.4-9.el9.x86_64 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-23T11:17:37.939 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: jq-1.6-16.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-23T11:17:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-13.11-1.el9.x86_64 2024-03-23T11:17:37.941 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-23T11:17:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-23T11:17:37.943 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply-3.11-14.el9.noarch 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-23T11:17:37.944 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-23T11:17:37.945 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: re2-1:20211101-3.el9.x86_64 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-23T11:17:37.946 INFO:teuthology.orchestra.run.smithi062.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-23T11:17:37.947 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-23T11:17:37.947 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-23T11:17:37.947 INFO:teuthology.orchestra.run.smithi062.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-23T11:17:37.947 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:37.947 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-23T11:17:38.161 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:18.2.2-820.gb630d6cd.el9.x86_64 1/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:18.2.2-820.gb630d6cd.el9.x86_64 2/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 3/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 4/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-820.gb630d6 5/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:18.2.2-820.gb630d6cd.el9.x86_64 6/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:18.2.2-820.gb630d6cd.el9.x86_64 7/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:18.2.2-820.gb630d6cd.el9.x86_64 8/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:18.2.2-820.gb630d6cd.el9.x86_64 9/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:18.2.2-820.gb630d6cd.el9.x86_64 10/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:18.2.2-820.gb630d6cd.el9.x86_64 11/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:18.2.2-820.gb630d6cd.el9.x86_64 12/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 13/180 2024-03-23T11:17:38.162 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:18.2.2-820.gb630d6cd.el9.x86_6 14/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:18.2.2-820.gb630d6cd.el9.x86_64 15/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephsqlite-2:18.2.2-820.gb630d6cd.el9.x86_64 16/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:18.2.2-820.gb630d6cd.el9.x86_64 17/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:18.2.2-820.gb630d6cd.el9.x86_ 18/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:18.2.2-820.gb630d6cd.el9.x86_64 19/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:18.2.2-820.gb630d6cd.el9 20/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:18.2.2-820.gb630d6cd.el9.x 21/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:18.2.2-820.gb630d6cd.el9.x86_64 22/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:18.2.2-820.gb630d6cd.el9.x86_64 23/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:18.2.2-820.gb630d6cd.el9.x86_64 24/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:18.2.2-820.gb630d6cd.el9.x86_64 25/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 26/180 2024-03-23T11:17:38.163 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 27/180 2024-03-23T11:17:38.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:18.2.2-820.gb630d6cd.el9.x86_64 28/180 2024-03-23T11:17:38.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-820.gb630d6cd.e 29/180 2024-03-23T11:17:38.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-820.gb630d6cd.el9.noar 30/180 2024-03-23T11:17:38.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-820.gb630d6cd.el9.no 31/180 2024-03-23T11:17:38.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-820.gb630 32/180 2024-03-23T11:17:38.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-820.gb630d6cd.el9 33/180 2024-03-23T11:17:38.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-rook-2:18.2.2-820.gb630d6cd.el9.noarch 34/180 2024-03-23T11:17:38.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-820.gb630d6cd.el 35/180 2024-03-23T11:17:38.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-volume-2:18.2.2-820.gb630d6cd.el9.noarch 36/180 2024-03-23T11:17:38.176 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:18.2.2-820.gb630d6cd.el9.noarch 37/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-top-2:18.2.2-820.gb630d6cd.el9.noarch 38/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flex-2.6.4-9.el9.x86_64 42/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 43/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 44/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 45/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 46/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 47/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 48/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 49/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 50/180 2024-03-23T11:17:38.177 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 51/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 52/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 53/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 54/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 55/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-13.11-1.el9.x86_64 56/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 57/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 58/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 59/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/180 2024-03-23T11:17:38.178 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 70/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 71/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 72/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 73/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 74/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 75/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 76/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 77/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 78/180 2024-03-23T11:17:38.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 79/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 80/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 81/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 82/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ply-3.11-14.el9.noarch 83/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 84/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 85/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : jq-1.6-16.el9.x86_64 86/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 87/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 88/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 89/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 90/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 91/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 92/180 2024-03-23T11:17:38.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 93/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 94/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 95/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 96/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 102/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 103/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 104/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 105/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 106/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 107/180 2024-03-23T11:17:38.181 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 108/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 109/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 110/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 111/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 112/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 113/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 114/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 115/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 116/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 117/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 118/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 119/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 120/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/180 2024-03-23T11:17:38.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 122/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 123/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 124/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 125/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 126/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 127/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 128/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 129/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 130/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : re2-1:20211101-3.el9.x86_64 131/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 132/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:18.2.2-820.gb630d6cd.el9.x86_64 133/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 134/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:18.2.2-820.gb630d6cd.el9.x86_64 135/180 2024-03-23T11:17:38.183 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 136/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 137/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 138/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 139/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 140/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 141/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 142/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 143/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 144/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 145/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 146/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : acl-2.3.1-4.el9.x86_64 147/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : acl-2.3.1-3.el9.x86_64 148/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 149/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 150/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 151/180 2024-03-23T11:17:38.184 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 152/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 153/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 154/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 155/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 156/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 157/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 158/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 159/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 160/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 161/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 162/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 163/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 164/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 165/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 166/180 2024-03-23T11:17:38.185 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 167/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 168/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 169/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 170/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 171/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 172/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 173/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 174/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 175/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 176/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 177/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 178/180 2024-03-23T11:17:38.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 179/180 2024-03-23T11:17:38.562 DEBUG:teuthology.parallel:result is None 2024-03-23T11:17:40.557 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 180/180 2024-03-23T11:17:40.557 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:40.557 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: acl-2.3.1-4.el9.x86_64 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-23T11:17:40.558 INFO:teuthology.orchestra.run.smithi120.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-23T11:17:40.559 INFO:teuthology.orchestra.run.smithi120.stdout: automake-1.16.2-8.el9.noarch 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: bison-3.7.4-5.el9.x86_64 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:40.560 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top-2:18.2.2-820.gb630d6cd.el9.noarch 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: flex-2.6.4-9.el9.x86_64 2024-03-23T11:17:40.561 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: jq-1.6-16.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.562 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-23T11:17:40.563 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-13.11-1.el9.x86_64 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-23T11:17:40.564 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-23T11:17:40.565 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply-3.11-14.el9.noarch 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-23T11:17:40.566 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-23T11:17:40.567 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:18.2.2-820.gb630d6cd.el9.x86_64 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: re2-1:20211101-3.el9.x86_64 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-23T11:17:40.568 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-23T11:17:40.569 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-23T11:17:40.569 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-23T11:17:40.569 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:40.569 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-23T11:17:41.263 DEBUG:teuthology.parallel:result is None 2024-03-23T11:17:41.264 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:17:41.484 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T11:17:41.542 INFO:teuthology.orchestra.run.smithi062.stdout:18.2.2-820.gb630d6cd.el9 2024-03-23T11:17:41.543 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-820.gb630d6cd.el9 2024-03-23T11:17:41.543 INFO:teuthology.task.install:The correct ceph version 18.2.2-820.gb630d6cd is installed. 2024-03-23T11:17:41.545 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b630d6cd702af63dfefcf906294a092ddce1c609 2024-03-23T11:17:41.685 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T11:17:41.745 INFO:teuthology.orchestra.run.smithi120.stdout:18.2.2-820.gb630d6cd.el9 2024-03-23T11:17:41.746 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-820.gb630d6cd.el9 2024-03-23T11:17:41.746 INFO:teuthology.task.install:The correct ceph version 18.2.2-820.gb630d6cd is installed. 2024-03-23T11:17:41.748 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-23T11:17:41.748 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:17:41.748 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T11:17:41.831 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:17:41.831 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T11:17:41.911 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-23T11:17:41.911 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:17:41.911 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T11:17:41.983 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T11:17:42.096 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:17:42.096 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T11:17:42.166 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T11:17:42.281 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-23T11:17:42.281 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:17:42.282 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T11:17:42.358 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T11:17:42.474 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:17:42.475 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T11:17:42.556 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T11:17:42.673 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-23T11:17:42.673 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:17:42.673 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T11:17:42.749 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T11:17:42.864 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:17:42.881 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T11:17:42.960 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T11:17:43.073 INFO:teuthology.run_tasks:Running task ceph... 2024-03-23T11:17:43.140 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-23T11:17:43.140 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /var/log/ceph 2024-03-23T11:17:43.146 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 777 /var/log/ceph 2024-03-23T11:17:43.220 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-23T11:17:43.220 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-23T11:17:43.225 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-23T11:17:43.298 INFO:tasks.ceph:Creating extra log directories... 2024-03-23T11:17:43.298 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-23T11:17:43.302 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-23T11:17:43.383 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-23T11:17:43.383 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-23T11:17:43.383 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-23T11:17:43.383 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:17:43.384 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-23T11:17:43.422 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-23T11:17:43.622 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-23T11:17:43.691 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:17:43.691 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-23T11:17:43.735 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-23T11:17:43.939 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-23T11:17:44.025 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-23T11:17:44.026 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-23T11:17:44.026 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234', 'branch': 'rishabh-yuri-RunCephCmd-base-reef-2', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub}', 'email': None, 'first_in_suite': False, 'job_id': '7618234', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'rishabh-yuri-RunCephCmd-base-reef-2'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609'}}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-yuri-RunCephCmd-base-reef-2', 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1956, 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'rishabh-yuri-RunCephCmd-base-reef-2', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_b630d6cd702af63dfefcf906294a092ddce1c609/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi062.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBfVklLINf8hI4SjP+XzEWierEPkHx2H2V2ejczUNn1z4nGxBS+XfW1RinZqevtqk6b+cm2PeBcbUbgepxvTjw=', 'smithi120.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMeZfkagDYOHSU0NTiM1iySUjN7prNazzwfIgxQoca6hVKsAjI6H2iOKJJ613zG69zBqLzNGYi5FLgQyo/nSRB4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': 'b630d6cd702af63dfefcf906294a092ddce1c609', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_forward_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-23_09:38:22', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-23T11:17:44.026 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-23T11:17:44.035 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-23T11:17:44.086 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T11:17:44.090 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T11:17:44.192 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:17:44.192 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T11:17:44.253 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T11:17:44.253 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2024-03-23T11:17:44.308 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T11:17:44.308 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:17:44.308 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-03-23T11:17:44.308 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:17:44.308 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:17:44.309 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-23 11:17:31.240710537 +0000 2024-03-23T11:17:44.309 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-23 11:15:12.919314983 +0000 2024-03-23T11:17:44.309 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-23 11:15:12.919314983 +0000 2024-03-23T11:17:44.309 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-23 11:15:12.919314983 +0000 2024-03-23T11:17:44.309 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T11:17:44.400 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-23T11:17:44.400 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-23T11:17:44.400 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000433969 s, 1.2 MB/s 2024-03-23T11:17:44.404 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T11:17:44.466 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2024-03-23T11:17:44.519 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T11:17:44.519 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:17:44.519 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-23T11:17:44.520 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:17:44.520 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:17:44.520 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-23 11:17:31.241710521 +0000 2024-03-23T11:17:44.520 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-23 11:15:13.552304683 +0000 2024-03-23T11:17:44.520 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-23 11:15:13.552304683 +0000 2024-03-23T11:17:44.520 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-23 11:15:13.552304683 +0000 2024-03-23T11:17:44.520 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T11:17:44.613 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-23T11:17:44.613 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-23T11:17:44.613 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000323349 s, 1.6 MB/s 2024-03-23T11:17:44.618 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T11:17:44.677 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2024-03-23T11:17:44.725 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T11:17:44.725 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:17:44.725 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-03-23T11:17:44.725 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:17:44.725 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:17:44.725 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-23 11:17:31.242710505 +0000 2024-03-23T11:17:44.725 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-23 11:15:14.140295115 +0000 2024-03-23T11:17:44.725 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-23 11:15:14.140295115 +0000 2024-03-23T11:17:44.725 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-23 11:15:14.140295115 +0000 2024-03-23T11:17:44.725 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T11:17:44.862 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-23T11:17:44.862 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-23T11:17:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000337169 s, 1.5 MB/s 2024-03-23T11:17:44.867 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T11:17:44.926 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2024-03-23T11:17:44.975 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T11:17:44.975 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:17:44.975 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 689 Links: 1 2024-03-23T11:17:44.975 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:17:44.975 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:17:44.975 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-23 11:17:31.243710489 +0000 2024-03-23T11:17:44.975 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-23 11:15:14.748285222 +0000 2024-03-23T11:17:44.975 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-23 11:15:14.748285222 +0000 2024-03-23T11:17:44.975 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-23 11:15:14.748285222 +0000 2024-03-23T11:17:44.975 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T11:17:45.064 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-23T11:17:45.064 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-23T11:17:45.064 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.00034508 s, 1.5 MB/s 2024-03-23T11:17:45.069 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T11:17:45.125 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T11:17:45.125 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:17:45.125 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T11:17:45.167 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T11:17:45.168 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 2024-03-23T11:17:45.248 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T11:17:45.249 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:17:45.249 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 640 Links: 1 2024-03-23T11:17:45.249 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:17:45.249 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:17:45.249 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-23 11:17:33.926377147 +0000 2024-03-23T11:17:45.249 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-23 11:15:10.492448792 +0000 2024-03-23T11:17:45.249 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-23 11:15:10.492448792 +0000 2024-03-23T11:17:45.249 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-23 11:15:10.492448792 +0000 2024-03-23T11:17:45.249 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T11:17:45.356 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-23T11:17:45.356 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-23T11:17:45.356 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000347115 s, 1.5 MB/s 2024-03-23T11:17:45.359 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T11:17:45.399 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 2024-03-23T11:17:45.479 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T11:17:45.479 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:17:45.479 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 656 Links: 1 2024-03-23T11:17:45.479 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:17:45.479 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:17:45.479 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-23 11:17:33.926377147 +0000 2024-03-23T11:17:45.479 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-23 11:15:11.070435466 +0000 2024-03-23T11:17:45.479 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-23 11:15:11.070435466 +0000 2024-03-23T11:17:45.479 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-23 11:15:11.070435466 +0000 2024-03-23T11:17:45.479 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T11:17:45.594 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-23T11:17:45.594 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-23T11:17:45.594 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000296418 s, 1.7 MB/s 2024-03-23T11:17:45.597 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T11:17:45.652 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 2024-03-23T11:17:45.696 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T11:17:45.696 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:17:45.696 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 676 Links: 1 2024-03-23T11:17:45.696 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:17:45.696 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:17:45.696 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-23 11:17:33.927377124 +0000 2024-03-23T11:17:45.697 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-23 11:15:11.632422509 +0000 2024-03-23T11:17:45.697 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-23 11:15:11.632422509 +0000 2024-03-23T11:17:45.697 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-23 11:15:11.632422509 +0000 2024-03-23T11:17:45.697 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T11:17:45.830 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-23T11:17:45.830 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-23T11:17:45.830 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.00031957 s, 1.6 MB/s 2024-03-23T11:17:45.834 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T11:17:45.888 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 2024-03-23T11:17:45.938 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T11:17:45.938 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T11:17:45.938 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 695 Links: 1 2024-03-23T11:17:45.938 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T11:17:45.938 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T11:17:45.938 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-23 11:17:33.928377102 +0000 2024-03-23T11:17:45.939 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-23 11:15:12.186409736 +0000 2024-03-23T11:17:45.939 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-23 11:15:12.186409736 +0000 2024-03-23T11:17:45.939 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-23 11:15:12.186409736 +0000 2024-03-23T11:17:45.939 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T11:17:46.024 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-23T11:17:46.024 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-23T11:17:46.024 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000326324 s, 1.6 MB/s 2024-03-23T11:17:46.028 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T11:17:46.079 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T11:17:46.079 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-23T11:17:46.079 INFO:tasks.ceph:Generating config... 2024-03-23T11:17:46.080 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-23T11:17:46.080 INFO:tasks.ceph:[client] debug client = 20 2024-03-23T11:17:46.080 INFO:tasks.ceph:[client] debug ms = 1 2024-03-23T11:17:46.080 INFO:tasks.ceph:[client] pid file = 2024-03-23T11:17:46.081 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-23T11:17:46.081 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-23T11:17:46.081 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-23T11:17:46.082 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-23T11:17:46.082 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-23T11:17:46.082 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-23T11:17:46.082 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-23T11:17:46.082 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-23T11:17:46.082 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-23T11:17:46.082 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-23T11:17:46.082 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-23T11:17:46.082 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-23T11:17:46.082 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-23T11:17:46.082 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-23T11:17:46.082 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-23T11:17:46.082 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-23T11:17:46.083 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-23T11:17:46.083 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-23T11:17:46.083 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-23T11:17:46.083 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-23T11:17:46.083 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-23T11:17:46.083 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-23T11:17:46.083 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-23T11:17:46.083 INFO:tasks.ceph:Setting up mon.a... 2024-03-23T11:17:46.083 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-23T11:17:46.228 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/ceph.keyring 2024-03-23T11:17:46.237 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-23T11:17:46.377 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T11:17:46.474 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.62'), ('mon.b', '172.21.15.120'), ('mon.c', '[v2:172.21.15.120:3301,v1:172.21.15.120:6790]')] 2024-03-23T11:17:46.475 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.62,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-23T11:17:46.475 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:17:46.475 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T11:17:46.528 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.62 --add b 172.21.15.120 --addv c '[v2:172.21.15.120:3301,v1:172.21.15.120:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:17:46.690 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:17:46.690 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: generated fsid e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:17:46.690 INFO:teuthology.orchestra.run.smithi062.stdout:setting min_mon_release = pacific 2024-03-23T11:17:46.691 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 0 2024-03-23T11:17:46.691 INFO:teuthology.orchestra.run.smithi062.stdout:fsid e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:17:46.691 INFO:teuthology.orchestra.run.smithi062.stdout:last_changed 2024-03-23T11:17:46.691179+0000 2024-03-23T11:17:46.691 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-03-23T11:17:46.691179+0000 2024-03-23T11:17:46.691 INFO:teuthology.orchestra.run.smithi062.stdout:min_mon_release 16 (pacific) 2024-03-23T11:17:46.691 INFO:teuthology.orchestra.run.smithi062.stdout:election_strategy: 1 2024-03-23T11:17:46.691 INFO:teuthology.orchestra.run.smithi062.stdout:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-03-23T11:17:46.691 INFO:teuthology.orchestra.run.smithi062.stdout:1: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-03-23T11:17:46.691 INFO:teuthology.orchestra.run.smithi062.stdout:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-03-23T11:17:46.691 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-23T11:17:46.694 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T11:17:46.746 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e8b2848c-f838-4b99-a310-de7a7069c38e... 2024-03-23T11:17:46.748 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T11:17:46.753 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2024-03-23T11:17:46.885 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.886 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: mon_allow_pool_size_one = true 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2024-03-23T11:17:46.887 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "172.21.15.62,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: osd mclock profile = high_recovery_ops 2024-03-23T11:17:46.888 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2024-03-23T11:17:46.889 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.890 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2024-03-23T11:17:46.890 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.890 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2024-03-23T11:17:46.890 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2024-03-23T11:17:46.890 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.890 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2024-03-23T11:17:46.890 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2024-03-23T11:17:46.890 INFO:teuthology.orchestra.run.smithi120.stdout: bdev async discard = True 2024-03-23T11:17:46.890 INFO:teuthology.orchestra.run.smithi120.stdout: bdev enable discard = True 2024-03-23T11:17:46.890 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore allocator = bitmap 2024-03-23T11:17:46.891 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2024-03-23T11:17:46.891 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2024-03-23T11:17:46.891 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 1/20 2024-03-23T11:17:46.891 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 1/20 2024-03-23T11:17:46.891 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:17:46.891 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2024-03-23T11:17:46.891 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 4/10 2024-03-23T11:17:46.891 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T11:17:46.891 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:17:46.892 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2024-03-23T11:17:46.893 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: # 1m isn't quite enough 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: mon_down_mkfs_grace = 2m 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.894 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_filestore_osds = false 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: mon op complaint time = 120 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: client mount timeout = 600 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: debug client = 20 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = "" 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-23T11:17:46.895 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.d] 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout:[mds] 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds = 20 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds balancer = 20 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug frag = True 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug scatterstat = True 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout: mds op complaint time = 180 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout: mds verify scatter = True 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-23T11:17:46.896 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-23T11:17:46.897 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-23T11:17:46.916 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-03-23T11:17:46.916 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2024-03-23T11:17:46.916 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T11:17:46.916 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2024-03-23T11:17:46.916 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.916 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T11:17:46.917 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T11:17:46.918 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = "172.21.15.62,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.919 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock profile = high_recovery_ops 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.920 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: bdev async discard = True 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: bdev enable discard = True 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore allocator = bitmap 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 1/20 2024-03-23T11:17:46.921 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 1/20 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 4/10 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:17:46.922 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2024-03-23T11:17:46.923 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: mon op complaint time = 120 2024-03-23T11:17:46.924 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout: client mount timeout = 600 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout: debug client = 20 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = "" 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.b] 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.c] 2024-03-23T11:17:46.925 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a] 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.c] 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b] 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.d] 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout:[mds] 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds = 20 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds balancer = 20 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug frag = True 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug scatterstat = True 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout: mds op complaint time = 180 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout: mds verify scatter = True 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-03-23T11:17:46.926 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-03-23T11:17:46.959 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-23T11:17:46.960 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-23T11:17:47.112 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-23T11:17:47.113 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:17:47.113 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-23T11:17:47.165 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:17:47.166 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-23T11:17:47.217 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:17:47.217 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:17:47.217 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-23T11:17:47.217 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T11:17:47.379 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:17:47.379 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:17:47.477 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:17:47.477 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:17:47.477 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-23T11:17:47.477 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T11:17:47.610 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:17:47.610 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:17:47.700 INFO:tasks.ceph:Setting up mon nodes... 2024-03-23T11:17:47.701 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-23T11:17:47.701 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-23T11:17:47.902 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-23T11:17:47.911 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-23T11:17:48.092 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-23T11:17:48.103 INFO:tasks.ceph:Setting up mds nodes... 2024-03-23T11:17:48.103 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-23T11:17:48.290 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-23T11:17:48.299 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-23T11:17:48.393 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-23T11:17:48.584 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-23T11:17:48.594 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-23T11:17:48.691 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-23T11:17:48.871 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-23T11:17:48.881 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-23T11:17:48.972 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-23T11:17:49.145 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-23T11:17:49.155 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-23T11:17:49.243 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-23T11:17:49.243 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-23T11:17:49.382 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-23T11:17:49.450 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-23T11:17:49.581 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-23T11:17:49.641 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-23T11:17:49.769 INFO:teuthology.orchestra.run.smithi120.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-23T11:17:49.824 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-23T11:17:49.946 INFO:teuthology.orchestra.run.smithi120.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-23T11:17:50.000 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-23T11:17:50.000 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-23T11:17:50.000 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-23T11:17:50.105 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T11:17:50.105 INFO:tasks.ceph:role: osd.0 2024-03-23T11:17:50.106 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:17:50.106 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-23T11:17:50.208 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T11:17:50.208 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T11:17:50.208 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T11:17:50.208 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T11:17:50.208 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T11:17:50.208 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-03-23T11:17:50.208 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T11:17:50.208 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T11:17:50.208 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T11:17:50.208 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T11:17:50.723 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-03-23T11:17:50.729 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-03-23T11:17:50.730 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-23T11:17:50.909 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-23T11:17:51.019 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-23T11:17:51.120 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T11:17:51.120 INFO:tasks.ceph:role: osd.1 2024-03-23T11:17:51.120 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:17:51.120 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-23T11:17:51.226 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T11:17:51.226 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T11:17:51.226 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T11:17:51.226 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T11:17:51.226 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T11:17:51.226 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-03-23T11:17:51.227 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T11:17:51.227 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T11:17:51.227 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T11:17:51.227 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T11:17:51.726 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-03-23T11:17:51.733 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-03-23T11:17:51.733 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-23T11:17:51.888 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-23T11:17:51.997 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-23T11:17:52.092 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T11:17:52.092 INFO:tasks.ceph:role: osd.2 2024-03-23T11:17:52.092 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:17:52.092 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-23T11:17:52.199 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T11:17:52.199 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T11:17:52.199 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T11:17:52.199 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T11:17:52.199 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T11:17:52.199 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-03-23T11:17:52.200 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T11:17:52.200 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T11:17:52.200 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T11:17:52.200 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T11:17:52.709 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-03-23T11:17:52.718 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-03-23T11:17:52.718 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-23T11:17:52.903 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-23T11:17:53.012 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-23T11:17:53.108 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T11:17:53.108 INFO:tasks.ceph:role: osd.3 2024-03-23T11:17:53.108 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:17:53.109 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-23T11:17:53.206 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T11:17:53.206 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T11:17:53.206 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T11:17:53.206 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T11:17:53.206 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T11:17:53.206 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-03-23T11:17:53.206 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T11:17:53.206 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T11:17:53.206 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T11:17:53.206 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T11:17:53.681 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-03-23T11:17:53.688 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-03-23T11:17:53.688 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-23T11:17:53.830 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-23T11:17:53.923 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:17:54.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:54.092+0000 7faf2b025500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-23T11:17:54.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:54.095+0000 7faf2b025500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-23T11:17:54.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:54.095+0000 7faf2b025500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T11:17:54.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:54.096+0000 7faf2b025500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T11:17:54.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:54.096+0000 7faf2b025500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T11:17:54.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:54.106+0000 7faf2b025500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-23T11:17:57.297 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-23T11:17:57.402 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:17:57.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:57.548+0000 7ffa43e69500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-23T11:17:57.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:57.549+0000 7ffa43e69500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-23T11:17:57.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:57.549+0000 7ffa43e69500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T11:17:57.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:57.549+0000 7ffa43e69500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T11:17:57.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:57.549+0000 7ffa43e69500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T11:17:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:17:57.551+0000 7ffa43e69500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-23T11:18:00.737 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-23T11:18:00.837 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:18:00.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:00.984+0000 7fea56014500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-23T11:18:00.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:00.984+0000 7fea56014500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-23T11:18:00.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:00.984+0000 7fea56014500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T11:18:00.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:00.984+0000 7fea56014500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T11:18:00.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:00.984+0000 7fea56014500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T11:18:00.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:00.987+0000 7fea56014500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-23T11:18:04.159 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-23T11:18:04.259 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:18:04.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:04.403+0000 7f58bf425500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-23T11:18:04.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:04.404+0000 7f58bf425500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-23T11:18:04.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:04.404+0000 7f58bf425500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T11:18:04.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:04.404+0000 7f58bf425500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T11:18:04.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:04.404+0000 7f58bf425500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T11:18:04.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:04.406+0000 7f58bf425500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-23T11:18:07.587 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-23T11:18:07.691 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-23T11:18:07.793 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T11:18:07.794 INFO:tasks.ceph:role: osd.4 2024-03-23T11:18:07.794 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:18:07.794 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-23T11:18:07.890 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T11:18:07.890 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T11:18:07.890 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T11:18:07.891 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T11:18:07.891 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T11:18:07.891 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-23T11:18:07.891 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T11:18:07.891 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T11:18:07.891 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T11:18:07.891 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T11:18:08.383 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-23T11:18:08.389 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-23T11:18:08.390 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-23T11:18:08.566 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-23T11:18:08.670 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-23T11:18:08.757 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T11:18:08.758 INFO:tasks.ceph:role: osd.5 2024-03-23T11:18:08.758 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:18:08.758 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-23T11:18:08.844 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T11:18:08.844 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T11:18:08.845 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T11:18:08.845 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T11:18:08.845 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T11:18:08.845 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-23T11:18:08.845 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T11:18:08.845 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T11:18:08.845 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T11:18:08.845 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T11:18:09.292 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-23T11:18:09.300 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-23T11:18:09.300 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-23T11:18:09.433 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-23T11:18:09.522 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-23T11:18:09.598 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T11:18:09.598 INFO:tasks.ceph:role: osd.6 2024-03-23T11:18:09.599 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:18:09.599 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-23T11:18:09.674 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T11:18:09.674 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T11:18:09.674 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T11:18:09.674 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T11:18:09.674 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T11:18:09.674 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-23T11:18:09.674 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T11:18:09.674 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T11:18:09.674 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T11:18:09.675 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T11:18:10.191 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-23T11:18:10.197 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-23T11:18:10.197 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-23T11:18:10.352 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-23T11:18:10.449 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-23T11:18:10.539 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T11:18:10.539 INFO:tasks.ceph:role: osd.7 2024-03-23T11:18:10.539 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:18:10.539 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-23T11:18:10.637 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T11:18:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T11:18:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T11:18:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T11:18:10.638 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T11:18:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-23T11:18:10.638 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T11:18:10.638 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T11:18:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T11:18:10.638 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T11:18:11.093 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-23T11:18:11.099 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-23T11:18:11.099 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-23T11:18:11.212 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-23T11:18:11.286 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:18:11.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:11.439+0000 7f5401621500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-23T11:18:11.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:11.440+0000 7f5401621500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-23T11:18:11.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:11.440+0000 7f5401621500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T11:18:11.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:11.440+0000 7f5401621500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T11:18:11.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:11.440+0000 7f5401621500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T11:18:11.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:11.443+0000 7f5401621500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-23T11:18:14.028 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:18:14.035 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:18:14.357 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-23T11:18:14.450 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:18:14.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:14.583+0000 7f0b8439b500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-23T11:18:14.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:14.583+0000 7f0b8439b500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-23T11:18:14.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:14.583+0000 7f0b8439b500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T11:18:14.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:14.584+0000 7f0b8439b500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T11:18:14.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:14.584+0000 7f0b8439b500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T11:18:14.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:14.586+0000 7f0b8439b500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-23T11:18:17.517 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-23T11:18:17.608 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:18:17.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:17.743+0000 7f6367307500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-23T11:18:17.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:17.744+0000 7f6367307500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-23T11:18:17.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:17.744+0000 7f6367307500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T11:18:17.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:17.744+0000 7f6367307500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T11:18:17.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:17.744+0000 7f6367307500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T11:18:17.747 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:17.746+0000 7f6367307500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-23T11:18:20.916 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-23T11:18:21.014 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:18:21.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:21.149+0000 7f014499f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-23T11:18:21.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:21.149+0000 7f014499f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-23T11:18:21.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:21.149+0000 7f014499f500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T11:18:21.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:21.149+0000 7f014499f500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T11:18:21.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:21.149+0000 7f014499f500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T11:18:21.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:21.151+0000 7f014499f500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-23T11:18:24.318 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-23T11:18:24.407 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-23T11:18:24.407 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:24.407 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-23T11:18:24.516 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:24.517 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-23T11:18:24.615 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:24.615 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-23T11:18:24.712 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:24.713 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-23T11:18:24.812 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:24.812 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-23T11:18:24.913 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:24.913 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-23T11:18:25.009 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:25.009 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-23T11:18:25.107 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:25.107 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-23T11:18:25.207 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:25.207 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-23T11:18:25.295 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:25.295 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-23T11:18:25.386 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:25.386 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-23T11:18:25.522 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:25.522 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-23T11:18:25.613 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:25.613 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-23T11:18:25.707 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:25.707 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-23T11:18:25.795 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:25.795 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-23T11:18:25.855 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:25.855 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-23T11:18:25.908 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:25.908 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-23T11:18:25.962 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:25.962 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-23T11:18:26.013 INFO:tasks.ceph:Adding keys to all mons... 2024-03-23T11:18:26.013 DEBUG:teuthology.orchestra.run.smithi062:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-23T11:18:26.020 DEBUG:teuthology.orchestra.run.smithi120:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-23T11:18:26.112 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.y] 2024-03-23T11:18:26.112 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBbuv5lqnG8NRAAvyswMz5kpW0rGjzWYRTAAg== 2024-03-23T11:18:26.112 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a] 2024-03-23T11:18:26.112 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBcuv5lYNxYERAAPVHWwRHM5V7OejL294b2zg== 2024-03-23T11:18:26.112 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.c] 2024-03-23T11:18:26.112 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBcuv5ltqrjIhAA9+f82fvlrFlIEXqswPE0ig== 2024-03-23T11:18:26.112 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.0] 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBiuv5lRfaQBRAAvrtCZXwt00PUQfCqqKBzMQ== 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.1] 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBluv5lzwS/IBAAmYU8yMLtCUJNr/AyE0MJDA== 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.2] 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBouv5lyAy1OhAAJoUsg6DFpEPxAcH5oI2exQ== 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.3] 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBsuv5lb0YcGBAA220D0uRhAZpG4w97wqsB5A== 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.x] 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBcuv5lT6+KBRAAl5TZsJGNr4gAB2cEDZo/hQ== 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b] 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBcuv5lmOUANBAAifuZ0DUPD9az/mturTndSA== 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.d] 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBduv5l7ly4CBAA8+oim0z+bbljfP0VH7XSaw== 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.4] 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBzuv5lC8RIGhAAH+5HaeCgQhrQMrYm9iWaYA== 2024-03-23T11:18:26.113 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.5] 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQB2uv5lYi7ZIhAAMNIm/jPckCAbX9oEx5Li/Q== 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.6] 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQB5uv5lgoRtLBAAFT7uJtyqHtHAegOYyDH/WQ== 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.7] 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQB9uv5l/dzrCBAA3jsua0LvBPI7I551/rRHrg== 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBduv5ltavQFhAAvV2RWuHOPK2T9Q7gWkYA4w== 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout:[client.2] 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBduv5lDRG1IhAAjVaXRR30G1985oMou7Ambg== 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout:[client.1] 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBduv5luQ/zLRAAwjDtV64TCKnQliPq4iUAYg== 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout:[client.3] 2024-03-23T11:18:26.114 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBduv5lomZyOBAA8BMIBD+Zhtp6Fa2RcEXCJg== 2024-03-23T11:18:26.144 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.y] 2024-03-23T11:18:26.144 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBbuv5lqnG8NRAAvyswMz5kpW0rGjzWYRTAAg== 2024-03-23T11:18:26.144 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2024-03-23T11:18:26.144 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBcuv5lYNxYERAAPVHWwRHM5V7OejL294b2zg== 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBcuv5ltqrjIhAA9+f82fvlrFlIEXqswPE0ig== 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.0] 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBiuv5lRfaQBRAAvrtCZXwt00PUQfCqqKBzMQ== 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.1] 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBluv5lzwS/IBAAmYU8yMLtCUJNr/AyE0MJDA== 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.2] 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBouv5lyAy1OhAAJoUsg6DFpEPxAcH5oI2exQ== 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.3] 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBsuv5lb0YcGBAA220D0uRhAZpG4w97wqsB5A== 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.x] 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBcuv5lT6+KBRAAl5TZsJGNr4gAB2cEDZo/hQ== 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBcuv5lmOUANBAAifuZ0DUPD9az/mturTndSA== 2024-03-23T11:18:26.145 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.d] 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBduv5l7ly4CBAA8+oim0z+bbljfP0VH7XSaw== 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.4] 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBzuv5lC8RIGhAAH+5HaeCgQhrQMrYm9iWaYA== 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.5] 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB2uv5lYi7ZIhAAMNIm/jPckCAbX9oEx5Li/Q== 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.6] 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB5uv5lgoRtLBAAFT7uJtyqHtHAegOYyDH/WQ== 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.7] 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB9uv5l/dzrCBAA3jsua0LvBPI7I551/rRHrg== 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout:[client.0] 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBduv5ltavQFhAAvV2RWuHOPK2T9Q7gWkYA4w== 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout:[client.2] 2024-03-23T11:18:26.146 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBduv5lDRG1IhAAjVaXRR30G1985oMou7Ambg== 2024-03-23T11:18:26.147 INFO:teuthology.orchestra.run.smithi120.stdout:[client.1] 2024-03-23T11:18:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBduv5luQ/zLRAAwjDtV64TCKnQliPq4iUAYg== 2024-03-23T11:18:26.147 INFO:teuthology.orchestra.run.smithi120.stdout:[client.3] 2024-03-23T11:18:26.147 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBduv5lomZyOBAA8BMIBD+Zhtp6Fa2RcEXCJg== 2024-03-23T11:18:26.148 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T11:18:26.164 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T11:18:26.311 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T11:18:26.315 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T11:18:26.458 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T11:18:26.462 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T11:18:26.611 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:26.615 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:26.755 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:26.759 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:26.899 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:26.906 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:27.055 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:27.059 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:27.202 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T11:18:27.209 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T11:18:27.350 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T11:18:27.356 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T11:18:27.504 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T11:18:27.509 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T11:18:27.654 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:27.659 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:27.802 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:27.808 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:27.943 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:27.949 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:28.089 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:28.097 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T11:18:28.244 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T11:18:28.251 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T11:18:28.400 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T11:18:28.404 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T11:18:28.548 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T11:18:28.555 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T11:18:28.698 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T11:18:28.705 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T11:18:28.852 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-23T11:18:28.852 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-23T11:18:28.949 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T11:18:29.113 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-23T11:18:29.188 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-23T11:18:29.284 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T11:18:29.437 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-23T11:18:29.521 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-23T11:18:29.614 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T11:18:29.769 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-23T11:18:29.855 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:18:29.863 DEBUG:teuthology.orchestra.run.smithi120:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-23T11:18:29.915 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-23T11:18:29.916 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-23T11:18:29.916 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-23T11:18:29.922 INFO:tasks.ceph.mon.a:Started 2024-03-23T11:18:29.922 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-23T11:18:29.922 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-23T11:18:29.927 INFO:tasks.ceph.mon.b:Started 2024-03-23T11:18:29.927 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-23T11:18:29.927 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-23T11:18:29.931 INFO:tasks.ceph.mon.c:Started 2024-03-23T11:18:29.931 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-23T11:18:29.931 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-23T11:18:29.931 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-23T11:18:29.937 INFO:tasks.ceph.mgr.y:Started 2024-03-23T11:18:29.937 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-23T11:18:29.937 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-23T11:18:29.941 INFO:tasks.ceph.mgr.x:Started 2024-03-23T11:18:29.941 INFO:tasks.ceph:Setting crush tunables to default 2024-03-23T11:18:29.942 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd crush tunables default 2024-03-23T11:18:30.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.126+0000 7f22ee400640 1 Processor -- start 2024-03-23T11:18:30.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.126+0000 7f22ee400640 1 -- start start 2024-03-23T11:18:30.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.127+0000 7f22ee400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:30.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.127+0000 7f22ee400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e815cd70 con 0x7f22e806b3f0 2024-03-23T11:18:30.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.127+0000 7f22ee400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e80ce2f0 con 0x7f22e80cefa0 2024-03-23T11:18:30.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.127+0000 7f22ee400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e80ce5d0 con 0x7f22e815c990 2024-03-23T11:18:30.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.128+0000 7f22e7e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f22e806b3f0 0x7f22e80d0e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:34472/0 (socket says 172.21.15.62:34472) 2024-03-23T11:18:30.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.128+0000 7f22e7e00640 1 -- 172.21.15.62:0/2579666411 learned_addr learned my addr 172.21.15.62:0/2579666411 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:18:30.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.128+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:30.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.128+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 msgr2=0x7f22e806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T11:18:30.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.128+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 msgr2=0x7f22e806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:30.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.128+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:30.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.128+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:18:30.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.329+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:30.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.330+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 msgr2=0x7f22e806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T11:18:30.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.330+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 msgr2=0x7f22e806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:30.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.330+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:30.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.330+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T11:18:30.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.731+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:30.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.732+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 msgr2=0x7f22e806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T11:18:30.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.732+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 msgr2=0x7f22e806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:30.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.732+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:30.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:30.732+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T11:18:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:31.534+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:31.534+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 msgr2=0x7f22e806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T11:18:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:31.534+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 msgr2=0x7f22e806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:31.534+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:31.534+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T11:18:33.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.127+0000 7f22e6000640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 msgr2=0x7f22e806aeb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T11:18:33.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.127+0000 7f22e6000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22e806aeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:33.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.127+0000 7f22e6000640 1 -- 172.21.15.62:0/2579666411 >> v1:172.21.15.120:6789/0 conn(0x7f22e80cefa0 legacy=0x7f22e80cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:33.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.127+0000 7f22e6000640 1 -- 172.21.15.62:0/2579666411 >> v1:172.21.15.62:6789/0 conn(0x7f22e806b3f0 legacy=0x7f22e80d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:33.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.127+0000 7f22e6000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:33.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.127+0000 7f22e6000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:33.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.128+0000 7f22e6000640 1 -- 172.21.15.62:0/2579666411 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e80ce5d0 con 0x7f22c80061f0 2024-03-23T11:18:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.128+0000 7f22e6000640 1 -- 172.21.15.62:0/2579666411 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c8008d60 con 0x7f22c8003730 2024-03-23T11:18:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.128+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.128+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.129+0000 7f22e7e00640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 msgr2=0x7f22c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T11:18:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.129+0000 7f22e7e00640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 msgr2=0x7f22c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.129+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.129+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 msgr2=0x7f22c8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T11:18:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.129+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 msgr2=0x7f22c8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.129+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.129+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:18:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.129+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T11:18:33.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.330+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:33.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.330+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:33.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.331+0000 7f22e7e00640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 msgr2=0x7f22c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T11:18:33.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.331+0000 7f22e7e00640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 msgr2=0x7f22c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:33.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.331+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:33.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.331+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T11:18:33.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.331+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 msgr2=0x7f22c8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T11:18:33.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.331+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 msgr2=0x7f22c8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:33.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.331+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:33.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.331+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T11:18:33.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.732+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:33.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.732+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:33.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.733+0000 7f22e7e00640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 msgr2=0x7f22c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T11:18:33.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.733+0000 7f22e7e00640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 msgr2=0x7f22c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:33.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.733+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:33.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.733+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T11:18:33.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.733+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 msgr2=0x7f22c8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T11:18:33.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.733+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 msgr2=0x7f22c8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:33.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.733+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:33.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:33.733+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T11:18:34.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:34.534+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:34.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:34.535+0000 7f22e7e00640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 msgr2=0x7f22c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T11:18:34.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:34.535+0000 7f22e7e00640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 msgr2=0x7f22c80086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:34.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:34.535+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:34.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:34.535+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:34.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:34.535+0000 7f22e7e00640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T11:18:34.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:34.535+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 msgr2=0x7f22c8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T11:18:34.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:34.535+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 msgr2=0x7f22c8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T11:18:34.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:34.535+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T11:18:34.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:34.535+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T11:18:36.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.128+0000 7f22e6000640 1 -- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 msgr2=0x7f22c8005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T11:18:36.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.128+0000 7f22e6000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22c8003730 0x7f22c8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:36.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.128+0000 7f22e6000640 1 -- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 msgr2=0x7f22c80086b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T11:18:36.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.128+0000 7f22e6000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.62:3300/0 conn(0x7f22c80061f0 0x7f22c80086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:36.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.128+0000 7f22e6000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22c800b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:36.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.128+0000 7f22eda00640 1 -- 172.21.15.62:0/2579666411 reap_dead start 2024-03-23T11:18:36.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.128+0000 7f22e6000640 1 -- 172.21.15.62:0/2579666411 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e80d0360 con 0x7f22e806b3f0 2024-03-23T11:18:36.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.128+0000 7f22e6000640 1 -- 172.21.15.62:0/2579666411 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c8013610 con 0x7f22e80cefa0 2024-03-23T11:18:36.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.128+0000 7f22e6000640 1 -- 172.21.15.62:0/2579666411 --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e80ce5d0 con 0x7f22e815c990 2024-03-23T11:18:36.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.128+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> v2:172.21.15.120:3300/0 conn(0x7f22e815c990 0x7f22c800b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:36.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.129+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> v1:172.21.15.120:6789/0 conn(0x7f22e80cefa0 legacy=0x7f22c8010e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:36.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.129+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 >> v1:172.21.15.62:6789/0 conn(0x7f22e806b3f0 legacy=0x7f22c800d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:36.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.129+0000 7f22ed000640 1 -- 172.21.15.62:0/2579666411 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c80066a0 con 0x7f22e815c990 2024-03-23T11:18:36.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.129+0000 7f22e7400640 1 -- 172.21.15.62:0/2579666411 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941279037 0 0) 0x7f22e80d0360 con 0x7f22e806b3f0 2024-03-23T11:18:36.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.131+0000 7f22ed000640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e815c990 0x7f22c800b170 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f22dc006fd0 tx=0x7f22dc02dbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3cdded1b28b9a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:36.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.131+0000 7f22e7400640 1 -- 172.21.15.62:0/2579666411 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc02e560 con 0x7f22e815c990 2024-03-23T11:18:36.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.131+0000 7f22e7400640 1 -- 172.21.15.62:0/2579666411 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22dc032070 con 0x7f22e815c990 2024-03-23T11:18:36.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.131+0000 7f22e7400640 1 -- 172.21.15.62:0/2579666411 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22dc02e8c0 con 0x7f22e815c990 2024-03-23T11:18:36.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.132+0000 7f22ee400640 1 -- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e815c990 msgr2=0x7f22c800b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:36.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.132+0000 7f22ee400640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e815c990 0x7f22c800b170 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f22dc006fd0 tx=0x7f22dc02dbc0 comp rx=0 tx=0).stop 2024-03-23T11:18:36.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.133+0000 7f22ee400640 1 -- 172.21.15.62:0/2579666411 shutdown_connections 2024-03-23T11:18:36.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.133+0000 7f22ee400640 1 --2- 172.21.15.62:0/2579666411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e815c990 0x7f22c800b170 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:36.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.133+0000 7f22ee400640 1 -- 172.21.15.62:0/2579666411 >> 172.21.15.62:0/2579666411 conn(0x7f22e8089f60 msgr2=0x7f22e80d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:36.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.133+0000 7f22ee400640 1 -- 172.21.15.62:0/2579666411 shutdown_connections 2024-03-23T11:18:36.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.133+0000 7f22ee400640 1 -- 172.21.15.62:0/2579666411 wait complete. 2024-03-23T11:18:36.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.136+0000 7f22ee400640 1 Processor -- start 2024-03-23T11:18:36.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.136+0000 7f22ee400640 1 -- start start 2024-03-23T11:18:36.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.137+0000 7f22ee400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22e806b3f0 0x7f22e814f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.137+0000 7f22ee400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e80cefa0 0x7f22e81524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.137+0000 7f22ee400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e815c990 0x7f22e80cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.137+0000 7f22ee400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e80cc160 con 0x7f22e815c990 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.137+0000 7f22ee400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e814f890 con 0x7f22e80cefa0 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.137+0000 7f22ee400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e8152a30 con 0x7f22e806b3f0 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.138+0000 7f22eda00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e815c990 0x7f22e80cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.138+0000 7f22ed000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22e806b3f0 0x7f22e814f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.138+0000 7f22eda00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e815c990 0x7f22e80cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33652/0 (socket says 172.21.15.62:33652) 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.138+0000 7f22eda00640 1 -- 172.21.15.62:0/275198666 learned_addr learned my addr 172.21.15.62:0/275198666 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.138+0000 7f22ed000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22e806b3f0 0x7f22e814f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:50400/0 (socket says 172.21.15.62:50400) 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.138+0000 7f22e7e00640 1 --2- 172.21.15.62:0/275198666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e80cefa0 0x7f22e81524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:36.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.138+0000 7f22eda00640 1 -- 172.21.15.62:0/275198666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22e806b3f0 msgr2=0x7f22e814f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:36.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.138+0000 7f22eda00640 1 --2- 172.21.15.62:0/275198666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22e806b3f0 0x7f22e814f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:36.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.138+0000 7f22eda00640 1 -- 172.21.15.62:0/275198666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e80cefa0 msgr2=0x7f22e81524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:36.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.138+0000 7f22eda00640 1 --2- 172.21.15.62:0/275198666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e80cefa0 0x7f22e81524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:36.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.138+0000 7f22eda00640 1 -- 172.21.15.62:0/275198666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22dc00e040 con 0x7f22e815c990 2024-03-23T11:18:36.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.139+0000 7f22eda00640 1 --2- 172.21.15.62:0/275198666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e815c990 0x7f22e80cc580 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f22c8008c40 tx=0x7f22e000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:36.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.139+0000 7f22e4c00640 1 -- 172.21.15.62:0/275198666 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e0007070 con 0x7f22e815c990 2024-03-23T11:18:36.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.139+0000 7f22e4c00640 1 -- 172.21.15.62:0/275198666 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e00071d0 con 0x7f22e815c990 2024-03-23T11:18:36.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.139+0000 7f22e4c00640 1 -- 172.21.15.62:0/275198666 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e00075c0 con 0x7f22e815c990 2024-03-23T11:18:36.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.139+0000 7f22ee400640 1 -- 172.21.15.62:0/275198666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e80cb4e0 con 0x7f22e815c990 2024-03-23T11:18:36.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.139+0000 7f22ee400640 1 -- 172.21.15.62:0/275198666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e80cbaa0 con 0x7f22e815c990 2024-03-23T11:18:36.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.140+0000 7f22e4c00640 1 -- 172.21.15.62:0/275198666 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f22e0003970 con 0x7f22e815c990 2024-03-23T11:18:36.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.140+0000 7f22e4c00640 1 -- 172.21.15.62:0/275198666 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f22e00045a0 con 0x7f22e815c990 2024-03-23T11:18:36.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.141+0000 7f22ee400640 1 -- 172.21.15.62:0/275198666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e80cc2a0 con 0x7f22e815c990 2024-03-23T11:18:36.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.146+0000 7f22e4c00640 1 -- 172.21.15.62:0/275198666 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f22e80cc2a0 con 0x7f22e815c990 2024-03-23T11:18:36.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.221+0000 7f22ee400640 1 -- 172.21.15.62:0/275198666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f22e80672c0 con 0x7f22e815c990 2024-03-23T11:18:36.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.231+0000 7f22e4c00640 1 -- 172.21.15.62:0/275198666 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f22e0017070 con 0x7f22e815c990 2024-03-23T11:18:36.231 INFO:teuthology.orchestra.run.smithi062.stderr:adjusted tunables profile to default 2024-03-23T11:18:36.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.232+0000 7f22ee400640 1 -- 172.21.15.62:0/275198666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e815c990 msgr2=0x7f22e80cc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:36.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.232+0000 7f22ee400640 1 --2- 172.21.15.62:0/275198666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e815c990 0x7f22e80cc580 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f22c8008c40 tx=0x7f22e000c960 comp rx=0 tx=0).stop 2024-03-23T11:18:36.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.232+0000 7f22ee400640 1 -- 172.21.15.62:0/275198666 shutdown_connections 2024-03-23T11:18:36.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.232+0000 7f22ee400640 1 --2- 172.21.15.62:0/275198666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e815c990 0x7f22e80cc580 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:36.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.232+0000 7f22ee400640 1 --2- 172.21.15.62:0/275198666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e80cefa0 0x7f22e81524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.232+0000 7f22ee400640 1 --2- 172.21.15.62:0/275198666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22e806b3f0 0x7f22e814f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.232+0000 7f22ee400640 1 -- 172.21.15.62:0/275198666 >> 172.21.15.62:0/275198666 conn(0x7f22e8089f60 msgr2=0x7f22e806b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.233+0000 7f22ee400640 1 -- 172.21.15.62:0/275198666 shutdown_connections 2024-03-23T11:18:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:36.233+0000 7f22ee400640 1 -- 172.21.15.62:0/275198666 wait complete. 2024-03-23T11:18:36.250 INFO:tasks.ceph:check_enable_crimson: False 2024-03-23T11:18:36.250 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-23T11:18:36.251 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:36.251 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-23T11:18:36.325 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:36.325 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-23T11:18:36.461 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:36.461 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-23T11:18:36.547 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:18:36.547 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-23T11:18:36.684 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:36.685 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-23T11:18:36.704 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:36.704+0000 7f50f2e14040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T11:18:36.738 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:36.737+0000 7fef97c42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T11:18:36.773 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:36.773+0000 7f50f2e14040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T11:18:36.774 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:36.774 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-23T11:18:36.802 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:36.802+0000 7fef97c42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T11:18:36.907 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:36.907 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-23T11:18:37.031 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:18:37.031 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-23T11:18:37.160 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new c85aebac-7bbe-48fe-8ff2-bab379b1927f 0 2024-03-23T11:18:37.346 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.345+0000 7fb2c1e00640 1 Processor -- start 2024-03-23T11:18:37.346 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.345+0000 7fb2c1e00640 1 -- start start 2024-03-23T11:18:37.348 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.347+0000 7fb2c1e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb2bc15cce0 0x7fb2bc0b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.348 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.347+0000 7fb2c1e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc0cd690 0x7fb2bc0cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.348 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.347+0000 7fb2c1e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc0d1100 con 0x7fb2bc15c900 2024-03-23T11:18:37.348 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.347+0000 7fb2c1e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2bc0ce060 con 0x7fb2bc15cce0 2024-03-23T11:18:37.348 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.347+0000 7fb2c1e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc0d13e0 con 0x7fb2bc0cd690 2024-03-23T11:18:37.348 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.347+0000 7fb2baa00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fb2bc15c900 0x7fb2bc0d21a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:38750/0 (socket says 172.21.15.120:38750) 2024-03-23T11:18:37.349 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.347+0000 7fb2baa00640 1 -- 172.21.15.120:0/1085983620 learned_addr learned my addr 172.21.15.120:0/1085983620 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-23T11:18:37.349 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.347+0000 7fb2bb400640 1 --2- 172.21.15.120:0/1085983620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc0cd690 0x7fb2bc0cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.351 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.348+0000 7fb2b3e00640 1 --2- 172.21.15.120:0/1085983620 >> v2:172.21.15.62:3300/0 conn(0x7fb2bc15cce0 0x7fb2bc0b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.348+0000 7fb2bb400640 1 -- 172.21.15.120:0/1085983620 >> v2:172.21.15.62:3300/0 conn(0x7fb2bc15cce0 msgr2=0x7fb2bc0b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:37.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.348+0000 7fb2bb400640 1 --2- 172.21.15.120:0/1085983620 >> v2:172.21.15.62:3300/0 conn(0x7fb2bc15cce0 0x7fb2bc0b82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.348+0000 7fb2bb400640 1 -- 172.21.15.120:0/1085983620 >> v1:172.21.15.120:6789/0 conn(0x7fb2bc15c900 legacy=0x7fb2bc0d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.348+0000 7fb2bb400640 1 -- 172.21.15.120:0/1085983620 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc0d16c0 con 0x7fb2bc0cd690 2024-03-23T11:18:37.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.348+0000 7fb2ba000640 1 -- 172.21.15.120:0/1085983620 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585554373 0 0) 0x7fb2bc0d1100 con 0x7fb2bc15c900 2024-03-23T11:18:37.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.349+0000 7fb2bb400640 1 --2- 172.21.15.120:0/1085983620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc0cd690 0x7fb2bc0cdb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009d90 tx=0x7fb2b402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98bb60340866d33d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:37.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.349+0000 7fb2ba000640 1 -- 172.21.15.120:0/1085983620 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b402cc50 con 0x7fb2bc0cd690 2024-03-23T11:18:37.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.349+0000 7fb2ba000640 1 -- 172.21.15.120:0/1085983620 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b402cdb0 con 0x7fb2bc0cd690 2024-03-23T11:18:37.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.349+0000 7fb2ba000640 1 -- 172.21.15.120:0/1085983620 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4034660 con 0x7fb2bc0cd690 2024-03-23T11:18:37.352 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.350+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1085983620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc0cd690 msgr2=0x7fb2bc0cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.350+0000 7fb2c1e00640 1 --2- 172.21.15.120:0/1085983620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc0cd690 0x7fb2bc0cdb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009d90 tx=0x7fb2b402bea0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.350+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1085983620 shutdown_connections 2024-03-23T11:18:37.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.350+0000 7fb2c1e00640 1 --2- 172.21.15.120:0/1085983620 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc0cd690 0x7fb2bc0cdb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.350+0000 7fb2c1e00640 1 --2- 172.21.15.120:0/1085983620 >> v2:172.21.15.62:3300/0 conn(0x7fb2bc15cce0 0x7fb2bc0b82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.350+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1085983620 >> 172.21.15.120:0/1085983620 conn(0x7fb2bc071390 msgr2=0x7fb2bc071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:37.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.350+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1085983620 shutdown_connections 2024-03-23T11:18:37.353 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.350+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1085983620 wait complete. 2024-03-23T11:18:37.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.352+0000 7fb2c1e00640 1 Processor -- start 2024-03-23T11:18:37.358 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.352+0000 7fb2c1e00640 1 -- start start 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2c1e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2bc0cd690 0x7fb2bc1533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2c1e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc15c900 0x7fb2bc0cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2c1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2bc15cce0 0x7fb2bc152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2c1e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc1521e0 con 0x7fb2bc15cce0 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2c1e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc151560 con 0x7fb2bc0cd690 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2c1e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc151840 con 0x7fb2bc15c900 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2baa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2bc0cd690 0x7fb2bc1533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2b3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc15c900 0x7fb2bc0cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2b3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc15c900 0x7fb2bc0cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:45558/0 (socket says 172.21.15.120:45558) 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2baa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2bc0cd690 0x7fb2bc1533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:49980/0 (socket says 172.21.15.120:49980) 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2b3e00640 1 -- 172.21.15.120:0/1668530340 learned_addr learned my addr 172.21.15.120:0/1668530340 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2bb400640 1 --2- 172.21.15.120:0/1668530340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2bc15cce0 0x7fb2bc152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.359 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2baa00640 1 -- 172.21.15.120:0/1668530340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc15c900 msgr2=0x7fb2bc0cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2baa00640 1 --2- 172.21.15.120:0/1668530340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc15c900 0x7fb2bc0cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2baa00640 1 -- 172.21.15.120:0/1668530340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2bc15cce0 msgr2=0x7fb2bc152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2baa00640 1 --2- 172.21.15.120:0/1668530340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2bc15cce0 0x7fb2bc152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2baa00640 1 -- 172.21.15.120:0/1668530340 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b4009a70 con 0x7fb2bc0cd690 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.353+0000 7fb2b3e00640 1 --2- 172.21.15.120:0/1668530340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc15c900 0x7fb2bc0cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.354+0000 7fb2baa00640 1 --2- 172.21.15.120:0/1668530340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2bc0cd690 0x7fb2bc1533d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00c450 tx=0x7fb2ac00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.354+0000 7fb2b3400640 1 -- 172.21.15.120:0/1668530340 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac0074e0 con 0x7fb2bc0cd690 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.354+0000 7fb2b3400640 1 -- 172.21.15.120:0/1668530340 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac007640 con 0x7fb2bc0cd690 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.354+0000 7fb2b3400640 1 -- 172.21.15.120:0/1668530340 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac004a20 con 0x7fb2bc0cd690 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.354+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1668530340 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2bc151b20 con 0x7fb2bc0cd690 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.354+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1668530340 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2bc151180 con 0x7fb2bc0cd690 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.355+0000 7fb2b3400640 1 -- 172.21.15.120:0/1668530340 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb2ac012400 con 0x7fb2bc0cd690 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.355+0000 7fb2b3400640 1 -- 172.21.15.120:0/1668530340 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb2ac01c3f0 con 0x7fb2bc0cd690 2024-03-23T11:18:37.360 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.355+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1668530340 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2bc0672c0 con 0x7fb2bc0cd690 2024-03-23T11:18:37.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.358+0000 7fb2b3400640 1 -- 172.21.15.120:0/1668530340 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fb2ac018070 con 0x7fb2bc0cd690 2024-03-23T11:18:37.408 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:37.408+0000 7fef97c42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T11:18:37.411 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:37.410+0000 7f50f2e14040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T11:18:37.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.436+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1668530340 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c85aebac-7bbe-48fe-8ff2-bab379b1927f", "id": 0} v 0) v1 -- 0x7fb2bc15aa10 con 0x7fb2bc0cd690 2024-03-23T11:18:37.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.445+0000 7fb2b3400640 1 -- 172.21.15.120:0/1668530340 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c85aebac-7bbe-48fe-8ff2-bab379b1927f", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb2ac012840 con 0x7fb2bc0cd690 2024-03-23T11:18:37.446 INFO:teuthology.orchestra.run.smithi120.stdout:0 2024-03-23T11:18:37.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.447+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1668530340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2bc0cd690 msgr2=0x7fb2bc1533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.447+0000 7fb2c1e00640 1 --2- 172.21.15.120:0/1668530340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2bc0cd690 0x7fb2bc1533d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00c450 tx=0x7fb2ac00c910 comp rx=0 tx=0).stop 2024-03-23T11:18:37.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.447+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1668530340 shutdown_connections 2024-03-23T11:18:37.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.447+0000 7fb2c1e00640 1 --2- 172.21.15.120:0/1668530340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2bc15cce0 0x7fb2bc152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.447+0000 7fb2c1e00640 1 --2- 172.21.15.120:0/1668530340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb2bc15c900 0x7fb2bc0cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.447+0000 7fb2c1e00640 1 --2- 172.21.15.120:0/1668530340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2bc0cd690 0x7fb2bc1533d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.447+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1668530340 >> 172.21.15.120:0/1668530340 conn(0x7fb2bc071390 msgr2=0x7fb2bc0c8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:37.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.447+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1668530340 shutdown_connections 2024-03-23T11:18:37.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.447+0000 7fb2c1e00640 1 -- 172.21.15.120:0/1668530340 wait complete. 2024-03-23T11:18:37.464 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 64153585-2982-4f26-98c5-b4cc9036dab9 1 2024-03-23T11:18:37.472 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:37.472+0000 7fef97c42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T11:18:37.480 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:37.480+0000 7f50f2e14040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T11:18:37.526 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:37.526+0000 7fef97c42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T11:18:37.535 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:37.535+0000 7f50f2e14040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T11:18:37.591 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:37.590+0000 7fef97c42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T11:18:37.612 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:37.611+0000 7f50f2e14040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T11:18:37.642 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:37.642+0000 7fef97c42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T11:18:37.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.644+0000 7fb480800640 1 Processor -- start 2024-03-23T11:18:37.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.644+0000 7fb480800640 1 -- start start 2024-03-23T11:18:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.645+0000 7fb480800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815c9e0 0x7fb4781607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.645+0000 7fb480800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb4780cf650 0x7fb4780cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.645+0000 7fb480800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47806bfc0 con 0x7fb47815cdc0 2024-03-23T11:18:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.645+0000 7fb480800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4780d0020 con 0x7fb4780cf650 2024-03-23T11:18:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.645+0000 7fb480800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4780d2630 con 0x7fb47815c9e0 2024-03-23T11:18:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.645+0000 7fb47de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815c9e0 0x7fb4781607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.645+0000 7fb47de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815c9e0 0x7fb4781607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:45570/0 (socket says 172.21.15.120:45570) 2024-03-23T11:18:37.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.645+0000 7fb47de00640 1 -- 172.21.15.120:0/3059669849 learned_addr learned my addr 172.21.15.120:0/3059669849 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:37.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.646+0000 7fb47e800640 1 --2- 172.21.15.120:0/3059669849 >> v2:172.21.15.62:3300/0 conn(0x7fb4780cf650 0x7fb4780cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.646+0000 7fb47de00640 1 -- 172.21.15.120:0/3059669849 >> v2:172.21.15.62:3300/0 conn(0x7fb4780cf650 msgr2=0x7fb4780cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:37.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.646+0000 7fb47de00640 1 --2- 172.21.15.120:0/3059669849 >> v2:172.21.15.62:3300/0 conn(0x7fb4780cf650 0x7fb4780cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.646+0000 7fb47de00640 1 -- 172.21.15.120:0/3059669849 >> v1:172.21.15.120:6789/0 conn(0x7fb47815cdc0 legacy=0x7fb4780d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.646+0000 7fb47de00640 1 -- 172.21.15.120:0/3059669849 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4780b5260 con 0x7fb47815c9e0 2024-03-23T11:18:37.647 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.647+0000 7fb47de00640 1 --2- 172.21.15.120:0/3059669849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815c9e0 0x7fb4781607c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009870 tx=0x7fb46c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1e7c282658972d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.647+0000 7fb47ca00640 1 -- 172.21.15.120:0/3059669849 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c030840 con 0x7fb47815c9e0 2024-03-23T11:18:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.647+0000 7fb47ca00640 1 -- 172.21.15.120:0/3059669849 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c0309a0 con 0x7fb47815c9e0 2024-03-23T11:18:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.647+0000 7fb47ca00640 1 -- 172.21.15.120:0/3059669849 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c030d30 con 0x7fb47815c9e0 2024-03-23T11:18:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.648+0000 7fb480800640 1 -- 172.21.15.120:0/3059669849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815c9e0 msgr2=0x7fb4781607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.648+0000 7fb480800640 1 --2- 172.21.15.120:0/3059669849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815c9e0 0x7fb4781607c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009870 tx=0x7fb46c02b980 comp rx=0 tx=0).stop 2024-03-23T11:18:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.648+0000 7fb480800640 1 -- 172.21.15.120:0/3059669849 shutdown_connections 2024-03-23T11:18:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.648+0000 7fb480800640 1 --2- 172.21.15.120:0/3059669849 >> v2:172.21.15.62:3300/0 conn(0x7fb4780cf650 0x7fb4780cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.648+0000 7fb480800640 1 --2- 172.21.15.120:0/3059669849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815c9e0 0x7fb4781607c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.648+0000 7fb480800640 1 -- 172.21.15.120:0/3059669849 >> 172.21.15.120:0/3059669849 conn(0x7fb478071570 msgr2=0x7fb478071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.648+0000 7fb480800640 1 -- 172.21.15.120:0/3059669849 shutdown_connections 2024-03-23T11:18:37.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.648+0000 7fb480800640 1 -- 172.21.15.120:0/3059669849 wait complete. 2024-03-23T11:18:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.649+0000 7fb480800640 1 Processor -- start 2024-03-23T11:18:37.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.649+0000 7fb480800640 1 -- start start 2024-03-23T11:18:37.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.650+0000 7fb480800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4780cf650 0x7fb478159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.650+0000 7fb480800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb47815c9e0 0x7fb4780abaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.650+0000 7fb480800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815cdc0 0x7fb4780abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.650+0000 7fb480800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47815a440 con 0x7fb4780cf650 2024-03-23T11:18:37.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.650+0000 7fb480800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4780aa950 con 0x7fb47815c9e0 2024-03-23T11:18:37.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.650+0000 7fb480800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4780aa0b0 con 0x7fb47815cdc0 2024-03-23T11:18:37.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47d400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb47815c9e0 0x7fb4780abaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47e800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815cdc0 0x7fb4780abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47d400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb47815c9e0 0x7fb4780abaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:49996/0 (socket says 172.21.15.120:49996) 2024-03-23T11:18:37.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47e800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815cdc0 0x7fb4780abfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:45586/0 (socket says 172.21.15.120:45586) 2024-03-23T11:18:37.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47d400640 1 -- 172.21.15.120:0/2140632739 learned_addr learned my addr 172.21.15.120:0/2140632739 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47de00640 1 --2- 172.21.15.120:0/2140632739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4780cf650 0x7fb478159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47e800640 1 -- 172.21.15.120:0/2140632739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb47815c9e0 msgr2=0x7fb4780abaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47e800640 1 --2- 172.21.15.120:0/2140632739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb47815c9e0 0x7fb4780abaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47e800640 1 -- 172.21.15.120:0/2140632739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4780cf650 msgr2=0x7fb478159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47e800640 1 --2- 172.21.15.120:0/2140632739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4780cf650 0x7fb478159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47e800640 1 -- 172.21.15.120:0/2140632739 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c0094d0 con 0x7fb47815cdc0 2024-03-23T11:18:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47d400640 1 --2- 172.21.15.120:0/2140632739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb47815c9e0 0x7fb4780abaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:18:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47de00640 1 --2- 172.21.15.120:0/2140632739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4780cf650 0x7fb478159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:18:37.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.651+0000 7fb47e800640 1 --2- 172.21.15.120:0/2140632739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815cdc0 0x7fb4780abfe0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb474007d90 tx=0x7fb47400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:37.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.652+0000 7fb472000640 1 -- 172.21.15.120:0/2140632739 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474018070 con 0x7fb47815cdc0 2024-03-23T11:18:37.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.652+0000 7fb472000640 1 -- 172.21.15.120:0/2140632739 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47400edd0 con 0x7fb47815cdc0 2024-03-23T11:18:37.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.652+0000 7fb472000640 1 -- 172.21.15.120:0/2140632739 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474003bb0 con 0x7fb47815cdc0 2024-03-23T11:18:37.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.652+0000 7fb480800640 1 -- 172.21.15.120:0/2140632739 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4780aa390 con 0x7fb47815cdc0 2024-03-23T11:18:37.654 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.652+0000 7fb480800640 1 -- 172.21.15.120:0/2140632739 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4781f2920 con 0x7fb47815cdc0 2024-03-23T11:18:37.655 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.652+0000 7fb472000640 1 -- 172.21.15.120:0/2140632739 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb474004320 con 0x7fb47815cdc0 2024-03-23T11:18:37.655 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.652+0000 7fb472000640 1 -- 172.21.15.120:0/2140632739 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fb4740148b0 con 0x7fb47815cdc0 2024-03-23T11:18:37.655 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.653+0000 7fb480800640 1 -- 172.21.15.120:0/2140632739 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb478066c90 con 0x7fb47815cdc0 2024-03-23T11:18:37.656 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.656+0000 7fb472000640 1 -- 172.21.15.120:0/2140632739 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fb4740047b0 con 0x7fb47815cdc0 2024-03-23T11:18:37.665 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:37.664+0000 7f50f2e14040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T11:18:37.728 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.727+0000 7fb480800640 1 -- 172.21.15.120:0/2140632739 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "64153585-2982-4f26-98c5-b4cc9036dab9", "id": 1} v 0) v1 -- 0x7fb4781f2ae0 con 0x7fb47815cdc0 2024-03-23T11:18:37.737 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.736+0000 7fb472000640 1 -- 172.21.15.120:0/2140632739 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "64153585-2982-4f26-98c5-b4cc9036dab9", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb47402c0b0 con 0x7fb47815cdc0 2024-03-23T11:18:37.737 INFO:teuthology.orchestra.run.smithi120.stdout:1 2024-03-23T11:18:37.738 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.737+0000 7fb480800640 1 -- 172.21.15.120:0/2140632739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815cdc0 msgr2=0x7fb4780abfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.738 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.737+0000 7fb480800640 1 --2- 172.21.15.120:0/2140632739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815cdc0 0x7fb4780abfe0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb474007d90 tx=0x7fb47400e4b0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.739 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.738+0000 7fb480800640 1 -- 172.21.15.120:0/2140632739 shutdown_connections 2024-03-23T11:18:37.739 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.738+0000 7fb480800640 1 --2- 172.21.15.120:0/2140632739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb47815cdc0 0x7fb4780abfe0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.739 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.738+0000 7fb480800640 1 --2- 172.21.15.120:0/2140632739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb47815c9e0 0x7fb4780abaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.739 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.738+0000 7fb480800640 1 --2- 172.21.15.120:0/2140632739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4780cf650 0x7fb478159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.739 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.738+0000 7fb480800640 1 -- 172.21.15.120:0/2140632739 >> 172.21.15.120:0/2140632739 conn(0x7fb478071570 msgr2=0x7fb478071950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:37.739 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.738+0000 7fb480800640 1 -- 172.21.15.120:0/2140632739 shutdown_connections 2024-03-23T11:18:37.739 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.738+0000 7fb480800640 1 -- 172.21.15.120:0/2140632739 wait complete. 2024-03-23T11:18:37.755 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 0ddbbec9-5271-4946-a229-c4b7924b2573 2 2024-03-23T11:18:37.778 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:37.778+0000 7fef97c42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T11:18:37.802 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:37.797+0000 7f50f2e14040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T11:18:37.861 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:37.861+0000 7fef97c42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T11:18:37.883 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:37.882+0000 7f50f2e14040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T11:18:37.917 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:37.917+0000 7fef97c42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T11:18:37.937 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:37.937+0000 7f50f2e14040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T11:18:37.979 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:37.979+0000 7fef97c42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T11:18:37.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.982+0000 7f6059200640 1 Processor -- start 2024-03-23T11:18:37.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.982+0000 7f6059200640 1 -- start start 2024-03-23T11:18:37.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.984+0000 7f6059200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60540b25a0 0x7f60540b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.984+0000 7f6059200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60540cbb10 con 0x7f60540cb700 2024-03-23T11:18:37.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.984+0000 7f6059200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60540b2ec0 con 0x7f6054142f10 2024-03-23T11:18:37.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.984+0000 7f6059200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60540cbd40 con 0x7f60540b25a0 2024-03-23T11:18:37.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.984+0000 7f6052a00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f6054142f10 0x7f60540caff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:38762/0 (socket says 172.21.15.120:38762) 2024-03-23T11:18:37.985 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.984+0000 7f6052a00640 1 -- 172.21.15.120:0/1494001797 learned_addr learned my addr 172.21.15.120:0/1494001797 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-23T11:18:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.984+0000 7f6053400640 1 --2- 172.21.15.120:0/1494001797 >> v2:172.21.15.62:3300/0 conn(0x7f60540b25a0 0x7f60540b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.984+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682124615 0 0) 0x7f60540b2ec0 con 0x7f6054142f10 2024-03-23T11:18:37.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6038003520 con 0x7f6054142f10 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716913202 0 0) 0x7f6038003520 con 0x7f6054142f10 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 >> v2:172.21.15.62:3300/0 conn(0x7f60540b25a0 msgr2=0x7f60540b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 --2- 172.21.15.120:0/1494001797 >> v2:172.21.15.62:3300/0 conn(0x7f60540b25a0 0x7f60540b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 >> v1:172.21.15.62:6789/0 conn(0x7f60540cb700 legacy=0x7f605406c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6054159d00 con 0x7f6054142f10 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f6040001670 con 0x7f6054142f10 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 >> v1:172.21.15.120:6789/0 conn(0x7f6054142f10 legacy=0x7f60540caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 --2- 172.21.15.120:0/1494001797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60380039c0 0x7f6038003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 --2- 172.21.15.120:0/1494001797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60380047c0 0x7f6038004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 --2- 172.21.15.120:0/1494001797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6038005180 0x7f603800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60540cbd40 con 0x7f60380039c0 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603800c080 con 0x7f60380047c0 2024-03-23T11:18:37.988 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603800e450 con 0x7f6038005180 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6052a00640 1 --2- 172.21.15.120:0/1494001797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60380047c0 0x7f6038004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6053e00640 1 --2- 172.21.15.120:0/1494001797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6038005180 0x7f603800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.985+0000 7f6053400640 1 --2- 172.21.15.120:0/1494001797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60380039c0 0x7f6038003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.986+0000 7f6052a00640 1 -- 172.21.15.120:0/1494001797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6038005180 msgr2=0x7f603800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.986+0000 7f6052a00640 1 --2- 172.21.15.120:0/1494001797 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6038005180 0x7f603800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.986+0000 7f6052a00640 1 -- 172.21.15.120:0/1494001797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60380039c0 msgr2=0x7f6038003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.986+0000 7f6052a00640 1 --2- 172.21.15.120:0/1494001797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60380039c0 0x7f6038003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.986+0000 7f6052a00640 1 -- 172.21.15.120:0/1494001797 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6038003520 con 0x7f60380047c0 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.986+0000 7f6053e00640 1 -- 172.21.15.120:0/1494001797 reap_dead start 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.987+0000 7f6052a00640 1 --2- 172.21.15.120:0/1494001797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60380047c0 0x7f6038004c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f604000c0e0 tx=0x7f60380057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd2a5469d77173de server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.987+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040005ad0 con 0x7f60380047c0 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.987+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6040005c30 con 0x7f60380047c0 2024-03-23T11:18:37.989 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.987+0000 7f6052000640 1 -- 172.21.15.120:0/1494001797 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604002ca00 con 0x7f60380047c0 2024-03-23T11:18:37.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.988+0000 7f6059200640 1 -- 172.21.15.120:0/1494001797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60380047c0 msgr2=0x7f6038004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.988+0000 7f6059200640 1 --2- 172.21.15.120:0/1494001797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60380047c0 0x7f6038004c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f604000c0e0 tx=0x7f60380057f0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.988+0000 7f6059200640 1 -- 172.21.15.120:0/1494001797 shutdown_connections 2024-03-23T11:18:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.988+0000 7f6059200640 1 --2- 172.21.15.120:0/1494001797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60380047c0 0x7f6038004c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.988+0000 7f6059200640 1 -- 172.21.15.120:0/1494001797 >> 172.21.15.120:0/1494001797 conn(0x7f60540b0090 msgr2=0x7f60540b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.988+0000 7f6059200640 1 -- 172.21.15.120:0/1494001797 shutdown_connections 2024-03-23T11:18:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.988+0000 7f6059200640 1 -- 172.21.15.120:0/1494001797 wait complete. 2024-03-23T11:18:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.989+0000 7f6059200640 1 Processor -- start 2024-03-23T11:18:37.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.989+0000 7f6059200640 1 -- start start 2024-03-23T11:18:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.990+0000 7f6059200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60380047c0 0x7f60541ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.990+0000 7f6059200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60540ab7a0 0x7f6054069ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.990+0000 7f6059200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605413eb80 0x7f605413f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.990+0000 7f6059200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605413f540 con 0x7f60380047c0 2024-03-23T11:18:37.993 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.990+0000 7f6059200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60540abc20 con 0x7f60540ab7a0 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.990+0000 7f6059200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60540abed0 con 0x7f605413eb80 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6052a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60540ab7a0 0x7f6054069ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6052a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60540ab7a0 0x7f6054069ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:50020/0 (socket says 172.21.15.120:50020) 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6052a00640 1 -- 172.21.15.120:0/1296369567 learned_addr learned my addr 172.21.15.120:0/1296369567 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6053e00640 1 --2- 172.21.15.120:0/1296369567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605413eb80 0x7f605413f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6053400640 1 --2- 172.21.15.120:0/1296369567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60380047c0 0x7f60541ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6052a00640 1 -- 172.21.15.120:0/1296369567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605413eb80 msgr2=0x7f605413f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6052a00640 1 --2- 172.21.15.120:0/1296369567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605413eb80 0x7f605413f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6052a00640 1 -- 172.21.15.120:0/1296369567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60380047c0 msgr2=0x7f60541ee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6052a00640 1 --2- 172.21.15.120:0/1296369567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60380047c0 0x7f60541ee750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6052a00640 1 -- 172.21.15.120:0/1296369567 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60540ac1b0 con 0x7f60540ab7a0 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6053e00640 1 --2- 172.21.15.120:0/1296369567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605413eb80 0x7f605413f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:18:37.994 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.991+0000 7f6052a00640 1 --2- 172.21.15.120:0/1296369567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60540ab7a0 0x7f6054069ed0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6040000c00 tx=0x7f604002e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.992+0000 7f604b400640 1 -- 172.21.15.120:0/1296369567 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604003c040 con 0x7f60540ab7a0 2024-03-23T11:18:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.992+0000 7f604b400640 1 -- 172.21.15.120:0/1296369567 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604002ccb0 con 0x7f60540ab7a0 2024-03-23T11:18:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.992+0000 7f604b400640 1 -- 172.21.15.120:0/1296369567 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60400072b0 con 0x7f60540ab7a0 2024-03-23T11:18:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.992+0000 7f6059200640 1 -- 172.21.15.120:0/1296369567 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60541eedb0 con 0x7f60540ab7a0 2024-03-23T11:18:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.992+0000 7f6059200640 1 -- 172.21.15.120:0/1296369567 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60541ef370 con 0x7f60540ab7a0 2024-03-23T11:18:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.992+0000 7f604b400640 1 -- 172.21.15.120:0/1296369567 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6040032070 con 0x7f60540ab7a0 2024-03-23T11:18:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.992+0000 7f604b400640 1 -- 172.21.15.120:0/1296369567 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f6040007710 con 0x7f60540ab7a0 2024-03-23T11:18:37.995 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.993+0000 7f6059200640 1 -- 172.21.15.120:0/1296369567 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605404e030 con 0x7f60540ab7a0 2024-03-23T11:18:37.996 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:37.995+0000 7f604b400640 1 -- 172.21.15.120:0/1296369567 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f6040007c40 con 0x7f60540ab7a0 2024-03-23T11:18:38.002 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:38.001+0000 7f50f2e14040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T11:18:38.037 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:38.036+0000 7fef97c42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T11:18:38.057 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:38.057+0000 7f50f2e14040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T11:18:38.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.073+0000 7f6059200640 1 -- 172.21.15.120:0/1296369567 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0ddbbec9-5271-4946-a229-c4b7924b2573", "id": 2} v 0) v1 -- 0x7f60541ef500 con 0x7f60540ab7a0 2024-03-23T11:18:38.082 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.081+0000 7f604b400640 1 -- 172.21.15.120:0/1296369567 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0ddbbec9-5271-4946-a229-c4b7924b2573", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f604002ce20 con 0x7f60540ab7a0 2024-03-23T11:18:38.082 INFO:teuthology.orchestra.run.smithi120.stdout:2 2024-03-23T11:18:38.083 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.082+0000 7f6059200640 1 -- 172.21.15.120:0/1296369567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60540ab7a0 msgr2=0x7f6054069ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.082+0000 7f6059200640 1 --2- 172.21.15.120:0/1296369567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60540ab7a0 0x7f6054069ed0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6040000c00 tx=0x7f604002e070 comp rx=0 tx=0).stop 2024-03-23T11:18:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.083+0000 7f6059200640 1 -- 172.21.15.120:0/1296369567 shutdown_connections 2024-03-23T11:18:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.083+0000 7f6059200640 1 --2- 172.21.15.120:0/1296369567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f605413eb80 0x7f605413f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.083+0000 7f6059200640 1 --2- 172.21.15.120:0/1296369567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60540ab7a0 0x7f6054069ed0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.083+0000 7f6059200640 1 --2- 172.21.15.120:0/1296369567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60380047c0 0x7f60541ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.083+0000 7f6059200640 1 -- 172.21.15.120:0/1296369567 >> 172.21.15.120:0/1296369567 conn(0x7f60540b0090 msgr2=0x7f605415c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.083+0000 7f6059200640 1 -- 172.21.15.120:0/1296369567 shutdown_connections 2024-03-23T11:18:38.084 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.083+0000 7f6059200640 1 -- 172.21.15.120:0/1296369567 wait complete. 2024-03-23T11:18:38.100 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new c7e6782f-d9c6-4e49-8191-bfc31bc3b274 3 2024-03-23T11:18:38.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.282+0000 7f4a0d000640 1 Processor -- start 2024-03-23T11:18:38.283 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.282+0000 7f4a0d000640 1 -- start start 2024-03-23T11:18:38.285 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.283+0000 7f4a0d000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4a080ceaf0 0x7f4a080989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.285 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.283+0000 7f4a0d000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a0815b8f0 0x7f4a080ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.285 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.283+0000 7f4a0d000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a0815c210 con 0x7f4a0815d3b0 2024-03-23T11:18:38.285 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.283+0000 7f4a0d000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a08099c30 con 0x7f4a080ceaf0 2024-03-23T11:18:38.285 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.283+0000 7f4a0d000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a080cf410 con 0x7f4a0815b8f0 2024-03-23T11:18:38.285 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.284+0000 7f49fea00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a0815b8f0 0x7f4a080ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.285 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.284+0000 7f49fea00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a0815b8f0 0x7f4a080ceed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:50026/0 (socket says 172.21.15.120:50026) 2024-03-23T11:18:38.285 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.284+0000 7f49fea00640 1 -- 172.21.15.120:0/1631833284 learned_addr learned my addr 172.21.15.120:0/1631833284 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:38.285 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.284+0000 7f4a07e00640 1 --1- 172.21.15.120:0/1631833284 >> v1:172.21.15.120:6789/0 conn(0x7f4a0815d3b0 0x7f4a0815d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:38774/0 (socket says 172.21.15.120:38774) 2024-03-23T11:18:38.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.284+0000 7f4a07400640 1 --2- 172.21.15.120:0/1631833284 >> v2:172.21.15.62:3300/0 conn(0x7f4a080ceaf0 0x7f4a080989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.284+0000 7f49fea00640 1 -- 172.21.15.120:0/1631833284 >> v2:172.21.15.62:3300/0 conn(0x7f4a080ceaf0 msgr2=0x7f4a080989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:38.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.284+0000 7f49fea00640 1 --2- 172.21.15.120:0/1631833284 >> v2:172.21.15.62:3300/0 conn(0x7f4a080ceaf0 0x7f4a080989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.284+0000 7f49fea00640 1 -- 172.21.15.120:0/1631833284 >> v1:172.21.15.120:6789/0 conn(0x7f4a0815d3b0 legacy=0x7f4a0815d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.284+0000 7f49fea00640 1 -- 172.21.15.120:0/1631833284 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a08098fb0 con 0x7f4a0815b8f0 2024-03-23T11:18:38.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.284+0000 7f4a06a00640 1 -- 172.21.15.120:0/1631833284 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984262837 0 0) 0x7f4a0815c210 con 0x7f4a0815d3b0 2024-03-23T11:18:38.287 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.285+0000 7f49fea00640 1 --2- 172.21.15.120:0/1631833284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a0815b8f0 0x7f4a080ceed0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4a080cab10 tx=0x7f49f002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=445bb1202e25dff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:38.287 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.285+0000 7f4a06a00640 1 -- 172.21.15.120:0/1631833284 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f0030860 con 0x7f4a0815b8f0 2024-03-23T11:18:38.287 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.286+0000 7f4a06a00640 1 -- 172.21.15.120:0/1631833284 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f00309c0 con 0x7f4a0815b8f0 2024-03-23T11:18:38.287 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.286+0000 7f4a06a00640 1 -- 172.21.15.120:0/1631833284 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f0030d50 con 0x7f4a0815b8f0 2024-03-23T11:18:38.287 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.286+0000 7f4a0d000640 1 -- 172.21.15.120:0/1631833284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a0815b8f0 msgr2=0x7f4a080ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.287 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.286+0000 7f4a0d000640 1 --2- 172.21.15.120:0/1631833284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a0815b8f0 0x7f4a080ceed0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4a080cab10 tx=0x7f49f002b9a0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.288 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.286+0000 7f4a0d000640 1 -- 172.21.15.120:0/1631833284 shutdown_connections 2024-03-23T11:18:38.288 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.286+0000 7f4a0d000640 1 --2- 172.21.15.120:0/1631833284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a0815b8f0 0x7f4a080ceed0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.288 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.286+0000 7f4a0d000640 1 --2- 172.21.15.120:0/1631833284 >> v2:172.21.15.62:3300/0 conn(0x7f4a080ceaf0 0x7f4a080989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.288 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.286+0000 7f4a0d000640 1 -- 172.21.15.120:0/1631833284 >> 172.21.15.120:0/1631833284 conn(0x7f4a080b2020 msgr2=0x7f4a080b2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:38.288 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.286+0000 7f4a0d000640 1 -- 172.21.15.120:0/1631833284 shutdown_connections 2024-03-23T11:18:38.288 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.286+0000 7f4a0d000640 1 -- 172.21.15.120:0/1631833284 wait complete. 2024-03-23T11:18:38.288 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.288+0000 7f4a0d000640 1 Processor -- start 2024-03-23T11:18:38.288 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.288+0000 7f4a0d000640 1 -- start start 2024-03-23T11:18:38.291 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a0d000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a080ceaf0 0x7f4a0815df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.291 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a0d000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a0815b8f0 0x7f4a080c4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.291 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a0d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a0815d3b0 0x7f4a081535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.291 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a0d000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080c4560 con 0x7f4a0815d3b0 2024-03-23T11:18:38.291 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a0d000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080c7730 con 0x7f4a080ceaf0 2024-03-23T11:18:38.291 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a0d000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a08153af0 con 0x7f4a0815b8f0 2024-03-23T11:18:38.291 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a07400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a080ceaf0 0x7f4a0815df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a07400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a080ceaf0 0x7f4a0815df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:50036/0 (socket says 172.21.15.120:50036) 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a07400640 1 -- 172.21.15.120:0/1079174901 learned_addr learned my addr 172.21.15.120:0/1079174901 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f49fea00640 1 --2- 172.21.15.120:0/1079174901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a0815b8f0 0x7f4a080c4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a07e00640 1 --2- 172.21.15.120:0/1079174901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a0815d3b0 0x7f4a081535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a07400640 1 -- 172.21.15.120:0/1079174901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a0815b8f0 msgr2=0x7f4a080c4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.289+0000 7f4a07400640 1 --2- 172.21.15.120:0/1079174901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a0815b8f0 0x7f4a080c4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.290+0000 7f4a07400640 1 -- 172.21.15.120:0/1079174901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a0815d3b0 msgr2=0x7f4a081535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.290+0000 7f4a07400640 1 --2- 172.21.15.120:0/1079174901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a0815d3b0 0x7f4a081535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.290+0000 7f4a07400640 1 -- 172.21.15.120:0/1079174901 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f00094d0 con 0x7f4a080ceaf0 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.290+0000 7f4a07e00640 1 --2- 172.21.15.120:0/1079174901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a0815d3b0 0x7f4a081535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.290+0000 7f4a07400640 1 --2- 172.21.15.120:0/1079174901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a080ceaf0 0x7f4a0815df50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f49f8007e40 tx=0x7f49f800e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.290+0000 7f49ffe00640 1 -- 172.21.15.120:0/1079174901 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f8014620 con 0x7f4a080ceaf0 2024-03-23T11:18:38.292 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.290+0000 7f49ffe00640 1 -- 172.21.15.120:0/1079174901 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f800ee90 con 0x7f4a080ceaf0 2024-03-23T11:18:38.293 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.290+0000 7f49ffe00640 1 -- 172.21.15.120:0/1079174901 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f80149c0 con 0x7f4a080ceaf0 2024-03-23T11:18:38.293 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.290+0000 7f4a0d000640 1 -- 172.21.15.120:0/1079174901 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a080c4770 con 0x7f4a080ceaf0 2024-03-23T11:18:38.293 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.290+0000 7f4a0d000640 1 -- 172.21.15.120:0/1079174901 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a080c79d0 con 0x7f4a080ceaf0 2024-03-23T11:18:38.293 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.290+0000 7f49ffe00640 1 -- 172.21.15.120:0/1079174901 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f49f801e460 con 0x7f4a080ceaf0 2024-03-23T11:18:38.293 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.291+0000 7f49ffe00640 1 -- 172.21.15.120:0/1079174901 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f49f801eba0 con 0x7f4a080ceaf0 2024-03-23T11:18:38.293 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.291+0000 7f4a0d000640 1 -- 172.21.15.120:0/1079174901 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a0815b710 con 0x7f4a080ceaf0 2024-03-23T11:18:38.294 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.294+0000 7f49ffe00640 1 -- 172.21.15.120:0/1079174901 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f49f801e710 con 0x7f4a080ceaf0 2024-03-23T11:18:38.371 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.371+0000 7f4a0d000640 1 -- 172.21.15.120:0/1079174901 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c7e6782f-d9c6-4e49-8191-bfc31bc3b274", "id": 3} v 0) v1 -- 0x7f4a080672c0 con 0x7f4a080ceaf0 2024-03-23T11:18:38.381 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.380+0000 7f49ffe00640 1 -- 172.21.15.120:0/1079174901 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c7e6782f-d9c6-4e49-8191-bfc31bc3b274", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f49f802d0b0 con 0x7f4a080ceaf0 2024-03-23T11:18:38.381 INFO:teuthology.orchestra.run.smithi120.stdout:3 2024-03-23T11:18:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.382+0000 7f4a0d000640 1 -- 172.21.15.120:0/1079174901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a080ceaf0 msgr2=0x7f4a0815df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.382+0000 7f4a0d000640 1 --2- 172.21.15.120:0/1079174901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a080ceaf0 0x7f4a0815df50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f49f8007e40 tx=0x7f49f800e4b0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.382+0000 7f4a0d000640 1 -- 172.21.15.120:0/1079174901 shutdown_connections 2024-03-23T11:18:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.382+0000 7f4a0d000640 1 --2- 172.21.15.120:0/1079174901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a0815d3b0 0x7f4a081535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.382+0000 7f4a0d000640 1 --2- 172.21.15.120:0/1079174901 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a0815b8f0 0x7f4a080c4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.382+0000 7f4a0d000640 1 --2- 172.21.15.120:0/1079174901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a080ceaf0 0x7f4a0815df50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.382+0000 7f4a0d000640 1 -- 172.21.15.120:0/1079174901 >> 172.21.15.120:0/1079174901 conn(0x7f4a080b2020 msgr2=0x7f4a080cc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.382+0000 7f4a0d000640 1 -- 172.21.15.120:0/1079174901 shutdown_connections 2024-03-23T11:18:38.383 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.382+0000 7f4a0d000640 1 -- 172.21.15.120:0/1079174901 wait complete. 2024-03-23T11:18:38.397 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9 4 2024-03-23T11:18:38.507 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:38.507+0000 7fef97c42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T11:18:38.529 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:38.528+0000 7f50f2e14040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T11:18:38.585 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.584+0000 7f7ebda00640 1 Processor -- start 2024-03-23T11:18:38.585 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.584+0000 7f7ebda00640 1 -- start start 2024-03-23T11:18:38.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7ebda00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815c900 0x7f7eb80d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7ebda00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7eb815cce0 0x7f7eb80b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7ebda00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb80d1100 con 0x7f7eb80cd690 2024-03-23T11:18:38.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7ebda00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb80d13e0 con 0x7f7eb815cce0 2024-03-23T11:18:38.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7ebda00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb80d16c0 con 0x7f7eb815c900 2024-03-23T11:18:38.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7eb6a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815c900 0x7f7eb80d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7eb6a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815c900 0x7f7eb80d21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:45616/0 (socket says 172.21.15.120:45616) 2024-03-23T11:18:38.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7eb6a00640 1 -- 172.21.15.120:0/4263650795 learned_addr learned my addr 172.21.15.120:0/4263650795 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:38.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7eafe00640 1 --2- 172.21.15.120:0/4263650795 >> v2:172.21.15.62:3300/0 conn(0x7f7eb815cce0 0x7f7eb80b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7eb6a00640 1 -- 172.21.15.120:0/4263650795 >> v2:172.21.15.62:3300/0 conn(0x7f7eb815cce0 msgr2=0x7f7eb80b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:38.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7eb6a00640 1 --2- 172.21.15.120:0/4263650795 >> v2:172.21.15.62:3300/0 conn(0x7f7eb815cce0 0x7f7eb80b82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7eb6a00640 1 -- 172.21.15.120:0/4263650795 >> v1:172.21.15.62:6789/0 conn(0x7f7eb80cd690 legacy=0x7f7eb80b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:38.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7eb6a00640 1 -- 172.21.15.120:0/4263650795 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb80d19a0 con 0x7f7eb815c900 2024-03-23T11:18:38.588 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.586+0000 7f7eafe00640 1 --2- 172.21.15.120:0/4263650795 >> v2:172.21.15.62:3300/0 conn(0x7f7eb815cce0 0x7f7eb80b82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:18:38.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.588+0000 7f7eb6a00640 1 --2- 172.21.15.120:0/4263650795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815c900 0x7f7eb80d21a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8009870 tx=0x7f7ea802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2309634deef271 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:38.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.588+0000 7f7eb6000640 1 -- 172.21.15.120:0/4263650795 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea8030840 con 0x7f7eb815c900 2024-03-23T11:18:38.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.588+0000 7f7eb6000640 1 -- 172.21.15.120:0/4263650795 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea80309a0 con 0x7f7eb815c900 2024-03-23T11:18:38.589 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.588+0000 7f7eb6000640 1 -- 172.21.15.120:0/4263650795 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea8030d30 con 0x7f7eb815c900 2024-03-23T11:18:38.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.589+0000 7f7ebda00640 1 -- 172.21.15.120:0/4263650795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815c900 msgr2=0x7f7eb80d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.589+0000 7f7ebda00640 1 --2- 172.21.15.120:0/4263650795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815c900 0x7f7eb80d21a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8009870 tx=0x7f7ea802b980 comp rx=0 tx=0).stop 2024-03-23T11:18:38.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.589+0000 7f7ebda00640 1 -- 172.21.15.120:0/4263650795 shutdown_connections 2024-03-23T11:18:38.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.589+0000 7f7ebda00640 1 --2- 172.21.15.120:0/4263650795 >> v2:172.21.15.62:3300/0 conn(0x7f7eb815cce0 0x7f7eb80b82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.589+0000 7f7ebda00640 1 --2- 172.21.15.120:0/4263650795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815c900 0x7f7eb80d21a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.589+0000 7f7ebda00640 1 -- 172.21.15.120:0/4263650795 >> 172.21.15.120:0/4263650795 conn(0x7f7eb8071390 msgr2=0x7f7eb8071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:38.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.589+0000 7f7ebda00640 1 -- 172.21.15.120:0/4263650795 shutdown_connections 2024-03-23T11:18:38.590 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.589+0000 7f7ebda00640 1 -- 172.21.15.120:0/4263650795 wait complete. 2024-03-23T11:18:38.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.591+0000 7f7ebda00640 1 Processor -- start 2024-03-23T11:18:38.591 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.591+0000 7f7ebda00640 1 -- start start 2024-03-23T11:18:38.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.591+0000 7f7ebda00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7eb80cd690 0x7f7eb8157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7ebda00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eb815c900 0x7f7eb8154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7ebda00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815cce0 0x7f7eb80cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7ebda00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb80c8480 con 0x7f7eb80cd690 2024-03-23T11:18:38.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7ebda00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb80c8760 con 0x7f7eb815c900 2024-03-23T11:18:38.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7ebda00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb80c8a40 con 0x7f7eb815cce0 2024-03-23T11:18:38.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7eafe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eb815c900 0x7f7eb8154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7eb7400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815cce0 0x7f7eb80cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7eafe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eb815c900 0x7f7eb8154090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:50044/0 (socket says 172.21.15.120:50044) 2024-03-23T11:18:38.594 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7eafe00640 1 -- 172.21.15.120:0/2896965603 learned_addr learned my addr 172.21.15.120:0/2896965603 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7eafe00640 1 -- 172.21.15.120:0/2896965603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815cce0 msgr2=0x7f7eb80cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7eafe00640 1 --2- 172.21.15.120:0/2896965603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815cce0 0x7f7eb80cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7eafe00640 1 -- 172.21.15.120:0/2896965603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7eb80cd690 msgr2=0x7f7eb8157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7eafe00640 1 --2- 172.21.15.120:0/2896965603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7eb80cd690 0x7f7eb8157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.592+0000 7f7eafe00640 1 -- 172.21.15.120:0/2896965603 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea80094d0 con 0x7f7eb815c900 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.593+0000 7f7eafe00640 1 --2- 172.21.15.120:0/2896965603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eb815c900 0x7f7eb8154090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000c7f0 tx=0x7f7ea000ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.593+0000 7f7eaea00640 1 -- 172.21.15.120:0/2896965603 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0007820 con 0x7f7eb815c900 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.593+0000 7f7eaea00640 1 -- 172.21.15.120:0/2896965603 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea0007980 con 0x7f7eb815c900 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.593+0000 7f7eaea00640 1 -- 172.21.15.120:0/2896965603 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0007cf0 con 0x7f7eb815c900 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.593+0000 7f7ebda00640 1 -- 172.21.15.120:0/2896965603 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb80c8d20 con 0x7f7eb815c900 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.593+0000 7f7ebda00640 1 -- 172.21.15.120:0/2896965603 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb81545d0 con 0x7f7eb815c900 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.593+0000 7f7eaea00640 1 -- 172.21.15.120:0/2896965603 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7ea001a450 con 0x7f7eb815c900 2024-03-23T11:18:38.595 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.593+0000 7f7eaea00640 1 -- 172.21.15.120:0/2896965603 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f7ea0011c90 con 0x7f7eb815c900 2024-03-23T11:18:38.596 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.594+0000 7f7ebda00640 1 -- 172.21.15.120:0/2896965603 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb80c9d70 con 0x7f7eb815c900 2024-03-23T11:18:38.597 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.596+0000 7f7eaea00640 1 -- 172.21.15.120:0/2896965603 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f7ea001f060 con 0x7f7eb815c900 2024-03-23T11:18:38.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.673+0000 7f7ebda00640 1 -- 172.21.15.120:0/2896965603 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9", "id": 4} v 0) v1 -- 0x7f7eb80672c0 con 0x7f7eb815c900 2024-03-23T11:18:38.674 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:38.674+0000 7fef97c42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T11:18:38.684 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.683+0000 7f7eaea00640 1 -- 172.21.15.120:0/2896965603 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7ea0015070 con 0x7f7eb815c900 2024-03-23T11:18:38.684 INFO:teuthology.orchestra.run.smithi120.stdout:4 2024-03-23T11:18:38.685 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.685+0000 7f7ebda00640 1 -- 172.21.15.120:0/2896965603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eb815c900 msgr2=0x7f7eb8154090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.685+0000 7f7ebda00640 1 --2- 172.21.15.120:0/2896965603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eb815c900 0x7f7eb8154090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000c7f0 tx=0x7f7ea000ccb0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.685+0000 7f7ebda00640 1 -- 172.21.15.120:0/2896965603 shutdown_connections 2024-03-23T11:18:38.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.685+0000 7f7ebda00640 1 --2- 172.21.15.120:0/2896965603 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7eb815cce0 0x7f7eb80cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.685+0000 7f7ebda00640 1 --2- 172.21.15.120:0/2896965603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7eb815c900 0x7f7eb8154090 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.685+0000 7f7ebda00640 1 --2- 172.21.15.120:0/2896965603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7eb80cd690 0x7f7eb8157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.685+0000 7f7ebda00640 1 -- 172.21.15.120:0/2896965603 >> 172.21.15.120:0/2896965603 conn(0x7f7eb8071390 msgr2=0x7f7eb815b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:38.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.685+0000 7f7ebda00640 1 -- 172.21.15.120:0/2896965603 shutdown_connections 2024-03-23T11:18:38.686 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.685+0000 7f7ebda00640 1 -- 172.21.15.120:0/2896965603 wait complete. 2024-03-23T11:18:38.694 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:38.694+0000 7f50f2e14040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T11:18:38.702 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 865988b2-5f37-4722-a2fc-8c611abd221a 5 2024-03-23T11:18:38.817 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:38.817+0000 7fef97c42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T11:18:38.850 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:38.849+0000 7f50f2e14040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T11:18:38.873 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:38.872+0000 7fef97c42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T11:18:38.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.888+0000 7f7a0b200640 1 Processor -- start 2024-03-23T11:18:38.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.888+0000 7f7a0b200640 1 -- start start 2024-03-23T11:18:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.889+0000 7f7a0b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 0x7f7a040d3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.890+0000 7f7a0b200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a040d3600 con 0x7f7a0415cba0 2024-03-23T11:18:38.891 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.890+0000 7f7a0b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0415e030 con 0x7f7a040ce8e0 2024-03-23T11:18:38.891 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.890+0000 7f7a0b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0415dc10 con 0x7f7a040b4880 2024-03-23T11:18:38.891 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.890+0000 7f7a0a800640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f7a040ce8e0 0x7f7a040cd100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:49264/0 (socket says 172.21.15.120:49264) 2024-03-23T11:18:38.891 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.890+0000 7f7a0a800640 1 -- 172.21.15.120:0/1521965861 learned_addr learned my addr 172.21.15.120:0/1521965861 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-23T11:18:38.891 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.890+0000 7f7a09400640 1 --2- 172.21.15.120:0/1521965861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 0x7f7a040d3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.891 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.890+0000 7f7a09400640 1 -- 172.21.15.120:0/1521965861 >> v1:172.21.15.120:6789/0 conn(0x7f7a040ce8e0 legacy=0x7f7a040cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.891 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.890+0000 7f7a09400640 1 -- 172.21.15.120:0/1521965861 >> v1:172.21.15.62:6789/0 conn(0x7f7a0415cba0 legacy=0x7f7a040b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:38.891 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.890+0000 7f7a08a00640 1 -- 172.21.15.120:0/1521965861 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928594068 0 0) 0x7f7a0415e030 con 0x7f7a040ce8e0 2024-03-23T11:18:38.891 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.890+0000 7f7a09400640 1 -- 172.21.15.120:0/1521965861 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0415d270 con 0x7f7a040b4880 2024-03-23T11:18:38.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.891+0000 7f7a09400640 1 --2- 172.21.15.120:0/1521965861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 0x7f7a040d3020 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f79f40097d0 tx=0x7f79f402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb042b49315decc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:38.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.891+0000 7f7a08a00640 1 -- 172.21.15.120:0/1521965861 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4031770 con 0x7f7a040b4880 2024-03-23T11:18:38.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.892+0000 7f7a08a00640 1 -- 172.21.15.120:0/1521965861 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f40318d0 con 0x7f7a040b4880 2024-03-23T11:18:38.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.892+0000 7f7a08a00640 1 -- 172.21.15.120:0/1521965861 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4031c60 con 0x7f7a040b4880 2024-03-23T11:18:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.892+0000 7f7a0b200640 1 -- 172.21.15.120:0/1521965861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 msgr2=0x7f7a040d3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.892+0000 7f7a0b200640 1 --2- 172.21.15.120:0/1521965861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 0x7f7a040d3020 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f79f40097d0 tx=0x7f79f402b810 comp rx=0 tx=0).stop 2024-03-23T11:18:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.892+0000 7f7a0b200640 1 -- 172.21.15.120:0/1521965861 shutdown_connections 2024-03-23T11:18:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.892+0000 7f7a0b200640 1 --2- 172.21.15.120:0/1521965861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 0x7f7a040d3020 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.892+0000 7f7a0b200640 1 -- 172.21.15.120:0/1521965861 >> 172.21.15.120:0/1521965861 conn(0x7f7a04089f60 msgr2=0x7f7a040d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.892+0000 7f7a0b200640 1 -- 172.21.15.120:0/1521965861 shutdown_connections 2024-03-23T11:18:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.892+0000 7f7a0b200640 1 -- 172.21.15.120:0/1521965861 wait complete. 2024-03-23T11:18:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.894+0000 7f7a0b200640 1 Processor -- start 2024-03-23T11:18:38.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.894+0000 7f7a0b200640 1 -- start start 2024-03-23T11:18:38.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.895+0000 7f7a0b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 0x7f7a04158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.895+0000 7f7a0b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a040ce8e0 0x7f7a04152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.895+0000 7f7a0b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a0415cba0 0x7f7a04152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:38.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.895+0000 7f7a0b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a04153410 con 0x7f7a040ce8e0 2024-03-23T11:18:38.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.895+0000 7f7a0b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a040b21d0 con 0x7f7a0415cba0 2024-03-23T11:18:38.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.895+0000 7f7a0b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a040b24b0 con 0x7f7a040b4880 2024-03-23T11:18:38.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.895+0000 7f7a09e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 0x7f7a04158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.895+0000 7f7a0a800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a0415cba0 0x7f7a04152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.895+0000 7f7a09e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 0x7f7a04158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57450/0 (socket says 172.21.15.120:57450) 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.895+0000 7f7a09e00640 1 -- 172.21.15.120:0/2704831915 learned_addr learned my addr 172.21.15.120:0/2704831915 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f7a09400640 1 --2- 172.21.15.120:0/2704831915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a040ce8e0 0x7f7a04152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f7a09e00640 1 -- 172.21.15.120:0/2704831915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a0415cba0 msgr2=0x7f7a04152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f7a09e00640 1 --2- 172.21.15.120:0/2704831915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a0415cba0 0x7f7a04152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f7a09e00640 1 -- 172.21.15.120:0/2704831915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a040ce8e0 msgr2=0x7f7a04152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f7a09e00640 1 --2- 172.21.15.120:0/2704831915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a040ce8e0 0x7f7a04152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f7a09e00640 1 -- 172.21.15.120:0/2704831915 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f4009400 con 0x7f7a040b4880 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f7a09400640 1 --2- 172.21.15.120:0/2704831915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a040ce8e0 0x7f7a04152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f7a09e00640 1 --2- 172.21.15.120:0/2704831915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 0x7f7a04158300 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f79f800c450 tx=0x7f79f800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f79fe000640 1 -- 172.21.15.120:0/2704831915 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f80052d0 con 0x7f7a040b4880 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f79fe000640 1 -- 172.21.15.120:0/2704831915 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f8005430 con 0x7f7a040b4880 2024-03-23T11:18:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f7a0b200640 1 -- 172.21.15.120:0/2704831915 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a040b2790 con 0x7f7a040b4880 2024-03-23T11:18:38.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f79fe000640 1 -- 172.21.15.120:0/2704831915 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8012870 con 0x7f7a040b4880 2024-03-23T11:18:38.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.896+0000 7f7a0b200640 1 -- 172.21.15.120:0/2704831915 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a04158840 con 0x7f7a040b4880 2024-03-23T11:18:38.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.897+0000 7f79fe000640 1 -- 172.21.15.120:0/2704831915 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f79f8012cd0 con 0x7f7a040b4880 2024-03-23T11:18:38.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.897+0000 7f79fe000640 1 -- 172.21.15.120:0/2704831915 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f79f8011380 con 0x7f7a040b4880 2024-03-23T11:18:38.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.897+0000 7f7a0b200640 1 -- 172.21.15.120:0/2704831915 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a040b2cb0 con 0x7f7a040b4880 2024-03-23T11:18:38.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.900+0000 7f79fe000640 1 -- 172.21.15.120:0/2704831915 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f7a040b2cb0 con 0x7f7a040b4880 2024-03-23T11:18:38.907 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:38.906+0000 7f50f2e14040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T11:18:38.977 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.976+0000 7f7a0b200640 1 -- 172.21.15.120:0/2704831915 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "865988b2-5f37-4722-a2fc-8c611abd221a", "id": 5} v 0) v1 -- 0x7f7a041589d0 con 0x7f7a040b4880 2024-03-23T11:18:38.986 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.986+0000 7f79fe000640 1 -- 172.21.15.120:0/2704831915 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "865988b2-5f37-4722-a2fc-8c611abd221a", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f79f80035f0 con 0x7f7a040b4880 2024-03-23T11:18:38.989 INFO:teuthology.orchestra.run.smithi120.stdout:5 2024-03-23T11:18:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.987+0000 7f7a0b200640 1 -- 172.21.15.120:0/2704831915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 msgr2=0x7f7a04158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.987+0000 7f7a0b200640 1 --2- 172.21.15.120:0/2704831915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 0x7f7a04158300 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f79f800c450 tx=0x7f79f800c910 comp rx=0 tx=0).stop 2024-03-23T11:18:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.987+0000 7f7a0b200640 1 -- 172.21.15.120:0/2704831915 shutdown_connections 2024-03-23T11:18:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.987+0000 7f7a0b200640 1 --2- 172.21.15.120:0/2704831915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a040ce8e0 0x7f7a04152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.987+0000 7f7a0b200640 1 --2- 172.21.15.120:0/2704831915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a0415cba0 0x7f7a04152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.987+0000 7f7a0b200640 1 --2- 172.21.15.120:0/2704831915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b4880 0x7f7a04158300 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.987+0000 7f7a0b200640 1 -- 172.21.15.120:0/2704831915 >> 172.21.15.120:0/2704831915 conn(0x7f7a04089f60 msgr2=0x7f7a040af280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:38.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.987+0000 7f7a0b200640 1 -- 172.21.15.120:0/2704831915 shutdown_connections 2024-03-23T11:18:38.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:38.987+0000 7f7a0b200640 1 -- 172.21.15.120:0/2704831915 wait complete. 2024-03-23T11:18:39.004 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new b76f68e1-ec2c-438e-8275-90f2c3df293e 6 2024-03-23T11:18:39.005 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:39.004+0000 7fef97c42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T11:18:39.037 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:39.037+0000 7f50f2e14040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T11:18:39.120 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:39.120+0000 7fef97c42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T11:18:39.154 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:39.153+0000 7f50f2e14040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T11:18:39.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.180+0000 7f4531400640 1 Processor -- start 2024-03-23T11:18:39.181 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.181+0000 7f4531400640 1 -- start start 2024-03-23T11:18:39.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.182+0000 7f4531400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f452c0d0e40 0x7f452c0d12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.182+0000 7f4531400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f452c0cefa0 0x7f452c0cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.182+0000 7f4531400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c15cd70 con 0x7f452c15c990 2024-03-23T11:18:39.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.182+0000 7f4531400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452c0d17e0 con 0x7f452c0cefa0 2024-03-23T11:18:39.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.182+0000 7f4531400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452c0cf970 con 0x7f452c0d0e40 2024-03-23T11:18:39.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.182+0000 7f452b400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f452c0d0e40 0x7f452c0d12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.183+0000 7f452b400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f452c0d0e40 0x7f452c0d12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:34908/0 (socket says 172.21.15.120:34908) 2024-03-23T11:18:39.184 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.183+0000 7f452b400640 1 -- 172.21.15.120:0/2470795026 learned_addr learned my addr 172.21.15.120:0/2470795026 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:39.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.183+0000 7f4530a00640 1 --2- 172.21.15.120:0/2470795026 >> v2:172.21.15.62:3300/0 conn(0x7f452c0cefa0 0x7f452c0cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.183+0000 7f452b400640 1 -- 172.21.15.120:0/2470795026 >> v2:172.21.15.62:3300/0 conn(0x7f452c0cefa0 msgr2=0x7f452c0cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:39.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.183+0000 7f452b400640 1 --2- 172.21.15.120:0/2470795026 >> v2:172.21.15.62:3300/0 conn(0x7f452c0cefa0 0x7f452c0cf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.183+0000 7f452b400640 1 -- 172.21.15.120:0/2470795026 >> v1:172.21.15.62:6789/0 conn(0x7f452c15c990 legacy=0x7f452c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:39.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.183+0000 7f452b400640 1 -- 172.21.15.120:0/2470795026 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c0ce2f0 con 0x7f452c0d0e40 2024-03-23T11:18:39.185 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.184+0000 7f452b400640 1 --2- 172.21.15.120:0/2470795026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c0d0e40 0x7f452c0d12a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f452c08c3a0 tx=0x7f451802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b8a87d7d98f2a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:39.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.186+0000 7f452aa00640 1 -- 172.21.15.120:0/2470795026 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518031870 con 0x7f452c0d0e40 2024-03-23T11:18:39.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.186+0000 7f452aa00640 1 -- 172.21.15.120:0/2470795026 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45180319d0 con 0x7f452c0d0e40 2024-03-23T11:18:39.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.186+0000 7f452aa00640 1 -- 172.21.15.120:0/2470795026 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518031d60 con 0x7f452c0d0e40 2024-03-23T11:18:39.187 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.187+0000 7f4531400640 1 -- 172.21.15.120:0/2470795026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c0d0e40 msgr2=0x7f452c0d12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.187+0000 7f4531400640 1 --2- 172.21.15.120:0/2470795026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c0d0e40 0x7f452c0d12a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f452c08c3a0 tx=0x7f451802b910 comp rx=0 tx=0).stop 2024-03-23T11:18:39.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.187+0000 7f4531400640 1 -- 172.21.15.120:0/2470795026 shutdown_connections 2024-03-23T11:18:39.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.187+0000 7f4531400640 1 --2- 172.21.15.120:0/2470795026 >> v2:172.21.15.62:3300/0 conn(0x7f452c0cefa0 0x7f452c0cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.187+0000 7f4531400640 1 --2- 172.21.15.120:0/2470795026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c0d0e40 0x7f452c0d12a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.188 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.187+0000 7f4531400640 1 -- 172.21.15.120:0/2470795026 >> 172.21.15.120:0/2470795026 conn(0x7f452c089f60 msgr2=0x7f452c0d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:39.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.187+0000 7f4531400640 1 -- 172.21.15.120:0/2470795026 shutdown_connections 2024-03-23T11:18:39.189 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.187+0000 7f4531400640 1 -- 172.21.15.120:0/2470795026 wait complete. 2024-03-23T11:18:39.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.189+0000 7f4531400640 1 Processor -- start 2024-03-23T11:18:39.190 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.189+0000 7f4531400640 1 -- start start 2024-03-23T11:18:39.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.189+0000 7f4531400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c0cefa0 0x7f452c0b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f4531400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f452c0d0e40 0x7f452c15dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f4531400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f452c15c990 0x7f452c1f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f4531400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c0b8200 con 0x7f452c15c990 2024-03-23T11:18:39.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f4531400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c15e4e0 con 0x7f452c0cefa0 2024-03-23T11:18:39.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f4531400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c15e730 con 0x7f452c0d0e40 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f452b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f452c0d0e40 0x7f452c15dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f452be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c0cefa0 0x7f452c0b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f452b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f452c0d0e40 0x7f452c15dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57458/0 (socket says 172.21.15.120:57458) 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f452b400640 1 -- 172.21.15.120:0/306150167 learned_addr learned my addr 172.21.15.120:0/306150167 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f4530a00640 1 --2- 172.21.15.120:0/306150167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f452c15c990 0x7f452c1f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f452b400640 1 -- 172.21.15.120:0/306150167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c0cefa0 msgr2=0x7f452c0b7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f452b400640 1 --2- 172.21.15.120:0/306150167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c0cefa0 0x7f452c0b7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f452b400640 1 -- 172.21.15.120:0/306150167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f452c15c990 msgr2=0x7f452c1f17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f452b400640 1 --2- 172.21.15.120:0/306150167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f452c15c990 0x7f452c1f17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.190+0000 7f452b400640 1 -- 172.21.15.120:0/306150167 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45180094d0 con 0x7f452c0d0e40 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.191+0000 7f452b400640 1 --2- 172.21.15.120:0/306150167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f452c0d0e40 0x7f452c15dfa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4518000c00 tx=0x7f45180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.191+0000 7f4523e00640 1 -- 172.21.15.120:0/306150167 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451803ad30 con 0x7f452c0d0e40 2024-03-23T11:18:39.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.191+0000 7f4523e00640 1 -- 172.21.15.120:0/306150167 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451803ae90 con 0x7f452c0d0e40 2024-03-23T11:18:39.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.191+0000 7f4531400640 1 -- 172.21.15.120:0/306150167 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452c15ea10 con 0x7f452c0d0e40 2024-03-23T11:18:39.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.191+0000 7f4523e00640 1 -- 172.21.15.120:0/306150167 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518044630 con 0x7f452c0d0e40 2024-03-23T11:18:39.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.191+0000 7f4531400640 1 -- 172.21.15.120:0/306150167 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452c15efd0 con 0x7f452c0d0e40 2024-03-23T11:18:39.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.191+0000 7f4523e00640 1 -- 172.21.15.120:0/306150167 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4518044a90 con 0x7f452c0d0e40 2024-03-23T11:18:39.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.191+0000 7f4523e00640 1 -- 172.21.15.120:0/306150167 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f451802d040 con 0x7f452c0d0e40 2024-03-23T11:18:39.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.193+0000 7f4531400640 1 -- 172.21.15.120:0/306150167 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452c0672c0 con 0x7f452c0d0e40 2024-03-23T11:18:39.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.195+0000 7f4523e00640 1 -- 172.21.15.120:0/306150167 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f4518035020 con 0x7f452c0d0e40 2024-03-23T11:18:39.273 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.273+0000 7f4531400640 1 -- 172.21.15.120:0/306150167 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b76f68e1-ec2c-438e-8275-90f2c3df293e", "id": 6} v 0) v1 -- 0x7f452c0cfd70 con 0x7f452c0d0e40 2024-03-23T11:18:39.284 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.283+0000 7f4523e00640 1 -- 172.21.15.120:0/306150167 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b76f68e1-ec2c-438e-8275-90f2c3df293e", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f451804d450 con 0x7f452c0d0e40 2024-03-23T11:18:39.284 INFO:teuthology.orchestra.run.smithi120.stdout:6 2024-03-23T11:18:39.285 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.284+0000 7f4531400640 1 -- 172.21.15.120:0/306150167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f452c0d0e40 msgr2=0x7f452c15dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.285 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.284+0000 7f4531400640 1 --2- 172.21.15.120:0/306150167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f452c0d0e40 0x7f452c15dfa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4518000c00 tx=0x7f45180046c0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.284+0000 7f4531400640 1 -- 172.21.15.120:0/306150167 shutdown_connections 2024-03-23T11:18:39.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.284+0000 7f4531400640 1 --2- 172.21.15.120:0/306150167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f452c15c990 0x7f452c1f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.284+0000 7f4531400640 1 --2- 172.21.15.120:0/306150167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f452c0d0e40 0x7f452c15dfa0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.284+0000 7f4531400640 1 --2- 172.21.15.120:0/306150167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f452c0cefa0 0x7f452c0b7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.284+0000 7f4531400640 1 -- 172.21.15.120:0/306150167 >> 172.21.15.120:0/306150167 conn(0x7f452c089f60 msgr2=0x7f452c0d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:39.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.284+0000 7f4531400640 1 -- 172.21.15.120:0/306150167 shutdown_connections 2024-03-23T11:18:39.286 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.285+0000 7f4531400640 1 -- 172.21.15.120:0/306150167 wait complete. 2024-03-23T11:18:39.305 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:39.304+0000 7fef97c42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T11:18:39.307 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 87fbbab4-1560-4530-8400-f57ca7f08fe6 7 2024-03-23T11:18:39.346 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:39.345+0000 7f50f2e14040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T11:18:39.356 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:39.356+0000 7fef97c42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T11:18:39.397 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:39.397+0000 7f50f2e14040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T11:18:39.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.536+0000 7f352da00640 1 Processor -- start 2024-03-23T11:18:39.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.536+0000 7f352da00640 1 -- start start 2024-03-23T11:18:39.538 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:39.537+0000 7fef97c42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T11:18:39.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.538+0000 7f352da00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f35280ceb50 0x7f35280cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.539 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.538+0000 7f352da00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f35280b0680 0x7f35280b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.538+0000 7f352da00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35280cf4f0 con 0x7f352815a7f0 2024-03-23T11:18:39.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.538+0000 7f352da00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35280cbf70 con 0x7f35280b0680 2024-03-23T11:18:39.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.538+0000 7f352da00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35280cc250 con 0x7f35280ceb50 2024-03-23T11:18:39.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.538+0000 7f351fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f35280ceb50 0x7f35280cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.538+0000 7f351fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f35280ceb50 0x7f35280cefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:34930/0 (socket says 172.21.15.120:34930) 2024-03-23T11:18:39.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.538+0000 7f351fe00640 1 -- 172.21.15.120:0/503482343 learned_addr learned my addr 172.21.15.120:0/503482343 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:39.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.538+0000 7f352d000640 1 --2- 172.21.15.120:0/503482343 >> v2:172.21.15.62:3300/0 conn(0x7f35280b0680 0x7f35280b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.539+0000 7f351fe00640 1 -- 172.21.15.120:0/503482343 >> v2:172.21.15.62:3300/0 conn(0x7f35280b0680 msgr2=0x7f35280b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:39.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.539+0000 7f351fe00640 1 --2- 172.21.15.120:0/503482343 >> v2:172.21.15.62:3300/0 conn(0x7f35280b0680 0x7f35280b0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.539+0000 7f351fe00640 1 -- 172.21.15.120:0/503482343 >> v1:172.21.15.62:6789/0 conn(0x7f352815a7f0 legacy=0x7f35280ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:39.540 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.539+0000 7f351fe00640 1 -- 172.21.15.120:0/503482343 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35280cc530 con 0x7f35280ceb50 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.540+0000 7f351fe00640 1 --2- 172.21.15.120:0/503482343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35280ceb50 0x7f35280cefb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3528071b00 tx=0x7f351002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3522b0681c4cefdd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.540+0000 7f3527400640 1 -- 172.21.15.120:0/503482343 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510030860 con 0x7f35280ceb50 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.540+0000 7f3527400640 1 -- 172.21.15.120:0/503482343 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35100309c0 con 0x7f35280ceb50 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.540+0000 7f3527400640 1 -- 172.21.15.120:0/503482343 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510030d50 con 0x7f35280ceb50 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.541+0000 7f352da00640 1 -- 172.21.15.120:0/503482343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35280ceb50 msgr2=0x7f35280cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.541+0000 7f352da00640 1 --2- 172.21.15.120:0/503482343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35280ceb50 0x7f35280cefb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3528071b00 tx=0x7f351002b9a0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.541+0000 7f352da00640 1 -- 172.21.15.120:0/503482343 shutdown_connections 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.541+0000 7f352da00640 1 --2- 172.21.15.120:0/503482343 >> v2:172.21.15.62:3300/0 conn(0x7f35280b0680 0x7f35280b0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.541+0000 7f352da00640 1 --2- 172.21.15.120:0/503482343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35280ceb50 0x7f35280cefb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.541+0000 7f352da00640 1 -- 172.21.15.120:0/503482343 >> 172.21.15.120:0/503482343 conn(0x7f35280b2060 msgr2=0x7f35280b2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.541+0000 7f352da00640 1 -- 172.21.15.120:0/503482343 shutdown_connections 2024-03-23T11:18:39.542 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.541+0000 7f352da00640 1 -- 172.21.15.120:0/503482343 wait complete. 2024-03-23T11:18:39.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.542+0000 7f352da00640 1 Processor -- start 2024-03-23T11:18:39.543 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.543+0000 7f352da00640 1 -- start start 2024-03-23T11:18:39.546 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.543+0000 7f352da00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35280b0680 0x7f3528156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.546 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.543+0000 7f352da00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35280ceb50 0x7f352815d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.546 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.543+0000 7f352da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352815a7f0 0x7f3528143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.546 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f352da00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352815dea0 con 0x7f352815a7f0 2024-03-23T11:18:39.546 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f352da00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3528157ec0 con 0x7f35280ceb50 2024-03-23T11:18:39.546 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f352da00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3528144420 con 0x7f35280b0680 2024-03-23T11:18:39.546 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f351fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35280ceb50 0x7f352815d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f3527e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35280b0680 0x7f3528156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f351fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35280ceb50 0x7f352815d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:34938/0 (socket says 172.21.15.120:34938) 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f351fe00640 1 -- 172.21.15.120:0/384734683 learned_addr learned my addr 172.21.15.120:0/384734683 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f3527e00640 1 --2- 172.21.15.120:0/384734683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35280b0680 0x7f3528156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:57466/0 (socket says 172.21.15.120:57466) 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f352d000640 1 --2- 172.21.15.120:0/384734683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352815a7f0 0x7f3528143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f351fe00640 1 -- 172.21.15.120:0/384734683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35280b0680 msgr2=0x7f3528156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f351fe00640 1 --2- 172.21.15.120:0/384734683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35280b0680 0x7f3528156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f351fe00640 1 -- 172.21.15.120:0/384734683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352815a7f0 msgr2=0x7f3528143ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f351fe00640 1 --2- 172.21.15.120:0/384734683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352815a7f0 0x7f3528143ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f351fe00640 1 -- 172.21.15.120:0/384734683 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35100094d0 con 0x7f35280ceb50 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.544+0000 7f352d000640 1 --2- 172.21.15.120:0/384734683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352815a7f0 0x7f3528143ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.545+0000 7f351fe00640 1 --2- 172.21.15.120:0/384734683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35280ceb50 0x7f352815d960 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3510000c00 tx=0x7f3510004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:39.547 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.545+0000 7f3524c00640 1 -- 172.21.15.120:0/384734683 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510039d60 con 0x7f35280ceb50 2024-03-23T11:18:39.548 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.545+0000 7f3524c00640 1 -- 172.21.15.120:0/384734683 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3510044480 con 0x7f35280ceb50 2024-03-23T11:18:39.548 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.545+0000 7f352da00640 1 -- 172.21.15.120:0/384734683 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35281572b0 con 0x7f35280ceb50 2024-03-23T11:18:39.548 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.545+0000 7f3524c00640 1 -- 172.21.15.120:0/384734683 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510044830 con 0x7f35280ceb50 2024-03-23T11:18:39.548 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.545+0000 7f352da00640 1 -- 172.21.15.120:0/384734683 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3528157870 con 0x7f35280ceb50 2024-03-23T11:18:39.548 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.545+0000 7f3524c00640 1 -- 172.21.15.120:0/384734683 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3510044c90 con 0x7f35280ceb50 2024-03-23T11:18:39.548 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.545+0000 7f3524c00640 1 -- 172.21.15.120:0/384734683 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f351003e070 con 0x7f35280ceb50 2024-03-23T11:18:39.548 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.546+0000 7f352da00640 1 -- 172.21.15.120:0/384734683 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3528158050 con 0x7f35280ceb50 2024-03-23T11:18:39.549 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.548+0000 7f3524c00640 1 -- 172.21.15.120:0/384734683 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f3510049070 con 0x7f35280ceb50 2024-03-23T11:18:39.578 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:39.577+0000 7f50f2e14040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T11:18:39.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.626+0000 7f352da00640 1 -- 172.21.15.120:0/384734683 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "87fbbab4-1560-4530-8400-f57ca7f08fe6", "id": 7} v 0) v1 -- 0x7f35280672c0 con 0x7f35280ceb50 2024-03-23T11:18:39.637 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.637+0000 7f3524c00640 1 -- 172.21.15.120:0/384734683 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "87fbbab4-1560-4530-8400-f57ca7f08fe6", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3510034070 con 0x7f35280ceb50 2024-03-23T11:18:39.637 INFO:teuthology.orchestra.run.smithi120.stdout:7 2024-03-23T11:18:39.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.638+0000 7f352da00640 1 -- 172.21.15.120:0/384734683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35280ceb50 msgr2=0x7f352815d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.638+0000 7f352da00640 1 --2- 172.21.15.120:0/384734683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35280ceb50 0x7f352815d960 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3510000c00 tx=0x7f3510004730 comp rx=0 tx=0).stop 2024-03-23T11:18:39.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.638+0000 7f352da00640 1 -- 172.21.15.120:0/384734683 shutdown_connections 2024-03-23T11:18:39.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.638+0000 7f352da00640 1 --2- 172.21.15.120:0/384734683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f352815a7f0 0x7f3528143ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.638+0000 7f352da00640 1 --2- 172.21.15.120:0/384734683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35280ceb50 0x7f352815d960 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.638+0000 7f352da00640 1 --2- 172.21.15.120:0/384734683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35280b0680 0x7f3528156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.638+0000 7f352da00640 1 -- 172.21.15.120:0/384734683 >> 172.21.15.120:0/384734683 conn(0x7f35280b2060 msgr2=0x7f35280b5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:39.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.638+0000 7f352da00640 1 -- 172.21.15.120:0/384734683 shutdown_connections 2024-03-23T11:18:39.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-23T11:18:39.638+0000 7f352da00640 1 -- 172.21.15.120:0/384734683 wait complete. 2024-03-23T11:18:39.657 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-23T11:18:39.657 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-23T11:18:39.664 INFO:tasks.ceph.osd.0:Started 2024-03-23T11:18:39.665 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-23T11:18:39.665 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-23T11:18:39.672 INFO:tasks.ceph.osd.1:Started 2024-03-23T11:18:39.672 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-23T11:18:39.672 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-23T11:18:39.679 INFO:tasks.ceph.osd.2:Started 2024-03-23T11:18:39.679 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-23T11:18:39.679 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-23T11:18:39.686 INFO:tasks.ceph.osd.3:Started 2024-03-23T11:18:39.686 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-23T11:18:39.686 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-23T11:18:39.693 INFO:tasks.ceph.osd.4:Started 2024-03-23T11:18:39.693 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-23T11:18:39.693 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-23T11:18:39.700 INFO:tasks.ceph.osd.5:Started 2024-03-23T11:18:39.700 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-23T11:18:39.700 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-23T11:18:39.705 INFO:tasks.ceph.osd.6:Started 2024-03-23T11:18:39.705 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-23T11:18:39.705 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-23T11:18:39.710 INFO:tasks.ceph.osd.7:Started 2024-03-23T11:18:39.711 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:18:39.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.931+0000 7f62a3000640 1 Processor -- start 2024-03-23T11:18:39.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.931+0000 7f62a3000640 1 -- start start 2024-03-23T11:18:39.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.932+0000 7f62a3000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c15fcf0 0x7f629c0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.932+0000 7f62a3000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f629c0d14f0 con 0x7f629c160420 2024-03-23T11:18:39.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.932+0000 7f62a3000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f629c0d3540 con 0x7f629c0b6dc0 2024-03-23T11:18:39.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.932+0000 7f62a3000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629c0ce510 con 0x7f629c15fcf0 2024-03-23T11:18:39.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.932+0000 7f62a1200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f629c160420 0x7f629c0d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:46646/0 (socket says 172.21.15.62:46646) 2024-03-23T11:18:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.932+0000 7f62a1200640 1 -- 172.21.15.62:0/1432951847 learned_addr learned my addr 172.21.15.62:0/1432951847 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:18:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.932+0000 7f62a1c00640 1 --2- 172.21.15.62:0/1432951847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c15fcf0 0x7f629c0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.933+0000 7f629be00640 1 -- 172.21.15.62:0/1432951847 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923798409 0 0) 0x7f629c0d14f0 con 0x7f629c160420 2024-03-23T11:18:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.933+0000 7f629be00640 1 -- 172.21.15.62:0/1432951847 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6284003520 con 0x7f629c160420 2024-03-23T11:18:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.933+0000 7f629be00640 1 -- 172.21.15.62:0/1432951847 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348759108 0 0) 0x7f629c0d3540 con 0x7f629c0b6dc0 2024-03-23T11:18:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.933+0000 7f62a1c00640 1 -- 172.21.15.62:0/1432951847 >> v1:172.21.15.120:6789/0 conn(0x7f629c0b6dc0 legacy=0x7f629c0d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.933+0000 7f62a1c00640 1 -- 172.21.15.62:0/1432951847 >> v1:172.21.15.62:6789/0 conn(0x7f629c160420 legacy=0x7f629c0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.933+0000 7f62a1c00640 1 -- 172.21.15.62:0/1432951847 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629c0cdc70 con 0x7f629c15fcf0 2024-03-23T11:18:39.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.934+0000 7f62a1c00640 1 --2- 172.21.15.62:0/1432951847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c15fcf0 0x7f629c0d3000 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f62900097d0 tx=0x7f629002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef23efd2041f9023 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:39.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.934+0000 7f629be00640 1 -- 172.21.15.62:0/1432951847 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290031770 con 0x7f629c15fcf0 2024-03-23T11:18:39.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.934+0000 7f629be00640 1 -- 172.21.15.62:0/1432951847 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62900318d0 con 0x7f629c15fcf0 2024-03-23T11:18:39.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.934+0000 7f629be00640 1 -- 172.21.15.62:0/1432951847 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290031c60 con 0x7f629c15fcf0 2024-03-23T11:18:39.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.935+0000 7f62a3000640 1 -- 172.21.15.62:0/1432951847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c15fcf0 msgr2=0x7f629c0d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.935+0000 7f62a3000640 1 --2- 172.21.15.62:0/1432951847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c15fcf0 0x7f629c0d3000 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f62900097d0 tx=0x7f629002b810 comp rx=0 tx=0).stop 2024-03-23T11:18:39.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.935+0000 7f62a3000640 1 -- 172.21.15.62:0/1432951847 shutdown_connections 2024-03-23T11:18:39.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.935+0000 7f62a3000640 1 --2- 172.21.15.62:0/1432951847 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c15fcf0 0x7f629c0d3000 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.935+0000 7f62a3000640 1 -- 172.21.15.62:0/1432951847 >> 172.21.15.62:0/1432951847 conn(0x7f629c089f60 msgr2=0x7f629c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:39.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.935+0000 7f62a3000640 1 -- 172.21.15.62:0/1432951847 shutdown_connections 2024-03-23T11:18:39.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.935+0000 7f62a3000640 1 -- 172.21.15.62:0/1432951847 wait complete. 2024-03-23T11:18:39.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.937+0000 7f62a3000640 1 Processor -- start 2024-03-23T11:18:39.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.937+0000 7f62a3000640 1 -- start start 2024-03-23T11:18:39.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.938+0000 7f62a3000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f629c0b6dc0 0x7f629c159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.938+0000 7f62a3000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f629c15fcf0 0x7f629c09fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.938+0000 7f62a3000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c160420 0x7f629c09c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:39.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.938+0000 7f62a3000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629c1f2120 con 0x7f629c15fcf0 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.938+0000 7f62a3000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629c09d740 con 0x7f629c0b6dc0 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.938+0000 7f62a3000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629c09da20 con 0x7f629c160420 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.938+0000 7f62a1c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f629c0b6dc0 0x7f629c159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.938+0000 7f62a1c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f629c0b6dc0 0x7f629c159570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:48158/0 (socket says 172.21.15.62:48158) 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.938+0000 7f62a1c00640 1 -- 172.21.15.62:0/1044428112 learned_addr learned my addr 172.21.15.62:0/1044428112 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.938+0000 7f62a1200640 1 --2- 172.21.15.62:0/1044428112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f629c15fcf0 0x7f629c09fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.938+0000 7f62a2600640 1 --2- 172.21.15.62:0/1044428112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c160420 0x7f629c09c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f62a1c00640 1 -- 172.21.15.62:0/1044428112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c160420 msgr2=0x7f629c09c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f62a1c00640 1 --2- 172.21.15.62:0/1044428112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c160420 0x7f629c09c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f62a1c00640 1 -- 172.21.15.62:0/1044428112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f629c15fcf0 msgr2=0x7f629c09fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f62a1c00640 1 --2- 172.21.15.62:0/1044428112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f629c15fcf0 0x7f629c09fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f62a1c00640 1 -- 172.21.15.62:0/1044428112 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6290009400 con 0x7f629c0b6dc0 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f62a1200640 1 --2- 172.21.15.62:0/1044428112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f629c15fcf0 0x7f629c09fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:18:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f62a2600640 1 --2- 172.21.15.62:0/1044428112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c160420 0x7f629c09c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:18:39.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f62a1c00640 1 --2- 172.21.15.62:0/1044428112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f629c0b6dc0 0x7f629c159570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f62900097d0 tx=0x7f62900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:39.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f6299600640 1 -- 172.21.15.62:0/1044428112 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290004220 con 0x7f629c0b6dc0 2024-03-23T11:18:39.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f6299600640 1 -- 172.21.15.62:0/1044428112 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62900043a0 con 0x7f629c0b6dc0 2024-03-23T11:18:39.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f6299600640 1 -- 172.21.15.62:0/1044428112 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290003850 con 0x7f629c0b6dc0 2024-03-23T11:18:39.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f62a3000640 1 -- 172.21.15.62:0/1044428112 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629c1f23b0 con 0x7f629c0b6dc0 2024-03-23T11:18:39.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.939+0000 7f62a3000640 1 -- 172.21.15.62:0/1044428112 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629c09dd00 con 0x7f629c0b6dc0 2024-03-23T11:18:39.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.940+0000 7f6299600640 1 -- 172.21.15.62:0/1044428112 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6290043070 con 0x7f629c0b6dc0 2024-03-23T11:18:39.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.940+0000 7f6299600640 1 -- 172.21.15.62:0/1044428112 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6290045af0 con 0x7f629c0b6dc0 2024-03-23T11:18:39.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.940+0000 7f62a3000640 1 -- 172.21.15.62:0/1044428112 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629c066bb0 con 0x7f629c0b6dc0 2024-03-23T11:18:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:39.943+0000 7f6299600640 1 -- 172.21.15.62:0/1044428112 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f6290031eb0 con 0x7f629c0b6dc0 2024-03-23T11:18:40.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:40.026+0000 7f62a3000640 1 -- 172.21.15.62:0/1044428112 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f629c0cea40 con 0x7f629c0b6dc0 2024-03-23T11:18:40.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:40.027+0000 7f6299600640 1 -- 172.21.15.62:0/1044428112 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f6290003e40 con 0x7f629c0b6dc0 2024-03-23T11:18:40.028 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:18:40.028 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":10,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:39.629234+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:18:40.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:40.029+0000 7f62a3000640 1 -- 172.21.15.62:0/1044428112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f629c0b6dc0 msgr2=0x7f629c159570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:40.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:40.029+0000 7f62a3000640 1 --2- 172.21.15.62:0/1044428112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f629c0b6dc0 0x7f629c159570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f62900097d0 tx=0x7f62900046c0 comp rx=0 tx=0).stop 2024-03-23T11:18:40.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:40.029+0000 7f62a3000640 1 -- 172.21.15.62:0/1044428112 shutdown_connections 2024-03-23T11:18:40.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:40.029+0000 7f62a3000640 1 --2- 172.21.15.62:0/1044428112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f629c15fcf0 0x7f629c09fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:40.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:40.029+0000 7f62a3000640 1 --2- 172.21.15.62:0/1044428112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f629c160420 0x7f629c09c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:40.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:40.029+0000 7f62a3000640 1 --2- 172.21.15.62:0/1044428112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f629c0b6dc0 0x7f629c159570 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:40.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:40.029+0000 7f62a3000640 1 -- 172.21.15.62:0/1044428112 >> 172.21.15.62:0/1044428112 conn(0x7f629c089f60 msgr2=0x7f629c0cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:40.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:40.029+0000 7f62a3000640 1 -- 172.21.15.62:0/1044428112 shutdown_connections 2024-03-23T11:18:40.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:40.029+0000 7f62a3000640 1 -- 172.21.15.62:0/1044428112 wait complete. 2024-03-23T11:18:40.045 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-23T11:18:40.045 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-23T11:18:40.381 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-03-23T11:18:40.381+0000 7f61a4616500 -1 Falling back to public interface 2024-03-23T11:18:40.388 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-23T11:18:40.388+0000 7f49e2605500 -1 Falling back to public interface 2024-03-23T11:18:40.389 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-23T11:18:40.388+0000 7f541d00f500 -1 Falling back to public interface 2024-03-23T11:18:40.406 INFO:tasks.ceph.osd.3.smithi062.stderr:2024-03-23T11:18:40.406+0000 7fc668ca5500 -1 Falling back to public interface 2024-03-23T11:18:40.408 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-03-23T11:18:40.408+0000 7f86df5e0500 -1 Falling back to public interface 2024-03-23T11:18:40.408 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-23T11:18:40.408+0000 7f85a3cc9500 -1 Falling back to public interface 2024-03-23T11:18:40.417 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-23T11:18:40.416+0000 7f92d261f500 -1 Falling back to public interface 2024-03-23T11:18:40.418 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-03-23T11:18:40.418+0000 7fd38f638500 -1 Falling back to public interface 2024-03-23T11:18:40.694 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:40.693+0000 7f50f2e14040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T11:18:40.733 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:40.733+0000 7fef97c42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T11:18:40.850 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-23T11:18:41.010 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.010+0000 7fda9e000640 1 Processor -- start 2024-03-23T11:18:41.010 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.010+0000 7fda9e000640 1 -- start start 2024-03-23T11:18:41.012 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.011+0000 7fda9e000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fda9815cb90 0x7fda980d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:41.012 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.011+0000 7fda9e000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fda980b8580 0x7fda980cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:41.012 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.011+0000 7fda9e000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda980d22a0 0x7fda980cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:41.012 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.011+0000 7fda9e000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9815cf70 con 0x7fda980b8580 2024-03-23T11:18:41.012 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.011+0000 7fda9e000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda980d1200 con 0x7fda9815cb90 2024-03-23T11:18:41.012 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.011+0000 7fda9e000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda980d14e0 con 0x7fda980d22a0 2024-03-23T11:18:41.012 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.011+0000 7fda97400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fda9815cb90 0x7fda980d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:41.013 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.011+0000 7fda97400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fda9815cb90 0x7fda980d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:48170/0 (socket says 172.21.15.62:48170) 2024-03-23T11:18:41.013 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.011+0000 7fda97400640 1 -- 172.21.15.62:0/2369519251 learned_addr learned my addr 172.21.15.62:0/2369519251 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:41.013 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.011+0000 7fda97e00640 1 --2- 172.21.15.62:0/2369519251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda980d22a0 0x7fda980cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:41.013 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.012+0000 7fda96a00640 1 --2- 172.21.15.62:0/2369519251 >> v2:172.21.15.62:3300/0 conn(0x7fda980b8580 0x7fda980cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:41.013 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.012+0000 7fda97400640 1 -- 172.21.15.62:0/2369519251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda980d22a0 msgr2=0x7fda980cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:41.013 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.012+0000 7fda97400640 1 --2- 172.21.15.62:0/2369519251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda980d22a0 0x7fda980cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:41.013 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.012+0000 7fda97400640 1 -- 172.21.15.62:0/2369519251 >> v2:172.21.15.62:3300/0 conn(0x7fda980b8580 msgr2=0x7fda980cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:41.013 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.012+0000 7fda97400640 1 --2- 172.21.15.62:0/2369519251 >> v2:172.21.15.62:3300/0 conn(0x7fda980b8580 0x7fda980cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:41.013 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.012+0000 7fda97400640 1 -- 172.21.15.62:0/2369519251 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda980d17c0 con 0x7fda9815cb90 2024-03-23T11:18:41.013 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.012+0000 7fda96a00640 1 --2- 172.21.15.62:0/2369519251 >> v2:172.21.15.62:3300/0 conn(0x7fda980b8580 0x7fda980cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:18:41.013 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.012+0000 7fda97e00640 1 --2- 172.21.15.62:0/2369519251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda980d22a0 0x7fda980cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:18:41.014 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:41.013+0000 7f50f2e14040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T11:18:41.015 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.013+0000 7fda97400640 1 --2- 172.21.15.62:0/2369519251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda9815cb90 0x7fda980d08b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fda980b5420 tx=0x7fda8802b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6a8cf822efe8bc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:41.015 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.013+0000 7fda96000640 1 -- 172.21.15.62:0/2369519251 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88031880 con 0x7fda9815cb90 2024-03-23T11:18:41.015 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.013+0000 7fda96000640 1 -- 172.21.15.62:0/2369519251 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda880319e0 con 0x7fda9815cb90 2024-03-23T11:18:41.015 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.013+0000 7fda96000640 1 -- 172.21.15.62:0/2369519251 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88031d70 con 0x7fda9815cb90 2024-03-23T11:18:41.015 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.014+0000 7fda9e000640 1 -- 172.21.15.62:0/2369519251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda9815cb90 msgr2=0x7fda980d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:41.015 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.014+0000 7fda9e000640 1 --2- 172.21.15.62:0/2369519251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda9815cb90 0x7fda980d08b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fda980b5420 tx=0x7fda8802b920 comp rx=0 tx=0).stop 2024-03-23T11:18:41.016 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.014+0000 7fda9e000640 1 -- 172.21.15.62:0/2369519251 shutdown_connections 2024-03-23T11:18:41.016 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.014+0000 7fda9e000640 1 --2- 172.21.15.62:0/2369519251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda980d22a0 0x7fda980cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:41.016 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.014+0000 7fda9e000640 1 --2- 172.21.15.62:0/2369519251 >> v2:172.21.15.62:3300/0 conn(0x7fda980b8580 0x7fda980cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:41.016 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.014+0000 7fda9e000640 1 --2- 172.21.15.62:0/2369519251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda9815cb90 0x7fda980d08b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:41.016 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.014+0000 7fda9e000640 1 -- 172.21.15.62:0/2369519251 >> 172.21.15.62:0/2369519251 conn(0x7fda98071830 msgr2=0x7fda98071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:41.016 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.014+0000 7fda9e000640 1 -- 172.21.15.62:0/2369519251 shutdown_connections 2024-03-23T11:18:41.016 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.014+0000 7fda9e000640 1 -- 172.21.15.62:0/2369519251 wait complete. 2024-03-23T11:18:41.018 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.016+0000 7fda9e000640 1 Processor -- start 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.016+0000 7fda9e000640 1 -- start start 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda9e000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda980b8580 0x7fda980c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda9e000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda980d22a0 0x7fda9813ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda9e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9815cb90 0x7fda9813cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda9e000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980c9230 con 0x7fda9815cb90 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda9e000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980ca150 con 0x7fda980d22a0 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda9e000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda980cb880 con 0x7fda980b8580 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda97e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9815cb90 0x7fda9813cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda97e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9815cb90 0x7fda9813cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49020/0 (socket says 172.21.15.62:49020) 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda97e00640 1 -- 172.21.15.62:0/1643059023 learned_addr learned my addr 172.21.15.62:0/1643059023 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda96a00640 1 --2- 172.21.15.62:0/1643059023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda980d22a0 0x7fda9813ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:41.019 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda97400640 1 --2- 172.21.15.62:0/1643059023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda980b8580 0x7fda980c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda97e00640 1 -- 172.21.15.62:0/1643059023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda980b8580 msgr2=0x7fda980c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda97e00640 1 --2- 172.21.15.62:0/1643059023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda980b8580 0x7fda980c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda97e00640 1 -- 172.21.15.62:0/1643059023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda980d22a0 msgr2=0x7fda9813ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda97e00640 1 --2- 172.21.15.62:0/1643059023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda980d22a0 0x7fda9813ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.017+0000 7fda97e00640 1 -- 172.21.15.62:0/1643059023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda90009a70 con 0x7fda9815cb90 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.018+0000 7fda97e00640 1 --2- 172.21.15.62:0/1643059023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9815cb90 0x7fda9813cf90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fda90009f00 tx=0x7fda9000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.018+0000 7fda8f400640 1 -- 172.21.15.62:0/1643059023 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90009d70 con 0x7fda9815cb90 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.018+0000 7fda8f400640 1 -- 172.21.15.62:0/1643059023 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9000f040 con 0x7fda9815cb90 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.018+0000 7fda8f400640 1 -- 172.21.15.62:0/1643059023 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90003bb0 con 0x7fda9815cb90 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.018+0000 7fda9e000640 1 -- 172.21.15.62:0/1643059023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda880094d0 con 0x7fda9815cb90 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.018+0000 7fda9e000640 1 -- 172.21.15.62:0/1643059023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda980cbe40 con 0x7fda9815cb90 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.018+0000 7fda8f400640 1 -- 172.21.15.62:0/1643059023 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fda90014780 con 0x7fda9815cb90 2024-03-23T11:18:41.020 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.018+0000 7fda8f400640 1 -- 172.21.15.62:0/1643059023 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fda9001d410 con 0x7fda9815cb90 2024-03-23T11:18:41.021 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.019+0000 7fda9e000640 1 -- 172.21.15.62:0/1643059023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda98066bb0 con 0x7fda9815cb90 2024-03-23T11:18:41.022 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.022+0000 7fda8f400640 1 -- 172.21.15.62:0/1643059023 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fda90018030 con 0x7fda9815cb90 2024-03-23T11:18:41.056 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:41.056+0000 7fef97c42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T11:18:41.074 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:41.073+0000 7f50f2e14040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T11:18:41.089 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.089+0000 7fda9e000640 1 -- 172.21.15.62:0/1643059023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda9815a080 con 0x7fda9815cb90 2024-03-23T11:18:41.090 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.090+0000 7fda8f400640 1 -- 172.21.15.62:0/1643059023 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fda90014a30 con 0x7fda9815cb90 2024-03-23T11:18:41.090 INFO:teuthology.misc.health.smithi062.stdout: 2024-03-23T11:18:41.090 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":10,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:39.629234+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:18:41.092 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.091+0000 7fda9e000640 1 -- 172.21.15.62:0/1643059023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9815cb90 msgr2=0x7fda9813cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:41.092 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.091+0000 7fda9e000640 1 --2- 172.21.15.62:0/1643059023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9815cb90 0x7fda9813cf90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fda90009f00 tx=0x7fda9000c490 comp rx=0 tx=0).stop 2024-03-23T11:18:41.092 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.091+0000 7fda9e000640 1 -- 172.21.15.62:0/1643059023 shutdown_connections 2024-03-23T11:18:41.092 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.091+0000 7fda9e000640 1 --2- 172.21.15.62:0/1643059023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda9815cb90 0x7fda9813cf90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:41.092 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.091+0000 7fda9e000640 1 --2- 172.21.15.62:0/1643059023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda980d22a0 0x7fda9813ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:41.092 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.091+0000 7fda9e000640 1 --2- 172.21.15.62:0/1643059023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda980b8580 0x7fda980c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:41.092 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.091+0000 7fda9e000640 1 -- 172.21.15.62:0/1643059023 >> 172.21.15.62:0/1643059023 conn(0x7fda98071830 msgr2=0x7fda9806b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:41.092 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.091+0000 7fda9e000640 1 -- 172.21.15.62:0/1643059023 shutdown_connections 2024-03-23T11:18:41.092 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:41.092+0000 7fda9e000640 1 -- 172.21.15.62:0/1643059023 wait complete. 2024-03-23T11:18:41.104 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-23T11:18:41.119 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:41.119+0000 7fef97c42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T11:18:41.204 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:41.203+0000 7f50f2e14040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T11:18:41.260 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-03-23T11:18:41.260+0000 7fef97c42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T11:18:42.252 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:42.251+0000 7f5093600640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-23T11:18:42.258 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-23T11:18:42.255+0000 7f5093600640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-23T11:18:43.950 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-23T11:18:43.949+0000 7f49e2605500 -1 osd.6 0 log_to_monitors true 2024-03-23T11:18:43.979 INFO:tasks.ceph.osd.3.smithi062.stderr:2024-03-23T11:18:43.978+0000 7fc668ca5500 -1 osd.3 0 log_to_monitors true 2024-03-23T11:18:43.999 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-23T11:18:43.999+0000 7f92d261f500 -1 osd.5 0 log_to_monitors true 2024-03-23T11:18:44.174 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-03-23T11:18:44.174+0000 7f61a4616500 -1 osd.2 0 log_to_monitors true 2024-03-23T11:18:44.183 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-23T11:18:44.182+0000 7f541d00f500 -1 osd.4 0 log_to_monitors true 2024-03-23T11:18:44.191 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:18:44.196 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:18:44.204 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-03-23T11:18:44.204+0000 7f86df5e0500 -1 osd.0 0 log_to_monitors true 2024-03-23T11:18:44.248 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-03-23T11:18:44.248+0000 7fd38f638500 -1 osd.1 0 log_to_monitors true 2024-03-23T11:18:44.263 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-23T11:18:44.262+0000 7f85a3cc9500 -1 osd.7 0 log_to_monitors true 2024-03-23T11:18:47.209 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-03-23T11:18:47.209+0000 7f86d9a00640 -1 osd.0 0 waiting for initial osdmap 2024-03-23T11:18:47.218 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-03-23T11:18:47.216+0000 7f86d0400640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:18:47.226 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-23T11:18:47.226+0000 7f92ce200640 -1 osd.5 0 waiting for initial osdmap 2024-03-23T11:18:47.233 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-23T11:18:47.232+0000 7f92c2e00640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:18:47.267 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-23T11:18:47.267+0000 7f5417000640 -1 osd.4 0 waiting for initial osdmap 2024-03-23T11:18:47.279 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-23T11:18:47.278+0000 7f540da00640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:18:47.285 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-23T11:18:47.285+0000 7f49dca00640 -1 osd.6 0 waiting for initial osdmap 2024-03-23T11:18:47.299 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-23T11:18:47.298+0000 7f49d3400640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:18:47.334 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-03-23T11:18:47.334+0000 7f619e400640 -1 osd.2 0 waiting for initial osdmap 2024-03-23T11:18:47.339 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-03-23T11:18:47.339+0000 7f6194e00640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:18:47.397 INFO:tasks.ceph.osd.3.smithi062.stderr:2024-03-23T11:18:47.397+0000 7fc663200640 -1 osd.3 0 waiting for initial osdmap 2024-03-23T11:18:47.410 INFO:tasks.ceph.osd.3.smithi062.stderr:2024-03-23T11:18:47.410+0000 7fc659c00640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:18:47.690 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-23T11:18:47.689+0000 7f85a0000640 -1 osd.7 0 waiting for initial osdmap 2024-03-23T11:18:47.699 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-23T11:18:47.699+0000 7f8594c00640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:18:47.773 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-03-23T11:18:47.773+0000 7fd389a00640 -1 osd.1 0 waiting for initial osdmap 2024-03-23T11:18:47.785 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-03-23T11:18:47.785+0000 7fd380400640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T11:18:47.909 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-23T11:18:48.095 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.095+0000 7f0abf200640 1 Processor -- start 2024-03-23T11:18:48.095 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.095+0000 7f0abf200640 1 -- start start 2024-03-23T11:18:48.097 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.097+0000 7f0abf200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0ac00ced30 0x7f0ac00ab240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:48.097 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.097+0000 7f0abf200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac015e910 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.097 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.097+0000 7f0abf200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac015ebf0 con 0x7f0ac015a550 2024-03-23T11:18:48.097 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.097+0000 7f0abf200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac015eed0 con 0x7f0ac00ced30 2024-03-23T11:18:48.097 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.097+0000 7f0ab7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0ac00ced30 0x7f0ac00ab240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:48.097 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.097+0000 7f0abe800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0ac00ccbb0 0x7f0ac00715c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:38164/0 (socket says 172.21.15.62:38164) 2024-03-23T11:18:48.098 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.097+0000 7f0abe800640 1 -- 172.21.15.62:0/671588767 learned_addr learned my addr 172.21.15.62:0/671588767 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:18:48.098 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.098+0000 7f0ab7e00640 1 -- 172.21.15.62:0/671588767 >> v1:172.21.15.120:6789/0 conn(0x7f0ac015a550 legacy=0x7f0ac00ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:48.098 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.098+0000 7f0ab7e00640 1 -- 172.21.15.62:0/671588767 >> v1:172.21.15.62:6789/0 conn(0x7f0ac00ccbb0 legacy=0x7f0ac00715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:48.098 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.098+0000 7f0abd400640 1 -- 172.21.15.62:0/671588767 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378723800 0 0) 0x7f0ac015e910 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.098 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.098+0000 7f0ab7e00640 1 -- 172.21.15.62:0/671588767 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac0070dc0 con 0x7f0ac00ced30 2024-03-23T11:18:48.100 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.099+0000 7f0ab7e00640 1 --2- 172.21.15.62:0/671588767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ac00ced30 0x7f0ac00ab240 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80097a0 tx=0x7f0aa802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea084241604e6da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:48.100 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.099+0000 7f0abd400640 1 -- 172.21.15.62:0/671588767 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8031830 con 0x7f0ac00ced30 2024-03-23T11:18:48.100 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.099+0000 7f0abd400640 1 -- 172.21.15.62:0/671588767 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa8031990 con 0x7f0ac00ced30 2024-03-23T11:18:48.100 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.099+0000 7f0abd400640 1 -- 172.21.15.62:0/671588767 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8031d20 con 0x7f0ac00ced30 2024-03-23T11:18:48.100 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.100+0000 7f0abf200640 1 -- 172.21.15.62:0/671588767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ac00ced30 msgr2=0x7f0ac00ab240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:48.100 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.100+0000 7f0abf200640 1 --2- 172.21.15.62:0/671588767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ac00ced30 0x7f0ac00ab240 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80097a0 tx=0x7f0aa802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.100 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.100+0000 7f0abf200640 1 -- 172.21.15.62:0/671588767 shutdown_connections 2024-03-23T11:18:48.100 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.100+0000 7f0abf200640 1 --2- 172.21.15.62:0/671588767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ac00ced30 0x7f0ac00ab240 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.100 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.100+0000 7f0abf200640 1 -- 172.21.15.62:0/671588767 >> 172.21.15.62:0/671588767 conn(0x7f0ac006a100 msgr2=0x7f0ac006a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:48.101 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.100+0000 7f0abf200640 1 -- 172.21.15.62:0/671588767 shutdown_connections 2024-03-23T11:18:48.101 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.100+0000 7f0abf200640 1 -- 172.21.15.62:0/671588767 wait complete. 2024-03-23T11:18:48.101 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.101+0000 7f0abf200640 1 Processor -- start 2024-03-23T11:18:48.102 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.102+0000 7f0abf200640 1 -- start start 2024-03-23T11:18:48.103 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.102+0000 7f0abf200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ac00ccbb0 0x7f0ac014ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abf200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ac00ced30 0x7f0ac0159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abf200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ac015a550 0x7f0ac01519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abf200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0158df0 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abf200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0159040 con 0x7f0ac015a550 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abf200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0159320 con 0x7f0ac00ced30 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ac00ccbb0 0x7f0ac014ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ac00ccbb0 0x7f0ac014ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52532/0 (socket says 172.21.15.62:52532) 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abde00640 1 -- 172.21.15.62:0/199985686 learned_addr learned my addr 172.21.15.62:0/199985686 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abe800640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ac015a550 0x7f0ac01519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abde00640 1 -- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ac00ced30 msgr2=0x7f0ac0159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abde00640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ac00ced30 0x7f0ac0159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abde00640 1 -- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ac015a550 msgr2=0x7f0ac01519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abde00640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ac015a550 0x7f0ac01519e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.103+0000 7f0abde00640 1 -- 172.21.15.62:0/199985686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa8009400 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.104 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.104+0000 7f0abde00640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ac00ccbb0 0x7f0ac014ff60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000c400 tx=0x7f0ab000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:48.105 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.104+0000 7f0ab6000640 1 -- 172.21.15.62:0/199985686 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00052d0 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.105 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.104+0000 7f0ab6000640 1 -- 172.21.15.62:0/199985686 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab0005430 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.105 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.104+0000 7f0ab6000640 1 -- 172.21.15.62:0/199985686 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0003830 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.105 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.104+0000 7f0abf200640 1 -- 172.21.15.62:0/199985686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac01511e0 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.105 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.104+0000 7f0abf200640 1 -- 172.21.15.62:0/199985686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac01504a0 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.106 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.105+0000 7f0abf200640 1 -- 172.21.15.62:0/199985686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac0151040 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.106 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.106+0000 7f0ab6000640 1 -- 172.21.15.62:0/199985686 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 88763+0+0 (secure 0 0 0) 0x7f0ab0003990 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.107 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.107+0000 7f0ab6000640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0a9c06a710 0x7f0a9c06cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:48.107 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.107+0000 7f0ab6000640 1 -- 172.21.15.62:0/199985686 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f0ab0087510 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.108 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.108+0000 7f0ab7e00640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0a9c06a710 0x7f0a9c06cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:48.109 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.109+0000 7f0ab7e00640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0a9c06a710 0x7f0a9c06cbd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8036000 tx=0x7f0aa8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:48.112 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.112+0000 7f0ab6000640 1 -- 172.21.15.62:0/199985686 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0ac0151040 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.328 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.327+0000 7f0abf200640 1 -- 172.21.15.62:0/199985686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ac0151700 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.329 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.329+0000 7f0ab6000640 1 -- 172.21.15.62:0/199985686 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10440 (secure 0 0 0) 0x7f0ac0151700 con 0x7f0ac00ccbb0 2024-03-23T11:18:48.330 INFO:teuthology.misc.health.smithi062.stdout: 2024-03-23T11:18:48.330 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":14,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:48.216045+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:18:48.333 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.332+0000 7f0abf200640 1 -- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0a9c06a710 msgr2=0x7f0a9c06cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:48.333 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.332+0000 7f0abf200640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0a9c06a710 0x7f0a9c06cbd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8036000 tx=0x7f0aa8030000 comp rx=0 tx=0).stop 2024-03-23T11:18:48.333 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.332+0000 7f0abf200640 1 -- 172.21.15.62:0/199985686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ac00ccbb0 msgr2=0x7f0ac014ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:48.333 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.332+0000 7f0abf200640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ac00ccbb0 0x7f0ac014ff60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000c400 tx=0x7f0ab000c8c0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.333 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.332+0000 7f0abf200640 1 -- 172.21.15.62:0/199985686 shutdown_connections 2024-03-23T11:18:48.333 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.332+0000 7f0abf200640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0a9c06a710 0x7f0a9c06cbd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.333 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.332+0000 7f0abf200640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ac015a550 0x7f0ac01519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.333 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.332+0000 7f0abf200640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ac00ced30 0x7f0ac0159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.333 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.332+0000 7f0abf200640 1 --2- 172.21.15.62:0/199985686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ac00ccbb0 0x7f0ac014ff60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.333 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.333+0000 7f0abf200640 1 -- 172.21.15.62:0/199985686 >> 172.21.15.62:0/199985686 conn(0x7f0ac006a100 msgr2=0x7f0ac0156f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:48.334 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.333+0000 7f0abf200640 1 -- 172.21.15.62:0/199985686 shutdown_connections 2024-03-23T11:18:48.334 INFO:teuthology.misc.health.smithi062.stderr:2024-03-23T11:18:48.333+0000 7f0abf200640 1 -- 172.21.15.62:0/199985686 wait complete. 2024-03-23T11:18:48.347 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-23T11:18:48.347 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-23T11:18:48.347 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T11:18:48.347 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T11:18:48.357 INFO:tasks.ceph.mds.a:Started 2024-03-23T11:18:48.357 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T11:18:48.357 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T11:18:48.364 INFO:tasks.ceph.mds.c:Started 2024-03-23T11:18:48.365 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T11:18:48.365 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T11:18:48.372 INFO:tasks.ceph.mds.b:Started 2024-03-23T11:18:48.372 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T11:18:48.372 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T11:18:48.379 INFO:tasks.ceph.mds.d:Started 2024-03-23T11:18:48.379 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-23T11:18:48.380 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:18:48.507 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:18:48.510 INFO:tasks.ceph.mds.c.smithi062.stdout:starting mds.c at 2024-03-23T11:18:48.533 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-23T11:18:48.534 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-23T11:18:48.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.561+0000 7f2df1400640 1 Processor -- start 2024-03-23T11:18:48.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.562+0000 7f2df1400640 1 -- start start 2024-03-23T11:18:48.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2df1400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2dec15cdc0 0x7f2dec0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:48.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2df1400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2dec0cf650 0x7f2dec0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:48.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2df1400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec06bfc0 con 0x7f2dec15c9e0 2024-03-23T11:18:48.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2df1400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dec0d0000 con 0x7f2dec0cf650 2024-03-23T11:18:48.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2df1400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dec0ce140 con 0x7f2dec15cdc0 2024-03-23T11:18:48.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2deb200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2dec0cf650 0x7f2dec0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:48.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2deb200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2dec0cf650 0x7f2dec0cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52562/0 (socket says 172.21.15.62:52562) 2024-03-23T11:18:48.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2deb200640 1 -- 172.21.15.62:0/1893989018 learned_addr learned my addr 172.21.15.62:0/1893989018 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:48.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2de9e00640 1 --2- 172.21.15.62:0/1893989018 >> v2:172.21.15.120:3300/0 conn(0x7f2dec15cdc0 0x7f2dec0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:48.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2deb200640 1 -- 172.21.15.62:0/1893989018 >> v2:172.21.15.120:3300/0 conn(0x7f2dec15cdc0 msgr2=0x7f2dec0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:48.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2deb200640 1 --2- 172.21.15.62:0/1893989018 >> v2:172.21.15.120:3300/0 conn(0x7f2dec15cdc0 0x7f2dec0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2deb200640 1 -- 172.21.15.62:0/1893989018 >> v1:172.21.15.120:6789/0 conn(0x7f2dec15c9e0 legacy=0x7f2dec1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:48.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.563+0000 7f2deb200640 1 -- 172.21.15.62:0/1893989018 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec0b5260 con 0x7f2dec0cf650 2024-03-23T11:18:48.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.564+0000 7f2deb200640 1 --2- 172.21.15.62:0/1893989018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec0cf650 0x7f2dec0cfac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0b89a0 tx=0x7f2de402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e390546a22ee6033 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:48.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.565+0000 7f2de9400640 1 -- 172.21.15.62:0/1893989018 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de402ca80 con 0x7f2dec0cf650 2024-03-23T11:18:48.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.565+0000 7f2de9400640 1 -- 172.21.15.62:0/1893989018 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de402cbe0 con 0x7f2dec0cf650 2024-03-23T11:18:48.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.565+0000 7f2de9400640 1 -- 172.21.15.62:0/1893989018 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de40343f0 con 0x7f2dec0cf650 2024-03-23T11:18:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.565+0000 7f2df1400640 1 -- 172.21.15.62:0/1893989018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec0cf650 msgr2=0x7f2dec0cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.565+0000 7f2df1400640 1 --2- 172.21.15.62:0/1893989018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec0cf650 0x7f2dec0cfac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0b89a0 tx=0x7f2de402bee0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.565+0000 7f2df1400640 1 -- 172.21.15.62:0/1893989018 shutdown_connections 2024-03-23T11:18:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.565+0000 7f2df1400640 1 --2- 172.21.15.62:0/1893989018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec0cf650 0x7f2dec0cfac0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.565+0000 7f2df1400640 1 --2- 172.21.15.62:0/1893989018 >> v2:172.21.15.120:3300/0 conn(0x7f2dec15cdc0 0x7f2dec0d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.565+0000 7f2df1400640 1 -- 172.21.15.62:0/1893989018 >> 172.21.15.62:0/1893989018 conn(0x7f2dec071570 msgr2=0x7f2dec071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.565+0000 7f2df1400640 1 -- 172.21.15.62:0/1893989018 shutdown_connections 2024-03-23T11:18:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.565+0000 7f2df1400640 1 -- 172.21.15.62:0/1893989018 wait complete. 2024-03-23T11:18:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.567+0000 7f2df1400640 1 Processor -- start 2024-03-23T11:18:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.567+0000 7f2df1400640 1 -- start start 2024-03-23T11:18:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.568+0000 7f2df1400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dec0cf650 0x7f2dec151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.568+0000 7f2df1400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dec15c9e0 0x7f2dec15e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.568+0000 7f2df1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec15cdc0 0x7f2dec15ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.568+0000 7f2df1400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec152760 con 0x7f2dec15cdc0 2024-03-23T11:18:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.568+0000 7f2df1400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec1523a0 con 0x7f2dec15c9e0 2024-03-23T11:18:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.568+0000 7f2df1400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec155b90 con 0x7f2dec0cf650 2024-03-23T11:18:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.568+0000 7f2deb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec15cdc0 0x7f2dec15ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.568+0000 7f2deb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec15cdc0 0x7f2dec15ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52574/0 (socket says 172.21.15.62:52574) 2024-03-23T11:18:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.568+0000 7f2deb200640 1 -- 172.21.15.62:0/1435665714 learned_addr learned my addr 172.21.15.62:0/1435665714 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.568+0000 7f2de9e00640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dec15c9e0 0x7f2dec15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.568+0000 7f2dea800640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dec0cf650 0x7f2dec151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.569+0000 7f2deb200640 1 -- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dec0cf650 msgr2=0x7f2dec151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.569+0000 7f2deb200640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dec0cf650 0x7f2dec151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.569+0000 7f2deb200640 1 -- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dec15c9e0 msgr2=0x7f2dec15e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.569+0000 7f2deb200640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dec15c9e0 0x7f2dec15e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.569+0000 7f2deb200640 1 -- 172.21.15.62:0/1435665714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de4009b30 con 0x7f2dec15cdc0 2024-03-23T11:18:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.569+0000 7f2deb200640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec15cdc0 0x7f2dec15ead0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2de4036040 tx=0x7f2de4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.569+0000 7f2de2a00640 1 -- 172.21.15.62:0/1435665714 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4009510 con 0x7f2dec15cdc0 2024-03-23T11:18:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.569+0000 7f2de2a00640 1 -- 172.21.15.62:0/1435665714 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de403e450 con 0x7f2dec15cdc0 2024-03-23T11:18:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.569+0000 7f2de2a00640 1 -- 172.21.15.62:0/1435665714 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de403e7e0 con 0x7f2dec15cdc0 2024-03-23T11:18:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.569+0000 7f2df1400640 1 -- 172.21.15.62:0/1435665714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dec1558b0 con 0x7f2dec15cdc0 2024-03-23T11:18:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.570+0000 7f2df1400640 1 -- 172.21.15.62:0/1435665714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dec152ce0 con 0x7f2dec15cdc0 2024-03-23T11:18:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.571+0000 7f2df1400640 1 -- 172.21.15.62:0/1435665714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dec066c90 con 0x7f2dec15cdc0 2024-03-23T11:18:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.571+0000 7f2de2a00640 1 -- 172.21.15.62:0/1435665714 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 88763+0+0 (secure 0 0 0) 0x7f2de402cd50 con 0x7f2dec15cdc0 2024-03-23T11:18:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.573+0000 7f2de2a00640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2dcc06a710 0x7f2dcc06cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.573+0000 7f2de2a00640 1 -- 172.21.15.62:0/1435665714 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f2de40ab210 con 0x7f2dec15cdc0 2024-03-23T11:18:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.573+0000 7f2dea800640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2dcc06a710 0x7f2dcc06cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.574+0000 7f2dea800640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2dcc06a710 0x7f2dcc06cbd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc004f60 tx=0x7f2ddc007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:48.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.577+0000 7f2de2a00640 1 -- 172.21.15.62:0/1435665714 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2de4076bf0 con 0x7f2dec15cdc0 2024-03-23T11:18:48.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.770+0000 7f2df1400640 1 -- 172.21.15.62:0/1435665714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dec071b60 con 0x7f2dec15cdc0 2024-03-23T11:18:48.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.772+0000 7f2de2a00640 1 -- 172.21.15.62:0/1435665714 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10440 (secure 0 0 0) 0x7f2de407a210 con 0x7f2dec15cdc0 2024-03-23T11:18:48.773 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:18:48.773 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":14,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:48.216045+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:18:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 -- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2dcc06a710 msgr2=0x7f2dcc06cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2dcc06a710 0x7f2dcc06cbd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc004f60 tx=0x7f2ddc007e30 comp rx=0 tx=0).stop 2024-03-23T11:18:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 -- 172.21.15.62:0/1435665714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec15cdc0 msgr2=0x7f2dec15ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec15cdc0 0x7f2dec15ead0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2de4036040 tx=0x7f2de4004a40 comp rx=0 tx=0).stop 2024-03-23T11:18:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 -- 172.21.15.62:0/1435665714 shutdown_connections 2024-03-23T11:18:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2dcc06a710 0x7f2dcc06cbd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec15cdc0 0x7f2dec15ead0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dec15c9e0 0x7f2dec15e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 --2- 172.21.15.62:0/1435665714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dec0cf650 0x7f2dec151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 -- 172.21.15.62:0/1435665714 >> 172.21.15.62:0/1435665714 conn(0x7f2dec071570 msgr2=0x7f2dec0b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 -- 172.21.15.62:0/1435665714 shutdown_connections 2024-03-23T11:18:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.775+0000 7f2df1400640 1 -- 172.21.15.62:0/1435665714 wait complete. 2024-03-23T11:18:48.791 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-23T11:18:48.791 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:18:48.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.997+0000 7f653f200640 1 Processor -- start 2024-03-23T11:18:48.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.997+0000 7f653f200640 1 -- start start 2024-03-23T11:18:48.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.999+0000 7f653f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f65400b2f30 0x7f6540070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:49.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.999+0000 7f653f200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65400cd140 con 0x7f6540143dd0 2024-03-23T11:18:49.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.999+0000 7f653f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65400cd3e0 con 0x7f6540070090 2024-03-23T11:18:49.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.999+0000 7f653f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6540070a60 con 0x7f65400b2f30 2024-03-23T11:18:49.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.999+0000 7f653de00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6540143dd0 0x7f654015abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:38180/0 (socket says 172.21.15.62:38180) 2024-03-23T11:18:49.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.999+0000 7f653de00640 1 -- 172.21.15.62:0/4225669691 learned_addr learned my addr 172.21.15.62:0/4225669691 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:18:49.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.999+0000 7f653e800640 1 --2- 172.21.15.62:0/4225669691 >> v2:172.21.15.120:3300/0 conn(0x7f65400b2f30 0x7f6540070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:49.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:48.999+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583076449 0 0) 0x7f65400cd140 con 0x7f6540143dd0 2024-03-23T11:18:49.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6524003520 con 0x7f6540143dd0 2024-03-23T11:18:49.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1290231294 0 0) 0x7f6524003520 con 0x7f6540143dd0 2024-03-23T11:18:49.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 >> v2:172.21.15.120:3300/0 conn(0x7f65400b2f30 msgr2=0x7f6540070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:49.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 --2- 172.21.15.62:0/4225669691 >> v2:172.21.15.120:3300/0 conn(0x7f65400b2f30 0x7f6540070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 >> v1:172.21.15.120:6789/0 conn(0x7f6540070090 legacy=0x7f65400b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:49.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65400cd6c0 con 0x7f6540143dd0 2024-03-23T11:18:49.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f6530001670 con 0x7f6540143dd0 2024-03-23T11:18:49.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 >> v1:172.21.15.62:6789/0 conn(0x7f6540143dd0 legacy=0x7f654015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 --2- 172.21.15.62:0/4225669691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6524003970 0x7f6524003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 --2- 172.21.15.62:0/4225669691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65240047c0 0x7f6524004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 --2- 172.21.15.62:0/4225669691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 0x7f652400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540070a60 con 0x7f6524005180 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652400bfd0 con 0x7f65240047c0 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f652400c190 con 0x7f6524003970 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.000+0000 7f653e800640 1 --2- 172.21.15.62:0/4225669691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 0x7f652400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.001+0000 7f653de00640 1 --2- 172.21.15.62:0/4225669691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6524003970 0x7f6524003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.001+0000 7f6537e00640 1 --2- 172.21.15.62:0/4225669691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65240047c0 0x7f6524004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.001+0000 7f653e800640 1 -- 172.21.15.62:0/4225669691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6524003970 msgr2=0x7f6524003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.001+0000 7f653e800640 1 --2- 172.21.15.62:0/4225669691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6524003970 0x7f6524003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.001+0000 7f653e800640 1 -- 172.21.15.62:0/4225669691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65240047c0 msgr2=0x7f6524004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:49.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.001+0000 7f653e800640 1 --2- 172.21.15.62:0/4225669691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65240047c0 0x7f6524004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.001+0000 7f653e800640 1 -- 172.21.15.62:0/4225669691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6524003520 con 0x7f6524005180 2024-03-23T11:18:49.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.002+0000 7f653e800640 1 -- 172.21.15.62:0/4225669691 reap_dead start 2024-03-23T11:18:49.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.002+0000 7f653e800640 1 --2- 172.21.15.62:0/4225669691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 0x7f652400b980 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6538009fc0 tx=0x7f65240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf4f42197be1250 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:49.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.002+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653802cc60 con 0x7f6524005180 2024-03-23T11:18:49.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.002+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653800f040 con 0x7f6524005180 2024-03-23T11:18:49.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.002+0000 7f653d400640 1 -- 172.21.15.62:0/4225669691 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538003990 con 0x7f6524005180 2024-03-23T11:18:49.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.003+0000 7f653f200640 1 -- 172.21.15.62:0/4225669691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 msgr2=0x7f652400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.003+0000 7f653f200640 1 --2- 172.21.15.62:0/4225669691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 0x7f652400b980 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6538009fc0 tx=0x7f65240057f0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.003+0000 7f653f200640 1 -- 172.21.15.62:0/4225669691 shutdown_connections 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.003+0000 7f653f200640 1 --2- 172.21.15.62:0/4225669691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 0x7f652400b980 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.003+0000 7f653f200640 1 -- 172.21.15.62:0/4225669691 >> 172.21.15.62:0/4225669691 conn(0x7f6540066050 msgr2=0x7f6540069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.003+0000 7f653f200640 1 -- 172.21.15.62:0/4225669691 shutdown_connections 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.003+0000 7f653f200640 1 -- 172.21.15.62:0/4225669691 wait complete. 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.005+0000 7f653f200640 1 Processor -- start 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.005+0000 7f653f200640 1 -- start start 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 0x7f65400afed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f654015c640 0x7f654015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65400c3730 0x7f65400c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654015d000 con 0x7f6524005180 2024-03-23T11:18:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401f2c90 con 0x7f65400c3730 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b0410 con 0x7f654015c640 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 0x7f65400afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 0x7f65400afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52596/0 (socket says 172.21.15.62:52596) 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653de00640 1 -- 172.21.15.62:0/2723345026 learned_addr learned my addr 172.21.15.62:0/2723345026 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653de00640 1 -- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f654015c640 msgr2=0x7f654015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653de00640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f654015c640 0x7f654015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653de00640 1 -- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65400c3730 msgr2=0x7f65400c3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653de00640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65400c3730 0x7f65400c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.006+0000 7f653de00640 1 -- 172.21.15.62:0/2723345026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65401f29b0 con 0x7f6524005180 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.007+0000 7f653de00640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 0x7f65400afed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f653000e700 tx=0x7f653000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.007+0000 7f6536000640 1 -- 172.21.15.62:0/2723345026 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653000c6c0 con 0x7f6524005180 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.007+0000 7f6536000640 1 -- 172.21.15.62:0/2723345026 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653000c820 con 0x7f6524005180 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.007+0000 7f6536000640 1 -- 172.21.15.62:0/2723345026 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653000cbd0 con 0x7f6524005180 2024-03-23T11:18:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.007+0000 7f653f200640 1 -- 172.21.15.62:0/2723345026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65400af3f0 con 0x7f6524005180 2024-03-23T11:18:49.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.007+0000 7f653f200640 1 -- 172.21.15.62:0/2723345026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65400af9b0 con 0x7f6524005180 2024-03-23T11:18:49.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.008+0000 7f653f200640 1 -- 172.21.15.62:0/2723345026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65400afbd0 con 0x7f6524005180 2024-03-23T11:18:49.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.009+0000 7f6536000640 1 -- 172.21.15.62:0/2723345026 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 88763+0+0 (secure 0 0 0) 0x7f653000cd30 con 0x7f6524005180 2024-03-23T11:18:49.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.010+0000 7f6536000640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f651c06a740 0x7f651c06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:49.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.011+0000 7f6536000640 1 -- 172.21.15.62:0/2723345026 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f65300885d0 con 0x7f6524005180 2024-03-23T11:18:49.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.011+0000 7f6537e00640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f651c06a740 0x7f651c06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:49.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.012+0000 7f6537e00640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f651c06a740 0x7f651c06cc00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f654015b380 tx=0x7f6528009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:49.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.015+0000 7f6536000640 1 -- 172.21.15.62:0/2723345026 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6530054030 con 0x7f6524005180 2024-03-23T11:18:49.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.255+0000 7f653f200640 1 -- 172.21.15.62:0/2723345026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6540069d40 con 0x7f6524005180 2024-03-23T11:18:49.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.304+0000 7f6536000640 1 -- 172.21.15.62:0/2723345026 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1253 (secure 0 0 0) 0x7f6530057650 con 0x7f6524005180 2024-03-23T11:18:49.304 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:18:49.304 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/496399527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":496399527},{"type":"v1","addr":"172.21.15.62:6833","nonce":496399527}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-03-23T11:18:49.304 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 2 2024-03-23T11:18:49.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.306+0000 7f653f200640 1 -- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f651c06a740 msgr2=0x7f651c06cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:49.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.306+0000 7f653f200640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f651c06a740 0x7f651c06cc00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f654015b380 tx=0x7f6528009040 comp rx=0 tx=0).stop 2024-03-23T11:18:49.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.306+0000 7f653f200640 1 -- 172.21.15.62:0/2723345026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 msgr2=0x7f65400afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:49.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.306+0000 7f653f200640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 0x7f65400afed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f653000e700 tx=0x7f653000ebc0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.307+0000 7f653f200640 1 -- 172.21.15.62:0/2723345026 shutdown_connections 2024-03-23T11:18:49.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.307+0000 7f653f200640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f651c06a740 0x7f651c06cc00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.307+0000 7f653f200640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65400c3730 0x7f65400c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.307+0000 7f653f200640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f654015c640 0x7f654015cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.307+0000 7f653f200640 1 --2- 172.21.15.62:0/2723345026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524005180 0x7f65400afed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:49.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.307+0000 7f653f200640 1 -- 172.21.15.62:0/2723345026 >> 172.21.15.62:0/2723345026 conn(0x7f6540066050 msgr2=0x7f65400cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:49.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.307+0000 7f653f200640 1 -- 172.21.15.62:0/2723345026 shutdown_connections 2024-03-23T11:18:49.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:49.307+0000 7f653f200640 1 -- 172.21.15.62:0/2723345026 wait complete. 2024-03-23T11:18:51.326 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:18:51.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.551+0000 7fba78a00640 1 Processor -- start 2024-03-23T11:18:51.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.551+0000 7fba78a00640 1 -- start start 2024-03-23T11:18:51.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.553+0000 7fba78a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba74143dd0 0x7fba7415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:51.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.553+0000 7fba78a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fba74070090 0x7fba740704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:51.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.553+0000 7fba78a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba74070a30 con 0x7fba740b2820 2024-03-23T11:18:51.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.553+0000 7fba78a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba740cd140 con 0x7fba74070090 2024-03-23T11:18:51.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.553+0000 7fba78a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba740cd3e0 con 0x7fba74143dd0 2024-03-23T11:18:51.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.553+0000 7fba73e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fba740b2820 0x7fba740b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:38196/0 (socket says 172.21.15.62:38196) 2024-03-23T11:18:51.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.553+0000 7fba73e00640 1 -- 172.21.15.62:0/2508452747 learned_addr learned my addr 172.21.15.62:0/2508452747 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:18:51.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba72a00640 1 --2- 172.21.15.62:0/2508452747 >> v2:172.21.15.120:3300/0 conn(0x7fba74070090 0x7fba740704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:51.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba73400640 1 --2- 172.21.15.62:0/2508452747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba74143dd0 0x7fba7415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:51.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349329560 0 0) 0x7fba74070a30 con 0x7fba740b2820 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba58003520 con 0x7fba740b2820 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1510441060 0 0) 0x7fba58003520 con 0x7fba740b2820 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba74143dd0 msgr2=0x7fba7415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba72000640 1 --2- 172.21.15.62:0/2508452747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba74143dd0 0x7fba7415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 >> v2:172.21.15.120:3300/0 conn(0x7fba74070090 msgr2=0x7fba740704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba72000640 1 --2- 172.21.15.62:0/2508452747 >> v2:172.21.15.120:3300/0 conn(0x7fba74070090 0x7fba740704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba740cd6c0 con 0x7fba740b2820 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fba6c0019f0 con 0x7fba740b2820 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.554+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 >> v1:172.21.15.62:6789/0 conn(0x7fba740b2820 legacy=0x7fba740b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72000640 1 --2- 172.21.15.62:0/2508452747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba74143dd0 0x7fba58003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72000640 1 --2- 172.21.15.62:0/2508452747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba58003ef0 0x7fba580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72000640 1 --2- 172.21.15.62:0/2508452747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba58004d00 0x7fba58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba740cd140 con 0x7fba58003ef0 2024-03-23T11:18:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba740cd3e0 con 0x7fba58004d00 2024-03-23T11:18:51.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5800b980 con 0x7fba74143dd0 2024-03-23T11:18:51.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72a00640 1 --2- 172.21.15.62:0/2508452747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba58003ef0 0x7fba580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:51.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72a00640 1 -- 172.21.15.62:0/2508452747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba74143dd0 msgr2=0x7fba58003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:51.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72a00640 1 --2- 172.21.15.62:0/2508452747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba74143dd0 0x7fba58003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:51.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72a00640 1 -- 172.21.15.62:0/2508452747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba58004d00 msgr2=0x7fba58005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:51.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72a00640 1 --2- 172.21.15.62:0/2508452747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba58004d00 0x7fba58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:51.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.555+0000 7fba72a00640 1 -- 172.21.15.62:0/2508452747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba58003520 con 0x7fba58003ef0 2024-03-23T11:18:51.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.557+0000 7fba72a00640 1 --2- 172.21.15.62:0/2508452747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba58003ef0 0x7fba580047c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba60009ac0 tx=0x7fba580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338e53aabbf5689d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:51.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.557+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6002c6d0 con 0x7fba58003ef0 2024-03-23T11:18:51.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.557+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6002c830 con 0x7fba58003ef0 2024-03-23T11:18:51.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.557+0000 7fba72000640 1 -- 172.21.15.62:0/2508452747 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6002cbc0 con 0x7fba58003ef0 2024-03-23T11:18:51.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.558+0000 7fba78a00640 1 -- 172.21.15.62:0/2508452747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba58003ef0 msgr2=0x7fba580047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:51.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.558+0000 7fba78a00640 1 --2- 172.21.15.62:0/2508452747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba58003ef0 0x7fba580047c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba60009ac0 tx=0x7fba580057f0 comp rx=0 tx=0).stop 2024-03-23T11:18:51.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.558+0000 7fba73e00640 1 -- 172.21.15.62:0/2508452747 reap_dead start 2024-03-23T11:18:51.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.558+0000 7fba78a00640 1 -- 172.21.15.62:0/2508452747 shutdown_connections 2024-03-23T11:18:51.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.558+0000 7fba78a00640 1 -- 172.21.15.62:0/2508452747 >> 172.21.15.62:0/2508452747 conn(0x7fba74066050 msgr2=0x7fba74069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:51.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.558+0000 7fba78a00640 1 -- 172.21.15.62:0/2508452747 shutdown_connections 2024-03-23T11:18:51.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.558+0000 7fba78a00640 1 -- 172.21.15.62:0/2508452747 wait complete. 2024-03-23T11:18:51.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.560+0000 7fba78a00640 1 Processor -- start 2024-03-23T11:18:51.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.560+0000 7fba78a00640 1 -- start start 2024-03-23T11:18:51.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.561+0000 7fba78a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba58003ef0 0x7fba74159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:51.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.561+0000 7fba78a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba741f1b80 0x7fba740ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:51.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.561+0000 7fba78a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba740abd80 0x7fba74141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:51.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.561+0000 7fba78a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba740ac290 con 0x7fba741f1b80 2024-03-23T11:18:51.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.561+0000 7fba78a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba740ac570 con 0x7fba740abd80 2024-03-23T11:18:51.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.561+0000 7fba78a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba740ac850 con 0x7fba58003ef0 2024-03-23T11:18:51.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.561+0000 7fba72a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba741f1b80 0x7fba740ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.561+0000 7fba72a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba741f1b80 0x7fba740ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52610/0 (socket says 172.21.15.62:52610) 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.561+0000 7fba72a00640 1 -- 172.21.15.62:0/3154194763 learned_addr learned my addr 172.21.15.62:0/3154194763 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.561+0000 7fba73400640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba58003ef0 0x7fba74159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.561+0000 7fba73e00640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba740abd80 0x7fba74141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba72a00640 1 -- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba58003ef0 msgr2=0x7fba74159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba72a00640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba58003ef0 0x7fba74159f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba72a00640 1 -- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba740abd80 msgr2=0x7fba74141c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba72a00640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba740abd80 0x7fba74141c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba72a00640 1 -- 172.21.15.62:0/3154194763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7406c4a0 con 0x7fba741f1b80 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba73400640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba58003ef0 0x7fba74159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba73e00640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba740abd80 0x7fba74141c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:18:51.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba72a00640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba741f1b80 0x7fba740ab840 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fba6002c2d0 tx=0x7fba60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:51.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba6b400640 1 -- 172.21.15.62:0/3154194763 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60003810 con 0x7fba741f1b80 2024-03-23T11:18:51.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba6b400640 1 -- 172.21.15.62:0/3154194763 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba60003970 con 0x7fba741f1b80 2024-03-23T11:18:51.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba6b400640 1 -- 172.21.15.62:0/3154194763 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60003ce0 con 0x7fba741f1b80 2024-03-23T11:18:51.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.562+0000 7fba78a00640 1 -- 172.21.15.62:0/3154194763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7406c780 con 0x7fba741f1b80 2024-03-23T11:18:51.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.563+0000 7fba78a00640 1 -- 172.21.15.62:0/3154194763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7413c6c0 con 0x7fba741f1b80 2024-03-23T11:18:51.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.564+0000 7fba78a00640 1 -- 172.21.15.62:0/3154194763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba74066bb0 con 0x7fba741f1b80 2024-03-23T11:18:51.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.565+0000 7fba6b400640 1 -- 172.21.15.62:0/3154194763 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 88763+0+0 (secure 0 0 0) 0x7fba60040660 con 0x7fba741f1b80 2024-03-23T11:18:51.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.566+0000 7fba6b400640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba5406a710 0x7fba5406cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:51.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.566+0000 7fba6b400640 1 -- 172.21.15.62:0/3154194763 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fba600aaa90 con 0x7fba741f1b80 2024-03-23T11:18:51.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.567+0000 7fba73400640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba5406a710 0x7fba5406cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:51.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.568+0000 7fba73400640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba5406a710 0x7fba5406cbd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fba640099b0 tx=0x7fba64008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:51.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.572+0000 7fba6b400640 1 -- 172.21.15.62:0/3154194763 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba600762f0 con 0x7fba741f1b80 2024-03-23T11:18:51.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.822+0000 7fba78a00640 1 -- 172.21.15.62:0/3154194763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba74143180 con 0x7fba741f1b80 2024-03-23T11:18:51.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.823+0000 7fba6b400640 1 -- 172.21.15.62:0/3154194763 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3524 (secure 0 0 0) 0x7fba60039de0 con 0x7fba741f1b80 2024-03-23T11:18:51.823 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:18:51.823 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/496399527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":496399527},{"type":"v1","addr":"172.21.15.62:6833","nonce":496399527}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/156755611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":156755611},{"type":"v1","addr":"172.21.15.120:6835","nonce":156755611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2391380456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2391380456},{"type":"v1","addr":"172.21.15.62:6835","nonce":2391380456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3936303103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3936303103},{"type":"v1","addr":"172.21.15.120:6837","nonce":3936303103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-23T11:18:51.824 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 3 2024-03-23T11:18:51.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 -- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba5406a710 msgr2=0x7fba5406cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:51.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba5406a710 0x7fba5406cbd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fba640099b0 tx=0x7fba64008040 comp rx=0 tx=0).stop 2024-03-23T11:18:51.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 -- 172.21.15.62:0/3154194763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba741f1b80 msgr2=0x7fba740ab840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:51.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba741f1b80 0x7fba740ab840 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fba6002c2d0 tx=0x7fba60004730 comp rx=0 tx=0).stop 2024-03-23T11:18:51.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 -- 172.21.15.62:0/3154194763 shutdown_connections 2024-03-23T11:18:51.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba5406a710 0x7fba5406cbd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:51.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba741f1b80 0x7fba740ab840 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:51.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba740abd80 0x7fba74141c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:51.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 --2- 172.21.15.62:0/3154194763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba58003ef0 0x7fba74159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:51.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 -- 172.21.15.62:0/3154194763 >> 172.21.15.62:0/3154194763 conn(0x7fba74066050 msgr2=0x7fba740b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:51.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 -- 172.21.15.62:0/3154194763 shutdown_connections 2024-03-23T11:18:51.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:51.826+0000 7fba78a00640 1 -- 172.21.15.62:0/3154194763 wait complete. 2024-03-23T11:18:51.844 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:18:52.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.060+0000 7f19e2e00640 1 Processor -- start 2024-03-23T11:18:52.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.060+0000 7f19e2e00640 1 -- start start 2024-03-23T11:18:52.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.062+0000 7f19e2e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f19dc15e120 0x7f19dc099910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.062+0000 7f19e2e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f19dc0b0970 0x7f19dc15e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.062+0000 7f19e2e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc099480 con 0x7f19dc0cee40 2024-03-23T11:18:52.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.062+0000 7f19e2e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc098bc0 con 0x7f19dc0b0970 2024-03-23T11:18:52.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.062+0000 7f19e2e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc0cf2b0 con 0x7f19dc15e120 2024-03-23T11:18:52.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.062+0000 7f19e0e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f19dc0b0970 0x7f19dc15e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.062+0000 7f19e0e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f19dc0b0970 0x7f19dc15e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52614/0 (socket says 172.21.15.62:52614) 2024-03-23T11:18:52.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.062+0000 7f19d3e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f19dc0cee40 0x7f19dc0b0260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:38198/0 (socket says 172.21.15.62:38198) 2024-03-23T11:18:52.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.062+0000 7f19e0e00640 1 -- 172.21.15.62:0/2786505628 learned_addr learned my addr 172.21.15.62:0/2786505628 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:52.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.063+0000 7f19dbe00640 1 --2- 172.21.15.62:0/2786505628 >> v2:172.21.15.120:3300/0 conn(0x7f19dc15e120 0x7f19dc099910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.063+0000 7f19e0e00640 1 -- 172.21.15.62:0/2786505628 >> v2:172.21.15.120:3300/0 conn(0x7f19dc15e120 msgr2=0x7f19dc099910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:52.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.063+0000 7f19e0e00640 1 --2- 172.21.15.62:0/2786505628 >> v2:172.21.15.120:3300/0 conn(0x7f19dc15e120 0x7f19dc099910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.063+0000 7f19e0e00640 1 -- 172.21.15.62:0/2786505628 >> v1:172.21.15.62:6789/0 conn(0x7f19dc0cee40 legacy=0x7f19dc0b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.063+0000 7f19e0e00640 1 -- 172.21.15.62:0/2786505628 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc098e30 con 0x7f19dc0b0970 2024-03-23T11:18:52.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.064+0000 7f19e0e00640 1 --2- 172.21.15.62:0/2786505628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19dc0b0970 0x7f19dc15e600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0718e0 tx=0x7f19d402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d40d879c3bac84 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:52.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.064+0000 7f19db400640 1 -- 172.21.15.62:0/2786505628 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d402cb10 con 0x7f19dc0b0970 2024-03-23T11:18:52.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.064+0000 7f19db400640 1 -- 172.21.15.62:0/2786505628 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d402cc70 con 0x7f19dc0b0970 2024-03-23T11:18:52.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.064+0000 7f19db400640 1 -- 172.21.15.62:0/2786505628 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d40343f0 con 0x7f19dc0b0970 2024-03-23T11:18:52.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.065+0000 7f19e2e00640 1 -- 172.21.15.62:0/2786505628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19dc0b0970 msgr2=0x7f19dc15e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.065+0000 7f19e2e00640 1 --2- 172.21.15.62:0/2786505628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19dc0b0970 0x7f19dc15e600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0718e0 tx=0x7f19d402bf70 comp rx=0 tx=0).stop 2024-03-23T11:18:52.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.065+0000 7f19e2e00640 1 -- 172.21.15.62:0/2786505628 shutdown_connections 2024-03-23T11:18:52.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.065+0000 7f19e2e00640 1 --2- 172.21.15.62:0/2786505628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19dc0b0970 0x7f19dc15e600 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.065+0000 7f19e2e00640 1 --2- 172.21.15.62:0/2786505628 >> v2:172.21.15.120:3300/0 conn(0x7f19dc15e120 0x7f19dc099910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.065+0000 7f19e2e00640 1 -- 172.21.15.62:0/2786505628 >> 172.21.15.62:0/2786505628 conn(0x7f19dc0b23f0 msgr2=0x7f19dc0b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:52.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.065+0000 7f19e2e00640 1 -- 172.21.15.62:0/2786505628 shutdown_connections 2024-03-23T11:18:52.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.065+0000 7f19e2e00640 1 -- 172.21.15.62:0/2786505628 wait complete. 2024-03-23T11:18:52.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.067+0000 7f19e2e00640 1 Processor -- start 2024-03-23T11:18:52.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.067+0000 7f19e2e00640 1 -- start start 2024-03-23T11:18:52.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.068+0000 7f19e2e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19dc0b0970 0x7f19dc1f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.068+0000 7f19e2e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19dc0cee40 0x7f19dc0ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.068+0000 7f19e2e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19dc15e120 0x7f19dc0c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.068+0000 7f19e2e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc1f3b50 con 0x7f19dc0b0970 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.068+0000 7f19e2e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc154330 con 0x7f19dc0cee40 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.068+0000 7f19e2e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0b66e0 con 0x7f19dc15e120 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.068+0000 7f19dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19dc0b0970 0x7f19dc1f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.068+0000 7f19dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19dc0b0970 0x7f19dc1f3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52620/0 (socket says 172.21.15.62:52620) 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.068+0000 7f19dbe00640 1 -- 172.21.15.62:0/1141505368 learned_addr learned my addr 172.21.15.62:0/1141505368 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19d3e00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19dc0cee40 0x7f19dc0ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19e0e00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19dc15e120 0x7f19dc0c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19dbe00640 1 -- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19dc15e120 msgr2=0x7f19dc0c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19dbe00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19dc15e120 0x7f19dc0c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19dbe00640 1 -- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19dc0cee40 msgr2=0x7f19dc0ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19dbe00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19dc0cee40 0x7f19dc0ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19dbe00640 1 -- 172.21.15.62:0/1141505368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d4009b30 con 0x7f19dc0b0970 2024-03-23T11:18:52.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19e0e00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19dc15e120 0x7f19dc0c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:18:52.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19dbe00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19dc0b0970 0x7f19dc1f3610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00d5f0 tx=0x7f19cc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:52.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19d8c00640 1 -- 172.21.15.62:0/1141505368 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc0125f0 con 0x7f19dc0b0970 2024-03-23T11:18:52.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19d8c00640 1 -- 172.21.15.62:0/1141505368 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc012750 con 0x7f19dc0b0970 2024-03-23T11:18:52.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.069+0000 7f19d8c00640 1 -- 172.21.15.62:0/1141505368 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc012b00 con 0x7f19dc0b0970 2024-03-23T11:18:52.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.070+0000 7f19e2e00640 1 -- 172.21.15.62:0/1141505368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc0b5ad0 con 0x7f19dc0b0970 2024-03-23T11:18:52.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.070+0000 7f19e2e00640 1 -- 172.21.15.62:0/1141505368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc0b6090 con 0x7f19dc0b0970 2024-03-23T11:18:52.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.071+0000 7f19e2e00640 1 -- 172.21.15.62:0/1141505368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc0672c0 con 0x7f19dc0b0970 2024-03-23T11:18:52.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.072+0000 7f19d8c00640 1 -- 172.21.15.62:0/1141505368 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 88763+0+0 (secure 0 0 0) 0x7f19cc012c60 con 0x7f19dc0b0970 2024-03-23T11:18:52.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.072+0000 7f19d8c00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19b806a710 0x7f19b806cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.073+0000 7f19d8c00640 1 -- 172.21.15.62:0/1141505368 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f19cc088ea0 con 0x7f19dc0b0970 2024-03-23T11:18:52.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.073+0000 7f19d3e00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19b806a710 0x7f19b806cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.074+0000 7f19d3e00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19b806a710 0x7f19b806cbd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0c9cc0 tx=0x7f19c4007c80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:52.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.077+0000 7f19d8c00640 1 -- 172.21.15.62:0/1141505368 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f19cc054650 con 0x7f19dc0b0970 2024-03-23T11:18:52.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.288+0000 7f19e2e00640 1 -- 172.21.15.62:0/1141505368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19dc0b21d0 con 0x7f19dc0b0970 2024-03-23T11:18:52.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.289+0000 7f19d8c00640 1 -- 172.21.15.62:0/1141505368 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12395 (secure 0 0 0) 0x7f19cc057c70 con 0x7f19dc0b0970 2024-03-23T11:18:52.289 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:18:52.289 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":17,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:51.235907+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:18:52.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 -- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19b806a710 msgr2=0x7f19b806cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19b806a710 0x7f19b806cbd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0c9cc0 tx=0x7f19c4007c80 comp rx=0 tx=0).stop 2024-03-23T11:18:52.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 -- 172.21.15.62:0/1141505368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19dc0b0970 msgr2=0x7f19dc1f3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19dc0b0970 0x7f19dc1f3610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00d5f0 tx=0x7f19cc00dab0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 -- 172.21.15.62:0/1141505368 shutdown_connections 2024-03-23T11:18:52.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19b806a710 0x7f19b806cbd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19dc15e120 0x7f19dc0c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19dc0cee40 0x7f19dc0ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 --2- 172.21.15.62:0/1141505368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19dc0b0970 0x7f19dc1f3610 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 -- 172.21.15.62:0/1141505368 >> 172.21.15.62:0/1141505368 conn(0x7f19dc0b23f0 msgr2=0x7f19dc0af580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:52.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 -- 172.21.15.62:0/1141505368 shutdown_connections 2024-03-23T11:18:52.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.292+0000 7f19e2e00640 1 -- 172.21.15.62:0/1141505368 wait complete. 2024-03-23T11:18:52.312 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:18:52.312 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:18:52.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.509+0000 7fdbb4c00640 1 Processor -- start 2024-03-23T11:18:52.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.510+0000 7fdbb4c00640 1 -- start start 2024-03-23T11:18:52.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.511+0000 7fdbb4c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdbb015b850 0x7fdbb00b0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.511+0000 7fdbb4c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb00b07a0 0x7fdbb00aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.511+0000 7fdbb4c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb015b3c0 con 0x7fdbb015a8b0 2024-03-23T11:18:52.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.511+0000 7fdbb4c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb00b0c30 con 0x7fdbb015b850 2024-03-23T11:18:52.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.511+0000 7fdbb4c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb015bc90 con 0x7fdbb00b07a0 2024-03-23T11:18:52.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.511+0000 7fdba6a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdbb015b850 0x7fdbb00b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.511+0000 7fdba6a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdbb015b850 0x7fdbb00b0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52628/0 (socket says 172.21.15.62:52628) 2024-03-23T11:18:52.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.511+0000 7fdba6a00640 1 -- 172.21.15.62:0/408211181 learned_addr learned my addr 172.21.15.62:0/408211181 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:52.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.512+0000 7fdbafe00640 1 --2- 172.21.15.62:0/408211181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb00b07a0 0x7fdbb00aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.512+0000 7fdba6a00640 1 -- 172.21.15.62:0/408211181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb00b07a0 msgr2=0x7fdbb00aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:52.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.512+0000 7fdba6a00640 1 --2- 172.21.15.62:0/408211181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb00b07a0 0x7fdbb00aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.512+0000 7fdba6a00640 1 -- 172.21.15.62:0/408211181 >> v1:172.21.15.120:6789/0 conn(0x7fdbb015a8b0 legacy=0x7fdbb00cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.512+0000 7fdba6a00640 1 -- 172.21.15.62:0/408211181 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb00cca40 con 0x7fdbb015b850 2024-03-23T11:18:52.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.513+0000 7fdba6a00640 1 --2- 172.21.15.62:0/408211181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb015b850 0x7fdbb00b0260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009870 tx=0x7fdb9802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51229471a5f65721 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:52.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.514+0000 7fdbaea00640 1 -- 172.21.15.62:0/408211181 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98030840 con 0x7fdbb015b850 2024-03-23T11:18:52.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.514+0000 7fdbaea00640 1 -- 172.21.15.62:0/408211181 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb980309a0 con 0x7fdbb015b850 2024-03-23T11:18:52.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.514+0000 7fdbaea00640 1 -- 172.21.15.62:0/408211181 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98030d30 con 0x7fdbb015b850 2024-03-23T11:18:52.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.514+0000 7fdbb4c00640 1 -- 172.21.15.62:0/408211181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb015b850 msgr2=0x7fdbb00b0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.514+0000 7fdbb4c00640 1 --2- 172.21.15.62:0/408211181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb015b850 0x7fdbb00b0260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009870 tx=0x7fdb9802b980 comp rx=0 tx=0).stop 2024-03-23T11:18:52.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.514+0000 7fdbb4c00640 1 -- 172.21.15.62:0/408211181 shutdown_connections 2024-03-23T11:18:52.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.514+0000 7fdbb4c00640 1 --2- 172.21.15.62:0/408211181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb00b07a0 0x7fdbb00aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.514+0000 7fdbb4c00640 1 --2- 172.21.15.62:0/408211181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb015b850 0x7fdbb00b0260 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.514+0000 7fdbb4c00640 1 -- 172.21.15.62:0/408211181 >> 172.21.15.62:0/408211181 conn(0x7fdbb00989d0 msgr2=0x7fdbb00cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:52.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.514+0000 7fdbb4c00640 1 -- 172.21.15.62:0/408211181 shutdown_connections 2024-03-23T11:18:52.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.514+0000 7fdbb4c00640 1 -- 172.21.15.62:0/408211181 wait complete. 2024-03-23T11:18:52.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.517+0000 7fdbb4c00640 1 Processor -- start 2024-03-23T11:18:52.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.517+0000 7fdbb4c00640 1 -- start start 2024-03-23T11:18:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.519+0000 7fdbb4c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb00b07a0 0x7fdbb01511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.519+0000 7fdbb4c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb015a8b0 0x7fdbb01efa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.519+0000 7fdbb4c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb015b850 0x7fdbb01eff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.519+0000 7fdbb4c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb0152850 con 0x7fdbb015a8b0 2024-03-23T11:18:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.519+0000 7fdbb4c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb0151bd0 con 0x7fdbb015b850 2024-03-23T11:18:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.519+0000 7fdbb4c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb0151eb0 con 0x7fdbb00b07a0 2024-03-23T11:18:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.519+0000 7fdba6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb015a8b0 0x7fdbb01efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.519+0000 7fdba6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb015a8b0 0x7fdbb01efa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52630/0 (socket says 172.21.15.62:52630) 2024-03-23T11:18:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.519+0000 7fdba6a00640 1 -- 172.21.15.62:0/831219642 learned_addr learned my addr 172.21.15.62:0/831219642 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.519+0000 7fdbaf400640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb00b07a0 0x7fdbb01511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.520+0000 7fdbafe00640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb015b850 0x7fdbb01eff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.520+0000 7fdba6a00640 1 -- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb00b07a0 msgr2=0x7fdbb01511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.520+0000 7fdba6a00640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb00b07a0 0x7fdbb01511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.520+0000 7fdba6a00640 1 -- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb015b850 msgr2=0x7fdbb01eff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.520+0000 7fdba6a00640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb015b850 0x7fdbb01eff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.520+0000 7fdba6a00640 1 -- 172.21.15.62:0/831219642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb980094d0 con 0x7fdbb015a8b0 2024-03-23T11:18:52.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.520+0000 7fdba6a00640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb015a8b0 0x7fdbb01efa20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb98000c00 tx=0x7fdb98004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:52.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.520+0000 7fdba7e00640 1 -- 172.21.15.62:0/831219642 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98039c90 con 0x7fdbb015a8b0 2024-03-23T11:18:52.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.520+0000 7fdba7e00640 1 -- 172.21.15.62:0/831219642 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb98039df0 con 0x7fdbb015a8b0 2024-03-23T11:18:52.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.520+0000 7fdba7e00640 1 -- 172.21.15.62:0/831219642 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98043640 con 0x7fdbb015a8b0 2024-03-23T11:18:52.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.520+0000 7fdbb4c00640 1 -- 172.21.15.62:0/831219642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb0152190 con 0x7fdbb015a8b0 2024-03-23T11:18:52.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.521+0000 7fdbb4c00640 1 -- 172.21.15.62:0/831219642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb00c95f0 con 0x7fdbb015a8b0 2024-03-23T11:18:52.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.522+0000 7fdbb4c00640 1 -- 172.21.15.62:0/831219642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb00c9780 con 0x7fdbb015a8b0 2024-03-23T11:18:52.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.523+0000 7fdba7e00640 1 -- 172.21.15.62:0/831219642 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 88763+0+0 (secure 0 0 0) 0x7fdb980437a0 con 0x7fdbb015a8b0 2024-03-23T11:18:52.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.524+0000 7fdba7e00640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdb8c06a710 0x7fdb8c06cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.524+0000 7fdba7e00640 1 -- 172.21.15.62:0/831219642 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdb9803e070 con 0x7fdbb015a8b0 2024-03-23T11:18:52.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.525+0000 7fdbaf400640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdb8c06a710 0x7fdb8c06cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.526+0000 7fdbaf400640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdb8c06a710 0x7fdb8c06cbd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdbb015ca40 tx=0x7fdba0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:52.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.528+0000 7fdba7e00640 1 -- 172.21.15.62:0/831219642 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdb98077180 con 0x7fdbb015a8b0 2024-03-23T11:18:52.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.732+0000 7fdbb4c00640 1 -- 172.21.15.62:0/831219642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdbb00672c0 con 0x7fdbb015a8b0 2024-03-23T11:18:52.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.733+0000 7fdba7e00640 1 -- 172.21.15.62:0/831219642 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdb9807a7a0 con 0x7fdbb015a8b0 2024-03-23T11:18:52.734 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:18:52.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 -- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdb8c06a710 msgr2=0x7fdb8c06cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdb8c06a710 0x7fdb8c06cbd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdbb015ca40 tx=0x7fdba0009040 comp rx=0 tx=0).stop 2024-03-23T11:18:52.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 -- 172.21.15.62:0/831219642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb015a8b0 msgr2=0x7fdbb01efa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb015a8b0 0x7fdbb01efa20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb98000c00 tx=0x7fdb98004730 comp rx=0 tx=0).stop 2024-03-23T11:18:52.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 -- 172.21.15.62:0/831219642 shutdown_connections 2024-03-23T11:18:52.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdb8c06a710 0x7fdb8c06cbd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb015a8b0 0x7fdbb01efa20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdbb015b850 0x7fdbb01eff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 --2- 172.21.15.62:0/831219642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdbb00b07a0 0x7fdbb01511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 -- 172.21.15.62:0/831219642 >> 172.21.15.62:0/831219642 conn(0x7fdbb00989d0 msgr2=0x7fdbb00b0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:52.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 -- 172.21.15.62:0/831219642 shutdown_connections 2024-03-23T11:18:52.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.736+0000 7fdbb4c00640 1 -- 172.21.15.62:0/831219642 wait complete. 2024-03-23T11:18:52.753 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T11:18:52.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.954+0000 7ff6e3200640 1 Processor -- start 2024-03-23T11:18:52.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.954+0000 7ff6e3200640 1 -- start start 2024-03-23T11:18:52.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.956+0000 7ff6e3200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff6e406b150 0x7ff6e41626c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.956+0000 7ff6e3200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 0x7ff6e40c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.956+0000 7ff6e3200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e4161b40 con 0x7ff6e40b0b30 2024-03-23T11:18:52.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.956+0000 7ff6e3200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e4098ce0 con 0x7ff6e406b150 2024-03-23T11:18:52.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.956+0000 7ff6e3200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40989d0 con 0x7ff6e4162c00 2024-03-23T11:18:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.957+0000 7ff6e1400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff6e406b150 0x7ff6e41626c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.957+0000 7ff6e2800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 0x7ff6e40c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.957+0000 7ff6e2800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 0x7ff6e40c34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:57378/0 (socket says 172.21.15.62:57378) 2024-03-23T11:18:52.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.957+0000 7ff6e1400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff6e406b150 0x7ff6e41626c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:41110/0 (socket says 172.21.15.62:41110) 2024-03-23T11:18:52.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.957+0000 7ff6e2800640 1 -- 172.21.15.62:0/931134357 learned_addr learned my addr 172.21.15.62:0/931134357 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:52.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.957+0000 7ff6e2800640 1 -- 172.21.15.62:0/931134357 >> v2:172.21.15.120:3300/0 conn(0x7ff6e406b150 msgr2=0x7ff6e41626c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:52.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.957+0000 7ff6e2800640 1 --2- 172.21.15.62:0/931134357 >> v2:172.21.15.120:3300/0 conn(0x7ff6e406b150 0x7ff6e41626c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.957+0000 7ff6e2800640 1 -- 172.21.15.62:0/931134357 >> v1:172.21.15.120:6789/0 conn(0x7ff6e40b0b30 legacy=0x7ff6e4161350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.957+0000 7ff6e2800640 1 -- 172.21.15.62:0/931134357 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4161e20 con 0x7ff6e4162c00 2024-03-23T11:18:52.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.957+0000 7ff6e1400640 1 --2- 172.21.15.62:0/931134357 >> v2:172.21.15.120:3300/0 conn(0x7ff6e406b150 0x7ff6e41626c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:18:52.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.959+0000 7ff6e2800640 1 --2- 172.21.15.62:0/931134357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 0x7ff6e40c34f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc009e50 tx=0x7ff6dc02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=272df663d85ecebf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:52.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.959+0000 7ff6e0a00640 1 -- 172.21.15.62:0/931134357 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc02ca70 con 0x7ff6e4162c00 2024-03-23T11:18:52.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.959+0000 7ff6e0a00640 1 -- 172.21.15.62:0/931134357 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc02cbd0 con 0x7ff6e4162c00 2024-03-23T11:18:52.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.959+0000 7ff6e0a00640 1 -- 172.21.15.62:0/931134357 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0343f0 con 0x7ff6e4162c00 2024-03-23T11:18:52.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.960+0000 7ff6e3200640 1 -- 172.21.15.62:0/931134357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 msgr2=0x7ff6e40c34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.960+0000 7ff6e3200640 1 --2- 172.21.15.62:0/931134357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 0x7ff6e40c34f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc009e50 tx=0x7ff6dc02bed0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.960+0000 7ff6e3200640 1 -- 172.21.15.62:0/931134357 shutdown_connections 2024-03-23T11:18:52.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.960+0000 7ff6e3200640 1 --2- 172.21.15.62:0/931134357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 0x7ff6e40c34f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.960+0000 7ff6e3200640 1 --2- 172.21.15.62:0/931134357 >> v2:172.21.15.120:3300/0 conn(0x7ff6e406b150 0x7ff6e41626c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.960+0000 7ff6e3200640 1 -- 172.21.15.62:0/931134357 >> 172.21.15.62:0/931134357 conn(0x7ff6e40b2010 msgr2=0x7ff6e40b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:52.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.961+0000 7ff6e3200640 1 -- 172.21.15.62:0/931134357 shutdown_connections 2024-03-23T11:18:52.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.961+0000 7ff6e3200640 1 -- 172.21.15.62:0/931134357 wait complete. 2024-03-23T11:18:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.963+0000 7ff6e3200640 1 Processor -- start 2024-03-23T11:18:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.964+0000 7ff6e3200640 1 -- start start 2024-03-23T11:18:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.965+0000 7ff6e3200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6e406b150 0x7ff6e41f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.965+0000 7ff6e3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40b0b30 0x7ff6e40cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.965+0000 7ff6e3200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 0x7ff6e40cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.965+0000 7ff6e3200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40c83a0 con 0x7ff6e40b0b30 2024-03-23T11:18:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.965+0000 7ff6e3200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40c7720 con 0x7ff6e406b150 2024-03-23T11:18:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.965+0000 7ff6e3200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40c7a00 con 0x7ff6e4162c00 2024-03-23T11:18:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.965+0000 7ff6e1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40b0b30 0x7ff6e40cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.966+0000 7ff6e1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40b0b30 0x7ff6e40cba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52632/0 (socket says 172.21.15.62:52632) 2024-03-23T11:18:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.966+0000 7ff6e1400640 1 -- 172.21.15.62:0/10696420 learned_addr learned my addr 172.21.15.62:0/10696420 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.966+0000 7ff6e2800640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 0x7ff6e40cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.966+0000 7ff6e1e00640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6e406b150 0x7ff6e41f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.966+0000 7ff6e1400640 1 -- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 msgr2=0x7ff6e40cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.966+0000 7ff6e1400640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 0x7ff6e40cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.966+0000 7ff6e1400640 1 -- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6e406b150 msgr2=0x7ff6e41f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.966+0000 7ff6e1400640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6e406b150 0x7ff6e41f6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.966+0000 7ff6e1400640 1 -- 172.21.15.62:0/10696420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d0009400 con 0x7ff6e40b0b30 2024-03-23T11:18:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.966+0000 7ff6e1e00640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6e406b150 0x7ff6e41f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:18:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.966+0000 7ff6e1400640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40b0b30 0x7ff6e40cba60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000ebe0 tx=0x7ff6d000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.967+0000 7ff6da000640 1 -- 172.21.15.62:0/10696420 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0009760 con 0x7ff6e40b0b30 2024-03-23T11:18:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.967+0000 7ff6da000640 1 -- 172.21.15.62:0/10696420 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d000cc30 con 0x7ff6e40b0b30 2024-03-23T11:18:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.967+0000 7ff6da000640 1 -- 172.21.15.62:0/10696420 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d000eee0 con 0x7ff6e40b0b30 2024-03-23T11:18:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.967+0000 7ff6e3200640 1 -- 172.21.15.62:0/10696420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc009b30 con 0x7ff6e40b0b30 2024-03-23T11:18:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.967+0000 7ff6e3200640 1 -- 172.21.15.62:0/10696420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e40c7fc0 con 0x7ff6e40b0b30 2024-03-23T11:18:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.969+0000 7ff6e3200640 1 -- 172.21.15.62:0/10696420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e406ce90 con 0x7ff6e40b0b30 2024-03-23T11:18:52.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.969+0000 7ff6da000640 1 -- 172.21.15.62:0/10696420 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 88763+0+0 (secure 0 0 0) 0x7ff6d0013470 con 0x7ff6e40b0b30 2024-03-23T11:18:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.970+0000 7ff6da000640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6c406a740 0x7ff6c406cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.970+0000 7ff6da000640 1 -- 172.21.15.62:0/10696420 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff6d0088b70 con 0x7ff6e40b0b30 2024-03-23T11:18:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.970+0000 7ff6e1e00640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6c406a740 0x7ff6c406cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:52.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.972+0000 7ff6e1e00640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6c406a740 0x7ff6c406cc00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4001700 tx=0x7ff6d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:52.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:52.974+0000 7ff6da000640 1 -- 172.21.15.62:0/10696420 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6d0054460 con 0x7ff6e40b0b30 2024-03-23T11:18:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.181+0000 7ff6e3200640 1 -- 172.21.15.62:0/10696420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff6e404e110 con 0x7ff6e40b0b30 2024-03-23T11:18:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.182+0000 7ff6da000640 1 -- 172.21.15.62:0/10696420 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff6d0057a80 con 0x7ff6e40b0b30 2024-03-23T11:18:53.182 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:18:53.183 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:18:53.183 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:18:53.183 INFO:teuthology.orchestra.run.smithi062.stdout: "poolnum": 1, 2024-03-23T11:18:53.183 INFO:teuthology.orchestra.run.smithi062.stdout: "poolname": ".mgr" 2024-03-23T11:18:53.183 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:18:53.183 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:18:53.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 -- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6c406a740 msgr2=0x7ff6c406cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6c406a740 0x7ff6c406cc00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4001700 tx=0x7ff6d4009040 comp rx=0 tx=0).stop 2024-03-23T11:18:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 -- 172.21.15.62:0/10696420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40b0b30 msgr2=0x7ff6e40cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40b0b30 0x7ff6e40cba60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000ebe0 tx=0x7ff6d000c520 comp rx=0 tx=0).stop 2024-03-23T11:18:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 -- 172.21.15.62:0/10696420 shutdown_connections 2024-03-23T11:18:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6c406a740 0x7ff6c406cc00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6e40b0b30 0x7ff6e40cba60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6e4162c00 0x7ff6e40cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 --2- 172.21.15.62:0/10696420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6e406b150 0x7ff6e41f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 -- 172.21.15.62:0/10696420 >> 172.21.15.62:0/10696420 conn(0x7ff6e40b2010 msgr2=0x7ff6e4072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 -- 172.21.15.62:0/10696420 shutdown_connections 2024-03-23T11:18:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.185+0000 7ff6e3200640 1 -- 172.21.15.62:0/10696420 wait complete. 2024-03-23T11:18:53.203 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T11:18:53.204 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T11:18:53.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.424+0000 7f9197e00640 1 Processor -- start 2024-03-23T11:18:53.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.424+0000 7f9197e00640 1 -- start start 2024-03-23T11:18:53.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.425+0000 7f9197e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f91980ce060 0x7f91980af950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:53.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.425+0000 7f9197e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91980989d0 con 0x7f919815a8a0 2024-03-23T11:18:53.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.425+0000 7f9197e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91980ce4f0 con 0x7f91980af550 2024-03-23T11:18:53.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.425+0000 7f9197e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91980cbbe0 con 0x7f91980ce060 2024-03-23T11:18:53.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.425+0000 7f9196a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f919815a8a0 0x7f91980ce720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:38202/0 (socket says 172.21.15.62:38202) 2024-03-23T11:18:53.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.425+0000 7f9196a00640 1 -- 172.21.15.62:0/788329667 learned_addr learned my addr 172.21.15.62:0/788329667 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:18:53.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9197400640 1 --2- 172.21.15.62:0/788329667 >> v2:172.21.15.120:3300/0 conn(0x7f91980ce060 0x7f91980af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:53.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510413136 0 0) 0x7f91980989d0 con 0x7f919815a8a0 2024-03-23T11:18:53.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f917c003520 con 0x7f919815a8a0 2024-03-23T11:18:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 28379629 0 0) 0x7f917c003520 con 0x7f919815a8a0 2024-03-23T11:18:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 >> v2:172.21.15.120:3300/0 conn(0x7f91980ce060 msgr2=0x7f91980af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 --2- 172.21.15.62:0/788329667 >> v2:172.21.15.120:3300/0 conn(0x7f91980ce060 0x7f91980af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 >> v1:172.21.15.120:6789/0 conn(0x7f91980af550 legacy=0x7f9198098e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91980cbec0 con 0x7f919815a8a0 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f91880019f0 con 0x7f919815a8a0 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 >> v1:172.21.15.62:6789/0 conn(0x7f919815a8a0 legacy=0x7f91980ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 --2- 172.21.15.62:0/788329667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f917c003970 0x7f917c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 --2- 172.21.15.62:0/788329667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f917c0047c0 0x7f917c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 --2- 172.21.15.62:0/788329667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 0x7f917c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91980cbbe0 con 0x7f917c0052a0 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c00c170 con 0x7f917c003970 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.426+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917c00c330 con 0x7f917c0047c0 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.427+0000 7f9197400640 1 --2- 172.21.15.62:0/788329667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 0x7f917c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.427+0000 7f9196a00640 1 --2- 172.21.15.62:0/788329667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f917c003970 0x7f917c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.427+0000 7f918e000640 1 --2- 172.21.15.62:0/788329667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f917c0047c0 0x7f917c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.427+0000 7f9197400640 1 -- 172.21.15.62:0/788329667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f917c0047c0 msgr2=0x7f917c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.427+0000 7f9197400640 1 --2- 172.21.15.62:0/788329667 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f917c0047c0 0x7f917c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.427+0000 7f9197400640 1 -- 172.21.15.62:0/788329667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f917c003970 msgr2=0x7f917c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.427+0000 7f9197400640 1 --2- 172.21.15.62:0/788329667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f917c003970 0x7f917c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.427+0000 7f9197400640 1 -- 172.21.15.62:0/788329667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91980cbec0 con 0x7f917c0052a0 2024-03-23T11:18:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.428+0000 7f9197400640 1 -- 172.21.15.62:0/788329667 reap_dead start 2024-03-23T11:18:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.428+0000 7f9197400640 1 --2- 172.21.15.62:0/788329667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 0x7f917c00b980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9190009a90 tx=0x7f917c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88731ca25922092 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.428+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190004020 con 0x7f917c0052a0 2024-03-23T11:18:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.428+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9190004180 con 0x7f917c0052a0 2024-03-23T11:18:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.428+0000 7f9196000640 1 -- 172.21.15.62:0/788329667 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919002c6b0 con 0x7f917c0052a0 2024-03-23T11:18:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.429+0000 7f9197e00640 1 -- 172.21.15.62:0/788329667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 msgr2=0x7f917c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.429+0000 7f9197e00640 1 --2- 172.21.15.62:0/788329667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 0x7f917c00b980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9190009a90 tx=0x7f917c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:18:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.429+0000 7f9197e00640 1 -- 172.21.15.62:0/788329667 shutdown_connections 2024-03-23T11:18:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.429+0000 7f9197e00640 1 --2- 172.21.15.62:0/788329667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 0x7f917c00b980 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.429+0000 7f9197e00640 1 -- 172.21.15.62:0/788329667 >> 172.21.15.62:0/788329667 conn(0x7f91980b1f80 msgr2=0x7f91980b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:53.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.429+0000 7f9197e00640 1 -- 172.21.15.62:0/788329667 shutdown_connections 2024-03-23T11:18:53.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.429+0000 7f9197e00640 1 -- 172.21.15.62:0/788329667 wait complete. 2024-03-23T11:18:53.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.431+0000 7f9197e00640 1 Processor -- start 2024-03-23T11:18:53.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.431+0000 7f9197e00640 1 -- start start 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9197e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 0x7f91980b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9197e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91980b3950 0x7f91980ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9197e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91981e7c10 0x7f91981e8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9197e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91981e85c0 con 0x7f917c0052a0 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9197e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91980b3e50 con 0x7f91981e7c10 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9197e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91980b4130 con 0x7f91980b3950 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9196a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 0x7f91980b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9196a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 0x7f91980b57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52648/0 (socket says 172.21.15.62:52648) 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9196a00640 1 -- 172.21.15.62:0/968437275 learned_addr learned my addr 172.21.15.62:0/968437275 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9197400640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91981e7c10 0x7f91981e8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9196a00640 1 -- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91980b3950 msgr2=0x7f91980ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9196a00640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91980b3950 0x7f91980ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.432+0000 7f9196a00640 1 -- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91981e7c10 msgr2=0x7f91981e8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:53.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.433+0000 7f9196a00640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91981e7c10 0x7f91981e8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:53.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.433+0000 7f9196a00640 1 -- 172.21.15.62:0/968437275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91980b4410 con 0x7f917c0052a0 2024-03-23T11:18:53.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.433+0000 7f9196a00640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 0x7f91980b57e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f91880079e0 tx=0x7f918800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:53.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.433+0000 7f918f400640 1 -- 172.21.15.62:0/968437275 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9188007ad0 con 0x7f917c0052a0 2024-03-23T11:18:53.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.433+0000 7f918f400640 1 -- 172.21.15.62:0/968437275 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918800ee50 con 0x7f917c0052a0 2024-03-23T11:18:53.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.433+0000 7f918f400640 1 -- 172.21.15.62:0/968437275 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9188013620 con 0x7f917c0052a0 2024-03-23T11:18:53.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.433+0000 7f9197e00640 1 -- 172.21.15.62:0/968437275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91980b46f0 con 0x7f917c0052a0 2024-03-23T11:18:53.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.433+0000 7f9197e00640 1 -- 172.21.15.62:0/968437275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91980ca750 con 0x7f917c0052a0 2024-03-23T11:18:53.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.434+0000 7f9197e00640 1 -- 172.21.15.62:0/968437275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91980cfcd0 con 0x7f917c0052a0 2024-03-23T11:18:53.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.435+0000 7f918f400640 1 -- 172.21.15.62:0/968437275 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 88763+0+0 (secure 0 0 0) 0x7f9188013780 con 0x7f917c0052a0 2024-03-23T11:18:53.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.436+0000 7f918f400640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f917406a740 0x7f917406cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:53.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.436+0000 7f918f400640 1 -- 172.21.15.62:0/968437275 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9188088ff0 con 0x7f917c0052a0 2024-03-23T11:18:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.437+0000 7f918e000640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f917406a740 0x7f917406cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.438+0000 7f918e000640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f917406a740 0x7f917406cc00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f91980b5770 tx=0x7f9180009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.441+0000 7f918f400640 1 -- 172.21.15.62:0/968437275 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f91880bc580 con 0x7f917c0052a0 2024-03-23T11:18:53.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.562+0000 7f918f400640 1 -- 172.21.15.62:0/968437275 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9188054bd0 con 0x7f917c0052a0 2024-03-23T11:18:53.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:53.650+0000 7f9197e00640 1 -- 172.21.15.62:0/968437275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f91980b48e0 con 0x7f917c0052a0 2024-03-23T11:18:54.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.738+0000 7f918f400640 1 -- 172.21.15.62:0/968437275 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9188087560 con 0x7f917c0052a0 2024-03-23T11:18:54.739 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' created 2024-03-23T11:18:54.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.743+0000 7f9197e00640 1 -- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f917406a740 msgr2=0x7f917406cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:54.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.743+0000 7f9197e00640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f917406a740 0x7f917406cc00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f91980b5770 tx=0x7f9180009040 comp rx=0 tx=0).stop 2024-03-23T11:18:54.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.743+0000 7f9197e00640 1 -- 172.21.15.62:0/968437275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 msgr2=0x7f91980b57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:54.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.743+0000 7f9197e00640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 0x7f91980b57e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f91880079e0 tx=0x7f918800e530 comp rx=0 tx=0).stop 2024-03-23T11:18:54.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.744+0000 7f9197e00640 1 -- 172.21.15.62:0/968437275 shutdown_connections 2024-03-23T11:18:54.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.744+0000 7f9197e00640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f917406a740 0x7f917406cc00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:54.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.744+0000 7f9197e00640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91981e7c10 0x7f91981e8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:54.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.744+0000 7f9197e00640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91980b3950 0x7f91980ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:54.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.744+0000 7f9197e00640 1 --2- 172.21.15.62:0/968437275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f917c0052a0 0x7f91980b57e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:54.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.744+0000 7f9197e00640 1 -- 172.21.15.62:0/968437275 >> 172.21.15.62:0/968437275 conn(0x7f91980b1f80 msgr2=0x7f91980b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:54.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.745+0000 7f9197e00640 1 -- 172.21.15.62:0/968437275 shutdown_connections 2024-03-23T11:18:54.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.745+0000 7f9197e00640 1 -- 172.21.15.62:0/968437275 wait complete. 2024-03-23T11:18:54.772 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T11:18:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.976+0000 7efce7800640 1 Processor -- start 2024-03-23T11:18:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.977+0000 7efce7800640 1 -- start start 2024-03-23T11:18:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.978+0000 7efce7800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efce0144320 0x7efce00cc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.978+0000 7efce7800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce00cff10 0x7efce0071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efce7800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce00cfb10 con 0x7efce0071f60 2024-03-23T11:18:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efce7800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce00cc750 con 0x7efce0144320 2024-03-23T11:18:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efce7800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce00cca30 con 0x7efce00cff10 2024-03-23T11:18:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efce4e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efce0144320 0x7efce00cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efcdfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce00cff10 0x7efce0071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:54.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efce4e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efce0144320 0x7efce00cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52658/0 (socket says 172.21.15.62:52658) 2024-03-23T11:18:54.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efcdfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce00cff10 0x7efce0071a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:57420/0 (socket says 172.21.15.62:57420) 2024-03-23T11:18:54.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efce4e00640 1 -- 172.21.15.62:0/3149924738 learned_addr learned my addr 172.21.15.62:0/3149924738 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:54.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efce4e00640 1 -- 172.21.15.62:0/3149924738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce00cff10 msgr2=0x7efce0071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:54.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efce4e00640 1 --2- 172.21.15.62:0/3149924738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce00cff10 0x7efce0071a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:54.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efce4e00640 1 -- 172.21.15.62:0/3149924738 >> v1:172.21.15.120:6789/0 conn(0x7efce0071f60 legacy=0x7efce006eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:54.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.979+0000 7efce4e00640 1 -- 172.21.15.62:0/3149924738 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce00ccd10 con 0x7efce0144320 2024-03-23T11:18:54.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.981+0000 7efce4e00640 1 --2- 172.21.15.62:0/3149924738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efce0144320 0x7efce00cc050 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009e50 tx=0x7efcd002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f233e1788f69c88 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:54.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.981+0000 7efcdf400640 1 -- 172.21.15.62:0/3149924738 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd002cb00 con 0x7efce0144320 2024-03-23T11:18:54.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.981+0000 7efcdf400640 1 -- 172.21.15.62:0/3149924738 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd002cc60 con 0x7efce0144320 2024-03-23T11:18:54.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.981+0000 7efcdf400640 1 -- 172.21.15.62:0/3149924738 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd00343f0 con 0x7efce0144320 2024-03-23T11:18:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.982+0000 7efce7800640 1 -- 172.21.15.62:0/3149924738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efce0144320 msgr2=0x7efce00cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.982+0000 7efce7800640 1 --2- 172.21.15.62:0/3149924738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efce0144320 0x7efce00cc050 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009e50 tx=0x7efcd002bf60 comp rx=0 tx=0).stop 2024-03-23T11:18:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.982+0000 7efce7800640 1 -- 172.21.15.62:0/3149924738 shutdown_connections 2024-03-23T11:18:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.982+0000 7efce7800640 1 --2- 172.21.15.62:0/3149924738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce00cff10 0x7efce0071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.982+0000 7efce7800640 1 --2- 172.21.15.62:0/3149924738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efce0144320 0x7efce00cc050 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.982+0000 7efce7800640 1 -- 172.21.15.62:0/3149924738 >> 172.21.15.62:0/3149924738 conn(0x7efce006d450 msgr2=0x7efce006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:54.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.982+0000 7efce7800640 1 -- 172.21.15.62:0/3149924738 shutdown_connections 2024-03-23T11:18:54.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.982+0000 7efce7800640 1 -- 172.21.15.62:0/3149924738 wait complete. 2024-03-23T11:18:54.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.984+0000 7efce7800640 1 Processor -- start 2024-03-23T11:18:54.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.984+0000 7efce7800640 1 -- start start 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.985+0000 7efce7800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce0071f60 0x7efce0133c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.985+0000 7efce7800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efce00cff10 0x7efce00b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.985+0000 7efce7800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efce0144320 0x7efce01422b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.985+0000 7efce7800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce00b1df0 con 0x7efce00cff10 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.985+0000 7efce7800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce0130810 con 0x7efce0144320 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.985+0000 7efce7800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce0130a80 con 0x7efce0071f60 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.985+0000 7efcdfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efce00cff10 0x7efce00b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.985+0000 7efcdfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efce00cff10 0x7efce00b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52670/0 (socket says 172.21.15.62:52670) 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.985+0000 7efcdfe00640 1 -- 172.21.15.62:0/600822247 learned_addr learned my addr 172.21.15.62:0/600822247 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.985+0000 7efce4e00640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce0071f60 0x7efce0133c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.985+0000 7efce5800640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efce0144320 0x7efce01422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.986+0000 7efcdfe00640 1 -- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce0071f60 msgr2=0x7efce0133c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:54.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.986+0000 7efcdfe00640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce0071f60 0x7efce0133c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:54.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.986+0000 7efcdfe00640 1 -- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efce0144320 msgr2=0x7efce01422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:54.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.986+0000 7efcdfe00640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efce0144320 0x7efce01422b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:54.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.986+0000 7efcdfe00640 1 -- 172.21.15.62:0/600822247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd0009b30 con 0x7efce00cff10 2024-03-23T11:18:54.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.986+0000 7efcdfe00640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efce00cff10 0x7efce00b18b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efcc800e9b0 tx=0x7efcc800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:54.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.986+0000 7efcdcc00640 1 -- 172.21.15.62:0/600822247 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc8009670 con 0x7efce00cff10 2024-03-23T11:18:54.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.986+0000 7efcdcc00640 1 -- 172.21.15.62:0/600822247 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc800c9a0 con 0x7efce00cff10 2024-03-23T11:18:54.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.987+0000 7efcdcc00640 1 -- 172.21.15.62:0/600822247 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc800cd50 con 0x7efce00cff10 2024-03-23T11:18:54.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.987+0000 7efce7800640 1 -- 172.21.15.62:0/600822247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce0130d60 con 0x7efce00cff10 2024-03-23T11:18:54.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.987+0000 7efce7800640 1 -- 172.21.15.62:0/600822247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce0131320 con 0x7efce00cff10 2024-03-23T11:18:54.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.988+0000 7efce7800640 1 -- 172.21.15.62:0/600822247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce00672c0 con 0x7efce00cff10 2024-03-23T11:18:54.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.989+0000 7efcdcc00640 1 -- 172.21.15.62:0/600822247 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7efcc8013470 con 0x7efce00cff10 2024-03-23T11:18:54.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.990+0000 7efcdcc00640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efcbc06a7d0 0x7efcbc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:54.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.991+0000 7efcdcc00640 1 -- 172.21.15.62:0/600822247 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7efcc8088df0 con 0x7efce00cff10 2024-03-23T11:18:54.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.991+0000 7efce4e00640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efcbc06a7d0 0x7efcbc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:54.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.992+0000 7efce4e00640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efcbc06a7d0 0x7efcbc06cc90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efcd0035000 tx=0x7efcd0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:54.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:54.996+0000 7efcdcc00640 1 -- 172.21.15.62:0/600822247 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efcc8054400 con 0x7efce00cff10 2024-03-23T11:18:55.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.196+0000 7efce7800640 1 -- 172.21.15.62:0/600822247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7efce006a860 con 0x7efce00cff10 2024-03-23T11:18:55.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.744+0000 7efcdcc00640 1 -- 172.21.15.62:0/600822247 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7efcc8057a20 con 0x7efce00cff10 2024-03-23T11:18:55.745 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' created 2024-03-23T11:18:55.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.752+0000 7efce7800640 1 -- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efcbc06a7d0 msgr2=0x7efcbc06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:55.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.752+0000 7efce7800640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efcbc06a7d0 0x7efcbc06cc90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efcd0035000 tx=0x7efcd0030000 comp rx=0 tx=0).stop 2024-03-23T11:18:55.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.752+0000 7efce7800640 1 -- 172.21.15.62:0/600822247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efce00cff10 msgr2=0x7efce00b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:55.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.752+0000 7efce7800640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efce00cff10 0x7efce00b18b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efcc800e9b0 tx=0x7efcc800ee70 comp rx=0 tx=0).stop 2024-03-23T11:18:55.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.753+0000 7efce7800640 1 -- 172.21.15.62:0/600822247 shutdown_connections 2024-03-23T11:18:55.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.753+0000 7efce7800640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efcbc06a7d0 0x7efcbc06cc90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:55.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.753+0000 7efce7800640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efce00cff10 0x7efce00b18b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:55.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.753+0000 7efce7800640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efce0144320 0x7efce01422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:55.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.753+0000 7efce7800640 1 --2- 172.21.15.62:0/600822247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efce0071f60 0x7efce0133c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:55.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.753+0000 7efce7800640 1 -- 172.21.15.62:0/600822247 >> 172.21.15.62:0/600822247 conn(0x7efce006d450 msgr2=0x7efce006eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:55.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.754+0000 7efce7800640 1 -- 172.21.15.62:0/600822247 shutdown_connections 2024-03-23T11:18:55.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.754+0000 7efce7800640 1 -- 172.21.15.62:0/600822247 wait complete. 2024-03-23T11:18:55.788 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T11:18:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.974+0000 7f57c7000640 1 Processor -- start 2024-03-23T11:18:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.974+0000 7f57c7000640 1 -- start start 2024-03-23T11:18:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.975+0000 7f57c7000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 0x7f57c0161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.975+0000 7f57c7000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f57c006c300 0x7f57c006c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.975+0000 7f57c7000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c00ae0c0 con 0x7f57c00b2f90 2024-03-23T11:18:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.975+0000 7f57c7000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c00b2740 con 0x7f57c006c300 2024-03-23T11:18:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.975+0000 7f57c7000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c01623b0 con 0x7f57c00b15e0 2024-03-23T11:18:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.976+0000 7f57bf400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 0x7f57c0161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.976+0000 7f57c5000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f57c006c300 0x7f57c006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.976+0000 7f57bf400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 0x7f57c0161e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:57452/0 (socket says 172.21.15.62:57452) 2024-03-23T11:18:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.976+0000 7f57bf400640 1 -- 172.21.15.62:0/1492281621 learned_addr learned my addr 172.21.15.62:0/1492281621 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.977+0000 7f57c5000640 1 -- 172.21.15.62:0/1492281621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 msgr2=0x7f57c0161e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.977+0000 7f57c5000640 1 --2- 172.21.15.62:0/1492281621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 0x7f57c0161e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.977+0000 7f57c5000640 1 -- 172.21.15.62:0/1492281621 >> v1:172.21.15.120:6789/0 conn(0x7f57c00b2f90 legacy=0x7f57c00b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.977+0000 7f57c5000640 1 -- 172.21.15.62:0/1492281621 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c00b1100 con 0x7f57c006c300 2024-03-23T11:18:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.977+0000 7f57bf400640 1 --2- 172.21.15.62:0/1492281621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 0x7f57c0161e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:18:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.978+0000 7f57c5000640 1 --2- 172.21.15.62:0/1492281621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57c006c300 0x7f57c006c790 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009d90 tx=0x7f57b802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b9c18da5bd75ca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.979+0000 7f57bea00640 1 -- 172.21.15.62:0/1492281621 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b802cc60 con 0x7f57c006c300 2024-03-23T11:18:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.979+0000 7f57bea00640 1 -- 172.21.15.62:0/1492281621 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b802cdc0 con 0x7f57c006c300 2024-03-23T11:18:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.979+0000 7f57bea00640 1 -- 172.21.15.62:0/1492281621 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8034660 con 0x7f57c006c300 2024-03-23T11:18:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.979+0000 7f57c7000640 1 -- 172.21.15.62:0/1492281621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57c006c300 msgr2=0x7f57c006c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.979+0000 7f57c7000640 1 --2- 172.21.15.62:0/1492281621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57c006c300 0x7f57c006c790 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009d90 tx=0x7f57b802beb0 comp rx=0 tx=0).stop 2024-03-23T11:18:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.980+0000 7f57c7000640 1 -- 172.21.15.62:0/1492281621 shutdown_connections 2024-03-23T11:18:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.980+0000 7f57c7000640 1 --2- 172.21.15.62:0/1492281621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57c006c300 0x7f57c006c790 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.980+0000 7f57c7000640 1 --2- 172.21.15.62:0/1492281621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 0x7f57c0161e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.980+0000 7f57c7000640 1 -- 172.21.15.62:0/1492281621 >> 172.21.15.62:0/1492281621 conn(0x7f57c00b02e0 msgr2=0x7f57c00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.980+0000 7f57c7000640 1 -- 172.21.15.62:0/1492281621 shutdown_connections 2024-03-23T11:18:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.980+0000 7f57c7000640 1 -- 172.21.15.62:0/1492281621 wait complete. 2024-03-23T11:18:55.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.982+0000 7f57c7000640 1 Processor -- start 2024-03-23T11:18:55.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.982+0000 7f57c7000640 1 -- start start 2024-03-23T11:18:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.983+0000 7f57c7000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57c006c300 0x7f57c0155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.983+0000 7f57c7000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 0x7f57c015adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.983+0000 7f57c7000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c00b2f90 0x7f57c01ec840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.983+0000 7f57c7000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c015e470 con 0x7f57c00b2f90 2024-03-23T11:18:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.983+0000 7f57c7000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c015a7a0 con 0x7f57c006c300 2024-03-23T11:18:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.983+0000 7f57c7000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c015e710 con 0x7f57c00b15e0 2024-03-23T11:18:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.984+0000 7f57c5000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c00b2f90 0x7f57c01ec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.984+0000 7f57c5000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c00b2f90 0x7f57c01ec840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52676/0 (socket says 172.21.15.62:52676) 2024-03-23T11:18:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.984+0000 7f57c5000640 1 -- 172.21.15.62:0/3426501835 learned_addr learned my addr 172.21.15.62:0/3426501835 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.984+0000 7f57bfe00640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57c006c300 0x7f57c0155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.984+0000 7f57bf400640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 0x7f57c015adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:55.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.984+0000 7f57c5000640 1 -- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 msgr2=0x7f57c015adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:55.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.984+0000 7f57c5000640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 0x7f57c015adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:55.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.984+0000 7f57c5000640 1 -- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57c006c300 msgr2=0x7f57c0155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:55.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.984+0000 7f57c5000640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57c006c300 0x7f57c0155890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:55.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.984+0000 7f57c5000640 1 -- 172.21.15.62:0/3426501835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a80094d0 con 0x7f57c00b2f90 2024-03-23T11:18:55.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.985+0000 7f57c5000640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c00b2f90 0x7f57c01ec840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009d90 tx=0x7f57b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:55.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.985+0000 7f57b7e00640 1 -- 172.21.15.62:0/3426501835 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8040650 con 0x7f57c00b2f90 2024-03-23T11:18:55.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.985+0000 7f57b7e00640 1 -- 172.21.15.62:0/3426501835 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b80407b0 con 0x7f57c00b2f90 2024-03-23T11:18:55.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.985+0000 7f57c7000640 1 -- 172.21.15.62:0/3426501835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b8009a70 con 0x7f57c00b2f90 2024-03-23T11:18:55.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.985+0000 7f57b7e00640 1 -- 172.21.15.62:0/3426501835 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8040b60 con 0x7f57c00b2f90 2024-03-23T11:18:55.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.985+0000 7f57c7000640 1 -- 172.21.15.62:0/3426501835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c015ecd0 con 0x7f57c00b2f90 2024-03-23T11:18:55.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.986+0000 7f57c7000640 1 -- 172.21.15.62:0/3426501835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c0073860 con 0x7f57c00b2f90 2024-03-23T11:18:55.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.987+0000 7f57b7e00640 1 -- 172.21.15.62:0/3426501835 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f57b8040cc0 con 0x7f57c00b2f90 2024-03-23T11:18:55.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.988+0000 7f57b7e00640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f579c06a7a0 0x7f579c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:55.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.988+0000 7f57b7e00640 1 -- 172.21.15.62:0/3426501835 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f57b80ab990 con 0x7f57c00b2f90 2024-03-23T11:18:55.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.989+0000 7f57bfe00640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f579c06a7a0 0x7f579c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:55.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.990+0000 7f57bfe00640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f579c06a7a0 0x7f579c06cc60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57b0000f60 tx=0x7f57b0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:55.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:55.993+0000 7f57b7e00640 1 -- 172.21.15.62:0/3426501835 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f57b8076fd0 con 0x7f57c00b2f90 2024-03-23T11:18:56.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.231+0000 7f57c7000640 1 -- 172.21.15.62:0/3426501835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f57c00b1df0 con 0x7f57c00b2f90 2024-03-23T11:18:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.242+0000 7f57b7e00640 1 -- 172.21.15.62:0/3426501835 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f57c00b1df0 con 0x7f57c00b2f90 2024-03-23T11:18:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:new fs with metadata pool 2 and data pool 3 2024-03-23T11:18:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.247+0000 7f57c7000640 1 -- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f579c06a7a0 msgr2=0x7f579c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.247+0000 7f57c7000640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f579c06a7a0 0x7f579c06cc60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57b0000f60 tx=0x7f57b0009040 comp rx=0 tx=0).stop 2024-03-23T11:18:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.247+0000 7f57c7000640 1 -- 172.21.15.62:0/3426501835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c00b2f90 msgr2=0x7f57c01ec840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.247+0000 7f57c7000640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c00b2f90 0x7f57c01ec840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009d90 tx=0x7f57b8004a40 comp rx=0 tx=0).stop 2024-03-23T11:18:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.247+0000 7f57c7000640 1 -- 172.21.15.62:0/3426501835 shutdown_connections 2024-03-23T11:18:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.247+0000 7f57c7000640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f579c06a7a0 0x7f579c06cc60 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57b0000f60 tx=0x7f57b0009040 comp rx=0 tx=0).stop 2024-03-23T11:18:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.247+0000 7f57c7000640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57c00b2f90 0x7f57c01ec840 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.247+0000 7f57c7000640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57c00b15e0 0x7f57c015adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.247+0000 7f57c7000640 1 --2- 172.21.15.62:0/3426501835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57c006c300 0x7f57c0155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.248+0000 7f57c7000640 1 -- 172.21.15.62:0/3426501835 >> 172.21.15.62:0/3426501835 conn(0x7f57c00b02e0 msgr2=0x7f57c00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.248+0000 7f57c7000640 1 -- 172.21.15.62:0/3426501835 shutdown_connections 2024-03-23T11:18:56.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.249+0000 7f57c7000640 1 -- 172.21.15.62:0/3426501835 wait complete. 2024-03-23T11:18:56.278 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:18:56.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.450+0000 7f941d400640 1 Processor -- start 2024-03-23T11:18:56.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.450+0000 7f941d400640 1 -- start start 2024-03-23T11:18:56.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f941d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f941806c300 0x7f94180ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f941d400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f94180cafa0 0x7f94180cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f941d400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94180cbc60 con 0x7f941806c6e0 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f941d400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94180cbf40 con 0x7f941806c300 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f941d400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94180cc220 con 0x7f94180cafa0 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f9416800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f941806c300 0x7f94180ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f9416800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f941806c300 0x7f94180ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52678/0 (socket says 172.21.15.62:52678) 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f9416800640 1 -- 172.21.15.62:0/3222585608 learned_addr learned my addr 172.21.15.62:0/3222585608 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f9416800640 1 -- 172.21.15.62:0/3222585608 >> v2:172.21.15.120:3300/0 conn(0x7f94180cafa0 msgr2=0x7f94180cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f9415e00640 1 --2- 172.21.15.62:0/3222585608 >> v2:172.21.15.120:3300/0 conn(0x7f94180cafa0 0x7f94180cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f9416800640 1 --2- 172.21.15.62:0/3222585608 >> v2:172.21.15.120:3300/0 conn(0x7f94180cafa0 0x7f94180cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.452+0000 7f9416800640 1 -- 172.21.15.62:0/3222585608 >> v1:172.21.15.62:6789/0 conn(0x7f941806c6e0 legacy=0x7f941806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.453+0000 7f9416800640 1 -- 172.21.15.62:0/3222585608 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94180cc500 con 0x7f941806c300 2024-03-23T11:18:56.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.453+0000 7f9415400640 1 -- 172.21.15.62:0/3222585608 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272324877 0 0) 0x7f94180cbc60 con 0x7f941806c6e0 2024-03-23T11:18:56.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.454+0000 7f9416800640 1 --2- 172.21.15.62:0/3222585608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f941806c300 0x7f94180ccd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f941806e370 tx=0x7f940802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad33edfba3cd3c0d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:56.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.454+0000 7f9415400640 1 -- 172.21.15.62:0/3222585608 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408030860 con 0x7f941806c300 2024-03-23T11:18:56.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.454+0000 7f9415400640 1 -- 172.21.15.62:0/3222585608 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94080309c0 con 0x7f941806c300 2024-03-23T11:18:56.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.454+0000 7f9415400640 1 -- 172.21.15.62:0/3222585608 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408030d50 con 0x7f941806c300 2024-03-23T11:18:56.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.454+0000 7f941d400640 1 -- 172.21.15.62:0/3222585608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f941806c300 msgr2=0x7f94180ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.454+0000 7f941d400640 1 --2- 172.21.15.62:0/3222585608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f941806c300 0x7f94180ccd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f941806e370 tx=0x7f940802b9a0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.454+0000 7f941d400640 1 -- 172.21.15.62:0/3222585608 shutdown_connections 2024-03-23T11:18:56.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.454+0000 7f941d400640 1 --2- 172.21.15.62:0/3222585608 >> v2:172.21.15.120:3300/0 conn(0x7f94180cafa0 0x7f94180cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.454+0000 7f941d400640 1 --2- 172.21.15.62:0/3222585608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f941806c300 0x7f94180ccd00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.455+0000 7f941d400640 1 -- 172.21.15.62:0/3222585608 >> 172.21.15.62:0/3222585608 conn(0x7f94180b02e0 msgr2=0x7f94180b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:56.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.455+0000 7f941d400640 1 -- 172.21.15.62:0/3222585608 shutdown_connections 2024-03-23T11:18:56.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.455+0000 7f941d400640 1 -- 172.21.15.62:0/3222585608 wait complete. 2024-03-23T11:18:56.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.456+0000 7f941d400640 1 Processor -- start 2024-03-23T11:18:56.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.456+0000 7f941d400640 1 -- start start 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.457+0000 7f941d400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f941806c300 0x7f9418155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.457+0000 7f941d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f941806c6e0 0x7f9418156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.457+0000 7f941d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94180cafa0 0x7f9418158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.457+0000 7f941d400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94181f6160 con 0x7f94180cafa0 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.457+0000 7f941d400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94181f6580 con 0x7f941806c300 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.457+0000 7f941d400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94181f57c0 con 0x7f941806c6e0 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.458+0000 7f9417200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94180cafa0 0x7f9418158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.458+0000 7f9417200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94180cafa0 0x7f9418158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52686/0 (socket says 172.21.15.62:52686) 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.458+0000 7f9417200640 1 -- 172.21.15.62:0/2935827217 learned_addr learned my addr 172.21.15.62:0/2935827217 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.458+0000 7f9415e00640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f941806c6e0 0x7f9418156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.458+0000 7f9416800640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f941806c300 0x7f9418155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.458+0000 7f9417200640 1 -- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f941806c6e0 msgr2=0x7f9418156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.458+0000 7f9417200640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f941806c6e0 0x7f9418156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.458+0000 7f9417200640 1 -- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f941806c300 msgr2=0x7f9418155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.458+0000 7f9417200640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f941806c300 0x7f9418155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.458+0000 7f9417200640 1 -- 172.21.15.62:0/2935827217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94080094d0 con 0x7f94180cafa0 2024-03-23T11:18:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.458+0000 7f9417200640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94180cafa0 0x7f9418158cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9410001560 tx=0x7f941000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.459+0000 7f940ea00640 1 -- 172.21.15.62:0/2935827217 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410003da0 con 0x7f94180cafa0 2024-03-23T11:18:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.459+0000 7f940ea00640 1 -- 172.21.15.62:0/2935827217 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9410005070 con 0x7f94180cafa0 2024-03-23T11:18:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.459+0000 7f940ea00640 1 -- 172.21.15.62:0/2935827217 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410005420 con 0x7f94180cafa0 2024-03-23T11:18:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.459+0000 7f941d400640 1 -- 172.21.15.62:0/2935827217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94181f5aa0 con 0x7f94180cafa0 2024-03-23T11:18:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.459+0000 7f941d400640 1 -- 172.21.15.62:0/2935827217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9418156100 con 0x7f94180cafa0 2024-03-23T11:18:56.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.460+0000 7f941d400640 1 -- 172.21.15.62:0/2935827217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94181f62a0 con 0x7f94180cafa0 2024-03-23T11:18:56.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.461+0000 7f940ea00640 1 -- 172.21.15.62:0/2935827217 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9410005580 con 0x7f94180cafa0 2024-03-23T11:18:56.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.462+0000 7f940ea00640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f806a7a0 0x7f93f806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:56.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.462+0000 7f940ea00640 1 -- 172.21.15.62:0/2935827217 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9410088680 con 0x7f94180cafa0 2024-03-23T11:18:56.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.462+0000 7f9416800640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f806a7a0 0x7f93f806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:56.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.464+0000 7f9416800640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f806a7a0 0x7f93f806cc60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9408036000 tx=0x7f940803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:56.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.466+0000 7f940ea00640 1 -- 172.21.15.62:0/2935827217 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f94181f62a0 con 0x7f94180cafa0 2024-03-23T11:18:56.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.662+0000 7f941d400640 1 -- 172.21.15.62:0/2935827217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94181f62a0 con 0x7f94180cafa0 2024-03-23T11:18:56.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.663+0000 7f940ea00640 1 -- 172.21.15.62:0/2935827217 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16050 (secure 0 0 0) 0x7f94181f62a0 con 0x7f94180cafa0 2024-03-23T11:18:56.664 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:18:56.664 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:56.233086+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:18:56.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.665+0000 7f941d400640 1 -- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f806a7a0 msgr2=0x7f93f806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.665+0000 7f941d400640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f806a7a0 0x7f93f806cc60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9408036000 tx=0x7f940803b040 comp rx=0 tx=0).stop 2024-03-23T11:18:56.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.666+0000 7f941d400640 1 -- 172.21.15.62:0/2935827217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94180cafa0 msgr2=0x7f9418158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.666+0000 7f941d400640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94180cafa0 0x7f9418158cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9410001560 tx=0x7f941000cc70 comp rx=0 tx=0).stop 2024-03-23T11:18:56.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.666+0000 7f941d400640 1 -- 172.21.15.62:0/2935827217 shutdown_connections 2024-03-23T11:18:56.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.666+0000 7f941d400640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f806a7a0 0x7f93f806cc60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.666+0000 7f941d400640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94180cafa0 0x7f9418158cd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.666+0000 7f941d400640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f941806c6e0 0x7f9418156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.666+0000 7f941d400640 1 --2- 172.21.15.62:0/2935827217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f941806c300 0x7f9418155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.666+0000 7f941d400640 1 -- 172.21.15.62:0/2935827217 >> 172.21.15.62:0/2935827217 conn(0x7f94180b02e0 msgr2=0x7f9418072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:56.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.666+0000 7f941d400640 1 -- 172.21.15.62:0/2935827217 shutdown_connections 2024-03-23T11:18:56.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.666+0000 7f941d400640 1 -- 172.21.15.62:0/2935827217 wait complete. 2024-03-23T11:18:56.685 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:18:56.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.897+0000 7fd915800640 1 Processor -- start 2024-03-23T11:18:56.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.897+0000 7fd915800640 1 -- start start 2024-03-23T11:18:56.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.899+0000 7fd915800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd9100cafa0 0x7fd9100cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:56.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.899+0000 7fd915800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9100cbc60 con 0x7fd91006c6e0 2024-03-23T11:18:56.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.899+0000 7fd915800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9100cbf40 con 0x7fd91006c300 2024-03-23T11:18:56.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.899+0000 7fd915800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9100cd410 con 0x7fd9100cafa0 2024-03-23T11:18:56.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.899+0000 7fd907e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd9100cafa0 0x7fd9100cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:56.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.899+0000 7fd907e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd9100cafa0 0x7fd9100cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52700/0 (socket says 172.21.15.62:52700) 2024-03-23T11:18:56.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.899+0000 7fd907e00640 1 -- 172.21.15.62:0/1161782915 learned_addr learned my addr 172.21.15.62:0/1161782915 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:56.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.899+0000 7fd90f400640 1 --1- 172.21.15.62:0/1161782915 >> v1:172.21.15.62:6789/0 conn(0x7fd91006c6e0 0x7fd91006cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:38230/0 (socket says 172.21.15.62:38230) 2024-03-23T11:18:56.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.899+0000 7fd907e00640 1 -- 172.21.15.62:0/1161782915 >> v1:172.21.15.120:6789/0 conn(0x7fd91006c300 legacy=0x7fd9100ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:56.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.899+0000 7fd907e00640 1 -- 172.21.15.62:0/1161782915 >> v1:172.21.15.62:6789/0 conn(0x7fd91006c6e0 legacy=0x7fd91006cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.899+0000 7fd907e00640 1 -- 172.21.15.62:0/1161782915 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9100cc220 con 0x7fd9100cafa0 2024-03-23T11:18:56.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.900+0000 7fd907e00640 1 --2- 172.21.15.62:0/1161782915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9100cafa0 0x7fd9100cb430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009870 tx=0x7fd8f802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d88ce8d6b3f7388 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:56.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.901+0000 7fd90e000640 1 -- 172.21.15.62:0/1161782915 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8030860 con 0x7fd9100cafa0 2024-03-23T11:18:56.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.901+0000 7fd90e000640 1 -- 172.21.15.62:0/1161782915 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f80309c0 con 0x7fd9100cafa0 2024-03-23T11:18:56.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.901+0000 7fd90e000640 1 -- 172.21.15.62:0/1161782915 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8030d50 con 0x7fd9100cafa0 2024-03-23T11:18:56.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.901+0000 7fd915800640 1 -- 172.21.15.62:0/1161782915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9100cafa0 msgr2=0x7fd9100cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.901+0000 7fd915800640 1 --2- 172.21.15.62:0/1161782915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9100cafa0 0x7fd9100cb430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009870 tx=0x7fd8f802b9a0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.901+0000 7fd915800640 1 -- 172.21.15.62:0/1161782915 shutdown_connections 2024-03-23T11:18:56.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.901+0000 7fd915800640 1 --2- 172.21.15.62:0/1161782915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9100cafa0 0x7fd9100cb430 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.901+0000 7fd915800640 1 -- 172.21.15.62:0/1161782915 >> 172.21.15.62:0/1161782915 conn(0x7fd9100b02e0 msgr2=0x7fd9100b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:56.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.901+0000 7fd915800640 1 -- 172.21.15.62:0/1161782915 shutdown_connections 2024-03-23T11:18:56.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.902+0000 7fd915800640 1 -- 172.21.15.62:0/1161782915 wait complete. 2024-03-23T11:18:56.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.903+0000 7fd915800640 1 Processor -- start 2024-03-23T11:18:56.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.903+0000 7fd915800640 1 -- start start 2024-03-23T11:18:56.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.904+0000 7fd915800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91006c300 0x7fd91013ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.904+0000 7fd915800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd91006c6e0 0x7fd9100b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.904+0000 7fd915800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9100cafa0 0x7fd91013d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.904+0000 7fd915800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd910150e00 con 0x7fd9100cafa0 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.904+0000 7fd915800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd910151660 con 0x7fd91006c6e0 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.904+0000 7fd915800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91013b320 con 0x7fd91006c300 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.905+0000 7fd90f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9100cafa0 0x7fd91013d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.905+0000 7fd90f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9100cafa0 0x7fd91013d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52706/0 (socket says 172.21.15.62:52706) 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.905+0000 7fd90f400640 1 -- 172.21.15.62:0/640055908 learned_addr learned my addr 172.21.15.62:0/640055908 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.905+0000 7fd90f400640 1 -- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91006c300 msgr2=0x7fd91013ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.905+0000 7fd90ea00640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91006c300 0x7fd91013ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.905+0000 7fd90f400640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91006c300 0x7fd91013ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.905+0000 7fd90f400640 1 -- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd91006c6e0 msgr2=0x7fd9100b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:56.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.905+0000 7fd90f400640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd91006c6e0 0x7fd9100b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:56.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.905+0000 7fd90f400640 1 -- 172.21.15.62:0/640055908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f80094d0 con 0x7fd9100cafa0 2024-03-23T11:18:56.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.905+0000 7fd90f400640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9100cafa0 0x7fd91013d760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd90800c780 tx=0x7fd90800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:56.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.906+0000 7fd906a00640 1 -- 172.21.15.62:0/640055908 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90800ce90 con 0x7fd9100cafa0 2024-03-23T11:18:56.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.906+0000 7fd906a00640 1 -- 172.21.15.62:0/640055908 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90800a640 con 0x7fd9100cafa0 2024-03-23T11:18:56.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.906+0000 7fd906a00640 1 -- 172.21.15.62:0/640055908 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90800a9f0 con 0x7fd9100cafa0 2024-03-23T11:18:56.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.906+0000 7fd915800640 1 -- 172.21.15.62:0/640055908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9100b3510 con 0x7fd9100cafa0 2024-03-23T11:18:56.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.906+0000 7fd915800640 1 -- 172.21.15.62:0/640055908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9101512f0 con 0x7fd9100cafa0 2024-03-23T11:18:56.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.907+0000 7fd915800640 1 -- 172.21.15.62:0/640055908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd910066c90 con 0x7fd9100cafa0 2024-03-23T11:18:56.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.908+0000 7fd906a00640 1 -- 172.21.15.62:0/640055908 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd90800ab50 con 0x7fd9100cafa0 2024-03-23T11:18:56.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.909+0000 7fd906a00640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8ec06a7d0 0x7fd8ec06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:56.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.909+0000 7fd906a00640 1 -- 172.21.15.62:0/640055908 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9080885a0 con 0x7fd9100cafa0 2024-03-23T11:18:56.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.910+0000 7fd90ea00640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8ec06a7d0 0x7fd8ec06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:56.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.911+0000 7fd90ea00640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8ec06a7d0 0x7fd8ec06cc90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd900003d90 tx=0x7fd900009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:56.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:56.914+0000 7fd906a00640 1 -- 172.21.15.62:0/640055908 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9080539d0 con 0x7fd9100cafa0 2024-03-23T11:18:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.106+0000 7fd915800640 1 -- 172.21.15.62:0/640055908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9100af300 con 0x7fd9100cafa0 2024-03-23T11:18:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.109+0000 7fd906a00640 1 -- 172.21.15.62:0/640055908 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16050 (secure 0 0 0) 0x7fd908056ff0 con 0x7fd9100cafa0 2024-03-23T11:18:57.110 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:18:57.110 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:56.233086+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:18:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 -- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8ec06a7d0 msgr2=0x7fd8ec06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8ec06a7d0 0x7fd8ec06cc90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd900003d90 tx=0x7fd900009040 comp rx=0 tx=0).stop 2024-03-23T11:18:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 -- 172.21.15.62:0/640055908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9100cafa0 msgr2=0x7fd91013d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9100cafa0 0x7fd91013d760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd90800c780 tx=0x7fd90800cc40 comp rx=0 tx=0).stop 2024-03-23T11:18:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 -- 172.21.15.62:0/640055908 shutdown_connections 2024-03-23T11:18:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8ec06a7d0 0x7fd8ec06cc90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9100cafa0 0x7fd91013d760 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd91006c6e0 0x7fd9100b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 --2- 172.21.15.62:0/640055908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd91006c300 0x7fd91013ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 -- 172.21.15.62:0/640055908 >> 172.21.15.62:0/640055908 conn(0x7fd9100b02e0 msgr2=0x7fd9100afb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 -- 172.21.15.62:0/640055908 shutdown_connections 2024-03-23T11:18:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.112+0000 7fd915800640 1 -- 172.21.15.62:0/640055908 wait complete. 2024-03-23T11:18:57.129 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T11:18:57.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.337+0000 7f68df000640 1 Processor -- start 2024-03-23T11:18:57.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.337+0000 7f68df000640 1 -- start start 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.339+0000 7f68df000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f68d8070090 0x7f68d8070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.339+0000 7f68df000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f68d80b2820 0x7f68d80b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.339+0000 7f68df000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d80b31b0 con 0x7f68d8143dd0 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.339+0000 7f68df000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d8070a60 con 0x7f68d80b2820 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.339+0000 7f68df000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d80cd140 con 0x7f68d8070090 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.339+0000 7f68de600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f68d80b2820 0x7f68d80b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.339+0000 7f68de600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f68d80b2820 0x7f68d80b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37626/0 (socket says 172.21.15.62:37626) 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.339+0000 7f68de600640 1 -- 172.21.15.62:0/1738542063 learned_addr learned my addr 172.21.15.62:0/1738542063 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.339+0000 7f68dd200640 1 --2- 172.21.15.62:0/1738542063 >> v2:172.21.15.120:3300/0 conn(0x7f68d8070090 0x7f68d8070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.340+0000 7f68de600640 1 -- 172.21.15.62:0/1738542063 >> v2:172.21.15.120:3300/0 conn(0x7f68d8070090 msgr2=0x7f68d8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.340+0000 7f68de600640 1 --2- 172.21.15.62:0/1738542063 >> v2:172.21.15.120:3300/0 conn(0x7f68d8070090 0x7f68d8070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.340+0000 7f68de600640 1 -- 172.21.15.62:0/1738542063 >> v1:172.21.15.120:6789/0 conn(0x7f68d8143dd0 legacy=0x7f68d815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:57.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.340+0000 7f68de600640 1 -- 172.21.15.62:0/1738542063 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d80cd3e0 con 0x7f68d80b2820 2024-03-23T11:18:57.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.341+0000 7f68de600640 1 --2- 172.21.15.62:0/1738542063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68d80b2820 0x7f68d80b2c70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f68d8071530 tx=0x7f68d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33e4e6cf227dbae server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:57.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.341+0000 7f68d7e00640 1 -- 172.21.15.62:0/1738542063 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0031870 con 0x7f68d80b2820 2024-03-23T11:18:57.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.341+0000 7f68d7e00640 1 -- 172.21.15.62:0/1738542063 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d00319d0 con 0x7f68d80b2820 2024-03-23T11:18:57.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.341+0000 7f68d7e00640 1 -- 172.21.15.62:0/1738542063 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0031d60 con 0x7f68d80b2820 2024-03-23T11:18:57.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.342+0000 7f68df000640 1 -- 172.21.15.62:0/1738542063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68d80b2820 msgr2=0x7f68d80b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:57.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.342+0000 7f68df000640 1 --2- 172.21.15.62:0/1738542063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68d80b2820 0x7f68d80b2c70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f68d8071530 tx=0x7f68d002b910 comp rx=0 tx=0).stop 2024-03-23T11:18:57.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.342+0000 7f68df000640 1 -- 172.21.15.62:0/1738542063 shutdown_connections 2024-03-23T11:18:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.342+0000 7f68df000640 1 --2- 172.21.15.62:0/1738542063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68d80b2820 0x7f68d80b2c70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.342+0000 7f68df000640 1 --2- 172.21.15.62:0/1738542063 >> v2:172.21.15.120:3300/0 conn(0x7f68d8070090 0x7f68d8070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.342+0000 7f68df000640 1 -- 172.21.15.62:0/1738542063 >> 172.21.15.62:0/1738542063 conn(0x7f68d8066050 msgr2=0x7f68d8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.342+0000 7f68df000640 1 -- 172.21.15.62:0/1738542063 shutdown_connections 2024-03-23T11:18:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.342+0000 7f68df000640 1 -- 172.21.15.62:0/1738542063 wait complete. 2024-03-23T11:18:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.344+0000 7f68df000640 1 Processor -- start 2024-03-23T11:18:57.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.344+0000 7f68df000640 1 -- start start 2024-03-23T11:18:57.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68df000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68d8070090 0x7f68d8136d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68df000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68d80b2820 0x7f68d806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68df000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68d8143dd0 0x7f68d806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68df000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d815a760 con 0x7f68d8070090 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68df000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d80c4000 con 0x7f68d80b2820 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68df000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d806d660 con 0x7f68d8143dd0 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68ddc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68d8070090 0x7f68d8136d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68ddc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68d8070090 0x7f68d8136d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37642/0 (socket says 172.21.15.62:37642) 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68ddc00640 1 -- 172.21.15.62:0/229092567 learned_addr learned my addr 172.21.15.62:0/229092567 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68ddc00640 1 -- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68d8143dd0 msgr2=0x7f68d806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68ddc00640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68d8143dd0 0x7f68d806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68ddc00640 1 -- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68d80b2820 msgr2=0x7f68d806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68ddc00640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68d80b2820 0x7f68d806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.345+0000 7f68ddc00640 1 -- 172.21.15.62:0/229092567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d00094d0 con 0x7f68d8070090 2024-03-23T11:18:57.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.346+0000 7f68ddc00640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68d8070090 0x7f68d8136d20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00ba80 tx=0x7f68cc00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:57.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.346+0000 7f68d5600640 1 -- 172.21.15.62:0/229092567 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc00cc90 con 0x7f68d8070090 2024-03-23T11:18:57.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.346+0000 7f68d5600640 1 -- 172.21.15.62:0/229092567 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc00cdf0 con 0x7f68d8070090 2024-03-23T11:18:57.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.346+0000 7f68d5600640 1 -- 172.21.15.62:0/229092567 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc014680 con 0x7f68d8070090 2024-03-23T11:18:57.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.346+0000 7f68df000640 1 -- 172.21.15.62:0/229092567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d806d940 con 0x7f68d8070090 2024-03-23T11:18:57.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.346+0000 7f68df000640 1 -- 172.21.15.62:0/229092567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d806df00 con 0x7f68d8070090 2024-03-23T11:18:57.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.347+0000 7f68df000640 1 -- 172.21.15.62:0/229092567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d80cca40 con 0x7f68d8070090 2024-03-23T11:18:57.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.348+0000 7f68d5600640 1 -- 172.21.15.62:0/229092567 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f68cc01d3b0 con 0x7f68d8070090 2024-03-23T11:18:57.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.349+0000 7f68d5600640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f68b806a7a0 0x7f68b806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:57.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.349+0000 7f68d5600640 1 -- 172.21.15.62:0/229092567 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68cc088cc0 con 0x7f68d8070090 2024-03-23T11:18:57.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.350+0000 7f68dd200640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f68b806a7a0 0x7f68b806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:57.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.351+0000 7f68dd200640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f68b806a7a0 0x7f68b806cc60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f68d80c3ab0 tx=0x7f68c4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:57.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.354+0000 7f68d5600640 1 -- 172.21.15.62:0/229092567 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f68cc054170 con 0x7f68d8070090 2024-03-23T11:18:57.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:57.592+0000 7f68df000640 1 -- 172.21.15.62:0/229092567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f68d8066bb0 con 0x7f68d8070090 2024-03-23T11:18:58.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.254+0000 7f68d5600640 1 -- 172.21.15.62:0/229092567 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f68cc0190b0 con 0x7f68d8070090 2024-03-23T11:18:58.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.258+0000 7f68df000640 1 -- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f68b806a7a0 msgr2=0x7f68b806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:58.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.258+0000 7f68df000640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f68b806a7a0 0x7f68b806cc60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f68d80c3ab0 tx=0x7f68c4008040 comp rx=0 tx=0).stop 2024-03-23T11:18:58.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.258+0000 7f68df000640 1 -- 172.21.15.62:0/229092567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68d8070090 msgr2=0x7f68d8136d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:58.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.258+0000 7f68df000640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68d8070090 0x7f68d8136d20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00ba80 tx=0x7f68cc00bf40 comp rx=0 tx=0).stop 2024-03-23T11:18:58.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.259+0000 7f68df000640 1 -- 172.21.15.62:0/229092567 shutdown_connections 2024-03-23T11:18:58.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.259+0000 7f68df000640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f68b806a7a0 0x7f68b806cc60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:58.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.259+0000 7f68df000640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68d8143dd0 0x7f68d806cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:58.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.259+0000 7f68df000640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68d80b2820 0x7f68d806caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:58.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.259+0000 7f68df000640 1 --2- 172.21.15.62:0/229092567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68d8070090 0x7f68d8136d20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:58.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.259+0000 7f68df000640 1 -- 172.21.15.62:0/229092567 >> 172.21.15.62:0/229092567 conn(0x7f68d8066050 msgr2=0x7f68d8143090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:58.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.259+0000 7f68df000640 1 -- 172.21.15.62:0/229092567 shutdown_connections 2024-03-23T11:18:58.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.259+0000 7f68df000640 1 -- 172.21.15.62:0/229092567 wait complete. 2024-03-23T11:18:58.282 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T11:18:58.283 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T11:18:58.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.511+0000 7fc187200640 1 Processor -- start 2024-03-23T11:18:58.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.511+0000 7fc187200640 1 -- start start 2024-03-23T11:18:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.513+0000 7fc187200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc188143dd0 0x7fc18815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.513+0000 7fc187200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc188070090 0x7fc1880704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.513+0000 7fc187200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc1880b2820 0x7fc1880b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.513+0000 7fc187200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc188070a30 con 0x7fc188143dd0 2024-03-23T11:18:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.513+0000 7fc187200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1880cd140 con 0x7fc1880b2820 2024-03-23T11:18:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.513+0000 7fc187200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1880b3180 con 0x7fc188070090 2024-03-23T11:18:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.513+0000 7fc185e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc188143dd0 0x7fc18815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.513+0000 7fc185e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc188143dd0 0x7fc18815abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37656/0 (socket says 172.21.15.62:37656) 2024-03-23T11:18:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.513+0000 7fc185e00640 1 -- 172.21.15.62:0/3146138039 learned_addr learned my addr 172.21.15.62:0/3146138039 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.513+0000 7fc185400640 1 --2- 172.21.15.62:0/3146138039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc188070090 0x7fc1880704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.513+0000 7fc186800640 1 --2- 172.21.15.62:0/3146138039 >> v2:172.21.15.120:3300/0 conn(0x7fc1880b2820 0x7fc1880b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.514+0000 7fc185e00640 1 -- 172.21.15.62:0/3146138039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc188070090 msgr2=0x7fc1880704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.514+0000 7fc185e00640 1 --2- 172.21.15.62:0/3146138039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc188070090 0x7fc1880704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.514+0000 7fc185e00640 1 -- 172.21.15.62:0/3146138039 >> v2:172.21.15.120:3300/0 conn(0x7fc1880b2820 msgr2=0x7fc1880b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:18:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.514+0000 7fc185e00640 1 --2- 172.21.15.62:0/3146138039 >> v2:172.21.15.120:3300/0 conn(0x7fc1880b2820 0x7fc1880b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.514+0000 7fc185e00640 1 -- 172.21.15.62:0/3146138039 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1880cd3e0 con 0x7fc188143dd0 2024-03-23T11:18:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.514+0000 7fc186800640 1 --2- 172.21.15.62:0/3146138039 >> v2:172.21.15.120:3300/0 conn(0x7fc1880b2820 0x7fc1880b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:18:58.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.515+0000 7fc185e00640 1 --2- 172.21.15.62:0/3146138039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc188143dd0 0x7fc18815abe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc188071500 tx=0x7fc17802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29918bfb7eed003 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:58.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.515+0000 7fc184a00640 1 -- 172.21.15.62:0/3146138039 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178031870 con 0x7fc188143dd0 2024-03-23T11:18:58.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.515+0000 7fc184a00640 1 -- 172.21.15.62:0/3146138039 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1780319d0 con 0x7fc188143dd0 2024-03-23T11:18:58.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.515+0000 7fc184a00640 1 -- 172.21.15.62:0/3146138039 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178031d60 con 0x7fc188143dd0 2024-03-23T11:18:58.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.516+0000 7fc187200640 1 -- 172.21.15.62:0/3146138039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc188143dd0 msgr2=0x7fc18815abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:58.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.516+0000 7fc187200640 1 --2- 172.21.15.62:0/3146138039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc188143dd0 0x7fc18815abe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc188071500 tx=0x7fc17802b910 comp rx=0 tx=0).stop 2024-03-23T11:18:58.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.516+0000 7fc187200640 1 -- 172.21.15.62:0/3146138039 shutdown_connections 2024-03-23T11:18:58.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.516+0000 7fc187200640 1 --2- 172.21.15.62:0/3146138039 >> v2:172.21.15.120:3300/0 conn(0x7fc1880b2820 0x7fc1880b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:58.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.516+0000 7fc187200640 1 --2- 172.21.15.62:0/3146138039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc188070090 0x7fc1880704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:58.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.516+0000 7fc187200640 1 --2- 172.21.15.62:0/3146138039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc188143dd0 0x7fc18815abe0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:58.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.516+0000 7fc187200640 1 -- 172.21.15.62:0/3146138039 >> 172.21.15.62:0/3146138039 conn(0x7fc188066050 msgr2=0x7fc188069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:58.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.516+0000 7fc187200640 1 -- 172.21.15.62:0/3146138039 shutdown_connections 2024-03-23T11:18:58.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.516+0000 7fc187200640 1 -- 172.21.15.62:0/3146138039 wait complete. 2024-03-23T11:18:58.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.518+0000 7fc187200640 1 Processor -- start 2024-03-23T11:18:58.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.519+0000 7fc187200640 1 -- start start 2024-03-23T11:18:58.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.520+0000 7fc187200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc188070090 0x7fc188159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:58.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.520+0000 7fc187200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1880b2820 0x7fc1881f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:58.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.520+0000 7fc187200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc188143dd0 0x7fc1880a7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:58.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.520+0000 7fc187200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18815a1c0 con 0x7fc1880b2820 2024-03-23T11:18:58.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.520+0000 7fc187200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1881f18c0 con 0x7fc188070090 2024-03-23T11:18:58.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.520+0000 7fc187200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1881f1ba0 con 0x7fc188143dd0 2024-03-23T11:18:58.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.520+0000 7fc185400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1880b2820 0x7fc1881f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:58.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.520+0000 7fc185400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1880b2820 0x7fc1881f2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37670/0 (socket says 172.21.15.62:37670) 2024-03-23T11:18:58.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.520+0000 7fc185400640 1 -- 172.21.15.62:0/777021975 learned_addr learned my addr 172.21.15.62:0/777021975 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:18:58.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.520+0000 7fc185e00640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc188070090 0x7fc188159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:58.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc186800640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc188143dd0 0x7fc1880a7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:58.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc185400640 1 -- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc188143dd0 msgr2=0x7fc1880a7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:58.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc185400640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc188143dd0 0x7fc1880a7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:58.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc185400640 1 -- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc188070090 msgr2=0x7fc188159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:58.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc185400640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc188070090 0x7fc188159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:58.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc185400640 1 -- 172.21.15.62:0/777021975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1780094d0 con 0x7fc1880b2820 2024-03-23T11:18:58.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc186800640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc188143dd0 0x7fc1880a7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:18:58.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc185400640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1880b2820 0x7fc1881f2960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc17400d5f0 tx=0x7fc17400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:58.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc17e000640 1 -- 172.21.15.62:0/777021975 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1740125f0 con 0x7fc1880b2820 2024-03-23T11:18:58.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc17e000640 1 -- 172.21.15.62:0/777021975 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc174012750 con 0x7fc1880b2820 2024-03-23T11:18:58.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc17e000640 1 -- 172.21.15.62:0/777021975 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174012b00 con 0x7fc1880b2820 2024-03-23T11:18:58.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.521+0000 7fc187200640 1 -- 172.21.15.62:0/777021975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1881f1e80 con 0x7fc1880b2820 2024-03-23T11:18:58.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.522+0000 7fc187200640 1 -- 172.21.15.62:0/777021975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1881f2440 con 0x7fc1880b2820 2024-03-23T11:18:58.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.523+0000 7fc187200640 1 -- 172.21.15.62:0/777021975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc188066bb0 con 0x7fc1880b2820 2024-03-23T11:18:58.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.524+0000 7fc17e000640 1 -- 172.21.15.62:0/777021975 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc174012c60 con 0x7fc1880b2820 2024-03-23T11:18:58.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.524+0000 7fc17e000640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc16806a7a0 0x7fc16806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:18:58.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.525+0000 7fc17e000640 1 -- 172.21.15.62:0/777021975 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc174089230 con 0x7fc1880b2820 2024-03-23T11:18:58.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.525+0000 7fc185e00640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc16806a7a0 0x7fc16806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:18:58.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.526+0000 7fc185e00640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc16806a7a0 0x7fc16806cc60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc178036000 tx=0x7fc17803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:18:58.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.531+0000 7fc17e000640 1 -- 172.21.15.62:0/777021975 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc174054640 con 0x7fc1880b2820 2024-03-23T11:18:58.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:58.768+0000 7fc187200640 1 -- 172.21.15.62:0/777021975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc18815c650 con 0x7fc1880b2820 2024-03-23T11:18:59.549 INFO:tasks.ceph.mds.d.smithi120.stderr:2024-03-23T11:18:59.548+0000 7fd3ff400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:18:59.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.774+0000 7fc17e000640 1 -- 172.21.15.62:0/777021975 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc174057c60 con 0x7fc1880b2820 2024-03-23T11:18:59.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.777+0000 7fc187200640 1 -- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc16806a7a0 msgr2=0x7fc16806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:59.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.777+0000 7fc187200640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc16806a7a0 0x7fc16806cc60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc178036000 tx=0x7fc17803c040 comp rx=0 tx=0).stop 2024-03-23T11:18:59.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.777+0000 7fc187200640 1 -- 172.21.15.62:0/777021975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1880b2820 msgr2=0x7fc1881f2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:18:59.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.777+0000 7fc187200640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1880b2820 0x7fc1881f2960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc17400d5f0 tx=0x7fc17400dab0 comp rx=0 tx=0).stop 2024-03-23T11:18:59.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.777+0000 7fc187200640 1 -- 172.21.15.62:0/777021975 shutdown_connections 2024-03-23T11:18:59.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.777+0000 7fc187200640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc16806a7a0 0x7fc16806cc60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:59.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.777+0000 7fc187200640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1880b2820 0x7fc1881f2960 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:59.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.777+0000 7fc187200640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc188143dd0 0x7fc1880a7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:59.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.777+0000 7fc187200640 1 --2- 172.21.15.62:0/777021975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc188070090 0x7fc188159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:18:59.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.777+0000 7fc187200640 1 -- 172.21.15.62:0/777021975 >> 172.21.15.62:0/777021975 conn(0x7fc188066050 msgr2=0x7fc1880ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:18:59.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.777+0000 7fc187200640 1 -- 172.21.15.62:0/777021975 shutdown_connections 2024-03-23T11:18:59.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:18:59.778+0000 7fc187200640 1 -- 172.21.15.62:0/777021975 wait complete. 2024-03-23T11:18:59.799 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T11:18:59.799 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:19:00.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.012+0000 7f0404e00640 1 Processor -- start 2024-03-23T11:19:00.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.012+0000 7f0404e00640 1 -- start start 2024-03-23T11:19:00.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.014+0000 7f0404e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0400160c00 0x7f040006d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.014+0000 7f0404e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0400160fe0 con 0x7f04000cde90 2024-03-23T11:19:00.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.014+0000 7f0404e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04000b1510 con 0x7f04000cda20 2024-03-23T11:19:00.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.014+0000 7f0404e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04000b1ad0 con 0x7f0400160c00 2024-03-23T11:19:00.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.014+0000 7f03ffe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f04000cde90 0x7f04000caf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53396/0 (socket says 172.21.15.62:53396) 2024-03-23T11:19:00.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.014+0000 7f03ffe00640 1 -- 172.21.15.62:0/2058906668 learned_addr learned my addr 172.21.15.62:0/2058906668 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:00.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03ff400640 1 --2- 172.21.15.62:0/2058906668 >> v2:172.21.15.120:3300/0 conn(0x7f0400160c00 0x7f040006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522770941 0 0) 0x7f0400160fe0 con 0x7f04000cde90 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e4003520 con 0x7f04000cde90 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 185708546 0 0) 0x7f03e4003520 con 0x7f04000cde90 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 >> v2:172.21.15.120:3300/0 conn(0x7f0400160c00 msgr2=0x7f040006d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 --2- 172.21.15.62:0/2058906668 >> v2:172.21.15.120:3300/0 conn(0x7f0400160c00 0x7f040006d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 >> v1:172.21.15.120:6789/0 conn(0x7f04000cda20 legacy=0x7f040006de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04000b17f0 con 0x7f04000cde90 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f03f80019f0 con 0x7f04000cde90 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 >> v1:172.21.15.62:6789/0 conn(0x7f04000cde90 legacy=0x7f04000caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03e4003970 0x7f03e4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03e40047c0 0x7f03e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03e4005180 0x7f03e400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04000b1ad0 con 0x7f03e4005180 2024-03-23T11:19:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e400c020 con 0x7f03e4003970 2024-03-23T11:19:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.015+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e400c1e0 con 0x7f03e40047c0 2024-03-23T11:19:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.016+0000 7f03ffe00640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03e4005180 0x7f03e400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.016+0000 7f03ff400640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03e4003970 0x7f03e4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.016+0000 7f03f7e00640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03e40047c0 0x7f03e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.016+0000 7f03ffe00640 1 -- 172.21.15.62:0/2058906668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03e40047c0 msgr2=0x7f03e4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.016+0000 7f03ffe00640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03e40047c0 0x7f03e4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.016+0000 7f03ffe00640 1 -- 172.21.15.62:0/2058906668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03e4003970 msgr2=0x7f03e4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.016+0000 7f03ffe00640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03e4003970 0x7f03e4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.016+0000 7f03ffe00640 1 -- 172.21.15.62:0/2058906668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e4003520 con 0x7f03e4005180 2024-03-23T11:19:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.016+0000 7f03f7e00640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03e40047c0 0x7f03e4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.016+0000 7f03ff400640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03e4003970 0x7f03e4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:00.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.017+0000 7f03ffe00640 1 -- 172.21.15.62:0/2058906668 reap_dead start 2024-03-23T11:19:00.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.017+0000 7f03ffe00640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03e4005180 0x7f03e400b980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03f800a830 tx=0x7f03e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c72b5c4ec39499 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:00.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.017+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f80072d0 con 0x7f03e4005180 2024-03-23T11:19:00.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.017+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f802f070 con 0x7f03e4005180 2024-03-23T11:19:00.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.017+0000 7f03fea00640 1 -- 172.21.15.62:0/2058906668 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f80076b0 con 0x7f03e4005180 2024-03-23T11:19:00.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.018+0000 7f0404e00640 1 -- 172.21.15.62:0/2058906668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03e4005180 msgr2=0x7f03e400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.018+0000 7f0404e00640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03e4005180 0x7f03e400b980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f03f800a830 tx=0x7f03e40057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.018+0000 7f0404e00640 1 -- 172.21.15.62:0/2058906668 shutdown_connections 2024-03-23T11:19:00.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.018+0000 7f0404e00640 1 --2- 172.21.15.62:0/2058906668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03e4005180 0x7f03e400b980 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.018+0000 7f0404e00640 1 -- 172.21.15.62:0/2058906668 >> 172.21.15.62:0/2058906668 conn(0x7f0400066050 msgr2=0x7f0400069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:00.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.018+0000 7f0404e00640 1 -- 172.21.15.62:0/2058906668 shutdown_connections 2024-03-23T11:19:00.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.018+0000 7f0404e00640 1 -- 172.21.15.62:0/2058906668 wait complete. 2024-03-23T11:19:00.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.020+0000 7f0404e00640 1 Processor -- start 2024-03-23T11:19:00.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.020+0000 7f0404e00640 1 -- start start 2024-03-23T11:19:00.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.021+0000 7f0404e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03e4005180 0x7f040014d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.021+0000 7f0404e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040014e990 0x7f04000cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.021+0000 7f0404e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f040014ee10 0x7f04000cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.021+0000 7f0404e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040014dd50 con 0x7f040014ee10 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.021+0000 7f0404e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040014f3a0 con 0x7f03e4005180 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.021+0000 7f0404e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040014f680 con 0x7f040014e990 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.021+0000 7f03ffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f040014ee10 0x7f04000cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.021+0000 7f03ffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f040014ee10 0x7f04000cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37686/0 (socket says 172.21.15.62:37686) 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.021+0000 7f03ffe00640 1 -- 172.21.15.62:0/2731989838 learned_addr learned my addr 172.21.15.62:0/2731989838 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.022+0000 7f03f7e00640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040014e990 0x7f04000cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.022+0000 7f03ff400640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03e4005180 0x7f040014d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.022+0000 7f03ffe00640 1 -- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040014e990 msgr2=0x7f04000cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.022+0000 7f03ffe00640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040014e990 0x7f04000cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.022+0000 7f03ffe00640 1 -- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03e4005180 msgr2=0x7f040014d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.022+0000 7f03ffe00640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03e4005180 0x7f040014d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.022+0000 7f03ffe00640 1 -- 172.21.15.62:0/2731989838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040014f960 con 0x7f040014ee10 2024-03-23T11:19:00.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.022+0000 7f03ff400640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03e4005180 0x7f040014d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:00.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.022+0000 7f03ffe00640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f040014ee10 0x7f04000cc470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f03f800a950 tx=0x7f03f8001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:00.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.023+0000 7f03f7400640 1 -- 172.21.15.62:0/2731989838 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f80019b0 con 0x7f040014ee10 2024-03-23T11:19:00.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.023+0000 7f03f7400640 1 -- 172.21.15.62:0/2731989838 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f8001250 con 0x7f040014ee10 2024-03-23T11:19:00.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.023+0000 7f03f7400640 1 -- 172.21.15.62:0/2731989838 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8004dc0 con 0x7f040014ee10 2024-03-23T11:19:00.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.023+0000 7f0404e00640 1 -- 172.21.15.62:0/2731989838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040014fc40 con 0x7f040014ee10 2024-03-23T11:19:00.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.023+0000 7f0404e00640 1 -- 172.21.15.62:0/2731989838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04000a7760 con 0x7f040014ee10 2024-03-23T11:19:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.024+0000 7f0404e00640 1 -- 172.21.15.62:0/2731989838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0400066bb0 con 0x7f040014ee10 2024-03-23T11:19:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.025+0000 7f03f7400640 1 -- 172.21.15.62:0/2731989838 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f03f8031030 con 0x7f040014ee10 2024-03-23T11:19:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.025+0000 7f03f7400640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03dc06a7d0 0x7f03dc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.026+0000 7f03f7400640 1 -- 172.21.15.62:0/2731989838 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03f80aa910 con 0x7f040014ee10 2024-03-23T11:19:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.026+0000 7f03ff400640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03dc06a7d0 0x7f03dc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.027+0000 7f03ff400640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03dc06a7d0 0x7f03dc06cc90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f040015dd50 tx=0x7f03f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:00.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.030+0000 7f03f7400640 1 -- 172.21.15.62:0/2731989838 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f03f8075c90 con 0x7f040014ee10 2024-03-23T11:19:00.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.269+0000 7f0404e00640 1 -- 172.21.15.62:0/2731989838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0400066bb0 con 0x7f040014ee10 2024-03-23T11:19:00.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.270+0000 7f03f7400640 1 -- 172.21.15.62:0/2731989838 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f03f80792b0 con 0x7f040014ee10 2024-03-23T11:19:00.271 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:00.271 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/496399527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":496399527},{"type":"v1","addr":"172.21.15.62:6833","nonce":496399527}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/156755611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":156755611},{"type":"v1","addr":"172.21.15.120:6835","nonce":156755611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2391380456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2391380456},{"type":"v1","addr":"172.21.15.62:6835","nonce":2391380456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:18:56.233595+0000","modified":"2024-03-23T11:18:59.758284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/3936303103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3936303103},{"type":"v1","addr":"172.21.15.120:6837","nonce":3936303103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T11:19:00.271 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-03-23T11:19:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 -- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03dc06a7d0 msgr2=0x7f03dc06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03dc06a7d0 0x7f03dc06cc90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f040015dd50 tx=0x7f03f0008040 comp rx=0 tx=0).stop 2024-03-23T11:19:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 -- 172.21.15.62:0/2731989838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f040014ee10 msgr2=0x7f04000cc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f040014ee10 0x7f04000cc470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f03f800a950 tx=0x7f03f8001d80 comp rx=0 tx=0).stop 2024-03-23T11:19:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 -- 172.21.15.62:0/2731989838 shutdown_connections 2024-03-23T11:19:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03dc06a7d0 0x7f03dc06cc90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f040014ee10 0x7f04000cc470 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f040014e990 0x7f04000cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 --2- 172.21.15.62:0/2731989838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03e4005180 0x7f040014d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 -- 172.21.15.62:0/2731989838 >> 172.21.15.62:0/2731989838 conn(0x7f0400066050 msgr2=0x7f04000ad830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:00.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 -- 172.21.15.62:0/2731989838 shutdown_connections 2024-03-23T11:19:00.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.273+0000 7f0404e00640 1 -- 172.21.15.62:0/2731989838 wait complete. 2024-03-23T11:19:00.293 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:00.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.513+0000 7f3d16400640 1 Processor -- start 2024-03-23T11:19:00.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.513+0000 7f3d16400640 1 -- start start 2024-03-23T11:19:00.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.515+0000 7f3d16400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1006cc60 0x7f3d100cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.515+0000 7f3d16400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d100ce8b0 con 0x7f3d10159490 2024-03-23T11:19:00.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.515+0000 7f3d16400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d100ced90 con 0x7f3d100ce420 2024-03-23T11:19:00.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.515+0000 7f3d16400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1006e3e0 con 0x7f3d1006cc60 2024-03-23T11:19:00.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.515+0000 7f3d15000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3d10159490 0x7f3d1015da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53400/0 (socket says 172.21.15.62:53400) 2024-03-23T11:19:00.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.515+0000 7f3d15000640 1 -- 172.21.15.62:0/2812592538 learned_addr learned my addr 172.21.15.62:0/2812592538 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:00.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.515+0000 7f3d0fe00640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1006cc60 0x7f3d100cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.515+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190511479 0 0) 0x7f3d100ce8b0 con 0x7f3d10159490 2024-03-23T11:19:00.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.515+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf4005730 con 0x7f3d10159490 2024-03-23T11:19:00.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3847536625 0 0) 0x7f3cf4005730 con 0x7f3d10159490 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1006cc60 msgr2=0x7f3d100cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1006cc60 0x7f3d100cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 >> v1:172.21.15.120:6789/0 conn(0x7f3d100ce420 legacy=0x7f3d100b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1006c1c0 con 0x7f3d10159490 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f3d00001670 con 0x7f3d10159490 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 >> v1:172.21.15.62:6789/0 conn(0x7f3d10159490 legacy=0x7f3d1015da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 0x7f3cf4005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1006cc60 0x7f3cf40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cf4006f10 0x7f3cf4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1006e3e0 con 0x7f3cf4005bb0 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf400dd20 con 0x7f3cf4006f10 2024-03-23T11:19:00.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf400dee0 con 0x7f3d1006cc60 2024-03-23T11:19:00.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.516+0000 7f3d15000640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 0x7f3cf4005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.517+0000 7f3d0fe00640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1006cc60 0x7f3cf40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.517+0000 7f3d15a00640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cf4006f10 0x7f3cf4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.517+0000 7f3d15000640 1 -- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1006cc60 msgr2=0x7f3cf40069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:00.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.517+0000 7f3d15000640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1006cc60 0x7f3cf40069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.517+0000 7f3d15000640 1 -- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cf4006f10 msgr2=0x7f3cf4007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:00.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.517+0000 7f3d15000640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cf4006f10 0x7f3cf4007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.517+0000 7f3d15000640 1 -- 172.21.15.62:0/2812592538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf4005730 con 0x7f3cf4005bb0 2024-03-23T11:19:00.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.517+0000 7f3d0fe00640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d1006cc60 0x7f3cf40069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:00.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.518+0000 7f3d15000640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 0x7f3cf4005fd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3d00007fc0 tx=0x7f3cf4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92fc76e92dd5fcaa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:00.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.518+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00005ad0 con 0x7f3cf4005bb0 2024-03-23T11:19:00.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.518+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d00005c30 con 0x7f3cf4005bb0 2024-03-23T11:19:00.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.519+0000 7f3d0f400640 1 -- 172.21.15.62:0/2812592538 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0002c7c0 con 0x7f3cf4005bb0 2024-03-23T11:19:00.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.519+0000 7f3d16400640 1 -- 172.21.15.62:0/2812592538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 msgr2=0x7f3cf4005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.519+0000 7f3d16400640 1 --2- 172.21.15.62:0/2812592538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 0x7f3cf4005fd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3d00007fc0 tx=0x7f3cf4007a00 comp rx=0 tx=0).stop 2024-03-23T11:19:00.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.519+0000 7f3d15a00640 1 -- 172.21.15.62:0/2812592538 reap_dead start 2024-03-23T11:19:00.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.519+0000 7f3d16400640 1 -- 172.21.15.62:0/2812592538 shutdown_connections 2024-03-23T11:19:00.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.519+0000 7f3d16400640 1 -- 172.21.15.62:0/2812592538 >> 172.21.15.62:0/2812592538 conn(0x7f3d10066050 msgr2=0x7f3d10069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:00.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.519+0000 7f3d16400640 1 -- 172.21.15.62:0/2812592538 shutdown_connections 2024-03-23T11:19:00.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.519+0000 7f3d16400640 1 -- 172.21.15.62:0/2812592538 wait complete. 2024-03-23T11:19:00.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.521+0000 7f3d16400640 1 Processor -- start 2024-03-23T11:19:00.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.521+0000 7f3d16400640 1 -- start start 2024-03-23T11:19:00.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.522+0000 7f3d16400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 0x7f3d100acf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.522+0000 7f3d16400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d10153640 0x7f3d100b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.522+0000 7f3d16400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d10151810 0x7f3d10151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.522+0000 7f3d16400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d10152200 con 0x7f3cf4005bb0 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.522+0000 7f3d16400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d10153c90 con 0x7f3d10151810 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.522+0000 7f3d16400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d10153f70 con 0x7f3d10153640 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.522+0000 7f3d15000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 0x7f3d100acf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.522+0000 7f3d15000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 0x7f3d100acf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37702/0 (socket says 172.21.15.62:37702) 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.522+0000 7f3d15000640 1 -- 172.21.15.62:0/3986620750 learned_addr learned my addr 172.21.15.62:0/3986620750 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d0fe00640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d10153640 0x7f3d100b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d15a00640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d10151810 0x7f3d10151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d15000640 1 -- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d10153640 msgr2=0x7f3d100b2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d15000640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d10153640 0x7f3d100b2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d15000640 1 -- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d10151810 msgr2=0x7f3d10151cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d15000640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d10151810 0x7f3d10151cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d15000640 1 -- 172.21.15.62:0/3986620750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d10154250 con 0x7f3cf4005bb0 2024-03-23T11:19:00.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d15a00640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d10151810 0x7f3d10151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:00.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d15000640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 0x7f3d100acf20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3d0002c360 tx=0x7f3d1006c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:00.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d0cc00640 1 -- 172.21.15.62:0/3986620750 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0002cbd0 con 0x7f3cf4005bb0 2024-03-23T11:19:00.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d0cc00640 1 -- 172.21.15.62:0/3986620750 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0002cd30 con 0x7f3cf4005bb0 2024-03-23T11:19:00.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d0cc00640 1 -- 172.21.15.62:0/3986620750 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00007290 con 0x7f3cf4005bb0 2024-03-23T11:19:00.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.523+0000 7f3d16400640 1 -- 172.21.15.62:0/3986620750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d10154530 con 0x7f3cf4005bb0 2024-03-23T11:19:00.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.524+0000 7f3d16400640 1 -- 172.21.15.62:0/3986620750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1014a5c0 con 0x7f3cf4005bb0 2024-03-23T11:19:00.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.525+0000 7f3d16400640 1 -- 172.21.15.62:0/3986620750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d10153a20 con 0x7f3cf4005bb0 2024-03-23T11:19:00.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.526+0000 7f3d0cc00640 1 -- 172.21.15.62:0/3986620750 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3d00032070 con 0x7f3cf4005bb0 2024-03-23T11:19:00.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.526+0000 7f3d0cc00640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cf006a7d0 0x7f3cf006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.527+0000 7f3d0cc00640 1 -- 172.21.15.62:0/3986620750 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d000aa1d0 con 0x7f3cf4005bb0 2024-03-23T11:19:00.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.527+0000 7f3d0fe00640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cf006a7d0 0x7f3cf006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.528+0000 7f3d0fe00640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cf006a7d0 0x7f3cf006cc90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3d10158370 tx=0x7f3cfc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:00.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.531+0000 7f3d0cc00640 1 -- 172.21.15.62:0/3986620750 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3d000755e0 con 0x7f3cf4005bb0 2024-03-23T11:19:00.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.729+0000 7f3d16400640 1 -- 172.21.15.62:0/3986620750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d10066bb0 con 0x7f3cf4005bb0 2024-03-23T11:19:00.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.731+0000 7f3d0cc00640 1 -- 172.21.15.62:0/3986620750 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f3d00045090 con 0x7f3cf4005bb0 2024-03-23T11:19:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:00.732 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:57.238201+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:00.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.734+0000 7f3d16400640 1 -- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cf006a7d0 msgr2=0x7f3cf006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.734+0000 7f3d16400640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cf006a7d0 0x7f3cf006cc90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3d10158370 tx=0x7f3cfc008040 comp rx=0 tx=0).stop 2024-03-23T11:19:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.734+0000 7f3d16400640 1 -- 172.21.15.62:0/3986620750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 msgr2=0x7f3d100acf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.734+0000 7f3d16400640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 0x7f3d100acf20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3d0002c360 tx=0x7f3d1006c350 comp rx=0 tx=0).stop 2024-03-23T11:19:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.734+0000 7f3d16400640 1 -- 172.21.15.62:0/3986620750 shutdown_connections 2024-03-23T11:19:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.734+0000 7f3d16400640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cf006a7d0 0x7f3cf006cc90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.734+0000 7f3d16400640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d10151810 0x7f3d10151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.734+0000 7f3d16400640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d10153640 0x7f3d100b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.734+0000 7f3d16400640 1 --2- 172.21.15.62:0/3986620750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf4005bb0 0x7f3d100acf20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.734+0000 7f3d16400640 1 -- 172.21.15.62:0/3986620750 >> 172.21.15.62:0/3986620750 conn(0x7f3d10066050 msgr2=0x7f3d100ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.734+0000 7f3d16400640 1 -- 172.21.15.62:0/3986620750 shutdown_connections 2024-03-23T11:19:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.735+0000 7f3d16400640 1 -- 172.21.15.62:0/3986620750 wait complete. 2024-03-23T11:19:00.754 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T11:19:00.754 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:19:00.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.966+0000 7f6413e00640 1 Processor -- start 2024-03-23T11:19:00.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.967+0000 7f6413e00640 1 -- start start 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.968+0000 7f6413e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f64140ce5f0 0x7f64140b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.969+0000 7f6413e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64140ced90 con 0x7f641406cc60 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.969+0000 7f6413e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641406e3e0 con 0x7f6414159490 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.969+0000 7f6413e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f641406c1c0 con 0x7f64140ce5f0 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.969+0000 7f6412000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f641406cc60 0x7f64140cdee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53410/0 (socket says 172.21.15.62:53410) 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.969+0000 7f6412000640 1 -- 172.21.15.62:0/1157860842 learned_addr learned my addr 172.21.15.62:0/1157860842 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.969+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502902335 0 0) 0x7f64140ced90 con 0x7f641406cc60 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.969+0000 7f6413400640 1 --2- 172.21.15.62:0/1157860842 >> v2:172.21.15.120:3300/0 conn(0x7f64140ce5f0 0x7f64140b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.969+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f8005730 con 0x7f641406cc60 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3942238521 0 0) 0x7f63f8005730 con 0x7f641406cc60 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 >> v2:172.21.15.120:3300/0 conn(0x7f64140ce5f0 msgr2=0x7f64140b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 --2- 172.21.15.62:0/1157860842 >> v2:172.21.15.120:3300/0 conn(0x7f64140ce5f0 0x7f64140b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 >> v1:172.21.15.120:6789/0 conn(0x7f6414159490 legacy=0x7f641415da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641406d0c0 con 0x7f641406cc60 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f63fc001670 con 0x7f641406cc60 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 >> v1:172.21.15.62:6789/0 conn(0x7f641406cc60 legacy=0x7f64140cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f8005b80 0x7f63f8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f80069d0 0x7f63f8006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8007390 0x7f63f800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641406c1c0 con 0x7f63f8007390 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f800e330 con 0x7f63f8005b80 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f800e4f0 con 0x7f63f80069d0 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6413400640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8007390 0x7f63f800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6412a00640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f8005b80 0x7f63f8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.970+0000 7f6412000640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f80069d0 0x7f63f8006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.971+0000 7f6413400640 1 -- 172.21.15.62:0/1157860842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f80069d0 msgr2=0x7f63f8006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:00.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.971+0000 7f6413400640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63f80069d0 0x7f63f8006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.971+0000 7f6413400640 1 -- 172.21.15.62:0/1157860842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f8005b80 msgr2=0x7f63f8005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:00.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.971+0000 7f6413400640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f8005b80 0x7f63f8005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.971+0000 7f6413400640 1 -- 172.21.15.62:0/1157860842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f8005730 con 0x7f63f8007390 2024-03-23T11:19:00.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.971+0000 7f6412a00640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f8005b80 0x7f63f8005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:00.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.972+0000 7f6413400640 1 -- 172.21.15.62:0/1157860842 reap_dead start 2024-03-23T11:19:00.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.972+0000 7f6413400640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8007390 0x7f63f800db90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f640c009ac0 tx=0x7f63f8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2cb08e39445238 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:00.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.972+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c004020 con 0x7f63f8007390 2024-03-23T11:19:00.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.972+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c004180 con 0x7f63f8007390 2024-03-23T11:19:00.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.972+0000 7f6411600640 1 -- 172.21.15.62:0/1157860842 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c02c710 con 0x7f63f8007390 2024-03-23T11:19:00.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.973+0000 7f6413e00640 1 -- 172.21.15.62:0/1157860842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8007390 msgr2=0x7f63f800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.973+0000 7f6413e00640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8007390 0x7f63f800db90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f640c009ac0 tx=0x7f63f8007a00 comp rx=0 tx=0).stop 2024-03-23T11:19:00.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.973+0000 7f6413e00640 1 -- 172.21.15.62:0/1157860842 shutdown_connections 2024-03-23T11:19:00.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.973+0000 7f6413e00640 1 --2- 172.21.15.62:0/1157860842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8007390 0x7f63f800db90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.973+0000 7f6413e00640 1 -- 172.21.15.62:0/1157860842 >> 172.21.15.62:0/1157860842 conn(0x7f6414066050 msgr2=0x7f6414069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:00.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.973+0000 7f6413e00640 1 -- 172.21.15.62:0/1157860842 shutdown_connections 2024-03-23T11:19:00.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.973+0000 7f6413e00640 1 -- 172.21.15.62:0/1157860842 wait complete. 2024-03-23T11:19:00.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.975+0000 7f6413e00640 1 Processor -- start 2024-03-23T11:19:00.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.975+0000 7f6413e00640 1 -- start start 2024-03-23T11:19:00.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.976+0000 7f6413e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f8007390 0x7f64140ad040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.976+0000 7f6413e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6414155240 0x7f64140b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.976+0000 7f6413e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64141eda80 0x7f64141edf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.976+0000 7f6413e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64141563a0 con 0x7f64141eda80 2024-03-23T11:19:00.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.976+0000 7f6413e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64140ad580 con 0x7f63f8007390 2024-03-23T11:19:00.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.976+0000 7f6413e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6414155790 con 0x7f6414155240 2024-03-23T11:19:00.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.976+0000 7f6413400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64141eda80 0x7f64141edf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.976+0000 7f6413400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64141eda80 0x7f64141edf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37716/0 (socket says 172.21.15.62:37716) 2024-03-23T11:19:00.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.976+0000 7f6413400640 1 -- 172.21.15.62:0/2716059834 learned_addr learned my addr 172.21.15.62:0/2716059834 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:00.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f6412000640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6414155240 0x7f64140b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f6412a00640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f8007390 0x7f64140ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f6413400640 1 -- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6414155240 msgr2=0x7f64140b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f6413400640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6414155240 0x7f64140b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f6413400640 1 -- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f8007390 msgr2=0x7f64140ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:00.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f6413400640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f8007390 0x7f64140ad040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:00.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f6413400640 1 -- 172.21.15.62:0/2716059834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6414155a70 con 0x7f64141eda80 2024-03-23T11:19:00.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f6413400640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64141eda80 0x7f64141edf00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f640c02c380 tx=0x7f640c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:00.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f640aa00640 1 -- 172.21.15.62:0/2716059834 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c03a070 con 0x7f64141eda80 2024-03-23T11:19:00.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f640aa00640 1 -- 172.21.15.62:0/2716059834 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c004500 con 0x7f64141eda80 2024-03-23T11:19:00.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f640aa00640 1 -- 172.21.15.62:0/2716059834 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c003830 con 0x7f64141eda80 2024-03-23T11:19:00.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.977+0000 7f6413e00640 1 -- 172.21.15.62:0/2716059834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6414155d50 con 0x7f64141eda80 2024-03-23T11:19:00.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.978+0000 7f6413e00640 1 -- 172.21.15.62:0/2716059834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64140b0820 con 0x7f64141eda80 2024-03-23T11:19:00.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.979+0000 7f6413e00640 1 -- 172.21.15.62:0/2716059834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6414156550 con 0x7f64141eda80 2024-03-23T11:19:00.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.980+0000 7f640aa00640 1 -- 172.21.15.62:0/2716059834 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f640c003990 con 0x7f64141eda80 2024-03-23T11:19:00.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.980+0000 7f640aa00640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63f006a7d0 0x7f63f006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:00.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.980+0000 7f640aa00640 1 -- 172.21.15.62:0/2716059834 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f640c0a9670 con 0x7f64141eda80 2024-03-23T11:19:00.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.981+0000 7f6412a00640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63f006a7d0 0x7f63f006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:00.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.982+0000 7f6412a00640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63f006a7d0 0x7f63f006cc90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6404007fd0 tx=0x7f6404009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:00.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:00.985+0000 7f640aa00640 1 -- 172.21.15.62:0/2716059834 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6414156550 con 0x7f64141eda80 2024-03-23T11:19:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.182+0000 7f6413e00640 1 -- 172.21.15.62:0/2716059834 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f641406c740 con 0x7f63f006a7d0 2024-03-23T11:19:01.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.207+0000 7f640aa00640 1 -- 172.21.15.62:0/2716059834 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264446 (secure 0 0 0) 0x7f6404005740 con 0x7f63f006a7d0 2024-03-23T11:19:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:01.213 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.213+0000 7f6413e00640 1 -- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63f006a7d0 msgr2=0x7f63f006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.213+0000 7f6413e00640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63f006a7d0 0x7f63f006cc90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6404007fd0 tx=0x7f6404009040 comp rx=0 tx=0).stop 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.213+0000 7f6413e00640 1 -- 172.21.15.62:0/2716059834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64141eda80 msgr2=0x7f64141edf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.213+0000 7f6413e00640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64141eda80 0x7f64141edf00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f640c02c380 tx=0x7f640c004730 comp rx=0 tx=0).stop 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.214+0000 7f6413e00640 1 -- 172.21.15.62:0/2716059834 shutdown_connections 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.214+0000 7f6413e00640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63f006a7d0 0x7f63f006cc90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.214+0000 7f6413e00640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64141eda80 0x7f64141edf00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.214+0000 7f6413e00640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6414155240 0x7f64140b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.214+0000 7f6413e00640 1 --2- 172.21.15.62:0/2716059834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63f8007390 0x7f64140ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.214+0000 7f6413e00640 1 -- 172.21.15.62:0/2716059834 >> 172.21.15.62:0/2716059834 conn(0x7f6414066050 msgr2=0x7f64140af770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.214+0000 7f6413e00640 1 -- 172.21.15.62:0/2716059834 shutdown_connections 2024-03-23T11:19:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.214+0000 7f6413e00640 1 -- 172.21.15.62:0/2716059834 wait complete. 2024-03-23T11:19:01.233 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-23T11:18:59.232607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":2972,"kb_used_omap":13,"kb_used_meta":216050,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3043328,"data_stored":1813773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":221235288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":80,"num_write_kb":798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.496694"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257670+0000","last_change":"2024-03-23T11:18:56.274088+0000","last_active":"2024-03-23T11:18:57.257670+0000","last_peered":"2024-03-23T11:18:57.257670+0000","last_clean":"2024-03-23T11:18:57.257670+0000","last_became_active":"2024-03-23T11:18:56.273097+0000","last_became_peered":"2024-03-23T11:18:56.273097+0000","last_unstale":"2024-03-23T11:18:57.257670+0000","last_undegraded":"2024-03-23T11:18:57.257670+0000","last_fullsized":"2024-03-23T11:18:57.257670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:48:24.807457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253337+0000","last_change":"2024-03-23T11:18:56.260586+0000","last_active":"2024-03-23T11:18:57.253337+0000","last_peered":"2024-03-23T11:18:57.253337+0000","last_clean":"2024-03-23T11:18:57.253337+0000","last_became_active":"2024-03-23T11:18:56.260321+0000","last_became_peered":"2024-03-23T11:18:56.260321+0000","last_unstale":"2024-03-23T11:18:57.253337+0000","last_undegraded":"2024-03-23T11:18:57.253337+0000","last_fullsized":"2024-03-23T11:18:57.253337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:24:40.203305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099527+0000","last_change":"2024-03-23T11:18:56.264999+0000","last_active":"2024-03-23T11:18:58.099527+0000","last_peered":"2024-03-23T11:18:58.099527+0000","last_clean":"2024-03-23T11:18:58.099527+0000","last_became_active":"2024-03-23T11:18:56.264689+0000","last_became_peered":"2024-03-23T11:18:56.264689+0000","last_unstale":"2024-03-23T11:18:58.099527+0000","last_undegraded":"2024-03-23T11:18:58.099527+0000","last_fullsized":"2024-03-23T11:18:58.099527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.747240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315279+0000","last_change":"2024-03-23T11:18:56.283608+0000","last_active":"2024-03-23T11:18:57.315279+0000","last_peered":"2024-03-23T11:18:57.315279+0000","last_clean":"2024-03-23T11:18:57.315279+0000","last_became_active":"2024-03-23T11:18:56.277048+0000","last_became_peered":"2024-03-23T11:18:56.277048+0000","last_unstale":"2024-03-23T11:18:57.315279+0000","last_undegraded":"2024-03-23T11:18:57.315279+0000","last_fullsized":"2024-03-23T11:18:57.315279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:28.809711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256138+0000","last_change":"2024-03-23T11:18:56.264915+0000","last_active":"2024-03-23T11:18:57.256138+0000","last_peered":"2024-03-23T11:18:57.256138+0000","last_clean":"2024-03-23T11:18:57.256138+0000","last_became_active":"2024-03-23T11:18:56.264696+0000","last_became_peered":"2024-03-23T11:18:56.264696+0000","last_unstale":"2024-03-23T11:18:57.256138+0000","last_undegraded":"2024-03-23T11:18:57.256138+0000","last_fullsized":"2024-03-23T11:18:57.256138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:59.578102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314423+0000","last_change":"2024-03-23T11:18:56.282941+0000","last_active":"2024-03-23T11:18:57.314423+0000","last_peered":"2024-03-23T11:18:57.314423+0000","last_clean":"2024-03-23T11:18:57.314423+0000","last_became_active":"2024-03-23T11:18:56.282556+0000","last_became_peered":"2024-03-23T11:18:56.282556+0000","last_unstale":"2024-03-23T11:18:57.314423+0000","last_undegraded":"2024-03-23T11:18:57.314423+0000","last_fullsized":"2024-03-23T11:18:57.314423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:01:46.194360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255145+0000","last_change":"2024-03-23T11:18:56.274568+0000","last_active":"2024-03-23T11:18:57.255145+0000","last_peered":"2024-03-23T11:18:57.255145+0000","last_clean":"2024-03-23T11:18:57.255145+0000","last_became_active":"2024-03-23T11:18:56.273524+0000","last_became_peered":"2024-03-23T11:18:56.273524+0000","last_unstale":"2024-03-23T11:18:57.255145+0000","last_undegraded":"2024-03-23T11:18:57.255145+0000","last_fullsized":"2024-03-23T11:18:57.255145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:48:43.299392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099491+0000","last_change":"2024-03-23T11:18:56.272280+0000","last_active":"2024-03-23T11:18:58.099491+0000","last_peered":"2024-03-23T11:18:58.099491+0000","last_clean":"2024-03-23T11:18:58.099491+0000","last_became_active":"2024-03-23T11:18:56.272096+0000","last_became_peered":"2024-03-23T11:18:56.272096+0000","last_unstale":"2024-03-23T11:18:58.099491+0000","last_undegraded":"2024-03-23T11:18:58.099491+0000","last_fullsized":"2024-03-23T11:18:58.099491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:28.809711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257785+0000","last_change":"2024-03-23T11:18:56.282997+0000","last_active":"2024-03-23T11:18:57.257785+0000","last_peered":"2024-03-23T11:18:57.257785+0000","last_clean":"2024-03-23T11:18:57.257785+0000","last_became_active":"2024-03-23T11:18:56.282768+0000","last_became_peered":"2024-03-23T11:18:56.282768+0000","last_unstale":"2024-03-23T11:18:57.257785+0000","last_undegraded":"2024-03-23T11:18:57.257785+0000","last_fullsized":"2024-03-23T11:18:57.257785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:47:39.199128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461969+0000","last_change":"2024-03-23T11:18:56.269707+0000","last_active":"2024-03-23T11:18:57.461969+0000","last_peered":"2024-03-23T11:18:57.461969+0000","last_clean":"2024-03-23T11:18:57.461969+0000","last_became_active":"2024-03-23T11:18:56.266078+0000","last_became_peered":"2024-03-23T11:18:56.266078+0000","last_unstale":"2024-03-23T11:18:57.461969+0000","last_undegraded":"2024-03-23T11:18:57.461969+0000","last_fullsized":"2024-03-23T11:18:57.461969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461775+0000","last_change":"2024-03-23T11:18:56.266542+0000","last_active":"2024-03-23T11:18:57.461775+0000","last_peered":"2024-03-23T11:18:57.461775+0000","last_clean":"2024-03-23T11:18:57.461775+0000","last_became_active":"2024-03-23T11:18:56.266026+0000","last_became_peered":"2024-03-23T11:18:56.266026+0000","last_unstale":"2024-03-23T11:18:57.461775+0000","last_undegraded":"2024-03-23T11:18:57.461775+0000","last_fullsized":"2024-03-23T11:18:57.461775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255738+0000","last_change":"2024-03-23T11:18:56.272690+0000","last_active":"2024-03-23T11:18:57.255738+0000","last_peered":"2024-03-23T11:18:57.255738+0000","last_clean":"2024-03-23T11:18:57.255738+0000","last_became_active":"2024-03-23T11:18:56.272153+0000","last_became_peered":"2024-03-23T11:18:56.272153+0000","last_unstale":"2024-03-23T11:18:57.255738+0000","last_undegraded":"2024-03-23T11:18:57.255738+0000","last_fullsized":"2024-03-23T11:18:57.255738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:44:23.104322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101098+0000","last_change":"2024-03-23T11:18:56.264317+0000","last_active":"2024-03-23T11:18:58.101098+0000","last_peered":"2024-03-23T11:18:58.101098+0000","last_clean":"2024-03-23T11:18:58.101098+0000","last_became_active":"2024-03-23T11:18:56.263977+0000","last_became_peered":"2024-03-23T11:18:56.263977+0000","last_unstale":"2024-03-23T11:18:58.101098+0000","last_undegraded":"2024-03-23T11:18:58.101098+0000","last_fullsized":"2024-03-23T11:18:58.101098+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:33.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098064+0000","last_change":"2024-03-23T11:18:56.267303+0000","last_active":"2024-03-23T11:18:58.098064+0000","last_peered":"2024-03-23T11:18:58.098064+0000","last_clean":"2024-03-23T11:18:58.098064+0000","last_became_active":"2024-03-23T11:18:56.267130+0000","last_became_peered":"2024-03-23T11:18:56.267130+0000","last_unstale":"2024-03-23T11:18:58.098064+0000","last_undegraded":"2024-03-23T11:18:58.098064+0000","last_fullsized":"2024-03-23T11:18:58.098064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252603+0000","last_change":"2024-03-23T11:18:56.275581+0000","last_active":"2024-03-23T11:18:57.252603+0000","last_peered":"2024-03-23T11:18:57.252603+0000","last_clean":"2024-03-23T11:18:57.252603+0000","last_became_active":"2024-03-23T11:18:56.274566+0000","last_became_peered":"2024-03-23T11:18:56.274566+0000","last_unstale":"2024-03-23T11:18:57.252603+0000","last_undegraded":"2024-03-23T11:18:57.252603+0000","last_fullsized":"2024-03-23T11:18:57.252603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:31:43.650115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314345+0000","last_change":"2024-03-23T11:18:56.282803+0000","last_active":"2024-03-23T11:18:57.314345+0000","last_peered":"2024-03-23T11:18:57.314345+0000","last_clean":"2024-03-23T11:18:57.314345+0000","last_became_active":"2024-03-23T11:18:56.282500+0000","last_became_peered":"2024-03-23T11:18:56.282500+0000","last_unstale":"2024-03-23T11:18:57.314345+0000","last_undegraded":"2024-03-23T11:18:57.314345+0000","last_fullsized":"2024-03-23T11:18:57.314345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315513+0000","last_change":"2024-03-23T11:18:56.287431+0000","last_active":"2024-03-23T11:18:57.315513+0000","last_peered":"2024-03-23T11:18:57.315513+0000","last_clean":"2024-03-23T11:18:57.315513+0000","last_became_active":"2024-03-23T11:18:56.286853+0000","last_became_peered":"2024-03-23T11:18:56.286853+0000","last_unstale":"2024-03-23T11:18:57.315513+0000","last_undegraded":"2024-03-23T11:18:57.315513+0000","last_fullsized":"2024-03-23T11:18:57.315513+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:54:17.561394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100445+0000","last_change":"2024-03-23T11:18:56.260336+0000","last_active":"2024-03-23T11:18:58.100445+0000","last_peered":"2024-03-23T11:18:58.100445+0000","last_clean":"2024-03-23T11:18:58.100445+0000","last_became_active":"2024-03-23T11:18:56.259638+0000","last_became_peered":"2024-03-23T11:18:56.259638+0000","last_unstale":"2024-03-23T11:18:58.100445+0000","last_undegraded":"2024-03-23T11:18:58.100445+0000","last_fullsized":"2024-03-23T11:18:58.100445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:36:11.188974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257119+0000","last_change":"2024-03-23T11:18:56.268940+0000","last_active":"2024-03-23T11:18:57.257119+0000","last_peered":"2024-03-23T11:18:57.257119+0000","last_clean":"2024-03-23T11:18:57.257119+0000","last_became_active":"2024-03-23T11:18:56.268168+0000","last_became_peered":"2024-03-23T11:18:56.268168+0000","last_unstale":"2024-03-23T11:18:57.257119+0000","last_undegraded":"2024-03-23T11:18:57.257119+0000","last_fullsized":"2024-03-23T11:18:57.257119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:22.782018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461751+0000","last_change":"2024-03-23T11:18:56.271021+0000","last_active":"2024-03-23T11:18:57.461751+0000","last_peered":"2024-03-23T11:18:57.461751+0000","last_clean":"2024-03-23T11:18:57.461751+0000","last_became_active":"2024-03-23T11:18:56.270766+0000","last_became_peered":"2024-03-23T11:18:56.270766+0000","last_unstale":"2024-03-23T11:18:57.461751+0000","last_undegraded":"2024-03-23T11:18:57.461751+0000","last_fullsized":"2024-03-23T11:18:57.461751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098354+0000","last_change":"2024-03-23T11:18:56.261635+0000","last_active":"2024-03-23T11:18:58.098354+0000","last_peered":"2024-03-23T11:18:58.098354+0000","last_clean":"2024-03-23T11:18:58.098354+0000","last_became_active":"2024-03-23T11:18:56.260741+0000","last_became_peered":"2024-03-23T11:18:56.260741+0000","last_unstale":"2024-03-23T11:18:58.098354+0000","last_undegraded":"2024-03-23T11:18:58.098354+0000","last_fullsized":"2024-03-23T11:18:58.098354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314890+0000","last_change":"2024-03-23T11:18:56.286882+0000","last_active":"2024-03-23T11:18:57.314890+0000","last_peered":"2024-03-23T11:18:57.314890+0000","last_clean":"2024-03-23T11:18:57.314890+0000","last_became_active":"2024-03-23T11:18:56.285291+0000","last_became_peered":"2024-03-23T11:18:56.285291+0000","last_unstale":"2024-03-23T11:18:57.314890+0000","last_undegraded":"2024-03-23T11:18:57.314890+0000","last_fullsized":"2024-03-23T11:18:57.314890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:56.858523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101535+0000","last_change":"2024-03-23T11:18:56.276050+0000","last_active":"2024-03-23T11:18:58.101535+0000","last_peered":"2024-03-23T11:18:58.101535+0000","last_clean":"2024-03-23T11:18:58.101535+0000","last_became_active":"2024-03-23T11:18:56.275264+0000","last_became_peered":"2024-03-23T11:18:56.275264+0000","last_unstale":"2024-03-23T11:18:58.101535+0000","last_undegraded":"2024-03-23T11:18:58.101535+0000","last_fullsized":"2024-03-23T11:18:58.101535+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314573+0000","last_change":"2024-03-23T11:18:56.283292+0000","last_active":"2024-03-23T11:18:57.314573+0000","last_peered":"2024-03-23T11:18:57.314573+0000","last_clean":"2024-03-23T11:18:57.314573+0000","last_became_active":"2024-03-23T11:18:56.282493+0000","last_became_peered":"2024-03-23T11:18:56.282493+0000","last_unstale":"2024-03-23T11:18:57.314573+0000","last_undegraded":"2024-03-23T11:18:57.314573+0000","last_fullsized":"2024-03-23T11:18:57.314573+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101895+0000","last_change":"2024-03-23T11:18:56.282679+0000","last_active":"2024-03-23T11:18:58.101895+0000","last_peered":"2024-03-23T11:18:58.101895+0000","last_clean":"2024-03-23T11:18:58.101895+0000","last_became_active":"2024-03-23T11:18:56.282434+0000","last_became_peered":"2024-03-23T11:18:56.282434+0000","last_unstale":"2024-03-23T11:18:58.101895+0000","last_undegraded":"2024-03-23T11:18:58.101895+0000","last_fullsized":"2024-03-23T11:18:58.101895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:00.423263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462105+0000","last_change":"2024-03-23T11:18:56.260538+0000","last_active":"2024-03-23T11:18:57.462105+0000","last_peered":"2024-03-23T11:18:57.462105+0000","last_clean":"2024-03-23T11:18:57.462105+0000","last_became_active":"2024-03-23T11:18:56.259279+0000","last_became_peered":"2024-03-23T11:18:56.259279+0000","last_unstale":"2024-03-23T11:18:57.462105+0000","last_undegraded":"2024-03-23T11:18:57.462105+0000","last_fullsized":"2024-03-23T11:18:57.462105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253224+0000","last_change":"2024-03-23T11:18:56.263336+0000","last_active":"2024-03-23T11:18:57.253224+0000","last_peered":"2024-03-23T11:18:57.253224+0000","last_clean":"2024-03-23T11:18:57.253224+0000","last_became_active":"2024-03-23T11:18:56.262573+0000","last_became_peered":"2024-03-23T11:18:56.262573+0000","last_unstale":"2024-03-23T11:18:57.253224+0000","last_undegraded":"2024-03-23T11:18:57.253224+0000","last_fullsized":"2024-03-23T11:18:57.253224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:41:50.064223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101071+0000","last_change":"2024-03-23T11:18:56.284862+0000","last_active":"2024-03-23T11:18:58.101071+0000","last_peered":"2024-03-23T11:18:58.101071+0000","last_clean":"2024-03-23T11:18:58.101071+0000","last_became_active":"2024-03-23T11:18:56.284599+0000","last_became_peered":"2024-03-23T11:18:56.284599+0000","last_unstale":"2024-03-23T11:18:58.101071+0000","last_undegraded":"2024-03-23T11:18:58.101071+0000","last_fullsized":"2024-03-23T11:18:58.101071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.705001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098596+0000","last_change":"2024-03-23T11:18:56.271496+0000","last_active":"2024-03-23T11:18:58.098596+0000","last_peered":"2024-03-23T11:18:58.098596+0000","last_clean":"2024-03-23T11:18:58.098596+0000","last_became_active":"2024-03-23T11:18:56.271243+0000","last_became_peered":"2024-03-23T11:18:56.271243+0000","last_unstale":"2024-03-23T11:18:58.098596+0000","last_undegraded":"2024-03-23T11:18:58.098596+0000","last_fullsized":"2024-03-23T11:18:58.098596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:16:46.738802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314711+0000","last_change":"2024-03-23T11:18:56.282267+0000","last_active":"2024-03-23T11:18:57.314711+0000","last_peered":"2024-03-23T11:18:57.314711+0000","last_clean":"2024-03-23T11:18:57.314711+0000","last_became_active":"2024-03-23T11:18:56.277544+0000","last_became_peered":"2024-03-23T11:18:56.277544+0000","last_unstale":"2024-03-23T11:18:57.314711+0000","last_undegraded":"2024-03-23T11:18:57.314711+0000","last_fullsized":"2024-03-23T11:18:57.314711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:12.916824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098163+0000","last_change":"2024-03-23T11:18:56.267136+0000","last_active":"2024-03-23T11:18:58.098163+0000","last_peered":"2024-03-23T11:18:58.098163+0000","last_clean":"2024-03-23T11:18:58.098163+0000","last_became_active":"2024-03-23T11:18:56.266640+0000","last_became_peered":"2024-03-23T11:18:56.266640+0000","last_unstale":"2024-03-23T11:18:58.098163+0000","last_undegraded":"2024-03-23T11:18:58.098163+0000","last_fullsized":"2024-03-23T11:18:58.098163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461393+0000","last_change":"2024-03-23T11:18:56.266829+0000","last_active":"2024-03-23T11:18:57.461393+0000","last_peered":"2024-03-23T11:18:57.461393+0000","last_clean":"2024-03-23T11:18:57.461393+0000","last_became_active":"2024-03-23T11:18:56.266536+0000","last_became_peered":"2024-03-23T11:18:56.266536+0000","last_unstale":"2024-03-23T11:18:57.461393+0000","last_undegraded":"2024-03-23T11:18:57.461393+0000","last_fullsized":"2024-03-23T11:18:57.461393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:00.423263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255237+0000","last_change":"2024-03-23T11:18:56.267024+0000","last_active":"2024-03-23T11:18:57.255237+0000","last_peered":"2024-03-23T11:18:57.255237+0000","last_clean":"2024-03-23T11:18:57.255237+0000","last_became_active":"2024-03-23T11:18:56.266730+0000","last_became_peered":"2024-03-23T11:18:56.266730+0000","last_unstale":"2024-03-23T11:18:57.255237+0000","last_undegraded":"2024-03-23T11:18:57.255237+0000","last_fullsized":"2024-03-23T11:18:57.255237+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:19.533798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101417+0000","last_change":"2024-03-23T11:18:55.778856+0000","last_active":"2024-03-23T11:18:58.101417+0000","last_peered":"2024-03-23T11:18:58.101417+0000","last_clean":"2024-03-23T11:18:58.101417+0000","last_became_active":"2024-03-23T11:18:55.778487+0000","last_became_peered":"2024-03-23T11:18:55.778487+0000","last_unstale":"2024-03-23T11:18:58.101417+0000","last_undegraded":"2024-03-23T11:18:58.101417+0000","last_fullsized":"2024-03-23T11:18:58.101417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098890+0000","last_change":"2024-03-23T11:18:56.280876+0000","last_active":"2024-03-23T11:18:58.098890+0000","last_peered":"2024-03-23T11:18:58.098890+0000","last_clean":"2024-03-23T11:18:58.098890+0000","last_became_active":"2024-03-23T11:18:56.280574+0000","last_became_peered":"2024-03-23T11:18:56.280574+0000","last_unstale":"2024-03-23T11:18:58.098890+0000","last_undegraded":"2024-03-23T11:18:58.098890+0000","last_fullsized":"2024-03-23T11:18:58.098890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:25.559049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462037+0000","last_change":"2024-03-23T11:18:55.771275+0000","last_active":"2024-03-23T11:18:57.462037+0000","last_peered":"2024-03-23T11:18:57.462037+0000","last_clean":"2024-03-23T11:18:57.462037+0000","last_became_active":"2024-03-23T11:18:55.770948+0000","last_became_peered":"2024-03-23T11:18:55.770948+0000","last_unstale":"2024-03-23T11:18:57.462037+0000","last_undegraded":"2024-03-23T11:18:57.462037+0000","last_fullsized":"2024-03-23T11:18:57.462037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101598+0000","last_change":"2024-03-23T11:18:55.780718+0000","last_active":"2024-03-23T11:18:58.101598+0000","last_peered":"2024-03-23T11:18:58.101598+0000","last_clean":"2024-03-23T11:18:58.101598+0000","last_became_active":"2024-03-23T11:18:55.779357+0000","last_became_peered":"2024-03-23T11:18:55.779357+0000","last_unstale":"2024-03-23T11:18:58.101598+0000","last_undegraded":"2024-03-23T11:18:58.101598+0000","last_fullsized":"2024-03-23T11:18:58.101598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:36:21.274877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099752+0000","last_change":"2024-03-23T11:18:56.266172+0000","last_active":"2024-03-23T11:18:58.099752+0000","last_peered":"2024-03-23T11:18:58.099752+0000","last_clean":"2024-03-23T11:18:58.099752+0000","last_became_active":"2024-03-23T11:18:56.265882+0000","last_became_peered":"2024-03-23T11:18:56.265882+0000","last_unstale":"2024-03-23T11:18:58.099752+0000","last_undegraded":"2024-03-23T11:18:58.099752+0000","last_fullsized":"2024-03-23T11:18:58.099752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314638+0000","last_change":"2024-03-23T11:18:55.774144+0000","last_active":"2024-03-23T11:18:57.314638+0000","last_peered":"2024-03-23T11:18:57.314638+0000","last_clean":"2024-03-23T11:18:57.314638+0000","last_became_active":"2024-03-23T11:18:55.773880+0000","last_became_peered":"2024-03-23T11:18:55.773880+0000","last_unstale":"2024-03-23T11:18:57.314638+0000","last_undegraded":"2024-03-23T11:18:57.314638+0000","last_fullsized":"2024-03-23T11:18:57.314638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252538+0000","last_change":"2024-03-23T11:18:56.260698+0000","last_active":"2024-03-23T11:18:57.252538+0000","last_peered":"2024-03-23T11:18:57.252538+0000","last_clean":"2024-03-23T11:18:57.252538+0000","last_became_active":"2024-03-23T11:18:56.258466+0000","last_became_peered":"2024-03-23T11:18:56.258466+0000","last_unstale":"2024-03-23T11:18:57.252538+0000","last_undegraded":"2024-03-23T11:18:57.252538+0000","last_fullsized":"2024-03-23T11:18:57.252538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:55:22.272329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462086+0000","last_change":"2024-03-23T11:18:55.783554+0000","last_active":"2024-03-23T11:18:57.462086+0000","last_peered":"2024-03-23T11:18:57.462086+0000","last_clean":"2024-03-23T11:18:57.462086+0000","last_became_active":"2024-03-23T11:18:55.783248+0000","last_became_peered":"2024-03-23T11:18:55.783248+0000","last_unstale":"2024-03-23T11:18:57.462086+0000","last_undegraded":"2024-03-23T11:18:57.462086+0000","last_fullsized":"2024-03-23T11:18:57.462086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:18.352259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314908+0000","last_change":"2024-03-23T11:18:56.287651+0000","last_active":"2024-03-23T11:18:57.314908+0000","last_peered":"2024-03-23T11:18:57.314908+0000","last_clean":"2024-03-23T11:18:57.314908+0000","last_became_active":"2024-03-23T11:18:56.287312+0000","last_became_peered":"2024-03-23T11:18:56.287312+0000","last_unstale":"2024-03-23T11:18:57.314908+0000","last_undegraded":"2024-03-23T11:18:57.314908+0000","last_fullsized":"2024-03-23T11:18:57.314908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:24:29.439358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314950+0000","last_change":"2024-03-23T11:18:55.783117+0000","last_active":"2024-03-23T11:18:57.314950+0000","last_peered":"2024-03-23T11:18:57.314950+0000","last_clean":"2024-03-23T11:18:57.314950+0000","last_became_active":"2024-03-23T11:18:55.782341+0000","last_became_peered":"2024-03-23T11:18:55.782341+0000","last_unstale":"2024-03-23T11:18:57.314950+0000","last_undegraded":"2024-03-23T11:18:57.314950+0000","last_fullsized":"2024-03-23T11:18:57.314950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:10:08.943114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099470+0000","last_change":"2024-03-23T11:18:56.290381+0000","last_active":"2024-03-23T11:18:58.099470+0000","last_peered":"2024-03-23T11:18:58.099470+0000","last_clean":"2024-03-23T11:18:58.099470+0000","last_became_active":"2024-03-23T11:18:56.290070+0000","last_became_peered":"2024-03-23T11:18:56.290070+0000","last_unstale":"2024-03-23T11:18:58.099470+0000","last_undegraded":"2024-03-23T11:18:58.099470+0000","last_fullsized":"2024-03-23T11:18:58.099470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253428+0000","last_change":"2024-03-23T11:18:55.783461+0000","last_active":"2024-03-23T11:18:57.253428+0000","last_peered":"2024-03-23T11:18:57.253428+0000","last_clean":"2024-03-23T11:18:57.253428+0000","last_became_active":"2024-03-23T11:18:55.783214+0000","last_became_peered":"2024-03-23T11:18:55.783214+0000","last_unstale":"2024-03-23T11:18:57.253428+0000","last_undegraded":"2024-03-23T11:18:57.253428+0000","last_fullsized":"2024-03-23T11:18:57.253428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:28:51.146588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255996+0000","last_change":"2024-03-23T11:18:56.268161+0000","last_active":"2024-03-23T11:18:57.255996+0000","last_peered":"2024-03-23T11:18:57.255996+0000","last_clean":"2024-03-23T11:18:57.255996+0000","last_became_active":"2024-03-23T11:18:56.263992+0000","last_became_peered":"2024-03-23T11:18:56.263992+0000","last_unstale":"2024-03-23T11:18:57.255996+0000","last_undegraded":"2024-03-23T11:18:57.255996+0000","last_fullsized":"2024-03-23T11:18:57.255996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461733+0000","last_change":"2024-03-23T11:18:55.782334+0000","last_active":"2024-03-23T11:18:57.461733+0000","last_peered":"2024-03-23T11:18:57.461733+0000","last_clean":"2024-03-23T11:18:57.461733+0000","last_became_active":"2024-03-23T11:18:55.780347+0000","last_became_peered":"2024-03-23T11:18:55.780347+0000","last_unstale":"2024-03-23T11:18:57.461733+0000","last_undegraded":"2024-03-23T11:18:57.461733+0000","last_fullsized":"2024-03-23T11:18:57.461733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:10:08.943114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256551+0000","last_change":"2024-03-23T11:18:56.272684+0000","last_active":"2024-03-23T11:18:57.256551+0000","last_peered":"2024-03-23T11:18:57.256551+0000","last_clean":"2024-03-23T11:18:57.256551+0000","last_became_active":"2024-03-23T11:18:56.268603+0000","last_became_peered":"2024-03-23T11:18:56.268603+0000","last_unstale":"2024-03-23T11:18:57.256551+0000","last_undegraded":"2024-03-23T11:18:57.256551+0000","last_fullsized":"2024-03-23T11:18:57.256551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:22.095583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100170+0000","last_change":"2024-03-23T11:18:55.783962+0000","last_active":"2024-03-23T11:18:58.100170+0000","last_peered":"2024-03-23T11:18:58.100170+0000","last_clean":"2024-03-23T11:18:58.100170+0000","last_became_active":"2024-03-23T11:18:55.783729+0000","last_became_peered":"2024-03-23T11:18:55.783729+0000","last_unstale":"2024-03-23T11:18:58.100170+0000","last_undegraded":"2024-03-23T11:18:58.100170+0000","last_fullsized":"2024-03-23T11:18:58.100170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:42.674448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098331+0000","last_change":"2024-03-23T11:18:56.261635+0000","last_active":"2024-03-23T11:18:58.098331+0000","last_peered":"2024-03-23T11:18:58.098331+0000","last_clean":"2024-03-23T11:18:58.098331+0000","last_became_active":"2024-03-23T11:18:56.261334+0000","last_became_peered":"2024-03-23T11:18:56.261334+0000","last_unstale":"2024-03-23T11:18:58.098331+0000","last_undegraded":"2024-03-23T11:18:58.098331+0000","last_fullsized":"2024-03-23T11:18:58.098331+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314958+0000","last_change":"2024-03-23T11:18:55.782301+0000","last_active":"2024-03-23T11:18:57.314958+0000","last_peered":"2024-03-23T11:18:57.314958+0000","last_clean":"2024-03-23T11:18:57.314958+0000","last_became_active":"2024-03-23T11:18:55.782005+0000","last_became_peered":"2024-03-23T11:18:55.782005+0000","last_unstale":"2024-03-23T11:18:57.314958+0000","last_undegraded":"2024-03-23T11:18:57.314958+0000","last_fullsized":"2024-03-23T11:18:57.314958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256246+0000","last_change":"2024-03-23T11:18:56.275199+0000","last_active":"2024-03-23T11:18:57.256246+0000","last_peered":"2024-03-23T11:18:57.256246+0000","last_clean":"2024-03-23T11:18:57.256246+0000","last_became_active":"2024-03-23T11:18:56.274944+0000","last_became_peered":"2024-03-23T11:18:56.274944+0000","last_unstale":"2024-03-23T11:18:57.256246+0000","last_undegraded":"2024-03-23T11:18:57.256246+0000","last_fullsized":"2024-03-23T11:18:57.256246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:00:18.525010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461578+0000","last_change":"2024-03-23T11:18:55.775042+0000","last_active":"2024-03-23T11:18:57.461578+0000","last_peered":"2024-03-23T11:18:57.461578+0000","last_clean":"2024-03-23T11:18:57.461578+0000","last_became_active":"2024-03-23T11:18:55.774721+0000","last_became_peered":"2024-03-23T11:18:55.774721+0000","last_unstale":"2024-03-23T11:18:57.461578+0000","last_undegraded":"2024-03-23T11:18:57.461578+0000","last_fullsized":"2024-03-23T11:18:57.461578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461383+0000","last_change":"2024-03-23T11:18:56.263485+0000","last_active":"2024-03-23T11:18:57.461383+0000","last_peered":"2024-03-23T11:18:57.461383+0000","last_clean":"2024-03-23T11:18:57.461383+0000","last_became_active":"2024-03-23T11:18:56.261677+0000","last_became_peered":"2024-03-23T11:18:56.261677+0000","last_unstale":"2024-03-23T11:18:57.461383+0000","last_undegraded":"2024-03-23T11:18:57.461383+0000","last_fullsized":"2024-03-23T11:18:57.461383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098165+0000","last_change":"2024-03-23T11:18:55.770098+0000","last_active":"2024-03-23T11:18:58.098165+0000","last_peered":"2024-03-23T11:18:58.098165+0000","last_clean":"2024-03-23T11:18:58.098165+0000","last_became_active":"2024-03-23T11:18:55.769814+0000","last_became_peered":"2024-03-23T11:18:55.769814+0000","last_unstale":"2024-03-23T11:18:58.098165+0000","last_undegraded":"2024-03-23T11:18:58.098165+0000","last_fullsized":"2024-03-23T11:18:58.098165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255419+0000","last_change":"2024-03-23T11:18:56.274715+0000","last_active":"2024-03-23T11:18:57.255419+0000","last_peered":"2024-03-23T11:18:57.255419+0000","last_clean":"2024-03-23T11:18:57.255419+0000","last_became_active":"2024-03-23T11:18:56.274437+0000","last_became_peered":"2024-03-23T11:18:56.274437+0000","last_unstale":"2024-03-23T11:18:57.255419+0000","last_undegraded":"2024-03-23T11:18:57.255419+0000","last_fullsized":"2024-03-23T11:18:57.255419+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:28:00.964447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256032+0000","last_change":"2024-03-23T11:18:55.789682+0000","last_active":"2024-03-23T11:18:57.256032+0000","last_peered":"2024-03-23T11:18:57.256032+0000","last_clean":"2024-03-23T11:18:57.256032+0000","last_became_active":"2024-03-23T11:18:55.789278+0000","last_became_peered":"2024-03-23T11:18:55.789278+0000","last_unstale":"2024-03-23T11:18:57.256032+0000","last_undegraded":"2024-03-23T11:18:57.256032+0000","last_fullsized":"2024-03-23T11:18:57.256032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:42.674448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256896+0000","last_change":"2024-03-23T11:18:56.267163+0000","last_active":"2024-03-23T11:18:57.256896+0000","last_peered":"2024-03-23T11:18:57.256896+0000","last_clean":"2024-03-23T11:18:57.256896+0000","last_became_active":"2024-03-23T11:18:56.266907+0000","last_became_peered":"2024-03-23T11:18:56.266907+0000","last_unstale":"2024-03-23T11:18:57.256896+0000","last_undegraded":"2024-03-23T11:18:57.256896+0000","last_fullsized":"2024-03-23T11:18:57.256896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:51:52.682495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100710+0000","last_change":"2024-03-23T11:18:55.779353+0000","last_active":"2024-03-23T11:18:58.100710+0000","last_peered":"2024-03-23T11:18:58.100710+0000","last_clean":"2024-03-23T11:18:58.100710+0000","last_became_active":"2024-03-23T11:18:55.779045+0000","last_became_peered":"2024-03-23T11:18:55.779045+0000","last_unstale":"2024-03-23T11:18:58.100710+0000","last_undegraded":"2024-03-23T11:18:58.100710+0000","last_fullsized":"2024-03-23T11:18:58.100710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257949+0000","last_change":"2024-03-23T11:18:56.275906+0000","last_active":"2024-03-23T11:18:57.257949+0000","last_peered":"2024-03-23T11:18:57.257949+0000","last_clean":"2024-03-23T11:18:57.257949+0000","last_became_active":"2024-03-23T11:18:56.274500+0000","last_became_peered":"2024-03-23T11:18:56.274500+0000","last_unstale":"2024-03-23T11:18:57.257949+0000","last_undegraded":"2024-03-23T11:18:57.257949+0000","last_fullsized":"2024-03-23T11:18:57.257949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:13:34.436113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253366+0000","last_change":"2024-03-23T11:18:55.763585+0000","last_active":"2024-03-23T11:18:57.253366+0000","last_peered":"2024-03-23T11:18:57.253366+0000","last_clean":"2024-03-23T11:18:57.253366+0000","last_became_active":"2024-03-23T11:18:55.762854+0000","last_became_peered":"2024-03-23T11:18:55.762854+0000","last_unstale":"2024-03-23T11:18:57.253366+0000","last_undegraded":"2024-03-23T11:18:57.253366+0000","last_fullsized":"2024-03-23T11:18:57.253366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:38.995255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315476+0000","last_change":"2024-03-23T11:18:56.276682+0000","last_active":"2024-03-23T11:18:57.315476+0000","last_peered":"2024-03-23T11:18:57.315476+0000","last_clean":"2024-03-23T11:18:57.315476+0000","last_became_active":"2024-03-23T11:18:56.276094+0000","last_became_peered":"2024-03-23T11:18:56.276094+0000","last_unstale":"2024-03-23T11:18:57.315476+0000","last_undegraded":"2024-03-23T11:18:57.315476+0000","last_fullsized":"2024-03-23T11:18:57.315476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:26.126571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098569+0000","last_change":"2024-03-23T11:18:55.783331+0000","last_active":"2024-03-23T11:18:58.098569+0000","last_peered":"2024-03-23T11:18:58.098569+0000","last_clean":"2024-03-23T11:18:58.098569+0000","last_became_active":"2024-03-23T11:18:55.783030+0000","last_became_peered":"2024-03-23T11:18:55.783030+0000","last_unstale":"2024-03-23T11:18:58.098569+0000","last_undegraded":"2024-03-23T11:18:58.098569+0000","last_fullsized":"2024-03-23T11:18:58.098569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256897+0000","last_change":"2024-03-23T11:18:56.284587+0000","last_active":"2024-03-23T11:18:57.256897+0000","last_peered":"2024-03-23T11:18:57.256897+0000","last_clean":"2024-03-23T11:18:57.256897+0000","last_became_active":"2024-03-23T11:18:56.283582+0000","last_became_peered":"2024-03-23T11:18:56.283582+0000","last_unstale":"2024-03-23T11:18:57.256897+0000","last_undegraded":"2024-03-23T11:18:57.256897+0000","last_fullsized":"2024-03-23T11:18:57.256897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:01:11.724895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":164,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T11:18:56.246747+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:18:56.246747+0000","last_peered":"2024-03-23T11:18:56.246747+0000","last_clean":"2024-03-23T11:18:56.246747+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:18:56.246747+0000","last_undegraded":"2024-03-23T11:18:56.246747+0000","last_fullsized":"2024-03-23T11:18:56.246747+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314507+0000","last_change":"2024-03-23T11:18:55.778697+0000","last_active":"2024-03-23T11:18:57.314507+0000","last_peered":"2024-03-23T11:18:57.314507+0000","last_clean":"2024-03-23T11:18:57.314507+0000","last_became_active":"2024-03-23T11:18:55.778387+0000","last_became_peered":"2024-03-23T11:18:55.778387+0000","last_unstale":"2024-03-23T11:18:57.314507+0000","last_undegraded":"2024-03-23T11:18:57.314507+0000","last_fullsized":"2024-03-23T11:18:57.314507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100582+0000","last_change":"2024-03-23T11:18:56.268261+0000","last_active":"2024-03-23T11:18:58.100582+0000","last_peered":"2024-03-23T11:18:58.100582+0000","last_clean":"2024-03-23T11:18:58.100582+0000","last_became_active":"2024-03-23T11:18:56.267974+0000","last_became_peered":"2024-03-23T11:18:56.267974+0000","last_unstale":"2024-03-23T11:18:58.100582+0000","last_undegraded":"2024-03-23T11:18:58.100582+0000","last_fullsized":"2024-03-23T11:18:58.100582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:44.879627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255801+0000","last_change":"2024-03-23T11:18:55.775332+0000","last_active":"2024-03-23T11:18:57.255801+0000","last_peered":"2024-03-23T11:18:57.255801+0000","last_clean":"2024-03-23T11:18:57.255801+0000","last_became_active":"2024-03-23T11:18:55.775009+0000","last_became_peered":"2024-03-23T11:18:55.775009+0000","last_unstale":"2024-03-23T11:18:57.255801+0000","last_undegraded":"2024-03-23T11:18:57.255801+0000","last_fullsized":"2024-03-23T11:18:57.255801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:08:12.342783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098600+0000","last_change":"2024-03-23T11:18:56.279541+0000","last_active":"2024-03-23T11:18:58.098600+0000","last_peered":"2024-03-23T11:18:58.098600+0000","last_clean":"2024-03-23T11:18:58.098600+0000","last_became_active":"2024-03-23T11:18:56.279231+0000","last_became_peered":"2024-03-23T11:18:56.279231+0000","last_unstale":"2024-03-23T11:18:58.098600+0000","last_undegraded":"2024-03-23T11:18:58.098600+0000","last_fullsized":"2024-03-23T11:18:58.098600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098389+0000","last_change":"2024-03-23T11:18:55.777288+0000","last_active":"2024-03-23T11:18:58.098389+0000","last_peered":"2024-03-23T11:18:58.098389+0000","last_clean":"2024-03-23T11:18:58.098389+0000","last_became_active":"2024-03-23T11:18:55.776998+0000","last_became_peered":"2024-03-23T11:18:55.776998+0000","last_unstale":"2024-03-23T11:18:58.098389+0000","last_undegraded":"2024-03-23T11:18:58.098389+0000","last_fullsized":"2024-03-23T11:18:58.098389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252646+0000","last_change":"2024-03-23T11:18:56.257547+0000","last_active":"2024-03-23T11:18:57.252646+0000","last_peered":"2024-03-23T11:18:57.252646+0000","last_clean":"2024-03-23T11:18:57.252646+0000","last_became_active":"2024-03-23T11:18:56.257307+0000","last_became_peered":"2024-03-23T11:18:56.257307+0000","last_unstale":"2024-03-23T11:18:57.252646+0000","last_undegraded":"2024-03-23T11:18:57.252646+0000","last_fullsized":"2024-03-23T11:18:57.252646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:32.498133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315167+0000","last_change":"2024-03-23T11:18:55.780573+0000","last_active":"2024-03-23T11:18:57.315167+0000","last_peered":"2024-03-23T11:18:57.315167+0000","last_clean":"2024-03-23T11:18:57.315167+0000","last_became_active":"2024-03-23T11:18:55.779963+0000","last_became_peered":"2024-03-23T11:18:55.779963+0000","last_unstale":"2024-03-23T11:18:57.315167+0000","last_undegraded":"2024-03-23T11:18:57.315167+0000","last_fullsized":"2024-03-23T11:18:57.315167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:18.352259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255291+0000","last_change":"2024-03-23T11:18:56.264913+0000","last_active":"2024-03-23T11:18:57.255291+0000","last_peered":"2024-03-23T11:18:57.255291+0000","last_clean":"2024-03-23T11:18:57.255291+0000","last_became_active":"2024-03-23T11:18:56.264679+0000","last_became_peered":"2024-03-23T11:18:56.264679+0000","last_unstale":"2024-03-23T11:18:57.255291+0000","last_undegraded":"2024-03-23T11:18:57.255291+0000","last_fullsized":"2024-03-23T11:18:57.255291+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:16:46.738802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253124+0000","last_change":"2024-03-23T11:18:55.785635+0000","last_active":"2024-03-23T11:18:57.253124+0000","last_peered":"2024-03-23T11:18:57.253124+0000","last_clean":"2024-03-23T11:18:57.253124+0000","last_became_active":"2024-03-23T11:18:55.785064+0000","last_became_peered":"2024-03-23T11:18:55.785064+0000","last_unstale":"2024-03-23T11:18:57.253124+0000","last_undegraded":"2024-03-23T11:18:57.253124+0000","last_fullsized":"2024-03-23T11:18:57.253124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:04:27.564911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315028+0000","last_change":"2024-03-23T11:18:56.276583+0000","last_active":"2024-03-23T11:18:57.315028+0000","last_peered":"2024-03-23T11:18:57.315028+0000","last_clean":"2024-03-23T11:18:57.315028+0000","last_became_active":"2024-03-23T11:18:56.276294+0000","last_became_peered":"2024-03-23T11:18:56.276294+0000","last_unstale":"2024-03-23T11:18:57.315028+0000","last_undegraded":"2024-03-23T11:18:57.315028+0000","last_fullsized":"2024-03-23T11:18:57.315028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:36.411878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257226+0000","last_change":"2024-03-23T11:18:55.784141+0000","last_active":"2024-03-23T11:18:57.257226+0000","last_peered":"2024-03-23T11:18:57.257226+0000","last_clean":"2024-03-23T11:18:57.257226+0000","last_became_active":"2024-03-23T11:18:55.783843+0000","last_became_peered":"2024-03-23T11:18:55.783843+0000","last_unstale":"2024-03-23T11:18:57.257226+0000","last_undegraded":"2024-03-23T11:18:57.257226+0000","last_fullsized":"2024-03-23T11:18:57.257226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:57:29.038916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252794+0000","last_change":"2024-03-23T11:18:56.258329+0000","last_active":"2024-03-23T11:18:57.252794+0000","last_peered":"2024-03-23T11:18:57.252794+0000","last_clean":"2024-03-23T11:18:57.252794+0000","last_became_active":"2024-03-23T11:18:56.258033+0000","last_became_peered":"2024-03-23T11:18:56.258033+0000","last_unstale":"2024-03-23T11:18:57.252794+0000","last_undegraded":"2024-03-23T11:18:57.252794+0000","last_fullsized":"2024-03-23T11:18:57.252794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:24:37.470160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253467+0000","last_change":"2024-03-23T11:18:55.778103+0000","last_active":"2024-03-23T11:18:57.253467+0000","last_peered":"2024-03-23T11:18:57.253467+0000","last_clean":"2024-03-23T11:18:57.253467+0000","last_became_active":"2024-03-23T11:18:55.777604+0000","last_became_peered":"2024-03-23T11:18:55.777604+0000","last_unstale":"2024-03-23T11:18:57.253467+0000","last_undegraded":"2024-03-23T11:18:57.253467+0000","last_fullsized":"2024-03-23T11:18:57.253467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:42:44.181703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101839+0000","last_change":"2024-03-23T11:18:56.280720+0000","last_active":"2024-03-23T11:18:58.101839+0000","last_peered":"2024-03-23T11:18:58.101839+0000","last_clean":"2024-03-23T11:18:58.101839+0000","last_became_active":"2024-03-23T11:18:56.278952+0000","last_became_peered":"2024-03-23T11:18:56.278952+0000","last_unstale":"2024-03-23T11:18:58.101839+0000","last_undegraded":"2024-03-23T11:18:58.101839+0000","last_fullsized":"2024-03-23T11:18:58.101839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462270+0000","last_change":"2024-03-23T11:18:55.776925+0000","last_active":"2024-03-23T11:18:57.462270+0000","last_peered":"2024-03-23T11:18:57.462270+0000","last_clean":"2024-03-23T11:18:57.462270+0000","last_became_active":"2024-03-23T11:18:55.776651+0000","last_became_peered":"2024-03-23T11:18:55.776651+0000","last_unstale":"2024-03-23T11:18:57.462270+0000","last_undegraded":"2024-03-23T11:18:57.462270+0000","last_fullsized":"2024-03-23T11:18:57.462270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099552+0000","last_change":"2024-03-23T11:18:56.268554+0000","last_active":"2024-03-23T11:18:58.099552+0000","last_peered":"2024-03-23T11:18:58.099552+0000","last_clean":"2024-03-23T11:18:58.099552+0000","last_became_active":"2024-03-23T11:18:56.267201+0000","last_became_peered":"2024-03-23T11:18:56.267201+0000","last_unstale":"2024-03-23T11:18:58.099552+0000","last_undegraded":"2024-03-23T11:18:58.099552+0000","last_fullsized":"2024-03-23T11:18:58.099552+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.705001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255562+0000","last_change":"2024-03-23T11:18:55.789376+0000","last_active":"2024-03-23T11:18:57.255562+0000","last_peered":"2024-03-23T11:18:57.255562+0000","last_clean":"2024-03-23T11:18:57.255562+0000","last_became_active":"2024-03-23T11:18:55.789122+0000","last_became_peered":"2024-03-23T11:18:55.789122+0000","last_unstale":"2024-03-23T11:18:57.255562+0000","last_undegraded":"2024-03-23T11:18:57.255562+0000","last_fullsized":"2024-03-23T11:18:57.255562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:26.469433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257058+0000","last_change":"2024-03-23T11:18:56.267674+0000","last_active":"2024-03-23T11:18:57.257058+0000","last_peered":"2024-03-23T11:18:57.257058+0000","last_clean":"2024-03-23T11:18:57.257058+0000","last_became_active":"2024-03-23T11:18:56.267418+0000","last_became_peered":"2024-03-23T11:18:56.267418+0000","last_unstale":"2024-03-23T11:18:57.257058+0000","last_undegraded":"2024-03-23T11:18:57.257058+0000","last_fullsized":"2024-03-23T11:18:57.257058+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:36:11.188974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252776+0000","last_change":"2024-03-23T11:18:55.775660+0000","last_active":"2024-03-23T11:18:57.252776+0000","last_peered":"2024-03-23T11:18:57.252776+0000","last_clean":"2024-03-23T11:18:57.252776+0000","last_became_active":"2024-03-23T11:18:55.775378+0000","last_became_peered":"2024-03-23T11:18:55.775378+0000","last_unstale":"2024-03-23T11:18:57.252776+0000","last_undegraded":"2024-03-23T11:18:57.252776+0000","last_fullsized":"2024-03-23T11:18:57.252776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:25:31.204286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100033+0000","last_change":"2024-03-23T11:18:56.273073+0000","last_active":"2024-03-23T11:18:58.100033+0000","last_peered":"2024-03-23T11:18:58.100033+0000","last_clean":"2024-03-23T11:18:58.100033+0000","last_became_active":"2024-03-23T11:18:56.265279+0000","last_became_peered":"2024-03-23T11:18:56.265279+0000","last_unstale":"2024-03-23T11:18:58.100033+0000","last_undegraded":"2024-03-23T11:18:58.100033+0000","last_fullsized":"2024-03-23T11:18:58.100033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252994+0000","last_change":"2024-03-23T11:18:55.773923+0000","last_active":"2024-03-23T11:18:57.252994+0000","last_peered":"2024-03-23T11:18:57.252994+0000","last_clean":"2024-03-23T11:18:57.252994+0000","last_became_active":"2024-03-23T11:18:55.773155+0000","last_became_peered":"2024-03-23T11:18:55.773155+0000","last_unstale":"2024-03-23T11:18:57.252994+0000","last_undegraded":"2024-03-23T11:18:57.252994+0000","last_fullsized":"2024-03-23T11:18:57.252994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:26.469433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101280+0000","last_change":"2024-03-23T11:18:56.265422+0000","last_active":"2024-03-23T11:18:58.101280+0000","last_peered":"2024-03-23T11:18:58.101280+0000","last_clean":"2024-03-23T11:18:58.101280+0000","last_became_active":"2024-03-23T11:18:56.265137+0000","last_became_peered":"2024-03-23T11:18:56.265137+0000","last_unstale":"2024-03-23T11:18:58.101280+0000","last_undegraded":"2024-03-23T11:18:58.101280+0000","last_fullsized":"2024-03-23T11:18:58.101280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:26.126571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256335+0000","last_change":"2024-03-23T11:18:55.778476+0000","last_active":"2024-03-23T11:18:57.256335+0000","last_peered":"2024-03-23T11:18:57.256335+0000","last_clean":"2024-03-23T11:18:57.256335+0000","last_became_active":"2024-03-23T11:18:55.778184+0000","last_became_peered":"2024-03-23T11:18:55.778184+0000","last_unstale":"2024-03-23T11:18:57.256335+0000","last_undegraded":"2024-03-23T11:18:57.256335+0000","last_fullsized":"2024-03-23T11:18:57.256335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:38.995255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462125+0000","last_change":"2024-03-23T11:18:56.277871+0000","last_active":"2024-03-23T11:18:57.462125+0000","last_peered":"2024-03-23T11:18:57.462125+0000","last_clean":"2024-03-23T11:18:57.462125+0000","last_became_active":"2024-03-23T11:18:56.277547+0000","last_became_peered":"2024-03-23T11:18:56.277547+0000","last_unstale":"2024-03-23T11:18:57.462125+0000","last_undegraded":"2024-03-23T11:18:57.462125+0000","last_fullsized":"2024-03-23T11:18:57.462125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.747240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315015+0000","last_change":"2024-03-23T11:18:56.288533+0000","last_active":"2024-03-23T11:18:57.315015+0000","last_peered":"2024-03-23T11:18:57.315015+0000","last_clean":"2024-03-23T11:18:57.315015+0000","last_became_active":"2024-03-23T11:18:56.288099+0000","last_became_peered":"2024-03-23T11:18:56.288099+0000","last_unstale":"2024-03-23T11:18:57.315015+0000","last_undegraded":"2024-03-23T11:18:57.315015+0000","last_fullsized":"2024-03-23T11:18:57.315015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:31:13.473526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256142+0000","last_change":"2024-03-23T11:18:55.784333+0000","last_active":"2024-03-23T11:18:57.256142+0000","last_peered":"2024-03-23T11:18:57.256142+0000","last_clean":"2024-03-23T11:18:57.256142+0000","last_became_active":"2024-03-23T11:18:55.783931+0000","last_became_peered":"2024-03-23T11:18:55.783931+0000","last_unstale":"2024-03-23T11:18:57.256142+0000","last_undegraded":"2024-03-23T11:18:57.256142+0000","last_fullsized":"2024-03-23T11:18:57.256142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:04:27.564911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315497+0000","last_change":"2024-03-23T11:18:56.287212+0000","last_active":"2024-03-23T11:18:57.315497+0000","last_peered":"2024-03-23T11:18:57.315497+0000","last_clean":"2024-03-23T11:18:57.315497+0000","last_became_active":"2024-03-23T11:18:56.284693+0000","last_became_peered":"2024-03-23T11:18:56.284693+0000","last_unstale":"2024-03-23T11:18:57.315497+0000","last_undegraded":"2024-03-23T11:18:57.315497+0000","last_fullsized":"2024-03-23T11:18:57.315497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:33.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256937+0000","last_change":"2024-03-23T11:18:55.779221+0000","last_active":"2024-03-23T11:18:57.256937+0000","last_peered":"2024-03-23T11:18:57.256937+0000","last_clean":"2024-03-23T11:18:57.256937+0000","last_became_active":"2024-03-23T11:18:55.778912+0000","last_became_peered":"2024-03-23T11:18:55.778912+0000","last_unstale":"2024-03-23T11:18:57.256937+0000","last_undegraded":"2024-03-23T11:18:57.256937+0000","last_fullsized":"2024-03-23T11:18:57.256937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:28:51.146588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099197+0000","last_change":"2024-03-23T11:18:56.291118+0000","last_active":"2024-03-23T11:18:58.099197+0000","last_peered":"2024-03-23T11:18:58.099197+0000","last_clean":"2024-03-23T11:18:58.099197+0000","last_became_active":"2024-03-23T11:18:56.290750+0000","last_became_peered":"2024-03-23T11:18:56.290750+0000","last_unstale":"2024-03-23T11:18:58.099197+0000","last_undegraded":"2024-03-23T11:18:58.099197+0000","last_fullsized":"2024-03-23T11:18:58.099197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101527+0000","last_change":"2024-03-23T11:18:55.784254+0000","last_active":"2024-03-23T11:18:58.101527+0000","last_peered":"2024-03-23T11:18:58.101527+0000","last_clean":"2024-03-23T11:18:58.101527+0000","last_became_active":"2024-03-23T11:18:55.781763+0000","last_became_peered":"2024-03-23T11:18:55.781763+0000","last_unstale":"2024-03-23T11:18:58.101527+0000","last_undegraded":"2024-03-23T11:18:58.101527+0000","last_fullsized":"2024-03-23T11:18:58.101527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101387+0000","last_change":"2024-03-23T11:18:56.267771+0000","last_active":"2024-03-23T11:18:58.101387+0000","last_peered":"2024-03-23T11:18:58.101387+0000","last_clean":"2024-03-23T11:18:58.101387+0000","last_became_active":"2024-03-23T11:18:56.267515+0000","last_became_peered":"2024-03-23T11:18:56.267515+0000","last_unstale":"2024-03-23T11:18:58.101387+0000","last_undegraded":"2024-03-23T11:18:58.101387+0000","last_fullsized":"2024-03-23T11:18:58.101387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099857+0000","last_change":"2024-03-23T11:18:55.785970+0000","last_active":"2024-03-23T11:18:58.099857+0000","last_peered":"2024-03-23T11:18:58.099857+0000","last_clean":"2024-03-23T11:18:58.099857+0000","last_became_active":"2024-03-23T11:18:55.784347+0000","last_became_peered":"2024-03-23T11:18:55.784347+0000","last_unstale":"2024-03-23T11:18:58.099857+0000","last_undegraded":"2024-03-23T11:18:58.099857+0000","last_fullsized":"2024-03-23T11:18:58.099857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:36:21.274877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":823296,"data_stored":669740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8580000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.764}]}]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0659999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.879}]}]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8029999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.75}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0750000000000002}]}]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1040000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.591}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.768}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.966}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0659999999999998}]}]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.976}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.659}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.909}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0459999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.585}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.206}]}]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6060000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.675}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":78281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.369}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4390000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542147,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":827392,"data_stored":669127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8719999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:19:01.237 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:19:01.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.498+0000 7fa2ef000640 1 Processor -- start 2024-03-23T11:19:01.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.498+0000 7fa2ef000640 1 -- start start 2024-03-23T11:19:01.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.500+0000 7fa2ef000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa2e8143dd0 0x7fa2e815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:01.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.500+0000 7fa2ef000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2e80b2f30 0x7fa2e8070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:01.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.500+0000 7fa2ef000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e80cd140 con 0x7fa2e8070090 2024-03-23T11:19:01.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.500+0000 7fa2ef000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e8070a60 con 0x7fa2e8143dd0 2024-03-23T11:19:01.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.500+0000 7fa2ef000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80cd3e0 con 0x7fa2e80b2f30 2024-03-23T11:19:01.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.500+0000 7fa2edc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa2e8143dd0 0x7fa2e815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:01.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.500+0000 7fa2edc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa2e8143dd0 0x7fa2e815abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:52960/0 (socket says 172.21.15.62:52960) 2024-03-23T11:19:01.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.500+0000 7fa2edc00640 1 -- 172.21.15.62:0/4021480997 learned_addr learned my addr 172.21.15.62:0/4021480997 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:01.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.500+0000 7fa2ed200640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fa2e8070090 0x7fa2e80b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41544/0 (socket says 172.21.15.62:41544) 2024-03-23T11:19:01.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.500+0000 7fa2ee600640 1 --2- 172.21.15.62:0/4021480997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2e80b2f30 0x7fa2e8070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:01.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.501+0000 7fa2edc00640 1 -- 172.21.15.62:0/4021480997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2e80b2f30 msgr2=0x7fa2e8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:01.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.501+0000 7fa2edc00640 1 --2- 172.21.15.62:0/4021480997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2e80b2f30 0x7fa2e8070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.501+0000 7fa2edc00640 1 -- 172.21.15.62:0/4021480997 >> v1:172.21.15.120:6789/0 conn(0x7fa2e8070090 legacy=0x7fa2e80b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:01.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.501+0000 7fa2edc00640 1 -- 172.21.15.62:0/4021480997 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80cd6c0 con 0x7fa2e8143dd0 2024-03-23T11:19:01.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.502+0000 7fa2edc00640 1 --2- 172.21.15.62:0/4021480997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2e8143dd0 0x7fa2e815abe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009870 tx=0x7fa2d802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbca2966bba50b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:01.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.503+0000 7fa2e7e00640 1 -- 172.21.15.62:0/4021480997 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8030840 con 0x7fa2e8143dd0 2024-03-23T11:19:01.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.503+0000 7fa2e7e00640 1 -- 172.21.15.62:0/4021480997 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d80309a0 con 0x7fa2e8143dd0 2024-03-23T11:19:01.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.503+0000 7fa2e7e00640 1 -- 172.21.15.62:0/4021480997 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8030d30 con 0x7fa2e8143dd0 2024-03-23T11:19:01.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.503+0000 7fa2ef000640 1 -- 172.21.15.62:0/4021480997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2e8143dd0 msgr2=0x7fa2e815abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:01.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.503+0000 7fa2ef000640 1 --2- 172.21.15.62:0/4021480997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2e8143dd0 0x7fa2e815abe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009870 tx=0x7fa2d802b980 comp rx=0 tx=0).stop 2024-03-23T11:19:01.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.504+0000 7fa2ef000640 1 -- 172.21.15.62:0/4021480997 shutdown_connections 2024-03-23T11:19:01.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.504+0000 7fa2ef000640 1 --2- 172.21.15.62:0/4021480997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2e80b2f30 0x7fa2e8070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.504+0000 7fa2ef000640 1 --2- 172.21.15.62:0/4021480997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2e8143dd0 0x7fa2e815abe0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.504+0000 7fa2ef000640 1 -- 172.21.15.62:0/4021480997 >> 172.21.15.62:0/4021480997 conn(0x7fa2e8066050 msgr2=0x7fa2e8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:01.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.504+0000 7fa2ef000640 1 -- 172.21.15.62:0/4021480997 shutdown_connections 2024-03-23T11:19:01.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.504+0000 7fa2ef000640 1 -- 172.21.15.62:0/4021480997 wait complete. 2024-03-23T11:19:01.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.505+0000 7fa2ef000640 1 Processor -- start 2024-03-23T11:19:01.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.506+0000 7fa2ef000640 1 -- start start 2024-03-23T11:19:01.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.506+0000 7fa2ef000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8070090 0x7fa2e81f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:01.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.506+0000 7fa2ef000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2e80b2f30 0x7fa2e813fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:01.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.506+0000 7fa2ef000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2e8143dd0 0x7fa2e80aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:01.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.506+0000 7fa2ef000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81f61b0 con 0x7fa2e8070090 2024-03-23T11:19:01.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2ef000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80c33a0 con 0x7fa2e8143dd0 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2ef000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80c3680 con 0x7fa2e80b2f30 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2edc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8070090 0x7fa2e81f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2edc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8070090 0x7fa2e81f5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37720/0 (socket says 172.21.15.62:37720) 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2edc00640 1 -- 172.21.15.62:0/370596599 learned_addr learned my addr 172.21.15.62:0/370596599 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2ed200640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2e80b2f30 0x7fa2e813fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2ee600640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2e8143dd0 0x7fa2e80aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2edc00640 1 -- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2e80b2f30 msgr2=0x7fa2e813fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2edc00640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2e80b2f30 0x7fa2e813fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2edc00640 1 -- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2e8143dd0 msgr2=0x7fa2e80aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2edc00640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2e8143dd0 0x7fa2e80aee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2edc00640 1 -- 172.21.15.62:0/370596599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d80094d0 con 0x7fa2e8070090 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2ed200640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2e80b2f30 0x7fa2e813fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.507+0000 7fa2ee600640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2e8143dd0 0x7fa2e80aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.508+0000 7fa2edc00640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8070090 0x7fa2e81f5c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8000c00 tx=0x7fa2d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:01.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.508+0000 7fa2e5600640 1 -- 172.21.15.62:0/370596599 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8039a90 con 0x7fa2e8070090 2024-03-23T11:19:01.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.508+0000 7fa2e5600640 1 -- 172.21.15.62:0/370596599 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d8039bf0 con 0x7fa2e8070090 2024-03-23T11:19:01.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.508+0000 7fa2e5600640 1 -- 172.21.15.62:0/370596599 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8043410 con 0x7fa2e8070090 2024-03-23T11:19:01.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.508+0000 7fa2ef000640 1 -- 172.21.15.62:0/370596599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e80c3960 con 0x7fa2e8070090 2024-03-23T11:19:01.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.508+0000 7fa2ef000640 1 -- 172.21.15.62:0/370596599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e80c3f20 con 0x7fa2e8070090 2024-03-23T11:19:01.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.509+0000 7fa2ef000640 1 -- 172.21.15.62:0/370596599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e8066bb0 con 0x7fa2e8070090 2024-03-23T11:19:01.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.510+0000 7fa2e5600640 1 -- 172.21.15.62:0/370596599 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa2d802f390 con 0x7fa2e8070090 2024-03-23T11:19:01.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.511+0000 7fa2e5600640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa2c806a7a0 0x7fa2c806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:01.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.511+0000 7fa2e5600640 1 -- 172.21.15.62:0/370596599 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2d80aafa0 con 0x7fa2e8070090 2024-03-23T11:19:01.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.512+0000 7fa2ed200640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa2c806a7a0 0x7fa2c806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:01.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.513+0000 7fa2ed200640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa2c806a7a0 0x7fa2c806cc60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4007fd0 tx=0x7fa2d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:01.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.516+0000 7fa2e5600640 1 -- 172.21.15.62:0/370596599 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2d8076440 con 0x7fa2e8070090 2024-03-23T11:19:01.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.711+0000 7fa2ef000640 1 -- 172.21.15.62:0/370596599 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2e80cdf60 con 0x7fa2c806a7a0 2024-03-23T11:19:01.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.732+0000 7fa2e5600640 1 -- 172.21.15.62:0/370596599 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264446 (secure 0 0 0) 0x7fa2d4005b90 con 0x7fa2c806a7a0 2024-03-23T11:19:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:01.737 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:19:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.737+0000 7fa2ef000640 1 -- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa2c806a7a0 msgr2=0x7fa2c806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.737+0000 7fa2ef000640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa2c806a7a0 0x7fa2c806cc60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4007fd0 tx=0x7fa2d4009040 comp rx=0 tx=0).stop 2024-03-23T11:19:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.737+0000 7fa2ef000640 1 -- 172.21.15.62:0/370596599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8070090 msgr2=0x7fa2e81f5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.737+0000 7fa2ef000640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8070090 0x7fa2e81f5c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8000c00 tx=0x7fa2d8004730 comp rx=0 tx=0).stop 2024-03-23T11:19:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.737+0000 7fa2ef000640 1 -- 172.21.15.62:0/370596599 shutdown_connections 2024-03-23T11:19:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.737+0000 7fa2ef000640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa2c806a7a0 0x7fa2c806cc60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.737+0000 7fa2ef000640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2e8143dd0 0x7fa2e80aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.737+0000 7fa2ef000640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2e80b2f30 0x7fa2e813fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.737+0000 7fa2ef000640 1 --2- 172.21.15.62:0/370596599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8070090 0x7fa2e81f5c70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.737+0000 7fa2ef000640 1 -- 172.21.15.62:0/370596599 >> 172.21.15.62:0/370596599 conn(0x7fa2e8066050 msgr2=0x7fa2e815c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.737+0000 7fa2ef000640 1 -- 172.21.15.62:0/370596599 shutdown_connections 2024-03-23T11:19:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:01.738+0000 7fa2ef000640 1 -- 172.21.15.62:0/370596599 wait complete. 2024-03-23T11:19:01.759 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-23T11:19:01.233459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":2972,"kb_used_omap":13,"kb_used_meta":216050,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3043328,"data_stored":1813773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":221235288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":80,"num_write_kb":798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002195"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257670+0000","last_change":"2024-03-23T11:18:56.274088+0000","last_active":"2024-03-23T11:18:57.257670+0000","last_peered":"2024-03-23T11:18:57.257670+0000","last_clean":"2024-03-23T11:18:57.257670+0000","last_became_active":"2024-03-23T11:18:56.273097+0000","last_became_peered":"2024-03-23T11:18:56.273097+0000","last_unstale":"2024-03-23T11:18:57.257670+0000","last_undegraded":"2024-03-23T11:18:57.257670+0000","last_fullsized":"2024-03-23T11:18:57.257670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:48:24.807457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253337+0000","last_change":"2024-03-23T11:18:56.260586+0000","last_active":"2024-03-23T11:18:57.253337+0000","last_peered":"2024-03-23T11:18:57.253337+0000","last_clean":"2024-03-23T11:18:57.253337+0000","last_became_active":"2024-03-23T11:18:56.260321+0000","last_became_peered":"2024-03-23T11:18:56.260321+0000","last_unstale":"2024-03-23T11:18:57.253337+0000","last_undegraded":"2024-03-23T11:18:57.253337+0000","last_fullsized":"2024-03-23T11:18:57.253337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:24:40.203305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099527+0000","last_change":"2024-03-23T11:18:56.264999+0000","last_active":"2024-03-23T11:18:58.099527+0000","last_peered":"2024-03-23T11:18:58.099527+0000","last_clean":"2024-03-23T11:18:58.099527+0000","last_became_active":"2024-03-23T11:18:56.264689+0000","last_became_peered":"2024-03-23T11:18:56.264689+0000","last_unstale":"2024-03-23T11:18:58.099527+0000","last_undegraded":"2024-03-23T11:18:58.099527+0000","last_fullsized":"2024-03-23T11:18:58.099527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.747240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315279+0000","last_change":"2024-03-23T11:18:56.283608+0000","last_active":"2024-03-23T11:18:57.315279+0000","last_peered":"2024-03-23T11:18:57.315279+0000","last_clean":"2024-03-23T11:18:57.315279+0000","last_became_active":"2024-03-23T11:18:56.277048+0000","last_became_peered":"2024-03-23T11:18:56.277048+0000","last_unstale":"2024-03-23T11:18:57.315279+0000","last_undegraded":"2024-03-23T11:18:57.315279+0000","last_fullsized":"2024-03-23T11:18:57.315279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:28.809711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256138+0000","last_change":"2024-03-23T11:18:56.264915+0000","last_active":"2024-03-23T11:18:57.256138+0000","last_peered":"2024-03-23T11:18:57.256138+0000","last_clean":"2024-03-23T11:18:57.256138+0000","last_became_active":"2024-03-23T11:18:56.264696+0000","last_became_peered":"2024-03-23T11:18:56.264696+0000","last_unstale":"2024-03-23T11:18:57.256138+0000","last_undegraded":"2024-03-23T11:18:57.256138+0000","last_fullsized":"2024-03-23T11:18:57.256138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:59.578102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314423+0000","last_change":"2024-03-23T11:18:56.282941+0000","last_active":"2024-03-23T11:18:57.314423+0000","last_peered":"2024-03-23T11:18:57.314423+0000","last_clean":"2024-03-23T11:18:57.314423+0000","last_became_active":"2024-03-23T11:18:56.282556+0000","last_became_peered":"2024-03-23T11:18:56.282556+0000","last_unstale":"2024-03-23T11:18:57.314423+0000","last_undegraded":"2024-03-23T11:18:57.314423+0000","last_fullsized":"2024-03-23T11:18:57.314423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:01:46.194360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255145+0000","last_change":"2024-03-23T11:18:56.274568+0000","last_active":"2024-03-23T11:18:57.255145+0000","last_peered":"2024-03-23T11:18:57.255145+0000","last_clean":"2024-03-23T11:18:57.255145+0000","last_became_active":"2024-03-23T11:18:56.273524+0000","last_became_peered":"2024-03-23T11:18:56.273524+0000","last_unstale":"2024-03-23T11:18:57.255145+0000","last_undegraded":"2024-03-23T11:18:57.255145+0000","last_fullsized":"2024-03-23T11:18:57.255145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:48:43.299392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099491+0000","last_change":"2024-03-23T11:18:56.272280+0000","last_active":"2024-03-23T11:18:58.099491+0000","last_peered":"2024-03-23T11:18:58.099491+0000","last_clean":"2024-03-23T11:18:58.099491+0000","last_became_active":"2024-03-23T11:18:56.272096+0000","last_became_peered":"2024-03-23T11:18:56.272096+0000","last_unstale":"2024-03-23T11:18:58.099491+0000","last_undegraded":"2024-03-23T11:18:58.099491+0000","last_fullsized":"2024-03-23T11:18:58.099491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:28.809711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257785+0000","last_change":"2024-03-23T11:18:56.282997+0000","last_active":"2024-03-23T11:18:57.257785+0000","last_peered":"2024-03-23T11:18:57.257785+0000","last_clean":"2024-03-23T11:18:57.257785+0000","last_became_active":"2024-03-23T11:18:56.282768+0000","last_became_peered":"2024-03-23T11:18:56.282768+0000","last_unstale":"2024-03-23T11:18:57.257785+0000","last_undegraded":"2024-03-23T11:18:57.257785+0000","last_fullsized":"2024-03-23T11:18:57.257785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:47:39.199128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461969+0000","last_change":"2024-03-23T11:18:56.269707+0000","last_active":"2024-03-23T11:18:57.461969+0000","last_peered":"2024-03-23T11:18:57.461969+0000","last_clean":"2024-03-23T11:18:57.461969+0000","last_became_active":"2024-03-23T11:18:56.266078+0000","last_became_peered":"2024-03-23T11:18:56.266078+0000","last_unstale":"2024-03-23T11:18:57.461969+0000","last_undegraded":"2024-03-23T11:18:57.461969+0000","last_fullsized":"2024-03-23T11:18:57.461969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461775+0000","last_change":"2024-03-23T11:18:56.266542+0000","last_active":"2024-03-23T11:18:57.461775+0000","last_peered":"2024-03-23T11:18:57.461775+0000","last_clean":"2024-03-23T11:18:57.461775+0000","last_became_active":"2024-03-23T11:18:56.266026+0000","last_became_peered":"2024-03-23T11:18:56.266026+0000","last_unstale":"2024-03-23T11:18:57.461775+0000","last_undegraded":"2024-03-23T11:18:57.461775+0000","last_fullsized":"2024-03-23T11:18:57.461775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255738+0000","last_change":"2024-03-23T11:18:56.272690+0000","last_active":"2024-03-23T11:18:57.255738+0000","last_peered":"2024-03-23T11:18:57.255738+0000","last_clean":"2024-03-23T11:18:57.255738+0000","last_became_active":"2024-03-23T11:18:56.272153+0000","last_became_peered":"2024-03-23T11:18:56.272153+0000","last_unstale":"2024-03-23T11:18:57.255738+0000","last_undegraded":"2024-03-23T11:18:57.255738+0000","last_fullsized":"2024-03-23T11:18:57.255738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:44:23.104322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101098+0000","last_change":"2024-03-23T11:18:56.264317+0000","last_active":"2024-03-23T11:18:58.101098+0000","last_peered":"2024-03-23T11:18:58.101098+0000","last_clean":"2024-03-23T11:18:58.101098+0000","last_became_active":"2024-03-23T11:18:56.263977+0000","last_became_peered":"2024-03-23T11:18:56.263977+0000","last_unstale":"2024-03-23T11:18:58.101098+0000","last_undegraded":"2024-03-23T11:18:58.101098+0000","last_fullsized":"2024-03-23T11:18:58.101098+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:33.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098064+0000","last_change":"2024-03-23T11:18:56.267303+0000","last_active":"2024-03-23T11:18:58.098064+0000","last_peered":"2024-03-23T11:18:58.098064+0000","last_clean":"2024-03-23T11:18:58.098064+0000","last_became_active":"2024-03-23T11:18:56.267130+0000","last_became_peered":"2024-03-23T11:18:56.267130+0000","last_unstale":"2024-03-23T11:18:58.098064+0000","last_undegraded":"2024-03-23T11:18:58.098064+0000","last_fullsized":"2024-03-23T11:18:58.098064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252603+0000","last_change":"2024-03-23T11:18:56.275581+0000","last_active":"2024-03-23T11:18:57.252603+0000","last_peered":"2024-03-23T11:18:57.252603+0000","last_clean":"2024-03-23T11:18:57.252603+0000","last_became_active":"2024-03-23T11:18:56.274566+0000","last_became_peered":"2024-03-23T11:18:56.274566+0000","last_unstale":"2024-03-23T11:18:57.252603+0000","last_undegraded":"2024-03-23T11:18:57.252603+0000","last_fullsized":"2024-03-23T11:18:57.252603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:31:43.650115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314345+0000","last_change":"2024-03-23T11:18:56.282803+0000","last_active":"2024-03-23T11:18:57.314345+0000","last_peered":"2024-03-23T11:18:57.314345+0000","last_clean":"2024-03-23T11:18:57.314345+0000","last_became_active":"2024-03-23T11:18:56.282500+0000","last_became_peered":"2024-03-23T11:18:56.282500+0000","last_unstale":"2024-03-23T11:18:57.314345+0000","last_undegraded":"2024-03-23T11:18:57.314345+0000","last_fullsized":"2024-03-23T11:18:57.314345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315513+0000","last_change":"2024-03-23T11:18:56.287431+0000","last_active":"2024-03-23T11:18:57.315513+0000","last_peered":"2024-03-23T11:18:57.315513+0000","last_clean":"2024-03-23T11:18:57.315513+0000","last_became_active":"2024-03-23T11:18:56.286853+0000","last_became_peered":"2024-03-23T11:18:56.286853+0000","last_unstale":"2024-03-23T11:18:57.315513+0000","last_undegraded":"2024-03-23T11:18:57.315513+0000","last_fullsized":"2024-03-23T11:18:57.315513+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:54:17.561394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100445+0000","last_change":"2024-03-23T11:18:56.260336+0000","last_active":"2024-03-23T11:18:58.100445+0000","last_peered":"2024-03-23T11:18:58.100445+0000","last_clean":"2024-03-23T11:18:58.100445+0000","last_became_active":"2024-03-23T11:18:56.259638+0000","last_became_peered":"2024-03-23T11:18:56.259638+0000","last_unstale":"2024-03-23T11:18:58.100445+0000","last_undegraded":"2024-03-23T11:18:58.100445+0000","last_fullsized":"2024-03-23T11:18:58.100445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:36:11.188974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257119+0000","last_change":"2024-03-23T11:18:56.268940+0000","last_active":"2024-03-23T11:18:57.257119+0000","last_peered":"2024-03-23T11:18:57.257119+0000","last_clean":"2024-03-23T11:18:57.257119+0000","last_became_active":"2024-03-23T11:18:56.268168+0000","last_became_peered":"2024-03-23T11:18:56.268168+0000","last_unstale":"2024-03-23T11:18:57.257119+0000","last_undegraded":"2024-03-23T11:18:57.257119+0000","last_fullsized":"2024-03-23T11:18:57.257119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:22.782018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461751+0000","last_change":"2024-03-23T11:18:56.271021+0000","last_active":"2024-03-23T11:18:57.461751+0000","last_peered":"2024-03-23T11:18:57.461751+0000","last_clean":"2024-03-23T11:18:57.461751+0000","last_became_active":"2024-03-23T11:18:56.270766+0000","last_became_peered":"2024-03-23T11:18:56.270766+0000","last_unstale":"2024-03-23T11:18:57.461751+0000","last_undegraded":"2024-03-23T11:18:57.461751+0000","last_fullsized":"2024-03-23T11:18:57.461751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098354+0000","last_change":"2024-03-23T11:18:56.261635+0000","last_active":"2024-03-23T11:18:58.098354+0000","last_peered":"2024-03-23T11:18:58.098354+0000","last_clean":"2024-03-23T11:18:58.098354+0000","last_became_active":"2024-03-23T11:18:56.260741+0000","last_became_peered":"2024-03-23T11:18:56.260741+0000","last_unstale":"2024-03-23T11:18:58.098354+0000","last_undegraded":"2024-03-23T11:18:58.098354+0000","last_fullsized":"2024-03-23T11:18:58.098354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314890+0000","last_change":"2024-03-23T11:18:56.286882+0000","last_active":"2024-03-23T11:18:57.314890+0000","last_peered":"2024-03-23T11:18:57.314890+0000","last_clean":"2024-03-23T11:18:57.314890+0000","last_became_active":"2024-03-23T11:18:56.285291+0000","last_became_peered":"2024-03-23T11:18:56.285291+0000","last_unstale":"2024-03-23T11:18:57.314890+0000","last_undegraded":"2024-03-23T11:18:57.314890+0000","last_fullsized":"2024-03-23T11:18:57.314890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:56.858523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101535+0000","last_change":"2024-03-23T11:18:56.276050+0000","last_active":"2024-03-23T11:18:58.101535+0000","last_peered":"2024-03-23T11:18:58.101535+0000","last_clean":"2024-03-23T11:18:58.101535+0000","last_became_active":"2024-03-23T11:18:56.275264+0000","last_became_peered":"2024-03-23T11:18:56.275264+0000","last_unstale":"2024-03-23T11:18:58.101535+0000","last_undegraded":"2024-03-23T11:18:58.101535+0000","last_fullsized":"2024-03-23T11:18:58.101535+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314573+0000","last_change":"2024-03-23T11:18:56.283292+0000","last_active":"2024-03-23T11:18:57.314573+0000","last_peered":"2024-03-23T11:18:57.314573+0000","last_clean":"2024-03-23T11:18:57.314573+0000","last_became_active":"2024-03-23T11:18:56.282493+0000","last_became_peered":"2024-03-23T11:18:56.282493+0000","last_unstale":"2024-03-23T11:18:57.314573+0000","last_undegraded":"2024-03-23T11:18:57.314573+0000","last_fullsized":"2024-03-23T11:18:57.314573+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101895+0000","last_change":"2024-03-23T11:18:56.282679+0000","last_active":"2024-03-23T11:18:58.101895+0000","last_peered":"2024-03-23T11:18:58.101895+0000","last_clean":"2024-03-23T11:18:58.101895+0000","last_became_active":"2024-03-23T11:18:56.282434+0000","last_became_peered":"2024-03-23T11:18:56.282434+0000","last_unstale":"2024-03-23T11:18:58.101895+0000","last_undegraded":"2024-03-23T11:18:58.101895+0000","last_fullsized":"2024-03-23T11:18:58.101895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:00.423263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462105+0000","last_change":"2024-03-23T11:18:56.260538+0000","last_active":"2024-03-23T11:18:57.462105+0000","last_peered":"2024-03-23T11:18:57.462105+0000","last_clean":"2024-03-23T11:18:57.462105+0000","last_became_active":"2024-03-23T11:18:56.259279+0000","last_became_peered":"2024-03-23T11:18:56.259279+0000","last_unstale":"2024-03-23T11:18:57.462105+0000","last_undegraded":"2024-03-23T11:18:57.462105+0000","last_fullsized":"2024-03-23T11:18:57.462105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253224+0000","last_change":"2024-03-23T11:18:56.263336+0000","last_active":"2024-03-23T11:18:57.253224+0000","last_peered":"2024-03-23T11:18:57.253224+0000","last_clean":"2024-03-23T11:18:57.253224+0000","last_became_active":"2024-03-23T11:18:56.262573+0000","last_became_peered":"2024-03-23T11:18:56.262573+0000","last_unstale":"2024-03-23T11:18:57.253224+0000","last_undegraded":"2024-03-23T11:18:57.253224+0000","last_fullsized":"2024-03-23T11:18:57.253224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:41:50.064223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101071+0000","last_change":"2024-03-23T11:18:56.284862+0000","last_active":"2024-03-23T11:18:58.101071+0000","last_peered":"2024-03-23T11:18:58.101071+0000","last_clean":"2024-03-23T11:18:58.101071+0000","last_became_active":"2024-03-23T11:18:56.284599+0000","last_became_peered":"2024-03-23T11:18:56.284599+0000","last_unstale":"2024-03-23T11:18:58.101071+0000","last_undegraded":"2024-03-23T11:18:58.101071+0000","last_fullsized":"2024-03-23T11:18:58.101071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.705001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098596+0000","last_change":"2024-03-23T11:18:56.271496+0000","last_active":"2024-03-23T11:18:58.098596+0000","last_peered":"2024-03-23T11:18:58.098596+0000","last_clean":"2024-03-23T11:18:58.098596+0000","last_became_active":"2024-03-23T11:18:56.271243+0000","last_became_peered":"2024-03-23T11:18:56.271243+0000","last_unstale":"2024-03-23T11:18:58.098596+0000","last_undegraded":"2024-03-23T11:18:58.098596+0000","last_fullsized":"2024-03-23T11:18:58.098596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:16:46.738802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314711+0000","last_change":"2024-03-23T11:18:56.282267+0000","last_active":"2024-03-23T11:18:57.314711+0000","last_peered":"2024-03-23T11:18:57.314711+0000","last_clean":"2024-03-23T11:18:57.314711+0000","last_became_active":"2024-03-23T11:18:56.277544+0000","last_became_peered":"2024-03-23T11:18:56.277544+0000","last_unstale":"2024-03-23T11:18:57.314711+0000","last_undegraded":"2024-03-23T11:18:57.314711+0000","last_fullsized":"2024-03-23T11:18:57.314711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:12.916824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098163+0000","last_change":"2024-03-23T11:18:56.267136+0000","last_active":"2024-03-23T11:18:58.098163+0000","last_peered":"2024-03-23T11:18:58.098163+0000","last_clean":"2024-03-23T11:18:58.098163+0000","last_became_active":"2024-03-23T11:18:56.266640+0000","last_became_peered":"2024-03-23T11:18:56.266640+0000","last_unstale":"2024-03-23T11:18:58.098163+0000","last_undegraded":"2024-03-23T11:18:58.098163+0000","last_fullsized":"2024-03-23T11:18:58.098163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461393+0000","last_change":"2024-03-23T11:18:56.266829+0000","last_active":"2024-03-23T11:18:57.461393+0000","last_peered":"2024-03-23T11:18:57.461393+0000","last_clean":"2024-03-23T11:18:57.461393+0000","last_became_active":"2024-03-23T11:18:56.266536+0000","last_became_peered":"2024-03-23T11:18:56.266536+0000","last_unstale":"2024-03-23T11:18:57.461393+0000","last_undegraded":"2024-03-23T11:18:57.461393+0000","last_fullsized":"2024-03-23T11:18:57.461393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:00.423263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255237+0000","last_change":"2024-03-23T11:18:56.267024+0000","last_active":"2024-03-23T11:18:57.255237+0000","last_peered":"2024-03-23T11:18:57.255237+0000","last_clean":"2024-03-23T11:18:57.255237+0000","last_became_active":"2024-03-23T11:18:56.266730+0000","last_became_peered":"2024-03-23T11:18:56.266730+0000","last_unstale":"2024-03-23T11:18:57.255237+0000","last_undegraded":"2024-03-23T11:18:57.255237+0000","last_fullsized":"2024-03-23T11:18:57.255237+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:19.533798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101417+0000","last_change":"2024-03-23T11:18:55.778856+0000","last_active":"2024-03-23T11:18:58.101417+0000","last_peered":"2024-03-23T11:18:58.101417+0000","last_clean":"2024-03-23T11:18:58.101417+0000","last_became_active":"2024-03-23T11:18:55.778487+0000","last_became_peered":"2024-03-23T11:18:55.778487+0000","last_unstale":"2024-03-23T11:18:58.101417+0000","last_undegraded":"2024-03-23T11:18:58.101417+0000","last_fullsized":"2024-03-23T11:18:58.101417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098890+0000","last_change":"2024-03-23T11:18:56.280876+0000","last_active":"2024-03-23T11:18:58.098890+0000","last_peered":"2024-03-23T11:18:58.098890+0000","last_clean":"2024-03-23T11:18:58.098890+0000","last_became_active":"2024-03-23T11:18:56.280574+0000","last_became_peered":"2024-03-23T11:18:56.280574+0000","last_unstale":"2024-03-23T11:18:58.098890+0000","last_undegraded":"2024-03-23T11:18:58.098890+0000","last_fullsized":"2024-03-23T11:18:58.098890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:25.559049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462037+0000","last_change":"2024-03-23T11:18:55.771275+0000","last_active":"2024-03-23T11:18:57.462037+0000","last_peered":"2024-03-23T11:18:57.462037+0000","last_clean":"2024-03-23T11:18:57.462037+0000","last_became_active":"2024-03-23T11:18:55.770948+0000","last_became_peered":"2024-03-23T11:18:55.770948+0000","last_unstale":"2024-03-23T11:18:57.462037+0000","last_undegraded":"2024-03-23T11:18:57.462037+0000","last_fullsized":"2024-03-23T11:18:57.462037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101598+0000","last_change":"2024-03-23T11:18:55.780718+0000","last_active":"2024-03-23T11:18:58.101598+0000","last_peered":"2024-03-23T11:18:58.101598+0000","last_clean":"2024-03-23T11:18:58.101598+0000","last_became_active":"2024-03-23T11:18:55.779357+0000","last_became_peered":"2024-03-23T11:18:55.779357+0000","last_unstale":"2024-03-23T11:18:58.101598+0000","last_undegraded":"2024-03-23T11:18:58.101598+0000","last_fullsized":"2024-03-23T11:18:58.101598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:36:21.274877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099752+0000","last_change":"2024-03-23T11:18:56.266172+0000","last_active":"2024-03-23T11:18:58.099752+0000","last_peered":"2024-03-23T11:18:58.099752+0000","last_clean":"2024-03-23T11:18:58.099752+0000","last_became_active":"2024-03-23T11:18:56.265882+0000","last_became_peered":"2024-03-23T11:18:56.265882+0000","last_unstale":"2024-03-23T11:18:58.099752+0000","last_undegraded":"2024-03-23T11:18:58.099752+0000","last_fullsized":"2024-03-23T11:18:58.099752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314638+0000","last_change":"2024-03-23T11:18:55.774144+0000","last_active":"2024-03-23T11:18:57.314638+0000","last_peered":"2024-03-23T11:18:57.314638+0000","last_clean":"2024-03-23T11:18:57.314638+0000","last_became_active":"2024-03-23T11:18:55.773880+0000","last_became_peered":"2024-03-23T11:18:55.773880+0000","last_unstale":"2024-03-23T11:18:57.314638+0000","last_undegraded":"2024-03-23T11:18:57.314638+0000","last_fullsized":"2024-03-23T11:18:57.314638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252538+0000","last_change":"2024-03-23T11:18:56.260698+0000","last_active":"2024-03-23T11:18:57.252538+0000","last_peered":"2024-03-23T11:18:57.252538+0000","last_clean":"2024-03-23T11:18:57.252538+0000","last_became_active":"2024-03-23T11:18:56.258466+0000","last_became_peered":"2024-03-23T11:18:56.258466+0000","last_unstale":"2024-03-23T11:18:57.252538+0000","last_undegraded":"2024-03-23T11:18:57.252538+0000","last_fullsized":"2024-03-23T11:18:57.252538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:55:22.272329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462086+0000","last_change":"2024-03-23T11:18:55.783554+0000","last_active":"2024-03-23T11:18:57.462086+0000","last_peered":"2024-03-23T11:18:57.462086+0000","last_clean":"2024-03-23T11:18:57.462086+0000","last_became_active":"2024-03-23T11:18:55.783248+0000","last_became_peered":"2024-03-23T11:18:55.783248+0000","last_unstale":"2024-03-23T11:18:57.462086+0000","last_undegraded":"2024-03-23T11:18:57.462086+0000","last_fullsized":"2024-03-23T11:18:57.462086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:18.352259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314908+0000","last_change":"2024-03-23T11:18:56.287651+0000","last_active":"2024-03-23T11:18:57.314908+0000","last_peered":"2024-03-23T11:18:57.314908+0000","last_clean":"2024-03-23T11:18:57.314908+0000","last_became_active":"2024-03-23T11:18:56.287312+0000","last_became_peered":"2024-03-23T11:18:56.287312+0000","last_unstale":"2024-03-23T11:18:57.314908+0000","last_undegraded":"2024-03-23T11:18:57.314908+0000","last_fullsized":"2024-03-23T11:18:57.314908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:24:29.439358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314950+0000","last_change":"2024-03-23T11:18:55.783117+0000","last_active":"2024-03-23T11:18:57.314950+0000","last_peered":"2024-03-23T11:18:57.314950+0000","last_clean":"2024-03-23T11:18:57.314950+0000","last_became_active":"2024-03-23T11:18:55.782341+0000","last_became_peered":"2024-03-23T11:18:55.782341+0000","last_unstale":"2024-03-23T11:18:57.314950+0000","last_undegraded":"2024-03-23T11:18:57.314950+0000","last_fullsized":"2024-03-23T11:18:57.314950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:10:08.943114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099470+0000","last_change":"2024-03-23T11:18:56.290381+0000","last_active":"2024-03-23T11:18:58.099470+0000","last_peered":"2024-03-23T11:18:58.099470+0000","last_clean":"2024-03-23T11:18:58.099470+0000","last_became_active":"2024-03-23T11:18:56.290070+0000","last_became_peered":"2024-03-23T11:18:56.290070+0000","last_unstale":"2024-03-23T11:18:58.099470+0000","last_undegraded":"2024-03-23T11:18:58.099470+0000","last_fullsized":"2024-03-23T11:18:58.099470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253428+0000","last_change":"2024-03-23T11:18:55.783461+0000","last_active":"2024-03-23T11:18:57.253428+0000","last_peered":"2024-03-23T11:18:57.253428+0000","last_clean":"2024-03-23T11:18:57.253428+0000","last_became_active":"2024-03-23T11:18:55.783214+0000","last_became_peered":"2024-03-23T11:18:55.783214+0000","last_unstale":"2024-03-23T11:18:57.253428+0000","last_undegraded":"2024-03-23T11:18:57.253428+0000","last_fullsized":"2024-03-23T11:18:57.253428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:28:51.146588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255996+0000","last_change":"2024-03-23T11:18:56.268161+0000","last_active":"2024-03-23T11:18:57.255996+0000","last_peered":"2024-03-23T11:18:57.255996+0000","last_clean":"2024-03-23T11:18:57.255996+0000","last_became_active":"2024-03-23T11:18:56.263992+0000","last_became_peered":"2024-03-23T11:18:56.263992+0000","last_unstale":"2024-03-23T11:18:57.255996+0000","last_undegraded":"2024-03-23T11:18:57.255996+0000","last_fullsized":"2024-03-23T11:18:57.255996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461733+0000","last_change":"2024-03-23T11:18:55.782334+0000","last_active":"2024-03-23T11:18:57.461733+0000","last_peered":"2024-03-23T11:18:57.461733+0000","last_clean":"2024-03-23T11:18:57.461733+0000","last_became_active":"2024-03-23T11:18:55.780347+0000","last_became_peered":"2024-03-23T11:18:55.780347+0000","last_unstale":"2024-03-23T11:18:57.461733+0000","last_undegraded":"2024-03-23T11:18:57.461733+0000","last_fullsized":"2024-03-23T11:18:57.461733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:10:08.943114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256551+0000","last_change":"2024-03-23T11:18:56.272684+0000","last_active":"2024-03-23T11:18:57.256551+0000","last_peered":"2024-03-23T11:18:57.256551+0000","last_clean":"2024-03-23T11:18:57.256551+0000","last_became_active":"2024-03-23T11:18:56.268603+0000","last_became_peered":"2024-03-23T11:18:56.268603+0000","last_unstale":"2024-03-23T11:18:57.256551+0000","last_undegraded":"2024-03-23T11:18:57.256551+0000","last_fullsized":"2024-03-23T11:18:57.256551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:22.095583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100170+0000","last_change":"2024-03-23T11:18:55.783962+0000","last_active":"2024-03-23T11:18:58.100170+0000","last_peered":"2024-03-23T11:18:58.100170+0000","last_clean":"2024-03-23T11:18:58.100170+0000","last_became_active":"2024-03-23T11:18:55.783729+0000","last_became_peered":"2024-03-23T11:18:55.783729+0000","last_unstale":"2024-03-23T11:18:58.100170+0000","last_undegraded":"2024-03-23T11:18:58.100170+0000","last_fullsized":"2024-03-23T11:18:58.100170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:42.674448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098331+0000","last_change":"2024-03-23T11:18:56.261635+0000","last_active":"2024-03-23T11:18:58.098331+0000","last_peered":"2024-03-23T11:18:58.098331+0000","last_clean":"2024-03-23T11:18:58.098331+0000","last_became_active":"2024-03-23T11:18:56.261334+0000","last_became_peered":"2024-03-23T11:18:56.261334+0000","last_unstale":"2024-03-23T11:18:58.098331+0000","last_undegraded":"2024-03-23T11:18:58.098331+0000","last_fullsized":"2024-03-23T11:18:58.098331+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314958+0000","last_change":"2024-03-23T11:18:55.782301+0000","last_active":"2024-03-23T11:18:57.314958+0000","last_peered":"2024-03-23T11:18:57.314958+0000","last_clean":"2024-03-23T11:18:57.314958+0000","last_became_active":"2024-03-23T11:18:55.782005+0000","last_became_peered":"2024-03-23T11:18:55.782005+0000","last_unstale":"2024-03-23T11:18:57.314958+0000","last_undegraded":"2024-03-23T11:18:57.314958+0000","last_fullsized":"2024-03-23T11:18:57.314958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256246+0000","last_change":"2024-03-23T11:18:56.275199+0000","last_active":"2024-03-23T11:18:57.256246+0000","last_peered":"2024-03-23T11:18:57.256246+0000","last_clean":"2024-03-23T11:18:57.256246+0000","last_became_active":"2024-03-23T11:18:56.274944+0000","last_became_peered":"2024-03-23T11:18:56.274944+0000","last_unstale":"2024-03-23T11:18:57.256246+0000","last_undegraded":"2024-03-23T11:18:57.256246+0000","last_fullsized":"2024-03-23T11:18:57.256246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:00:18.525010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461578+0000","last_change":"2024-03-23T11:18:55.775042+0000","last_active":"2024-03-23T11:18:57.461578+0000","last_peered":"2024-03-23T11:18:57.461578+0000","last_clean":"2024-03-23T11:18:57.461578+0000","last_became_active":"2024-03-23T11:18:55.774721+0000","last_became_peered":"2024-03-23T11:18:55.774721+0000","last_unstale":"2024-03-23T11:18:57.461578+0000","last_undegraded":"2024-03-23T11:18:57.461578+0000","last_fullsized":"2024-03-23T11:18:57.461578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461383+0000","last_change":"2024-03-23T11:18:56.263485+0000","last_active":"2024-03-23T11:18:57.461383+0000","last_peered":"2024-03-23T11:18:57.461383+0000","last_clean":"2024-03-23T11:18:57.461383+0000","last_became_active":"2024-03-23T11:18:56.261677+0000","last_became_peered":"2024-03-23T11:18:56.261677+0000","last_unstale":"2024-03-23T11:18:57.461383+0000","last_undegraded":"2024-03-23T11:18:57.461383+0000","last_fullsized":"2024-03-23T11:18:57.461383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098165+0000","last_change":"2024-03-23T11:18:55.770098+0000","last_active":"2024-03-23T11:18:58.098165+0000","last_peered":"2024-03-23T11:18:58.098165+0000","last_clean":"2024-03-23T11:18:58.098165+0000","last_became_active":"2024-03-23T11:18:55.769814+0000","last_became_peered":"2024-03-23T11:18:55.769814+0000","last_unstale":"2024-03-23T11:18:58.098165+0000","last_undegraded":"2024-03-23T11:18:58.098165+0000","last_fullsized":"2024-03-23T11:18:58.098165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255419+0000","last_change":"2024-03-23T11:18:56.274715+0000","last_active":"2024-03-23T11:18:57.255419+0000","last_peered":"2024-03-23T11:18:57.255419+0000","last_clean":"2024-03-23T11:18:57.255419+0000","last_became_active":"2024-03-23T11:18:56.274437+0000","last_became_peered":"2024-03-23T11:18:56.274437+0000","last_unstale":"2024-03-23T11:18:57.255419+0000","last_undegraded":"2024-03-23T11:18:57.255419+0000","last_fullsized":"2024-03-23T11:18:57.255419+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:28:00.964447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256032+0000","last_change":"2024-03-23T11:18:55.789682+0000","last_active":"2024-03-23T11:18:57.256032+0000","last_peered":"2024-03-23T11:18:57.256032+0000","last_clean":"2024-03-23T11:18:57.256032+0000","last_became_active":"2024-03-23T11:18:55.789278+0000","last_became_peered":"2024-03-23T11:18:55.789278+0000","last_unstale":"2024-03-23T11:18:57.256032+0000","last_undegraded":"2024-03-23T11:18:57.256032+0000","last_fullsized":"2024-03-23T11:18:57.256032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:42.674448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256896+0000","last_change":"2024-03-23T11:18:56.267163+0000","last_active":"2024-03-23T11:18:57.256896+0000","last_peered":"2024-03-23T11:18:57.256896+0000","last_clean":"2024-03-23T11:18:57.256896+0000","last_became_active":"2024-03-23T11:18:56.266907+0000","last_became_peered":"2024-03-23T11:18:56.266907+0000","last_unstale":"2024-03-23T11:18:57.256896+0000","last_undegraded":"2024-03-23T11:18:57.256896+0000","last_fullsized":"2024-03-23T11:18:57.256896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:51:52.682495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100710+0000","last_change":"2024-03-23T11:18:55.779353+0000","last_active":"2024-03-23T11:18:58.100710+0000","last_peered":"2024-03-23T11:18:58.100710+0000","last_clean":"2024-03-23T11:18:58.100710+0000","last_became_active":"2024-03-23T11:18:55.779045+0000","last_became_peered":"2024-03-23T11:18:55.779045+0000","last_unstale":"2024-03-23T11:18:58.100710+0000","last_undegraded":"2024-03-23T11:18:58.100710+0000","last_fullsized":"2024-03-23T11:18:58.100710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257949+0000","last_change":"2024-03-23T11:18:56.275906+0000","last_active":"2024-03-23T11:18:57.257949+0000","last_peered":"2024-03-23T11:18:57.257949+0000","last_clean":"2024-03-23T11:18:57.257949+0000","last_became_active":"2024-03-23T11:18:56.274500+0000","last_became_peered":"2024-03-23T11:18:56.274500+0000","last_unstale":"2024-03-23T11:18:57.257949+0000","last_undegraded":"2024-03-23T11:18:57.257949+0000","last_fullsized":"2024-03-23T11:18:57.257949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:13:34.436113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253366+0000","last_change":"2024-03-23T11:18:55.763585+0000","last_active":"2024-03-23T11:18:57.253366+0000","last_peered":"2024-03-23T11:18:57.253366+0000","last_clean":"2024-03-23T11:18:57.253366+0000","last_became_active":"2024-03-23T11:18:55.762854+0000","last_became_peered":"2024-03-23T11:18:55.762854+0000","last_unstale":"2024-03-23T11:18:57.253366+0000","last_undegraded":"2024-03-23T11:18:57.253366+0000","last_fullsized":"2024-03-23T11:18:57.253366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:38.995255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315476+0000","last_change":"2024-03-23T11:18:56.276682+0000","last_active":"2024-03-23T11:18:57.315476+0000","last_peered":"2024-03-23T11:18:57.315476+0000","last_clean":"2024-03-23T11:18:57.315476+0000","last_became_active":"2024-03-23T11:18:56.276094+0000","last_became_peered":"2024-03-23T11:18:56.276094+0000","last_unstale":"2024-03-23T11:18:57.315476+0000","last_undegraded":"2024-03-23T11:18:57.315476+0000","last_fullsized":"2024-03-23T11:18:57.315476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:26.126571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098569+0000","last_change":"2024-03-23T11:18:55.783331+0000","last_active":"2024-03-23T11:18:58.098569+0000","last_peered":"2024-03-23T11:18:58.098569+0000","last_clean":"2024-03-23T11:18:58.098569+0000","last_became_active":"2024-03-23T11:18:55.783030+0000","last_became_peered":"2024-03-23T11:18:55.783030+0000","last_unstale":"2024-03-23T11:18:58.098569+0000","last_undegraded":"2024-03-23T11:18:58.098569+0000","last_fullsized":"2024-03-23T11:18:58.098569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256897+0000","last_change":"2024-03-23T11:18:56.284587+0000","last_active":"2024-03-23T11:18:57.256897+0000","last_peered":"2024-03-23T11:18:57.256897+0000","last_clean":"2024-03-23T11:18:57.256897+0000","last_became_active":"2024-03-23T11:18:56.283582+0000","last_became_peered":"2024-03-23T11:18:56.283582+0000","last_unstale":"2024-03-23T11:18:57.256897+0000","last_undegraded":"2024-03-23T11:18:57.256897+0000","last_fullsized":"2024-03-23T11:18:57.256897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:01:11.724895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":164,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T11:18:56.246747+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:18:56.246747+0000","last_peered":"2024-03-23T11:18:56.246747+0000","last_clean":"2024-03-23T11:18:56.246747+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:18:56.246747+0000","last_undegraded":"2024-03-23T11:18:56.246747+0000","last_fullsized":"2024-03-23T11:18:56.246747+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314507+0000","last_change":"2024-03-23T11:18:55.778697+0000","last_active":"2024-03-23T11:18:57.314507+0000","last_peered":"2024-03-23T11:18:57.314507+0000","last_clean":"2024-03-23T11:18:57.314507+0000","last_became_active":"2024-03-23T11:18:55.778387+0000","last_became_peered":"2024-03-23T11:18:55.778387+0000","last_unstale":"2024-03-23T11:18:57.314507+0000","last_undegraded":"2024-03-23T11:18:57.314507+0000","last_fullsized":"2024-03-23T11:18:57.314507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100582+0000","last_change":"2024-03-23T11:18:56.268261+0000","last_active":"2024-03-23T11:18:58.100582+0000","last_peered":"2024-03-23T11:18:58.100582+0000","last_clean":"2024-03-23T11:18:58.100582+0000","last_became_active":"2024-03-23T11:18:56.267974+0000","last_became_peered":"2024-03-23T11:18:56.267974+0000","last_unstale":"2024-03-23T11:18:58.100582+0000","last_undegraded":"2024-03-23T11:18:58.100582+0000","last_fullsized":"2024-03-23T11:18:58.100582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:44.879627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255801+0000","last_change":"2024-03-23T11:18:55.775332+0000","last_active":"2024-03-23T11:18:57.255801+0000","last_peered":"2024-03-23T11:18:57.255801+0000","last_clean":"2024-03-23T11:18:57.255801+0000","last_became_active":"2024-03-23T11:18:55.775009+0000","last_became_peered":"2024-03-23T11:18:55.775009+0000","last_unstale":"2024-03-23T11:18:57.255801+0000","last_undegraded":"2024-03-23T11:18:57.255801+0000","last_fullsized":"2024-03-23T11:18:57.255801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:08:12.342783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098600+0000","last_change":"2024-03-23T11:18:56.279541+0000","last_active":"2024-03-23T11:18:58.098600+0000","last_peered":"2024-03-23T11:18:58.098600+0000","last_clean":"2024-03-23T11:18:58.098600+0000","last_became_active":"2024-03-23T11:18:56.279231+0000","last_became_peered":"2024-03-23T11:18:56.279231+0000","last_unstale":"2024-03-23T11:18:58.098600+0000","last_undegraded":"2024-03-23T11:18:58.098600+0000","last_fullsized":"2024-03-23T11:18:58.098600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098389+0000","last_change":"2024-03-23T11:18:55.777288+0000","last_active":"2024-03-23T11:18:58.098389+0000","last_peered":"2024-03-23T11:18:58.098389+0000","last_clean":"2024-03-23T11:18:58.098389+0000","last_became_active":"2024-03-23T11:18:55.776998+0000","last_became_peered":"2024-03-23T11:18:55.776998+0000","last_unstale":"2024-03-23T11:18:58.098389+0000","last_undegraded":"2024-03-23T11:18:58.098389+0000","last_fullsized":"2024-03-23T11:18:58.098389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252646+0000","last_change":"2024-03-23T11:18:56.257547+0000","last_active":"2024-03-23T11:18:57.252646+0000","last_peered":"2024-03-23T11:18:57.252646+0000","last_clean":"2024-03-23T11:18:57.252646+0000","last_became_active":"2024-03-23T11:18:56.257307+0000","last_became_peered":"2024-03-23T11:18:56.257307+0000","last_unstale":"2024-03-23T11:18:57.252646+0000","last_undegraded":"2024-03-23T11:18:57.252646+0000","last_fullsized":"2024-03-23T11:18:57.252646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:32.498133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315167+0000","last_change":"2024-03-23T11:18:55.780573+0000","last_active":"2024-03-23T11:18:57.315167+0000","last_peered":"2024-03-23T11:18:57.315167+0000","last_clean":"2024-03-23T11:18:57.315167+0000","last_became_active":"2024-03-23T11:18:55.779963+0000","last_became_peered":"2024-03-23T11:18:55.779963+0000","last_unstale":"2024-03-23T11:18:57.315167+0000","last_undegraded":"2024-03-23T11:18:57.315167+0000","last_fullsized":"2024-03-23T11:18:57.315167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:18.352259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255291+0000","last_change":"2024-03-23T11:18:56.264913+0000","last_active":"2024-03-23T11:18:57.255291+0000","last_peered":"2024-03-23T11:18:57.255291+0000","last_clean":"2024-03-23T11:18:57.255291+0000","last_became_active":"2024-03-23T11:18:56.264679+0000","last_became_peered":"2024-03-23T11:18:56.264679+0000","last_unstale":"2024-03-23T11:18:57.255291+0000","last_undegraded":"2024-03-23T11:18:57.255291+0000","last_fullsized":"2024-03-23T11:18:57.255291+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:16:46.738802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253124+0000","last_change":"2024-03-23T11:18:55.785635+0000","last_active":"2024-03-23T11:18:57.253124+0000","last_peered":"2024-03-23T11:18:57.253124+0000","last_clean":"2024-03-23T11:18:57.253124+0000","last_became_active":"2024-03-23T11:18:55.785064+0000","last_became_peered":"2024-03-23T11:18:55.785064+0000","last_unstale":"2024-03-23T11:18:57.253124+0000","last_undegraded":"2024-03-23T11:18:57.253124+0000","last_fullsized":"2024-03-23T11:18:57.253124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:04:27.564911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315028+0000","last_change":"2024-03-23T11:18:56.276583+0000","last_active":"2024-03-23T11:18:57.315028+0000","last_peered":"2024-03-23T11:18:57.315028+0000","last_clean":"2024-03-23T11:18:57.315028+0000","last_became_active":"2024-03-23T11:18:56.276294+0000","last_became_peered":"2024-03-23T11:18:56.276294+0000","last_unstale":"2024-03-23T11:18:57.315028+0000","last_undegraded":"2024-03-23T11:18:57.315028+0000","last_fullsized":"2024-03-23T11:18:57.315028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:36.411878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257226+0000","last_change":"2024-03-23T11:18:55.784141+0000","last_active":"2024-03-23T11:18:57.257226+0000","last_peered":"2024-03-23T11:18:57.257226+0000","last_clean":"2024-03-23T11:18:57.257226+0000","last_became_active":"2024-03-23T11:18:55.783843+0000","last_became_peered":"2024-03-23T11:18:55.783843+0000","last_unstale":"2024-03-23T11:18:57.257226+0000","last_undegraded":"2024-03-23T11:18:57.257226+0000","last_fullsized":"2024-03-23T11:18:57.257226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:57:29.038916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252794+0000","last_change":"2024-03-23T11:18:56.258329+0000","last_active":"2024-03-23T11:18:57.252794+0000","last_peered":"2024-03-23T11:18:57.252794+0000","last_clean":"2024-03-23T11:18:57.252794+0000","last_became_active":"2024-03-23T11:18:56.258033+0000","last_became_peered":"2024-03-23T11:18:56.258033+0000","last_unstale":"2024-03-23T11:18:57.252794+0000","last_undegraded":"2024-03-23T11:18:57.252794+0000","last_fullsized":"2024-03-23T11:18:57.252794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:24:37.470160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253467+0000","last_change":"2024-03-23T11:18:55.778103+0000","last_active":"2024-03-23T11:18:57.253467+0000","last_peered":"2024-03-23T11:18:57.253467+0000","last_clean":"2024-03-23T11:18:57.253467+0000","last_became_active":"2024-03-23T11:18:55.777604+0000","last_became_peered":"2024-03-23T11:18:55.777604+0000","last_unstale":"2024-03-23T11:18:57.253467+0000","last_undegraded":"2024-03-23T11:18:57.253467+0000","last_fullsized":"2024-03-23T11:18:57.253467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:42:44.181703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101839+0000","last_change":"2024-03-23T11:18:56.280720+0000","last_active":"2024-03-23T11:18:58.101839+0000","last_peered":"2024-03-23T11:18:58.101839+0000","last_clean":"2024-03-23T11:18:58.101839+0000","last_became_active":"2024-03-23T11:18:56.278952+0000","last_became_peered":"2024-03-23T11:18:56.278952+0000","last_unstale":"2024-03-23T11:18:58.101839+0000","last_undegraded":"2024-03-23T11:18:58.101839+0000","last_fullsized":"2024-03-23T11:18:58.101839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462270+0000","last_change":"2024-03-23T11:18:55.776925+0000","last_active":"2024-03-23T11:18:57.462270+0000","last_peered":"2024-03-23T11:18:57.462270+0000","last_clean":"2024-03-23T11:18:57.462270+0000","last_became_active":"2024-03-23T11:18:55.776651+0000","last_became_peered":"2024-03-23T11:18:55.776651+0000","last_unstale":"2024-03-23T11:18:57.462270+0000","last_undegraded":"2024-03-23T11:18:57.462270+0000","last_fullsized":"2024-03-23T11:18:57.462270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099552+0000","last_change":"2024-03-23T11:18:56.268554+0000","last_active":"2024-03-23T11:18:58.099552+0000","last_peered":"2024-03-23T11:18:58.099552+0000","last_clean":"2024-03-23T11:18:58.099552+0000","last_became_active":"2024-03-23T11:18:56.267201+0000","last_became_peered":"2024-03-23T11:18:56.267201+0000","last_unstale":"2024-03-23T11:18:58.099552+0000","last_undegraded":"2024-03-23T11:18:58.099552+0000","last_fullsized":"2024-03-23T11:18:58.099552+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.705001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255562+0000","last_change":"2024-03-23T11:18:55.789376+0000","last_active":"2024-03-23T11:18:57.255562+0000","last_peered":"2024-03-23T11:18:57.255562+0000","last_clean":"2024-03-23T11:18:57.255562+0000","last_became_active":"2024-03-23T11:18:55.789122+0000","last_became_peered":"2024-03-23T11:18:55.789122+0000","last_unstale":"2024-03-23T11:18:57.255562+0000","last_undegraded":"2024-03-23T11:18:57.255562+0000","last_fullsized":"2024-03-23T11:18:57.255562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:26.469433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257058+0000","last_change":"2024-03-23T11:18:56.267674+0000","last_active":"2024-03-23T11:18:57.257058+0000","last_peered":"2024-03-23T11:18:57.257058+0000","last_clean":"2024-03-23T11:18:57.257058+0000","last_became_active":"2024-03-23T11:18:56.267418+0000","last_became_peered":"2024-03-23T11:18:56.267418+0000","last_unstale":"2024-03-23T11:18:57.257058+0000","last_undegraded":"2024-03-23T11:18:57.257058+0000","last_fullsized":"2024-03-23T11:18:57.257058+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:36:11.188974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252776+0000","last_change":"2024-03-23T11:18:55.775660+0000","last_active":"2024-03-23T11:18:57.252776+0000","last_peered":"2024-03-23T11:18:57.252776+0000","last_clean":"2024-03-23T11:18:57.252776+0000","last_became_active":"2024-03-23T11:18:55.775378+0000","last_became_peered":"2024-03-23T11:18:55.775378+0000","last_unstale":"2024-03-23T11:18:57.252776+0000","last_undegraded":"2024-03-23T11:18:57.252776+0000","last_fullsized":"2024-03-23T11:18:57.252776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:25:31.204286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100033+0000","last_change":"2024-03-23T11:18:56.273073+0000","last_active":"2024-03-23T11:18:58.100033+0000","last_peered":"2024-03-23T11:18:58.100033+0000","last_clean":"2024-03-23T11:18:58.100033+0000","last_became_active":"2024-03-23T11:18:56.265279+0000","last_became_peered":"2024-03-23T11:18:56.265279+0000","last_unstale":"2024-03-23T11:18:58.100033+0000","last_undegraded":"2024-03-23T11:18:58.100033+0000","last_fullsized":"2024-03-23T11:18:58.100033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252994+0000","last_change":"2024-03-23T11:18:55.773923+0000","last_active":"2024-03-23T11:18:57.252994+0000","last_peered":"2024-03-23T11:18:57.252994+0000","last_clean":"2024-03-23T11:18:57.252994+0000","last_became_active":"2024-03-23T11:18:55.773155+0000","last_became_peered":"2024-03-23T11:18:55.773155+0000","last_unstale":"2024-03-23T11:18:57.252994+0000","last_undegraded":"2024-03-23T11:18:57.252994+0000","last_fullsized":"2024-03-23T11:18:57.252994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:26.469433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101280+0000","last_change":"2024-03-23T11:18:56.265422+0000","last_active":"2024-03-23T11:18:58.101280+0000","last_peered":"2024-03-23T11:18:58.101280+0000","last_clean":"2024-03-23T11:18:58.101280+0000","last_became_active":"2024-03-23T11:18:56.265137+0000","last_became_peered":"2024-03-23T11:18:56.265137+0000","last_unstale":"2024-03-23T11:18:58.101280+0000","last_undegraded":"2024-03-23T11:18:58.101280+0000","last_fullsized":"2024-03-23T11:18:58.101280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:26.126571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256335+0000","last_change":"2024-03-23T11:18:55.778476+0000","last_active":"2024-03-23T11:18:57.256335+0000","last_peered":"2024-03-23T11:18:57.256335+0000","last_clean":"2024-03-23T11:18:57.256335+0000","last_became_active":"2024-03-23T11:18:55.778184+0000","last_became_peered":"2024-03-23T11:18:55.778184+0000","last_unstale":"2024-03-23T11:18:57.256335+0000","last_undegraded":"2024-03-23T11:18:57.256335+0000","last_fullsized":"2024-03-23T11:18:57.256335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:38.995255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462125+0000","last_change":"2024-03-23T11:18:56.277871+0000","last_active":"2024-03-23T11:18:57.462125+0000","last_peered":"2024-03-23T11:18:57.462125+0000","last_clean":"2024-03-23T11:18:57.462125+0000","last_became_active":"2024-03-23T11:18:56.277547+0000","last_became_peered":"2024-03-23T11:18:56.277547+0000","last_unstale":"2024-03-23T11:18:57.462125+0000","last_undegraded":"2024-03-23T11:18:57.462125+0000","last_fullsized":"2024-03-23T11:18:57.462125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.747240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315015+0000","last_change":"2024-03-23T11:18:56.288533+0000","last_active":"2024-03-23T11:18:57.315015+0000","last_peered":"2024-03-23T11:18:57.315015+0000","last_clean":"2024-03-23T11:18:57.315015+0000","last_became_active":"2024-03-23T11:18:56.288099+0000","last_became_peered":"2024-03-23T11:18:56.288099+0000","last_unstale":"2024-03-23T11:18:57.315015+0000","last_undegraded":"2024-03-23T11:18:57.315015+0000","last_fullsized":"2024-03-23T11:18:57.315015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:31:13.473526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256142+0000","last_change":"2024-03-23T11:18:55.784333+0000","last_active":"2024-03-23T11:18:57.256142+0000","last_peered":"2024-03-23T11:18:57.256142+0000","last_clean":"2024-03-23T11:18:57.256142+0000","last_became_active":"2024-03-23T11:18:55.783931+0000","last_became_peered":"2024-03-23T11:18:55.783931+0000","last_unstale":"2024-03-23T11:18:57.256142+0000","last_undegraded":"2024-03-23T11:18:57.256142+0000","last_fullsized":"2024-03-23T11:18:57.256142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:04:27.564911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315497+0000","last_change":"2024-03-23T11:18:56.287212+0000","last_active":"2024-03-23T11:18:57.315497+0000","last_peered":"2024-03-23T11:18:57.315497+0000","last_clean":"2024-03-23T11:18:57.315497+0000","last_became_active":"2024-03-23T11:18:56.284693+0000","last_became_peered":"2024-03-23T11:18:56.284693+0000","last_unstale":"2024-03-23T11:18:57.315497+0000","last_undegraded":"2024-03-23T11:18:57.315497+0000","last_fullsized":"2024-03-23T11:18:57.315497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:33.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256937+0000","last_change":"2024-03-23T11:18:55.779221+0000","last_active":"2024-03-23T11:18:57.256937+0000","last_peered":"2024-03-23T11:18:57.256937+0000","last_clean":"2024-03-23T11:18:57.256937+0000","last_became_active":"2024-03-23T11:18:55.778912+0000","last_became_peered":"2024-03-23T11:18:55.778912+0000","last_unstale":"2024-03-23T11:18:57.256937+0000","last_undegraded":"2024-03-23T11:18:57.256937+0000","last_fullsized":"2024-03-23T11:18:57.256937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:28:51.146588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099197+0000","last_change":"2024-03-23T11:18:56.291118+0000","last_active":"2024-03-23T11:18:58.099197+0000","last_peered":"2024-03-23T11:18:58.099197+0000","last_clean":"2024-03-23T11:18:58.099197+0000","last_became_active":"2024-03-23T11:18:56.290750+0000","last_became_peered":"2024-03-23T11:18:56.290750+0000","last_unstale":"2024-03-23T11:18:58.099197+0000","last_undegraded":"2024-03-23T11:18:58.099197+0000","last_fullsized":"2024-03-23T11:18:58.099197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101527+0000","last_change":"2024-03-23T11:18:55.784254+0000","last_active":"2024-03-23T11:18:58.101527+0000","last_peered":"2024-03-23T11:18:58.101527+0000","last_clean":"2024-03-23T11:18:58.101527+0000","last_became_active":"2024-03-23T11:18:55.781763+0000","last_became_peered":"2024-03-23T11:18:55.781763+0000","last_unstale":"2024-03-23T11:18:58.101527+0000","last_undegraded":"2024-03-23T11:18:58.101527+0000","last_fullsized":"2024-03-23T11:18:58.101527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101387+0000","last_change":"2024-03-23T11:18:56.267771+0000","last_active":"2024-03-23T11:18:58.101387+0000","last_peered":"2024-03-23T11:18:58.101387+0000","last_clean":"2024-03-23T11:18:58.101387+0000","last_became_active":"2024-03-23T11:18:56.267515+0000","last_became_peered":"2024-03-23T11:18:56.267515+0000","last_unstale":"2024-03-23T11:18:58.101387+0000","last_undegraded":"2024-03-23T11:18:58.101387+0000","last_fullsized":"2024-03-23T11:18:58.101387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099857+0000","last_change":"2024-03-23T11:18:55.785970+0000","last_active":"2024-03-23T11:18:58.099857+0000","last_peered":"2024-03-23T11:18:58.099857+0000","last_clean":"2024-03-23T11:18:58.099857+0000","last_became_active":"2024-03-23T11:18:55.784347+0000","last_became_peered":"2024-03-23T11:18:55.784347+0000","last_unstale":"2024-03-23T11:18:58.099857+0000","last_undegraded":"2024-03-23T11:18:58.099857+0000","last_fullsized":"2024-03-23T11:18:58.099857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:36:21.274877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":823296,"data_stored":669740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8580000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.764}]}]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0659999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.879}]}]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8029999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.75}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0750000000000002}]}]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1040000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.591}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.768}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.966}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0659999999999998}]}]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.976}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.659}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.909}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0459999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.585}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.206}]}]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6060000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.675}]}]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":78281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.369}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4390000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542147,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":827392,"data_stored":669127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8719999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:19:01.763 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T11:19:01.763 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-23T11:19:01.763 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-23T11:19:01.763 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-23T11:19:01.764 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-23T11:19:02.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.026+0000 7fec62200640 1 Processor -- start 2024-03-23T11:19:02.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.026+0000 7fec62200640 1 -- start start 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.028+0000 7fec62200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fec5c143dd0 0x7fec5c15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.028+0000 7fec62200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fec5c0b2f30 0x7fec5c070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.028+0000 7fec62200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5c0cd140 con 0x7fec5c070090 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.028+0000 7fec62200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5c070a60 con 0x7fec5c0b2f30 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.028+0000 7fec62200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5c0cd3e0 con 0x7fec5c143dd0 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.028+0000 7fec61800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fec5c0b2f30 0x7fec5c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.028+0000 7fec61800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fec5c0b2f30 0x7fec5c070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37728/0 (socket says 172.21.15.62:37728) 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.028+0000 7fec61800640 1 -- 172.21.15.62:0/3326705320 learned_addr learned my addr 172.21.15.62:0/3326705320 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.029+0000 7fec60e00640 1 --2- 172.21.15.62:0/3326705320 >> v2:172.21.15.120:3300/0 conn(0x7fec5c143dd0 0x7fec5c15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.029+0000 7fec61800640 1 -- 172.21.15.62:0/3326705320 >> v2:172.21.15.120:3300/0 conn(0x7fec5c143dd0 msgr2=0x7fec5c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.029+0000 7fec61800640 1 --2- 172.21.15.62:0/3326705320 >> v2:172.21.15.120:3300/0 conn(0x7fec5c143dd0 0x7fec5c15abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.029+0000 7fec61800640 1 -- 172.21.15.62:0/3326705320 >> v1:172.21.15.120:6789/0 conn(0x7fec5c070090 legacy=0x7fec5c0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.029+0000 7fec61800640 1 -- 172.21.15.62:0/3326705320 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c0cd6c0 con 0x7fec5c0b2f30 2024-03-23T11:19:02.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.030+0000 7fec61800640 1 --2- 172.21.15.62:0/3326705320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec5c0b2f30 0x7fec5c070520 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fec5c06e380 tx=0x7fec5402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d56bfc63f5d4cd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:02.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.030+0000 7fec5b400640 1 -- 172.21.15.62:0/3326705320 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5402ca80 con 0x7fec5c0b2f30 2024-03-23T11:19:02.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.030+0000 7fec5b400640 1 -- 172.21.15.62:0/3326705320 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5402cbe0 con 0x7fec5c0b2f30 2024-03-23T11:19:02.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.030+0000 7fec5b400640 1 -- 172.21.15.62:0/3326705320 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec540343f0 con 0x7fec5c0b2f30 2024-03-23T11:19:02.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.031+0000 7fec62200640 1 -- 172.21.15.62:0/3326705320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec5c0b2f30 msgr2=0x7fec5c070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.031+0000 7fec62200640 1 --2- 172.21.15.62:0/3326705320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec5c0b2f30 0x7fec5c070520 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fec5c06e380 tx=0x7fec5402bee0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.031+0000 7fec62200640 1 -- 172.21.15.62:0/3326705320 shutdown_connections 2024-03-23T11:19:02.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.031+0000 7fec62200640 1 --2- 172.21.15.62:0/3326705320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec5c0b2f30 0x7fec5c070520 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.031+0000 7fec62200640 1 --2- 172.21.15.62:0/3326705320 >> v2:172.21.15.120:3300/0 conn(0x7fec5c143dd0 0x7fec5c15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.031+0000 7fec62200640 1 -- 172.21.15.62:0/3326705320 >> 172.21.15.62:0/3326705320 conn(0x7fec5c066050 msgr2=0x7fec5c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:02.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.031+0000 7fec62200640 1 -- 172.21.15.62:0/3326705320 shutdown_connections 2024-03-23T11:19:02.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.031+0000 7fec62200640 1 -- 172.21.15.62:0/3326705320 wait complete. 2024-03-23T11:19:02.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.033+0000 7fec62200640 1 Processor -- start 2024-03-23T11:19:02.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.033+0000 7fec62200640 1 -- start start 2024-03-23T11:19:02.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec62200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec5c070090 0x7fec5c132090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec62200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec5c0b2f30 0x7fec5c1f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec62200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec5c143dd0 0x7fec5c159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec62200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c12f850 con 0x7fec5c143dd0 2024-03-23T11:19:02.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec62200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c1f4380 con 0x7fec5c070090 2024-03-23T11:19:02.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec62200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c12ec40 con 0x7fec5c0b2f30 2024-03-23T11:19:02.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec61800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec5c143dd0 0x7fec5c159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec61800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec5c143dd0 0x7fec5c159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37738/0 (socket says 172.21.15.62:37738) 2024-03-23T11:19:02.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec61800640 1 -- 172.21.15.62:0/4058094338 learned_addr learned my addr 172.21.15.62:0/4058094338 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:02.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec5be00640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec5c0b2f30 0x7fec5c1f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec61800640 1 -- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec5c0b2f30 msgr2=0x7fec5c1f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec60e00640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec5c070090 0x7fec5c132090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec61800640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec5c0b2f30 0x7fec5c1f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec61800640 1 -- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec5c070090 msgr2=0x7fec5c132090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec61800640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec5c070090 0x7fec5c132090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec61800640 1 -- 172.21.15.62:0/4058094338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec54009b30 con 0x7fec5c143dd0 2024-03-23T11:19:02.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.034+0000 7fec5be00640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec5c0b2f30 0x7fec5c1f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:02.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.035+0000 7fec61800640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec5c143dd0 0x7fec5c159bf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fec54009f40 tx=0x7fec54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:02.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.035+0000 7fec58c00640 1 -- 172.21.15.62:0/4058094338 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54009510 con 0x7fec5c143dd0 2024-03-23T11:19:02.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.035+0000 7fec58c00640 1 -- 172.21.15.62:0/4058094338 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5403e450 con 0x7fec5c143dd0 2024-03-23T11:19:02.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.035+0000 7fec58c00640 1 -- 172.21.15.62:0/4058094338 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5403e7e0 con 0x7fec5c143dd0 2024-03-23T11:19:02.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.035+0000 7fec62200640 1 -- 172.21.15.62:0/4058094338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5c12ef20 con 0x7fec5c143dd0 2024-03-23T11:19:02.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.035+0000 7fec62200640 1 -- 172.21.15.62:0/4058094338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5c12f4e0 con 0x7fec5c143dd0 2024-03-23T11:19:02.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.036+0000 7fec62200640 1 -- 172.21.15.62:0/4058094338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5c12fa00 con 0x7fec5c143dd0 2024-03-23T11:19:02.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.037+0000 7fec58c00640 1 -- 172.21.15.62:0/4058094338 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fec5402cd50 con 0x7fec5c143dd0 2024-03-23T11:19:02.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.038+0000 7fec58c00640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec3c06a7a0 0x7fec3c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.038+0000 7fec58c00640 1 -- 172.21.15.62:0/4058094338 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec540ab950 con 0x7fec5c143dd0 2024-03-23T11:19:02.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.039+0000 7fec60e00640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec3c06a7a0 0x7fec3c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.040+0000 7fec60e00640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec3c06a7a0 0x7fec3c06cc60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fec50006fd0 tx=0x7fec50008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:02.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.042+0000 7fec58c00640 1 -- 172.21.15.62:0/4058094338 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec5c12fa00 con 0x7fec5c143dd0 2024-03-23T11:19:02.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.293+0000 7fec62200640 1 -- 172.21.15.62:0/4058094338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fec5c12fa00 con 0x7fec5c143dd0 2024-03-23T11:19:02.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.301+0000 7fec58c00640 1 -- 172.21.15.62:0/4058094338 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170396 (secure 0 0 0) 0x7fec5c12fa00 con 0x7fec5c143dd0 2024-03-23T11:19:02.308 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:02.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.305+0000 7fec62200640 1 -- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec3c06a7a0 msgr2=0x7fec3c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.305+0000 7fec62200640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec3c06a7a0 0x7fec3c06cc60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fec50006fd0 tx=0x7fec50008040 comp rx=0 tx=0).stop 2024-03-23T11:19:02.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.305+0000 7fec62200640 1 -- 172.21.15.62:0/4058094338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec5c143dd0 msgr2=0x7fec5c159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.305+0000 7fec62200640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec5c143dd0 0x7fec5c159bf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fec54009f40 tx=0x7fec54004a40 comp rx=0 tx=0).stop 2024-03-23T11:19:02.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.305+0000 7fec62200640 1 -- 172.21.15.62:0/4058094338 shutdown_connections 2024-03-23T11:19:02.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.305+0000 7fec62200640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec3c06a7a0 0x7fec3c06cc60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.305+0000 7fec62200640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec5c143dd0 0x7fec5c159bf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.305+0000 7fec62200640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec5c0b2f30 0x7fec5c1f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.305+0000 7fec62200640 1 --2- 172.21.15.62:0/4058094338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec5c070090 0x7fec5c132090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.305+0000 7fec62200640 1 -- 172.21.15.62:0/4058094338 >> 172.21.15.62:0/4058094338 conn(0x7fec5c066050 msgr2=0x7fec5c0b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:02.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.306+0000 7fec62200640 1 -- 172.21.15.62:0/4058094338 shutdown_connections 2024-03-23T11:19:02.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.306+0000 7fec62200640 1 -- 172.21.15.62:0/4058094338 wait complete. 2024-03-23T11:19:02.325 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":5,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":37071},{"type":"v1","addr":"172.21.15.120:6833","nonce":37071}]},"active_addr":"172.21.15.120:6833/37071","active_change":"2024-03-23T11:18:41.209191+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4106,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.120:0","nonce":1403053187}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.120:0","nonce":4272110029}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.120:0","nonce":3865210680}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.120:0","nonce":3710360632}]}]} 2024-03-23T11:19:02.329 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-23T11:19:02.329 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-23T11:19:02.329 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:02.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.547+0000 7f6403e00640 1 Processor -- start 2024-03-23T11:19:02.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.547+0000 7f6403e00640 1 -- start start 2024-03-23T11:19:02.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.549+0000 7f6403e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f64040b2f30 0x7f6404070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.549+0000 7f6403e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64040cd140 con 0x7f6404143dd0 2024-03-23T11:19:02.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.549+0000 7f6403e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64040cd3e0 con 0x7f6404070090 2024-03-23T11:19:02.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.549+0000 7f6403e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6404070a60 con 0x7f64040b2f30 2024-03-23T11:19:02.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.550+0000 7f6402a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6404143dd0 0x7f640415abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53426/0 (socket says 172.21.15.62:53426) 2024-03-23T11:19:02.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.550+0000 7f6402a00640 1 -- 172.21.15.62:0/1154996787 learned_addr learned my addr 172.21.15.62:0/1154996787 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:02.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.550+0000 7f6403400640 1 --2- 172.21.15.62:0/1154996787 >> v2:172.21.15.120:3300/0 conn(0x7f64040b2f30 0x7f6404070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.550+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582668005 0 0) 0x7f64040cd140 con 0x7f6404143dd0 2024-03-23T11:19:02.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.550+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e8003520 con 0x7f6404143dd0 2024-03-23T11:19:02.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.550+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3402359664 0 0) 0x7f63e8003520 con 0x7f6404143dd0 2024-03-23T11:19:02.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.550+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 >> v2:172.21.15.120:3300/0 conn(0x7f64040b2f30 msgr2=0x7f6404070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.550+0000 7f6402000640 1 --2- 172.21.15.62:0/1154996787 >> v2:172.21.15.120:3300/0 conn(0x7f64040b2f30 0x7f6404070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.550+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 >> v1:172.21.15.120:6789/0 conn(0x7f6404070090 legacy=0x7f64040b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.550+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64040cd6c0 con 0x7f6404143dd0 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.550+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f63f4001670 con 0x7f6404143dd0 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 >> v1:172.21.15.62:6789/0 conn(0x7f6404143dd0 legacy=0x7f640415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6402000640 1 --2- 172.21.15.62:0/1154996787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e8003970 0x7f63e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6402000640 1 --2- 172.21.15.62:0/1154996787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e80047c0 0x7f63e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6402000640 1 --2- 172.21.15.62:0/1154996787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e8005180 0x7f63e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6404070a60 con 0x7f63e8005180 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e800c020 con 0x7f63e8003970 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e800c1e0 con 0x7f63e80047c0 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6403400640 1 --2- 172.21.15.62:0/1154996787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e8005180 0x7f63e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6402a00640 1 --2- 172.21.15.62:0/1154996787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e8003970 0x7f63e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f63fbe00640 1 --2- 172.21.15.62:0/1154996787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e80047c0 0x7f63e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6403400640 1 -- 172.21.15.62:0/1154996787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e80047c0 msgr2=0x7f63e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:02.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6403400640 1 --2- 172.21.15.62:0/1154996787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e80047c0 0x7f63e8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6403400640 1 -- 172.21.15.62:0/1154996787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e8003970 msgr2=0x7f63e8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:02.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6403400640 1 --2- 172.21.15.62:0/1154996787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e8003970 0x7f63e8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.551+0000 7f6403400640 1 -- 172.21.15.62:0/1154996787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e8003520 con 0x7f63e8005180 2024-03-23T11:19:02.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.552+0000 7f6403400640 1 -- 172.21.15.62:0/1154996787 reap_dead start 2024-03-23T11:19:02.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.553+0000 7f6403400640 1 --2- 172.21.15.62:0/1154996787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e8005180 0x7f63e800b980 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f63fc009fc0 tx=0x7f63e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26baed48597519d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:02.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.553+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02cc60 con 0x7f63e8005180 2024-03-23T11:19:02.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.553+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00f040 con 0x7f63e8005180 2024-03-23T11:19:02.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.553+0000 7f6402000640 1 -- 172.21.15.62:0/1154996787 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc003990 con 0x7f63e8005180 2024-03-23T11:19:02.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.553+0000 7f6403e00640 1 -- 172.21.15.62:0/1154996787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e8005180 msgr2=0x7f63e800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.553+0000 7f6403e00640 1 --2- 172.21.15.62:0/1154996787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e8005180 0x7f63e800b980 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f63fc009fc0 tx=0x7f63e80057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.554+0000 7f6403e00640 1 -- 172.21.15.62:0/1154996787 shutdown_connections 2024-03-23T11:19:02.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.554+0000 7f6403e00640 1 --2- 172.21.15.62:0/1154996787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e8005180 0x7f63e800b980 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.554+0000 7f6403e00640 1 -- 172.21.15.62:0/1154996787 >> 172.21.15.62:0/1154996787 conn(0x7f6404066050 msgr2=0x7f6404069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:02.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.554+0000 7f6403e00640 1 -- 172.21.15.62:0/1154996787 shutdown_connections 2024-03-23T11:19:02.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.554+0000 7f6403e00640 1 -- 172.21.15.62:0/1154996787 wait complete. 2024-03-23T11:19:02.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.555+0000 7f6403e00640 1 Processor -- start 2024-03-23T11:19:02.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.556+0000 7f6403e00640 1 -- start start 2024-03-23T11:19:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f6403e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e8005180 0x7f640406c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f6403e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640413ac40 0x7f6404139200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f6403e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64040b1920 0x7f64040b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f6403e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64040b22f0 con 0x7f640413ac40 2024-03-23T11:19:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f6403e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640406c890 con 0x7f64040b1920 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f6403e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64041f64b0 con 0x7f63e8005180 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f63fbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640413ac40 0x7f6404139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f63fbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640413ac40 0x7f6404139200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37760/0 (socket says 172.21.15.62:37760) 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f63fbe00640 1 -- 172.21.15.62:0/3382104783 learned_addr learned my addr 172.21.15.62:0/3382104783 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f6402a00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e8005180 0x7f640406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f6403400640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64040b1920 0x7f64040b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f63fbe00640 1 -- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e8005180 msgr2=0x7f640406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f63fbe00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e8005180 0x7f640406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f63fbe00640 1 -- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64040b1920 msgr2=0x7f64040b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f63fbe00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64040b1920 0x7f64040b1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f63fbe00640 1 -- 172.21.15.62:0/3382104783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64041f5930 con 0x7f640413ac40 2024-03-23T11:19:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.557+0000 7f6402a00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e8005180 0x7f640406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.558+0000 7f63fbe00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640413ac40 0x7f6404139200 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00e4e0 tx=0x7f63ec00e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.558+0000 7f63faa00640 1 -- 172.21.15.62:0/3382104783 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec0052e0 con 0x7f640413ac40 2024-03-23T11:19:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.558+0000 7f63faa00640 1 -- 172.21.15.62:0/3382104783 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ec005440 con 0x7f640413ac40 2024-03-23T11:19:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.558+0000 7f6403e00640 1 -- 172.21.15.62:0/3382104783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64041f5c10 con 0x7f640413ac40 2024-03-23T11:19:02.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.558+0000 7f63faa00640 1 -- 172.21.15.62:0/3382104783 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec003830 con 0x7f640413ac40 2024-03-23T11:19:02.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.558+0000 7f6403e00640 1 -- 172.21.15.62:0/3382104783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64041f61d0 con 0x7f640413ac40 2024-03-23T11:19:02.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.559+0000 7f6403e00640 1 -- 172.21.15.62:0/3382104783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64041f1f60 con 0x7f640413ac40 2024-03-23T11:19:02.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.560+0000 7f63faa00640 1 -- 172.21.15.62:0/3382104783 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f63ec003990 con 0x7f640413ac40 2024-03-23T11:19:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.561+0000 7f63faa00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63e006a7a0 0x7f63e006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.561+0000 7f63faa00640 1 -- 172.21.15.62:0/3382104783 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63ec088b10 con 0x7f640413ac40 2024-03-23T11:19:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.561+0000 7f6402a00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63e006a7a0 0x7f63e006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:02.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.562+0000 7f6402a00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63e006a7a0 0x7f63e006cc60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f63f4001200 tx=0x7f63f4000f70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:02.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.565+0000 7f63faa00640 1 -- 172.21.15.62:0/3382104783 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f63ec053e90 con 0x7f640413ac40 2024-03-23T11:19:02.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.765+0000 7f6403e00640 1 -- 172.21.15.62:0/3382104783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64041f1f60 con 0x7f640413ac40 2024-03-23T11:19:02.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.768+0000 7f63faa00640 1 -- 172.21.15.62:0/3382104783 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f63ec0574b0 con 0x7f640413ac40 2024-03-23T11:19:02.768 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:02.768 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:57.238201+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:02.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 -- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63e006a7a0 msgr2=0x7f63e006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63e006a7a0 0x7f63e006cc60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f63f4001200 tx=0x7f63f4000f70 comp rx=0 tx=0).stop 2024-03-23T11:19:02.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 -- 172.21.15.62:0/3382104783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640413ac40 msgr2=0x7f6404139200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:02.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640413ac40 0x7f6404139200 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00e4e0 tx=0x7f63ec00e9a0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 -- 172.21.15.62:0/3382104783 shutdown_connections 2024-03-23T11:19:02.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63e006a7a0 0x7f63e006cc60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640413ac40 0x7f6404139200 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64040b1920 0x7f64040b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 --2- 172.21.15.62:0/3382104783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e8005180 0x7f640406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:02.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 -- 172.21.15.62:0/3382104783 >> 172.21.15.62:0/3382104783 conn(0x7f6404066050 msgr2=0x7f64040b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:02.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 -- 172.21.15.62:0/3382104783 shutdown_connections 2024-03-23T11:19:02.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:02.771+0000 7f6403e00640 1 -- 172.21.15.62:0/3382104783 wait complete. 2024-03-23T11:19:02.790 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-23T11:19:02.790 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:03.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.014+0000 7f05a7e00640 1 Processor -- start 2024-03-23T11:19:03.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.014+0000 7f05a7e00640 1 -- start start 2024-03-23T11:19:03.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.016+0000 7f05a7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f05a8070090 0x7f05a8070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.016+0000 7f05a7e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a80cd140 con 0x7f05a80b2820 2024-03-23T11:19:03.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.016+0000 7f05a7e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a80cd3e0 con 0x7f05a8143dd0 2024-03-23T11:19:03.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.016+0000 7f05a7e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a8070a60 con 0x7f05a8070090 2024-03-23T11:19:03.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.016+0000 7f05a6000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f05a8070090 0x7f05a8070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.016+0000 7f05a6000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f05a8070090 0x7f05a8070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37772/0 (socket says 172.21.15.62:37772) 2024-03-23T11:19:03.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.016+0000 7f05a6000640 1 -- 172.21.15.62:0/1794718706 learned_addr learned my addr 172.21.15.62:0/1794718706 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.016+0000 7f05a7400640 1 --1- 172.21.15.62:0/1794718706 >> v1:172.21.15.62:6789/0 conn(0x7f05a80b2820 0x7f05a80b2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53442/0 (socket says 172.21.15.62:53442) 2024-03-23T11:19:03.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.017+0000 7f05a6000640 1 -- 172.21.15.62:0/1794718706 >> v1:172.21.15.120:6789/0 conn(0x7f05a8143dd0 legacy=0x7f05a815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.017+0000 7f05a6000640 1 -- 172.21.15.62:0/1794718706 >> v1:172.21.15.62:6789/0 conn(0x7f05a80b2820 legacy=0x7f05a80b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.017+0000 7f05a6000640 1 -- 172.21.15.62:0/1794718706 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a80cd6c0 con 0x7f05a8070090 2024-03-23T11:19:03.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.017+0000 7f05a5600640 1 -- 172.21.15.62:0/1794718706 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343650471 0 0) 0x7f05a80cd140 con 0x7f05a80b2820 2024-03-23T11:19:03.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.018+0000 7f05a6000640 1 --2- 172.21.15.62:0/1794718706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05a8070090 0x7f05a8070520 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0594009870 tx=0x7f059402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61824009d66f18c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.018+0000 7f05a5600640 1 -- 172.21.15.62:0/1794718706 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594030850 con 0x7f05a8070090 2024-03-23T11:19:03.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.018+0000 7f05a5600640 1 -- 172.21.15.62:0/1794718706 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05940309b0 con 0x7f05a8070090 2024-03-23T11:19:03.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.018+0000 7f05a5600640 1 -- 172.21.15.62:0/1794718706 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594030d40 con 0x7f05a8070090 2024-03-23T11:19:03.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.019+0000 7f05a7e00640 1 -- 172.21.15.62:0/1794718706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05a8070090 msgr2=0x7f05a8070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.019+0000 7f05a7e00640 1 --2- 172.21.15.62:0/1794718706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05a8070090 0x7f05a8070520 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0594009870 tx=0x7f059402b990 comp rx=0 tx=0).stop 2024-03-23T11:19:03.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.019+0000 7f05a7e00640 1 -- 172.21.15.62:0/1794718706 shutdown_connections 2024-03-23T11:19:03.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.019+0000 7f05a7e00640 1 --2- 172.21.15.62:0/1794718706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05a8070090 0x7f05a8070520 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.019+0000 7f05a7e00640 1 -- 172.21.15.62:0/1794718706 >> 172.21.15.62:0/1794718706 conn(0x7f05a8066050 msgr2=0x7f05a8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.019+0000 7f05a7e00640 1 -- 172.21.15.62:0/1794718706 shutdown_connections 2024-03-23T11:19:03.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.019+0000 7f05a7e00640 1 -- 172.21.15.62:0/1794718706 wait complete. 2024-03-23T11:19:03.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.021+0000 7f05a7e00640 1 Processor -- start 2024-03-23T11:19:03.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.021+0000 7f05a7e00640 1 -- start start 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.022+0000 7f05a7e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05a8070090 0x7f05a80c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.022+0000 7f05a7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05a80b2820 0x7f05a815a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.022+0000 7f05a7e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05a8143dd0 0x7f05a80ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.022+0000 7f05a7e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8134c90 con 0x7f05a80b2820 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.022+0000 7f05a7e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8142050 con 0x7f05a8070090 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.022+0000 7f05a7e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80c42f0 con 0x7f05a8143dd0 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.022+0000 7f05a6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05a80b2820 0x7f05a815a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.022+0000 7f05a6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05a80b2820 0x7f05a815a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37778/0 (socket says 172.21.15.62:37778) 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.022+0000 7f05a6000640 1 -- 172.21.15.62:0/284880877 learned_addr learned my addr 172.21.15.62:0/284880877 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f05a6a00640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05a8070090 0x7f05a80c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f05a7400640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05a8143dd0 0x7f05a80ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f05a6000640 1 -- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05a8143dd0 msgr2=0x7f05a80ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f05a6000640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05a8143dd0 0x7f05a80ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f05a6000640 1 -- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05a8070090 msgr2=0x7f05a80c3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f05a6000640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05a8070090 0x7f05a80c3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f05a6000640 1 -- 172.21.15.62:0/284880877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05940094d0 con 0x7f05a80b2820 2024-03-23T11:19:03.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f05a6a00640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05a8070090 0x7f05a80c3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:03.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f05a6000640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05a80b2820 0x7f05a815a5b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0594009870 tx=0x7f0594004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f059ea00640 1 -- 172.21.15.62:0/284880877 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594039d60 con 0x7f05a80b2820 2024-03-23T11:19:03.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f059ea00640 1 -- 172.21.15.62:0/284880877 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0594043400 con 0x7f05a80b2820 2024-03-23T11:19:03.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f059ea00640 1 -- 172.21.15.62:0/284880877 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05940437b0 con 0x7f05a80b2820 2024-03-23T11:19:03.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.023+0000 7f05a7e00640 1 -- 172.21.15.62:0/284880877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a80c45d0 con 0x7f05a80b2820 2024-03-23T11:19:03.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.024+0000 7f05a7e00640 1 -- 172.21.15.62:0/284880877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a80c4b90 con 0x7f05a80b2820 2024-03-23T11:19:03.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.025+0000 7f05a7e00640 1 -- 172.21.15.62:0/284880877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a8066bb0 con 0x7f05a80b2820 2024-03-23T11:19:03.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.026+0000 7f059ea00640 1 -- 172.21.15.62:0/284880877 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f059402e070 con 0x7f05a80b2820 2024-03-23T11:19:03.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.026+0000 7f059ea00640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f058806a7a0 0x7f058806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.027+0000 7f059ea00640 1 -- 172.21.15.62:0/284880877 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f059403e070 con 0x7f05a80b2820 2024-03-23T11:19:03.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.027+0000 7f05a6a00640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f058806a7a0 0x7f058806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.028+0000 7f05a6a00640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f058806a7a0 0x7f058806cc60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f059800a830 tx=0x7f0598008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.031+0000 7f059ea00640 1 -- 172.21.15.62:0/284880877 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0594077120 con 0x7f05a80b2820 2024-03-23T11:19:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.229+0000 7f05a7e00640 1 -- 172.21.15.62:0/284880877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05a80b35a0 con 0x7f05a80b2820 2024-03-23T11:19:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.232+0000 7f059ea00640 1 -- 172.21.15.62:0/284880877 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f0594034090 con 0x7f05a80b2820 2024-03-23T11:19:03.232 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:03.232 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:57.238201+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:03.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.234+0000 7f05a7e00640 1 -- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f058806a7a0 msgr2=0x7f058806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.234+0000 7f05a7e00640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f058806a7a0 0x7f058806cc60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f059800a830 tx=0x7f0598008040 comp rx=0 tx=0).stop 2024-03-23T11:19:03.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.235+0000 7f05a7e00640 1 -- 172.21.15.62:0/284880877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05a80b2820 msgr2=0x7f05a815a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.235+0000 7f05a7e00640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05a80b2820 0x7f05a815a5b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0594009870 tx=0x7f0594004730 comp rx=0 tx=0).stop 2024-03-23T11:19:03.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.235+0000 7f05a7e00640 1 -- 172.21.15.62:0/284880877 shutdown_connections 2024-03-23T11:19:03.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.235+0000 7f05a7e00640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f058806a7a0 0x7f058806cc60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.235+0000 7f05a7e00640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05a80b2820 0x7f05a815a5b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.235+0000 7f05a7e00640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05a8143dd0 0x7f05a80ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.235+0000 7f05a7e00640 1 --2- 172.21.15.62:0/284880877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05a8070090 0x7f05a80c3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.235+0000 7f05a7e00640 1 -- 172.21.15.62:0/284880877 >> 172.21.15.62:0/284880877 conn(0x7f05a8066050 msgr2=0x7f05a806d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.235+0000 7f05a7e00640 1 -- 172.21.15.62:0/284880877 shutdown_connections 2024-03-23T11:19:03.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.235+0000 7f05a7e00640 1 -- 172.21.15.62:0/284880877 wait complete. 2024-03-23T11:19:03.253 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-23T11:19:03.253 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-23T11:19:03.253 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-23T11:19:03.254 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-23T11:19:03.254 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-23T11:19:03.255 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-23T11:19:03.255 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-23T11:19:03.255 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-23T11:19:03.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.551+0000 7f6071200640 1 Processor -- start 2024-03-23T11:19:03.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.551+0000 7f6071200640 1 -- start start 2024-03-23T11:19:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.554+0000 7f6071200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f606c15c9e0 0x7f606c1607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.554+0000 7f6071200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f606c0cf650 0x7f606c0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.554+0000 7f6071200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c06bfc0 con 0x7f606c15cdc0 2024-03-23T11:19:03.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.555+0000 7f6071200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c0d0000 con 0x7f606c15c9e0 2024-03-23T11:19:03.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.555+0000 7f6071200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c0d2630 con 0x7f606c0cf650 2024-03-23T11:19:03.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.555+0000 7f6069400640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f606c15cdc0 0x7f606c0d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41590/0 (socket says 172.21.15.62:41590) 2024-03-23T11:19:03.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.555+0000 7f6069400640 1 -- 172.21.15.62:0/2392768385 learned_addr learned my addr 172.21.15.62:0/2392768385 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:03.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.555+0000 7f6069e00640 1 --2- 172.21.15.62:0/2392768385 >> v2:172.21.15.62:3300/0 conn(0x7f606c15c9e0 0x7f606c1607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.556+0000 7f606a800640 1 --2- 172.21.15.62:0/2392768385 >> v2:172.21.15.120:3300/0 conn(0x7f606c0cf650 0x7f606c0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.556+0000 7f6069e00640 1 -- 172.21.15.62:0/2392768385 >> v2:172.21.15.120:3300/0 conn(0x7f606c0cf650 msgr2=0x7f606c0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.556+0000 7f6069e00640 1 --2- 172.21.15.62:0/2392768385 >> v2:172.21.15.120:3300/0 conn(0x7f606c0cf650 0x7f606c0cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.556+0000 7f6069e00640 1 -- 172.21.15.62:0/2392768385 >> v1:172.21.15.120:6789/0 conn(0x7f606c15cdc0 legacy=0x7f606c0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.556+0000 7f6069e00640 1 -- 172.21.15.62:0/2392768385 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c0b5260 con 0x7f606c15c9e0 2024-03-23T11:19:03.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.558+0000 7f6069e00640 1 --2- 172.21.15.62:0/2392768385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f606c15c9e0 0x7f606c1607c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f606c06b570 tx=0x7f606402bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a743c60c0808ad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.559+0000 7f6068a00640 1 -- 172.21.15.62:0/2392768385 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606402ccb0 con 0x7f606c15c9e0 2024-03-23T11:19:03.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.559+0000 7f6068a00640 1 -- 172.21.15.62:0/2392768385 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606402ce10 con 0x7f606c15c9e0 2024-03-23T11:19:03.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.559+0000 7f6068a00640 1 -- 172.21.15.62:0/2392768385 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6064034660 con 0x7f606c15c9e0 2024-03-23T11:19:03.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.560+0000 7f6071200640 1 -- 172.21.15.62:0/2392768385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f606c15c9e0 msgr2=0x7f606c1607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.560+0000 7f6071200640 1 --2- 172.21.15.62:0/2392768385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f606c15c9e0 0x7f606c1607c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f606c06b570 tx=0x7f606402bf00 comp rx=0 tx=0).stop 2024-03-23T11:19:03.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.561+0000 7f6071200640 1 -- 172.21.15.62:0/2392768385 shutdown_connections 2024-03-23T11:19:03.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.561+0000 7f6071200640 1 --2- 172.21.15.62:0/2392768385 >> v2:172.21.15.120:3300/0 conn(0x7f606c0cf650 0x7f606c0cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.561+0000 7f6071200640 1 --2- 172.21.15.62:0/2392768385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f606c15c9e0 0x7f606c1607c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.561+0000 7f6071200640 1 -- 172.21.15.62:0/2392768385 >> 172.21.15.62:0/2392768385 conn(0x7f606c071570 msgr2=0x7f606c071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.561+0000 7f6071200640 1 -- 172.21.15.62:0/2392768385 shutdown_connections 2024-03-23T11:19:03.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.561+0000 7f6071200640 1 -- 172.21.15.62:0/2392768385 wait complete. 2024-03-23T11:19:03.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.561+0000 7f765ea00640 1 Processor -- start 2024-03-23T11:19:03.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.561+0000 7f765ea00640 1 -- start start 2024-03-23T11:19:03.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.563+0000 7f765ea00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f765815c9e0 0x7f76581607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.563+0000 7f765ea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f765815cdc0 0x7f76580d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.563+0000 7f765ea00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765806bfc0 con 0x7f76580cf650 2024-03-23T11:19:03.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.563+0000 7f765ea00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76580ce140 con 0x7f765815c9e0 2024-03-23T11:19:03.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.563+0000 7f765ea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76580b4370 con 0x7f765815cdc0 2024-03-23T11:19:03.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.564+0000 7f952ce00640 1 Processor -- start 2024-03-23T11:19:03.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.564+0000 7f6071200640 1 Processor -- start 2024-03-23T11:19:03.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.564+0000 7f952ce00640 1 -- start start 2024-03-23T11:19:03.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.564+0000 7f765ca00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f76580cf650 0x7f765815d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41604/0 (socket says 172.21.15.62:41604) 2024-03-23T11:19:03.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.564+0000 7f765ca00640 1 -- 172.21.15.62:0/225235271 learned_addr learned my addr 172.21.15.62:0/225235271 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:03.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.564+0000 7f6071200640 1 -- start start 2024-03-23T11:19:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.564+0000 7f7657400640 1 --2- 172.21.15.62:0/225235271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f765815cdc0 0x7f76580d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.565+0000 7f7657e00640 1 --2- 172.21.15.62:0/225235271 >> v2:172.21.15.120:3300/0 conn(0x7f765815c9e0 0x7f76581607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.565+0000 7f7656a00640 1 -- 172.21.15.62:0/225235271 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40947620 0 0) 0x7f765806bfc0 con 0x7f76580cf650 2024-03-23T11:19:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.565+0000 7f7657400640 1 -- 172.21.15.62:0/225235271 >> v2:172.21.15.120:3300/0 conn(0x7f765815c9e0 msgr2=0x7f76581607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.565+0000 7f7657400640 1 --2- 172.21.15.62:0/225235271 >> v2:172.21.15.120:3300/0 conn(0x7f765815c9e0 0x7f76581607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.565+0000 7f7657400640 1 -- 172.21.15.62:0/225235271 >> v1:172.21.15.120:6789/0 conn(0x7f76580cf650 legacy=0x7f765815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.565+0000 7f7657400640 1 -- 172.21.15.62:0/225235271 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76580b5260 con 0x7f765815cdc0 2024-03-23T11:19:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.565+0000 7f7657e00640 1 --2- 172.21.15.62:0/225235271 >> v2:172.21.15.120:3300/0 conn(0x7f765815c9e0 0x7f76581607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:03.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.567+0000 7f6071200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f606c0cf650 0x7f606c1f1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.567+0000 7f6071200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f606c15c9e0 0x7f606c0cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.567+0000 7f6071200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f606c15cdc0 0x7f606c160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.567+0000 7f952ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f95280ad6a0 0x7f95280adb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.567+0000 7f952ce00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952815a040 con 0x7f95281532d0 2024-03-23T11:19:03.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.567+0000 7f952ce00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9528159a80 con 0x7f9528152ef0 2024-03-23T11:19:03.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.567+0000 7f6071200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c156970 con 0x7f606c15c9e0 2024-03-23T11:19:03.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.567+0000 7f6071200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c156c50 con 0x7f606c0cf650 2024-03-23T11:19:03.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.567+0000 7f7657400640 1 --2- 172.21.15.62:0/225235271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f765815cdc0 0x7f76580d1f20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f76480097a0 tx=0x7f764802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a1defd83e6535a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.567+0000 7f6071200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606c156f30 con 0x7f606c15cdc0 2024-03-23T11:19:03.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.568+0000 7f9525400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f95281532d0 0x7f9528159050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53458/0 (socket says 172.21.15.62:53458) 2024-03-23T11:19:03.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.568+0000 7f9525400640 1 -- 172.21.15.62:0/2501579475 learned_addr learned my addr 172.21.15.62:0/2501579475 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:03.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.568+0000 7f6069400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f606c15c9e0 0x7f606c0cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.568+0000 7f6069e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f606c0cf650 0x7f606c1f1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.568+0000 7f952ce00640 1 -- 172.21.15.62:0/2501579475 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95280ae070 con 0x7f95280ad6a0 2024-03-23T11:19:03.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.568+0000 7efebf600640 1 Processor -- start 2024-03-23T11:19:03.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.569+0000 7f3269c00640 1 Processor -- start 2024-03-23T11:19:03.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.569+0000 7efebf600640 1 -- start start 2024-03-23T11:19:03.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.569+0000 7f7656a00640 1 -- 172.21.15.62:0/225235271 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764802d040 con 0x7f765815cdc0 2024-03-23T11:19:03.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.569+0000 7f7656a00640 1 -- 172.21.15.62:0/225235271 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7648031810 con 0x7f765815cdc0 2024-03-23T11:19:03.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.569+0000 7f7656a00640 1 -- 172.21.15.62:0/225235271 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648031ba0 con 0x7f765815cdc0 2024-03-23T11:19:03.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.569+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912483936 0 0) 0x7f9528159a80 con 0x7f9528152ef0 2024-03-23T11:19:03.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.569+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f950c003440 con 0x7f9528152ef0 2024-03-23T11:19:03.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.569+0000 7f9526800640 1 --2- 172.21.15.62:0/2501579475 >> v2:172.21.15.62:3300/0 conn(0x7f95280ad6a0 0x7f95280adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.569+0000 7f3269c00640 1 -- start start 2024-03-23T11:19:03.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612120687 0 0) 0x7f952815a040 con 0x7f95281532d0 2024-03-23T11:19:03.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9528159a80 con 0x7f95281532d0 2024-03-23T11:19:03.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3781631590 0 0) 0x7f950c003440 con 0x7f9528152ef0 2024-03-23T11:19:03.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 >> v2:172.21.15.62:3300/0 conn(0x7f95280ad6a0 msgr2=0x7f95280adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 --2- 172.21.15.62:0/2501579475 >> v2:172.21.15.62:3300/0 conn(0x7f95280ad6a0 0x7f95280adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.568+0000 7f606a800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f606c15cdc0 0x7f606c160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.568+0000 7f6069400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f606c15c9e0 0x7f606c0cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37790/0 (socket says 172.21.15.62:37790) 2024-03-23T11:19:03.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.568+0000 7f6069400640 1 -- 172.21.15.62:0/470724860 learned_addr learned my addr 172.21.15.62:0/470724860 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 >> v1:172.21.15.62:6789/0 conn(0x7f95281532d0 legacy=0x7f9528159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9528159d60 con 0x7f9528152ef0 2024-03-23T11:19:03.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9514001670 con 0x7f9528152ef0 2024-03-23T11:19:03.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 >> v1:172.21.15.120:6789/0 conn(0x7f9528152ef0 legacy=0x7f952815cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f6069e00640 1 -- 172.21.15.62:0/470724860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f606c15cdc0 msgr2=0x7f606c160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f6069e00640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f606c15cdc0 0x7f606c160730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 --2- 172.21.15.62:0/2501579475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f950c003bf0 0x7f950c003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 --2- 172.21.15.62:0/2501579475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f950c00ac50 0x7f950c00d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 --2- 172.21.15.62:0/2501579475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f950c00d7c0 0x7f950c00fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95280ae070 con 0x7f950c003bf0 2024-03-23T11:19:03.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c010560 con 0x7f950c00ac50 2024-03-23T11:19:03.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f6069e00640 1 -- 172.21.15.62:0/470724860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f606c15c9e0 msgr2=0x7f606c0cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c010720 con 0x7f950c00d7c0 2024-03-23T11:19:03.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f765ea00640 1 -- 172.21.15.62:0/225235271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f765815cdc0 msgr2=0x7f76580d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f765ea00640 1 --2- 172.21.15.62:0/225235271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f765815cdc0 0x7f76580d1f20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f76480097a0 tx=0x7f764802b8b0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f765ea00640 1 -- 172.21.15.62:0/225235271 shutdown_connections 2024-03-23T11:19:03.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f765ea00640 1 --2- 172.21.15.62:0/225235271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f765815cdc0 0x7f76580d1f20 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f76480097a0 tx=0x7f764802b8b0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f765ea00640 1 --2- 172.21.15.62:0/225235271 >> v2:172.21.15.120:3300/0 conn(0x7f765815c9e0 0x7f76581607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f765ea00640 1 -- 172.21.15.62:0/225235271 >> 172.21.15.62:0/225235271 conn(0x7f7658071570 msgr2=0x7f7658071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f6069e00640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f606c15c9e0 0x7f606c0cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f765ea00640 1 -- 172.21.15.62:0/225235271 shutdown_connections 2024-03-23T11:19:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f6069e00640 1 -- 172.21.15.62:0/470724860 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064009a70 con 0x7f606c0cf650 2024-03-23T11:19:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.570+0000 7f9526800640 1 --2- 172.21.15.62:0/2501579475 >> v2:172.21.15.62:3300/0 conn(0x7f95280ad6a0 0x7f95280adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f9525e00640 1 --2- 172.21.15.62:0/2501579475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f950c003bf0 0x7f950c003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f9525400640 1 --2- 172.21.15.62:0/2501579475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f950c00ac50 0x7f950c00d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f9525e00640 1 -- 172.21.15.62:0/2501579475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f950c00d7c0 msgr2=0x7f950c00fc80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T11:19:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f9525e00640 1 --2- 172.21.15.62:0/2501579475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f950c00d7c0 0x7f950c00fc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f9525e00640 1 -- 172.21.15.62:0/2501579475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f950c00ac50 msgr2=0x7f950c00d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f9525e00640 1 --2- 172.21.15.62:0/2501579475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f950c00ac50 0x7f950c00d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f9525e00640 1 -- 172.21.15.62:0/2501579475 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9528159d60 con 0x7f950c003bf0 2024-03-23T11:19:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f9526800640 1 -- 172.21.15.62:0/2501579475 reap_dead start 2024-03-23T11:19:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f6069e00640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f606c0cf650 0x7f606c1f1cd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6064033000 tx=0x7f6064004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f3269c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32640b15e0 0x7f3264161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f3269c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f326406c300 0x7f326406c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f3269c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32640ae0c0 con 0x7f32640b2f90 2024-03-23T11:19:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f3269c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32640b2740 con 0x7f326406c300 2024-03-23T11:19:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.571+0000 7f3269c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32641623b0 con 0x7f32640b15e0 2024-03-23T11:19:03.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.572+0000 7f52e6600640 1 Processor -- start 2024-03-23T11:19:03.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.572+0000 7f3262a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f32640b2f90 0x7f32640b3370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53466/0 (socket says 172.21.15.62:53466) 2024-03-23T11:19:03.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.572+0000 7f325a000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32640b15e0 0x7f3264161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.572+0000 7f3262a00640 1 -- 172.21.15.62:0/568176023 learned_addr learned my addr 172.21.15.62:0/568176023 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:03.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.572+0000 7f6062000640 1 -- 172.21.15.62:0/470724860 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606403e610 con 0x7f606c0cf650 2024-03-23T11:19:03.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.572+0000 7f52e6600640 1 -- start start 2024-03-23T11:19:03.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.572+0000 7f3263400640 1 --2- 172.21.15.62:0/568176023 >> v2:172.21.15.120:3300/0 conn(0x7f326406c300 0x7f326406c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.572+0000 7f6062000640 1 -- 172.21.15.62:0/470724860 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606403e770 con 0x7f606c0cf650 2024-03-23T11:19:03.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.572+0000 7f6062000640 1 -- 172.21.15.62:0/470724860 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606403eb20 con 0x7f606c0cf650 2024-03-23T11:19:03.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7f765ea00640 1 -- 172.21.15.62:0/225235271 wait complete. 2024-03-23T11:19:03.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7f6071200640 1 -- 172.21.15.62:0/470724860 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c157210 con 0x7f606c0cf650 2024-03-23T11:19:03.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7f6071200640 1 -- 172.21.15.62:0/470724860 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c1f2210 con 0x7f606c0cf650 2024-03-23T11:19:03.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7f3263400640 1 -- 172.21.15.62:0/568176023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32640b15e0 msgr2=0x7f3264161e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7f3263400640 1 --2- 172.21.15.62:0/568176023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32640b15e0 0x7f3264161e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7f3262000640 1 -- 172.21.15.62:0/568176023 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798085007 0 0) 0x7f32640ae0c0 con 0x7f32640b2f90 2024-03-23T11:19:03.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7f3263400640 1 -- 172.21.15.62:0/568176023 >> v1:172.21.15.62:6789/0 conn(0x7f32640b2f90 legacy=0x7f32640b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7f3263400640 1 -- 172.21.15.62:0/568176023 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32640b1100 con 0x7f326406c300 2024-03-23T11:19:03.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7f52e6600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f52e015c900 0x7f52e00d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7f52e6600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e00d1100 con 0x7f52e00cd690 2024-03-23T11:19:03.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7f52e6600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e00d13e0 con 0x7f52e015cce0 2024-03-23T11:19:03.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7f52e6600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e00b89f0 con 0x7f52e015c900 2024-03-23T11:19:03.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7f9525e00640 1 --2- 172.21.15.62:0/2501579475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f950c003bf0 0x7f950c003ff0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9514001eb0 tx=0x7f950c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e6dd77fa12be81 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7f52df400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f52e015c900 0x7f52e00d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7f52df400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f52e015c900 0x7f52e00d21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53076/0 (socket says 172.21.15.62:53076) 2024-03-23T11:19:03.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7f52df400640 1 -- 172.21.15.62:0/3732660250 learned_addr learned my addr 172.21.15.62:0/3732660250 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7efebf600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efeb815c9e0 0x7efeb81607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7efebf600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efeb80cf650 0x7efeb80cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7efebf600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb806bfc0 con 0x7efeb815cdc0 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7efebf600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb80d0000 con 0x7efeb80cf650 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7efebf600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb80d2630 con 0x7efeb815c9e0 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7efebd600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efeb80cf650 0x7efeb80cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7efebd600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efeb80cf650 0x7efeb80cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37800/0 (socket says 172.21.15.62:37800) 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.573+0000 7efebd600640 1 -- 172.21.15.62:0/3810365813 learned_addr learned my addr 172.21.15.62:0/3810365813 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7efebcc00640 1 --2- 172.21.15.62:0/3810365813 >> v2:172.21.15.120:3300/0 conn(0x7efeb815c9e0 0x7efeb81607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7efebd600640 1 -- 172.21.15.62:0/3810365813 >> v2:172.21.15.120:3300/0 conn(0x7efeb815c9e0 msgr2=0x7efeb81607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7efebd600640 1 --2- 172.21.15.62:0/3810365813 >> v2:172.21.15.120:3300/0 conn(0x7efeb815c9e0 0x7efeb81607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7efebd600640 1 -- 172.21.15.62:0/3810365813 >> v1:172.21.15.120:6789/0 conn(0x7efeb815cdc0 legacy=0x7efeb80d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.574+0000 7efebd600640 1 -- 172.21.15.62:0/3810365813 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb80b5260 con 0x7efeb80cf650 2024-03-23T11:19:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.575+0000 7f765ea00640 1 Processor -- start 2024-03-23T11:19:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.576+0000 7f765ea00640 1 -- start start 2024-03-23T11:19:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.576+0000 7f6071200640 1 -- 172.21.15.62:0/470724860 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6044000fc0 con 0x7f606c0cf650 2024-03-23T11:19:03.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.575+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951402c980 con 0x7f950c003bf0 2024-03-23T11:19:03.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.575+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951402cae0 con 0x7f950c003bf0 2024-03-23T11:19:03.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.575+0000 7f9524a00640 1 -- 172.21.15.62:0/2501579475 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9514034440 con 0x7f950c003bf0 2024-03-23T11:19:03.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.576+0000 7f952ce00640 1 -- 172.21.15.62:0/2501579475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f950c003bf0 msgr2=0x7f950c003ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.576+0000 7f952ce00640 1 --2- 172.21.15.62:0/2501579475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f950c003bf0 0x7f950c003ff0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9514001eb0 tx=0x7f950c0036f0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.576+0000 7f952ce00640 1 -- 172.21.15.62:0/2501579475 shutdown_connections 2024-03-23T11:19:03.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.576+0000 7f952ce00640 1 --2- 172.21.15.62:0/2501579475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f950c003bf0 0x7f950c003ff0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.576+0000 7f952ce00640 1 -- 172.21.15.62:0/2501579475 >> 172.21.15.62:0/2501579475 conn(0x7f9528069c60 msgr2=0x7f952806a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.575+0000 7f52df400640 1 -- 172.21.15.62:0/3732660250 >> v1:172.21.15.120:6789/0 conn(0x7f52e015cce0 legacy=0x7f52e00b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.575+0000 7f52df400640 1 -- 172.21.15.62:0/3732660250 >> v1:172.21.15.62:6789/0 conn(0x7f52e00cd690 legacy=0x7f52e00b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.575+0000 7f52df400640 1 -- 172.21.15.62:0/3732660250 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e00d16c0 con 0x7f52e015c900 2024-03-23T11:19:03.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.576+0000 7efebd600640 1 --2- 172.21.15.62:0/3810365813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb80cf650 0x7efeb80cfac0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efeb80d0380 tx=0x7efeb003cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fbce33c74e35fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.577+0000 7f952ce00640 1 -- 172.21.15.62:0/2501579475 shutdown_connections 2024-03-23T11:19:03.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.577+0000 7efeb7400640 1 -- 172.21.15.62:0/3810365813 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb003da90 con 0x7efeb80cf650 2024-03-23T11:19:03.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.577+0000 7efeb7400640 1 -- 172.21.15.62:0/3810365813 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb003dbf0 con 0x7efeb80cf650 2024-03-23T11:19:03.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.577+0000 7efeb7400640 1 -- 172.21.15.62:0/3810365813 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb00453f0 con 0x7efeb80cf650 2024-03-23T11:19:03.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.577+0000 7f52df400640 1 --2- 172.21.15.62:0/3732660250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e015c900 0x7f52e00d21a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009e50 tx=0x7f52d802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e74a05dcbc69469e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.577+0000 7f765ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76580cf650 0x7f76580acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.577+0000 7f765ea00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f765815c9e0 0x7f76580ad100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.577+0000 7f952ce00640 1 -- 172.21.15.62:0/2501579475 wait complete. 2024-03-23T11:19:03.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.578+0000 7f3263400640 1 --2- 172.21.15.62:0/568176023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f326406c300 0x7f326406c790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f325c009d90 tx=0x7f325c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39c1f49272386864 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.577+0000 7f765ea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76580aa1a0 0x7f765814cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.578+0000 7f765ea00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765815ff60 con 0x7f76580cf650 2024-03-23T11:19:03.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.578+0000 7f765ea00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765814aae0 con 0x7f765815c9e0 2024-03-23T11:19:03.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.578+0000 7f765ea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765814aed0 con 0x7f76580aa1a0 2024-03-23T11:19:03.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.578+0000 7f7657e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76580cf650 0x7f76580acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.578+0000 7f52de000640 1 -- 172.21.15.62:0/3732660250 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d802cb20 con 0x7f52e015c900 2024-03-23T11:19:03.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.578+0000 7efebf600640 1 -- 172.21.15.62:0/3810365813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb80cf650 msgr2=0x7efeb80cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.578+0000 7efebf600640 1 --2- 172.21.15.62:0/3810365813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb80cf650 0x7efeb80cfac0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efeb80d0380 tx=0x7efeb003cef0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.578+0000 7f765ca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76580aa1a0 0x7f765814cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.578+0000 7f765ca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76580aa1a0 0x7f765814cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:55912/0 (socket says 172.21.15.62:55912) 2024-03-23T11:19:03.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.578+0000 7f765ca00640 1 -- 172.21.15.62:0/2728551428 learned_addr learned my addr 172.21.15.62:0/2728551428 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7efebf600640 1 -- 172.21.15.62:0/3810365813 shutdown_connections 2024-03-23T11:19:03.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7efebf600640 1 --2- 172.21.15.62:0/3810365813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb80cf650 0x7efeb80cfac0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7efebf600640 1 --2- 172.21.15.62:0/3810365813 >> v2:172.21.15.120:3300/0 conn(0x7efeb815c9e0 0x7efeb81607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7efebf600640 1 -- 172.21.15.62:0/3810365813 >> 172.21.15.62:0/3810365813 conn(0x7efeb8071570 msgr2=0x7efeb8071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7efebf600640 1 -- 172.21.15.62:0/3810365813 shutdown_connections 2024-03-23T11:19:03.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7efebf600640 1 -- 172.21.15.62:0/3810365813 wait complete. 2024-03-23T11:19:03.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7f7657400640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f765815c9e0 0x7f76580ad100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7f52de000640 1 -- 172.21.15.62:0/3732660250 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d802cc80 con 0x7f52e015c900 2024-03-23T11:19:03.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7f52de000640 1 -- 172.21.15.62:0/3732660250 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d80343f0 con 0x7f52e015c900 2024-03-23T11:19:03.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7f6062000640 1 -- 172.21.15.62:0/470724860 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f606403b050 con 0x7f606c0cf650 2024-03-23T11:19:03.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.580+0000 7f952ce00640 1 Processor -- start 2024-03-23T11:19:03.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7f7657e00640 1 -- 172.21.15.62:0/2728551428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76580aa1a0 msgr2=0x7f765814cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7f7657e00640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76580aa1a0 0x7f765814cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7f7657e00640 1 -- 172.21.15.62:0/2728551428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f765815c9e0 msgr2=0x7f76580ad100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7f7657e00640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f765815c9e0 0x7f76580ad100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.579+0000 7f7657e00640 1 -- 172.21.15.62:0/2728551428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7648009400 con 0x7f76580cf650 2024-03-23T11:19:03.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.580+0000 7f952ce00640 1 -- start start 2024-03-23T11:19:03.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.580+0000 7f7657e00640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76580cf650 0x7f76580acbc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f764400d5f0 tx=0x7f764400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.580+0000 7f3262000640 1 -- 172.21.15.62:0/568176023 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c02cc50 con 0x7f326406c300 2024-03-23T11:19:03.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.580+0000 7f3262000640 1 -- 172.21.15.62:0/568176023 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325c02cdb0 con 0x7f326406c300 2024-03-23T11:19:03.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.580+0000 7f3262000640 1 -- 172.21.15.62:0/568176023 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c034660 con 0x7f326406c300 2024-03-23T11:19:03.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.580+0000 7ff322200640 1 Processor -- start 2024-03-23T11:19:03.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.580+0000 7f52e6600640 1 -- 172.21.15.62:0/3732660250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e015c900 msgr2=0x7f52e00d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.580+0000 7ff322200640 1 -- start start 2024-03-23T11:19:03.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.581+0000 7f764fe00640 1 -- 172.21.15.62:0/2728551428 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644004030 con 0x7f76580cf650 2024-03-23T11:19:03.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.581+0000 7f764fe00640 1 -- 172.21.15.62:0/2728551428 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7644004190 con 0x7f76580cf650 2024-03-23T11:19:03.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.581+0000 7f764fe00640 1 -- 172.21.15.62:0/2728551428 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644014620 con 0x7f76580cf650 2024-03-23T11:19:03.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.581+0000 7f6062000640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f604c06a7a0 0x7f604c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.581+0000 7f952ce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f950c003bf0 0x7f9528150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.581+0000 7f765ea00640 1 -- 172.21.15.62:0/2728551428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765814b1b0 con 0x7f76580cf650 2024-03-23T11:19:03.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.581+0000 7f765ea00640 1 -- 172.21.15.62:0/2728551428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765814b770 con 0x7f76580cf650 2024-03-23T11:19:03.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.581+0000 7efebf600640 1 Processor -- start 2024-03-23T11:19:03.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7f52e6600640 1 --2- 172.21.15.62:0/3732660250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e015c900 0x7f52e00d21a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009e50 tx=0x7f52d802bf80 comp rx=0 tx=0).stop 2024-03-23T11:19:03.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.581+0000 7f952ce00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9528155550 0x7f952814aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.581+0000 7f952ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f952814b010 0x7f9528153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.581+0000 7f6062000640 1 -- 172.21.15.62:0/470724860 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60640abbc0 con 0x7f606c0cf650 2024-03-23T11:19:03.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7f6062000640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x7f604c072940 0x7f604c074d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7f6062000640 1 -- 172.21.15.62:0/470724860 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f604c075410 con 0x7f604c072940 2024-03-23T11:19:03.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7f6062000640 1 -- 172.21.15.62:0/470724860 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60640ac010 con 0x7f606c0cf650 2024-03-23T11:19:03.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7f6069400640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f604c06a7a0 0x7f604c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7efebf600640 1 -- start start 2024-03-23T11:19:03.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7f52e6600640 1 -- 172.21.15.62:0/3732660250 shutdown_connections 2024-03-23T11:19:03.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7f52e6600640 1 --2- 172.21.15.62:0/3732660250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e015c900 0x7f52e00d21a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7f52e6600640 1 -- 172.21.15.62:0/3732660250 >> 172.21.15.62:0/3732660250 conn(0x7f52e0071390 msgr2=0x7f52e0071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7f606a800640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x7f604c072940 0x7f604c074d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7f3269c00640 1 -- 172.21.15.62:0/568176023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f326406c300 msgr2=0x7f326406c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7f3269c00640 1 --2- 172.21.15.62:0/568176023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f326406c300 0x7f326406c790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f325c009d90 tx=0x7f325c02bea0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7ff322200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff31c161e70 0x7ff31c06ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7ff322200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c0ae0c0 con 0x7ff31c0b2f90 2024-03-23T11:19:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7ff322200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c0b2740 con 0x7ff31c0b15e0 2024-03-23T11:19:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.582+0000 7ff322200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31c0b1100 con 0x7ff31c161e70 2024-03-23T11:19:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.583+0000 7f9525e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f950c003bf0 0x7f9528150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.583+0000 7f9525e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f950c003bf0 0x7f9528150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:55924/0 (socket says 172.21.15.62:55924) 2024-03-23T11:19:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.583+0000 7f952ce00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952814b4c0 con 0x7f952814b010 2024-03-23T11:19:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.583+0000 7f952ce00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95281544b0 con 0x7f9528155550 2024-03-23T11:19:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.583+0000 7f952ce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9528154790 con 0x7f950c003bf0 2024-03-23T11:19:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.583+0000 7ff31aa00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ff31c0b15e0 0x7ff31c06c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41636/0 (socket says 172.21.15.62:41636) 2024-03-23T11:19:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.583+0000 7ff31aa00640 1 -- 172.21.15.62:0/1881953245 learned_addr learned my addr 172.21.15.62:0/1881953245 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.583+0000 7ff31be00640 1 --2- 172.21.15.62:0/1881953245 >> v2:172.21.15.62:3300/0 conn(0x7ff31c161e70 0x7ff31c06ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7f3269c00640 1 -- 172.21.15.62:0/568176023 shutdown_connections 2024-03-23T11:19:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.583+0000 7f9525400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9528155550 0x7f952814aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7f3269c00640 1 --2- 172.21.15.62:0/568176023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f326406c300 0x7f326406c790 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785991930 0 0) 0x7ff31c0b2740 con 0x7ff31c0b15e0 2024-03-23T11:19:03.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff300003520 con 0x7ff31c0b15e0 2024-03-23T11:19:03.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7f3269c00640 1 --2- 172.21.15.62:0/568176023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32640b15e0 0x7f3264161e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7efebf600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeb80cf650 0x7efeb80c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7f6069400640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f604c06a7a0 0x7f604c06cc60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f606c0b3d90 tx=0x7f605c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7f606a800640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x7f604c072940 0x7f604c074d60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7f9525400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9528155550 0x7f952814aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53092/0 (socket says 172.21.15.62:53092) 2024-03-23T11:19:03.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7efebf600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb815c9e0 0x7efeb815e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7f9525400640 1 -- 172.21.15.62:0/4111614750 learned_addr learned my addr 172.21.15.62:0/4111614750 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7f9526800640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f952814b010 0x7f9528153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585382263 0 0) 0x7ff31c0ae0c0 con 0x7ff31c0b2f90 2024-03-23T11:19:03.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff31c0b2740 con 0x7ff31c0b2f90 2024-03-23T11:19:03.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7f3269c00640 1 -- 172.21.15.62:0/568176023 >> 172.21.15.62:0/568176023 conn(0x7f32640b02e0 msgr2=0x7f32640b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7f9525400640 1 -- 172.21.15.62:0/4111614750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f950c003bf0 msgr2=0x7f9528150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7f9525400640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f950c003bf0 0x7f9528150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7f9525400640 1 -- 172.21.15.62:0/4111614750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f952814b010 msgr2=0x7f9528153780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7efebf600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb815cdc0 0x7efeb815ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7f52e6600640 1 -- 172.21.15.62:0/3732660250 shutdown_connections 2024-03-23T11:19:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2871381650 0 0) 0x7ff300003520 con 0x7ff31c0b15e0 2024-03-23T11:19:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7f3269c00640 1 -- 172.21.15.62:0/568176023 shutdown_connections 2024-03-23T11:19:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7efebcc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeb80cf650 0x7efeb80c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7efebf600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb80c9af0 con 0x7efeb815c9e0 2024-03-23T11:19:03.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7f52e6600640 1 -- 172.21.15.62:0/3732660250 wait complete. 2024-03-23T11:19:03.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7efebcc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeb80cf650 0x7efeb80c8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:55932/0 (socket says 172.21.15.62:55932) 2024-03-23T11:19:03.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7efebcc00640 1 -- 172.21.15.62:0/3748520465 learned_addr learned my addr 172.21.15.62:0/3748520465 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb806d2a0 con 0x7efeb815cdc0 2024-03-23T11:19:03.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb8070470 con 0x7efeb80cf650 2024-03-23T11:19:03.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.584+0000 7f765ea00640 1 -- 172.21.15.62:0/2728551428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f76580b4c10 con 0x7f76580cf650 2024-03-23T11:19:03.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7f3269c00640 1 -- 172.21.15.62:0/568176023 wait complete. 2024-03-23T11:19:03.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.585+0000 7efebd600640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb815cdc0 0x7efeb815ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 >> v2:172.21.15.62:3300/0 conn(0x7ff31c161e70 msgr2=0x7ff31c06ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 --2- 172.21.15.62:0/1881953245 >> v2:172.21.15.62:3300/0 conn(0x7ff31c161e70 0x7ff31c06ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 >> v1:172.21.15.62:6789/0 conn(0x7ff31c0b2f90 legacy=0x7ff31c0b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c0b1b30 con 0x7ff31c0b15e0 2024-03-23T11:19:03.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7ff308001670 con 0x7ff31c0b15e0 2024-03-23T11:19:03.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 >> v1:172.21.15.120:6789/0 conn(0x7ff31c0b15e0 legacy=0x7ff31c06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7f6062000640 1 -- 172.21.15.62:0/470724860 <== osd.2 v2:172.21.15.62:6800/36996 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f604c075410 con 0x7f604c072940 2024-03-23T11:19:03.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7efeb7e00640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb815c9e0 0x7efeb815e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 --2- 172.21.15.62:0/1881953245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300003b00 0x7ff300003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 --2- 172.21.15.62:0/1881953245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3000047c0 0x7ff300004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 --2- 172.21.15.62:0/1881953245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff300005180 0x7ff30000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c001230 con 0x7ff300005180 2024-03-23T11:19:03.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c0b1100 con 0x7ff300003b00 2024-03-23T11:19:03.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30000c020 con 0x7ff3000047c0 2024-03-23T11:19:03.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7f764fe00640 1 -- 172.21.15.62:0/2728551428 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7644014780 con 0x7f76580cf650 2024-03-23T11:19:03.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7ff31be00640 1 --2- 172.21.15.62:0/1881953245 >> v2:172.21.15.62:3300/0 conn(0x7ff31c161e70 0x7ff31c06ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:03.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7f9525400640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f952814b010 0x7f9528153780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.587+0000 7f9525400640 1 -- 172.21.15.62:0/4111614750 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9528154a70 con 0x7f9528155550 2024-03-23T11:19:03.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7efebd600640 1 -- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeb80cf650 msgr2=0x7efeb80c8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7efebd600640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeb80cf650 0x7efeb80c8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7efebd600640 1 -- 172.21.15.62:0/3748520465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb815c9e0 msgr2=0x7efeb815e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7efebd600640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb815c9e0 0x7efeb815e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.586+0000 7efebd600640 1 -- 172.21.15.62:0/3748520465 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0009b30 con 0x7efeb815cdc0 2024-03-23T11:19:03.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.587+0000 7ff31b400640 1 --2- 172.21.15.62:0/1881953245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300003b00 0x7ff300003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.587+0000 7efebd600640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb815cdc0 0x7efeb815ec50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efeb0043000 tx=0x7efeb0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.587+0000 7ff31aa00640 1 --2- 172.21.15.62:0/1881953245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3000047c0 0x7ff300004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.587+0000 7f9526800640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f952814b010 0x7f9528153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:03.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7ff31be00640 1 --2- 172.21.15.62:0/1881953245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff300005180 0x7ff30000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7f3269c00640 1 Processor -- start 2024-03-23T11:19:03.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7efeb4c00640 1 -- 172.21.15.62:0/3748520465 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0009510 con 0x7efeb815cdc0 2024-03-23T11:19:03.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7f764fe00640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f763806a870 0x7f763806cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7f764fe00640 1 -- 172.21.15.62:0/2728551428 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76440893c0 con 0x7f76580cf650 2024-03-23T11:19:03.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7f764fe00640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x7f7638072a90 0x7f7638074eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7f764fe00640 1 -- 172.21.15.62:0/2728551428 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7638075560 con 0x7f7638072a90 2024-03-23T11:19:03.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7f764fe00640 1 -- 172.21.15.62:0/2728551428 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f764408b9e0 con 0x7f76580cf650 2024-03-23T11:19:03.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7efeb4c00640 1 -- 172.21.15.62:0/3748520465 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb004f450 con 0x7efeb815cdc0 2024-03-23T11:19:03.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7efeb4c00640 1 -- 172.21.15.62:0/3748520465 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb004f800 con 0x7efeb815cdc0 2024-03-23T11:19:03.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb8070750 con 0x7efeb815cdc0 2024-03-23T11:19:03.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7f3269c00640 1 -- start start 2024-03-23T11:19:03.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb8070d10 con 0x7efeb815cdc0 2024-03-23T11:19:03.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7f9525400640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9528155550 0x7f952814aad0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f951800d540 tx=0x7f951800da00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7ff31b400640 1 -- 172.21.15.62:0/1881953245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3000047c0 msgr2=0x7ff300004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7ff31b400640 1 --2- 172.21.15.62:0/1881953245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3000047c0 0x7ff300004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7ff31b400640 1 -- 172.21.15.62:0/1881953245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff300005180 msgr2=0x7ff30000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7ff31b400640 1 --2- 172.21.15.62:0/1881953245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff300005180 0x7ff30000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.588+0000 7ff31b400640 1 -- 172.21.15.62:0/1881953245 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c0b1b30 con 0x7ff300003b00 2024-03-23T11:19:03.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.589+0000 7ff31be00640 1 -- 172.21.15.62:0/1881953245 reap_dead start 2024-03-23T11:19:03.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.589+0000 7f86ff600640 1 Processor -- start 2024-03-23T11:19:03.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.589+0000 7f52e6600640 1 Processor -- start 2024-03-23T11:19:03.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.589+0000 7f7657400640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f763806a870 0x7f763806cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.589+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efe94000fc0 con 0x7efeb815cdc0 2024-03-23T11:19:03.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f86ff600640 1 -- start start 2024-03-23T11:19:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f3269c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f326406c300 0x7f3264148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f3269c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32640b15e0 0x7f32640a9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f3269c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32640b2f90 0x7f32640aa410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f3269c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3264158e60 con 0x7f32640b15e0 2024-03-23T11:19:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f3269c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32640a75c0 con 0x7f326406c300 2024-03-23T11:19:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f3269c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3264148b80 con 0x7f32640b2f90 2024-03-23T11:19:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f951e000640 1 -- 172.21.15.62:0/4111614750 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9518005ce0 con 0x7f9528155550 2024-03-23T11:19:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f951e000640 1 -- 172.21.15.62:0/4111614750 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9518005e40 con 0x7f9528155550 2024-03-23T11:19:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f951e000640 1 -- 172.21.15.62:0/4111614750 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9518007270 con 0x7f9528155550 2024-03-23T11:19:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f952ce00640 1 -- 172.21.15.62:0/4111614750 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9528154d50 con 0x7f9528155550 2024-03-23T11:19:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.589+0000 7f765ca00640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x7f7638072a90 0x7f7638074eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f952ce00640 1 -- 172.21.15.62:0/4111614750 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9528150540 con 0x7f9528155550 2024-03-23T11:19:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f325a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32640b15e0 0x7f32640a9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f325a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32640b15e0 0x7f32640a9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37830/0 (socket says 172.21.15.62:37830) 2024-03-23T11:19:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f325a000640 1 -- 172.21.15.62:0/2505688845 learned_addr learned my addr 172.21.15.62:0/2505688845 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.590+0000 7f52e6600640 1 -- start start 2024-03-23T11:19:03.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.591+0000 7ff31b400640 1 --2- 172.21.15.62:0/1881953245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300003b00 0x7ff300003ee0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff30c007ec0 tx=0x7ff3000057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cec475a7cd025f25 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.591+0000 7f3263400640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32640b2f90 0x7f32640aa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.591+0000 7f3262a00640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f326406c300 0x7f3264148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.591+0000 7f325a000640 1 -- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32640b2f90 msgr2=0x7f32640aa410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.591+0000 7f325a000640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32640b2f90 0x7f32640aa410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.591+0000 7f325a000640 1 -- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f326406c300 msgr2=0x7f3264148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.591+0000 7f325a000640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f326406c300 0x7f3264148640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.591+0000 7f325a000640 1 -- 172.21.15.62:0/2505688845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c009a70 con 0x7f32640b15e0 2024-03-23T11:19:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.591+0000 7f3262a00640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f326406c300 0x7f3264148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.591+0000 7f86ff600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f86f815c9e0 0x7f86f81607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f86ff600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f86f815cdc0 0x7f86f80d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f86ff600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f80cf650 0x7f86f80cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f86ff600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f80d0020 con 0x7f86f815cdc0 2024-03-23T11:19:03.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f86ff600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f806bfc0 con 0x7f86f815c9e0 2024-03-23T11:19:03.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f86ff600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f80b5260 con 0x7f86f80cf650 2024-03-23T11:19:03.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.591+0000 7f765ca00640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x7f7638072a90 0x7f7638074eb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f325a000640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32640b15e0 0x7f32640a9ed0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f324c00e9e0 tx=0x7f324c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f86fd600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f80cf650 0x7f86f80cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f86fd600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f80cf650 0x7f86f80cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:55962/0 (socket says 172.21.15.62:55962) 2024-03-23T11:19:03.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f86fd600640 1 -- 172.21.15.62:0/1209195217 learned_addr learned my addr 172.21.15.62:0/1209195217 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f952ce00640 1 -- 172.21.15.62:0/4111614750 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f952815ca50 con 0x7f9528155550 2024-03-23T11:19:03.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f764fe00640 1 -- 172.21.15.62:0/2728551428 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7638075560 con 0x7f7638072a90 2024-03-23T11:19:03.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f52e6600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e00cd690 0x7f52e00a9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f52e6600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e015c900 0x7f52e01f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f52e6600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e015cce0 0x7f52e01f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f52e6600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00ca7b0 con 0x7f52e015cce0 2024-03-23T11:19:03.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f52e6600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00aa2b0 con 0x7f52e00cd690 2024-03-23T11:19:03.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7f52e6600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00ca9c0 con 0x7f52e015c900 2024-03-23T11:19:03.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.592+0000 7efeb4c00640 1 -- 172.21.15.62:0/3748520465 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7efeb004f960 con 0x7efeb815cdc0 2024-03-23T11:19:03.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.593+0000 7f7657400640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f763806a870 0x7f763806cd30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7648036000 tx=0x7f764803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.593+0000 7f86f7e00640 1 --2- 172.21.15.62:0/1209195217 >> v2:172.21.15.62:3300/0 conn(0x7f86f815cdc0 0x7f86f80d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.593+0000 7f86fd600640 1 -- 172.21.15.62:0/1209195217 >> v2:172.21.15.120:3300/0 conn(0x7f86f815c9e0 msgr2=0x7f86f81607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.593+0000 7f86fd600640 1 --2- 172.21.15.62:0/1209195217 >> v2:172.21.15.120:3300/0 conn(0x7f86f815c9e0 0x7f86f81607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7f325b400640 1 -- 172.21.15.62:0/2505688845 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c009670 con 0x7f32640b15e0 2024-03-23T11:19:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7f325b400640 1 -- 172.21.15.62:0/2505688845 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324c00c9d0 con 0x7f32640b15e0 2024-03-23T11:19:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7f325b400640 1 -- 172.21.15.62:0/2505688845 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c00cd80 con 0x7f32640b15e0 2024-03-23T11:19:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c005270 con 0x7ff300003b00 2024-03-23T11:19:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30c0053d0 con 0x7ff300003b00 2024-03-23T11:19:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7ff31a000640 1 -- 172.21.15.62:0/1881953245 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c02c830 con 0x7ff300003b00 2024-03-23T11:19:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7efeb4c00640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe9806a7a0 0x7efe9806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7efeb4c00640 1 -- 172.21.15.62:0/3748520465 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efeb00bd100 con 0x7efeb815cdc0 2024-03-23T11:19:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7efeb4c00640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x7efe98072940 0x7efe98074d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7efeb4c00640 1 -- 172.21.15.62:0/3748520465 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efe98075410 con 0x7efe98072940 2024-03-23T11:19:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7efeb4c00640 1 -- 172.21.15.62:0/3748520465 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7efeb00bd470 con 0x7efeb815cdc0 2024-03-23T11:19:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.595+0000 7f86fd600640 1 -- 172.21.15.62:0/1209195217 >> v2:172.21.15.62:3300/0 conn(0x7f86f815cdc0 msgr2=0x7f86f80d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.595+0000 7f86fd600640 1 --2- 172.21.15.62:0/1209195217 >> v2:172.21.15.62:3300/0 conn(0x7f86f815cdc0 0x7f86f80d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.595+0000 7f86fd600640 1 -- 172.21.15.62:0/1209195217 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f80ce140 con 0x7f86f80cf650 2024-03-23T11:19:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.595+0000 7ff322200640 1 -- 172.21.15.62:0/1881953245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300003b00 msgr2=0x7ff300003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.595+0000 7ff322200640 1 --2- 172.21.15.62:0/1881953245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300003b00 0x7ff300003ee0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff30c007ec0 tx=0x7ff3000057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.595+0000 7f951e000640 1 -- 172.21.15.62:0/4111614750 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f95180073d0 con 0x7f9528155550 2024-03-23T11:19:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.595+0000 7f3269c00640 1 -- 172.21.15.62:0/2505688845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326415bf90 con 0x7f32640b15e0 2024-03-23T11:19:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.593+0000 7f52df400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e00cd690 0x7f52e00a9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.593+0000 7f52dfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e015cce0 0x7f52e01f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.593+0000 7f52dfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e015cce0 0x7f52e01f5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37858/0 (socket says 172.21.15.62:37858) 2024-03-23T11:19:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.593+0000 7f52dfe00640 1 -- 172.21.15.62:0/1099003624 learned_addr learned my addr 172.21.15.62:0/1099003624 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.594+0000 7f52dea00640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e015c900 0x7f52e01f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.595+0000 7ff322200640 1 -- 172.21.15.62:0/1881953245 shutdown_connections 2024-03-23T11:19:03.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.595+0000 7f86f7e00640 1 --2- 172.21.15.62:0/1209195217 >> v2:172.21.15.62:3300/0 conn(0x7f86f815cdc0 0x7f86f80d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:03.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.596+0000 7ff322200640 1 --2- 172.21.15.62:0/1881953245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300003b00 0x7ff300003ee0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.596+0000 7ff322200640 1 -- 172.21.15.62:0/1881953245 >> 172.21.15.62:0/1881953245 conn(0x7ff31c0b02e0 msgr2=0x7ff31c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.595+0000 7efeb7e00640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x7efe98072940 0x7efe98074d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.596+0000 7efebcc00640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe9806a7a0 0x7efe9806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.595+0000 7f3269c00640 1 -- 172.21.15.62:0/2505688845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32640a7810 con 0x7f32640b15e0 2024-03-23T11:19:03.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.596+0000 7f52dea00640 1 -- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e00cd690 msgr2=0x7f52e00a9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.596+0000 7f52dea00640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e00cd690 0x7f52e00a9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.596+0000 7f52dea00640 1 -- 172.21.15.62:0/1099003624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e015cce0 msgr2=0x7f52e01f5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.596+0000 7f52dea00640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e015cce0 0x7f52e01f5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.596+0000 7f52dea00640 1 -- 172.21.15.62:0/1099003624 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d8009b30 con 0x7f52e015c900 2024-03-23T11:19:03.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.597+0000 7ff322200640 1 -- 172.21.15.62:0/1881953245 shutdown_connections 2024-03-23T11:19:03.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.596+0000 7f951e000640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f950806a820 0x7f950806cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.596+0000 7f951e000640 1 -- 172.21.15.62:0/4111614750 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9518088de0 con 0x7f9528155550 2024-03-23T11:19:03.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.597+0000 7f951e000640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x7f9508072a40 0x7f9508074e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.597+0000 7f951e000640 1 -- 172.21.15.62:0/4111614750 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9508075510 con 0x7f9508072a40 2024-03-23T11:19:03.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.597+0000 7f951e000640 1 -- 172.21.15.62:0/4111614750 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9518089280 con 0x7f9528155550 2024-03-23T11:19:03.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.597+0000 7f9525e00640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f950806a820 0x7f950806cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.597+0000 7ff322200640 1 -- 172.21.15.62:0/1881953245 wait complete. 2024-03-23T11:19:03.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.596+0000 7f3269c00640 1 -- 172.21.15.62:0/2505688845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f32640b20f0 con 0x7f32640b15e0 2024-03-23T11:19:03.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.597+0000 7f9526800640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x7f9508072a40 0x7f9508074e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.597+0000 7efebcc00640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe9806a7a0 0x7efe9806cc60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efeac006fd0 tx=0x7efeac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.597+0000 7f52dea00640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e015c900 0x7f52e01f58a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f52d000b3e0 tx=0x7f52d000b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.598+0000 7f9526800640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x7f9508072a40 0x7f9508074e60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.598+0000 7f52d7400640 1 -- 172.21.15.62:0/1099003624 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0004020 con 0x7f52e015c900 2024-03-23T11:19:03.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.598+0000 7f52d7400640 1 -- 172.21.15.62:0/1099003624 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d0004180 con 0x7f52e015c900 2024-03-23T11:19:03.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.598+0000 7f52d7400640 1 -- 172.21.15.62:0/1099003624 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0011630 con 0x7f52e015c900 2024-03-23T11:19:03.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.598+0000 7f52e6600640 1 -- 172.21.15.62:0/1099003624 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e00caca0 con 0x7f52e015c900 2024-03-23T11:19:03.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.599+0000 7f52e6600640 1 -- 172.21.15.62:0/1099003624 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e00cb260 con 0x7f52e015c900 2024-03-23T11:19:03.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.599+0000 7f951e000640 1 -- 172.21.15.62:0/4111614750 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f9508075510 con 0x7f9508072a40 2024-03-23T11:19:03.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.599+0000 7f9525e00640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f950806a820 0x7f950806cce0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9514036000 tx=0x7f9514009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.599+0000 7f325b400640 1 -- 172.21.15.62:0/2505688845 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f324c013470 con 0x7f32640b15e0 2024-03-23T11:19:03.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.598+0000 7efeb7e00640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x7efe98072940 0x7efe98074d60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.599+0000 7f86fd600640 1 --2- 172.21.15.62:0/1209195217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f80cf650 0x7f86f80cfae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f86f80d0380 tx=0x7f86f002bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bc1c8b78fbb3fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.600+0000 7f86f7400640 1 -- 172.21.15.62:0/1209195217 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f002ca80 con 0x7f86f80cf650 2024-03-23T11:19:03.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.600+0000 7f86f7400640 1 -- 172.21.15.62:0/1209195217 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f002cbe0 con 0x7f86f80cf650 2024-03-23T11:19:03.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.600+0000 7f86f7400640 1 -- 172.21.15.62:0/1209195217 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f00343f0 con 0x7f86f80cf650 2024-03-23T11:19:03.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7f52e6600640 1 -- 172.21.15.62:0/1099003624 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f52e00d26e0 con 0x7f52e015c900 2024-03-23T11:19:03.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7ff322200640 1 Processor -- start 2024-03-23T11:19:03.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7f86ff600640 1 -- 172.21.15.62:0/1209195217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f80cf650 msgr2=0x7f86f80cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7f86ff600640 1 --2- 172.21.15.62:0/1209195217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f80cf650 0x7f86f80cfae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f86f80d0380 tx=0x7f86f002bee0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7efeb4c00640 1 -- 172.21.15.62:0/3748520465 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7efe98075410 con 0x7efe98072940 2024-03-23T11:19:03.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7ff322200640 1 -- start start 2024-03-23T11:19:03.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7f86ff600640 1 -- 172.21.15.62:0/1209195217 shutdown_connections 2024-03-23T11:19:03.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7f86ff600640 1 --2- 172.21.15.62:0/1209195217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f80cf650 0x7f86f80cfae0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7f86ff600640 1 --2- 172.21.15.62:0/1209195217 >> v2:172.21.15.62:3300/0 conn(0x7f86f815cdc0 0x7f86f80d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7f86ff600640 1 --2- 172.21.15.62:0/1209195217 >> v2:172.21.15.120:3300/0 conn(0x7f86f815c9e0 0x7f86f81607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7f86ff600640 1 -- 172.21.15.62:0/1209195217 >> 172.21.15.62:0/1209195217 conn(0x7f86f8071570 msgr2=0x7f86f8071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7f325b400640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f324006a870 0x7f324006cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7f86ff600640 1 -- 172.21.15.62:0/1209195217 shutdown_connections 2024-03-23T11:19:03.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.601+0000 7f86ff600640 1 -- 172.21.15.62:0/1209195217 wait complete. 2024-03-23T11:19:03.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.602+0000 7f325b400640 1 -- 172.21.15.62:0/2505688845 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f324c0892c0 con 0x7f32640b15e0 2024-03-23T11:19:03.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.602+0000 7f3262a00640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f324006a870 0x7f324006cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.602+0000 7f325b400640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x7f3240072a90 0x7f3240074eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.602+0000 7f325b400640 1 -- 172.21.15.62:0/2505688845 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3240075560 con 0x7f3240072a90 2024-03-23T11:19:03.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.602+0000 7f325b400640 1 -- 172.21.15.62:0/2505688845 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f324c089670 con 0x7f32640b15e0 2024-03-23T11:19:03.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.602+0000 7f3263400640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x7f3240072a90 0x7f3240074eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.603+0000 7f3262a00640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f324006a870 0x7f324006cd30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3254000f60 tx=0x7f3254007ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.604+0000 7f3263400640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x7f3240072a90 0x7f3240074eb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.604+0000 7f86ff600640 1 Processor -- start 2024-03-23T11:19:03.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.604+0000 7f86ff600640 1 -- start start 2024-03-23T11:19:03.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.605+0000 7f325b400640 1 -- 172.21.15.62:0/2505688845 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3240075560 con 0x7f3240072a90 2024-03-23T11:19:03.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.605+0000 7f52d7400640 1 -- 172.21.15.62:0/1099003624 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f52d0011790 con 0x7f52e015c900 2024-03-23T11:19:03.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f52d7400640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52c006a8a0 0x7f52c006cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f52d7400640 1 -- 172.21.15.62:0/1099003624 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52d0089280 con 0x7f52e015c900 2024-03-23T11:19:03.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f52d7400640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x7f52c0072ac0 0x7f52c0074ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f52d7400640 1 -- 172.21.15.62:0/1099003624 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52c0075590 con 0x7f52c0072ac0 2024-03-23T11:19:03.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f52d7400640 1 -- 172.21.15.62:0/1099003624 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52d0089630 con 0x7f52e015c900 2024-03-23T11:19:03.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.607+0000 7f52dfe00640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x7f52c0072ac0 0x7f52c0074ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.608+0000 7f52df400640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52c006a8a0 0x7f52c006cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.605+0000 7ff322200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff300003b00 0x7ff31c1eac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.605+0000 7ff322200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff31c1eb190 0x7ff31c148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.605+0000 7ff322200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff31c148aa0 0x7ff31c148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.605+0000 7ff322200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c1eb610 con 0x7ff300003b00 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.605+0000 7ff322200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c15d360 con 0x7ff31c1eb190 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.605+0000 7ff322200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c15d080 con 0x7ff31c148aa0 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.605+0000 7ff31b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff300003b00 0x7ff31c1eac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.605+0000 7ff31b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff300003b00 0x7ff31c1eac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37874/0 (socket says 172.21.15.62:37874) 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.605+0000 7ff31b400640 1 -- 172.21.15.62:0/3243948809 learned_addr learned my addr 172.21.15.62:0/3243948809 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7ff31be00640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff31c148aa0 0x7ff31c148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7ff31aa00640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff31c1eb190 0x7ff31c148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7ff31b400640 1 -- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff31c148aa0 msgr2=0x7ff31c148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7ff31b400640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff31c148aa0 0x7ff31c148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7ff31b400640 1 -- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff31c1eb190 msgr2=0x7ff31c148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7ff31b400640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff31c1eb190 0x7ff31c148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7ff31b400640 1 -- 172.21.15.62:0/3243948809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c146570 con 0x7ff300003b00 2024-03-23T11:19:03.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.608+0000 7ff31b400640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff300003b00 0x7ff31c1eac50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff30c007ec0 tx=0x7ff30c001750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f86ff600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f80cf650 0x7f86f81f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f86ff600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f815c9e0 0x7f86f80cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f86ff600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86f815cdc0 0x7f86f80c7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f86ff600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f80cbb60 con 0x7f86f815cdc0 2024-03-23T11:19:03.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f86ff600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f80c6dd0 con 0x7f86f815c9e0 2024-03-23T11:19:03.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f86ff600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f80c64c0 con 0x7f86f80cf650 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f86fd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86f815cdc0 0x7f86f80c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f86fd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86f815cdc0 0x7f86f80c7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37880/0 (socket says 172.21.15.62:37880) 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.606+0000 7f86fd600640 1 -- 172.21.15.62:0/2060518319 learned_addr learned my addr 172.21.15.62:0/2060518319 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.607+0000 7f86fcc00640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f80cf650 0x7f86f81f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.607+0000 7f86f7e00640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f815c9e0 0x7f86f80cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.607+0000 7f86fd600640 1 -- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f80cf650 msgr2=0x7f86f81f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.607+0000 7f86fd600640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86f80cf650 0x7f86f81f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.607+0000 7f86fd600640 1 -- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f815c9e0 msgr2=0x7f86f80cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.607+0000 7f86fd600640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f815c9e0 0x7f86f80cb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.607+0000 7f86fd600640 1 -- 172.21.15.62:0/2060518319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f0009b30 con 0x7f86f815cdc0 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.607+0000 7f86f7e00640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f815c9e0 0x7f86f80cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.608+0000 7f86fd600640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86f815cdc0 0x7f86f80c7260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f86f0036040 tx=0x7f86f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.608+0000 7f86f4c00640 1 -- 172.21.15.62:0/2060518319 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f003e450 con 0x7f86f815cdc0 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.608+0000 7f86f4c00640 1 -- 172.21.15.62:0/2060518319 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f003e5d0 con 0x7f86f815cdc0 2024-03-23T11:19:03.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.608+0000 7f86f4c00640 1 -- 172.21.15.62:0/2060518319 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f003e980 con 0x7f86f815cdc0 2024-03-23T11:19:03.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.608+0000 7f86ff600640 1 -- 172.21.15.62:0/2060518319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f80c6780 con 0x7f86f815cdc0 2024-03-23T11:19:03.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.609+0000 7ff313400640 1 -- 172.21.15.62:0/3243948809 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c0015f0 con 0x7ff300003b00 2024-03-23T11:19:03.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.608+0000 7f86ff600640 1 -- 172.21.15.62:0/2060518319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f80cbd70 con 0x7f86f815cdc0 2024-03-23T11:19:03.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.609+0000 7ff313400640 1 -- 172.21.15.62:0/3243948809 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30c03a050 con 0x7ff300003b00 2024-03-23T11:19:03.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.609+0000 7ff322200640 1 -- 172.21.15.62:0/3243948809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31c146850 con 0x7ff300003b00 2024-03-23T11:19:03.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.609+0000 7ff313400640 1 -- 172.21.15.62:0/3243948809 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c003810 con 0x7ff300003b00 2024-03-23T11:19:03.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.609+0000 7f86ff600640 1 -- 172.21.15.62:0/2060518319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f86f80b4930 con 0x7f86f815cdc0 2024-03-23T11:19:03.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.610+0000 7ff322200640 1 -- 172.21.15.62:0/3243948809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31c15bbb0 con 0x7ff300003b00 2024-03-23T11:19:03.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.612+0000 7f52dfe00640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x7f52c0072ac0 0x7f52c0074ee0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.613+0000 7f52df400640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52c006a8a0 0x7f52c006cd60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f52d8035000 tx=0x7f52d8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.613+0000 7ff322200640 1 -- 172.21.15.62:0/3243948809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff2f4000fc0 con 0x7ff300003b00 2024-03-23T11:19:03.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.614+0000 7f52d7400640 1 -- 172.21.15.62:0/1099003624 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f52c0075590 con 0x7f52c0072ac0 2024-03-23T11:19:03.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.613+0000 7f86f4c00640 1 -- 172.21.15.62:0/2060518319 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f86f002cd50 con 0x7f86f815cdc0 2024-03-23T11:19:03.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.614+0000 7f86f4c00640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f86d406a7a0 0x7f86d406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.615+0000 7f6071200640 1 -- 172.21.15.62:0/470724860 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6044002cb0 con 0x7f604c072940 2024-03-23T11:19:03.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.614+0000 7ff313400640 1 -- 172.21.15.62:0/3243948809 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff30c032070 con 0x7ff300003b00 2024-03-23T11:19:03.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.615+0000 7ff313400640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff2fc06a7a0 0x7ff2fc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.615+0000 7ff313400640 1 -- 172.21.15.62:0/3243948809 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff30c0aa7d0 con 0x7ff300003b00 2024-03-23T11:19:03.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.616+0000 7ff313400640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x7ff2fc072940 0x7ff2fc074d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.616+0000 7ff313400640 1 -- 172.21.15.62:0/3243948809 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2fc075410 con 0x7ff2fc072940 2024-03-23T11:19:03.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.616+0000 7ff313400640 1 -- 172.21.15.62:0/3243948809 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff30c075d00 con 0x7ff300003b00 2024-03-23T11:19:03.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.615+0000 7f86f4c00640 1 -- 172.21.15.62:0/2060518319 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86f00ab8e0 con 0x7f86f815cdc0 2024-03-23T11:19:03.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.615+0000 7f86f4c00640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x7f86d4072940 0x7f86d4074d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.615+0000 7f86fcc00640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f86d406a7a0 0x7f86d406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.615+0000 7f86f4c00640 1 -- 172.21.15.62:0/2060518319 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86d4075410 con 0x7f86d4072940 2024-03-23T11:19:03.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.615+0000 7f86f4c00640 1 -- 172.21.15.62:0/2060518319 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86f0076c60 con 0x7f86f815cdc0 2024-03-23T11:19:03.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.616+0000 7ff31aa00640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff2fc06a7a0 0x7ff2fc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.617+0000 7f86fcc00640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f86d406a7a0 0x7f86d406cc60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f86f8159630 tx=0x7f86e0009110 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.617+0000 7f86f7e00640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x7f86d4072940 0x7f86d4074d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.617+0000 7ff31be00640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x7ff2fc072940 0x7ff2fc074d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.617+0000 7f6062000640 1 -- 172.21.15.62:0/470724860 <== osd.2 v2:172.21.15.62:6800/36996 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6044002cb0 con 0x7f604c072940 2024-03-23T11:19:03.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.617+0000 7f6071200640 1 -- 172.21.15.62:0/470724860 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x7f604c072940 msgr2=0x7f604c074d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.617+0000 7f6071200640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x7f604c072940 0x7f604c074d60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.618+0000 7ff31aa00640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff2fc06a7a0 0x7ff2fc06cc60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff308000ff0 tx=0x7ff308009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.618+0000 7f6071200640 1 -- 172.21.15.62:0/470724860 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f604c06a7a0 msgr2=0x7f604c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.618+0000 7f6071200640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f604c06a7a0 0x7f604c06cc60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f606c0b3d90 tx=0x7f605c009040 comp rx=0 tx=0).stop 2024-03-23T11:19:03.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.618+0000 7f6071200640 1 -- 172.21.15.62:0/470724860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f606c0cf650 msgr2=0x7f606c1f1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.618+0000 7f6071200640 1 --2- 172.21.15.62:0/470724860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f606c0cf650 0x7f606c1f1cd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6064033000 tx=0x7f6064004a40 comp rx=0 tx=0).stop 2024-03-23T11:19:03.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.618+0000 7f606a800640 1 -- 172.21.15.62:0/470724860 reap_dead start 2024-03-23T11:19:03.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.618+0000 7f6071200640 1 -- 172.21.15.62:0/470724860 shutdown_connections 2024-03-23T11:19:03.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.618+0000 7f6071200640 1 -- 172.21.15.62:0/470724860 >> 172.21.15.62:0/470724860 conn(0x7f606c071570 msgr2=0x7f606c071950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.618+0000 7ff31be00640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x7ff2fc072940 0x7ff2fc074d60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.618+0000 7f86f7e00640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x7f86d4072940 0x7f86d4074d60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.619+0000 7f6071200640 1 -- 172.21.15.62:0/470724860 shutdown_connections 2024-03-23T11:19:03.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.619+0000 7f6071200640 1 -- 172.21.15.62:0/470724860 wait complete. 2024-03-23T11:19:03.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.622+0000 7f952ce00640 1 -- 172.21.15.62:0/4111614750 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9528155270 con 0x7f9508072a40 2024-03-23T11:19:03.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.622+0000 7ff313400640 1 -- 172.21.15.62:0/3243948809 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff2fc075410 con 0x7ff2fc072940 2024-03-23T11:19:03.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.622+0000 7f951e000640 1 -- 172.21.15.62:0/4111614750 <== osd.0 v2:172.21.15.62:6811/36998 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9528155270 con 0x7f9508072a40 2024-03-23T11:19:03.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.622+0000 7f86f4c00640 1 -- 172.21.15.62:0/2060518319 <== osd.7 v2:172.21.15.120:6816/37655 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f86d4075410 con 0x7f86d4072940 2024-03-23T11:19:03.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.623+0000 7f952ce00640 1 -- 172.21.15.62:0/4111614750 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x7f9508072a40 msgr2=0x7f9508074e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.623+0000 7f952ce00640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x7f9508072a40 0x7f9508074e60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.623+0000 7f952ce00640 1 -- 172.21.15.62:0/4111614750 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f950806a820 msgr2=0x7f950806cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.623+0000 7f952ce00640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f950806a820 0x7f950806cce0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9514036000 tx=0x7f9514009030 comp rx=0 tx=0).stop 2024-03-23T11:19:03.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.623+0000 7f952ce00640 1 -- 172.21.15.62:0/4111614750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9528155550 msgr2=0x7f952814aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.623+0000 7f952ce00640 1 --2- 172.21.15.62:0/4111614750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9528155550 0x7f952814aad0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f951800d540 tx=0x7f951800da00 comp rx=0 tx=0).stop 2024-03-23T11:19:03.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.623+0000 7f9526800640 1 -- 172.21.15.62:0/4111614750 reap_dead start 2024-03-23T11:19:03.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.623+0000 7f952ce00640 1 -- 172.21.15.62:0/4111614750 shutdown_connections 2024-03-23T11:19:03.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.623+0000 7f952ce00640 1 -- 172.21.15.62:0/4111614750 >> 172.21.15.62:0/4111614750 conn(0x7f9528069c60 msgr2=0x7f952806a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.623+0000 7f952ce00640 1 -- 172.21.15.62:0/4111614750 shutdown_connections 2024-03-23T11:19:03.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.624+0000 7f952ce00640 1 -- 172.21.15.62:0/4111614750 wait complete. 2024-03-23T11:19:03.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.627+0000 7f765ea00640 1 -- 172.21.15.62:0/2728551428 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f765814b040 con 0x7f7638072a90 2024-03-23T11:19:03.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.629+0000 7f764fe00640 1 -- 172.21.15.62:0/2728551428 <== osd.1 v2:172.21.15.62:6824/37001 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f765814b040 con 0x7f7638072a90 2024-03-23T11:19:03.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.633+0000 7f765ea00640 1 -- 172.21.15.62:0/2728551428 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x7f7638072a90 msgr2=0x7f7638074eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.633+0000 7f765ea00640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x7f7638072a90 0x7f7638074eb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.635+0000 7f765ea00640 1 -- 172.21.15.62:0/2728551428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f763806a870 msgr2=0x7f763806cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.635+0000 7f765ea00640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f763806a870 0x7f763806cd30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7648036000 tx=0x7f764803c040 comp rx=0 tx=0).stop 2024-03-23T11:19:03.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.635+0000 7f765ea00640 1 -- 172.21.15.62:0/2728551428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76580cf650 msgr2=0x7f76580acbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.635+0000 7f765ea00640 1 --2- 172.21.15.62:0/2728551428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76580cf650 0x7f76580acbc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f764400d5f0 tx=0x7f764400dab0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.635+0000 7f765ca00640 1 -- 172.21.15.62:0/2728551428 reap_dead start 2024-03-23T11:19:03.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.635+0000 7f765ea00640 1 -- 172.21.15.62:0/2728551428 shutdown_connections 2024-03-23T11:19:03.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.635+0000 7f765ea00640 1 -- 172.21.15.62:0/2728551428 >> 172.21.15.62:0/2728551428 conn(0x7f7658071570 msgr2=0x7f76580b8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.635+0000 7f765ea00640 1 -- 172.21.15.62:0/2728551428 shutdown_connections 2024-03-23T11:19:03.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.635+0000 7f765ea00640 1 -- 172.21.15.62:0/2728551428 wait complete. 2024-03-23T11:19:03.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.636+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efe94002cb0 con 0x7efe98072940 2024-03-23T11:19:03.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.639+0000 7efeb4c00640 1 -- 172.21.15.62:0/3748520465 <== osd.5 v2:172.21.15.120:6824/37642 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efe94002cb0 con 0x7efe98072940 2024-03-23T11:19:03.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.639+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x7efe98072940 msgr2=0x7efe98074d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.639+0000 7efebf600640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x7efe98072940 0x7efe98074d60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.640+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe9806a7a0 msgr2=0x7efe9806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.640+0000 7efebf600640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe9806a7a0 0x7efe9806cc60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efeac006fd0 tx=0x7efeac008040 comp rx=0 tx=0).stop 2024-03-23T11:19:03.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.640+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb815cdc0 msgr2=0x7efeb815ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.640+0000 7efebf600640 1 --2- 172.21.15.62:0/3748520465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb815cdc0 0x7efeb815ec50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efeb0043000 tx=0x7efeb0004a40 comp rx=0 tx=0).stop 2024-03-23T11:19:03.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.640+0000 7efebd600640 1 -- 172.21.15.62:0/3748520465 reap_dead start 2024-03-23T11:19:03.659 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:03.660 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-23T11:19:03.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.640+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 shutdown_connections 2024-03-23T11:19:03.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.641+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 >> 172.21.15.62:0/3748520465 conn(0x7efeb8071570 msgr2=0x7efeb80b7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.642+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 shutdown_connections 2024-03-23T11:19:03.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.642+0000 7efebf600640 1 -- 172.21.15.62:0/3748520465 wait complete. 2024-03-23T11:19:03.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.642+0000 7f3269c00640 1 -- 172.21.15.62:0/2505688845 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f32640b1e10 con 0x7f3240072a90 2024-03-23T11:19:03.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.643+0000 7f325b400640 1 -- 172.21.15.62:0/2505688845 <== osd.3 v2:172.21.15.62:6808/37000 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f325c034440 con 0x7f3240072a90 2024-03-23T11:19:03.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.644+0000 7f3269c00640 1 -- 172.21.15.62:0/2505688845 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x7f3240072a90 msgr2=0x7f3240074eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.644+0000 7f3269c00640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x7f3240072a90 0x7f3240074eb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.644+0000 7f3269c00640 1 -- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f324006a870 msgr2=0x7f324006cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.644+0000 7f3269c00640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f324006a870 0x7f324006cd30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3254000f60 tx=0x7f3254007ca0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.644+0000 7f3269c00640 1 -- 172.21.15.62:0/2505688845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32640b15e0 msgr2=0x7f32640a9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.644+0000 7f3269c00640 1 --2- 172.21.15.62:0/2505688845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32640b15e0 0x7f32640a9ed0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f324c00e9e0 tx=0x7f324c00eea0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.644+0000 7f3263400640 1 -- 172.21.15.62:0/2505688845 reap_dead start 2024-03-23T11:19:03.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.645+0000 7f3269c00640 1 -- 172.21.15.62:0/2505688845 shutdown_connections 2024-03-23T11:19:03.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.645+0000 7f3269c00640 1 -- 172.21.15.62:0/2505688845 >> 172.21.15.62:0/2505688845 conn(0x7f32640b02e0 msgr2=0x7f32640ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.645+0000 7f3269c00640 1 -- 172.21.15.62:0/2505688845 shutdown_connections 2024-03-23T11:19:03.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.645+0000 7f3269c00640 1 -- 172.21.15.62:0/2505688845 wait complete. 2024-03-23T11:19:03.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.648+0000 7ff322200640 1 -- 172.21.15.62:0/3243948809 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff2f4002cb0 con 0x7ff2fc072940 2024-03-23T11:19:03.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.649+0000 7f52e6600640 1 -- 172.21.15.62:0/1099003624 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f52e00d1c80 con 0x7f52c0072ac0 2024-03-23T11:19:03.664 INFO:teuthology.orchestra.run.smithi062.stdout:60129542149 2024-03-23T11:19:03.664 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-23T11:19:03.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.649+0000 7ff313400640 1 -- 172.21.15.62:0/3243948809 <== osd.6 v2:172.21.15.120:6800/37641 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff2f4002cb0 con 0x7ff2fc072940 2024-03-23T11:19:03.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.650+0000 7f52d7400640 1 -- 172.21.15.62:0/1099003624 <== osd.4 v2:172.21.15.120:6803/37640 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f52cc001480 con 0x7f52c0072ac0 2024-03-23T11:19:03.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.650+0000 7ff322200640 1 -- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x7ff2fc072940 msgr2=0x7ff2fc074d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.650+0000 7ff322200640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x7ff2fc072940 0x7ff2fc074d60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.650+0000 7ff322200640 1 -- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff2fc06a7a0 msgr2=0x7ff2fc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.650+0000 7ff322200640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff2fc06a7a0 0x7ff2fc06cc60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff308000ff0 tx=0x7ff308009040 comp rx=0 tx=0).stop 2024-03-23T11:19:03.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.650+0000 7ff322200640 1 -- 172.21.15.62:0/3243948809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff300003b00 msgr2=0x7ff31c1eac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.650+0000 7ff322200640 1 --2- 172.21.15.62:0/3243948809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff300003b00 0x7ff31c1eac50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff30c007ec0 tx=0x7ff30c001750 comp rx=0 tx=0).stop 2024-03-23T11:19:03.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.650+0000 7ff31be00640 1 -- 172.21.15.62:0/3243948809 reap_dead start 2024-03-23T11:19:03.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.651+0000 7ff322200640 1 -- 172.21.15.62:0/3243948809 shutdown_connections 2024-03-23T11:19:03.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.651+0000 7ff322200640 1 -- 172.21.15.62:0/3243948809 >> 172.21.15.62:0/3243948809 conn(0x7ff31c0b02e0 msgr2=0x7ff31c0afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.651+0000 7ff322200640 1 -- 172.21.15.62:0/3243948809 shutdown_connections 2024-03-23T11:19:03.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.651+0000 7ff322200640 1 -- 172.21.15.62:0/3243948809 wait complete. 2024-03-23T11:19:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.651+0000 7f52e6600640 1 -- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x7f52c0072ac0 msgr2=0x7f52c0074ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.651+0000 7f52e6600640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x7f52c0072ac0 0x7f52c0074ee0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.651+0000 7f52e6600640 1 -- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52c006a8a0 msgr2=0x7f52c006cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.651+0000 7f52e6600640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52c006a8a0 0x7f52c006cd60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f52d8035000 tx=0x7f52d8030000 comp rx=0 tx=0).stop 2024-03-23T11:19:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.651+0000 7f52e6600640 1 -- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e015c900 msgr2=0x7f52e01f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.651+0000 7f52e6600640 1 --2- 172.21.15.62:0/1099003624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e015c900 0x7f52e01f58a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f52d000b3e0 tx=0x7f52d000b8a0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.651+0000 7f52dfe00640 1 -- 172.21.15.62:0/1099003624 reap_dead start 2024-03-23T11:19:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.652+0000 7f52e6600640 1 -- 172.21.15.62:0/1099003624 shutdown_connections 2024-03-23T11:19:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.652+0000 7f52e6600640 1 -- 172.21.15.62:0/1099003624 >> 172.21.15.62:0/1099003624 conn(0x7f52e0071390 msgr2=0x7f52e00b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.652+0000 7f52e6600640 1 -- 172.21.15.62:0/1099003624 shutdown_connections 2024-03-23T11:19:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.652+0000 7f52e6600640 1 -- 172.21.15.62:0/1099003624 wait complete. 2024-03-23T11:19:03.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.652+0000 7f86ff600640 1 -- 172.21.15.62:0/2060518319 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f86f80b4650 con 0x7f86d4072940 2024-03-23T11:19:03.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.654+0000 7f86f4c00640 1 -- 172.21.15.62:0/2060518319 <== osd.7 v2:172.21.15.120:6816/37655 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f86e8009490 con 0x7f86d4072940 2024-03-23T11:19:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.654+0000 7f86ff600640 1 -- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x7f86d4072940 msgr2=0x7f86d4074d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.654+0000 7f86ff600640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x7f86d4072940 0x7f86d4074d60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.655+0000 7f86ff600640 1 -- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f86d406a7a0 msgr2=0x7f86d406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.655+0000 7f86ff600640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f86d406a7a0 0x7f86d406cc60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f86f8159630 tx=0x7f86e0009110 comp rx=0 tx=0).stop 2024-03-23T11:19:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.655+0000 7f86ff600640 1 -- 172.21.15.62:0/2060518319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86f815cdc0 msgr2=0x7f86f80c7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.655+0000 7f86ff600640 1 --2- 172.21.15.62:0/2060518319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86f815cdc0 0x7f86f80c7260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f86f0036040 tx=0x7f86f0004a40 comp rx=0 tx=0).stop 2024-03-23T11:19:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.655+0000 7f86fd600640 1 -- 172.21.15.62:0/2060518319 reap_dead start 2024-03-23T11:19:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.655+0000 7f86ff600640 1 -- 172.21.15.62:0/2060518319 shutdown_connections 2024-03-23T11:19:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.655+0000 7f86ff600640 1 -- 172.21.15.62:0/2060518319 >> 172.21.15.62:0/2060518319 conn(0x7f86f8071570 msgr2=0x7f86f80b5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.655+0000 7f86ff600640 1 -- 172.21.15.62:0/2060518319 shutdown_connections 2024-03-23T11:19:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.655+0000 7f86ff600640 1 -- 172.21.15.62:0/2060518319 wait complete. 2024-03-23T11:19:03.670 INFO:teuthology.orchestra.run.smithi062.stdout:60129542151 2024-03-23T11:19:03.670 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-23T11:19:03.671 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:03.672 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-23T11:19:03.672 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:03.672 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-23T11:19:03.673 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:03.674 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-23T11:19:03.678 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:03.679 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:03.679 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-23T11:19:03.680 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-23T11:19:03.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.965+0000 7fb0aca00640 1 Processor -- start 2024-03-23T11:19:03.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.966+0000 7fb0aca00640 1 -- start start 2024-03-23T11:19:03.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.967+0000 7fc30f200640 1 Processor -- start 2024-03-23T11:19:03.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.968+0000 7fc30f200640 1 -- start start 2024-03-23T11:19:03.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.968+0000 7fb0aca00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb0a004fa60 0x7fb0a004fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.968+0000 7fb0aca00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb0a0054240 0x7fb0a00389a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.969+0000 7fb0aca00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a0006870 con 0x7fb0a00df3f0 2024-03-23T11:19:03.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.969+0000 7fb0a7400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb0a004fa60 0x7fb0a004fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.969+0000 7fb0aca00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a0007740 con 0x7fb0a004fa60 2024-03-23T11:19:03.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.969+0000 7fb0a7400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb0a004fa60 0x7fb0a004fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37896/0 (socket says 172.21.15.62:37896) 2024-03-23T11:19:03.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.969+0000 7fb0a7400640 1 -- 172.21.15.62:0/586916813 learned_addr learned my addr 172.21.15.62:0/586916813 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.969+0000 7fb0aca00640 1 -- 172.21.15.62:0/586916813 --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a0050380 con 0x7fb0a0054240 2024-03-23T11:19:03.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.969+0000 7fb0a7e00640 1 --2- 172.21.15.62:0/586916813 >> v2:172.21.15.120:3300/0 conn(0x7fb0a0054240 0x7fb0a00389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.970+0000 7fb0a7e00640 1 -- 172.21.15.62:0/586916813 >> v2:172.21.15.62:3300/0 conn(0x7fb0a004fa60 msgr2=0x7fb0a004fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.971+0000 7fb0a7e00640 1 --2- 172.21.15.62:0/586916813 >> v2:172.21.15.62:3300/0 conn(0x7fb0a004fa60 0x7fb0a004fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.971+0000 7fb0a7e00640 1 -- 172.21.15.62:0/586916813 >> v1:172.21.15.62:6789/0 conn(0x7fb0a00df3f0 legacy=0x7fb0a0038290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.971+0000 7fb0a7e00640 1 -- 172.21.15.62:0/586916813 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a0006ac0 con 0x7fb0a0054240 2024-03-23T11:19:03.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.971+0000 7fc30f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc308052e70 0x7fc308008c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.971+0000 7fc30f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308055880 0x7fc308054250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.971+0000 7fc30f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc308009ba0 con 0x7fc3080e7bb0 2024-03-23T11:19:03.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.971+0000 7fc30f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3080096e0 con 0x7fc308052e70 2024-03-23T11:19:03.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.971+0000 7fc30f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080098c0 con 0x7fc308055880 2024-03-23T11:19:03.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.971+0000 7fb0a7400640 1 --2- 172.21.15.62:0/586916813 >> v2:172.21.15.62:3300/0 conn(0x7fb0a004fa60 0x7fb0a004fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:03.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.972+0000 7fb0a7e00640 1 --2- 172.21.15.62:0/586916813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0a0054240 0x7fb0a00389a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8068a10 tx=0x7fb0a80a5950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c617a05559ed753 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.972+0000 7fc30d400640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fc3080e7bb0 0x7fc308055170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41640/0 (socket says 172.21.15.62:41640) 2024-03-23T11:19:03.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.972+0000 7fc30d400640 1 -- 172.21.15.62:0/2877107980 learned_addr learned my addr 172.21.15.62:0/2877107980 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:03.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.972+0000 7fc30e800640 1 --2- 172.21.15.62:0/2877107980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308055880 0x7fc308054250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.972+0000 7fc30de00640 1 --2- 172.21.15.62:0/2877107980 >> v2:172.21.15.120:3300/0 conn(0x7fc308052e70 0x7fc308008c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.973+0000 7fb0a6000640 1 -- 172.21.15.62:0/586916813 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80aa4c0 con 0x7fb0a0054240 2024-03-23T11:19:03.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.973+0000 7fc64b000640 1 Processor -- start 2024-03-23T11:19:03.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.973+0000 7fb0a6000640 1 -- 172.21.15.62:0/586916813 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a80aa620 con 0x7fb0a0054240 2024-03-23T11:19:03.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.973+0000 7fb0a6000640 1 -- 172.21.15.62:0/586916813 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80aaa10 con 0x7fb0a0054240 2024-03-23T11:19:03.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.974+0000 7fb0aca00640 1 -- 172.21.15.62:0/586916813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0a0054240 msgr2=0x7fb0a00389a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.974+0000 7fb0aca00640 1 --2- 172.21.15.62:0/586916813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0a0054240 0x7fb0a00389a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8068a10 tx=0x7fb0a80a5950 comp rx=0 tx=0).stop 2024-03-23T11:19:03.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.973+0000 7fc30ca00640 1 -- 172.21.15.62:0/2877107980 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524794214 0 0) 0x7fc308009ba0 con 0x7fc3080e7bb0 2024-03-23T11:19:03.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.973+0000 7fc30ca00640 1 -- 172.21.15.62:0/2877107980 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f4003520 con 0x7fc3080e7bb0 2024-03-23T11:19:03.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.973+0000 7fc30e800640 1 -- 172.21.15.62:0/2877107980 >> v2:172.21.15.120:3300/0 conn(0x7fc308052e70 msgr2=0x7fc308008c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.973+0000 7fc30e800640 1 --2- 172.21.15.62:0/2877107980 >> v2:172.21.15.120:3300/0 conn(0x7fc308052e70 0x7fc308008c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.973+0000 7fc30e800640 1 -- 172.21.15.62:0/2877107980 >> v1:172.21.15.120:6789/0 conn(0x7fc3080e7bb0 legacy=0x7fc308055170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.973+0000 7fc30e800640 1 -- 172.21.15.62:0/2877107980 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3080e70d0 con 0x7fc308055880 2024-03-23T11:19:03.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.974+0000 7fb0aca00640 1 -- 172.21.15.62:0/586916813 shutdown_connections 2024-03-23T11:19:03.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.974+0000 7fb0aca00640 1 --2- 172.21.15.62:0/586916813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0a0054240 0x7fb0a00389a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.974+0000 7fb0aca00640 1 --2- 172.21.15.62:0/586916813 >> v2:172.21.15.62:3300/0 conn(0x7fb0a004fa60 0x7fb0a004fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.974+0000 7fb0aca00640 1 -- 172.21.15.62:0/586916813 >> 172.21.15.62:0/586916813 conn(0x7fb0a0009f80 msgr2=0x7fb0a000a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.973+0000 7fc64b000640 1 -- start start 2024-03-23T11:19:03.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.975+0000 7fc64b000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc6440d3130 0x7fc6440b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.975+0000 7fc64b000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc6440cd340 0x7fc6440cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.975+0000 7fc64b000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6440d3590 con 0x7fc6440b4190 2024-03-23T11:19:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.975+0000 7fb0aca00640 1 -- 172.21.15.62:0/586916813 shutdown_connections 2024-03-23T11:19:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.975+0000 7fc64b000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64415da10 con 0x7fc6440d3130 2024-03-23T11:19:03.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.975+0000 7fc64b000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64415d120 con 0x7fc6440cd340 2024-03-23T11:19:03.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.975+0000 7fc643e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc6440b4190 0x7fc6440d0250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53502/0 (socket says 172.21.15.62:53502) 2024-03-23T11:19:03.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.975+0000 7fc643e00640 1 -- 172.21.15.62:0/28462305 learned_addr learned my addr 172.21.15.62:0/28462305 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:03.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.976+0000 7fb0aca00640 1 -- 172.21.15.62:0/586916813 wait complete. 2024-03-23T11:19:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.976+0000 7fc649000640 1 --2- 172.21.15.62:0/28462305 >> v2:172.21.15.120:3300/0 conn(0x7fc6440cd340 0x7fc6440cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.976+0000 7fc63be00640 1 --2- 172.21.15.62:0/28462305 >> v2:172.21.15.62:3300/0 conn(0x7fc6440d3130 0x7fc6440b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.976+0000 7fc643400640 1 -- 172.21.15.62:0/28462305 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500498508 0 0) 0x7fc62c001110 con 0x7fc6440b4190 2024-03-23T11:19:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.976+0000 7fc643400640 1 -- 172.21.15.62:0/28462305 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc628003520 con 0x7fc6440b4190 2024-03-23T11:19:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.977+0000 7fc63be00640 1 -- 172.21.15.62:0/28462305 >> v2:172.21.15.120:3300/0 conn(0x7fc6440cd340 msgr2=0x7fc6440cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.977+0000 7fc63be00640 1 --2- 172.21.15.62:0/28462305 >> v2:172.21.15.120:3300/0 conn(0x7fc6440cd340 0x7fc6440cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.977+0000 7fc63be00640 1 -- 172.21.15.62:0/28462305 >> v1:172.21.15.62:6789/0 conn(0x7fc6440b4190 legacy=0x7fc6440d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.977+0000 7fc643400640 1 -- 172.21.15.62:0/28462305 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2238469902 0 0) 0x7fc628003520 con 0x7fc6440b4190 2024-03-23T11:19:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.977+0000 7fc63be00640 1 -- 172.21.15.62:0/28462305 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6440d0a30 con 0x7fc6440d3130 2024-03-23T11:19:03.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.977+0000 7fc30e800640 1 --2- 172.21.15.62:0/2877107980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308055880 0x7fc308054250 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc310068a10 tx=0x7fc3100a58c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d08db204cb116adc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.978+0000 7fc30ca00640 1 -- 172.21.15.62:0/2877107980 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3100ab6d0 con 0x7fc308055880 2024-03-23T11:19:03.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.978+0000 7fc30ca00640 1 -- 172.21.15.62:0/2877107980 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3100a7040 con 0x7fc308055880 2024-03-23T11:19:03.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.978+0000 7fc30ca00640 1 -- 172.21.15.62:0/2877107980 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3100abab0 con 0x7fc308055880 2024-03-23T11:19:03.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.978+0000 7fb0aca00640 1 Processor -- start 2024-03-23T11:19:03.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.979+0000 7fc30f200640 1 -- 172.21.15.62:0/2877107980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308055880 msgr2=0x7fc308054250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.979+0000 7fc30f200640 1 --2- 172.21.15.62:0/2877107980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308055880 0x7fc308054250 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc310068a10 tx=0x7fc3100a58c0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.979+0000 7fc30f200640 1 -- 172.21.15.62:0/2877107980 shutdown_connections 2024-03-23T11:19:03.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.979+0000 7fc30f200640 1 --2- 172.21.15.62:0/2877107980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308055880 0x7fc308054250 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.979+0000 7fc30f200640 1 --2- 172.21.15.62:0/2877107980 >> v2:172.21.15.120:3300/0 conn(0x7fc308052e70 0x7fc308008c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.979+0000 7fc30f200640 1 -- 172.21.15.62:0/2877107980 >> 172.21.15.62:0/2877107980 conn(0x7fc308053730 msgr2=0x7fc308053b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.978+0000 7fc63be00640 1 --2- 172.21.15.62:0/28462305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6440d3130 0x7fc6440b4570 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc644160430 tx=0x7fc63402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180eb56a7b164a9d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.978+0000 7fb0aca00640 1 -- start start 2024-03-23T11:19:03.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.979+0000 7fc643400640 1 -- 172.21.15.62:0/28462305 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c001800 con 0x7fc6440d3130 2024-03-23T11:19:03.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.979+0000 7fc643400640 1 -- 172.21.15.62:0/28462305 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc634033440 con 0x7fc6440d3130 2024-03-23T11:19:03.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.979+0000 7fc30f200640 1 -- 172.21.15.62:0/2877107980 shutdown_connections 2024-03-23T11:19:03.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.979+0000 7fc30f200640 1 -- 172.21.15.62:0/2877107980 wait complete. 2024-03-23T11:19:03.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.979+0000 7fc643400640 1 -- 172.21.15.62:0/28462305 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c001800 con 0x7fc6440d3130 2024-03-23T11:19:03.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.980+0000 7fb0aca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0a004fa60 0x7fb0a0035640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.981+0000 7fc64b000640 1 -- 172.21.15.62:0/28462305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6440d3130 msgr2=0x7fc6440b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.981+0000 7fb0aca00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0a0054240 0x7fb0a0035b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.981+0000 7fc64b000640 1 --2- 172.21.15.62:0/28462305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6440d3130 0x7fc6440b4570 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc644160430 tx=0x7fc63402b7a0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.981+0000 7fb0aca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0a00df3f0 0x7fb0a00d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.981+0000 7fb0aca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a003a130 con 0x7fb0a00df3f0 2024-03-23T11:19:03.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.981+0000 7fb0aca00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00d6010 con 0x7fb0a0054240 2024-03-23T11:19:03.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.981+0000 7fb0a7400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0a004fa60 0x7fb0a0035640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.981+0000 7fb0a7400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0a004fa60 0x7fb0a0035640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:56016/0 (socket says 172.21.15.62:56016) 2024-03-23T11:19:03.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.981+0000 7fb0a7400640 1 -- 172.21.15.62:0/2279030274 learned_addr learned my addr 172.21.15.62:0/2279030274 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.981+0000 7fb0aca00640 1 -- 172.21.15.62:0/2279030274 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00d62f0 con 0x7fb0a004fa60 2024-03-23T11:19:03.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.981+0000 7fb0a6a00640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0a0054240 0x7fb0a0035b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.982+0000 7fb0a7e00640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0a00df3f0 0x7fb0a00d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.982+0000 7fcf1fc00640 1 Processor -- start 2024-03-23T11:19:03.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.982+0000 7fcf1fc00640 1 -- start start 2024-03-23T11:19:03.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.982+0000 7fc30f200640 1 Processor -- start 2024-03-23T11:19:03.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fb0a7e00640 1 -- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0a004fa60 msgr2=0x7fb0a0035640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fb0a7e00640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0a004fa60 0x7fb0a0035640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fb0a7e00640 1 -- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0a0054240 msgr2=0x7fb0a0035b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fb0a7e00640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0a0054240 0x7fb0a0035b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fb0a7e00640 1 -- 172.21.15.62:0/2279030274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c0094d0 con 0x7fb0a00df3f0 2024-03-23T11:19:03.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fb0a7400640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0a004fa60 0x7fb0a0035640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:03.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fc64b000640 1 -- 172.21.15.62:0/28462305 shutdown_connections 2024-03-23T11:19:03.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fc64b000640 1 --2- 172.21.15.62:0/28462305 >> v2:172.21.15.120:3300/0 conn(0x7fc6440cd340 0x7fc6440cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fc64b000640 1 --2- 172.21.15.62:0/28462305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6440d3130 0x7fc6440b4570 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fc64b000640 1 -- 172.21.15.62:0/28462305 >> 172.21.15.62:0/28462305 conn(0x7fc644089f60 msgr2=0x7fc6440d2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:03.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fb0a7e00640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0a00df3f0 0x7fb0a00d16d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb0a804f2a0 tx=0x7fb0a8067fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fc64b000640 1 -- 172.21.15.62:0/28462305 shutdown_connections 2024-03-23T11:19:03.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fb09b400640 1 -- 172.21.15.62:0/2279030274 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80b8070 con 0x7fb0a00df3f0 2024-03-23T11:19:03.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fb09b400640 1 -- 172.21.15.62:0/2279030274 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a80b3990 con 0x7fb0a00df3f0 2024-03-23T11:19:03.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fb09b400640 1 -- 172.21.15.62:0/2279030274 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80aa4c0 con 0x7fb0a00df3f0 2024-03-23T11:19:03.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.983+0000 7fc30f200640 1 -- start start 2024-03-23T11:19:03.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fc30f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308052e70 0x7fc308032670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fc64b000640 1 -- 172.21.15.62:0/28462305 wait complete. 2024-03-23T11:19:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fc30f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc308055880 0x7fc30804f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fc30f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3080e7bb0 0x7fc30804fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fc30f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc308032250 con 0x7fc3080e7bb0 2024-03-23T11:19:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fc30f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30803a870 con 0x7fc308055880 2024-03-23T11:19:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fc30f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc30803a590 con 0x7fc308052e70 2024-03-23T11:19:03.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fcf4bc00640 1 Processor -- start 2024-03-23T11:19:03.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fcf1fc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcf1815cdc0 0x7fcf180d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fb0aca00640 1 -- 172.21.15.62:0/2279030274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a804e550 con 0x7fb0a00df3f0 2024-03-23T11:19:03.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.984+0000 7fb0aca00640 1 -- 172.21.15.62:0/2279030274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a00d68b0 con 0x7fb0a00df3f0 2024-03-23T11:19:03.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fc30e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3080e7bb0 0x7fc30804fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fc30de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308052e70 0x7fc308032670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fc30de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308052e70 0x7fc308032670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:56026/0 (socket says 172.21.15.62:56026) 2024-03-23T11:19:03.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fc30de00640 1 -- 172.21.15.62:0/4189708919 learned_addr learned my addr 172.21.15.62:0/4189708919 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:03.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fc30d400640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc308055880 0x7fc30804f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.986+0000 7fc30e800640 1 -- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308052e70 msgr2=0x7fc308032670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.986+0000 7fc30e800640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308052e70 0x7fc308032670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.986+0000 7fc30e800640 1 -- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc308055880 msgr2=0x7fc30804f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:03.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.986+0000 7fc30e800640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc308055880 0x7fc30804f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.986+0000 7fc30e800640 1 -- 172.21.15.62:0/4189708919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31004e550 con 0x7fc3080e7bb0 2024-03-23T11:19:03.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fcf1fc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf180cf650 0x7fcf180cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:03.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fcf1fc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1806bfc0 con 0x7fcf1815c9e0 2024-03-23T11:19:03.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fcf1fc00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf180d0000 con 0x7fcf180cf650 2024-03-23T11:19:03.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fcf1fc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf180ce140 con 0x7fcf1815cdc0 2024-03-23T11:19:03.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fcf1d200640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fcf1815c9e0 0x7fcf181607c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41644/0 (socket says 172.21.15.62:41644) 2024-03-23T11:19:03.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fcf1d200640 1 -- 172.21.15.62:0/2893123054 learned_addr learned my addr 172.21.15.62:0/2893123054 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:03.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.987+0000 7fc30d400640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc308055880 0x7fc30804f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:03.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.987+0000 7fc30e800640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3080e7bb0 0x7fc30804fe50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc31004f2a0 tx=0x7fc310067fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:03.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.987+0000 7fc64b000640 1 Processor -- start 2024-03-23T11:19:03.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.986+0000 7fb0aca00640 1 -- 172.21.15.62:0/2279030274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb084005250 con 0x7fb0a00df3f0 2024-03-23T11:19:03.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.987+0000 7f2e9b200640 1 Processor -- start 2024-03-23T11:19:03.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.987+0000 7fc306000640 1 -- 172.21.15.62:0/4189708919 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3100b4830 con 0x7fc3080e7bb0 2024-03-23T11:19:03.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.988+0000 7f2e9b200640 1 -- start start 2024-03-23T11:19:03.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.988+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807650335 0 0) 0x7fcf1806bfc0 con 0x7fcf1815c9e0 2024-03-23T11:19:03.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.988+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcefc003520 con 0x7fcf1815c9e0 2024-03-23T11:19:03.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.988+0000 7fc306000640 1 -- 172.21.15.62:0/4189708919 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3100b4990 con 0x7fc3080e7bb0 2024-03-23T11:19:03.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.988+0000 7fc306000640 1 -- 172.21.15.62:0/4189708919 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3100b4d00 con 0x7fc3080e7bb0 2024-03-23T11:19:03.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.988+0000 7fcf1dc00640 1 --2- 172.21.15.62:0/2893123054 >> v2:172.21.15.62:3300/0 conn(0x7fcf180cf650 0x7fcf180cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:03.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.988+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2787365679 0 0) 0x7fcefc003520 con 0x7fcf1815c9e0 2024-03-23T11:19:03.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fc30f200640 1 -- 172.21.15.62:0/4189708919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc308031b90 con 0x7fc3080e7bb0 2024-03-23T11:19:03.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 >> v2:172.21.15.120:3300/0 conn(0x7fcf1815cdc0 msgr2=0x7fcf180d1f20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T11:19:03.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 --2- 172.21.15.62:0/2893123054 >> v2:172.21.15.120:3300/0 conn(0x7fcf1815cdc0 0x7fcf180d1f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 >> v2:172.21.15.62:3300/0 conn(0x7fcf180cf650 msgr2=0x7fcf180cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:03.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 --2- 172.21.15.62:0/2893123054 >> v2:172.21.15.62:3300/0 conn(0x7fcf180cf650 0x7fcf180cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:03.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf180b5260 con 0x7fcf1815c9e0 2024-03-23T11:19:03.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7f2e9b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2e9c15c990 0x7f2e9c06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fcf100019f0 con 0x7fcf1815c9e0 2024-03-23T11:19:04.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 >> v1:172.21.15.120:6789/0 conn(0x7fcf1815c9e0 legacy=0x7fcf181607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcefc0039c0 0x7fcefc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcefc0047c0 0x7fcefc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0051d0 0x7fcefc014190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180d0000 con 0x7fcefc0039c0 2024-03-23T11:19:04.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180ce140 con 0x7fcefc0047c0 2024-03-23T11:19:04.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7f2e9b200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9c15cd70 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.990+0000 7f2e9b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9c0ce2f0 con 0x7f2e9c0cefa0 2024-03-23T11:19:04.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.990+0000 7f2e9b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e9c0ce5d0 con 0x7f2e9c15c990 2024-03-23T11:19:04.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.990+0000 7f2e99e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2e9c15c990 0x7f2e9c06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.990+0000 7f2e99e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2e9c15c990 0x7f2e9c06aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53206/0 (socket says 172.21.15.62:53206) 2024-03-23T11:19:04.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.990+0000 7f2e99e00640 1 -- 172.21.15.62:0/2202271309 learned_addr learned my addr 172.21.15.62:0/2202271309 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:04.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.985+0000 7fcf4bc00640 1 -- start start 2024-03-23T11:19:04.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf4bc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 0x7fcf4415cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf4bc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcf441532d0 0x7fcf44159050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf4bc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4415a040 con 0x7fcf44159590 2024-03-23T11:19:04.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf4bc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf44159a80 con 0x7fcf441532d0 2024-03-23T11:19:04.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf4bc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf44159d60 con 0x7fcf44152ef0 2024-03-23T11:19:04.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.990+0000 7fcf49200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 0x7fcf4415cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.989+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0149b0 con 0x7fcefc0051d0 2024-03-23T11:19:04.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.990+0000 7fb09b400640 1 -- 172.21.15.62:0/2279030274 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb0a80b3d00 con 0x7fb0a00df3f0 2024-03-23T11:19:04.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.990+0000 7fc30f200640 1 -- 172.21.15.62:0/4189708919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc308032bb0 con 0x7fc3080e7bb0 2024-03-23T11:19:04.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.990+0000 7fcf1d200640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcefc0039c0 0x7fcefc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.991+0000 7f2e98a00640 1 -- 172.21.15.62:0/2202271309 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707870578 0 0) 0x7f2e9c0ce2f0 con 0x7f2e9c0cefa0 2024-03-23T11:19:04.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.991+0000 7fcf17e00640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcefc0047c0 0x7fcefc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.991+0000 7fcf1dc00640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0051d0 0x7fcefc014190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.991+0000 7f2e98a00640 1 -- 172.21.15.62:0/2202271309 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e80003520 con 0x7f2e9c0cefa0 2024-03-23T11:19:04.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.991+0000 7fcf1d200640 1 -- 172.21.15.62:0/2893123054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0051d0 msgr2=0x7fcefc014190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:04.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.991+0000 7fb09b400640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb08806a8a0 0x7fb08806cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.991+0000 7fcf1d200640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0051d0 0x7fcefc014190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.991+0000 7fcf49200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 0x7fcf4415cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:56038/0 (socket says 172.21.15.62:56038) 2024-03-23T11:19:04.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.991+0000 7fcf49200640 1 -- 172.21.15.62:0/187275530 learned_addr learned my addr 172.21.15.62:0/187275530 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:04.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.991+0000 7fcf43e00640 1 --2- 172.21.15.62:0/187275530 >> v2:172.21.15.120:3300/0 conn(0x7fcf441532d0 0x7fcf44159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7fb09b400640 1 -- 172.21.15.62:0/2279030274 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0a81247b0 con 0x7fb0a00df3f0 2024-03-23T11:19:04.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7fa35f000640 1 Processor -- start 2024-03-23T11:19:04.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7fcf1d200640 1 -- 172.21.15.62:0/2893123054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcefc0047c0 msgr2=0x7fcefc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:04.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7fcf1d200640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcefc0047c0 0x7fcefc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7fcf1d200640 1 -- 172.21.15.62:0/2893123054 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc003520 con 0x7fcefc0039c0 2024-03-23T11:19:04.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7fcf1dc00640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0051d0 0x7fcefc014190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:04.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7fcf1dc00640 1 -- 172.21.15.62:0/2893123054 reap_dead start 2024-03-23T11:19:04.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7fa35f000640 1 -- start start 2024-03-23T11:19:04.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.991+0000 7fc64b000640 1 -- start start 2024-03-23T11:19:04.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.993+0000 7fc64b000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6440b4190 0x7fc6441fc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.993+0000 7fc64b000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6440cd340 0x7fc6440b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.993+0000 7fc64b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6440d3130 0x7fc6440b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.993+0000 7fc64b000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6440b6350 con 0x7fc6440cd340 2024-03-23T11:19:04.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.993+0000 7fc64b000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6440b6630 con 0x7fc6440b4190 2024-03-23T11:19:04.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.993+0000 7fc64b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6440b6910 con 0x7fc6440d3130 2024-03-23T11:19:04.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7f2e99e00640 1 -- 172.21.15.62:0/2202271309 >> v1:172.21.15.120:6789/0 conn(0x7f2e9c0cefa0 legacy=0x7f2e9c0cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7f2e99e00640 1 -- 172.21.15.62:0/2202271309 >> v1:172.21.15.62:6789/0 conn(0x7f2e9c06b3f0 legacy=0x7f2e9c0d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7f2e99e00640 1 -- 172.21.15.62:0/2202271309 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9c0d0360 con 0x7f2e9c15c990 2024-03-23T11:19:04.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.992+0000 7fc30f200640 1 -- 172.21.15.62:0/4189708919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc308032390 con 0x7fc3080e7bb0 2024-03-23T11:19:04.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fc649000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6440d3130 0x7fc6440b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fc643e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6440b4190 0x7fc6441fc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fc643e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6440b4190 0x7fc6441fc5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53226/0 (socket says 172.21.15.62:53226) 2024-03-23T11:19:04.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fc643e00640 1 -- 172.21.15.62:0/1205734320 learned_addr learned my addr 172.21.15.62:0/1205734320 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:04.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7f2e99e00640 1 --2- 172.21.15.62:0/2202271309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c15c990 0x7f2e9c06aeb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009870 tx=0x7f2e8c02ba80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0f797653a0ca14 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fc63be00640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6440cd340 0x7fc6440b7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fcf1d200640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcefc0039c0 0x7fcefc003de0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000a7b0 tx=0x7fcefc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f681f7dfc384ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fb0a7400640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb08806a8a0 0x7fb08806cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fa35f000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa35815c900 0x7fa3580d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fa35f000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3580d1100 con 0x7fa35815cce0 2024-03-23T11:19:04.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fa35f000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3580d13e0 con 0x7fa3580cd690 2024-03-23T11:19:04.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fa35f000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3580b89f0 con 0x7fa35815c900 2024-03-23T11:19:04.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7f2e98a00640 1 -- 172.21.15.62:0/2202271309 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c02d840 con 0x7f2e9c15c990 2024-03-23T11:19:04.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7f2e98a00640 1 -- 172.21.15.62:0/2202271309 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8c02d9a0 con 0x7f2e9c15c990 2024-03-23T11:19:04.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fcf49200640 1 -- 172.21.15.62:0/187275530 >> v2:172.21.15.120:3300/0 conn(0x7fcf441532d0 msgr2=0x7fcf44159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:04.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fcf49200640 1 --2- 172.21.15.62:0/187275530 >> v2:172.21.15.120:3300/0 conn(0x7fcf441532d0 0x7fcf44159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fcf49200640 1 -- 172.21.15.62:0/187275530 >> v1:172.21.15.120:6789/0 conn(0x7fcf44159590 legacy=0x7fcf440ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:04.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.994+0000 7fcf49200640 1 -- 172.21.15.62:0/187275530 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf4415c770 con 0x7fcf44152ef0 2024-03-23T11:19:04.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7f2e98a00640 1 -- 172.21.15.62:0/2202271309 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c02dd50 con 0x7f2e9c15c990 2024-03-23T11:19:04.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7fc643e00640 1 -- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6440d3130 msgr2=0x7fc6440b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7fc643e00640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6440d3130 0x7fc6440b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7fc643e00640 1 -- 172.21.15.62:0/1205734320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6440cd340 msgr2=0x7fc6440b7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7fa357e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa35815c900 0x7fa3580d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7fc643e00640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6440cd340 0x7fc6440b7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7fc643e00640 1 -- 172.21.15.62:0/1205734320 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc634009400 con 0x7fc6440b4190 2024-03-23T11:19:04.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7fa357e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa35815c900 0x7fa3580d21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53238/0 (socket says 172.21.15.62:53238) 2024-03-23T11:19:04.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7fa357e00640 1 -- 172.21.15.62:0/4073358893 learned_addr learned my addr 172.21.15.62:0/4073358893 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:04.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7fc63be00640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6440cd340 0x7fc6440b7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:04.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7f2e9b200640 1 -- 172.21.15.62:0/2202271309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c15c990 msgr2=0x7f2e9c06aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7f2e9b200640 1 --2- 172.21.15.62:0/2202271309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c15c990 0x7f2e9c06aeb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009870 tx=0x7f2e8c02ba80 comp rx=0 tx=0).stop 2024-03-23T11:19:04.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7fa357400640 1 --1- 172.21.15.62:0/4073358893 >> v1:172.21.15.62:6789/0 conn(0x7fa35815cce0 0x7fa3580b82e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53516/0 (socket says 172.21.15.62:53516) 2024-03-23T11:19:04.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.996+0000 7f2e9b200640 1 -- 172.21.15.62:0/2202271309 shutdown_connections 2024-03-23T11:19:04.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.996+0000 7f2e9b200640 1 --2- 172.21.15.62:0/2202271309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c15c990 0x7f2e9c06aeb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.996+0000 7f2e9b200640 1 -- 172.21.15.62:0/2202271309 >> 172.21.15.62:0/2202271309 conn(0x7f2e9c089f60 msgr2=0x7f2e9c0d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.996+0000 7fa357e00640 1 -- 172.21.15.62:0/4073358893 >> v1:172.21.15.120:6789/0 conn(0x7fa3580cd690 legacy=0x7fa3580b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.996+0000 7fa357e00640 1 -- 172.21.15.62:0/4073358893 >> v1:172.21.15.62:6789/0 conn(0x7fa35815cce0 legacy=0x7fa3580b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.996+0000 7fa357e00640 1 -- 172.21.15.62:0/4073358893 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3580d16c0 con 0x7fa35815c900 2024-03-23T11:19:04.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.995+0000 7fc306000640 1 -- 172.21.15.62:0/4189708919 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc3100bf3f0 con 0x7fc3080e7bb0 2024-03-23T11:19:04.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.996+0000 7fc306000640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc2f006a7f0 0x7fc2f006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.996+0000 7fb0a7400640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb08806a8a0 0x7fb08806cd60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00dec00 tx=0x7fb09c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.996+0000 7f2e9b200640 1 -- 172.21.15.62:0/2202271309 shutdown_connections 2024-03-23T11:19:04.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.996+0000 7fc643e00640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6440b4190 0x7fc6441fc5a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00c750 tx=0x7fc62c00cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.996+0000 7fcf49200640 1 --2- 172.21.15.62:0/187275530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 0x7fcf4415cf70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf3400a910 tx=0x7fcf3400a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=662642861b1b7dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.997+0000 7fc640c00640 1 -- 172.21.15.62:0/1205734320 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c005ad0 con 0x7fc6440b4190 2024-03-23T11:19:04.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.997+0000 7fc640c00640 1 -- 172.21.15.62:0/1205734320 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c005c30 con 0x7fc6440b4190 2024-03-23T11:19:04.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.997+0000 7fc640c00640 1 -- 172.21.15.62:0/1205734320 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c007070 con 0x7fc6440b4190 2024-03-23T11:19:04.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.997+0000 7fcf43400640 1 -- 172.21.15.62:0/187275530 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3402ca10 con 0x7fcf44152ef0 2024-03-23T11:19:04.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.997+0000 7fcf43400640 1 -- 172.21.15.62:0/187275530 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf34030070 con 0x7fcf44152ef0 2024-03-23T11:19:04.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.997+0000 7fcf43400640 1 -- 172.21.15.62:0/187275530 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3402cd90 con 0x7fcf44152ef0 2024-03-23T11:19:04.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.997+0000 7fc30de00640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc2f006a7f0 0x7fc2f006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.997+0000 7fc64b000640 1 -- 172.21.15.62:0/1205734320 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6440b6bf0 con 0x7fc6440b4190 2024-03-23T11:19:04.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.997+0000 7fc64b000640 1 -- 172.21.15.62:0/1205734320 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6440b71b0 con 0x7fc6440b4190 2024-03-23T11:19:04.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.997+0000 7f2e9b200640 1 -- 172.21.15.62:0/2202271309 wait complete. 2024-03-23T11:19:04.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.998+0000 7f349b600640 1 Processor -- start 2024-03-23T11:19:04.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.998+0000 7fcf4bc00640 1 -- 172.21.15.62:0/187275530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 msgr2=0x7fcf4415cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.998+0000 7fcf4bc00640 1 --2- 172.21.15.62:0/187275530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 0x7fcf4415cf70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf3400a910 tx=0x7fcf3400a250 comp rx=0 tx=0).stop 2024-03-23T11:19:04.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.998+0000 7f349b600640 1 -- start start 2024-03-23T11:19:04.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.998+0000 7fc30de00640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc2f006a7f0 0x7fc2f006ccb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009b60 tx=0x7fc2fc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.999+0000 7fcf4bc00640 1 -- 172.21.15.62:0/187275530 shutdown_connections 2024-03-23T11:19:04.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.999+0000 7fcf4bc00640 1 --2- 172.21.15.62:0/187275530 >> v2:172.21.15.120:3300/0 conn(0x7fcf441532d0 0x7fcf44159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.999+0000 7fcf4bc00640 1 --2- 172.21.15.62:0/187275530 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 0x7fcf4415cf70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.999+0000 7fcf4bc00640 1 -- 172.21.15.62:0/187275530 >> 172.21.15.62:0/187275530 conn(0x7fcf44069c60 msgr2=0x7fcf4406a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.999+0000 7fcf4bc00640 1 -- 172.21.15.62:0/187275530 shutdown_connections 2024-03-23T11:19:04.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.999+0000 7f2e9b200640 1 Processor -- start 2024-03-23T11:19:04.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.999+0000 7fa357e00640 1 --2- 172.21.15.62:0/4073358893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa35815c900 0x7fa3580d21a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa348009870 tx=0x7fa34802bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c69bfc113b16ec20 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.999+0000 7fc64b000640 1 -- 172.21.15.62:0/1205734320 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6440cadc0 con 0x7fc6440b4190 2024-03-23T11:19:04.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.999+0000 7f2e9b200640 1 -- start start 2024-03-23T11:19:04.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:03.999+0000 7fcf4bc00640 1 -- 172.21.15.62:0/187275530 wait complete. 2024-03-23T11:19:04.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.000+0000 7fa356a00640 1 -- 172.21.15.62:0/4073358893 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34802d870 con 0x7fa35815c900 2024-03-23T11:19:04.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.000+0000 7fa356a00640 1 -- 172.21.15.62:0/4073358893 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34802d9d0 con 0x7fa35815c900 2024-03-23T11:19:04.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.000+0000 7fa356a00640 1 -- 172.21.15.62:0/4073358893 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34802dd60 con 0x7fa35815c900 2024-03-23T11:19:04.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.000+0000 7f349b600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 0x7f34940b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.000+0000 7f349b600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3494162620 0x7f34940d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.000+0000 7f349b600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f34940d1600 0x7f34940b4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.000+0000 7f349b600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34940b20b0 con 0x7f3494162620 2024-03-23T11:19:04.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.000+0000 7f349b600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34940d1380 con 0x7f34940d1600 2024-03-23T11:19:04.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.000+0000 7f349b600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3494161460 con 0x7f34940d0a60 2024-03-23T11:19:04.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.000+0000 7f2e9b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e9c06b3f0 0x7f2e9c0ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f2e9b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c0cefa0 0x7f2e9c0b6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f3498c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 0x7f34940b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f2e9b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e9c15c990 0x7f2e9c156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f2e9b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9c0caf00 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f2e9b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9c0b6980 con 0x7f2e9c0cefa0 2024-03-23T11:19:04.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f2e9b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9c0cb0f0 con 0x7f2e9c15c990 2024-03-23T11:19:04.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7fa35f000640 1 -- 172.21.15.62:0/4073358893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa35815c900 msgr2=0x7fa3580d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7fa35f000640 1 --2- 172.21.15.62:0/4073358893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa35815c900 0x7fa3580d21a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa348009870 tx=0x7fa34802bab0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7fc640c00640 1 -- 172.21.15.62:0/1205734320 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc62c0071d0 con 0x7fc6440b4190 2024-03-23T11:19:04.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7fa35f000640 1 -- 172.21.15.62:0/4073358893 shutdown_connections 2024-03-23T11:19:04.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7fa35f000640 1 --2- 172.21.15.62:0/4073358893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa35815c900 0x7fa3580d21a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7fa35f000640 1 -- 172.21.15.62:0/4073358893 >> 172.21.15.62:0/4073358893 conn(0x7fa358071390 msgr2=0x7fa358071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f2e99400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c0cefa0 0x7f2e9c0b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f2e99e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e9c06b3f0 0x7f2e9c0ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f2e99e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e9c06b3f0 0x7f2e9c0ca9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37982/0 (socket says 172.21.15.62:37982) 2024-03-23T11:19:04.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f2e99e00640 1 -- 172.21.15.62:0/948692474 learned_addr learned my addr 172.21.15.62:0/948692474 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:04.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f3493e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3494162620 0x7f34940d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f3493e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3494162620 0x7f34940d0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37974/0 (socket says 172.21.15.62:37974) 2024-03-23T11:19:04.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f3493e00640 1 -- 172.21.15.62:0/4007760820 learned_addr learned my addr 172.21.15.62:0/4007760820 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:04.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.001+0000 7f3499600640 1 --2- 172.21.15.62:0/4007760820 >> v2:172.21.15.120:3300/0 conn(0x7f34940d1600 0x7f34940b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7fa35f000640 1 -- 172.21.15.62:0/4073358893 shutdown_connections 2024-03-23T11:19:04.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f2e9a800640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e9c15c990 0x7f2e9c156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f2e99e00640 1 -- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e9c15c990 msgr2=0x7f2e9c156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f2e99e00640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e9c15c990 0x7f2e9c156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f2e99e00640 1 -- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c0cefa0 msgr2=0x7f2e9c0b6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f2e99e00640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c0cefa0 0x7f2e9c0b6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f2e99e00640 1 -- 172.21.15.62:0/948692474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c0094d0 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f3493e00640 1 -- 172.21.15.62:0/4007760820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 msgr2=0x7f34940b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f3493e00640 1 --2- 172.21.15.62:0/4007760820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 0x7f34940b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f3493e00640 1 -- 172.21.15.62:0/4007760820 >> v2:172.21.15.120:3300/0 conn(0x7f34940d1600 msgr2=0x7f34940b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f3493e00640 1 --2- 172.21.15.62:0/4007760820 >> v2:172.21.15.120:3300/0 conn(0x7f34940d1600 0x7f34940b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f3493e00640 1 -- 172.21.15.62:0/4007760820 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3494161720 con 0x7f3494162620 2024-03-23T11:19:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7fc640c00640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc62006a7d0 0x7fc62006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7fcf4bc00640 1 Processor -- start 2024-03-23T11:19:04.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7fa35f000640 1 -- 172.21.15.62:0/4073358893 wait complete. 2024-03-23T11:19:04.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7fcf4bc00640 1 -- start start 2024-03-23T11:19:04.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7fc640c00640 1 -- 172.21.15.62:0/1205734320 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc62c088240 con 0x7fc6440b4190 2024-03-23T11:19:04.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.002+0000 7f2e99e00640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e9c06b3f0 0x7f2e9c0ca9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009870 tx=0x7f2e8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.003+0000 7fc63be00640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc62006a7d0 0x7fc62006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.003+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10034630 con 0x7fcefc0039c0 2024-03-23T11:19:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.003+0000 7fb09b400640 1 -- 172.21.15.62:0/2279030274 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb0a80efd80 con 0x7fb0a00df3f0 2024-03-23T11:19:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.003+0000 7f3493e00640 1 --2- 172.21.15.62:0/4007760820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494162620 0x7f34940d0e40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f349408aef0 tx=0x7f348402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c9872b366fc1e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.004+0000 7fc63be00640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc62006a7d0 0x7fc62006cc90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc634035000 tx=0x7fc63402f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.004+0000 7fc306000640 1 -- 172.21.15.62:0/4189708919 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3100f3870 con 0x7fc3080e7bb0 2024-03-23T11:19:04.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fcf4bc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 0x7fcf441423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fa35f000640 1 Processor -- start 2024-03-23T11:19:04.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fcf4bc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf441532d0 0x7fcf44154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fa35f000640 1 -- start start 2024-03-23T11:19:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fcf4bc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf44159590 0x7fcf4415b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fcf4bc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf44153dc0 con 0x7fcf44159590 2024-03-23T11:19:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fcf4bc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf44153800 con 0x7fcf441532d0 2024-03-23T11:19:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fcf4bc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf44153ae0 con 0x7fcf44152ef0 2024-03-23T11:19:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fcf49200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 0x7fcf441423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fcf49c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf44159590 0x7fcf4415b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fcf49c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf44159590 0x7fcf4415b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37986/0 (socket says 172.21.15.62:37986) 2024-03-23T11:19:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fcf49c00640 1 -- 172.21.15.62:0/3497912396 learned_addr learned my addr 172.21.15.62:0/3497912396 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.005+0000 7fcf43e00640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf441532d0 0x7fcf44154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.006+0000 7fcf49c00640 1 -- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 msgr2=0x7fcf441423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.006+0000 7fcf49c00640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 0x7fcf441423a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.006+0000 7fcf49c00640 1 -- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf441532d0 msgr2=0x7fcf44154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.006+0000 7fcf49c00640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf441532d0 0x7fcf44154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.006+0000 7fcf49c00640 1 -- 172.21.15.62:0/3497912396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3400a570 con 0x7fcf44159590 2024-03-23T11:19:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.006+0000 7fcf49c00640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf44159590 0x7fcf4415b5a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00b9e0 tx=0x7fcf3c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.006+0000 7fa35f000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3580cd690 0x7fa3581f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.006+0000 7fa35f000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa35815c900 0x7fa35815d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.006+0000 7fa35f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa35815cce0 0x7fa35815ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa35f000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35815e760 con 0x7fa3580cd690 2024-03-23T11:19:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa35f000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35815dbe0 con 0x7fa35815c900 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa35f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35815dec0 con 0x7fa35815cce0 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa357e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3580cd690 0x7fa3581f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa357e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3580cd690 0x7fa3581f18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37988/0 (socket says 172.21.15.62:37988) 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa357e00640 1 -- 172.21.15.62:0/3734146765 learned_addr learned my addr 172.21.15.62:0/3734146765 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa357400640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa35815c900 0x7fa35815d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa35d000640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa35815cce0 0x7fa35815ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa357e00640 1 -- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa35815cce0 msgr2=0x7fa35815ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa357e00640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa35815cce0 0x7fa35815ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa357e00640 1 -- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa35815c900 msgr2=0x7fa35815d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa357e00640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa35815c900 0x7fa35815d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.007+0000 7fa357e00640 1 -- 172.21.15.62:0/3734146765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3480094d0 con 0x7fa3580cd690 2024-03-23T11:19:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.008+0000 7fa357e00640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3580cd690 0x7fa3581f18f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa348009990 tx=0x7fa3480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1002f070 con 0x7fcefc0039c0 2024-03-23T11:19:04.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7fcf17400640 1 -- 172.21.15.62:0/2893123054 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10034a30 con 0x7fcefc0039c0 2024-03-23T11:19:04.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7fa34fe00640 1 -- 172.21.15.62:0/3734146765 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34803b040 con 0x7fa3580cd690 2024-03-23T11:19:04.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7fcf40c00640 1 -- 172.21.15.62:0/3497912396 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c003b70 con 0x7fcf44159590 2024-03-23T11:19:04.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7fcf40c00640 1 -- 172.21.15.62:0/3497912396 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3c003cd0 con 0x7fcf44159590 2024-03-23T11:19:04.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7fcf40c00640 1 -- 172.21.15.62:0/3497912396 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c00cb60 con 0x7fcf44159590 2024-03-23T11:19:04.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7fa34fe00640 1 -- 172.21.15.62:0/3734146765 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa348037c40 con 0x7fa3580cd690 2024-03-23T11:19:04.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7f2e92000640 1 -- 172.21.15.62:0/948692474 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c004230 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7f2e92000640 1 -- 172.21.15.62:0/948692474 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8c004390 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7f2e92000640 1 -- 172.21.15.62:0/948692474 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c037890 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7fcf4bc00640 1 -- 172.21.15.62:0/3497912396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf4415ada0 con 0x7fcf44159590 2024-03-23T11:19:04.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7f3493400640 1 -- 172.21.15.62:0/4007760820 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348402c9f0 con 0x7f3494162620 2024-03-23T11:19:04.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7f3493400640 1 -- 172.21.15.62:0/4007760820 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348402cb50 con 0x7f3494162620 2024-03-23T11:19:04.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7f3493400640 1 -- 172.21.15.62:0/4007760820 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34840343f0 con 0x7f3494162620 2024-03-23T11:19:04.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.009+0000 7f2e9b200640 1 -- 172.21.15.62:0/948692474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9c0cb3d0 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.010+0000 7f2e9b200640 1 -- 172.21.15.62:0/948692474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9c0cb990 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.010+0000 7fa34fe00640 1 -- 172.21.15.62:0/3734146765 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34802d7f0 con 0x7fa3580cd690 2024-03-23T11:19:04.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.010+0000 7f349b600640 1 -- 172.21.15.62:0/4007760820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494162620 msgr2=0x7f34940d0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.010+0000 7f349b600640 1 --2- 172.21.15.62:0/4007760820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494162620 0x7f34940d0e40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f349408aef0 tx=0x7f348402be50 comp rx=0 tx=0).stop 2024-03-23T11:19:04.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.010+0000 7fcf4bc00640 1 -- 172.21.15.62:0/3497912396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf441545a0 con 0x7fcf44159590 2024-03-23T11:19:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.010+0000 7fa35f000640 1 -- 172.21.15.62:0/3734146765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa35815e1a0 con 0x7fa3580cd690 2024-03-23T11:19:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.010+0000 7fa35f000640 1 -- 172.21.15.62:0/3734146765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3581470e0 con 0x7fa3580cd690 2024-03-23T11:19:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.010+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2893123054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcefc0039c0 msgr2=0x7fcefc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.010+0000 7fcf1fc00640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcefc0039c0 0x7fcefc003de0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000a7b0 tx=0x7fcefc0057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.011+0000 7fc306000640 1 -- 172.21.15.62:0/4189708919 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc308032390 con 0x7fc3080e7bb0 2024-03-23T11:19:04.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.011+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2893123054 shutdown_connections 2024-03-23T11:19:04.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.011+0000 7fcf1fc00640 1 --2- 172.21.15.62:0/2893123054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcefc0039c0 0x7fcefc003de0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.011+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2893123054 >> 172.21.15.62:0/2893123054 conn(0x7fcf18071570 msgr2=0x7fcf18071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.011+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2893123054 shutdown_connections 2024-03-23T11:19:04.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.011+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2893123054 wait complete. 2024-03-23T11:19:04.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.011+0000 7f2e9b200640 1 -- 172.21.15.62:0/948692474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e74005250 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.012+0000 7fcf4bc00640 1 -- 172.21.15.62:0/3497912396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf4415ac00 con 0x7fcf44159590 2024-03-23T11:19:04.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.012+0000 7f349b600640 1 -- 172.21.15.62:0/4007760820 shutdown_connections 2024-03-23T11:19:04.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.012+0000 7f349b600640 1 --2- 172.21.15.62:0/4007760820 >> v2:172.21.15.120:3300/0 conn(0x7f34940d1600 0x7f34940b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.012+0000 7f349b600640 1 --2- 172.21.15.62:0/4007760820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494162620 0x7f34940d0e40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.012+0000 7f349b600640 1 --2- 172.21.15.62:0/4007760820 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 0x7f34940b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.012+0000 7f349b600640 1 -- 172.21.15.62:0/4007760820 >> 172.21.15.62:0/4007760820 conn(0x7f3494071390 msgr2=0x7f3494071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.012+0000 7fa35f000640 1 -- 172.21.15.62:0/3734146765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3580d0ca0 con 0x7fa3580cd690 2024-03-23T11:19:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.013+0000 7f349b600640 1 -- 172.21.15.62:0/4007760820 shutdown_connections 2024-03-23T11:19:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.013+0000 7f349b600640 1 -- 172.21.15.62:0/4007760820 wait complete. 2024-03-23T11:19:04.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.013+0000 7fcf1fc00640 1 Processor -- start 2024-03-23T11:19:04.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.013+0000 7fcf1fc00640 1 -- start start 2024-03-23T11:19:04.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.014+0000 7fcf1fc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcefc0039c0 0x7fcf180c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.014+0000 7fcf1fc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf180706e0 0x7fcf180d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.014+0000 7fcf1fc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf18070b60 0x7fcf180d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.014+0000 7fcf1fc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180cc280 con 0x7fcefc0039c0 2024-03-23T11:19:04.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.014+0000 7fcf1fc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180cb420 con 0x7fcf18070b60 2024-03-23T11:19:04.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.014+0000 7fcf1fc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180cb700 con 0x7fcf180706e0 2024-03-23T11:19:04.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.015+0000 7fcf17e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf180706e0 0x7fcf180d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.015+0000 7fcf1dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf18070b60 0x7fcf180d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.015+0000 7fcf17e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf180706e0 0x7fcf180d28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:56102/0 (socket says 172.21.15.62:56102) 2024-03-23T11:19:04.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.015+0000 7fcf17e00640 1 -- 172.21.15.62:0/2315655800 learned_addr learned my addr 172.21.15.62:0/2315655800 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:04.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.015+0000 7f349b600640 1 Processor -- start 2024-03-23T11:19:04.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.016+0000 7fcf40c00640 1 -- 172.21.15.62:0/3497912396 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fcf3c00ccc0 con 0x7fcf44159590 2024-03-23T11:19:04.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.016+0000 7fcf1dc00640 1 -- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf180706e0 msgr2=0x7fcf180d28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.016+0000 7fcf1dc00640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf180706e0 0x7fcf180d28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.016+0000 7fcf1dc00640 1 -- 172.21.15.62:0/2315655800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcefc0039c0 msgr2=0x7fcf180c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.016+0000 7fcf1dc00640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcefc0039c0 0x7fcf180c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.016+0000 7fcf1dc00640 1 -- 172.21.15.62:0/2315655800 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf180cb9e0 con 0x7fcf18070b60 2024-03-23T11:19:04.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.016+0000 7fcf17e00640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf180706e0 0x7fcf180d28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:04.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.017+0000 7fcf1dc00640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf18070b60 0x7fcf180d2e10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcf180ce280 tx=0x7fcf0400f670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.016+0000 7f349b600640 1 -- start start 2024-03-23T11:19:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.017+0000 7f349b600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 0x7f349414ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.017+0000 7f349b600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940d1600 0x7f34940b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.017+0000 7f349b600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494162620 0x7f34940b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.017+0000 7f349b600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349414c320 con 0x7f3494162620 2024-03-23T11:19:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.017+0000 7f349b600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349414b710 con 0x7f34940d1600 2024-03-23T11:19:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.017+0000 7f349b600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349414b9f0 con 0x7f34940d0a60 2024-03-23T11:19:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.017+0000 7fcf40c00640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf2406a7f0 0x7fcf2406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.018+0000 7fcf49200640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf2406a7f0 0x7fcf2406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.017+0000 7f2e92000640 1 -- 172.21.15.62:0/948692474 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2e8c0379f0 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.018+0000 7f3493e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940d1600 0x7f34940b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.018+0000 7f3493e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940d1600 0x7f34940b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53304/0 (socket says 172.21.15.62:53304) 2024-03-23T11:19:04.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.018+0000 7f3498c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 0x7f349414ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.018+0000 7fcf40c00640 1 -- 172.21.15.62:0/3497912396 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf3c057c20 con 0x7fcf44159590 2024-03-23T11:19:04.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.018+0000 7f3498c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 0x7f349414ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:56118/0 (socket says 172.21.15.62:56118) 2024-03-23T11:19:04.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.018+0000 7f3498c00640 1 -- 172.21.15.62:0/1763918751 learned_addr learned my addr 172.21.15.62:0/1763918751 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:04.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.019+0000 7f2e92000640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2e7c06a7f0 0x7f2e7c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7fcf49200640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf2406a7f0 0x7fcf2406ccb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcf34037000 tx=0x7fcf34032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.019+0000 7fcf14c00640 1 -- 172.21.15.62:0/2315655800 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf04014660 con 0x7fcf18070b60 2024-03-23T11:19:04.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7fcf14c00640 1 -- 172.21.15.62:0/2315655800 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf040147c0 con 0x7fcf18070b60 2024-03-23T11:19:04.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7fcf14c00640 1 -- 172.21.15.62:0/2315655800 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf04014b50 con 0x7fcf18070b60 2024-03-23T11:19:04.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7f3498c00640 1 -- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940d1600 msgr2=0x7f34940b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7f3498c00640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940d1600 0x7f34940b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7f3498c00640 1 -- 172.21.15.62:0/1763918751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494162620 msgr2=0x7f34940b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7f3498c00640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494162620 0x7f34940b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2315655800 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf180cbcc0 con 0x7fcf18070b60 2024-03-23T11:19:04.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2315655800 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf180c8920 con 0x7fcf18070b60 2024-03-23T11:19:04.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7f2e92000640 1 -- 172.21.15.62:0/948692474 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e8c032070 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7fa34fe00640 1 -- 172.21.15.62:0/3734146765 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa348042480 con 0x7fa3580cd690 2024-03-23T11:19:04.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.020+0000 7f3498c00640 1 -- 172.21.15.62:0/1763918751 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3484009b30 con 0x7f34940d0a60 2024-03-23T11:19:04.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.021+0000 7f3498c00640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 0x7f349414ad50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f348c00e870 tx=0x7f348c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.021+0000 7f2e99400640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2e7c06a7f0 0x7f2e7c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.021+0000 7fc640c00640 1 -- 172.21.15.62:0/1205734320 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc62c053700 con 0x7fc6440b4190 2024-03-23T11:19:04.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.021+0000 7f3490c00640 1 -- 172.21.15.62:0/1763918751 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c00ee60 con 0x7f34940d0a60 2024-03-23T11:19:04.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.021+0000 7f3490c00640 1 -- 172.21.15.62:0/1763918751 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348c00c940 con 0x7f34940d0a60 2024-03-23T11:19:04.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.022+0000 7f3490c00640 1 -- 172.21.15.62:0/1763918751 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c00ccb0 con 0x7f34940d0a60 2024-03-23T11:19:04.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.022+0000 7f3493e00640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940d1600 0x7f34940b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:04.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.022+0000 7f349b600640 1 -- 172.21.15.62:0/1763918751 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349414bcd0 con 0x7f34940d0a60 2024-03-23T11:19:04.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.022+0000 7fa34fe00640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa33806a7a0 0x7fa33806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.022+0000 7fa34fe00640 1 -- 172.21.15.62:0/3734146765 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa348032070 con 0x7fa3580cd690 2024-03-23T11:19:04.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.023+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2315655800 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1806d010 con 0x7fcf18070b60 2024-03-23T11:19:04.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.023+0000 7f2e99400640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2e7c06a7f0 0x7f2e7c06ccb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c1575a0 tx=0x7f2e88009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.024+0000 7fa357400640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa33806a7a0 0x7fa33806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.022+0000 7f349b600640 1 -- 172.21.15.62:0/1763918751 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349414b290 con 0x7f34940d0a60 2024-03-23T11:19:04.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.023+0000 7f349b600640 1 -- 172.21.15.62:0/1763918751 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34940672c0 con 0x7f34940d0a60 2024-03-23T11:19:04.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.025+0000 7fa357400640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa33806a7a0 0x7fa33806cc60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa3580cbb30 tx=0x7fa344007c60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.024+0000 7fcf14c00640 1 -- 172.21.15.62:0/2315655800 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fcf04014cb0 con 0x7fcf18070b60 2024-03-23T11:19:04.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.025+0000 7fcf14c00640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcef806a7a0 0x7fcef806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.025+0000 7fcf14c00640 1 -- 172.21.15.62:0/2315655800 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf04088bf0 con 0x7fcf18070b60 2024-03-23T11:19:04.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.026+0000 7fcf1d200640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcef806a7a0 0x7fcef806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.028+0000 7f3490c00640 1 -- 172.21.15.62:0/1763918751 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f348c013470 con 0x7f34940d0a60 2024-03-23T11:19:04.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.028+0000 7f2e92000640 1 -- 172.21.15.62:0/948692474 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e8c0768a0 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.029+0000 7fcf1d200640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcef806a7a0 0x7fcef806cc60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcf10034000 tx=0x7fcf10007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.030+0000 7f3490c00640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347006a870 0x7f347006cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:04.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.030+0000 7f3490c00640 1 -- 172.21.15.62:0/1763918751 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f348c0894a0 con 0x7f34940d0a60 2024-03-23T11:19:04.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.033+0000 7f3493e00640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347006a870 0x7f347006cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:04.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.034+0000 7f3493e00640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347006a870 0x7f347006cd30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3484035000 tx=0x7f3484030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:04.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.039+0000 7fa34fe00640 1 -- 172.21.15.62:0/3734146765 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa3480774c0 con 0x7fa3580cd690 2024-03-23T11:19:04.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.041+0000 7fcf14c00640 1 -- 172.21.15.62:0/2315655800 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf1806d010 con 0x7fcf18070b60 2024-03-23T11:19:04.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.045+0000 7f3490c00640 1 -- 172.21.15.62:0/1763918751 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f348c0549e0 con 0x7f34940d0a60 2024-03-23T11:19:04.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.050+0000 7fcf40c00640 1 -- 172.21.15.62:0/3497912396 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf3c0ba580 con 0x7fcf44159590 2024-03-23T11:19:04.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.442+0000 7fc30f200640 1 -- 172.21.15.62:0/4189708919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc308032390 con 0x7fc3080e7bb0 2024-03-23T11:19:04.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.443+0000 7fc306000640 1 -- 172.21.15.62:0/4189708919 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc308032390 con 0x7fc3080e7bb0 2024-03-23T11:19:04.443 INFO:teuthology.orchestra.run.smithi062.stdout:60129542149 2024-03-23T11:19:04.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.444+0000 7fc64b000640 1 -- 172.21.15.62:0/1205734320 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc64415de80 con 0x7fc6440b4190 2024-03-23T11:19:04.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.445+0000 7f2e9b200640 1 -- 172.21.15.62:0/948692474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2e740057d0 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.445+0000 7fb0aca00640 1 -- 172.21.15.62:0/2279030274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb0840050c0 con 0x7fb0a00df3f0 2024-03-23T11:19:04.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.445+0000 7fc640c00640 1 -- 172.21.15.62:0/1205734320 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc64415de80 con 0x7fc6440b4190 2024-03-23T11:19:04.447 INFO:teuthology.orchestra.run.smithi062.stdout:60129542147 2024-03-23T11:19:04.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.446+0000 7fb09b400640 1 -- 172.21.15.62:0/2279030274 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb0a80f33a0 con 0x7fb0a00df3f0 2024-03-23T11:19:04.447 INFO:teuthology.orchestra.run.smithi062.stdout:60129542148 2024-03-23T11:19:04.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.446+0000 7f2e92000640 1 -- 172.21.15.62:0/948692474 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e8c079ec0 con 0x7f2e9c06b3f0 2024-03-23T11:19:04.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.449+0000 7fb0aca00640 1 -- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb08806a8a0 msgr2=0x7fb08806cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fb0aca00640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb08806a8a0 0x7fb08806cd60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00dec00 tx=0x7fb09c009240 comp rx=0 tx=0).stop 2024-03-23T11:19:04.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fb0aca00640 1 -- 172.21.15.62:0/2279030274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0a00df3f0 msgr2=0x7fb0a00d16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fb0aca00640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0a00df3f0 0x7fb0a00d16d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb0a804f2a0 tx=0x7fb0a8067fd0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.447+0000 7fc30f200640 1 -- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc2f006a7f0 msgr2=0x7fc2f006ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.447+0000 7fc30f200640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc2f006a7f0 0x7fc2f006ccb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009b60 tx=0x7fc2fc009190 comp rx=0 tx=0).stop 2024-03-23T11:19:04.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.447+0000 7fc30f200640 1 -- 172.21.15.62:0/4189708919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3080e7bb0 msgr2=0x7fc30804fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.447+0000 7fc30f200640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3080e7bb0 0x7fc30804fe50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc31004f2a0 tx=0x7fc310067fd0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.448+0000 7fc30f200640 1 -- 172.21.15.62:0/4189708919 shutdown_connections 2024-03-23T11:19:04.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.448+0000 7fc30f200640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc2f006a7f0 0x7fc2f006ccb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.448+0000 7fc30f200640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3080e7bb0 0x7fc30804fe50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.448+0000 7fc30f200640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc308055880 0x7fc30804f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.448+0000 7fc30f200640 1 --2- 172.21.15.62:0/4189708919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc308052e70 0x7fc308032670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.448+0000 7fc30f200640 1 -- 172.21.15.62:0/4189708919 >> 172.21.15.62:0/4189708919 conn(0x7fc308053730 msgr2=0x7fc3080374b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.448+0000 7fc30f200640 1 -- 172.21.15.62:0/4189708919 shutdown_connections 2024-03-23T11:19:04.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.448+0000 7fc30f200640 1 -- 172.21.15.62:0/4189708919 wait complete. 2024-03-23T11:19:04.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.449+0000 7fc64b000640 1 -- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc62006a7d0 msgr2=0x7fc62006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.449+0000 7fc64b000640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc62006a7d0 0x7fc62006cc90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc634035000 tx=0x7fc63402f000 comp rx=0 tx=0).stop 2024-03-23T11:19:04.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.449+0000 7fc64b000640 1 -- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6440b4190 msgr2=0x7fc6441fc5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.449+0000 7fc64b000640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6440b4190 0x7fc6441fc5a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00c750 tx=0x7fc62c00cc10 comp rx=0 tx=0).stop 2024-03-23T11:19:04.452 INFO:teuthology.orchestra.run.smithi062.stdout:60129542148 2024-03-23T11:19:04.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fc64b000640 1 -- 172.21.15.62:0/1205734320 shutdown_connections 2024-03-23T11:19:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fb0aca00640 1 -- 172.21.15.62:0/2279030274 shutdown_connections 2024-03-23T11:19:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fb0aca00640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb08806a8a0 0x7fb08806cd60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fb0aca00640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0a00df3f0 0x7fb0a00d16d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fb0aca00640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0a0054240 0x7fb0a0035b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fb0aca00640 1 --2- 172.21.15.62:0/2279030274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0a004fa60 0x7fb0a0035640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fb0aca00640 1 -- 172.21.15.62:0/2279030274 >> 172.21.15.62:0/2279030274 conn(0x7fb0a0009f80 msgr2=0x7fb0a0036a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fc64b000640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc62006a7d0 0x7fc62006cc90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fc64b000640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6440cd340 0x7fc6440b7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fc64b000640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6440d3130 0x7fc6440b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fc64b000640 1 --2- 172.21.15.62:0/1205734320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6440b4190 0x7fc6441fc5a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7fc64b000640 1 -- 172.21.15.62:0/1205734320 >> 172.21.15.62:0/1205734320 conn(0x7fc644089f60 msgr2=0x7fc6440d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7f2e9b200640 1 -- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2e7c06a7f0 msgr2=0x7f2e7c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.450+0000 7f2e9b200640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2e7c06a7f0 0x7f2e7c06ccb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c1575a0 tx=0x7f2e88009040 comp rx=0 tx=0).stop 2024-03-23T11:19:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7f2e9b200640 1 -- 172.21.15.62:0/948692474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e9c06b3f0 msgr2=0x7f2e9c0ca9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7f2e9b200640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e9c06b3f0 0x7f2e9c0ca9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009870 tx=0x7f2e8c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7fc64b000640 1 -- 172.21.15.62:0/1205734320 shutdown_connections 2024-03-23T11:19:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7fc64b000640 1 -- 172.21.15.62:0/1205734320 wait complete. 2024-03-23T11:19:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7fb0aca00640 1 -- 172.21.15.62:0/2279030274 shutdown_connections 2024-03-23T11:19:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7fb0aca00640 1 -- 172.21.15.62:0/2279030274 wait complete. 2024-03-23T11:19:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7f2e9b200640 1 -- 172.21.15.62:0/948692474 shutdown_connections 2024-03-23T11:19:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7f2e9b200640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2e7c06a7f0 0x7f2e7c06ccb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7f2e9b200640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e9c15c990 0x7f2e9c156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7f2e9b200640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c0cefa0 0x7f2e9c0b6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7f2e9b200640 1 --2- 172.21.15.62:0/948692474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e9c06b3f0 0x7f2e9c0ca9c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.451+0000 7f2e9b200640 1 -- 172.21.15.62:0/948692474 >> 172.21.15.62:0/948692474 conn(0x7f2e9c089f60 msgr2=0x7f2e9c0d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.452+0000 7f2e9b200640 1 -- 172.21.15.62:0/948692474 shutdown_connections 2024-03-23T11:19:04.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.452+0000 7f2e9b200640 1 -- 172.21.15.62:0/948692474 wait complete. 2024-03-23T11:19:04.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.474+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2315655800 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fcf1806d010 con 0x7fcf18070b60 2024-03-23T11:19:04.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.475+0000 7fcf14c00640 1 -- 172.21.15.62:0/2315655800 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcf1806d010 con 0x7fcf18070b60 2024-03-23T11:19:04.476 INFO:teuthology.orchestra.run.smithi062.stdout:60129542148 2024-03-23T11:19:04.476 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.1 2024-03-23T11:19:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.477+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcef806a7a0 msgr2=0x7fcef806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.478+0000 7fcf1fc00640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcef806a7a0 0x7fcef806cc60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcf10034000 tx=0x7fcf10007000 comp rx=0 tx=0).stop 2024-03-23T11:19:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.478+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf18070b60 msgr2=0x7fcf180d2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.478+0000 7fcf1fc00640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf18070b60 0x7fcf180d2e10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcf180ce280 tx=0x7fcf0400f670 comp rx=0 tx=0).stop 2024-03-23T11:19:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.478+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2315655800 shutdown_connections 2024-03-23T11:19:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.478+0000 7fcf1fc00640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcef806a7a0 0x7fcef806cc60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.478+0000 7fcf1fc00640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf18070b60 0x7fcf180d2e10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.478+0000 7fcf1fc00640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf180706e0 0x7fcf180d28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.478+0000 7fcf1fc00640 1 --2- 172.21.15.62:0/2315655800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcefc0039c0 0x7fcf180c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.478+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2315655800 >> 172.21.15.62:0/2315655800 conn(0x7fcf18071570 msgr2=0x7fcf180b6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.478+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2315655800 shutdown_connections 2024-03-23T11:19:04.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.478+0000 7fcf1fc00640 1 -- 172.21.15.62:0/2315655800 wait complete. 2024-03-23T11:19:04.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.479+0000 7fa35f000640 1 -- 172.21.15.62:0/3734146765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa3580672c0 con 0x7fa3580cd690 2024-03-23T11:19:04.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.2 2024-03-23T11:19:04.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.480+0000 7fa34fe00640 1 -- 172.21.15.62:0/3734146765 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa34807aae0 con 0x7fa3580cd690 2024-03-23T11:19:04.480 INFO:teuthology.orchestra.run.smithi062.stdout:60129542148 2024-03-23T11:19:04.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.0 2024-03-23T11:19:04.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.482+0000 7f349b600640 1 -- 172.21.15.62:0/1763918751 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f3494163e20 con 0x7f34940d0a60 2024-03-23T11:19:04.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7fcf4bc00640 1 -- 172.21.15.62:0/3497912396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fcf4415b2c0 con 0x7fcf44159590 2024-03-23T11:19:04.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7fa35f000640 1 -- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa33806a7a0 msgr2=0x7fa33806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7fa35f000640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa33806a7a0 0x7fa33806cc60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa3580cbb30 tx=0x7fa344007c60 comp rx=0 tx=0).stop 2024-03-23T11:19:04.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7fa35f000640 1 -- 172.21.15.62:0/3734146765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3580cd690 msgr2=0x7fa3581f18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7fa35f000640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3580cd690 0x7fa3581f18f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa348009990 tx=0x7fa3480046c0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7f3490c00640 1 -- 172.21.15.62:0/1763918751 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f348c058000 con 0x7f34940d0a60 2024-03-23T11:19:04.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7fa35f000640 1 -- 172.21.15.62:0/3734146765 shutdown_connections 2024-03-23T11:19:04.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7fa35f000640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa33806a7a0 0x7fa33806cc60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7fa35f000640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa35815cce0 0x7fa35815ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7fa35f000640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa35815c900 0x7fa35815d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7fa35f000640 1 --2- 172.21.15.62:0/3734146765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3580cd690 0x7fa3581f18f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.483+0000 7fa35f000640 1 -- 172.21.15.62:0/3734146765 >> 172.21.15.62:0/3734146765 conn(0x7fa358071390 msgr2=0x7fa3580b6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.485+0000 7fcf40c00640 1 -- 172.21.15.62:0/3497912396 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcf4415b2c0 con 0x7fcf44159590 2024-03-23T11:19:04.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.485+0000 7fa35f000640 1 -- 172.21.15.62:0/3734146765 shutdown_connections 2024-03-23T11:19:04.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.485+0000 7fa35f000640 1 -- 172.21.15.62:0/3734146765 wait complete. 2024-03-23T11:19:04.485 INFO:teuthology.orchestra.run.smithi062.stdout:60129542148 2024-03-23T11:19:04.486 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.4 2024-03-23T11:19:04.486 INFO:teuthology.orchestra.run.smithi062.stdout:60129542148 2024-03-23T11:19:04.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 -- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347006a870 msgr2=0x7f347006cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347006a870 0x7f347006cd30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3484035000 tx=0x7f3484030000 comp rx=0 tx=0).stop 2024-03-23T11:19:04.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 -- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 msgr2=0x7f349414ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 0x7f349414ad50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f348c00e870 tx=0x7f348c00ed30 comp rx=0 tx=0).stop 2024-03-23T11:19:04.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 -- 172.21.15.62:0/1763918751 shutdown_connections 2024-03-23T11:19:04.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347006a870 0x7f347006cd30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494162620 0x7f34940b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940d1600 0x7f34940b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 --2- 172.21.15.62:0/1763918751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34940d0a60 0x7f349414ad50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 -- 172.21.15.62:0/1763918751 >> 172.21.15.62:0/1763918751 conn(0x7f3494071390 msgr2=0x7f3494162a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 -- 172.21.15.62:0/1763918751 shutdown_connections 2024-03-23T11:19:04.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.486+0000 7f349b600640 1 -- 172.21.15.62:0/1763918751 wait complete. 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.487+0000 7fcf4bc00640 1 -- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf2406a7f0 msgr2=0x7fcf2406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.487+0000 7fcf4bc00640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf2406a7f0 0x7fcf2406ccb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcf34037000 tx=0x7fcf34032000 comp rx=0 tx=0).stop 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.487+0000 7fcf4bc00640 1 -- 172.21.15.62:0/3497912396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf44159590 msgr2=0x7fcf4415b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.487+0000 7fcf4bc00640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf44159590 0x7fcf4415b5a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00b9e0 tx=0x7fcf3c00bea0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.487+0000 7fcf4bc00640 1 -- 172.21.15.62:0/3497912396 shutdown_connections 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.487+0000 7fcf4bc00640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf2406a7f0 0x7fcf2406ccb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.487+0000 7fcf4bc00640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf44159590 0x7fcf4415b5a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.487+0000 7fcf4bc00640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf441532d0 0x7fcf44154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.487+0000 7fcf4bc00640 1 --2- 172.21.15.62:0/3497912396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf44152ef0 0x7fcf441423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.487+0000 7fcf4bc00640 1 -- 172.21.15.62:0/3497912396 >> 172.21.15.62:0/3497912396 conn(0x7fcf44069c60 msgr2=0x7fcf44152100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.488+0000 7fcf4bc00640 1 -- 172.21.15.62:0/3497912396 shutdown_connections 2024-03-23T11:19:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:04.488+0000 7fcf4bc00640 1 -- 172.21.15.62:0/3497912396 wait complete. 2024-03-23T11:19:04.500 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.5 2024-03-23T11:19:04.504 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.6 2024-03-23T11:19:04.511 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.7 2024-03-23T11:19:04.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.3 2024-03-23T11:19:05.477 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-23T11:19:05.481 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-23T11:19:05.482 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-23T11:19:05.487 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-23T11:19:05.501 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-23T11:19:05.505 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-23T11:19:05.512 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-23T11:19:05.513 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-23T11:19:05.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.757+0000 7f41be200640 1 Processor -- start 2024-03-23T11:19:05.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.757+0000 7f41be200640 1 -- start start 2024-03-23T11:19:05.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.759+0000 7f41be200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41b80d2260 0x7f41b80b5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.759+0000 7f41be200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b8165060 con 0x7f41b80b4d80 2024-03-23T11:19:05.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.759+0000 7f41be200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80cda40 con 0x7f41b814aea0 2024-03-23T11:19:05.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.759+0000 7f41be200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b8164df0 con 0x7f41b80d2260 2024-03-23T11:19:05.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.760+0000 7f41b7400640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f41b814aea0 0x7f41b81654f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41694/0 (socket says 172.21.15.62:41694) 2024-03-23T11:19:05.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.760+0000 7f41b7400640 1 -- 172.21.15.62:0/3125769670 learned_addr learned my addr 172.21.15.62:0/3125769670 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:05.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.760+0000 7f41b6a00640 1 --2- 172.21.15.62:0/3125769670 >> v2:172.21.15.62:3300/0 conn(0x7f41b80d2260 0x7f41b80b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.762+0000 7f41b6a00640 1 -- 172.21.15.62:0/3125769670 >> v1:172.21.15.120:6789/0 conn(0x7f41b814aea0 legacy=0x7f41b81654f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.762+0000 7f41b6a00640 1 -- 172.21.15.62:0/3125769670 >> v1:172.21.15.62:6789/0 conn(0x7f41b80b4d80 legacy=0x7f41b80cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.762+0000 7f41b6a00640 1 -- 172.21.15.62:0/3125769670 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b81616f0 con 0x7f41b80d2260 2024-03-23T11:19:05.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.764+0000 7f41b6a00640 1 --2- 172.21.15.62:0/3125769670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b80d2260 0x7f41b80b5740 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009a00 tx=0x7f41a802ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34aa2709cb2a2036 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.764+0000 7f41b6000640 1 -- 172.21.15.62:0/3125769670 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a802d640 con 0x7f41b80d2260 2024-03-23T11:19:05.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.764+0000 7f41b6000640 1 -- 172.21.15.62:0/3125769670 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a802d7a0 con 0x7f41b80d2260 2024-03-23T11:19:05.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.764+0000 7f41b6000640 1 -- 172.21.15.62:0/3125769670 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a802db30 con 0x7f41b80d2260 2024-03-23T11:19:05.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.766+0000 7f41be200640 1 -- 172.21.15.62:0/3125769670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b80d2260 msgr2=0x7f41b80b5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.766+0000 7f41be200640 1 --2- 172.21.15.62:0/3125769670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b80d2260 0x7f41b80b5740 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009a00 tx=0x7f41a802ba90 comp rx=0 tx=0).stop 2024-03-23T11:19:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.766+0000 7f41be200640 1 -- 172.21.15.62:0/3125769670 shutdown_connections 2024-03-23T11:19:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.766+0000 7f41be200640 1 --2- 172.21.15.62:0/3125769670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b80d2260 0x7f41b80b5740 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.766+0000 7f41be200640 1 -- 172.21.15.62:0/3125769670 >> 172.21.15.62:0/3125769670 conn(0x7f41b8071570 msgr2=0x7f41b8071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.766+0000 7f41be200640 1 -- 172.21.15.62:0/3125769670 shutdown_connections 2024-03-23T11:19:05.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.766+0000 7f41be200640 1 -- 172.21.15.62:0/3125769670 wait complete. 2024-03-23T11:19:05.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.769+0000 7f41be200640 1 Processor -- start 2024-03-23T11:19:05.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.769+0000 7f41be200640 1 -- start start 2024-03-23T11:19:05.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.771+0000 7f41be200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b80b4d80 0x7f41b806dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.772+0000 7f41be200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41b80d2260 0x7f41b80c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.772+0000 7f41be200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b814aea0 0x7f41b80ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.772+0000 7f41b7400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b80b4d80 0x7f41b806dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.773+0000 7f41b7400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b80b4d80 0x7f41b806dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53308/0 (socket says 172.21.15.62:53308) 2024-03-23T11:19:05.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.773+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41b80d2260 0x7f41b80c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.773+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41b80d2260 0x7f41b80c8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:56124/0 (socket says 172.21.15.62:56124) 2024-03-23T11:19:05.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.774+0000 7f41b7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b814aea0 0x7f41b80ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.774+0000 7f41b7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b814aea0 0x7f41b80ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38022/0 (socket says 172.21.15.62:38022) 2024-03-23T11:19:05.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.774+0000 7f41b6a00640 1 -- 172.21.15.62:0/159301967 learned_addr learned my addr 172.21.15.62:0/159301967 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:05.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.774+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80c99c0 con 0x7f41b814aea0 2024-03-23T11:19:05.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.774+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80c8fc0 con 0x7f41b80b4d80 2024-03-23T11:19:05.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.774+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80c9c10 con 0x7f41b80d2260 2024-03-23T11:19:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.775+0000 7f41b7e00640 1 -- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41b80d2260 msgr2=0x7f41b80c8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.775+0000 7f41b7e00640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41b80d2260 0x7f41b80c8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.775+0000 7f41b7e00640 1 -- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b80b4d80 msgr2=0x7f41b806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.775+0000 7f41b7e00640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b80b4d80 0x7f41b806dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.775+0000 7f41b7e00640 1 -- 172.21.15.62:0/159301967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a8009400 con 0x7f41b814aea0 2024-03-23T11:19:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.776+0000 7f41b7e00640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b814aea0 0x7f41b80ca6f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f41b000c730 tx=0x7f41b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.777+0000 7f41b7400640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b80b4d80 0x7f41b806dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:05.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.777+0000 7f41af400640 1 -- 172.21.15.62:0/159301967 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0003b70 con 0x7f41b814aea0 2024-03-23T11:19:05.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.777+0000 7f41af400640 1 -- 172.21.15.62:0/159301967 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b0003cf0 con 0x7f41b814aea0 2024-03-23T11:19:05.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.777+0000 7f41af400640 1 -- 172.21.15.62:0/159301967 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0005310 con 0x7f41b814aea0 2024-03-23T11:19:05.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.777+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b80c9ef0 con 0x7f41b814aea0 2024-03-23T11:19:05.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.777+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b80cc570 con 0x7f41b814aea0 2024-03-23T11:19:05.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.778+0000 7f41b6a00640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41b80d2260 0x7f41b80c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:05.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.782+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b80ca410 con 0x7f41b814aea0 2024-03-23T11:19:05.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.782+0000 7f41af400640 1 -- 172.21.15.62:0/159301967 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f41b0005470 con 0x7f41b814aea0 2024-03-23T11:19:05.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.786+0000 7f41af400640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f419806a7a0 0x7f419806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.786+0000 7f41af400640 1 -- 172.21.15.62:0/159301967 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41b0088b20 con 0x7f41b814aea0 2024-03-23T11:19:05.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.787+0000 7f41b7400640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f419806a7a0 0x7f419806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.790+0000 7f36eb800640 1 Processor -- start 2024-03-23T11:19:05.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.792+0000 7f41b7400640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f419806a7a0 0x7f419806cc60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f41a4005880 tx=0x7f41a4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.791+0000 7f36eb800640 1 -- start start 2024-03-23T11:19:05.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.793+0000 7f36eb800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f36e40cf0e0 0x7f36e40d3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.794+0000 7f36e8e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f36e40cf0e0 0x7f36e40d3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.794+0000 7f36e8e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f36e40cf0e0 0x7f36e40d3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38026/0 (socket says 172.21.15.62:38026) 2024-03-23T11:19:05.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.794+0000 7f36e8e00640 1 -- 172.21.15.62:0/2000288939 learned_addr learned my addr 172.21.15.62:0/2000288939 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:05.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.794+0000 7f36eb800640 1 --2- 172.21.15.62:0/2000288939 >> v2:172.21.15.120:3300/0 conn(0x7f36e40cf4c0 0x7f36e40b5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.795+0000 7f36eb800640 1 -- 172.21.15.62:0/2000288939 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e40d1860 con 0x7f36e40b5ad0 2024-03-23T11:19:05.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.795+0000 7f36eb800640 1 -- 172.21.15.62:0/2000288939 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e40cd350 con 0x7f36e40cf0e0 2024-03-23T11:19:05.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.795+0000 7f36eb800640 1 -- 172.21.15.62:0/2000288939 --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e40d12a0 con 0x7f36e40cf4c0 2024-03-23T11:19:05.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.795+0000 7f36e3e00640 1 --2- 172.21.15.62:0/2000288939 >> v2:172.21.15.120:3300/0 conn(0x7f36e40cf4c0 0x7f36e40b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.796+0000 7f41af400640 1 -- 172.21.15.62:0/159301967 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f41b80ca410 con 0x7f41b814aea0 2024-03-23T11:19:05.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.797+0000 7f36e8e00640 1 -- 172.21.15.62:0/2000288939 >> v2:172.21.15.120:3300/0 conn(0x7f36e40cf4c0 msgr2=0x7f36e40b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:05.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.797+0000 7f36e8e00640 1 --2- 172.21.15.62:0/2000288939 >> v2:172.21.15.120:3300/0 conn(0x7f36e40cf4c0 0x7f36e40b5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.797+0000 7f36e8e00640 1 -- 172.21.15.62:0/2000288939 >> v1:172.21.15.120:6789/0 conn(0x7f36e40b5ad0 legacy=0x7f36e40cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.797+0000 7f36e8e00640 1 -- 172.21.15.62:0/2000288939 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e40d1580 con 0x7f36e40cf0e0 2024-03-23T11:19:05.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.797+0000 7f85f4a00640 1 Processor -- start 2024-03-23T11:19:05.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.797+0000 7f36e3e00640 1 --2- 172.21.15.62:0/2000288939 >> v2:172.21.15.120:3300/0 conn(0x7f36e40cf4c0 0x7f36e40b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:05.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.798+0000 7f85f4a00640 1 -- start start 2024-03-23T11:19:05.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.799+0000 7f85f4a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e80e12e0 0x7f85e803a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.799+0000 7f36e8e00640 1 --2- 172.21.15.62:0/2000288939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e40cf0e0 0x7f36e40d3060 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f36e40d26a0 tx=0x7f36d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3e59829bd7bec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.800+0000 7f85f4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f85e800a430 0x7f85e8055d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.800+0000 7f85f4a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e80558f0 con 0x7f85e800ab10 2024-03-23T11:19:05.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.800+0000 7f85f4a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e80e16c0 con 0x7f85e800a430 2024-03-23T11:19:05.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.800+0000 7f85f4a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e803a680 con 0x7f85e80e12e0 2024-03-23T11:19:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.800+0000 7f85eea00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f85e800ab10 0x7f85e8009d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41724/0 (socket says 172.21.15.62:41724) 2024-03-23T11:19:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.801+0000 7f85eea00640 1 -- 172.21.15.62:0/2981064787 learned_addr learned my addr 172.21.15.62:0/2981064787 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.801+0000 7f85ef400640 1 --2- 172.21.15.62:0/2981064787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e80e12e0 0x7f85e803a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.801+0000 7f36e3400640 1 -- 172.21.15.62:0/2000288939 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d0004230 con 0x7f36e40cf0e0 2024-03-23T11:19:05.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.801+0000 7f85efe00640 1 --2- 172.21.15.62:0/2981064787 >> v2:172.21.15.62:3300/0 conn(0x7f85e800a430 0x7f85e8055d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.801+0000 7f85ef400640 1 -- 172.21.15.62:0/2981064787 >> v2:172.21.15.62:3300/0 conn(0x7f85e800a430 msgr2=0x7f85e8055d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:05.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.801+0000 7f85ef400640 1 --2- 172.21.15.62:0/2981064787 >> v2:172.21.15.62:3300/0 conn(0x7f85e800a430 0x7f85e8055d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.801+0000 7f85ee000640 1 -- 172.21.15.62:0/2981064787 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269083404 0 0) 0x7f85e80558f0 con 0x7f85e800ab10 2024-03-23T11:19:05.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.801+0000 7f85ef400640 1 -- 172.21.15.62:0/2981064787 >> v1:172.21.15.120:6789/0 conn(0x7f85e800ab10 legacy=0x7f85e8009d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.802+0000 7f85ef400640 1 -- 172.21.15.62:0/2981064787 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e8054ce0 con 0x7f85e80e12e0 2024-03-23T11:19:05.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.801+0000 7f36e3400640 1 -- 172.21.15.62:0/2000288939 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d0004390 con 0x7f36e40cf0e0 2024-03-23T11:19:05.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.802+0000 7f36e3400640 1 -- 172.21.15.62:0/2000288939 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d0003810 con 0x7f36e40cf0e0 2024-03-23T11:19:05.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.803+0000 7f36eb800640 1 -- 172.21.15.62:0/2000288939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e40cf0e0 msgr2=0x7f36e40d3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.803+0000 7f36eb800640 1 --2- 172.21.15.62:0/2000288939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e40cf0e0 0x7f36e40d3060 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f36e40d26a0 tx=0x7f36d002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.803+0000 7f36eb800640 1 -- 172.21.15.62:0/2000288939 shutdown_connections 2024-03-23T11:19:05.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.804+0000 7f36eb800640 1 --2- 172.21.15.62:0/2000288939 >> v2:172.21.15.120:3300/0 conn(0x7f36e40cf4c0 0x7f36e40b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.804+0000 7f36eb800640 1 --2- 172.21.15.62:0/2000288939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e40cf0e0 0x7f36e40d3060 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.804+0000 7f85ef400640 1 --2- 172.21.15.62:0/2981064787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e80e12e0 0x7f85e803a140 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f85dc0097a0 tx=0x7f85dc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=473cefa946421185 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.804+0000 7f36eb800640 1 -- 172.21.15.62:0/2000288939 >> 172.21.15.62:0/2000288939 conn(0x7f36e4089f60 msgr2=0x7f36e40d2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:05.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.804+0000 7f36eb800640 1 -- 172.21.15.62:0/2000288939 shutdown_connections 2024-03-23T11:19:05.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.804+0000 7f85ee000640 1 -- 172.21.15.62:0/2981064787 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc031830 con 0x7f85e80e12e0 2024-03-23T11:19:05.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.804+0000 7f85ee000640 1 -- 172.21.15.62:0/2981064787 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc031990 con 0x7f85e80e12e0 2024-03-23T11:19:05.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.804+0000 7f85ee000640 1 -- 172.21.15.62:0/2981064787 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc031830 con 0x7f85e80e12e0 2024-03-23T11:19:05.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.804+0000 7f36eb800640 1 -- 172.21.15.62:0/2000288939 wait complete. 2024-03-23T11:19:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.806+0000 7f85f4a00640 1 -- 172.21.15.62:0/2981064787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e80e12e0 msgr2=0x7f85e803a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.806+0000 7f85f4a00640 1 --2- 172.21.15.62:0/2981064787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e80e12e0 0x7f85e803a140 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f85dc0097a0 tx=0x7f85dc02b8d0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.806+0000 7f85f4a00640 1 -- 172.21.15.62:0/2981064787 shutdown_connections 2024-03-23T11:19:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.806+0000 7f85f4a00640 1 --2- 172.21.15.62:0/2981064787 >> v2:172.21.15.62:3300/0 conn(0x7f85e800a430 0x7f85e8055d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.806+0000 7f85f4a00640 1 --2- 172.21.15.62:0/2981064787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e80e12e0 0x7f85e803a140 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.806+0000 7f85f4a00640 1 -- 172.21.15.62:0/2981064787 >> 172.21.15.62:0/2981064787 conn(0x7f85e8056ca0 msgr2=0x7f85e80570b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.806+0000 7f85f4a00640 1 -- 172.21.15.62:0/2981064787 shutdown_connections 2024-03-23T11:19:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.806+0000 7f85f4a00640 1 -- 172.21.15.62:0/2981064787 wait complete. 2024-03-23T11:19:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.808+0000 7f36eb800640 1 Processor -- start 2024-03-23T11:19:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.809+0000 7f36eb800640 1 -- start start 2024-03-23T11:19:05.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.810+0000 7f85f4a00640 1 Processor -- start 2024-03-23T11:19:05.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.810+0000 7f85f4a00640 1 -- start start 2024-03-23T11:19:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.811+0000 7f36eb800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e40b5ad0 0x7f36e415acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.811+0000 7f36eb800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e40cf0e0 0x7f36e40b7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.812+0000 7f36eb800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40cf4c0 0x7f36e40b7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.812+0000 7f85f4a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e800a430 0x7f85e80de060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f85f4a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e800ab10 0x7f85e8176d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f36eb800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e40cb160 con 0x7f36e40b5ad0 2024-03-23T11:19:05.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f36eb800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e40b0220 con 0x7f36e40cf4c0 2024-03-23T11:19:05.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f85f4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85e80e12e0 0x7f85e80534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f36e9800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40cf4c0 0x7f36e40b7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f36e9800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40cf4c0 0x7f36e40b7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53330/0 (socket says 172.21.15.62:53330) 2024-03-23T11:19:05.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f36e9800640 1 -- 172.21.15.62:0/1985558274 learned_addr learned my addr 172.21.15.62:0/1985558274 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:05.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f36e3e00640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e40cf0e0 0x7f36e40b7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f85efe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85e80e12e0 0x7f85e80534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f36eb800640 1 -- 172.21.15.62:0/1985558274 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e40b0500 con 0x7f36e40cf0e0 2024-03-23T11:19:05.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f85efe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85e80e12e0 0x7f85e80534d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38052/0 (socket says 172.21.15.62:38052) 2024-03-23T11:19:05.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.814+0000 7f85f4a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e8050360 con 0x7f85e80e12e0 2024-03-23T11:19:05.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.814+0000 7f85f4a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e804c660 con 0x7f85e800ab10 2024-03-23T11:19:05.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.814+0000 7f85f4a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e8051b60 con 0x7f85e800a430 2024-03-23T11:19:05.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.813+0000 7f36e8e00640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e40b5ad0 0x7f36e415acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.814+0000 7f85efe00640 1 -- 172.21.15.62:0/1237306739 learned_addr learned my addr 172.21.15.62:0/1237306739 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:05.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.814+0000 7f85ef400640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e800a430 0x7f85e80de060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.814+0000 7f85eea00640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e800ab10 0x7f85e8176d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.815+0000 7f85efe00640 1 -- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e800a430 msgr2=0x7f85e80de060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.815+0000 7f85efe00640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e800a430 0x7f85e80de060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.815+0000 7f36e3e00640 1 -- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40cf4c0 msgr2=0x7f36e40b7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.815+0000 7f36e3e00640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40cf4c0 0x7f36e40b7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.815+0000 7f36e3e00640 1 -- 172.21.15.62:0/1985558274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e40b5ad0 msgr2=0x7f36e415acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.815+0000 7f36e3e00640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e40b5ad0 0x7f36e415acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.815+0000 7f36e3e00640 1 -- 172.21.15.62:0/1985558274 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8009400 con 0x7f36e40cf0e0 2024-03-23T11:19:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.816+0000 7f85efe00640 1 -- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e800ab10 msgr2=0x7f85e8176d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.816+0000 7f85efe00640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e800ab10 0x7f85e8176d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.816+0000 7f85efe00640 1 -- 172.21.15.62:0/1237306739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc009400 con 0x7f85e80e12e0 2024-03-23T11:19:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.816+0000 7f85eea00640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e800ab10 0x7f85e8176d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:05.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.816+0000 7f85efe00640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85e80e12e0 0x7f85e80534d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f85f004f2a0 tx=0x7f85f0067690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.816+0000 7f36e3e00640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e40cf0e0 0x7f36e40b7440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f36d800e870 tx=0x7f36d800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.817+0000 7f36e0c00640 1 -- 172.21.15.62:0/1985558274 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d800c940 con 0x7f36e40cf0e0 2024-03-23T11:19:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.817+0000 7f85e3400640 1 -- 172.21.15.62:0/1237306739 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f00717a0 con 0x7f85e80e12e0 2024-03-23T11:19:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.817+0000 7f85e3400640 1 -- 172.21.15.62:0/1237306739 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f0071900 con 0x7f85e80e12e0 2024-03-23T11:19:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.817+0000 7f85e3400640 1 -- 172.21.15.62:0/1237306739 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0071c70 con 0x7f85e80e12e0 2024-03-23T11:19:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.817+0000 7f36eb800640 1 -- 172.21.15.62:0/1985558274 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d0009460 con 0x7f36e40cf0e0 2024-03-23T11:19:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.817+0000 7f36eb800640 1 -- 172.21.15.62:0/1985558274 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e40b0ac0 con 0x7f36e40cf0e0 2024-03-23T11:19:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.818+0000 7f85f4a00640 1 -- 172.21.15.62:0/1237306739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e804ba50 con 0x7f85e80e12e0 2024-03-23T11:19:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.819+0000 7f85f4a00640 1 -- 172.21.15.62:0/1237306739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e804c010 con 0x7f85e80e12e0 2024-03-23T11:19:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.817+0000 7f36e0c00640 1 -- 172.21.15.62:0/1985558274 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d800caa0 con 0x7f36e40cf0e0 2024-03-23T11:19:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.817+0000 7f36e0c00640 1 -- 172.21.15.62:0/1985558274 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d800c940 con 0x7f36e40cf0e0 2024-03-23T11:19:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.820+0000 7f36eb800640 1 -- 172.21.15.62:0/1985558274 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c0005250 con 0x7f36e40cf0e0 2024-03-23T11:19:05.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.822+0000 7f36e0c00640 1 -- 172.21.15.62:0/1985558274 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f36d800cc10 con 0x7f36e40cf0e0 2024-03-23T11:19:05.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.824+0000 7f36e0c00640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a800 0x7f36c406ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.825+0000 7f36e0c00640 1 -- 172.21.15.62:0/1985558274 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36d8088a10 con 0x7f36e40cf0e0 2024-03-23T11:19:05.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.825+0000 7f36e8e00640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a800 0x7f36c406ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.824+0000 7f85e3400640 1 -- 172.21.15.62:0/1237306739 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f85f009b3f0 con 0x7f85e80e12e0 2024-03-23T11:19:05.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.824+0000 7f85e0c00640 1 -- 172.21.15.62:0/1237306739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c4005250 con 0x7f85e80e12e0 2024-03-23T11:19:05.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.825+0000 7f85e3400640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f85d006ebe0 0x7f85d00710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.825+0000 7f85e3400640 1 -- 172.21.15.62:0/1237306739 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85f0101d80 con 0x7f85e80e12e0 2024-03-23T11:19:05.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.827+0000 7f85ef400640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f85d006ebe0 0x7f85d00710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.827+0000 7f36e8e00640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a800 0x7f36c406ccc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f36d0035000 tx=0x7f36d0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.828+0000 7f85ef400640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f85d006ebe0 0x7f85d00710a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f85dc036000 tx=0x7f85dc03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.835+0000 7f85e3400640 1 -- 172.21.15.62:0/1237306739 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85f00cd240 con 0x7f85e80e12e0 2024-03-23T11:19:05.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.838+0000 7f36e0c00640 1 -- 172.21.15.62:0/1985558274 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f36d8053ec0 con 0x7f36e40cf0e0 2024-03-23T11:19:05.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.873+0000 7f7be6200640 1 Processor -- start 2024-03-23T11:19:05.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.873+0000 7f7be6200640 1 -- start start 2024-03-23T11:19:05.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.875+0000 7f7be6200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be015cba0 0x7f7be00d3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.875+0000 7f7be6200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7be00b4880 0x7f7be00cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.876+0000 7f7be6200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be00d3600 con 0x7f7be00ce8e0 2024-03-23T11:19:05.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.876+0000 7f7be6200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be015dc10 con 0x7f7be00b4880 2024-03-23T11:19:05.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.876+0000 7f7be6200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be015d270 con 0x7f7be015cba0 2024-03-23T11:19:05.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.876+0000 7f7be4e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be015cba0 0x7f7be00d3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.876+0000 7f7be4e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be015cba0 0x7f7be00d3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:56160/0 (socket says 172.21.15.62:56160) 2024-03-23T11:19:05.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.876+0000 7f7bdfe00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f7be00ce8e0 0x7f7be00b4170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41734/0 (socket says 172.21.15.62:41734) 2024-03-23T11:19:05.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.876+0000 7f7bdfe00640 1 -- 172.21.15.62:0/3062270872 learned_addr learned my addr 172.21.15.62:0/3062270872 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:05.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.877+0000 7f7be5800640 1 --2- 172.21.15.62:0/3062270872 >> v2:172.21.15.120:3300/0 conn(0x7f7be00b4880 0x7f7be00cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.877+0000 7fa7f8a00640 1 Processor -- start 2024-03-23T11:19:05.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.877+0000 7fa7f8a00640 1 -- start start 2024-03-23T11:19:05.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.877+0000 7f7be4e00640 1 -- 172.21.15.62:0/3062270872 >> v2:172.21.15.120:3300/0 conn(0x7f7be00b4880 msgr2=0x7f7be00cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:05.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.877+0000 7f7be4e00640 1 --2- 172.21.15.62:0/3062270872 >> v2:172.21.15.120:3300/0 conn(0x7f7be00b4880 0x7f7be00cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.878+0000 7f7be4e00640 1 -- 172.21.15.62:0/3062270872 >> v1:172.21.15.120:6789/0 conn(0x7f7be00ce8e0 legacy=0x7f7be00b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.879+0000 7fa7f8a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa7ec0e32a0 0x7fa7ec052c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.878+0000 7f7be4e00640 1 -- 172.21.15.62:0/3062270872 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be015e030 con 0x7f7be015cba0 2024-03-23T11:19:05.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.879+0000 7fa7f8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7ec04ff50 0x7fa7ec037b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.880+0000 7f1efbe00640 1 Processor -- start 2024-03-23T11:19:05.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.880+0000 7fa7f8a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ec0e3680 con 0x7fa7ec053160 2024-03-23T11:19:05.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.880+0000 7fa7f8a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7ec053670 con 0x7fa7ec0e32a0 2024-03-23T11:19:05.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.880+0000 7fa7f8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec053950 con 0x7fa7ec04ff50 2024-03-23T11:19:05.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.880+0000 7f1efbe00640 1 -- start start 2024-03-23T11:19:05.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.880+0000 7f7be4e00640 1 --2- 172.21.15.62:0/3062270872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be015cba0 0x7f7be00d3020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009d90 tx=0x7f7bd802bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbf2f515a7a7e857 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.880+0000 7fa7f1e00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fa7ec053160 0x7fa7ec04f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41738/0 (socket says 172.21.15.62:41738) 2024-03-23T11:19:05.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.880+0000 7fa7f1e00640 1 -- 172.21.15.62:0/1537242875 learned_addr learned my addr 172.21.15.62:0/1537242875 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:05.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.880+0000 7fa7f3200640 1 --2- 172.21.15.62:0/1537242875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7ec04ff50 0x7fa7ec037b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.880+0000 7fa7f2800640 1 --2- 172.21.15.62:0/1537242875 >> v2:172.21.15.120:3300/0 conn(0x7fa7ec0e32a0 0x7fa7ec052c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.881+0000 7f7bdf400640 1 -- 172.21.15.62:0/3062270872 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd802c980 con 0x7f7be015cba0 2024-03-23T11:19:05.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.881+0000 7f7bdf400640 1 -- 172.21.15.62:0/3062270872 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd802cae0 con 0x7f7be015cba0 2024-03-23T11:19:05.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.881+0000 7f7bdf400640 1 -- 172.21.15.62:0/3062270872 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd802ce70 con 0x7f7be015cba0 2024-03-23T11:19:05.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.881+0000 7fa7f1400640 1 -- 172.21.15.62:0/1537242875 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894239375 0 0) 0x7fa7ec0e3680 con 0x7fa7ec053160 2024-03-23T11:19:05.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.881+0000 7fa7f1400640 1 -- 172.21.15.62:0/1537242875 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d8003520 con 0x7fa7ec053160 2024-03-23T11:19:05.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.881+0000 7fa7f3200640 1 -- 172.21.15.62:0/1537242875 >> v2:172.21.15.120:3300/0 conn(0x7fa7ec0e32a0 msgr2=0x7fa7ec052c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:05.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.881+0000 7fa7f3200640 1 --2- 172.21.15.62:0/1537242875 >> v2:172.21.15.120:3300/0 conn(0x7fa7ec0e32a0 0x7fa7ec052c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.881+0000 7fa7f3200640 1 -- 172.21.15.62:0/1537242875 >> v1:172.21.15.120:6789/0 conn(0x7fa7ec053160 legacy=0x7fa7ec04f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.881+0000 7fa7f3200640 1 -- 172.21.15.62:0/1537242875 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ec053c30 con 0x7fa7ec04ff50 2024-03-23T11:19:05.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.882+0000 7f1efbe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1efc15c990 0x7f1efc06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.882+0000 7f1efbe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1efc15cd70 con 0x7f1efc0cefa0 2024-03-23T11:19:05.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.882+0000 7f1efbe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1efc0ce2f0 con 0x7f1efc06b3f0 2024-03-23T11:19:05.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.882+0000 7f1efbe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1efc0ce5d0 con 0x7f1efc15c990 2024-03-23T11:19:05.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.882+0000 7f1efa000640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f1efc06b3f0 0x7f1efc0d0e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41748/0 (socket says 172.21.15.62:41748) 2024-03-23T11:19:05.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.882+0000 7f7be6200640 1 -- 172.21.15.62:0/3062270872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be015cba0 msgr2=0x7f7be00d3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.882+0000 7f7be6200640 1 --2- 172.21.15.62:0/3062270872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be015cba0 0x7f7be00d3020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009d90 tx=0x7f7bd802bc50 comp rx=0 tx=0).stop 2024-03-23T11:19:05.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.882+0000 7f1efa000640 1 -- 172.21.15.62:0/3659307036 learned_addr learned my addr 172.21.15.62:0/3659307036 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:05.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.882+0000 7f1efaa00640 1 --2- 172.21.15.62:0/3659307036 >> v2:172.21.15.62:3300/0 conn(0x7f1efc15c990 0x7f1efc06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.883+0000 7fa7f3200640 1 --2- 172.21.15.62:0/1537242875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7ec04ff50 0x7fa7ec037b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4068a10 tx=0x7fa7f40a5990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9947ad35fc92b4f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.883+0000 7f7be6200640 1 -- 172.21.15.62:0/3062270872 shutdown_connections 2024-03-23T11:19:05.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.883+0000 7f7be6200640 1 --2- 172.21.15.62:0/3062270872 >> v2:172.21.15.120:3300/0 conn(0x7f7be00b4880 0x7f7be00cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.883+0000 7f7be6200640 1 --2- 172.21.15.62:0/3062270872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be015cba0 0x7f7be00d3020 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.883+0000 7f7be6200640 1 -- 172.21.15.62:0/3062270872 >> 172.21.15.62:0/3062270872 conn(0x7f7be0089f60 msgr2=0x7f7be00d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:05.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.883+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001496705 0 0) 0x7f1efc0ce2f0 con 0x7f1efc06b3f0 2024-03-23T11:19:05.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.883+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee0003520 con 0x7f1efc06b3f0 2024-03-23T11:19:05.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.883+0000 7f7be6200640 1 -- 172.21.15.62:0/3062270872 shutdown_connections 2024-03-23T11:19:05.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2133006043 0 0) 0x7f1ee0003520 con 0x7f1efc06b3f0 2024-03-23T11:19:05.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 >> v2:172.21.15.62:3300/0 conn(0x7f1efc15c990 msgr2=0x7f1efc06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 --2- 172.21.15.62:0/3659307036 >> v2:172.21.15.62:3300/0 conn(0x7f1efc15c990 0x7f1efc06aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 >> v1:172.21.15.62:6789/0 conn(0x7f1efc0cefa0 legacy=0x7f1efc0cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc0d0360 con 0x7f1efc06b3f0 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f1eec0019f0 con 0x7f1efc06b3f0 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 >> v1:172.21.15.120:6789/0 conn(0x7f1efc06b3f0 legacy=0x7f1efc0d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ee0003970 0x7f1ee0003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee0004370 0x7f1ee0004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee0005040 0x7f1ee000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc0ce5d0 con 0x7f1ee0003970 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee000bfa0 con 0x7f1ee0004370 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee000c160 con 0x7f1ee0005040 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.884+0000 7f1efaa00640 1 --2- 172.21.15.62:0/3659307036 >> v2:172.21.15.62:3300/0 conn(0x7f1efc15c990 0x7f1efc06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.885+0000 7f1efb400640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee0005040 0x7f1ee000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.885+0000 7f1efaa00640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ee0003970 0x7f1ee0003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.885+0000 7f1efa000640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee0004370 0x7f1ee0004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7f1efaa00640 1 -- 172.21.15.62:0/3659307036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee0005040 msgr2=0x7f1ee000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:05.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7f1efaa00640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee0005040 0x7f1ee000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7f1efaa00640 1 -- 172.21.15.62:0/3659307036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee0004370 msgr2=0x7f1ee0004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:05.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.883+0000 7f7be6200640 1 -- 172.21.15.62:0/3062270872 wait complete. 2024-03-23T11:19:05.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7f1efaa00640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee0004370 0x7f1ee0004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7f1efaa00640 1 -- 172.21.15.62:0/3659307036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc0d0360 con 0x7f1ee0003970 2024-03-23T11:19:05.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7fa7f1400640 1 -- 172.21.15.62:0/1537242875 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40aa700 con 0x7fa7ec04ff50 2024-03-23T11:19:05.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7fa7f1400640 1 -- 172.21.15.62:0/1537242875 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f40aa860 con 0x7fa7ec04ff50 2024-03-23T11:19:05.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7fa7f1400640 1 -- 172.21.15.62:0/1537242875 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40aac90 con 0x7fa7ec04ff50 2024-03-23T11:19:05.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7f7be6200640 1 Processor -- start 2024-03-23T11:19:05.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7f7be6200640 1 -- start start 2024-03-23T11:19:05.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7f1efb400640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1ee0005040 0x7f1ee000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:05.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.886+0000 7f1efb400640 1 -- 172.21.15.62:0/3659307036 reap_dead start 2024-03-23T11:19:05.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.888+0000 7f010de00640 1 Processor -- start 2024-03-23T11:19:05.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.888+0000 7f1efaa00640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ee0003970 0x7f1ee0003d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009ed0 tx=0x7f1ee00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebea4b5c164da0b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.888+0000 7f010de00640 1 -- start start 2024-03-23T11:19:05.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.888+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef402cb40 con 0x7f1ee0003970 2024-03-23T11:19:05.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.888+0000 7f7be6200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be00b4880 0x7f7be01fefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.888+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef402cca0 con 0x7f1ee0003970 2024-03-23T11:19:05.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.889+0000 7f7be6200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be00ce8e0 0x7f7be01f1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.889+0000 7f7be6200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7be015cba0 0x7f7be00cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.889+0000 7f7be6200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be00b6ea0 con 0x7f7be00ce8e0 2024-03-23T11:19:05.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.889+0000 7f7be6200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be00b6570 con 0x7f7be015cba0 2024-03-23T11:19:05.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.889+0000 7fa7f8a00640 1 -- 172.21.15.62:0/1537242875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7ec04ff50 msgr2=0x7fa7ec037b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.889+0000 7fa7f8a00640 1 --2- 172.21.15.62:0/1537242875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7ec04ff50 0x7fa7ec037b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4068a10 tx=0x7fa7f40a5990 comp rx=0 tx=0).stop 2024-03-23T11:19:05.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.888+0000 7f1ef9600640 1 -- 172.21.15.62:0/3659307036 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4004220 con 0x7f1ee0003970 2024-03-23T11:19:05.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f7be6200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be00b6850 con 0x7f7be00b4880 2024-03-23T11:19:05.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7fa7f8a00640 1 -- 172.21.15.62:0/1537242875 shutdown_connections 2024-03-23T11:19:05.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7fa7f8a00640 1 --2- 172.21.15.62:0/1537242875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7ec04ff50 0x7fa7ec037b80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7fa7f8a00640 1 --2- 172.21.15.62:0/1537242875 >> v2:172.21.15.120:3300/0 conn(0x7fa7ec0e32a0 0x7fa7ec052c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7fa7f8a00640 1 -- 172.21.15.62:0/1537242875 >> 172.21.15.62:0/1537242875 conn(0x7fa7ec009f80 msgr2=0x7fa7ec00a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:05.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f1efbe00640 1 -- 172.21.15.62:0/3659307036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ee0003970 msgr2=0x7f1ee0003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f7be4e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be00b4880 0x7f7be01fefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f7be5800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7be015cba0 0x7f7be00cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f7be5800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7be015cba0 0x7f7be00cc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53376/0 (socket says 172.21.15.62:53376) 2024-03-23T11:19:05.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f7be5800640 1 -- 172.21.15.62:0/1710127566 learned_addr learned my addr 172.21.15.62:0/1710127566 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:05.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f1efbe00640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ee0003970 0x7f1ee0003d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009ed0 tx=0x7f1ee00057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7fa7f8a00640 1 -- 172.21.15.62:0/1537242875 shutdown_connections 2024-03-23T11:19:05.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f010de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f010815cdc0 0x7f01080d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f010de00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f01080cf650 0x7f01080cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f010de00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010806bfc0 con 0x7f010815c9e0 2024-03-23T11:19:05.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f010de00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01080d0020 con 0x7f01080cf650 2024-03-23T11:19:05.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f010de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01080ce140 con 0x7f010815cdc0 2024-03-23T11:19:05.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f7bdfe00640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be00ce8e0 0x7f7be01f1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7fa7f8a00640 1 -- 172.21.15.62:0/1537242875 wait complete. 2024-03-23T11:19:05.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f1efbe00640 1 -- 172.21.15.62:0/3659307036 shutdown_connections 2024-03-23T11:19:05.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.891+0000 7f0106a00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f010815c9e0 0x7f01081607c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:41760/0 (socket says 172.21.15.62:41760) 2024-03-23T11:19:05.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.891+0000 7f0106a00640 1 -- 172.21.15.62:0/170493281 learned_addr learned my addr 172.21.15.62:0/170493281 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:05.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.891+0000 7f7be5800640 1 -- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be00b4880 msgr2=0x7f7be01fefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.891+0000 7f7be5800640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be00b4880 0x7f7be01fefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.891+0000 7f7be5800640 1 -- 172.21.15.62:0/1710127566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be00ce8e0 msgr2=0x7f7be01f1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.891+0000 7f7be5800640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be00ce8e0 0x7f7be01f1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.891+0000 7f7be5800640 1 -- 172.21.15.62:0/1710127566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd8009a70 con 0x7f7be015cba0 2024-03-23T11:19:05.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.891+0000 7f0106000640 1 --2- 172.21.15.62:0/170493281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f010815cdc0 0x7f01080d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.891+0000 7f0107400640 1 --2- 172.21.15.62:0/170493281 >> v2:172.21.15.120:3300/0 conn(0x7f01080cf650 0x7f01080cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.891+0000 7f7bdfe00640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be00ce8e0 0x7f7be01f1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:05.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f1efbe00640 1 --2- 172.21.15.62:0/3659307036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ee0003970 0x7f1ee0003d50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.890+0000 7f1efbe00640 1 -- 172.21.15.62:0/3659307036 >> 172.21.15.62:0/3659307036 conn(0x7f1efc089f60 msgr2=0x7f1efc0d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:05.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.892+0000 7f0106000640 1 -- 172.21.15.62:0/170493281 >> v2:172.21.15.120:3300/0 conn(0x7f01080cf650 msgr2=0x7f01080cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:05.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.892+0000 7f0106000640 1 --2- 172.21.15.62:0/170493281 >> v2:172.21.15.120:3300/0 conn(0x7f01080cf650 0x7f01080cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.892+0000 7f7be5800640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7be015cba0 0x7f7be00cc130 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000d5a0 tx=0x7f7bd000da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.892+0000 7f0106000640 1 -- 172.21.15.62:0/170493281 >> v1:172.21.15.120:6789/0 conn(0x7f010815c9e0 legacy=0x7f01081607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.892+0000 7f0105600640 1 -- 172.21.15.62:0/170493281 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463529747 0 0) 0x7f010806bfc0 con 0x7f010815c9e0 2024-03-23T11:19:05.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.892+0000 7f1efbe00640 1 -- 172.21.15.62:0/3659307036 shutdown_connections 2024-03-23T11:19:05.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.892+0000 7f1efbe00640 1 -- 172.21.15.62:0/3659307036 wait complete. 2024-03-23T11:19:05.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.892+0000 7f0106000640 1 -- 172.21.15.62:0/170493281 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01080b5260 con 0x7f010815cdc0 2024-03-23T11:19:05.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.892+0000 7f7bdcc00640 1 -- 172.21.15.62:0/1710127566 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd00125f0 con 0x7f7be015cba0 2024-03-23T11:19:05.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.893+0000 7f7be6200640 1 -- 172.21.15.62:0/1710127566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be00b6b30 con 0x7f7be015cba0 2024-03-23T11:19:05.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.893+0000 7f7be6200640 1 -- 172.21.15.62:0/1710127566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be01f1be0 con 0x7f7be015cba0 2024-03-23T11:19:05.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.893+0000 7f7bdcc00640 1 -- 172.21.15.62:0/1710127566 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd0012750 con 0x7f7be015cba0 2024-03-23T11:19:05.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.894+0000 7f0107400640 1 --2- 172.21.15.62:0/170493281 >> v2:172.21.15.120:3300/0 conn(0x7f01080cf650 0x7f01080cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:05.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.894+0000 7f5ff5000640 1 Processor -- start 2024-03-23T11:19:05.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.894+0000 7fa7f8a00640 1 Processor -- start 2024-03-23T11:19:05.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.894+0000 7f0106000640 1 --2- 172.21.15.62:0/170493281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f010815cdc0 0x7f01080d1f20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f00f40097a0 tx=0x7f00f402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e402713928cf5a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.894+0000 7f5ff5000640 1 -- start start 2024-03-23T11:19:05.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.894+0000 7fa7f8a00640 1 -- start start 2024-03-23T11:19:05.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.894+0000 7f1efbe00640 1 Processor -- start 2024-03-23T11:19:05.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.895+0000 7f1efbe00640 1 -- start start 2024-03-23T11:19:05.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.893+0000 7f7bdcc00640 1 -- 172.21.15.62:0/1710127566 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0012ae0 con 0x7f7be015cba0 2024-03-23T11:19:05.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.896+0000 7f1efbe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee0003970 0x7f1efc0b7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.896+0000 7f1efbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1efc0cc0f0 0x7f1efc0b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.896+0000 7f1efbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1efc202130 0x7f1efc2025b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.896+0000 7fa7f8a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7ec04ff50 0x7fa7ec0e4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7fa7f8a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7ec053160 0x7fa7ec04c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f1efbe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc202af0 con 0x7f1efc202130 2024-03-23T11:19:05.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f1efbe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc0b7d40 con 0x7f1ee0003970 2024-03-23T11:19:05.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f1efbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc0b58b0 con 0x7f1efc0cc0f0 2024-03-23T11:19:05.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f0105600640 1 -- 172.21.15.62:0/170493281 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f4031830 con 0x7f010815cdc0 2024-03-23T11:19:05.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f1efa000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1efc0cc0f0 0x7f1efc0b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f0105600640 1 -- 172.21.15.62:0/170493281 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f4031990 con 0x7f010815cdc0 2024-03-23T11:19:05.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f1efaa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee0003970 0x7f1efc0b7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f0105600640 1 -- 172.21.15.62:0/170493281 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f4031d20 con 0x7f010815cdc0 2024-03-23T11:19:05.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f1efaa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee0003970 0x7f1efc0b7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53386/0 (socket says 172.21.15.62:53386) 2024-03-23T11:19:05.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f1efb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1efc202130 0x7f1efc2025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f1efb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1efc202130 0x7f1efc2025b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38084/0 (socket says 172.21.15.62:38084) 2024-03-23T11:19:05.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7f1efaa00640 1 -- 172.21.15.62:0/2615592805 learned_addr learned my addr 172.21.15.62:0/2615592805 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:05.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7fa7f2800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7ec04ff50 0x7fa7ec0e4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7fa7f2800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7ec04ff50 0x7fa7ec0e4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53400/0 (socket says 172.21.15.62:53400) 2024-03-23T11:19:05.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7fa7f2800640 1 -- 172.21.15.62:0/3269145188 learned_addr learned my addr 172.21.15.62:0/3269145188 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:05.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.897+0000 7fa7f1e00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7ec053160 0x7fa7ec04c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.898+0000 7fa7f8a00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7ec0e32a0 0x7fa7ec038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.898+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec04cff0 con 0x7fa7ec053160 2024-03-23T11:19:05.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.898+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec0e37b0 con 0x7fa7ec04ff50 2024-03-23T11:19:05.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.898+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec0e3a90 con 0x7fa7ec0e32a0 2024-03-23T11:19:05.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.898+0000 7fa7f3200640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7ec0e32a0 0x7fa7ec038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.898+0000 7f1efb400640 1 -- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1efc0cc0f0 msgr2=0x7f1efc0b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.898+0000 7f1efb400640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1efc0cc0f0 0x7f1efc0b5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.898+0000 7f1efb400640 1 -- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee0003970 msgr2=0x7f1efc0b7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.898+0000 7f1efb400640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee0003970 0x7f1efc0b7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.898+0000 7f1efb400640 1 -- 172.21.15.62:0/2615592805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc0b7fa0 con 0x7f1efc202130 2024-03-23T11:19:05.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7f1efb400640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1efc202130 0x7f1efc2025b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1efc08ac50 tx=0x7f1ee400feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7f5ff5000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5ff00b2f90 0x7f5ff0161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7f5ff5000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 0x7f5ff00b3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7f5ff5000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff00ae0c0 con 0x7f5ff0164a30 2024-03-23T11:19:05.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7f5ff5000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff00b2740 con 0x7f5ff00b2f90 2024-03-23T11:19:05.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7f5ff5000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff01623b0 con 0x7f5ff00b15e0 2024-03-23T11:19:05.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7fa7f2800640 1 -- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7ec0e32a0 msgr2=0x7fa7ec038b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7fa7f2800640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7ec0e32a0 0x7fa7ec038b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7fa7f2800640 1 -- 172.21.15.62:0/3269145188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7ec053160 msgr2=0x7fa7ec04c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7fa7f2800640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7ec053160 0x7fa7ec04c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7fa7f2800640 1 -- 172.21.15.62:0/3269145188 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f404e550 con 0x7fa7ec04ff50 2024-03-23T11:19:05.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7fa7f1e00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7ec053160 0x7fa7ec04c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:05.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7f010de00640 1 -- 172.21.15.62:0/170493281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f010815cdc0 msgr2=0x7f01080d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7f010de00640 1 --2- 172.21.15.62:0/170493281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f010815cdc0 0x7f01080d1f20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f00f40097a0 tx=0x7f00f402b8d0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.899+0000 7f5fed400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 0x7f5ff00b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f5fed400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 0x7f5ff00b3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:56238/0 (socket says 172.21.15.62:56238) 2024-03-23T11:19:05.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f5fed400640 1 -- 172.21.15.62:0/3887570996 learned_addr learned my addr 172.21.15.62:0/3887570996 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:05.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f5fede00640 1 --2- 172.21.15.62:0/3887570996 >> v2:172.21.15.62:3300/0 conn(0x7f5ff00b2f90 0x7f5ff0161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f1ef2a00640 1 -- 172.21.15.62:0/2615592805 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4018070 con 0x7f1efc202130 2024-03-23T11:19:05.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f1ef2a00640 1 -- 172.21.15.62:0/2615592805 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee40050d0 con 0x7f1efc202130 2024-03-23T11:19:05.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f1ef2a00640 1 -- 172.21.15.62:0/2615592805 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4005280 con 0x7f1efc202130 2024-03-23T11:19:05.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f1efbe00640 1 -- 172.21.15.62:0/2615592805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1efc0b8280 con 0x7f1efc202130 2024-03-23T11:19:05.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f7be6200640 1 -- 172.21.15.62:0/1710127566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb8005250 con 0x7f7be015cba0 2024-03-23T11:19:05.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f1efbe00640 1 -- 172.21.15.62:0/2615592805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1efc15d490 con 0x7f1efc202130 2024-03-23T11:19:05.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f010de00640 1 -- 172.21.15.62:0/170493281 shutdown_connections 2024-03-23T11:19:05.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f010de00640 1 --2- 172.21.15.62:0/170493281 >> v2:172.21.15.120:3300/0 conn(0x7f01080cf650 0x7f01080cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f010de00640 1 --2- 172.21.15.62:0/170493281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f010815cdc0 0x7f01080d1f20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.901+0000 7fa7f2800640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7ec04ff50 0x7fa7ec0e4570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000e9b0 tx=0x7fa7e000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f010de00640 1 -- 172.21.15.62:0/170493281 >> 172.21.15.62:0/170493281 conn(0x7f0108071570 msgr2=0x7f0108071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:05.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.900+0000 7f5fed400640 1 -- 172.21.15.62:0/3887570996 >> v2:172.21.15.62:3300/0 conn(0x7f5ff00b2f90 msgr2=0x7f5ff0161e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.901+0000 7f7bdcc00640 1 -- 172.21.15.62:0/1710127566 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7bd0012c40 con 0x7f7be015cba0 2024-03-23T11:19:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.901+0000 7fa7eaa00640 1 -- 172.21.15.62:0/3269145188 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0009670 con 0x7fa7ec04ff50 2024-03-23T11:19:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.901+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ec0e3d70 con 0x7fa7ec04ff50 2024-03-23T11:19:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.901+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ec0e4ab0 con 0x7fa7ec04ff50 2024-03-23T11:19:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.902+0000 7f010de00640 1 -- 172.21.15.62:0/170493281 shutdown_connections 2024-03-23T11:19:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.901+0000 7f5fed400640 1 --2- 172.21.15.62:0/3887570996 >> v2:172.21.15.62:3300/0 conn(0x7f5ff00b2f90 0x7f5ff0161e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.901+0000 7f5fed400640 1 -- 172.21.15.62:0/3887570996 >> v1:172.21.15.120:6789/0 conn(0x7f5ff0164a30 legacy=0x7f5ff0164ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T11:19:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.901+0000 7f5fed400640 1 -- 172.21.15.62:0/3887570996 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff00b1100 con 0x7f5ff00b15e0 2024-03-23T11:19:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.902+0000 7f5fede00640 1 --2- 172.21.15.62:0/3887570996 >> v2:172.21.15.62:3300/0 conn(0x7f5ff00b2f90 0x7f5ff0161e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.904+0000 7f5fed400640 1 --2- 172.21.15.62:0/3887570996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 0x7f5ff00b3370 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009870 tx=0x7f5fdc02b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84ace605d3b1a1b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.902+0000 7fa7eaa00640 1 -- 172.21.15.62:0/3269145188 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e000c9a0 con 0x7fa7ec04ff50 2024-03-23T11:19:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.902+0000 7fa7eaa00640 1 -- 172.21.15.62:0/3269145188 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0009670 con 0x7fa7ec04ff50 2024-03-23T11:19:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.903+0000 7f1ef2a00640 1 -- 172.21.15.62:0/2615592805 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f1ee4007030 con 0x7f1efc202130 2024-03-23T11:19:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.904+0000 7f1ef2a00640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ed806a7f0 0x7f1ed806ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.904+0000 7f1ef2a00640 1 -- 172.21.15.62:0/2615592805 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ee4088130 con 0x7f1efc202130 2024-03-23T11:19:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.905+0000 7f5feca00640 1 -- 172.21.15.62:0/3887570996 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc031870 con 0x7f5ff00b15e0 2024-03-23T11:19:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.905+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ec0089a0 con 0x7fa7ec04ff50 2024-03-23T11:19:05.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.905+0000 7f5feca00640 1 -- 172.21.15.62:0/3887570996 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fdc0319d0 con 0x7f5ff00b15e0 2024-03-23T11:19:05.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.905+0000 7f5feca00640 1 -- 172.21.15.62:0/3887570996 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc031870 con 0x7f5ff00b15e0 2024-03-23T11:19:05.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.902+0000 7f010de00640 1 -- 172.21.15.62:0/170493281 wait complete. 2024-03-23T11:19:05.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.905+0000 7f010de00640 1 Processor -- start 2024-03-23T11:19:05.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.905+0000 7f1efaa00640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ed806a7f0 0x7f1ed806ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.905+0000 7f010de00640 1 -- start start 2024-03-23T11:19:05.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.906+0000 7f7bdcc00640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bbc06a7d0 0x7f7bbc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.906+0000 7f1efbe00640 1 -- 172.21.15.62:0/2615592805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed4005250 con 0x7f1efc202130 2024-03-23T11:19:05.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.906+0000 7f5ff5000640 1 -- 172.21.15.62:0/3887570996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 msgr2=0x7f5ff00b3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.906+0000 7f5ff5000640 1 --2- 172.21.15.62:0/3887570996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 0x7f5ff00b3370 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009870 tx=0x7f5fdc02b910 comp rx=0 tx=0).stop 2024-03-23T11:19:05.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.907+0000 7f5ff5000640 1 -- 172.21.15.62:0/3887570996 shutdown_connections 2024-03-23T11:19:05.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.907+0000 7fa7eaa00640 1 -- 172.21.15.62:0/3269145188 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa7e000cb10 con 0x7fa7ec04ff50 2024-03-23T11:19:05.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.907+0000 7f5ff5000640 1 --2- 172.21.15.62:0/3887570996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 0x7f5ff00b3370 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.907+0000 7f5ff5000640 1 --2- 172.21.15.62:0/3887570996 >> v2:172.21.15.62:3300/0 conn(0x7f5ff00b2f90 0x7f5ff0161e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.907+0000 7f5ff5000640 1 -- 172.21.15.62:0/3887570996 >> 172.21.15.62:0/3887570996 conn(0x7f5ff00b02e0 msgr2=0x7f5ff00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:05.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.907+0000 7f5ff5000640 1 -- 172.21.15.62:0/3887570996 shutdown_connections 2024-03-23T11:19:05.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.908+0000 7f5ff5000640 1 -- 172.21.15.62:0/3887570996 wait complete. 2024-03-23T11:19:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.908+0000 7f010de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01080cf650 0x7f010815d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.908+0000 7f010de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f010815c9e0 0x7f010806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.908+0000 7f010de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f010815cdc0 0x7f01081f8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f0106a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01080cf650 0x7f010815d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f0107400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f010815cdc0 0x7f01081f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f0107400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f010815cdc0 0x7f01081f8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38114/0 (socket says 172.21.15.62:38114) 2024-03-23T11:19:05.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f0107400640 1 -- 172.21.15.62:0/3649056074 learned_addr learned my addr 172.21.15.62:0/3649056074 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:05.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f0106a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01080cf650 0x7f010815d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:56248/0 (socket says 172.21.15.62:56248) 2024-03-23T11:19:05.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f010de00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01082025a0 con 0x7f010815cdc0 2024-03-23T11:19:05.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0108202880 con 0x7f010815c9e0 2024-03-23T11:19:05.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0108202b60 con 0x7f01080cf650 2024-03-23T11:19:05.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.907+0000 7f7be4e00640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bbc06a7d0 0x7f7bbc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.907+0000 7f7bdcc00640 1 -- 172.21.15.62:0/1710127566 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bd0089130 con 0x7f7be015cba0 2024-03-23T11:19:05.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.910+0000 7fa7eaa00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a800 0x7fa7d406ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f0106000640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f010815c9e0 0x7f010806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.910+0000 7fa7eaa00640 1 -- 172.21.15.62:0/3269145188 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7e0088bd0 con 0x7fa7ec04ff50 2024-03-23T11:19:05.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.910+0000 7f1efaa00640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ed806a7f0 0x7f1ed806ccb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40091d0 tx=0x7f1ef4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f0107400640 1 -- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01080cf650 msgr2=0x7f010815d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f0107400640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01080cf650 0x7f010815d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f0107400640 1 -- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f010815c9e0 msgr2=0x7f010806ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f0107400640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f010815c9e0 0x7f010806ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.909+0000 7f0107400640 1 -- 172.21.15.62:0/3649056074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0100009a70 con 0x7f010815cdc0 2024-03-23T11:19:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.910+0000 7f0106000640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f010815c9e0 0x7f010806ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.910+0000 7f0107400640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f010815cdc0 0x7f01081f8860 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f010000efc0 tx=0x7f0100009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.911+0000 7fa7f1e00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a800 0x7fa7d406ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.911+0000 7f7be4e00640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bbc06a7d0 0x7f7bbc06cc90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8036000 tx=0x7f7bd8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.912+0000 7fa7f1e00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a800 0x7fa7d406ccc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec04b680 tx=0x7fa7e4007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.912+0000 7f00ff400640 1 -- 172.21.15.62:0/3649056074 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100009d70 con 0x7f010815cdc0 2024-03-23T11:19:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.912+0000 7f00ff400640 1 -- 172.21.15.62:0/3649056074 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010000f040 con 0x7f010815cdc0 2024-03-23T11:19:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.912+0000 7f00ff400640 1 -- 172.21.15.62:0/3649056074 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100003990 con 0x7f010815cdc0 2024-03-23T11:19:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.912+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f4009400 con 0x7f010815cdc0 2024-03-23T11:19:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.912+0000 7f5ff5000640 1 Processor -- start 2024-03-23T11:19:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.912+0000 7f5ff5000640 1 -- start start 2024-03-23T11:19:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.913+0000 7f1ef2a00640 1 -- 172.21.15.62:0/2615592805 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ee40536e0 con 0x7f1efc202130 2024-03-23T11:19:05.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.913+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0108203120 con 0x7f010815cdc0 2024-03-23T11:19:05.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.914+0000 7f5ff5000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 0x7f5ff0156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.914+0000 7f5ff5000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ff00b2f90 0x7f5ff01f9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.914+0000 7f5ff5000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff0164a30 0x7f5ff00cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.914+0000 7f5ff5000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff01f6300 con 0x7f5ff00b2f90 2024-03-23T11:19:05.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.914+0000 7f5ff5000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff015a130 con 0x7f5ff0164a30 2024-03-23T11:19:05.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.914+0000 7f5ff5000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff01565a0 con 0x7f5ff00b15e0 2024-03-23T11:19:05.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.914+0000 7f5fed400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ff00b2f90 0x7f5ff01f9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.915+0000 7f5fed400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ff00b2f90 0x7f5ff01f9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38118/0 (socket says 172.21.15.62:38118) 2024-03-23T11:19:05.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.915+0000 7f5fed400640 1 -- 172.21.15.62:0/1978653568 learned_addr learned my addr 172.21.15.62:0/1978653568 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:05.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.915+0000 7f5fede00640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 0x7f5ff0156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.915+0000 7f5fed400640 1 -- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 msgr2=0x7f5ff0156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:05.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.915+0000 7f5fed400640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 0x7f5ff0156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.915+0000 7f5fed400640 1 -- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff0164a30 msgr2=0x7f5ff00cd750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T11:19:05.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.915+0000 7f5fed400640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff0164a30 0x7f5ff00cd750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:05.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.915+0000 7f5fed400640 1 -- 172.21.15.62:0/1978653568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe0009400 con 0x7f5ff00b2f90 2024-03-23T11:19:05.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.916+0000 7f5fed400640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ff00b2f90 0x7f5ff01f9920 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0090b0 tx=0x7f5fdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.916+0000 7f00ff400640 1 -- 172.21.15.62:0/3649056074 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f01000144a0 con 0x7f010815cdc0 2024-03-23T11:19:05.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.916+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f010806c1a0 con 0x7f010815cdc0 2024-03-23T11:19:05.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.916+0000 7f5fe6000640 1 -- 172.21.15.62:0/1978653568 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc031870 con 0x7f5ff00b2f90 2024-03-23T11:19:05.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.916+0000 7f5fe6000640 1 -- 172.21.15.62:0/1978653568 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fdc03ae50 con 0x7f5ff00b2f90 2024-03-23T11:19:05.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.916+0000 7f5fe6000640 1 -- 172.21.15.62:0/1978653568 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc031870 con 0x7f5ff00b2f90 2024-03-23T11:19:05.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.916+0000 7f5ff5000640 1 -- 172.21.15.62:0/1978653568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc0094d0 con 0x7f5ff00b2f90 2024-03-23T11:19:05.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.917+0000 7f5ff5000640 1 -- 172.21.15.62:0/1978653568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff0159890 con 0x7f5ff00b2f90 2024-03-23T11:19:05.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.918+0000 7f00ff400640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f00e806a870 0x7f00e806cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.918+0000 7f5ff5000640 1 -- 172.21.15.62:0/1978653568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff0073860 con 0x7f5ff00b2f90 2024-03-23T11:19:05.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.918+0000 7f00ff400640 1 -- 172.21.15.62:0/3649056074 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0100089950 con 0x7f010815cdc0 2024-03-23T11:19:05.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.919+0000 7f0106a00640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f00e806a870 0x7f00e806cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.920+0000 7f0106a00640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f00e806a870 0x7f00e806cd30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f00f8000f60 tx=0x7f00f8007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.921+0000 7f7bdcc00640 1 -- 172.21.15.62:0/1710127566 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7bd0054500 con 0x7f7be015cba0 2024-03-23T11:19:05.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.923+0000 7f00ff400640 1 -- 172.21.15.62:0/3649056074 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0100054de0 con 0x7f010815cdc0 2024-03-23T11:19:05.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.924+0000 7f5fe6000640 1 -- 172.21.15.62:0/1978653568 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5fdc042070 con 0x7f5ff00b2f90 2024-03-23T11:19:05.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.926+0000 7f5fe6000640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5fd006a830 0x7f5fd006ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:05.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.926+0000 7f5fe6000640 1 -- 172.21.15.62:0/1978653568 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fdc02d040 con 0x7f5ff00b2f90 2024-03-23T11:19:05.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.927+0000 7f5fede00640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5fd006a830 0x7f5fd006ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:05.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.928+0000 7f5fede00640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5fd006a830 0x7f5fd006ccf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0009cd0 tx=0x7f5fe0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:05.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.932+0000 7f5fe6000640 1 -- 172.21.15.62:0/1978653568 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5fdc076dd0 con 0x7f5ff00b2f90 2024-03-23T11:19:05.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:05.936+0000 7fa7eaa00640 1 -- 172.21.15.62:0/3269145188 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7e0054090 con 0x7fa7ec04ff50 2024-03-23T11:19:06.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.222+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f41b80ca410 con 0x7f41b814aea0 2024-03-23T11:19:06.224 INFO:teuthology.orchestra.run.smithi062.stdout:60129542151 2024-03-23T11:19:06.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.223+0000 7f41af400640 1 -- 172.21.15.62:0/159301967 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f41b80ca410 con 0x7f41b814aea0 2024-03-23T11:19:06.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.227+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f419806a7a0 msgr2=0x7f419806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.227+0000 7f41be200640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f419806a7a0 0x7f419806cc60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f41a4005880 tx=0x7f41a4009040 comp rx=0 tx=0).stop 2024-03-23T11:19:06.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.227+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b814aea0 msgr2=0x7f41b80ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.227+0000 7f41be200640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b814aea0 0x7f41b80ca6f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f41b000c730 tx=0x7f41b000cbf0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.228+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 shutdown_connections 2024-03-23T11:19:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.228+0000 7f41be200640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f419806a7a0 0x7f419806cc60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.228+0000 7f41be200640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41b814aea0 0x7f41b80ca6f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.228+0000 7f41be200640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41b80d2260 0x7f41b80c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.228+0000 7f41be200640 1 --2- 172.21.15.62:0/159301967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b80b4d80 0x7f41b806dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.228+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 >> 172.21.15.62:0/159301967 conn(0x7f41b8071570 msgr2=0x7f41b8161a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:06.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.228+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 shutdown_connections 2024-03-23T11:19:06.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.229+0000 7f41be200640 1 -- 172.21.15.62:0/159301967 wait complete. 2024-03-23T11:19:06.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.1 2024-03-23T11:19:06.254 DEBUG:teuthology.parallel:result is None 2024-03-23T11:19:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.257+0000 7f36eb800640 1 -- 172.21.15.62:0/1985558274 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f36c00057d0 con 0x7f36e40cf0e0 2024-03-23T11:19:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.258+0000 7f85e0c00640 1 -- 172.21.15.62:0/1237306739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f85c4005500 con 0x7f85e80e12e0 2024-03-23T11:19:06.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.258+0000 7f36e0c00640 1 -- 172.21.15.62:0/1985558274 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f36d80574e0 con 0x7f36e40cf0e0 2024-03-23T11:19:06.258 INFO:teuthology.orchestra.run.smithi062.stdout:60129542149 2024-03-23T11:19:06.258 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:06.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.258+0000 7f85e3400640 1 -- 172.21.15.62:0/1237306739 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f85f0097070 con 0x7f85e80e12e0 2024-03-23T11:19:06.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.261+0000 7f36eb800640 1 -- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a800 msgr2=0x7f36c406ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.261+0000 7f36eb800640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a800 0x7f36c406ccc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f36d0035000 tx=0x7f36d0008030 comp rx=0 tx=0).stop 2024-03-23T11:19:06.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f36eb800640 1 -- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e40cf0e0 msgr2=0x7f36e40b7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f36eb800640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e40cf0e0 0x7f36e40b7440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f36d800e870 tx=0x7f36d800ed30 comp rx=0 tx=0).stop 2024-03-23T11:19:06.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f85e0c00640 1 -- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f85d006ebe0 msgr2=0x7f85d00710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f85e0c00640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f85d006ebe0 0x7f85d00710a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f85dc036000 tx=0x7f85dc03d040 comp rx=0 tx=0).stop 2024-03-23T11:19:06.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f85e0c00640 1 -- 172.21.15.62:0/1237306739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85e80e12e0 msgr2=0x7f85e80534d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f85e0c00640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85e80e12e0 0x7f85e80534d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f85f004f2a0 tx=0x7f85f0067690 comp rx=0 tx=0).stop 2024-03-23T11:19:06.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f36eb800640 1 -- 172.21.15.62:0/1985558274 shutdown_connections 2024-03-23T11:19:06.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f36eb800640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a800 0x7f36c406ccc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f36eb800640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40cf4c0 0x7f36e40b7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f36eb800640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e40cf0e0 0x7f36e40b7440 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f36eb800640 1 --2- 172.21.15.62:0/1985558274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e40b5ad0 0x7f36e415acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f36eb800640 1 -- 172.21.15.62:0/1985558274 >> 172.21.15.62:0/1985558274 conn(0x7f36e4089f60 msgr2=0x7f36e406c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:06.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f85e0c00640 1 -- 172.21.15.62:0/1237306739 shutdown_connections 2024-03-23T11:19:06.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f85e0c00640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f85d006ebe0 0x7f85d00710a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f85e0c00640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85e80e12e0 0x7f85e80534d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f85e0c00640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e800ab10 0x7f85e8176d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f85e0c00640 1 --2- 172.21.15.62:0/1237306739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e800a430 0x7f85e80de060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f85e0c00640 1 -- 172.21.15.62:0/1237306739 >> 172.21.15.62:0/1237306739 conn(0x7f85e8056ca0 msgr2=0x7f85e80deef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:06.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f36eb800640 1 -- 172.21.15.62:0/1985558274 shutdown_connections 2024-03-23T11:19:06.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.262+0000 7f36eb800640 1 -- 172.21.15.62:0/1985558274 wait complete. 2024-03-23T11:19:06.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.263+0000 7f85e0c00640 1 -- 172.21.15.62:0/1237306739 shutdown_connections 2024-03-23T11:19:06.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.263+0000 7f85e0c00640 1 -- 172.21.15.62:0/1237306739 wait complete. 2024-03-23T11:19:06.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.0 2024-03-23T11:19:06.289 DEBUG:teuthology.parallel:result is None 2024-03-23T11:19:06.293 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.2 2024-03-23T11:19:06.293 DEBUG:teuthology.parallel:result is None 2024-03-23T11:19:06.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.302+0000 7f7be6200640 1 -- 172.21.15.62:0/1710127566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f7bb8005500 con 0x7f7be015cba0 2024-03-23T11:19:06.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.303+0000 7f7bdcc00640 1 -- 172.21.15.62:0/1710127566 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7bd0057b20 con 0x7f7be015cba0 2024-03-23T11:19:06.303 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:06.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 -- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bbc06a7d0 msgr2=0x7f7bbc06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bbc06a7d0 0x7f7bbc06cc90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8036000 tx=0x7f7bd8030000 comp rx=0 tx=0).stop 2024-03-23T11:19:06.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 -- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7be015cba0 msgr2=0x7f7be00cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7be015cba0 0x7f7be00cc130 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000d5a0 tx=0x7f7bd000da60 comp rx=0 tx=0).stop 2024-03-23T11:19:06.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 -- 172.21.15.62:0/1710127566 shutdown_connections 2024-03-23T11:19:06.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bbc06a7d0 0x7f7bbc06cc90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be00ce8e0 0x7f7be01f1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7be015cba0 0x7f7be00cc130 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 --2- 172.21.15.62:0/1710127566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7be00b4880 0x7f7be01fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 -- 172.21.15.62:0/1710127566 >> 172.21.15.62:0/1710127566 conn(0x7f7be0089f60 msgr2=0x7f7be006aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:06.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 -- 172.21.15.62:0/1710127566 shutdown_connections 2024-03-23T11:19:06.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.306+0000 7f7be6200640 1 -- 172.21.15.62:0/1710127566 wait complete. 2024-03-23T11:19:06.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.310+0000 7f1efbe00640 1 -- 172.21.15.62:0/2615592805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f1ed40057d0 con 0x7f1efc202130 2024-03-23T11:19:06.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.310+0000 7f1ef2a00640 1 -- 172.21.15.62:0/2615592805 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1ee4001430 con 0x7f1efc202130 2024-03-23T11:19:06.310 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:06.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.312+0000 7f1efbe00640 1 -- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ed806a7f0 msgr2=0x7f1ed806ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.312+0000 7f1efbe00640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ed806a7f0 0x7f1ed806ccb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40091d0 tx=0x7f1ef4030000 comp rx=0 tx=0).stop 2024-03-23T11:19:06.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.312+0000 7f1efbe00640 1 -- 172.21.15.62:0/2615592805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1efc202130 msgr2=0x7f1efc2025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.312+0000 7f1efbe00640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1efc202130 0x7f1efc2025b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1efc08ac50 tx=0x7f1ee400feb0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.313+0000 7f1efbe00640 1 -- 172.21.15.62:0/2615592805 shutdown_connections 2024-03-23T11:19:06.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.313+0000 7f1efbe00640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ed806a7f0 0x7f1ed806ccb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.313+0000 7f1efbe00640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1efc202130 0x7f1efc2025b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.313+0000 7f1efbe00640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1efc0cc0f0 0x7f1efc0b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.313+0000 7f1efbe00640 1 --2- 172.21.15.62:0/2615592805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ee0003970 0x7f1efc0b7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.313+0000 7f1efbe00640 1 -- 172.21.15.62:0/2615592805 >> 172.21.15.62:0/2615592805 conn(0x7f1efc089f60 msgr2=0x7f1efc0b2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:06.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.313+0000 7f1efbe00640 1 -- 172.21.15.62:0/2615592805 shutdown_connections 2024-03-23T11:19:06.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.313+0000 7f1efbe00640 1 -- 172.21.15.62:0/2615592805 wait complete. 2024-03-23T11:19:06.322 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.4 2024-03-23T11:19:06.323 DEBUG:teuthology.parallel:result is None 2024-03-23T11:19:06.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.335+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa7ec005360 con 0x7fa7ec04ff50 2024-03-23T11:19:06.336 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:06.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.336+0000 7fa7eaa00640 1 -- 172.21.15.62:0/3269145188 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa7e0017090 con 0x7fa7ec04ff50 2024-03-23T11:19:06.337 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.7 2024-03-23T11:19:06.337 DEBUG:teuthology.parallel:result is None 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a800 msgr2=0x7fa7d406ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a800 0x7fa7d406ccc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec04b680 tx=0x7fa7e4007d30 comp rx=0 tx=0).stop 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7ec04ff50 msgr2=0x7fa7ec0e4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7ec04ff50 0x7fa7ec0e4570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000e9b0 tx=0x7fa7e000ee70 comp rx=0 tx=0).stop 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 shutdown_connections 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a800 0x7fa7d406ccc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7ec053160 0x7fa7ec04c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7ec0e32a0 0x7fa7ec038b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 --2- 172.21.15.62:0/3269145188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7ec04ff50 0x7fa7ec0e4570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 >> 172.21.15.62:0/3269145188 conn(0x7fa7ec009f80 msgr2=0x7fa7ec04ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 shutdown_connections 2024-03-23T11:19:06.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.338+0000 7fa7f8a00640 1 -- 172.21.15.62:0/3269145188 wait complete. 2024-03-23T11:19:06.353 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.6 2024-03-23T11:19:06.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.354+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f010806b9a0 con 0x7f010815cdc0 2024-03-23T11:19:06.354 DEBUG:teuthology.parallel:result is None 2024-03-23T11:19:06.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.354+0000 7f00ff400640 1 -- 172.21.15.62:0/3649056074 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0100018090 con 0x7f010815cdc0 2024-03-23T11:19:06.354 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.356+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f00e806a870 msgr2=0x7f00e806cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.356+0000 7f010de00640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f00e806a870 0x7f00e806cd30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f00f8000f60 tx=0x7f00f8007d50 comp rx=0 tx=0).stop 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.356+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f010815cdc0 msgr2=0x7f01081f8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.356+0000 7f010de00640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f010815cdc0 0x7f01081f8860 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f010000efc0 tx=0x7f0100009f90 comp rx=0 tx=0).stop 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.356+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 shutdown_connections 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.356+0000 7f010de00640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f00e806a870 0x7f00e806cd30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.356+0000 7f010de00640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f010815cdc0 0x7f01081f8860 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.356+0000 7f010de00640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f010815c9e0 0x7f010806ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.356+0000 7f010de00640 1 --2- 172.21.15.62:0/3649056074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01080cf650 0x7f010815d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.356+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 >> 172.21.15.62:0/3649056074 conn(0x7f0108071570 msgr2=0x7f01080c91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.357+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 shutdown_connections 2024-03-23T11:19:06.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.357+0000 7f010de00640 1 -- 172.21.15.62:0/3649056074 wait complete. 2024-03-23T11:19:06.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.359+0000 7f5ff5000640 1 -- 172.21.15.62:0/1978653568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5ff0066c90 con 0x7f5ff00b2f90 2024-03-23T11:19:06.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.359+0000 7f5fe6000640 1 -- 172.21.15.62:0/1978653568 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5fdc049090 con 0x7f5ff00b2f90 2024-03-23T11:19:06.359 INFO:teuthology.orchestra.run.smithi062.stdout:60129542150 2024-03-23T11:19:06.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.361+0000 7f5ff5000640 1 -- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5fd006a830 msgr2=0x7f5fd006ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.361+0000 7f5ff5000640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5fd006a830 0x7f5fd006ccf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0009cd0 tx=0x7f5fe0009190 comp rx=0 tx=0).stop 2024-03-23T11:19:06.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.362+0000 7f5ff5000640 1 -- 172.21.15.62:0/1978653568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ff00b2f90 msgr2=0x7f5ff01f9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.362+0000 7f5ff5000640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ff00b2f90 0x7f5ff01f9920 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0090b0 tx=0x7f5fdc0046c0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.362+0000 7f5ff5000640 1 -- 172.21.15.62:0/1978653568 shutdown_connections 2024-03-23T11:19:06.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.362+0000 7f5ff5000640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5fd006a830 0x7f5fd006ccf0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.362+0000 7f5ff5000640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ff00b2f90 0x7f5ff01f9920 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.362+0000 7f5ff5000640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff0164a30 0x7f5ff00cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.362+0000 7f5ff5000640 1 --2- 172.21.15.62:0/1978653568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ff00b15e0 0x7f5ff0156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.362+0000 7f5ff5000640 1 -- 172.21.15.62:0/1978653568 >> 172.21.15.62:0/1978653568 conn(0x7f5ff00b02e0 msgr2=0x7f5ff0163790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:06.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.362+0000 7f5ff5000640 1 -- 172.21.15.62:0/1978653568 shutdown_connections 2024-03-23T11:19:06.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.362+0000 7f5ff5000640 1 -- 172.21.15.62:0/1978653568 wait complete. 2024-03-23T11:19:06.373 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.3 2024-03-23T11:19:06.374 DEBUG:teuthology.parallel:result is None 2024-03-23T11:19:06.378 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.5 2024-03-23T11:19:06.378 DEBUG:teuthology.parallel:result is None 2024-03-23T11:19:06.378 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-23T11:19:06.378 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:19:06.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.635+0000 7efe9ec00640 1 Processor -- start 2024-03-23T11:19:06.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.635+0000 7efe9ec00640 1 -- start start 2024-03-23T11:19:06.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9ec00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efe9815fcf0 0x7efe980d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:06.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9ec00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efe98160420 0x7efe981608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:06.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe980d06d0 0x7efe980d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:06.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9ec00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe98160df0 con 0x7efe9815fcf0 2024-03-23T11:19:06.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9ec00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe980d3540 con 0x7efe98160420 2024-03-23T11:19:06.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe980ce510 con 0x7efe980d06d0 2024-03-23T11:19:06.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efe9815fcf0 0x7efe980d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:06.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efe9815fcf0 0x7efe980d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38134/0 (socket says 172.21.15.62:38134) 2024-03-23T11:19:06.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9d800640 1 -- 172.21.15.62:0/3140161500 learned_addr learned my addr 172.21.15.62:0/3140161500 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:06.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9d800640 1 -- 172.21.15.62:0/3140161500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe980d06d0 msgr2=0x7efe980d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:06.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9d800640 1 --2- 172.21.15.62:0/3140161500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe980d06d0 0x7efe980d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9d800640 1 -- 172.21.15.62:0/3140161500 >> v2:172.21.15.120:3300/0 conn(0x7efe98160420 msgr2=0x7efe981608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:06.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9d800640 1 --2- 172.21.15.62:0/3140161500 >> v2:172.21.15.120:3300/0 conn(0x7efe98160420 0x7efe981608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.637+0000 7efe9d800640 1 -- 172.21.15.62:0/3140161500 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe980cdc70 con 0x7efe9815fcf0 2024-03-23T11:19:06.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.638+0000 7efe9d800640 1 --2- 172.21.15.62:0/3140161500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe9815fcf0 0x7efe980d3000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efe98161540 tx=0x7efe8c02b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269b6d8631f9e5a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:06.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.639+0000 7efe97e00640 1 -- 172.21.15.62:0/3140161500 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c0317f0 con 0x7efe9815fcf0 2024-03-23T11:19:06.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.639+0000 7efe97e00640 1 -- 172.21.15.62:0/3140161500 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c031950 con 0x7efe9815fcf0 2024-03-23T11:19:06.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.639+0000 7efe97e00640 1 -- 172.21.15.62:0/3140161500 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c031ce0 con 0x7efe9815fcf0 2024-03-23T11:19:06.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.639+0000 7efe9ec00640 1 -- 172.21.15.62:0/3140161500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe9815fcf0 msgr2=0x7efe980d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.639+0000 7efe9ec00640 1 --2- 172.21.15.62:0/3140161500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe9815fcf0 0x7efe980d3000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efe98161540 tx=0x7efe8c02b890 comp rx=0 tx=0).stop 2024-03-23T11:19:06.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.639+0000 7efe9ec00640 1 -- 172.21.15.62:0/3140161500 shutdown_connections 2024-03-23T11:19:06.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.639+0000 7efe9ec00640 1 --2- 172.21.15.62:0/3140161500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe980d06d0 0x7efe980d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.639+0000 7efe9ec00640 1 --2- 172.21.15.62:0/3140161500 >> v2:172.21.15.120:3300/0 conn(0x7efe98160420 0x7efe981608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.639+0000 7efe9ec00640 1 --2- 172.21.15.62:0/3140161500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe9815fcf0 0x7efe980d3000 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.639+0000 7efe9ec00640 1 -- 172.21.15.62:0/3140161500 >> 172.21.15.62:0/3140161500 conn(0x7efe98089f60 msgr2=0x7efe980d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:06.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.639+0000 7efe9ec00640 1 -- 172.21.15.62:0/3140161500 shutdown_connections 2024-03-23T11:19:06.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.640+0000 7efe9ec00640 1 -- 172.21.15.62:0/3140161500 wait complete. 2024-03-23T11:19:06.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.641+0000 7efe9ec00640 1 Processor -- start 2024-03-23T11:19:06.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.641+0000 7efe9ec00640 1 -- start start 2024-03-23T11:19:06.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.642+0000 7efe9ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe980d06d0 0x7efe9815b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:06.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.642+0000 7efe9ec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9815fcf0 0x7efe98158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:06.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.642+0000 7efe9ec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe98160420 0x7efe981588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:06.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.642+0000 7efe9ec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe980cc170 con 0x7efe98160420 2024-03-23T11:19:06.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.642+0000 7efe9ec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe980cb560 con 0x7efe9815fcf0 2024-03-23T11:19:06.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.642+0000 7efe9ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe980cb840 con 0x7efe980d06d0 2024-03-23T11:19:06.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe9e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe98160420 0x7efe981588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:06.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe9e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe98160420 0x7efe981588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38150/0 (socket says 172.21.15.62:38150) 2024-03-23T11:19:06.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe9e200640 1 -- 172.21.15.62:0/3474325745 learned_addr learned my addr 172.21.15.62:0/3474325745 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:06.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe9ce00640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9815fcf0 0x7efe98158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:06.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe9d800640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe980d06d0 0x7efe9815b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:06.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe9e200640 1 -- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe980d06d0 msgr2=0x7efe9815b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe9e200640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe980d06d0 0x7efe9815b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe9e200640 1 -- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9815fcf0 msgr2=0x7efe98158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe9e200640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9815fcf0 0x7efe98158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe9e200640 1 -- 172.21.15.62:0/3474325745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c0094d0 con 0x7efe98160420 2024-03-23T11:19:06.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe9e200640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe98160420 0x7efe981588c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efe9000b970 tx=0x7efe9000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:06.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.643+0000 7efe95600640 1 -- 172.21.15.62:0/3474325745 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9000cb40 con 0x7efe98160420 2024-03-23T11:19:06.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.644+0000 7efe95600640 1 -- 172.21.15.62:0/3474325745 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9000ccc0 con 0x7efe98160420 2024-03-23T11:19:06.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.644+0000 7efe95600640 1 -- 172.21.15.62:0/3474325745 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe90014680 con 0x7efe98160420 2024-03-23T11:19:06.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.644+0000 7efe9ec00640 1 -- 172.21.15.62:0/3474325745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe980cbb20 con 0x7efe98160420 2024-03-23T11:19:06.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.644+0000 7efe9ec00640 1 -- 172.21.15.62:0/3474325745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9815b5c0 con 0x7efe98160420 2024-03-23T11:19:06.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.645+0000 7efe9ec00640 1 -- 172.21.15.62:0/3474325745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe98066bb0 con 0x7efe98160420 2024-03-23T11:19:06.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.646+0000 7efe95600640 1 -- 172.21.15.62:0/3474325745 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7efe9001d3b0 con 0x7efe98160420 2024-03-23T11:19:06.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.646+0000 7efe95600640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe7806a7a0 0x7efe7806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:06.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.647+0000 7efe95600640 1 -- 172.21.15.62:0/3474325745 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe90088d90 con 0x7efe98160420 2024-03-23T11:19:06.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.647+0000 7efe9d800640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe7806a7a0 0x7efe7806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:06.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.648+0000 7efe9d800640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe7806a7a0 0x7efe7806cc60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efe8c036000 tx=0x7efe8c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:06.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.651+0000 7efe95600640 1 -- 172.21.15.62:0/3474325745 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efe90054110 con 0x7efe98160420 2024-03-23T11:19:06.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.840+0000 7efe9ec00640 1 -- 172.21.15.62:0/3474325745 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe980b55d0 con 0x7efe7806a7a0 2024-03-23T11:19:06.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.861+0000 7efe95600640 1 -- 172.21.15.62:0/3474325745 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264436 (secure 0 0 0) 0x7efe8c031bc0 con 0x7efe7806a7a0 2024-03-23T11:19:06.866 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:06.867 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:19:06.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 -- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe7806a7a0 msgr2=0x7efe7806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe7806a7a0 0x7efe7806cc60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efe8c036000 tx=0x7efe8c03d040 comp rx=0 tx=0).stop 2024-03-23T11:19:06.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 -- 172.21.15.62:0/3474325745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe98160420 msgr2=0x7efe981588c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:06.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe98160420 0x7efe981588c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efe9000b970 tx=0x7efe9000be30 comp rx=0 tx=0).stop 2024-03-23T11:19:06.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 -- 172.21.15.62:0/3474325745 shutdown_connections 2024-03-23T11:19:06.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe7806a7a0 0x7efe7806cc60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe98160420 0x7efe981588c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe9815fcf0 0x7efe98158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 --2- 172.21.15.62:0/3474325745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe980d06d0 0x7efe9815b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:06.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 -- 172.21.15.62:0/3474325745 >> 172.21.15.62:0/3474325745 conn(0x7efe98089f60 msgr2=0x7efe980b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:06.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 -- 172.21.15.62:0/3474325745 shutdown_connections 2024-03-23T11:19:06.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:06.868+0000 7efe9ec00640 1 -- 172.21.15.62:0/3474325745 wait complete. 2024-03-23T11:19:06.885 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-23T11:19:05.234713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":2972,"kb_used_omap":13,"kb_used_meta":216050,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3043328,"data_stored":1813773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":221235288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":81,"num_write_kb":798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.994449"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257670+0000","last_change":"2024-03-23T11:18:56.274088+0000","last_active":"2024-03-23T11:18:57.257670+0000","last_peered":"2024-03-23T11:18:57.257670+0000","last_clean":"2024-03-23T11:18:57.257670+0000","last_became_active":"2024-03-23T11:18:56.273097+0000","last_became_peered":"2024-03-23T11:18:56.273097+0000","last_unstale":"2024-03-23T11:18:57.257670+0000","last_undegraded":"2024-03-23T11:18:57.257670+0000","last_fullsized":"2024-03-23T11:18:57.257670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:48:24.807457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253337+0000","last_change":"2024-03-23T11:18:56.260586+0000","last_active":"2024-03-23T11:18:57.253337+0000","last_peered":"2024-03-23T11:18:57.253337+0000","last_clean":"2024-03-23T11:18:57.253337+0000","last_became_active":"2024-03-23T11:18:56.260321+0000","last_became_peered":"2024-03-23T11:18:56.260321+0000","last_unstale":"2024-03-23T11:18:57.253337+0000","last_undegraded":"2024-03-23T11:18:57.253337+0000","last_fullsized":"2024-03-23T11:18:57.253337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:24:40.203305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099527+0000","last_change":"2024-03-23T11:18:56.264999+0000","last_active":"2024-03-23T11:18:58.099527+0000","last_peered":"2024-03-23T11:18:58.099527+0000","last_clean":"2024-03-23T11:18:58.099527+0000","last_became_active":"2024-03-23T11:18:56.264689+0000","last_became_peered":"2024-03-23T11:18:56.264689+0000","last_unstale":"2024-03-23T11:18:58.099527+0000","last_undegraded":"2024-03-23T11:18:58.099527+0000","last_fullsized":"2024-03-23T11:18:58.099527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.747240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315279+0000","last_change":"2024-03-23T11:18:56.283608+0000","last_active":"2024-03-23T11:18:57.315279+0000","last_peered":"2024-03-23T11:18:57.315279+0000","last_clean":"2024-03-23T11:18:57.315279+0000","last_became_active":"2024-03-23T11:18:56.277048+0000","last_became_peered":"2024-03-23T11:18:56.277048+0000","last_unstale":"2024-03-23T11:18:57.315279+0000","last_undegraded":"2024-03-23T11:18:57.315279+0000","last_fullsized":"2024-03-23T11:18:57.315279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:28.809711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256138+0000","last_change":"2024-03-23T11:18:56.264915+0000","last_active":"2024-03-23T11:18:57.256138+0000","last_peered":"2024-03-23T11:18:57.256138+0000","last_clean":"2024-03-23T11:18:57.256138+0000","last_became_active":"2024-03-23T11:18:56.264696+0000","last_became_peered":"2024-03-23T11:18:56.264696+0000","last_unstale":"2024-03-23T11:18:57.256138+0000","last_undegraded":"2024-03-23T11:18:57.256138+0000","last_fullsized":"2024-03-23T11:18:57.256138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:59.578102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314423+0000","last_change":"2024-03-23T11:18:56.282941+0000","last_active":"2024-03-23T11:18:57.314423+0000","last_peered":"2024-03-23T11:18:57.314423+0000","last_clean":"2024-03-23T11:18:57.314423+0000","last_became_active":"2024-03-23T11:18:56.282556+0000","last_became_peered":"2024-03-23T11:18:56.282556+0000","last_unstale":"2024-03-23T11:18:57.314423+0000","last_undegraded":"2024-03-23T11:18:57.314423+0000","last_fullsized":"2024-03-23T11:18:57.314423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:01:46.194360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255145+0000","last_change":"2024-03-23T11:18:56.274568+0000","last_active":"2024-03-23T11:18:57.255145+0000","last_peered":"2024-03-23T11:18:57.255145+0000","last_clean":"2024-03-23T11:18:57.255145+0000","last_became_active":"2024-03-23T11:18:56.273524+0000","last_became_peered":"2024-03-23T11:18:56.273524+0000","last_unstale":"2024-03-23T11:18:57.255145+0000","last_undegraded":"2024-03-23T11:18:57.255145+0000","last_fullsized":"2024-03-23T11:18:57.255145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:48:43.299392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099491+0000","last_change":"2024-03-23T11:18:56.272280+0000","last_active":"2024-03-23T11:18:58.099491+0000","last_peered":"2024-03-23T11:18:58.099491+0000","last_clean":"2024-03-23T11:18:58.099491+0000","last_became_active":"2024-03-23T11:18:56.272096+0000","last_became_peered":"2024-03-23T11:18:56.272096+0000","last_unstale":"2024-03-23T11:18:58.099491+0000","last_undegraded":"2024-03-23T11:18:58.099491+0000","last_fullsized":"2024-03-23T11:18:58.099491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:28.809711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257785+0000","last_change":"2024-03-23T11:18:56.282997+0000","last_active":"2024-03-23T11:18:57.257785+0000","last_peered":"2024-03-23T11:18:57.257785+0000","last_clean":"2024-03-23T11:18:57.257785+0000","last_became_active":"2024-03-23T11:18:56.282768+0000","last_became_peered":"2024-03-23T11:18:56.282768+0000","last_unstale":"2024-03-23T11:18:57.257785+0000","last_undegraded":"2024-03-23T11:18:57.257785+0000","last_fullsized":"2024-03-23T11:18:57.257785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:47:39.199128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461969+0000","last_change":"2024-03-23T11:18:56.269707+0000","last_active":"2024-03-23T11:18:57.461969+0000","last_peered":"2024-03-23T11:18:57.461969+0000","last_clean":"2024-03-23T11:18:57.461969+0000","last_became_active":"2024-03-23T11:18:56.266078+0000","last_became_peered":"2024-03-23T11:18:56.266078+0000","last_unstale":"2024-03-23T11:18:57.461969+0000","last_undegraded":"2024-03-23T11:18:57.461969+0000","last_fullsized":"2024-03-23T11:18:57.461969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461775+0000","last_change":"2024-03-23T11:18:56.266542+0000","last_active":"2024-03-23T11:18:57.461775+0000","last_peered":"2024-03-23T11:18:57.461775+0000","last_clean":"2024-03-23T11:18:57.461775+0000","last_became_active":"2024-03-23T11:18:56.266026+0000","last_became_peered":"2024-03-23T11:18:56.266026+0000","last_unstale":"2024-03-23T11:18:57.461775+0000","last_undegraded":"2024-03-23T11:18:57.461775+0000","last_fullsized":"2024-03-23T11:18:57.461775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255738+0000","last_change":"2024-03-23T11:18:56.272690+0000","last_active":"2024-03-23T11:18:57.255738+0000","last_peered":"2024-03-23T11:18:57.255738+0000","last_clean":"2024-03-23T11:18:57.255738+0000","last_became_active":"2024-03-23T11:18:56.272153+0000","last_became_peered":"2024-03-23T11:18:56.272153+0000","last_unstale":"2024-03-23T11:18:57.255738+0000","last_undegraded":"2024-03-23T11:18:57.255738+0000","last_fullsized":"2024-03-23T11:18:57.255738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:44:23.104322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101098+0000","last_change":"2024-03-23T11:18:56.264317+0000","last_active":"2024-03-23T11:18:58.101098+0000","last_peered":"2024-03-23T11:18:58.101098+0000","last_clean":"2024-03-23T11:18:58.101098+0000","last_became_active":"2024-03-23T11:18:56.263977+0000","last_became_peered":"2024-03-23T11:18:56.263977+0000","last_unstale":"2024-03-23T11:18:58.101098+0000","last_undegraded":"2024-03-23T11:18:58.101098+0000","last_fullsized":"2024-03-23T11:18:58.101098+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:33.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098064+0000","last_change":"2024-03-23T11:18:56.267303+0000","last_active":"2024-03-23T11:18:58.098064+0000","last_peered":"2024-03-23T11:18:58.098064+0000","last_clean":"2024-03-23T11:18:58.098064+0000","last_became_active":"2024-03-23T11:18:56.267130+0000","last_became_peered":"2024-03-23T11:18:56.267130+0000","last_unstale":"2024-03-23T11:18:58.098064+0000","last_undegraded":"2024-03-23T11:18:58.098064+0000","last_fullsized":"2024-03-23T11:18:58.098064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252603+0000","last_change":"2024-03-23T11:18:56.275581+0000","last_active":"2024-03-23T11:18:57.252603+0000","last_peered":"2024-03-23T11:18:57.252603+0000","last_clean":"2024-03-23T11:18:57.252603+0000","last_became_active":"2024-03-23T11:18:56.274566+0000","last_became_peered":"2024-03-23T11:18:56.274566+0000","last_unstale":"2024-03-23T11:18:57.252603+0000","last_undegraded":"2024-03-23T11:18:57.252603+0000","last_fullsized":"2024-03-23T11:18:57.252603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:31:43.650115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314345+0000","last_change":"2024-03-23T11:18:56.282803+0000","last_active":"2024-03-23T11:18:57.314345+0000","last_peered":"2024-03-23T11:18:57.314345+0000","last_clean":"2024-03-23T11:18:57.314345+0000","last_became_active":"2024-03-23T11:18:56.282500+0000","last_became_peered":"2024-03-23T11:18:56.282500+0000","last_unstale":"2024-03-23T11:18:57.314345+0000","last_undegraded":"2024-03-23T11:18:57.314345+0000","last_fullsized":"2024-03-23T11:18:57.314345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315513+0000","last_change":"2024-03-23T11:18:56.287431+0000","last_active":"2024-03-23T11:18:57.315513+0000","last_peered":"2024-03-23T11:18:57.315513+0000","last_clean":"2024-03-23T11:18:57.315513+0000","last_became_active":"2024-03-23T11:18:56.286853+0000","last_became_peered":"2024-03-23T11:18:56.286853+0000","last_unstale":"2024-03-23T11:18:57.315513+0000","last_undegraded":"2024-03-23T11:18:57.315513+0000","last_fullsized":"2024-03-23T11:18:57.315513+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:54:17.561394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100445+0000","last_change":"2024-03-23T11:18:56.260336+0000","last_active":"2024-03-23T11:18:58.100445+0000","last_peered":"2024-03-23T11:18:58.100445+0000","last_clean":"2024-03-23T11:18:58.100445+0000","last_became_active":"2024-03-23T11:18:56.259638+0000","last_became_peered":"2024-03-23T11:18:56.259638+0000","last_unstale":"2024-03-23T11:18:58.100445+0000","last_undegraded":"2024-03-23T11:18:58.100445+0000","last_fullsized":"2024-03-23T11:18:58.100445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:36:11.188974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257119+0000","last_change":"2024-03-23T11:18:56.268940+0000","last_active":"2024-03-23T11:18:57.257119+0000","last_peered":"2024-03-23T11:18:57.257119+0000","last_clean":"2024-03-23T11:18:57.257119+0000","last_became_active":"2024-03-23T11:18:56.268168+0000","last_became_peered":"2024-03-23T11:18:56.268168+0000","last_unstale":"2024-03-23T11:18:57.257119+0000","last_undegraded":"2024-03-23T11:18:57.257119+0000","last_fullsized":"2024-03-23T11:18:57.257119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:22.782018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461751+0000","last_change":"2024-03-23T11:18:56.271021+0000","last_active":"2024-03-23T11:18:57.461751+0000","last_peered":"2024-03-23T11:18:57.461751+0000","last_clean":"2024-03-23T11:18:57.461751+0000","last_became_active":"2024-03-23T11:18:56.270766+0000","last_became_peered":"2024-03-23T11:18:56.270766+0000","last_unstale":"2024-03-23T11:18:57.461751+0000","last_undegraded":"2024-03-23T11:18:57.461751+0000","last_fullsized":"2024-03-23T11:18:57.461751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098354+0000","last_change":"2024-03-23T11:18:56.261635+0000","last_active":"2024-03-23T11:18:58.098354+0000","last_peered":"2024-03-23T11:18:58.098354+0000","last_clean":"2024-03-23T11:18:58.098354+0000","last_became_active":"2024-03-23T11:18:56.260741+0000","last_became_peered":"2024-03-23T11:18:56.260741+0000","last_unstale":"2024-03-23T11:18:58.098354+0000","last_undegraded":"2024-03-23T11:18:58.098354+0000","last_fullsized":"2024-03-23T11:18:58.098354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314890+0000","last_change":"2024-03-23T11:18:56.286882+0000","last_active":"2024-03-23T11:18:57.314890+0000","last_peered":"2024-03-23T11:18:57.314890+0000","last_clean":"2024-03-23T11:18:57.314890+0000","last_became_active":"2024-03-23T11:18:56.285291+0000","last_became_peered":"2024-03-23T11:18:56.285291+0000","last_unstale":"2024-03-23T11:18:57.314890+0000","last_undegraded":"2024-03-23T11:18:57.314890+0000","last_fullsized":"2024-03-23T11:18:57.314890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:56.858523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101535+0000","last_change":"2024-03-23T11:18:56.276050+0000","last_active":"2024-03-23T11:18:58.101535+0000","last_peered":"2024-03-23T11:18:58.101535+0000","last_clean":"2024-03-23T11:18:58.101535+0000","last_became_active":"2024-03-23T11:18:56.275264+0000","last_became_peered":"2024-03-23T11:18:56.275264+0000","last_unstale":"2024-03-23T11:18:58.101535+0000","last_undegraded":"2024-03-23T11:18:58.101535+0000","last_fullsized":"2024-03-23T11:18:58.101535+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314573+0000","last_change":"2024-03-23T11:18:56.283292+0000","last_active":"2024-03-23T11:18:57.314573+0000","last_peered":"2024-03-23T11:18:57.314573+0000","last_clean":"2024-03-23T11:18:57.314573+0000","last_became_active":"2024-03-23T11:18:56.282493+0000","last_became_peered":"2024-03-23T11:18:56.282493+0000","last_unstale":"2024-03-23T11:18:57.314573+0000","last_undegraded":"2024-03-23T11:18:57.314573+0000","last_fullsized":"2024-03-23T11:18:57.314573+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101895+0000","last_change":"2024-03-23T11:18:56.282679+0000","last_active":"2024-03-23T11:18:58.101895+0000","last_peered":"2024-03-23T11:18:58.101895+0000","last_clean":"2024-03-23T11:18:58.101895+0000","last_became_active":"2024-03-23T11:18:56.282434+0000","last_became_peered":"2024-03-23T11:18:56.282434+0000","last_unstale":"2024-03-23T11:18:58.101895+0000","last_undegraded":"2024-03-23T11:18:58.101895+0000","last_fullsized":"2024-03-23T11:18:58.101895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:00.423263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462105+0000","last_change":"2024-03-23T11:18:56.260538+0000","last_active":"2024-03-23T11:18:57.462105+0000","last_peered":"2024-03-23T11:18:57.462105+0000","last_clean":"2024-03-23T11:18:57.462105+0000","last_became_active":"2024-03-23T11:18:56.259279+0000","last_became_peered":"2024-03-23T11:18:56.259279+0000","last_unstale":"2024-03-23T11:18:57.462105+0000","last_undegraded":"2024-03-23T11:18:57.462105+0000","last_fullsized":"2024-03-23T11:18:57.462105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253224+0000","last_change":"2024-03-23T11:18:56.263336+0000","last_active":"2024-03-23T11:18:57.253224+0000","last_peered":"2024-03-23T11:18:57.253224+0000","last_clean":"2024-03-23T11:18:57.253224+0000","last_became_active":"2024-03-23T11:18:56.262573+0000","last_became_peered":"2024-03-23T11:18:56.262573+0000","last_unstale":"2024-03-23T11:18:57.253224+0000","last_undegraded":"2024-03-23T11:18:57.253224+0000","last_fullsized":"2024-03-23T11:18:57.253224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:41:50.064223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101071+0000","last_change":"2024-03-23T11:18:56.284862+0000","last_active":"2024-03-23T11:18:58.101071+0000","last_peered":"2024-03-23T11:18:58.101071+0000","last_clean":"2024-03-23T11:18:58.101071+0000","last_became_active":"2024-03-23T11:18:56.284599+0000","last_became_peered":"2024-03-23T11:18:56.284599+0000","last_unstale":"2024-03-23T11:18:58.101071+0000","last_undegraded":"2024-03-23T11:18:58.101071+0000","last_fullsized":"2024-03-23T11:18:58.101071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.705001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098596+0000","last_change":"2024-03-23T11:18:56.271496+0000","last_active":"2024-03-23T11:18:58.098596+0000","last_peered":"2024-03-23T11:18:58.098596+0000","last_clean":"2024-03-23T11:18:58.098596+0000","last_became_active":"2024-03-23T11:18:56.271243+0000","last_became_peered":"2024-03-23T11:18:56.271243+0000","last_unstale":"2024-03-23T11:18:58.098596+0000","last_undegraded":"2024-03-23T11:18:58.098596+0000","last_fullsized":"2024-03-23T11:18:58.098596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:16:46.738802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314711+0000","last_change":"2024-03-23T11:18:56.282267+0000","last_active":"2024-03-23T11:18:57.314711+0000","last_peered":"2024-03-23T11:18:57.314711+0000","last_clean":"2024-03-23T11:18:57.314711+0000","last_became_active":"2024-03-23T11:18:56.277544+0000","last_became_peered":"2024-03-23T11:18:56.277544+0000","last_unstale":"2024-03-23T11:18:57.314711+0000","last_undegraded":"2024-03-23T11:18:57.314711+0000","last_fullsized":"2024-03-23T11:18:57.314711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:12.916824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098163+0000","last_change":"2024-03-23T11:18:56.267136+0000","last_active":"2024-03-23T11:18:58.098163+0000","last_peered":"2024-03-23T11:18:58.098163+0000","last_clean":"2024-03-23T11:18:58.098163+0000","last_became_active":"2024-03-23T11:18:56.266640+0000","last_became_peered":"2024-03-23T11:18:56.266640+0000","last_unstale":"2024-03-23T11:18:58.098163+0000","last_undegraded":"2024-03-23T11:18:58.098163+0000","last_fullsized":"2024-03-23T11:18:58.098163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461393+0000","last_change":"2024-03-23T11:18:56.266829+0000","last_active":"2024-03-23T11:18:57.461393+0000","last_peered":"2024-03-23T11:18:57.461393+0000","last_clean":"2024-03-23T11:18:57.461393+0000","last_became_active":"2024-03-23T11:18:56.266536+0000","last_became_peered":"2024-03-23T11:18:56.266536+0000","last_unstale":"2024-03-23T11:18:57.461393+0000","last_undegraded":"2024-03-23T11:18:57.461393+0000","last_fullsized":"2024-03-23T11:18:57.461393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:00.423263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255237+0000","last_change":"2024-03-23T11:18:56.267024+0000","last_active":"2024-03-23T11:18:57.255237+0000","last_peered":"2024-03-23T11:18:57.255237+0000","last_clean":"2024-03-23T11:18:57.255237+0000","last_became_active":"2024-03-23T11:18:56.266730+0000","last_became_peered":"2024-03-23T11:18:56.266730+0000","last_unstale":"2024-03-23T11:18:57.255237+0000","last_undegraded":"2024-03-23T11:18:57.255237+0000","last_fullsized":"2024-03-23T11:18:57.255237+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:19.533798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101417+0000","last_change":"2024-03-23T11:18:55.778856+0000","last_active":"2024-03-23T11:18:58.101417+0000","last_peered":"2024-03-23T11:18:58.101417+0000","last_clean":"2024-03-23T11:18:58.101417+0000","last_became_active":"2024-03-23T11:18:55.778487+0000","last_became_peered":"2024-03-23T11:18:55.778487+0000","last_unstale":"2024-03-23T11:18:58.101417+0000","last_undegraded":"2024-03-23T11:18:58.101417+0000","last_fullsized":"2024-03-23T11:18:58.101417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098890+0000","last_change":"2024-03-23T11:18:56.280876+0000","last_active":"2024-03-23T11:18:58.098890+0000","last_peered":"2024-03-23T11:18:58.098890+0000","last_clean":"2024-03-23T11:18:58.098890+0000","last_became_active":"2024-03-23T11:18:56.280574+0000","last_became_peered":"2024-03-23T11:18:56.280574+0000","last_unstale":"2024-03-23T11:18:58.098890+0000","last_undegraded":"2024-03-23T11:18:58.098890+0000","last_fullsized":"2024-03-23T11:18:58.098890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:25.559049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462037+0000","last_change":"2024-03-23T11:18:55.771275+0000","last_active":"2024-03-23T11:18:57.462037+0000","last_peered":"2024-03-23T11:18:57.462037+0000","last_clean":"2024-03-23T11:18:57.462037+0000","last_became_active":"2024-03-23T11:18:55.770948+0000","last_became_peered":"2024-03-23T11:18:55.770948+0000","last_unstale":"2024-03-23T11:18:57.462037+0000","last_undegraded":"2024-03-23T11:18:57.462037+0000","last_fullsized":"2024-03-23T11:18:57.462037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101598+0000","last_change":"2024-03-23T11:18:55.780718+0000","last_active":"2024-03-23T11:18:58.101598+0000","last_peered":"2024-03-23T11:18:58.101598+0000","last_clean":"2024-03-23T11:18:58.101598+0000","last_became_active":"2024-03-23T11:18:55.779357+0000","last_became_peered":"2024-03-23T11:18:55.779357+0000","last_unstale":"2024-03-23T11:18:58.101598+0000","last_undegraded":"2024-03-23T11:18:58.101598+0000","last_fullsized":"2024-03-23T11:18:58.101598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:36:21.274877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099752+0000","last_change":"2024-03-23T11:18:56.266172+0000","last_active":"2024-03-23T11:18:58.099752+0000","last_peered":"2024-03-23T11:18:58.099752+0000","last_clean":"2024-03-23T11:18:58.099752+0000","last_became_active":"2024-03-23T11:18:56.265882+0000","last_became_peered":"2024-03-23T11:18:56.265882+0000","last_unstale":"2024-03-23T11:18:58.099752+0000","last_undegraded":"2024-03-23T11:18:58.099752+0000","last_fullsized":"2024-03-23T11:18:58.099752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314638+0000","last_change":"2024-03-23T11:18:55.774144+0000","last_active":"2024-03-23T11:18:57.314638+0000","last_peered":"2024-03-23T11:18:57.314638+0000","last_clean":"2024-03-23T11:18:57.314638+0000","last_became_active":"2024-03-23T11:18:55.773880+0000","last_became_peered":"2024-03-23T11:18:55.773880+0000","last_unstale":"2024-03-23T11:18:57.314638+0000","last_undegraded":"2024-03-23T11:18:57.314638+0000","last_fullsized":"2024-03-23T11:18:57.314638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252538+0000","last_change":"2024-03-23T11:18:56.260698+0000","last_active":"2024-03-23T11:18:57.252538+0000","last_peered":"2024-03-23T11:18:57.252538+0000","last_clean":"2024-03-23T11:18:57.252538+0000","last_became_active":"2024-03-23T11:18:56.258466+0000","last_became_peered":"2024-03-23T11:18:56.258466+0000","last_unstale":"2024-03-23T11:18:57.252538+0000","last_undegraded":"2024-03-23T11:18:57.252538+0000","last_fullsized":"2024-03-23T11:18:57.252538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:55:22.272329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462086+0000","last_change":"2024-03-23T11:18:55.783554+0000","last_active":"2024-03-23T11:18:57.462086+0000","last_peered":"2024-03-23T11:18:57.462086+0000","last_clean":"2024-03-23T11:18:57.462086+0000","last_became_active":"2024-03-23T11:18:55.783248+0000","last_became_peered":"2024-03-23T11:18:55.783248+0000","last_unstale":"2024-03-23T11:18:57.462086+0000","last_undegraded":"2024-03-23T11:18:57.462086+0000","last_fullsized":"2024-03-23T11:18:57.462086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:18.352259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314908+0000","last_change":"2024-03-23T11:18:56.287651+0000","last_active":"2024-03-23T11:18:57.314908+0000","last_peered":"2024-03-23T11:18:57.314908+0000","last_clean":"2024-03-23T11:18:57.314908+0000","last_became_active":"2024-03-23T11:18:56.287312+0000","last_became_peered":"2024-03-23T11:18:56.287312+0000","last_unstale":"2024-03-23T11:18:57.314908+0000","last_undegraded":"2024-03-23T11:18:57.314908+0000","last_fullsized":"2024-03-23T11:18:57.314908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:24:29.439358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314950+0000","last_change":"2024-03-23T11:18:55.783117+0000","last_active":"2024-03-23T11:18:57.314950+0000","last_peered":"2024-03-23T11:18:57.314950+0000","last_clean":"2024-03-23T11:18:57.314950+0000","last_became_active":"2024-03-23T11:18:55.782341+0000","last_became_peered":"2024-03-23T11:18:55.782341+0000","last_unstale":"2024-03-23T11:18:57.314950+0000","last_undegraded":"2024-03-23T11:18:57.314950+0000","last_fullsized":"2024-03-23T11:18:57.314950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:10:08.943114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099470+0000","last_change":"2024-03-23T11:18:56.290381+0000","last_active":"2024-03-23T11:18:58.099470+0000","last_peered":"2024-03-23T11:18:58.099470+0000","last_clean":"2024-03-23T11:18:58.099470+0000","last_became_active":"2024-03-23T11:18:56.290070+0000","last_became_peered":"2024-03-23T11:18:56.290070+0000","last_unstale":"2024-03-23T11:18:58.099470+0000","last_undegraded":"2024-03-23T11:18:58.099470+0000","last_fullsized":"2024-03-23T11:18:58.099470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253428+0000","last_change":"2024-03-23T11:18:55.783461+0000","last_active":"2024-03-23T11:18:57.253428+0000","last_peered":"2024-03-23T11:18:57.253428+0000","last_clean":"2024-03-23T11:18:57.253428+0000","last_became_active":"2024-03-23T11:18:55.783214+0000","last_became_peered":"2024-03-23T11:18:55.783214+0000","last_unstale":"2024-03-23T11:18:57.253428+0000","last_undegraded":"2024-03-23T11:18:57.253428+0000","last_fullsized":"2024-03-23T11:18:57.253428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:28:51.146588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255996+0000","last_change":"2024-03-23T11:18:56.268161+0000","last_active":"2024-03-23T11:18:57.255996+0000","last_peered":"2024-03-23T11:18:57.255996+0000","last_clean":"2024-03-23T11:18:57.255996+0000","last_became_active":"2024-03-23T11:18:56.263992+0000","last_became_peered":"2024-03-23T11:18:56.263992+0000","last_unstale":"2024-03-23T11:18:57.255996+0000","last_undegraded":"2024-03-23T11:18:57.255996+0000","last_fullsized":"2024-03-23T11:18:57.255996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461733+0000","last_change":"2024-03-23T11:18:55.782334+0000","last_active":"2024-03-23T11:18:57.461733+0000","last_peered":"2024-03-23T11:18:57.461733+0000","last_clean":"2024-03-23T11:18:57.461733+0000","last_became_active":"2024-03-23T11:18:55.780347+0000","last_became_peered":"2024-03-23T11:18:55.780347+0000","last_unstale":"2024-03-23T11:18:57.461733+0000","last_undegraded":"2024-03-23T11:18:57.461733+0000","last_fullsized":"2024-03-23T11:18:57.461733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:10:08.943114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256551+0000","last_change":"2024-03-23T11:18:56.272684+0000","last_active":"2024-03-23T11:18:57.256551+0000","last_peered":"2024-03-23T11:18:57.256551+0000","last_clean":"2024-03-23T11:18:57.256551+0000","last_became_active":"2024-03-23T11:18:56.268603+0000","last_became_peered":"2024-03-23T11:18:56.268603+0000","last_unstale":"2024-03-23T11:18:57.256551+0000","last_undegraded":"2024-03-23T11:18:57.256551+0000","last_fullsized":"2024-03-23T11:18:57.256551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:22.095583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100170+0000","last_change":"2024-03-23T11:18:55.783962+0000","last_active":"2024-03-23T11:18:58.100170+0000","last_peered":"2024-03-23T11:18:58.100170+0000","last_clean":"2024-03-23T11:18:58.100170+0000","last_became_active":"2024-03-23T11:18:55.783729+0000","last_became_peered":"2024-03-23T11:18:55.783729+0000","last_unstale":"2024-03-23T11:18:58.100170+0000","last_undegraded":"2024-03-23T11:18:58.100170+0000","last_fullsized":"2024-03-23T11:18:58.100170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:42.674448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098331+0000","last_change":"2024-03-23T11:18:56.261635+0000","last_active":"2024-03-23T11:18:58.098331+0000","last_peered":"2024-03-23T11:18:58.098331+0000","last_clean":"2024-03-23T11:18:58.098331+0000","last_became_active":"2024-03-23T11:18:56.261334+0000","last_became_peered":"2024-03-23T11:18:56.261334+0000","last_unstale":"2024-03-23T11:18:58.098331+0000","last_undegraded":"2024-03-23T11:18:58.098331+0000","last_fullsized":"2024-03-23T11:18:58.098331+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314958+0000","last_change":"2024-03-23T11:18:55.782301+0000","last_active":"2024-03-23T11:18:57.314958+0000","last_peered":"2024-03-23T11:18:57.314958+0000","last_clean":"2024-03-23T11:18:57.314958+0000","last_became_active":"2024-03-23T11:18:55.782005+0000","last_became_peered":"2024-03-23T11:18:55.782005+0000","last_unstale":"2024-03-23T11:18:57.314958+0000","last_undegraded":"2024-03-23T11:18:57.314958+0000","last_fullsized":"2024-03-23T11:18:57.314958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256246+0000","last_change":"2024-03-23T11:18:56.275199+0000","last_active":"2024-03-23T11:18:57.256246+0000","last_peered":"2024-03-23T11:18:57.256246+0000","last_clean":"2024-03-23T11:18:57.256246+0000","last_became_active":"2024-03-23T11:18:56.274944+0000","last_became_peered":"2024-03-23T11:18:56.274944+0000","last_unstale":"2024-03-23T11:18:57.256246+0000","last_undegraded":"2024-03-23T11:18:57.256246+0000","last_fullsized":"2024-03-23T11:18:57.256246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:00:18.525010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461578+0000","last_change":"2024-03-23T11:18:55.775042+0000","last_active":"2024-03-23T11:18:57.461578+0000","last_peered":"2024-03-23T11:18:57.461578+0000","last_clean":"2024-03-23T11:18:57.461578+0000","last_became_active":"2024-03-23T11:18:55.774721+0000","last_became_peered":"2024-03-23T11:18:55.774721+0000","last_unstale":"2024-03-23T11:18:57.461578+0000","last_undegraded":"2024-03-23T11:18:57.461578+0000","last_fullsized":"2024-03-23T11:18:57.461578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461383+0000","last_change":"2024-03-23T11:18:56.263485+0000","last_active":"2024-03-23T11:18:57.461383+0000","last_peered":"2024-03-23T11:18:57.461383+0000","last_clean":"2024-03-23T11:18:57.461383+0000","last_became_active":"2024-03-23T11:18:56.261677+0000","last_became_peered":"2024-03-23T11:18:56.261677+0000","last_unstale":"2024-03-23T11:18:57.461383+0000","last_undegraded":"2024-03-23T11:18:57.461383+0000","last_fullsized":"2024-03-23T11:18:57.461383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098165+0000","last_change":"2024-03-23T11:18:55.770098+0000","last_active":"2024-03-23T11:18:58.098165+0000","last_peered":"2024-03-23T11:18:58.098165+0000","last_clean":"2024-03-23T11:18:58.098165+0000","last_became_active":"2024-03-23T11:18:55.769814+0000","last_became_peered":"2024-03-23T11:18:55.769814+0000","last_unstale":"2024-03-23T11:18:58.098165+0000","last_undegraded":"2024-03-23T11:18:58.098165+0000","last_fullsized":"2024-03-23T11:18:58.098165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255419+0000","last_change":"2024-03-23T11:18:56.274715+0000","last_active":"2024-03-23T11:18:57.255419+0000","last_peered":"2024-03-23T11:18:57.255419+0000","last_clean":"2024-03-23T11:18:57.255419+0000","last_became_active":"2024-03-23T11:18:56.274437+0000","last_became_peered":"2024-03-23T11:18:56.274437+0000","last_unstale":"2024-03-23T11:18:57.255419+0000","last_undegraded":"2024-03-23T11:18:57.255419+0000","last_fullsized":"2024-03-23T11:18:57.255419+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:28:00.964447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256032+0000","last_change":"2024-03-23T11:18:55.789682+0000","last_active":"2024-03-23T11:18:57.256032+0000","last_peered":"2024-03-23T11:18:57.256032+0000","last_clean":"2024-03-23T11:18:57.256032+0000","last_became_active":"2024-03-23T11:18:55.789278+0000","last_became_peered":"2024-03-23T11:18:55.789278+0000","last_unstale":"2024-03-23T11:18:57.256032+0000","last_undegraded":"2024-03-23T11:18:57.256032+0000","last_fullsized":"2024-03-23T11:18:57.256032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:42.674448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256896+0000","last_change":"2024-03-23T11:18:56.267163+0000","last_active":"2024-03-23T11:18:57.256896+0000","last_peered":"2024-03-23T11:18:57.256896+0000","last_clean":"2024-03-23T11:18:57.256896+0000","last_became_active":"2024-03-23T11:18:56.266907+0000","last_became_peered":"2024-03-23T11:18:56.266907+0000","last_unstale":"2024-03-23T11:18:57.256896+0000","last_undegraded":"2024-03-23T11:18:57.256896+0000","last_fullsized":"2024-03-23T11:18:57.256896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:51:52.682495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100710+0000","last_change":"2024-03-23T11:18:55.779353+0000","last_active":"2024-03-23T11:18:58.100710+0000","last_peered":"2024-03-23T11:18:58.100710+0000","last_clean":"2024-03-23T11:18:58.100710+0000","last_became_active":"2024-03-23T11:18:55.779045+0000","last_became_peered":"2024-03-23T11:18:55.779045+0000","last_unstale":"2024-03-23T11:18:58.100710+0000","last_undegraded":"2024-03-23T11:18:58.100710+0000","last_fullsized":"2024-03-23T11:18:58.100710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257949+0000","last_change":"2024-03-23T11:18:56.275906+0000","last_active":"2024-03-23T11:18:57.257949+0000","last_peered":"2024-03-23T11:18:57.257949+0000","last_clean":"2024-03-23T11:18:57.257949+0000","last_became_active":"2024-03-23T11:18:56.274500+0000","last_became_peered":"2024-03-23T11:18:56.274500+0000","last_unstale":"2024-03-23T11:18:57.257949+0000","last_undegraded":"2024-03-23T11:18:57.257949+0000","last_fullsized":"2024-03-23T11:18:57.257949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:13:34.436113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253366+0000","last_change":"2024-03-23T11:18:55.763585+0000","last_active":"2024-03-23T11:18:57.253366+0000","last_peered":"2024-03-23T11:18:57.253366+0000","last_clean":"2024-03-23T11:18:57.253366+0000","last_became_active":"2024-03-23T11:18:55.762854+0000","last_became_peered":"2024-03-23T11:18:55.762854+0000","last_unstale":"2024-03-23T11:18:57.253366+0000","last_undegraded":"2024-03-23T11:18:57.253366+0000","last_fullsized":"2024-03-23T11:18:57.253366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:38.995255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315476+0000","last_change":"2024-03-23T11:18:56.276682+0000","last_active":"2024-03-23T11:18:57.315476+0000","last_peered":"2024-03-23T11:18:57.315476+0000","last_clean":"2024-03-23T11:18:57.315476+0000","last_became_active":"2024-03-23T11:18:56.276094+0000","last_became_peered":"2024-03-23T11:18:56.276094+0000","last_unstale":"2024-03-23T11:18:57.315476+0000","last_undegraded":"2024-03-23T11:18:57.315476+0000","last_fullsized":"2024-03-23T11:18:57.315476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:26.126571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098569+0000","last_change":"2024-03-23T11:18:55.783331+0000","last_active":"2024-03-23T11:18:58.098569+0000","last_peered":"2024-03-23T11:18:58.098569+0000","last_clean":"2024-03-23T11:18:58.098569+0000","last_became_active":"2024-03-23T11:18:55.783030+0000","last_became_peered":"2024-03-23T11:18:55.783030+0000","last_unstale":"2024-03-23T11:18:58.098569+0000","last_undegraded":"2024-03-23T11:18:58.098569+0000","last_fullsized":"2024-03-23T11:18:58.098569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256897+0000","last_change":"2024-03-23T11:18:56.284587+0000","last_active":"2024-03-23T11:18:57.256897+0000","last_peered":"2024-03-23T11:18:57.256897+0000","last_clean":"2024-03-23T11:18:57.256897+0000","last_became_active":"2024-03-23T11:18:56.283582+0000","last_became_peered":"2024-03-23T11:18:56.283582+0000","last_unstale":"2024-03-23T11:18:57.256897+0000","last_undegraded":"2024-03-23T11:18:57.256897+0000","last_fullsized":"2024-03-23T11:18:57.256897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:01:11.724895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":164,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T11:18:56.246747+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:18:56.246747+0000","last_peered":"2024-03-23T11:18:56.246747+0000","last_clean":"2024-03-23T11:18:56.246747+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:18:56.246747+0000","last_undegraded":"2024-03-23T11:18:56.246747+0000","last_fullsized":"2024-03-23T11:18:56.246747+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314507+0000","last_change":"2024-03-23T11:18:55.778697+0000","last_active":"2024-03-23T11:18:57.314507+0000","last_peered":"2024-03-23T11:18:57.314507+0000","last_clean":"2024-03-23T11:18:57.314507+0000","last_became_active":"2024-03-23T11:18:55.778387+0000","last_became_peered":"2024-03-23T11:18:55.778387+0000","last_unstale":"2024-03-23T11:18:57.314507+0000","last_undegraded":"2024-03-23T11:18:57.314507+0000","last_fullsized":"2024-03-23T11:18:57.314507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100582+0000","last_change":"2024-03-23T11:18:56.268261+0000","last_active":"2024-03-23T11:18:58.100582+0000","last_peered":"2024-03-23T11:18:58.100582+0000","last_clean":"2024-03-23T11:18:58.100582+0000","last_became_active":"2024-03-23T11:18:56.267974+0000","last_became_peered":"2024-03-23T11:18:56.267974+0000","last_unstale":"2024-03-23T11:18:58.100582+0000","last_undegraded":"2024-03-23T11:18:58.100582+0000","last_fullsized":"2024-03-23T11:18:58.100582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:44.879627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255801+0000","last_change":"2024-03-23T11:18:55.775332+0000","last_active":"2024-03-23T11:18:57.255801+0000","last_peered":"2024-03-23T11:18:57.255801+0000","last_clean":"2024-03-23T11:18:57.255801+0000","last_became_active":"2024-03-23T11:18:55.775009+0000","last_became_peered":"2024-03-23T11:18:55.775009+0000","last_unstale":"2024-03-23T11:18:57.255801+0000","last_undegraded":"2024-03-23T11:18:57.255801+0000","last_fullsized":"2024-03-23T11:18:57.255801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:08:12.342783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098600+0000","last_change":"2024-03-23T11:18:56.279541+0000","last_active":"2024-03-23T11:18:58.098600+0000","last_peered":"2024-03-23T11:18:58.098600+0000","last_clean":"2024-03-23T11:18:58.098600+0000","last_became_active":"2024-03-23T11:18:56.279231+0000","last_became_peered":"2024-03-23T11:18:56.279231+0000","last_unstale":"2024-03-23T11:18:58.098600+0000","last_undegraded":"2024-03-23T11:18:58.098600+0000","last_fullsized":"2024-03-23T11:18:58.098600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098389+0000","last_change":"2024-03-23T11:18:55.777288+0000","last_active":"2024-03-23T11:18:58.098389+0000","last_peered":"2024-03-23T11:18:58.098389+0000","last_clean":"2024-03-23T11:18:58.098389+0000","last_became_active":"2024-03-23T11:18:55.776998+0000","last_became_peered":"2024-03-23T11:18:55.776998+0000","last_unstale":"2024-03-23T11:18:58.098389+0000","last_undegraded":"2024-03-23T11:18:58.098389+0000","last_fullsized":"2024-03-23T11:18:58.098389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252646+0000","last_change":"2024-03-23T11:18:56.257547+0000","last_active":"2024-03-23T11:18:57.252646+0000","last_peered":"2024-03-23T11:18:57.252646+0000","last_clean":"2024-03-23T11:18:57.252646+0000","last_became_active":"2024-03-23T11:18:56.257307+0000","last_became_peered":"2024-03-23T11:18:56.257307+0000","last_unstale":"2024-03-23T11:18:57.252646+0000","last_undegraded":"2024-03-23T11:18:57.252646+0000","last_fullsized":"2024-03-23T11:18:57.252646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:32.498133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315167+0000","last_change":"2024-03-23T11:18:55.780573+0000","last_active":"2024-03-23T11:18:57.315167+0000","last_peered":"2024-03-23T11:18:57.315167+0000","last_clean":"2024-03-23T11:18:57.315167+0000","last_became_active":"2024-03-23T11:18:55.779963+0000","last_became_peered":"2024-03-23T11:18:55.779963+0000","last_unstale":"2024-03-23T11:18:57.315167+0000","last_undegraded":"2024-03-23T11:18:57.315167+0000","last_fullsized":"2024-03-23T11:18:57.315167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:18.352259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255291+0000","last_change":"2024-03-23T11:18:56.264913+0000","last_active":"2024-03-23T11:18:57.255291+0000","last_peered":"2024-03-23T11:18:57.255291+0000","last_clean":"2024-03-23T11:18:57.255291+0000","last_became_active":"2024-03-23T11:18:56.264679+0000","last_became_peered":"2024-03-23T11:18:56.264679+0000","last_unstale":"2024-03-23T11:18:57.255291+0000","last_undegraded":"2024-03-23T11:18:57.255291+0000","last_fullsized":"2024-03-23T11:18:57.255291+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:16:46.738802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253124+0000","last_change":"2024-03-23T11:18:55.785635+0000","last_active":"2024-03-23T11:18:57.253124+0000","last_peered":"2024-03-23T11:18:57.253124+0000","last_clean":"2024-03-23T11:18:57.253124+0000","last_became_active":"2024-03-23T11:18:55.785064+0000","last_became_peered":"2024-03-23T11:18:55.785064+0000","last_unstale":"2024-03-23T11:18:57.253124+0000","last_undegraded":"2024-03-23T11:18:57.253124+0000","last_fullsized":"2024-03-23T11:18:57.253124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:04:27.564911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315028+0000","last_change":"2024-03-23T11:18:56.276583+0000","last_active":"2024-03-23T11:18:57.315028+0000","last_peered":"2024-03-23T11:18:57.315028+0000","last_clean":"2024-03-23T11:18:57.315028+0000","last_became_active":"2024-03-23T11:18:56.276294+0000","last_became_peered":"2024-03-23T11:18:56.276294+0000","last_unstale":"2024-03-23T11:18:57.315028+0000","last_undegraded":"2024-03-23T11:18:57.315028+0000","last_fullsized":"2024-03-23T11:18:57.315028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:36.411878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257226+0000","last_change":"2024-03-23T11:18:55.784141+0000","last_active":"2024-03-23T11:18:57.257226+0000","last_peered":"2024-03-23T11:18:57.257226+0000","last_clean":"2024-03-23T11:18:57.257226+0000","last_became_active":"2024-03-23T11:18:55.783843+0000","last_became_peered":"2024-03-23T11:18:55.783843+0000","last_unstale":"2024-03-23T11:18:57.257226+0000","last_undegraded":"2024-03-23T11:18:57.257226+0000","last_fullsized":"2024-03-23T11:18:57.257226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:57:29.038916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252794+0000","last_change":"2024-03-23T11:18:56.258329+0000","last_active":"2024-03-23T11:18:57.252794+0000","last_peered":"2024-03-23T11:18:57.252794+0000","last_clean":"2024-03-23T11:18:57.252794+0000","last_became_active":"2024-03-23T11:18:56.258033+0000","last_became_peered":"2024-03-23T11:18:56.258033+0000","last_unstale":"2024-03-23T11:18:57.252794+0000","last_undegraded":"2024-03-23T11:18:57.252794+0000","last_fullsized":"2024-03-23T11:18:57.252794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:24:37.470160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.314065+0000","last_change":"2024-03-23T11:18:55.778103+0000","last_active":"2024-03-23T11:18:58.314065+0000","last_peered":"2024-03-23T11:18:58.314065+0000","last_clean":"2024-03-23T11:18:58.314065+0000","last_became_active":"2024-03-23T11:18:55.777604+0000","last_became_peered":"2024-03-23T11:18:55.777604+0000","last_unstale":"2024-03-23T11:18:58.314065+0000","last_undegraded":"2024-03-23T11:18:58.314065+0000","last_fullsized":"2024-03-23T11:18:58.314065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:42:44.181703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101839+0000","last_change":"2024-03-23T11:18:56.280720+0000","last_active":"2024-03-23T11:18:58.101839+0000","last_peered":"2024-03-23T11:18:58.101839+0000","last_clean":"2024-03-23T11:18:58.101839+0000","last_became_active":"2024-03-23T11:18:56.278952+0000","last_became_peered":"2024-03-23T11:18:56.278952+0000","last_unstale":"2024-03-23T11:18:58.101839+0000","last_undegraded":"2024-03-23T11:18:58.101839+0000","last_fullsized":"2024-03-23T11:18:58.101839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462270+0000","last_change":"2024-03-23T11:18:55.776925+0000","last_active":"2024-03-23T11:18:57.462270+0000","last_peered":"2024-03-23T11:18:57.462270+0000","last_clean":"2024-03-23T11:18:57.462270+0000","last_became_active":"2024-03-23T11:18:55.776651+0000","last_became_peered":"2024-03-23T11:18:55.776651+0000","last_unstale":"2024-03-23T11:18:57.462270+0000","last_undegraded":"2024-03-23T11:18:57.462270+0000","last_fullsized":"2024-03-23T11:18:57.462270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099552+0000","last_change":"2024-03-23T11:18:56.268554+0000","last_active":"2024-03-23T11:18:58.099552+0000","last_peered":"2024-03-23T11:18:58.099552+0000","last_clean":"2024-03-23T11:18:58.099552+0000","last_became_active":"2024-03-23T11:18:56.267201+0000","last_became_peered":"2024-03-23T11:18:56.267201+0000","last_unstale":"2024-03-23T11:18:58.099552+0000","last_undegraded":"2024-03-23T11:18:58.099552+0000","last_fullsized":"2024-03-23T11:18:58.099552+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.705001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255562+0000","last_change":"2024-03-23T11:18:55.789376+0000","last_active":"2024-03-23T11:18:57.255562+0000","last_peered":"2024-03-23T11:18:57.255562+0000","last_clean":"2024-03-23T11:18:57.255562+0000","last_became_active":"2024-03-23T11:18:55.789122+0000","last_became_peered":"2024-03-23T11:18:55.789122+0000","last_unstale":"2024-03-23T11:18:57.255562+0000","last_undegraded":"2024-03-23T11:18:57.255562+0000","last_fullsized":"2024-03-23T11:18:57.255562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:26.469433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257058+0000","last_change":"2024-03-23T11:18:56.267674+0000","last_active":"2024-03-23T11:18:57.257058+0000","last_peered":"2024-03-23T11:18:57.257058+0000","last_clean":"2024-03-23T11:18:57.257058+0000","last_became_active":"2024-03-23T11:18:56.267418+0000","last_became_peered":"2024-03-23T11:18:56.267418+0000","last_unstale":"2024-03-23T11:18:57.257058+0000","last_undegraded":"2024-03-23T11:18:57.257058+0000","last_fullsized":"2024-03-23T11:18:57.257058+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:36:11.188974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252776+0000","last_change":"2024-03-23T11:18:55.775660+0000","last_active":"2024-03-23T11:18:57.252776+0000","last_peered":"2024-03-23T11:18:57.252776+0000","last_clean":"2024-03-23T11:18:57.252776+0000","last_became_active":"2024-03-23T11:18:55.775378+0000","last_became_peered":"2024-03-23T11:18:55.775378+0000","last_unstale":"2024-03-23T11:18:57.252776+0000","last_undegraded":"2024-03-23T11:18:57.252776+0000","last_fullsized":"2024-03-23T11:18:57.252776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:25:31.204286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100033+0000","last_change":"2024-03-23T11:18:56.273073+0000","last_active":"2024-03-23T11:18:58.100033+0000","last_peered":"2024-03-23T11:18:58.100033+0000","last_clean":"2024-03-23T11:18:58.100033+0000","last_became_active":"2024-03-23T11:18:56.265279+0000","last_became_peered":"2024-03-23T11:18:56.265279+0000","last_unstale":"2024-03-23T11:18:58.100033+0000","last_undegraded":"2024-03-23T11:18:58.100033+0000","last_fullsized":"2024-03-23T11:18:58.100033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252994+0000","last_change":"2024-03-23T11:18:55.773923+0000","last_active":"2024-03-23T11:18:57.252994+0000","last_peered":"2024-03-23T11:18:57.252994+0000","last_clean":"2024-03-23T11:18:57.252994+0000","last_became_active":"2024-03-23T11:18:55.773155+0000","last_became_peered":"2024-03-23T11:18:55.773155+0000","last_unstale":"2024-03-23T11:18:57.252994+0000","last_undegraded":"2024-03-23T11:18:57.252994+0000","last_fullsized":"2024-03-23T11:18:57.252994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:26.469433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101280+0000","last_change":"2024-03-23T11:18:56.265422+0000","last_active":"2024-03-23T11:18:58.101280+0000","last_peered":"2024-03-23T11:18:58.101280+0000","last_clean":"2024-03-23T11:18:58.101280+0000","last_became_active":"2024-03-23T11:18:56.265137+0000","last_became_peered":"2024-03-23T11:18:56.265137+0000","last_unstale":"2024-03-23T11:18:58.101280+0000","last_undegraded":"2024-03-23T11:18:58.101280+0000","last_fullsized":"2024-03-23T11:18:58.101280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:26.126571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256335+0000","last_change":"2024-03-23T11:18:55.778476+0000","last_active":"2024-03-23T11:18:57.256335+0000","last_peered":"2024-03-23T11:18:57.256335+0000","last_clean":"2024-03-23T11:18:57.256335+0000","last_became_active":"2024-03-23T11:18:55.778184+0000","last_became_peered":"2024-03-23T11:18:55.778184+0000","last_unstale":"2024-03-23T11:18:57.256335+0000","last_undegraded":"2024-03-23T11:18:57.256335+0000","last_fullsized":"2024-03-23T11:18:57.256335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:38.995255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462125+0000","last_change":"2024-03-23T11:18:56.277871+0000","last_active":"2024-03-23T11:18:57.462125+0000","last_peered":"2024-03-23T11:18:57.462125+0000","last_clean":"2024-03-23T11:18:57.462125+0000","last_became_active":"2024-03-23T11:18:56.277547+0000","last_became_peered":"2024-03-23T11:18:56.277547+0000","last_unstale":"2024-03-23T11:18:57.462125+0000","last_undegraded":"2024-03-23T11:18:57.462125+0000","last_fullsized":"2024-03-23T11:18:57.462125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.747240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315015+0000","last_change":"2024-03-23T11:18:56.288533+0000","last_active":"2024-03-23T11:18:57.315015+0000","last_peered":"2024-03-23T11:18:57.315015+0000","last_clean":"2024-03-23T11:18:57.315015+0000","last_became_active":"2024-03-23T11:18:56.288099+0000","last_became_peered":"2024-03-23T11:18:56.288099+0000","last_unstale":"2024-03-23T11:18:57.315015+0000","last_undegraded":"2024-03-23T11:18:57.315015+0000","last_fullsized":"2024-03-23T11:18:57.315015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:31:13.473526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256142+0000","last_change":"2024-03-23T11:18:55.784333+0000","last_active":"2024-03-23T11:18:57.256142+0000","last_peered":"2024-03-23T11:18:57.256142+0000","last_clean":"2024-03-23T11:18:57.256142+0000","last_became_active":"2024-03-23T11:18:55.783931+0000","last_became_peered":"2024-03-23T11:18:55.783931+0000","last_unstale":"2024-03-23T11:18:57.256142+0000","last_undegraded":"2024-03-23T11:18:57.256142+0000","last_fullsized":"2024-03-23T11:18:57.256142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:04:27.564911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315497+0000","last_change":"2024-03-23T11:18:56.287212+0000","last_active":"2024-03-23T11:18:57.315497+0000","last_peered":"2024-03-23T11:18:57.315497+0000","last_clean":"2024-03-23T11:18:57.315497+0000","last_became_active":"2024-03-23T11:18:56.284693+0000","last_became_peered":"2024-03-23T11:18:56.284693+0000","last_unstale":"2024-03-23T11:18:57.315497+0000","last_undegraded":"2024-03-23T11:18:57.315497+0000","last_fullsized":"2024-03-23T11:18:57.315497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:33.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256937+0000","last_change":"2024-03-23T11:18:55.779221+0000","last_active":"2024-03-23T11:18:57.256937+0000","last_peered":"2024-03-23T11:18:57.256937+0000","last_clean":"2024-03-23T11:18:57.256937+0000","last_became_active":"2024-03-23T11:18:55.778912+0000","last_became_peered":"2024-03-23T11:18:55.778912+0000","last_unstale":"2024-03-23T11:18:57.256937+0000","last_undegraded":"2024-03-23T11:18:57.256937+0000","last_fullsized":"2024-03-23T11:18:57.256937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:28:51.146588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099197+0000","last_change":"2024-03-23T11:18:56.291118+0000","last_active":"2024-03-23T11:18:58.099197+0000","last_peered":"2024-03-23T11:18:58.099197+0000","last_clean":"2024-03-23T11:18:58.099197+0000","last_became_active":"2024-03-23T11:18:56.290750+0000","last_became_peered":"2024-03-23T11:18:56.290750+0000","last_unstale":"2024-03-23T11:18:58.099197+0000","last_undegraded":"2024-03-23T11:18:58.099197+0000","last_fullsized":"2024-03-23T11:18:58.099197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101527+0000","last_change":"2024-03-23T11:18:55.784254+0000","last_active":"2024-03-23T11:18:58.101527+0000","last_peered":"2024-03-23T11:18:58.101527+0000","last_clean":"2024-03-23T11:18:58.101527+0000","last_became_active":"2024-03-23T11:18:55.781763+0000","last_became_peered":"2024-03-23T11:18:55.781763+0000","last_unstale":"2024-03-23T11:18:58.101527+0000","last_undegraded":"2024-03-23T11:18:58.101527+0000","last_fullsized":"2024-03-23T11:18:58.101527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101387+0000","last_change":"2024-03-23T11:18:56.267771+0000","last_active":"2024-03-23T11:18:58.101387+0000","last_peered":"2024-03-23T11:18:58.101387+0000","last_clean":"2024-03-23T11:18:58.101387+0000","last_became_active":"2024-03-23T11:18:56.267515+0000","last_became_peered":"2024-03-23T11:18:56.267515+0000","last_unstale":"2024-03-23T11:18:58.101387+0000","last_undegraded":"2024-03-23T11:18:58.101387+0000","last_fullsized":"2024-03-23T11:18:58.101387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099857+0000","last_change":"2024-03-23T11:18:55.785970+0000","last_active":"2024-03-23T11:18:58.099857+0000","last_peered":"2024-03-23T11:18:58.099857+0000","last_clean":"2024-03-23T11:18:58.099857+0000","last_became_active":"2024-03-23T11:18:55.784347+0000","last_became_peered":"2024-03-23T11:18:55.784347+0000","last_unstale":"2024-03-23T11:18:58.099857+0000","last_undegraded":"2024-03-23T11:18:58.099857+0000","last_fullsized":"2024-03-23T11:18:58.099857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:36:21.274877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":823296,"data_stored":669740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.577}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7849999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1859999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.254}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.764}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.702}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.845}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0419999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0529999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3460000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1819999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":78281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2889999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5069999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6060000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":827392,"data_stored":669127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:19:06.890 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:19:07.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.148+0000 7f3176e00640 1 Processor -- start 2024-03-23T11:19:07.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.148+0000 7f3176e00640 1 -- start start 2024-03-23T11:19:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3176e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3170143dd0 0x7f317015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3176e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3170070090 0x7f31700704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3176e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3170070a30 con 0x7f31700b2820 2024-03-23T11:19:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3176e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31700cd140 con 0x7f3170070090 2024-03-23T11:19:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3176e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31700cd3e0 con 0x7f3170143dd0 2024-03-23T11:19:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3175000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3170070090 0x7f31700704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3176400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f31700b2820 0x7f31700b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53556/0 (socket says 172.21.15.62:53556) 2024-03-23T11:19:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3176400640 1 -- 172.21.15.62:0/3552537880 learned_addr learned my addr 172.21.15.62:0/3552537880 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3175000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3170070090 0x7f31700704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38154/0 (socket says 172.21.15.62:38154) 2024-03-23T11:19:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3175a00640 1 --2- 172.21.15.62:0/3552537880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3170143dd0 0x7f317015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3175000640 1 -- 172.21.15.62:0/3552537880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3170143dd0 msgr2=0x7f317015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3175000640 1 --2- 172.21.15.62:0/3552537880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3170143dd0 0x7f317015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3175000640 1 -- 172.21.15.62:0/3552537880 >> v1:172.21.15.62:6789/0 conn(0x7f31700b2820 legacy=0x7f31700b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f316fe00640 1 -- 172.21.15.62:0/3552537880 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505632928 0 0) 0x7f3170070a30 con 0x7f31700b2820 2024-03-23T11:19:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3175000640 1 -- 172.21.15.62:0/3552537880 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31700cd6c0 con 0x7f3170070090 2024-03-23T11:19:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.150+0000 7f3175a00640 1 --2- 172.21.15.62:0/3552537880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3170143dd0 0x7f317015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.152+0000 7f3175000640 1 --2- 172.21.15.62:0/3552537880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3170070090 0x7f31700704f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f315c0097a0 tx=0x7f315c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe6c5ce47f05510 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.152+0000 7f316fe00640 1 -- 172.21.15.62:0/3552537880 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c02d040 con 0x7f3170070090 2024-03-23T11:19:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.152+0000 7f316fe00640 1 -- 172.21.15.62:0/3552537880 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315c031810 con 0x7f3170070090 2024-03-23T11:19:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.152+0000 7f316fe00640 1 -- 172.21.15.62:0/3552537880 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c031ba0 con 0x7f3170070090 2024-03-23T11:19:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.152+0000 7f3176e00640 1 -- 172.21.15.62:0/3552537880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3170070090 msgr2=0x7f31700704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.152+0000 7f3176e00640 1 --2- 172.21.15.62:0/3552537880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3170070090 0x7f31700704f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f315c0097a0 tx=0x7f315c02b8b0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.153+0000 7f3176e00640 1 -- 172.21.15.62:0/3552537880 shutdown_connections 2024-03-23T11:19:07.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.153+0000 7f3176e00640 1 --2- 172.21.15.62:0/3552537880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3170070090 0x7f31700704f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.153+0000 7f3176e00640 1 --2- 172.21.15.62:0/3552537880 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3170143dd0 0x7f317015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.153+0000 7f3176e00640 1 -- 172.21.15.62:0/3552537880 >> 172.21.15.62:0/3552537880 conn(0x7f3170066050 msgr2=0x7f3170069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:07.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.153+0000 7f3176e00640 1 -- 172.21.15.62:0/3552537880 shutdown_connections 2024-03-23T11:19:07.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.153+0000 7f3176e00640 1 -- 172.21.15.62:0/3552537880 wait complete. 2024-03-23T11:19:07.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.154+0000 7f3176e00640 1 Processor -- start 2024-03-23T11:19:07.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.154+0000 7f3176e00640 1 -- start start 2024-03-23T11:19:07.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.155+0000 7f3176e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3170070090 0x7f317015a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.155+0000 7f3176e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31700b2820 0x7f31701356a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.155+0000 7f3176e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3170143dd0 0x7f31700c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.155+0000 7f3176e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31700c3300 con 0x7f3170070090 2024-03-23T11:19:07.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.155+0000 7f3176e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31700c35e0 con 0x7f3170143dd0 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.155+0000 7f3176e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31700c38c0 con 0x7f31700b2820 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3175a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3170070090 0x7f317015a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3175a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3170070090 0x7f317015a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38162/0 (socket says 172.21.15.62:38162) 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3175a00640 1 -- 172.21.15.62:0/3017298277 learned_addr learned my addr 172.21.15.62:0/3017298277 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3175000640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31700b2820 0x7f31701356a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3176400640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3170143dd0 0x7f31700c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3175a00640 1 -- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31700b2820 msgr2=0x7f31701356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3175a00640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31700b2820 0x7f31701356a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3175a00640 1 -- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3170143dd0 msgr2=0x7f31700c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3175a00640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3170143dd0 0x7f31700c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3175a00640 1 -- 172.21.15.62:0/3017298277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c009400 con 0x7f3170070090 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3175000640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31700b2820 0x7f31701356a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3176400640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3170143dd0 0x7f31700c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:07.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.156+0000 7f3175a00640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3170070090 0x7f317015a310 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f316400c7f0 tx=0x7f316400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:07.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.157+0000 7f316d600640 1 -- 172.21.15.62:0/3017298277 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3164007820 con 0x7f3170070090 2024-03-23T11:19:07.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.157+0000 7f316d600640 1 -- 172.21.15.62:0/3017298277 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3164007980 con 0x7f3170070090 2024-03-23T11:19:07.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.157+0000 7f316d600640 1 -- 172.21.15.62:0/3017298277 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3164007cf0 con 0x7f3170070090 2024-03-23T11:19:07.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.157+0000 7f3176e00640 1 -- 172.21.15.62:0/3017298277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31700c3ba0 con 0x7f3170070090 2024-03-23T11:19:07.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.157+0000 7f3176e00640 1 -- 172.21.15.62:0/3017298277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3170138500 con 0x7f3170070090 2024-03-23T11:19:07.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.158+0000 7f3176e00640 1 -- 172.21.15.62:0/3017298277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3170066bb0 con 0x7f3170070090 2024-03-23T11:19:07.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.159+0000 7f316d600640 1 -- 172.21.15.62:0/3017298277 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f31640113f0 con 0x7f3170070090 2024-03-23T11:19:07.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.159+0000 7f316d600640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f315006a7a0 0x7f315006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.159+0000 7f316d600640 1 -- 172.21.15.62:0/3017298277 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31640890c0 con 0x7f3170070090 2024-03-23T11:19:07.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.160+0000 7f3175000640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f315006a7a0 0x7f315006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.161+0000 7f3175000640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f315006a7a0 0x7f315006cc60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f315c036000 tx=0x7f315c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:07.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.164+0000 7f316d600640 1 -- 172.21.15.62:0/3017298277 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31640544d0 con 0x7f3170070090 2024-03-23T11:19:07.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.349+0000 7f3176e00640 1 -- 172.21.15.62:0/3017298277 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f31700cdf60 con 0x7f315006a7a0 2024-03-23T11:19:07.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.369+0000 7f316d600640 1 -- 172.21.15.62:0/3017298277 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264437 (secure 0 0 0) 0x7f315c031610 con 0x7f315006a7a0 2024-03-23T11:19:07.374 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:07.375 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 -- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f315006a7a0 msgr2=0x7f315006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f315006a7a0 0x7f315006cc60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f315c036000 tx=0x7f315c008030 comp rx=0 tx=0).stop 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 -- 172.21.15.62:0/3017298277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3170070090 msgr2=0x7f317015a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3170070090 0x7f317015a310 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f316400c7f0 tx=0x7f316400ccb0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 -- 172.21.15.62:0/3017298277 shutdown_connections 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f315006a7a0 0x7f315006cc60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3170143dd0 0x7f31700c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31700b2820 0x7f31701356a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 --2- 172.21.15.62:0/3017298277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3170070090 0x7f317015a310 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 -- 172.21.15.62:0/3017298277 >> 172.21.15.62:0/3017298277 conn(0x7f3170066050 msgr2=0x7f317015b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 -- 172.21.15.62:0/3017298277 shutdown_connections 2024-03-23T11:19:07.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.375+0000 7f3176e00640 1 -- 172.21.15.62:0/3017298277 wait complete. 2024-03-23T11:19:07.392 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-23T11:19:07.235716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":2972,"kb_used_omap":13,"kb_used_meta":216050,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3043328,"data_stored":1813773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":221235288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":81,"num_write_kb":798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003806"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257670+0000","last_change":"2024-03-23T11:18:56.274088+0000","last_active":"2024-03-23T11:18:57.257670+0000","last_peered":"2024-03-23T11:18:57.257670+0000","last_clean":"2024-03-23T11:18:57.257670+0000","last_became_active":"2024-03-23T11:18:56.273097+0000","last_became_peered":"2024-03-23T11:18:56.273097+0000","last_unstale":"2024-03-23T11:18:57.257670+0000","last_undegraded":"2024-03-23T11:18:57.257670+0000","last_fullsized":"2024-03-23T11:18:57.257670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:48:24.807457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253337+0000","last_change":"2024-03-23T11:18:56.260586+0000","last_active":"2024-03-23T11:18:57.253337+0000","last_peered":"2024-03-23T11:18:57.253337+0000","last_clean":"2024-03-23T11:18:57.253337+0000","last_became_active":"2024-03-23T11:18:56.260321+0000","last_became_peered":"2024-03-23T11:18:56.260321+0000","last_unstale":"2024-03-23T11:18:57.253337+0000","last_undegraded":"2024-03-23T11:18:57.253337+0000","last_fullsized":"2024-03-23T11:18:57.253337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:24:40.203305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099527+0000","last_change":"2024-03-23T11:18:56.264999+0000","last_active":"2024-03-23T11:18:58.099527+0000","last_peered":"2024-03-23T11:18:58.099527+0000","last_clean":"2024-03-23T11:18:58.099527+0000","last_became_active":"2024-03-23T11:18:56.264689+0000","last_became_peered":"2024-03-23T11:18:56.264689+0000","last_unstale":"2024-03-23T11:18:58.099527+0000","last_undegraded":"2024-03-23T11:18:58.099527+0000","last_fullsized":"2024-03-23T11:18:58.099527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.747240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315279+0000","last_change":"2024-03-23T11:18:56.283608+0000","last_active":"2024-03-23T11:18:57.315279+0000","last_peered":"2024-03-23T11:18:57.315279+0000","last_clean":"2024-03-23T11:18:57.315279+0000","last_became_active":"2024-03-23T11:18:56.277048+0000","last_became_peered":"2024-03-23T11:18:56.277048+0000","last_unstale":"2024-03-23T11:18:57.315279+0000","last_undegraded":"2024-03-23T11:18:57.315279+0000","last_fullsized":"2024-03-23T11:18:57.315279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:28.809711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256138+0000","last_change":"2024-03-23T11:18:56.264915+0000","last_active":"2024-03-23T11:18:57.256138+0000","last_peered":"2024-03-23T11:18:57.256138+0000","last_clean":"2024-03-23T11:18:57.256138+0000","last_became_active":"2024-03-23T11:18:56.264696+0000","last_became_peered":"2024-03-23T11:18:56.264696+0000","last_unstale":"2024-03-23T11:18:57.256138+0000","last_undegraded":"2024-03-23T11:18:57.256138+0000","last_fullsized":"2024-03-23T11:18:57.256138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:59.578102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314423+0000","last_change":"2024-03-23T11:18:56.282941+0000","last_active":"2024-03-23T11:18:57.314423+0000","last_peered":"2024-03-23T11:18:57.314423+0000","last_clean":"2024-03-23T11:18:57.314423+0000","last_became_active":"2024-03-23T11:18:56.282556+0000","last_became_peered":"2024-03-23T11:18:56.282556+0000","last_unstale":"2024-03-23T11:18:57.314423+0000","last_undegraded":"2024-03-23T11:18:57.314423+0000","last_fullsized":"2024-03-23T11:18:57.314423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:01:46.194360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255145+0000","last_change":"2024-03-23T11:18:56.274568+0000","last_active":"2024-03-23T11:18:57.255145+0000","last_peered":"2024-03-23T11:18:57.255145+0000","last_clean":"2024-03-23T11:18:57.255145+0000","last_became_active":"2024-03-23T11:18:56.273524+0000","last_became_peered":"2024-03-23T11:18:56.273524+0000","last_unstale":"2024-03-23T11:18:57.255145+0000","last_undegraded":"2024-03-23T11:18:57.255145+0000","last_fullsized":"2024-03-23T11:18:57.255145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:48:43.299392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099491+0000","last_change":"2024-03-23T11:18:56.272280+0000","last_active":"2024-03-23T11:18:58.099491+0000","last_peered":"2024-03-23T11:18:58.099491+0000","last_clean":"2024-03-23T11:18:58.099491+0000","last_became_active":"2024-03-23T11:18:56.272096+0000","last_became_peered":"2024-03-23T11:18:56.272096+0000","last_unstale":"2024-03-23T11:18:58.099491+0000","last_undegraded":"2024-03-23T11:18:58.099491+0000","last_fullsized":"2024-03-23T11:18:58.099491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:28.809711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257785+0000","last_change":"2024-03-23T11:18:56.282997+0000","last_active":"2024-03-23T11:18:57.257785+0000","last_peered":"2024-03-23T11:18:57.257785+0000","last_clean":"2024-03-23T11:18:57.257785+0000","last_became_active":"2024-03-23T11:18:56.282768+0000","last_became_peered":"2024-03-23T11:18:56.282768+0000","last_unstale":"2024-03-23T11:18:57.257785+0000","last_undegraded":"2024-03-23T11:18:57.257785+0000","last_fullsized":"2024-03-23T11:18:57.257785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:47:39.199128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461969+0000","last_change":"2024-03-23T11:18:56.269707+0000","last_active":"2024-03-23T11:18:57.461969+0000","last_peered":"2024-03-23T11:18:57.461969+0000","last_clean":"2024-03-23T11:18:57.461969+0000","last_became_active":"2024-03-23T11:18:56.266078+0000","last_became_peered":"2024-03-23T11:18:56.266078+0000","last_unstale":"2024-03-23T11:18:57.461969+0000","last_undegraded":"2024-03-23T11:18:57.461969+0000","last_fullsized":"2024-03-23T11:18:57.461969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461775+0000","last_change":"2024-03-23T11:18:56.266542+0000","last_active":"2024-03-23T11:18:57.461775+0000","last_peered":"2024-03-23T11:18:57.461775+0000","last_clean":"2024-03-23T11:18:57.461775+0000","last_became_active":"2024-03-23T11:18:56.266026+0000","last_became_peered":"2024-03-23T11:18:56.266026+0000","last_unstale":"2024-03-23T11:18:57.461775+0000","last_undegraded":"2024-03-23T11:18:57.461775+0000","last_fullsized":"2024-03-23T11:18:57.461775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255738+0000","last_change":"2024-03-23T11:18:56.272690+0000","last_active":"2024-03-23T11:18:57.255738+0000","last_peered":"2024-03-23T11:18:57.255738+0000","last_clean":"2024-03-23T11:18:57.255738+0000","last_became_active":"2024-03-23T11:18:56.272153+0000","last_became_peered":"2024-03-23T11:18:56.272153+0000","last_unstale":"2024-03-23T11:18:57.255738+0000","last_undegraded":"2024-03-23T11:18:57.255738+0000","last_fullsized":"2024-03-23T11:18:57.255738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:44:23.104322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101098+0000","last_change":"2024-03-23T11:18:56.264317+0000","last_active":"2024-03-23T11:18:58.101098+0000","last_peered":"2024-03-23T11:18:58.101098+0000","last_clean":"2024-03-23T11:18:58.101098+0000","last_became_active":"2024-03-23T11:18:56.263977+0000","last_became_peered":"2024-03-23T11:18:56.263977+0000","last_unstale":"2024-03-23T11:18:58.101098+0000","last_undegraded":"2024-03-23T11:18:58.101098+0000","last_fullsized":"2024-03-23T11:18:58.101098+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:33.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098064+0000","last_change":"2024-03-23T11:18:56.267303+0000","last_active":"2024-03-23T11:18:58.098064+0000","last_peered":"2024-03-23T11:18:58.098064+0000","last_clean":"2024-03-23T11:18:58.098064+0000","last_became_active":"2024-03-23T11:18:56.267130+0000","last_became_peered":"2024-03-23T11:18:56.267130+0000","last_unstale":"2024-03-23T11:18:58.098064+0000","last_undegraded":"2024-03-23T11:18:58.098064+0000","last_fullsized":"2024-03-23T11:18:58.098064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252603+0000","last_change":"2024-03-23T11:18:56.275581+0000","last_active":"2024-03-23T11:18:57.252603+0000","last_peered":"2024-03-23T11:18:57.252603+0000","last_clean":"2024-03-23T11:18:57.252603+0000","last_became_active":"2024-03-23T11:18:56.274566+0000","last_became_peered":"2024-03-23T11:18:56.274566+0000","last_unstale":"2024-03-23T11:18:57.252603+0000","last_undegraded":"2024-03-23T11:18:57.252603+0000","last_fullsized":"2024-03-23T11:18:57.252603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:31:43.650115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314345+0000","last_change":"2024-03-23T11:18:56.282803+0000","last_active":"2024-03-23T11:18:57.314345+0000","last_peered":"2024-03-23T11:18:57.314345+0000","last_clean":"2024-03-23T11:18:57.314345+0000","last_became_active":"2024-03-23T11:18:56.282500+0000","last_became_peered":"2024-03-23T11:18:56.282500+0000","last_unstale":"2024-03-23T11:18:57.314345+0000","last_undegraded":"2024-03-23T11:18:57.314345+0000","last_fullsized":"2024-03-23T11:18:57.314345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315513+0000","last_change":"2024-03-23T11:18:56.287431+0000","last_active":"2024-03-23T11:18:57.315513+0000","last_peered":"2024-03-23T11:18:57.315513+0000","last_clean":"2024-03-23T11:18:57.315513+0000","last_became_active":"2024-03-23T11:18:56.286853+0000","last_became_peered":"2024-03-23T11:18:56.286853+0000","last_unstale":"2024-03-23T11:18:57.315513+0000","last_undegraded":"2024-03-23T11:18:57.315513+0000","last_fullsized":"2024-03-23T11:18:57.315513+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:54:17.561394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100445+0000","last_change":"2024-03-23T11:18:56.260336+0000","last_active":"2024-03-23T11:18:58.100445+0000","last_peered":"2024-03-23T11:18:58.100445+0000","last_clean":"2024-03-23T11:18:58.100445+0000","last_became_active":"2024-03-23T11:18:56.259638+0000","last_became_peered":"2024-03-23T11:18:56.259638+0000","last_unstale":"2024-03-23T11:18:58.100445+0000","last_undegraded":"2024-03-23T11:18:58.100445+0000","last_fullsized":"2024-03-23T11:18:58.100445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:36:11.188974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257119+0000","last_change":"2024-03-23T11:18:56.268940+0000","last_active":"2024-03-23T11:18:57.257119+0000","last_peered":"2024-03-23T11:18:57.257119+0000","last_clean":"2024-03-23T11:18:57.257119+0000","last_became_active":"2024-03-23T11:18:56.268168+0000","last_became_peered":"2024-03-23T11:18:56.268168+0000","last_unstale":"2024-03-23T11:18:57.257119+0000","last_undegraded":"2024-03-23T11:18:57.257119+0000","last_fullsized":"2024-03-23T11:18:57.257119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:22.782018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461751+0000","last_change":"2024-03-23T11:18:56.271021+0000","last_active":"2024-03-23T11:18:57.461751+0000","last_peered":"2024-03-23T11:18:57.461751+0000","last_clean":"2024-03-23T11:18:57.461751+0000","last_became_active":"2024-03-23T11:18:56.270766+0000","last_became_peered":"2024-03-23T11:18:56.270766+0000","last_unstale":"2024-03-23T11:18:57.461751+0000","last_undegraded":"2024-03-23T11:18:57.461751+0000","last_fullsized":"2024-03-23T11:18:57.461751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098354+0000","last_change":"2024-03-23T11:18:56.261635+0000","last_active":"2024-03-23T11:18:58.098354+0000","last_peered":"2024-03-23T11:18:58.098354+0000","last_clean":"2024-03-23T11:18:58.098354+0000","last_became_active":"2024-03-23T11:18:56.260741+0000","last_became_peered":"2024-03-23T11:18:56.260741+0000","last_unstale":"2024-03-23T11:18:58.098354+0000","last_undegraded":"2024-03-23T11:18:58.098354+0000","last_fullsized":"2024-03-23T11:18:58.098354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314890+0000","last_change":"2024-03-23T11:18:56.286882+0000","last_active":"2024-03-23T11:18:57.314890+0000","last_peered":"2024-03-23T11:18:57.314890+0000","last_clean":"2024-03-23T11:18:57.314890+0000","last_became_active":"2024-03-23T11:18:56.285291+0000","last_became_peered":"2024-03-23T11:18:56.285291+0000","last_unstale":"2024-03-23T11:18:57.314890+0000","last_undegraded":"2024-03-23T11:18:57.314890+0000","last_fullsized":"2024-03-23T11:18:57.314890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:56.858523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101535+0000","last_change":"2024-03-23T11:18:56.276050+0000","last_active":"2024-03-23T11:18:58.101535+0000","last_peered":"2024-03-23T11:18:58.101535+0000","last_clean":"2024-03-23T11:18:58.101535+0000","last_became_active":"2024-03-23T11:18:56.275264+0000","last_became_peered":"2024-03-23T11:18:56.275264+0000","last_unstale":"2024-03-23T11:18:58.101535+0000","last_undegraded":"2024-03-23T11:18:58.101535+0000","last_fullsized":"2024-03-23T11:18:58.101535+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314573+0000","last_change":"2024-03-23T11:18:56.283292+0000","last_active":"2024-03-23T11:18:57.314573+0000","last_peered":"2024-03-23T11:18:57.314573+0000","last_clean":"2024-03-23T11:18:57.314573+0000","last_became_active":"2024-03-23T11:18:56.282493+0000","last_became_peered":"2024-03-23T11:18:56.282493+0000","last_unstale":"2024-03-23T11:18:57.314573+0000","last_undegraded":"2024-03-23T11:18:57.314573+0000","last_fullsized":"2024-03-23T11:18:57.314573+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101895+0000","last_change":"2024-03-23T11:18:56.282679+0000","last_active":"2024-03-23T11:18:58.101895+0000","last_peered":"2024-03-23T11:18:58.101895+0000","last_clean":"2024-03-23T11:18:58.101895+0000","last_became_active":"2024-03-23T11:18:56.282434+0000","last_became_peered":"2024-03-23T11:18:56.282434+0000","last_unstale":"2024-03-23T11:18:58.101895+0000","last_undegraded":"2024-03-23T11:18:58.101895+0000","last_fullsized":"2024-03-23T11:18:58.101895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:00.423263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462105+0000","last_change":"2024-03-23T11:18:56.260538+0000","last_active":"2024-03-23T11:18:57.462105+0000","last_peered":"2024-03-23T11:18:57.462105+0000","last_clean":"2024-03-23T11:18:57.462105+0000","last_became_active":"2024-03-23T11:18:56.259279+0000","last_became_peered":"2024-03-23T11:18:56.259279+0000","last_unstale":"2024-03-23T11:18:57.462105+0000","last_undegraded":"2024-03-23T11:18:57.462105+0000","last_fullsized":"2024-03-23T11:18:57.462105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253224+0000","last_change":"2024-03-23T11:18:56.263336+0000","last_active":"2024-03-23T11:18:57.253224+0000","last_peered":"2024-03-23T11:18:57.253224+0000","last_clean":"2024-03-23T11:18:57.253224+0000","last_became_active":"2024-03-23T11:18:56.262573+0000","last_became_peered":"2024-03-23T11:18:56.262573+0000","last_unstale":"2024-03-23T11:18:57.253224+0000","last_undegraded":"2024-03-23T11:18:57.253224+0000","last_fullsized":"2024-03-23T11:18:57.253224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:41:50.064223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101071+0000","last_change":"2024-03-23T11:18:56.284862+0000","last_active":"2024-03-23T11:18:58.101071+0000","last_peered":"2024-03-23T11:18:58.101071+0000","last_clean":"2024-03-23T11:18:58.101071+0000","last_became_active":"2024-03-23T11:18:56.284599+0000","last_became_peered":"2024-03-23T11:18:56.284599+0000","last_unstale":"2024-03-23T11:18:58.101071+0000","last_undegraded":"2024-03-23T11:18:58.101071+0000","last_fullsized":"2024-03-23T11:18:58.101071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.705001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098596+0000","last_change":"2024-03-23T11:18:56.271496+0000","last_active":"2024-03-23T11:18:58.098596+0000","last_peered":"2024-03-23T11:18:58.098596+0000","last_clean":"2024-03-23T11:18:58.098596+0000","last_became_active":"2024-03-23T11:18:56.271243+0000","last_became_peered":"2024-03-23T11:18:56.271243+0000","last_unstale":"2024-03-23T11:18:58.098596+0000","last_undegraded":"2024-03-23T11:18:58.098596+0000","last_fullsized":"2024-03-23T11:18:58.098596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:16:46.738802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314711+0000","last_change":"2024-03-23T11:18:56.282267+0000","last_active":"2024-03-23T11:18:57.314711+0000","last_peered":"2024-03-23T11:18:57.314711+0000","last_clean":"2024-03-23T11:18:57.314711+0000","last_became_active":"2024-03-23T11:18:56.277544+0000","last_became_peered":"2024-03-23T11:18:56.277544+0000","last_unstale":"2024-03-23T11:18:57.314711+0000","last_undegraded":"2024-03-23T11:18:57.314711+0000","last_fullsized":"2024-03-23T11:18:57.314711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:12.916824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098163+0000","last_change":"2024-03-23T11:18:56.267136+0000","last_active":"2024-03-23T11:18:58.098163+0000","last_peered":"2024-03-23T11:18:58.098163+0000","last_clean":"2024-03-23T11:18:58.098163+0000","last_became_active":"2024-03-23T11:18:56.266640+0000","last_became_peered":"2024-03-23T11:18:56.266640+0000","last_unstale":"2024-03-23T11:18:58.098163+0000","last_undegraded":"2024-03-23T11:18:58.098163+0000","last_fullsized":"2024-03-23T11:18:58.098163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461393+0000","last_change":"2024-03-23T11:18:56.266829+0000","last_active":"2024-03-23T11:18:57.461393+0000","last_peered":"2024-03-23T11:18:57.461393+0000","last_clean":"2024-03-23T11:18:57.461393+0000","last_became_active":"2024-03-23T11:18:56.266536+0000","last_became_peered":"2024-03-23T11:18:56.266536+0000","last_unstale":"2024-03-23T11:18:57.461393+0000","last_undegraded":"2024-03-23T11:18:57.461393+0000","last_fullsized":"2024-03-23T11:18:57.461393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:00.423263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255237+0000","last_change":"2024-03-23T11:18:56.267024+0000","last_active":"2024-03-23T11:18:57.255237+0000","last_peered":"2024-03-23T11:18:57.255237+0000","last_clean":"2024-03-23T11:18:57.255237+0000","last_became_active":"2024-03-23T11:18:56.266730+0000","last_became_peered":"2024-03-23T11:18:56.266730+0000","last_unstale":"2024-03-23T11:18:57.255237+0000","last_undegraded":"2024-03-23T11:18:57.255237+0000","last_fullsized":"2024-03-23T11:18:57.255237+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:19.533798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101417+0000","last_change":"2024-03-23T11:18:55.778856+0000","last_active":"2024-03-23T11:18:58.101417+0000","last_peered":"2024-03-23T11:18:58.101417+0000","last_clean":"2024-03-23T11:18:58.101417+0000","last_became_active":"2024-03-23T11:18:55.778487+0000","last_became_peered":"2024-03-23T11:18:55.778487+0000","last_unstale":"2024-03-23T11:18:58.101417+0000","last_undegraded":"2024-03-23T11:18:58.101417+0000","last_fullsized":"2024-03-23T11:18:58.101417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098890+0000","last_change":"2024-03-23T11:18:56.280876+0000","last_active":"2024-03-23T11:18:58.098890+0000","last_peered":"2024-03-23T11:18:58.098890+0000","last_clean":"2024-03-23T11:18:58.098890+0000","last_became_active":"2024-03-23T11:18:56.280574+0000","last_became_peered":"2024-03-23T11:18:56.280574+0000","last_unstale":"2024-03-23T11:18:58.098890+0000","last_undegraded":"2024-03-23T11:18:58.098890+0000","last_fullsized":"2024-03-23T11:18:58.098890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:25.559049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462037+0000","last_change":"2024-03-23T11:18:55.771275+0000","last_active":"2024-03-23T11:18:57.462037+0000","last_peered":"2024-03-23T11:18:57.462037+0000","last_clean":"2024-03-23T11:18:57.462037+0000","last_became_active":"2024-03-23T11:18:55.770948+0000","last_became_peered":"2024-03-23T11:18:55.770948+0000","last_unstale":"2024-03-23T11:18:57.462037+0000","last_undegraded":"2024-03-23T11:18:57.462037+0000","last_fullsized":"2024-03-23T11:18:57.462037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101598+0000","last_change":"2024-03-23T11:18:55.780718+0000","last_active":"2024-03-23T11:18:58.101598+0000","last_peered":"2024-03-23T11:18:58.101598+0000","last_clean":"2024-03-23T11:18:58.101598+0000","last_became_active":"2024-03-23T11:18:55.779357+0000","last_became_peered":"2024-03-23T11:18:55.779357+0000","last_unstale":"2024-03-23T11:18:58.101598+0000","last_undegraded":"2024-03-23T11:18:58.101598+0000","last_fullsized":"2024-03-23T11:18:58.101598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:36:21.274877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099752+0000","last_change":"2024-03-23T11:18:56.266172+0000","last_active":"2024-03-23T11:18:58.099752+0000","last_peered":"2024-03-23T11:18:58.099752+0000","last_clean":"2024-03-23T11:18:58.099752+0000","last_became_active":"2024-03-23T11:18:56.265882+0000","last_became_peered":"2024-03-23T11:18:56.265882+0000","last_unstale":"2024-03-23T11:18:58.099752+0000","last_undegraded":"2024-03-23T11:18:58.099752+0000","last_fullsized":"2024-03-23T11:18:58.099752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314638+0000","last_change":"2024-03-23T11:18:55.774144+0000","last_active":"2024-03-23T11:18:57.314638+0000","last_peered":"2024-03-23T11:18:57.314638+0000","last_clean":"2024-03-23T11:18:57.314638+0000","last_became_active":"2024-03-23T11:18:55.773880+0000","last_became_peered":"2024-03-23T11:18:55.773880+0000","last_unstale":"2024-03-23T11:18:57.314638+0000","last_undegraded":"2024-03-23T11:18:57.314638+0000","last_fullsized":"2024-03-23T11:18:57.314638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252538+0000","last_change":"2024-03-23T11:18:56.260698+0000","last_active":"2024-03-23T11:18:57.252538+0000","last_peered":"2024-03-23T11:18:57.252538+0000","last_clean":"2024-03-23T11:18:57.252538+0000","last_became_active":"2024-03-23T11:18:56.258466+0000","last_became_peered":"2024-03-23T11:18:56.258466+0000","last_unstale":"2024-03-23T11:18:57.252538+0000","last_undegraded":"2024-03-23T11:18:57.252538+0000","last_fullsized":"2024-03-23T11:18:57.252538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:55:22.272329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462086+0000","last_change":"2024-03-23T11:18:55.783554+0000","last_active":"2024-03-23T11:18:57.462086+0000","last_peered":"2024-03-23T11:18:57.462086+0000","last_clean":"2024-03-23T11:18:57.462086+0000","last_became_active":"2024-03-23T11:18:55.783248+0000","last_became_peered":"2024-03-23T11:18:55.783248+0000","last_unstale":"2024-03-23T11:18:57.462086+0000","last_undegraded":"2024-03-23T11:18:57.462086+0000","last_fullsized":"2024-03-23T11:18:57.462086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:18.352259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314908+0000","last_change":"2024-03-23T11:18:56.287651+0000","last_active":"2024-03-23T11:18:57.314908+0000","last_peered":"2024-03-23T11:18:57.314908+0000","last_clean":"2024-03-23T11:18:57.314908+0000","last_became_active":"2024-03-23T11:18:56.287312+0000","last_became_peered":"2024-03-23T11:18:56.287312+0000","last_unstale":"2024-03-23T11:18:57.314908+0000","last_undegraded":"2024-03-23T11:18:57.314908+0000","last_fullsized":"2024-03-23T11:18:57.314908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:24:29.439358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314950+0000","last_change":"2024-03-23T11:18:55.783117+0000","last_active":"2024-03-23T11:18:57.314950+0000","last_peered":"2024-03-23T11:18:57.314950+0000","last_clean":"2024-03-23T11:18:57.314950+0000","last_became_active":"2024-03-23T11:18:55.782341+0000","last_became_peered":"2024-03-23T11:18:55.782341+0000","last_unstale":"2024-03-23T11:18:57.314950+0000","last_undegraded":"2024-03-23T11:18:57.314950+0000","last_fullsized":"2024-03-23T11:18:57.314950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:10:08.943114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099470+0000","last_change":"2024-03-23T11:18:56.290381+0000","last_active":"2024-03-23T11:18:58.099470+0000","last_peered":"2024-03-23T11:18:58.099470+0000","last_clean":"2024-03-23T11:18:58.099470+0000","last_became_active":"2024-03-23T11:18:56.290070+0000","last_became_peered":"2024-03-23T11:18:56.290070+0000","last_unstale":"2024-03-23T11:18:58.099470+0000","last_undegraded":"2024-03-23T11:18:58.099470+0000","last_fullsized":"2024-03-23T11:18:58.099470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253428+0000","last_change":"2024-03-23T11:18:55.783461+0000","last_active":"2024-03-23T11:18:57.253428+0000","last_peered":"2024-03-23T11:18:57.253428+0000","last_clean":"2024-03-23T11:18:57.253428+0000","last_became_active":"2024-03-23T11:18:55.783214+0000","last_became_peered":"2024-03-23T11:18:55.783214+0000","last_unstale":"2024-03-23T11:18:57.253428+0000","last_undegraded":"2024-03-23T11:18:57.253428+0000","last_fullsized":"2024-03-23T11:18:57.253428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:28:51.146588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255996+0000","last_change":"2024-03-23T11:18:56.268161+0000","last_active":"2024-03-23T11:18:57.255996+0000","last_peered":"2024-03-23T11:18:57.255996+0000","last_clean":"2024-03-23T11:18:57.255996+0000","last_became_active":"2024-03-23T11:18:56.263992+0000","last_became_peered":"2024-03-23T11:18:56.263992+0000","last_unstale":"2024-03-23T11:18:57.255996+0000","last_undegraded":"2024-03-23T11:18:57.255996+0000","last_fullsized":"2024-03-23T11:18:57.255996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461733+0000","last_change":"2024-03-23T11:18:55.782334+0000","last_active":"2024-03-23T11:18:57.461733+0000","last_peered":"2024-03-23T11:18:57.461733+0000","last_clean":"2024-03-23T11:18:57.461733+0000","last_became_active":"2024-03-23T11:18:55.780347+0000","last_became_peered":"2024-03-23T11:18:55.780347+0000","last_unstale":"2024-03-23T11:18:57.461733+0000","last_undegraded":"2024-03-23T11:18:57.461733+0000","last_fullsized":"2024-03-23T11:18:57.461733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:10:08.943114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256551+0000","last_change":"2024-03-23T11:18:56.272684+0000","last_active":"2024-03-23T11:18:57.256551+0000","last_peered":"2024-03-23T11:18:57.256551+0000","last_clean":"2024-03-23T11:18:57.256551+0000","last_became_active":"2024-03-23T11:18:56.268603+0000","last_became_peered":"2024-03-23T11:18:56.268603+0000","last_unstale":"2024-03-23T11:18:57.256551+0000","last_undegraded":"2024-03-23T11:18:57.256551+0000","last_fullsized":"2024-03-23T11:18:57.256551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:22.095583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100170+0000","last_change":"2024-03-23T11:18:55.783962+0000","last_active":"2024-03-23T11:18:58.100170+0000","last_peered":"2024-03-23T11:18:58.100170+0000","last_clean":"2024-03-23T11:18:58.100170+0000","last_became_active":"2024-03-23T11:18:55.783729+0000","last_became_peered":"2024-03-23T11:18:55.783729+0000","last_unstale":"2024-03-23T11:18:58.100170+0000","last_undegraded":"2024-03-23T11:18:58.100170+0000","last_fullsized":"2024-03-23T11:18:58.100170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:42.674448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098331+0000","last_change":"2024-03-23T11:18:56.261635+0000","last_active":"2024-03-23T11:18:58.098331+0000","last_peered":"2024-03-23T11:18:58.098331+0000","last_clean":"2024-03-23T11:18:58.098331+0000","last_became_active":"2024-03-23T11:18:56.261334+0000","last_became_peered":"2024-03-23T11:18:56.261334+0000","last_unstale":"2024-03-23T11:18:58.098331+0000","last_undegraded":"2024-03-23T11:18:58.098331+0000","last_fullsized":"2024-03-23T11:18:58.098331+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.959980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314958+0000","last_change":"2024-03-23T11:18:55.782301+0000","last_active":"2024-03-23T11:18:57.314958+0000","last_peered":"2024-03-23T11:18:57.314958+0000","last_clean":"2024-03-23T11:18:57.314958+0000","last_became_active":"2024-03-23T11:18:55.782005+0000","last_became_peered":"2024-03-23T11:18:55.782005+0000","last_unstale":"2024-03-23T11:18:57.314958+0000","last_undegraded":"2024-03-23T11:18:57.314958+0000","last_fullsized":"2024-03-23T11:18:57.314958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256246+0000","last_change":"2024-03-23T11:18:56.275199+0000","last_active":"2024-03-23T11:18:57.256246+0000","last_peered":"2024-03-23T11:18:57.256246+0000","last_clean":"2024-03-23T11:18:57.256246+0000","last_became_active":"2024-03-23T11:18:56.274944+0000","last_became_peered":"2024-03-23T11:18:56.274944+0000","last_unstale":"2024-03-23T11:18:57.256246+0000","last_undegraded":"2024-03-23T11:18:57.256246+0000","last_fullsized":"2024-03-23T11:18:57.256246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:00:18.525010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461578+0000","last_change":"2024-03-23T11:18:55.775042+0000","last_active":"2024-03-23T11:18:57.461578+0000","last_peered":"2024-03-23T11:18:57.461578+0000","last_clean":"2024-03-23T11:18:57.461578+0000","last_became_active":"2024-03-23T11:18:55.774721+0000","last_became_peered":"2024-03-23T11:18:55.774721+0000","last_unstale":"2024-03-23T11:18:57.461578+0000","last_undegraded":"2024-03-23T11:18:57.461578+0000","last_fullsized":"2024-03-23T11:18:57.461578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.461383+0000","last_change":"2024-03-23T11:18:56.263485+0000","last_active":"2024-03-23T11:18:57.461383+0000","last_peered":"2024-03-23T11:18:57.461383+0000","last_clean":"2024-03-23T11:18:57.461383+0000","last_became_active":"2024-03-23T11:18:56.261677+0000","last_became_peered":"2024-03-23T11:18:56.261677+0000","last_unstale":"2024-03-23T11:18:57.461383+0000","last_undegraded":"2024-03-23T11:18:57.461383+0000","last_fullsized":"2024-03-23T11:18:57.461383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098165+0000","last_change":"2024-03-23T11:18:55.770098+0000","last_active":"2024-03-23T11:18:58.098165+0000","last_peered":"2024-03-23T11:18:58.098165+0000","last_clean":"2024-03-23T11:18:58.098165+0000","last_became_active":"2024-03-23T11:18:55.769814+0000","last_became_peered":"2024-03-23T11:18:55.769814+0000","last_unstale":"2024-03-23T11:18:58.098165+0000","last_undegraded":"2024-03-23T11:18:58.098165+0000","last_fullsized":"2024-03-23T11:18:58.098165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.906819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255419+0000","last_change":"2024-03-23T11:18:56.274715+0000","last_active":"2024-03-23T11:18:57.255419+0000","last_peered":"2024-03-23T11:18:57.255419+0000","last_clean":"2024-03-23T11:18:57.255419+0000","last_became_active":"2024-03-23T11:18:56.274437+0000","last_became_peered":"2024-03-23T11:18:56.274437+0000","last_unstale":"2024-03-23T11:18:57.255419+0000","last_undegraded":"2024-03-23T11:18:57.255419+0000","last_fullsized":"2024-03-23T11:18:57.255419+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:28:00.964447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256032+0000","last_change":"2024-03-23T11:18:55.789682+0000","last_active":"2024-03-23T11:18:57.256032+0000","last_peered":"2024-03-23T11:18:57.256032+0000","last_clean":"2024-03-23T11:18:57.256032+0000","last_became_active":"2024-03-23T11:18:55.789278+0000","last_became_peered":"2024-03-23T11:18:55.789278+0000","last_unstale":"2024-03-23T11:18:57.256032+0000","last_undegraded":"2024-03-23T11:18:57.256032+0000","last_fullsized":"2024-03-23T11:18:57.256032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:42.674448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256896+0000","last_change":"2024-03-23T11:18:56.267163+0000","last_active":"2024-03-23T11:18:57.256896+0000","last_peered":"2024-03-23T11:18:57.256896+0000","last_clean":"2024-03-23T11:18:57.256896+0000","last_became_active":"2024-03-23T11:18:56.266907+0000","last_became_peered":"2024-03-23T11:18:56.266907+0000","last_unstale":"2024-03-23T11:18:57.256896+0000","last_undegraded":"2024-03-23T11:18:57.256896+0000","last_fullsized":"2024-03-23T11:18:57.256896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:51:52.682495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100710+0000","last_change":"2024-03-23T11:18:55.779353+0000","last_active":"2024-03-23T11:18:58.100710+0000","last_peered":"2024-03-23T11:18:58.100710+0000","last_clean":"2024-03-23T11:18:58.100710+0000","last_became_active":"2024-03-23T11:18:55.779045+0000","last_became_peered":"2024-03-23T11:18:55.779045+0000","last_unstale":"2024-03-23T11:18:58.100710+0000","last_undegraded":"2024-03-23T11:18:58.100710+0000","last_fullsized":"2024-03-23T11:18:58.100710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257949+0000","last_change":"2024-03-23T11:18:56.275906+0000","last_active":"2024-03-23T11:18:57.257949+0000","last_peered":"2024-03-23T11:18:57.257949+0000","last_clean":"2024-03-23T11:18:57.257949+0000","last_became_active":"2024-03-23T11:18:56.274500+0000","last_became_peered":"2024-03-23T11:18:56.274500+0000","last_unstale":"2024-03-23T11:18:57.257949+0000","last_undegraded":"2024-03-23T11:18:57.257949+0000","last_fullsized":"2024-03-23T11:18:57.257949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:13:34.436113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253366+0000","last_change":"2024-03-23T11:18:55.763585+0000","last_active":"2024-03-23T11:18:57.253366+0000","last_peered":"2024-03-23T11:18:57.253366+0000","last_clean":"2024-03-23T11:18:57.253366+0000","last_became_active":"2024-03-23T11:18:55.762854+0000","last_became_peered":"2024-03-23T11:18:55.762854+0000","last_unstale":"2024-03-23T11:18:57.253366+0000","last_undegraded":"2024-03-23T11:18:57.253366+0000","last_fullsized":"2024-03-23T11:18:57.253366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:38.995255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315476+0000","last_change":"2024-03-23T11:18:56.276682+0000","last_active":"2024-03-23T11:18:57.315476+0000","last_peered":"2024-03-23T11:18:57.315476+0000","last_clean":"2024-03-23T11:18:57.315476+0000","last_became_active":"2024-03-23T11:18:56.276094+0000","last_became_peered":"2024-03-23T11:18:56.276094+0000","last_unstale":"2024-03-23T11:18:57.315476+0000","last_undegraded":"2024-03-23T11:18:57.315476+0000","last_fullsized":"2024-03-23T11:18:57.315476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:26.126571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098569+0000","last_change":"2024-03-23T11:18:55.783331+0000","last_active":"2024-03-23T11:18:58.098569+0000","last_peered":"2024-03-23T11:18:58.098569+0000","last_clean":"2024-03-23T11:18:58.098569+0000","last_became_active":"2024-03-23T11:18:55.783030+0000","last_became_peered":"2024-03-23T11:18:55.783030+0000","last_unstale":"2024-03-23T11:18:58.098569+0000","last_undegraded":"2024-03-23T11:18:58.098569+0000","last_fullsized":"2024-03-23T11:18:58.098569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:27:34.105727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256897+0000","last_change":"2024-03-23T11:18:56.284587+0000","last_active":"2024-03-23T11:18:57.256897+0000","last_peered":"2024-03-23T11:18:57.256897+0000","last_clean":"2024-03-23T11:18:57.256897+0000","last_became_active":"2024-03-23T11:18:56.283582+0000","last_became_peered":"2024-03-23T11:18:56.283582+0000","last_unstale":"2024-03-23T11:18:57.256897+0000","last_undegraded":"2024-03-23T11:18:57.256897+0000","last_fullsized":"2024-03-23T11:18:57.256897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:01:11.724895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":164,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T11:18:56.246747+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:18:56.246747+0000","last_peered":"2024-03-23T11:18:56.246747+0000","last_clean":"2024-03-23T11:18:56.246747+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:18:56.246747+0000","last_undegraded":"2024-03-23T11:18:56.246747+0000","last_fullsized":"2024-03-23T11:18:56.246747+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.314507+0000","last_change":"2024-03-23T11:18:55.778697+0000","last_active":"2024-03-23T11:18:57.314507+0000","last_peered":"2024-03-23T11:18:57.314507+0000","last_clean":"2024-03-23T11:18:57.314507+0000","last_became_active":"2024-03-23T11:18:55.778387+0000","last_became_peered":"2024-03-23T11:18:55.778387+0000","last_unstale":"2024-03-23T11:18:57.314507+0000","last_undegraded":"2024-03-23T11:18:57.314507+0000","last_fullsized":"2024-03-23T11:18:57.314507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100582+0000","last_change":"2024-03-23T11:18:56.268261+0000","last_active":"2024-03-23T11:18:58.100582+0000","last_peered":"2024-03-23T11:18:58.100582+0000","last_clean":"2024-03-23T11:18:58.100582+0000","last_became_active":"2024-03-23T11:18:56.267974+0000","last_became_peered":"2024-03-23T11:18:56.267974+0000","last_unstale":"2024-03-23T11:18:58.100582+0000","last_undegraded":"2024-03-23T11:18:58.100582+0000","last_fullsized":"2024-03-23T11:18:58.100582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:44.879627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255801+0000","last_change":"2024-03-23T11:18:55.775332+0000","last_active":"2024-03-23T11:18:57.255801+0000","last_peered":"2024-03-23T11:18:57.255801+0000","last_clean":"2024-03-23T11:18:57.255801+0000","last_became_active":"2024-03-23T11:18:55.775009+0000","last_became_peered":"2024-03-23T11:18:55.775009+0000","last_unstale":"2024-03-23T11:18:57.255801+0000","last_undegraded":"2024-03-23T11:18:57.255801+0000","last_fullsized":"2024-03-23T11:18:57.255801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:08:12.342783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098600+0000","last_change":"2024-03-23T11:18:56.279541+0000","last_active":"2024-03-23T11:18:58.098600+0000","last_peered":"2024-03-23T11:18:58.098600+0000","last_clean":"2024-03-23T11:18:58.098600+0000","last_became_active":"2024-03-23T11:18:56.279231+0000","last_became_peered":"2024-03-23T11:18:56.279231+0000","last_unstale":"2024-03-23T11:18:58.098600+0000","last_undegraded":"2024-03-23T11:18:58.098600+0000","last_fullsized":"2024-03-23T11:18:58.098600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:19.216031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.098389+0000","last_change":"2024-03-23T11:18:55.777288+0000","last_active":"2024-03-23T11:18:58.098389+0000","last_peered":"2024-03-23T11:18:58.098389+0000","last_clean":"2024-03-23T11:18:58.098389+0000","last_became_active":"2024-03-23T11:18:55.776998+0000","last_became_peered":"2024-03-23T11:18:55.776998+0000","last_unstale":"2024-03-23T11:18:58.098389+0000","last_undegraded":"2024-03-23T11:18:58.098389+0000","last_fullsized":"2024-03-23T11:18:58.098389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252646+0000","last_change":"2024-03-23T11:18:56.257547+0000","last_active":"2024-03-23T11:18:57.252646+0000","last_peered":"2024-03-23T11:18:57.252646+0000","last_clean":"2024-03-23T11:18:57.252646+0000","last_became_active":"2024-03-23T11:18:56.257307+0000","last_became_peered":"2024-03-23T11:18:56.257307+0000","last_unstale":"2024-03-23T11:18:57.252646+0000","last_undegraded":"2024-03-23T11:18:57.252646+0000","last_fullsized":"2024-03-23T11:18:57.252646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:32.498133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315167+0000","last_change":"2024-03-23T11:18:55.780573+0000","last_active":"2024-03-23T11:18:57.315167+0000","last_peered":"2024-03-23T11:18:57.315167+0000","last_clean":"2024-03-23T11:18:57.315167+0000","last_became_active":"2024-03-23T11:18:55.779963+0000","last_became_peered":"2024-03-23T11:18:55.779963+0000","last_unstale":"2024-03-23T11:18:57.315167+0000","last_undegraded":"2024-03-23T11:18:57.315167+0000","last_fullsized":"2024-03-23T11:18:57.315167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:18.352259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255291+0000","last_change":"2024-03-23T11:18:56.264913+0000","last_active":"2024-03-23T11:18:57.255291+0000","last_peered":"2024-03-23T11:18:57.255291+0000","last_clean":"2024-03-23T11:18:57.255291+0000","last_became_active":"2024-03-23T11:18:56.264679+0000","last_became_peered":"2024-03-23T11:18:56.264679+0000","last_unstale":"2024-03-23T11:18:57.255291+0000","last_undegraded":"2024-03-23T11:18:57.255291+0000","last_fullsized":"2024-03-23T11:18:57.255291+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:16:46.738802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.253124+0000","last_change":"2024-03-23T11:18:55.785635+0000","last_active":"2024-03-23T11:18:57.253124+0000","last_peered":"2024-03-23T11:18:57.253124+0000","last_clean":"2024-03-23T11:18:57.253124+0000","last_became_active":"2024-03-23T11:18:55.785064+0000","last_became_peered":"2024-03-23T11:18:55.785064+0000","last_unstale":"2024-03-23T11:18:57.253124+0000","last_undegraded":"2024-03-23T11:18:57.253124+0000","last_fullsized":"2024-03-23T11:18:57.253124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:04:27.564911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315028+0000","last_change":"2024-03-23T11:18:56.276583+0000","last_active":"2024-03-23T11:18:57.315028+0000","last_peered":"2024-03-23T11:18:57.315028+0000","last_clean":"2024-03-23T11:18:57.315028+0000","last_became_active":"2024-03-23T11:18:56.276294+0000","last_became_peered":"2024-03-23T11:18:56.276294+0000","last_unstale":"2024-03-23T11:18:57.315028+0000","last_undegraded":"2024-03-23T11:18:57.315028+0000","last_fullsized":"2024-03-23T11:18:57.315028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:36.411878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257226+0000","last_change":"2024-03-23T11:18:55.784141+0000","last_active":"2024-03-23T11:18:57.257226+0000","last_peered":"2024-03-23T11:18:57.257226+0000","last_clean":"2024-03-23T11:18:57.257226+0000","last_became_active":"2024-03-23T11:18:55.783843+0000","last_became_peered":"2024-03-23T11:18:55.783843+0000","last_unstale":"2024-03-23T11:18:57.257226+0000","last_undegraded":"2024-03-23T11:18:57.257226+0000","last_fullsized":"2024-03-23T11:18:57.257226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:57:29.038916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252794+0000","last_change":"2024-03-23T11:18:56.258329+0000","last_active":"2024-03-23T11:18:57.252794+0000","last_peered":"2024-03-23T11:18:57.252794+0000","last_clean":"2024-03-23T11:18:57.252794+0000","last_became_active":"2024-03-23T11:18:56.258033+0000","last_became_peered":"2024-03-23T11:18:56.258033+0000","last_unstale":"2024-03-23T11:18:57.252794+0000","last_undegraded":"2024-03-23T11:18:57.252794+0000","last_fullsized":"2024-03-23T11:18:57.252794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:24:37.470160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.314065+0000","last_change":"2024-03-23T11:18:55.778103+0000","last_active":"2024-03-23T11:18:58.314065+0000","last_peered":"2024-03-23T11:18:58.314065+0000","last_clean":"2024-03-23T11:18:58.314065+0000","last_became_active":"2024-03-23T11:18:55.777604+0000","last_became_peered":"2024-03-23T11:18:55.777604+0000","last_unstale":"2024-03-23T11:18:58.314065+0000","last_undegraded":"2024-03-23T11:18:58.314065+0000","last_fullsized":"2024-03-23T11:18:58.314065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:42:44.181703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101839+0000","last_change":"2024-03-23T11:18:56.280720+0000","last_active":"2024-03-23T11:18:58.101839+0000","last_peered":"2024-03-23T11:18:58.101839+0000","last_clean":"2024-03-23T11:18:58.101839+0000","last_became_active":"2024-03-23T11:18:56.278952+0000","last_became_peered":"2024-03-23T11:18:56.278952+0000","last_unstale":"2024-03-23T11:18:58.101839+0000","last_undegraded":"2024-03-23T11:18:58.101839+0000","last_fullsized":"2024-03-23T11:18:58.101839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:57.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462270+0000","last_change":"2024-03-23T11:18:55.776925+0000","last_active":"2024-03-23T11:18:57.462270+0000","last_peered":"2024-03-23T11:18:57.462270+0000","last_clean":"2024-03-23T11:18:57.462270+0000","last_became_active":"2024-03-23T11:18:55.776651+0000","last_became_peered":"2024-03-23T11:18:55.776651+0000","last_unstale":"2024-03-23T11:18:57.462270+0000","last_undegraded":"2024-03-23T11:18:57.462270+0000","last_fullsized":"2024-03-23T11:18:57.462270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:39.750345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099552+0000","last_change":"2024-03-23T11:18:56.268554+0000","last_active":"2024-03-23T11:18:58.099552+0000","last_peered":"2024-03-23T11:18:58.099552+0000","last_clean":"2024-03-23T11:18:58.099552+0000","last_became_active":"2024-03-23T11:18:56.267201+0000","last_became_peered":"2024-03-23T11:18:56.267201+0000","last_unstale":"2024-03-23T11:18:58.099552+0000","last_undegraded":"2024-03-23T11:18:58.099552+0000","last_fullsized":"2024-03-23T11:18:58.099552+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.705001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.255562+0000","last_change":"2024-03-23T11:18:55.789376+0000","last_active":"2024-03-23T11:18:57.255562+0000","last_peered":"2024-03-23T11:18:57.255562+0000","last_clean":"2024-03-23T11:18:57.255562+0000","last_became_active":"2024-03-23T11:18:55.789122+0000","last_became_peered":"2024-03-23T11:18:55.789122+0000","last_unstale":"2024-03-23T11:18:57.255562+0000","last_undegraded":"2024-03-23T11:18:57.255562+0000","last_fullsized":"2024-03-23T11:18:57.255562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:26.469433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.257058+0000","last_change":"2024-03-23T11:18:56.267674+0000","last_active":"2024-03-23T11:18:57.257058+0000","last_peered":"2024-03-23T11:18:57.257058+0000","last_clean":"2024-03-23T11:18:57.257058+0000","last_became_active":"2024-03-23T11:18:56.267418+0000","last_became_peered":"2024-03-23T11:18:56.267418+0000","last_unstale":"2024-03-23T11:18:57.257058+0000","last_undegraded":"2024-03-23T11:18:57.257058+0000","last_fullsized":"2024-03-23T11:18:57.257058+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:36:11.188974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252776+0000","last_change":"2024-03-23T11:18:55.775660+0000","last_active":"2024-03-23T11:18:57.252776+0000","last_peered":"2024-03-23T11:18:57.252776+0000","last_clean":"2024-03-23T11:18:57.252776+0000","last_became_active":"2024-03-23T11:18:55.775378+0000","last_became_peered":"2024-03-23T11:18:55.775378+0000","last_unstale":"2024-03-23T11:18:57.252776+0000","last_undegraded":"2024-03-23T11:18:57.252776+0000","last_fullsized":"2024-03-23T11:18:57.252776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:25:31.204286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.100033+0000","last_change":"2024-03-23T11:18:56.273073+0000","last_active":"2024-03-23T11:18:58.100033+0000","last_peered":"2024-03-23T11:18:58.100033+0000","last_clean":"2024-03-23T11:18:58.100033+0000","last_became_active":"2024-03-23T11:18:56.265279+0000","last_became_peered":"2024-03-23T11:18:56.265279+0000","last_unstale":"2024-03-23T11:18:58.100033+0000","last_undegraded":"2024-03-23T11:18:58.100033+0000","last_fullsized":"2024-03-23T11:18:58.100033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:08.274920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.252994+0000","last_change":"2024-03-23T11:18:55.773923+0000","last_active":"2024-03-23T11:18:57.252994+0000","last_peered":"2024-03-23T11:18:57.252994+0000","last_clean":"2024-03-23T11:18:57.252994+0000","last_became_active":"2024-03-23T11:18:55.773155+0000","last_became_peered":"2024-03-23T11:18:55.773155+0000","last_unstale":"2024-03-23T11:18:57.252994+0000","last_undegraded":"2024-03-23T11:18:57.252994+0000","last_fullsized":"2024-03-23T11:18:57.252994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:26.469433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101280+0000","last_change":"2024-03-23T11:18:56.265422+0000","last_active":"2024-03-23T11:18:58.101280+0000","last_peered":"2024-03-23T11:18:58.101280+0000","last_clean":"2024-03-23T11:18:58.101280+0000","last_became_active":"2024-03-23T11:18:56.265137+0000","last_became_peered":"2024-03-23T11:18:56.265137+0000","last_unstale":"2024-03-23T11:18:58.101280+0000","last_undegraded":"2024-03-23T11:18:58.101280+0000","last_fullsized":"2024-03-23T11:18:58.101280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:26.126571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256335+0000","last_change":"2024-03-23T11:18:55.778476+0000","last_active":"2024-03-23T11:18:57.256335+0000","last_peered":"2024-03-23T11:18:57.256335+0000","last_clean":"2024-03-23T11:18:57.256335+0000","last_became_active":"2024-03-23T11:18:55.778184+0000","last_became_peered":"2024-03-23T11:18:55.778184+0000","last_unstale":"2024-03-23T11:18:57.256335+0000","last_undegraded":"2024-03-23T11:18:57.256335+0000","last_fullsized":"2024-03-23T11:18:57.256335+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:38.995255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.462125+0000","last_change":"2024-03-23T11:18:56.277871+0000","last_active":"2024-03-23T11:18:57.462125+0000","last_peered":"2024-03-23T11:18:57.462125+0000","last_clean":"2024-03-23T11:18:57.462125+0000","last_became_active":"2024-03-23T11:18:56.277547+0000","last_became_peered":"2024-03-23T11:18:56.277547+0000","last_unstale":"2024-03-23T11:18:57.462125+0000","last_undegraded":"2024-03-23T11:18:57.462125+0000","last_fullsized":"2024-03-23T11:18:57.462125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.747240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315015+0000","last_change":"2024-03-23T11:18:56.288533+0000","last_active":"2024-03-23T11:18:57.315015+0000","last_peered":"2024-03-23T11:18:57.315015+0000","last_clean":"2024-03-23T11:18:57.315015+0000","last_became_active":"2024-03-23T11:18:56.288099+0000","last_became_peered":"2024-03-23T11:18:56.288099+0000","last_unstale":"2024-03-23T11:18:57.315015+0000","last_undegraded":"2024-03-23T11:18:57.315015+0000","last_fullsized":"2024-03-23T11:18:57.315015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:31:13.473526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256142+0000","last_change":"2024-03-23T11:18:55.784333+0000","last_active":"2024-03-23T11:18:57.256142+0000","last_peered":"2024-03-23T11:18:57.256142+0000","last_clean":"2024-03-23T11:18:57.256142+0000","last_became_active":"2024-03-23T11:18:55.783931+0000","last_became_peered":"2024-03-23T11:18:55.783931+0000","last_unstale":"2024-03-23T11:18:57.256142+0000","last_undegraded":"2024-03-23T11:18:57.256142+0000","last_fullsized":"2024-03-23T11:18:57.256142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:04:27.564911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.315497+0000","last_change":"2024-03-23T11:18:56.287212+0000","last_active":"2024-03-23T11:18:57.315497+0000","last_peered":"2024-03-23T11:18:57.315497+0000","last_clean":"2024-03-23T11:18:57.315497+0000","last_became_active":"2024-03-23T11:18:56.284693+0000","last_became_peered":"2024-03-23T11:18:56.284693+0000","last_unstale":"2024-03-23T11:18:57.315497+0000","last_undegraded":"2024-03-23T11:18:57.315497+0000","last_fullsized":"2024-03-23T11:18:57.315497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:33.038484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:57.256937+0000","last_change":"2024-03-23T11:18:55.779221+0000","last_active":"2024-03-23T11:18:57.256937+0000","last_peered":"2024-03-23T11:18:57.256937+0000","last_clean":"2024-03-23T11:18:57.256937+0000","last_became_active":"2024-03-23T11:18:55.778912+0000","last_became_peered":"2024-03-23T11:18:55.778912+0000","last_unstale":"2024-03-23T11:18:57.256937+0000","last_undegraded":"2024-03-23T11:18:57.256937+0000","last_fullsized":"2024-03-23T11:18:57.256937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:28:51.146588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099197+0000","last_change":"2024-03-23T11:18:56.291118+0000","last_active":"2024-03-23T11:18:58.099197+0000","last_peered":"2024-03-23T11:18:58.099197+0000","last_clean":"2024-03-23T11:18:58.099197+0000","last_became_active":"2024-03-23T11:18:56.290750+0000","last_became_peered":"2024-03-23T11:18:56.290750+0000","last_unstale":"2024-03-23T11:18:58.099197+0000","last_undegraded":"2024-03-23T11:18:58.099197+0000","last_fullsized":"2024-03-23T11:18:58.099197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:06.430105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101527+0000","last_change":"2024-03-23T11:18:55.784254+0000","last_active":"2024-03-23T11:18:58.101527+0000","last_peered":"2024-03-23T11:18:58.101527+0000","last_clean":"2024-03-23T11:18:58.101527+0000","last_became_active":"2024-03-23T11:18:55.781763+0000","last_became_peered":"2024-03-23T11:18:55.781763+0000","last_unstale":"2024-03-23T11:18:58.101527+0000","last_undegraded":"2024-03-23T11:18:58.101527+0000","last_fullsized":"2024-03-23T11:18:58.101527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:14.847953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.101387+0000","last_change":"2024-03-23T11:18:56.267771+0000","last_active":"2024-03-23T11:18:58.101387+0000","last_peered":"2024-03-23T11:18:58.101387+0000","last_clean":"2024-03-23T11:18:58.101387+0000","last_became_active":"2024-03-23T11:18:56.267515+0000","last_became_peered":"2024-03-23T11:18:56.267515+0000","last_unstale":"2024-03-23T11:18:58.101387+0000","last_undegraded":"2024-03-23T11:18:58.101387+0000","last_fullsized":"2024-03-23T11:18:58.101387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:55.733931+0000","last_clean_scrub_stamp":"2024-03-23T11:18:55.733931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:11.915522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-23T11:18:58.099857+0000","last_change":"2024-03-23T11:18:55.785970+0000","last_active":"2024-03-23T11:18:58.099857+0000","last_peered":"2024-03-23T11:18:58.099857+0000","last_clean":"2024-03-23T11:18:58.099857+0000","last_became_active":"2024-03-23T11:18:55.784347+0000","last_became_peered":"2024-03-23T11:18:55.784347+0000","last_unstale":"2024-03-23T11:18:58.099857+0000","last_undegraded":"2024-03-23T11:18:58.099857+0000","last_fullsized":"2024-03-23T11:18:58.099857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:54.552392+0000","last_clean_scrub_stamp":"2024-03-23T11:18:54.552392+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:36:21.274877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":823296,"data_stored":669740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.577}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7849999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1859999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.254}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.764}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":78916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.702}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.845}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0419999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0529999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3460000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1819999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":78281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2889999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5069999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6060000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":827392,"data_stored":669127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:19:07.396 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-23T11:19:07.397 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-23T11:19:07.397 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-23T11:19:07.397 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-23T11:19:07.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.652+0000 7f4157e00640 1 Processor -- start 2024-03-23T11:19:07.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.653+0000 7f4157e00640 1 -- start start 2024-03-23T11:19:07.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.654+0000 7f4157e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41501571e0 0x7f41500ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.654+0000 7f4157e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f41500ade70 0x7f415006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.654+0000 7f4157e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415006cf40 con 0x7f41501613a0 2024-03-23T11:19:07.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.654+0000 7f4157e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41501575c0 con 0x7f41500ade70 2024-03-23T11:19:07.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.654+0000 7f4157e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415006c3a0 con 0x7f41501571e0 2024-03-23T11:19:07.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.655+0000 7f4154a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f41500ade70 0x7f415006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.655+0000 7f4155400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41501571e0 0x7f41500ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.655+0000 7f4154a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f41500ade70 0x7f415006d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53588/0 (socket says 172.21.15.62:53588) 2024-03-23T11:19:07.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.655+0000 7f4154a00640 1 -- 172.21.15.62:0/2566518096 learned_addr learned my addr 172.21.15.62:0/2566518096 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:07.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.656+0000 7f4154a00640 1 -- 172.21.15.62:0/2566518096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41501571e0 msgr2=0x7f41500ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:07.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.656+0000 7f4154a00640 1 --2- 172.21.15.62:0/2566518096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41501571e0 0x7f41500ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.656+0000 7f4154a00640 1 -- 172.21.15.62:0/2566518096 >> v1:172.21.15.120:6789/0 conn(0x7f41501613a0 legacy=0x7f415015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.656+0000 7f4154a00640 1 -- 172.21.15.62:0/2566518096 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415006c610 con 0x7f41500ade70 2024-03-23T11:19:07.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.657+0000 7f4154a00640 1 --2- 172.21.15.62:0/2566518096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41500ade70 0x7f415006d3d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f413c009870 tx=0x7f413c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9ef5e0c1c540ef8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:07.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.657+0000 7f414fe00640 1 -- 172.21.15.62:0/2566518096 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c031640 con 0x7f41500ade70 2024-03-23T11:19:07.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.657+0000 7f414fe00640 1 -- 172.21.15.62:0/2566518096 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413c0317a0 con 0x7f41500ade70 2024-03-23T11:19:07.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.657+0000 7f414fe00640 1 -- 172.21.15.62:0/2566518096 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c031b30 con 0x7f41500ade70 2024-03-23T11:19:07.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.658+0000 7f4157e00640 1 -- 172.21.15.62:0/2566518096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41500ade70 msgr2=0x7f415006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.658+0000 7f4157e00640 1 --2- 172.21.15.62:0/2566518096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41500ade70 0x7f415006d3d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f413c009870 tx=0x7f413c02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.658+0000 7f4157e00640 1 -- 172.21.15.62:0/2566518096 shutdown_connections 2024-03-23T11:19:07.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.658+0000 7f4157e00640 1 --2- 172.21.15.62:0/2566518096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41500ade70 0x7f415006d3d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.658+0000 7f4157e00640 1 --2- 172.21.15.62:0/2566518096 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41501571e0 0x7f41500ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.658+0000 7f4157e00640 1 -- 172.21.15.62:0/2566518096 >> 172.21.15.62:0/2566518096 conn(0x7f41500b02e0 msgr2=0x7f41500b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:07.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.658+0000 7f4157e00640 1 -- 172.21.15.62:0/2566518096 shutdown_connections 2024-03-23T11:19:07.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.658+0000 7f4157e00640 1 -- 172.21.15.62:0/2566518096 wait complete. 2024-03-23T11:19:07.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.660+0000 7f4157e00640 1 Processor -- start 2024-03-23T11:19:07.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.660+0000 7f4157e00640 1 -- start start 2024-03-23T11:19:07.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4157e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41500ade70 0x7f4150130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4157e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41501571e0 0x7f41500af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4157e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41501613a0 0x7f41501f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4157e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41500afb50 con 0x7f41500ade70 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4157e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4150131160 con 0x7f41501613a0 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4157e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4150131440 con 0x7f41501571e0 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4155400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41500ade70 0x7f4150130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4155400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41500ade70 0x7f4150130b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52304/0 (socket says 172.21.15.62:52304) 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4155400640 1 -- 172.21.15.62:0/3828430672 learned_addr learned my addr 172.21.15.62:0/3828430672 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4155e00640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41501613a0 0x7f41501f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4154a00640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41501571e0 0x7f41500af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4155400640 1 -- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41501571e0 msgr2=0x7f41500af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4155400640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41501571e0 0x7f41500af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4155400640 1 -- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41501613a0 msgr2=0x7f41501f1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4155400640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41501613a0 0x7f41501f1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.661+0000 7f4155400640 1 -- 172.21.15.62:0/3828430672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c0094d0 con 0x7f41500ade70 2024-03-23T11:19:07.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.662+0000 7f4155e00640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41501613a0 0x7f41501f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:07.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.662+0000 7f4155400640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41500ade70 0x7f4150130b80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f414400e9b0 tx=0x7f414400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:07.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.662+0000 7f414d600640 1 -- 172.21.15.62:0/3828430672 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4144009670 con 0x7f41500ade70 2024-03-23T11:19:07.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.662+0000 7f414d600640 1 -- 172.21.15.62:0/3828430672 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414400c9a0 con 0x7f41500ade70 2024-03-23T11:19:07.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.662+0000 7f414d600640 1 -- 172.21.15.62:0/3828430672 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414400cd50 con 0x7f41500ade70 2024-03-23T11:19:07.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.662+0000 7f4157e00640 1 -- 172.21.15.62:0/3828430672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4150131720 con 0x7f41500ade70 2024-03-23T11:19:07.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.662+0000 7f4157e00640 1 -- 172.21.15.62:0/3828430672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4150131ce0 con 0x7f41500ade70 2024-03-23T11:19:07.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.663+0000 7f4157e00640 1 -- 172.21.15.62:0/3828430672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4150066c90 con 0x7f41500ade70 2024-03-23T11:19:07.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.664+0000 7f414d600640 1 -- 172.21.15.62:0/3828430672 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4144013470 con 0x7f41500ade70 2024-03-23T11:19:07.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.665+0000 7f414d600640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f413006a7a0 0x7f413006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:07.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.665+0000 7f414d600640 1 -- 172.21.15.62:0/3828430672 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4144089090 con 0x7f41500ade70 2024-03-23T11:19:07.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.666+0000 7f4154a00640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f413006a7a0 0x7f413006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:07.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.667+0000 7f4154a00640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f413006a7a0 0x7f413006cc60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f413c036000 tx=0x7f413c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:07.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.669+0000 7f414d600640 1 -- 172.21.15.62:0/3828430672 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f41440544b0 con 0x7f41500ade70 2024-03-23T11:19:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.899+0000 7f4157e00640 1 -- 172.21.15.62:0/3828430672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f41500b38c0 con 0x7f41500ade70 2024-03-23T11:19:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.900+0000 7f414d600640 1 -- 172.21.15.62:0/3828430672 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4144057ad0 con 0x7f41500ade70 2024-03-23T11:19:07.900 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:07.901 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-23T11:19:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.902+0000 7f4157e00640 1 -- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f413006a7a0 msgr2=0x7f413006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.902+0000 7f4157e00640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f413006a7a0 0x7f413006cc60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f413c036000 tx=0x7f413c03c040 comp rx=0 tx=0).stop 2024-03-23T11:19:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.902+0000 7f4157e00640 1 -- 172.21.15.62:0/3828430672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41500ade70 msgr2=0x7f4150130b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.902+0000 7f4157e00640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41500ade70 0x7f4150130b80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f414400e9b0 tx=0x7f414400ee70 comp rx=0 tx=0).stop 2024-03-23T11:19:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.903+0000 7f4157e00640 1 -- 172.21.15.62:0/3828430672 shutdown_connections 2024-03-23T11:19:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.903+0000 7f4157e00640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f413006a7a0 0x7f413006cc60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.903+0000 7f4157e00640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41501613a0 0x7f41501f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.903+0000 7f4157e00640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41501571e0 0x7f41500af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.903+0000 7f4157e00640 1 --2- 172.21.15.62:0/3828430672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41500ade70 0x7f4150130b80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:07.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.903+0000 7f4157e00640 1 -- 172.21.15.62:0/3828430672 >> 172.21.15.62:0/3828430672 conn(0x7f41500b02e0 msgr2=0x7f4150144c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:07.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.903+0000 7f4157e00640 1 -- 172.21.15.62:0/3828430672 shutdown_connections 2024-03-23T11:19:07.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:07.903+0000 7f4157e00640 1 -- 172.21.15.62:0/3828430672 wait complete. 2024-03-23T11:19:07.921 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-23T11:19:07.922 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:08.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.135+0000 7f1bc7200640 1 Processor -- start 2024-03-23T11:19:08.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.136+0000 7f1bc7200640 1 -- start start 2024-03-23T11:19:08.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.137+0000 7f1bc7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc015c9e0 0x7f1bc01607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.137+0000 7f1bc7200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc006bfc0 con 0x7f1bc015cdc0 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.137+0000 7f1bc7200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc00b5260 con 0x7f1bc00cf650 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.137+0000 7f1bc7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc00d2630 con 0x7f1bc015c9e0 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.138+0000 7f1bbf400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1bc015cdc0 0x7f1bc00d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35986/0 (socket says 172.21.15.62:35986) 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.138+0000 7f1bbf400640 1 -- 172.21.15.62:0/1532766073 learned_addr learned my addr 172.21.15.62:0/1532766073 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.138+0000 7f1bbfe00640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc015c9e0 0x7f1bc01607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.138+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040393186 0 0) 0x7f1bc006bfc0 con 0x7f1bc015cdc0 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.138+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba4003520 con 0x7f1bc015cdc0 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.138+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 824407743 0 0) 0x7f1ba4003520 con 0x7f1bc015cdc0 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.138+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc015c9e0 msgr2=0x7f1bc01607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.138+0000 7f1bbea00640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc015c9e0 0x7f1bc01607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.138+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 >> v1:172.21.15.120:6789/0 conn(0x7f1bc00cf650 legacy=0x7f1bc015d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.138+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc00ce140 con 0x7f1bc015cdc0 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.139+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f1bac001670 con 0x7f1bc015cdc0 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.139+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 >> v1:172.21.15.62:6789/0 conn(0x7f1bc015cdc0 legacy=0x7f1bc00d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.139+0000 7f1bbea00640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ba40039a0 0x7f1ba4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.139+0000 7f1bbea00640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ba40047c0 0x7f1ba4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.139+0000 7f1bbea00640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc015c9e0 0x7f1ba4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.139+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc00d2630 con 0x7f1ba40047c0 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.139+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba400ba60 con 0x7f1ba40039a0 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.139+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba400bc20 con 0x7f1bc015c9e0 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.139+0000 7f1bbf400640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ba40047c0 0x7f1ba4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.139+0000 7f1bbfe00640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ba40039a0 0x7f1ba4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.139+0000 7f1bc5200640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc015c9e0 0x7f1ba4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.140+0000 7f1bbf400640 1 -- 172.21.15.62:0/1532766073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc015c9e0 msgr2=0x7f1ba4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.140+0000 7f1bbf400640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc015c9e0 0x7f1ba4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.140+0000 7f1bbf400640 1 -- 172.21.15.62:0/1532766073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ba40039a0 msgr2=0x7f1ba4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:08.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.140+0000 7f1bbf400640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ba40039a0 0x7f1ba4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.140+0000 7f1bbf400640 1 -- 172.21.15.62:0/1532766073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba4003520 con 0x7f1ba40047c0 2024-03-23T11:19:08.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.141+0000 7f1bbf400640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ba40047c0 0x7f1ba4004c40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1bac001eb0 tx=0x7f1ba40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=279b6de47c2a5d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:08.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.141+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac005ad0 con 0x7f1ba40047c0 2024-03-23T11:19:08.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.141+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac005c50 con 0x7f1ba40047c0 2024-03-23T11:19:08.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.141+0000 7f1bbea00640 1 -- 172.21.15.62:0/1532766073 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac02c7c0 con 0x7f1ba40047c0 2024-03-23T11:19:08.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.142+0000 7f1bc7200640 1 -- 172.21.15.62:0/1532766073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ba40047c0 msgr2=0x7f1ba4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.142+0000 7f1bc7200640 1 --2- 172.21.15.62:0/1532766073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ba40047c0 0x7f1ba4004c40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1bac001eb0 tx=0x7f1ba40057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.142+0000 7f1bc5200640 1 -- 172.21.15.62:0/1532766073 reap_dead start 2024-03-23T11:19:08.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.142+0000 7f1bc7200640 1 -- 172.21.15.62:0/1532766073 shutdown_connections 2024-03-23T11:19:08.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.142+0000 7f1bc7200640 1 -- 172.21.15.62:0/1532766073 >> 172.21.15.62:0/1532766073 conn(0x7f1bc0071570 msgr2=0x7f1bc0071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:08.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.142+0000 7f1bc7200640 1 -- 172.21.15.62:0/1532766073 shutdown_connections 2024-03-23T11:19:08.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.142+0000 7f1bc7200640 1 -- 172.21.15.62:0/1532766073 wait complete. 2024-03-23T11:19:08.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.144+0000 7f1bc7200640 1 Processor -- start 2024-03-23T11:19:08.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.144+0000 7f1bc7200640 1 -- start start 2024-03-23T11:19:08.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.145+0000 7f1bc7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ba40047c0 0x7f1bc015ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.145+0000 7f1bc7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc00d1760 0x7f1bc00d1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.145+0000 7f1bc7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bc00aab10 0x7f1bc00aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.145+0000 7f1bc7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc01f56b0 con 0x7f1bc00aab10 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.145+0000 7f1bc7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc014f470 con 0x7f1ba40047c0 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.145+0000 7f1bc7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc00d22d0 con 0x7f1bc00d1760 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.146+0000 7f1bc5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bc00aab10 0x7f1bc00aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.146+0000 7f1bc5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bc00aab10 0x7f1bc00aaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52322/0 (socket says 172.21.15.62:52322) 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.146+0000 7f1bc5200640 1 -- 172.21.15.62:0/4091276082 learned_addr learned my addr 172.21.15.62:0/4091276082 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.146+0000 7f1bbf400640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc00d1760 0x7f1bc00d1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.146+0000 7f1bbfe00640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ba40047c0 0x7f1bc015ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.146+0000 7f1bc5200640 1 -- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc00d1760 msgr2=0x7f1bc00d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.146+0000 7f1bc5200640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc00d1760 0x7f1bc00d1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.146+0000 7f1bc5200640 1 -- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ba40047c0 msgr2=0x7f1bc015ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.146+0000 7f1bc5200640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ba40047c0 0x7f1bc015ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.146+0000 7f1bc5200640 1 -- 172.21.15.62:0/4091276082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc01f5900 con 0x7f1bc00aab10 2024-03-23T11:19:08.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.146+0000 7f1bc5200640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bc00aab10 0x7f1bc00aaf70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1bc00b5910 tx=0x7f1bb800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:08.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.147+0000 7f1bb7e00640 1 -- 172.21.15.62:0/4091276082 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8003970 con 0x7f1bc00aab10 2024-03-23T11:19:08.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.148+0000 7f1bb7e00640 1 -- 172.21.15.62:0/4091276082 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb8003ad0 con 0x7f1bc00aab10 2024-03-23T11:19:08.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.148+0000 7f1bb7e00640 1 -- 172.21.15.62:0/4091276082 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8003e80 con 0x7f1bc00aab10 2024-03-23T11:19:08.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.148+0000 7f1bc7200640 1 -- 172.21.15.62:0/4091276082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc01f5be0 con 0x7f1bc00aab10 2024-03-23T11:19:08.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.148+0000 7f1bc7200640 1 -- 172.21.15.62:0/4091276082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc015f130 con 0x7f1bc00aab10 2024-03-23T11:19:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.149+0000 7f1bc7200640 1 -- 172.21.15.62:0/4091276082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc015f2c0 con 0x7f1bc00aab10 2024-03-23T11:19:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.150+0000 7f1bb7e00640 1 -- 172.21.15.62:0/4091276082 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f1bb80050d0 con 0x7f1bc00aab10 2024-03-23T11:19:08.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.150+0000 7f1bb7e00640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ba006a7a0 0x7f1ba006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.150+0000 7f1bb7e00640 1 -- 172.21.15.62:0/4091276082 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1bb8088c50 con 0x7f1bc00aab10 2024-03-23T11:19:08.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.151+0000 7f1bbfe00640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ba006a7a0 0x7f1ba006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.152+0000 7f1bbfe00640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ba006a7a0 0x7f1ba006cc60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009a90 tx=0x7f1bb0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:08.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.155+0000 7f1bb7e00640 1 -- 172.21.15.62:0/4091276082 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1bb8053fd0 con 0x7f1bc00aab10 2024-03-23T11:19:08.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.350+0000 7f1bc7200640 1 -- 172.21.15.62:0/4091276082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bc0066c90 con 0x7f1bc00aab10 2024-03-23T11:19:08.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.353+0000 7f1bb7e00640 1 -- 172.21.15.62:0/4091276082 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f1bb80575f0 con 0x7f1bc00aab10 2024-03-23T11:19:08.353 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:08.353 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:57.238201+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:08.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.355+0000 7f1bc7200640 1 -- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ba006a7a0 msgr2=0x7f1ba006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.355+0000 7f1bc7200640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ba006a7a0 0x7f1ba006cc60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009a90 tx=0x7f1bb0008040 comp rx=0 tx=0).stop 2024-03-23T11:19:08.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.355+0000 7f1bc7200640 1 -- 172.21.15.62:0/4091276082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bc00aab10 msgr2=0x7f1bc00aaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.356+0000 7f1bc7200640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bc00aab10 0x7f1bc00aaf70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1bc00b5910 tx=0x7f1bb800ec10 comp rx=0 tx=0).stop 2024-03-23T11:19:08.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.356+0000 7f1bc7200640 1 -- 172.21.15.62:0/4091276082 shutdown_connections 2024-03-23T11:19:08.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.356+0000 7f1bc7200640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ba006a7a0 0x7f1ba006cc60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.356+0000 7f1bc7200640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bc00aab10 0x7f1bc00aaf70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.356+0000 7f1bc7200640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1bc00d1760 0x7f1bc00d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.356+0000 7f1bc7200640 1 --2- 172.21.15.62:0/4091276082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1ba40047c0 0x7f1bc015ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.356+0000 7f1bc7200640 1 -- 172.21.15.62:0/4091276082 >> 172.21.15.62:0/4091276082 conn(0x7f1bc0071570 msgr2=0x7f1bc0071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:08.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.356+0000 7f1bc7200640 1 -- 172.21.15.62:0/4091276082 shutdown_connections 2024-03-23T11:19:08.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.356+0000 7f1bc7200640 1 -- 172.21.15.62:0/4091276082 wait complete. 2024-03-23T11:19:08.373 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:08.373 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:08.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.591+0000 7f2d49c00640 1 Processor -- start 2024-03-23T11:19:08.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.591+0000 7f2d49c00640 1 -- start start 2024-03-23T11:19:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.592+0000 7f2d49c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2d44160420 0x7f2d441608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.592+0000 7f2d49c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d440ce510 con 0x7f2d440b5db0 2024-03-23T11:19:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.592+0000 7f2d49c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d440cdc70 con 0x7f2d4415fcf0 2024-03-23T11:19:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.592+0000 7f2d49c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d44160df0 con 0x7f2d44160420 2024-03-23T11:19:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.593+0000 7f2d43400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2d44160420 0x7f2d441608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.593+0000 7f2d43400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2d44160420 0x7f2d441608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52330/0 (socket says 172.21.15.62:52330) 2024-03-23T11:19:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.593+0000 7f2d43400640 1 -- 172.21.15.62:0/2401700580 learned_addr learned my addr 172.21.15.62:0/2401700580 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.593+0000 7f2d49200640 1 --1- 172.21.15.62:0/2401700580 >> v1:172.21.15.62:6789/0 conn(0x7f2d440b5db0 0x7f2d440b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35992/0 (socket says 172.21.15.62:35992) 2024-03-23T11:19:08.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.593+0000 7f2d43400640 1 -- 172.21.15.62:0/2401700580 >> v1:172.21.15.120:6789/0 conn(0x7f2d4415fcf0 legacy=0x7f2d440d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:08.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.593+0000 7f2d43400640 1 -- 172.21.15.62:0/2401700580 >> v1:172.21.15.62:6789/0 conn(0x7f2d440b5db0 legacy=0x7f2d440b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.593+0000 7f2d43400640 1 -- 172.21.15.62:0/2401700580 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d440cdf50 con 0x7f2d44160420 2024-03-23T11:19:08.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.593+0000 7f2d42a00640 1 -- 172.21.15.62:0/2401700580 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248942904 0 0) 0x7f2d440ce510 con 0x7f2d440b5db0 2024-03-23T11:19:08.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.594+0000 7f2d43400640 1 --2- 172.21.15.62:0/2401700580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d44160420 0x7f2d441608b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009e50 tx=0x7f2d2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d825ddf5dc47ba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:08.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.595+0000 7f2d42a00640 1 -- 172.21.15.62:0/2401700580 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c02ca70 con 0x7f2d44160420 2024-03-23T11:19:08.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.595+0000 7f2d42a00640 1 -- 172.21.15.62:0/2401700580 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2c02cbd0 con 0x7f2d44160420 2024-03-23T11:19:08.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.595+0000 7f2d42a00640 1 -- 172.21.15.62:0/2401700580 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c0343f0 con 0x7f2d44160420 2024-03-23T11:19:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.595+0000 7f2d49c00640 1 -- 172.21.15.62:0/2401700580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d44160420 msgr2=0x7f2d441608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.595+0000 7f2d49c00640 1 --2- 172.21.15.62:0/2401700580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d44160420 0x7f2d441608b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009e50 tx=0x7f2d2c02bed0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.595+0000 7f2d49c00640 1 -- 172.21.15.62:0/2401700580 shutdown_connections 2024-03-23T11:19:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.595+0000 7f2d49c00640 1 --2- 172.21.15.62:0/2401700580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d44160420 0x7f2d441608b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.595+0000 7f2d49c00640 1 -- 172.21.15.62:0/2401700580 >> 172.21.15.62:0/2401700580 conn(0x7f2d44089f60 msgr2=0x7f2d440d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.596+0000 7f2d49c00640 1 -- 172.21.15.62:0/2401700580 shutdown_connections 2024-03-23T11:19:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.596+0000 7f2d49c00640 1 -- 172.21.15.62:0/2401700580 wait complete. 2024-03-23T11:19:08.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.597+0000 7f2d49c00640 1 Processor -- start 2024-03-23T11:19:08.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.597+0000 7f2d49c00640 1 -- start start 2024-03-23T11:19:08.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.598+0000 7f2d49c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d440b5db0 0x7f2d440c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.598+0000 7f2d49c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d4415fcf0 0x7f2d440ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.598+0000 7f2d49c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d44160420 0x7f2d441587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d49c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d441594c0 con 0x7f2d440b5db0 2024-03-23T11:19:08.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d49c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440a0470 con 0x7f2d4415fcf0 2024-03-23T11:19:08.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d49c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440c9420 con 0x7f2d44160420 2024-03-23T11:19:08.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d43e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d440b5db0 0x7f2d440c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d43e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d440b5db0 0x7f2d440c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52334/0 (socket says 172.21.15.62:52334) 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d43e00640 1 -- 172.21.15.62:0/644421640 learned_addr learned my addr 172.21.15.62:0/644421640 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d43400640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d4415fcf0 0x7f2d440ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d49200640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d44160420 0x7f2d441587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d43e00640 1 -- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d44160420 msgr2=0x7f2d441587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d43e00640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d44160420 0x7f2d441587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d43e00640 1 -- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d4415fcf0 msgr2=0x7f2d440ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d43e00640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d4415fcf0 0x7f2d440ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d43e00640 1 -- 172.21.15.62:0/644421640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c009b30 con 0x7f2d440b5db0 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.599+0000 7f2d49200640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d44160420 0x7f2d441587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.600+0000 7f2d43e00640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d440b5db0 0x7f2d440c8ee0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400c460 tx=0x7f2d3400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.600+0000 7f2d3be00640 1 -- 172.21.15.62:0/644421640 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d34011790 con 0x7f2d440b5db0 2024-03-23T11:19:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.600+0000 7f2d3be00640 1 -- 172.21.15.62:0/644421640 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d34011930 con 0x7f2d440b5db0 2024-03-23T11:19:08.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.600+0000 7f2d3be00640 1 -- 172.21.15.62:0/644421640 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d34011ce0 con 0x7f2d440b5db0 2024-03-23T11:19:08.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.600+0000 7f2d49c00640 1 -- 172.21.15.62:0/644421640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d440c5f70 con 0x7f2d440b5db0 2024-03-23T11:19:08.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.600+0000 7f2d49c00640 1 -- 172.21.15.62:0/644421640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d440c64f0 con 0x7f2d440b5db0 2024-03-23T11:19:08.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.601+0000 7f2d49c00640 1 -- 172.21.15.62:0/644421640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d44066bb0 con 0x7f2d440b5db0 2024-03-23T11:19:08.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.602+0000 7f2d3be00640 1 -- 172.21.15.62:0/644421640 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2d3401c460 con 0x7f2d440b5db0 2024-03-23T11:19:08.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.603+0000 7f2d3be00640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d2006a7a0 0x7f2d2006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:08.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.603+0000 7f2d3be00640 1 -- 172.21.15.62:0/644421640 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d34088bb0 con 0x7f2d440b5db0 2024-03-23T11:19:08.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.604+0000 7f2d43400640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d2006a7a0 0x7f2d2006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:08.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.605+0000 7f2d43400640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d2006a7a0 0x7f2d2006cc60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c035000 tx=0x7f2d2c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:08.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.608+0000 7f2d3be00640 1 -- 172.21.15.62:0/644421640 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d34054060 con 0x7f2d440b5db0 2024-03-23T11:19:08.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.804+0000 7f2d49c00640 1 -- 172.21.15.62:0/644421640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d440cea40 con 0x7f2d440b5db0 2024-03-23T11:19:08.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.805+0000 7f2d3be00640 1 -- 172.21.15.62:0/644421640 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d34057680 con 0x7f2d440b5db0 2024-03-23T11:19:08.806 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:08.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 -- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d2006a7a0 msgr2=0x7f2d2006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d2006a7a0 0x7f2d2006cc60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c035000 tx=0x7f2d2c030000 comp rx=0 tx=0).stop 2024-03-23T11:19:08.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 -- 172.21.15.62:0/644421640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d440b5db0 msgr2=0x7f2d440c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:08.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d440b5db0 0x7f2d440c8ee0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400c460 tx=0x7f2d3400c920 comp rx=0 tx=0).stop 2024-03-23T11:19:08.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 -- 172.21.15.62:0/644421640 shutdown_connections 2024-03-23T11:19:08.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d2006a7a0 0x7f2d2006cc60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d44160420 0x7f2d441587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d4415fcf0 0x7f2d440ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 --2- 172.21.15.62:0/644421640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d440b5db0 0x7f2d440c8ee0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:08.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 -- 172.21.15.62:0/644421640 >> 172.21.15.62:0/644421640 conn(0x7f2d44089f60 msgr2=0x7f2d440cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:08.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 -- 172.21.15.62:0/644421640 shutdown_connections 2024-03-23T11:19:08.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:08.809+0000 7f2d49c00640 1 -- 172.21.15.62:0/644421640 wait complete. 2024-03-23T11:19:08.826 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:09.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.048+0000 7f4607200640 1 Processor -- start 2024-03-23T11:19:09.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.048+0000 7f4607200640 1 -- start start 2024-03-23T11:19:09.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.050+0000 7f4607200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f460815fcf0 0x7f46080d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.050+0000 7f4607200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4608160420 0x7f4608160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.050+0000 7f4607200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f46080d06d0 0x7f46080d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.050+0000 7f4607200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4608160dc0 con 0x7f46080d06d0 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.050+0000 7f4607200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46080d3540 con 0x7f4608160420 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.050+0000 7f4607200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46080ce510 con 0x7f460815fcf0 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.051+0000 7f4606800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f46080d06d0 0x7f46080d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.051+0000 7f4606800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f46080d06d0 0x7f46080d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52336/0 (socket says 172.21.15.62:52336) 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.051+0000 7f4606800640 1 -- 172.21.15.62:0/3387156220 learned_addr learned my addr 172.21.15.62:0/3387156220 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.051+0000 7f4605400640 1 --2- 172.21.15.62:0/3387156220 >> v2:172.21.15.120:3300/0 conn(0x7f4608160420 0x7f4608160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.051+0000 7f4605e00640 1 --2- 172.21.15.62:0/3387156220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f460815fcf0 0x7f46080d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.051+0000 7f4606800640 1 -- 172.21.15.62:0/3387156220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f460815fcf0 msgr2=0x7f46080d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.051+0000 7f4606800640 1 --2- 172.21.15.62:0/3387156220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f460815fcf0 0x7f46080d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.051+0000 7f4606800640 1 -- 172.21.15.62:0/3387156220 >> v2:172.21.15.120:3300/0 conn(0x7f4608160420 msgr2=0x7f4608160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.051+0000 7f4606800640 1 --2- 172.21.15.62:0/3387156220 >> v2:172.21.15.120:3300/0 conn(0x7f4608160420 0x7f4608160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.051+0000 7f4606800640 1 -- 172.21.15.62:0/3387156220 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46080cdc70 con 0x7f46080d06d0 2024-03-23T11:19:09.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.052+0000 7f4606800640 1 --2- 172.21.15.62:0/3387156220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46080d06d0 0x7f46080d0b20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f46080664e0 tx=0x7f460002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca7b74ffb4b48f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:09.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.053+0000 7f4604a00640 1 -- 172.21.15.62:0/3387156220 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460002c9e0 con 0x7f46080d06d0 2024-03-23T11:19:09.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.053+0000 7f4604a00640 1 -- 172.21.15.62:0/3387156220 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460002cb40 con 0x7f46080d06d0 2024-03-23T11:19:09.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.053+0000 7f4604a00640 1 -- 172.21.15.62:0/3387156220 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46000343f0 con 0x7f46080d06d0 2024-03-23T11:19:09.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.053+0000 7f4607200640 1 -- 172.21.15.62:0/3387156220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46080d06d0 msgr2=0x7f46080d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.053+0000 7f4607200640 1 --2- 172.21.15.62:0/3387156220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46080d06d0 0x7f46080d0b20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f46080664e0 tx=0x7f460002be40 comp rx=0 tx=0).stop 2024-03-23T11:19:09.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.054+0000 7f4607200640 1 -- 172.21.15.62:0/3387156220 shutdown_connections 2024-03-23T11:19:09.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.054+0000 7f4607200640 1 --2- 172.21.15.62:0/3387156220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46080d06d0 0x7f46080d0b20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.054+0000 7f4607200640 1 --2- 172.21.15.62:0/3387156220 >> v2:172.21.15.120:3300/0 conn(0x7f4608160420 0x7f4608160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.054+0000 7f4607200640 1 --2- 172.21.15.62:0/3387156220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f460815fcf0 0x7f46080d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.054+0000 7f4607200640 1 -- 172.21.15.62:0/3387156220 >> 172.21.15.62:0/3387156220 conn(0x7f4608089f60 msgr2=0x7f46080d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:09.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.054+0000 7f4607200640 1 -- 172.21.15.62:0/3387156220 shutdown_connections 2024-03-23T11:19:09.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.054+0000 7f4607200640 1 -- 172.21.15.62:0/3387156220 wait complete. 2024-03-23T11:19:09.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.055+0000 7f4607200640 1 Processor -- start 2024-03-23T11:19:09.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.056+0000 7f4607200640 1 -- start start 2024-03-23T11:19:09.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.056+0000 7f4607200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46080d06d0 0x7f46081f3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.056+0000 7f4607200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f460815fcf0 0x7f46081f2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4607200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4608160420 0x7f46081f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4607200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46080c9f80 con 0x7f460815fcf0 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4607200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46080a04a0 con 0x7f46080d06d0 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4607200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46081f3110 con 0x7f4608160420 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4605400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f460815fcf0 0x7f46081f2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4605400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f460815fcf0 0x7f46081f2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52350/0 (socket says 172.21.15.62:52350) 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4605400640 1 -- 172.21.15.62:0/3969249211 learned_addr learned my addr 172.21.15.62:0/3969249211 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4605400640 1 -- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4608160420 msgr2=0x7f46081f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4605400640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4608160420 0x7f46081f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4605e00640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46080d06d0 0x7f46081f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4605400640 1 -- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46080d06d0 msgr2=0x7f46081f3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4605400640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46080d06d0 0x7f46081f3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4605400640 1 -- 172.21.15.62:0/3969249211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4600009b30 con 0x7f460815fcf0 2024-03-23T11:19:09.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.057+0000 7f4605400640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f460815fcf0 0x7f46081f2bd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f45f400e9b0 tx=0x7f45f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:09.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.058+0000 7f45fe000640 1 -- 172.21.15.62:0/3969249211 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f4009670 con 0x7f460815fcf0 2024-03-23T11:19:09.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.058+0000 7f45fe000640 1 -- 172.21.15.62:0/3969249211 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f400c9a0 con 0x7f460815fcf0 2024-03-23T11:19:09.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.058+0000 7f4607200640 1 -- 172.21.15.62:0/3969249211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46080cc380 con 0x7f460815fcf0 2024-03-23T11:19:09.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.058+0000 7f45fe000640 1 -- 172.21.15.62:0/3969249211 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f400cd50 con 0x7f460815fcf0 2024-03-23T11:19:09.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.058+0000 7f4607200640 1 -- 172.21.15.62:0/3969249211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46080a01c0 con 0x7f460815fcf0 2024-03-23T11:19:09.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.059+0000 7f4607200640 1 -- 172.21.15.62:0/3969249211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46080ca0c0 con 0x7f460815fcf0 2024-03-23T11:19:09.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.060+0000 7f45fe000640 1 -- 172.21.15.62:0/3969249211 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f45f4013470 con 0x7f460815fcf0 2024-03-23T11:19:09.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.061+0000 7f45fe000640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45e806a7a0 0x7f45e806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.061+0000 7f45fe000640 1 -- 172.21.15.62:0/3969249211 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45f40890d0 con 0x7f460815fcf0 2024-03-23T11:19:09.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.062+0000 7f4605e00640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45e806a7a0 0x7f45e806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.063+0000 7f4605e00640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45e806a7a0 0x7f45e806cc60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f45f8006fd0 tx=0x7f45f8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:09.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.065+0000 7f45fe000640 1 -- 172.21.15.62:0/3969249211 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f46080ca0c0 con 0x7f460815fcf0 2024-03-23T11:19:09.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.262+0000 7f4607200640 1 -- 172.21.15.62:0/3969249211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f46080ca0c0 con 0x7f460815fcf0 2024-03-23T11:19:09.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.263+0000 7f45fe000640 1 -- 172.21.15.62:0/3969249211 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f46080ca0c0 con 0x7f460815fcf0 2024-03-23T11:19:09.263 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:09.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 -- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45e806a7a0 msgr2=0x7f45e806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45e806a7a0 0x7f45e806cc60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f45f8006fd0 tx=0x7f45f8008040 comp rx=0 tx=0).stop 2024-03-23T11:19:09.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 -- 172.21.15.62:0/3969249211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f460815fcf0 msgr2=0x7f46081f2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f460815fcf0 0x7f46081f2bd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f45f400e9b0 tx=0x7f45f400ee70 comp rx=0 tx=0).stop 2024-03-23T11:19:09.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 -- 172.21.15.62:0/3969249211 shutdown_connections 2024-03-23T11:19:09.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45e806a7a0 0x7f45e806cc60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f460815fcf0 0x7f46081f2bd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4608160420 0x7f46081f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 --2- 172.21.15.62:0/3969249211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46080d06d0 0x7f46081f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 -- 172.21.15.62:0/3969249211 >> 172.21.15.62:0/3969249211 conn(0x7f4608089f60 msgr2=0x7f46080b7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:09.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 -- 172.21.15.62:0/3969249211 shutdown_connections 2024-03-23T11:19:09.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.266+0000 7f4607200640 1 -- 172.21.15.62:0/3969249211 wait complete. 2024-03-23T11:19:09.287 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:09.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.508+0000 7fba3d600640 1 Processor -- start 2024-03-23T11:19:09.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.509+0000 7fba3d600640 1 -- start start 2024-03-23T11:19:09.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.510+0000 7fba3d600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fba38070090 0x7fba38070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.510+0000 7fba3d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 0x7fba380b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.510+0000 7fba3d600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba380b31b0 con 0x7fba38143dd0 2024-03-23T11:19:09.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.510+0000 7fba3d600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba38070a60 con 0x7fba38070090 2024-03-23T11:19:09.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.510+0000 7fba3d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba380cd140 con 0x7fba380b2820 2024-03-23T11:19:09.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.511+0000 7fba3cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 0x7fba380b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.511+0000 7fba37400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fba38070090 0x7fba38070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.511+0000 7fba3cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 0x7fba380b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:55710/0 (socket says 172.21.15.62:55710) 2024-03-23T11:19:09.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.511+0000 7fba37400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fba38070090 0x7fba38070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53674/0 (socket says 172.21.15.62:53674) 2024-03-23T11:19:09.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.511+0000 7fba3cc00640 1 -- 172.21.15.62:0/1670013152 learned_addr learned my addr 172.21.15.62:0/1670013152 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:09.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.511+0000 7fba3cc00640 1 -- 172.21.15.62:0/1670013152 >> v2:172.21.15.120:3300/0 conn(0x7fba38070090 msgr2=0x7fba38070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:09.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.511+0000 7fba3cc00640 1 --2- 172.21.15.62:0/1670013152 >> v2:172.21.15.120:3300/0 conn(0x7fba38070090 0x7fba38070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.511+0000 7fba3cc00640 1 -- 172.21.15.62:0/1670013152 >> v1:172.21.15.120:6789/0 conn(0x7fba38143dd0 legacy=0x7fba3815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.511+0000 7fba3cc00640 1 -- 172.21.15.62:0/1670013152 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba380cd3e0 con 0x7fba380b2820 2024-03-23T11:19:09.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.511+0000 7fba37400640 1 --2- 172.21.15.62:0/1670013152 >> v2:172.21.15.120:3300/0 conn(0x7fba38070090 0x7fba38070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:09.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.513+0000 7fba3cc00640 1 --2- 172.21.15.62:0/1670013152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 0x7fba380b2c70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba30009e50 tx=0x7fba3002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52eaed122f3a18da server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:09.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.513+0000 7fba36a00640 1 -- 172.21.15.62:0/1670013152 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3002cb00 con 0x7fba380b2820 2024-03-23T11:19:09.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.513+0000 7fba36a00640 1 -- 172.21.15.62:0/1670013152 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3002cc60 con 0x7fba380b2820 2024-03-23T11:19:09.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.513+0000 7fba36a00640 1 -- 172.21.15.62:0/1670013152 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba300343f0 con 0x7fba380b2820 2024-03-23T11:19:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.514+0000 7fba3d600640 1 -- 172.21.15.62:0/1670013152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 msgr2=0x7fba380b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.514+0000 7fba3d600640 1 --2- 172.21.15.62:0/1670013152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 0x7fba380b2c70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba30009e50 tx=0x7fba3002bf60 comp rx=0 tx=0).stop 2024-03-23T11:19:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.514+0000 7fba3d600640 1 -- 172.21.15.62:0/1670013152 shutdown_connections 2024-03-23T11:19:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.514+0000 7fba3d600640 1 --2- 172.21.15.62:0/1670013152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 0x7fba380b2c70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.514+0000 7fba3d600640 1 --2- 172.21.15.62:0/1670013152 >> v2:172.21.15.120:3300/0 conn(0x7fba38070090 0x7fba38070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.514+0000 7fba3d600640 1 -- 172.21.15.62:0/1670013152 >> 172.21.15.62:0/1670013152 conn(0x7fba38066050 msgr2=0x7fba38069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:09.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.514+0000 7fba3d600640 1 -- 172.21.15.62:0/1670013152 shutdown_connections 2024-03-23T11:19:09.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.514+0000 7fba3d600640 1 -- 172.21.15.62:0/1670013152 wait complete. 2024-03-23T11:19:09.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.516+0000 7fba3d600640 1 Processor -- start 2024-03-23T11:19:09.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.516+0000 7fba3d600640 1 -- start start 2024-03-23T11:19:09.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.517+0000 7fba3d600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba38070090 0x7fba380a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.517+0000 7fba3d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 0x7fba3813adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.517+0000 7fba3d600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba38143dd0 0x7fba38142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.517+0000 7fba3d600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba380a9330 con 0x7fba38143dd0 2024-03-23T11:19:09.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.517+0000 7fba3d600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba380a8720 con 0x7fba38070090 2024-03-23T11:19:09.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.517+0000 7fba3d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba380a8a00 con 0x7fba380b2820 2024-03-23T11:19:09.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.517+0000 7fba3cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba38143dd0 0x7fba38142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.517+0000 7fba3cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba38143dd0 0x7fba38142020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52352/0 (socket says 172.21.15.62:52352) 2024-03-23T11:19:09.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.517+0000 7fba3cc00640 1 -- 172.21.15.62:0/2643229778 learned_addr learned my addr 172.21.15.62:0/2643229778 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:09.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.517+0000 7fba37400640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 0x7fba3813adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.517+0000 7fba37e00640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba38070090 0x7fba380a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.518+0000 7fba3cc00640 1 -- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 msgr2=0x7fba3813adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.518+0000 7fba3cc00640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 0x7fba3813adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.518+0000 7fba3cc00640 1 -- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba38070090 msgr2=0x7fba380a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.518+0000 7fba3cc00640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba38070090 0x7fba380a7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.518+0000 7fba3cc00640 1 -- 172.21.15.62:0/2643229778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20009400 con 0x7fba38143dd0 2024-03-23T11:19:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.518+0000 7fba37e00640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba38070090 0x7fba380a7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.518+0000 7fba3cc00640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba38143dd0 0x7fba38142020 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fba30009e50 tx=0x7fba30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.518+0000 7fba2fe00640 1 -- 172.21.15.62:0/2643229778 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba30009510 con 0x7fba38143dd0 2024-03-23T11:19:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.518+0000 7fba2fe00640 1 -- 172.21.15.62:0/2643229778 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3003e450 con 0x7fba38143dd0 2024-03-23T11:19:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.518+0000 7fba3d600640 1 -- 172.21.15.62:0/2643229778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba30009b30 con 0x7fba38143dd0 2024-03-23T11:19:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.518+0000 7fba2fe00640 1 -- 172.21.15.62:0/2643229778 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3003e800 con 0x7fba38143dd0 2024-03-23T11:19:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.519+0000 7fba3d600640 1 -- 172.21.15.62:0/2643229778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba380a8fc0 con 0x7fba38143dd0 2024-03-23T11:19:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.520+0000 7fba3d600640 1 -- 172.21.15.62:0/2643229778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba380a94e0 con 0x7fba38143dd0 2024-03-23T11:19:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.521+0000 7fba2fe00640 1 -- 172.21.15.62:0/2643229778 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fba3003e960 con 0x7fba38143dd0 2024-03-23T11:19:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.521+0000 7fba2fe00640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba1406a7a0 0x7fba1406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.522+0000 7fba2fe00640 1 -- 172.21.15.62:0/2643229778 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba300abba0 con 0x7fba38143dd0 2024-03-23T11:19:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.522+0000 7fba37e00640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba1406a7a0 0x7fba1406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.523+0000 7fba37e00640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba1406a7a0 0x7fba1406cc60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fba28007fd0 tx=0x7fba28009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.526+0000 7fba2fe00640 1 -- 172.21.15.62:0/2643229778 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba380a94e0 con 0x7fba38143dd0 2024-03-23T11:19:09.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.725+0000 7fba3d600640 1 -- 172.21.15.62:0/2643229778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fba38066bb0 con 0x7fba38143dd0 2024-03-23T11:19:09.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.726+0000 7fba2fe00640 1 -- 172.21.15.62:0/2643229778 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fba30076ea0 con 0x7fba38143dd0 2024-03-23T11:19:09.727 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:09.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 -- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba1406a7a0 msgr2=0x7fba1406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba1406a7a0 0x7fba1406cc60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fba28007fd0 tx=0x7fba28009040 comp rx=0 tx=0).stop 2024-03-23T11:19:09.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 -- 172.21.15.62:0/2643229778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba38143dd0 msgr2=0x7fba38142020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba38143dd0 0x7fba38142020 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fba30009e50 tx=0x7fba30004a40 comp rx=0 tx=0).stop 2024-03-23T11:19:09.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 -- 172.21.15.62:0/2643229778 shutdown_connections 2024-03-23T11:19:09.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba1406a7a0 0x7fba1406cc60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba38143dd0 0x7fba38142020 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba380b2820 0x7fba3813adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 --2- 172.21.15.62:0/2643229778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba38070090 0x7fba380a7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 -- 172.21.15.62:0/2643229778 >> 172.21.15.62:0/2643229778 conn(0x7fba38066050 msgr2=0x7fba380aee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:09.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 -- 172.21.15.62:0/2643229778 shutdown_connections 2024-03-23T11:19:09.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.729+0000 7fba3d600640 1 -- 172.21.15.62:0/2643229778 wait complete. 2024-03-23T11:19:09.749 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:19:09.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.969+0000 7f2ee4400640 1 Processor -- start 2024-03-23T11:19:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.969+0000 7f2ee4400640 1 -- start start 2024-03-23T11:19:09.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.970+0000 7f2ee4400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2edc1613a0 0x7f2edc06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.970+0000 7f2ee4400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2edc06d910 0x7f2edc15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.970+0000 7f2ee4400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc06cf40 con 0x7f2edc1571e0 2024-03-23T11:19:09.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.970+0000 7f2ee4400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2edc1575c0 con 0x7f2edc06d910 2024-03-23T11:19:09.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.970+0000 7f2ee4400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc06c3a0 con 0x7f2edc1613a0 2024-03-23T11:19:09.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.971+0000 7f2ee1a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2edc1571e0 0x7f2edc0ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35998/0 (socket says 172.21.15.62:35998) 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.971+0000 7f2ee1a00640 1 -- 172.21.15.62:0/1544676011 learned_addr learned my addr 172.21.15.62:0/1544676011 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.971+0000 7f2ee1000640 1 --2- 172.21.15.62:0/1544676011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2edc1613a0 0x7f2edc06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.971+0000 7f2ee2400640 1 --2- 172.21.15.62:0/1544676011 >> v2:172.21.15.120:3300/0 conn(0x7f2edc06d910 0x7f2edc15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.971+0000 7f2edbe00640 1 -- 172.21.15.62:0/1544676011 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508569108 0 0) 0x7f2edc06cf40 con 0x7f2edc1571e0 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.971+0000 7f2edbe00640 1 -- 172.21.15.62:0/1544676011 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec4003520 con 0x7f2edc1571e0 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.972+0000 7f2edbe00640 1 -- 172.21.15.62:0/1544676011 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83421105 0 0) 0x7f2ec4003520 con 0x7f2edc1571e0 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.972+0000 7f2ee1000640 1 -- 172.21.15.62:0/1544676011 >> v2:172.21.15.120:3300/0 conn(0x7f2edc06d910 msgr2=0x7f2edc15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.972+0000 7f2ee1000640 1 --2- 172.21.15.62:0/1544676011 >> v2:172.21.15.120:3300/0 conn(0x7f2edc06d910 0x7f2edc15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.972+0000 7f2ee1000640 1 -- 172.21.15.62:0/1544676011 >> v1:172.21.15.62:6789/0 conn(0x7f2edc1571e0 legacy=0x7f2edc0ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.972+0000 7f2ee1000640 1 -- 172.21.15.62:0/1544676011 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc0ae040 con 0x7f2edc1613a0 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.972+0000 7f2ee2400640 1 --2- 172.21.15.62:0/1544676011 >> v2:172.21.15.120:3300/0 conn(0x7f2edc06d910 0x7f2edc15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.973+0000 7f2ee1000640 1 --2- 172.21.15.62:0/1544676011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2edc1613a0 0x7f2edc06d3d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80097a0 tx=0x7f2ec802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3cb389f76bac238 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.974+0000 7f2edbe00640 1 -- 172.21.15.62:0/1544676011 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0001670 con 0x7f2edc1613a0 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.974+0000 7f2edbe00640 1 -- 172.21.15.62:0/1544676011 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec802e050 con 0x7f2edc1613a0 2024-03-23T11:19:09.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.974+0000 7f2edbe00640 1 -- 172.21.15.62:0/1544676011 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8030710 con 0x7f2edc1613a0 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.974+0000 7f2ee4400640 1 -- 172.21.15.62:0/1544676011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2edc1613a0 msgr2=0x7f2edc06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.974+0000 7f2ee4400640 1 --2- 172.21.15.62:0/1544676011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2edc1613a0 0x7f2edc06d3d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80097a0 tx=0x7f2ec802b710 comp rx=0 tx=0).stop 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.974+0000 7f2ee4400640 1 -- 172.21.15.62:0/1544676011 shutdown_connections 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.974+0000 7f2ee4400640 1 --2- 172.21.15.62:0/1544676011 >> v2:172.21.15.120:3300/0 conn(0x7f2edc06d910 0x7f2edc15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.974+0000 7f2ee4400640 1 --2- 172.21.15.62:0/1544676011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2edc1613a0 0x7f2edc06d3d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.974+0000 7f2ee4400640 1 -- 172.21.15.62:0/1544676011 >> 172.21.15.62:0/1544676011 conn(0x7f2edc0b02e0 msgr2=0x7f2edc0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.974+0000 7f2ee4400640 1 -- 172.21.15.62:0/1544676011 shutdown_connections 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.974+0000 7f2ee4400640 1 -- 172.21.15.62:0/1544676011 wait complete. 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.977+0000 7f2ee4400640 1 Processor -- start 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.977+0000 7f2ee4400640 1 -- start start 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.978+0000 7f2ee4400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2edc06d910 0x7f2edc14f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.978+0000 7f2ee4400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2edc1571e0 0x7f2edc1517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.978+0000 7f2ee4400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2edc1613a0 0x7f2edc151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.978+0000 7f2ee4400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc153410 con 0x7f2edc06d910 2024-03-23T11:19:09.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.978+0000 7f2ee4400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc152920 con 0x7f2edc1571e0 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.978+0000 7f2ee4400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc1530e0 con 0x7f2edc1613a0 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.978+0000 7f2ee1a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2edc06d910 0x7f2edc14f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.978+0000 7f2ee1a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2edc06d910 0x7f2edc14f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52360/0 (socket says 172.21.15.62:52360) 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.978+0000 7f2ee1a00640 1 -- 172.21.15.62:0/907398747 learned_addr learned my addr 172.21.15.62:0/907398747 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.979+0000 7f2ee1000640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2edc1571e0 0x7f2edc1517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.979+0000 7f2ee2400640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2edc1613a0 0x7f2edc151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.979+0000 7f2ee1a00640 1 -- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2edc1613a0 msgr2=0x7f2edc151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.979+0000 7f2ee1a00640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2edc1613a0 0x7f2edc151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.979+0000 7f2ee1a00640 1 -- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2edc1571e0 msgr2=0x7f2edc1517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.979+0000 7f2ee1a00640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2edc1571e0 0x7f2edc1517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.979+0000 7f2ee1a00640 1 -- 172.21.15.62:0/907398747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed4009a70 con 0x7f2edc06d910 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.979+0000 7f2ee1000640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2edc1571e0 0x7f2edc1517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.979+0000 7f2ee1a00640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2edc06d910 0x7f2edc14f820 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000c750 tx=0x7f2ed000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.979+0000 7f2ed9600640 1 -- 172.21.15.62:0/907398747 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0005ad0 con 0x7f2edc06d910 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.980+0000 7f2ed9600640 1 -- 172.21.15.62:0/907398747 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed0005c30 con 0x7f2edc06d910 2024-03-23T11:19:09.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.980+0000 7f2ed9600640 1 -- 172.21.15.62:0/907398747 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0007070 con 0x7f2edc06d910 2024-03-23T11:19:09.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.980+0000 7f2ee4400640 1 -- 172.21.15.62:0/907398747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec8009400 con 0x7f2edc06d910 2024-03-23T11:19:09.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.980+0000 7f2ee4400640 1 -- 172.21.15.62:0/907398747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc152640 con 0x7f2edc06d910 2024-03-23T11:19:09.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.981+0000 7f2ee4400640 1 -- 172.21.15.62:0/907398747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2edc0aeb70 con 0x7f2edc06d910 2024-03-23T11:19:09.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.982+0000 7f2ed9600640 1 -- 172.21.15.62:0/907398747 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2ed00071d0 con 0x7f2edc06d910 2024-03-23T11:19:09.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.983+0000 7f2ed9600640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ebc06a7a0 0x7f2ebc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:09.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.983+0000 7f2ed9600640 1 -- 172.21.15.62:0/907398747 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ed0088560 con 0x7f2edc06d910 2024-03-23T11:19:09.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.984+0000 7f2ee1000640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ebc06a7a0 0x7f2ebc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:09.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.985+0000 7f2ee1000640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ebc06a7a0 0x7f2ebc06cc60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8034000 tx=0x7f2ec8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:09.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:09.988+0000 7f2ed9600640 1 -- 172.21.15.62:0/907398747 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ed0053970 con 0x7f2edc06d910 2024-03-23T11:19:10.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.225+0000 7f2ee4400640 1 -- 172.21.15.62:0/907398747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2edc066c90 con 0x7f2edc06d910 2024-03-23T11:19:10.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.226+0000 7f2ed9600640 1 -- 172.21.15.62:0/907398747 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f2ed0056f90 con 0x7f2edc06d910 2024-03-23T11:19:10.227 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:10.227 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/496399527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":496399527},{"type":"v1","addr":"172.21.15.62:6833","nonce":496399527}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/156755611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":156755611},{"type":"v1","addr":"172.21.15.120:6835","nonce":156755611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2391380456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2391380456},{"type":"v1","addr":"172.21.15.62:6835","nonce":2391380456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:18:56.233595+0000","modified":"2024-03-23T11:18:59.758284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/3936303103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3936303103},{"type":"v1","addr":"172.21.15.120:6837","nonce":3936303103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T11:19:10.227 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-03-23T11:19:10.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 -- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ebc06a7a0 msgr2=0x7f2ebc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ebc06a7a0 0x7f2ebc06cc60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8034000 tx=0x7f2ec8008030 comp rx=0 tx=0).stop 2024-03-23T11:19:10.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 -- 172.21.15.62:0/907398747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2edc06d910 msgr2=0x7f2edc14f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2edc06d910 0x7f2edc14f820 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000c750 tx=0x7f2ed000cc10 comp rx=0 tx=0).stop 2024-03-23T11:19:10.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 -- 172.21.15.62:0/907398747 shutdown_connections 2024-03-23T11:19:10.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ebc06a7a0 0x7f2ebc06cc60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2edc1613a0 0x7f2edc151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2edc1571e0 0x7f2edc1517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 --2- 172.21.15.62:0/907398747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2edc06d910 0x7f2edc14f820 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 -- 172.21.15.62:0/907398747 >> 172.21.15.62:0/907398747 conn(0x7f2edc0b02e0 msgr2=0x7f2edc0af170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:10.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 -- 172.21.15.62:0/907398747 shutdown_connections 2024-03-23T11:19:10.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.229+0000 7f2ee4400640 1 -- 172.21.15.62:0/907398747 wait complete. 2024-03-23T11:19:10.248 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.467+0000 7f7819800640 1 Processor -- start 2024-03-23T11:19:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.468+0000 7f7819800640 1 -- start start 2024-03-23T11:19:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.469+0000 7f7819800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7814160420 0x7f78141608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.469+0000 7f7819800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140ce510 con 0x7f78140b5db0 2024-03-23T11:19:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.469+0000 7f7819800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140cdc70 con 0x7f781415fcf0 2024-03-23T11:19:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.469+0000 7f7819800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7814160df0 con 0x7f7814160420 2024-03-23T11:19:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.470+0000 7f7813400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7814160420 0x7f78141608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.470+0000 7f7818e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f78140b5db0 0x7f78140b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36006/0 (socket says 172.21.15.62:36006) 2024-03-23T11:19:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.470+0000 7f7818e00640 1 -- 172.21.15.62:0/1377963754 learned_addr learned my addr 172.21.15.62:0/1377963754 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.470+0000 7f7812a00640 1 -- 172.21.15.62:0/1377963754 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211733980 0 0) 0x7f78140ce510 con 0x7f78140b5db0 2024-03-23T11:19:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.470+0000 7f7812a00640 1 -- 172.21.15.62:0/1377963754 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f4003520 con 0x7f78140b5db0 2024-03-23T11:19:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.470+0000 7f7813400640 1 -- 172.21.15.62:0/1377963754 >> v1:172.21.15.120:6789/0 conn(0x7f781415fcf0 legacy=0x7f78140d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.470+0000 7f7813400640 1 -- 172.21.15.62:0/1377963754 >> v1:172.21.15.62:6789/0 conn(0x7f78140b5db0 legacy=0x7f78140b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.470+0000 7f7813400640 1 -- 172.21.15.62:0/1377963754 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78140cdf50 con 0x7f7814160420 2024-03-23T11:19:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.472+0000 7f7813400640 1 --2- 172.21.15.62:0/1377963754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7814160420 0x7f78141608b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009d90 tx=0x7f77fc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b999de70e350eb3c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.473+0000 7f7812a00640 1 -- 172.21.15.62:0/1377963754 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc02cc60 con 0x7f7814160420 2024-03-23T11:19:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.473+0000 7f7812a00640 1 -- 172.21.15.62:0/1377963754 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77fc02cdc0 con 0x7f7814160420 2024-03-23T11:19:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.473+0000 7f7812a00640 1 -- 172.21.15.62:0/1377963754 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc034640 con 0x7f7814160420 2024-03-23T11:19:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.473+0000 7f7819800640 1 -- 172.21.15.62:0/1377963754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7814160420 msgr2=0x7f78141608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.473+0000 7f7819800640 1 --2- 172.21.15.62:0/1377963754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7814160420 0x7f78141608b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009d90 tx=0x7f77fc02beb0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.474+0000 7f7819800640 1 -- 172.21.15.62:0/1377963754 shutdown_connections 2024-03-23T11:19:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.474+0000 7f7819800640 1 --2- 172.21.15.62:0/1377963754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7814160420 0x7f78141608b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.474+0000 7f7819800640 1 -- 172.21.15.62:0/1377963754 >> 172.21.15.62:0/1377963754 conn(0x7f7814089f60 msgr2=0x7f78140d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.474+0000 7f7819800640 1 -- 172.21.15.62:0/1377963754 shutdown_connections 2024-03-23T11:19:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.474+0000 7f7819800640 1 -- 172.21.15.62:0/1377963754 wait complete. 2024-03-23T11:19:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.475+0000 7f7819800640 1 Processor -- start 2024-03-23T11:19:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.476+0000 7f7819800640 1 -- start start 2024-03-23T11:19:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.476+0000 7f7819800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78140b5db0 0x7f781414bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.476+0000 7f7819800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f781415fcf0 0x7f78140cf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.476+0000 7f7819800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7814160420 0x7f78140cf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.476+0000 7f7819800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78141e7ba0 con 0x7f78140b5db0 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.476+0000 7f7819800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781414c0e0 con 0x7f781415fcf0 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.476+0000 7f7819800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78141e7030 con 0x7f7814160420 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.477+0000 7f7813e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78140b5db0 0x7f781414bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.477+0000 7f7813e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78140b5db0 0x7f781414bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52374/0 (socket says 172.21.15.62:52374) 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.477+0000 7f7813e00640 1 -- 172.21.15.62:0/3347158604 learned_addr learned my addr 172.21.15.62:0/3347158604 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.477+0000 7f7813400640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f781415fcf0 0x7f78140cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.477+0000 7f7818e00640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7814160420 0x7f78140cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.477+0000 7f7813e00640 1 -- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7814160420 msgr2=0x7f78140cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.477+0000 7f7813e00640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7814160420 0x7f78140cf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.477+0000 7f7813e00640 1 -- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f781415fcf0 msgr2=0x7f78140cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.477+0000 7f7813e00640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f781415fcf0 0x7f78140cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.477+0000 7f7813e00640 1 -- 172.21.15.62:0/3347158604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc009a70 con 0x7f78140b5db0 2024-03-23T11:19:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.478+0000 7f7813e00640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78140b5db0 0x7f781414bba0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f780400c430 tx=0x7f780400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:10.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.478+0000 7f780be00640 1 -- 172.21.15.62:0/3347158604 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78040052d0 con 0x7f78140b5db0 2024-03-23T11:19:10.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.478+0000 7f780be00640 1 -- 172.21.15.62:0/3347158604 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7804005450 con 0x7f78140b5db0 2024-03-23T11:19:10.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.478+0000 7f780be00640 1 -- 172.21.15.62:0/3347158604 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804007260 con 0x7f78140b5db0 2024-03-23T11:19:10.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.478+0000 7f7819800640 1 -- 172.21.15.62:0/3347158604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78141e7df0 con 0x7f78140b5db0 2024-03-23T11:19:10.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.478+0000 7f7819800640 1 -- 172.21.15.62:0/3347158604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78141e83b0 con 0x7f78140b5db0 2024-03-23T11:19:10.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.479+0000 7f7819800640 1 -- 172.21.15.62:0/3347158604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7814066bb0 con 0x7f78140b5db0 2024-03-23T11:19:10.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.480+0000 7f780be00640 1 -- 172.21.15.62:0/3347158604 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f78040073c0 con 0x7f78140b5db0 2024-03-23T11:19:10.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.481+0000 7f780be00640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77f006a7a0 0x7f77f006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:10.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.481+0000 7f780be00640 1 -- 172.21.15.62:0/3347158604 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78040881f0 con 0x7f78140b5db0 2024-03-23T11:19:10.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.482+0000 7f7813400640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77f006a7a0 0x7f77f006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.483+0000 7f7813400640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77f006a7a0 0x7f77f006cc60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f77fc036000 tx=0x7f77fc030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:10.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.486+0000 7f780be00640 1 -- 172.21.15.62:0/3347158604 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7804053600 con 0x7f78140b5db0 2024-03-23T11:19:10.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.685+0000 7f7819800640 1 -- 172.21.15.62:0/3347158604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78140c94e0 con 0x7f78140b5db0 2024-03-23T11:19:10.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.687+0000 7f780be00640 1 -- 172.21.15.62:0/3347158604 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f7804056c20 con 0x7f78140b5db0 2024-03-23T11:19:10.687 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:10.687 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:57.238201+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:10.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.689+0000 7f7819800640 1 -- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77f006a7a0 msgr2=0x7f77f006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.689+0000 7f7819800640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77f006a7a0 0x7f77f006cc60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f77fc036000 tx=0x7f77fc030000 comp rx=0 tx=0).stop 2024-03-23T11:19:10.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.689+0000 7f7819800640 1 -- 172.21.15.62:0/3347158604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78140b5db0 msgr2=0x7f781414bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.689+0000 7f7819800640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78140b5db0 0x7f781414bba0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f780400c430 tx=0x7f780400c8f0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.689+0000 7f7819800640 1 -- 172.21.15.62:0/3347158604 shutdown_connections 2024-03-23T11:19:10.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.689+0000 7f7819800640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77f006a7a0 0x7f77f006cc60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.689+0000 7f7819800640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7814160420 0x7f78140cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.689+0000 7f7819800640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f781415fcf0 0x7f78140cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.689+0000 7f7819800640 1 --2- 172.21.15.62:0/3347158604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78140b5db0 0x7f781414bba0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.689+0000 7f7819800640 1 -- 172.21.15.62:0/3347158604 >> 172.21.15.62:0/3347158604 conn(0x7f7814089f60 msgr2=0x7f781406ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:10.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.690+0000 7f7819800640 1 -- 172.21.15.62:0/3347158604 shutdown_connections 2024-03-23T11:19:10.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.690+0000 7f7819800640 1 -- 172.21.15.62:0/3347158604 wait complete. 2024-03-23T11:19:10.708 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:10.708 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:10.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.928+0000 7f6647e00640 1 Processor -- start 2024-03-23T11:19:10.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.928+0000 7f6647e00640 1 -- start start 2024-03-23T11:19:10.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.929+0000 7f6647e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f664815ca00 0x7f66480ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:10.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.929+0000 7f6647e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66480cfe40 con 0x7f66480d0d40 2024-03-23T11:19:10.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.929+0000 7f6647e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66480cd680 con 0x7f66480b2bf0 2024-03-23T11:19:10.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.929+0000 7f6647e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66480cd960 con 0x7f664815ca00 2024-03-23T11:19:10.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.930+0000 7f6646000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f66480d0d40 0x7f66480b24e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36020/0 (socket says 172.21.15.62:36020) 2024-03-23T11:19:10.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.930+0000 7f6646000640 1 -- 172.21.15.62:0/3163683059 learned_addr learned my addr 172.21.15.62:0/3163683059 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:10.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.930+0000 7f6646a00640 1 --2- 172.21.15.62:0/3163683059 >> v2:172.21.15.120:3300/0 conn(0x7f664815ca00 0x7f66480ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:10.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.930+0000 7f6645600640 1 -- 172.21.15.62:0/3163683059 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292597159 0 0) 0x7f66480cfe40 con 0x7f66480d0d40 2024-03-23T11:19:10.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.930+0000 7f6645600640 1 -- 172.21.15.62:0/3163683059 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f662c003520 con 0x7f66480d0d40 2024-03-23T11:19:10.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.931+0000 7f6645600640 1 -- 172.21.15.62:0/3163683059 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 997389755 0 0) 0x7f662c003520 con 0x7f66480d0d40 2024-03-23T11:19:10.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.931+0000 7f6646a00640 1 -- 172.21.15.62:0/3163683059 >> v1:172.21.15.120:6789/0 conn(0x7f66480b2bf0 legacy=0x7f66480cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.931+0000 7f6646a00640 1 -- 172.21.15.62:0/3163683059 >> v1:172.21.15.62:6789/0 conn(0x7f66480d0d40 legacy=0x7f66480b24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.931+0000 7f6646a00640 1 -- 172.21.15.62:0/3163683059 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66480cdc40 con 0x7f664815ca00 2024-03-23T11:19:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.932+0000 7f6646a00640 1 --2- 172.21.15.62:0/3163683059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664815ca00 0x7f66480ce160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6630009790 tx=0x7f663002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b9a59bf968b9fe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.932+0000 7f6645600640 1 -- 172.21.15.62:0/3163683059 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663002f050 con 0x7f664815ca00 2024-03-23T11:19:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.932+0000 7f6645600640 1 -- 172.21.15.62:0/3163683059 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6630031500 con 0x7f664815ca00 2024-03-23T11:19:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.932+0000 7f6645600640 1 -- 172.21.15.62:0/3163683059 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380019f0 con 0x7f664815ca00 2024-03-23T11:19:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.933+0000 7f6647e00640 1 -- 172.21.15.62:0/3163683059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664815ca00 msgr2=0x7f66480ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.933+0000 7f6647e00640 1 --2- 172.21.15.62:0/3163683059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664815ca00 0x7f66480ce160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6630009790 tx=0x7f663002b820 comp rx=0 tx=0).stop 2024-03-23T11:19:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.933+0000 7f6647e00640 1 -- 172.21.15.62:0/3163683059 shutdown_connections 2024-03-23T11:19:10.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.933+0000 7f6647e00640 1 --2- 172.21.15.62:0/3163683059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664815ca00 0x7f66480ce160 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.933+0000 7f6647e00640 1 -- 172.21.15.62:0/3163683059 >> 172.21.15.62:0/3163683059 conn(0x7f664809ba10 msgr2=0x7f6648089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:10.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.933+0000 7f6647e00640 1 -- 172.21.15.62:0/3163683059 shutdown_connections 2024-03-23T11:19:10.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.933+0000 7f6647e00640 1 -- 172.21.15.62:0/3163683059 wait complete. 2024-03-23T11:19:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.935+0000 7f6647e00640 1 Processor -- start 2024-03-23T11:19:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.935+0000 7f6647e00640 1 -- start start 2024-03-23T11:19:10.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.936+0000 7f6647e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480b2bf0 0x7f664815f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:10.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.936+0000 7f6647e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66480d0d40 0x7f66481483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:10.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.936+0000 7f6647e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f664815ca00 0x7f6648148920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:10.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.936+0000 7f6647e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66481584c0 con 0x7f66480b2bf0 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.936+0000 7f6647e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66481587a0 con 0x7f66480d0d40 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.936+0000 7f6647e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6648158a80 con 0x7f664815ca00 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.936+0000 7f6646a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480b2bf0 0x7f664815f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.936+0000 7f6646a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480b2bf0 0x7f664815f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52378/0 (socket says 172.21.15.62:52378) 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.936+0000 7f6646a00640 1 -- 172.21.15.62:0/3880513202 learned_addr learned my addr 172.21.15.62:0/3880513202 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.937+0000 7f6646000640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66480d0d40 0x7f66481483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.937+0000 7f6647400640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f664815ca00 0x7f6648148920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.937+0000 7f6646a00640 1 -- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f664815ca00 msgr2=0x7f6648148920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.937+0000 7f6646a00640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f664815ca00 0x7f6648148920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.937+0000 7f6646a00640 1 -- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66480d0d40 msgr2=0x7f66481483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.937+0000 7f6646a00640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66480d0d40 0x7f66481483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.937+0000 7f6646a00640 1 -- 172.21.15.62:0/3880513202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66300093a0 con 0x7f66480b2bf0 2024-03-23T11:19:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.937+0000 7f6646000640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66480d0d40 0x7f66481483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.937+0000 7f6646a00640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480b2bf0 0x7f664815f8a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6630009790 tx=0x7f66300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.938+0000 7f663ea00640 1 -- 172.21.15.62:0/3880513202 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630005b90 con 0x7f66480b2bf0 2024-03-23T11:19:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.938+0000 7f663ea00640 1 -- 172.21.15.62:0/3880513202 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6630005d10 con 0x7f66480b2bf0 2024-03-23T11:19:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.938+0000 7f663ea00640 1 -- 172.21.15.62:0/3880513202 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630033980 con 0x7f66480b2bf0 2024-03-23T11:19:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.938+0000 7f6647e00640 1 -- 172.21.15.62:0/3880513202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6648158d60 con 0x7f66480b2bf0 2024-03-23T11:19:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.938+0000 7f6647e00640 1 -- 172.21.15.62:0/3880513202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6648157fc0 con 0x7f66480b2bf0 2024-03-23T11:19:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.939+0000 7f6647e00640 1 -- 172.21.15.62:0/3880513202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6648158fa0 con 0x7f66480b2bf0 2024-03-23T11:19:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.940+0000 7f663ea00640 1 -- 172.21.15.62:0/3880513202 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f663003a040 con 0x7f66480b2bf0 2024-03-23T11:19:10.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.941+0000 7f663ea00640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 0x7f662406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:10.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.941+0000 7f663ea00640 1 -- 172.21.15.62:0/3880513202 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66300aa450 con 0x7f66480b2bf0 2024-03-23T11:19:10.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.941+0000 7f6646000640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 0x7f662406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:10.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.942+0000 7f6646000640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 0x7f662406cc60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f66481494a0 tx=0x7f6638009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:10.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:10.945+0000 7f663ea00640 1 -- 172.21.15.62:0/3880513202 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f66300757d0 con 0x7f66480b2bf0 2024-03-23T11:19:11.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.143+0000 7f6647e00640 1 -- 172.21.15.62:0/3880513202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66480b6c00 con 0x7f66480b2bf0 2024-03-23T11:19:11.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.144+0000 7f663ea00640 1 -- 172.21.15.62:0/3880513202 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6630038020 con 0x7f66480b2bf0 2024-03-23T11:19:11.144 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 -- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 msgr2=0x7f662406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 0x7f662406cc60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f66481494a0 tx=0x7f6638009040 comp rx=0 tx=0).stop 2024-03-23T11:19:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 -- 172.21.15.62:0/3880513202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480b2bf0 msgr2=0x7f664815f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480b2bf0 0x7f664815f8a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6630009790 tx=0x7f66300046c0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 -- 172.21.15.62:0/3880513202 shutdown_connections 2024-03-23T11:19:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 0x7f662406cc60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f664815ca00 0x7f6648148920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66480d0d40 0x7f66481483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 --2- 172.21.15.62:0/3880513202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480b2bf0 0x7f664815f8a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 -- 172.21.15.62:0/3880513202 >> 172.21.15.62:0/3880513202 conn(0x7f664809ba10 msgr2=0x7f664815bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 -- 172.21.15.62:0/3880513202 shutdown_connections 2024-03-23T11:19:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.147+0000 7f6647e00640 1 -- 172.21.15.62:0/3880513202 wait complete. 2024-03-23T11:19:11.169 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:11.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.378+0000 7fb8a9a00640 1 Processor -- start 2024-03-23T11:19:11.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.378+0000 7fb8a9a00640 1 -- start start 2024-03-23T11:19:11.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.380+0000 7fb8a9a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8a4143dd0 0x7fb8a415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.380+0000 7fb8a9a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb8a4070090 0x7fb8a40704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.380+0000 7fb8a9a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a4070a30 con 0x7fb8a40b2820 2024-03-23T11:19:11.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.380+0000 7fb8a9a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a40cd140 con 0x7fb8a4070090 2024-03-23T11:19:11.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.380+0000 7fb8a9a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40cd3e0 con 0x7fb8a4143dd0 2024-03-23T11:19:11.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.381+0000 7fb8a9000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb8a40b2820 0x7fb8a40b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36026/0 (socket says 172.21.15.62:36026) 2024-03-23T11:19:11.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.381+0000 7fb8a9000640 1 -- 172.21.15.62:0/1548353184 learned_addr learned my addr 172.21.15.62:0/1548353184 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:11.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.381+0000 7fb8a3400640 1 --2- 172.21.15.62:0/1548353184 >> v2:172.21.15.120:3300/0 conn(0x7fb8a4070090 0x7fb8a40704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.381+0000 7fb8a3e00640 1 --2- 172.21.15.62:0/1548353184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8a4143dd0 0x7fb8a415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.381+0000 7fb8a2a00640 1 -- 172.21.15.62:0/1548353184 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950208335 0 0) 0x7fb8a4070a30 con 0x7fb8a40b2820 2024-03-23T11:19:11.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.381+0000 7fb8a2a00640 1 -- 172.21.15.62:0/1548353184 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb888003520 con 0x7fb8a40b2820 2024-03-23T11:19:11.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.382+0000 7fb8a2a00640 1 -- 172.21.15.62:0/1548353184 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3104741180 0 0) 0x7fb888003520 con 0x7fb8a40b2820 2024-03-23T11:19:11.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.382+0000 7fb8a3400640 1 -- 172.21.15.62:0/1548353184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8a4143dd0 msgr2=0x7fb8a415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:11.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.382+0000 7fb8a3400640 1 --2- 172.21.15.62:0/1548353184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8a4143dd0 0x7fb8a415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.382+0000 7fb8a3400640 1 -- 172.21.15.62:0/1548353184 >> v1:172.21.15.62:6789/0 conn(0x7fb8a40b2820 legacy=0x7fb8a40b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.382+0000 7fb8a3400640 1 -- 172.21.15.62:0/1548353184 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a40cd6c0 con 0x7fb8a4070090 2024-03-23T11:19:11.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.383+0000 7fb8a3400640 1 --2- 172.21.15.62:0/1548353184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8a4070090 0x7fb8a40704f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0097a0 tx=0x7fb88c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=785b3e90fb660c9d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:11.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.383+0000 7fb8a2a00640 1 -- 172.21.15.62:0/1548353184 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c0019f0 con 0x7fb8a4070090 2024-03-23T11:19:11.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.383+0000 7fb8a2a00640 1 -- 172.21.15.62:0/1548353184 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c02e050 con 0x7fb8a4070090 2024-03-23T11:19:11.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.383+0000 7fb8a2a00640 1 -- 172.21.15.62:0/1548353184 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c030730 con 0x7fb8a4070090 2024-03-23T11:19:11.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.384+0000 7fb8a9a00640 1 -- 172.21.15.62:0/1548353184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8a4070090 msgr2=0x7fb8a40704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.384+0000 7fb8a9a00640 1 --2- 172.21.15.62:0/1548353184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8a4070090 0x7fb8a40704f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0097a0 tx=0x7fb88c02b790 comp rx=0 tx=0).stop 2024-03-23T11:19:11.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.384+0000 7fb8a9a00640 1 -- 172.21.15.62:0/1548353184 shutdown_connections 2024-03-23T11:19:11.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.384+0000 7fb8a9a00640 1 --2- 172.21.15.62:0/1548353184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8a4070090 0x7fb8a40704f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.384+0000 7fb8a9a00640 1 --2- 172.21.15.62:0/1548353184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8a4143dd0 0x7fb8a415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.384+0000 7fb8a9a00640 1 -- 172.21.15.62:0/1548353184 >> 172.21.15.62:0/1548353184 conn(0x7fb8a4066050 msgr2=0x7fb8a4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:11.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.384+0000 7fb8a9a00640 1 -- 172.21.15.62:0/1548353184 shutdown_connections 2024-03-23T11:19:11.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.384+0000 7fb8a9a00640 1 -- 172.21.15.62:0/1548353184 wait complete. 2024-03-23T11:19:11.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.386+0000 7fb8a9a00640 1 Processor -- start 2024-03-23T11:19:11.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.386+0000 7fb8a9a00640 1 -- start start 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.387+0000 7fb8a9a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8a4070090 0x7fb8a4159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.387+0000 7fb8a9a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8a40b2820 0x7fb8a40affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.387+0000 7fb8a9a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8a4143dd0 0x7fb8a40b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.387+0000 7fb8a9a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a41427e0 con 0x7fb8a4143dd0 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.387+0000 7fb8a9a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40ac650 con 0x7fb8a4070090 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.387+0000 7fb8a9a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a4141f40 con 0x7fb8a40b2820 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb8a9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8a4143dd0 0x7fb8a40b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb8a9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8a4143dd0 0x7fb8a40b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52394/0 (socket says 172.21.15.62:52394) 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb8a9000640 1 -- 172.21.15.62:0/2054744412 learned_addr learned my addr 172.21.15.62:0/2054744412 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb8a3400640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8a40b2820 0x7fb8a40affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb8a3e00640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8a4070090 0x7fb8a4159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb8a9000640 1 -- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8a40b2820 msgr2=0x7fb8a40affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb8a9000640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8a40b2820 0x7fb8a40affb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb8a9000640 1 -- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8a4070090 msgr2=0x7fb8a4159b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb8a9000640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8a4070090 0x7fb8a4159b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb8a9000640 1 -- 172.21.15.62:0/2054744412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c009400 con 0x7fb8a4143dd0 2024-03-23T11:19:11.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb8a9000640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8a4143dd0 0x7fb8a40b04f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00cc90 tx=0x7fb89c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:11.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.388+0000 7fb89be00640 1 -- 172.21.15.62:0/2054744412 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c00cea0 con 0x7fb8a4143dd0 2024-03-23T11:19:11.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.389+0000 7fb89be00640 1 -- 172.21.15.62:0/2054744412 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c012400 con 0x7fb8a4143dd0 2024-03-23T11:19:11.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.389+0000 7fb89be00640 1 -- 172.21.15.62:0/2054744412 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c0127b0 con 0x7fb8a4143dd0 2024-03-23T11:19:11.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.389+0000 7fb8a9a00640 1 -- 172.21.15.62:0/2054744412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a4142220 con 0x7fb8a4143dd0 2024-03-23T11:19:11.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.389+0000 7fb8a9a00640 1 -- 172.21.15.62:0/2054744412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a413c030 con 0x7fb8a4143dd0 2024-03-23T11:19:11.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.390+0000 7fb8a9a00640 1 -- 172.21.15.62:0/2054744412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a413be90 con 0x7fb8a4143dd0 2024-03-23T11:19:11.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.391+0000 7fb89be00640 1 -- 172.21.15.62:0/2054744412 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb89c007070 con 0x7fb8a4143dd0 2024-03-23T11:19:11.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.392+0000 7fb89be00640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb88006a7a0 0x7fb88006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.392+0000 7fb89be00640 1 -- 172.21.15.62:0/2054744412 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb89c088e90 con 0x7fb8a4143dd0 2024-03-23T11:19:11.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.392+0000 7fb8a3e00640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb88006a7a0 0x7fb88006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.393+0000 7fb8a3e00640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb88006a7a0 0x7fb88006cc60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb894009a30 tx=0x7fb894008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:11.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.396+0000 7fb89be00640 1 -- 172.21.15.62:0/2054744412 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb8a413be90 con 0x7fb8a4143dd0 2024-03-23T11:19:11.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.596+0000 7fb8a9a00640 1 -- 172.21.15.62:0/2054744412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb8a413be90 con 0x7fb8a4143dd0 2024-03-23T11:19:11.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.597+0000 7fb89be00640 1 -- 172.21.15.62:0/2054744412 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb8a413be90 con 0x7fb8a4143dd0 2024-03-23T11:19:11.598 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:11.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.600+0000 7fb8a9a00640 1 -- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb88006a7a0 msgr2=0x7fb88006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.600+0000 7fb8a9a00640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb88006a7a0 0x7fb88006cc60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb894009a30 tx=0x7fb894008040 comp rx=0 tx=0).stop 2024-03-23T11:19:11.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.600+0000 7fb8a9a00640 1 -- 172.21.15.62:0/2054744412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8a4143dd0 msgr2=0x7fb8a40b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.600+0000 7fb8a9a00640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8a4143dd0 0x7fb8a40b04f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00cc90 tx=0x7fb89c00a430 comp rx=0 tx=0).stop 2024-03-23T11:19:11.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.600+0000 7fb8a9a00640 1 -- 172.21.15.62:0/2054744412 shutdown_connections 2024-03-23T11:19:11.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.600+0000 7fb8a9a00640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb88006a7a0 0x7fb88006cc60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.600+0000 7fb8a9a00640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8a4143dd0 0x7fb8a40b04f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.600+0000 7fb8a9a00640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8a40b2820 0x7fb8a40affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.600+0000 7fb8a9a00640 1 --2- 172.21.15.62:0/2054744412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8a4070090 0x7fb8a4159b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.600+0000 7fb8a9a00640 1 -- 172.21.15.62:0/2054744412 >> 172.21.15.62:0/2054744412 conn(0x7fb8a4066050 msgr2=0x7fb8a415d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:11.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.601+0000 7fb8a9a00640 1 -- 172.21.15.62:0/2054744412 shutdown_connections 2024-03-23T11:19:11.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.601+0000 7fb8a9a00640 1 -- 172.21.15.62:0/2054744412 wait complete. 2024-03-23T11:19:11.620 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:11.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.838+0000 7f5b03200640 1 Processor -- start 2024-03-23T11:19:11.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.839+0000 7f5b03200640 1 -- start start 2024-03-23T11:19:11.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.840+0000 7f5b03200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5afc0cde00 0x7f5afc0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.840+0000 7f5b03200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5afc06fd20 0x7f5afc070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.840+0000 7f5b03200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0b5120 con 0x7f5afc0cf780 2024-03-23T11:19:11.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.840+0000 7f5b03200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5afc0cd110 con 0x7f5afc06fd20 2024-03-23T11:19:11.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.840+0000 7f5b03200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5afc0d00a0 con 0x7f5afc0cde00 2024-03-23T11:19:11.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.841+0000 7f5b01200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5afc06fd20 0x7f5afc070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.841+0000 7f5b01200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5afc06fd20 0x7f5afc070150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52398/0 (socket says 172.21.15.62:52398) 2024-03-23T11:19:11.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.841+0000 7f5b01200640 1 -- 172.21.15.62:0/4249736643 learned_addr learned my addr 172.21.15.62:0/4249736643 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:11.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.841+0000 7f5afb400640 1 --2- 172.21.15.62:0/4249736643 >> v2:172.21.15.120:3300/0 conn(0x7f5afc0cde00 0x7f5afc0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.841+0000 7f5b01200640 1 -- 172.21.15.62:0/4249736643 >> v2:172.21.15.120:3300/0 conn(0x7f5afc0cde00 msgr2=0x7f5afc0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:11.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.841+0000 7f5b01200640 1 --2- 172.21.15.62:0/4249736643 >> v2:172.21.15.120:3300/0 conn(0x7f5afc0cde00 0x7f5afc0cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.841+0000 7f5b01200640 1 -- 172.21.15.62:0/4249736643 >> v1:172.21.15.120:6789/0 conn(0x7f5afc0cf780 legacy=0x7f5afc15b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.841+0000 7f5b01200640 1 -- 172.21.15.62:0/4249736643 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc0b57c0 con 0x7f5afc06fd20 2024-03-23T11:19:11.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.843+0000 7f5b01200640 1 --2- 172.21.15.62:0/4249736643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5afc06fd20 0x7f5afc070150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5afc074000 tx=0x7f5af402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cccf33313b1b486 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:11.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.843+0000 7f5afaa00640 1 -- 172.21.15.62:0/4249736643 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af402cb20 con 0x7f5afc06fd20 2024-03-23T11:19:11.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.843+0000 7f5afaa00640 1 -- 172.21.15.62:0/4249736643 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af402cc80 con 0x7f5afc06fd20 2024-03-23T11:19:11.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.843+0000 7f5afaa00640 1 -- 172.21.15.62:0/4249736643 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af40343f0 con 0x7f5afc06fd20 2024-03-23T11:19:11.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.844+0000 7f5b03200640 1 -- 172.21.15.62:0/4249736643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5afc06fd20 msgr2=0x7f5afc070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.844+0000 7f5b03200640 1 --2- 172.21.15.62:0/4249736643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5afc06fd20 0x7f5afc070150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5afc074000 tx=0x7f5af402bf80 comp rx=0 tx=0).stop 2024-03-23T11:19:11.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.844+0000 7f5b03200640 1 -- 172.21.15.62:0/4249736643 shutdown_connections 2024-03-23T11:19:11.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.844+0000 7f5b03200640 1 --2- 172.21.15.62:0/4249736643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5afc06fd20 0x7f5afc070150 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.844+0000 7f5b03200640 1 --2- 172.21.15.62:0/4249736643 >> v2:172.21.15.120:3300/0 conn(0x7f5afc0cde00 0x7f5afc0cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.844+0000 7f5b03200640 1 -- 172.21.15.62:0/4249736643 >> 172.21.15.62:0/4249736643 conn(0x7f5afc06e580 msgr2=0x7f5afc06e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:11.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.844+0000 7f5b03200640 1 -- 172.21.15.62:0/4249736643 shutdown_connections 2024-03-23T11:19:11.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.844+0000 7f5b03200640 1 -- 172.21.15.62:0/4249736643 wait complete. 2024-03-23T11:19:11.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.846+0000 7f5b03200640 1 Processor -- start 2024-03-23T11:19:11.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.847+0000 7f5b03200640 1 -- start start 2024-03-23T11:19:11.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.848+0000 7f5b03200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5afc06fd20 0x7f5afc143f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.848+0000 7f5b03200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5afc0cde00 0x7f5afc1444d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.848+0000 7f5b03200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5afc0cf780 0x7f5afc144a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.848+0000 7f5b03200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc1576a0 con 0x7f5afc0cf780 2024-03-23T11:19:11.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.848+0000 7f5b03200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc149610 con 0x7f5afc0cde00 2024-03-23T11:19:11.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.848+0000 7f5b03200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc148a40 con 0x7f5afc06fd20 2024-03-23T11:19:11.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.848+0000 7f5b01200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5afc0cf780 0x7f5afc144a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.848+0000 7f5b01200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5afc0cf780 0x7f5afc144a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52402/0 (socket says 172.21.15.62:52402) 2024-03-23T11:19:11.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.848+0000 7f5b01200640 1 -- 172.21.15.62:0/2863049524 learned_addr learned my addr 172.21.15.62:0/2863049524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:11.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.849+0000 7f5afb400640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5afc0cde00 0x7f5afc1444d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.849+0000 7f5afbe00640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5afc06fd20 0x7f5afc143f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.849+0000 7f5b01200640 1 -- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5afc06fd20 msgr2=0x7f5afc143f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.849+0000 7f5b01200640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5afc06fd20 0x7f5afc143f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.849+0000 7f5b01200640 1 -- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5afc0cde00 msgr2=0x7f5afc1444d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:11.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.849+0000 7f5b01200640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5afc0cde00 0x7f5afc1444d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:11.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.849+0000 7f5b01200640 1 -- 172.21.15.62:0/2863049524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af4009b30 con 0x7f5afc0cf780 2024-03-23T11:19:11.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.849+0000 7f5b01200640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5afc0cf780 0x7f5afc144a10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5af4036040 tx=0x7f5af4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:11.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.849+0000 7f5af3e00640 1 -- 172.21.15.62:0/2863049524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af4009510 con 0x7f5afc0cf780 2024-03-23T11:19:11.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.850+0000 7f5af3e00640 1 -- 172.21.15.62:0/2863049524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af402c920 con 0x7f5afc0cf780 2024-03-23T11:19:11.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.850+0000 7f5af3e00640 1 -- 172.21.15.62:0/2863049524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af403e690 con 0x7f5afc0cf780 2024-03-23T11:19:11.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.850+0000 7f5b03200640 1 -- 172.21.15.62:0/2863049524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc148ce0 con 0x7f5afc0cf780 2024-03-23T11:19:11.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.850+0000 7f5b03200640 1 -- 172.21.15.62:0/2863049524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc1492a0 con 0x7f5afc0cf780 2024-03-23T11:19:11.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.851+0000 7f5b03200640 1 -- 172.21.15.62:0/2863049524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc1497a0 con 0x7f5afc0cf780 2024-03-23T11:19:11.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.852+0000 7f5af3e00640 1 -- 172.21.15.62:0/2863049524 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5af403e7f0 con 0x7f5afc0cf780 2024-03-23T11:19:11.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.853+0000 7f5af3e00640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5adc06a7a0 0x7f5adc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:11.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.853+0000 7f5af3e00640 1 -- 172.21.15.62:0/2863049524 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5af40ab8e0 con 0x7f5afc0cf780 2024-03-23T11:19:11.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.853+0000 7f5afbe00640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5adc06a7a0 0x7f5adc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:11.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.854+0000 7f5afbe00640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5adc06a7a0 0x7f5adc06cc60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5afc14e600 tx=0x7f5ae8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:11.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:11.857+0000 7f5af3e00640 1 -- 172.21.15.62:0/2863049524 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5af4076ca0 con 0x7f5afc0cf780 2024-03-23T11:19:12.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.053+0000 7f5b03200640 1 -- 172.21.15.62:0/2863049524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5afc0672c0 con 0x7f5afc0cf780 2024-03-23T11:19:12.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.054+0000 7f5af3e00640 1 -- 172.21.15.62:0/2863049524 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5af407a2c0 con 0x7f5afc0cf780 2024-03-23T11:19:12.055 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:12.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 -- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5adc06a7a0 msgr2=0x7f5adc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5adc06a7a0 0x7f5adc06cc60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5afc14e600 tx=0x7f5ae8009040 comp rx=0 tx=0).stop 2024-03-23T11:19:12.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 -- 172.21.15.62:0/2863049524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5afc0cf780 msgr2=0x7f5afc144a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5afc0cf780 0x7f5afc144a10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5af4036040 tx=0x7f5af4004a40 comp rx=0 tx=0).stop 2024-03-23T11:19:12.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 -- 172.21.15.62:0/2863049524 shutdown_connections 2024-03-23T11:19:12.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5adc06a7a0 0x7f5adc06cc60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5afc0cf780 0x7f5afc144a10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5afc0cde00 0x7f5afc1444d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 --2- 172.21.15.62:0/2863049524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5afc06fd20 0x7f5afc143f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 -- 172.21.15.62:0/2863049524 >> 172.21.15.62:0/2863049524 conn(0x7f5afc06e580 msgr2=0x7f5afc0b06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:12.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 -- 172.21.15.62:0/2863049524 shutdown_connections 2024-03-23T11:19:12.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.057+0000 7f5b03200640 1 -- 172.21.15.62:0/2863049524 wait complete. 2024-03-23T11:19:12.076 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:19:12.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.302+0000 7fe3ab000640 1 Processor -- start 2024-03-23T11:19:12.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.302+0000 7fe3ab000640 1 -- start start 2024-03-23T11:19:12.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.304+0000 7fe3ab000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a406c300 0x7fe3a40ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.304+0000 7fe3ab000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a40cbc60 con 0x7fe3a406c6e0 2024-03-23T11:19:12.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.304+0000 7fe3ab000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a40cbf40 con 0x7fe3a40cafa0 2024-03-23T11:19:12.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.304+0000 7fe3ab000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40cc220 con 0x7fe3a406c300 2024-03-23T11:19:12.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.304+0000 7fe3a9000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe3a406c6e0 0x7fe3a406cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36030/0 (socket says 172.21.15.62:36030) 2024-03-23T11:19:12.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.304+0000 7fe3a9000640 1 -- 172.21.15.62:0/2645380732 learned_addr learned my addr 172.21.15.62:0/2645380732 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:12.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.304+0000 7fe3a3e00640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a406c300 0x7fe3a40ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.304+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049154094 0 0) 0x7fe3a40cbc60 con 0x7fe3a406c6e0 2024-03-23T11:19:12.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe388003520 con 0x7fe3a406c6e0 2024-03-23T11:19:12.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3537052592 0 0) 0x7fe388003520 con 0x7fe3a406c6e0 2024-03-23T11:19:12.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a406c300 msgr2=0x7fe3a40ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:12.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a406c300 0x7fe3a40ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 >> v1:172.21.15.120:6789/0 conn(0x7fe3a40cafa0 legacy=0x7fe3a40cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40cc500 con 0x7fe3a406c6e0 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fe39c0019f0 con 0x7fe3a406c6e0 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 >> v1:172.21.15.62:6789/0 conn(0x7fe3a406c6e0 legacy=0x7fe3a406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3880039a0 0x7fe388003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a406c300 0x7fe3880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388004d00 0x7fe388005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40cc220 con 0x7fe388004d00 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38800bab0 con 0x7fe3880039a0 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38800bc70 con 0x7fe3a406c300 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.305+0000 7fe3a9000640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388004d00 0x7fe388005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.306+0000 7fe3a3e00640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3880039a0 0x7fe388003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.306+0000 7fe3a3400640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a406c300 0x7fe3880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.306+0000 7fe3a9000640 1 -- 172.21.15.62:0/2645380732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a406c300 msgr2=0x7fe3880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:12.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.306+0000 7fe3a9000640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a406c300 0x7fe3880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.306+0000 7fe3a9000640 1 -- 172.21.15.62:0/2645380732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3880039a0 msgr2=0x7fe388003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:12.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.306+0000 7fe3a9000640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3880039a0 0x7fe388003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.306+0000 7fe3a9000640 1 -- 172.21.15.62:0/2645380732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388003520 con 0x7fe388004d00 2024-03-23T11:19:12.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.307+0000 7fe3a9000640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388004d00 0x7fe388005180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00a830 tx=0x7fe3880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=defd467f104d1494 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:12.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.307+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0072d0 con 0x7fe388004d00 2024-03-23T11:19:12.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.307+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39c02f070 con 0x7fe388004d00 2024-03-23T11:19:12.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.307+0000 7fe3a2a00640 1 -- 172.21.15.62:0/2645380732 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0076b0 con 0x7fe388004d00 2024-03-23T11:19:12.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.308+0000 7fe3ab000640 1 -- 172.21.15.62:0/2645380732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388004d00 msgr2=0x7fe388005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.308+0000 7fe3ab000640 1 --2- 172.21.15.62:0/2645380732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388004d00 0x7fe388005180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00a830 tx=0x7fe3880057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.308+0000 7fe3a9000640 1 -- 172.21.15.62:0/2645380732 reap_dead start 2024-03-23T11:19:12.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.308+0000 7fe3ab000640 1 -- 172.21.15.62:0/2645380732 shutdown_connections 2024-03-23T11:19:12.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.308+0000 7fe3ab000640 1 -- 172.21.15.62:0/2645380732 >> 172.21.15.62:0/2645380732 conn(0x7fe3a40b02e0 msgr2=0x7fe3a40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:12.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.308+0000 7fe3ab000640 1 -- 172.21.15.62:0/2645380732 shutdown_connections 2024-03-23T11:19:12.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.308+0000 7fe3ab000640 1 -- 172.21.15.62:0/2645380732 wait complete. 2024-03-23T11:19:12.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.310+0000 7fe3ab000640 1 Processor -- start 2024-03-23T11:19:12.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.310+0000 7fe3ab000640 1 -- start start 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.311+0000 7fe3ab000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe388004d00 0x7fe3a4152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.311+0000 7fe3ab000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3a4153500 0x7fe3a415ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.311+0000 7fe3ab000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a415b2c0 0x7fe3a4133410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.311+0000 7fe3ab000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a415b750 con 0x7fe3a4153500 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.311+0000 7fe3ab000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4152b30 con 0x7fe3a415b2c0 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.311+0000 7fe3ab000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4150af0 con 0x7fe388004d00 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.311+0000 7fe3a3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3a4153500 0x7fe3a415ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.311+0000 7fe3a3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3a4153500 0x7fe3a415ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52418/0 (socket says 172.21.15.62:52418) 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.311+0000 7fe3a3400640 1 -- 172.21.15.62:0/2498274304 learned_addr learned my addr 172.21.15.62:0/2498274304 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.312+0000 7fe3a3400640 1 -- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe388004d00 msgr2=0x7fe3a4152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.312+0000 7fe3a3400640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe388004d00 0x7fe3a4152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.312+0000 7fe3a3400640 1 -- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a415b2c0 msgr2=0x7fe3a4133410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.312+0000 7fe3a3400640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a415b2c0 0x7fe3a4133410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.312+0000 7fe3a3400640 1 -- 172.21.15.62:0/2498274304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a4151f20 con 0x7fe3a4153500 2024-03-23T11:19:12.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.312+0000 7fe3a3400640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3a4153500 0x7fe3a415ad80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe39000e550 tx=0x7fe39000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:12.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.312+0000 7fe39be00640 1 -- 172.21.15.62:0/2498274304 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3900052e0 con 0x7fe3a4153500 2024-03-23T11:19:12.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.312+0000 7fe39be00640 1 -- 172.21.15.62:0/2498274304 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe390005440 con 0x7fe3a4153500 2024-03-23T11:19:12.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.312+0000 7fe39be00640 1 -- 172.21.15.62:0/2498274304 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe390003810 con 0x7fe3a4153500 2024-03-23T11:19:12.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.312+0000 7fe3ab000640 1 -- 172.21.15.62:0/2498274304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a4152200 con 0x7fe3a4153500 2024-03-23T11:19:12.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.312+0000 7fe3ab000640 1 -- 172.21.15.62:0/2498274304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a41527c0 con 0x7fe3a4153500 2024-03-23T11:19:12.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.313+0000 7fe3ab000640 1 -- 172.21.15.62:0/2498274304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a4066c90 con 0x7fe3a4153500 2024-03-23T11:19:12.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.314+0000 7fe39be00640 1 -- 172.21.15.62:0/2498274304 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe390003970 con 0x7fe3a4153500 2024-03-23T11:19:12.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.315+0000 7fe39be00640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe38406a7a0 0x7fe38406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.315+0000 7fe39be00640 1 -- 172.21.15.62:0/2498274304 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe390088b40 con 0x7fe3a4153500 2024-03-23T11:19:12.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.316+0000 7fe3a3e00640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe38406a7a0 0x7fe38406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.316+0000 7fe3a3e00640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe38406a7a0 0x7fe38406cc60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe394009910 tx=0x7fe394008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:12.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.320+0000 7fe39be00640 1 -- 172.21.15.62:0/2498274304 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe390053ec0 con 0x7fe3a4153500 2024-03-23T11:19:12.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.554+0000 7fe3ab000640 1 -- 172.21.15.62:0/2498274304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3a4152980 con 0x7fe3a4153500 2024-03-23T11:19:12.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.555+0000 7fe39be00640 1 -- 172.21.15.62:0/2498274304 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7fe3900574e0 con 0x7fe3a4153500 2024-03-23T11:19:12.555 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:12.555 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/496399527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":496399527},{"type":"v1","addr":"172.21.15.62:6833","nonce":496399527}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/156755611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":156755611},{"type":"v1","addr":"172.21.15.120:6835","nonce":156755611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2391380456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2391380456},{"type":"v1","addr":"172.21.15.62:6835","nonce":2391380456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:18:56.233595+0000","modified":"2024-03-23T11:18:59.758284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/3936303103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3936303103},{"type":"v1","addr":"172.21.15.120:6837","nonce":3936303103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T11:19:12.555 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-03-23T11:19:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.557+0000 7fe3ab000640 1 -- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe38406a7a0 msgr2=0x7fe38406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.557+0000 7fe3ab000640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe38406a7a0 0x7fe38406cc60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe394009910 tx=0x7fe394008040 comp rx=0 tx=0).stop 2024-03-23T11:19:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.557+0000 7fe3ab000640 1 -- 172.21.15.62:0/2498274304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3a4153500 msgr2=0x7fe3a415ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.557+0000 7fe3ab000640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3a4153500 0x7fe3a415ad80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe39000e550 tx=0x7fe39000ea10 comp rx=0 tx=0).stop 2024-03-23T11:19:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.557+0000 7fe3ab000640 1 -- 172.21.15.62:0/2498274304 shutdown_connections 2024-03-23T11:19:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.557+0000 7fe3ab000640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe38406a7a0 0x7fe38406cc60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.557+0000 7fe3ab000640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3a4153500 0x7fe3a415ad80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.557+0000 7fe3ab000640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a415b2c0 0x7fe3a4133410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.558+0000 7fe3ab000640 1 --2- 172.21.15.62:0/2498274304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe388004d00 0x7fe3a4152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.558+0000 7fe3ab000640 1 -- 172.21.15.62:0/2498274304 >> 172.21.15.62:0/2498274304 conn(0x7fe3a40b02e0 msgr2=0x7fe3a406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.558+0000 7fe3ab000640 1 -- 172.21.15.62:0/2498274304 shutdown_connections 2024-03-23T11:19:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.558+0000 7fe3ab000640 1 -- 172.21.15.62:0/2498274304 wait complete. 2024-03-23T11:19:12.577 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:12.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.789+0000 7fb097200640 1 Processor -- start 2024-03-23T11:19:12.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.789+0000 7fb097200640 1 -- start start 2024-03-23T11:19:12.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.790+0000 7fb097200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb090159490 0x7fb09006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb097200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb0900ce5f0 0x7fb0900b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb097200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0900ced90 con 0x7fb09015da90 2024-03-23T11:19:12.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb097200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb09006e3e0 con 0x7fb090159490 2024-03-23T11:19:12.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb097200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb09006c1c0 con 0x7fb0900ce5f0 2024-03-23T11:19:12.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb095e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb090159490 0x7fb09006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb095400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb09015da90 0x7fb0900cdee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36036/0 (socket says 172.21.15.62:36036) 2024-03-23T11:19:12.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb095400640 1 -- 172.21.15.62:0/997101046 learned_addr learned my addr 172.21.15.62:0/997101046 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:12.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb095e00640 1 -- 172.21.15.62:0/997101046 >> v2:172.21.15.120:3300/0 conn(0x7fb0900ce5f0 msgr2=0x7fb0900b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:12.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb096800640 1 --2- 172.21.15.62:0/997101046 >> v2:172.21.15.120:3300/0 conn(0x7fb0900ce5f0 0x7fb0900b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb094a00640 1 -- 172.21.15.62:0/997101046 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364815969 0 0) 0x7fb0900ced90 con 0x7fb09015da90 2024-03-23T11:19:12.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb095e00640 1 --2- 172.21.15.62:0/997101046 >> v2:172.21.15.120:3300/0 conn(0x7fb0900ce5f0 0x7fb0900b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb095e00640 1 -- 172.21.15.62:0/997101046 >> v1:172.21.15.62:6789/0 conn(0x7fb09015da90 legacy=0x7fb0900cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb095e00640 1 -- 172.21.15.62:0/997101046 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09006e100 con 0x7fb090159490 2024-03-23T11:19:12.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.791+0000 7fb096800640 1 --2- 172.21.15.62:0/997101046 >> v2:172.21.15.120:3300/0 conn(0x7fb0900ce5f0 0x7fb0900b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:12.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.793+0000 7fb095e00640 1 --2- 172.21.15.62:0/997101046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb090159490 0x7fb09006cc60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb0900ad990 tx=0x7fb08402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dda78658cc476c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:12.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.793+0000 7fb094a00640 1 -- 172.21.15.62:0/997101046 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08402e070 con 0x7fb090159490 2024-03-23T11:19:12.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.793+0000 7fb094a00640 1 -- 172.21.15.62:0/997101046 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0840315a0 con 0x7fb090159490 2024-03-23T11:19:12.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.793+0000 7fb094a00640 1 -- 172.21.15.62:0/997101046 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084031930 con 0x7fb090159490 2024-03-23T11:19:12.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.793+0000 7fb097200640 1 -- 172.21.15.62:0/997101046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb090159490 msgr2=0x7fb09006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.793+0000 7fb097200640 1 --2- 172.21.15.62:0/997101046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb090159490 0x7fb09006cc60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb0900ad990 tx=0x7fb08402b840 comp rx=0 tx=0).stop 2024-03-23T11:19:12.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.794+0000 7fb097200640 1 -- 172.21.15.62:0/997101046 shutdown_connections 2024-03-23T11:19:12.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.794+0000 7fb097200640 1 --2- 172.21.15.62:0/997101046 >> v2:172.21.15.120:3300/0 conn(0x7fb0900ce5f0 0x7fb0900b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.794+0000 7fb097200640 1 --2- 172.21.15.62:0/997101046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb090159490 0x7fb09006cc60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.794+0000 7fb097200640 1 -- 172.21.15.62:0/997101046 >> 172.21.15.62:0/997101046 conn(0x7fb090066050 msgr2=0x7fb090069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:12.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.794+0000 7fb097200640 1 -- 172.21.15.62:0/997101046 shutdown_connections 2024-03-23T11:19:12.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.794+0000 7fb097200640 1 -- 172.21.15.62:0/997101046 wait complete. 2024-03-23T11:19:12.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.796+0000 7fb097200640 1 Processor -- start 2024-03-23T11:19:12.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.796+0000 7fb097200640 1 -- start start 2024-03-23T11:19:12.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.797+0000 7fb097200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0900ce5f0 0x7fb09014ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.797+0000 7fb097200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb090159490 0x7fb090155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.797+0000 7fb097200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09015da90 0x7fb0901577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.797+0000 7fb097200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09014d380 con 0x7fb09015da90 2024-03-23T11:19:12.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.797+0000 7fb097200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb090156fa0 con 0x7fb0900ce5f0 2024-03-23T11:19:12.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.797+0000 7fb097200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb090155f00 con 0x7fb090159490 2024-03-23T11:19:12.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.797+0000 7fb096800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09015da90 0x7fb0901577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.797+0000 7fb096800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09015da90 0x7fb0901577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52444/0 (socket says 172.21.15.62:52444) 2024-03-23T11:19:12.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.797+0000 7fb096800640 1 -- 172.21.15.62:0/742617107 learned_addr learned my addr 172.21.15.62:0/742617107 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:12.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.797+0000 7fb095e00640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0900ce5f0 0x7fb09014ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.797+0000 7fb095400640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb090159490 0x7fb090155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.798+0000 7fb096800640 1 -- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb090159490 msgr2=0x7fb090155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.798+0000 7fb096800640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb090159490 0x7fb090155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.798+0000 7fb096800640 1 -- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0900ce5f0 msgr2=0x7fb09014ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:12.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.798+0000 7fb096800640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0900ce5f0 0x7fb09014ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:12.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.798+0000 7fb096800640 1 -- 172.21.15.62:0/742617107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb084009400 con 0x7fb09015da90 2024-03-23T11:19:12.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.798+0000 7fb096800640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09015da90 0x7fb0901577a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00cd50 tx=0x7fb08c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:12.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.798+0000 7fb08a000640 1 -- 172.21.15.62:0/742617107 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08c00f680 con 0x7fb09015da90 2024-03-23T11:19:12.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.798+0000 7fb08a000640 1 -- 172.21.15.62:0/742617107 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08c00f800 con 0x7fb09015da90 2024-03-23T11:19:12.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.798+0000 7fb097200640 1 -- 172.21.15.62:0/742617107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0901561e0 con 0x7fb09015da90 2024-03-23T11:19:12.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.798+0000 7fb08a000640 1 -- 172.21.15.62:0/742617107 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08c00fbb0 con 0x7fb09015da90 2024-03-23T11:19:12.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.798+0000 7fb097200640 1 -- 172.21.15.62:0/742617107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0901567a0 con 0x7fb09015da90 2024-03-23T11:19:12.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.800+0000 7fb097200640 1 -- 172.21.15.62:0/742617107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb090066bb0 con 0x7fb09015da90 2024-03-23T11:19:12.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.800+0000 7fb08a000640 1 -- 172.21.15.62:0/742617107 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb08c00b040 con 0x7fb09015da90 2024-03-23T11:19:12.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.801+0000 7fb08a000640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07406a7a0 0x7fb07406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:12.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.801+0000 7fb08a000640 1 -- 172.21.15.62:0/742617107 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb08c089800 con 0x7fb09015da90 2024-03-23T11:19:12.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.802+0000 7fb095e00640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07406a7a0 0x7fb07406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:12.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.803+0000 7fb095e00640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07406a7a0 0x7fb07406cc60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb084036000 tx=0x7fb08403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:12.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:12.806+0000 7fb08a000640 1 -- 172.21.15.62:0/742617107 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb08c054c20 con 0x7fb09015da90 2024-03-23T11:19:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.002+0000 7fb097200640 1 -- 172.21.15.62:0/742617107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb090158970 con 0x7fb09015da90 2024-03-23T11:19:13.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.004+0000 7fb08a000640 1 -- 172.21.15.62:0/742617107 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16037 (secure 0 0 0) 0x7fb08c058240 con 0x7fb09015da90 2024-03-23T11:19:13.005 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:13.005 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:57.238201+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:13.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.007+0000 7fb097200640 1 -- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07406a7a0 msgr2=0x7fb07406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:13.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.007+0000 7fb097200640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07406a7a0 0x7fb07406cc60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb084036000 tx=0x7fb08403d040 comp rx=0 tx=0).stop 2024-03-23T11:19:13.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.007+0000 7fb097200640 1 -- 172.21.15.62:0/742617107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09015da90 msgr2=0x7fb0901577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:13.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.007+0000 7fb097200640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09015da90 0x7fb0901577a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00cd50 tx=0x7fb08c00a650 comp rx=0 tx=0).stop 2024-03-23T11:19:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.007+0000 7fb097200640 1 -- 172.21.15.62:0/742617107 shutdown_connections 2024-03-23T11:19:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.007+0000 7fb097200640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07406a7a0 0x7fb07406cc60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.007+0000 7fb097200640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb09015da90 0x7fb0901577a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.007+0000 7fb097200640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb090159490 0x7fb090155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.007+0000 7fb097200640 1 --2- 172.21.15.62:0/742617107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0900ce5f0 0x7fb09014ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.007+0000 7fb097200640 1 -- 172.21.15.62:0/742617107 >> 172.21.15.62:0/742617107 conn(0x7fb090066050 msgr2=0x7fb0900b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.008+0000 7fb097200640 1 -- 172.21.15.62:0/742617107 shutdown_connections 2024-03-23T11:19:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.008+0000 7fb097200640 1 -- 172.21.15.62:0/742617107 wait complete. 2024-03-23T11:19:13.026 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:19:13.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.232+0000 7fbec2200640 1 Processor -- start 2024-03-23T11:19:13.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.233+0000 7fbec2200640 1 -- start start 2024-03-23T11:19:13.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.234+0000 7fbec2200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbebc15cdc0 0x7fbebc0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:13.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.234+0000 7fbec2200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbebc0cf650 0x7fbebc0cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:13.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.234+0000 7fbec2200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc06bfc0 con 0x7fbebc15c9e0 2024-03-23T11:19:13.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.234+0000 7fbec2200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbebc0d0020 con 0x7fbebc0cf650 2024-03-23T11:19:13.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.234+0000 7fbec2200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc0ce140 con 0x7fbebc15cdc0 2024-03-23T11:19:13.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.235+0000 7fbebbe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbebc0cf650 0x7fbebc0cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:13.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.235+0000 7fbebbe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbebc0cf650 0x7fbebc0cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52448/0 (socket says 172.21.15.62:52448) 2024-03-23T11:19:13.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.235+0000 7fbebbe00640 1 -- 172.21.15.62:0/2670119361 learned_addr learned my addr 172.21.15.62:0/2670119361 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:13.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.235+0000 7fbebaa00640 1 --2- 172.21.15.62:0/2670119361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbebc15cdc0 0x7fbebc0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:13.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.235+0000 7fbebbe00640 1 -- 172.21.15.62:0/2670119361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbebc15cdc0 msgr2=0x7fbebc0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:13.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.235+0000 7fbebbe00640 1 --2- 172.21.15.62:0/2670119361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbebc15cdc0 0x7fbebc0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.235+0000 7fbebbe00640 1 -- 172.21.15.62:0/2670119361 >> v1:172.21.15.120:6789/0 conn(0x7fbebc15c9e0 legacy=0x7fbebc1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:13.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.235+0000 7fbebbe00640 1 -- 172.21.15.62:0/2670119361 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc0b5260 con 0x7fbebc0cf650 2024-03-23T11:19:13.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.236+0000 7fbebbe00640 1 --2- 172.21.15.62:0/2670119361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbebc0cf650 0x7fbebc0cfae0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009e50 tx=0x7fbeb402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5df4ec957e1b116 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:13.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.237+0000 7fbeba000640 1 -- 172.21.15.62:0/2670119361 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb402cb00 con 0x7fbebc0cf650 2024-03-23T11:19:13.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.237+0000 7fbeba000640 1 -- 172.21.15.62:0/2670119361 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb402cc60 con 0x7fbebc0cf650 2024-03-23T11:19:13.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.237+0000 7fbeba000640 1 -- 172.21.15.62:0/2670119361 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40343f0 con 0x7fbebc0cf650 2024-03-23T11:19:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.237+0000 7fbec2200640 1 -- 172.21.15.62:0/2670119361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbebc0cf650 msgr2=0x7fbebc0cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.237+0000 7fbec2200640 1 --2- 172.21.15.62:0/2670119361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbebc0cf650 0x7fbebc0cfae0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009e50 tx=0x7fbeb402bf60 comp rx=0 tx=0).stop 2024-03-23T11:19:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.238+0000 7fbec2200640 1 -- 172.21.15.62:0/2670119361 shutdown_connections 2024-03-23T11:19:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.238+0000 7fbec2200640 1 --2- 172.21.15.62:0/2670119361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbebc0cf650 0x7fbebc0cfae0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.238+0000 7fbec2200640 1 --2- 172.21.15.62:0/2670119361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbebc15cdc0 0x7fbebc0d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.238+0000 7fbec2200640 1 -- 172.21.15.62:0/2670119361 >> 172.21.15.62:0/2670119361 conn(0x7fbebc071570 msgr2=0x7fbebc071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.238+0000 7fbec2200640 1 -- 172.21.15.62:0/2670119361 shutdown_connections 2024-03-23T11:19:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.238+0000 7fbec2200640 1 -- 172.21.15.62:0/2670119361 wait complete. 2024-03-23T11:19:13.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.240+0000 7fbec2200640 1 Processor -- start 2024-03-23T11:19:13.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.240+0000 7fbec2200640 1 -- start start 2024-03-23T11:19:13.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.242+0000 7fbec2200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbebc0cf650 0x7fbebc0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:13.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.242+0000 7fbec2200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbebc15c9e0 0x7fbebc160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:13.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.242+0000 7fbec2200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbebc15cdc0 0x7fbebc160c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:13.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.242+0000 7fbec2200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc1f38e0 con 0x7fbebc0cf650 2024-03-23T11:19:13.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.242+0000 7fbec2200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc1596a0 con 0x7fbebc15cdc0 2024-03-23T11:19:13.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.242+0000 7fbec2200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc0aa990 con 0x7fbebc15c9e0 2024-03-23T11:19:13.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.242+0000 7fbebb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbebc0cf650 0x7fbebc0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:13.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.242+0000 7fbebb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbebc0cf650 0x7fbebc0b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52452/0 (socket says 172.21.15.62:52452) 2024-03-23T11:19:13.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.242+0000 7fbebb400640 1 -- 172.21.15.62:0/1050391657 learned_addr learned my addr 172.21.15.62:0/1050391657 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:13.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.242+0000 7fbebaa00640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbebc15c9e0 0x7fbebc160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:13.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.242+0000 7fbebbe00640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbebc15cdc0 0x7fbebc160c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:13.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.243+0000 7fbebb400640 1 -- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbebc15c9e0 msgr2=0x7fbebc160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:13.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.243+0000 7fbebb400640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbebc15c9e0 0x7fbebc160730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.243+0000 7fbebb400640 1 -- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbebc15cdc0 msgr2=0x7fbebc160c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:13.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.243+0000 7fbebb400640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbebc15cdc0 0x7fbebc160c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.243+0000 7fbebb400640 1 -- 172.21.15.62:0/1050391657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb4009b30 con 0x7fbebc0cf650 2024-03-23T11:19:13.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.243+0000 7fbebbe00640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbebc15cdc0 0x7fbebc160c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:13.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.243+0000 7fbebb400640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbebc0cf650 0x7fbebc0b2f20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbeac00c450 tx=0x7fbeac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:13.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.243+0000 7fbeb3400640 1 -- 172.21.15.62:0/1050391657 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac0052d0 con 0x7fbebc0cf650 2024-03-23T11:19:13.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.243+0000 7fbeb3400640 1 -- 172.21.15.62:0/1050391657 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeac005430 con 0x7fbebc0cf650 2024-03-23T11:19:13.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.243+0000 7fbec2200640 1 -- 172.21.15.62:0/1050391657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc0aac70 con 0x7fbebc0cf650 2024-03-23T11:19:13.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.244+0000 7fbeb3400640 1 -- 172.21.15.62:0/1050391657 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac012870 con 0x7fbebc0cf650 2024-03-23T11:19:13.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.244+0000 7fbec2200640 1 -- 172.21.15.62:0/1050391657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc0ab230 con 0x7fbebc0cf650 2024-03-23T11:19:13.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.245+0000 7fbec2200640 1 -- 172.21.15.62:0/1050391657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc1f3640 con 0x7fbebc0cf650 2024-03-23T11:19:13.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.246+0000 7fbeb3400640 1 -- 172.21.15.62:0/1050391657 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbeac0129d0 con 0x7fbebc0cf650 2024-03-23T11:19:13.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.247+0000 7fbeb3400640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe9806a7a0 0x7fbe9806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:13.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.247+0000 7fbeb3400640 1 -- 172.21.15.62:0/1050391657 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbeac088770 con 0x7fbebc0cf650 2024-03-23T11:19:13.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.247+0000 7fbebaa00640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe9806a7a0 0x7fbe9806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:13.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.248+0000 7fbebaa00640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe9806a7a0 0x7fbe9806cc60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0b6960 tx=0x7fbea4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:13.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.255+0000 7fbeb3400640 1 -- 172.21.15.62:0/1050391657 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbebc1f3640 con 0x7fbebc0cf650 2024-03-23T11:19:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.493+0000 7fbec2200640 1 -- 172.21.15.62:0/1050391657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbebc1f3a20 con 0x7fbebc0cf650 2024-03-23T11:19:13.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.494+0000 7fbeb3400640 1 -- 172.21.15.62:0/1050391657 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7fbebc1f3a20 con 0x7fbebc0cf650 2024-03-23T11:19:13.494 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:13.494 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/496399527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":496399527},{"type":"v1","addr":"172.21.15.62:6833","nonce":496399527}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/156755611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":156755611},{"type":"v1","addr":"172.21.15.120:6835","nonce":156755611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2391380456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2391380456},{"type":"v1","addr":"172.21.15.62:6835","nonce":2391380456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:18:56.233595+0000","modified":"2024-03-23T11:18:59.758284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/3936303103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3936303103},{"type":"v1","addr":"172.21.15.120:6837","nonce":3936303103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T11:19:13.495 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-03-23T11:19:13.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 -- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe9806a7a0 msgr2=0x7fbe9806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:13.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe9806a7a0 0x7fbe9806cc60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0b6960 tx=0x7fbea4009190 comp rx=0 tx=0).stop 2024-03-23T11:19:13.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 -- 172.21.15.62:0/1050391657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbebc0cf650 msgr2=0x7fbebc0b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:13.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbebc0cf650 0x7fbebc0b2f20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbeac00c450 tx=0x7fbeac00c910 comp rx=0 tx=0).stop 2024-03-23T11:19:13.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 -- 172.21.15.62:0/1050391657 shutdown_connections 2024-03-23T11:19:13.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe9806a7a0 0x7fbe9806cc60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbebc15cdc0 0x7fbebc160c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbebc15c9e0 0x7fbebc160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 --2- 172.21.15.62:0/1050391657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbebc0cf650 0x7fbebc0b2f20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:13.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 -- 172.21.15.62:0/1050391657 >> 172.21.15.62:0/1050391657 conn(0x7fbebc071570 msgr2=0x7fbebc0b7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:13.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 -- 172.21.15.62:0/1050391657 shutdown_connections 2024-03-23T11:19:13.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:13.497+0000 7fbec2200640 1 -- 172.21.15.62:0/1050391657 wait complete. 2024-03-23T11:19:13.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:18:56.233595+0000', 'modified': '2024-03-23T11:18:59.758284+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4263}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4263': {'gid': 4263, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6837/3936303103', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 3936303103}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 3936303103}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:19:13.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:19:13.918 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-23T11:19:14.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.151+0000 7f2afb200640 1 Processor -- start 2024-03-23T11:19:14.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.151+0000 7f2afb200640 1 -- start start 2024-03-23T11:19:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.152+0000 7f2afb200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2afc0b2d50 0x7f2afc0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.152+0000 7f2afb200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc0d0810 con 0x7f2afc15c960 2024-03-23T11:19:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.152+0000 7f2afb200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc0cd9d0 con 0x7f2afc0d13b0 2024-03-23T11:19:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.152+0000 7f2afb200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2afc0cdcb0 con 0x7f2afc0b2d50 2024-03-23T11:19:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.153+0000 7f2af9e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2afc15c960 0x7f2afc0d0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36038/0 (socket says 172.21.15.62:36038) 2024-03-23T11:19:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.153+0000 7f2af9e00640 1 -- 172.21.15.62:0/1179397433 learned_addr learned my addr 172.21.15.62:0/1179397433 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.153+0000 7f2afa800640 1 --2- 172.21.15.62:0/1179397433 >> v2:172.21.15.62:3300/0 conn(0x7f2afc0b2d50 0x7f2afc0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.153+0000 7f2afa800640 1 -- 172.21.15.62:0/1179397433 >> v1:172.21.15.120:6789/0 conn(0x7f2afc0d13b0 legacy=0x7f2afc0b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.153+0000 7f2afa800640 1 -- 172.21.15.62:0/1179397433 >> v1:172.21.15.62:6789/0 conn(0x7f2afc15c960 legacy=0x7f2afc0d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.153+0000 7f2afa800640 1 -- 172.21.15.62:0/1179397433 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc0cdf90 con 0x7f2afc0b2d50 2024-03-23T11:19:14.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.154+0000 7f2afa800640 1 --2- 172.21.15.62:0/1179397433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2afc0b2d50 0x7f2afc0cf790 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2af40097a0 tx=0x7f2af402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233271c7fccee337 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:14.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.154+0000 7f2af8a00640 1 -- 172.21.15.62:0/1179397433 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af402f810 con 0x7f2afc0b2d50 2024-03-23T11:19:14.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.155+0000 7f2af8a00640 1 -- 172.21.15.62:0/1179397433 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af402f970 con 0x7f2afc0b2d50 2024-03-23T11:19:14.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.155+0000 7f2af8a00640 1 -- 172.21.15.62:0/1179397433 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af402fd00 con 0x7f2afc0b2d50 2024-03-23T11:19:14.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.155+0000 7f2afb200640 1 -- 172.21.15.62:0/1179397433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2afc0b2d50 msgr2=0x7f2afc0cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.155+0000 7f2afb200640 1 --2- 172.21.15.62:0/1179397433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2afc0b2d50 0x7f2afc0cf790 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2af40097a0 tx=0x7f2af402b9f0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.155+0000 7f2afb200640 1 -- 172.21.15.62:0/1179397433 shutdown_connections 2024-03-23T11:19:14.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.155+0000 7f2afb200640 1 --2- 172.21.15.62:0/1179397433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2afc0b2d50 0x7f2afc0cf790 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.155+0000 7f2afb200640 1 -- 172.21.15.62:0/1179397433 >> 172.21.15.62:0/1179397433 conn(0x7f2afc089f60 msgr2=0x7f2afc0d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:14.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.155+0000 7f2afb200640 1 -- 172.21.15.62:0/1179397433 shutdown_connections 2024-03-23T11:19:14.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.155+0000 7f2afb200640 1 -- 172.21.15.62:0/1179397433 wait complete. 2024-03-23T11:19:14.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.157+0000 7f2afb200640 1 Processor -- start 2024-03-23T11:19:14.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.157+0000 7f2afb200640 1 -- start start 2024-03-23T11:19:14.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.158+0000 7f2afb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2afc0b2d50 0x7f2afc0ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.158+0000 7f2afb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2afc0d13b0 0x7f2afc15fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.158+0000 7f2afb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2afc15c960 0x7f2afc15e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.158+0000 7f2afb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc156700 con 0x7f2afc15c960 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.158+0000 7f2afb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc15ff70 con 0x7f2afc0b2d50 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.158+0000 7f2afb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc15ea40 con 0x7f2afc0d13b0 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.159+0000 7f2afa800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2afc15c960 0x7f2afc15e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.159+0000 7f2afa800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2afc15c960 0x7f2afc15e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52458/0 (socket says 172.21.15.62:52458) 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.159+0000 7f2afa800640 1 -- 172.21.15.62:0/51282525 learned_addr learned my addr 172.21.15.62:0/51282525 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.159+0000 7f2af9e00640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2afc0b2d50 0x7f2afc0ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.159+0000 7f2af9400640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2afc0d13b0 0x7f2afc15fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.159+0000 7f2afa800640 1 -- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2afc0d13b0 msgr2=0x7f2afc15fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.159+0000 7f2afa800640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2afc0d13b0 0x7f2afc15fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.159+0000 7f2afa800640 1 -- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2afc0b2d50 msgr2=0x7f2afc0ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.159+0000 7f2afa800640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2afc0b2d50 0x7f2afc0ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.159+0000 7f2afa800640 1 -- 172.21.15.62:0/51282525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af4009400 con 0x7f2afc15c960 2024-03-23T11:19:14.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.159+0000 7f2afa800640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2afc15c960 0x7f2afc15e500 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2af40098c0 tx=0x7f2af4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:14.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.160+0000 7f2af2000640 1 -- 172.21.15.62:0/51282525 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af403c040 con 0x7f2afc15c960 2024-03-23T11:19:14.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.160+0000 7f2af2000640 1 -- 172.21.15.62:0/51282525 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af4037890 con 0x7f2afc15c960 2024-03-23T11:19:14.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.160+0000 7f2af2000640 1 -- 172.21.15.62:0/51282525 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af4037c40 con 0x7f2afc15c960 2024-03-23T11:19:14.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.160+0000 7f2afb200640 1 -- 172.21.15.62:0/51282525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2afc15dab0 con 0x7f2afc15c960 2024-03-23T11:19:14.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.160+0000 7f2afb200640 1 -- 172.21.15.62:0/51282525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2afc0cc3e0 con 0x7f2afc15c960 2024-03-23T11:19:14.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.162+0000 7f2af2000640 1 -- 172.21.15.62:0/51282525 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2af404b3b0 con 0x7f2afc15c960 2024-03-23T11:19:14.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.163+0000 7f2af2000640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ad806a7a0 0x7f2ad806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.164+0000 7f2af2000640 1 -- 172.21.15.62:0/51282525 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2af40ab060 con 0x7f2afc15c960 2024-03-23T11:19:14.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.164+0000 7f2af9e00640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ad806a7a0 0x7f2ad806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.164+0000 7f2b03de0740 1 Processor -- start 2024-03-23T11:19:14.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.164+0000 7f2b03de0740 1 -- start start 2024-03-23T11:19:14.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.165+0000 7f2af9e00640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ad806a7a0 0x7f2ad806cc60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4005880 tx=0x7f2ae4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:14.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.165+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 0x56232a453b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.165+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a4525a0 0x56232a454090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.165+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a44ea40 0x56232a47e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.165+0000 7f2b03de0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232a3bf110 con 0x56232a4525a0 2024-03-23T11:19:14.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.165+0000 7f2b03de0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232a3bc8a0 con 0x56232a49c520 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.165+0000 7f2b03de0740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56232a3b8190 con 0x56232a44ea40 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2af9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a4525a0 0x56232a454090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2af9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a4525a0 0x56232a454090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52472/0 (socket says 172.21.15.62:52472) 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2af9400640 1 -- 172.21.15.62:0/524948572 learned_addr learned my addr 172.21.15.62:0/524948572 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2af9e00640 1 --2- 172.21.15.62:0/524948572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 0x56232a453b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2af9400640 1 -- 172.21.15.62:0/524948572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a44ea40 msgr2=0x56232a47e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2af9400640 1 --2- 172.21.15.62:0/524948572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a44ea40 0x56232a47e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2af9400640 1 -- 172.21.15.62:0/524948572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 msgr2=0x56232a453b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2af9400640 1 --2- 172.21.15.62:0/524948572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 0x56232a453b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2af9400640 1 -- 172.21.15.62:0/524948572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56232a383230 con 0x56232a4525a0 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2af9e00640 1 --2- 172.21.15.62:0/524948572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 0x56232a453b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2af9400640 1 --2- 172.21.15.62:0/524948572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a4525a0 0x56232a454090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2aec001ca0 tx=0x7f2aec002160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2083bf64d3d19b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:14.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.166+0000 7f2aeaa00640 1 -- 172.21.15.62:0/524948572 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec003bd0 con 0x56232a4525a0 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.167+0000 7f2aeaa00640 1 -- 172.21.15.62:0/524948572 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aec003d30 con 0x56232a4525a0 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.167+0000 7f2aeaa00640 1 -- 172.21.15.62:0/524948572 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec003ea0 con 0x56232a4525a0 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.167+0000 7f2b03de0740 1 -- 172.21.15.62:0/524948572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a4525a0 msgr2=0x56232a454090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.167+0000 7f2b03de0740 1 --2- 172.21.15.62:0/524948572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a4525a0 0x56232a454090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2aec001ca0 tx=0x7f2aec002160 comp rx=0 tx=0).stop 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.167+0000 7f2b03de0740 1 -- 172.21.15.62:0/524948572 shutdown_connections 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.167+0000 7f2b03de0740 1 --2- 172.21.15.62:0/524948572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a4525a0 0x56232a454090 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.167+0000 7f2b03de0740 1 --2- 172.21.15.62:0/524948572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a44ea40 0x56232a47e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.167+0000 7f2b03de0740 1 --2- 172.21.15.62:0/524948572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 0x56232a453b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.167+0000 7f2b03de0740 1 -- 172.21.15.62:0/524948572 >> 172.21.15.62:0/524948572 conn(0x56232a450cb0 msgr2=0x56232a499830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.167+0000 7f2b03de0740 1 -- 172.21.15.62:0/524948572 shutdown_connections 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.167+0000 7f2b03de0740 1 -- 172.21.15.62:0/524948572 wait complete. 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.168+0000 7f2b03de0740 1 -- start start 2024-03-23T11:19:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.168+0000 7f2b03de0740 1 Processor -- start 2024-03-23T11:19:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.169+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a4525a0 0x56232a47ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.169+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a49f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.169+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a476dc0 0x56232a464620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.169+0000 7f2b03de0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232a3598c0 con 0x56232a49c520 2024-03-23T11:19:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.169+0000 7f2b03de0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232a359560 con 0x56232a4525a0 2024-03-23T11:19:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.169+0000 7f2b03de0740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56232a11ff20 con 0x56232a476dc0 2024-03-23T11:19:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.169+0000 7f2b03de0740 10 client.0 _ll_register_callbacks cb 0x56232a3b1800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:19:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.169+0000 7f2af9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a49f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.169+0000 7f2af9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a49f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52486/0 (socket says 172.21.15.62:52486) 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.169+0000 7f2af9400640 1 -- 172.21.15.62:0/223029036 learned_addr learned my addr 172.21.15.62:0/223029036 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2afa800640 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a476dc0 0x56232a464620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2af9e00640 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a4525a0 0x56232a47ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2af9400640 1 -- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a476dc0 msgr2=0x56232a464620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2af9400640 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a476dc0 0x56232a464620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2af9400640 1 -- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a4525a0 msgr2=0x56232a47ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2af9400640 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a4525a0 0x56232a47ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2af9400640 1 -- 172.21.15.62:0/223029036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aec00f040 con 0x56232a49c520 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2af9400640 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a49f420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00e530 tx=0x7f2aec0012a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e0c043495d39e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2adf400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2adf400640 1 -- 172.21.15.62:0/223029036 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec003ea0 con 0x56232a49c520 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2adf400640 1 -- 172.21.15.62:0/223029036 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2aec00ee50 con 0x56232a49c520 2024-03-23T11:19:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.170+0000 7f2b03de0740 1 -- 172.21.15.62:0/223029036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56232a3e2280 con 0x56232a49c520 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.171+0000 7f2adf400640 1 -- 172.21.15.62:0/223029036 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2aec006ba0 con 0x56232a49c520 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.171+0000 7f2b03de0740 10 client.4632 fetch_fsmap learned FSMap version 8 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.171+0000 7f2b03de0740 1 -- 172.21.15.62:0/223029036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x56232a3b6810 con 0x56232a49c520 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.171+0000 7f2adf400640 1 -- 172.21.15.62:0/223029036 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56232a3e2280 con 0x56232a49c520 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.171+0000 7f2adf400640 1 -- 172.21.15.62:0/223029036 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2aec006d00 con 0x56232a49c520 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.171+0000 7f2b03de0740 10 client.4632 fetch_fsmap finished waiting for FSMap version 8 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.171+0000 7f2b03de0740 10 client.4632 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.171+0000 7f2b03de0740 20 client.4632 populate_metadata read hostname 'smithi062' 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.171+0000 7f2b03de0740 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:6836/3936303103,v1:172.21.15.120:6837/3936303103] conn(0x56232a4636e0 0x56232a49df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.171+0000 7f2b03de0740 4 client.4632 mds_command: new command op to 4263 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.171+0000 7f2b03de0740 1 -- 172.21.15.62:0/223029036 --> [v2:172.21.15.120:6836/3936303103,v1:172.21.15.120:6837/3936303103] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56232a3aa9d0 con 0x56232a4636e0 2024-03-23T11:19:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.172+0000 7f2af9e00640 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:6836/3936303103,v1:172.21.15.120:6837/3936303103] conn(0x56232a4636e0 0x56232a49df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.173+0000 7f2af9e00640 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:6836/3936303103,v1:172.21.15.120:6837/3936303103] conn(0x56232a4636e0 0x56232a49df00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7be1022c73292dad server_cookie=89e6f6b40d03c507 in_seq=0 out_seq=0 2024-03-23T11:19:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.173+0000 7f2adf400640 10 client.4632 ms_handle_connect on v2:172.21.15.120:6836/3936303103 2024-03-23T11:19:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.174+0000 7f2adf400640 1 -- 172.21.15.62:0/223029036 <== mds.0 v2:172.21.15.120:6836/3936303103 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56232a3aa9d0 con 0x56232a4636e0 2024-03-23T11:19:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.174+0000 7f2adf400640 10 client.4632 handle_command_reply: tid=0 2024-03-23T11:19:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 1 client.4632 shutdown 2024-03-23T11:19:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 1 -- 172.21.15.62:0/223029036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 msgr2=0x56232a49f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a49f420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00e530 tx=0x7f2aec0012a0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 1 -- 172.21.15.62:0/223029036 shutdown_connections 2024-03-23T11:19:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:6836/3936303103,v1:172.21.15.120:6837/3936303103] conn(0x56232a4636e0 0x56232a49df00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a49f420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a476dc0 0x56232a464620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 1 --2- 172.21.15.62:0/223029036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a4525a0 0x56232a47ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2adf400640 0 client.4632 ms_handle_reset on v2:172.21.15.120:6836/3936303103 2024-03-23T11:19:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 1 -- 172.21.15.62:0/223029036 >> 172.21.15.62:0/223029036 conn(0x56232a44ea40 msgr2=0x56232a452d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 1 -- 172.21.15.62:0/223029036 shutdown_connections 2024-03-23T11:19:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 1 -- 172.21.15.62:0/223029036 wait complete. 2024-03-23T11:19:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.175+0000 7f2b03de0740 20 client.4632 trim_cache size 0 max 16384 2024-03-23T11:19:14.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.192+0000 7f2b03de0740 1 Processor -- start 2024-03-23T11:19:14.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.192+0000 7f2b03de0740 1 -- start start 2024-03-23T11:19:14.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a4525a0 0x56232a463ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a476dc0 0x56232a47c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 0x56232a47cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2b03de0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232a47bef0 con 0x56232a476dc0 2024-03-23T11:19:14.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2b03de0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232a49bd60 con 0x56232a49c520 2024-03-23T11:19:14.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2b03de0740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56232a47d410 con 0x56232a4525a0 2024-03-23T11:19:14.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2af9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a476dc0 0x56232a47c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2af9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a476dc0 0x56232a47c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52498/0 (socket says 172.21.15.62:52498) 2024-03-23T11:19:14.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2af9400640 1 -- 172.21.15.62:0/202759020 learned_addr learned my addr 172.21.15.62:0/202759020 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:14.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2af9e00640 1 --2- 172.21.15.62:0/202759020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a4525a0 0x56232a463ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2afa800640 1 --2- 172.21.15.62:0/202759020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 0x56232a47cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.193+0000 7f2af9400640 1 -- 172.21.15.62:0/202759020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a4525a0 msgr2=0x56232a463ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.194+0000 7f2af9400640 1 --2- 172.21.15.62:0/202759020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a4525a0 0x56232a463ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.194+0000 7f2af9400640 1 -- 172.21.15.62:0/202759020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 msgr2=0x56232a47cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.194+0000 7f2af9400640 1 --2- 172.21.15.62:0/202759020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 0x56232a47cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.194+0000 7f2af9400640 1 -- 172.21.15.62:0/202759020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56232a358ad0 con 0x56232a476dc0 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.194+0000 7f2afa800640 1 --2- 172.21.15.62:0/202759020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 0x56232a47cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.194+0000 7f2af9400640 1 --2- 172.21.15.62:0/202759020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a476dc0 0x56232a47c710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2aec007dd0 tx=0x7f2aec01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c7667b33152d45 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.194+0000 7f2aeb400640 1 -- 172.21.15.62:0/202759020 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec001b40 con 0x56232a476dc0 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.194+0000 7f2aeb400640 1 -- 172.21.15.62:0/202759020 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aec001910 con 0x56232a476dc0 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.194+0000 7f2aeb400640 1 -- 172.21.15.62:0/202759020 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec00ea70 con 0x56232a476dc0 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.195+0000 7f2b03de0740 1 -- 172.21.15.62:0/202759020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a476dc0 msgr2=0x56232a47c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.195+0000 7f2b03de0740 1 --2- 172.21.15.62:0/202759020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a476dc0 0x56232a47c710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2aec007dd0 tx=0x7f2aec01c000 comp rx=0 tx=0).stop 2024-03-23T11:19:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.195+0000 7f2b03de0740 1 -- 172.21.15.62:0/202759020 shutdown_connections 2024-03-23T11:19:14.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.195+0000 7f2b03de0740 1 --2- 172.21.15.62:0/202759020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a476dc0 0x56232a47c710 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2aec007dd0 tx=0x7f2aec01c000 comp rx=0 tx=0).stop 2024-03-23T11:19:14.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.195+0000 7f2b03de0740 1 --2- 172.21.15.62:0/202759020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a49c520 0x56232a47cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.195+0000 7f2b03de0740 1 --2- 172.21.15.62:0/202759020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a4525a0 0x56232a463ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.195+0000 7f2b03de0740 1 -- 172.21.15.62:0/202759020 >> 172.21.15.62:0/202759020 conn(0x56232a44ea40 msgr2=0x56232a499630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:14.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.195+0000 7f2b03de0740 1 -- 172.21.15.62:0/202759020 shutdown_connections 2024-03-23T11:19:14.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.195+0000 7f2b03de0740 1 -- 172.21.15.62:0/202759020 wait complete. 2024-03-23T11:19:14.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.196+0000 7f2b03de0740 1 -- start start 2024-03-23T11:19:14.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.196+0000 7f2b03de0740 1 Processor -- start 2024-03-23T11:19:14.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a4525a0 0x56232a49d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a4a81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2b03de0740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a4642b0 0x56232a47c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2b03de0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232a49d150 con 0x56232a49c520 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2b03de0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232a4a8ed0 con 0x56232a4525a0 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2b03de0740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56232a47d740 con 0x56232a4642b0 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2b03de0740 10 client.0 _ll_register_callbacks cb 0x56232a49cce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2af9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a4a81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2af9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a4a81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52514/0 (socket says 172.21.15.62:52514) 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2af9400640 1 -- 172.21.15.62:0/3891626921 learned_addr learned my addr 172.21.15.62:0/3891626921 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2af9e00640 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a4525a0 0x56232a49d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2af9400640 1 -- 172.21.15.62:0/3891626921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a4642b0 msgr2=0x56232a47c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.197+0000 7f2af9400640 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a4642b0 0x56232a47c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.198+0000 7f2af9400640 1 -- 172.21.15.62:0/3891626921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a4525a0 msgr2=0x56232a49d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.198+0000 7f2af9400640 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a4525a0 0x56232a49d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.198+0000 7f2af9400640 1 -- 172.21.15.62:0/3891626921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56232a3b6810 con 0x56232a49c520 2024-03-23T11:19:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.198+0000 7f2af9400640 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a4a81f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2aec001a80 tx=0x7f2aec009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe305a883851dbd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.198+0000 7f2adf400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.198+0000 7f2adf400640 1 -- 172.21.15.62:0/3891626921 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec00ea70 con 0x56232a49c520 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.198+0000 7f2adf400640 1 -- 172.21.15.62:0/3891626921 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2aec00ee50 con 0x56232a49c520 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.198+0000 7f2b03de0740 1 -- 172.21.15.62:0/3891626921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56232a4a1d50 con 0x56232a49c520 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.198+0000 7f2adf400640 1 -- 172.21.15.62:0/3891626921 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2aec005320 con 0x56232a49c520 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.198+0000 7f2b03de0740 10 client.4638 fetch_fsmap learned FSMap version 8 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.198+0000 7f2b03de0740 1 -- 172.21.15.62:0/3891626921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x56232a37dfe0 con 0x56232a49c520 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.199+0000 7f2adf400640 1 -- 172.21.15.62:0/3891626921 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56232a4a1d50 con 0x56232a49c520 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.199+0000 7f2adf400640 1 -- 172.21.15.62:0/3891626921 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2aec035070 con 0x56232a49c520 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.199+0000 7f2b03de0740 10 client.4638 fetch_fsmap finished waiting for FSMap version 8 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.199+0000 7f2b03de0740 10 client.4638 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.199+0000 7f2b03de0740 20 client.4638 populate_metadata read hostname 'smithi062' 2024-03-23T11:19:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.199+0000 7f2b03de0740 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.120:6836/3936303103,v1:172.21.15.120:6837/3936303103] conn(0x56232a4a8730 0x56232a47cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.199+0000 7f2b03de0740 4 client.4638 mds_command: new command op to 4263 tid=0[{"prefix": "status"}] 2024-03-23T11:19:14.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.199+0000 7f2b03de0740 1 -- 172.21.15.62:0/3891626921 --> [v2:172.21.15.120:6836/3936303103,v1:172.21.15.120:6837/3936303103] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f2aec003d30 con 0x56232a4a8730 2024-03-23T11:19:14.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.200+0000 7f2af9e00640 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.120:6836/3936303103,v1:172.21.15.120:6837/3936303103] conn(0x56232a4a8730 0x56232a47cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.200+0000 7f2af9e00640 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.120:6836/3936303103,v1:172.21.15.120:6837/3936303103] conn(0x56232a4a8730 0x56232a47cc50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc65f26ca089af8b server_cookie=e7ea06fbcb3c7738 in_seq=0 out_seq=0 2024-03-23T11:19:14.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.200+0000 7f2adf400640 10 client.4638 ms_handle_connect on v2:172.21.15.120:6836/3936303103 2024-03-23T11:19:14.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2adf400640 1 -- 172.21.15.62:0/3891626921 <== mds.0 v2:172.21.15.120:6836/3936303103 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+323 (crc 0 0 0) 0x7f2aec003d30 con 0x56232a4a8730 2024-03-23T11:19:14.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2adf400640 10 client.4638 handle_command_reply: tid=0 2024-03-23T11:19:14.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 1 client.4638 shutdown 2024-03-23T11:19:14.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 1 -- 172.21.15.62:0/3891626921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 msgr2=0x56232a4a81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a4a81f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2aec001a80 tx=0x7f2aec009040 comp rx=0 tx=0).stop 2024-03-23T11:19:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 1 -- 172.21.15.62:0/3891626921 shutdown_connections 2024-03-23T11:19:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.120:6836/3936303103,v1:172.21.15.120:6837/3936303103] conn(0x56232a4a8730 0x56232a47cc50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56232a49c520 0x56232a4a81f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56232a4642b0 0x56232a47c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 1 --2- 172.21.15.62:0/3891626921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56232a4525a0 0x56232a49d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2adf400640 0 client.4638 ms_handle_reset on v2:172.21.15.120:6836/3936303103 2024-03-23T11:19:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 1 -- 172.21.15.62:0/3891626921 >> 172.21.15.62:0/3891626921 conn(0x56232a44ea40 msgr2=0x56232a499420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 1 -- 172.21.15.62:0/3891626921 shutdown_connections 2024-03-23T11:19:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 1 -- 172.21.15.62:0/3891626921 wait complete. 2024-03-23T11:19:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.201+0000 7f2b03de0740 20 client.4638 trim_cache size 0 max 16384 2024-03-23T11:19:14.204 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:19:14.205 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:19:14.205 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:19:14.205 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 4263, 2024-03-23T11:19:14.205 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:19:14.205 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:19:14.205 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:19:14.205 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 17.96245708, 2024-03-23T11:19:14.205 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 8, 2024-03-23T11:19:14.205 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 22, 2024-03-23T11:19:14.205 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 22, 2024-03-23T11:19:14.205 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 25.662103948999999 2024-03-23T11:19:14.206 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:19:14.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 -- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ad806a7a0 msgr2=0x7f2ad806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ad806a7a0 0x7f2ad806cc60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4005880 tx=0x7f2ae4009040 comp rx=0 tx=0).stop 2024-03-23T11:19:14.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 -- 172.21.15.62:0/51282525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2afc15c960 msgr2=0x7f2afc15e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2afc15c960 0x7f2afc15e500 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2af40098c0 tx=0x7f2af4004730 comp rx=0 tx=0).stop 2024-03-23T11:19:14.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 -- 172.21.15.62:0/51282525 shutdown_connections 2024-03-23T11:19:14.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ad806a7a0 0x7f2ad806cc60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2afc15c960 0x7f2afc15e500 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2afc0d13b0 0x7f2afc15fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 --2- 172.21.15.62:0/51282525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2afc0b2d50 0x7f2afc0ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 -- 172.21.15.62:0/51282525 >> 172.21.15.62:0/51282525 conn(0x7f2afc089f60 msgr2=0x7f2afc0af300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:14.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 -- 172.21.15.62:0/51282525 shutdown_connections 2024-03-23T11:19:14.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.202+0000 7f2afb200640 1 -- 172.21.15.62:0/51282525 wait complete. 2024-03-23T11:19:14.218 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-23T11:19:14.240 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-23T11:19:14.241 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-23T11:19:14.241 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-23T11:19:14.241 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T11:19:14.241 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:19:14.241 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-23T11:19:14.241 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T11:19:14.241 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-23T11:19:14.241 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-23T11:19:14.241 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T11:19:14.241 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T11:19:14.241 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-23T11:19:14.242 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T11:19:14.242 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-23T11:19:14.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:14.242 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:19:14.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:14.296 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link delete ceph-brx 2024-03-23T11:19:14.302 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:19:14.348 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:19:14.396 INFO:teuthology.orchestra.run.smithi062.stderr:Cannot find device "ceph-brx" 2024-03-23T11:19:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:19:14.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:14.400 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2024-03-23T11:19:14.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:14.447 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link delete ceph-brx 2024-03-23T11:19:14.586 INFO:teuthology.orchestra.run.smithi120.stderr:Cannot find device "ceph-brx" 2024-03-23T11:19:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:19:14.590 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-23T11:19:14.590 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:19:14.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.819+0000 7ff535e00640 1 Processor -- start 2024-03-23T11:19:14.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.820+0000 7ff535e00640 1 -- start start 2024-03-23T11:19:14.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.821+0000 7ff535e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff53015c9e0 0x7ff5301607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.821+0000 7ff535e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53006bfc0 con 0x7ff5300cf650 2024-03-23T11:19:14.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.821+0000 7ff535e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5300b5260 con 0x7ff53015cdc0 2024-03-23T11:19:14.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.821+0000 7ff535e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5300d2630 con 0x7ff53015c9e0 2024-03-23T11:19:14.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.821+0000 7ff52f400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff5300cf650 0x7ff53015d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36046/0 (socket says 172.21.15.62:36046) 2024-03-23T11:19:14.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.821+0000 7ff52f400640 1 -- 172.21.15.62:0/3842962191 learned_addr learned my addr 172.21.15.62:0/3842962191 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:14.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.821+0000 7ff52ea00640 1 --2- 172.21.15.62:0/3842962191 >> v2:172.21.15.120:3300/0 conn(0x7ff53015c9e0 0x7ff5301607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863586033 0 0) 0x7ff53006bfc0 con 0x7ff5300cf650 2024-03-23T11:19:14.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff514003520 con 0x7ff5300cf650 2024-03-23T11:19:14.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3713190535 0 0) 0x7ff514003520 con 0x7ff5300cf650 2024-03-23T11:19:14.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 >> v2:172.21.15.120:3300/0 conn(0x7ff53015c9e0 msgr2=0x7ff5301607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:14.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 --2- 172.21.15.62:0/3842962191 >> v2:172.21.15.120:3300/0 conn(0x7ff53015c9e0 0x7ff5301607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 >> v1:172.21.15.120:6789/0 conn(0x7ff53015cdc0 legacy=0x7ff5300d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5300ce140 con 0x7ff5300cf650 2024-03-23T11:19:14.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7ff5280019f0 con 0x7ff5300cf650 2024-03-23T11:19:14.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 >> v1:172.21.15.62:6789/0 conn(0x7ff5300cf650 legacy=0x7ff53015d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff514003970 0x7ff514003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5140047c0 0x7ff514004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff514005180 0x7ff51400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5300d2630 con 0x7ff514003970 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51400c020 con 0x7ff514005180 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.822+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51400c1e0 con 0x7ff5140047c0 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.823+0000 7ff52ea00640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff514003970 0x7ff514003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.823+0000 7ff526000640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5140047c0 0x7ff514004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.823+0000 7ff52f400640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff514005180 0x7ff51400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.823+0000 7ff52ea00640 1 -- 172.21.15.62:0/3842962191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5140047c0 msgr2=0x7ff514004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.823+0000 7ff52ea00640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5140047c0 0x7ff514004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.823+0000 7ff52ea00640 1 -- 172.21.15.62:0/3842962191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff514005180 msgr2=0x7ff51400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.823+0000 7ff52ea00640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff514005180 0x7ff51400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.823+0000 7ff52ea00640 1 -- 172.21.15.62:0/3842962191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff514003520 con 0x7ff514003970 2024-03-23T11:19:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.823+0000 7ff52f400640 1 -- 172.21.15.62:0/3842962191 reap_dead start 2024-03-23T11:19:14.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.823+0000 7ff526000640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5140047c0 0x7ff514004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:14.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.825+0000 7ff52ea00640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff514003970 0x7ff514003d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff520009ac0 tx=0x7ff5140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f9f7d89c16c8c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:14.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.825+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52002c6b0 con 0x7ff514003970 2024-03-23T11:19:14.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.825+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52002c810 con 0x7ff514003970 2024-03-23T11:19:14.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.825+0000 7ff52e000640 1 -- 172.21.15.62:0/3842962191 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52002cba0 con 0x7ff514003970 2024-03-23T11:19:14.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.826+0000 7ff535e00640 1 -- 172.21.15.62:0/3842962191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff514003970 msgr2=0x7ff514003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.826+0000 7ff535e00640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff514003970 0x7ff514003d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff520009ac0 tx=0x7ff5140057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.826+0000 7ff535e00640 1 -- 172.21.15.62:0/3842962191 shutdown_connections 2024-03-23T11:19:14.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.826+0000 7ff535e00640 1 --2- 172.21.15.62:0/3842962191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff514003970 0x7ff514003d90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.826+0000 7ff535e00640 1 -- 172.21.15.62:0/3842962191 >> 172.21.15.62:0/3842962191 conn(0x7ff530071570 msgr2=0x7ff530071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:14.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.826+0000 7ff535e00640 1 -- 172.21.15.62:0/3842962191 shutdown_connections 2024-03-23T11:19:14.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.826+0000 7ff535e00640 1 -- 172.21.15.62:0/3842962191 wait complete. 2024-03-23T11:19:14.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.828+0000 7ff535e00640 1 Processor -- start 2024-03-23T11:19:14.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.828+0000 7ff535e00640 1 -- start start 2024-03-23T11:19:14.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.829+0000 7ff535e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff514003970 0x7ff5301612c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.829+0000 7ff535e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5300a9300 0x7ff5300ace50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.829+0000 7ff535e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5300cbb90 0x7ff5300cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.829+0000 7ff535e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5300cc580 con 0x7ff5300a9300 2024-03-23T11:19:14.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.829+0000 7ff535e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5300ad390 con 0x7ff514003970 2024-03-23T11:19:14.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.829+0000 7ff535e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5300a9790 con 0x7ff5300cbb90 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.829+0000 7ff526000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5300a9300 0x7ff5300ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.829+0000 7ff526000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5300a9300 0x7ff5300ace50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52528/0 (socket says 172.21.15.62:52528) 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.829+0000 7ff526000640 1 -- 172.21.15.62:0/2008780496 learned_addr learned my addr 172.21.15.62:0/2008780496 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.830+0000 7ff52f400640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5300cbb90 0x7ff5300cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.830+0000 7ff52ea00640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff514003970 0x7ff5301612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.830+0000 7ff526000640 1 -- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5300cbb90 msgr2=0x7ff5300cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.830+0000 7ff526000640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5300cbb90 0x7ff5300cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.830+0000 7ff526000640 1 -- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff514003970 msgr2=0x7ff5301612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.830+0000 7ff526000640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff514003970 0x7ff5301612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.830+0000 7ff526000640 1 -- 172.21.15.62:0/2008780496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5300a9a70 con 0x7ff5300a9300 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.830+0000 7ff52f400640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5300cbb90 0x7ff5300cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:14.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.830+0000 7ff526000640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5300a9300 0x7ff5300ace50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff51800e410 tx=0x7ff51800e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:14.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.830+0000 7ff527e00640 1 -- 172.21.15.62:0/2008780496 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5180052e0 con 0x7ff5300a9300 2024-03-23T11:19:14.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.831+0000 7ff527e00640 1 -- 172.21.15.62:0/2008780496 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff518005440 con 0x7ff5300a9300 2024-03-23T11:19:14.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.831+0000 7ff527e00640 1 -- 172.21.15.62:0/2008780496 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518003810 con 0x7ff5300a9300 2024-03-23T11:19:14.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.831+0000 7ff535e00640 1 -- 172.21.15.62:0/2008780496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5300a9d50 con 0x7ff5300a9300 2024-03-23T11:19:14.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.831+0000 7ff535e00640 1 -- 172.21.15.62:0/2008780496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5300aa310 con 0x7ff5300a9300 2024-03-23T11:19:14.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.832+0000 7ff535e00640 1 -- 172.21.15.62:0/2008780496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5300aa4a0 con 0x7ff5300a9300 2024-03-23T11:19:14.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.833+0000 7ff527e00640 1 -- 172.21.15.62:0/2008780496 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff518003970 con 0x7ff5300a9300 2024-03-23T11:19:14.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.834+0000 7ff527e00640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff50c06a7a0 0x7ff50c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:14.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.834+0000 7ff527e00640 1 -- 172.21.15.62:0/2008780496 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5180885f0 con 0x7ff5300a9300 2024-03-23T11:19:14.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.834+0000 7ff52ea00640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff50c06a7a0 0x7ff50c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:14.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.835+0000 7ff52ea00640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff50c06a7a0 0x7ff50c06cc60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff520036000 tx=0x7ff520030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:14.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:14.838+0000 7ff527e00640 1 -- 172.21.15.62:0/2008780496 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff518053a20 con 0x7ff5300a9300 2024-03-23T11:19:15.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.077+0000 7ff535e00640 1 -- 172.21.15.62:0/2008780496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff530066c90 con 0x7ff5300a9300 2024-03-23T11:19:15.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.077+0000 7ff527e00640 1 -- 172.21.15.62:0/2008780496 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff518057040 con 0x7ff5300a9300 2024-03-23T11:19:15.078 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:19:15.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 -- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff50c06a7a0 msgr2=0x7ff50c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:15.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff50c06a7a0 0x7ff50c06cc60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff520036000 tx=0x7ff520030000 comp rx=0 tx=0).stop 2024-03-23T11:19:15.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 -- 172.21.15.62:0/2008780496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5300a9300 msgr2=0x7ff5300ace50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:15.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5300a9300 0x7ff5300ace50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff51800e410 tx=0x7ff51800e8d0 comp rx=0 tx=0).stop 2024-03-23T11:19:15.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 -- 172.21.15.62:0/2008780496 shutdown_connections 2024-03-23T11:19:15.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff50c06a7a0 0x7ff50c06cc60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:15.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5300a9300 0x7ff5300ace50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:15.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5300cbb90 0x7ff5300cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:15.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 --2- 172.21.15.62:0/2008780496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff514003970 0x7ff5301612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:15.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 -- 172.21.15.62:0/2008780496 >> 172.21.15.62:0/2008780496 conn(0x7ff530071570 msgr2=0x7ff530072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:15.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 -- 172.21.15.62:0/2008780496 shutdown_connections 2024-03-23T11:19:15.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:15.080+0000 7ff535e00640 1 -- 172.21.15.62:0/2008780496 wait complete. 2024-03-23T11:19:15.101 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:19:15.101 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:19:15.101 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:19:15.101 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:19:15.101 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:19:15.102 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:19:15.102 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:19:15.102 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:19:15.102 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:19:15.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:15.102 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:19:15.154 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:19:15.155 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:19:15.155 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:19:15.155 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:15.155 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:19:15.155 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:15.155 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:15.155 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:15.155 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:19:15.155 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:15.155 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:15.155 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:19:15.156 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:15.156 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:15.156 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:19:15.156 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:15.156 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:19:15.156 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42602sec preferred_lft 42602sec 2024-03-23T11:19:15.156 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:19:15.156 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:15.157 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-23T11:19:15.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:15.157 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:19:15.158 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:19:15.158 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add name ceph-brx type bridge 2024-03-23T11:19:15.158 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr flush dev ceph-brx 2024-03-23T11:19:15.158 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set ceph-brx up 2024-03-23T11:19:15.158 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-23T11:19:15.158 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:19:15.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:15.607 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:19:15.696 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:19:15.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:15.700 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:19:15.761 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:19:15.761 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:19:15.761 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:19:15.761 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:19:15.761 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:19:15.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:15.762 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:19:15.762 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:19:15.762 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:19:15.762 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:19:15.762 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:19:15.762 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:19:16.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:16.510 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:19:16.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:16.572 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list-id 2024-03-23T11:19:16.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:16.628 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:19:16.629 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:19:16.629 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:19:16.629 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-23T11:19:16.629 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:19:16.784 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-23T11:19:16.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:16.784 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:19:16.785 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:19:16.785 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-23T11:19:16.785 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-23T11:19:16.785 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-23T11:19:16.785 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-23T11:19:16.785 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-23T11:19:16.785 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:19:17.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:17.087 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:19:17.087 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:19:17.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:19:17.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set dev brx.0 master ceph-brx 2024-03-23T11:19:17.087 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:19:17.257 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:19:17.258 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:19:17.258 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:19:17.310 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:19:17.310 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:19:17.311 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:19:17.363 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:19:17.468 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:19:17.468 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:19:17.560 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:19:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:19:17.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:17.565 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:19:17.621 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:19:17.621 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:19:17.627 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:19:17.702 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:19:17.702 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:19:17.775 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:19:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:19:17.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:17.779 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:19:17.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:19:17.816+0000 7fc4e723e480 -1 init, newargv = 0x55e41321acd0 newargc=15 2024-03-23T11:19:17.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[44718]: starting ceph client 2024-03-23T11:19:17.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[44718]: starting fuse 2024-03-23T11:19:18.925 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:19:19.034 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:19:19.034 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:19:19.131 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:19:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:19:19.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:19.136 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:19:19.190 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:19:19.191 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:19:19.191 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:19:19.191 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:19:19.191 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:19:19.191 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:19:19.191 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:19:19.191 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:19:19.191 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:19:19.191 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:19:19.191 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:19:19.191 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:19:19.191 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:19:19.192 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:19:19.376 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.44718.asok 2024-03-23T11:19:19.555 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44718.asok 2024-03-23T11:19:19.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:19.555 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44718.asok status 2024-03-23T11:19:19.810 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:19:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:19:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:19:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:19:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:19:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:19:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:19:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "44718", 2024-03-23T11:19:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 4650, 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 4650 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1136440437 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:19:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1136440437 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.4650 192.168.0.1:0/1136440437", 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/1136440437", 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 8, 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 22, 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:19:19.812 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:19:19.826 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:19:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.047+0000 7f5e3da00640 1 Processor -- start 2024-03-23T11:19:20.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.047+0000 7f5e3da00640 1 -- start start 2024-03-23T11:19:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.049+0000 7f5e3da00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5e38143dd0 0x7f5e3815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.049+0000 7f5e3da00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5e38070090 0x7f5e38070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.049+0000 7f5e3da00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e380cd140 con 0x7f5e380b2820 2024-03-23T11:19:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.049+0000 7f5e3da00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e38070a60 con 0x7f5e38143dd0 2024-03-23T11:19:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.049+0000 7f5e3da00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e380cd3e0 con 0x7f5e38070090 2024-03-23T11:19:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.049+0000 7f5e37e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5e38143dd0 0x7f5e3815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.049+0000 7f5e3d000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5e380b2820 0x7f5e380b2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50648/0 (socket says 172.21.15.62:50648) 2024-03-23T11:19:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.049+0000 7f5e3d000640 1 -- 172.21.15.62:0/3765110068 learned_addr learned my addr 172.21.15.62:0/3765110068 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.050+0000 7f5e2f400640 1 --2- 172.21.15.62:0/3765110068 >> v2:172.21.15.120:3300/0 conn(0x7f5e38070090 0x7f5e38070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.050+0000 7f5e37e00640 1 -- 172.21.15.62:0/3765110068 >> v2:172.21.15.120:3300/0 conn(0x7f5e38070090 msgr2=0x7f5e38070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.050+0000 7f5e37e00640 1 --2- 172.21.15.62:0/3765110068 >> v2:172.21.15.120:3300/0 conn(0x7f5e38070090 0x7f5e38070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.050+0000 7f5e37e00640 1 -- 172.21.15.62:0/3765110068 >> v1:172.21.15.62:6789/0 conn(0x7f5e380b2820 legacy=0x7f5e380b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.050+0000 7f5e37e00640 1 -- 172.21.15.62:0/3765110068 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e380cd6c0 con 0x7f5e38143dd0 2024-03-23T11:19:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.050+0000 7f5e37400640 1 -- 172.21.15.62:0/3765110068 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314621399 0 0) 0x7f5e380cd140 con 0x7f5e380b2820 2024-03-23T11:19:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.051+0000 7f5e37e00640 1 --2- 172.21.15.62:0/3765110068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e38143dd0 0x7f5e3815abe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5e38071530 tx=0x7f5e2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b1ca5395efecc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.051+0000 7f5e37400640 1 -- 172.21.15.62:0/3765110068 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28031830 con 0x7f5e38143dd0 2024-03-23T11:19:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.051+0000 7f5e37400640 1 -- 172.21.15.62:0/3765110068 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e28031990 con 0x7f5e38143dd0 2024-03-23T11:19:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.051+0000 7f5e37400640 1 -- 172.21.15.62:0/3765110068 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28031d20 con 0x7f5e38143dd0 2024-03-23T11:19:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.052+0000 7f5e3da00640 1 -- 172.21.15.62:0/3765110068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e38143dd0 msgr2=0x7f5e3815abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.052+0000 7f5e3da00640 1 --2- 172.21.15.62:0/3765110068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e38143dd0 0x7f5e3815abe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5e38071530 tx=0x7f5e2802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:19:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.052+0000 7f5e3da00640 1 -- 172.21.15.62:0/3765110068 shutdown_connections 2024-03-23T11:19:20.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.052+0000 7f5e3da00640 1 --2- 172.21.15.62:0/3765110068 >> v2:172.21.15.120:3300/0 conn(0x7f5e38070090 0x7f5e38070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:20.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.052+0000 7f5e3da00640 1 --2- 172.21.15.62:0/3765110068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e38143dd0 0x7f5e3815abe0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:20.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.052+0000 7f5e3da00640 1 -- 172.21.15.62:0/3765110068 >> 172.21.15.62:0/3765110068 conn(0x7f5e38066050 msgr2=0x7f5e38069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:20.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.052+0000 7f5e3da00640 1 -- 172.21.15.62:0/3765110068 shutdown_connections 2024-03-23T11:19:20.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.052+0000 7f5e3da00640 1 -- 172.21.15.62:0/3765110068 wait complete. 2024-03-23T11:19:20.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.054+0000 7f5e3da00640 1 Processor -- start 2024-03-23T11:19:20.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.054+0000 7f5e3da00640 1 -- start start 2024-03-23T11:19:20.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.055+0000 7f5e3da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e38070090 0x7f5e38139870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:20.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.055+0000 7f5e3da00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e380b2820 0x7f5e3813f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:20.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.055+0000 7f5e3da00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e38143dd0 0x7f5e3813ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:20.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.055+0000 7f5e3da00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380b0400 con 0x7f5e38070090 2024-03-23T11:19:20.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.055+0000 7f5e3da00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380afad0 con 0x7f5e38143dd0 2024-03-23T11:19:20.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.055+0000 7f5e3da00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380afdb0 con 0x7f5e380b2820 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e37e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e38070090 0x7f5e38139870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e37e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e38070090 0x7f5e38139870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43166/0 (socket says 172.21.15.62:43166) 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e37e00640 1 -- 172.21.15.62:0/4168734583 learned_addr learned my addr 172.21.15.62:0/4168734583 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e3d000640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e38143dd0 0x7f5e3813ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e2f400640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e380b2820 0x7f5e3813f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e37e00640 1 -- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e380b2820 msgr2=0x7f5e3813f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e37e00640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e380b2820 0x7f5e3813f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e37e00640 1 -- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e38143dd0 msgr2=0x7f5e3813ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e37e00640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e38143dd0 0x7f5e3813ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e37e00640 1 -- 172.21.15.62:0/4168734583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e28009400 con 0x7f5e38070090 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e2f400640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e380b2820 0x7f5e3813f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:20.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.056+0000 7f5e37e00640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e38070090 0x7f5e38139870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5e28000c00 tx=0x7f5e280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:20.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.057+0000 7f5e34c00640 1 -- 172.21.15.62:0/4168734583 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2803aa60 con 0x7f5e38070090 2024-03-23T11:19:20.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.057+0000 7f5e34c00640 1 -- 172.21.15.62:0/4168734583 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2803abe0 con 0x7f5e38070090 2024-03-23T11:19:20.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.057+0000 7f5e34c00640 1 -- 172.21.15.62:0/4168734583 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e280443f0 con 0x7f5e38070090 2024-03-23T11:19:20.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.057+0000 7f5e3da00640 1 -- 172.21.15.62:0/4168734583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e380b0090 con 0x7f5e38070090 2024-03-23T11:19:20.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.057+0000 7f5e3da00640 1 -- 172.21.15.62:0/4168734583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e380b0890 con 0x7f5e38070090 2024-03-23T11:19:20.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.058+0000 7f5e3da00640 1 -- 172.21.15.62:0/4168734583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e380b0250 con 0x7f5e38070090 2024-03-23T11:19:20.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.059+0000 7f5e34c00640 1 -- 172.21.15.62:0/4168734583 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5e2804d450 con 0x7f5e38070090 2024-03-23T11:19:20.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.060+0000 7f5e34c00640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5e1406a7a0 0x7f5e1406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:20.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.060+0000 7f5e34c00640 1 -- 172.21.15.62:0/4168734583 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e280aaf90 con 0x7f5e38070090 2024-03-23T11:19:20.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.061+0000 7f5e2f400640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5e1406a7a0 0x7f5e1406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:20.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.062+0000 7f5e2f400640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5e1406a7a0 0x7f5e1406cc60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5e3815a930 tx=0x7f5e20008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:20.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.065+0000 7f5e34c00640 1 -- 172.21.15.62:0/4168734583 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e28076430 con 0x7f5e38070090 2024-03-23T11:19:20.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.309+0000 7f5e3da00640 1 -- 172.21.15.62:0/4168734583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5e38066bb0 con 0x7f5e38070090 2024-03-23T11:19:20.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.309+0000 7f5e34c00640 1 -- 172.21.15.62:0/4168734583 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5e28035020 con 0x7f5e38070090 2024-03-23T11:19:20.311 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:19:20.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 -- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5e1406a7a0 msgr2=0x7f5e1406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:20.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5e1406a7a0 0x7f5e1406cc60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5e3815a930 tx=0x7f5e20008040 comp rx=0 tx=0).stop 2024-03-23T11:19:20.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 -- 172.21.15.62:0/4168734583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e38070090 msgr2=0x7f5e38139870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:20.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e38070090 0x7f5e38139870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5e28000c00 tx=0x7f5e280046c0 comp rx=0 tx=0).stop 2024-03-23T11:19:20.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 -- 172.21.15.62:0/4168734583 shutdown_connections 2024-03-23T11:19:20.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5e1406a7a0 0x7f5e1406cc60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:20.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e38143dd0 0x7f5e3813ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:20.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e380b2820 0x7f5e3813f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:20.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 --2- 172.21.15.62:0/4168734583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e38070090 0x7f5e38139870 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:20.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 -- 172.21.15.62:0/4168734583 >> 172.21.15.62:0/4168734583 conn(0x7f5e38066050 msgr2=0x7f5e380b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:20.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 -- 172.21.15.62:0/4168734583 shutdown_connections 2024-03-23T11:19:20.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:20.312+0000 7f5e3da00640 1 -- 172.21.15.62:0/4168734583 wait complete. 2024-03-23T11:19:20.331 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:19:20.331 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:19:20.331 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:19:20.331 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-23T11:19:20.331 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-23T11:19:20.331 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:19:20.331 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:19:20.331 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:19:20.331 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-23T11:19:20.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:20.332 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:19:20.383 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:19:20.383 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:19:20.383 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:19:20.383 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:20.383 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:19:20.383 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42597sec preferred_lft 42597sec 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:19:20.384 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:20.385 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:19:20.385 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:20.385 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:19:20.385 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:20.385 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:19:20.385 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:19:20.385 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:19:20.385 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:20.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:20.386 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:19:20.474 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:19:20.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:20.479 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:19:20.534 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:19:20.534 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:19:20.534 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:19:20.534 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:19:20.534 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:19:20.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:20.535 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:19:20.535 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:19:20.536 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:19:20.536 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:19:20.536 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:19:20.536 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:19:20.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:20.731 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:19:20.783 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:19:20.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:20.785 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list-id 2024-03-23T11:19:20.842 INFO:teuthology.orchestra.run.smithi062.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-23T11:19:20.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:20.843 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:19:20.843 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:19:20.843 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:19:20.843 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-23T11:19:20.843 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:19:20.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:20.994 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether a2:ab:34:bd:ba:d3 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::a0ab:34ff:febd:bad3/64 scope link 2024-03-23T11:19:21.085 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:21.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:21.090 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-23T11:19:21.183 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:19:21.184 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:19:21.184 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:19:21.184 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:21.184 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:19:21.184 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:21.184 INFO:teuthology.orchestra.run.smithi062.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:19:21.184 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether a2:ab:34:bd:ba:d3 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T11:19:21.184 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T11:19:21.184 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:21.184 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::a0ab:34ff:febd:bad3/64 scope link 2024-03-23T11:19:21.184 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:21.188 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-23T11:19:21.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:21.189 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:19:21.189 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:19:21.189 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-23T11:19:21.189 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-23T11:19:21.189 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-23T11:19:21.189 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-23T11:19:21.189 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-23T11:19:21.189 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:19:21.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:21.454 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:19:21.454 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:19:21.454 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.1 up 2024-03-23T11:19:21.454 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set dev brx.1 master ceph-brx 2024-03-23T11:19:21.454 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:19:21.621 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-23T11:19:21.621 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:19:21.621 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-23T11:19:21.671 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-23T11:19:21.672 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:19:21.672 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-23T11:19:21.720 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:19:21.808 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:19:21.808 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:19:21.905 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:19:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:19:21.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:21.910 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:19:21.968 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:19:21.968 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [40] 2024-03-23T11:19:21.968 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-23T11:19:21.973 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:19:22.051 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:19:22.051 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:19:22.119 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:19:22.122 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:19:22.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:22.123 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:19:22.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi062.stderr:2024-03-23T11:19:22.153+0000 7f7586210480 -1 init, newargv = 0x56490a068b40 newargc=15 2024-03-23T11:19:22.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi062.stderr:ceph-fuse[45281]: starting ceph client 2024-03-23T11:19:22.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi062.stderr:ceph-fuse[45281]: starting fuse 2024-03-23T11:19:22.165 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:19:22.166 INFO:teuthology.orchestra.run.smithi062.stdout:43 2024-03-23T11:19:22.166 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40, 43] 2024-03-23T11:19:22.166 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:19:22.166 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:19:22.166 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:19:22.166 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:19:22.166 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:19:22.166 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:19:22.166 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:19:22.166 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:19:22.167 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:19:22.168 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.2")) 2024-03-23T11:19:22.168 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:19:22.364 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.2.45281.asok 2024-03-23T11:19:22.544 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.45281.asok 2024-03-23T11:19:22.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:22.544 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.45281.asok status 2024-03-23T11:19:22.790 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:19:22.790 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:19:22.790 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:19:22.790 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "2", 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "45281", 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 4665, 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:19:22.791 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 4665 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.2:0", 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 4096968495 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.2:0", 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 4096968495 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.4665 192.168.0.2:0/4096968495", 2024-03-23T11:19:22.792 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.2:0/4096968495", 2024-03-23T11:19:22.793 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:19:22.793 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 8, 2024-03-23T11:19:22.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 22, 2024-03-23T11:19:22.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:19:22.793 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:19:22.793 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:19:22.793 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:19:22.805 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:19:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.026+0000 7f4008c00640 1 Processor -- start 2024-03-23T11:19:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.026+0000 7f4008c00640 1 -- start start 2024-03-23T11:19:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4008c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f40040995b0 0x7f40040ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4008c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f40040989d0 0x7f4004098e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4008c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4004099360 con 0x7f4004160240 2024-03-23T11:19:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4008c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40040cf690 con 0x7f40040989d0 2024-03-23T11:19:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4008c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40040cb730 con 0x7f40040995b0 2024-03-23T11:19:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4002800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f40040989d0 0x7f4004098e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4002800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f40040989d0 0x7f4004098e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43176/0 (socket says 172.21.15.62:43176) 2024-03-23T11:19:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4002800640 1 -- 172.21.15.62:0/27987736 learned_addr learned my addr 172.21.15.62:0/27987736 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f3ffbe00640 1 --2- 172.21.15.62:0/27987736 >> v2:172.21.15.120:3300/0 conn(0x7f40040995b0 0x7f40040ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4002800640 1 -- 172.21.15.62:0/27987736 >> v2:172.21.15.120:3300/0 conn(0x7f40040995b0 msgr2=0x7f40040ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4002800640 1 --2- 172.21.15.62:0/27987736 >> v2:172.21.15.120:3300/0 conn(0x7f40040995b0 0x7f40040ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4002800640 1 -- 172.21.15.62:0/27987736 >> v1:172.21.15.120:6789/0 conn(0x7f4004160240 legacy=0x7f400406ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.028+0000 7f4002800640 1 -- 172.21.15.62:0/27987736 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40040cba10 con 0x7f40040989d0 2024-03-23T11:19:23.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.029+0000 7f4002800640 1 --2- 172.21.15.62:0/27987736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40040989d0 0x7f4004098e20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f40040716a0 tx=0x7f3ffc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6906e0a0e0f10584 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:23.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.030+0000 7f4001400640 1 -- 172.21.15.62:0/27987736 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc02c9f0 con 0x7f40040989d0 2024-03-23T11:19:23.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.030+0000 7f4001400640 1 -- 172.21.15.62:0/27987736 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ffc02cb50 con 0x7f40040989d0 2024-03-23T11:19:23.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.030+0000 7f4001400640 1 -- 172.21.15.62:0/27987736 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0343f0 con 0x7f40040989d0 2024-03-23T11:19:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.030+0000 7f4008c00640 1 -- 172.21.15.62:0/27987736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40040989d0 msgr2=0x7f4004098e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.030+0000 7f4008c00640 1 --2- 172.21.15.62:0/27987736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40040989d0 0x7f4004098e20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f40040716a0 tx=0x7f3ffc02be50 comp rx=0 tx=0).stop 2024-03-23T11:19:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.031+0000 7f4008c00640 1 -- 172.21.15.62:0/27987736 shutdown_connections 2024-03-23T11:19:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.031+0000 7f4008c00640 1 --2- 172.21.15.62:0/27987736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40040989d0 0x7f4004098e20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.031+0000 7f4008c00640 1 --2- 172.21.15.62:0/27987736 >> v2:172.21.15.120:3300/0 conn(0x7f40040995b0 0x7f40040ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.031+0000 7f4008c00640 1 -- 172.21.15.62:0/27987736 >> 172.21.15.62:0/27987736 conn(0x7f40040b22c0 msgr2=0x7f40040b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.031+0000 7f4008c00640 1 -- 172.21.15.62:0/27987736 shutdown_connections 2024-03-23T11:19:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.031+0000 7f4008c00640 1 -- 172.21.15.62:0/27987736 wait complete. 2024-03-23T11:19:23.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.032+0000 7f4008c00640 1 Processor -- start 2024-03-23T11:19:23.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.033+0000 7f4008c00640 1 -- start start 2024-03-23T11:19:23.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.033+0000 7f4008c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40040989d0 0x7f400415d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:23.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.033+0000 7f4008c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40040995b0 0x7f40040c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.033+0000 7f4008c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4004160240 0x7f40041f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.033+0000 7f4008c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40040c4a20 con 0x7f40040989d0 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.033+0000 7f4008c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40040c4d00 con 0x7f4004160240 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.033+0000 7f4008c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40040c4fe0 con 0x7f40040995b0 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.034+0000 7f4001e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40040989d0 0x7f400415d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.034+0000 7f4001e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40040989d0 0x7f400415d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43190/0 (socket says 172.21.15.62:43190) 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.034+0000 7f4001e00640 1 -- 172.21.15.62:0/3172566997 learned_addr learned my addr 172.21.15.62:0/3172566997 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.034+0000 7f4002800640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4004160240 0x7f40041f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.034+0000 7f3ffbe00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40040995b0 0x7f40040c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.034+0000 7f4001e00640 1 -- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40040995b0 msgr2=0x7f40040c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.034+0000 7f4001e00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40040995b0 0x7f40040c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.034+0000 7f4001e00640 1 -- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4004160240 msgr2=0x7f40041f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.034+0000 7f4001e00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4004160240 0x7f40041f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.034+0000 7f4001e00640 1 -- 172.21.15.62:0/3172566997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc009b30 con 0x7f40040989d0 2024-03-23T11:19:23.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.034+0000 7f4002800640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4004160240 0x7f40041f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:23.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.035+0000 7f4001e00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40040989d0 0x7f400415d830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400b3e0 tx=0x7f3ff400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:23.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.035+0000 7f3ffa000640 1 -- 172.21.15.62:0/3172566997 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4011830 con 0x7f40040989d0 2024-03-23T11:19:23.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.035+0000 7f3ffa000640 1 -- 172.21.15.62:0/3172566997 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff40119b0 con 0x7f40040989d0 2024-03-23T11:19:23.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.035+0000 7f4008c00640 1 -- 172.21.15.62:0/3172566997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40040c52c0 con 0x7f40040989d0 2024-03-23T11:19:23.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.035+0000 7f3ffa000640 1 -- 172.21.15.62:0/3172566997 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4011d60 con 0x7f40040989d0 2024-03-23T11:19:23.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.035+0000 7f4008c00640 1 -- 172.21.15.62:0/3172566997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40040c8720 con 0x7f40040989d0 2024-03-23T11:19:23.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.036+0000 7f4008c00640 1 -- 172.21.15.62:0/3172566997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40040738e0 con 0x7f40040989d0 2024-03-23T11:19:23.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.037+0000 7f3ffa000640 1 -- 172.21.15.62:0/3172566997 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3ff401a400 con 0x7f40040989d0 2024-03-23T11:19:23.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.038+0000 7f3ffa000640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe006a7a0 0x7f3fe006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:23.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.038+0000 7f3ffa000640 1 -- 172.21.15.62:0/3172566997 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ff4088f40 con 0x7f40040989d0 2024-03-23T11:19:23.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.039+0000 7f3ffbe00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe006a7a0 0x7f3fe006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:23.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.040+0000 7f3ffbe00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe006a7a0 0x7f3fe006cc60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f400415fb90 tx=0x7f3fec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.043+0000 7f3ffa000640 1 -- 172.21.15.62:0/3172566997 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ff40542c0 con 0x7f40040989d0 2024-03-23T11:19:23.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.289+0000 7f4008c00640 1 -- 172.21.15.62:0/3172566997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4004066c90 con 0x7f40040989d0 2024-03-23T11:19:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.290+0000 7f3ffa000640 1 -- 172.21.15.62:0/3172566997 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3ff4015070 con 0x7f40040989d0 2024-03-23T11:19:23.291 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:19:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.292+0000 7f4008c00640 1 -- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe006a7a0 msgr2=0x7f3fe006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.292+0000 7f4008c00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe006a7a0 0x7f3fe006cc60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f400415fb90 tx=0x7f3fec008040 comp rx=0 tx=0).stop 2024-03-23T11:19:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.292+0000 7f4008c00640 1 -- 172.21.15.62:0/3172566997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40040989d0 msgr2=0x7f400415d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.292+0000 7f4008c00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40040989d0 0x7f400415d830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400b3e0 tx=0x7f3ff400b8a0 comp rx=0 tx=0).stop 2024-03-23T11:19:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.292+0000 7f4008c00640 1 -- 172.21.15.62:0/3172566997 shutdown_connections 2024-03-23T11:19:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.292+0000 7f4008c00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe006a7a0 0x7f3fe006cc60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.292+0000 7f4008c00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4004160240 0x7f40041f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.292+0000 7f4008c00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40040995b0 0x7f40040c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.292+0000 7f4008c00640 1 --2- 172.21.15.62:0/3172566997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40040989d0 0x7f400415d830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.292+0000 7f4008c00640 1 -- 172.21.15.62:0/3172566997 >> 172.21.15.62:0/3172566997 conn(0x7f40040b22c0 msgr2=0x7f40040b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.293+0000 7f4008c00640 1 -- 172.21.15.62:0/3172566997 shutdown_connections 2024-03-23T11:19:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:23.293+0000 7f4008c00640 1 -- 172.21.15.62:0/3172566997 wait complete. 2024-03-23T11:19:23.314 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:19:23.314 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:19:23.315 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2024-03-23T11:19:23.315 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-23T11:19:23.315 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T11:19:23.315 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:19:23.315 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:19:23.315 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:19:23.315 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T11:19:23.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:23.315 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2024-03-23T11:19:23.369 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:19:23.369 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:19:23.369 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:19:23.369 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:19:23.370 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:23.371 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:19:23.371 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 42605sec preferred_lft 42605sec 2024-03-23T11:19:23.371 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2024-03-23T11:19:23.371 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:23.372 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-23T11:19:23.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:23.372 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-23T11:19:23.372 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-23T11:19:23.372 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link add name ceph-brx type bridge 2024-03-23T11:19:23.372 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip addr flush dev ceph-brx 2024-03-23T11:19:23.372 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set ceph-brx up 2024-03-23T11:19:23.373 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-23T11:19:23.373 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-23T11:19:23.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:23.827 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:19:23.906 INFO:teuthology.orchestra.run.smithi120.stdout:1 2024-03-23T11:19:23.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:23.910 DEBUG:teuthology.orchestra.run.smithi120:> route 2024-03-23T11:19:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2024-03-23T11:19:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:19:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:19:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:19:23.970 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:19:23.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:23.971 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-23T11:19:23.971 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-23T11:19:23.971 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:19:23.971 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:19:23.972 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:19:23.972 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-23T11:19:24.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:24.695 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2024-03-23T11:19:24.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:24.747 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list-id 2024-03-23T11:19:24.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:24.793 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-23T11:19:24.793 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-23T11:19:24.793 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T11:19:24.793 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-23T11:19:24.793 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-23T11:19:24.928 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-23T11:19:24.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:24.928 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-23T11:19:24.928 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-23T11:19:24.928 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-23T11:19:24.928 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-23T11:19:24.928 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-23T11:19:24.928 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-23T11:19:24.928 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-23T11:19:24.928 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-23T11:19:25.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:25.199 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-23T11:19:25.199 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-23T11:19:25.199 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2024-03-23T11:19:25.199 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set dev brx.0 master ceph-brx 2024-03-23T11:19:25.199 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-23T11:19:25.358 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-23T11:19:25.358 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:19:25.358 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-23T11:19:25.406 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-23T11:19:25.406 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:19:25.406 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-23T11:19:25.502 DEBUG:teuthology.orchestra.run.smithi120:> sudo modprobe fuse 2024-03-23T11:19:25.595 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:19:25.595 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:19:25.683 INFO:teuthology.orchestra.run.smithi120.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:19:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:19:25.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:25.687 DEBUG:teuthology.orchestra.run.smithi120:> ls /sys/fs/fuse/connections 2024-03-23T11:19:25.737 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:19:25.737 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-23T11:19:25.741 DEBUG:teuthology.orchestra.run.smithi120:> sudo modprobe fuse 2024-03-23T11:19:25.813 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:19:25.813 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:19:25.880 INFO:teuthology.orchestra.run.smithi120.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:19:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:19:25.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:25.883 DEBUG:teuthology.orchestra.run.smithi120:> ls /sys/fs/fuse/connections 2024-03-23T11:19:25.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi120.stderr:2024-03-23T11:19:25.918+0000 7fdac8244480 -1 init, newargv = 0x55abe1b41af0 newargc=15 2024-03-23T11:19:25.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi120.stderr:ceph-fuse[42559]: starting ceph client 2024-03-23T11:19:25.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi120.stderr:ceph-fuse[42559]: starting fuse 2024-03-23T11:19:27.026 DEBUG:teuthology.orchestra.run.smithi120:> sudo modprobe fuse 2024-03-23T11:19:27.126 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:19:27.126 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:19:27.218 INFO:teuthology.orchestra.run.smithi120.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:19:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:19:27.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:27.223 DEBUG:teuthology.orchestra.run.smithi120:> ls /sys/fs/fuse/connections 2024-03-23T11:19:27.278 INFO:teuthology.orchestra.run.smithi120.stdout:40 2024-03-23T11:19:27.278 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:19:27.278 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:19:27.278 DEBUG:teuthology.orchestra.run.smithi120:> import glob 2024-03-23T11:19:27.278 DEBUG:teuthology.orchestra.run.smithi120:> import re 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> import os 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> import subprocess 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> def _find_admin_socket(client_name): 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> files = glob.glob(asok_path) 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> # Given a non-glob path, it better be there 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> if "*" not in asok_path: 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> assert(len(files) == 1) 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> return files[0] 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-23T11:19:27.279 DEBUG:teuthology.orchestra.run.smithi120:> for f in files: 2024-03-23T11:19:27.280 DEBUG:teuthology.orchestra.run.smithi120:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:19:27.280 DEBUG:teuthology.orchestra.run.smithi120:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:19:27.280 DEBUG:teuthology.orchestra.run.smithi120:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:19:27.280 DEBUG:teuthology.orchestra.run.smithi120:> contents = proc_f.read() 2024-03-23T11:19:27.280 DEBUG:teuthology.orchestra.run.smithi120:> if mountpoint in contents: 2024-03-23T11:19:27.280 DEBUG:teuthology.orchestra.run.smithi120:> return f 2024-03-23T11:19:27.280 DEBUG:teuthology.orchestra.run.smithi120:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:19:27.280 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-23T11:19:27.280 DEBUG:teuthology.orchestra.run.smithi120:> print(_find_admin_socket("client.1")) 2024-03-23T11:19:27.280 DEBUG:teuthology.orchestra.run.smithi120:> ' 2024-03-23T11:19:27.453 INFO:teuthology.orchestra.run.smithi120.stdout:/var/run/ceph/ceph-client.1.42559.asok 2024-03-23T11:19:27.633 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42559.asok 2024-03-23T11:19:27.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:27.633 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42559.asok status 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout: "metadata": { 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout: "entity_id": "1", 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout: "hostname": "smithi120", 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout: "pid": "42559", 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout: "root": "/" 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout: "dentry_count": 0, 2024-03-23T11:19:27.873 INFO:teuthology.orchestra.run.smithi120.stdout: "dentry_pinned_count": 0, 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "id": 4648, 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "inst": { 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "name": { 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "client", 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "num": 4648 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": { 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "v1", 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "nonce": 988069672 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": { 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "v1", 2024-03-23T11:19:27.874 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:19:27.875 INFO:teuthology.orchestra.run.smithi120.stdout: "nonce": 988069672 2024-03-23T11:19:27.875 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-23T11:19:27.875 INFO:teuthology.orchestra.run.smithi120.stdout: "inst_str": "client.4648 192.168.0.1:0/988069672", 2024-03-23T11:19:27.875 INFO:teuthology.orchestra.run.smithi120.stdout: "addr_str": "192.168.0.1:0/988069672", 2024-03-23T11:19:27.875 INFO:teuthology.orchestra.run.smithi120.stdout: "inode_count": 1, 2024-03-23T11:19:27.875 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_epoch": 8, 2024-03-23T11:19:27.875 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_epoch": 22, 2024-03-23T11:19:27.875 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:19:27.875 INFO:teuthology.orchestra.run.smithi120.stdout: "blocklisted": false, 2024-03-23T11:19:27.875 INFO:teuthology.orchestra.run.smithi120.stdout: "fs_name": "cephfs" 2024-03-23T11:19:27.875 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-23T11:19:27.887 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:19:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.112+0000 7f54ace00640 1 Processor -- start 2024-03-23T11:19:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.112+0000 7f54ace00640 1 -- start start 2024-03-23T11:19:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.114+0000 7f54ace00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f54a8143dd0 0x7f54a815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.114+0000 7f54ace00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b2f30 0x7f54a8070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.114+0000 7f54ace00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a80cd140 con 0x7f54a8070090 2024-03-23T11:19:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.114+0000 7f54ace00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a8070a60 con 0x7f54a8143dd0 2024-03-23T11:19:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.114+0000 7f54ace00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a80cd3e0 con 0x7f54a80b2f30 2024-03-23T11:19:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.114+0000 7f549ea00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f54a8070090 0x7f54a80b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53222/0 (socket says 172.21.15.62:53222) 2024-03-23T11:19:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.114+0000 7f549ea00640 1 -- 172.21.15.62:0/3163016420 learned_addr learned my addr 172.21.15.62:0/3163016420 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a7400640 1 --2- 172.21.15.62:0/3163016420 >> v2:172.21.15.120:3300/0 conn(0x7f54a8143dd0 0x7f54a815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a7e00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b2f30 0x7f54a8070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278257888 0 0) 0x7f54a80cd140 con 0x7f54a8070090 2024-03-23T11:19:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f548c003520 con 0x7f54a8070090 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1227219577 0 0) 0x7f548c003520 con 0x7f54a8070090 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b2f30 msgr2=0x7f54a8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a6a00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b2f30 0x7f54a8070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 >> v2:172.21.15.120:3300/0 conn(0x7f54a8143dd0 msgr2=0x7f54a815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a6a00640 1 --2- 172.21.15.62:0/3163016420 >> v2:172.21.15.120:3300/0 conn(0x7f54a8143dd0 0x7f54a815abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a80cd6c0 con 0x7f54a8070090 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f5490001670 con 0x7f54a8070090 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 >> v1:172.21.15.62:6789/0 conn(0x7f54a8070090 legacy=0x7f54a80b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.115+0000 7f54a6a00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b2f30 0x7f548c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f54a6a00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c003ef0 0x7f548c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f54a6a00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c004d00 0x7f548c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a8070a60 con 0x7f548c003ef0 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a80cd3e0 con 0x7f548c004d00 2024-03-23T11:19:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548c00b980 con 0x7f54a80b2f30 2024-03-23T11:19:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f549ea00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c003ef0 0x7f548c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f54a7400640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b2f30 0x7f548c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f54a7e00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c004d00 0x7f548c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f549ea00640 1 -- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b2f30 msgr2=0x7f548c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f549ea00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b2f30 0x7f548c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f549ea00640 1 -- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c004d00 msgr2=0x7f548c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f549ea00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c004d00 0x7f548c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f549ea00640 1 -- 172.21.15.62:0/3163016420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c003520 con 0x7f548c003ef0 2024-03-23T11:19:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f54a7400640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a80b2f30 0x7f548c003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.116+0000 7f54a7e00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c004d00 0x7f548c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:28.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.118+0000 7f549ea00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c003ef0 0x7f548c0047c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f549000c290 tx=0x7f548c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b280ff5b9f2510 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:28.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.118+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490005ad0 con 0x7f548c003ef0 2024-03-23T11:19:28.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.118+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5490005c50 con 0x7f548c003ef0 2024-03-23T11:19:28.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.118+0000 7f54a6a00640 1 -- 172.21.15.62:0/3163016420 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549002c8c0 con 0x7f548c003ef0 2024-03-23T11:19:28.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.118+0000 7f54ace00640 1 -- 172.21.15.62:0/3163016420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c003ef0 msgr2=0x7f548c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:28.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.118+0000 7f54ace00640 1 --2- 172.21.15.62:0/3163016420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c003ef0 0x7f548c0047c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f549000c290 tx=0x7f548c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.118+0000 7f54a7e00640 1 -- 172.21.15.62:0/3163016420 reap_dead start 2024-03-23T11:19:28.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.119+0000 7f54ace00640 1 -- 172.21.15.62:0/3163016420 shutdown_connections 2024-03-23T11:19:28.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.119+0000 7f54ace00640 1 -- 172.21.15.62:0/3163016420 >> 172.21.15.62:0/3163016420 conn(0x7f54a8066050 msgr2=0x7f54a8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:28.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.119+0000 7f54ace00640 1 -- 172.21.15.62:0/3163016420 shutdown_connections 2024-03-23T11:19:28.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.119+0000 7f54ace00640 1 -- 172.21.15.62:0/3163016420 wait complete. 2024-03-23T11:19:28.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.121+0000 7f54ace00640 1 Processor -- start 2024-03-23T11:19:28.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.121+0000 7f54ace00640 1 -- start start 2024-03-23T11:19:28.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.122+0000 7f54ace00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c003ef0 0x7f54a8159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:28.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.122+0000 7f54ace00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a80ac250 0x7f54a806cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:28.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.122+0000 7f54ace00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a806d180 0x7f54a813e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:28.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.122+0000 7f54ace00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a806db20 con 0x7f54a80ac250 2024-03-23T11:19:28.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.122+0000 7f54ace00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a806d610 con 0x7f548c003ef0 2024-03-23T11:19:28.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.122+0000 7f54ace00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a815a0c0 con 0x7f54a806d180 2024-03-23T11:19:28.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.122+0000 7f549ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a80ac250 0x7f54a806cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:28.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.122+0000 7f549ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a80ac250 0x7f54a806cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50136/0 (socket says 172.21.15.62:50136) 2024-03-23T11:19:28.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.122+0000 7f549ea00640 1 -- 172.21.15.62:0/3673467613 learned_addr learned my addr 172.21.15.62:0/3673467613 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.122+0000 7f54a7400640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c003ef0 0x7f54a8159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.123+0000 7f54a7e00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a806d180 0x7f54a813e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.123+0000 7f549ea00640 1 -- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a806d180 msgr2=0x7f54a813e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.123+0000 7f549ea00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a806d180 0x7f54a813e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.123+0000 7f549ea00640 1 -- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c003ef0 msgr2=0x7f54a8159b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.123+0000 7f549ea00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c003ef0 0x7f54a8159b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.123+0000 7f549ea00640 1 -- 172.21.15.62:0/3673467613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a813ddf0 con 0x7f54a80ac250 2024-03-23T11:19:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.123+0000 7f54a7e00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a806d180 0x7f54a813e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.123+0000 7f549ea00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a80ac250 0x7f54a806cc40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5490001930 tx=0x7f5490000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.123+0000 7f549fe00640 1 -- 172.21.15.62:0/3673467613 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490039070 con 0x7f54a80ac250 2024-03-23T11:19:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.123+0000 7f549fe00640 1 -- 172.21.15.62:0/3673467613 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549002ccf0 con 0x7f54a80ac250 2024-03-23T11:19:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.123+0000 7f549fe00640 1 -- 172.21.15.62:0/3673467613 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54900072b0 con 0x7f54a80ac250 2024-03-23T11:19:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.124+0000 7f54ace00640 1 -- 172.21.15.62:0/3673467613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a80ac6e0 con 0x7f54a80ac250 2024-03-23T11:19:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.124+0000 7f54ace00640 1 -- 172.21.15.62:0/3673467613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a80acca0 con 0x7f54a80ac250 2024-03-23T11:19:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.125+0000 7f54ace00640 1 -- 172.21.15.62:0/3673467613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a813df80 con 0x7f54a80ac250 2024-03-23T11:19:28.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.126+0000 7f549fe00640 1 -- 172.21.15.62:0/3673467613 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5490032070 con 0x7f54a80ac250 2024-03-23T11:19:28.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.127+0000 7f549fe00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548406a7a0 0x7f548406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:28.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.127+0000 7f549fe00640 1 -- 172.21.15.62:0/3673467613 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f549003c040 con 0x7f54a80ac250 2024-03-23T11:19:28.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.127+0000 7f54a7400640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548406a7a0 0x7f548406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:28.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.128+0000 7f54a7400640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548406a7a0 0x7f548406cc60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f54a813ae00 tx=0x7f549800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:28.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.132+0000 7f549fe00640 1 -- 172.21.15.62:0/3673467613 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54900754b0 con 0x7f54a80ac250 2024-03-23T11:19:28.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.379+0000 7f54ace00640 1 -- 172.21.15.62:0/3673467613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f54a8066bb0 con 0x7f54a80ac250 2024-03-23T11:19:28.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.380+0000 7f549fe00640 1 -- 172.21.15.62:0/3673467613 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f549002ce60 con 0x7f54a80ac250 2024-03-23T11:19:28.381 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.382+0000 7f54ace00640 1 -- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548406a7a0 msgr2=0x7f548406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.382+0000 7f54ace00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548406a7a0 0x7f548406cc60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f54a813ae00 tx=0x7f549800b040 comp rx=0 tx=0).stop 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.382+0000 7f54ace00640 1 -- 172.21.15.62:0/3673467613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a80ac250 msgr2=0x7f54a806cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.382+0000 7f54ace00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a80ac250 0x7f54a806cc40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5490001930 tx=0x7f5490000fe0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.383+0000 7f54ace00640 1 -- 172.21.15.62:0/3673467613 shutdown_connections 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.383+0000 7f54ace00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548406a7a0 0x7f548406cc60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.383+0000 7f54ace00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a80ac250 0x7f54a806cc40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.383+0000 7f54ace00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54a806d180 0x7f54a813e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.383+0000 7f54ace00640 1 --2- 172.21.15.62:0/3673467613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f548c003ef0 0x7f54a8159b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.383+0000 7f54ace00640 1 -- 172.21.15.62:0/3673467613 >> 172.21.15.62:0/3673467613 conn(0x7f54a8066050 msgr2=0x7f54a80ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.383+0000 7f54ace00640 1 -- 172.21.15.62:0/3673467613 shutdown_connections 2024-03-23T11:19:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:28.383+0000 7f54ace00640 1 -- 172.21.15.62:0/3673467613 wait complete. 2024-03-23T11:19:28.403 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:19:28.403 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:19:28.403 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2024-03-23T11:19:28.403 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-23T11:19:28.403 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-23T11:19:28.403 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:19:28.403 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:19:28.403 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:19:28.403 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-23T11:19:28.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:28.404 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2024-03-23T11:19:28.459 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:19:28.459 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:19:28.459 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:19:28.460 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 42599sec preferred_lft 42599sec 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 66:a5:06:fb:03:aa brd ff:ff:ff:ff:ff:ff 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::dcb9:8cff:fe67:3ff5/64 scope link 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 66:a5:06:fb:03:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::64a5:6ff:fefb:3aa/64 scope link 2024-03-23T11:19:28.461 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:28.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:28.462 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:19:28.552 INFO:teuthology.orchestra.run.smithi120.stdout:1 2024-03-23T11:19:28.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:28.556 DEBUG:teuthology.orchestra.run.smithi120:> route 2024-03-23T11:19:28.608 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2024-03-23T11:19:28.608 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:19:28.608 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:19:28.608 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:19:28.608 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:19:28.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:28.609 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-23T11:19:28.609 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-23T11:19:28.609 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:19:28.609 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:19:28.609 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:19:28.609 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-23T11:19:28.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:28.782 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2024-03-23T11:19:28.832 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T11:19:28.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:28.833 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list-id 2024-03-23T11:19:28.884 INFO:teuthology.orchestra.run.smithi120.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-23T11:19:28.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:28.885 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-23T11:19:28.885 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-23T11:19:28.885 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T11:19:28.885 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-23T11:19:28.885 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-23T11:19:29.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:29.023 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-23T11:19:29.111 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:19:29.111 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:19:29.111 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:19:29.111 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:29.111 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2024-03-23T11:19:29.111 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:29.111 INFO:teuthology.orchestra.run.smithi120.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:19:29.111 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether d6:a9:73:35:b4:39 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T11:19:29.112 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T11:19:29.112 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:29.112 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::d4a9:73ff:fe35:b439/64 scope link 2024-03-23T11:19:29.112 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:29.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:29.115 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-23T11:19:29.201 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:19:29.201 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:19:29.201 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:19:29.201 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:29.201 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2024-03-23T11:19:29.201 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:29.202 INFO:teuthology.orchestra.run.smithi120.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:19:29.202 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether d6:a9:73:35:b4:39 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T11:19:29.202 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T11:19:29.202 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:29.202 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::d4a9:73ff:fe35:b439/64 scope link 2024-03-23T11:19:29.202 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:19:29.206 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-23T11:19:29.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:29.206 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-23T11:19:29.206 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-23T11:19:29.206 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-23T11:19:29.206 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-23T11:19:29.206 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-23T11:19:29.206 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-23T11:19:29.206 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-23T11:19:29.206 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-23T11:19:29.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:29.456 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-23T11:19:29.456 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-23T11:19:29.456 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.1 up 2024-03-23T11:19:29.456 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set dev brx.1 master ceph-brx 2024-03-23T11:19:29.456 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-23T11:19:29.619 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-23T11:19:29.619 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:19:29.619 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-23T11:19:29.663 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-23T11:19:29.663 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:19:29.663 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-23T11:19:29.757 DEBUG:teuthology.orchestra.run.smithi120:> sudo modprobe fuse 2024-03-23T11:19:29.851 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:19:29.851 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:19:29.940 INFO:teuthology.orchestra.run.smithi120.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:19:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:19:29.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:29.944 DEBUG:teuthology.orchestra.run.smithi120:> ls /sys/fs/fuse/connections 2024-03-23T11:19:29.993 INFO:teuthology.orchestra.run.smithi120.stdout:40 2024-03-23T11:19:29.994 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [40] 2024-03-23T11:19:29.994 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-23T11:19:30.040 DEBUG:teuthology.orchestra.run.smithi120:> sudo modprobe fuse 2024-03-23T11:19:30.118 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:19:30.118 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:19:30.192 INFO:teuthology.orchestra.run.smithi120.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:19:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:19:30.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:30.196 DEBUG:teuthology.orchestra.run.smithi120:> ls /sys/fs/fuse/connections 2024-03-23T11:19:30.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi120.stderr:2024-03-23T11:19:30.228+0000 7fd53a23f480 -1 init, newargv = 0x563ab1d36d30 newargc=15 2024-03-23T11:19:30.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi120.stderr:ceph-fuse[43080]: starting ceph client 2024-03-23T11:19:30.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi120.stderr:ceph-fuse[43080]: starting fuse 2024-03-23T11:19:30.240 INFO:teuthology.orchestra.run.smithi120.stdout:40 2024-03-23T11:19:30.240 INFO:teuthology.orchestra.run.smithi120.stdout:43 2024-03-23T11:19:30.241 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40, 43] 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> import glob 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> import re 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> import os 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> import subprocess 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> def _find_admin_socket(client_name): 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> files = glob.glob(asok_path) 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-23T11:19:30.241 DEBUG:teuthology.orchestra.run.smithi120:> # Given a non-glob path, it better be there 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> if "*" not in asok_path: 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> assert(len(files) == 1) 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> return files[0] 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> for f in files: 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> contents = proc_f.read() 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> if mountpoint in contents: 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> return f 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> print(_find_admin_socket("client.3")) 2024-03-23T11:19:30.242 DEBUG:teuthology.orchestra.run.smithi120:> ' 2024-03-23T11:19:30.430 INFO:teuthology.orchestra.run.smithi120.stdout:/var/run/ceph/ceph-client.3.43080.asok 2024-03-23T11:19:30.611 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.43080.asok 2024-03-23T11:19:30.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:30.611 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.43080.asok status 2024-03-23T11:19:30.844 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-23T11:19:30.844 INFO:teuthology.orchestra.run.smithi120.stdout: "metadata": { 2024-03-23T11:19:30.844 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:19:30.844 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:19:30.844 INFO:teuthology.orchestra.run.smithi120.stdout: "entity_id": "3", 2024-03-23T11:19:30.844 INFO:teuthology.orchestra.run.smithi120.stdout: "hostname": "smithi120", 2024-03-23T11:19:30.844 INFO:teuthology.orchestra.run.smithi120.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "pid": "43080", 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "root": "/" 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "dentry_count": 0, 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "dentry_pinned_count": 0, 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "id": 4660, 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "inst": { 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "name": { 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "client", 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "num": 4660 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": { 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "v1", 2024-03-23T11:19:30.845 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": "192.168.0.2:0", 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "nonce": 3452253489 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": { 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "v1", 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": "192.168.0.2:0", 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "nonce": 3452253489 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "inst_str": "client.4660 192.168.0.2:0/3452253489", 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "addr_str": "192.168.0.2:0/3452253489", 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "inode_count": 1, 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_epoch": 8, 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_epoch": 22, 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:19:30.846 INFO:teuthology.orchestra.run.smithi120.stdout: "blocklisted": false, 2024-03-23T11:19:30.847 INFO:teuthology.orchestra.run.smithi120.stdout: "fs_name": "cephfs" 2024-03-23T11:19:30.847 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-23T11:19:30.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:30.859 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:19:30.859 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:19:30.920 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:19:30.920 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:19:30.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:30.921 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:19:31.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:31.026 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:19:31.026 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-23T11:19:31.084 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:19:31.084 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-23T11:19:31.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:31.084 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-23T11:19:31.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:31.192 DEBUG:teuthology.orchestra.run.smithi120:> stat --file-system '--printf=%T 2024-03-23T11:19:31.192 DEBUG:teuthology.orchestra.run.smithi120:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-23T11:19:31.251 INFO:teuthology.orchestra.run.smithi120.stdout:fuseblk 2024-03-23T11:19:31.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-23T11:19:31.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:31.251 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-23T11:19:31.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:31.347 DEBUG:teuthology.orchestra.run.smithi120:> stat --file-system '--printf=%T 2024-03-23T11:19:31.348 DEBUG:teuthology.orchestra.run.smithi120:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-23T11:19:31.403 INFO:teuthology.orchestra.run.smithi120.stdout:fuseblk 2024-03-23T11:19:31.403 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-23T11:19:31.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:31.403 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-23T11:19:31.492 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-23T11:19:31.525 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:31.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.756+0000 7f7c48a00640 1 Processor -- start 2024-03-23T11:19:31.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.756+0000 7f7c48a00640 1 -- start start 2024-03-23T11:19:31.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.758+0000 7f7c48a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c44143dd0 0x7f7c4415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:31.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.758+0000 7f7c48a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c440cd140 con 0x7f7c440b2f30 2024-03-23T11:19:31.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.758+0000 7f7c48a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c440cd3e0 con 0x7f7c44070090 2024-03-23T11:19:31.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.758+0000 7f7c48a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c440cd6c0 con 0x7f7c44143dd0 2024-03-23T11:19:31.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.758+0000 7f7c43200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7c440b2f30 0x7f7c440ce480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53230/0 (socket says 172.21.15.62:53230) 2024-03-23T11:19:31.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.758+0000 7f7c43200640 1 -- 172.21.15.62:0/1540285098 learned_addr learned my addr 172.21.15.62:0/1540285098 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:31.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.758+0000 7f7c42800640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c44143dd0 0x7f7c4415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:31.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.758+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043643348 0 0) 0x7f7c440cd140 con 0x7f7c440b2f30 2024-03-23T11:19:31.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.758+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c28003520 con 0x7f7c440b2f30 2024-03-23T11:19:31.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2541098758 0 0) 0x7f7c28003520 con 0x7f7c440b2f30 2024-03-23T11:19:31.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c44143dd0 msgr2=0x7f7c4415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:31.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c44143dd0 0x7f7c4415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 >> v1:172.21.15.120:6789/0 conn(0x7f7c44070090 legacy=0x7f7c440b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c440cd9a0 con 0x7f7c440b2f30 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7c3c0019f0 con 0x7f7c440b2f30 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 >> v1:172.21.15.62:6789/0 conn(0x7f7c440b2f30 legacy=0x7f7c440ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c280039a0 0x7f7c28003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c44143dd0 0x7f7c280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 0x7f7c28005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c440cd6c0 con 0x7f7c28004d00 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2800ba30 con 0x7f7c280039a0 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2800bbf0 con 0x7f7c44143dd0 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.759+0000 7f7c43200640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 0x7f7c28005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.760+0000 7f7c42800640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c280039a0 0x7f7c28003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:31.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.760+0000 7f7c3be00640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c44143dd0 0x7f7c280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:31.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.760+0000 7f7c43200640 1 -- 172.21.15.62:0/1540285098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c44143dd0 msgr2=0x7f7c280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:31.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.760+0000 7f7c43200640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c44143dd0 0x7f7c280047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:31.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.760+0000 7f7c43200640 1 -- 172.21.15.62:0/1540285098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c280039a0 msgr2=0x7f7c28003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:31.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.760+0000 7f7c43200640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c280039a0 0x7f7c28003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:31.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.760+0000 7f7c43200640 1 -- 172.21.15.62:0/1540285098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c28003520 con 0x7f7c28004d00 2024-03-23T11:19:31.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.761+0000 7f7c43200640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 0x7f7c28005180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7c440b3350 tx=0x7f7c280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ff9d4441614dab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:31.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.761+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c0072d0 con 0x7f7c28004d00 2024-03-23T11:19:31.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.761+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3c02f070 con 0x7f7c28004d00 2024-03-23T11:19:31.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.761+0000 7f7c41e00640 1 -- 172.21.15.62:0/1540285098 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c007690 con 0x7f7c28004d00 2024-03-23T11:19:31.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.762+0000 7f7c48a00640 1 -- 172.21.15.62:0/1540285098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 msgr2=0x7f7c28005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:31.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.762+0000 7f7c48a00640 1 --2- 172.21.15.62:0/1540285098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 0x7f7c28005180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7c440b3350 tx=0x7f7c280057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:31.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.762+0000 7f7c43200640 1 -- 172.21.15.62:0/1540285098 reap_dead start 2024-03-23T11:19:31.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.762+0000 7f7c48a00640 1 -- 172.21.15.62:0/1540285098 shutdown_connections 2024-03-23T11:19:31.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.762+0000 7f7c48a00640 1 -- 172.21.15.62:0/1540285098 >> 172.21.15.62:0/1540285098 conn(0x7f7c44066050 msgr2=0x7f7c44069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:31.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.762+0000 7f7c48a00640 1 -- 172.21.15.62:0/1540285098 shutdown_connections 2024-03-23T11:19:31.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.762+0000 7f7c48a00640 1 -- 172.21.15.62:0/1540285098 wait complete. 2024-03-23T11:19:31.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.764+0000 7f7c48a00640 1 Processor -- start 2024-03-23T11:19:31.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.764+0000 7f7c48a00640 1 -- start start 2024-03-23T11:19:31.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.765+0000 7f7c48a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 0x7f7c44159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:31.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.765+0000 7f7c48a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c4413af80 0x7f7c44159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:31.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.765+0000 7f7c48a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4413c570 0x7f7c4413f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:31.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.765+0000 7f7c48a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4413dbe0 con 0x7f7c28004d00 2024-03-23T11:19:31.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.765+0000 7f7c48a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4413b4d0 con 0x7f7c4413c570 2024-03-23T11:19:31.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.765+0000 7f7c48a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4413b7b0 con 0x7f7c4413af80 2024-03-23T11:19:31.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.765+0000 7f7c42800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 0x7f7c44159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:31.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.766+0000 7f7c42800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 0x7f7c44159c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50148/0 (socket says 172.21.15.62:50148) 2024-03-23T11:19:31.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.766+0000 7f7c42800640 1 -- 172.21.15.62:0/2556461407 learned_addr learned my addr 172.21.15.62:0/2556461407 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.766+0000 7f7c3be00640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c4413af80 0x7f7c44159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.766+0000 7f7c43200640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4413c570 0x7f7c4413f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.766+0000 7f7c42800640 1 -- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c4413af80 msgr2=0x7f7c44159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.766+0000 7f7c42800640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c4413af80 0x7f7c44159410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.766+0000 7f7c42800640 1 -- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4413c570 msgr2=0x7f7c4413f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.766+0000 7f7c42800640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4413c570 0x7f7c4413f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.766+0000 7f7c42800640 1 -- 172.21.15.62:0/2556461407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4413ba90 con 0x7f7c28004d00 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.766+0000 7f7c43200640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4413c570 0x7f7c4413f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.766+0000 7f7c42800640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 0x7f7c44159c60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7c34011600 tx=0x7f7c34011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.767+0000 7f7c3aa00640 1 -- 172.21.15.62:0/2556461407 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3400f920 con 0x7f7c28004d00 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.767+0000 7f7c3aa00640 1 -- 172.21.15.62:0/2556461407 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3400fa80 con 0x7f7c28004d00 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.767+0000 7f7c3aa00640 1 -- 172.21.15.62:0/2556461407 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3400fdf0 con 0x7f7c28004d00 2024-03-23T11:19:31.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.767+0000 7f7c48a00640 1 -- 172.21.15.62:0/2556461407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4413bd70 con 0x7f7c28004d00 2024-03-23T11:19:31.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.767+0000 7f7c48a00640 1 -- 172.21.15.62:0/2556461407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4415a3d0 con 0x7f7c28004d00 2024-03-23T11:19:31.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.768+0000 7f7c48a00640 1 -- 172.21.15.62:0/2556461407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c44066bb0 con 0x7f7c28004d00 2024-03-23T11:19:31.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.769+0000 7f7c3aa00640 1 -- 172.21.15.62:0/2556461407 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7c34015460 con 0x7f7c28004d00 2024-03-23T11:19:31.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.770+0000 7f7c3aa00640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c2006a7d0 0x7f7c2006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:31.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.770+0000 7f7c3aa00640 1 -- 172.21.15.62:0/2556461407 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c340892f0 con 0x7f7c28004d00 2024-03-23T11:19:31.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.770+0000 7f7c3be00640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c2006a7d0 0x7f7c2006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:31.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.772+0000 7f7c3be00640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c2006a7d0 0x7f7c2006cc90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7c440ccbf0 tx=0x7f7c2c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:31.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.774+0000 7f7c3aa00640 1 -- 172.21.15.62:0/2556461407 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7c34054670 con 0x7f7c28004d00 2024-03-23T11:19:31.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.981+0000 7f7c48a00640 1 -- 172.21.15.62:0/2556461407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c4406e0a0 con 0x7f7c28004d00 2024-03-23T11:19:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.983+0000 7f7c3aa00640 1 -- 172.21.15.62:0/2556461407 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f7c34057c90 con 0x7f7c28004d00 2024-03-23T11:19:31.984 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:31.984 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:57.238201+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.986+0000 7f7c48a00640 1 -- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c2006a7d0 msgr2=0x7f7c2006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.986+0000 7f7c48a00640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c2006a7d0 0x7f7c2006cc90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7c440ccbf0 tx=0x7f7c2c009040 comp rx=0 tx=0).stop 2024-03-23T11:19:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.986+0000 7f7c48a00640 1 -- 172.21.15.62:0/2556461407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 msgr2=0x7f7c44159c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.986+0000 7f7c48a00640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 0x7f7c44159c60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7c34011600 tx=0x7f7c34011ac0 comp rx=0 tx=0).stop 2024-03-23T11:19:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.986+0000 7f7c48a00640 1 -- 172.21.15.62:0/2556461407 shutdown_connections 2024-03-23T11:19:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.986+0000 7f7c48a00640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c2006a7d0 0x7f7c2006cc90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.986+0000 7f7c48a00640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c4413c570 0x7f7c4413f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.986+0000 7f7c48a00640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c4413af80 0x7f7c44159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.986+0000 7f7c48a00640 1 --2- 172.21.15.62:0/2556461407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c28004d00 0x7f7c44159c60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.986+0000 7f7c48a00640 1 -- 172.21.15.62:0/2556461407 >> 172.21.15.62:0/2556461407 conn(0x7f7c44066050 msgr2=0x7f7c44069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.986+0000 7f7c48a00640 1 -- 172.21.15.62:0/2556461407 shutdown_connections 2024-03-23T11:19:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:31.987+0000 7f7c48a00640 1 -- 172.21.15.62:0/2556461407 wait complete. 2024-03-23T11:19:32.004 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:32.005 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.222+0000 7fc466000640 1 Processor -- start 2024-03-23T11:19:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.223+0000 7fc466000640 1 -- start start 2024-03-23T11:19:32.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.224+0000 7fc466000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46006c300 0x7fc4600ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.224+0000 7fc466000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc46006c6e0 0x7fc46006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.224+0000 7fc466000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4600cbc60 con 0x7fc4600cafa0 2024-03-23T11:19:32.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.224+0000 7fc466000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4600cbf40 con 0x7fc46006c6e0 2024-03-23T11:19:32.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.224+0000 7fc466000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4600cc220 con 0x7fc46006c300 2024-03-23T11:19:32.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.224+0000 7fc45ea00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc4600cafa0 0x7fc4600cb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53246/0 (socket says 172.21.15.62:53246) 2024-03-23T11:19:32.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.224+0000 7fc45ea00640 1 -- 172.21.15.62:0/1166989888 learned_addr learned my addr 172.21.15.62:0/1166989888 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:32.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45f400640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46006c300 0x7fc4600ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45fe00640 1 --2- 172.21.15.62:0/1166989888 >> v2:172.21.15.120:3300/0 conn(0x7fc46006c6e0 0x7fc46006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522061185 0 0) 0x7fc4600cbc60 con 0x7fc4600cafa0 2024-03-23T11:19:32.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc444003520 con 0x7fc4600cafa0 2024-03-23T11:19:32.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 64894122 0 0) 0x7fc444003520 con 0x7fc4600cafa0 2024-03-23T11:19:32.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46006c300 msgr2=0x7fc4600ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:32.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46006c300 0x7fc4600ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 >> v2:172.21.15.120:3300/0 conn(0x7fc46006c6e0 msgr2=0x7fc46006cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:32.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 --2- 172.21.15.62:0/1166989888 >> v2:172.21.15.120:3300/0 conn(0x7fc46006c6e0 0x7fc46006cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4600cc500 con 0x7fc4600cafa0 2024-03-23T11:19:32.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fc44c001670 con 0x7fc4600cafa0 2024-03-23T11:19:32.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 >> v1:172.21.15.62:6789/0 conn(0x7fc4600cafa0 legacy=0x7fc4600cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:32.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46006c300 0x7fc444003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc444003ef0 0x7fc4440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 0x7fc444005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4600cbf40 con 0x7fc444004d00 2024-03-23T11:19:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4600cc220 con 0x7fc444003ef0 2024-03-23T11:19:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.225+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44400b980 con 0x7fc46006c300 2024-03-23T11:19:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.226+0000 7fc45fe00640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 0x7fc444005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.226+0000 7fc45f400640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46006c300 0x7fc444003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.226+0000 7fc45ea00640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc444003ef0 0x7fc4440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.226+0000 7fc45fe00640 1 -- 172.21.15.62:0/1166989888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46006c300 msgr2=0x7fc444003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.226+0000 7fc45fe00640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46006c300 0x7fc444003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.226+0000 7fc45fe00640 1 -- 172.21.15.62:0/1166989888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc444003ef0 msgr2=0x7fc4440047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.226+0000 7fc45fe00640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc444003ef0 0x7fc4440047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.226+0000 7fc45fe00640 1 -- 172.21.15.62:0/1166989888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444003520 con 0x7fc444004d00 2024-03-23T11:19:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.227+0000 7fc45fe00640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 0x7fc444005180 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc458009ac0 tx=0x7fc4440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=507d8671001608d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.227+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45802c6d0 con 0x7fc444004d00 2024-03-23T11:19:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.227+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45802c830 con 0x7fc444004d00 2024-03-23T11:19:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.227+0000 7fc45e000640 1 -- 172.21.15.62:0/1166989888 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45802cbc0 con 0x7fc444004d00 2024-03-23T11:19:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.228+0000 7fc466000640 1 -- 172.21.15.62:0/1166989888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 msgr2=0x7fc444005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.228+0000 7fc466000640 1 --2- 172.21.15.62:0/1166989888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 0x7fc444005180 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc458009ac0 tx=0x7fc4440057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.228+0000 7fc45fe00640 1 -- 172.21.15.62:0/1166989888 reap_dead start 2024-03-23T11:19:32.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.228+0000 7fc466000640 1 -- 172.21.15.62:0/1166989888 shutdown_connections 2024-03-23T11:19:32.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.228+0000 7fc466000640 1 -- 172.21.15.62:0/1166989888 >> 172.21.15.62:0/1166989888 conn(0x7fc4600b02e0 msgr2=0x7fc4600b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:32.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.228+0000 7fc466000640 1 -- 172.21.15.62:0/1166989888 shutdown_connections 2024-03-23T11:19:32.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.228+0000 7fc466000640 1 -- 172.21.15.62:0/1166989888 wait complete. 2024-03-23T11:19:32.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.230+0000 7fc466000640 1 Processor -- start 2024-03-23T11:19:32.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.230+0000 7fc466000640 1 -- start start 2024-03-23T11:19:32.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.231+0000 7fc466000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 0x7fc46015c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.231+0000 7fc466000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc46013c180 0x7fc4600b16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.231+0000 7fc466000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46013c9d0 0x7fc46013ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.231+0000 7fc466000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46013d390 con 0x7fc444004d00 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.231+0000 7fc466000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46015c420 con 0x7fc46013c180 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.231+0000 7fc466000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc46015d950 con 0x7fc46013c9d0 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.231+0000 7fc45f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 0x7fc46015c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.231+0000 7fc45f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 0x7fc46015c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50172/0 (socket says 172.21.15.62:50172) 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.231+0000 7fc45f400640 1 -- 172.21.15.62:0/2118505309 learned_addr learned my addr 172.21.15.62:0/2118505309 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.232+0000 7fc45ea00640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc46013c180 0x7fc4600b16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.232+0000 7fc45fe00640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46013c9d0 0x7fc46013ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.232+0000 7fc45f400640 1 -- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46013c9d0 msgr2=0x7fc46013ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.232+0000 7fc45f400640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46013c9d0 0x7fc46013ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.232+0000 7fc45f400640 1 -- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc46013c180 msgr2=0x7fc4600b16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:32.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.232+0000 7fc45f400640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc46013c180 0x7fc4600b16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.232+0000 7fc45f400640 1 -- 172.21.15.62:0/2118505309 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46015dc30 con 0x7fc444004d00 2024-03-23T11:19:32.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.232+0000 7fc45f400640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 0x7fc46015c8b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc450007f30 tx=0x7fc450011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:32.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.233+0000 7fc457400640 1 -- 172.21.15.62:0/2118505309 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45000e030 con 0x7fc444004d00 2024-03-23T11:19:32.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.233+0000 7fc457400640 1 -- 172.21.15.62:0/2118505309 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450016470 con 0x7fc444004d00 2024-03-23T11:19:32.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.233+0000 7fc457400640 1 -- 172.21.15.62:0/2118505309 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450016820 con 0x7fc444004d00 2024-03-23T11:19:32.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.233+0000 7fc466000640 1 -- 172.21.15.62:0/2118505309 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46015bdd0 con 0x7fc444004d00 2024-03-23T11:19:32.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.233+0000 7fc466000640 1 -- 172.21.15.62:0/2118505309 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46015d660 con 0x7fc444004d00 2024-03-23T11:19:32.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.234+0000 7fc466000640 1 -- 172.21.15.62:0/2118505309 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc460066c90 con 0x7fc444004d00 2024-03-23T11:19:32.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.235+0000 7fc457400640 1 -- 172.21.15.62:0/2118505309 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc450016980 con 0x7fc444004d00 2024-03-23T11:19:32.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.236+0000 7fc457400640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc44006a7a0 0x7fc44006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.236+0000 7fc457400640 1 -- 172.21.15.62:0/2118505309 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc45000f070 con 0x7fc444004d00 2024-03-23T11:19:32.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.236+0000 7fc45ea00640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc44006a7a0 0x7fc44006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.237+0000 7fc45ea00640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc44006a7a0 0x7fc44006cc60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc44c001eb0 tx=0x7fc44c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:32.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.240+0000 7fc457400640 1 -- 172.21.15.62:0/2118505309 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc450055190 con 0x7fc444004d00 2024-03-23T11:19:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.439+0000 7fc466000640 1 -- 172.21.15.62:0/2118505309 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc46015a4f0 con 0x7fc444004d00 2024-03-23T11:19:32.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.440+0000 7fc457400640 1 -- 172.21.15.62:0/2118505309 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4500587b0 con 0x7fc444004d00 2024-03-23T11:19:32.440 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:32.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.442+0000 7fc466000640 1 -- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc44006a7a0 msgr2=0x7fc44006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:32.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.442+0000 7fc466000640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc44006a7a0 0x7fc44006cc60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc44c001eb0 tx=0x7fc44c009040 comp rx=0 tx=0).stop 2024-03-23T11:19:32.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.443+0000 7fc466000640 1 -- 172.21.15.62:0/2118505309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 msgr2=0x7fc46015c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:32.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.443+0000 7fc466000640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 0x7fc46015c8b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc450007f30 tx=0x7fc450011660 comp rx=0 tx=0).stop 2024-03-23T11:19:32.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.443+0000 7fc466000640 1 -- 172.21.15.62:0/2118505309 shutdown_connections 2024-03-23T11:19:32.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.443+0000 7fc466000640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc44006a7a0 0x7fc44006cc60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.443+0000 7fc466000640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc46013c9d0 0x7fc46013ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.443+0000 7fc466000640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc46013c180 0x7fc4600b16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.443+0000 7fc466000640 1 --2- 172.21.15.62:0/2118505309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc444004d00 0x7fc46015c8b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.443+0000 7fc466000640 1 -- 172.21.15.62:0/2118505309 >> 172.21.15.62:0/2118505309 conn(0x7fc4600b02e0 msgr2=0x7fc460072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:32.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.443+0000 7fc466000640 1 -- 172.21.15.62:0/2118505309 shutdown_connections 2024-03-23T11:19:32.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.443+0000 7fc466000640 1 -- 172.21.15.62:0/2118505309 wait complete. 2024-03-23T11:19:32.461 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.671+0000 7f0518a00640 1 Processor -- start 2024-03-23T11:19:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.672+0000 7f0518a00640 1 -- start start 2024-03-23T11:19:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.673+0000 7f0518a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f051415fcf0 0x7f05140d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.673+0000 7f0518a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0514160420 0x7f0514160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.673+0000 7f0518a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f05140d06d0 0x7f05140d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.673+0000 7f0518a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0514160dc0 con 0x7f05140d06d0 2024-03-23T11:19:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.673+0000 7f0518a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05140d3540 con 0x7f0514160420 2024-03-23T11:19:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.673+0000 7f0518a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05140ce510 con 0x7f051415fcf0 2024-03-23T11:19:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.674+0000 7f0513e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f05140d06d0 0x7f05140d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.674+0000 7f0513e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f05140d06d0 0x7f05140d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50182/0 (socket says 172.21.15.62:50182) 2024-03-23T11:19:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.674+0000 7f0513e00640 1 -- 172.21.15.62:0/121499266 learned_addr learned my addr 172.21.15.62:0/121499266 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.674+0000 7f050be00640 1 --2- 172.21.15.62:0/121499266 >> v2:172.21.15.120:3300/0 conn(0x7f0514160420 0x7f0514160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.674+0000 7f0513400640 1 --2- 172.21.15.62:0/121499266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f051415fcf0 0x7f05140d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.674+0000 7f0513e00640 1 -- 172.21.15.62:0/121499266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f051415fcf0 msgr2=0x7f05140d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.674+0000 7f0513e00640 1 --2- 172.21.15.62:0/121499266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f051415fcf0 0x7f05140d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.674+0000 7f0513e00640 1 -- 172.21.15.62:0/121499266 >> v2:172.21.15.120:3300/0 conn(0x7f0514160420 msgr2=0x7f0514160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.674+0000 7f0513e00640 1 --2- 172.21.15.62:0/121499266 >> v2:172.21.15.120:3300/0 conn(0x7f0514160420 0x7f0514160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.674+0000 7f0513e00640 1 -- 172.21.15.62:0/121499266 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05140cdc70 con 0x7f05140d06d0 2024-03-23T11:19:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.676+0000 7f0513e00640 1 --2- 172.21.15.62:0/121499266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05140d06d0 0x7f05140d0b20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f05140664e0 tx=0x7f050c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14a877562df0b83 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.676+0000 7f0512a00640 1 -- 172.21.15.62:0/121499266 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c02c9e0 con 0x7f05140d06d0 2024-03-23T11:19:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.676+0000 7f0512a00640 1 -- 172.21.15.62:0/121499266 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050c02cb40 con 0x7f05140d06d0 2024-03-23T11:19:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.676+0000 7f0512a00640 1 -- 172.21.15.62:0/121499266 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c0343f0 con 0x7f05140d06d0 2024-03-23T11:19:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.676+0000 7f0518a00640 1 -- 172.21.15.62:0/121499266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05140d06d0 msgr2=0x7f05140d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.676+0000 7f0518a00640 1 --2- 172.21.15.62:0/121499266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05140d06d0 0x7f05140d0b20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f05140664e0 tx=0x7f050c02be40 comp rx=0 tx=0).stop 2024-03-23T11:19:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.677+0000 7f0518a00640 1 -- 172.21.15.62:0/121499266 shutdown_connections 2024-03-23T11:19:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.677+0000 7f0518a00640 1 --2- 172.21.15.62:0/121499266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05140d06d0 0x7f05140d0b20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.677+0000 7f0518a00640 1 --2- 172.21.15.62:0/121499266 >> v2:172.21.15.120:3300/0 conn(0x7f0514160420 0x7f0514160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.677+0000 7f0518a00640 1 --2- 172.21.15.62:0/121499266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f051415fcf0 0x7f05140d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.677+0000 7f0518a00640 1 -- 172.21.15.62:0/121499266 >> 172.21.15.62:0/121499266 conn(0x7f0514089f60 msgr2=0x7f05140d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.677+0000 7f0518a00640 1 -- 172.21.15.62:0/121499266 shutdown_connections 2024-03-23T11:19:32.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.677+0000 7f0518a00640 1 -- 172.21.15.62:0/121499266 wait complete. 2024-03-23T11:19:32.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.678+0000 7f0518a00640 1 Processor -- start 2024-03-23T11:19:32.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.679+0000 7f0518a00640 1 -- start start 2024-03-23T11:19:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.679+0000 7f0518a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05140d06d0 0x7f05141f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0518a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f051415fcf0 0x7f05140c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0518a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0514160420 0x7f051406c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0518a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05141f6640 con 0x7f05140d06d0 2024-03-23T11:19:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0518a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05141566d0 con 0x7f051415fcf0 2024-03-23T11:19:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0518a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05141f5da0 con 0x7f0514160420 2024-03-23T11:19:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0513400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05140d06d0 0x7f05141f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0513400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05140d06d0 0x7f05141f5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50192/0 (socket says 172.21.15.62:50192) 2024-03-23T11:19:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0513400640 1 -- 172.21.15.62:0/2693724971 learned_addr learned my addr 172.21.15.62:0/2693724971 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f050be00640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f051415fcf0 0x7f05140c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0513e00640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0514160420 0x7f051406c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0513400640 1 -- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0514160420 msgr2=0x7f051406c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0513400640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0514160420 0x7f051406c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0513400640 1 -- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f051415fcf0 msgr2=0x7f05140c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0513400640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f051415fcf0 0x7f05140c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.680+0000 7f0513400640 1 -- 172.21.15.62:0/2693724971 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c009b30 con 0x7f05140d06d0 2024-03-23T11:19:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.681+0000 7f0513400640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05140d06d0 0x7f05141f5700 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f050400d800 tx=0x7f050400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.681+0000 7f050b400640 1 -- 172.21.15.62:0/2693724971 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05040096a0 con 0x7f05140d06d0 2024-03-23T11:19:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.681+0000 7f050b400640 1 -- 172.21.15.62:0/2693724971 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0504009800 con 0x7f05140d06d0 2024-03-23T11:19:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.681+0000 7f050b400640 1 -- 172.21.15.62:0/2693724971 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504009990 con 0x7f05140d06d0 2024-03-23T11:19:32.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.681+0000 7f0518a00640 1 -- 172.21.15.62:0/2693724971 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05141f6080 con 0x7f05140d06d0 2024-03-23T11:19:32.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.681+0000 7f0518a00640 1 -- 172.21.15.62:0/2693724971 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051406ca40 con 0x7f05140d06d0 2024-03-23T11:19:32.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.682+0000 7f0518a00640 1 -- 172.21.15.62:0/2693724971 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0514156530 con 0x7f05140d06d0 2024-03-23T11:19:32.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.683+0000 7f050b400640 1 -- 172.21.15.62:0/2693724971 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0504009af0 con 0x7f05140d06d0 2024-03-23T11:19:32.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.684+0000 7f050b400640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f04f006a7a0 0x7f04f006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:32.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.684+0000 7f050b400640 1 -- 172.21.15.62:0/2693724971 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05040891b0 con 0x7f05140d06d0 2024-03-23T11:19:32.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.685+0000 7f050be00640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f04f006a7a0 0x7f04f006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:32.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.686+0000 7f050be00640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f04f006a7a0 0x7f04f006cc60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f05140b8400 tx=0x7f04fc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.688+0000 7f050b400640 1 -- 172.21.15.62:0/2693724971 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0514156530 con 0x7f05140d06d0 2024-03-23T11:19:32.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.883+0000 7f0518a00640 1 -- 172.21.15.62:0/2693724971 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f05140ce210 con 0x7f05140d06d0 2024-03-23T11:19:32.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.884+0000 7f050b400640 1 -- 172.21.15.62:0/2693724971 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f05140ce210 con 0x7f05140d06d0 2024-03-23T11:19:32.884 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:32.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 -- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f04f006a7a0 msgr2=0x7f04f006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:32.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f04f006a7a0 0x7f04f006cc60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f05140b8400 tx=0x7f04fc008040 comp rx=0 tx=0).stop 2024-03-23T11:19:32.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 -- 172.21.15.62:0/2693724971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05140d06d0 msgr2=0x7f05141f5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05140d06d0 0x7f05141f5700 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f050400d800 tx=0x7f050400dcc0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 -- 172.21.15.62:0/2693724971 shutdown_connections 2024-03-23T11:19:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f04f006a7a0 0x7f04f006cc60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0514160420 0x7f051406c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f051415fcf0 0x7f05140c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 --2- 172.21.15.62:0/2693724971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05140d06d0 0x7f05141f5700 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 -- 172.21.15.62:0/2693724971 >> 172.21.15.62:0/2693724971 conn(0x7f0514089f60 msgr2=0x7f051415f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 -- 172.21.15.62:0/2693724971 shutdown_connections 2024-03-23T11:19:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:32.887+0000 7f0518a00640 1 -- 172.21.15.62:0/2693724971 wait complete. 2024-03-23T11:19:32.906 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:33.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.126+0000 7f0298c00640 1 Processor -- start 2024-03-23T11:19:33.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.127+0000 7f0298c00640 1 -- start start 2024-03-23T11:19:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.128+0000 7f0298c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02941571e0 0x7f02940ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.128+0000 7f0298c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f02941613a0 0x7f029415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.128+0000 7f0298c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029406cf40 con 0x7f02940ade70 2024-03-23T11:19:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.128+0000 7f0298c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02941575c0 con 0x7f02941613a0 2024-03-23T11:19:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.128+0000 7f0298c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029406c3a0 con 0x7f02941571e0 2024-03-23T11:19:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.128+0000 7f0292800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f02941613a0 0x7f029415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.128+0000 7f0292800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f02941613a0 0x7f029415e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50194/0 (socket says 172.21.15.62:50194) 2024-03-23T11:19:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.128+0000 7f0292800640 1 -- 172.21.15.62:0/80107718 learned_addr learned my addr 172.21.15.62:0/80107718 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.129+0000 7f0291e00640 1 --2- 172.21.15.62:0/80107718 >> v2:172.21.15.120:3300/0 conn(0x7f02941571e0 0x7f02940ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.129+0000 7f0292800640 1 -- 172.21.15.62:0/80107718 >> v2:172.21.15.120:3300/0 conn(0x7f02941571e0 msgr2=0x7f02940ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.129+0000 7f0292800640 1 --2- 172.21.15.62:0/80107718 >> v2:172.21.15.120:3300/0 conn(0x7f02941571e0 0x7f02940ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.129+0000 7f0292800640 1 -- 172.21.15.62:0/80107718 >> v1:172.21.15.120:6789/0 conn(0x7f02940ade70 legacy=0x7f029406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.129+0000 7f0292800640 1 -- 172.21.15.62:0/80107718 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029406c610 con 0x7f02941613a0 2024-03-23T11:19:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.130+0000 7f0292800640 1 --2- 172.21.15.62:0/80107718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02941613a0 0x7f029415e280 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f029406c1b0 tx=0x7f028c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4dcba427b218b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:33.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.130+0000 7f0290a00640 1 -- 172.21.15.62:0/80107718 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c0317e0 con 0x7f02941613a0 2024-03-23T11:19:33.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.130+0000 7f0290a00640 1 -- 172.21.15.62:0/80107718 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c031940 con 0x7f02941613a0 2024-03-23T11:19:33.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.130+0000 7f0290a00640 1 -- 172.21.15.62:0/80107718 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c031cd0 con 0x7f02941613a0 2024-03-23T11:19:33.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.131+0000 7f0298c00640 1 -- 172.21.15.62:0/80107718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02941613a0 msgr2=0x7f029415e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.131+0000 7f0298c00640 1 --2- 172.21.15.62:0/80107718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02941613a0 0x7f029415e280 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f029406c1b0 tx=0x7f028c02b880 comp rx=0 tx=0).stop 2024-03-23T11:19:33.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.131+0000 7f0298c00640 1 -- 172.21.15.62:0/80107718 shutdown_connections 2024-03-23T11:19:33.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.131+0000 7f0298c00640 1 --2- 172.21.15.62:0/80107718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02941613a0 0x7f029415e280 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.131+0000 7f0298c00640 1 --2- 172.21.15.62:0/80107718 >> v2:172.21.15.120:3300/0 conn(0x7f02941571e0 0x7f02940ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.131+0000 7f0298c00640 1 -- 172.21.15.62:0/80107718 >> 172.21.15.62:0/80107718 conn(0x7f02940b02e0 msgr2=0x7f02940b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:33.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.131+0000 7f0298c00640 1 -- 172.21.15.62:0/80107718 shutdown_connections 2024-03-23T11:19:33.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.131+0000 7f0298c00640 1 -- 172.21.15.62:0/80107718 wait complete. 2024-03-23T11:19:33.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.133+0000 7f0298c00640 1 Processor -- start 2024-03-23T11:19:33.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.133+0000 7f0298c00640 1 -- start start 2024-03-23T11:19:33.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.134+0000 7f0298c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02940ade70 0x7f0294144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.134+0000 7f0298c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02941571e0 0x7f0294154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.134+0000 7f0298c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02941613a0 0x7f0294154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.134+0000 7f0298c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02941454c0 con 0x7f02940ade70 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.134+0000 7f0298c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0294150570 con 0x7f02941613a0 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.134+0000 7f0298c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029414f960 con 0x7f02941571e0 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.134+0000 7f0291e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02940ade70 0x7f0294144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.134+0000 7f0291e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02940ade70 0x7f0294144f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50204/0 (socket says 172.21.15.62:50204) 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.134+0000 7f0291e00640 1 -- 172.21.15.62:0/3325847936 learned_addr learned my addr 172.21.15.62:0/3325847936 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.135+0000 7f0291400640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02941571e0 0x7f0294154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.135+0000 7f0291e00640 1 -- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02941571e0 msgr2=0x7f0294154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.135+0000 7f0291e00640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02941571e0 0x7f0294154140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.135+0000 7f0291e00640 1 -- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02941613a0 msgr2=0x7f0294154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.135+0000 7f0291e00640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02941613a0 0x7f0294154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.135+0000 7f0291e00640 1 -- 172.21.15.62:0/3325847936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028c0094d0 con 0x7f02940ade70 2024-03-23T11:19:33.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.135+0000 7f0291e00640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02940ade70 0x7f0294144f80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0284007e70 tx=0x7f028400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:33.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.135+0000 7f028a000640 1 -- 172.21.15.62:0/3325847936 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284014620 con 0x7f02940ade70 2024-03-23T11:19:33.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.135+0000 7f028a000640 1 -- 172.21.15.62:0/3325847936 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0284014780 con 0x7f02940ade70 2024-03-23T11:19:33.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.136+0000 7f0298c00640 1 -- 172.21.15.62:0/3325847936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029414fc40 con 0x7f02940ade70 2024-03-23T11:19:33.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.136+0000 7f028a000640 1 -- 172.21.15.62:0/3325847936 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284014b30 con 0x7f02940ade70 2024-03-23T11:19:33.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.136+0000 7f0298c00640 1 -- 172.21.15.62:0/3325847936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0294150200 con 0x7f02940ade70 2024-03-23T11:19:33.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.137+0000 7f0298c00640 1 -- 172.21.15.62:0/3325847936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0294066c90 con 0x7f02940ade70 2024-03-23T11:19:33.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.138+0000 7f028a000640 1 -- 172.21.15.62:0/3325847936 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0284014c90 con 0x7f02940ade70 2024-03-23T11:19:33.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.139+0000 7f028a000640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f027006a7a0 0x7f027006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.139+0000 7f028a000640 1 -- 172.21.15.62:0/3325847936 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02840897f0 con 0x7f02940ade70 2024-03-23T11:19:33.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.140+0000 7f0291400640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f027006a7a0 0x7f027006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.141+0000 7f0291400640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f027006a7a0 0x7f027006cc60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0294155290 tx=0x7f027c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:33.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.144+0000 7f028a000640 1 -- 172.21.15.62:0/3325847936 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0284054b70 con 0x7f02940ade70 2024-03-23T11:19:33.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.344+0000 7f0298c00640 1 -- 172.21.15.62:0/3325847936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02940b38c0 con 0x7f02940ade70 2024-03-23T11:19:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.345+0000 7f028a000640 1 -- 172.21.15.62:0/3325847936 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0284058190 con 0x7f02940ade70 2024-03-23T11:19:33.345 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:33.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.347+0000 7f0298c00640 1 -- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f027006a7a0 msgr2=0x7f027006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.347+0000 7f0298c00640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f027006a7a0 0x7f027006cc60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0294155290 tx=0x7f027c008040 comp rx=0 tx=0).stop 2024-03-23T11:19:33.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.347+0000 7f0298c00640 1 -- 172.21.15.62:0/3325847936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02940ade70 msgr2=0x7f0294144f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.347+0000 7f0298c00640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02940ade70 0x7f0294144f80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0284007e70 tx=0x7f028400e4b0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.348+0000 7f0298c00640 1 -- 172.21.15.62:0/3325847936 shutdown_connections 2024-03-23T11:19:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.348+0000 7f0298c00640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f027006a7a0 0x7f027006cc60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.348+0000 7f0298c00640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02941613a0 0x7f0294154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.348+0000 7f0298c00640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02941571e0 0x7f0294154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.348+0000 7f0298c00640 1 --2- 172.21.15.62:0/3325847936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02940ade70 0x7f0294144f80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.348+0000 7f0298c00640 1 -- 172.21.15.62:0/3325847936 >> 172.21.15.62:0/3325847936 conn(0x7f02940b02e0 msgr2=0x7f02940ae250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.348+0000 7f0298c00640 1 -- 172.21.15.62:0/3325847936 shutdown_connections 2024-03-23T11:19:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.348+0000 7f0298c00640 1 -- 172.21.15.62:0/3325847936 wait complete. 2024-03-23T11:19:33.368 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:33.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.588+0000 7fcebb200640 1 Processor -- start 2024-03-23T11:19:33.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.588+0000 7fcebb200640 1 -- start start 2024-03-23T11:19:33.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.590+0000 7fcebb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0b2f30 0x7fcebc070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.590+0000 7fcebb200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc0cd140 con 0x7fcebc143dd0 2024-03-23T11:19:33.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.590+0000 7fcebb200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc0cd3e0 con 0x7fcebc070090 2024-03-23T11:19:33.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.590+0000 7fcebb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc070a60 con 0x7fcebc0b2f30 2024-03-23T11:19:33.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.590+0000 7fceb9e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fcebc143dd0 0x7fcebc15abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53256/0 (socket says 172.21.15.62:53256) 2024-03-23T11:19:33.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.590+0000 7fceb9e00640 1 -- 172.21.15.62:0/3362770808 learned_addr learned my addr 172.21.15.62:0/3362770808 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:33.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceba800640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0b2f30 0x7fcebc070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065287154 0 0) 0x7fcebc0cd140 con 0x7fcebc143dd0 2024-03-23T11:19:33.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce9c003520 con 0x7fcebc143dd0 2024-03-23T11:19:33.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843197481 0 0) 0x7fcebc0cd3e0 con 0x7fcebc070090 2024-03-23T11:19:33.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcebc0cd140 con 0x7fcebc070090 2024-03-23T11:19:33.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3774724849 0 0) 0x7fce9c003520 con 0x7fcebc143dd0 2024-03-23T11:19:33.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0b2f30 msgr2=0x7fcebc070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:33.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0b2f30 0x7fcebc070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 >> v1:172.21.15.120:6789/0 conn(0x7fcebc070090 legacy=0x7fcebc0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc0cd6c0 con 0x7fcebc143dd0 2024-03-23T11:19:33.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fceac0015f0 con 0x7fcebc143dd0 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 >> v1:172.21.15.62:6789/0 conn(0x7fcebc143dd0 legacy=0x7fcebc15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce9c003b30 0x7fce9c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0047c0 0x7fce9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9c005180 0x7fce9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc070a60 con 0x7fce9c0047c0 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c00bfa0 con 0x7fce9c005180 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.591+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c00c160 con 0x7fce9c003b30 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.592+0000 7fceb1400640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0047c0 0x7fce9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.592+0000 7fceba800640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9c005180 0x7fce9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.592+0000 7fceb9e00640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce9c003b30 0x7fce9c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.592+0000 7fceb1400640 1 -- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce9c003b30 msgr2=0x7fce9c003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:33.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.592+0000 7fceb1400640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce9c003b30 0x7fce9c003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.592+0000 7fceb1400640 1 -- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9c005180 msgr2=0x7fce9c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:33.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.592+0000 7fceb1400640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9c005180 0x7fce9c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.592+0000 7fceb1400640 1 -- 172.21.15.62:0/3362770808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c003520 con 0x7fce9c0047c0 2024-03-23T11:19:33.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.592+0000 7fceba800640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9c005180 0x7fce9c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:33.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.592+0000 7fceb9e00640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce9c003b30 0x7fce9c003f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:33.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.593+0000 7fceb1400640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0047c0 0x7fce9c004c40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcea40018d0 tx=0x7fce9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9381c8a80fdcb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:33.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.594+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea40052d0 con 0x7fce9c0047c0 2024-03-23T11:19:33.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.594+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea4005430 con 0x7fce9c0047c0 2024-03-23T11:19:33.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.594+0000 7fceb9400640 1 -- 172.21.15.62:0/3362770808 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4003810 con 0x7fce9c0047c0 2024-03-23T11:19:33.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.594+0000 7fcebb200640 1 -- 172.21.15.62:0/3362770808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0047c0 msgr2=0x7fce9c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.594+0000 7fcebb200640 1 --2- 172.21.15.62:0/3362770808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0047c0 0x7fce9c004c40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcea40018d0 tx=0x7fce9c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.594+0000 7fceba800640 1 -- 172.21.15.62:0/3362770808 reap_dead start 2024-03-23T11:19:33.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.594+0000 7fcebb200640 1 -- 172.21.15.62:0/3362770808 shutdown_connections 2024-03-23T11:19:33.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.595+0000 7fcebb200640 1 -- 172.21.15.62:0/3362770808 >> 172.21.15.62:0/3362770808 conn(0x7fcebc066050 msgr2=0x7fcebc069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:33.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.595+0000 7fcebb200640 1 -- 172.21.15.62:0/3362770808 shutdown_connections 2024-03-23T11:19:33.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.595+0000 7fcebb200640 1 -- 172.21.15.62:0/3362770808 wait complete. 2024-03-23T11:19:33.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.596+0000 7fcebb200640 1 Processor -- start 2024-03-23T11:19:33.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.597+0000 7fcebb200640 1 -- start start 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fcebb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9c0047c0 0x7fcebc1f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fcebb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0c3680 0x7fcebc13df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fcebb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc1f5f40 0x7fcebc15c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fcebb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc13e4c0 con 0x7fcebc1f5f40 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fcebb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc1f2630 con 0x7fce9c0047c0 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fcebb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0c3b00 con 0x7fcebc0c3680 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fceba800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc1f5f40 0x7fcebc15c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fceba800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc1f5f40 0x7fcebc15c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50214/0 (socket says 172.21.15.62:50214) 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fceba800640 1 -- 172.21.15.62:0/1674239153 learned_addr learned my addr 172.21.15.62:0/1674239153 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fceb9e00640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9c0047c0 0x7fcebc1f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fceb1400640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0c3680 0x7fcebc13df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fceba800640 1 -- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0c3680 msgr2=0x7fcebc13df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fceba800640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0c3680 0x7fcebc13df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fceba800640 1 -- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9c0047c0 msgr2=0x7fcebc1f2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fceba800640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9c0047c0 0x7fcebc1f2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fceba800640 1 -- 172.21.15.62:0/1674239153 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc0c3d80 con 0x7fcebc1f5f40 2024-03-23T11:19:33.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.598+0000 7fceb9e00640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9c0047c0 0x7fcebc1f2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:33.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.599+0000 7fceba800640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc1f5f40 0x7fcebc15c050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0cbe40 tx=0x7fceb400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:33.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.599+0000 7fceb2a00640 1 -- 172.21.15.62:0/1674239153 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4017400 con 0x7fcebc1f5f40 2024-03-23T11:19:33.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.599+0000 7fceb2a00640 1 -- 172.21.15.62:0/1674239153 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb4011040 con 0x7fcebc1f5f40 2024-03-23T11:19:33.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.599+0000 7fcebb200640 1 -- 172.21.15.62:0/1674239153 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc0c4060 con 0x7fcebc1f5f40 2024-03-23T11:19:33.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.599+0000 7fceb2a00640 1 -- 172.21.15.62:0/1674239153 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4017780 con 0x7fcebc1f5f40 2024-03-23T11:19:33.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.599+0000 7fcebb200640 1 -- 172.21.15.62:0/1674239153 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc1f63c0 con 0x7fcebc1f5f40 2024-03-23T11:19:33.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.600+0000 7fcebb200640 1 -- 172.21.15.62:0/1674239153 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc1f27c0 con 0x7fcebc1f5f40 2024-03-23T11:19:33.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.601+0000 7fceb2a00640 1 -- 172.21.15.62:0/1674239153 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fceb40203b0 con 0x7fcebc1f5f40 2024-03-23T11:19:33.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.602+0000 7fceb2a00640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9806a7d0 0x7fce9806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:33.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.602+0000 7fceb2a00640 1 -- 172.21.15.62:0/1674239153 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fceb4088ec0 con 0x7fcebc1f5f40 2024-03-23T11:19:33.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.603+0000 7fceb9e00640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9806a7d0 0x7fce9806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:33.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.604+0000 7fceb9e00640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9806a7d0 0x7fce9806cc90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fceac0078d0 tx=0x7fceac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:33.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.607+0000 7fceb2a00640 1 -- 172.21.15.62:0/1674239153 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fceb40542d0 con 0x7fcebc1f5f40 2024-03-23T11:19:33.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.807+0000 7fcebb200640 1 -- 172.21.15.62:0/1674239153 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcebc0cca40 con 0x7fcebc1f5f40 2024-03-23T11:19:33.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.809+0000 7fceb2a00640 1 -- 172.21.15.62:0/1674239153 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16037 (secure 0 0 0) 0x7fceb40578f0 con 0x7fcebc1f5f40 2024-03-23T11:19:33.809 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:33.809 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:57.238201+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.811+0000 7fcebb200640 1 -- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9806a7d0 msgr2=0x7fce9806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.811+0000 7fcebb200640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9806a7d0 0x7fce9806cc90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fceac0078d0 tx=0x7fceac009040 comp rx=0 tx=0).stop 2024-03-23T11:19:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.811+0000 7fcebb200640 1 -- 172.21.15.62:0/1674239153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc1f5f40 msgr2=0x7fcebc15c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.811+0000 7fcebb200640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc1f5f40 0x7fcebc15c050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0cbe40 tx=0x7fceb400f4a0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.811+0000 7fcebb200640 1 -- 172.21.15.62:0/1674239153 shutdown_connections 2024-03-23T11:19:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.811+0000 7fcebb200640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9806a7d0 0x7fce9806cc90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.811+0000 7fcebb200640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcebc1f5f40 0x7fcebc15c050 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.811+0000 7fcebb200640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcebc0c3680 0x7fcebc13df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.811+0000 7fcebb200640 1 --2- 172.21.15.62:0/1674239153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce9c0047c0 0x7fcebc1f2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.811+0000 7fcebb200640 1 -- 172.21.15.62:0/1674239153 >> 172.21.15.62:0/1674239153 conn(0x7fcebc066050 msgr2=0x7fcebc13e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:33.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.812+0000 7fcebb200640 1 -- 172.21.15.62:0/1674239153 shutdown_connections 2024-03-23T11:19:33.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:33.812+0000 7fcebb200640 1 -- 172.21.15.62:0/1674239153 wait complete. 2024-03-23T11:19:33.832 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:33.832 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:34.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.047+0000 7f4b76000640 1 Processor -- start 2024-03-23T11:19:34.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.047+0000 7f4b76000640 1 -- start start 2024-03-23T11:19:34.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.048+0000 7f4b76000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4b70143dd0 0x7f4b7015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.048+0000 7f4b76000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4b70070090 0x7f4b70070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.048+0000 7f4b76000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b700cd140 con 0x7f4b700b2820 2024-03-23T11:19:34.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.048+0000 7f4b76000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b70070a60 con 0x7f4b70143dd0 2024-03-23T11:19:34.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.048+0000 7f4b76000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b700cd3e0 con 0x7f4b70070090 2024-03-23T11:19:34.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.049+0000 7f4b74c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4b70143dd0 0x7f4b7015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.049+0000 7f4b75600640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4b700b2820 0x7f4b700b2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53272/0 (socket says 172.21.15.62:53272) 2024-03-23T11:19:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.049+0000 7f4b75600640 1 -- 172.21.15.62:0/2398244499 learned_addr learned my addr 172.21.15.62:0/2398244499 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.049+0000 7f4b6fe00640 1 --2- 172.21.15.62:0/2398244499 >> v2:172.21.15.120:3300/0 conn(0x7f4b70070090 0x7f4b70070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.049+0000 7f4b74c00640 1 -- 172.21.15.62:0/2398244499 >> v2:172.21.15.120:3300/0 conn(0x7f4b70070090 msgr2=0x7f4b70070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.049+0000 7f4b74c00640 1 --2- 172.21.15.62:0/2398244499 >> v2:172.21.15.120:3300/0 conn(0x7f4b70070090 0x7f4b70070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.049+0000 7f4b74c00640 1 -- 172.21.15.62:0/2398244499 >> v1:172.21.15.62:6789/0 conn(0x7f4b700b2820 legacy=0x7f4b700b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.049+0000 7f4b6f400640 1 -- 172.21.15.62:0/2398244499 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789163877 0 0) 0x7f4b700cd140 con 0x7f4b700b2820 2024-03-23T11:19:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.049+0000 7f4b74c00640 1 -- 172.21.15.62:0/2398244499 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b700cd6c0 con 0x7f4b70143dd0 2024-03-23T11:19:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.051+0000 7f4b74c00640 1 --2- 172.21.15.62:0/2398244499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70143dd0 0x7f4b7015abe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4b70071530 tx=0x7f4b6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9778ac7109221294 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:34.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.051+0000 7f4b6f400640 1 -- 172.21.15.62:0/2398244499 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b64031830 con 0x7f4b70143dd0 2024-03-23T11:19:34.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.051+0000 7f4b6f400640 1 -- 172.21.15.62:0/2398244499 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b64031990 con 0x7f4b70143dd0 2024-03-23T11:19:34.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.051+0000 7f4b6f400640 1 -- 172.21.15.62:0/2398244499 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b64031d20 con 0x7f4b70143dd0 2024-03-23T11:19:34.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.051+0000 7f4b76000640 1 -- 172.21.15.62:0/2398244499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70143dd0 msgr2=0x7f4b7015abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.051+0000 7f4b76000640 1 --2- 172.21.15.62:0/2398244499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70143dd0 0x7f4b7015abe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4b70071530 tx=0x7f4b6402b8d0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.052+0000 7f4b76000640 1 -- 172.21.15.62:0/2398244499 shutdown_connections 2024-03-23T11:19:34.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.052+0000 7f4b76000640 1 --2- 172.21.15.62:0/2398244499 >> v2:172.21.15.120:3300/0 conn(0x7f4b70070090 0x7f4b70070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.052+0000 7f4b76000640 1 --2- 172.21.15.62:0/2398244499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70143dd0 0x7f4b7015abe0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.052+0000 7f4b76000640 1 -- 172.21.15.62:0/2398244499 >> 172.21.15.62:0/2398244499 conn(0x7f4b70066050 msgr2=0x7f4b70069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:34.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.052+0000 7f4b76000640 1 -- 172.21.15.62:0/2398244499 shutdown_connections 2024-03-23T11:19:34.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.052+0000 7f4b76000640 1 -- 172.21.15.62:0/2398244499 wait complete. 2024-03-23T11:19:34.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.053+0000 7f4b76000640 1 Processor -- start 2024-03-23T11:19:34.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.054+0000 7f4b76000640 1 -- start start 2024-03-23T11:19:34.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.054+0000 7f4b76000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b70070090 0x7f4b70142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.054+0000 7f4b76000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b700b2820 0x7f4b700c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.054+0000 7f4b76000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70143dd0 0x7f4b7015c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.054+0000 7f4b76000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700c40b0 con 0x7f4b70143dd0 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.054+0000 7f4b76000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701426a0 con 0x7f4b700b2820 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.054+0000 7f4b76000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700c34a0 con 0x7f4b70070090 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.055+0000 7f4b75600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70143dd0 0x7f4b7015c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.055+0000 7f4b75600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70143dd0 0x7f4b7015c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50228/0 (socket says 172.21.15.62:50228) 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.055+0000 7f4b75600640 1 -- 172.21.15.62:0/3845899552 learned_addr learned my addr 172.21.15.62:0/3845899552 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.055+0000 7f4b74c00640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b70070090 0x7f4b70142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.055+0000 7f4b6fe00640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b700b2820 0x7f4b700c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.055+0000 7f4b75600640 1 -- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b70070090 msgr2=0x7f4b70142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.055+0000 7f4b75600640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b70070090 0x7f4b70142160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.055+0000 7f4b75600640 1 -- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b700b2820 msgr2=0x7f4b700c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.055+0000 7f4b75600640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b700b2820 0x7f4b700c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.055+0000 7f4b75600640 1 -- 172.21.15.62:0/3845899552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b64009400 con 0x7f4b70143dd0 2024-03-23T11:19:34.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.056+0000 7f4b75600640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70143dd0 0x7f4b7015c410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800c7a0 tx=0x7f4b6800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:34.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.056+0000 7f4b6cc00640 1 -- 172.21.15.62:0/3845899552 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6800a640 con 0x7f4b70143dd0 2024-03-23T11:19:34.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.056+0000 7f4b6cc00640 1 -- 172.21.15.62:0/3845899552 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6800a7a0 con 0x7f4b70143dd0 2024-03-23T11:19:34.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.056+0000 7f4b6cc00640 1 -- 172.21.15.62:0/3845899552 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6800abd0 con 0x7f4b70143dd0 2024-03-23T11:19:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.056+0000 7f4b76000640 1 -- 172.21.15.62:0/3845899552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b700c3780 con 0x7f4b70143dd0 2024-03-23T11:19:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.056+0000 7f4b76000640 1 -- 172.21.15.62:0/3845899552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b700c3d40 con 0x7f4b70143dd0 2024-03-23T11:19:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.057+0000 7f4b76000640 1 -- 172.21.15.62:0/3845899552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b700c4220 con 0x7f4b70143dd0 2024-03-23T11:19:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.058+0000 7f4b6cc00640 1 -- 172.21.15.62:0/3845899552 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4b6800ad30 con 0x7f4b70143dd0 2024-03-23T11:19:34.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.060+0000 7f4b6cc00640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b5006a7a0 0x7f4b5006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.060+0000 7f4b6cc00640 1 -- 172.21.15.62:0/3845899552 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b68088330 con 0x7f4b70143dd0 2024-03-23T11:19:34.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.060+0000 7f4b74c00640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b5006a7a0 0x7f4b5006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.061+0000 7f4b74c00640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b5006a7a0 0x7f4b5006cc60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4b64036000 tx=0x7f4b64008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:34.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.064+0000 7f4b6cc00640 1 -- 172.21.15.62:0/3845899552 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4b68053740 con 0x7f4b70143dd0 2024-03-23T11:19:34.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.263+0000 7f4b76000640 1 -- 172.21.15.62:0/3845899552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4b7015b0a0 con 0x7f4b70143dd0 2024-03-23T11:19:34.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.264+0000 7f4b6cc00640 1 -- 172.21.15.62:0/3845899552 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4b680180a0 con 0x7f4b70143dd0 2024-03-23T11:19:34.265 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:34.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.267+0000 7f4b76000640 1 -- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b5006a7a0 msgr2=0x7f4b5006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.267+0000 7f4b76000640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b5006a7a0 0x7f4b5006cc60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4b64036000 tx=0x7f4b64008030 comp rx=0 tx=0).stop 2024-03-23T11:19:34.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.267+0000 7f4b76000640 1 -- 172.21.15.62:0/3845899552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70143dd0 msgr2=0x7f4b7015c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.267+0000 7f4b76000640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70143dd0 0x7f4b7015c410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800c7a0 tx=0x7f4b6800cc60 comp rx=0 tx=0).stop 2024-03-23T11:19:34.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.267+0000 7f4b76000640 1 -- 172.21.15.62:0/3845899552 shutdown_connections 2024-03-23T11:19:34.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.267+0000 7f4b76000640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b5006a7a0 0x7f4b5006cc60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.267+0000 7f4b76000640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70143dd0 0x7f4b7015c410 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.267+0000 7f4b76000640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b700b2820 0x7f4b700c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.267+0000 7f4b76000640 1 --2- 172.21.15.62:0/3845899552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4b70070090 0x7f4b70142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.267+0000 7f4b76000640 1 -- 172.21.15.62:0/3845899552 >> 172.21.15.62:0/3845899552 conn(0x7f4b70066050 msgr2=0x7f4b700afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:34.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.268+0000 7f4b76000640 1 -- 172.21.15.62:0/3845899552 shutdown_connections 2024-03-23T11:19:34.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.268+0000 7f4b76000640 1 -- 172.21.15.62:0/3845899552 wait complete. 2024-03-23T11:19:34.287 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.505+0000 7f0643e00640 1 Processor -- start 2024-03-23T11:19:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.505+0000 7f0643e00640 1 -- start start 2024-03-23T11:19:34.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.507+0000 7f0643e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0644143dd0 0x7f064415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.507+0000 7f0643e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f06440b2f30 0x7f0644070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.507+0000 7f0643e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06440cd140 con 0x7f0644070090 2024-03-23T11:19:34.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.507+0000 7f0643e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0644070a60 con 0x7f0644143dd0 2024-03-23T11:19:34.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.507+0000 7f0643e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06440cd3e0 con 0x7f06440b2f30 2024-03-23T11:19:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.507+0000 7f0642000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0644070090 0x7f06440b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53282/0 (socket says 172.21.15.62:53282) 2024-03-23T11:19:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.507+0000 7f0642000640 1 -- 172.21.15.62:0/1922640986 learned_addr learned my addr 172.21.15.62:0/1922640986 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.507+0000 7f0642a00640 1 --2- 172.21.15.62:0/1922640986 >> v2:172.21.15.62:3300/0 conn(0x7f0644143dd0 0x7f064415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.508+0000 7f0643400640 1 --2- 172.21.15.62:0/1922640986 >> v2:172.21.15.120:3300/0 conn(0x7f06440b2f30 0x7f0644070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.508+0000 7f0641600640 1 -- 172.21.15.62:0/1922640986 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424642647 0 0) 0x7f06440cd140 con 0x7f0644070090 2024-03-23T11:19:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.508+0000 7f0641600640 1 -- 172.21.15.62:0/1922640986 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0628003520 con 0x7f0644070090 2024-03-23T11:19:34.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.508+0000 7f0642a00640 1 -- 172.21.15.62:0/1922640986 >> v2:172.21.15.120:3300/0 conn(0x7f06440b2f30 msgr2=0x7f0644070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:34.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.508+0000 7f0642a00640 1 --2- 172.21.15.62:0/1922640986 >> v2:172.21.15.120:3300/0 conn(0x7f06440b2f30 0x7f0644070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.508+0000 7f0642a00640 1 -- 172.21.15.62:0/1922640986 >> v1:172.21.15.62:6789/0 conn(0x7f0644070090 legacy=0x7f06440b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.508+0000 7f0642a00640 1 -- 172.21.15.62:0/1922640986 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06440cd6c0 con 0x7f0644143dd0 2024-03-23T11:19:34.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.508+0000 7f0643400640 1 --2- 172.21.15.62:0/1922640986 >> v2:172.21.15.120:3300/0 conn(0x7f06440b2f30 0x7f0644070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:34.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.509+0000 7f0642a00640 1 --2- 172.21.15.62:0/1922640986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0644143dd0 0x7f064415abe0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f06440cbe50 tx=0x7f063402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f8a9de10d16e19 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:34.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.510+0000 7f0641600640 1 -- 172.21.15.62:0/1922640986 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634031830 con 0x7f0644143dd0 2024-03-23T11:19:34.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.510+0000 7f0641600640 1 -- 172.21.15.62:0/1922640986 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0634031990 con 0x7f0644143dd0 2024-03-23T11:19:34.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.510+0000 7f0641600640 1 -- 172.21.15.62:0/1922640986 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634031d20 con 0x7f0644143dd0 2024-03-23T11:19:34.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.510+0000 7f0643e00640 1 -- 172.21.15.62:0/1922640986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0644143dd0 msgr2=0x7f064415abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.510+0000 7f0643e00640 1 --2- 172.21.15.62:0/1922640986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0644143dd0 0x7f064415abe0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f06440cbe50 tx=0x7f063402b8d0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.510+0000 7f0643e00640 1 -- 172.21.15.62:0/1922640986 shutdown_connections 2024-03-23T11:19:34.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.510+0000 7f0643e00640 1 --2- 172.21.15.62:0/1922640986 >> v2:172.21.15.120:3300/0 conn(0x7f06440b2f30 0x7f0644070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.510+0000 7f0643e00640 1 --2- 172.21.15.62:0/1922640986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0644143dd0 0x7f064415abe0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.510+0000 7f0643e00640 1 -- 172.21.15.62:0/1922640986 >> 172.21.15.62:0/1922640986 conn(0x7f0644066050 msgr2=0x7f0644069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:34.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.510+0000 7f0643e00640 1 -- 172.21.15.62:0/1922640986 shutdown_connections 2024-03-23T11:19:34.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.510+0000 7f0643e00640 1 -- 172.21.15.62:0/1922640986 wait complete. 2024-03-23T11:19:34.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.512+0000 7f0643e00640 1 Processor -- start 2024-03-23T11:19:34.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.512+0000 7f0643e00640 1 -- start start 2024-03-23T11:19:34.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.513+0000 7f0643e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0644070090 0x7f064412fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.513+0000 7f0643e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06440b2f30 0x7f064415c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.513+0000 7f0643e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0644143dd0 0x7f064415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.513+0000 7f0643e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06441f3f50 con 0x7f0644143dd0 2024-03-23T11:19:34.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.513+0000 7f0643e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0644130aa0 con 0x7f06440b2f30 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.513+0000 7f0643e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0644130d80 con 0x7f0644070090 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0643400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0644143dd0 0x7f064415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0643400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0644143dd0 0x7f064415cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50252/0 (socket says 172.21.15.62:50252) 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0643400640 1 -- 172.21.15.62:0/339891774 learned_addr learned my addr 172.21.15.62:0/339891774 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0642a00640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0644070090 0x7f064412fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0642000640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06440b2f30 0x7f064415c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0643400640 1 -- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0644070090 msgr2=0x7f064412fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0643400640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0644070090 0x7f064412fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0643400640 1 -- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06440b2f30 msgr2=0x7f064415c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0643400640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06440b2f30 0x7f064415c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0643400640 1 -- 172.21.15.62:0/339891774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0634009400 con 0x7f0644143dd0 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0642000640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06440b2f30 0x7f064415c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:34.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.514+0000 7f0643400640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0644143dd0 0x7f064415cd70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f063c009fa0 tx=0x7f063c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:34.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.515+0000 7f063aa00640 1 -- 172.21.15.62:0/339891774 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c003b90 con 0x7f0644143dd0 2024-03-23T11:19:34.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.515+0000 7f063aa00640 1 -- 172.21.15.62:0/339891774 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063c00f040 con 0x7f0644143dd0 2024-03-23T11:19:34.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.515+0000 7f063aa00640 1 -- 172.21.15.62:0/339891774 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c004030 con 0x7f0644143dd0 2024-03-23T11:19:34.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.515+0000 7f0643e00640 1 -- 172.21.15.62:0/339891774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0644131060 con 0x7f0644143dd0 2024-03-23T11:19:34.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.515+0000 7f0643e00640 1 -- 172.21.15.62:0/339891774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0644131740 con 0x7f0644143dd0 2024-03-23T11:19:34.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.516+0000 7f0643e00640 1 -- 172.21.15.62:0/339891774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0644066bb0 con 0x7f0644143dd0 2024-03-23T11:19:34.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.517+0000 7f063aa00640 1 -- 172.21.15.62:0/339891774 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f063c004190 con 0x7f0644143dd0 2024-03-23T11:19:34.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.518+0000 7f063aa00640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f062406a7a0 0x7f062406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.518+0000 7f063aa00640 1 -- 172.21.15.62:0/339891774 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f063c089160 con 0x7f0644143dd0 2024-03-23T11:19:34.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.519+0000 7f0642a00640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f062406a7a0 0x7f062406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.520+0000 7f0642a00640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f062406a7a0 0x7f062406cc60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0634036000 tx=0x7f063403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:34.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.522+0000 7f063aa00640 1 -- 172.21.15.62:0/339891774 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f063c054580 con 0x7f0644143dd0 2024-03-23T11:19:34.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.720+0000 7f0643e00640 1 -- 172.21.15.62:0/339891774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f064415adf0 con 0x7f0644143dd0 2024-03-23T11:19:34.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.721+0000 7f063aa00640 1 -- 172.21.15.62:0/339891774 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f063c057ba0 con 0x7f0644143dd0 2024-03-23T11:19:34.722 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:34.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.724+0000 7f0643e00640 1 -- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f062406a7a0 msgr2=0x7f062406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.725+0000 7f0643e00640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f062406a7a0 0x7f062406cc60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0634036000 tx=0x7f063403c040 comp rx=0 tx=0).stop 2024-03-23T11:19:34.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.725+0000 7f0643e00640 1 -- 172.21.15.62:0/339891774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0644143dd0 msgr2=0x7f064415cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.725+0000 7f0643e00640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0644143dd0 0x7f064415cd70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f063c009fa0 tx=0x7f063c00d560 comp rx=0 tx=0).stop 2024-03-23T11:19:34.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.725+0000 7f0643e00640 1 -- 172.21.15.62:0/339891774 shutdown_connections 2024-03-23T11:19:34.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.725+0000 7f0643e00640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f062406a7a0 0x7f062406cc60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.725+0000 7f0643e00640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0644143dd0 0x7f064415cd70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.725+0000 7f0643e00640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06440b2f30 0x7f064415c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.725+0000 7f0643e00640 1 --2- 172.21.15.62:0/339891774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0644070090 0x7f064412fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.725+0000 7f0643e00640 1 -- 172.21.15.62:0/339891774 >> 172.21.15.62:0/339891774 conn(0x7f0644066050 msgr2=0x7f0644069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:34.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.725+0000 7f0643e00640 1 -- 172.21.15.62:0/339891774 shutdown_connections 2024-03-23T11:19:34.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.725+0000 7f0643e00640 1 -- 172.21.15.62:0/339891774 wait complete. 2024-03-23T11:19:34.746 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:34.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.973+0000 7f06eb200640 1 Processor -- start 2024-03-23T11:19:34.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.973+0000 7f06eb200640 1 -- start start 2024-03-23T11:19:34.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.975+0000 7f06eb200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f06ec15c830 0x7f06ec0ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.975+0000 7f06eb200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ec1608d0 con 0x7f06ec160310 2024-03-23T11:19:34.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.975+0000 7f06eb200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ec160bb0 con 0x7f06ec0ce250 2024-03-23T11:19:34.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.975+0000 7f06eb200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ec0addb0 con 0x7f06ec15c830 2024-03-23T11:19:34.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.975+0000 7f06e1400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f06ec160310 0x7f06ec0ab980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53292/0 (socket says 172.21.15.62:53292) 2024-03-23T11:19:34.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.975+0000 7f06e1400640 1 -- 172.21.15.62:0/4266179467 learned_addr learned my addr 172.21.15.62:0/4266179467 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:34.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9e00640 1 --2- 172.21.15.62:0/4266179467 >> v2:172.21.15.120:3300/0 conn(0x7f06ec15c830 0x7f06ec0ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970489019 0 0) 0x7f06ec1608d0 con 0x7f06ec160310 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d0003520 con 0x7f06ec160310 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 62281985 0 0) 0x7f06d0003520 con 0x7f06ec160310 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 >> v2:172.21.15.120:3300/0 conn(0x7f06ec15c830 msgr2=0x7f06ec0ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 --2- 172.21.15.62:0/4266179467 >> v2:172.21.15.120:3300/0 conn(0x7f06ec15c830 0x7f06ec0ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 >> v1:172.21.15.120:6789/0 conn(0x7f06ec0ce250 legacy=0x7f06ec15d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec160e90 con 0x7f06ec160310 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f06d4001670 con 0x7f06ec160310 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 >> v1:172.21.15.62:6789/0 conn(0x7f06ec160310 legacy=0x7f06ec0ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 --2- 172.21.15.62:0/4266179467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06d00039c0 0x7f06d0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 --2- 172.21.15.62:0/4266179467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 0x7f06d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 --2- 172.21.15.62:0/4266179467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06d0005180 0x7f06d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d000c050 con 0x7f06d00047c0 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d000c1c0 con 0x7f06d0005180 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.976+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d000c380 con 0x7f06d00039c0 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.977+0000 7f06e1400640 1 --2- 172.21.15.62:0/4266179467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 0x7f06d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.977+0000 7f06e9e00640 1 --2- 172.21.15.62:0/4266179467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06d00039c0 0x7f06d0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.977+0000 7f06ea800640 1 --2- 172.21.15.62:0/4266179467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06d0005180 0x7f06d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.977+0000 7f06e1400640 1 -- 172.21.15.62:0/4266179467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06d00039c0 msgr2=0x7f06d0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.977+0000 7f06e1400640 1 --2- 172.21.15.62:0/4266179467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06d00039c0 0x7f06d0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.977+0000 7f06e1400640 1 -- 172.21.15.62:0/4266179467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06d0005180 msgr2=0x7f06d000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.977+0000 7f06e1400640 1 --2- 172.21.15.62:0/4266179467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06d0005180 0x7f06d000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.977+0000 7f06e1400640 1 -- 172.21.15.62:0/4266179467 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d0003520 con 0x7f06d00047c0 2024-03-23T11:19:34.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.977+0000 7f06ea800640 1 -- 172.21.15.62:0/4266179467 reap_dead start 2024-03-23T11:19:34.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.978+0000 7f06e1400640 1 --2- 172.21.15.62:0/4266179467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 0x7f06d0004c40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f06d400c0e0 tx=0x7f06d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4881a99f4f48b5ee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:34.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.979+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d4005ad0 con 0x7f06d00047c0 2024-03-23T11:19:34.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.979+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d4005c30 con 0x7f06d00047c0 2024-03-23T11:19:34.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.979+0000 7f06e9400640 1 -- 172.21.15.62:0/4266179467 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d402ca50 con 0x7f06d00047c0 2024-03-23T11:19:34.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.979+0000 7f06eb200640 1 -- 172.21.15.62:0/4266179467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 msgr2=0x7f06d0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.979+0000 7f06eb200640 1 --2- 172.21.15.62:0/4266179467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 0x7f06d0004c40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f06d400c0e0 tx=0x7f06d00057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.979+0000 7f06eb200640 1 -- 172.21.15.62:0/4266179467 shutdown_connections 2024-03-23T11:19:34.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.979+0000 7f06eb200640 1 --2- 172.21.15.62:0/4266179467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 0x7f06d0004c40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.979+0000 7f06eb200640 1 -- 172.21.15.62:0/4266179467 >> 172.21.15.62:0/4266179467 conn(0x7f06ec066050 msgr2=0x7f06ec069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:34.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.979+0000 7f06eb200640 1 -- 172.21.15.62:0/4266179467 shutdown_connections 2024-03-23T11:19:34.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.980+0000 7f06eb200640 1 -- 172.21.15.62:0/4266179467 wait complete. 2024-03-23T11:19:34.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.981+0000 7f06eb200640 1 Processor -- start 2024-03-23T11:19:34.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.981+0000 7f06eb200640 1 -- start start 2024-03-23T11:19:34.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.982+0000 7f06eb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 0x7f06ec15f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.982+0000 7f06eb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ec0c3300 0x7f06ec0aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.982+0000 7f06eb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ec0af430 0x7f06ec0c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.982+0000 7f06eb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec15fe50 con 0x7f06d00047c0 2024-03-23T11:19:34.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.982+0000 7f06eb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec0af8c0 con 0x7f06ec0af430 2024-03-23T11:19:34.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.982+0000 7f06eb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1577b0 con 0x7f06ec0c3300 2024-03-23T11:19:34.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.983+0000 7f06e9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 0x7f06ec15f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.983+0000 7f06e9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 0x7f06ec15f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50278/0 (socket says 172.21.15.62:50278) 2024-03-23T11:19:34.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.983+0000 7f06e9e00640 1 -- 172.21.15.62:0/3515658926 learned_addr learned my addr 172.21.15.62:0/3515658926 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:34.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.983+0000 7f06e1400640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ec0c3300 0x7f06ec0aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.983+0000 7f06ea800640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ec0af430 0x7f06ec0c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.983+0000 7f06e9e00640 1 -- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ec0c3300 msgr2=0x7f06ec0aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.983+0000 7f06e9e00640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ec0c3300 0x7f06ec0aeef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.983+0000 7f06e9e00640 1 -- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ec0af430 msgr2=0x7f06ec0c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:34.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.983+0000 7f06e9e00640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ec0af430 0x7f06ec0c4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:34.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.983+0000 7f06e9e00640 1 -- 172.21.15.62:0/3515658926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec0c37e0 con 0x7f06d00047c0 2024-03-23T11:19:34.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.983+0000 7f06e9e00640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 0x7f06ec15f910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0114e0 tx=0x7f06dc0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:34.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.984+0000 7f06e2a00640 1 -- 172.21.15.62:0/3515658926 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc015870 con 0x7f06d00047c0 2024-03-23T11:19:34.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.984+0000 7f06e2a00640 1 -- 172.21.15.62:0/3515658926 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06dc0159d0 con 0x7f06d00047c0 2024-03-23T11:19:34.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.984+0000 7f06e2a00640 1 -- 172.21.15.62:0/3515658926 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc015d40 con 0x7f06d00047c0 2024-03-23T11:19:34.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.984+0000 7f06eb200640 1 -- 172.21.15.62:0/3515658926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec0c3ac0 con 0x7f06d00047c0 2024-03-23T11:19:34.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.984+0000 7f06eb200640 1 -- 172.21.15.62:0/3515658926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec0c4080 con 0x7f06d00047c0 2024-03-23T11:19:34.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.985+0000 7f06eb200640 1 -- 172.21.15.62:0/3515658926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec066bb0 con 0x7f06d00047c0 2024-03-23T11:19:34.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.986+0000 7f06e2a00640 1 -- 172.21.15.62:0/3515658926 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f06dc013400 con 0x7f06d00047c0 2024-03-23T11:19:34.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.987+0000 7f06e2a00640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f06c806a7d0 0x7f06c806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:34.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.987+0000 7f06e2a00640 1 -- 172.21.15.62:0/3515658926 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06dc089250 con 0x7f06d00047c0 2024-03-23T11:19:34.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.987+0000 7f06e1400640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f06c806a7d0 0x7f06c806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:34.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.988+0000 7f06e1400640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f06c806a7d0 0x7f06c806cc90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f06d4034000 tx=0x7f06d4009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:34.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:34.991+0000 7f06e2a00640 1 -- 172.21.15.62:0/3515658926 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06dc0545d0 con 0x7f06d00047c0 2024-03-23T11:19:35.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.193+0000 7f06eb200640 1 -- 172.21.15.62:0/3515658926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f06ec069fe0 con 0x7f06d00047c0 2024-03-23T11:19:35.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.193+0000 7f06e2a00640 1 -- 172.21.15.62:0/3515658926 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f06dc057bf0 con 0x7f06d00047c0 2024-03-23T11:19:35.194 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:35.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 -- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f06c806a7d0 msgr2=0x7f06c806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:35.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f06c806a7d0 0x7f06c806cc90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f06d4034000 tx=0x7f06d4009030 comp rx=0 tx=0).stop 2024-03-23T11:19:35.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 -- 172.21.15.62:0/3515658926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 msgr2=0x7f06ec15f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:35.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 0x7f06ec15f910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0114e0 tx=0x7f06dc0119a0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 -- 172.21.15.62:0/3515658926 shutdown_connections 2024-03-23T11:19:35.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f06c806a7d0 0x7f06c806cc90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ec0af430 0x7f06ec0c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06ec0c3300 0x7f06ec0aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 --2- 172.21.15.62:0/3515658926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06d00047c0 0x7f06ec15f910 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 -- 172.21.15.62:0/3515658926 >> 172.21.15.62:0/3515658926 conn(0x7f06ec066050 msgr2=0x7f06ec06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:35.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 -- 172.21.15.62:0/3515658926 shutdown_connections 2024-03-23T11:19:35.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.196+0000 7f06eb200640 1 -- 172.21.15.62:0/3515658926 wait complete. 2024-03-23T11:19:35.217 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:35.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.433+0000 7f78f5c00640 1 Processor -- start 2024-03-23T11:19:35.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.433+0000 7f78f5c00640 1 -- start start 2024-03-23T11:19:35.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.435+0000 7f78f5c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f78f0159490 0x7f78f006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.435+0000 7f78f5c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f00ced90 con 0x7f78f00ce5f0 2024-03-23T11:19:35.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.435+0000 7f78f5c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f006e3e0 con 0x7f78f015da90 2024-03-23T11:19:35.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.435+0000 7f78f5c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f006c1c0 con 0x7f78f0159490 2024-03-23T11:19:35.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.435+0000 7f78efe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f78f0159490 0x7f78f006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.435+0000 7f78efe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f78f0159490 0x7f78f006cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50290/0 (socket says 172.21.15.62:50290) 2024-03-23T11:19:35.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.435+0000 7f78f5200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f78f00ce5f0 0x7f78f00b26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53304/0 (socket says 172.21.15.62:53304) 2024-03-23T11:19:35.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.435+0000 7f78efe00640 1 -- 172.21.15.62:0/648679503 learned_addr learned my addr 172.21.15.62:0/648679503 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:35.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.436+0000 7f78efe00640 1 -- 172.21.15.62:0/648679503 >> v1:172.21.15.120:6789/0 conn(0x7f78f015da90 legacy=0x7f78f00cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:35.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.436+0000 7f78efe00640 1 -- 172.21.15.62:0/648679503 >> v1:172.21.15.62:6789/0 conn(0x7f78f00ce5f0 legacy=0x7f78f00b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:35.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.436+0000 7f78efe00640 1 -- 172.21.15.62:0/648679503 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f006e100 con 0x7f78f0159490 2024-03-23T11:19:35.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.437+0000 7f78efe00640 1 --2- 172.21.15.62:0/648679503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78f0159490 0x7f78f006cc60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f78e000bb10 tx=0x7f78e002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803899d207153bc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:35.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.437+0000 7f78eea00640 1 -- 172.21.15.62:0/648679503 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e002ee90 con 0x7f78f0159490 2024-03-23T11:19:35.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.437+0000 7f78eea00640 1 -- 172.21.15.62:0/648679503 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e0036440 con 0x7f78f0159490 2024-03-23T11:19:35.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.437+0000 7f78eea00640 1 -- 172.21.15.62:0/648679503 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e00367d0 con 0x7f78f0159490 2024-03-23T11:19:35.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.438+0000 7f78f5c00640 1 -- 172.21.15.62:0/648679503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78f0159490 msgr2=0x7f78f006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:35.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.438+0000 7f78f5c00640 1 --2- 172.21.15.62:0/648679503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78f0159490 0x7f78f006cc60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f78e000bb10 tx=0x7f78e002e0e0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.438+0000 7f78f5c00640 1 -- 172.21.15.62:0/648679503 shutdown_connections 2024-03-23T11:19:35.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.438+0000 7f78f5c00640 1 --2- 172.21.15.62:0/648679503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78f0159490 0x7f78f006cc60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.438+0000 7f78f5c00640 1 -- 172.21.15.62:0/648679503 >> 172.21.15.62:0/648679503 conn(0x7f78f0066050 msgr2=0x7f78f0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:35.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.438+0000 7f78f5c00640 1 -- 172.21.15.62:0/648679503 shutdown_connections 2024-03-23T11:19:35.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.438+0000 7f78f5c00640 1 -- 172.21.15.62:0/648679503 wait complete. 2024-03-23T11:19:35.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.439+0000 7f78f5c00640 1 Processor -- start 2024-03-23T11:19:35.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.440+0000 7f78f5c00640 1 -- start start 2024-03-23T11:19:35.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.440+0000 7f78f5c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78f00ce5f0 0x7f78f015b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78f5c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78f0159490 0x7f78f0142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78f5c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78f015da90 0x7f78f01501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78f5c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f015c5b0 con 0x7f78f0159490 2024-03-23T11:19:35.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78f5c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f0156100 con 0x7f78f015da90 2024-03-23T11:19:35.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78f5c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f00ccaa0 con 0x7f78f00ce5f0 2024-03-23T11:19:35.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78ef400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78f0159490 0x7f78f0142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78ef400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78f0159490 0x7f78f0142ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50302/0 (socket says 172.21.15.62:50302) 2024-03-23T11:19:35.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78ef400640 1 -- 172.21.15.62:0/712958335 learned_addr learned my addr 172.21.15.62:0/712958335 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:35.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78efe00640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78f00ce5f0 0x7f78f015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78f5200640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78f015da90 0x7f78f01501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78ef400640 1 -- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78f00ce5f0 msgr2=0x7f78f015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78ef400640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78f00ce5f0 0x7f78f015b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78ef400640 1 -- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78f015da90 msgr2=0x7f78f01501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78ef400640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78f015da90 0x7f78f01501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78ef400640 1 -- 172.21.15.62:0/712958335 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e000b770 con 0x7f78f0159490 2024-03-23T11:19:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.441+0000 7f78f5200640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78f015da90 0x7f78f01501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:35.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.442+0000 7f78ef400640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78f0159490 0x7f78f0142ce0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f78d800b420 tx=0x7f78d800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:35.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.442+0000 7f78e7e00640 1 -- 172.21.15.62:0/712958335 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d8011830 con 0x7f78f0159490 2024-03-23T11:19:35.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.442+0000 7f78e7e00640 1 -- 172.21.15.62:0/712958335 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d8011990 con 0x7f78f0159490 2024-03-23T11:19:35.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.442+0000 7f78e7e00640 1 -- 172.21.15.62:0/712958335 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d8011d40 con 0x7f78f0159490 2024-03-23T11:19:35.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.442+0000 7f78f5c00640 1 -- 172.21.15.62:0/712958335 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f015ba30 con 0x7f78f0159490 2024-03-23T11:19:35.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.442+0000 7f78f5c00640 1 -- 172.21.15.62:0/712958335 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f015bff0 con 0x7f78f0159490 2024-03-23T11:19:35.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.444+0000 7f78f5c00640 1 -- 172.21.15.62:0/712958335 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f0156290 con 0x7f78f0159490 2024-03-23T11:19:35.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.444+0000 7f78e7e00640 1 -- 172.21.15.62:0/712958335 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f78d801b460 con 0x7f78f0159490 2024-03-23T11:19:35.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.445+0000 7f78e7e00640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f78cc06a7d0 0x7f78cc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.445+0000 7f78e7e00640 1 -- 172.21.15.62:0/712958335 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78d8088940 con 0x7f78f0159490 2024-03-23T11:19:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.446+0000 7f78efe00640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f78cc06a7d0 0x7f78cc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.447+0000 7f78efe00640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f78cc06a7d0 0x7f78cc06cc90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f78e0038000 tx=0x7f78e0032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:35.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.450+0000 7f78e7e00640 1 -- 172.21.15.62:0/712958335 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f78d8053cd0 con 0x7f78f0159490 2024-03-23T11:19:35.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.647+0000 7f78f5c00640 1 -- 172.21.15.62:0/712958335 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78f0066bb0 con 0x7f78f0159490 2024-03-23T11:19:35.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.650+0000 7f78e7e00640 1 -- 172.21.15.62:0/712958335 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f78d80572f0 con 0x7f78f0159490 2024-03-23T11:19:35.651 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:35.651 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:57.238201+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:35.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 -- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f78cc06a7d0 msgr2=0x7f78cc06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:35.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f78cc06a7d0 0x7f78cc06cc90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f78e0038000 tx=0x7f78e0032000 comp rx=0 tx=0).stop 2024-03-23T11:19:35.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 -- 172.21.15.62:0/712958335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78f0159490 msgr2=0x7f78f0142ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:35.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78f0159490 0x7f78f0142ce0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f78d800b420 tx=0x7f78d800b8e0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 -- 172.21.15.62:0/712958335 shutdown_connections 2024-03-23T11:19:35.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f78cc06a7d0 0x7f78cc06cc90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78f0159490 0x7f78f0142ce0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78f015da90 0x7f78f01501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 --2- 172.21.15.62:0/712958335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78f00ce5f0 0x7f78f015b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 -- 172.21.15.62:0/712958335 >> 172.21.15.62:0/712958335 conn(0x7f78f0066050 msgr2=0x7f78f00b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:35.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 -- 172.21.15.62:0/712958335 shutdown_connections 2024-03-23T11:19:35.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.654+0000 7f78f5c00640 1 -- 172.21.15.62:0/712958335 wait complete. 2024-03-23T11:19:35.674 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:35.675 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:35.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.899+0000 7fa0cda00640 1 Processor -- start 2024-03-23T11:19:35.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.900+0000 7fa0cda00640 1 -- start start 2024-03-23T11:19:35.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.901+0000 7fa0cda00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0c80ce8d0 0x7fa0c80af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.901+0000 7fa0cda00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa0c80cecb0 0x7fa0c80ad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.901+0000 7fa0cda00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0c80addb0 0x7fa0c80ae240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.901+0000 7fa0cda00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c80cac10 con 0x7fa0c80addb0 2024-03-23T11:19:35.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.901+0000 7fa0cda00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c80cc860 con 0x7fa0c80cecb0 2024-03-23T11:19:35.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.901+0000 7fa0cda00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80cbfc0 con 0x7fa0c80ce8d0 2024-03-23T11:19:35.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.902+0000 7fa0cd000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0c80addb0 0x7fa0c80ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.902+0000 7fa0cd000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0c80addb0 0x7fa0c80ae240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50310/0 (socket says 172.21.15.62:50310) 2024-03-23T11:19:35.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.902+0000 7fa0cd000640 1 -- 172.21.15.62:0/639919049 learned_addr learned my addr 172.21.15.62:0/639919049 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:35.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.902+0000 7fa0c7400640 1 --2- 172.21.15.62:0/639919049 >> v2:172.21.15.120:3300/0 conn(0x7fa0c80cecb0 0x7fa0c80ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.902+0000 7fa0c7e00640 1 --2- 172.21.15.62:0/639919049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0c80ce8d0 0x7fa0c80af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.902+0000 7fa0cd000640 1 -- 172.21.15.62:0/639919049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0c80ce8d0 msgr2=0x7fa0c80af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:35.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.902+0000 7fa0cd000640 1 --2- 172.21.15.62:0/639919049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0c80ce8d0 0x7fa0c80af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.902+0000 7fa0cd000640 1 -- 172.21.15.62:0/639919049 >> v2:172.21.15.120:3300/0 conn(0x7fa0c80cecb0 msgr2=0x7fa0c80ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:35.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.902+0000 7fa0cd000640 1 --2- 172.21.15.62:0/639919049 >> v2:172.21.15.120:3300/0 conn(0x7fa0c80cecb0 0x7fa0c80ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.902+0000 7fa0cd000640 1 -- 172.21.15.62:0/639919049 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c80cc2a0 con 0x7fa0c80addb0 2024-03-23T11:19:35.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.902+0000 7fa0c7e00640 1 --2- 172.21.15.62:0/639919049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0c80ce8d0 0x7fa0c80af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:35.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.903+0000 7fa0cd000640 1 --2- 172.21.15.62:0/639919049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0c80addb0 0x7fa0c80ae240 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8071e00 tx=0x7fa0c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c09eb3f4e40f446 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:35.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.904+0000 7fa0c6a00640 1 -- 172.21.15.62:0/639919049 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0031870 con 0x7fa0c80addb0 2024-03-23T11:19:35.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.904+0000 7fa0c6a00640 1 -- 172.21.15.62:0/639919049 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c00319d0 con 0x7fa0c80addb0 2024-03-23T11:19:35.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.904+0000 7fa0c6a00640 1 -- 172.21.15.62:0/639919049 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0031d60 con 0x7fa0c80addb0 2024-03-23T11:19:35.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.904+0000 7fa0cda00640 1 -- 172.21.15.62:0/639919049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0c80addb0 msgr2=0x7fa0c80ae240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:35.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.904+0000 7fa0cda00640 1 --2- 172.21.15.62:0/639919049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0c80addb0 0x7fa0c80ae240 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8071e00 tx=0x7fa0c002b910 comp rx=0 tx=0).stop 2024-03-23T11:19:35.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.905+0000 7fa0cda00640 1 -- 172.21.15.62:0/639919049 shutdown_connections 2024-03-23T11:19:35.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.905+0000 7fa0cda00640 1 --2- 172.21.15.62:0/639919049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0c80addb0 0x7fa0c80ae240 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.905+0000 7fa0cda00640 1 --2- 172.21.15.62:0/639919049 >> v2:172.21.15.120:3300/0 conn(0x7fa0c80cecb0 0x7fa0c80ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.905+0000 7fa0cda00640 1 --2- 172.21.15.62:0/639919049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0c80ce8d0 0x7fa0c80af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.905+0000 7fa0cda00640 1 -- 172.21.15.62:0/639919049 >> 172.21.15.62:0/639919049 conn(0x7fa0c80b1fb0 msgr2=0x7fa0c80b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:35.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.905+0000 7fa0cda00640 1 -- 172.21.15.62:0/639919049 shutdown_connections 2024-03-23T11:19:35.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.905+0000 7fa0cda00640 1 -- 172.21.15.62:0/639919049 wait complete. 2024-03-23T11:19:35.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.907+0000 7fa0cda00640 1 Processor -- start 2024-03-23T11:19:35.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.907+0000 7fa0cda00640 1 -- start start 2024-03-23T11:19:35.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.908+0000 7fa0cda00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0c80addb0 0x7fa0c80c7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.908+0000 7fa0cda00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0c80ce8d0 0x7fa0c8146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.908+0000 7fa0cda00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0c80cecb0 0x7fa0c81470d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.908+0000 7fa0cda00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80c4b60 con 0x7fa0c80ce8d0 2024-03-23T11:19:35.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.908+0000 7fa0cda00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c8147700 con 0x7fa0c80cecb0 2024-03-23T11:19:35.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.908+0000 7fa0cda00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c81479e0 con 0x7fa0c80addb0 2024-03-23T11:19:35.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.908+0000 7fa0c7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0c80ce8d0 0x7fa0c8146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.908+0000 7fa0c7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0c80ce8d0 0x7fa0c8146b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50320/0 (socket says 172.21.15.62:50320) 2024-03-23T11:19:35.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.908+0000 7fa0c7400640 1 -- 172.21.15.62:0/806412470 learned_addr learned my addr 172.21.15.62:0/806412470 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:35.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.908+0000 7fa0c7e00640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0c80addb0 0x7fa0c80c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.909+0000 7fa0cd000640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0c80cecb0 0x7fa0c81470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.909+0000 7fa0c7400640 1 -- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0c80addb0 msgr2=0x7fa0c80c7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:35.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.909+0000 7fa0c7400640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0c80addb0 0x7fa0c80c7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.909+0000 7fa0c7400640 1 -- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0c80cecb0 msgr2=0x7fa0c81470d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:35.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.909+0000 7fa0c7400640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0c80cecb0 0x7fa0c81470d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:35.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.909+0000 7fa0c7400640 1 -- 172.21.15.62:0/806412470 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c00094d0 con 0x7fa0c80ce8d0 2024-03-23T11:19:35.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.909+0000 7fa0c7400640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0c80ce8d0 0x7fa0c8146b90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000d5f0 tx=0x7fa0b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:35.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.909+0000 7fa0bfe00640 1 -- 172.21.15.62:0/806412470 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b00125f0 con 0x7fa0c80ce8d0 2024-03-23T11:19:35.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.909+0000 7fa0bfe00640 1 -- 172.21.15.62:0/806412470 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b0012750 con 0x7fa0c80ce8d0 2024-03-23T11:19:35.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.909+0000 7fa0bfe00640 1 -- 172.21.15.62:0/806412470 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0012b00 con 0x7fa0c80ce8d0 2024-03-23T11:19:35.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.910+0000 7fa0cda00640 1 -- 172.21.15.62:0/806412470 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c8147cc0 con 0x7fa0c80ce8d0 2024-03-23T11:19:35.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.910+0000 7fa0cda00640 1 -- 172.21.15.62:0/806412470 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c806c2f0 con 0x7fa0c80ce8d0 2024-03-23T11:19:35.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.911+0000 7fa0cda00640 1 -- 172.21.15.62:0/806412470 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c8073540 con 0x7fa0c80ce8d0 2024-03-23T11:19:35.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.912+0000 7fa0bfe00640 1 -- 172.21.15.62:0/806412470 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa0b0012c60 con 0x7fa0c80ce8d0 2024-03-23T11:19:35.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.912+0000 7fa0bfe00640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa0a406a7a0 0x7fa0a406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:35.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.913+0000 7fa0bfe00640 1 -- 172.21.15.62:0/806412470 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0b00891f0 con 0x7fa0c80ce8d0 2024-03-23T11:19:35.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.913+0000 7fa0c7e00640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa0a406a7a0 0x7fa0a406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:35.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.914+0000 7fa0c7e00640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa0a406a7a0 0x7fa0a406cc60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa0c80c72b0 tx=0x7fa0b80079c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:35.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:35.917+0000 7fa0bfe00640 1 -- 172.21.15.62:0/806412470 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa0b00546a0 con 0x7fa0c80ce8d0 2024-03-23T11:19:36.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.114+0000 7fa0cda00640 1 -- 172.21.15.62:0/806412470 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0c80b1d80 con 0x7fa0c80ce8d0 2024-03-23T11:19:36.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.115+0000 7fa0bfe00640 1 -- 172.21.15.62:0/806412470 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa0b0057cc0 con 0x7fa0c80ce8d0 2024-03-23T11:19:36.115 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:36.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.117+0000 7fa0cda00640 1 -- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa0a406a7a0 msgr2=0x7fa0a406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.117+0000 7fa0cda00640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa0a406a7a0 0x7fa0a406cc60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa0c80c72b0 tx=0x7fa0b80079c0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.118+0000 7fa0cda00640 1 -- 172.21.15.62:0/806412470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0c80ce8d0 msgr2=0x7fa0c8146b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.118+0000 7fa0cda00640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0c80ce8d0 0x7fa0c8146b90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000d5f0 tx=0x7fa0b000dab0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.118+0000 7fa0cda00640 1 -- 172.21.15.62:0/806412470 shutdown_connections 2024-03-23T11:19:36.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.118+0000 7fa0cda00640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa0a406a7a0 0x7fa0a406cc60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.118+0000 7fa0cda00640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0c80ce8d0 0x7fa0c8146b90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.118+0000 7fa0cda00640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0c80cecb0 0x7fa0c81470d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.118+0000 7fa0cda00640 1 --2- 172.21.15.62:0/806412470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0c80addb0 0x7fa0c80c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.118+0000 7fa0cda00640 1 -- 172.21.15.62:0/806412470 >> 172.21.15.62:0/806412470 conn(0x7fa0c80b1fb0 msgr2=0x7fa0c806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:36.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.118+0000 7fa0cda00640 1 -- 172.21.15.62:0/806412470 shutdown_connections 2024-03-23T11:19:36.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.118+0000 7fa0cda00640 1 -- 172.21.15.62:0/806412470 wait complete. 2024-03-23T11:19:36.138 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:36.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.368+0000 7fae7a800640 1 Processor -- start 2024-03-23T11:19:36.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.368+0000 7fae7a800640 1 -- start start 2024-03-23T11:19:36.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.370+0000 7fae7a800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fae74159490 0x7fae7406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.370+0000 7fae7a800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fae740ce5f0 0x7fae740b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.370+0000 7fae7a800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae740ced90 con 0x7fae7415da90 2024-03-23T11:19:36.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.370+0000 7fae7a800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7406e3e0 con 0x7fae74159490 2024-03-23T11:19:36.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.370+0000 7fae7a800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7406c1c0 con 0x7fae740ce5f0 2024-03-23T11:19:36.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.370+0000 7fae79400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fae74159490 0x7fae7406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.370+0000 7fae79400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fae74159490 0x7fae7406cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50328/0 (socket says 172.21.15.62:50328) 2024-03-23T11:19:36.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.370+0000 7fae79400640 1 -- 172.21.15.62:0/3777652481 learned_addr learned my addr 172.21.15.62:0/3777652481 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:36.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.370+0000 7fae79e00640 1 --2- 172.21.15.62:0/3777652481 >> v2:172.21.15.120:3300/0 conn(0x7fae740ce5f0 0x7fae740b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.371+0000 7fae79400640 1 -- 172.21.15.62:0/3777652481 >> v2:172.21.15.120:3300/0 conn(0x7fae740ce5f0 msgr2=0x7fae740b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:36.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.371+0000 7fae79400640 1 --2- 172.21.15.62:0/3777652481 >> v2:172.21.15.120:3300/0 conn(0x7fae740ce5f0 0x7fae740b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.371+0000 7fae79400640 1 -- 172.21.15.62:0/3777652481 >> v1:172.21.15.120:6789/0 conn(0x7fae7415da90 legacy=0x7fae740cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:36.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.371+0000 7fae79400640 1 -- 172.21.15.62:0/3777652481 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7406e100 con 0x7fae74159490 2024-03-23T11:19:36.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.372+0000 7fae79400640 1 --2- 172.21.15.62:0/3777652481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae74159490 0x7fae7406cc60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fae740ad990 tx=0x7fae6802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceac0ef5c9679998 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:36.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.372+0000 7fae73e00640 1 -- 172.21.15.62:0/3777652481 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae680317e0 con 0x7fae74159490 2024-03-23T11:19:36.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.373+0000 7fae73e00640 1 -- 172.21.15.62:0/3777652481 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae68031940 con 0x7fae74159490 2024-03-23T11:19:36.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.373+0000 7fae73e00640 1 -- 172.21.15.62:0/3777652481 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68031cd0 con 0x7fae74159490 2024-03-23T11:19:36.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.373+0000 7fae7a800640 1 -- 172.21.15.62:0/3777652481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae74159490 msgr2=0x7fae7406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.373+0000 7fae7a800640 1 --2- 172.21.15.62:0/3777652481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae74159490 0x7fae7406cc60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fae740ad990 tx=0x7fae6802b880 comp rx=0 tx=0).stop 2024-03-23T11:19:36.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.373+0000 7fae7a800640 1 -- 172.21.15.62:0/3777652481 shutdown_connections 2024-03-23T11:19:36.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.373+0000 7fae7a800640 1 --2- 172.21.15.62:0/3777652481 >> v2:172.21.15.120:3300/0 conn(0x7fae740ce5f0 0x7fae740b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.373+0000 7fae7a800640 1 --2- 172.21.15.62:0/3777652481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae74159490 0x7fae7406cc60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.373+0000 7fae7a800640 1 -- 172.21.15.62:0/3777652481 >> 172.21.15.62:0/3777652481 conn(0x7fae74066050 msgr2=0x7fae74069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:36.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.373+0000 7fae7a800640 1 -- 172.21.15.62:0/3777652481 shutdown_connections 2024-03-23T11:19:36.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.373+0000 7fae7a800640 1 -- 172.21.15.62:0/3777652481 wait complete. 2024-03-23T11:19:36.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.375+0000 7fae7a800640 1 Processor -- start 2024-03-23T11:19:36.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.375+0000 7fae7a800640 1 -- start start 2024-03-23T11:19:36.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.376+0000 7fae7a800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae740ce5f0 0x7fae7414de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.376+0000 7fae7a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae74159490 0x7fae74166c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.376+0000 7fae7a800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae7415da90 0x7fae74139870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.376+0000 7fae7a800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7415b380 con 0x7fae74159490 2024-03-23T11:19:36.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.376+0000 7fae7a800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7414e380 con 0x7fae7415da90 2024-03-23T11:19:36.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.376+0000 7fae7a800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7414e650 con 0x7fae740ce5f0 2024-03-23T11:19:36.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.377+0000 7fae78a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae74159490 0x7fae74166c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.377+0000 7fae78a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae74159490 0x7fae74166c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50330/0 (socket says 172.21.15.62:50330) 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.377+0000 7fae78a00640 1 -- 172.21.15.62:0/4203025970 learned_addr learned my addr 172.21.15.62:0/4203025970 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.377+0000 7fae79e00640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae7415da90 0x7fae74139870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.377+0000 7fae79400640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae740ce5f0 0x7fae7414de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.377+0000 7fae78a00640 1 -- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae740ce5f0 msgr2=0x7fae7414de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.377+0000 7fae78a00640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae740ce5f0 0x7fae7414de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.377+0000 7fae78a00640 1 -- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae7415da90 msgr2=0x7fae74139870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.377+0000 7fae78a00640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae7415da90 0x7fae74139870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.377+0000 7fae78a00640 1 -- 172.21.15.62:0/4203025970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae680094d0 con 0x7fae74159490 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.378+0000 7fae78a00640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae74159490 0x7fae74166c70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fae6000c430 tx=0x7fae6000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.378+0000 7fae71600640 1 -- 172.21.15.62:0/4203025970 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60011790 con 0x7fae74159490 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.378+0000 7fae71600640 1 -- 172.21.15.62:0/4203025970 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae600118f0 con 0x7fae74159490 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.378+0000 7fae71600640 1 -- 172.21.15.62:0/4203025970 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60011ce0 con 0x7fae74159490 2024-03-23T11:19:36.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.378+0000 7fae7a800640 1 -- 172.21.15.62:0/4203025970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7414e930 con 0x7fae74159490 2024-03-23T11:19:36.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.378+0000 7fae7a800640 1 -- 172.21.15.62:0/4203025970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7414eef0 con 0x7fae74159490 2024-03-23T11:19:36.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.379+0000 7fae7a800640 1 -- 172.21.15.62:0/4203025970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7414f110 con 0x7fae74159490 2024-03-23T11:19:36.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.380+0000 7fae71600640 1 -- 172.21.15.62:0/4203025970 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fae6001c460 con 0x7fae74159490 2024-03-23T11:19:36.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.381+0000 7fae71600640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae5406a7a0 0x7fae5406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.381+0000 7fae71600640 1 -- 172.21.15.62:0/4203025970 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae60088d50 con 0x7fae74159490 2024-03-23T11:19:36.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.382+0000 7fae79400640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae5406a7a0 0x7fae5406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.383+0000 7fae79400640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae5406a7a0 0x7fae5406cc60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fae68036000 tx=0x7fae6803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:36.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.385+0000 7fae71600640 1 -- 172.21.15.62:0/4203025970 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae6008c030 con 0x7fae74159490 2024-03-23T11:19:36.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.582+0000 7fae7a800640 1 -- 172.21.15.62:0/4203025970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae7406d320 con 0x7fae74159490 2024-03-23T11:19:36.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.583+0000 7fae71600640 1 -- 172.21.15.62:0/4203025970 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae60054170 con 0x7fae74159490 2024-03-23T11:19:36.583 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:36.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 -- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae5406a7a0 msgr2=0x7fae5406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae5406a7a0 0x7fae5406cc60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fae68036000 tx=0x7fae6803d040 comp rx=0 tx=0).stop 2024-03-23T11:19:36.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 -- 172.21.15.62:0/4203025970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae74159490 msgr2=0x7fae74166c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae74159490 0x7fae74166c70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fae6000c430 tx=0x7fae6000c8f0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 -- 172.21.15.62:0/4203025970 shutdown_connections 2024-03-23T11:19:36.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae5406a7a0 0x7fae5406cc60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae74159490 0x7fae74166c70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae7415da90 0x7fae74139870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 --2- 172.21.15.62:0/4203025970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae740ce5f0 0x7fae7414de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 -- 172.21.15.62:0/4203025970 >> 172.21.15.62:0/4203025970 conn(0x7fae74066050 msgr2=0x7fae74069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:36.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 -- 172.21.15.62:0/4203025970 shutdown_connections 2024-03-23T11:19:36.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.586+0000 7fae7a800640 1 -- 172.21.15.62:0/4203025970 wait complete. 2024-03-23T11:19:36.605 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:36.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.830+0000 7f9138c00640 1 Processor -- start 2024-03-23T11:19:36.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.830+0000 7f9138c00640 1 -- start start 2024-03-23T11:19:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.832+0000 7f9138c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f913415cab0 0x7f91340b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.832+0000 7f9138c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913415ce90 con 0x7f91340abd10 2024-03-23T11:19:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.832+0000 7f9138c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91340b22a0 con 0x7f913406ac70 2024-03-23T11:19:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.832+0000 7f9138c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91340b2580 con 0x7f913415cab0 2024-03-23T11:19:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.832+0000 7f9131400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f91340abd10 0x7f913406ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53320/0 (socket says 172.21.15.62:53320) 2024-03-23T11:19:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.832+0000 7f9131400640 1 -- 172.21.15.62:0/2055539151 learned_addr learned my addr 172.21.15.62:0/2055539151 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.832+0000 7f9131e00640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f913415cab0 0x7f91340b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885997476 0 0) 0x7f913415ce90 con 0x7f91340abd10 2024-03-23T11:19:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9118003520 con 0x7f91340abd10 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3667285012 0 0) 0x7f9118003520 con 0x7f91340abd10 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f913415cab0 msgr2=0x7f91340b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f913415cab0 0x7f91340b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 >> v1:172.21.15.120:6789/0 conn(0x7f913406ac70 legacy=0x7f91340b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91340b2860 con 0x7f91340abd10 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f911c001670 con 0x7f91340abd10 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 >> v1:172.21.15.62:6789/0 conn(0x7f91340abd10 legacy=0x7f913406ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91180039a0 0x7f9118003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 0x7f9118004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f913415cab0 0x7f9118005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91340b2580 con 0x7f91180047c0 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911800ba30 con 0x7f91180039a0 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911800bbf0 con 0x7f913415cab0 2024-03-23T11:19:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.833+0000 7f9131400640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 0x7f9118004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.834+0000 7f9131e00640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91180039a0 0x7f9118003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.834+0000 7f9132800640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f913415cab0 0x7f9118005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.834+0000 7f9131400640 1 -- 172.21.15.62:0/2055539151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f913415cab0 msgr2=0x7f9118005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.834+0000 7f9131400640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f913415cab0 0x7f9118005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.834+0000 7f9131400640 1 -- 172.21.15.62:0/2055539151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91180039a0 msgr2=0x7f9118003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.834+0000 7f9131400640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91180039a0 0x7f9118003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.834+0000 7f9131400640 1 -- 172.21.15.62:0/2055539151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9118003520 con 0x7f91180047c0 2024-03-23T11:19:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.835+0000 7f9131400640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 0x7f9118004c40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f911c00c210 tx=0x7f91180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa89b962e5cc832 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.835+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c005ad0 con 0x7f91180047c0 2024-03-23T11:19:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.835+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911c005c30 con 0x7f91180047c0 2024-03-23T11:19:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.835+0000 7f9130a00640 1 -- 172.21.15.62:0/2055539151 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c02c870 con 0x7f91180047c0 2024-03-23T11:19:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.836+0000 7f9138c00640 1 -- 172.21.15.62:0/2055539151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 msgr2=0x7f9118004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.836+0000 7f9138c00640 1 --2- 172.21.15.62:0/2055539151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 0x7f9118004c40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f911c00c210 tx=0x7f91180057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.836+0000 7f9132800640 1 -- 172.21.15.62:0/2055539151 reap_dead start 2024-03-23T11:19:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.836+0000 7f9138c00640 1 -- 172.21.15.62:0/2055539151 shutdown_connections 2024-03-23T11:19:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.836+0000 7f9138c00640 1 -- 172.21.15.62:0/2055539151 >> 172.21.15.62:0/2055539151 conn(0x7f9134070ae0 msgr2=0x7f9134070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.836+0000 7f9138c00640 1 -- 172.21.15.62:0/2055539151 shutdown_connections 2024-03-23T11:19:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.836+0000 7f9138c00640 1 -- 172.21.15.62:0/2055539151 wait complete. 2024-03-23T11:19:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.838+0000 7f9138c00640 1 Processor -- start 2024-03-23T11:19:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.838+0000 7f9138c00640 1 -- start start 2024-03-23T11:19:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.839+0000 7f9138c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 0x7f913414da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.839+0000 7f9138c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9134150040 0x7f913414d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.839+0000 7f9138c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9134150740 0x7f9134150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.839+0000 7f9138c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91341f20d0 con 0x7f91180047c0 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.839+0000 7f9138c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9134151100 con 0x7f9134150740 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.839+0000 7f9138c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91341504c0 con 0x7f9134150040 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.839+0000 7f9131e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 0x7f913414da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.839+0000 7f9131e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 0x7f913414da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50342/0 (socket says 172.21.15.62:50342) 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.839+0000 7f9131e00640 1 -- 172.21.15.62:0/2500103711 learned_addr learned my addr 172.21.15.62:0/2500103711 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f9132800640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9134150740 0x7f9134150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f9131400640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9134150040 0x7f913414d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f9131e00640 1 -- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9134150040 msgr2=0x7f913414d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f9131e00640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9134150040 0x7f913414d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f9131e00640 1 -- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9134150740 msgr2=0x7f9134150bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f9131e00640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9134150740 0x7f9134150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f9131e00640 1 -- 172.21.15.62:0/2500103711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913415d8a0 con 0x7f91180047c0 2024-03-23T11:19:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f9131e00640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 0x7f913414da00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9124007f30 tx=0x7f9124011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f912a000640 1 -- 172.21.15.62:0/2500103711 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912400e030 con 0x7f91180047c0 2024-03-23T11:19:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f912a000640 1 -- 172.21.15.62:0/2500103711 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9124016470 con 0x7f91180047c0 2024-03-23T11:19:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f912a000640 1 -- 172.21.15.62:0/2500103711 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124016820 con 0x7f91180047c0 2024-03-23T11:19:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f9138c00640 1 -- 172.21.15.62:0/2500103711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913415db80 con 0x7f91180047c0 2024-03-23T11:19:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.840+0000 7f9138c00640 1 -- 172.21.15.62:0/2500103711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913415e140 con 0x7f91180047c0 2024-03-23T11:19:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.842+0000 7f9138c00640 1 -- 172.21.15.62:0/2500103711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91340672c0 con 0x7f91180047c0 2024-03-23T11:19:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.842+0000 7f912a000640 1 -- 172.21.15.62:0/2500103711 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9124016980 con 0x7f91180047c0 2024-03-23T11:19:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.843+0000 7f912a000640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f911006a7a0 0x7f911006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.843+0000 7f912a000640 1 -- 172.21.15.62:0/2500103711 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f912400f070 con 0x7f91180047c0 2024-03-23T11:19:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.844+0000 7f9131400640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f911006a7a0 0x7f911006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.845+0000 7f9131400640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f911006a7a0 0x7f911006cc60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f911c034000 tx=0x7f911c009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:36.848+0000 7f912a000640 1 -- 172.21.15.62:0/2500103711 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f91240551a0 con 0x7f91180047c0 2024-03-23T11:19:37.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.048+0000 7f9138c00640 1 -- 172.21.15.62:0/2500103711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f913406f4a0 con 0x7f91180047c0 2024-03-23T11:19:37.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.049+0000 7f912a000640 1 -- 172.21.15.62:0/2500103711 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f91240587c0 con 0x7f91180047c0 2024-03-23T11:19:37.049 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.052+0000 7f9138c00640 1 -- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f911006a7a0 msgr2=0x7f911006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.052+0000 7f9138c00640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f911006a7a0 0x7f911006cc60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f911c034000 tx=0x7f911c009030 comp rx=0 tx=0).stop 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.052+0000 7f9138c00640 1 -- 172.21.15.62:0/2500103711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 msgr2=0x7f913414da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.052+0000 7f9138c00640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 0x7f913414da00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9124007f30 tx=0x7f9124011660 comp rx=0 tx=0).stop 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.052+0000 7f9138c00640 1 -- 172.21.15.62:0/2500103711 shutdown_connections 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.052+0000 7f9138c00640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f911006a7a0 0x7f911006cc60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.052+0000 7f9138c00640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9134150740 0x7f9134150bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.052+0000 7f9138c00640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9134150040 0x7f913414d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.052+0000 7f9138c00640 1 --2- 172.21.15.62:0/2500103711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91180047c0 0x7f913414da00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.052+0000 7f9138c00640 1 -- 172.21.15.62:0/2500103711 >> 172.21.15.62:0/2500103711 conn(0x7f9134070ae0 msgr2=0x7f9134157310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.053+0000 7f9138c00640 1 -- 172.21.15.62:0/2500103711 shutdown_connections 2024-03-23T11:19:37.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.053+0000 7f9138c00640 1 -- 172.21.15.62:0/2500103711 wait complete. 2024-03-23T11:19:37.071 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:37.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.286+0000 7f9793e00640 1 Processor -- start 2024-03-23T11:19:37.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.286+0000 7f9793e00640 1 -- start start 2024-03-23T11:19:37.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.287+0000 7f9793e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794070090 0x7f97940704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.287+0000 7f9793e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f97940b2820 0x7f97940b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.287+0000 7f9793e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9794070a30 con 0x7f9794143dd0 2024-03-23T11:19:37.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.287+0000 7f9793e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97940cd140 con 0x7f97940b2820 2024-03-23T11:19:37.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.287+0000 7f9793e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97940b3180 con 0x7f9794070090 2024-03-23T11:19:37.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.288+0000 7f9792a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9794143dd0 0x7f979415abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36216/0 (socket says 172.21.15.62:36216) 2024-03-23T11:19:37.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.288+0000 7f9792a00640 1 -- 172.21.15.62:0/3586740308 learned_addr learned my addr 172.21.15.62:0/3586740308 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:37.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.288+0000 7f9793400640 1 --2- 172.21.15.62:0/3586740308 >> v2:172.21.15.120:3300/0 conn(0x7f97940b2820 0x7f97940b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.288+0000 7f9792000640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794070090 0x7f97940704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.288+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959475297 0 0) 0x7f9794070a30 con 0x7f9794143dd0 2024-03-23T11:19:37.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.288+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9778003520 con 0x7f9794143dd0 2024-03-23T11:19:37.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1353009731 0 0) 0x7f9778003520 con 0x7f9794143dd0 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794070090 msgr2=0x7f97940704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794070090 0x7f97940704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 >> v2:172.21.15.120:3300/0 conn(0x7f97940b2820 msgr2=0x7f97940b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 --2- 172.21.15.62:0/3586740308 >> v2:172.21.15.120:3300/0 conn(0x7f97940b2820 0x7f97940b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97940cd3e0 con 0x7f9794143dd0 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9784001670 con 0x7f9794143dd0 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 >> v1:172.21.15.62:6789/0 conn(0x7f9794143dd0 legacy=0x7f979415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794070090 0x7f9778003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9778003f40 0x7f97780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9778004d00 0x7f9778005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97940cd140 con 0x7f9778004d00 2024-03-23T11:19:37.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97940b3180 con 0x7f9778003f40 2024-03-23T11:19:37.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977800b980 con 0x7f9794070090 2024-03-23T11:19:37.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9793400640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9778004d00 0x7f9778005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9792000640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9778003f40 0x7f97780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.289+0000 7f9792a00640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794070090 0x7f9778003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.290+0000 7f9793400640 1 -- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794070090 msgr2=0x7f9778003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:37.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.290+0000 7f9793400640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794070090 0x7f9778003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.290+0000 7f9793400640 1 -- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9778003f40 msgr2=0x7f97780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:37.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.290+0000 7f9793400640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9778003f40 0x7f97780047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.290+0000 7f9793400640 1 -- 172.21.15.62:0/3586740308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778003520 con 0x7f9778004d00 2024-03-23T11:19:37.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.290+0000 7f9792000640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9778003f40 0x7f97780047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:37.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.291+0000 7f9793400640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9778004d00 0x7f9778005180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f978c00e170 tx=0x7f97780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0e7e74dce92979 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:37.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.291+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c02cc20 con 0x7f9778004d00 2024-03-23T11:19:37.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.291+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978c00f040 con 0x7f9778004d00 2024-03-23T11:19:37.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.291+0000 7f9791600640 1 -- 172.21.15.62:0/3586740308 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c034440 con 0x7f9778004d00 2024-03-23T11:19:37.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.292+0000 7f9793e00640 1 -- 172.21.15.62:0/3586740308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9778004d00 msgr2=0x7f9778005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.292+0000 7f9793e00640 1 --2- 172.21.15.62:0/3586740308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9778004d00 0x7f9778005180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f978c00e170 tx=0x7f97780057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.292+0000 7f9793400640 1 -- 172.21.15.62:0/3586740308 reap_dead start 2024-03-23T11:19:37.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.292+0000 7f9793e00640 1 -- 172.21.15.62:0/3586740308 shutdown_connections 2024-03-23T11:19:37.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.292+0000 7f9793e00640 1 -- 172.21.15.62:0/3586740308 >> 172.21.15.62:0/3586740308 conn(0x7f9794066050 msgr2=0x7f9794069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:37.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.292+0000 7f9793e00640 1 -- 172.21.15.62:0/3586740308 shutdown_connections 2024-03-23T11:19:37.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.292+0000 7f9793e00640 1 -- 172.21.15.62:0/3586740308 wait complete. 2024-03-23T11:19:37.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.294+0000 7f9793e00640 1 Processor -- start 2024-03-23T11:19:37.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.294+0000 7f9793e00640 1 -- start start 2024-03-23T11:19:37.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.295+0000 7f9793e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97941599c0 0x7f979413dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.295+0000 7f9793e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794159da0 0x7f979413f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.295+0000 7f9793e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97940a7850 0x7f979413fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.295+0000 7f9793e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979413e8c0 con 0x7f97940a7850 2024-03-23T11:19:37.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.295+0000 7f9793e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979413eba0 con 0x7f97941599c0 2024-03-23T11:19:37.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.295+0000 7f9793e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979413ee80 con 0x7f9794159da0 2024-03-23T11:19:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.295+0000 7f9793400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97940a7850 0x7f979413fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.295+0000 7f9793400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97940a7850 0x7f979413fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51082/0 (socket says 172.21.15.62:51082) 2024-03-23T11:19:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.295+0000 7f9793400640 1 -- 172.21.15.62:0/4223171676 learned_addr learned my addr 172.21.15.62:0/4223171676 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.296+0000 7f9792a00640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97941599c0 0x7f979413dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.296+0000 7f9792000640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794159da0 0x7f979413f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.296+0000 7f9793400640 1 -- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794159da0 msgr2=0x7f979413f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.296+0000 7f9793400640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794159da0 0x7f979413f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.296+0000 7f9793400640 1 -- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97941599c0 msgr2=0x7f979413dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.296+0000 7f9793400640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97941599c0 0x7f979413dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.296+0000 7f9793400640 1 -- 172.21.15.62:0/4223171676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979413f160 con 0x7f97940a7850 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.296+0000 7f9792a00640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97941599c0 0x7f979413dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.296+0000 7f9793400640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97940a7850 0x7f979413fea0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f978c00e170 tx=0x7f978c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.297+0000 7f978aa00640 1 -- 172.21.15.62:0/4223171676 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c02cea0 con 0x7f97940a7850 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.297+0000 7f978aa00640 1 -- 172.21.15.62:0/4223171676 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978c034d50 con 0x7f97940a7850 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.297+0000 7f978aa00640 1 -- 172.21.15.62:0/4223171676 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c003b90 con 0x7f97940a7850 2024-03-23T11:19:37.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.297+0000 7f9793e00640 1 -- 172.21.15.62:0/4223171676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979413f440 con 0x7f97940a7850 2024-03-23T11:19:37.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.297+0000 7f9793e00640 1 -- 172.21.15.62:0/4223171676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97941efee0 con 0x7f97940a7850 2024-03-23T11:19:37.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.298+0000 7f9793e00640 1 -- 172.21.15.62:0/4223171676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979413f630 con 0x7f97940a7850 2024-03-23T11:19:37.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.299+0000 7f978aa00640 1 -- 172.21.15.62:0/4223171676 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f978c003cf0 con 0x7f97940a7850 2024-03-23T11:19:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.300+0000 7f978aa00640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f977406a7d0 0x7f977406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.300+0000 7f978aa00640 1 -- 172.21.15.62:0/4223171676 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f978c0aa320 con 0x7f97940a7850 2024-03-23T11:19:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.300+0000 7f9792a00640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f977406a7d0 0x7f977406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.302+0000 7f9792a00640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f977406a7d0 0x7f977406cc90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9784000c00 tx=0x7f9784009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:37.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.304+0000 7f978aa00640 1 -- 172.21.15.62:0/4223171676 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f978c0756c0 con 0x7f97940a7850 2024-03-23T11:19:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.504+0000 7f9793e00640 1 -- 172.21.15.62:0/4223171676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9794066bb0 con 0x7f97940a7850 2024-03-23T11:19:37.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.506+0000 7f978aa00640 1 -- 172.21.15.62:0/4223171676 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f978c045090 con 0x7f97940a7850 2024-03-23T11:19:37.506 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:37.507 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:57.238201+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 -- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f977406a7d0 msgr2=0x7f977406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f977406a7d0 0x7f977406cc90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9784000c00 tx=0x7f9784009040 comp rx=0 tx=0).stop 2024-03-23T11:19:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 -- 172.21.15.62:0/4223171676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97940a7850 msgr2=0x7f979413fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97940a7850 0x7f979413fea0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f978c00e170 tx=0x7f978c009750 comp rx=0 tx=0).stop 2024-03-23T11:19:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 -- 172.21.15.62:0/4223171676 shutdown_connections 2024-03-23T11:19:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f977406a7d0 0x7f977406cc90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97940a7850 0x7f979413fea0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9794159da0 0x7f979413f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 --2- 172.21.15.62:0/4223171676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97941599c0 0x7f979413dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 -- 172.21.15.62:0/4223171676 >> 172.21.15.62:0/4223171676 conn(0x7f9794066050 msgr2=0x7f9794070470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 -- 172.21.15.62:0/4223171676 shutdown_connections 2024-03-23T11:19:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.509+0000 7f9793e00640 1 -- 172.21.15.62:0/4223171676 wait complete. 2024-03-23T11:19:37.526 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:37.527 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:37.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.732+0000 7fdd35e00640 1 Processor -- start 2024-03-23T11:19:37.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.733+0000 7fdd35e00640 1 -- start start 2024-03-23T11:19:37.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.734+0000 7fdd35e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdd3015cdc0 0x7fdd300d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.734+0000 7fdd35e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd300cf650 0x7fdd300cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.734+0000 7fdd35e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3006bfc0 con 0x7fdd3015c9e0 2024-03-23T11:19:37.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.734+0000 7fdd35e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd300d0020 con 0x7fdd3015cdc0 2024-03-23T11:19:37.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.734+0000 7fdd35e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd300ce140 con 0x7fdd300cf650 2024-03-23T11:19:37.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.735+0000 7fdd2ea00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fdd3015c9e0 0x7fdd301607c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:36288/0 (socket says 172.21.15.62:36288) 2024-03-23T11:19:37.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.735+0000 7fdd2ea00640 1 -- 172.21.15.62:0/1734142459 learned_addr learned my addr 172.21.15.62:0/1734142459 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:37.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.735+0000 7fdd2f400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd300cf650 0x7fdd300cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.735+0000 7fdd26000640 1 --2- 172.21.15.62:0/1734142459 >> v2:172.21.15.120:3300/0 conn(0x7fdd3015cdc0 0x7fdd300d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.736+0000 7fdd2f400640 1 -- 172.21.15.62:0/1734142459 >> v2:172.21.15.120:3300/0 conn(0x7fdd3015cdc0 msgr2=0x7fdd300d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:37.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.736+0000 7fdd2f400640 1 --2- 172.21.15.62:0/1734142459 >> v2:172.21.15.120:3300/0 conn(0x7fdd3015cdc0 0x7fdd300d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.736+0000 7fdd2f400640 1 -- 172.21.15.62:0/1734142459 >> v1:172.21.15.120:6789/0 conn(0x7fdd3015c9e0 legacy=0x7fdd301607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.736+0000 7fdd2f400640 1 -- 172.21.15.62:0/1734142459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd300b5260 con 0x7fdd300cf650 2024-03-23T11:19:37.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.737+0000 7fdd2f400640 1 --2- 172.21.15.62:0/1734142459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd300cf650 0x7fdd300cfae0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009d90 tx=0x7fdd2802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d00fa91d28141f4a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:37.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.737+0000 7fdd2e000640 1 -- 172.21.15.62:0/1734142459 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2802cc50 con 0x7fdd300cf650 2024-03-23T11:19:37.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.737+0000 7fdd2e000640 1 -- 172.21.15.62:0/1734142459 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2802cdb0 con 0x7fdd300cf650 2024-03-23T11:19:37.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.737+0000 7fdd2e000640 1 -- 172.21.15.62:0/1734142459 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28034660 con 0x7fdd300cf650 2024-03-23T11:19:37.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.738+0000 7fdd35e00640 1 -- 172.21.15.62:0/1734142459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd300cf650 msgr2=0x7fdd300cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.738+0000 7fdd35e00640 1 --2- 172.21.15.62:0/1734142459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd300cf650 0x7fdd300cfae0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009d90 tx=0x7fdd2802bea0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.738+0000 7fdd35e00640 1 -- 172.21.15.62:0/1734142459 shutdown_connections 2024-03-23T11:19:37.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.738+0000 7fdd35e00640 1 --2- 172.21.15.62:0/1734142459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd300cf650 0x7fdd300cfae0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.738+0000 7fdd35e00640 1 --2- 172.21.15.62:0/1734142459 >> v2:172.21.15.120:3300/0 conn(0x7fdd3015cdc0 0x7fdd300d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.738+0000 7fdd35e00640 1 -- 172.21.15.62:0/1734142459 >> 172.21.15.62:0/1734142459 conn(0x7fdd30071570 msgr2=0x7fdd30071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:37.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.738+0000 7fdd35e00640 1 -- 172.21.15.62:0/1734142459 shutdown_connections 2024-03-23T11:19:37.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.738+0000 7fdd35e00640 1 -- 172.21.15.62:0/1734142459 wait complete. 2024-03-23T11:19:37.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.740+0000 7fdd35e00640 1 Processor -- start 2024-03-23T11:19:37.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.740+0000 7fdd35e00640 1 -- start start 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.741+0000 7fdd35e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd300cf650 0x7fdd300c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.741+0000 7fdd35e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd3015c9e0 0x7fdd300b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.741+0000 7fdd35e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd3015cdc0 0x7fdd30153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.741+0000 7fdd35e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd300cad40 con 0x7fdd3015c9e0 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.741+0000 7fdd35e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd300b8270 con 0x7fdd300cf650 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.741+0000 7fdd35e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd301542e0 con 0x7fdd3015cdc0 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.741+0000 7fdd26000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd3015c9e0 0x7fdd300b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.741+0000 7fdd26000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd3015c9e0 0x7fdd300b7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51088/0 (socket says 172.21.15.62:51088) 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.741+0000 7fdd26000640 1 -- 172.21.15.62:0/2428164846 learned_addr learned my addr 172.21.15.62:0/2428164846 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd2ea00640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd300cf650 0x7fdd300c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd2f400640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd3015cdc0 0x7fdd30153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd26000640 1 -- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd3015cdc0 msgr2=0x7fdd30153da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd26000640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd3015cdc0 0x7fdd30153da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd26000640 1 -- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd300cf650 msgr2=0x7fdd300c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd26000640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd300cf650 0x7fdd300c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd26000640 1 -- 172.21.15.62:0/2428164846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28009a70 con 0x7fdd3015c9e0 2024-03-23T11:19:37.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd2ea00640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd300cf650 0x7fdd300c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:37.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd2f400640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd3015cdc0 0x7fdd30153da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:37.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd26000640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd3015c9e0 0x7fdd300b7d30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800e870 tx=0x7fdd1800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:37.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd27e00640 1 -- 172.21.15.62:0/2428164846 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd180094f0 con 0x7fdd3015c9e0 2024-03-23T11:19:37.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd27e00640 1 -- 172.21.15.62:0/2428164846 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1800c940 con 0x7fdd3015c9e0 2024-03-23T11:19:37.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd27e00640 1 -- 172.21.15.62:0/2428164846 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1800ccf0 con 0x7fdd3015c9e0 2024-03-23T11:19:37.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.742+0000 7fdd35e00640 1 -- 172.21.15.62:0/2428164846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd300ca0c0 con 0x7fdd3015c9e0 2024-03-23T11:19:37.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.743+0000 7fdd35e00640 1 -- 172.21.15.62:0/2428164846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd300ca680 con 0x7fdd3015c9e0 2024-03-23T11:19:37.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.744+0000 7fdd35e00640 1 -- 172.21.15.62:0/2428164846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd30066c90 con 0x7fdd3015c9e0 2024-03-23T11:19:37.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.744+0000 7fdd27e00640 1 -- 172.21.15.62:0/2428164846 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fdd18013470 con 0x7fdd3015c9e0 2024-03-23T11:19:37.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.745+0000 7fdd27e00640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdd0c06a7a0 0x7fdd0c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:37.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.745+0000 7fdd27e00640 1 -- 172.21.15.62:0/2428164846 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd18088e80 con 0x7fdd3015c9e0 2024-03-23T11:19:37.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.746+0000 7fdd2ea00640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdd0c06a7a0 0x7fdd0c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.747+0000 7fdd2ea00640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdd0c06a7a0 0x7fdd0c06cc60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd20005880 tx=0x7fdd20009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:37.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.750+0000 7fdd27e00640 1 -- 172.21.15.62:0/2428164846 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd18054290 con 0x7fdd3015c9e0 2024-03-23T11:19:37.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.949+0000 7fdd35e00640 1 -- 172.21.15.62:0/2428164846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd300d2360 con 0x7fdd3015c9e0 2024-03-23T11:19:37.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.950+0000 7fdd27e00640 1 -- 172.21.15.62:0/2428164846 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd180578b0 con 0x7fdd3015c9e0 2024-03-23T11:19:37.951 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:37.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 -- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdd0c06a7a0 msgr2=0x7fdd0c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdd0c06a7a0 0x7fdd0c06cc60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd20005880 tx=0x7fdd20009040 comp rx=0 tx=0).stop 2024-03-23T11:19:37.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 -- 172.21.15.62:0/2428164846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd3015c9e0 msgr2=0x7fdd300b7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:37.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd3015c9e0 0x7fdd300b7d30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800e870 tx=0x7fdd1800ed30 comp rx=0 tx=0).stop 2024-03-23T11:19:37.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 -- 172.21.15.62:0/2428164846 shutdown_connections 2024-03-23T11:19:37.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdd0c06a7a0 0x7fdd0c06cc60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd3015c9e0 0x7fdd300b7d30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd3015cdc0 0x7fdd30153da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 --2- 172.21.15.62:0/2428164846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd300cf650 0x7fdd300c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:37.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 -- 172.21.15.62:0/2428164846 >> 172.21.15.62:0/2428164846 conn(0x7fdd30071570 msgr2=0x7fdd3006b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:37.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 -- 172.21.15.62:0/2428164846 shutdown_connections 2024-03-23T11:19:37.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:37.954+0000 7fdd35e00640 1 -- 172.21.15.62:0/2428164846 wait complete. 2024-03-23T11:19:37.973 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:38.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.194+0000 7f7b17200640 1 Processor -- start 2024-03-23T11:19:38.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.195+0000 7f7b17200640 1 -- start start 2024-03-23T11:19:38.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.196+0000 7f7b17200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7b18160c00 0x7f7b1806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.196+0000 7f7b17200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b18160fe0 con 0x7f7b180cde90 2024-03-23T11:19:38.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.196+0000 7f7b17200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b180b1510 con 0x7f7b180cda20 2024-03-23T11:19:38.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.196+0000 7f7b17200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b180b1ad0 con 0x7f7b18160c00 2024-03-23T11:19:38.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.197+0000 7f7b16800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7b180cde90 0x7f7b180caf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36224/0 (socket says 172.21.15.62:36224) 2024-03-23T11:19:38.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.197+0000 7f7b16800640 1 -- 172.21.15.62:0/2417080299 learned_addr learned my addr 172.21.15.62:0/2417080299 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:38.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.197+0000 7f7b15e00640 1 --2- 172.21.15.62:0/2417080299 >> v2:172.21.15.120:3300/0 conn(0x7f7b18160c00 0x7f7b1806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.197+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019239372 0 0) 0x7f7b18160fe0 con 0x7f7b180cde90 2024-03-23T11:19:38.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.197+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7afc003520 con 0x7f7b180cde90 2024-03-23T11:19:38.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2867660529 0 0) 0x7f7afc003520 con 0x7f7b180cde90 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 >> v2:172.21.15.120:3300/0 conn(0x7f7b18160c00 msgr2=0x7f7b1806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 --2- 172.21.15.62:0/2417080299 >> v2:172.21.15.120:3300/0 conn(0x7f7b18160c00 0x7f7b1806d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 >> v1:172.21.15.120:6789/0 conn(0x7f7b180cda20 legacy=0x7f7b1806de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b180b17f0 con 0x7f7b180cde90 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7b100019f0 con 0x7f7b180cde90 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 >> v1:172.21.15.62:6789/0 conn(0x7f7b180cde90 legacy=0x7f7b180caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 --2- 172.21.15.62:0/2417080299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc003970 0x7f7afc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 --2- 172.21.15.62:0/2417080299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7afc0047c0 0x7f7afc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 --2- 172.21.15.62:0/2417080299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7afc005180 0x7f7afc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b180b1ad0 con 0x7f7afc003970 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc00c020 con 0x7f7afc005180 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc00c1e0 con 0x7f7afc0047c0 2024-03-23T11:19:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.198+0000 7f7b15e00640 1 --2- 172.21.15.62:0/2417080299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc003970 0x7f7afc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.199+0000 7f7b15e00640 1 -- 172.21.15.62:0/2417080299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7afc0047c0 msgr2=0x7f7afc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:38.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.199+0000 7f7b15e00640 1 --2- 172.21.15.62:0/2417080299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7afc0047c0 0x7f7afc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.199+0000 7f7b15e00640 1 -- 172.21.15.62:0/2417080299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7afc005180 msgr2=0x7f7afc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:38.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.199+0000 7f7b15e00640 1 --2- 172.21.15.62:0/2417080299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7afc005180 0x7f7afc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.199+0000 7f7b15e00640 1 -- 172.21.15.62:0/2417080299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc003520 con 0x7f7afc003970 2024-03-23T11:19:38.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.199+0000 7f7b16800640 1 -- 172.21.15.62:0/2417080299 reap_dead start 2024-03-23T11:19:38.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.200+0000 7f7b15e00640 1 --2- 172.21.15.62:0/2417080299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc003970 0x7f7afc003d90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009ac0 tx=0x7f7afc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2213ffa4159977a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:38.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.200+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0802c710 con 0x7f7afc003970 2024-03-23T11:19:38.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.200+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0802c870 con 0x7f7afc003970 2024-03-23T11:19:38.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.200+0000 7f7b15400640 1 -- 172.21.15.62:0/2417080299 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0802cc00 con 0x7f7afc003970 2024-03-23T11:19:38.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.200+0000 7f7b17200640 1 -- 172.21.15.62:0/2417080299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc003970 msgr2=0x7f7afc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.200+0000 7f7b17200640 1 --2- 172.21.15.62:0/2417080299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc003970 0x7f7afc003d90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009ac0 tx=0x7f7afc0057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.201+0000 7f7b17200640 1 -- 172.21.15.62:0/2417080299 shutdown_connections 2024-03-23T11:19:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.201+0000 7f7b17200640 1 --2- 172.21.15.62:0/2417080299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc003970 0x7f7afc003d90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.201+0000 7f7b17200640 1 -- 172.21.15.62:0/2417080299 >> 172.21.15.62:0/2417080299 conn(0x7f7b18066050 msgr2=0x7f7b18069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.201+0000 7f7b17200640 1 -- 172.21.15.62:0/2417080299 shutdown_connections 2024-03-23T11:19:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.201+0000 7f7b17200640 1 -- 172.21.15.62:0/2417080299 wait complete. 2024-03-23T11:19:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.203+0000 7f7b17200640 1 Processor -- start 2024-03-23T11:19:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.203+0000 7f7b17200640 1 -- start start 2024-03-23T11:19:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b17200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7afc003970 0x7f7b181571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b17200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b181f62d0 0x7f7b18161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b17200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b181616d0 0x7f7b1815df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b17200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b18157710 con 0x7f7b181616d0 2024-03-23T11:19:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b17200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b18161c30 con 0x7f7afc003970 2024-03-23T11:19:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b17200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b18161f10 con 0x7f7b181f62d0 2024-03-23T11:19:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b16800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b181616d0 0x7f7b1815df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b16800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b181616d0 0x7f7b1815df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51112/0 (socket says 172.21.15.62:51112) 2024-03-23T11:19:38.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b16800640 1 -- 172.21.15.62:0/3309506449 learned_addr learned my addr 172.21.15.62:0/3309506449 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b15e00640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7afc003970 0x7f7b181571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b0fe00640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b181f62d0 0x7f7b18161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b16800640 1 -- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b181f62d0 msgr2=0x7f7b18161190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b16800640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b181f62d0 0x7f7b18161190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b16800640 1 -- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7afc003970 msgr2=0x7f7b181571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b16800640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7afc003970 0x7f7b181571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.204+0000 7f7b16800640 1 -- 172.21.15.62:0/3309506449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b181621f0 con 0x7f7b181616d0 2024-03-23T11:19:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.205+0000 7f7b16800640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b181616d0 0x7f7b1815df10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7b100012e0 tx=0x7f7b1000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.205+0000 7f7b0e000640 1 -- 172.21.15.62:0/3309506449 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1000cea0 con 0x7f7b181616d0 2024-03-23T11:19:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.205+0000 7f7b0e000640 1 -- 172.21.15.62:0/3309506449 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1000a610 con 0x7f7b181616d0 2024-03-23T11:19:38.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.205+0000 7f7b0e000640 1 -- 172.21.15.62:0/3309506449 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1000a9c0 con 0x7f7b181616d0 2024-03-23T11:19:38.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.205+0000 7f7b17200640 1 -- 172.21.15.62:0/3309506449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b18162b80 con 0x7f7b181616d0 2024-03-23T11:19:38.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.205+0000 7f7b17200640 1 -- 172.21.15.62:0/3309506449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1815e450 con 0x7f7b181616d0 2024-03-23T11:19:38.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.206+0000 7f7b17200640 1 -- 172.21.15.62:0/3309506449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b181629e0 con 0x7f7b181616d0 2024-03-23T11:19:38.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.207+0000 7f7b0e000640 1 -- 172.21.15.62:0/3309506449 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7b1000ab20 con 0x7f7b181616d0 2024-03-23T11:19:38.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.208+0000 7f7b0e000640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7af406a7a0 0x7f7af406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.208+0000 7f7b0e000640 1 -- 172.21.15.62:0/3309506449 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b10088b10 con 0x7f7b181616d0 2024-03-23T11:19:38.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.209+0000 7f7b15e00640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7af406a7a0 0x7f7af406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.210+0000 7f7b15e00640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7af406a7a0 0x7f7af406cc60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7b08036000 tx=0x7f7b08030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:38.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.213+0000 7f7b0e000640 1 -- 172.21.15.62:0/3309506449 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b181629e0 con 0x7f7b181616d0 2024-03-23T11:19:38.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.410+0000 7f7b17200640 1 -- 172.21.15.62:0/3309506449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7b181629e0 con 0x7f7b181616d0 2024-03-23T11:19:38.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.411+0000 7f7b0e000640 1 -- 172.21.15.62:0/3309506449 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7b181629e0 con 0x7f7b181616d0 2024-03-23T11:19:38.412 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:38.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 -- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7af406a7a0 msgr2=0x7f7af406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7af406a7a0 0x7f7af406cc60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7b08036000 tx=0x7f7b08030000 comp rx=0 tx=0).stop 2024-03-23T11:19:38.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 -- 172.21.15.62:0/3309506449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b181616d0 msgr2=0x7f7b1815df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b181616d0 0x7f7b1815df10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7b100012e0 tx=0x7f7b1000ccb0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 -- 172.21.15.62:0/3309506449 shutdown_connections 2024-03-23T11:19:38.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7af406a7a0 0x7f7af406cc60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b181616d0 0x7f7b1815df10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b181f62d0 0x7f7b18161190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 --2- 172.21.15.62:0/3309506449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7afc003970 0x7f7b181571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 -- 172.21.15.62:0/3309506449 >> 172.21.15.62:0/3309506449 conn(0x7f7b18066050 msgr2=0x7f7b1815fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:38.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 -- 172.21.15.62:0/3309506449 shutdown_connections 2024-03-23T11:19:38.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.414+0000 7f7b17200640 1 -- 172.21.15.62:0/3309506449 wait complete. 2024-03-23T11:19:38.434 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:38.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.655+0000 7f978b000640 1 Processor -- start 2024-03-23T11:19:38.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.656+0000 7f978b000640 1 -- start start 2024-03-23T11:19:38.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.657+0000 7f978b000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f978415cbb0 0x7f97840996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.657+0000 7f978b000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f97840990e0 0x7f97840b1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.657+0000 7f978b000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97840cab10 con 0x7f978406bfe0 2024-03-23T11:19:38.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.657+0000 7f978b000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97840cb3b0 con 0x7f97840990e0 2024-03-23T11:19:38.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.657+0000 7f978b000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97840afe20 con 0x7f978415cbb0 2024-03-23T11:19:38.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.658+0000 7f978a600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f97840990e0 0x7f97840b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.658+0000 7f978a600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f97840990e0 0x7f97840b1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51122/0 (socket says 172.21.15.62:51122) 2024-03-23T11:19:38.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.658+0000 7f978a600640 1 -- 172.21.15.62:0/1904250875 learned_addr learned my addr 172.21.15.62:0/1904250875 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:38.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.658+0000 7f9789c00640 1 --2- 172.21.15.62:0/1904250875 >> v2:172.21.15.120:3300/0 conn(0x7f978415cbb0 0x7f97840996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.658+0000 7f978a600640 1 -- 172.21.15.62:0/1904250875 >> v2:172.21.15.120:3300/0 conn(0x7f978415cbb0 msgr2=0x7f97840996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:38.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.658+0000 7f978a600640 1 --2- 172.21.15.62:0/1904250875 >> v2:172.21.15.120:3300/0 conn(0x7f978415cbb0 0x7f97840996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.658+0000 7f978a600640 1 -- 172.21.15.62:0/1904250875 >> v1:172.21.15.120:6789/0 conn(0x7f978406bfe0 legacy=0x7f97840989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:38.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.658+0000 7f978a600640 1 -- 172.21.15.62:0/1904250875 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978406c750 con 0x7f97840990e0 2024-03-23T11:19:38.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.660+0000 7f978a600640 1 --2- 172.21.15.62:0/1904250875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97840990e0 0x7f97840b1010 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f978406d410 tx=0x7f977c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce9eaab8eb64638 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:38.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.660+0000 7f9783e00640 1 -- 172.21.15.62:0/1904250875 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c031870 con 0x7f97840990e0 2024-03-23T11:19:38.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.660+0000 7f9783e00640 1 -- 172.21.15.62:0/1904250875 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c0319d0 con 0x7f97840990e0 2024-03-23T11:19:38.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.660+0000 7f9783e00640 1 -- 172.21.15.62:0/1904250875 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c031d60 con 0x7f97840990e0 2024-03-23T11:19:38.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.661+0000 7f978b000640 1 -- 172.21.15.62:0/1904250875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97840990e0 msgr2=0x7f97840b1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.661+0000 7f978b000640 1 --2- 172.21.15.62:0/1904250875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97840990e0 0x7f97840b1010 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f978406d410 tx=0x7f977c02b910 comp rx=0 tx=0).stop 2024-03-23T11:19:38.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.661+0000 7f978b000640 1 -- 172.21.15.62:0/1904250875 shutdown_connections 2024-03-23T11:19:38.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.661+0000 7f978b000640 1 --2- 172.21.15.62:0/1904250875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97840990e0 0x7f97840b1010 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.661+0000 7f978b000640 1 --2- 172.21.15.62:0/1904250875 >> v2:172.21.15.120:3300/0 conn(0x7f978415cbb0 0x7f97840996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.661+0000 7f978b000640 1 -- 172.21.15.62:0/1904250875 >> 172.21.15.62:0/1904250875 conn(0x7f97840b1fb0 msgr2=0x7f97840b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:38.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.661+0000 7f978b000640 1 -- 172.21.15.62:0/1904250875 shutdown_connections 2024-03-23T11:19:38.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.661+0000 7f978b000640 1 -- 172.21.15.62:0/1904250875 wait complete. 2024-03-23T11:19:38.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.663+0000 7f978b000640 1 Processor -- start 2024-03-23T11:19:38.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.663+0000 7f978b000640 1 -- start start 2024-03-23T11:19:38.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f978b000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f978406bfe0 0x7f97840b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f978b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97840990e0 0x7f9784159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f978b000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978415cbb0 0x7f978415e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f978b000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97840c7e30 con 0x7f978415cbb0 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f978b000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97840b40d0 con 0x7f978406bfe0 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f978b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97840b43b0 con 0x7f97840990e0 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f978a600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978415cbb0 0x7f978415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f978a600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978415cbb0 0x7f978415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51136/0 (socket says 172.21.15.62:51136) 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f978a600640 1 -- 172.21.15.62:0/3602904147 learned_addr learned my addr 172.21.15.62:0/3602904147 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f9789c00640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f978406bfe0 0x7f97840b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f9789200640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97840990e0 0x7f9784159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.664+0000 7f978a600640 1 -- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97840990e0 msgr2=0x7f9784159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.665+0000 7f978a600640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97840990e0 0x7f9784159af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.665+0000 7f978a600640 1 -- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f978406bfe0 msgr2=0x7f97840b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.665+0000 7f978a600640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f978406bfe0 0x7f97840b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.665+0000 7f978a600640 1 -- 172.21.15.62:0/3602904147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c0094d0 con 0x7f978415cbb0 2024-03-23T11:19:38.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.665+0000 7f9789200640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97840990e0 0x7f9784159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:38.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.665+0000 7f9789c00640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f978406bfe0 0x7f97840b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:38.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.665+0000 7f978a600640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978415cbb0 0x7f978415e590 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f977c0090b0 tx=0x7f977c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:38.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.665+0000 7f9781600640 1 -- 172.21.15.62:0/3602904147 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c03ac60 con 0x7f978415cbb0 2024-03-23T11:19:38.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.665+0000 7f9781600640 1 -- 172.21.15.62:0/3602904147 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c03adc0 con 0x7f978415cbb0 2024-03-23T11:19:38.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.665+0000 7f978b000640 1 -- 172.21.15.62:0/3602904147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97840b4690 con 0x7f978415cbb0 2024-03-23T11:19:38.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.665+0000 7f9781600640 1 -- 172.21.15.62:0/3602904147 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c044630 con 0x7f978415cbb0 2024-03-23T11:19:38.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.666+0000 7f978b000640 1 -- 172.21.15.62:0/3602904147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978415f230 con 0x7f978415cbb0 2024-03-23T11:19:38.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.667+0000 7f978b000640 1 -- 172.21.15.62:0/3602904147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97840672c0 con 0x7f978415cbb0 2024-03-23T11:19:38.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.668+0000 7f9781600640 1 -- 172.21.15.62:0/3602904147 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f977c04d450 con 0x7f978415cbb0 2024-03-23T11:19:38.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.669+0000 7f9781600640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f976806a7a0 0x7f976806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:38.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.669+0000 7f9781600640 1 -- 172.21.15.62:0/3602904147 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f977c02d040 con 0x7f978415cbb0 2024-03-23T11:19:38.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.669+0000 7f9789c00640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f976806a7a0 0x7f976806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:38.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.670+0000 7f9789c00640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f976806a7a0 0x7f976806cc60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9778006fd0 tx=0x7f9778008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:38.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.673+0000 7f9781600640 1 -- 172.21.15.62:0/3602904147 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f977c076810 con 0x7f978415cbb0 2024-03-23T11:19:38.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.877+0000 7f978b000640 1 -- 172.21.15.62:0/3602904147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f97840cf430 con 0x7f978415cbb0 2024-03-23T11:19:38.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.878+0000 7f9781600640 1 -- 172.21.15.62:0/3602904147 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f977c035020 con 0x7f978415cbb0 2024-03-23T11:19:38.878 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:38.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 -- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f976806a7a0 msgr2=0x7f976806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f976806a7a0 0x7f976806cc60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9778006fd0 tx=0x7f9778008040 comp rx=0 tx=0).stop 2024-03-23T11:19:38.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 -- 172.21.15.62:0/3602904147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978415cbb0 msgr2=0x7f978415e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:38.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978415cbb0 0x7f978415e590 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f977c0090b0 tx=0x7f977c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 -- 172.21.15.62:0/3602904147 shutdown_connections 2024-03-23T11:19:38.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f976806a7a0 0x7f976806cc60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f978415cbb0 0x7f978415e590 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f97840990e0 0x7f9784159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 --2- 172.21.15.62:0/3602904147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f978406bfe0 0x7f97840b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:38.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 -- 172.21.15.62:0/3602904147 >> 172.21.15.62:0/3602904147 conn(0x7f97840b1fb0 msgr2=0x7f97840af1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:38.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 -- 172.21.15.62:0/3602904147 shutdown_connections 2024-03-23T11:19:38.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:38.881+0000 7f978b000640 1 -- 172.21.15.62:0/3602904147 wait complete. 2024-03-23T11:19:38.934 INFO:tasks.cephfs_test_runner:Starting test: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-23T11:19:38.934 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-03-23T11:19:39.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.155+0000 7f82ce800640 1 Processor -- start 2024-03-23T11:19:39.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.156+0000 7f82ce800640 1 -- start start 2024-03-23T11:19:39.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.157+0000 7f82ce800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f82c8070090 0x7f82c8070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:39.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.157+0000 7f82ce800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82c80b2820 0x7f82c80b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:39.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.157+0000 7f82ce800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c80b31b0 con 0x7f82c8143dd0 2024-03-23T11:19:39.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.157+0000 7f82ce800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c8070a60 con 0x7f82c8070090 2024-03-23T11:19:39.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.157+0000 7f82ce800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80cd140 con 0x7f82c80b2820 2024-03-23T11:19:39.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.158+0000 7f82cca00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f82c8070090 0x7f82c8070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:39.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.158+0000 7f82cde00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82c80b2820 0x7f82c80b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:39.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.158+0000 7f82cca00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f82c8070090 0x7f82c8070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:46782/0 (socket says 172.21.15.62:46782) 2024-03-23T11:19:39.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.158+0000 7f82cca00640 1 -- 172.21.15.62:0/467008124 learned_addr learned my addr 172.21.15.62:0/467008124 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:39.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.158+0000 7f82cca00640 1 -- 172.21.15.62:0/467008124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82c80b2820 msgr2=0x7f82c80b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:39.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.158+0000 7f82cca00640 1 --2- 172.21.15.62:0/467008124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82c80b2820 0x7f82c80b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:39.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.158+0000 7f82cca00640 1 -- 172.21.15.62:0/467008124 >> v1:172.21.15.120:6789/0 conn(0x7f82c8143dd0 legacy=0x7f82c815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:39.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.158+0000 7f82cca00640 1 -- 172.21.15.62:0/467008124 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c80cd3e0 con 0x7f82c8070090 2024-03-23T11:19:39.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.160+0000 7f82cca00640 1 --2- 172.21.15.62:0/467008124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82c8070090 0x7f82c8070520 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009870 tx=0x7f82b402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22ff2649ae5954f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:39.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.160+0000 7f82c7e00640 1 -- 172.21.15.62:0/467008124 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4031640 con 0x7f82c8070090 2024-03-23T11:19:39.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.160+0000 7f82c7e00640 1 -- 172.21.15.62:0/467008124 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b40317a0 con 0x7f82c8070090 2024-03-23T11:19:39.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.160+0000 7f82c7e00640 1 -- 172.21.15.62:0/467008124 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4031b30 con 0x7f82c8070090 2024-03-23T11:19:39.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.161+0000 7f82ce800640 1 -- 172.21.15.62:0/467008124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82c8070090 msgr2=0x7f82c8070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:39.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.161+0000 7f82ce800640 1 --2- 172.21.15.62:0/467008124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82c8070090 0x7f82c8070520 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009870 tx=0x7f82b402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:19:39.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.161+0000 7f82ce800640 1 -- 172.21.15.62:0/467008124 shutdown_connections 2024-03-23T11:19:39.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.161+0000 7f82ce800640 1 --2- 172.21.15.62:0/467008124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82c80b2820 0x7f82c80b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:39.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.161+0000 7f82ce800640 1 --2- 172.21.15.62:0/467008124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82c8070090 0x7f82c8070520 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:39.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.161+0000 7f82ce800640 1 -- 172.21.15.62:0/467008124 >> 172.21.15.62:0/467008124 conn(0x7f82c8066050 msgr2=0x7f82c8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:39.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.161+0000 7f82ce800640 1 -- 172.21.15.62:0/467008124 shutdown_connections 2024-03-23T11:19:39.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.161+0000 7f82ce800640 1 -- 172.21.15.62:0/467008124 wait complete. 2024-03-23T11:19:39.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.163+0000 7f82ce800640 1 Processor -- start 2024-03-23T11:19:39.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.163+0000 7f82ce800640 1 -- start start 2024-03-23T11:19:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.164+0000 7f82ce800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82c8070090 0x7f82c81f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.164+0000 7f82ce800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82c80b2820 0x7f82c80ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.164+0000 7f82ce800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82c8143dd0 0x7f82c813f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.164+0000 7f82ce800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80acdc0 con 0x7f82c80b2820 2024-03-23T11:19:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.164+0000 7f82ce800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c81f2fe0 con 0x7f82c8143dd0 2024-03-23T11:19:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.164+0000 7f82ce800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80ac1b0 con 0x7f82c8070090 2024-03-23T11:19:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.164+0000 7f82cca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82c80b2820 0x7f82c80ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.164+0000 7f82cca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82c80b2820 0x7f82c80ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51140/0 (socket says 172.21.15.62:51140) 2024-03-23T11:19:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.164+0000 7f82cca00640 1 -- 172.21.15.62:0/1318079170 learned_addr learned my addr 172.21.15.62:0/1318079170 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.164+0000 7f82cde00640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82c8143dd0 0x7f82c813f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82cd400640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82c8070090 0x7f82c81f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82cca00640 1 -- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82c8070090 msgr2=0x7f82c81f2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82cca00640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82c8070090 0x7f82c81f2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82cca00640 1 -- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82c8143dd0 msgr2=0x7f82c813f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82cca00640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82c8143dd0 0x7f82c813f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82cca00640 1 -- 172.21.15.62:0/1318079170 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b40094d0 con 0x7f82c80b2820 2024-03-23T11:19:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82cd400640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82c8070090 0x7f82c81f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82cca00640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82c80b2820 0x7f82c80ab840 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f82b4000c00 tx=0x7f82b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82c5600640 1 -- 172.21.15.62:0/1318079170 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b403a930 con 0x7f82c80b2820 2024-03-23T11:19:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82c5600640 1 -- 172.21.15.62:0/1318079170 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b403aa90 con 0x7f82c80b2820 2024-03-23T11:19:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82ce800640 1 -- 172.21.15.62:0/1318079170 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c80ac490 con 0x7f82c80b2820 2024-03-23T11:19:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.165+0000 7f82c5600640 1 -- 172.21.15.62:0/1318079170 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b403ae40 con 0x7f82c80b2820 2024-03-23T11:19:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.166+0000 7f82ce800640 1 -- 172.21.15.62:0/1318079170 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c80aca50 con 0x7f82c80b2820 2024-03-23T11:19:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.167+0000 7f82ce800640 1 -- 172.21.15.62:0/1318079170 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c80acf70 con 0x7f82c80b2820 2024-03-23T11:19:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.168+0000 7f82c5600640 1 -- 172.21.15.62:0/1318079170 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f82b402f050 con 0x7f82c80b2820 2024-03-23T11:19:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.168+0000 7f82c5600640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f82a806a7a0 0x7f82a806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.169+0000 7f82c5600640 1 -- 172.21.15.62:0/1318079170 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82b40aad90 con 0x7f82c80b2820 2024-03-23T11:19:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.169+0000 7f82cd400640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f82a806a7a0 0x7f82a806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.170+0000 7f82cd400640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f82a806a7a0 0x7f82a806cc60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f82b8001730 tx=0x7f82b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:39.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.173+0000 7f82c5600640 1 -- 172.21.15.62:0/1318079170 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82c80acf70 con 0x7f82c80b2820 2024-03-23T11:19:39.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:39.446+0000 7f82ce800640 1 -- 172.21.15.62:0/1318079170 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) v1 -- 0x7f82c80acf70 con 0x7f82c80b2820 2024-03-23T11:19:40.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.375+0000 7f82c5600640 1 -- 172.21.15.62:0/1318079170 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v59) v1 ==== 145+0+0 (secure 0 0 0) 0x7f82c80acf70 con 0x7f82c80b2820 2024-03-23T11:19:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.379+0000 7f82ce800640 1 -- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f82a806a7a0 msgr2=0x7f82a806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.379+0000 7f82ce800640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f82a806a7a0 0x7f82a806cc60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f82b8001730 tx=0x7f82b8009040 comp rx=0 tx=0).stop 2024-03-23T11:19:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.379+0000 7f82ce800640 1 -- 172.21.15.62:0/1318079170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82c80b2820 msgr2=0x7f82c80ab840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.379+0000 7f82ce800640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82c80b2820 0x7f82c80ab840 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f82b4000c00 tx=0x7f82b40046c0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.380+0000 7f82ce800640 1 -- 172.21.15.62:0/1318079170 shutdown_connections 2024-03-23T11:19:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.380+0000 7f82ce800640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f82a806a7a0 0x7f82a806cc60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.380+0000 7f82ce800640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82c80b2820 0x7f82c80ab840 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.380+0000 7f82ce800640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82c8143dd0 0x7f82c813f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.380+0000 7f82ce800640 1 --2- 172.21.15.62:0/1318079170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82c8070090 0x7f82c81f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.380+0000 7f82ce800640 1 -- 172.21.15.62:0/1318079170 >> 172.21.15.62:0/1318079170 conn(0x7f82c8066050 msgr2=0x7f82c8072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:40.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.380+0000 7f82ce800640 1 -- 172.21.15.62:0/1318079170 shutdown_connections 2024-03-23T11:19:40.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.380+0000 7f82ce800640 1 -- 172.21.15.62:0/1318079170 wait complete. 2024-03-23T11:19:40.401 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T11:19:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.634+0000 7efd6dc00640 1 Processor -- start 2024-03-23T11:19:40.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.635+0000 7efd6dc00640 1 -- start start 2024-03-23T11:19:40.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.636+0000 7efd6dc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efd6815ca00 0x7efd680ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:40.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.636+0000 7efd6dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680b2bf0 0x7efd680d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:40.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.636+0000 7efd6dc00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd680cfe40 con 0x7efd680d0d40 2024-03-23T11:19:40.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.636+0000 7efd6dc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd680d1700 con 0x7efd6815ca00 2024-03-23T11:19:40.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.637+0000 7efd6dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd680cd680 con 0x7efd680b2bf0 2024-03-23T11:19:40.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.637+0000 7efd5f400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7efd680d0d40 0x7efd680b24e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36234/0 (socket says 172.21.15.62:36234) 2024-03-23T11:19:40.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.637+0000 7efd5f400640 1 -- 172.21.15.62:0/4040299384 learned_addr learned my addr 172.21.15.62:0/4040299384 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:40.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.637+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553552660 0 0) 0x7efd680cfe40 con 0x7efd680d0d40 2024-03-23T11:19:40.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.637+0000 7efd67e00640 1 --2- 172.21.15.62:0/4040299384 >> v2:172.21.15.120:3300/0 conn(0x7efd6815ca00 0x7efd680ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:40.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.637+0000 7efd6d200640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680b2bf0 0x7efd680d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:40.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.637+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd4c003520 con 0x7efd680d0d40 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1414447613 0 0) 0x7efd4c003520 con 0x7efd680d0d40 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680b2bf0 msgr2=0x7efd680d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680b2bf0 0x7efd680d11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 >> v2:172.21.15.120:3300/0 conn(0x7efd6815ca00 msgr2=0x7efd680ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 --2- 172.21.15.62:0/4040299384 >> v2:172.21.15.120:3300/0 conn(0x7efd6815ca00 0x7efd680ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd680cd960 con 0x7efd680d0d40 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7efd50001670 con 0x7efd680d0d40 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 >> v1:172.21.15.62:6789/0 conn(0x7efd680d0d40 legacy=0x7efd680b24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680b2bf0 0x7efd4c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd4c003f40 0x7efd4c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd4c004d00 0x7efd4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd680d1700 con 0x7efd4c003f40 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd680cd680 con 0x7efd4c004d00 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c00ba30 con 0x7efd680b2bf0 2024-03-23T11:19:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd67e00640 1 --2- 172.21.15.62:0/4040299384 >> v2:172.21.15.120:3300/0 conn(0x7efd6815ca00 0x7efd680ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.638+0000 7efd5f400640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd4c003f40 0x7efd4c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.639+0000 7efd6d200640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd4c004d00 0x7efd4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.639+0000 7efd5f400640 1 -- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680b2bf0 msgr2=0x7efd4c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.639+0000 7efd67e00640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680b2bf0 0x7efd4c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.639+0000 7efd5f400640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680b2bf0 0x7efd4c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.639+0000 7efd5f400640 1 -- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd4c004d00 msgr2=0x7efd4c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.639+0000 7efd5f400640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd4c004d00 0x7efd4c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.639+0000 7efd5f400640 1 -- 172.21.15.62:0/4040299384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4c003520 con 0x7efd4c003f40 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.639+0000 7efd67e00640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680b2bf0 0x7efd4c003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.640+0000 7efd5f400640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd4c003f40 0x7efd4c0047c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c110 tx=0x7efd4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23744afc41fd4882 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.640+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50005ad0 con 0x7efd4c003f40 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.640+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd50005c50 con 0x7efd4c003f40 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.640+0000 7efd67400640 1 -- 172.21.15.62:0/4040299384 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5002c830 con 0x7efd4c003f40 2024-03-23T11:19:40.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.641+0000 7efd6dc00640 1 -- 172.21.15.62:0/4040299384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd4c003f40 msgr2=0x7efd4c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.641+0000 7efd6dc00640 1 --2- 172.21.15.62:0/4040299384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd4c003f40 0x7efd4c0047c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c110 tx=0x7efd4c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.641+0000 7efd6d200640 1 -- 172.21.15.62:0/4040299384 reap_dead start 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.641+0000 7efd6dc00640 1 -- 172.21.15.62:0/4040299384 shutdown_connections 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.641+0000 7efd6dc00640 1 -- 172.21.15.62:0/4040299384 >> 172.21.15.62:0/4040299384 conn(0x7efd6809ba10 msgr2=0x7efd68089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.641+0000 7efd6dc00640 1 -- 172.21.15.62:0/4040299384 shutdown_connections 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.641+0000 7efd6dc00640 1 -- 172.21.15.62:0/4040299384 wait complete. 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.643+0000 7efd6dc00640 1 Processor -- start 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.643+0000 7efd6dc00640 1 -- start start 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.644+0000 7efd6dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd4c003f40 0x7efd6815ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.644+0000 7efd6dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680cba70 0x7efd68159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.644+0000 7efd6dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd680cef20 0x7efd680cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.644+0000 7efd6dc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6815d300 con 0x7efd4c003f40 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.644+0000 7efd6dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd680cf8e0 con 0x7efd680cef20 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.644+0000 7efd6dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd680b7c70 con 0x7efd680cba70 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.645+0000 7efd67e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd4c003f40 0x7efd6815ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:40.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.645+0000 7efd6d200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd680cef20 0x7efd680cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.645+0000 7efd6d200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd680cef20 0x7efd680cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:46814/0 (socket says 172.21.15.62:46814) 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.645+0000 7efd6d200640 1 -- 172.21.15.62:0/3317643862 learned_addr learned my addr 172.21.15.62:0/3317643862 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.645+0000 7efd5f400640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680cba70 0x7efd68159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.645+0000 7efd67e00640 1 -- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680cba70 msgr2=0x7efd68159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.645+0000 7efd67e00640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680cba70 0x7efd68159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.645+0000 7efd67e00640 1 -- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd680cef20 msgr2=0x7efd680cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.645+0000 7efd67e00640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd680cef20 0x7efd680cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.645+0000 7efd67e00640 1 -- 172.21.15.62:0/3317643862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6815cf60 con 0x7efd4c003f40 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.646+0000 7efd67e00640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd4c003f40 0x7efd6815ab80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efd5800f530 tx=0x7efd5800f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.646+0000 7efd64c00640 1 -- 172.21.15.62:0/3317643862 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58014890 con 0x7efd4c003f40 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.646+0000 7efd64c00640 1 -- 172.21.15.62:0/3317643862 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5800c030 con 0x7efd4c003f40 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.646+0000 7efd64c00640 1 -- 172.21.15.62:0/3317643862 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58014c30 con 0x7efd4c003f40 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.646+0000 7efd6dc00640 1 -- 172.21.15.62:0/3317643862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd680b7ec0 con 0x7efd4c003f40 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.646+0000 7efd6dc00640 1 -- 172.21.15.62:0/3317643862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd680cbef0 con 0x7efd4c003f40 2024-03-23T11:19:40.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.647+0000 7efd6dc00640 1 -- 172.21.15.62:0/3317643862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd680672c0 con 0x7efd4c003f40 2024-03-23T11:19:40.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.648+0000 7efd64c00640 1 -- 172.21.15.62:0/3317643862 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7efd580133e0 con 0x7efd4c003f40 2024-03-23T11:19:40.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.649+0000 7efd64c00640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd4406a7a0 0x7efd4406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:40.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.649+0000 7efd64c00640 1 -- 172.21.15.62:0/3317643862 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd58089110 con 0x7efd4c003f40 2024-03-23T11:19:40.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.650+0000 7efd5f400640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd4406a7a0 0x7efd4406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:40.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.651+0000 7efd5f400640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd4406a7a0 0x7efd4406cc60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efd50034000 tx=0x7efd50007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:40.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.654+0000 7efd64c00640 1 -- 172.21.15.62:0/3317643862 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efd580545d0 con 0x7efd4c003f40 2024-03-23T11:19:40.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.853+0000 7efd6dc00640 1 -- 172.21.15.62:0/3317643862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7efd680b8680 con 0x7efd4c003f40 2024-03-23T11:19:40.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.860+0000 7efd64c00640 1 -- 172.21.15.62:0/3317643862 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7efd58057bf0 con 0x7efd4c003f40 2024-03-23T11:19:40.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 -- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd4406a7a0 msgr2=0x7efd4406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:40.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd4406a7a0 0x7efd4406cc60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efd50034000 tx=0x7efd50007000 comp rx=0 tx=0).stop 2024-03-23T11:19:40.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 -- 172.21.15.62:0/3317643862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd4c003f40 msgr2=0x7efd6815ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:40.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd4c003f40 0x7efd6815ab80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efd5800f530 tx=0x7efd5800f9f0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 -- 172.21.15.62:0/3317643862 shutdown_connections 2024-03-23T11:19:40.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd4406a7a0 0x7efd4406cc60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd680cef20 0x7efd680cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd680cba70 0x7efd68159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 --2- 172.21.15.62:0/3317643862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd4c003f40 0x7efd6815ab80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:40.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 -- 172.21.15.62:0/3317643862 >> 172.21.15.62:0/3317643862 conn(0x7efd6809ba10 msgr2=0x7efd680b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:40.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 -- 172.21.15.62:0/3317643862 shutdown_connections 2024-03-23T11:19:40.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:40.863+0000 7efd6dc00640 1 -- 172.21.15.62:0/3317643862 wait complete. 2024-03-23T11:19:40.880 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:19:40.880 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:19:40.991 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:19:41.112 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:19:41.112 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:19:41.112 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:41.172 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:19:41.172 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:41.228 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:41.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.433+0000 7f800ce00640 1 Processor -- start 2024-03-23T11:19:41.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.433+0000 7f800ce00640 1 -- start start 2024-03-23T11:19:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.435+0000 7f800ce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8008160310 0x7f80080ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.435+0000 7f800ce00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f80080ce250 0x7f80080ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.435+0000 7f800ce00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80081608d0 con 0x7f800815c830 2024-03-23T11:19:41.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.435+0000 7f800ce00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80080addb0 con 0x7f80080ce250 2024-03-23T11:19:41.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.435+0000 7f800ce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800815cc10 con 0x7f8008160310 2024-03-23T11:19:41.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.435+0000 7f8007400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f800815c830 0x7f80080ab980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36238/0 (socket says 172.21.15.62:36238) 2024-03-23T11:19:41.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.435+0000 7f8007400640 1 -- 172.21.15.62:0/3685521156 learned_addr learned my addr 172.21.15.62:0/3685521156 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f8007e00640 1 --2- 172.21.15.62:0/3685521156 >> v2:172.21.15.120:3300/0 conn(0x7f80080ce250 0x7f80080ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f7ffea00640 1 --2- 172.21.15.62:0/3685521156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8008160310 0x7f80080ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679067773 0 0) 0x7f80081608d0 con 0x7f800815c830 2024-03-23T11:19:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fec003520 con 0x7f800815c830 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 731605181 0 0) 0x7f7fec003520 con 0x7f800815c830 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8008160310 msgr2=0x7f80080ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f8006a00640 1 --2- 172.21.15.62:0/3685521156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8008160310 0x7f80080ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 >> v2:172.21.15.120:3300/0 conn(0x7f80080ce250 msgr2=0x7f80080ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f8006a00640 1 --2- 172.21.15.62:0/3685521156 >> v2:172.21.15.120:3300/0 conn(0x7f80080ce250 0x7f80080ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8008160bb0 con 0x7f800815c830 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7ff80019f0 con 0x7f800815c830 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.436+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 >> v1:172.21.15.62:6789/0 conn(0x7f800815c830 legacy=0x7f80080ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f8006a00640 1 --2- 172.21.15.62:0/3685521156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fec0039a0 0x7f7fec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f8006a00640 1 --2- 172.21.15.62:0/3685521156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fec0047c0 0x7f7fec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f8006a00640 1 --2- 172.21.15.62:0/3685521156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8008160310 0x7f7fec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800815cc10 con 0x7f7fec0047c0 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec00b980 con 0x7f7fec0039a0 2024-03-23T11:19:41.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec00bac0 con 0x7f8008160310 2024-03-23T11:19:41.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f7ffea00640 1 --2- 172.21.15.62:0/3685521156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fec0047c0 0x7f7fec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f7ffea00640 1 -- 172.21.15.62:0/3685521156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8008160310 msgr2=0x7f7fec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:41.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f7ffea00640 1 --2- 172.21.15.62:0/3685521156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8008160310 0x7f7fec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f7ffea00640 1 -- 172.21.15.62:0/3685521156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fec0039a0 msgr2=0x7f7fec003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:41.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f7ffea00640 1 --2- 172.21.15.62:0/3685521156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fec0039a0 0x7f7fec003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.437+0000 7f7ffea00640 1 -- 172.21.15.62:0/3685521156 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec003520 con 0x7f7fec0047c0 2024-03-23T11:19:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.438+0000 7f7ffea00640 1 --2- 172.21.15.62:0/3685521156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fec0047c0 0x7f7fec004c40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0009ac0 tx=0x7f7fec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28eb6050c653b60a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.439+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff002c710 con 0x7f7fec0047c0 2024-03-23T11:19:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.439+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff002c870 con 0x7f7fec0047c0 2024-03-23T11:19:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.439+0000 7f8006a00640 1 -- 172.21.15.62:0/3685521156 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff002cc00 con 0x7f7fec0047c0 2024-03-23T11:19:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.439+0000 7f800ce00640 1 -- 172.21.15.62:0/3685521156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fec0047c0 msgr2=0x7f7fec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.439+0000 7f800ce00640 1 --2- 172.21.15.62:0/3685521156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fec0047c0 0x7f7fec004c40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0009ac0 tx=0x7f7fec0057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.439+0000 7f8007e00640 1 -- 172.21.15.62:0/3685521156 reap_dead start 2024-03-23T11:19:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.439+0000 7f800ce00640 1 -- 172.21.15.62:0/3685521156 shutdown_connections 2024-03-23T11:19:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.439+0000 7f800ce00640 1 -- 172.21.15.62:0/3685521156 >> 172.21.15.62:0/3685521156 conn(0x7f8008066050 msgr2=0x7f8008069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.439+0000 7f800ce00640 1 -- 172.21.15.62:0/3685521156 shutdown_connections 2024-03-23T11:19:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.439+0000 7f800ce00640 1 -- 172.21.15.62:0/3685521156 wait complete. 2024-03-23T11:19:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.441+0000 7f800ce00640 1 Processor -- start 2024-03-23T11:19:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.441+0000 7f800ce00640 1 -- start start 2024-03-23T11:19:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.442+0000 7f800ce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080aedc0 0x7f8008159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.442+0000 7f800ce00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80081409b0 0x7f8008151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.442+0000 7f800ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80080af1a0 0x7f8008153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.442+0000 7f800ce00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8008151340 con 0x7f80080af1a0 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.442+0000 7f800ce00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8008140e30 con 0x7f80081409b0 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.442+0000 7f800ce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8008151d40 con 0x7f80080aedc0 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f8007e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80080af1a0 0x7f8008153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f8007e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80080af1a0 0x7f8008153f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51184/0 (socket says 172.21.15.62:51184) 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f8007e00640 1 -- 172.21.15.62:0/2629234619 learned_addr learned my addr 172.21.15.62:0/2629234619 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f7ffea00640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80081409b0 0x7f8008151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f8007400640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080aedc0 0x7f8008159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f8007e00640 1 -- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080aedc0 msgr2=0x7f8008159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f8007e00640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080aedc0 0x7f8008159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f8007e00640 1 -- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80081409b0 msgr2=0x7f8008151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f8007e00640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80081409b0 0x7f8008151800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f8007e00640 1 -- 172.21.15.62:0/2629234619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80081531b0 con 0x7f80080af1a0 2024-03-23T11:19:41.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f8007400640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080aedc0 0x7f8008159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:41.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.443+0000 7f8007e00640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80080af1a0 0x7f8008153f30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f80080ce1f0 tx=0x7f800000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.444+0000 7f7fffe00640 1 -- 172.21.15.62:0/2629234619 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800000c680 con 0x7f80080af1a0 2024-03-23T11:19:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.444+0000 7f7fffe00640 1 -- 172.21.15.62:0/2629234619 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800000c7e0 con 0x7f80080af1a0 2024-03-23T11:19:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.444+0000 7f7fffe00640 1 -- 172.21.15.62:0/2629234619 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800000cb90 con 0x7f80080af1a0 2024-03-23T11:19:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.444+0000 7f800ce00640 1 -- 172.21.15.62:0/2629234619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800815a2b0 con 0x7f80080af1a0 2024-03-23T11:19:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.444+0000 7f800ce00640 1 -- 172.21.15.62:0/2629234619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8008153730 con 0x7f80080af1a0 2024-03-23T11:19:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.445+0000 7f800ce00640 1 -- 172.21.15.62:0/2629234619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8008161130 con 0x7f80080af1a0 2024-03-23T11:19:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.446+0000 7f7fffe00640 1 -- 172.21.15.62:0/2629234619 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f800000ccf0 con 0x7f80080af1a0 2024-03-23T11:19:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.447+0000 7f7fffe00640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fe406a7a0 0x7f7fe406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.447+0000 7f7fffe00640 1 -- 172.21.15.62:0/2629234619 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8000088b40 con 0x7f80080af1a0 2024-03-23T11:19:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.447+0000 7f8007400640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fe406a7a0 0x7f7fe406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.448+0000 7f8007400640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fe406a7a0 0x7f7fe406cc60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f80080ac750 tx=0x7f7ff8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:41.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.451+0000 7f7fffe00640 1 -- 172.21.15.62:0/2629234619 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8000053ec0 con 0x7f80080af1a0 2024-03-23T11:19:41.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.649+0000 7f800ce00640 1 -- 172.21.15.62:0/2629234619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8008066bb0 con 0x7f80080af1a0 2024-03-23T11:19:41.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.652+0000 7f7fffe00640 1 -- 172.21.15.62:0/2629234619 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16037 (secure 0 0 0) 0x7f80000574e0 con 0x7f80080af1a0 2024-03-23T11:19:41.652 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:41.652 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:18:57.238201+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.654+0000 7f800ce00640 1 -- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fe406a7a0 msgr2=0x7f7fe406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.654+0000 7f800ce00640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fe406a7a0 0x7f7fe406cc60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f80080ac750 tx=0x7f7ff8009040 comp rx=0 tx=0).stop 2024-03-23T11:19:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.654+0000 7f800ce00640 1 -- 172.21.15.62:0/2629234619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80080af1a0 msgr2=0x7f8008153f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.654+0000 7f800ce00640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80080af1a0 0x7f8008153f30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f80080ce1f0 tx=0x7f800000ec00 comp rx=0 tx=0).stop 2024-03-23T11:19:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.655+0000 7f800ce00640 1 -- 172.21.15.62:0/2629234619 shutdown_connections 2024-03-23T11:19:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.655+0000 7f800ce00640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fe406a7a0 0x7f7fe406cc60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.655+0000 7f800ce00640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80080af1a0 0x7f8008153f30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.655+0000 7f800ce00640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80081409b0 0x7f8008151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.655+0000 7f800ce00640 1 --2- 172.21.15.62:0/2629234619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80080aedc0 0x7f8008159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.655+0000 7f800ce00640 1 -- 172.21.15.62:0/2629234619 >> 172.21.15.62:0/2629234619 conn(0x7f8008066050 msgr2=0x7f80080adcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:41.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.655+0000 7f800ce00640 1 -- 172.21.15.62:0/2629234619 shutdown_connections 2024-03-23T11:19:41.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.655+0000 7f800ce00640 1 -- 172.21.15.62:0/2629234619 wait complete. 2024-03-23T11:19:41.674 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:41.674 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:41.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.897+0000 7f780b200640 1 Processor -- start 2024-03-23T11:19:41.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.897+0000 7f780b200640 1 -- start start 2024-03-23T11:19:41.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.898+0000 7f780b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f780c0996b0 0x7f780c06bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.898+0000 7f780b200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f780c06c520 con 0x7f780c15cbb0 2024-03-23T11:19:41.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.898+0000 7f780b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f780c0cab10 con 0x7f780c0990e0 2024-03-23T11:19:41.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.898+0000 7f780b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f780c0cb3b0 con 0x7f780c0996b0 2024-03-23T11:19:41.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.899+0000 7f7809e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f780c15cbb0 0x7f780c0989d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36250/0 (socket says 172.21.15.62:36250) 2024-03-23T11:19:41.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.899+0000 7f7809e00640 1 -- 172.21.15.62:0/3618929400 learned_addr learned my addr 172.21.15.62:0/3618929400 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:41.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.899+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464704000 0 0) 0x7f780c06c520 con 0x7f780c15cbb0 2024-03-23T11:19:41.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.899+0000 7f780a800640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f780c0996b0 0x7f780c06bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.899+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f0003520 con 0x7f780c15cbb0 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.899+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2019655930 0 0) 0x7f77f0003520 con 0x7f780c15cbb0 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.899+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f780c0996b0 msgr2=0x7f780c06bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.899+0000 7f7808a00640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f780c0996b0 0x7f780c06bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.899+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 >> v1:172.21.15.120:6789/0 conn(0x7f780c0990e0 legacy=0x7f780c0b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.899+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780c0afe20 con 0x7f780c15cbb0 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.899+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f77fc001670 con 0x7f780c15cbb0 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 >> v1:172.21.15.62:6789/0 conn(0x7f780c15cbb0 legacy=0x7f780c0989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7808a00640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77f00039a0 0x7f77f0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7808a00640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f780c0996b0 0x7f77f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7808a00640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77f0004d00 0x7f77f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c0cb3b0 con 0x7f77f00039a0 2024-03-23T11:19:41.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f000bb10 con 0x7f77f0004d00 2024-03-23T11:19:41.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f000bcd0 con 0x7f780c0996b0 2024-03-23T11:19:41.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7809e00640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77f00039a0 0x7f77f0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7809400640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f780c0996b0 0x7f77f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f780a800640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77f0004d00 0x7f77f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7809e00640 1 -- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f780c0996b0 msgr2=0x7f77f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:41.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7809e00640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f780c0996b0 0x7f77f00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7809e00640 1 -- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77f0004d00 msgr2=0x7f77f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:41.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7809e00640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77f0004d00 0x7f77f0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7809e00640 1 -- 172.21.15.62:0/3618929400 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f0003520 con 0x7f77f00039a0 2024-03-23T11:19:41.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.900+0000 7f7809400640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f780c0996b0 0x7f77f00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:41.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.901+0000 7f7809e00640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77f00039a0 0x7f77f0003dc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00c190 tx=0x7f77f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a839d44a385bf523 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:41.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.902+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc005ad0 con 0x7f77f00039a0 2024-03-23T11:19:41.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.902+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77fc005c30 con 0x7f77f00039a0 2024-03-23T11:19:41.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.902+0000 7f7808a00640 1 -- 172.21.15.62:0/3618929400 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc02c8a0 con 0x7f77f00039a0 2024-03-23T11:19:41.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.902+0000 7f780b200640 1 -- 172.21.15.62:0/3618929400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77f00039a0 msgr2=0x7f77f0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:41.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.902+0000 7f780b200640 1 --2- 172.21.15.62:0/3618929400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77f00039a0 0x7f77f0003dc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00c190 tx=0x7f77f00057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.902+0000 7f780a800640 1 -- 172.21.15.62:0/3618929400 reap_dead start 2024-03-23T11:19:41.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.902+0000 7f780b200640 1 -- 172.21.15.62:0/3618929400 shutdown_connections 2024-03-23T11:19:41.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.902+0000 7f780b200640 1 -- 172.21.15.62:0/3618929400 >> 172.21.15.62:0/3618929400 conn(0x7f780c0b1fb0 msgr2=0x7f780c0b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:41.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.903+0000 7f780b200640 1 -- 172.21.15.62:0/3618929400 shutdown_connections 2024-03-23T11:19:41.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.903+0000 7f780b200640 1 -- 172.21.15.62:0/3618929400 wait complete. 2024-03-23T11:19:41.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.904+0000 7f780b200640 1 Processor -- start 2024-03-23T11:19:41.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.904+0000 7f780b200640 1 -- start start 2024-03-23T11:19:41.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.905+0000 7f780b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77f00039a0 0x7f780c1581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.905+0000 7f780b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f780c158830 0x7f780c15fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.905+0000 7f780b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f780c159d00 0x7f780c15a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.905+0000 7f780b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c15a6f0 con 0x7f780c159d00 2024-03-23T11:19:41.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.905+0000 7f780b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c160060 con 0x7f780c158830 2024-03-23T11:19:41.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.905+0000 7f780b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f780c1f2a80 con 0x7f77f00039a0 2024-03-23T11:19:41.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.906+0000 7f780a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f780c159d00 0x7f780c15a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.906+0000 7f780a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f780c159d00 0x7f780c15a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51198/0 (socket says 172.21.15.62:51198) 2024-03-23T11:19:41.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.906+0000 7f780a800640 1 -- 172.21.15.62:0/426045993 learned_addr learned my addr 172.21.15.62:0/426045993 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:41.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.906+0000 7f7809e00640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77f00039a0 0x7f780c1581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.906+0000 7f7809400640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f780c158830 0x7f780c15fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.906+0000 7f780a800640 1 -- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77f00039a0 msgr2=0x7f780c1581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:41.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.906+0000 7f780a800640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77f00039a0 0x7f780c1581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.906+0000 7f780a800640 1 -- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f780c158830 msgr2=0x7f780c15fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:41.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.906+0000 7f780a800640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f780c158830 0x7f780c15fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:41.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.906+0000 7f780a800640 1 -- 172.21.15.62:0/426045993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780c1f2d60 con 0x7f780c159d00 2024-03-23T11:19:41.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.907+0000 7f780a800640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f780c159d00 0x7f780c15a1b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f780c0720c0 tx=0x7f7804009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:41.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.907+0000 7f7802000640 1 -- 172.21.15.62:0/426045993 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804007cf0 con 0x7f780c159d00 2024-03-23T11:19:41.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.907+0000 7f7802000640 1 -- 172.21.15.62:0/426045993 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7804007e70 con 0x7f780c159d00 2024-03-23T11:19:41.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.907+0000 7f780b200640 1 -- 172.21.15.62:0/426045993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f780c1f3040 con 0x7f780c159d00 2024-03-23T11:19:41.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.907+0000 7f7802000640 1 -- 172.21.15.62:0/426045993 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804014620 con 0x7f780c159d00 2024-03-23T11:19:41.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.907+0000 7f780b200640 1 -- 172.21.15.62:0/426045993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f780c1f3600 con 0x7f780c159d00 2024-03-23T11:19:41.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.908+0000 7f780b200640 1 -- 172.21.15.62:0/426045993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f780c1f3820 con 0x7f780c159d00 2024-03-23T11:19:41.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.909+0000 7f7802000640 1 -- 172.21.15.62:0/426045993 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f780401d3b0 con 0x7f780c159d00 2024-03-23T11:19:41.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.910+0000 7f7802000640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77e806a7a0 0x7f77e806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:41.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.910+0000 7f7802000640 1 -- 172.21.15.62:0/426045993 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7804088b00 con 0x7f780c159d00 2024-03-23T11:19:41.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.911+0000 7f7809e00640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77e806a7a0 0x7f77e806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:41.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.912+0000 7f7809e00640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77e806a7a0 0x7f77e806cc60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00c190 tx=0x7f77fc009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:41.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:41.915+0000 7f7802000640 1 -- 172.21.15.62:0/426045993 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f780408ae60 con 0x7f780c159d00 2024-03-23T11:19:42.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.113+0000 7f780b200640 1 -- 172.21.15.62:0/426045993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f780c0672c0 con 0x7f780c159d00 2024-03-23T11:19:42.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.113+0000 7f7802000640 1 -- 172.21.15.62:0/426045993 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7804019020 con 0x7f780c159d00 2024-03-23T11:19:42.114 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.116+0000 7f780b200640 1 -- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77e806a7a0 msgr2=0x7f77e806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.116+0000 7f780b200640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77e806a7a0 0x7f77e806cc60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00c190 tx=0x7f77fc009030 comp rx=0 tx=0).stop 2024-03-23T11:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.116+0000 7f780b200640 1 -- 172.21.15.62:0/426045993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f780c159d00 msgr2=0x7f780c15a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.116+0000 7f780b200640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f780c159d00 0x7f780c15a1b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f780c0720c0 tx=0x7f7804009d50 comp rx=0 tx=0).stop 2024-03-23T11:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.116+0000 7f780b200640 1 -- 172.21.15.62:0/426045993 shutdown_connections 2024-03-23T11:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.116+0000 7f780b200640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f77e806a7a0 0x7f77e806cc60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.116+0000 7f780b200640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f780c159d00 0x7f780c15a1b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.116+0000 7f780b200640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f780c158830 0x7f780c15fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.116+0000 7f780b200640 1 --2- 172.21.15.62:0/426045993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77f00039a0 0x7f780c1581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.116+0000 7f780b200640 1 -- 172.21.15.62:0/426045993 >> 172.21.15.62:0/426045993 conn(0x7f780c0b1fb0 msgr2=0x7f780c15bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.117+0000 7f780b200640 1 -- 172.21.15.62:0/426045993 shutdown_connections 2024-03-23T11:19:42.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.117+0000 7f780b200640 1 -- 172.21.15.62:0/426045993 wait complete. 2024-03-23T11:19:42.136 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:42.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.366+0000 7fb86c800640 1 Processor -- start 2024-03-23T11:19:42.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.366+0000 7fb86c800640 1 -- start start 2024-03-23T11:19:42.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.367+0000 7fb86c800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb86415c9e0 0x7fb8641607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.367+0000 7fb86c800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86406bfc0 con 0x7fb8640cf650 2024-03-23T11:19:42.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.367+0000 7fb86c800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8640b5260 con 0x7fb86415cdc0 2024-03-23T11:19:42.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.367+0000 7fb86c800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8640d2630 con 0x7fb86415c9e0 2024-03-23T11:19:42.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.367+0000 7fb86a800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb8640cf650 0x7fb86415d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36266/0 (socket says 172.21.15.62:36266) 2024-03-23T11:19:42.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.367+0000 7fb86a800640 1 -- 172.21.15.62:0/520393407 learned_addr learned my addr 172.21.15.62:0/520393407 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:42.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.368+0000 7fb869e00640 1 --2- 172.21.15.62:0/520393407 >> v2:172.21.15.120:3300/0 conn(0x7fb86415c9e0 0x7fb8641607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.368+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714783932 0 0) 0x7fb86406bfc0 con 0x7fb8640cf650 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.368+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84c003520 con 0x7fb8640cf650 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.368+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 218373135 0 0) 0x7fb84c003520 con 0x7fb8640cf650 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.368+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 >> v2:172.21.15.120:3300/0 conn(0x7fb86415c9e0 msgr2=0x7fb8641607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.368+0000 7fb868a00640 1 --2- 172.21.15.62:0/520393407 >> v2:172.21.15.120:3300/0 conn(0x7fb86415c9e0 0x7fb8641607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.368+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 >> v1:172.21.15.120:6789/0 conn(0x7fb86415cdc0 legacy=0x7fb8640d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.368+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8640ce140 con 0x7fb8640cf650 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.368+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fb8600019f0 con 0x7fb8640cf650 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 >> v1:172.21.15.62:6789/0 conn(0x7fb8640cf650 legacy=0x7fb86415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb868a00640 1 --2- 172.21.15.62:0/520393407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb84c0039c0 0x7fb84c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb868a00640 1 --2- 172.21.15.62:0/520393407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb84c0047c0 0x7fb84c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb868a00640 1 --2- 172.21.15.62:0/520393407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb84c005180 0x7fb84c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8640d2630 con 0x7fb84c005180 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c00c080 con 0x7fb84c0039c0 2024-03-23T11:19:42.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c00c240 con 0x7fb84c0047c0 2024-03-23T11:19:42.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb86a800640 1 --2- 172.21.15.62:0/520393407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb84c005180 0x7fb84c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb869e00640 1 --2- 172.21.15.62:0/520393407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb84c0039c0 0x7fb84c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb86a800640 1 -- 172.21.15.62:0/520393407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb84c0047c0 msgr2=0x7fb84c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:42.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb86a800640 1 --2- 172.21.15.62:0/520393407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb84c0047c0 0x7fb84c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb86a800640 1 -- 172.21.15.62:0/520393407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb84c0039c0 msgr2=0x7fb84c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:42.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb86a800640 1 --2- 172.21.15.62:0/520393407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb84c0039c0 0x7fb84c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb86a800640 1 -- 172.21.15.62:0/520393407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c003520 con 0x7fb84c005180 2024-03-23T11:19:42.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.369+0000 7fb869e00640 1 --2- 172.21.15.62:0/520393407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb84c0039c0 0x7fb84c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.370+0000 7fb86a800640 1 -- 172.21.15.62:0/520393407 reap_dead start 2024-03-23T11:19:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.371+0000 7fb86a800640 1 --2- 172.21.15.62:0/520393407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb84c005180 0x7fb84c00b980 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb86000a760 tx=0x7fb84c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe33ac2fec22e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.371+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb860034630 con 0x7fb84c005180 2024-03-23T11:19:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.371+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86002f070 con 0x7fb84c005180 2024-03-23T11:19:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.371+0000 7fb868a00640 1 -- 172.21.15.62:0/520393407 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb860034a10 con 0x7fb84c005180 2024-03-23T11:19:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.371+0000 7fb86c800640 1 -- 172.21.15.62:0/520393407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb84c005180 msgr2=0x7fb84c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.371+0000 7fb86c800640 1 --2- 172.21.15.62:0/520393407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb84c005180 0x7fb84c00b980 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb86000a760 tx=0x7fb84c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.372+0000 7fb86c800640 1 -- 172.21.15.62:0/520393407 shutdown_connections 2024-03-23T11:19:42.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.372+0000 7fb86c800640 1 --2- 172.21.15.62:0/520393407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb84c005180 0x7fb84c00b980 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.372+0000 7fb86c800640 1 -- 172.21.15.62:0/520393407 >> 172.21.15.62:0/520393407 conn(0x7fb864071570 msgr2=0x7fb864071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:42.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.372+0000 7fb86c800640 1 -- 172.21.15.62:0/520393407 shutdown_connections 2024-03-23T11:19:42.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.372+0000 7fb86c800640 1 -- 172.21.15.62:0/520393407 wait complete. 2024-03-23T11:19:42.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.374+0000 7fb86c800640 1 Processor -- start 2024-03-23T11:19:42.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.374+0000 7fb86c800640 1 -- start start 2024-03-23T11:19:42.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.375+0000 7fb86c800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb84c005180 0x7fb864158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.375+0000 7fb86c800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8640a9300 0x7fb8640b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.375+0000 7fb86c800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8640ca5d0 0x7fb86415d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.375+0000 7fb86c800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8640a9a70 con 0x7fb8640a9300 2024-03-23T11:19:42.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.375+0000 7fb86c800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8640a9790 con 0x7fb84c005180 2024-03-23T11:19:42.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.375+0000 7fb86c800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8640c9af0 con 0x7fb8640ca5d0 2024-03-23T11:19:42.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.375+0000 7fb869400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8640a9300 0x7fb8640b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.375+0000 7fb869400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8640a9300 0x7fb8640b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51218/0 (socket says 172.21.15.62:51218) 2024-03-23T11:19:42.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.375+0000 7fb869400640 1 -- 172.21.15.62:0/2976265341 learned_addr learned my addr 172.21.15.62:0/2976265341 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:42.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.375+0000 7fb86a800640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8640ca5d0 0x7fb86415d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.376+0000 7fb869400640 1 -- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8640ca5d0 msgr2=0x7fb86415d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.376+0000 7fb869e00640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb84c005180 0x7fb864158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.376+0000 7fb869400640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8640ca5d0 0x7fb86415d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.376+0000 7fb869400640 1 -- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb84c005180 msgr2=0x7fb864158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.376+0000 7fb869400640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb84c005180 0x7fb864158b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.376+0000 7fb869400640 1 -- 172.21.15.62:0/2976265341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8640c9dd0 con 0x7fb8640a9300 2024-03-23T11:19:42.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.376+0000 7fb86a800640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8640ca5d0 0x7fb86415d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:42.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.376+0000 7fb869400640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8640a9300 0x7fb8640b2c90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb85400c500 tx=0x7fb85400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:42.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.377+0000 7fb85e000640 1 -- 172.21.15.62:0/2976265341 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8540052d0 con 0x7fb8640a9300 2024-03-23T11:19:42.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.377+0000 7fb86c800640 1 -- 172.21.15.62:0/2976265341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8640ca0b0 con 0x7fb8640a9300 2024-03-23T11:19:42.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.377+0000 7fb85e000640 1 -- 172.21.15.62:0/2976265341 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb854005430 con 0x7fb8640a9300 2024-03-23T11:19:42.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.377+0000 7fb85e000640 1 -- 172.21.15.62:0/2976265341 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854012870 con 0x7fb8640a9300 2024-03-23T11:19:42.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.377+0000 7fb86c800640 1 -- 172.21.15.62:0/2976265341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8640b31d0 con 0x7fb8640a9300 2024-03-23T11:19:42.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.379+0000 7fb86c800640 1 -- 172.21.15.62:0/2976265341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb864066c90 con 0x7fb8640a9300 2024-03-23T11:19:42.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.379+0000 7fb85e000640 1 -- 172.21.15.62:0/2976265341 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb8540129d0 con 0x7fb8640a9300 2024-03-23T11:19:42.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.380+0000 7fb85e000640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb84806a7d0 0x7fb84806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.380+0000 7fb85e000640 1 -- 172.21.15.62:0/2976265341 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb854088a50 con 0x7fb8640a9300 2024-03-23T11:19:42.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.381+0000 7fb869e00640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb84806a7d0 0x7fb84806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.382+0000 7fb869e00640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb84806a7d0 0x7fb84806cc90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb8580098b0 tx=0x7fb858008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:42.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.385+0000 7fb85e000640 1 -- 172.21.15.62:0/2976265341 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb854057870 con 0x7fb8640a9300 2024-03-23T11:19:42.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.583+0000 7fb86c800640 1 -- 172.21.15.62:0/2976265341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb8640d2b80 con 0x7fb8640a9300 2024-03-23T11:19:42.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.585+0000 7fb85e000640 1 -- 172.21.15.62:0/2976265341 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb854057400 con 0x7fb8640a9300 2024-03-23T11:19:42.585 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:42.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.587+0000 7fb86c800640 1 -- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb84806a7d0 msgr2=0x7fb84806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.587+0000 7fb86c800640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb84806a7d0 0x7fb84806cc90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb8580098b0 tx=0x7fb858008040 comp rx=0 tx=0).stop 2024-03-23T11:19:42.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.587+0000 7fb86c800640 1 -- 172.21.15.62:0/2976265341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8640a9300 msgr2=0x7fb8640b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.587+0000 7fb86c800640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8640a9300 0x7fb8640b2c90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb85400c500 tx=0x7fb85400c9c0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.587+0000 7fb86c800640 1 -- 172.21.15.62:0/2976265341 shutdown_connections 2024-03-23T11:19:42.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.587+0000 7fb86c800640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb84806a7d0 0x7fb84806cc90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.587+0000 7fb86c800640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8640a9300 0x7fb8640b2c90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.587+0000 7fb86c800640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8640ca5d0 0x7fb86415d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.587+0000 7fb86c800640 1 --2- 172.21.15.62:0/2976265341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb84c005180 0x7fb864158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.587+0000 7fb86c800640 1 -- 172.21.15.62:0/2976265341 >> 172.21.15.62:0/2976265341 conn(0x7fb864071570 msgr2=0x7fb86415eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:42.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.588+0000 7fb86c800640 1 -- 172.21.15.62:0/2976265341 shutdown_connections 2024-03-23T11:19:42.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.588+0000 7fb86c800640 1 -- 172.21.15.62:0/2976265341 wait complete. 2024-03-23T11:19:42.607 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:42.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.817+0000 7f094d800640 1 Processor -- start 2024-03-23T11:19:42.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.817+0000 7f094d800640 1 -- start start 2024-03-23T11:19:42.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.819+0000 7f094d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0948143dd0 0x7f094815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.819+0000 7f094d800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0948070090 0x7f0948070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.819+0000 7f094d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09480b2820 0x7f09480b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.819+0000 7f094d800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0948070a60 con 0x7f0948143dd0 2024-03-23T11:19:42.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.819+0000 7f094d800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09480b31b0 con 0x7f0948070090 2024-03-23T11:19:42.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.819+0000 7f094d800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09480cd140 con 0x7f09480b2820 2024-03-23T11:19:42.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.819+0000 7f0947e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0948143dd0 0x7f094815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.819+0000 7f0947e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0948143dd0 0x7f094815abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51220/0 (socket says 172.21.15.62:51220) 2024-03-23T11:19:42.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.819+0000 7f0947e00640 1 -- 172.21.15.62:0/3762880331 learned_addr learned my addr 172.21.15.62:0/3762880331 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:42.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.820+0000 7f0947400640 1 --2- 172.21.15.62:0/3762880331 >> v2:172.21.15.120:3300/0 conn(0x7f0948070090 0x7f0948070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.820+0000 7f094ce00640 1 --2- 172.21.15.62:0/3762880331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09480b2820 0x7f09480b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.820+0000 7f0947e00640 1 -- 172.21.15.62:0/3762880331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09480b2820 msgr2=0x7f09480b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:42.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.820+0000 7f0947e00640 1 --2- 172.21.15.62:0/3762880331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09480b2820 0x7f09480b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.820+0000 7f0947e00640 1 -- 172.21.15.62:0/3762880331 >> v2:172.21.15.120:3300/0 conn(0x7f0948070090 msgr2=0x7f0948070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:42.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.820+0000 7f0947e00640 1 --2- 172.21.15.62:0/3762880331 >> v2:172.21.15.120:3300/0 conn(0x7f0948070090 0x7f0948070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.820+0000 7f0947e00640 1 -- 172.21.15.62:0/3762880331 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09480cd3e0 con 0x7f0948143dd0 2024-03-23T11:19:42.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.821+0000 7f0947e00640 1 --2- 172.21.15.62:0/3762880331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0948143dd0 0x7f094815abe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0948071530 tx=0x7f093802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb72f3c381fb5512 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:42.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.822+0000 7f0946a00640 1 -- 172.21.15.62:0/3762880331 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0938031870 con 0x7f0948143dd0 2024-03-23T11:19:42.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.822+0000 7f0946a00640 1 -- 172.21.15.62:0/3762880331 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09380319d0 con 0x7f0948143dd0 2024-03-23T11:19:42.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.822+0000 7f0946a00640 1 -- 172.21.15.62:0/3762880331 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0938031d60 con 0x7f0948143dd0 2024-03-23T11:19:42.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.822+0000 7f094d800640 1 -- 172.21.15.62:0/3762880331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0948143dd0 msgr2=0x7f094815abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.822+0000 7f094d800640 1 --2- 172.21.15.62:0/3762880331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0948143dd0 0x7f094815abe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0948071530 tx=0x7f093802b910 comp rx=0 tx=0).stop 2024-03-23T11:19:42.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.822+0000 7f094d800640 1 -- 172.21.15.62:0/3762880331 shutdown_connections 2024-03-23T11:19:42.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.822+0000 7f094d800640 1 --2- 172.21.15.62:0/3762880331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09480b2820 0x7f09480b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.822+0000 7f094d800640 1 --2- 172.21.15.62:0/3762880331 >> v2:172.21.15.120:3300/0 conn(0x7f0948070090 0x7f0948070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.822+0000 7f094d800640 1 --2- 172.21.15.62:0/3762880331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0948143dd0 0x7f094815abe0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.822+0000 7f094d800640 1 -- 172.21.15.62:0/3762880331 >> 172.21.15.62:0/3762880331 conn(0x7f0948066050 msgr2=0x7f0948069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:42.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.823+0000 7f094d800640 1 -- 172.21.15.62:0/3762880331 shutdown_connections 2024-03-23T11:19:42.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.823+0000 7f094d800640 1 -- 172.21.15.62:0/3762880331 wait complete. 2024-03-23T11:19:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.824+0000 7f094d800640 1 Processor -- start 2024-03-23T11:19:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.824+0000 7f094d800640 1 -- start start 2024-03-23T11:19:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.825+0000 7f094d800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0948070090 0x7f09481564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.825+0000 7f094d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09480b2820 0x7f0948155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.825+0000 7f094d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0948143dd0 0x7f0948138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.825+0000 7f094d800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0948156a30 con 0x7f09480b2820 2024-03-23T11:19:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.825+0000 7f094d800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094815a730 con 0x7f0948070090 2024-03-23T11:19:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.825+0000 7f094d800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094815aa10 con 0x7f0948143dd0 2024-03-23T11:19:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.826+0000 7f0947400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09480b2820 0x7f0948155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.826+0000 7f0947400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09480b2820 0x7f0948155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51230/0 (socket says 172.21.15.62:51230) 2024-03-23T11:19:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.826+0000 7f0947400640 1 -- 172.21.15.62:0/3382178806 learned_addr learned my addr 172.21.15.62:0/3382178806 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.826+0000 7f0947e00640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0948070090 0x7f09481564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.826+0000 7f094ce00640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0948143dd0 0x7f0948138bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.826+0000 7f0947400640 1 -- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0948143dd0 msgr2=0x7f0948138bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.826+0000 7f0947400640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0948143dd0 0x7f0948138bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.826+0000 7f0947400640 1 -- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0948070090 msgr2=0x7f09481564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.826+0000 7f0947400640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0948070090 0x7f09481564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.826+0000 7f0947400640 1 -- 172.21.15.62:0/3382178806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09380094d0 con 0x7f09480b2820 2024-03-23T11:19:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.826+0000 7f0947e00640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0948070090 0x7f09481564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.827+0000 7f0947400640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09480b2820 0x7f0948155bd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f093400d5f0 tx=0x7f093400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.827+0000 7f093fe00640 1 -- 172.21.15.62:0/3382178806 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09340125f0 con 0x7f09480b2820 2024-03-23T11:19:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.827+0000 7f093fe00640 1 -- 172.21.15.62:0/3382178806 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0934012750 con 0x7f09480b2820 2024-03-23T11:19:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.827+0000 7f093fe00640 1 -- 172.21.15.62:0/3382178806 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934012b00 con 0x7f09480b2820 2024-03-23T11:19:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.827+0000 7f094d800640 1 -- 172.21.15.62:0/3382178806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09480afc70 con 0x7f09480b2820 2024-03-23T11:19:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.827+0000 7f094d800640 1 -- 172.21.15.62:0/3382178806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0948156110 con 0x7f09480b2820 2024-03-23T11:19:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.828+0000 7f094d800640 1 -- 172.21.15.62:0/3382178806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09480afad0 con 0x7f09480b2820 2024-03-23T11:19:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.829+0000 7f093fe00640 1 -- 172.21.15.62:0/3382178806 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0934012c60 con 0x7f09480b2820 2024-03-23T11:19:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.830+0000 7f093fe00640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f092806a7a0 0x7f092806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.830+0000 7f093fe00640 1 -- 172.21.15.62:0/3382178806 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0934089310 con 0x7f09480b2820 2024-03-23T11:19:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.831+0000 7f0947e00640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f092806a7a0 0x7f092806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.832+0000 7f0947e00640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f092806a7a0 0x7f092806cc60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0938036000 tx=0x7f093803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:42.835+0000 7f093fe00640 1 -- 172.21.15.62:0/3382178806 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09480afad0 con 0x7f09480b2820 2024-03-23T11:19:43.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.038+0000 7f094d800640 1 -- 172.21.15.62:0/3382178806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0948156330 con 0x7f09480b2820 2024-03-23T11:19:43.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.039+0000 7f093fe00640 1 -- 172.21.15.62:0/3382178806 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0948156330 con 0x7f09480b2820 2024-03-23T11:19:43.039 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:43.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.041+0000 7f094d800640 1 -- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f092806a7a0 msgr2=0x7f092806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.041+0000 7f094d800640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f092806a7a0 0x7f092806cc60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0938036000 tx=0x7f093803c040 comp rx=0 tx=0).stop 2024-03-23T11:19:43.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.042+0000 7f094d800640 1 -- 172.21.15.62:0/3382178806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09480b2820 msgr2=0x7f0948155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.042+0000 7f094d800640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09480b2820 0x7f0948155bd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f093400d5f0 tx=0x7f093400dab0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.042+0000 7f094d800640 1 -- 172.21.15.62:0/3382178806 shutdown_connections 2024-03-23T11:19:43.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.042+0000 7f094d800640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f092806a7a0 0x7f092806cc60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.042+0000 7f094d800640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09480b2820 0x7f0948155bd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.042+0000 7f094d800640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0948143dd0 0x7f0948138bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.042+0000 7f094d800640 1 --2- 172.21.15.62:0/3382178806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0948070090 0x7f09481564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.042+0000 7f094d800640 1 -- 172.21.15.62:0/3382178806 >> 172.21.15.62:0/3382178806 conn(0x7f0948066050 msgr2=0x7f094806da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.042+0000 7f094d800640 1 -- 172.21.15.62:0/3382178806 shutdown_connections 2024-03-23T11:19:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.042+0000 7f094d800640 1 -- 172.21.15.62:0/3382178806 wait complete. 2024-03-23T11:19:43.062 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:19:43.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.291+0000 7f7abfe00640 1 Processor -- start 2024-03-23T11:19:43.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.292+0000 7f7abfe00640 1 -- start start 2024-03-23T11:19:43.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.293+0000 7f7abfe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7ac015c960 0x7f7ac00cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.293+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac00d13b0 0x7f7ac00b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.293+0000 7f7abfe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac00d0810 con 0x7f7ac00cd260 2024-03-23T11:19:43.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.293+0000 7f7abfe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac00cd9d0 con 0x7f7ac015c960 2024-03-23T11:19:43.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.293+0000 7f7abfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00cdcb0 con 0x7f7ac00d13b0 2024-03-23T11:19:43.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.294+0000 7f7abe000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7ac00cd260 0x7f7ac00d0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36276/0 (socket says 172.21.15.62:36276) 2024-03-23T11:19:43.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.294+0000 7f7abe000640 1 -- 172.21.15.62:0/1054482326 learned_addr learned my addr 172.21.15.62:0/1054482326 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:43.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.294+0000 7f7abf400640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac00d13b0 0x7f7ac00b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.294+0000 7f7abea00640 1 --2- 172.21.15.62:0/1054482326 >> v2:172.21.15.120:3300/0 conn(0x7f7ac015c960 0x7f7ac00cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.294+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113081228 0 0) 0x7f7aa8001290 con 0x7f7ac00cd260 2024-03-23T11:19:43.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.294+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aa0003520 con 0x7f7ac00cd260 2024-03-23T11:19:43.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2347868375 0 0) 0x7f7aa0003520 con 0x7f7ac00cd260 2024-03-23T11:19:43.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac00d13b0 msgr2=0x7f7ac00b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:43.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac00d13b0 0x7f7ac00b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 >> v2:172.21.15.120:3300/0 conn(0x7f7ac015c960 msgr2=0x7f7ac00cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:43.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 --2- 172.21.15.62:0/1054482326 >> v2:172.21.15.120:3300/0 conn(0x7f7ac015c960 0x7f7ac00cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac00cdf90 con 0x7f7ac00cd260 2024-03-23T11:19:43.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7aa8001980 con 0x7f7ac00cd260 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 >> v1:172.21.15.62:6789/0 conn(0x7f7ac00cd260 legacy=0x7f7ac00d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 0x7f7aa0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7aa00047c0 0x7f7aa0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac00d13b0 0x7f7aa0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00cd9d0 con 0x7f7aa0003970 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00cdcb0 con 0x7f7aa00047c0 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa000ba00 con 0x7f7ac00d13b0 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.295+0000 7f7abea00640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 0x7f7aa0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.296+0000 7f7abe000640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7aa00047c0 0x7f7aa0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.296+0000 7f7abf400640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac00d13b0 0x7f7aa0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.296+0000 7f7abea00640 1 -- 172.21.15.62:0/1054482326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac00d13b0 msgr2=0x7f7aa0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.296+0000 7f7abea00640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac00d13b0 0x7f7aa0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.296+0000 7f7abea00640 1 -- 172.21.15.62:0/1054482326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7aa00047c0 msgr2=0x7f7aa0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.296+0000 7f7abea00640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7aa00047c0 0x7f7aa0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.296+0000 7f7abea00640 1 -- 172.21.15.62:0/1054482326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa0003520 con 0x7f7aa0003970 2024-03-23T11:19:43.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.297+0000 7f7abea00640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 0x7f7aa0003d90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0009ac0 tx=0x7f7aa00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3c5badcc605f2a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:43.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.298+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab002c710 con 0x7f7aa0003970 2024-03-23T11:19:43.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.298+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab002c870 con 0x7f7aa0003970 2024-03-23T11:19:43.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.298+0000 7f7abd600640 1 -- 172.21.15.62:0/1054482326 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab002cc00 con 0x7f7aa0003970 2024-03-23T11:19:43.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.298+0000 7f7abfe00640 1 -- 172.21.15.62:0/1054482326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 msgr2=0x7f7aa0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.298+0000 7f7abfe00640 1 --2- 172.21.15.62:0/1054482326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 0x7f7aa0003d90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0009ac0 tx=0x7f7aa00057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.298+0000 7f7abf400640 1 -- 172.21.15.62:0/1054482326 reap_dead start 2024-03-23T11:19:43.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.298+0000 7f7abfe00640 1 -- 172.21.15.62:0/1054482326 shutdown_connections 2024-03-23T11:19:43.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.298+0000 7f7abfe00640 1 -- 172.21.15.62:0/1054482326 >> 172.21.15.62:0/1054482326 conn(0x7f7ac0089f60 msgr2=0x7f7ac00d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:43.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.298+0000 7f7abfe00640 1 -- 172.21.15.62:0/1054482326 shutdown_connections 2024-03-23T11:19:43.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.298+0000 7f7abfe00640 1 -- 172.21.15.62:0/1054482326 wait complete. 2024-03-23T11:19:43.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.300+0000 7f7abfe00640 1 Processor -- start 2024-03-23T11:19:43.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.300+0000 7f7abfe00640 1 -- start start 2024-03-23T11:19:43.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.301+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 0x7f7ac015ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.301+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac015f010 0x7f7ac01594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.301+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ac0159a20 0x7f7ac00c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.301+0000 7f7abfe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac0159ea0 con 0x7f7aa0003970 2024-03-23T11:19:43.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.301+0000 7f7abfe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac014c850 con 0x7f7ac0159a20 2024-03-23T11:19:43.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.301+0000 7f7abfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00cc840 con 0x7f7ac015f010 2024-03-23T11:19:43.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.301+0000 7f7abea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 0x7f7ac015ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.302+0000 7f7abea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 0x7f7ac015ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51242/0 (socket says 172.21.15.62:51242) 2024-03-23T11:19:43.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.302+0000 7f7abea00640 1 -- 172.21.15.62:0/2903651790 learned_addr learned my addr 172.21.15.62:0/2903651790 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:43.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.302+0000 7f7abe000640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac015f010 0x7f7ac01594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.302+0000 7f7abf400640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ac0159a20 0x7f7ac00c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.302+0000 7f7abea00640 1 -- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac015f010 msgr2=0x7f7ac01594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.302+0000 7f7abea00640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac015f010 0x7f7ac01594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.302+0000 7f7abea00640 1 -- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ac0159a20 msgr2=0x7f7ac00c8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.302+0000 7f7abea00640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ac0159a20 0x7f7ac00c8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.302+0000 7f7abea00640 1 -- 172.21.15.62:0/2903651790 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac00cb7a0 con 0x7f7aa0003970 2024-03-23T11:19:43.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.302+0000 7f7abea00640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 0x7f7ac015ead0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0037040 tx=0x7f7ab0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:43.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.303+0000 7f7ab6a00640 1 -- 172.21.15.62:0/2903651790 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0003810 con 0x7f7aa0003970 2024-03-23T11:19:43.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.303+0000 7f7ab6a00640 1 -- 172.21.15.62:0/2903651790 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab002de90 con 0x7f7aa0003970 2024-03-23T11:19:43.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.303+0000 7f7ab6a00640 1 -- 172.21.15.62:0/2903651790 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0003b70 con 0x7f7aa0003970 2024-03-23T11:19:43.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.303+0000 7f7abfe00640 1 -- 172.21.15.62:0/2903651790 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac00cba80 con 0x7f7aa0003970 2024-03-23T11:19:43.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.303+0000 7f7abfe00640 1 -- 172.21.15.62:0/2903651790 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac00cc040 con 0x7f7aa0003970 2024-03-23T11:19:43.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.304+0000 7f7abfe00640 1 -- 172.21.15.62:0/2903651790 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac0159e00 con 0x7f7aa0003970 2024-03-23T11:19:43.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.305+0000 7f7ab6a00640 1 -- 172.21.15.62:0/2903651790 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7ab0003cd0 con 0x7f7aa0003970 2024-03-23T11:19:43.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.306+0000 7f7ab6a00640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a9c06a7a0 0x7f7a9c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.306+0000 7f7ab6a00640 1 -- 172.21.15.62:0/2903651790 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7ab00aad80 con 0x7f7aa0003970 2024-03-23T11:19:43.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.307+0000 7f7abe000640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a9c06a7a0 0x7f7a9c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.308+0000 7f7abe000640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a9c06a7a0 0x7f7a9c06cc60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8001430 tx=0x7f7aa8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:43.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.311+0000 7f7ab6a00640 1 -- 172.21.15.62:0/2903651790 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7ab0076250 con 0x7f7aa0003970 2024-03-23T11:19:43.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.526+0000 7f7abfe00640 1 -- 172.21.15.62:0/2903651790 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7ac0066bb0 con 0x7f7aa0003970 2024-03-23T11:19:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.527+0000 7f7ab6a00640 1 -- 172.21.15.62:0/2903651790 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7ab0079870 con 0x7f7aa0003970 2024-03-23T11:19:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:19:43.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.530+0000 7f7abfe00640 1 -- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a9c06a7a0 msgr2=0x7f7a9c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.530+0000 7f7abfe00640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a9c06a7a0 0x7f7a9c06cc60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8001430 tx=0x7f7aa8009040 comp rx=0 tx=0).stop 2024-03-23T11:19:43.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.530+0000 7f7abfe00640 1 -- 172.21.15.62:0/2903651790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 msgr2=0x7f7ac015ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.530+0000 7f7abfe00640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 0x7f7ac015ead0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0037040 tx=0x7f7ab0004730 comp rx=0 tx=0).stop 2024-03-23T11:19:43.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.530+0000 7f7abfe00640 1 -- 172.21.15.62:0/2903651790 shutdown_connections 2024-03-23T11:19:43.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.530+0000 7f7abfe00640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a9c06a7a0 0x7f7a9c06cc60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.530+0000 7f7abfe00640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ac0159a20 0x7f7ac00c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.530+0000 7f7abfe00640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ac015f010 0x7f7ac01594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.530+0000 7f7abfe00640 1 --2- 172.21.15.62:0/2903651790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7aa0003970 0x7f7ac015ead0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.530+0000 7f7abfe00640 1 -- 172.21.15.62:0/2903651790 >> 172.21.15.62:0/2903651790 conn(0x7f7ac0089f60 msgr2=0x7f7ac00d0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:43.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.531+0000 7f7abfe00640 1 -- 172.21.15.62:0/2903651790 shutdown_connections 2024-03-23T11:19:43.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.531+0000 7f7abfe00640 1 -- 172.21.15.62:0/2903651790 wait complete. 2024-03-23T11:19:43.549 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:19:43.549 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:43.607 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:43.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.831+0000 7fdd14c00640 1 Processor -- start 2024-03-23T11:19:43.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.832+0000 7fdd14c00640 1 -- start start 2024-03-23T11:19:43.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.833+0000 7fdd14c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdd1015c9e0 0x7fdd101607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.833+0000 7fdd14c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdd1015cdc0 0x7fdd100d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.833+0000 7fdd14c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1006bfc0 con 0x7fdd100cf650 2024-03-23T11:19:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.833+0000 7fdd14c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd100ce140 con 0x7fdd1015c9e0 2024-03-23T11:19:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.833+0000 7fdd14c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd100b4370 con 0x7fdd1015cdc0 2024-03-23T11:19:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.833+0000 7fdd0de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdd1015c9e0 0x7fdd101607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.834+0000 7fdd0de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdd1015c9e0 0x7fdd101607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51252/0 (socket says 172.21.15.62:51252) 2024-03-23T11:19:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.834+0000 7fdd0de00640 1 -- 172.21.15.62:0/3964460931 learned_addr learned my addr 172.21.15.62:0/3964460931 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.834+0000 7fdd07e00640 1 --2- 172.21.15.62:0/3964460931 >> v2:172.21.15.120:3300/0 conn(0x7fdd1015cdc0 0x7fdd100d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.834+0000 7fdd0de00640 1 -- 172.21.15.62:0/3964460931 >> v2:172.21.15.120:3300/0 conn(0x7fdd1015cdc0 msgr2=0x7fdd100d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.834+0000 7fdd0de00640 1 --2- 172.21.15.62:0/3964460931 >> v2:172.21.15.120:3300/0 conn(0x7fdd1015cdc0 0x7fdd100d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.834+0000 7fdd0de00640 1 -- 172.21.15.62:0/3964460931 >> v1:172.21.15.62:6789/0 conn(0x7fdd100cf650 legacy=0x7fdd1015d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.834+0000 7fdd0de00640 1 -- 172.21.15.62:0/3964460931 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd100b5260 con 0x7fdd1015c9e0 2024-03-23T11:19:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.834+0000 7fdd07e00640 1 --2- 172.21.15.62:0/3964460931 >> v2:172.21.15.120:3300/0 conn(0x7fdd1015cdc0 0x7fdd100d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.835+0000 7fdd0de00640 1 --2- 172.21.15.62:0/3964460931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd1015c9e0 0x7fdd101607c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdd100b89a0 tx=0x7fdd0002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf34a9f8b8365e4c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.835+0000 7fdd0d400640 1 -- 172.21.15.62:0/3964460931 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00030870 con 0x7fdd1015c9e0 2024-03-23T11:19:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.835+0000 7fdd0d400640 1 -- 172.21.15.62:0/3964460931 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd000309d0 con 0x7fdd1015c9e0 2024-03-23T11:19:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.835+0000 7fdd0d400640 1 -- 172.21.15.62:0/3964460931 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00030d60 con 0x7fdd1015c9e0 2024-03-23T11:19:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.836+0000 7fdd14c00640 1 -- 172.21.15.62:0/3964460931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd1015c9e0 msgr2=0x7fdd101607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.836+0000 7fdd14c00640 1 --2- 172.21.15.62:0/3964460931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd1015c9e0 0x7fdd101607c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdd100b89a0 tx=0x7fdd0002b9b0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.836+0000 7fdd14c00640 1 -- 172.21.15.62:0/3964460931 shutdown_connections 2024-03-23T11:19:43.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.836+0000 7fdd14c00640 1 --2- 172.21.15.62:0/3964460931 >> v2:172.21.15.120:3300/0 conn(0x7fdd1015cdc0 0x7fdd100d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.836+0000 7fdd14c00640 1 --2- 172.21.15.62:0/3964460931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd1015c9e0 0x7fdd101607c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.836+0000 7fdd14c00640 1 -- 172.21.15.62:0/3964460931 >> 172.21.15.62:0/3964460931 conn(0x7fdd10071570 msgr2=0x7fdd10071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:43.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.836+0000 7fdd14c00640 1 -- 172.21.15.62:0/3964460931 shutdown_connections 2024-03-23T11:19:43.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.836+0000 7fdd14c00640 1 -- 172.21.15.62:0/3964460931 wait complete. 2024-03-23T11:19:43.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.838+0000 7fdd14c00640 1 Processor -- start 2024-03-23T11:19:43.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.838+0000 7fdd14c00640 1 -- start start 2024-03-23T11:19:43.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.839+0000 7fdd14c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd100cf650 0x7fdd1014ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.839+0000 7fdd14c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd1015c9e0 0x7fdd10158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.839+0000 7fdd14c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd1015cdc0 0x7fdd1015a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.839+0000 7fdd14c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd10152840 con 0x7fdd1015c9e0 2024-03-23T11:19:43.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.839+0000 7fdd14c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100cb280 con 0x7fdd1015cdc0 2024-03-23T11:19:43.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.839+0000 7fdd14c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100cafa0 con 0x7fdd100cf650 2024-03-23T11:19:43.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.839+0000 7fdd07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd1015c9e0 0x7fdd10158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.839+0000 7fdd07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd1015c9e0 0x7fdd10158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51254/0 (socket says 172.21.15.62:51254) 2024-03-23T11:19:43.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.839+0000 7fdd07e00640 1 -- 172.21.15.62:0/3932172325 learned_addr learned my addr 172.21.15.62:0/3932172325 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:43.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.840+0000 7fdd0de00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd100cf650 0x7fdd1014ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.840+0000 7fdd0e800640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd1015cdc0 0x7fdd1015a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.840+0000 7fdd07e00640 1 -- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd100cf650 msgr2=0x7fdd1014ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.840+0000 7fdd07e00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd100cf650 0x7fdd1014ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.840+0000 7fdd07e00640 1 -- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd1015cdc0 msgr2=0x7fdd1015a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:43.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.840+0000 7fdd07e00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd1015cdc0 0x7fdd1015a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:43.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.840+0000 7fdd07e00640 1 -- 172.21.15.62:0/3932172325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd000094d0 con 0x7fdd1015c9e0 2024-03-23T11:19:43.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.840+0000 7fdd07e00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd1015c9e0 0x7fdd10158e30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800c7f0 tx=0x7fdcf800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:43.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.840+0000 7fdd06000640 1 -- 172.21.15.62:0/3932172325 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8007820 con 0x7fdd1015c9e0 2024-03-23T11:19:43.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.841+0000 7fdd06000640 1 -- 172.21.15.62:0/3932172325 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf8007980 con 0x7fdd1015c9e0 2024-03-23T11:19:43.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.841+0000 7fdd06000640 1 -- 172.21.15.62:0/3932172325 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8007cf0 con 0x7fdd1015c9e0 2024-03-23T11:19:43.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.841+0000 7fdd14c00640 1 -- 172.21.15.62:0/3932172325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd101f58c0 con 0x7fdd1015c9e0 2024-03-23T11:19:43.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.841+0000 7fdd14c00640 1 -- 172.21.15.62:0/3932172325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd101f5e80 con 0x7fdd1015c9e0 2024-03-23T11:19:43.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.842+0000 7fdd14c00640 1 -- 172.21.15.62:0/3932172325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd10066550 con 0x7fdd1015c9e0 2024-03-23T11:19:43.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.843+0000 7fdd06000640 1 -- 172.21.15.62:0/3932172325 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fdcf80113f0 con 0x7fdd1015c9e0 2024-03-23T11:19:43.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.844+0000 7fdd06000640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcec06a7a0 0x7fdcec06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:43.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.844+0000 7fdd06000640 1 -- 172.21.15.62:0/3932172325 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdcf8089190 con 0x7fdd1015c9e0 2024-03-23T11:19:43.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.845+0000 7fdd0de00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcec06a7a0 0x7fdcec06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:43.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.846+0000 7fdd0de00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcec06a7a0 0x7fdcec06cc60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdd00036000 tx=0x7fdd00008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:43.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:43.849+0000 7fdd06000640 1 -- 172.21.15.62:0/3932172325 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdcf8054520 con 0x7fdd1015c9e0 2024-03-23T11:19:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.053+0000 7fdd14c00640 1 -- 172.21.15.62:0/3932172325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd10066c90 con 0x7fdd1015c9e0 2024-03-23T11:19:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.054+0000 7fdd06000640 1 -- 172.21.15.62:0/3932172325 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16226 (secure 0 0 0) 0x7fdcf8057b40 con 0x7fdd1015c9e0 2024-03-23T11:19:44.055 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:44.055 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:19:42.867981+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 -- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcec06a7a0 msgr2=0x7fdcec06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcec06a7a0 0x7fdcec06cc60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdd00036000 tx=0x7fdd00008030 comp rx=0 tx=0).stop 2024-03-23T11:19:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 -- 172.21.15.62:0/3932172325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd1015c9e0 msgr2=0x7fdd10158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd1015c9e0 0x7fdd10158e30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800c7f0 tx=0x7fdcf800ccb0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 -- 172.21.15.62:0/3932172325 shutdown_connections 2024-03-23T11:19:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcec06a7a0 0x7fdcec06cc60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd1015c9e0 0x7fdd10158e30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd1015cdc0 0x7fdd1015a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 --2- 172.21.15.62:0/3932172325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd100cf650 0x7fdd1014ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 -- 172.21.15.62:0/3932172325 >> 172.21.15.62:0/3932172325 conn(0x7fdd10071570 msgr2=0x7fdd1015fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 -- 172.21.15.62:0/3932172325 shutdown_connections 2024-03-23T11:19:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.057+0000 7fdd14c00640 1 -- 172.21.15.62:0/3932172325 wait complete. 2024-03-23T11:19:44.077 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:44.078 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:44.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.289+0000 7fa4a2200640 1 Processor -- start 2024-03-23T11:19:44.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.290+0000 7fa4a2200640 1 -- start start 2024-03-23T11:19:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.291+0000 7fa4a2200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa49c0b0980 0x7fa49c161100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.291+0000 7fa4a2200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa49c0cb8c0 0x7fa49c0ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.291+0000 7fa4a2200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c0ced60 con 0x7fa49c069c60 2024-03-23T11:19:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.291+0000 7fa4a2200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa49c0cf2a0 con 0x7fa49c0b0980 2024-03-23T11:19:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.291+0000 7fa4a2200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa49c0cefc0 con 0x7fa49c0cb8c0 2024-03-23T11:19:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.291+0000 7fa4a0e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa49c0b0980 0x7fa49c161100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.292+0000 7fa4a0e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa49c0b0980 0x7fa49c161100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51264/0 (socket says 172.21.15.62:51264) 2024-03-23T11:19:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.292+0000 7fa4a0e00640 1 -- 172.21.15.62:0/1203948263 learned_addr learned my addr 172.21.15.62:0/1203948263 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.292+0000 7fa49be00640 1 --2- 172.21.15.62:0/1203948263 >> v2:172.21.15.120:3300/0 conn(0x7fa49c0cb8c0 0x7fa49c0ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.292+0000 7fa4a0e00640 1 -- 172.21.15.62:0/1203948263 >> v2:172.21.15.120:3300/0 conn(0x7fa49c0cb8c0 msgr2=0x7fa49c0ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:44.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.292+0000 7fa4a0e00640 1 --2- 172.21.15.62:0/1203948263 >> v2:172.21.15.120:3300/0 conn(0x7fa49c0cb8c0 0x7fa49c0ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.292+0000 7fa4a0e00640 1 -- 172.21.15.62:0/1203948263 >> v1:172.21.15.120:6789/0 conn(0x7fa49c069c60 legacy=0x7fa49c15c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:44.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.292+0000 7fa4a0e00640 1 -- 172.21.15.62:0/1203948263 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c160620 con 0x7fa49c0b0980 2024-03-23T11:19:44.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.293+0000 7fa4a0e00640 1 --2- 172.21.15.62:0/1203948263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa49c0b0980 0x7fa49c161100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa49c073590 tx=0x7fa49002b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c28902d85826b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:44.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.293+0000 7fa49b400640 1 -- 172.21.15.62:0/1203948263 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490031890 con 0x7fa49c0b0980 2024-03-23T11:19:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.294+0000 7fa49b400640 1 -- 172.21.15.62:0/1203948263 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4900319f0 con 0x7fa49c0b0980 2024-03-23T11:19:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.294+0000 7fa49b400640 1 -- 172.21.15.62:0/1203948263 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490031d80 con 0x7fa49c0b0980 2024-03-23T11:19:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.294+0000 7fa4a2200640 1 -- 172.21.15.62:0/1203948263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa49c0b0980 msgr2=0x7fa49c161100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.294+0000 7fa4a2200640 1 --2- 172.21.15.62:0/1203948263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa49c0b0980 0x7fa49c161100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa49c073590 tx=0x7fa49002b930 comp rx=0 tx=0).stop 2024-03-23T11:19:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.294+0000 7fa4a2200640 1 -- 172.21.15.62:0/1203948263 shutdown_connections 2024-03-23T11:19:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.294+0000 7fa4a2200640 1 --2- 172.21.15.62:0/1203948263 >> v2:172.21.15.120:3300/0 conn(0x7fa49c0cb8c0 0x7fa49c0ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.294+0000 7fa4a2200640 1 --2- 172.21.15.62:0/1203948263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa49c0b0980 0x7fa49c161100 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.294+0000 7fa4a2200640 1 -- 172.21.15.62:0/1203948263 >> 172.21.15.62:0/1203948263 conn(0x7fa49c0989d0 msgr2=0x7fa49c0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.294+0000 7fa4a2200640 1 -- 172.21.15.62:0/1203948263 shutdown_connections 2024-03-23T11:19:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.294+0000 7fa4a2200640 1 -- 172.21.15.62:0/1203948263 wait complete. 2024-03-23T11:19:44.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.296+0000 7fa4a2200640 1 Processor -- start 2024-03-23T11:19:44.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.296+0000 7fa4a2200640 1 -- start start 2024-03-23T11:19:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.297+0000 7fa4a2200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c069c60 0x7fa49c0c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.297+0000 7fa4a2200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa49c0b0980 0x7fa49c0c5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.297+0000 7fa4a2200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa49c0cb8c0 0x7fa49c1514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.297+0000 7fa4a2200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c1f66a0 con 0x7fa49c0b0980 2024-03-23T11:19:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.297+0000 7fa4a2200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c1f5a90 con 0x7fa49c069c60 2024-03-23T11:19:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.297+0000 7fa4a2200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c1f5d70 con 0x7fa49c0cb8c0 2024-03-23T11:19:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.297+0000 7fa49be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa49c0b0980 0x7fa49c0c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.298+0000 7fa49be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa49c0b0980 0x7fa49c0c5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51278/0 (socket says 172.21.15.62:51278) 2024-03-23T11:19:44.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.298+0000 7fa49be00640 1 -- 172.21.15.62:0/2588254970 learned_addr learned my addr 172.21.15.62:0/2588254970 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:44.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.298+0000 7fa4a0e00640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c069c60 0x7fa49c0c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.298+0000 7fa4a1800640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa49c0cb8c0 0x7fa49c1514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.298+0000 7fa49be00640 1 -- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa49c0cb8c0 msgr2=0x7fa49c1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.298+0000 7fa49be00640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa49c0cb8c0 0x7fa49c1514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.298+0000 7fa49be00640 1 -- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c069c60 msgr2=0x7fa49c0c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.298+0000 7fa49be00640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c069c60 0x7fa49c0c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.298+0000 7fa49be00640 1 -- 172.21.15.62:0/2588254970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4900094d0 con 0x7fa49c0b0980 2024-03-23T11:19:44.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.298+0000 7fa49be00640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa49c0b0980 0x7fa49c0c5790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa48800d5f0 tx=0x7fa48800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:44.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.299+0000 7fa498c00640 1 -- 172.21.15.62:0/2588254970 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4880125f0 con 0x7fa49c0b0980 2024-03-23T11:19:44.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.299+0000 7fa498c00640 1 -- 172.21.15.62:0/2588254970 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa488012750 con 0x7fa49c0b0980 2024-03-23T11:19:44.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.299+0000 7fa498c00640 1 -- 172.21.15.62:0/2588254970 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488012b00 con 0x7fa49c0b0980 2024-03-23T11:19:44.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.299+0000 7fa4a2200640 1 -- 172.21.15.62:0/2588254970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa49c1f6050 con 0x7fa49c0b0980 2024-03-23T11:19:44.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.299+0000 7fa4a2200640 1 -- 172.21.15.62:0/2588254970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa49c155400 con 0x7fa49c0b0980 2024-03-23T11:19:44.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.300+0000 7fa4a2200640 1 -- 172.21.15.62:0/2588254970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa49c0672c0 con 0x7fa49c0b0980 2024-03-23T11:19:44.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.301+0000 7fa498c00640 1 -- 172.21.15.62:0/2588254970 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa488012c60 con 0x7fa49c0b0980 2024-03-23T11:19:44.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.302+0000 7fa498c00640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa47c06a7a0 0x7fa47c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.302+0000 7fa498c00640 1 -- 172.21.15.62:0/2588254970 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa488089370 con 0x7fa49c0b0980 2024-03-23T11:19:44.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.302+0000 7fa4a0e00640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa47c06a7a0 0x7fa47c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.304+0000 7fa4a0e00640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa47c06a7a0 0x7fa47c06cc60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa490036000 tx=0x7fa49003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:44.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.310+0000 7fa498c00640 1 -- 172.21.15.62:0/2588254970 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa488054670 con 0x7fa49c0b0980 2024-03-23T11:19:44.500 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:19:44.505 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:19:44.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.509+0000 7fa4a2200640 1 -- 172.21.15.62:0/2588254970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa49c06d700 con 0x7fa49c0b0980 2024-03-23T11:19:44.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.510+0000 7fa498c00640 1 -- 172.21.15.62:0/2588254970 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa488057c90 con 0x7fa49c0b0980 2024-03-23T11:19:44.510 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:44.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.512+0000 7fa4a2200640 1 -- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa47c06a7a0 msgr2=0x7fa47c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.512+0000 7fa4a2200640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa47c06a7a0 0x7fa47c06cc60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa490036000 tx=0x7fa49003c040 comp rx=0 tx=0).stop 2024-03-23T11:19:44.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.512+0000 7fa4a2200640 1 -- 172.21.15.62:0/2588254970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa49c0b0980 msgr2=0x7fa49c0c5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.512+0000 7fa4a2200640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa49c0b0980 0x7fa49c0c5790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa48800d5f0 tx=0x7fa48800dab0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.512+0000 7fa4a2200640 1 -- 172.21.15.62:0/2588254970 shutdown_connections 2024-03-23T11:19:44.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.512+0000 7fa4a2200640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa47c06a7a0 0x7fa47c06cc60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.512+0000 7fa4a2200640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa49c0b0980 0x7fa49c0c5790 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.512+0000 7fa4a2200640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa49c0cb8c0 0x7fa49c1514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.512+0000 7fa4a2200640 1 --2- 172.21.15.62:0/2588254970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c069c60 0x7fa49c0c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.512+0000 7fa4a2200640 1 -- 172.21.15.62:0/2588254970 >> 172.21.15.62:0/2588254970 conn(0x7fa49c0989d0 msgr2=0x7fa49c0ca900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:44.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.513+0000 7fa4a2200640 1 -- 172.21.15.62:0/2588254970 shutdown_connections 2024-03-23T11:19:44.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.513+0000 7fa4a2200640 1 -- 172.21.15.62:0/2588254970 wait complete. 2024-03-23T11:19:44.534 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:44.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.774+0000 7f935fa00640 1 Processor -- start 2024-03-23T11:19:44.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.775+0000 7f935fa00640 1 -- start start 2024-03-23T11:19:44.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.776+0000 7f935fa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f935815c8f0 0x7f93580cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.776+0000 7f935fa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9358071810 0x7f9358071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.776+0000 7f935fa00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93580aefe0 con 0x7f93580ae6b0 2024-03-23T11:19:44.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.776+0000 7f935fa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93580aeb30 con 0x7f9358071810 2024-03-23T11:19:44.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.776+0000 7f935fa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93580721e0 con 0x7f935815c8f0 2024-03-23T11:19:44.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.777+0000 7f934fe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f93580ae6b0 0x7f9358160ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36290/0 (socket says 172.21.15.62:36290) 2024-03-23T11:19:44.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.777+0000 7f934fe00640 1 -- 172.21.15.62:0/1699632043 learned_addr learned my addr 172.21.15.62:0/1699632043 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.777+0000 7f935d000640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f935815c8f0 0x7f93580cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.777+0000 7f935da00640 1 --2- 172.21.15.62:0/1699632043 >> v2:172.21.15.120:3300/0 conn(0x7f9358071810 0x7f9358071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.777+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010726111 0 0) 0x7f93580aefe0 con 0x7f93580ae6b0 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.777+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f933c003520 con 0x7f93580ae6b0 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.777+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1194188343 0 0) 0x7f933c003520 con 0x7f93580ae6b0 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.777+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f935815c8f0 msgr2=0x7f93580cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.777+0000 7f9357e00640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f935815c8f0 0x7f93580cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.777+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 >> v2:172.21.15.120:3300/0 conn(0x7f9358071810 msgr2=0x7f9358071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.777+0000 7f9357e00640 1 --2- 172.21.15.62:0/1699632043 >> v2:172.21.15.120:3300/0 conn(0x7f9358071810 0x7f9358071ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9358160570 con 0x7f93580ae6b0 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9340001670 con 0x7f93580ae6b0 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 >> v1:172.21.15.62:6789/0 conn(0x7f93580ae6b0 legacy=0x7f9358160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f9357e00640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f933c003970 0x7f933c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f9357e00640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f933c0047c0 0x7f933c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f9357e00640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f935815c8f0 0x7f933c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93580aeb30 con 0x7f933c0047c0 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93580721e0 con 0x7f933c003970 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c00b980 con 0x7f935815c8f0 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f934fe00640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f933c0047c0 0x7f933c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f935d000640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f933c003970 0x7f933c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f935da00640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f935815c8f0 0x7f933c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f934fe00640 1 -- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f935815c8f0 msgr2=0x7f933c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f934fe00640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f935815c8f0 0x7f933c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f934fe00640 1 -- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f933c003970 msgr2=0x7f933c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.778+0000 7f934fe00640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f933c003970 0x7f933c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.779+0000 7f934fe00640 1 -- 172.21.15.62:0/1699632043 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c003520 con 0x7f933c0047c0 2024-03-23T11:19:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.779+0000 7f935da00640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f935815c8f0 0x7f933c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:44.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.780+0000 7f934fe00640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f933c0047c0 0x7f933c004c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f934000c210 tx=0x7f933c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11c1052ee7aa863 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:44.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.780+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9340005ad0 con 0x7f933c0047c0 2024-03-23T11:19:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.780+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9340005c30 con 0x7f933c0047c0 2024-03-23T11:19:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.780+0000 7f9357e00640 1 -- 172.21.15.62:0/1699632043 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934002c8d0 con 0x7f933c0047c0 2024-03-23T11:19:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.781+0000 7f935fa00640 1 -- 172.21.15.62:0/1699632043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f933c0047c0 msgr2=0x7f933c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.781+0000 7f935fa00640 1 --2- 172.21.15.62:0/1699632043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f933c0047c0 0x7f933c004c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f934000c210 tx=0x7f933c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.781+0000 7f935da00640 1 -- 172.21.15.62:0/1699632043 reap_dead start 2024-03-23T11:19:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.781+0000 7f935fa00640 1 -- 172.21.15.62:0/1699632043 shutdown_connections 2024-03-23T11:19:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.781+0000 7f935fa00640 1 -- 172.21.15.62:0/1699632043 >> 172.21.15.62:0/1699632043 conn(0x7f93580b1370 msgr2=0x7f93580b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.781+0000 7f935fa00640 1 -- 172.21.15.62:0/1699632043 shutdown_connections 2024-03-23T11:19:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.781+0000 7f935fa00640 1 -- 172.21.15.62:0/1699632043 wait complete. 2024-03-23T11:19:44.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.783+0000 7f935fa00640 1 Processor -- start 2024-03-23T11:19:44.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.783+0000 7f935fa00640 1 -- start start 2024-03-23T11:19:44.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.784+0000 7f935fa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f933c0047c0 0x7f9358151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.784+0000 7f935fa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358151220 0x7f9358146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.784+0000 7f935fa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93580b42b0 0x7f935815bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.784+0000 7f935fa00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93580b3e20 con 0x7f9358151220 2024-03-23T11:19:44.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.784+0000 7f935fa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9358147520 con 0x7f933c0047c0 2024-03-23T11:19:44.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.784+0000 7f935fa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93580b2f30 con 0x7f93580b42b0 2024-03-23T11:19:44.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.784+0000 7f934fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358151220 0x7f9358146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.784+0000 7f934fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358151220 0x7f9358146fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51290/0 (socket says 172.21.15.62:51290) 2024-03-23T11:19:44.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.784+0000 7f934fe00640 1 -- 172.21.15.62:0/4193620946 learned_addr learned my addr 172.21.15.62:0/4193620946 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:44.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.784+0000 7f935d000640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f933c0047c0 0x7f9358151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.785+0000 7f935da00640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93580b42b0 0x7f935815bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.785+0000 7f934fe00640 1 -- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93580b42b0 msgr2=0x7f935815bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.785+0000 7f934fe00640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93580b42b0 0x7f935815bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.785+0000 7f934fe00640 1 -- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f933c0047c0 msgr2=0x7f9358151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.785+0000 7f934fe00640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f933c0047c0 0x7f9358151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.785+0000 7f934fe00640 1 -- 172.21.15.62:0/4193620946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93580b3210 con 0x7f9358151220 2024-03-23T11:19:44.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.785+0000 7f934fe00640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358151220 0x7f9358146fe0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f934000c210 tx=0x7f93580ce640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:44.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.786+0000 7f9355600640 1 -- 172.21.15.62:0/4193620946 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934002cce0 con 0x7f9358151220 2024-03-23T11:19:44.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.786+0000 7f9355600640 1 -- 172.21.15.62:0/4193620946 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934002ce40 con 0x7f9358151220 2024-03-23T11:19:44.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.786+0000 7f9355600640 1 -- 172.21.15.62:0/4193620946 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9340007270 con 0x7f9358151220 2024-03-23T11:19:44.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.786+0000 7f935fa00640 1 -- 172.21.15.62:0/4193620946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93580b34f0 con 0x7f9358151220 2024-03-23T11:19:44.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.786+0000 7f935fa00640 1 -- 172.21.15.62:0/4193620946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93580b3ab0 con 0x7f9358151220 2024-03-23T11:19:44.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.787+0000 7f935fa00640 1 -- 172.21.15.62:0/4193620946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93580672c0 con 0x7f9358151220 2024-03-23T11:19:44.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.788+0000 7f9355600640 1 -- 172.21.15.62:0/4193620946 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9340032070 con 0x7f9358151220 2024-03-23T11:19:44.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.789+0000 7f9355600640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f933406a7a0 0x7f933406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:44.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.789+0000 7f9355600640 1 -- 172.21.15.62:0/4193620946 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f934003c040 con 0x7f9358151220 2024-03-23T11:19:44.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.789+0000 7f935d000640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f933406a7a0 0x7f933406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:44.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.791+0000 7f935d000640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f933406a7a0 0x7f933406cc60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f93480098b0 tx=0x7f9348008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.793+0000 7f9355600640 1 -- 172.21.15.62:0/4193620946 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9340075280 con 0x7f9358151220 2024-03-23T11:19:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.993+0000 7f935fa00640 1 -- 172.21.15.62:0/4193620946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f93580713c0 con 0x7f9358151220 2024-03-23T11:19:44.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.994+0000 7f9355600640 1 -- 172.21.15.62:0/4193620946 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9340045020 con 0x7f9358151220 2024-03-23T11:19:44.994 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:44.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 -- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f933406a7a0 msgr2=0x7f933406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f933406a7a0 0x7f933406cc60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f93480098b0 tx=0x7f9348008040 comp rx=0 tx=0).stop 2024-03-23T11:19:44.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 -- 172.21.15.62:0/4193620946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358151220 msgr2=0x7f9358146fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:44.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358151220 0x7f9358146fe0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f934000c210 tx=0x7f93580ce640 comp rx=0 tx=0).stop 2024-03-23T11:19:44.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 -- 172.21.15.62:0/4193620946 shutdown_connections 2024-03-23T11:19:44.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f933406a7a0 0x7f933406cc60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358151220 0x7f9358146fe0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f93580b42b0 0x7f935815bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 --2- 172.21.15.62:0/4193620946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f933c0047c0 0x7f9358151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:44.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 -- 172.21.15.62:0/4193620946 >> 172.21.15.62:0/4193620946 conn(0x7f93580b1370 msgr2=0x7f9358158530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:44.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 -- 172.21.15.62:0/4193620946 shutdown_connections 2024-03-23T11:19:44.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:44.997+0000 7f935fa00640 1 -- 172.21.15.62:0/4193620946 wait complete. 2024-03-23T11:19:45.017 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:45.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.216+0000 7f8254000640 1 Processor -- start 2024-03-23T11:19:45.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.216+0000 7f8254000640 1 -- start start 2024-03-23T11:19:45.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.218+0000 7f8254000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f824c0989d0 0x7f824c098e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.218+0000 7f8254000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824c15c3f0 0x7f824c15c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.218+0000 7f8254000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c099370 con 0x7f824c0ccad0 2024-03-23T11:19:45.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.218+0000 7f8254000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f824c0d0210 con 0x7f824c0989d0 2024-03-23T11:19:45.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.218+0000 7f8254000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824c0cf600 con 0x7f824c15c3f0 2024-03-23T11:19:45.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.218+0000 7f8250c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f824c0989d0 0x7f824c098e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:45.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.218+0000 7f8251600640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f824c0ccad0 0x7f824c06b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36300/0 (socket says 172.21.15.62:36300) 2024-03-23T11:19:45.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.218+0000 7f8251600640 1 -- 172.21.15.62:0/1239326697 learned_addr learned my addr 172.21.15.62:0/1239326697 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:45.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.218+0000 7f8252000640 1 --2- 172.21.15.62:0/1239326697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824c15c3f0 0x7f824c15c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:45.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.218+0000 7f824be00640 1 -- 172.21.15.62:0/1239326697 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858291617 0 0) 0x7f824c099370 con 0x7f824c0ccad0 2024-03-23T11:19:45.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.219+0000 7f824be00640 1 -- 172.21.15.62:0/1239326697 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8234003520 con 0x7f824c0ccad0 2024-03-23T11:19:45.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.219+0000 7f8250c00640 1 -- 172.21.15.62:0/1239326697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824c15c3f0 msgr2=0x7f824c15c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:45.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.219+0000 7f8250c00640 1 --2- 172.21.15.62:0/1239326697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824c15c3f0 0x7f824c15c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.219+0000 7f8250c00640 1 -- 172.21.15.62:0/1239326697 >> v1:172.21.15.62:6789/0 conn(0x7f824c0ccad0 legacy=0x7f824c06b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:45.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.219+0000 7f8250c00640 1 -- 172.21.15.62:0/1239326697 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824c0cf8e0 con 0x7f824c0989d0 2024-03-23T11:19:45.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.220+0000 7f8250c00640 1 --2- 172.21.15.62:0/1239326697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f824c0989d0 0x7f824c098e30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f82380097a0 tx=0x7f823802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e953398709c761 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:45.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.220+0000 7f824be00640 1 -- 172.21.15.62:0/1239326697 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238031830 con 0x7f824c0989d0 2024-03-23T11:19:45.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.220+0000 7f824be00640 1 -- 172.21.15.62:0/1239326697 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8238031990 con 0x7f824c0989d0 2024-03-23T11:19:45.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.220+0000 7f824be00640 1 -- 172.21.15.62:0/1239326697 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238031d20 con 0x7f824c0989d0 2024-03-23T11:19:45.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.221+0000 7f8254000640 1 -- 172.21.15.62:0/1239326697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f824c0989d0 msgr2=0x7f824c098e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:45.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.221+0000 7f8254000640 1 --2- 172.21.15.62:0/1239326697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f824c0989d0 0x7f824c098e30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f82380097a0 tx=0x7f823802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.221+0000 7f8254000640 1 -- 172.21.15.62:0/1239326697 shutdown_connections 2024-03-23T11:19:45.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.221+0000 7f8254000640 1 --2- 172.21.15.62:0/1239326697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824c15c3f0 0x7f824c15c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.221+0000 7f8254000640 1 --2- 172.21.15.62:0/1239326697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f824c0989d0 0x7f824c098e30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.221+0000 7f8254000640 1 -- 172.21.15.62:0/1239326697 >> 172.21.15.62:0/1239326697 conn(0x7f824c0b2270 msgr2=0x7f824c0b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:45.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.221+0000 7f8254000640 1 -- 172.21.15.62:0/1239326697 shutdown_connections 2024-03-23T11:19:45.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.221+0000 7f8254000640 1 -- 172.21.15.62:0/1239326697 wait complete. 2024-03-23T11:19:45.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.223+0000 7f8254000640 1 Processor -- start 2024-03-23T11:19:45.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.223+0000 7f8254000640 1 -- start start 2024-03-23T11:19:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.224+0000 7f8254000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f824c0989d0 0x7f824c0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.224+0000 7f8254000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824c0ccad0 0x7f824c0b5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.224+0000 7f8254000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f824c15c3f0 0x7f824c0cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.224+0000 7f8254000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824c0ca120 con 0x7f824c15c3f0 2024-03-23T11:19:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.224+0000 7f8254000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824c0b0230 con 0x7f824c0989d0 2024-03-23T11:19:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.224+0000 7f8254000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824c0c94a0 con 0x7f824c0ccad0 2024-03-23T11:19:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.224+0000 7f8252000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f824c15c3f0 0x7f824c0cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.224+0000 7f8252000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f824c15c3f0 0x7f824c0cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51306/0 (socket says 172.21.15.62:51306) 2024-03-23T11:19:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.224+0000 7f8252000640 1 -- 172.21.15.62:0/3560440676 learned_addr learned my addr 172.21.15.62:0/3560440676 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.225+0000 7f8251600640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f824c0989d0 0x7f824c0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.225+0000 7f8252000640 1 -- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824c0ccad0 msgr2=0x7f824c0b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.225+0000 7f8250c00640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824c0ccad0 0x7f824c0b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.225+0000 7f8252000640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824c0ccad0 0x7f824c0b5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.225+0000 7f8252000640 1 -- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f824c0989d0 msgr2=0x7f824c0ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.225+0000 7f8252000640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f824c0989d0 0x7f824c0ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.225+0000 7f8252000640 1 -- 172.21.15.62:0/3560440676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8238009400 con 0x7f824c15c3f0 2024-03-23T11:19:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.225+0000 7f8252000640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f824c15c3f0 0x7f824c0cd650 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8244007e90 tx=0x7f824400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.226+0000 7f8249600640 1 -- 172.21.15.62:0/3560440676 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244014420 con 0x7f824c15c3f0 2024-03-23T11:19:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.226+0000 7f8249600640 1 -- 172.21.15.62:0/3560440676 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8244014580 con 0x7f824c15c3f0 2024-03-23T11:19:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.226+0000 7f8254000640 1 -- 172.21.15.62:0/3560440676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824c0c9780 con 0x7f824c15c3f0 2024-03-23T11:19:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.226+0000 7f8249600640 1 -- 172.21.15.62:0/3560440676 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244014930 con 0x7f824c15c3f0 2024-03-23T11:19:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.226+0000 7f8254000640 1 -- 172.21.15.62:0/3560440676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f824c0c9d40 con 0x7f824c15c3f0 2024-03-23T11:19:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.227+0000 7f8254000640 1 -- 172.21.15.62:0/3560440676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824c0ca260 con 0x7f824c15c3f0 2024-03-23T11:19:45.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.229+0000 7f8249600640 1 -- 172.21.15.62:0/3560440676 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8244014a90 con 0x7f824c15c3f0 2024-03-23T11:19:45.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.230+0000 7f8249600640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f822c06a7a0 0x7f822c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.230+0000 7f8249600640 1 -- 172.21.15.62:0/3560440676 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8244089780 con 0x7f824c15c3f0 2024-03-23T11:19:45.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.230+0000 7f8251600640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f822c06a7a0 0x7f822c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:45.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.231+0000 7f8251600640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f822c06a7a0 0x7f822c06cc60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f82400011e0 tx=0x7f8240009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:45.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.235+0000 7f8249600640 1 -- 172.21.15.62:0/3560440676 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f824c0ca260 con 0x7f824c15c3f0 2024-03-23T11:19:45.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.434+0000 7f8254000640 1 -- 172.21.15.62:0/3560440676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f824c0ca260 con 0x7f824c15c3f0 2024-03-23T11:19:45.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.434+0000 7f8249600640 1 -- 172.21.15.62:0/3560440676 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f824c0ca260 con 0x7f824c15c3f0 2024-03-23T11:19:45.435 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:45.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.436+0000 7f8254000640 1 -- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f822c06a7a0 msgr2=0x7f822c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:45.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.436+0000 7f8254000640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f822c06a7a0 0x7f822c06cc60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f82400011e0 tx=0x7f8240009040 comp rx=0 tx=0).stop 2024-03-23T11:19:45.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.436+0000 7f8254000640 1 -- 172.21.15.62:0/3560440676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f824c15c3f0 msgr2=0x7f824c0cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:45.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.437+0000 7f8254000640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f824c15c3f0 0x7f824c0cd650 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8244007e90 tx=0x7f824400e4b0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.437+0000 7f8254000640 1 -- 172.21.15.62:0/3560440676 shutdown_connections 2024-03-23T11:19:45.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.437+0000 7f8254000640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f822c06a7a0 0x7f822c06cc60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.437+0000 7f8254000640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f824c15c3f0 0x7f824c0cd650 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.437+0000 7f8254000640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f824c0ccad0 0x7f824c0b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.437+0000 7f8254000640 1 --2- 172.21.15.62:0/3560440676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f824c0989d0 0x7f824c0ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.437+0000 7f8254000640 1 -- 172.21.15.62:0/3560440676 >> 172.21.15.62:0/3560440676 conn(0x7f824c0b2270 msgr2=0x7f824c06f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:45.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.437+0000 7f8254000640 1 -- 172.21.15.62:0/3560440676 shutdown_connections 2024-03-23T11:19:45.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.437+0000 7f8254000640 1 -- 172.21.15.62:0/3560440676 wait complete. 2024-03-23T11:19:45.454 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:19:45.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.656+0000 7fec53000640 1 Processor -- start 2024-03-23T11:19:45.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.656+0000 7fec53000640 1 -- start start 2024-03-23T11:19:45.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec53000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fec4c15c9e0 0x7fec4c1607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec53000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fec4c15cdc0 0x7fec4c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec53000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec4c0cf650 0x7fec4c0cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec53000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4c0d0020 con 0x7fec4c15cdc0 2024-03-23T11:19:45.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec53000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4c06bfc0 con 0x7fec4c15c9e0 2024-03-23T11:19:45.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec53000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c0b5260 con 0x7fec4c0cf650 2024-03-23T11:19:45.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec43e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fec4c15cdc0 0x7fec4c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:45.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec43e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fec4c15cdc0 0x7fec4c0d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51314/0 (socket says 172.21.15.62:51314) 2024-03-23T11:19:45.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec43e00640 1 -- 172.21.15.62:0/4277812410 learned_addr learned my addr 172.21.15.62:0/4277812410 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:45.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec43e00640 1 -- 172.21.15.62:0/4277812410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec4c0cf650 msgr2=0x7fec4c0cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:45.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec43e00640 1 --2- 172.21.15.62:0/4277812410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec4c0cf650 0x7fec4c0cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec43e00640 1 -- 172.21.15.62:0/4277812410 >> v2:172.21.15.120:3300/0 conn(0x7fec4c15c9e0 msgr2=0x7fec4c1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:45.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec43e00640 1 --2- 172.21.15.62:0/4277812410 >> v2:172.21.15.120:3300/0 conn(0x7fec4c15c9e0 0x7fec4c1607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.658+0000 7fec43e00640 1 -- 172.21.15.62:0/4277812410 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c0ce140 con 0x7fec4c15cdc0 2024-03-23T11:19:45.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.659+0000 7fec43e00640 1 --2- 172.21.15.62:0/4277812410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec4c15cdc0 0x7fec4c0d1f20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0d0380 tx=0x7fec3402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f3bb1a31341456 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:45.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.660+0000 7fec4b400640 1 -- 172.21.15.62:0/4277812410 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34031870 con 0x7fec4c15cdc0 2024-03-23T11:19:45.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.660+0000 7fec4b400640 1 -- 172.21.15.62:0/4277812410 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec340319d0 con 0x7fec4c15cdc0 2024-03-23T11:19:45.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.660+0000 7fec4b400640 1 -- 172.21.15.62:0/4277812410 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34031d60 con 0x7fec4c15cdc0 2024-03-23T11:19:45.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.660+0000 7fec53000640 1 -- 172.21.15.62:0/4277812410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec4c15cdc0 msgr2=0x7fec4c0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:45.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.660+0000 7fec53000640 1 --2- 172.21.15.62:0/4277812410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec4c15cdc0 0x7fec4c0d1f20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0d0380 tx=0x7fec3402b910 comp rx=0 tx=0).stop 2024-03-23T11:19:45.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.661+0000 7fec53000640 1 -- 172.21.15.62:0/4277812410 shutdown_connections 2024-03-23T11:19:45.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.661+0000 7fec53000640 1 --2- 172.21.15.62:0/4277812410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec4c0cf650 0x7fec4c0cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.661+0000 7fec53000640 1 --2- 172.21.15.62:0/4277812410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec4c15cdc0 0x7fec4c0d1f20 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.661+0000 7fec53000640 1 --2- 172.21.15.62:0/4277812410 >> v2:172.21.15.120:3300/0 conn(0x7fec4c15c9e0 0x7fec4c1607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.661+0000 7fec53000640 1 -- 172.21.15.62:0/4277812410 >> 172.21.15.62:0/4277812410 conn(0x7fec4c071570 msgr2=0x7fec4c071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:45.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.661+0000 7fec53000640 1 -- 172.21.15.62:0/4277812410 shutdown_connections 2024-03-23T11:19:45.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.661+0000 7fec53000640 1 -- 172.21.15.62:0/4277812410 wait complete. 2024-03-23T11:19:45.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.662+0000 7fec53000640 1 Processor -- start 2024-03-23T11:19:45.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.663+0000 7fec53000640 1 -- start start 2024-03-23T11:19:45.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.663+0000 7fec53000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec4c0cf650 0x7fec4c0cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.663+0000 7fec53000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec4c15c9e0 0x7fec4c149870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.663+0000 7fec53000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec4c15cdc0 0x7fec4c146e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.663+0000 7fec53000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c149db0 con 0x7fec4c15c9e0 2024-03-23T11:19:45.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.663+0000 7fec53000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c147350 con 0x7fec4c15cdc0 2024-03-23T11:19:45.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec53000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c0cc660 con 0x7fec4c0cf650 2024-03-23T11:19:45.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec43e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec4c15c9e0 0x7fec4c149870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:45.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec43e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec4c15c9e0 0x7fec4c149870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51328/0 (socket says 172.21.15.62:51328) 2024-03-23T11:19:45.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec43e00640 1 -- 172.21.15.62:0/29810837 learned_addr learned my addr 172.21.15.62:0/29810837 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:45.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec4be00640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec4c0cf650 0x7fec4c0cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:45.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec51000640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec4c15cdc0 0x7fec4c146e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:45.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec43e00640 1 -- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec4c0cf650 msgr2=0x7fec4c0cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:45.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec43e00640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec4c0cf650 0x7fec4c0cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec43e00640 1 -- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec4c15cdc0 msgr2=0x7fec4c146e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:45.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec43e00640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec4c15cdc0 0x7fec4c146e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec43e00640 1 -- 172.21.15.62:0/29810837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec340094d0 con 0x7fec4c15c9e0 2024-03-23T11:19:45.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.664+0000 7fec51000640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec4c15cdc0 0x7fec4c146e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:45.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.665+0000 7fec43e00640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec4c15c9e0 0x7fec4c149870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fec34000c00 tx=0x7fec340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:45.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.665+0000 7fec48c00640 1 -- 172.21.15.62:0/29810837 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34031840 con 0x7fec4c15c9e0 2024-03-23T11:19:45.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.665+0000 7fec48c00640 1 -- 172.21.15.62:0/29810837 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec340443f0 con 0x7fec4c15c9e0 2024-03-23T11:19:45.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.665+0000 7fec48c00640 1 -- 172.21.15.62:0/29810837 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec340447a0 con 0x7fec4c15c9e0 2024-03-23T11:19:45.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.665+0000 7fec53000640 1 -- 172.21.15.62:0/29810837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4c0c7db0 con 0x7fec4c15c9e0 2024-03-23T11:19:45.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.665+0000 7fec53000640 1 -- 172.21.15.62:0/29810837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4c0c8370 con 0x7fec4c15c9e0 2024-03-23T11:19:45.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.666+0000 7fec53000640 1 -- 172.21.15.62:0/29810837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4c071c10 con 0x7fec4c15c9e0 2024-03-23T11:19:45.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.667+0000 7fec48c00640 1 -- 172.21.15.62:0/29810837 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fec3404d450 con 0x7fec4c15c9e0 2024-03-23T11:19:45.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.668+0000 7fec48c00640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec2806a7a0 0x7fec2806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:45.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.668+0000 7fec48c00640 1 -- 172.21.15.62:0/29810837 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fec3402d040 con 0x7fec4c15c9e0 2024-03-23T11:19:45.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.669+0000 7fec4be00640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec2806a7a0 0x7fec2806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:45.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.670+0000 7fec4be00640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec2806a7a0 0x7fec2806cc60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00a810 tx=0x7fec3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:45.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.672+0000 7fec48c00640 1 -- 172.21.15.62:0/29810837 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec340769a0 con 0x7fec4c15c9e0 2024-03-23T11:19:45.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.881+0000 7fec53000640 1 -- 172.21.15.62:0/29810837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fec4c0ceb80 con 0x7fec4c15c9e0 2024-03-23T11:19:45.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.882+0000 7fec48c00640 1 -- 172.21.15.62:0/29810837 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fec34079fc0 con 0x7fec4c15c9e0 2024-03-23T11:19:45.882 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:19:45.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.884+0000 7fec53000640 1 -- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec2806a7a0 msgr2=0x7fec2806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:45.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.885+0000 7fec53000640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec2806a7a0 0x7fec2806cc60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00a810 tx=0x7fec3c008040 comp rx=0 tx=0).stop 2024-03-23T11:19:45.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.885+0000 7fec53000640 1 -- 172.21.15.62:0/29810837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec4c15c9e0 msgr2=0x7fec4c149870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:45.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.885+0000 7fec53000640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec4c15c9e0 0x7fec4c149870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fec34000c00 tx=0x7fec340046c0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.885+0000 7fec53000640 1 -- 172.21.15.62:0/29810837 shutdown_connections 2024-03-23T11:19:45.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.885+0000 7fec53000640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fec2806a7a0 0x7fec2806cc60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.885+0000 7fec53000640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec4c15c9e0 0x7fec4c149870 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.885+0000 7fec53000640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fec4c15cdc0 0x7fec4c146e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.885+0000 7fec53000640 1 --2- 172.21.15.62:0/29810837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fec4c0cf650 0x7fec4c0cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:45.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.885+0000 7fec53000640 1 -- 172.21.15.62:0/29810837 >> 172.21.15.62:0/29810837 conn(0x7fec4c071570 msgr2=0x7fec4c15eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:45.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.885+0000 7fec53000640 1 -- 172.21.15.62:0/29810837 shutdown_connections 2024-03-23T11:19:45.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:45.885+0000 7fec53000640 1 -- 172.21.15.62:0/29810837 wait complete. 2024-03-23T11:19:45.902 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:19:45.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:45.902 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:19:46.001 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:19:46.001 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:19:46.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:46.001 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:19:46.060 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:19:46.060 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:46.118 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:19:46.118 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:46.171 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:46.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.399+0000 7fcdacc00640 1 Processor -- start 2024-03-23T11:19:46.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.399+0000 7fcdacc00640 1 -- start start 2024-03-23T11:19:46.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.401+0000 7fcdacc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcda80cda20 0x7fcda806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.401+0000 7fcdacc00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda8160fe0 con 0x7fcda8160c00 2024-03-23T11:19:46.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.401+0000 7fcdacc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda80b1510 con 0x7fcda80cde90 2024-03-23T11:19:46.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.401+0000 7fcdacc00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda80b1ad0 con 0x7fcda80cda20 2024-03-23T11:19:46.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.401+0000 7fcda7400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fcda8160c00 0x7fcda806de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36314/0 (socket says 172.21.15.62:36314) 2024-03-23T11:19:46.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.401+0000 7fcda7400640 1 -- 172.21.15.62:0/4175607049 learned_addr learned my addr 172.21.15.62:0/4175607049 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:46.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.401+0000 7fcda6a00640 1 --2- 172.21.15.62:0/4175607049 >> v2:172.21.15.62:3300/0 conn(0x7fcda80cda20 0x7fcda806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.401+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135900923 0 0) 0x7fcda8160fe0 con 0x7fcda8160c00 2024-03-23T11:19:46.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.401+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd8c003520 con 0x7fcda8160c00 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3093954618 0 0) 0x7fcd8c003520 con 0x7fcda8160c00 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 >> v2:172.21.15.62:3300/0 conn(0x7fcda80cda20 msgr2=0x7fcda806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 --2- 172.21.15.62:0/4175607049 >> v2:172.21.15.62:3300/0 conn(0x7fcda80cda20 0x7fcda806d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 >> v1:172.21.15.120:6789/0 conn(0x7fcda80cde90 legacy=0x7fcda80caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda80b17f0 con 0x7fcda8160c00 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fcd980019f0 con 0x7fcda8160c00 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 >> v1:172.21.15.62:6789/0 conn(0x7fcda8160c00 legacy=0x7fcda806de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 --2- 172.21.15.62:0/4175607049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd8c003970 0x7fcd8c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 --2- 172.21.15.62:0/4175607049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd8c0047c0 0x7fcd8c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 --2- 172.21.15.62:0/4175607049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd8c005180 0x7fcd8c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda80b1ad0 con 0x7fcd8c0047c0 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c00c000 con 0x7fcd8c003970 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c00c1c0 con 0x7fcd8c005180 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6a00640 1 --2- 172.21.15.62:0/4175607049 >> v2:172.21.15.62:3300/0 conn(0x7fcda80cda20 0x7fcda806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.402+0000 7fcda6a00640 1 --2- 172.21.15.62:0/4175607049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd8c0047c0 0x7fcd8c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.403+0000 7fcda7e00640 1 --2- 172.21.15.62:0/4175607049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd8c005180 0x7fcd8c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.403+0000 7fcda7400640 1 --2- 172.21.15.62:0/4175607049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd8c003970 0x7fcd8c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.403+0000 7fcda6a00640 1 -- 172.21.15.62:0/4175607049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd8c005180 msgr2=0x7fcd8c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:46.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.403+0000 7fcda6a00640 1 --2- 172.21.15.62:0/4175607049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd8c005180 0x7fcd8c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.403+0000 7fcda6a00640 1 -- 172.21.15.62:0/4175607049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd8c003970 msgr2=0x7fcd8c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:46.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.403+0000 7fcda6a00640 1 --2- 172.21.15.62:0/4175607049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd8c003970 0x7fcd8c003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.403+0000 7fcda6a00640 1 -- 172.21.15.62:0/4175607049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c003520 con 0x7fcd8c0047c0 2024-03-23T11:19:46.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.403+0000 7fcda7e00640 1 -- 172.21.15.62:0/4175607049 reap_dead start 2024-03-23T11:19:46.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.404+0000 7fcda6a00640 1 --2- 172.21.15.62:0/4175607049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd8c0047c0 0x7fcd8c004c40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009c60 tx=0x7fcd8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f513f8508fdac64b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:46.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.404+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9002c8d0 con 0x7fcd8c0047c0 2024-03-23T11:19:46.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.404+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9002ca30 con 0x7fcd8c0047c0 2024-03-23T11:19:46.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.404+0000 7fcda6000640 1 -- 172.21.15.62:0/4175607049 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9002cde0 con 0x7fcd8c0047c0 2024-03-23T11:19:46.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.405+0000 7fcdacc00640 1 -- 172.21.15.62:0/4175607049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd8c0047c0 msgr2=0x7fcd8c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:46.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.405+0000 7fcdacc00640 1 --2- 172.21.15.62:0/4175607049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd8c0047c0 0x7fcd8c004c40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009c60 tx=0x7fcd8c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.405+0000 7fcdacc00640 1 -- 172.21.15.62:0/4175607049 shutdown_connections 2024-03-23T11:19:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.405+0000 7fcdacc00640 1 --2- 172.21.15.62:0/4175607049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd8c0047c0 0x7fcd8c004c40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.405+0000 7fcdacc00640 1 -- 172.21.15.62:0/4175607049 >> 172.21.15.62:0/4175607049 conn(0x7fcda8066050 msgr2=0x7fcda8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.405+0000 7fcdacc00640 1 -- 172.21.15.62:0/4175607049 shutdown_connections 2024-03-23T11:19:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.405+0000 7fcdacc00640 1 -- 172.21.15.62:0/4175607049 wait complete. 2024-03-23T11:19:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.407+0000 7fcdacc00640 1 Processor -- start 2024-03-23T11:19:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.407+0000 7fcdacc00640 1 -- start start 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.408+0000 7fcdacc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd8c0047c0 0x7fcda814f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.408+0000 7fcdacc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda81f5c40 0x7fcda81f9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.408+0000 7fcdacc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcda81f9e60 0x7fcda815eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.408+0000 7fcdacc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda81fa3c0 con 0x7fcda81f5c40 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.408+0000 7fcdacc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda81fa6a0 con 0x7fcda81f9e60 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.408+0000 7fcdacc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda80ac820 con 0x7fcd8c0047c0 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.408+0000 7fcda6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda81f5c40 0x7fcda81f9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.408+0000 7fcda6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda81f5c40 0x7fcda81f9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51358/0 (socket says 172.21.15.62:51358) 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.408+0000 7fcda6a00640 1 -- 172.21.15.62:0/3285245708 learned_addr learned my addr 172.21.15.62:0/3285245708 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.409+0000 7fcda7e00640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcda81f9e60 0x7fcda815eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.409+0000 7fcda7400640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd8c0047c0 0x7fcda814f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.409+0000 7fcda6a00640 1 -- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd8c0047c0 msgr2=0x7fcda814f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:46.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.409+0000 7fcda6a00640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd8c0047c0 0x7fcda814f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.409+0000 7fcda6a00640 1 -- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcda81f9e60 msgr2=0x7fcda815eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:46.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.409+0000 7fcda6a00640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcda81f9e60 0x7fcda815eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.409+0000 7fcda6a00640 1 -- 172.21.15.62:0/3285245708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda80acb00 con 0x7fcda81f5c40 2024-03-23T11:19:46.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.409+0000 7fcda7e00640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcda81f9e60 0x7fcda815eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:46.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.409+0000 7fcda6a00640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda81f5c40 0x7fcda81f9920 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009c60 tx=0x7fcd90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:46.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.409+0000 7fcd9f400640 1 -- 172.21.15.62:0/3285245708 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90034c00 con 0x7fcda81f5c40 2024-03-23T11:19:46.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.410+0000 7fcd9f400640 1 -- 172.21.15.62:0/3285245708 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd90034d60 con 0x7fcda81f5c40 2024-03-23T11:19:46.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.410+0000 7fcdacc00640 1 -- 172.21.15.62:0/3285245708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda80ac510 con 0x7fcda81f5c40 2024-03-23T11:19:46.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.410+0000 7fcd9f400640 1 -- 172.21.15.62:0/3285245708 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90040690 con 0x7fcda81f5c40 2024-03-23T11:19:46.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.410+0000 7fcdacc00640 1 -- 172.21.15.62:0/3285245708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda81f60c0 con 0x7fcda81f5c40 2024-03-23T11:19:46.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.411+0000 7fcdacc00640 1 -- 172.21.15.62:0/3285245708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda80acc90 con 0x7fcda81f5c40 2024-03-23T11:19:46.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.412+0000 7fcd9f400640 1 -- 172.21.15.62:0/3285245708 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fcd900407f0 con 0x7fcda81f5c40 2024-03-23T11:19:46.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.413+0000 7fcd9f400640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd8406a7d0 0x7fcd8406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.413+0000 7fcd9f400640 1 -- 172.21.15.62:0/3285245708 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcd900aafa0 con 0x7fcda81f5c40 2024-03-23T11:19:46.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.413+0000 7fcda7400640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd8406a7d0 0x7fcd8406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.414+0000 7fcda7400640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd8406a7d0 0x7fcd8406cc90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcd98002230 tx=0x7fcd98002100 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:46.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.417+0000 7fcd9f400640 1 -- 172.21.15.62:0/3285245708 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd900de580 con 0x7fcda81f5c40 2024-03-23T11:19:46.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.621+0000 7fcdacc00640 1 -- 172.21.15.62:0/3285245708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcda81fa860 con 0x7fcda81f5c40 2024-03-23T11:19:46.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.622+0000 7fcd9f400640 1 -- 172.21.15.62:0/3285245708 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16226 (secure 0 0 0) 0x7fcd90076340 con 0x7fcda81f5c40 2024-03-23T11:19:46.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:46.623 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:19:42.867981+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:46.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.625+0000 7fcdacc00640 1 -- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd8406a7d0 msgr2=0x7fcd8406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:46.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.625+0000 7fcdacc00640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd8406a7d0 0x7fcd8406cc90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcd98002230 tx=0x7fcd98002100 comp rx=0 tx=0).stop 2024-03-23T11:19:46.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.625+0000 7fcdacc00640 1 -- 172.21.15.62:0/3285245708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda81f5c40 msgr2=0x7fcda81f9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:46.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.625+0000 7fcdacc00640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda81f5c40 0x7fcda81f9920 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009c60 tx=0x7fcd90004730 comp rx=0 tx=0).stop 2024-03-23T11:19:46.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.625+0000 7fcdacc00640 1 -- 172.21.15.62:0/3285245708 shutdown_connections 2024-03-23T11:19:46.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.625+0000 7fcdacc00640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd8406a7d0 0x7fcd8406cc90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.625+0000 7fcdacc00640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda81f5c40 0x7fcda81f9920 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.625+0000 7fcdacc00640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcda81f9e60 0x7fcda815eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.625+0000 7fcdacc00640 1 --2- 172.21.15.62:0/3285245708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd8c0047c0 0x7fcda814f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.625+0000 7fcdacc00640 1 -- 172.21.15.62:0/3285245708 >> 172.21.15.62:0/3285245708 conn(0x7fcda8066050 msgr2=0x7fcda8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:46.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.626+0000 7fcdacc00640 1 -- 172.21.15.62:0/3285245708 shutdown_connections 2024-03-23T11:19:46.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.626+0000 7fcdacc00640 1 -- 172.21.15.62:0/3285245708 wait complete. 2024-03-23T11:19:46.644 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:46.645 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:46.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.871+0000 7f32fb200640 1 Processor -- start 2024-03-23T11:19:46.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.872+0000 7f32fb200640 1 -- start start 2024-03-23T11:19:46.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.873+0000 7f32fb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc15fcf0 0x7f32fc0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.873+0000 7f32fb200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f32fc0b6dc0 0x7f32fc160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.873+0000 7f32fb200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32fc160dc0 con 0x7f32fc160420 2024-03-23T11:19:46.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.873+0000 7f32fb200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32fc0d3540 con 0x7f32fc0b6dc0 2024-03-23T11:19:46.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.873+0000 7f32fb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc0ce510 con 0x7f32fc15fcf0 2024-03-23T11:19:46.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.874+0000 7f32f9400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f32fc160420 0x7f32fc0d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:36328/0 (socket says 172.21.15.62:36328) 2024-03-23T11:19:46.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.874+0000 7f32f9400640 1 -- 172.21.15.62:0/4264483425 learned_addr learned my addr 172.21.15.62:0/4264483425 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:46.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.874+0000 7f32f9e00640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc15fcf0 0x7f32fc0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.874+0000 7f32fa800640 1 --2- 172.21.15.62:0/4264483425 >> v2:172.21.15.120:3300/0 conn(0x7f32fc0b6dc0 0x7f32fc160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.874+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821164577 0 0) 0x7f32fc160dc0 con 0x7f32fc160420 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.874+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e0003520 con 0x7f32fc160420 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1529747276 0 0) 0x7f32e0003520 con 0x7f32fc160420 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc15fcf0 msgr2=0x7f32fc0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc15fcf0 0x7f32fc0d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 >> v2:172.21.15.120:3300/0 conn(0x7f32fc0b6dc0 msgr2=0x7f32fc160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 --2- 172.21.15.62:0/4264483425 >> v2:172.21.15.120:3300/0 conn(0x7f32fc0b6dc0 0x7f32fc160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32fc0cdc70 con 0x7f32fc160420 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f32e8001670 con 0x7f32fc160420 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 >> v1:172.21.15.62:6789/0 conn(0x7f32fc160420 legacy=0x7f32fc0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e0003970 0x7f32e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 0x7f32e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc15fcf0 0x7f32e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc0d3540 con 0x7f32e00047c0 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc0ce510 con 0x7f32e0003970 2024-03-23T11:19:46.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e000ba60 con 0x7f32fc15fcf0 2024-03-23T11:19:46.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f9400640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 0x7f32e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.875+0000 7f32f9e00640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e0003970 0x7f32e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.876+0000 7f32fa800640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc15fcf0 0x7f32e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.876+0000 7f32f9400640 1 -- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc15fcf0 msgr2=0x7f32e0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:46.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.876+0000 7f32f9400640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc15fcf0 0x7f32e0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.876+0000 7f32f9400640 1 -- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e0003970 msgr2=0x7f32e0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:46.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.876+0000 7f32f9400640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e0003970 0x7f32e0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.876+0000 7f32f9400640 1 -- 172.21.15.62:0/4264483425 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e0003520 con 0x7f32e00047c0 2024-03-23T11:19:46.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.876+0000 7f32fa800640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc15fcf0 0x7f32e0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:46.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.876+0000 7f32f9e00640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e0003970 0x7f32e0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:46.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.877+0000 7f32f9400640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 0x7f32e0004c40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f32e800c0e0 tx=0x7f32e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9bfd0ef5984299 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:46.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.877+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8005ad0 con 0x7f32e00047c0 2024-03-23T11:19:46.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.877+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e8005c50 con 0x7f32e00047c0 2024-03-23T11:19:46.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.877+0000 7f32f8a00640 1 -- 172.21.15.62:0/4264483425 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e802c7d0 con 0x7f32e00047c0 2024-03-23T11:19:46.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.878+0000 7f32fb200640 1 -- 172.21.15.62:0/4264483425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 msgr2=0x7f32e0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:46.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.878+0000 7f32fb200640 1 --2- 172.21.15.62:0/4264483425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 0x7f32e0004c40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f32e800c0e0 tx=0x7f32e00057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.878+0000 7f32fa800640 1 -- 172.21.15.62:0/4264483425 reap_dead start 2024-03-23T11:19:46.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.878+0000 7f32fb200640 1 -- 172.21.15.62:0/4264483425 shutdown_connections 2024-03-23T11:19:46.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.878+0000 7f32fb200640 1 -- 172.21.15.62:0/4264483425 >> 172.21.15.62:0/4264483425 conn(0x7f32fc089f60 msgr2=0x7f32fc0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:46.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.878+0000 7f32fb200640 1 -- 172.21.15.62:0/4264483425 shutdown_connections 2024-03-23T11:19:46.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.878+0000 7f32fb200640 1 -- 172.21.15.62:0/4264483425 wait complete. 2024-03-23T11:19:46.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.880+0000 7f32fb200640 1 Processor -- start 2024-03-23T11:19:46.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.880+0000 7f32fb200640 1 -- start start 2024-03-23T11:19:46.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.881+0000 7f32fb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 0x7f32fc0ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.881+0000 7f32fb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc153d00 0x7f32fc0cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.881+0000 7f32fb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32fc0cf740 0x7f32fc0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.881+0000 7f32fb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc0ca0a0 con 0x7f32e00047c0 2024-03-23T11:19:46.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.881+0000 7f32fb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc0cfbf0 con 0x7f32fc0cf740 2024-03-23T11:19:46.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.881+0000 7f32fb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc0caa00 con 0x7f32fc153d00 2024-03-23T11:19:46.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.881+0000 7f32f9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 0x7f32fc0ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.881+0000 7f32f9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 0x7f32fc0ca4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51364/0 (socket says 172.21.15.62:51364) 2024-03-23T11:19:46.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.881+0000 7f32f9e00640 1 -- 172.21.15.62:0/2310022074 learned_addr learned my addr 172.21.15.62:0/2310022074 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:46.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.882+0000 7f32f9400640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc153d00 0x7f32fc0cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.882+0000 7f32f9e00640 1 -- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc153d00 msgr2=0x7f32fc0cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:46.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.882+0000 7f32f9e00640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc153d00 0x7f32fc0cf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.882+0000 7f32fa800640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32fc0cf740 0x7f32fc0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.882+0000 7f32f9e00640 1 -- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32fc0cf740 msgr2=0x7f32fc0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:46.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.882+0000 7f32f9e00640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32fc0cf740 0x7f32fc0cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:46.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.882+0000 7f32f9e00640 1 -- 172.21.15.62:0/2310022074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32fc0c9720 con 0x7f32e00047c0 2024-03-23T11:19:46.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.882+0000 7f32f9e00640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 0x7f32fc0ca4c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f32ec011570 tx=0x7f32ec011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:46.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.883+0000 7f32f2000640 1 -- 172.21.15.62:0/2310022074 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec015870 con 0x7f32e00047c0 2024-03-23T11:19:46.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.883+0000 7f32f2000640 1 -- 172.21.15.62:0/2310022074 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ec0159d0 con 0x7f32e00047c0 2024-03-23T11:19:46.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.883+0000 7f32f2000640 1 -- 172.21.15.62:0/2310022074 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec015d60 con 0x7f32e00047c0 2024-03-23T11:19:46.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.883+0000 7f32fb200640 1 -- 172.21.15.62:0/2310022074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32fc0cc1b0 con 0x7f32e00047c0 2024-03-23T11:19:46.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.883+0000 7f32fb200640 1 -- 172.21.15.62:0/2310022074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32fc0c9cc0 con 0x7f32e00047c0 2024-03-23T11:19:46.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.884+0000 7f32fb200640 1 -- 172.21.15.62:0/2310022074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32fc066bb0 con 0x7f32e00047c0 2024-03-23T11:19:46.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.885+0000 7f32f2000640 1 -- 172.21.15.62:0/2310022074 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f32ec00f460 con 0x7f32e00047c0 2024-03-23T11:19:46.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.886+0000 7f32f2000640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f32dc06a7a0 0x7f32dc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:46.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.886+0000 7f32f2000640 1 -- 172.21.15.62:0/2310022074 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f32ec0894b0 con 0x7f32e00047c0 2024-03-23T11:19:46.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.886+0000 7f32f9400640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f32dc06a7a0 0x7f32dc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:46.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.888+0000 7f32f9400640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f32dc06a7a0 0x7f32dc06cc60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f32e8034000 tx=0x7f32e8007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:46.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:46.891+0000 7f32f2000640 1 -- 172.21.15.62:0/2310022074 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32ec0548e0 con 0x7f32e00047c0 2024-03-23T11:19:47.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.089+0000 7f32fb200640 1 -- 172.21.15.62:0/2310022074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f32fc0cea40 con 0x7f32e00047c0 2024-03-23T11:19:47.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.090+0000 7f32f2000640 1 -- 172.21.15.62:0/2310022074 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f32ec057f00 con 0x7f32e00047c0 2024-03-23T11:19:47.091 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.093+0000 7f32fb200640 1 -- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f32dc06a7a0 msgr2=0x7f32dc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.093+0000 7f32fb200640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f32dc06a7a0 0x7f32dc06cc60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f32e8034000 tx=0x7f32e8007000 comp rx=0 tx=0).stop 2024-03-23T11:19:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.093+0000 7f32fb200640 1 -- 172.21.15.62:0/2310022074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 msgr2=0x7f32fc0ca4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.093+0000 7f32fb200640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 0x7f32fc0ca4c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f32ec011570 tx=0x7f32ec011a30 comp rx=0 tx=0).stop 2024-03-23T11:19:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.093+0000 7f32fb200640 1 -- 172.21.15.62:0/2310022074 shutdown_connections 2024-03-23T11:19:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.093+0000 7f32fb200640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f32dc06a7a0 0x7f32dc06cc60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.093+0000 7f32fb200640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32fc0cf740 0x7f32fc0cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.093+0000 7f32fb200640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32fc153d00 0x7f32fc0cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.093+0000 7f32fb200640 1 --2- 172.21.15.62:0/2310022074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e00047c0 0x7f32fc0ca4c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.093+0000 7f32fb200640 1 -- 172.21.15.62:0/2310022074 >> 172.21.15.62:0/2310022074 conn(0x7f32fc089f60 msgr2=0x7f32fc15f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.094+0000 7f32fb200640 1 -- 172.21.15.62:0/2310022074 shutdown_connections 2024-03-23T11:19:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.094+0000 7f32fb200640 1 -- 172.21.15.62:0/2310022074 wait complete. 2024-03-23T11:19:47.113 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:47.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.341+0000 7f9e9dc00640 1 Processor -- start 2024-03-23T11:19:47.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.341+0000 7f9e9dc00640 1 -- start start 2024-03-23T11:19:47.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.343+0000 7f9e9dc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9e980cd7c0 0x7f9e980b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:47.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.343+0000 7f9e9dc00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e980cad20 con 0x7f9e9806c300 2024-03-23T11:19:47.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.343+0000 7f9e9dc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9806e130 con 0x7f9e9806dca0 2024-03-23T11:19:47.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.343+0000 7f9e9dc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9815a8a0 con 0x7f9e980cd7c0 2024-03-23T11:19:47.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.343+0000 7f9e97400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9e9806c300 0x7f9e980cb490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60812/0 (socket says 172.21.15.62:60812) 2024-03-23T11:19:47.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.343+0000 7f9e97400640 1 -- 172.21.15.62:0/3362908913 learned_addr learned my addr 172.21.15.62:0/3362908913 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:47.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.343+0000 7f9e96a00640 1 --2- 172.21.15.62:0/3362908913 >> v2:172.21.15.120:3300/0 conn(0x7f9e980cd7c0 0x7f9e980b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:47.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.344+0000 7f9e96000640 1 -- 172.21.15.62:0/3362908913 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331360019 0 0) 0x7f9e980cad20 con 0x7f9e9806c300 2024-03-23T11:19:47.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.344+0000 7f9e96000640 1 -- 172.21.15.62:0/3362908913 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e7c005730 con 0x7f9e9806c300 2024-03-23T11:19:47.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.344+0000 7f9e96a00640 1 -- 172.21.15.62:0/3362908913 >> v1:172.21.15.120:6789/0 conn(0x7f9e9806dca0 legacy=0x7f9e980cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.344+0000 7f9e96000640 1 -- 172.21.15.62:0/3362908913 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 578377911 0 0) 0x7f9e7c005730 con 0x7f9e9806c300 2024-03-23T11:19:47.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.344+0000 7f9e96a00640 1 -- 172.21.15.62:0/3362908913 >> v1:172.21.15.62:6789/0 conn(0x7f9e9806c300 legacy=0x7f9e980cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.344+0000 7f9e96a00640 1 -- 172.21.15.62:0/3362908913 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9815ab80 con 0x7f9e980cd7c0 2024-03-23T11:19:47.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.346+0000 7f9e96a00640 1 --2- 172.21.15.62:0/3362908913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e980cd7c0 0x7f9e980b15e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9e880097a0 tx=0x7f9e8802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8199a33a94f26ee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:47.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.346+0000 7f9e96000640 1 -- 172.21.15.62:0/3362908913 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e900019f0 con 0x7f9e980cd7c0 2024-03-23T11:19:47.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.346+0000 7f9e96000640 1 -- 172.21.15.62:0/3362908913 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8802e050 con 0x7f9e980cd7c0 2024-03-23T11:19:47.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.346+0000 7f9e96000640 1 -- 172.21.15.62:0/3362908913 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e88030730 con 0x7f9e980cd7c0 2024-03-23T11:19:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.347+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3362908913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e980cd7c0 msgr2=0x7f9e980b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.347+0000 7f9e9dc00640 1 --2- 172.21.15.62:0/3362908913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e980cd7c0 0x7f9e980b15e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9e880097a0 tx=0x7f9e8802b790 comp rx=0 tx=0).stop 2024-03-23T11:19:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.347+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3362908913 shutdown_connections 2024-03-23T11:19:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.347+0000 7f9e9dc00640 1 --2- 172.21.15.62:0/3362908913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e980cd7c0 0x7f9e980b15e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.347+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3362908913 >> 172.21.15.62:0/3362908913 conn(0x7f9e980b02e0 msgr2=0x7f9e980b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.347+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3362908913 shutdown_connections 2024-03-23T11:19:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.347+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3362908913 wait complete. 2024-03-23T11:19:47.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.349+0000 7f9e9dc00640 1 Processor -- start 2024-03-23T11:19:47.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.349+0000 7f9e9dc00640 1 -- start start 2024-03-23T11:19:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.350+0000 7f9e9dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e9806c300 0x7f9e9814cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.350+0000 7f9e9dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e980cd7c0 0x7f9e981529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.350+0000 7f9e9dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e98152f20 0x7f9e980c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.350+0000 7f9e9dc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9814d4b0 con 0x7f9e9806c300 2024-03-23T11:19:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.350+0000 7f9e9dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e98151940 con 0x7f9e980cd7c0 2024-03-23T11:19:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.350+0000 7f9e9dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e98151c20 con 0x7f9e98152f20 2024-03-23T11:19:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.350+0000 7f9e96a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e9806c300 0x7f9e9814cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.350+0000 7f9e96a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e9806c300 0x7f9e9814cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59614/0 (socket says 172.21.15.62:59614) 2024-03-23T11:19:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.350+0000 7f9e96a00640 1 -- 172.21.15.62:0/3597188422 learned_addr learned my addr 172.21.15.62:0/3597188422 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e97400640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e98152f20 0x7f9e980c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e96a00640 1 -- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e98152f20 msgr2=0x7f9e980c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e8fe00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e980cd7c0 0x7f9e981529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e96a00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e98152f20 0x7f9e980c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e96a00640 1 -- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e980cd7c0 msgr2=0x7f9e981529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e96a00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e980cd7c0 0x7f9e981529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e96a00640 1 -- 172.21.15.62:0/3597188422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e88009400 con 0x7f9e9806c300 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e97400640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e98152f20 0x7f9e980c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e8fe00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e980cd7c0 0x7f9e981529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e96a00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e9806c300 0x7f9e9814cf70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9e880098c0 tx=0x7f9e880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e8ea00640 1 -- 172.21.15.62:0/3597188422 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8803c880 con 0x7f9e9806c300 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.351+0000 7f9e8ea00640 1 -- 172.21.15.62:0/3597188422 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8803c9e0 con 0x7f9e9806c300 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.352+0000 7f9e8ea00640 1 -- 172.21.15.62:0/3597188422 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8803cd90 con 0x7f9e9806c300 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.352+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3597188422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e98151f00 con 0x7f9e9806c300 2024-03-23T11:19:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.352+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3597188422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e981524c0 con 0x7f9e9806c300 2024-03-23T11:19:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.353+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3597188422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e981571a0 con 0x7f9e9806c300 2024-03-23T11:19:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.354+0000 7f9e8ea00640 1 -- 172.21.15.62:0/3597188422 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9e88039040 con 0x7f9e9806c300 2024-03-23T11:19:47.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.355+0000 7f9e8ea00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e7406a7a0 0x7f9e7406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:47.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.355+0000 7f9e8ea00640 1 -- 172.21.15.62:0/3597188422 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9e880aa9e0 con 0x7f9e9806c300 2024-03-23T11:19:47.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.356+0000 7f9e8fe00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e7406a7a0 0x7f9e7406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.357+0000 7f9e8fe00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e7406a7a0 0x7f9e7406cc60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9e981599c0 tx=0x7f9e80009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:47.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.360+0000 7f9e8ea00640 1 -- 172.21.15.62:0/3597188422 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e88075d70 con 0x7f9e9806c300 2024-03-23T11:19:47.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.562+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3597188422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9e9806c970 con 0x7f9e9806c300 2024-03-23T11:19:47.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.563+0000 7f9e8ea00640 1 -- 172.21.15.62:0/3597188422 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9e8804d020 con 0x7f9e9806c300 2024-03-23T11:19:47.563 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:47.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.565+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e7406a7a0 msgr2=0x7f9e7406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.565+0000 7f9e9dc00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e7406a7a0 0x7f9e7406cc60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9e981599c0 tx=0x7f9e80009040 comp rx=0 tx=0).stop 2024-03-23T11:19:47.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.565+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3597188422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e9806c300 msgr2=0x7f9e9814cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.565+0000 7f9e9dc00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e9806c300 0x7f9e9814cf70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9e880098c0 tx=0x7f9e880046c0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.565+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3597188422 shutdown_connections 2024-03-23T11:19:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.565+0000 7f9e9dc00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e7406a7a0 0x7f9e7406cc60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.565+0000 7f9e9dc00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e98152f20 0x7f9e980c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.565+0000 7f9e9dc00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e980cd7c0 0x7f9e981529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.565+0000 7f9e9dc00640 1 --2- 172.21.15.62:0/3597188422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e9806c300 0x7f9e9814cf70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.565+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3597188422 >> 172.21.15.62:0/3597188422 conn(0x7f9e980b02e0 msgr2=0x7f9e980702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.566+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3597188422 shutdown_connections 2024-03-23T11:19:47.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.566+0000 7f9e9dc00640 1 -- 172.21.15.62:0/3597188422 wait complete. 2024-03-23T11:19:47.585 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:47.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.809+0000 7f615b400640 1 Processor -- start 2024-03-23T11:19:47.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.809+0000 7f615b400640 1 -- start start 2024-03-23T11:19:47.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.810+0000 7f615b400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f61540cafa0 0x7f61540cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:47.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.810+0000 7f615b400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f615406c6e0 0x7f615406cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:47.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.810+0000 7f615b400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61540cbc60 con 0x7f615406c300 2024-03-23T11:19:47.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.810+0000 7f615b400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61540cd410 con 0x7f61540cafa0 2024-03-23T11:19:47.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.810+0000 7f615b400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61540cbf40 con 0x7f615406c6e0 2024-03-23T11:19:47.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.811+0000 7f6153e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f61540cafa0 0x7f61540cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:47.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.811+0000 7f6153e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f61540cafa0 0x7f61540cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59626/0 (socket says 172.21.15.62:59626) 2024-03-23T11:19:47.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.811+0000 7f6153e00640 1 -- 172.21.15.62:0/389655950 learned_addr learned my addr 172.21.15.62:0/389655950 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:47.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.811+0000 7f6159400640 1 --2- 172.21.15.62:0/389655950 >> v2:172.21.15.120:3300/0 conn(0x7f615406c6e0 0x7f615406cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:47.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.811+0000 7f6153e00640 1 -- 172.21.15.62:0/389655950 >> v2:172.21.15.120:3300/0 conn(0x7f615406c6e0 msgr2=0x7f615406cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:47.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.811+0000 7f6153e00640 1 --2- 172.21.15.62:0/389655950 >> v2:172.21.15.120:3300/0 conn(0x7f615406c6e0 0x7f615406cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.811+0000 7f6153e00640 1 -- 172.21.15.62:0/389655950 >> v1:172.21.15.62:6789/0 conn(0x7f615406c300 legacy=0x7f61540ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.811+0000 7f6153e00640 1 -- 172.21.15.62:0/389655950 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61540cc220 con 0x7f61540cafa0 2024-03-23T11:19:47.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.811+0000 7f6159400640 1 --2- 172.21.15.62:0/389655950 >> v2:172.21.15.120:3300/0 conn(0x7f615406c6e0 0x7f615406cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:47.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.811+0000 7f6153400640 1 -- 172.21.15.62:0/389655950 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786710109 0 0) 0x7f61540cbc60 con 0x7f615406c300 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.813+0000 7f6153e00640 1 --2- 172.21.15.62:0/389655950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61540cafa0 0x7f61540cb430 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6154070270 tx=0x7f614402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e13f07f25d68ced server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.813+0000 7f6153400640 1 -- 172.21.15.62:0/389655950 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6144030870 con 0x7f61540cafa0 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.813+0000 7f6153400640 1 -- 172.21.15.62:0/389655950 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61440309d0 con 0x7f61540cafa0 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.813+0000 7f6153400640 1 -- 172.21.15.62:0/389655950 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6144030d60 con 0x7f61540cafa0 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.813+0000 7f615b400640 1 -- 172.21.15.62:0/389655950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61540cafa0 msgr2=0x7f61540cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.813+0000 7f615b400640 1 --2- 172.21.15.62:0/389655950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61540cafa0 0x7f61540cb430 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6154070270 tx=0x7f614402b9b0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.813+0000 7f615b400640 1 -- 172.21.15.62:0/389655950 shutdown_connections 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.813+0000 7f615b400640 1 --2- 172.21.15.62:0/389655950 >> v2:172.21.15.120:3300/0 conn(0x7f615406c6e0 0x7f615406cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.813+0000 7f615b400640 1 --2- 172.21.15.62:0/389655950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61540cafa0 0x7f61540cb430 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.813+0000 7f615b400640 1 -- 172.21.15.62:0/389655950 >> 172.21.15.62:0/389655950 conn(0x7f61540b02e0 msgr2=0x7f61540b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.814+0000 7f615b400640 1 -- 172.21.15.62:0/389655950 shutdown_connections 2024-03-23T11:19:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.814+0000 7f615b400640 1 -- 172.21.15.62:0/389655950 wait complete. 2024-03-23T11:19:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.815+0000 7f615b400640 1 Processor -- start 2024-03-23T11:19:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.816+0000 7f615b400640 1 -- start start 2024-03-23T11:19:47.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f615b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f615406c300 0x7f61540ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f615b400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f615406c6e0 0x7f6154157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f615b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61540cafa0 0x7f615415bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f615b400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61540b30c0 con 0x7f615406c300 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f615b400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61540ca7b0 con 0x7f615406c6e0 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f615b400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61541575a0 con 0x7f61540cafa0 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f6158a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f615406c300 0x7f61540ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f6158a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f615406c300 0x7f61540ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59636/0 (socket says 172.21.15.62:59636) 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f6158a00640 1 -- 172.21.15.62:0/4203011755 learned_addr learned my addr 172.21.15.62:0/4203011755 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f6158a00640 1 -- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61540cafa0 msgr2=0x7f615415bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f6158a00640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61540cafa0 0x7f615415bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f6158a00640 1 -- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f615406c6e0 msgr2=0x7f6154157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f6158a00640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f615406c6e0 0x7f6154157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.817+0000 7f6158a00640 1 -- 172.21.15.62:0/4203011755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61440094d0 con 0x7f615406c300 2024-03-23T11:19:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.818+0000 7f6158a00640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f615406c300 0x7f61540ca270 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f614800c4e0 tx=0x7f614800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:47.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.818+0000 7f6150c00640 1 -- 172.21.15.62:0/4203011755 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148005480 con 0x7f615406c300 2024-03-23T11:19:47.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.818+0000 7f6150c00640 1 -- 172.21.15.62:0/4203011755 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61480035f0 con 0x7f615406c300 2024-03-23T11:19:47.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.818+0000 7f615b400640 1 -- 172.21.15.62:0/4203011755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61540b2440 con 0x7f615406c300 2024-03-23T11:19:47.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.818+0000 7f6150c00640 1 -- 172.21.15.62:0/4203011755 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61480039c0 con 0x7f615406c300 2024-03-23T11:19:47.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.818+0000 7f615b400640 1 -- 172.21.15.62:0/4203011755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61540b2a00 con 0x7f615406c300 2024-03-23T11:19:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.819+0000 7f615b400640 1 -- 172.21.15.62:0/4203011755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6154066c90 con 0x7f615406c300 2024-03-23T11:19:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.820+0000 7f6150c00640 1 -- 172.21.15.62:0/4203011755 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6148003b20 con 0x7f615406c300 2024-03-23T11:19:47.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.821+0000 7f6150c00640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f613806a7a0 0x7f613806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:47.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.821+0000 7f6150c00640 1 -- 172.21.15.62:0/4203011755 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f61480883b0 con 0x7f615406c300 2024-03-23T11:19:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.822+0000 7f6153e00640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f613806a7a0 0x7f613806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:47.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.823+0000 7f6153e00640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f613806a7a0 0x7f613806cc60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6144036000 tx=0x7f6144030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:47.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:47.825+0000 7f6150c00640 1 -- 172.21.15.62:0/4203011755 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6148053760 con 0x7f615406c300 2024-03-23T11:19:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.019+0000 7f615b400640 1 -- 172.21.15.62:0/4203011755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6154140c60 con 0x7f615406c300 2024-03-23T11:19:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.020+0000 7f6150c00640 1 -- 172.21.15.62:0/4203011755 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f61480170a0 con 0x7f615406c300 2024-03-23T11:19:48.020 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.022+0000 7f615b400640 1 -- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f613806a7a0 msgr2=0x7f613806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.022+0000 7f615b400640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f613806a7a0 0x7f613806cc60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6144036000 tx=0x7f6144030000 comp rx=0 tx=0).stop 2024-03-23T11:19:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.022+0000 7f615b400640 1 -- 172.21.15.62:0/4203011755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f615406c300 msgr2=0x7f61540ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.022+0000 7f615b400640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f615406c300 0x7f61540ca270 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f614800c4e0 tx=0x7f614800c9a0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.022+0000 7f615b400640 1 -- 172.21.15.62:0/4203011755 shutdown_connections 2024-03-23T11:19:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.022+0000 7f615b400640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f613806a7a0 0x7f613806cc60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.022+0000 7f615b400640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61540cafa0 0x7f615415bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.022+0000 7f615b400640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f615406c6e0 0x7f6154157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.022+0000 7f615b400640 1 --2- 172.21.15.62:0/4203011755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f615406c300 0x7f61540ca270 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.022+0000 7f615b400640 1 -- 172.21.15.62:0/4203011755 >> 172.21.15.62:0/4203011755 conn(0x7f61540b02e0 msgr2=0x7f61540cd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.023+0000 7f615b400640 1 -- 172.21.15.62:0/4203011755 shutdown_connections 2024-03-23T11:19:48.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.023+0000 7f615b400640 1 -- 172.21.15.62:0/4203011755 wait complete. 2024-03-23T11:19:48.041 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:19:48.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.253+0000 7f5c3ca00640 1 Processor -- start 2024-03-23T11:19:48.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.254+0000 7f5c3ca00640 1 -- start start 2024-03-23T11:19:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.255+0000 7f5c3ca00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5c381593e0 0x7f5c3815d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:48.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.256+0000 7f5c3ca00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c380b14c0 con 0x7f5c380b2b30 2024-03-23T11:19:48.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.256+0000 7f5c3ca00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3806a6f0 con 0x7f5c3806a290 2024-03-23T11:19:48.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.256+0000 7f5c3ca00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3806a020 con 0x7f5c381593e0 2024-03-23T11:19:48.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.256+0000 7f5c36800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5c381593e0 0x7f5c3815d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.256+0000 7f5c37200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5c380b2b30 0x7f5c380b2f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60838/0 (socket says 172.21.15.62:60838) 2024-03-23T11:19:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.256+0000 7f5c36800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5c381593e0 0x7f5c3815d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59640/0 (socket says 172.21.15.62:59640) 2024-03-23T11:19:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.256+0000 7f5c37200640 1 -- 172.21.15.62:0/1270223486 learned_addr learned my addr 172.21.15.62:0/1270223486 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.256+0000 7f5c36800640 1 -- 172.21.15.62:0/1270223486 >> v1:172.21.15.120:6789/0 conn(0x7f5c3806a290 legacy=0x7f5c380cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.256+0000 7f5c36800640 1 -- 172.21.15.62:0/1270223486 >> v1:172.21.15.62:6789/0 conn(0x7f5c380b2b30 legacy=0x7f5c380b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.256+0000 7f5c36800640 1 -- 172.21.15.62:0/1270223486 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3815dee0 con 0x7f5c381593e0 2024-03-23T11:19:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.256+0000 7f5c35e00640 1 -- 172.21.15.62:0/1270223486 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144367507 0 0) 0x7f5c380b14c0 con 0x7f5c380b2b30 2024-03-23T11:19:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.258+0000 7f5c36800640 1 --2- 172.21.15.62:0/1270223486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c381593e0 0x7f5c3815d060 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5c280097a0 tx=0x7f5c2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d35b7bfae90262e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:48.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.258+0000 7f5c35e00640 1 -- 172.21.15.62:0/1270223486 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c28031830 con 0x7f5c381593e0 2024-03-23T11:19:48.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.258+0000 7f5c35e00640 1 -- 172.21.15.62:0/1270223486 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c28031990 con 0x7f5c381593e0 2024-03-23T11:19:48.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.258+0000 7f5c35e00640 1 -- 172.21.15.62:0/1270223486 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c28031d20 con 0x7f5c381593e0 2024-03-23T11:19:48.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.259+0000 7f5c3ca00640 1 -- 172.21.15.62:0/1270223486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c381593e0 msgr2=0x7f5c3815d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:48.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.259+0000 7f5c3ca00640 1 --2- 172.21.15.62:0/1270223486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c381593e0 0x7f5c3815d060 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5c280097a0 tx=0x7f5c2802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.259+0000 7f5c3ca00640 1 -- 172.21.15.62:0/1270223486 shutdown_connections 2024-03-23T11:19:48.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.259+0000 7f5c3ca00640 1 --2- 172.21.15.62:0/1270223486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c381593e0 0x7f5c3815d060 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.259+0000 7f5c3ca00640 1 -- 172.21.15.62:0/1270223486 >> 172.21.15.62:0/1270223486 conn(0x7f5c380b0090 msgr2=0x7f5c380b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:48.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.259+0000 7f5c3ca00640 1 -- 172.21.15.62:0/1270223486 shutdown_connections 2024-03-23T11:19:48.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.259+0000 7f5c3ca00640 1 -- 172.21.15.62:0/1270223486 wait complete. 2024-03-23T11:19:48.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.261+0000 7f5c3ca00640 1 Processor -- start 2024-03-23T11:19:48.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.261+0000 7f5c3ca00640 1 -- start start 2024-03-23T11:19:48.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c3ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c3806a290 0x7f5c380cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:48.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c3ca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c380b2b30 0x7f5c380af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:48.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c3ca00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c381593e0 0x7f5c380afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:48.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c3ca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380cc9c0 con 0x7f5c3806a290 2024-03-23T11:19:48.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c3ca00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380ccca0 con 0x7f5c381593e0 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c3ca00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380ccf80 con 0x7f5c380b2b30 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c36800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c3806a290 0x7f5c380cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c36800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c3806a290 0x7f5c380cc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59654/0 (socket says 172.21.15.62:59654) 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c36800640 1 -- 172.21.15.62:0/2334464479 learned_addr learned my addr 172.21.15.62:0/2334464479 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c36800640 1 -- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c380b2b30 msgr2=0x7f5c380af4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c36800640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c380b2b30 0x7f5c380af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c37200640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c381593e0 0x7f5c380afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c36800640 1 -- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c381593e0 msgr2=0x7f5c380afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c36800640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c381593e0 0x7f5c380afa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.262+0000 7f5c36800640 1 -- 172.21.15.62:0/2334464479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c28009400 con 0x7f5c3806a290 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.263+0000 7f5c36800640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c3806a290 0x7f5c380cc0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5c280097a0 tx=0x7f5c280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.263+0000 7f5c2ea00640 1 -- 172.21.15.62:0/2334464479 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2803aa60 con 0x7f5c3806a290 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.263+0000 7f5c2ea00640 1 -- 172.21.15.62:0/2334464479 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2803abc0 con 0x7f5c3806a290 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.263+0000 7f5c2ea00640 1 -- 172.21.15.62:0/2334464479 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c280443f0 con 0x7f5c3806a290 2024-03-23T11:19:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.263+0000 7f5c3ca00640 1 -- 172.21.15.62:0/2334464479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c380cd260 con 0x7f5c3806a290 2024-03-23T11:19:48.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.263+0000 7f5c3ca00640 1 -- 172.21.15.62:0/2334464479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c380cc600 con 0x7f5c3806a290 2024-03-23T11:19:48.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.264+0000 7f5c3ca00640 1 -- 172.21.15.62:0/2334464479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c380cd780 con 0x7f5c3806a290 2024-03-23T11:19:48.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.265+0000 7f5c2ea00640 1 -- 172.21.15.62:0/2334464479 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5c2804d450 con 0x7f5c3806a290 2024-03-23T11:19:48.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.266+0000 7f5c2ea00640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c1406a7a0 0x7f5c1406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:48.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.266+0000 7f5c2ea00640 1 -- 172.21.15.62:0/2334464479 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5c280ab000 con 0x7f5c3806a290 2024-03-23T11:19:48.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.267+0000 7f5c2fe00640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c1406a7a0 0x7f5c1406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:48.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.268+0000 7f5c2fe00640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c1406a7a0 0x7f5c1406cc60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5c381f5ad0 tx=0x7f5c200079b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:48.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.271+0000 7f5c2ea00640 1 -- 172.21.15.62:0/2334464479 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c380cd780 con 0x7f5c3806a290 2024-03-23T11:19:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.478+0000 7f5c3ca00640 1 -- 172.21.15.62:0/2334464479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5c380cd780 con 0x7f5c3806a290 2024-03-23T11:19:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.478+0000 7f5c2ea00640 1 -- 172.21.15.62:0/2334464479 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5c380cd780 con 0x7f5c3806a290 2024-03-23T11:19:48.479 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:19:48.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 -- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c1406a7a0 msgr2=0x7f5c1406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:48.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c1406a7a0 0x7f5c1406cc60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5c381f5ad0 tx=0x7f5c200079b0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 -- 172.21.15.62:0/2334464479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c3806a290 msgr2=0x7f5c380cc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c3806a290 0x7f5c380cc0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5c280097a0 tx=0x7f5c280046c0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 -- 172.21.15.62:0/2334464479 shutdown_connections 2024-03-23T11:19:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c1406a7a0 0x7f5c1406cc60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c381593e0 0x7f5c380afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c380b2b30 0x7f5c380af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 --2- 172.21.15.62:0/2334464479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c3806a290 0x7f5c380cc0c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 -- 172.21.15.62:0/2334464479 >> 172.21.15.62:0/2334464479 conn(0x7f5c380b0090 msgr2=0x7f5c380ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 -- 172.21.15.62:0/2334464479 shutdown_connections 2024-03-23T11:19:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.481+0000 7f5c3ca00640 1 -- 172.21.15.62:0/2334464479 wait complete. 2024-03-23T11:19:48.500 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:19:48.500 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:48.557 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:48.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.777+0000 7fbf66600640 1 Processor -- start 2024-03-23T11:19:48.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.777+0000 7fbf66600640 1 -- start start 2024-03-23T11:19:48.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.779+0000 7fbf66600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf600d0d40 0x7fbf600d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:48.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.779+0000 7fbf66600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbf600b24e0 0x7fbf600b2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:48.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.779+0000 7fbf66600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf600ce870 con 0x7fbf6015ca00 2024-03-23T11:19:48.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.779+0000 7fbf66600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf600d1700 con 0x7fbf600b24e0 2024-03-23T11:19:48.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.779+0000 7fbf66600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600b2eb0 con 0x7fbf600d0d40 2024-03-23T11:19:48.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.779+0000 7fbf65c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbf600b24e0 0x7fbf600b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:48.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.779+0000 7fbf65c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbf600b24e0 0x7fbf600b2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59658/0 (socket says 172.21.15.62:59658) 2024-03-23T11:19:48.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.779+0000 7fbf65c00640 1 -- 172.21.15.62:0/752133311 learned_addr learned my addr 172.21.15.62:0/752133311 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:48.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.780+0000 7fbf5fe00640 1 --2- 172.21.15.62:0/752133311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf600d0d40 0x7fbf600d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:48.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.780+0000 7fbf65c00640 1 -- 172.21.15.62:0/752133311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf600d0d40 msgr2=0x7fbf600d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:48.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.780+0000 7fbf65c00640 1 --2- 172.21.15.62:0/752133311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf600d0d40 0x7fbf600d11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.780+0000 7fbf65c00640 1 -- 172.21.15.62:0/752133311 >> v1:172.21.15.120:6789/0 conn(0x7fbf6015ca00 legacy=0x7fbf600ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:48.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.780+0000 7fbf65c00640 1 -- 172.21.15.62:0/752133311 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf600cfe40 con 0x7fbf600b24e0 2024-03-23T11:19:48.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.780+0000 7fbf5fe00640 1 --2- 172.21.15.62:0/752133311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf600d0d40 0x7fbf600d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:48.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.781+0000 7fbf65c00640 1 --2- 172.21.15.62:0/752133311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf600b24e0 0x7fbf600b2970 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009870 tx=0x7fbf5002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135dc8b616d17c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:48.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.781+0000 7fbf5f400640 1 -- 172.21.15.62:0/752133311 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50031640 con 0x7fbf600b24e0 2024-03-23T11:19:48.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.781+0000 7fbf5f400640 1 -- 172.21.15.62:0/752133311 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf500317a0 con 0x7fbf600b24e0 2024-03-23T11:19:48.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.781+0000 7fbf5f400640 1 -- 172.21.15.62:0/752133311 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50031b30 con 0x7fbf600b24e0 2024-03-23T11:19:48.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.782+0000 7fbf66600640 1 -- 172.21.15.62:0/752133311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf600b24e0 msgr2=0x7fbf600b2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:48.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.782+0000 7fbf66600640 1 --2- 172.21.15.62:0/752133311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf600b24e0 0x7fbf600b2970 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009870 tx=0x7fbf5002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.782+0000 7fbf66600640 1 -- 172.21.15.62:0/752133311 shutdown_connections 2024-03-23T11:19:48.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.782+0000 7fbf66600640 1 --2- 172.21.15.62:0/752133311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf600b24e0 0x7fbf600b2970 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.782+0000 7fbf66600640 1 --2- 172.21.15.62:0/752133311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf600d0d40 0x7fbf600d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.782+0000 7fbf66600640 1 -- 172.21.15.62:0/752133311 >> 172.21.15.62:0/752133311 conn(0x7fbf6009ba10 msgr2=0x7fbf60089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:48.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.782+0000 7fbf66600640 1 -- 172.21.15.62:0/752133311 shutdown_connections 2024-03-23T11:19:48.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.782+0000 7fbf66600640 1 -- 172.21.15.62:0/752133311 wait complete. 2024-03-23T11:19:48.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.784+0000 7fbf66600640 1 Processor -- start 2024-03-23T11:19:48.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.784+0000 7fbf66600640 1 -- start start 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf66600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf600b24e0 0x7fbf60153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf66600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf600d0d40 0x7fbf6015cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf66600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf6015ca00 0x7fbf600c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf66600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf60154370 con 0x7fbf600b24e0 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf66600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6015d320 con 0x7fbf6015ca00 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf66600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6015d5f0 con 0x7fbf600d0d40 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf65200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf600b24e0 0x7fbf60153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf65200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf600b24e0 0x7fbf60153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59666/0 (socket says 172.21.15.62:59666) 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf65200640 1 -- 172.21.15.62:0/1079185433 learned_addr learned my addr 172.21.15.62:0/1079185433 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf65200640 1 -- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf600d0d40 msgr2=0x7fbf6015cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf65200640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf600d0d40 0x7fbf6015cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf65200640 1 -- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf6015ca00 msgr2=0x7fbf600c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:48.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf65200640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf6015ca00 0x7fbf600c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:48.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.785+0000 7fbf65200640 1 -- 172.21.15.62:0/1079185433 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf500094d0 con 0x7fbf600b24e0 2024-03-23T11:19:48.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.786+0000 7fbf65200640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf600b24e0 0x7fbf60153e30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800c7a0 tx=0x7fbf5800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:48.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.786+0000 7fbf5cc00640 1 -- 172.21.15.62:0/1079185433 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5800aa50 con 0x7fbf600b24e0 2024-03-23T11:19:48.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.786+0000 7fbf5cc00640 1 -- 172.21.15.62:0/1079185433 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5800abb0 con 0x7fbf600b24e0 2024-03-23T11:19:48.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.786+0000 7fbf66600640 1 -- 172.21.15.62:0/1079185433 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6015d8d0 con 0x7fbf600b24e0 2024-03-23T11:19:48.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.786+0000 7fbf5cc00640 1 -- 172.21.15.62:0/1079185433 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58012420 con 0x7fbf600b24e0 2024-03-23T11:19:48.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.786+0000 7fbf66600640 1 -- 172.21.15.62:0/1079185433 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf600b67f0 con 0x7fbf600b24e0 2024-03-23T11:19:48.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.787+0000 7fbf66600640 1 -- 172.21.15.62:0/1079185433 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf600b69e0 con 0x7fbf600b24e0 2024-03-23T11:19:48.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.788+0000 7fbf5cc00640 1 -- 172.21.15.62:0/1079185433 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbf58012580 con 0x7fbf600b24e0 2024-03-23T11:19:48.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.790+0000 7fbf5cc00640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf3c06a7a0 0x7fbf3c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:48.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.790+0000 7fbf5cc00640 1 -- 172.21.15.62:0/1079185433 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbf58089420 con 0x7fbf600b24e0 2024-03-23T11:19:48.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.790+0000 7fbf5fe00640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf3c06a7a0 0x7fbf3c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:48.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.792+0000 7fbf5fe00640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf3c06a7a0 0x7fbf3c06cc60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbf48006fd0 tx=0x7fbf48008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:48.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.794+0000 7fbf5cc00640 1 -- 172.21.15.62:0/1079185433 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf580547b0 con 0x7fbf600b24e0 2024-03-23T11:19:48.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.996+0000 7fbf66600640 1 -- 172.21.15.62:0/1079185433 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf600672c0 con 0x7fbf600b24e0 2024-03-23T11:19:48.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:48.999+0000 7fbf5cc00640 1 -- 172.21.15.62:0/1079185433 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16226 (secure 0 0 0) 0x7fbf58016090 con 0x7fbf600b24e0 2024-03-23T11:19:48.999 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:48.999 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:19:42.867981+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 -- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf3c06a7a0 msgr2=0x7fbf3c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf3c06a7a0 0x7fbf3c06cc60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbf48006fd0 tx=0x7fbf48008040 comp rx=0 tx=0).stop 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 -- 172.21.15.62:0/1079185433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf600b24e0 msgr2=0x7fbf60153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf600b24e0 0x7fbf60153e30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800c7a0 tx=0x7fbf5800cc60 comp rx=0 tx=0).stop 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 -- 172.21.15.62:0/1079185433 shutdown_connections 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf3c06a7a0 0x7fbf3c06cc60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf6015ca00 0x7fbf600c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf600d0d40 0x7fbf6015cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 --2- 172.21.15.62:0/1079185433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf600b24e0 0x7fbf60153e30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 -- 172.21.15.62:0/1079185433 >> 172.21.15.62:0/1079185433 conn(0x7fbf6009ba10 msgr2=0x7fbf60089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 -- 172.21.15.62:0/1079185433 shutdown_connections 2024-03-23T11:19:49.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.002+0000 7fbf66600640 1 -- 172.21.15.62:0/1079185433 wait complete. 2024-03-23T11:19:49.022 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:49.023 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:49.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.234+0000 7febf7200640 1 Processor -- start 2024-03-23T11:19:49.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.234+0000 7febf7200640 1 -- start start 2024-03-23T11:19:49.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.235+0000 7febf7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf815fcf0 0x7febf80d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.236+0000 7febf7200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7febf8160420 0x7febf8160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.236+0000 7febf7200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf8160dc0 con 0x7febf80d06d0 2024-03-23T11:19:49.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.236+0000 7febf7200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf80d3540 con 0x7febf8160420 2024-03-23T11:19:49.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.236+0000 7febf7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf80ce510 con 0x7febf815fcf0 2024-03-23T11:19:49.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.236+0000 7febf6800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7febf80d06d0 0x7febf80d0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60848/0 (socket says 172.21.15.62:60848) 2024-03-23T11:19:49.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.236+0000 7febf6800640 1 -- 172.21.15.62:0/2502588018 learned_addr learned my addr 172.21.15.62:0/2502588018 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:49.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.236+0000 7febf5400640 1 --2- 172.21.15.62:0/2502588018 >> v2:172.21.15.62:3300/0 conn(0x7febf8160420 0x7febf8160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:49.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.236+0000 7febf5e00640 1 --2- 172.21.15.62:0/2502588018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf815fcf0 0x7febf80d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:49.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.237+0000 7febf5400640 1 -- 172.21.15.62:0/2502588018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf815fcf0 msgr2=0x7febf80d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:49.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.237+0000 7febf5400640 1 --2- 172.21.15.62:0/2502588018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf815fcf0 0x7febf80d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.237+0000 7febf5400640 1 -- 172.21.15.62:0/2502588018 >> v1:172.21.15.62:6789/0 conn(0x7febf80d06d0 legacy=0x7febf80d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.237+0000 7febf5400640 1 -- 172.21.15.62:0/2502588018 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf80cdc70 con 0x7febf8160420 2024-03-23T11:19:49.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.238+0000 7febf5400640 1 --2- 172.21.15.62:0/2502588018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf8160420 0x7febf8160880 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7febe00097a0 tx=0x7febe002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f47d1782af44c9b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:49.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.238+0000 7febf4a00640 1 -- 172.21.15.62:0/2502588018 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0031780 con 0x7febf8160420 2024-03-23T11:19:49.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.238+0000 7febf4a00640 1 -- 172.21.15.62:0/2502588018 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe00318e0 con 0x7febf8160420 2024-03-23T11:19:49.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.238+0000 7febf4a00640 1 -- 172.21.15.62:0/2502588018 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0031c70 con 0x7febf8160420 2024-03-23T11:19:49.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.239+0000 7febf7200640 1 -- 172.21.15.62:0/2502588018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf8160420 msgr2=0x7febf8160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.239+0000 7febf7200640 1 --2- 172.21.15.62:0/2502588018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf8160420 0x7febf8160880 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7febe00097a0 tx=0x7febe002b820 comp rx=0 tx=0).stop 2024-03-23T11:19:49.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.239+0000 7febf7200640 1 -- 172.21.15.62:0/2502588018 shutdown_connections 2024-03-23T11:19:49.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.239+0000 7febf7200640 1 --2- 172.21.15.62:0/2502588018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf8160420 0x7febf8160880 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.239+0000 7febf7200640 1 --2- 172.21.15.62:0/2502588018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf815fcf0 0x7febf80d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.239+0000 7febf7200640 1 -- 172.21.15.62:0/2502588018 >> 172.21.15.62:0/2502588018 conn(0x7febf8089f60 msgr2=0x7febf80d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:49.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.239+0000 7febf7200640 1 -- 172.21.15.62:0/2502588018 shutdown_connections 2024-03-23T11:19:49.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.239+0000 7febf7200640 1 -- 172.21.15.62:0/2502588018 wait complete. 2024-03-23T11:19:49.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.241+0000 7febf7200640 1 Processor -- start 2024-03-23T11:19:49.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.241+0000 7febf7200640 1 -- start start 2024-03-23T11:19:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf80d06d0 0x7febf8158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febf815fcf0 0x7febf81f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf8160420 0x7febf8159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf81f3a40 con 0x7febf80d06d0 2024-03-23T11:19:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf81f3d20 con 0x7febf815fcf0 2024-03-23T11:19:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf81f4000 con 0x7febf8160420 2024-03-23T11:19:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf80d06d0 0x7febf8158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf80d06d0 0x7febf8158ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59684/0 (socket says 172.21.15.62:59684) 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf5e00640 1 -- 172.21.15.62:0/2900522574 learned_addr learned my addr 172.21.15.62:0/2900522574 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf5e00640 1 -- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf8160420 msgr2=0x7febf8159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf5e00640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf8160420 0x7febf8159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf5e00640 1 -- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febf815fcf0 msgr2=0x7febf81f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf5e00640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febf815fcf0 0x7febf81f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.242+0000 7febf5e00640 1 -- 172.21.15.62:0/2900522574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0009400 con 0x7febf80d06d0 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.243+0000 7febf5e00640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf80d06d0 0x7febf8158ce0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7febe800d800 tx=0x7febe800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.243+0000 7febee000640 1 -- 172.21.15.62:0/2900522574 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe800b6c0 con 0x7febf80d06d0 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.243+0000 7febee000640 1 -- 172.21.15.62:0/2900522574 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe800b820 con 0x7febf80d06d0 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.243+0000 7febf7200640 1 -- 172.21.15.62:0/2900522574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf81f42e0 con 0x7febf80d06d0 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.243+0000 7febee000640 1 -- 172.21.15.62:0/2900522574 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe800bbb0 con 0x7febf80d06d0 2024-03-23T11:19:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.243+0000 7febf7200640 1 -- 172.21.15.62:0/2900522574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf8159220 con 0x7febf80d06d0 2024-03-23T11:19:49.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.244+0000 7febf7200640 1 -- 172.21.15.62:0/2900522574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf8066bb0 con 0x7febf80d06d0 2024-03-23T11:19:49.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.245+0000 7febee000640 1 -- 172.21.15.62:0/2900522574 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7febe800bd10 con 0x7febf80d06d0 2024-03-23T11:19:49.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.247+0000 7febee000640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7febd406a7a0 0x7febd406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.247+0000 7febee000640 1 -- 172.21.15.62:0/2900522574 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7febe80893d0 con 0x7febf80d06d0 2024-03-23T11:19:49.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.248+0000 7febf5400640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7febd406a7a0 0x7febd406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:49.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.249+0000 7febf5400640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7febd406a7a0 0x7febd406cc60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7febe0036000 tx=0x7febe0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:49.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.253+0000 7febee000640 1 -- 172.21.15.62:0/2900522574 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7febe8054760 con 0x7febf80d06d0 2024-03-23T11:19:49.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.453+0000 7febf7200640 1 -- 172.21.15.62:0/2900522574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7febf80cea40 con 0x7febf80d06d0 2024-03-23T11:19:49.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.454+0000 7febee000640 1 -- 172.21.15.62:0/2900522574 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7febe805e030 con 0x7febf80d06d0 2024-03-23T11:19:49.454 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:49.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.456+0000 7febf7200640 1 -- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7febd406a7a0 msgr2=0x7febd406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.456+0000 7febf7200640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7febd406a7a0 0x7febd406cc60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7febe0036000 tx=0x7febe0030000 comp rx=0 tx=0).stop 2024-03-23T11:19:49.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.456+0000 7febf7200640 1 -- 172.21.15.62:0/2900522574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf80d06d0 msgr2=0x7febf8158ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.456+0000 7febf7200640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf80d06d0 0x7febf8158ce0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7febe800d800 tx=0x7febe800dcc0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.457+0000 7febf7200640 1 -- 172.21.15.62:0/2900522574 shutdown_connections 2024-03-23T11:19:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.457+0000 7febf7200640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7febd406a7a0 0x7febd406cc60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.457+0000 7febf7200640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7febf8160420 0x7febf8159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.457+0000 7febf7200640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febf815fcf0 0x7febf81f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.457+0000 7febf7200640 1 --2- 172.21.15.62:0/2900522574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf80d06d0 0x7febf8158ce0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.457+0000 7febf7200640 1 -- 172.21.15.62:0/2900522574 >> 172.21.15.62:0/2900522574 conn(0x7febf8089f60 msgr2=0x7febf80b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.457+0000 7febf7200640 1 -- 172.21.15.62:0/2900522574 shutdown_connections 2024-03-23T11:19:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.457+0000 7febf7200640 1 -- 172.21.15.62:0/2900522574 wait complete. 2024-03-23T11:19:49.476 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:49.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.692+0000 7fc7c5e00640 1 Processor -- start 2024-03-23T11:19:49.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.692+0000 7fc7c5e00640 1 -- start start 2024-03-23T11:19:49.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.694+0000 7fc7c5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc7c0159490 0x7fc7c006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.694+0000 7fc7c5e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7c00ce5f0 0x7fc7c00b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.694+0000 7fc7c5e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c00ced90 con 0x7fc7c015da90 2024-03-23T11:19:49.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.694+0000 7fc7c5e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7c006e3e0 con 0x7fc7c0159490 2024-03-23T11:19:49.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.694+0000 7fc7c5e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c006c1c0 con 0x7fc7c00ce5f0 2024-03-23T11:19:49.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.694+0000 7fc7c4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc7c0159490 0x7fc7c006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:49.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.694+0000 7fc7c4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc7c0159490 0x7fc7c006cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59690/0 (socket says 172.21.15.62:59690) 2024-03-23T11:19:49.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.694+0000 7fc7bfe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc7c015da90 0x7fc7c00cdee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60862/0 (socket says 172.21.15.62:60862) 2024-03-23T11:19:49.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.694+0000 7fc7c4a00640 1 -- 172.21.15.62:0/3338622998 learned_addr learned my addr 172.21.15.62:0/3338622998 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:49.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.695+0000 7fc7c5400640 1 --2- 172.21.15.62:0/3338622998 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7c00ce5f0 0x7fc7c00b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:49.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.695+0000 7fc7c4a00640 1 -- 172.21.15.62:0/3338622998 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7c00ce5f0 msgr2=0x7fc7c00b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:49.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.695+0000 7fc7c4a00640 1 --2- 172.21.15.62:0/3338622998 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7c00ce5f0 0x7fc7c00b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.695+0000 7fc7c4a00640 1 -- 172.21.15.62:0/3338622998 >> v1:172.21.15.62:6789/0 conn(0x7fc7c015da90 legacy=0x7fc7c00cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.695+0000 7fc7c4a00640 1 -- 172.21.15.62:0/3338622998 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c006e100 con 0x7fc7c0159490 2024-03-23T11:19:49.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.695+0000 7fc7bf400640 1 -- 172.21.15.62:0/3338622998 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176471342 0 0) 0x7fc7c00ced90 con 0x7fc7c015da90 2024-03-23T11:19:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.696+0000 7fc7c4a00640 1 --2- 172.21.15.62:0/3338622998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c0159490 0x7fc7c006cc60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009870 tx=0x7fc7b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fbef4b4246bde5d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.696+0000 7fc7bf400640 1 -- 172.21.15.62:0/3338622998 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4031640 con 0x7fc7c0159490 2024-03-23T11:19:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.696+0000 7fc7bf400640 1 -- 172.21.15.62:0/3338622998 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b40317a0 con 0x7fc7c0159490 2024-03-23T11:19:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.696+0000 7fc7bf400640 1 -- 172.21.15.62:0/3338622998 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4031b30 con 0x7fc7c0159490 2024-03-23T11:19:49.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.697+0000 7fc7c5e00640 1 -- 172.21.15.62:0/3338622998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c0159490 msgr2=0x7fc7c006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.697+0000 7fc7c5e00640 1 --2- 172.21.15.62:0/3338622998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c0159490 0x7fc7c006cc60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009870 tx=0x7fc7b402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.697+0000 7fc7c5e00640 1 -- 172.21.15.62:0/3338622998 shutdown_connections 2024-03-23T11:19:49.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.697+0000 7fc7c5e00640 1 --2- 172.21.15.62:0/3338622998 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7c00ce5f0 0x7fc7c00b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.697+0000 7fc7c5e00640 1 --2- 172.21.15.62:0/3338622998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c0159490 0x7fc7c006cc60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.697+0000 7fc7c5e00640 1 -- 172.21.15.62:0/3338622998 >> 172.21.15.62:0/3338622998 conn(0x7fc7c0066050 msgr2=0x7fc7c0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:49.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.697+0000 7fc7c5e00640 1 -- 172.21.15.62:0/3338622998 shutdown_connections 2024-03-23T11:19:49.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.697+0000 7fc7c5e00640 1 -- 172.21.15.62:0/3338622998 wait complete. 2024-03-23T11:19:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.699+0000 7fc7c5e00640 1 Processor -- start 2024-03-23T11:19:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.699+0000 7fc7c5e00640 1 -- start start 2024-03-23T11:19:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.700+0000 7fc7c5e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7c00ce5f0 0x7fc7c0140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.700+0000 7fc7c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c0159490 0x7fc7c015c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.700+0000 7fc7c5e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7c015da90 0x7fc7c015a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.700+0000 7fc7c5e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c01412e0 con 0x7fc7c0159490 2024-03-23T11:19:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.700+0000 7fc7c5e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c01653e0 con 0x7fc7c00ce5f0 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.700+0000 7fc7c5e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c01580b0 con 0x7fc7c015da90 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.700+0000 7fc7bfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c0159490 0x7fc7c015c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.700+0000 7fc7bfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c0159490 0x7fc7c015c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59694/0 (socket says 172.21.15.62:59694) 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.700+0000 7fc7bfe00640 1 -- 172.21.15.62:0/2199611473 learned_addr learned my addr 172.21.15.62:0/2199611473 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.701+0000 7fc7c4a00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7c00ce5f0 0x7fc7c0140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.701+0000 7fc7c5400640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7c015da90 0x7fc7c015a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.701+0000 7fc7bfe00640 1 -- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7c015da90 msgr2=0x7fc7c015a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.701+0000 7fc7bfe00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7c015da90 0x7fc7c015a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.701+0000 7fc7bfe00640 1 -- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7c00ce5f0 msgr2=0x7fc7c0140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.701+0000 7fc7bfe00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7c00ce5f0 0x7fc7c0140da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.701+0000 7fc7bfe00640 1 -- 172.21.15.62:0/2199611473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b40094d0 con 0x7fc7c0159490 2024-03-23T11:19:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.701+0000 7fc7c5400640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7c015da90 0x7fc7c015a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:49.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.701+0000 7fc7bfe00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c0159490 0x7fc7c015c200 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00c430 tx=0x7fc7ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:49.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.701+0000 7fc7bcc00640 1 -- 172.21.15.62:0/2199611473 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0074e0 con 0x7fc7c0159490 2024-03-23T11:19:49.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.702+0000 7fc7bcc00640 1 -- 172.21.15.62:0/2199611473 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac007660 con 0x7fc7c0159490 2024-03-23T11:19:49.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.702+0000 7fc7bcc00640 1 -- 172.21.15.62:0/2199611473 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac004a20 con 0x7fc7c0159490 2024-03-23T11:19:49.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.702+0000 7fc7c5e00640 1 -- 172.21.15.62:0/2199611473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c0158390 con 0x7fc7c0159490 2024-03-23T11:19:49.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.702+0000 7fc7c5e00640 1 -- 172.21.15.62:0/2199611473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c0158950 con 0x7fc7c0159490 2024-03-23T11:19:49.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.703+0000 7fc7c5e00640 1 -- 172.21.15.62:0/2199611473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c0066bb0 con 0x7fc7c0159490 2024-03-23T11:19:49.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.704+0000 7fc7bcc00640 1 -- 172.21.15.62:0/2199611473 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc7ac004b80 con 0x7fc7c0159490 2024-03-23T11:19:49.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.705+0000 7fc7bcc00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc7a006a7a0 0x7fc7a006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:49.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.705+0000 7fc7bcc00640 1 -- 172.21.15.62:0/2199611473 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc7ac088d50 con 0x7fc7c0159490 2024-03-23T11:19:49.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.705+0000 7fc7c4a00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc7a006a7a0 0x7fc7a006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.706+0000 7fc7c4a00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc7a006a7a0 0x7fc7a006cc60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4036000 tx=0x7fc7b403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:49.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.709+0000 7fc7bcc00640 1 -- 172.21.15.62:0/2199611473 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc7ac054050 con 0x7fc7c0159490 2024-03-23T11:19:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.909+0000 7fc7c5e00640 1 -- 172.21.15.62:0/2199611473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc7c006d560 con 0x7fc7c0159490 2024-03-23T11:19:49.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.911+0000 7fc7bcc00640 1 -- 172.21.15.62:0/2199611473 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc7ac057670 con 0x7fc7c0159490 2024-03-23T11:19:49.911 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:49.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.913+0000 7fc7c5e00640 1 -- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc7a006a7a0 msgr2=0x7fc7a006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.913+0000 7fc7c5e00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc7a006a7a0 0x7fc7a006cc60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4036000 tx=0x7fc7b403c040 comp rx=0 tx=0).stop 2024-03-23T11:19:49.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.913+0000 7fc7c5e00640 1 -- 172.21.15.62:0/2199611473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c0159490 msgr2=0x7fc7c015c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:49.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.913+0000 7fc7c5e00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c0159490 0x7fc7c015c200 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00c430 tx=0x7fc7ac00c8f0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.913+0000 7fc7c5e00640 1 -- 172.21.15.62:0/2199611473 shutdown_connections 2024-03-23T11:19:49.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.913+0000 7fc7c5e00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc7a006a7a0 0x7fc7a006cc60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.913+0000 7fc7c5e00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c0159490 0x7fc7c015c200 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.913+0000 7fc7c5e00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7c015da90 0x7fc7c015a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.913+0000 7fc7c5e00640 1 --2- 172.21.15.62:0/2199611473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7c00ce5f0 0x7fc7c0140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:49.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.913+0000 7fc7c5e00640 1 -- 172.21.15.62:0/2199611473 >> 172.21.15.62:0/2199611473 conn(0x7fc7c0066050 msgr2=0x7fc7c00af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:49.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.914+0000 7fc7c5e00640 1 -- 172.21.15.62:0/2199611473 shutdown_connections 2024-03-23T11:19:49.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:49.914+0000 7fc7c5e00640 1 -- 172.21.15.62:0/2199611473 wait complete. 2024-03-23T11:19:49.931 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:50.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.148+0000 7fd487800640 1 Processor -- start 2024-03-23T11:19:50.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.149+0000 7fd487800640 1 -- start start 2024-03-23T11:19:50.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.150+0000 7fd487800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4800cf650 0x7fd4800cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.150+0000 7fd487800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48006bfc0 con 0x7fd48015c9e0 2024-03-23T11:19:50.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.150+0000 7fd487800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4800b5260 con 0x7fd48015cdc0 2024-03-23T11:19:50.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.150+0000 7fd487800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800d0020 con 0x7fd4800cf650 2024-03-23T11:19:50.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.150+0000 7fd484e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd48015c9e0 0x7fd4801607c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60874/0 (socket says 172.21.15.62:60874) 2024-03-23T11:19:50.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.150+0000 7fd484e00640 1 -- 172.21.15.62:0/602969455 learned_addr learned my addr 172.21.15.62:0/602969455 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:50.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.151+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053407799 0 0) 0x7fd48006bfc0 con 0x7fd48015c9e0 2024-03-23T11:19:50.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.151+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd464003520 con 0x7fd48015c9e0 2024-03-23T11:19:50.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.151+0000 7fd485800640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4800cf650 0x7fd4800cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.151+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 323915452 0 0) 0x7fd464003520 con 0x7fd48015c9e0 2024-03-23T11:19:50.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.151+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4800cf650 msgr2=0x7fd4800cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:50.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.151+0000 7fd47f400640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4800cf650 0x7fd4800cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.151+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 >> v1:172.21.15.120:6789/0 conn(0x7fd48015cdc0 legacy=0x7fd4800d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.151+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4800d2630 con 0x7fd48015c9e0 2024-03-23T11:19:50.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fd4700019f0 con 0x7fd48015c9e0 2024-03-23T11:19:50.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 >> v1:172.21.15.62:6789/0 conn(0x7fd48015c9e0 legacy=0x7fd4801607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd47f400640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4640039a0 0x7fd464003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd47f400640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4800cf650 0x7fd4640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd47f400640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd464004d00 0x7fd464005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800d0020 con 0x7fd464004d00 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46400ba60 con 0x7fd4640039a0 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46400bc20 con 0x7fd4800cf650 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd485800640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd464004d00 0x7fd464005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd484e00640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4640039a0 0x7fd464003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd47fe00640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4800cf650 0x7fd4640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd485800640 1 -- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4800cf650 msgr2=0x7fd4640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd485800640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4800cf650 0x7fd4640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd485800640 1 -- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4640039a0 msgr2=0x7fd464003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:50.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd485800640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4640039a0 0x7fd464003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.152+0000 7fd485800640 1 -- 172.21.15.62:0/602969455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd464003520 con 0x7fd464004d00 2024-03-23T11:19:50.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.153+0000 7fd47fe00640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4800cf650 0x7fd4640047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:50.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.154+0000 7fd485800640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd464004d00 0x7fd464005180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd478009a90 tx=0x7fd4640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187c44072abbdb70 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:50.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.154+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47802c690 con 0x7fd464004d00 2024-03-23T11:19:50.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.154+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47802c7f0 con 0x7fd464004d00 2024-03-23T11:19:50.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.154+0000 7fd47f400640 1 -- 172.21.15.62:0/602969455 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47802cb80 con 0x7fd464004d00 2024-03-23T11:19:50.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.155+0000 7fd487800640 1 -- 172.21.15.62:0/602969455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd464004d00 msgr2=0x7fd464005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.155+0000 7fd487800640 1 --2- 172.21.15.62:0/602969455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd464004d00 0x7fd464005180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd478009a90 tx=0x7fd4640057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.155+0000 7fd485800640 1 -- 172.21.15.62:0/602969455 reap_dead start 2024-03-23T11:19:50.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.155+0000 7fd487800640 1 -- 172.21.15.62:0/602969455 shutdown_connections 2024-03-23T11:19:50.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.155+0000 7fd487800640 1 -- 172.21.15.62:0/602969455 >> 172.21.15.62:0/602969455 conn(0x7fd480071570 msgr2=0x7fd480071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:50.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.155+0000 7fd487800640 1 -- 172.21.15.62:0/602969455 shutdown_connections 2024-03-23T11:19:50.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.155+0000 7fd487800640 1 -- 172.21.15.62:0/602969455 wait complete. 2024-03-23T11:19:50.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.157+0000 7fd487800640 1 Processor -- start 2024-03-23T11:19:50.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.157+0000 7fd487800640 1 -- start start 2024-03-23T11:19:50.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.158+0000 7fd487800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd464004d00 0x7fd480152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.158+0000 7fd487800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4800c99c0 0x7fd48015f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.158+0000 7fd487800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4800c9e40 0x7fd48015fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.158+0000 7fd487800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800ca3a0 con 0x7fd4800c9e40 2024-03-23T11:19:50.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.158+0000 7fd487800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800ca680 con 0x7fd4800c99c0 2024-03-23T11:19:50.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.158+0000 7fd487800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800ca960 con 0x7fd464004d00 2024-03-23T11:19:50.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.158+0000 7fd485800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4800c9e40 0x7fd48015fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.158+0000 7fd485800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4800c9e40 0x7fd48015fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59726/0 (socket says 172.21.15.62:59726) 2024-03-23T11:19:50.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.158+0000 7fd485800640 1 -- 172.21.15.62:0/1065515675 learned_addr learned my addr 172.21.15.62:0/1065515675 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:50.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.158+0000 7fd484e00640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd464004d00 0x7fd480152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd485800640 1 -- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd464004d00 msgr2=0x7fd480152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd485800640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd464004d00 0x7fd480152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd47fe00640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4800c99c0 0x7fd48015f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd485800640 1 -- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4800c99c0 msgr2=0x7fd48015f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd485800640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4800c99c0 0x7fd48015f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd485800640 1 -- 172.21.15.62:0/1065515675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4800cac40 con 0x7fd4800c9e40 2024-03-23T11:19:50.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd484e00640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd464004d00 0x7fd480152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:50.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd485800640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4800c9e40 0x7fd48015fc50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd478037040 tx=0x7fd478004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:50.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd47cc00640 1 -- 172.21.15.62:0/1065515675 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47802c690 con 0x7fd4800c9e40 2024-03-23T11:19:50.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd47cc00640 1 -- 172.21.15.62:0/1065515675 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd478003830 con 0x7fd4800c9e40 2024-03-23T11:19:50.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd47cc00640 1 -- 172.21.15.62:0/1065515675 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd478003be0 con 0x7fd4800c9e40 2024-03-23T11:19:50.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.159+0000 7fd487800640 1 -- 172.21.15.62:0/1065515675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4800caf20 con 0x7fd4800c9e40 2024-03-23T11:19:50.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.160+0000 7fd487800640 1 -- 172.21.15.62:0/1065515675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4801f56a0 con 0x7fd4800c9e40 2024-03-23T11:19:50.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.161+0000 7fd487800640 1 -- 172.21.15.62:0/1065515675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd480066c90 con 0x7fd4800c9e40 2024-03-23T11:19:50.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.162+0000 7fd47cc00640 1 -- 172.21.15.62:0/1065515675 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd478003d40 con 0x7fd4800c9e40 2024-03-23T11:19:50.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.163+0000 7fd47cc00640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd46006a7d0 0x7fd46006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.163+0000 7fd47cc00640 1 -- 172.21.15.62:0/1065515675 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd478000e40 con 0x7fd4800c9e40 2024-03-23T11:19:50.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.163+0000 7fd484e00640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd46006a7d0 0x7fd46006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.164+0000 7fd484e00640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd46006a7d0 0x7fd46006cc90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd470001f30 tx=0x7fd470001310 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:50.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.168+0000 7fd47cc00640 1 -- 172.21.15.62:0/1065515675 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd478075ad0 con 0x7fd4800c9e40 2024-03-23T11:19:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.364+0000 7fd487800640 1 -- 172.21.15.62:0/1065515675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd48006b810 con 0x7fd4800c9e40 2024-03-23T11:19:50.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.365+0000 7fd47cc00640 1 -- 172.21.15.62:0/1065515675 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd4780790f0 con 0x7fd4800c9e40 2024-03-23T11:19:50.365 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 -- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd46006a7d0 msgr2=0x7fd46006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd46006a7d0 0x7fd46006cc90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd470001f30 tx=0x7fd470001310 comp rx=0 tx=0).stop 2024-03-23T11:19:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 -- 172.21.15.62:0/1065515675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4800c9e40 msgr2=0x7fd48015fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4800c9e40 0x7fd48015fc50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd478037040 tx=0x7fd478004730 comp rx=0 tx=0).stop 2024-03-23T11:19:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 -- 172.21.15.62:0/1065515675 shutdown_connections 2024-03-23T11:19:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd46006a7d0 0x7fd46006cc90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4800c9e40 0x7fd48015fc50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4800c99c0 0x7fd48015f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 --2- 172.21.15.62:0/1065515675 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd464004d00 0x7fd480152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 -- 172.21.15.62:0/1065515675 >> 172.21.15.62:0/1065515675 conn(0x7fd480071570 msgr2=0x7fd4800b3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 -- 172.21.15.62:0/1065515675 shutdown_connections 2024-03-23T11:19:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.367+0000 7fd487800640 1 -- 172.21.15.62:0/1065515675 wait complete. 2024-03-23T11:19:50.385 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:19:50.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.598+0000 7ff7c9400640 1 Processor -- start 2024-03-23T11:19:50.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.598+0000 7ff7c9400640 1 -- start start 2024-03-23T11:19:50.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.600+0000 7ff7c9400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff7c406c300 0x7ff7c40ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.600+0000 7ff7c9400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff7c40cafa0 0x7ff7c40cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.600+0000 7ff7c9400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7c406c6e0 0x7ff7c406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.600+0000 7ff7c9400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c40cbc60 con 0x7ff7c40cafa0 2024-03-23T11:19:50.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.600+0000 7ff7c9400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c40cbf40 con 0x7ff7c406c300 2024-03-23T11:19:50.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.600+0000 7ff7c9400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c40cc220 con 0x7ff7c406c6e0 2024-03-23T11:19:50.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.600+0000 7ff7c1e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff7c40cafa0 0x7ff7c40cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.600+0000 7ff7c1e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff7c40cafa0 0x7ff7c40cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59728/0 (socket says 172.21.15.62:59728) 2024-03-23T11:19:50.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.600+0000 7ff7c1e00640 1 -- 172.21.15.62:0/3669892663 learned_addr learned my addr 172.21.15.62:0/3669892663 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:50.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.600+0000 7ff7c2800640 1 --2- 172.21.15.62:0/3669892663 >> v2:172.21.15.120:3300/0 conn(0x7ff7c406c300 0x7ff7c40ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.601+0000 7ff7c3200640 1 --2- 172.21.15.62:0/3669892663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7c406c6e0 0x7ff7c406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.601+0000 7ff7c1e00640 1 -- 172.21.15.62:0/3669892663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7c406c6e0 msgr2=0x7ff7c406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:50.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.601+0000 7ff7c1e00640 1 --2- 172.21.15.62:0/3669892663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7c406c6e0 0x7ff7c406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.601+0000 7ff7c1e00640 1 -- 172.21.15.62:0/3669892663 >> v2:172.21.15.120:3300/0 conn(0x7ff7c406c300 msgr2=0x7ff7c40ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:50.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.601+0000 7ff7c1e00640 1 --2- 172.21.15.62:0/3669892663 >> v2:172.21.15.120:3300/0 conn(0x7ff7c406c300 0x7ff7c40ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.601+0000 7ff7c1e00640 1 -- 172.21.15.62:0/3669892663 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c40cc500 con 0x7ff7c40cafa0 2024-03-23T11:19:50.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.602+0000 7ff7c1e00640 1 --2- 172.21.15.62:0/3669892663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c40cafa0 0x7ff7c40cb430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff7c406c1b0 tx=0x7ff7b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f359764d1f696502 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:50.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.602+0000 7ff7c1400640 1 -- 172.21.15.62:0/3669892663 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0031870 con 0x7ff7c40cafa0 2024-03-23T11:19:50.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.602+0000 7ff7c1400640 1 -- 172.21.15.62:0/3669892663 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b00319d0 con 0x7ff7c40cafa0 2024-03-23T11:19:50.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.602+0000 7ff7c1400640 1 -- 172.21.15.62:0/3669892663 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0031d60 con 0x7ff7c40cafa0 2024-03-23T11:19:50.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.603+0000 7ff7c9400640 1 -- 172.21.15.62:0/3669892663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c40cafa0 msgr2=0x7ff7c40cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.603+0000 7ff7c9400640 1 --2- 172.21.15.62:0/3669892663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c40cafa0 0x7ff7c40cb430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff7c406c1b0 tx=0x7ff7b002b910 comp rx=0 tx=0).stop 2024-03-23T11:19:50.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.603+0000 7ff7c9400640 1 -- 172.21.15.62:0/3669892663 shutdown_connections 2024-03-23T11:19:50.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.603+0000 7ff7c9400640 1 --2- 172.21.15.62:0/3669892663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7c406c6e0 0x7ff7c406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.603+0000 7ff7c9400640 1 --2- 172.21.15.62:0/3669892663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c40cafa0 0x7ff7c40cb430 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.603+0000 7ff7c9400640 1 --2- 172.21.15.62:0/3669892663 >> v2:172.21.15.120:3300/0 conn(0x7ff7c406c300 0x7ff7c40ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.603+0000 7ff7c9400640 1 -- 172.21.15.62:0/3669892663 >> 172.21.15.62:0/3669892663 conn(0x7ff7c40b02e0 msgr2=0x7ff7c40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:50.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.603+0000 7ff7c9400640 1 -- 172.21.15.62:0/3669892663 shutdown_connections 2024-03-23T11:19:50.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.603+0000 7ff7c9400640 1 -- 172.21.15.62:0/3669892663 wait complete. 2024-03-23T11:19:50.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.605+0000 7ff7c9400640 1 Processor -- start 2024-03-23T11:19:50.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.605+0000 7ff7c9400640 1 -- start start 2024-03-23T11:19:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.606+0000 7ff7c9400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7c406c300 0x7ff7c4151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.606+0000 7ff7c9400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7c406c6e0 0x7ff7c414fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.606+0000 7ff7c9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c40cafa0 0x7ff7c415b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.606+0000 7ff7c9400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c415b040 con 0x7ff7c40cafa0 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.606+0000 7ff7c9400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c4151730 con 0x7ff7c406c6e0 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.606+0000 7ff7c9400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c413a700 con 0x7ff7c406c300 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.606+0000 7ff7c3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c40cafa0 0x7ff7c415b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.606+0000 7ff7c3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c40cafa0 0x7ff7c415b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59742/0 (socket says 172.21.15.62:59742) 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.606+0000 7ff7c3200640 1 -- 172.21.15.62:0/1579785512 learned_addr learned my addr 172.21.15.62:0/1579785512 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.607+0000 7ff7c1e00640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7c406c6e0 0x7ff7c414fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.607+0000 7ff7c2800640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7c406c300 0x7ff7c4151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.607+0000 7ff7c3200640 1 -- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7c406c300 msgr2=0x7ff7c4151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.607+0000 7ff7c3200640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7c406c300 0x7ff7c4151c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.607+0000 7ff7c3200640 1 -- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7c406c6e0 msgr2=0x7ff7c414fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.607+0000 7ff7c3200640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7c406c6e0 0x7ff7c414fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.607+0000 7ff7c3200640 1 -- 172.21.15.62:0/1579785512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b00094d0 con 0x7ff7c40cafa0 2024-03-23T11:19:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.607+0000 7ff7c3200640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c40cafa0 0x7ff7c415b4d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc007e40 tx=0x7ff7bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.607+0000 7ff7baa00640 1 -- 172.21.15.62:0/1579785512 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc014620 con 0x7ff7c40cafa0 2024-03-23T11:19:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.607+0000 7ff7baa00640 1 -- 172.21.15.62:0/1579785512 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7bc00ee90 con 0x7ff7c40cafa0 2024-03-23T11:19:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.607+0000 7ff7baa00640 1 -- 172.21.15.62:0/1579785512 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc0149e0 con 0x7ff7c40cafa0 2024-03-23T11:19:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.608+0000 7ff7c9400640 1 -- 172.21.15.62:0/1579785512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c413a9e0 con 0x7ff7c40cafa0 2024-03-23T11:19:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.608+0000 7ff7c9400640 1 -- 172.21.15.62:0/1579785512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c413b0e0 con 0x7ff7c40cafa0 2024-03-23T11:19:50.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.609+0000 7ff7c9400640 1 -- 172.21.15.62:0/1579785512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c415b1b0 con 0x7ff7c40cafa0 2024-03-23T11:19:50.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.610+0000 7ff7baa00640 1 -- 172.21.15.62:0/1579785512 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff7bc014b40 con 0x7ff7c40cafa0 2024-03-23T11:19:50.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.610+0000 7ff7baa00640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff7a406a7a0 0x7ff7a406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:50.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.611+0000 7ff7baa00640 1 -- 172.21.15.62:0/1579785512 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff7bc089760 con 0x7ff7c40cafa0 2024-03-23T11:19:50.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.611+0000 7ff7c2800640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff7a406a7a0 0x7ff7a406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:50.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.612+0000 7ff7c2800640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff7a406a7a0 0x7ff7a406cc60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4006fd0 tx=0x7ff7b4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:50.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.615+0000 7ff7baa00640 1 -- 172.21.15.62:0/1579785512 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff7bc054b90 con 0x7ff7c40cafa0 2024-03-23T11:19:50.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.820+0000 7ff7c9400640 1 -- 172.21.15.62:0/1579785512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff7c4066c90 con 0x7ff7c40cafa0 2024-03-23T11:19:50.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.820+0000 7ff7baa00640 1 -- 172.21.15.62:0/1579785512 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff7bc0581b0 con 0x7ff7c40cafa0 2024-03-23T11:19:50.820 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:19:50.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 -- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff7a406a7a0 msgr2=0x7ff7a406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff7a406a7a0 0x7ff7a406cc60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4006fd0 tx=0x7ff7b4008040 comp rx=0 tx=0).stop 2024-03-23T11:19:50.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 -- 172.21.15.62:0/1579785512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c40cafa0 msgr2=0x7ff7c415b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:50.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c40cafa0 0x7ff7c415b4d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc007e40 tx=0x7ff7bc00e4b0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 -- 172.21.15.62:0/1579785512 shutdown_connections 2024-03-23T11:19:50.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff7a406a7a0 0x7ff7a406cc60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c40cafa0 0x7ff7c415b4d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7c406c6e0 0x7ff7c414fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 --2- 172.21.15.62:0/1579785512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff7c406c300 0x7ff7c4151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:50.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 -- 172.21.15.62:0/1579785512 >> 172.21.15.62:0/1579785512 conn(0x7ff7c40b02e0 msgr2=0x7ff7c415a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:50.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 -- 172.21.15.62:0/1579785512 shutdown_connections 2024-03-23T11:19:50.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:50.823+0000 7ff7c9400640 1 -- 172.21.15.62:0/1579785512 wait complete. 2024-03-23T11:19:50.841 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi120.front.sepia.ceph.com... 2024-03-23T11:19:50.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:50.841 DEBUG:teuthology.orchestra.run.smithi120:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-23T11:19:50.941 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:19:50.941 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:19:50.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:50.942 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T11:19:50.996 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:19:50.996 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:51.049 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:19:51.049 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:51.105 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:51.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.324+0000 7faeede00640 1 Processor -- start 2024-03-23T11:19:51.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.324+0000 7faeede00640 1 -- start start 2024-03-23T11:19:51.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.325+0000 7faeede00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee8143dd0 0x7faee815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faeede00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faee8070090 0x7faee80704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faeede00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee8070a30 con 0x7faee80b2820 2024-03-23T11:19:51.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faeede00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee80cd140 con 0x7faee8070090 2024-03-23T11:19:51.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faeede00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee80cd3e0 con 0x7faee8143dd0 2024-03-23T11:19:51.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faee7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faee8070090 0x7faee80704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faee7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faee8070090 0x7faee80704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59752/0 (socket says 172.21.15.62:59752) 2024-03-23T11:19:51.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faee7e00640 1 -- 172.21.15.62:0/1264864779 learned_addr learned my addr 172.21.15.62:0/1264864779 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:51.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faeed400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7faee80b2820 0x7faee80b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60886/0 (socket says 172.21.15.62:60886) 2024-03-23T11:19:51.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faee7e00640 1 -- 172.21.15.62:0/1264864779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee8143dd0 msgr2=0x7faee815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:51.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faeeca00640 1 --2- 172.21.15.62:0/1264864779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee8143dd0 0x7faee815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faee7e00640 1 --2- 172.21.15.62:0/1264864779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee8143dd0 0x7faee815abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faee7e00640 1 -- 172.21.15.62:0/1264864779 >> v1:172.21.15.62:6789/0 conn(0x7faee80b2820 legacy=0x7faee80b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:51.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faee7e00640 1 -- 172.21.15.62:0/1264864779 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee80cd6c0 con 0x7faee8070090 2024-03-23T11:19:51.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.327+0000 7faeeca00640 1 --2- 172.21.15.62:0/1264864779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee8143dd0 0x7faee815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:51.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.326+0000 7faee7400640 1 -- 172.21.15.62:0/1264864779 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039711052 0 0) 0x7faee8070a30 con 0x7faee80b2820 2024-03-23T11:19:51.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.328+0000 7faee7e00640 1 --2- 172.21.15.62:0/1264864779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee8070090 0x7faee80704f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faed4009870 tx=0x7faed402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9894181c6327f5ca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.328+0000 7faee7400640 1 -- 172.21.15.62:0/1264864779 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4030840 con 0x7faee8070090 2024-03-23T11:19:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.328+0000 7faee7400640 1 -- 172.21.15.62:0/1264864779 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed40309a0 con 0x7faee8070090 2024-03-23T11:19:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.328+0000 7faee7400640 1 -- 172.21.15.62:0/1264864779 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4030d30 con 0x7faee8070090 2024-03-23T11:19:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.328+0000 7faeede00640 1 -- 172.21.15.62:0/1264864779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee8070090 msgr2=0x7faee80704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.328+0000 7faeede00640 1 --2- 172.21.15.62:0/1264864779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee8070090 0x7faee80704f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faed4009870 tx=0x7faed402b980 comp rx=0 tx=0).stop 2024-03-23T11:19:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.329+0000 7faeede00640 1 -- 172.21.15.62:0/1264864779 shutdown_connections 2024-03-23T11:19:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.329+0000 7faeede00640 1 --2- 172.21.15.62:0/1264864779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee8070090 0x7faee80704f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.329+0000 7faeede00640 1 --2- 172.21.15.62:0/1264864779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee8143dd0 0x7faee815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.329+0000 7faeede00640 1 -- 172.21.15.62:0/1264864779 >> 172.21.15.62:0/1264864779 conn(0x7faee8066050 msgr2=0x7faee8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:51.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.329+0000 7faeede00640 1 -- 172.21.15.62:0/1264864779 shutdown_connections 2024-03-23T11:19:51.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.329+0000 7faeede00640 1 -- 172.21.15.62:0/1264864779 wait complete. 2024-03-23T11:19:51.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.330+0000 7faeede00640 1 Processor -- start 2024-03-23T11:19:51.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.330+0000 7faeede00640 1 -- start start 2024-03-23T11:19:51.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.331+0000 7faeede00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee8070090 0x7faee815cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.331+0000 7faeede00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee80b2820 0x7faee81ef2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.331+0000 7faeede00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faee8143dd0 0x7faee81416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.331+0000 7faeede00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee813bc60 con 0x7faee8070090 2024-03-23T11:19:51.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.331+0000 7faeede00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee815d300 con 0x7faee8143dd0 2024-03-23T11:19:51.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.331+0000 7faeede00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee80aaa90 con 0x7faee80b2820 2024-03-23T11:19:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.332+0000 7faeeca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee8070090 0x7faee815cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.332+0000 7faeeca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee8070090 0x7faee815cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59764/0 (socket says 172.21.15.62:59764) 2024-03-23T11:19:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.332+0000 7faeeca00640 1 -- 172.21.15.62:0/175685364 learned_addr learned my addr 172.21.15.62:0/175685364 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.332+0000 7faee7e00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee80b2820 0x7faee81ef2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.332+0000 7faeed400640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faee8143dd0 0x7faee81416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.332+0000 7faeeca00640 1 -- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee80b2820 msgr2=0x7faee81ef2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.332+0000 7faeeca00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee80b2820 0x7faee81ef2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.332+0000 7faeeca00640 1 -- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faee8143dd0 msgr2=0x7faee81416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.332+0000 7faeeca00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faee8143dd0 0x7faee81416b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.332+0000 7faeeca00640 1 -- 172.21.15.62:0/175685364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed40094d0 con 0x7faee8070090 2024-03-23T11:19:51.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.332+0000 7faee7e00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee80b2820 0x7faee81ef2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:51.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.333+0000 7faeeca00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee8070090 0x7faee815cdc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7faedc00c7f0 tx=0x7faedc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:51.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.333+0000 7faee4c00640 1 -- 172.21.15.62:0/175685364 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc007820 con 0x7faee8070090 2024-03-23T11:19:51.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.333+0000 7faee4c00640 1 -- 172.21.15.62:0/175685364 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faedc007980 con 0x7faee8070090 2024-03-23T11:19:51.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.333+0000 7faeede00640 1 -- 172.21.15.62:0/175685364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee80aad70 con 0x7faee8070090 2024-03-23T11:19:51.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.333+0000 7faee4c00640 1 -- 172.21.15.62:0/175685364 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc007cf0 con 0x7faee8070090 2024-03-23T11:19:51.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.333+0000 7faeede00640 1 -- 172.21.15.62:0/175685364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee813beb0 con 0x7faee8070090 2024-03-23T11:19:51.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.334+0000 7faeede00640 1 -- 172.21.15.62:0/175685364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee80cca40 con 0x7faee8070090 2024-03-23T11:19:51.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.335+0000 7faee4c00640 1 -- 172.21.15.62:0/175685364 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7faedc0113f0 con 0x7faee8070090 2024-03-23T11:19:51.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.336+0000 7faee4c00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faec806a7a0 0x7faec806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.336+0000 7faee4c00640 1 -- 172.21.15.62:0/175685364 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faedc0890a0 con 0x7faee8070090 2024-03-23T11:19:51.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.336+0000 7faee7e00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faec806a7a0 0x7faec806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.338+0000 7faee7e00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faec806a7a0 0x7faec806cc60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faed4036000 tx=0x7faed403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:51.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.340+0000 7faee4c00640 1 -- 172.21.15.62:0/175685364 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faedc0544d0 con 0x7faee8070090 2024-03-23T11:19:51.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.537+0000 7faeede00640 1 -- 172.21.15.62:0/175685364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faee8066bb0 con 0x7faee8070090 2024-03-23T11:19:51.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.540+0000 7faee4c00640 1 -- 172.21.15.62:0/175685364 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16226 (secure 0 0 0) 0x7faedc057af0 con 0x7faee8070090 2024-03-23T11:19:51.541 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:51.541 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:19:42.867981+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:51.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 -- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faec806a7a0 msgr2=0x7faec806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:51.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faec806a7a0 0x7faec806cc60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faed4036000 tx=0x7faed403b040 comp rx=0 tx=0).stop 2024-03-23T11:19:51.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 -- 172.21.15.62:0/175685364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee8070090 msgr2=0x7faee815cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:51.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee8070090 0x7faee815cdc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7faedc00c7f0 tx=0x7faedc00ccb0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 -- 172.21.15.62:0/175685364 shutdown_connections 2024-03-23T11:19:51.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faec806a7a0 0x7faec806cc60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faee8143dd0 0x7faee81416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faee80b2820 0x7faee81ef2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 --2- 172.21.15.62:0/175685364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee8070090 0x7faee815cdc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 -- 172.21.15.62:0/175685364 >> 172.21.15.62:0/175685364 conn(0x7faee8066050 msgr2=0x7faee8070490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:51.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 -- 172.21.15.62:0/175685364 shutdown_connections 2024-03-23T11:19:51.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.543+0000 7faeede00640 1 -- 172.21.15.62:0/175685364 wait complete. 2024-03-23T11:19:51.561 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:51.561 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:51.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.784+0000 7f5a3aa00640 1 Processor -- start 2024-03-23T11:19:51.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.784+0000 7f5a3aa00640 1 -- start start 2024-03-23T11:19:51.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.786+0000 7f5a3aa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5a3415fcf0 0x7f5a340d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.786+0000 7f5a3aa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a34160420 0x7f5a34160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.786+0000 7f5a3aa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5a340d06d0 0x7f5a340d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.786+0000 7f5a3aa00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a34160dc0 con 0x7f5a340d06d0 2024-03-23T11:19:51.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.786+0000 7f5a3aa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a340d3540 con 0x7f5a3415fcf0 2024-03-23T11:19:51.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.786+0000 7f5a3aa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a340ce510 con 0x7f5a34160420 2024-03-23T11:19:51.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.786+0000 7f5a3a000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5a340d06d0 0x7f5a340d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.787+0000 7f5a3a000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5a340d06d0 0x7f5a340d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59766/0 (socket says 172.21.15.62:59766) 2024-03-23T11:19:51.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.787+0000 7f5a3a000640 1 -- 172.21.15.62:0/26789706 learned_addr learned my addr 172.21.15.62:0/26789706 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:51.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.787+0000 7f5a38c00640 1 --2- 172.21.15.62:0/26789706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a34160420 0x7f5a34160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.787+0000 7f5a39600640 1 --2- 172.21.15.62:0/26789706 >> v2:172.21.15.120:3300/0 conn(0x7f5a3415fcf0 0x7f5a340d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.787+0000 7f5a3a000640 1 -- 172.21.15.62:0/26789706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a34160420 msgr2=0x7f5a34160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:51.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.787+0000 7f5a3a000640 1 --2- 172.21.15.62:0/26789706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a34160420 0x7f5a34160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.787+0000 7f5a3a000640 1 -- 172.21.15.62:0/26789706 >> v2:172.21.15.120:3300/0 conn(0x7f5a3415fcf0 msgr2=0x7f5a340d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:51.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.787+0000 7f5a3a000640 1 --2- 172.21.15.62:0/26789706 >> v2:172.21.15.120:3300/0 conn(0x7f5a3415fcf0 0x7f5a340d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.787+0000 7f5a3a000640 1 -- 172.21.15.62:0/26789706 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a340cdc70 con 0x7f5a340d06d0 2024-03-23T11:19:51.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.788+0000 7f5a3a000640 1 --2- 172.21.15.62:0/26789706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a340d06d0 0x7f5a340d0b20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5a34066480 tx=0x7f5a2c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f23250ddc213e6a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:51.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.788+0000 7f5a33e00640 1 -- 172.21.15.62:0/26789706 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0317e0 con 0x7f5a340d06d0 2024-03-23T11:19:51.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.789+0000 7f5a33e00640 1 -- 172.21.15.62:0/26789706 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c031940 con 0x7f5a340d06d0 2024-03-23T11:19:51.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.789+0000 7f5a33e00640 1 -- 172.21.15.62:0/26789706 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c031cd0 con 0x7f5a340d06d0 2024-03-23T11:19:51.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.789+0000 7f5a3aa00640 1 -- 172.21.15.62:0/26789706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a340d06d0 msgr2=0x7f5a340d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:51.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.789+0000 7f5a3aa00640 1 --2- 172.21.15.62:0/26789706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a340d06d0 0x7f5a340d0b20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5a34066480 tx=0x7f5a2c02b880 comp rx=0 tx=0).stop 2024-03-23T11:19:51.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.789+0000 7f5a3aa00640 1 -- 172.21.15.62:0/26789706 shutdown_connections 2024-03-23T11:19:51.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.789+0000 7f5a3aa00640 1 --2- 172.21.15.62:0/26789706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a340d06d0 0x7f5a340d0b20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.789+0000 7f5a3aa00640 1 --2- 172.21.15.62:0/26789706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a34160420 0x7f5a34160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.789+0000 7f5a3aa00640 1 --2- 172.21.15.62:0/26789706 >> v2:172.21.15.120:3300/0 conn(0x7f5a3415fcf0 0x7f5a340d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.789+0000 7f5a3aa00640 1 -- 172.21.15.62:0/26789706 >> 172.21.15.62:0/26789706 conn(0x7f5a34089f60 msgr2=0x7f5a340d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:51.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.789+0000 7f5a3aa00640 1 -- 172.21.15.62:0/26789706 shutdown_connections 2024-03-23T11:19:51.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.789+0000 7f5a3aa00640 1 -- 172.21.15.62:0/26789706 wait complete. 2024-03-23T11:19:51.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.791+0000 7f5a3aa00640 1 Processor -- start 2024-03-23T11:19:51.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.791+0000 7f5a3aa00640 1 -- start start 2024-03-23T11:19:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.792+0000 7f5a3aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a340d06d0 0x7f5a34159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.792+0000 7f5a3aa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a3415fcf0 0x7f5a3415a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.792+0000 7f5a3aa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a34160420 0x7f5a341f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.792+0000 7f5a3aa00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a34155ae0 con 0x7f5a340d06d0 2024-03-23T11:19:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.792+0000 7f5a3aa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a341f1e10 con 0x7f5a34160420 2024-03-23T11:19:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.792+0000 7f5a3aa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a34154bc0 con 0x7f5a3415fcf0 2024-03-23T11:19:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.792+0000 7f5a39600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a340d06d0 0x7f5a34159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.792+0000 7f5a39600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a340d06d0 0x7f5a34159280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59776/0 (socket says 172.21.15.62:59776) 2024-03-23T11:19:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.793+0000 7f5a39600640 1 -- 172.21.15.62:0/3716714655 learned_addr learned my addr 172.21.15.62:0/3716714655 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.793+0000 7f5a38c00640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a3415fcf0 0x7f5a3415a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.793+0000 7f5a39600640 1 -- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a3415fcf0 msgr2=0x7f5a3415a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.793+0000 7f5a3a000640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a34160420 0x7f5a341f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.793+0000 7f5a39600640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a3415fcf0 0x7f5a3415a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.793+0000 7f5a39600640 1 -- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a34160420 msgr2=0x7f5a341f18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.793+0000 7f5a39600640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a34160420 0x7f5a341f18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.793+0000 7f5a39600640 1 -- 172.21.15.62:0/3716714655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c0094d0 con 0x7f5a340d06d0 2024-03-23T11:19:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.793+0000 7f5a39600640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a340d06d0 0x7f5a34159280 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5a28007e90 tx=0x7f5a2800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:51.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.793+0000 7f5a31600640 1 -- 172.21.15.62:0/3716714655 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28014620 con 0x7f5a340d06d0 2024-03-23T11:19:51.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.794+0000 7f5a31600640 1 -- 172.21.15.62:0/3716714655 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a28014780 con 0x7f5a340d06d0 2024-03-23T11:19:51.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.794+0000 7f5a31600640 1 -- 172.21.15.62:0/3716714655 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28014b30 con 0x7f5a340d06d0 2024-03-23T11:19:51.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.794+0000 7f5a3aa00640 1 -- 172.21.15.62:0/3716714655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a34154e60 con 0x7f5a340d06d0 2024-03-23T11:19:51.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.794+0000 7f5a3aa00640 1 -- 172.21.15.62:0/3716714655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a34155420 con 0x7f5a340d06d0 2024-03-23T11:19:51.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.795+0000 7f5a3aa00640 1 -- 172.21.15.62:0/3716714655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a34155d50 con 0x7f5a340d06d0 2024-03-23T11:19:51.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.796+0000 7f5a31600640 1 -- 172.21.15.62:0/3716714655 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5a28014c90 con 0x7f5a340d06d0 2024-03-23T11:19:51.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.797+0000 7f5a31600640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a1406a7a0 0x7f5a1406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:51.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.797+0000 7f5a31600640 1 -- 172.21.15.62:0/3716714655 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5a280897c0 con 0x7f5a340d06d0 2024-03-23T11:19:51.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.797+0000 7f5a38c00640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a1406a7a0 0x7f5a1406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:51.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.798+0000 7f5a38c00640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a1406a7a0 0x7f5a1406cc60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5a340b7250 tx=0x7f5a20008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:51.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.801+0000 7f5a31600640 1 -- 172.21.15.62:0/3716714655 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a28054b50 con 0x7f5a340d06d0 2024-03-23T11:19:51.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.999+0000 7f5a3aa00640 1 -- 172.21.15.62:0/3716714655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a340675b0 con 0x7f5a340d06d0 2024-03-23T11:19:52.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:51.999+0000 7f5a31600640 1 -- 172.21.15.62:0/3716714655 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a28058170 con 0x7f5a340d06d0 2024-03-23T11:19:52.000 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.002+0000 7f5a3aa00640 1 -- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a1406a7a0 msgr2=0x7f5a1406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.002+0000 7f5a3aa00640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a1406a7a0 0x7f5a1406cc60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5a340b7250 tx=0x7f5a20008040 comp rx=0 tx=0).stop 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.002+0000 7f5a3aa00640 1 -- 172.21.15.62:0/3716714655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a340d06d0 msgr2=0x7f5a34159280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.002+0000 7f5a3aa00640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a340d06d0 0x7f5a34159280 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5a28007e90 tx=0x7f5a2800e4b0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.002+0000 7f5a3aa00640 1 -- 172.21.15.62:0/3716714655 shutdown_connections 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.002+0000 7f5a3aa00640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a1406a7a0 0x7f5a1406cc60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.002+0000 7f5a3aa00640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a34160420 0x7f5a341f18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.002+0000 7f5a3aa00640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a3415fcf0 0x7f5a3415a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.002+0000 7f5a3aa00640 1 --2- 172.21.15.62:0/3716714655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a340d06d0 0x7f5a34159280 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.002+0000 7f5a3aa00640 1 -- 172.21.15.62:0/3716714655 >> 172.21.15.62:0/3716714655 conn(0x7f5a34089f60 msgr2=0x7f5a340d0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.002+0000 7f5a3aa00640 1 -- 172.21.15.62:0/3716714655 shutdown_connections 2024-03-23T11:19:52.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.003+0000 7f5a3aa00640 1 -- 172.21.15.62:0/3716714655 wait complete. 2024-03-23T11:19:52.021 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:52.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.240+0000 7f3fbe400640 1 Processor -- start 2024-03-23T11:19:52.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.240+0000 7f3fbe400640 1 -- start start 2024-03-23T11:19:52.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.241+0000 7f3fbe400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fb815fcf0 0x7f3fb80d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.241+0000 7f3fbe400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3fb80b6dc0 0x7f3fb8160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.241+0000 7f3fbe400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb8160dc0 con 0x7f3fb8160420 2024-03-23T11:19:52.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.241+0000 7f3fbe400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fb80d3540 con 0x7f3fb80b6dc0 2024-03-23T11:19:52.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.241+0000 7f3fbe400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb80ce510 con 0x7f3fb815fcf0 2024-03-23T11:19:52.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.242+0000 7f3fb7e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3fb8160420 0x7f3fb80d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60890/0 (socket says 172.21.15.62:60890) 2024-03-23T11:19:52.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.242+0000 7f3fb7e00640 1 -- 172.21.15.62:0/3990039322 learned_addr learned my addr 172.21.15.62:0/3990039322 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:52.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.242+0000 7f3fbd000640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fb815fcf0 0x7f3fb80d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.242+0000 7f3fbda00640 1 --2- 172.21.15.62:0/3990039322 >> v2:172.21.15.120:3300/0 conn(0x7f3fb80b6dc0 0x7f3fb8160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.242+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292217143 0 0) 0x7f3fb8160dc0 con 0x7f3fb8160420 2024-03-23T11:19:52.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.242+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f9c003520 con 0x7f3fb8160420 2024-03-23T11:19:52.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4151358516 0 0) 0x7f3f9c003520 con 0x7f3fb8160420 2024-03-23T11:19:52.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fb815fcf0 msgr2=0x7f3fb80d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:52.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fb815fcf0 0x7f3fb80d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 >> v2:172.21.15.120:3300/0 conn(0x7f3fb80b6dc0 msgr2=0x7f3fb8160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:52.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 --2- 172.21.15.62:0/3990039322 >> v2:172.21.15.120:3300/0 conn(0x7f3fb80b6dc0 0x7f3fb8160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb80cdc70 con 0x7f3fb8160420 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f3fa80019f0 con 0x7f3fb8160420 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 >> v1:172.21.15.62:6789/0 conn(0x7f3fb8160420 legacy=0x7f3fb80d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f9c003970 0x7f3f9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9c0047c0 0x7f3f9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f9c005180 0x7f3f9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb80d3540 con 0x7f3f9c005180 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb80ce510 con 0x7f3f9c0047c0 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c00bfa0 con 0x7f3f9c003970 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fbd000640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fb815fcf0 0x7f3fb80d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.243+0000 7f3fbda00640 1 --2- 172.21.15.62:0/3990039322 >> v2:172.21.15.120:3300/0 conn(0x7f3fb80b6dc0 0x7f3fb8160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.244+0000 7f3fbda00640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f9c005180 0x7f3f9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.244+0000 7f3fb7e00640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9c0047c0 0x7f3f9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.244+0000 7f3fbda00640 1 -- 172.21.15.62:0/3990039322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f9c003970 msgr2=0x7f3f9c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:52.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.244+0000 7f3fbda00640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f9c003970 0x7f3f9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.244+0000 7f3fbda00640 1 -- 172.21.15.62:0/3990039322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9c0047c0 msgr2=0x7f3f9c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:52.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.244+0000 7f3fbda00640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f9c0047c0 0x7f3f9c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.244+0000 7f3fbda00640 1 -- 172.21.15.62:0/3990039322 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c003520 con 0x7f3f9c005180 2024-03-23T11:19:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.245+0000 7f3fbda00640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f9c005180 0x7f3f9c00b980 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009c70 tx=0x7f3f9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa49a2b607a3bc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.245+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0004020 con 0x7f3f9c005180 2024-03-23T11:19:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.245+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb0004180 con 0x7f3f9c005180 2024-03-23T11:19:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.245+0000 7f3fb7400640 1 -- 172.21.15.62:0/3990039322 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb002c960 con 0x7f3f9c005180 2024-03-23T11:19:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.246+0000 7f3fbe400640 1 -- 172.21.15.62:0/3990039322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f9c005180 msgr2=0x7f3f9c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.246+0000 7f3fbe400640 1 --2- 172.21.15.62:0/3990039322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f9c005180 0x7f3f9c00b980 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009c70 tx=0x7f3f9c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.246+0000 7f3fbda00640 1 -- 172.21.15.62:0/3990039322 reap_dead start 2024-03-23T11:19:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.246+0000 7f3fbe400640 1 -- 172.21.15.62:0/3990039322 shutdown_connections 2024-03-23T11:19:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.246+0000 7f3fbe400640 1 -- 172.21.15.62:0/3990039322 >> 172.21.15.62:0/3990039322 conn(0x7f3fb8089f60 msgr2=0x7f3fb80d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.246+0000 7f3fbe400640 1 -- 172.21.15.62:0/3990039322 shutdown_connections 2024-03-23T11:19:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.246+0000 7f3fbe400640 1 -- 172.21.15.62:0/3990039322 wait complete. 2024-03-23T11:19:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.248+0000 7f3fbe400640 1 Processor -- start 2024-03-23T11:19:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.248+0000 7f3fbe400640 1 -- start start 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.249+0000 7f3fbe400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fb8150af0 0x7f3fb80c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.249+0000 7f3fbe400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fb80c8980 0x7f3fb80ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.249+0000 7f3fbe400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb80ca790 0x7f3fb80c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.249+0000 7f3fbe400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb80c8e00 con 0x7f3fb80ca790 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.249+0000 7f3fbe400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb80c5ed0 con 0x7f3fb8150af0 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.249+0000 7f3fbe400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb80caff0 con 0x7f3fb80c8980 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.249+0000 7f3fbda00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb80ca790 0x7f3fb80c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.249+0000 7f3fbda00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb80ca790 0x7f3fb80c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59794/0 (socket says 172.21.15.62:59794) 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.249+0000 7f3fbda00640 1 -- 172.21.15.62:0/866290751 learned_addr learned my addr 172.21.15.62:0/866290751 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fb7e00640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fb80c8980 0x7f3fb80ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fbd000640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fb8150af0 0x7f3fb80c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fbda00640 1 -- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fb80c8980 msgr2=0x7f3fb80ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fbda00640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fb80c8980 0x7f3fb80ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fbda00640 1 -- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fb8150af0 msgr2=0x7f3fb80c8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fbda00640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fb8150af0 0x7f3fb80c8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fbda00640 1 -- 172.21.15.62:0/866290751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb80c6120 con 0x7f3fb80ca790 2024-03-23T11:19:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fbda00640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb80ca790 0x7f3fb80c6c00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3fb002c510 tx=0x7f3fb0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fb4c00640 1 -- 172.21.15.62:0/866290751 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0006e10 con 0x7f3fb80ca790 2024-03-23T11:19:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fb4c00640 1 -- 172.21.15.62:0/866290751 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb0005e10 con 0x7f3fb80ca790 2024-03-23T11:19:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fb4c00640 1 -- 172.21.15.62:0/866290751 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb002cdb0 con 0x7f3fb80ca790 2024-03-23T11:19:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.250+0000 7f3fbe400640 1 -- 172.21.15.62:0/866290751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb80c6400 con 0x7f3fb80ca790 2024-03-23T11:19:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.251+0000 7f3fbe400640 1 -- 172.21.15.62:0/866290751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb80cac40 con 0x7f3fb80ca790 2024-03-23T11:19:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.252+0000 7f3fbe400640 1 -- 172.21.15.62:0/866290751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb80c6920 con 0x7f3fb80ca790 2024-03-23T11:19:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.253+0000 7f3fb4c00640 1 -- 172.21.15.62:0/866290751 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3fb0006040 con 0x7f3fb80ca790 2024-03-23T11:19:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.254+0000 7f3fb4c00640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3f9406a7a0 0x7f3f9406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.254+0000 7f3fb4c00640 1 -- 172.21.15.62:0/866290751 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3fb00aaed0 con 0x7f3fb80ca790 2024-03-23T11:19:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.255+0000 7f3fbd000640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3f9406a7a0 0x7f3f9406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.256+0000 7f3fbd000640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3f9406a7a0 0x7f3f9406cc60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0004790 tx=0x7f3fa0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.259+0000 7f3fb4c00640 1 -- 172.21.15.62:0/866290751 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3fb80c6920 con 0x7f3fb80ca790 2024-03-23T11:19:52.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.465+0000 7f3fbe400640 1 -- 172.21.15.62:0/866290751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3fb80c6920 con 0x7f3fb80ca790 2024-03-23T11:19:52.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.466+0000 7f3fb4c00640 1 -- 172.21.15.62:0/866290751 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3fb80c6920 con 0x7f3fb80ca790 2024-03-23T11:19:52.466 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:52.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 -- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3f9406a7a0 msgr2=0x7f3f9406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3f9406a7a0 0x7f3f9406cc60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0004790 tx=0x7f3fa0008040 comp rx=0 tx=0).stop 2024-03-23T11:19:52.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 -- 172.21.15.62:0/866290751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb80ca790 msgr2=0x7f3fb80c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb80ca790 0x7f3fb80c6c00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3fb002c510 tx=0x7f3fb0004730 comp rx=0 tx=0).stop 2024-03-23T11:19:52.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 -- 172.21.15.62:0/866290751 shutdown_connections 2024-03-23T11:19:52.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3f9406a7a0 0x7f3f9406cc60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb80ca790 0x7f3fb80c6c00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fb80c8980 0x7f3fb80ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 --2- 172.21.15.62:0/866290751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fb8150af0 0x7f3fb80c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 -- 172.21.15.62:0/866290751 >> 172.21.15.62:0/866290751 conn(0x7f3fb8089f60 msgr2=0x7f3fb80d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:52.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 -- 172.21.15.62:0/866290751 shutdown_connections 2024-03-23T11:19:52.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.469+0000 7f3fbe400640 1 -- 172.21.15.62:0/866290751 wait complete. 2024-03-23T11:19:52.488 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:52.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.712+0000 7fd0bce00640 1 Processor -- start 2024-03-23T11:19:52.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.712+0000 7fd0bce00640 1 -- start start 2024-03-23T11:19:52.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.714+0000 7fd0bce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b815c830 0x7fd0b80ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.714+0000 7fd0bce00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b81608d0 con 0x7fd0b8160310 2024-03-23T11:19:52.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.714+0000 7fd0bce00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b8160bb0 con 0x7fd0b80ce250 2024-03-23T11:19:52.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.714+0000 7fd0bce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80addb0 con 0x7fd0b815c830 2024-03-23T11:19:52.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.714+0000 7fd0aea00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd0b8160310 0x7fd0b80ab980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60904/0 (socket says 172.21.15.62:60904) 2024-03-23T11:19:52.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.714+0000 7fd0aea00640 1 -- 172.21.15.62:0/2766773794 learned_addr learned my addr 172.21.15.62:0/2766773794 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:52.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.714+0000 7fd0b7400640 1 --2- 172.21.15.62:0/2766773794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b815c830 0x7fd0b80ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.715+0000 7fd0b6a00640 1 -- 172.21.15.62:0/2766773794 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493305695 0 0) 0x7fd0b81608d0 con 0x7fd0b8160310 2024-03-23T11:19:52.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.715+0000 7fd0b6a00640 1 -- 172.21.15.62:0/2766773794 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd09c003520 con 0x7fd0b8160310 2024-03-23T11:19:52.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.715+0000 7fd0b7400640 1 -- 172.21.15.62:0/2766773794 >> v1:172.21.15.120:6789/0 conn(0x7fd0b80ce250 legacy=0x7fd0b815d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.715+0000 7fd0b7400640 1 -- 172.21.15.62:0/2766773794 >> v1:172.21.15.62:6789/0 conn(0x7fd0b8160310 legacy=0x7fd0b80ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.715+0000 7fd0b6a00640 1 -- 172.21.15.62:0/2766773794 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 681570018 0 0) 0x7fd09c003520 con 0x7fd0b8160310 2024-03-23T11:19:52.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.715+0000 7fd0b7400640 1 -- 172.21.15.62:0/2766773794 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b8160e90 con 0x7fd0b815c830 2024-03-23T11:19:52.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.717+0000 7fd0b7400640 1 --2- 172.21.15.62:0/2766773794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b815c830 0x7fd0b80ad310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009dc0 tx=0x7fd0a802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5dc09f00ab366b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:52.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.717+0000 7fd0b6a00640 1 -- 172.21.15.62:0/2766773794 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0001670 con 0x7fd0b815c830 2024-03-23T11:19:52.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.717+0000 7fd0b6a00640 1 -- 172.21.15.62:0/2766773794 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a802c910 con 0x7fd0b815c830 2024-03-23T11:19:52.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.717+0000 7fd0b6a00640 1 -- 172.21.15.62:0/2766773794 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a802cc80 con 0x7fd0b815c830 2024-03-23T11:19:52.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.718+0000 7fd0bce00640 1 -- 172.21.15.62:0/2766773794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b815c830 msgr2=0x7fd0b80ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.718+0000 7fd0bce00640 1 --2- 172.21.15.62:0/2766773794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b815c830 0x7fd0b80ad310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009dc0 tx=0x7fd0a802bde0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.718+0000 7fd0bce00640 1 -- 172.21.15.62:0/2766773794 shutdown_connections 2024-03-23T11:19:52.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.718+0000 7fd0bce00640 1 --2- 172.21.15.62:0/2766773794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b815c830 0x7fd0b80ad310 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.718+0000 7fd0bce00640 1 -- 172.21.15.62:0/2766773794 >> 172.21.15.62:0/2766773794 conn(0x7fd0b8066050 msgr2=0x7fd0b8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:52.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.718+0000 7fd0bce00640 1 -- 172.21.15.62:0/2766773794 shutdown_connections 2024-03-23T11:19:52.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.718+0000 7fd0bce00640 1 -- 172.21.15.62:0/2766773794 wait complete. 2024-03-23T11:19:52.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.719+0000 7fd0bce00640 1 Processor -- start 2024-03-23T11:19:52.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.720+0000 7fd0bce00640 1 -- start start 2024-03-23T11:19:52.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.720+0000 7fd0bce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b80ce250 0x7fd0b80c3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0bce00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0b815c830 0x7fd0b80b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0bce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8160310 0x7fd0b81f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0bce00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b8156ff0 con 0x7fd0b8160310 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0bce00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b81572d0 con 0x7fd0b815c830 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0bce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b81575b0 con 0x7fd0b80ce250 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0b7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8160310 0x7fd0b81f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0b7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8160310 0x7fd0b81f33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59796/0 (socket says 172.21.15.62:59796) 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0b7e00640 1 -- 172.21.15.62:0/3317720173 learned_addr learned my addr 172.21.15.62:0/3317720173 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0b7400640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b80ce250 0x7fd0b80c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0aea00640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0b815c830 0x7fd0b80b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0b7e00640 1 -- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b80ce250 msgr2=0x7fd0b80c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0b7e00640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b80ce250 0x7fd0b80c3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0b7e00640 1 -- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0b815c830 msgr2=0x7fd0b80b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0b7e00640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0b815c830 0x7fd0b80b21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.721+0000 7fd0b7e00640 1 -- 172.21.15.62:0/3317720173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a8009a70 con 0x7fd0b8160310 2024-03-23T11:19:52.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.722+0000 7fd0b7e00640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8160310 0x7fd0b81f33c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd0b000c390 tx=0x7fd0b000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:52.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.722+0000 7fd0afe00640 1 -- 172.21.15.62:0/3317720173 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b00052d0 con 0x7fd0b8160310 2024-03-23T11:19:52.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.722+0000 7fd0afe00640 1 -- 172.21.15.62:0/3317720173 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b0005430 con 0x7fd0b8160310 2024-03-23T11:19:52.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.722+0000 7fd0afe00640 1 -- 172.21.15.62:0/3317720173 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b0003810 con 0x7fd0b8160310 2024-03-23T11:19:52.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.722+0000 7fd0bce00640 1 -- 172.21.15.62:0/3317720173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b8157890 con 0x7fd0b8160310 2024-03-23T11:19:52.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.722+0000 7fd0bce00640 1 -- 172.21.15.62:0/3317720173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b80c3840 con 0x7fd0b8160310 2024-03-23T11:19:52.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.723+0000 7fd0bce00640 1 -- 172.21.15.62:0/3317720173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b8157db0 con 0x7fd0b8160310 2024-03-23T11:19:52.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.724+0000 7fd0afe00640 1 -- 172.21.15.62:0/3317720173 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd0b0003970 con 0x7fd0b8160310 2024-03-23T11:19:52.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.725+0000 7fd0afe00640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd09406a7a0 0x7fd09406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:52.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.725+0000 7fd0afe00640 1 -- 172.21.15.62:0/3317720173 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd0b00883a0 con 0x7fd0b8160310 2024-03-23T11:19:52.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.726+0000 7fd0b7400640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd09406a7a0 0x7fd09406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:52.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.727+0000 7fd0b7400640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd09406a7a0 0x7fd09406cc60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8036000 tx=0x7fd0a8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:52.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.730+0000 7fd0afe00640 1 -- 172.21.15.62:0/3317720173 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0b8157db0 con 0x7fd0b8160310 2024-03-23T11:19:52.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.933+0000 7fd0bce00640 1 -- 172.21.15.62:0/3317720173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd0b8157db0 con 0x7fd0b8160310 2024-03-23T11:19:52.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.933+0000 7fd0afe00640 1 -- 172.21.15.62:0/3317720173 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd0b8157db0 con 0x7fd0b8160310 2024-03-23T11:19:52.934 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.936+0000 7fd0bce00640 1 -- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd09406a7a0 msgr2=0x7fd09406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.936+0000 7fd0bce00640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd09406a7a0 0x7fd09406cc60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8036000 tx=0x7fd0a8030000 comp rx=0 tx=0).stop 2024-03-23T11:19:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.936+0000 7fd0bce00640 1 -- 172.21.15.62:0/3317720173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8160310 msgr2=0x7fd0b81f33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.937+0000 7fd0bce00640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8160310 0x7fd0b81f33c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd0b000c390 tx=0x7fd0b000c850 comp rx=0 tx=0).stop 2024-03-23T11:19:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.937+0000 7fd0bce00640 1 -- 172.21.15.62:0/3317720173 shutdown_connections 2024-03-23T11:19:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.937+0000 7fd0bce00640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd09406a7a0 0x7fd09406cc60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.937+0000 7fd0bce00640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8160310 0x7fd0b81f33c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.937+0000 7fd0bce00640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0b815c830 0x7fd0b80b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.937+0000 7fd0bce00640 1 --2- 172.21.15.62:0/3317720173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0b80ce250 0x7fd0b80c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.937+0000 7fd0bce00640 1 -- 172.21.15.62:0/3317720173 >> 172.21.15.62:0/3317720173 conn(0x7fd0b8066050 msgr2=0x7fd0b80aff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.937+0000 7fd0bce00640 1 -- 172.21.15.62:0/3317720173 shutdown_connections 2024-03-23T11:19:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:52.937+0000 7fd0bce00640 1 -- 172.21.15.62:0/3317720173 wait complete. 2024-03-23T11:19:52.957 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:19:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.176+0000 7f95a2a00640 1 Processor -- start 2024-03-23T11:19:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.176+0000 7f95a2a00640 1 -- start start 2024-03-23T11:19:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.178+0000 7f95a2a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f959c0cf650 0x7f959c0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.178+0000 7f95a2a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959c06bfc0 con 0x7f959c15c9e0 2024-03-23T11:19:53.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.178+0000 7f95a2a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959c0b5260 con 0x7f959c15cdc0 2024-03-23T11:19:53.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.178+0000 7f95a2a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f959c0d0000 con 0x7f959c0cf650 2024-03-23T11:19:53.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.178+0000 7f959be00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f959c15c9e0 0x7f959c1607c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60910/0 (socket says 172.21.15.62:60910) 2024-03-23T11:19:53.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.178+0000 7f959be00640 1 -- 172.21.15.62:0/1713418526 learned_addr learned my addr 172.21.15.62:0/1713418526 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:53.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.178+0000 7f95a0a00640 1 --2- 172.21.15.62:0/1713418526 >> v2:172.21.15.62:3300/0 conn(0x7f959c0cf650 0x7f959c0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.179+0000 7f959b400640 1 -- 172.21.15.62:0/1713418526 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397199058 0 0) 0x7f959c06bfc0 con 0x7f959c15c9e0 2024-03-23T11:19:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.179+0000 7f959b400640 1 -- 172.21.15.62:0/1713418526 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9580003520 con 0x7f959c15c9e0 2024-03-23T11:19:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.179+0000 7f95a0a00640 1 -- 172.21.15.62:0/1713418526 >> v1:172.21.15.120:6789/0 conn(0x7f959c15cdc0 legacy=0x7f959c0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.179+0000 7f95a0a00640 1 -- 172.21.15.62:0/1713418526 >> v1:172.21.15.62:6789/0 conn(0x7f959c15c9e0 legacy=0x7f959c1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.179+0000 7f95a0a00640 1 -- 172.21.15.62:0/1713418526 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959c0d2630 con 0x7f959c0cf650 2024-03-23T11:19:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.181+0000 7f95a0a00640 1 --2- 172.21.15.62:0/1713418526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f959c0cf650 0x7f959c0cfac0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9594009d90 tx=0x7f959402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c6303603f95ab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.181+0000 7f959b400640 1 -- 172.21.15.62:0/1713418526 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959402cd00 con 0x7f959c0cf650 2024-03-23T11:19:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.181+0000 7f959b400640 1 -- 172.21.15.62:0/1713418526 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959402ce60 con 0x7f959c0cf650 2024-03-23T11:19:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.181+0000 7f959b400640 1 -- 172.21.15.62:0/1713418526 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9594034660 con 0x7f959c0cf650 2024-03-23T11:19:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.182+0000 7f95a2a00640 1 -- 172.21.15.62:0/1713418526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f959c0cf650 msgr2=0x7f959c0cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.182+0000 7f95a2a00640 1 --2- 172.21.15.62:0/1713418526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f959c0cf650 0x7f959c0cfac0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9594009d90 tx=0x7f959402bf50 comp rx=0 tx=0).stop 2024-03-23T11:19:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.182+0000 7f95a2a00640 1 -- 172.21.15.62:0/1713418526 shutdown_connections 2024-03-23T11:19:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.182+0000 7f95a2a00640 1 --2- 172.21.15.62:0/1713418526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f959c0cf650 0x7f959c0cfac0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.182+0000 7f95a2a00640 1 -- 172.21.15.62:0/1713418526 >> 172.21.15.62:0/1713418526 conn(0x7f959c071570 msgr2=0x7f959c071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:53.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.182+0000 7f95a2a00640 1 -- 172.21.15.62:0/1713418526 shutdown_connections 2024-03-23T11:19:53.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.182+0000 7f95a2a00640 1 -- 172.21.15.62:0/1713418526 wait complete. 2024-03-23T11:19:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.184+0000 7f95a2a00640 1 Processor -- start 2024-03-23T11:19:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.184+0000 7f95a2a00640 1 -- start start 2024-03-23T11:19:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.185+0000 7f95a2a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f959c0cf650 0x7f959c1f2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.185+0000 7f95a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f959c15c9e0 0x7f959c158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.185+0000 7f95a2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f959c15cdc0 0x7f959c0c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.185+0000 7f95a2a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959c0c7640 con 0x7f959c15cdc0 2024-03-23T11:19:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.185+0000 7f95a2a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959c1593a0 con 0x7f959c0cf650 2024-03-23T11:19:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.185+0000 7f95a2a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959c0c6870 con 0x7f959c15c9e0 2024-03-23T11:19:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.185+0000 7f959be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f959c0cf650 0x7f959c1f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.186+0000 7f95a0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f959c15cdc0 0x7f959c0c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.186+0000 7f959be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f959c0cf650 0x7f959c1f2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:41502/0 (socket says 172.21.15.62:41502) 2024-03-23T11:19:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.186+0000 7f959be00640 1 -- 172.21.15.62:0/175042207 learned_addr learned my addr 172.21.15.62:0/175042207 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:53.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.186+0000 7f9593400640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f959c15c9e0 0x7f959c158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:53.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.186+0000 7f95a0a00640 1 -- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f959c15c9e0 msgr2=0x7f959c158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.186+0000 7f95a0a00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f959c15c9e0 0x7f959c158e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.186+0000 7f95a0a00640 1 -- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f959c0cf650 msgr2=0x7f959c1f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.186+0000 7f95a0a00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f959c0cf650 0x7f959c1f2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.186+0000 7f95a0a00640 1 -- 172.21.15.62:0/175042207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9594009a70 con 0x7f959c15cdc0 2024-03-23T11:19:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.186+0000 7f9593400640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f959c15c9e0 0x7f959c158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.186+0000 7f95a0a00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f959c15cdc0 0x7f959c0c5f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9594033000 tx=0x7f9594004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.187+0000 7f9598c00640 1 -- 172.21.15.62:0/175042207 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959403d600 con 0x7f959c15cdc0 2024-03-23T11:19:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.187+0000 7f9598c00640 1 -- 172.21.15.62:0/175042207 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959403d7a0 con 0x7f959c15cdc0 2024-03-23T11:19:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.187+0000 7f9598c00640 1 -- 172.21.15.62:0/175042207 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959403db50 con 0x7f959c15cdc0 2024-03-23T11:19:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.187+0000 7f95a2a00640 1 -- 172.21.15.62:0/175042207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f959c0c6ca0 con 0x7f959c15cdc0 2024-03-23T11:19:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.187+0000 7f95a2a00640 1 -- 172.21.15.62:0/175042207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959c0c7260 con 0x7f959c15cdc0 2024-03-23T11:19:53.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.188+0000 7f95a2a00640 1 -- 172.21.15.62:0/175042207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f959c066c90 con 0x7f959c15cdc0 2024-03-23T11:19:53.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.189+0000 7f9598c00640 1 -- 172.21.15.62:0/175042207 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f959403b040 con 0x7f959c15cdc0 2024-03-23T11:19:53.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.190+0000 7f9598c00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f957806a7a0 0x7f957806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:53.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.191+0000 7f9598c00640 1 -- 172.21.15.62:0/175042207 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f95940aaf00 con 0x7f959c15cdc0 2024-03-23T11:19:53.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.191+0000 7f959be00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f957806a7a0 0x7f957806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:53.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.192+0000 7f959be00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f957806a7a0 0x7f957806cc60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f958c0011e0 tx=0x7f958c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:53.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.195+0000 7f9598c00640 1 -- 172.21.15.62:0/175042207 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9594076400 con 0x7f959c15cdc0 2024-03-23T11:19:53.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.410+0000 7f95a2a00640 1 -- 172.21.15.62:0/175042207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f959c0b4b70 con 0x7f959c15cdc0 2024-03-23T11:19:53.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.411+0000 7f9598c00640 1 -- 172.21.15.62:0/175042207 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f959404d080 con 0x7f959c15cdc0 2024-03-23T11:19:53.411 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:19:53.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 -- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f957806a7a0 msgr2=0x7f957806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f957806a7a0 0x7f957806cc60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f958c0011e0 tx=0x7f958c009040 comp rx=0 tx=0).stop 2024-03-23T11:19:53.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 -- 172.21.15.62:0/175042207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f959c15cdc0 msgr2=0x7f959c0c5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f959c15cdc0 0x7f959c0c5f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9594033000 tx=0x7f9594004a40 comp rx=0 tx=0).stop 2024-03-23T11:19:53.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 -- 172.21.15.62:0/175042207 shutdown_connections 2024-03-23T11:19:53.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f957806a7a0 0x7f957806cc60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f959c15cdc0 0x7f959c0c5f00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f959c15c9e0 0x7f959c158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 --2- 172.21.15.62:0/175042207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f959c0cf650 0x7f959c1f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 -- 172.21.15.62:0/175042207 >> 172.21.15.62:0/175042207 conn(0x7f959c071570 msgr2=0x7f959c0c91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:53.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 -- 172.21.15.62:0/175042207 shutdown_connections 2024-03-23T11:19:53.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.414+0000 7f95a2a00640 1 -- 172.21.15.62:0/175042207 wait complete. 2024-03-23T11:19:53.437 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:19:53.438 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:53.493 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.716+0000 7f9f25a00640 1 Processor -- start 2024-03-23T11:19:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.716+0000 7f9f25a00640 1 -- start start 2024-03-23T11:19:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.718+0000 7f9f25a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9f20160310 0x7f9f200ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.718+0000 7f9f25a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f200ce250 0x7f9f200ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.718+0000 7f9f25a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f201608d0 con 0x7f9f2015c830 2024-03-23T11:19:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.718+0000 7f9f25a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f200addb0 con 0x7f9f20160310 2024-03-23T11:19:53.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.718+0000 7f9f25a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2015cc10 con 0x7f9f200ce250 2024-03-23T11:19:53.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.719+0000 7f9f25000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f200ce250 0x7f9f200ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:53.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.719+0000 7f9f17400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9f20160310 0x7f9f200ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:53.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.719+0000 7f9f1fe00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f9f2015c830 0x7f9f200ab980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:43098/0 (socket says 172.21.15.62:43098) 2024-03-23T11:19:53.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.719+0000 7f9f1fe00640 1 -- 172.21.15.62:0/3649713745 learned_addr learned my addr 172.21.15.62:0/3649713745 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:53.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.719+0000 7f9f17400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9f20160310 0x7f9f200ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:41508/0 (socket says 172.21.15.62:41508) 2024-03-23T11:19:53.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.719+0000 7f9f25000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f200ce250 0x7f9f200ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:39184/0 (socket says 172.21.15.62:39184) 2024-03-23T11:19:53.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.720+0000 7f9f25000640 1 -- 172.21.15.62:0/3649713745 >> v2:172.21.15.120:3300/0 conn(0x7f9f20160310 msgr2=0x7f9f200ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:53.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.720+0000 7f9f25000640 1 --2- 172.21.15.62:0/3649713745 >> v2:172.21.15.120:3300/0 conn(0x7f9f20160310 0x7f9f200ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.720+0000 7f9f25000640 1 -- 172.21.15.62:0/3649713745 >> v1:172.21.15.120:6789/0 conn(0x7f9f2015c830 legacy=0x7f9f200ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.720+0000 7f9f25000640 1 -- 172.21.15.62:0/3649713745 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f20160bb0 con 0x7f9f200ce250 2024-03-23T11:19:53.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.721+0000 7f9f25000640 1 --2- 172.21.15.62:0/3649713745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f200ce250 0x7f9f200ce6e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009d90 tx=0x7f9f1802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71fa18bbecf51780 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:53.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.722+0000 7f9f1f400640 1 -- 172.21.15.62:0/3649713745 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1802cc60 con 0x7f9f200ce250 2024-03-23T11:19:53.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.722+0000 7f9f1f400640 1 -- 172.21.15.62:0/3649713745 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1802cdc0 con 0x7f9f200ce250 2024-03-23T11:19:53.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.722+0000 7f9f1f400640 1 -- 172.21.15.62:0/3649713745 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18034660 con 0x7f9f200ce250 2024-03-23T11:19:53.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.722+0000 7f9f25a00640 1 -- 172.21.15.62:0/3649713745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f200ce250 msgr2=0x7f9f200ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.722+0000 7f9f25a00640 1 --2- 172.21.15.62:0/3649713745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f200ce250 0x7f9f200ce6e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009d90 tx=0x7f9f1802beb0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.723+0000 7f9f25a00640 1 -- 172.21.15.62:0/3649713745 shutdown_connections 2024-03-23T11:19:53.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.723+0000 7f9f25a00640 1 --2- 172.21.15.62:0/3649713745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f200ce250 0x7f9f200ce6e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.723+0000 7f9f25a00640 1 --2- 172.21.15.62:0/3649713745 >> v2:172.21.15.120:3300/0 conn(0x7f9f20160310 0x7f9f200ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.723+0000 7f9f25a00640 1 -- 172.21.15.62:0/3649713745 >> 172.21.15.62:0/3649713745 conn(0x7f9f20066050 msgr2=0x7f9f20069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:53.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.723+0000 7f9f25a00640 1 -- 172.21.15.62:0/3649713745 shutdown_connections 2024-03-23T11:19:53.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.723+0000 7f9f25a00640 1 -- 172.21.15.62:0/3649713745 wait complete. 2024-03-23T11:19:53.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.725+0000 7f9f25a00640 1 Processor -- start 2024-03-23T11:19:53.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.725+0000 7f9f25a00640 1 -- start start 2024-03-23T11:19:53.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.726+0000 7f9f25a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f200ce250 0x7f9f20155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:53.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f2015c830 0x7f9f20145b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:53.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f20160310 0x7f9f200cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:53.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2015dad0 con 0x7f9f20160310 2024-03-23T11:19:53.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201556d0 con 0x7f9f200ce250 2024-03-23T11:19:53.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f200cbf10 con 0x7f9f2015c830 2024-03-23T11:19:53.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f20160310 0x7f9f200cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:53.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f20160310 0x7f9f200cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59818/0 (socket says 172.21.15.62:59818) 2024-03-23T11:19:53.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25000640 1 -- 172.21.15.62:0/403608927 learned_addr learned my addr 172.21.15.62:0/403608927 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:53.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f1fe00640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f200ce250 0x7f9f20155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:53.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f17400640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f2015c830 0x7f9f20145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:53.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25000640 1 -- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f2015c830 msgr2=0x7f9f20145b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25000640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f2015c830 0x7f9f20145b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25000640 1 -- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f200ce250 msgr2=0x7f9f20155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.727+0000 7f9f25000640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f200ce250 0x7f9f20155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.728+0000 7f9f25000640 1 -- 172.21.15.62:0/403608927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f18009a70 con 0x7f9f20160310 2024-03-23T11:19:53.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.728+0000 7f9f25000640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f20160310 0x7f9f200cb400 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009eb0 tx=0x7f9f18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:53.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.728+0000 7f9f1cc00640 1 -- 172.21.15.62:0/403608927 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18040650 con 0x7f9f20160310 2024-03-23T11:19:53.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.728+0000 7f9f1cc00640 1 -- 172.21.15.62:0/403608927 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f180407b0 con 0x7f9f20160310 2024-03-23T11:19:53.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.728+0000 7f9f1cc00640 1 -- 172.21.15.62:0/403608927 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18040b60 con 0x7f9f20160310 2024-03-23T11:19:53.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.728+0000 7f9f25a00640 1 -- 172.21.15.62:0/403608927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2015d1a0 con 0x7f9f20160310 2024-03-23T11:19:53.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.728+0000 7f9f25a00640 1 -- 172.21.15.62:0/403608927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2015d760 con 0x7f9f20160310 2024-03-23T11:19:53.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.730+0000 7f9f25a00640 1 -- 172.21.15.62:0/403608927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2015dc40 con 0x7f9f20160310 2024-03-23T11:19:53.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.730+0000 7f9f1cc00640 1 -- 172.21.15.62:0/403608927 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9f18040cc0 con 0x7f9f20160310 2024-03-23T11:19:53.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.731+0000 7f9f1cc00640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9efc06a7a0 0x7f9efc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:53.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.732+0000 7f9f1cc00640 1 -- 172.21.15.62:0/403608927 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9f180abc50 con 0x7f9f20160310 2024-03-23T11:19:53.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.732+0000 7f9f1fe00640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9efc06a7a0 0x7f9efc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:53.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.733+0000 7f9f1fe00640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9efc06a7a0 0x7f9efc06cc60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9f10005880 tx=0x7f9f10009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:53.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.736+0000 7f9f1cc00640 1 -- 172.21.15.62:0/403608927 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9f18077070 con 0x7f9f20160310 2024-03-23T11:19:53.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.934+0000 7f9f25a00640 1 -- 172.21.15.62:0/403608927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f20066bb0 con 0x7f9f20160310 2024-03-23T11:19:53.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.936+0000 7f9f1cc00640 1 -- 172.21.15.62:0/403608927 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16226 (secure 0 0 0) 0x7f9f18009450 con 0x7f9f20160310 2024-03-23T11:19:53.937 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:53.938 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:19:42.867981+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:53.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.939+0000 7f9f25a00640 1 -- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9efc06a7a0 msgr2=0x7f9efc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.939+0000 7f9f25a00640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9efc06a7a0 0x7f9efc06cc60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9f10005880 tx=0x7f9f10009040 comp rx=0 tx=0).stop 2024-03-23T11:19:53.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.939+0000 7f9f25a00640 1 -- 172.21.15.62:0/403608927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f20160310 msgr2=0x7f9f200cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:53.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.940+0000 7f9f25a00640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f20160310 0x7f9f200cb400 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009eb0 tx=0x7f9f18004a40 comp rx=0 tx=0).stop 2024-03-23T11:19:53.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.940+0000 7f9f25a00640 1 -- 172.21.15.62:0/403608927 shutdown_connections 2024-03-23T11:19:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.940+0000 7f9f25a00640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9efc06a7a0 0x7f9efc06cc60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.940+0000 7f9f25a00640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f20160310 0x7f9f200cb400 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.940+0000 7f9f25a00640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f2015c830 0x7f9f20145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.940+0000 7f9f25a00640 1 --2- 172.21.15.62:0/403608927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f200ce250 0x7f9f20155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.940+0000 7f9f25a00640 1 -- 172.21.15.62:0/403608927 >> 172.21.15.62:0/403608927 conn(0x7f9f20066050 msgr2=0x7f9f200adf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.940+0000 7f9f25a00640 1 -- 172.21.15.62:0/403608927 shutdown_connections 2024-03-23T11:19:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:53.940+0000 7f9f25a00640 1 -- 172.21.15.62:0/403608927 wait complete. 2024-03-23T11:19:53.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:53.961 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:54.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.177+0000 7fa928c00640 1 Processor -- start 2024-03-23T11:19:54.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.177+0000 7fa928c00640 1 -- start start 2024-03-23T11:19:54.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.178+0000 7fa928c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa92415fcf0 0x7fa9240d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.178+0000 7fa928c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa924160420 0x7fa924160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.179+0000 7fa928c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa924160dc0 con 0x7fa9240d06d0 2024-03-23T11:19:54.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.179+0000 7fa928c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9240d3540 con 0x7fa924160420 2024-03-23T11:19:54.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.179+0000 7fa928c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240ce510 con 0x7fa92415fcf0 2024-03-23T11:19:54.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.179+0000 7fa922a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa924160420 0x7fa924160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.179+0000 7fa922a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa924160420 0x7fa924160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59820/0 (socket says 172.21.15.62:59820) 2024-03-23T11:19:54.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.179+0000 7fa922a00640 1 -- 172.21.15.62:0/3976948171 learned_addr learned my addr 172.21.15.62:0/3976948171 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:54.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.179+0000 7fa923400640 1 --2- 172.21.15.62:0/3976948171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa92415fcf0 0x7fa9240d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.180+0000 7fa922a00640 1 -- 172.21.15.62:0/3976948171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa92415fcf0 msgr2=0x7fa9240d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:54.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.180+0000 7fa922a00640 1 --2- 172.21.15.62:0/3976948171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa92415fcf0 0x7fa9240d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.180+0000 7fa922a00640 1 -- 172.21.15.62:0/3976948171 >> v1:172.21.15.120:6789/0 conn(0x7fa9240d06d0 legacy=0x7fa9240d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:54.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.180+0000 7fa922a00640 1 -- 172.21.15.62:0/3976948171 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9240cdc70 con 0x7fa924160420 2024-03-23T11:19:54.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.181+0000 7fa922a00640 1 --2- 172.21.15.62:0/3976948171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa924160420 0x7fa924160880 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009870 tx=0x7fa90c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15c4fcce1886fe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:54.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.181+0000 7fa922000640 1 -- 172.21.15.62:0/3976948171 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c0317c0 con 0x7fa924160420 2024-03-23T11:19:54.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.181+0000 7fa922000640 1 -- 172.21.15.62:0/3976948171 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c031920 con 0x7fa924160420 2024-03-23T11:19:54.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.181+0000 7fa922000640 1 -- 172.21.15.62:0/3976948171 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c031cb0 con 0x7fa924160420 2024-03-23T11:19:54.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.182+0000 7fa928c00640 1 -- 172.21.15.62:0/3976948171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa924160420 msgr2=0x7fa924160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:54.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.182+0000 7fa928c00640 1 --2- 172.21.15.62:0/3976948171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa924160420 0x7fa924160880 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009870 tx=0x7fa90c02b860 comp rx=0 tx=0).stop 2024-03-23T11:19:54.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.182+0000 7fa928c00640 1 -- 172.21.15.62:0/3976948171 shutdown_connections 2024-03-23T11:19:54.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.182+0000 7fa928c00640 1 --2- 172.21.15.62:0/3976948171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa924160420 0x7fa924160880 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.182+0000 7fa928c00640 1 --2- 172.21.15.62:0/3976948171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa92415fcf0 0x7fa9240d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.182+0000 7fa928c00640 1 -- 172.21.15.62:0/3976948171 >> 172.21.15.62:0/3976948171 conn(0x7fa924089f60 msgr2=0x7fa9240d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:54.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.182+0000 7fa928c00640 1 -- 172.21.15.62:0/3976948171 shutdown_connections 2024-03-23T11:19:54.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.182+0000 7fa928c00640 1 -- 172.21.15.62:0/3976948171 wait complete. 2024-03-23T11:19:54.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.184+0000 7fa928c00640 1 Processor -- start 2024-03-23T11:19:54.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.184+0000 7fa928c00640 1 -- start start 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.185+0000 7fa928c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9240d06d0 0x7fa9241f6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.185+0000 7fa928c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa92415fcf0 0x7fa92415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.185+0000 7fa928c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa924160420 0x7fa92414f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.185+0000 7fa928c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240ca860 con 0x7fa924160420 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.185+0000 7fa928c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240c8780 con 0x7fa92415fcf0 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.185+0000 7fa928c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9241f6580 con 0x7fa9240d06d0 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.185+0000 7fa923e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa924160420 0x7fa92414f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.185+0000 7fa923e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa924160420 0x7fa92414f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59836/0 (socket says 172.21.15.62:59836) 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.185+0000 7fa923e00640 1 -- 172.21.15.62:0/1792262223 learned_addr learned my addr 172.21.15.62:0/1792262223 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.186+0000 7fa922a00640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa92415fcf0 0x7fa92415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.186+0000 7fa923400640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9240d06d0 0x7fa9241f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.186+0000 7fa923e00640 1 -- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9240d06d0 msgr2=0x7fa9241f6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:54.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.186+0000 7fa923e00640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9240d06d0 0x7fa9241f6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.186+0000 7fa923e00640 1 -- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa92415fcf0 msgr2=0x7fa92415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:54.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.186+0000 7fa923e00640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa92415fcf0 0x7fa92415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.186+0000 7fa923e00640 1 -- 172.21.15.62:0/1792262223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c0094d0 con 0x7fa924160420 2024-03-23T11:19:54.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.186+0000 7fa923e00640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa924160420 0x7fa92414f830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00c440 tx=0x7fa91c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:54.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.187+0000 7fa91b400640 1 -- 172.21.15.62:0/1792262223 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c011790 con 0x7fa924160420 2024-03-23T11:19:54.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.187+0000 7fa91b400640 1 -- 172.21.15.62:0/1792262223 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c0118f0 con 0x7fa924160420 2024-03-23T11:19:54.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.187+0000 7fa91b400640 1 -- 172.21.15.62:0/1792262223 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c011ca0 con 0x7fa924160420 2024-03-23T11:19:54.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.187+0000 7fa928c00640 1 -- 172.21.15.62:0/1792262223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9240c9be0 con 0x7fa924160420 2024-03-23T11:19:54.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.187+0000 7fa928c00640 1 -- 172.21.15.62:0/1792262223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9240ca1a0 con 0x7fa924160420 2024-03-23T11:19:54.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.188+0000 7fa928c00640 1 -- 172.21.15.62:0/1792262223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9240ca9a0 con 0x7fa924160420 2024-03-23T11:19:54.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.189+0000 7fa91b400640 1 -- 172.21.15.62:0/1792262223 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa91c01c460 con 0x7fa924160420 2024-03-23T11:19:54.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.190+0000 7fa91b400640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa90006a7a0 0x7fa90006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.190+0000 7fa91b400640 1 -- 172.21.15.62:0/1792262223 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa91c088d30 con 0x7fa924160420 2024-03-23T11:19:54.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.190+0000 7fa923400640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa90006a7a0 0x7fa90006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.191+0000 7fa923400640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa90006a7a0 0x7fa90006cc60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa914009a60 tx=0x7fa914003990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:54.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.194+0000 7fa91b400640 1 -- 172.21.15.62:0/1792262223 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa91c0540c0 con 0x7fa924160420 2024-03-23T11:19:54.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.394+0000 7fa928c00640 1 -- 172.21.15.62:0/1792262223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa924066bb0 con 0x7fa924160420 2024-03-23T11:19:54.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.395+0000 7fa91b400640 1 -- 172.21.15.62:0/1792262223 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa91c0576e0 con 0x7fa924160420 2024-03-23T11:19:54.396 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:54.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 -- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa90006a7a0 msgr2=0x7fa90006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:54.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa90006a7a0 0x7fa90006cc60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa914009a60 tx=0x7fa914003990 comp rx=0 tx=0).stop 2024-03-23T11:19:54.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 -- 172.21.15.62:0/1792262223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa924160420 msgr2=0x7fa92414f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:54.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa924160420 0x7fa92414f830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00c440 tx=0x7fa91c00c900 comp rx=0 tx=0).stop 2024-03-23T11:19:54.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 -- 172.21.15.62:0/1792262223 shutdown_connections 2024-03-23T11:19:54.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa90006a7a0 0x7fa90006cc60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa924160420 0x7fa92414f830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa92415fcf0 0x7fa92415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 --2- 172.21.15.62:0/1792262223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9240d06d0 0x7fa9241f6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 -- 172.21.15.62:0/1792262223 >> 172.21.15.62:0/1792262223 conn(0x7fa924089f60 msgr2=0x7fa92415f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:54.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 -- 172.21.15.62:0/1792262223 shutdown_connections 2024-03-23T11:19:54.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.398+0000 7fa928c00640 1 -- 172.21.15.62:0/1792262223 wait complete. 2024-03-23T11:19:54.418 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:54.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.641+0000 7f9b27e00640 1 Processor -- start 2024-03-23T11:19:54.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.642+0000 7f9b27e00640 1 -- start start 2024-03-23T11:19:54.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.643+0000 7f9b27e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9b2815c830 0x7f9b280ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.643+0000 7f9b27e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b28160310 0x7f9b280ab980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.643+0000 7f9b27e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9b280abec0 0x7f9b280ce250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.643+0000 7f9b27e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b280addb0 con 0x7f9b280abec0 2024-03-23T11:19:54.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.643+0000 7f9b27e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b2815cc10 con 0x7f9b2815c830 2024-03-23T11:19:54.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.643+0000 7f9b27e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280ad850 con 0x7f9b28160310 2024-03-23T11:19:54.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.644+0000 7f9b27400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9b280abec0 0x7f9b280ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.644+0000 7f9b27400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9b280abec0 0x7f9b280ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59844/0 (socket says 172.21.15.62:59844) 2024-03-23T11:19:54.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.644+0000 7f9b27400640 1 -- 172.21.15.62:0/2297206859 learned_addr learned my addr 172.21.15.62:0/2297206859 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:54.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.644+0000 7f9b1fe00640 1 --2- 172.21.15.62:0/2297206859 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b28160310 0x7f9b280ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.644+0000 7f9b26a00640 1 --2- 172.21.15.62:0/2297206859 >> v2:172.21.15.120:3300/0 conn(0x7f9b2815c830 0x7f9b280ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.644+0000 7f9b27400640 1 -- 172.21.15.62:0/2297206859 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b28160310 msgr2=0x7f9b280ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:54.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.644+0000 7f9b27400640 1 --2- 172.21.15.62:0/2297206859 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b28160310 0x7f9b280ab980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.644+0000 7f9b27400640 1 -- 172.21.15.62:0/2297206859 >> v2:172.21.15.120:3300/0 conn(0x7f9b2815c830 msgr2=0x7f9b280ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:54.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.644+0000 7f9b27400640 1 --2- 172.21.15.62:0/2297206859 >> v2:172.21.15.120:3300/0 conn(0x7f9b2815c830 0x7f9b280ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.644+0000 7f9b27400640 1 -- 172.21.15.62:0/2297206859 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b281608d0 con 0x7f9b280abec0 2024-03-23T11:19:54.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.644+0000 7f9b1fe00640 1 --2- 172.21.15.62:0/2297206859 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b28160310 0x7f9b280ab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.645+0000 7f9b27400640 1 --2- 172.21.15.62:0/2297206859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b280abec0 0x7f9b280ce250 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9b280ce220 tx=0x7f9b2002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578bd8a45e4a2d73 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.646+0000 7f9b26000640 1 -- 172.21.15.62:0/2297206859 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2002ca50 con 0x7f9b280abec0 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.646+0000 7f9b26000640 1 -- 172.21.15.62:0/2297206859 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2002cbb0 con 0x7f9b280abec0 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.646+0000 7f9b26000640 1 -- 172.21.15.62:0/2297206859 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b200343f0 con 0x7f9b280abec0 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.646+0000 7f9b27e00640 1 -- 172.21.15.62:0/2297206859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b280abec0 msgr2=0x7f9b280ce250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.646+0000 7f9b27e00640 1 --2- 172.21.15.62:0/2297206859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b280abec0 0x7f9b280ce250 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9b280ce220 tx=0x7f9b2002beb0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.646+0000 7f9b27e00640 1 -- 172.21.15.62:0/2297206859 shutdown_connections 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.646+0000 7f9b27e00640 1 --2- 172.21.15.62:0/2297206859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b280abec0 0x7f9b280ce250 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.646+0000 7f9b27e00640 1 --2- 172.21.15.62:0/2297206859 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b28160310 0x7f9b280ab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.646+0000 7f9b27e00640 1 --2- 172.21.15.62:0/2297206859 >> v2:172.21.15.120:3300/0 conn(0x7f9b2815c830 0x7f9b280ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.646+0000 7f9b27e00640 1 -- 172.21.15.62:0/2297206859 >> 172.21.15.62:0/2297206859 conn(0x7f9b28066050 msgr2=0x7f9b28069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.647+0000 7f9b27e00640 1 -- 172.21.15.62:0/2297206859 shutdown_connections 2024-03-23T11:19:54.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.647+0000 7f9b27e00640 1 -- 172.21.15.62:0/2297206859 wait complete. 2024-03-23T11:19:54.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.648+0000 7f9b27e00640 1 Processor -- start 2024-03-23T11:19:54.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.648+0000 7f9b27e00640 1 -- start start 2024-03-23T11:19:54.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.649+0000 7f9b27e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b280abec0 0x7f9b280c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.649+0000 7f9b27e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b2815c830 0x7f9b280c3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.649+0000 7f9b27e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b28160310 0x7f9b2814f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.649+0000 7f9b27e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280cbd80 con 0x7f9b28160310 2024-03-23T11:19:54.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.649+0000 7f9b27e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280cc5e0 con 0x7f9b280abec0 2024-03-23T11:19:54.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.649+0000 7f9b27e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280c4220 con 0x7f9b2815c830 2024-03-23T11:19:54.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.650+0000 7f9b27400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b28160310 0x7f9b2814f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.650+0000 7f9b27400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b28160310 0x7f9b2814f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59856/0 (socket says 172.21.15.62:59856) 2024-03-23T11:19:54.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.650+0000 7f9b27400640 1 -- 172.21.15.62:0/1375484101 learned_addr learned my addr 172.21.15.62:0/1375484101 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:54.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.650+0000 7f9b27400640 1 -- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b2815c830 msgr2=0x7f9b280c3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:54.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.650+0000 7f9b27400640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b2815c830 0x7f9b280c3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.650+0000 7f9b27400640 1 -- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b280abec0 msgr2=0x7f9b280c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:54.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.650+0000 7f9b26a00640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b280abec0 0x7f9b280c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.650+0000 7f9b27400640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b280abec0 0x7f9b280c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.650+0000 7f9b27400640 1 -- 172.21.15.62:0/1375484101 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b20009b30 con 0x7f9b28160310 2024-03-23T11:19:54.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.650+0000 7f9b26a00640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b280abec0 0x7f9b280c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:54.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.650+0000 7f9b27400640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b28160310 0x7f9b2814f0f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9b20036040 tx=0x7f9b20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:54.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.651+0000 7f9b1ea00640 1 -- 172.21.15.62:0/1375484101 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2003e450 con 0x7f9b28160310 2024-03-23T11:19:54.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.651+0000 7f9b1ea00640 1 -- 172.21.15.62:0/1375484101 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2003e5b0 con 0x7f9b28160310 2024-03-23T11:19:54.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.651+0000 7f9b1ea00640 1 -- 172.21.15.62:0/1375484101 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2003e960 con 0x7f9b28160310 2024-03-23T11:19:54.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.651+0000 7f9b27e00640 1 -- 172.21.15.62:0/1375484101 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b280c44f0 con 0x7f9b28160310 2024-03-23T11:19:54.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.651+0000 7f9b27e00640 1 -- 172.21.15.62:0/1375484101 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b280cc270 con 0x7f9b28160310 2024-03-23T11:19:54.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.652+0000 7f9b27e00640 1 -- 172.21.15.62:0/1375484101 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b280cc790 con 0x7f9b28160310 2024-03-23T11:19:54.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.653+0000 7f9b1ea00640 1 -- 172.21.15.62:0/1375484101 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9b2003eac0 con 0x7f9b28160310 2024-03-23T11:19:54.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.654+0000 7f9b1ea00640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b0406a7d0 0x7f9b0406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:54.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.655+0000 7f9b1ea00640 1 -- 172.21.15.62:0/1375484101 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9b200abd00 con 0x7f9b28160310 2024-03-23T11:19:54.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.655+0000 7f9b26a00640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b0406a7d0 0x7f9b0406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:54.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.656+0000 7f9b26a00640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b0406a7d0 0x7f9b0406cc90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b280ad820 tx=0x7f9b18008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.659+0000 7f9b1ea00640 1 -- 172.21.15.62:0/1375484101 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9b280cc790 con 0x7f9b28160310 2024-03-23T11:19:54.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.856+0000 7f9b27e00640 1 -- 172.21.15.62:0/1375484101 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9b280cc750 con 0x7f9b28160310 2024-03-23T11:19:54.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.857+0000 7f9b1ea00640 1 -- 172.21.15.62:0/1375484101 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9b20043020 con 0x7f9b28160310 2024-03-23T11:19:54.858 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:54.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 -- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b0406a7d0 msgr2=0x7f9b0406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:54.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b0406a7d0 0x7f9b0406cc90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b280ad820 tx=0x7f9b18008040 comp rx=0 tx=0).stop 2024-03-23T11:19:54.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 -- 172.21.15.62:0/1375484101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b28160310 msgr2=0x7f9b2814f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:54.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b28160310 0x7f9b2814f0f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9b20036040 tx=0x7f9b20004a40 comp rx=0 tx=0).stop 2024-03-23T11:19:54.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 -- 172.21.15.62:0/1375484101 shutdown_connections 2024-03-23T11:19:54.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b0406a7d0 0x7f9b0406cc90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b28160310 0x7f9b2814f0f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b2815c830 0x7f9b280c3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 --2- 172.21.15.62:0/1375484101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b280abec0 0x7f9b280c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:54.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 -- 172.21.15.62:0/1375484101 >> 172.21.15.62:0/1375484101 conn(0x7f9b28066050 msgr2=0x7f9b280adf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:54.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 -- 172.21.15.62:0/1375484101 shutdown_connections 2024-03-23T11:19:54.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:54.860+0000 7f9b27e00640 1 -- 172.21.15.62:0/1375484101 wait complete. 2024-03-23T11:19:54.878 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:55.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.098+0000 7f6c07c00640 1 Processor -- start 2024-03-23T11:19:55.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.098+0000 7f6c07c00640 1 -- start start 2024-03-23T11:19:55.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.099+0000 7f6c07c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6c0015a740 0x7f6c0015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6c07c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6c0015ab20 0x7f6c000d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6c07c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015b580 0x7f6c0015ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6c07c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0015eb60 con 0x7f6c0015ab20 2024-03-23T11:19:55.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6c07c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0015ee40 con 0x7f6c0015a740 2024-03-23T11:19:55.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6c07c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0015f120 con 0x7f6c0015b580 2024-03-23T11:19:55.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6bf7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6c0015ab20 0x7f6c000d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6bf7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6c0015ab20 0x7f6c000d01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59866/0 (socket says 172.21.15.62:59866) 2024-03-23T11:19:55.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6bf7e00640 1 -- 172.21.15.62:0/3660521356 learned_addr learned my addr 172.21.15.62:0/3660521356 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:55.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6c05c00640 1 --2- 172.21.15.62:0/3660521356 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015b580 0x7f6c0015ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6c05200640 1 --2- 172.21.15.62:0/3660521356 >> v2:172.21.15.120:3300/0 conn(0x7f6c0015a740 0x7f6c0015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6bf7e00640 1 -- 172.21.15.62:0/3660521356 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015b580 msgr2=0x7f6c0015ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:55.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.100+0000 7f6bf7e00640 1 --2- 172.21.15.62:0/3660521356 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015b580 0x7f6c0015ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.101+0000 7f6bf7e00640 1 -- 172.21.15.62:0/3660521356 >> v2:172.21.15.120:3300/0 conn(0x7f6c0015a740 msgr2=0x7f6c0015e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:55.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.101+0000 7f6bf7e00640 1 --2- 172.21.15.62:0/3660521356 >> v2:172.21.15.120:3300/0 conn(0x7f6c0015a740 0x7f6c0015e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.101+0000 7f6bf7e00640 1 -- 172.21.15.62:0/3660521356 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0015b060 con 0x7f6c0015ab20 2024-03-23T11:19:55.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.101+0000 7f6c05c00640 1 --2- 172.21.15.62:0/3660521356 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015b580 0x7f6c0015ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:55.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.101+0000 7f6c05200640 1 --2- 172.21.15.62:0/3660521356 >> v2:172.21.15.120:3300/0 conn(0x7f6c0015a740 0x7f6c0015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:55.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.102+0000 7f6bf7e00640 1 --2- 172.21.15.62:0/3660521356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0015ab20 0x7f6c000d01b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6c0006e4d0 tx=0x7f6be802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e6acd92b235491 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:55.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.102+0000 7f6bffe00640 1 -- 172.21.15.62:0/3660521356 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8031860 con 0x7f6c0015ab20 2024-03-23T11:19:55.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.102+0000 7f6bffe00640 1 -- 172.21.15.62:0/3660521356 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be80319c0 con 0x7f6c0015ab20 2024-03-23T11:19:55.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.102+0000 7f6bffe00640 1 -- 172.21.15.62:0/3660521356 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8031d50 con 0x7f6c0015ab20 2024-03-23T11:19:55.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.103+0000 7f6c07c00640 1 -- 172.21.15.62:0/3660521356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0015ab20 msgr2=0x7f6c000d01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:55.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.103+0000 7f6c07c00640 1 --2- 172.21.15.62:0/3660521356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0015ab20 0x7f6c000d01b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6c0006e4d0 tx=0x7f6be802b900 comp rx=0 tx=0).stop 2024-03-23T11:19:55.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.103+0000 7f6c07c00640 1 -- 172.21.15.62:0/3660521356 shutdown_connections 2024-03-23T11:19:55.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.103+0000 7f6c07c00640 1 --2- 172.21.15.62:0/3660521356 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015b580 0x7f6c0015ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.103+0000 7f6c07c00640 1 --2- 172.21.15.62:0/3660521356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0015ab20 0x7f6c000d01b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.103+0000 7f6c07c00640 1 --2- 172.21.15.62:0/3660521356 >> v2:172.21.15.120:3300/0 conn(0x7f6c0015a740 0x7f6c0015e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.103+0000 7f6c07c00640 1 -- 172.21.15.62:0/3660521356 >> 172.21.15.62:0/3660521356 conn(0x7f6c00071470 msgr2=0x7f6c00071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:55.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.103+0000 7f6c07c00640 1 -- 172.21.15.62:0/3660521356 shutdown_connections 2024-03-23T11:19:55.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.103+0000 7f6c07c00640 1 -- 172.21.15.62:0/3660521356 wait complete. 2024-03-23T11:19:55.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.105+0000 7f6c07c00640 1 Processor -- start 2024-03-23T11:19:55.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.105+0000 7f6c07c00640 1 -- start start 2024-03-23T11:19:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.106+0000 7f6c07c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c0015a740 0x7f6c001eff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.106+0000 7f6c07c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015ab20 0x7f6c001458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.106+0000 7f6c07c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0015b580 0x7f6c001f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.106+0000 7f6c07c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c001f04c0 con 0x7f6c0015b580 2024-03-23T11:19:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.106+0000 7f6c07c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c00145e20 con 0x7f6c0015a740 2024-03-23T11:19:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.106+0000 7f6c07c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c001460c0 con 0x7f6c0015ab20 2024-03-23T11:19:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.106+0000 7f6c05c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0015b580 0x7f6c001f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.106+0000 7f6c05c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0015b580 0x7f6c001f2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59874/0 (socket says 172.21.15.62:59874) 2024-03-23T11:19:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.106+0000 7f6c05c00640 1 -- 172.21.15.62:0/575740577 learned_addr learned my addr 172.21.15.62:0/575740577 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.106+0000 7f6c05200640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c0015a740 0x7f6c001eff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6bf7e00640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015ab20 0x7f6c001458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6c05c00640 1 -- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015ab20 msgr2=0x7f6c001458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6c05c00640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015ab20 0x7f6c001458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6c05c00640 1 -- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c0015a740 msgr2=0x7f6c001eff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6c05c00640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c0015a740 0x7f6c001eff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6c05c00640 1 -- 172.21.15.62:0/575740577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be80094d0 con 0x7f6c0015b580 2024-03-23T11:19:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6bf7e00640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015ab20 0x7f6c001458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6c05200640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c0015a740 0x7f6c001eff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6c05c00640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0015b580 0x7f6c001f2d10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800cd50 tx=0x7f6bf800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:55.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6bfd600640 1 -- 172.21.15.62:0/575740577 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf800f680 con 0x7f6c0015b580 2024-03-23T11:19:55.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6bfd600640 1 -- 172.21.15.62:0/575740577 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf800f800 con 0x7f6c0015b580 2024-03-23T11:19:55.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6bfd600640 1 -- 172.21.15.62:0/575740577 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf800fbb0 con 0x7f6c0015b580 2024-03-23T11:19:55.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.107+0000 7f6c07c00640 1 -- 172.21.15.62:0/575740577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c001463a0 con 0x7f6c0015b580 2024-03-23T11:19:55.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.108+0000 7f6c07c00640 1 -- 172.21.15.62:0/575740577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c00146960 con 0x7f6c0015b580 2024-03-23T11:19:55.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.109+0000 7f6c07c00640 1 -- 172.21.15.62:0/575740577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c000672c0 con 0x7f6c0015b580 2024-03-23T11:19:55.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.110+0000 7f6bfd600640 1 -- 172.21.15.62:0/575740577 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6bf800b040 con 0x7f6c0015b580 2024-03-23T11:19:55.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.111+0000 7f6bfd600640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6bdc06a7a0 0x7f6bdc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.111+0000 7f6bfd600640 1 -- 172.21.15.62:0/575740577 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6bf8089890 con 0x7f6c0015b580 2024-03-23T11:19:55.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.111+0000 7f6c05200640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6bdc06a7a0 0x7f6bdc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.112+0000 7f6c05200640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6bdc06a7a0 0x7f6bdc06cc60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6bf00088b0 tx=0x7f6bf0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.115+0000 7f6bfd600640 1 -- 172.21.15.62:0/575740577 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6bf8054b90 con 0x7f6c0015b580 2024-03-23T11:19:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.315+0000 7f6c07c00640 1 -- 172.21.15.62:0/575740577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c000cfad0 con 0x7f6c0015b580 2024-03-23T11:19:55.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.316+0000 7f6bfd600640 1 -- 172.21.15.62:0/575740577 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6bf801d020 con 0x7f6c0015b580 2024-03-23T11:19:55.316 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.318+0000 7f6c07c00640 1 -- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6bdc06a7a0 msgr2=0x7f6bdc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.318+0000 7f6c07c00640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6bdc06a7a0 0x7f6bdc06cc60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6bf00088b0 tx=0x7f6bf0008040 comp rx=0 tx=0).stop 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.318+0000 7f6c07c00640 1 -- 172.21.15.62:0/575740577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0015b580 msgr2=0x7f6c001f2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.318+0000 7f6c07c00640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0015b580 0x7f6c001f2d10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800cd50 tx=0x7f6bf800a650 comp rx=0 tx=0).stop 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.318+0000 7f6c07c00640 1 -- 172.21.15.62:0/575740577 shutdown_connections 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.318+0000 7f6c07c00640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6bdc06a7a0 0x7f6bdc06cc60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.318+0000 7f6c07c00640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c0015b580 0x7f6c001f2d10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.318+0000 7f6c07c00640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6c0015ab20 0x7f6c001458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.318+0000 7f6c07c00640 1 --2- 172.21.15.62:0/575740577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c0015a740 0x7f6c001eff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.318+0000 7f6c07c00640 1 -- 172.21.15.62:0/575740577 >> 172.21.15.62:0/575740577 conn(0x7f6c00071470 msgr2=0x7f6c00072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.319+0000 7f6c07c00640 1 -- 172.21.15.62:0/575740577 shutdown_connections 2024-03-23T11:19:55.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.319+0000 7f6c07c00640 1 -- 172.21.15.62:0/575740577 wait complete. 2024-03-23T11:19:55.339 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:19:55.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.560+0000 7fd13c800640 1 Processor -- start 2024-03-23T11:19:55.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.560+0000 7fd13c800640 1 -- start start 2024-03-23T11:19:55.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.562+0000 7fd13c800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd13406c300 0x7fd1340ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.562+0000 7fd13c800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd13406c6e0 0x7fd13406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.562+0000 7fd13c800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1340cbc60 con 0x7fd1340cafa0 2024-03-23T11:19:55.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.562+0000 7fd13c800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1340cbf40 con 0x7fd13406c300 2024-03-23T11:19:55.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.562+0000 7fd13c800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1340cc220 con 0x7fd13406c6e0 2024-03-23T11:19:55.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.562+0000 7fd139400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd1340cafa0 0x7fd1340cb430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60922/0 (socket says 172.21.15.62:60922) 2024-03-23T11:19:55.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.562+0000 7fd139400640 1 -- 172.21.15.62:0/3159516021 learned_addr learned my addr 172.21.15.62:0/3159516021 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:55.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.562+0000 7fd139e00640 1 --2- 172.21.15.62:0/3159516021 >> v2:172.21.15.62:3300/0 conn(0x7fd13406c300 0x7fd1340ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.563+0000 7fd13a800640 1 --2- 172.21.15.62:0/3159516021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd13406c6e0 0x7fd13406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.563+0000 7fd138a00640 1 -- 172.21.15.62:0/3159516021 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704112503 0 0) 0x7fd1340cbc60 con 0x7fd1340cafa0 2024-03-23T11:19:55.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.563+0000 7fd138a00640 1 -- 172.21.15.62:0/3159516021 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd11c003520 con 0x7fd1340cafa0 2024-03-23T11:19:55.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.563+0000 7fd139e00640 1 -- 172.21.15.62:0/3159516021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd13406c6e0 msgr2=0x7fd13406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:55.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.563+0000 7fd139e00640 1 --2- 172.21.15.62:0/3159516021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd13406c6e0 0x7fd13406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.563+0000 7fd139e00640 1 -- 172.21.15.62:0/3159516021 >> v1:172.21.15.62:6789/0 conn(0x7fd1340cafa0 legacy=0x7fd1340cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:55.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.563+0000 7fd139e00640 1 -- 172.21.15.62:0/3159516021 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1340cc500 con 0x7fd13406c300 2024-03-23T11:19:55.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.565+0000 7fd139e00640 1 --2- 172.21.15.62:0/3159516021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd13406c300 0x7fd1340ccd00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd1200097a0 tx=0x7fd12002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42607eb790bdd23a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:55.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.566+0000 7fd138a00640 1 -- 172.21.15.62:0/3159516021 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd120031830 con 0x7fd13406c300 2024-03-23T11:19:55.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.566+0000 7fd138a00640 1 -- 172.21.15.62:0/3159516021 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd120031990 con 0x7fd13406c300 2024-03-23T11:19:55.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.566+0000 7fd138a00640 1 -- 172.21.15.62:0/3159516021 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd120031d20 con 0x7fd13406c300 2024-03-23T11:19:55.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.566+0000 7fd13c800640 1 -- 172.21.15.62:0/3159516021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd13406c300 msgr2=0x7fd1340ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:55.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.566+0000 7fd13c800640 1 --2- 172.21.15.62:0/3159516021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd13406c300 0x7fd1340ccd00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd1200097a0 tx=0x7fd12002b8d0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.566+0000 7fd13c800640 1 -- 172.21.15.62:0/3159516021 shutdown_connections 2024-03-23T11:19:55.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.566+0000 7fd13c800640 1 --2- 172.21.15.62:0/3159516021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd13406c6e0 0x7fd13406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.566+0000 7fd13c800640 1 --2- 172.21.15.62:0/3159516021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd13406c300 0x7fd1340ccd00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.566+0000 7fd13c800640 1 -- 172.21.15.62:0/3159516021 >> 172.21.15.62:0/3159516021 conn(0x7fd1340b02e0 msgr2=0x7fd1340b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:55.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.566+0000 7fd13c800640 1 -- 172.21.15.62:0/3159516021 shutdown_connections 2024-03-23T11:19:55.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.566+0000 7fd13c800640 1 -- 172.21.15.62:0/3159516021 wait complete. 2024-03-23T11:19:55.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.568+0000 7fd13c800640 1 Processor -- start 2024-03-23T11:19:55.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.568+0000 7fd13c800640 1 -- start start 2024-03-23T11:19:55.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.569+0000 7fd13c800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd13406c300 0x7fd13412ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.569+0000 7fd13c800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd13406c6e0 0x7fd1341f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.569+0000 7fd13c800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1340cafa0 0x7fd13415d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.569+0000 7fd13c800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1341f47f0 con 0x7fd13406c6e0 2024-03-23T11:19:55.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.569+0000 7fd13c800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1341f03c0 con 0x7fd13406c300 2024-03-23T11:19:55.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.569+0000 7fd13c800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13412f290 con 0x7fd1340cafa0 2024-03-23T11:19:55.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.569+0000 7fd139400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd13406c6e0 0x7fd1341f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.570+0000 7fd139400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd13406c6e0 0x7fd1341f42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59906/0 (socket says 172.21.15.62:59906) 2024-03-23T11:19:55.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.570+0000 7fd139400640 1 -- 172.21.15.62:0/3734124275 learned_addr learned my addr 172.21.15.62:0/3734124275 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.570+0000 7fd139e00640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd13406c300 0x7fd13412ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.570+0000 7fd13a800640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1340cafa0 0x7fd13415d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.570+0000 7fd139400640 1 -- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1340cafa0 msgr2=0x7fd13415d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.570+0000 7fd139400640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1340cafa0 0x7fd13415d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.570+0000 7fd139400640 1 -- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd13406c300 msgr2=0x7fd13412ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.570+0000 7fd139400640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd13406c300 0x7fd13412ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.570+0000 7fd139400640 1 -- 172.21.15.62:0/3734124275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd120009400 con 0x7fd13406c6e0 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.570+0000 7fd139e00640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd13406c300 0x7fd13412ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.570+0000 7fd139400640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd13406c6e0 0x7fd1341f42b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd12800c3c0 tx=0x7fd12800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.571+0000 7fd12e000640 1 -- 172.21.15.62:0/3734124275 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd128005ce0 con 0x7fd13406c6e0 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.571+0000 7fd12e000640 1 -- 172.21.15.62:0/3734124275 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd128005e40 con 0x7fd13406c6e0 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.571+0000 7fd12e000640 1 -- 172.21.15.62:0/3734124275 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1280072f0 con 0x7fd13406c6e0 2024-03-23T11:19:55.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.571+0000 7fd13c800640 1 -- 172.21.15.62:0/3734124275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13412f570 con 0x7fd13406c6e0 2024-03-23T11:19:55.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.571+0000 7fd13c800640 1 -- 172.21.15.62:0/3734124275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13412fb30 con 0x7fd13406c6e0 2024-03-23T11:19:55.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.572+0000 7fd13c800640 1 -- 172.21.15.62:0/3734124275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1341f0590 con 0x7fd13406c6e0 2024-03-23T11:19:55.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.573+0000 7fd12e000640 1 -- 172.21.15.62:0/3734124275 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd128007450 con 0x7fd13406c6e0 2024-03-23T11:19:55.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.574+0000 7fd12e000640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd11406a7a0 0x7fd11406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:55.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.574+0000 7fd12e000640 1 -- 172.21.15.62:0/3734124275 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd128088370 con 0x7fd13406c6e0 2024-03-23T11:19:55.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.575+0000 7fd139e00640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd11406a7a0 0x7fd11406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:55.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.576+0000 7fd139e00640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd11406a7a0 0x7fd11406cc60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd120036000 tx=0x7fd120008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:55.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.579+0000 7fd12e000640 1 -- 172.21.15.62:0/3734124275 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1341f0590 con 0x7fd13406c6e0 2024-03-23T11:19:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.792+0000 7fd13c800640 1 -- 172.21.15.62:0/3734124275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd1341f0590 con 0x7fd13406c6e0 2024-03-23T11:19:55.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.793+0000 7fd12e000640 1 -- 172.21.15.62:0/3734124275 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd1341f0590 con 0x7fd13406c6e0 2024-03-23T11:19:55.793 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:19:55.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.795+0000 7fd13c800640 1 -- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd11406a7a0 msgr2=0x7fd11406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:55.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.795+0000 7fd13c800640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd11406a7a0 0x7fd11406cc60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd120036000 tx=0x7fd120008030 comp rx=0 tx=0).stop 2024-03-23T11:19:55.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.795+0000 7fd13c800640 1 -- 172.21.15.62:0/3734124275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd13406c6e0 msgr2=0x7fd1341f42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:55.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.795+0000 7fd13c800640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd13406c6e0 0x7fd1341f42b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd12800c3c0 tx=0x7fd12800c880 comp rx=0 tx=0).stop 2024-03-23T11:19:55.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.796+0000 7fd13c800640 1 -- 172.21.15.62:0/3734124275 shutdown_connections 2024-03-23T11:19:55.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.796+0000 7fd13c800640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd11406a7a0 0x7fd11406cc60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.796+0000 7fd13c800640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd13406c6e0 0x7fd1341f42b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.796+0000 7fd13c800640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1340cafa0 0x7fd13415d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.796+0000 7fd13c800640 1 --2- 172.21.15.62:0/3734124275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd13406c300 0x7fd13412ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.796+0000 7fd13c800640 1 -- 172.21.15.62:0/3734124275 >> 172.21.15.62:0/3734124275 conn(0x7fd1340b02e0 msgr2=0x7fd1340cb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.796+0000 7fd13c800640 1 -- 172.21.15.62:0/3734124275 shutdown_connections 2024-03-23T11:19:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:55.796+0000 7fd13c800640 1 -- 172.21.15.62:0/3734124275 wait complete. 2024-03-23T11:19:55.816 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:19:55.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:55.817 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-23T11:19:55.914 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:19:55.914 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:19:55.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:19:55.914 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T11:19:55.967 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:19:56.000 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:56.058 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:19:56.058 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:56.112 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:56.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.345+0000 7f0322e00640 1 Processor -- start 2024-03-23T11:19:56.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.345+0000 7f0322e00640 1 -- start start 2024-03-23T11:19:56.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.346+0000 7f0322e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f031c15c900 0x7f031c0d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.346+0000 7f0322e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f031c15cce0 0x7f031c0b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.347+0000 7f0322e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c0d1100 con 0x7f031c0cd690 2024-03-23T11:19:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.347+0000 7f0322e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031c0d13e0 con 0x7f031c15c900 2024-03-23T11:19:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.347+0000 7f0322e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031c0d16c0 con 0x7f031c15cce0 2024-03-23T11:19:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.347+0000 7f031be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f031c15c900 0x7f031c0d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.347+0000 7f031be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f031c15c900 0x7f031c0d21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59916/0 (socket says 172.21.15.62:59916) 2024-03-23T11:19:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.347+0000 7f031be00640 1 -- 172.21.15.62:0/1821584213 learned_addr learned my addr 172.21.15.62:0/1821584213 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.347+0000 7f031be00640 1 -- 172.21.15.62:0/1821584213 >> v2:172.21.15.120:3300/0 conn(0x7f031c15cce0 msgr2=0x7f031c0b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.347+0000 7f031b400640 1 --2- 172.21.15.62:0/1821584213 >> v2:172.21.15.120:3300/0 conn(0x7f031c15cce0 0x7f031c0b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.347+0000 7f031be00640 1 --2- 172.21.15.62:0/1821584213 >> v2:172.21.15.120:3300/0 conn(0x7f031c15cce0 0x7f031c0b82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.347+0000 7f031be00640 1 -- 172.21.15.62:0/1821584213 >> v1:172.21.15.120:6789/0 conn(0x7f031c0cd690 legacy=0x7f031c0b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.347+0000 7f031be00640 1 -- 172.21.15.62:0/1821584213 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c0d19a0 con 0x7f031c15c900 2024-03-23T11:19:56.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.349+0000 7f031be00640 1 --2- 172.21.15.62:0/1821584213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f031c15c900 0x7f031c0d21a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f031c072290 tx=0x7f030c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c580398bb4865fe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:56.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.349+0000 7f031aa00640 1 -- 172.21.15.62:0/1821584213 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030c030860 con 0x7f031c15c900 2024-03-23T11:19:56.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.349+0000 7f031aa00640 1 -- 172.21.15.62:0/1821584213 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030c0309c0 con 0x7f031c15c900 2024-03-23T11:19:56.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.349+0000 7f031aa00640 1 -- 172.21.15.62:0/1821584213 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030c030d50 con 0x7f031c15c900 2024-03-23T11:19:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.350+0000 7f0322e00640 1 -- 172.21.15.62:0/1821584213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f031c15c900 msgr2=0x7f031c0d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.350+0000 7f0322e00640 1 --2- 172.21.15.62:0/1821584213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f031c15c900 0x7f031c0d21a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f031c072290 tx=0x7f030c02b9a0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.350+0000 7f0322e00640 1 -- 172.21.15.62:0/1821584213 shutdown_connections 2024-03-23T11:19:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.350+0000 7f0322e00640 1 --2- 172.21.15.62:0/1821584213 >> v2:172.21.15.120:3300/0 conn(0x7f031c15cce0 0x7f031c0b82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.350+0000 7f0322e00640 1 --2- 172.21.15.62:0/1821584213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f031c15c900 0x7f031c0d21a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.350+0000 7f0322e00640 1 -- 172.21.15.62:0/1821584213 >> 172.21.15.62:0/1821584213 conn(0x7f031c071390 msgr2=0x7f031c071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.350+0000 7f0322e00640 1 -- 172.21.15.62:0/1821584213 shutdown_connections 2024-03-23T11:19:56.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.350+0000 7f0322e00640 1 -- 172.21.15.62:0/1821584213 wait complete. 2024-03-23T11:19:56.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.352+0000 7f0322e00640 1 Processor -- start 2024-03-23T11:19:56.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.352+0000 7f0322e00640 1 -- start start 2024-03-23T11:19:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f0322e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f031c0cd690 0x7f031c1535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f0322e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f031c15c900 0x7f031c0c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f0322e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f031c15cce0 0x7f031c1f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f0322e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c154fe0 con 0x7f031c0cd690 2024-03-23T11:19:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f0322e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c0c7dd0 con 0x7f031c15cce0 2024-03-23T11:19:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f0322e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031c0c8060 con 0x7f031c15c900 2024-03-23T11:19:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f031be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f031c0cd690 0x7f031c1535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f031be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f031c0cd690 0x7f031c1535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59932/0 (socket says 172.21.15.62:59932) 2024-03-23T11:19:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f031be00640 1 -- 172.21.15.62:0/1168933599 learned_addr learned my addr 172.21.15.62:0/1168933599 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f031be00640 1 -- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f031c15c900 msgr2=0x7f031c0c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f031be00640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f031c15c900 0x7f031c0c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f031be00640 1 -- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f031c15cce0 msgr2=0x7f031c1f5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f031be00640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f031c15cce0 0x7f031c1f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.353+0000 7f031be00640 1 -- 172.21.15.62:0/1168933599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c0094d0 con 0x7f031c0cd690 2024-03-23T11:19:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.354+0000 7f031be00640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f031c0cd690 0x7f031c1535f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f030c000c00 tx=0x7f030c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.354+0000 7f0313e00640 1 -- 172.21.15.62:0/1168933599 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030c030830 con 0x7f031c0cd690 2024-03-23T11:19:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.354+0000 7f0313e00640 1 -- 172.21.15.62:0/1168933599 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030c043400 con 0x7f031c0cd690 2024-03-23T11:19:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.354+0000 7f0313e00640 1 -- 172.21.15.62:0/1168933599 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030c0437b0 con 0x7f031c0cd690 2024-03-23T11:19:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.354+0000 7f0322e00640 1 -- 172.21.15.62:0/1168933599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031c0c8340 con 0x7f031c0cd690 2024-03-23T11:19:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.354+0000 7f0322e00640 1 -- 172.21.15.62:0/1168933599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031c0c8900 con 0x7f031c0cd690 2024-03-23T11:19:56.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.356+0000 7f0322e00640 1 -- 172.21.15.62:0/1168933599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031c0672c0 con 0x7f031c0cd690 2024-03-23T11:19:56.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.356+0000 7f0313e00640 1 -- 172.21.15.62:0/1168933599 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f030c043910 con 0x7f031c0cd690 2024-03-23T11:19:56.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.357+0000 7f0313e00640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02fc06a7a0 0x7f02fc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.357+0000 7f0313e00640 1 -- 172.21.15.62:0/1168933599 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f030c03e070 con 0x7f031c0cd690 2024-03-23T11:19:56.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.358+0000 7f031b400640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02fc06a7a0 0x7f02fc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:56.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.359+0000 7f031b400640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02fc06a7a0 0x7f02fc06cc60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f031c153b30 tx=0x7f0308008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:56.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.362+0000 7f0313e00640 1 -- 172.21.15.62:0/1168933599 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f030c0df580 con 0x7f031c0cd690 2024-03-23T11:19:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.567+0000 7f0322e00640 1 -- 172.21.15.62:0/1168933599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f031c1582e0 con 0x7f031c0cd690 2024-03-23T11:19:56.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.570+0000 7f0313e00640 1 -- 172.21.15.62:0/1168933599 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16226 (secure 0 0 0) 0x7f030c077520 con 0x7f031c0cd690 2024-03-23T11:19:56.571 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:56.571 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:19:42.867981+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:56.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.573+0000 7f0322e00640 1 -- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02fc06a7a0 msgr2=0x7f02fc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:56.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.573+0000 7f0322e00640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02fc06a7a0 0x7f02fc06cc60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f031c153b30 tx=0x7f0308008040 comp rx=0 tx=0).stop 2024-03-23T11:19:56.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.573+0000 7f0322e00640 1 -- 172.21.15.62:0/1168933599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f031c0cd690 msgr2=0x7f031c1535f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:56.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.573+0000 7f0322e00640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f031c0cd690 0x7f031c1535f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f030c000c00 tx=0x7f030c004730 comp rx=0 tx=0).stop 2024-03-23T11:19:56.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.573+0000 7f0322e00640 1 -- 172.21.15.62:0/1168933599 shutdown_connections 2024-03-23T11:19:56.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.573+0000 7f0322e00640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02fc06a7a0 0x7f02fc06cc60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.573+0000 7f0322e00640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f031c15cce0 0x7f031c1f5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.573+0000 7f0322e00640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f031c15c900 0x7f031c0c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.573+0000 7f0322e00640 1 --2- 172.21.15.62:0/1168933599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f031c0cd690 0x7f031c1535f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.573+0000 7f0322e00640 1 -- 172.21.15.62:0/1168933599 >> 172.21.15.62:0/1168933599 conn(0x7f031c071390 msgr2=0x7f031c158800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:56.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.574+0000 7f0322e00640 1 -- 172.21.15.62:0/1168933599 shutdown_connections 2024-03-23T11:19:56.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.574+0000 7f0322e00640 1 -- 172.21.15.62:0/1168933599 wait complete. 2024-03-23T11:19:56.592 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:56.593 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:56.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.813+0000 7f40ff200640 1 Processor -- start 2024-03-23T11:19:56.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.813+0000 7f40ff200640 1 -- start start 2024-03-23T11:19:56.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.815+0000 7f40ff200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f40f80cafa0 0x7f40f80cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.815+0000 7f40ff200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f40f806c6e0 0x7f40f806cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.815+0000 7f40ff200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f80cbc60 con 0x7f40f806c300 2024-03-23T11:19:56.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.815+0000 7f40ff200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f80cd410 con 0x7f40f806c6e0 2024-03-23T11:19:56.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.815+0000 7f40ff200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f80cbf40 con 0x7f40f80cafa0 2024-03-23T11:19:56.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.815+0000 7f40f7e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f40f806c300 0x7f40f80ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60926/0 (socket says 172.21.15.62:60926) 2024-03-23T11:19:56.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.815+0000 7f40f7e00640 1 -- 172.21.15.62:0/632108643 learned_addr learned my addr 172.21.15.62:0/632108643 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:56.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.815+0000 7f40fd200640 1 --2- 172.21.15.62:0/632108643 >> v2:172.21.15.62:3300/0 conn(0x7f40f806c6e0 0x7f40f806cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:56.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.816+0000 7f40f7400640 1 --2- 172.21.15.62:0/632108643 >> v2:172.21.15.120:3300/0 conn(0x7f40f80cafa0 0x7f40f80cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:56.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.816+0000 7f40f6a00640 1 -- 172.21.15.62:0/632108643 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026753103 0 0) 0x7f40f80cbc60 con 0x7f40f806c300 2024-03-23T11:19:56.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.816+0000 7f40f6a00640 1 -- 172.21.15.62:0/632108643 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40dc003520 con 0x7f40f806c300 2024-03-23T11:19:56.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.816+0000 7f40fd200640 1 -- 172.21.15.62:0/632108643 >> v2:172.21.15.120:3300/0 conn(0x7f40f80cafa0 msgr2=0x7f40f80cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:56.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.816+0000 7f40fd200640 1 --2- 172.21.15.62:0/632108643 >> v2:172.21.15.120:3300/0 conn(0x7f40f80cafa0 0x7f40f80cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.816+0000 7f40fd200640 1 -- 172.21.15.62:0/632108643 >> v1:172.21.15.62:6789/0 conn(0x7f40f806c300 legacy=0x7f40f80ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:56.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.816+0000 7f40fd200640 1 -- 172.21.15.62:0/632108643 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f80cc220 con 0x7f40f806c6e0 2024-03-23T11:19:56.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.817+0000 7f40fd200640 1 --2- 172.21.15.62:0/632108643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f806c6e0 0x7f40f806cb30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f40f806e370 tx=0x7f40f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3dd3b629359473f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.817+0000 7f40f6a00640 1 -- 172.21.15.62:0/632108643 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0031830 con 0x7f40f806c6e0 2024-03-23T11:19:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.817+0000 7f40f6a00640 1 -- 172.21.15.62:0/632108643 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f0031990 con 0x7f40f806c6e0 2024-03-23T11:19:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.817+0000 7f40f6a00640 1 -- 172.21.15.62:0/632108643 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0031d20 con 0x7f40f806c6e0 2024-03-23T11:19:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.818+0000 7f40ff200640 1 -- 172.21.15.62:0/632108643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f806c6e0 msgr2=0x7f40f806cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.818+0000 7f40ff200640 1 --2- 172.21.15.62:0/632108643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f806c6e0 0x7f40f806cb30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f40f806e370 tx=0x7f40f002b8d0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.818+0000 7f40ff200640 1 -- 172.21.15.62:0/632108643 shutdown_connections 2024-03-23T11:19:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.818+0000 7f40ff200640 1 --2- 172.21.15.62:0/632108643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f806c6e0 0x7f40f806cb30 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.818+0000 7f40ff200640 1 --2- 172.21.15.62:0/632108643 >> v2:172.21.15.120:3300/0 conn(0x7f40f80cafa0 0x7f40f80cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.818+0000 7f40ff200640 1 -- 172.21.15.62:0/632108643 >> 172.21.15.62:0/632108643 conn(0x7f40f80b02e0 msgr2=0x7f40f80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.818+0000 7f40ff200640 1 -- 172.21.15.62:0/632108643 shutdown_connections 2024-03-23T11:19:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.818+0000 7f40ff200640 1 -- 172.21.15.62:0/632108643 wait complete. 2024-03-23T11:19:56.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.820+0000 7f40ff200640 1 Processor -- start 2024-03-23T11:19:56.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.820+0000 7f40ff200640 1 -- start start 2024-03-23T11:19:56.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.821+0000 7f40ff200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40f806c300 0x7f40f81578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.821+0000 7f40ff200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40f806c6e0 0x7f40f813f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.821+0000 7f40ff200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f80cafa0 0x7f40f813bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.821+0000 7f40ff200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8157e10 con 0x7f40f80cafa0 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.821+0000 7f40ff200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80b3710 con 0x7f40f806c300 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.821+0000 7f40ff200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f813e540 con 0x7f40f806c6e0 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.821+0000 7f40fd200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f80cafa0 0x7f40f813bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.822+0000 7f40fd200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f80cafa0 0x7f40f813bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59956/0 (socket says 172.21.15.62:59956) 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.822+0000 7f40fd200640 1 -- 172.21.15.62:0/1169566181 learned_addr learned my addr 172.21.15.62:0/1169566181 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.822+0000 7f40f7400640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40f806c6e0 0x7f40f813f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.822+0000 7f40f7e00640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40f806c300 0x7f40f81578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.822+0000 7f40fd200640 1 -- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40f806c6e0 msgr2=0x7f40f813f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.822+0000 7f40fd200640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40f806c6e0 0x7f40f813f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.822+0000 7f40fd200640 1 -- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40f806c300 msgr2=0x7f40f81578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:56.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.822+0000 7f40fd200640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40f806c300 0x7f40f81578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:56.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.822+0000 7f40fd200640 1 -- 172.21.15.62:0/1169566181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f0009400 con 0x7f40f80cafa0 2024-03-23T11:19:56.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.823+0000 7f40fd200640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f80cafa0 0x7f40f813bfa0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f40f002bee0 tx=0x7f40f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:56.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.823+0000 7f40efe00640 1 -- 172.21.15.62:0/1169566181 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f003a630 con 0x7f40f80cafa0 2024-03-23T11:19:56.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.823+0000 7f40efe00640 1 -- 172.21.15.62:0/1169566181 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f003a7b0 con 0x7f40f80cafa0 2024-03-23T11:19:56.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.823+0000 7f40efe00640 1 -- 172.21.15.62:0/1169566181 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f003ab60 con 0x7f40f80cafa0 2024-03-23T11:19:56.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.823+0000 7f40ff200640 1 -- 172.21.15.62:0/1169566181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f813e820 con 0x7f40f80cafa0 2024-03-23T11:19:56.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.823+0000 7f40ff200640 1 -- 172.21.15.62:0/1169566181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f813ede0 con 0x7f40f80cafa0 2024-03-23T11:19:56.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.824+0000 7f40ff200640 1 -- 172.21.15.62:0/1169566181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f8066c90 con 0x7f40f80cafa0 2024-03-23T11:19:56.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.825+0000 7f40efe00640 1 -- 172.21.15.62:0/1169566181 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f40f003acc0 con 0x7f40f80cafa0 2024-03-23T11:19:56.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.826+0000 7f40efe00640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f40d406a7a0 0x7f40d406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:56.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.826+0000 7f40efe00640 1 -- 172.21.15.62:0/1169566181 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f40f00aa950 con 0x7f40f80cafa0 2024-03-23T11:19:56.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.827+0000 7f40f7e00640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f40d406a7a0 0x7f40d406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:56.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.828+0000 7f40f7e00640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f40d406a7a0 0x7f40d406cc60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f40e8001560 tx=0x7f40e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:56.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:56.830+0000 7f40efe00640 1 -- 172.21.15.62:0/1169566181 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40f0075cf0 con 0x7f40f80cafa0 2024-03-23T11:19:57.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.026+0000 7f40ff200640 1 -- 172.21.15.62:0/1169566181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f40f815a7c0 con 0x7f40f80cafa0 2024-03-23T11:19:57.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.027+0000 7f40efe00640 1 -- 172.21.15.62:0/1169566181 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f40f0035020 con 0x7f40f80cafa0 2024-03-23T11:19:57.028 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:57.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 -- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f40d406a7a0 msgr2=0x7f40d406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f40d406a7a0 0x7f40d406cc60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f40e8001560 tx=0x7f40e8009040 comp rx=0 tx=0).stop 2024-03-23T11:19:57.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 -- 172.21.15.62:0/1169566181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f80cafa0 msgr2=0x7f40f813bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f80cafa0 0x7f40f813bfa0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f40f002bee0 tx=0x7f40f00046c0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 -- 172.21.15.62:0/1169566181 shutdown_connections 2024-03-23T11:19:57.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f40d406a7a0 0x7f40d406cc60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f80cafa0 0x7f40f813bfa0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40f806c6e0 0x7f40f813f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 --2- 172.21.15.62:0/1169566181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40f806c300 0x7f40f81578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 -- 172.21.15.62:0/1169566181 >> 172.21.15.62:0/1169566181 conn(0x7f40f80b02e0 msgr2=0x7f40f8072c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:57.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 -- 172.21.15.62:0/1169566181 shutdown_connections 2024-03-23T11:19:57.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.030+0000 7f40ff200640 1 -- 172.21.15.62:0/1169566181 wait complete. 2024-03-23T11:19:57.049 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:57.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.276+0000 7f9993200640 1 Processor -- start 2024-03-23T11:19:57.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.276+0000 7f9993200640 1 -- start start 2024-03-23T11:19:57.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.278+0000 7f9993200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9994070090 0x7f9994070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.278+0000 7f9993200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f99940b2820 0x7f99940b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.278+0000 7f9993200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99940b31b0 con 0x7f9994143dd0 2024-03-23T11:19:57.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.278+0000 7f9993200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9994070a60 con 0x7f9994070090 2024-03-23T11:19:57.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.278+0000 7f9993200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99940cd140 con 0x7f99940b2820 2024-03-23T11:19:57.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.278+0000 7f9991400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9994070090 0x7f9994070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.279+0000 7f9991400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9994070090 0x7f9994070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59958/0 (socket says 172.21.15.62:59958) 2024-03-23T11:19:57.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.279+0000 7f9991400640 1 -- 172.21.15.62:0/137076183 learned_addr learned my addr 172.21.15.62:0/137076183 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:57.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.279+0000 7f9992800640 1 --2- 172.21.15.62:0/137076183 >> v2:172.21.15.120:3300/0 conn(0x7f99940b2820 0x7f99940b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.279+0000 7f9991400640 1 -- 172.21.15.62:0/137076183 >> v2:172.21.15.120:3300/0 conn(0x7f99940b2820 msgr2=0x7f99940b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:57.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.279+0000 7f9991400640 1 --2- 172.21.15.62:0/137076183 >> v2:172.21.15.120:3300/0 conn(0x7f99940b2820 0x7f99940b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.279+0000 7f9991400640 1 -- 172.21.15.62:0/137076183 >> v1:172.21.15.120:6789/0 conn(0x7f9994143dd0 legacy=0x7f999415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.279+0000 7f9991400640 1 -- 172.21.15.62:0/137076183 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99940cd3e0 con 0x7f9994070090 2024-03-23T11:19:57.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.280+0000 7f9991400640 1 --2- 172.21.15.62:0/137076183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9994070090 0x7f9994070520 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f99940cbe50 tx=0x7f998002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7c4fcccec7c5c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:57.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.280+0000 7f9990a00640 1 -- 172.21.15.62:0/137076183 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9980030860 con 0x7f9994070090 2024-03-23T11:19:57.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.281+0000 7f9990a00640 1 -- 172.21.15.62:0/137076183 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99800309c0 con 0x7f9994070090 2024-03-23T11:19:57.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.281+0000 7f9990a00640 1 -- 172.21.15.62:0/137076183 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9980030d50 con 0x7f9994070090 2024-03-23T11:19:57.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.281+0000 7f9993200640 1 -- 172.21.15.62:0/137076183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9994070090 msgr2=0x7f9994070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.281+0000 7f9993200640 1 --2- 172.21.15.62:0/137076183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9994070090 0x7f9994070520 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f99940cbe50 tx=0x7f998002b9a0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.281+0000 7f9993200640 1 -- 172.21.15.62:0/137076183 shutdown_connections 2024-03-23T11:19:57.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.281+0000 7f9993200640 1 --2- 172.21.15.62:0/137076183 >> v2:172.21.15.120:3300/0 conn(0x7f99940b2820 0x7f99940b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.281+0000 7f9993200640 1 --2- 172.21.15.62:0/137076183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9994070090 0x7f9994070520 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.281+0000 7f9993200640 1 -- 172.21.15.62:0/137076183 >> 172.21.15.62:0/137076183 conn(0x7f9994066050 msgr2=0x7f9994069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:57.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.281+0000 7f9993200640 1 -- 172.21.15.62:0/137076183 shutdown_connections 2024-03-23T11:19:57.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.281+0000 7f9993200640 1 -- 172.21.15.62:0/137076183 wait complete. 2024-03-23T11:19:57.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.283+0000 7f9993200640 1 Processor -- start 2024-03-23T11:19:57.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.283+0000 7f9993200640 1 -- start start 2024-03-23T11:19:57.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.284+0000 7f9993200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9994070090 0x7f99940c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.284+0000 7f9993200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99940b2820 0x7f99940c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.284+0000 7f9993200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9994143dd0 0x7f9994134b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.284+0000 7f9993200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999415a8d0 con 0x7f9994143dd0 2024-03-23T11:19:57.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.284+0000 7f9993200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9994132a90 con 0x7f99940b2820 2024-03-23T11:19:57.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.284+0000 7f9993200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99940c3490 con 0x7f9994070090 2024-03-23T11:19:57.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.285+0000 7f9992800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9994143dd0 0x7f9994134b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.285+0000 7f9992800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9994143dd0 0x7f9994134b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59444/0 (socket says 172.21.15.62:59444) 2024-03-23T11:19:57.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.285+0000 7f9992800640 1 -- 172.21.15.62:0/3576365624 learned_addr learned my addr 172.21.15.62:0/3576365624 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.285+0000 7f9991e00640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9994070090 0x7f99940c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.285+0000 7f9991400640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99940b2820 0x7f99940c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.285+0000 7f9992800640 1 -- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9994070090 msgr2=0x7f99940c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.285+0000 7f9992800640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9994070090 0x7f99940c5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.285+0000 7f9992800640 1 -- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99940b2820 msgr2=0x7f99940c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.285+0000 7f9992800640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99940b2820 0x7f99940c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.285+0000 7f9992800640 1 -- 172.21.15.62:0/3576365624 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99800094d0 con 0x7f9994143dd0 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.285+0000 7f9992800640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9994143dd0 0x7f9994134b20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f998c00cd70 tx=0x7f998c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.286+0000 7f998a000640 1 -- 172.21.15.62:0/3576365624 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c003b90 con 0x7f9994143dd0 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.286+0000 7f998a000640 1 -- 172.21.15.62:0/3576365624 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c003cf0 con 0x7f9994143dd0 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.286+0000 7f998a000640 1 -- 172.21.15.62:0/3576365624 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c004240 con 0x7f9994143dd0 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.286+0000 7f9993200640 1 -- 172.21.15.62:0/3576365624 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99940b0650 con 0x7f9994143dd0 2024-03-23T11:19:57.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.286+0000 7f9993200640 1 -- 172.21.15.62:0/3576365624 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9994132fc0 con 0x7f9994143dd0 2024-03-23T11:19:57.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.287+0000 7f9993200640 1 -- 172.21.15.62:0/3576365624 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999406deb0 con 0x7f9994143dd0 2024-03-23T11:19:57.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.288+0000 7f998a000640 1 -- 172.21.15.62:0/3576365624 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f998c0043a0 con 0x7f9994143dd0 2024-03-23T11:19:57.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.289+0000 7f998a000640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997406a7a0 0x7f997406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.289+0000 7f998a000640 1 -- 172.21.15.62:0/3576365624 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f998c012070 con 0x7f9994143dd0 2024-03-23T11:19:57.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.290+0000 7f9991e00640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997406a7a0 0x7f997406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.291+0000 7f9991e00640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997406a7a0 0x7f997406cc60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9984000f60 tx=0x7f9984009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:57.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.293+0000 7f998a000640 1 -- 172.21.15.62:0/3576365624 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f998c054bc0 con 0x7f9994143dd0 2024-03-23T11:19:57.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.494+0000 7f9993200640 1 -- 172.21.15.62:0/3576365624 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9994066bb0 con 0x7f9994143dd0 2024-03-23T11:19:57.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.495+0000 7f998a000640 1 -- 172.21.15.62:0/3576365624 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f998c0581e0 con 0x7f9994143dd0 2024-03-23T11:19:57.495 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:57.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 -- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997406a7a0 msgr2=0x7f997406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997406a7a0 0x7f997406cc60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9984000f60 tx=0x7f9984009040 comp rx=0 tx=0).stop 2024-03-23T11:19:57.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 -- 172.21.15.62:0/3576365624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9994143dd0 msgr2=0x7f9994134b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9994143dd0 0x7f9994134b20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f998c00cd70 tx=0x7f998c00a660 comp rx=0 tx=0).stop 2024-03-23T11:19:57.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 -- 172.21.15.62:0/3576365624 shutdown_connections 2024-03-23T11:19:57.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997406a7a0 0x7f997406cc60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9994143dd0 0x7f9994134b20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99940b2820 0x7f99940c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 --2- 172.21.15.62:0/3576365624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9994070090 0x7f99940c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 -- 172.21.15.62:0/3576365624 >> 172.21.15.62:0/3576365624 conn(0x7f9994066050 msgr2=0x7f999415b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:57.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 -- 172.21.15.62:0/3576365624 shutdown_connections 2024-03-23T11:19:57.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.498+0000 7f9993200640 1 -- 172.21.15.62:0/3576365624 wait complete. 2024-03-23T11:19:57.518 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:19:57.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.745+0000 7f044ca00640 1 Processor -- start 2024-03-23T11:19:57.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.745+0000 7f044ca00640 1 -- start start 2024-03-23T11:19:57.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.747+0000 7f044ca00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f04480cd7c0 0x7f04480b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.747+0000 7f044ca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f044806dca0 0x7f044806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.747+0000 7f044ca00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04480cad20 con 0x7f04480cdba0 2024-03-23T11:19:57.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.747+0000 7f044ca00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044806e130 con 0x7f04480cd7c0 2024-03-23T11:19:57.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.747+0000 7f044ca00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044815a8a0 con 0x7f044806dca0 2024-03-23T11:19:57.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.747+0000 7f0445e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f04480cd7c0 0x7f04480b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.748+0000 7f0445e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f04480cd7c0 0x7f04480b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59448/0 (socket says 172.21.15.62:59448) 2024-03-23T11:19:57.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.748+0000 7f0445e00640 1 -- 172.21.15.62:0/97193211 learned_addr learned my addr 172.21.15.62:0/97193211 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:57.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.748+0000 7f0445400640 1 --2- 172.21.15.62:0/97193211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f044806dca0 0x7f044806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.748+0000 7f0445e00640 1 -- 172.21.15.62:0/97193211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f044806dca0 msgr2=0x7f044806c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:57.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.748+0000 7f0445e00640 1 --2- 172.21.15.62:0/97193211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f044806dca0 0x7f044806c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.748+0000 7f0445e00640 1 -- 172.21.15.62:0/97193211 >> v1:172.21.15.62:6789/0 conn(0x7f04480cdba0 legacy=0x7f04480cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.748+0000 7f0445e00640 1 -- 172.21.15.62:0/97193211 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044815ab80 con 0x7f04480cd7c0 2024-03-23T11:19:57.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.748+0000 7f0444a00640 1 -- 172.21.15.62:0/97193211 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957795656 0 0) 0x7f04480cad20 con 0x7f04480cdba0 2024-03-23T11:19:57.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.749+0000 7f0445e00640 1 --2- 172.21.15.62:0/97193211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04480cd7c0 0x7f04480b15e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f043800bb10 tx=0x7f043802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3af2c8f74740baa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:57.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.750+0000 7f0444a00640 1 -- 172.21.15.62:0/97193211 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043802ee70 con 0x7f04480cd7c0 2024-03-23T11:19:57.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.750+0000 7f0444a00640 1 -- 172.21.15.62:0/97193211 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0438036440 con 0x7f04480cd7c0 2024-03-23T11:19:57.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.750+0000 7f0444a00640 1 -- 172.21.15.62:0/97193211 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04380367d0 con 0x7f04480cd7c0 2024-03-23T11:19:57.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.750+0000 7f044ca00640 1 -- 172.21.15.62:0/97193211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04480cd7c0 msgr2=0x7f04480b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.750+0000 7f044ca00640 1 --2- 172.21.15.62:0/97193211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04480cd7c0 0x7f04480b15e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f043800bb10 tx=0x7f043802e0c0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.750+0000 7f044ca00640 1 -- 172.21.15.62:0/97193211 shutdown_connections 2024-03-23T11:19:57.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.750+0000 7f044ca00640 1 --2- 172.21.15.62:0/97193211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f044806dca0 0x7f044806c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.750+0000 7f044ca00640 1 --2- 172.21.15.62:0/97193211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04480cd7c0 0x7f04480b15e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.750+0000 7f044ca00640 1 -- 172.21.15.62:0/97193211 >> 172.21.15.62:0/97193211 conn(0x7f04480b02e0 msgr2=0x7f04480b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:57.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.751+0000 7f044ca00640 1 -- 172.21.15.62:0/97193211 shutdown_connections 2024-03-23T11:19:57.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.751+0000 7f044ca00640 1 -- 172.21.15.62:0/97193211 wait complete. 2024-03-23T11:19:57.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.752+0000 7f044ca00640 1 Processor -- start 2024-03-23T11:19:57.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.753+0000 7f044ca00640 1 -- start start 2024-03-23T11:19:57.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f044ca00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04480cd7c0 0x7f04480accc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f044ca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04480cdba0 0x7f0448146e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f044ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448159320 0x7f04481549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f044ca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480ad200 con 0x7f0448159320 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f044ca00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480abc20 con 0x7f04480cd7c0 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f044ca00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04480abf00 con 0x7f04480cdba0 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f0446800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448159320 0x7f04481549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f0446800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448159320 0x7f04481549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59454/0 (socket says 172.21.15.62:59454) 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f0446800640 1 -- 172.21.15.62:0/4202135079 learned_addr learned my addr 172.21.15.62:0/4202135079 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f0445400640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04480cdba0 0x7f0448146e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f0445e00640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04480cd7c0 0x7f04480accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f0446800640 1 -- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04480cdba0 msgr2=0x7f0448146e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.754+0000 7f0446800640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04480cdba0 0x7f0448146e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.755+0000 7f0446800640 1 -- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04480cd7c0 msgr2=0x7f04480accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.755+0000 7f0446800640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04480cd7c0 0x7f04480accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.755+0000 7f0446800640 1 -- 172.21.15.62:0/4202135079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043800b770 con 0x7f0448159320 2024-03-23T11:19:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.755+0000 7f0446800640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448159320 0x7f04481549a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f044000c710 tx=0x7f044000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:57.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.755+0000 7f043e000640 1 -- 172.21.15.62:0/4202135079 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440003da0 con 0x7f0448159320 2024-03-23T11:19:57.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.755+0000 7f043e000640 1 -- 172.21.15.62:0/4202135079 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044000ce20 con 0x7f0448159320 2024-03-23T11:19:57.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.755+0000 7f044ca00640 1 -- 172.21.15.62:0/4202135079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04480ac1e0 con 0x7f0448159320 2024-03-23T11:19:57.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.755+0000 7f043e000640 1 -- 172.21.15.62:0/4202135079 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440005290 con 0x7f0448159320 2024-03-23T11:19:57.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.755+0000 7f044ca00640 1 -- 172.21.15.62:0/4202135079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04480ac7a0 con 0x7f0448159320 2024-03-23T11:19:57.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.756+0000 7f044ca00640 1 -- 172.21.15.62:0/4202135079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04480ac9c0 con 0x7f0448159320 2024-03-23T11:19:57.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.757+0000 7f043e000640 1 -- 172.21.15.62:0/4202135079 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f04400053f0 con 0x7f0448159320 2024-03-23T11:19:57.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.758+0000 7f043e000640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042806a7d0 0x7f042806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:57.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.758+0000 7f043e000640 1 -- 172.21.15.62:0/4202135079 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0440088570 con 0x7f0448159320 2024-03-23T11:19:57.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.759+0000 7f0445e00640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042806a7d0 0x7f042806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:57.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.760+0000 7f0445e00640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042806a7d0 0x7f042806cc90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0438038000 tx=0x7f0438008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:57.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.763+0000 7f043e000640 1 -- 172.21.15.62:0/4202135079 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f04400539a0 con 0x7f0448159320 2024-03-23T11:19:57.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.962+0000 7f044ca00640 1 -- 172.21.15.62:0/4202135079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0448066c90 con 0x7f0448159320 2024-03-23T11:19:57.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.963+0000 7f043e000640 1 -- 172.21.15.62:0/4202135079 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f04400170a0 con 0x7f0448159320 2024-03-23T11:19:57.964 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:19:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 -- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042806a7d0 msgr2=0x7f042806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042806a7d0 0x7f042806cc90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0438038000 tx=0x7f0438008040 comp rx=0 tx=0).stop 2024-03-23T11:19:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 -- 172.21.15.62:0/4202135079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448159320 msgr2=0x7f04481549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448159320 0x7f04481549a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f044000c710 tx=0x7f044000cbd0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 -- 172.21.15.62:0/4202135079 shutdown_connections 2024-03-23T11:19:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042806a7d0 0x7f042806cc90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448159320 0x7f04481549a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04480cdba0 0x7f0448146e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 --2- 172.21.15.62:0/4202135079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04480cd7c0 0x7f04480accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 -- 172.21.15.62:0/4202135079 >> 172.21.15.62:0/4202135079 conn(0x7f04480b02e0 msgr2=0x7f04480702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 -- 172.21.15.62:0/4202135079 shutdown_connections 2024-03-23T11:19:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:57.966+0000 7f044ca00640 1 -- 172.21.15.62:0/4202135079 wait complete. 2024-03-23T11:19:57.985 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:19:58.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.199+0000 7f996f200640 1 Processor -- start 2024-03-23T11:19:58.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.199+0000 7f996f200640 1 -- start start 2024-03-23T11:19:58.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.201+0000 7f996f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f99700cb070 0x7f99700989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.201+0000 7f996f200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9970098f10 0x7f99700b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.201+0000 7f996f200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9970099e20 con 0x7f997015a770 2024-03-23T11:19:58.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.201+0000 7f996f200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99700992f0 con 0x7f9970098f10 2024-03-23T11:19:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.201+0000 7f996f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99700cb5c0 con 0x7f99700cb070 2024-03-23T11:19:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.201+0000 7f996de00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f997015a770 0x7f9970099710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50366/0 (socket says 172.21.15.62:50366) 2024-03-23T11:19:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.201+0000 7f996de00640 1 -- 172.21.15.62:0/3528278980 learned_addr learned my addr 172.21.15.62:0/3528278980 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.201+0000 7f996e800640 1 --2- 172.21.15.62:0/3528278980 >> v2:172.21.15.62:3300/0 conn(0x7f9970098f10 0x7f99700b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.201+0000 7f996d400640 1 --2- 172.21.15.62:0/3528278980 >> v2:172.21.15.120:3300/0 conn(0x7f99700cb070 0x7f99700989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.202+0000 7f996ca00640 1 -- 172.21.15.62:0/3528278980 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849569974 0 0) 0x7f9970099e20 con 0x7f997015a770 2024-03-23T11:19:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.202+0000 7f996ca00640 1 -- 172.21.15.62:0/3528278980 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9954003520 con 0x7f997015a770 2024-03-23T11:19:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.202+0000 7f996e800640 1 -- 172.21.15.62:0/3528278980 >> v2:172.21.15.120:3300/0 conn(0x7f99700cb070 msgr2=0x7f99700989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.202+0000 7f996e800640 1 --2- 172.21.15.62:0/3528278980 >> v2:172.21.15.120:3300/0 conn(0x7f99700cb070 0x7f99700989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.202+0000 7f996e800640 1 -- 172.21.15.62:0/3528278980 >> v1:172.21.15.62:6789/0 conn(0x7f997015a770 legacy=0x7f9970099710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.202+0000 7f996e800640 1 -- 172.21.15.62:0/3528278980 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99700cb8a0 con 0x7f9970098f10 2024-03-23T11:19:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.203+0000 7f996e800640 1 --2- 172.21.15.62:0/3528278980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9970098f10 0x7f99700b10f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9970071d80 tx=0x7f996002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b881b249b48b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.203+0000 7f996ca00640 1 -- 172.21.15.62:0/3528278980 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996002e070 con 0x7f9970098f10 2024-03-23T11:19:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.203+0000 7f996ca00640 1 -- 172.21.15.62:0/3528278980 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99600315a0 con 0x7f9970098f10 2024-03-23T11:19:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.203+0000 7f996ca00640 1 -- 172.21.15.62:0/3528278980 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960031930 con 0x7f9970098f10 2024-03-23T11:19:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.204+0000 7f996f200640 1 -- 172.21.15.62:0/3528278980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9970098f10 msgr2=0x7f99700b10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.204+0000 7f996f200640 1 --2- 172.21.15.62:0/3528278980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9970098f10 0x7f99700b10f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9970071d80 tx=0x7f996002b840 comp rx=0 tx=0).stop 2024-03-23T11:19:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.204+0000 7f996f200640 1 -- 172.21.15.62:0/3528278980 shutdown_connections 2024-03-23T11:19:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.204+0000 7f996f200640 1 --2- 172.21.15.62:0/3528278980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9970098f10 0x7f99700b10f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.204+0000 7f996f200640 1 --2- 172.21.15.62:0/3528278980 >> v2:172.21.15.120:3300/0 conn(0x7f99700cb070 0x7f99700989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.204+0000 7f996f200640 1 -- 172.21.15.62:0/3528278980 >> 172.21.15.62:0/3528278980 conn(0x7f99700b1f50 msgr2=0x7f99700b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.204+0000 7f996f200640 1 -- 172.21.15.62:0/3528278980 shutdown_connections 2024-03-23T11:19:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.204+0000 7f996f200640 1 -- 172.21.15.62:0/3528278980 wait complete. 2024-03-23T11:19:58.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.206+0000 7f996f200640 1 Processor -- start 2024-03-23T11:19:58.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.206+0000 7f996f200640 1 -- start start 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.207+0000 7f996f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9970098f10 0x7f99701f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.207+0000 7f996f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99700cb070 0x7f997015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.207+0000 7f996f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f997015a770 0x7f997015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.207+0000 7f996f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99701f2040 con 0x7f9970098f10 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.207+0000 7f996f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9970145100 con 0x7f997015a770 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.207+0000 7f996f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9970145370 con 0x7f99700cb070 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.207+0000 7f996de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9970098f10 0x7f99701f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.208+0000 7f996de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9970098f10 0x7f99701f1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59484/0 (socket says 172.21.15.62:59484) 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.208+0000 7f996de00640 1 -- 172.21.15.62:0/3027091512 learned_addr learned my addr 172.21.15.62:0/3027091512 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.208+0000 7f996e800640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f997015a770 0x7f997015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.208+0000 7f996de00640 1 -- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99700cb070 msgr2=0x7f997015d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.208+0000 7f996de00640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99700cb070 0x7f997015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.208+0000 7f996de00640 1 -- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f997015a770 msgr2=0x7f997015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.208+0000 7f996de00640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f997015a770 0x7f997015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.208+0000 7f996de00640 1 -- 172.21.15.62:0/3027091512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9960009400 con 0x7f9970098f10 2024-03-23T11:19:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.208+0000 7f996de00640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9970098f10 0x7f99701f1b00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f996800cbf0 tx=0x7f996800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.208+0000 7f9966000640 1 -- 172.21.15.62:0/3027091512 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996800ad40 con 0x7f9970098f10 2024-03-23T11:19:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.209+0000 7f9966000640 1 -- 172.21.15.62:0/3027091512 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9968012400 con 0x7f9970098f10 2024-03-23T11:19:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.209+0000 7f996f200640 1 -- 172.21.15.62:0/3027091512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9970145650 con 0x7f9970098f10 2024-03-23T11:19:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.209+0000 7f9966000640 1 -- 172.21.15.62:0/3027091512 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996800aea0 con 0x7f9970098f10 2024-03-23T11:19:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.209+0000 7f996f200640 1 -- 172.21.15.62:0/3027091512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9970148550 con 0x7f9970098f10 2024-03-23T11:19:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.210+0000 7f996f200640 1 -- 172.21.15.62:0/3027091512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99700672c0 con 0x7f9970098f10 2024-03-23T11:19:58.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.211+0000 7f9966000640 1 -- 172.21.15.62:0/3027091512 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9968007070 con 0x7f9970098f10 2024-03-23T11:19:58.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.212+0000 7f9966000640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f994c06a7a0 0x7f994c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.212+0000 7f9966000640 1 -- 172.21.15.62:0/3027091512 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9968088ec0 con 0x7f9970098f10 2024-03-23T11:19:58.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.212+0000 7f996d400640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f994c06a7a0 0x7f994c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:58.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.214+0000 7f996d400640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f994c06a7a0 0x7f994c06cc60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9958006a90 tx=0x7f9958006a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:58.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.216+0000 7f9966000640 1 -- 172.21.15.62:0/3027091512 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f99680541c0 con 0x7f9970098f10 2024-03-23T11:19:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.430+0000 7f996f200640 1 -- 172.21.15.62:0/3027091512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f99700ce820 con 0x7f9970098f10 2024-03-23T11:19:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.431+0000 7f9966000640 1 -- 172.21.15.62:0/3027091512 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9968017080 con 0x7f9970098f10 2024-03-23T11:19:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:19:58.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.433+0000 7f996f200640 1 -- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f994c06a7a0 msgr2=0x7f994c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.433+0000 7f996f200640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f994c06a7a0 0x7f994c06cc60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9958006a90 tx=0x7f9958006a20 comp rx=0 tx=0).stop 2024-03-23T11:19:58.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.434+0000 7f996f200640 1 -- 172.21.15.62:0/3027091512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9970098f10 msgr2=0x7f99701f1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.434+0000 7f996f200640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9970098f10 0x7f99701f1b00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f996800cbf0 tx=0x7f996800a430 comp rx=0 tx=0).stop 2024-03-23T11:19:58.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.434+0000 7f996f200640 1 -- 172.21.15.62:0/3027091512 shutdown_connections 2024-03-23T11:19:58.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.434+0000 7f996f200640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f994c06a7a0 0x7f994c06cc60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.434+0000 7f996f200640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f997015a770 0x7f997015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.434+0000 7f996f200640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99700cb070 0x7f997015d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.434+0000 7f996f200640 1 --2- 172.21.15.62:0/3027091512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9970098f10 0x7f99701f1b00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.434+0000 7f996f200640 1 -- 172.21.15.62:0/3027091512 >> 172.21.15.62:0/3027091512 conn(0x7f99700b1f50 msgr2=0x7f9970099710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.434+0000 7f996f200640 1 -- 172.21.15.62:0/3027091512 shutdown_connections 2024-03-23T11:19:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.434+0000 7f996f200640 1 -- 172.21.15.62:0/3027091512 wait complete. 2024-03-23T11:19:58.454 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:19:58.454 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:19:58.513 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:19:58.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.739+0000 7f989a200640 1 Processor -- start 2024-03-23T11:19:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.740+0000 7f989a200640 1 -- start start 2024-03-23T11:19:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.741+0000 7f989a200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f989415c9e0 0x7f98941607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.741+0000 7f989a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f989415cdc0 0x7f98940d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.741+0000 7f989a200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f989406bfc0 con 0x7f98940cf650 2024-03-23T11:19:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.741+0000 7f989a200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98940ce140 con 0x7f989415c9e0 2024-03-23T11:19:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.741+0000 7f989a200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98940b4370 con 0x7f989415cdc0 2024-03-23T11:19:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.741+0000 7f9893400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f989415c9e0 0x7f98941607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.741+0000 7f9893e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f98940cf650 0x7f989415d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50374/0 (socket says 172.21.15.62:50374) 2024-03-23T11:19:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.742+0000 7f9893e00640 1 -- 172.21.15.62:0/2898312318 learned_addr learned my addr 172.21.15.62:0/2898312318 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:19:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.742+0000 7f9892a00640 1 --2- 172.21.15.62:0/2898312318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f989415cdc0 0x7f98940d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:58.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.742+0000 7f9893400640 1 -- 172.21.15.62:0/2898312318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f989415cdc0 msgr2=0x7f98940d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:58.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.742+0000 7f9893400640 1 --2- 172.21.15.62:0/2898312318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f989415cdc0 0x7f98940d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.742+0000 7f9893400640 1 -- 172.21.15.62:0/2898312318 >> v1:172.21.15.62:6789/0 conn(0x7f98940cf650 legacy=0x7f989415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.742+0000 7f9893400640 1 -- 172.21.15.62:0/2898312318 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98940b5260 con 0x7f989415c9e0 2024-03-23T11:19:58.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.742+0000 7f9892000640 1 -- 172.21.15.62:0/2898312318 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481795948 0 0) 0x7f989406bfc0 con 0x7f98940cf650 2024-03-23T11:19:58.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.743+0000 7f9893400640 1 --2- 172.21.15.62:0/2898312318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f989415c9e0 0x7f98941607c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9884009d90 tx=0x7f988402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189ad29456833751 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:58.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.744+0000 7f9892000640 1 -- 172.21.15.62:0/2898312318 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988402cc50 con 0x7f989415c9e0 2024-03-23T11:19:58.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.744+0000 7f9892000640 1 -- 172.21.15.62:0/2898312318 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988402cdb0 con 0x7f989415c9e0 2024-03-23T11:19:58.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.744+0000 7f9892000640 1 -- 172.21.15.62:0/2898312318 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9884034660 con 0x7f989415c9e0 2024-03-23T11:19:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.744+0000 7f989a200640 1 -- 172.21.15.62:0/2898312318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f989415c9e0 msgr2=0x7f98941607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.744+0000 7f989a200640 1 --2- 172.21.15.62:0/2898312318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f989415c9e0 0x7f98941607c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9884009d90 tx=0x7f988402bea0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.745+0000 7f989a200640 1 -- 172.21.15.62:0/2898312318 shutdown_connections 2024-03-23T11:19:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.745+0000 7f989a200640 1 --2- 172.21.15.62:0/2898312318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f989415cdc0 0x7f98940d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.745+0000 7f989a200640 1 --2- 172.21.15.62:0/2898312318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f989415c9e0 0x7f98941607c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.745+0000 7f989a200640 1 -- 172.21.15.62:0/2898312318 >> 172.21.15.62:0/2898312318 conn(0x7f9894071570 msgr2=0x7f9894071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.745+0000 7f989a200640 1 -- 172.21.15.62:0/2898312318 shutdown_connections 2024-03-23T11:19:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.745+0000 7f989a200640 1 -- 172.21.15.62:0/2898312318 wait complete. 2024-03-23T11:19:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.746+0000 7f989a200640 1 Processor -- start 2024-03-23T11:19:58.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.747+0000 7f989a200640 1 -- start start 2024-03-23T11:19:58.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f989a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98940cf650 0x7f98941f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f989a200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f989415c9e0 0x7f98940abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f989a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f989415cdc0 0x7f989415d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f989a200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98941f60f0 con 0x7f98940cf650 2024-03-23T11:19:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f989a200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98940ac210 con 0x7f989415c9e0 2024-03-23T11:19:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f989a200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98940a9530 con 0x7f989415cdc0 2024-03-23T11:19:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f9893400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98940cf650 0x7f98941f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f9893400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98940cf650 0x7f98941f5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59512/0 (socket says 172.21.15.62:59512) 2024-03-23T11:19:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f9893400640 1 -- 172.21.15.62:0/3334108073 learned_addr learned my addr 172.21.15.62:0/3334108073 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f9893400640 1 -- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f989415cdc0 msgr2=0x7f989415d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f9893400640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f989415cdc0 0x7f989415d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f9893400640 1 -- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f989415c9e0 msgr2=0x7f98940abcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f9893400640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f989415c9e0 0x7f98940abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.748+0000 7f9893400640 1 -- 172.21.15.62:0/3334108073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9884009a70 con 0x7f98940cf650 2024-03-23T11:19:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.749+0000 7f9893400640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98940cf650 0x7f98941f5bb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9884009d90 tx=0x7f9884004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.749+0000 7f988b400640 1 -- 172.21.15.62:0/3334108073 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9884040650 con 0x7f98940cf650 2024-03-23T11:19:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.749+0000 7f988b400640 1 -- 172.21.15.62:0/3334108073 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98840407b0 con 0x7f98940cf650 2024-03-23T11:19:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.749+0000 7f988b400640 1 -- 172.21.15.62:0/3334108073 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9884040b60 con 0x7f98940cf650 2024-03-23T11:19:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.749+0000 7f989a200640 1 -- 172.21.15.62:0/3334108073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98941f6320 con 0x7f98940cf650 2024-03-23T11:19:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.749+0000 7f989a200640 1 -- 172.21.15.62:0/3334108073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98941f68e0 con 0x7f98940cf650 2024-03-23T11:19:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.750+0000 7f989a200640 1 -- 172.21.15.62:0/3334108073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98940a9300 con 0x7f98940cf650 2024-03-23T11:19:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.751+0000 7f988b400640 1 -- 172.21.15.62:0/3334108073 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9884040cc0 con 0x7f98940cf650 2024-03-23T11:19:58.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.752+0000 7f988b400640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f987006a7a0 0x7f987006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:58.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.752+0000 7f988b400640 1 -- 172.21.15.62:0/3334108073 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f98840abd30 con 0x7f98940cf650 2024-03-23T11:19:58.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.753+0000 7f9892a00640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f987006a7a0 0x7f987006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:58.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.754+0000 7f9892a00640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f987006a7a0 0x7f987006cc60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f987c006fd0 tx=0x7f987c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:58.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.757+0000 7f988b400640 1 -- 172.21.15.62:0/3334108073 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9884077150 con 0x7f98940cf650 2024-03-23T11:19:58.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.957+0000 7f989a200640 1 -- 172.21.15.62:0/3334108073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9894066c90 con 0x7f98940cf650 2024-03-23T11:19:58.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.958+0000 7f988b400640 1 -- 172.21.15.62:0/3334108073 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16226 (secure 0 0 0) 0x7f988407a770 con 0x7f98940cf650 2024-03-23T11:19:58.959 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:19:58.959 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:19:42.867981+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:19:58.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.961+0000 7f989a200640 1 -- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f987006a7a0 msgr2=0x7f987006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.961+0000 7f989a200640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f987006a7a0 0x7f987006cc60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f987c006fd0 tx=0x7f987c008040 comp rx=0 tx=0).stop 2024-03-23T11:19:58.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.961+0000 7f989a200640 1 -- 172.21.15.62:0/3334108073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98940cf650 msgr2=0x7f98941f5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:58.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.961+0000 7f989a200640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98940cf650 0x7f98941f5bb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9884009d90 tx=0x7f9884004a40 comp rx=0 tx=0).stop 2024-03-23T11:19:58.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.961+0000 7f989a200640 1 -- 172.21.15.62:0/3334108073 shutdown_connections 2024-03-23T11:19:58.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.961+0000 7f989a200640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f987006a7a0 0x7f987006cc60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.961+0000 7f989a200640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f989415cdc0 0x7f989415d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.961+0000 7f989a200640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f989415c9e0 0x7f98940abcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.961+0000 7f989a200640 1 --2- 172.21.15.62:0/3334108073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98940cf650 0x7f98941f5bb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:58.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.961+0000 7f989a200640 1 -- 172.21.15.62:0/3334108073 >> 172.21.15.62:0/3334108073 conn(0x7f9894071570 msgr2=0x7f989415e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:58.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.962+0000 7f989a200640 1 -- 172.21.15.62:0/3334108073 shutdown_connections 2024-03-23T11:19:58.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:58.962+0000 7f989a200640 1 -- 172.21.15.62:0/3334108073 wait complete. 2024-03-23T11:19:58.982 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:19:58.982 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:19:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.209+0000 7fd9cb200640 1 Processor -- start 2024-03-23T11:19:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.209+0000 7fd9cb200640 1 -- start start 2024-03-23T11:19:59.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.210+0000 7fd9cb200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd9c40ce8d0 0x7fd9c40af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.211+0000 7fd9cb200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd9c40999e0 0x7fd9c40adf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.211+0000 7fd9cb200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c40cc860 con 0x7fd9c40cecb0 2024-03-23T11:19:59.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.211+0000 7fd9cb200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c40cac10 con 0x7fd9c40999e0 2024-03-23T11:19:59.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.211+0000 7fd9cb200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c40cbfc0 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.211+0000 7fd9ca800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd9c40999e0 0x7fd9c40adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.211+0000 7fd9ca800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd9c40999e0 0x7fd9c40adf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59522/0 (socket says 172.21.15.62:59522) 2024-03-23T11:19:59.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.211+0000 7fd9ca800640 1 -- 172.21.15.62:0/90652257 learned_addr learned my addr 172.21.15.62:0/90652257 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:59.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.211+0000 7fd9c9e00640 1 --2- 172.21.15.62:0/90652257 >> v2:172.21.15.120:3300/0 conn(0x7fd9c40ce8d0 0x7fd9c40af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.212+0000 7fd9ca800640 1 -- 172.21.15.62:0/90652257 >> v2:172.21.15.120:3300/0 conn(0x7fd9c40ce8d0 msgr2=0x7fd9c40af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:59.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.212+0000 7fd9ca800640 1 --2- 172.21.15.62:0/90652257 >> v2:172.21.15.120:3300/0 conn(0x7fd9c40ce8d0 0x7fd9c40af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.212+0000 7fd9ca800640 1 -- 172.21.15.62:0/90652257 >> v1:172.21.15.120:6789/0 conn(0x7fd9c40cecb0 legacy=0x7fd9c40ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.212+0000 7fd9ca800640 1 -- 172.21.15.62:0/90652257 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c40cc2a0 con 0x7fd9c40999e0 2024-03-23T11:19:59.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.213+0000 7fd9ca800640 1 --2- 172.21.15.62:0/90652257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9c40999e0 0x7fd9c40adf80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40716d0 tx=0x7fd9c002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280c3a22d54921d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:59.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.213+0000 7fd9c8a00640 1 -- 172.21.15.62:0/90652257 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c002cb10 con 0x7fd9c40999e0 2024-03-23T11:19:59.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.213+0000 7fd9c8a00640 1 -- 172.21.15.62:0/90652257 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c002cc70 con 0x7fd9c40999e0 2024-03-23T11:19:59.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.213+0000 7fd9c8a00640 1 -- 172.21.15.62:0/90652257 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00343f0 con 0x7fd9c40999e0 2024-03-23T11:19:59.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.214+0000 7fd9cb200640 1 -- 172.21.15.62:0/90652257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9c40999e0 msgr2=0x7fd9c40adf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.214+0000 7fd9cb200640 1 --2- 172.21.15.62:0/90652257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9c40999e0 0x7fd9c40adf80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40716d0 tx=0x7fd9c002bf70 comp rx=0 tx=0).stop 2024-03-23T11:19:59.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.214+0000 7fd9cb200640 1 -- 172.21.15.62:0/90652257 shutdown_connections 2024-03-23T11:19:59.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.214+0000 7fd9cb200640 1 --2- 172.21.15.62:0/90652257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9c40999e0 0x7fd9c40adf80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.214+0000 7fd9cb200640 1 --2- 172.21.15.62:0/90652257 >> v2:172.21.15.120:3300/0 conn(0x7fd9c40ce8d0 0x7fd9c40af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.214+0000 7fd9cb200640 1 -- 172.21.15.62:0/90652257 >> 172.21.15.62:0/90652257 conn(0x7fd9c40b1fb0 msgr2=0x7fd9c40b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:59.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.214+0000 7fd9cb200640 1 -- 172.21.15.62:0/90652257 shutdown_connections 2024-03-23T11:19:59.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.214+0000 7fd9cb200640 1 -- 172.21.15.62:0/90652257 wait complete. 2024-03-23T11:19:59.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.216+0000 7fd9cb200640 1 Processor -- start 2024-03-23T11:19:59.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.216+0000 7fd9cb200640 1 -- start start 2024-03-23T11:19:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.217+0000 7fd9cb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9c40999e0 0x7fd9c4157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.217+0000 7fd9cb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9c40ce8d0 0x7fd9c40c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.217+0000 7fd9cb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c40cecb0 0x7fd9c4159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.217+0000 7fd9cb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c40c5350 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.217+0000 7fd9cb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c40c6470 con 0x7fd9c40cecb0 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.217+0000 7fd9cb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c40c6750 con 0x7fd9c40999e0 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.217+0000 7fd9c9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9c40ce8d0 0x7fd9c40c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.218+0000 7fd9c9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9c40ce8d0 0x7fd9c40c4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59530/0 (socket says 172.21.15.62:59530) 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.218+0000 7fd9c9400640 1 -- 172.21.15.62:0/75855079 learned_addr learned my addr 172.21.15.62:0/75855079 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.218+0000 7fd9c9e00640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9c40999e0 0x7fd9c4157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.218+0000 7fd9ca800640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c40cecb0 0x7fd9c4159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.218+0000 7fd9c9400640 1 -- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9c40999e0 msgr2=0x7fd9c4157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.218+0000 7fd9c9400640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9c40999e0 0x7fd9c4157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.218+0000 7fd9c9400640 1 -- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c40cecb0 msgr2=0x7fd9c4159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.218+0000 7fd9c9400640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c40cecb0 0x7fd9c4159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.218+0000 7fd9c9400640 1 -- 172.21.15.62:0/75855079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c0009b30 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.218+0000 7fd9c9400640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9c40ce8d0 0x7fd9c40c4a00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400c450 tx=0x7fd9b400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.219+0000 7fd9be000640 1 -- 172.21.15.62:0/75855079 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40052d0 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.219+0000 7fd9be000640 1 -- 172.21.15.62:0/75855079 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b4005430 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.219+0000 7fd9be000640 1 -- 172.21.15.62:0/75855079 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4012870 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.219+0000 7fd9cb200640 1 -- 172.21.15.62:0/75855079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c40c6a30 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.219+0000 7fd9cb200640 1 -- 172.21.15.62:0/75855079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c40c4f40 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.220+0000 7fd9cb200640 1 -- 172.21.15.62:0/75855079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c4073540 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.221+0000 7fd9be000640 1 -- 172.21.15.62:0/75855079 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd9b40129d0 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.222+0000 7fd9be000640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd9a806a7d0 0x7fd9a806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.222+0000 7fd9be000640 1 -- 172.21.15.62:0/75855079 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd9b40889a0 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.222+0000 7fd9c9e00640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd9a806a7d0 0x7fd9a806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.223+0000 7fd9c9e00640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd9a806a7d0 0x7fd9a806cc90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8006fd0 tx=0x7fd9b8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:59.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.226+0000 7fd9be000640 1 -- 172.21.15.62:0/75855079 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9b4053ca0 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.423+0000 7fd9cb200640 1 -- 172.21.15.62:0/75855079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9c406ef60 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.424+0000 7fd9be000640 1 -- 172.21.15.62:0/75855079 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9b40572c0 con 0x7fd9c40ce8d0 2024-03-23T11:19:59.424 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:19:59.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 -- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd9a806a7d0 msgr2=0x7fd9a806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd9a806a7d0 0x7fd9a806cc90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8006fd0 tx=0x7fd9b8008040 comp rx=0 tx=0).stop 2024-03-23T11:19:59.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 -- 172.21.15.62:0/75855079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9c40ce8d0 msgr2=0x7fd9c40c4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9c40ce8d0 0x7fd9c40c4a00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400c450 tx=0x7fd9b400c910 comp rx=0 tx=0).stop 2024-03-23T11:19:59.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 -- 172.21.15.62:0/75855079 shutdown_connections 2024-03-23T11:19:59.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd9a806a7d0 0x7fd9a806cc90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9c40ce8d0 0x7fd9c40c4a00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c40cecb0 0x7fd9c4159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 --2- 172.21.15.62:0/75855079 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd9c40999e0 0x7fd9c4157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 -- 172.21.15.62:0/75855079 >> 172.21.15.62:0/75855079 conn(0x7fd9c40b1fb0 msgr2=0x7fd9c406f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:59.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 -- 172.21.15.62:0/75855079 shutdown_connections 2024-03-23T11:19:59.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.427+0000 7fd9cb200640 1 -- 172.21.15.62:0/75855079 wait complete. 2024-03-23T11:19:59.448 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:19:59.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.660+0000 7f5faf200640 1 Processor -- start 2024-03-23T11:19:59.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.660+0000 7f5faf200640 1 -- start start 2024-03-23T11:19:59.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.662+0000 7f5faf200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5fb0070090 0x7f5fb0070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.662+0000 7f5faf200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fb00b2820 0x7f5fb00b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.662+0000 7f5faf200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb00b31b0 con 0x7f5fb0143dd0 2024-03-23T11:19:59.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.662+0000 7f5faf200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb0070a60 con 0x7f5fb0070090 2024-03-23T11:19:59.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.662+0000 7f5faf200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00cd140 con 0x7f5fb00b2820 2024-03-23T11:19:59.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.663+0000 7f5fad400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5fb0070090 0x7f5fb0070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.663+0000 7f5fad400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5fb0070090 0x7f5fb0070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:42618/0 (socket says 172.21.15.62:42618) 2024-03-23T11:19:59.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.663+0000 7f5fad400640 1 -- 172.21.15.62:0/3224535310 learned_addr learned my addr 172.21.15.62:0/3224535310 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:59.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.663+0000 7f5fae800640 1 --2- 172.21.15.62:0/3224535310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fb00b2820 0x7f5fb00b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.664+0000 7f5fae800640 1 -- 172.21.15.62:0/3224535310 >> v2:172.21.15.120:3300/0 conn(0x7f5fb0070090 msgr2=0x7f5fb0070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:19:59.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.664+0000 7f5fae800640 1 --2- 172.21.15.62:0/3224535310 >> v2:172.21.15.120:3300/0 conn(0x7f5fb0070090 0x7f5fb0070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.664+0000 7f5fae800640 1 -- 172.21.15.62:0/3224535310 >> v1:172.21.15.120:6789/0 conn(0x7f5fb0143dd0 legacy=0x7f5fb015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.664+0000 7f5fae800640 1 -- 172.21.15.62:0/3224535310 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb00cd3e0 con 0x7f5fb00b2820 2024-03-23T11:19:59.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.666+0000 7f5fae800640 1 --2- 172.21.15.62:0/3224535310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fb00b2820 0x7f5fb00b2c70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009d90 tx=0x7f5fa802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ff2d4219ec23e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:59.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.666+0000 7f5faca00640 1 -- 172.21.15.62:0/3224535310 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa802cc50 con 0x7f5fb00b2820 2024-03-23T11:19:59.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.666+0000 7f5faca00640 1 -- 172.21.15.62:0/3224535310 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa802cdb0 con 0x7f5fb00b2820 2024-03-23T11:19:59.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.666+0000 7f5faca00640 1 -- 172.21.15.62:0/3224535310 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8034660 con 0x7f5fb00b2820 2024-03-23T11:19:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.667+0000 7f5faf200640 1 -- 172.21.15.62:0/3224535310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fb00b2820 msgr2=0x7f5fb00b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.667+0000 7f5faf200640 1 --2- 172.21.15.62:0/3224535310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fb00b2820 0x7f5fb00b2c70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009d90 tx=0x7f5fa802bea0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.667+0000 7f5faf200640 1 -- 172.21.15.62:0/3224535310 shutdown_connections 2024-03-23T11:19:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.667+0000 7f5faf200640 1 --2- 172.21.15.62:0/3224535310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fb00b2820 0x7f5fb00b2c70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.667+0000 7f5faf200640 1 --2- 172.21.15.62:0/3224535310 >> v2:172.21.15.120:3300/0 conn(0x7f5fb0070090 0x7f5fb0070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.667+0000 7f5faf200640 1 -- 172.21.15.62:0/3224535310 >> 172.21.15.62:0/3224535310 conn(0x7f5fb0066050 msgr2=0x7f5fb0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.667+0000 7f5faf200640 1 -- 172.21.15.62:0/3224535310 shutdown_connections 2024-03-23T11:19:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.668+0000 7f5faf200640 1 -- 172.21.15.62:0/3224535310 wait complete. 2024-03-23T11:19:59.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.670+0000 7f5faf200640 1 Processor -- start 2024-03-23T11:19:59.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.670+0000 7f5faf200640 1 -- start start 2024-03-23T11:19:59.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.671+0000 7f5faf200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb0070090 0x7f5fb00c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.671+0000 7f5faf200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb00b2820 0x7f5fb00c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.671+0000 7f5faf200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fb0143dd0 0x7f5fb013dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.671+0000 7f5faf200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00c6640 con 0x7f5fb00b2820 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.671+0000 7f5faf200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00c4210 con 0x7f5fb0070090 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.671+0000 7f5faf200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00c4440 con 0x7f5fb0143dd0 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.672+0000 7f5fad400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb00b2820 0x7f5fb00c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.672+0000 7f5fad400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb00b2820 0x7f5fb00c3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59534/0 (socket says 172.21.15.62:59534) 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.672+0000 7f5fad400640 1 -- 172.21.15.62:0/1615178714 learned_addr learned my addr 172.21.15.62:0/1615178714 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.672+0000 7f5fade00640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb0070090 0x7f5fb00c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.672+0000 7f5fae800640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fb0143dd0 0x7f5fb013dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.672+0000 7f5fad400640 1 -- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fb0143dd0 msgr2=0x7f5fb013dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.672+0000 7f5fad400640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fb0143dd0 0x7f5fb013dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.672+0000 7f5fad400640 1 -- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb0070090 msgr2=0x7f5fb00c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.672+0000 7f5fad400640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb0070090 0x7f5fb00c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.672+0000 7f5fad400640 1 -- 172.21.15.62:0/1615178714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa8009a70 con 0x7f5fb00b2820 2024-03-23T11:19:59.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.672+0000 7f5fade00640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb0070090 0x7f5fb00c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:19:59.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.673+0000 7f5fad400640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb00b2820 0x7f5fb00c3cd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c00e840 tx=0x7f5f9c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:59.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.673+0000 7f5fa6000640 1 -- 172.21.15.62:0/1615178714 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c00c8e0 con 0x7f5fb00b2820 2024-03-23T11:19:59.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.673+0000 7f5fa6000640 1 -- 172.21.15.62:0/1615178714 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9c00ca40 con 0x7f5fb00b2820 2024-03-23T11:19:59.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.673+0000 7f5fa6000640 1 -- 172.21.15.62:0/1615178714 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c00cdf0 con 0x7f5fb00b2820 2024-03-23T11:19:59.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.673+0000 7f5faf200640 1 -- 172.21.15.62:0/1615178714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb00c4720 con 0x7f5fb00b2820 2024-03-23T11:19:59.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.673+0000 7f5faf200640 1 -- 172.21.15.62:0/1615178714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb00c4ce0 con 0x7f5fb00b2820 2024-03-23T11:19:59.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.675+0000 7f5faf200640 1 -- 172.21.15.62:0/1615178714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb0066bb0 con 0x7f5fb00b2820 2024-03-23T11:19:59.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.675+0000 7f5fa6000640 1 -- 172.21.15.62:0/1615178714 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5f9c013470 con 0x7f5fb00b2820 2024-03-23T11:19:59.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.676+0000 7f5fa6000640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f9006a7a0 0x7f5f9006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:19:59.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.676+0000 7f5fa6000640 1 -- 172.21.15.62:0/1615178714 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5f9c088ff0 con 0x7f5fb00b2820 2024-03-23T11:19:59.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.677+0000 7f5fade00640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f9006a7a0 0x7f5f9006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:19:59.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.678+0000 7f5fade00640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f9006a7a0 0x7f5f9006cc60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0001700 tx=0x7f5fa0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:19:59.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.681+0000 7f5fa6000640 1 -- 172.21.15.62:0/1615178714 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5f9c0542c0 con 0x7f5fb00b2820 2024-03-23T11:19:59.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.881+0000 7f5faf200640 1 -- 172.21.15.62:0/1615178714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5fb00cb590 con 0x7f5fb00b2820 2024-03-23T11:19:59.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.882+0000 7f5fa6000640 1 -- 172.21.15.62:0/1615178714 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5f9c017020 con 0x7f5fb00b2820 2024-03-23T11:19:59.882 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:19:59.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.884+0000 7f5faf200640 1 -- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f9006a7a0 msgr2=0x7f5f9006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.884+0000 7f5faf200640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f9006a7a0 0x7f5f9006cc60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0001700 tx=0x7f5fa0009040 comp rx=0 tx=0).stop 2024-03-23T11:19:59.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.884+0000 7f5faf200640 1 -- 172.21.15.62:0/1615178714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb00b2820 msgr2=0x7f5fb00c3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:19:59.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.884+0000 7f5faf200640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb00b2820 0x7f5fb00c3cd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c00e840 tx=0x7f5f9c00ed00 comp rx=0 tx=0).stop 2024-03-23T11:19:59.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.884+0000 7f5faf200640 1 -- 172.21.15.62:0/1615178714 shutdown_connections 2024-03-23T11:19:59.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.884+0000 7f5faf200640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f9006a7a0 0x7f5f9006cc60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.884+0000 7f5faf200640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb00b2820 0x7f5fb00c3cd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.884+0000 7f5faf200640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fb0143dd0 0x7f5fb013dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.884+0000 7f5faf200640 1 --2- 172.21.15.62:0/1615178714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fb0070090 0x7f5fb00c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:19:59.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.884+0000 7f5faf200640 1 -- 172.21.15.62:0/1615178714 >> 172.21.15.62:0/1615178714 conn(0x7f5fb0066050 msgr2=0x7f5fb00adde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:19:59.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.885+0000 7f5faf200640 1 -- 172.21.15.62:0/1615178714 shutdown_connections 2024-03-23T11:19:59.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:19:59.885+0000 7f5faf200640 1 -- 172.21.15.62:0/1615178714 wait complete. 2024-03-23T11:19:59.904 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:20:00.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.129+0000 7f8027200640 1 Processor -- start 2024-03-23T11:20:00.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.130+0000 7f8027200640 1 -- start start 2024-03-23T11:20:00.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.131+0000 7f8027200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8028160420 0x7f8028160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:00.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.131+0000 7f8027200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f80280b5db0 0x7f80280b6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:00.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.131+0000 7f8027200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8028160dc0 con 0x7f802815fcf0 2024-03-23T11:20:00.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.131+0000 7f8027200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80280ce510 con 0x7f80280b5db0 2024-03-23T11:20:00.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.131+0000 7f8027200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80280cdc70 con 0x7f8028160420 2024-03-23T11:20:00.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.132+0000 7f8026800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f80280b5db0 0x7f80280b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:00.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.132+0000 7f8026800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f80280b5db0 0x7f80280b6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:42636/0 (socket says 172.21.15.62:42636) 2024-03-23T11:20:00.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.132+0000 7f8026800640 1 -- 172.21.15.62:0/3653765146 learned_addr learned my addr 172.21.15.62:0/3653765146 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:00.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.132+0000 7f801fe00640 1 --2- 172.21.15.62:0/3653765146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8028160420 0x7f8028160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:00.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.133+0000 7f8026800640 1 -- 172.21.15.62:0/3653765146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8028160420 msgr2=0x7f8028160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:00.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.133+0000 7f8026800640 1 --2- 172.21.15.62:0/3653765146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8028160420 0x7f8028160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.133+0000 7f8026800640 1 -- 172.21.15.62:0/3653765146 >> v1:172.21.15.120:6789/0 conn(0x7f802815fcf0 legacy=0x7f80280d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.133+0000 7f8026800640 1 -- 172.21.15.62:0/3653765146 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80280cdf50 con 0x7f80280b5db0 2024-03-23T11:20:00.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.135+0000 7f8026800640 1 --2- 172.21.15.62:0/3653765146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80280b5db0 0x7f80280b6220 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8020009e50 tx=0x7f802002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bc87df5c069b9ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:00.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.139+0000 7f8025400640 1 -- 172.21.15.62:0/3653765146 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802002ca70 con 0x7f80280b5db0 2024-03-23T11:20:00.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.139+0000 7f8025400640 1 -- 172.21.15.62:0/3653765146 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802002cbd0 con 0x7f80280b5db0 2024-03-23T11:20:00.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.139+0000 7f8025400640 1 -- 172.21.15.62:0/3653765146 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80200343f0 con 0x7f80280b5db0 2024-03-23T11:20:00.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.140+0000 7f8027200640 1 -- 172.21.15.62:0/3653765146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80280b5db0 msgr2=0x7f80280b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.140+0000 7f8027200640 1 --2- 172.21.15.62:0/3653765146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80280b5db0 0x7f80280b6220 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8020009e50 tx=0x7f802002bed0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.140+0000 7f8027200640 1 -- 172.21.15.62:0/3653765146 shutdown_connections 2024-03-23T11:20:00.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.140+0000 7f8027200640 1 --2- 172.21.15.62:0/3653765146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80280b5db0 0x7f80280b6220 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.140+0000 7f8027200640 1 --2- 172.21.15.62:0/3653765146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8028160420 0x7f8028160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.140+0000 7f8027200640 1 -- 172.21.15.62:0/3653765146 >> 172.21.15.62:0/3653765146 conn(0x7f8028089f60 msgr2=0x7f80280d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:00.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.140+0000 7f8027200640 1 -- 172.21.15.62:0/3653765146 shutdown_connections 2024-03-23T11:20:00.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.140+0000 7f8027200640 1 -- 172.21.15.62:0/3653765146 wait complete. 2024-03-23T11:20:00.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.142+0000 7f8027200640 1 Processor -- start 2024-03-23T11:20:00.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.142+0000 7f8027200640 1 -- start start 2024-03-23T11:20:00.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.143+0000 7f8027200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80280b5db0 0x7f8028150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:00.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.143+0000 7f8027200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f802815fcf0 0x7f802814d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:00.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.143+0000 7f8027200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8028160420 0x7f802815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:00.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.143+0000 7f8027200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80280c8760 con 0x7f802815fcf0 2024-03-23T11:20:00.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.143+0000 7f8027200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802809c880 con 0x7f80280b5db0 2024-03-23T11:20:00.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.143+0000 7f8027200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802814cc20 con 0x7f8028160420 2024-03-23T11:20:00.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.143+0000 7f801fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f802815fcf0 0x7f802814d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.144+0000 7f801fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f802815fcf0 0x7f802814d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59548/0 (socket says 172.21.15.62:59548) 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.144+0000 7f801fe00640 1 -- 172.21.15.62:0/217962412 learned_addr learned my addr 172.21.15.62:0/217962412 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.144+0000 7f8026800640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8028160420 0x7f802815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.144+0000 7f8025e00640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80280b5db0 0x7f8028150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.144+0000 7f801fe00640 1 -- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8028160420 msgr2=0x7f802815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.144+0000 7f801fe00640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8028160420 0x7f802815c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.144+0000 7f801fe00640 1 -- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80280b5db0 msgr2=0x7f8028150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.144+0000 7f801fe00640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80280b5db0 0x7f8028150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.144+0000 7f801fe00640 1 -- 172.21.15.62:0/217962412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8020009b30 con 0x7f802815fcf0 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.144+0000 7f801fe00640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f802815fcf0 0x7f802814d0b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f801000e8a0 tx=0x7f801000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.145+0000 7f801e000640 1 -- 172.21.15.62:0/217962412 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80100094f0 con 0x7f802815fcf0 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.145+0000 7f801e000640 1 -- 172.21.15.62:0/217962412 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801000c940 con 0x7f802815fcf0 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.145+0000 7f801e000640 1 -- 172.21.15.62:0/217962412 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801000ccf0 con 0x7f802815fcf0 2024-03-23T11:20:00.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.145+0000 7f8027200640 1 -- 172.21.15.62:0/217962412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80280c8970 con 0x7f802815fcf0 2024-03-23T11:20:00.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.145+0000 7f8027200640 1 -- 172.21.15.62:0/217962412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802814c5d0 con 0x7f802815fcf0 2024-03-23T11:20:00.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.146+0000 7f8027200640 1 -- 172.21.15.62:0/217962412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8028066bb0 con 0x7f802815fcf0 2024-03-23T11:20:00.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.147+0000 7f801e000640 1 -- 172.21.15.62:0/217962412 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8010013470 con 0x7f802815fcf0 2024-03-23T11:20:00.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.148+0000 7f801e000640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f800406a7d0 0x7f800406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:00.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.148+0000 7f801e000640 1 -- 172.21.15.62:0/217962412 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8010088f30 con 0x7f802815fcf0 2024-03-23T11:20:00.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.149+0000 7f8025e00640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f800406a7d0 0x7f800406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:00.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.150+0000 7f8025e00640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f800406a7d0 0x7f800406cc90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f80281f5810 tx=0x7f8018009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:00.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.153+0000 7f801e000640 1 -- 172.21.15.62:0/217962412 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8010054230 con 0x7f802815fcf0 2024-03-23T11:20:00.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.355+0000 7f8027200640 1 -- 172.21.15.62:0/217962412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f802814cd60 con 0x7f802815fcf0 2024-03-23T11:20:00.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.356+0000 7f801e000640 1 -- 172.21.15.62:0/217962412 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8010057850 con 0x7f802815fcf0 2024-03-23T11:20:00.357 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:20:00.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 -- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f800406a7d0 msgr2=0x7f800406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f800406a7d0 0x7f800406cc90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f80281f5810 tx=0x7f8018009040 comp rx=0 tx=0).stop 2024-03-23T11:20:00.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 -- 172.21.15.62:0/217962412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f802815fcf0 msgr2=0x7f802814d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f802815fcf0 0x7f802814d0b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f801000e8a0 tx=0x7f801000ed60 comp rx=0 tx=0).stop 2024-03-23T11:20:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 -- 172.21.15.62:0/217962412 shutdown_connections 2024-03-23T11:20:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f800406a7d0 0x7f800406cc90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f802815fcf0 0x7f802814d0b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8028160420 0x7f802815c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 --2- 172.21.15.62:0/217962412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80280b5db0 0x7f8028150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 -- 172.21.15.62:0/217962412 >> 172.21.15.62:0/217962412 conn(0x7f8028089f60 msgr2=0x7f80280cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 -- 172.21.15.62:0/217962412 shutdown_connections 2024-03-23T11:20:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.359+0000 7f8027200640 1 -- 172.21.15.62:0/217962412 wait complete. 2024-03-23T11:20:00.380 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:20:00.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.606+0000 7f3251400640 1 Processor -- start 2024-03-23T11:20:00.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.606+0000 7f3251400640 1 -- start start 2024-03-23T11:20:00.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.608+0000 7f3251400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f324c0caff0 0x7f324c0cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:00.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.608+0000 7f3251400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f324c15bb10 con 0x7f324c143450 2024-03-23T11:20:00.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.608+0000 7f3251400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f324c0cb9e0 con 0x7f324c069c60 2024-03-23T11:20:00.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.608+0000 7f3251400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f324c0ce1a0 con 0x7f324c0caff0 2024-03-23T11:20:00.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.608+0000 7f3250a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f324c143450 0x7f324c1438c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50378/0 (socket says 172.21.15.62:50378) 2024-03-23T11:20:00.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.608+0000 7f3250a00640 1 -- 172.21.15.62:0/939538079 learned_addr learned my addr 172.21.15.62:0/939538079 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:00.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.608+0000 7f324b400640 1 --2- 172.21.15.62:0/939538079 >> v2:172.21.15.62:3300/0 conn(0x7f324c0caff0 0x7f324c0cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:00.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.608+0000 7f324aa00640 1 -- 172.21.15.62:0/939538079 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659288895 0 0) 0x7f324c15bb10 con 0x7f324c143450 2024-03-23T11:20:00.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.609+0000 7f324aa00640 1 -- 172.21.15.62:0/939538079 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3230003520 con 0x7f324c143450 2024-03-23T11:20:00.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.609+0000 7f324b400640 1 -- 172.21.15.62:0/939538079 >> v1:172.21.15.120:6789/0 conn(0x7f324c069c60 legacy=0x7f324c06a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:00.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.609+0000 7f324b400640 1 -- 172.21.15.62:0/939538079 >> v1:172.21.15.62:6789/0 conn(0x7f324c143450 legacy=0x7f324c1438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.609+0000 7f324b400640 1 -- 172.21.15.62:0/939538079 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324c15ad50 con 0x7f324c0caff0 2024-03-23T11:20:00.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.610+0000 7f324b400640 1 --2- 172.21.15.62:0/939538079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f324c0caff0 0x7f324c0cb4a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f32340097a0 tx=0x7f323402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e37c88a67c8425 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:00.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.610+0000 7f324aa00640 1 -- 172.21.15.62:0/939538079 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3234031830 con 0x7f324c0caff0 2024-03-23T11:20:00.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.610+0000 7f324aa00640 1 -- 172.21.15.62:0/939538079 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3234031990 con 0x7f324c0caff0 2024-03-23T11:20:00.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.610+0000 7f324aa00640 1 -- 172.21.15.62:0/939538079 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3234031d20 con 0x7f324c0caff0 2024-03-23T11:20:00.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.610+0000 7f3251400640 1 -- 172.21.15.62:0/939538079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f324c0caff0 msgr2=0x7f324c0cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.610+0000 7f3251400640 1 --2- 172.21.15.62:0/939538079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f324c0caff0 0x7f324c0cb4a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f32340097a0 tx=0x7f323402b8d0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.611+0000 7f3251400640 1 -- 172.21.15.62:0/939538079 shutdown_connections 2024-03-23T11:20:00.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.611+0000 7f3251400640 1 --2- 172.21.15.62:0/939538079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f324c0caff0 0x7f324c0cb4a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.611+0000 7f3251400640 1 -- 172.21.15.62:0/939538079 >> 172.21.15.62:0/939538079 conn(0x7f324c0b0090 msgr2=0x7f324c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:00.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.611+0000 7f3251400640 1 -- 172.21.15.62:0/939538079 shutdown_connections 2024-03-23T11:20:00.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.611+0000 7f3251400640 1 -- 172.21.15.62:0/939538079 wait complete. 2024-03-23T11:20:00.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.612+0000 7f3251400640 1 Processor -- start 2024-03-23T11:20:00.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.613+0000 7f3251400640 1 -- start start 2024-03-23T11:20:00.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f3251400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f324c069c60 0x7f324c158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:00.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f3251400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f324c0caff0 0x7f324c13fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:00.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f3251400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f324c143450 0x7f324c13f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:00.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f3251400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c0cbcb0 con 0x7f324c069c60 2024-03-23T11:20:00.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f3251400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c140ea0 con 0x7f324c0caff0 2024-03-23T11:20:00.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f3251400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324c159630 con 0x7f324c143450 2024-03-23T11:20:00.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f324be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f324c069c60 0x7f324c158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:00.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f324be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f324c069c60 0x7f324c158650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59560/0 (socket says 172.21.15.62:59560) 2024-03-23T11:20:00.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f324be00640 1 -- 172.21.15.62:0/304281812 learned_addr learned my addr 172.21.15.62:0/304281812 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f324be00640 1 -- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f324c143450 msgr2=0x7f324c13f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f324be00640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f324c143450 0x7f324c13f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f324b400640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f324c0caff0 0x7f324c13fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f324be00640 1 -- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f324c0caff0 msgr2=0x7f324c13fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f324be00640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f324c0caff0 0x7f324c13fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.614+0000 7f324be00640 1 -- 172.21.15.62:0/304281812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234009400 con 0x7f324c069c60 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.615+0000 7f324be00640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f324c069c60 0x7f324c158650 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f323c00b420 tx=0x7f323c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.615+0000 7f3243e00640 1 -- 172.21.15.62:0/304281812 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c011830 con 0x7f324c069c60 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.615+0000 7f3243e00640 1 -- 172.21.15.62:0/304281812 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323c011990 con 0x7f324c069c60 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.615+0000 7f3251400640 1 -- 172.21.15.62:0/304281812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324c140230 con 0x7f324c069c60 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.615+0000 7f3243e00640 1 -- 172.21.15.62:0/304281812 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c011d40 con 0x7f324c069c60 2024-03-23T11:20:00.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.615+0000 7f3251400640 1 -- 172.21.15.62:0/304281812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324c1413d0 con 0x7f324c069c60 2024-03-23T11:20:00.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.616+0000 7f3251400640 1 -- 172.21.15.62:0/304281812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324c066bb0 con 0x7f324c069c60 2024-03-23T11:20:00.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.617+0000 7f3243e00640 1 -- 172.21.15.62:0/304281812 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f323c01b460 con 0x7f324c069c60 2024-03-23T11:20:00.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.618+0000 7f3243e00640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f322806a7d0 0x7f322806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:00.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.618+0000 7f3243e00640 1 -- 172.21.15.62:0/304281812 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f323c088940 con 0x7f324c069c60 2024-03-23T11:20:00.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.619+0000 7f324b400640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f322806a7d0 0x7f322806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:00.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.620+0000 7f324b400640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f322806a7d0 0x7f322806cc90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3234036000 tx=0x7f3234008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:00.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.623+0000 7f3243e00640 1 -- 172.21.15.62:0/304281812 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f323c053cd0 con 0x7f324c069c60 2024-03-23T11:20:00.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.832+0000 7f3251400640 1 -- 172.21.15.62:0/304281812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f324c15dad0 con 0x7f324c069c60 2024-03-23T11:20:00.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.833+0000 7f3243e00640 1 -- 172.21.15.62:0/304281812 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f323c088020 con 0x7f324c069c60 2024-03-23T11:20:00.833 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:20:00.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.835+0000 7f3251400640 1 -- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f322806a7d0 msgr2=0x7f322806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.835+0000 7f3251400640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f322806a7d0 0x7f322806cc90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3234036000 tx=0x7f3234008030 comp rx=0 tx=0).stop 2024-03-23T11:20:00.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.835+0000 7f3251400640 1 -- 172.21.15.62:0/304281812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f324c069c60 msgr2=0x7f324c158650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:00.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.835+0000 7f3251400640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f324c069c60 0x7f324c158650 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f323c00b420 tx=0x7f323c00b8e0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.836+0000 7f3251400640 1 -- 172.21.15.62:0/304281812 shutdown_connections 2024-03-23T11:20:00.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.836+0000 7f3251400640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f322806a7d0 0x7f322806cc90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.836+0000 7f3251400640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f324c143450 0x7f324c13f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.836+0000 7f3251400640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f324c0caff0 0x7f324c13fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.836+0000 7f3251400640 1 --2- 172.21.15.62:0/304281812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f324c069c60 0x7f324c158650 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:00.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.836+0000 7f3251400640 1 -- 172.21.15.62:0/304281812 >> 172.21.15.62:0/304281812 conn(0x7f324c0b0090 msgr2=0x7f324c0ca8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:00.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.836+0000 7f3251400640 1 -- 172.21.15.62:0/304281812 shutdown_connections 2024-03-23T11:20:00.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:00.836+0000 7f3251400640 1 -- 172.21.15.62:0/304281812 wait complete. 2024-03-23T11:20:00.855 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi120.front.sepia.ceph.com... 2024-03-23T11:20:00.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:00.855 DEBUG:teuthology.orchestra.run.smithi120:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-23T11:20:00.956 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:20:00.957 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:20:00.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:00.957 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T11:20:01.001 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T11:20:01.001 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:20:01.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.225+0000 7f8c77800640 1 Processor -- start 2024-03-23T11:20:01.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.226+0000 7f8c77800640 1 -- start start 2024-03-23T11:20:01.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.227+0000 7f8c77800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8c7006c300 0x7f8c700ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.227+0000 7f8c77800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8c7006c6e0 0x7f8c7006cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.227+0000 7f8c77800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c700cbc60 con 0x7f8c700cafa0 2024-03-23T11:20:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.227+0000 7f8c77800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c700cbf40 con 0x7f8c7006c6e0 2024-03-23T11:20:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.227+0000 7f8c77800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c700cc220 con 0x7f8c7006c300 2024-03-23T11:20:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.228+0000 7f8c75800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8c7006c6e0 0x7f8c7006cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.228+0000 7f8c75800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8c7006c6e0 0x7f8c7006cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59570/0 (socket says 172.21.15.62:59570) 2024-03-23T11:20:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.228+0000 7f8c75800640 1 -- 172.21.15.62:0/2474203586 learned_addr learned my addr 172.21.15.62:0/2474203586 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.228+0000 7f8c74e00640 1 --2- 172.21.15.62:0/2474203586 >> v2:172.21.15.120:3300/0 conn(0x7f8c7006c300 0x7f8c700ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.228+0000 7f8c75800640 1 -- 172.21.15.62:0/2474203586 >> v2:172.21.15.120:3300/0 conn(0x7f8c7006c300 msgr2=0x7f8c700ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.228+0000 7f8c75800640 1 --2- 172.21.15.62:0/2474203586 >> v2:172.21.15.120:3300/0 conn(0x7f8c7006c300 0x7f8c700ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.228+0000 7f8c75800640 1 -- 172.21.15.62:0/2474203586 >> v1:172.21.15.120:6789/0 conn(0x7f8c700cafa0 legacy=0x7f8c700cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.228+0000 7f8c75800640 1 -- 172.21.15.62:0/2474203586 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c700cc500 con 0x7f8c7006c6e0 2024-03-23T11:20:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.229+0000 7f8c75800640 1 --2- 172.21.15.62:0/2474203586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7006c6e0 0x7f8c7006cb30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8c7006c1b0 tx=0x7f8c6802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4079a38f61dcbd11 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.230+0000 7f8c6f400640 1 -- 172.21.15.62:0/2474203586 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6802cb20 con 0x7f8c7006c6e0 2024-03-23T11:20:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.230+0000 7f8c6f400640 1 -- 172.21.15.62:0/2474203586 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6802cc80 con 0x7f8c7006c6e0 2024-03-23T11:20:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.230+0000 7f8c6f400640 1 -- 172.21.15.62:0/2474203586 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c680343f0 con 0x7f8c7006c6e0 2024-03-23T11:20:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.230+0000 7f8c77800640 1 -- 172.21.15.62:0/2474203586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7006c6e0 msgr2=0x7f8c7006cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.230+0000 7f8c77800640 1 --2- 172.21.15.62:0/2474203586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7006c6e0 0x7f8c7006cb30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8c7006c1b0 tx=0x7f8c6802bf80 comp rx=0 tx=0).stop 2024-03-23T11:20:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.230+0000 7f8c77800640 1 -- 172.21.15.62:0/2474203586 shutdown_connections 2024-03-23T11:20:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.230+0000 7f8c77800640 1 --2- 172.21.15.62:0/2474203586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7006c6e0 0x7f8c7006cb30 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.230+0000 7f8c77800640 1 --2- 172.21.15.62:0/2474203586 >> v2:172.21.15.120:3300/0 conn(0x7f8c7006c300 0x7f8c700ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.230+0000 7f8c77800640 1 -- 172.21.15.62:0/2474203586 >> 172.21.15.62:0/2474203586 conn(0x7f8c700b02e0 msgr2=0x7f8c700b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:01.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.231+0000 7f8c77800640 1 -- 172.21.15.62:0/2474203586 shutdown_connections 2024-03-23T11:20:01.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.231+0000 7f8c77800640 1 -- 172.21.15.62:0/2474203586 wait complete. 2024-03-23T11:20:01.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.232+0000 7f8c77800640 1 Processor -- start 2024-03-23T11:20:01.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.232+0000 7f8c77800640 1 -- start start 2024-03-23T11:20:01.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.233+0000 7f8c77800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7006c300 0x7f8c700ad000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.233+0000 7f8c77800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c7006c6e0 0x7f8c700ad540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.233+0000 7f8c77800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c700cafa0 0x7f8c700abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.233+0000 7f8c77800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7015e040 con 0x7f8c7006c300 2024-03-23T11:20:01.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.233+0000 7f8c77800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c700b1af0 con 0x7f8c700cafa0 2024-03-23T11:20:01.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.233+0000 7f8c77800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7015d400 con 0x7f8c7006c6e0 2024-03-23T11:20:01.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.233+0000 7f8c74e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7006c300 0x7f8c700ad000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c74e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7006c300 0x7f8c700ad000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59572/0 (socket says 172.21.15.62:59572) 2024-03-23T11:20:01.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c74e00640 1 -- 172.21.15.62:0/2715334081 learned_addr learned my addr 172.21.15.62:0/2715334081 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:01.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c74e00640 1 -- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c7006c6e0 msgr2=0x7f8c700ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c74e00640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c7006c6e0 0x7f8c700ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c74e00640 1 -- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c700cafa0 msgr2=0x7f8c700abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c74e00640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c700cafa0 0x7f8c700abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c74e00640 1 -- 172.21.15.62:0/2715334081 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c68009b30 con 0x7f8c7006c300 2024-03-23T11:20:01.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c74e00640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7006c300 0x7f8c700ad000 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400d800 tx=0x7f8c6400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:01.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c6cc00640 1 -- 172.21.15.62:0/2715334081 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6400b6c0 con 0x7f8c7006c300 2024-03-23T11:20:01.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c6cc00640 1 -- 172.21.15.62:0/2715334081 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6400b820 con 0x7f8c7006c300 2024-03-23T11:20:01.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c6cc00640 1 -- 172.21.15.62:0/2715334081 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6400bbb0 con 0x7f8c7006c300 2024-03-23T11:20:01.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.234+0000 7f8c77800640 1 -- 172.21.15.62:0/2715334081 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7015cb60 con 0x7f8c7006c300 2024-03-23T11:20:01.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.235+0000 7f8c77800640 1 -- 172.21.15.62:0/2715334081 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7015d120 con 0x7f8c7006c300 2024-03-23T11:20:01.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.236+0000 7f8c77800640 1 -- 172.21.15.62:0/2715334081 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c70066c90 con 0x7f8c7006c300 2024-03-23T11:20:01.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.237+0000 7f8c6cc00640 1 -- 172.21.15.62:0/2715334081 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8c6400bd10 con 0x7f8c7006c300 2024-03-23T11:20:01.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.237+0000 7f8c6cc00640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5006a7a0 0x7f8c5006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.238+0000 7f8c6cc00640 1 -- 172.21.15.62:0/2715334081 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8c640893b0 con 0x7f8c7006c300 2024-03-23T11:20:01.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.238+0000 7f8c6fe00640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5006a7a0 0x7f8c5006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.239+0000 7f8c6fe00640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5006a7a0 0x7f8c5006cc60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c000f60 tx=0x7f8c5c0079b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:01.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.242+0000 7f8c6cc00640 1 -- 172.21.15.62:0/2715334081 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8c64054750 con 0x7f8c7006c300 2024-03-23T11:20:01.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.481+0000 7f8c77800640 1 -- 172.21.15.62:0/2715334081 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c7006d040 con 0x7f8c7006c300 2024-03-23T11:20:01.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.487+0000 7f8c6cc00640 1 -- 172.21.15.62:0/2715334081 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f8c64057d70 con 0x7f8c7006c300 2024-03-23T11:20:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:01.488 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/496399527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":496399527},{"type":"v1","addr":"172.21.15.62:6833","nonce":496399527}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/156755611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":156755611},{"type":"v1","addr":"172.21.15.120:6835","nonce":156755611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2391380456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2391380456},{"type":"v1","addr":"172.21.15.62:6835","nonce":2391380456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:18:56.233595+0000","modified":"2024-03-23T11:18:59.758284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/3936303103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3936303103},{"type":"v1","addr":"172.21.15.120:6837","nonce":3936303103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T11:20:01.488 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-03-23T11:20:01.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.490+0000 7f8c77800640 1 -- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5006a7a0 msgr2=0x7f8c5006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.490+0000 7f8c77800640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5006a7a0 0x7f8c5006cc60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c000f60 tx=0x7f8c5c0079b0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.490+0000 7f8c77800640 1 -- 172.21.15.62:0/2715334081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7006c300 msgr2=0x7f8c700ad000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.490+0000 7f8c77800640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7006c300 0x7f8c700ad000 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400d800 tx=0x7f8c6400dcc0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.490+0000 7f8c77800640 1 -- 172.21.15.62:0/2715334081 shutdown_connections 2024-03-23T11:20:01.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.490+0000 7f8c77800640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5006a7a0 0x7f8c5006cc60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.490+0000 7f8c77800640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c700cafa0 0x7f8c700abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.490+0000 7f8c77800640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c7006c6e0 0x7f8c700ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.490+0000 7f8c77800640 1 --2- 172.21.15.62:0/2715334081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7006c300 0x7f8c700ad000 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.490+0000 7f8c77800640 1 -- 172.21.15.62:0/2715334081 >> 172.21.15.62:0/2715334081 conn(0x7f8c700b02e0 msgr2=0x7f8c7015a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:01.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.491+0000 7f8c77800640 1 -- 172.21.15.62:0/2715334081 shutdown_connections 2024-03-23T11:20:01.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.491+0000 7f8c77800640 1 -- 172.21.15.62:0/2715334081 wait complete. 2024-03-23T11:20:01.510 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:20:01.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.731+0000 7f9bd1400640 1 Processor -- start 2024-03-23T11:20:01.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.731+0000 7f9bd1400640 1 -- start start 2024-03-23T11:20:01.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.732+0000 7f9bd1400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9bcc0af340 0x7f9bcc0990e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.732+0000 7f9bd1400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0cc860 con 0x7f9bcc0ce8d0 2024-03-23T11:20:01.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.732+0000 7f9bd1400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0cbfc0 con 0x7f9bcc0ae880 2024-03-23T11:20:01.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.732+0000 7f9bd1400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bcc0cac10 con 0x7f9bcc0af340 2024-03-23T11:20:01.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.733+0000 7f9bcbe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9bcc0ce8d0 0x7f9bcc0cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50394/0 (socket says 172.21.15.62:50394) 2024-03-23T11:20:01.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.733+0000 7f9bcbe00640 1 -- 172.21.15.62:0/2488874949 learned_addr learned my addr 172.21.15.62:0/2488874949 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:01.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.733+0000 7f9bd0a00640 1 --2- 172.21.15.62:0/2488874949 >> v2:172.21.15.120:3300/0 conn(0x7f9bcc0af340 0x7f9bcc0990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.733+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908348978 0 0) 0x7f9bcc0cc860 con 0x7f9bcc0ce8d0 2024-03-23T11:20:01.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.733+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb0003520 con 0x7f9bcc0ce8d0 2024-03-23T11:20:01.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3734409885 0 0) 0x7f9bb0003520 con 0x7f9bcc0ce8d0 2024-03-23T11:20:01.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 >> v2:172.21.15.120:3300/0 conn(0x7f9bcc0af340 msgr2=0x7f9bcc0990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:01.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 --2- 172.21.15.62:0/2488874949 >> v2:172.21.15.120:3300/0 conn(0x7f9bcc0af340 0x7f9bcc0990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 >> v1:172.21.15.120:6789/0 conn(0x7f9bcc0ae880 legacy=0x7f9bcc0989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0cc2a0 con 0x7f9bcc0ce8d0 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9bbc001670 con 0x7f9bcc0ce8d0 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 >> v1:172.21.15.62:6789/0 conn(0x7f9bcc0ce8d0 legacy=0x7f9bcc0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 --2- 172.21.15.62:0/2488874949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bb0003970 0x7f9bb0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 --2- 172.21.15.62:0/2488874949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bb00047c0 0x7f9bb0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 --2- 172.21.15.62:0/2488874949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bb0005180 0x7f9bb000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb000c0a0 con 0x7f9bb0003970 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb000c210 con 0x7f9bb00047c0 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb000c3d0 con 0x7f9bb0005180 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcbe00640 1 --2- 172.21.15.62:0/2488874949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bb0003970 0x7f9bb0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bc3e00640 1 --2- 172.21.15.62:0/2488874949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bb00047c0 0x7f9bb0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bd0a00640 1 --2- 172.21.15.62:0/2488874949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bb0005180 0x7f9bb000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcbe00640 1 -- 172.21.15.62:0/2488874949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bb0005180 msgr2=0x7f9bb000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcbe00640 1 --2- 172.21.15.62:0/2488874949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bb0005180 0x7f9bb000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcbe00640 1 -- 172.21.15.62:0/2488874949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bb00047c0 msgr2=0x7f9bb0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:01.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.734+0000 7f9bcbe00640 1 --2- 172.21.15.62:0/2488874949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bb00047c0 0x7f9bb0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.735+0000 7f9bcbe00640 1 -- 172.21.15.62:0/2488874949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb0003520 con 0x7f9bb0003970 2024-03-23T11:20:01.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.735+0000 7f9bd0a00640 1 -- 172.21.15.62:0/2488874949 reap_dead start 2024-03-23T11:20:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.736+0000 7f9bcbe00640 1 --2- 172.21.15.62:0/2488874949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bb0003970 0x7f9bb0003d90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00c240 tx=0x7f9bb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39134ac8ded936f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.736+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc005ad0 con 0x7f9bb0003970 2024-03-23T11:20:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.736+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bbc005c30 con 0x7f9bb0003970 2024-03-23T11:20:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.736+0000 7f9bcb400640 1 -- 172.21.15.62:0/2488874949 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc02cab0 con 0x7f9bb0003970 2024-03-23T11:20:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.736+0000 7f9bd1400640 1 -- 172.21.15.62:0/2488874949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bb0003970 msgr2=0x7f9bb0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.736+0000 7f9bd1400640 1 --2- 172.21.15.62:0/2488874949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bb0003970 0x7f9bb0003d90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00c240 tx=0x7f9bb00057f0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.737+0000 7f9bd1400640 1 -- 172.21.15.62:0/2488874949 shutdown_connections 2024-03-23T11:20:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.737+0000 7f9bd1400640 1 --2- 172.21.15.62:0/2488874949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bb0003970 0x7f9bb0003d90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.737+0000 7f9bd1400640 1 -- 172.21.15.62:0/2488874949 >> 172.21.15.62:0/2488874949 conn(0x7f9bcc0b1fb0 msgr2=0x7f9bcc0b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.737+0000 7f9bd1400640 1 -- 172.21.15.62:0/2488874949 shutdown_connections 2024-03-23T11:20:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.737+0000 7f9bd1400640 1 -- 172.21.15.62:0/2488874949 wait complete. 2024-03-23T11:20:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.738+0000 7f9bd1400640 1 Processor -- start 2024-03-23T11:20:01.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.739+0000 7f9bd1400640 1 -- start start 2024-03-23T11:20:01.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd1400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bb0003970 0x7f9bcc0c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd1400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bcc099490 0x7f9bcc1f55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bcc0c82c0 0x7f9bcc0c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd1400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0c7590 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd1400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc06c790 con 0x7f9bcc099490 2024-03-23T11:20:01.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd1400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0ca140 con 0x7f9bb0003970 2024-03-23T11:20:01.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bcc0c82c0 0x7f9bcc0c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bcc0c82c0 0x7f9bcc0c8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59594/0 (socket says 172.21.15.62:59594) 2024-03-23T11:20:01.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd0a00640 1 -- 172.21.15.62:0/2547428154 learned_addr learned my addr 172.21.15.62:0/2547428154 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:01.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bcbe00640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bb0003970 0x7f9bcc0c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bc3e00640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bcc099490 0x7f9bcc1f55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd0a00640 1 -- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bb0003970 msgr2=0x7f9bcc0c98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd0a00640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bb0003970 0x7f9bcc0c98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd0a00640 1 -- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bcc099490 msgr2=0x7f9bcc1f55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd0a00640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bcc099490 0x7f9bcc1f55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bd0a00640 1 -- 172.21.15.62:0/2547428154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0c77e0 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.740+0000 7f9bc3e00640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bcc099490 0x7f9bcc1f55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:01.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.741+0000 7f9bd0a00640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bcc0c82c0 0x7f9bcc0c8740 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0b30b0 tx=0x7f9bc4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:01.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.741+0000 7f9bc8c00640 1 -- 172.21.15.62:0/2547428154 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4007db0 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.741+0000 7f9bc8c00640 1 -- 172.21.15.62:0/2547428154 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc4014400 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.741+0000 7f9bc8c00640 1 -- 172.21.15.62:0/2547428154 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40147b0 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.741+0000 7f9bd1400640 1 -- 172.21.15.62:0/2547428154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bcc0c7ac0 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.741+0000 7f9bd1400640 1 -- 172.21.15.62:0/2547428154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc0c9e30 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.742+0000 7f9bd1400640 1 -- 172.21.15.62:0/2547428154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc073540 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.743+0000 7f9bc8c00640 1 -- 172.21.15.62:0/2547428154 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9bc401d3b0 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.744+0000 7f9bc8c00640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ba806a7a0 0x7f9ba806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:01.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.744+0000 7f9bc8c00640 1 -- 172.21.15.62:0/2547428154 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9bc4088cf0 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.745+0000 7f9bcbe00640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ba806a7a0 0x7f9ba806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:01.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.746+0000 7f9bcbe00640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ba806a7a0 0x7f9ba806cc60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc034000 tx=0x7f9bbc009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:01.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.748+0000 7f9bc8c00640 1 -- 172.21.15.62:0/2547428154 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9bc4053ff0 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.953+0000 7f9bd1400640 1 -- 172.21.15.62:0/2547428154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bcc0672c0 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.955+0000 7f9bc8c00640 1 -- 172.21.15.62:0/2547428154 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16226 (secure 0 0 0) 0x7f9bc4057610 con 0x7f9bcc0c82c0 2024-03-23T11:20:01.956 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:01.956 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:19:42.867981+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:20:01.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 -- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ba806a7a0 msgr2=0x7f9ba806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ba806a7a0 0x7f9ba806cc60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc034000 tx=0x7f9bbc009030 comp rx=0 tx=0).stop 2024-03-23T11:20:01.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 -- 172.21.15.62:0/2547428154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bcc0c82c0 msgr2=0x7f9bcc0c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:01.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bcc0c82c0 0x7f9bcc0c8740 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0b30b0 tx=0x7f9bc4009da0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 -- 172.21.15.62:0/2547428154 shutdown_connections 2024-03-23T11:20:01.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ba806a7a0 0x7f9ba806cc60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bcc0c82c0 0x7f9bcc0c8740 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bcc099490 0x7f9bcc1f55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 --2- 172.21.15.62:0/2547428154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bb0003970 0x7f9bcc0c98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:01.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 -- 172.21.15.62:0/2547428154 >> 172.21.15.62:0/2547428154 conn(0x7f9bcc0b1fb0 msgr2=0x7f9bcc0b1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:01.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 -- 172.21.15.62:0/2547428154 shutdown_connections 2024-03-23T11:20:01.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:01.959+0000 7f9bd1400640 1 -- 172.21.15.62:0/2547428154 wait complete. 2024-03-23T11:20:01.979 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:18:53.705929+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:18:55.252182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:20:01.980 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:20:02.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.208+0000 7fa977200640 1 Processor -- start 2024-03-23T11:20:02.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.209+0000 7fa977200640 1 -- start start 2024-03-23T11:20:02.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.210+0000 7fa977200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa97815fcf0 0x7fa9780d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.210+0000 7fa977200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa9780b6dc0 0x7fa9781608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.210+0000 7fa977200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9780d3540 con 0x7fa978160420 2024-03-23T11:20:02.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.211+0000 7fa977200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa978160df0 con 0x7fa97815fcf0 2024-03-23T11:20:02.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.211+0000 7fa977200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9780ce510 con 0x7fa9780b6dc0 2024-03-23T11:20:02.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.211+0000 7fa975e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa97815fcf0 0x7fa9780d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.211+0000 7fa975400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa978160420 0x7fa9780d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50400/0 (socket says 172.21.15.62:50400) 2024-03-23T11:20:02.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.211+0000 7fa975400640 1 -- 172.21.15.62:0/1883642473 learned_addr learned my addr 172.21.15.62:0/1883642473 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:02.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.211+0000 7fa976800640 1 --2- 172.21.15.62:0/1883642473 >> v2:172.21.15.120:3300/0 conn(0x7fa9780b6dc0 0x7fa9781608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.211+0000 7fa975e00640 1 -- 172.21.15.62:0/1883642473 >> v2:172.21.15.120:3300/0 conn(0x7fa9780b6dc0 msgr2=0x7fa9781608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:02.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.211+0000 7fa975e00640 1 --2- 172.21.15.62:0/1883642473 >> v2:172.21.15.120:3300/0 conn(0x7fa9780b6dc0 0x7fa9781608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.211+0000 7fa975e00640 1 -- 172.21.15.62:0/1883642473 >> v1:172.21.15.62:6789/0 conn(0x7fa978160420 legacy=0x7fa9780d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.212+0000 7fa975e00640 1 -- 172.21.15.62:0/1883642473 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9780cdc70 con 0x7fa97815fcf0 2024-03-23T11:20:02.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.211+0000 7fa974a00640 1 -- 172.21.15.62:0/1883642473 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432394564 0 0) 0x7fa9780d3540 con 0x7fa978160420 2024-03-23T11:20:02.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.213+0000 7fa975e00640 1 --2- 172.21.15.62:0/1883642473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa97815fcf0 0x7fa9780d3000 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa97808bb20 tx=0x7fa96802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a0d0f67f6bdad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:02.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.213+0000 7fa974a00640 1 -- 172.21.15.62:0/1883642473 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96802e070 con 0x7fa97815fcf0 2024-03-23T11:20:02.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.213+0000 7fa974a00640 1 -- 172.21.15.62:0/1883642473 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9680315a0 con 0x7fa97815fcf0 2024-03-23T11:20:02.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.213+0000 7fa974a00640 1 -- 172.21.15.62:0/1883642473 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968031930 con 0x7fa97815fcf0 2024-03-23T11:20:02.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.214+0000 7fa977200640 1 -- 172.21.15.62:0/1883642473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa97815fcf0 msgr2=0x7fa9780d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.214+0000 7fa977200640 1 --2- 172.21.15.62:0/1883642473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa97815fcf0 0x7fa9780d3000 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa97808bb20 tx=0x7fa96802b840 comp rx=0 tx=0).stop 2024-03-23T11:20:02.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.214+0000 7fa977200640 1 -- 172.21.15.62:0/1883642473 shutdown_connections 2024-03-23T11:20:02.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.214+0000 7fa977200640 1 --2- 172.21.15.62:0/1883642473 >> v2:172.21.15.120:3300/0 conn(0x7fa9780b6dc0 0x7fa9781608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.214+0000 7fa977200640 1 --2- 172.21.15.62:0/1883642473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa97815fcf0 0x7fa9780d3000 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.214+0000 7fa977200640 1 -- 172.21.15.62:0/1883642473 >> 172.21.15.62:0/1883642473 conn(0x7fa978089f60 msgr2=0x7fa9780d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:02.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.214+0000 7fa977200640 1 -- 172.21.15.62:0/1883642473 shutdown_connections 2024-03-23T11:20:02.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.214+0000 7fa977200640 1 -- 172.21.15.62:0/1883642473 wait complete. 2024-03-23T11:20:02.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.216+0000 7fa977200640 1 Processor -- start 2024-03-23T11:20:02.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.216+0000 7fa977200640 1 -- start start 2024-03-23T11:20:02.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.217+0000 7fa977200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9780b6dc0 0x7fa9780cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.217+0000 7fa977200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa97815fcf0 0x7fa9781f1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.217+0000 7fa977200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa978160420 0x7fa9781f1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.217+0000 7fa977200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780cc160 con 0x7fa978160420 2024-03-23T11:20:02.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.217+0000 7fa977200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780c94f0 con 0x7fa97815fcf0 2024-03-23T11:20:02.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.217+0000 7fa977200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780c9210 con 0x7fa9780b6dc0 2024-03-23T11:20:02.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.217+0000 7fa976800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa978160420 0x7fa9781f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.217+0000 7fa976800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa978160420 0x7fa9781f1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59606/0 (socket says 172.21.15.62:59606) 2024-03-23T11:20:02.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.217+0000 7fa976800640 1 -- 172.21.15.62:0/858468684 learned_addr learned my addr 172.21.15.62:0/858468684 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:02.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa975e00640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9780b6dc0 0x7fa9780cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa975400640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa97815fcf0 0x7fa9781f1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa976800640 1 -- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9780b6dc0 msgr2=0x7fa9780cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa976800640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9780b6dc0 0x7fa9780cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa976800640 1 -- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa97815fcf0 msgr2=0x7fa9781f1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa976800640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa97815fcf0 0x7fa9781f1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa976800640 1 -- 172.21.15.62:0/858468684 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa968009400 con 0x7fa978160420 2024-03-23T11:20:02.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa975e00640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9780b6dc0 0x7fa9780cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:02.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa976800640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa978160420 0x7fa9781f1870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa97000d800 tx=0x7fa97000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:02.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa96e000640 1 -- 172.21.15.62:0/858468684 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97000b6c0 con 0x7fa978160420 2024-03-23T11:20:02.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa96e000640 1 -- 172.21.15.62:0/858468684 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97000b840 con 0x7fa978160420 2024-03-23T11:20:02.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa96e000640 1 -- 172.21.15.62:0/858468684 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97000bbf0 con 0x7fa978160420 2024-03-23T11:20:02.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa977200640 1 -- 172.21.15.62:0/858468684 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97814d960 con 0x7fa978160420 2024-03-23T11:20:02.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.218+0000 7fa977200640 1 -- 172.21.15.62:0/858468684 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97814df20 con 0x7fa978160420 2024-03-23T11:20:02.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.220+0000 7fa977200640 1 -- 172.21.15.62:0/858468684 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa978066bb0 con 0x7fa978160420 2024-03-23T11:20:02.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.220+0000 7fa96e000640 1 -- 172.21.15.62:0/858468684 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa97000bd50 con 0x7fa978160420 2024-03-23T11:20:02.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.221+0000 7fa96e000640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa95406a7a0 0x7fa95406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.221+0000 7fa96e000640 1 -- 172.21.15.62:0/858468684 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa970089360 con 0x7fa978160420 2024-03-23T11:20:02.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.222+0000 7fa975e00640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa95406a7a0 0x7fa95406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.223+0000 7fa975e00640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa95406a7a0 0x7fa95406cc60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa968036000 tx=0x7fa96803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:02.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.226+0000 7fa96e000640 1 -- 172.21.15.62:0/858468684 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa9700546f0 con 0x7fa978160420 2024-03-23T11:20:02.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.425+0000 7fa977200640 1 -- 172.21.15.62:0/858468684 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9780b4d90 con 0x7fa978160420 2024-03-23T11:20:02.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.426+0000 7fa96e000640 1 -- 172.21.15.62:0/858468684 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa970057d10 con 0x7fa978160420 2024-03-23T11:20:02.426 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:20:02.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 -- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa95406a7a0 msgr2=0x7fa95406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa95406a7a0 0x7fa95406cc60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa968036000 tx=0x7fa96803d040 comp rx=0 tx=0).stop 2024-03-23T11:20:02.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 -- 172.21.15.62:0/858468684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa978160420 msgr2=0x7fa9781f1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa978160420 0x7fa9781f1870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa97000d800 tx=0x7fa97000dcc0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 -- 172.21.15.62:0/858468684 shutdown_connections 2024-03-23T11:20:02.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa95406a7a0 0x7fa95406cc60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa978160420 0x7fa9781f1870 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa97815fcf0 0x7fa9781f1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 --2- 172.21.15.62:0/858468684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9780b6dc0 0x7fa9780cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 -- 172.21.15.62:0/858468684 >> 172.21.15.62:0/858468684 conn(0x7fa978089f60 msgr2=0x7fa97815f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:02.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 -- 172.21.15.62:0/858468684 shutdown_connections 2024-03-23T11:20:02.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.429+0000 7fa977200640 1 -- 172.21.15.62:0/858468684 wait complete. 2024-03-23T11:20:02.450 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:20:02.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.675+0000 7fa958c00640 1 Processor -- start 2024-03-23T11:20:02.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.675+0000 7fa958c00640 1 -- start start 2024-03-23T11:20:02.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.676+0000 7fa958c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9540cf650 0x7fa9540cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.676+0000 7fa958c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95406bfc0 con 0x7fa95415cdc0 2024-03-23T11:20:02.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.677+0000 7fa958c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9540b5260 con 0x7fa95415c9e0 2024-03-23T11:20:02.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.677+0000 7fa958c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9540d0020 con 0x7fa9540cf650 2024-03-23T11:20:02.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.677+0000 7fa951400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa95415cdc0 0x7fa9540d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50410/0 (socket says 172.21.15.62:50410) 2024-03-23T11:20:02.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.677+0000 7fa951400640 1 -- 172.21.15.62:0/3244416183 learned_addr learned my addr 172.21.15.62:0/3244416183 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:02.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.677+0000 7fa952800640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9540cf650 0x7fa9540cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854727856 0 0) 0x7fa95406bfc0 con 0x7fa95415cdc0 2024-03-23T11:20:02.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa938003520 con 0x7fa95415cdc0 2024-03-23T11:20:02.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3709034431 0 0) 0x7fa938003520 con 0x7fa95415cdc0 2024-03-23T11:20:02.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9540cf650 msgr2=0x7fa9540cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:02.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9540cf650 0x7fa9540cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 >> v1:172.21.15.120:6789/0 conn(0x7fa95415c9e0 legacy=0x7fa9541607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9540d2630 con 0x7fa95415cdc0 2024-03-23T11:20:02.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fa940001670 con 0x7fa95415cdc0 2024-03-23T11:20:02.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 >> v1:172.21.15.62:6789/0 conn(0x7fa95415cdc0 legacy=0x7fa9540d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9540cf650 0x7fa9380039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa938003f70 0x7fa9380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa938004d00 0x7fa938005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9540d0020 con 0x7fa938004d00 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93800bac0 con 0x7fa938003f70 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93800bc80 con 0x7fa9540cf650 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.678+0000 7fa952800640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa938004d00 0x7fa938005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.679+0000 7fa951400640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa938003f70 0x7fa9380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.679+0000 7fa951e00640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9540cf650 0x7fa9380039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.679+0000 7fa952800640 1 -- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9540cf650 msgr2=0x7fa9380039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.679+0000 7fa952800640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9540cf650 0x7fa9380039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.679+0000 7fa952800640 1 -- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa938003f70 msgr2=0x7fa9380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.679+0000 7fa952800640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa938003f70 0x7fa9380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.679+0000 7fa952800640 1 -- 172.21.15.62:0/3244416183 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa938003520 con 0x7fa938004d00 2024-03-23T11:20:02.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.679+0000 7fa951e00640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9540cf650 0x7fa9380039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:02.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.679+0000 7fa951400640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa938003f70 0x7fa9380047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:20:02.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.680+0000 7fa952800640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa938004d00 0x7fa938005180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009ac0 tx=0x7fa9380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d99320c5313557 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:02.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.681+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c02c6b0 con 0x7fa938004d00 2024-03-23T11:20:02.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.681+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94c02c810 con 0x7fa938004d00 2024-03-23T11:20:02.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.681+0000 7fa950a00640 1 -- 172.21.15.62:0/3244416183 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c02cba0 con 0x7fa938004d00 2024-03-23T11:20:02.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.681+0000 7fa958c00640 1 -- 172.21.15.62:0/3244416183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa938004d00 msgr2=0x7fa938005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.681+0000 7fa958c00640 1 --2- 172.21.15.62:0/3244416183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa938004d00 0x7fa938005180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009ac0 tx=0x7fa9380057f0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.681+0000 7fa952800640 1 -- 172.21.15.62:0/3244416183 reap_dead start 2024-03-23T11:20:02.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.681+0000 7fa958c00640 1 -- 172.21.15.62:0/3244416183 shutdown_connections 2024-03-23T11:20:02.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.681+0000 7fa958c00640 1 -- 172.21.15.62:0/3244416183 >> 172.21.15.62:0/3244416183 conn(0x7fa954071570 msgr2=0x7fa954071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:02.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.681+0000 7fa958c00640 1 -- 172.21.15.62:0/3244416183 shutdown_connections 2024-03-23T11:20:02.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.681+0000 7fa958c00640 1 -- 172.21.15.62:0/3244416183 wait complete. 2024-03-23T11:20:02.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.683+0000 7fa958c00640 1 Processor -- start 2024-03-23T11:20:02.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.683+0000 7fa958c00640 1 -- start start 2024-03-23T11:20:02.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.684+0000 7fa958c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa954156fe0 0x7fa954158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.684+0000 7fa958c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9541573c0 0x7fa95415aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.684+0000 7fa958c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa954157840 0x7fa95415af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.684+0000 7fa958c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9541f6680 con 0x7fa954157840 2024-03-23T11:20:02.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.684+0000 7fa958c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9541f58c0 con 0x7fa9541573c0 2024-03-23T11:20:02.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.684+0000 7fa958c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9541f5ba0 con 0x7fa954156fe0 2024-03-23T11:20:02.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.684+0000 7fa952800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa954157840 0x7fa95415af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.684+0000 7fa952800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa954157840 0x7fa95415af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59616/0 (socket says 172.21.15.62:59616) 2024-03-23T11:20:02.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.684+0000 7fa952800640 1 -- 172.21.15.62:0/2578719263 learned_addr learned my addr 172.21.15.62:0/2578719263 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:02.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.685+0000 7fa951e00640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa954156fe0 0x7fa954158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.685+0000 7fa951400640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9541573c0 0x7fa95415aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.685+0000 7fa952800640 1 -- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa954156fe0 msgr2=0x7fa954158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.685+0000 7fa952800640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa954156fe0 0x7fa954158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.685+0000 7fa952800640 1 -- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9541573c0 msgr2=0x7fa95415aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.685+0000 7fa952800640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9541573c0 0x7fa95415aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.685+0000 7fa952800640 1 -- 172.21.15.62:0/2578719263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9541f5e80 con 0x7fa954157840 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.685+0000 7fa951400640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9541573c0 0x7fa95415aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.685+0000 7fa952800640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa954157840 0x7fa95415af60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa94c037040 tx=0x7fa94c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.685+0000 7fa94a000640 1 -- 172.21.15.62:0/2578719263 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c02c6b0 con 0x7fa954157840 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.686+0000 7fa94a000640 1 -- 172.21.15.62:0/2578719263 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94c003810 con 0x7fa954157840 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.686+0000 7fa958c00640 1 -- 172.21.15.62:0/2578719263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9541f6160 con 0x7fa954157840 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.686+0000 7fa94a000640 1 -- 172.21.15.62:0/2578719263 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c003bc0 con 0x7fa954157840 2024-03-23T11:20:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.686+0000 7fa958c00640 1 -- 172.21.15.62:0/2578719263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9540c99c0 con 0x7fa954157840 2024-03-23T11:20:02.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.687+0000 7fa958c00640 1 -- 172.21.15.62:0/2578719263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa954066c90 con 0x7fa954157840 2024-03-23T11:20:02.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.688+0000 7fa94a000640 1 -- 172.21.15.62:0/2578719263 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa94c003d20 con 0x7fa954157840 2024-03-23T11:20:02.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.689+0000 7fa94a000640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa93406a7d0 0x7fa93406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:02.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.689+0000 7fa94a000640 1 -- 172.21.15.62:0/2578719263 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa94c0aa1b0 con 0x7fa954157840 2024-03-23T11:20:02.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.689+0000 7fa951e00640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa93406a7d0 0x7fa93406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:02.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.690+0000 7fa951e00640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa93406a7d0 0x7fa93406cc90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa944007fd0 tx=0x7fa944009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:02.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.693+0000 7fa94a000640 1 -- 172.21.15.62:0/2578719263 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa94c075540 con 0x7fa954157840 2024-03-23T11:20:02.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.893+0000 7fa958c00640 1 -- 172.21.15.62:0/2578719263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa9541f6320 con 0x7fa954157840 2024-03-23T11:20:02.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.894+0000 7fa94a000640 1 -- 172.21.15.62:0/2578719263 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa94c078b60 con 0x7fa954157840 2024-03-23T11:20:02.894 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:20:02.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.896+0000 7fa958c00640 1 -- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa93406a7d0 msgr2=0x7fa93406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.896+0000 7fa958c00640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa93406a7d0 0x7fa93406cc90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa944007fd0 tx=0x7fa944009040 comp rx=0 tx=0).stop 2024-03-23T11:20:02.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.896+0000 7fa958c00640 1 -- 172.21.15.62:0/2578719263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa954157840 msgr2=0x7fa95415af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:02.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.896+0000 7fa958c00640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa954157840 0x7fa95415af60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa94c037040 tx=0x7fa94c004730 comp rx=0 tx=0).stop 2024-03-23T11:20:02.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.897+0000 7fa958c00640 1 -- 172.21.15.62:0/2578719263 shutdown_connections 2024-03-23T11:20:02.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.897+0000 7fa958c00640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa93406a7d0 0x7fa93406cc90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.897+0000 7fa958c00640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa954157840 0x7fa95415af60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.897+0000 7fa958c00640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9541573c0 0x7fa95415aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.897+0000 7fa958c00640 1 --2- 172.21.15.62:0/2578719263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa954156fe0 0x7fa954158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:02.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.897+0000 7fa958c00640 1 -- 172.21.15.62:0/2578719263 >> 172.21.15.62:0/2578719263 conn(0x7fa954071570 msgr2=0x7fa95415d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:02.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.897+0000 7fa958c00640 1 -- 172.21.15.62:0/2578719263 shutdown_connections 2024-03-23T11:20:02.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:02.897+0000 7fa958c00640 1 -- 172.21.15.62:0/2578719263 wait complete. 2024-03-23T11:20:02.915 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:20:03.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.133+0000 7fd972a00640 1 Processor -- start 2024-03-23T11:20:03.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.133+0000 7fd972a00640 1 -- start start 2024-03-23T11:20:03.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.134+0000 7fd972a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd96c160420 0x7fd96c160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.134+0000 7fd972a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd96c0b5db0 0x7fd96c0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.134+0000 7fd972a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96c160dc0 con 0x7fd96c15fcf0 2024-03-23T11:20:03.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.134+0000 7fd972a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96c0ce510 con 0x7fd96c0b5db0 2024-03-23T11:20:03.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.134+0000 7fd972a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0cdc70 con 0x7fd96c160420 2024-03-23T11:20:03.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.135+0000 7fd971600640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd96c15fcf0 0x7fd96c0d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50426/0 (socket says 172.21.15.62:50426) 2024-03-23T11:20:03.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.135+0000 7fd971600640 1 -- 172.21.15.62:0/2185033365 learned_addr learned my addr 172.21.15.62:0/2185033365 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:03.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.135+0000 7fd972000640 1 --2- 172.21.15.62:0/2185033365 >> v2:172.21.15.62:3300/0 conn(0x7fd96c0b5db0 0x7fd96c0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.135+0000 7fd970c00640 1 --2- 172.21.15.62:0/2185033365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd96c160420 0x7fd96c160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.135+0000 7fd96be00640 1 -- 172.21.15.62:0/2185033365 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445732677 0 0) 0x7fd96c160dc0 con 0x7fd96c15fcf0 2024-03-23T11:20:03.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.135+0000 7fd96be00640 1 -- 172.21.15.62:0/2185033365 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd954003520 con 0x7fd96c15fcf0 2024-03-23T11:20:03.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.135+0000 7fd972000640 1 -- 172.21.15.62:0/2185033365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd96c160420 msgr2=0x7fd96c160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:03.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.135+0000 7fd972000640 1 --2- 172.21.15.62:0/2185033365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd96c160420 0x7fd96c160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.135+0000 7fd972000640 1 -- 172.21.15.62:0/2185033365 >> v1:172.21.15.62:6789/0 conn(0x7fd96c15fcf0 legacy=0x7fd96c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.136+0000 7fd972000640 1 -- 172.21.15.62:0/2185033365 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c0cdf50 con 0x7fd96c0b5db0 2024-03-23T11:20:03.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.137+0000 7fd972000640 1 --2- 172.21.15.62:0/2185033365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd96c0b5db0 0x7fd96c0b6220 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd964009d90 tx=0x7fd96402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6acbca6327b8acb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:03.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.137+0000 7fd96be00640 1 -- 172.21.15.62:0/2185033365 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96402cbc0 con 0x7fd96c0b5db0 2024-03-23T11:20:03.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.137+0000 7fd96be00640 1 -- 172.21.15.62:0/2185033365 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96402cd20 con 0x7fd96c0b5db0 2024-03-23T11:20:03.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.137+0000 7fd96be00640 1 -- 172.21.15.62:0/2185033365 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd964034660 con 0x7fd96c0b5db0 2024-03-23T11:20:03.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.138+0000 7fd972a00640 1 -- 172.21.15.62:0/2185033365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd96c0b5db0 msgr2=0x7fd96c0b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.138+0000 7fd972a00640 1 --2- 172.21.15.62:0/2185033365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd96c0b5db0 0x7fd96c0b6220 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd964009d90 tx=0x7fd96402be10 comp rx=0 tx=0).stop 2024-03-23T11:20:03.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.138+0000 7fd972a00640 1 -- 172.21.15.62:0/2185033365 shutdown_connections 2024-03-23T11:20:03.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.138+0000 7fd972a00640 1 --2- 172.21.15.62:0/2185033365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd96c0b5db0 0x7fd96c0b6220 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.138+0000 7fd972a00640 1 --2- 172.21.15.62:0/2185033365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd96c160420 0x7fd96c160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.138+0000 7fd972a00640 1 -- 172.21.15.62:0/2185033365 >> 172.21.15.62:0/2185033365 conn(0x7fd96c089f60 msgr2=0x7fd96c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:03.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.138+0000 7fd972a00640 1 -- 172.21.15.62:0/2185033365 shutdown_connections 2024-03-23T11:20:03.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.138+0000 7fd972a00640 1 -- 172.21.15.62:0/2185033365 wait complete. 2024-03-23T11:20:03.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.140+0000 7fd972a00640 1 Processor -- start 2024-03-23T11:20:03.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.140+0000 7fd972a00640 1 -- start start 2024-03-23T11:20:03.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.141+0000 7fd972a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96c0b5db0 0x7fd96c09e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.141+0000 7fd972a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd96c15fcf0 0x7fd96c153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.141+0000 7fd972a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd96c160420 0x7fd96c06b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.141+0000 7fd972a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c09eaf0 con 0x7fd96c160420 2024-03-23T11:20:03.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.141+0000 7fd972a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c09e120 con 0x7fd96c0b5db0 2024-03-23T11:20:03.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.141+0000 7fd972a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c09d510 con 0x7fd96c15fcf0 2024-03-23T11:20:03.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.142+0000 7fd972000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd96c160420 0x7fd96c06b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.142+0000 7fd972000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd96c160420 0x7fd96c06b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59644/0 (socket says 172.21.15.62:59644) 2024-03-23T11:20:03.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.142+0000 7fd972000640 1 -- 172.21.15.62:0/2553890914 learned_addr learned my addr 172.21.15.62:0/2553890914 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:03.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.142+0000 7fd970c00640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd96c15fcf0 0x7fd96c153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.142+0000 7fd971600640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96c0b5db0 0x7fd96c09e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.142+0000 7fd972000640 1 -- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd96c15fcf0 msgr2=0x7fd96c153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.142+0000 7fd972000640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd96c15fcf0 0x7fd96c153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.142+0000 7fd972000640 1 -- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96c0b5db0 msgr2=0x7fd96c09e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.142+0000 7fd972000640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96c0b5db0 0x7fd96c09e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.142+0000 7fd972000640 1 -- 172.21.15.62:0/2553890914 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd964009a70 con 0x7fd96c160420 2024-03-23T11:20:03.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.143+0000 7fd972000640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd96c160420 0x7fd96c06b1b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd964009d90 tx=0x7fd964004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:03.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.143+0000 7fd969600640 1 -- 172.21.15.62:0/2553890914 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd964040650 con 0x7fd96c160420 2024-03-23T11:20:03.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.143+0000 7fd969600640 1 -- 172.21.15.62:0/2553890914 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96402ce90 con 0x7fd96c160420 2024-03-23T11:20:03.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.143+0000 7fd972a00640 1 -- 172.21.15.62:0/2553890914 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96c09d7f0 con 0x7fd96c160420 2024-03-23T11:20:03.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.143+0000 7fd969600640 1 -- 172.21.15.62:0/2553890914 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9640409f0 con 0x7fd96c160420 2024-03-23T11:20:03.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.143+0000 7fd972a00640 1 -- 172.21.15.62:0/2553890914 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96c09ddb0 con 0x7fd96c160420 2024-03-23T11:20:03.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.144+0000 7fd972a00640 1 -- 172.21.15.62:0/2553890914 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96c09e2b0 con 0x7fd96c160420 2024-03-23T11:20:03.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.145+0000 7fd969600640 1 -- 172.21.15.62:0/2553890914 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd964040b50 con 0x7fd96c160420 2024-03-23T11:20:03.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.146+0000 7fd969600640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd94c06a7a0 0x7fd94c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.146+0000 7fd969600640 1 -- 172.21.15.62:0/2553890914 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd9640ab990 con 0x7fd96c160420 2024-03-23T11:20:03.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.147+0000 7fd971600640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd94c06a7a0 0x7fd94c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.148+0000 7fd971600640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd94c06a7a0 0x7fd94c06cc60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd9600011e0 tx=0x7fd960009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:03.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.150+0000 7fd969600640 1 -- 172.21.15.62:0/2553890914 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd964076c90 con 0x7fd96c160420 2024-03-23T11:20:03.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.351+0000 7fd972a00640 1 -- 172.21.15.62:0/2553890914 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd96c0b5af0 con 0x7fd96c160420 2024-03-23T11:20:03.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.352+0000 7fd969600640 1 -- 172.21.15.62:0/2553890914 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd964009450 con 0x7fd96c160420 2024-03-23T11:20:03.353 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.355+0000 7fd972a00640 1 -- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd94c06a7a0 msgr2=0x7fd94c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.355+0000 7fd972a00640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd94c06a7a0 0x7fd94c06cc60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd9600011e0 tx=0x7fd960009040 comp rx=0 tx=0).stop 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.355+0000 7fd972a00640 1 -- 172.21.15.62:0/2553890914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd96c160420 msgr2=0x7fd96c06b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.355+0000 7fd972a00640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd96c160420 0x7fd96c06b1b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd964009d90 tx=0x7fd964004a40 comp rx=0 tx=0).stop 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.355+0000 7fd972a00640 1 -- 172.21.15.62:0/2553890914 shutdown_connections 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.355+0000 7fd972a00640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd94c06a7a0 0x7fd94c06cc60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.355+0000 7fd972a00640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd96c160420 0x7fd96c06b1b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.355+0000 7fd972a00640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd96c15fcf0 0x7fd96c153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.355+0000 7fd972a00640 1 --2- 172.21.15.62:0/2553890914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96c0b5db0 0x7fd96c09e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.355+0000 7fd972a00640 1 -- 172.21.15.62:0/2553890914 >> 172.21.15.62:0/2553890914 conn(0x7fd96c089f60 msgr2=0x7fd96c0c7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.356+0000 7fd972a00640 1 -- 172.21.15.62:0/2553890914 shutdown_connections 2024-03-23T11:20:03.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.356+0000 7fd972a00640 1 -- 172.21.15.62:0/2553890914 wait complete. 2024-03-23T11:20:03.374 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:20:03.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.595+0000 7f8e4ba00640 1 Processor -- start 2024-03-23T11:20:03.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.595+0000 7f8e4ba00640 1 -- start start 2024-03-23T11:20:03.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.596+0000 7f8e4ba00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8e4406c300 0x7f8e440ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.596+0000 7f8e4ba00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e440cbc60 con 0x7f8e440cafa0 2024-03-23T11:20:03.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.596+0000 7f8e4ba00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e440cbf40 con 0x7f8e4406c6e0 2024-03-23T11:20:03.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.596+0000 7f8e4ba00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e440cc220 con 0x7f8e4406c300 2024-03-23T11:20:03.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.597+0000 7f8e43e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8e440cafa0 0x7f8e440cb430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50432/0 (socket says 172.21.15.62:50432) 2024-03-23T11:20:03.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.597+0000 7f8e43e00640 1 -- 172.21.15.62:0/1680265428 learned_addr learned my addr 172.21.15.62:0/1680265428 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:03.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.597+0000 7f8e49000640 1 --2- 172.21.15.62:0/1680265428 >> v2:172.21.15.120:3300/0 conn(0x7f8e4406c300 0x7f8e440ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.597+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903031307 0 0) 0x7f8e440cbc60 con 0x7f8e440cafa0 2024-03-23T11:20:03.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.597+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e2c003520 con 0x7f8e440cafa0 2024-03-23T11:20:03.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1024795465 0 0) 0x7f8e2c003520 con 0x7f8e440cafa0 2024-03-23T11:20:03.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 >> v2:172.21.15.120:3300/0 conn(0x7f8e4406c300 msgr2=0x7f8e440ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 --2- 172.21.15.62:0/1680265428 >> v2:172.21.15.120:3300/0 conn(0x7f8e4406c300 0x7f8e440ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 >> v1:172.21.15.120:6789/0 conn(0x7f8e4406c6e0 legacy=0x7f8e4406cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e440cc500 con 0x7f8e440cafa0 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f8e30001670 con 0x7f8e440cafa0 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 >> v1:172.21.15.62:6789/0 conn(0x7f8e440cafa0 legacy=0x7f8e440cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 --2- 172.21.15.62:0/1680265428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e2c0039c0 0x7f8e2c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 --2- 172.21.15.62:0/1680265428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e2c0047c0 0x7f8e2c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 --2- 172.21.15.62:0/1680265428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 0x7f8e2c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e440cc220 con 0x7f8e2c005180 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c00c080 con 0x7f8e2c0039c0 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c00c240 con 0x7f8e2c0047c0 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.598+0000 7f8e49a00640 1 --2- 172.21.15.62:0/1680265428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 0x7f8e2c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.599+0000 7f8e49000640 1 --2- 172.21.15.62:0/1680265428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e2c0039c0 0x7f8e2c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.599+0000 7f8e43e00640 1 --2- 172.21.15.62:0/1680265428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e2c0047c0 0x7f8e2c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.599+0000 7f8e49a00640 1 -- 172.21.15.62:0/1680265428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e2c0047c0 msgr2=0x7f8e2c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.599+0000 7f8e49a00640 1 --2- 172.21.15.62:0/1680265428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e2c0047c0 0x7f8e2c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.599+0000 7f8e49a00640 1 -- 172.21.15.62:0/1680265428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e2c0039c0 msgr2=0x7f8e2c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.599+0000 7f8e49a00640 1 --2- 172.21.15.62:0/1680265428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e2c0039c0 0x7f8e2c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.599+0000 7f8e49a00640 1 -- 172.21.15.62:0/1680265428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2c003520 con 0x7f8e2c005180 2024-03-23T11:20:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.600+0000 7f8e49a00640 1 -- 172.21.15.62:0/1680265428 reap_dead start 2024-03-23T11:20:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.600+0000 7f8e49a00640 1 --2- 172.21.15.62:0/1680265428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 0x7f8e2c00b980 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c001e50 tx=0x7f8e2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500d489c70006b73 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.600+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c003b70 con 0x7f8e2c005180 2024-03-23T11:20:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.600+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c003d10 con 0x7f8e2c005180 2024-03-23T11:20:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.600+0000 7f8e43400640 1 -- 172.21.15.62:0/1680265428 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c005310 con 0x7f8e2c005180 2024-03-23T11:20:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.601+0000 7f8e4ba00640 1 -- 172.21.15.62:0/1680265428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 msgr2=0x7f8e2c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.601+0000 7f8e4ba00640 1 --2- 172.21.15.62:0/1680265428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 0x7f8e2c00b980 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c001e50 tx=0x7f8e2c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.601+0000 7f8e4ba00640 1 -- 172.21.15.62:0/1680265428 shutdown_connections 2024-03-23T11:20:03.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.601+0000 7f8e4ba00640 1 --2- 172.21.15.62:0/1680265428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 0x7f8e2c00b980 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.601+0000 7f8e4ba00640 1 -- 172.21.15.62:0/1680265428 >> 172.21.15.62:0/1680265428 conn(0x7f8e440b02e0 msgr2=0x7f8e440b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:03.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.601+0000 7f8e4ba00640 1 -- 172.21.15.62:0/1680265428 shutdown_connections 2024-03-23T11:20:03.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.601+0000 7f8e4ba00640 1 -- 172.21.15.62:0/1680265428 wait complete. 2024-03-23T11:20:03.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.603+0000 7f8e4ba00640 1 Processor -- start 2024-03-23T11:20:03.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.603+0000 7f8e4ba00640 1 -- start start 2024-03-23T11:20:03.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.604+0000 7f8e4ba00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 0x7f8e4413ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.604+0000 7f8e4ba00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e44140120 0x7f8e441405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.604+0000 7f8e4ba00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e4414e9b0 0x7f8e4414ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.604+0000 7f8e4ba00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e44152a40 con 0x7f8e2c005180 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.604+0000 7f8e4ba00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4414f370 con 0x7f8e44140120 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.604+0000 7f8e4ba00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4413dad0 con 0x7f8e4414e9b0 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.604+0000 7f8e49000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 0x7f8e4413ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.604+0000 7f8e49000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 0x7f8e4413ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59664/0 (socket says 172.21.15.62:59664) 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.604+0000 7f8e49000640 1 -- 172.21.15.62:0/3882388774 learned_addr learned my addr 172.21.15.62:0/3882388774 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e43e00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e44140120 0x7f8e441405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e49a00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e4414e9b0 0x7f8e4414ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e49000640 1 -- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e4414e9b0 msgr2=0x7f8e4414ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e49000640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e4414e9b0 0x7f8e4414ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e49000640 1 -- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e44140120 msgr2=0x7f8e441405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e49000640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e44140120 0x7f8e441405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e49000640 1 -- 172.21.15.62:0/3882388774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e44151460 con 0x7f8e2c005180 2024-03-23T11:20:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e43e00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e44140120 0x7f8e441405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e49000640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 0x7f8e4413ca40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8e38011910 tx=0x7f8e38011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e40c00640 1 -- 172.21.15.62:0/3882388774 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3800fa90 con 0x7f8e2c005180 2024-03-23T11:20:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e40c00640 1 -- 172.21.15.62:0/3882388774 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3800fbf0 con 0x7f8e2c005180 2024-03-23T11:20:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e40c00640 1 -- 172.21.15.62:0/3882388774 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38014440 con 0x7f8e2c005180 2024-03-23T11:20:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.605+0000 7f8e4ba00640 1 -- 172.21.15.62:0/3882388774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4413cf80 con 0x7f8e2c005180 2024-03-23T11:20:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.606+0000 7f8e4ba00640 1 -- 172.21.15.62:0/3882388774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e44150ea0 con 0x7f8e2c005180 2024-03-23T11:20:03.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.607+0000 7f8e4ba00640 1 -- 172.21.15.62:0/3882388774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4413d8c0 con 0x7f8e2c005180 2024-03-23T11:20:03.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.608+0000 7f8e40c00640 1 -- 172.21.15.62:0/3882388774 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8e380145a0 con 0x7f8e2c005180 2024-03-23T11:20:03.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.609+0000 7f8e40c00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e2406a7a0 0x7f8e2406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:03.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.609+0000 7f8e40c00640 1 -- 172.21.15.62:0/3882388774 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8e38089660 con 0x7f8e2c005180 2024-03-23T11:20:03.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.610+0000 7f8e43e00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e2406a7a0 0x7f8e2406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:03.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.611+0000 7f8e43e00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e2406a7a0 0x7f8e2406cc60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8e440ca830 tx=0x7f8e30000f10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:03.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.613+0000 7f8e40c00640 1 -- 172.21.15.62:0/3882388774 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8e380549f0 con 0x7f8e2c005180 2024-03-23T11:20:03.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.842+0000 7f8e4ba00640 1 -- 172.21.15.62:0/3882388774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e44066c90 con 0x7f8e2c005180 2024-03-23T11:20:03.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.843+0000 7f8e40c00640 1 -- 172.21.15.62:0/3882388774 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f8e38058010 con 0x7f8e2c005180 2024-03-23T11:20:03.843 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:03.844 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/496399527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":496399527},{"type":"v1","addr":"172.21.15.62:6833","nonce":496399527}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/156755611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":156755611},{"type":"v1","addr":"172.21.15.120:6835","nonce":156755611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2391380456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2391380456},{"type":"v1","addr":"172.21.15.62:6835","nonce":2391380456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:18:56.233595+0000","modified":"2024-03-23T11:18:59.758284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/3936303103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3936303103},{"type":"v1","addr":"172.21.15.120:6837","nonce":3936303103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T11:20:03.844 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-03-23T11:20:03.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.845+0000 7f8e4ba00640 1 -- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e2406a7a0 msgr2=0x7f8e2406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.845+0000 7f8e4ba00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e2406a7a0 0x7f8e2406cc60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8e440ca830 tx=0x7f8e30000f10 comp rx=0 tx=0).stop 2024-03-23T11:20:03.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.845+0000 7f8e4ba00640 1 -- 172.21.15.62:0/3882388774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 msgr2=0x7f8e4413ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:03.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.845+0000 7f8e4ba00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 0x7f8e4413ca40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8e38011910 tx=0x7f8e38011dd0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.845+0000 7f8e4ba00640 1 -- 172.21.15.62:0/3882388774 shutdown_connections 2024-03-23T11:20:03.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.845+0000 7f8e4ba00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e2406a7a0 0x7f8e2406cc60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.845+0000 7f8e4ba00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e4414e9b0 0x7f8e4414ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.846+0000 7f8e4ba00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e44140120 0x7f8e441405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.846+0000 7f8e4ba00640 1 --2- 172.21.15.62:0/3882388774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e2c005180 0x7f8e4413ca40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:03.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.846+0000 7f8e4ba00640 1 -- 172.21.15.62:0/3882388774 >> 172.21.15.62:0/3882388774 conn(0x7f8e440b02e0 msgr2=0x7f8e4406d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:03.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.846+0000 7f8e4ba00640 1 -- 172.21.15.62:0/3882388774 shutdown_connections 2024-03-23T11:20:03.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:03.846+0000 7f8e4ba00640 1 -- 172.21.15.62:0/3882388774 wait complete. 2024-03-23T11:20:03.862 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:20:04.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.079+0000 7fcf77400640 1 Processor -- start 2024-03-23T11:20:04.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.079+0000 7fcf77400640 1 -- start start 2024-03-23T11:20:04.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.080+0000 7fcf77400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf700cd7c0 0x7fcf700b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.080+0000 7fcf77400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf7006dca0 0x7fcf7006c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.080+0000 7fcf77400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf700cad20 con 0x7fcf700cdba0 2024-03-23T11:20:04.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.080+0000 7fcf77400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf7006e130 con 0x7fcf700cd7c0 2024-03-23T11:20:04.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.080+0000 7fcf77400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7015a8a0 con 0x7fcf7006dca0 2024-03-23T11:20:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.081+0000 7fcf74a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf700cd7c0 0x7fcf700b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.081+0000 7fcf74a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf700cd7c0 0x7fcf700b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59674/0 (socket says 172.21.15.62:59674) 2024-03-23T11:20:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.081+0000 7fcf74a00640 1 -- 172.21.15.62:0/1030084410 learned_addr learned my addr 172.21.15.62:0/1030084410 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.081+0000 7fcf75400640 1 --1- 172.21.15.62:0/1030084410 >> v1:172.21.15.62:6789/0 conn(0x7fcf700cdba0 0x7fcf700cb490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50444/0 (socket says 172.21.15.62:50444) 2024-03-23T11:20:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.081+0000 7fcf6fe00640 1 --2- 172.21.15.62:0/1030084410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf7006dca0 0x7fcf7006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.082+0000 7fcf74a00640 1 -- 172.21.15.62:0/1030084410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf7006dca0 msgr2=0x7fcf7006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.082+0000 7fcf74a00640 1 --2- 172.21.15.62:0/1030084410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf7006dca0 0x7fcf7006c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.082+0000 7fcf74a00640 1 -- 172.21.15.62:0/1030084410 >> v1:172.21.15.62:6789/0 conn(0x7fcf700cdba0 legacy=0x7fcf700cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.082+0000 7fcf6f400640 1 -- 172.21.15.62:0/1030084410 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819279389 0 0) 0x7fcf700cad20 con 0x7fcf700cdba0 2024-03-23T11:20:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.082+0000 7fcf74a00640 1 -- 172.21.15.62:0/1030084410 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7015ab80 con 0x7fcf700cd7c0 2024-03-23T11:20:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.083+0000 7fcf74a00640 1 --2- 172.21.15.62:0/1030084410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf700cd7c0 0x7fcf700b15e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcf64009870 tx=0x7fcf6402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16744f4d8233ed96 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.083+0000 7fcf6f400640 1 -- 172.21.15.62:0/1030084410 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64031640 con 0x7fcf700cd7c0 2024-03-23T11:20:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.083+0000 7fcf6f400640 1 -- 172.21.15.62:0/1030084410 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf640317a0 con 0x7fcf700cd7c0 2024-03-23T11:20:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.083+0000 7fcf6f400640 1 -- 172.21.15.62:0/1030084410 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64031b30 con 0x7fcf700cd7c0 2024-03-23T11:20:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.084+0000 7fcf77400640 1 -- 172.21.15.62:0/1030084410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf700cd7c0 msgr2=0x7fcf700b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.084+0000 7fcf77400640 1 --2- 172.21.15.62:0/1030084410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf700cd7c0 0x7fcf700b15e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcf64009870 tx=0x7fcf6402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.084+0000 7fcf77400640 1 -- 172.21.15.62:0/1030084410 shutdown_connections 2024-03-23T11:20:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.084+0000 7fcf77400640 1 --2- 172.21.15.62:0/1030084410 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf7006dca0 0x7fcf7006c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.084+0000 7fcf77400640 1 --2- 172.21.15.62:0/1030084410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf700cd7c0 0x7fcf700b15e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.084+0000 7fcf77400640 1 -- 172.21.15.62:0/1030084410 >> 172.21.15.62:0/1030084410 conn(0x7fcf700b02e0 msgr2=0x7fcf700b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:04.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.084+0000 7fcf77400640 1 -- 172.21.15.62:0/1030084410 shutdown_connections 2024-03-23T11:20:04.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.084+0000 7fcf77400640 1 -- 172.21.15.62:0/1030084410 wait complete. 2024-03-23T11:20:04.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.086+0000 7fcf77400640 1 Processor -- start 2024-03-23T11:20:04.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.086+0000 7fcf77400640 1 -- start start 2024-03-23T11:20:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf77400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf700cd7c0 0x7fcf701e2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf77400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf700cdba0 0x7fcf70153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf77400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf70155cc0 0x7fcf700ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf77400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf70156ea0 con 0x7fcf700cdba0 2024-03-23T11:20:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf77400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf700c3780 con 0x7fcf70155cc0 2024-03-23T11:20:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf77400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf70156600 con 0x7fcf700cd7c0 2024-03-23T11:20:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf6fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf700cdba0 0x7fcf70153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf6fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf700cdba0 0x7fcf70153040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59686/0 (socket says 172.21.15.62:59686) 2024-03-23T11:20:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf6fe00640 1 -- 172.21.15.62:0/4235908868 learned_addr learned my addr 172.21.15.62:0/4235908868 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf74a00640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf700cd7c0 0x7fcf701e2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf6fe00640 1 -- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf700cd7c0 msgr2=0x7fcf701e2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:04.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf75400640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf70155cc0 0x7fcf700ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf6fe00640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf700cd7c0 0x7fcf701e2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.087+0000 7fcf6fe00640 1 -- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf70155cc0 msgr2=0x7fcf700ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:04.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.088+0000 7fcf6fe00640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf70155cc0 0x7fcf700ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.088+0000 7fcf6fe00640 1 -- 172.21.15.62:0/4235908868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf640094d0 con 0x7fcf700cdba0 2024-03-23T11:20:04.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.088+0000 7fcf75400640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf70155cc0 0x7fcf700ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:04.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.088+0000 7fcf6fe00640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf700cdba0 0x7fcf70153040 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00e870 tx=0x7fcf5c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:04.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.088+0000 7fcf6cc00640 1 -- 172.21.15.62:0/4235908868 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c0094f0 con 0x7fcf700cdba0 2024-03-23T11:20:04.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.088+0000 7fcf6cc00640 1 -- 172.21.15.62:0/4235908868 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5c00c940 con 0x7fcf700cdba0 2024-03-23T11:20:04.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.088+0000 7fcf77400640 1 -- 172.21.15.62:0/4235908868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf701568e0 con 0x7fcf700cdba0 2024-03-23T11:20:04.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.088+0000 7fcf6cc00640 1 -- 172.21.15.62:0/4235908868 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c00ccf0 con 0x7fcf700cdba0 2024-03-23T11:20:04.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.088+0000 7fcf77400640 1 -- 172.21.15.62:0/4235908868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf70153580 con 0x7fcf700cdba0 2024-03-23T11:20:04.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.090+0000 7fcf77400640 1 -- 172.21.15.62:0/4235908868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf70157150 con 0x7fcf700cdba0 2024-03-23T11:20:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.091+0000 7fcf6cc00640 1 -- 172.21.15.62:0/4235908868 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fcf5c013470 con 0x7fcf700cdba0 2024-03-23T11:20:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.091+0000 7fcf6cc00640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf5006a7a0 0x7fcf5006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.091+0000 7fcf6cc00640 1 -- 172.21.15.62:0/4235908868 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcf5c089280 con 0x7fcf700cdba0 2024-03-23T11:20:04.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.092+0000 7fcf74a00640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf5006a7a0 0x7fcf5006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.093+0000 7fcf74a00640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf5006a7a0 0x7fcf5006cc60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcf64036000 tx=0x7fcf6403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:04.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.096+0000 7fcf6cc00640 1 -- 172.21.15.62:0/4235908868 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf5c054620 con 0x7fcf700cdba0 2024-03-23T11:20:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.295+0000 7fcf77400640 1 -- 172.21.15.62:0/4235908868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf700c38f0 con 0x7fcf700cdba0 2024-03-23T11:20:04.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.298+0000 7fcf6cc00640 1 -- 172.21.15.62:0/4235908868 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16226 (secure 0 0 0) 0x7fcf5c057c40 con 0x7fcf700cdba0 2024-03-23T11:20:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:04.299 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:19:42.867981+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:20:04.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.300+0000 7fcf77400640 1 -- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf5006a7a0 msgr2=0x7fcf5006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:04.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.300+0000 7fcf77400640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf5006a7a0 0x7fcf5006cc60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcf64036000 tx=0x7fcf6403c040 comp rx=0 tx=0).stop 2024-03-23T11:20:04.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.300+0000 7fcf77400640 1 -- 172.21.15.62:0/4235908868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf700cdba0 msgr2=0x7fcf70153040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:04.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.300+0000 7fcf77400640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf700cdba0 0x7fcf70153040 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00e870 tx=0x7fcf5c00ed30 comp rx=0 tx=0).stop 2024-03-23T11:20:04.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.301+0000 7fcf77400640 1 -- 172.21.15.62:0/4235908868 shutdown_connections 2024-03-23T11:20:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.301+0000 7fcf77400640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcf5006a7a0 0x7fcf5006cc60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.301+0000 7fcf77400640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf700cdba0 0x7fcf70153040 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.301+0000 7fcf77400640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf70155cc0 0x7fcf700ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.301+0000 7fcf77400640 1 --2- 172.21.15.62:0/4235908868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf700cd7c0 0x7fcf701e2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.301+0000 7fcf77400640 1 -- 172.21.15.62:0/4235908868 >> 172.21.15.62:0/4235908868 conn(0x7fcf700b02e0 msgr2=0x7fcf7015aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.301+0000 7fcf77400640 1 -- 172.21.15.62:0/4235908868 shutdown_connections 2024-03-23T11:20:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.301+0000 7fcf77400640 1 -- 172.21.15.62:0/4235908868 wait complete. 2024-03-23T11:20:04.321 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T11:20:04.321 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:20:04.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.540+0000 7fb416000640 1 Processor -- start 2024-03-23T11:20:04.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.540+0000 7fb416000640 1 -- start start 2024-03-23T11:20:04.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.542+0000 7fb416000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb410143dd0 0x7fb41015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.542+0000 7fb416000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb4100b2f30 0x7fb4100704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.542+0000 7fb416000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb410070a30 con 0x7fb410070090 2024-03-23T11:20:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.542+0000 7fb416000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4100cd140 con 0x7fb4100b2f30 2024-03-23T11:20:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.542+0000 7fb416000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100cd3e0 con 0x7fb410143dd0 2024-03-23T11:20:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.542+0000 7fb415600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb4100b2f30 0x7fb4100704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.542+0000 7fb415600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb4100b2f30 0x7fb4100704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59692/0 (socket says 172.21.15.62:59692) 2024-03-23T11:20:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.542+0000 7fb415600640 1 -- 172.21.15.62:0/2892474524 learned_addr learned my addr 172.21.15.62:0/2892474524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.543+0000 7fb414c00640 1 --2- 172.21.15.62:0/2892474524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb410143dd0 0x7fb41015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.543+0000 7fb415600640 1 -- 172.21.15.62:0/2892474524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb410143dd0 msgr2=0x7fb41015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.543+0000 7fb415600640 1 --2- 172.21.15.62:0/2892474524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb410143dd0 0x7fb41015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.543+0000 7fb415600640 1 -- 172.21.15.62:0/2892474524 >> v1:172.21.15.120:6789/0 conn(0x7fb410070090 legacy=0x7fb4100b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.543+0000 7fb415600640 1 -- 172.21.15.62:0/2892474524 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4100cd6c0 con 0x7fb4100b2f30 2024-03-23T11:20:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.544+0000 7fb415600640 1 --2- 172.21.15.62:0/2892474524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4100b2f30 0x7fb4100704f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb408009870 tx=0x7fb40802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b87437d1ca2b6319 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.544+0000 7fb40fe00640 1 -- 172.21.15.62:0/2892474524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408031640 con 0x7fb4100b2f30 2024-03-23T11:20:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.544+0000 7fb40fe00640 1 -- 172.21.15.62:0/2892474524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4080317a0 con 0x7fb4100b2f30 2024-03-23T11:20:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.544+0000 7fb40fe00640 1 -- 172.21.15.62:0/2892474524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408031b30 con 0x7fb4100b2f30 2024-03-23T11:20:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.545+0000 7fb416000640 1 -- 172.21.15.62:0/2892474524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4100b2f30 msgr2=0x7fb4100704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.545+0000 7fb416000640 1 --2- 172.21.15.62:0/2892474524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4100b2f30 0x7fb4100704f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb408009870 tx=0x7fb40802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.545+0000 7fb416000640 1 -- 172.21.15.62:0/2892474524 shutdown_connections 2024-03-23T11:20:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.545+0000 7fb416000640 1 --2- 172.21.15.62:0/2892474524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4100b2f30 0x7fb4100704f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.545+0000 7fb416000640 1 --2- 172.21.15.62:0/2892474524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb410143dd0 0x7fb41015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.545+0000 7fb416000640 1 -- 172.21.15.62:0/2892474524 >> 172.21.15.62:0/2892474524 conn(0x7fb410066050 msgr2=0x7fb410069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.545+0000 7fb416000640 1 -- 172.21.15.62:0/2892474524 shutdown_connections 2024-03-23T11:20:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.545+0000 7fb416000640 1 -- 172.21.15.62:0/2892474524 wait complete. 2024-03-23T11:20:04.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.547+0000 7fb416000640 1 Processor -- start 2024-03-23T11:20:04.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.547+0000 7fb416000640 1 -- start start 2024-03-23T11:20:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.548+0000 7fb416000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb410070090 0x7fb4100a8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.548+0000 7fb416000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4100b2f30 0x7fb41013b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.548+0000 7fb416000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410143dd0 0x7fb4100a8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.548+0000 7fb416000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41013b9c0 con 0x7fb410143dd0 2024-03-23T11:20:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.548+0000 7fb416000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41013bbf0 con 0x7fb4100b2f30 2024-03-23T11:20:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.548+0000 7fb416000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41013bed0 con 0x7fb410070090 2024-03-23T11:20:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.548+0000 7fb415600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410143dd0 0x7fb4100a8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.548+0000 7fb415600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410143dd0 0x7fb4100a8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59696/0 (socket says 172.21.15.62:59696) 2024-03-23T11:20:04.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.548+0000 7fb415600640 1 -- 172.21.15.62:0/770319815 learned_addr learned my addr 172.21.15.62:0/770319815 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb407e00640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4100b2f30 0x7fb41013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb414c00640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb410070090 0x7fb4100a8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb415600640 1 -- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb410070090 msgr2=0x7fb4100a8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb415600640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb410070090 0x7fb4100a8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb415600640 1 -- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4100b2f30 msgr2=0x7fb41013b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb415600640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4100b2f30 0x7fb41013b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb415600640 1 -- 172.21.15.62:0/770319815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4080094d0 con 0x7fb410143dd0 2024-03-23T11:20:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb407e00640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4100b2f30 0x7fb41013b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb415600640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410143dd0 0x7fb4100a8d50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb40802b930 tx=0x7fb4080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb40d600640 1 -- 172.21.15.62:0/770319815 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40803ac60 con 0x7fb410143dd0 2024-03-23T11:20:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb40d600640 1 -- 172.21.15.62:0/770319815 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40803adc0 con 0x7fb410143dd0 2024-03-23T11:20:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb416000640 1 -- 172.21.15.62:0/770319815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41013c1b0 con 0x7fb410143dd0 2024-03-23T11:20:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.549+0000 7fb40d600640 1 -- 172.21.15.62:0/770319815 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408044630 con 0x7fb410143dd0 2024-03-23T11:20:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.550+0000 7fb416000640 1 -- 172.21.15.62:0/770319815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41013c770 con 0x7fb410143dd0 2024-03-23T11:20:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.551+0000 7fb416000640 1 -- 172.21.15.62:0/770319815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41013c960 con 0x7fb410143dd0 2024-03-23T11:20:04.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.552+0000 7fb40d600640 1 -- 172.21.15.62:0/770319815 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb40802f050 con 0x7fb410143dd0 2024-03-23T11:20:04.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.553+0000 7fb40d600640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3ec06a7d0 0x7fb3ec06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:04.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.553+0000 7fb40d600640 1 -- 172.21.15.62:0/770319815 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb4080aad20 con 0x7fb410143dd0 2024-03-23T11:20:04.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.553+0000 7fb414c00640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3ec06a7d0 0x7fb3ec06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:04.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.554+0000 7fb414c00640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3ec06a7d0 0x7fb3ec06cc90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb4000099c0 tx=0x7fb400003990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:04.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.557+0000 7fb40d600640 1 -- 172.21.15.62:0/770319815 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb408076150 con 0x7fb410143dd0 2024-03-23T11:20:04.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.793+0000 7fb416000640 1 -- 172.21.15.62:0/770319815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb410066bb0 con 0x7fb410143dd0 2024-03-23T11:20:04.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.794+0000 7fb40d600640 1 -- 172.21.15.62:0/770319815 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7fb408035070 con 0x7fb410143dd0 2024-03-23T11:20:04.795 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:04.795 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/496399527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":496399527},{"type":"v1","addr":"172.21.15.62:6833","nonce":496399527}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/156755611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":156755611},{"type":"v1","addr":"172.21.15.120:6835","nonce":156755611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2391380456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2391380456},{"type":"v1","addr":"172.21.15.62:6835","nonce":2391380456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:18:56.233595+0000","modified":"2024-03-23T11:18:59.758284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/3936303103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3936303103},{"type":"v1","addr":"172.21.15.120:6837","nonce":3936303103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T11:20:04.795 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-03-23T11:20:04.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.797+0000 7fb416000640 1 -- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3ec06a7d0 msgr2=0x7fb3ec06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:04.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.797+0000 7fb416000640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3ec06a7d0 0x7fb3ec06cc90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb4000099c0 tx=0x7fb400003990 comp rx=0 tx=0).stop 2024-03-23T11:20:04.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.797+0000 7fb416000640 1 -- 172.21.15.62:0/770319815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410143dd0 msgr2=0x7fb4100a8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:04.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.797+0000 7fb416000640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410143dd0 0x7fb4100a8d50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb40802b930 tx=0x7fb4080046c0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.797+0000 7fb416000640 1 -- 172.21.15.62:0/770319815 shutdown_connections 2024-03-23T11:20:04.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.797+0000 7fb416000640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3ec06a7d0 0x7fb3ec06cc90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.797+0000 7fb416000640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410143dd0 0x7fb4100a8d50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.797+0000 7fb416000640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4100b2f30 0x7fb41013b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.797+0000 7fb416000640 1 --2- 172.21.15.62:0/770319815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb410070090 0x7fb4100a8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.797+0000 7fb416000640 1 -- 172.21.15.62:0/770319815 >> 172.21.15.62:0/770319815 conn(0x7fb410066050 msgr2=0x7fb4100adde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.797+0000 7fb416000640 1 -- 172.21.15.62:0/770319815 shutdown_connections 2024-03-23T11:20:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:04.798+0000 7fb416000640 1 -- 172.21.15.62:0/770319815 wait complete. 2024-03-23T11:20:04.818 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:20:05.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.040+0000 7fe8e2200640 1 Processor -- start 2024-03-23T11:20:05.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.040+0000 7fe8e2200640 1 -- start start 2024-03-23T11:20:05.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.041+0000 7fe8e2200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe8dc15fcf0 0x7fe8dc0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:05.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.041+0000 7fe8e2200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8dc0d14f0 con 0x7fe8dc160420 2024-03-23T11:20:05.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.041+0000 7fe8e2200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8dc0d3540 con 0x7fe8dc0b6dc0 2024-03-23T11:20:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.041+0000 7fe8e2200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8dc0ce510 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.042+0000 7fe8e0e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe8dc15fcf0 0x7fe8dc0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.042+0000 7fe8e0e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe8dc15fcf0 0x7fe8dc0d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59702/0 (socket says 172.21.15.62:59702) 2024-03-23T11:20:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.042+0000 7fe8e0e00640 1 -- 172.21.15.62:0/2711255476 learned_addr learned my addr 172.21.15.62:0/2711255476 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.042+0000 7fe8d3e00640 1 --1- 172.21.15.62:0/2711255476 >> v1:172.21.15.62:6789/0 conn(0x7fe8dc160420 0x7fe8dc0d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50456/0 (socket says 172.21.15.62:50456) 2024-03-23T11:20:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.042+0000 7fe8e0e00640 1 -- 172.21.15.62:0/2711255476 >> v1:172.21.15.120:6789/0 conn(0x7fe8dc0b6dc0 legacy=0x7fe8dc0d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.042+0000 7fe8e0e00640 1 -- 172.21.15.62:0/2711255476 >> v1:172.21.15.62:6789/0 conn(0x7fe8dc160420 legacy=0x7fe8dc0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.042+0000 7fe8e0e00640 1 -- 172.21.15.62:0/2711255476 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc0cdc70 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.044+0000 7fe8e0e00640 1 --2- 172.21.15.62:0/2711255476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8dc15fcf0 0x7fe8dc0d3000 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4009870 tx=0x7fe8c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c49956d54bc98d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:05.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.044+0000 7fe8dbe00640 1 -- 172.21.15.62:0/2711255476 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c4031870 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.044+0000 7fe8dbe00640 1 -- 172.21.15.62:0/2711255476 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c40319d0 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.044+0000 7fe8dbe00640 1 -- 172.21.15.62:0/2711255476 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c4031d60 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.045+0000 7fe8e2200640 1 -- 172.21.15.62:0/2711255476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8dc15fcf0 msgr2=0x7fe8dc0d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.045+0000 7fe8e2200640 1 --2- 172.21.15.62:0/2711255476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8dc15fcf0 0x7fe8dc0d3000 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4009870 tx=0x7fe8c402b910 comp rx=0 tx=0).stop 2024-03-23T11:20:05.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.045+0000 7fe8e2200640 1 -- 172.21.15.62:0/2711255476 shutdown_connections 2024-03-23T11:20:05.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.045+0000 7fe8e2200640 1 --2- 172.21.15.62:0/2711255476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8dc15fcf0 0x7fe8dc0d3000 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.045+0000 7fe8e2200640 1 -- 172.21.15.62:0/2711255476 >> 172.21.15.62:0/2711255476 conn(0x7fe8dc089f60 msgr2=0x7fe8dc0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:05.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.045+0000 7fe8e2200640 1 -- 172.21.15.62:0/2711255476 shutdown_connections 2024-03-23T11:20:05.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.045+0000 7fe8e2200640 1 -- 172.21.15.62:0/2711255476 wait complete. 2024-03-23T11:20:05.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.047+0000 7fe8e2200640 1 Processor -- start 2024-03-23T11:20:05.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.047+0000 7fe8e2200640 1 -- start start 2024-03-23T11:20:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8e2200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8dc0b6dc0 0x7fe8dc09e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8e2200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8dc15fcf0 0x7fe8dc09c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8e2200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8dc160420 0x7fe8dc09f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8e2200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc09e920 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8e2200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc09ec00 con 0x7fe8dc160420 2024-03-23T11:20:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8e2200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc09eee0 con 0x7fe8dc0b6dc0 2024-03-23T11:20:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8d3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8dc15fcf0 0x7fe8dc09c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:05.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8d3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8dc15fcf0 0x7fe8dc09c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59714/0 (socket says 172.21.15.62:59714) 2024-03-23T11:20:05.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8d3e00640 1 -- 172.21.15.62:0/4214213858 learned_addr learned my addr 172.21.15.62:0/4214213858 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8e1800640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8dc160420 0x7fe8dc09f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8d3e00640 1 -- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8dc0b6dc0 msgr2=0x7fe8dc09e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8e0e00640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8dc0b6dc0 0x7fe8dc09e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8d3e00640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8dc0b6dc0 0x7fe8dc09e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8d3e00640 1 -- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8dc160420 msgr2=0x7fe8dc09f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8d3e00640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8dc160420 0x7fe8dc09f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.048+0000 7fe8d3e00640 1 -- 172.21.15.62:0/4214213858 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c40094d0 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.049+0000 7fe8e1800640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8dc160420 0x7fe8dc09f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.049+0000 7fe8d3e00640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8dc15fcf0 0x7fe8dc09c860 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc00c3e0 tx=0x7fe8cc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.049+0000 7fe8d9600640 1 -- 172.21.15.62:0/4214213858 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc0074e0 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.049+0000 7fe8d9600640 1 -- 172.21.15.62:0/4214213858 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8cc007640 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.049+0000 7fe8d9600640 1 -- 172.21.15.62:0/4214213858 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc0049e0 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.049+0000 7fe8e2200640 1 -- 172.21.15.62:0/4214213858 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8dc09f1c0 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.049+0000 7fe8e2200640 1 -- 172.21.15.62:0/4214213858 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8dc09cda0 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.050+0000 7fe8e2200640 1 -- 172.21.15.62:0/4214213858 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8dc066bb0 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.051+0000 7fe8d9600640 1 -- 172.21.15.62:0/4214213858 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe8cc004b40 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.052+0000 7fe8d9600640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8b806a7a0 0x7fe8b806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.052+0000 7fe8d9600640 1 -- 172.21.15.62:0/4214213858 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe8cc088770 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.053+0000 7fe8e0e00640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8b806a7a0 0x7fe8b806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:05.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.054+0000 7fe8e0e00640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8b806a7a0 0x7fe8b806cc60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4036000 tx=0x7fe8c403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:05.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.059+0000 7fe8d9600640 1 -- 172.21.15.62:0/4214213858 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe8cc053ba0 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.300+0000 7fe8e2200640 1 -- 172.21.15.62:0/4214213858 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8dc0b4d90 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.302+0000 7fe8d9600640 1 -- 172.21.15.62:0/4214213858 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7fe8cc0571c0 con 0x7fe8dc15fcf0 2024-03-23T11:20:05.302 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:05.302 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/496399527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":496399527},{"type":"v1","addr":"172.21.15.62:6833","nonce":496399527}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/156755611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":156755611},{"type":"v1","addr":"172.21.15.120:6835","nonce":156755611}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2391380456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2391380456},{"type":"v1","addr":"172.21.15.62:6835","nonce":2391380456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:18:56.233595+0000","modified":"2024-03-23T11:18:59.758284+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/3936303103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3936303103},{"type":"v1","addr":"172.21.15.120:6837","nonce":3936303103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T11:20:05.302 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-03-23T11:20:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.304+0000 7fe8e2200640 1 -- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8b806a7a0 msgr2=0x7fe8b806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.304+0000 7fe8e2200640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8b806a7a0 0x7fe8b806cc60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4036000 tx=0x7fe8c403c040 comp rx=0 tx=0).stop 2024-03-23T11:20:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.304+0000 7fe8e2200640 1 -- 172.21.15.62:0/4214213858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8dc15fcf0 msgr2=0x7fe8dc09c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.304+0000 7fe8e2200640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8dc15fcf0 0x7fe8dc09c860 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc00c3e0 tx=0x7fe8cc00c8a0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.304+0000 7fe8e2200640 1 -- 172.21.15.62:0/4214213858 shutdown_connections 2024-03-23T11:20:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.304+0000 7fe8e2200640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8b806a7a0 0x7fe8b806cc60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.305+0000 7fe8e2200640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8dc15fcf0 0x7fe8dc09c860 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.305+0000 7fe8e2200640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8dc160420 0x7fe8dc09f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.305+0000 7fe8e2200640 1 --2- 172.21.15.62:0/4214213858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8dc0b6dc0 0x7fe8dc09e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.305+0000 7fe8e2200640 1 -- 172.21.15.62:0/4214213858 >> 172.21.15.62:0/4214213858 conn(0x7fe8dc089f60 msgr2=0x7fe8dc06afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:05.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.305+0000 7fe8e2200640 1 -- 172.21.15.62:0/4214213858 shutdown_connections 2024-03-23T11:20:05.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.305+0000 7fe8e2200640 1 -- 172.21.15.62:0/4214213858 wait complete. 2024-03-23T11:20:05.323 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:20:05.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.543+0000 7fba6e200640 1 Processor -- start 2024-03-23T11:20:05.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.543+0000 7fba6e200640 1 -- start start 2024-03-23T11:20:05.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.545+0000 7fba6e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b15e0 0x7fba68161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:05.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.545+0000 7fba6e200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba680ae0c0 con 0x7fba6806c300 2024-03-23T11:20:05.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.545+0000 7fba6e200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba680b2740 con 0x7fba680b2f90 2024-03-23T11:20:05.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.545+0000 7fba6e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba681623b0 con 0x7fba680b15e0 2024-03-23T11:20:05.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.545+0000 7fba67e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fba6806c300 0x7fba6806c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50460/0 (socket says 172.21.15.62:50460) 2024-03-23T11:20:05.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.545+0000 7fba67e00640 1 -- 172.21.15.62:0/2359999727 learned_addr learned my addr 172.21.15.62:0/2359999727 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:05.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.545+0000 7fba66a00640 1 --2- 172.21.15.62:0/2359999727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b15e0 0x7fba68161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:05.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.546+0000 7fba66000640 1 -- 172.21.15.62:0/2359999727 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874417042 0 0) 0x7fba680ae0c0 con 0x7fba6806c300 2024-03-23T11:20:05.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.546+0000 7fba66000640 1 -- 172.21.15.62:0/2359999727 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba4c003520 con 0x7fba6806c300 2024-03-23T11:20:05.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.546+0000 7fba66000640 1 -- 172.21.15.62:0/2359999727 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865555727 0 0) 0x7fba680b2740 con 0x7fba680b2f90 2024-03-23T11:20:05.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.546+0000 7fba66a00640 1 -- 172.21.15.62:0/2359999727 >> v1:172.21.15.120:6789/0 conn(0x7fba680b2f90 legacy=0x7fba680b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.546+0000 7fba66a00640 1 -- 172.21.15.62:0/2359999727 >> v1:172.21.15.62:6789/0 conn(0x7fba6806c300 legacy=0x7fba6806c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.546+0000 7fba66a00640 1 -- 172.21.15.62:0/2359999727 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba680b1100 con 0x7fba680b15e0 2024-03-23T11:20:05.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.548+0000 7fba66a00640 1 --2- 172.21.15.62:0/2359999727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b15e0 0x7fba68161e70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba54009dc0 tx=0x7fba5402bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57da34438dd306b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.548+0000 7fba66000640 1 -- 172.21.15.62:0/2359999727 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5402ccc0 con 0x7fba680b15e0 2024-03-23T11:20:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.548+0000 7fba66000640 1 -- 172.21.15.62:0/2359999727 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5402ce20 con 0x7fba680b15e0 2024-03-23T11:20:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.548+0000 7fba66000640 1 -- 172.21.15.62:0/2359999727 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54034660 con 0x7fba680b15e0 2024-03-23T11:20:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.549+0000 7fba6e200640 1 -- 172.21.15.62:0/2359999727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b15e0 msgr2=0x7fba68161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.549+0000 7fba6e200640 1 --2- 172.21.15.62:0/2359999727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b15e0 0x7fba68161e70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba54009dc0 tx=0x7fba5402bf10 comp rx=0 tx=0).stop 2024-03-23T11:20:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.549+0000 7fba6e200640 1 -- 172.21.15.62:0/2359999727 shutdown_connections 2024-03-23T11:20:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.549+0000 7fba6e200640 1 --2- 172.21.15.62:0/2359999727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b15e0 0x7fba68161e70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.549+0000 7fba6e200640 1 -- 172.21.15.62:0/2359999727 >> 172.21.15.62:0/2359999727 conn(0x7fba680b02e0 msgr2=0x7fba680b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.549+0000 7fba6e200640 1 -- 172.21.15.62:0/2359999727 shutdown_connections 2024-03-23T11:20:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.549+0000 7fba6e200640 1 -- 172.21.15.62:0/2359999727 wait complete. 2024-03-23T11:20:05.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.551+0000 7fba6e200640 1 Processor -- start 2024-03-23T11:20:05.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.551+0000 7fba6e200640 1 -- start start 2024-03-23T11:20:05.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.552+0000 7fba6e200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba6806c300 0x7fba68161890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:05.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.552+0000 7fba6e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba680b15e0 0x7fba681f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:05.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.552+0000 7fba6e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b2f90 0x7fba6815ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:05.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.552+0000 7fba6e200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba68159580 con 0x7fba680b15e0 2024-03-23T11:20:05.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.552+0000 7fba6e200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba68159860 con 0x7fba6806c300 2024-03-23T11:20:05.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.552+0000 7fba6e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba68159b40 con 0x7fba680b2f90 2024-03-23T11:20:05.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.552+0000 7fba66a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba680b15e0 0x7fba681f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:05.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.552+0000 7fba66a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba680b15e0 0x7fba681f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59730/0 (socket says 172.21.15.62:59730) 2024-03-23T11:20:05.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.552+0000 7fba66a00640 1 -- 172.21.15.62:0/3267505782 learned_addr learned my addr 172.21.15.62:0/3267505782 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:05.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.552+0000 7fba67400640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba6806c300 0x7fba68161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:05.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.552+0000 7fba67e00640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b2f90 0x7fba6815ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:05.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.553+0000 7fba66a00640 1 -- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b2f90 msgr2=0x7fba6815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.553+0000 7fba66a00640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b2f90 0x7fba6815ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.553+0000 7fba66a00640 1 -- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba6806c300 msgr2=0x7fba68161890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.553+0000 7fba66a00640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba6806c300 0x7fba68161890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.553+0000 7fba66a00640 1 -- 172.21.15.62:0/3267505782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba54009a70 con 0x7fba680b15e0 2024-03-23T11:20:05.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.553+0000 7fba67e00640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b2f90 0x7fba6815ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:05.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.553+0000 7fba66a00640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba680b15e0 0x7fba681f5f50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fba54009ee0 tx=0x7fba54004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:05.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.553+0000 7fba5f400640 1 -- 172.21.15.62:0/3267505782 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54040650 con 0x7fba680b15e0 2024-03-23T11:20:05.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.553+0000 7fba5f400640 1 -- 172.21.15.62:0/3267505782 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba540407d0 con 0x7fba680b15e0 2024-03-23T11:20:05.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.553+0000 7fba5f400640 1 -- 172.21.15.62:0/3267505782 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54040b80 con 0x7fba680b15e0 2024-03-23T11:20:05.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.553+0000 7fba6e200640 1 -- 172.21.15.62:0/3267505782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba68159e20 con 0x7fba680b15e0 2024-03-23T11:20:05.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.554+0000 7fba6e200640 1 -- 172.21.15.62:0/3267505782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba681f6490 con 0x7fba680b15e0 2024-03-23T11:20:05.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.555+0000 7fba6e200640 1 -- 172.21.15.62:0/3267505782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba68159fb0 con 0x7fba680b15e0 2024-03-23T11:20:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.556+0000 7fba5f400640 1 -- 172.21.15.62:0/3267505782 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fba54040ce0 con 0x7fba680b15e0 2024-03-23T11:20:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.557+0000 7fba5f400640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba4806a7a0 0x7fba4806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.557+0000 7fba5f400640 1 -- 172.21.15.62:0/3267505782 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fba540ab810 con 0x7fba680b15e0 2024-03-23T11:20:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.558+0000 7fba67400640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba4806a7a0 0x7fba4806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:05.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.559+0000 7fba67400640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba4806a7a0 0x7fba4806cc60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fba58007fd0 tx=0x7fba58009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:05.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.562+0000 7fba5f400640 1 -- 172.21.15.62:0/3267505782 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba54076c40 con 0x7fba680b15e0 2024-03-23T11:20:05.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.759+0000 7fba6e200640 1 -- 172.21.15.62:0/3267505782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba68066c90 con 0x7fba680b15e0 2024-03-23T11:20:05.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.762+0000 7fba5f400640 1 -- 172.21.15.62:0/3267505782 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16226 (secure 0 0 0) 0x7fba5407a260 con 0x7fba680b15e0 2024-03-23T11:20:05.763 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:05.763 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:19:42.867981+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:18:53.705929+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T11:18:55.252182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:20:05.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.764+0000 7fba6e200640 1 -- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba4806a7a0 msgr2=0x7fba4806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.765+0000 7fba6e200640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba4806a7a0 0x7fba4806cc60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fba58007fd0 tx=0x7fba58009040 comp rx=0 tx=0).stop 2024-03-23T11:20:05.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.765+0000 7fba6e200640 1 -- 172.21.15.62:0/3267505782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba680b15e0 msgr2=0x7fba681f5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:05.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.765+0000 7fba6e200640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba680b15e0 0x7fba681f5f50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fba54009ee0 tx=0x7fba54004ab0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.765+0000 7fba6e200640 1 -- 172.21.15.62:0/3267505782 shutdown_connections 2024-03-23T11:20:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.765+0000 7fba6e200640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fba4806a7a0 0x7fba4806cc60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.765+0000 7fba6e200640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba680b15e0 0x7fba681f5f50 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.765+0000 7fba6e200640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fba680b2f90 0x7fba6815ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.765+0000 7fba6e200640 1 --2- 172.21.15.62:0/3267505782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba6806c300 0x7fba68161890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.765+0000 7fba6e200640 1 -- 172.21.15.62:0/3267505782 >> 172.21.15.62:0/3267505782 conn(0x7fba680b02e0 msgr2=0x7fba680ae710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.765+0000 7fba6e200640 1 -- 172.21.15.62:0/3267505782 shutdown_connections 2024-03-23T11:20:05.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:05.765+0000 7fba6e200640 1 -- 172.21.15.62:0/3267505782 wait complete. 2024-03-23T11:20:05.785 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:20:06.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.011+0000 7fb864c00640 1 Processor -- start 2024-03-23T11:20:06.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.011+0000 7fb864c00640 1 -- start start 2024-03-23T11:20:06.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.013+0000 7fb864c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015d060 0x7fb86006a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.013+0000 7fb864c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb8600b2b30 0x7fb8600b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.013+0000 7fb864c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8600b14c0 con 0x7fb8601593e0 2024-03-23T11:20:06.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.013+0000 7fb864c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86006a020 con 0x7fb8600b2b30 2024-03-23T11:20:06.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.013+0000 7fb864c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8600b34e0 con 0x7fb86015d060 2024-03-23T11:20:06.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.013+0000 7fb85f400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb8601593e0 0x7fb8600cc100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50470/0 (socket says 172.21.15.62:50470) 2024-03-23T11:20:06.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.013+0000 7fb85f400640 1 -- 172.21.15.62:0/792632149 learned_addr learned my addr 172.21.15.62:0/792632149 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb856a00640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015d060 0x7fb86006a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85fe00640 1 --2- 172.21.15.62:0/792632149 >> v2:172.21.15.120:3300/0 conn(0x7fb8600b2b30 0x7fb8600b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688204587 0 0) 0x7fb8600b14c0 con 0x7fb8601593e0 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb840003420 con 0x7fb8601593e0 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3454997220 0 0) 0x7fb840003420 con 0x7fb8601593e0 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015d060 msgr2=0x7fb86006a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015d060 0x7fb86006a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 >> v2:172.21.15.120:3300/0 conn(0x7fb8600b2b30 msgr2=0x7fb8600b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 --2- 172.21.15.62:0/792632149 >> v2:172.21.15.120:3300/0 conn(0x7fb8600b2b30 0x7fb8600b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86015dee0 con 0x7fb8601593e0 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fb850001670 con 0x7fb8601593e0 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 >> v1:172.21.15.62:6789/0 conn(0x7fb8601593e0 legacy=0x7fb8600cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 0x7fb840004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb840003660 0x7fb840003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015d060 0x7fb840004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8600b34e0 con 0x7fb840004a50 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84000fca0 con 0x7fb840003660 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.014+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84000fe60 con 0x7fb86015d060 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.015+0000 7fb85f400640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 0x7fb840004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.015+0000 7fb856a00640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb840003660 0x7fb840003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.015+0000 7fb85fe00640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015d060 0x7fb840004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.015+0000 7fb85f400640 1 -- 172.21.15.62:0/792632149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015d060 msgr2=0x7fb840004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.015+0000 7fb85f400640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015d060 0x7fb840004060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.015+0000 7fb85f400640 1 -- 172.21.15.62:0/792632149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb840003660 msgr2=0x7fb840003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.015+0000 7fb85f400640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb840003660 0x7fb840003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.015+0000 7fb85f400640 1 -- 172.21.15.62:0/792632149 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb840003420 con 0x7fb840004a50 2024-03-23T11:20:06.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.017+0000 7fb85f400640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 0x7fb840004e70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb85000c190 tx=0x7fb8400046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd38521d248341 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:06.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.017+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850005ad0 con 0x7fb840004a50 2024-03-23T11:20:06.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.017+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb850005c50 con 0x7fb840004a50 2024-03-23T11:20:06.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.017+0000 7fb85ea00640 1 -- 172.21.15.62:0/792632149 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85002c810 con 0x7fb840004a50 2024-03-23T11:20:06.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.017+0000 7fb864c00640 1 -- 172.21.15.62:0/792632149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 msgr2=0x7fb840004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:06.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.017+0000 7fb864c00640 1 --2- 172.21.15.62:0/792632149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 0x7fb840004e70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb85000c190 tx=0x7fb8400046e0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.017+0000 7fb85fe00640 1 -- 172.21.15.62:0/792632149 reap_dead start 2024-03-23T11:20:06.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.018+0000 7fb864c00640 1 -- 172.21.15.62:0/792632149 shutdown_connections 2024-03-23T11:20:06.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.018+0000 7fb864c00640 1 -- 172.21.15.62:0/792632149 >> 172.21.15.62:0/792632149 conn(0x7fb8600b0090 msgr2=0x7fb8600b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:06.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.018+0000 7fb864c00640 1 -- 172.21.15.62:0/792632149 shutdown_connections 2024-03-23T11:20:06.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.018+0000 7fb864c00640 1 -- 172.21.15.62:0/792632149 wait complete. 2024-03-23T11:20:06.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.020+0000 7fb864c00640 1 Processor -- start 2024-03-23T11:20:06.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.020+0000 7fb864c00640 1 -- start start 2024-03-23T11:20:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.021+0000 7fb864c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 0x7fb8600cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.021+0000 7fb864c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015c2e0 0x7fb8600707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.021+0000 7fb864c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb860151a80 0x7fb860151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.021+0000 7fb864c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb860152470 con 0x7fb840004a50 2024-03-23T11:20:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.021+0000 7fb864c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8601edd70 con 0x7fb860151a80 2024-03-23T11:20:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.021+0000 7fb864c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8600cca10 con 0x7fb86015c2e0 2024-03-23T11:20:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.021+0000 7fb85f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 0x7fb8600cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.021+0000 7fb85f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 0x7fb8600cc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59738/0 (socket says 172.21.15.62:59738) 2024-03-23T11:20:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.021+0000 7fb85f400640 1 -- 172.21.15.62:0/3899993390 learned_addr learned my addr 172.21.15.62:0/3899993390 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.021+0000 7fb856a00640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015c2e0 0x7fb8600707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.021+0000 7fb85fe00640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb860151a80 0x7fb860151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.022+0000 7fb85f400640 1 -- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015c2e0 msgr2=0x7fb8600707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.022+0000 7fb85f400640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015c2e0 0x7fb8600707a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.022+0000 7fb85f400640 1 -- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb860151a80 msgr2=0x7fb860151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.022+0000 7fb85f400640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb860151a80 0x7fb860151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.022+0000 7fb85f400640 1 -- 172.21.15.62:0/3899993390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8600cccf0 con 0x7fb840004a50 2024-03-23T11:20:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.022+0000 7fb85f400640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 0x7fb8600cc0c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb85002c3b0 tx=0x7fb8600b1430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.022+0000 7fb857e00640 1 -- 172.21.15.62:0/3899993390 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850039070 con 0x7fb840004a50 2024-03-23T11:20:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.022+0000 7fb857e00640 1 -- 172.21.15.62:0/3899993390 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85002cc20 con 0x7fb840004a50 2024-03-23T11:20:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.022+0000 7fb857e00640 1 -- 172.21.15.62:0/3899993390 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85003c040 con 0x7fb840004a50 2024-03-23T11:20:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.022+0000 7fb864c00640 1 -- 172.21.15.62:0/3899993390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8600ccfd0 con 0x7fb840004a50 2024-03-23T11:20:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.022+0000 7fb864c00640 1 -- 172.21.15.62:0/3899993390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8600cc600 con 0x7fb840004a50 2024-03-23T11:20:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.024+0000 7fb864c00640 1 -- 172.21.15.62:0/3899993390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8601562f0 con 0x7fb840004a50 2024-03-23T11:20:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.024+0000 7fb857e00640 1 -- 172.21.15.62:0/3899993390 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb850032070 con 0x7fb840004a50 2024-03-23T11:20:06.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.025+0000 7fb857e00640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb83c06a7d0 0x7fb83c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.025+0000 7fb857e00640 1 -- 172.21.15.62:0/3899993390 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb8500aa0c0 con 0x7fb840004a50 2024-03-23T11:20:06.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.026+0000 7fb856a00640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb83c06a7d0 0x7fb83c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.027+0000 7fb856a00640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb83c06a7d0 0x7fb83c06cc90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb860154de0 tx=0x7fb848008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:06.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.030+0000 7fb857e00640 1 -- 172.21.15.62:0/3899993390 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb850075570 con 0x7fb840004a50 2024-03-23T11:20:06.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.272+0000 7fb864c00640 1 -- 172.21.15.62:0/3899993390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb860066bb0 con 0x7fb840004a50 2024-03-23T11:20:06.275 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:20:06.275+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:20:06.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.291+0000 7fb857e00640 1 -- 172.21.15.62:0/3899993390 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb850078b90 con 0x7fb840004a50 2024-03-23T11:20:06.292 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:20:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.294+0000 7fb864c00640 1 -- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb83c06a7d0 msgr2=0x7fb83c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.294+0000 7fb864c00640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb83c06a7d0 0x7fb83c06cc90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb860154de0 tx=0x7fb848008040 comp rx=0 tx=0).stop 2024-03-23T11:20:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.294+0000 7fb864c00640 1 -- 172.21.15.62:0/3899993390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 msgr2=0x7fb8600cc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.294+0000 7fb864c00640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 0x7fb8600cc0c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb85002c3b0 tx=0x7fb8600b1430 comp rx=0 tx=0).stop 2024-03-23T11:20:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.295+0000 7fb864c00640 1 -- 172.21.15.62:0/3899993390 shutdown_connections 2024-03-23T11:20:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.295+0000 7fb864c00640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb83c06a7d0 0x7fb83c06cc90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.295+0000 7fb864c00640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb860151a80 0x7fb860151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.295+0000 7fb864c00640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb86015c2e0 0x7fb8600707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.295+0000 7fb864c00640 1 --2- 172.21.15.62:0/3899993390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb840004a50 0x7fb8600cc0c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.295+0000 7fb864c00640 1 -- 172.21.15.62:0/3899993390 >> 172.21.15.62:0/3899993390 conn(0x7fb8600b0090 msgr2=0x7fb860156d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:06.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.295+0000 7fb864c00640 1 -- 172.21.15.62:0/3899993390 shutdown_connections 2024-03-23T11:20:06.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.295+0000 7fb864c00640 1 -- 172.21.15.62:0/3899993390 wait complete. 2024-03-23T11:20:06.314 INFO:tasks.ceph.mds.d.smithi120.stderr: -3852> 2024-03-23T11:18:59.548+0000 7fd3ff400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:20:06.315 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T11:20:06.397 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-23T11:20:06.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.538+0000 7fa391000640 1 Processor -- start 2024-03-23T11:20:06.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.538+0000 7fa391000640 1 -- start start 2024-03-23T11:20:06.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.540+0000 7fa391000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c1593e0 0x7fa38c15d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.540+0000 7fa391000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38c0b14c0 con 0x7fa38c0b2b30 2024-03-23T11:20:06.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.540+0000 7fa391000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38c06a6f0 con 0x7fa38c06a290 2024-03-23T11:20:06.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.540+0000 7fa391000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c06a020 con 0x7fa38c1593e0 2024-03-23T11:20:06.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.540+0000 7fa38be00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa38c0b2b30 0x7fa38c0b1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50486/0 (socket says 172.21.15.62:50486) 2024-03-23T11:20:06.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.540+0000 7fa38be00640 1 -- 172.21.15.62:0/3079333222 learned_addr learned my addr 172.21.15.62:0/3079333222 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:06.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38b400640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c1593e0 0x7fa38c15d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352573082 0 0) 0x7fa38c0b14c0 con 0x7fa38c0b2b30 2024-03-23T11:20:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa370005730 con 0x7fa38c0b2b30 2024-03-23T11:20:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3656455265 0 0) 0x7fa370005730 con 0x7fa38c0b2b30 2024-03-23T11:20:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c1593e0 msgr2=0x7fa38c15d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c1593e0 0x7fa38c15d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 >> v1:172.21.15.120:6789/0 conn(0x7fa38c06a290 legacy=0x7fa38c0cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c15dee0 con 0x7fa38c0b2b30 2024-03-23T11:20:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fa3840019f0 con 0x7fa38c0b2b30 2024-03-23T11:20:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 >> v1:172.21.15.62:6789/0 conn(0x7fa38c0b2b30 legacy=0x7fa38c0b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c1593e0 0x7fa370005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa370006130 0x7fa3700069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.541+0000 7fa38aa00640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa370006f10 0x7fa370007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.542+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37000dc40 con 0x7fa370006f10 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.542+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37000ddb0 con 0x7fa370006130 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.542+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37000df70 con 0x7fa38c1593e0 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.542+0000 7fa38be00640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa370006f10 0x7fa370007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.542+0000 7fa38b400640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c1593e0 0x7fa370005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.542+0000 7fa38be00640 1 -- 172.21.15.62:0/3079333222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c1593e0 msgr2=0x7fa370005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.542+0000 7fa38be00640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c1593e0 0x7fa370005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.542+0000 7fa38be00640 1 -- 172.21.15.62:0/3079333222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa370006130 msgr2=0x7fa3700069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.542+0000 7fa38be00640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa370006130 0x7fa3700069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.542+0000 7fa38be00640 1 -- 172.21.15.62:0/3079333222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa370005730 con 0x7fa370006f10 2024-03-23T11:20:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.542+0000 7fa38b400640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa38c1593e0 0x7fa370005bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:20:06.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.543+0000 7fa38be00640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa370006f10 0x7fa370007390 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa38400a820 tx=0x7fa370007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3672423434ea042 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:06.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.544+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840072d0 con 0x7fa370006f10 2024-03-23T11:20:06.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.544+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38402f070 con 0x7fa370006f10 2024-03-23T11:20:06.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.544+0000 7fa38aa00640 1 -- 172.21.15.62:0/3079333222 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840076f0 con 0x7fa370006f10 2024-03-23T11:20:06.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.544+0000 7fa391000640 1 -- 172.21.15.62:0/3079333222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa370006f10 msgr2=0x7fa370007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:06.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.544+0000 7fa391000640 1 --2- 172.21.15.62:0/3079333222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa370006f10 0x7fa370007390 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa38400a820 tx=0x7fa370007a00 comp rx=0 tx=0).stop 2024-03-23T11:20:06.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.544+0000 7fa38be00640 1 -- 172.21.15.62:0/3079333222 reap_dead start 2024-03-23T11:20:06.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.544+0000 7fa391000640 1 -- 172.21.15.62:0/3079333222 shutdown_connections 2024-03-23T11:20:06.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.544+0000 7fa391000640 1 -- 172.21.15.62:0/3079333222 >> 172.21.15.62:0/3079333222 conn(0x7fa38c0b0090 msgr2=0x7fa38c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:06.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.544+0000 7fa391000640 1 -- 172.21.15.62:0/3079333222 shutdown_connections 2024-03-23T11:20:06.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.544+0000 7fa391000640 1 -- 172.21.15.62:0/3079333222 wait complete. 2024-03-23T11:20:06.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.546+0000 7fa391000640 1 Processor -- start 2024-03-23T11:20:06.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.546+0000 7fa391000640 1 -- start start 2024-03-23T11:20:06.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.547+0000 7fa391000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa370006f10 0x7fa38c070200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.547+0000 7fa391000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38c150300 0x7fa38c150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.547+0000 7fa391000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c0ce2b0 0x7fa38c0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.547+0000 7fa391000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c0cc100 con 0x7fa38c150300 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.547+0000 7fa391000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c0cc350 con 0x7fa38c0ce2b0 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.547+0000 7fa391000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c0cc630 con 0x7fa370006f10 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.548+0000 7fa383e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38c150300 0x7fa38c150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.548+0000 7fa383e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38c150300 0x7fa38c150760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59762/0 (socket says 172.21.15.62:59762) 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.548+0000 7fa383e00640 1 -- 172.21.15.62:0/332990627 learned_addr learned my addr 172.21.15.62:0/332990627 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.548+0000 7fa38b400640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa370006f10 0x7fa38c070200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.548+0000 7fa383e00640 1 -- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa370006f10 msgr2=0x7fa38c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.548+0000 7fa383e00640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa370006f10 0x7fa38c070200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.548+0000 7fa383e00640 1 -- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c0ce2b0 msgr2=0x7fa38c0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.548+0000 7fa383e00640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c0ce2b0 0x7fa38c0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.548+0000 7fa383e00640 1 -- 172.21.15.62:0/332990627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c0cc910 con 0x7fa38c150300 2024-03-23T11:20:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.549+0000 7fa383e00640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38c150300 0x7fa38c150760 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa37400c450 tx=0x7fa37400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:06.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.549+0000 7fa383400640 1 -- 172.21.15.62:0/332990627 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3740052d0 con 0x7fa38c150300 2024-03-23T11:20:06.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.549+0000 7fa383400640 1 -- 172.21.15.62:0/332990627 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa374005430 con 0x7fa38c150300 2024-03-23T11:20:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.549+0000 7fa391000640 1 -- 172.21.15.62:0/332990627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38c149000 con 0x7fa38c150300 2024-03-23T11:20:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.549+0000 7fa383400640 1 -- 172.21.15.62:0/332990627 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374012870 con 0x7fa38c150300 2024-03-23T11:20:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.549+0000 7fa391000640 1 -- 172.21.15.62:0/332990627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c070920 con 0x7fa38c150300 2024-03-23T11:20:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.550+0000 7fa391000640 1 -- 172.21.15.62:0/332990627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c066bb0 con 0x7fa38c150300 2024-03-23T11:20:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.551+0000 7fa383400640 1 -- 172.21.15.62:0/332990627 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa3740129d0 con 0x7fa38c150300 2024-03-23T11:20:06.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.552+0000 7fa383400640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36806a7d0 0x7fa36806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:06.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.552+0000 7fa383400640 1 -- 172.21.15.62:0/332990627 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fa374088730 con 0x7fa38c150300 2024-03-23T11:20:06.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.552+0000 7fa38b400640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36806a7d0 0x7fa36806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:06.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.554+0000 7fa38b400640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36806a7d0 0x7fa36806cc90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009a90 tx=0x7fa37c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:06.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.556+0000 7fa383400640 1 -- 172.21.15.62:0/332990627 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa374053a70 con 0x7fa38c150300 2024-03-23T11:20:06.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:06.795+0000 7fa391000640 1 -- 172.21.15.62:0/332990627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa38c070920 con 0x7fa38c150300 2024-03-23T11:20:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.295+0000 7fa383400640 1 -- 172.21.15.62:0/332990627 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa38c070920 con 0x7fa38c150300 2024-03-23T11:20:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.299+0000 7fa391000640 1 -- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36806a7d0 msgr2=0x7fa36806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.299+0000 7fa391000640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36806a7d0 0x7fa36806cc90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009a90 tx=0x7fa37c008040 comp rx=0 tx=0).stop 2024-03-23T11:20:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.299+0000 7fa391000640 1 -- 172.21.15.62:0/332990627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38c150300 msgr2=0x7fa38c150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.299+0000 7fa391000640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38c150300 0x7fa38c150760 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa37400c450 tx=0x7fa37400c910 comp rx=0 tx=0).stop 2024-03-23T11:20:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.299+0000 7fa391000640 1 -- 172.21.15.62:0/332990627 shutdown_connections 2024-03-23T11:20:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.299+0000 7fa391000640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36806a7d0 0x7fa36806cc90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.299+0000 7fa391000640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa38c150300 0x7fa38c150760 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.299+0000 7fa391000640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa38c0ce2b0 0x7fa38c0ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.299+0000 7fa391000640 1 --2- 172.21.15.62:0/332990627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa370006f10 0x7fa38c070200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.299+0000 7fa391000640 1 -- 172.21.15.62:0/332990627 >> 172.21.15.62:0/332990627 conn(0x7fa38c0b0090 msgr2=0x7fa38c0b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.300+0000 7fa391000640 1 -- 172.21.15.62:0/332990627 shutdown_connections 2024-03-23T11:20:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.300+0000 7fa391000640 1 -- 172.21.15.62:0/332990627 wait complete. 2024-03-23T11:20:07.323 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T11:20:07.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.549+0000 7fe230a00640 1 Processor -- start 2024-03-23T11:20:07.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.550+0000 7fe230a00640 1 -- start start 2024-03-23T11:20:07.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.551+0000 7fe230a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe22c154d30 0x7fe22c0ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:07.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.551+0000 7fe230a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe22c0ab850 0x7fe22c06c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:07.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.551+0000 7fe230a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c15bcb0 con 0x7fe22c0cebd0 2024-03-23T11:20:07.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.551+0000 7fe230a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22c0cf050 con 0x7fe22c0ab850 2024-03-23T11:20:07.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.551+0000 7fe230a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c0abc70 con 0x7fe22c154d30 2024-03-23T11:20:07.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.552+0000 7fe22b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe22c0ab850 0x7fe22c06c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:07.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.552+0000 7fe22a800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe22c154d30 0x7fe22c0ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:07.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.552+0000 7fe22b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe22c0ab850 0x7fe22c06c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:42726/0 (socket says 172.21.15.62:42726) 2024-03-23T11:20:07.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.552+0000 7fe22b200640 1 -- 172.21.15.62:0/986057588 learned_addr learned my addr 172.21.15.62:0/986057588 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:07.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.552+0000 7fe22b200640 1 -- 172.21.15.62:0/986057588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe22c154d30 msgr2=0x7fe22c0ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:07.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.552+0000 7fe22b200640 1 --2- 172.21.15.62:0/986057588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe22c154d30 0x7fe22c0ce690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:07.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.552+0000 7fe22b200640 1 -- 172.21.15.62:0/986057588 >> v1:172.21.15.120:6789/0 conn(0x7fe22c0cebd0 legacy=0x7fe22c0ab140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:07.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.553+0000 7fe22b200640 1 -- 172.21.15.62:0/986057588 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c15ac90 con 0x7fe22c0ab850 2024-03-23T11:20:07.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.554+0000 7fe22b200640 1 --2- 172.21.15.62:0/986057588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe22c0ab850 0x7fe22c06c2b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe224009e50 tx=0x7fe22402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5f4a88d4afbe93c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:07.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.554+0000 7fe229e00640 1 -- 172.21.15.62:0/986057588 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22402cb00 con 0x7fe22c0ab850 2024-03-23T11:20:07.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.554+0000 7fe229e00640 1 -- 172.21.15.62:0/986057588 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22402cc60 con 0x7fe22c0ab850 2024-03-23T11:20:07.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.554+0000 7fe229e00640 1 -- 172.21.15.62:0/986057588 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2240343f0 con 0x7fe22c0ab850 2024-03-23T11:20:07.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.555+0000 7fe230a00640 1 -- 172.21.15.62:0/986057588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe22c0ab850 msgr2=0x7fe22c06c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:07.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.555+0000 7fe230a00640 1 --2- 172.21.15.62:0/986057588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe22c0ab850 0x7fe22c06c2b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe224009e50 tx=0x7fe22402bf60 comp rx=0 tx=0).stop 2024-03-23T11:20:07.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.555+0000 7fe230a00640 1 -- 172.21.15.62:0/986057588 shutdown_connections 2024-03-23T11:20:07.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.555+0000 7fe230a00640 1 --2- 172.21.15.62:0/986057588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe22c0ab850 0x7fe22c06c2b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:07.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.555+0000 7fe230a00640 1 --2- 172.21.15.62:0/986057588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe22c154d30 0x7fe22c0ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:07.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.555+0000 7fe230a00640 1 -- 172.21.15.62:0/986057588 >> 172.21.15.62:0/986057588 conn(0x7fe22c0b0280 msgr2=0x7fe22c0b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:07.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.555+0000 7fe230a00640 1 -- 172.21.15.62:0/986057588 shutdown_connections 2024-03-23T11:20:07.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.555+0000 7fe230a00640 1 -- 172.21.15.62:0/986057588 wait complete. 2024-03-23T11:20:07.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.558+0000 7fe230a00640 1 Processor -- start 2024-03-23T11:20:07.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.558+0000 7fe230a00640 1 -- start start 2024-03-23T11:20:07.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.559+0000 7fe230a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe22c0ab850 0x7fe22c150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:07.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.559+0000 7fe230a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe22c0cebd0 0x7fe22c158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:07.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.559+0000 7fe230a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe22c154d30 0x7fe22c0cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.560+0000 7fe230a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c0cb690 con 0x7fe22c154d30 2024-03-23T11:20:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.560+0000 7fe230a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c0cc1d0 con 0x7fe22c0cebd0 2024-03-23T11:20:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.560+0000 7fe230a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c0cb8a0 con 0x7fe22c0ab850 2024-03-23T11:20:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.560+0000 7fe22b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe22c154d30 0x7fe22c0cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.560+0000 7fe22b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe22c154d30 0x7fe22c0cc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33244/0 (socket says 172.21.15.62:33244) 2024-03-23T11:20:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.560+0000 7fe22b200640 1 -- 172.21.15.62:0/1563350942 learned_addr learned my addr 172.21.15.62:0/1563350942 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.560+0000 7fe22a800640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe22c0ab850 0x7fe22c150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.560+0000 7fe221e00640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe22c0cebd0 0x7fe22c158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:07.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.561+0000 7fe22b200640 1 -- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe22c0ab850 msgr2=0x7fe22c150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:07.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.561+0000 7fe22b200640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe22c0ab850 0x7fe22c150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:07.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.561+0000 7fe22b200640 1 -- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe22c0cebd0 msgr2=0x7fe22c158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:07.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.561+0000 7fe22b200640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe22c0cebd0 0x7fe22c158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:07.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.561+0000 7fe22b200640 1 -- 172.21.15.62:0/1563350942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe224009b30 con 0x7fe22c154d30 2024-03-23T11:20:07.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.561+0000 7fe221e00640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe22c0cebd0 0x7fe22c158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:20:07.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.561+0000 7fe22b200640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe22c154d30 0x7fe22c0cc660 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe224036040 tx=0x7fe224004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:07.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.561+0000 7fe223400640 1 -- 172.21.15.62:0/1563350942 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224009510 con 0x7fe22c154d30 2024-03-23T11:20:07.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.561+0000 7fe223400640 1 -- 172.21.15.62:0/1563350942 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22403e450 con 0x7fe22c154d30 2024-03-23T11:20:07.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.561+0000 7fe223400640 1 -- 172.21.15.62:0/1563350942 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22403e800 con 0x7fe22c154d30 2024-03-23T11:20:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.561+0000 7fe230a00640 1 -- 172.21.15.62:0/1563350942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c0cbb80 con 0x7fe22c154d30 2024-03-23T11:20:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.562+0000 7fe230a00640 1 -- 172.21.15.62:0/1563350942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c150860 con 0x7fe22c154d30 2024-03-23T11:20:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.563+0000 7fe230a00640 1 -- 172.21.15.62:0/1563350942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22c0cc380 con 0x7fe22c154d30 2024-03-23T11:20:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.564+0000 7fe223400640 1 -- 172.21.15.62:0/1563350942 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe22403e960 con 0x7fe22c154d30 2024-03-23T11:20:07.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.565+0000 7fe223400640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe20806a7a0 0x7fe20806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:07.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.565+0000 7fe223400640 1 -- 172.21.15.62:0/1563350942 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fe2240abb60 con 0x7fe22c154d30 2024-03-23T11:20:07.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.566+0000 7fe22a800640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe20806a7a0 0x7fe20806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.567+0000 7fe22a800640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe20806a7a0 0x7fe20806cc60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe22c159fa0 tx=0x7fe21c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:07.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.570+0000 7fe223400640 1 -- 172.21.15.62:0/1563350942 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe22c0cc380 con 0x7fe22c154d30 2024-03-23T11:20:07.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:07.768+0000 7fe230a00640 1 -- 172.21.15.62:0/1563350942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe22c0cc340 con 0x7fe22c154d30 2024-03-23T11:20:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.533+0000 7fe223400640 1 -- 172.21.15.62:0/1563350942 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe224076e10 con 0x7fe22c154d30 2024-03-23T11:20:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' removed 2024-03-23T11:20:08.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.537+0000 7fe230a00640 1 -- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe20806a7a0 msgr2=0x7fe20806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:08.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.537+0000 7fe230a00640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe20806a7a0 0x7fe20806cc60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe22c159fa0 tx=0x7fe21c009190 comp rx=0 tx=0).stop 2024-03-23T11:20:08.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.537+0000 7fe230a00640 1 -- 172.21.15.62:0/1563350942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe22c154d30 msgr2=0x7fe22c0cc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:08.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.537+0000 7fe230a00640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe22c154d30 0x7fe22c0cc660 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe224036040 tx=0x7fe224004a40 comp rx=0 tx=0).stop 2024-03-23T11:20:08.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.537+0000 7fe230a00640 1 -- 172.21.15.62:0/1563350942 shutdown_connections 2024-03-23T11:20:08.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.537+0000 7fe230a00640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe20806a7a0 0x7fe20806cc60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:08.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.537+0000 7fe230a00640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe22c154d30 0x7fe22c0cc660 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:08.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.537+0000 7fe230a00640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe22c0cebd0 0x7fe22c158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:08.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.537+0000 7fe230a00640 1 --2- 172.21.15.62:0/1563350942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe22c0ab850 0x7fe22c150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:08.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.538+0000 7fe230a00640 1 -- 172.21.15.62:0/1563350942 >> 172.21.15.62:0/1563350942 conn(0x7fe22c0b0280 msgr2=0x7fe22c0b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:08.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.538+0000 7fe230a00640 1 -- 172.21.15.62:0/1563350942 shutdown_connections 2024-03-23T11:20:08.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.538+0000 7fe230a00640 1 -- 172.21.15.62:0/1563350942 wait complete. 2024-03-23T11:20:08.561 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T11:20:08.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.784+0000 7f9e2a600640 1 Processor -- start 2024-03-23T11:20:08.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.784+0000 7f9e2a600640 1 -- start start 2024-03-23T11:20:08.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.786+0000 7f9e2a600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9e24154d30 0x7f9e240ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:08.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.786+0000 7f9e2a600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9e240ab850 0x7f9e2406c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:08.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.786+0000 7f9e2a600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2415bcb0 con 0x7f9e240cebd0 2024-03-23T11:20:08.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.786+0000 7f9e2a600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e240cf080 con 0x7f9e240ab850 2024-03-23T11:20:08.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.786+0000 7f9e2a600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e240abca0 con 0x7f9e24154d30 2024-03-23T11:20:08.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.786+0000 7f9e29c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9e240ab850 0x7f9e2406c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:08.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.786+0000 7f9e29c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9e240ab850 0x7f9e2406c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33256/0 (socket says 172.21.15.62:33256) 2024-03-23T11:20:08.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.786+0000 7f9e29c00640 1 -- 172.21.15.62:0/943437541 learned_addr learned my addr 172.21.15.62:0/943437541 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:08.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.786+0000 7f9e23e00640 1 --1- 172.21.15.62:0/943437541 >> v1:172.21.15.62:6789/0 conn(0x7f9e240cebd0 0x7f9e240ab140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35576/0 (socket says 172.21.15.62:35576) 2024-03-23T11:20:08.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.786+0000 7f9e29200640 1 --2- 172.21.15.62:0/943437541 >> v2:172.21.15.120:3300/0 conn(0x7f9e24154d30 0x7f9e240ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:08.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.787+0000 7f9e29c00640 1 -- 172.21.15.62:0/943437541 >> v2:172.21.15.120:3300/0 conn(0x7f9e24154d30 msgr2=0x7f9e240ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:08.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.787+0000 7f9e29c00640 1 --2- 172.21.15.62:0/943437541 >> v2:172.21.15.120:3300/0 conn(0x7f9e24154d30 0x7f9e240ce690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:08.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.787+0000 7f9e29c00640 1 -- 172.21.15.62:0/943437541 >> v1:172.21.15.62:6789/0 conn(0x7f9e240cebd0 legacy=0x7f9e240ab140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:08.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.787+0000 7f9e29c00640 1 -- 172.21.15.62:0/943437541 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2415ac90 con 0x7f9e240ab850 2024-03-23T11:20:08.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.788+0000 7f9e29c00640 1 --2- 172.21.15.62:0/943437541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e240ab850 0x7f9e2406c2b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9e240b2a50 tx=0x7f9e1c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a83a4e1a7a7f2d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:08.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.788+0000 7f9e23400640 1 -- 172.21.15.62:0/943437541 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c02cb20 con 0x7f9e240ab850 2024-03-23T11:20:08.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.788+0000 7f9e23400640 1 -- 172.21.15.62:0/943437541 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c02cc80 con 0x7f9e240ab850 2024-03-23T11:20:08.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.788+0000 7f9e23400640 1 -- 172.21.15.62:0/943437541 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c0343f0 con 0x7f9e240ab850 2024-03-23T11:20:08.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.789+0000 7f9e2a600640 1 -- 172.21.15.62:0/943437541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e240ab850 msgr2=0x7f9e2406c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:08.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.789+0000 7f9e2a600640 1 --2- 172.21.15.62:0/943437541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e240ab850 0x7f9e2406c2b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9e240b2a50 tx=0x7f9e1c02bf80 comp rx=0 tx=0).stop 2024-03-23T11:20:08.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.789+0000 7f9e2a600640 1 -- 172.21.15.62:0/943437541 shutdown_connections 2024-03-23T11:20:08.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.789+0000 7f9e2a600640 1 --2- 172.21.15.62:0/943437541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e240ab850 0x7f9e2406c2b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:08.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.789+0000 7f9e2a600640 1 --2- 172.21.15.62:0/943437541 >> v2:172.21.15.120:3300/0 conn(0x7f9e24154d30 0x7f9e240ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:08.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.789+0000 7f9e2a600640 1 -- 172.21.15.62:0/943437541 >> 172.21.15.62:0/943437541 conn(0x7f9e240b0280 msgr2=0x7f9e240b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:08.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.789+0000 7f9e2a600640 1 -- 172.21.15.62:0/943437541 shutdown_connections 2024-03-23T11:20:08.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.789+0000 7f9e2a600640 1 -- 172.21.15.62:0/943437541 wait complete. 2024-03-23T11:20:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.791+0000 7f9e2a600640 1 Processor -- start 2024-03-23T11:20:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.791+0000 7f9e2a600640 1 -- start start 2024-03-23T11:20:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.792+0000 7f9e2a600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e240ab850 0x7f9e2414f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.792+0000 7f9e2a600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e240cebd0 0x7f9e241ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.792+0000 7f9e2a600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e24154d30 0x7f9e240cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.792+0000 7f9e2a600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24150a70 con 0x7f9e240cebd0 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.792+0000 7f9e2a600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24150d50 con 0x7f9e24154d30 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.792+0000 7f9e2a600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24151030 con 0x7f9e240ab850 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.792+0000 7f9e23e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e240cebd0 0x7f9e241ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.793+0000 7f9e23e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e240cebd0 0x7f9e241ef600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33262/0 (socket says 172.21.15.62:33262) 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.793+0000 7f9e23e00640 1 -- 172.21.15.62:0/3987303538 learned_addr learned my addr 172.21.15.62:0/3987303538 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.793+0000 7f9e29200640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e240ab850 0x7f9e2414f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.793+0000 7f9e29c00640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e24154d30 0x7f9e240cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.793+0000 7f9e23e00640 1 -- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e240ab850 msgr2=0x7f9e2414f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.793+0000 7f9e23e00640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e240ab850 0x7f9e2414f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.793+0000 7f9e23e00640 1 -- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e24154d30 msgr2=0x7f9e240cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.793+0000 7f9e23e00640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e24154d30 0x7f9e240cb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.793+0000 7f9e23e00640 1 -- 172.21.15.62:0/3987303538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c009b30 con 0x7f9e240cebd0 2024-03-23T11:20:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.793+0000 7f9e23e00640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e240cebd0 0x7f9e241ef600 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000c4d0 tx=0x7f9e1000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:08.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.794+0000 7f9e20c00640 1 -- 172.21.15.62:0/3987303538 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e10007480 con 0x7f9e240cebd0 2024-03-23T11:20:08.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.794+0000 7f9e20c00640 1 -- 172.21.15.62:0/3987303538 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e100075e0 con 0x7f9e240cebd0 2024-03-23T11:20:08.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.794+0000 7f9e2a600640 1 -- 172.21.15.62:0/3987303538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e24151310 con 0x7f9e240cebd0 2024-03-23T11:20:08.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.794+0000 7f9e20c00640 1 -- 172.21.15.62:0/3987303538 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e100049e0 con 0x7f9e240cebd0 2024-03-23T11:20:08.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.794+0000 7f9e2a600640 1 -- 172.21.15.62:0/3987303538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e240caea0 con 0x7f9e240cebd0 2024-03-23T11:20:08.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.795+0000 7f9e2a600640 1 -- 172.21.15.62:0/3987303538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e24158710 con 0x7f9e240cebd0 2024-03-23T11:20:08.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.796+0000 7f9e20c00640 1 -- 172.21.15.62:0/3987303538 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9e10004b40 con 0x7f9e240cebd0 2024-03-23T11:20:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.797+0000 7f9e20c00640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0406a7d0 0x7f9e0406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.797+0000 7f9e20c00640 1 -- 172.21.15.62:0/3987303538 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5473+0+0 (secure 0 0 0) 0x7f9e10088240 con 0x7f9e240cebd0 2024-03-23T11:20:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.797+0000 7f9e29200640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0406a7d0 0x7f9e0406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:08.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.799+0000 7f9e29200640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0406a7d0 0x7f9e0406cc90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9e18006fd0 tx=0x7f9e18008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:08.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:08.801+0000 7f9e20c00640 1 -- 172.21.15.62:0/3987303538 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e24158710 con 0x7f9e240cebd0 2024-03-23T11:20:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.002+0000 7f9e2a600640 1 -- 172.21.15.62:0/3987303538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9e24158710 con 0x7f9e240cebd0 2024-03-23T11:20:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.547+0000 7f9e20c00640 1 -- 172.21.15.62:0/3987303538 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9e24158710 con 0x7f9e240cebd0 2024-03-23T11:20:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' removed 2024-03-23T11:20:09.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.552+0000 7f9e2a600640 1 -- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0406a7d0 msgr2=0x7f9e0406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:09.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.552+0000 7f9e2a600640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0406a7d0 0x7f9e0406cc90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9e18006fd0 tx=0x7f9e18008040 comp rx=0 tx=0).stop 2024-03-23T11:20:09.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.552+0000 7f9e2a600640 1 -- 172.21.15.62:0/3987303538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e240cebd0 msgr2=0x7f9e241ef600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:09.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.552+0000 7f9e2a600640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e240cebd0 0x7f9e241ef600 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000c4d0 tx=0x7f9e1000c990 comp rx=0 tx=0).stop 2024-03-23T11:20:09.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.552+0000 7f9e2a600640 1 -- 172.21.15.62:0/3987303538 shutdown_connections 2024-03-23T11:20:09.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.552+0000 7f9e2a600640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0406a7d0 0x7f9e0406cc90 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9e18006fd0 tx=0x7f9e18008040 comp rx=0 tx=0).stop 2024-03-23T11:20:09.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.552+0000 7f9e2a600640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e240cebd0 0x7f9e241ef600 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:09.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.552+0000 7f9e2a600640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e24154d30 0x7f9e240cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:09.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.552+0000 7f9e2a600640 1 --2- 172.21.15.62:0/3987303538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e240ab850 0x7f9e2414f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:09.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.552+0000 7f9e2a600640 1 -- 172.21.15.62:0/3987303538 >> 172.21.15.62:0/3987303538 conn(0x7f9e240b0280 msgr2=0x7f9e24069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:09.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.553+0000 7f9e2a600640 1 -- 172.21.15.62:0/3987303538 shutdown_connections 2024-03-23T11:20:09.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:09.553+0000 7f9e2a600640 1 -- 172.21.15.62:0/3987303538 wait complete. 2024-03-23T11:20:09.577 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T11:20:09.578 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T11:20:09.578 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T11:20:09.578 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:20:09.578 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T11:20:09.578 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T11:20:09.578 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T11:20:09.579 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:20:09.579 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T11:20:09.579 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T11:20:09.579 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T11:20:09.579 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:20:09.579 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T11:20:09.579 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T11:20:09.579 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T11:20:09.580 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:20:09.598 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T11:20:09.598 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T11:20:09.601 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T11:20:09.601 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T11:20:09.602 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T11:20:09.602 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T11:20:09.647 INFO:tasks.ceph.mds.a:Started 2024-03-23T11:20:09.648 DEBUG:teuthology.parallel:result is None 2024-03-23T11:20:09.649 INFO:tasks.ceph.mds.d:Started 2024-03-23T11:20:09.650 DEBUG:teuthology.parallel:result is None 2024-03-23T11:20:09.654 INFO:tasks.ceph.mds.c:Started 2024-03-23T11:20:09.654 DEBUG:teuthology.parallel:result is None 2024-03-23T11:20:09.810 INFO:tasks.ceph.mds.c.smithi062.stdout:starting mds.c at 2024-03-23T11:20:09.811 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:20:09.835 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-23T11:20:10.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-23T11:20:14.644 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:20:14.653 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:20:15.602 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T11:20:15.602 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T11:20:15.610 INFO:tasks.ceph.mds.b:Started 2024-03-23T11:20:15.610 DEBUG:teuthology.parallel:result is None 2024-03-23T11:20:15.610 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T11:20:15.789 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-23T11:20:15.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.844+0000 7f7844a00640 1 Processor -- start 2024-03-23T11:20:15.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.844+0000 7f7844a00640 1 -- start start 2024-03-23T11:20:15.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.845+0000 7f7844a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f78400b2970 0x7f7840158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.845+0000 7f7844a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f78400b2210 0x7f78400cb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.845+0000 7f7844a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7840069c60 0x7f7840070090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f7844a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78400b35c0 con 0x7f78400b2970 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f7844a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78400b2d50 con 0x7f78400b2210 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f7844a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78400b3000 con 0x7f7840069c60 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f783f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f78400b2970 0x7f7840158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f783f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f78400b2970 0x7f7840158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33302/0 (socket says 172.21.15.62:33302) 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f783f400640 1 -- 172.21.15.62:0/42011184 learned_addr learned my addr 172.21.15.62:0/42011184 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f783f400640 1 -- 172.21.15.62:0/42011184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7840069c60 msgr2=0x7f7840070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f783fe00640 1 --2- 172.21.15.62:0/42011184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7840069c60 0x7f7840070090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f7836a00640 1 --2- 172.21.15.62:0/42011184 >> v2:172.21.15.120:3300/0 conn(0x7f78400b2210 0x7f78400cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f783f400640 1 --2- 172.21.15.62:0/42011184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7840069c60 0x7f7840070090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:15.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f783f400640 1 -- 172.21.15.62:0/42011184 >> v2:172.21.15.120:3300/0 conn(0x7f78400b2210 msgr2=0x7f78400cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:15.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f783f400640 1 --2- 172.21.15.62:0/42011184 >> v2:172.21.15.120:3300/0 conn(0x7f78400b2210 0x7f78400cb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:15.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f783f400640 1 -- 172.21.15.62:0/42011184 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78400b32e0 con 0x7f78400b2970 2024-03-23T11:20:15.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f783fe00640 1 --2- 172.21.15.62:0/42011184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7840069c60 0x7f7840070090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:15.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.846+0000 7f7836a00640 1 --2- 172.21.15.62:0/42011184 >> v2:172.21.15.120:3300/0 conn(0x7f78400b2210 0x7f78400cb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.847+0000 7f783f400640 1 --2- 172.21.15.62:0/42011184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78400b2970 0x7f7840158cd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f78400b0bd0 tx=0x7f783002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa201305971ef8e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.848+0000 7f783ea00640 1 -- 172.21.15.62:0/42011184 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783002edf0 con 0x7f78400b2970 2024-03-23T11:20:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.848+0000 7f783ea00640 1 -- 172.21.15.62:0/42011184 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7830036440 con 0x7f78400b2970 2024-03-23T11:20:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.848+0000 7f783ea00640 1 -- 172.21.15.62:0/42011184 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78300367d0 con 0x7f78400b2970 2024-03-23T11:20:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.848+0000 7f7844a00640 1 -- 172.21.15.62:0/42011184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78400b2970 msgr2=0x7f7840158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.848+0000 7f7844a00640 1 --2- 172.21.15.62:0/42011184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78400b2970 0x7f7840158cd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f78400b0bd0 tx=0x7f783002e040 comp rx=0 tx=0).stop 2024-03-23T11:20:15.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.849+0000 7f7844a00640 1 -- 172.21.15.62:0/42011184 shutdown_connections 2024-03-23T11:20:15.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.849+0000 7f7844a00640 1 --2- 172.21.15.62:0/42011184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7840069c60 0x7f7840070090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:15.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.849+0000 7f7844a00640 1 --2- 172.21.15.62:0/42011184 >> v2:172.21.15.120:3300/0 conn(0x7f78400b2210 0x7f78400cb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:15.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.849+0000 7f7844a00640 1 --2- 172.21.15.62:0/42011184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78400b2970 0x7f7840158cd0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:15.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.849+0000 7f7844a00640 1 -- 172.21.15.62:0/42011184 >> 172.21.15.62:0/42011184 conn(0x7f78400b0090 msgr2=0x7f78400b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:15.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.849+0000 7f7844a00640 1 -- 172.21.15.62:0/42011184 shutdown_connections 2024-03-23T11:20:15.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.849+0000 7f7844a00640 1 -- 172.21.15.62:0/42011184 wait complete. 2024-03-23T11:20:15.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.850+0000 7f7844a00640 1 Processor -- start 2024-03-23T11:20:15.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.851+0000 7f7844a00640 1 -- start start 2024-03-23T11:20:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f7844a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7840069c60 0x7f78401456e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f7844a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78400b2210 0x7f784015ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f7844a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78400b2970 0x7f784015bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f7844a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7840145c20 con 0x7f78400b2970 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f7844a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7840159fb0 con 0x7f7840069c60 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f7844a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784015a290 con 0x7f78400b2210 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f783fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78400b2970 0x7f784015bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f783fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78400b2970 0x7f784015bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33310/0 (socket says 172.21.15.62:33310) 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f783fe00640 1 -- 172.21.15.62:0/3674138761 learned_addr learned my addr 172.21.15.62:0/3674138761 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f783f400640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7840069c60 0x7f78401456e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f7836a00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78400b2210 0x7f784015ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f783fe00640 1 -- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78400b2210 msgr2=0x7f784015ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f783fe00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78400b2210 0x7f784015ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.852+0000 7f783fe00640 1 -- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7840069c60 msgr2=0x7f78401456e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.853+0000 7f783fe00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7840069c60 0x7f78401456e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.853+0000 7f783fe00640 1 -- 172.21.15.62:0/3674138761 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783000b770 con 0x7f78400b2970 2024-03-23T11:20:15.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.853+0000 7f7836a00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78400b2210 0x7f784015ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:15.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.853+0000 7f783fe00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78400b2970 0x7f784015bfb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f783800cef0 tx=0x7f783800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:15.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.853+0000 7f7837e00640 1 -- 172.21.15.62:0/3674138761 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783800d840 con 0x7f78400b2970 2024-03-23T11:20:15.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.853+0000 7f7837e00640 1 -- 172.21.15.62:0/3674138761 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783800d9a0 con 0x7f78400b2970 2024-03-23T11:20:15.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.853+0000 7f7844a00640 1 -- 172.21.15.62:0/3674138761 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784015a570 con 0x7f78400b2970 2024-03-23T11:20:15.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.853+0000 7f7837e00640 1 -- 172.21.15.62:0/3674138761 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783800dd50 con 0x7f78400b2970 2024-03-23T11:20:15.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.853+0000 7f7844a00640 1 -- 172.21.15.62:0/3674138761 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784015ab30 con 0x7f78400b2970 2024-03-23T11:20:15.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.855+0000 7f7844a00640 1 -- 172.21.15.62:0/3674138761 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7840066bb0 con 0x7f78400b2970 2024-03-23T11:20:15.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.855+0000 7f7837e00640 1 -- 172.21.15.62:0/3674138761 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f783801e3b0 con 0x7f78400b2970 2024-03-23T11:20:15.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.856+0000 7f7837e00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f781c06a7a0 0x7f781c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:15.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.856+0000 7f7837e00640 1 -- 172.21.15.62:0/3674138761 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7838089560 con 0x7f78400b2970 2024-03-23T11:20:15.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.857+0000 7f783f400640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f781c06a7a0 0x7f781c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:15.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.858+0000 7f783f400640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f781c06a7a0 0x7f781c06cc60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7830038000 tx=0x7f7830032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:15.861+0000 7f7837e00640 1 -- 172.21.15.62:0/3674138761 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7838054be0 con 0x7f78400b2970 2024-03-23T11:20:16.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.075+0000 7f7844a00640 1 -- 172.21.15.62:0/3674138761 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f78400b1ba0 con 0x7f78400b2970 2024-03-23T11:20:16.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.814+0000 7f7837e00640 1 -- 172.21.15.62:0/3674138761 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) v1 ==== 97+0+0 (secure 0 0 0) 0x7f783801a080 con 0x7f78400b2970 2024-03-23T11:20:16.815 INFO:teuthology.orchestra.run.smithi062.stderr: removed all blocklist entries 2024-03-23T11:20:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.817+0000 7f7844a00640 1 -- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f781c06a7a0 msgr2=0x7f781c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.817+0000 7f7844a00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f781c06a7a0 0x7f781c06cc60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7830038000 tx=0x7f7830032000 comp rx=0 tx=0).stop 2024-03-23T11:20:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.817+0000 7f7844a00640 1 -- 172.21.15.62:0/3674138761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78400b2970 msgr2=0x7f784015bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.817+0000 7f7844a00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78400b2970 0x7f784015bfb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f783800cef0 tx=0x7f783800a710 comp rx=0 tx=0).stop 2024-03-23T11:20:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.818+0000 7f7844a00640 1 -- 172.21.15.62:0/3674138761 shutdown_connections 2024-03-23T11:20:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.818+0000 7f7844a00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f781c06a7a0 0x7f781c06cc60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.818+0000 7f7844a00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78400b2970 0x7f784015bfb0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:16.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.818+0000 7f7844a00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78400b2210 0x7f784015ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:16.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.818+0000 7f7844a00640 1 --2- 172.21.15.62:0/3674138761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7840069c60 0x7f78401456e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:16.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.818+0000 7f7844a00640 1 -- 172.21.15.62:0/3674138761 >> 172.21.15.62:0/3674138761 conn(0x7f78400b0090 msgr2=0x7f78400ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:16.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.818+0000 7f7844a00640 1 -- 172.21.15.62:0/3674138761 shutdown_connections 2024-03-23T11:20:16.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:16.818+0000 7f7844a00640 1 -- 172.21.15.62:0/3674138761 wait complete. 2024-03-23T11:20:16.840 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T11:20:17.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.049+0000 7f1643e00640 1 Processor -- start 2024-03-23T11:20:17.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.049+0000 7f1643e00640 1 -- start start 2024-03-23T11:20:17.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.051+0000 7f1643e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1644143dd0 0x7f164415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.051+0000 7f1643e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1644070090 0x7f16440704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.051+0000 7f1643e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1644070a30 con 0x7f16440b2820 2024-03-23T11:20:17.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.051+0000 7f1643e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16440cd140 con 0x7f1644143dd0 2024-03-23T11:20:17.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.051+0000 7f1643e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16440cd3e0 con 0x7f1644070090 2024-03-23T11:20:17.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.051+0000 7f1642a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1644143dd0 0x7f164415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.051+0000 7f1643400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f16440b2820 0x7f16440b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35590/0 (socket says 172.21.15.62:35590) 2024-03-23T11:20:17.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.051+0000 7f1643400640 1 -- 172.21.15.62:0/1695431432 learned_addr learned my addr 172.21.15.62:0/1695431432 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:17.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.051+0000 7f1642a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1644143dd0 0x7f164415abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33312/0 (socket says 172.21.15.62:33312) 2024-03-23T11:20:17.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.051+0000 7f163be00640 1 --2- 172.21.15.62:0/1695431432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1644070090 0x7f16440704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.052+0000 7f1642a00640 1 -- 172.21.15.62:0/1695431432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1644070090 msgr2=0x7f16440704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:17.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.052+0000 7f1642a00640 1 --2- 172.21.15.62:0/1695431432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1644070090 0x7f16440704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.052+0000 7f1642a00640 1 -- 172.21.15.62:0/1695431432 >> v1:172.21.15.62:6789/0 conn(0x7f16440b2820 legacy=0x7f16440b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.052+0000 7f1642a00640 1 -- 172.21.15.62:0/1695431432 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16440cd6c0 con 0x7f1644143dd0 2024-03-23T11:20:17.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.052+0000 7f1642000640 1 -- 172.21.15.62:0/1695431432 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034956624 0 0) 0x7f1644070a30 con 0x7f16440b2820 2024-03-23T11:20:17.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.052+0000 7f163be00640 1 --2- 172.21.15.62:0/1695431432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1644070090 0x7f16440704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:17.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.053+0000 7f1642a00640 1 --2- 172.21.15.62:0/1695431432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1644143dd0 0x7f164415abe0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f16340097a0 tx=0x7f163402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5613be857c34c1fb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:17.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.053+0000 7f1642000640 1 -- 172.21.15.62:0/1695431432 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1634031830 con 0x7f1644143dd0 2024-03-23T11:20:17.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.053+0000 7f1642000640 1 -- 172.21.15.62:0/1695431432 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1634031990 con 0x7f1644143dd0 2024-03-23T11:20:17.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.053+0000 7f1642000640 1 -- 172.21.15.62:0/1695431432 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1634031d20 con 0x7f1644143dd0 2024-03-23T11:20:17.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.054+0000 7f1643e00640 1 -- 172.21.15.62:0/1695431432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1644143dd0 msgr2=0x7f164415abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.054+0000 7f1643e00640 1 --2- 172.21.15.62:0/1695431432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1644143dd0 0x7f164415abe0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f16340097a0 tx=0x7f163402b8d0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.054+0000 7f1643e00640 1 -- 172.21.15.62:0/1695431432 shutdown_connections 2024-03-23T11:20:17.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.054+0000 7f1643e00640 1 --2- 172.21.15.62:0/1695431432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1644070090 0x7f16440704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.054+0000 7f1643e00640 1 --2- 172.21.15.62:0/1695431432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1644143dd0 0x7f164415abe0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.054+0000 7f1643e00640 1 -- 172.21.15.62:0/1695431432 >> 172.21.15.62:0/1695431432 conn(0x7f1644066050 msgr2=0x7f1644069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:17.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.054+0000 7f1643e00640 1 -- 172.21.15.62:0/1695431432 shutdown_connections 2024-03-23T11:20:17.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.054+0000 7f1643e00640 1 -- 172.21.15.62:0/1695431432 wait complete. 2024-03-23T11:20:17.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.056+0000 7f1643e00640 1 Processor -- start 2024-03-23T11:20:17.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.056+0000 7f1643e00640 1 -- start start 2024-03-23T11:20:17.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.057+0000 7f1643e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1644070090 0x7f164406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.057+0000 7f1643e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16440b2820 0x7f164413ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.057+0000 7f1643e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1644143dd0 0x7f1644141720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.057+0000 7f1643e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164406e080 con 0x7f1644143dd0 2024-03-23T11:20:17.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.057+0000 7f1643e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164413dd00 con 0x7f1644070090 2024-03-23T11:20:17.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.057+0000 7f1643e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164413dfe0 con 0x7f16440b2820 2024-03-23T11:20:17.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.057+0000 7f1643400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1644143dd0 0x7f1644141720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.057+0000 7f1643400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1644143dd0 0x7f1644141720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33328/0 (socket says 172.21.15.62:33328) 2024-03-23T11:20:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.057+0000 7f1643400640 1 -- 172.21.15.62:0/2414390147 learned_addr learned my addr 172.21.15.62:0/2414390147 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.058+0000 7f163be00640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16440b2820 0x7f164413ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.058+0000 7f1642a00640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1644070090 0x7f164406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.058+0000 7f1643400640 1 -- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16440b2820 msgr2=0x7f164413ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.058+0000 7f1643400640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16440b2820 0x7f164413ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.058+0000 7f1643400640 1 -- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1644070090 msgr2=0x7f164406db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.058+0000 7f1643400640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1644070090 0x7f164406db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.058+0000 7f1643400640 1 -- 172.21.15.62:0/2414390147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1634009400 con 0x7f1644143dd0 2024-03-23T11:20:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.058+0000 7f163be00640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16440b2820 0x7f164413ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:20:17.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.058+0000 7f1643400640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1644143dd0 0x7f1644141720 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f163c001560 tx=0x7f163c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:17.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.059+0000 7f163b400640 1 -- 172.21.15.62:0/2414390147 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c00a640 con 0x7f1644143dd0 2024-03-23T11:20:17.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.059+0000 7f163b400640 1 -- 172.21.15.62:0/2414390147 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c00a7a0 con 0x7f1644143dd0 2024-03-23T11:20:17.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.059+0000 7f163b400640 1 -- 172.21.15.62:0/2414390147 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c00ab30 con 0x7f1644143dd0 2024-03-23T11:20:17.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.059+0000 7f1643e00640 1 -- 172.21.15.62:0/2414390147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164413e2c0 con 0x7f1644143dd0 2024-03-23T11:20:17.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.059+0000 7f1643e00640 1 -- 172.21.15.62:0/2414390147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164413b7c0 con 0x7f1644143dd0 2024-03-23T11:20:17.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.060+0000 7f1643e00640 1 -- 172.21.15.62:0/2414390147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16440b2640 con 0x7f1644143dd0 2024-03-23T11:20:17.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.061+0000 7f163b400640 1 -- 172.21.15.62:0/2414390147 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f163c00ac90 con 0x7f1644143dd0 2024-03-23T11:20:17.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.062+0000 7f163b400640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f162006a7a0 0x7f162006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.062+0000 7f163b400640 1 -- 172.21.15.62:0/2414390147 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f163c087fe0 con 0x7f1644143dd0 2024-03-23T11:20:17.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.062+0000 7f1642a00640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f162006a7a0 0x7f162006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.064+0000 7f1642a00640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f162006a7a0 0x7f162006cc60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1634036000 tx=0x7f163403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:17.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.066+0000 7f163b400640 1 -- 172.21.15.62:0/2414390147 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f163c053760 con 0x7f1644143dd0 2024-03-23T11:20:17.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-23T11:20:17.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.339+0000 7f1643e00640 1 -- 172.21.15.62:0/2414390147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f16440cbec0 con 0x7f1644143dd0 2024-03-23T11:20:17.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.340+0000 7f163b400640 1 -- 172.21.15.62:0/2414390147 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f163c018070 con 0x7f1644143dd0 2024-03-23T11:20:17.340 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:17.340 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:20:17.340 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_dump": [ 2024-03-23T11:20:17.340 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.a", 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lYNxYERAAPVHWwRHM5V7OejL294b2zg==", 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.b", 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lmOUANBAAifuZ0DUPD9az/mturTndSA==", 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.341 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.c", 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5ltqrjIhAA9+f82fvlrFlIEXqswPE0ig==", 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:20:17.342 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.d", 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5l7ly4CBAA8+oim0z+bbljfP0VH7XSaw==", 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.343 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.0", 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBiuv5lRfaQBRAAvrtCZXwt00PUQfCqqKBzMQ==", 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.1", 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBluv5lzwS/IBAAmYU8yMLtCUJNr/AyE0MJDA==", 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.344 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.2", 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBouv5lyAy1OhAAJoUsg6DFpEPxAcH5oI2exQ==", 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.345 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.3", 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBsuv5lb0YcGBAA220D0uRhAZpG4w97wqsB5A==", 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.4", 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBzuv5lC8RIGhAAH+5HaeCgQhrQMrYm9iWaYA==", 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.346 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.5", 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB2uv5lYi7ZIhAAMNIm/jPckCAbX9oEx5Li/Q==", 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.347 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.6", 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB5uv5lgoRtLBAAFT7uJtyqHtHAegOYyDH/WQ==", 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.7", 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB9uv5l/dzrCBAA3jsua0LvBPI7I551/rRHrg==", 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.348 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.0", 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5ltavQFhAAvV2RWuHOPK2T9Q7gWkYA4w==", 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.349 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.1", 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5luQ/zLRAAwjDtV64TCKnQliPq4iUAYg==", 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.2", 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lDRG1IhAAjVaXRR30G1985oMou7Ambg==", 2024-03-23T11:20:17.350 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.3", 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lomZyOBAA8BMIBD+Zhtp6Fa2RcEXCJg==", 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:20:17.351 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.admin", 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lNtZHBhAAhiAC1rdmANClOSwzrX9iSQ==", 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-mds", 2024-03-23T11:20:17.352 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQCLuv5linMhCRAA/tpSEJ5Tog30R0tqIKG7jA==", 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-mds" 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-mgr", 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQCLuv5ldpAhCRAAl5PBQsAfPyRdCXHruUDfMg==", 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-osd", 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQCLuv5lkKkhCRAAipavfYxDdeIr6ZFNPG9VXg==", 2024-03-23T11:20:17.353 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-osd" 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-rbd", 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQCLuv5lgMMhCRAAtzNNqCXmKuFkgkYMi0TiAQ==", 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-23T11:20:17.354 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQCLuv5lStwhCRAALUsBzlaFVEMP0gWyx2fJIQ==", 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-rgw", 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQCLuv5lgPYhCRAAyjtaF3bmTnDZIUndgFdPrQ==", 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.355 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.x", 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lT6+KBRAAl5TZsJGNr4gAB2cEDZo/hQ==", 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.y", 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lqnG8NRAAvyswMz5kpW0rGjzWYRTAAg==", 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.356 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:17.357 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-03-23T11:20:17.357 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:20:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 -- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f162006a7a0 msgr2=0x7f162006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f162006a7a0 0x7f162006cc60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1634036000 tx=0x7f163403c040 comp rx=0 tx=0).stop 2024-03-23T11:20:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 -- 172.21.15.62:0/2414390147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1644143dd0 msgr2=0x7f1644141720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1644143dd0 0x7f1644141720 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f163c001560 tx=0x7f163c00cc70 comp rx=0 tx=0).stop 2024-03-23T11:20:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 -- 172.21.15.62:0/2414390147 shutdown_connections 2024-03-23T11:20:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f162006a7a0 0x7f162006cc60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1644143dd0 0x7f1644141720 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16440b2820 0x7f164413ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 --2- 172.21.15.62:0/2414390147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1644070090 0x7f164406db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 -- 172.21.15.62:0/2414390147 >> 172.21.15.62:0/2414390147 conn(0x7f1644066050 msgr2=0x7f1644072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 -- 172.21.15.62:0/2414390147 shutdown_connections 2024-03-23T11:20:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.343+0000 7f1643e00640 1 -- 172.21.15.62:0/2414390147 wait complete. 2024-03-23T11:20:17.363 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-23T11:20:17.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.580+0000 7f1d03e00640 1 Processor -- start 2024-03-23T11:20:17.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.580+0000 7f1d03e00640 1 -- start start 2024-03-23T11:20:17.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.582+0000 7f1d03e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1d04159490 0x7f1d0406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.582+0000 7f1d03e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d0415da90 0x7f1d040cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.582+0000 7f1d03e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d040ce8b0 con 0x7f1d040ce420 2024-03-23T11:20:17.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.582+0000 7f1d03e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d040ced90 con 0x7f1d04159490 2024-03-23T11:20:17.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.582+0000 7f1d03e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0406e3e0 con 0x7f1d0415da90 2024-03-23T11:20:17.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.583+0000 7f1d02000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d0415da90 0x7f1d040cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.583+0000 7f1d02000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d0415da90 0x7f1d040cdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:48702/0 (socket says 172.21.15.62:48702) 2024-03-23T11:20:17.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.583+0000 7f1d02000640 1 -- 172.21.15.62:0/1270269816 learned_addr learned my addr 172.21.15.62:0/1270269816 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:17.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.583+0000 7f1d02a00640 1 --2- 172.21.15.62:0/1270269816 >> v2:172.21.15.120:3300/0 conn(0x7f1d04159490 0x7f1d0406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.583+0000 7f1d02000640 1 -- 172.21.15.62:0/1270269816 >> v2:172.21.15.120:3300/0 conn(0x7f1d04159490 msgr2=0x7f1d0406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:17.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.583+0000 7f1d02000640 1 --2- 172.21.15.62:0/1270269816 >> v2:172.21.15.120:3300/0 conn(0x7f1d04159490 0x7f1d0406cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.583+0000 7f1d02000640 1 -- 172.21.15.62:0/1270269816 >> v1:172.21.15.120:6789/0 conn(0x7f1d040ce420 legacy=0x7f1d040b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.583+0000 7f1d02000640 1 -- 172.21.15.62:0/1270269816 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0406c1c0 con 0x7f1d0415da90 2024-03-23T11:20:17.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.584+0000 7f1d02000640 1 --2- 172.21.15.62:0/1270269816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d0415da90 0x7f1d040cdee0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009870 tx=0x7f1cec02db00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c19d1bc45ca2a9be server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:17.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.585+0000 7f1d01600640 1 -- 172.21.15.62:0/1270269816 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec02f8c0 con 0x7f1d0415da90 2024-03-23T11:20:17.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.585+0000 7f1d01600640 1 -- 172.21.15.62:0/1270269816 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec02fa20 con 0x7f1d0415da90 2024-03-23T11:20:17.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.585+0000 7f1d01600640 1 -- 172.21.15.62:0/1270269816 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec02fdb0 con 0x7f1d0415da90 2024-03-23T11:20:17.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.585+0000 7f1d03e00640 1 -- 172.21.15.62:0/1270269816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d0415da90 msgr2=0x7f1d040cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.585+0000 7f1d03e00640 1 --2- 172.21.15.62:0/1270269816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d0415da90 0x7f1d040cdee0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009870 tx=0x7f1cec02db00 comp rx=0 tx=0).stop 2024-03-23T11:20:17.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.586+0000 7f1d03e00640 1 -- 172.21.15.62:0/1270269816 shutdown_connections 2024-03-23T11:20:17.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.586+0000 7f1d03e00640 1 --2- 172.21.15.62:0/1270269816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d0415da90 0x7f1d040cdee0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.586+0000 7f1d03e00640 1 --2- 172.21.15.62:0/1270269816 >> v2:172.21.15.120:3300/0 conn(0x7f1d04159490 0x7f1d0406cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.586+0000 7f1d03e00640 1 -- 172.21.15.62:0/1270269816 >> 172.21.15.62:0/1270269816 conn(0x7f1d04066050 msgr2=0x7f1d04069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:17.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.586+0000 7f1d03e00640 1 -- 172.21.15.62:0/1270269816 shutdown_connections 2024-03-23T11:20:17.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.586+0000 7f1d03e00640 1 -- 172.21.15.62:0/1270269816 wait complete. 2024-03-23T11:20:17.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.588+0000 7f1d03e00640 1 Processor -- start 2024-03-23T11:20:17.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.588+0000 7f1d03e00640 1 -- start start 2024-03-23T11:20:17.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.589+0000 7f1d03e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d040ce420 0x7f1d04157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.589+0000 7f1d03e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d04159490 0x7f1d04152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.589+0000 7f1d03e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d0415da90 0x7f1d04148880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.589+0000 7f1d03e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040ac3d0 con 0x7f1d040ce420 2024-03-23T11:20:17.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.589+0000 7f1d03e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040ac6b0 con 0x7f1d0415da90 2024-03-23T11:20:17.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.589+0000 7f1d03e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040ac990 con 0x7f1d04159490 2024-03-23T11:20:17.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.589+0000 7f1d02a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d040ce420 0x7f1d04157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.589+0000 7f1d02a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d040ce420 0x7f1d04157470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59386/0 (socket says 172.21.15.62:59386) 2024-03-23T11:20:17.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.589+0000 7f1d02a00640 1 -- 172.21.15.62:0/176297528 learned_addr learned my addr 172.21.15.62:0/176297528 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:17.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.589+0000 7f1d03400640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d0415da90 0x7f1d04148880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.589+0000 7f1d02000640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d04159490 0x7f1d04152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1d02a00640 1 -- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d04159490 msgr2=0x7f1d04152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1d02a00640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d04159490 0x7f1d04152f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1d02a00640 1 -- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d0415da90 msgr2=0x7f1d04148880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1d02a00640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d0415da90 0x7f1d04148880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1d02a00640 1 -- 172.21.15.62:0/176297528 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0094d0 con 0x7f1d040ce420 2024-03-23T11:20:17.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1d03400640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d0415da90 0x7f1d04148880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:17.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1d02000640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d04159490 0x7f1d04152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:17.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1d02a00640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d040ce420 0x7f1d04157470 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400e9b0 tx=0x7f1cf400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:17.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1cfaa00640 1 -- 172.21.15.62:0/176297528 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4009670 con 0x7f1d040ce420 2024-03-23T11:20:17.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1cfaa00640 1 -- 172.21.15.62:0/176297528 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf400c9a0 con 0x7f1d040ce420 2024-03-23T11:20:17.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1d03e00640 1 -- 172.21.15.62:0/176297528 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d040acc70 con 0x7f1d040ce420 2024-03-23T11:20:17.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1cfaa00640 1 -- 172.21.15.62:0/176297528 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf400cd50 con 0x7f1d040ce420 2024-03-23T11:20:17.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.590+0000 7f1d03e00640 1 -- 172.21.15.62:0/176297528 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d041579b0 con 0x7f1d040ce420 2024-03-23T11:20:17.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.592+0000 7f1d03e00640 1 -- 172.21.15.62:0/176297528 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d04066bb0 con 0x7f1d040ce420 2024-03-23T11:20:17.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.592+0000 7f1cfaa00640 1 -- 172.21.15.62:0/176297528 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f1cf4013470 con 0x7f1d040ce420 2024-03-23T11:20:17.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.593+0000 7f1cfaa00640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ce006a7d0 0x7f1ce006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:17.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.594+0000 7f1cfaa00640 1 -- 172.21.15.62:0/176297528 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1cf4088dc0 con 0x7f1d040ce420 2024-03-23T11:20:17.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.594+0000 7f1d02000640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ce006a7d0 0x7f1ce006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:17.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.595+0000 7f1d02000640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ce006a7d0 0x7f1ce006cc90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1cec038000 tx=0x7f1cec008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:17.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.598+0000 7f1cfaa00640 1 -- 172.21.15.62:0/176297528 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1cf4054400 con 0x7f1d040ce420 2024-03-23T11:20:17.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.875+0000 7f1d03e00640 1 -- 172.21.15.62:0/176297528 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f1d0406d560 con 0x7f1d040ce420 2024-03-23T11:20:17.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.880+0000 7f1cfaa00640 1 -- 172.21.15.62:0/176297528 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1cf40210a0 con 0x7f1d040ce420 2024-03-23T11:20:17.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 -- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ce006a7d0 msgr2=0x7f1ce006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ce006a7d0 0x7f1ce006cc90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1cec038000 tx=0x7f1cec008030 comp rx=0 tx=0).stop 2024-03-23T11:20:17.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 -- 172.21.15.62:0/176297528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d040ce420 msgr2=0x7f1d04157470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:17.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d040ce420 0x7f1d04157470 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400e9b0 tx=0x7f1cf400ee70 comp rx=0 tx=0).stop 2024-03-23T11:20:17.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 -- 172.21.15.62:0/176297528 shutdown_connections 2024-03-23T11:20:17.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1ce006a7d0 0x7f1ce006cc90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d0415da90 0x7f1d04148880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d04159490 0x7f1d04152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 --2- 172.21.15.62:0/176297528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d040ce420 0x7f1d04157470 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:17.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 -- 172.21.15.62:0/176297528 >> 172.21.15.62:0/176297528 conn(0x7f1d04066050 msgr2=0x7f1d040cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:17.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 -- 172.21.15.62:0/176297528 shutdown_connections 2024-03-23T11:20:17.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:17.883+0000 7f1d03e00640 1 -- 172.21.15.62:0/176297528 wait complete. 2024-03-23T11:20:17.902 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-23T11:20:18.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.112+0000 7f0d72800640 1 Processor -- start 2024-03-23T11:20:18.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.112+0000 7f0d72800640 1 -- start start 2024-03-23T11:20:18.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.113+0000 7f0d72800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cd7c0 0x7f0d6c0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.114+0000 7f0d72800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0d6c06dca0 0x7f0d6c06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.114+0000 7f0d72800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c0cad20 con 0x7f0d6c0cdba0 2024-03-23T11:20:18.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.114+0000 7f0d72800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6c06e130 con 0x7f0d6c06dca0 2024-03-23T11:20:18.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.114+0000 7f0d72800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c15a8a0 con 0x7f0d6c0cd7c0 2024-03-23T11:20:18.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.114+0000 7f0d63e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0d6c06dca0 0x7f0d6c06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:18.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.114+0000 7f0d6b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cd7c0 0x7f0d6c0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:18.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.114+0000 7f0d63e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0d6c06dca0 0x7f0d6c06c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:45480/0 (socket says 172.21.15.62:45480) 2024-03-23T11:20:18.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.114+0000 7f0d63e00640 1 -- 172.21.15.62:0/790084225 learned_addr learned my addr 172.21.15.62:0/790084225 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:18.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.114+0000 7f0d6b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cd7c0 0x7f0d6c0b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:48722/0 (socket says 172.21.15.62:48722) 2024-03-23T11:20:18.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.115+0000 7f0d6b400640 1 -- 172.21.15.62:0/790084225 >> v2:172.21.15.120:3300/0 conn(0x7f0d6c06dca0 msgr2=0x7f0d6c06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:18.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.115+0000 7f0d6b400640 1 --2- 172.21.15.62:0/790084225 >> v2:172.21.15.120:3300/0 conn(0x7f0d6c06dca0 0x7f0d6c06c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.115+0000 7f0d6b400640 1 -- 172.21.15.62:0/790084225 >> v1:172.21.15.120:6789/0 conn(0x7f0d6c0cdba0 legacy=0x7f0d6c0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:18.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.115+0000 7f0d6b400640 1 -- 172.21.15.62:0/790084225 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c15ab80 con 0x7f0d6c0cd7c0 2024-03-23T11:20:18.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.117+0000 7f0d6b400640 1 --2- 172.21.15.62:0/790084225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cd7c0 0x7f0d6c0b15e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0097a0 tx=0x7f0d5c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b97c00fa125b01b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.117+0000 7f0d6aa00640 1 -- 172.21.15.62:0/790084225 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c02e070 con 0x7f0d6c0cd7c0 2024-03-23T11:20:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.117+0000 7f0d6aa00640 1 -- 172.21.15.62:0/790084225 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c0315a0 con 0x7f0d6c0cd7c0 2024-03-23T11:20:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.117+0000 7f0d6aa00640 1 -- 172.21.15.62:0/790084225 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c031930 con 0x7f0d6c0cd7c0 2024-03-23T11:20:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.118+0000 7f0d72800640 1 -- 172.21.15.62:0/790084225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cd7c0 msgr2=0x7f0d6c0b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.118+0000 7f0d72800640 1 --2- 172.21.15.62:0/790084225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cd7c0 0x7f0d6c0b15e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0097a0 tx=0x7f0d5c02b840 comp rx=0 tx=0).stop 2024-03-23T11:20:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.118+0000 7f0d72800640 1 -- 172.21.15.62:0/790084225 shutdown_connections 2024-03-23T11:20:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.118+0000 7f0d72800640 1 --2- 172.21.15.62:0/790084225 >> v2:172.21.15.120:3300/0 conn(0x7f0d6c06dca0 0x7f0d6c06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.118+0000 7f0d72800640 1 --2- 172.21.15.62:0/790084225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cd7c0 0x7f0d6c0b15e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.118+0000 7f0d72800640 1 -- 172.21.15.62:0/790084225 >> 172.21.15.62:0/790084225 conn(0x7f0d6c0b02e0 msgr2=0x7f0d6c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.118+0000 7f0d72800640 1 -- 172.21.15.62:0/790084225 shutdown_connections 2024-03-23T11:20:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.118+0000 7f0d72800640 1 -- 172.21.15.62:0/790084225 wait complete. 2024-03-23T11:20:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.120+0000 7f0d72800640 1 Processor -- start 2024-03-23T11:20:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.120+0000 7f0d72800640 1 -- start start 2024-03-23T11:20:18.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.121+0000 7f0d72800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d6c0cd7c0 0x7f0d6c155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.121+0000 7f0d72800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cdba0 0x7f0d6c0a9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.121+0000 7f0d72800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d6c156a20 0x7f0d6c06cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.121+0000 7f0d72800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c0aa8e0 con 0x7f0d6c156a20 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.121+0000 7f0d72800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c1561a0 con 0x7f0d6c0cd7c0 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.121+0000 7f0d72800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c0aab80 con 0x7f0d6c0cdba0 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.121+0000 7f0d6be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d6c156a20 0x7f0d6c06cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.122+0000 7f0d6be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d6c156a20 0x7f0d6c06cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59396/0 (socket says 172.21.15.62:59396) 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.122+0000 7f0d6be00640 1 -- 172.21.15.62:0/3943391819 learned_addr learned my addr 172.21.15.62:0/3943391819 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.122+0000 7f0d6b400640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d6c0cd7c0 0x7f0d6c155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.122+0000 7f0d63e00640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cdba0 0x7f0d6c0a9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.122+0000 7f0d6be00640 1 -- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cdba0 msgr2=0x7f0d6c0a9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.122+0000 7f0d6be00640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cdba0 0x7f0d6c0a9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.122+0000 7f0d6be00640 1 -- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d6c0cd7c0 msgr2=0x7f0d6c155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.122+0000 7f0d6be00640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d6c0cd7c0 0x7f0d6c155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.122+0000 7f0d6be00640 1 -- 172.21.15.62:0/3943391819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c009400 con 0x7f0d6c156a20 2024-03-23T11:20:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.122+0000 7f0d6be00640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d6c156a20 0x7f0d6c06cc20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400c6b0 tx=0x7f0d6400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.123+0000 7f0d63400640 1 -- 172.21.15.62:0/3943391819 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d64003b70 con 0x7f0d6c156a20 2024-03-23T11:20:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.123+0000 7f0d63400640 1 -- 172.21.15.62:0/3943391819 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d64003cf0 con 0x7f0d6c156a20 2024-03-23T11:20:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.123+0000 7f0d72800640 1 -- 172.21.15.62:0/3943391819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c0aae60 con 0x7f0d6c156a20 2024-03-23T11:20:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.123+0000 7f0d63400640 1 -- 172.21.15.62:0/3943391819 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d64003ea0 con 0x7f0d6c156a20 2024-03-23T11:20:18.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.123+0000 7f0d72800640 1 -- 172.21.15.62:0/3943391819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c0ab420 con 0x7f0d6c156a20 2024-03-23T11:20:18.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.124+0000 7f0d72800640 1 -- 172.21.15.62:0/3943391819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c066c90 con 0x7f0d6c156a20 2024-03-23T11:20:18.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.125+0000 7f0d63400640 1 -- 172.21.15.62:0/3943391819 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0d64005310 con 0x7f0d6c156a20 2024-03-23T11:20:18.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.126+0000 7f0d63400640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0d4806a7a0 0x7f0d4806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.127+0000 7f0d63400640 1 -- 172.21.15.62:0/3943391819 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0d64088850 con 0x7f0d6c156a20 2024-03-23T11:20:18.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.127+0000 7f0d6b400640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0d4806a7a0 0x7f0d4806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:18.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.128+0000 7f0d6b400640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0d4806a7a0 0x7f0d4806cc60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c036000 tx=0x7f0d5c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:18.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.131+0000 7f0d63400640 1 -- 172.21.15.62:0/3943391819 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0d64053ea0 con 0x7f0d6c156a20 2024-03-23T11:20:18.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.399+0000 7f0d72800640 1 -- 172.21.15.62:0/3943391819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f0d6c06c970 con 0x7f0d6c156a20 2024-03-23T11:20:18.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.406+0000 7f0d63400640 1 -- 172.21.15.62:0/3943391819 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0d640574c0 con 0x7f0d6c156a20 2024-03-23T11:20:18.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.410+0000 7f0d72800640 1 -- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0d4806a7a0 msgr2=0x7f0d4806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:18.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.410+0000 7f0d72800640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0d4806a7a0 0x7f0d4806cc60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c036000 tx=0x7f0d5c03d040 comp rx=0 tx=0).stop 2024-03-23T11:20:18.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.410+0000 7f0d72800640 1 -- 172.21.15.62:0/3943391819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d6c156a20 msgr2=0x7f0d6c06cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:18.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.410+0000 7f0d72800640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d6c156a20 0x7f0d6c06cc20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400c6b0 tx=0x7f0d6400cb70 comp rx=0 tx=0).stop 2024-03-23T11:20:18.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.410+0000 7f0d72800640 1 -- 172.21.15.62:0/3943391819 shutdown_connections 2024-03-23T11:20:18.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.410+0000 7f0d72800640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0d4806a7a0 0x7f0d4806cc60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.410+0000 7f0d72800640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d6c156a20 0x7f0d6c06cc20 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.410+0000 7f0d72800640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d6c0cdba0 0x7f0d6c0a9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.410+0000 7f0d72800640 1 --2- 172.21.15.62:0/3943391819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d6c0cd7c0 0x7f0d6c155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.410+0000 7f0d72800640 1 -- 172.21.15.62:0/3943391819 >> 172.21.15.62:0/3943391819 conn(0x7f0d6c0b02e0 msgr2=0x7f0d6c15afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:18.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.411+0000 7f0d72800640 1 -- 172.21.15.62:0/3943391819 shutdown_connections 2024-03-23T11:20:18.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.411+0000 7f0d72800640 1 -- 172.21.15.62:0/3943391819 wait complete. 2024-03-23T11:20:18.428 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-23T11:20:18.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.642+0000 7f5dd4c00640 1 Processor -- start 2024-03-23T11:20:18.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.642+0000 7f5dd4c00640 1 -- start start 2024-03-23T11:20:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.644+0000 7f5dd4c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5dd0143dd0 0x7f5dd015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.644+0000 7f5dd4c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5dd0070090 0x7f5dd0070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.644+0000 7f5dd4c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2820 0x7f5dd00b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.644+0000 7f5dd4c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd0070a60 con 0x7f5dd0143dd0 2024-03-23T11:20:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.644+0000 7f5dd4c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd00b31b0 con 0x7f5dd0070090 2024-03-23T11:20:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.644+0000 7f5dd4c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00cd140 con 0x7f5dd00b2820 2024-03-23T11:20:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.644+0000 7f5dcf400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5dd0143dd0 0x7f5dd015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.645+0000 7f5dcf400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5dd0143dd0 0x7f5dd015abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59398/0 (socket says 172.21.15.62:59398) 2024-03-23T11:20:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.645+0000 7f5dcf400640 1 -- 172.21.15.62:0/2521211134 learned_addr learned my addr 172.21.15.62:0/2521211134 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.645+0000 7f5dcf400640 1 -- 172.21.15.62:0/2521211134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2820 msgr2=0x7f5dd00b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:18.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.645+0000 7f5dcf400640 1 --2- 172.21.15.62:0/2521211134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2820 0x7f5dd00b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.645+0000 7f5dcf400640 1 -- 172.21.15.62:0/2521211134 >> v2:172.21.15.120:3300/0 conn(0x7f5dd0070090 msgr2=0x7f5dd0070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:18.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.645+0000 7f5dcf400640 1 --2- 172.21.15.62:0/2521211134 >> v2:172.21.15.120:3300/0 conn(0x7f5dd0070090 0x7f5dd0070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.645+0000 7f5dcf400640 1 -- 172.21.15.62:0/2521211134 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd00cd3e0 con 0x7f5dd0143dd0 2024-03-23T11:20:18.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.646+0000 7f5dcf400640 1 --2- 172.21.15.62:0/2521211134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd0143dd0 0x7f5dd015abe0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0071530 tx=0x7f5dc002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ae9e234b69acdb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:18.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.646+0000 7f5dcea00640 1 -- 172.21.15.62:0/2521211134 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0031860 con 0x7f5dd0143dd0 2024-03-23T11:20:18.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.647+0000 7f5dcea00640 1 -- 172.21.15.62:0/2521211134 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc00319c0 con 0x7f5dd0143dd0 2024-03-23T11:20:18.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.647+0000 7f5dcea00640 1 -- 172.21.15.62:0/2521211134 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0031d50 con 0x7f5dd0143dd0 2024-03-23T11:20:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.647+0000 7f5dd4c00640 1 -- 172.21.15.62:0/2521211134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd0143dd0 msgr2=0x7f5dd015abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.647+0000 7f5dd4c00640 1 --2- 172.21.15.62:0/2521211134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd0143dd0 0x7f5dd015abe0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0071530 tx=0x7f5dc002b900 comp rx=0 tx=0).stop 2024-03-23T11:20:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.647+0000 7f5dd4c00640 1 -- 172.21.15.62:0/2521211134 shutdown_connections 2024-03-23T11:20:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.647+0000 7f5dd4c00640 1 --2- 172.21.15.62:0/2521211134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd00b2820 0x7f5dd00b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.647+0000 7f5dd4c00640 1 --2- 172.21.15.62:0/2521211134 >> v2:172.21.15.120:3300/0 conn(0x7f5dd0070090 0x7f5dd0070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.647+0000 7f5dd4c00640 1 --2- 172.21.15.62:0/2521211134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd0143dd0 0x7f5dd015abe0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.647+0000 7f5dd4c00640 1 -- 172.21.15.62:0/2521211134 >> 172.21.15.62:0/2521211134 conn(0x7f5dd0066050 msgr2=0x7f5dd0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.647+0000 7f5dd4c00640 1 -- 172.21.15.62:0/2521211134 shutdown_connections 2024-03-23T11:20:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.648+0000 7f5dd4c00640 1 -- 172.21.15.62:0/2521211134 wait complete. 2024-03-23T11:20:18.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.649+0000 7f5dd4c00640 1 Processor -- start 2024-03-23T11:20:18.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.649+0000 7f5dd4c00640 1 -- start start 2024-03-23T11:20:18.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.650+0000 7f5dd4c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5dd0070090 0x7f5dd015a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.650+0000 7f5dd4c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd00b2820 0x7f5dd00afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.650+0000 7f5dd4c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd0143dd0 0x7f5dd01e4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.650+0000 7f5dd4c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00ae9f0 con 0x7f5dd00b2820 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.650+0000 7f5dd4c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00aecd0 con 0x7f5dd0070090 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.650+0000 7f5dd4c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00aefb0 con 0x7f5dd0143dd0 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.651+0000 7f5dc7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd00b2820 0x7f5dd00afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.651+0000 7f5dc7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd00b2820 0x7f5dd00afd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59406/0 (socket says 172.21.15.62:59406) 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.651+0000 7f5dc7e00640 1 -- 172.21.15.62:0/3638144732 learned_addr learned my addr 172.21.15.62:0/3638144732 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.651+0000 7f5dcf400640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5dd0070090 0x7f5dd015a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.651+0000 7f5dcfe00640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd0143dd0 0x7f5dd01e4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.651+0000 7f5dc7e00640 1 -- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd0143dd0 msgr2=0x7f5dd01e4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.651+0000 7f5dc7e00640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd0143dd0 0x7f5dd01e4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.651+0000 7f5dc7e00640 1 -- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5dd0070090 msgr2=0x7f5dd015a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.651+0000 7f5dc7e00640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5dd0070090 0x7f5dd015a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.651+0000 7f5dc7e00640 1 -- 172.21.15.62:0/3638144732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00094d0 con 0x7f5dd00b2820 2024-03-23T11:20:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.651+0000 7f5dcfe00640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd0143dd0 0x7f5dd01e4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:18.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.652+0000 7f5dc7e00640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd00b2820 0x7f5dd00afd60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5db800b410 tx=0x7f5db800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:18.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.652+0000 7f5dc7400640 1 -- 172.21.15.62:0/3638144732 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db8011830 con 0x7f5dd00b2820 2024-03-23T11:20:18.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.652+0000 7f5dc7400640 1 -- 172.21.15.62:0/3638144732 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db80119b0 con 0x7f5dd00b2820 2024-03-23T11:20:18.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.652+0000 7f5dd4c00640 1 -- 172.21.15.62:0/3638144732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd00af290 con 0x7f5dd00b2820 2024-03-23T11:20:18.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.652+0000 7f5dc7400640 1 -- 172.21.15.62:0/3638144732 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db8011d60 con 0x7f5dd00b2820 2024-03-23T11:20:18.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.652+0000 7f5dd4c00640 1 -- 172.21.15.62:0/3638144732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd015c660 con 0x7f5dd00b2820 2024-03-23T11:20:18.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.653+0000 7f5dd4c00640 1 -- 172.21.15.62:0/3638144732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd015c880 con 0x7f5dd00b2820 2024-03-23T11:20:18.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.654+0000 7f5dc7400640 1 -- 172.21.15.62:0/3638144732 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5db801a400 con 0x7f5dd00b2820 2024-03-23T11:20:18.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.655+0000 7f5dc7400640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5dac06a7d0 0x7f5dac06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:18.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.655+0000 7f5dc7400640 1 -- 172.21.15.62:0/3638144732 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5db8088b10 con 0x7f5dd00b2820 2024-03-23T11:20:18.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.656+0000 7f5dcf400640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5dac06a7d0 0x7f5dac06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:18.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.657+0000 7f5dcf400640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5dac06a7d0 0x7f5dac06cc90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0036000 tx=0x7f5dc003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:18.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.659+0000 7f5dc7400640 1 -- 172.21.15.62:0/3638144732 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5dd015c880 con 0x7f5dd00b2820 2024-03-23T11:20:18.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.930+0000 7f5dd4c00640 1 -- 172.21.15.62:0/3638144732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f5dd00af7b0 con 0x7f5dd00b2820 2024-03-23T11:20:18.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.936+0000 7f5dc7400640 1 -- 172.21.15.62:0/3638144732 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5dd00af7b0 con 0x7f5dd00b2820 2024-03-23T11:20:18.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 -- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5dac06a7d0 msgr2=0x7f5dac06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:18.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5dac06a7d0 0x7f5dac06cc90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0036000 tx=0x7f5dc003c040 comp rx=0 tx=0).stop 2024-03-23T11:20:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 -- 172.21.15.62:0/3638144732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd00b2820 msgr2=0x7f5dd00afd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd00b2820 0x7f5dd00afd60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5db800b410 tx=0x7f5db800b8d0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 -- 172.21.15.62:0/3638144732 shutdown_connections 2024-03-23T11:20:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5dac06a7d0 0x7f5dac06cc90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd00b2820 0x7f5dd00afd60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5dd0143dd0 0x7f5dd01e4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 --2- 172.21.15.62:0/3638144732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5dd0070090 0x7f5dd015a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 -- 172.21.15.62:0/3638144732 >> 172.21.15.62:0/3638144732 conn(0x7f5dd0066050 msgr2=0x7f5dd00ab840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 -- 172.21.15.62:0/3638144732 shutdown_connections 2024-03-23T11:20:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:18.939+0000 7f5dd4c00640 1 -- 172.21.15.62:0/3638144732 wait complete. 2024-03-23T11:20:18.958 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-23T11:20:19.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.185+0000 7fdcdbc00640 1 Processor -- start 2024-03-23T11:20:19.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.186+0000 7fdcdbc00640 1 -- start start 2024-03-23T11:20:19.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.187+0000 7fdcdbc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdcd40cd7c0 0x7fdcd40b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.187+0000 7fdcdbc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd406dca0 0x7fdcd406c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.187+0000 7fdcdbc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdcd40cdba0 0x7fdcd40cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.187+0000 7fdcdbc00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd40cad20 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.187+0000 7fdcdbc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd406e130 con 0x7fdcd40cdba0 2024-03-23T11:20:19.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.187+0000 7fdcdbc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd415a8a0 con 0x7fdcd406dca0 2024-03-23T11:20:19.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd9200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdcd40cd7c0 0x7fdcd40b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:19.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd9200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdcd40cd7c0 0x7fdcd40b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59420/0 (socket says 172.21.15.62:59420) 2024-03-23T11:20:19.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd9200640 1 -- 172.21.15.62:0/96463057 learned_addr learned my addr 172.21.15.62:0/96463057 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:19.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd3e00640 1 --2- 172.21.15.62:0/96463057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd406dca0 0x7fdcd406c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:19.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd9c00640 1 --2- 172.21.15.62:0/96463057 >> v2:172.21.15.120:3300/0 conn(0x7fdcd40cdba0 0x7fdcd40cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:19.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd9200640 1 -- 172.21.15.62:0/96463057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd406dca0 msgr2=0x7fdcd406c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:19.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd9200640 1 --2- 172.21.15.62:0/96463057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd406dca0 0x7fdcd406c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd9200640 1 -- 172.21.15.62:0/96463057 >> v2:172.21.15.120:3300/0 conn(0x7fdcd40cdba0 msgr2=0x7fdcd40cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:19.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd9200640 1 --2- 172.21.15.62:0/96463057 >> v2:172.21.15.120:3300/0 conn(0x7fdcd40cdba0 0x7fdcd40cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd9200640 1 -- 172.21.15.62:0/96463057 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd415ab80 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd9c00640 1 --2- 172.21.15.62:0/96463057 >> v2:172.21.15.120:3300/0 conn(0x7fdcd40cdba0 0x7fdcd40cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:19.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.188+0000 7fdcd3e00640 1 --2- 172.21.15.62:0/96463057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd406dca0 0x7fdcd406c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:19.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.190+0000 7fdcd9200640 1 --2- 172.21.15.62:0/96463057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd40cd7c0 0x7fdcd40b15e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4070270 tx=0x7fdcc802b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653d7233b536142 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:19.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.190+0000 7fdcd3400640 1 -- 172.21.15.62:0/96463057 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80317f0 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.190+0000 7fdcd3400640 1 -- 172.21.15.62:0/96463057 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc8031950 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.190+0000 7fdcd3400640 1 -- 172.21.15.62:0/96463057 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8031ce0 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.191+0000 7fdcdbc00640 1 -- 172.21.15.62:0/96463057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd40cd7c0 msgr2=0x7fdcd40b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:19.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.191+0000 7fdcdbc00640 1 --2- 172.21.15.62:0/96463057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd40cd7c0 0x7fdcd40b15e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4070270 tx=0x7fdcc802b890 comp rx=0 tx=0).stop 2024-03-23T11:20:19.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.191+0000 7fdcdbc00640 1 -- 172.21.15.62:0/96463057 shutdown_connections 2024-03-23T11:20:19.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.191+0000 7fdcdbc00640 1 --2- 172.21.15.62:0/96463057 >> v2:172.21.15.120:3300/0 conn(0x7fdcd40cdba0 0x7fdcd40cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.191+0000 7fdcdbc00640 1 --2- 172.21.15.62:0/96463057 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd406dca0 0x7fdcd406c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.191+0000 7fdcdbc00640 1 --2- 172.21.15.62:0/96463057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd40cd7c0 0x7fdcd40b15e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.191+0000 7fdcdbc00640 1 -- 172.21.15.62:0/96463057 >> 172.21.15.62:0/96463057 conn(0x7fdcd40b02e0 msgr2=0x7fdcd40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:19.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.191+0000 7fdcdbc00640 1 -- 172.21.15.62:0/96463057 shutdown_connections 2024-03-23T11:20:19.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.191+0000 7fdcdbc00640 1 -- 172.21.15.62:0/96463057 wait complete. 2024-03-23T11:20:19.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.193+0000 7fdcdbc00640 1 Processor -- start 2024-03-23T11:20:19.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.193+0000 7fdcdbc00640 1 -- start start 2024-03-23T11:20:19.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcdbc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd40cd7c0 0x7fdcd41526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcdbc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd40cdba0 0x7fdcd40a8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcdbc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcd40acca0 0x7fdcd4154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcdbc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40a9d40 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcdbc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40a74d0 con 0x7fdcd40acca0 2024-03-23T11:20:19.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcdbc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40a77b0 con 0x7fdcd40cdba0 2024-03-23T11:20:19.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcd9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd40cd7c0 0x7fdcd41526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:19.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcd9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd40cd7c0 0x7fdcd41526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59424/0 (socket says 172.21.15.62:59424) 2024-03-23T11:20:19.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcd9200640 1 -- 172.21.15.62:0/3191968689 learned_addr learned my addr 172.21.15.62:0/3191968689 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:19.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcd9200640 1 -- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd40cdba0 msgr2=0x7fdcd40a8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:19.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcd9200640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd40cdba0 0x7fdcd40a8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcd9200640 1 -- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcd40acca0 msgr2=0x7fdcd4154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:19.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcd9200640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcd40acca0 0x7fdcd4154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.194+0000 7fdcd9200640 1 -- 172.21.15.62:0/3191968689 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc80094d0 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.195+0000 7fdcd9200640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd40cd7c0 0x7fdcd41526e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8000c00 tx=0x7fdcc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:19.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.195+0000 7fdcd0c00640 1 -- 172.21.15.62:0/3191968689 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8032910 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.195+0000 7fdcd0c00640 1 -- 172.21.15.62:0/3191968689 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc8032a70 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.195+0000 7fdcd0c00640 1 -- 172.21.15.62:0/3191968689 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8032e20 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.195+0000 7fdcdbc00640 1 -- 172.21.15.62:0/3191968689 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd40a7a90 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.195+0000 7fdcdbc00640 1 -- 172.21.15.62:0/3191968689 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd40a87d0 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.196+0000 7fdcdbc00640 1 -- 172.21.15.62:0/3191968689 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd40a7fb0 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.197+0000 7fdcd0c00640 1 -- 172.21.15.62:0/3191968689 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fdcc804e450 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.198+0000 7fdcd0c00640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcb406a7a0 0x7fdcb406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.198+0000 7fdcd0c00640 1 -- 172.21.15.62:0/3191968689 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdcc80aaff0 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.199+0000 7fdcd3e00640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcb406a7a0 0x7fdcb406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:19.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.200+0000 7fdcd3e00640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcb406a7a0 0x7fdcb406cc60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0006b10 tx=0x7fdcc0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:19.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.203+0000 7fdcd0c00640 1 -- 172.21.15.62:0/3191968689 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdcc80de580 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.469+0000 7fdcdbc00640 1 -- 172.21.15.62:0/3191968689 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fdcd4066c90 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.477+0000 7fdcd0c00640 1 -- 172.21.15.62:0/3191968689 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7fdcd40a7fb0 con 0x7fdcd40cd7c0 2024-03-23T11:20:19.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 -- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcb406a7a0 msgr2=0x7fdcb406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:19.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcb406a7a0 0x7fdcb406cc60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0006b10 tx=0x7fdcc0008040 comp rx=0 tx=0).stop 2024-03-23T11:20:19.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 -- 172.21.15.62:0/3191968689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd40cd7c0 msgr2=0x7fdcd41526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:19.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd40cd7c0 0x7fdcd41526e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8000c00 tx=0x7fdcc80046c0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 -- 172.21.15.62:0/3191968689 shutdown_connections 2024-03-23T11:20:19.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdcb406a7a0 0x7fdcb406cc60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdcd40acca0 0x7fdcd4154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdcd40cdba0 0x7fdcd40a8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 --2- 172.21.15.62:0/3191968689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd40cd7c0 0x7fdcd41526e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 -- 172.21.15.62:0/3191968689 >> 172.21.15.62:0/3191968689 conn(0x7fdcd40b02e0 msgr2=0x7fdcd415b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:19.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 -- 172.21.15.62:0/3191968689 shutdown_connections 2024-03-23T11:20:19.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.480+0000 7fdcdbc00640 1 -- 172.21.15.62:0/3191968689 wait complete. 2024-03-23T11:20:19.497 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-23T11:20:19.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.713+0000 7fa6bf800640 1 Processor -- start 2024-03-23T11:20:19.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.713+0000 7fa6bf800640 1 -- start start 2024-03-23T11:20:19.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.715+0000 7fa6bf800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa6b815cdc0 0x7fa6b80d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.715+0000 7fa6bf800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b806bfc0 con 0x7fa6b815c9e0 2024-03-23T11:20:19.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.715+0000 7fa6bf800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b80b5260 con 0x7fa6b80cf650 2024-03-23T11:20:19.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.715+0000 7fa6bf800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b80ce140 con 0x7fa6b815cdc0 2024-03-23T11:20:19.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.715+0000 7fa6bce00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa6b815c9e0 0x7fa6b81607c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43024/0 (socket says 172.21.15.62:43024) 2024-03-23T11:20:19.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.715+0000 7fa6bce00640 1 -- 172.21.15.62:0/4053365803 learned_addr learned my addr 172.21.15.62:0/4053365803 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:19.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.715+0000 7fa6b7e00640 1 --2- 172.21.15.62:0/4053365803 >> v2:172.21.15.62:3300/0 conn(0x7fa6b815cdc0 0x7fa6b80d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:19.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.716+0000 7fa6b7400640 1 -- 172.21.15.62:0/4053365803 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427404717 0 0) 0x7fa6b806bfc0 con 0x7fa6b815c9e0 2024-03-23T11:20:19.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.716+0000 7fa6b7400640 1 -- 172.21.15.62:0/4053365803 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a0003520 con 0x7fa6b815c9e0 2024-03-23T11:20:19.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.716+0000 7fa6b7e00640 1 -- 172.21.15.62:0/4053365803 >> v1:172.21.15.120:6789/0 conn(0x7fa6b80cf650 legacy=0x7fa6b815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:19.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.716+0000 7fa6b7e00640 1 -- 172.21.15.62:0/4053365803 >> v1:172.21.15.62:6789/0 conn(0x7fa6b815c9e0 legacy=0x7fa6b81607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:19.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.716+0000 7fa6b7e00640 1 -- 172.21.15.62:0/4053365803 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b80b4370 con 0x7fa6b815cdc0 2024-03-23T11:20:19.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.717+0000 7fa6b7e00640 1 --2- 172.21.15.62:0/4053365803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6b815cdc0 0x7fa6b80d1f20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40097a0 tx=0x7fa6a402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4f13362fb44739 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:19.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.717+0000 7fa6b7400640 1 -- 172.21.15.62:0/4053365803 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4030610 con 0x7fa6b815cdc0 2024-03-23T11:20:19.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.717+0000 7fa6b7400640 1 -- 172.21.15.62:0/4053365803 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a4030770 con 0x7fa6b815cdc0 2024-03-23T11:20:19.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.717+0000 7fa6b7400640 1 -- 172.21.15.62:0/4053365803 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4030b00 con 0x7fa6b815cdc0 2024-03-23T11:20:19.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.718+0000 7fa6bf800640 1 -- 172.21.15.62:0/4053365803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6b815cdc0 msgr2=0x7fa6b80d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:19.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.718+0000 7fa6bf800640 1 --2- 172.21.15.62:0/4053365803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6b815cdc0 0x7fa6b80d1f20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40097a0 tx=0x7fa6a402b960 comp rx=0 tx=0).stop 2024-03-23T11:20:19.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.718+0000 7fa6bf800640 1 -- 172.21.15.62:0/4053365803 shutdown_connections 2024-03-23T11:20:19.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.718+0000 7fa6bf800640 1 --2- 172.21.15.62:0/4053365803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6b815cdc0 0x7fa6b80d1f20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.718+0000 7fa6bf800640 1 -- 172.21.15.62:0/4053365803 >> 172.21.15.62:0/4053365803 conn(0x7fa6b8071570 msgr2=0x7fa6b8071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:19.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.718+0000 7fa6bf800640 1 -- 172.21.15.62:0/4053365803 shutdown_connections 2024-03-23T11:20:19.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.718+0000 7fa6bf800640 1 -- 172.21.15.62:0/4053365803 wait complete. 2024-03-23T11:20:19.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.720+0000 7fa6bf800640 1 Processor -- start 2024-03-23T11:20:19.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.720+0000 7fa6bf800640 1 -- start start 2024-03-23T11:20:19.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.721+0000 7fa6bf800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6b80cf650 0x7fa6b80cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.721+0000 7fa6bf800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6b815c9e0 0x7fa6b806b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.721+0000 7fa6bf800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6b815cdc0 0x7fa6b80c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.721+0000 7fa6bf800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b80c68e0 con 0x7fa6b80cf650 2024-03-23T11:20:19.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.721+0000 7fa6bf800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b806ba40 con 0x7fa6b815cdc0 2024-03-23T11:20:19.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.721+0000 7fa6bf800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b80c62b0 con 0x7fa6b815c9e0 2024-03-23T11:20:19.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.721+0000 7fa6bce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6b80cf650 0x7fa6b80cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:19.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.721+0000 7fa6bce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6b80cf650 0x7fa6b80cbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59442/0 (socket says 172.21.15.62:59442) 2024-03-23T11:20:19.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.721+0000 7fa6bce00640 1 -- 172.21.15.62:0/512144862 learned_addr learned my addr 172.21.15.62:0/512144862 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:19.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6b7e00640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6b815c9e0 0x7fa6b806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:19.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6bce00640 1 -- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6b815c9e0 msgr2=0x7fa6b806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:19.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6bd800640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6b815cdc0 0x7fa6b80c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:19.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6bce00640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6b815c9e0 0x7fa6b806b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6bce00640 1 -- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6b815cdc0 msgr2=0x7fa6b80c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:19.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6bce00640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6b815cdc0 0x7fa6b80c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:19.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6bce00640 1 -- 172.21.15.62:0/512144862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a4009400 con 0x7fa6b80cf650 2024-03-23T11:20:19.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6bd800640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6b815cdc0 0x7fa6b80c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:19.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6bce00640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6b80cf650 0x7fa6b80cbef0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00c500 tx=0x7fa6ac00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:19.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6b4c00640 1 -- 172.21.15.62:0/512144862 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac005ce0 con 0x7fa6b80cf650 2024-03-23T11:20:19.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6b4c00640 1 -- 172.21.15.62:0/512144862 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac005e40 con 0x7fa6b80cf650 2024-03-23T11:20:19.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6bf800640 1 -- 172.21.15.62:0/512144862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b80b3040 con 0x7fa6b80cf650 2024-03-23T11:20:19.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.722+0000 7fa6b4c00640 1 -- 172.21.15.62:0/512144862 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac007330 con 0x7fa6b80cf650 2024-03-23T11:20:19.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.723+0000 7fa6bf800640 1 -- 172.21.15.62:0/512144862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b80c6500 con 0x7fa6b80cf650 2024-03-23T11:20:19.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.724+0000 7fa6bf800640 1 -- 172.21.15.62:0/512144862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b80c6a20 con 0x7fa6b80cf650 2024-03-23T11:20:19.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.725+0000 7fa6b4c00640 1 -- 172.21.15.62:0/512144862 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa6ac007550 con 0x7fa6b80cf650 2024-03-23T11:20:19.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.725+0000 7fa6b4c00640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa69806a7a0 0x7fa69806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:19.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.725+0000 7fa6b4c00640 1 -- 172.21.15.62:0/512144862 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa6ac088130 con 0x7fa6b80cf650 2024-03-23T11:20:19.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.726+0000 7fa6b7e00640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa69806a7a0 0x7fa69806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:19.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.727+0000 7fa6b7e00640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa69806a7a0 0x7fa69806cc60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4036000 tx=0x7fa6a403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:19.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.730+0000 7fa6b4c00640 1 -- 172.21.15.62:0/512144862 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa6b80c6a20 con 0x7fa6b80cf650 2024-03-23T11:20:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:19.996+0000 7fa6bf800640 1 -- 172.21.15.62:0/512144862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fa6b80c6a20 con 0x7fa6b80cf650 2024-03-23T11:20:20.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.002+0000 7fa6b4c00640 1 -- 172.21.15.62:0/512144862 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa6b80c6a20 con 0x7fa6b80cf650 2024-03-23T11:20:20.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 -- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa69806a7a0 msgr2=0x7fa69806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa69806a7a0 0x7fa69806cc60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4036000 tx=0x7fa6a403b040 comp rx=0 tx=0).stop 2024-03-23T11:20:20.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 -- 172.21.15.62:0/512144862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6b80cf650 msgr2=0x7fa6b80cbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6b80cf650 0x7fa6b80cbef0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00c500 tx=0x7fa6ac00c9c0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 -- 172.21.15.62:0/512144862 shutdown_connections 2024-03-23T11:20:20.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa69806a7a0 0x7fa69806cc60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6b815cdc0 0x7fa6b80c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6b815c9e0 0x7fa6b806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 --2- 172.21.15.62:0/512144862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6b80cf650 0x7fa6b80cbef0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 -- 172.21.15.62:0/512144862 >> 172.21.15.62:0/512144862 conn(0x7fa6b8071570 msgr2=0x7fa6b815e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:20.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 -- 172.21.15.62:0/512144862 shutdown_connections 2024-03-23T11:20:20.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.005+0000 7fa6bf800640 1 -- 172.21.15.62:0/512144862 wait complete. 2024-03-23T11:20:20.024 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-23T11:20:20.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.242+0000 7f673f000640 1 Processor -- start 2024-03-23T11:20:20.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.242+0000 7f673f000640 1 -- start start 2024-03-23T11:20:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.244+0000 7f673f000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f673806dca0 0x7f67380b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.244+0000 7f673f000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f673806c300 0x7f67380cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.244+0000 7f673f000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67380cad20 con 0x7f67380cd7c0 2024-03-23T11:20:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.244+0000 7f673f000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673806e130 con 0x7f673806c300 2024-03-23T11:20:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.244+0000 7f673f000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673815a8a0 con 0x7f673806dca0 2024-03-23T11:20:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.244+0000 7f673d000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f673806c300 0x7f67380cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.244+0000 7f6737e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f67380cd7c0 0x7f67380cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43038/0 (socket says 172.21.15.62:43038) 2024-03-23T11:20:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.244+0000 7f673d000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f673806c300 0x7f67380cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59452/0 (socket says 172.21.15.62:59452) 2024-03-23T11:20:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.244+0000 7f6737e00640 1 -- 172.21.15.62:0/2846213117 learned_addr learned my addr 172.21.15.62:0/2846213117 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.245+0000 7f672fe00640 1 --2- 172.21.15.62:0/2846213117 >> v2:172.21.15.120:3300/0 conn(0x7f673806dca0 0x7f67380b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.245+0000 7f673d000640 1 -- 172.21.15.62:0/2846213117 >> v2:172.21.15.120:3300/0 conn(0x7f673806dca0 msgr2=0x7f67380b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:20.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.245+0000 7f673d000640 1 --2- 172.21.15.62:0/2846213117 >> v2:172.21.15.120:3300/0 conn(0x7f673806dca0 0x7f67380b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.245+0000 7f673d000640 1 -- 172.21.15.62:0/2846213117 >> v1:172.21.15.62:6789/0 conn(0x7f67380cd7c0 legacy=0x7f67380cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.245+0000 7f6737400640 1 -- 172.21.15.62:0/2846213117 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731536851 0 0) 0x7f67380cad20 con 0x7f67380cd7c0 2024-03-23T11:20:20.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.245+0000 7f673d000640 1 -- 172.21.15.62:0/2846213117 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673815ab80 con 0x7f673806c300 2024-03-23T11:20:20.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.245+0000 7f672fe00640 1 --2- 172.21.15.62:0/2846213117 >> v2:172.21.15.120:3300/0 conn(0x7f673806dca0 0x7f67380b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:20.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.246+0000 7f673d000640 1 --2- 172.21.15.62:0/2846213117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673806c300 0x7f67380cb490 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6738074040 tx=0x7f673002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de79f4b4377c8e5f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:20.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.246+0000 7f6737400640 1 -- 172.21.15.62:0/2846213117 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673002cbd0 con 0x7f673806c300 2024-03-23T11:20:20.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.246+0000 7f6737400640 1 -- 172.21.15.62:0/2846213117 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673002cd30 con 0x7f673806c300 2024-03-23T11:20:20.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.246+0000 7f6737400640 1 -- 172.21.15.62:0/2846213117 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673002cea0 con 0x7f673806c300 2024-03-23T11:20:20.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.247+0000 7f673f000640 1 -- 172.21.15.62:0/2846213117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673806c300 msgr2=0x7f67380cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.247+0000 7f673f000640 1 --2- 172.21.15.62:0/2846213117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673806c300 0x7f67380cb490 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6738074040 tx=0x7f673002be20 comp rx=0 tx=0).stop 2024-03-23T11:20:20.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.247+0000 7f673f000640 1 -- 172.21.15.62:0/2846213117 shutdown_connections 2024-03-23T11:20:20.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.247+0000 7f673f000640 1 --2- 172.21.15.62:0/2846213117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673806c300 0x7f67380cb490 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.247+0000 7f673f000640 1 --2- 172.21.15.62:0/2846213117 >> v2:172.21.15.120:3300/0 conn(0x7f673806dca0 0x7f67380b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.247+0000 7f673f000640 1 -- 172.21.15.62:0/2846213117 >> 172.21.15.62:0/2846213117 conn(0x7f67380b02e0 msgr2=0x7f67380b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:20.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.247+0000 7f673f000640 1 -- 172.21.15.62:0/2846213117 shutdown_connections 2024-03-23T11:20:20.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.247+0000 7f673f000640 1 -- 172.21.15.62:0/2846213117 wait complete. 2024-03-23T11:20:20.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.248+0000 7f673f000640 1 Processor -- start 2024-03-23T11:20:20.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.249+0000 7f673f000640 1 -- start start 2024-03-23T11:20:20.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.249+0000 7f673f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f673806c300 0x7f6738158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.249+0000 7f673f000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67380cd7c0 0x7f6738149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.249+0000 7f673f000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6738148ae0 0x7f67380ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.249+0000 7f673f000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6738159090 con 0x7f67380cd7c0 2024-03-23T11:20:20.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.249+0000 7f673f000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6738159370 con 0x7f6738148ae0 2024-03-23T11:20:20.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.249+0000 7f673f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6738159650 con 0x7f673806c300 2024-03-23T11:20:20.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.250+0000 7f672fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67380cd7c0 0x7f6738149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.250+0000 7f672fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67380cd7c0 0x7f6738149020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59456/0 (socket says 172.21.15.62:59456) 2024-03-23T11:20:20.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.250+0000 7f672fe00640 1 -- 172.21.15.62:0/513787485 learned_addr learned my addr 172.21.15.62:0/513787485 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:20.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.250+0000 7f672fe00640 1 -- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f673806c300 msgr2=0x7f6738158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.250+0000 7f672fe00640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f673806c300 0x7f6738158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.250+0000 7f672fe00640 1 -- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6738148ae0 msgr2=0x7f67380ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.250+0000 7f673d000640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6738148ae0 0x7f67380ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.250+0000 7f672fe00640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6738148ae0 0x7f67380ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.250+0000 7f672fe00640 1 -- 172.21.15.62:0/513787485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6730009a70 con 0x7f67380cd7c0 2024-03-23T11:20:20.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.250+0000 7f672fe00640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67380cd7c0 0x7f6738149020 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f672000c7f0 tx=0x7f672000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:20.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.251+0000 7f6734c00640 1 -- 172.21.15.62:0/513787485 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720007a30 con 0x7f67380cd7c0 2024-03-23T11:20:20.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.251+0000 7f6734c00640 1 -- 172.21.15.62:0/513787485 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6720007b90 con 0x7f67380cd7c0 2024-03-23T11:20:20.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.251+0000 7f6734c00640 1 -- 172.21.15.62:0/513787485 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672000ce40 con 0x7f67380cd7c0 2024-03-23T11:20:20.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.251+0000 7f673f000640 1 -- 172.21.15.62:0/513787485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6738159930 con 0x7f67380cd7c0 2024-03-23T11:20:20.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.251+0000 7f673f000640 1 -- 172.21.15.62:0/513787485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6738149560 con 0x7f67380cd7c0 2024-03-23T11:20:20.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.252+0000 7f673f000640 1 -- 172.21.15.62:0/513787485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673814f3b0 con 0x7f67380cd7c0 2024-03-23T11:20:20.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.253+0000 7f6734c00640 1 -- 172.21.15.62:0/513787485 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f67200113f0 con 0x7f67380cd7c0 2024-03-23T11:20:20.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.254+0000 7f6734c00640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f671406a7d0 0x7f671406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.254+0000 7f6734c00640 1 -- 172.21.15.62:0/513787485 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6720088c10 con 0x7f67380cd7c0 2024-03-23T11:20:20.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.255+0000 7f6737e00640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f671406a7d0 0x7f671406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.256+0000 7f6737e00640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f671406a7d0 0x7f671406cc90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6728007fd0 tx=0x7f6728007960 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:20.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.258+0000 7f6734c00640 1 -- 172.21.15.62:0/513787485 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f673814f3b0 con 0x7f67380cd7c0 2024-03-23T11:20:20.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.526+0000 7f673f000640 1 -- 172.21.15.62:0/513787485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f6738159e50 con 0x7f67380cd7c0 2024-03-23T11:20:20.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.534+0000 7f6734c00640 1 -- 172.21.15.62:0/513787485 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f6738159e50 con 0x7f67380cd7c0 2024-03-23T11:20:20.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 -- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f671406a7d0 msgr2=0x7f671406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f671406a7d0 0x7f671406cc90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6728007fd0 tx=0x7f6728007960 comp rx=0 tx=0).stop 2024-03-23T11:20:20.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 -- 172.21.15.62:0/513787485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67380cd7c0 msgr2=0x7f6738149020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67380cd7c0 0x7f6738149020 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f672000c7f0 tx=0x7f672000cbf0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 -- 172.21.15.62:0/513787485 shutdown_connections 2024-03-23T11:20:20.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f671406a7d0 0x7f671406cc90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67380cd7c0 0x7f6738149020 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6738148ae0 0x7f67380ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 --2- 172.21.15.62:0/513787485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f673806c300 0x7f6738158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 -- 172.21.15.62:0/513787485 >> 172.21.15.62:0/513787485 conn(0x7f67380b02e0 msgr2=0x7f67380702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 -- 172.21.15.62:0/513787485 shutdown_connections 2024-03-23T11:20:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.537+0000 7f673f000640 1 -- 172.21.15.62:0/513787485 wait complete. 2024-03-23T11:20:20.557 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:20:20.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.766+0000 7f8d6f200640 1 Processor -- start 2024-03-23T11:20:20.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.766+0000 7f8d6f200640 1 -- start start 2024-03-23T11:20:20.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.768+0000 7f8d6f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7015c830 0x7f8d700ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.768+0000 7f8d6f200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d701608d0 con 0x7f8d700ce250 2024-03-23T11:20:20.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.768+0000 7f8d6f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70160bb0 con 0x7f8d70160310 2024-03-23T11:20:20.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.768+0000 7f8d6f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700addb0 con 0x7f8d7015c830 2024-03-23T11:20:20.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.768+0000 7f8d6e800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8d700ce250 0x7f8d7015d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43040/0 (socket says 172.21.15.62:43040) 2024-03-23T11:20:20.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.769+0000 7f8d6e800640 1 -- 172.21.15.62:0/651615378 learned_addr learned my addr 172.21.15.62:0/651615378 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:20.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.769+0000 7f8d6de00640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7015c830 0x7f8d700ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.769+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861352021 0 0) 0x7f8d701608d0 con 0x7f8d700ce250 2024-03-23T11:20:20.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.769+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d54003520 con 0x7f8d700ce250 2024-03-23T11:20:20.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.769+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396971309 0 0) 0x7f8d70160bb0 con 0x7f8d70160310 2024-03-23T11:20:20.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.769+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d701608d0 con 0x7f8d70160310 2024-03-23T11:20:20.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.769+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 156245280 0 0) 0x7f8d54003520 con 0x7f8d700ce250 2024-03-23T11:20:20.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7015c830 msgr2=0x7f8d700ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:20.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7015c830 0x7f8d700ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 >> v1:172.21.15.120:6789/0 conn(0x7f8d70160310 legacy=0x7f8d700ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70160e90 con 0x7f8d700ce250 2024-03-23T11:20:20.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f8d680019f0 con 0x7f8d700ce250 2024-03-23T11:20:20.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 >> v1:172.21.15.62:6789/0 conn(0x7f8d700ce250 legacy=0x7f8d7015d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7015c830 0x7f8d54003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 0x7f8d540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d54004d00 0x7f8d54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5400ba90 con 0x7f8d540040e0 2024-03-23T11:20:20.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5400bc00 con 0x7f8d54004d00 2024-03-23T11:20:20.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5400bdc0 con 0x7f8d7015c830 2024-03-23T11:20:20.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6d400640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 0x7f8d540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6e800640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d54004d00 0x7f8d54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.770+0000 7f8d6de00640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7015c830 0x7f8d54003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.771+0000 7f8d6d400640 1 -- 172.21.15.62:0/651615378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7015c830 msgr2=0x7f8d54003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:20.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.771+0000 7f8d6d400640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d7015c830 0x7f8d54003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.771+0000 7f8d6d400640 1 -- 172.21.15.62:0/651615378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d54004d00 msgr2=0x7f8d54005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:20.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.771+0000 7f8d6d400640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d54004d00 0x7f8d54005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.771+0000 7f8d6d400640 1 -- 172.21.15.62:0/651615378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d54003520 con 0x7f8d540040e0 2024-03-23T11:20:20.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.772+0000 7f8d6d400640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 0x7f8d540047c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0018d0 tx=0x7f8d540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbbb33478f98321 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:20.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.772+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c02f050 con 0x7f8d540040e0 2024-03-23T11:20:20.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.772+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5c0052d0 con 0x7f8d540040e0 2024-03-23T11:20:20.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.772+0000 7f8d6ca00640 1 -- 172.21.15.62:0/651615378 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c003810 con 0x7f8d540040e0 2024-03-23T11:20:20.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.772+0000 7f8d6f200640 1 -- 172.21.15.62:0/651615378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 msgr2=0x7f8d540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.772+0000 7f8d6f200640 1 --2- 172.21.15.62:0/651615378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 0x7f8d540047c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0018d0 tx=0x7f8d540057f0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.772+0000 7f8d6e800640 1 -- 172.21.15.62:0/651615378 reap_dead start 2024-03-23T11:20:20.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.773+0000 7f8d6f200640 1 -- 172.21.15.62:0/651615378 shutdown_connections 2024-03-23T11:20:20.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.773+0000 7f8d6f200640 1 -- 172.21.15.62:0/651615378 >> 172.21.15.62:0/651615378 conn(0x7f8d70066050 msgr2=0x7f8d70069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:20.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.773+0000 7f8d6f200640 1 -- 172.21.15.62:0/651615378 shutdown_connections 2024-03-23T11:20:20.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.773+0000 7f8d6f200640 1 -- 172.21.15.62:0/651615378 wait complete. 2024-03-23T11:20:20.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.774+0000 7f8d6f200640 1 Processor -- start 2024-03-23T11:20:20.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.775+0000 7f8d6f200640 1 -- start start 2024-03-23T11:20:20.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.775+0000 7f8d6f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 0x7f8d70145810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.775+0000 7f8d6f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d700b0050 0x7f8d700b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.775+0000 7f8d6f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d700cae10 0x7f8d700cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.775+0000 7f8d6f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700b0a10 con 0x7f8d540040e0 2024-03-23T11:20:20.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.775+0000 7f8d6f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700ae0c0 con 0x7f8d700cae10 2024-03-23T11:20:20.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.775+0000 7f8d6f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70145d50 con 0x7f8d700b0050 2024-03-23T11:20:20.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.776+0000 7f8d6de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 0x7f8d70145810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.776+0000 7f8d6de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 0x7f8d70145810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59482/0 (socket says 172.21.15.62:59482) 2024-03-23T11:20:20.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.776+0000 7f8d6de00640 1 -- 172.21.15.62:0/2377900993 learned_addr learned my addr 172.21.15.62:0/2377900993 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.776+0000 7f8d6d400640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d700b0050 0x7f8d700b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.776+0000 7f8d6de00640 1 -- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d700b0050 msgr2=0x7f8d700b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.776+0000 7f8d6de00640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d700b0050 0x7f8d700b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.776+0000 7f8d6de00640 1 -- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d700cae10 msgr2=0x7f8d700cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.776+0000 7f8d6de00640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d700cae10 0x7f8d700cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.776+0000 7f8d6de00640 1 -- 172.21.15.62:0/2377900993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70154e70 con 0x7f8d540040e0 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.776+0000 7f8d6de00640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 0x7f8d70145810 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8d60011600 tx=0x7f8d60011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.777+0000 7f8d66000640 1 -- 172.21.15.62:0/2377900993 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000f8c0 con 0x7f8d540040e0 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.777+0000 7f8d66000640 1 -- 172.21.15.62:0/2377900993 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6000fa20 con 0x7f8d540040e0 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.777+0000 7f8d66000640 1 -- 172.21.15.62:0/2377900993 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000fd90 con 0x7f8d540040e0 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.777+0000 7f8d6f200640 1 -- 172.21.15.62:0/2377900993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d700cbb80 con 0x7f8d540040e0 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.777+0000 7f8d6f200640 1 -- 172.21.15.62:0/2377900993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d700add50 con 0x7f8d540040e0 2024-03-23T11:20:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.778+0000 7f8d6f200640 1 -- 172.21.15.62:0/2377900993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d700b1830 con 0x7f8d540040e0 2024-03-23T11:20:20.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.779+0000 7f8d66000640 1 -- 172.21.15.62:0/2377900993 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8d60015460 con 0x7f8d540040e0 2024-03-23T11:20:20.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.780+0000 7f8d66000640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d5006a7d0 0x7f8d5006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:20.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.780+0000 7f8d66000640 1 -- 172.21.15.62:0/2377900993 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8d600891c0 con 0x7f8d540040e0 2024-03-23T11:20:20.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.780+0000 7f8d6d400640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d5006a7d0 0x7f8d5006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:20.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.782+0000 7f8d6d400640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d5006a7d0 0x7f8d5006cc90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c034000 tx=0x7f8d5c031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:20.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.784+0000 7f8d66000640 1 -- 172.21.15.62:0/2377900993 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d60054920 con 0x7f8d540040e0 2024-03-23T11:20:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.973+0000 7f8d6f200640 1 -- 172.21.15.62:0/2377900993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d70066bb0 con 0x7f8d540040e0 2024-03-23T11:20:20.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.975+0000 7f8d66000640 1 -- 172.21.15.62:0/2377900993 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+12389 (secure 0 0 0) 0x7f8d60057f40 con 0x7f8d540040e0 2024-03-23T11:20:20.976 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:20.977 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":32,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:20:16.806629+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:20:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.977+0000 7f8d6f200640 1 -- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d5006a7d0 msgr2=0x7f8d5006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.977+0000 7f8d6f200640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d5006a7d0 0x7f8d5006cc90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c034000 tx=0x7f8d5c031000 comp rx=0 tx=0).stop 2024-03-23T11:20:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.977+0000 7f8d6f200640 1 -- 172.21.15.62:0/2377900993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 msgr2=0x7f8d70145810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.977+0000 7f8d6f200640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 0x7f8d70145810 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8d60011600 tx=0x7f8d60011ac0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.978+0000 7f8d6f200640 1 -- 172.21.15.62:0/2377900993 shutdown_connections 2024-03-23T11:20:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.978+0000 7f8d6f200640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d5006a7d0 0x7f8d5006cc90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.978+0000 7f8d6f200640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d700cae10 0x7f8d700cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.978+0000 7f8d6f200640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d700b0050 0x7f8d700b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.978+0000 7f8d6f200640 1 --2- 172.21.15.62:0/2377900993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d540040e0 0x7f8d70145810 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.978+0000 7f8d6f200640 1 -- 172.21.15.62:0/2377900993 >> 172.21.15.62:0/2377900993 conn(0x7f8d70066050 msgr2=0x7f8d7015bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.978+0000 7f8d6f200640 1 -- 172.21.15.62:0/2377900993 shutdown_connections 2024-03-23T11:20:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:20.978+0000 7f8d6f200640 1 -- 172.21.15.62:0/2377900993 wait complete. 2024-03-23T11:20:20.996 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:20:20.997 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:20:21.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.206+0000 7efcb0a00640 1 Processor -- start 2024-03-23T11:20:21.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.206+0000 7efcb0a00640 1 -- start start 2024-03-23T11:20:21.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.208+0000 7efcb0a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efcac0ce5f0 0x7efcac0b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:21.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.208+0000 7efcb0a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcac0ced90 con 0x7efcac06cc60 2024-03-23T11:20:21.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.208+0000 7efcb0a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcac06e3e0 con 0x7efcac159490 2024-03-23T11:20:21.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.208+0000 7efcb0a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcac06c1c0 con 0x7efcac0ce5f0 2024-03-23T11:20:21.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.208+0000 7efca9e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7efcac06cc60 0x7efcac0cdee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43050/0 (socket says 172.21.15.62:43050) 2024-03-23T11:20:21.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.208+0000 7efca9e00640 1 -- 172.21.15.62:0/2963508872 learned_addr learned my addr 172.21.15.62:0/2963508872 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:21.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.208+0000 7efcab200640 1 --2- 172.21.15.62:0/2963508872 >> v2:172.21.15.62:3300/0 conn(0x7efcac0ce5f0 0x7efcac0b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:21.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.209+0000 7efca9400640 1 -- 172.21.15.62:0/2963508872 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625555033 0 0) 0x7efcac0ced90 con 0x7efcac06cc60 2024-03-23T11:20:21.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.209+0000 7efca9400640 1 -- 172.21.15.62:0/2963508872 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc90003520 con 0x7efcac06cc60 2024-03-23T11:20:21.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.209+0000 7efcab200640 1 -- 172.21.15.62:0/2963508872 >> v1:172.21.15.120:6789/0 conn(0x7efcac159490 legacy=0x7efcac15da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.209+0000 7efcab200640 1 -- 172.21.15.62:0/2963508872 >> v1:172.21.15.62:6789/0 conn(0x7efcac06cc60 legacy=0x7efcac0cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.209+0000 7efcab200640 1 -- 172.21.15.62:0/2963508872 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac06d0c0 con 0x7efcac0ce5f0 2024-03-23T11:20:21.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.210+0000 7efcab200640 1 --2- 172.21.15.62:0/2963508872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcac0ce5f0 0x7efcac0b26b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7efca4009d90 tx=0x7efca402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d99b39495815ec server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:21.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.210+0000 7efca9400640 1 -- 172.21.15.62:0/2963508872 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca402cc60 con 0x7efcac0ce5f0 2024-03-23T11:20:21.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.210+0000 7efca9400640 1 -- 172.21.15.62:0/2963508872 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca402cdc0 con 0x7efcac0ce5f0 2024-03-23T11:20:21.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.210+0000 7efca9400640 1 -- 172.21.15.62:0/2963508872 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4034660 con 0x7efcac0ce5f0 2024-03-23T11:20:21.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.211+0000 7efcb0a00640 1 -- 172.21.15.62:0/2963508872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcac0ce5f0 msgr2=0x7efcac0b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.211+0000 7efcb0a00640 1 --2- 172.21.15.62:0/2963508872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcac0ce5f0 0x7efcac0b26b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7efca4009d90 tx=0x7efca402beb0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.211+0000 7efcb0a00640 1 -- 172.21.15.62:0/2963508872 shutdown_connections 2024-03-23T11:20:21.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.211+0000 7efcb0a00640 1 --2- 172.21.15.62:0/2963508872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcac0ce5f0 0x7efcac0b26b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.211+0000 7efcb0a00640 1 -- 172.21.15.62:0/2963508872 >> 172.21.15.62:0/2963508872 conn(0x7efcac066050 msgr2=0x7efcac069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:21.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.211+0000 7efcb0a00640 1 -- 172.21.15.62:0/2963508872 shutdown_connections 2024-03-23T11:20:21.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.211+0000 7efcb0a00640 1 -- 172.21.15.62:0/2963508872 wait complete. 2024-03-23T11:20:21.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.213+0000 7efcb0a00640 1 Processor -- start 2024-03-23T11:20:21.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.213+0000 7efcb0a00640 1 -- start start 2024-03-23T11:20:21.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcb0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcac06cc60 0x7efcac0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:21.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcb0a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcac0ce5f0 0x7efcac0cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:21.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcb0a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcac159490 0x7efcac15b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:21.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcb0a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac15c660 con 0x7efcac06cc60 2024-03-23T11:20:21.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcb0a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac15bdc0 con 0x7efcac0ce5f0 2024-03-23T11:20:21.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcb0a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcac15c0a0 con 0x7efcac159490 2024-03-23T11:20:21.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcaa800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcac06cc60 0x7efcac0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:21.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcaa800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcac06cc60 0x7efcac0cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59510/0 (socket says 172.21.15.62:59510) 2024-03-23T11:20:21.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcaa800640 1 -- 172.21.15.62:0/22670796 learned_addr learned my addr 172.21.15.62:0/22670796 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:21.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcab200640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcac159490 0x7efcac15b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:21.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcaa800640 1 -- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcac159490 msgr2=0x7efcac15b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcaa800640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcac159490 0x7efcac15b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efca9e00640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcac0ce5f0 0x7efcac0cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:21.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcaa800640 1 -- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcac0ce5f0 msgr2=0x7efcac0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcaa800640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcac0ce5f0 0x7efcac0cddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.214+0000 7efcaa800640 1 -- 172.21.15.62:0/22670796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca4009a70 con 0x7efcac06cc60 2024-03-23T11:20:21.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.215+0000 7efcaa800640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcac06cc60 0x7efcac0cd8b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00c400 tx=0x7efc9c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:21.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.215+0000 7efca2a00640 1 -- 172.21.15.62:0/22670796 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c0052d0 con 0x7efcac06cc60 2024-03-23T11:20:21.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.215+0000 7efca2a00640 1 -- 172.21.15.62:0/22670796 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9c005430 con 0x7efcac06cc60 2024-03-23T11:20:21.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.215+0000 7efca2a00640 1 -- 172.21.15.62:0/22670796 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c003830 con 0x7efcac06cc60 2024-03-23T11:20:21.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.215+0000 7efcb0a00640 1 -- 172.21.15.62:0/22670796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcac15c380 con 0x7efcac06cc60 2024-03-23T11:20:21.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.215+0000 7efcb0a00640 1 -- 172.21.15.62:0/22670796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcac14d1f0 con 0x7efcac06cc60 2024-03-23T11:20:21.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.216+0000 7efcb0a00640 1 -- 172.21.15.62:0/22670796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcac14d470 con 0x7efcac06cc60 2024-03-23T11:20:21.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.217+0000 7efca2a00640 1 -- 172.21.15.62:0/22670796 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7efc9c003990 con 0x7efcac06cc60 2024-03-23T11:20:21.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.218+0000 7efca2a00640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efc8806a7d0 0x7efc8806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:21.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.218+0000 7efca2a00640 1 -- 172.21.15.62:0/22670796 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efc9c087fc0 con 0x7efcac06cc60 2024-03-23T11:20:21.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.219+0000 7efca9e00640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efc8806a7d0 0x7efc8806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:21.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.220+0000 7efca9e00640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efc8806a7d0 0x7efc8806cc90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efcac154df0 tx=0x7efc94009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:21.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.222+0000 7efca2a00640 1 -- 172.21.15.62:0/22670796 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efcac14d470 con 0x7efcac06cc60 2024-03-23T11:20:21.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.421+0000 7efcb0a00640 1 -- 172.21.15.62:0/22670796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efcac0ab840 con 0x7efcac06cc60 2024-03-23T11:20:21.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.422+0000 7efca2a00640 1 -- 172.21.15.62:0/22670796 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v32) v1 ==== 93+0+10 (secure 0 0 0) 0x7efcac0ab840 con 0x7efcac06cc60 2024-03-23T11:20:21.424 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:20:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.425+0000 7efcb0a00640 1 -- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efc8806a7d0 msgr2=0x7efc8806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.425+0000 7efcb0a00640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efc8806a7d0 0x7efc8806cc90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efcac154df0 tx=0x7efc94009040 comp rx=0 tx=0).stop 2024-03-23T11:20:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.425+0000 7efcb0a00640 1 -- 172.21.15.62:0/22670796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcac06cc60 msgr2=0x7efcac0cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.425+0000 7efcb0a00640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcac06cc60 0x7efcac0cd8b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00c400 tx=0x7efc9c00c8c0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.425+0000 7efcb0a00640 1 -- 172.21.15.62:0/22670796 shutdown_connections 2024-03-23T11:20:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.426+0000 7efcb0a00640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efc8806a7d0 0x7efc8806cc90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.426+0000 7efcb0a00640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efcac159490 0x7efcac15b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.426+0000 7efcb0a00640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcac0ce5f0 0x7efcac0cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.426+0000 7efcb0a00640 1 --2- 172.21.15.62:0/22670796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcac06cc60 0x7efcac0cd8b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.426+0000 7efcb0a00640 1 -- 172.21.15.62:0/22670796 >> 172.21.15.62:0/22670796 conn(0x7efcac066050 msgr2=0x7efcac154f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:21.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.426+0000 7efcb0a00640 1 -- 172.21.15.62:0/22670796 shutdown_connections 2024-03-23T11:20:21.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.426+0000 7efcb0a00640 1 -- 172.21.15.62:0/22670796 wait complete. 2024-03-23T11:20:21.446 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T11:20:21.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.659+0000 7fa1bfe00640 1 Processor -- start 2024-03-23T11:20:21.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.660+0000 7fa1bfe00640 1 -- start start 2024-03-23T11:20:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.661+0000 7fa1bfe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa1b815cae0 0x7fa1b80d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.661+0000 7fa1bfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1b815cec0 0x7fa1b80b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.661+0000 7fa1bfe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b8160d50 con 0x7fa1b8160820 2024-03-23T11:20:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.661+0000 7fa1bfe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b8161030 con 0x7fa1b815cae0 2024-03-23T11:20:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.661+0000 7fa1bfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b8161310 con 0x7fa1b815cec0 2024-03-23T11:20:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.661+0000 7fa1bde00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa1b8160820 0x7fa1b80cf6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43054/0 (socket says 172.21.15.62:43054) 2024-03-23T11:20:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.661+0000 7fa1bde00640 1 -- 172.21.15.62:0/585499837 learned_addr learned my addr 172.21.15.62:0/585499837 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.662+0000 7fa1bca00640 1 --2- 172.21.15.62:0/585499837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1b815cec0 0x7fa1b80b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.662+0000 7fa1bd400640 1 --2- 172.21.15.62:0/585499837 >> v2:172.21.15.120:3300/0 conn(0x7fa1b815cae0 0x7fa1b80d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.662+0000 7fa1b7e00640 1 -- 172.21.15.62:0/585499837 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041541393 0 0) 0x7fa1b8160d50 con 0x7fa1b8160820 2024-03-23T11:20:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.662+0000 7fa1b7e00640 1 -- 172.21.15.62:0/585499837 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a0003520 con 0x7fa1b8160820 2024-03-23T11:20:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.662+0000 7fa1bca00640 1 -- 172.21.15.62:0/585499837 >> v2:172.21.15.120:3300/0 conn(0x7fa1b815cae0 msgr2=0x7fa1b80d2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.662+0000 7fa1bca00640 1 --2- 172.21.15.62:0/585499837 >> v2:172.21.15.120:3300/0 conn(0x7fa1b815cae0 0x7fa1b80d2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.662+0000 7fa1bca00640 1 -- 172.21.15.62:0/585499837 >> v1:172.21.15.62:6789/0 conn(0x7fa1b8160820 legacy=0x7fa1b80cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.662+0000 7fa1bca00640 1 -- 172.21.15.62:0/585499837 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b81615f0 con 0x7fa1b815cec0 2024-03-23T11:20:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.664+0000 7fa1bca00640 1 --2- 172.21.15.62:0/585499837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1b815cec0 0x7fa1b80b5bb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8009d90 tx=0x7fa1a802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c92cd5af9851f213 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.664+0000 7fa1b7e00640 1 -- 172.21.15.62:0/585499837 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a802cc50 con 0x7fa1b815cec0 2024-03-23T11:20:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.664+0000 7fa1b7e00640 1 -- 172.21.15.62:0/585499837 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a802cdb0 con 0x7fa1b815cec0 2024-03-23T11:20:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.664+0000 7fa1b7e00640 1 -- 172.21.15.62:0/585499837 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8034640 con 0x7fa1b815cec0 2024-03-23T11:20:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.665+0000 7fa1bfe00640 1 -- 172.21.15.62:0/585499837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1b815cec0 msgr2=0x7fa1b80b5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.665+0000 7fa1bfe00640 1 --2- 172.21.15.62:0/585499837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1b815cec0 0x7fa1b80b5bb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8009d90 tx=0x7fa1a802bea0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.665+0000 7fa1bfe00640 1 -- 172.21.15.62:0/585499837 shutdown_connections 2024-03-23T11:20:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.665+0000 7fa1bfe00640 1 --2- 172.21.15.62:0/585499837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1b815cec0 0x7fa1b80b5bb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.665+0000 7fa1bfe00640 1 --2- 172.21.15.62:0/585499837 >> v2:172.21.15.120:3300/0 conn(0x7fa1b815cae0 0x7fa1b80d2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.665+0000 7fa1bfe00640 1 -- 172.21.15.62:0/585499837 >> 172.21.15.62:0/585499837 conn(0x7fa1b8071570 msgr2=0x7fa1b8071950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.665+0000 7fa1bfe00640 1 -- 172.21.15.62:0/585499837 shutdown_connections 2024-03-23T11:20:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.665+0000 7fa1bfe00640 1 -- 172.21.15.62:0/585499837 wait complete. 2024-03-23T11:20:21.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.667+0000 7fa1bfe00640 1 Processor -- start 2024-03-23T11:20:21.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.667+0000 7fa1bfe00640 1 -- start start 2024-03-23T11:20:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.668+0000 7fa1bfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1b815cae0 0x7fa1b81f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.668+0000 7fa1bfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b815cec0 0x7fa1b8157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.668+0000 7fa1bfe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1b8160820 0x7fa1b80b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.668+0000 7fa1bfe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b80b37d0 con 0x7fa1b815cec0 2024-03-23T11:20:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.668+0000 7fa1bfe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b8157ba0 con 0x7fa1b8160820 2024-03-23T11:20:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.668+0000 7fa1bfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b80b34a0 con 0x7fa1b815cae0 2024-03-23T11:20:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.668+0000 7fa1bca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b815cec0 0x7fa1b8157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.668+0000 7fa1bca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b815cec0 0x7fa1b8157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59518/0 (socket says 172.21.15.62:59518) 2024-03-23T11:20:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.668+0000 7fa1bca00640 1 -- 172.21.15.62:0/1486675416 learned_addr learned my addr 172.21.15.62:0/1486675416 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:21.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1bd400640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1b815cae0 0x7fa1b81f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:21.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1bde00640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1b8160820 0x7fa1b80b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:21.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1bca00640 1 -- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1b815cae0 msgr2=0x7fa1b81f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1bca00640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1b815cae0 0x7fa1b81f1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1bca00640 1 -- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1b8160820 msgr2=0x7fa1b80b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1bca00640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1b8160820 0x7fa1b80b2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1bca00640 1 -- 172.21.15.62:0/1486675416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a8009a70 con 0x7fa1b815cec0 2024-03-23T11:20:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1bde00640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1b8160820 0x7fa1b80b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1bca00640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b815cec0 0x7fa1b8157660 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8000f80 tx=0x7fa1a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1b5600640 1 -- 172.21.15.62:0/1486675416 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8004880 con 0x7fa1b815cec0 2024-03-23T11:20:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1b5600640 1 -- 172.21.15.62:0/1486675416 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a8009450 con 0x7fa1b815cec0 2024-03-23T11:20:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1b5600640 1 -- 172.21.15.62:0/1486675416 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8034640 con 0x7fa1b815cec0 2024-03-23T11:20:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.669+0000 7fa1bfe00640 1 -- 172.21.15.62:0/1486675416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b80cc690 con 0x7fa1b815cec0 2024-03-23T11:20:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.670+0000 7fa1bfe00640 1 -- 172.21.15.62:0/1486675416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b80ccc50 con 0x7fa1b815cec0 2024-03-23T11:20:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.671+0000 7fa1bfe00640 1 -- 172.21.15.62:0/1486675416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b8160170 con 0x7fa1b815cec0 2024-03-23T11:20:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.672+0000 7fa1b5600640 1 -- 172.21.15.62:0/1486675416 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa1a803b030 con 0x7fa1b815cec0 2024-03-23T11:20:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.672+0000 7fa1b5600640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa19c06a7a0 0x7fa19c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.673+0000 7fa1b5600640 1 -- 172.21.15.62:0/1486675416 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa1a80aaff0 con 0x7fa1b815cec0 2024-03-23T11:20:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.673+0000 7fa1bd400640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa19c06a7a0 0x7fa19c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.674+0000 7fa1bd400640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa19c06a7a0 0x7fa19c06cc60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009a10 tx=0x7fa1ac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.677+0000 7fa1b5600640 1 -- 172.21.15.62:0/1486675416 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa1a80de580 con 0x7fa1b815cec0 2024-03-23T11:20:21.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.881+0000 7fa1bfe00640 1 -- 172.21.15.62:0/1486675416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa1b8066c20 con 0x7fa1b815cec0 2024-03-23T11:20:21.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.882+0000 7fa1b5600640 1 -- 172.21.15.62:0/1486675416 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v32) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa1a804b060 con 0x7fa1b815cec0 2024-03-23T11:20:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:21.884 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:20:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:20:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "poolnum": 1, 2024-03-23T11:20:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "poolname": ".mgr" 2024-03-23T11:20:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:21.884 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:20:21.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.885+0000 7fa1bfe00640 1 -- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa19c06a7a0 msgr2=0x7fa19c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.885+0000 7fa1bfe00640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa19c06a7a0 0x7fa19c06cc60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009a10 tx=0x7fa1ac008040 comp rx=0 tx=0).stop 2024-03-23T11:20:21.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.885+0000 7fa1bfe00640 1 -- 172.21.15.62:0/1486675416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b815cec0 msgr2=0x7fa1b8157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:21.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.885+0000 7fa1bfe00640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b815cec0 0x7fa1b8157660 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8000f80 tx=0x7fa1a8004a40 comp rx=0 tx=0).stop 2024-03-23T11:20:21.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.885+0000 7fa1bfe00640 1 -- 172.21.15.62:0/1486675416 shutdown_connections 2024-03-23T11:20:21.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.885+0000 7fa1bfe00640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa19c06a7a0 0x7fa19c06cc60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.885+0000 7fa1bfe00640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b815cec0 0x7fa1b8157660 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.885+0000 7fa1bfe00640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1b8160820 0x7fa1b80b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.885+0000 7fa1bfe00640 1 --2- 172.21.15.62:0/1486675416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1b815cae0 0x7fa1b81f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:21.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.885+0000 7fa1bfe00640 1 -- 172.21.15.62:0/1486675416 >> 172.21.15.62:0/1486675416 conn(0x7fa1b8071570 msgr2=0x7fa1b815d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:21.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.885+0000 7fa1bfe00640 1 -- 172.21.15.62:0/1486675416 shutdown_connections 2024-03-23T11:20:21.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:21.886+0000 7fa1bfe00640 1 -- 172.21.15.62:0/1486675416 wait complete. 2024-03-23T11:20:21.904 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T11:20:21.904 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T11:20:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.117+0000 7f4e1b000640 1 Processor -- start 2024-03-23T11:20:22.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.118+0000 7f4e1b000640 1 -- start start 2024-03-23T11:20:22.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.119+0000 7f4e1b000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4e14143270 0x7f4e14159d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:22.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.119+0000 7f4e1b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e140cd1a0 0x7f4e140cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:22.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.119+0000 7f4e1b000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e140cdb30 con 0x7f4e14142e90 2024-03-23T11:20:22.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.119+0000 7f4e1b000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e1406cf70 con 0x7f4e14143270 2024-03-23T11:20:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.119+0000 7f4e1b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e140c4310 con 0x7f4e140cd1a0 2024-03-23T11:20:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.120+0000 7f4e13400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4e14143270 0x7f4e14159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.120+0000 7f4e13e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4e14142e90 0x7f4e140c38c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43066/0 (socket says 172.21.15.62:43066) 2024-03-23T11:20:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.120+0000 7f4e13e00640 1 -- 172.21.15.62:0/3449756890 learned_addr learned my addr 172.21.15.62:0/3449756890 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.120+0000 7f4e19000640 1 --2- 172.21.15.62:0/3449756890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e140cd1a0 0x7f4e140cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.120+0000 7f4e13400640 1 -- 172.21.15.62:0/3449756890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e140cd1a0 msgr2=0x7f4e140cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.120+0000 7f4e13400640 1 --2- 172.21.15.62:0/3449756890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e140cd1a0 0x7f4e140cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.120+0000 7f4e13400640 1 -- 172.21.15.62:0/3449756890 >> v1:172.21.15.62:6789/0 conn(0x7f4e14142e90 legacy=0x7f4e140c38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.120+0000 7f4e13400640 1 -- 172.21.15.62:0/3449756890 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1406c8b0 con 0x7f4e14143270 2024-03-23T11:20:22.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.121+0000 7f4e13400640 1 --2- 172.21.15.62:0/3449756890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e14143270 0x7f4e14159d30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0097a0 tx=0x7f4dfc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e49e39f78d6a24af server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:22.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.122+0000 7f4e12a00640 1 -- 172.21.15.62:0/3449756890 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc0316f0 con 0x7f4e14143270 2024-03-23T11:20:22.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.122+0000 7f4e12a00640 1 -- 172.21.15.62:0/3449756890 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dfc031850 con 0x7f4e14143270 2024-03-23T11:20:22.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.122+0000 7f4e12a00640 1 -- 172.21.15.62:0/3449756890 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc031be0 con 0x7f4e14143270 2024-03-23T11:20:22.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.122+0000 7f4e1b000640 1 -- 172.21.15.62:0/3449756890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e14143270 msgr2=0x7f4e14159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:22.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.122+0000 7f4e1b000640 1 --2- 172.21.15.62:0/3449756890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e14143270 0x7f4e14159d30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0097a0 tx=0x7f4dfc02b790 comp rx=0 tx=0).stop 2024-03-23T11:20:22.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.123+0000 7f4e1b000640 1 -- 172.21.15.62:0/3449756890 shutdown_connections 2024-03-23T11:20:22.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.123+0000 7f4e1b000640 1 --2- 172.21.15.62:0/3449756890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e140cd1a0 0x7f4e140cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:22.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.123+0000 7f4e1b000640 1 --2- 172.21.15.62:0/3449756890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e14143270 0x7f4e14159d30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:22.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.123+0000 7f4e1b000640 1 -- 172.21.15.62:0/3449756890 >> 172.21.15.62:0/3449756890 conn(0x7f4e140b17d0 msgr2=0x7f4e140b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:22.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.123+0000 7f4e1b000640 1 -- 172.21.15.62:0/3449756890 shutdown_connections 2024-03-23T11:20:22.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.123+0000 7f4e1b000640 1 -- 172.21.15.62:0/3449756890 wait complete. 2024-03-23T11:20:22.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.124+0000 7f4e1b000640 1 Processor -- start 2024-03-23T11:20:22.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.125+0000 7f4e1b000640 1 -- start start 2024-03-23T11:20:22.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e1b000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e140cd1a0 0x7f4e140b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:22.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e1b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e14142e90 0x7f4e140a9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:22.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e1b000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e14143270 0x7f4e140aa380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:22.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e1b000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e140ca5b0 con 0x7f4e140cd1a0 2024-03-23T11:20:22.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e1b000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e140c9a30 con 0x7f4e14143270 2024-03-23T11:20:22.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e1b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e140c9d10 con 0x7f4e14142e90 2024-03-23T11:20:22.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e13e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e140cd1a0 0x7f4e140b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:22.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e13e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e140cd1a0 0x7f4e140b0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59538/0 (socket says 172.21.15.62:59538) 2024-03-23T11:20:22.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e13e00640 1 -- 172.21.15.62:0/1684725177 learned_addr learned my addr 172.21.15.62:0/1684725177 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:22.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e13e00640 1 -- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e14142e90 msgr2=0x7f4e140a9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:22.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e13400640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e14142e90 0x7f4e140a9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:22.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e13e00640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e14142e90 0x7f4e140a9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:22.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e13e00640 1 -- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e14143270 msgr2=0x7f4e140aa380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:22.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e19000640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e14143270 0x7f4e140aa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:22.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e13e00640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e14143270 0x7f4e140aa380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:22.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e13e00640 1 -- 172.21.15.62:0/1684725177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc009400 con 0x7f4e140cd1a0 2024-03-23T11:20:22.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.126+0000 7f4e13400640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e14142e90 0x7f4e140a9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:22.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.127+0000 7f4e13e00640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e140cd1a0 0x7f4e140b0580 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400cba0 tx=0x7f4e0400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:22.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.127+0000 7f4e0be00640 1 -- 172.21.15.62:0/1684725177 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0400ad40 con 0x7f4e140cd1a0 2024-03-23T11:20:22.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.127+0000 7f4e0be00640 1 -- 172.21.15.62:0/1684725177 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e04012400 con 0x7f4e140cd1a0 2024-03-23T11:20:22.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.127+0000 7f4e1b000640 1 -- 172.21.15.62:0/1684725177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e140c9ff0 con 0x7f4e140cd1a0 2024-03-23T11:20:22.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.127+0000 7f4e0be00640 1 -- 172.21.15.62:0/1684725177 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0400aea0 con 0x7f4e140cd1a0 2024-03-23T11:20:22.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.127+0000 7f4e1b000640 1 -- 172.21.15.62:0/1684725177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e140b0ac0 con 0x7f4e140cd1a0 2024-03-23T11:20:22.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.128+0000 7f4e1b000640 1 -- 172.21.15.62:0/1684725177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e14066c90 con 0x7f4e140cd1a0 2024-03-23T11:20:22.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.129+0000 7f4e0be00640 1 -- 172.21.15.62:0/1684725177 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4e04007070 con 0x7f4e140cd1a0 2024-03-23T11:20:22.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.130+0000 7f4e0be00640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4df006a7a0 0x7f4df006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:22.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.130+0000 7f4e0be00640 1 -- 172.21.15.62:0/1684725177 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4e04088810 con 0x7f4e140cd1a0 2024-03-23T11:20:22.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.131+0000 7f4e13400640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4df006a7a0 0x7f4df006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:22.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.132+0000 7f4e13400640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4df006a7a0 0x7f4df006cc60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc036000 tx=0x7f4dfc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:22.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.135+0000 7f4e0be00640 1 -- 172.21.15.62:0/1684725177 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4e04054030 con 0x7f4e140cd1a0 2024-03-23T11:20:22.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:22.338+0000 7f4e1b000640 1 -- 172.21.15.62:0/1684725177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4e140cdb30 con 0x7f4e140cd1a0 2024-03-23T11:20:23.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.219+0000 7f4e0be00640 1 -- 172.21.15.62:0/1684725177 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v33) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4e140cdb30 con 0x7f4e140cd1a0 2024-03-23T11:20:23.221 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' created 2024-03-23T11:20:23.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.223+0000 7f4e1b000640 1 -- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4df006a7a0 msgr2=0x7f4df006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:23.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.223+0000 7f4e1b000640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4df006a7a0 0x7f4df006cc60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc036000 tx=0x7f4dfc008030 comp rx=0 tx=0).stop 2024-03-23T11:20:23.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.223+0000 7f4e1b000640 1 -- 172.21.15.62:0/1684725177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e140cd1a0 msgr2=0x7f4e140b0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:23.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.224+0000 7f4e1b000640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e140cd1a0 0x7f4e140b0580 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400cba0 tx=0x7f4e0400a430 comp rx=0 tx=0).stop 2024-03-23T11:20:23.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.224+0000 7f4e1b000640 1 -- 172.21.15.62:0/1684725177 shutdown_connections 2024-03-23T11:20:23.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.225+0000 7f4e1b000640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4df006a7a0 0x7f4df006cc60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:23.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.225+0000 7f4e1b000640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e14143270 0x7f4e140aa380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:23.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.225+0000 7f4e1b000640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e14142e90 0x7f4e140a9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:23.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.225+0000 7f4e1b000640 1 --2- 172.21.15.62:0/1684725177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e140cd1a0 0x7f4e140b0580 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:23.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.225+0000 7f4e1b000640 1 -- 172.21.15.62:0/1684725177 >> 172.21.15.62:0/1684725177 conn(0x7f4e140b17d0 msgr2=0x7f4e14141400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:23.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.226+0000 7f4e1b000640 1 -- 172.21.15.62:0/1684725177 shutdown_connections 2024-03-23T11:20:23.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.226+0000 7f4e1b000640 1 -- 172.21.15.62:0/1684725177 wait complete. 2024-03-23T11:20:23.249 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T11:20:23.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.464+0000 7f7cff200640 1 Processor -- start 2024-03-23T11:20:23.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.464+0000 7f7cff200640 1 -- start start 2024-03-23T11:20:23.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.466+0000 7f7cff200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7cf815de20 0x7f7cf80b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.466+0000 7f7cff200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf80c5390 0x7f7cf815e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.466+0000 7f7cff200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf8070230 con 0x7f7cf815e820 2024-03-23T11:20:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.466+0000 7f7cff200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf806ce30 con 0x7f7cf815de20 2024-03-23T11:20:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.466+0000 7f7cff200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf806d110 con 0x7f7cf80c5390 2024-03-23T11:20:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.466+0000 7f7cf7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7cf815de20 0x7f7cf80b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.467+0000 7f7cf7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7cf815de20 0x7f7cf80b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59556/0 (socket says 172.21.15.62:59556) 2024-03-23T11:20:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.467+0000 7f7cf7e00640 1 -- 172.21.15.62:0/2500032478 learned_addr learned my addr 172.21.15.62:0/2500032478 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:23.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.467+0000 7f7cf7400640 1 --2- 172.21.15.62:0/2500032478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf80c5390 0x7f7cf815e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:23.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.467+0000 7f7cf7e00640 1 -- 172.21.15.62:0/2500032478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf80c5390 msgr2=0x7f7cf815e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:23.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.467+0000 7f7cf7e00640 1 --2- 172.21.15.62:0/2500032478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf80c5390 0x7f7cf815e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:23.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.467+0000 7f7cf7e00640 1 -- 172.21.15.62:0/2500032478 >> v1:172.21.15.120:6789/0 conn(0x7f7cf815e820 legacy=0x7f7cf80cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:23.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.467+0000 7f7cf7e00640 1 -- 172.21.15.62:0/2500032478 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf806d3f0 con 0x7f7cf815de20 2024-03-23T11:20:23.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.468+0000 7f7cf7e00640 1 --2- 172.21.15.62:0/2500032478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf815de20 0x7f7cf80b06d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009870 tx=0x7f7ce802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeec02c2d5a65c94 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:23.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.469+0000 7f7cf6a00640 1 -- 172.21.15.62:0/2500032478 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80317c0 con 0x7f7cf815de20 2024-03-23T11:20:23.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.469+0000 7f7cf6a00640 1 -- 172.21.15.62:0/2500032478 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce8031920 con 0x7f7cf815de20 2024-03-23T11:20:23.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.469+0000 7f7cf6a00640 1 -- 172.21.15.62:0/2500032478 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8031cb0 con 0x7f7cf815de20 2024-03-23T11:20:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.469+0000 7f7cff200640 1 -- 172.21.15.62:0/2500032478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf815de20 msgr2=0x7f7cf80b06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.469+0000 7f7cff200640 1 --2- 172.21.15.62:0/2500032478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf815de20 0x7f7cf80b06d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009870 tx=0x7f7ce802b860 comp rx=0 tx=0).stop 2024-03-23T11:20:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.470+0000 7f7cff200640 1 -- 172.21.15.62:0/2500032478 shutdown_connections 2024-03-23T11:20:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.470+0000 7f7cff200640 1 --2- 172.21.15.62:0/2500032478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf80c5390 0x7f7cf815e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.470+0000 7f7cff200640 1 --2- 172.21.15.62:0/2500032478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf815de20 0x7f7cf80b06d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.470+0000 7f7cff200640 1 -- 172.21.15.62:0/2500032478 >> 172.21.15.62:0/2500032478 conn(0x7f7cf8065a20 msgr2=0x7f7cf8065e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.470+0000 7f7cff200640 1 -- 172.21.15.62:0/2500032478 shutdown_connections 2024-03-23T11:20:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.470+0000 7f7cff200640 1 -- 172.21.15.62:0/2500032478 wait complete. 2024-03-23T11:20:23.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.471+0000 7f7cff200640 1 Processor -- start 2024-03-23T11:20:23.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.472+0000 7f7cff200640 1 -- start start 2024-03-23T11:20:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.472+0000 7f7cff200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf80c5390 0x7f7cf80a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.472+0000 7f7cff200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cf815de20 0x7f7cf80ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cff200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf815e820 0x7f7cf8147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cff200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf80cb8d0 con 0x7f7cf815e820 2024-03-23T11:20:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cff200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf80a8e20 con 0x7f7cf815de20 2024-03-23T11:20:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cff200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf80cd100 con 0x7f7cf80c5390 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cfd200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf815e820 0x7f7cf8147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cfd200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf815e820 0x7f7cf8147f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59564/0 (socket says 172.21.15.62:59564) 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cfd200640 1 -- 172.21.15.62:0/3765078208 learned_addr learned my addr 172.21.15.62:0/3765078208 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cf7e00640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf80c5390 0x7f7cf80a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cfd200640 1 -- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf80c5390 msgr2=0x7f7cf80a88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cfd200640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf80c5390 0x7f7cf80a88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cfd200640 1 -- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cf815de20 msgr2=0x7f7cf80ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cf7400640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cf815de20 0x7f7cf80ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cfd200640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cf815de20 0x7f7cf80ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.473+0000 7f7cfd200640 1 -- 172.21.15.62:0/3765078208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce80094d0 con 0x7f7cf815e820 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.474+0000 7f7cfd200640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf815e820 0x7f7cf8147f10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000b9e0 tx=0x7f7cf000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.474+0000 7f7cefe00640 1 -- 172.21.15.62:0/3765078208 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0003b70 con 0x7f7cf815e820 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.474+0000 7f7cefe00640 1 -- 172.21.15.62:0/3765078208 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf0003cd0 con 0x7f7cf815e820 2024-03-23T11:20:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.474+0000 7f7cefe00640 1 -- 172.21.15.62:0/3765078208 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf000cb60 con 0x7f7cf815e820 2024-03-23T11:20:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.474+0000 7f7cff200640 1 -- 172.21.15.62:0/3765078208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf80cbb20 con 0x7f7cf815e820 2024-03-23T11:20:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.474+0000 7f7cff200640 1 -- 172.21.15.62:0/3765078208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf80cc0e0 con 0x7f7cf815e820 2024-03-23T11:20:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.475+0000 7f7cff200640 1 -- 172.21.15.62:0/3765078208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf80cdfb0 con 0x7f7cf815e820 2024-03-23T11:20:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.476+0000 7f7cefe00640 1 -- 172.21.15.62:0/3765078208 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7cf000ccc0 con 0x7f7cf815e820 2024-03-23T11:20:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.477+0000 7f7cefe00640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cd806a7a0 0x7f7cd806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.477+0000 7f7cefe00640 1 -- 172.21.15.62:0/3765078208 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f7cf0088960 con 0x7f7cf815e820 2024-03-23T11:20:23.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.478+0000 7f7cf7e00640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cd806a7a0 0x7f7cd806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:23.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.479+0000 7f7cf7e00640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cd806a7a0 0x7f7cd806cc60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8036000 tx=0x7f7ce803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:23.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.482+0000 7f7cefe00640 1 -- 172.21.15.62:0/3765078208 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7cf0053f50 con 0x7f7cf815e820 2024-03-23T11:20:23.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:23.672+0000 7f7cff200640 1 -- 172.21.15.62:0/3765078208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7cf8066bb0 con 0x7f7cf815e820 2024-03-23T11:20:24.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.239+0000 7f7cefe00640 1 -- 172.21.15.62:0/3765078208 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v34) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7cf0057570 con 0x7f7cf815e820 2024-03-23T11:20:24.243 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' created 2024-03-23T11:20:24.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.247+0000 7f7cff200640 1 -- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cd806a7a0 msgr2=0x7f7cd806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:24.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.247+0000 7f7cff200640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cd806a7a0 0x7f7cd806cc60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8036000 tx=0x7f7ce803d040 comp rx=0 tx=0).stop 2024-03-23T11:20:24.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.247+0000 7f7cff200640 1 -- 172.21.15.62:0/3765078208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf815e820 msgr2=0x7f7cf8147f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:24.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.247+0000 7f7cff200640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf815e820 0x7f7cf8147f10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000b9e0 tx=0x7f7cf000bea0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.248+0000 7f7cff200640 1 -- 172.21.15.62:0/3765078208 shutdown_connections 2024-03-23T11:20:24.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.248+0000 7f7cff200640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cd806a7a0 0x7f7cd806cc60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.248+0000 7f7cff200640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf815e820 0x7f7cf8147f10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.248+0000 7f7cff200640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cf815de20 0x7f7cf80ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.248+0000 7f7cff200640 1 --2- 172.21.15.62:0/3765078208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cf80c5390 0x7f7cf80a88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.248+0000 7f7cff200640 1 -- 172.21.15.62:0/3765078208 >> 172.21.15.62:0/3765078208 conn(0x7f7cf8065a20 msgr2=0x7f7cf80c3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:24.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.253+0000 7f7cff200640 1 -- 172.21.15.62:0/3765078208 shutdown_connections 2024-03-23T11:20:24.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.254+0000 7f7cff200640 1 -- 172.21.15.62:0/3765078208 wait complete. 2024-03-23T11:20:24.287 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T11:20:24.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.472+0000 7f609d800640 1 Processor -- start 2024-03-23T11:20:24.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.472+0000 7f609d800640 1 -- start start 2024-03-23T11:20:24.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.473+0000 7f609d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f609815fcf0 0x7f60980d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.473+0000 7f609d800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6098160420 0x7f6098160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.473+0000 7f609d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60980d06d0 0x7f60980d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.473+0000 7f609d800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6098160dc0 con 0x7f60980d06d0 2024-03-23T11:20:24.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f609d800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60980d3540 con 0x7f6098160420 2024-03-23T11:20:24.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f609d800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60980ce510 con 0x7f609815fcf0 2024-03-23T11:20:24.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f609ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60980d06d0 0x7f60980d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f609ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60980d06d0 0x7f60980d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59574/0 (socket says 172.21.15.62:59574) 2024-03-23T11:20:24.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f609ce00640 1 -- 172.21.15.62:0/3979078750 learned_addr learned my addr 172.21.15.62:0/3979078750 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:24.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f6097e00640 1 --2- 172.21.15.62:0/3979078750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f609815fcf0 0x7f60980d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f608f400640 1 --2- 172.21.15.62:0/3979078750 >> v2:172.21.15.120:3300/0 conn(0x7f6098160420 0x7f6098160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f609ce00640 1 -- 172.21.15.62:0/3979078750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f609815fcf0 msgr2=0x7f60980d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f609ce00640 1 --2- 172.21.15.62:0/3979078750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f609815fcf0 0x7f60980d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f609ce00640 1 -- 172.21.15.62:0/3979078750 >> v2:172.21.15.120:3300/0 conn(0x7f6098160420 msgr2=0x7f6098160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f609ce00640 1 --2- 172.21.15.62:0/3979078750 >> v2:172.21.15.120:3300/0 conn(0x7f6098160420 0x7f6098160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f609ce00640 1 -- 172.21.15.62:0/3979078750 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60980cdc70 con 0x7f60980d06d0 2024-03-23T11:20:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.474+0000 7f608f400640 1 --2- 172.21.15.62:0/3979078750 >> v2:172.21.15.120:3300/0 conn(0x7f6098160420 0x7f6098160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.476+0000 7f609ce00640 1 --2- 172.21.15.62:0/3979078750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60980d06d0 0x7f60980d0b20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f60980664e0 tx=0x7f609002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fdc11257269bfb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.476+0000 7f6097400640 1 -- 172.21.15.62:0/3979078750 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609002c9e0 con 0x7f60980d06d0 2024-03-23T11:20:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.476+0000 7f6097400640 1 -- 172.21.15.62:0/3979078750 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609002cb40 con 0x7f60980d06d0 2024-03-23T11:20:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.476+0000 7f6097400640 1 -- 172.21.15.62:0/3979078750 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60900343f0 con 0x7f60980d06d0 2024-03-23T11:20:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.476+0000 7f609d800640 1 -- 172.21.15.62:0/3979078750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60980d06d0 msgr2=0x7f60980d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.476+0000 7f609d800640 1 --2- 172.21.15.62:0/3979078750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60980d06d0 0x7f60980d0b20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f60980664e0 tx=0x7f609002be40 comp rx=0 tx=0).stop 2024-03-23T11:20:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.477+0000 7f609d800640 1 -- 172.21.15.62:0/3979078750 shutdown_connections 2024-03-23T11:20:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.477+0000 7f609d800640 1 --2- 172.21.15.62:0/3979078750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60980d06d0 0x7f60980d0b20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.477+0000 7f609d800640 1 --2- 172.21.15.62:0/3979078750 >> v2:172.21.15.120:3300/0 conn(0x7f6098160420 0x7f6098160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.477+0000 7f609d800640 1 --2- 172.21.15.62:0/3979078750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f609815fcf0 0x7f60980d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.477+0000 7f609d800640 1 -- 172.21.15.62:0/3979078750 >> 172.21.15.62:0/3979078750 conn(0x7f6098089f60 msgr2=0x7f60980d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.477+0000 7f609d800640 1 -- 172.21.15.62:0/3979078750 shutdown_connections 2024-03-23T11:20:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.477+0000 7f609d800640 1 -- 172.21.15.62:0/3979078750 wait complete. 2024-03-23T11:20:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.479+0000 7f609d800640 1 Processor -- start 2024-03-23T11:20:24.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.479+0000 7f609d800640 1 -- start start 2024-03-23T11:20:24.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.480+0000 7f609d800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60980d06d0 0x7f60981f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.480+0000 7f609d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f609815fcf0 0x7f60980c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.480+0000 7f609d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6098160420 0x7f609806c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.480+0000 7f609d800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60981f6640 con 0x7f6098160420 2024-03-23T11:20:24.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.480+0000 7f609d800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60981566d0 con 0x7f60980d06d0 2024-03-23T11:20:24.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.480+0000 7f609d800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60981f5da0 con 0x7f609815fcf0 2024-03-23T11:20:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.480+0000 7f609ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6098160420 0x7f609806c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.480+0000 7f609ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6098160420 0x7f609806c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59584/0 (socket says 172.21.15.62:59584) 2024-03-23T11:20:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.481+0000 7f609ce00640 1 -- 172.21.15.62:0/4103280289 learned_addr learned my addr 172.21.15.62:0/4103280289 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.481+0000 7f6097e00640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60980d06d0 0x7f60981f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.481+0000 7f608f400640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f609815fcf0 0x7f60980c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.481+0000 7f609ce00640 1 -- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f609815fcf0 msgr2=0x7f60980c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.481+0000 7f609ce00640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f609815fcf0 0x7f60980c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.481+0000 7f609ce00640 1 -- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60980d06d0 msgr2=0x7f60981f5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.481+0000 7f609ce00640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60980d06d0 0x7f60981f5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.481+0000 7f609ce00640 1 -- 172.21.15.62:0/4103280289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6090009b30 con 0x7f6098160420 2024-03-23T11:20:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.481+0000 7f609ce00640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6098160420 0x7f609806c500 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6090036040 tx=0x7f6090004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.482+0000 7f6094c00640 1 -- 172.21.15.62:0/4103280289 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6090009510 con 0x7f6098160420 2024-03-23T11:20:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.482+0000 7f6094c00640 1 -- 172.21.15.62:0/4103280289 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609003e450 con 0x7f6098160420 2024-03-23T11:20:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.482+0000 7f6094c00640 1 -- 172.21.15.62:0/4103280289 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609003e800 con 0x7f6098160420 2024-03-23T11:20:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.482+0000 7f609d800640 1 -- 172.21.15.62:0/4103280289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60981f6080 con 0x7f6098160420 2024-03-23T11:20:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.482+0000 7f609d800640 1 -- 172.21.15.62:0/4103280289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609806ca40 con 0x7f6098160420 2024-03-23T11:20:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.483+0000 7f609d800640 1 -- 172.21.15.62:0/4103280289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6098156530 con 0x7f6098160420 2024-03-23T11:20:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.485+0000 7f6094c00640 1 -- 172.21.15.62:0/4103280289 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f609003e960 con 0x7f6098160420 2024-03-23T11:20:24.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.486+0000 7f6094c00640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f607406a7a0 0x7f607406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.486+0000 7f6094c00640 1 -- 172.21.15.62:0/4103280289 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f60900ac220 con 0x7f6098160420 2024-03-23T11:20:24.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.486+0000 7f6097e00640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f607406a7a0 0x7f607406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.487+0000 7f6097e00640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f607406a7a0 0x7f607406cc60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6088006fd0 tx=0x7f6088008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:24.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.490+0000 7f6094c00640 1 -- 172.21.15.62:0/4103280289 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6090077710 con 0x7f6098160420 2024-03-23T11:20:24.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.728+0000 7f609d800640 1 -- 172.21.15.62:0/4103280289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f60980ce210 con 0x7f6098160420 2024-03-23T11:20:24.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.744+0000 7f6094c00640 1 -- 172.21.15.62:0/4103280289 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f60980ce210 con 0x7f6098160420 2024-03-23T11:20:24.745 INFO:teuthology.orchestra.run.smithi062.stderr:new fs with metadata pool 4 and data pool 5 2024-03-23T11:20:24.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.749+0000 7f609d800640 1 -- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f607406a7a0 msgr2=0x7f607406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:24.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.749+0000 7f609d800640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f607406a7a0 0x7f607406cc60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6088006fd0 tx=0x7f6088008040 comp rx=0 tx=0).stop 2024-03-23T11:20:24.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.749+0000 7f609d800640 1 -- 172.21.15.62:0/4103280289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6098160420 msgr2=0x7f609806c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:24.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.749+0000 7f609d800640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6098160420 0x7f609806c500 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6090036040 tx=0x7f6090004a40 comp rx=0 tx=0).stop 2024-03-23T11:20:24.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.750+0000 7f609d800640 1 -- 172.21.15.62:0/4103280289 shutdown_connections 2024-03-23T11:20:24.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.750+0000 7f609d800640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f607406a7a0 0x7f607406cc60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.750+0000 7f609d800640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6098160420 0x7f609806c500 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.750+0000 7f609d800640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f609815fcf0 0x7f60980c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.750+0000 7f609d800640 1 --2- 172.21.15.62:0/4103280289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60980d06d0 0x7f60981f5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.750+0000 7f609d800640 1 -- 172.21.15.62:0/4103280289 >> 172.21.15.62:0/4103280289 conn(0x7f6098089f60 msgr2=0x7f609815f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:24.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.750+0000 7f609d800640 1 -- 172.21.15.62:0/4103280289 shutdown_connections 2024-03-23T11:20:24.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.751+0000 7f609d800640 1 -- 172.21.15.62:0/4103280289 wait complete. 2024-03-23T11:20:24.783 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:20:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.953+0000 7f076e200640 1 Processor -- start 2024-03-23T11:20:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.954+0000 7f076e200640 1 -- start start 2024-03-23T11:20:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.955+0000 7f076e200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f076806c300 0x7f07680ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.955+0000 7f076e200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f07680cafa0 0x7f07680cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.955+0000 7f076e200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07680cbc60 con 0x7f076806c6e0 2024-03-23T11:20:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.955+0000 7f076e200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07680cbf40 con 0x7f07680cafa0 2024-03-23T11:20:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.955+0000 7f076e200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07680cc220 con 0x7f076806c300 2024-03-23T11:20:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.956+0000 7f0766a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f07680cafa0 0x7f07680cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.956+0000 7f0767e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f076806c6e0 0x7f076806cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43074/0 (socket says 172.21.15.62:43074) 2024-03-23T11:20:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.956+0000 7f0767e00640 1 -- 172.21.15.62:0/185387523 learned_addr learned my addr 172.21.15.62:0/185387523 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.956+0000 7f0766a00640 1 --2- 172.21.15.62:0/185387523 >> v2:172.21.15.62:3300/0 conn(0x7f07680cafa0 0x7f07680cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59588/0 (socket says 172.21.15.62:59588) 2024-03-23T11:20:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.956+0000 7f0767400640 1 --2- 172.21.15.62:0/185387523 >> v2:172.21.15.120:3300/0 conn(0x7f076806c300 0x7f07680ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.956+0000 7f0766a00640 1 -- 172.21.15.62:0/185387523 >> v2:172.21.15.120:3300/0 conn(0x7f076806c300 msgr2=0x7f07680ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.956+0000 7f0766a00640 1 --2- 172.21.15.62:0/185387523 >> v2:172.21.15.120:3300/0 conn(0x7f076806c300 0x7f07680ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.956+0000 7f0766a00640 1 -- 172.21.15.62:0/185387523 >> v1:172.21.15.62:6789/0 conn(0x7f076806c6e0 legacy=0x7f076806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.956+0000 7f0766a00640 1 -- 172.21.15.62:0/185387523 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07680cc500 con 0x7f07680cafa0 2024-03-23T11:20:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.956+0000 7f0767400640 1 --2- 172.21.15.62:0/185387523 >> v2:172.21.15.120:3300/0 conn(0x7f076806c300 0x7f07680ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:24.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.957+0000 7f0766a00640 1 --2- 172.21.15.62:0/185387523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07680cafa0 0x7f07680cb430 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f076806c1b0 tx=0x7f075402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb38b48042c50df1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.958+0000 7f0766000640 1 -- 172.21.15.62:0/185387523 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754031830 con 0x7f07680cafa0 2024-03-23T11:20:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.958+0000 7f0766000640 1 -- 172.21.15.62:0/185387523 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0754031990 con 0x7f07680cafa0 2024-03-23T11:20:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.958+0000 7f0766000640 1 -- 172.21.15.62:0/185387523 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754031d20 con 0x7f07680cafa0 2024-03-23T11:20:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.958+0000 7f076e200640 1 -- 172.21.15.62:0/185387523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07680cafa0 msgr2=0x7f07680cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.958+0000 7f076e200640 1 --2- 172.21.15.62:0/185387523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07680cafa0 0x7f07680cb430 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f076806c1b0 tx=0x7f075402b8d0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.958+0000 7f076e200640 1 -- 172.21.15.62:0/185387523 shutdown_connections 2024-03-23T11:20:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.958+0000 7f076e200640 1 --2- 172.21.15.62:0/185387523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07680cafa0 0x7f07680cb430 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.958+0000 7f076e200640 1 --2- 172.21.15.62:0/185387523 >> v2:172.21.15.120:3300/0 conn(0x7f076806c300 0x7f07680ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.958+0000 7f076e200640 1 -- 172.21.15.62:0/185387523 >> 172.21.15.62:0/185387523 conn(0x7f07680b02e0 msgr2=0x7f07680b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.959+0000 7f076e200640 1 -- 172.21.15.62:0/185387523 shutdown_connections 2024-03-23T11:20:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.959+0000 7f076e200640 1 -- 172.21.15.62:0/185387523 wait complete. 2024-03-23T11:20:24.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.960+0000 7f076e200640 1 Processor -- start 2024-03-23T11:20:24.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.960+0000 7f076e200640 1 -- start start 2024-03-23T11:20:24.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.961+0000 7f076e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076806c300 0x7f076814b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.961+0000 7f076e200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f076806c6e0 0x7f07680ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.961+0000 7f076e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07680cafa0 0x7f07680abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.961+0000 7f076e200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07680ad700 con 0x7f076806c300 2024-03-23T11:20:24.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.961+0000 7f076e200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076814bc80 con 0x7f076806c6e0 2024-03-23T11:20:24.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.961+0000 7f076e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076814bf60 con 0x7f07680cafa0 2024-03-23T11:20:24.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.961+0000 7f0767400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076806c300 0x7f076814b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.961+0000 7f0767400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076806c300 0x7f076814b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59596/0 (socket says 172.21.15.62:59596) 2024-03-23T11:20:24.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.962+0000 7f0767400640 1 -- 172.21.15.62:0/3617414408 learned_addr learned my addr 172.21.15.62:0/3617414408 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:24.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.962+0000 7f0766a00640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f076806c6e0 0x7f07680ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.962+0000 7f0767400640 1 -- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07680cafa0 msgr2=0x7f07680abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:24.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.962+0000 7f0767400640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07680cafa0 0x7f07680abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.962+0000 7f0767400640 1 -- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f076806c6e0 msgr2=0x7f07680ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:24.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.962+0000 7f0767400640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f076806c6e0 0x7f07680ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:24.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.962+0000 7f0767400640 1 -- 172.21.15.62:0/3617414408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0754009400 con 0x7f076806c300 2024-03-23T11:20:24.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.962+0000 7f0767400640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076806c300 0x7f076814b630 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f075800c7f0 tx=0x7f075800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:24.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.962+0000 7f075f400640 1 -- 172.21.15.62:0/3617414408 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0758007a30 con 0x7f076806c300 2024-03-23T11:20:24.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.962+0000 7f075f400640 1 -- 172.21.15.62:0/3617414408 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0758007b90 con 0x7f076806c300 2024-03-23T11:20:24.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.963+0000 7f075f400640 1 -- 172.21.15.62:0/3617414408 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075800ce40 con 0x7f076806c300 2024-03-23T11:20:24.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.963+0000 7f076e200640 1 -- 172.21.15.62:0/3617414408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076814c240 con 0x7f076806c300 2024-03-23T11:20:24.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.963+0000 7f076e200640 1 -- 172.21.15.62:0/3617414408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076813d800 con 0x7f076806c300 2024-03-23T11:20:24.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.964+0000 7f076e200640 1 -- 172.21.15.62:0/3617414408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076813d660 con 0x7f076806c300 2024-03-23T11:20:24.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.965+0000 7f075f400640 1 -- 172.21.15.62:0/3617414408 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f07580113f0 con 0x7f076806c300 2024-03-23T11:20:24.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.966+0000 7f075f400640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f074806a7a0 0x7f074806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:24.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.966+0000 7f075f400640 1 -- 172.21.15.62:0/3617414408 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0758088f90 con 0x7f076806c300 2024-03-23T11:20:24.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.967+0000 7f0766a00640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f074806a7a0 0x7f074806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:24.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.968+0000 7f0766a00640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f074806a7a0 0x7f074806cc60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0754036000 tx=0x7f075403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:24.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:24.971+0000 7f075f400640 1 -- 172.21.15.62:0/3617414408 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f076813d660 con 0x7f076806c300 2024-03-23T11:20:25.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.164+0000 7f076e200640 1 -- 172.21.15.62:0/3617414408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f076813d660 con 0x7f076806c300 2024-03-23T11:20:25.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.167+0000 7f075f400640 1 -- 172.21.15.62:0/3617414408 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16114 (secure 0 0 0) 0x7f076813d660 con 0x7f076806c300 2024-03-23T11:20:25.168 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:25.168 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:20:24.731272+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:20:22.395497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T11:20:23.725162+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:20:25.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.169+0000 7f076e200640 1 -- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f074806a7a0 msgr2=0x7f074806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.169+0000 7f076e200640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f074806a7a0 0x7f074806cc60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0754036000 tx=0x7f075403c040 comp rx=0 tx=0).stop 2024-03-23T11:20:25.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.169+0000 7f076e200640 1 -- 172.21.15.62:0/3617414408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076806c300 msgr2=0x7f076814b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.169+0000 7f076e200640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076806c300 0x7f076814b630 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f075800c7f0 tx=0x7f075800cbf0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.170+0000 7f076e200640 1 -- 172.21.15.62:0/3617414408 shutdown_connections 2024-03-23T11:20:25.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.170+0000 7f076e200640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f074806a7a0 0x7f074806cc60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.170+0000 7f076e200640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07680cafa0 0x7f07680abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.170+0000 7f076e200640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f076806c6e0 0x7f07680ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.170+0000 7f076e200640 1 --2- 172.21.15.62:0/3617414408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076806c300 0x7f076814b630 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.170+0000 7f076e200640 1 -- 172.21.15.62:0/3617414408 >> 172.21.15.62:0/3617414408 conn(0x7f07680b02e0 msgr2=0x7f076813ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:25.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.170+0000 7f076e200640 1 -- 172.21.15.62:0/3617414408 shutdown_connections 2024-03-23T11:20:25.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.170+0000 7f076e200640 1 -- 172.21.15.62:0/3617414408 wait complete. 2024-03-23T11:20:25.189 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:20:25.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.387+0000 7fe310400640 1 Processor -- start 2024-03-23T11:20:25.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.388+0000 7fe310400640 1 -- start start 2024-03-23T11:20:25.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.389+0000 7fe310400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080cf650 0x7fe3080cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.389+0000 7fe310400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30806bfc0 con 0x7fe30815cdc0 2024-03-23T11:20:25.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.389+0000 7fe310400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3080b5260 con 0x7fe30815c9e0 2024-03-23T11:20:25.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.389+0000 7fe310400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080d0020 con 0x7fe3080cf650 2024-03-23T11:20:25.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.390+0000 7fe30d000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe30815cdc0 0x7fe3080d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43080/0 (socket says 172.21.15.62:43080) 2024-03-23T11:20:25.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.390+0000 7fe30d000640 1 -- 172.21.15.62:0/268926709 learned_addr learned my addr 172.21.15.62:0/268926709 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.390+0000 7fe30e400640 1 --2- 172.21.15.62:0/268926709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080cf650 0x7fe3080cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.390+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863057791 0 0) 0x7fe30806bfc0 con 0x7fe30815cdc0 2024-03-23T11:20:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.390+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f0003520 con 0x7fe30815cdc0 2024-03-23T11:20:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2291878397 0 0) 0x7fe2f0003520 con 0x7fe30815cdc0 2024-03-23T11:20:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080cf650 msgr2=0x7fe3080cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 --2- 172.21.15.62:0/268926709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080cf650 0x7fe3080cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 >> v1:172.21.15.120:6789/0 conn(0x7fe30815c9e0 legacy=0x7fe3081607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3080d2630 con 0x7fe30815cdc0 2024-03-23T11:20:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fe2f8001670 con 0x7fe30815cdc0 2024-03-23T11:20:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 >> v1:172.21.15.62:6789/0 conn(0x7fe30815cdc0 legacy=0x7fe3080d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 --2- 172.21.15.62:0/268926709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f00039a0 0x7fe2f0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 --2- 172.21.15.62:0/268926709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080cf650 0x7fe2f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 --2- 172.21.15.62:0/268926709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2f0004d00 0x7fe2f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080d0020 con 0x7fe2f0004d00 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f000ba30 con 0x7fe2f00039a0 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f000bba0 con 0x7fe3080cf650 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.391+0000 7fe30e400640 1 --2- 172.21.15.62:0/268926709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2f0004d00 0x7fe2f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.392+0000 7fe30e400640 1 -- 172.21.15.62:0/268926709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080cf650 msgr2=0x7fe2f00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.392+0000 7fe30e400640 1 --2- 172.21.15.62:0/268926709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080cf650 0x7fe2f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.392+0000 7fe30e400640 1 -- 172.21.15.62:0/268926709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f00039a0 msgr2=0x7fe2f0003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.392+0000 7fe30e400640 1 --2- 172.21.15.62:0/268926709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f00039a0 0x7fe2f0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.392+0000 7fe30e400640 1 -- 172.21.15.62:0/268926709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f0003520 con 0x7fe2f0004d00 2024-03-23T11:20:25.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.393+0000 7fe30e400640 1 --2- 172.21.15.62:0/268926709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2f0004d00 0x7fe2f0005180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe300009fc0 tx=0x7fe2f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f0d69dcd73f973 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.393+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30002cc50 con 0x7fe2f0004d00 2024-03-23T11:20:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.393+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30000f040 con 0x7fe2f0004d00 2024-03-23T11:20:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.393+0000 7fe307e00640 1 -- 172.21.15.62:0/268926709 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300034440 con 0x7fe2f0004d00 2024-03-23T11:20:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.393+0000 7fe310400640 1 -- 172.21.15.62:0/268926709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2f0004d00 msgr2=0x7fe2f0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.393+0000 7fe310400640 1 --2- 172.21.15.62:0/268926709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2f0004d00 0x7fe2f0005180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe300009fc0 tx=0x7fe2f00057f0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.393+0000 7fe30e400640 1 -- 172.21.15.62:0/268926709 reap_dead start 2024-03-23T11:20:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.394+0000 7fe310400640 1 -- 172.21.15.62:0/268926709 shutdown_connections 2024-03-23T11:20:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.394+0000 7fe310400640 1 -- 172.21.15.62:0/268926709 >> 172.21.15.62:0/268926709 conn(0x7fe308071570 msgr2=0x7fe308071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.394+0000 7fe310400640 1 -- 172.21.15.62:0/268926709 shutdown_connections 2024-03-23T11:20:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.394+0000 7fe310400640 1 -- 172.21.15.62:0/268926709 wait complete. 2024-03-23T11:20:25.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.395+0000 7fe310400640 1 Processor -- start 2024-03-23T11:20:25.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.396+0000 7fe310400640 1 -- start start 2024-03-23T11:20:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe310400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f0004d00 0x7fe30815ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe310400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe308160f50 0x7fe3080cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe310400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080c8fb0 0x7fe3080cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe310400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080c6420 con 0x7fe308160f50 2024-03-23T11:20:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe310400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30815b6f0 con 0x7fe2f0004d00 2024-03-23T11:20:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe310400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080c6670 con 0x7fe3080c8fb0 2024-03-23T11:20:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe30d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe308160f50 0x7fe3080cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe30d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe308160f50 0x7fe3080cbb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59604/0 (socket says 172.21.15.62:59604) 2024-03-23T11:20:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe30d000640 1 -- 172.21.15.62:0/1766881539 learned_addr learned my addr 172.21.15.62:0/1766881539 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe30e400640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080c8fb0 0x7fe3080cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe30da00640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f0004d00 0x7fe30815ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe30d000640 1 -- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080c8fb0 msgr2=0x7fe3080cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe30d000640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080c8fb0 0x7fe3080cc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe30d000640 1 -- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f0004d00 msgr2=0x7fe30815ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe30d000640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f0004d00 0x7fe30815ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe30d000640 1 -- 172.21.15.62:0/1766881539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3080c6950 con 0x7fe308160f50 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.397+0000 7fe30e400640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080c8fb0 0x7fe3080cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.398+0000 7fe30d000640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe308160f50 0x7fe3080cbb90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800c730 tx=0x7fe2f800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.398+0000 7fe305600640 1 -- 172.21.15.62:0/1766881539 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80074e0 con 0x7fe308160f50 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.398+0000 7fe305600640 1 -- 172.21.15.62:0/1766881539 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f8007640 con 0x7fe308160f50 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.398+0000 7fe305600640 1 -- 172.21.15.62:0/1766881539 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8004a20 con 0x7fe308160f50 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.398+0000 7fe310400640 1 -- 172.21.15.62:0/1766881539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3080c6c30 con 0x7fe308160f50 2024-03-23T11:20:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.398+0000 7fe310400640 1 -- 172.21.15.62:0/1766881539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3080c71f0 con 0x7fe308160f50 2024-03-23T11:20:25.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.399+0000 7fe310400640 1 -- 172.21.15.62:0/1766881539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3080c7410 con 0x7fe308160f50 2024-03-23T11:20:25.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.400+0000 7fe305600640 1 -- 172.21.15.62:0/1766881539 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe2f8004b80 con 0x7fe308160f50 2024-03-23T11:20:25.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.401+0000 7fe305600640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2ec06a7d0 0x7fe2ec06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.401+0000 7fe305600640 1 -- 172.21.15.62:0/1766881539 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2f80881a0 con 0x7fe308160f50 2024-03-23T11:20:25.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.402+0000 7fe30da00640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2ec06a7d0 0x7fe2ec06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.403+0000 7fe30da00640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2ec06a7d0 0x7fe2ec06cc90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc001770 tx=0x7fe2fc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:25.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.406+0000 7fe305600640 1 -- 172.21.15.62:0/1766881539 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2f8053520 con 0x7fe308160f50 2024-03-23T11:20:25.560 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:20:25.558+0000 7f2e77e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:20:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.608+0000 7fe310400640 1 -- 172.21.15.62:0/1766881539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3080b7880 con 0x7fe308160f50 2024-03-23T11:20:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.611+0000 7fe305600640 1 -- 172.21.15.62:0/1766881539 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16114 (secure 0 0 0) 0x7fe2f8001bf0 con 0x7fe308160f50 2024-03-23T11:20:25.613 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:25.613 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:20:24.731272+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:20:22.395497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T11:20:23.725162+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:20:25.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 -- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2ec06a7d0 msgr2=0x7fe2ec06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2ec06a7d0 0x7fe2ec06cc90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc001770 tx=0x7fe2fc009040 comp rx=0 tx=0).stop 2024-03-23T11:20:25.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 -- 172.21.15.62:0/1766881539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe308160f50 msgr2=0x7fe3080cbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe308160f50 0x7fe3080cbb90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800c730 tx=0x7fe2f800cbf0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 -- 172.21.15.62:0/1766881539 shutdown_connections 2024-03-23T11:20:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2ec06a7d0 0x7fe2ec06cc90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe308160f50 0x7fe3080cbb90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3080c8fb0 0x7fe3080cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 --2- 172.21.15.62:0/1766881539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2f0004d00 0x7fe30815ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 -- 172.21.15.62:0/1766881539 >> 172.21.15.62:0/1766881539 conn(0x7fe308071570 msgr2=0x7fe30815bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 -- 172.21.15.62:0/1766881539 shutdown_connections 2024-03-23T11:20:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.614+0000 7fe310400640 1 -- 172.21.15.62:0/1766881539 wait complete. 2024-03-23T11:20:25.634 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T11:20:25.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.861+0000 7f9f01800640 1 Processor -- start 2024-03-23T11:20:25.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.861+0000 7f9f01800640 1 -- start start 2024-03-23T11:20:25.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.863+0000 7f9f01800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc0b2f30 0x7f9efc070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.863+0000 7f9f01800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc0cd140 con 0x7f9efc143dd0 2024-03-23T11:20:25.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.863+0000 7f9f01800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc0cd3e0 con 0x7f9efc070090 2024-03-23T11:20:25.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.863+0000 7f9f01800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc070a60 con 0x7f9efc0b2f30 2024-03-23T11:20:25.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.863+0000 7f9efbe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9efc143dd0 0x7f9efc15abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43090/0 (socket says 172.21.15.62:43090) 2024-03-23T11:20:25.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.863+0000 7f9efbe00640 1 -- 172.21.15.62:0/91278702 learned_addr learned my addr 172.21.15.62:0/91278702 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:25.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.864+0000 7f9f00e00640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc0b2f30 0x7f9efc070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.864+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233261475 0 0) 0x7f9efc0cd140 con 0x7f9efc143dd0 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.864+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee0003520 con 0x7f9efc143dd0 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.864+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 929624547 0 0) 0x7f9ee0003520 con 0x7f9efc143dd0 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.864+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc0b2f30 msgr2=0x7f9efc070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.864+0000 7f9efaa00640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc0b2f30 0x7f9efc070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.864+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 >> v1:172.21.15.120:6789/0 conn(0x7f9efc070090 legacy=0x7f9efc0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.864+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc0cd6c0 con 0x7f9efc143dd0 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.864+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9eec001670 con 0x7f9efc143dd0 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.864+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 >> v1:172.21.15.62:6789/0 conn(0x7f9efc143dd0 legacy=0x7f9efc15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.864+0000 7f9efaa00640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc0b2f30 0x7f9ee00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9efaa00640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ee0003f70 0x7f9ee00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9efaa00640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 0x7f9ee0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc070a60 con 0x7f9ee0004d00 2024-03-23T11:20:25.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee000bac0 con 0x7f9ee0003f70 2024-03-23T11:20:25.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee000bc80 con 0x7f9efc0b2f30 2024-03-23T11:20:25.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9f00e00640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 0x7f9ee0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9efb400640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ee0003f70 0x7f9ee00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9efbe00640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc0b2f30 0x7f9ee00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9f00e00640 1 -- 172.21.15.62:0/91278702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc0b2f30 msgr2=0x7f9ee00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:25.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9f00e00640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc0b2f30 0x7f9ee00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9f00e00640 1 -- 172.21.15.62:0/91278702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ee0003f70 msgr2=0x7f9ee00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:25.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9f00e00640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ee0003f70 0x7f9ee00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.865+0000 7f9f00e00640 1 -- 172.21.15.62:0/91278702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee0003520 con 0x7f9ee0004d00 2024-03-23T11:20:25.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.867+0000 7f9f00e00640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 0x7f9ee0005180 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009fc0 tx=0x7f9ee00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adcc895de76d8ea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:25.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.867+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef402cc20 con 0x7f9ee0004d00 2024-03-23T11:20:25.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.867+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef400f040 con 0x7f9ee0004d00 2024-03-23T11:20:25.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.867+0000 7f9efaa00640 1 -- 172.21.15.62:0/91278702 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4034440 con 0x7f9ee0004d00 2024-03-23T11:20:25.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.867+0000 7f9f01800640 1 -- 172.21.15.62:0/91278702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 msgr2=0x7f9ee0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.867+0000 7f9f01800640 1 --2- 172.21.15.62:0/91278702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 0x7f9ee0005180 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009fc0 tx=0x7f9ee00057f0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.867+0000 7f9f00e00640 1 -- 172.21.15.62:0/91278702 reap_dead start 2024-03-23T11:20:25.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.867+0000 7f9f01800640 1 -- 172.21.15.62:0/91278702 shutdown_connections 2024-03-23T11:20:25.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.867+0000 7f9f01800640 1 -- 172.21.15.62:0/91278702 >> 172.21.15.62:0/91278702 conn(0x7f9efc066050 msgr2=0x7f9efc069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:25.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.867+0000 7f9f01800640 1 -- 172.21.15.62:0/91278702 shutdown_connections 2024-03-23T11:20:25.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.868+0000 7f9f01800640 1 -- 172.21.15.62:0/91278702 wait complete. 2024-03-23T11:20:25.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.869+0000 7f9f01800640 1 Processor -- start 2024-03-23T11:20:25.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.869+0000 7f9f01800640 1 -- start start 2024-03-23T11:20:25.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.870+0000 7f9f01800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 0x7f9efc15c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.870+0000 7f9f01800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9efc13fcb0 0x7f9efc140130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.870+0000 7f9f01800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc1f2f40 0x7f9efc1f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.870+0000 7f9f01800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc13d480 con 0x7f9ee0004d00 2024-03-23T11:20:25.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.870+0000 7f9f01800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc1f3900 con 0x7f9efc13fcb0 2024-03-23T11:20:25.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.870+0000 7f9f01800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc13d130 con 0x7f9efc1f2f40 2024-03-23T11:20:25.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9efbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 0x7f9efc15c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9efbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 0x7f9efc15c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59618/0 (socket says 172.21.15.62:59618) 2024-03-23T11:20:25.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9efbe00640 1 -- 172.21.15.62:0/3795545194 learned_addr learned my addr 172.21.15.62:0/3795545194 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:25.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9efb400640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9efc13fcb0 0x7f9efc140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9efbe00640 1 -- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc1f2f40 msgr2=0x7f9efc1f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9f00e00640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc1f2f40 0x7f9efc1f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9efbe00640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc1f2f40 0x7f9efc1f33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9efbe00640 1 -- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9efc13fcb0 msgr2=0x7f9efc140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9efbe00640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9efc13fcb0 0x7f9efc140130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9efbe00640 1 -- 172.21.15.62:0/3795545194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc0ac0d0 con 0x7f9ee0004d00 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9efb400640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9efc13fcb0 0x7f9efc140130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.871+0000 7f9efbe00640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 0x7f9efc15c630 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9eec00e7e0 tx=0x7f9eec00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.872+0000 7f9ef3e00640 1 -- 172.21.15.62:0/3795545194 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec00c870 con 0x7f9ee0004d00 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.872+0000 7f9ef3e00640 1 -- 172.21.15.62:0/3795545194 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eec00c9d0 con 0x7f9ee0004d00 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.872+0000 7f9ef3e00640 1 -- 172.21.15.62:0/3795545194 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec00cd80 con 0x7f9ee0004d00 2024-03-23T11:20:25.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.872+0000 7f9f01800640 1 -- 172.21.15.62:0/3795545194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc0ac3b0 con 0x7f9ee0004d00 2024-03-23T11:20:25.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.872+0000 7f9f01800640 1 -- 172.21.15.62:0/3795545194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc0abd60 con 0x7f9ee0004d00 2024-03-23T11:20:25.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.873+0000 7f9f01800640 1 -- 172.21.15.62:0/3795545194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc13d5c0 con 0x7f9ee0004d00 2024-03-23T11:20:25.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.874+0000 7f9ef3e00640 1 -- 172.21.15.62:0/3795545194 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9eec01d460 con 0x7f9ee0004d00 2024-03-23T11:20:25.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.875+0000 7f9ef3e00640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ed806a7a0 0x7f9ed806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:25.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.875+0000 7f9ef3e00640 1 -- 172.21.15.62:0/3795545194 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9eec088d20 con 0x7f9ee0004d00 2024-03-23T11:20:25.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.876+0000 7f9efb400640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ed806a7a0 0x7f9ed806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:25.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.877+0000 7f9efb400640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ed806a7a0 0x7f9ed806cc60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40016b0 tx=0x7f9ee4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:25.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:25.880+0000 7f9ef3e00640 1 -- 172.21.15.62:0/3795545194 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9efc13d5c0 con 0x7f9ee0004d00 2024-03-23T11:20:26.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.123+0000 7f9f01800640 1 -- 172.21.15.62:0/3795545194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9efc13d5c0 con 0x7f9ee0004d00 2024-03-23T11:20:26.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.744+0000 7f9ef3e00640 1 -- 172.21.15.62:0/3795545194 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9efc13d5c0 con 0x7f9ee0004d00 2024-03-23T11:20:26.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.747+0000 7f9f01800640 1 -- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ed806a7a0 msgr2=0x7f9ed806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:26.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.747+0000 7f9f01800640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ed806a7a0 0x7f9ed806cc60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40016b0 tx=0x7f9ee4009040 comp rx=0 tx=0).stop 2024-03-23T11:20:26.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.747+0000 7f9f01800640 1 -- 172.21.15.62:0/3795545194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 msgr2=0x7f9efc15c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:26.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.747+0000 7f9f01800640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 0x7f9efc15c630 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9eec00e7e0 tx=0x7f9eec00eca0 comp rx=0 tx=0).stop 2024-03-23T11:20:26.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.748+0000 7f9f01800640 1 -- 172.21.15.62:0/3795545194 shutdown_connections 2024-03-23T11:20:26.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.748+0000 7f9f01800640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ed806a7a0 0x7f9ed806cc60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:26.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.748+0000 7f9f01800640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9efc1f2f40 0x7f9efc1f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:26.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.748+0000 7f9f01800640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9efc13fcb0 0x7f9efc140130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:26.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.748+0000 7f9f01800640 1 --2- 172.21.15.62:0/3795545194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ee0004d00 0x7f9efc15c630 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:26.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.748+0000 7f9f01800640 1 -- 172.21.15.62:0/3795545194 >> 172.21.15.62:0/3795545194 conn(0x7f9efc066050 msgr2=0x7f9efc0ae4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:26.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.748+0000 7f9f01800640 1 -- 172.21.15.62:0/3795545194 shutdown_connections 2024-03-23T11:20:26.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.748+0000 7f9f01800640 1 -- 172.21.15.62:0/3795545194 wait complete. 2024-03-23T11:20:26.768 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T11:20:26.769 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T11:20:26.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.984+0000 7f3ece000640 1 Processor -- start 2024-03-23T11:20:26.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.985+0000 7f3ece000640 1 -- start start 2024-03-23T11:20:26.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.986+0000 7f3ece000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3ec815c830 0x7f3ec80ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:26.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.986+0000 7f3ece000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec80ce250 0x7f3ec80ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:26.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3ece000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec81608d0 con 0x7f3ec8160310 2024-03-23T11:20:26.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3ece000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec80addb0 con 0x7f3ec815c830 2024-03-23T11:20:26.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3ece000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec815cc10 con 0x7f3ec80ce250 2024-03-23T11:20:26.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3eccc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3ec815c830 0x7f3ec80ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:26.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3eccc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3ec815c830 0x7f3ec80ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59624/0 (socket says 172.21.15.62:59624) 2024-03-23T11:20:26.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3eccc00640 1 -- 172.21.15.62:0/195884992 learned_addr learned my addr 172.21.15.62:0/195884992 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:26.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3ecd600640 1 --2- 172.21.15.62:0/195884992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec80ce250 0x7f3ec80ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:26.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3eccc00640 1 -- 172.21.15.62:0/195884992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec80ce250 msgr2=0x7f3ec80ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:26.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3eccc00640 1 --2- 172.21.15.62:0/195884992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec80ce250 0x7f3ec80ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:26.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3eccc00640 1 -- 172.21.15.62:0/195884992 >> v1:172.21.15.120:6789/0 conn(0x7f3ec8160310 legacy=0x7f3ec80ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:26.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3eccc00640 1 -- 172.21.15.62:0/195884992 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec8160bb0 con 0x7f3ec815c830 2024-03-23T11:20:26.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.987+0000 7f3ecd600640 1 --2- 172.21.15.62:0/195884992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec80ce250 0x7f3ec80ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:26.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.989+0000 7f3eccc00640 1 --2- 172.21.15.62:0/195884992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ec815c830 0x7f3ec80ad310 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8009e80 tx=0x7f3eb802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98476e0c11f85b36 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:26.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.989+0000 7f3ec7400640 1 -- 172.21.15.62:0/195884992 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb802ca50 con 0x7f3ec815c830 2024-03-23T11:20:26.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.989+0000 7f3ec7400640 1 -- 172.21.15.62:0/195884992 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb802cbb0 con 0x7f3ec815c830 2024-03-23T11:20:26.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.989+0000 7f3ec7400640 1 -- 172.21.15.62:0/195884992 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb80343f0 con 0x7f3ec815c830 2024-03-23T11:20:26.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.990+0000 7f3ece000640 1 -- 172.21.15.62:0/195884992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ec815c830 msgr2=0x7f3ec80ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:26.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.990+0000 7f3ece000640 1 --2- 172.21.15.62:0/195884992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ec815c830 0x7f3ec80ad310 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8009e80 tx=0x7f3eb802beb0 comp rx=0 tx=0).stop 2024-03-23T11:20:26.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.990+0000 7f3ece000640 1 -- 172.21.15.62:0/195884992 shutdown_connections 2024-03-23T11:20:26.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.990+0000 7f3ece000640 1 --2- 172.21.15.62:0/195884992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec80ce250 0x7f3ec80ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:26.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.990+0000 7f3ece000640 1 --2- 172.21.15.62:0/195884992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ec815c830 0x7f3ec80ad310 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:26.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.990+0000 7f3ece000640 1 -- 172.21.15.62:0/195884992 >> 172.21.15.62:0/195884992 conn(0x7f3ec8066050 msgr2=0x7f3ec8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:26.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.990+0000 7f3ece000640 1 -- 172.21.15.62:0/195884992 shutdown_connections 2024-03-23T11:20:26.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.990+0000 7f3ece000640 1 -- 172.21.15.62:0/195884992 wait complete. 2024-03-23T11:20:26.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.992+0000 7f3ece000640 1 Processor -- start 2024-03-23T11:20:26.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.992+0000 7f3ece000640 1 -- start start 2024-03-23T11:20:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.993+0000 7f3ece000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec80ce250 0x7f3ec8157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.993+0000 7f3ece000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ec815c830 0x7f3ec80adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.993+0000 7f3ece000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ec8160310 0x7f3ec80ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.993+0000 7f3ece000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec81f33b0 con 0x7f3ec8160310 2024-03-23T11:20:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.993+0000 7f3ece000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec81f30d0 con 0x7f3ec815c830 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.993+0000 7f3ece000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec81f5b10 con 0x7f3ec80ce250 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.994+0000 7f3ecd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ec8160310 0x7f3ec80ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.994+0000 7f3ecd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ec8160310 0x7f3ec80ae4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59642/0 (socket says 172.21.15.62:59642) 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.994+0000 7f3ecd600640 1 -- 172.21.15.62:0/3587290525 learned_addr learned my addr 172.21.15.62:0/3587290525 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.994+0000 7f3eccc00640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec80ce250 0x7f3ec8157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.994+0000 7f3ec7e00640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ec815c830 0x7f3ec80adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.994+0000 7f3ecd600640 1 -- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec80ce250 msgr2=0x7f3ec8157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.994+0000 7f3ecd600640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec80ce250 0x7f3ec8157e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.994+0000 7f3ecd600640 1 -- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ec815c830 msgr2=0x7f3ec80adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.994+0000 7f3ecd600640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ec815c830 0x7f3ec80adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.994+0000 7f3ecd600640 1 -- 172.21.15.62:0/3587290525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb8009b30 con 0x7f3ec8160310 2024-03-23T11:20:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.995+0000 7f3ecd600640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ec8160310 0x7f3ec80ae4d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000c7c0 tx=0x7f3ec000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:26.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.995+0000 7f3ec4c00640 1 -- 172.21.15.62:0/3587290525 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0004230 con 0x7f3ec8160310 2024-03-23T11:20:26.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.995+0000 7f3ec4c00640 1 -- 172.21.15.62:0/3587290525 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec0004390 con 0x7f3ec8160310 2024-03-23T11:20:26.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.995+0000 7f3ec4c00640 1 -- 172.21.15.62:0/3587290525 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0007860 con 0x7f3ec8160310 2024-03-23T11:20:26.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.995+0000 7f3ece000640 1 -- 172.21.15.62:0/3587290525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec81f5df0 con 0x7f3ec8160310 2024-03-23T11:20:26.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.995+0000 7f3ece000640 1 -- 172.21.15.62:0/3587290525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec8158350 con 0x7f3ec8160310 2024-03-23T11:20:26.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.996+0000 7f3ece000640 1 -- 172.21.15.62:0/3587290525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec81f5970 con 0x7f3ec8160310 2024-03-23T11:20:26.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.997+0000 7f3ec4c00640 1 -- 172.21.15.62:0/3587290525 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3ec00079c0 con 0x7f3ec8160310 2024-03-23T11:20:26.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.998+0000 7f3ec4c00640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3ea406a7a0 0x7f3ea406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:26.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.999+0000 7f3ec4c00640 1 -- 172.21.15.62:0/3587290525 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ec00890f0 con 0x7f3ec8160310 2024-03-23T11:20:27.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:26.999+0000 7f3eccc00640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3ea406a7a0 0x7f3ea406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:27.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:27.000+0000 7f3eccc00640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3ea406a7a0 0x7f3ea406cc60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8035000 tx=0x7f3eb8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:27.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:27.004+0000 7f3ec4c00640 1 -- 172.21.15.62:0/3587290525 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ec81f5970 con 0x7f3ec8160310 2024-03-23T11:20:27.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:27.250+0000 7f3ece000640 1 -- 172.21.15.62:0/3587290525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3ec8158570 con 0x7f3ec8160310 2024-03-23T11:20:28.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.262+0000 7f3ec4c00640 1 -- 172.21.15.62:0/3587290525 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3ec8158570 con 0x7f3ec8160310 2024-03-23T11:20:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 -- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3ea406a7a0 msgr2=0x7f3ea406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3ea406a7a0 0x7f3ea406cc60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8035000 tx=0x7f3eb8030000 comp rx=0 tx=0).stop 2024-03-23T11:20:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 -- 172.21.15.62:0/3587290525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ec8160310 msgr2=0x7f3ec80ae4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ec8160310 0x7f3ec80ae4d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000c7c0 tx=0x7f3ec000cc80 comp rx=0 tx=0).stop 2024-03-23T11:20:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 -- 172.21.15.62:0/3587290525 shutdown_connections 2024-03-23T11:20:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3ea406a7a0 0x7f3ea406cc60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ec8160310 0x7f3ec80ae4d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ec815c830 0x7f3ec80adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 --2- 172.21.15.62:0/3587290525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3ec80ce250 0x7f3ec8157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 -- 172.21.15.62:0/3587290525 >> 172.21.15.62:0/3587290525 conn(0x7f3ec8066050 msgr2=0x7f3ec815b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:28.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 -- 172.21.15.62:0/3587290525 shutdown_connections 2024-03-23T11:20:28.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.266+0000 7f3ece000640 1 -- 172.21.15.62:0/3587290525 wait complete. 2024-03-23T11:20:28.290 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T11:20:28.290 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:20:28.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.508+0000 7f3167200640 1 Processor -- start 2024-03-23T11:20:28.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.508+0000 7f3167200640 1 -- start start 2024-03-23T11:20:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.509+0000 7f3167200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3160143dd0 0x7f316015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.510+0000 7f3167200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3160070090 0x7f3160070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:28.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.510+0000 7f3167200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31600cd140 con 0x7f31600b2820 2024-03-23T11:20:28.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.510+0000 7f3167200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3160070a60 con 0x7f3160070090 2024-03-23T11:20:28.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.510+0000 7f3167200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31600cd3e0 con 0x7f3160143dd0 2024-03-23T11:20:28.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.510+0000 7f3165400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3160070090 0x7f3160070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:28.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.510+0000 7f3165400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3160070090 0x7f3160070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55758/0 (socket says 172.21.15.62:55758) 2024-03-23T11:20:28.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.510+0000 7f3165400640 1 -- 172.21.15.62:0/2478902568 learned_addr learned my addr 172.21.15.62:0/2478902568 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:28.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.510+0000 7f3165e00640 1 --2- 172.21.15.62:0/2478902568 >> v2:172.21.15.120:3300/0 conn(0x7f3160143dd0 0x7f316015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:28.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.511+0000 7f3165400640 1 -- 172.21.15.62:0/2478902568 >> v2:172.21.15.120:3300/0 conn(0x7f3160143dd0 msgr2=0x7f316015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:28.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.511+0000 7f3165400640 1 --2- 172.21.15.62:0/2478902568 >> v2:172.21.15.120:3300/0 conn(0x7f3160143dd0 0x7f316015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.511+0000 7f3165400640 1 -- 172.21.15.62:0/2478902568 >> v1:172.21.15.120:6789/0 conn(0x7f31600b2820 legacy=0x7f31600b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:28.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.511+0000 7f3165400640 1 -- 172.21.15.62:0/2478902568 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31600cd6c0 con 0x7f3160070090 2024-03-23T11:20:28.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.512+0000 7f3165400640 1 --2- 172.21.15.62:0/2478902568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3160070090 0x7f3160070520 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f316006e380 tx=0x7f315002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b1550c2dc90894 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:28.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.512+0000 7f3164a00640 1 -- 172.21.15.62:0/2478902568 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150031870 con 0x7f3160070090 2024-03-23T11:20:28.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.512+0000 7f3164a00640 1 -- 172.21.15.62:0/2478902568 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31500319d0 con 0x7f3160070090 2024-03-23T11:20:28.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.512+0000 7f3164a00640 1 -- 172.21.15.62:0/2478902568 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150031d60 con 0x7f3160070090 2024-03-23T11:20:28.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.513+0000 7f3167200640 1 -- 172.21.15.62:0/2478902568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3160070090 msgr2=0x7f3160070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:28.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.513+0000 7f3167200640 1 --2- 172.21.15.62:0/2478902568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3160070090 0x7f3160070520 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f316006e380 tx=0x7f315002b910 comp rx=0 tx=0).stop 2024-03-23T11:20:28.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.513+0000 7f3167200640 1 -- 172.21.15.62:0/2478902568 shutdown_connections 2024-03-23T11:20:28.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.513+0000 7f3167200640 1 --2- 172.21.15.62:0/2478902568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3160070090 0x7f3160070520 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.513+0000 7f3167200640 1 --2- 172.21.15.62:0/2478902568 >> v2:172.21.15.120:3300/0 conn(0x7f3160143dd0 0x7f316015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.513+0000 7f3167200640 1 -- 172.21.15.62:0/2478902568 >> 172.21.15.62:0/2478902568 conn(0x7f3160066050 msgr2=0x7f3160069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:28.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.513+0000 7f3167200640 1 -- 172.21.15.62:0/2478902568 shutdown_connections 2024-03-23T11:20:28.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.513+0000 7f3167200640 1 -- 172.21.15.62:0/2478902568 wait complete. 2024-03-23T11:20:28.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.515+0000 7f3167200640 1 Processor -- start 2024-03-23T11:20:28.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.515+0000 7f3167200640 1 -- start start 2024-03-23T11:20:28.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3167200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3160070090 0x7f3160159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:28.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3167200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31600b2820 0x7f316013bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:28.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3167200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3160143dd0 0x7f316013c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:28.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3167200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316015a2c0 con 0x7f3160070090 2024-03-23T11:20:28.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3167200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316013b4d0 con 0x7f3160143dd0 2024-03-23T11:20:28.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3167200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316013a8c0 con 0x7f31600b2820 2024-03-23T11:20:28.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3165e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3160070090 0x7f3160159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:28.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3165e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3160070090 0x7f3160159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55772/0 (socket says 172.21.15.62:55772) 2024-03-23T11:20:28.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3165e00640 1 -- 172.21.15.62:0/310016092 learned_addr learned my addr 172.21.15.62:0/310016092 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:28.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3165e00640 1 -- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31600b2820 msgr2=0x7f316013bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:28.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3165e00640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31600b2820 0x7f316013bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3165e00640 1 -- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3160143dd0 msgr2=0x7f316013c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:28.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.516+0000 7f3166800640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3160143dd0 0x7f316013c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:28.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.517+0000 7f3165e00640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3160143dd0 0x7f316013c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.517+0000 7f3165e00640 1 -- 172.21.15.62:0/310016092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31500094d0 con 0x7f3160070090 2024-03-23T11:20:28.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.517+0000 7f3166800640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3160143dd0 0x7f316013c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:28.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.517+0000 7f3165e00640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3160070090 0x7f3160159d80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f315400d620 tx=0x7f315400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:28.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.517+0000 7f315a000640 1 -- 172.21.15.62:0/310016092 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315400b620 con 0x7f3160070090 2024-03-23T11:20:28.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.517+0000 7f315a000640 1 -- 172.21.15.62:0/310016092 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315400b780 con 0x7f3160070090 2024-03-23T11:20:28.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.517+0000 7f3167200640 1 -- 172.21.15.62:0/310016092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316013aba0 con 0x7f3160070090 2024-03-23T11:20:28.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.517+0000 7f315a000640 1 -- 172.21.15.62:0/310016092 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315400bb30 con 0x7f3160070090 2024-03-23T11:20:28.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.517+0000 7f3167200640 1 -- 172.21.15.62:0/310016092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316013b160 con 0x7f3160070090 2024-03-23T11:20:28.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.519+0000 7f3167200640 1 -- 172.21.15.62:0/310016092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3160066bb0 con 0x7f3160070090 2024-03-23T11:20:28.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.519+0000 7f315a000640 1 -- 172.21.15.62:0/310016092 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f315400bc90 con 0x7f3160070090 2024-03-23T11:20:28.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.520+0000 7f315a000640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f314406a7d0 0x7f314406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:28.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.520+0000 7f315a000640 1 -- 172.21.15.62:0/310016092 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3154089300 con 0x7f3160070090 2024-03-23T11:20:28.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.521+0000 7f3165400640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f314406a7d0 0x7f314406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:28.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.522+0000 7f3165400640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f314406a7d0 0x7f314406cc90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3150036000 tx=0x7f3150030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:28.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.525+0000 7f315a000640 1 -- 172.21.15.62:0/310016092 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3154054680 con 0x7f3160070090 2024-03-23T11:20:28.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.766+0000 7f3167200640 1 -- 172.21.15.62:0/310016092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31600ceeb0 con 0x7f3160070090 2024-03-23T11:20:28.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.767+0000 7f315a000640 1 -- 172.21.15.62:0/310016092 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f3154057ca0 con 0x7f3160070090 2024-03-23T11:20:28.769 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:28.769 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3421279564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3421279564},{"type":"v1","addr":"172.21.15.120:6835","nonce":3421279564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2052651236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2052651236},{"type":"v1","addr":"172.21.15.120:6837","nonce":2052651236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5193,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/1599029025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1599029025},{"type":"v1","addr":"172.21.15.62:6834","nonce":1599029025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:20:24.731603+0000","modified":"2024-03-23T11:20:28.249457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":5,"addr":"172.21.15.62:6835/3537214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":3537214366},{"type":"v1","addr":"172.21.15.62:6835","nonce":3537214366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T11:20:28.769 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.770+0000 7f3167200640 1 -- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f314406a7d0 msgr2=0x7f314406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.770+0000 7f3167200640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f314406a7d0 0x7f314406cc90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3150036000 tx=0x7f3150030000 comp rx=0 tx=0).stop 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.770+0000 7f3167200640 1 -- 172.21.15.62:0/310016092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3160070090 msgr2=0x7f3160159d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.770+0000 7f3167200640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3160070090 0x7f3160159d80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f315400d620 tx=0x7f315400dae0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.770+0000 7f3167200640 1 -- 172.21.15.62:0/310016092 shutdown_connections 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.770+0000 7f3167200640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f314406a7d0 0x7f314406cc90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.770+0000 7f3167200640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3160143dd0 0x7f316013c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.770+0000 7f3167200640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f31600b2820 0x7f316013bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.770+0000 7f3167200640 1 --2- 172.21.15.62:0/310016092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3160070090 0x7f3160159d80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.770+0000 7f3167200640 1 -- 172.21.15.62:0/310016092 >> 172.21.15.62:0/310016092 conn(0x7f3160066050 msgr2=0x7f316015a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.771+0000 7f3167200640 1 -- 172.21.15.62:0/310016092 shutdown_connections 2024-03-23T11:20:28.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.771+0000 7f3167200640 1 -- 172.21.15.62:0/310016092 wait complete. 2024-03-23T11:20:28.792 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:20:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.998+0000 7f819cc00640 1 Processor -- start 2024-03-23T11:20:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.998+0000 7f819cc00640 1 -- start start 2024-03-23T11:20:29.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:28.999+0000 7f819cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8198159490 0x7f819806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.000+0000 7f819cc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f819815da90 0x7f81980cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.000+0000 7f819cc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81980ce8b0 con 0x7f81980ce420 2024-03-23T11:20:29.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.000+0000 7f819cc00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81980ced90 con 0x7f819815da90 2024-03-23T11:20:29.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.000+0000 7f819cc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819806e3e0 con 0x7f8198159490 2024-03-23T11:20:29.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.000+0000 7f8196a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f819815da90 0x7f81980cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.000+0000 7f8196a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f819815da90 0x7f81980cdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55788/0 (socket says 172.21.15.62:55788) 2024-03-23T11:20:29.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.000+0000 7f8196a00640 1 -- 172.21.15.62:0/641027129 learned_addr learned my addr 172.21.15.62:0/641027129 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:29.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.000+0000 7f8197400640 1 --2- 172.21.15.62:0/641027129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8198159490 0x7f819806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.001+0000 7f8196a00640 1 -- 172.21.15.62:0/641027129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8198159490 msgr2=0x7f819806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:29.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.001+0000 7f8196a00640 1 --2- 172.21.15.62:0/641027129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8198159490 0x7f819806cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.001+0000 7f8196a00640 1 -- 172.21.15.62:0/641027129 >> v1:172.21.15.120:6789/0 conn(0x7f81980ce420 legacy=0x7f81980b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:29.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.001+0000 7f8196a00640 1 -- 172.21.15.62:0/641027129 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819806c1c0 con 0x7f819815da90 2024-03-23T11:20:29.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.002+0000 7f8196a00640 1 --2- 172.21.15.62:0/641027129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f819815da90 0x7f81980cdee0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8184009870 tx=0x7f818402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30f8a2d56365094 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:29.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.003+0000 7f8196000640 1 -- 172.21.15.62:0/641027129 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81840317c0 con 0x7f819815da90 2024-03-23T11:20:29.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.003+0000 7f8196000640 1 -- 172.21.15.62:0/641027129 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8184031920 con 0x7f819815da90 2024-03-23T11:20:29.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.003+0000 7f8196000640 1 -- 172.21.15.62:0/641027129 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184031cb0 con 0x7f819815da90 2024-03-23T11:20:29.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.003+0000 7f819cc00640 1 -- 172.21.15.62:0/641027129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f819815da90 msgr2=0x7f81980cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.003+0000 7f819cc00640 1 --2- 172.21.15.62:0/641027129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f819815da90 0x7f81980cdee0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8184009870 tx=0x7f818402b860 comp rx=0 tx=0).stop 2024-03-23T11:20:29.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.003+0000 7f819cc00640 1 -- 172.21.15.62:0/641027129 shutdown_connections 2024-03-23T11:20:29.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.003+0000 7f819cc00640 1 --2- 172.21.15.62:0/641027129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f819815da90 0x7f81980cdee0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.003+0000 7f819cc00640 1 --2- 172.21.15.62:0/641027129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8198159490 0x7f819806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.003+0000 7f819cc00640 1 -- 172.21.15.62:0/641027129 >> 172.21.15.62:0/641027129 conn(0x7f8198066050 msgr2=0x7f8198069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:29.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.003+0000 7f819cc00640 1 -- 172.21.15.62:0/641027129 shutdown_connections 2024-03-23T11:20:29.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.003+0000 7f819cc00640 1 -- 172.21.15.62:0/641027129 wait complete. 2024-03-23T11:20:29.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.005+0000 7f819cc00640 1 Processor -- start 2024-03-23T11:20:29.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.005+0000 7f819cc00640 1 -- start start 2024-03-23T11:20:29.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.006+0000 7f819cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81980ce420 0x7f8198159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.006+0000 7f819cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8198159490 0x7f81980ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.006+0000 7f819cc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f819815da90 0x7f81980aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.006+0000 7f819cc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819815a020 con 0x7f8198159490 2024-03-23T11:20:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.006+0000 7f819cc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81981575c0 con 0x7f819815da90 2024-03-23T11:20:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.006+0000 7f819cc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81980ad280 con 0x7f81980ce420 2024-03-23T11:20:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.007+0000 7f8196a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8198159490 0x7f81980ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.007+0000 7f8196a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8198159490 0x7f81980ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55800/0 (socket says 172.21.15.62:55800) 2024-03-23T11:20:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.007+0000 7f8196a00640 1 -- 172.21.15.62:0/3410626682 learned_addr learned my addr 172.21.15.62:0/3410626682 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.007+0000 7f8197e00640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f819815da90 0x7f81980aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.007+0000 7f8197400640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81980ce420 0x7f8198159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.007+0000 7f8196a00640 1 -- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81980ce420 msgr2=0x7f8198159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.007+0000 7f8196a00640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81980ce420 0x7f8198159ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.007+0000 7f8196a00640 1 -- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f819815da90 msgr2=0x7f81980aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.007+0000 7f8196a00640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f819815da90 0x7f81980aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.007+0000 7f8196a00640 1 -- 172.21.15.62:0/3410626682 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81840094d0 con 0x7f8198159490 2024-03-23T11:20:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.008+0000 7f8196a00640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8198159490 0x7f81980ae890 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8184009870 tx=0x7f81840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.008+0000 7f818f400640 1 -- 172.21.15.62:0/3410626682 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184032860 con 0x7f8198159490 2024-03-23T11:20:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.008+0000 7f818f400640 1 -- 172.21.15.62:0/3410626682 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81840329c0 con 0x7f8198159490 2024-03-23T11:20:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.008+0000 7f818f400640 1 -- 172.21.15.62:0/3410626682 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184032d70 con 0x7f8198159490 2024-03-23T11:20:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.008+0000 7f819cc00640 1 -- 172.21.15.62:0/3410626682 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8198158c60 con 0x7f8198159490 2024-03-23T11:20:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.008+0000 7f819cc00640 1 -- 172.21.15.62:0/3410626682 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81980acfa0 con 0x7f8198159490 2024-03-23T11:20:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.009+0000 7f819cc00640 1 -- 172.21.15.62:0/3410626682 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81980ad3c0 con 0x7f8198159490 2024-03-23T11:20:29.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.010+0000 7f818f400640 1 -- 172.21.15.62:0/3410626682 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f818404e450 con 0x7f8198159490 2024-03-23T11:20:29.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.011+0000 7f818f400640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817806a7a0 0x7f817806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.011+0000 7f818f400640 1 -- 172.21.15.62:0/3410626682 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81840ab0c0 con 0x7f8198159490 2024-03-23T11:20:29.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.012+0000 7f8197400640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817806a7a0 0x7f817806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.013+0000 7f8197400640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817806a7a0 0x7f817806cc60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8188006fd0 tx=0x7f8188008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:29.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.015+0000 7f818f400640 1 -- 172.21.15.62:0/3410626682 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8184076440 con 0x7f8198159490 2024-03-23T11:20:29.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.214+0000 7f819cc00640 1 -- 172.21.15.62:0/3410626682 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8198066bb0 con 0x7f8198159490 2024-03-23T11:20:29.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.217+0000 7f818f400640 1 -- 172.21.15.62:0/3410626682 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16101 (secure 0 0 0) 0x7f8184079a60 con 0x7f8198159490 2024-03-23T11:20:29.219 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:29.219 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:20:25.735105+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:20:22.395497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T11:20:23.725162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:20:29.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.220+0000 7f819cc00640 1 -- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817806a7a0 msgr2=0x7f817806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.220+0000 7f819cc00640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817806a7a0 0x7f817806cc60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8188006fd0 tx=0x7f8188008040 comp rx=0 tx=0).stop 2024-03-23T11:20:29.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.220+0000 7f819cc00640 1 -- 172.21.15.62:0/3410626682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8198159490 msgr2=0x7f81980ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.220+0000 7f819cc00640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8198159490 0x7f81980ae890 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8184009870 tx=0x7f81840046c0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.220+0000 7f819cc00640 1 -- 172.21.15.62:0/3410626682 shutdown_connections 2024-03-23T11:20:29.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.220+0000 7f819cc00640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817806a7a0 0x7f817806cc60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.220+0000 7f819cc00640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8198159490 0x7f81980ae890 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.220+0000 7f819cc00640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f819815da90 0x7f81980aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.220+0000 7f819cc00640 1 --2- 172.21.15.62:0/3410626682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81980ce420 0x7f8198159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.220+0000 7f819cc00640 1 -- 172.21.15.62:0/3410626682 >> 172.21.15.62:0/3410626682 conn(0x7f8198066050 msgr2=0x7f8198069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:29.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.221+0000 7f819cc00640 1 -- 172.21.15.62:0/3410626682 shutdown_connections 2024-03-23T11:20:29.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.221+0000 7f819cc00640 1 -- 172.21.15.62:0/3410626682 wait complete. 2024-03-23T11:20:29.238 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T11:20:29.238 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:20:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.454+0000 7f7759a00640 1 Processor -- start 2024-03-23T11:20:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.454+0000 7f7759a00640 1 -- start start 2024-03-23T11:20:29.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.456+0000 7f7759a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f775415fcf0 0x7f77540d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.456+0000 7f7759a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7754160420 0x7f7754160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.456+0000 7f7759a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7754160dc0 con 0x7f77540d06d0 2024-03-23T11:20:29.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.456+0000 7f7759a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77540d3540 con 0x7f775415fcf0 2024-03-23T11:20:29.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.456+0000 7f7759a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77540ce510 con 0x7f7754160420 2024-03-23T11:20:29.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.456+0000 7f7759000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f77540d06d0 0x7f77540d0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35220/0 (socket says 172.21.15.62:35220) 2024-03-23T11:20:29.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.456+0000 7f7759000640 1 -- 172.21.15.62:0/157254064 learned_addr learned my addr 172.21.15.62:0/157254064 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:29.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7753400640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7754160420 0x7f7754160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7753e00640 1 --2- 172.21.15.62:0/157254064 >> v2:172.21.15.120:3300/0 conn(0x7f775415fcf0 0x7f77540d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166473175 0 0) 0x7f7754160dc0 con 0x7f77540d06d0 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7738003520 con 0x7f77540d06d0 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4215599628 0 0) 0x7f7738003520 con 0x7f77540d06d0 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7754160420 msgr2=0x7f7754160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7754160420 0x7f7754160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 >> v2:172.21.15.120:3300/0 conn(0x7f775415fcf0 msgr2=0x7f77540d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 --2- 172.21.15.62:0/157254064 >> v2:172.21.15.120:3300/0 conn(0x7f775415fcf0 0x7f77540d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77540cdc70 con 0x7f77540d06d0 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f774c0019f0 con 0x7f77540d06d0 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 >> v1:172.21.15.62:6789/0 conn(0x7f77540d06d0 legacy=0x7f77540d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7738003970 0x7f7738003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7754160420 0x7f77380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7738004d00 0x7f7738005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77540d3540 con 0x7f7738004d00 2024-03-23T11:20:29.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77540ce510 con 0x7f7738003970 2024-03-23T11:20:29.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.457+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773800b980 con 0x7f7754160420 2024-03-23T11:20:29.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.458+0000 7f7759000640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7738004d00 0x7f7738005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.458+0000 7f7753e00640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7738003970 0x7f7738003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.458+0000 7f7753400640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7754160420 0x7f77380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.458+0000 7f7759000640 1 -- 172.21.15.62:0/157254064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7754160420 msgr2=0x7f77380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:29.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.458+0000 7f7759000640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7754160420 0x7f77380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.458+0000 7f7759000640 1 -- 172.21.15.62:0/157254064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7738003970 msgr2=0x7f7738003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:29.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.458+0000 7f7759000640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7738003970 0x7f7738003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.458+0000 7f7759000640 1 -- 172.21.15.62:0/157254064 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7738003520 con 0x7f7738004d00 2024-03-23T11:20:29.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.459+0000 7f7759000640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7738004d00 0x7f7738005180 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f774c00a860 tx=0x7f77380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632c9aec7e2d7bd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:29.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.460+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c034630 con 0x7f7738004d00 2024-03-23T11:20:29.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.460+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774c02f070 con 0x7f7738004d00 2024-03-23T11:20:29.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.460+0000 7f7752a00640 1 -- 172.21.15.62:0/157254064 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c0349f0 con 0x7f7738004d00 2024-03-23T11:20:29.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.460+0000 7f7759a00640 1 -- 172.21.15.62:0/157254064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7738004d00 msgr2=0x7f7738005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.460+0000 7f7759a00640 1 --2- 172.21.15.62:0/157254064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7738004d00 0x7f7738005180 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f774c00a860 tx=0x7f77380057f0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.460+0000 7f7759000640 1 -- 172.21.15.62:0/157254064 reap_dead start 2024-03-23T11:20:29.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.460+0000 7f7759a00640 1 -- 172.21.15.62:0/157254064 shutdown_connections 2024-03-23T11:20:29.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.460+0000 7f7759a00640 1 -- 172.21.15.62:0/157254064 >> 172.21.15.62:0/157254064 conn(0x7f7754089f60 msgr2=0x7f77540d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:29.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.460+0000 7f7759a00640 1 -- 172.21.15.62:0/157254064 shutdown_connections 2024-03-23T11:20:29.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.460+0000 7f7759a00640 1 -- 172.21.15.62:0/157254064 wait complete. 2024-03-23T11:20:29.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.462+0000 7f7759a00640 1 Processor -- start 2024-03-23T11:20:29.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.462+0000 7f7759a00640 1 -- start start 2024-03-23T11:20:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.463+0000 7f7759a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f775414d870 0x7f775414df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.463+0000 7f7759a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f775409edb0 0x7f775409f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.463+0000 7f7759a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77541f5930 0x7f77541f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.463+0000 7f7759a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775414dc50 con 0x7f77541f5930 2024-03-23T11:20:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.463+0000 7f7759a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77540c6f20 con 0x7f775409edb0 2024-03-23T11:20:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.463+0000 7f7759a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775409f770 con 0x7f775414d870 2024-03-23T11:20:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.463+0000 7f7759000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77541f5930 0x7f77541f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.463+0000 7f7759000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77541f5930 0x7f77541f5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55820/0 (socket says 172.21.15.62:55820) 2024-03-23T11:20:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.463+0000 7f7759000640 1 -- 172.21.15.62:0/1025355223 learned_addr learned my addr 172.21.15.62:0/1025355223 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.464+0000 7f7753400640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f775409edb0 0x7f775409f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.464+0000 7f7753e00640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f775414d870 0x7f775414df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.464+0000 7f7759000640 1 -- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f775414d870 msgr2=0x7f775414df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.464+0000 7f7759000640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f775414d870 0x7f775414df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.464+0000 7f7759000640 1 -- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f775409edb0 msgr2=0x7f775409f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.464+0000 7f7759000640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f775409edb0 0x7f775409f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.464+0000 7f7759000640 1 -- 172.21.15.62:0/1025355223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77541f66a0 con 0x7f77541f5930 2024-03-23T11:20:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.464+0000 7f7759000640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77541f5930 0x7f77541f5de0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f774c02cba0 tx=0x7f774c001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.465+0000 7f774be00640 1 -- 172.21.15.62:0/1025355223 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c034840 con 0x7f77541f5930 2024-03-23T11:20:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.465+0000 7f774be00640 1 -- 172.21.15.62:0/1025355223 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774c0349a0 con 0x7f77541f5930 2024-03-23T11:20:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.465+0000 7f774be00640 1 -- 172.21.15.62:0/1025355223 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c034d10 con 0x7f77541f5930 2024-03-23T11:20:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.465+0000 7f7759a00640 1 -- 172.21.15.62:0/1025355223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77540c7170 con 0x7f77541f5930 2024-03-23T11:20:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.465+0000 7f7759a00640 1 -- 172.21.15.62:0/1025355223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77540c7730 con 0x7f77541f5930 2024-03-23T11:20:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.466+0000 7f7759a00640 1 -- 172.21.15.62:0/1025355223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7754066bb0 con 0x7f77541f5930 2024-03-23T11:20:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.467+0000 7f774be00640 1 -- 172.21.15.62:0/1025355223 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f774c031030 con 0x7f77541f5930 2024-03-23T11:20:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.468+0000 7f774be00640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f773406a7d0 0x7f773406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.468+0000 7f774be00640 1 -- 172.21.15.62:0/1025355223 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f774c0aafa0 con 0x7f77541f5930 2024-03-23T11:20:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.468+0000 7f7753e00640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f773406a7d0 0x7f773406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.469+0000 7f7753e00640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f773406a7d0 0x7f773406cc90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7754151090 tx=0x7f7744008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:29.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.472+0000 7f774be00640 1 -- 172.21.15.62:0/1025355223 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f774c076320 con 0x7f77541f5930 2024-03-23T11:20:29.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.670+0000 7f7759a00640 1 -- 172.21.15.62:0/1025355223 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77540b55d0 con 0x7f773406a7d0 2024-03-23T11:20:29.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.697+0000 7f774be00640 1 -- 172.21.15.62:0/1025355223 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286552 (secure 0 0 0) 0x7f7744004030 con 0x7f773406a7d0 2024-03-23T11:20:29.703 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:29.704 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:20:29.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 -- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f773406a7d0 msgr2=0x7f773406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f773406a7d0 0x7f773406cc90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7754151090 tx=0x7f7744008040 comp rx=0 tx=0).stop 2024-03-23T11:20:29.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 -- 172.21.15.62:0/1025355223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77541f5930 msgr2=0x7f77541f5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77541f5930 0x7f77541f5de0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f774c02cba0 tx=0x7f774c001d80 comp rx=0 tx=0).stop 2024-03-23T11:20:29.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 -- 172.21.15.62:0/1025355223 shutdown_connections 2024-03-23T11:20:29.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f773406a7d0 0x7f773406cc90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77541f5930 0x7f77541f5de0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f775409edb0 0x7f775409f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 --2- 172.21.15.62:0/1025355223 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f775414d870 0x7f775414df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 -- 172.21.15.62:0/1025355223 >> 172.21.15.62:0/1025355223 conn(0x7f7754089f60 msgr2=0x7f77540d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:29.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 -- 172.21.15.62:0/1025355223 shutdown_connections 2024-03-23T11:20:29.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.704+0000 7f7759a00640 1 -- 172.21.15.62:0/1025355223 wait complete. 2024-03-23T11:20:29.726 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-03-23T11:20:29.264090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224644,"kb_used_data":4380,"kb_used_omap":13,"kb_used_meta":220146,"kb_avail":754750076,"statfs":{"total":773094113280,"available":772864077824,"internally_reserved":0,"allocated":4485120,"data_stored":2461685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":225429592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001789"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.885685+0000","last_change":"2024-03-23T11:20:24.780788+0000","last_active":"2024-03-23T11:20:25.885685+0000","last_peered":"2024-03-23T11:20:25.885685+0000","last_clean":"2024-03-23T11:20:25.885685+0000","last_became_active":"2024-03-23T11:20:24.780256+0000","last_became_peered":"2024-03-23T11:20:24.780256+0000","last_unstale":"2024-03-23T11:20:25.885685+0000","last_undegraded":"2024-03-23T11:20:25.885685+0000","last_fullsized":"2024-03-23T11:20:25.885685+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:41:17.793193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.811027+0000","last_change":"2024-03-23T11:20:24.773252+0000","last_active":"2024-03-23T11:20:25.811027+0000","last_peered":"2024-03-23T11:20:25.811027+0000","last_clean":"2024-03-23T11:20:25.811027+0000","last_became_active":"2024-03-23T11:20:24.772717+0000","last_became_peered":"2024-03-23T11:20:24.772717+0000","last_unstale":"2024-03-23T11:20:25.811027+0000","last_undegraded":"2024-03-23T11:20:25.811027+0000","last_fullsized":"2024-03-23T11:20:25.811027+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:58:08.558907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295986+0000","last_change":"2024-03-23T11:20:24.764765+0000","last_active":"2024-03-23T11:20:26.295986+0000","last_peered":"2024-03-23T11:20:26.295986+0000","last_clean":"2024-03-23T11:20:26.295986+0000","last_became_active":"2024-03-23T11:20:24.758563+0000","last_became_peered":"2024-03-23T11:20:24.758563+0000","last_unstale":"2024-03-23T11:20:26.295986+0000","last_undegraded":"2024-03-23T11:20:26.295986+0000","last_fullsized":"2024-03-23T11:20:26.295986+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:15.019627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.743025+0000","last_change":"2024-03-23T11:20:24.777436+0000","last_active":"2024-03-23T11:20:25.743025+0000","last_peered":"2024-03-23T11:20:25.743025+0000","last_clean":"2024-03-23T11:20:25.743025+0000","last_became_active":"2024-03-23T11:20:24.777221+0000","last_became_peered":"2024-03-23T11:20:24.777221+0000","last_unstale":"2024-03-23T11:20:25.743025+0000","last_undegraded":"2024-03-23T11:20:25.743025+0000","last_fullsized":"2024-03-23T11:20:25.743025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.152151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.209710+0000","last_change":"2024-03-23T11:20:24.780185+0000","last_active":"2024-03-23T11:20:26.209710+0000","last_peered":"2024-03-23T11:20:26.209710+0000","last_clean":"2024-03-23T11:20:26.209710+0000","last_became_active":"2024-03-23T11:20:24.779904+0000","last_became_peered":"2024-03-23T11:20:24.779904+0000","last_unstale":"2024-03-23T11:20:26.209710+0000","last_undegraded":"2024-03-23T11:20:26.209710+0000","last_fullsized":"2024-03-23T11:20:26.209710+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:42:55.664530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.810924+0000","last_change":"2024-03-23T11:20:24.774166+0000","last_active":"2024-03-23T11:20:25.810924+0000","last_peered":"2024-03-23T11:20:25.810924+0000","last_clean":"2024-03-23T11:20:25.810924+0000","last_became_active":"2024-03-23T11:20:24.773926+0000","last_became_peered":"2024-03-23T11:20:24.773926+0000","last_unstale":"2024-03-23T11:20:25.810924+0000","last_undegraded":"2024-03-23T11:20:25.810924+0000","last_fullsized":"2024-03-23T11:20:25.810924+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:30.582986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747296+0000","last_change":"2024-03-23T11:20:24.782449+0000","last_active":"2024-03-23T11:20:25.747296+0000","last_peered":"2024-03-23T11:20:25.747296+0000","last_clean":"2024-03-23T11:20:25.747296+0000","last_became_active":"2024-03-23T11:20:24.782082+0000","last_became_peered":"2024-03-23T11:20:24.782082+0000","last_unstale":"2024-03-23T11:20:25.747296+0000","last_undegraded":"2024-03-23T11:20:25.747296+0000","last_fullsized":"2024-03-23T11:20:25.747296+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.878125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.745869+0000","last_change":"2024-03-23T11:20:24.761096+0000","last_active":"2024-03-23T11:20:25.745869+0000","last_peered":"2024-03-23T11:20:25.745869+0000","last_clean":"2024-03-23T11:20:25.745869+0000","last_became_active":"2024-03-23T11:20:24.760771+0000","last_became_peered":"2024-03-23T11:20:24.760771+0000","last_unstale":"2024-03-23T11:20:25.745869+0000","last_undegraded":"2024-03-23T11:20:25.745869+0000","last_fullsized":"2024-03-23T11:20:25.745869+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:14.345002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747878+0000","last_change":"2024-03-23T11:20:24.783056+0000","last_active":"2024-03-23T11:20:25.747878+0000","last_peered":"2024-03-23T11:20:25.747878+0000","last_clean":"2024-03-23T11:20:25.747878+0000","last_became_active":"2024-03-23T11:20:24.782703+0000","last_became_peered":"2024-03-23T11:20:24.782703+0000","last_unstale":"2024-03-23T11:20:25.747878+0000","last_undegraded":"2024-03-23T11:20:25.747878+0000","last_fullsized":"2024-03-23T11:20:25.747878+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:00.613118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.742921+0000","last_change":"2024-03-23T11:20:24.778368+0000","last_active":"2024-03-23T11:20:25.742921+0000","last_peered":"2024-03-23T11:20:25.742921+0000","last_clean":"2024-03-23T11:20:25.742921+0000","last_became_active":"2024-03-23T11:20:24.777711+0000","last_became_peered":"2024-03-23T11:20:24.777711+0000","last_unstale":"2024-03-23T11:20:25.742921+0000","last_undegraded":"2024-03-23T11:20:25.742921+0000","last_fullsized":"2024-03-23T11:20:25.742921+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:57.399282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746388+0000","last_change":"2024-03-23T11:20:24.768454+0000","last_active":"2024-03-23T11:20:25.746388+0000","last_peered":"2024-03-23T11:20:25.746388+0000","last_clean":"2024-03-23T11:20:25.746388+0000","last_became_active":"2024-03-23T11:20:24.768167+0000","last_became_peered":"2024-03-23T11:20:24.768167+0000","last_unstale":"2024-03-23T11:20:25.746388+0000","last_undegraded":"2024-03-23T11:20:25.746388+0000","last_fullsized":"2024-03-23T11:20:25.746388+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:57.399282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295829+0000","last_change":"2024-03-23T11:20:24.770561+0000","last_active":"2024-03-23T11:20:26.295829+0000","last_peered":"2024-03-23T11:20:26.295829+0000","last_clean":"2024-03-23T11:20:26.295829+0000","last_became_active":"2024-03-23T11:20:24.770207+0000","last_became_peered":"2024-03-23T11:20:24.770207+0000","last_unstale":"2024-03-23T11:20:26.295829+0000","last_undegraded":"2024-03-23T11:20:26.295829+0000","last_fullsized":"2024-03-23T11:20:26.295829+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:51:36.049584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.209856+0000","last_change":"2024-03-23T11:20:24.762498+0000","last_active":"2024-03-23T11:20:26.209856+0000","last_peered":"2024-03-23T11:20:26.209856+0000","last_clean":"2024-03-23T11:20:26.209856+0000","last_became_active":"2024-03-23T11:20:24.762155+0000","last_became_peered":"2024-03-23T11:20:24.762155+0000","last_unstale":"2024-03-23T11:20:26.209856+0000","last_undegraded":"2024-03-23T11:20:26.209856+0000","last_fullsized":"2024-03-23T11:20:26.209856+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:22:08.135604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.811259+0000","last_change":"2024-03-23T11:20:24.769694+0000","last_active":"2024-03-23T11:20:25.811259+0000","last_peered":"2024-03-23T11:20:25.811259+0000","last_clean":"2024-03-23T11:20:25.811259+0000","last_became_active":"2024-03-23T11:20:24.769411+0000","last_became_peered":"2024-03-23T11:20:24.769411+0000","last_unstale":"2024-03-23T11:20:25.811259+0000","last_undegraded":"2024-03-23T11:20:25.811259+0000","last_fullsized":"2024-03-23T11:20:25.811259+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:53:56.922627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295947+0000","last_change":"2024-03-23T11:20:24.766167+0000","last_active":"2024-03-23T11:20:26.295947+0000","last_peered":"2024-03-23T11:20:26.295947+0000","last_clean":"2024-03-23T11:20:26.295947+0000","last_became_active":"2024-03-23T11:20:24.765916+0000","last_became_peered":"2024-03-23T11:20:24.765916+0000","last_unstale":"2024-03-23T11:20:26.295947+0000","last_undegraded":"2024-03-23T11:20:26.295947+0000","last_fullsized":"2024-03-23T11:20:26.295947+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:08:28.525913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746590+0000","last_change":"2024-03-23T11:20:24.781114+0000","last_active":"2024-03-23T11:20:25.746590+0000","last_peered":"2024-03-23T11:20:25.746590+0000","last_clean":"2024-03-23T11:20:25.746590+0000","last_became_active":"2024-03-23T11:20:24.780855+0000","last_became_peered":"2024-03-23T11:20:24.780855+0000","last_unstale":"2024-03-23T11:20:25.746590+0000","last_undegraded":"2024-03-23T11:20:25.746590+0000","last_fullsized":"2024-03-23T11:20:25.746590+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:03:04.312998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747960+0000","last_change":"2024-03-23T11:20:24.781658+0000","last_active":"2024-03-23T11:20:25.747960+0000","last_peered":"2024-03-23T11:20:25.747960+0000","last_clean":"2024-03-23T11:20:25.747960+0000","last_became_active":"2024-03-23T11:20:24.781368+0000","last_became_peered":"2024-03-23T11:20:24.781368+0000","last_unstale":"2024-03-23T11:20:25.747960+0000","last_undegraded":"2024-03-23T11:20:25.747960+0000","last_fullsized":"2024-03-23T11:20:25.747960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:38.280685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.811120+0000","last_change":"2024-03-23T11:20:24.765059+0000","last_active":"2024-03-23T11:20:25.811120+0000","last_peered":"2024-03-23T11:20:25.811120+0000","last_clean":"2024-03-23T11:20:25.811120+0000","last_became_active":"2024-03-23T11:20:24.764735+0000","last_became_peered":"2024-03-23T11:20:24.764735+0000","last_unstale":"2024-03-23T11:20:25.811120+0000","last_undegraded":"2024-03-23T11:20:25.811120+0000","last_fullsized":"2024-03-23T11:20:25.811120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:22.166605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.810562+0000","last_change":"2024-03-23T11:20:24.768957+0000","last_active":"2024-03-23T11:20:25.810562+0000","last_peered":"2024-03-23T11:20:25.810562+0000","last_clean":"2024-03-23T11:20:25.810562+0000","last_became_active":"2024-03-23T11:20:24.768659+0000","last_became_peered":"2024-03-23T11:20:24.768659+0000","last_unstale":"2024-03-23T11:20:25.810562+0000","last_undegraded":"2024-03-23T11:20:25.810562+0000","last_fullsized":"2024-03-23T11:20:25.810562+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:51:36.049584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.209875+0000","last_change":"2024-03-23T11:20:24.774367+0000","last_active":"2024-03-23T11:20:26.209875+0000","last_peered":"2024-03-23T11:20:26.209875+0000","last_clean":"2024-03-23T11:20:26.209875+0000","last_became_active":"2024-03-23T11:20:24.774099+0000","last_became_peered":"2024-03-23T11:20:24.774099+0000","last_unstale":"2024-03-23T11:20:26.209875+0000","last_undegraded":"2024-03-23T11:20:26.209875+0000","last_fullsized":"2024-03-23T11:20:26.209875+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:14:57.844505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747244+0000","last_change":"2024-03-23T11:20:24.778194+0000","last_active":"2024-03-23T11:20:25.747244+0000","last_peered":"2024-03-23T11:20:25.747244+0000","last_clean":"2024-03-23T11:20:25.747244+0000","last_became_active":"2024-03-23T11:20:24.777870+0000","last_became_peered":"2024-03-23T11:20:24.777870+0000","last_unstale":"2024-03-23T11:20:25.747244+0000","last_undegraded":"2024-03-23T11:20:25.747244+0000","last_fullsized":"2024-03-23T11:20:25.747244+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:08:28.525913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746543+0000","last_change":"2024-03-23T11:20:24.783528+0000","last_active":"2024-03-23T11:20:25.746543+0000","last_peered":"2024-03-23T11:20:25.746543+0000","last_clean":"2024-03-23T11:20:25.746543+0000","last_became_active":"2024-03-23T11:20:24.779012+0000","last_became_peered":"2024-03-23T11:20:24.779012+0000","last_unstale":"2024-03-23T11:20:25.746543+0000","last_undegraded":"2024-03-23T11:20:25.746543+0000","last_fullsized":"2024-03-23T11:20:25.746543+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:41:17.793193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295520+0000","last_change":"2024-03-23T11:20:24.772686+0000","last_active":"2024-03-23T11:20:26.295520+0000","last_peered":"2024-03-23T11:20:26.295520+0000","last_clean":"2024-03-23T11:20:26.295520+0000","last_became_active":"2024-03-23T11:20:24.770924+0000","last_became_peered":"2024-03-23T11:20:24.770924+0000","last_unstale":"2024-03-23T11:20:26.295520+0000","last_undegraded":"2024-03-23T11:20:26.295520+0000","last_fullsized":"2024-03-23T11:20:26.295520+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:16.669316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295497+0000","last_change":"2024-03-23T11:20:24.763547+0000","last_active":"2024-03-23T11:20:26.295497+0000","last_peered":"2024-03-23T11:20:26.295497+0000","last_clean":"2024-03-23T11:20:26.295497+0000","last_became_active":"2024-03-23T11:20:24.763083+0000","last_became_peered":"2024-03-23T11:20:24.763083+0000","last_unstale":"2024-03-23T11:20:26.295497+0000","last_undegraded":"2024-03-23T11:20:26.295497+0000","last_fullsized":"2024-03-23T11:20:26.295497+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:44:59.527355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.743120+0000","last_change":"2024-03-23T11:20:24.765205+0000","last_active":"2024-03-23T11:20:25.743120+0000","last_peered":"2024-03-23T11:20:25.743120+0000","last_clean":"2024-03-23T11:20:25.743120+0000","last_became_active":"2024-03-23T11:20:24.764983+0000","last_became_peered":"2024-03-23T11:20:24.764983+0000","last_unstale":"2024-03-23T11:20:25.743120+0000","last_undegraded":"2024-03-23T11:20:25.743120+0000","last_fullsized":"2024-03-23T11:20:25.743120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:11:54.038528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.811219+0000","last_change":"2024-03-23T11:20:24.754291+0000","last_active":"2024-03-23T11:20:25.811219+0000","last_peered":"2024-03-23T11:20:25.811219+0000","last_clean":"2024-03-23T11:20:25.811219+0000","last_became_active":"2024-03-23T11:20:24.754023+0000","last_became_peered":"2024-03-23T11:20:24.754023+0000","last_unstale":"2024-03-23T11:20:25.811219+0000","last_undegraded":"2024-03-23T11:20:25.811219+0000","last_fullsized":"2024-03-23T11:20:25.811219+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:15.019627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.743233+0000","last_change":"2024-03-23T11:20:24.772136+0000","last_active":"2024-03-23T11:20:25.743233+0000","last_peered":"2024-03-23T11:20:25.743233+0000","last_clean":"2024-03-23T11:20:25.743233+0000","last_became_active":"2024-03-23T11:20:24.771864+0000","last_became_peered":"2024-03-23T11:20:24.771864+0000","last_unstale":"2024-03-23T11:20:25.743233+0000","last_undegraded":"2024-03-23T11:20:25.743233+0000","last_fullsized":"2024-03-23T11:20:25.743233+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:03:04.312998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296060+0000","last_change":"2024-03-23T11:20:24.771662+0000","last_active":"2024-03-23T11:20:26.296060+0000","last_peered":"2024-03-23T11:20:26.296060+0000","last_clean":"2024-03-23T11:20:26.296060+0000","last_became_active":"2024-03-23T11:20:24.770452+0000","last_became_peered":"2024-03-23T11:20:24.770452+0000","last_unstale":"2024-03-23T11:20:26.296060+0000","last_undegraded":"2024-03-23T11:20:26.296060+0000","last_fullsized":"2024-03-23T11:20:26.296060+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:13:42.990421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747956+0000","last_change":"2024-03-23T11:20:24.774529+0000","last_active":"2024-03-23T11:20:25.747956+0000","last_peered":"2024-03-23T11:20:25.747956+0000","last_clean":"2024-03-23T11:20:25.747956+0000","last_became_active":"2024-03-23T11:20:24.773297+0000","last_became_peered":"2024-03-23T11:20:24.773297+0000","last_unstale":"2024-03-23T11:20:25.747956+0000","last_undegraded":"2024-03-23T11:20:25.747956+0000","last_fullsized":"2024-03-23T11:20:25.747956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:00.267569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746339+0000","last_change":"2024-03-23T11:20:24.764027+0000","last_active":"2024-03-23T11:20:25.746339+0000","last_peered":"2024-03-23T11:20:25.746339+0000","last_clean":"2024-03-23T11:20:25.746339+0000","last_became_active":"2024-03-23T11:20:24.762832+0000","last_became_peered":"2024-03-23T11:20:24.762832+0000","last_unstale":"2024-03-23T11:20:25.746339+0000","last_undegraded":"2024-03-23T11:20:25.746339+0000","last_fullsized":"2024-03-23T11:20:25.746339+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:30:39.630549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.885475+0000","last_change":"2024-03-23T11:20:24.770934+0000","last_active":"2024-03-23T11:20:25.885475+0000","last_peered":"2024-03-23T11:20:25.885475+0000","last_clean":"2024-03-23T11:20:25.885475+0000","last_became_active":"2024-03-23T11:20:24.770577+0000","last_became_peered":"2024-03-23T11:20:24.770577+0000","last_unstale":"2024-03-23T11:20:25.885475+0000","last_undegraded":"2024-03-23T11:20:25.885475+0000","last_fullsized":"2024-03-23T11:20:25.885475+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:08:28.525913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746121+0000","last_change":"2024-03-23T11:20:24.767795+0000","last_active":"2024-03-23T11:20:25.746121+0000","last_peered":"2024-03-23T11:20:25.746121+0000","last_clean":"2024-03-23T11:20:25.746121+0000","last_became_active":"2024-03-23T11:20:24.767433+0000","last_became_peered":"2024-03-23T11:20:24.767433+0000","last_unstale":"2024-03-23T11:20:25.746121+0000","last_undegraded":"2024-03-23T11:20:25.746121+0000","last_fullsized":"2024-03-23T11:20:25.746121+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.152151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296163+0000","last_change":"2024-03-23T11:20:24.769847+0000","last_active":"2024-03-23T11:20:26.296163+0000","last_peered":"2024-03-23T11:20:26.296163+0000","last_clean":"2024-03-23T11:20:26.296163+0000","last_became_active":"2024-03-23T11:20:24.769576+0000","last_became_peered":"2024-03-23T11:20:24.769576+0000","last_unstale":"2024-03-23T11:20:26.296163+0000","last_undegraded":"2024-03-23T11:20:26.296163+0000","last_fullsized":"2024-03-23T11:20:26.296163+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:00:28.054426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"35'4","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.782881+0000","last_change":"2024-03-23T11:20:24.274189+0000","last_active":"2024-03-23T11:20:24.782881+0000","last_peered":"2024-03-23T11:20:24.782881+0000","last_clean":"2024-03-23T11:20:24.782881+0000","last_became_active":"2024-03-23T11:20:24.273908+0000","last_became_peered":"2024-03-23T11:20:24.273908+0000","last_unstale":"2024-03-23T11:20:24.782881+0000","last_undegraded":"2024-03-23T11:20:24.782881+0000","last_fullsized":"2024-03-23T11:20:24.782881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:11.707558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.885987+0000","last_change":"2024-03-23T11:20:24.780587+0000","last_active":"2024-03-23T11:20:25.885987+0000","last_peered":"2024-03-23T11:20:25.885987+0000","last_clean":"2024-03-23T11:20:25.885987+0000","last_became_active":"2024-03-23T11:20:24.777836+0000","last_became_peered":"2024-03-23T11:20:24.777836+0000","last_unstale":"2024-03-23T11:20:25.885987+0000","last_undegraded":"2024-03-23T11:20:25.885987+0000","last_fullsized":"2024-03-23T11:20:25.885987+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:13.726077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296257+0000","last_change":"2024-03-23T11:20:24.263698+0000","last_active":"2024-03-23T11:20:26.296257+0000","last_peered":"2024-03-23T11:20:26.296257+0000","last_clean":"2024-03-23T11:20:26.296257+0000","last_became_active":"2024-03-23T11:20:24.263412+0000","last_became_peered":"2024-03-23T11:20:24.263412+0000","last_unstale":"2024-03-23T11:20:26.296257+0000","last_undegraded":"2024-03-23T11:20:26.296257+0000","last_fullsized":"2024-03-23T11:20:26.296257+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:29:31.328499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746704+0000","last_change":"2024-03-23T11:20:24.777302+0000","last_active":"2024-03-23T11:20:25.746704+0000","last_peered":"2024-03-23T11:20:25.746704+0000","last_clean":"2024-03-23T11:20:25.746704+0000","last_became_active":"2024-03-23T11:20:24.777028+0000","last_became_peered":"2024-03-23T11:20:24.777028+0000","last_unstale":"2024-03-23T11:20:25.746704+0000","last_undegraded":"2024-03-23T11:20:25.746704+0000","last_fullsized":"2024-03-23T11:20:25.746704+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:14:57.844505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.775960+0000","last_change":"2024-03-23T11:20:24.270930+0000","last_active":"2024-03-23T11:20:24.775960+0000","last_peered":"2024-03-23T11:20:24.775960+0000","last_clean":"2024-03-23T11:20:24.775960+0000","last_became_active":"2024-03-23T11:20:24.270535+0000","last_became_peered":"2024-03-23T11:20:24.270535+0000","last_unstale":"2024-03-23T11:20:24.775960+0000","last_undegraded":"2024-03-23T11:20:24.775960+0000","last_fullsized":"2024-03-23T11:20:24.775960+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:22:07.120173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.743392+0000","last_change":"2024-03-23T11:20:24.763737+0000","last_active":"2024-03-23T11:20:25.743392+0000","last_peered":"2024-03-23T11:20:25.743392+0000","last_clean":"2024-03-23T11:20:25.743392+0000","last_became_active":"2024-03-23T11:20:24.763511+0000","last_became_peered":"2024-03-23T11:20:24.763511+0000","last_unstale":"2024-03-23T11:20:25.743392+0000","last_undegraded":"2024-03-23T11:20:25.743392+0000","last_fullsized":"2024-03-23T11:20:25.743392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:14.345002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.767793+0000","last_change":"2024-03-23T11:20:24.264541+0000","last_active":"2024-03-23T11:20:24.767793+0000","last_peered":"2024-03-23T11:20:24.767793+0000","last_clean":"2024-03-23T11:20:24.767793+0000","last_became_active":"2024-03-23T11:20:24.264244+0000","last_became_peered":"2024-03-23T11:20:24.264244+0000","last_unstale":"2024-03-23T11:20:24.767793+0000","last_undegraded":"2024-03-23T11:20:24.767793+0000","last_fullsized":"2024-03-23T11:20:24.767793+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:31.618689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746592+0000","last_change":"2024-03-23T11:20:24.776629+0000","last_active":"2024-03-23T11:20:25.746592+0000","last_peered":"2024-03-23T11:20:25.746592+0000","last_clean":"2024-03-23T11:20:25.746592+0000","last_became_active":"2024-03-23T11:20:24.776314+0000","last_became_peered":"2024-03-23T11:20:24.776314+0000","last_unstale":"2024-03-23T11:20:25.746592+0000","last_undegraded":"2024-03-23T11:20:25.746592+0000","last_fullsized":"2024-03-23T11:20:25.746592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:06:48.779863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"35'1","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.783724+0000","last_change":"2024-03-23T11:20:24.269524+0000","last_active":"2024-03-23T11:20:24.783724+0000","last_peered":"2024-03-23T11:20:24.783724+0000","last_clean":"2024-03-23T11:20:24.783724+0000","last_became_active":"2024-03-23T11:20:24.268972+0000","last_became_peered":"2024-03-23T11:20:24.268972+0000","last_unstale":"2024-03-23T11:20:24.783724+0000","last_undegraded":"2024-03-23T11:20:24.783724+0000","last_fullsized":"2024-03-23T11:20:24.783724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:16:16.741588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295558+0000","last_change":"2024-03-23T11:20:24.766147+0000","last_active":"2024-03-23T11:20:26.295558+0000","last_peered":"2024-03-23T11:20:26.295558+0000","last_clean":"2024-03-23T11:20:26.295558+0000","last_became_active":"2024-03-23T11:20:24.765773+0000","last_became_peered":"2024-03-23T11:20:24.765773+0000","last_unstale":"2024-03-23T11:20:26.295558+0000","last_undegraded":"2024-03-23T11:20:26.295558+0000","last_fullsized":"2024-03-23T11:20:26.295558+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:00.613118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"35'1","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.783821+0000","last_change":"2024-03-23T11:20:24.269384+0000","last_active":"2024-03-23T11:20:24.783821+0000","last_peered":"2024-03-23T11:20:24.783821+0000","last_clean":"2024-03-23T11:20:24.783821+0000","last_became_active":"2024-03-23T11:20:24.268752+0000","last_became_peered":"2024-03-23T11:20:24.268752+0000","last_unstale":"2024-03-23T11:20:24.783821+0000","last_undegraded":"2024-03-23T11:20:24.783821+0000","last_fullsized":"2024-03-23T11:20:24.783821+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:31.618689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747224+0000","last_change":"2024-03-23T11:20:24.784288+0000","last_active":"2024-03-23T11:20:25.747224+0000","last_peered":"2024-03-23T11:20:25.747224+0000","last_clean":"2024-03-23T11:20:25.747224+0000","last_became_active":"2024-03-23T11:20:24.784012+0000","last_became_peered":"2024-03-23T11:20:24.784012+0000","last_unstale":"2024-03-23T11:20:25.747224+0000","last_undegraded":"2024-03-23T11:20:25.747224+0000","last_fullsized":"2024-03-23T11:20:25.747224+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:04:49.659198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.739922+0000","last_change":"2024-03-23T11:20:24.262869+0000","last_active":"2024-03-23T11:20:24.739922+0000","last_peered":"2024-03-23T11:20:24.739922+0000","last_clean":"2024-03-23T11:20:24.739922+0000","last_became_active":"2024-03-23T11:20:24.262525+0000","last_became_peered":"2024-03-23T11:20:24.262525+0000","last_unstale":"2024-03-23T11:20:24.739922+0000","last_undegraded":"2024-03-23T11:20:24.739922+0000","last_fullsized":"2024-03-23T11:20:24.739922+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:29:31.328499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.742476+0000","last_change":"2024-03-23T11:20:24.776155+0000","last_active":"2024-03-23T11:20:25.742476+0000","last_peered":"2024-03-23T11:20:25.742476+0000","last_clean":"2024-03-23T11:20:25.742476+0000","last_became_active":"2024-03-23T11:20:24.773562+0000","last_became_peered":"2024-03-23T11:20:24.773562+0000","last_unstale":"2024-03-23T11:20:25.742476+0000","last_undegraded":"2024-03-23T11:20:25.742476+0000","last_fullsized":"2024-03-23T11:20:25.742476+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:00:21.168162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.763039+0000","last_change":"2024-03-23T11:20:24.266106+0000","last_active":"2024-03-23T11:20:24.763039+0000","last_peered":"2024-03-23T11:20:24.763039+0000","last_clean":"2024-03-23T11:20:24.763039+0000","last_became_active":"2024-03-23T11:20:24.265744+0000","last_became_peered":"2024-03-23T11:20:24.265744+0000","last_unstale":"2024-03-23T11:20:24.763039+0000","last_undegraded":"2024-03-23T11:20:24.763039+0000","last_fullsized":"2024-03-23T11:20:24.763039+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.609703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296212+0000","last_change":"2024-03-23T11:20:24.771313+0000","last_active":"2024-03-23T11:20:26.296212+0000","last_peered":"2024-03-23T11:20:26.296212+0000","last_clean":"2024-03-23T11:20:26.296212+0000","last_became_active":"2024-03-23T11:20:24.770842+0000","last_became_peered":"2024-03-23T11:20:24.770842+0000","last_unstale":"2024-03-23T11:20:26.296212+0000","last_undegraded":"2024-03-23T11:20:26.296212+0000","last_fullsized":"2024-03-23T11:20:26.296212+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:22.166605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.740881+0000","last_change":"2024-03-23T11:20:24.265733+0000","last_active":"2024-03-23T11:20:24.740881+0000","last_peered":"2024-03-23T11:20:24.740881+0000","last_clean":"2024-03-23T11:20:24.740881+0000","last_became_active":"2024-03-23T11:20:24.265389+0000","last_became_peered":"2024-03-23T11:20:24.265389+0000","last_unstale":"2024-03-23T11:20:24.740881+0000","last_undegraded":"2024-03-23T11:20:24.740881+0000","last_fullsized":"2024-03-23T11:20:24.740881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:11.707558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.811309+0000","last_change":"2024-03-23T11:20:24.764620+0000","last_active":"2024-03-23T11:20:25.811309+0000","last_peered":"2024-03-23T11:20:25.811309+0000","last_clean":"2024-03-23T11:20:25.811309+0000","last_became_active":"2024-03-23T11:20:24.764420+0000","last_became_peered":"2024-03-23T11:20:24.764420+0000","last_unstale":"2024-03-23T11:20:25.811309+0000","last_undegraded":"2024-03-23T11:20:25.811309+0000","last_fullsized":"2024-03-23T11:20:25.811309+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:25:01.738465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296388+0000","last_change":"2024-03-23T11:20:24.262933+0000","last_active":"2024-03-23T11:20:26.296388+0000","last_peered":"2024-03-23T11:20:26.296388+0000","last_clean":"2024-03-23T11:20:26.296388+0000","last_became_active":"2024-03-23T11:20:24.262659+0000","last_became_peered":"2024-03-23T11:20:24.262659+0000","last_unstale":"2024-03-23T11:20:26.296388+0000","last_undegraded":"2024-03-23T11:20:26.296388+0000","last_fullsized":"2024-03-23T11:20:26.296388+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:06.287580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.810951+0000","last_change":"2024-03-23T11:20:24.756469+0000","last_active":"2024-03-23T11:20:25.810951+0000","last_peered":"2024-03-23T11:20:25.810951+0000","last_clean":"2024-03-23T11:20:25.810951+0000","last_became_active":"2024-03-23T11:20:24.756225+0000","last_became_peered":"2024-03-23T11:20:24.756225+0000","last_unstale":"2024-03-23T11:20:25.810951+0000","last_undegraded":"2024-03-23T11:20:25.810951+0000","last_fullsized":"2024-03-23T11:20:25.810951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:00.267569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.775039+0000","last_change":"2024-03-23T11:20:24.271111+0000","last_active":"2024-03-23T11:20:24.775039+0000","last_peered":"2024-03-23T11:20:24.775039+0000","last_clean":"2024-03-23T11:20:24.775039+0000","last_became_active":"2024-03-23T11:20:24.270844+0000","last_became_peered":"2024-03-23T11:20:24.270844+0000","last_unstale":"2024-03-23T11:20:24.775039+0000","last_undegraded":"2024-03-23T11:20:24.775039+0000","last_fullsized":"2024-03-23T11:20:24.775039+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:08:34.466237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295814+0000","last_change":"2024-03-23T11:20:24.764978+0000","last_active":"2024-03-23T11:20:26.295814+0000","last_peered":"2024-03-23T11:20:26.295814+0000","last_clean":"2024-03-23T11:20:26.295814+0000","last_became_active":"2024-03-23T11:20:24.763142+0000","last_became_peered":"2024-03-23T11:20:24.763142+0000","last_unstale":"2024-03-23T11:20:26.295814+0000","last_undegraded":"2024-03-23T11:20:26.295814+0000","last_fullsized":"2024-03-23T11:20:26.295814+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:30.582986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":17,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.787653+0000","last_change":"2024-03-23T11:20:24.268566+0000","last_active":"2024-03-23T11:20:24.787653+0000","last_peered":"2024-03-23T11:20:24.787653+0000","last_clean":"2024-03-23T11:20:24.787653+0000","last_became_active":"2024-03-23T11:20:24.268288+0000","last_became_peered":"2024-03-23T11:20:24.268288+0000","last_unstale":"2024-03-23T11:20:24.787653+0000","last_undegraded":"2024-03-23T11:20:24.787653+0000","last_fullsized":"2024-03-23T11:20:24.787653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:15.653884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.742556+0000","last_change":"2024-03-23T11:20:24.775094+0000","last_active":"2024-03-23T11:20:25.742556+0000","last_peered":"2024-03-23T11:20:25.742556+0000","last_clean":"2024-03-23T11:20:25.742556+0000","last_became_active":"2024-03-23T11:20:24.774861+0000","last_became_peered":"2024-03-23T11:20:24.774861+0000","last_unstale":"2024-03-23T11:20:25.742556+0000","last_undegraded":"2024-03-23T11:20:25.742556+0000","last_fullsized":"2024-03-23T11:20:25.742556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:25.103486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"35'2","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.784123+0000","last_change":"2024-03-23T11:20:24.273073+0000","last_active":"2024-03-23T11:20:24.784123+0000","last_peered":"2024-03-23T11:20:24.784123+0000","last_clean":"2024-03-23T11:20:24.784123+0000","last_became_active":"2024-03-23T11:20:24.272435+0000","last_became_peered":"2024-03-23T11:20:24.272435+0000","last_unstale":"2024-03-23T11:20:24.784123+0000","last_undegraded":"2024-03-23T11:20:24.784123+0000","last_fullsized":"2024-03-23T11:20:24.784123+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:54.976104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.810727+0000","last_change":"2024-03-23T11:20:24.764404+0000","last_active":"2024-03-23T11:20:25.810727+0000","last_peered":"2024-03-23T11:20:25.810727+0000","last_clean":"2024-03-23T11:20:25.810727+0000","last_became_active":"2024-03-23T11:20:24.762524+0000","last_became_peered":"2024-03-23T11:20:24.762524+0000","last_unstale":"2024-03-23T11:20:25.810727+0000","last_undegraded":"2024-03-23T11:20:25.810727+0000","last_fullsized":"2024-03-23T11:20:25.810727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:41:17.793193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.749531+0000","last_change":"2024-03-23T11:20:24.267580+0000","last_active":"2024-03-23T11:20:24.749531+0000","last_peered":"2024-03-23T11:20:24.749531+0000","last_clean":"2024-03-23T11:20:24.749531+0000","last_became_active":"2024-03-23T11:20:24.267272+0000","last_became_peered":"2024-03-23T11:20:24.267272+0000","last_unstale":"2024-03-23T11:20:24.749531+0000","last_undegraded":"2024-03-23T11:20:24.749531+0000","last_fullsized":"2024-03-23T11:20:24.749531+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:11:10.777710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746374+0000","last_change":"2024-03-23T11:20:24.774621+0000","last_active":"2024-03-23T11:20:25.746374+0000","last_peered":"2024-03-23T11:20:25.746374+0000","last_clean":"2024-03-23T11:20:25.746374+0000","last_became_active":"2024-03-23T11:20:24.774167+0000","last_became_peered":"2024-03-23T11:20:24.774167+0000","last_unstale":"2024-03-23T11:20:25.746374+0000","last_undegraded":"2024-03-23T11:20:25.746374+0000","last_fullsized":"2024-03-23T11:20:25.746374+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:57.318031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.740097+0000","last_change":"2024-03-23T11:20:24.265822+0000","last_active":"2024-03-23T11:20:24.740097+0000","last_peered":"2024-03-23T11:20:24.740097+0000","last_clean":"2024-03-23T11:20:24.740097+0000","last_became_active":"2024-03-23T11:20:24.264936+0000","last_became_peered":"2024-03-23T11:20:24.264936+0000","last_unstale":"2024-03-23T11:20:24.740097+0000","last_undegraded":"2024-03-23T11:20:24.740097+0000","last_fullsized":"2024-03-23T11:20:24.740097+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:40:11.857041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.885637+0000","last_change":"2024-03-23T11:20:24.776569+0000","last_active":"2024-03-23T11:20:25.885637+0000","last_peered":"2024-03-23T11:20:25.885637+0000","last_clean":"2024-03-23T11:20:25.885637+0000","last_became_active":"2024-03-23T11:20:24.776257+0000","last_became_peered":"2024-03-23T11:20:24.776257+0000","last_unstale":"2024-03-23T11:20:25.885637+0000","last_undegraded":"2024-03-23T11:20:25.885637+0000","last_fullsized":"2024-03-23T11:20:25.885637+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:00.267569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":19,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.784818+0000","last_change":"2024-03-23T11:20:24.269918+0000","last_active":"2024-03-23T11:20:24.784818+0000","last_peered":"2024-03-23T11:20:24.784818+0000","last_clean":"2024-03-23T11:20:24.784818+0000","last_became_active":"2024-03-23T11:20:24.269576+0000","last_became_peered":"2024-03-23T11:20:24.269576+0000","last_unstale":"2024-03-23T11:20:24.784818+0000","last_undegraded":"2024-03-23T11:20:24.784818+0000","last_fullsized":"2024-03-23T11:20:24.784818+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:56.780601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747467+0000","last_change":"2024-03-23T11:20:24.773626+0000","last_active":"2024-03-23T11:20:25.747467+0000","last_peered":"2024-03-23T11:20:25.747467+0000","last_clean":"2024-03-23T11:20:25.747467+0000","last_became_active":"2024-03-23T11:20:24.773265+0000","last_became_peered":"2024-03-23T11:20:24.773265+0000","last_unstale":"2024-03-23T11:20:25.747467+0000","last_undegraded":"2024-03-23T11:20:25.747467+0000","last_fullsized":"2024-03-23T11:20:25.747467+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:45.264945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.778831+0000","last_change":"2024-03-23T11:20:24.269588+0000","last_active":"2024-03-23T11:20:24.778831+0000","last_peered":"2024-03-23T11:20:24.778831+0000","last_clean":"2024-03-23T11:20:24.778831+0000","last_became_active":"2024-03-23T11:20:24.268693+0000","last_became_peered":"2024-03-23T11:20:24.268693+0000","last_unstale":"2024-03-23T11:20:24.778831+0000","last_undegraded":"2024-03-23T11:20:24.778831+0000","last_fullsized":"2024-03-23T11:20:24.778831+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:12:56.580754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296240+0000","last_change":"2024-03-23T11:20:24.776102+0000","last_active":"2024-03-23T11:20:26.296240+0000","last_peered":"2024-03-23T11:20:26.296240+0000","last_clean":"2024-03-23T11:20:26.296240+0000","last_became_active":"2024-03-23T11:20:24.775711+0000","last_became_peered":"2024-03-23T11:20:24.775711+0000","last_unstale":"2024-03-23T11:20:26.296240+0000","last_undegraded":"2024-03-23T11:20:26.296240+0000","last_fullsized":"2024-03-23T11:20:26.296240+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:58:08.558907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"35'2","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.784237+0000","last_change":"2024-03-23T11:20:24.269723+0000","last_active":"2024-03-23T11:20:24.784237+0000","last_peered":"2024-03-23T11:20:24.784237+0000","last_clean":"2024-03-23T11:20:24.784237+0000","last_became_active":"2024-03-23T11:20:24.269450+0000","last_became_peered":"2024-03-23T11:20:24.269450+0000","last_unstale":"2024-03-23T11:20:24.784237+0000","last_undegraded":"2024-03-23T11:20:24.784237+0000","last_fullsized":"2024-03-23T11:20:24.784237+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:06.287580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.743163+0000","last_change":"2024-03-23T11:20:24.766463+0000","last_active":"2024-03-23T11:20:25.743163+0000","last_peered":"2024-03-23T11:20:25.743163+0000","last_clean":"2024-03-23T11:20:25.743163+0000","last_became_active":"2024-03-23T11:20:24.766173+0000","last_became_peered":"2024-03-23T11:20:24.766173+0000","last_unstale":"2024-03-23T11:20:25.743163+0000","last_undegraded":"2024-03-23T11:20:25.743163+0000","last_fullsized":"2024-03-23T11:20:25.743163+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:57.318031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'1","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.775309+0000","last_change":"2024-03-23T11:20:24.271441+0000","last_active":"2024-03-23T11:20:24.775309+0000","last_peered":"2024-03-23T11:20:24.775309+0000","last_clean":"2024-03-23T11:20:24.775309+0000","last_became_active":"2024-03-23T11:20:24.270146+0000","last_became_peered":"2024-03-23T11:20:24.270146+0000","last_unstale":"2024-03-23T11:20:24.775309+0000","last_undegraded":"2024-03-23T11:20:24.775309+0000","last_fullsized":"2024-03-23T11:20:24.775309+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:49:18.386429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.210227+0000","last_change":"2024-03-23T11:20:24.774987+0000","last_active":"2024-03-23T11:20:26.210227+0000","last_peered":"2024-03-23T11:20:26.210227+0000","last_clean":"2024-03-23T11:20:26.210227+0000","last_became_active":"2024-03-23T11:20:24.774705+0000","last_became_peered":"2024-03-23T11:20:24.774705+0000","last_unstale":"2024-03-23T11:20:26.210227+0000","last_undegraded":"2024-03-23T11:20:26.210227+0000","last_fullsized":"2024-03-23T11:20:26.210227+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:00:28.054426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.766533+0000","last_change":"2024-03-23T11:20:24.263872+0000","last_active":"2024-03-23T11:20:24.766533+0000","last_peered":"2024-03-23T11:20:24.766533+0000","last_clean":"2024-03-23T11:20:24.766533+0000","last_became_active":"2024-03-23T11:20:24.263540+0000","last_became_peered":"2024-03-23T11:20:24.263540+0000","last_unstale":"2024-03-23T11:20:24.766533+0000","last_undegraded":"2024-03-23T11:20:24.766533+0000","last_fullsized":"2024-03-23T11:20:24.766533+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:17:45.841557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.209614+0000","last_change":"2024-03-23T11:20:24.774237+0000","last_active":"2024-03-23T11:20:26.209614+0000","last_peered":"2024-03-23T11:20:26.209614+0000","last_clean":"2024-03-23T11:20:26.209614+0000","last_became_active":"2024-03-23T11:20:24.773941+0000","last_became_peered":"2024-03-23T11:20:24.773941+0000","last_unstale":"2024-03-23T11:20:26.209614+0000","last_undegraded":"2024-03-23T11:20:26.209614+0000","last_fullsized":"2024-03-23T11:20:26.209614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:04:49.659198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295939+0000","last_change":"2024-03-23T11:20:24.258520+0000","last_active":"2024-03-23T11:20:26.295939+0000","last_peered":"2024-03-23T11:20:26.295939+0000","last_clean":"2024-03-23T11:20:26.295939+0000","last_became_active":"2024-03-23T11:20:24.258044+0000","last_became_peered":"2024-03-23T11:20:24.258044+0000","last_unstale":"2024-03-23T11:20:26.295939+0000","last_undegraded":"2024-03-23T11:20:26.295939+0000","last_fullsized":"2024-03-23T11:20:26.295939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:40:11.857041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.210026+0000","last_change":"2024-03-23T11:20:24.763014+0000","last_active":"2024-03-23T11:20:26.210026+0000","last_peered":"2024-03-23T11:20:26.210026+0000","last_clean":"2024-03-23T11:20:26.210026+0000","last_became_active":"2024-03-23T11:20:24.762743+0000","last_became_peered":"2024-03-23T11:20:24.762743+0000","last_unstale":"2024-03-23T11:20:26.210026+0000","last_undegraded":"2024-03-23T11:20:26.210026+0000","last_fullsized":"2024-03-23T11:20:26.210026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:08:35.481669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.766920+0000","last_change":"2024-03-23T11:20:24.270394+0000","last_active":"2024-03-23T11:20:24.766920+0000","last_peered":"2024-03-23T11:20:24.766920+0000","last_clean":"2024-03-23T11:20:24.766920+0000","last_became_active":"2024-03-23T11:20:24.270017+0000","last_became_peered":"2024-03-23T11:20:24.270017+0000","last_unstale":"2024-03-23T11:20:24.766920+0000","last_undegraded":"2024-03-23T11:20:24.766920+0000","last_fullsized":"2024-03-23T11:20:24.766920+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:56.780601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.209658+0000","last_change":"2024-03-23T11:20:24.767387+0000","last_active":"2024-03-23T11:20:26.209658+0000","last_peered":"2024-03-23T11:20:26.209658+0000","last_clean":"2024-03-23T11:20:26.209658+0000","last_became_active":"2024-03-23T11:20:24.767061+0000","last_became_peered":"2024-03-23T11:20:24.767061+0000","last_unstale":"2024-03-23T11:20:26.209658+0000","last_undegraded":"2024-03-23T11:20:26.209658+0000","last_fullsized":"2024-03-23T11:20:26.209658+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:57.796033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.763652+0000","last_change":"2024-03-23T11:20:24.268829+0000","last_active":"2024-03-23T11:20:24.763652+0000","last_peered":"2024-03-23T11:20:24.763652+0000","last_clean":"2024-03-23T11:20:24.763652+0000","last_became_active":"2024-03-23T11:20:24.268449+0000","last_became_peered":"2024-03-23T11:20:24.268449+0000","last_unstale":"2024-03-23T11:20:24.763652+0000","last_undegraded":"2024-03-23T11:20:24.763652+0000","last_fullsized":"2024-03-23T11:20:24.763652+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:18:02.329265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.745824+0000","last_change":"2024-03-23T11:20:24.779856+0000","last_active":"2024-03-23T11:20:25.745824+0000","last_peered":"2024-03-23T11:20:25.745824+0000","last_clean":"2024-03-23T11:20:25.745824+0000","last_became_active":"2024-03-23T11:20:24.779505+0000","last_became_peered":"2024-03-23T11:20:24.779505+0000","last_unstale":"2024-03-23T11:20:25.745824+0000","last_undegraded":"2024-03-23T11:20:25.745824+0000","last_fullsized":"2024-03-23T11:20:25.745824+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:18:45.214042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.748976+0000","last_change":"2024-03-23T11:20:24.270722+0000","last_active":"2024-03-23T11:20:24.748976+0000","last_peered":"2024-03-23T11:20:24.748976+0000","last_clean":"2024-03-23T11:20:24.748976+0000","last_became_active":"2024-03-23T11:20:24.270020+0000","last_became_peered":"2024-03-23T11:20:24.270020+0000","last_unstale":"2024-03-23T11:20:24.748976+0000","last_undegraded":"2024-03-23T11:20:24.748976+0000","last_fullsized":"2024-03-23T11:20:24.748976+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:15.653884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.210242+0000","last_change":"2024-03-23T11:20:24.775999+0000","last_active":"2024-03-23T11:20:26.210242+0000","last_peered":"2024-03-23T11:20:26.210242+0000","last_clean":"2024-03-23T11:20:26.210242+0000","last_became_active":"2024-03-23T11:20:24.775730+0000","last_became_peered":"2024-03-23T11:20:24.775730+0000","last_unstale":"2024-03-23T11:20:26.210242+0000","last_undegraded":"2024-03-23T11:20:26.210242+0000","last_fullsized":"2024-03-23T11:20:26.210242+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:08:28.525913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.776530+0000","last_change":"2024-03-23T11:20:24.271790+0000","last_active":"2024-03-23T11:20:24.776530+0000","last_peered":"2024-03-23T11:20:24.776530+0000","last_clean":"2024-03-23T11:20:24.776530+0000","last_became_active":"2024-03-23T11:20:24.271405+0000","last_became_peered":"2024-03-23T11:20:24.271405+0000","last_unstale":"2024-03-23T11:20:24.776530+0000","last_undegraded":"2024-03-23T11:20:24.776530+0000","last_fullsized":"2024-03-23T11:20:24.776530+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:29:57.348200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296288+0000","last_change":"2024-03-23T11:20:24.773335+0000","last_active":"2024-03-23T11:20:26.296288+0000","last_peered":"2024-03-23T11:20:26.296288+0000","last_clean":"2024-03-23T11:20:26.296288+0000","last_became_active":"2024-03-23T11:20:24.773061+0000","last_became_peered":"2024-03-23T11:20:24.773061+0000","last_unstale":"2024-03-23T11:20:26.296288+0000","last_undegraded":"2024-03-23T11:20:26.296288+0000","last_fullsized":"2024-03-23T11:20:26.296288+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:25:35.850555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296391+0000","last_change":"2024-03-23T11:20:24.261390+0000","last_active":"2024-03-23T11:20:26.296391+0000","last_peered":"2024-03-23T11:20:26.296391+0000","last_clean":"2024-03-23T11:20:26.296391+0000","last_became_active":"2024-03-23T11:20:24.259776+0000","last_became_peered":"2024-03-23T11:20:24.259776+0000","last_unstale":"2024-03-23T11:20:26.296391+0000","last_undegraded":"2024-03-23T11:20:26.296391+0000","last_fullsized":"2024-03-23T11:20:26.296391+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:54.976104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296019+0000","last_change":"2024-03-23T11:20:24.765429+0000","last_active":"2024-03-23T11:20:26.296019+0000","last_peered":"2024-03-23T11:20:26.296019+0000","last_clean":"2024-03-23T11:20:26.296019+0000","last_became_active":"2024-03-23T11:20:24.765192+0000","last_became_peered":"2024-03-23T11:20:24.765192+0000","last_unstale":"2024-03-23T11:20:26.296019+0000","last_undegraded":"2024-03-23T11:20:26.296019+0000","last_fullsized":"2024-03-23T11:20:26.296019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:42:55.664530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.748349+0000","last_change":"2024-03-23T11:20:24.261689+0000","last_active":"2024-03-23T11:20:24.748349+0000","last_peered":"2024-03-23T11:20:24.748349+0000","last_clean":"2024-03-23T11:20:24.748349+0000","last_became_active":"2024-03-23T11:20:24.261283+0000","last_became_peered":"2024-03-23T11:20:24.261283+0000","last_unstale":"2024-03-23T11:20:24.748349+0000","last_undegraded":"2024-03-23T11:20:24.748349+0000","last_fullsized":"2024-03-23T11:20:24.748349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:45:05.172226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.885392+0000","last_change":"2024-03-23T11:20:24.769221+0000","last_active":"2024-03-23T11:20:25.885392+0000","last_peered":"2024-03-23T11:20:25.885392+0000","last_clean":"2024-03-23T11:20:25.885392+0000","last_became_active":"2024-03-23T11:20:24.768903+0000","last_became_peered":"2024-03-23T11:20:24.768903+0000","last_unstale":"2024-03-23T11:20:25.885392+0000","last_undegraded":"2024-03-23T11:20:25.885392+0000","last_fullsized":"2024-03-23T11:20:25.885392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:45.264945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296197+0000","last_change":"2024-03-23T11:20:24.273155+0000","last_active":"2024-03-23T11:20:26.296197+0000","last_peered":"2024-03-23T11:20:26.296197+0000","last_clean":"2024-03-23T11:20:26.296197+0000","last_became_active":"2024-03-23T11:20:24.272765+0000","last_became_peered":"2024-03-23T11:20:24.272765+0000","last_unstale":"2024-03-23T11:20:26.296197+0000","last_undegraded":"2024-03-23T11:20:26.296197+0000","last_fullsized":"2024-03-23T11:20:26.296197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.609703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295786+0000","last_change":"2024-03-23T11:20:24.762140+0000","last_active":"2024-03-23T11:20:26.295786+0000","last_peered":"2024-03-23T11:20:26.295786+0000","last_clean":"2024-03-23T11:20:26.295786+0000","last_became_active":"2024-03-23T11:20:24.761475+0000","last_became_peered":"2024-03-23T11:20:24.761475+0000","last_unstale":"2024-03-23T11:20:26.295786+0000","last_undegraded":"2024-03-23T11:20:26.295786+0000","last_fullsized":"2024-03-23T11:20:26.295786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:04:49.659198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.749666+0000","last_change":"2024-03-23T11:20:24.272555+0000","last_active":"2024-03-23T11:20:24.749666+0000","last_peered":"2024-03-23T11:20:24.749666+0000","last_clean":"2024-03-23T11:20:24.749666+0000","last_became_active":"2024-03-23T11:20:24.272270+0000","last_became_peered":"2024-03-23T11:20:24.272270+0000","last_unstale":"2024-03-23T11:20:24.749666+0000","last_undegraded":"2024-03-23T11:20:24.749666+0000","last_fullsized":"2024-03-23T11:20:24.749666+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:25:14.851311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.810984+0000","last_change":"2024-03-23T11:20:24.780005+0000","last_active":"2024-03-23T11:20:25.810984+0000","last_peered":"2024-03-23T11:20:25.810984+0000","last_clean":"2024-03-23T11:20:25.810984+0000","last_became_active":"2024-03-23T11:20:24.775265+0000","last_became_peered":"2024-03-23T11:20:24.775265+0000","last_unstale":"2024-03-23T11:20:25.810984+0000","last_undegraded":"2024-03-23T11:20:25.810984+0000","last_fullsized":"2024-03-23T11:20:25.810984+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:25:35.850555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.763799+0000","last_change":"2024-03-23T11:20:24.275279+0000","last_active":"2024-03-23T11:20:24.763799+0000","last_peered":"2024-03-23T11:20:24.763799+0000","last_clean":"2024-03-23T11:20:24.763799+0000","last_became_active":"2024-03-23T11:20:24.274581+0000","last_became_peered":"2024-03-23T11:20:24.274581+0000","last_unstale":"2024-03-23T11:20:24.763799+0000","last_undegraded":"2024-03-23T11:20:24.763799+0000","last_fullsized":"2024-03-23T11:20:24.763799+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:50.493955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746341+0000","last_change":"2024-03-23T11:20:24.765049+0000","last_active":"2024-03-23T11:20:25.746341+0000","last_peered":"2024-03-23T11:20:25.746341+0000","last_clean":"2024-03-23T11:20:25.746341+0000","last_became_active":"2024-03-23T11:20:24.764610+0000","last_became_peered":"2024-03-23T11:20:24.764610+0000","last_unstale":"2024-03-23T11:20:25.746341+0000","last_undegraded":"2024-03-23T11:20:25.746341+0000","last_fullsized":"2024-03-23T11:20:25.746341+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:13.726077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.739921+0000","last_change":"2024-03-23T11:20:24.265565+0000","last_active":"2024-03-23T11:20:24.739921+0000","last_peered":"2024-03-23T11:20:24.739921+0000","last_clean":"2024-03-23T11:20:24.739921+0000","last_became_active":"2024-03-23T11:20:24.265287+0000","last_became_peered":"2024-03-23T11:20:24.265287+0000","last_unstale":"2024-03-23T11:20:24.739921+0000","last_undegraded":"2024-03-23T11:20:24.739921+0000","last_fullsized":"2024-03-23T11:20:24.739921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:25:01.633915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747410+0000","last_change":"2024-03-23T11:20:24.766725+0000","last_active":"2024-03-23T11:20:25.747410+0000","last_peered":"2024-03-23T11:20:25.747410+0000","last_clean":"2024-03-23T11:20:25.747410+0000","last_became_active":"2024-03-23T11:20:24.765503+0000","last_became_peered":"2024-03-23T11:20:24.765503+0000","last_unstale":"2024-03-23T11:20:25.747410+0000","last_undegraded":"2024-03-23T11:20:25.747410+0000","last_fullsized":"2024-03-23T11:20:25.747410+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:40:24.562542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.767926+0000","last_change":"2024-03-23T11:20:24.269202+0000","last_active":"2024-03-23T11:20:24.767926+0000","last_peered":"2024-03-23T11:20:24.767926+0000","last_clean":"2024-03-23T11:20:24.767926+0000","last_became_active":"2024-03-23T11:20:24.268912+0000","last_became_peered":"2024-03-23T11:20:24.268912+0000","last_unstale":"2024-03-23T11:20:24.767926+0000","last_undegraded":"2024-03-23T11:20:24.767926+0000","last_fullsized":"2024-03-23T11:20:24.767926+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:22:07.120173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.755776+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:20:24.755776+0000","last_peered":"2024-03-23T11:20:24.755776+0000","last_clean":"2024-03-23T11:20:24.755776+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:20:24.755776+0000","last_undegraded":"2024-03-23T11:20:24.755776+0000","last_fullsized":"2024-03-23T11:20:24.755776+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542167,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":1003520,"data_stored":750237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.751}]},{"osd":1,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":2.3820000000000001,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.3700000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":1.081}]},{"osd":3,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":1.3140000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.345,"5min":1.345,"15min":1.345},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.29}]},{"osd":5,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.0859999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.248}]},{"osd":6,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.421,"15min":1.421},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.129}]}]},{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":159726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":0.84399999999999997}]},{"osd":1,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.0859999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.331}]},{"osd":2,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.3439999999999999,"5min":2.3439999999999999,"15min":2.3439999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.1520000000000001},"last":0.97999999999999998}]},{"osd":3,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":1.173}]},{"osd":4,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.113},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":1.5620000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":3.097,"5min":3.097,"15min":3.097},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.8109999999999999,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.26}]},{"osd":7,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":4.0880000000000001,"5min":4.0880000000000001,"15min":4.0880000000000001},"last":1.653},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":2.581,"5min":2.581,"15min":2.581},"last":1.5209999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":160451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.355,"5min":2.355,"15min":2.355},"last":1.508}]},{"osd":1,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.6559999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.345}]},{"osd":2,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":1.242},{"interface":"front","average":{"1min":1.248,"5min":1.248,"15min":1.248},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":1.552}]},{"osd":3,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.147}]},{"osd":4,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.8129999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.73}]},{"osd":7,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.611,"15min":1.611},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":2.516,"5min":2.516,"15min":2.516},"last":1.7729999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542167,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":159883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.651,"5min":1.651,"15min":1.651},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.6339999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.385}]},{"osd":2,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":1.419}]},{"osd":3,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.131},{"interface":"front","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.86899999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":2.1819999999999999,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.0720000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.319,"5min":2.319,"15min":2.319},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.2549999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.587},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":1.02,"5min":1.02,"15min":1.02},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.0269999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542167,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":421888,"data_stored":161365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.226},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":1.2929999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.163},{"interface":"front","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.391}]},{"osd":2,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":3.3109999999999999,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":1.274},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.7109999999999999,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.4850000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.3490000000000002,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.544},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"min":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"max":{"1min":4.8970000000000002,"5min":4.8970000000000002,"15min":4.8970000000000002},"last":1.6180000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.552,"15min":1.552},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":3.4910000000000001,"5min":3.4910000000000001,"15min":3.4910000000000001},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.504,"5min":1.504,"15min":1.504},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.125,"5min":3.125,"15min":3.125},"last":1.458}]},{"osd":6,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":4.5499999999999998,"5min":4.5499999999999998,"15min":4.5499999999999998},"last":1.8089999999999999},{"interface":"front","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":1.677}]},{"osd":7,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":2.952,"5min":2.952,"15min":2.952},"last":1.645},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":4.798,"5min":4.798,"15min":4.798},"last":1.877}]}]},{"osd":2,"up_from":14,"seq":60129542167,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":159849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":1.1970000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":1.1519999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.087},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.93600000000000005}]},{"osd":3,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":1.343}]},{"osd":4,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.595},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":1.647}]},{"osd":5,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.272},{"interface":"front","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.464}]},{"osd":6,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":2.758,"5min":2.758,"15min":2.758},"last":1.401},{"interface":"front","average":{"1min":1.546,"5min":1.546,"15min":1.546},"min":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"max":{"1min":2.8580000000000001,"5min":2.8580000000000001,"15min":2.8580000000000001},"last":1.5289999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.484},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.8320000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":417792,"data_stored":160696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":0.877}]},{"osd":2,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.2090000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.4169999999999998,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.458}]},{"osd":4,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":1.8959999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.353,"15min":1.353},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":2.758,"5min":2.758,"15min":2.758},"last":1.2569999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.6589999999999998,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.5129999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.778},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"max":{"1min":2.9630000000000001,"5min":2.9630000000000001,"15min":2.9630000000000001},"last":1.3500000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"max":{"1min":3.2749999999999999,"5min":3.2749999999999999,"15min":3.2749999999999999},"last":1.595},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.7829999999999999,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.9970000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":749478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":1.034},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.125}]},{"osd":2,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.224},{"interface":"front","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.3919999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.103,"15min":1.103},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.1819999999999999,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.9670000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.1739999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.8719999999999999,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":2.335}]},{"osd":6,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":2.5499999999999998,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":2.4399999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.446,"15min":1.446},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.677}]},{"osd":7,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.617,"5min":2.617,"15min":2.617},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":3.081,"5min":3.081,"15min":3.081},"last":1.4810000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:20:29.731 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:20:29.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.987+0000 7fbc16000640 1 Processor -- start 2024-03-23T11:20:29.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.987+0000 7fbc16000640 1 -- start start 2024-03-23T11:20:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.989+0000 7fbc16000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc10070090 0x7fbc100704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.989+0000 7fbc16000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbc100b2820 0x7fbc100b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.989+0000 7fbc16000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc10070a30 con 0x7fbc10143dd0 2024-03-23T11:20:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.989+0000 7fbc16000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc100cd140 con 0x7fbc100b2820 2024-03-23T11:20:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.989+0000 7fbc16000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc100b3180 con 0x7fbc10070090 2024-03-23T11:20:29.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.989+0000 7fbc15600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbc100b2820 0x7fbc100b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.990+0000 7fbc14c00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbc10143dd0 0x7fbc1015abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35226/0 (socket says 172.21.15.62:35226) 2024-03-23T11:20:29.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.990+0000 7fbc14c00640 1 -- 172.21.15.62:0/1922820702 learned_addr learned my addr 172.21.15.62:0/1922820702 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:29.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.990+0000 7fbc0fe00640 1 --2- 172.21.15.62:0/1922820702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc10070090 0x7fbc100704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.990+0000 7fbc0f400640 1 -- 172.21.15.62:0/1922820702 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219774172 0 0) 0x7fbc10070a30 con 0x7fbc10143dd0 2024-03-23T11:20:29.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.990+0000 7fbc0f400640 1 -- 172.21.15.62:0/1922820702 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf8003520 con 0x7fbc10143dd0 2024-03-23T11:20:29.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.990+0000 7fbc15600640 1 -- 172.21.15.62:0/1922820702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc10070090 msgr2=0x7fbc100704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:29.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.990+0000 7fbc15600640 1 --2- 172.21.15.62:0/1922820702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc10070090 0x7fbc100704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.990+0000 7fbc15600640 1 -- 172.21.15.62:0/1922820702 >> v1:172.21.15.62:6789/0 conn(0x7fbc10143dd0 legacy=0x7fbc1015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.990+0000 7fbc15600640 1 -- 172.21.15.62:0/1922820702 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc100cd3e0 con 0x7fbc100b2820 2024-03-23T11:20:29.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.991+0000 7fbc15600640 1 --2- 172.21.15.62:0/1922820702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc100b2820 0x7fbc100b2c40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009d90 tx=0x7fbc0802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9d77544f989028 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:29.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.992+0000 7fbc0f400640 1 -- 172.21.15.62:0/1922820702 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0802cc50 con 0x7fbc100b2820 2024-03-23T11:20:29.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.992+0000 7fbc0f400640 1 -- 172.21.15.62:0/1922820702 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0802cdb0 con 0x7fbc100b2820 2024-03-23T11:20:29.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.992+0000 7fbc0f400640 1 -- 172.21.15.62:0/1922820702 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08034660 con 0x7fbc100b2820 2024-03-23T11:20:29.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.992+0000 7fbc16000640 1 -- 172.21.15.62:0/1922820702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc100b2820 msgr2=0x7fbc100b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.992+0000 7fbc16000640 1 --2- 172.21.15.62:0/1922820702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc100b2820 0x7fbc100b2c40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009d90 tx=0x7fbc0802bea0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.992+0000 7fbc16000640 1 -- 172.21.15.62:0/1922820702 shutdown_connections 2024-03-23T11:20:29.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.992+0000 7fbc16000640 1 --2- 172.21.15.62:0/1922820702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc100b2820 0x7fbc100b2c40 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.992+0000 7fbc16000640 1 --2- 172.21.15.62:0/1922820702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc10070090 0x7fbc100704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.992+0000 7fbc16000640 1 -- 172.21.15.62:0/1922820702 >> 172.21.15.62:0/1922820702 conn(0x7fbc10066050 msgr2=0x7fbc10069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:29.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.992+0000 7fbc16000640 1 -- 172.21.15.62:0/1922820702 shutdown_connections 2024-03-23T11:20:29.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.992+0000 7fbc16000640 1 -- 172.21.15.62:0/1922820702 wait complete. 2024-03-23T11:20:29.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.994+0000 7fbc16000640 1 Processor -- start 2024-03-23T11:20:29.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.994+0000 7fbc16000640 1 -- start start 2024-03-23T11:20:29.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.995+0000 7fbc16000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc10070090 0x7fbc100c5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.995+0000 7fbc16000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc100b2820 0x7fbc10159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.995+0000 7fbc16000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc10143dd0 0x7fbc10159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:29.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.995+0000 7fbc16000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc100c6680 con 0x7fbc10070090 2024-03-23T11:20:29.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.995+0000 7fbc16000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc100c3b60 con 0x7fbc100b2820 2024-03-23T11:20:29.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.995+0000 7fbc16000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc100b0a10 con 0x7fbc10143dd0 2024-03-23T11:20:29.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.996+0000 7fbc14c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc10070090 0x7fbc100c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.996+0000 7fbc14c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc10070090 0x7fbc100c5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55828/0 (socket says 172.21.15.62:55828) 2024-03-23T11:20:29.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.996+0000 7fbc14c00640 1 -- 172.21.15.62:0/3225384611 learned_addr learned my addr 172.21.15.62:0/3225384611 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:29.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.996+0000 7fbc15600640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc10143dd0 0x7fbc10159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.996+0000 7fbc14c00640 1 -- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc10143dd0 msgr2=0x7fbc10159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.996+0000 7fbc14c00640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc10143dd0 0x7fbc10159d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.996+0000 7fbc14c00640 1 -- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc100b2820 msgr2=0x7fbc10159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:29.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.996+0000 7fbc0fe00640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc100b2820 0x7fbc10159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:29.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.996+0000 7fbc14c00640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc100b2820 0x7fbc10159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:29.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.996+0000 7fbc14c00640 1 -- 172.21.15.62:0/3225384611 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc08009a70 con 0x7fbc10070090 2024-03-23T11:20:29.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.996+0000 7fbc14c00640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc10070090 0x7fbc100c5950 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400c410 tx=0x7fbc0400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:29.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.997+0000 7fbc0cc00640 1 -- 172.21.15.62:0/3225384611 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc04005ce0 con 0x7fbc10070090 2024-03-23T11:20:29.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.997+0000 7fbc0cc00640 1 -- 172.21.15.62:0/3225384611 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc04005e40 con 0x7fbc10070090 2024-03-23T11:20:29.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.997+0000 7fbc0cc00640 1 -- 172.21.15.62:0/3225384611 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc040072f0 con 0x7fbc10070090 2024-03-23T11:20:29.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.997+0000 7fbc16000640 1 -- 172.21.15.62:0/3225384611 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc100b0730 con 0x7fbc10070090 2024-03-23T11:20:29.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.997+0000 7fbc16000640 1 -- 172.21.15.62:0/3225384611 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc100c3780 con 0x7fbc10070090 2024-03-23T11:20:29.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.998+0000 7fbc16000640 1 -- 172.21.15.62:0/3225384611 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc100c3300 con 0x7fbc10070090 2024-03-23T11:20:29.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:29.999+0000 7fbc0cc00640 1 -- 172.21.15.62:0/3225384611 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbc04007450 con 0x7fbc10070090 2024-03-23T11:20:30.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.000+0000 7fbc0cc00640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbf006a7d0 0x7fbbf006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:30.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.000+0000 7fbc0cc00640 1 -- 172.21.15.62:0/3225384611 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc04088420 con 0x7fbc10070090 2024-03-23T11:20:30.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.000+0000 7fbc0fe00640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbf006a7d0 0x7fbbf006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:30.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.001+0000 7fbc0fe00640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbf006a7d0 0x7fbbf006cc90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbc1015c540 tx=0x7fbbfc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:30.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.004+0000 7fbc0cc00640 1 -- 172.21.15.62:0/3225384611 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc100c3300 con 0x7fbc10070090 2024-03-23T11:20:30.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.205+0000 7fbc16000640 1 -- 172.21.15.62:0/3225384611 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbc100cdc80 con 0x7fbbf006a7d0 2024-03-23T11:20:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.227+0000 7fbc0cc00640 1 -- 172.21.15.62:0/3225384611 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286552 (secure 0 0 0) 0x7fbbfc010870 con 0x7fbbf006a7d0 2024-03-23T11:20:30.233 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:20:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 -- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbf006a7d0 msgr2=0x7fbbf006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbf006a7d0 0x7fbbf006cc90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbc1015c540 tx=0x7fbbfc008040 comp rx=0 tx=0).stop 2024-03-23T11:20:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 -- 172.21.15.62:0/3225384611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc10070090 msgr2=0x7fbc100c5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc10070090 0x7fbc100c5950 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400c410 tx=0x7fbc0400c8d0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 -- 172.21.15.62:0/3225384611 shutdown_connections 2024-03-23T11:20:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbf006a7d0 0x7fbbf006cc90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc10143dd0 0x7fbc10159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc100b2820 0x7fbc10159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 --2- 172.21.15.62:0/3225384611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc10070090 0x7fbc100c5950 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 -- 172.21.15.62:0/3225384611 >> 172.21.15.62:0/3225384611 conn(0x7fbc10066050 msgr2=0x7fbc100704d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 -- 172.21.15.62:0/3225384611 shutdown_connections 2024-03-23T11:20:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.234+0000 7fbc16000640 1 -- 172.21.15.62:0/3225384611 wait complete. 2024-03-23T11:20:30.255 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-03-23T11:20:29.264090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224644,"kb_used_data":4380,"kb_used_omap":13,"kb_used_meta":220146,"kb_avail":754750076,"statfs":{"total":773094113280,"available":772864077824,"internally_reserved":0,"allocated":4485120,"data_stored":2461685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":225429592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001789"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.885685+0000","last_change":"2024-03-23T11:20:24.780788+0000","last_active":"2024-03-23T11:20:25.885685+0000","last_peered":"2024-03-23T11:20:25.885685+0000","last_clean":"2024-03-23T11:20:25.885685+0000","last_became_active":"2024-03-23T11:20:24.780256+0000","last_became_peered":"2024-03-23T11:20:24.780256+0000","last_unstale":"2024-03-23T11:20:25.885685+0000","last_undegraded":"2024-03-23T11:20:25.885685+0000","last_fullsized":"2024-03-23T11:20:25.885685+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:41:17.793193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.811027+0000","last_change":"2024-03-23T11:20:24.773252+0000","last_active":"2024-03-23T11:20:25.811027+0000","last_peered":"2024-03-23T11:20:25.811027+0000","last_clean":"2024-03-23T11:20:25.811027+0000","last_became_active":"2024-03-23T11:20:24.772717+0000","last_became_peered":"2024-03-23T11:20:24.772717+0000","last_unstale":"2024-03-23T11:20:25.811027+0000","last_undegraded":"2024-03-23T11:20:25.811027+0000","last_fullsized":"2024-03-23T11:20:25.811027+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:58:08.558907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295986+0000","last_change":"2024-03-23T11:20:24.764765+0000","last_active":"2024-03-23T11:20:26.295986+0000","last_peered":"2024-03-23T11:20:26.295986+0000","last_clean":"2024-03-23T11:20:26.295986+0000","last_became_active":"2024-03-23T11:20:24.758563+0000","last_became_peered":"2024-03-23T11:20:24.758563+0000","last_unstale":"2024-03-23T11:20:26.295986+0000","last_undegraded":"2024-03-23T11:20:26.295986+0000","last_fullsized":"2024-03-23T11:20:26.295986+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:15.019627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.743025+0000","last_change":"2024-03-23T11:20:24.777436+0000","last_active":"2024-03-23T11:20:25.743025+0000","last_peered":"2024-03-23T11:20:25.743025+0000","last_clean":"2024-03-23T11:20:25.743025+0000","last_became_active":"2024-03-23T11:20:24.777221+0000","last_became_peered":"2024-03-23T11:20:24.777221+0000","last_unstale":"2024-03-23T11:20:25.743025+0000","last_undegraded":"2024-03-23T11:20:25.743025+0000","last_fullsized":"2024-03-23T11:20:25.743025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.152151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.209710+0000","last_change":"2024-03-23T11:20:24.780185+0000","last_active":"2024-03-23T11:20:26.209710+0000","last_peered":"2024-03-23T11:20:26.209710+0000","last_clean":"2024-03-23T11:20:26.209710+0000","last_became_active":"2024-03-23T11:20:24.779904+0000","last_became_peered":"2024-03-23T11:20:24.779904+0000","last_unstale":"2024-03-23T11:20:26.209710+0000","last_undegraded":"2024-03-23T11:20:26.209710+0000","last_fullsized":"2024-03-23T11:20:26.209710+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:42:55.664530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.810924+0000","last_change":"2024-03-23T11:20:24.774166+0000","last_active":"2024-03-23T11:20:25.810924+0000","last_peered":"2024-03-23T11:20:25.810924+0000","last_clean":"2024-03-23T11:20:25.810924+0000","last_became_active":"2024-03-23T11:20:24.773926+0000","last_became_peered":"2024-03-23T11:20:24.773926+0000","last_unstale":"2024-03-23T11:20:25.810924+0000","last_undegraded":"2024-03-23T11:20:25.810924+0000","last_fullsized":"2024-03-23T11:20:25.810924+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:30.582986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747296+0000","last_change":"2024-03-23T11:20:24.782449+0000","last_active":"2024-03-23T11:20:25.747296+0000","last_peered":"2024-03-23T11:20:25.747296+0000","last_clean":"2024-03-23T11:20:25.747296+0000","last_became_active":"2024-03-23T11:20:24.782082+0000","last_became_peered":"2024-03-23T11:20:24.782082+0000","last_unstale":"2024-03-23T11:20:25.747296+0000","last_undegraded":"2024-03-23T11:20:25.747296+0000","last_fullsized":"2024-03-23T11:20:25.747296+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.878125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.745869+0000","last_change":"2024-03-23T11:20:24.761096+0000","last_active":"2024-03-23T11:20:25.745869+0000","last_peered":"2024-03-23T11:20:25.745869+0000","last_clean":"2024-03-23T11:20:25.745869+0000","last_became_active":"2024-03-23T11:20:24.760771+0000","last_became_peered":"2024-03-23T11:20:24.760771+0000","last_unstale":"2024-03-23T11:20:25.745869+0000","last_undegraded":"2024-03-23T11:20:25.745869+0000","last_fullsized":"2024-03-23T11:20:25.745869+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:14.345002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747878+0000","last_change":"2024-03-23T11:20:24.783056+0000","last_active":"2024-03-23T11:20:25.747878+0000","last_peered":"2024-03-23T11:20:25.747878+0000","last_clean":"2024-03-23T11:20:25.747878+0000","last_became_active":"2024-03-23T11:20:24.782703+0000","last_became_peered":"2024-03-23T11:20:24.782703+0000","last_unstale":"2024-03-23T11:20:25.747878+0000","last_undegraded":"2024-03-23T11:20:25.747878+0000","last_fullsized":"2024-03-23T11:20:25.747878+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:00.613118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.742921+0000","last_change":"2024-03-23T11:20:24.778368+0000","last_active":"2024-03-23T11:20:25.742921+0000","last_peered":"2024-03-23T11:20:25.742921+0000","last_clean":"2024-03-23T11:20:25.742921+0000","last_became_active":"2024-03-23T11:20:24.777711+0000","last_became_peered":"2024-03-23T11:20:24.777711+0000","last_unstale":"2024-03-23T11:20:25.742921+0000","last_undegraded":"2024-03-23T11:20:25.742921+0000","last_fullsized":"2024-03-23T11:20:25.742921+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:57.399282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746388+0000","last_change":"2024-03-23T11:20:24.768454+0000","last_active":"2024-03-23T11:20:25.746388+0000","last_peered":"2024-03-23T11:20:25.746388+0000","last_clean":"2024-03-23T11:20:25.746388+0000","last_became_active":"2024-03-23T11:20:24.768167+0000","last_became_peered":"2024-03-23T11:20:24.768167+0000","last_unstale":"2024-03-23T11:20:25.746388+0000","last_undegraded":"2024-03-23T11:20:25.746388+0000","last_fullsized":"2024-03-23T11:20:25.746388+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:57.399282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295829+0000","last_change":"2024-03-23T11:20:24.770561+0000","last_active":"2024-03-23T11:20:26.295829+0000","last_peered":"2024-03-23T11:20:26.295829+0000","last_clean":"2024-03-23T11:20:26.295829+0000","last_became_active":"2024-03-23T11:20:24.770207+0000","last_became_peered":"2024-03-23T11:20:24.770207+0000","last_unstale":"2024-03-23T11:20:26.295829+0000","last_undegraded":"2024-03-23T11:20:26.295829+0000","last_fullsized":"2024-03-23T11:20:26.295829+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:51:36.049584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.209856+0000","last_change":"2024-03-23T11:20:24.762498+0000","last_active":"2024-03-23T11:20:26.209856+0000","last_peered":"2024-03-23T11:20:26.209856+0000","last_clean":"2024-03-23T11:20:26.209856+0000","last_became_active":"2024-03-23T11:20:24.762155+0000","last_became_peered":"2024-03-23T11:20:24.762155+0000","last_unstale":"2024-03-23T11:20:26.209856+0000","last_undegraded":"2024-03-23T11:20:26.209856+0000","last_fullsized":"2024-03-23T11:20:26.209856+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:22:08.135604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.811259+0000","last_change":"2024-03-23T11:20:24.769694+0000","last_active":"2024-03-23T11:20:25.811259+0000","last_peered":"2024-03-23T11:20:25.811259+0000","last_clean":"2024-03-23T11:20:25.811259+0000","last_became_active":"2024-03-23T11:20:24.769411+0000","last_became_peered":"2024-03-23T11:20:24.769411+0000","last_unstale":"2024-03-23T11:20:25.811259+0000","last_undegraded":"2024-03-23T11:20:25.811259+0000","last_fullsized":"2024-03-23T11:20:25.811259+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:53:56.922627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295947+0000","last_change":"2024-03-23T11:20:24.766167+0000","last_active":"2024-03-23T11:20:26.295947+0000","last_peered":"2024-03-23T11:20:26.295947+0000","last_clean":"2024-03-23T11:20:26.295947+0000","last_became_active":"2024-03-23T11:20:24.765916+0000","last_became_peered":"2024-03-23T11:20:24.765916+0000","last_unstale":"2024-03-23T11:20:26.295947+0000","last_undegraded":"2024-03-23T11:20:26.295947+0000","last_fullsized":"2024-03-23T11:20:26.295947+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:08:28.525913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746590+0000","last_change":"2024-03-23T11:20:24.781114+0000","last_active":"2024-03-23T11:20:25.746590+0000","last_peered":"2024-03-23T11:20:25.746590+0000","last_clean":"2024-03-23T11:20:25.746590+0000","last_became_active":"2024-03-23T11:20:24.780855+0000","last_became_peered":"2024-03-23T11:20:24.780855+0000","last_unstale":"2024-03-23T11:20:25.746590+0000","last_undegraded":"2024-03-23T11:20:25.746590+0000","last_fullsized":"2024-03-23T11:20:25.746590+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:03:04.312998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747960+0000","last_change":"2024-03-23T11:20:24.781658+0000","last_active":"2024-03-23T11:20:25.747960+0000","last_peered":"2024-03-23T11:20:25.747960+0000","last_clean":"2024-03-23T11:20:25.747960+0000","last_became_active":"2024-03-23T11:20:24.781368+0000","last_became_peered":"2024-03-23T11:20:24.781368+0000","last_unstale":"2024-03-23T11:20:25.747960+0000","last_undegraded":"2024-03-23T11:20:25.747960+0000","last_fullsized":"2024-03-23T11:20:25.747960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:38.280685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.811120+0000","last_change":"2024-03-23T11:20:24.765059+0000","last_active":"2024-03-23T11:20:25.811120+0000","last_peered":"2024-03-23T11:20:25.811120+0000","last_clean":"2024-03-23T11:20:25.811120+0000","last_became_active":"2024-03-23T11:20:24.764735+0000","last_became_peered":"2024-03-23T11:20:24.764735+0000","last_unstale":"2024-03-23T11:20:25.811120+0000","last_undegraded":"2024-03-23T11:20:25.811120+0000","last_fullsized":"2024-03-23T11:20:25.811120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:22.166605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.810562+0000","last_change":"2024-03-23T11:20:24.768957+0000","last_active":"2024-03-23T11:20:25.810562+0000","last_peered":"2024-03-23T11:20:25.810562+0000","last_clean":"2024-03-23T11:20:25.810562+0000","last_became_active":"2024-03-23T11:20:24.768659+0000","last_became_peered":"2024-03-23T11:20:24.768659+0000","last_unstale":"2024-03-23T11:20:25.810562+0000","last_undegraded":"2024-03-23T11:20:25.810562+0000","last_fullsized":"2024-03-23T11:20:25.810562+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:51:36.049584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.209875+0000","last_change":"2024-03-23T11:20:24.774367+0000","last_active":"2024-03-23T11:20:26.209875+0000","last_peered":"2024-03-23T11:20:26.209875+0000","last_clean":"2024-03-23T11:20:26.209875+0000","last_became_active":"2024-03-23T11:20:24.774099+0000","last_became_peered":"2024-03-23T11:20:24.774099+0000","last_unstale":"2024-03-23T11:20:26.209875+0000","last_undegraded":"2024-03-23T11:20:26.209875+0000","last_fullsized":"2024-03-23T11:20:26.209875+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:14:57.844505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747244+0000","last_change":"2024-03-23T11:20:24.778194+0000","last_active":"2024-03-23T11:20:25.747244+0000","last_peered":"2024-03-23T11:20:25.747244+0000","last_clean":"2024-03-23T11:20:25.747244+0000","last_became_active":"2024-03-23T11:20:24.777870+0000","last_became_peered":"2024-03-23T11:20:24.777870+0000","last_unstale":"2024-03-23T11:20:25.747244+0000","last_undegraded":"2024-03-23T11:20:25.747244+0000","last_fullsized":"2024-03-23T11:20:25.747244+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:08:28.525913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746543+0000","last_change":"2024-03-23T11:20:24.783528+0000","last_active":"2024-03-23T11:20:25.746543+0000","last_peered":"2024-03-23T11:20:25.746543+0000","last_clean":"2024-03-23T11:20:25.746543+0000","last_became_active":"2024-03-23T11:20:24.779012+0000","last_became_peered":"2024-03-23T11:20:24.779012+0000","last_unstale":"2024-03-23T11:20:25.746543+0000","last_undegraded":"2024-03-23T11:20:25.746543+0000","last_fullsized":"2024-03-23T11:20:25.746543+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:41:17.793193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295520+0000","last_change":"2024-03-23T11:20:24.772686+0000","last_active":"2024-03-23T11:20:26.295520+0000","last_peered":"2024-03-23T11:20:26.295520+0000","last_clean":"2024-03-23T11:20:26.295520+0000","last_became_active":"2024-03-23T11:20:24.770924+0000","last_became_peered":"2024-03-23T11:20:24.770924+0000","last_unstale":"2024-03-23T11:20:26.295520+0000","last_undegraded":"2024-03-23T11:20:26.295520+0000","last_fullsized":"2024-03-23T11:20:26.295520+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:16.669316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295497+0000","last_change":"2024-03-23T11:20:24.763547+0000","last_active":"2024-03-23T11:20:26.295497+0000","last_peered":"2024-03-23T11:20:26.295497+0000","last_clean":"2024-03-23T11:20:26.295497+0000","last_became_active":"2024-03-23T11:20:24.763083+0000","last_became_peered":"2024-03-23T11:20:24.763083+0000","last_unstale":"2024-03-23T11:20:26.295497+0000","last_undegraded":"2024-03-23T11:20:26.295497+0000","last_fullsized":"2024-03-23T11:20:26.295497+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:44:59.527355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.743120+0000","last_change":"2024-03-23T11:20:24.765205+0000","last_active":"2024-03-23T11:20:25.743120+0000","last_peered":"2024-03-23T11:20:25.743120+0000","last_clean":"2024-03-23T11:20:25.743120+0000","last_became_active":"2024-03-23T11:20:24.764983+0000","last_became_peered":"2024-03-23T11:20:24.764983+0000","last_unstale":"2024-03-23T11:20:25.743120+0000","last_undegraded":"2024-03-23T11:20:25.743120+0000","last_fullsized":"2024-03-23T11:20:25.743120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:11:54.038528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.811219+0000","last_change":"2024-03-23T11:20:24.754291+0000","last_active":"2024-03-23T11:20:25.811219+0000","last_peered":"2024-03-23T11:20:25.811219+0000","last_clean":"2024-03-23T11:20:25.811219+0000","last_became_active":"2024-03-23T11:20:24.754023+0000","last_became_peered":"2024-03-23T11:20:24.754023+0000","last_unstale":"2024-03-23T11:20:25.811219+0000","last_undegraded":"2024-03-23T11:20:25.811219+0000","last_fullsized":"2024-03-23T11:20:25.811219+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:15.019627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.743233+0000","last_change":"2024-03-23T11:20:24.772136+0000","last_active":"2024-03-23T11:20:25.743233+0000","last_peered":"2024-03-23T11:20:25.743233+0000","last_clean":"2024-03-23T11:20:25.743233+0000","last_became_active":"2024-03-23T11:20:24.771864+0000","last_became_peered":"2024-03-23T11:20:24.771864+0000","last_unstale":"2024-03-23T11:20:25.743233+0000","last_undegraded":"2024-03-23T11:20:25.743233+0000","last_fullsized":"2024-03-23T11:20:25.743233+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:03:04.312998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296060+0000","last_change":"2024-03-23T11:20:24.771662+0000","last_active":"2024-03-23T11:20:26.296060+0000","last_peered":"2024-03-23T11:20:26.296060+0000","last_clean":"2024-03-23T11:20:26.296060+0000","last_became_active":"2024-03-23T11:20:24.770452+0000","last_became_peered":"2024-03-23T11:20:24.770452+0000","last_unstale":"2024-03-23T11:20:26.296060+0000","last_undegraded":"2024-03-23T11:20:26.296060+0000","last_fullsized":"2024-03-23T11:20:26.296060+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:13:42.990421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747956+0000","last_change":"2024-03-23T11:20:24.774529+0000","last_active":"2024-03-23T11:20:25.747956+0000","last_peered":"2024-03-23T11:20:25.747956+0000","last_clean":"2024-03-23T11:20:25.747956+0000","last_became_active":"2024-03-23T11:20:24.773297+0000","last_became_peered":"2024-03-23T11:20:24.773297+0000","last_unstale":"2024-03-23T11:20:25.747956+0000","last_undegraded":"2024-03-23T11:20:25.747956+0000","last_fullsized":"2024-03-23T11:20:25.747956+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:00.267569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746339+0000","last_change":"2024-03-23T11:20:24.764027+0000","last_active":"2024-03-23T11:20:25.746339+0000","last_peered":"2024-03-23T11:20:25.746339+0000","last_clean":"2024-03-23T11:20:25.746339+0000","last_became_active":"2024-03-23T11:20:24.762832+0000","last_became_peered":"2024-03-23T11:20:24.762832+0000","last_unstale":"2024-03-23T11:20:25.746339+0000","last_undegraded":"2024-03-23T11:20:25.746339+0000","last_fullsized":"2024-03-23T11:20:25.746339+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:30:39.630549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.885475+0000","last_change":"2024-03-23T11:20:24.770934+0000","last_active":"2024-03-23T11:20:25.885475+0000","last_peered":"2024-03-23T11:20:25.885475+0000","last_clean":"2024-03-23T11:20:25.885475+0000","last_became_active":"2024-03-23T11:20:24.770577+0000","last_became_peered":"2024-03-23T11:20:24.770577+0000","last_unstale":"2024-03-23T11:20:25.885475+0000","last_undegraded":"2024-03-23T11:20:25.885475+0000","last_fullsized":"2024-03-23T11:20:25.885475+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:08:28.525913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746121+0000","last_change":"2024-03-23T11:20:24.767795+0000","last_active":"2024-03-23T11:20:25.746121+0000","last_peered":"2024-03-23T11:20:25.746121+0000","last_clean":"2024-03-23T11:20:25.746121+0000","last_became_active":"2024-03-23T11:20:24.767433+0000","last_became_peered":"2024-03-23T11:20:24.767433+0000","last_unstale":"2024-03-23T11:20:25.746121+0000","last_undegraded":"2024-03-23T11:20:25.746121+0000","last_fullsized":"2024-03-23T11:20:25.746121+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.152151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296163+0000","last_change":"2024-03-23T11:20:24.769847+0000","last_active":"2024-03-23T11:20:26.296163+0000","last_peered":"2024-03-23T11:20:26.296163+0000","last_clean":"2024-03-23T11:20:26.296163+0000","last_became_active":"2024-03-23T11:20:24.769576+0000","last_became_peered":"2024-03-23T11:20:24.769576+0000","last_unstale":"2024-03-23T11:20:26.296163+0000","last_undegraded":"2024-03-23T11:20:26.296163+0000","last_fullsized":"2024-03-23T11:20:26.296163+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:00:28.054426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"35'4","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.782881+0000","last_change":"2024-03-23T11:20:24.274189+0000","last_active":"2024-03-23T11:20:24.782881+0000","last_peered":"2024-03-23T11:20:24.782881+0000","last_clean":"2024-03-23T11:20:24.782881+0000","last_became_active":"2024-03-23T11:20:24.273908+0000","last_became_peered":"2024-03-23T11:20:24.273908+0000","last_unstale":"2024-03-23T11:20:24.782881+0000","last_undegraded":"2024-03-23T11:20:24.782881+0000","last_fullsized":"2024-03-23T11:20:24.782881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:11.707558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.885987+0000","last_change":"2024-03-23T11:20:24.780587+0000","last_active":"2024-03-23T11:20:25.885987+0000","last_peered":"2024-03-23T11:20:25.885987+0000","last_clean":"2024-03-23T11:20:25.885987+0000","last_became_active":"2024-03-23T11:20:24.777836+0000","last_became_peered":"2024-03-23T11:20:24.777836+0000","last_unstale":"2024-03-23T11:20:25.885987+0000","last_undegraded":"2024-03-23T11:20:25.885987+0000","last_fullsized":"2024-03-23T11:20:25.885987+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:13.726077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296257+0000","last_change":"2024-03-23T11:20:24.263698+0000","last_active":"2024-03-23T11:20:26.296257+0000","last_peered":"2024-03-23T11:20:26.296257+0000","last_clean":"2024-03-23T11:20:26.296257+0000","last_became_active":"2024-03-23T11:20:24.263412+0000","last_became_peered":"2024-03-23T11:20:24.263412+0000","last_unstale":"2024-03-23T11:20:26.296257+0000","last_undegraded":"2024-03-23T11:20:26.296257+0000","last_fullsized":"2024-03-23T11:20:26.296257+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:29:31.328499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746704+0000","last_change":"2024-03-23T11:20:24.777302+0000","last_active":"2024-03-23T11:20:25.746704+0000","last_peered":"2024-03-23T11:20:25.746704+0000","last_clean":"2024-03-23T11:20:25.746704+0000","last_became_active":"2024-03-23T11:20:24.777028+0000","last_became_peered":"2024-03-23T11:20:24.777028+0000","last_unstale":"2024-03-23T11:20:25.746704+0000","last_undegraded":"2024-03-23T11:20:25.746704+0000","last_fullsized":"2024-03-23T11:20:25.746704+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:14:57.844505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.775960+0000","last_change":"2024-03-23T11:20:24.270930+0000","last_active":"2024-03-23T11:20:24.775960+0000","last_peered":"2024-03-23T11:20:24.775960+0000","last_clean":"2024-03-23T11:20:24.775960+0000","last_became_active":"2024-03-23T11:20:24.270535+0000","last_became_peered":"2024-03-23T11:20:24.270535+0000","last_unstale":"2024-03-23T11:20:24.775960+0000","last_undegraded":"2024-03-23T11:20:24.775960+0000","last_fullsized":"2024-03-23T11:20:24.775960+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:22:07.120173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.743392+0000","last_change":"2024-03-23T11:20:24.763737+0000","last_active":"2024-03-23T11:20:25.743392+0000","last_peered":"2024-03-23T11:20:25.743392+0000","last_clean":"2024-03-23T11:20:25.743392+0000","last_became_active":"2024-03-23T11:20:24.763511+0000","last_became_peered":"2024-03-23T11:20:24.763511+0000","last_unstale":"2024-03-23T11:20:25.743392+0000","last_undegraded":"2024-03-23T11:20:25.743392+0000","last_fullsized":"2024-03-23T11:20:25.743392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:14.345002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.767793+0000","last_change":"2024-03-23T11:20:24.264541+0000","last_active":"2024-03-23T11:20:24.767793+0000","last_peered":"2024-03-23T11:20:24.767793+0000","last_clean":"2024-03-23T11:20:24.767793+0000","last_became_active":"2024-03-23T11:20:24.264244+0000","last_became_peered":"2024-03-23T11:20:24.264244+0000","last_unstale":"2024-03-23T11:20:24.767793+0000","last_undegraded":"2024-03-23T11:20:24.767793+0000","last_fullsized":"2024-03-23T11:20:24.767793+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:31.618689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746592+0000","last_change":"2024-03-23T11:20:24.776629+0000","last_active":"2024-03-23T11:20:25.746592+0000","last_peered":"2024-03-23T11:20:25.746592+0000","last_clean":"2024-03-23T11:20:25.746592+0000","last_became_active":"2024-03-23T11:20:24.776314+0000","last_became_peered":"2024-03-23T11:20:24.776314+0000","last_unstale":"2024-03-23T11:20:25.746592+0000","last_undegraded":"2024-03-23T11:20:25.746592+0000","last_fullsized":"2024-03-23T11:20:25.746592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:06:48.779863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"35'1","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.783724+0000","last_change":"2024-03-23T11:20:24.269524+0000","last_active":"2024-03-23T11:20:24.783724+0000","last_peered":"2024-03-23T11:20:24.783724+0000","last_clean":"2024-03-23T11:20:24.783724+0000","last_became_active":"2024-03-23T11:20:24.268972+0000","last_became_peered":"2024-03-23T11:20:24.268972+0000","last_unstale":"2024-03-23T11:20:24.783724+0000","last_undegraded":"2024-03-23T11:20:24.783724+0000","last_fullsized":"2024-03-23T11:20:24.783724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:16:16.741588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295558+0000","last_change":"2024-03-23T11:20:24.766147+0000","last_active":"2024-03-23T11:20:26.295558+0000","last_peered":"2024-03-23T11:20:26.295558+0000","last_clean":"2024-03-23T11:20:26.295558+0000","last_became_active":"2024-03-23T11:20:24.765773+0000","last_became_peered":"2024-03-23T11:20:24.765773+0000","last_unstale":"2024-03-23T11:20:26.295558+0000","last_undegraded":"2024-03-23T11:20:26.295558+0000","last_fullsized":"2024-03-23T11:20:26.295558+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:00.613118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"35'1","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.783821+0000","last_change":"2024-03-23T11:20:24.269384+0000","last_active":"2024-03-23T11:20:24.783821+0000","last_peered":"2024-03-23T11:20:24.783821+0000","last_clean":"2024-03-23T11:20:24.783821+0000","last_became_active":"2024-03-23T11:20:24.268752+0000","last_became_peered":"2024-03-23T11:20:24.268752+0000","last_unstale":"2024-03-23T11:20:24.783821+0000","last_undegraded":"2024-03-23T11:20:24.783821+0000","last_fullsized":"2024-03-23T11:20:24.783821+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:31.618689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747224+0000","last_change":"2024-03-23T11:20:24.784288+0000","last_active":"2024-03-23T11:20:25.747224+0000","last_peered":"2024-03-23T11:20:25.747224+0000","last_clean":"2024-03-23T11:20:25.747224+0000","last_became_active":"2024-03-23T11:20:24.784012+0000","last_became_peered":"2024-03-23T11:20:24.784012+0000","last_unstale":"2024-03-23T11:20:25.747224+0000","last_undegraded":"2024-03-23T11:20:25.747224+0000","last_fullsized":"2024-03-23T11:20:25.747224+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:04:49.659198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.739922+0000","last_change":"2024-03-23T11:20:24.262869+0000","last_active":"2024-03-23T11:20:24.739922+0000","last_peered":"2024-03-23T11:20:24.739922+0000","last_clean":"2024-03-23T11:20:24.739922+0000","last_became_active":"2024-03-23T11:20:24.262525+0000","last_became_peered":"2024-03-23T11:20:24.262525+0000","last_unstale":"2024-03-23T11:20:24.739922+0000","last_undegraded":"2024-03-23T11:20:24.739922+0000","last_fullsized":"2024-03-23T11:20:24.739922+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:29:31.328499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.742476+0000","last_change":"2024-03-23T11:20:24.776155+0000","last_active":"2024-03-23T11:20:25.742476+0000","last_peered":"2024-03-23T11:20:25.742476+0000","last_clean":"2024-03-23T11:20:25.742476+0000","last_became_active":"2024-03-23T11:20:24.773562+0000","last_became_peered":"2024-03-23T11:20:24.773562+0000","last_unstale":"2024-03-23T11:20:25.742476+0000","last_undegraded":"2024-03-23T11:20:25.742476+0000","last_fullsized":"2024-03-23T11:20:25.742476+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:00:21.168162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.763039+0000","last_change":"2024-03-23T11:20:24.266106+0000","last_active":"2024-03-23T11:20:24.763039+0000","last_peered":"2024-03-23T11:20:24.763039+0000","last_clean":"2024-03-23T11:20:24.763039+0000","last_became_active":"2024-03-23T11:20:24.265744+0000","last_became_peered":"2024-03-23T11:20:24.265744+0000","last_unstale":"2024-03-23T11:20:24.763039+0000","last_undegraded":"2024-03-23T11:20:24.763039+0000","last_fullsized":"2024-03-23T11:20:24.763039+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.609703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296212+0000","last_change":"2024-03-23T11:20:24.771313+0000","last_active":"2024-03-23T11:20:26.296212+0000","last_peered":"2024-03-23T11:20:26.296212+0000","last_clean":"2024-03-23T11:20:26.296212+0000","last_became_active":"2024-03-23T11:20:24.770842+0000","last_became_peered":"2024-03-23T11:20:24.770842+0000","last_unstale":"2024-03-23T11:20:26.296212+0000","last_undegraded":"2024-03-23T11:20:26.296212+0000","last_fullsized":"2024-03-23T11:20:26.296212+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:22.166605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.740881+0000","last_change":"2024-03-23T11:20:24.265733+0000","last_active":"2024-03-23T11:20:24.740881+0000","last_peered":"2024-03-23T11:20:24.740881+0000","last_clean":"2024-03-23T11:20:24.740881+0000","last_became_active":"2024-03-23T11:20:24.265389+0000","last_became_peered":"2024-03-23T11:20:24.265389+0000","last_unstale":"2024-03-23T11:20:24.740881+0000","last_undegraded":"2024-03-23T11:20:24.740881+0000","last_fullsized":"2024-03-23T11:20:24.740881+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:11.707558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.811309+0000","last_change":"2024-03-23T11:20:24.764620+0000","last_active":"2024-03-23T11:20:25.811309+0000","last_peered":"2024-03-23T11:20:25.811309+0000","last_clean":"2024-03-23T11:20:25.811309+0000","last_became_active":"2024-03-23T11:20:24.764420+0000","last_became_peered":"2024-03-23T11:20:24.764420+0000","last_unstale":"2024-03-23T11:20:25.811309+0000","last_undegraded":"2024-03-23T11:20:25.811309+0000","last_fullsized":"2024-03-23T11:20:25.811309+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:25:01.738465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296388+0000","last_change":"2024-03-23T11:20:24.262933+0000","last_active":"2024-03-23T11:20:26.296388+0000","last_peered":"2024-03-23T11:20:26.296388+0000","last_clean":"2024-03-23T11:20:26.296388+0000","last_became_active":"2024-03-23T11:20:24.262659+0000","last_became_peered":"2024-03-23T11:20:24.262659+0000","last_unstale":"2024-03-23T11:20:26.296388+0000","last_undegraded":"2024-03-23T11:20:26.296388+0000","last_fullsized":"2024-03-23T11:20:26.296388+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:06.287580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.810951+0000","last_change":"2024-03-23T11:20:24.756469+0000","last_active":"2024-03-23T11:20:25.810951+0000","last_peered":"2024-03-23T11:20:25.810951+0000","last_clean":"2024-03-23T11:20:25.810951+0000","last_became_active":"2024-03-23T11:20:24.756225+0000","last_became_peered":"2024-03-23T11:20:24.756225+0000","last_unstale":"2024-03-23T11:20:25.810951+0000","last_undegraded":"2024-03-23T11:20:25.810951+0000","last_fullsized":"2024-03-23T11:20:25.810951+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:00.267569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.775039+0000","last_change":"2024-03-23T11:20:24.271111+0000","last_active":"2024-03-23T11:20:24.775039+0000","last_peered":"2024-03-23T11:20:24.775039+0000","last_clean":"2024-03-23T11:20:24.775039+0000","last_became_active":"2024-03-23T11:20:24.270844+0000","last_became_peered":"2024-03-23T11:20:24.270844+0000","last_unstale":"2024-03-23T11:20:24.775039+0000","last_undegraded":"2024-03-23T11:20:24.775039+0000","last_fullsized":"2024-03-23T11:20:24.775039+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:08:34.466237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295814+0000","last_change":"2024-03-23T11:20:24.764978+0000","last_active":"2024-03-23T11:20:26.295814+0000","last_peered":"2024-03-23T11:20:26.295814+0000","last_clean":"2024-03-23T11:20:26.295814+0000","last_became_active":"2024-03-23T11:20:24.763142+0000","last_became_peered":"2024-03-23T11:20:24.763142+0000","last_unstale":"2024-03-23T11:20:26.295814+0000","last_undegraded":"2024-03-23T11:20:26.295814+0000","last_fullsized":"2024-03-23T11:20:26.295814+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:30.582986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":17,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.787653+0000","last_change":"2024-03-23T11:20:24.268566+0000","last_active":"2024-03-23T11:20:24.787653+0000","last_peered":"2024-03-23T11:20:24.787653+0000","last_clean":"2024-03-23T11:20:24.787653+0000","last_became_active":"2024-03-23T11:20:24.268288+0000","last_became_peered":"2024-03-23T11:20:24.268288+0000","last_unstale":"2024-03-23T11:20:24.787653+0000","last_undegraded":"2024-03-23T11:20:24.787653+0000","last_fullsized":"2024-03-23T11:20:24.787653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:15.653884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.742556+0000","last_change":"2024-03-23T11:20:24.775094+0000","last_active":"2024-03-23T11:20:25.742556+0000","last_peered":"2024-03-23T11:20:25.742556+0000","last_clean":"2024-03-23T11:20:25.742556+0000","last_became_active":"2024-03-23T11:20:24.774861+0000","last_became_peered":"2024-03-23T11:20:24.774861+0000","last_unstale":"2024-03-23T11:20:25.742556+0000","last_undegraded":"2024-03-23T11:20:25.742556+0000","last_fullsized":"2024-03-23T11:20:25.742556+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:07:25.103486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"35'2","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.784123+0000","last_change":"2024-03-23T11:20:24.273073+0000","last_active":"2024-03-23T11:20:24.784123+0000","last_peered":"2024-03-23T11:20:24.784123+0000","last_clean":"2024-03-23T11:20:24.784123+0000","last_became_active":"2024-03-23T11:20:24.272435+0000","last_became_peered":"2024-03-23T11:20:24.272435+0000","last_unstale":"2024-03-23T11:20:24.784123+0000","last_undegraded":"2024-03-23T11:20:24.784123+0000","last_fullsized":"2024-03-23T11:20:24.784123+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:54.976104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.810727+0000","last_change":"2024-03-23T11:20:24.764404+0000","last_active":"2024-03-23T11:20:25.810727+0000","last_peered":"2024-03-23T11:20:25.810727+0000","last_clean":"2024-03-23T11:20:25.810727+0000","last_became_active":"2024-03-23T11:20:24.762524+0000","last_became_peered":"2024-03-23T11:20:24.762524+0000","last_unstale":"2024-03-23T11:20:25.810727+0000","last_undegraded":"2024-03-23T11:20:25.810727+0000","last_fullsized":"2024-03-23T11:20:25.810727+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:41:17.793193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.749531+0000","last_change":"2024-03-23T11:20:24.267580+0000","last_active":"2024-03-23T11:20:24.749531+0000","last_peered":"2024-03-23T11:20:24.749531+0000","last_clean":"2024-03-23T11:20:24.749531+0000","last_became_active":"2024-03-23T11:20:24.267272+0000","last_became_peered":"2024-03-23T11:20:24.267272+0000","last_unstale":"2024-03-23T11:20:24.749531+0000","last_undegraded":"2024-03-23T11:20:24.749531+0000","last_fullsized":"2024-03-23T11:20:24.749531+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:11:10.777710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746374+0000","last_change":"2024-03-23T11:20:24.774621+0000","last_active":"2024-03-23T11:20:25.746374+0000","last_peered":"2024-03-23T11:20:25.746374+0000","last_clean":"2024-03-23T11:20:25.746374+0000","last_became_active":"2024-03-23T11:20:24.774167+0000","last_became_peered":"2024-03-23T11:20:24.774167+0000","last_unstale":"2024-03-23T11:20:25.746374+0000","last_undegraded":"2024-03-23T11:20:25.746374+0000","last_fullsized":"2024-03-23T11:20:25.746374+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:57.318031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.740097+0000","last_change":"2024-03-23T11:20:24.265822+0000","last_active":"2024-03-23T11:20:24.740097+0000","last_peered":"2024-03-23T11:20:24.740097+0000","last_clean":"2024-03-23T11:20:24.740097+0000","last_became_active":"2024-03-23T11:20:24.264936+0000","last_became_peered":"2024-03-23T11:20:24.264936+0000","last_unstale":"2024-03-23T11:20:24.740097+0000","last_undegraded":"2024-03-23T11:20:24.740097+0000","last_fullsized":"2024-03-23T11:20:24.740097+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:40:11.857041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.885637+0000","last_change":"2024-03-23T11:20:24.776569+0000","last_active":"2024-03-23T11:20:25.885637+0000","last_peered":"2024-03-23T11:20:25.885637+0000","last_clean":"2024-03-23T11:20:25.885637+0000","last_became_active":"2024-03-23T11:20:24.776257+0000","last_became_peered":"2024-03-23T11:20:24.776257+0000","last_unstale":"2024-03-23T11:20:25.885637+0000","last_undegraded":"2024-03-23T11:20:25.885637+0000","last_fullsized":"2024-03-23T11:20:25.885637+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:37:00.267569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":19,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.784818+0000","last_change":"2024-03-23T11:20:24.269918+0000","last_active":"2024-03-23T11:20:24.784818+0000","last_peered":"2024-03-23T11:20:24.784818+0000","last_clean":"2024-03-23T11:20:24.784818+0000","last_became_active":"2024-03-23T11:20:24.269576+0000","last_became_peered":"2024-03-23T11:20:24.269576+0000","last_unstale":"2024-03-23T11:20:24.784818+0000","last_undegraded":"2024-03-23T11:20:24.784818+0000","last_fullsized":"2024-03-23T11:20:24.784818+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:56.780601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747467+0000","last_change":"2024-03-23T11:20:24.773626+0000","last_active":"2024-03-23T11:20:25.747467+0000","last_peered":"2024-03-23T11:20:25.747467+0000","last_clean":"2024-03-23T11:20:25.747467+0000","last_became_active":"2024-03-23T11:20:24.773265+0000","last_became_peered":"2024-03-23T11:20:24.773265+0000","last_unstale":"2024-03-23T11:20:25.747467+0000","last_undegraded":"2024-03-23T11:20:25.747467+0000","last_fullsized":"2024-03-23T11:20:25.747467+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:45.264945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.778831+0000","last_change":"2024-03-23T11:20:24.269588+0000","last_active":"2024-03-23T11:20:24.778831+0000","last_peered":"2024-03-23T11:20:24.778831+0000","last_clean":"2024-03-23T11:20:24.778831+0000","last_became_active":"2024-03-23T11:20:24.268693+0000","last_became_peered":"2024-03-23T11:20:24.268693+0000","last_unstale":"2024-03-23T11:20:24.778831+0000","last_undegraded":"2024-03-23T11:20:24.778831+0000","last_fullsized":"2024-03-23T11:20:24.778831+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:12:56.580754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296240+0000","last_change":"2024-03-23T11:20:24.776102+0000","last_active":"2024-03-23T11:20:26.296240+0000","last_peered":"2024-03-23T11:20:26.296240+0000","last_clean":"2024-03-23T11:20:26.296240+0000","last_became_active":"2024-03-23T11:20:24.775711+0000","last_became_peered":"2024-03-23T11:20:24.775711+0000","last_unstale":"2024-03-23T11:20:26.296240+0000","last_undegraded":"2024-03-23T11:20:26.296240+0000","last_fullsized":"2024-03-23T11:20:26.296240+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:58:08.558907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"35'2","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.784237+0000","last_change":"2024-03-23T11:20:24.269723+0000","last_active":"2024-03-23T11:20:24.784237+0000","last_peered":"2024-03-23T11:20:24.784237+0000","last_clean":"2024-03-23T11:20:24.784237+0000","last_became_active":"2024-03-23T11:20:24.269450+0000","last_became_peered":"2024-03-23T11:20:24.269450+0000","last_unstale":"2024-03-23T11:20:24.784237+0000","last_undegraded":"2024-03-23T11:20:24.784237+0000","last_fullsized":"2024-03-23T11:20:24.784237+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:06.287580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.743163+0000","last_change":"2024-03-23T11:20:24.766463+0000","last_active":"2024-03-23T11:20:25.743163+0000","last_peered":"2024-03-23T11:20:25.743163+0000","last_clean":"2024-03-23T11:20:25.743163+0000","last_became_active":"2024-03-23T11:20:24.766173+0000","last_became_peered":"2024-03-23T11:20:24.766173+0000","last_unstale":"2024-03-23T11:20:25.743163+0000","last_undegraded":"2024-03-23T11:20:25.743163+0000","last_fullsized":"2024-03-23T11:20:25.743163+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:57.318031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'1","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.775309+0000","last_change":"2024-03-23T11:20:24.271441+0000","last_active":"2024-03-23T11:20:24.775309+0000","last_peered":"2024-03-23T11:20:24.775309+0000","last_clean":"2024-03-23T11:20:24.775309+0000","last_became_active":"2024-03-23T11:20:24.270146+0000","last_became_peered":"2024-03-23T11:20:24.270146+0000","last_unstale":"2024-03-23T11:20:24.775309+0000","last_undegraded":"2024-03-23T11:20:24.775309+0000","last_fullsized":"2024-03-23T11:20:24.775309+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:49:18.386429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.210227+0000","last_change":"2024-03-23T11:20:24.774987+0000","last_active":"2024-03-23T11:20:26.210227+0000","last_peered":"2024-03-23T11:20:26.210227+0000","last_clean":"2024-03-23T11:20:26.210227+0000","last_became_active":"2024-03-23T11:20:24.774705+0000","last_became_peered":"2024-03-23T11:20:24.774705+0000","last_unstale":"2024-03-23T11:20:26.210227+0000","last_undegraded":"2024-03-23T11:20:26.210227+0000","last_fullsized":"2024-03-23T11:20:26.210227+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:00:28.054426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.766533+0000","last_change":"2024-03-23T11:20:24.263872+0000","last_active":"2024-03-23T11:20:24.766533+0000","last_peered":"2024-03-23T11:20:24.766533+0000","last_clean":"2024-03-23T11:20:24.766533+0000","last_became_active":"2024-03-23T11:20:24.263540+0000","last_became_peered":"2024-03-23T11:20:24.263540+0000","last_unstale":"2024-03-23T11:20:24.766533+0000","last_undegraded":"2024-03-23T11:20:24.766533+0000","last_fullsized":"2024-03-23T11:20:24.766533+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:17:45.841557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.209614+0000","last_change":"2024-03-23T11:20:24.774237+0000","last_active":"2024-03-23T11:20:26.209614+0000","last_peered":"2024-03-23T11:20:26.209614+0000","last_clean":"2024-03-23T11:20:26.209614+0000","last_became_active":"2024-03-23T11:20:24.773941+0000","last_became_peered":"2024-03-23T11:20:24.773941+0000","last_unstale":"2024-03-23T11:20:26.209614+0000","last_undegraded":"2024-03-23T11:20:26.209614+0000","last_fullsized":"2024-03-23T11:20:26.209614+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:04:49.659198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295939+0000","last_change":"2024-03-23T11:20:24.258520+0000","last_active":"2024-03-23T11:20:26.295939+0000","last_peered":"2024-03-23T11:20:26.295939+0000","last_clean":"2024-03-23T11:20:26.295939+0000","last_became_active":"2024-03-23T11:20:24.258044+0000","last_became_peered":"2024-03-23T11:20:24.258044+0000","last_unstale":"2024-03-23T11:20:26.295939+0000","last_undegraded":"2024-03-23T11:20:26.295939+0000","last_fullsized":"2024-03-23T11:20:26.295939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:40:11.857041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.210026+0000","last_change":"2024-03-23T11:20:24.763014+0000","last_active":"2024-03-23T11:20:26.210026+0000","last_peered":"2024-03-23T11:20:26.210026+0000","last_clean":"2024-03-23T11:20:26.210026+0000","last_became_active":"2024-03-23T11:20:24.762743+0000","last_became_peered":"2024-03-23T11:20:24.762743+0000","last_unstale":"2024-03-23T11:20:26.210026+0000","last_undegraded":"2024-03-23T11:20:26.210026+0000","last_fullsized":"2024-03-23T11:20:26.210026+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:08:35.481669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.766920+0000","last_change":"2024-03-23T11:20:24.270394+0000","last_active":"2024-03-23T11:20:24.766920+0000","last_peered":"2024-03-23T11:20:24.766920+0000","last_clean":"2024-03-23T11:20:24.766920+0000","last_became_active":"2024-03-23T11:20:24.270017+0000","last_became_peered":"2024-03-23T11:20:24.270017+0000","last_unstale":"2024-03-23T11:20:24.766920+0000","last_undegraded":"2024-03-23T11:20:24.766920+0000","last_fullsized":"2024-03-23T11:20:24.766920+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:56.780601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.209658+0000","last_change":"2024-03-23T11:20:24.767387+0000","last_active":"2024-03-23T11:20:26.209658+0000","last_peered":"2024-03-23T11:20:26.209658+0000","last_clean":"2024-03-23T11:20:26.209658+0000","last_became_active":"2024-03-23T11:20:24.767061+0000","last_became_peered":"2024-03-23T11:20:24.767061+0000","last_unstale":"2024-03-23T11:20:26.209658+0000","last_undegraded":"2024-03-23T11:20:26.209658+0000","last_fullsized":"2024-03-23T11:20:26.209658+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:57.796033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.763652+0000","last_change":"2024-03-23T11:20:24.268829+0000","last_active":"2024-03-23T11:20:24.763652+0000","last_peered":"2024-03-23T11:20:24.763652+0000","last_clean":"2024-03-23T11:20:24.763652+0000","last_became_active":"2024-03-23T11:20:24.268449+0000","last_became_peered":"2024-03-23T11:20:24.268449+0000","last_unstale":"2024-03-23T11:20:24.763652+0000","last_undegraded":"2024-03-23T11:20:24.763652+0000","last_fullsized":"2024-03-23T11:20:24.763652+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:18:02.329265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.745824+0000","last_change":"2024-03-23T11:20:24.779856+0000","last_active":"2024-03-23T11:20:25.745824+0000","last_peered":"2024-03-23T11:20:25.745824+0000","last_clean":"2024-03-23T11:20:25.745824+0000","last_became_active":"2024-03-23T11:20:24.779505+0000","last_became_peered":"2024-03-23T11:20:24.779505+0000","last_unstale":"2024-03-23T11:20:25.745824+0000","last_undegraded":"2024-03-23T11:20:25.745824+0000","last_fullsized":"2024-03-23T11:20:25.745824+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:18:45.214042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.748976+0000","last_change":"2024-03-23T11:20:24.270722+0000","last_active":"2024-03-23T11:20:24.748976+0000","last_peered":"2024-03-23T11:20:24.748976+0000","last_clean":"2024-03-23T11:20:24.748976+0000","last_became_active":"2024-03-23T11:20:24.270020+0000","last_became_peered":"2024-03-23T11:20:24.270020+0000","last_unstale":"2024-03-23T11:20:24.748976+0000","last_undegraded":"2024-03-23T11:20:24.748976+0000","last_fullsized":"2024-03-23T11:20:24.748976+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:12:15.653884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.210242+0000","last_change":"2024-03-23T11:20:24.775999+0000","last_active":"2024-03-23T11:20:26.210242+0000","last_peered":"2024-03-23T11:20:26.210242+0000","last_clean":"2024-03-23T11:20:26.210242+0000","last_became_active":"2024-03-23T11:20:24.775730+0000","last_became_peered":"2024-03-23T11:20:24.775730+0000","last_unstale":"2024-03-23T11:20:26.210242+0000","last_undegraded":"2024-03-23T11:20:26.210242+0000","last_fullsized":"2024-03-23T11:20:26.210242+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:08:28.525913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.776530+0000","last_change":"2024-03-23T11:20:24.271790+0000","last_active":"2024-03-23T11:20:24.776530+0000","last_peered":"2024-03-23T11:20:24.776530+0000","last_clean":"2024-03-23T11:20:24.776530+0000","last_became_active":"2024-03-23T11:20:24.271405+0000","last_became_peered":"2024-03-23T11:20:24.271405+0000","last_unstale":"2024-03-23T11:20:24.776530+0000","last_undegraded":"2024-03-23T11:20:24.776530+0000","last_fullsized":"2024-03-23T11:20:24.776530+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:29:57.348200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296288+0000","last_change":"2024-03-23T11:20:24.773335+0000","last_active":"2024-03-23T11:20:26.296288+0000","last_peered":"2024-03-23T11:20:26.296288+0000","last_clean":"2024-03-23T11:20:26.296288+0000","last_became_active":"2024-03-23T11:20:24.773061+0000","last_became_peered":"2024-03-23T11:20:24.773061+0000","last_unstale":"2024-03-23T11:20:26.296288+0000","last_undegraded":"2024-03-23T11:20:26.296288+0000","last_fullsized":"2024-03-23T11:20:26.296288+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:25:35.850555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296391+0000","last_change":"2024-03-23T11:20:24.261390+0000","last_active":"2024-03-23T11:20:26.296391+0000","last_peered":"2024-03-23T11:20:26.296391+0000","last_clean":"2024-03-23T11:20:26.296391+0000","last_became_active":"2024-03-23T11:20:24.259776+0000","last_became_peered":"2024-03-23T11:20:24.259776+0000","last_unstale":"2024-03-23T11:20:26.296391+0000","last_undegraded":"2024-03-23T11:20:26.296391+0000","last_fullsized":"2024-03-23T11:20:26.296391+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:54.976104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296019+0000","last_change":"2024-03-23T11:20:24.765429+0000","last_active":"2024-03-23T11:20:26.296019+0000","last_peered":"2024-03-23T11:20:26.296019+0000","last_clean":"2024-03-23T11:20:26.296019+0000","last_became_active":"2024-03-23T11:20:24.765192+0000","last_became_peered":"2024-03-23T11:20:24.765192+0000","last_unstale":"2024-03-23T11:20:26.296019+0000","last_undegraded":"2024-03-23T11:20:26.296019+0000","last_fullsized":"2024-03-23T11:20:26.296019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:42:55.664530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.748349+0000","last_change":"2024-03-23T11:20:24.261689+0000","last_active":"2024-03-23T11:20:24.748349+0000","last_peered":"2024-03-23T11:20:24.748349+0000","last_clean":"2024-03-23T11:20:24.748349+0000","last_became_active":"2024-03-23T11:20:24.261283+0000","last_became_peered":"2024-03-23T11:20:24.261283+0000","last_unstale":"2024-03-23T11:20:24.748349+0000","last_undegraded":"2024-03-23T11:20:24.748349+0000","last_fullsized":"2024-03-23T11:20:24.748349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:45:05.172226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.885392+0000","last_change":"2024-03-23T11:20:24.769221+0000","last_active":"2024-03-23T11:20:25.885392+0000","last_peered":"2024-03-23T11:20:25.885392+0000","last_clean":"2024-03-23T11:20:25.885392+0000","last_became_active":"2024-03-23T11:20:24.768903+0000","last_became_peered":"2024-03-23T11:20:24.768903+0000","last_unstale":"2024-03-23T11:20:25.885392+0000","last_undegraded":"2024-03-23T11:20:25.885392+0000","last_fullsized":"2024-03-23T11:20:25.885392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:45.264945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.296197+0000","last_change":"2024-03-23T11:20:24.273155+0000","last_active":"2024-03-23T11:20:26.296197+0000","last_peered":"2024-03-23T11:20:26.296197+0000","last_clean":"2024-03-23T11:20:26.296197+0000","last_became_active":"2024-03-23T11:20:24.272765+0000","last_became_peered":"2024-03-23T11:20:24.272765+0000","last_unstale":"2024-03-23T11:20:26.296197+0000","last_undegraded":"2024-03-23T11:20:26.296197+0000","last_fullsized":"2024-03-23T11:20:26.296197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.609703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:26.295786+0000","last_change":"2024-03-23T11:20:24.762140+0000","last_active":"2024-03-23T11:20:26.295786+0000","last_peered":"2024-03-23T11:20:26.295786+0000","last_clean":"2024-03-23T11:20:26.295786+0000","last_became_active":"2024-03-23T11:20:24.761475+0000","last_became_peered":"2024-03-23T11:20:24.761475+0000","last_unstale":"2024-03-23T11:20:26.295786+0000","last_undegraded":"2024-03-23T11:20:26.295786+0000","last_fullsized":"2024-03-23T11:20:26.295786+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:04:49.659198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.749666+0000","last_change":"2024-03-23T11:20:24.272555+0000","last_active":"2024-03-23T11:20:24.749666+0000","last_peered":"2024-03-23T11:20:24.749666+0000","last_clean":"2024-03-23T11:20:24.749666+0000","last_became_active":"2024-03-23T11:20:24.272270+0000","last_became_peered":"2024-03-23T11:20:24.272270+0000","last_unstale":"2024-03-23T11:20:24.749666+0000","last_undegraded":"2024-03-23T11:20:24.749666+0000","last_fullsized":"2024-03-23T11:20:24.749666+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:25:14.851311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.810984+0000","last_change":"2024-03-23T11:20:24.780005+0000","last_active":"2024-03-23T11:20:25.810984+0000","last_peered":"2024-03-23T11:20:25.810984+0000","last_clean":"2024-03-23T11:20:25.810984+0000","last_became_active":"2024-03-23T11:20:24.775265+0000","last_became_peered":"2024-03-23T11:20:24.775265+0000","last_unstale":"2024-03-23T11:20:25.810984+0000","last_undegraded":"2024-03-23T11:20:25.810984+0000","last_fullsized":"2024-03-23T11:20:25.810984+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:25:35.850555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.763799+0000","last_change":"2024-03-23T11:20:24.275279+0000","last_active":"2024-03-23T11:20:24.763799+0000","last_peered":"2024-03-23T11:20:24.763799+0000","last_clean":"2024-03-23T11:20:24.763799+0000","last_became_active":"2024-03-23T11:20:24.274581+0000","last_became_peered":"2024-03-23T11:20:24.274581+0000","last_unstale":"2024-03-23T11:20:24.763799+0000","last_undegraded":"2024-03-23T11:20:24.763799+0000","last_fullsized":"2024-03-23T11:20:24.763799+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:50.493955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.746341+0000","last_change":"2024-03-23T11:20:24.765049+0000","last_active":"2024-03-23T11:20:25.746341+0000","last_peered":"2024-03-23T11:20:25.746341+0000","last_clean":"2024-03-23T11:20:25.746341+0000","last_became_active":"2024-03-23T11:20:24.764610+0000","last_became_peered":"2024-03-23T11:20:24.764610+0000","last_unstale":"2024-03-23T11:20:25.746341+0000","last_undegraded":"2024-03-23T11:20:25.746341+0000","last_fullsized":"2024-03-23T11:20:25.746341+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:13.726077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.739921+0000","last_change":"2024-03-23T11:20:24.265565+0000","last_active":"2024-03-23T11:20:24.739921+0000","last_peered":"2024-03-23T11:20:24.739921+0000","last_clean":"2024-03-23T11:20:24.739921+0000","last_became_active":"2024-03-23T11:20:24.265287+0000","last_became_peered":"2024-03-23T11:20:24.265287+0000","last_unstale":"2024-03-23T11:20:24.739921+0000","last_undegraded":"2024-03-23T11:20:24.739921+0000","last_fullsized":"2024-03-23T11:20:24.739921+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:25:01.633915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-23T11:20:25.747410+0000","last_change":"2024-03-23T11:20:24.766725+0000","last_active":"2024-03-23T11:20:25.747410+0000","last_peered":"2024-03-23T11:20:25.747410+0000","last_clean":"2024-03-23T11:20:25.747410+0000","last_became_active":"2024-03-23T11:20:24.765503+0000","last_became_peered":"2024-03-23T11:20:24.765503+0000","last_unstale":"2024-03-23T11:20:25.747410+0000","last_undegraded":"2024-03-23T11:20:25.747410+0000","last_fullsized":"2024-03-23T11:20:25.747410+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:24.217719+0000","last_clean_scrub_stamp":"2024-03-23T11:20:24.217719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:40:24.562542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.767926+0000","last_change":"2024-03-23T11:20:24.269202+0000","last_active":"2024-03-23T11:20:24.767926+0000","last_peered":"2024-03-23T11:20:24.767926+0000","last_clean":"2024-03-23T11:20:24.767926+0000","last_became_active":"2024-03-23T11:20:24.268912+0000","last_became_peered":"2024-03-23T11:20:24.268912+0000","last_unstale":"2024-03-23T11:20:24.767926+0000","last_undegraded":"2024-03-23T11:20:24.767926+0000","last_fullsized":"2024-03-23T11:20:24.767926+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:20:23.202288+0000","last_clean_scrub_stamp":"2024-03-23T11:20:23.202288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:22:07.120173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-23T11:20:24.755776+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:20:24.755776+0000","last_peered":"2024-03-23T11:20:24.755776+0000","last_clean":"2024-03-23T11:20:24.755776+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:20:24.755776+0000","last_undegraded":"2024-03-23T11:20:24.755776+0000","last_fullsized":"2024-03-23T11:20:24.755776+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542167,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":1003520,"data_stored":750237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.751}]},{"osd":1,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":2.3820000000000001,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.4350000000000001,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.3700000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":1.081}]},{"osd":3,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":1.3140000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.345,"5min":1.345,"15min":1.345},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.29}]},{"osd":5,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.0859999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.248}]},{"osd":6,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.421,"15min":1.421},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.129}]}]},{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":159726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":0.84399999999999997}]},{"osd":1,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.0859999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.331}]},{"osd":2,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.3439999999999999,"5min":2.3439999999999999,"15min":2.3439999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.1520000000000001},"last":0.97999999999999998}]},{"osd":3,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":1.173}]},{"osd":4,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":1.113},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":1.5620000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":3.097,"5min":3.097,"15min":3.097},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.8109999999999999,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.26}]},{"osd":7,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":4.0880000000000001,"5min":4.0880000000000001,"15min":4.0880000000000001},"last":1.653},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":2.581,"5min":2.581,"15min":2.581},"last":1.5209999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":160451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.355,"5min":2.355,"15min":2.355},"last":1.508}]},{"osd":1,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.6559999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.345}]},{"osd":2,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":1.242},{"interface":"front","average":{"1min":1.248,"5min":1.248,"15min":1.248},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":1.552}]},{"osd":3,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.147}]},{"osd":4,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.8129999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.73}]},{"osd":7,"last update":"Sat Mar 23 11:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.611,"15min":1.611},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":2.516,"5min":2.516,"15min":2.516},"last":1.7729999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542167,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":159883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.651,"5min":1.651,"15min":1.651},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.6339999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.385}]},{"osd":2,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":1.419}]},{"osd":3,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.131},{"interface":"front","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.86899999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":2.1819999999999999,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.0720000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.319,"5min":2.319,"15min":2.319},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.2549999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.587},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":1.02,"5min":1.02,"15min":1.02},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.0269999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542167,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":421888,"data_stored":161365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.226},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":1.2929999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.163},{"interface":"front","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.391}]},{"osd":2,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":3.3109999999999999,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":1.274},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.7109999999999999,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.4850000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.3490000000000002,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.544},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"min":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"max":{"1min":4.8970000000000002,"5min":4.8970000000000002,"15min":4.8970000000000002},"last":1.6180000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.552,"15min":1.552},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":3.4910000000000001,"5min":3.4910000000000001,"15min":3.4910000000000001},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.504,"5min":1.504,"15min":1.504},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.125,"5min":3.125,"15min":3.125},"last":1.458}]},{"osd":6,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":4.5499999999999998,"5min":4.5499999999999998,"15min":4.5499999999999998},"last":1.8089999999999999},{"interface":"front","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":1.677}]},{"osd":7,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":2.952,"5min":2.952,"15min":2.952},"last":1.645},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":4.798,"5min":4.798,"15min":4.798},"last":1.877}]}]},{"osd":2,"up_from":14,"seq":60129542167,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":159849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":1.1970000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":1.1519999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.087},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.93600000000000005}]},{"osd":3,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":1.343}]},{"osd":4,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.595},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":1.647}]},{"osd":5,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.5830000000000002,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.272},{"interface":"front","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.464}]},{"osd":6,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":2.758,"5min":2.758,"15min":2.758},"last":1.401},{"interface":"front","average":{"1min":1.546,"5min":1.546,"15min":1.546},"min":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"max":{"1min":2.8580000000000001,"5min":2.8580000000000001,"15min":2.8580000000000001},"last":1.5289999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.484},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"min":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.8320000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":417792,"data_stored":160696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":0.877}]},{"osd":2,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.2090000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.4169999999999998,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.458}]},{"osd":4,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":1.8959999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.353,"15min":1.353},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":2.758,"5min":2.758,"15min":2.758},"last":1.2569999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.6589999999999998,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.5129999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.778},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"max":{"1min":2.9630000000000001,"5min":2.9630000000000001,"15min":2.9630000000000001},"last":1.3500000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"max":{"1min":3.2749999999999999,"5min":3.2749999999999999,"15min":3.2749999999999999},"last":1.595},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.7829999999999999,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.9970000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":749478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":1.034},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.125}]},{"osd":2,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.224},{"interface":"front","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":1.3919999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.103,"15min":1.103},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.1819999999999999,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.9670000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.1739999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.8719999999999999,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":2.335}]},{"osd":6,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":2.5499999999999998,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":2.4399999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.446,"15min":1.446},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.677}]},{"osd":7,"last update":"Sat Mar 23 11:19:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.617,"5min":2.617,"15min":2.617},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":3.081,"5min":3.081,"15min":3.081},"last":1.4810000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:20:30.259 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T11:20:30.259 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T11:20:30.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.526+0000 7f077ee00640 1 Processor -- start 2024-03-23T11:20:30.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.526+0000 7f077ee00640 1 -- start start 2024-03-23T11:20:30.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.528+0000 7f077ee00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0778143fe0 0x7f07780ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:30.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.528+0000 7f077ee00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07780cebd0 con 0x7f07780ce780 2024-03-23T11:20:30.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.528+0000 7f077ee00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077815eec0 con 0x7f0778143c00 2024-03-23T11:20:30.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.528+0000 7f077ee00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07780b1a50 con 0x7f0778143fe0 2024-03-23T11:20:30.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.528+0000 7f077e400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f07780ce780 0x7f077815e190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35228/0 (socket says 172.21.15.62:35228) 2024-03-23T11:20:30.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.528+0000 7f077e400640 1 -- 172.21.15.62:0/1659189077 learned_addr learned my addr 172.21.15.62:0/1659189077 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:30.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.528+0000 7f077d000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0778143fe0 0x7f07780ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:30.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.529+0000 7f0777e00640 1 -- 172.21.15.62:0/1659189077 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492827424 0 0) 0x7f0770001110 con 0x7f07780ce780 2024-03-23T11:20:30.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.529+0000 7f0777e00640 1 -- 172.21.15.62:0/1659189077 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0760003520 con 0x7f07780ce780 2024-03-23T11:20:30.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.529+0000 7f077d000640 1 -- 172.21.15.62:0/1659189077 >> v1:172.21.15.120:6789/0 conn(0x7f0778143c00 legacy=0x7f077815abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:30.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.529+0000 7f077d000640 1 -- 172.21.15.62:0/1659189077 >> v1:172.21.15.62:6789/0 conn(0x7f07780ce780 legacy=0x7f077815e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:30.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.529+0000 7f077d000640 1 -- 172.21.15.62:0/1659189077 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07780b1d30 con 0x7f0778143fe0 2024-03-23T11:20:30.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.530+0000 7f077d000640 1 --2- 172.21.15.62:0/1659189077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0778143fe0 0x7f07780ce240 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f07640097a0 tx=0x7f076402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7f024ad8e3cf1c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:30.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.530+0000 7f0777e00640 1 -- 172.21.15.62:0/1659189077 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764030610 con 0x7f0778143fe0 2024-03-23T11:20:30.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.530+0000 7f0777e00640 1 -- 172.21.15.62:0/1659189077 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0764030770 con 0x7f0778143fe0 2024-03-23T11:20:30.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.530+0000 7f0777e00640 1 -- 172.21.15.62:0/1659189077 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764030b00 con 0x7f0778143fe0 2024-03-23T11:20:30.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.531+0000 7f077ee00640 1 -- 172.21.15.62:0/1659189077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0778143fe0 msgr2=0x7f07780ce240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:30.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.531+0000 7f077ee00640 1 --2- 172.21.15.62:0/1659189077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0778143fe0 0x7f07780ce240 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f07640097a0 tx=0x7f076402b960 comp rx=0 tx=0).stop 2024-03-23T11:20:30.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.531+0000 7f077ee00640 1 -- 172.21.15.62:0/1659189077 shutdown_connections 2024-03-23T11:20:30.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.531+0000 7f077ee00640 1 --2- 172.21.15.62:0/1659189077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0778143fe0 0x7f07780ce240 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.531+0000 7f077ee00640 1 -- 172.21.15.62:0/1659189077 >> 172.21.15.62:0/1659189077 conn(0x7f0778066050 msgr2=0x7f0778069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:30.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.531+0000 7f077ee00640 1 -- 172.21.15.62:0/1659189077 shutdown_connections 2024-03-23T11:20:30.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.531+0000 7f077ee00640 1 -- 172.21.15.62:0/1659189077 wait complete. 2024-03-23T11:20:30.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.533+0000 7f077ee00640 1 Processor -- start 2024-03-23T11:20:30.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.533+0000 7f077ee00640 1 -- start start 2024-03-23T11:20:30.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.534+0000 7f077ee00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07780ce780 0x7f0778159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:30.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.534+0000 7f077ee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0778143c00 0x7f07780afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:30.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.534+0000 7f077ee00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0778143fe0 0x7f07780b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:30.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.534+0000 7f077ee00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07781f11c0 con 0x7f07780ce780 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.534+0000 7f077ee00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07781f14a0 con 0x7f0778143fe0 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.534+0000 7f077ee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07781f1780 con 0x7f0778143c00 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.534+0000 7f077da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07780ce780 0x7f0778159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.534+0000 7f077da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07780ce780 0x7f0778159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55842/0 (socket says 172.21.15.62:55842) 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.534+0000 7f077da00640 1 -- 172.21.15.62:0/1283882012 learned_addr learned my addr 172.21.15.62:0/1283882012 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f077d000640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0778143c00 0x7f07780afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f077da00640 1 -- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0778143c00 msgr2=0x7f07780afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f077e400640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0778143fe0 0x7f07780b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f077da00640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0778143c00 0x7f07780afdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f077da00640 1 -- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0778143fe0 msgr2=0x7f07780b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f077da00640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0778143fe0 0x7f07780b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f077da00640 1 -- 172.21.15.62:0/1283882012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0764009400 con 0x7f07780ce780 2024-03-23T11:20:30.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f077d000640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0778143c00 0x7f07780afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:30.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f077da00640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07780ce780 0x7f0778159b90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f076c00c730 tx=0x7f076c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:30.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f0775600640 1 -- 172.21.15.62:0/1283882012 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c003b70 con 0x7f07780ce780 2024-03-23T11:20:30.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f0775600640 1 -- 172.21.15.62:0/1283882012 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c003cf0 con 0x7f07780ce780 2024-03-23T11:20:30.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f0775600640 1 -- 172.21.15.62:0/1283882012 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c003ea0 con 0x7f07780ce780 2024-03-23T11:20:30.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.535+0000 7f077ee00640 1 -- 172.21.15.62:0/1283882012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07781f1a60 con 0x7f07780ce780 2024-03-23T11:20:30.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.536+0000 7f077ee00640 1 -- 172.21.15.62:0/1283882012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077815a0d0 con 0x7f07780ce780 2024-03-23T11:20:30.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.537+0000 7f077ee00640 1 -- 172.21.15.62:0/1283882012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07780ced70 con 0x7f07780ce780 2024-03-23T11:20:30.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.538+0000 7f0775600640 1 -- 172.21.15.62:0/1283882012 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f076c00aa90 con 0x7f07780ce780 2024-03-23T11:20:30.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.539+0000 7f0775600640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f075806a7a0 0x7f075806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:30.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.539+0000 7f0775600640 1 -- 172.21.15.62:0/1283882012 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f076c088c40 con 0x7f07780ce780 2024-03-23T11:20:30.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.539+0000 7f077d000640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f075806a7a0 0x7f075806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:30.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.540+0000 7f077d000640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f075806a7a0 0x7f075806cc60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0764036000 tx=0x7f076403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:30.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.543+0000 7f0775600640 1 -- 172.21.15.62:0/1283882012 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f076c053fc0 con 0x7f07780ce780 2024-03-23T11:20:30.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.817+0000 7f077ee00640 1 -- 172.21.15.62:0/1283882012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f07781f1300 con 0x7f07780ce780 2024-03-23T11:20:30.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.823+0000 7f0775600640 1 -- 172.21.15.62:0/1283882012 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 190+0+0 (secure 0 0 0) 0x7f07781f1300 con 0x7f07780ce780 2024-03-23T11:20:30.824 INFO:teuthology.orchestra.run.smithi062.stderr:updated caps for client.0 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.826+0000 7f077ee00640 1 -- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f075806a7a0 msgr2=0x7f075806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.826+0000 7f077ee00640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f075806a7a0 0x7f075806cc60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0764036000 tx=0x7f076403b040 comp rx=0 tx=0).stop 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.826+0000 7f077ee00640 1 -- 172.21.15.62:0/1283882012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07780ce780 msgr2=0x7f0778159b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.826+0000 7f077ee00640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07780ce780 0x7f0778159b90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f076c00c730 tx=0x7f076c00cbf0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.826+0000 7f077ee00640 1 -- 172.21.15.62:0/1283882012 shutdown_connections 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.826+0000 7f077ee00640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f075806a7a0 0x7f075806cc60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.826+0000 7f077ee00640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0778143fe0 0x7f07780b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.826+0000 7f077ee00640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0778143c00 0x7f07780afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.826+0000 7f077ee00640 1 --2- 172.21.15.62:0/1283882012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07780ce780 0x7f0778159b90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.826+0000 7f077ee00640 1 -- 172.21.15.62:0/1283882012 >> 172.21.15.62:0/1283882012 conn(0x7f0778066050 msgr2=0x7f07780add90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.827+0000 7f077ee00640 1 -- 172.21.15.62:0/1283882012 shutdown_connections 2024-03-23T11:20:30.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:30.827+0000 7f077ee00640 1 -- 172.21.15.62:0/1283882012 wait complete. 2024-03-23T11:20:30.846 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:20:31.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.056+0000 7f1956200640 1 Processor -- start 2024-03-23T11:20:31.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.056+0000 7f1956200640 1 -- start start 2024-03-23T11:20:31.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.057+0000 7f1956200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1950159490 0x7f195006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.058+0000 7f1956200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f19500ce5f0 0x7f19500b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.058+0000 7f1956200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19500ced90 con 0x7f195015da90 2024-03-23T11:20:31.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.058+0000 7f1956200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195006e3e0 con 0x7f19500ce5f0 2024-03-23T11:20:31.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.058+0000 7f1956200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195006c1c0 con 0x7f1950159490 2024-03-23T11:20:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.058+0000 7f1955800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f19500ce5f0 0x7f19500b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.058+0000 7f194fe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f195015da90 0x7f19500cdee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35236/0 (socket says 172.21.15.62:35236) 2024-03-23T11:20:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.058+0000 7f194fe00640 1 -- 172.21.15.62:0/1344631338 learned_addr learned my addr 172.21.15.62:0/1344631338 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.058+0000 7f1954e00640 1 --2- 172.21.15.62:0/1344631338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1950159490 0x7f195006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.059+0000 7f1955800640 1 -- 172.21.15.62:0/1344631338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1950159490 msgr2=0x7f195006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.059+0000 7f1955800640 1 --2- 172.21.15.62:0/1344631338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1950159490 0x7f195006cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.059+0000 7f1955800640 1 -- 172.21.15.62:0/1344631338 >> v1:172.21.15.62:6789/0 conn(0x7f195015da90 legacy=0x7f19500cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.059+0000 7f1955800640 1 -- 172.21.15.62:0/1344631338 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195006e100 con 0x7f19500ce5f0 2024-03-23T11:20:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.059+0000 7f194f400640 1 -- 172.21.15.62:0/1344631338 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679076544 0 0) 0x7f19500ced90 con 0x7f195015da90 2024-03-23T11:20:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.060+0000 7f1955800640 1 --2- 172.21.15.62:0/1344631338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19500ce5f0 0x7f19500b26b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1948009d90 tx=0x7f194802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eff0f521a931dcd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.060+0000 7f194f400640 1 -- 172.21.15.62:0/1344631338 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194802cbc0 con 0x7f19500ce5f0 2024-03-23T11:20:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.060+0000 7f194f400640 1 -- 172.21.15.62:0/1344631338 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194802cd20 con 0x7f19500ce5f0 2024-03-23T11:20:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.060+0000 7f194f400640 1 -- 172.21.15.62:0/1344631338 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1948034660 con 0x7f19500ce5f0 2024-03-23T11:20:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.061+0000 7f1956200640 1 -- 172.21.15.62:0/1344631338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19500ce5f0 msgr2=0x7f19500b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.061+0000 7f1956200640 1 --2- 172.21.15.62:0/1344631338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19500ce5f0 0x7f19500b26b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1948009d90 tx=0x7f194802be10 comp rx=0 tx=0).stop 2024-03-23T11:20:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.061+0000 7f1956200640 1 -- 172.21.15.62:0/1344631338 shutdown_connections 2024-03-23T11:20:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.061+0000 7f1956200640 1 --2- 172.21.15.62:0/1344631338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19500ce5f0 0x7f19500b26b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.061+0000 7f1956200640 1 --2- 172.21.15.62:0/1344631338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1950159490 0x7f195006cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.061+0000 7f1956200640 1 -- 172.21.15.62:0/1344631338 >> 172.21.15.62:0/1344631338 conn(0x7f1950066050 msgr2=0x7f1950069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.061+0000 7f1956200640 1 -- 172.21.15.62:0/1344631338 shutdown_connections 2024-03-23T11:20:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.061+0000 7f1956200640 1 -- 172.21.15.62:0/1344631338 wait complete. 2024-03-23T11:20:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.063+0000 7f1956200640 1 Processor -- start 2024-03-23T11:20:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.063+0000 7f1956200640 1 -- start start 2024-03-23T11:20:31.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.064+0000 7f1956200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19500ce5f0 0x7f19501f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.064+0000 7f1956200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1950159490 0x7f1950146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.064+0000 7f1956200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f195015da90 0x7f195015b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.064+0000 7f1956200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19500ad100 con 0x7f1950159490 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.064+0000 7f1956200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1950156a80 con 0x7f195015da90 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.064+0000 7f1956200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1950158630 con 0x7f19500ce5f0 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.064+0000 7f194fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1950159490 0x7f1950146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.064+0000 7f194fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1950159490 0x7f1950146d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55848/0 (socket says 172.21.15.62:55848) 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.064+0000 7f194fe00640 1 -- 172.21.15.62:0/3875250462 learned_addr learned my addr 172.21.15.62:0/3875250462 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.064+0000 7f1954e00640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19500ce5f0 0x7f19501f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.064+0000 7f1955800640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f195015da90 0x7f195015b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f194fe00640 1 -- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19500ce5f0 msgr2=0x7f19501f5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f194fe00640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19500ce5f0 0x7f19501f5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f194fe00640 1 -- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f195015da90 msgr2=0x7f195015b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f194fe00640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f195015da90 0x7f195015b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f194fe00640 1 -- 172.21.15.62:0/3875250462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1948009a70 con 0x7f1950159490 2024-03-23T11:20:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f1955800640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f195015da90 0x7f195015b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f194fe00640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1950159490 0x7f1950146d00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f193c00c500 tx=0x7f193c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f194cc00640 1 -- 172.21.15.62:0/3875250462 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c007560 con 0x7f1950159490 2024-03-23T11:20:31.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f194cc00640 1 -- 172.21.15.62:0/3875250462 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193c0047e0 con 0x7f1950159490 2024-03-23T11:20:31.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f194cc00640 1 -- 172.21.15.62:0/3875250462 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c004b90 con 0x7f1950159490 2024-03-23T11:20:31.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f1956200640 1 -- 172.21.15.62:0/3875250462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1950158910 con 0x7f1950159490 2024-03-23T11:20:31.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.065+0000 7f1956200640 1 -- 172.21.15.62:0/3875250462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1950147240 con 0x7f1950159490 2024-03-23T11:20:31.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.067+0000 7f1956200640 1 -- 172.21.15.62:0/3875250462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1950156f10 con 0x7f1950159490 2024-03-23T11:20:31.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.067+0000 7f194cc00640 1 -- 172.21.15.62:0/3875250462 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f193c004cf0 con 0x7f1950159490 2024-03-23T11:20:31.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.068+0000 7f194cc00640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f193006a7a0 0x7f193006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.068+0000 7f194cc00640 1 -- 172.21.15.62:0/3875250462 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f193c087f30 con 0x7f1950159490 2024-03-23T11:20:31.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.069+0000 7f1954e00640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f193006a7a0 0x7f193006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.070+0000 7f1954e00640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f193006a7a0 0x7f193006cc60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f19500b1f80 tx=0x7f1940008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:31.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.073+0000 7f194cc00640 1 -- 172.21.15.62:0/3875250462 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1950156f10 con 0x7f1950159490 2024-03-23T11:20:31.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.315+0000 7f1956200640 1 -- 172.21.15.62:0/3875250462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1950156f10 con 0x7f1950159490 2024-03-23T11:20:31.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.316+0000 7f194cc00640 1 -- 172.21.15.62:0/3875250462 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f1950156f10 con 0x7f1950159490 2024-03-23T11:20:31.318 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:31.318 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3421279564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3421279564},{"type":"v1","addr":"172.21.15.120:6835","nonce":3421279564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2052651236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2052651236},{"type":"v1","addr":"172.21.15.120:6837","nonce":2052651236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5193,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/1599029025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1599029025},{"type":"v1","addr":"172.21.15.62:6834","nonce":1599029025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:20:24.731603+0000","modified":"2024-03-23T11:20:28.249457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":5,"addr":"172.21.15.62:6835/3537214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":3537214366},{"type":"v1","addr":"172.21.15.62:6835","nonce":3537214366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T11:20:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-03-23T11:20:31.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.319+0000 7f1956200640 1 -- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f193006a7a0 msgr2=0x7f193006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:31.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.320+0000 7f1956200640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f193006a7a0 0x7f193006cc60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f19500b1f80 tx=0x7f1940008040 comp rx=0 tx=0).stop 2024-03-23T11:20:31.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.320+0000 7f1956200640 1 -- 172.21.15.62:0/3875250462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1950159490 msgr2=0x7f1950146d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:31.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.320+0000 7f1956200640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1950159490 0x7f1950146d00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f193c00c500 tx=0x7f193c00c9c0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.320+0000 7f1956200640 1 -- 172.21.15.62:0/3875250462 shutdown_connections 2024-03-23T11:20:31.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.320+0000 7f1956200640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f193006a7a0 0x7f193006cc60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.320+0000 7f1956200640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1950159490 0x7f1950146d00 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.320+0000 7f1956200640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f195015da90 0x7f195015b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.320+0000 7f1956200640 1 --2- 172.21.15.62:0/3875250462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19500ce5f0 0x7f19501f5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.320+0000 7f1956200640 1 -- 172.21.15.62:0/3875250462 >> 172.21.15.62:0/3875250462 conn(0x7f1950066050 msgr2=0x7f19500ae000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:31.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.320+0000 7f1956200640 1 -- 172.21.15.62:0/3875250462 shutdown_connections 2024-03-23T11:20:31.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.320+0000 7f1956200640 1 -- 172.21.15.62:0/3875250462 wait complete. 2024-03-23T11:20:31.741 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:20:24.731603+0000', 'modified': '2024-03-23T11:20:28.249457+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5196}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5196': {'gid': 5196, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.62:6835/3537214366', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6833', 'nonce': 3537214366}, {'type': 'v1', 'addr': '172.21.15.62:6835', 'nonce': 3537214366}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:20:31.741 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:20:31.741 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T11:20:31.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.972+0000 7ff588000640 1 Processor -- start 2024-03-23T11:20:31.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.973+0000 7ff588000640 1 -- start start 2024-03-23T11:20:31.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.974+0000 7ff588000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff5801571e0 0x7ff5800ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.974+0000 7ff588000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58006cf40 con 0x7ff5800ade70 2024-03-23T11:20:31.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.974+0000 7ff588000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5801575c0 con 0x7ff5801613a0 2024-03-23T11:20:31.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.974+0000 7ff588000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58006c3a0 con 0x7ff5801571e0 2024-03-23T11:20:31.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.975+0000 7ff585600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff5801571e0 0x7ff5800ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.975+0000 7ff584c00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff5800ade70 0x7ff58006d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35248/0 (socket says 172.21.15.62:35248) 2024-03-23T11:20:31.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.975+0000 7ff584c00640 1 -- 172.21.15.62:0/1393903915 learned_addr learned my addr 172.21.15.62:0/1393903915 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:31.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.975+0000 7ff57fe00640 1 -- 172.21.15.62:0/1393903915 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407263928 0 0) 0x7ff58006cf40 con 0x7ff5800ade70 2024-03-23T11:20:31.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.975+0000 7ff585600640 1 -- 172.21.15.62:0/1393903915 >> v1:172.21.15.120:6789/0 conn(0x7ff5801613a0 legacy=0x7ff58015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:31.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.975+0000 7ff585600640 1 -- 172.21.15.62:0/1393903915 >> v1:172.21.15.62:6789/0 conn(0x7ff5800ade70 legacy=0x7ff58006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:31.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.975+0000 7ff585600640 1 -- 172.21.15.62:0/1393903915 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58006c610 con 0x7ff5801571e0 2024-03-23T11:20:31.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.977+0000 7ff585600640 1 --2- 172.21.15.62:0/1393903915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5801571e0 0x7ff5800ad930 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff5740097a0 tx=0x7ff57402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0081159a34d69 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:31.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.977+0000 7ff57fe00640 1 -- 172.21.15.62:0/1393903915 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57402e070 con 0x7ff5801571e0 2024-03-23T11:20:31.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.977+0000 7ff57fe00640 1 -- 172.21.15.62:0/1393903915 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5740315a0 con 0x7ff5801571e0 2024-03-23T11:20:31.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.977+0000 7ff57fe00640 1 -- 172.21.15.62:0/1393903915 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574031930 con 0x7ff5801571e0 2024-03-23T11:20:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.978+0000 7ff588000640 1 -- 172.21.15.62:0/1393903915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5801571e0 msgr2=0x7ff5800ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.978+0000 7ff588000640 1 --2- 172.21.15.62:0/1393903915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5801571e0 0x7ff5800ad930 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff5740097a0 tx=0x7ff57402b840 comp rx=0 tx=0).stop 2024-03-23T11:20:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.978+0000 7ff588000640 1 -- 172.21.15.62:0/1393903915 shutdown_connections 2024-03-23T11:20:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.978+0000 7ff588000640 1 --2- 172.21.15.62:0/1393903915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5801571e0 0x7ff5800ad930 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.978+0000 7ff588000640 1 -- 172.21.15.62:0/1393903915 >> 172.21.15.62:0/1393903915 conn(0x7ff5800b02e0 msgr2=0x7ff5800b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.978+0000 7ff588000640 1 -- 172.21.15.62:0/1393903915 shutdown_connections 2024-03-23T11:20:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.978+0000 7ff588000640 1 -- 172.21.15.62:0/1393903915 wait complete. 2024-03-23T11:20:31.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.980+0000 7ff588000640 1 Processor -- start 2024-03-23T11:20:31.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.980+0000 7ff588000640 1 -- start start 2024-03-23T11:20:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.981+0000 7ff588000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5800ade70 0x7ff5800cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.981+0000 7ff588000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5801571e0 0x7ff5801f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.981+0000 7ff588000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5801613a0 0x7ff5801f63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.981+0000 7ff588000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5801f6930 con 0x7ff5801571e0 2024-03-23T11:20:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.981+0000 7ff588000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58014d7c0 con 0x7ff5800ade70 2024-03-23T11:20:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.981+0000 7ff588000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800cbbe0 con 0x7ff5801613a0 2024-03-23T11:20:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.981+0000 7ff584c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5801571e0 0x7ff5801f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.981+0000 7ff584c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5801571e0 0x7ff5801f2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55866/0 (socket says 172.21.15.62:55866) 2024-03-23T11:20:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.981+0000 7ff584c00640 1 -- 172.21.15.62:0/529225852 learned_addr learned my addr 172.21.15.62:0/529225852 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.982+0000 7ff584c00640 1 -- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5801613a0 msgr2=0x7ff5801f63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.982+0000 7ff584c00640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5801613a0 0x7ff5801f63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.982+0000 7ff584c00640 1 -- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5800ade70 msgr2=0x7ff5800cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.982+0000 7ff584c00640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5800ade70 0x7ff5800cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.982+0000 7ff584c00640 1 -- 172.21.15.62:0/529225852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff574009400 con 0x7ff5801571e0 2024-03-23T11:20:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.982+0000 7ff584c00640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5801571e0 0x7ff5801f2400 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00c3e0 tx=0x7ff56c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.982+0000 7ff57d600640 1 -- 172.21.15.62:0/529225852 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c0074e0 con 0x7ff5801571e0 2024-03-23T11:20:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.982+0000 7ff57d600640 1 -- 172.21.15.62:0/529225852 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c007660 con 0x7ff5801571e0 2024-03-23T11:20:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.982+0000 7ff57d600640 1 -- 172.21.15.62:0/529225852 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c004a20 con 0x7ff5801571e0 2024-03-23T11:20:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.982+0000 7ff588000640 1 -- 172.21.15.62:0/529225852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5800cbec0 con 0x7ff5801571e0 2024-03-23T11:20:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.982+0000 7ff588000640 1 -- 172.21.15.62:0/529225852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5800cc480 con 0x7ff5801571e0 2024-03-23T11:20:31.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.985+0000 7ff57d600640 1 -- 172.21.15.62:0/529225852 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff56c004b80 con 0x7ff5801571e0 2024-03-23T11:20:31.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.985+0000 7ff57d600640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff56006a7a0 0x7ff56006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.986+0000 7ff57d600640 1 -- 172.21.15.62:0/529225852 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff56c088b70 con 0x7ff5801571e0 2024-03-23T11:20:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.986+0000 7ff585600640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff56006a7a0 0x7ff56006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.986+0000 7ff58ad77740 1 Processor -- start 2024-03-23T11:20:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.987+0000 7ff58ad77740 1 -- start start 2024-03-23T11:20:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.987+0000 7ff585600640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff56006a7a0 0x7ff56006cc60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff574036000 tx=0x7ff574030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x5628741c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741c7410 0x5628741c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c38b0 0x5628741f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff58ad77740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562874134110 con 0x5628741c7410 2024-03-23T11:20:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff58ad77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628741318a0 con 0x562874211390 2024-03-23T11:20:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff58ad77740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56287412d190 con 0x5628741c38b0 2024-03-23T11:20:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff584c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741c7410 0x5628741c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff584c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741c7410 0x5628741c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55878/0 (socket says 172.21.15.62:55878) 2024-03-23T11:20:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff584c00640 1 -- 172.21.15.62:0/3909017377 learned_addr learned my addr 172.21.15.62:0/3909017377 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff586000640 1 --2- 172.21.15.62:0/3909017377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c38b0 0x5628741f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff585600640 1 --2- 172.21.15.62:0/3909017377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x5628741c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff584c00640 1 -- 172.21.15.62:0/3909017377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c38b0 msgr2=0x5628741f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff584c00640 1 --2- 172.21.15.62:0/3909017377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c38b0 0x5628741f36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff584c00640 1 -- 172.21.15.62:0/3909017377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 msgr2=0x5628741c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff584c00640 1 --2- 172.21.15.62:0/3909017377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x5628741c89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff584c00640 1 -- 172.21.15.62:0/3909017377 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628740f8230 con 0x5628741c7410 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff586000640 1 --2- 172.21.15.62:0/3909017377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c38b0 0x5628741f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.988+0000 7ff585600640 1 --2- 172.21.15.62:0/3909017377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x5628741c89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.989+0000 7ff584c00640 1 --2- 172.21.15.62:0/3909017377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741c7410 0x5628741c8f00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff56c054150 tx=0x7ff56c058350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1a490866fb4625 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.989+0000 7ff572000640 1 -- 172.21.15.62:0/3909017377 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c060040 con 0x5628741c7410 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.989+0000 7ff572000640 1 -- 172.21.15.62:0/3909017377 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c001080 con 0x5628741c7410 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.989+0000 7ff572000640 1 -- 172.21.15.62:0/3909017377 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c054360 con 0x5628741c7410 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.990+0000 7ff58ad77740 1 -- 172.21.15.62:0/3909017377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741c7410 msgr2=0x5628741c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.990+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3909017377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741c7410 0x5628741c8f00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff56c054150 tx=0x7ff56c058350 comp rx=0 tx=0).stop 2024-03-23T11:20:31.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.990+0000 7ff58ad77740 1 -- 172.21.15.62:0/3909017377 shutdown_connections 2024-03-23T11:20:31.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.990+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3909017377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741c7410 0x5628741c8f00 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.990+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3909017377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c38b0 0x5628741f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.990+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3909017377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x5628741c89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.990+0000 7ff58ad77740 1 -- 172.21.15.62:0/3909017377 >> 172.21.15.62:0/3909017377 conn(0x5628741c5b20 msgr2=0x56287420e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:31.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.990+0000 7ff58ad77740 1 -- 172.21.15.62:0/3909017377 shutdown_connections 2024-03-23T11:20:31.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.990+0000 7ff58ad77740 1 -- 172.21.15.62:0/3909017377 wait complete. 2024-03-23T11:20:31.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.991+0000 7ff58ad77740 1 -- start start 2024-03-23T11:20:31.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.991+0000 7ff58ad77740 1 Processor -- start 2024-03-23T11:20:31.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.992+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 0x5628741f1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.992+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x56287420fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.992+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741d7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.992+0000 7ff58ad77740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628740ce8c0 con 0x5628741d7aa0 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.992+0000 7ff58ad77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628740ce560 con 0x562874211390 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.992+0000 7ff58ad77740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562873e94f20 con 0x5628741c7410 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.992+0000 7ff58ad77740 10 client.0 _ll_register_callbacks cb 0x562874126800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.992+0000 7ff586000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741d7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.992+0000 7ff586000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741d7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55884/0 (socket says 172.21.15.62:55884) 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.992+0000 7ff586000640 1 -- 172.21.15.62:0/3885624807 learned_addr learned my addr 172.21.15.62:0/3885624807 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.993+0000 7ff585600640 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 0x5628741f1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.993+0000 7ff584c00640 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x56287420fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.993+0000 7ff586000640 1 -- 172.21.15.62:0/3885624807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 msgr2=0x5628741f1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.993+0000 7ff586000640 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 0x5628741f1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.993+0000 7ff586000640 1 -- 172.21.15.62:0/3885624807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 msgr2=0x56287420fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.993+0000 7ff586000640 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x56287420fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.993+0000 7ff586000640 1 -- 172.21.15.62:0/3885624807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5800cc480 con 0x5628741d7aa0 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.993+0000 7ff586000640 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741d7f00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff57800d4b0 tx=0x7ff57800d970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3712cc820a165f5e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.993+0000 7ff566a00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.993+0000 7ff566a00640 1 -- 172.21.15.62:0/3885624807 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578012800 con 0x5628741d7aa0 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.993+0000 7ff566a00640 1 -- 172.21.15.62:0/3885624807 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff578012960 con 0x5628741d7aa0 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff58ad77740 1 -- 172.21.15.62:0/3885624807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562874157280 con 0x5628741d7aa0 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff566a00640 1 -- 172.21.15.62:0/3885624807 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff578012b20 con 0x5628741d7aa0 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff58ad77740 10 client.5379 fetch_fsmap learned FSMap version 16 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff58ad77740 1 -- 172.21.15.62:0/3885624807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x56287412b810 con 0x5628741d7aa0 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff566a00640 1 -- 172.21.15.62:0/3885624807 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562874157280 con 0x5628741d7aa0 2024-03-23T11:20:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff566a00640 1 -- 172.21.15.62:0/3885624807 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff578012c80 con 0x5628741d7aa0 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff58ad77740 10 client.5379 fetch_fsmap finished waiting for FSMap version 16 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff58ad77740 10 client.5379 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff58ad77740 20 client.5379 populate_metadata read hostname 'smithi062' 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.62:6833/3537214366,v1:172.21.15.62:6835/3537214366] conn(0x562874213650 0x562874221b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff58ad77740 4 client.5379 mds_command: new command op to 5196 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.994+0000 7ff58ad77740 1 -- 172.21.15.62:0/3885624807 --> [v2:172.21.15.62:6833/3537214366,v1:172.21.15.62:6835/3537214366] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56287411f9d0 con 0x562874213650 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.995+0000 7ff585600640 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.62:6833/3537214366,v1:172.21.15.62:6835/3537214366] conn(0x562874213650 0x562874221b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.995+0000 7ff585600640 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.62:6833/3537214366,v1:172.21.15.62:6835/3537214366] conn(0x562874213650 0x562874221b50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a8a8079ebc9eaeb server_cookie=82ee0340a47943fd in_seq=0 out_seq=0 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.995+0000 7ff566a00640 10 client.5379 ms_handle_connect on v2:172.21.15.62:6833/3537214366 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff566a00640 1 -- 172.21.15.62:0/3885624807 <== mds.0 v2:172.21.15.62:6833/3537214366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56287411f9d0 con 0x562874213650 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff566a00640 10 client.5379 handle_command_reply: tid=0 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff58ad77740 1 client.5379 shutdown 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff58ad77740 1 -- 172.21.15.62:0/3885624807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 msgr2=0x5628741d7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:32.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741d7f00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff57800d4b0 tx=0x7ff57800d970 comp rx=0 tx=0).stop 2024-03-23T11:20:32.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff58ad77740 1 -- 172.21.15.62:0/3885624807 shutdown_connections 2024-03-23T11:20:32.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.62:6833/3537214366,v1:172.21.15.62:6835/3537214366] conn(0x562874213650 0x562874221b50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741d7f00 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x56287420fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff566a00640 0 client.5379 ms_handle_reset on v2:172.21.15.62:6833/3537214366 2024-03-23T11:20:32.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3885624807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 0x5628741f1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.996+0000 7ff58ad77740 1 -- 172.21.15.62:0/3885624807 >> 172.21.15.62:0/3885624807 conn(0x5628741c38b0 msgr2=0x5628741c7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:32.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.997+0000 7ff58ad77740 1 -- 172.21.15.62:0/3885624807 shutdown_connections 2024-03-23T11:20:32.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.997+0000 7ff58ad77740 1 -- 172.21.15.62:0/3885624807 wait complete. 2024-03-23T11:20:32.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:31.997+0000 7ff58ad77740 20 client.5379 trim_cache size 0 max 16384 2024-03-23T11:20:32.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.014+0000 7ff58ad77740 1 Processor -- start 2024-03-23T11:20:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.014+0000 7ff58ad77740 1 -- start start 2024-03-23T11:20:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.014+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 0x5628741c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628741d7aa0 0x5628741ebe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562874211390 0x5628741ec3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff58ad77740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628741c7980 con 0x562874211390 2024-03-23T11:20:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff58ad77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628741ca010 con 0x5628741d7aa0 2024-03-23T11:20:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff58ad77740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5628741f2000 con 0x5628741c7410 2024-03-23T11:20:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff586000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562874211390 0x5628741ec3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff586000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562874211390 0x5628741ec3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55888/0 (socket says 172.21.15.62:55888) 2024-03-23T11:20:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff586000640 1 -- 172.21.15.62:0/332827619 learned_addr learned my addr 172.21.15.62:0/332827619 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:32.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff586000640 1 -- 172.21.15.62:0/332827619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 msgr2=0x5628741c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:32.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff586000640 1 --2- 172.21.15.62:0/332827619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 0x5628741c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff586000640 1 -- 172.21.15.62:0/332827619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628741d7aa0 msgr2=0x5628741ebe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:32.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff584c00640 1 --2- 172.21.15.62:0/332827619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628741d7aa0 0x5628741ebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:32.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff586000640 1 --2- 172.21.15.62:0/332827619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628741d7aa0 0x5628741ebe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.015+0000 7ff586000640 1 -- 172.21.15.62:0/332827619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58014e930 con 0x562874211390 2024-03-23T11:20:32.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.016+0000 7ff586000640 1 --2- 172.21.15.62:0/332827619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562874211390 0x5628741ec3a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff57800d9b0 tx=0x7ff578012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83f9b734ef1d412 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:32.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.016+0000 7ff572a00640 1 -- 172.21.15.62:0/332827619 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578003d90 con 0x562874211390 2024-03-23T11:20:32.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.016+0000 7ff572a00640 1 -- 172.21.15.62:0/332827619 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff578028070 con 0x562874211390 2024-03-23T11:20:32.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.016+0000 7ff572a00640 1 -- 172.21.15.62:0/332827619 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57801aeb0 con 0x562874211390 2024-03-23T11:20:32.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.017+0000 7ff58ad77740 1 -- 172.21.15.62:0/332827619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562874211390 msgr2=0x5628741ec3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:32.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.017+0000 7ff58ad77740 1 --2- 172.21.15.62:0/332827619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562874211390 0x5628741ec3a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff57800d9b0 tx=0x7ff578012000 comp rx=0 tx=0).stop 2024-03-23T11:20:32.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.017+0000 7ff58ad77740 1 -- 172.21.15.62:0/332827619 shutdown_connections 2024-03-23T11:20:32.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.017+0000 7ff58ad77740 1 --2- 172.21.15.62:0/332827619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562874211390 0x5628741ec3a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.017+0000 7ff58ad77740 1 --2- 172.21.15.62:0/332827619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628741d7aa0 0x5628741ebe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.017+0000 7ff58ad77740 1 --2- 172.21.15.62:0/332827619 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 0x5628741c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.017+0000 7ff58ad77740 1 -- 172.21.15.62:0/332827619 >> 172.21.15.62:0/332827619 conn(0x5628741c38b0 msgr2=0x5628741d63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:32.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.017+0000 7ff58ad77740 1 -- 172.21.15.62:0/332827619 shutdown_connections 2024-03-23T11:20:32.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.017+0000 7ff58ad77740 1 -- 172.21.15.62:0/332827619 wait complete. 2024-03-23T11:20:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.018+0000 7ff58ad77740 1 -- start start 2024-03-23T11:20:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.018+0000 7ff58ad77740 1 Processor -- start 2024-03-23T11:20:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 0x562874274bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff58ad77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x562874204030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff58ad77740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562874227ee0 con 0x5628741d7aa0 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff58ad77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562874204a40 con 0x562874211390 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff58ad77740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562874227c90 con 0x5628741c7410 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff58ad77740 10 client.0 _ll_register_callbacks cb 0x7ff578003cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff584c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff584c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741c8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55898/0 (socket says 172.21.15.62:55898) 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff584c00640 1 -- 172.21.15.62:0/3198575672 learned_addr learned my addr 172.21.15.62:0/3198575672 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff584c00640 1 -- 172.21.15.62:0/3198575672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 msgr2=0x562874274bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff584c00640 1 --2- 172.21.15.62:0/3198575672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 0x562874274bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff584c00640 1 -- 172.21.15.62:0/3198575672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 msgr2=0x562874204030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff584c00640 1 --2- 172.21.15.62:0/3198575672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x562874204030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.019+0000 7ff584c00640 1 -- 172.21.15.62:0/3198575672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56287412b810 con 0x5628741d7aa0 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.020+0000 7ff584c00640 1 --2- 172.21.15.62:0/3198575672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741c8a50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff56c057be0 tx=0x7ff56c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a8a4e7652288ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.020+0000 7ff566a00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.020+0000 7ff566a00640 1 -- 172.21.15.62:0/3198575672 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c074070 con 0x5628741d7aa0 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.020+0000 7ff566a00640 1 -- 172.21.15.62:0/3198575672 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff56c060040 con 0x5628741d7aa0 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.020+0000 7ff58ad77740 1 -- 172.21.15.62:0/3198575672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff56c001080 con 0x5628741d7aa0 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.020+0000 7ff566a00640 1 -- 172.21.15.62:0/3198575672 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff56c057a20 con 0x5628741d7aa0 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.020+0000 7ff58ad77740 10 client.5385 fetch_fsmap learned FSMap version 16 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.020+0000 7ff58ad77740 1 -- 172.21.15.62:0/3198575672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5628740cdad0 con 0x5628741d7aa0 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.020+0000 7ff566a00640 1 -- 172.21.15.62:0/3198575672 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c001080 con 0x5628741d7aa0 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.020+0000 7ff566a00640 1 -- 172.21.15.62:0/3198575672 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff56c01b060 con 0x5628741d7aa0 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.021+0000 7ff58ad77740 10 client.5385 fetch_fsmap finished waiting for FSMap version 16 2024-03-23T11:20:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.021+0000 7ff58ad77740 10 client.5385 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:20:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.021+0000 7ff58ad77740 20 client.5385 populate_metadata read hostname 'smithi062' 2024-03-23T11:20:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.021+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3198575672 >> [v2:172.21.15.62:6833/3537214366,v1:172.21.15.62:6835/3537214366] conn(0x5628741c3d90 0x5628741c4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.021+0000 7ff58ad77740 4 client.5385 mds_command: new command op to 5196 tid=0[{"prefix": "status"}] 2024-03-23T11:20:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.021+0000 7ff58ad77740 1 -- 172.21.15.62:0/3198575672 --> [v2:172.21.15.62:6833/3537214366,v1:172.21.15.62:6835/3537214366] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562873e213c0 con 0x5628741c3d90 2024-03-23T11:20:32.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.021+0000 7ff585600640 1 --2- 172.21.15.62:0/3198575672 >> [v2:172.21.15.62:6833/3537214366,v1:172.21.15.62:6835/3537214366] conn(0x5628741c3d90 0x5628741c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:32.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.021+0000 7ff585600640 1 --2- 172.21.15.62:0/3198575672 >> [v2:172.21.15.62:6833/3537214366,v1:172.21.15.62:6835/3537214366] conn(0x5628741c3d90 0x5628741c4170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=902069d43f88b44f server_cookie=56f891145618cd3c in_seq=0 out_seq=0 2024-03-23T11:20:32.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.021+0000 7ff566a00640 10 client.5385 ms_handle_connect on v2:172.21.15.62:6833/3537214366 2024-03-23T11:20:32.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff566a00640 1 -- 172.21.15.62:0/3198575672 <== mds.0 v2:172.21.15.62:6833/3537214366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x562873e213c0 con 0x5628741c3d90 2024-03-23T11:20:32.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff566a00640 10 client.5385 handle_command_reply: tid=0 2024-03-23T11:20:32.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 1 client.5385 shutdown 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 1 -- 172.21.15.62:0/3198575672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 msgr2=0x5628741c8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3198575672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741c8a50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff56c057be0 tx=0x7ff56c000ed0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 1 -- 172.21.15.62:0/3198575672 shutdown_connections 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3198575672 >> [v2:172.21.15.62:6833/3537214366,v1:172.21.15.62:6835/3537214366] conn(0x5628741c3d90 0x5628741c4170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3198575672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628741d7aa0 0x5628741c8a50 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3198575672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562874211390 0x562874204030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 1 --2- 172.21.15.62:0/3198575672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5628741c7410 0x562874274bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff566a00640 0 client.5385 ms_handle_reset on v2:172.21.15.62:6833/3537214366 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 1 -- 172.21.15.62:0/3198575672 >> 172.21.15.62:0/3198575672 conn(0x5628741c38b0 msgr2=0x5628742129a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 1 -- 172.21.15.62:0/3198575672 shutdown_connections 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 1 -- 172.21.15.62:0/3198575672 wait complete. 2024-03-23T11:20:32.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.022+0000 7ff58ad77740 20 client.5385 trim_cache size 0 max 16384 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 5196, 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 7.2873835419999997, 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 16, 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 36, 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 36, 2024-03-23T11:20:32.027 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 22.207608049000001 2024-03-23T11:20:32.028 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:20:32.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 -- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff56006a7a0 msgr2=0x7ff56006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:32.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff56006a7a0 0x7ff56006cc60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff574036000 tx=0x7ff574030000 comp rx=0 tx=0).stop 2024-03-23T11:20:32.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 -- 172.21.15.62:0/529225852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5801571e0 msgr2=0x7ff5801f2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5801571e0 0x7ff5801f2400 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00c3e0 tx=0x7ff56c00c8a0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 -- 172.21.15.62:0/529225852 shutdown_connections 2024-03-23T11:20:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff56006a7a0 0x7ff56006cc60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5801571e0 0x7ff5801f2400 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5801613a0 0x7ff5801f63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 --2- 172.21.15.62:0/529225852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5800ade70 0x7ff5800cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 -- 172.21.15.62:0/529225852 >> 172.21.15.62:0/529225852 conn(0x7ff5800b02e0 msgr2=0x7ff5800b19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 -- 172.21.15.62:0/529225852 shutdown_connections 2024-03-23T11:20:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.023+0000 7ff588000640 1 -- 172.21.15.62:0/529225852 wait complete. 2024-03-23T11:20:32.044 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:20:32.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.259+0000 7fd59b200640 1 Processor -- start 2024-03-23T11:20:32.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.259+0000 7fd59b200640 1 -- start start 2024-03-23T11:20:32.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.261+0000 7fd59b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd59c159490 0x7fd59c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.261+0000 7fd59b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59c15da90 0x7fd59c0cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.261+0000 7fd59b200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59c0ce8b0 con 0x7fd59c0ce420 2024-03-23T11:20:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.261+0000 7fd59b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59c0ced90 con 0x7fd59c159490 2024-03-23T11:20:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.261+0000 7fd59b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c06e3e0 con 0x7fd59c15da90 2024-03-23T11:20:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.262+0000 7fd59a800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd59c0ce420 0x7fd59c0b26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35264/0 (socket says 172.21.15.62:35264) 2024-03-23T11:20:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.262+0000 7fd59a800640 1 -- 172.21.15.62:0/3009552501 learned_addr learned my addr 172.21.15.62:0/3009552501 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.262+0000 7fd599e00640 1 --2- 172.21.15.62:0/3009552501 >> v2:172.21.15.120:3300/0 conn(0x7fd59c159490 0x7fd59c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:32.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.262+0000 7fd593e00640 1 --2- 172.21.15.62:0/3009552501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59c15da90 0x7fd59c0cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:32.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.262+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111586993 0 0) 0x7fd59c0ce8b0 con 0x7fd59c0ce420 2024-03-23T11:20:32.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.262+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd580003520 con 0x7fd59c0ce420 2024-03-23T11:20:32.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2752213556 0 0) 0x7fd580003520 con 0x7fd59c0ce420 2024-03-23T11:20:32.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59c15da90 msgr2=0x7fd59c0cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:32.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 --2- 172.21.15.62:0/3009552501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59c15da90 0x7fd59c0cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 >> v2:172.21.15.120:3300/0 conn(0x7fd59c159490 msgr2=0x7fd59c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:32.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 --2- 172.21.15.62:0/3009552501 >> v2:172.21.15.120:3300/0 conn(0x7fd59c159490 0x7fd59c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c06c1c0 con 0x7fd59c0ce420 2024-03-23T11:20:32.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fd5940019f0 con 0x7fd59c0ce420 2024-03-23T11:20:32.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 >> v1:172.21.15.62:6789/0 conn(0x7fd59c0ce420 legacy=0x7fd59c0b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 --2- 172.21.15.62:0/3009552501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd580003970 0x7fd580003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 --2- 172.21.15.62:0/3009552501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5800047c0 0x7fd580004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 --2- 172.21.15.62:0/3009552501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59c15da90 0x7fd580005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0ced90 con 0x7fd580003970 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c06e3e0 con 0x7fd5800047c0 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58000b980 con 0x7fd59c15da90 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599e00640 1 --2- 172.21.15.62:0/3009552501 >> v2:172.21.15.120:3300/0 conn(0x7fd59c159490 0x7fd59c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.263+0000 7fd599e00640 1 --2- 172.21.15.62:0/3009552501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd580003970 0x7fd580003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.264+0000 7fd593e00640 1 --2- 172.21.15.62:0/3009552501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5800047c0 0x7fd580004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.264+0000 7fd599e00640 1 -- 172.21.15.62:0/3009552501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59c15da90 msgr2=0x7fd580005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.264+0000 7fd599e00640 1 --2- 172.21.15.62:0/3009552501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd59c15da90 0x7fd580005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.264+0000 7fd599e00640 1 -- 172.21.15.62:0/3009552501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5800047c0 msgr2=0x7fd580004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.264+0000 7fd599e00640 1 --2- 172.21.15.62:0/3009552501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5800047c0 0x7fd580004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.264+0000 7fd599e00640 1 -- 172.21.15.62:0/3009552501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd580003520 con 0x7fd580003970 2024-03-23T11:20:32.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.265+0000 7fd599e00640 1 --2- 172.21.15.62:0/3009552501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd580003970 0x7fd580003d90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009ca0 tx=0x7fd5800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1f25e9af6b8b78 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:32.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.265+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c004020 con 0x7fd580003970 2024-03-23T11:20:32.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.265+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c004180 con 0x7fd580003970 2024-03-23T11:20:32.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.265+0000 7fd599400640 1 -- 172.21.15.62:0/3009552501 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c02c950 con 0x7fd580003970 2024-03-23T11:20:32.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.266+0000 7fd59b200640 1 -- 172.21.15.62:0/3009552501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd580003970 msgr2=0x7fd580003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:32.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.266+0000 7fd59b200640 1 --2- 172.21.15.62:0/3009552501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd580003970 0x7fd580003d90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009ca0 tx=0x7fd5800057f0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.266+0000 7fd59a800640 1 -- 172.21.15.62:0/3009552501 reap_dead start 2024-03-23T11:20:32.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.266+0000 7fd59b200640 1 -- 172.21.15.62:0/3009552501 shutdown_connections 2024-03-23T11:20:32.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.266+0000 7fd59b200640 1 -- 172.21.15.62:0/3009552501 >> 172.21.15.62:0/3009552501 conn(0x7fd59c066050 msgr2=0x7fd59c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:32.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.266+0000 7fd59b200640 1 -- 172.21.15.62:0/3009552501 shutdown_connections 2024-03-23T11:20:32.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.266+0000 7fd59b200640 1 -- 172.21.15.62:0/3009552501 wait complete. 2024-03-23T11:20:32.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.268+0000 7fd59b200640 1 Processor -- start 2024-03-23T11:20:32.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.268+0000 7fd59b200640 1 -- start start 2024-03-23T11:20:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.269+0000 7fd59b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd580003970 0x7fd59c15a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.269+0000 7fd59b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd59c15c5d0 0x7fd59c156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.269+0000 7fd59b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59c152420 0x7fd59c0b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.269+0000 7fd59b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c1554d0 con 0x7fd59c15c5d0 2024-03-23T11:20:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.269+0000 7fd59b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c157500 con 0x7fd59c152420 2024-03-23T11:20:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.269+0000 7fd59b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c155ff0 con 0x7fd580003970 2024-03-23T11:20:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.269+0000 7fd593e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd59c15c5d0 0x7fd59c156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.270+0000 7fd593e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd59c15c5d0 0x7fd59c156fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55918/0 (socket says 172.21.15.62:55918) 2024-03-23T11:20:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.270+0000 7fd593e00640 1 -- 172.21.15.62:0/244188633 learned_addr learned my addr 172.21.15.62:0/244188633 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:32.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.270+0000 7fd593e00640 1 -- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd580003970 msgr2=0x7fd59c15a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:32.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.270+0000 7fd593e00640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd580003970 0x7fd59c15a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.270+0000 7fd593e00640 1 -- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59c152420 msgr2=0x7fd59c0b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:32.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.270+0000 7fd593e00640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59c152420 0x7fd59c0b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.270+0000 7fd593e00640 1 -- 172.21.15.62:0/244188633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c155750 con 0x7fd59c15c5d0 2024-03-23T11:20:32.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.270+0000 7fd593e00640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd59c15c5d0 0x7fd59c156fc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd584010860 tx=0x7fd584010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:32.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.271+0000 7fd592000640 1 -- 172.21.15.62:0/244188633 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584009610 con 0x7fd59c15c5d0 2024-03-23T11:20:32.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.271+0000 7fd592000640 1 -- 172.21.15.62:0/244188633 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58400eb60 con 0x7fd59c15c5d0 2024-03-23T11:20:32.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.271+0000 7fd592000640 1 -- 172.21.15.62:0/244188633 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58401c400 con 0x7fd59c15c5d0 2024-03-23T11:20:32.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.271+0000 7fd59b200640 1 -- 172.21.15.62:0/244188633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c155a30 con 0x7fd59c15c5d0 2024-03-23T11:20:32.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.271+0000 7fd59b200640 1 -- 172.21.15.62:0/244188633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59c1528a0 con 0x7fd59c15c5d0 2024-03-23T11:20:32.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.272+0000 7fd59b200640 1 -- 172.21.15.62:0/244188633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c066bb0 con 0x7fd59c15c5d0 2024-03-23T11:20:32.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.273+0000 7fd592000640 1 -- 172.21.15.62:0/244188633 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd58401c560 con 0x7fd59c15c5d0 2024-03-23T11:20:32.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.274+0000 7fd592000640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd57806a7a0 0x7fd57806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:32.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.274+0000 7fd592000640 1 -- 172.21.15.62:0/244188633 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd584089300 con 0x7fd59c15c5d0 2024-03-23T11:20:32.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.274+0000 7fd599e00640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd57806a7a0 0x7fd57806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:32.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.276+0000 7fd599e00640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd57806a7a0 0x7fd57806cc60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd58c035000 tx=0x7fd58c02e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:32.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.279+0000 7fd592000640 1 -- 172.21.15.62:0/244188633 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd5840547c0 con 0x7fd59c15c5d0 2024-03-23T11:20:32.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.517+0000 7fd59b200640 1 -- 172.21.15.62:0/244188633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd59c0ce200 con 0x7fd59c15c5d0 2024-03-23T11:20:32.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.517+0000 7fd592000640 1 -- 172.21.15.62:0/244188633 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd584057de0 con 0x7fd59c15c5d0 2024-03-23T11:20:32.518 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.520+0000 7fd59b200640 1 -- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd57806a7a0 msgr2=0x7fd57806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.520+0000 7fd59b200640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd57806a7a0 0x7fd57806cc60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd58c035000 tx=0x7fd58c02e000 comp rx=0 tx=0).stop 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.520+0000 7fd59b200640 1 -- 172.21.15.62:0/244188633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd59c15c5d0 msgr2=0x7fd59c156fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.520+0000 7fd59b200640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd59c15c5d0 0x7fd59c156fc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd584010860 tx=0x7fd584010d20 comp rx=0 tx=0).stop 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.520+0000 7fd59b200640 1 -- 172.21.15.62:0/244188633 shutdown_connections 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.520+0000 7fd59b200640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd57806a7a0 0x7fd57806cc60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.520+0000 7fd59b200640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd59c15c5d0 0x7fd59c156fc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.520+0000 7fd59b200640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd59c152420 0x7fd59c0b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.520+0000 7fd59b200640 1 --2- 172.21.15.62:0/244188633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd580003970 0x7fd59c15a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.520+0000 7fd59b200640 1 -- 172.21.15.62:0/244188633 >> 172.21.15.62:0/244188633 conn(0x7fd59c066050 msgr2=0x7fd59c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.521+0000 7fd59b200640 1 -- 172.21.15.62:0/244188633 shutdown_connections 2024-03-23T11:20:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:32.521+0000 7fd59b200640 1 -- 172.21.15.62:0/244188633 wait complete. 2024-03-23T11:20:32.539 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:20:32.540 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:20:32.540 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:20:32.540 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:20:32.540 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:20:32.540 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:20:32.540 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:20:32.540 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:20:32.540 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:20:32.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:32.540 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:20:32.593 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:20:32.593 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:20:32.593 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:20:32.593 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:20:32.593 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:20:32.593 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:20:32.593 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:20:32.593 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:20:32.593 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42524sec preferred_lft 42524sec 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:20:32.594 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:20:32.595 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:20:32.595 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:20:32.595 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:20:32.595 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:20:32.595 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:20:32.595 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:20:32.595 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:20:32.595 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:20:32.595 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:20:32.595 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:20:32.595 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:20:32.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:32.596 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:20:32.690 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:20:32.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:32.695 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:20:32.753 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:20:32.753 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:20:32.753 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:20:32.753 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:20:32.753 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:20:32.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:32.754 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:20:32.754 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:20:32.754 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:20:32.755 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:20:32.755 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:20:32.755 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:20:32.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:32.948 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:20:32.999 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:20:33.000 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:20:33.000 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:20:33.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:33.001 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:20:33.105 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:20:33.106 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:20:33.106 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:20:33.166 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:20:33.166 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:20:33.166 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:20:33.222 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:20:33.326 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:20:33.326 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:20:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:20:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:20:33.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:33.427 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:20:33.482 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:20:33.482 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:20:33.487 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:20:33.571 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:20:33.571 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:20:33.650 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:20:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:20:33.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:33.655 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:20:33.684 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:20:33.684+0000 7f5eee4d3480 -1 init, newargv = 0x5646a44bfc60 newargc=15 2024-03-23T11:20:33.684 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[50243]: starting ceph client 2024-03-23T11:20:33.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[50243]: starting fuse 2024-03-23T11:20:33.701 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:20:33.702 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:20:33.702 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:20:33.702 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:20:33.702 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:20:33.702 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:20:33.702 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:20:33.702 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:20:33.702 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:20:33.702 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:20:33.702 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:20:33.702 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:20:33.703 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:20:33.704 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:20:33.704 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:20:33.704 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:20:33.903 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.50243.asok 2024-03-23T11:20:34.082 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.50243.asok 2024-03-23T11:20:34.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:34.083 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.50243.asok status 2024-03-23T11:20:34.330 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:20:34.330 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:20:34.330 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "50243", 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 5400, 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:20:34.331 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 5400 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 3550691231 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 3550691231 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.5400 192.168.0.1:0/3550691231", 2024-03-23T11:20:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/3550691231", 2024-03-23T11:20:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:20:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 16, 2024-03-23T11:20:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 36, 2024-03-23T11:20:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:20:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:20:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:20:34.333 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:20:34.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:34.346 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:20:34.346 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:20:34.402 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:20:34.402 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:20:34.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:34.403 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:20:34.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:34.501 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir) 2024-03-23T11:20:34.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:34.569 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir/childdir) 2024-03-23T11:20:34.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:34.623 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch rfile) 2024-03-23T11:20:34.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:34.682 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/pfile) 2024-03-23T11:20:34.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:34.741 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/childdir/cfile) 2024-03-23T11:20:34.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:34.799 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-23T11:20:34.874 INFO:teuthology.orchestra.run.smithi062.stdout:./ 2024-03-23T11:20:34.875 INFO:teuthology.orchestra.run.smithi062.stdout:./rfile 2024-03-23T11:20:34.875 INFO:teuthology.orchestra.run.smithi062.stdout:./parentdir 2024-03-23T11:20:34.875 INFO:teuthology.orchestra.run.smithi062.stdout:./parentdir/pfile 2024-03-23T11:20:34.875 INFO:teuthology.orchestra.run.smithi062.stdout:./parentdir/childdir 2024-03-23T11:20:34.875 INFO:teuthology.orchestra.run.smithi062.stdout:./parentdir/childdir/cfile 2024-03-23T11:20:34.876 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:20:34.876 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:20:34.877 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:20:34.877 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:20:34.877 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-23T11:20:34.877 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:20:35.023 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:20:35.203 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:20:35.203 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:20:35.204 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:20:35.204 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:20:35.204 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./rfile").st_ino) 2024-03-23T11:20:35.204 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:20:35.397 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627778 2024-03-23T11:20:35.577 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:20:35.577 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:20:35.577 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:20:35.577 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:20:35.577 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir").st_ino) 2024-03-23T11:20:35.577 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:20:35.772 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627776 2024-03-23T11:20:35.951 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:20:35.952 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:20:35.952 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:20:35.952 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:20:35.952 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/pfile").st_ino) 2024-03-23T11:20:35.952 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:20:36.152 INFO:teuthology.orchestra.run.smithi062.stdout:1099511628279 2024-03-23T11:20:36.330 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:20:36.330 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:20:36.331 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:20:36.331 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:20:36.331 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir").st_ino) 2024-03-23T11:20:36.331 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:20:36.529 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627777 2024-03-23T11:20:36.708 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:20:36.708 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:20:36.708 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:20:36.708 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:20:36.708 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir/cfile").st_ino) 2024-03-23T11:20:36.708 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:20:36.902 INFO:teuthology.orchestra.run.smithi062.stdout:1099511628280 2024-03-23T11:20:37.081 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:20:37.081 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:20:37.183 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:20:37.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.413+0000 7f9b51800640 1 Processor -- start 2024-03-23T11:20:37.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.413+0000 7f9b51800640 1 -- start start 2024-03-23T11:20:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.414+0000 7f9b51800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9b4c160420 0x7f9b4c1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.414+0000 7f9b51800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b4c0b5db0 0x7f9b4c0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.414+0000 7f9b51800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b4c0ce510 con 0x7f9b4c15fcf0 2024-03-23T11:20:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.414+0000 7f9b51800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b4c160df0 con 0x7f9b4c160420 2024-03-23T11:20:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.414+0000 7f9b51800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c0cdc70 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.415+0000 7f9b4b400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9b4c160420 0x7f9b4c1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.415+0000 7f9b50e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b4c0b5db0 0x7f9b4c0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.415+0000 7f9b4b400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9b4c160420 0x7f9b4c1608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:36032/0 (socket says 172.21.15.62:36032) 2024-03-23T11:20:37.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.415+0000 7f9b4b400640 1 -- 172.21.15.62:0/1674904311 learned_addr learned my addr 172.21.15.62:0/1674904311 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:37.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.416+0000 7f9b4b400640 1 -- 172.21.15.62:0/1674904311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b4c0b5db0 msgr2=0x7f9b4c0b6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:37.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.416+0000 7f9b4b400640 1 --2- 172.21.15.62:0/1674904311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b4c0b5db0 0x7f9b4c0b6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.416+0000 7f9b4b400640 1 -- 172.21.15.62:0/1674904311 >> v1:172.21.15.120:6789/0 conn(0x7f9b4c15fcf0 legacy=0x7f9b4c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:37.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.416+0000 7f9b4b400640 1 -- 172.21.15.62:0/1674904311 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4c0cdf50 con 0x7f9b4c160420 2024-03-23T11:20:37.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.417+0000 7f9b4b400640 1 --2- 172.21.15.62:0/1674904311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b4c160420 0x7f9b4c1608b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9b34009870 tx=0x7f9b3402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b95adc3012f30131 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:37.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.418+0000 7f9b4aa00640 1 -- 172.21.15.62:0/1674904311 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34031640 con 0x7f9b4c160420 2024-03-23T11:20:37.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.418+0000 7f9b4aa00640 1 -- 172.21.15.62:0/1674904311 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b340317a0 con 0x7f9b4c160420 2024-03-23T11:20:37.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.418+0000 7f9b4aa00640 1 -- 172.21.15.62:0/1674904311 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34031b30 con 0x7f9b4c160420 2024-03-23T11:20:37.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.419+0000 7f9b51800640 1 -- 172.21.15.62:0/1674904311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b4c160420 msgr2=0x7f9b4c1608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:37.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.419+0000 7f9b51800640 1 --2- 172.21.15.62:0/1674904311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b4c160420 0x7f9b4c1608b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9b34009870 tx=0x7f9b3402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.419+0000 7f9b51800640 1 -- 172.21.15.62:0/1674904311 shutdown_connections 2024-03-23T11:20:37.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.419+0000 7f9b51800640 1 --2- 172.21.15.62:0/1674904311 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b4c0b5db0 0x7f9b4c0b6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.419+0000 7f9b51800640 1 --2- 172.21.15.62:0/1674904311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b4c160420 0x7f9b4c1608b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.419+0000 7f9b51800640 1 -- 172.21.15.62:0/1674904311 >> 172.21.15.62:0/1674904311 conn(0x7f9b4c089f60 msgr2=0x7f9b4c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:37.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.419+0000 7f9b51800640 1 -- 172.21.15.62:0/1674904311 shutdown_connections 2024-03-23T11:20:37.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.419+0000 7f9b51800640 1 -- 172.21.15.62:0/1674904311 wait complete. 2024-03-23T11:20:37.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.422+0000 7f9b51800640 1 Processor -- start 2024-03-23T11:20:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.422+0000 7f9b51800640 1 -- start start 2024-03-23T11:20:37.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.423+0000 7f9b51800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b4c0b5db0 0x7f9b4c09ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.423+0000 7f9b51800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b4c15fcf0 0x7f9b4c0b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.423+0000 7f9b51800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b4c160420 0x7f9b4c09e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.423+0000 7f9b51800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c09eef0 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.423+0000 7f9b51800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c09f1d0 con 0x7f9b4c160420 2024-03-23T11:20:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.423+0000 7f9b51800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c09f4b0 con 0x7f9b4c15fcf0 2024-03-23T11:20:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.424+0000 7f9b4be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b4c0b5db0 0x7f9b4c09ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.424+0000 7f9b4be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b4c0b5db0 0x7f9b4c09ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41458/0 (socket says 172.21.15.62:41458) 2024-03-23T11:20:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.424+0000 7f9b4be00640 1 -- 172.21.15.62:0/1805525010 learned_addr learned my addr 172.21.15.62:0/1805525010 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.424+0000 7f9b50e00640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b4c160420 0x7f9b4c09e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.424+0000 7f9b4b400640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b4c15fcf0 0x7f9b4c0b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.424+0000 7f9b4be00640 1 -- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b4c15fcf0 msgr2=0x7f9b4c0b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:37.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.424+0000 7f9b4be00640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b4c15fcf0 0x7f9b4c0b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.424+0000 7f9b4be00640 1 -- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b4c160420 msgr2=0x7f9b4c09e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:37.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.424+0000 7f9b4be00640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b4c160420 0x7f9b4c09e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.424+0000 7f9b4be00640 1 -- 172.21.15.62:0/1805525010 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b340094d0 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.425+0000 7f9b4be00640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b4c0b5db0 0x7f9b4c09ff90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00c400 tx=0x7f9b3c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:37.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.425+0000 7f9b43e00640 1 -- 172.21.15.62:0/1805525010 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0052d0 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.425+0000 7f9b43e00640 1 -- 172.21.15.62:0/1805525010 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3c005430 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.425+0000 7f9b51800640 1 -- 172.21.15.62:0/1805525010 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b4c09f790 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.425+0000 7f9b43e00640 1 -- 172.21.15.62:0/1805525010 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c003830 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.425+0000 7f9b51800640 1 -- 172.21.15.62:0/1805525010 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4c0b6d50 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.426+0000 7f9b51800640 1 -- 172.21.15.62:0/1805525010 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b4c09fcb0 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.428+0000 7f9b43e00640 1 -- 172.21.15.62:0/1805525010 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9b3c003990 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.428+0000 7f9b43e00640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b2806a7a0 0x7f9b2806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.429+0000 7f9b43e00640 1 -- 172.21.15.62:0/1805525010 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b3c088470 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.429+0000 7f9b4b400640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b2806a7a0 0x7f9b2806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.430+0000 7f9b4b400640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b2806a7a0 0x7f9b2806cc60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9b34036000 tx=0x7f9b3403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:37.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.433+0000 7f9b43e00640 1 -- 172.21.15.62:0/1805525010 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9b4c09fcb0 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.633+0000 7f9b51800640 1 -- 172.21.15.62:0/1805525010 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b4c09fcb0 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.635+0000 7f9b43e00640 1 -- 172.21.15.62:0/1805525010 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16101 (secure 0 0 0) 0x7f9b4c09fcb0 con 0x7f9b4c0b5db0 2024-03-23T11:20:37.635 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:37.635 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:20:25.735105+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:20:22.395497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T11:20:23.725162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:20:37.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 -- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b2806a7a0 msgr2=0x7f9b2806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:37.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b2806a7a0 0x7f9b2806cc60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9b34036000 tx=0x7f9b3403c040 comp rx=0 tx=0).stop 2024-03-23T11:20:37.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 -- 172.21.15.62:0/1805525010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b4c0b5db0 msgr2=0x7f9b4c09ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:37.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b4c0b5db0 0x7f9b4c09ff90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00c400 tx=0x7f9b3c00c8c0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 -- 172.21.15.62:0/1805525010 shutdown_connections 2024-03-23T11:20:37.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b2806a7a0 0x7f9b2806cc60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b4c160420 0x7f9b4c09e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b4c15fcf0 0x7f9b4c0b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 --2- 172.21.15.62:0/1805525010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b4c0b5db0 0x7f9b4c09ff90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 -- 172.21.15.62:0/1805525010 >> 172.21.15.62:0/1805525010 conn(0x7f9b4c089f60 msgr2=0x7f9b4c0cb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:37.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 -- 172.21.15.62:0/1805525010 shutdown_connections 2024-03-23T11:20:37.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.638+0000 7f9b51800640 1 -- 172.21.15.62:0/1805525010 wait complete. 2024-03-23T11:20:37.658 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:20:22.395497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:20:23.725162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:20:37.658 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:20:37.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.876+0000 7f7b2de00640 1 Processor -- start 2024-03-23T11:20:37.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.877+0000 7f7b2de00640 1 -- start start 2024-03-23T11:20:37.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.879+0000 7f7b2de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b2815fcf0 0x7f7b280d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.879+0000 7f7b2de00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7b280b6dc0 0x7f7b28160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.879+0000 7f7b2de00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b28160dc0 con 0x7f7b28160420 2024-03-23T11:20:37.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.879+0000 7f7b2de00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b280d3540 con 0x7f7b280b6dc0 2024-03-23T11:20:37.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.879+0000 7f7b2de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b280ce510 con 0x7f7b2815fcf0 2024-03-23T11:20:37.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.879+0000 7f7b27e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7b28160420 0x7f7b280d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:45374/0 (socket says 172.21.15.62:45374) 2024-03-23T11:20:37.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.879+0000 7f7b27e00640 1 -- 172.21.15.62:0/4205122386 learned_addr learned my addr 172.21.15.62:0/4205122386 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:37.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.879+0000 7f7b2d400640 1 --2- 172.21.15.62:0/4205122386 >> v2:172.21.15.120:3300/0 conn(0x7f7b280b6dc0 0x7f7b28160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.879+0000 7f7b2ca00640 1 --2- 172.21.15.62:0/4205122386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b2815fcf0 0x7f7b280d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.880+0000 7f7b27400640 1 -- 172.21.15.62:0/4205122386 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102183394 0 0) 0x7f7b28160dc0 con 0x7f7b28160420 2024-03-23T11:20:37.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.880+0000 7f7b27400640 1 -- 172.21.15.62:0/4205122386 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b0c003520 con 0x7f7b28160420 2024-03-23T11:20:37.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.880+0000 7f7b2d400640 1 -- 172.21.15.62:0/4205122386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b2815fcf0 msgr2=0x7f7b280d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:37.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.880+0000 7f7b2d400640 1 --2- 172.21.15.62:0/4205122386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b2815fcf0 0x7f7b280d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.880+0000 7f7b2d400640 1 -- 172.21.15.62:0/4205122386 >> v1:172.21.15.62:6789/0 conn(0x7f7b28160420 legacy=0x7f7b280d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:37.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.880+0000 7f7b27400640 1 -- 172.21.15.62:0/4205122386 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3405879318 0 0) 0x7f7b0c003520 con 0x7f7b28160420 2024-03-23T11:20:37.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.880+0000 7f7b2d400640 1 -- 172.21.15.62:0/4205122386 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b280cdc70 con 0x7f7b280b6dc0 2024-03-23T11:20:37.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.882+0000 7f7b2d400640 1 --2- 172.21.15.62:0/4205122386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b280b6dc0 0x7f7b28160880 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7b200097a0 tx=0x7f7b2002b660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc80ef6077c76889 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:37.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.882+0000 7f7b27400640 1 -- 172.21.15.62:0/4205122386 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b180019f0 con 0x7f7b280b6dc0 2024-03-23T11:20:37.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.882+0000 7f7b27400640 1 -- 172.21.15.62:0/4205122386 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2002e050 con 0x7f7b280b6dc0 2024-03-23T11:20:37.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.882+0000 7f7b27400640 1 -- 172.21.15.62:0/4205122386 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20030710 con 0x7f7b280b6dc0 2024-03-23T11:20:37.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.883+0000 7f7b2de00640 1 -- 172.21.15.62:0/4205122386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b280b6dc0 msgr2=0x7f7b28160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:37.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.883+0000 7f7b2de00640 1 --2- 172.21.15.62:0/4205122386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b280b6dc0 0x7f7b28160880 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7b200097a0 tx=0x7f7b2002b660 comp rx=0 tx=0).stop 2024-03-23T11:20:37.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.883+0000 7f7b2de00640 1 -- 172.21.15.62:0/4205122386 shutdown_connections 2024-03-23T11:20:37.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.883+0000 7f7b2de00640 1 --2- 172.21.15.62:0/4205122386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b280b6dc0 0x7f7b28160880 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.883+0000 7f7b2de00640 1 --2- 172.21.15.62:0/4205122386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b2815fcf0 0x7f7b280d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.883+0000 7f7b2de00640 1 -- 172.21.15.62:0/4205122386 >> 172.21.15.62:0/4205122386 conn(0x7f7b28089f60 msgr2=0x7f7b280d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:37.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.883+0000 7f7b2de00640 1 -- 172.21.15.62:0/4205122386 shutdown_connections 2024-03-23T11:20:37.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.883+0000 7f7b2de00640 1 -- 172.21.15.62:0/4205122386 wait complete. 2024-03-23T11:20:37.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.885+0000 7f7b2de00640 1 Processor -- start 2024-03-23T11:20:37.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.885+0000 7f7b2de00640 1 -- start start 2024-03-23T11:20:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b2de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b280b6dc0 0x7f7b28149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b2de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b2815fcf0 0x7f7b28158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b2de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b28160420 0x7f7b2815a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b2de00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b28159630 con 0x7f7b280b6dc0 2024-03-23T11:20:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b2de00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b28159910 con 0x7f7b2815fcf0 2024-03-23T11:20:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b2de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b280c6050 con 0x7f7b28160420 2024-03-23T11:20:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b2ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b280b6dc0 0x7f7b28149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b2ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b280b6dc0 0x7f7b28149430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41470/0 (socket says 172.21.15.62:41470) 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b2ca00640 1 -- 172.21.15.62:0/115619727 learned_addr learned my addr 172.21.15.62:0/115619727 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b27e00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b2815fcf0 0x7f7b28158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b2d400640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b28160420 0x7f7b2815a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.886+0000 7f7b2ca00640 1 -- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b28160420 msgr2=0x7f7b2815a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.887+0000 7f7b2ca00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b28160420 0x7f7b2815a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.887+0000 7f7b2ca00640 1 -- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b2815fcf0 msgr2=0x7f7b28158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.887+0000 7f7b2ca00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b2815fcf0 0x7f7b28158be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.887+0000 7f7b2ca00640 1 -- 172.21.15.62:0/115619727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20009400 con 0x7f7b280b6dc0 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.887+0000 7f7b27e00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b2815fcf0 0x7f7b28158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.887+0000 7f7b2ca00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b280b6dc0 0x7f7b28149430 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7b1000e9b0 tx=0x7f7b1000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.887+0000 7f7b24c00640 1 -- 172.21.15.62:0/115619727 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b10009670 con 0x7f7b280b6dc0 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.887+0000 7f7b24c00640 1 -- 172.21.15.62:0/115619727 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1000c9a0 con 0x7f7b280b6dc0 2024-03-23T11:20:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.887+0000 7f7b24c00640 1 -- 172.21.15.62:0/115619727 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1000cd50 con 0x7f7b280b6dc0 2024-03-23T11:20:37.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.887+0000 7f7b2de00640 1 -- 172.21.15.62:0/115619727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b280c6330 con 0x7f7b280b6dc0 2024-03-23T11:20:37.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.887+0000 7f7b2de00640 1 -- 172.21.15.62:0/115619727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b28149de0 con 0x7f7b280b6dc0 2024-03-23T11:20:37.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.888+0000 7f7b2de00640 1 -- 172.21.15.62:0/115619727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b280c5eb0 con 0x7f7b280b6dc0 2024-03-23T11:20:37.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.889+0000 7f7b24c00640 1 -- 172.21.15.62:0/115619727 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7b10013470 con 0x7f7b280b6dc0 2024-03-23T11:20:37.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.891+0000 7f7b24c00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7b0406a7a0 0x7f7b0406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:37.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.891+0000 7f7b24c00640 1 -- 172.21.15.62:0/115619727 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b10089000 con 0x7f7b280b6dc0 2024-03-23T11:20:37.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.891+0000 7f7b27e00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7b0406a7a0 0x7f7b0406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:37.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.892+0000 7f7b27e00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7b0406a7a0 0x7f7b0406cc60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7b2808a760 tx=0x7f7b18009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:37.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:37.895+0000 7f7b24c00640 1 -- 172.21.15.62:0/115619727 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b280c5eb0 con 0x7f7b280b6dc0 2024-03-23T11:20:38.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.101+0000 7f7b2de00640 1 -- 172.21.15.62:0/115619727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b280c5eb0 con 0x7f7b280b6dc0 2024-03-23T11:20:38.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.102+0000 7f7b24c00640 1 -- 172.21.15.62:0/115619727 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b280c5eb0 con 0x7f7b280b6dc0 2024-03-23T11:20:38.102 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:20:38.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 -- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7b0406a7a0 msgr2=0x7f7b0406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:38.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7b0406a7a0 0x7f7b0406cc60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7b2808a760 tx=0x7f7b18009040 comp rx=0 tx=0).stop 2024-03-23T11:20:38.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 -- 172.21.15.62:0/115619727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b280b6dc0 msgr2=0x7f7b28149430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:38.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b280b6dc0 0x7f7b28149430 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7b1000e9b0 tx=0x7f7b1000ee70 comp rx=0 tx=0).stop 2024-03-23T11:20:38.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 -- 172.21.15.62:0/115619727 shutdown_connections 2024-03-23T11:20:38.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7b0406a7a0 0x7f7b0406cc60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7b28160420 0x7f7b2815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b2815fcf0 0x7f7b28158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 --2- 172.21.15.62:0/115619727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b280b6dc0 0x7f7b28149430 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 -- 172.21.15.62:0/115619727 >> 172.21.15.62:0/115619727 conn(0x7f7b28089f60 msgr2=0x7f7b2806bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:38.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 -- 172.21.15.62:0/115619727 shutdown_connections 2024-03-23T11:20:38.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.105+0000 7f7b2de00640 1 -- 172.21.15.62:0/115619727 wait complete. 2024-03-23T11:20:38.124 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:20:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.349+0000 7fee5f800640 1 Processor -- start 2024-03-23T11:20:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.350+0000 7fee5f800640 1 -- start start 2024-03-23T11:20:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.351+0000 7fee5f800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee5815c9e0 0x7fee581607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.351+0000 7fee5f800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fee5815cdc0 0x7fee580d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.351+0000 7fee5f800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fee580cf650 0x7fee580cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.351+0000 7fee5f800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee580d0020 con 0x7fee580cf650 2024-03-23T11:20:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.351+0000 7fee5f800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5806bfc0 con 0x7fee5815cdc0 2024-03-23T11:20:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.351+0000 7fee5f800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee580b5260 con 0x7fee5815c9e0 2024-03-23T11:20:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.352+0000 7fee5d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fee580cf650 0x7fee580cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.352+0000 7fee5d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fee580cf650 0x7fee580cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41474/0 (socket says 172.21.15.62:41474) 2024-03-23T11:20:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.352+0000 7fee5d800640 1 -- 172.21.15.62:0/2537137840 learned_addr learned my addr 172.21.15.62:0/2537137840 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.352+0000 7fee57e00640 1 --2- 172.21.15.62:0/2537137840 >> v2:172.21.15.120:3300/0 conn(0x7fee5815cdc0 0x7fee580d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.352+0000 7fee5ce00640 1 --2- 172.21.15.62:0/2537137840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee5815c9e0 0x7fee581607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.352+0000 7fee5d800640 1 -- 172.21.15.62:0/2537137840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee5815c9e0 msgr2=0x7fee581607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.352+0000 7fee5d800640 1 --2- 172.21.15.62:0/2537137840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee5815c9e0 0x7fee581607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.352+0000 7fee5d800640 1 -- 172.21.15.62:0/2537137840 >> v2:172.21.15.120:3300/0 conn(0x7fee5815cdc0 msgr2=0x7fee580d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.352+0000 7fee5d800640 1 --2- 172.21.15.62:0/2537137840 >> v2:172.21.15.120:3300/0 conn(0x7fee5815cdc0 0x7fee580d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.352+0000 7fee5d800640 1 -- 172.21.15.62:0/2537137840 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee580ce140 con 0x7fee580cf650 2024-03-23T11:20:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.352+0000 7fee5ce00640 1 --2- 172.21.15.62:0/2537137840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee5815c9e0 0x7fee581607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:38.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.354+0000 7fee5d800640 1 --2- 172.21.15.62:0/2537137840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee580cf650 0x7fee580cfae0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fee580b53f0 tx=0x7fee5002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f79a5e2990de107 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:38.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.354+0000 7fee57400640 1 -- 172.21.15.62:0/2537137840 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50031640 con 0x7fee580cf650 2024-03-23T11:20:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.354+0000 7fee57400640 1 -- 172.21.15.62:0/2537137840 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee500317a0 con 0x7fee580cf650 2024-03-23T11:20:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.354+0000 7fee57400640 1 -- 172.21.15.62:0/2537137840 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50031b30 con 0x7fee580cf650 2024-03-23T11:20:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.355+0000 7fee5f800640 1 -- 172.21.15.62:0/2537137840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee580cf650 msgr2=0x7fee580cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.355+0000 7fee5f800640 1 --2- 172.21.15.62:0/2537137840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee580cf650 0x7fee580cfae0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fee580b53f0 tx=0x7fee5002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.355+0000 7fee5f800640 1 -- 172.21.15.62:0/2537137840 shutdown_connections 2024-03-23T11:20:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.355+0000 7fee5f800640 1 --2- 172.21.15.62:0/2537137840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee580cf650 0x7fee580cfae0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.355+0000 7fee5f800640 1 --2- 172.21.15.62:0/2537137840 >> v2:172.21.15.120:3300/0 conn(0x7fee5815cdc0 0x7fee580d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.355+0000 7fee5f800640 1 --2- 172.21.15.62:0/2537137840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee5815c9e0 0x7fee581607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.355+0000 7fee5f800640 1 -- 172.21.15.62:0/2537137840 >> 172.21.15.62:0/2537137840 conn(0x7fee58071570 msgr2=0x7fee58071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.355+0000 7fee5f800640 1 -- 172.21.15.62:0/2537137840 shutdown_connections 2024-03-23T11:20:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.355+0000 7fee5f800640 1 -- 172.21.15.62:0/2537137840 wait complete. 2024-03-23T11:20:38.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.357+0000 7fee5f800640 1 Processor -- start 2024-03-23T11:20:38.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.357+0000 7fee5f800640 1 -- start start 2024-03-23T11:20:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.358+0000 7fee5f800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee580cf650 0x7fee580acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.358+0000 7fee5f800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee5815c9e0 0x7fee5815b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.358+0000 7fee5f800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5815cdc0 0x7fee580ab340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.358+0000 7fee5f800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee580ac930 con 0x7fee5815cdc0 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.358+0000 7fee5f800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee580c97e0 con 0x7fee580cf650 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.358+0000 7fee5f800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee580abd20 con 0x7fee5815c9e0 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.358+0000 7fee5d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5815cdc0 0x7fee580ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.358+0000 7fee5d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5815cdc0 0x7fee580ab340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41484/0 (socket says 172.21.15.62:41484) 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.358+0000 7fee5d800640 1 -- 172.21.15.62:0/3677202458 learned_addr learned my addr 172.21.15.62:0/3677202458 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.358+0000 7fee5ce00640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee580cf650 0x7fee580acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.358+0000 7fee57e00640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee5815c9e0 0x7fee5815b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.359+0000 7fee5d800640 1 -- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee5815c9e0 msgr2=0x7fee5815b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.359+0000 7fee5d800640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee5815c9e0 0x7fee5815b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.359+0000 7fee5d800640 1 -- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee580cf650 msgr2=0x7fee580acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:38.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.359+0000 7fee5d800640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee580cf650 0x7fee580acdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.359+0000 7fee5d800640 1 -- 172.21.15.62:0/3677202458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee500094d0 con 0x7fee5815cdc0 2024-03-23T11:20:38.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.359+0000 7fee5d800640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5815cdc0 0x7fee580ab340 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fee5002b930 tx=0x7fee500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:38.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.359+0000 7fee54c00640 1 -- 172.21.15.62:0/3677202458 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5003ac60 con 0x7fee5815cdc0 2024-03-23T11:20:38.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.359+0000 7fee54c00640 1 -- 172.21.15.62:0/3677202458 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5003adc0 con 0x7fee5815cdc0 2024-03-23T11:20:38.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.359+0000 7fee5f800640 1 -- 172.21.15.62:0/3677202458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee580ac000 con 0x7fee5815cdc0 2024-03-23T11:20:38.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.359+0000 7fee54c00640 1 -- 172.21.15.62:0/3677202458 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50044630 con 0x7fee5815cdc0 2024-03-23T11:20:38.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.360+0000 7fee5f800640 1 -- 172.21.15.62:0/3677202458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee580ac5c0 con 0x7fee5815cdc0 2024-03-23T11:20:38.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.361+0000 7fee5f800640 1 -- 172.21.15.62:0/3677202458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee580acae0 con 0x7fee5815cdc0 2024-03-23T11:20:38.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.362+0000 7fee54c00640 1 -- 172.21.15.62:0/3677202458 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fee5002f050 con 0x7fee5815cdc0 2024-03-23T11:20:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.363+0000 7fee54c00640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee3806a7a0 0x7fee3806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.363+0000 7fee54c00640 1 -- 172.21.15.62:0/3677202458 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee500aad40 con 0x7fee5815cdc0 2024-03-23T11:20:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.364+0000 7fee5ce00640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee3806a7a0 0x7fee3806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.365+0000 7fee5ce00640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee3806a7a0 0x7fee3806cc60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fee580ad3a0 tx=0x7fee480079c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:38.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.368+0000 7fee54c00640 1 -- 172.21.15.62:0/3677202458 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fee580acae0 con 0x7fee5815cdc0 2024-03-23T11:20:38.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.564+0000 7fee5f800640 1 -- 172.21.15.62:0/3677202458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fee580ac750 con 0x7fee5815cdc0 2024-03-23T11:20:38.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.565+0000 7fee54c00640 1 -- 172.21.15.62:0/3677202458 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7fee50076270 con 0x7fee5815cdc0 2024-03-23T11:20:38.565 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:20:38.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.568+0000 7fee5f800640 1 -- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee3806a7a0 msgr2=0x7fee3806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:38.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.568+0000 7fee5f800640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee3806a7a0 0x7fee3806cc60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fee580ad3a0 tx=0x7fee480079c0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.568+0000 7fee5f800640 1 -- 172.21.15.62:0/3677202458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5815cdc0 msgr2=0x7fee580ab340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:38.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.568+0000 7fee5f800640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5815cdc0 0x7fee580ab340 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fee5002b930 tx=0x7fee500046c0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.568+0000 7fee5f800640 1 -- 172.21.15.62:0/3677202458 shutdown_connections 2024-03-23T11:20:38.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.569+0000 7fee5f800640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee3806a7a0 0x7fee3806cc60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.569+0000 7fee5f800640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee5815cdc0 0x7fee580ab340 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.569+0000 7fee5f800640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee5815c9e0 0x7fee5815b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.569+0000 7fee5f800640 1 --2- 172.21.15.62:0/3677202458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee580cf650 0x7fee580acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.569+0000 7fee5f800640 1 -- 172.21.15.62:0/3677202458 >> 172.21.15.62:0/3677202458 conn(0x7fee58071570 msgr2=0x7fee5815f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:38.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.569+0000 7fee5f800640 1 -- 172.21.15.62:0/3677202458 shutdown_connections 2024-03-23T11:20:38.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.569+0000 7fee5f800640 1 -- 172.21.15.62:0/3677202458 wait complete. 2024-03-23T11:20:38.588 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:20:38.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.815+0000 7f0fefe00640 1 Processor -- start 2024-03-23T11:20:38.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.815+0000 7f0fefe00640 1 -- start start 2024-03-23T11:20:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.816+0000 7f0fefe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0ff015ca00 0x7f0ff00ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.816+0000 7f0fefe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ff00d0d40 0x7f0ff00d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.816+0000 7f0fefe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0ff00b24e0 0x7f0ff00b2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.816+0000 7f0fefe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff00d1700 con 0x7f0ff015ca00 2024-03-23T11:20:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.816+0000 7f0fefe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff00b2eb0 con 0x7f0ff00b24e0 2024-03-23T11:20:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.816+0000 7f0fefe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff00cfe40 con 0x7f0ff00d0d40 2024-03-23T11:20:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.817+0000 7f0feea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0ff015ca00 0x7f0ff00ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.817+0000 7f0feea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0ff015ca00 0x7f0ff00ce160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41488/0 (socket says 172.21.15.62:41488) 2024-03-23T11:20:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.817+0000 7f0feea00640 1 -- 172.21.15.62:0/3096116146 learned_addr learned my addr 172.21.15.62:0/3096116146 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.817+0000 7f0feea00640 1 -- 172.21.15.62:0/3096116146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ff00d0d40 msgr2=0x7f0ff00d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.817+0000 7f0feea00640 1 --2- 172.21.15.62:0/3096116146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ff00d0d40 0x7f0ff00d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.817+0000 7f0fef400640 1 --2- 172.21.15.62:0/3096116146 >> v2:172.21.15.120:3300/0 conn(0x7f0ff00b24e0 0x7f0ff00b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.817+0000 7f0feea00640 1 -- 172.21.15.62:0/3096116146 >> v2:172.21.15.120:3300/0 conn(0x7f0ff00b24e0 msgr2=0x7f0ff00b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:38.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.817+0000 7f0feea00640 1 --2- 172.21.15.62:0/3096116146 >> v2:172.21.15.120:3300/0 conn(0x7f0ff00b24e0 0x7f0ff00b2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.817+0000 7f0feea00640 1 -- 172.21.15.62:0/3096116146 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff00cd680 con 0x7f0ff015ca00 2024-03-23T11:20:38.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.818+0000 7f0feea00640 1 --2- 172.21.15.62:0/3096116146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ff015ca00 0x7f0ff00ce160 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0160530 tx=0x7f0fe002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225b51fa295946ce server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:38.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.819+0000 7f0fee000640 1 -- 172.21.15.62:0/3096116146 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0004230 con 0x7f0ff015ca00 2024-03-23T11:20:38.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.819+0000 7f0fee000640 1 -- 172.21.15.62:0/3096116146 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe0004390 con 0x7f0ff015ca00 2024-03-23T11:20:38.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.819+0000 7f0fee000640 1 -- 172.21.15.62:0/3096116146 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0003810 con 0x7f0ff015ca00 2024-03-23T11:20:38.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.819+0000 7f0fefe00640 1 -- 172.21.15.62:0/3096116146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ff015ca00 msgr2=0x7f0ff00ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:38.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.819+0000 7f0fefe00640 1 --2- 172.21.15.62:0/3096116146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ff015ca00 0x7f0ff00ce160 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0160530 tx=0x7f0fe002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.819+0000 7f0fefe00640 1 -- 172.21.15.62:0/3096116146 shutdown_connections 2024-03-23T11:20:38.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.819+0000 7f0fefe00640 1 --2- 172.21.15.62:0/3096116146 >> v2:172.21.15.120:3300/0 conn(0x7f0ff00b24e0 0x7f0ff00b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.819+0000 7f0fefe00640 1 --2- 172.21.15.62:0/3096116146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ff00d0d40 0x7f0ff00d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.819+0000 7f0fefe00640 1 --2- 172.21.15.62:0/3096116146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ff015ca00 0x7f0ff00ce160 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.819+0000 7f0fefe00640 1 -- 172.21.15.62:0/3096116146 >> 172.21.15.62:0/3096116146 conn(0x7f0ff009ba10 msgr2=0x7f0ff0089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:38.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.819+0000 7f0fefe00640 1 -- 172.21.15.62:0/3096116146 shutdown_connections 2024-03-23T11:20:38.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.820+0000 7f0fefe00640 1 -- 172.21.15.62:0/3096116146 wait complete. 2024-03-23T11:20:38.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.821+0000 7f0fefe00640 1 Processor -- start 2024-03-23T11:20:38.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.821+0000 7f0fefe00640 1 -- start start 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.822+0000 7f0fefe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ff00b24e0 0x7f0ff01f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.822+0000 7f0fefe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ff00d0d40 0x7f0ff015f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.822+0000 7f0fefe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff015ca00 0x7f0ff00cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.822+0000 7f0fefe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff014e460 con 0x7f0ff00b24e0 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.822+0000 7f0fefe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff015da60 con 0x7f0ff015ca00 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.822+0000 7f0fefe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff01f1ff0 con 0x7f0ff00d0d40 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.823+0000 7f0feea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ff00b24e0 0x7f0ff01f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.823+0000 7f0feea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ff00b24e0 0x7f0ff01f1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41490/0 (socket says 172.21.15.62:41490) 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.823+0000 7f0feea00640 1 -- 172.21.15.62:0/1540808155 learned_addr learned my addr 172.21.15.62:0/1540808155 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.823+0000 7f0fef400640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff015ca00 0x7f0ff00cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.823+0000 7f0fe6000640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ff00d0d40 0x7f0ff015f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.823+0000 7f0feea00640 1 -- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ff00d0d40 msgr2=0x7f0ff015f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:38.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.823+0000 7f0feea00640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ff00d0d40 0x7f0ff015f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.823+0000 7f0feea00640 1 -- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff015ca00 msgr2=0x7f0ff00cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:38.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.823+0000 7f0feea00640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff015ca00 0x7f0ff00cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:38.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.823+0000 7f0feea00640 1 -- 172.21.15.62:0/1540808155 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe0009460 con 0x7f0ff00b24e0 2024-03-23T11:20:38.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.824+0000 7f0feea00640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ff00b24e0 0x7f0ff01f1ab0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0000c00 tx=0x7f0fe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:38.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.824+0000 7f0fe7400640 1 -- 172.21.15.62:0/1540808155 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0003c40 con 0x7f0ff00b24e0 2024-03-23T11:20:38.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.824+0000 7f0fe7400640 1 -- 172.21.15.62:0/1540808155 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe003f070 con 0x7f0ff00b24e0 2024-03-23T11:20:38.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.824+0000 7f0fefe00640 1 -- 172.21.15.62:0/1540808155 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff015dcb0 con 0x7f0ff00b24e0 2024-03-23T11:20:38.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.824+0000 7f0fe7400640 1 -- 172.21.15.62:0/1540808155 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0004500 con 0x7f0ff00b24e0 2024-03-23T11:20:38.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.824+0000 7f0fefe00640 1 -- 172.21.15.62:0/1540808155 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff015e040 con 0x7f0ff00b24e0 2024-03-23T11:20:38.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.825+0000 7f0fefe00640 1 -- 172.21.15.62:0/1540808155 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff0152e40 con 0x7f0ff00b24e0 2024-03-23T11:20:38.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.826+0000 7f0fe7400640 1 -- 172.21.15.62:0/1540808155 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0fe0041030 con 0x7f0ff00b24e0 2024-03-23T11:20:38.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.827+0000 7f0fe7400640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0fcc06a7a0 0x7f0fcc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:38.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.827+0000 7f0fe7400640 1 -- 172.21.15.62:0/1540808155 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fe00ab0b0 con 0x7f0ff00b24e0 2024-03-23T11:20:38.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.828+0000 7f0fe6000640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0fcc06a7a0 0x7f0fcc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:38.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.829+0000 7f0fe6000640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0fcc06a7a0 0x7f0fcc06cc60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00b2290 tx=0x7f0fd8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:38.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:38.831+0000 7f0fe7400640 1 -- 172.21.15.62:0/1540808155 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0ff0152e40 con 0x7f0ff00b24e0 2024-03-23T11:20:39.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.031+0000 7f0fefe00640 1 -- 172.21.15.62:0/1540808155 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0ff015de40 con 0x7f0ff00b24e0 2024-03-23T11:20:39.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.032+0000 7f0fe7400640 1 -- 172.21.15.62:0/1540808155 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0fe0076560 con 0x7f0ff00b24e0 2024-03-23T11:20:39.032 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 -- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0fcc06a7a0 msgr2=0x7f0fcc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0fcc06a7a0 0x7f0fcc06cc60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00b2290 tx=0x7f0fd8008040 comp rx=0 tx=0).stop 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 -- 172.21.15.62:0/1540808155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ff00b24e0 msgr2=0x7f0ff01f1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ff00b24e0 0x7f0ff01f1ab0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0000c00 tx=0x7f0fe00046c0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 -- 172.21.15.62:0/1540808155 shutdown_connections 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0fcc06a7a0 0x7f0fcc06cc60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ff015ca00 0x7f0ff00cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ff00d0d40 0x7f0ff015f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 --2- 172.21.15.62:0/1540808155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ff00b24e0 0x7f0ff01f1ab0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 -- 172.21.15.62:0/1540808155 >> 172.21.15.62:0/1540808155 conn(0x7f0ff009ba10 msgr2=0x7f0ff0089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 -- 172.21.15.62:0/1540808155 shutdown_connections 2024-03-23T11:20:39.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.035+0000 7f0fefe00640 1 -- 172.21.15.62:0/1540808155 wait complete. 2024-03-23T11:20:39.055 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:20:39.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.279+0000 7fe034e00640 1 Processor -- start 2024-03-23T11:20:39.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.280+0000 7fe034e00640 1 -- start start 2024-03-23T11:20:39.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.283+0000 7fe034e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe03015cdc0 0x7fe0300d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.283+0000 7fe034e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe0300cf650 0x7fe0300cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.283+0000 7fe034e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03006bfc0 con 0x7fe03015c9e0 2024-03-23T11:20:39.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.283+0000 7fe034e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0300d0000 con 0x7fe0300cf650 2024-03-23T11:20:39.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.283+0000 7fe034e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0300ce140 con 0x7fe03015cdc0 2024-03-23T11:20:39.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.284+0000 7fe02e800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe0300cf650 0x7fe0300cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:39.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.284+0000 7fe02e800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe0300cf650 0x7fe0300cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41494/0 (socket says 172.21.15.62:41494) 2024-03-23T11:20:39.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.284+0000 7fe02e800640 1 -- 172.21.15.62:0/2624666825 learned_addr learned my addr 172.21.15.62:0/2624666825 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:39.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.284+0000 7fe02d400640 1 --2- 172.21.15.62:0/2624666825 >> v2:172.21.15.120:3300/0 conn(0x7fe03015cdc0 0x7fe0300d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:39.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.284+0000 7fe02e800640 1 -- 172.21.15.62:0/2624666825 >> v2:172.21.15.120:3300/0 conn(0x7fe03015cdc0 msgr2=0x7fe0300d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:39.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.284+0000 7fe02e800640 1 --2- 172.21.15.62:0/2624666825 >> v2:172.21.15.120:3300/0 conn(0x7fe03015cdc0 0x7fe0300d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.284+0000 7fe02e800640 1 -- 172.21.15.62:0/2624666825 >> v1:172.21.15.120:6789/0 conn(0x7fe03015c9e0 legacy=0x7fe0301607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:39.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.284+0000 7fe02e800640 1 -- 172.21.15.62:0/2624666825 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0300b5260 con 0x7fe0300cf650 2024-03-23T11:20:39.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.285+0000 7fe02e800640 1 --2- 172.21.15.62:0/2624666825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0300cf650 0x7fe0300cfac0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe0300b89a0 tx=0x7fe02802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68dc9b6f33506ee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:39.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.286+0000 7fe02ca00640 1 -- 172.21.15.62:0/2624666825 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02802ca80 con 0x7fe0300cf650 2024-03-23T11:20:39.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.286+0000 7fe02ca00640 1 -- 172.21.15.62:0/2624666825 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02802cbe0 con 0x7fe0300cf650 2024-03-23T11:20:39.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.286+0000 7fe02ca00640 1 -- 172.21.15.62:0/2624666825 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0280343f0 con 0x7fe0300cf650 2024-03-23T11:20:39.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.286+0000 7fe034e00640 1 -- 172.21.15.62:0/2624666825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0300cf650 msgr2=0x7fe0300cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:39.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.286+0000 7fe034e00640 1 --2- 172.21.15.62:0/2624666825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0300cf650 0x7fe0300cfac0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe0300b89a0 tx=0x7fe02802bee0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.286+0000 7fe034e00640 1 -- 172.21.15.62:0/2624666825 shutdown_connections 2024-03-23T11:20:39.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.286+0000 7fe034e00640 1 --2- 172.21.15.62:0/2624666825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0300cf650 0x7fe0300cfac0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.286+0000 7fe034e00640 1 --2- 172.21.15.62:0/2624666825 >> v2:172.21.15.120:3300/0 conn(0x7fe03015cdc0 0x7fe0300d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.286+0000 7fe034e00640 1 -- 172.21.15.62:0/2624666825 >> 172.21.15.62:0/2624666825 conn(0x7fe030071570 msgr2=0x7fe030071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:39.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.287+0000 7fe034e00640 1 -- 172.21.15.62:0/2624666825 shutdown_connections 2024-03-23T11:20:39.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.287+0000 7fe034e00640 1 -- 172.21.15.62:0/2624666825 wait complete. 2024-03-23T11:20:39.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.288+0000 7fe034e00640 1 Processor -- start 2024-03-23T11:20:39.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.288+0000 7fe034e00640 1 -- start start 2024-03-23T11:20:39.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.289+0000 7fe034e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0300cf650 0x7fe030152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.289+0000 7fe034e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe03015c9e0 0x7fe0301522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.289+0000 7fe034e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe03015cdc0 0x7fe0300cc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.289+0000 7fe034e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe030157420 con 0x7fe03015cdc0 2024-03-23T11:20:39.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.289+0000 7fe034e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe030157140 con 0x7fe03015c9e0 2024-03-23T11:20:39.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.289+0000 7fe034e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe03015e970 con 0x7fe0300cf650 2024-03-23T11:20:39.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.290+0000 7fe02e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe03015cdc0 0x7fe0300cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:39.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.290+0000 7fe02e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe03015cdc0 0x7fe0300cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41500/0 (socket says 172.21.15.62:41500) 2024-03-23T11:20:39.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.290+0000 7fe02e800640 1 -- 172.21.15.62:0/1082736502 learned_addr learned my addr 172.21.15.62:0/1082736502 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:39.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.290+0000 7fe02de00640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0300cf650 0x7fe030152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:39.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.290+0000 7fe02d400640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe03015c9e0 0x7fe0301522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:39.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.290+0000 7fe02e800640 1 -- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0300cf650 msgr2=0x7fe030152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:39.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.290+0000 7fe02e800640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0300cf650 0x7fe030152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.290+0000 7fe02e800640 1 -- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe03015c9e0 msgr2=0x7fe0301522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:39.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.290+0000 7fe02e800640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe03015c9e0 0x7fe0301522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.290+0000 7fe02e800640 1 -- 172.21.15.62:0/1082736502 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe028009b30 con 0x7fe03015cdc0 2024-03-23T11:20:39.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.290+0000 7fe02de00640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0300cf650 0x7fe030152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:20:39.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.291+0000 7fe02e800640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe03015cdc0 0x7fe0300cc380 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe028036040 tx=0x7fe028004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:39.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.291+0000 7fe026000640 1 -- 172.21.15.62:0/1082736502 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe028009510 con 0x7fe03015cdc0 2024-03-23T11:20:39.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.291+0000 7fe026000640 1 -- 172.21.15.62:0/1082736502 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02803e450 con 0x7fe03015cdc0 2024-03-23T11:20:39.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.291+0000 7fe034e00640 1 -- 172.21.15.62:0/1082736502 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03015ec50 con 0x7fe03015cdc0 2024-03-23T11:20:39.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.291+0000 7fe026000640 1 -- 172.21.15.62:0/1082736502 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02803e7e0 con 0x7fe03015cdc0 2024-03-23T11:20:39.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.291+0000 7fe034e00640 1 -- 172.21.15.62:0/1082736502 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03015f450 con 0x7fe03015cdc0 2024-03-23T11:20:39.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.292+0000 7fe034e00640 1 -- 172.21.15.62:0/1082736502 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03015f110 con 0x7fe03015cdc0 2024-03-23T11:20:39.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.293+0000 7fe026000640 1 -- 172.21.15.62:0/1082736502 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe02802cd50 con 0x7fe03015cdc0 2024-03-23T11:20:39.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.294+0000 7fe026000640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 0x7fe00c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.294+0000 7fe026000640 1 -- 172.21.15.62:0/1082736502 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0280ab8b0 con 0x7fe03015cdc0 2024-03-23T11:20:39.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.295+0000 7fe02de00640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 0x7fe00c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:39.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.296+0000 7fe02de00640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 0x7fe00c06cc60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe020003eb0 tx=0x7fe020009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:39.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.299+0000 7fe026000640 1 -- 172.21.15.62:0/1082736502 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe028076c30 con 0x7fe03015cdc0 2024-03-23T11:20:39.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.512+0000 7fe034e00640 1 -- 172.21.15.62:0/1082736502 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe030066c90 con 0x7fe03015cdc0 2024-03-23T11:20:39.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.513+0000 7fe026000640 1 -- 172.21.15.62:0/1082736502 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe028043080 con 0x7fe03015cdc0 2024-03-23T11:20:39.513 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:20:39.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 -- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 msgr2=0x7fe00c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:39.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 0x7fe00c06cc60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe020003eb0 tx=0x7fe020009040 comp rx=0 tx=0).stop 2024-03-23T11:20:39.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 -- 172.21.15.62:0/1082736502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe03015cdc0 msgr2=0x7fe0300cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:39.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe03015cdc0 0x7fe0300cc380 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe028036040 tx=0x7fe028004a40 comp rx=0 tx=0).stop 2024-03-23T11:20:39.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 -- 172.21.15.62:0/1082736502 shutdown_connections 2024-03-23T11:20:39.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 0x7fe00c06cc60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe03015cdc0 0x7fe0300cc380 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe03015c9e0 0x7fe0301522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 --2- 172.21.15.62:0/1082736502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0300cf650 0x7fe030152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 -- 172.21.15.62:0/1082736502 >> 172.21.15.62:0/1082736502 conn(0x7fe030071570 msgr2=0x7fe030160f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:39.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 -- 172.21.15.62:0/1082736502 shutdown_connections 2024-03-23T11:20:39.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.516+0000 7fe034e00640 1 -- 172.21.15.62:0/1082736502 wait complete. 2024-03-23T11:20:39.537 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:20:39.537 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:20:39.590 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:20:39.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.814+0000 7f6b3e600640 1 Processor -- start 2024-03-23T11:20:39.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.814+0000 7f6b3e600640 1 -- start start 2024-03-23T11:20:39.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.816+0000 7f6b3e600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6b380cda20 0x7f6b3806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.816+0000 7f6b3e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b380cde90 0x7f6b380ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.816+0000 7f6b3e600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b380ce820 con 0x7f6b38160c00 2024-03-23T11:20:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.816+0000 7f6b3e600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b380b1ad0 con 0x7f6b380cda20 2024-03-23T11:20:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.816+0000 7f6b3e600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b38160fe0 con 0x7f6b380cde90 2024-03-23T11:20:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.816+0000 7f6b3d200640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f6b38160c00 0x7f6b3806de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:55540/0 (socket says 172.21.15.62:55540) 2024-03-23T11:20:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.817+0000 7f6b3d200640 1 -- 172.21.15.62:0/1685393401 learned_addr learned my addr 172.21.15.62:0/1685393401 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.817+0000 7f6b3dc00640 1 --2- 172.21.15.62:0/1685393401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b380cde90 0x7f6b380ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.817+0000 7f6b37e00640 1 --2- 172.21.15.62:0/1685393401 >> v2:172.21.15.120:3300/0 conn(0x7f6b380cda20 0x7f6b3806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.817+0000 7f6b37400640 1 -- 172.21.15.62:0/1685393401 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496375126 0 0) 0x7f6b28001110 con 0x7f6b38160c00 2024-03-23T11:20:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.817+0000 7f6b37400640 1 -- 172.21.15.62:0/1685393401 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b20003520 con 0x7f6b38160c00 2024-03-23T11:20:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.817+0000 7f6b37e00640 1 -- 172.21.15.62:0/1685393401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b380cde90 msgr2=0x7f6b380ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.817+0000 7f6b37e00640 1 --2- 172.21.15.62:0/1685393401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b380cde90 0x7f6b380ce2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.817+0000 7f6b37e00640 1 -- 172.21.15.62:0/1685393401 >> v1:172.21.15.120:6789/0 conn(0x7f6b38160c00 legacy=0x7f6b3806de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.817+0000 7f6b37e00640 1 -- 172.21.15.62:0/1685393401 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b380b1510 con 0x7f6b380cda20 2024-03-23T11:20:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.819+0000 7f6b37e00640 1 --2- 172.21.15.62:0/1685393401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b380cda20 0x7f6b3806d530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0097a0 tx=0x7f6b2c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7242e1f14128eb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.819+0000 7f6b37400640 1 -- 172.21.15.62:0/1685393401 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c031830 con 0x7f6b380cda20 2024-03-23T11:20:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.819+0000 7f6b37400640 1 -- 172.21.15.62:0/1685393401 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2c031990 con 0x7f6b380cda20 2024-03-23T11:20:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.819+0000 7f6b37400640 1 -- 172.21.15.62:0/1685393401 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c031d20 con 0x7f6b380cda20 2024-03-23T11:20:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.820+0000 7f6b3e600640 1 -- 172.21.15.62:0/1685393401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b380cda20 msgr2=0x7f6b3806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.820+0000 7f6b3e600640 1 --2- 172.21.15.62:0/1685393401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b380cda20 0x7f6b3806d530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0097a0 tx=0x7f6b2c02b8d0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.820+0000 7f6b3e600640 1 -- 172.21.15.62:0/1685393401 shutdown_connections 2024-03-23T11:20:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.820+0000 7f6b3e600640 1 --2- 172.21.15.62:0/1685393401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b380cde90 0x7f6b380ce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.820+0000 7f6b3e600640 1 --2- 172.21.15.62:0/1685393401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b380cda20 0x7f6b3806d530 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.820+0000 7f6b3e600640 1 -- 172.21.15.62:0/1685393401 >> 172.21.15.62:0/1685393401 conn(0x7f6b38066050 msgr2=0x7f6b38069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.821+0000 7f6b3e600640 1 -- 172.21.15.62:0/1685393401 shutdown_connections 2024-03-23T11:20:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.821+0000 7f6b3e600640 1 -- 172.21.15.62:0/1685393401 wait complete. 2024-03-23T11:20:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.823+0000 7f6b3e600640 1 Processor -- start 2024-03-23T11:20:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.823+0000 7f6b3e600640 1 -- start start 2024-03-23T11:20:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.824+0000 7f6b3e600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b380cde90 0x7f6b3814fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.824+0000 7f6b3e600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b38160c00 0x7f6b3815ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.824+0000 7f6b3e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b3815f480 0x7f6b381eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.824+0000 7f6b3e600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3815d700 con 0x7f6b38160c00 2024-03-23T11:20:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.824+0000 7f6b3e600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3815f900 con 0x7f6b380cde90 2024-03-23T11:20:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.824+0000 7f6b3e600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b381eb950 con 0x7f6b3815f480 2024-03-23T11:20:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.824+0000 7f6b37e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b38160c00 0x7f6b3815ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:39.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.824+0000 7f6b37e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b38160c00 0x7f6b3815ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41510/0 (socket says 172.21.15.62:41510) 2024-03-23T11:20:39.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.824+0000 7f6b37e00640 1 -- 172.21.15.62:0/4028245420 learned_addr learned my addr 172.21.15.62:0/4028245420 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:39.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.825+0000 7f6b3dc00640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b3815f480 0x7f6b381eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:39.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.825+0000 7f6b37e00640 1 -- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b3815f480 msgr2=0x7f6b381eaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:39.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.825+0000 7f6b37e00640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b3815f480 0x7f6b381eaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.825+0000 7f6b37e00640 1 -- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b380cde90 msgr2=0x7f6b3814fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:39.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.825+0000 7f6b37e00640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b380cde90 0x7f6b3814fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:39.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.825+0000 7f6b37e00640 1 -- 172.21.15.62:0/4028245420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c009400 con 0x7f6b38160c00 2024-03-23T11:20:39.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.826+0000 7f6b37e00640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b38160c00 0x7f6b3815ef40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c02be60 tx=0x7f6b2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:39.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.826+0000 7f6b34c00640 1 -- 172.21.15.62:0/4028245420 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c004230 con 0x7f6b38160c00 2024-03-23T11:20:39.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.826+0000 7f6b34c00640 1 -- 172.21.15.62:0/4028245420 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2c004390 con 0x7f6b38160c00 2024-03-23T11:20:39.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.826+0000 7f6b34c00640 1 -- 172.21.15.62:0/4028245420 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c03a830 con 0x7f6b38160c00 2024-03-23T11:20:39.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.826+0000 7f6b3e600640 1 -- 172.21.15.62:0/4028245420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b381ebc30 con 0x7f6b38160c00 2024-03-23T11:20:39.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.826+0000 7f6b3e600640 1 -- 172.21.15.62:0/4028245420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b381ec1f0 con 0x7f6b38160c00 2024-03-23T11:20:39.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.828+0000 7f6b3e600640 1 -- 172.21.15.62:0/4028245420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b38066bb0 con 0x7f6b38160c00 2024-03-23T11:20:39.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.829+0000 7f6b34c00640 1 -- 172.21.15.62:0/4028245420 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6b2c03a990 con 0x7f6b38160c00 2024-03-23T11:20:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.831+0000 7f6b34c00640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6b1c06a7a0 0x7f6b1c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.831+0000 7f6b34c00640 1 -- 172.21.15.62:0/4028245420 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b2c0aad30 con 0x7f6b38160c00 2024-03-23T11:20:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.832+0000 7f6b3d200640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6b1c06a7a0 0x7f6b1c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:39.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.833+0000 7f6b3d200640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6b1c06a7a0 0x7f6b1c06cc60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6b28005880 tx=0x7f6b28001150 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:39.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:39.836+0000 7f6b34c00640 1 -- 172.21.15.62:0/4028245420 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6b2c0761d0 con 0x7f6b38160c00 2024-03-23T11:20:40.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.034+0000 7f6b3e600640 1 -- 172.21.15.62:0/4028245420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b380b1d40 con 0x7f6b38160c00 2024-03-23T11:20:40.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.035+0000 7f6b34c00640 1 -- 172.21.15.62:0/4028245420 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16101 (secure 0 0 0) 0x7f6b2c035090 con 0x7f6b38160c00 2024-03-23T11:20:40.035 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:40.036 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:20:25.735105+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:20:22.395497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T11:20:23.725162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:20:40.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 -- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6b1c06a7a0 msgr2=0x7f6b1c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6b1c06a7a0 0x7f6b1c06cc60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6b28005880 tx=0x7f6b28001150 comp rx=0 tx=0).stop 2024-03-23T11:20:40.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 -- 172.21.15.62:0/4028245420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b38160c00 msgr2=0x7f6b3815ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b38160c00 0x7f6b3815ef40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c02be60 tx=0x7f6b2c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 -- 172.21.15.62:0/4028245420 shutdown_connections 2024-03-23T11:20:40.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6b1c06a7a0 0x7f6b1c06cc60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b38160c00 0x7f6b3815ef40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b3815f480 0x7f6b381eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 --2- 172.21.15.62:0/4028245420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b380cde90 0x7f6b3814fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 -- 172.21.15.62:0/4028245420 >> 172.21.15.62:0/4028245420 conn(0x7f6b38066050 msgr2=0x7f6b3806de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:40.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 -- 172.21.15.62:0/4028245420 shutdown_connections 2024-03-23T11:20:40.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.038+0000 7f6b3e600640 1 -- 172.21.15.62:0/4028245420 wait complete. 2024-03-23T11:20:40.057 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:20:22.395497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:20:23.725162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:20:40.057 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:20:40.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.283+0000 7f571ac00640 1 Processor -- start 2024-03-23T11:20:40.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.283+0000 7f571ac00640 1 -- start start 2024-03-23T11:20:40.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.285+0000 7f571ac00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f571406ce70 0x7f5714065c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.285+0000 7f571ac00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57140ccd90 con 0x7f57140cc930 2024-03-23T11:20:40.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.285+0000 7f571ac00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57140ccfc0 con 0x7f571415c8e0 2024-03-23T11:20:40.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.285+0000 7f571ac00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571415ccc0 con 0x7f571406ce70 2024-03-23T11:20:40.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.285+0000 7f571a200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f571406ce70 0x7f5714065c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:40.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.285+0000 7f5718e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f57140cc930 0x7f571406c760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:45380/0 (socket says 172.21.15.62:45380) 2024-03-23T11:20:40.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.285+0000 7f5718e00640 1 -- 172.21.15.62:0/4201869807 learned_addr learned my addr 172.21.15.62:0/4201869807 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:40.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.285+0000 7f571a200640 1 -- 172.21.15.62:0/4201869807 >> v1:172.21.15.120:6789/0 conn(0x7f571415c8e0 legacy=0x7f57140c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:40.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.286+0000 7f571a200640 1 -- 172.21.15.62:0/4201869807 >> v1:172.21.15.62:6789/0 conn(0x7f57140cc930 legacy=0x7f571406c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.286+0000 7f571a200640 1 -- 172.21.15.62:0/4201869807 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5714069c60 con 0x7f571406ce70 2024-03-23T11:20:40.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.286+0000 7f5713e00640 1 -- 172.21.15.62:0/4201869807 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671320045 0 0) 0x7f57140ccd90 con 0x7f57140cc930 2024-03-23T11:20:40.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.287+0000 7f571a200640 1 --2- 172.21.15.62:0/4201869807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f571406ce70 0x7f5714065c40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f570c0097a0 tx=0x7f570c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29911d814640889 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:40.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.287+0000 7f5713e00640 1 -- 172.21.15.62:0/4201869807 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c031830 con 0x7f571406ce70 2024-03-23T11:20:40.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.287+0000 7f5713e00640 1 -- 172.21.15.62:0/4201869807 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c031990 con 0x7f571406ce70 2024-03-23T11:20:40.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.287+0000 7f5713e00640 1 -- 172.21.15.62:0/4201869807 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c031d20 con 0x7f571406ce70 2024-03-23T11:20:40.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.288+0000 7f571ac00640 1 -- 172.21.15.62:0/4201869807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f571406ce70 msgr2=0x7f5714065c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.288+0000 7f571ac00640 1 --2- 172.21.15.62:0/4201869807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f571406ce70 0x7f5714065c40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f570c0097a0 tx=0x7f570c02b8d0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.288+0000 7f571ac00640 1 -- 172.21.15.62:0/4201869807 shutdown_connections 2024-03-23T11:20:40.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.288+0000 7f571ac00640 1 --2- 172.21.15.62:0/4201869807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f571406ce70 0x7f5714065c40 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.288+0000 7f571ac00640 1 -- 172.21.15.62:0/4201869807 >> 172.21.15.62:0/4201869807 conn(0x7f57140cb7e0 msgr2=0x7f57140cbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:40.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.288+0000 7f571ac00640 1 -- 172.21.15.62:0/4201869807 shutdown_connections 2024-03-23T11:20:40.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.288+0000 7f571ac00640 1 -- 172.21.15.62:0/4201869807 wait complete. 2024-03-23T11:20:40.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.289+0000 7f571ac00640 1 Processor -- start 2024-03-23T11:20:40.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.290+0000 7f571ac00640 1 -- start start 2024-03-23T11:20:40.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f571ac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f571406ce70 0x7f571415f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f571ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57140cc930 0x7f57140ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f571ac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f571415c8e0 0x7f571414df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f571ac00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57140aee70 con 0x7f571406ce70 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f571ac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57140afcc0 con 0x7f571415c8e0 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f571ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57140af0b0 con 0x7f57140cc930 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f5719800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f571406ce70 0x7f571415f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f5719800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f571406ce70 0x7f571415f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41530/0 (socket says 172.21.15.62:41530) 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f5719800640 1 -- 172.21.15.62:0/633452928 learned_addr learned my addr 172.21.15.62:0/633452928 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f5719800640 1 -- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57140cc930 msgr2=0x7f57140ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f5719800640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57140cc930 0x7f57140ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f5719800640 1 -- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f571415c8e0 msgr2=0x7f571414df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f5719800640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f571415c8e0 0x7f571414df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.291+0000 7f5719800640 1 -- 172.21.15.62:0/633452928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c009400 con 0x7f571406ce70 2024-03-23T11:20:40.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.292+0000 7f5719800640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f571406ce70 0x7f571415f790 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f570400ba50 tx=0x7f570400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:40.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.292+0000 7f5711600640 1 -- 172.21.15.62:0/633452928 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570400cbe0 con 0x7f571406ce70 2024-03-23T11:20:40.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.292+0000 7f5711600640 1 -- 172.21.15.62:0/633452928 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570400cd80 con 0x7f571406ce70 2024-03-23T11:20:40.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.292+0000 7f5711600640 1 -- 172.21.15.62:0/633452928 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704014680 con 0x7f571406ce70 2024-03-23T11:20:40.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.292+0000 7f571ac00640 1 -- 172.21.15.62:0/633452928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57140af390 con 0x7f571406ce70 2024-03-23T11:20:40.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.292+0000 7f571ac00640 1 -- 172.21.15.62:0/633452928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57140af950 con 0x7f571406ce70 2024-03-23T11:20:40.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.293+0000 7f571ac00640 1 -- 172.21.15.62:0/633452928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57140672c0 con 0x7f571406ce70 2024-03-23T11:20:40.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.294+0000 7f5711600640 1 -- 172.21.15.62:0/633452928 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f57040147e0 con 0x7f571406ce70 2024-03-23T11:20:40.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.295+0000 7f5711600640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f56f406a7a0 0x7f56f406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.295+0000 7f5711600640 1 -- 172.21.15.62:0/633452928 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5704088d40 con 0x7f571406ce70 2024-03-23T11:20:40.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.296+0000 7f5718e00640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f56f406a7a0 0x7f56f406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:40.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.297+0000 7f5718e00640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f56f406a7a0 0x7f56f406cc60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5700007fd0 tx=0x7f5700007920 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:40.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.300+0000 7f5711600640 1 -- 172.21.15.62:0/633452928 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5704054170 con 0x7f571406ce70 2024-03-23T11:20:40.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.497+0000 7f571ac00640 1 -- 172.21.15.62:0/633452928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5714065a20 con 0x7f571406ce70 2024-03-23T11:20:40.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.498+0000 7f5711600640 1 -- 172.21.15.62:0/633452928 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f57040190a0 con 0x7f571406ce70 2024-03-23T11:20:40.498 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:20:40.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.500+0000 7f571ac00640 1 -- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f56f406a7a0 msgr2=0x7f56f406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.500+0000 7f571ac00640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f56f406a7a0 0x7f56f406cc60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5700007fd0 tx=0x7f5700007920 comp rx=0 tx=0).stop 2024-03-23T11:20:40.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.500+0000 7f571ac00640 1 -- 172.21.15.62:0/633452928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f571406ce70 msgr2=0x7f571415f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.500+0000 7f571ac00640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f571406ce70 0x7f571415f790 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f570400ba50 tx=0x7f570400bf10 comp rx=0 tx=0).stop 2024-03-23T11:20:40.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.500+0000 7f571ac00640 1 -- 172.21.15.62:0/633452928 shutdown_connections 2024-03-23T11:20:40.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.500+0000 7f571ac00640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f56f406a7a0 0x7f56f406cc60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.500+0000 7f571ac00640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f571415c8e0 0x7f571414df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.500+0000 7f571ac00640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57140cc930 0x7f57140ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.500+0000 7f571ac00640 1 --2- 172.21.15.62:0/633452928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f571406ce70 0x7f571415f790 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.500+0000 7f571ac00640 1 -- 172.21.15.62:0/633452928 >> 172.21.15.62:0/633452928 conn(0x7f57140cb7e0 msgr2=0x7f57140cbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:40.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.501+0000 7f571ac00640 1 -- 172.21.15.62:0/633452928 shutdown_connections 2024-03-23T11:20:40.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.501+0000 7f571ac00640 1 -- 172.21.15.62:0/633452928 wait complete. 2024-03-23T11:20:40.519 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:20:40.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.742+0000 7f4dc0000640 1 Processor -- start 2024-03-23T11:20:40.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.742+0000 7f4dc0000640 1 -- start start 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.743+0000 7f4dc0000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4db815c900 0x7f4db80d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.743+0000 7f4dc0000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4db815cce0 0x7f4db80b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.743+0000 7f4dc0000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db80cd690 0x7f4db80cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.744+0000 7f4dc0000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db80ce060 con 0x7f4db815c900 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.744+0000 7f4dc0000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db80d1100 con 0x7f4db815cce0 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.744+0000 7f4dc0000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db80d13e0 con 0x7f4db80cd690 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.744+0000 7f4dbd600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4db815c900 0x7f4db80d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.744+0000 7f4dbd600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4db815c900 0x7f4db80d21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41532/0 (socket says 172.21.15.62:41532) 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.744+0000 7f4dbd600640 1 -- 172.21.15.62:0/259404397 learned_addr learned my addr 172.21.15.62:0/259404397 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.744+0000 7f4dbd600640 1 -- 172.21.15.62:0/259404397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db80cd690 msgr2=0x7f4db80cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.744+0000 7f4dbd600640 1 --2- 172.21.15.62:0/259404397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db80cd690 0x7f4db80cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.744+0000 7f4dbd600640 1 -- 172.21.15.62:0/259404397 >> v2:172.21.15.120:3300/0 conn(0x7f4db815cce0 msgr2=0x7f4db80b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:40.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.744+0000 7f4dbd600640 1 --2- 172.21.15.62:0/259404397 >> v2:172.21.15.120:3300/0 conn(0x7f4db815cce0 0x7f4db80b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.744+0000 7f4dbd600640 1 -- 172.21.15.62:0/259404397 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db80d16c0 con 0x7f4db815c900 2024-03-23T11:20:40.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.745+0000 7f4dbd600640 1 --2- 172.21.15.62:0/259404397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4db815c900 0x7f4db80d21a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4db8072290 tx=0x7f4dac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28516266cce191c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:40.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.746+0000 7f4db7e00640 1 -- 172.21.15.62:0/259404397 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac030860 con 0x7f4db815c900 2024-03-23T11:20:40.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.746+0000 7f4db7e00640 1 -- 172.21.15.62:0/259404397 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac0309c0 con 0x7f4db815c900 2024-03-23T11:20:40.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.746+0000 7f4db7e00640 1 -- 172.21.15.62:0/259404397 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac030d50 con 0x7f4db815c900 2024-03-23T11:20:40.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.746+0000 7f4dc0000640 1 -- 172.21.15.62:0/259404397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4db815c900 msgr2=0x7f4db80d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.746+0000 7f4dc0000640 1 --2- 172.21.15.62:0/259404397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4db815c900 0x7f4db80d21a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4db8072290 tx=0x7f4dac02b9a0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.747+0000 7f4dc0000640 1 -- 172.21.15.62:0/259404397 shutdown_connections 2024-03-23T11:20:40.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.747+0000 7f4dc0000640 1 --2- 172.21.15.62:0/259404397 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db80cd690 0x7f4db80cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.747+0000 7f4dc0000640 1 --2- 172.21.15.62:0/259404397 >> v2:172.21.15.120:3300/0 conn(0x7f4db815cce0 0x7f4db80b82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.747+0000 7f4dc0000640 1 --2- 172.21.15.62:0/259404397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4db815c900 0x7f4db80d21a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.747+0000 7f4dc0000640 1 -- 172.21.15.62:0/259404397 >> 172.21.15.62:0/259404397 conn(0x7f4db8071390 msgr2=0x7f4db8071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:40.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.747+0000 7f4dc0000640 1 -- 172.21.15.62:0/259404397 shutdown_connections 2024-03-23T11:20:40.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.747+0000 7f4dc0000640 1 -- 172.21.15.62:0/259404397 wait complete. 2024-03-23T11:20:40.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.748+0000 7f4dc0000640 1 Processor -- start 2024-03-23T11:20:40.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.749+0000 7f4dc0000640 1 -- start start 2024-03-23T11:20:40.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dc0000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4db80cd690 0x7f4db80cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dc0000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db815c900 0x7f4db80a9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dc0000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db815cce0 0x7f4db814d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dc0000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db80a9e20 con 0x7f4db80cd690 2024-03-23T11:20:40.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dc0000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db80aa100 con 0x7f4db815c900 2024-03-23T11:20:40.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dc0000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db80aa3e0 con 0x7f4db815cce0 2024-03-23T11:20:40.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dbd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4db80cd690 0x7f4db80cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:40.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dbd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4db80cd690 0x7f4db80cbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41538/0 (socket says 172.21.15.62:41538) 2024-03-23T11:20:40.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dbd600640 1 -- 172.21.15.62:0/3093152511 learned_addr learned my addr 172.21.15.62:0/3093152511 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:40.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dbd600640 1 -- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db815cce0 msgr2=0x7f4db814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dbd600640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db815cce0 0x7f4db814d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dbd600640 1 -- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db815c900 msgr2=0x7f4db80a9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dbd600640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db815c900 0x7f4db80a9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.750+0000 7f4dbd600640 1 -- 172.21.15.62:0/3093152511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac0094d0 con 0x7f4db80cd690 2024-03-23T11:20:40.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.751+0000 7f4dbd600640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4db80cd690 0x7f4db80cbd10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4dac000c00 tx=0x7f4dac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:40.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.751+0000 7f4db5600640 1 -- 172.21.15.62:0/3093152511 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac039d40 con 0x7f4db80cd690 2024-03-23T11:20:40.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.751+0000 7f4db5600640 1 -- 172.21.15.62:0/3093152511 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac043400 con 0x7f4db80cd690 2024-03-23T11:20:40.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.751+0000 7f4dc0000640 1 -- 172.21.15.62:0/3093152511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db80aa6c0 con 0x7f4db80cd690 2024-03-23T11:20:40.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.751+0000 7f4db5600640 1 -- 172.21.15.62:0/3093152511 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac039ea0 con 0x7f4db80cd690 2024-03-23T11:20:40.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.751+0000 7f4dc0000640 1 -- 172.21.15.62:0/3093152511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db80c7960 con 0x7f4db80cd690 2024-03-23T11:20:40.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.752+0000 7f4dc0000640 1 -- 172.21.15.62:0/3093152511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db80d0ca0 con 0x7f4db80cd690 2024-03-23T11:20:40.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.753+0000 7f4db5600640 1 -- 172.21.15.62:0/3093152511 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4dac0437b0 con 0x7f4db80cd690 2024-03-23T11:20:40.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.754+0000 7f4db5600640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4d9c06a7a0 0x7f4d9c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:40.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.754+0000 7f4db5600640 1 -- 172.21.15.62:0/3093152511 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4dac03e070 con 0x7f4db80cd690 2024-03-23T11:20:40.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.755+0000 7f4dbcc00640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4d9c06a7a0 0x7f4d9c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:40.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.756+0000 7f4dbcc00640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4d9c06a7a0 0x7f4d9c06cc60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4db80d2590 tx=0x7f4da8009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:40.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.759+0000 7f4db5600640 1 -- 172.21.15.62:0/3093152511 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4dac077470 con 0x7f4db80cd690 2024-03-23T11:20:40.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.960+0000 7f4dc0000640 1 -- 172.21.15.62:0/3093152511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4db80672c0 con 0x7f4db80cd690 2024-03-23T11:20:40.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.961+0000 7f4db5600640 1 -- 172.21.15.62:0/3093152511 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4dac07aa90 con 0x7f4db80cd690 2024-03-23T11:20:40.961 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 -- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4d9c06a7a0 msgr2=0x7f4d9c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4d9c06a7a0 0x7f4d9c06cc60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4db80d2590 tx=0x7f4da8009250 comp rx=0 tx=0).stop 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 -- 172.21.15.62:0/3093152511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4db80cd690 msgr2=0x7f4db80cbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4db80cd690 0x7f4db80cbd10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4dac000c00 tx=0x7f4dac004730 comp rx=0 tx=0).stop 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 -- 172.21.15.62:0/3093152511 shutdown_connections 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4d9c06a7a0 0x7f4d9c06cc60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4db815cce0 0x7f4db814d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db815c900 0x7f4db80a9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 --2- 172.21.15.62:0/3093152511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4db80cd690 0x7f4db80cbd10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 -- 172.21.15.62:0/3093152511 >> 172.21.15.62:0/3093152511 conn(0x7f4db8071390 msgr2=0x7f4db8071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 -- 172.21.15.62:0/3093152511 shutdown_connections 2024-03-23T11:20:40.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:40.964+0000 7f4dc0000640 1 -- 172.21.15.62:0/3093152511 wait complete. 2024-03-23T11:20:40.984 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:20:41.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.202+0000 7f2a14800640 1 Processor -- start 2024-03-23T11:20:41.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.203+0000 7f2a14800640 1 -- start start 2024-03-23T11:20:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.204+0000 7f2a14800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2a0c0cafa0 0x7f2a0c0cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.204+0000 7f2a14800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0cbc60 con 0x7f2a0c06c300 2024-03-23T11:20:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.204+0000 7f2a14800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0cbf40 con 0x7f2a0c06c6e0 2024-03-23T11:20:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.204+0000 7f2a14800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0c0cd410 con 0x7f2a0c0cafa0 2024-03-23T11:20:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.205+0000 7f2a11e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2a0c06c300 0x7f2a0c0ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:45396/0 (socket says 172.21.15.62:45396) 2024-03-23T11:20:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.205+0000 7f2a11e00640 1 -- 172.21.15.62:0/1542012724 learned_addr learned my addr 172.21.15.62:0/1542012724 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.205+0000 7f2a11400640 1 --2- 172.21.15.62:0/1542012724 >> v2:172.21.15.120:3300/0 conn(0x7f2a0c0cafa0 0x7f2a0c0cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.205+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699359026 0 0) 0x7f2a0c0cbc60 con 0x7f2a0c06c300 2024-03-23T11:20:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.205+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f8003520 con 0x7f2a0c06c300 2024-03-23T11:20:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 542872558 0 0) 0x7f29f8003520 con 0x7f2a0c06c300 2024-03-23T11:20:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 >> v2:172.21.15.120:3300/0 conn(0x7f2a0c0cafa0 msgr2=0x7f2a0c0cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 --2- 172.21.15.62:0/1542012724 >> v2:172.21.15.120:3300/0 conn(0x7f2a0c0cafa0 0x7f2a0c0cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 >> v1:172.21.15.120:6789/0 conn(0x7f2a0c06c6e0 legacy=0x7f2a0c06cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c0cc220 con 0x7f2a0c06c300 2024-03-23T11:20:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f2a040019f0 con 0x7f2a0c06c300 2024-03-23T11:20:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 >> v1:172.21.15.62:6789/0 conn(0x7f2a0c06c300 legacy=0x7f2a0c0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 --2- 172.21.15.62:0/1542012724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f8003970 0x7f29f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 --2- 172.21.15.62:0/1542012724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f80047c0 0x7f29f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 --2- 172.21.15.62:0/1542012724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29f8005180 0x7f29f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c0cd410 con 0x7f29f8003970 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f800c020 con 0x7f29f80047c0 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f800c1e0 con 0x7f29f8005180 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a11e00640 1 --2- 172.21.15.62:0/1542012724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f8003970 0x7f29f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a11400640 1 --2- 172.21.15.62:0/1542012724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f80047c0 0x7f29f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a11e00640 1 -- 172.21.15.62:0/1542012724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29f8005180 msgr2=0x7f29f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.206+0000 7f2a11e00640 1 --2- 172.21.15.62:0/1542012724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29f8005180 0x7f29f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.207+0000 7f2a11e00640 1 -- 172.21.15.62:0/1542012724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f80047c0 msgr2=0x7f29f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.207+0000 7f2a11e00640 1 --2- 172.21.15.62:0/1542012724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f80047c0 0x7f29f8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.207+0000 7f2a11e00640 1 -- 172.21.15.62:0/1542012724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f8003520 con 0x7f29f8003970 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.207+0000 7f2a12800640 1 -- 172.21.15.62:0/1542012724 reap_dead start 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.208+0000 7f2a11e00640 1 --2- 172.21.15.62:0/1542012724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f8003970 0x7f29f8003d90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400a830 tx=0x7f29f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58403f82e313c5fd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.208+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a040072d0 con 0x7f29f8003970 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.208+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0402f070 con 0x7f29f8003970 2024-03-23T11:20:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.208+0000 7f2a10a00640 1 -- 172.21.15.62:0/1542012724 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a040076f0 con 0x7f29f8003970 2024-03-23T11:20:41.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.208+0000 7f2a14800640 1 -- 172.21.15.62:0/1542012724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f8003970 msgr2=0x7f29f8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.208+0000 7f2a14800640 1 --2- 172.21.15.62:0/1542012724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f8003970 0x7f29f8003d90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400a830 tx=0x7f29f80057f0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.209+0000 7f2a14800640 1 -- 172.21.15.62:0/1542012724 shutdown_connections 2024-03-23T11:20:41.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.209+0000 7f2a14800640 1 --2- 172.21.15.62:0/1542012724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f8003970 0x7f29f8003d90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.209+0000 7f2a14800640 1 -- 172.21.15.62:0/1542012724 >> 172.21.15.62:0/1542012724 conn(0x7f2a0c0b02e0 msgr2=0x7f2a0c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:41.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.209+0000 7f2a14800640 1 -- 172.21.15.62:0/1542012724 shutdown_connections 2024-03-23T11:20:41.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.209+0000 7f2a14800640 1 -- 172.21.15.62:0/1542012724 wait complete. 2024-03-23T11:20:41.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.211+0000 7f2a14800640 1 Processor -- start 2024-03-23T11:20:41.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.211+0000 7f2a14800640 1 -- start start 2024-03-23T11:20:41.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.212+0000 7f2a14800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f8003970 0x7f2a0c1372f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.212+0000 7f2a14800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c155060 0x7f2a0c1571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.212+0000 7f2a14800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c15c7b0 0x7f2a0c15cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.212+0000 7f2a14800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c137830 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.212+0000 7f2a14800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c155590 con 0x7f29f8003970 2024-03-23T11:20:41.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.212+0000 7f2a14800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c155870 con 0x7f2a0c155060 2024-03-23T11:20:41.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.212+0000 7f2a12800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c15c7b0 0x7f2a0c15cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.212+0000 7f2a12800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c15c7b0 0x7f2a0c15cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41566/0 (socket says 172.21.15.62:41566) 2024-03-23T11:20:41.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.212+0000 7f2a12800640 1 -- 172.21.15.62:0/3634017783 learned_addr learned my addr 172.21.15.62:0/3634017783 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:41.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a11e00640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f8003970 0x7f2a0c1372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a11400640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c155060 0x7f2a0c1571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a12800640 1 -- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c155060 msgr2=0x7f2a0c1571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a12800640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c155060 0x7f2a0c1571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a12800640 1 -- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f8003970 msgr2=0x7f2a0c1372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a12800640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f8003970 0x7f2a0c1372f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a12800640 1 -- 172.21.15.62:0/3634017783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c155b50 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a11400640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c155060 0x7f2a0c1571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:41.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a11e00640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f8003970 0x7f2a0c1372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:41.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a12800640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c15c7b0 0x7f2a0c15cc30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c15e3a0 tx=0x7f2a0800c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:41.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a02000640 1 -- 172.21.15.62:0/3634017783 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a08004220 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a02000640 1 -- 172.21.15.62:0/3634017783 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a080043a0 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a02000640 1 -- 172.21.15.62:0/3634017783 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a08007310 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.213+0000 7f2a14800640 1 -- 172.21.15.62:0/3634017783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c155e30 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.214+0000 7f2a14800640 1 -- 172.21.15.62:0/3634017783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c1576e0 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.215+0000 7f2a14800640 1 -- 172.21.15.62:0/3634017783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c066c90 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.215+0000 7f2a02000640 1 -- 172.21.15.62:0/3634017783 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2a08007470 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.217+0000 7f2a02000640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29f006a7d0 0x7f29f006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.217+0000 7f2a02000640 1 -- 172.21.15.62:0/3634017783 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a08087da0 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.218+0000 7f2a11e00640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29f006a7d0 0x7f29f006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.219+0000 7f2a11e00640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29f006a7d0 0x7f29f006cc90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2a04034000 tx=0x7f2a04007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:41.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.222+0000 7f2a02000640 1 -- 172.21.15.62:0/3634017783 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2a08053120 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.422+0000 7f2a14800640 1 -- 172.21.15.62:0/3634017783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2a0c157870 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.423+0000 7f2a02000640 1 -- 172.21.15.62:0/3634017783 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2a08056740 con 0x7f2a0c15c7b0 2024-03-23T11:20:41.423 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:20:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.426+0000 7f2a14800640 1 -- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29f006a7d0 msgr2=0x7f29f006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.426+0000 7f2a14800640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29f006a7d0 0x7f29f006cc90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2a04034000 tx=0x7f2a04007000 comp rx=0 tx=0).stop 2024-03-23T11:20:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.426+0000 7f2a14800640 1 -- 172.21.15.62:0/3634017783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c15c7b0 msgr2=0x7f2a0c15cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.426+0000 7f2a14800640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c15c7b0 0x7f2a0c15cc30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c15e3a0 tx=0x7f2a0800c5f0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.426+0000 7f2a14800640 1 -- 172.21.15.62:0/3634017783 shutdown_connections 2024-03-23T11:20:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.426+0000 7f2a14800640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29f006a7d0 0x7f29f006cc90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.426+0000 7f2a14800640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c15c7b0 0x7f2a0c15cc30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.426+0000 7f2a14800640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c155060 0x7f2a0c1571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.426+0000 7f2a14800640 1 --2- 172.21.15.62:0/3634017783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f8003970 0x7f2a0c1372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.426+0000 7f2a14800640 1 -- 172.21.15.62:0/3634017783 >> 172.21.15.62:0/3634017783 conn(0x7f2a0c0b02e0 msgr2=0x7f2a0c13ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.426+0000 7f2a14800640 1 -- 172.21.15.62:0/3634017783 shutdown_connections 2024-03-23T11:20:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.427+0000 7f2a14800640 1 -- 172.21.15.62:0/3634017783 wait complete. 2024-03-23T11:20:41.445 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:20:41.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.662+0000 7fdca9a00640 1 Processor -- start 2024-03-23T11:20:41.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.663+0000 7fdca9a00640 1 -- start start 2024-03-23T11:20:41.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.664+0000 7fdca9a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdca415c960 0x7fdca40cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.664+0000 7fdca9a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40d0810 con 0x7fdca40d13b0 2024-03-23T11:20:41.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.664+0000 7fdca9a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40cd9d0 con 0x7fdca40cd260 2024-03-23T11:20:41.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.664+0000 7fdca9a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca40cdcb0 con 0x7fdca415c960 2024-03-23T11:20:41.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.665+0000 7fdca9000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fdca40d13b0 0x7fdca40b3310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:45398/0 (socket says 172.21.15.62:45398) 2024-03-23T11:20:41.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.665+0000 7fdca9000640 1 -- 172.21.15.62:0/4078066651 learned_addr learned my addr 172.21.15.62:0/4078066651 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:41.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.665+0000 7fdca3e00640 1 --2- 172.21.15.62:0/4078066651 >> v2:172.21.15.120:3300/0 conn(0x7fdca415c960 0x7fdca40cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.665+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983775838 0 0) 0x7fdc9c001110 con 0x7fdca40d13b0 2024-03-23T11:20:41.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.665+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc88003520 con 0x7fdca40d13b0 2024-03-23T11:20:41.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 406164599 0 0) 0x7fdc88003520 con 0x7fdca40d13b0 2024-03-23T11:20:41.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 >> v2:172.21.15.120:3300/0 conn(0x7fdca415c960 msgr2=0x7fdca40cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:41.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 --2- 172.21.15.62:0/4078066651 >> v2:172.21.15.120:3300/0 conn(0x7fdca415c960 0x7fdca40cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 >> v1:172.21.15.120:6789/0 conn(0x7fdca40cd260 legacy=0x7fdca40d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca40cdf90 con 0x7fdca40d13b0 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fdc9c001800 con 0x7fdca40d13b0 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 >> v1:172.21.15.62:6789/0 conn(0x7fdca40d13b0 legacy=0x7fdca40b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 --2- 172.21.15.62:0/4078066651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc880039c0 0x7fdc88003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 --2- 172.21.15.62:0/4078066651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc880047c0 0x7fdc88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 --2- 172.21.15.62:0/4078066651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc88005180 0x7fdc8800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca40cdcb0 con 0x7fdc88005180 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8800c060 con 0x7fdc880047c0 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8800c220 con 0x7fdc880039c0 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdca9000640 1 --2- 172.21.15.62:0/4078066651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc88005180 0x7fdc8800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.666+0000 7fdc9b400640 1 --2- 172.21.15.62:0/4078066651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc880047c0 0x7fdc88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.667+0000 7fdca9000640 1 -- 172.21.15.62:0/4078066651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc880039c0 msgr2=0x7fdc88003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:41.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.667+0000 7fdca9000640 1 --2- 172.21.15.62:0/4078066651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc880039c0 0x7fdc88003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.667+0000 7fdca9000640 1 -- 172.21.15.62:0/4078066651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc880047c0 msgr2=0x7fdc88004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:41.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.667+0000 7fdca9000640 1 --2- 172.21.15.62:0/4078066651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc880047c0 0x7fdc88004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.667+0000 7fdca9000640 1 -- 172.21.15.62:0/4078066651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc88003520 con 0x7fdc88005180 2024-03-23T11:20:41.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.667+0000 7fdc9b400640 1 --2- 172.21.15.62:0/4078066651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc880047c0 0x7fdc88004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:41.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.667+0000 7fdca9000640 1 -- 172.21.15.62:0/4078066651 reap_dead start 2024-03-23T11:20:41.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.668+0000 7fdca9000640 1 --2- 172.21.15.62:0/4078066651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc88005180 0x7fdc8800b980 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c007fc0 tx=0x7fdc880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b52c484787cc4232 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:41.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.668+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c005ad0 con 0x7fdc88005180 2024-03-23T11:20:41.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.668+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9c005c70 con 0x7fdc88005180 2024-03-23T11:20:41.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.668+0000 7fdca3400640 1 -- 172.21.15.62:0/4078066651 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c02c7e0 con 0x7fdc88005180 2024-03-23T11:20:41.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.669+0000 7fdca9a00640 1 -- 172.21.15.62:0/4078066651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc88005180 msgr2=0x7fdc8800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.669+0000 7fdca9a00640 1 --2- 172.21.15.62:0/4078066651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc88005180 0x7fdc8800b980 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c007fc0 tx=0x7fdc880057f0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.669+0000 7fdca9a00640 1 -- 172.21.15.62:0/4078066651 shutdown_connections 2024-03-23T11:20:41.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.669+0000 7fdca9a00640 1 --2- 172.21.15.62:0/4078066651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc88005180 0x7fdc8800b980 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.669+0000 7fdca9a00640 1 -- 172.21.15.62:0/4078066651 >> 172.21.15.62:0/4078066651 conn(0x7fdca4089f60 msgr2=0x7fdca40d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:41.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.669+0000 7fdca9a00640 1 -- 172.21.15.62:0/4078066651 shutdown_connections 2024-03-23T11:20:41.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.669+0000 7fdca9a00640 1 -- 172.21.15.62:0/4078066651 wait complete. 2024-03-23T11:20:41.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.670+0000 7fdca9a00640 1 Processor -- start 2024-03-23T11:20:41.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.671+0000 7fdca9a00640 1 -- start start 2024-03-23T11:20:41.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdca9a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc88005180 0x7fdca415fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdca9a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdca4158490 0x7fdca40cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdca9a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdca40cbc10 0x7fdca409d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdca9a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca40cc090 con 0x7fdca4158490 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdca9a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca415e6f0 con 0x7fdc88005180 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdca9a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca415ddc0 con 0x7fdca40cbc10 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdc9b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdca4158490 0x7fdca40cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdc9b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdca4158490 0x7fdca40cb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41590/0 (socket says 172.21.15.62:41590) 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdc9b400640 1 -- 172.21.15.62:0/3065737547 learned_addr learned my addr 172.21.15.62:0/3065737547 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdc9b400640 1 -- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdca40cbc10 msgr2=0x7fdca409d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdc9b400640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdca40cbc10 0x7fdca409d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdc9b400640 1 -- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc88005180 msgr2=0x7fdca415fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdca3e00640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc88005180 0x7fdca415fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdc9b400640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc88005180 0x7fdca415fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.672+0000 7fdc9b400640 1 -- 172.21.15.62:0/3065737547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca415e0a0 con 0x7fdca4158490 2024-03-23T11:20:41.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.673+0000 7fdc9b400640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdca4158490 0x7fdca40cb6d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00d620 tx=0x7fdc8c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:41.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.673+0000 7fdca0c00640 1 -- 172.21.15.62:0/3065737547 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c017070 con 0x7fdca4158490 2024-03-23T11:20:41.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.673+0000 7fdca0c00640 1 -- 172.21.15.62:0/3065737547 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8c005280 con 0x7fdca4158490 2024-03-23T11:20:41.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.673+0000 7fdca0c00640 1 -- 172.21.15.62:0/3065737547 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c006820 con 0x7fdca4158490 2024-03-23T11:20:41.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.673+0000 7fdca9a00640 1 -- 172.21.15.62:0/3065737547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca415e380 con 0x7fdca4158490 2024-03-23T11:20:41.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.673+0000 7fdca9a00640 1 -- 172.21.15.62:0/3065737547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca4158910 con 0x7fdca4158490 2024-03-23T11:20:41.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.674+0000 7fdca9a00640 1 -- 172.21.15.62:0/3065737547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca415e510 con 0x7fdca4158490 2024-03-23T11:20:41.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.675+0000 7fdca0c00640 1 -- 172.21.15.62:0/3065737547 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fdc8c006980 con 0x7fdca4158490 2024-03-23T11:20:41.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.676+0000 7fdca0c00640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc8006a7a0 0x7fdc8006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:41.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.676+0000 7fdca0c00640 1 -- 172.21.15.62:0/3065737547 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc8c0886f0 con 0x7fdca4158490 2024-03-23T11:20:41.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.677+0000 7fdca3e00640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc8006a7a0 0x7fdc8006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:41.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.678+0000 7fdca3e00640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc8006a7a0 0x7fdc8006cc60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdc940073f0 tx=0x7fdc9400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:41.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.680+0000 7fdca0c00640 1 -- 172.21.15.62:0/3065737547 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdc8c053a70 con 0x7fdca4158490 2024-03-23T11:20:41.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.894+0000 7fdca9a00640 1 -- 172.21.15.62:0/3065737547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdca4066bb0 con 0x7fdca4158490 2024-03-23T11:20:41.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.895+0000 7fdca0c00640 1 -- 172.21.15.62:0/3065737547 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdc8c001280 con 0x7fdca4158490 2024-03-23T11:20:41.894 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:20:41.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.897+0000 7fdca9a00640 1 -- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc8006a7a0 msgr2=0x7fdc8006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.897+0000 7fdca9a00640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc8006a7a0 0x7fdc8006cc60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdc940073f0 tx=0x7fdc9400b040 comp rx=0 tx=0).stop 2024-03-23T11:20:41.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.898+0000 7fdca9a00640 1 -- 172.21.15.62:0/3065737547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdca4158490 msgr2=0x7fdca40cb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:41.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.898+0000 7fdca9a00640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdca4158490 0x7fdca40cb6d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00d620 tx=0x7fdc8c00dae0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.898+0000 7fdca9a00640 1 -- 172.21.15.62:0/3065737547 shutdown_connections 2024-03-23T11:20:41.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.898+0000 7fdca9a00640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc8006a7a0 0x7fdc8006cc60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.898+0000 7fdca9a00640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdca4158490 0x7fdca40cb6d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.898+0000 7fdca9a00640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdca40cbc10 0x7fdca409d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.898+0000 7fdca9a00640 1 --2- 172.21.15.62:0/3065737547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc88005180 0x7fdca415fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:41.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.898+0000 7fdca9a00640 1 -- 172.21.15.62:0/3065737547 >> 172.21.15.62:0/3065737547 conn(0x7fdca4089f60 msgr2=0x7fdca40b7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:41.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.898+0000 7fdca9a00640 1 -- 172.21.15.62:0/3065737547 shutdown_connections 2024-03-23T11:20:41.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:41.898+0000 7fdca9a00640 1 -- 172.21.15.62:0/3065737547 wait complete. 2024-03-23T11:20:41.918 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:20:41.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:41.918 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:20:42.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[50243]: fuse finished with error 0 and tester_r 0 2024-03-23T11:20:42.017 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:20:44.820 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:20:44.830 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:20:48.119 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:20:48.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:20:48.119 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:20:48.178 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:20:48.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.406+0000 7f0381000640 1 Processor -- start 2024-03-23T11:20:48.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.406+0000 7f0381000640 1 -- start start 2024-03-23T11:20:48.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.408+0000 7f0381000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c159490 0x7f037c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:48.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.408+0000 7f0381000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c0ced90 con 0x7f037c15da90 2024-03-23T11:20:48.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.408+0000 7f0381000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c06e3e0 con 0x7f037c0ce5f0 2024-03-23T11:20:48.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.408+0000 7f0381000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037c06c1c0 con 0x7f037c159490 2024-03-23T11:20:48.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.408+0000 7f037aa00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f037c15da90 0x7f037c0cdee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40110/0 (socket says 172.21.15.62:40110) 2024-03-23T11:20:48.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.408+0000 7f037aa00640 1 -- 172.21.15.62:0/3978471556 learned_addr learned my addr 172.21.15.62:0/3978471556 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:20:48.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.408+0000 7f037b400640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c159490 0x7f037c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:48.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.408+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123901631 0 0) 0x7f037c0ced90 con 0x7f037c15da90 2024-03-23T11:20:48.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.408+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0360003520 con 0x7f037c15da90 2024-03-23T11:20:48.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 380402583 0 0) 0x7f0360003520 con 0x7f037c15da90 2024-03-23T11:20:48.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c159490 msgr2=0x7f037c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:48.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c159490 0x7f037c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:48.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 >> v1:172.21.15.120:6789/0 conn(0x7f037c0ce5f0 legacy=0x7f037c0b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:48.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c06e100 con 0x7f037c15da90 2024-03-23T11:20:48.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f0368001670 con 0x7f037c15da90 2024-03-23T11:20:48.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 >> v1:172.21.15.62:6789/0 conn(0x7f037c15da90 legacy=0x7f037c0cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:48.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03600039a0 0x7f0360003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:48.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03600047c0 0x7f0360004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:48.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c159490 0x7f0360005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:48.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c06c1c0 con 0x7f03600039a0 2024-03-23T11:20:48.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036000bb10 con 0x7f03600047c0 2024-03-23T11:20:48.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036000bcd0 con 0x7f037c159490 2024-03-23T11:20:48.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.409+0000 7f037b400640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03600039a0 0x7f0360003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:48.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.410+0000 7f037aa00640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03600047c0 0x7f0360004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:48.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.410+0000 7f037b400640 1 -- 172.21.15.62:0/3978471556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c159490 msgr2=0x7f0360005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:48.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.410+0000 7f037b400640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c159490 0x7f0360005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:48.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.410+0000 7f037b400640 1 -- 172.21.15.62:0/3978471556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03600047c0 msgr2=0x7f0360004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:48.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.410+0000 7f037b400640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03600047c0 0x7f0360004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:48.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.410+0000 7f037b400640 1 -- 172.21.15.62:0/3978471556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0360003520 con 0x7f03600039a0 2024-03-23T11:20:48.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.410+0000 7f037aa00640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03600047c0 0x7f0360004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:48.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.411+0000 7f037b400640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03600039a0 0x7f0360003dc0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f036c009ac0 tx=0x7f03600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de8df05af72a532 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:48.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.411+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c02c710 con 0x7f03600039a0 2024-03-23T11:20:48.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.411+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036c02c870 con 0x7f03600039a0 2024-03-23T11:20:48.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.411+0000 7f037a000640 1 -- 172.21.15.62:0/3978471556 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c02cc00 con 0x7f03600039a0 2024-03-23T11:20:48.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.412+0000 7f0381000640 1 -- 172.21.15.62:0/3978471556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03600039a0 msgr2=0x7f0360003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:48.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.412+0000 7f0381000640 1 --2- 172.21.15.62:0/3978471556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03600039a0 0x7f0360003dc0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f036c009ac0 tx=0x7f03600057f0 comp rx=0 tx=0).stop 2024-03-23T11:20:48.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.412+0000 7f037be00640 1 -- 172.21.15.62:0/3978471556 reap_dead start 2024-03-23T11:20:48.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.412+0000 7f0381000640 1 -- 172.21.15.62:0/3978471556 shutdown_connections 2024-03-23T11:20:48.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.412+0000 7f0381000640 1 -- 172.21.15.62:0/3978471556 >> 172.21.15.62:0/3978471556 conn(0x7f037c066050 msgr2=0x7f037c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:48.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.412+0000 7f0381000640 1 -- 172.21.15.62:0/3978471556 shutdown_connections 2024-03-23T11:20:48.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.412+0000 7f0381000640 1 -- 172.21.15.62:0/3978471556 wait complete. 2024-03-23T11:20:48.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.414+0000 7f0381000640 1 Processor -- start 2024-03-23T11:20:48.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.414+0000 7f0381000640 1 -- start start 2024-03-23T11:20:48.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.415+0000 7f0381000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03600039a0 0x7f037c157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:48.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.415+0000 7f0381000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c14ee60 0x7f037c15afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:48.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.415+0000 7f0381000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f037c15b520 0x7f037c155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:48.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.415+0000 7f0381000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c15b9a0 con 0x7f037c15b520 2024-03-23T11:20:48.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.415+0000 7f0381000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c1564a0 con 0x7f03600039a0 2024-03-23T11:20:48.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.415+0000 7f0381000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037c156780 con 0x7f037c14ee60 2024-03-23T11:20:48.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.415+0000 7f037be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f037c15b520 0x7f037c155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:48.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.415+0000 7f037be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f037c15b520 0x7f037c155770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36740/0 (socket says 172.21.15.62:36740) 2024-03-23T11:20:48.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.415+0000 7f037be00640 1 -- 172.21.15.62:0/2756603712 learned_addr learned my addr 172.21.15.62:0/2756603712 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:48.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f037b400640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03600039a0 0x7f037c157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f037aa00640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c14ee60 0x7f037c15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f037be00640 1 -- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c14ee60 msgr2=0x7f037c15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f037be00640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c14ee60 0x7f037c15afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f037be00640 1 -- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03600039a0 msgr2=0x7f037c157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f037be00640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03600039a0 0x7f037c157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f037be00640 1 -- 172.21.15.62:0/2756603712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c156a60 con 0x7f037c15b520 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f037b400640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03600039a0 0x7f037c157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f037be00640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f037c15b520 0x7f037c155770 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f037c06dbb0 tx=0x7f037400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f0373400640 1 -- 172.21.15.62:0/2756603712 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0374005d40 con 0x7f037c15b520 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f0373400640 1 -- 172.21.15.62:0/2756603712 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037400ced0 con 0x7f037c15b520 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f0373400640 1 -- 172.21.15.62:0/2756603712 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037400a850 con 0x7f037c15b520 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.416+0000 7f0381000640 1 -- 172.21.15.62:0/2756603712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037c156d40 con 0x7f037c15b520 2024-03-23T11:20:48.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.417+0000 7f0381000640 1 -- 172.21.15.62:0/2756603712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037c14f2e0 con 0x7f037c15b520 2024-03-23T11:20:48.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.418+0000 7f0381000640 1 -- 172.21.15.62:0/2756603712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037c157260 con 0x7f037c15b520 2024-03-23T11:20:48.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.419+0000 7f0373400640 1 -- 172.21.15.62:0/2756603712 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f037400a9b0 con 0x7f037c15b520 2024-03-23T11:20:48.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.420+0000 7f0373400640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f035c06a7d0 0x7f035c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:48.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.420+0000 7f0373400640 1 -- 172.21.15.62:0/2756603712 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0374087cd0 con 0x7f037c15b520 2024-03-23T11:20:48.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.420+0000 7f037b400640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f035c06a7d0 0x7f035c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:48.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.422+0000 7f037b400640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f035c06a7d0 0x7f035c06cc90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f036c009ac0 tx=0x7f036c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.425+0000 7f0373400640 1 -- 172.21.15.62:0/2756603712 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f037c157260 con 0x7f037c15b520 2024-03-23T11:20:48.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.664+0000 7f0381000640 1 -- 172.21.15.62:0/2756603712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f037c0cb5b0 con 0x7f037c15b520 2024-03-23T11:20:48.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.665+0000 7f0373400640 1 -- 172.21.15.62:0/2756603712 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f0374053240 con 0x7f037c15b520 2024-03-23T11:20:48.665 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:48.665 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3421279564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3421279564},{"type":"v1","addr":"172.21.15.120:6835","nonce":3421279564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2052651236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2052651236},{"type":"v1","addr":"172.21.15.120:6837","nonce":2052651236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5193,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/1599029025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1599029025},{"type":"v1","addr":"172.21.15.62:6834","nonce":1599029025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:20:24.731603+0000","modified":"2024-03-23T11:20:28.249457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":5,"addr":"172.21.15.62:6835/3537214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":3537214366},{"type":"v1","addr":"172.21.15.62:6835","nonce":3537214366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T11:20:48.665 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-03-23T11:20:48.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.667+0000 7f0381000640 1 -- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f035c06a7d0 msgr2=0x7f035c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.667+0000 7f0381000640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f035c06a7d0 0x7f035c06cc90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f036c009ac0 tx=0x7f036c030000 comp rx=0 tx=0).stop 2024-03-23T11:20:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.667+0000 7f0381000640 1 -- 172.21.15.62:0/2756603712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f037c15b520 msgr2=0x7f037c155770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.667+0000 7f0381000640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f037c15b520 0x7f037c155770 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f037c06dbb0 tx=0x7f037400cb90 comp rx=0 tx=0).stop 2024-03-23T11:20:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.668+0000 7f0381000640 1 -- 172.21.15.62:0/2756603712 shutdown_connections 2024-03-23T11:20:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.668+0000 7f0381000640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f035c06a7d0 0x7f035c06cc90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.668+0000 7f0381000640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f037c15b520 0x7f037c155770 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.668+0000 7f0381000640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f037c14ee60 0x7f037c15afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.668+0000 7f0381000640 1 --2- 172.21.15.62:0/2756603712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03600039a0 0x7f037c157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.668+0000 7f0381000640 1 -- 172.21.15.62:0/2756603712 >> 172.21.15.62:0/2756603712 conn(0x7f037c066050 msgr2=0x7f037c0b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.668+0000 7f0381000640 1 -- 172.21.15.62:0/2756603712 shutdown_connections 2024-03-23T11:20:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:48.668+0000 7f0381000640 1 -- 172.21.15.62:0/2756603712 wait complete. 2024-03-23T11:20:48.687 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-23T11:20:48.946 INFO:teuthology.orchestra.run.smithi062.stdout:{"message":"","return_code":0} 2024-03-23T11:20:48.947 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-23T11:20:48.947 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.162+0000 7f1312c00640 1 Processor -- start 2024-03-23T11:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.162+0000 7f1312c00640 1 -- start start 2024-03-23T11:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.164+0000 7f1312c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f130c143dd0 0x7f130c15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.164+0000 7f1312c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f130c070090 0x7f130c070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.164+0000 7f1312c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f130c0b2820 0x7f130c0b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.164+0000 7f1312c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130c070a60 con 0x7f130c070090 2024-03-23T11:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.164+0000 7f1312c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130c0b31b0 con 0x7f130c143dd0 2024-03-23T11:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.164+0000 7f1312c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c0cd140 con 0x7f130c0b2820 2024-03-23T11:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.164+0000 7f1310e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f130c070090 0x7f130c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.164+0000 7f1310e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f130c070090 0x7f130c070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36746/0 (socket says 172.21.15.62:36746) 2024-03-23T11:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.164+0000 7f1310e00640 1 -- 172.21.15.62:0/4057941509 learned_addr learned my addr 172.21.15.62:0/4057941509 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.165+0000 7f1312200640 1 --2- 172.21.15.62:0/4057941509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f130c0b2820 0x7f130c0b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.165+0000 7f1311800640 1 --2- 172.21.15.62:0/4057941509 >> v2:172.21.15.120:3300/0 conn(0x7f130c143dd0 0x7f130c15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.165+0000 7f1310e00640 1 -- 172.21.15.62:0/4057941509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f130c0b2820 msgr2=0x7f130c0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.165+0000 7f1310e00640 1 --2- 172.21.15.62:0/4057941509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f130c0b2820 0x7f130c0b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.165+0000 7f1310e00640 1 -- 172.21.15.62:0/4057941509 >> v2:172.21.15.120:3300/0 conn(0x7f130c143dd0 msgr2=0x7f130c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.165+0000 7f1310e00640 1 --2- 172.21.15.62:0/4057941509 >> v2:172.21.15.120:3300/0 conn(0x7f130c143dd0 0x7f130c15abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.165+0000 7f1310e00640 1 -- 172.21.15.62:0/4057941509 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c0cd3e0 con 0x7f130c070090 2024-03-23T11:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.165+0000 7f1311800640 1 --2- 172.21.15.62:0/4057941509 >> v2:172.21.15.120:3300/0 conn(0x7f130c143dd0 0x7f130c15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.166+0000 7f1310e00640 1 --2- 172.21.15.62:0/4057941509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f130c070090 0x7f130c070520 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f130c06e380 tx=0x7f12f802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d49cd4c1b6491d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.166+0000 7f130be00640 1 -- 172.21.15.62:0/4057941509 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8031860 con 0x7f130c070090 2024-03-23T11:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.166+0000 7f130be00640 1 -- 172.21.15.62:0/4057941509 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f80319c0 con 0x7f130c070090 2024-03-23T11:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.166+0000 7f130be00640 1 -- 172.21.15.62:0/4057941509 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8031d50 con 0x7f130c070090 2024-03-23T11:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.167+0000 7f1312c00640 1 -- 172.21.15.62:0/4057941509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f130c070090 msgr2=0x7f130c070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.167+0000 7f1312c00640 1 --2- 172.21.15.62:0/4057941509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f130c070090 0x7f130c070520 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f130c06e380 tx=0x7f12f802b900 comp rx=0 tx=0).stop 2024-03-23T11:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.167+0000 7f1312c00640 1 -- 172.21.15.62:0/4057941509 shutdown_connections 2024-03-23T11:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.167+0000 7f1312c00640 1 --2- 172.21.15.62:0/4057941509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f130c0b2820 0x7f130c0b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.167+0000 7f1312c00640 1 --2- 172.21.15.62:0/4057941509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f130c070090 0x7f130c070520 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.167+0000 7f1312c00640 1 --2- 172.21.15.62:0/4057941509 >> v2:172.21.15.120:3300/0 conn(0x7f130c143dd0 0x7f130c15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.167+0000 7f1312c00640 1 -- 172.21.15.62:0/4057941509 >> 172.21.15.62:0/4057941509 conn(0x7f130c066050 msgr2=0x7f130c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:49.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.167+0000 7f1312c00640 1 -- 172.21.15.62:0/4057941509 shutdown_connections 2024-03-23T11:20:49.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.167+0000 7f1312c00640 1 -- 172.21.15.62:0/4057941509 wait complete. 2024-03-23T11:20:49.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.169+0000 7f1312c00640 1 Processor -- start 2024-03-23T11:20:49.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.169+0000 7f1312c00640 1 -- start start 2024-03-23T11:20:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.170+0000 7f1312c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f130c070090 0x7f130c1f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:49.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.170+0000 7f1312c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f130c0b2820 0x7f130c15baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:49.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.170+0000 7f1312c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f130c143dd0 0x7f130c15c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:49.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.170+0000 7f1312c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c1f2e50 con 0x7f130c0b2820 2024-03-23T11:20:49.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.170+0000 7f1312c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c12f520 con 0x7f130c143dd0 2024-03-23T11:20:49.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.170+0000 7f1312c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c12f800 con 0x7f130c070090 2024-03-23T11:20:49.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.170+0000 7f1310e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f130c0b2820 0x7f130c15baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:49.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.170+0000 7f1310e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f130c0b2820 0x7f130c15baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36756/0 (socket says 172.21.15.62:36756) 2024-03-23T11:20:49.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.170+0000 7f1310e00640 1 -- 172.21.15.62:0/48242137 learned_addr learned my addr 172.21.15.62:0/48242137 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:49.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.171+0000 7f1311800640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f130c070090 0x7f130c1f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:49.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.171+0000 7f1312200640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f130c143dd0 0x7f130c15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:49.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.171+0000 7f1310e00640 1 -- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f130c070090 msgr2=0x7f130c1f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:49.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.171+0000 7f1310e00640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f130c070090 0x7f130c1f2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.171+0000 7f1310e00640 1 -- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f130c143dd0 msgr2=0x7f130c15c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:49.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.171+0000 7f1310e00640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f130c143dd0 0x7f130c15c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.171+0000 7f1310e00640 1 -- 172.21.15.62:0/48242137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f80094d0 con 0x7f130c0b2820 2024-03-23T11:20:49.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.171+0000 7f1312200640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f130c143dd0 0x7f130c15c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:49.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.171+0000 7f1310e00640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f130c0b2820 0x7f130c15baf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f12f802b940 tx=0x7f12f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:49.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.172+0000 7f1309600640 1 -- 172.21.15.62:0/48242137 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f803ad30 con 0x7f130c0b2820 2024-03-23T11:20:49.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.172+0000 7f1309600640 1 -- 172.21.15.62:0/48242137 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f80443f0 con 0x7f130c0b2820 2024-03-23T11:20:49.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.172+0000 7f1309600640 1 -- 172.21.15.62:0/48242137 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f80447a0 con 0x7f130c0b2820 2024-03-23T11:20:49.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.172+0000 7f1312c00640 1 -- 172.21.15.62:0/48242137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c12fae0 con 0x7f130c0b2820 2024-03-23T11:20:49.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.172+0000 7f1312c00640 1 -- 172.21.15.62:0/48242137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c1300a0 con 0x7f130c0b2820 2024-03-23T11:20:49.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.173+0000 7f1312c00640 1 -- 172.21.15.62:0/48242137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c130290 con 0x7f130c0b2820 2024-03-23T11:20:49.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.174+0000 7f1309600640 1 -- 172.21.15.62:0/48242137 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f12f804d450 con 0x7f130c0b2820 2024-03-23T11:20:49.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.175+0000 7f1309600640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12ec06a820 0x7f12ec06cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:49.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.175+0000 7f1309600640 1 -- 172.21.15.62:0/48242137 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12f80ab390 con 0x7f130c0b2820 2024-03-23T11:20:49.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.176+0000 7f1311800640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12ec06a820 0x7f12ec06cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:49.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.177+0000 7f1311800640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12ec06a820 0x7f12ec06cce0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f130000a910 tx=0x7f1300008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:49.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.180+0000 7f1309600640 1 -- 172.21.15.62:0/48242137 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f12f8076790 con 0x7f130c0b2820 2024-03-23T11:20:49.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.424+0000 7f1312c00640 1 -- 172.21.15.62:0/48242137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f130c12f940 con 0x7f130c0b2820 2024-03-23T11:20:49.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.425+0000 7f1309600640 1 -- 172.21.15.62:0/48242137 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f130c12f940 con 0x7f130c0b2820 2024-03-23T11:20:49.425 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:20:49.425 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3421279564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3421279564},{"type":"v1","addr":"172.21.15.120:6835","nonce":3421279564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2052651236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2052651236},{"type":"v1","addr":"172.21.15.120:6837","nonce":2052651236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5193,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/1599029025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1599029025},{"type":"v1","addr":"172.21.15.62:6834","nonce":1599029025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:20:24.731603+0000","modified":"2024-03-23T11:20:28.249457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":5,"addr":"172.21.15.62:6835/3537214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":3537214366},{"type":"v1","addr":"172.21.15.62:6835","nonce":3537214366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T11:20:49.425 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-03-23T11:20:49.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 -- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12ec06a820 msgr2=0x7f12ec06cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:49.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12ec06a820 0x7f12ec06cce0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f130000a910 tx=0x7f1300008040 comp rx=0 tx=0).stop 2024-03-23T11:20:49.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 -- 172.21.15.62:0/48242137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f130c0b2820 msgr2=0x7f130c15baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:49.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f130c0b2820 0x7f130c15baf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f12f802b940 tx=0x7f12f80046c0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 -- 172.21.15.62:0/48242137 shutdown_connections 2024-03-23T11:20:49.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12ec06a820 0x7f12ec06cce0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f130c0b2820 0x7f130c15baf0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f130c143dd0 0x7f130c15c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 --2- 172.21.15.62:0/48242137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f130c070090 0x7f130c1f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:49.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 -- 172.21.15.62:0/48242137 >> 172.21.15.62:0/48242137 conn(0x7f130c066050 msgr2=0x7f130c0ae6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:49.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 -- 172.21.15.62:0/48242137 shutdown_connections 2024-03-23T11:20:49.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:49.429+0000 7f1312c00640 1 -- 172.21.15.62:0/48242137 wait complete. 2024-03-23T11:20:49.448 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json tag path /parentdir mytag 2024-03-23T11:20:49.674 INFO:teuthology.orchestra.run.smithi062.stdout:{"return_code":0,"scrub_tag":"mytag","mode":"asynchronous"} 2024-03-23T11:20:49.674 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag" } 2024-03-23T11:20:59.675 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000000.00000000 scrub_tag 2024-03-23T11:20:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.792+0000 7f8df3837f40 1 Processor -- start 2024-03-23T11:20:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.793+0000 7f8df3837f40 1 -- start start 2024-03-23T11:20:59.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.793+0000 7f8df3837f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555aad980320 0x555aad9c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.793+0000 7f8df3837f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aad9c81c0 0x555aada4a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.793+0000 7f8df3837f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555aada55080 0x555aada4a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8df3837f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aad8c8800 con 0x555aad9c81c0 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8df3837f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aad8c84a0 con 0x555aada55080 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8df3837f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555aad8bf110 con 0x555aad980320 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8de7a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aad9c81c0 0x555aada4a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8de7a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aad9c81c0 0x555aada4a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46256/0 (socket says 172.21.15.62:46256) 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8de7a00640 1 -- 172.21.15.62:0/3485185802 learned_addr learned my addr 172.21.15.62:0/3485185802 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8df0400640 1 --2- 172.21.15.62:0/3485185802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555aad980320 0x555aad9c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8df0e00640 1 --2- 172.21.15.62:0/3485185802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555aada55080 0x555aada4a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8de7a00640 1 -- 172.21.15.62:0/3485185802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555aad980320 msgr2=0x555aad9c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8de7a00640 1 --2- 172.21.15.62:0/3485185802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555aad980320 0x555aad9c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8de7a00640 1 -- 172.21.15.62:0/3485185802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555aada55080 msgr2=0x555aada4a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:59.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8de7a00640 1 --2- 172.21.15.62:0/3485185802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555aada55080 0x555aada4a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.794+0000 7f8de7a00640 1 -- 172.21.15.62:0/3485185802 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd8005de0 con 0x555aad9c81c0 2024-03-23T11:20:59.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.795+0000 7f8df0400640 1 --2- 172.21.15.62:0/3485185802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555aad980320 0x555aad9c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:59.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.795+0000 7f8df0e00640 1 --2- 172.21.15.62:0/3485185802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555aada55080 0x555aada4a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.795+0000 7f8de7a00640 1 --2- 172.21.15.62:0/3485185802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aad9c81c0 0x555aada4a260 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8055040 tx=0x7f8dd804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d745a57ae02418 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.795+0000 7f8def000640 1 -- 172.21.15.62:0/3485185802 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd80527b0 con 0x555aad9c81c0 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.795+0000 7f8def000640 1 -- 172.21.15.62:0/3485185802 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd8052930 con 0x555aad9c81c0 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.795+0000 7f8def000640 1 -- 172.21.15.62:0/3485185802 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8052ce0 con 0x555aad9c81c0 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.796+0000 7f8df3837f40 1 -- 172.21.15.62:0/3485185802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aad9c81c0 msgr2=0x555aada4a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.796+0000 7f8df3837f40 1 --2- 172.21.15.62:0/3485185802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aad9c81c0 0x555aada4a260 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8055040 tx=0x7f8dd804c000 comp rx=0 tx=0).stop 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.796+0000 7f8df3837f40 1 -- 172.21.15.62:0/3485185802 shutdown_connections 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.796+0000 7f8df3837f40 1 --2- 172.21.15.62:0/3485185802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aad9c81c0 0x555aada4a260 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.796+0000 7f8df3837f40 1 --2- 172.21.15.62:0/3485185802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555aada55080 0x555aada4a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.796+0000 7f8df3837f40 1 --2- 172.21.15.62:0/3485185802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555aad980320 0x555aad9c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.796+0000 7f8df3837f40 1 -- 172.21.15.62:0/3485185802 >> 172.21.15.62:0/3485185802 conn(0x555aad9abae0 msgr2=0x555aad9abec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.796+0000 7f8df3837f40 1 -- 172.21.15.62:0/3485185802 shutdown_connections 2024-03-23T11:20:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.796+0000 7f8df3837f40 1 -- 172.21.15.62:0/3485185802 wait complete. 2024-03-23T11:20:59.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.797+0000 7f8df3837f40 1 Processor -- start 2024-03-23T11:20:59.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.797+0000 7f8df3837f40 1 -- start start 2024-03-23T11:20:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.798+0000 7f8df3837f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555aad980320 0x555aada4c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.798+0000 7f8df3837f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555aad9c81c0 0x555aada4f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.798+0000 7f8df3837f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aada55080 0x555aada4fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.798+0000 7f8df3837f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aada4b000 con 0x555aada55080 2024-03-23T11:20:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.798+0000 7f8df3837f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555aada47530 con 0x555aad9c81c0 2024-03-23T11:20:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.798+0000 7f8df3837f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555aad982660 con 0x555aad980320 2024-03-23T11:20:59.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.799+0000 7f8df0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aada55080 0x555aada4fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.799+0000 7f8df0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aada55080 0x555aada4fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46262/0 (socket says 172.21.15.62:46262) 2024-03-23T11:20:59.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.799+0000 7f8df0e00640 1 -- 172.21.15.62:0/1064476393 learned_addr learned my addr 172.21.15.62:0/1064476393 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:59.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.799+0000 7f8df0400640 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555aad980320 0x555aada4c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.799+0000 7f8de7a00640 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555aad9c81c0 0x555aada4f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.799+0000 7f8df0e00640 1 -- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555aad980320 msgr2=0x555aada4c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.799+0000 7f8df0e00640 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555aad980320 0x555aada4c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.799+0000 7f8df0e00640 1 -- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555aad9c81c0 msgr2=0x555aada4f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.799+0000 7f8df0e00640 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555aad9c81c0 0x555aada4f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.799+0000 7f8df0e00640 1 -- 172.21.15.62:0/1064476393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555aad8f14e0 con 0x555aada55080 2024-03-23T11:20:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.799+0000 7f8df0400640 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555aad980320 0x555aada4c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:20:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.800+0000 7f8df0e00640 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aada55080 0x555aada4fc90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8de8002210 tx=0x7f8de800c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.800+0000 7f8dedc00640 1 -- 172.21.15.62:0/1064476393 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8003650 con 0x555aada55080 2024-03-23T11:20:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.800+0000 7f8dedc00640 1 -- 172.21.15.62:0/1064476393 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de80037b0 con 0x555aada55080 2024-03-23T11:20:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.800+0000 7f8dedc00640 1 -- 172.21.15.62:0/1064476393 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8003b40 con 0x555aada55080 2024-03-23T11:20:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.800+0000 7f8df3837f40 1 -- 172.21.15.62:0/1064476393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555aad8d7340 con 0x555aada55080 2024-03-23T11:20:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.800+0000 7f8df3837f40 1 -- 172.21.15.62:0/1064476393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555aad8ca780 con 0x555aada55080 2024-03-23T11:20:59.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.802+0000 7f8dedc00640 1 -- 172.21.15.62:0/1064476393 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8de8003ca0 con 0x555aada55080 2024-03-23T11:20:59.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.803+0000 7f8dedc00640 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8dcc069180 0x7f8dcc06b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.803+0000 7f8dedc00640 1 -- 172.21.15.62:0/1064476393 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8de8087900 con 0x555aada55080 2024-03-23T11:20:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.804+0000 7f8df3837f40 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x555aada46b00 0x555aad9c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.804+0000 7f8df3837f40 1 -- 172.21.15.62:0/1064476393 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x555aad9c5e80 con 0x555aada46b00 2024-03-23T11:20:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.804+0000 7f8df0400640 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8dcc069180 0x7f8dcc06b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.804+0000 7f8de7a00640 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x555aada46b00 0x555aad9c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.805+0000 7f8de7a00640 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x555aada46b00 0x555aad9c5940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:59.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.805+0000 7f8df0400640 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8dcc069180 0x7f8dcc06b640 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8de0005fd0 tx=0x7f8de0005e00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:59.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.805+0000 7f8de7a00640 1 -- 172.21.15.62:0/1064476393 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (9) out=9b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f8dd8062020 con 0x555aada46b00 2024-03-23T11:20:59.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.805+0000 7f8df3837f40 1 -- 172.21.15.62:0/1064476393 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x555aada46b00 msgr2=0x555aad9c5940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.805+0000 7f8df3837f40 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x555aada46b00 0x555aad9c5940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.805+0000 7f8df3837f40 1 -- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8dcc069180 msgr2=0x7f8dcc06b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.805+0000 7f8df3837f40 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8dcc069180 0x7f8dcc06b640 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8de0005fd0 tx=0x7f8de0005e00 comp rx=0 tx=0).stop 2024-03-23T11:20:59.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.805+0000 7f8df3837f40 1 -- 172.21.15.62:0/1064476393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aada55080 msgr2=0x555aada4fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.805+0000 7f8df3837f40 1 --2- 172.21.15.62:0/1064476393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555aada55080 0x555aada4fc90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8de8002210 tx=0x7f8de800c5e0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.805+0000 7f8df0e00640 1 -- 172.21.15.62:0/1064476393 reap_dead start 2024-03-23T11:20:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.806+0000 7f8df3837f40 1 -- 172.21.15.62:0/1064476393 shutdown_connections 2024-03-23T11:20:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.806+0000 7f8df3837f40 1 -- 172.21.15.62:0/1064476393 >> 172.21.15.62:0/1064476393 conn(0x555aad9abae0 msgr2=0x555aad9abec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.806+0000 7f8df3837f40 1 -- 172.21.15.62:0/1064476393 shutdown_connections 2024-03-23T11:20:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.806+0000 7f8df3837f40 1 -- 172.21.15.62:0/1064476393 wait complete. 2024-03-23T11:20:59.810 INFO:teuthology.orchestra.run.smithi062.stdout:mytag 2024-03-23T11:20:59.810 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000001.00000000 scrub_tag 2024-03-23T11:20:59.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.920+0000 7f45c59fef40 1 Processor -- start 2024-03-23T11:20:59.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.920+0000 7f45c59fef40 1 -- start start 2024-03-23T11:20:59.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.920+0000 7f45c59fef40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45a8002260 0x55d7c8ce7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.920+0000 7f45c59fef40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8c49930 0x55d7c8c4b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.920+0000 7f45c59fef40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7c8cedc40 0x55d7c8c4ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.920+0000 7f45c59fef40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c8bc6c20 con 0x55d7c8c49930 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.921+0000 7f45c59fef40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c8b62800 con 0x55d7c8cedc40 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.921+0000 7f45c59fef40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d7c8b624a0 con 0x7f45a8002260 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.921+0000 7f45bbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8c49930 0x55d7c8c4b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.921+0000 7f45bbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8c49930 0x55d7c8c4b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46274/0 (socket says 172.21.15.62:46274) 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.921+0000 7f45bbe00640 1 -- 172.21.15.62:0/1373996216 learned_addr learned my addr 172.21.15.62:0/1373996216 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.921+0000 7f45bbe00640 1 -- 172.21.15.62:0/1373996216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45a8002260 msgr2=0x55d7c8ce7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.921+0000 7f45bbe00640 1 --2- 172.21.15.62:0/1373996216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45a8002260 0x55d7c8ce7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.921+0000 7f45bbe00640 1 -- 172.21.15.62:0/1373996216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7c8cedc40 msgr2=0x55d7c8c4ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.921+0000 7f45bbe00640 1 --2- 172.21.15.62:0/1373996216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7c8cedc40 0x55d7c8c4ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.921+0000 7f45bbe00640 1 -- 172.21.15.62:0/1373996216 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7c8b64780 con 0x55d7c8c49930 2024-03-23T11:20:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45bbe00640 1 --2- 172.21.15.62:0/1373996216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8c49930 0x55d7c8c4b520 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0097c0 tx=0x7f45ac0098b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=589c1e6480af980a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:59.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c1800640 1 -- 172.21.15.62:0/1373996216 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac0088d0 con 0x55d7c8c49930 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c1800640 1 -- 172.21.15.62:0/1373996216 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac010040 con 0x55d7c8c49930 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c1800640 1 -- 172.21.15.62:0/1373996216 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac008c70 con 0x55d7c8c49930 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c59fef40 1 -- 172.21.15.62:0/1373996216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8c49930 msgr2=0x55d7c8c4b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c59fef40 1 --2- 172.21.15.62:0/1373996216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8c49930 0x55d7c8c4b520 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0097c0 tx=0x7f45ac0098b0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c59fef40 1 -- 172.21.15.62:0/1373996216 shutdown_connections 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c59fef40 1 --2- 172.21.15.62:0/1373996216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8c49930 0x55d7c8c4b520 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c59fef40 1 --2- 172.21.15.62:0/1373996216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7c8cedc40 0x55d7c8c4ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c59fef40 1 --2- 172.21.15.62:0/1373996216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45a8002260 0x55d7c8ce7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c59fef40 1 -- 172.21.15.62:0/1373996216 >> 172.21.15.62:0/1373996216 conn(0x55d7c8c67ff0 msgr2=0x55d7c8cd4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c59fef40 1 -- 172.21.15.62:0/1373996216 shutdown_connections 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.922+0000 7f45c59fef40 1 -- 172.21.15.62:0/1373996216 wait complete. 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.923+0000 7f45c59fef40 1 Processor -- start 2024-03-23T11:20:59.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.923+0000 7f45c59fef40 1 -- start start 2024-03-23T11:20:59.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.924+0000 7f45c59fef40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d7c8cedc40 0x55d7c8d8aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.924+0000 7f45c59fef40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45a8002260 0x55d7c8cec390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.924+0000 7f45c59fef40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8cf11c0 0x55d7c8ceb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.924+0000 7f45c59fef40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c8b59110 con 0x55d7c8cf11c0 2024-03-23T11:20:59.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.924+0000 7f45c59fef40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7c8d85430 con 0x7f45a8002260 2024-03-23T11:20:59.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.924+0000 7f45c59fef40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d7c8c4ca00 con 0x55d7c8cedc40 2024-03-23T11:20:59.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.925+0000 7f45c3600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8cf11c0 0x55d7c8ceb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.925+0000 7f45c3600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8cf11c0 0x55d7c8ceb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46280/0 (socket says 172.21.15.62:46280) 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.925+0000 7f45c3600640 1 -- 172.21.15.62:0/421334615 learned_addr learned my addr 172.21.15.62:0/421334615 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.925+0000 7f45bbe00640 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45a8002260 0x55d7c8cec390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.925+0000 7f45c2c00640 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d7c8cedc40 0x55d7c8d8aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.925+0000 7f45c3600640 1 -- 172.21.15.62:0/421334615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d7c8cedc40 msgr2=0x55d7c8d8aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.925+0000 7f45c3600640 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d7c8cedc40 0x55d7c8d8aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.925+0000 7f45c3600640 1 -- 172.21.15.62:0/421334615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45a8002260 msgr2=0x55d7c8cec390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.925+0000 7f45c3600640 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45a8002260 0x55d7c8cec390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.925+0000 7f45c3600640 1 -- 172.21.15.62:0/421334615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac009410 con 0x55d7c8cf11c0 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.926+0000 7f45c3600640 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8cf11c0 0x55d7c8ceb000 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x55d7c8c4adf0 tx=0x7f45bc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.926+0000 7f45bb400640 1 -- 172.21.15.62:0/421334615 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45bc00a070 con 0x55d7c8cf11c0 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.926+0000 7f45bb400640 1 -- 172.21.15.62:0/421334615 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45bc00cdc0 con 0x55d7c8cf11c0 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.926+0000 7f45bb400640 1 -- 172.21.15.62:0/421334615 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45bc003310 con 0x55d7c8cf11c0 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.926+0000 7f45c59fef40 1 -- 172.21.15.62:0/421334615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d7c8b641e0 con 0x55d7c8cf11c0 2024-03-23T11:20:59.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.926+0000 7f45c59fef40 1 -- 172.21.15.62:0/421334615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d7c8ceab70 con 0x55d7c8cf11c0 2024-03-23T11:20:59.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.928+0000 7f45bb400640 1 -- 172.21.15.62:0/421334615 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f45bc001610 con 0x55d7c8cf11c0 2024-03-23T11:20:59.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.929+0000 7f45bb400640 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45a0069180 0x7f45a006b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.929+0000 7f45bb400640 1 -- 172.21.15.62:0/421334615 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45bc086e70 con 0x55d7c8cf11c0 2024-03-23T11:20:59.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.929+0000 7f45c59fef40 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55d7c8cec8d0 0x55d7c8d8e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:20:59.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.929+0000 7f45c59fef40 1 -- 172.21.15.62:0/421334615 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:1 4.7 4:e2e07930:::10000000001.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55d7c8d84950 con 0x55d7c8cec8d0 2024-03-23T11:20:59.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.930+0000 7f45bbe00640 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55d7c8cec8d0 0x55d7c8d8e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.930+0000 7f45c2c00640 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45a0069180 0x7f45a006b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:20:59.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.930+0000 7f45bbe00640 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55d7c8cec8d0 0x55d7c8d8e4f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:20:59.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45bbe00640 1 -- 172.21.15.62:0/421334615 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (9) out=9b] v0'0 uv5 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f45ac01a020 con 0x55d7c8cec8d0 2024-03-23T11:20:59.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45c59fef40 1 -- 172.21.15.62:0/421334615 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55d7c8cec8d0 msgr2=0x55d7c8d8e4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45c59fef40 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55d7c8cec8d0 0x55d7c8d8e4f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:20:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45c59fef40 1 -- 172.21.15.62:0/421334615 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45a0069180 msgr2=0x7f45a006b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45c59fef40 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45a0069180 0x7f45a006b640 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f45b4054000 tx=0x7f45b404d000 comp rx=0 tx=0).stop 2024-03-23T11:20:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45c59fef40 1 -- 172.21.15.62:0/421334615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8cf11c0 msgr2=0x55d7c8ceb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:20:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45c59fef40 1 --2- 172.21.15.62:0/421334615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7c8cf11c0 0x55d7c8ceb000 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x55d7c8c4adf0 tx=0x7f45bc00c520 comp rx=0 tx=0).stop 2024-03-23T11:20:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45c3600640 1 -- 172.21.15.62:0/421334615 reap_dead start 2024-03-23T11:20:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45c59fef40 1 -- 172.21.15.62:0/421334615 shutdown_connections 2024-03-23T11:20:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45c59fef40 1 -- 172.21.15.62:0/421334615 >> 172.21.15.62:0/421334615 conn(0x55d7c8c67ff0 msgr2=0x55d7c8ce7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:20:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45c59fef40 1 -- 172.21.15.62:0/421334615 shutdown_connections 2024-03-23T11:20:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:20:59.931+0000 7f45c59fef40 1 -- 172.21.15.62:0/421334615 wait complete. 2024-03-23T11:20:59.936 INFO:teuthology.orchestra.run.smithi062.stdout:mytag 2024-03-23T11:20:59.936 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-03-23T11:21:00.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.030+0000 7f11f2e46f40 1 Processor -- start 2024-03-23T11:21:00.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.031+0000 7f11f2e46f40 1 -- start start 2024-03-23T11:21:00.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.031+0000 7f11f2e46f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560da75f3a20 0x560da76cac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f2e46f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560da75f4980 0x560da76b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f2e46f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da76cced0 0x560da76b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f2e46f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da7539800 con 0x560da76cced0 2024-03-23T11:21:00.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f2e46f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da75394a0 con 0x560da75f4980 2024-03-23T11:21:00.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f2e46f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560da7530110 con 0x560da75f3a20 2024-03-23T11:21:00.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f0400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da76cced0 0x560da76b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f0400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da76cced0 0x560da76b4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46282/0 (socket says 172.21.15.62:46282) 2024-03-23T11:21:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f0400640 1 -- 172.21.15.62:0/2238574191 learned_addr learned my addr 172.21.15.62:0/2238574191 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11ef000640 1 --2- 172.21.15.62:0/2238574191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560da75f4980 0x560da76b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11efa00640 1 --2- 172.21.15.62:0/2238574191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560da75f3a20 0x560da76cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f0400640 1 -- 172.21.15.62:0/2238574191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560da75f3a20 msgr2=0x560da76cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f0400640 1 --2- 172.21.15.62:0/2238574191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560da75f3a20 0x560da76cac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f0400640 1 -- 172.21.15.62:0/2238574191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560da75f4980 msgr2=0x560da76b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.032+0000 7f11f0400640 1 --2- 172.21.15.62:0/2238574191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560da75f4980 0x560da76b04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.033+0000 7f11f0400640 1 -- 172.21.15.62:0/2238574191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e000a100 con 0x560da76cced0 2024-03-23T11:21:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.033+0000 7f11f0400640 1 --2- 172.21.15.62:0/2238574191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da76cced0 0x560da76b4060 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f11e8001e40 tx=0x7f11e8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4369075e3cec04 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.033+0000 7f11edc00640 1 -- 172.21.15.62:0/2238574191 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e800d6b0 con 0x560da76cced0 2024-03-23T11:21:00.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.033+0000 7f11edc00640 1 -- 172.21.15.62:0/2238574191 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e800a040 con 0x560da76cced0 2024-03-23T11:21:00.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.033+0000 7f11edc00640 1 -- 172.21.15.62:0/2238574191 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e800da70 con 0x560da76cced0 2024-03-23T11:21:00.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.034+0000 7f11f2e46f40 1 -- 172.21.15.62:0/2238574191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da76cced0 msgr2=0x560da76b4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.034+0000 7f11f2e46f40 1 --2- 172.21.15.62:0/2238574191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da76cced0 0x560da76b4060 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f11e8001e40 tx=0x7f11e8001e70 comp rx=0 tx=0).stop 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.034+0000 7f11f2e46f40 1 -- 172.21.15.62:0/2238574191 shutdown_connections 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.034+0000 7f11f2e46f40 1 --2- 172.21.15.62:0/2238574191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da76cced0 0x560da76b4060 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.034+0000 7f11f2e46f40 1 --2- 172.21.15.62:0/2238574191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560da75f4980 0x560da76b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.034+0000 7f11f2e46f40 1 --2- 172.21.15.62:0/2238574191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560da75f3a20 0x560da76cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.034+0000 7f11f2e46f40 1 -- 172.21.15.62:0/2238574191 >> 172.21.15.62:0/2238574191 conn(0x560da75ef6f0 msgr2=0x560da75f0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.034+0000 7f11f2e46f40 1 -- 172.21.15.62:0/2238574191 shutdown_connections 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.034+0000 7f11f2e46f40 1 -- 172.21.15.62:0/2238574191 wait complete. 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.035+0000 7f11f2e46f40 1 Processor -- start 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.035+0000 7f11f2e46f40 1 -- start start 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.036+0000 7f11f2e46f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da75f3a20 0x560da76aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.036+0000 7f11f2e46f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560da75f4980 0x560da76c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.036+0000 7f11f2e46f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560da76cced0 0x560da76ac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.036+0000 7f11f2e46f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da76cac70 con 0x560da75f3a20 2024-03-23T11:21:00.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.036+0000 7f11f2e46f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da763c510 con 0x560da75f4980 2024-03-23T11:21:00.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.036+0000 7f11f2e46f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560da76cb380 con 0x560da76cced0 2024-03-23T11:21:00.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.036+0000 7f11efa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da75f3a20 0x560da76aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.036+0000 7f11efa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da75f3a20 0x560da76aabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46286/0 (socket says 172.21.15.62:46286) 2024-03-23T11:21:00.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.036+0000 7f11efa00640 1 -- 172.21.15.62:0/3299355890 learned_addr learned my addr 172.21.15.62:0/3299355890 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:00.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11efa00640 1 -- 172.21.15.62:0/3299355890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560da76cced0 msgr2=0x560da76ac9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11efa00640 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560da76cced0 0x560da76ac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11efa00640 1 -- 172.21.15.62:0/3299355890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560da75f4980 msgr2=0x560da76c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11ef000640 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560da75f4980 0x560da76c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11efa00640 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560da75f4980 0x560da76c46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11efa00640 1 -- 172.21.15.62:0/3299355890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e8001a10 con 0x560da75f3a20 2024-03-23T11:21:00.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11ef000640 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560da75f4980 0x560da76c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:00.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11efa00640 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da75f3a20 0x560da76aabe0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f11e000e300 tx=0x7f11e000e7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11e7e00640 1 -- 172.21.15.62:0/3299355890 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0013930 con 0x560da75f3a20 2024-03-23T11:21:00.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11e7e00640 1 -- 172.21.15.62:0/3299355890 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e0013a90 con 0x560da75f3a20 2024-03-23T11:21:00.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11e7e00640 1 -- 172.21.15.62:0/3299355890 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0013e40 con 0x560da75f3a20 2024-03-23T11:21:00.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.037+0000 7f11f2e46f40 1 -- 172.21.15.62:0/3299355890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d8005de0 con 0x560da75f3a20 2024-03-23T11:21:00.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.038+0000 7f11f2e46f40 1 -- 172.21.15.62:0/3299355890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560da7548340 con 0x560da75f3a20 2024-03-23T11:21:00.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.039+0000 7f11e7e00640 1 -- 172.21.15.62:0/3299355890 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f11e0014410 con 0x560da75f3a20 2024-03-23T11:21:00.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.040+0000 7f11e7e00640 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f11cc069180 0x7f11cc06b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.040+0000 7f11e7e00640 1 -- 172.21.15.62:0/3299355890 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11e0088310 con 0x560da75f3a20 2024-03-23T11:21:00.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.041+0000 7f11f2e46f40 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x560da76b4670 0x560da76b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.041+0000 7f11f2e46f40 1 -- 172.21.15.62:0/3299355890 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:1 5.36 5:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x560da76b3690 con 0x560da76b4670 2024-03-23T11:21:00.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.041+0000 7f11ef000640 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f11cc069180 0x7f11cc06b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.041+0000 7f11f0400640 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x560da76b4670 0x560da76b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.042+0000 7f11f0400640 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x560da76b4670 0x560da76b3150 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.042+0000 7f11ef000640 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f11cc069180 0x7f11cc06b640 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f11d8057000 tx=0x7f11d8052000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.042+0000 7f11f0400640 1 -- 172.21.15.62:0/3299355890 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr (9) out=9b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f11e8018040 con 0x560da76b4670 2024-03-23T11:21:00.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.042+0000 7f11f2e46f40 1 -- 172.21.15.62:0/3299355890 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x560da76b4670 msgr2=0x560da76b3150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.042+0000 7f11f2e46f40 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x560da76b4670 0x560da76b3150 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.042+0000 7f11f2e46f40 1 -- 172.21.15.62:0/3299355890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f11cc069180 msgr2=0x7f11cc06b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.042+0000 7f11f2e46f40 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f11cc069180 0x7f11cc06b640 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f11d8057000 tx=0x7f11d8052000 comp rx=0 tx=0).stop 2024-03-23T11:21:00.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.042+0000 7f11f2e46f40 1 -- 172.21.15.62:0/3299355890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da75f3a20 msgr2=0x560da76aabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.042+0000 7f11f2e46f40 1 --2- 172.21.15.62:0/3299355890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560da75f3a20 0x560da76aabe0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f11e000e300 tx=0x7f11e000e7c0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.042+0000 7f11f0400640 1 -- 172.21.15.62:0/3299355890 reap_dead start 2024-03-23T11:21:00.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.043+0000 7f11f2e46f40 1 -- 172.21.15.62:0/3299355890 shutdown_connections 2024-03-23T11:21:00.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.043+0000 7f11f2e46f40 1 -- 172.21.15.62:0/3299355890 >> 172.21.15.62:0/3299355890 conn(0x560da75ef6f0 msgr2=0x560da76cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:00.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.043+0000 7f11f2e46f40 1 -- 172.21.15.62:0/3299355890 shutdown_connections 2024-03-23T11:21:00.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.043+0000 7f11f2e46f40 1 -- 172.21.15.62:0/3299355890 wait complete. 2024-03-23T11:21:00.046 INFO:teuthology.orchestra.run.smithi062.stdout:mytag 2024-03-23T11:21:00.046 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-03-23T11:21:00.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.152+0000 7fbbe004af40 1 Processor -- start 2024-03-23T11:21:00.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.153+0000 7fbbe004af40 1 -- start start 2024-03-23T11:21:00.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.153+0000 7fbbe004af40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c85e2910 0x55d6c8669be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.153+0000 7fbbe004af40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d6c8686c20 0x55d6c86850e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.153+0000 7fbbe004af40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 0x55d6c8685620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.153+0000 7fbbe004af40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6c84f7800 con 0x55d6c8687540 2024-03-23T11:21:00.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.153+0000 7fbbe004af40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6c84f74a0 con 0x55d6c85e2910 2024-03-23T11:21:00.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.153+0000 7fbbe004af40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d6c84ee110 con 0x55d6c8686c20 2024-03-23T11:21:00.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.154+0000 7fbbdd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 0x55d6c8685620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.154+0000 7fbbdd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 0x55d6c8685620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46306/0 (socket says 172.21.15.62:46306) 2024-03-23T11:21:00.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.154+0000 7fbbdd600640 1 -- 172.21.15.62:0/840376746 learned_addr learned my addr 172.21.15.62:0/840376746 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:00.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.154+0000 7fbbdcc00640 1 --2- 172.21.15.62:0/840376746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c85e2910 0x55d6c8669be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.154+0000 7fbbdd600640 1 -- 172.21.15.62:0/840376746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d6c8686c20 msgr2=0x55d6c86850e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:00.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.154+0000 7fbbd4200640 1 --2- 172.21.15.62:0/840376746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d6c8686c20 0x55d6c86850e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.154+0000 7fbbdd600640 1 --2- 172.21.15.62:0/840376746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d6c8686c20 0x55d6c86850e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.154+0000 7fbbdd600640 1 -- 172.21.15.62:0/840376746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c85e2910 msgr2=0x55d6c8669be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:00.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.154+0000 7fbbdd600640 1 --2- 172.21.15.62:0/840376746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c85e2910 0x55d6c8669be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.154+0000 7fbbdd600640 1 -- 172.21.15.62:0/840376746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbcc005de0 con 0x55d6c8687540 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.155+0000 7fbbdd600640 1 --2- 172.21.15.62:0/840376746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 0x55d6c8685620 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800a510 tx=0x7fbbd800a9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb832481cf9ae7b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.155+0000 7fbbd7400640 1 -- 172.21.15.62:0/840376746 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd800e710 con 0x55d6c8687540 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.155+0000 7fbbd7400640 1 -- 172.21.15.62:0/840376746 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd800e870 con 0x55d6c8687540 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.155+0000 7fbbd7400640 1 -- 172.21.15.62:0/840376746 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd800ec20 con 0x55d6c8687540 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.156+0000 7fbbe004af40 1 -- 172.21.15.62:0/840376746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 msgr2=0x55d6c8685620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.156+0000 7fbbe004af40 1 --2- 172.21.15.62:0/840376746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 0x55d6c8685620 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800a510 tx=0x7fbbd800a9d0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.156+0000 7fbbe004af40 1 -- 172.21.15.62:0/840376746 shutdown_connections 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.156+0000 7fbbe004af40 1 --2- 172.21.15.62:0/840376746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 0x55d6c8685620 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.156+0000 7fbbe004af40 1 --2- 172.21.15.62:0/840376746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d6c8686c20 0x55d6c86850e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.156+0000 7fbbe004af40 1 --2- 172.21.15.62:0/840376746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c85e2910 0x55d6c8669be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.156+0000 7fbbe004af40 1 -- 172.21.15.62:0/840376746 >> 172.21.15.62:0/840376746 conn(0x55d6c85fcff0 msgr2=0x55d6c85ddbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.156+0000 7fbbe004af40 1 -- 172.21.15.62:0/840376746 shutdown_connections 2024-03-23T11:21:00.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.156+0000 7fbbe004af40 1 -- 172.21.15.62:0/840376746 wait complete. 2024-03-23T11:21:00.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.157+0000 7fbbe004af40 1 Processor -- start 2024-03-23T11:21:00.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.157+0000 7fbbe004af40 1 -- start start 2024-03-23T11:21:00.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.158+0000 7fbbe004af40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d6c85e2910 0x55d6c868e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.158+0000 7fbbe004af40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c8686c20 0x55d6c868d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.158+0000 7fbbe004af40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 0x55d6c86901a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.158+0000 7fbbe004af40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6c86859f0 con 0x55d6c8687540 2024-03-23T11:21:00.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.158+0000 7fbbe004af40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6c86895a0 con 0x55d6c8686c20 2024-03-23T11:21:00.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.158+0000 7fbbe004af40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d6c85de1e0 con 0x55d6c85e2910 2024-03-23T11:21:00.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.158+0000 7fbbdd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 0x55d6c86901a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.159+0000 7fbbdd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 0x55d6c86901a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46322/0 (socket says 172.21.15.62:46322) 2024-03-23T11:21:00.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.159+0000 7fbbdd600640 1 -- 172.21.15.62:0/2943105684 learned_addr learned my addr 172.21.15.62:0/2943105684 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:00.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.159+0000 7fbbd4200640 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c8686c20 0x55d6c868d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.159+0000 7fbbdcc00640 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d6c85e2910 0x55d6c868e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.159+0000 7fbbdd600640 1 -- 172.21.15.62:0/2943105684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d6c85e2910 msgr2=0x55d6c868e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.159+0000 7fbbdd600640 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d6c85e2910 0x55d6c868e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.159+0000 7fbbdd600640 1 -- 172.21.15.62:0/2943105684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c8686c20 msgr2=0x55d6c868d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.159+0000 7fbbdd600640 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c8686c20 0x55d6c868d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.159+0000 7fbbdd600640 1 -- 172.21.15.62:0/2943105684 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd800a0e0 con 0x55d6c8687540 2024-03-23T11:21:00.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.159+0000 7fbbdd600640 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 0x55d6c86901a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800aa10 tx=0x7fbbd80032b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.160+0000 7fbbd6000640 1 -- 172.21.15.62:0/2943105684 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd80109b0 con 0x55d6c8687540 2024-03-23T11:21:00.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.160+0000 7fbbd6000640 1 -- 172.21.15.62:0/2943105684 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd8010b10 con 0x55d6c8687540 2024-03-23T11:21:00.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.160+0000 7fbbd6000640 1 -- 172.21.15.62:0/2943105684 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8024400 con 0x55d6c8687540 2024-03-23T11:21:00.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.160+0000 7fbbe004af40 1 -- 172.21.15.62:0/2943105684 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d6c8506340 con 0x55d6c8687540 2024-03-23T11:21:00.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.160+0000 7fbbe004af40 1 -- 172.21.15.62:0/2943105684 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d6c84f9780 con 0x55d6c8687540 2024-03-23T11:21:00.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.162+0000 7fbbd6000640 1 -- 172.21.15.62:0/2943105684 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbbd8017030 con 0x55d6c8687540 2024-03-23T11:21:00.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.163+0000 7fbbd6000640 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbb8069180 0x7fbbb806b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.163+0000 7fbbd6000640 1 -- 172.21.15.62:0/2943105684 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbd8088f80 con 0x55d6c8687540 2024-03-23T11:21:00.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.163+0000 7fbbe004af40 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55d6c86906e0 0x55d6c872bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.163+0000 7fbbe004af40 1 -- 172.21.15.62:0/2943105684 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:1 5.19 5:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55d6c85e1d30 con 0x55d6c86906e0 2024-03-23T11:21:00.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.164+0000 7fbbdcc00640 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbb8069180 0x7fbbb806b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.164+0000 7fbbd4200640 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55d6c86906e0 0x55d6c872bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.164+0000 7fbbd4200640 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55d6c86906e0 0x55d6c872bb00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbd4200640 1 -- 172.21.15.62:0/2943105684 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (9) out=9b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7fbbc4007710 con 0x55d6c86906e0 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbe004af40 1 -- 172.21.15.62:0/2943105684 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55d6c86906e0 msgr2=0x55d6c872bb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbe004af40 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55d6c86906e0 0x55d6c872bb00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbdcc00640 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbb8069180 0x7fbbb806b640 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc054000 tx=0x7fbbcc04e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbe004af40 1 -- 172.21.15.62:0/2943105684 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbb8069180 msgr2=0x7fbbb806b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbe004af40 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbb8069180 0x7fbbb806b640 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc054000 tx=0x7fbbcc04e000 comp rx=0 tx=0).stop 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbe004af40 1 -- 172.21.15.62:0/2943105684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 msgr2=0x55d6c86901a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbe004af40 1 --2- 172.21.15.62:0/2943105684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6c8687540 0x55d6c86901a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800aa10 tx=0x7fbbd80032b0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbdd600640 1 -- 172.21.15.62:0/2943105684 reap_dead start 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbe004af40 1 -- 172.21.15.62:0/2943105684 shutdown_connections 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbe004af40 1 -- 172.21.15.62:0/2943105684 >> 172.21.15.62:0/2943105684 conn(0x55d6c85fcff0 msgr2=0x55d6c868a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbe004af40 1 -- 172.21.15.62:0/2943105684 shutdown_connections 2024-03-23T11:21:00.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.165+0000 7fbbe004af40 1 -- 172.21.15.62:0/2943105684 wait complete. 2024-03-23T11:21:00.169 INFO:teuthology.orchestra.run.smithi062.stdout:mytag 2024-03-23T11:21:00.169 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-03-23T11:21:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.273+0000 7f36a0a37f40 1 Processor -- start 2024-03-23T11:21:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.273+0000 7f36a0a37f40 1 -- start start 2024-03-23T11:21:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.274+0000 7f36a0a37f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36840035f0 0x5626c038b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.274+0000 7f36a0a37f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5626c0390150 0x5626c038f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.274+0000 7f36a0a37f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5626c02e7f00 0x5626c0391a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.274+0000 7f36a0a37f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c01ff800 con 0x7f36840035f0 2024-03-23T11:21:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.274+0000 7f36a0a37f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c01ff4a0 con 0x5626c02e7f00 2024-03-23T11:21:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.274+0000 7f36a0a37f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626c01f6110 con 0x5626c0390150 2024-03-23T11:21:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.274+0000 7f369d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36840035f0 0x5626c038b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.274+0000 7f369d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36840035f0 0x5626c038b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46344/0 (socket says 172.21.15.62:46344) 2024-03-23T11:21:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.274+0000 7f369d400640 1 -- 172.21.15.62:0/3250460948 learned_addr learned my addr 172.21.15.62:0/3250460948 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.275+0000 7f369de00640 1 --2- 172.21.15.62:0/3250460948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5626c02e7f00 0x5626c0391a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.275+0000 7f3694a00640 1 --2- 172.21.15.62:0/3250460948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5626c0390150 0x5626c038f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.275+0000 7f369d400640 1 -- 172.21.15.62:0/3250460948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5626c0390150 msgr2=0x5626c038f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.275+0000 7f369d400640 1 --2- 172.21.15.62:0/3250460948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5626c0390150 0x5626c038f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.275+0000 7f369d400640 1 -- 172.21.15.62:0/3250460948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5626c02e7f00 msgr2=0x5626c0391a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.275+0000 7f369d400640 1 --2- 172.21.15.62:0/3250460948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5626c02e7f00 0x5626c0391a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.275+0000 7f369d400640 1 -- 172.21.15.62:0/3250460948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626c020e340 con 0x7f36840035f0 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.275+0000 7f3694a00640 1 --2- 172.21.15.62:0/3250460948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5626c0390150 0x5626c038f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.275+0000 7f369de00640 1 --2- 172.21.15.62:0/3250460948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5626c02e7f00 0x5626c0391a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.275+0000 7f369d400640 1 --2- 172.21.15.62:0/3250460948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36840035f0 0x5626c038b0d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f36900028f0 tx=0x7f3690005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d883646952607193 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f3697e00640 1 -- 172.21.15.62:0/3250460948 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369000f9e0 con 0x7f36840035f0 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f3697e00640 1 -- 172.21.15.62:0/3250460948 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369000fb40 con 0x7f36840035f0 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f3697e00640 1 -- 172.21.15.62:0/3250460948 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3690013430 con 0x7f36840035f0 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f36a0a37f40 1 -- 172.21.15.62:0/3250460948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36840035f0 msgr2=0x5626c038b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f36a0a37f40 1 --2- 172.21.15.62:0/3250460948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36840035f0 0x5626c038b0d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f36900028f0 tx=0x7f3690005d80 comp rx=0 tx=0).stop 2024-03-23T11:21:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f36a0a37f40 1 -- 172.21.15.62:0/3250460948 shutdown_connections 2024-03-23T11:21:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f36a0a37f40 1 --2- 172.21.15.62:0/3250460948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5626c02e7f00 0x5626c0391a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f36a0a37f40 1 --2- 172.21.15.62:0/3250460948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5626c0390150 0x5626c038f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f36a0a37f40 1 --2- 172.21.15.62:0/3250460948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36840035f0 0x5626c038b0d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f36a0a37f40 1 -- 172.21.15.62:0/3250460948 >> 172.21.15.62:0/3250460948 conn(0x5626c0304ff0 msgr2=0x5626c02e9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f36a0a37f40 1 -- 172.21.15.62:0/3250460948 shutdown_connections 2024-03-23T11:21:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.276+0000 7f36a0a37f40 1 -- 172.21.15.62:0/3250460948 wait complete. 2024-03-23T11:21:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.277+0000 7f36a0a37f40 1 Processor -- start 2024-03-23T11:21:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.277+0000 7f36a0a37f40 1 -- start start 2024-03-23T11:21:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.278+0000 7f36a0a37f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626c02e7f00 0x5626c036f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.278+0000 7f36a0a37f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5626c0390150 0x5626c0384370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.278+0000 7f36a0a37f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840035f0 0x5626c0391a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.278+0000 7f36a0a37f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c036ec20 con 0x5626c02e7f00 2024-03-23T11:21:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.278+0000 7f36a0a37f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626c0385220 con 0x5626c0390150 2024-03-23T11:21:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.278+0000 7f36a0a37f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626c0392120 con 0x7f36840035f0 2024-03-23T11:21:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.278+0000 7f369d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626c02e7f00 0x5626c036f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.279+0000 7f369d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626c02e7f00 0x5626c036f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46348/0 (socket says 172.21.15.62:46348) 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.279+0000 7f369d400640 1 -- 172.21.15.62:0/1350975904 learned_addr learned my addr 172.21.15.62:0/1350975904 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.279+0000 7f3694a00640 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5626c0390150 0x5626c0384370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.279+0000 7f369de00640 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840035f0 0x5626c0391a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.279+0000 7f369d400640 1 -- 172.21.15.62:0/1350975904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840035f0 msgr2=0x5626c0391a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.279+0000 7f369d400640 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36840035f0 0x5626c0391a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.279+0000 7f369d400640 1 -- 172.21.15.62:0/1350975904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5626c0390150 msgr2=0x5626c0384370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.279+0000 7f369d400640 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5626c0390150 0x5626c0384370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.279+0000 7f369d400640 1 -- 172.21.15.62:0/1350975904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36900024f0 con 0x5626c02e7f00 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.279+0000 7f369d400640 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626c02e7f00 0x5626c036f0b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f369001dee0 tx=0x7f3690003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.280+0000 7f3696a00640 1 -- 172.21.15.62:0/1350975904 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3690022070 con 0x5626c02e7f00 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.280+0000 7f3696a00640 1 -- 172.21.15.62:0/1350975904 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369000fec0 con 0x5626c02e7f00 2024-03-23T11:21:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.280+0000 7f3696a00640 1 -- 172.21.15.62:0/1350975904 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369000f9e0 con 0x5626c02e7f00 2024-03-23T11:21:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.280+0000 7f36a0a37f40 1 -- 172.21.15.62:0/1350975904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5626c0201780 con 0x5626c02e7f00 2024-03-23T11:21:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.280+0000 7f36a0a37f40 1 -- 172.21.15.62:0/1350975904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626c02011e0 con 0x5626c02e7f00 2024-03-23T11:21:00.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.282+0000 7f3696a00640 1 -- 172.21.15.62:0/1350975904 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3690003040 con 0x5626c02e7f00 2024-03-23T11:21:00.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.283+0000 7f3696a00640 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f367c069180 0x7f367c06b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.283+0000 7f3696a00640 1 -- 172.21.15.62:0/1350975904 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36900896c0 con 0x5626c02e7f00 2024-03-23T11:21:00.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.283+0000 7f36a0a37f40 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5626c03849d0 0x5626c038c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:00.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.283+0000 7f36a0a37f40 1 -- 172.21.15.62:0/1350975904 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:1 5.12 5:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x5626c038cb50 con 0x5626c03849d0 2024-03-23T11:21:00.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.283+0000 7f369de00640 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5626c03849d0 0x5626c038c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.283+0000 7f3694a00640 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f367c069180 0x7f367c06b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:00.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.284+0000 7f369de00640 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5626c03849d0 0x5626c038c610 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.284+0000 7f3694a00640 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f367c069180 0x7f367c06b640 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f36880017a0 tx=0x7f3688001710 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.284+0000 7f369de00640 1 -- 172.21.15.62:0/1350975904 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f3698059040 con 0x5626c03849d0 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:error getting xattr cephfs_data/10000000002.00000000/scrub_tag: (61) No data available 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.285+0000 7f36a0a37f40 1 -- 172.21.15.62:0/1350975904 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5626c03849d0 msgr2=0x5626c038c610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.285+0000 7f36a0a37f40 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5626c03849d0 0x5626c038c610 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.285+0000 7f36a0a37f40 1 -- 172.21.15.62:0/1350975904 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f367c069180 msgr2=0x7f367c06b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.285+0000 7f36a0a37f40 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f367c069180 0x7f367c06b640 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f36880017a0 tx=0x7f3688001710 comp rx=0 tx=0).stop 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.285+0000 7f36a0a37f40 1 -- 172.21.15.62:0/1350975904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626c02e7f00 msgr2=0x5626c036f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.285+0000 7f36a0a37f40 1 --2- 172.21.15.62:0/1350975904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626c02e7f00 0x5626c036f0b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f369001dee0 tx=0x7f3690003330 comp rx=0 tx=0).stop 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.285+0000 7f369de00640 1 -- 172.21.15.62:0/1350975904 reap_dead start 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.285+0000 7f36a0a37f40 1 -- 172.21.15.62:0/1350975904 shutdown_connections 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.285+0000 7f36a0a37f40 1 -- 172.21.15.62:0/1350975904 >> 172.21.15.62:0/1350975904 conn(0x5626c0304ff0 msgr2=0x5626c0303d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.285+0000 7f36a0a37f40 1 -- 172.21.15.62:0/1350975904 shutdown_connections 2024-03-23T11:21:00.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:00.285+0000 7f36a0a37f40 1 -- 172.21.15.62:0/1350975904 wait complete. 2024-03-23T11:21:00.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:21:00.289 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:21:00.289 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:21:00.389 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:21:00.508 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:21:00.508 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:21:00.508 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:21:00.569 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:21:00.569 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:21:00.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:00.569 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:21:00.626 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-23T11:21:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:21:00.628 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:21:00.628 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:21:00.683 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:21:00.684 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:21:00.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:00.684 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T11:21:00.737 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-23T11:21:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:21:00.738 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:21:00.738 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:21:00.798 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:21:00.798 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:21:00.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:00.798 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T11:21:00.855 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-23T11:21:00.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:21:00.856 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:21:00.856 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:21:00.914 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:21:00.914 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:21:00.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:00.915 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T11:21:00.963 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-23T11:21:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:21:00.965 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:21:01.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.191+0000 7f8c81c00640 1 Processor -- start 2024-03-23T11:21:01.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.192+0000 7f8c81c00640 1 -- start start 2024-03-23T11:21:01.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.193+0000 7f8c81c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8c7c0d2440 0x7f8c7c167a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.193+0000 7f8c81c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c0b5ad0 con 0x7f8c7c15cb90 2024-03-23T11:21:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.193+0000 7f8c81c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c06b820 con 0x7f8c7c0b84f0 2024-03-23T11:21:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.193+0000 7f8c81c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c06c0c0 con 0x7f8c7c0d2440 2024-03-23T11:21:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.193+0000 7f8c7aa00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8c7c15cb90 0x7f8c7c06b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57216/0 (socket says 172.21.15.62:57216) 2024-03-23T11:21:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.193+0000 7f8c7aa00640 1 -- 172.21.15.62:0/507864284 learned_addr learned my addr 172.21.15.62:0/507864284 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:01.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.194+0000 7f8c7b400640 1 --2- 172.21.15.62:0/507864284 >> v2:172.21.15.120:3300/0 conn(0x7f8c7c0d2440 0x7f8c7c167a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.194+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852778167 0 0) 0x7f8c7c0b5ad0 con 0x7f8c7c15cb90 2024-03-23T11:21:01.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.194+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c60003520 con 0x7f8c7c15cb90 2024-03-23T11:21:01.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.194+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3343653520 0 0) 0x7f8c60003520 con 0x7f8c7c15cb90 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.194+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 >> v2:172.21.15.120:3300/0 conn(0x7f8c7c0d2440 msgr2=0x7f8c7c167a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.194+0000 7f8c79600640 1 --2- 172.21.15.62:0/507864284 >> v2:172.21.15.120:3300/0 conn(0x7f8c7c0d2440 0x7f8c7c167a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.194+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 >> v1:172.21.15.120:6789/0 conn(0x7f8c7c0b84f0 legacy=0x7f8c7c0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.194+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c06bb00 con 0x7f8c7c15cb90 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.194+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f8c64001670 con 0x7f8c7c15cb90 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.194+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 >> v1:172.21.15.62:6789/0 conn(0x7f8c7c15cb90 legacy=0x7f8c7c06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.195+0000 7f8c79600640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c60003970 0x7f8c60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.195+0000 7f8c79600640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c600047c0 0x7f8c60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.195+0000 7f8c79600640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 0x7f8c6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.195+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c06c0c0 con 0x7f8c60005180 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.195+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6000c020 con 0x7f8c60003970 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.195+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6000c1e0 con 0x7f8c600047c0 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.195+0000 7f8c7b400640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 0x7f8c6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.196+0000 7f8c7aa00640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c60003970 0x7f8c60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.196+0000 7f8c7a000640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c600047c0 0x7f8c60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.196+0000 7f8c7b400640 1 -- 172.21.15.62:0/507864284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c600047c0 msgr2=0x7f8c60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:01.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.196+0000 7f8c7b400640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c600047c0 0x7f8c60004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.196+0000 7f8c7b400640 1 -- 172.21.15.62:0/507864284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c60003970 msgr2=0x7f8c60003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:01.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.196+0000 7f8c7b400640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c60003970 0x7f8c60003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.196+0000 7f8c7b400640 1 -- 172.21.15.62:0/507864284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c60003520 con 0x7f8c60005180 2024-03-23T11:21:01.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.196+0000 7f8c7aa00640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c60003970 0x7f8c60003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:01.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.196+0000 7f8c7a000640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c600047c0 0x7f8c60004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:01.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.197+0000 7f8c7b400640 1 -- 172.21.15.62:0/507864284 reap_dead start 2024-03-23T11:21:01.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.197+0000 7f8c7b400640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 0x7f8c6000b980 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009a90 tx=0x7f8c600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e6d5056bbb7259b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:01.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.198+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7402c6b0 con 0x7f8c60005180 2024-03-23T11:21:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.198+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7402c830 con 0x7f8c60005180 2024-03-23T11:21:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.198+0000 7f8c79600640 1 -- 172.21.15.62:0/507864284 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7402cbe0 con 0x7f8c60005180 2024-03-23T11:21:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.198+0000 7f8c81c00640 1 -- 172.21.15.62:0/507864284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 msgr2=0x7f8c6000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.198+0000 7f8c81c00640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 0x7f8c6000b980 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009a90 tx=0x7f8c600057f0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.198+0000 7f8c81c00640 1 -- 172.21.15.62:0/507864284 shutdown_connections 2024-03-23T11:21:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.198+0000 7f8c81c00640 1 --2- 172.21.15.62:0/507864284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 0x7f8c6000b980 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.198+0000 7f8c81c00640 1 -- 172.21.15.62:0/507864284 >> 172.21.15.62:0/507864284 conn(0x7f8c7c0cf7d0 msgr2=0x7f8c7c0cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.199+0000 7f8c81c00640 1 -- 172.21.15.62:0/507864284 shutdown_connections 2024-03-23T11:21:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.199+0000 7f8c81c00640 1 -- 172.21.15.62:0/507864284 wait complete. 2024-03-23T11:21:01.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.200+0000 7f8c81c00640 1 Processor -- start 2024-03-23T11:21:01.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.200+0000 7f8c81c00640 1 -- start start 2024-03-23T11:21:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.201+0000 7f8c81c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 0x7f8c7c1590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.201+0000 7f8c81c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c7c1f1ea0 0x7f8c7c1e1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.201+0000 7f8c81c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c7c1e2340 0x7f8c7c1e34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.201+0000 7f8c81c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0d1010 con 0x7f8c60005180 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.201+0000 7f8c81c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c159630 con 0x7f8c7c1e2340 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.201+0000 7f8c81c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c1f2470 con 0x7f8c7c1f1ea0 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.202+0000 7f8c7aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 0x7f8c7c1590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.202+0000 7f8c7aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 0x7f8c7c1590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46370/0 (socket says 172.21.15.62:46370) 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.202+0000 7f8c7aa00640 1 -- 172.21.15.62:0/927625350 learned_addr learned my addr 172.21.15.62:0/927625350 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.202+0000 7f8c7a000640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c7c1f1ea0 0x7f8c7c1e1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.202+0000 7f8c7b400640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c7c1e2340 0x7f8c7c1e34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.202+0000 7f8c7aa00640 1 -- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c7c1f1ea0 msgr2=0x7f8c7c1e1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.202+0000 7f8c7aa00640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c7c1f1ea0 0x7f8c7c1e1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.202+0000 7f8c7aa00640 1 -- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c7c1e2340 msgr2=0x7f8c7c1e34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.202+0000 7f8c7aa00640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c7c1e2340 0x7f8c7c1e34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.202+0000 7f8c7aa00640 1 -- 172.21.15.62:0/927625350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c1f2750 con 0x7f8c60005180 2024-03-23T11:21:01.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.202+0000 7f8c7b400640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c7c1e2340 0x7f8c7c1e34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:01.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.203+0000 7f8c7aa00640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 0x7f8c7c1590f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8c64001200 tx=0x7f8c6400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:01.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.203+0000 7f8c72a00640 1 -- 172.21.15.62:0/927625350 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c640074e0 con 0x7f8c60005180 2024-03-23T11:21:01.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.203+0000 7f8c72a00640 1 -- 172.21.15.62:0/927625350 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c64007640 con 0x7f8c60005180 2024-03-23T11:21:01.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.203+0000 7f8c72a00640 1 -- 172.21.15.62:0/927625350 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c640049e0 con 0x7f8c60005180 2024-03-23T11:21:01.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.203+0000 7f8c81c00640 1 -- 172.21.15.62:0/927625350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c1f2a30 con 0x7f8c60005180 2024-03-23T11:21:01.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.203+0000 7f8c81c00640 1 -- 172.21.15.62:0/927625350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c1f54d0 con 0x7f8c60005180 2024-03-23T11:21:01.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.204+0000 7f8c81c00640 1 -- 172.21.15.62:0/927625350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c066c90 con 0x7f8c60005180 2024-03-23T11:21:01.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.205+0000 7f8c72a00640 1 -- 172.21.15.62:0/927625350 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8c64004b40 con 0x7f8c60005180 2024-03-23T11:21:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.206+0000 7f8c72a00640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5806a7a0 0x7f8c5806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.206+0000 7f8c72a00640 1 -- 172.21.15.62:0/927625350 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c64088650 con 0x7f8c60005180 2024-03-23T11:21:01.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.207+0000 7f8c7a000640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5806a7a0 0x7f8c5806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.208+0000 7f8c7a000640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5806a7a0 0x7f8c5806cc60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c158a50 tx=0x7f8c6c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:01.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.212+0000 7f8c72a00640 1 -- 172.21.15.62:0/927625350 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8c64053a80 con 0x7f8c60005180 2024-03-23T11:21:01.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.451+0000 7f8c81c00640 1 -- 172.21.15.62:0/927625350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c7c0d0640 con 0x7f8c60005180 2024-03-23T11:21:01.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.452+0000 7f8c72a00640 1 -- 172.21.15.62:0/927625350 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f8c640570a0 con 0x7f8c60005180 2024-03-23T11:21:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:01.452 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3421279564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3421279564},{"type":"v1","addr":"172.21.15.120:6835","nonce":3421279564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2052651236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2052651236},{"type":"v1","addr":"172.21.15.120:6837","nonce":2052651236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5193,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/1599029025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1599029025},{"type":"v1","addr":"172.21.15.62:6834","nonce":1599029025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:20:24.731603+0000","modified":"2024-03-23T11:20:28.249457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":5,"addr":"172.21.15.62:6835/3537214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":3537214366},{"type":"v1","addr":"172.21.15.62:6835","nonce":3537214366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T11:21:01.452 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-03-23T11:21:01.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 -- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5806a7a0 msgr2=0x7f8c5806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5806a7a0 0x7f8c5806cc60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c158a50 tx=0x7f8c6c009040 comp rx=0 tx=0).stop 2024-03-23T11:21:01.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 -- 172.21.15.62:0/927625350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 msgr2=0x7f8c7c1590f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 0x7f8c7c1590f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8c64001200 tx=0x7f8c6400dd40 comp rx=0 tx=0).stop 2024-03-23T11:21:01.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 -- 172.21.15.62:0/927625350 shutdown_connections 2024-03-23T11:21:01.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8c5806a7a0 0x7f8c5806cc60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c7c1e2340 0x7f8c7c1e34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c7c1f1ea0 0x7f8c7c1e1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 --2- 172.21.15.62:0/927625350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c60005180 0x7f8c7c1590f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 -- 172.21.15.62:0/927625350 >> 172.21.15.62:0/927625350 conn(0x7f8c7c0cf7d0 msgr2=0x7f8c7c0b7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:01.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 -- 172.21.15.62:0/927625350 shutdown_connections 2024-03-23T11:21:01.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.455+0000 7f8c81c00640 1 -- 172.21.15.62:0/927625350 wait complete. 2024-03-23T11:21:01.475 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:21:01.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.697+0000 7f6e16400640 1 Processor -- start 2024-03-23T11:21:01.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.697+0000 7f6e16400640 1 -- start start 2024-03-23T11:21:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.699+0000 7f6e16400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6e100cd7c0 0x7f6e100b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.699+0000 7f6e16400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e1006dca0 0x7f6e1006c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.699+0000 7f6e16400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e100cad20 con 0x7f6e100cdba0 2024-03-23T11:21:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.699+0000 7f6e16400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1006e130 con 0x7f6e100cd7c0 2024-03-23T11:21:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.699+0000 7f6e16400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1015a8a0 con 0x7f6e1006dca0 2024-03-23T11:21:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.699+0000 7f6e0f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6e100cd7c0 0x7f6e100b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.700+0000 7f6e0f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6e100cd7c0 0x7f6e100b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46372/0 (socket says 172.21.15.62:46372) 2024-03-23T11:21:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.700+0000 7f6e0f400640 1 -- 172.21.15.62:0/1316008463 learned_addr learned my addr 172.21.15.62:0/1316008463 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.700+0000 7f6e07e00640 1 --2- 172.21.15.62:0/1316008463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e1006dca0 0x7f6e1006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.700+0000 7f6e0f400640 1 -- 172.21.15.62:0/1316008463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e1006dca0 msgr2=0x7f6e1006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.700+0000 7f6e0f400640 1 --2- 172.21.15.62:0/1316008463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e1006dca0 0x7f6e1006c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.700+0000 7f6e0f400640 1 -- 172.21.15.62:0/1316008463 >> v1:172.21.15.120:6789/0 conn(0x7f6e100cdba0 legacy=0x7f6e100cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.700+0000 7f6e0f400640 1 -- 172.21.15.62:0/1316008463 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1015ab80 con 0x7f6e100cd7c0 2024-03-23T11:21:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.701+0000 7f6e0f400640 1 --2- 172.21.15.62:0/1316008463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e100cd7c0 0x7f6e100b15e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009870 tx=0x7f6e0002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3770bef0b905e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.701+0000 7f6e0ea00640 1 -- 172.21.15.62:0/1316008463 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e000317c0 con 0x7f6e100cd7c0 2024-03-23T11:21:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.702+0000 7f6e0ea00640 1 -- 172.21.15.62:0/1316008463 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e00031920 con 0x7f6e100cd7c0 2024-03-23T11:21:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.702+0000 7f6e0ea00640 1 -- 172.21.15.62:0/1316008463 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00031cb0 con 0x7f6e100cd7c0 2024-03-23T11:21:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.702+0000 7f6e16400640 1 -- 172.21.15.62:0/1316008463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e100cd7c0 msgr2=0x7f6e100b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.702+0000 7f6e16400640 1 --2- 172.21.15.62:0/1316008463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e100cd7c0 0x7f6e100b15e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009870 tx=0x7f6e0002b860 comp rx=0 tx=0).stop 2024-03-23T11:21:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.702+0000 7f6e16400640 1 -- 172.21.15.62:0/1316008463 shutdown_connections 2024-03-23T11:21:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.702+0000 7f6e16400640 1 --2- 172.21.15.62:0/1316008463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e1006dca0 0x7f6e1006c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.702+0000 7f6e16400640 1 --2- 172.21.15.62:0/1316008463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e100cd7c0 0x7f6e100b15e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.702+0000 7f6e16400640 1 -- 172.21.15.62:0/1316008463 >> 172.21.15.62:0/1316008463 conn(0x7f6e100b02e0 msgr2=0x7f6e100b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.702+0000 7f6e16400640 1 -- 172.21.15.62:0/1316008463 shutdown_connections 2024-03-23T11:21:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.702+0000 7f6e16400640 1 -- 172.21.15.62:0/1316008463 wait complete. 2024-03-23T11:21:01.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.705+0000 7f6e16400640 1 Processor -- start 2024-03-23T11:21:01.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.705+0000 7f6e16400640 1 -- start start 2024-03-23T11:21:01.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.706+0000 7f6e16400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e100cd7c0 0x7f6e100acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.706+0000 7f6e16400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e100cdba0 0x7f6e10154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.706+0000 7f6e16400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e101552e0 0x7f6e10153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.706+0000 7f6e16400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10155b10 con 0x7f6e100cdba0 2024-03-23T11:21:01.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.706+0000 7f6e16400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10155df0 con 0x7f6e101552e0 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.706+0000 7f6e16400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e101560d0 con 0x7f6e100cd7c0 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.706+0000 7f6e07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e100cdba0 0x7f6e10154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.706+0000 7f6e07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e100cdba0 0x7f6e10154da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46384/0 (socket says 172.21.15.62:46384) 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.707+0000 7f6e07e00640 1 -- 172.21.15.62:0/2248289742 learned_addr learned my addr 172.21.15.62:0/2248289742 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.707+0000 7f6e0f400640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e100cd7c0 0x7f6e100acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.707+0000 7f6e0fe00640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e101552e0 0x7f6e10153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.707+0000 7f6e07e00640 1 -- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e100cd7c0 msgr2=0x7f6e100acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.707+0000 7f6e07e00640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e100cd7c0 0x7f6e100acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.707+0000 7f6e07e00640 1 -- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e101552e0 msgr2=0x7f6e10153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.707+0000 7f6e07e00640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e101552e0 0x7f6e10153810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.707+0000 7f6e07e00640 1 -- 172.21.15.62:0/2248289742 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e000094d0 con 0x7f6e100cdba0 2024-03-23T11:21:01.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.707+0000 7f6e0f400640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e100cd7c0 0x7f6e100acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:21:01.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.707+0000 7f6e07e00640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e100cdba0 0x7f6e10154da0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6df800d5f0 tx=0x7f6df800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:01.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.708+0000 7f6e07400640 1 -- 172.21.15.62:0/2248289742 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df80125f0 con 0x7f6e100cdba0 2024-03-23T11:21:01.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.708+0000 7f6e07400640 1 -- 172.21.15.62:0/2248289742 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df8012750 con 0x7f6e100cdba0 2024-03-23T11:21:01.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.708+0000 7f6e07400640 1 -- 172.21.15.62:0/2248289742 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8012b00 con 0x7f6e100cdba0 2024-03-23T11:21:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.708+0000 7f6e16400640 1 -- 172.21.15.62:0/2248289742 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e101563b0 con 0x7f6e100cdba0 2024-03-23T11:21:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.708+0000 7f6e16400640 1 -- 172.21.15.62:0/2248289742 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e10156b90 con 0x7f6e100cdba0 2024-03-23T11:21:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.709+0000 7f6e16400640 1 -- 172.21.15.62:0/2248289742 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e10066c90 con 0x7f6e100cdba0 2024-03-23T11:21:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.710+0000 7f6e07400640 1 -- 172.21.15.62:0/2248289742 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6df8012c60 con 0x7f6e100cdba0 2024-03-23T11:21:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.711+0000 7f6e07400640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6dec06a7a0 0x7f6dec06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.711+0000 7f6e07400640 1 -- 172.21.15.62:0/2248289742 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6df8089320 con 0x7f6e100cdba0 2024-03-23T11:21:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.711+0000 7f6e0f400640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6dec06a7a0 0x7f6dec06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:01.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.713+0000 7f6e0f400640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6dec06a7a0 0x7f6dec06cc60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6e00036000 tx=0x7f6e0003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.715+0000 7f6e07400640 1 -- 172.21.15.62:0/2248289742 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6df80546a0 con 0x7f6e100cdba0 2024-03-23T11:21:01.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.914+0000 7f6e16400640 1 -- 172.21.15.62:0/2248289742 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e1006c970 con 0x7f6e100cdba0 2024-03-23T11:21:01.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.915+0000 7f6e07400640 1 -- 172.21.15.62:0/2248289742 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16101 (secure 0 0 0) 0x7f6df8057cc0 con 0x7f6e100cdba0 2024-03-23T11:21:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:01.915 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:20:25.735105+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:20:22.395497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T11:20:23.725162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:21:01.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 -- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6dec06a7a0 msgr2=0x7f6dec06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6dec06a7a0 0x7f6dec06cc60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6e00036000 tx=0x7f6e0003d040 comp rx=0 tx=0).stop 2024-03-23T11:21:01.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 -- 172.21.15.62:0/2248289742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e100cdba0 msgr2=0x7f6e10154da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:01.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e100cdba0 0x7f6e10154da0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6df800d5f0 tx=0x7f6df800dab0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 -- 172.21.15.62:0/2248289742 shutdown_connections 2024-03-23T11:21:01.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6dec06a7a0 0x7f6dec06cc60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e100cdba0 0x7f6e10154da0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e101552e0 0x7f6e10153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 --2- 172.21.15.62:0/2248289742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e100cd7c0 0x7f6e100acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:01.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 -- 172.21.15.62:0/2248289742 >> 172.21.15.62:0/2248289742 conn(0x7f6e100b02e0 msgr2=0x7f6e1015af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:01.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 -- 172.21.15.62:0/2248289742 shutdown_connections 2024-03-23T11:21:01.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:01.918+0000 7f6e16400640 1 -- 172.21.15.62:0/2248289742 wait complete. 2024-03-23T11:21:01.936 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:20:22.395497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:20:23.725162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:21:01.937 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:21:02.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.161+0000 7f0d02200640 1 Processor -- start 2024-03-23T11:21:02.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.161+0000 7f0d02200640 1 -- start start 2024-03-23T11:21:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.163+0000 7f0d02200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0cfc15c830 0x7f0cfc0ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.163+0000 7f0d02200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0cfc160310 0x7f0cfc0ab980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.163+0000 7f0d02200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc1608d0 con 0x7f0cfc0abec0 2024-03-23T11:21:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.163+0000 7f0d02200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cfc0addb0 con 0x7f0cfc160310 2024-03-23T11:21:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.163+0000 7f0d02200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cfc15cc10 con 0x7f0cfc15c830 2024-03-23T11:21:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.164+0000 7f0cfbe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0cfc160310 0x7f0cfc0ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.164+0000 7f0cfbe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0cfc160310 0x7f0cfc0ab980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46390/0 (socket says 172.21.15.62:46390) 2024-03-23T11:21:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.164+0000 7f0cfbe00640 1 -- 172.21.15.62:0/3358487160 learned_addr learned my addr 172.21.15.62:0/3358487160 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.164+0000 7f0d00e00640 1 --2- 172.21.15.62:0/3358487160 >> v2:172.21.15.120:3300/0 conn(0x7f0cfc15c830 0x7f0cfc0ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.164+0000 7f0cfbe00640 1 -- 172.21.15.62:0/3358487160 >> v2:172.21.15.120:3300/0 conn(0x7f0cfc15c830 msgr2=0x7f0cfc0ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.164+0000 7f0cfbe00640 1 --2- 172.21.15.62:0/3358487160 >> v2:172.21.15.120:3300/0 conn(0x7f0cfc15c830 0x7f0cfc0ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.164+0000 7f0cfbe00640 1 -- 172.21.15.62:0/3358487160 >> v1:172.21.15.120:6789/0 conn(0x7f0cfc0abec0 legacy=0x7f0cfc0ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.164+0000 7f0cfbe00640 1 -- 172.21.15.62:0/3358487160 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc160bb0 con 0x7f0cfc160310 2024-03-23T11:21:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.165+0000 7f0cfbe00640 1 --2- 172.21.15.62:0/3358487160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cfc160310 0x7f0cfc0ab980 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0ce220 tx=0x7f0ce802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d56424e1eff4df server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:02.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.166+0000 7f0cfb400640 1 -- 172.21.15.62:0/3358487160 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8031870 con 0x7f0cfc160310 2024-03-23T11:21:02.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.166+0000 7f0cfb400640 1 -- 172.21.15.62:0/3358487160 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce80319d0 con 0x7f0cfc160310 2024-03-23T11:21:02.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.166+0000 7f0cfb400640 1 -- 172.21.15.62:0/3358487160 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8031d60 con 0x7f0cfc160310 2024-03-23T11:21:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.166+0000 7f0d02200640 1 -- 172.21.15.62:0/3358487160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cfc160310 msgr2=0x7f0cfc0ab980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.166+0000 7f0d02200640 1 --2- 172.21.15.62:0/3358487160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cfc160310 0x7f0cfc0ab980 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0ce220 tx=0x7f0ce802b910 comp rx=0 tx=0).stop 2024-03-23T11:21:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.166+0000 7f0d02200640 1 -- 172.21.15.62:0/3358487160 shutdown_connections 2024-03-23T11:21:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.166+0000 7f0d02200640 1 --2- 172.21.15.62:0/3358487160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cfc160310 0x7f0cfc0ab980 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.166+0000 7f0d02200640 1 --2- 172.21.15.62:0/3358487160 >> v2:172.21.15.120:3300/0 conn(0x7f0cfc15c830 0x7f0cfc0ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.166+0000 7f0d02200640 1 -- 172.21.15.62:0/3358487160 >> 172.21.15.62:0/3358487160 conn(0x7f0cfc066050 msgr2=0x7f0cfc069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.167+0000 7f0d02200640 1 -- 172.21.15.62:0/3358487160 shutdown_connections 2024-03-23T11:21:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.167+0000 7f0d02200640 1 -- 172.21.15.62:0/3358487160 wait complete. 2024-03-23T11:21:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.168+0000 7f0d02200640 1 Processor -- start 2024-03-23T11:21:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.168+0000 7f0d02200640 1 -- start start 2024-03-23T11:21:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.169+0000 7f0d02200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cfc0abec0 0x7f0cfc1537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.169+0000 7f0d02200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cfc15c830 0x7f0cfc15c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.169+0000 7f0d02200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cfc160310 0x7f0cfc0af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.169+0000 7f0d02200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc153d20 con 0x7f0cfc15c830 2024-03-23T11:21:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.169+0000 7f0d02200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc15c620 con 0x7f0cfc160310 2024-03-23T11:21:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.169+0000 7f0d02200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc1f40a0 con 0x7f0cfc0abec0 2024-03-23T11:21:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.170+0000 7f0cfbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cfc15c830 0x7f0cfc15c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.170+0000 7f0cfbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cfc15c830 0x7f0cfc15c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46400/0 (socket says 172.21.15.62:46400) 2024-03-23T11:21:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.170+0000 7f0cfbe00640 1 -- 172.21.15.62:0/709624922 learned_addr learned my addr 172.21.15.62:0/709624922 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.170+0000 7f0d01800640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cfc160310 0x7f0cfc0af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.170+0000 7f0cfbe00640 1 -- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cfc0abec0 msgr2=0x7f0cfc1537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.170+0000 7f0cfbe00640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cfc0abec0 0x7f0cfc1537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.170+0000 7f0cfbe00640 1 -- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cfc160310 msgr2=0x7f0cfc0af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.170+0000 7f0cfbe00640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cfc160310 0x7f0cfc0af0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.170+0000 7f0cfbe00640 1 -- 172.21.15.62:0/709624922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce80094d0 con 0x7f0cfc15c830 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.170+0000 7f0d01800640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cfc160310 0x7f0cfc0af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.171+0000 7f0cfbe00640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cfc15c830 0x7f0cfc15c0e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8000c00 tx=0x7f0ce80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.171+0000 7f0cf8c00640 1 -- 172.21.15.62:0/709624922 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce803abe0 con 0x7f0cfc15c830 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.171+0000 7f0cf8c00640 1 -- 172.21.15.62:0/709624922 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce803ad40 con 0x7f0cfc15c830 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.171+0000 7f0d02200640 1 -- 172.21.15.62:0/709624922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cfc1f4380 con 0x7f0cfc15c830 2024-03-23T11:21:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.171+0000 7f0cf8c00640 1 -- 172.21.15.62:0/709624922 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8044630 con 0x7f0cfc15c830 2024-03-23T11:21:02.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.171+0000 7f0d02200640 1 -- 172.21.15.62:0/709624922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cfc1f4940 con 0x7f0cfc15c830 2024-03-23T11:21:02.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.172+0000 7f0d02200640 1 -- 172.21.15.62:0/709624922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cfc066bb0 con 0x7f0cfc15c830 2024-03-23T11:21:02.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.173+0000 7f0cf8c00640 1 -- 172.21.15.62:0/709624922 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0ce804d450 con 0x7f0cfc15c830 2024-03-23T11:21:02.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.174+0000 7f0cf8c00640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0cdc06a7d0 0x7f0cdc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.174+0000 7f0cf8c00640 1 -- 172.21.15.62:0/709624922 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ce802d040 con 0x7f0cfc15c830 2024-03-23T11:21:02.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.175+0000 7f0d00e00640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0cdc06a7d0 0x7f0cdc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.176+0000 7f0d00e00640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0cdc06a7d0 0x7f0cdc06cc90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00098f0 tx=0x7f0cf0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:02.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.178+0000 7f0cf8c00640 1 -- 172.21.15.62:0/709624922 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0ce80767e0 con 0x7f0cfc15c830 2024-03-23T11:21:02.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.377+0000 7f0d02200640 1 -- 172.21.15.62:0/709624922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0cfc0b1250 con 0x7f0cfc15c830 2024-03-23T11:21:02.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.378+0000 7f0cf8c00640 1 -- 172.21.15.62:0/709624922 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ce8079e00 con 0x7f0cfc15c830 2024-03-23T11:21:02.377 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:21:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.380+0000 7f0d02200640 1 -- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0cdc06a7d0 msgr2=0x7f0cdc06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.380+0000 7f0d02200640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0cdc06a7d0 0x7f0cdc06cc90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00098f0 tx=0x7f0cf0008040 comp rx=0 tx=0).stop 2024-03-23T11:21:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.380+0000 7f0d02200640 1 -- 172.21.15.62:0/709624922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cfc15c830 msgr2=0x7f0cfc15c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.380+0000 7f0d02200640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cfc15c830 0x7f0cfc15c0e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8000c00 tx=0x7f0ce80046c0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.381+0000 7f0d02200640 1 -- 172.21.15.62:0/709624922 shutdown_connections 2024-03-23T11:21:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.381+0000 7f0d02200640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0cdc06a7d0 0x7f0cdc06cc90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.381+0000 7f0d02200640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cfc15c830 0x7f0cfc15c0e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.381+0000 7f0d02200640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cfc160310 0x7f0cfc0af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.381+0000 7f0d02200640 1 --2- 172.21.15.62:0/709624922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cfc0abec0 0x7f0cfc1537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.381+0000 7f0d02200640 1 -- 172.21.15.62:0/709624922 >> 172.21.15.62:0/709624922 conn(0x7f0cfc066050 msgr2=0x7f0cfc15fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:02.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.381+0000 7f0d02200640 1 -- 172.21.15.62:0/709624922 shutdown_connections 2024-03-23T11:21:02.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.381+0000 7f0d02200640 1 -- 172.21.15.62:0/709624922 wait complete. 2024-03-23T11:21:02.399 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:21:02.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.624+0000 7f9b3dc00640 1 Processor -- start 2024-03-23T11:21:02.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.624+0000 7f9b3dc00640 1 -- start start 2024-03-23T11:21:02.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.626+0000 7f9b3dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b38160310 0x7f9b380ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.626+0000 7f9b3dc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9b380ce250 0x7f9b380ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.626+0000 7f9b3dc00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b381608d0 con 0x7f9b3815c830 2024-03-23T11:21:02.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.626+0000 7f9b3dc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b380addb0 con 0x7f9b380ce250 2024-03-23T11:21:02.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.626+0000 7f9b3dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3815cc10 con 0x7f9b38160310 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.626+0000 7f9b37e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9b3815c830 0x7f9b380ab980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57218/0 (socket says 172.21.15.62:57218) 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.626+0000 7f9b37e00640 1 -- 172.21.15.62:0/1901887455 learned_addr learned my addr 172.21.15.62:0/1901887455 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.627+0000 7f9b3d200640 1 --2- 172.21.15.62:0/1901887455 >> v2:172.21.15.120:3300/0 conn(0x7f9b380ce250 0x7f9b380ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.627+0000 7f9b37400640 1 --2- 172.21.15.62:0/1901887455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b38160310 0x7f9b380ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.627+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651621671 0 0) 0x7f9b381608d0 con 0x7f9b3815c830 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.627+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b1c003520 con 0x7f9b3815c830 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.627+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 808365472 0 0) 0x7f9b1c003520 con 0x7f9b3815c830 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.627+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b38160310 msgr2=0x7f9b380ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.627+0000 7f9b36a00640 1 --2- 172.21.15.62:0/1901887455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b38160310 0x7f9b380ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.627+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 >> v2:172.21.15.120:3300/0 conn(0x7f9b380ce250 msgr2=0x7f9b380ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.627+0000 7f9b36a00640 1 --2- 172.21.15.62:0/1901887455 >> v2:172.21.15.120:3300/0 conn(0x7f9b380ce250 0x7f9b380ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.627+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b38160bb0 con 0x7f9b3815c830 2024-03-23T11:21:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.627+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9b280019f0 con 0x7f9b3815c830 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 >> v1:172.21.15.62:6789/0 conn(0x7f9b3815c830 legacy=0x7f9b380ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b36a00640 1 --2- 172.21.15.62:0/1901887455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 0x7f9b1c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b36a00640 1 --2- 172.21.15.62:0/1901887455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b38160310 0x7f9b1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b36a00640 1 --2- 172.21.15.62:0/1901887455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b1c004d00 0x7f9b1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3815cc10 con 0x7f9b1c0039a0 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c00b980 con 0x7f9b1c004d00 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c00bac0 con 0x7f9b38160310 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b37e00640 1 --2- 172.21.15.62:0/1901887455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 0x7f9b1c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b37e00640 1 -- 172.21.15.62:0/1901887455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b38160310 msgr2=0x7f9b1c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b37e00640 1 --2- 172.21.15.62:0/1901887455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b38160310 0x7f9b1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b37e00640 1 -- 172.21.15.62:0/1901887455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b1c004d00 msgr2=0x7f9b1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b37e00640 1 --2- 172.21.15.62:0/1901887455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b1c004d00 0x7f9b1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.628+0000 7f9b37e00640 1 -- 172.21.15.62:0/1901887455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c003520 con 0x7f9b1c0039a0 2024-03-23T11:21:02.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.629+0000 7f9b37e00640 1 --2- 172.21.15.62:0/1901887455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 0x7f9b1c003dc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800a860 tx=0x7f9b1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3cbbf9ddbf6555a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:02.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.630+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b280072d0 con 0x7f9b1c0039a0 2024-03-23T11:21:02.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.630+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2802f070 con 0x7f9b1c0039a0 2024-03-23T11:21:02.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.630+0000 7f9b36a00640 1 -- 172.21.15.62:0/1901887455 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b280076b0 con 0x7f9b1c0039a0 2024-03-23T11:21:02.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.630+0000 7f9b3dc00640 1 -- 172.21.15.62:0/1901887455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 msgr2=0x7f9b1c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:02.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.630+0000 7f9b3dc00640 1 --2- 172.21.15.62:0/1901887455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 0x7f9b1c003dc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800a860 tx=0x7f9b1c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.630+0000 7f9b3d200640 1 -- 172.21.15.62:0/1901887455 reap_dead start 2024-03-23T11:21:02.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.630+0000 7f9b3dc00640 1 -- 172.21.15.62:0/1901887455 shutdown_connections 2024-03-23T11:21:02.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.630+0000 7f9b3dc00640 1 -- 172.21.15.62:0/1901887455 >> 172.21.15.62:0/1901887455 conn(0x7f9b38066050 msgr2=0x7f9b38069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:02.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.630+0000 7f9b3dc00640 1 -- 172.21.15.62:0/1901887455 shutdown_connections 2024-03-23T11:21:02.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.630+0000 7f9b3dc00640 1 -- 172.21.15.62:0/1901887455 wait complete. 2024-03-23T11:21:02.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.632+0000 7f9b3dc00640 1 Processor -- start 2024-03-23T11:21:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.632+0000 7f9b3dc00640 1 -- start start 2024-03-23T11:21:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.633+0000 7f9b3dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 0x7f9b38151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.633+0000 7f9b3dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b380af580 0x7f9b3814d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.633+0000 7f9b3dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3815e2a0 0x7f9b3815e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.633+0000 7f9b3dc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3815ec60 con 0x7f9b1c0039a0 2024-03-23T11:21:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.633+0000 7f9b3dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b380cb0a0 con 0x7f9b3815e2a0 2024-03-23T11:21:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.633+0000 7f9b3dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b381f2030 con 0x7f9b380af580 2024-03-23T11:21:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.634+0000 7f9b37e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 0x7f9b38151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.634+0000 7f9b37e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 0x7f9b38151870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46414/0 (socket says 172.21.15.62:46414) 2024-03-23T11:21:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.634+0000 7f9b37e00640 1 -- 172.21.15.62:0/3560342214 learned_addr learned my addr 172.21.15.62:0/3560342214 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:02.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.634+0000 7f9b37400640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b380af580 0x7f9b3814d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.634+0000 7f9b3d200640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3815e2a0 0x7f9b3815e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.634+0000 7f9b37e00640 1 -- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b380af580 msgr2=0x7f9b3814d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.634+0000 7f9b37e00640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b380af580 0x7f9b3814d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.634+0000 7f9b37e00640 1 -- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3815e2a0 msgr2=0x7f9b3815e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.634+0000 7f9b37e00640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3815e2a0 0x7f9b3815e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.634+0000 7f9b37e00640 1 -- 172.21.15.62:0/3560342214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b380cd640 con 0x7f9b1c0039a0 2024-03-23T11:21:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.634+0000 7f9b37e00640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 0x7f9b38151870 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9b28001cb0 tx=0x7f9b28001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.635+0000 7f9b2fe00640 1 -- 172.21.15.62:0/3560342214 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b280073a0 con 0x7f9b1c0039a0 2024-03-23T11:21:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.635+0000 7f9b2fe00640 1 -- 172.21.15.62:0/3560342214 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b28007500 con 0x7f9b1c0039a0 2024-03-23T11:21:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.635+0000 7f9b2fe00640 1 -- 172.21.15.62:0/3560342214 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28007890 con 0x7f9b1c0039a0 2024-03-23T11:21:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.635+0000 7f9b3dc00640 1 -- 172.21.15.62:0/3560342214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b380cd900 con 0x7f9b1c0039a0 2024-03-23T11:21:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.635+0000 7f9b3dc00640 1 -- 172.21.15.62:0/3560342214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b380cdec0 con 0x7f9b1c0039a0 2024-03-23T11:21:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.636+0000 7f9b3dc00640 1 -- 172.21.15.62:0/3560342214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b380ce9e0 con 0x7f9b1c0039a0 2024-03-23T11:21:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.637+0000 7f9b2fe00640 1 -- 172.21.15.62:0/3560342214 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9b28031030 con 0x7f9b1c0039a0 2024-03-23T11:21:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.638+0000 7f9b2fe00640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b1406a7d0 0x7f9b1406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.639+0000 7f9b2fe00640 1 -- 172.21.15.62:0/3560342214 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b280aafb0 con 0x7f9b1c0039a0 2024-03-23T11:21:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.639+0000 7f9b37400640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b1406a7d0 0x7f9b1406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.640+0000 7f9b37400640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b1406a7d0 0x7f9b1406cc90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9b3815f250 tx=0x7f9b20008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.643+0000 7f9b2fe00640 1 -- 172.21.15.62:0/3560342214 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9b280de580 con 0x7f9b1c0039a0 2024-03-23T11:21:02.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.844+0000 7f9b3dc00640 1 -- 172.21.15.62:0/3560342214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9b38066bb0 con 0x7f9b1c0039a0 2024-03-23T11:21:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.845+0000 7f9b2fe00640 1 -- 172.21.15.62:0/3560342214 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9b28076460 con 0x7f9b1c0039a0 2024-03-23T11:21:02.844 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.847+0000 7f9b3dc00640 1 -- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b1406a7d0 msgr2=0x7f9b1406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.847+0000 7f9b3dc00640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b1406a7d0 0x7f9b1406cc90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9b3815f250 tx=0x7f9b20008040 comp rx=0 tx=0).stop 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.847+0000 7f9b3dc00640 1 -- 172.21.15.62:0/3560342214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 msgr2=0x7f9b38151870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.847+0000 7f9b3dc00640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 0x7f9b38151870 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9b28001cb0 tx=0x7f9b28001d80 comp rx=0 tx=0).stop 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.847+0000 7f9b3dc00640 1 -- 172.21.15.62:0/3560342214 shutdown_connections 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.847+0000 7f9b3dc00640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b1406a7d0 0x7f9b1406cc90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.847+0000 7f9b3dc00640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b3815e2a0 0x7f9b3815e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.847+0000 7f9b3dc00640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b380af580 0x7f9b3814d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.847+0000 7f9b3dc00640 1 --2- 172.21.15.62:0/3560342214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c0039a0 0x7f9b38151870 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.847+0000 7f9b3dc00640 1 -- 172.21.15.62:0/3560342214 >> 172.21.15.62:0/3560342214 conn(0x7f9b38066050 msgr2=0x7f9b380ad100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.847+0000 7f9b3dc00640 1 -- 172.21.15.62:0/3560342214 shutdown_connections 2024-03-23T11:21:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:02.848+0000 7f9b3dc00640 1 -- 172.21.15.62:0/3560342214 wait complete. 2024-03-23T11:21:02.865 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:21:03.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.082+0000 7f4189a00640 1 Processor -- start 2024-03-23T11:21:03.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.082+0000 7f4189a00640 1 -- start start 2024-03-23T11:21:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.084+0000 7f4189a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4184143dd0 0x7f418415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.084+0000 7f4189a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41840cd140 con 0x7f41840b2f30 2024-03-23T11:21:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.084+0000 7f4189a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41840cd3e0 con 0x7f4184070090 2024-03-23T11:21:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.084+0000 7f4189a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41840cd6c0 con 0x7f4184143dd0 2024-03-23T11:21:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.084+0000 7f4189000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f41840b2f30 0x7f41840ce480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57228/0 (socket says 172.21.15.62:57228) 2024-03-23T11:21:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.084+0000 7f4189000640 1 -- 172.21.15.62:0/413365583 learned_addr learned my addr 172.21.15.62:0/413365583 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:03.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4183e00640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4184143dd0 0x7f418415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195723604 0 0) 0x7f41840cd140 con 0x7f41840b2f30 2024-03-23T11:21:03.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4168003520 con 0x7f41840b2f30 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1242953248 0 0) 0x7f4168003520 con 0x7f41840b2f30 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4184143dd0 msgr2=0x7f418415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4184143dd0 0x7f418415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 >> v1:172.21.15.120:6789/0 conn(0x7f4184070090 legacy=0x7f41840b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41840cd9a0 con 0x7f41840b2f30 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f417c0019f0 con 0x7f41840b2f30 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 >> v1:172.21.15.62:6789/0 conn(0x7f41840b2f30 legacy=0x7f41840ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41680039a0 0x7f4168003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41680047c0 0x7f4168004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4184143dd0 0x7f4168005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41840cd6c0 con 0x7f41680039a0 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416800ba60 con 0x7f41680047c0 2024-03-23T11:21:03.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.085+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416800bc20 con 0x7f4184143dd0 2024-03-23T11:21:03.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.086+0000 7f4183e00640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41680039a0 0x7f4168003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.086+0000 7f4183400640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41680047c0 0x7f4168004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.086+0000 7f4189000640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4184143dd0 0x7f4168005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.086+0000 7f4183e00640 1 -- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4184143dd0 msgr2=0x7f4168005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:03.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.086+0000 7f4183e00640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4184143dd0 0x7f4168005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.086+0000 7f4183e00640 1 -- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41680047c0 msgr2=0x7f4168004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:03.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.086+0000 7f4183e00640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41680047c0 0x7f4168004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.086+0000 7f4183e00640 1 -- 172.21.15.62:0/413365583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4168003520 con 0x7f41680039a0 2024-03-23T11:21:03.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.086+0000 7f4189000640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4184143dd0 0x7f4168005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:03.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.087+0000 7f4183e00640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41680039a0 0x7f4168003dc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4174009a90 tx=0x7f41680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a88af36bd2c382e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:03.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.088+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417402c680 con 0x7f41680039a0 2024-03-23T11:21:03.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.088+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417402c7e0 con 0x7f41680039a0 2024-03-23T11:21:03.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.088+0000 7f4182a00640 1 -- 172.21.15.62:0/413365583 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417402cb70 con 0x7f41680039a0 2024-03-23T11:21:03.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.088+0000 7f4189a00640 1 -- 172.21.15.62:0/413365583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41680039a0 msgr2=0x7f4168003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.088+0000 7f4189a00640 1 --2- 172.21.15.62:0/413365583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41680039a0 0x7f4168003dc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4174009a90 tx=0x7f41680057f0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.088+0000 7f4189000640 1 -- 172.21.15.62:0/413365583 reap_dead start 2024-03-23T11:21:03.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.089+0000 7f4189a00640 1 -- 172.21.15.62:0/413365583 shutdown_connections 2024-03-23T11:21:03.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.089+0000 7f4189a00640 1 -- 172.21.15.62:0/413365583 >> 172.21.15.62:0/413365583 conn(0x7f4184066050 msgr2=0x7f4184069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:03.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.089+0000 7f4189a00640 1 -- 172.21.15.62:0/413365583 shutdown_connections 2024-03-23T11:21:03.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.089+0000 7f4189a00640 1 -- 172.21.15.62:0/413365583 wait complete. 2024-03-23T11:21:03.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.090+0000 7f4189a00640 1 Processor -- start 2024-03-23T11:21:03.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.091+0000 7f4189a00640 1 -- start start 2024-03-23T11:21:03.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.091+0000 7f4189a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41680039a0 0x7f41841f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41840af040 0x7f41840b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41841ef2e0 0x7f4184155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41840b00f0 con 0x7f41841ef2e0 2024-03-23T11:21:03.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41841f2e50 con 0x7f41840af040 2024-03-23T11:21:03.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41840af4e0 con 0x7f41680039a0 2024-03-23T11:21:03.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41841ef2e0 0x7f4184155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41841ef2e0 0x7f4184155500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46424/0 (socket says 172.21.15.62:46424) 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189000640 1 -- 172.21.15.62:0/2561889841 learned_addr learned my addr 172.21.15.62:0/2561889841 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4183400640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41840af040 0x7f41840b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189000640 1 -- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41680039a0 msgr2=0x7f41841f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189000640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41680039a0 0x7f41841f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189000640 1 -- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41840af040 msgr2=0x7f41840b0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189000640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41840af040 0x7f41840b0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4189000640 1 -- 172.21.15.62:0/2561889841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41840af7c0 con 0x7f41841ef2e0 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.092+0000 7f4183400640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41840af040 0x7f41840b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.093+0000 7f4189000640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41841ef2e0 0x7f4184155500 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f41840b3350 tx=0x7f417c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.093+0000 7f417be00640 1 -- 172.21.15.62:0/2561889841 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c00deb0 con 0x7f41841ef2e0 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.093+0000 7f417be00640 1 -- 172.21.15.62:0/2561889841 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c004b60 con 0x7f41841ef2e0 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.093+0000 7f417be00640 1 -- 172.21.15.62:0/2561889841 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c007070 con 0x7f41841ef2e0 2024-03-23T11:21:03.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.093+0000 7f4189a00640 1 -- 172.21.15.62:0/2561889841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41840afaa0 con 0x7f41841ef2e0 2024-03-23T11:21:03.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.093+0000 7f4189a00640 1 -- 172.21.15.62:0/2561889841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41840b0ac0 con 0x7f41841ef2e0 2024-03-23T11:21:03.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.095+0000 7f4189a00640 1 -- 172.21.15.62:0/2561889841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4184066bb0 con 0x7f41841ef2e0 2024-03-23T11:21:03.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.095+0000 7f417be00640 1 -- 172.21.15.62:0/2561889841 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f417c0071d0 con 0x7f41841ef2e0 2024-03-23T11:21:03.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.096+0000 7f417be00640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f416406a7a0 0x7f416406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.096+0000 7f417be00640 1 -- 172.21.15.62:0/2561889841 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f417c089180 con 0x7f41841ef2e0 2024-03-23T11:21:03.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.097+0000 7f4183e00640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f416406a7a0 0x7f416406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.098+0000 7f4183e00640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f416406a7a0 0x7f416406cc60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4174036000 tx=0x7f417402e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:03.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.104+0000 7f417be00640 1 -- 172.21.15.62:0/2561889841 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f417c0545a0 con 0x7f41841ef2e0 2024-03-23T11:21:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.309+0000 7f4189a00640 1 -- 172.21.15.62:0/2561889841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f41840b0230 con 0x7f41841ef2e0 2024-03-23T11:21:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.310+0000 7f417be00640 1 -- 172.21.15.62:0/2561889841 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7f417c057bc0 con 0x7f41841ef2e0 2024-03-23T11:21:03.309 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:21:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 -- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f416406a7a0 msgr2=0x7f416406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f416406a7a0 0x7f416406cc60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4174036000 tx=0x7f417402e000 comp rx=0 tx=0).stop 2024-03-23T11:21:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 -- 172.21.15.62:0/2561889841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41841ef2e0 msgr2=0x7f4184155500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41841ef2e0 0x7f4184155500 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f41840b3350 tx=0x7f417c00b4a0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 -- 172.21.15.62:0/2561889841 shutdown_connections 2024-03-23T11:21:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f416406a7a0 0x7f416406cc60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41841ef2e0 0x7f4184155500 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41840af040 0x7f41840b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 --2- 172.21.15.62:0/2561889841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41680039a0 0x7f41841f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 -- 172.21.15.62:0/2561889841 >> 172.21.15.62:0/2561889841 conn(0x7f4184066050 msgr2=0x7f418406dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 -- 172.21.15.62:0/2561889841 shutdown_connections 2024-03-23T11:21:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.313+0000 7f4189a00640 1 -- 172.21.15.62:0/2561889841 wait complete. 2024-03-23T11:21:03.332 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:21:03.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.545+0000 7fa327200640 1 Processor -- start 2024-03-23T11:21:03.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.545+0000 7fa327200640 1 -- start start 2024-03-23T11:21:03.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.547+0000 7fa327200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa3200cafa0 0x7fa3200cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.547+0000 7fa327200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa32006c6e0 0x7fa32006cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.547+0000 7fa327200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3200cbc60 con 0x7fa32006c300 2024-03-23T11:21:03.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.547+0000 7fa327200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3200cd410 con 0x7fa32006c6e0 2024-03-23T11:21:03.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.547+0000 7fa327200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3200cbf40 con 0x7fa3200cafa0 2024-03-23T11:21:03.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.547+0000 7fa325200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa32006c6e0 0x7fa32006cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.547+0000 7fa325200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa32006c6e0 0x7fa32006cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46426/0 (socket says 172.21.15.62:46426) 2024-03-23T11:21:03.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.547+0000 7fa325200640 1 -- 172.21.15.62:0/1491038187 learned_addr learned my addr 172.21.15.62:0/1491038187 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:03.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.547+0000 7fa31f400640 1 --2- 172.21.15.62:0/1491038187 >> v2:172.21.15.120:3300/0 conn(0x7fa3200cafa0 0x7fa3200cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.548+0000 7fa325200640 1 -- 172.21.15.62:0/1491038187 >> v2:172.21.15.120:3300/0 conn(0x7fa3200cafa0 msgr2=0x7fa3200cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:03.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.548+0000 7fa325200640 1 --2- 172.21.15.62:0/1491038187 >> v2:172.21.15.120:3300/0 conn(0x7fa3200cafa0 0x7fa3200cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.548+0000 7fa325200640 1 -- 172.21.15.62:0/1491038187 >> v1:172.21.15.120:6789/0 conn(0x7fa32006c300 legacy=0x7fa3200ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:03.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.548+0000 7fa325200640 1 -- 172.21.15.62:0/1491038187 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3200cc220 con 0x7fa32006c6e0 2024-03-23T11:21:03.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.549+0000 7fa325200640 1 --2- 172.21.15.62:0/1491038187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa32006c6e0 0x7fa32006cb30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa32006e370 tx=0x7fa31802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7988d507a6927139 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:03.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.549+0000 7fa31ea00640 1 -- 172.21.15.62:0/1491038187 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31802ca80 con 0x7fa32006c6e0 2024-03-23T11:21:03.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.549+0000 7fa31ea00640 1 -- 172.21.15.62:0/1491038187 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31802cbe0 con 0x7fa32006c6e0 2024-03-23T11:21:03.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.549+0000 7fa31ea00640 1 -- 172.21.15.62:0/1491038187 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3180343f0 con 0x7fa32006c6e0 2024-03-23T11:21:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.550+0000 7fa327200640 1 -- 172.21.15.62:0/1491038187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa32006c6e0 msgr2=0x7fa32006cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.550+0000 7fa327200640 1 --2- 172.21.15.62:0/1491038187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa32006c6e0 0x7fa32006cb30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa32006e370 tx=0x7fa31802bee0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.550+0000 7fa327200640 1 -- 172.21.15.62:0/1491038187 shutdown_connections 2024-03-23T11:21:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.550+0000 7fa327200640 1 --2- 172.21.15.62:0/1491038187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa32006c6e0 0x7fa32006cb30 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.550+0000 7fa327200640 1 --2- 172.21.15.62:0/1491038187 >> v2:172.21.15.120:3300/0 conn(0x7fa3200cafa0 0x7fa3200cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.550+0000 7fa327200640 1 -- 172.21.15.62:0/1491038187 >> 172.21.15.62:0/1491038187 conn(0x7fa3200b02e0 msgr2=0x7fa3200b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.550+0000 7fa327200640 1 -- 172.21.15.62:0/1491038187 shutdown_connections 2024-03-23T11:21:03.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.550+0000 7fa327200640 1 -- 172.21.15.62:0/1491038187 wait complete. 2024-03-23T11:21:03.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.552+0000 7fa327200640 1 Processor -- start 2024-03-23T11:21:03.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.552+0000 7fa327200640 1 -- start start 2024-03-23T11:21:03.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa327200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa32006c300 0x7fa32013dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa327200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa32006c6e0 0x7fa320133030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa327200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3200cafa0 0x7fa3201f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa327200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3201f1ef0 con 0x7fa32006c300 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa327200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3200abcc0 con 0x7fa32006c6e0 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa327200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3200abfa0 con 0x7fa3200cafa0 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa31fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa32006c300 0x7fa32013dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa31fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa32006c300 0x7fa32013dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46438/0 (socket says 172.21.15.62:46438) 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa31fe00640 1 -- 172.21.15.62:0/2490224771 learned_addr learned my addr 172.21.15.62:0/2490224771 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa31f400640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa32006c6e0 0x7fa320133030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa31fe00640 1 -- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3200cafa0 msgr2=0x7fa3201f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa31fe00640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3200cafa0 0x7fa3201f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa31fe00640 1 -- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa32006c6e0 msgr2=0x7fa320133030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa31fe00640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa32006c6e0 0x7fa320133030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa31fe00640 1 -- 172.21.15.62:0/2490224771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa318009b30 con 0x7fa32006c300 2024-03-23T11:21:03.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.554+0000 7fa31f400640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa32006c6e0 0x7fa320133030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:03.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.555+0000 7fa31fe00640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa32006c300 0x7fa32013dc00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa31000c430 tx=0x7fa31000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:03.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.555+0000 7fa317e00640 1 -- 172.21.15.62:0/2490224771 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310011790 con 0x7fa32006c300 2024-03-23T11:21:03.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.555+0000 7fa317e00640 1 -- 172.21.15.62:0/2490224771 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3100118f0 con 0x7fa32006c300 2024-03-23T11:21:03.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.555+0000 7fa327200640 1 -- 172.21.15.62:0/2490224771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3200ac280 con 0x7fa32006c300 2024-03-23T11:21:03.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.555+0000 7fa317e00640 1 -- 172.21.15.62:0/2490224771 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310011ce0 con 0x7fa32006c300 2024-03-23T11:21:03.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.555+0000 7fa327200640 1 -- 172.21.15.62:0/2490224771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3200ac840 con 0x7fa32006c300 2024-03-23T11:21:03.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.556+0000 7fa327200640 1 -- 172.21.15.62:0/2490224771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa320066c90 con 0x7fa32006c300 2024-03-23T11:21:03.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.557+0000 7fa317e00640 1 -- 172.21.15.62:0/2490224771 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa31001c460 con 0x7fa32006c300 2024-03-23T11:21:03.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.559+0000 7fa317e00640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa30006a7d0 0x7fa30006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:03.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.559+0000 7fa317e00640 1 -- 172.21.15.62:0/2490224771 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa310088cc0 con 0x7fa32006c300 2024-03-23T11:21:03.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.559+0000 7fa31f400640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa30006a7d0 0x7fa30006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:03.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.561+0000 7fa31f400640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa30006a7d0 0x7fa30006cc90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa30c006fd0 tx=0x7fa30c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:03.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.563+0000 7fa317e00640 1 -- 172.21.15.62:0/2490224771 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa31008f080 con 0x7fa32006c300 2024-03-23T11:21:03.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.799+0000 7fa327200640 1 -- 172.21.15.62:0/2490224771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3200ade50 con 0x7fa32006c300 2024-03-23T11:21:03.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.800+0000 7fa317e00640 1 -- 172.21.15.62:0/2490224771 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fa3100540d0 con 0x7fa32006c300 2024-03-23T11:21:03.800 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:03.800 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3421279564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3421279564},{"type":"v1","addr":"172.21.15.120:6835","nonce":3421279564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2052651236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2052651236},{"type":"v1","addr":"172.21.15.120:6837","nonce":2052651236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5193,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/1599029025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1599029025},{"type":"v1","addr":"172.21.15.62:6834","nonce":1599029025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:20:24.731603+0000","modified":"2024-03-23T11:20:28.249457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":5,"addr":"172.21.15.62:6835/3537214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":3537214366},{"type":"v1","addr":"172.21.15.62:6835","nonce":3537214366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T11:21:03.800 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.803+0000 7fa327200640 1 -- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa30006a7d0 msgr2=0x7fa30006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.803+0000 7fa327200640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa30006a7d0 0x7fa30006cc90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa30c006fd0 tx=0x7fa30c008040 comp rx=0 tx=0).stop 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.803+0000 7fa327200640 1 -- 172.21.15.62:0/2490224771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa32006c300 msgr2=0x7fa32013dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.803+0000 7fa327200640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa32006c300 0x7fa32013dc00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa31000c430 tx=0x7fa31000c8f0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.803+0000 7fa327200640 1 -- 172.21.15.62:0/2490224771 shutdown_connections 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.803+0000 7fa327200640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa30006a7d0 0x7fa30006cc90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.803+0000 7fa327200640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3200cafa0 0x7fa3201f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.803+0000 7fa327200640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa32006c6e0 0x7fa320133030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.803+0000 7fa327200640 1 --2- 172.21.15.62:0/2490224771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa32006c300 0x7fa32013dc00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.803+0000 7fa327200640 1 -- 172.21.15.62:0/2490224771 >> 172.21.15.62:0/2490224771 conn(0x7fa3200b02e0 msgr2=0x7fa320157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.804+0000 7fa327200640 1 -- 172.21.15.62:0/2490224771 shutdown_connections 2024-03-23T11:21:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:03.804+0000 7fa327200640 1 -- 172.21.15.62:0/2490224771 wait complete. 2024-03-23T11:21:03.822 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:21:04.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.043+0000 7eff6aa00640 1 Processor -- start 2024-03-23T11:21:04.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.043+0000 7eff6aa00640 1 -- start start 2024-03-23T11:21:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.044+0000 7eff6aa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff64143dd0 0x7eff6415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff6aa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7eff64070090 0x7eff64070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff6aa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff640b2820 0x7eff640b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff6aa00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff64070a60 con 0x7eff64143dd0 2024-03-23T11:21:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff6aa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff640b31b0 con 0x7eff64070090 2024-03-23T11:21:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff6aa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff640cd140 con 0x7eff640b2820 2024-03-23T11:21:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff69600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff64143dd0 0x7eff6415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff69600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff64143dd0 0x7eff6415abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46450/0 (socket says 172.21.15.62:46450) 2024-03-23T11:21:04.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff69600640 1 -- 172.21.15.62:0/2631101174 learned_addr learned my addr 172.21.15.62:0/2631101174 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:04.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff69600640 1 -- 172.21.15.62:0/2631101174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff640b2820 msgr2=0x7eff640b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:04.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff69600640 1 --2- 172.21.15.62:0/2631101174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff640b2820 0x7eff640b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff69600640 1 -- 172.21.15.62:0/2631101174 >> v2:172.21.15.120:3300/0 conn(0x7eff64070090 msgr2=0x7eff64070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:04.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff69600640 1 --2- 172.21.15.62:0/2631101174 >> v2:172.21.15.120:3300/0 conn(0x7eff64070090 0x7eff64070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.045+0000 7eff69600640 1 -- 172.21.15.62:0/2631101174 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff640cd3e0 con 0x7eff64143dd0 2024-03-23T11:21:04.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.046+0000 7eff69600640 1 --2- 172.21.15.62:0/2631101174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64143dd0 0x7eff6415abe0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7eff64071530 tx=0x7eff5402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458c7fec80144dcd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.047+0000 7eff63e00640 1 -- 172.21.15.62:0/2631101174 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54031870 con 0x7eff64143dd0 2024-03-23T11:21:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.047+0000 7eff63e00640 1 -- 172.21.15.62:0/2631101174 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff540319d0 con 0x7eff64143dd0 2024-03-23T11:21:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.047+0000 7eff63e00640 1 -- 172.21.15.62:0/2631101174 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54031d60 con 0x7eff64143dd0 2024-03-23T11:21:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.047+0000 7eff6aa00640 1 -- 172.21.15.62:0/2631101174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64143dd0 msgr2=0x7eff6415abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.047+0000 7eff6aa00640 1 --2- 172.21.15.62:0/2631101174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64143dd0 0x7eff6415abe0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7eff64071530 tx=0x7eff5402b910 comp rx=0 tx=0).stop 2024-03-23T11:21:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.047+0000 7eff6aa00640 1 -- 172.21.15.62:0/2631101174 shutdown_connections 2024-03-23T11:21:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.047+0000 7eff6aa00640 1 --2- 172.21.15.62:0/2631101174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff640b2820 0x7eff640b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.047+0000 7eff6aa00640 1 --2- 172.21.15.62:0/2631101174 >> v2:172.21.15.120:3300/0 conn(0x7eff64070090 0x7eff64070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.047+0000 7eff6aa00640 1 --2- 172.21.15.62:0/2631101174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64143dd0 0x7eff6415abe0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.047+0000 7eff6aa00640 1 -- 172.21.15.62:0/2631101174 >> 172.21.15.62:0/2631101174 conn(0x7eff64066050 msgr2=0x7eff64069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:04.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.048+0000 7eff6aa00640 1 -- 172.21.15.62:0/2631101174 shutdown_connections 2024-03-23T11:21:04.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.048+0000 7eff6aa00640 1 -- 172.21.15.62:0/2631101174 wait complete. 2024-03-23T11:21:04.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.049+0000 7eff6aa00640 1 Processor -- start 2024-03-23T11:21:04.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.049+0000 7eff6aa00640 1 -- start start 2024-03-23T11:21:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.050+0000 7eff6aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64070090 0x7eff640b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.050+0000 7eff6aa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff640b2820 0x7eff6413fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.050+0000 7eff6aa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff64143dd0 0x7eff6413f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.050+0000 7eff6aa00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff640b0670 con 0x7eff64070090 2024-03-23T11:21:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.050+0000 7eff6aa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff640aea40 con 0x7eff640b2820 2024-03-23T11:21:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.050+0000 7eff6aa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff640aed20 con 0x7eff64143dd0 2024-03-23T11:21:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.050+0000 7eff69600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64070090 0x7eff640b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.051+0000 7eff69600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64070090 0x7eff640b0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46460/0 (socket says 172.21.15.62:46460) 2024-03-23T11:21:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.051+0000 7eff69600640 1 -- 172.21.15.62:0/4287111023 learned_addr learned my addr 172.21.15.62:0/4287111023 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.051+0000 7eff68c00640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff640b2820 0x7eff6413fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.051+0000 7eff69600640 1 -- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff64143dd0 msgr2=0x7eff6413f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.051+0000 7eff69600640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff64143dd0 0x7eff6413f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.051+0000 7eff69600640 1 -- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff640b2820 msgr2=0x7eff6413fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.051+0000 7eff69600640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff640b2820 0x7eff6413fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.051+0000 7eff69600640 1 -- 172.21.15.62:0/4287111023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff540094d0 con 0x7eff64070090 2024-03-23T11:21:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.051+0000 7eff68c00640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff640b2820 0x7eff6413fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.051+0000 7eff69600640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64070090 0x7eff640b0090 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7eff54000c00 tx=0x7eff540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.052+0000 7eff61600640 1 -- 172.21.15.62:0/4287111023 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5403ad10 con 0x7eff64070090 2024-03-23T11:21:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.052+0000 7eff61600640 1 -- 172.21.15.62:0/4287111023 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5403ae70 con 0x7eff64070090 2024-03-23T11:21:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.052+0000 7eff61600640 1 -- 172.21.15.62:0/4287111023 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54044610 con 0x7eff64070090 2024-03-23T11:21:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.052+0000 7eff6aa00640 1 -- 172.21.15.62:0/4287111023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff640af000 con 0x7eff64070090 2024-03-23T11:21:04.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.052+0000 7eff6aa00640 1 -- 172.21.15.62:0/4287111023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff64140330 con 0x7eff64070090 2024-03-23T11:21:04.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.053+0000 7eff6aa00640 1 -- 172.21.15.62:0/4287111023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff640af520 con 0x7eff64070090 2024-03-23T11:21:04.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.054+0000 7eff61600640 1 -- 172.21.15.62:0/4287111023 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7eff5404d450 con 0x7eff64070090 2024-03-23T11:21:04.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.055+0000 7eff61600640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff4806a7d0 0x7eff4806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.055+0000 7eff61600640 1 -- 172.21.15.62:0/4287111023 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff5402d040 con 0x7eff64070090 2024-03-23T11:21:04.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.055+0000 7eff68c00640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff4806a7d0 0x7eff4806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.056+0000 7eff68c00640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff4806a7d0 0x7eff4806cc90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7eff6415cab0 tx=0x7eff580073c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:04.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.059+0000 7eff61600640 1 -- 172.21.15.62:0/4287111023 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff54076900 con 0x7eff64070090 2024-03-23T11:21:04.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.263+0000 7eff6aa00640 1 -- 172.21.15.62:0/4287111023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff64066bb0 con 0x7eff64070090 2024-03-23T11:21:04.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.265+0000 7eff61600640 1 -- 172.21.15.62:0/4287111023 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16101 (secure 0 0 0) 0x7eff54079f20 con 0x7eff64070090 2024-03-23T11:21:04.264 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:04.264 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:20:25.735105+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:20:22.395497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T11:20:23.725162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:21:04.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.267+0000 7eff6aa00640 1 -- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff4806a7d0 msgr2=0x7eff4806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.267+0000 7eff6aa00640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff4806a7d0 0x7eff4806cc90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7eff6415cab0 tx=0x7eff580073c0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.267+0000 7eff6aa00640 1 -- 172.21.15.62:0/4287111023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64070090 msgr2=0x7eff640b0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.267+0000 7eff6aa00640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64070090 0x7eff640b0090 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7eff54000c00 tx=0x7eff540046c0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.268+0000 7eff6aa00640 1 -- 172.21.15.62:0/4287111023 shutdown_connections 2024-03-23T11:21:04.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.268+0000 7eff6aa00640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff4806a7d0 0x7eff4806cc90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.268+0000 7eff6aa00640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff64143dd0 0x7eff6413f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.268+0000 7eff6aa00640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff640b2820 0x7eff6413fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.268+0000 7eff6aa00640 1 --2- 172.21.15.62:0/4287111023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff64070090 0x7eff640b0090 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.268+0000 7eff6aa00640 1 -- 172.21.15.62:0/4287111023 >> 172.21.15.62:0/4287111023 conn(0x7eff64066050 msgr2=0x7eff6415ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:04.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.268+0000 7eff6aa00640 1 -- 172.21.15.62:0/4287111023 shutdown_connections 2024-03-23T11:21:04.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.268+0000 7eff6aa00640 1 -- 172.21.15.62:0/4287111023 wait complete. 2024-03-23T11:21:04.286 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T11:21:04.286 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:21:04.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.492+0000 7f5a5cc00640 1 Processor -- start 2024-03-23T11:21:04.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.492+0000 7f5a5cc00640 1 -- start start 2024-03-23T11:21:04.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.494+0000 7f5a5cc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5a58156920 0x7f5a581594c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.494+0000 7f5a5cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 0x7f5a580b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.494+0000 7f5a5cc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a580cd3d0 con 0x7f5a58156d00 2024-03-23T11:21:04.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.494+0000 7f5a5cc00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a580cd6b0 con 0x7f5a58156920 2024-03-23T11:21:04.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.494+0000 7f5a5cc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a580cd990 con 0x7f5a5806de90 2024-03-23T11:21:04.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.494+0000 7f5a55e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5a58156920 0x7f5a581594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.494+0000 7f5a55e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5a58156920 0x7f5a581594c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46468/0 (socket says 172.21.15.62:46468) 2024-03-23T11:21:04.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.494+0000 7f5a55e00640 1 -- 172.21.15.62:0/2596090361 learned_addr learned my addr 172.21.15.62:0/2596090361 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:04.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.495+0000 7f5a56800640 1 --2- 172.21.15.62:0/2596090361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 0x7f5a580b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.495+0000 7f5a55e00640 1 -- 172.21.15.62:0/2596090361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 msgr2=0x7f5a580b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:04.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.495+0000 7f5a55e00640 1 --2- 172.21.15.62:0/2596090361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 0x7f5a580b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.495+0000 7f5a55e00640 1 -- 172.21.15.62:0/2596090361 >> v1:172.21.15.120:6789/0 conn(0x7f5a58156d00 legacy=0x7f5a5806cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.495+0000 7f5a55e00640 1 -- 172.21.15.62:0/2596090361 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a580cdc70 con 0x7f5a58156920 2024-03-23T11:21:04.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.495+0000 7f5a56800640 1 --2- 172.21.15.62:0/2596090361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 0x7f5a580b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:21:04.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.496+0000 7f5a55e00640 1 --2- 172.21.15.62:0/2596090361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a58156920 0x7f5a581594c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5a480098a0 tx=0x7f5a4802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f3ef97821dee32a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:04.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.497+0000 7f5a54a00640 1 -- 172.21.15.62:0/2596090361 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a48031640 con 0x7f5a58156920 2024-03-23T11:21:04.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.497+0000 7f5a54a00640 1 -- 172.21.15.62:0/2596090361 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a480317a0 con 0x7f5a58156920 2024-03-23T11:21:04.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.497+0000 7f5a54a00640 1 -- 172.21.15.62:0/2596090361 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a48031b30 con 0x7f5a58156920 2024-03-23T11:21:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.497+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2596090361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a58156920 msgr2=0x7f5a581594c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.497+0000 7f5a5cc00640 1 --2- 172.21.15.62:0/2596090361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a58156920 0x7f5a581594c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5a480098a0 tx=0x7f5a4802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.498+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2596090361 shutdown_connections 2024-03-23T11:21:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.498+0000 7f5a5cc00640 1 --2- 172.21.15.62:0/2596090361 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 0x7f5a580b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.498+0000 7f5a5cc00640 1 --2- 172.21.15.62:0/2596090361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a58156920 0x7f5a581594c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.498+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2596090361 >> 172.21.15.62:0/2596090361 conn(0x7f5a580b02e0 msgr2=0x7f5a580b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.498+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2596090361 shutdown_connections 2024-03-23T11:21:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.498+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2596090361 wait complete. 2024-03-23T11:21:04.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.500+0000 7f5a5cc00640 1 Processor -- start 2024-03-23T11:21:04.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.500+0000 7f5a5cc00640 1 -- start start 2024-03-23T11:21:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.501+0000 7f5a5cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 0x7f5a58165f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.501+0000 7f5a5cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a58156920 0x7f5a58146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.501+0000 7f5a5cc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a58156d00 0x7f5a581476e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.501+0000 7f5a5cc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a581474e0 con 0x7f5a58156920 2024-03-23T11:21:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.501+0000 7f5a5cc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a58157320 con 0x7f5a58156d00 2024-03-23T11:21:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.501+0000 7f5a5cc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a581665f0 con 0x7f5a5806de90 2024-03-23T11:21:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.501+0000 7f5a55400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a58156920 0x7f5a58146fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.501+0000 7f5a55400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a58156920 0x7f5a58146fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46476/0 (socket says 172.21.15.62:46476) 2024-03-23T11:21:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.501+0000 7f5a55400640 1 -- 172.21.15.62:0/2345805091 learned_addr learned my addr 172.21.15.62:0/2345805091 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a56800640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a58156d00 0x7f5a581476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a55e00640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 0x7f5a58165f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a55400640 1 -- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 msgr2=0x7f5a58165f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a55400640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 0x7f5a58165f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a55400640 1 -- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a58156d00 msgr2=0x7f5a581476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a55400640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a58156d00 0x7f5a581476e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a55400640 1 -- 172.21.15.62:0/2345805091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a480094d0 con 0x7f5a58156920 2024-03-23T11:21:04.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a55e00640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 0x7f5a58165f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:04.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a55400640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a58156920 0x7f5a58146fa0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000c3c0 tx=0x7f5a4000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:04.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a4e000640 1 -- 172.21.15.62:0/2345805091 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a400052d0 con 0x7f5a58156920 2024-03-23T11:21:04.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a4e000640 1 -- 172.21.15.62:0/2345805091 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a40005450 con 0x7f5a58156920 2024-03-23T11:21:04.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a4e000640 1 -- 172.21.15.62:0/2345805091 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a40003830 con 0x7f5a58156920 2024-03-23T11:21:04.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.502+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2345805091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a581668d0 con 0x7f5a58156920 2024-03-23T11:21:04.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.503+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2345805091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5814ad00 con 0x7f5a58156920 2024-03-23T11:21:04.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.504+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2345805091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a58066c90 con 0x7f5a58156920 2024-03-23T11:21:04.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.504+0000 7f5a4e000640 1 -- 172.21.15.62:0/2345805091 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5a40003990 con 0x7f5a58156920 2024-03-23T11:21:04.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.505+0000 7f5a4e000640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a3406a7d0 0x7f5a3406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.505+0000 7f5a4e000640 1 -- 172.21.15.62:0/2345805091 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a400882e0 con 0x7f5a58156920 2024-03-23T11:21:04.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.506+0000 7f5a55e00640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a3406a7d0 0x7f5a3406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.507+0000 7f5a55e00640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a3406a7d0 0x7f5a3406cc90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5a48036000 tx=0x7f5a4803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:04.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.510+0000 7f5a4e000640 1 -- 172.21.15.62:0/2345805091 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a40053660 con 0x7f5a58156920 2024-03-23T11:21:04.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.747+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2345805091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a5806e510 con 0x7f5a58156920 2024-03-23T11:21:04.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.748+0000 7f5a4e000640 1 -- 172.21.15.62:0/2345805091 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f5a40056c80 con 0x7f5a58156920 2024-03-23T11:21:04.747 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:04.747 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3421279564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3421279564},{"type":"v1","addr":"172.21.15.120:6835","nonce":3421279564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2052651236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2052651236},{"type":"v1","addr":"172.21.15.120:6837","nonce":2052651236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5193,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/1599029025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1599029025},{"type":"v1","addr":"172.21.15.62:6834","nonce":1599029025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:20:24.731603+0000","modified":"2024-03-23T11:20:28.249457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":5,"addr":"172.21.15.62:6835/3537214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":3537214366},{"type":"v1","addr":"172.21.15.62:6835","nonce":3537214366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T11:21:04.748 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-03-23T11:21:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.750+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a3406a7d0 msgr2=0x7f5a3406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.750+0000 7f5a5cc00640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a3406a7d0 0x7f5a3406cc90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5a48036000 tx=0x7f5a4803c040 comp rx=0 tx=0).stop 2024-03-23T11:21:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.750+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2345805091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a58156920 msgr2=0x7f5a58146fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.750+0000 7f5a5cc00640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a58156920 0x7f5a58146fa0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000c3c0 tx=0x7f5a4000c880 comp rx=0 tx=0).stop 2024-03-23T11:21:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.751+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2345805091 shutdown_connections 2024-03-23T11:21:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.751+0000 7f5a5cc00640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a3406a7d0 0x7f5a3406cc90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.751+0000 7f5a5cc00640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a58156920 0x7f5a58146fa0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.751+0000 7f5a5cc00640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a58156d00 0x7f5a581476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.751+0000 7f5a5cc00640 1 --2- 172.21.15.62:0/2345805091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5a5806de90 0x7f5a58165f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.751+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2345805091 >> 172.21.15.62:0/2345805091 conn(0x7f5a580b02e0 msgr2=0x7f5a5815b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:04.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.751+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2345805091 shutdown_connections 2024-03-23T11:21:04.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.751+0000 7f5a5cc00640 1 -- 172.21.15.62:0/2345805091 wait complete. 2024-03-23T11:21:04.770 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:21:04.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.987+0000 7fdef7600640 1 Processor -- start 2024-03-23T11:21:04.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.988+0000 7fdef7600640 1 -- start start 2024-03-23T11:21:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.989+0000 7fdef7600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdef015cdc0 0x7fdef00d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.989+0000 7fdef7600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef006bfc0 con 0x7fdef00cf650 2024-03-23T11:21:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.989+0000 7fdef7600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef00b5260 con 0x7fdef015c9e0 2024-03-23T11:21:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.989+0000 7fdef7600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef00ce140 con 0x7fdef015cdc0 2024-03-23T11:21:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.990+0000 7fdeefe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdef015cdc0 0x7fdef00d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.990+0000 7fdeefe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdef015cdc0 0x7fdef00d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46480/0 (socket says 172.21.15.62:46480) 2024-03-23T11:21:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.990+0000 7fdeefe00640 1 -- 172.21.15.62:0/4056437522 learned_addr learned my addr 172.21.15.62:0/4056437522 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.990+0000 7fdef5600640 1 --1- 172.21.15.62:0/4056437522 >> v1:172.21.15.62:6789/0 conn(0x7fdef00cf650 0x7fdef015d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57230/0 (socket says 172.21.15.62:57230) 2024-03-23T11:21:04.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.990+0000 7fdeefe00640 1 -- 172.21.15.62:0/4056437522 >> v1:172.21.15.120:6789/0 conn(0x7fdef015c9e0 legacy=0x7fdef01607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.990+0000 7fdeefe00640 1 -- 172.21.15.62:0/4056437522 >> v1:172.21.15.62:6789/0 conn(0x7fdef00cf650 legacy=0x7fdef015d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.990+0000 7fdeefe00640 1 -- 172.21.15.62:0/4056437522 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef00b4370 con 0x7fdef015cdc0 2024-03-23T11:21:04.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.992+0000 7fdeefe00640 1 --2- 172.21.15.62:0/4056437522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdef015cdc0 0x7fdef00d1f20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009870 tx=0x7fdedc02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ce76cd947c372b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.992+0000 7fdeef400640 1 -- 172.21.15.62:0/4056437522 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc02f850 con 0x7fdef015cdc0 2024-03-23T11:21:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.992+0000 7fdeef400640 1 -- 172.21.15.62:0/4056437522 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdedc02f9b0 con 0x7fdef015cdc0 2024-03-23T11:21:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.992+0000 7fdeef400640 1 -- 172.21.15.62:0/4056437522 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc02fd40 con 0x7fdef015cdc0 2024-03-23T11:21:04.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.993+0000 7fdef7600640 1 -- 172.21.15.62:0/4056437522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdef015cdc0 msgr2=0x7fdef00d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.993+0000 7fdef7600640 1 --2- 172.21.15.62:0/4056437522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdef015cdc0 0x7fdef00d1f20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009870 tx=0x7fdedc02ba30 comp rx=0 tx=0).stop 2024-03-23T11:21:04.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.993+0000 7fdef7600640 1 -- 172.21.15.62:0/4056437522 shutdown_connections 2024-03-23T11:21:04.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.993+0000 7fdef7600640 1 --2- 172.21.15.62:0/4056437522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdef015cdc0 0x7fdef00d1f20 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.993+0000 7fdef7600640 1 -- 172.21.15.62:0/4056437522 >> 172.21.15.62:0/4056437522 conn(0x7fdef0071570 msgr2=0x7fdef0071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:04.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.993+0000 7fdef7600640 1 -- 172.21.15.62:0/4056437522 shutdown_connections 2024-03-23T11:21:04.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.993+0000 7fdef7600640 1 -- 172.21.15.62:0/4056437522 wait complete. 2024-03-23T11:21:04.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.995+0000 7fdef7600640 1 Processor -- start 2024-03-23T11:21:04.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.995+0000 7fdef7600640 1 -- start start 2024-03-23T11:21:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.996+0000 7fdef7600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdef00cf650 0x7fdef0158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.996+0000 7fdef7600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdef015c9e0 0x7fdef01f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.996+0000 7fdef7600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdef015cdc0 0x7fdef00b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.996+0000 7fdef7600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef00cc730 con 0x7fdef015cdc0 2024-03-23T11:21:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.996+0000 7fdef7600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef00c7400 con 0x7fdef015c9e0 2024-03-23T11:21:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.996+0000 7fdef7600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef0159430 con 0x7fdef00cf650 2024-03-23T11:21:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.996+0000 7fdef5600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdef015cdc0 0x7fdef00b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.996+0000 7fdef5600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdef015cdc0 0x7fdef00b6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46484/0 (socket says 172.21.15.62:46484) 2024-03-23T11:21:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.996+0000 7fdef5600640 1 -- 172.21.15.62:0/3190290823 learned_addr learned my addr 172.21.15.62:0/3190290823 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdef5600640 1 -- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdef00cf650 msgr2=0x7fdef0158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdeefe00640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdef015c9e0 0x7fdef01f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdef5600640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdef00cf650 0x7fdef0158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdef5600640 1 -- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdef015c9e0 msgr2=0x7fdef01f2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdef5600640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdef015c9e0 0x7fdef01f2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdef5600640 1 -- 172.21.15.62:0/3190290823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc0094d0 con 0x7fdef015cdc0 2024-03-23T11:21:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdeefe00640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdef015c9e0 0x7fdef01f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdef5600640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdef015cdc0 0x7fdef00b6c60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdee800c7b0 tx=0x7fdee800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:04.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdeecc00640 1 -- 172.21.15.62:0/3190290823 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee800a640 con 0x7fdef015cdc0 2024-03-23T11:21:04.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdeecc00640 1 -- 172.21.15.62:0/3190290823 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee800a7a0 con 0x7fdef015cdc0 2024-03-23T11:21:04.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdef7600640 1 -- 172.21.15.62:0/3190290823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef00c6ad0 con 0x7fdef015cdc0 2024-03-23T11:21:04.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.997+0000 7fdeecc00640 1 -- 172.21.15.62:0/3190290823 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee800ab10 con 0x7fdef015cdc0 2024-03-23T11:21:04.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.998+0000 7fdef7600640 1 -- 172.21.15.62:0/3190290823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef00c7090 con 0x7fdef015cdc0 2024-03-23T11:21:04.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:04.999+0000 7fdef7600640 1 -- 172.21.15.62:0/3190290823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef0066c90 con 0x7fdef015cdc0 2024-03-23T11:21:04.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.000+0000 7fdeecc00640 1 -- 172.21.15.62:0/3190290823 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fdee800ac70 con 0x7fdef015cdc0 2024-03-23T11:21:04.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.001+0000 7fdeecc00640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fded006a7d0 0x7fded006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.001+0000 7fdeecc00640 1 -- 172.21.15.62:0/3190290823 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdee8088490 con 0x7fdef015cdc0 2024-03-23T11:21:05.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.001+0000 7fdef4c00640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fded006a7d0 0x7fded006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.002+0000 7fdef4c00640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fded006a7d0 0x7fded006cc90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdee4000f60 tx=0x7fdee40079b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.006+0000 7fdeecc00640 1 -- 172.21.15.62:0/3190290823 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdee8053950 con 0x7fdef015cdc0 2024-03-23T11:21:05.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.243+0000 7fdef7600640 1 -- 172.21.15.62:0/3190290823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdef00d2220 con 0x7fdef015cdc0 2024-03-23T11:21:05.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.244+0000 7fdeecc00640 1 -- 172.21.15.62:0/3190290823 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fdee8056f70 con 0x7fdef015cdc0 2024-03-23T11:21:05.243 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:05.243 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3421279564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3421279564},{"type":"v1","addr":"172.21.15.120:6835","nonce":3421279564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2052651236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2052651236},{"type":"v1","addr":"172.21.15.120:6837","nonce":2052651236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5193,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/1599029025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1599029025},{"type":"v1","addr":"172.21.15.62:6834","nonce":1599029025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:20:24.731603+0000","modified":"2024-03-23T11:20:28.249457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":5,"addr":"172.21.15.62:6835/3537214366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":3537214366},{"type":"v1","addr":"172.21.15.62:6835","nonce":3537214366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T11:21:05.243 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-03-23T11:21:05.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.246+0000 7fdef7600640 1 -- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fded006a7d0 msgr2=0x7fded006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.246+0000 7fdef7600640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fded006a7d0 0x7fded006cc90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdee4000f60 tx=0x7fdee40079b0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.246+0000 7fdef7600640 1 -- 172.21.15.62:0/3190290823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdef015cdc0 msgr2=0x7fdef00b6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.246+0000 7fdef7600640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdef015cdc0 0x7fdef00b6c60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdee800c7b0 tx=0x7fdee800cc70 comp rx=0 tx=0).stop 2024-03-23T11:21:05.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.247+0000 7fdef7600640 1 -- 172.21.15.62:0/3190290823 shutdown_connections 2024-03-23T11:21:05.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.247+0000 7fdef7600640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fded006a7d0 0x7fded006cc90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.247+0000 7fdef7600640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdef015cdc0 0x7fdef00b6c60 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.247+0000 7fdef7600640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdef015c9e0 0x7fdef01f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.247+0000 7fdef7600640 1 --2- 172.21.15.62:0/3190290823 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdef00cf650 0x7fdef0158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.247+0000 7fdef7600640 1 -- 172.21.15.62:0/3190290823 >> 172.21.15.62:0/3190290823 conn(0x7fdef0071570 msgr2=0x7fdef00b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:05.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.247+0000 7fdef7600640 1 -- 172.21.15.62:0/3190290823 shutdown_connections 2024-03-23T11:21:05.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.247+0000 7fdef7600640 1 -- 172.21.15.62:0/3190290823 wait complete. 2024-03-23T11:21:05.265 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:21:05.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.484+0000 7f4c1e600640 1 Processor -- start 2024-03-23T11:21:05.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.484+0000 7f4c1e600640 1 -- start start 2024-03-23T11:21:05.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.486+0000 7f4c1e600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4c18070090 0x7f4c18070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.486+0000 7f4c1e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c180b2820 0x7f4c180b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.486+0000 7f4c1e600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c180b31b0 con 0x7f4c18143dd0 2024-03-23T11:21:05.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.486+0000 7f4c1e600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c18070a60 con 0x7f4c18070090 2024-03-23T11:21:05.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.486+0000 7f4c1e600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c180cd140 con 0x7f4c180b2820 2024-03-23T11:21:05.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.486+0000 7f4c1d200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4c18143dd0 0x7f4c1815abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57246/0 (socket says 172.21.15.62:57246) 2024-03-23T11:21:05.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.486+0000 7f4c1d200640 1 -- 172.21.15.62:0/3853777551 learned_addr learned my addr 172.21.15.62:0/3853777551 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:05.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.486+0000 7f4c17e00640 1 --2- 172.21.15.62:0/3853777551 >> v2:172.21.15.62:3300/0 conn(0x7f4c18070090 0x7f4c18070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.486+0000 7f4c1dc00640 1 --2- 172.21.15.62:0/3853777551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c180b2820 0x7f4c180b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.487+0000 7f4c17e00640 1 -- 172.21.15.62:0/3853777551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c180b2820 msgr2=0x7f4c180b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:05.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.487+0000 7f4c17e00640 1 --2- 172.21.15.62:0/3853777551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c180b2820 0x7f4c180b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.487+0000 7f4c17e00640 1 -- 172.21.15.62:0/3853777551 >> v1:172.21.15.62:6789/0 conn(0x7f4c18143dd0 legacy=0x7f4c1815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.487+0000 7f4c17e00640 1 -- 172.21.15.62:0/3853777551 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c180cd3e0 con 0x7f4c18070090 2024-03-23T11:21:05.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.488+0000 7f4c17e00640 1 --2- 172.21.15.62:0/3853777551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c18070090 0x7f4c18070520 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4c040097a0 tx=0x7f4c0402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7579b06d3a76c8e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:05.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.488+0000 7f4c17400640 1 -- 172.21.15.62:0/3853777551 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0402d040 con 0x7f4c18070090 2024-03-23T11:21:05.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.488+0000 7f4c17400640 1 -- 172.21.15.62:0/3853777551 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c04031810 con 0x7f4c18070090 2024-03-23T11:21:05.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.488+0000 7f4c17400640 1 -- 172.21.15.62:0/3853777551 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04031ba0 con 0x7f4c18070090 2024-03-23T11:21:05.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.489+0000 7f4c1e600640 1 -- 172.21.15.62:0/3853777551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c18070090 msgr2=0x7f4c18070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.489+0000 7f4c1e600640 1 --2- 172.21.15.62:0/3853777551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c18070090 0x7f4c18070520 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4c040097a0 tx=0x7f4c0402b8b0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.489+0000 7f4c1e600640 1 -- 172.21.15.62:0/3853777551 shutdown_connections 2024-03-23T11:21:05.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.489+0000 7f4c1e600640 1 --2- 172.21.15.62:0/3853777551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c180b2820 0x7f4c180b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.489+0000 7f4c1e600640 1 --2- 172.21.15.62:0/3853777551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c18070090 0x7f4c18070520 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.489+0000 7f4c1e600640 1 -- 172.21.15.62:0/3853777551 >> 172.21.15.62:0/3853777551 conn(0x7f4c18066050 msgr2=0x7f4c18069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:05.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.489+0000 7f4c1e600640 1 -- 172.21.15.62:0/3853777551 shutdown_connections 2024-03-23T11:21:05.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.489+0000 7f4c1e600640 1 -- 172.21.15.62:0/3853777551 wait complete. 2024-03-23T11:21:05.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.491+0000 7f4c1e600640 1 Processor -- start 2024-03-23T11:21:05.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.491+0000 7f4c1e600640 1 -- start start 2024-03-23T11:21:05.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.492+0000 7f4c1e600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c18070090 0x7f4c1806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.492+0000 7f4c1e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c180b2820 0x7f4c1813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.492+0000 7f4c1e600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c18143dd0 0x7f4c1813cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.492+0000 7f4c1e600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1806e120 con 0x7f4c18143dd0 2024-03-23T11:21:05.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.492+0000 7f4c1e600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c180ab9e0 con 0x7f4c18070090 2024-03-23T11:21:05.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.492+0000 7f4c1e600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1806d880 con 0x7f4c180b2820 2024-03-23T11:21:05.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.492+0000 7f4c1dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c18143dd0 0x7f4c1813cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.492+0000 7f4c1dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c18143dd0 0x7f4c1813cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46504/0 (socket says 172.21.15.62:46504) 2024-03-23T11:21:05.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.492+0000 7f4c1dc00640 1 -- 172.21.15.62:0/2263658717 learned_addr learned my addr 172.21.15.62:0/2263658717 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:05.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c1d200640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c18070090 0x7f4c1806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c1dc00640 1 -- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c180b2820 msgr2=0x7f4c1813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c17e00640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c180b2820 0x7f4c1813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c1dc00640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c180b2820 0x7f4c1813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c1dc00640 1 -- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c18070090 msgr2=0x7f4c1806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c1dc00640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c18070090 0x7f4c1806ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c1dc00640 1 -- 172.21.15.62:0/2263658717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c04009400 con 0x7f4c18143dd0 2024-03-23T11:21:05.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c1d200640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c18070090 0x7f4c1806ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:05.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c1dc00640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c18143dd0 0x7f4c1813cef0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4c10007e40 tx=0x7f4c1000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:05.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c14c00640 1 -- 172.21.15.62:0/2263658717 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c10014620 con 0x7f4c18143dd0 2024-03-23T11:21:05.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c14c00640 1 -- 172.21.15.62:0/2263658717 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1000ee90 con 0x7f4c18143dd0 2024-03-23T11:21:05.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c14c00640 1 -- 172.21.15.62:0/2263658717 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c100149e0 con 0x7f4c18143dd0 2024-03-23T11:21:05.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.493+0000 7f4c1e600640 1 -- 172.21.15.62:0/2263658717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1806db60 con 0x7f4c18143dd0 2024-03-23T11:21:05.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.494+0000 7f4c1e600640 1 -- 172.21.15.62:0/2263658717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1813b8a0 con 0x7f4c18143dd0 2024-03-23T11:21:05.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.495+0000 7f4c1e600640 1 -- 172.21.15.62:0/2263658717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c180ab840 con 0x7f4c18143dd0 2024-03-23T11:21:05.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.495+0000 7f4c14c00640 1 -- 172.21.15.62:0/2263658717 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4c10014b40 con 0x7f4c18143dd0 2024-03-23T11:21:05.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.496+0000 7f4c14c00640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4bf806a7a0 0x7f4bf806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.496+0000 7f4c14c00640 1 -- 172.21.15.62:0/2263658717 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c100897c0 con 0x7f4c18143dd0 2024-03-23T11:21:05.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.497+0000 7f4c1d200640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4bf806a7a0 0x7f4bf806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.498+0000 7f4c1d200640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4bf806a7a0 0x7f4bf806cc60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c005880 tx=0x7f4c0c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:05.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.502+0000 7f4c14c00640 1 -- 172.21.15.62:0/2263658717 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c180ab840 con 0x7f4c18143dd0 2024-03-23T11:21:05.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.702+0000 7f4c1e600640 1 -- 172.21.15.62:0/2263658717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c180ab840 con 0x7f4c18143dd0 2024-03-23T11:21:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.705+0000 7f4c14c00640 1 -- 172.21.15.62:0/2263658717 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16101 (secure 0 0 0) 0x7f4c180ab840 con 0x7f4c18143dd0 2024-03-23T11:21:05.705 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:05.705 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:20:25.735105+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:20:22.395497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T11:20:23.725162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:21:05.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 -- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4bf806a7a0 msgr2=0x7f4bf806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4bf806a7a0 0x7f4bf806cc60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c005880 tx=0x7f4c0c009040 comp rx=0 tx=0).stop 2024-03-23T11:21:05.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 -- 172.21.15.62:0/2263658717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c18143dd0 msgr2=0x7f4c1813cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c18143dd0 0x7f4c1813cef0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4c10007e40 tx=0x7f4c1000e4b0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 -- 172.21.15.62:0/2263658717 shutdown_connections 2024-03-23T11:21:05.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4bf806a7a0 0x7f4bf806cc60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c18143dd0 0x7f4c1813cef0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c180b2820 0x7f4c1813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 --2- 172.21.15.62:0/2263658717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c18070090 0x7f4c1806ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 -- 172.21.15.62:0/2263658717 >> 172.21.15.62:0/2263658717 conn(0x7f4c18066050 msgr2=0x7f4c180b1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:05.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 -- 172.21.15.62:0/2263658717 shutdown_connections 2024-03-23T11:21:05.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.708+0000 7f4c1e600640 1 -- 172.21.15.62:0/2263658717 wait complete. 2024-03-23T11:21:05.726 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:21:05.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.950+0000 7fc12e000640 1 Processor -- start 2024-03-23T11:21:05.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.950+0000 7fc12e000640 1 -- start start 2024-03-23T11:21:05.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.952+0000 7fc12e000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc1280cafa0 0x7fc1280cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.952+0000 7fc12e000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c6e0 0x7fc12806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.952+0000 7fc12e000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1280cbc60 con 0x7fc12806c300 2024-03-23T11:21:05.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.952+0000 7fc12e000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1280cd410 con 0x7fc1280cafa0 2024-03-23T11:21:05.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.952+0000 7fc12e000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1280cbf40 con 0x7fc12806c6e0 2024-03-23T11:21:05.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.953+0000 7fc126a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc1280cafa0 0x7fc1280cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.953+0000 7fc127e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c6e0 0x7fc12806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.953+0000 7fc126a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc1280cafa0 0x7fc1280cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:55932/0 (socket says 172.21.15.62:55932) 2024-03-23T11:21:05.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.953+0000 7fc127e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c6e0 0x7fc12806cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:41902/0 (socket says 172.21.15.62:41902) 2024-03-23T11:21:05.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.953+0000 7fc127e00640 1 -- 172.21.15.62:0/1660381250 learned_addr learned my addr 172.21.15.62:0/1660381250 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:05.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.953+0000 7fc127400640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fc12806c300 0x7fc1280ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:53298/0 (socket says 172.21.15.62:53298) 2024-03-23T11:21:05.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.954+0000 7fc127e00640 1 -- 172.21.15.62:0/1660381250 >> v2:172.21.15.120:3300/0 conn(0x7fc1280cafa0 msgr2=0x7fc1280cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:05.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.954+0000 7fc127e00640 1 --2- 172.21.15.62:0/1660381250 >> v2:172.21.15.120:3300/0 conn(0x7fc1280cafa0 0x7fc1280cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.954+0000 7fc127e00640 1 -- 172.21.15.62:0/1660381250 >> v1:172.21.15.120:6789/0 conn(0x7fc12806c300 legacy=0x7fc1280ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.954+0000 7fc127e00640 1 -- 172.21.15.62:0/1660381250 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1280cc220 con 0x7fc12806c6e0 2024-03-23T11:21:05.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.954+0000 7fc126a00640 1 --2- 172.21.15.62:0/1660381250 >> v2:172.21.15.120:3300/0 conn(0x7fc1280cafa0 0x7fc1280cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:21:05.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.955+0000 7fc127e00640 1 --2- 172.21.15.62:0/1660381250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c6e0 0x7fc12806cb70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc120009e50 tx=0x7fc12002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=568971db58ff9b50 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:05.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.956+0000 7fc126000640 1 -- 172.21.15.62:0/1660381250 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12002cb00 con 0x7fc12806c6e0 2024-03-23T11:21:05.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.956+0000 7fc126000640 1 -- 172.21.15.62:0/1660381250 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12002cc60 con 0x7fc12806c6e0 2024-03-23T11:21:05.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.956+0000 7fc126000640 1 -- 172.21.15.62:0/1660381250 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1200343f0 con 0x7fc12806c6e0 2024-03-23T11:21:05.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.956+0000 7fc12e000640 1 -- 172.21.15.62:0/1660381250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c6e0 msgr2=0x7fc12806cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.956+0000 7fc12e000640 1 --2- 172.21.15.62:0/1660381250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c6e0 0x7fc12806cb70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc120009e50 tx=0x7fc12002bf60 comp rx=0 tx=0).stop 2024-03-23T11:21:05.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.956+0000 7fc12e000640 1 -- 172.21.15.62:0/1660381250 shutdown_connections 2024-03-23T11:21:05.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.956+0000 7fc12e000640 1 --2- 172.21.15.62:0/1660381250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c6e0 0x7fc12806cb70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.956+0000 7fc12e000640 1 --2- 172.21.15.62:0/1660381250 >> v2:172.21.15.120:3300/0 conn(0x7fc1280cafa0 0x7fc1280cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.956+0000 7fc12e000640 1 -- 172.21.15.62:0/1660381250 >> 172.21.15.62:0/1660381250 conn(0x7fc1280b02e0 msgr2=0x7fc1280b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:05.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.957+0000 7fc12e000640 1 -- 172.21.15.62:0/1660381250 shutdown_connections 2024-03-23T11:21:05.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.957+0000 7fc12e000640 1 -- 172.21.15.62:0/1660381250 wait complete. 2024-03-23T11:21:05.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.959+0000 7fc12e000640 1 Processor -- start 2024-03-23T11:21:05.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.959+0000 7fc12e000640 1 -- start start 2024-03-23T11:21:05.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.960+0000 7fc12e000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c300 0x7fc128158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.960+0000 7fc12e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12806c6e0 0x7fc12813fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc12e000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1280cafa0 0x7fc1280c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc12e000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12813dee0 con 0x7fc12806c6e0 2024-03-23T11:21:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc12e000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12813e1c0 con 0x7fc1280cafa0 2024-03-23T11:21:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc12e000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12813e4a0 con 0x7fc12806c300 2024-03-23T11:21:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc126a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12806c6e0 0x7fc12813fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc126a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12806c6e0 0x7fc12813fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46518/0 (socket says 172.21.15.62:46518) 2024-03-23T11:21:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc126a00640 1 -- 172.21.15.62:0/1802189926 learned_addr learned my addr 172.21.15.62:0/1802189926 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc127e00640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1280cafa0 0x7fc1280c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc127400640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c300 0x7fc128158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc126a00640 1 -- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c300 msgr2=0x7fc128158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc126a00640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c300 0x7fc128158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc126a00640 1 -- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1280cafa0 msgr2=0x7fc1280c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:05.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc126a00640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1280cafa0 0x7fc1280c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:05.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.961+0000 7fc126a00640 1 -- 172.21.15.62:0/1802189926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc114009400 con 0x7fc12806c6e0 2024-03-23T11:21:05.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.962+0000 7fc126a00640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12806c6e0 0x7fc12813fe90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc11400e960 tx=0x7fc11400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:05.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.962+0000 7fc11f400640 1 -- 172.21.15.62:0/1802189926 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11400ca00 con 0x7fc12806c6e0 2024-03-23T11:21:05.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.962+0000 7fc11f400640 1 -- 172.21.15.62:0/1802189926 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11400cb60 con 0x7fc12806c6e0 2024-03-23T11:21:05.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.962+0000 7fc12e000640 1 -- 172.21.15.62:0/1802189926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc120009b30 con 0x7fc12806c6e0 2024-03-23T11:21:05.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.962+0000 7fc11f400640 1 -- 172.21.15.62:0/1802189926 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc114013470 con 0x7fc12806c6e0 2024-03-23T11:21:05.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.962+0000 7fc12e000640 1 -- 172.21.15.62:0/1802189926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1281403d0 con 0x7fc12806c6e0 2024-03-23T11:21:05.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.963+0000 7fc12e000640 1 -- 172.21.15.62:0/1802189926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1280cb640 con 0x7fc12806c6e0 2024-03-23T11:21:05.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.964+0000 7fc11f400640 1 -- 172.21.15.62:0/1802189926 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc1140135d0 con 0x7fc12806c6e0 2024-03-23T11:21:05.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.966+0000 7fc11f400640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc10806a7a0 0x7fc10806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:05.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.966+0000 7fc11f400640 1 -- 172.21.15.62:0/1802189926 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc114089040 con 0x7fc12806c6e0 2024-03-23T11:21:05.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.966+0000 7fc127400640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc10806a7a0 0x7fc10806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.967+0000 7fc127400640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc10806a7a0 0x7fc10806cc60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc118000f60 tx=0x7fc118007cc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:05.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:05.971+0000 7fc11f400640 1 -- 172.21.15.62:0/1802189926 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc114054460 con 0x7fc12806c6e0 2024-03-23T11:21:06.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.207+0000 7fc12e000640 1 -- 172.21.15.62:0/1802189926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc1280cb940 con 0x7fc12806c6e0 2024-03-23T11:21:06.208 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:21:06.210+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:21:06.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.226+0000 7fc11f400640 1 -- 172.21.15.62:0/1802189926 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc114057a80 con 0x7fc12806c6e0 2024-03-23T11:21:06.225 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:21:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.228+0000 7fc12e000640 1 -- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc10806a7a0 msgr2=0x7fc10806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.228+0000 7fc12e000640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc10806a7a0 0x7fc10806cc60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc118000f60 tx=0x7fc118007cc0 comp rx=0 tx=0).stop 2024-03-23T11:21:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.228+0000 7fc12e000640 1 -- 172.21.15.62:0/1802189926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12806c6e0 msgr2=0x7fc12813fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.228+0000 7fc12e000640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12806c6e0 0x7fc12813fe90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc11400e960 tx=0x7fc11400ee20 comp rx=0 tx=0).stop 2024-03-23T11:21:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.229+0000 7fc12e000640 1 -- 172.21.15.62:0/1802189926 shutdown_connections 2024-03-23T11:21:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.229+0000 7fc12e000640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc10806a7a0 0x7fc10806cc60 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.229+0000 7fc12e000640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc12806c6e0 0x7fc12813fe90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.229+0000 7fc12e000640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1280cafa0 0x7fc1280c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:06.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.229+0000 7fc12e000640 1 --2- 172.21.15.62:0/1802189926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc12806c300 0x7fc128158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:06.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.229+0000 7fc12e000640 1 -- 172.21.15.62:0/1802189926 >> 172.21.15.62:0/1802189926 conn(0x7fc1280b02e0 msgr2=0x7fc12813c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:06.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.229+0000 7fc12e000640 1 -- 172.21.15.62:0/1802189926 shutdown_connections 2024-03-23T11:21:06.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.229+0000 7fc12e000640 1 -- 172.21.15.62:0/1802189926 wait complete. 2024-03-23T11:21:06.246 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T11:21:06.251 INFO:tasks.ceph.mds.a.smithi062.stderr: -6201> 2024-03-23T11:20:25.558+0000 7f2e77e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:21:06.320 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:21:06.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.438+0000 7f3504200640 1 Processor -- start 2024-03-23T11:21:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.439+0000 7f3504200640 1 -- start start 2024-03-23T11:21:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.440+0000 7f3504200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc15c9e0 0x7f34fc1607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.440+0000 7f3504200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34fc06bfc0 con 0x7f34fc0cf650 2024-03-23T11:21:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.440+0000 7f3504200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34fc0b5260 con 0x7f34fc15cdc0 2024-03-23T11:21:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.440+0000 7f3504200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc0d2630 con 0x7f34fc15c9e0 2024-03-23T11:21:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.440+0000 7f3502200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f34fc0cf650 0x7f34fc15d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57260/0 (socket says 172.21.15.62:57260) 2024-03-23T11:21:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.440+0000 7f3502200640 1 -- 172.21.15.62:0/1033102013 learned_addr learned my addr 172.21.15.62:0/1033102013 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:06.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f3501800640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc15c9e0 0x7f34fc1607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:06.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785030212 0 0) 0x7f34fc06bfc0 con 0x7f34fc0cf650 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e4003520 con 0x7f34fc0cf650 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 281295038 0 0) 0x7f34e4003520 con 0x7f34fc0cf650 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc15c9e0 msgr2=0x7f34fc1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc15c9e0 0x7f34fc1607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 >> v1:172.21.15.120:6789/0 conn(0x7f34fc15cdc0 legacy=0x7f34fc0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc0ce140 con 0x7f34fc0cf650 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f34f40019f0 con 0x7f34fc0cf650 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 >> v1:172.21.15.62:6789/0 conn(0x7f34fc0cf650 legacy=0x7f34fc15d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34e40039a0 0x7f34e4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc15c9e0 0x7f34e40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.441+0000 7f34fbe00640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34e4004d00 0x7f34e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.442+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc0d2630 con 0x7f34e40039a0 2024-03-23T11:21:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.442+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e400ba60 con 0x7f34e4004d00 2024-03-23T11:21:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.442+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e400bc20 con 0x7f34fc15c9e0 2024-03-23T11:21:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.442+0000 7f3501800640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34e40039a0 0x7f34e4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.442+0000 7f3502200640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34e4004d00 0x7f34e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.442+0000 7f3500e00640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc15c9e0 0x7f34e40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.442+0000 7f3501800640 1 -- 172.21.15.62:0/1033102013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc15c9e0 msgr2=0x7f34e40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.442+0000 7f3501800640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc15c9e0 0x7f34e40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.442+0000 7f3501800640 1 -- 172.21.15.62:0/1033102013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34e4004d00 msgr2=0x7f34e4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.442+0000 7f3501800640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34e4004d00 0x7f34e4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.442+0000 7f3501800640 1 -- 172.21.15.62:0/1033102013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e4003520 con 0x7f34e40039a0 2024-03-23T11:21:06.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.443+0000 7f3501800640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34e40039a0 0x7f34e4003dc0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009a90 tx=0x7f34e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2d1fd1bc561472 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:06.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.444+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f002c690 con 0x7f34e40039a0 2024-03-23T11:21:06.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.444+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f002c7f0 con 0x7f34e40039a0 2024-03-23T11:21:06.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.444+0000 7f34fbe00640 1 -- 172.21.15.62:0/1033102013 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f002cb80 con 0x7f34e40039a0 2024-03-23T11:21:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.444+0000 7f3504200640 1 -- 172.21.15.62:0/1033102013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34e40039a0 msgr2=0x7f34e4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.444+0000 7f3504200640 1 --2- 172.21.15.62:0/1033102013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34e40039a0 0x7f34e4003dc0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009a90 tx=0x7f34e40057f0 comp rx=0 tx=0).stop 2024-03-23T11:21:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.444+0000 7f3502200640 1 -- 172.21.15.62:0/1033102013 reap_dead start 2024-03-23T11:21:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.444+0000 7f3504200640 1 -- 172.21.15.62:0/1033102013 shutdown_connections 2024-03-23T11:21:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.444+0000 7f3504200640 1 -- 172.21.15.62:0/1033102013 >> 172.21.15.62:0/1033102013 conn(0x7f34fc071570 msgr2=0x7f34fc071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.444+0000 7f3504200640 1 -- 172.21.15.62:0/1033102013 shutdown_connections 2024-03-23T11:21:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.444+0000 7f3504200640 1 -- 172.21.15.62:0/1033102013 wait complete. 2024-03-23T11:21:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.446+0000 7f3504200640 1 Processor -- start 2024-03-23T11:21:06.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.446+0000 7f3504200640 1 -- start start 2024-03-23T11:21:06.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.447+0000 7f3504200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34e40039a0 0x7f34fc15f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:06.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.447+0000 7f3504200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc150e60 0x7f34fc1506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:06.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.447+0000 7f3504200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34fc160730 0x7f34fc160bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:06.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.447+0000 7f3504200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc15f990 con 0x7f34fc160730 2024-03-23T11:21:06.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.447+0000 7f3504200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc1610f0 con 0x7f34e40039a0 2024-03-23T11:21:06.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.447+0000 7f3504200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc150c30 con 0x7f34fc150e60 2024-03-23T11:21:06.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.447+0000 7f3502200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34fc160730 0x7f34fc160bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:06.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.448+0000 7f3502200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34fc160730 0x7f34fc160bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46562/0 (socket says 172.21.15.62:46562) 2024-03-23T11:21:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.448+0000 7f3502200640 1 -- 172.21.15.62:0/3372264919 learned_addr learned my addr 172.21.15.62:0/3372264919 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.448+0000 7f3501800640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34e40039a0 0x7f34fc15f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.448+0000 7f3502200640 1 -- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc150e60 msgr2=0x7f34fc1506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.448+0000 7f3500e00640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc150e60 0x7f34fc1506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.448+0000 7f3502200640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc150e60 0x7f34fc1506f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.448+0000 7f3502200640 1 -- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34e40039a0 msgr2=0x7f34fc15f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.448+0000 7f3502200640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34e40039a0 0x7f34fc15f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.448+0000 7f3502200640 1 -- 172.21.15.62:0/3372264919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc1f2480 con 0x7f34fc160730 2024-03-23T11:21:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.448+0000 7f3502200640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34fc160730 0x7f34fc160bb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0cced0 tx=0x7f34f400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:06.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.448+0000 7f34f9600640 1 -- 172.21.15.62:0/3372264919 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f400ebe0 con 0x7f34fc160730 2024-03-23T11:21:06.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.449+0000 7f34f9600640 1 -- 172.21.15.62:0/3372264919 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f400ed40 con 0x7f34fc160730 2024-03-23T11:21:06.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.449+0000 7f34f9600640 1 -- 172.21.15.62:0/3372264919 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f4014670 con 0x7f34fc160730 2024-03-23T11:21:06.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.449+0000 7f3504200640 1 -- 172.21.15.62:0/3372264919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34fc0aad60 con 0x7f34fc160730 2024-03-23T11:21:06.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.449+0000 7f3504200640 1 -- 172.21.15.62:0/3372264919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34fc0abbc0 con 0x7f34fc160730 2024-03-23T11:21:06.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.450+0000 7f3504200640 1 -- 172.21.15.62:0/3372264919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34fc0abd50 con 0x7f34fc160730 2024-03-23T11:21:06.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.451+0000 7f34f9600640 1 -- 172.21.15.62:0/3372264919 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f34f40147d0 con 0x7f34fc160730 2024-03-23T11:21:06.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.451+0000 7f34f9600640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34dc06a7a0 0x7f34dc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:06.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.452+0000 7f34f9600640 1 -- 172.21.15.62:0/3372264919 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f34f4088bc0 con 0x7f34fc160730 2024-03-23T11:21:06.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.452+0000 7f3501800640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34dc06a7a0 0x7f34dc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:06.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.453+0000 7f3501800640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34dc06a7a0 0x7f34dc06cc60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f34f0036000 tx=0x7f34f0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:06.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.456+0000 7f34f9600640 1 -- 172.21.15.62:0/3372264919 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f34f4053ee0 con 0x7f34fc160730 2024-03-23T11:21:06.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:06.692+0000 7f3504200640 1 -- 172.21.15.62:0/3372264919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f34fc066c90 con 0x7f34fc160730 2024-03-23T11:21:07.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.230+0000 7f34f9600640 1 -- 172.21.15.62:0/3372264919 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7f34f4057500 con 0x7f34fc160730 2024-03-23T11:21:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.233+0000 7f3504200640 1 -- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34dc06a7a0 msgr2=0x7f34dc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.233+0000 7f3504200640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34dc06a7a0 0x7f34dc06cc60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f34f0036000 tx=0x7f34f0030000 comp rx=0 tx=0).stop 2024-03-23T11:21:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.233+0000 7f3504200640 1 -- 172.21.15.62:0/3372264919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34fc160730 msgr2=0x7f34fc160bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.233+0000 7f3504200640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34fc160730 0x7f34fc160bb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0cced0 tx=0x7f34f400e530 comp rx=0 tx=0).stop 2024-03-23T11:21:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.233+0000 7f3504200640 1 -- 172.21.15.62:0/3372264919 shutdown_connections 2024-03-23T11:21:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.233+0000 7f3504200640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34dc06a7a0 0x7f34dc06cc60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.233+0000 7f3504200640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34fc160730 0x7f34fc160bb0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.233+0000 7f3504200640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34fc150e60 0x7f34fc1506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.233+0000 7f3504200640 1 --2- 172.21.15.62:0/3372264919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34e40039a0 0x7f34fc15f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.233+0000 7f3504200640 1 -- 172.21.15.62:0/3372264919 >> 172.21.15.62:0/3372264919 conn(0x7f34fc071570 msgr2=0x7f34fc071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.234+0000 7f3504200640 1 -- 172.21.15.62:0/3372264919 shutdown_connections 2024-03-23T11:21:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.234+0000 7f3504200640 1 -- 172.21.15.62:0/3372264919 wait complete. 2024-03-23T11:21:07.257 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T11:21:07.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.480+0000 7f5681e00640 1 Processor -- start 2024-03-23T11:21:07.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.480+0000 7f5681e00640 1 -- start start 2024-03-23T11:21:07.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.481+0000 7f5681e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f567c06dd30 0x7f567c06c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:07.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.482+0000 7f5681e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f567c0ce690 0x7f567c0ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:07.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.482+0000 7f5681e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c15cfd0 con 0x7f567c06d950 2024-03-23T11:21:07.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.482+0000 7f5681e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c0ceff0 con 0x7f567c0ce690 2024-03-23T11:21:07.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.482+0000 7f5681e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c15d220 con 0x7f567c06dd30 2024-03-23T11:21:07.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.482+0000 7f5681400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f567c0ce690 0x7f567c0ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:07.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.482+0000 7f5681400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f567c0ce690 0x7f567c0ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:59522/0 (socket says 172.21.15.62:59522) 2024-03-23T11:21:07.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.482+0000 7f5681400640 1 -- 172.21.15.62:0/2396594389 learned_addr learned my addr 172.21.15.62:0/2396594389 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:07.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.482+0000 7f567be00640 1 --2- 172.21.15.62:0/2396594389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f567c06dd30 0x7f567c06c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:07.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.483+0000 7f5681400640 1 -- 172.21.15.62:0/2396594389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f567c06dd30 msgr2=0x7f567c06c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:07.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.483+0000 7f5681400640 1 --2- 172.21.15.62:0/2396594389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f567c06dd30 0x7f567c06c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:07.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.483+0000 7f5681400640 1 -- 172.21.15.62:0/2396594389 >> v1:172.21.15.120:6789/0 conn(0x7f567c06d950 legacy=0x7f567c070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:07.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.483+0000 7f5681400640 1 -- 172.21.15.62:0/2396594389 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c15d500 con 0x7f567c0ce690 2024-03-23T11:21:07.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.484+0000 7f5681400640 1 --2- 172.21.15.62:0/2396594389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f567c0ce690 0x7f567c0ceab0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5674009e50 tx=0x7f567402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44a84ee8f9095760 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:07.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.485+0000 7f567b400640 1 -- 172.21.15.62:0/2396594389 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567402ca70 con 0x7f567c0ce690 2024-03-23T11:21:07.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.485+0000 7f567b400640 1 -- 172.21.15.62:0/2396594389 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567402cbd0 con 0x7f567c0ce690 2024-03-23T11:21:07.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.485+0000 7f567b400640 1 -- 172.21.15.62:0/2396594389 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56740343f0 con 0x7f567c0ce690 2024-03-23T11:21:07.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.485+0000 7f5681e00640 1 -- 172.21.15.62:0/2396594389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f567c0ce690 msgr2=0x7f567c0ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:07.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.485+0000 7f5681e00640 1 --2- 172.21.15.62:0/2396594389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f567c0ce690 0x7f567c0ceab0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5674009e50 tx=0x7f567402bed0 comp rx=0 tx=0).stop 2024-03-23T11:21:07.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.486+0000 7f5681e00640 1 -- 172.21.15.62:0/2396594389 shutdown_connections 2024-03-23T11:21:07.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.486+0000 7f5681e00640 1 --2- 172.21.15.62:0/2396594389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f567c0ce690 0x7f567c0ceab0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:07.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.486+0000 7f5681e00640 1 --2- 172.21.15.62:0/2396594389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f567c06dd30 0x7f567c06c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:07.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.486+0000 7f5681e00640 1 -- 172.21.15.62:0/2396594389 >> 172.21.15.62:0/2396594389 conn(0x7f567c0b0280 msgr2=0x7f567c0b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:07.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.486+0000 7f5681e00640 1 -- 172.21.15.62:0/2396594389 shutdown_connections 2024-03-23T11:21:07.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.486+0000 7f5681e00640 1 -- 172.21.15.62:0/2396594389 wait complete. 2024-03-23T11:21:07.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.487+0000 7f5681e00640 1 Processor -- start 2024-03-23T11:21:07.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.488+0000 7f5681e00640 1 -- start start 2024-03-23T11:21:07.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.488+0000 7f5681e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567c06d950 0x7f567c09bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:07.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.488+0000 7f5681e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f567c06dd30 0x7f567c1ef3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5681e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f567c0ce690 0x7f567c0cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5681e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c14dfd0 con 0x7f567c06d950 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5681e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c09c1b0 con 0x7f567c06dd30 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5681e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c155b00 con 0x7f567c0ce690 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5680a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567c06d950 0x7f567c09bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5680a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567c06d950 0x7f567c09bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53042/0 (socket says 172.21.15.62:53042) 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5680a00640 1 -- 172.21.15.62:0/4112647172 learned_addr learned my addr 172.21.15.62:0/4112647172 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f567be00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f567c06dd30 0x7f567c1ef3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5681400640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f567c0ce690 0x7f567c0cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5680a00640 1 -- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f567c0ce690 msgr2=0x7f567c0cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5680a00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f567c0ce690 0x7f567c0cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5680a00640 1 -- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f567c06dd30 msgr2=0x7f567c1ef3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:07.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5680a00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f567c06dd30 0x7f567c1ef3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:07.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f5680a00640 1 -- 172.21.15.62:0/4112647172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5674009b30 con 0x7f567c06d950 2024-03-23T11:21:07.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.489+0000 7f567be00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f567c06dd30 0x7f567c1ef3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:07.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.490+0000 7f5680a00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567c06d950 0x7f567c09bbd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f567000c450 tx=0x7f567000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:07.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.490+0000 7f5678c00640 1 -- 172.21.15.62:0/4112647172 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56700052d0 con 0x7f567c06d950 2024-03-23T11:21:07.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.490+0000 7f5678c00640 1 -- 172.21.15.62:0/4112647172 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5670005430 con 0x7f567c06d950 2024-03-23T11:21:07.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.490+0000 7f5678c00640 1 -- 172.21.15.62:0/4112647172 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670012870 con 0x7f567c06d950 2024-03-23T11:21:07.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.490+0000 7f5681e00640 1 -- 172.21.15.62:0/4112647172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567c155de0 con 0x7f567c06d950 2024-03-23T11:21:07.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.490+0000 7f5681e00640 1 -- 172.21.15.62:0/4112647172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567c1563a0 con 0x7f567c06d950 2024-03-23T11:21:07.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.491+0000 7f5681e00640 1 -- 172.21.15.62:0/4112647172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567c066bb0 con 0x7f567c06d950 2024-03-23T11:21:07.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.492+0000 7f5678c00640 1 -- 172.21.15.62:0/4112647172 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f56700129d0 con 0x7f567c06d950 2024-03-23T11:21:07.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.493+0000 7f5678c00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f565c06a7a0 0x7f565c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:07.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.493+0000 7f5678c00640 1 -- 172.21.15.62:0/4112647172 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5670088980 con 0x7f567c06d950 2024-03-23T11:21:07.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.494+0000 7f567be00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f565c06a7a0 0x7f565c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:07.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.495+0000 7f567be00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f565c06a7a0 0x7f565c06cc60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f567c0caf10 tx=0x7f5668008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:07.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.498+0000 7f5678c00640 1 -- 172.21.15.62:0/4112647172 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5670053ca0 con 0x7f567c06d950 2024-03-23T11:21:07.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:07.696+0000 7f5681e00640 1 -- 172.21.15.62:0/4112647172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f567c072170 con 0x7f567c06d950 2024-03-23T11:21:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.376+0000 7f5678c00640 1 -- 172.21.15.62:0/4112647172 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v38) v1 ==== 181+0+0 (secure 0 0 0) 0x7f56700572c0 con 0x7f567c06d950 2024-03-23T11:21:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' removed 2024-03-23T11:21:08.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.381+0000 7f5681e00640 1 -- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f565c06a7a0 msgr2=0x7f565c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.381+0000 7f5681e00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f565c06a7a0 0x7f565c06cc60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f567c0caf10 tx=0x7f5668008040 comp rx=0 tx=0).stop 2024-03-23T11:21:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.381+0000 7f5681e00640 1 -- 172.21.15.62:0/4112647172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567c06d950 msgr2=0x7f567c09bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.381+0000 7f5681e00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567c06d950 0x7f567c09bbd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f567000c450 tx=0x7f567000c910 comp rx=0 tx=0).stop 2024-03-23T11:21:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.381+0000 7f5681e00640 1 -- 172.21.15.62:0/4112647172 shutdown_connections 2024-03-23T11:21:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.381+0000 7f5681e00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f565c06a7a0 0x7f565c06cc60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.381+0000 7f5681e00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f567c0ce690 0x7f567c0cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.381+0000 7f5681e00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f567c06dd30 0x7f567c1ef3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.381+0000 7f5681e00640 1 --2- 172.21.15.62:0/4112647172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567c06d950 0x7f567c09bbd0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.381+0000 7f5681e00640 1 -- 172.21.15.62:0/4112647172 >> 172.21.15.62:0/4112647172 conn(0x7f567c0b0280 msgr2=0x7f567c06ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.382+0000 7f5681e00640 1 -- 172.21.15.62:0/4112647172 shutdown_connections 2024-03-23T11:21:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.382+0000 7f5681e00640 1 -- 172.21.15.62:0/4112647172 wait complete. 2024-03-23T11:21:08.404 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T11:21:08.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.620+0000 7f71c8a00640 1 Processor -- start 2024-03-23T11:21:08.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.620+0000 7f71c8a00640 1 -- start start 2024-03-23T11:21:08.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.622+0000 7f71c8a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71c415cba0 0x7f71c4099710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:08.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.622+0000 7f71c8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c40b1240 0x7f71c40cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:08.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.622+0000 7f71c8a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71c40cb040 0x7f71c40989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:08.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.622+0000 7f71c8a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c40ce780 con 0x7f71c40cb040 2024-03-23T11:21:08.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.622+0000 7f71c8a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c40ced00 con 0x7f71c415cba0 2024-03-23T11:21:08.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.622+0000 7f71c8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c406bb50 con 0x7f71c40b1240 2024-03-23T11:21:08.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.622+0000 7f71c3200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71c40cb040 0x7f71c40989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:08.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.623+0000 7f71c3200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71c40cb040 0x7f71c40989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53044/0 (socket says 172.21.15.62:53044) 2024-03-23T11:21:08.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.623+0000 7f71c3200640 1 -- 172.21.15.62:0/3083979651 learned_addr learned my addr 172.21.15.62:0/3083979651 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.623+0000 7f71c2800640 1 --2- 172.21.15.62:0/3083979651 >> v2:172.21.15.120:3300/0 conn(0x7f71c415cba0 0x7f71c4099710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.623+0000 7f71c1e00640 1 --2- 172.21.15.62:0/3083979651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c40b1240 0x7f71c40cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.623+0000 7f71c3200640 1 -- 172.21.15.62:0/3083979651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c40b1240 msgr2=0x7f71c40cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.623+0000 7f71c3200640 1 --2- 172.21.15.62:0/3083979651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c40b1240 0x7f71c40cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.623+0000 7f71c3200640 1 -- 172.21.15.62:0/3083979651 >> v2:172.21.15.120:3300/0 conn(0x7f71c415cba0 msgr2=0x7f71c4099710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.623+0000 7f71c3200640 1 --2- 172.21.15.62:0/3083979651 >> v2:172.21.15.120:3300/0 conn(0x7f71c415cba0 0x7f71c4099710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.623+0000 7f71c3200640 1 -- 172.21.15.62:0/3083979651 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c406be30 con 0x7f71c40cb040 2024-03-23T11:21:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.624+0000 7f71c3200640 1 --2- 172.21.15.62:0/3083979651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40cb040 0x7f71c40989d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f71c40d0480 tx=0x7f71bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906feade058bdddf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c1400640 1 -- 172.21.15.62:0/3083979651 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc0317e0 con 0x7f71c40cb040 2024-03-23T11:21:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c1400640 1 -- 172.21.15.62:0/3083979651 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71bc031940 con 0x7f71c40cb040 2024-03-23T11:21:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c1400640 1 -- 172.21.15.62:0/3083979651 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc031cd0 con 0x7f71c40cb040 2024-03-23T11:21:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c8a00640 1 -- 172.21.15.62:0/3083979651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40cb040 msgr2=0x7f71c40989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c8a00640 1 --2- 172.21.15.62:0/3083979651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40cb040 0x7f71c40989d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f71c40d0480 tx=0x7f71bc02b880 comp rx=0 tx=0).stop 2024-03-23T11:21:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c8a00640 1 -- 172.21.15.62:0/3083979651 shutdown_connections 2024-03-23T11:21:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c8a00640 1 --2- 172.21.15.62:0/3083979651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40cb040 0x7f71c40989d0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c8a00640 1 --2- 172.21.15.62:0/3083979651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c40b1240 0x7f71c40cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c8a00640 1 --2- 172.21.15.62:0/3083979651 >> v2:172.21.15.120:3300/0 conn(0x7f71c415cba0 0x7f71c4099710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c8a00640 1 -- 172.21.15.62:0/3083979651 >> 172.21.15.62:0/3083979651 conn(0x7f71c40b1fa0 msgr2=0x7f71c40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c8a00640 1 -- 172.21.15.62:0/3083979651 shutdown_connections 2024-03-23T11:21:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.625+0000 7f71c8a00640 1 -- 172.21.15.62:0/3083979651 wait complete. 2024-03-23T11:21:08.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.627+0000 7f71c8a00640 1 Processor -- start 2024-03-23T11:21:08.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.627+0000 7f71c8a00640 1 -- start start 2024-03-23T11:21:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.628+0000 7f71c8a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40b1240 0x7f71c40b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.628+0000 7f71c8a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c40cb040 0x7f71c415f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.628+0000 7f71c8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415cba0 0x7f71c415fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.628+0000 7f71c8a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c41f2490 con 0x7f71c40b1240 2024-03-23T11:21:08.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.628+0000 7f71c8a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40c6190 con 0x7f71c40cb040 2024-03-23T11:21:08.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.628+0000 7f71c8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40c50f0 con 0x7f71c415cba0 2024-03-23T11:21:08.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.628+0000 7f71c2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40b1240 0x7f71c40b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:08.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.628+0000 7f71c2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40b1240 0x7f71c40b5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53056/0 (socket says 172.21.15.62:53056) 2024-03-23T11:21:08.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.628+0000 7f71c2800640 1 -- 172.21.15.62:0/3848158737 learned_addr learned my addr 172.21.15.62:0/3848158737 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:08.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.629+0000 7f71c3200640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415cba0 0x7f71c415fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:08.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.629+0000 7f71c2800640 1 -- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415cba0 msgr2=0x7f71c415fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:08.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.629+0000 7f71c2800640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415cba0 0x7f71c415fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:08.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.629+0000 7f71c2800640 1 -- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c40cb040 msgr2=0x7f71c415f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:08.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.629+0000 7f71c2800640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c40cb040 0x7f71c415f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:08.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.629+0000 7f71c2800640 1 -- 172.21.15.62:0/3848158737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc0094d0 con 0x7f71c40b1240 2024-03-23T11:21:08.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.629+0000 7f71c3200640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415cba0 0x7f71c415fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.629+0000 7f71c2800640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40b1240 0x7f71c40b5370 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f71b4007e40 tx=0x7f71b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.629+0000 7f71baa00640 1 -- 172.21.15.62:0/3848158737 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b4014620 con 0x7f71c40b1240 2024-03-23T11:21:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.630+0000 7f71baa00640 1 -- 172.21.15.62:0/3848158737 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b400ee90 con 0x7f71c40b1240 2024-03-23T11:21:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.630+0000 7f71baa00640 1 -- 172.21.15.62:0/3848158737 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b40149a0 con 0x7f71c40b1240 2024-03-23T11:21:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.630+0000 7f71c8a00640 1 -- 172.21.15.62:0/3848158737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c40c53d0 con 0x7f71c40b1240 2024-03-23T11:21:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.630+0000 7f71c8a00640 1 -- 172.21.15.62:0/3848158737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c40c5990 con 0x7f71c40b1240 2024-03-23T11:21:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.631+0000 7f71c8a00640 1 -- 172.21.15.62:0/3848158737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c40672c0 con 0x7f71c40b1240 2024-03-23T11:21:08.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.632+0000 7f71baa00640 1 -- 172.21.15.62:0/3848158737 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f71b4014b00 con 0x7f71c40b1240 2024-03-23T11:21:08.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.633+0000 7f71baa00640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a006a7d0 0x7f71a006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:08.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.633+0000 7f71baa00640 1 -- 172.21.15.62:0/3848158737 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f71b4089500 con 0x7f71c40b1240 2024-03-23T11:21:08.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.633+0000 7f71c1e00640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a006a7d0 0x7f71a006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:08.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.635+0000 7f71c1e00640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a006a7d0 0x7f71a006cc90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f71c415f1a0 tx=0x7f71ac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:08.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.638+0000 7f71baa00640 1 -- 172.21.15.62:0/3848158737 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f71b4054a80 con 0x7f71c40b1240 2024-03-23T11:21:08.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:08.839+0000 7f71c8a00640 1 -- 172.21.15.62:0/3848158737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f71c40b2b90 con 0x7f71c40b1240 2024-03-23T11:21:09.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.400+0000 7f71baa00640 1 -- 172.21.15.62:0/3848158737 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v39) v1 ==== 169+0+0 (secure 0 0 0) 0x7f71b4014db0 con 0x7f71c40b1240 2024-03-23T11:21:09.399 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' removed 2024-03-23T11:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.405+0000 7f71c8a00640 1 -- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a006a7d0 msgr2=0x7f71a006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.405+0000 7f71c8a00640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a006a7d0 0x7f71a006cc90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f71c415f1a0 tx=0x7f71ac008040 comp rx=0 tx=0).stop 2024-03-23T11:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.405+0000 7f71c8a00640 1 -- 172.21.15.62:0/3848158737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40b1240 msgr2=0x7f71c40b5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.405+0000 7f71c8a00640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40b1240 0x7f71c40b5370 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f71b4007e40 tx=0x7f71b400e4b0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.406+0000 7f71c8a00640 1 -- 172.21.15.62:0/3848158737 shutdown_connections 2024-03-23T11:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.406+0000 7f71c8a00640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a006a7d0 0x7f71a006cc90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.406+0000 7f71c8a00640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415cba0 0x7f71c415fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.406+0000 7f71c8a00640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c40cb040 0x7f71c415f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.406+0000 7f71c8a00640 1 --2- 172.21.15.62:0/3848158737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40b1240 0x7f71c40b5370 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.406+0000 7f71c8a00640 1 -- 172.21.15.62:0/3848158737 >> 172.21.15.62:0/3848158737 conn(0x7f71c40b1fa0 msgr2=0x7f71c4160bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.406+0000 7f71c8a00640 1 -- 172.21.15.62:0/3848158737 shutdown_connections 2024-03-23T11:21:09.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.406+0000 7f71c8a00640 1 -- 172.21.15.62:0/3848158737 wait complete. 2024-03-23T11:21:09.429 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T11:21:09.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.642+0000 7fa352e00640 1 Processor -- start 2024-03-23T11:21:09.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.642+0000 7fa352e00640 1 -- start start 2024-03-23T11:21:09.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.644+0000 7fa352e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa34c0d30b0 0x7fa34c0ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:09.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.644+0000 7fa352e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c0cd280 con 0x7fa34c15c8d0 2024-03-23T11:21:09.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.644+0000 7fa352e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c0cd520 con 0x7fa34c0d0ca0 2024-03-23T11:21:09.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.644+0000 7fa352e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34c0d0b00 con 0x7fa34c0d30b0 2024-03-23T11:21:09.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.644+0000 7fa351a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa34c15c8d0 0x7fa34c0d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47940/0 (socket says 172.21.15.62:47940) 2024-03-23T11:21:09.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.644+0000 7fa351a00640 1 -- 172.21.15.62:0/289880589 learned_addr learned my addr 172.21.15.62:0/289880589 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:09.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.644+0000 7fa351000640 1 --2- 172.21.15.62:0/289880589 >> v2:172.21.15.62:3300/0 conn(0x7fa34c0d30b0 0x7fa34c0ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:09.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.645+0000 7fa34be00640 1 -- 172.21.15.62:0/289880589 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923488546 0 0) 0x7fa34c0cd280 con 0x7fa34c15c8d0 2024-03-23T11:21:09.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.645+0000 7fa34be00640 1 -- 172.21.15.62:0/289880589 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa334003520 con 0x7fa34c15c8d0 2024-03-23T11:21:09.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.645+0000 7fa351000640 1 -- 172.21.15.62:0/289880589 >> v1:172.21.15.120:6789/0 conn(0x7fa34c0d0ca0 legacy=0x7fa34c0d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:09.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.645+0000 7fa351000640 1 -- 172.21.15.62:0/289880589 >> v1:172.21.15.62:6789/0 conn(0x7fa34c15c8d0 legacy=0x7fa34c0d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.645+0000 7fa351000640 1 -- 172.21.15.62:0/289880589 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c0cd7c0 con 0x7fa34c0d30b0 2024-03-23T11:21:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.646+0000 7fa351000640 1 --2- 172.21.15.62:0/289880589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa34c0d30b0 0x7fa34c0ce2a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa3380097a0 tx=0x7fa33802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a8b682fa2c8f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.646+0000 7fa34be00640 1 -- 172.21.15.62:0/289880589 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33802e070 con 0x7fa34c0d30b0 2024-03-23T11:21:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.646+0000 7fa34be00640 1 -- 172.21.15.62:0/289880589 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3380315a0 con 0x7fa34c0d30b0 2024-03-23T11:21:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.646+0000 7fa34be00640 1 -- 172.21.15.62:0/289880589 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338031930 con 0x7fa34c0d30b0 2024-03-23T11:21:09.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.647+0000 7fa352e00640 1 -- 172.21.15.62:0/289880589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa34c0d30b0 msgr2=0x7fa34c0ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:09.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.647+0000 7fa352e00640 1 --2- 172.21.15.62:0/289880589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa34c0d30b0 0x7fa34c0ce2a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa3380097a0 tx=0x7fa33802b840 comp rx=0 tx=0).stop 2024-03-23T11:21:09.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.647+0000 7fa352e00640 1 -- 172.21.15.62:0/289880589 shutdown_connections 2024-03-23T11:21:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.647+0000 7fa352e00640 1 --2- 172.21.15.62:0/289880589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa34c0d30b0 0x7fa34c0ce2a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.647+0000 7fa352e00640 1 -- 172.21.15.62:0/289880589 >> 172.21.15.62:0/289880589 conn(0x7fa34c089f60 msgr2=0x7fa34c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.647+0000 7fa352e00640 1 -- 172.21.15.62:0/289880589 shutdown_connections 2024-03-23T11:21:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.647+0000 7fa352e00640 1 -- 172.21.15.62:0/289880589 wait complete. 2024-03-23T11:21:09.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.649+0000 7fa352e00640 1 Processor -- start 2024-03-23T11:21:09.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.649+0000 7fa352e00640 1 -- start start 2024-03-23T11:21:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.650+0000 7fa352e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c0d0ca0 0x7fa34c15f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.650+0000 7fa352e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c0d30b0 0x7fa34c1f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.650+0000 7fa352e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa34c15c8d0 0x7fa34c15d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.650+0000 7fa352e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0c7720 con 0x7fa34c15c8d0 2024-03-23T11:21:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.650+0000 7fa352e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0c6da0 con 0x7fa34c0d30b0 2024-03-23T11:21:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.650+0000 7fa352e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0c7060 con 0x7fa34c0d0ca0 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.650+0000 7fa352400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa34c15c8d0 0x7fa34c15d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.650+0000 7fa352400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa34c15c8d0 0x7fa34c15d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53072/0 (socket says 172.21.15.62:53072) 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.651+0000 7fa352400640 1 -- 172.21.15.62:0/581874642 learned_addr learned my addr 172.21.15.62:0/581874642 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.651+0000 7fa351a00640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c0d0ca0 0x7fa34c15f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.651+0000 7fa351000640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c0d30b0 0x7fa34c1f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.651+0000 7fa352400640 1 -- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c0d0ca0 msgr2=0x7fa34c15f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.651+0000 7fa352400640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c0d0ca0 0x7fa34c15f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.651+0000 7fa352400640 1 -- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c0d30b0 msgr2=0x7fa34c1f16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.651+0000 7fa352400640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c0d30b0 0x7fa34c1f16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.651+0000 7fa352400640 1 -- 172.21.15.62:0/581874642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa338009400 con 0x7fa34c15c8d0 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.651+0000 7fa351000640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c0d30b0 0x7fa34c1f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.651+0000 7fa352400640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa34c15c8d0 0x7fa34c15d6b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa34400c780 tx=0x7fa34400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:09.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.652+0000 7fa349600640 1 -- 172.21.15.62:0/581874642 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344004b60 con 0x7fa34c15c8d0 2024-03-23T11:21:09.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.652+0000 7fa349600640 1 -- 172.21.15.62:0/581874642 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34400ce90 con 0x7fa34c15c8d0 2024-03-23T11:21:09.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.652+0000 7fa349600640 1 -- 172.21.15.62:0/581874642 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34400a850 con 0x7fa34c15c8d0 2024-03-23T11:21:09.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.652+0000 7fa352e00640 1 -- 172.21.15.62:0/581874642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34c0c7340 con 0x7fa34c15c8d0 2024-03-23T11:21:09.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.652+0000 7fa352e00640 1 -- 172.21.15.62:0/581874642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c0ceb30 con 0x7fa34c15c8d0 2024-03-23T11:21:09.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.653+0000 7fa352e00640 1 -- 172.21.15.62:0/581874642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34c0c7860 con 0x7fa34c15c8d0 2024-03-23T11:21:09.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.654+0000 7fa349600640 1 -- 172.21.15.62:0/581874642 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa34400a9b0 con 0x7fa34c15c8d0 2024-03-23T11:21:09.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.655+0000 7fa349600640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa32c06a7d0 0x7fa32c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:09.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.656+0000 7fa349600640 1 -- 172.21.15.62:0/581874642 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa344088cb0 con 0x7fa34c15c8d0 2024-03-23T11:21:09.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.656+0000 7fa351a00640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa32c06a7d0 0x7fa32c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:09.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.657+0000 7fa351a00640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa32c06a7d0 0x7fa32c06cc90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa3400011e0 tx=0x7fa340009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:09.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.660+0000 7fa349600640 1 -- 172.21.15.62:0/581874642 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa34c0c7860 con 0x7fa34c15c8d0 2024-03-23T11:21:09.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.859+0000 7fa352e00640 1 -- 172.21.15.62:0/581874642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa34c0c7860 con 0x7fa34c15c8d0 2024-03-23T11:21:09.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.868+0000 7fa349600640 1 -- 172.21.15.62:0/581874642 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa34c0c7860 con 0x7fa34c15c8d0 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 -- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa32c06a7d0 msgr2=0x7fa32c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa32c06a7d0 0x7fa32c06cc90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa3400011e0 tx=0x7fa340009040 comp rx=0 tx=0).stop 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 -- 172.21.15.62:0/581874642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa34c15c8d0 msgr2=0x7fa34c15d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa34c15c8d0 0x7fa34c15d6b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa34400c780 tx=0x7fa34400cc40 comp rx=0 tx=0).stop 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 -- 172.21.15.62:0/581874642 shutdown_connections 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa32c06a7d0 0x7fa32c06cc90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa34c15c8d0 0x7fa34c15d6b0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c0d30b0 0x7fa34c1f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 --2- 172.21.15.62:0/581874642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa34c0d0ca0 0x7fa34c15f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 -- 172.21.15.62:0/581874642 >> 172.21.15.62:0/581874642 conn(0x7fa34c089f60 msgr2=0x7fa34c15bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 -- 172.21.15.62:0/581874642 shutdown_connections 2024-03-23T11:21:09.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:09.871+0000 7fa352e00640 1 -- 172.21.15.62:0/581874642 wait complete. 2024-03-23T11:21:09.889 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-03-23T11:21:10.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.111+0000 7f9a56200640 1 Processor -- start 2024-03-23T11:21:10.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.111+0000 7f9a56200640 1 -- start start 2024-03-23T11:21:10.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.112+0000 7f9a56200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9a50143dd0 0x7f9a5015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:10.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.112+0000 7f9a56200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a50070090 0x7f9a500704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:10.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.112+0000 7f9a56200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9a500b2820 0x7f9a500b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:10.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.112+0000 7f9a56200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a50070a30 con 0x7f9a50143dd0 2024-03-23T11:21:10.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.112+0000 7f9a56200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a500cd140 con 0x7f9a500b2820 2024-03-23T11:21:10.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.112+0000 7f9a56200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a500b3180 con 0x7f9a50070090 2024-03-23T11:21:10.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.113+0000 7f9a54e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9a50143dd0 0x7f9a5015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:10.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.113+0000 7f9a54e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9a50143dd0 0x7f9a5015abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53074/0 (socket says 172.21.15.62:53074) 2024-03-23T11:21:10.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.113+0000 7f9a54e00640 1 -- 172.21.15.62:0/1079387642 learned_addr learned my addr 172.21.15.62:0/1079387642 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:10.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.113+0000 7f9a55800640 1 --2- 172.21.15.62:0/1079387642 >> v2:172.21.15.120:3300/0 conn(0x7f9a500b2820 0x7f9a500b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:10.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.113+0000 7f9a4fe00640 1 --2- 172.21.15.62:0/1079387642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a50070090 0x7f9a500704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:10.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.113+0000 7f9a54e00640 1 -- 172.21.15.62:0/1079387642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a50070090 msgr2=0x7f9a500704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:10.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.114+0000 7f9a54e00640 1 --2- 172.21.15.62:0/1079387642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a50070090 0x7f9a500704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:10.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.114+0000 7f9a54e00640 1 -- 172.21.15.62:0/1079387642 >> v2:172.21.15.120:3300/0 conn(0x7f9a500b2820 msgr2=0x7f9a500b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:10.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.114+0000 7f9a54e00640 1 --2- 172.21.15.62:0/1079387642 >> v2:172.21.15.120:3300/0 conn(0x7f9a500b2820 0x7f9a500b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:10.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.114+0000 7f9a54e00640 1 -- 172.21.15.62:0/1079387642 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a500cd3e0 con 0x7f9a50143dd0 2024-03-23T11:21:10.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.114+0000 7f9a4fe00640 1 --2- 172.21.15.62:0/1079387642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a50070090 0x7f9a500704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:10.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.115+0000 7f9a54e00640 1 --2- 172.21.15.62:0/1079387642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a50143dd0 0x7f9a5015abe0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9a50071500 tx=0x7f9a4002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0c2950f2aa2efa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:10.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.115+0000 7f9a4f400640 1 -- 172.21.15.62:0/1079387642 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4002ca70 con 0x7f9a50143dd0 2024-03-23T11:21:10.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.115+0000 7f9a4f400640 1 -- 172.21.15.62:0/1079387642 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4002cbd0 con 0x7f9a50143dd0 2024-03-23T11:21:10.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.115+0000 7f9a4f400640 1 -- 172.21.15.62:0/1079387642 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a400343f0 con 0x7f9a50143dd0 2024-03-23T11:21:10.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.116+0000 7f9a56200640 1 -- 172.21.15.62:0/1079387642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a50143dd0 msgr2=0x7f9a5015abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:10.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.116+0000 7f9a56200640 1 --2- 172.21.15.62:0/1079387642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a50143dd0 0x7f9a5015abe0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9a50071500 tx=0x7f9a4002bed0 comp rx=0 tx=0).stop 2024-03-23T11:21:10.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.116+0000 7f9a56200640 1 -- 172.21.15.62:0/1079387642 shutdown_connections 2024-03-23T11:21:10.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.116+0000 7f9a56200640 1 --2- 172.21.15.62:0/1079387642 >> v2:172.21.15.120:3300/0 conn(0x7f9a500b2820 0x7f9a500b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:10.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.116+0000 7f9a56200640 1 --2- 172.21.15.62:0/1079387642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a50070090 0x7f9a500704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:10.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.116+0000 7f9a56200640 1 --2- 172.21.15.62:0/1079387642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a50143dd0 0x7f9a5015abe0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:10.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.116+0000 7f9a56200640 1 -- 172.21.15.62:0/1079387642 >> 172.21.15.62:0/1079387642 conn(0x7f9a50066050 msgr2=0x7f9a50069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:10.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.116+0000 7f9a56200640 1 -- 172.21.15.62:0/1079387642 shutdown_connections 2024-03-23T11:21:10.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.116+0000 7f9a56200640 1 -- 172.21.15.62:0/1079387642 wait complete. 2024-03-23T11:21:10.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.118+0000 7f9a56200640 1 Processor -- start 2024-03-23T11:21:10.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.118+0000 7f9a56200640 1 -- start start 2024-03-23T11:21:10.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.119+0000 7f9a56200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a50070090 0x7f9a50136270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:10.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.119+0000 7f9a56200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a500b2820 0x7f9a5006c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:10.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.119+0000 7f9a56200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a50143dd0 0x7f9a501f5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:10.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.119+0000 7f9a56200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5006d680 con 0x7f9a50143dd0 2024-03-23T11:21:10.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.119+0000 7f9a56200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5006db10 con 0x7f9a500b2820 2024-03-23T11:21:10.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.119+0000 7f9a56200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5006ca70 con 0x7f9a50070090 2024-03-23T11:21:10.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.120+0000 7f9a55800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a50143dd0 0x7f9a501f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:10.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.120+0000 7f9a55800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a50143dd0 0x7f9a501f5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53082/0 (socket says 172.21.15.62:53082) 2024-03-23T11:21:10.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.120+0000 7f9a55800640 1 -- 172.21.15.62:0/1492691492 learned_addr learned my addr 172.21.15.62:0/1492691492 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:10.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.120+0000 7f9a54e00640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a50070090 0x7f9a50136270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:10.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.120+0000 7f9a4fe00640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a500b2820 0x7f9a5006c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:10.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.120+0000 7f9a55800640 1 -- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a50070090 msgr2=0x7f9a50136270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:10.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.120+0000 7f9a55800640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a50070090 0x7f9a50136270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:10.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.120+0000 7f9a55800640 1 -- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a500b2820 msgr2=0x7f9a5006c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:10.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.120+0000 7f9a55800640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a500b2820 0x7f9a5006c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:10.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.120+0000 7f9a55800640 1 -- 172.21.15.62:0/1492691492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a40009b30 con 0x7f9a50143dd0 2024-03-23T11:21:10.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.120+0000 7f9a55800640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a50143dd0 0x7f9a501f5280 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800d800 tx=0x7f9a4800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:10.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.121+0000 7f9a4cc00640 1 -- 172.21.15.62:0/1492691492 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4800b6c0 con 0x7f9a50143dd0 2024-03-23T11:21:10.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.121+0000 7f9a4cc00640 1 -- 172.21.15.62:0/1492691492 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4800b840 con 0x7f9a50143dd0 2024-03-23T11:21:10.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.121+0000 7f9a56200640 1 -- 172.21.15.62:0/1492691492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5006cd50 con 0x7f9a50143dd0 2024-03-23T11:21:10.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.121+0000 7f9a4cc00640 1 -- 172.21.15.62:0/1492691492 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4800bbf0 con 0x7f9a50143dd0 2024-03-23T11:21:10.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.121+0000 7f9a56200640 1 -- 172.21.15.62:0/1492691492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5006d310 con 0x7f9a50143dd0 2024-03-23T11:21:10.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.122+0000 7f9a56200640 1 -- 172.21.15.62:0/1492691492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5006d830 con 0x7f9a50143dd0 2024-03-23T11:21:10.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.123+0000 7f9a4cc00640 1 -- 172.21.15.62:0/1492691492 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9a4800bd50 con 0x7f9a50143dd0 2024-03-23T11:21:10.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.124+0000 7f9a4cc00640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a3006a7a0 0x7f9a3006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:10.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.124+0000 7f9a4cc00640 1 -- 172.21.15.62:0/1492691492 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9a48088ea0 con 0x7f9a50143dd0 2024-03-23T11:21:10.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.125+0000 7f9a54e00640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a3006a7a0 0x7f9a3006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:10.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.126+0000 7f9a54e00640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a3006a7a0 0x7f9a3006cc60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9a40035000 tx=0x7f9a40030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:10.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.128+0000 7f9a4cc00640 1 -- 172.21.15.62:0/1492691492 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9a5006d830 con 0x7f9a50143dd0 2024-03-23T11:21:10.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:10.400+0000 7f9a56200640 1 -- 172.21.15.62:0/1492691492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) v1 -- 0x7f9a50066bb0 con 0x7f9a50143dd0 2024-03-23T11:21:11.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.409+0000 7f9a4cc00640 1 -- 172.21.15.62:0/1492691492 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v137) v1 ==== 142+0+0 (secure 0 0 0) 0x7f9a480545c0 con 0x7f9a50143dd0 2024-03-23T11:21:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.412+0000 7f9a56200640 1 -- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a3006a7a0 msgr2=0x7f9a3006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.412+0000 7f9a56200640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a3006a7a0 0x7f9a3006cc60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9a40035000 tx=0x7f9a40030000 comp rx=0 tx=0).stop 2024-03-23T11:21:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.412+0000 7f9a56200640 1 -- 172.21.15.62:0/1492691492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a50143dd0 msgr2=0x7f9a501f5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.413+0000 7f9a56200640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a50143dd0 0x7f9a501f5280 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800d800 tx=0x7f9a4800dcc0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.413+0000 7f9a56200640 1 -- 172.21.15.62:0/1492691492 shutdown_connections 2024-03-23T11:21:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.413+0000 7f9a56200640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a3006a7a0 0x7f9a3006cc60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.413+0000 7f9a56200640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a50143dd0 0x7f9a501f5280 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.413+0000 7f9a56200640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a500b2820 0x7f9a5006c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.413+0000 7f9a56200640 1 --2- 172.21.15.62:0/1492691492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a50070090 0x7f9a50136270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.413+0000 7f9a56200640 1 -- 172.21.15.62:0/1492691492 >> 172.21.15.62:0/1492691492 conn(0x7f9a50066050 msgr2=0x7f9a500ae670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.413+0000 7f9a56200640 1 -- 172.21.15.62:0/1492691492 shutdown_connections 2024-03-23T11:21:11.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.413+0000 7f9a56200640 1 -- 172.21.15.62:0/1492691492 wait complete. 2024-03-23T11:21:11.435 INFO:tasks.cephfs_test_runner:test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-03-23T11:21:11.435 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) That the MDS can repair an inodes backtrace in the data pool 2024-03-23T11:21:11.435 INFO:tasks.cephfs_test_runner:test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-23T11:21:11.435 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-03-23T11:21:11.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.648+0000 7f28d9800640 1 Processor -- start 2024-03-23T11:21:11.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.649+0000 7f28d9800640 1 -- start start 2024-03-23T11:21:11.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.650+0000 7f28d9800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f28d4070090 0x7f28d4070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:11.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.650+0000 7f28d9800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40b2820 0x7f28d40b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:11.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.650+0000 7f28d9800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d40b31b0 con 0x7f28d4143dd0 2024-03-23T11:21:11.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.650+0000 7f28d9800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d4070a60 con 0x7f28d4070090 2024-03-23T11:21:11.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.650+0000 7f28d9800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40cd140 con 0x7f28d40b2820 2024-03-23T11:21:11.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.650+0000 7f28d3e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f28d4143dd0 0x7f28d415abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47946/0 (socket says 172.21.15.62:47946) 2024-03-23T11:21:11.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.650+0000 7f28d3e00640 1 -- 172.21.15.62:0/4175688562 learned_addr learned my addr 172.21.15.62:0/4175688562 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:11.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28cbe00640 1 --2- 172.21.15.62:0/4175688562 >> v2:172.21.15.120:3300/0 conn(0x7f28d4070090 0x7f28d4070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:11.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d8e00640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40b2820 0x7f28d40b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:11.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502276797 0 0) 0x7f28d40b31b0 con 0x7f28d4143dd0 2024-03-23T11:21:11.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b8003520 con 0x7f28d4143dd0 2024-03-23T11:21:11.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2830673027 0 0) 0x7f28b8003520 con 0x7f28d4143dd0 2024-03-23T11:21:11.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40b2820 msgr2=0x7f28d40b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:11.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40b2820 0x7f28d40b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 >> v2:172.21.15.120:3300/0 conn(0x7f28d4070090 msgr2=0x7f28d4070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:11.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 --2- 172.21.15.62:0/4175688562 >> v2:172.21.15.120:3300/0 conn(0x7f28d4070090 0x7f28d4070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d40cd3e0 con 0x7f28d4143dd0 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f28c4001670 con 0x7f28d4143dd0 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 >> v1:172.21.15.62:6789/0 conn(0x7f28d4143dd0 legacy=0x7f28d415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40b2820 0x7f28b8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28b8003ef0 0x7f28b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28b8004d00 0x7f28b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d4070a60 con 0x7f28b8004d00 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40cd140 con 0x7f28b8003ef0 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.651+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b800b980 con 0x7f28d40b2820 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.652+0000 7f28d8e00640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28b8004d00 0x7f28b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.652+0000 7f28d3e00640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40b2820 0x7f28b8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.652+0000 7f28cbe00640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28b8003ef0 0x7f28b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.652+0000 7f28d8e00640 1 -- 172.21.15.62:0/4175688562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40b2820 msgr2=0x7f28b8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.652+0000 7f28d8e00640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d40b2820 0x7f28b8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.652+0000 7f28d8e00640 1 -- 172.21.15.62:0/4175688562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28b8003ef0 msgr2=0x7f28b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.652+0000 7f28d8e00640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28b8003ef0 0x7f28b80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.652+0000 7f28d8e00640 1 -- 172.21.15.62:0/4175688562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b8003520 con 0x7f28b8004d00 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.653+0000 7f28d8e00640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28b8004d00 0x7f28b8005180 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009fc0 tx=0x7f28b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60cc5d520ed59b3d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.654+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc02cc00 con 0x7f28b8004d00 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.654+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc00f040 con 0x7f28b8004d00 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.654+0000 7f28d3400640 1 -- 172.21.15.62:0/4175688562 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc034440 con 0x7f28b8004d00 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.654+0000 7f28d9800640 1 -- 172.21.15.62:0/4175688562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28b8004d00 msgr2=0x7f28b8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.654+0000 7f28d9800640 1 --2- 172.21.15.62:0/4175688562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28b8004d00 0x7f28b8005180 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009fc0 tx=0x7f28b80057f0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.654+0000 7f28d8e00640 1 -- 172.21.15.62:0/4175688562 reap_dead start 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.654+0000 7f28d9800640 1 -- 172.21.15.62:0/4175688562 shutdown_connections 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.654+0000 7f28d9800640 1 -- 172.21.15.62:0/4175688562 >> 172.21.15.62:0/4175688562 conn(0x7f28d4066050 msgr2=0x7f28d4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.654+0000 7f28d9800640 1 -- 172.21.15.62:0/4175688562 shutdown_connections 2024-03-23T11:21:11.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.654+0000 7f28d9800640 1 -- 172.21.15.62:0/4175688562 wait complete. 2024-03-23T11:21:11.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.656+0000 7f28d9800640 1 Processor -- start 2024-03-23T11:21:11.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.656+0000 7f28d9800640 1 -- start start 2024-03-23T11:21:11.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.657+0000 7f28d9800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d406cc20 0x7f28d4150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:11.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.657+0000 7f28d9800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28d413a930 0x7f28d413ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:11.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.657+0000 7f28d9800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d406d000 0x7f28d406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:11.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.657+0000 7f28d9800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d413b4f0 con 0x7f28d413a930 2024-03-23T11:21:11.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.657+0000 7f28d9800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d413b2d0 con 0x7f28d406cc20 2024-03-23T11:21:11.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.657+0000 7f28d9800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d413b740 con 0x7f28d406d000 2024-03-23T11:21:11.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28cbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28d413a930 0x7f28d413ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:11.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28cbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28d413a930 0x7f28d413ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53106/0 (socket says 172.21.15.62:53106) 2024-03-23T11:21:11.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28cbe00640 1 -- 172.21.15.62:0/1028268714 learned_addr learned my addr 172.21.15.62:0/1028268714 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:11.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28d8e00640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d406d000 0x7f28d406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28cbe00640 1 -- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d406d000 msgr2=0x7f28d406d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28d3e00640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d406cc20 0x7f28d4150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28cbe00640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d406d000 0x7f28d406d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28cbe00640 1 -- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d406cc20 msgr2=0x7f28d4150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28cbe00640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d406cc20 0x7f28d4150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28cbe00640 1 -- 172.21.15.62:0/1028268714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d413ba20 con 0x7f28d413a930 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28d3e00640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d406cc20 0x7f28d4150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.658+0000 7f28cbe00640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28d413a930 0x7f28d413ad90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f28bc011600 tx=0x7f28bc011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.659+0000 7f28d0c00640 1 -- 172.21.15.62:0/1028268714 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc00f920 con 0x7f28d413a930 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.659+0000 7f28d0c00640 1 -- 172.21.15.62:0/1028268714 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28bc00fa80 con 0x7f28d413a930 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.659+0000 7f28d0c00640 1 -- 172.21.15.62:0/1028268714 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc00fdf0 con 0x7f28d413a930 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.659+0000 7f28d9800640 1 -- 172.21.15.62:0/1028268714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d413bd00 con 0x7f28d413a930 2024-03-23T11:21:11.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.659+0000 7f28d9800640 1 -- 172.21.15.62:0/1028268714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d415a1a0 con 0x7f28d413a930 2024-03-23T11:21:11.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.660+0000 7f28d9800640 1 -- 172.21.15.62:0/1028268714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d415a3c0 con 0x7f28d413a930 2024-03-23T11:21:11.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.661+0000 7f28d0c00640 1 -- 172.21.15.62:0/1028268714 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f28bc015460 con 0x7f28d413a930 2024-03-23T11:21:11.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.662+0000 7f28d0c00640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28b006a7a0 0x7f28b006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:11.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.662+0000 7f28d0c00640 1 -- 172.21.15.62:0/1028268714 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f28bc089010 con 0x7f28d413a930 2024-03-23T11:21:11.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.663+0000 7f28d3e00640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28b006a7a0 0x7f28b006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:11.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.664+0000 7f28d3e00640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28b006a7a0 0x7f28b006cc60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f28c4007d90 tx=0x7f28c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:11.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.667+0000 7f28d0c00640 1 -- 172.21.15.62:0/1028268714 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f28d415a3c0 con 0x7f28d413a930 2024-03-23T11:21:11.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:11.938+0000 7f28d9800640 1 -- 172.21.15.62:0/1028268714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) v1 -- 0x7f28d415a3c0 con 0x7f28d413a930 2024-03-23T11:21:12.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.413+0000 7f28d0c00640 1 -- 172.21.15.62:0/1028268714 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v138) v1 ==== 152+0+0 (secure 0 0 0) 0x7f28d415a3c0 con 0x7f28d413a930 2024-03-23T11:21:12.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 -- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28b006a7a0 msgr2=0x7f28b006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:12.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28b006a7a0 0x7f28b006cc60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f28c4007d90 tx=0x7f28c4009040 comp rx=0 tx=0).stop 2024-03-23T11:21:12.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 -- 172.21.15.62:0/1028268714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28d413a930 msgr2=0x7f28d413ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:12.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28d413a930 0x7f28d413ad90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f28bc011600 tx=0x7f28bc011ac0 comp rx=0 tx=0).stop 2024-03-23T11:21:12.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 -- 172.21.15.62:0/1028268714 shutdown_connections 2024-03-23T11:21:12.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28b006a7a0 0x7f28b006cc60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:12.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28d413a930 0x7f28d413ad90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:12.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28d406d000 0x7f28d406d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:12.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 --2- 172.21.15.62:0/1028268714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28d406cc20 0x7f28d4150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:12.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 -- 172.21.15.62:0/1028268714 >> 172.21.15.62:0/1028268714 conn(0x7f28d4066050 msgr2=0x7f28d40ae300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:12.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 -- 172.21.15.62:0/1028268714 shutdown_connections 2024-03-23T11:21:12.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.417+0000 7f28d9800640 1 -- 172.21.15.62:0/1028268714 wait complete. 2024-03-23T11:21:12.440 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T11:21:12.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.665+0000 7efeffe00640 1 Processor -- start 2024-03-23T11:21:12.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.665+0000 7efeffe00640 1 -- start start 2024-03-23T11:21:12.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.667+0000 7efeffe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efef80cd7c0 0x7efef80b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:12.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.667+0000 7efeffe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efef806c300 0x7efef80cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:12.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.667+0000 7efeffe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef80cad20 con 0x7efef806dca0 2024-03-23T11:21:12.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.667+0000 7efeffe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef806c770 con 0x7efef806c300 2024-03-23T11:21:12.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.667+0000 7efeffe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef806e130 con 0x7efef80cd7c0 2024-03-23T11:21:12.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.667+0000 7efefde00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efef806c300 0x7efef80cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:12.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.667+0000 7efefde00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efef806c300 0x7efef80cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53116/0 (socket says 172.21.15.62:53116) 2024-03-23T11:21:12.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.667+0000 7efefde00640 1 -- 172.21.15.62:0/1801016814 learned_addr learned my addr 172.21.15.62:0/1801016814 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:12.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.668+0000 7efefd400640 1 --2- 172.21.15.62:0/1801016814 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efef80cd7c0 0x7efef80b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:12.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.668+0000 7efefde00640 1 -- 172.21.15.62:0/1801016814 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efef80cd7c0 msgr2=0x7efef80b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:12.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.668+0000 7efefde00640 1 --2- 172.21.15.62:0/1801016814 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efef80cd7c0 0x7efef80b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:12.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.668+0000 7efefde00640 1 -- 172.21.15.62:0/1801016814 >> v1:172.21.15.120:6789/0 conn(0x7efef806dca0 legacy=0x7efef80cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:12.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.668+0000 7efefde00640 1 -- 172.21.15.62:0/1801016814 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef815a8a0 con 0x7efef806c300 2024-03-23T11:21:12.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.669+0000 7efefde00640 1 --2- 172.21.15.62:0/1801016814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efef806c300 0x7efef80cb490 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7efef0009e50 tx=0x7efef002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95c1d395797fc82 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:12.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.670+0000 7efef7e00640 1 -- 172.21.15.62:0/1801016814 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef002c9e0 con 0x7efef806c300 2024-03-23T11:21:12.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.670+0000 7efef7e00640 1 -- 172.21.15.62:0/1801016814 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef002cb40 con 0x7efef806c300 2024-03-23T11:21:12.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.670+0000 7efef7e00640 1 -- 172.21.15.62:0/1801016814 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef00343f0 con 0x7efef806c300 2024-03-23T11:21:12.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.670+0000 7efeffe00640 1 -- 172.21.15.62:0/1801016814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efef806c300 msgr2=0x7efef80cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:12.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.670+0000 7efeffe00640 1 --2- 172.21.15.62:0/1801016814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efef806c300 0x7efef80cb490 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7efef0009e50 tx=0x7efef002be40 comp rx=0 tx=0).stop 2024-03-23T11:21:12.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.670+0000 7efeffe00640 1 -- 172.21.15.62:0/1801016814 shutdown_connections 2024-03-23T11:21:12.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.670+0000 7efeffe00640 1 --2- 172.21.15.62:0/1801016814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efef806c300 0x7efef80cb490 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:12.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.670+0000 7efeffe00640 1 --2- 172.21.15.62:0/1801016814 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efef80cd7c0 0x7efef80b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:12.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.670+0000 7efeffe00640 1 -- 172.21.15.62:0/1801016814 >> 172.21.15.62:0/1801016814 conn(0x7efef80b02e0 msgr2=0x7efef80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:12.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.670+0000 7efeffe00640 1 -- 172.21.15.62:0/1801016814 shutdown_connections 2024-03-23T11:21:12.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.670+0000 7efeffe00640 1 -- 172.21.15.62:0/1801016814 wait complete. 2024-03-23T11:21:12.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.672+0000 7efeffe00640 1 Processor -- start 2024-03-23T11:21:12.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.672+0000 7efeffe00640 1 -- start start 2024-03-23T11:21:12.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.673+0000 7efeffe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efef806c300 0x7efef81528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:12.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.673+0000 7efeffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efef80cd7c0 0x7efef8156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:12.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.673+0000 7efeffe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efef8151c80 0x7efef815be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:12.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.673+0000 7efeffe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef8150900 con 0x7efef80cd7c0 2024-03-23T11:21:12.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.673+0000 7efeffe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef8155c00 con 0x7efef806c300 2024-03-23T11:21:12.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.673+0000 7efeffe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef8155060 con 0x7efef8151c80 2024-03-23T11:21:12.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.673+0000 7efefca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efef80cd7c0 0x7efef8156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:12.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.673+0000 7efefca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efef80cd7c0 0x7efef8156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53126/0 (socket says 172.21.15.62:53126) 2024-03-23T11:21:12.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.673+0000 7efefca00640 1 -- 172.21.15.62:0/1046920396 learned_addr learned my addr 172.21.15.62:0/1046920396 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:12.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.674+0000 7efefd400640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efef806c300 0x7efef81528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:12.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.674+0000 7efefde00640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efef8151c80 0x7efef815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:12.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.674+0000 7efefca00640 1 -- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efef8151c80 msgr2=0x7efef815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:12.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.674+0000 7efefca00640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efef8151c80 0x7efef815be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:12.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.674+0000 7efefca00640 1 -- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efef806c300 msgr2=0x7efef81528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:12.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.674+0000 7efefca00640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efef806c300 0x7efef81528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:12.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.674+0000 7efefca00640 1 -- 172.21.15.62:0/1046920396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef0009b30 con 0x7efef80cd7c0 2024-03-23T11:21:12.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.674+0000 7efefde00640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efef8151c80 0x7efef815be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:12.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.674+0000 7efefca00640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efef80cd7c0 0x7efef8156090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7efee400c460 tx=0x7efee400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:12.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.675+0000 7efef5600640 1 -- 172.21.15.62:0/1046920396 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee4012830 con 0x7efef80cd7c0 2024-03-23T11:21:12.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.675+0000 7efef5600640 1 -- 172.21.15.62:0/1046920396 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee40129d0 con 0x7efef80cd7c0 2024-03-23T11:21:12.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.675+0000 7efef5600640 1 -- 172.21.15.62:0/1046920396 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee4012d80 con 0x7efef80cd7c0 2024-03-23T11:21:12.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.675+0000 7efeffe00640 1 -- 172.21.15.62:0/1046920396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef8152df0 con 0x7efef80cd7c0 2024-03-23T11:21:12.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.675+0000 7efeffe00640 1 -- 172.21.15.62:0/1046920396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef81555b0 con 0x7efef80cd7c0 2024-03-23T11:21:12.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.676+0000 7efeffe00640 1 -- 172.21.15.62:0/1046920396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef8155d90 con 0x7efef80cd7c0 2024-03-23T11:21:12.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.677+0000 7efef5600640 1 -- 172.21.15.62:0/1046920396 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7efee401a3b0 con 0x7efef80cd7c0 2024-03-23T11:21:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.678+0000 7efef5600640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efed806a7a0 0x7efed806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.678+0000 7efef5600640 1 -- 172.21.15.62:0/1046920396 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7efee40887f0 con 0x7efef80cd7c0 2024-03-23T11:21:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.679+0000 7efefd400640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efed806a7a0 0x7efed806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:12.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.680+0000 7efefd400640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efed806a7a0 0x7efed806cc60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efeec006fd0 tx=0x7efeec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:12.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.684+0000 7efef5600640 1 -- 172.21.15.62:0/1046920396 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efee4053e70 con 0x7efef80cd7c0 2024-03-23T11:21:12.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.881+0000 7efeffe00640 1 -- 172.21.15.62:0/1046920396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7efef8066c90 con 0x7efef80cd7c0 2024-03-23T11:21:13.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.888+0000 7efef5600640 1 -- 172.21.15.62:0/1046920396 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7efee4057490 con 0x7efef80cd7c0 2024-03-23T11:21:13.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 -- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efed806a7a0 msgr2=0x7efed806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:13.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efed806a7a0 0x7efed806cc60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efeec006fd0 tx=0x7efeec008040 comp rx=0 tx=0).stop 2024-03-23T11:21:13.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 -- 172.21.15.62:0/1046920396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efef80cd7c0 msgr2=0x7efef8156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:13.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efef80cd7c0 0x7efef8156090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7efee400c460 tx=0x7efee400c920 comp rx=0 tx=0).stop 2024-03-23T11:21:13.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 -- 172.21.15.62:0/1046920396 shutdown_connections 2024-03-23T11:21:13.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efed806a7a0 0x7efed806cc60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:13.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efef80cd7c0 0x7efef8156090 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:13.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efef8151c80 0x7efef815be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:13.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 --2- 172.21.15.62:0/1046920396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efef806c300 0x7efef81528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:13.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 -- 172.21.15.62:0/1046920396 >> 172.21.15.62:0/1046920396 conn(0x7efef80b02e0 msgr2=0x7efef80b2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:13.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 -- 172.21.15.62:0/1046920396 shutdown_connections 2024-03-23T11:21:13.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:12.891+0000 7efeffe00640 1 -- 172.21.15.62:0/1046920396 wait complete. 2024-03-23T11:21:13.114 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:21:13.115 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:21:13.240 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:21:13.358 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:21:13.358 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:21:13.358 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:21:13.418 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:21:13.418 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:21:13.419 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:21:13.477 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:21:13.477 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:21:13.477 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:21:13.537 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:21:13.537 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:21:13.537 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:21:13.593 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:21:13.593 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T11:21:13.594 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:21:13.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.826+0000 7f4d0e400640 1 Processor -- start 2024-03-23T11:21:13.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.826+0000 7f4d0e400640 1 -- start start 2024-03-23T11:21:13.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.827+0000 7f4d0e400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4d080990e0 0x7f4d080996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:13.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.827+0000 7f4d0e400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4d0806bfe0 0x7f4d080b1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:13.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.827+0000 7f4d0e400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d080cab10 con 0x7f4d0815cbb0 2024-03-23T11:21:13.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.827+0000 7f4d0e400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d080cb3b0 con 0x7f4d080990e0 2024-03-23T11:21:13.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.827+0000 7f4d0e400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d080afe20 con 0x7f4d0806bfe0 2024-03-23T11:21:13.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.828+0000 7f4d07e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4d080990e0 0x7f4d080996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:13.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.828+0000 7f4d07e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4d080990e0 0x7f4d080996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53130/0 (socket says 172.21.15.62:53130) 2024-03-23T11:21:13.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.828+0000 7f4d07e00640 1 -- 172.21.15.62:0/744209994 learned_addr learned my addr 172.21.15.62:0/744209994 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:13.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.828+0000 7f4d0da00640 1 --2- 172.21.15.62:0/744209994 >> v2:172.21.15.120:3300/0 conn(0x7f4d0806bfe0 0x7f4d080b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:13.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.828+0000 7f4d07e00640 1 -- 172.21.15.62:0/744209994 >> v2:172.21.15.120:3300/0 conn(0x7f4d0806bfe0 msgr2=0x7f4d080b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:13.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.828+0000 7f4d07e00640 1 --2- 172.21.15.62:0/744209994 >> v2:172.21.15.120:3300/0 conn(0x7f4d0806bfe0 0x7f4d080b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:13.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.828+0000 7f4d07e00640 1 -- 172.21.15.62:0/744209994 >> v1:172.21.15.120:6789/0 conn(0x7f4d0815cbb0 legacy=0x7f4d080989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:13.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.828+0000 7f4d07e00640 1 -- 172.21.15.62:0/744209994 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0806c490 con 0x7f4d080990e0 2024-03-23T11:21:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.829+0000 7f4d07e00640 1 --2- 172.21.15.62:0/744209994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d080990e0 0x7f4d080996b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4d0806dbd0 tx=0x7f4cf002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcd6111f20a6e07f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.830+0000 7f4d07400640 1 -- 172.21.15.62:0/744209994 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0031870 con 0x7f4d080990e0 2024-03-23T11:21:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.830+0000 7f4d07400640 1 -- 172.21.15.62:0/744209994 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf00319d0 con 0x7f4d080990e0 2024-03-23T11:21:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.830+0000 7f4d07400640 1 -- 172.21.15.62:0/744209994 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0031d60 con 0x7f4d080990e0 2024-03-23T11:21:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.831+0000 7f4d0e400640 1 -- 172.21.15.62:0/744209994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d080990e0 msgr2=0x7f4d080996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.831+0000 7f4d0e400640 1 --2- 172.21.15.62:0/744209994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d080990e0 0x7f4d080996b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4d0806dbd0 tx=0x7f4cf002b910 comp rx=0 tx=0).stop 2024-03-23T11:21:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.831+0000 7f4d0e400640 1 -- 172.21.15.62:0/744209994 shutdown_connections 2024-03-23T11:21:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.831+0000 7f4d0e400640 1 --2- 172.21.15.62:0/744209994 >> v2:172.21.15.120:3300/0 conn(0x7f4d0806bfe0 0x7f4d080b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.831+0000 7f4d0e400640 1 --2- 172.21.15.62:0/744209994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d080990e0 0x7f4d080996b0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.831+0000 7f4d0e400640 1 -- 172.21.15.62:0/744209994 >> 172.21.15.62:0/744209994 conn(0x7f4d080b1fb0 msgr2=0x7f4d080b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.831+0000 7f4d0e400640 1 -- 172.21.15.62:0/744209994 shutdown_connections 2024-03-23T11:21:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.831+0000 7f4d0e400640 1 -- 172.21.15.62:0/744209994 wait complete. 2024-03-23T11:21:13.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.832+0000 7f4d0e400640 1 Processor -- start 2024-03-23T11:21:13.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.833+0000 7f4d0e400640 1 -- start start 2024-03-23T11:21:13.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0e400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d0806bfe0 0x7f4d080c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:13.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0e400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d080990e0 0x7f4d080c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:13.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0e400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0815cbb0 0x7f4d0814c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:13.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0e400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080b5f40 con 0x7f4d0815cbb0 2024-03-23T11:21:13.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0e400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0815dd10 con 0x7f4d080990e0 2024-03-23T11:21:13.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0e400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d080c5a70 con 0x7f4d0806bfe0 2024-03-23T11:21:13.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0815cbb0 0x7f4d0814c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:13.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0815cbb0 0x7f4d0814c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53140/0 (socket says 172.21.15.62:53140) 2024-03-23T11:21:13.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0da00640 1 -- 172.21.15.62:0/1543308964 learned_addr learned my addr 172.21.15.62:0/1543308964 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:13.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d07e00640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d080990e0 0x7f4d080c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0d000640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d0806bfe0 0x7f4d080c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0da00640 1 -- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d0806bfe0 msgr2=0x7f4d080c4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0da00640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d0806bfe0 0x7f4d080c4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0da00640 1 -- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d080990e0 msgr2=0x7f4d080c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0da00640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d080990e0 0x7f4d080c5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0da00640 1 -- 172.21.15.62:0/1543308964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf00094d0 con 0x7f4d0815cbb0 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d07e00640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d080990e0 0x7f4d080c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.834+0000 7f4d0d000640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d0806bfe0 0x7f4d080c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.835+0000 7f4d0da00640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0815cbb0 0x7f4d0814c1c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000d800 tx=0x7f4d0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.835+0000 7f4d04c00640 1 -- 172.21.15.62:0/1543308964 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0000b6c0 con 0x7f4d0815cbb0 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.835+0000 7f4d04c00640 1 -- 172.21.15.62:0/1543308964 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0000b840 con 0x7f4d0815cbb0 2024-03-23T11:21:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.835+0000 7f4d04c00640 1 -- 172.21.15.62:0/1543308964 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0000bbf0 con 0x7f4d0815cbb0 2024-03-23T11:21:13.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.835+0000 7f4d0e400640 1 -- 172.21.15.62:0/1543308964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0815d3e0 con 0x7f4d0815cbb0 2024-03-23T11:21:13.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.835+0000 7f4d0e400640 1 -- 172.21.15.62:0/1543308964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0815d9a0 con 0x7f4d0815cbb0 2024-03-23T11:21:13.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.836+0000 7f4d0e400640 1 -- 172.21.15.62:0/1543308964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0815d570 con 0x7f4d0815cbb0 2024-03-23T11:21:13.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.837+0000 7f4d04c00640 1 -- 172.21.15.62:0/1543308964 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4d0000bd50 con 0x7f4d0815cbb0 2024-03-23T11:21:13.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.838+0000 7f4d04c00640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4ce406a7a0 0x7f4ce406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:13.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.838+0000 7f4d04c00640 1 -- 172.21.15.62:0/1543308964 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4d000890f0 con 0x7f4d0815cbb0 2024-03-23T11:21:13.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.839+0000 7f4d0d000640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4ce406a7a0 0x7f4ce406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:13.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.840+0000 7f4d0d000640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4ce406a7a0 0x7f4ce406cc60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8007e90 tx=0x7f4cf80073a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:13.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:13.843+0000 7f4d04c00640 1 -- 172.21.15.62:0/1543308964 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4d00054770 con 0x7f4d0815cbb0 2024-03-23T11:21:14.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.081+0000 7f4d0e400640 1 -- 172.21.15.62:0/1543308964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d0815de80 con 0x7f4d0815cbb0 2024-03-23T11:21:14.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.082+0000 7f4d04c00640 1 -- 172.21.15.62:0/1543308964 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f4d00057d90 con 0x7f4d0815cbb0 2024-03-23T11:21:14.081 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:14.082 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3421279564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3421279564},{"type":"v1","addr":"172.21.15.120:6835","nonce":3421279564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2052651236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2052651236},{"type":"v1","addr":"172.21.15.120:6837","nonce":2052651236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5193,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/1599029025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1599029025},{"type":"v1","addr":"172.21.15.62:6834","nonce":1599029025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1387107251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":1387107251},{"type":"v1","addr":"172.21.15.62:6835","nonce":1387107251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[]} 2024-03-23T11:21:14.082 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-03-23T11:21:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 -- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4ce406a7a0 msgr2=0x7f4ce406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4ce406a7a0 0x7f4ce406cc60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8007e90 tx=0x7f4cf80073a0 comp rx=0 tx=0).stop 2024-03-23T11:21:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 -- 172.21.15.62:0/1543308964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0815cbb0 msgr2=0x7f4d0814c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0815cbb0 0x7f4d0814c1c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000d800 tx=0x7f4d0000dcc0 comp rx=0 tx=0).stop 2024-03-23T11:21:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 -- 172.21.15.62:0/1543308964 shutdown_connections 2024-03-23T11:21:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4ce406a7a0 0x7f4ce406cc60 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0815cbb0 0x7f4d0814c1c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d080990e0 0x7f4d080c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 --2- 172.21.15.62:0/1543308964 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d0806bfe0 0x7f4d080c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:14.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 -- 172.21.15.62:0/1543308964 >> 172.21.15.62:0/1543308964 conn(0x7f4d080b1fb0 msgr2=0x7f4d080b2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:14.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 -- 172.21.15.62:0/1543308964 shutdown_connections 2024-03-23T11:21:14.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:14.085+0000 7f4d0e400640 1 -- 172.21.15.62:0/1543308964 wait complete. 2024-03-23T11:21:14.105 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T11:21:14.105 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T11:21:14.106 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T11:21:14.106 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:21:14.106 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T11:21:14.106 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T11:21:14.106 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T11:21:14.106 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:21:14.106 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T11:21:14.107 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T11:21:14.107 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T11:21:14.107 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:21:14.107 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T11:21:14.107 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T11:21:14.107 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T11:21:14.107 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:21:14.127 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T11:21:14.127 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T11:21:14.128 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T11:21:14.128 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T11:21:14.174 INFO:tasks.ceph.mds.d:Started 2024-03-23T11:21:14.174 DEBUG:teuthology.parallel:result is None 2024-03-23T11:21:14.175 INFO:tasks.ceph.mds.c:Started 2024-03-23T11:21:14.175 DEBUG:teuthology.parallel:result is None 2024-03-23T11:21:14.349 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-23T11:21:14.372 INFO:tasks.ceph.mds.c.smithi062.stdout:starting mds.c at 2024-03-23T11:21:14.970 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:21:14.978 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:21:17.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-23T11:21:17.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-23T11:21:20.127 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T11:21:20.127 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T11:21:20.129 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T11:21:20.130 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T11:21:20.135 INFO:tasks.ceph.mds.a:Started 2024-03-23T11:21:20.135 DEBUG:teuthology.parallel:result is None 2024-03-23T11:21:20.137 INFO:tasks.ceph.mds.b:Started 2024-03-23T11:21:20.137 DEBUG:teuthology.parallel:result is None 2024-03-23T11:21:20.138 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T11:21:20.305 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:21:20.321 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-23T11:21:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.342+0000 7feac9800640 1 Processor -- start 2024-03-23T11:21:20.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.342+0000 7feac9800640 1 -- start start 2024-03-23T11:21:20.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.343+0000 7feac9800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feac406c300 0x7feac40ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:20.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.343+0000 7feac9800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feac406c6e0 0x7feac406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:20.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.343+0000 7feac9800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac40cbc60 con 0x7feac40cafa0 2024-03-23T11:21:20.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.343+0000 7feac9800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac40cbf40 con 0x7feac406c300 2024-03-23T11:21:20.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.343+0000 7feac9800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac40cc220 con 0x7feac406c6e0 2024-03-23T11:21:20.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.344+0000 7feac2000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7feac40cafa0 0x7feac40cb430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43712/0 (socket says 172.21.15.62:43712) 2024-03-23T11:21:20.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.344+0000 7feac2000640 1 -- 172.21.15.62:0/2422398129 learned_addr learned my addr 172.21.15.62:0/2422398129 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:20.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.344+0000 7feac3400640 1 --2- 172.21.15.62:0/2422398129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feac406c6e0 0x7feac406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:20.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.344+0000 7feac2a00640 1 --2- 172.21.15.62:0/2422398129 >> v2:172.21.15.120:3300/0 conn(0x7feac406c300 0x7feac40ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:20.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.344+0000 7feac1600640 1 -- 172.21.15.62:0/2422398129 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668310191 0 0) 0x7feac40cbc60 con 0x7feac40cafa0 2024-03-23T11:21:20.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.344+0000 7feac1600640 1 -- 172.21.15.62:0/2422398129 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa8003520 con 0x7feac40cafa0 2024-03-23T11:21:20.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.344+0000 7feac3400640 1 -- 172.21.15.62:0/2422398129 >> v2:172.21.15.120:3300/0 conn(0x7feac406c300 msgr2=0x7feac40ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:20.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.344+0000 7feac3400640 1 --2- 172.21.15.62:0/2422398129 >> v2:172.21.15.120:3300/0 conn(0x7feac406c300 0x7feac40ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:20.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.344+0000 7feac3400640 1 -- 172.21.15.62:0/2422398129 >> v1:172.21.15.62:6789/0 conn(0x7feac40cafa0 legacy=0x7feac40cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:20.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.344+0000 7feac3400640 1 -- 172.21.15.62:0/2422398129 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac40cc500 con 0x7feac406c6e0 2024-03-23T11:21:20.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.344+0000 7feac2a00640 1 --2- 172.21.15.62:0/2422398129 >> v2:172.21.15.120:3300/0 conn(0x7feac406c300 0x7feac40ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:21:20.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.345+0000 7feac3400640 1 --2- 172.21.15.62:0/2422398129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feac406c6e0 0x7feac406cb70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feabc009d90 tx=0x7feabc02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1843dbb43d0b3a38 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:20.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.346+0000 7feac1600640 1 -- 172.21.15.62:0/2422398129 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc02cc60 con 0x7feac406c6e0 2024-03-23T11:21:20.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.346+0000 7feac1600640 1 -- 172.21.15.62:0/2422398129 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feabc02cdc0 con 0x7feac406c6e0 2024-03-23T11:21:20.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.346+0000 7feac1600640 1 -- 172.21.15.62:0/2422398129 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc034640 con 0x7feac406c6e0 2024-03-23T11:21:20.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.346+0000 7feac9800640 1 -- 172.21.15.62:0/2422398129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feac406c6e0 msgr2=0x7feac406cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:20.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.346+0000 7feac9800640 1 --2- 172.21.15.62:0/2422398129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feac406c6e0 0x7feac406cb70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feabc009d90 tx=0x7feabc02beb0 comp rx=0 tx=0).stop 2024-03-23T11:21:20.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.346+0000 7feac9800640 1 -- 172.21.15.62:0/2422398129 shutdown_connections 2024-03-23T11:21:20.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.346+0000 7feac9800640 1 --2- 172.21.15.62:0/2422398129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feac406c6e0 0x7feac406cb70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:20.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.346+0000 7feac9800640 1 --2- 172.21.15.62:0/2422398129 >> v2:172.21.15.120:3300/0 conn(0x7feac406c300 0x7feac40ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:20.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.346+0000 7feac9800640 1 -- 172.21.15.62:0/2422398129 >> 172.21.15.62:0/2422398129 conn(0x7feac40b02e0 msgr2=0x7feac40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:20.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.347+0000 7feac9800640 1 -- 172.21.15.62:0/2422398129 shutdown_connections 2024-03-23T11:21:20.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.347+0000 7feac9800640 1 -- 172.21.15.62:0/2422398129 wait complete. 2024-03-23T11:21:20.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.348+0000 7feac9800640 1 Processor -- start 2024-03-23T11:21:20.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.348+0000 7feac9800640 1 -- start start 2024-03-23T11:21:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.349+0000 7feac9800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac406c300 0x7feac413ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.349+0000 7feac9800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feac406c6e0 0x7feac40ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.349+0000 7feac9800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feac40cafa0 0x7feac40ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.349+0000 7feac9800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac413cf60 con 0x7feac406c300 2024-03-23T11:21:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.349+0000 7feac9800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac413b390 con 0x7feac40cafa0 2024-03-23T11:21:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.349+0000 7feac9800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac414aca0 con 0x7feac406c6e0 2024-03-23T11:21:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac406c300 0x7feac413ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac406c300 0x7feac413ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46698/0 (socket says 172.21.15.62:46698) 2024-03-23T11:21:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac2a00640 1 -- 172.21.15.62:0/3180154209 learned_addr learned my addr 172.21.15.62:0/3180154209 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac3400640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feac40cafa0 0x7feac40ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac2000640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feac406c6e0 0x7feac40ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac2a00640 1 -- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feac406c6e0 msgr2=0x7feac40ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac2a00640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feac406c6e0 0x7feac40ae250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac2a00640 1 -- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feac40cafa0 msgr2=0x7feac40ae790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac2a00640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feac40cafa0 0x7feac40ae790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac2a00640 1 -- 172.21.15.62:0/3180154209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab4009400 con 0x7feac406c300 2024-03-23T11:21:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac3400640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feac40cafa0 0x7feac40ae790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:21:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.350+0000 7feac2a00640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac406c300 0x7feac413ae50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7feab400ebe0 tx=0x7feab400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:20.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.351+0000 7feabaa00640 1 -- 172.21.15.62:0/3180154209 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab4009760 con 0x7feac406c300 2024-03-23T11:21:20.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.351+0000 7feabaa00640 1 -- 172.21.15.62:0/3180154209 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab400cc30 con 0x7feac406c300 2024-03-23T11:21:20.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.351+0000 7feabaa00640 1 -- 172.21.15.62:0/3180154209 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab400eee0 con 0x7feac406c300 2024-03-23T11:21:20.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.351+0000 7feac9800640 1 -- 172.21.15.62:0/3180154209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feabc009a70 con 0x7feac406c300 2024-03-23T11:21:20.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.351+0000 7feac9800640 1 -- 172.21.15.62:0/3180154209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac40a86b0 con 0x7feac406c300 2024-03-23T11:21:20.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.352+0000 7feac9800640 1 -- 172.21.15.62:0/3180154209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac4066c90 con 0x7feac406c300 2024-03-23T11:21:20.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.353+0000 7feabaa00640 1 -- 172.21.15.62:0/3180154209 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7feab4013470 con 0x7feac406c300 2024-03-23T11:21:20.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.354+0000 7feabaa00640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feaa406a7a0 0x7feaa406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:20.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.354+0000 7feabaa00640 1 -- 172.21.15.62:0/3180154209 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4943+0+0 (secure 0 0 0) 0x7feab4088dc0 con 0x7feac406c300 2024-03-23T11:21:20.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.354+0000 7feac2000640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feaa406a7a0 0x7feaa406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.355+0000 7feac2000640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feaa406a7a0 0x7feaa406cc60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feac41598e0 tx=0x7feab0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:20.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.359+0000 7feabaa00640 1 -- 172.21.15.62:0/3180154209 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feab40544d0 con 0x7feac406c300 2024-03-23T11:21:20.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:20.567+0000 7feac9800640 1 -- 172.21.15.62:0/3180154209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7feac40afd90 con 0x7feac406c300 2024-03-23T11:21:21.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.349+0000 7feabaa00640 1 -- 172.21.15.62:0/3180154209 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v44) v1 ==== 97+0+0 (secure 0 0 0) 0x7feab4057af0 con 0x7feac406c300 2024-03-23T11:21:21.348 INFO:teuthology.orchestra.run.smithi062.stderr: removed all blocklist entries 2024-03-23T11:21:21.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.353+0000 7feac9800640 1 -- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feaa406a7a0 msgr2=0x7feaa406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:21.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.353+0000 7feac9800640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feaa406a7a0 0x7feaa406cc60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feac41598e0 tx=0x7feab0009040 comp rx=0 tx=0).stop 2024-03-23T11:21:21.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.353+0000 7feac9800640 1 -- 172.21.15.62:0/3180154209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac406c300 msgr2=0x7feac413ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:21.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.353+0000 7feac9800640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac406c300 0x7feac413ae50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7feab400ebe0 tx=0x7feab400c520 comp rx=0 tx=0).stop 2024-03-23T11:21:21.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.353+0000 7feac9800640 1 -- 172.21.15.62:0/3180154209 shutdown_connections 2024-03-23T11:21:21.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.353+0000 7feac9800640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feaa406a7a0 0x7feaa406cc60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.353+0000 7feac9800640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feac40cafa0 0x7feac40ae790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.353+0000 7feac9800640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feac406c6e0 0x7feac40ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.353+0000 7feac9800640 1 --2- 172.21.15.62:0/3180154209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac406c300 0x7feac413ae50 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.353+0000 7feac9800640 1 -- 172.21.15.62:0/3180154209 >> 172.21.15.62:0/3180154209 conn(0x7feac40b02e0 msgr2=0x7feac413eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:21.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.353+0000 7feac9800640 1 -- 172.21.15.62:0/3180154209 shutdown_connections 2024-03-23T11:21:21.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.354+0000 7feac9800640 1 -- 172.21.15.62:0/3180154209 wait complete. 2024-03-23T11:21:21.372 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T11:21:21.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.589+0000 7fd04ec00640 1 Processor -- start 2024-03-23T11:21:21.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.590+0000 7fd04ec00640 1 -- start start 2024-03-23T11:21:21.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.591+0000 7fd04ec00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd04815fcf0 0x7fd0480d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:21.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.591+0000 7fd04ec00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd0480b6dc0 0x7fd0481608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.591+0000 7fd04ec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0480d3540 con 0x7fd048160420 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.591+0000 7fd04ec00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd048160df0 con 0x7fd04815fcf0 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.591+0000 7fd04ec00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0480ce510 con 0x7fd0480b6dc0 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.591+0000 7fd04d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd04815fcf0 0x7fd0480d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.592+0000 7fd04d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd04815fcf0 0x7fd0480d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46714/0 (socket says 172.21.15.62:46714) 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.592+0000 7fd04d800640 1 -- 172.21.15.62:0/10226906 learned_addr learned my addr 172.21.15.62:0/10226906 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.592+0000 7fd04e200640 1 --2- 172.21.15.62:0/10226906 >> v2:172.21.15.120:3300/0 conn(0x7fd0480b6dc0 0x7fd0481608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.592+0000 7fd04d800640 1 -- 172.21.15.62:0/10226906 >> v2:172.21.15.120:3300/0 conn(0x7fd0480b6dc0 msgr2=0x7fd0481608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.592+0000 7fd04d800640 1 --2- 172.21.15.62:0/10226906 >> v2:172.21.15.120:3300/0 conn(0x7fd0480b6dc0 0x7fd0481608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.592+0000 7fd04d800640 1 -- 172.21.15.62:0/10226906 >> v1:172.21.15.120:6789/0 conn(0x7fd048160420 legacy=0x7fd0480d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.592+0000 7fd04d800640 1 -- 172.21.15.62:0/10226906 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0480cdc70 con 0x7fd04815fcf0 2024-03-23T11:21:21.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.592+0000 7fd04e200640 1 --2- 172.21.15.62:0/10226906 >> v2:172.21.15.120:3300/0 conn(0x7fd0480b6dc0 0x7fd0481608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.593+0000 7fd04d800640 1 --2- 172.21.15.62:0/10226906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04815fcf0 0x7fd0480d3000 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd04808bb20 tx=0x7fd03c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974ca09af258248b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.594+0000 7fd047e00640 1 -- 172.21.15.62:0/10226906 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c0317e0 con 0x7fd04815fcf0 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.594+0000 7fd047e00640 1 -- 172.21.15.62:0/10226906 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03c031940 con 0x7fd04815fcf0 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.594+0000 7fd047e00640 1 -- 172.21.15.62:0/10226906 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c031cd0 con 0x7fd04815fcf0 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.594+0000 7fd04ec00640 1 -- 172.21.15.62:0/10226906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04815fcf0 msgr2=0x7fd0480d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.594+0000 7fd04ec00640 1 --2- 172.21.15.62:0/10226906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04815fcf0 0x7fd0480d3000 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd04808bb20 tx=0x7fd03c02b880 comp rx=0 tx=0).stop 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.594+0000 7fd04ec00640 1 -- 172.21.15.62:0/10226906 shutdown_connections 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.594+0000 7fd04ec00640 1 --2- 172.21.15.62:0/10226906 >> v2:172.21.15.120:3300/0 conn(0x7fd0480b6dc0 0x7fd0481608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.594+0000 7fd04ec00640 1 --2- 172.21.15.62:0/10226906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd04815fcf0 0x7fd0480d3000 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.594+0000 7fd04ec00640 1 -- 172.21.15.62:0/10226906 >> 172.21.15.62:0/10226906 conn(0x7fd048089f60 msgr2=0x7fd0480d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.595+0000 7fd04ec00640 1 -- 172.21.15.62:0/10226906 shutdown_connections 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.595+0000 7fd04ec00640 1 -- 172.21.15.62:0/10226906 wait complete. 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.596+0000 7fd04ec00640 1 Processor -- start 2024-03-23T11:21:21.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.596+0000 7fd04ec00640 1 -- start start 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.597+0000 7fd04ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0480b6dc0 0x7fd048158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.597+0000 7fd04ec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd04815fcf0 0x7fd048159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.597+0000 7fd04ec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd048160420 0x7fd0481f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.597+0000 7fd04ec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04815a8a0 con 0x7fd048160420 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.597+0000 7fd04ec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd048159c90 con 0x7fd04815fcf0 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.597+0000 7fd04ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd048159f70 con 0x7fd0480b6dc0 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd048160420 0x7fd0481f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd048160420 0x7fd0481f55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46724/0 (socket says 172.21.15.62:46724) 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04e200640 1 -- 172.21.15.62:0/2706453275 learned_addr learned my addr 172.21.15.62:0/2706453275 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04d800640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0480b6dc0 0x7fd048158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04ce00640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd04815fcf0 0x7fd048159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04e200640 1 -- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0480b6dc0 msgr2=0x7fd048158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:21.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04e200640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0480b6dc0 0x7fd048158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04e200640 1 -- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd04815fcf0 msgr2=0x7fd048159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:21.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04e200640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd04815fcf0 0x7fd048159390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04e200640 1 -- 172.21.15.62:0/2706453275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c0094d0 con 0x7fd048160420 2024-03-23T11:21:21.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04d800640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0480b6dc0 0x7fd048158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:21.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.598+0000 7fd04e200640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd048160420 0x7fd0481f55e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd04000c7f0 tx=0x7fd04000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:21.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.599+0000 7fd045600640 1 -- 172.21.15.62:0/2706453275 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040007820 con 0x7fd048160420 2024-03-23T11:21:21.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.599+0000 7fd045600640 1 -- 172.21.15.62:0/2706453275 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd040007980 con 0x7fd048160420 2024-03-23T11:21:21.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.599+0000 7fd045600640 1 -- 172.21.15.62:0/2706453275 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040007d70 con 0x7fd048160420 2024-03-23T11:21:21.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.599+0000 7fd04ec00640 1 -- 172.21.15.62:0/2706453275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04815a250 con 0x7fd048160420 2024-03-23T11:21:21.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.599+0000 7fd04ec00640 1 -- 172.21.15.62:0/2706453275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd048159090 con 0x7fd048160420 2024-03-23T11:21:21.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.600+0000 7fd04ec00640 1 -- 172.21.15.62:0/2706453275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0480675b0 con 0x7fd048160420 2024-03-23T11:21:21.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.601+0000 7fd045600640 1 -- 172.21.15.62:0/2706453275 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd0400113f0 con 0x7fd048160420 2024-03-23T11:21:21.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.602+0000 7fd045600640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd02c06a7a0 0x7fd02c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:21.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.602+0000 7fd045600640 1 -- 172.21.15.62:0/2706453275 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd040088ee0 con 0x7fd048160420 2024-03-23T11:21:21.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.603+0000 7fd04d800640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd02c06a7a0 0x7fd02c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:21.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.604+0000 7fd04d800640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd02c06a7a0 0x7fd02c06cc60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd03c036000 tx=0x7fd03c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:21.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.607+0000 7fd045600640 1 -- 172.21.15.62:0/2706453275 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0400545b0 con 0x7fd048160420 2024-03-23T11:21:21.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.877+0000 7fd04ec00640 1 -- 172.21.15.62:0/2706453275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd048066bb0 con 0x7fd048160420 2024-03-23T11:21:21.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.878+0000 7fd045600640 1 -- 172.21.15.62:0/2706453275 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v9) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd040057bd0 con 0x7fd048160420 2024-03-23T11:21:21.877 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:21.877 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:21:21.877 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_dump": [ 2024-03-23T11:21:21.877 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.877 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.a", 2024-03-23T11:21:21.877 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lYNxYERAAPVHWwRHM5V7OejL294b2zg==", 2024-03-23T11:21:21.877 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.877 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:21:21.877 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:21:21.877 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.b", 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lmOUANBAAifuZ0DUPD9az/mturTndSA==", 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.878 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.c", 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5ltqrjIhAA9+f82fvlrFlIEXqswPE0ig==", 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.d", 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5l7ly4CBAA8+oim0z+bbljfP0VH7XSaw==", 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.879 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.0", 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBiuv5lRfaQBRAAvrtCZXwt00PUQfCqqKBzMQ==", 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.880 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.1", 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBluv5lzwS/IBAAmYU8yMLtCUJNr/AyE0MJDA==", 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.2", 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBouv5lyAy1OhAAJoUsg6DFpEPxAcH5oI2exQ==", 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:21:21.881 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.3", 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBsuv5lb0YcGBAA220D0uRhAZpG4w97wqsB5A==", 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.882 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.4", 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBzuv5lC8RIGhAAH+5HaeCgQhrQMrYm9iWaYA==", 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.5", 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB2uv5lYi7ZIhAAMNIm/jPckCAbX9oEx5Li/Q==", 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:21:21.883 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.6", 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB5uv5lgoRtLBAAFT7uJtyqHtHAegOYyDH/WQ==", 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.7", 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB9uv5l/dzrCBAA3jsua0LvBPI7I551/rRHrg==", 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.884 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.0", 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5ltavQFhAAvV2RWuHOPK2T9Q7gWkYA4w==", 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow r", 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.1", 2024-03-23T11:21:21.885 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5luQ/zLRAAwjDtV64TCKnQliPq4iUAYg==", 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.2", 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lDRG1IhAAjVaXRR30G1985oMou7Ambg==", 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:21:21.886 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.3", 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lomZyOBAA8BMIBD+Zhtp6Fa2RcEXCJg==", 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:21:21.887 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.admin", 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lNtZHBhAAhiAC1rdmANClOSwzrX9iSQ==", 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.x", 2024-03-23T11:21:21.888 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lT6+KBRAAl5TZsJGNr4gAB2cEDZo/hQ==", 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.y", 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lqnG8NRAAvyswMz5kpW0rGjzWYRTAAg==", 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-03-23T11:21:21.889 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:21:21.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.881+0000 7fd04ec00640 1 -- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd02c06a7a0 msgr2=0x7fd02c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:21.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.881+0000 7fd04ec00640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd02c06a7a0 0x7fd02c06cc60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd03c036000 tx=0x7fd03c03d040 comp rx=0 tx=0).stop 2024-03-23T11:21:21.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.881+0000 7fd04ec00640 1 -- 172.21.15.62:0/2706453275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd048160420 msgr2=0x7fd0481f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:21.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.881+0000 7fd04ec00640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd048160420 0x7fd0481f55e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd04000c7f0 tx=0x7fd04000ccb0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.881+0000 7fd04ec00640 1 -- 172.21.15.62:0/2706453275 shutdown_connections 2024-03-23T11:21:21.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.881+0000 7fd04ec00640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd02c06a7a0 0x7fd02c06cc60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.881+0000 7fd04ec00640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd048160420 0x7fd0481f55e0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.881+0000 7fd04ec00640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd04815fcf0 0x7fd048159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.881+0000 7fd04ec00640 1 --2- 172.21.15.62:0/2706453275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0480b6dc0 0x7fd048158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:21.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.881+0000 7fd04ec00640 1 -- 172.21.15.62:0/2706453275 >> 172.21.15.62:0/2706453275 conn(0x7fd048089f60 msgr2=0x7fd04806b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:21.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.882+0000 7fd04ec00640 1 -- 172.21.15.62:0/2706453275 shutdown_connections 2024-03-23T11:21:21.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:21.882+0000 7fd04ec00640 1 -- 172.21.15.62:0/2706453275 wait complete. 2024-03-23T11:21:21.898 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:21:22.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.109+0000 7f561f200640 1 Processor -- start 2024-03-23T11:21:22.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.109+0000 7f561f200640 1 -- start start 2024-03-23T11:21:22.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.111+0000 7f561f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f561806c300 0x7f56180ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.111+0000 7f561f200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f561806c6e0 0x7f561806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.111+0000 7f561f200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56180cbc60 con 0x7f56180cafa0 2024-03-23T11:21:22.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.111+0000 7f561f200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56180cbf40 con 0x7f561806c6e0 2024-03-23T11:21:22.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.111+0000 7f561f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56180cc220 con 0x7f561806c300 2024-03-23T11:21:22.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.111+0000 7f561d200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f561806c6e0 0x7f561806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:22.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.111+0000 7f561d200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f561806c6e0 0x7f561806cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46728/0 (socket says 172.21.15.62:46728) 2024-03-23T11:21:22.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.111+0000 7f5617400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f56180cafa0 0x7f56180cb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43724/0 (socket says 172.21.15.62:43724) 2024-03-23T11:21:22.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.111+0000 7f561d200640 1 -- 172.21.15.62:0/353306145 learned_addr learned my addr 172.21.15.62:0/353306145 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:22.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.112+0000 7f5617e00640 1 --2- 172.21.15.62:0/353306145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f561806c300 0x7f56180ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:22.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.112+0000 7f561d200640 1 -- 172.21.15.62:0/353306145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f561806c300 msgr2=0x7f56180ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:22.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.112+0000 7f561d200640 1 --2- 172.21.15.62:0/353306145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f561806c300 0x7f56180ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.112+0000 7f561d200640 1 -- 172.21.15.62:0/353306145 >> v1:172.21.15.62:6789/0 conn(0x7f56180cafa0 legacy=0x7f56180cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:22.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.112+0000 7f561d200640 1 -- 172.21.15.62:0/353306145 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56180cc500 con 0x7f561806c6e0 2024-03-23T11:21:22.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.112+0000 7f5616a00640 1 -- 172.21.15.62:0/353306145 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852987507 0 0) 0x7f56180cbc60 con 0x7f56180cafa0 2024-03-23T11:21:22.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.113+0000 7f561d200640 1 --2- 172.21.15.62:0/353306145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f561806c6e0 0x7f561806cb70 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5610009e50 tx=0x7f561002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb6a3a7768ed29 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:22.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.113+0000 7f5616a00640 1 -- 172.21.15.62:0/353306145 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561002cb00 con 0x7f561806c6e0 2024-03-23T11:21:22.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.113+0000 7f5616a00640 1 -- 172.21.15.62:0/353306145 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561002cc60 con 0x7f561806c6e0 2024-03-23T11:21:22.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.113+0000 7f5616a00640 1 -- 172.21.15.62:0/353306145 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56100343f0 con 0x7f561806c6e0 2024-03-23T11:21:22.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.114+0000 7f561f200640 1 -- 172.21.15.62:0/353306145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f561806c6e0 msgr2=0x7f561806cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:22.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.114+0000 7f561f200640 1 --2- 172.21.15.62:0/353306145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f561806c6e0 0x7f561806cb70 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5610009e50 tx=0x7f561002bf60 comp rx=0 tx=0).stop 2024-03-23T11:21:22.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.114+0000 7f561f200640 1 -- 172.21.15.62:0/353306145 shutdown_connections 2024-03-23T11:21:22.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.114+0000 7f561f200640 1 --2- 172.21.15.62:0/353306145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f561806c6e0 0x7f561806cb70 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.114+0000 7f561f200640 1 --2- 172.21.15.62:0/353306145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f561806c300 0x7f56180ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.114+0000 7f561f200640 1 -- 172.21.15.62:0/353306145 >> 172.21.15.62:0/353306145 conn(0x7f56180b02e0 msgr2=0x7f56180b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:22.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.114+0000 7f561f200640 1 -- 172.21.15.62:0/353306145 shutdown_connections 2024-03-23T11:21:22.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.114+0000 7f561f200640 1 -- 172.21.15.62:0/353306145 wait complete. 2024-03-23T11:21:22.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.116+0000 7f561f200640 1 Processor -- start 2024-03-23T11:21:22.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.116+0000 7f561f200640 1 -- start start 2024-03-23T11:21:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.117+0000 7f561f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f561806c300 0x7f5618156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.117+0000 7f561f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f561806c6e0 0x7f56181513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.117+0000 7f561f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56180cafa0 0x7f5618151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.117+0000 7f561f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56181efbf0 con 0x7f561806c6e0 2024-03-23T11:21:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.117+0000 7f561f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561813e550 con 0x7f56180cafa0 2024-03-23T11:21:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.117+0000 7f561f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5618151f50 con 0x7f561806c300 2024-03-23T11:21:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.117+0000 7f5617400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f561806c6e0 0x7f56181513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.117+0000 7f5617400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f561806c6e0 0x7f56181513e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46732/0 (socket says 172.21.15.62:46732) 2024-03-23T11:21:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.117+0000 7f5617400640 1 -- 172.21.15.62:0/508895374 learned_addr learned my addr 172.21.15.62:0/508895374 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:22.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f5617e00640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f561806c300 0x7f5618156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:22.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f561d200640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56180cafa0 0x7f5618151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f5617400640 1 -- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f561806c300 msgr2=0x7f5618156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f5617400640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f561806c300 0x7f5618156f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f5617400640 1 -- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56180cafa0 msgr2=0x7f5618151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f5617400640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56180cafa0 0x7f5618151920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f5617400640 1 -- 172.21.15.62:0/508895374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5610009b30 con 0x7f561806c6e0 2024-03-23T11:21:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f5617400640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f561806c6e0 0x7f56181513e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f560400c430 tx=0x7f560400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f560fe00640 1 -- 172.21.15.62:0/508895374 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56040074e0 con 0x7f561806c6e0 2024-03-23T11:21:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f560fe00640 1 -- 172.21.15.62:0/508895374 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5604007660 con 0x7f561806c6e0 2024-03-23T11:21:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f560fe00640 1 -- 172.21.15.62:0/508895374 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5604004a20 con 0x7f561806c6e0 2024-03-23T11:21:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.118+0000 7f561f200640 1 -- 172.21.15.62:0/508895374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5618154170 con 0x7f561806c6e0 2024-03-23T11:21:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.119+0000 7f561f200640 1 -- 172.21.15.62:0/508895374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5618154730 con 0x7f561806c6e0 2024-03-23T11:21:22.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.120+0000 7f561f200640 1 -- 172.21.15.62:0/508895374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5618066c90 con 0x7f561806c6e0 2024-03-23T11:21:22.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.121+0000 7f560fe00640 1 -- 172.21.15.62:0/508895374 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5604004b80 con 0x7f561806c6e0 2024-03-23T11:21:22.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.122+0000 7f560fe00640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55f806a7a0 0x7f55f806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.122+0000 7f560fe00640 1 -- 172.21.15.62:0/508895374 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5604088980 con 0x7f561806c6e0 2024-03-23T11:21:22.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.122+0000 7f5617e00640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55f806a7a0 0x7f55f806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.123+0000 7f5617e00640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55f806a7a0 0x7f55f806cc60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5608006fd0 tx=0x7f5608008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:22.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.126+0000 7f560fe00640 1 -- 172.21.15.62:0/508895374 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5604054050 con 0x7f561806c6e0 2024-03-23T11:21:22.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.325+0000 7f561f200640 1 -- 172.21.15.62:0/508895374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f561813b350 con 0x7f561806c6e0 2024-03-23T11:21:22.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.326+0000 7f560fe00640 1 -- 172.21.15.62:0/508895374 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+12389 (secure 0 0 0) 0x7f5604057670 con 0x7f561806c6e0 2024-03-23T11:21:22.325 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:22.325 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":44,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:21:21.341061+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:21:22.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.328+0000 7f561f200640 1 -- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55f806a7a0 msgr2=0x7f55f806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:22.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.328+0000 7f561f200640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55f806a7a0 0x7f55f806cc60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5608006fd0 tx=0x7f5608008040 comp rx=0 tx=0).stop 2024-03-23T11:21:22.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.328+0000 7f561f200640 1 -- 172.21.15.62:0/508895374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f561806c6e0 msgr2=0x7f56181513e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:22.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.328+0000 7f561f200640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f561806c6e0 0x7f56181513e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f560400c430 tx=0x7f560400c8f0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.328+0000 7f561f200640 1 -- 172.21.15.62:0/508895374 shutdown_connections 2024-03-23T11:21:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.328+0000 7f561f200640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55f806a7a0 0x7f55f806cc60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.328+0000 7f561f200640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f561806c6e0 0x7f56181513e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.328+0000 7f561f200640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56180cafa0 0x7f5618151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.328+0000 7f561f200640 1 --2- 172.21.15.62:0/508895374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f561806c300 0x7f5618156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.328+0000 7f561f200640 1 -- 172.21.15.62:0/508895374 >> 172.21.15.62:0/508895374 conn(0x7f56180b02e0 msgr2=0x7f56181407d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.329+0000 7f561f200640 1 -- 172.21.15.62:0/508895374 shutdown_connections 2024-03-23T11:21:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.329+0000 7f561f200640 1 -- 172.21.15.62:0/508895374 wait complete. 2024-03-23T11:21:22.346 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:21:22.347 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:21:22.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.569+0000 7fd2a8e00640 1 Processor -- start 2024-03-23T11:21:22.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.569+0000 7fd2a8e00640 1 -- start start 2024-03-23T11:21:22.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.571+0000 7fd2a8e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2a406dca0 0x7fd2a40b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.571+0000 7fd2a8e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2a406c300 0x7fd2a40cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.571+0000 7fd2a8e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a40cad20 con 0x7fd2a40cd7c0 2024-03-23T11:21:22.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.571+0000 7fd2a8e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a406c770 con 0x7fd2a406c300 2024-03-23T11:21:22.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.571+0000 7fd2a8e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a406e130 con 0x7fd2a406dca0 2024-03-23T11:21:22.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.571+0000 7fd2a2800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2a406c300 0x7fd2a40cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:22.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.571+0000 7fd2a2800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2a406c300 0x7fd2a40cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46736/0 (socket says 172.21.15.62:46736) 2024-03-23T11:21:22.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.571+0000 7fd2a2800640 1 -- 172.21.15.62:0/999315954 learned_addr learned my addr 172.21.15.62:0/999315954 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:22.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.572+0000 7fd2a2800640 1 -- 172.21.15.62:0/999315954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2a406dca0 msgr2=0x7fd2a40b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:22.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.572+0000 7fd2a2800640 1 --2- 172.21.15.62:0/999315954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2a406dca0 0x7fd2a40b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.572+0000 7fd2a2800640 1 -- 172.21.15.62:0/999315954 >> v1:172.21.15.120:6789/0 conn(0x7fd2a40cd7c0 legacy=0x7fd2a40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:22.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.572+0000 7fd2a2800640 1 -- 172.21.15.62:0/999315954 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a415a8a0 con 0x7fd2a406c300 2024-03-23T11:21:22.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.573+0000 7fd2a2800640 1 --2- 172.21.15.62:0/999315954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2a406c300 0x7fd2a40cb490 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009e50 tx=0x7fd29c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83edb97ec8e12b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:22.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.573+0000 7fd2a0a00640 1 -- 172.21.15.62:0/999315954 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c02c9e0 con 0x7fd2a406c300 2024-03-23T11:21:22.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.573+0000 7fd2a0a00640 1 -- 172.21.15.62:0/999315954 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c02cb40 con 0x7fd2a406c300 2024-03-23T11:21:22.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.573+0000 7fd2a0a00640 1 -- 172.21.15.62:0/999315954 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c0343f0 con 0x7fd2a406c300 2024-03-23T11:21:22.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.574+0000 7fd2a8e00640 1 -- 172.21.15.62:0/999315954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2a406c300 msgr2=0x7fd2a40cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:22.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.574+0000 7fd2a8e00640 1 --2- 172.21.15.62:0/999315954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2a406c300 0x7fd2a40cb490 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009e50 tx=0x7fd29c02be40 comp rx=0 tx=0).stop 2024-03-23T11:21:22.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.574+0000 7fd2a8e00640 1 -- 172.21.15.62:0/999315954 shutdown_connections 2024-03-23T11:21:22.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.574+0000 7fd2a8e00640 1 --2- 172.21.15.62:0/999315954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2a406c300 0x7fd2a40cb490 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.574+0000 7fd2a8e00640 1 --2- 172.21.15.62:0/999315954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2a406dca0 0x7fd2a40b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.574+0000 7fd2a8e00640 1 -- 172.21.15.62:0/999315954 >> 172.21.15.62:0/999315954 conn(0x7fd2a40b02e0 msgr2=0x7fd2a40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:22.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.574+0000 7fd2a8e00640 1 -- 172.21.15.62:0/999315954 shutdown_connections 2024-03-23T11:21:22.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.574+0000 7fd2a8e00640 1 -- 172.21.15.62:0/999315954 wait complete. 2024-03-23T11:21:22.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.576+0000 7fd2a8e00640 1 Processor -- start 2024-03-23T11:21:22.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.576+0000 7fd2a8e00640 1 -- start start 2024-03-23T11:21:22.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.577+0000 7fd2a8e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2a406c300 0x7fd2a4148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.577+0000 7fd2a8e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2a40cd7c0 0x7fd2a41394e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.577+0000 7fd2a8e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2a40aee70 0x7fd2a414c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.577+0000 7fd2a8e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a4157c00 con 0x7fd2a40aee70 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.577+0000 7fd2a8e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41492c0 con 0x7fd2a406c300 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.577+0000 7fd2a8e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41495a0 con 0x7fd2a40cd7c0 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.577+0000 7fd2a2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2a40aee70 0x7fd2a414c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.577+0000 7fd2a2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2a40aee70 0x7fd2a414c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46742/0 (socket says 172.21.15.62:46742) 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.577+0000 7fd2a2800640 1 -- 172.21.15.62:0/2426637287 learned_addr learned my addr 172.21.15.62:0/2426637287 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.577+0000 7fd2a1e00640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2a406c300 0x7fd2a4148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.578+0000 7fd2a1400640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2a40cd7c0 0x7fd2a41394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.578+0000 7fd2a2800640 1 -- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2a40cd7c0 msgr2=0x7fd2a41394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.578+0000 7fd2a2800640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2a40cd7c0 0x7fd2a41394e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.578+0000 7fd2a2800640 1 -- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2a406c300 msgr2=0x7fd2a4148c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:22.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.578+0000 7fd2a2800640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2a406c300 0x7fd2a4148c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.578+0000 7fd2a2800640 1 -- 172.21.15.62:0/2426637287 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c009b30 con 0x7fd2a40aee70 2024-03-23T11:21:22.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.578+0000 7fd2a2800640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2a40aee70 0x7fd2a414c4b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd29c036040 tx=0x7fd29c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:22.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.578+0000 7fd29a000640 1 -- 172.21.15.62:0/2426637287 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c03e450 con 0x7fd2a40aee70 2024-03-23T11:21:22.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.578+0000 7fd29a000640 1 -- 172.21.15.62:0/2426637287 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c03e5d0 con 0x7fd2a40aee70 2024-03-23T11:21:22.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.578+0000 7fd29a000640 1 -- 172.21.15.62:0/2426637287 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c03e980 con 0x7fd2a40aee70 2024-03-23T11:21:22.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.579+0000 7fd2a8e00640 1 -- 172.21.15.62:0/2426637287 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a4149880 con 0x7fd2a40aee70 2024-03-23T11:21:22.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.579+0000 7fd2a8e00640 1 -- 172.21.15.62:0/2426637287 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a4139a20 con 0x7fd2a40aee70 2024-03-23T11:21:22.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.580+0000 7fd2a8e00640 1 -- 172.21.15.62:0/2426637287 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a4157db0 con 0x7fd2a40aee70 2024-03-23T11:21:22.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.581+0000 7fd29a000640 1 -- 172.21.15.62:0/2426637287 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd29c03eae0 con 0x7fd2a40aee70 2024-03-23T11:21:22.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.582+0000 7fd29a000640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd28406a7a0 0x7fd28406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:22.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.582+0000 7fd29a000640 1 -- 172.21.15.62:0/2426637287 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd29c0abb80 con 0x7fd2a40aee70 2024-03-23T11:21:22.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.583+0000 7fd2a1e00640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd28406a7a0 0x7fd28406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:22.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.584+0000 7fd2a1e00640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd28406a7a0 0x7fd28406cc60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd294006fd0 tx=0x7fd294008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:22.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.585+0000 7fd29a000640 1 -- 172.21.15.62:0/2426637287 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd29c077360 con 0x7fd2a40aee70 2024-03-23T11:21:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.785+0000 7fd2a8e00640 1 -- 172.21.15.62:0/2426637287 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd2a4157db0 con 0x7fd2a40aee70 2024-03-23T11:21:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.786+0000 7fd29a000640 1 -- 172.21.15.62:0/2426637287 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd2a4157db0 con 0x7fd2a40aee70 2024-03-23T11:21:22.785 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:21:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.788+0000 7fd2a8e00640 1 -- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd28406a7a0 msgr2=0x7fd28406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.788+0000 7fd2a8e00640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd28406a7a0 0x7fd28406cc60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd294006fd0 tx=0x7fd294008040 comp rx=0 tx=0).stop 2024-03-23T11:21:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.788+0000 7fd2a8e00640 1 -- 172.21.15.62:0/2426637287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2a40aee70 msgr2=0x7fd2a414c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.788+0000 7fd2a8e00640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2a40aee70 0x7fd2a414c4b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd29c036040 tx=0x7fd29c004a40 comp rx=0 tx=0).stop 2024-03-23T11:21:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.788+0000 7fd2a8e00640 1 -- 172.21.15.62:0/2426637287 shutdown_connections 2024-03-23T11:21:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.788+0000 7fd2a8e00640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd28406a7a0 0x7fd28406cc60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.788+0000 7fd2a8e00640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2a40aee70 0x7fd2a414c4b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.788+0000 7fd2a8e00640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2a40cd7c0 0x7fd2a41394e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.788+0000 7fd2a8e00640 1 --2- 172.21.15.62:0/2426637287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2a406c300 0x7fd2a4148c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.788+0000 7fd2a8e00640 1 -- 172.21.15.62:0/2426637287 >> 172.21.15.62:0/2426637287 conn(0x7fd2a40b02e0 msgr2=0x7fd2a40cdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.789+0000 7fd2a8e00640 1 -- 172.21.15.62:0/2426637287 shutdown_connections 2024-03-23T11:21:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:22.789+0000 7fd2a8e00640 1 -- 172.21.15.62:0/2426637287 wait complete. 2024-03-23T11:21:22.805 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T11:21:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.022+0000 7f7ae9200640 1 Processor -- start 2024-03-23T11:21:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.022+0000 7f7ae9200640 1 -- start start 2024-03-23T11:21:23.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.024+0000 7f7ae9200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4390 0x7f7ae40b4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:23.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.024+0000 7f7ae9200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae40c6670 con 0x7f7ae415ce50 2024-03-23T11:21:23.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.024+0000 7f7ae9200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae40ce1f0 con 0x7f7ae40b4cb0 2024-03-23T11:21:23.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.024+0000 7f7ae9200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae40ce7b0 con 0x7f7ae40b4390 2024-03-23T11:21:23.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.024+0000 7f7ae3e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7ae415ce50 0x7f7ae40c6b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43738/0 (socket says 172.21.15.62:43738) 2024-03-23T11:21:23.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.024+0000 7f7ae3e00640 1 -- 172.21.15.62:0/3378787826 learned_addr learned my addr 172.21.15.62:0/3378787826 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:23.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.025+0000 7f7ae3400640 1 --2- 172.21.15.62:0/3378787826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4390 0x7f7ae40b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:23.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.025+0000 7f7ae2000640 1 -- 172.21.15.62:0/3378787826 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147855894 0 0) 0x7f7ae40c6670 con 0x7f7ae415ce50 2024-03-23T11:21:23.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.025+0000 7f7ae2000640 1 -- 172.21.15.62:0/3378787826 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ac8003520 con 0x7f7ae415ce50 2024-03-23T11:21:23.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.025+0000 7f7ae3400640 1 -- 172.21.15.62:0/3378787826 >> v1:172.21.15.120:6789/0 conn(0x7f7ae40b4cb0 legacy=0x7f7ae415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:23.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.025+0000 7f7ae2000640 1 -- 172.21.15.62:0/3378787826 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3819660070 0 0) 0x7f7ac8003520 con 0x7f7ae415ce50 2024-03-23T11:21:23.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.025+0000 7f7ae3400640 1 -- 172.21.15.62:0/3378787826 >> v1:172.21.15.62:6789/0 conn(0x7f7ae415ce50 legacy=0x7f7ae40c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:23.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.025+0000 7f7ae3400640 1 -- 172.21.15.62:0/3378787826 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae40ce4d0 con 0x7f7ae40b4390 2024-03-23T11:21:23.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.027+0000 7f7ae3400640 1 --2- 172.21.15.62:0/3378787826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4390 0x7f7ae40b4770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40097d0 tx=0x7f7ad402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5455ff27e26e96ab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:23.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.027+0000 7f7ae2000640 1 -- 172.21.15.62:0/3378787826 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0019f0 con 0x7f7ae40b4390 2024-03-23T11:21:23.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.027+0000 7f7ae2000640 1 -- 172.21.15.62:0/3378787826 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad402e050 con 0x7f7ae40b4390 2024-03-23T11:21:23.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.027+0000 7f7ae2000640 1 -- 172.21.15.62:0/3378787826 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4030710 con 0x7f7ae40b4390 2024-03-23T11:21:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.028+0000 7f7ae9200640 1 -- 172.21.15.62:0/3378787826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4390 msgr2=0x7f7ae40b4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.028+0000 7f7ae9200640 1 --2- 172.21.15.62:0/3378787826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4390 0x7f7ae40b4770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40097d0 tx=0x7f7ad402b800 comp rx=0 tx=0).stop 2024-03-23T11:21:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.028+0000 7f7ae9200640 1 -- 172.21.15.62:0/3378787826 shutdown_connections 2024-03-23T11:21:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.028+0000 7f7ae9200640 1 --2- 172.21.15.62:0/3378787826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4390 0x7f7ae40b4770 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.028+0000 7f7ae9200640 1 -- 172.21.15.62:0/3378787826 >> 172.21.15.62:0/3378787826 conn(0x7f7ae4089f60 msgr2=0x7f7ae40d26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.028+0000 7f7ae9200640 1 -- 172.21.15.62:0/3378787826 shutdown_connections 2024-03-23T11:21:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.028+0000 7f7ae9200640 1 -- 172.21.15.62:0/3378787826 wait complete. 2024-03-23T11:21:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.031+0000 7f7ae9200640 1 Processor -- start 2024-03-23T11:21:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.031+0000 7f7ae9200640 1 -- start start 2024-03-23T11:21:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.032+0000 7f7ae9200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ae40b4390 0x7f7ae40b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.032+0000 7f7ae9200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4cb0 0x7f7ae4152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.032+0000 7f7ae9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae415ce50 0x7f7ae41e6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.032+0000 7f7ae9200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae41f6370 con 0x7f7ae415ce50 2024-03-23T11:21:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.032+0000 7f7ae9200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae40b6f40 con 0x7f7ae40b4390 2024-03-23T11:21:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.032+0000 7f7ae9200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae40b6610 con 0x7f7ae40b4cb0 2024-03-23T11:21:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.032+0000 7f7ae3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae415ce50 0x7f7ae41e6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.032+0000 7f7ae3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae415ce50 0x7f7ae41e6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46758/0 (socket says 172.21.15.62:46758) 2024-03-23T11:21:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.032+0000 7f7ae3e00640 1 -- 172.21.15.62:0/1201100561 learned_addr learned my addr 172.21.15.62:0/1201100561 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.033+0000 7f7ae2a00640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4cb0 0x7f7ae4152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:23.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.033+0000 7f7ae3400640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ae40b4390 0x7f7ae40b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:23.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.033+0000 7f7ae3e00640 1 -- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4cb0 msgr2=0x7f7ae4152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:23.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.033+0000 7f7ae3e00640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4cb0 0x7f7ae4152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:23.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.033+0000 7f7ae3e00640 1 -- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ae40b4390 msgr2=0x7f7ae40b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:23.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.033+0000 7f7ae3e00640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ae40b4390 0x7f7ae40b73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:23.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.033+0000 7f7ae3e00640 1 -- 172.21.15.62:0/1201100561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad4009400 con 0x7f7ae415ce50 2024-03-23T11:21:23.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.033+0000 7f7ae2a00640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4cb0 0x7f7ae4152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:21:23.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.033+0000 7f7ae3e00640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae415ce50 0x7f7ae41e6400 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00cca0 tx=0x7f7adc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:23.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.034+0000 7f7adb400640 1 -- 172.21.15.62:0/1201100561 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc012400 con 0x7f7ae415ce50 2024-03-23T11:21:23.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.034+0000 7f7adb400640 1 -- 172.21.15.62:0/1201100561 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7adc012560 con 0x7f7ae415ce50 2024-03-23T11:21:23.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.034+0000 7f7adb400640 1 -- 172.21.15.62:0/1201100561 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0128f0 con 0x7f7ae415ce50 2024-03-23T11:21:23.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.034+0000 7f7ae9200640 1 -- 172.21.15.62:0/1201100561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae40d16d0 con 0x7f7ae415ce50 2024-03-23T11:21:23.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.034+0000 7f7ae9200640 1 -- 172.21.15.62:0/1201100561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae40b6bd0 con 0x7f7ae415ce50 2024-03-23T11:21:23.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.035+0000 7f7ae9200640 1 -- 172.21.15.62:0/1201100561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae40b70d0 con 0x7f7ae415ce50 2024-03-23T11:21:23.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.036+0000 7f7adb400640 1 -- 172.21.15.62:0/1201100561 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7adc012a50 con 0x7f7ae415ce50 2024-03-23T11:21:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.037+0000 7f7adb400640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7ac406a7a0 0x7f7ac406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.038+0000 7f7adb400640 1 -- 172.21.15.62:0/1201100561 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7adc0890c0 con 0x7f7ae415ce50 2024-03-23T11:21:23.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.038+0000 7f7ae3400640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7ac406a7a0 0x7f7ac406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:23.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.039+0000 7f7ae3400640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7ac406a7a0 0x7f7ac406cc60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4034000 tx=0x7f7ad4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:23.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.042+0000 7f7adb400640 1 -- 172.21.15.62:0/1201100561 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7adc0548c0 con 0x7f7ae415ce50 2024-03-23T11:21:23.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.244+0000 7f7ae9200640 1 -- 172.21.15.62:0/1201100561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7ae4066bb0 con 0x7f7ae415ce50 2024-03-23T11:21:23.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.244+0000 7f7adb400640 1 -- 172.21.15.62:0/1201100561 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v44) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7adc057ee0 con 0x7f7ae415ce50 2024-03-23T11:21:23.244 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:23.244 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:21:23.244 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:23.245 INFO:teuthology.orchestra.run.smithi062.stdout: "poolnum": 1, 2024-03-23T11:21:23.245 INFO:teuthology.orchestra.run.smithi062.stdout: "poolname": ".mgr" 2024-03-23T11:21:23.245 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:23.245 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:21:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 -- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7ac406a7a0 msgr2=0x7f7ac406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7ac406a7a0 0x7f7ac406cc60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4034000 tx=0x7f7ad4008030 comp rx=0 tx=0).stop 2024-03-23T11:21:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 -- 172.21.15.62:0/1201100561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae415ce50 msgr2=0x7f7ae41e6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae415ce50 0x7f7ae41e6400 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00cca0 tx=0x7f7adc00a430 comp rx=0 tx=0).stop 2024-03-23T11:21:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 -- 172.21.15.62:0/1201100561 shutdown_connections 2024-03-23T11:21:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7ac406a7a0 0x7f7ac406cc60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae415ce50 0x7f7ae41e6400 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ae40b4cb0 0x7f7ae4152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 --2- 172.21.15.62:0/1201100561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ae40b4390 0x7f7ae40b73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 -- 172.21.15.62:0/1201100561 >> 172.21.15.62:0/1201100561 conn(0x7f7ae4089f60 msgr2=0x7f7ae40b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 -- 172.21.15.62:0/1201100561 shutdown_connections 2024-03-23T11:21:23.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.247+0000 7f7ae9200640 1 -- 172.21.15.62:0/1201100561 wait complete. 2024-03-23T11:21:23.265 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T11:21:23.265 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T11:21:23.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.487+0000 7f01e5e00640 1 Processor -- start 2024-03-23T11:21:23.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.487+0000 7f01e5e00640 1 -- start start 2024-03-23T11:21:23.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.488+0000 7f01e5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f01e006d950 0x7f01e006c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:23.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.488+0000 7f01e5e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e015cfd0 con 0x7f01e006dd30 2024-03-23T11:21:23.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.489+0000 7f01e5e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e015d220 con 0x7f01e00ce690 2024-03-23T11:21:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.489+0000 7f01e5e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e015d500 con 0x7f01e006d950 2024-03-23T11:21:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.489+0000 7f01e4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f01e006d950 0x7f01e006c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.489+0000 7f01e4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f01e006d950 0x7f01e006c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46770/0 (socket says 172.21.15.62:46770) 2024-03-23T11:21:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.489+0000 7f01e4a00640 1 -- 172.21.15.62:0/2916287375 learned_addr learned my addr 172.21.15.62:0/2916287375 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.489+0000 7f01dfe00640 1 --1- 172.21.15.62:0/2916287375 >> v1:172.21.15.62:6789/0 conn(0x7f01e006dd30 0x7f01e0070cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43752/0 (socket says 172.21.15.62:43752) 2024-03-23T11:21:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.489+0000 7f01e4a00640 1 -- 172.21.15.62:0/2916287375 >> v1:172.21.15.120:6789/0 conn(0x7f01e00ce690 legacy=0x7f01e0159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.489+0000 7f01e4a00640 1 -- 172.21.15.62:0/2916287375 >> v1:172.21.15.62:6789/0 conn(0x7f01e006dd30 legacy=0x7f01e0070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.489+0000 7f01e4a00640 1 -- 172.21.15.62:0/2916287375 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e015d7e0 con 0x7f01e006d950 2024-03-23T11:21:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.489+0000 7f01df400640 1 -- 172.21.15.62:0/2916287375 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538089266 0 0) 0x7f01e015cfd0 con 0x7f01e006dd30 2024-03-23T11:21:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.491+0000 7f01e4a00640 1 --2- 172.21.15.62:0/2916287375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01e006d950 0x7f01e006c010 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f01d4009870 tx=0x7f01d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fcdb2f5d4167fc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.491+0000 7f01df400640 1 -- 172.21.15.62:0/2916287375 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d4031870 con 0x7f01e006d950 2024-03-23T11:21:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.491+0000 7f01df400640 1 -- 172.21.15.62:0/2916287375 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d40319d0 con 0x7f01e006d950 2024-03-23T11:21:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.491+0000 7f01df400640 1 -- 172.21.15.62:0/2916287375 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d4031d60 con 0x7f01e006d950 2024-03-23T11:21:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.491+0000 7f01e5e00640 1 -- 172.21.15.62:0/2916287375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01e006d950 msgr2=0x7f01e006c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.491+0000 7f01e5e00640 1 --2- 172.21.15.62:0/2916287375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01e006d950 0x7f01e006c010 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f01d4009870 tx=0x7f01d402b910 comp rx=0 tx=0).stop 2024-03-23T11:21:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.491+0000 7f01e5e00640 1 -- 172.21.15.62:0/2916287375 shutdown_connections 2024-03-23T11:21:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.492+0000 7f01e5e00640 1 --2- 172.21.15.62:0/2916287375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01e006d950 0x7f01e006c010 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.492+0000 7f01e5e00640 1 -- 172.21.15.62:0/2916287375 >> 172.21.15.62:0/2916287375 conn(0x7f01e00b0280 msgr2=0x7f01e00b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.492+0000 7f01e5e00640 1 -- 172.21.15.62:0/2916287375 shutdown_connections 2024-03-23T11:21:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.492+0000 7f01e5e00640 1 -- 172.21.15.62:0/2916287375 wait complete. 2024-03-23T11:21:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.493+0000 7f01e5e00640 1 Processor -- start 2024-03-23T11:21:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.493+0000 7f01e5e00640 1 -- start start 2024-03-23T11:21:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.494+0000 7f01e5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01e006d950 0x7f01e01489d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.494+0000 7f01e5e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01e006dd30 0x7f01e014feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.494+0000 7f01e5e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01e00ce690 0x7f01e00ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.494+0000 7f01e5e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e0150de0 con 0x7f01e006d950 2024-03-23T11:21:23.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.494+0000 7f01e5e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e0150820 con 0x7f01e00ce690 2024-03-23T11:21:23.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.494+0000 7f01e5e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e0150b00 con 0x7f01e006dd30 2024-03-23T11:21:23.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.495+0000 7f01e4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01e006d950 0x7f01e01489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:23.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.495+0000 7f01e4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01e006d950 0x7f01e01489d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46780/0 (socket says 172.21.15.62:46780) 2024-03-23T11:21:23.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.495+0000 7f01e4a00640 1 -- 172.21.15.62:0/667750234 learned_addr learned my addr 172.21.15.62:0/667750234 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:23.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.495+0000 7f01e4a00640 1 -- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01e006dd30 msgr2=0x7f01e014feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:23.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.495+0000 7f01e4a00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01e006dd30 0x7f01e014feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:23.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.495+0000 7f01e4a00640 1 -- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01e00ce690 msgr2=0x7f01e00ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:23.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.495+0000 7f01e5400640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01e00ce690 0x7f01e00ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:23.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.495+0000 7f01e4a00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01e00ce690 0x7f01e00ac030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:23.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.495+0000 7f01e4a00640 1 -- 172.21.15.62:0/667750234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d40094d0 con 0x7f01e006d950 2024-03-23T11:21:23.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.495+0000 7f01e5400640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01e00ce690 0x7f01e00ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:23.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.496+0000 7f01e4a00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01e006d950 0x7f01e01489d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f01d4009870 tx=0x7f01d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:23.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.496+0000 7f01dcc00640 1 -- 172.21.15.62:0/667750234 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d403ad30 con 0x7f01e006d950 2024-03-23T11:21:23.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.496+0000 7f01dcc00640 1 -- 172.21.15.62:0/667750234 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d403ae90 con 0x7f01e006d950 2024-03-23T11:21:23.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.496+0000 7f01e5e00640 1 -- 172.21.15.62:0/667750234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e01e2cd0 con 0x7f01e006d950 2024-03-23T11:21:23.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.496+0000 7f01dcc00640 1 -- 172.21.15.62:0/667750234 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d4044630 con 0x7f01e006d950 2024-03-23T11:21:23.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.496+0000 7f01e5e00640 1 -- 172.21.15.62:0/667750234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e0148f10 con 0x7f01e006d950 2024-03-23T11:21:23.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.497+0000 7f01e5e00640 1 -- 172.21.15.62:0/667750234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e00b0aa0 con 0x7f01e006d950 2024-03-23T11:21:23.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.498+0000 7f01dcc00640 1 -- 172.21.15.62:0/667750234 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f01d404d450 con 0x7f01e006d950 2024-03-23T11:21:23.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.499+0000 7f01dcc00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f01c006a7a0 0x7f01c006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:23.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.499+0000 7f01dcc00640 1 -- 172.21.15.62:0/667750234 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f01d402d040 con 0x7f01e006d950 2024-03-23T11:21:23.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.500+0000 7f01dfe00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f01c006a7a0 0x7f01c006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:23.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.501+0000 7f01dfe00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f01c006a7a0 0x7f01c006cc60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f01cc007fd0 tx=0x7f01cc007920 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:23.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.503+0000 7f01dcc00640 1 -- 172.21.15.62:0/667750234 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f01d4076900 con 0x7f01e006d950 2024-03-23T11:21:23.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:23.702+0000 7f01e5e00640 1 -- 172.21.15.62:0/667750234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f01e01e31f0 con 0x7f01e006d950 2024-03-23T11:21:24.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-23T11:21:24.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-23T11:21:24.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.201+0000 7f01dcc00640 1 -- 172.21.15.62:0/667750234 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v45) v1 ==== 138+0+0 (secure 0 0 0) 0x7f01e01e31f0 con 0x7f01e006d950 2024-03-23T11:21:24.201 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' created 2024-03-23T11:21:24.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.205+0000 7f01e5e00640 1 -- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f01c006a7a0 msgr2=0x7f01c006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:24.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.205+0000 7f01e5e00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f01c006a7a0 0x7f01c006cc60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f01cc007fd0 tx=0x7f01cc007920 comp rx=0 tx=0).stop 2024-03-23T11:21:24.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.205+0000 7f01e5e00640 1 -- 172.21.15.62:0/667750234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01e006d950 msgr2=0x7f01e01489d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:24.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.205+0000 7f01e5e00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01e006d950 0x7f01e01489d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f01d4009870 tx=0x7f01d40046c0 comp rx=0 tx=0).stop 2024-03-23T11:21:24.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.206+0000 7f01e5e00640 1 -- 172.21.15.62:0/667750234 shutdown_connections 2024-03-23T11:21:24.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.206+0000 7f01e5e00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f01c006a7a0 0x7f01c006cc60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:24.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.206+0000 7f01e5e00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01e00ce690 0x7f01e00ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:24.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.206+0000 7f01e5e00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01e006dd30 0x7f01e014feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:24.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.206+0000 7f01e5e00640 1 --2- 172.21.15.62:0/667750234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01e006d950 0x7f01e01489d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:24.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.206+0000 7f01e5e00640 1 -- 172.21.15.62:0/667750234 >> 172.21.15.62:0/667750234 conn(0x7f01e00b0280 msgr2=0x7f01e00b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:24.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.206+0000 7f01e5e00640 1 -- 172.21.15.62:0/667750234 shutdown_connections 2024-03-23T11:21:24.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.206+0000 7f01e5e00640 1 -- 172.21.15.62:0/667750234 wait complete. 2024-03-23T11:21:24.231 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T11:21:24.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.449+0000 7fd260200640 1 Processor -- start 2024-03-23T11:21:24.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.449+0000 7fd260200640 1 -- start start 2024-03-23T11:21:24.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.451+0000 7fd260200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd258070230 0x7fd258070610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:24.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.451+0000 7fd260200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd258159bf0 0x7fd2580cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:24.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.451+0000 7fd260200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd258070b50 con 0x7fd2580cd920 2024-03-23T11:21:24.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.451+0000 7fd260200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25815a080 con 0x7fd258159bf0 2024-03-23T11:21:24.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.451+0000 7fd260200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25815a360 con 0x7fd258070230 2024-03-23T11:21:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.451+0000 7fd25e200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd2580cd920 0x7fd2580ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43760/0 (socket says 172.21.15.62:43760) 2024-03-23T11:21:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.451+0000 7fd25e200640 1 -- 172.21.15.62:0/593956216 learned_addr learned my addr 172.21.15.62:0/593956216 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.452+0000 7fd25d800640 1 --2- 172.21.15.62:0/593956216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd258070230 0x7fd258070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.452+0000 7fd25ce00640 1 --2- 172.21.15.62:0/593956216 >> v2:172.21.15.120:3300/0 conn(0x7fd258159bf0 0x7fd2580cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.452+0000 7fd257e00640 1 -- 172.21.15.62:0/593956216 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66555801 0 0) 0x7fd258070b50 con 0x7fd2580cd920 2024-03-23T11:21:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.452+0000 7fd257e00640 1 -- 172.21.15.62:0/593956216 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd240003520 con 0x7fd2580cd920 2024-03-23T11:21:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.452+0000 7fd25d800640 1 -- 172.21.15.62:0/593956216 >> v2:172.21.15.120:3300/0 conn(0x7fd258159bf0 msgr2=0x7fd2580cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.452+0000 7fd25d800640 1 --2- 172.21.15.62:0/593956216 >> v2:172.21.15.120:3300/0 conn(0x7fd258159bf0 0x7fd2580cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.452+0000 7fd257e00640 1 -- 172.21.15.62:0/593956216 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4101407740 0 0) 0x7fd240003520 con 0x7fd2580cd920 2024-03-23T11:21:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.452+0000 7fd25d800640 1 -- 172.21.15.62:0/593956216 >> v1:172.21.15.62:6789/0 conn(0x7fd2580cd920 legacy=0x7fd2580ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.452+0000 7fd25d800640 1 -- 172.21.15.62:0/593956216 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25815a640 con 0x7fd258070230 2024-03-23T11:21:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.454+0000 7fd25d800640 1 --2- 172.21.15.62:0/593956216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd258070230 0x7fd258070610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0097a0 tx=0x7fd24c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=405d39aa4f75f1c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:24.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.454+0000 7fd257e00640 1 -- 172.21.15.62:0/593956216 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2500019f0 con 0x7fd258070230 2024-03-23T11:21:24.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.455+0000 7fd257e00640 1 -- 172.21.15.62:0/593956216 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24c02e050 con 0x7fd258070230 2024-03-23T11:21:24.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.455+0000 7fd257e00640 1 -- 172.21.15.62:0/593956216 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c030710 con 0x7fd258070230 2024-03-23T11:21:24.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.455+0000 7fd260200640 1 -- 172.21.15.62:0/593956216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd258070230 msgr2=0x7fd258070610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:24.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.455+0000 7fd260200640 1 --2- 172.21.15.62:0/593956216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd258070230 0x7fd258070610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0097a0 tx=0x7fd24c02b790 comp rx=0 tx=0).stop 2024-03-23T11:21:24.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.455+0000 7fd260200640 1 -- 172.21.15.62:0/593956216 shutdown_connections 2024-03-23T11:21:24.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.455+0000 7fd260200640 1 --2- 172.21.15.62:0/593956216 >> v2:172.21.15.120:3300/0 conn(0x7fd258159bf0 0x7fd2580cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:24.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.455+0000 7fd260200640 1 --2- 172.21.15.62:0/593956216 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd258070230 0x7fd258070610 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:24.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.456+0000 7fd260200640 1 -- 172.21.15.62:0/593956216 >> 172.21.15.62:0/593956216 conn(0x7fd258065a20 msgr2=0x7fd258065e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:24.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.456+0000 7fd260200640 1 -- 172.21.15.62:0/593956216 shutdown_connections 2024-03-23T11:21:24.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.456+0000 7fd260200640 1 -- 172.21.15.62:0/593956216 wait complete. 2024-03-23T11:21:24.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.458+0000 7fd260200640 1 Processor -- start 2024-03-23T11:21:24.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.458+0000 7fd260200640 1 -- start start 2024-03-23T11:21:24.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.459+0000 7fd260200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd258070230 0x7fd25813c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:24.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.459+0000 7fd260200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cd920 0x7fd25815baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:24.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.459+0000 7fd260200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25813e580 0x7fd25813b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:24.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.459+0000 7fd260200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25813e160 con 0x7fd2580cd920 2024-03-23T11:21:24.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.459+0000 7fd260200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd258159480 con 0x7fd258070230 2024-03-23T11:21:24.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.459+0000 7fd260200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25813d4e0 con 0x7fd25813e580 2024-03-23T11:21:24.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cd920 0x7fd25815baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:24.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cd920 0x7fd25815baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46792/0 (socket says 172.21.15.62:46792) 2024-03-23T11:21:24.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25ce00640 1 -- 172.21.15.62:0/4209390992 learned_addr learned my addr 172.21.15.62:0/4209390992 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:24.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25d800640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd258070230 0x7fd25813c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:24.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25e200640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25813e580 0x7fd25813b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:24.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25ce00640 1 -- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25813e580 msgr2=0x7fd25813b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:24.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25ce00640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25813e580 0x7fd25813b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:24.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25ce00640 1 -- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd258070230 msgr2=0x7fd25813c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:24.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25ce00640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd258070230 0x7fd25813c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:24.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25ce00640 1 -- 172.21.15.62:0/4209390992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c009400 con 0x7fd2580cd920 2024-03-23T11:21:24.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25e200640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25813e580 0x7fd25813b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:24.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.460+0000 7fd25d800640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd258070230 0x7fd25813c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:24.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.461+0000 7fd25ce00640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cd920 0x7fd25815baa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd24400e870 tx=0x7fd24400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:24.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.461+0000 7fd255600640 1 -- 172.21.15.62:0/4209390992 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2440094f0 con 0x7fd2580cd920 2024-03-23T11:21:24.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.461+0000 7fd255600640 1 -- 172.21.15.62:0/4209390992 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24400c940 con 0x7fd2580cd920 2024-03-23T11:21:24.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.461+0000 7fd255600640 1 -- 172.21.15.62:0/4209390992 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24400ccf0 con 0x7fd2580cd920 2024-03-23T11:21:24.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.461+0000 7fd260200640 1 -- 172.21.15.62:0/4209390992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25813d7c0 con 0x7fd2580cd920 2024-03-23T11:21:24.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.461+0000 7fd260200640 1 -- 172.21.15.62:0/4209390992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25813dd80 con 0x7fd2580cd920 2024-03-23T11:21:24.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.462+0000 7fd260200640 1 -- 172.21.15.62:0/4209390992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd258066bb0 con 0x7fd2580cd920 2024-03-23T11:21:24.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.464+0000 7fd255600640 1 -- 172.21.15.62:0/4209390992 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd244013470 con 0x7fd2580cd920 2024-03-23T11:21:24.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.465+0000 7fd255600640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 0x7fd23806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:24.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.465+0000 7fd255600640 1 -- 172.21.15.62:0/4209390992 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd244089450 con 0x7fd2580cd920 2024-03-23T11:21:24.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.466+0000 7fd25d800640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 0x7fd23806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:24.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.467+0000 7fd25d800640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 0x7fd23806cc60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd24c034000 tx=0x7fd24c02f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:24.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.469+0000 7fd255600640 1 -- 172.21.15.62:0/4209390992 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd244054ab0 con 0x7fd2580cd920 2024-03-23T11:21:24.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:24.659+0000 7fd260200640 1 -- 172.21.15.62:0/4209390992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd2580714e0 con 0x7fd2580cd920 2024-03-23T11:21:25.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.208+0000 7fd255600640 1 -- 172.21.15.62:0/4209390992 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v46) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd2440580d0 con 0x7fd2580cd920 2024-03-23T11:21:25.209 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' created 2024-03-23T11:21:25.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.214+0000 7fd260200640 1 -- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 msgr2=0x7fd23806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.214+0000 7fd260200640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 0x7fd23806cc60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd24c034000 tx=0x7fd24c02f000 comp rx=0 tx=0).stop 2024-03-23T11:21:25.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.215+0000 7fd260200640 1 -- 172.21.15.62:0/4209390992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cd920 msgr2=0x7fd25815baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.215+0000 7fd260200640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cd920 0x7fd25815baa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd24400e870 tx=0x7fd24400ed30 comp rx=0 tx=0).stop 2024-03-23T11:21:25.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.216+0000 7fd260200640 1 -- 172.21.15.62:0/4209390992 shutdown_connections 2024-03-23T11:21:25.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.216+0000 7fd260200640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 0x7fd23806cc60 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd24c034000 tx=0x7fd24c02f000 comp rx=0 tx=0).stop 2024-03-23T11:21:25.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.216+0000 7fd260200640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cd920 0x7fd25815baa0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.216+0000 7fd260200640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25813e580 0x7fd25813b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.216+0000 7fd260200640 1 --2- 172.21.15.62:0/4209390992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd258070230 0x7fd25813c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.216+0000 7fd260200640 1 -- 172.21.15.62:0/4209390992 >> 172.21.15.62:0/4209390992 conn(0x7fd258065a20 msgr2=0x7fd25813f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:25.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.216+0000 7fd260200640 1 -- 172.21.15.62:0/4209390992 shutdown_connections 2024-03-23T11:21:25.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.217+0000 7fd260200640 1 -- 172.21.15.62:0/4209390992 wait complete. 2024-03-23T11:21:25.251 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T11:21:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.435+0000 7f66d3200640 1 Processor -- start 2024-03-23T11:21:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.435+0000 7f66d3200640 1 -- start start 2024-03-23T11:21:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.436+0000 7f66d3200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66cc15fcf0 0x7f66cc0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.437+0000 7f66d3200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f66cc0b6dc0 0x7f66cc160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.437+0000 7f66d3200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66cc160dc0 con 0x7f66cc160420 2024-03-23T11:21:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.437+0000 7f66d3200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66cc0d3540 con 0x7f66cc0b6dc0 2024-03-23T11:21:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.437+0000 7f66d3200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc0ce510 con 0x7f66cc15fcf0 2024-03-23T11:21:25.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.437+0000 7f66d2800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f66cc0b6dc0 0x7f66cc160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.437+0000 7f66d1e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66cc15fcf0 0x7f66cc0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.437+0000 7f66d2800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f66cc0b6dc0 0x7f66cc160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:46136/0 (socket says 172.21.15.62:46136) 2024-03-23T11:21:25.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.437+0000 7f66d2800640 1 -- 172.21.15.62:0/3572365570 learned_addr learned my addr 172.21.15.62:0/3572365570 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.438+0000 7f66d1e00640 1 -- 172.21.15.62:0/3572365570 >> v2:172.21.15.120:3300/0 conn(0x7f66cc0b6dc0 msgr2=0x7f66cc160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.438+0000 7f66d1e00640 1 --2- 172.21.15.62:0/3572365570 >> v2:172.21.15.120:3300/0 conn(0x7f66cc0b6dc0 0x7f66cc160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.438+0000 7f66d1e00640 1 -- 172.21.15.62:0/3572365570 >> v1:172.21.15.120:6789/0 conn(0x7f66cc160420 legacy=0x7f66cc0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.438+0000 7f66d1e00640 1 -- 172.21.15.62:0/3572365570 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc0cdc70 con 0x7f66cc15fcf0 2024-03-23T11:21:25.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.440+0000 7f66d1e00640 1 --2- 172.21.15.62:0/3572365570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66cc15fcf0 0x7f66cc0d3000 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f66c00097a0 tx=0x7f66c002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d57b44d9c9ecac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:25.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.440+0000 7f66d0a00640 1 -- 172.21.15.62:0/3572365570 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c002f050 con 0x7f66cc15fcf0 2024-03-23T11:21:25.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.440+0000 7f66d0a00640 1 -- 172.21.15.62:0/3572365570 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c0031700 con 0x7f66cc15fcf0 2024-03-23T11:21:25.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.441+0000 7f66d0a00640 1 -- 172.21.15.62:0/3572365570 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0031a90 con 0x7f66cc15fcf0 2024-03-23T11:21:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.441+0000 7f66d3200640 1 -- 172.21.15.62:0/3572365570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66cc15fcf0 msgr2=0x7f66cc0d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.441+0000 7f66d3200640 1 --2- 172.21.15.62:0/3572365570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66cc15fcf0 0x7f66cc0d3000 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f66c00097a0 tx=0x7f66c002b7a0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.441+0000 7f66d3200640 1 -- 172.21.15.62:0/3572365570 shutdown_connections 2024-03-23T11:21:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.441+0000 7f66d3200640 1 --2- 172.21.15.62:0/3572365570 >> v2:172.21.15.120:3300/0 conn(0x7f66cc0b6dc0 0x7f66cc160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.441+0000 7f66d3200640 1 --2- 172.21.15.62:0/3572365570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66cc15fcf0 0x7f66cc0d3000 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.441+0000 7f66d3200640 1 -- 172.21.15.62:0/3572365570 >> 172.21.15.62:0/3572365570 conn(0x7f66cc089f60 msgr2=0x7f66cc0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.441+0000 7f66d3200640 1 -- 172.21.15.62:0/3572365570 shutdown_connections 2024-03-23T11:21:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.441+0000 7f66d3200640 1 -- 172.21.15.62:0/3572365570 wait complete. 2024-03-23T11:21:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.443+0000 7f66d3200640 1 Processor -- start 2024-03-23T11:21:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.443+0000 7f66d3200640 1 -- start start 2024-03-23T11:21:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.444+0000 7f66d3200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66cc0b6dc0 0x7f66cc1f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.444+0000 7f66d3200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66cc15fcf0 0x7f66cc06c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.444+0000 7f66d3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66cc160420 0x7f66cc1f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.444+0000 7f66d3200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc06c5a0 con 0x7f66cc160420 2024-03-23T11:21:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.444+0000 7f66d3200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc1f30c0 con 0x7f66cc15fcf0 2024-03-23T11:21:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.444+0000 7f66d3200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc1f4350 con 0x7f66cc0b6dc0 2024-03-23T11:21:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.444+0000 7f66d2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66cc160420 0x7f66cc1f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.444+0000 7f66d2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66cc160420 0x7f66cc1f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46798/0 (socket says 172.21.15.62:46798) 2024-03-23T11:21:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.444+0000 7f66d2800640 1 -- 172.21.15.62:0/443745258 learned_addr learned my addr 172.21.15.62:0/443745258 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.445+0000 7f66d1400640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66cc15fcf0 0x7f66cc06c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.445+0000 7f66d1e00640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66cc0b6dc0 0x7f66cc1f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.445+0000 7f66d2800640 1 -- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66cc0b6dc0 msgr2=0x7f66cc1f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.445+0000 7f66d2800640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66cc0b6dc0 0x7f66cc1f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.445+0000 7f66d2800640 1 -- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66cc15fcf0 msgr2=0x7f66cc06c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.445+0000 7f66d2800640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66cc15fcf0 0x7f66cc06c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.445+0000 7f66d2800640 1 -- 172.21.15.62:0/443745258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c0009400 con 0x7f66cc160420 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.445+0000 7f66d2800640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66cc160420 0x7f66cc1f2b80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f66c800efc0 tx=0x7f66c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.445+0000 7f66c6000640 1 -- 172.21.15.62:0/443745258 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8009b60 con 0x7f66cc160420 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.445+0000 7f66c6000640 1 -- 172.21.15.62:0/443745258 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c800f040 con 0x7f66cc160420 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.446+0000 7f66d3200640 1 -- 172.21.15.62:0/443745258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66cc1f4630 con 0x7f66cc160420 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.446+0000 7f66c6000640 1 -- 172.21.15.62:0/443745258 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8003990 con 0x7f66cc160420 2024-03-23T11:21:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.446+0000 7f66d3200640 1 -- 172.21.15.62:0/443745258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc1f4bf0 con 0x7f66cc160420 2024-03-23T11:21:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.447+0000 7f66d3200640 1 -- 172.21.15.62:0/443745258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66cc066bb0 con 0x7f66cc160420 2024-03-23T11:21:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.448+0000 7f66c6000640 1 -- 172.21.15.62:0/443745258 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f66c8003af0 con 0x7f66cc160420 2024-03-23T11:21:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.448+0000 7f66c6000640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f66b006a7a0 0x7f66b006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.449+0000 7f66c6000640 1 -- 172.21.15.62:0/443745258 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f66c8089450 con 0x7f66cc160420 2024-03-23T11:21:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.449+0000 7f66d1e00640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f66b006a7a0 0x7f66b006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.450+0000 7f66d1e00640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f66b006a7a0 0x7f66b006cc60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f66c0035000 tx=0x7f66c0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.454+0000 7f66c6000640 1 -- 172.21.15.62:0/443745258 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f66c80547b0 con 0x7f66cc160420 2024-03-23T11:21:25.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.683+0000 7f66d3200640 1 -- 172.21.15.62:0/443745258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f66cc0cbef0 con 0x7f66cc160420 2024-03-23T11:21:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.691+0000 7f66c6000640 1 -- 172.21.15.62:0/443745258 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f66c8057dd0 con 0x7f66cc160420 2024-03-23T11:21:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:new fs with metadata pool 6 and data pool 7 2024-03-23T11:21:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.695+0000 7f66d3200640 1 -- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f66b006a7a0 msgr2=0x7f66b006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.695+0000 7f66d3200640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f66b006a7a0 0x7f66b006cc60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f66c0035000 tx=0x7f66c0008030 comp rx=0 tx=0).stop 2024-03-23T11:21:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.695+0000 7f66d3200640 1 -- 172.21.15.62:0/443745258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66cc160420 msgr2=0x7f66cc1f2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.695+0000 7f66d3200640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66cc160420 0x7f66cc1f2b80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f66c800efc0 tx=0x7f66c800c490 comp rx=0 tx=0).stop 2024-03-23T11:21:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.696+0000 7f66d3200640 1 -- 172.21.15.62:0/443745258 shutdown_connections 2024-03-23T11:21:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.696+0000 7f66d3200640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f66b006a7a0 0x7f66b006cc60 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.696+0000 7f66d3200640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66cc160420 0x7f66cc1f2b80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.696+0000 7f66d3200640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66cc15fcf0 0x7f66cc06c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.696+0000 7f66d3200640 1 --2- 172.21.15.62:0/443745258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66cc0b6dc0 0x7f66cc1f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.696+0000 7f66d3200640 1 -- 172.21.15.62:0/443745258 >> 172.21.15.62:0/443745258 conn(0x7f66cc089f60 msgr2=0x7f66cc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.697+0000 7f66d3200640 1 -- 172.21.15.62:0/443745258 shutdown_connections 2024-03-23T11:21:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.697+0000 7f66d3200640 1 -- 172.21.15.62:0/443745258 wait complete. 2024-03-23T11:21:25.730 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:21:25.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.903+0000 7f7c10a00640 1 Processor -- start 2024-03-23T11:21:25.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.903+0000 7f7c10a00640 1 -- start start 2024-03-23T11:21:25.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.904+0000 7f7c10a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7c0c1571e0 0x7f7c0c0ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.904+0000 7f7c10a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7c0c1613a0 0x7f7c0c15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.904+0000 7f7c10a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c06cf40 con 0x7f7c0c0ade70 2024-03-23T11:21:25.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.904+0000 7f7c10a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0c1575c0 con 0x7f7c0c1571e0 2024-03-23T11:21:25.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.904+0000 7f7c10a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0c06c3a0 con 0x7f7c0c1613a0 2024-03-23T11:21:25.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.905+0000 7f7c09e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7c0c1571e0 0x7f7c0c0ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.905+0000 7f7c09400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7c0c0ade70 0x7f7c0c06d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43770/0 (socket says 172.21.15.62:43770) 2024-03-23T11:21:25.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.905+0000 7f7c09400640 1 -- 172.21.15.62:0/3817891801 learned_addr learned my addr 172.21.15.62:0/3817891801 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:25.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.905+0000 7f7c0a800640 1 --2- 172.21.15.62:0/3817891801 >> v2:172.21.15.120:3300/0 conn(0x7f7c0c1613a0 0x7f7c0c15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.905+0000 7f7c09e00640 1 -- 172.21.15.62:0/3817891801 >> v2:172.21.15.120:3300/0 conn(0x7f7c0c1613a0 msgr2=0x7f7c0c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:25.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.905+0000 7f7c09e00640 1 --2- 172.21.15.62:0/3817891801 >> v2:172.21.15.120:3300/0 conn(0x7f7c0c1613a0 0x7f7c0c15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.905+0000 7f7c09e00640 1 -- 172.21.15.62:0/3817891801 >> v1:172.21.15.62:6789/0 conn(0x7f7c0c0ade70 legacy=0x7f7c0c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.905+0000 7f7c09e00640 1 -- 172.21.15.62:0/3817891801 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c06c610 con 0x7f7c0c1571e0 2024-03-23T11:21:25.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.905+0000 7f7c08a00640 1 -- 172.21.15.62:0/3817891801 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969505906 0 0) 0x7f7c0c06cf40 con 0x7f7c0c0ade70 2024-03-23T11:21:25.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.905+0000 7f7c0a800640 1 --2- 172.21.15.62:0/3817891801 >> v2:172.21.15.120:3300/0 conn(0x7f7c0c1613a0 0x7f7c0c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:25.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.907+0000 7f7c09e00640 1 --2- 172.21.15.62:0/3817891801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c0c1571e0 0x7f7c0c0ad930 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0ac760 tx=0x7f7bfc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573b727b22c568a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:25.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.907+0000 7f7c08a00640 1 -- 172.21.15.62:0/3817891801 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc02e070 con 0x7f7c0c1571e0 2024-03-23T11:21:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.907+0000 7f7c08a00640 1 -- 172.21.15.62:0/3817891801 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bfc0315a0 con 0x7f7c0c1571e0 2024-03-23T11:21:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.907+0000 7f7c08a00640 1 -- 172.21.15.62:0/3817891801 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc031930 con 0x7f7c0c1571e0 2024-03-23T11:21:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.908+0000 7f7c10a00640 1 -- 172.21.15.62:0/3817891801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c0c1571e0 msgr2=0x7f7c0c0ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.908+0000 7f7c10a00640 1 --2- 172.21.15.62:0/3817891801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c0c1571e0 0x7f7c0c0ad930 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0ac760 tx=0x7f7bfc02b840 comp rx=0 tx=0).stop 2024-03-23T11:21:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.908+0000 7f7c10a00640 1 -- 172.21.15.62:0/3817891801 shutdown_connections 2024-03-23T11:21:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.908+0000 7f7c10a00640 1 --2- 172.21.15.62:0/3817891801 >> v2:172.21.15.120:3300/0 conn(0x7f7c0c1613a0 0x7f7c0c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.908+0000 7f7c10a00640 1 --2- 172.21.15.62:0/3817891801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c0c1571e0 0x7f7c0c0ad930 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.908+0000 7f7c10a00640 1 -- 172.21.15.62:0/3817891801 >> 172.21.15.62:0/3817891801 conn(0x7f7c0c0b02e0 msgr2=0x7f7c0c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.908+0000 7f7c10a00640 1 -- 172.21.15.62:0/3817891801 shutdown_connections 2024-03-23T11:21:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.908+0000 7f7c10a00640 1 -- 172.21.15.62:0/3817891801 wait complete. 2024-03-23T11:21:25.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.909+0000 7f7c10a00640 1 Processor -- start 2024-03-23T11:21:25.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.910+0000 7f7c10a00640 1 -- start start 2024-03-23T11:21:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.910+0000 7f7c10a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c0c0ade70 0x7f7c0c0cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.910+0000 7f7c10a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c0c1571e0 0x7f7c0c1f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.910+0000 7f7c10a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c0c1613a0 0x7f7c0c151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.910+0000 7f7c10a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c152420 con 0x7f7c0c1613a0 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.910+0000 7f7c10a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0cb960 con 0x7f7c0c0ade70 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.910+0000 7f7c10a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0cbc40 con 0x7f7c0c1571e0 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.911+0000 7f7c0a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c0c1613a0 0x7f7c0c151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.911+0000 7f7c0a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c0c1613a0 0x7f7c0c151ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46818/0 (socket says 172.21.15.62:46818) 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.911+0000 7f7c0a800640 1 -- 172.21.15.62:0/667213574 learned_addr learned my addr 172.21.15.62:0/667213574 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.911+0000 7f7c09400640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c0c1571e0 0x7f7c0c1f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.911+0000 7f7c09e00640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c0c0ade70 0x7f7c0c0cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.911+0000 7f7c0a800640 1 -- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c0c1571e0 msgr2=0x7f7c0c1f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.911+0000 7f7c0a800640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c0c1571e0 0x7f7c0c1f2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.911+0000 7f7c0a800640 1 -- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c0c0ade70 msgr2=0x7f7c0c0cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.911+0000 7f7c0a800640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c0c0ade70 0x7f7c0c0cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:25.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.911+0000 7f7c0a800640 1 -- 172.21.15.62:0/667213574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc009400 con 0x7f7c0c1613a0 2024-03-23T11:21:25.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.912+0000 7f7c0a800640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c0c1613a0 0x7f7c0c151ee0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400ce80 tx=0x7f7c0400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:25.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.912+0000 7f7c02000640 1 -- 172.21.15.62:0/667213574 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400d840 con 0x7f7c0c1613a0 2024-03-23T11:21:25.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.912+0000 7f7c02000640 1 -- 172.21.15.62:0/667213574 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0400d9e0 con 0x7f7c0c1613a0 2024-03-23T11:21:25.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.912+0000 7f7c02000640 1 -- 172.21.15.62:0/667213574 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400dd90 con 0x7f7c0c1613a0 2024-03-23T11:21:25.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.912+0000 7f7c10a00640 1 -- 172.21.15.62:0/667213574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0c158210 con 0x7f7c0c1613a0 2024-03-23T11:21:25.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.912+0000 7f7c10a00640 1 -- 172.21.15.62:0/667213574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c15c550 con 0x7f7c0c1613a0 2024-03-23T11:21:25.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.913+0000 7f7c10a00640 1 -- 172.21.15.62:0/667213574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0c066c90 con 0x7f7c0c1613a0 2024-03-23T11:21:25.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.914+0000 7f7c02000640 1 -- 172.21.15.62:0/667213574 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7c04015440 con 0x7f7c0c1613a0 2024-03-23T11:21:25.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.915+0000 7f7c02000640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bec06a7d0 0x7f7bec06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:25.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.915+0000 7f7c02000640 1 -- 172.21.15.62:0/667213574 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c04089710 con 0x7f7c0c1613a0 2024-03-23T11:21:25.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.915+0000 7f7c09e00640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bec06a7d0 0x7f7bec06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:25.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.917+0000 7f7c09e00640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bec06a7d0 0x7f7bec06cc90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc036000 tx=0x7f7bfc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:25.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:25.919+0000 7f7c02000640 1 -- 172.21.15.62:0/667213574 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7c04054b40 con 0x7f7c0c1613a0 2024-03-23T11:21:26.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.118+0000 7f7c10a00640 1 -- 172.21.15.62:0/667213574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c0c0b38c0 con 0x7f7c0c1613a0 2024-03-23T11:21:26.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.120+0000 7f7c02000640 1 -- 172.21.15.62:0/667213574 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16058 (secure 0 0 0) 0x7f7c04058160 con 0x7f7c0c1613a0 2024-03-23T11:21:26.120 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:26.120 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":47,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:21:25.686476+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:21:23.756222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-23T11:21:24.715577+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:21:26.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 -- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bec06a7d0 msgr2=0x7f7bec06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:26.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bec06a7d0 0x7f7bec06cc90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc036000 tx=0x7f7bfc008030 comp rx=0 tx=0).stop 2024-03-23T11:21:26.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 -- 172.21.15.62:0/667213574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c0c1613a0 msgr2=0x7f7c0c151ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:26.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c0c1613a0 0x7f7c0c151ee0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400ce80 tx=0x7f7c0400a710 comp rx=0 tx=0).stop 2024-03-23T11:21:26.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 -- 172.21.15.62:0/667213574 shutdown_connections 2024-03-23T11:21:26.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7bec06a7d0 0x7f7bec06cc90 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c0c1613a0 0x7f7c0c151ee0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c0c1571e0 0x7f7c0c1f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 --2- 172.21.15.62:0/667213574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c0c0ade70 0x7f7c0c0cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 -- 172.21.15.62:0/667213574 >> 172.21.15.62:0/667213574 conn(0x7f7c0c0b02e0 msgr2=0x7f7c0c155010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:26.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 -- 172.21.15.62:0/667213574 shutdown_connections 2024-03-23T11:21:26.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.123+0000 7f7c10a00640 1 -- 172.21.15.62:0/667213574 wait complete. 2024-03-23T11:21:26.142 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:21:26.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.365+0000 7f36cec00640 1 Processor -- start 2024-03-23T11:21:26.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.365+0000 7f36cec00640 1 -- start start 2024-03-23T11:21:26.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.367+0000 7f36cec00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f36c8160c00 0x7f36c806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:26.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.367+0000 7f36cec00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c8160fe0 con 0x7f36c80cde90 2024-03-23T11:21:26.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.367+0000 7f36cec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c80b1510 con 0x7f36c80cda20 2024-03-23T11:21:26.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.367+0000 7f36cec00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c80b1ad0 con 0x7f36c8160c00 2024-03-23T11:21:26.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.367+0000 7f36cd800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f36c8160c00 0x7f36c806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:26.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.367+0000 7f36cd800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f36c8160c00 0x7f36c806d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46824/0 (socket says 172.21.15.62:46824) 2024-03-23T11:21:26.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.367+0000 7f36cd800640 1 -- 172.21.15.62:0/1452828594 learned_addr learned my addr 172.21.15.62:0/1452828594 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:26.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.367+0000 7f36cd800640 1 -- 172.21.15.62:0/1452828594 >> v1:172.21.15.120:6789/0 conn(0x7f36c80cda20 legacy=0x7f36c806de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:26.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.367+0000 7f36cd800640 1 -- 172.21.15.62:0/1452828594 >> v1:172.21.15.62:6789/0 conn(0x7f36c80cde90 legacy=0x7f36c80caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:26.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.367+0000 7f36cd800640 1 -- 172.21.15.62:0/1452828594 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c80b17f0 con 0x7f36c8160c00 2024-03-23T11:21:26.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.369+0000 7f36cd800640 1 --2- 172.21.15.62:0/1452828594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c8160c00 0x7f36c806d530 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009870 tx=0x7f36bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e1e736f8a0b031 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:26.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.369+0000 7f36c7e00640 1 -- 172.21.15.62:0/1452828594 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc030860 con 0x7f36c8160c00 2024-03-23T11:21:26.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.369+0000 7f36c7e00640 1 -- 172.21.15.62:0/1452828594 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36bc0309c0 con 0x7f36c8160c00 2024-03-23T11:21:26.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.369+0000 7f36c7e00640 1 -- 172.21.15.62:0/1452828594 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc030d50 con 0x7f36c8160c00 2024-03-23T11:21:26.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.369+0000 7f36cec00640 1 -- 172.21.15.62:0/1452828594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c8160c00 msgr2=0x7f36c806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:26.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.369+0000 7f36cec00640 1 --2- 172.21.15.62:0/1452828594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c8160c00 0x7f36c806d530 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009870 tx=0x7f36bc02b9a0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.370+0000 7f36cec00640 1 -- 172.21.15.62:0/1452828594 shutdown_connections 2024-03-23T11:21:26.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.370+0000 7f36cec00640 1 --2- 172.21.15.62:0/1452828594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c8160c00 0x7f36c806d530 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.370+0000 7f36cec00640 1 -- 172.21.15.62:0/1452828594 >> 172.21.15.62:0/1452828594 conn(0x7f36c8066050 msgr2=0x7f36c8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:26.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.370+0000 7f36cec00640 1 -- 172.21.15.62:0/1452828594 shutdown_connections 2024-03-23T11:21:26.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.370+0000 7f36cec00640 1 -- 172.21.15.62:0/1452828594 wait complete. 2024-03-23T11:21:26.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.371+0000 7f36cec00640 1 Processor -- start 2024-03-23T11:21:26.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.371+0000 7f36cec00640 1 -- start start 2024-03-23T11:21:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.372+0000 7f36cec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c80cda20 0x7f36c81e93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.372+0000 7f36cec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36c80cde90 0x7f36c81466a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.372+0000 7f36cec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36c8160c00 0x7f36c8159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.372+0000 7f36cec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c8158a70 con 0x7f36c80cda20 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.372+0000 7f36cec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c8144e30 con 0x7f36c80cde90 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.372+0000 7f36cec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c8145600 con 0x7f36c8160c00 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.373+0000 7f36cd800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c80cda20 0x7f36c81e93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.373+0000 7f36cd800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c80cda20 0x7f36c81e93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46838/0 (socket says 172.21.15.62:46838) 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.373+0000 7f36cd800640 1 -- 172.21.15.62:0/231760291 learned_addr learned my addr 172.21.15.62:0/231760291 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.373+0000 7f36cd800640 1 -- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36c8160c00 msgr2=0x7f36c8159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.373+0000 7f36cd800640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36c8160c00 0x7f36c8159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.373+0000 7f36cce00640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36c80cde90 0x7f36c81466a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.373+0000 7f36cd800640 1 -- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36c80cde90 msgr2=0x7f36c81466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.373+0000 7f36cd800640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36c80cde90 0x7f36c81466a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.373+0000 7f36cd800640 1 -- 172.21.15.62:0/231760291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc0094d0 con 0x7f36c80cda20 2024-03-23T11:21:26.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.373+0000 7f36cce00640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36c80cde90 0x7f36c81466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.373+0000 7f36cd800640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c80cda20 0x7f36c81e93b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009870 tx=0x7f36bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.374+0000 7f36c5600640 1 -- 172.21.15.62:0/231760291 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc030810 con 0x7f36c80cda20 2024-03-23T11:21:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.374+0000 7f36c5600640 1 -- 172.21.15.62:0/231760291 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36bc039d90 con 0x7f36c80cda20 2024-03-23T11:21:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.374+0000 7f36c5600640 1 -- 172.21.15.62:0/231760291 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc043630 con 0x7f36c80cda20 2024-03-23T11:21:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.374+0000 7f36cec00640 1 -- 172.21.15.62:0/231760291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c81458e0 con 0x7f36c80cda20 2024-03-23T11:21:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.374+0000 7f36cec00640 1 -- 172.21.15.62:0/231760291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c8145ea0 con 0x7f36c80cda20 2024-03-23T11:21:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.375+0000 7f36cec00640 1 -- 172.21.15.62:0/231760291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c8066bb0 con 0x7f36c80cda20 2024-03-23T11:21:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.376+0000 7f36c5600640 1 -- 172.21.15.62:0/231760291 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f36bc02f3b0 con 0x7f36c80cda20 2024-03-23T11:21:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.377+0000 7f36c5600640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36ac06a7a0 0x7f36ac06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.377+0000 7f36c5600640 1 -- 172.21.15.62:0/231760291 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36bc0ab1c0 con 0x7f36c80cda20 2024-03-23T11:21:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.378+0000 7f36cce00640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36ac06a7a0 0x7f36ac06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:26.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.379+0000 7f36cce00640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36ac06a7a0 0x7f36ac06cc60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f36c8159420 tx=0x7f36b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.381+0000 7f36c5600640 1 -- 172.21.15.62:0/231760291 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f36bc0765e0 con 0x7f36c80cda20 2024-03-23T11:21:26.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.576+0000 7f36cec00640 1 -- 172.21.15.62:0/231760291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36c80cabe0 con 0x7f36c80cda20 2024-03-23T11:21:26.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.578+0000 7f36c5600640 1 -- 172.21.15.62:0/231760291 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16058 (secure 0 0 0) 0x7f36bc079c00 con 0x7f36c80cda20 2024-03-23T11:21:26.578 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:26.578 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":47,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:21:25.686476+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:21:23.756222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-23T11:21:24.715577+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:21:26.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.580+0000 7f36cec00640 1 -- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36ac06a7a0 msgr2=0x7f36ac06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.581+0000 7f36cec00640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36ac06a7a0 0x7f36ac06cc60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f36c8159420 tx=0x7f36b8009040 comp rx=0 tx=0).stop 2024-03-23T11:21:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.581+0000 7f36cec00640 1 -- 172.21.15.62:0/231760291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c80cda20 msgr2=0x7f36c81e93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.581+0000 7f36cec00640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c80cda20 0x7f36c81e93b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009870 tx=0x7f36bc004730 comp rx=0 tx=0).stop 2024-03-23T11:21:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.581+0000 7f36cec00640 1 -- 172.21.15.62:0/231760291 shutdown_connections 2024-03-23T11:21:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.581+0000 7f36cec00640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36ac06a7a0 0x7f36ac06cc60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.581+0000 7f36cec00640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36c8160c00 0x7f36c8159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.581+0000 7f36cec00640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36c80cde90 0x7f36c81466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.581+0000 7f36cec00640 1 --2- 172.21.15.62:0/231760291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36c80cda20 0x7f36c81e93b0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.581+0000 7f36cec00640 1 -- 172.21.15.62:0/231760291 >> 172.21.15.62:0/231760291 conn(0x7f36c8066050 msgr2=0x7f36c80cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.581+0000 7f36cec00640 1 -- 172.21.15.62:0/231760291 shutdown_connections 2024-03-23T11:21:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.581+0000 7f36cec00640 1 -- 172.21.15.62:0/231760291 wait complete. 2024-03-23T11:21:26.600 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T11:21:26.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.825+0000 7fdfb4a00640 1 Processor -- start 2024-03-23T11:21:26.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.825+0000 7fdfb4a00640 1 -- start start 2024-03-23T11:21:26.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.827+0000 7fdfb4a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdfb006c300 0x7fdfb00ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:26.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.827+0000 7fdfb4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdfb00cafa0 0x7fdfb00cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:26.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.827+0000 7fdfb4a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb00cbc60 con 0x7fdfb006c6e0 2024-03-23T11:21:26.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.827+0000 7fdfb4a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb00cbf40 con 0x7fdfb00cafa0 2024-03-23T11:21:26.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.827+0000 7fdfb4a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb00cc220 con 0x7fdfb006c300 2024-03-23T11:21:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.827+0000 7fdfa7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdfb00cafa0 0x7fdfb00cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.828+0000 7fdfa7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdfb00cafa0 0x7fdfb00cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46852/0 (socket says 172.21.15.62:46852) 2024-03-23T11:21:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.828+0000 7fdfa7e00640 1 -- 172.21.15.62:0/2850737731 learned_addr learned my addr 172.21.15.62:0/2850737731 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.828+0000 7fdfade00640 1 --2- 172.21.15.62:0/2850737731 >> v2:172.21.15.120:3300/0 conn(0x7fdfb006c300 0x7fdfb00ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.828+0000 7fdfa7e00640 1 -- 172.21.15.62:0/2850737731 >> v2:172.21.15.120:3300/0 conn(0x7fdfb006c300 msgr2=0x7fdfb00ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.828+0000 7fdfa7e00640 1 --2- 172.21.15.62:0/2850737731 >> v2:172.21.15.120:3300/0 conn(0x7fdfb006c300 0x7fdfb00ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.828+0000 7fdfa7e00640 1 -- 172.21.15.62:0/2850737731 >> v1:172.21.15.120:6789/0 conn(0x7fdfb006c6e0 legacy=0x7fdfb006cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.828+0000 7fdfa7e00640 1 -- 172.21.15.62:0/2850737731 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb00cc500 con 0x7fdfb00cafa0 2024-03-23T11:21:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.828+0000 7fdfade00640 1 --2- 172.21.15.62:0/2850737731 >> v2:172.21.15.120:3300/0 conn(0x7fdfb006c300 0x7fdfb00ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.829+0000 7fdfa7e00640 1 --2- 172.21.15.62:0/2850737731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb00cafa0 0x7fdfb00cb430 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdfb006c1b0 tx=0x7fdf9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212b98e5e5d84829 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:26.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.830+0000 7fdfad400640 1 -- 172.21.15.62:0/2850737731 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98031870 con 0x7fdfb00cafa0 2024-03-23T11:21:26.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.830+0000 7fdfad400640 1 -- 172.21.15.62:0/2850737731 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf980319d0 con 0x7fdfb00cafa0 2024-03-23T11:21:26.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.830+0000 7fdfad400640 1 -- 172.21.15.62:0/2850737731 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98031d60 con 0x7fdfb00cafa0 2024-03-23T11:21:26.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.830+0000 7fdfb4a00640 1 -- 172.21.15.62:0/2850737731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb00cafa0 msgr2=0x7fdfb00cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:26.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.830+0000 7fdfb4a00640 1 --2- 172.21.15.62:0/2850737731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb00cafa0 0x7fdfb00cb430 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdfb006c1b0 tx=0x7fdf9802b910 comp rx=0 tx=0).stop 2024-03-23T11:21:26.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.830+0000 7fdfb4a00640 1 -- 172.21.15.62:0/2850737731 shutdown_connections 2024-03-23T11:21:26.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.830+0000 7fdfb4a00640 1 --2- 172.21.15.62:0/2850737731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb00cafa0 0x7fdfb00cb430 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.830+0000 7fdfb4a00640 1 --2- 172.21.15.62:0/2850737731 >> v2:172.21.15.120:3300/0 conn(0x7fdfb006c300 0x7fdfb00ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.830+0000 7fdfb4a00640 1 -- 172.21.15.62:0/2850737731 >> 172.21.15.62:0/2850737731 conn(0x7fdfb00b02e0 msgr2=0x7fdfb00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:26.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.830+0000 7fdfb4a00640 1 -- 172.21.15.62:0/2850737731 shutdown_connections 2024-03-23T11:21:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.830+0000 7fdfb4a00640 1 -- 172.21.15.62:0/2850737731 wait complete. 2024-03-23T11:21:26.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.832+0000 7fdfb4a00640 1 Processor -- start 2024-03-23T11:21:26.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.832+0000 7fdfb4a00640 1 -- start start 2024-03-23T11:21:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.833+0000 7fdfb4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb006c300 0x7fdfb013ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.833+0000 7fdfb4a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfb006c6e0 0x7fdfb01efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.833+0000 7fdfb4a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfb00cafa0 0x7fdfb015af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.833+0000 7fdfb4a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb0154870 con 0x7fdfb006c300 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.833+0000 7fdfb4a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb0154e30 con 0x7fdfb00cafa0 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.833+0000 7fdfb4a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb01f08b0 con 0x7fdfb006c6e0 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.833+0000 7fdfade00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb006c300 0x7fdfb013ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.833+0000 7fdfade00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb006c300 0x7fdfb013ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46856/0 (socket says 172.21.15.62:46856) 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.833+0000 7fdfade00640 1 -- 172.21.15.62:0/1510479167 learned_addr learned my addr 172.21.15.62:0/1510479167 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.834+0000 7fdfade00640 1 -- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfb006c6e0 msgr2=0x7fdfb01efdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.834+0000 7fdfade00640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfb006c6e0 0x7fdfb01efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.834+0000 7fdfade00640 1 -- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfb00cafa0 msgr2=0x7fdfb015af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.834+0000 7fdfae800640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfb00cafa0 0x7fdfb015af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.834+0000 7fdfade00640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfb00cafa0 0x7fdfb015af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:26.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.834+0000 7fdfade00640 1 -- 172.21.15.62:0/1510479167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf980094d0 con 0x7fdfb006c300 2024-03-23T11:21:26.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.834+0000 7fdfae800640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfb00cafa0 0x7fdfb015af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:26.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.834+0000 7fdfade00640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb006c300 0x7fdfb013ab40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000c820 tx=0x7fdfa000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:26.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.835+0000 7fdfa6000640 1 -- 172.21.15.62:0/1510479167 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0007a30 con 0x7fdfb006c300 2024-03-23T11:21:26.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.835+0000 7fdfa6000640 1 -- 172.21.15.62:0/1510479167 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa0007b90 con 0x7fdfb006c300 2024-03-23T11:21:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.835+0000 7fdfa6000640 1 -- 172.21.15.62:0/1510479167 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa000ce70 con 0x7fdfb006c300 2024-03-23T11:21:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.835+0000 7fdfb4a00640 1 -- 172.21.15.62:0/1510479167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb01f0b90 con 0x7fdfb006c300 2024-03-23T11:21:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.835+0000 7fdfb4a00640 1 -- 172.21.15.62:0/1510479167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb013b080 con 0x7fdfb006c300 2024-03-23T11:21:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.836+0000 7fdfb4a00640 1 -- 172.21.15.62:0/1510479167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb0066c90 con 0x7fdfb006c300 2024-03-23T11:21:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.837+0000 7fdfa6000640 1 -- 172.21.15.62:0/1510479167 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fdfa00113f0 con 0x7fdfb006c300 2024-03-23T11:21:26.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.838+0000 7fdfa6000640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdf8c06a7a0 0x7fdf8c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:26.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.838+0000 7fdfa6000640 1 -- 172.21.15.62:0/1510479167 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdfa0088f60 con 0x7fdfb006c300 2024-03-23T11:21:26.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.839+0000 7fdfa7e00640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdf8c06a7a0 0x7fdf8c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:26.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.840+0000 7fdfa7e00640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdf8c06a7a0 0x7fdf8c06cc60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdf98036000 tx=0x7fdf98030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:26.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:26.843+0000 7fdfa6000640 1 -- 172.21.15.62:0/1510479167 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdfa0054370 con 0x7fdfb006c300 2024-03-23T11:21:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.085+0000 7fdfb4a00640 1 -- 172.21.15.62:0/1510479167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdfb00af400 con 0x7fdfb006c300 2024-03-23T11:21:27.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.706+0000 7fdfa6000640 1 -- 172.21.15.62:0/1510479167 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdfa001f0b0 con 0x7fdfb006c300 2024-03-23T11:21:27.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 -- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdf8c06a7a0 msgr2=0x7fdf8c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:27.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdf8c06a7a0 0x7fdf8c06cc60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdf98036000 tx=0x7fdf98030000 comp rx=0 tx=0).stop 2024-03-23T11:21:27.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 -- 172.21.15.62:0/1510479167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb006c300 msgr2=0x7fdfb013ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:27.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb006c300 0x7fdfb013ab40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000c820 tx=0x7fdfa000cc20 comp rx=0 tx=0).stop 2024-03-23T11:21:27.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 -- 172.21.15.62:0/1510479167 shutdown_connections 2024-03-23T11:21:27.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdf8c06a7a0 0x7fdf8c06cc60 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:27.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfb00cafa0 0x7fdfb015af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:27.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdfb006c6e0 0x7fdfb01efdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:27.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 --2- 172.21.15.62:0/1510479167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb006c300 0x7fdfb013ab40 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:27.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 -- 172.21.15.62:0/1510479167 >> 172.21.15.62:0/1510479167 conn(0x7fdfb00b02e0 msgr2=0x7fdfb013eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:27.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 -- 172.21.15.62:0/1510479167 shutdown_connections 2024-03-23T11:21:27.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.710+0000 7fdfb4a00640 1 -- 172.21.15.62:0/1510479167 wait complete. 2024-03-23T11:21:27.735 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T11:21:27.735 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T11:21:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.962+0000 7f71b3200640 1 Processor -- start 2024-03-23T11:21:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.962+0000 7f71b3200640 1 -- start start 2024-03-23T11:21:27.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.964+0000 7f71b3200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71b4143dd0 0x7f71b415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:27.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.964+0000 7f71b3200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71b4070090 0x7f71b40704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:27.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.964+0000 7f71b3200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b4070a30 con 0x7f71b40b2820 2024-03-23T11:21:27.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.964+0000 7f71b3200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b40cd140 con 0x7f71b4143dd0 2024-03-23T11:21:27.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.964+0000 7f71b3200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b40cd3e0 con 0x7f71b4070090 2024-03-23T11:21:27.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.964+0000 7f71b1e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71b4143dd0 0x7f71b415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:27.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.964+0000 7f71b2800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f71b40b2820 0x7f71b40b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59574/0 (socket says 172.21.15.62:59574) 2024-03-23T11:21:27.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.964+0000 7f71b2800640 1 -- 172.21.15.62:0/1441831151 learned_addr learned my addr 172.21.15.62:0/1441831151 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:27.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.965+0000 7f71b1400640 1 --2- 172.21.15.62:0/1441831151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71b4070090 0x7f71b40704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:27.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.965+0000 7f71b1e00640 1 -- 172.21.15.62:0/1441831151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71b4070090 msgr2=0x7f71b40704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:27.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.965+0000 7f71b1e00640 1 --2- 172.21.15.62:0/1441831151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71b4070090 0x7f71b40704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:27.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.965+0000 7f71b1e00640 1 -- 172.21.15.62:0/1441831151 >> v1:172.21.15.62:6789/0 conn(0x7f71b40b2820 legacy=0x7f71b40b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:27.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.965+0000 7f71b1e00640 1 -- 172.21.15.62:0/1441831151 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b40cd6c0 con 0x7f71b4143dd0 2024-03-23T11:21:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.966+0000 7f71b1e00640 1 --2- 172.21.15.62:0/1441831151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4143dd0 0x7f71b415abe0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f71a40097a0 tx=0x7f71a402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805654a3c64670f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.966+0000 7f71b0a00640 1 -- 172.21.15.62:0/1441831151 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a402d040 con 0x7f71b4143dd0 2024-03-23T11:21:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.966+0000 7f71b0a00640 1 -- 172.21.15.62:0/1441831151 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a4031810 con 0x7f71b4143dd0 2024-03-23T11:21:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.966+0000 7f71b0a00640 1 -- 172.21.15.62:0/1441831151 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4031ba0 con 0x7f71b4143dd0 2024-03-23T11:21:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.967+0000 7f71b3200640 1 -- 172.21.15.62:0/1441831151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4143dd0 msgr2=0x7f71b415abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.967+0000 7f71b3200640 1 --2- 172.21.15.62:0/1441831151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4143dd0 0x7f71b415abe0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f71a40097a0 tx=0x7f71a402b8b0 comp rx=0 tx=0).stop 2024-03-23T11:21:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.967+0000 7f71b3200640 1 -- 172.21.15.62:0/1441831151 shutdown_connections 2024-03-23T11:21:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.967+0000 7f71b3200640 1 --2- 172.21.15.62:0/1441831151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71b4070090 0x7f71b40704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.967+0000 7f71b3200640 1 --2- 172.21.15.62:0/1441831151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4143dd0 0x7f71b415abe0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.967+0000 7f71b3200640 1 -- 172.21.15.62:0/1441831151 >> 172.21.15.62:0/1441831151 conn(0x7f71b4066050 msgr2=0x7f71b4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.967+0000 7f71b3200640 1 -- 172.21.15.62:0/1441831151 shutdown_connections 2024-03-23T11:21:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.967+0000 7f71b3200640 1 -- 172.21.15.62:0/1441831151 wait complete. 2024-03-23T11:21:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.969+0000 7f71b3200640 1 Processor -- start 2024-03-23T11:21:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.969+0000 7f71b3200640 1 -- start start 2024-03-23T11:21:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.970+0000 7f71b3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4070090 0x7f71b40acce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.970+0000 7f71b3200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71b40b2820 0x7f71b413e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.970+0000 7f71b3200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71b4143dd0 0x7f71b413eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.970+0000 7f71b3200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b413f160 con 0x7f71b4070090 2024-03-23T11:21:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.970+0000 7f71b3200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b413f440 con 0x7f71b4143dd0 2024-03-23T11:21:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.970+0000 7f71b3200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b413f720 con 0x7f71b40b2820 2024-03-23T11:21:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.970+0000 7f71b1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4070090 0x7f71b40acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.970+0000 7f71b1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4070090 0x7f71b40acce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34548/0 (socket says 172.21.15.62:34548) 2024-03-23T11:21:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.970+0000 7f71b1e00640 1 -- 172.21.15.62:0/3439716281 learned_addr learned my addr 172.21.15.62:0/3439716281 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.971+0000 7f71b1e00640 1 -- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71b40b2820 msgr2=0x7f71b413e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.971+0000 7f71b1e00640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71b40b2820 0x7f71b413e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.971+0000 7f71b1e00640 1 -- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71b4143dd0 msgr2=0x7f71b413eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.971+0000 7f71b1e00640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71b4143dd0 0x7f71b413eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.971+0000 7f71b1e00640 1 -- 172.21.15.62:0/3439716281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a4009400 con 0x7f71b4070090 2024-03-23T11:21:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.971+0000 7f71b1e00640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4070090 0x7f71b40acce0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f71a4000c00 tx=0x7f71a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.971+0000 7f71aa000640 1 -- 172.21.15.62:0/3439716281 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a403ad60 con 0x7f71b4070090 2024-03-23T11:21:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.971+0000 7f71aa000640 1 -- 172.21.15.62:0/3439716281 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a40443f0 con 0x7f71b4070090 2024-03-23T11:21:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.971+0000 7f71b3200640 1 -- 172.21.15.62:0/3439716281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b413fa00 con 0x7f71b4070090 2024-03-23T11:21:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.971+0000 7f71aa000640 1 -- 172.21.15.62:0/3439716281 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a40447a0 con 0x7f71b4070090 2024-03-23T11:21:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.972+0000 7f71b3200640 1 -- 172.21.15.62:0/3439716281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b413c2f0 con 0x7f71b4070090 2024-03-23T11:21:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.973+0000 7f71b3200640 1 -- 172.21.15.62:0/3439716281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b413ff20 con 0x7f71b4070090 2024-03-23T11:21:27.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.974+0000 7f71aa000640 1 -- 172.21.15.62:0/3439716281 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f71a404d450 con 0x7f71b4070090 2024-03-23T11:21:27.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.974+0000 7f71aa000640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f719006a7a0 0x7f719006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:27.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.975+0000 7f71aa000640 1 -- 172.21.15.62:0/3439716281 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71a40ab0b0 con 0x7f71b4070090 2024-03-23T11:21:27.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.975+0000 7f71b1400640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f719006a7a0 0x7f719006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.977+0000 7f71b1400640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f719006a7a0 0x7f719006cc60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f71b415d850 tx=0x7f719c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:27.979+0000 7f71aa000640 1 -- 172.21.15.62:0/3439716281 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f71b413ff20 con 0x7f71b4070090 2024-03-23T11:21:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.216+0000 7f71b3200640 1 -- 172.21.15.62:0/3439716281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f71b413ff20 con 0x7f71b4070090 2024-03-23T11:21:28.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.717+0000 7f71aa000640 1 -- 172.21.15.62:0/3439716281 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7f71a4076550 con 0x7f71b4070090 2024-03-23T11:21:28.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.720+0000 7f71b3200640 1 -- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f719006a7a0 msgr2=0x7f719006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:28.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.720+0000 7f71b3200640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f719006a7a0 0x7f719006cc60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f71b415d850 tx=0x7f719c008040 comp rx=0 tx=0).stop 2024-03-23T11:21:28.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.720+0000 7f71b3200640 1 -- 172.21.15.62:0/3439716281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4070090 msgr2=0x7f71b40acce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:28.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.720+0000 7f71b3200640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4070090 0x7f71b40acce0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f71a4000c00 tx=0x7f71a40046c0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.720+0000 7f71b3200640 1 -- 172.21.15.62:0/3439716281 shutdown_connections 2024-03-23T11:21:28.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.720+0000 7f71b3200640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f719006a7a0 0x7f719006cc60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.720+0000 7f71b3200640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71b4143dd0 0x7f71b413eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.720+0000 7f71b3200640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71b40b2820 0x7f71b413e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.720+0000 7f71b3200640 1 --2- 172.21.15.62:0/3439716281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4070090 0x7f71b40acce0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.720+0000 7f71b3200640 1 -- 172.21.15.62:0/3439716281 >> 172.21.15.62:0/3439716281 conn(0x7f71b4066050 msgr2=0x7f71b40ae9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:28.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.720+0000 7f71b3200640 1 -- 172.21.15.62:0/3439716281 shutdown_connections 2024-03-23T11:21:28.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.721+0000 7f71b3200640 1 -- 172.21.15.62:0/3439716281 wait complete. 2024-03-23T11:21:28.742 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T11:21:28.742 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:21:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.968+0000 7f6e9f000640 1 Processor -- start 2024-03-23T11:21:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.968+0000 7f6e9f000640 1 -- start start 2024-03-23T11:21:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.970+0000 7f6e9f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e980cd7c0 0x7f6e980b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.970+0000 7f6e9f000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6e9806dca0 0x7f6e9806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.970+0000 7f6e9f000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e980cad20 con 0x7f6e980cdba0 2024-03-23T11:21:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.970+0000 7f6e9f000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9806e130 con 0x7f6e9806dca0 2024-03-23T11:21:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.970+0000 7f6e9f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9815a8a0 con 0x7f6e980cd7c0 2024-03-23T11:21:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.970+0000 7f6e9d000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6e980cdba0 0x7f6e980cb490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59580/0 (socket says 172.21.15.62:59580) 2024-03-23T11:21:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.970+0000 7f6e9d000640 1 -- 172.21.15.62:0/4215113616 learned_addr learned my addr 172.21.15.62:0/4215113616 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e97400640 1 --2- 172.21.15.62:0/4215113616 >> v2:172.21.15.120:3300/0 conn(0x7f6e9806dca0 0x7f6e9806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e97e00640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e980cd7c0 0x7f6e980b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488905696 0 0) 0x7f6e980cad20 con 0x7f6e980cdba0 2024-03-23T11:21:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e7c003520 con 0x7f6e980cdba0 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1895496037 0 0) 0x7f6e7c003520 con 0x7f6e980cdba0 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e980cd7c0 msgr2=0x7f6e980b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e980cd7c0 0x7f6e980b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 >> v2:172.21.15.120:3300/0 conn(0x7f6e9806dca0 msgr2=0x7f6e9806c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 --2- 172.21.15.62:0/4215113616 >> v2:172.21.15.120:3300/0 conn(0x7f6e9806dca0 0x7f6e9806c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9815ab80 con 0x7f6e980cdba0 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f6e900019f0 con 0x7f6e980cdba0 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 >> v1:172.21.15.62:6789/0 conn(0x7f6e980cdba0 legacy=0x7f6e980cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e7c003970 0x7f6e7c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.971+0000 7f6e96a00640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e7c0047c0 0x7f6e7c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e96a00640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e980cd7c0 0x7f6e7c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9806e130 con 0x7f6e7c003970 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9815a8a0 con 0x7f6e7c0047c0 2024-03-23T11:21:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c00dbe0 con 0x7f6e980cd7c0 2024-03-23T11:21:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e97e00640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e7c003970 0x7f6e7c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e97e00640 1 -- 172.21.15.62:0/4215113616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e980cd7c0 msgr2=0x7f6e7c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e97e00640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e980cd7c0 0x7f6e7c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e97e00640 1 -- 172.21.15.62:0/4215113616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e7c0047c0 msgr2=0x7f6e7c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e97400640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e7c0047c0 0x7f6e7c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e97e00640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e7c0047c0 0x7f6e7c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e97e00640 1 -- 172.21.15.62:0/4215113616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c003520 con 0x7f6e7c003970 2024-03-23T11:21:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.972+0000 7f6e97400640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e7c0047c0 0x7f6e7c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.973+0000 7f6e97e00640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e7c003970 0x7f6e7c003d90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009af0 tx=0x7f6e7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8009ce8e07b4fdf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.974+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8802c710 con 0x7f6e7c003970 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.974+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8802c870 con 0x7f6e7c003970 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.974+0000 7f6e96a00640 1 -- 172.21.15.62:0/4215113616 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8802cc00 con 0x7f6e7c003970 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.974+0000 7f6e9f000640 1 -- 172.21.15.62:0/4215113616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e7c003970 msgr2=0x7f6e7c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.974+0000 7f6e9f000640 1 --2- 172.21.15.62:0/4215113616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e7c003970 0x7f6e7c003d90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009af0 tx=0x7f6e7c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.974+0000 7f6e9d000640 1 -- 172.21.15.62:0/4215113616 reap_dead start 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.974+0000 7f6e9f000640 1 -- 172.21.15.62:0/4215113616 shutdown_connections 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.974+0000 7f6e9f000640 1 -- 172.21.15.62:0/4215113616 >> 172.21.15.62:0/4215113616 conn(0x7f6e980b02e0 msgr2=0x7f6e980b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.974+0000 7f6e9f000640 1 -- 172.21.15.62:0/4215113616 shutdown_connections 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.974+0000 7f6e9f000640 1 -- 172.21.15.62:0/4215113616 wait complete. 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.976+0000 7f6e9f000640 1 Processor -- start 2024-03-23T11:21:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.976+0000 7f6e9f000640 1 -- start start 2024-03-23T11:21:28.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.977+0000 7f6e9f000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e7c003970 0x7f6e98152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.977+0000 7f6e9f000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e98151950 0x7f6e98144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.977+0000 7f6e9f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e981e1ff0 0x7f6e98158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.977+0000 7f6e9f000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e981514c0 con 0x7f6e98151950 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.977+0000 7f6e9f000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e98150b90 con 0x7f6e7c003970 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.977+0000 7f6e9f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e98150e70 con 0x7f6e981e1ff0 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e97400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e98151950 0x7f6e98144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e97400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e98151950 0x7f6e98144f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34558/0 (socket says 172.21.15.62:34558) 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e97400640 1 -- 172.21.15.62:0/1228944184 learned_addr learned my addr 172.21.15.62:0/1228944184 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e97e00640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e7c003970 0x7f6e98152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e9d000640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e981e1ff0 0x7f6e98158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e97400640 1 -- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e981e1ff0 msgr2=0x7f6e98158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e97400640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e981e1ff0 0x7f6e98158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e97400640 1 -- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e7c003970 msgr2=0x7f6e98152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e97400640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e7c003970 0x7f6e98152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e97400640 1 -- 172.21.15.62:0/1228944184 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98151150 con 0x7f6e98151950 2024-03-23T11:21:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e97e00640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e7c003970 0x7f6e98152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e9d000640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e981e1ff0 0x7f6e98158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.978+0000 7f6e97400640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e98151950 0x7f6e98144f80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6e840109d0 tx=0x7f6e84010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.979+0000 7f6e8fe00640 1 -- 172.21.15.62:0/1228944184 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8400ec20 con 0x7f6e98151950 2024-03-23T11:21:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.979+0000 7f6e8fe00640 1 -- 172.21.15.62:0/1228944184 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8400ed80 con 0x7f6e98151950 2024-03-23T11:21:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.979+0000 7f6e9f000640 1 -- 172.21.15.62:0/1228944184 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e980b32f0 con 0x7f6e98151950 2024-03-23T11:21:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.979+0000 7f6e8fe00640 1 -- 172.21.15.62:0/1228944184 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8401c620 con 0x7f6e98151950 2024-03-23T11:21:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.979+0000 7f6e9f000640 1 -- 172.21.15.62:0/1228944184 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e98158d60 con 0x7f6e98151950 2024-03-23T11:21:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.980+0000 7f6e9f000640 1 -- 172.21.15.62:0/1228944184 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e98158fe0 con 0x7f6e98151950 2024-03-23T11:21:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.981+0000 7f6e8fe00640 1 -- 172.21.15.62:0/1228944184 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6e8401c780 con 0x7f6e98151950 2024-03-23T11:21:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.982+0000 7f6e8fe00640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6e7806a7a0 0x7f6e7806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.982+0000 7f6e8fe00640 1 -- 172.21.15.62:0/1228944184 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e84089410 con 0x7f6e98151950 2024-03-23T11:21:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.982+0000 7f6e97e00640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6e7806a7a0 0x7f6e7806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.983+0000 7f6e97e00640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6e7806a7a0 0x7f6e7806cc60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6e88036000 tx=0x7f6e88030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:28.986+0000 7f6e8fe00640 1 -- 172.21.15.62:0/1228944184 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6e84054840 con 0x7f6e98151950 2024-03-23T11:21:29.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.223+0000 7f6e9f000640 1 -- 172.21.15.62:0/1228944184 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e98066c90 con 0x7f6e98151950 2024-03-23T11:21:29.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.224+0000 7f6e8fe00640 1 -- 172.21.15.62:0/1228944184 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f6e84057e60 con 0x7f6e98151950 2024-03-23T11:21:29.224 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:29.224 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3787987198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3787987198},{"type":"v1","addr":"172.21.15.120:6835","nonce":3787987198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2749498408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2749498408},{"type":"v1","addr":"172.21.15.120:6837","nonce":2749498408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2325988870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2325988870},{"type":"v1","addr":"172.21.15.62:6833","nonce":2325988870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:21:25.687136+0000","modified":"2024-03-23T11:21:28.704192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6835/1159419089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1159419089},{"type":"v1","addr":"172.21.15.62:6835","nonce":1159419089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T11:21:29.224 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-03-23T11:21:29.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 -- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6e7806a7a0 msgr2=0x7f6e7806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6e7806a7a0 0x7f6e7806cc60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6e88036000 tx=0x7f6e88030000 comp rx=0 tx=0).stop 2024-03-23T11:21:29.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 -- 172.21.15.62:0/1228944184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e98151950 msgr2=0x7f6e98144f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e98151950 0x7f6e98144f80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6e840109d0 tx=0x7f6e84010e90 comp rx=0 tx=0).stop 2024-03-23T11:21:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 -- 172.21.15.62:0/1228944184 shutdown_connections 2024-03-23T11:21:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6e7806a7a0 0x7f6e7806cc60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e98151950 0x7f6e98144f80 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e981e1ff0 0x7f6e98158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 --2- 172.21.15.62:0/1228944184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e7c003970 0x7f6e98152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 -- 172.21.15.62:0/1228944184 >> 172.21.15.62:0/1228944184 conn(0x7f6e980b02e0 msgr2=0x7f6e980ae690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 -- 172.21.15.62:0/1228944184 shutdown_connections 2024-03-23T11:21:29.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.227+0000 7f6e9f000640 1 -- 172.21.15.62:0/1228944184 wait complete. 2024-03-23T11:21:29.245 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:21:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.465+0000 7f442e000640 1 Processor -- start 2024-03-23T11:21:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.465+0000 7f442e000640 1 -- start start 2024-03-23T11:21:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.467+0000 7f442e000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4428160240 0x7f442806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.467+0000 7f442e000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f44280989d0 0x7f4428098e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.467+0000 7f442e000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442806b2b0 con 0x7f44280995b0 2024-03-23T11:21:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.467+0000 7f442e000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44280cf490 con 0x7f44280989d0 2024-03-23T11:21:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.467+0000 7f442e000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44280993a0 con 0x7f4428160240 2024-03-23T11:21:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.467+0000 7f4427e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f44280989d0 0x7f4428098e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.467+0000 7f4427e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f44280989d0 0x7f4428098e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34570/0 (socket says 172.21.15.62:34570) 2024-03-23T11:21:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.467+0000 7f4427e00640 1 -- 172.21.15.62:0/2591062985 learned_addr learned my addr 172.21.15.62:0/2591062985 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.467+0000 7f4427400640 1 --2- 172.21.15.62:0/2591062985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4428160240 0x7f442806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.468+0000 7f4427e00640 1 -- 172.21.15.62:0/2591062985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4428160240 msgr2=0x7f442806ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.468+0000 7f4427e00640 1 --2- 172.21.15.62:0/2591062985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4428160240 0x7f442806ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.468+0000 7f4427e00640 1 -- 172.21.15.62:0/2591062985 >> v1:172.21.15.120:6789/0 conn(0x7f44280995b0 legacy=0x7f44280ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.468+0000 7f4427e00640 1 -- 172.21.15.62:0/2591062985 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44280cf690 con 0x7f44280989d0 2024-03-23T11:21:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.469+0000 7f4427e00640 1 --2- 172.21.15.62:0/2591062985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44280989d0 0x7f4428098e60 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f44200098a0 tx=0x7f442002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d90211f4cc92d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.469+0000 7f4426000640 1 -- 172.21.15.62:0/2591062985 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420031640 con 0x7f44280989d0 2024-03-23T11:21:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.469+0000 7f4426000640 1 -- 172.21.15.62:0/2591062985 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44200317a0 con 0x7f44280989d0 2024-03-23T11:21:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.469+0000 7f4426000640 1 -- 172.21.15.62:0/2591062985 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420031b30 con 0x7f44280989d0 2024-03-23T11:21:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.470+0000 7f442e000640 1 -- 172.21.15.62:0/2591062985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44280989d0 msgr2=0x7f4428098e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.470+0000 7f442e000640 1 --2- 172.21.15.62:0/2591062985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44280989d0 0x7f4428098e60 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f44200098a0 tx=0x7f442002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.470+0000 7f442e000640 1 -- 172.21.15.62:0/2591062985 shutdown_connections 2024-03-23T11:21:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.470+0000 7f442e000640 1 --2- 172.21.15.62:0/2591062985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44280989d0 0x7f4428098e60 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.470+0000 7f442e000640 1 --2- 172.21.15.62:0/2591062985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4428160240 0x7f442806ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.470+0000 7f442e000640 1 -- 172.21.15.62:0/2591062985 >> 172.21.15.62:0/2591062985 conn(0x7f44280b22c0 msgr2=0x7f44280b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.470+0000 7f442e000640 1 -- 172.21.15.62:0/2591062985 shutdown_connections 2024-03-23T11:21:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.470+0000 7f442e000640 1 -- 172.21.15.62:0/2591062985 wait complete. 2024-03-23T11:21:29.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.472+0000 7f442e000640 1 Processor -- start 2024-03-23T11:21:29.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.472+0000 7f442e000640 1 -- start start 2024-03-23T11:21:29.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.473+0000 7f442e000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44280989d0 0x7f4428150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.473+0000 7f442e000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44280995b0 0x7f44280c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.473+0000 7f442e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4428160240 0x7f44280c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.473+0000 7f442e000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44280c6b10 con 0x7f4428160240 2024-03-23T11:21:29.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.473+0000 7f442e000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44280c5e90 con 0x7f44280989d0 2024-03-23T11:21:29.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.473+0000 7f442e000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44280c6170 con 0x7f44280995b0 2024-03-23T11:21:29.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.473+0000 7f4427e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4428160240 0x7f44280c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:29.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.473+0000 7f4427e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4428160240 0x7f44280c7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34584/0 (socket says 172.21.15.62:34584) 2024-03-23T11:21:29.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.473+0000 7f4427e00640 1 -- 172.21.15.62:0/2895473487 learned_addr learned my addr 172.21.15.62:0/2895473487 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:29.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f4427400640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44280989d0 0x7f4428150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f4426a00640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44280995b0 0x7f44280c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f4427e00640 1 -- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44280995b0 msgr2=0x7f44280c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f4427e00640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44280995b0 0x7f44280c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f4427e00640 1 -- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44280989d0 msgr2=0x7f4428150940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f4427e00640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44280989d0 0x7f4428150940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f4427e00640 1 -- 172.21.15.62:0/2895473487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44200094d0 con 0x7f4428160240 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f4427400640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44280989d0 0x7f4428150940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f4426a00640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44280995b0 0x7f44280c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f4427e00640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4428160240 0x7f44280c7de0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f44200098a0 tx=0x7f44200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f441f400640 1 -- 172.21.15.62:0/2895473487 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442003ac60 con 0x7f4428160240 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f441f400640 1 -- 172.21.15.62:0/2895473487 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442003adc0 con 0x7f4428160240 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f441f400640 1 -- 172.21.15.62:0/2895473487 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420044610 con 0x7f4428160240 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.474+0000 7f442e000640 1 -- 172.21.15.62:0/2895473487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44280c6450 con 0x7f4428160240 2024-03-23T11:21:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.475+0000 7f442e000640 1 -- 172.21.15.62:0/2895473487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4428151100 con 0x7f4428160240 2024-03-23T11:21:29.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.476+0000 7f442e000640 1 -- 172.21.15.62:0/2895473487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44280c6c50 con 0x7f4428160240 2024-03-23T11:21:29.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.477+0000 7f441f400640 1 -- 172.21.15.62:0/2895473487 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f442002f050 con 0x7f4428160240 2024-03-23T11:21:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.477+0000 7f441f400640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440406a7a0 0x7f440406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.478+0000 7f441f400640 1 -- 172.21.15.62:0/2895473487 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44200aade0 con 0x7f4428160240 2024-03-23T11:21:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.478+0000 7f4427400640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440406a7a0 0x7f440406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:29.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.479+0000 7f4427400640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440406a7a0 0x7f440406cc60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4418009f50 tx=0x7f441800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:29.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.482+0000 7f441f400640 1 -- 172.21.15.62:0/2895473487 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f44280c6c50 con 0x7f4428160240 2024-03-23T11:21:29.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.681+0000 7f442e000640 1 -- 172.21.15.62:0/2895473487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44280c6c50 con 0x7f4428160240 2024-03-23T11:21:29.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.683+0000 7f441f400640 1 -- 172.21.15.62:0/2895473487 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16045 (secure 0 0 0) 0x7f44280c6c50 con 0x7f4428160240 2024-03-23T11:21:29.683 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:29.683 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":48,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:21:26.691298+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:21:23.756222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-23T11:21:24.715577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:21:29.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.685+0000 7f442e000640 1 -- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440406a7a0 msgr2=0x7f440406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.685+0000 7f442e000640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440406a7a0 0x7f440406cc60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4418009f50 tx=0x7f441800b040 comp rx=0 tx=0).stop 2024-03-23T11:21:29.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.685+0000 7f442e000640 1 -- 172.21.15.62:0/2895473487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4428160240 msgr2=0x7f44280c7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.685+0000 7f442e000640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4428160240 0x7f44280c7de0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f44200098a0 tx=0x7f44200046c0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.686+0000 7f442e000640 1 -- 172.21.15.62:0/2895473487 shutdown_connections 2024-03-23T11:21:29.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.686+0000 7f442e000640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440406a7a0 0x7f440406cc60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.686+0000 7f442e000640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4428160240 0x7f44280c7de0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.686+0000 7f442e000640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f44280995b0 0x7f44280c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.686+0000 7f442e000640 1 --2- 172.21.15.62:0/2895473487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44280989d0 0x7f4428150940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.686+0000 7f442e000640 1 -- 172.21.15.62:0/2895473487 >> 172.21.15.62:0/2895473487 conn(0x7f44280b22c0 msgr2=0x7f44280cd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:29.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.686+0000 7f442e000640 1 -- 172.21.15.62:0/2895473487 shutdown_connections 2024-03-23T11:21:29.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.686+0000 7f442e000640 1 -- 172.21.15.62:0/2895473487 wait complete. 2024-03-23T11:21:29.705 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T11:21:29.705 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:21:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.926+0000 7f39b2e00640 1 Processor -- start 2024-03-23T11:21:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.926+0000 7f39b2e00640 1 -- start start 2024-03-23T11:21:29.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.928+0000 7f39b2e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f39ac143dd0 0x7f39ac15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.928+0000 7f39b2e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39ac070090 0x7f39ac0704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.928+0000 7f39b2e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ac070a30 con 0x7f39ac0b2820 2024-03-23T11:21:29.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.928+0000 7f39b2e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39ac0cd140 con 0x7f39ac143dd0 2024-03-23T11:21:29.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.928+0000 7f39b2e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac0cd3e0 con 0x7f39ac070090 2024-03-23T11:21:29.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.928+0000 7f39b2400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f39ac0b2820 0x7f39ac0b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59590/0 (socket says 172.21.15.62:59590) 2024-03-23T11:21:29.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.928+0000 7f39b2400640 1 -- 172.21.15.62:0/1285516128 learned_addr learned my addr 172.21.15.62:0/1285516128 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39b1a00640 1 --2- 172.21.15.62:0/1285516128 >> v2:172.21.15.120:3300/0 conn(0x7f39ac143dd0 0x7f39ac15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39b1000640 1 --2- 172.21.15.62:0/1285516128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39ac070090 0x7f39ac0704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654923742 0 0) 0x7f39ac070a30 con 0x7f39ac0b2820 2024-03-23T11:21:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3994003520 con 0x7f39ac0b2820 2024-03-23T11:21:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1360505678 0 0) 0x7f3994003520 con 0x7f39ac0b2820 2024-03-23T11:21:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39ac070090 msgr2=0x7f39ac0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 --2- 172.21.15.62:0/1285516128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39ac070090 0x7f39ac0704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 >> v2:172.21.15.120:3300/0 conn(0x7f39ac143dd0 msgr2=0x7f39ac15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 --2- 172.21.15.62:0/1285516128 >> v2:172.21.15.120:3300/0 conn(0x7f39ac143dd0 0x7f39ac15abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac0cd6c0 con 0x7f39ac0b2820 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f39a40019f0 con 0x7f39ac0b2820 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 >> v1:172.21.15.62:6789/0 conn(0x7f39ac0b2820 legacy=0x7f39ac0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 --2- 172.21.15.62:0/1285516128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3994003970 0x7f3994003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 --2- 172.21.15.62:0/1285516128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39940047c0 0x7f3994004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.929+0000 7f39abe00640 1 --2- 172.21.15.62:0/1285516128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39ac070090 0x7f3994005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.930+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac0cd140 con 0x7f3994003970 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.930+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac0cd3e0 con 0x7f39940047c0 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.930+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399400b980 con 0x7f39ac070090 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.930+0000 7f39b1a00640 1 --2- 172.21.15.62:0/1285516128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3994003970 0x7f3994003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.930+0000 7f39b1a00640 1 -- 172.21.15.62:0/1285516128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39ac070090 msgr2=0x7f3994005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.930+0000 7f39b1a00640 1 --2- 172.21.15.62:0/1285516128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f39ac070090 0x7f3994005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.930+0000 7f39b1a00640 1 -- 172.21.15.62:0/1285516128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39940047c0 msgr2=0x7f3994004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.930+0000 7f39b1a00640 1 --2- 172.21.15.62:0/1285516128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39940047c0 0x7f3994004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.930+0000 7f39b1a00640 1 -- 172.21.15.62:0/1285516128 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3994003520 con 0x7f3994003970 2024-03-23T11:21:29.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.931+0000 7f39b1a00640 1 --2- 172.21.15.62:0/1285516128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3994003970 0x7f3994003d90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009ac0 tx=0x7f39940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a0e3df4d205290 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:29.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.932+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a002c710 con 0x7f3994003970 2024-03-23T11:21:29.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.932+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a002c870 con 0x7f3994003970 2024-03-23T11:21:29.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.932+0000 7f39abe00640 1 -- 172.21.15.62:0/1285516128 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a002cc00 con 0x7f3994003970 2024-03-23T11:21:29.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.932+0000 7f39b2e00640 1 -- 172.21.15.62:0/1285516128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3994003970 msgr2=0x7f3994003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.932+0000 7f39b2e00640 1 --2- 172.21.15.62:0/1285516128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3994003970 0x7f3994003d90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009ac0 tx=0x7f39940057f0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.932+0000 7f39b2400640 1 -- 172.21.15.62:0/1285516128 reap_dead start 2024-03-23T11:21:29.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.932+0000 7f39b2e00640 1 -- 172.21.15.62:0/1285516128 shutdown_connections 2024-03-23T11:21:29.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.933+0000 7f39b2e00640 1 -- 172.21.15.62:0/1285516128 >> 172.21.15.62:0/1285516128 conn(0x7f39ac066050 msgr2=0x7f39ac069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:29.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.933+0000 7f39b2e00640 1 -- 172.21.15.62:0/1285516128 shutdown_connections 2024-03-23T11:21:29.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.933+0000 7f39b2e00640 1 -- 172.21.15.62:0/1285516128 wait complete. 2024-03-23T11:21:29.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.934+0000 7f39b2e00640 1 Processor -- start 2024-03-23T11:21:29.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.935+0000 7f39b2e00640 1 -- start start 2024-03-23T11:21:29.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.935+0000 7f39b2e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3994003970 0x7f39ac0a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39ac1f0970 0x7f39ac1ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39ac0ac3d0 0x7f39ac0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac1f1090 con 0x7f39ac0ac3d0 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac0acd90 con 0x7f39ac1f0970 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac1f0df0 con 0x7f3994003970 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39ac0ac3d0 0x7f39ac0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39ac0ac3d0 0x7f39ac0ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34608/0 (socket says 172.21.15.62:34608) 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2400640 1 -- 172.21.15.62:0/2853460863 learned_addr learned my addr 172.21.15.62:0/2853460863 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2400640 1 -- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3994003970 msgr2=0x7f39ac0a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2400640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3994003970 0x7f39ac0a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2400640 1 -- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39ac1f0970 msgr2=0x7f39ac1ef5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2400640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39ac1f0970 0x7f39ac1ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:29.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.936+0000 7f39b2400640 1 -- 172.21.15.62:0/2853460863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac0a8500 con 0x7f39ac0ac3d0 2024-03-23T11:21:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.937+0000 7f39b2400640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39ac0ac3d0 0x7f39ac0ac850 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f39a40013d0 tx=0x7f39a400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.937+0000 7f39a9600640 1 -- 172.21.15.62:0/2853460863 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a400a610 con 0x7f39ac0ac3d0 2024-03-23T11:21:29.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.937+0000 7f39a9600640 1 -- 172.21.15.62:0/2853460863 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a400a790 con 0x7f39ac0ac3d0 2024-03-23T11:21:29.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.937+0000 7f39a9600640 1 -- 172.21.15.62:0/2853460863 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a400ab40 con 0x7f39ac0ac3d0 2024-03-23T11:21:29.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.937+0000 7f39b2e00640 1 -- 172.21.15.62:0/2853460863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ac0a87e0 con 0x7f39ac0ac3d0 2024-03-23T11:21:29.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.937+0000 7f39b2e00640 1 -- 172.21.15.62:0/2853460863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ac0a8da0 con 0x7f39ac0ac3d0 2024-03-23T11:21:29.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.938+0000 7f39b2e00640 1 -- 172.21.15.62:0/2853460863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ac0cbec0 con 0x7f39ac0ac3d0 2024-03-23T11:21:29.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.939+0000 7f39a9600640 1 -- 172.21.15.62:0/2853460863 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f39a400aca0 con 0x7f39ac0ac3d0 2024-03-23T11:21:29.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.940+0000 7f39a9600640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f398c06a7d0 0x7f398c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:29.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.940+0000 7f39a9600640 1 -- 172.21.15.62:0/2853460863 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39a4088630 con 0x7f39ac0ac3d0 2024-03-23T11:21:29.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.941+0000 7f39b1a00640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f398c06a7d0 0x7f398c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:29.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.942+0000 7f39b1a00640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f398c06a7d0 0x7f398c06cc90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f39a0036000 tx=0x7f39a002e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:29.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:29.945+0000 7f39a9600640 1 -- 172.21.15.62:0/2853460863 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39a40539b0 con 0x7f39ac0ac3d0 2024-03-23T11:21:30.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.137+0000 7f39b2e00640 1 -- 172.21.15.62:0/2853460863 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39ac0cdf60 con 0x7f398c06a7d0 2024-03-23T11:21:30.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.161+0000 7f39a9600640 1 -- 172.21.15.62:0/2853460863 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286801 (secure 0 0 0) 0x7f39a002c9e0 con 0x7f398c06a7d0 2024-03-23T11:21:30.165 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:30.167 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.167+0000 7f39b2e00640 1 -- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f398c06a7d0 msgr2=0x7f398c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.167+0000 7f39b2e00640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f398c06a7d0 0x7f398c06cc90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f39a0036000 tx=0x7f39a002e000 comp rx=0 tx=0).stop 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.167+0000 7f39b2e00640 1 -- 172.21.15.62:0/2853460863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39ac0ac3d0 msgr2=0x7f39ac0ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.167+0000 7f39b2e00640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39ac0ac3d0 0x7f39ac0ac850 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f39a40013d0 tx=0x7f39a400cf00 comp rx=0 tx=0).stop 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.168+0000 7f39b2e00640 1 -- 172.21.15.62:0/2853460863 shutdown_connections 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.168+0000 7f39b2e00640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f398c06a7d0 0x7f398c06cc90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.168+0000 7f39b2e00640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39ac0ac3d0 0x7f39ac0ac850 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.168+0000 7f39b2e00640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39ac1f0970 0x7f39ac1ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.168+0000 7f39b2e00640 1 --2- 172.21.15.62:0/2853460863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3994003970 0x7f39ac0a7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.168+0000 7f39b2e00640 1 -- 172.21.15.62:0/2853460863 >> 172.21.15.62:0/2853460863 conn(0x7f39ac066050 msgr2=0x7f39ac069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.168+0000 7f39b2e00640 1 -- 172.21.15.62:0/2853460863 shutdown_connections 2024-03-23T11:21:30.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.168+0000 7f39b2e00640 1 -- 172.21.15.62:0/2853460863 wait complete. 2024-03-23T11:21:30.188 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-03-23T11:21:29.284640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":229968,"kb_used_data":5596,"kb_used_omap":13,"kb_used_meta":224242,"kb_avail":754744752,"statfs":{"total":773094113280,"available":772858626048,"internally_reserved":0,"allocated":5730304,"data_stored":3009741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":229623896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.084970"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708444+0000","last_change":"2024-03-23T11:21:25.724317+0000","last_active":"2024-03-23T11:21:26.708444+0000","last_peered":"2024-03-23T11:21:26.708444+0000","last_clean":"2024-03-23T11:21:26.708444+0000","last_became_active":"2024-03-23T11:21:25.723999+0000","last_became_peered":"2024-03-23T11:21:25.723999+0000","last_unstale":"2024-03-23T11:21:26.708444+0000","last_undegraded":"2024-03-23T11:21:26.708444+0000","last_fullsized":"2024-03-23T11:21:26.708444+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:20:38.705711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153295+0000","last_change":"2024-03-23T11:21:25.736776+0000","last_active":"2024-03-23T11:21:27.153295+0000","last_peered":"2024-03-23T11:21:27.153295+0000","last_clean":"2024-03-23T11:21:27.153295+0000","last_became_active":"2024-03-23T11:21:25.736444+0000","last_became_peered":"2024-03-23T11:21:25.736444+0000","last_unstale":"2024-03-23T11:21:27.153295+0000","last_undegraded":"2024-03-23T11:21:27.153295+0000","last_fullsized":"2024-03-23T11:21:27.153295+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:10:58.542179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708959+0000","last_change":"2024-03-23T11:21:25.713672+0000","last_active":"2024-03-23T11:21:26.708959+0000","last_peered":"2024-03-23T11:21:26.708959+0000","last_clean":"2024-03-23T11:21:26.708959+0000","last_became_active":"2024-03-23T11:21:25.713239+0000","last_became_peered":"2024-03-23T11:21:25.713239+0000","last_unstale":"2024-03-23T11:21:26.708959+0000","last_undegraded":"2024-03-23T11:21:26.708959+0000","last_fullsized":"2024-03-23T11:21:26.708959+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:59.704598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414281+0000","last_change":"2024-03-23T11:21:25.732972+0000","last_active":"2024-03-23T11:21:27.414281+0000","last_peered":"2024-03-23T11:21:27.414281+0000","last_clean":"2024-03-23T11:21:27.414281+0000","last_became_active":"2024-03-23T11:21:25.732767+0000","last_became_peered":"2024-03-23T11:21:25.732767+0000","last_unstale":"2024-03-23T11:21:27.414281+0000","last_undegraded":"2024-03-23T11:21:27.414281+0000","last_fullsized":"2024-03-23T11:21:27.414281+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:26:11.643044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153464+0000","last_change":"2024-03-23T11:21:25.738428+0000","last_active":"2024-03-23T11:21:27.153464+0000","last_peered":"2024-03-23T11:21:27.153464+0000","last_clean":"2024-03-23T11:21:27.153464+0000","last_became_active":"2024-03-23T11:21:25.738122+0000","last_became_peered":"2024-03-23T11:21:25.738122+0000","last_unstale":"2024-03-23T11:21:27.153464+0000","last_undegraded":"2024-03-23T11:21:27.153464+0000","last_fullsized":"2024-03-23T11:21:27.153464+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:45:22.587347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705323+0000","last_change":"2024-03-23T11:21:25.719207+0000","last_active":"2024-03-23T11:21:26.705323+0000","last_peered":"2024-03-23T11:21:26.705323+0000","last_clean":"2024-03-23T11:21:26.705323+0000","last_became_active":"2024-03-23T11:21:25.718962+0000","last_became_peered":"2024-03-23T11:21:25.718962+0000","last_unstale":"2024-03-23T11:21:26.705323+0000","last_undegraded":"2024-03-23T11:21:26.705323+0000","last_fullsized":"2024-03-23T11:21:26.705323+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:34:17.443990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.707497+0000","last_change":"2024-03-23T11:21:25.710267+0000","last_active":"2024-03-23T11:21:26.707497+0000","last_peered":"2024-03-23T11:21:26.707497+0000","last_clean":"2024-03-23T11:21:26.707497+0000","last_became_active":"2024-03-23T11:21:25.709808+0000","last_became_peered":"2024-03-23T11:21:25.709808+0000","last_unstale":"2024-03-23T11:21:26.707497+0000","last_undegraded":"2024-03-23T11:21:26.707497+0000","last_fullsized":"2024-03-23T11:21:26.707497+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:49.587791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412250+0000","last_change":"2024-03-23T11:21:25.729487+0000","last_active":"2024-03-23T11:21:27.412250+0000","last_peered":"2024-03-23T11:21:27.412250+0000","last_clean":"2024-03-23T11:21:27.412250+0000","last_became_active":"2024-03-23T11:21:25.729226+0000","last_became_peered":"2024-03-23T11:21:25.729226+0000","last_unstale":"2024-03-23T11:21:27.412250+0000","last_undegraded":"2024-03-23T11:21:27.412250+0000","last_fullsized":"2024-03-23T11:21:27.412250+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:38:40.371641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415104+0000","last_change":"2024-03-23T11:21:25.716142+0000","last_active":"2024-03-23T11:21:27.415104+0000","last_peered":"2024-03-23T11:21:27.415104+0000","last_clean":"2024-03-23T11:21:27.415104+0000","last_became_active":"2024-03-23T11:21:25.715891+0000","last_became_peered":"2024-03-23T11:21:25.715891+0000","last_unstale":"2024-03-23T11:21:27.415104+0000","last_undegraded":"2024-03-23T11:21:27.415104+0000","last_fullsized":"2024-03-23T11:21:27.415104+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:58.142972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415180+0000","last_change":"2024-03-23T11:21:25.729941+0000","last_active":"2024-03-23T11:21:27.415180+0000","last_peered":"2024-03-23T11:21:27.415180+0000","last_clean":"2024-03-23T11:21:27.415180+0000","last_became_active":"2024-03-23T11:21:25.729669+0000","last_became_peered":"2024-03-23T11:21:25.729669+0000","last_unstale":"2024-03-23T11:21:27.415180+0000","last_undegraded":"2024-03-23T11:21:27.415180+0000","last_fullsized":"2024-03-23T11:21:27.415180+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:01.037491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413244+0000","last_change":"2024-03-23T11:21:25.715370+0000","last_active":"2024-03-23T11:21:27.413244+0000","last_peered":"2024-03-23T11:21:27.413244+0000","last_clean":"2024-03-23T11:21:27.413244+0000","last_became_active":"2024-03-23T11:21:25.715141+0000","last_became_peered":"2024-03-23T11:21:25.715141+0000","last_unstale":"2024-03-23T11:21:27.413244+0000","last_undegraded":"2024-03-23T11:21:27.413244+0000","last_fullsized":"2024-03-23T11:21:27.413244+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:14.281844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153271+0000","last_change":"2024-03-23T11:21:25.734475+0000","last_active":"2024-03-23T11:21:27.153271+0000","last_peered":"2024-03-23T11:21:27.153271+0000","last_clean":"2024-03-23T11:21:27.153271+0000","last_became_active":"2024-03-23T11:21:25.734082+0000","last_became_peered":"2024-03-23T11:21:25.734082+0000","last_unstale":"2024-03-23T11:21:27.153271+0000","last_undegraded":"2024-03-23T11:21:27.153271+0000","last_fullsized":"2024-03-23T11:21:27.153271+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:01.037491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415224+0000","last_change":"2024-03-23T11:21:25.714553+0000","last_active":"2024-03-23T11:21:27.415224+0000","last_peered":"2024-03-23T11:21:27.415224+0000","last_clean":"2024-03-23T11:21:27.415224+0000","last_became_active":"2024-03-23T11:21:25.710516+0000","last_became_peered":"2024-03-23T11:21:25.710516+0000","last_unstale":"2024-03-23T11:21:27.415224+0000","last_undegraded":"2024-03-23T11:21:27.415224+0000","last_fullsized":"2024-03-23T11:21:27.415224+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:14:33.578227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413636+0000","last_change":"2024-03-23T11:21:25.714172+0000","last_active":"2024-03-23T11:21:27.413636+0000","last_peered":"2024-03-23T11:21:27.413636+0000","last_clean":"2024-03-23T11:21:27.413636+0000","last_became_active":"2024-03-23T11:21:25.713924+0000","last_became_peered":"2024-03-23T11:21:25.713924+0000","last_unstale":"2024-03-23T11:21:27.413636+0000","last_undegraded":"2024-03-23T11:21:27.413636+0000","last_fullsized":"2024-03-23T11:21:27.413636+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:15.957774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705165+0000","last_change":"2024-03-23T11:21:25.720955+0000","last_active":"2024-03-23T11:21:26.705165+0000","last_peered":"2024-03-23T11:21:26.705165+0000","last_clean":"2024-03-23T11:21:26.705165+0000","last_became_active":"2024-03-23T11:21:25.720693+0000","last_became_peered":"2024-03-23T11:21:25.720693+0000","last_unstale":"2024-03-23T11:21:26.705165+0000","last_undegraded":"2024-03-23T11:21:26.705165+0000","last_fullsized":"2024-03-23T11:21:26.705165+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:21:29.280006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411479+0000","last_change":"2024-03-23T11:21:25.738494+0000","last_active":"2024-03-23T11:21:27.411479+0000","last_peered":"2024-03-23T11:21:27.411479+0000","last_clean":"2024-03-23T11:21:27.411479+0000","last_became_active":"2024-03-23T11:21:25.738073+0000","last_became_peered":"2024-03-23T11:21:25.738073+0000","last_unstale":"2024-03-23T11:21:27.411479+0000","last_undegraded":"2024-03-23T11:21:27.411479+0000","last_fullsized":"2024-03-23T11:21:27.411479+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:14.281844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709662+0000","last_change":"2024-03-23T11:21:25.734121+0000","last_active":"2024-03-23T11:21:26.709662+0000","last_peered":"2024-03-23T11:21:26.709662+0000","last_clean":"2024-03-23T11:21:26.709662+0000","last_became_active":"2024-03-23T11:21:25.733850+0000","last_became_peered":"2024-03-23T11:21:25.733850+0000","last_unstale":"2024-03-23T11:21:26.709662+0000","last_undegraded":"2024-03-23T11:21:26.709662+0000","last_fullsized":"2024-03-23T11:21:26.709662+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:14.281844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708939+0000","last_change":"2024-03-23T11:21:25.717626+0000","last_active":"2024-03-23T11:21:26.708939+0000","last_peered":"2024-03-23T11:21:26.708939+0000","last_clean":"2024-03-23T11:21:26.708939+0000","last_became_active":"2024-03-23T11:21:25.717163+0000","last_became_peered":"2024-03-23T11:21:25.717163+0000","last_unstale":"2024-03-23T11:21:26.708939+0000","last_undegraded":"2024-03-23T11:21:26.708939+0000","last_fullsized":"2024-03-23T11:21:26.708939+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:14.264816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412300+0000","last_change":"2024-03-23T11:21:25.737723+0000","last_active":"2024-03-23T11:21:27.412300+0000","last_peered":"2024-03-23T11:21:27.412300+0000","last_clean":"2024-03-23T11:21:27.412300+0000","last_became_active":"2024-03-23T11:21:25.737232+0000","last_became_peered":"2024-03-23T11:21:25.737232+0000","last_unstale":"2024-03-23T11:21:27.412300+0000","last_undegraded":"2024-03-23T11:21:27.412300+0000","last_fullsized":"2024-03-23T11:21:27.412300+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:59.704598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415692+0000","last_change":"2024-03-23T11:21:25.722849+0000","last_active":"2024-03-23T11:21:27.415692+0000","last_peered":"2024-03-23T11:21:27.415692+0000","last_clean":"2024-03-23T11:21:27.415692+0000","last_became_active":"2024-03-23T11:21:25.721770+0000","last_became_peered":"2024-03-23T11:21:25.721770+0000","last_unstale":"2024-03-23T11:21:27.415692+0000","last_undegraded":"2024-03-23T11:21:27.415692+0000","last_fullsized":"2024-03-23T11:21:27.415692+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:22:45.890443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705345+0000","last_change":"2024-03-23T11:21:25.726876+0000","last_active":"2024-03-23T11:21:26.705345+0000","last_peered":"2024-03-23T11:21:26.705345+0000","last_clean":"2024-03-23T11:21:26.705345+0000","last_became_active":"2024-03-23T11:21:25.726491+0000","last_became_peered":"2024-03-23T11:21:25.726491+0000","last_unstale":"2024-03-23T11:21:26.705345+0000","last_undegraded":"2024-03-23T11:21:26.705345+0000","last_fullsized":"2024-03-23T11:21:26.705345+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:53:32.675975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413747+0000","last_change":"2024-03-23T11:21:25.729488+0000","last_active":"2024-03-23T11:21:27.413747+0000","last_peered":"2024-03-23T11:21:27.413747+0000","last_clean":"2024-03-23T11:21:27.413747+0000","last_became_active":"2024-03-23T11:21:25.729224+0000","last_became_peered":"2024-03-23T11:21:25.729224+0000","last_unstale":"2024-03-23T11:21:27.413747+0000","last_undegraded":"2024-03-23T11:21:27.413747+0000","last_fullsized":"2024-03-23T11:21:27.413747+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:46:30.086218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411714+0000","last_change":"2024-03-23T11:21:25.734805+0000","last_active":"2024-03-23T11:21:27.411714+0000","last_peered":"2024-03-23T11:21:27.411714+0000","last_clean":"2024-03-23T11:21:27.411714+0000","last_became_active":"2024-03-23T11:21:25.734522+0000","last_became_peered":"2024-03-23T11:21:25.734522+0000","last_unstale":"2024-03-23T11:21:27.411714+0000","last_undegraded":"2024-03-23T11:21:27.411714+0000","last_fullsized":"2024-03-23T11:21:27.411714+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:14.264816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411481+0000","last_change":"2024-03-23T11:21:25.734712+0000","last_active":"2024-03-23T11:21:27.411481+0000","last_peered":"2024-03-23T11:21:27.411481+0000","last_clean":"2024-03-23T11:21:27.411481+0000","last_became_active":"2024-03-23T11:21:25.734428+0000","last_became_peered":"2024-03-23T11:21:25.734428+0000","last_unstale":"2024-03-23T11:21:27.411481+0000","last_undegraded":"2024-03-23T11:21:27.411481+0000","last_fullsized":"2024-03-23T11:21:27.411481+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:30:28.372767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412339+0000","last_change":"2024-03-23T11:21:25.728371+0000","last_active":"2024-03-23T11:21:27.412339+0000","last_peered":"2024-03-23T11:21:27.412339+0000","last_clean":"2024-03-23T11:21:27.412339+0000","last_became_active":"2024-03-23T11:21:25.728086+0000","last_became_peered":"2024-03-23T11:21:25.728086+0000","last_unstale":"2024-03-23T11:21:27.412339+0000","last_undegraded":"2024-03-23T11:21:27.412339+0000","last_fullsized":"2024-03-23T11:21:27.412339+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:44.633633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415697+0000","last_change":"2024-03-23T11:21:25.710597+0000","last_active":"2024-03-23T11:21:27.415697+0000","last_peered":"2024-03-23T11:21:27.415697+0000","last_clean":"2024-03-23T11:21:27.415697+0000","last_became_active":"2024-03-23T11:21:25.708337+0000","last_became_peered":"2024-03-23T11:21:25.708337+0000","last_unstale":"2024-03-23T11:21:27.415697+0000","last_undegraded":"2024-03-23T11:21:27.415697+0000","last_fullsized":"2024-03-23T11:21:27.415697+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:51:50.518844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709091+0000","last_change":"2024-03-23T11:21:25.735447+0000","last_active":"2024-03-23T11:21:26.709091+0000","last_peered":"2024-03-23T11:21:26.709091+0000","last_clean":"2024-03-23T11:21:26.709091+0000","last_became_active":"2024-03-23T11:21:25.735137+0000","last_became_peered":"2024-03-23T11:21:25.735137+0000","last_unstale":"2024-03-23T11:21:26.709091+0000","last_undegraded":"2024-03-23T11:21:26.709091+0000","last_fullsized":"2024-03-23T11:21:26.709091+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.172566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153538+0000","last_change":"2024-03-23T11:21:25.735411+0000","last_active":"2024-03-23T11:21:27.153538+0000","last_peered":"2024-03-23T11:21:27.153538+0000","last_clean":"2024-03-23T11:21:27.153538+0000","last_became_active":"2024-03-23T11:21:25.733746+0000","last_became_peered":"2024-03-23T11:21:25.733746+0000","last_unstale":"2024-03-23T11:21:27.153538+0000","last_undegraded":"2024-03-23T11:21:27.153538+0000","last_fullsized":"2024-03-23T11:21:27.153538+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:15.957774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411397+0000","last_change":"2024-03-23T11:21:25.729414+0000","last_active":"2024-03-23T11:21:27.411397+0000","last_peered":"2024-03-23T11:21:27.411397+0000","last_clean":"2024-03-23T11:21:27.411397+0000","last_became_active":"2024-03-23T11:21:25.727348+0000","last_became_peered":"2024-03-23T11:21:25.727348+0000","last_unstale":"2024-03-23T11:21:27.411397+0000","last_undegraded":"2024-03-23T11:21:27.411397+0000","last_fullsized":"2024-03-23T11:21:27.411397+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:16:19.087133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708769+0000","last_change":"2024-03-23T11:21:25.724688+0000","last_active":"2024-03-23T11:21:26.708769+0000","last_peered":"2024-03-23T11:21:26.708769+0000","last_clean":"2024-03-23T11:21:26.708769+0000","last_became_active":"2024-03-23T11:21:25.724429+0000","last_became_peered":"2024-03-23T11:21:25.724429+0000","last_unstale":"2024-03-23T11:21:26.708769+0000","last_undegraded":"2024-03-23T11:21:26.708769+0000","last_fullsized":"2024-03-23T11:21:26.708769+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:58.142972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705129+0000","last_change":"2024-03-23T11:21:25.736926+0000","last_active":"2024-03-23T11:21:26.705129+0000","last_peered":"2024-03-23T11:21:26.705129+0000","last_clean":"2024-03-23T11:21:26.705129+0000","last_became_active":"2024-03-23T11:21:25.736611+0000","last_became_peered":"2024-03-23T11:21:25.736611+0000","last_unstale":"2024-03-23T11:21:26.705129+0000","last_undegraded":"2024-03-23T11:21:26.705129+0000","last_fullsized":"2024-03-23T11:21:26.705129+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:24.644963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415260+0000","last_change":"2024-03-23T11:21:25.702397+0000","last_active":"2024-03-23T11:21:27.415260+0000","last_peered":"2024-03-23T11:21:27.415260+0000","last_clean":"2024-03-23T11:21:27.415260+0000","last_became_active":"2024-03-23T11:21:25.702163+0000","last_became_peered":"2024-03-23T11:21:25.702163+0000","last_unstale":"2024-03-23T11:21:27.415260+0000","last_undegraded":"2024-03-23T11:21:27.415260+0000","last_fullsized":"2024-03-23T11:21:27.415260+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:44:31.035576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.416134+0000","last_change":"2024-03-23T11:21:25.708704+0000","last_active":"2024-03-23T11:21:27.416134+0000","last_peered":"2024-03-23T11:21:27.416134+0000","last_clean":"2024-03-23T11:21:27.416134+0000","last_became_active":"2024-03-23T11:21:25.708466+0000","last_became_peered":"2024-03-23T11:21:25.708466+0000","last_unstale":"2024-03-23T11:21:27.416134+0000","last_undegraded":"2024-03-23T11:21:27.416134+0000","last_fullsized":"2024-03-23T11:21:27.416134+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:01:06.863295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.706019+0000","last_change":"2024-03-23T11:21:25.247616+0000","last_active":"2024-03-23T11:21:26.706019+0000","last_peered":"2024-03-23T11:21:26.706019+0000","last_clean":"2024-03-23T11:21:26.706019+0000","last_became_active":"2024-03-23T11:21:25.247330+0000","last_became_peered":"2024-03-23T11:21:25.247330+0000","last_unstale":"2024-03-23T11:21:26.706019+0000","last_undegraded":"2024-03-23T11:21:26.706019+0000","last_fullsized":"2024-03-23T11:21:26.706019+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:30:27.358845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412266+0000","last_change":"2024-03-23T11:21:25.727724+0000","last_active":"2024-03-23T11:21:27.412266+0000","last_peered":"2024-03-23T11:21:27.412266+0000","last_clean":"2024-03-23T11:21:27.412266+0000","last_became_active":"2024-03-23T11:21:25.727414+0000","last_became_peered":"2024-03-23T11:21:25.727414+0000","last_unstale":"2024-03-23T11:21:27.412266+0000","last_undegraded":"2024-03-23T11:21:27.412266+0000","last_fullsized":"2024-03-23T11:21:27.412266+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:17:10.080996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709272+0000","last_change":"2024-03-23T11:21:25.238118+0000","last_active":"2024-03-23T11:21:26.709272+0000","last_peered":"2024-03-23T11:21:26.709272+0000","last_clean":"2024-03-23T11:21:26.709272+0000","last_became_active":"2024-03-23T11:21:25.237838+0000","last_became_peered":"2024-03-23T11:21:25.237838+0000","last_unstale":"2024-03-23T11:21:26.709272+0000","last_undegraded":"2024-03-23T11:21:26.709272+0000","last_fullsized":"2024-03-23T11:21:26.709272+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:44:30.021654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153652+0000","last_change":"2024-03-23T11:21:25.733483+0000","last_active":"2024-03-23T11:21:27.153652+0000","last_peered":"2024-03-23T11:21:27.153652+0000","last_clean":"2024-03-23T11:21:27.153652+0000","last_became_active":"2024-03-23T11:21:25.732952+0000","last_became_peered":"2024-03-23T11:21:25.732952+0000","last_unstale":"2024-03-23T11:21:27.153652+0000","last_undegraded":"2024-03-23T11:21:27.153652+0000","last_fullsized":"2024-03-23T11:21:27.153652+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:38:40.371641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.416034+0000","last_change":"2024-03-23T11:21:25.233779+0000","last_active":"2024-03-23T11:21:27.416034+0000","last_peered":"2024-03-23T11:21:27.416034+0000","last_clean":"2024-03-23T11:21:27.416034+0000","last_became_active":"2024-03-23T11:21:25.232737+0000","last_became_peered":"2024-03-23T11:21:25.232737+0000","last_unstale":"2024-03-23T11:21:27.416034+0000","last_undegraded":"2024-03-23T11:21:27.416034+0000","last_fullsized":"2024-03-23T11:21:27.416034+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:54:30.771333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415777+0000","last_change":"2024-03-23T11:21:25.724496+0000","last_active":"2024-03-23T11:21:27.415777+0000","last_peered":"2024-03-23T11:21:27.415777+0000","last_clean":"2024-03-23T11:21:27.415777+0000","last_became_active":"2024-03-23T11:21:25.723664+0000","last_became_peered":"2024-03-23T11:21:25.723664+0000","last_unstale":"2024-03-23T11:21:27.415777+0000","last_undegraded":"2024-03-23T11:21:27.415777+0000","last_fullsized":"2024-03-23T11:21:27.415777+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:55:29.694090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708016+0000","last_change":"2024-03-23T11:21:25.236923+0000","last_active":"2024-03-23T11:21:26.708016+0000","last_peered":"2024-03-23T11:21:26.708016+0000","last_clean":"2024-03-23T11:21:26.708016+0000","last_became_active":"2024-03-23T11:21:25.236655+0000","last_became_peered":"2024-03-23T11:21:25.236655+0000","last_unstale":"2024-03-23T11:21:26.708016+0000","last_undegraded":"2024-03-23T11:21:26.708016+0000","last_fullsized":"2024-03-23T11:21:26.708016+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:32:51.302700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708769+0000","last_change":"2024-03-23T11:21:25.730281+0000","last_active":"2024-03-23T11:21:26.708769+0000","last_peered":"2024-03-23T11:21:26.708769+0000","last_clean":"2024-03-23T11:21:26.708769+0000","last_became_active":"2024-03-23T11:21:25.729682+0000","last_became_peered":"2024-03-23T11:21:25.729682+0000","last_unstale":"2024-03-23T11:21:26.708769+0000","last_undegraded":"2024-03-23T11:21:26.708769+0000","last_fullsized":"2024-03-23T11:21:26.708769+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:15.957774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705459+0000","last_change":"2024-03-23T11:21:25.243050+0000","last_active":"2024-03-23T11:21:26.705459+0000","last_peered":"2024-03-23T11:21:26.705459+0000","last_clean":"2024-03-23T11:21:26.705459+0000","last_became_active":"2024-03-23T11:21:25.242725+0000","last_became_peered":"2024-03-23T11:21:25.242725+0000","last_unstale":"2024-03-23T11:21:26.705459+0000","last_undegraded":"2024-03-23T11:21:26.705459+0000","last_fullsized":"2024-03-23T11:21:26.705459+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:43.619711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413721+0000","last_change":"2024-03-23T11:21:25.718451+0000","last_active":"2024-03-23T11:21:27.413721+0000","last_peered":"2024-03-23T11:21:27.413721+0000","last_clean":"2024-03-23T11:21:27.413721+0000","last_became_active":"2024-03-23T11:21:25.718187+0000","last_became_peered":"2024-03-23T11:21:25.718187+0000","last_unstale":"2024-03-23T11:21:27.413721+0000","last_undegraded":"2024-03-23T11:21:27.413721+0000","last_fullsized":"2024-03-23T11:21:27.413721+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:59.704598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709034+0000","last_change":"2024-03-23T11:21:25.235406+0000","last_active":"2024-03-23T11:21:26.709034+0000","last_peered":"2024-03-23T11:21:26.709034+0000","last_clean":"2024-03-23T11:21:26.709034+0000","last_became_active":"2024-03-23T11:21:25.235031+0000","last_became_peered":"2024-03-23T11:21:25.235031+0000","last_unstale":"2024-03-23T11:21:26.709034+0000","last_undegraded":"2024-03-23T11:21:26.709034+0000","last_fullsized":"2024-03-23T11:21:26.709034+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:51:49.504922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412525+0000","last_change":"2024-03-23T11:21:25.728909+0000","last_active":"2024-03-23T11:21:27.412525+0000","last_peered":"2024-03-23T11:21:27.412525+0000","last_clean":"2024-03-23T11:21:27.412525+0000","last_became_active":"2024-03-23T11:21:25.728658+0000","last_became_peered":"2024-03-23T11:21:25.728658+0000","last_unstale":"2024-03-23T11:21:27.412525+0000","last_undegraded":"2024-03-23T11:21:27.412525+0000","last_fullsized":"2024-03-23T11:21:27.412525+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:45:22.587347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.152794+0000","last_change":"2024-03-23T11:21:25.233936+0000","last_active":"2024-03-23T11:21:27.152794+0000","last_peered":"2024-03-23T11:21:27.152794+0000","last_clean":"2024-03-23T11:21:27.152794+0000","last_became_active":"2024-03-23T11:21:25.233492+0000","last_became_peered":"2024-03-23T11:21:25.233492+0000","last_unstale":"2024-03-23T11:21:27.152794+0000","last_undegraded":"2024-03-23T11:21:27.152794+0000","last_fullsized":"2024-03-23T11:21:27.152794+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:35.431511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411598+0000","last_change":"2024-03-23T11:21:25.735573+0000","last_active":"2024-03-23T11:21:27.411598+0000","last_peered":"2024-03-23T11:21:27.411598+0000","last_clean":"2024-03-23T11:21:27.411598+0000","last_became_active":"2024-03-23T11:21:25.735148+0000","last_became_peered":"2024-03-23T11:21:25.735148+0000","last_unstale":"2024-03-23T11:21:27.411598+0000","last_undegraded":"2024-03-23T11:21:27.411598+0000","last_fullsized":"2024-03-23T11:21:27.411598+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:58.142972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412763+0000","last_change":"2024-03-23T11:21:25.246416+0000","last_active":"2024-03-23T11:21:27.412763+0000","last_peered":"2024-03-23T11:21:27.412763+0000","last_clean":"2024-03-23T11:21:27.412763+0000","last_became_active":"2024-03-23T11:21:25.246120+0000","last_became_peered":"2024-03-23T11:21:25.246120+0000","last_unstale":"2024-03-23T11:21:27.412763+0000","last_undegraded":"2024-03-23T11:21:27.412763+0000","last_fullsized":"2024-03-23T11:21:27.412763+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:22:31.683260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709184+0000","last_change":"2024-03-23T11:21:25.717681+0000","last_active":"2024-03-23T11:21:26.709184+0000","last_peered":"2024-03-23T11:21:26.709184+0000","last_clean":"2024-03-23T11:21:26.709184+0000","last_became_active":"2024-03-23T11:21:25.716834+0000","last_became_peered":"2024-03-23T11:21:25.716834+0000","last_unstale":"2024-03-23T11:21:26.709184+0000","last_undegraded":"2024-03-23T11:21:26.709184+0000","last_fullsized":"2024-03-23T11:21:26.709184+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:24:44.892188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709775+0000","last_change":"2024-03-23T11:21:25.238113+0000","last_active":"2024-03-23T11:21:26.709775+0000","last_peered":"2024-03-23T11:21:26.709775+0000","last_clean":"2024-03-23T11:21:26.709775+0000","last_became_active":"2024-03-23T11:21:25.237821+0000","last_became_peered":"2024-03-23T11:21:25.237821+0000","last_unstale":"2024-03-23T11:21:26.709775+0000","last_undegraded":"2024-03-23T11:21:26.709775+0000","last_fullsized":"2024-03-23T11:21:26.709775+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:57:12.921335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708483+0000","last_change":"2024-03-23T11:21:25.735163+0000","last_active":"2024-03-23T11:21:26.708483+0000","last_peered":"2024-03-23T11:21:26.708483+0000","last_clean":"2024-03-23T11:21:26.708483+0000","last_became_active":"2024-03-23T11:21:25.734900+0000","last_became_peered":"2024-03-23T11:21:25.734900+0000","last_unstale":"2024-03-23T11:21:26.708483+0000","last_undegraded":"2024-03-23T11:21:26.708483+0000","last_fullsized":"2024-03-23T11:21:26.708483+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:59:52.540372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414903+0000","last_change":"2024-03-23T11:21:25.248303+0000","last_active":"2024-03-23T11:21:27.414903+0000","last_peered":"2024-03-23T11:21:27.414903+0000","last_clean":"2024-03-23T11:21:27.414903+0000","last_became_active":"2024-03-23T11:21:25.247968+0000","last_became_peered":"2024-03-23T11:21:25.247968+0000","last_unstale":"2024-03-23T11:21:27.414903+0000","last_undegraded":"2024-03-23T11:21:27.414903+0000","last_fullsized":"2024-03-23T11:21:27.414903+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:03.518519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705702+0000","last_change":"2024-03-23T11:21:25.722033+0000","last_active":"2024-03-23T11:21:26.705702+0000","last_peered":"2024-03-23T11:21:26.705702+0000","last_clean":"2024-03-23T11:21:26.705702+0000","last_became_active":"2024-03-23T11:21:25.721778+0000","last_became_peered":"2024-03-23T11:21:25.721778+0000","last_unstale":"2024-03-23T11:21:26.705702+0000","last_undegraded":"2024-03-23T11:21:26.705702+0000","last_fullsized":"2024-03-23T11:21:26.705702+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:07:35.908037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708387+0000","last_change":"2024-03-23T11:21:25.243533+0000","last_active":"2024-03-23T11:21:26.708387+0000","last_peered":"2024-03-23T11:21:26.708387+0000","last_clean":"2024-03-23T11:21:26.708387+0000","last_became_active":"2024-03-23T11:21:25.243042+0000","last_became_peered":"2024-03-23T11:21:25.243042+0000","last_unstale":"2024-03-23T11:21:26.708387+0000","last_undegraded":"2024-03-23T11:21:26.708387+0000","last_fullsized":"2024-03-23T11:21:26.708387+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:57:21.481260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708112+0000","last_change":"2024-03-23T11:21:25.716460+0000","last_active":"2024-03-23T11:21:26.708112+0000","last_peered":"2024-03-23T11:21:26.708112+0000","last_clean":"2024-03-23T11:21:26.708112+0000","last_became_active":"2024-03-23T11:21:25.715952+0000","last_became_peered":"2024-03-23T11:21:25.715952+0000","last_unstale":"2024-03-23T11:21:26.708112+0000","last_undegraded":"2024-03-23T11:21:26.708112+0000","last_fullsized":"2024-03-23T11:21:26.708112+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:46:32.663913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415779+0000","last_change":"2024-03-23T11:21:25.236579+0000","last_active":"2024-03-23T11:21:27.415779+0000","last_peered":"2024-03-23T11:21:27.415779+0000","last_clean":"2024-03-23T11:21:27.415779+0000","last_became_active":"2024-03-23T11:21:25.236311+0000","last_became_peered":"2024-03-23T11:21:25.236311+0000","last_unstale":"2024-03-23T11:21:27.415779+0000","last_undegraded":"2024-03-23T11:21:27.415779+0000","last_fullsized":"2024-03-23T11:21:27.415779+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:30.813941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411576+0000","last_change":"2024-03-23T11:21:25.739517+0000","last_active":"2024-03-23T11:21:27.411576+0000","last_peered":"2024-03-23T11:21:27.411576+0000","last_clean":"2024-03-23T11:21:27.411576+0000","last_became_active":"2024-03-23T11:21:25.738986+0000","last_became_peered":"2024-03-23T11:21:25.738986+0000","last_unstale":"2024-03-23T11:21:27.411576+0000","last_undegraded":"2024-03-23T11:21:27.411576+0000","last_fullsized":"2024-03-23T11:21:27.411576+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.172566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705530+0000","last_change":"2024-03-23T11:21:25.247411+0000","last_active":"2024-03-23T11:21:26.705530+0000","last_peered":"2024-03-23T11:21:26.705530+0000","last_clean":"2024-03-23T11:21:26.705530+0000","last_became_active":"2024-03-23T11:21:25.246344+0000","last_became_peered":"2024-03-23T11:21:25.246344+0000","last_unstale":"2024-03-23T11:21:26.705530+0000","last_undegraded":"2024-03-23T11:21:26.705530+0000","last_fullsized":"2024-03-23T11:21:26.705530+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:36.158644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414535+0000","last_change":"2024-03-23T11:21:25.725147+0000","last_active":"2024-03-23T11:21:27.414535+0000","last_peered":"2024-03-23T11:21:27.414535+0000","last_clean":"2024-03-23T11:21:27.414535+0000","last_became_active":"2024-03-23T11:21:25.723200+0000","last_became_peered":"2024-03-23T11:21:25.723200+0000","last_unstale":"2024-03-23T11:21:27.414535+0000","last_undegraded":"2024-03-23T11:21:27.414535+0000","last_fullsized":"2024-03-23T11:21:27.414535+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:30:28.372767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.707681+0000","last_change":"2024-03-23T11:21:25.241367+0000","last_active":"2024-03-23T11:21:26.707681+0000","last_peered":"2024-03-23T11:21:26.707681+0000","last_clean":"2024-03-23T11:21:26.707681+0000","last_became_active":"2024-03-23T11:21:25.239130+0000","last_became_peered":"2024-03-23T11:21:25.239130+0000","last_unstale":"2024-03-23T11:21:26.707681+0000","last_undegraded":"2024-03-23T11:21:26.707681+0000","last_fullsized":"2024-03-23T11:21:26.707681+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:27.495291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705021+0000","last_change":"2024-03-23T11:21:25.705982+0000","last_active":"2024-03-23T11:21:26.705021+0000","last_peered":"2024-03-23T11:21:26.705021+0000","last_clean":"2024-03-23T11:21:26.705021+0000","last_became_active":"2024-03-23T11:21:25.705728+0000","last_became_peered":"2024-03-23T11:21:25.705728+0000","last_unstale":"2024-03-23T11:21:26.705021+0000","last_undegraded":"2024-03-23T11:21:26.705021+0000","last_fullsized":"2024-03-23T11:21:26.705021+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:46:30.086218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705295+0000","last_change":"2024-03-23T11:21:25.242606+0000","last_active":"2024-03-23T11:21:26.705295+0000","last_peered":"2024-03-23T11:21:26.705295+0000","last_clean":"2024-03-23T11:21:26.705295+0000","last_became_active":"2024-03-23T11:21:25.241597+0000","last_became_peered":"2024-03-23T11:21:25.241597+0000","last_unstale":"2024-03-23T11:21:26.705295+0000","last_undegraded":"2024-03-23T11:21:26.705295+0000","last_fullsized":"2024-03-23T11:21:26.705295+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:58.690677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.707675+0000","last_change":"2024-03-23T11:21:25.732635+0000","last_active":"2024-03-23T11:21:26.707675+0000","last_peered":"2024-03-23T11:21:26.707675+0000","last_clean":"2024-03-23T11:21:26.707675+0000","last_became_active":"2024-03-23T11:21:25.732329+0000","last_became_peered":"2024-03-23T11:21:25.732329+0000","last_unstale":"2024-03-23T11:21:26.707675+0000","last_undegraded":"2024-03-23T11:21:26.707675+0000","last_fullsized":"2024-03-23T11:21:26.707675+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:13:55.868430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":19,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414564+0000","last_change":"2024-03-23T11:21:25.240149+0000","last_active":"2024-03-23T11:21:27.414564+0000","last_peered":"2024-03-23T11:21:27.414564+0000","last_clean":"2024-03-23T11:21:27.414564+0000","last_became_active":"2024-03-23T11:21:25.239122+0000","last_became_peered":"2024-03-23T11:21:25.239122+0000","last_unstale":"2024-03-23T11:21:27.414564+0000","last_undegraded":"2024-03-23T11:21:27.414564+0000","last_fullsized":"2024-03-23T11:21:27.414564+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:07:21.407189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.416011+0000","last_change":"2024-03-23T11:21:25.739132+0000","last_active":"2024-03-23T11:21:27.416011+0000","last_peered":"2024-03-23T11:21:27.416011+0000","last_clean":"2024-03-23T11:21:27.416011+0000","last_became_active":"2024-03-23T11:21:25.738843+0000","last_became_peered":"2024-03-23T11:21:25.738843+0000","last_unstale":"2024-03-23T11:21:27.416011+0000","last_undegraded":"2024-03-23T11:21:27.416011+0000","last_fullsized":"2024-03-23T11:21:27.416011+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:13.362315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.416309+0000","last_change":"2024-03-23T11:21:25.238639+0000","last_active":"2024-03-23T11:21:27.416309+0000","last_peered":"2024-03-23T11:21:27.416309+0000","last_clean":"2024-03-23T11:21:27.416309+0000","last_became_active":"2024-03-23T11:21:25.238243+0000","last_became_peered":"2024-03-23T11:21:25.238243+0000","last_unstale":"2024-03-23T11:21:27.416309+0000","last_undegraded":"2024-03-23T11:21:27.416309+0000","last_fullsized":"2024-03-23T11:21:27.416309+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:03.518519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415269+0000","last_change":"2024-03-23T11:21:25.720463+0000","last_active":"2024-03-23T11:21:27.415269+0000","last_peered":"2024-03-23T11:21:27.415269+0000","last_clean":"2024-03-23T11:21:27.415269+0000","last_became_active":"2024-03-23T11:21:25.720213+0000","last_became_peered":"2024-03-23T11:21:25.720213+0000","last_unstale":"2024-03-23T11:21:27.415269+0000","last_undegraded":"2024-03-23T11:21:27.415269+0000","last_fullsized":"2024-03-23T11:21:27.415269+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.172566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709454+0000","last_change":"2024-03-23T11:21:25.240590+0000","last_active":"2024-03-23T11:21:26.709454+0000","last_peered":"2024-03-23T11:21:26.709454+0000","last_clean":"2024-03-23T11:21:26.709454+0000","last_became_active":"2024-03-23T11:21:25.240321+0000","last_became_peered":"2024-03-23T11:21:25.240321+0000","last_unstale":"2024-03-23T11:21:26.709454+0000","last_undegraded":"2024-03-23T11:21:26.709454+0000","last_fullsized":"2024-03-23T11:21:26.709454+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:12.348393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412068+0000","last_change":"2024-03-23T11:21:25.730698+0000","last_active":"2024-03-23T11:21:27.412068+0000","last_peered":"2024-03-23T11:21:27.412068+0000","last_clean":"2024-03-23T11:21:27.412068+0000","last_became_active":"2024-03-23T11:21:25.730318+0000","last_became_peered":"2024-03-23T11:21:25.730318+0000","last_unstale":"2024-03-23T11:21:27.412068+0000","last_undegraded":"2024-03-23T11:21:27.412068+0000","last_fullsized":"2024-03-23T11:21:27.412068+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:22:45.890443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153241+0000","last_change":"2024-03-23T11:21:25.242644+0000","last_active":"2024-03-23T11:21:27.153241+0000","last_peered":"2024-03-23T11:21:27.153241+0000","last_clean":"2024-03-23T11:21:27.153241+0000","last_became_active":"2024-03-23T11:21:25.241302+0000","last_became_peered":"2024-03-23T11:21:25.241302+0000","last_unstale":"2024-03-23T11:21:27.153241+0000","last_undegraded":"2024-03-23T11:21:27.153241+0000","last_fullsized":"2024-03-23T11:21:27.153241+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:43.619711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412120+0000","last_change":"2024-03-23T11:21:25.723798+0000","last_active":"2024-03-23T11:21:27.412120+0000","last_peered":"2024-03-23T11:21:27.412120+0000","last_clean":"2024-03-23T11:21:27.412120+0000","last_became_active":"2024-03-23T11:21:25.723081+0000","last_became_peered":"2024-03-23T11:21:25.723081+0000","last_unstale":"2024-03-23T11:21:27.412120+0000","last_undegraded":"2024-03-23T11:21:27.412120+0000","last_fullsized":"2024-03-23T11:21:27.412120+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:51:23.351270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705842+0000","last_change":"2024-03-23T11:21:25.244531+0000","last_active":"2024-03-23T11:21:26.705842+0000","last_peered":"2024-03-23T11:21:26.705842+0000","last_clean":"2024-03-23T11:21:26.705842+0000","last_became_active":"2024-03-23T11:21:25.243903+0000","last_became_peered":"2024-03-23T11:21:25.243903+0000","last_unstale":"2024-03-23T11:21:26.705842+0000","last_undegraded":"2024-03-23T11:21:26.705842+0000","last_fullsized":"2024-03-23T11:21:26.705842+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:17:09.067074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705369+0000","last_change":"2024-03-23T11:21:25.707063+0000","last_active":"2024-03-23T11:21:26.705369+0000","last_peered":"2024-03-23T11:21:26.705369+0000","last_clean":"2024-03-23T11:21:26.705369+0000","last_became_active":"2024-03-23T11:21:25.706793+0000","last_became_peered":"2024-03-23T11:21:25.706793+0000","last_unstale":"2024-03-23T11:21:26.705369+0000","last_undegraded":"2024-03-23T11:21:26.705369+0000","last_fullsized":"2024-03-23T11:21:26.705369+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:10:58.542179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705667+0000","last_change":"2024-03-23T11:21:25.241834+0000","last_active":"2024-03-23T11:21:26.705667+0000","last_peered":"2024-03-23T11:21:26.705667+0000","last_clean":"2024-03-23T11:21:26.705667+0000","last_became_active":"2024-03-23T11:21:25.240492+0000","last_became_peered":"2024-03-23T11:21:25.240492+0000","last_unstale":"2024-03-23T11:21:26.705667+0000","last_undegraded":"2024-03-23T11:21:26.705667+0000","last_fullsized":"2024-03-23T11:21:26.705667+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:51:22.337348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413227+0000","last_change":"2024-03-23T11:21:25.739549+0000","last_active":"2024-03-23T11:21:27.413227+0000","last_peered":"2024-03-23T11:21:27.413227+0000","last_clean":"2024-03-23T11:21:27.413227+0000","last_became_active":"2024-03-23T11:21:25.739225+0000","last_became_peered":"2024-03-23T11:21:25.739225+0000","last_unstale":"2024-03-23T11:21:27.413227+0000","last_undegraded":"2024-03-23T11:21:27.413227+0000","last_fullsized":"2024-03-23T11:21:27.413227+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:46:45.399765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.152852+0000","last_change":"2024-03-23T11:21:25.240479+0000","last_active":"2024-03-23T11:21:27.152852+0000","last_peered":"2024-03-23T11:21:27.152852+0000","last_clean":"2024-03-23T11:21:27.152852+0000","last_became_active":"2024-03-23T11:21:25.240183+0000","last_became_peered":"2024-03-23T11:21:25.240183+0000","last_unstale":"2024-03-23T11:21:27.152852+0000","last_undegraded":"2024-03-23T11:21:27.152852+0000","last_fullsized":"2024-03-23T11:21:27.152852+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:51:22.337348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415538+0000","last_change":"2024-03-23T11:21:25.726452+0000","last_active":"2024-03-23T11:21:27.415538+0000","last_peered":"2024-03-23T11:21:27.415538+0000","last_clean":"2024-03-23T11:21:27.415538+0000","last_became_active":"2024-03-23T11:21:25.726164+0000","last_became_peered":"2024-03-23T11:21:25.726164+0000","last_unstale":"2024-03-23T11:21:27.415538+0000","last_undegraded":"2024-03-23T11:21:27.415538+0000","last_fullsized":"2024-03-23T11:21:27.415538+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:28:32.903553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412648+0000","last_change":"2024-03-23T11:21:25.248848+0000","last_active":"2024-03-23T11:21:27.412648+0000","last_peered":"2024-03-23T11:21:27.412648+0000","last_clean":"2024-03-23T11:21:27.412648+0000","last_became_active":"2024-03-23T11:21:25.248550+0000","last_became_peered":"2024-03-23T11:21:25.248550+0000","last_unstale":"2024-03-23T11:21:27.412648+0000","last_undegraded":"2024-03-23T11:21:27.412648+0000","last_fullsized":"2024-03-23T11:21:27.412648+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:30.813941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412410+0000","last_change":"2024-03-23T11:21:25.713688+0000","last_active":"2024-03-23T11:21:27.412410+0000","last_peered":"2024-03-23T11:21:27.412410+0000","last_clean":"2024-03-23T11:21:27.412410+0000","last_became_active":"2024-03-23T11:21:25.713432+0000","last_became_peered":"2024-03-23T11:21:25.713432+0000","last_unstale":"2024-03-23T11:21:27.412410+0000","last_undegraded":"2024-03-23T11:21:27.412410+0000","last_fullsized":"2024-03-23T11:21:27.412410+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:14.264816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.707532+0000","last_change":"2024-03-23T11:21:25.239911+0000","last_active":"2024-03-23T11:21:26.707532+0000","last_peered":"2024-03-23T11:21:26.707532+0000","last_clean":"2024-03-23T11:21:26.707532+0000","last_became_active":"2024-03-23T11:21:25.239085+0000","last_became_peered":"2024-03-23T11:21:25.239085+0000","last_unstale":"2024-03-23T11:21:26.707532+0000","last_undegraded":"2024-03-23T11:21:26.707532+0000","last_fullsized":"2024-03-23T11:21:26.707532+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:10.047255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412439+0000","last_change":"2024-03-23T11:21:25.737196+0000","last_active":"2024-03-23T11:21:27.412439+0000","last_peered":"2024-03-23T11:21:27.412439+0000","last_clean":"2024-03-23T11:21:27.412439+0000","last_became_active":"2024-03-23T11:21:25.736819+0000","last_became_peered":"2024-03-23T11:21:25.736819+0000","last_unstale":"2024-03-23T11:21:27.412439+0000","last_undegraded":"2024-03-23T11:21:27.412439+0000","last_fullsized":"2024-03-23T11:21:27.412439+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:24:44.892188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705812+0000","last_change":"2024-03-23T11:21:25.245691+0000","last_active":"2024-03-23T11:21:26.705812+0000","last_peered":"2024-03-23T11:21:26.705812+0000","last_clean":"2024-03-23T11:21:26.705812+0000","last_became_active":"2024-03-23T11:21:25.245427+0000","last_became_peered":"2024-03-23T11:21:25.245427+0000","last_unstale":"2024-03-23T11:21:26.705812+0000","last_undegraded":"2024-03-23T11:21:26.705812+0000","last_fullsized":"2024-03-23T11:21:26.705812+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:35.431511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414541+0000","last_change":"2024-03-23T11:21:25.719724+0000","last_active":"2024-03-23T11:21:27.414541+0000","last_peered":"2024-03-23T11:21:27.414541+0000","last_clean":"2024-03-23T11:21:27.414541+0000","last_became_active":"2024-03-23T11:21:25.719520+0000","last_became_peered":"2024-03-23T11:21:25.719520+0000","last_unstale":"2024-03-23T11:21:27.414541+0000","last_undegraded":"2024-03-23T11:21:27.414541+0000","last_fullsized":"2024-03-23T11:21:27.414541+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:17:10.080996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709461+0000","last_change":"2024-03-23T11:21:25.243289+0000","last_active":"2024-03-23T11:21:26.709461+0000","last_peered":"2024-03-23T11:21:26.709461+0000","last_clean":"2024-03-23T11:21:26.709461+0000","last_became_active":"2024-03-23T11:21:25.242599+0000","last_became_peered":"2024-03-23T11:21:25.242599+0000","last_unstale":"2024-03-23T11:21:26.709461+0000","last_undegraded":"2024-03-23T11:21:26.709461+0000","last_fullsized":"2024-03-23T11:21:26.709461+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:51:11.971960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708297+0000","last_change":"2024-03-23T11:21:25.716247+0000","last_active":"2024-03-23T11:21:26.708297+0000","last_peered":"2024-03-23T11:21:26.708297+0000","last_clean":"2024-03-23T11:21:26.708297+0000","last_became_active":"2024-03-23T11:21:25.715513+0000","last_became_peered":"2024-03-23T11:21:25.715513+0000","last_unstale":"2024-03-23T11:21:26.708297+0000","last_undegraded":"2024-03-23T11:21:26.708297+0000","last_fullsized":"2024-03-23T11:21:26.708297+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:35:32.092319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413529+0000","last_change":"2024-03-23T11:21:25.232342+0000","last_active":"2024-03-23T11:21:27.413529+0000","last_peered":"2024-03-23T11:21:27.413529+0000","last_clean":"2024-03-23T11:21:27.413529+0000","last_became_active":"2024-03-23T11:21:25.230334+0000","last_became_peered":"2024-03-23T11:21:25.230334+0000","last_unstale":"2024-03-23T11:21:27.413529+0000","last_undegraded":"2024-03-23T11:21:27.413529+0000","last_fullsized":"2024-03-23T11:21:27.413529+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:29:39.675931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705598+0000","last_change":"2024-03-23T11:21:25.707016+0000","last_active":"2024-03-23T11:21:26.705598+0000","last_peered":"2024-03-23T11:21:26.705598+0000","last_clean":"2024-03-23T11:21:26.705598+0000","last_became_active":"2024-03-23T11:21:25.706715+0000","last_became_peered":"2024-03-23T11:21:25.706715+0000","last_unstale":"2024-03-23T11:21:26.705598+0000","last_undegraded":"2024-03-23T11:21:26.705598+0000","last_fullsized":"2024-03-23T11:21:26.705598+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:13:49.965517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708093+0000","last_change":"2024-03-23T11:21:25.241625+0000","last_active":"2024-03-23T11:21:26.708093+0000","last_peered":"2024-03-23T11:21:26.708093+0000","last_clean":"2024-03-23T11:21:26.708093+0000","last_became_active":"2024-03-23T11:21:25.241001+0000","last_became_peered":"2024-03-23T11:21:25.241001+0000","last_unstale":"2024-03-23T11:21:26.708093+0000","last_undegraded":"2024-03-23T11:21:26.708093+0000","last_fullsized":"2024-03-23T11:21:26.708093+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:22:03.599640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705409+0000","last_change":"2024-03-23T11:21:25.721698+0000","last_active":"2024-03-23T11:21:26.705409+0000","last_peered":"2024-03-23T11:21:26.705409+0000","last_clean":"2024-03-23T11:21:26.705409+0000","last_became_active":"2024-03-23T11:21:25.721426+0000","last_became_peered":"2024-03-23T11:21:25.721426+0000","last_unstale":"2024-03-23T11:21:26.705409+0000","last_undegraded":"2024-03-23T11:21:26.705409+0000","last_fullsized":"2024-03-23T11:21:26.705409+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:30:26.838709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709019+0000","last_change":"2024-03-23T11:21:25.242174+0000","last_active":"2024-03-23T11:21:26.709019+0000","last_peered":"2024-03-23T11:21:26.709019+0000","last_clean":"2024-03-23T11:21:26.709019+0000","last_became_active":"2024-03-23T11:21:25.241908+0000","last_became_peered":"2024-03-23T11:21:25.241908+0000","last_unstale":"2024-03-23T11:21:26.709019+0000","last_undegraded":"2024-03-23T11:21:26.709019+0000","last_fullsized":"2024-03-23T11:21:26.709019+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:18:15.869974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.152811+0000","last_change":"2024-03-23T11:21:25.734053+0000","last_active":"2024-03-23T11:21:27.152811+0000","last_peered":"2024-03-23T11:21:27.152811+0000","last_clean":"2024-03-23T11:21:27.152811+0000","last_became_active":"2024-03-23T11:21:25.733771+0000","last_became_peered":"2024-03-23T11:21:25.733771+0000","last_unstale":"2024-03-23T11:21:27.152811+0000","last_undegraded":"2024-03-23T11:21:27.152811+0000","last_fullsized":"2024-03-23T11:21:27.152811+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:26:11.643044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413391+0000","last_change":"2024-03-23T11:21:25.240986+0000","last_active":"2024-03-23T11:21:27.413391+0000","last_peered":"2024-03-23T11:21:27.413391+0000","last_clean":"2024-03-23T11:21:27.413391+0000","last_became_active":"2024-03-23T11:21:25.240746+0000","last_became_peered":"2024-03-23T11:21:25.240746+0000","last_unstale":"2024-03-23T11:21:27.413391+0000","last_undegraded":"2024-03-23T11:21:27.413391+0000","last_fullsized":"2024-03-23T11:21:27.413391+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:17:58.351453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709284+0000","last_change":"2024-03-23T11:21:25.716592+0000","last_active":"2024-03-23T11:21:26.709284+0000","last_peered":"2024-03-23T11:21:26.709284+0000","last_clean":"2024-03-23T11:21:26.709284+0000","last_became_active":"2024-03-23T11:21:25.716232+0000","last_became_peered":"2024-03-23T11:21:25.716232+0000","last_unstale":"2024-03-23T11:21:26.709284+0000","last_undegraded":"2024-03-23T11:21:26.709284+0000","last_fullsized":"2024-03-23T11:21:26.709284+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:30:28.372767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413002+0000","last_change":"2024-03-23T11:21:25.245195+0000","last_active":"2024-03-23T11:21:27.413002+0000","last_peered":"2024-03-23T11:21:27.413002+0000","last_clean":"2024-03-23T11:21:27.413002+0000","last_became_active":"2024-03-23T11:21:25.243949+0000","last_became_peered":"2024-03-23T11:21:25.243949+0000","last_unstale":"2024-03-23T11:21:27.413002+0000","last_undegraded":"2024-03-23T11:21:27.413002+0000","last_fullsized":"2024-03-23T11:21:27.413002+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:06:11.083524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414623+0000","last_change":"2024-03-23T11:21:25.724043+0000","last_active":"2024-03-23T11:21:27.414623+0000","last_peered":"2024-03-23T11:21:27.414623+0000","last_clean":"2024-03-23T11:21:27.414623+0000","last_became_active":"2024-03-23T11:21:25.721286+0000","last_became_peered":"2024-03-23T11:21:25.721286+0000","last_unstale":"2024-03-23T11:21:27.414623+0000","last_undegraded":"2024-03-23T11:21:27.414623+0000","last_fullsized":"2024-03-23T11:21:27.414623+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:24:44.892188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153468+0000","last_change":"2024-03-23T11:21:25.239406+0000","last_active":"2024-03-23T11:21:27.153468+0000","last_peered":"2024-03-23T11:21:27.153468+0000","last_clean":"2024-03-23T11:21:27.153468+0000","last_became_active":"2024-03-23T11:21:25.238460+0000","last_became_peered":"2024-03-23T11:21:25.238460+0000","last_unstale":"2024-03-23T11:21:27.153468+0000","last_undegraded":"2024-03-23T11:21:27.153468+0000","last_fullsized":"2024-03-23T11:21:27.153468+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:58.690677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":191,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-23T11:21:25.711939+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:21:25.711939+0000","last_peered":"2024-03-23T11:21:25.711939+0000","last_clean":"2024-03-23T11:21:25.711939+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:21:25.711939+0000","last_undegraded":"2024-03-23T11:21:25.711939+0000","last_fullsized":"2024-03-23T11:21:25.711939+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542179,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1159168,"data_stored":818744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.90800000000000003,"15min":0.89400000000000002},"min":{"1min":0.626,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.998,"5min":0.97599999999999998,"15min":0.97199999999999998},"min":{"1min":0.68600000000000005,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.361,"5min":1.411,"15min":1.411},"last":1.0329999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.171,"15min":1.1759999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.542,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":1.516},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.238,"15min":1.248},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.5589999999999999,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.3480000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.04,"15min":1.0469999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.345,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1259999999999999,"15min":1.123},"min":{"1min":0.82599999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":0.97399999999999998}]},{"osd":3,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1160000000000001,"15min":1.1140000000000001},"min":{"1min":0.76900000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.502,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.244,"15min":1.252},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.661,"5min":1.778,"15min":1.778},"last":1.306}]},{"osd":4,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.212,"15min":1.214},"min":{"1min":0.77100000000000002,"5min":0.627,"15min":0.627},"max":{"1min":1.5720000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.367,"5min":1.349,"15min":1.3460000000000001},"min":{"1min":1.1970000000000001,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.6080000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.476}]},{"osd":5,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.3120000000000001,"15min":1.3280000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.6040000000000001,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.409},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3360000000000001,"15min":1.3400000000000001},"min":{"1min":0.90400000000000003,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.6459999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.5569999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.407,"15min":1.4159999999999999},"min":{"1min":1.04,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7310000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3360000000000001,"15min":1.3300000000000001},"min":{"1min":1.1000000000000001,"5min":0.747,"15min":0.747},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.389}]}]},{"osd":6,"up_from":14,"seq":60129542179,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":565248,"data_stored":228289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.0900000000000001,"15min":1.0680000000000001},"min":{"1min":0.64100000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":3.9140000000000001,"5min":3.9140000000000001,"15min":3.9140000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.1419999999999999,"15min":1.1120000000000001},"min":{"1min":0.71899999999999997,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":6.6539999999999999,"5min":6.6539999999999999,"15min":6.6539999999999999},"last":0.75800000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.206,"15min":1.2190000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.7490000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.2629999999999999,"15min":1.2769999999999999},"min":{"1min":0.78400000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.5369999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.125}]},{"osd":2,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.268,"15min":1.2569999999999999},"min":{"1min":0.83799999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":3.73,"5min":3.73,"15min":3.73},"last":1.085},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2909999999999999,"15min":1.302},"min":{"1min":0.95899999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.2280000000000002,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":0.83699999999999997}]},{"osd":3,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7050000000000001,"5min":1.321,"15min":1.2569999999999999},"min":{"1min":0.81999999999999995,"5min":0.749,"15min":0.749},"max":{"1min":12.914,"5min":12.914,"15min":12.914},"last":1.034},{"interface":"front","average":{"1min":1.361,"5min":1.3879999999999999,"15min":1.3919999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":3.8100000000000001,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.89300000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.3979999999999999,"15min":1.423},"min":{"1min":0.85099999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7310000000000001,"5min":2.702,"15min":2.702},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.4630000000000001,"15min":1.448},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":7.2140000000000004,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":0.86599999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.5549999999999999,"15min":1.581},"min":{"1min":0.95999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.798,"5min":3.097,"15min":3.097},"last":1.212},{"interface":"front","average":{"1min":1.373,"5min":1.5,"15min":1.5209999999999999},"min":{"1min":1.016,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.9219999999999999,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.3580000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.6339999999999999,"5min":1.706,"15min":1.718},"min":{"1min":0.871,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":4.9470000000000001,"5min":4.9470000000000001,"15min":4.9470000000000001},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.5369999999999999,"15min":1.5409999999999999},"min":{"1min":1.0149999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.7749999999999999,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":1.198}]}]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":565248,"data_stored":228356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.1950000000000001,"15min":1.224},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.591,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.339},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.157,"15min":1.159},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.508,"5min":2.355,"15min":2.355},"last":0.98599999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2030000000000001,"15min":1.2050000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.2559999999999998,"5min":2.2559999999999998,"15min":2.2559999999999998},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.228,"15min":1.2390000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.7669999999999999,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.6299999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.7769999999999999,"5min":2.012,"15min":2.012},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.236,"15min":1.244},"min":{"1min":0.74199999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.6120000000000001,"5min":1.946,"15min":1.946},"last":1.2470000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.304,"15min":1.3140000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.716,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.4039999999999999,"15min":1.425},"min":{"1min":0.85399999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.6679999999999999,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.5600000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.4319999999999999,"15min":1.45},"min":{"1min":0.81899999999999995,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1949999999999998,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.54,"15min":1.5640000000000001},"min":{"1min":0.71499999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8300000000000001,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.5960000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.494,"15min":1.5169999999999999},"min":{"1min":0.89300000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.881,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.431},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.55,"15min":1.5780000000000001},"min":{"1min":0.89500000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":2.02}]},{"osd":7,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.5760000000000001,"15min":1.599},"min":{"1min":0.79000000000000004,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.9590000000000001,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.538},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.6220000000000001,"15min":1.6459999999999999},"min":{"1min":0.97399999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":2.3780000000000001,"5min":2.516,"15min":2.516},"last":1.7949999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":573440,"data_stored":229850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0880000000000001,"15min":1.0920000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.554,"5min":1.651,"15min":1.651},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.194,"15min":1.1970000000000001},"min":{"1min":0.79300000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.6910000000000001,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.2370000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.083,"15min":1.0940000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.5029999999999999,"5min":1.591,"15min":1.591},"last":1.409},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.097,"15min":1.093},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.3759999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.22,"15min":1.232},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.617,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2,"15min":1.2050000000000001},"min":{"1min":0.92400000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.5349999999999999,"5min":1.696,"15min":1.696},"last":1.52}]},{"osd":3,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.216,"15min":1.2170000000000001},"min":{"1min":0.88800000000000001,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.415,"5min":1.635,"15min":1.635},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.224,"5min":1.232,"15min":1.2330000000000001},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.629,"5min":1.909,"15min":1.909},"last":1.641}]},{"osd":5,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.294,"15min":1.302},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.524,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.248,"5min":1.3340000000000001,"15min":1.3480000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7490000000000001,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.593}]},{"osd":6,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.375,"15min":1.3759999999999999},"min":{"1min":1.008,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.7410000000000001,"5min":2.319,"15min":2.319},"last":1.333},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4750000000000001,"15min":1.48},"min":{"1min":1.1439999999999999,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.8979999999999999,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.6879999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.466,"15min":1.468},"min":{"1min":1.1759999999999999,"5min":0.754,"15min":0.754},"max":{"1min":1.944,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.474},{"interface":"front","average":{"1min":1.27,"5min":1.403,"15min":1.425},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.8300000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.286}]}]},{"osd":3,"up_from":14,"seq":60129542179,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":573440,"data_stored":228924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.95199999999999996,"15min":0.95499999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.3959999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.099,"15min":1.103},"min":{"1min":0.58099999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.1060000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.996,"15min":1.002},"min":{"1min":0.73699999999999999,"5min":0.442,"15min":0.442},"max":{"1min":1.3360000000000001,"5min":1.976,"15min":1.976},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.254,"15min":1.266},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.867,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":0.82999999999999996}]},{"osd":2,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.165,"15min":1.1579999999999999},"min":{"1min":0.89300000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.6839999999999999,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":0.88},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.2569999999999999,"15min":1.282},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.4850000000000001,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.228}]},{"osd":4,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3009999999999999,"15min":1.3},"min":{"1min":0.95799999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.714,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.387},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.53,"15min":1.554},"min":{"1min":1.1339999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"max":{"1min":1.784,"5min":4.8970000000000002,"15min":4.8970000000000002},"last":1.4159999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.5309999999999999,"15min":1.5449999999999999},"min":{"1min":1.0880000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":1.849,"5min":3.4910000000000001,"15min":3.4910000000000001},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.357,"5min":1.474,"15min":1.494},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.71,"5min":3.125,"15min":3.125},"last":1.4570000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.544,"15min":1.5669999999999999},"min":{"1min":0.95699999999999996,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.8560000000000001,"5min":4.5499999999999998,"15min":4.5499999999999998},"last":1.147},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.5660000000000001,"15min":1.585},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.976,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":1.4830000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.534,"15min":1.5489999999999999},"min":{"1min":1.0229999999999999,"5min":0.877,"15min":0.877},"max":{"1min":1.843,"5min":2.952,"15min":2.952},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.6519999999999999,"15min":1.6659999999999999},"min":{"1min":1.3080000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.929,"5min":4.798,"15min":4.798},"last":1.272}]}]},{"osd":2,"up_from":14,"seq":60129542179,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":577536,"data_stored":229872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.88100000000000001,"15min":0.90100000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1970000000000001,"5min":1.371,"15min":1.371},"last":0.745},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.97699999999999998,"15min":0.96899999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":1.2210000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.96199999999999997,"15min":0.97599999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.2310000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.028,"5min":0.98199999999999998,"15min":0.97399999999999998},"min":{"1min":0.46000000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4870000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.98199999999999998}]},{"osd":3,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.1120000000000001,"15min":1.117},"min":{"1min":0.55300000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.746,"5min":1.98,"15min":1.98},"last":1.456},{"interface":"front","average":{"1min":1.087,"5min":1.091,"15min":1.0920000000000001},"min":{"1min":0.47499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.3939999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.345,"15min":1.347},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.756,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.538},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3440000000000001,"15min":1.3380000000000001},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":1.8200000000000001,"5min":1.871,"15min":1.871},"last":1.103}]},{"osd":5,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.4219999999999999,"15min":1.4359999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.0379999999999998,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.81399999999999995,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.0510000000000002,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.6919999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3720000000000001,"15min":1.3680000000000001},"min":{"1min":0.83599999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.8939999999999999,"5min":2.758,"15min":2.758},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5249999999999999,"15min":1.5389999999999999},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.169,"5min":2.8580000000000001,"15min":2.8580000000000001},"last":1.258}]},{"osd":7,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.411,"15min":1.389},"min":{"1min":0.80400000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.246,"5min":2.246,"15min":2.246},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.6080000000000001,"15min":1.6180000000000001},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.1120000000000001,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.621}]}]},{"osd":1,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":561152,"data_stored":227687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.876,"15min":0.85799999999999998},"min":{"1min":0.42499999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.2959999999999998,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.97099999999999997,"15min":0.98399999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.75,"5min":1.99,"15min":1.99},"last":0.64700000000000002}]},{"osd":2,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.97499999999999998,"15min":0.98199999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.696,"5min":1.738,"15min":1.738},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.085,"5min":1.1220000000000001,"15min":1.1279999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.2849999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.2050000000000001,"15min":1.22},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.1190000000000002,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1779999999999999},"min":{"1min":0.61499999999999999,"5min":0.434,"15min":0.434},"max":{"1min":1.736,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.415}]},{"osd":4,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.2450000000000001,"15min":1.2390000000000001},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.8959999999999999,"5min":2.153,"15min":2.153},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.337,"15min":1.347},"min":{"1min":0.93600000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.9730000000000001,"5min":2.758,"15min":2.758},"last":1.4790000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.4299999999999999,"15min":1.4530000000000001},"min":{"1min":1.075,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.514,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.647},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.45,"15min":1.47},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.9279999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.46}]},{"osd":6,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4350000000000001,"15min":1.4390000000000001},"min":{"1min":0.94799999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.631},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.5640000000000001,"15min":1.5980000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.869,"5min":2.9630000000000001,"15min":2.9630000000000001},"last":1.5640000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.5680000000000001,"15min":1.595},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":2.0049999999999999,"5min":3.2749999999999999,"15min":3.2749999999999999},"last":1.369},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.587,"15min":1.601},"min":{"1min":1.0569999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.4710000000000001,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.581}]}]},{"osd":0,"up_from":14,"seq":60129542178,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28152,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343688,"statfs":{"total":96636764160,"available":96607936512,"internally_reserved":0,"allocated":1155072,"data_stored":818019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.85999999999999999,"15min":0.85499999999999998},"min":{"1min":0.46500000000000002,"5min":0.379,"15min":0.379},"max":{"1min":1.2729999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.376},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.93000000000000005,"15min":0.92000000000000004},"min":{"1min":0.61199999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.542,"5min":1.694,"15min":1.694},"last":0.30399999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.03,"15min":1.026},"min":{"1min":0.57399999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.615,"5min":1.635,"15min":1.635},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.077,"5min":1.014,"15min":1.004},"min":{"1min":0.77500000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.3919999999999999,"5min":1.802,"15min":1.802},"last":0.432}]},{"osd":3,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0820000000000001,"15min":1.0960000000000001},"min":{"1min":0.66500000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.302,"15min":1.3129999999999999},"min":{"1min":0.70499999999999996,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9670000000000001,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":0.57499999999999996}]},{"osd":4,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.3859999999999999,"15min":1.381},"min":{"1min":1.0640000000000001,"5min":0.874,"15min":0.874},"max":{"1min":2.4239999999999999,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.274,"15min":1.2709999999999999},"min":{"1min":1.0049999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.9039999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":0.39900000000000002}]},{"osd":5,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.444,"15min":1.4670000000000001},"min":{"1min":0.85699999999999998,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.7210000000000001,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.458,"5min":1.5469999999999999,"15min":1.5620000000000001},"min":{"1min":0.92400000000000004,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":2.335,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":0.52700000000000002}]},{"osd":6,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.5680000000000001,"15min":1.583},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.4399999999999999,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.393,"5min":1.4350000000000001,"15min":1.4419999999999999},"min":{"1min":0.94399999999999995,"5min":0.753,"15min":0.753},"max":{"1min":1.841,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":0.58399999999999996}]},{"osd":7,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.5589999999999999,"15min":1.5609999999999999},"min":{"1min":1.026,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.5419999999999998,"5min":2.617,"15min":2.617},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.508,"15min":1.5189999999999999},"min":{"1min":1.0269999999999999,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.0169999999999999,"5min":3.081,"15min":3.081},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:21:30.194 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:21:30.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.454+0000 7f188e200640 1 Processor -- start 2024-03-23T11:21:30.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.454+0000 7f188e200640 1 -- start start 2024-03-23T11:21:30.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.455+0000 7f188e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1888143dd0 0x7f188815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:30.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.456+0000 7f188e200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f18880b2f30 0x7f18880704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:30.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.456+0000 7f188e200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1888070a30 con 0x7f1888070090 2024-03-23T11:21:30.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.456+0000 7f188e200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18880cd140 con 0x7f18880b2f30 2024-03-23T11:21:30.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.456+0000 7f188e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18880cd3e0 con 0x7f1888143dd0 2024-03-23T11:21:30.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.456+0000 7f1887e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1888070090 0x7f18880b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59602/0 (socket says 172.21.15.62:59602) 2024-03-23T11:21:30.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.456+0000 7f1887e00640 1 -- 172.21.15.62:0/4257544003 learned_addr learned my addr 172.21.15.62:0/4257544003 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.456+0000 7f188ce00640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1888143dd0 0x7f188815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.456+0000 7f188d800640 1 --2- 172.21.15.62:0/4257544003 >> v2:172.21.15.120:3300/0 conn(0x7f18880b2f30 0x7f18880704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.456+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355111478 0 0) 0x7f1888070a30 con 0x7f1888070090 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f186c003520 con 0x7f1888070090 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3496606904 0 0) 0x7f186c003520 con 0x7f1888070090 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1888143dd0 msgr2=0x7f188815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1888143dd0 0x7f188815abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 >> v2:172.21.15.120:3300/0 conn(0x7f18880b2f30 msgr2=0x7f18880704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 --2- 172.21.15.62:0/4257544003 >> v2:172.21.15.120:3300/0 conn(0x7f18880b2f30 0x7f18880704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18880cd6c0 con 0x7f1888070090 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f1874001670 con 0x7f1888070090 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 >> v1:172.21.15.62:6789/0 conn(0x7f1888070090 legacy=0x7f18880b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1888143dd0 0x7f186c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f186c003f40 0x7f186c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 0x7f186c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18880cd140 con 0x7f186c004d00 2024-03-23T11:21:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18880cd3e0 con 0x7f186c003f40 2024-03-23T11:21:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186c00b980 con 0x7f1888143dd0 2024-03-23T11:21:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.457+0000 7f188d800640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 0x7f186c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.458+0000 7f1887e00640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f186c003f40 0x7f186c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.458+0000 7f188ce00640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1888143dd0 0x7f186c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.458+0000 7f188d800640 1 -- 172.21.15.62:0/4257544003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1888143dd0 msgr2=0x7f186c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.458+0000 7f188d800640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1888143dd0 0x7f186c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.458+0000 7f188d800640 1 -- 172.21.15.62:0/4257544003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f186c003f40 msgr2=0x7f186c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.458+0000 7f188d800640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f186c003f40 0x7f186c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.458+0000 7f188d800640 1 -- 172.21.15.62:0/4257544003 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c003520 con 0x7f186c004d00 2024-03-23T11:21:30.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.459+0000 7f188d800640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 0x7f186c005180 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1880009fc0 tx=0x7f186c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871043745d3a735 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:30.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.459+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188002cbf0 con 0x7f186c004d00 2024-03-23T11:21:30.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.459+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188000f040 con 0x7f186c004d00 2024-03-23T11:21:30.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.459+0000 7f1887400640 1 -- 172.21.15.62:0/4257544003 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1880034440 con 0x7f186c004d00 2024-03-23T11:21:30.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.460+0000 7f188e200640 1 -- 172.21.15.62:0/4257544003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 msgr2=0x7f186c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.460+0000 7f188e200640 1 --2- 172.21.15.62:0/4257544003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 0x7f186c005180 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1880009fc0 tx=0x7f186c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.460+0000 7f188d800640 1 -- 172.21.15.62:0/4257544003 reap_dead start 2024-03-23T11:21:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.460+0000 7f188e200640 1 -- 172.21.15.62:0/4257544003 shutdown_connections 2024-03-23T11:21:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.460+0000 7f188e200640 1 -- 172.21.15.62:0/4257544003 >> 172.21.15.62:0/4257544003 conn(0x7f1888066050 msgr2=0x7f1888069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.460+0000 7f188e200640 1 -- 172.21.15.62:0/4257544003 shutdown_connections 2024-03-23T11:21:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.460+0000 7f188e200640 1 -- 172.21.15.62:0/4257544003 wait complete. 2024-03-23T11:21:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.462+0000 7f188e200640 1 Processor -- start 2024-03-23T11:21:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.462+0000 7f188e200640 1 -- start start 2024-03-23T11:21:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.463+0000 7f188e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 0x7f18880a76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.463+0000 7f188e200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18881f0040 0x7f188813d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.463+0000 7f188e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f188813def0 0x7f1888159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.463+0000 7f188e200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188813e450 con 0x7f186c004d00 2024-03-23T11:21:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.463+0000 7f188e200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188813e730 con 0x7f18881f0040 2024-03-23T11:21:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.463+0000 7f188e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f188813ea10 con 0x7f188813def0 2024-03-23T11:21:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.463+0000 7f188ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 0x7f18880a76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.463+0000 7f188ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 0x7f18880a76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34624/0 (socket says 172.21.15.62:34624) 2024-03-23T11:21:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.463+0000 7f188ce00640 1 -- 172.21.15.62:0/3988111772 learned_addr learned my addr 172.21.15.62:0/3988111772 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:30.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.464+0000 7f1887e00640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18881f0040 0x7f188813d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.464+0000 7f188d800640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f188813def0 0x7f1888159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.464+0000 7f188ce00640 1 -- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f188813def0 msgr2=0x7f1888159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.464+0000 7f188ce00640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f188813def0 0x7f1888159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.464+0000 7f188ce00640 1 -- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18881f0040 msgr2=0x7f188813d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.464+0000 7f188ce00640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18881f0040 0x7f188813d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.464+0000 7f188ce00640 1 -- 172.21.15.62:0/3988111772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188813ecf0 con 0x7f186c004d00 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.464+0000 7f188d800640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f188813def0 0x7f1888159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.464+0000 7f1887e00640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18881f0040 0x7f188813d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.464+0000 7f188ce00640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 0x7f18880a76a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1878011600 tx=0x7f1878011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.465+0000 7f1884c00640 1 -- 172.21.15.62:0/3988111772 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187800f920 con 0x7f186c004d00 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.465+0000 7f1884c00640 1 -- 172.21.15.62:0/3988111772 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f187800fa80 con 0x7f186c004d00 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.465+0000 7f1884c00640 1 -- 172.21.15.62:0/3988111772 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187800fdf0 con 0x7f186c004d00 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.465+0000 7f188e200640 1 -- 172.21.15.62:0/3988111772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188813efd0 con 0x7f186c004d00 2024-03-23T11:21:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.465+0000 7f188e200640 1 -- 172.21.15.62:0/3988111772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18880a7be0 con 0x7f186c004d00 2024-03-23T11:21:30.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.466+0000 7f188e200640 1 -- 172.21.15.62:0/3988111772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18880cca40 con 0x7f186c004d00 2024-03-23T11:21:30.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.467+0000 7f1884c00640 1 -- 172.21.15.62:0/3988111772 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f1878015460 con 0x7f186c004d00 2024-03-23T11:21:30.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.468+0000 7f1884c00640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f186806a7d0 0x7f186806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:30.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.468+0000 7f1884c00640 1 -- 172.21.15.62:0/3988111772 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18780891c0 con 0x7f186c004d00 2024-03-23T11:21:30.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.468+0000 7f1887e00640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f186806a7d0 0x7f186806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:30.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.470+0000 7f1887e00640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f186806a7d0 0x7f186806cc90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f188813cbd0 tx=0x7f1874001010 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:30.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.473+0000 7f1884c00640 1 -- 172.21.15.62:0/3988111772 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1878054670 con 0x7f186c004d00 2024-03-23T11:21:30.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.669+0000 7f188e200640 1 -- 172.21.15.62:0/3988111772 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f18880cdf60 con 0x7f186806a7d0 2024-03-23T11:21:30.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.691+0000 7f1884c00640 1 -- 172.21.15.62:0/3988111772 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286801 (secure 0 0 0) 0x7f1874001c80 con 0x7f186806a7d0 2024-03-23T11:21:30.695 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:30.696 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.697+0000 7f188e200640 1 -- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f186806a7d0 msgr2=0x7f186806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.697+0000 7f188e200640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f186806a7d0 0x7f186806cc90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f188813cbd0 tx=0x7f1874001010 comp rx=0 tx=0).stop 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.697+0000 7f188e200640 1 -- 172.21.15.62:0/3988111772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 msgr2=0x7f18880a76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.697+0000 7f188e200640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 0x7f18880a76a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1878011600 tx=0x7f1878011ac0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.698+0000 7f188e200640 1 -- 172.21.15.62:0/3988111772 shutdown_connections 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.698+0000 7f188e200640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f186806a7d0 0x7f186806cc90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.698+0000 7f188e200640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f188813def0 0x7f1888159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.698+0000 7f188e200640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18881f0040 0x7f188813d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.698+0000 7f188e200640 1 --2- 172.21.15.62:0/3988111772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f186c004d00 0x7f18880a76a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.698+0000 7f188e200640 1 -- 172.21.15.62:0/3988111772 >> 172.21.15.62:0/3988111772 conn(0x7f1888066050 msgr2=0x7f18880b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.698+0000 7f188e200640 1 -- 172.21.15.62:0/3988111772 shutdown_connections 2024-03-23T11:21:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.698+0000 7f188e200640 1 -- 172.21.15.62:0/3988111772 wait complete. 2024-03-23T11:21:30.718 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-03-23T11:21:29.284640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":229968,"kb_used_data":5596,"kb_used_omap":13,"kb_used_meta":224242,"kb_avail":754744752,"statfs":{"total":773094113280,"available":772858626048,"internally_reserved":0,"allocated":5730304,"data_stored":3009741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":229623896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.084970"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708444+0000","last_change":"2024-03-23T11:21:25.724317+0000","last_active":"2024-03-23T11:21:26.708444+0000","last_peered":"2024-03-23T11:21:26.708444+0000","last_clean":"2024-03-23T11:21:26.708444+0000","last_became_active":"2024-03-23T11:21:25.723999+0000","last_became_peered":"2024-03-23T11:21:25.723999+0000","last_unstale":"2024-03-23T11:21:26.708444+0000","last_undegraded":"2024-03-23T11:21:26.708444+0000","last_fullsized":"2024-03-23T11:21:26.708444+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:20:38.705711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153295+0000","last_change":"2024-03-23T11:21:25.736776+0000","last_active":"2024-03-23T11:21:27.153295+0000","last_peered":"2024-03-23T11:21:27.153295+0000","last_clean":"2024-03-23T11:21:27.153295+0000","last_became_active":"2024-03-23T11:21:25.736444+0000","last_became_peered":"2024-03-23T11:21:25.736444+0000","last_unstale":"2024-03-23T11:21:27.153295+0000","last_undegraded":"2024-03-23T11:21:27.153295+0000","last_fullsized":"2024-03-23T11:21:27.153295+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:10:58.542179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708959+0000","last_change":"2024-03-23T11:21:25.713672+0000","last_active":"2024-03-23T11:21:26.708959+0000","last_peered":"2024-03-23T11:21:26.708959+0000","last_clean":"2024-03-23T11:21:26.708959+0000","last_became_active":"2024-03-23T11:21:25.713239+0000","last_became_peered":"2024-03-23T11:21:25.713239+0000","last_unstale":"2024-03-23T11:21:26.708959+0000","last_undegraded":"2024-03-23T11:21:26.708959+0000","last_fullsized":"2024-03-23T11:21:26.708959+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:59.704598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414281+0000","last_change":"2024-03-23T11:21:25.732972+0000","last_active":"2024-03-23T11:21:27.414281+0000","last_peered":"2024-03-23T11:21:27.414281+0000","last_clean":"2024-03-23T11:21:27.414281+0000","last_became_active":"2024-03-23T11:21:25.732767+0000","last_became_peered":"2024-03-23T11:21:25.732767+0000","last_unstale":"2024-03-23T11:21:27.414281+0000","last_undegraded":"2024-03-23T11:21:27.414281+0000","last_fullsized":"2024-03-23T11:21:27.414281+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:26:11.643044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153464+0000","last_change":"2024-03-23T11:21:25.738428+0000","last_active":"2024-03-23T11:21:27.153464+0000","last_peered":"2024-03-23T11:21:27.153464+0000","last_clean":"2024-03-23T11:21:27.153464+0000","last_became_active":"2024-03-23T11:21:25.738122+0000","last_became_peered":"2024-03-23T11:21:25.738122+0000","last_unstale":"2024-03-23T11:21:27.153464+0000","last_undegraded":"2024-03-23T11:21:27.153464+0000","last_fullsized":"2024-03-23T11:21:27.153464+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:45:22.587347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705323+0000","last_change":"2024-03-23T11:21:25.719207+0000","last_active":"2024-03-23T11:21:26.705323+0000","last_peered":"2024-03-23T11:21:26.705323+0000","last_clean":"2024-03-23T11:21:26.705323+0000","last_became_active":"2024-03-23T11:21:25.718962+0000","last_became_peered":"2024-03-23T11:21:25.718962+0000","last_unstale":"2024-03-23T11:21:26.705323+0000","last_undegraded":"2024-03-23T11:21:26.705323+0000","last_fullsized":"2024-03-23T11:21:26.705323+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:34:17.443990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.707497+0000","last_change":"2024-03-23T11:21:25.710267+0000","last_active":"2024-03-23T11:21:26.707497+0000","last_peered":"2024-03-23T11:21:26.707497+0000","last_clean":"2024-03-23T11:21:26.707497+0000","last_became_active":"2024-03-23T11:21:25.709808+0000","last_became_peered":"2024-03-23T11:21:25.709808+0000","last_unstale":"2024-03-23T11:21:26.707497+0000","last_undegraded":"2024-03-23T11:21:26.707497+0000","last_fullsized":"2024-03-23T11:21:26.707497+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:49.587791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412250+0000","last_change":"2024-03-23T11:21:25.729487+0000","last_active":"2024-03-23T11:21:27.412250+0000","last_peered":"2024-03-23T11:21:27.412250+0000","last_clean":"2024-03-23T11:21:27.412250+0000","last_became_active":"2024-03-23T11:21:25.729226+0000","last_became_peered":"2024-03-23T11:21:25.729226+0000","last_unstale":"2024-03-23T11:21:27.412250+0000","last_undegraded":"2024-03-23T11:21:27.412250+0000","last_fullsized":"2024-03-23T11:21:27.412250+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:38:40.371641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415104+0000","last_change":"2024-03-23T11:21:25.716142+0000","last_active":"2024-03-23T11:21:27.415104+0000","last_peered":"2024-03-23T11:21:27.415104+0000","last_clean":"2024-03-23T11:21:27.415104+0000","last_became_active":"2024-03-23T11:21:25.715891+0000","last_became_peered":"2024-03-23T11:21:25.715891+0000","last_unstale":"2024-03-23T11:21:27.415104+0000","last_undegraded":"2024-03-23T11:21:27.415104+0000","last_fullsized":"2024-03-23T11:21:27.415104+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:58.142972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415180+0000","last_change":"2024-03-23T11:21:25.729941+0000","last_active":"2024-03-23T11:21:27.415180+0000","last_peered":"2024-03-23T11:21:27.415180+0000","last_clean":"2024-03-23T11:21:27.415180+0000","last_became_active":"2024-03-23T11:21:25.729669+0000","last_became_peered":"2024-03-23T11:21:25.729669+0000","last_unstale":"2024-03-23T11:21:27.415180+0000","last_undegraded":"2024-03-23T11:21:27.415180+0000","last_fullsized":"2024-03-23T11:21:27.415180+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:01.037491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413244+0000","last_change":"2024-03-23T11:21:25.715370+0000","last_active":"2024-03-23T11:21:27.413244+0000","last_peered":"2024-03-23T11:21:27.413244+0000","last_clean":"2024-03-23T11:21:27.413244+0000","last_became_active":"2024-03-23T11:21:25.715141+0000","last_became_peered":"2024-03-23T11:21:25.715141+0000","last_unstale":"2024-03-23T11:21:27.413244+0000","last_undegraded":"2024-03-23T11:21:27.413244+0000","last_fullsized":"2024-03-23T11:21:27.413244+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:14.281844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153271+0000","last_change":"2024-03-23T11:21:25.734475+0000","last_active":"2024-03-23T11:21:27.153271+0000","last_peered":"2024-03-23T11:21:27.153271+0000","last_clean":"2024-03-23T11:21:27.153271+0000","last_became_active":"2024-03-23T11:21:25.734082+0000","last_became_peered":"2024-03-23T11:21:25.734082+0000","last_unstale":"2024-03-23T11:21:27.153271+0000","last_undegraded":"2024-03-23T11:21:27.153271+0000","last_fullsized":"2024-03-23T11:21:27.153271+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:01.037491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415224+0000","last_change":"2024-03-23T11:21:25.714553+0000","last_active":"2024-03-23T11:21:27.415224+0000","last_peered":"2024-03-23T11:21:27.415224+0000","last_clean":"2024-03-23T11:21:27.415224+0000","last_became_active":"2024-03-23T11:21:25.710516+0000","last_became_peered":"2024-03-23T11:21:25.710516+0000","last_unstale":"2024-03-23T11:21:27.415224+0000","last_undegraded":"2024-03-23T11:21:27.415224+0000","last_fullsized":"2024-03-23T11:21:27.415224+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:14:33.578227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413636+0000","last_change":"2024-03-23T11:21:25.714172+0000","last_active":"2024-03-23T11:21:27.413636+0000","last_peered":"2024-03-23T11:21:27.413636+0000","last_clean":"2024-03-23T11:21:27.413636+0000","last_became_active":"2024-03-23T11:21:25.713924+0000","last_became_peered":"2024-03-23T11:21:25.713924+0000","last_unstale":"2024-03-23T11:21:27.413636+0000","last_undegraded":"2024-03-23T11:21:27.413636+0000","last_fullsized":"2024-03-23T11:21:27.413636+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:15.957774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705165+0000","last_change":"2024-03-23T11:21:25.720955+0000","last_active":"2024-03-23T11:21:26.705165+0000","last_peered":"2024-03-23T11:21:26.705165+0000","last_clean":"2024-03-23T11:21:26.705165+0000","last_became_active":"2024-03-23T11:21:25.720693+0000","last_became_peered":"2024-03-23T11:21:25.720693+0000","last_unstale":"2024-03-23T11:21:26.705165+0000","last_undegraded":"2024-03-23T11:21:26.705165+0000","last_fullsized":"2024-03-23T11:21:26.705165+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:21:29.280006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411479+0000","last_change":"2024-03-23T11:21:25.738494+0000","last_active":"2024-03-23T11:21:27.411479+0000","last_peered":"2024-03-23T11:21:27.411479+0000","last_clean":"2024-03-23T11:21:27.411479+0000","last_became_active":"2024-03-23T11:21:25.738073+0000","last_became_peered":"2024-03-23T11:21:25.738073+0000","last_unstale":"2024-03-23T11:21:27.411479+0000","last_undegraded":"2024-03-23T11:21:27.411479+0000","last_fullsized":"2024-03-23T11:21:27.411479+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:14.281844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709662+0000","last_change":"2024-03-23T11:21:25.734121+0000","last_active":"2024-03-23T11:21:26.709662+0000","last_peered":"2024-03-23T11:21:26.709662+0000","last_clean":"2024-03-23T11:21:26.709662+0000","last_became_active":"2024-03-23T11:21:25.733850+0000","last_became_peered":"2024-03-23T11:21:25.733850+0000","last_unstale":"2024-03-23T11:21:26.709662+0000","last_undegraded":"2024-03-23T11:21:26.709662+0000","last_fullsized":"2024-03-23T11:21:26.709662+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:14.281844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708939+0000","last_change":"2024-03-23T11:21:25.717626+0000","last_active":"2024-03-23T11:21:26.708939+0000","last_peered":"2024-03-23T11:21:26.708939+0000","last_clean":"2024-03-23T11:21:26.708939+0000","last_became_active":"2024-03-23T11:21:25.717163+0000","last_became_peered":"2024-03-23T11:21:25.717163+0000","last_unstale":"2024-03-23T11:21:26.708939+0000","last_undegraded":"2024-03-23T11:21:26.708939+0000","last_fullsized":"2024-03-23T11:21:26.708939+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:14.264816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412300+0000","last_change":"2024-03-23T11:21:25.737723+0000","last_active":"2024-03-23T11:21:27.412300+0000","last_peered":"2024-03-23T11:21:27.412300+0000","last_clean":"2024-03-23T11:21:27.412300+0000","last_became_active":"2024-03-23T11:21:25.737232+0000","last_became_peered":"2024-03-23T11:21:25.737232+0000","last_unstale":"2024-03-23T11:21:27.412300+0000","last_undegraded":"2024-03-23T11:21:27.412300+0000","last_fullsized":"2024-03-23T11:21:27.412300+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:59.704598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415692+0000","last_change":"2024-03-23T11:21:25.722849+0000","last_active":"2024-03-23T11:21:27.415692+0000","last_peered":"2024-03-23T11:21:27.415692+0000","last_clean":"2024-03-23T11:21:27.415692+0000","last_became_active":"2024-03-23T11:21:25.721770+0000","last_became_peered":"2024-03-23T11:21:25.721770+0000","last_unstale":"2024-03-23T11:21:27.415692+0000","last_undegraded":"2024-03-23T11:21:27.415692+0000","last_fullsized":"2024-03-23T11:21:27.415692+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:22:45.890443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705345+0000","last_change":"2024-03-23T11:21:25.726876+0000","last_active":"2024-03-23T11:21:26.705345+0000","last_peered":"2024-03-23T11:21:26.705345+0000","last_clean":"2024-03-23T11:21:26.705345+0000","last_became_active":"2024-03-23T11:21:25.726491+0000","last_became_peered":"2024-03-23T11:21:25.726491+0000","last_unstale":"2024-03-23T11:21:26.705345+0000","last_undegraded":"2024-03-23T11:21:26.705345+0000","last_fullsized":"2024-03-23T11:21:26.705345+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:53:32.675975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413747+0000","last_change":"2024-03-23T11:21:25.729488+0000","last_active":"2024-03-23T11:21:27.413747+0000","last_peered":"2024-03-23T11:21:27.413747+0000","last_clean":"2024-03-23T11:21:27.413747+0000","last_became_active":"2024-03-23T11:21:25.729224+0000","last_became_peered":"2024-03-23T11:21:25.729224+0000","last_unstale":"2024-03-23T11:21:27.413747+0000","last_undegraded":"2024-03-23T11:21:27.413747+0000","last_fullsized":"2024-03-23T11:21:27.413747+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:46:30.086218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411714+0000","last_change":"2024-03-23T11:21:25.734805+0000","last_active":"2024-03-23T11:21:27.411714+0000","last_peered":"2024-03-23T11:21:27.411714+0000","last_clean":"2024-03-23T11:21:27.411714+0000","last_became_active":"2024-03-23T11:21:25.734522+0000","last_became_peered":"2024-03-23T11:21:25.734522+0000","last_unstale":"2024-03-23T11:21:27.411714+0000","last_undegraded":"2024-03-23T11:21:27.411714+0000","last_fullsized":"2024-03-23T11:21:27.411714+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:14.264816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411481+0000","last_change":"2024-03-23T11:21:25.734712+0000","last_active":"2024-03-23T11:21:27.411481+0000","last_peered":"2024-03-23T11:21:27.411481+0000","last_clean":"2024-03-23T11:21:27.411481+0000","last_became_active":"2024-03-23T11:21:25.734428+0000","last_became_peered":"2024-03-23T11:21:25.734428+0000","last_unstale":"2024-03-23T11:21:27.411481+0000","last_undegraded":"2024-03-23T11:21:27.411481+0000","last_fullsized":"2024-03-23T11:21:27.411481+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:30:28.372767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412339+0000","last_change":"2024-03-23T11:21:25.728371+0000","last_active":"2024-03-23T11:21:27.412339+0000","last_peered":"2024-03-23T11:21:27.412339+0000","last_clean":"2024-03-23T11:21:27.412339+0000","last_became_active":"2024-03-23T11:21:25.728086+0000","last_became_peered":"2024-03-23T11:21:25.728086+0000","last_unstale":"2024-03-23T11:21:27.412339+0000","last_undegraded":"2024-03-23T11:21:27.412339+0000","last_fullsized":"2024-03-23T11:21:27.412339+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:44.633633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415697+0000","last_change":"2024-03-23T11:21:25.710597+0000","last_active":"2024-03-23T11:21:27.415697+0000","last_peered":"2024-03-23T11:21:27.415697+0000","last_clean":"2024-03-23T11:21:27.415697+0000","last_became_active":"2024-03-23T11:21:25.708337+0000","last_became_peered":"2024-03-23T11:21:25.708337+0000","last_unstale":"2024-03-23T11:21:27.415697+0000","last_undegraded":"2024-03-23T11:21:27.415697+0000","last_fullsized":"2024-03-23T11:21:27.415697+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:51:50.518844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709091+0000","last_change":"2024-03-23T11:21:25.735447+0000","last_active":"2024-03-23T11:21:26.709091+0000","last_peered":"2024-03-23T11:21:26.709091+0000","last_clean":"2024-03-23T11:21:26.709091+0000","last_became_active":"2024-03-23T11:21:25.735137+0000","last_became_peered":"2024-03-23T11:21:25.735137+0000","last_unstale":"2024-03-23T11:21:26.709091+0000","last_undegraded":"2024-03-23T11:21:26.709091+0000","last_fullsized":"2024-03-23T11:21:26.709091+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.172566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153538+0000","last_change":"2024-03-23T11:21:25.735411+0000","last_active":"2024-03-23T11:21:27.153538+0000","last_peered":"2024-03-23T11:21:27.153538+0000","last_clean":"2024-03-23T11:21:27.153538+0000","last_became_active":"2024-03-23T11:21:25.733746+0000","last_became_peered":"2024-03-23T11:21:25.733746+0000","last_unstale":"2024-03-23T11:21:27.153538+0000","last_undegraded":"2024-03-23T11:21:27.153538+0000","last_fullsized":"2024-03-23T11:21:27.153538+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:15.957774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411397+0000","last_change":"2024-03-23T11:21:25.729414+0000","last_active":"2024-03-23T11:21:27.411397+0000","last_peered":"2024-03-23T11:21:27.411397+0000","last_clean":"2024-03-23T11:21:27.411397+0000","last_became_active":"2024-03-23T11:21:25.727348+0000","last_became_peered":"2024-03-23T11:21:25.727348+0000","last_unstale":"2024-03-23T11:21:27.411397+0000","last_undegraded":"2024-03-23T11:21:27.411397+0000","last_fullsized":"2024-03-23T11:21:27.411397+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:16:19.087133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708769+0000","last_change":"2024-03-23T11:21:25.724688+0000","last_active":"2024-03-23T11:21:26.708769+0000","last_peered":"2024-03-23T11:21:26.708769+0000","last_clean":"2024-03-23T11:21:26.708769+0000","last_became_active":"2024-03-23T11:21:25.724429+0000","last_became_peered":"2024-03-23T11:21:25.724429+0000","last_unstale":"2024-03-23T11:21:26.708769+0000","last_undegraded":"2024-03-23T11:21:26.708769+0000","last_fullsized":"2024-03-23T11:21:26.708769+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:58.142972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705129+0000","last_change":"2024-03-23T11:21:25.736926+0000","last_active":"2024-03-23T11:21:26.705129+0000","last_peered":"2024-03-23T11:21:26.705129+0000","last_clean":"2024-03-23T11:21:26.705129+0000","last_became_active":"2024-03-23T11:21:25.736611+0000","last_became_peered":"2024-03-23T11:21:25.736611+0000","last_unstale":"2024-03-23T11:21:26.705129+0000","last_undegraded":"2024-03-23T11:21:26.705129+0000","last_fullsized":"2024-03-23T11:21:26.705129+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:24.644963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415260+0000","last_change":"2024-03-23T11:21:25.702397+0000","last_active":"2024-03-23T11:21:27.415260+0000","last_peered":"2024-03-23T11:21:27.415260+0000","last_clean":"2024-03-23T11:21:27.415260+0000","last_became_active":"2024-03-23T11:21:25.702163+0000","last_became_peered":"2024-03-23T11:21:25.702163+0000","last_unstale":"2024-03-23T11:21:27.415260+0000","last_undegraded":"2024-03-23T11:21:27.415260+0000","last_fullsized":"2024-03-23T11:21:27.415260+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:44:31.035576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.416134+0000","last_change":"2024-03-23T11:21:25.708704+0000","last_active":"2024-03-23T11:21:27.416134+0000","last_peered":"2024-03-23T11:21:27.416134+0000","last_clean":"2024-03-23T11:21:27.416134+0000","last_became_active":"2024-03-23T11:21:25.708466+0000","last_became_peered":"2024-03-23T11:21:25.708466+0000","last_unstale":"2024-03-23T11:21:27.416134+0000","last_undegraded":"2024-03-23T11:21:27.416134+0000","last_fullsized":"2024-03-23T11:21:27.416134+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:01:06.863295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.706019+0000","last_change":"2024-03-23T11:21:25.247616+0000","last_active":"2024-03-23T11:21:26.706019+0000","last_peered":"2024-03-23T11:21:26.706019+0000","last_clean":"2024-03-23T11:21:26.706019+0000","last_became_active":"2024-03-23T11:21:25.247330+0000","last_became_peered":"2024-03-23T11:21:25.247330+0000","last_unstale":"2024-03-23T11:21:26.706019+0000","last_undegraded":"2024-03-23T11:21:26.706019+0000","last_fullsized":"2024-03-23T11:21:26.706019+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:30:27.358845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412266+0000","last_change":"2024-03-23T11:21:25.727724+0000","last_active":"2024-03-23T11:21:27.412266+0000","last_peered":"2024-03-23T11:21:27.412266+0000","last_clean":"2024-03-23T11:21:27.412266+0000","last_became_active":"2024-03-23T11:21:25.727414+0000","last_became_peered":"2024-03-23T11:21:25.727414+0000","last_unstale":"2024-03-23T11:21:27.412266+0000","last_undegraded":"2024-03-23T11:21:27.412266+0000","last_fullsized":"2024-03-23T11:21:27.412266+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:17:10.080996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709272+0000","last_change":"2024-03-23T11:21:25.238118+0000","last_active":"2024-03-23T11:21:26.709272+0000","last_peered":"2024-03-23T11:21:26.709272+0000","last_clean":"2024-03-23T11:21:26.709272+0000","last_became_active":"2024-03-23T11:21:25.237838+0000","last_became_peered":"2024-03-23T11:21:25.237838+0000","last_unstale":"2024-03-23T11:21:26.709272+0000","last_undegraded":"2024-03-23T11:21:26.709272+0000","last_fullsized":"2024-03-23T11:21:26.709272+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:44:30.021654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153652+0000","last_change":"2024-03-23T11:21:25.733483+0000","last_active":"2024-03-23T11:21:27.153652+0000","last_peered":"2024-03-23T11:21:27.153652+0000","last_clean":"2024-03-23T11:21:27.153652+0000","last_became_active":"2024-03-23T11:21:25.732952+0000","last_became_peered":"2024-03-23T11:21:25.732952+0000","last_unstale":"2024-03-23T11:21:27.153652+0000","last_undegraded":"2024-03-23T11:21:27.153652+0000","last_fullsized":"2024-03-23T11:21:27.153652+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:38:40.371641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.416034+0000","last_change":"2024-03-23T11:21:25.233779+0000","last_active":"2024-03-23T11:21:27.416034+0000","last_peered":"2024-03-23T11:21:27.416034+0000","last_clean":"2024-03-23T11:21:27.416034+0000","last_became_active":"2024-03-23T11:21:25.232737+0000","last_became_peered":"2024-03-23T11:21:25.232737+0000","last_unstale":"2024-03-23T11:21:27.416034+0000","last_undegraded":"2024-03-23T11:21:27.416034+0000","last_fullsized":"2024-03-23T11:21:27.416034+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:54:30.771333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415777+0000","last_change":"2024-03-23T11:21:25.724496+0000","last_active":"2024-03-23T11:21:27.415777+0000","last_peered":"2024-03-23T11:21:27.415777+0000","last_clean":"2024-03-23T11:21:27.415777+0000","last_became_active":"2024-03-23T11:21:25.723664+0000","last_became_peered":"2024-03-23T11:21:25.723664+0000","last_unstale":"2024-03-23T11:21:27.415777+0000","last_undegraded":"2024-03-23T11:21:27.415777+0000","last_fullsized":"2024-03-23T11:21:27.415777+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:55:29.694090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708016+0000","last_change":"2024-03-23T11:21:25.236923+0000","last_active":"2024-03-23T11:21:26.708016+0000","last_peered":"2024-03-23T11:21:26.708016+0000","last_clean":"2024-03-23T11:21:26.708016+0000","last_became_active":"2024-03-23T11:21:25.236655+0000","last_became_peered":"2024-03-23T11:21:25.236655+0000","last_unstale":"2024-03-23T11:21:26.708016+0000","last_undegraded":"2024-03-23T11:21:26.708016+0000","last_fullsized":"2024-03-23T11:21:26.708016+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:32:51.302700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708769+0000","last_change":"2024-03-23T11:21:25.730281+0000","last_active":"2024-03-23T11:21:26.708769+0000","last_peered":"2024-03-23T11:21:26.708769+0000","last_clean":"2024-03-23T11:21:26.708769+0000","last_became_active":"2024-03-23T11:21:25.729682+0000","last_became_peered":"2024-03-23T11:21:25.729682+0000","last_unstale":"2024-03-23T11:21:26.708769+0000","last_undegraded":"2024-03-23T11:21:26.708769+0000","last_fullsized":"2024-03-23T11:21:26.708769+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:15.957774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705459+0000","last_change":"2024-03-23T11:21:25.243050+0000","last_active":"2024-03-23T11:21:26.705459+0000","last_peered":"2024-03-23T11:21:26.705459+0000","last_clean":"2024-03-23T11:21:26.705459+0000","last_became_active":"2024-03-23T11:21:25.242725+0000","last_became_peered":"2024-03-23T11:21:25.242725+0000","last_unstale":"2024-03-23T11:21:26.705459+0000","last_undegraded":"2024-03-23T11:21:26.705459+0000","last_fullsized":"2024-03-23T11:21:26.705459+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:43.619711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413721+0000","last_change":"2024-03-23T11:21:25.718451+0000","last_active":"2024-03-23T11:21:27.413721+0000","last_peered":"2024-03-23T11:21:27.413721+0000","last_clean":"2024-03-23T11:21:27.413721+0000","last_became_active":"2024-03-23T11:21:25.718187+0000","last_became_peered":"2024-03-23T11:21:25.718187+0000","last_unstale":"2024-03-23T11:21:27.413721+0000","last_undegraded":"2024-03-23T11:21:27.413721+0000","last_fullsized":"2024-03-23T11:21:27.413721+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:59.704598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709034+0000","last_change":"2024-03-23T11:21:25.235406+0000","last_active":"2024-03-23T11:21:26.709034+0000","last_peered":"2024-03-23T11:21:26.709034+0000","last_clean":"2024-03-23T11:21:26.709034+0000","last_became_active":"2024-03-23T11:21:25.235031+0000","last_became_peered":"2024-03-23T11:21:25.235031+0000","last_unstale":"2024-03-23T11:21:26.709034+0000","last_undegraded":"2024-03-23T11:21:26.709034+0000","last_fullsized":"2024-03-23T11:21:26.709034+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:51:49.504922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412525+0000","last_change":"2024-03-23T11:21:25.728909+0000","last_active":"2024-03-23T11:21:27.412525+0000","last_peered":"2024-03-23T11:21:27.412525+0000","last_clean":"2024-03-23T11:21:27.412525+0000","last_became_active":"2024-03-23T11:21:25.728658+0000","last_became_peered":"2024-03-23T11:21:25.728658+0000","last_unstale":"2024-03-23T11:21:27.412525+0000","last_undegraded":"2024-03-23T11:21:27.412525+0000","last_fullsized":"2024-03-23T11:21:27.412525+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:45:22.587347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.152794+0000","last_change":"2024-03-23T11:21:25.233936+0000","last_active":"2024-03-23T11:21:27.152794+0000","last_peered":"2024-03-23T11:21:27.152794+0000","last_clean":"2024-03-23T11:21:27.152794+0000","last_became_active":"2024-03-23T11:21:25.233492+0000","last_became_peered":"2024-03-23T11:21:25.233492+0000","last_unstale":"2024-03-23T11:21:27.152794+0000","last_undegraded":"2024-03-23T11:21:27.152794+0000","last_fullsized":"2024-03-23T11:21:27.152794+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:35.431511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411598+0000","last_change":"2024-03-23T11:21:25.735573+0000","last_active":"2024-03-23T11:21:27.411598+0000","last_peered":"2024-03-23T11:21:27.411598+0000","last_clean":"2024-03-23T11:21:27.411598+0000","last_became_active":"2024-03-23T11:21:25.735148+0000","last_became_peered":"2024-03-23T11:21:25.735148+0000","last_unstale":"2024-03-23T11:21:27.411598+0000","last_undegraded":"2024-03-23T11:21:27.411598+0000","last_fullsized":"2024-03-23T11:21:27.411598+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:58.142972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412763+0000","last_change":"2024-03-23T11:21:25.246416+0000","last_active":"2024-03-23T11:21:27.412763+0000","last_peered":"2024-03-23T11:21:27.412763+0000","last_clean":"2024-03-23T11:21:27.412763+0000","last_became_active":"2024-03-23T11:21:25.246120+0000","last_became_peered":"2024-03-23T11:21:25.246120+0000","last_unstale":"2024-03-23T11:21:27.412763+0000","last_undegraded":"2024-03-23T11:21:27.412763+0000","last_fullsized":"2024-03-23T11:21:27.412763+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:22:31.683260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709184+0000","last_change":"2024-03-23T11:21:25.717681+0000","last_active":"2024-03-23T11:21:26.709184+0000","last_peered":"2024-03-23T11:21:26.709184+0000","last_clean":"2024-03-23T11:21:26.709184+0000","last_became_active":"2024-03-23T11:21:25.716834+0000","last_became_peered":"2024-03-23T11:21:25.716834+0000","last_unstale":"2024-03-23T11:21:26.709184+0000","last_undegraded":"2024-03-23T11:21:26.709184+0000","last_fullsized":"2024-03-23T11:21:26.709184+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:24:44.892188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709775+0000","last_change":"2024-03-23T11:21:25.238113+0000","last_active":"2024-03-23T11:21:26.709775+0000","last_peered":"2024-03-23T11:21:26.709775+0000","last_clean":"2024-03-23T11:21:26.709775+0000","last_became_active":"2024-03-23T11:21:25.237821+0000","last_became_peered":"2024-03-23T11:21:25.237821+0000","last_unstale":"2024-03-23T11:21:26.709775+0000","last_undegraded":"2024-03-23T11:21:26.709775+0000","last_fullsized":"2024-03-23T11:21:26.709775+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:57:12.921335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708483+0000","last_change":"2024-03-23T11:21:25.735163+0000","last_active":"2024-03-23T11:21:26.708483+0000","last_peered":"2024-03-23T11:21:26.708483+0000","last_clean":"2024-03-23T11:21:26.708483+0000","last_became_active":"2024-03-23T11:21:25.734900+0000","last_became_peered":"2024-03-23T11:21:25.734900+0000","last_unstale":"2024-03-23T11:21:26.708483+0000","last_undegraded":"2024-03-23T11:21:26.708483+0000","last_fullsized":"2024-03-23T11:21:26.708483+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:59:52.540372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414903+0000","last_change":"2024-03-23T11:21:25.248303+0000","last_active":"2024-03-23T11:21:27.414903+0000","last_peered":"2024-03-23T11:21:27.414903+0000","last_clean":"2024-03-23T11:21:27.414903+0000","last_became_active":"2024-03-23T11:21:25.247968+0000","last_became_peered":"2024-03-23T11:21:25.247968+0000","last_unstale":"2024-03-23T11:21:27.414903+0000","last_undegraded":"2024-03-23T11:21:27.414903+0000","last_fullsized":"2024-03-23T11:21:27.414903+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:03.518519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705702+0000","last_change":"2024-03-23T11:21:25.722033+0000","last_active":"2024-03-23T11:21:26.705702+0000","last_peered":"2024-03-23T11:21:26.705702+0000","last_clean":"2024-03-23T11:21:26.705702+0000","last_became_active":"2024-03-23T11:21:25.721778+0000","last_became_peered":"2024-03-23T11:21:25.721778+0000","last_unstale":"2024-03-23T11:21:26.705702+0000","last_undegraded":"2024-03-23T11:21:26.705702+0000","last_fullsized":"2024-03-23T11:21:26.705702+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:07:35.908037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708387+0000","last_change":"2024-03-23T11:21:25.243533+0000","last_active":"2024-03-23T11:21:26.708387+0000","last_peered":"2024-03-23T11:21:26.708387+0000","last_clean":"2024-03-23T11:21:26.708387+0000","last_became_active":"2024-03-23T11:21:25.243042+0000","last_became_peered":"2024-03-23T11:21:25.243042+0000","last_unstale":"2024-03-23T11:21:26.708387+0000","last_undegraded":"2024-03-23T11:21:26.708387+0000","last_fullsized":"2024-03-23T11:21:26.708387+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:57:21.481260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708112+0000","last_change":"2024-03-23T11:21:25.716460+0000","last_active":"2024-03-23T11:21:26.708112+0000","last_peered":"2024-03-23T11:21:26.708112+0000","last_clean":"2024-03-23T11:21:26.708112+0000","last_became_active":"2024-03-23T11:21:25.715952+0000","last_became_peered":"2024-03-23T11:21:25.715952+0000","last_unstale":"2024-03-23T11:21:26.708112+0000","last_undegraded":"2024-03-23T11:21:26.708112+0000","last_fullsized":"2024-03-23T11:21:26.708112+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:46:32.663913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415779+0000","last_change":"2024-03-23T11:21:25.236579+0000","last_active":"2024-03-23T11:21:27.415779+0000","last_peered":"2024-03-23T11:21:27.415779+0000","last_clean":"2024-03-23T11:21:27.415779+0000","last_became_active":"2024-03-23T11:21:25.236311+0000","last_became_peered":"2024-03-23T11:21:25.236311+0000","last_unstale":"2024-03-23T11:21:27.415779+0000","last_undegraded":"2024-03-23T11:21:27.415779+0000","last_fullsized":"2024-03-23T11:21:27.415779+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:30.813941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.411576+0000","last_change":"2024-03-23T11:21:25.739517+0000","last_active":"2024-03-23T11:21:27.411576+0000","last_peered":"2024-03-23T11:21:27.411576+0000","last_clean":"2024-03-23T11:21:27.411576+0000","last_became_active":"2024-03-23T11:21:25.738986+0000","last_became_peered":"2024-03-23T11:21:25.738986+0000","last_unstale":"2024-03-23T11:21:27.411576+0000","last_undegraded":"2024-03-23T11:21:27.411576+0000","last_fullsized":"2024-03-23T11:21:27.411576+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.172566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705530+0000","last_change":"2024-03-23T11:21:25.247411+0000","last_active":"2024-03-23T11:21:26.705530+0000","last_peered":"2024-03-23T11:21:26.705530+0000","last_clean":"2024-03-23T11:21:26.705530+0000","last_became_active":"2024-03-23T11:21:25.246344+0000","last_became_peered":"2024-03-23T11:21:25.246344+0000","last_unstale":"2024-03-23T11:21:26.705530+0000","last_undegraded":"2024-03-23T11:21:26.705530+0000","last_fullsized":"2024-03-23T11:21:26.705530+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:36.158644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414535+0000","last_change":"2024-03-23T11:21:25.725147+0000","last_active":"2024-03-23T11:21:27.414535+0000","last_peered":"2024-03-23T11:21:27.414535+0000","last_clean":"2024-03-23T11:21:27.414535+0000","last_became_active":"2024-03-23T11:21:25.723200+0000","last_became_peered":"2024-03-23T11:21:25.723200+0000","last_unstale":"2024-03-23T11:21:27.414535+0000","last_undegraded":"2024-03-23T11:21:27.414535+0000","last_fullsized":"2024-03-23T11:21:27.414535+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:30:28.372767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.707681+0000","last_change":"2024-03-23T11:21:25.241367+0000","last_active":"2024-03-23T11:21:26.707681+0000","last_peered":"2024-03-23T11:21:26.707681+0000","last_clean":"2024-03-23T11:21:26.707681+0000","last_became_active":"2024-03-23T11:21:25.239130+0000","last_became_peered":"2024-03-23T11:21:25.239130+0000","last_unstale":"2024-03-23T11:21:26.707681+0000","last_undegraded":"2024-03-23T11:21:26.707681+0000","last_fullsized":"2024-03-23T11:21:26.707681+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:27.495291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705021+0000","last_change":"2024-03-23T11:21:25.705982+0000","last_active":"2024-03-23T11:21:26.705021+0000","last_peered":"2024-03-23T11:21:26.705021+0000","last_clean":"2024-03-23T11:21:26.705021+0000","last_became_active":"2024-03-23T11:21:25.705728+0000","last_became_peered":"2024-03-23T11:21:25.705728+0000","last_unstale":"2024-03-23T11:21:26.705021+0000","last_undegraded":"2024-03-23T11:21:26.705021+0000","last_fullsized":"2024-03-23T11:21:26.705021+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:46:30.086218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705295+0000","last_change":"2024-03-23T11:21:25.242606+0000","last_active":"2024-03-23T11:21:26.705295+0000","last_peered":"2024-03-23T11:21:26.705295+0000","last_clean":"2024-03-23T11:21:26.705295+0000","last_became_active":"2024-03-23T11:21:25.241597+0000","last_became_peered":"2024-03-23T11:21:25.241597+0000","last_unstale":"2024-03-23T11:21:26.705295+0000","last_undegraded":"2024-03-23T11:21:26.705295+0000","last_fullsized":"2024-03-23T11:21:26.705295+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:58.690677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.707675+0000","last_change":"2024-03-23T11:21:25.732635+0000","last_active":"2024-03-23T11:21:26.707675+0000","last_peered":"2024-03-23T11:21:26.707675+0000","last_clean":"2024-03-23T11:21:26.707675+0000","last_became_active":"2024-03-23T11:21:25.732329+0000","last_became_peered":"2024-03-23T11:21:25.732329+0000","last_unstale":"2024-03-23T11:21:26.707675+0000","last_undegraded":"2024-03-23T11:21:26.707675+0000","last_fullsized":"2024-03-23T11:21:26.707675+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:13:55.868430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":19,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414564+0000","last_change":"2024-03-23T11:21:25.240149+0000","last_active":"2024-03-23T11:21:27.414564+0000","last_peered":"2024-03-23T11:21:27.414564+0000","last_clean":"2024-03-23T11:21:27.414564+0000","last_became_active":"2024-03-23T11:21:25.239122+0000","last_became_peered":"2024-03-23T11:21:25.239122+0000","last_unstale":"2024-03-23T11:21:27.414564+0000","last_undegraded":"2024-03-23T11:21:27.414564+0000","last_fullsized":"2024-03-23T11:21:27.414564+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:07:21.407189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.416011+0000","last_change":"2024-03-23T11:21:25.739132+0000","last_active":"2024-03-23T11:21:27.416011+0000","last_peered":"2024-03-23T11:21:27.416011+0000","last_clean":"2024-03-23T11:21:27.416011+0000","last_became_active":"2024-03-23T11:21:25.738843+0000","last_became_peered":"2024-03-23T11:21:25.738843+0000","last_unstale":"2024-03-23T11:21:27.416011+0000","last_undegraded":"2024-03-23T11:21:27.416011+0000","last_fullsized":"2024-03-23T11:21:27.416011+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:13.362315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.416309+0000","last_change":"2024-03-23T11:21:25.238639+0000","last_active":"2024-03-23T11:21:27.416309+0000","last_peered":"2024-03-23T11:21:27.416309+0000","last_clean":"2024-03-23T11:21:27.416309+0000","last_became_active":"2024-03-23T11:21:25.238243+0000","last_became_peered":"2024-03-23T11:21:25.238243+0000","last_unstale":"2024-03-23T11:21:27.416309+0000","last_undegraded":"2024-03-23T11:21:27.416309+0000","last_fullsized":"2024-03-23T11:21:27.416309+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:17:03.518519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415269+0000","last_change":"2024-03-23T11:21:25.720463+0000","last_active":"2024-03-23T11:21:27.415269+0000","last_peered":"2024-03-23T11:21:27.415269+0000","last_clean":"2024-03-23T11:21:27.415269+0000","last_became_active":"2024-03-23T11:21:25.720213+0000","last_became_peered":"2024-03-23T11:21:25.720213+0000","last_unstale":"2024-03-23T11:21:27.415269+0000","last_undegraded":"2024-03-23T11:21:27.415269+0000","last_fullsized":"2024-03-23T11:21:27.415269+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.172566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709454+0000","last_change":"2024-03-23T11:21:25.240590+0000","last_active":"2024-03-23T11:21:26.709454+0000","last_peered":"2024-03-23T11:21:26.709454+0000","last_clean":"2024-03-23T11:21:26.709454+0000","last_became_active":"2024-03-23T11:21:25.240321+0000","last_became_peered":"2024-03-23T11:21:25.240321+0000","last_unstale":"2024-03-23T11:21:26.709454+0000","last_undegraded":"2024-03-23T11:21:26.709454+0000","last_fullsized":"2024-03-23T11:21:26.709454+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:12.348393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412068+0000","last_change":"2024-03-23T11:21:25.730698+0000","last_active":"2024-03-23T11:21:27.412068+0000","last_peered":"2024-03-23T11:21:27.412068+0000","last_clean":"2024-03-23T11:21:27.412068+0000","last_became_active":"2024-03-23T11:21:25.730318+0000","last_became_peered":"2024-03-23T11:21:25.730318+0000","last_unstale":"2024-03-23T11:21:27.412068+0000","last_undegraded":"2024-03-23T11:21:27.412068+0000","last_fullsized":"2024-03-23T11:21:27.412068+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:22:45.890443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153241+0000","last_change":"2024-03-23T11:21:25.242644+0000","last_active":"2024-03-23T11:21:27.153241+0000","last_peered":"2024-03-23T11:21:27.153241+0000","last_clean":"2024-03-23T11:21:27.153241+0000","last_became_active":"2024-03-23T11:21:25.241302+0000","last_became_peered":"2024-03-23T11:21:25.241302+0000","last_unstale":"2024-03-23T11:21:27.153241+0000","last_undegraded":"2024-03-23T11:21:27.153241+0000","last_fullsized":"2024-03-23T11:21:27.153241+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:43.619711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412120+0000","last_change":"2024-03-23T11:21:25.723798+0000","last_active":"2024-03-23T11:21:27.412120+0000","last_peered":"2024-03-23T11:21:27.412120+0000","last_clean":"2024-03-23T11:21:27.412120+0000","last_became_active":"2024-03-23T11:21:25.723081+0000","last_became_peered":"2024-03-23T11:21:25.723081+0000","last_unstale":"2024-03-23T11:21:27.412120+0000","last_undegraded":"2024-03-23T11:21:27.412120+0000","last_fullsized":"2024-03-23T11:21:27.412120+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:51:23.351270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705842+0000","last_change":"2024-03-23T11:21:25.244531+0000","last_active":"2024-03-23T11:21:26.705842+0000","last_peered":"2024-03-23T11:21:26.705842+0000","last_clean":"2024-03-23T11:21:26.705842+0000","last_became_active":"2024-03-23T11:21:25.243903+0000","last_became_peered":"2024-03-23T11:21:25.243903+0000","last_unstale":"2024-03-23T11:21:26.705842+0000","last_undegraded":"2024-03-23T11:21:26.705842+0000","last_fullsized":"2024-03-23T11:21:26.705842+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:17:09.067074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705369+0000","last_change":"2024-03-23T11:21:25.707063+0000","last_active":"2024-03-23T11:21:26.705369+0000","last_peered":"2024-03-23T11:21:26.705369+0000","last_clean":"2024-03-23T11:21:26.705369+0000","last_became_active":"2024-03-23T11:21:25.706793+0000","last_became_peered":"2024-03-23T11:21:25.706793+0000","last_unstale":"2024-03-23T11:21:26.705369+0000","last_undegraded":"2024-03-23T11:21:26.705369+0000","last_fullsized":"2024-03-23T11:21:26.705369+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:10:58.542179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705667+0000","last_change":"2024-03-23T11:21:25.241834+0000","last_active":"2024-03-23T11:21:26.705667+0000","last_peered":"2024-03-23T11:21:26.705667+0000","last_clean":"2024-03-23T11:21:26.705667+0000","last_became_active":"2024-03-23T11:21:25.240492+0000","last_became_peered":"2024-03-23T11:21:25.240492+0000","last_unstale":"2024-03-23T11:21:26.705667+0000","last_undegraded":"2024-03-23T11:21:26.705667+0000","last_fullsized":"2024-03-23T11:21:26.705667+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:51:22.337348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413227+0000","last_change":"2024-03-23T11:21:25.739549+0000","last_active":"2024-03-23T11:21:27.413227+0000","last_peered":"2024-03-23T11:21:27.413227+0000","last_clean":"2024-03-23T11:21:27.413227+0000","last_became_active":"2024-03-23T11:21:25.739225+0000","last_became_peered":"2024-03-23T11:21:25.739225+0000","last_unstale":"2024-03-23T11:21:27.413227+0000","last_undegraded":"2024-03-23T11:21:27.413227+0000","last_fullsized":"2024-03-23T11:21:27.413227+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:46:45.399765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.152852+0000","last_change":"2024-03-23T11:21:25.240479+0000","last_active":"2024-03-23T11:21:27.152852+0000","last_peered":"2024-03-23T11:21:27.152852+0000","last_clean":"2024-03-23T11:21:27.152852+0000","last_became_active":"2024-03-23T11:21:25.240183+0000","last_became_peered":"2024-03-23T11:21:25.240183+0000","last_unstale":"2024-03-23T11:21:27.152852+0000","last_undegraded":"2024-03-23T11:21:27.152852+0000","last_fullsized":"2024-03-23T11:21:27.152852+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:51:22.337348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.415538+0000","last_change":"2024-03-23T11:21:25.726452+0000","last_active":"2024-03-23T11:21:27.415538+0000","last_peered":"2024-03-23T11:21:27.415538+0000","last_clean":"2024-03-23T11:21:27.415538+0000","last_became_active":"2024-03-23T11:21:25.726164+0000","last_became_peered":"2024-03-23T11:21:25.726164+0000","last_unstale":"2024-03-23T11:21:27.415538+0000","last_undegraded":"2024-03-23T11:21:27.415538+0000","last_fullsized":"2024-03-23T11:21:27.415538+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:28:32.903553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412648+0000","last_change":"2024-03-23T11:21:25.248848+0000","last_active":"2024-03-23T11:21:27.412648+0000","last_peered":"2024-03-23T11:21:27.412648+0000","last_clean":"2024-03-23T11:21:27.412648+0000","last_became_active":"2024-03-23T11:21:25.248550+0000","last_became_peered":"2024-03-23T11:21:25.248550+0000","last_unstale":"2024-03-23T11:21:27.412648+0000","last_undegraded":"2024-03-23T11:21:27.412648+0000","last_fullsized":"2024-03-23T11:21:27.412648+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:30.813941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412410+0000","last_change":"2024-03-23T11:21:25.713688+0000","last_active":"2024-03-23T11:21:27.412410+0000","last_peered":"2024-03-23T11:21:27.412410+0000","last_clean":"2024-03-23T11:21:27.412410+0000","last_became_active":"2024-03-23T11:21:25.713432+0000","last_became_peered":"2024-03-23T11:21:25.713432+0000","last_unstale":"2024-03-23T11:21:27.412410+0000","last_undegraded":"2024-03-23T11:21:27.412410+0000","last_fullsized":"2024-03-23T11:21:27.412410+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:14.264816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.707532+0000","last_change":"2024-03-23T11:21:25.239911+0000","last_active":"2024-03-23T11:21:26.707532+0000","last_peered":"2024-03-23T11:21:26.707532+0000","last_clean":"2024-03-23T11:21:26.707532+0000","last_became_active":"2024-03-23T11:21:25.239085+0000","last_became_peered":"2024-03-23T11:21:25.239085+0000","last_unstale":"2024-03-23T11:21:26.707532+0000","last_undegraded":"2024-03-23T11:21:26.707532+0000","last_fullsized":"2024-03-23T11:21:26.707532+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:10.047255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.412439+0000","last_change":"2024-03-23T11:21:25.737196+0000","last_active":"2024-03-23T11:21:27.412439+0000","last_peered":"2024-03-23T11:21:27.412439+0000","last_clean":"2024-03-23T11:21:27.412439+0000","last_became_active":"2024-03-23T11:21:25.736819+0000","last_became_peered":"2024-03-23T11:21:25.736819+0000","last_unstale":"2024-03-23T11:21:27.412439+0000","last_undegraded":"2024-03-23T11:21:27.412439+0000","last_fullsized":"2024-03-23T11:21:27.412439+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:24:44.892188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705812+0000","last_change":"2024-03-23T11:21:25.245691+0000","last_active":"2024-03-23T11:21:26.705812+0000","last_peered":"2024-03-23T11:21:26.705812+0000","last_clean":"2024-03-23T11:21:26.705812+0000","last_became_active":"2024-03-23T11:21:25.245427+0000","last_became_peered":"2024-03-23T11:21:25.245427+0000","last_unstale":"2024-03-23T11:21:26.705812+0000","last_undegraded":"2024-03-23T11:21:26.705812+0000","last_fullsized":"2024-03-23T11:21:26.705812+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:35.431511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414541+0000","last_change":"2024-03-23T11:21:25.719724+0000","last_active":"2024-03-23T11:21:27.414541+0000","last_peered":"2024-03-23T11:21:27.414541+0000","last_clean":"2024-03-23T11:21:27.414541+0000","last_became_active":"2024-03-23T11:21:25.719520+0000","last_became_peered":"2024-03-23T11:21:25.719520+0000","last_unstale":"2024-03-23T11:21:27.414541+0000","last_undegraded":"2024-03-23T11:21:27.414541+0000","last_fullsized":"2024-03-23T11:21:27.414541+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:17:10.080996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709461+0000","last_change":"2024-03-23T11:21:25.243289+0000","last_active":"2024-03-23T11:21:26.709461+0000","last_peered":"2024-03-23T11:21:26.709461+0000","last_clean":"2024-03-23T11:21:26.709461+0000","last_became_active":"2024-03-23T11:21:25.242599+0000","last_became_peered":"2024-03-23T11:21:25.242599+0000","last_unstale":"2024-03-23T11:21:26.709461+0000","last_undegraded":"2024-03-23T11:21:26.709461+0000","last_fullsized":"2024-03-23T11:21:26.709461+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:51:11.971960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708297+0000","last_change":"2024-03-23T11:21:25.716247+0000","last_active":"2024-03-23T11:21:26.708297+0000","last_peered":"2024-03-23T11:21:26.708297+0000","last_clean":"2024-03-23T11:21:26.708297+0000","last_became_active":"2024-03-23T11:21:25.715513+0000","last_became_peered":"2024-03-23T11:21:25.715513+0000","last_unstale":"2024-03-23T11:21:26.708297+0000","last_undegraded":"2024-03-23T11:21:26.708297+0000","last_fullsized":"2024-03-23T11:21:26.708297+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:35:32.092319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413529+0000","last_change":"2024-03-23T11:21:25.232342+0000","last_active":"2024-03-23T11:21:27.413529+0000","last_peered":"2024-03-23T11:21:27.413529+0000","last_clean":"2024-03-23T11:21:27.413529+0000","last_became_active":"2024-03-23T11:21:25.230334+0000","last_became_peered":"2024-03-23T11:21:25.230334+0000","last_unstale":"2024-03-23T11:21:27.413529+0000","last_undegraded":"2024-03-23T11:21:27.413529+0000","last_fullsized":"2024-03-23T11:21:27.413529+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:29:39.675931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705598+0000","last_change":"2024-03-23T11:21:25.707016+0000","last_active":"2024-03-23T11:21:26.705598+0000","last_peered":"2024-03-23T11:21:26.705598+0000","last_clean":"2024-03-23T11:21:26.705598+0000","last_became_active":"2024-03-23T11:21:25.706715+0000","last_became_peered":"2024-03-23T11:21:25.706715+0000","last_unstale":"2024-03-23T11:21:26.705598+0000","last_undegraded":"2024-03-23T11:21:26.705598+0000","last_fullsized":"2024-03-23T11:21:26.705598+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:13:49.965517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.708093+0000","last_change":"2024-03-23T11:21:25.241625+0000","last_active":"2024-03-23T11:21:26.708093+0000","last_peered":"2024-03-23T11:21:26.708093+0000","last_clean":"2024-03-23T11:21:26.708093+0000","last_became_active":"2024-03-23T11:21:25.241001+0000","last_became_peered":"2024-03-23T11:21:25.241001+0000","last_unstale":"2024-03-23T11:21:26.708093+0000","last_undegraded":"2024-03-23T11:21:26.708093+0000","last_fullsized":"2024-03-23T11:21:26.708093+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:22:03.599640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.705409+0000","last_change":"2024-03-23T11:21:25.721698+0000","last_active":"2024-03-23T11:21:26.705409+0000","last_peered":"2024-03-23T11:21:26.705409+0000","last_clean":"2024-03-23T11:21:26.705409+0000","last_became_active":"2024-03-23T11:21:25.721426+0000","last_became_peered":"2024-03-23T11:21:25.721426+0000","last_unstale":"2024-03-23T11:21:26.705409+0000","last_undegraded":"2024-03-23T11:21:26.705409+0000","last_fullsized":"2024-03-23T11:21:26.705409+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:30:26.838709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709019+0000","last_change":"2024-03-23T11:21:25.242174+0000","last_active":"2024-03-23T11:21:26.709019+0000","last_peered":"2024-03-23T11:21:26.709019+0000","last_clean":"2024-03-23T11:21:26.709019+0000","last_became_active":"2024-03-23T11:21:25.241908+0000","last_became_peered":"2024-03-23T11:21:25.241908+0000","last_unstale":"2024-03-23T11:21:26.709019+0000","last_undegraded":"2024-03-23T11:21:26.709019+0000","last_fullsized":"2024-03-23T11:21:26.709019+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:18:15.869974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.152811+0000","last_change":"2024-03-23T11:21:25.734053+0000","last_active":"2024-03-23T11:21:27.152811+0000","last_peered":"2024-03-23T11:21:27.152811+0000","last_clean":"2024-03-23T11:21:27.152811+0000","last_became_active":"2024-03-23T11:21:25.733771+0000","last_became_peered":"2024-03-23T11:21:25.733771+0000","last_unstale":"2024-03-23T11:21:27.152811+0000","last_undegraded":"2024-03-23T11:21:27.152811+0000","last_fullsized":"2024-03-23T11:21:27.152811+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:26:11.643044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413391+0000","last_change":"2024-03-23T11:21:25.240986+0000","last_active":"2024-03-23T11:21:27.413391+0000","last_peered":"2024-03-23T11:21:27.413391+0000","last_clean":"2024-03-23T11:21:27.413391+0000","last_became_active":"2024-03-23T11:21:25.240746+0000","last_became_peered":"2024-03-23T11:21:25.240746+0000","last_unstale":"2024-03-23T11:21:27.413391+0000","last_undegraded":"2024-03-23T11:21:27.413391+0000","last_fullsized":"2024-03-23T11:21:27.413391+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:17:58.351453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:26.709284+0000","last_change":"2024-03-23T11:21:25.716592+0000","last_active":"2024-03-23T11:21:26.709284+0000","last_peered":"2024-03-23T11:21:26.709284+0000","last_clean":"2024-03-23T11:21:26.709284+0000","last_became_active":"2024-03-23T11:21:25.716232+0000","last_became_peered":"2024-03-23T11:21:25.716232+0000","last_unstale":"2024-03-23T11:21:26.709284+0000","last_undegraded":"2024-03-23T11:21:26.709284+0000","last_fullsized":"2024-03-23T11:21:26.709284+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:30:28.372767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.413002+0000","last_change":"2024-03-23T11:21:25.245195+0000","last_active":"2024-03-23T11:21:27.413002+0000","last_peered":"2024-03-23T11:21:27.413002+0000","last_clean":"2024-03-23T11:21:27.413002+0000","last_became_active":"2024-03-23T11:21:25.243949+0000","last_became_peered":"2024-03-23T11:21:25.243949+0000","last_unstale":"2024-03-23T11:21:27.413002+0000","last_undegraded":"2024-03-23T11:21:27.413002+0000","last_fullsized":"2024-03-23T11:21:27.413002+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:06:11.083524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.414623+0000","last_change":"2024-03-23T11:21:25.724043+0000","last_active":"2024-03-23T11:21:27.414623+0000","last_peered":"2024-03-23T11:21:27.414623+0000","last_clean":"2024-03-23T11:21:27.414623+0000","last_became_active":"2024-03-23T11:21:25.721286+0000","last_became_peered":"2024-03-23T11:21:25.721286+0000","last_unstale":"2024-03-23T11:21:27.414623+0000","last_undegraded":"2024-03-23T11:21:27.414623+0000","last_fullsized":"2024-03-23T11:21:27.414623+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:25.198311+0000","last_clean_scrub_stamp":"2024-03-23T11:21:25.198311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:24:44.892188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T11:21:27.153468+0000","last_change":"2024-03-23T11:21:25.239406+0000","last_active":"2024-03-23T11:21:27.153468+0000","last_peered":"2024-03-23T11:21:27.153468+0000","last_clean":"2024-03-23T11:21:27.153468+0000","last_became_active":"2024-03-23T11:21:25.238460+0000","last_became_peered":"2024-03-23T11:21:25.238460+0000","last_unstale":"2024-03-23T11:21:27.153468+0000","last_undegraded":"2024-03-23T11:21:27.153468+0000","last_fullsized":"2024-03-23T11:21:27.153468+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:21:24.184389+0000","last_clean_scrub_stamp":"2024-03-23T11:21:24.184389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:58.690677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":191,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-23T11:21:25.711939+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:21:25.711939+0000","last_peered":"2024-03-23T11:21:25.711939+0000","last_clean":"2024-03-23T11:21:25.711939+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:21:25.711939+0000","last_undegraded":"2024-03-23T11:21:25.711939+0000","last_fullsized":"2024-03-23T11:21:25.711939+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542179,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1159168,"data_stored":818744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.90800000000000003,"15min":0.89400000000000002},"min":{"1min":0.626,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.998,"5min":0.97599999999999998,"15min":0.97199999999999998},"min":{"1min":0.68600000000000005,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.361,"5min":1.411,"15min":1.411},"last":1.0329999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.171,"15min":1.1759999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.542,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":1.516},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.238,"15min":1.248},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.5589999999999999,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.3480000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.04,"15min":1.0469999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.345,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1259999999999999,"15min":1.123},"min":{"1min":0.82599999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":0.97399999999999998}]},{"osd":3,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1160000000000001,"15min":1.1140000000000001},"min":{"1min":0.76900000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.502,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.244,"15min":1.252},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.661,"5min":1.778,"15min":1.778},"last":1.306}]},{"osd":4,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.212,"15min":1.214},"min":{"1min":0.77100000000000002,"5min":0.627,"15min":0.627},"max":{"1min":1.5720000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.367,"5min":1.349,"15min":1.3460000000000001},"min":{"1min":1.1970000000000001,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.6080000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.476}]},{"osd":5,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.3120000000000001,"15min":1.3280000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.6040000000000001,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.409},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3360000000000001,"15min":1.3400000000000001},"min":{"1min":0.90400000000000003,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.6459999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.5569999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.407,"15min":1.4159999999999999},"min":{"1min":1.04,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7310000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3360000000000001,"15min":1.3300000000000001},"min":{"1min":1.1000000000000001,"5min":0.747,"15min":0.747},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.389}]}]},{"osd":6,"up_from":14,"seq":60129542179,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":565248,"data_stored":228289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.0900000000000001,"15min":1.0680000000000001},"min":{"1min":0.64100000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":3.9140000000000001,"5min":3.9140000000000001,"15min":3.9140000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.1419999999999999,"15min":1.1120000000000001},"min":{"1min":0.71899999999999997,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":6.6539999999999999,"5min":6.6539999999999999,"15min":6.6539999999999999},"last":0.75800000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.206,"15min":1.2190000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.7490000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.2629999999999999,"15min":1.2769999999999999},"min":{"1min":0.78400000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.5369999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.125}]},{"osd":2,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.268,"15min":1.2569999999999999},"min":{"1min":0.83799999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":3.73,"5min":3.73,"15min":3.73},"last":1.085},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2909999999999999,"15min":1.302},"min":{"1min":0.95899999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.2280000000000002,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":0.83699999999999997}]},{"osd":3,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7050000000000001,"5min":1.321,"15min":1.2569999999999999},"min":{"1min":0.81999999999999995,"5min":0.749,"15min":0.749},"max":{"1min":12.914,"5min":12.914,"15min":12.914},"last":1.034},{"interface":"front","average":{"1min":1.361,"5min":1.3879999999999999,"15min":1.3919999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":3.8100000000000001,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.89300000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.3979999999999999,"15min":1.423},"min":{"1min":0.85099999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.7310000000000001,"5min":2.702,"15min":2.702},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.4630000000000001,"15min":1.448},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":7.2140000000000004,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":0.86599999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.5549999999999999,"15min":1.581},"min":{"1min":0.95999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.798,"5min":3.097,"15min":3.097},"last":1.212},{"interface":"front","average":{"1min":1.373,"5min":1.5,"15min":1.5209999999999999},"min":{"1min":1.016,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.9219999999999999,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.3580000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.6339999999999999,"5min":1.706,"15min":1.718},"min":{"1min":0.871,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":4.9470000000000001,"5min":4.9470000000000001,"15min":4.9470000000000001},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.5369999999999999,"15min":1.5409999999999999},"min":{"1min":1.0149999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.7749999999999999,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":1.198}]}]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":565248,"data_stored":228356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.1950000000000001,"15min":1.224},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.591,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.339},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.157,"15min":1.159},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.508,"5min":2.355,"15min":2.355},"last":0.98599999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2030000000000001,"15min":1.2050000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.2559999999999998,"5min":2.2559999999999998,"15min":2.2559999999999998},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.228,"15min":1.2390000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.7669999999999999,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.6299999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.7769999999999999,"5min":2.012,"15min":2.012},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.236,"15min":1.244},"min":{"1min":0.74199999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.6120000000000001,"5min":1.946,"15min":1.946},"last":1.2470000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.304,"15min":1.3140000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.716,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.4039999999999999,"15min":1.425},"min":{"1min":0.85399999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.6679999999999999,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.5600000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.4319999999999999,"15min":1.45},"min":{"1min":0.81899999999999995,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1949999999999998,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.54,"15min":1.5640000000000001},"min":{"1min":0.71499999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8300000000000001,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.5960000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.494,"15min":1.5169999999999999},"min":{"1min":0.89300000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.881,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.431},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.55,"15min":1.5780000000000001},"min":{"1min":0.89500000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":2.02}]},{"osd":7,"last update":"Sat Mar 23 11:20:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.5760000000000001,"15min":1.599},"min":{"1min":0.79000000000000004,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.9590000000000001,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.538},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.6220000000000001,"15min":1.6459999999999999},"min":{"1min":0.97399999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":2.3780000000000001,"5min":2.516,"15min":2.516},"last":1.7949999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":573440,"data_stored":229850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0880000000000001,"15min":1.0920000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.554,"5min":1.651,"15min":1.651},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.194,"15min":1.1970000000000001},"min":{"1min":0.79300000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.6910000000000001,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.2370000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.083,"15min":1.0940000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.5029999999999999,"5min":1.591,"15min":1.591},"last":1.409},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.097,"15min":1.093},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.3759999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.22,"15min":1.232},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.617,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2,"15min":1.2050000000000001},"min":{"1min":0.92400000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.5349999999999999,"5min":1.696,"15min":1.696},"last":1.52}]},{"osd":3,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.216,"15min":1.2170000000000001},"min":{"1min":0.88800000000000001,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.415,"5min":1.635,"15min":1.635},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.224,"5min":1.232,"15min":1.2330000000000001},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.629,"5min":1.909,"15min":1.909},"last":1.641}]},{"osd":5,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.294,"15min":1.302},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.524,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.248,"5min":1.3340000000000001,"15min":1.3480000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7490000000000001,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.593}]},{"osd":6,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.375,"15min":1.3759999999999999},"min":{"1min":1.008,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.7410000000000001,"5min":2.319,"15min":2.319},"last":1.333},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4750000000000001,"15min":1.48},"min":{"1min":1.1439999999999999,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.8979999999999999,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.6879999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.466,"15min":1.468},"min":{"1min":1.1759999999999999,"5min":0.754,"15min":0.754},"max":{"1min":1.944,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.474},{"interface":"front","average":{"1min":1.27,"5min":1.403,"15min":1.425},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.8300000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.286}]}]},{"osd":3,"up_from":14,"seq":60129542179,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":573440,"data_stored":228924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.95199999999999996,"15min":0.95499999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.3959999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.099,"15min":1.103},"min":{"1min":0.58099999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.1060000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.996,"15min":1.002},"min":{"1min":0.73699999999999999,"5min":0.442,"15min":0.442},"max":{"1min":1.3360000000000001,"5min":1.976,"15min":1.976},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.254,"15min":1.266},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.867,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":0.82999999999999996}]},{"osd":2,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.165,"15min":1.1579999999999999},"min":{"1min":0.89300000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.6839999999999999,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":0.88},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.2569999999999999,"15min":1.282},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.4850000000000001,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.228}]},{"osd":4,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3009999999999999,"15min":1.3},"min":{"1min":0.95799999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.714,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.387},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.53,"15min":1.554},"min":{"1min":1.1339999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"max":{"1min":1.784,"5min":4.8970000000000002,"15min":4.8970000000000002},"last":1.4159999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.5309999999999999,"15min":1.5449999999999999},"min":{"1min":1.0880000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":1.849,"5min":3.4910000000000001,"15min":3.4910000000000001},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.357,"5min":1.474,"15min":1.494},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.71,"5min":3.125,"15min":3.125},"last":1.4570000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.544,"15min":1.5669999999999999},"min":{"1min":0.95699999999999996,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.8560000000000001,"5min":4.5499999999999998,"15min":4.5499999999999998},"last":1.147},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.5660000000000001,"15min":1.585},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.976,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":1.4830000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:20:53 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.534,"15min":1.5489999999999999},"min":{"1min":1.0229999999999999,"5min":0.877,"15min":0.877},"max":{"1min":1.843,"5min":2.952,"15min":2.952},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.6519999999999999,"15min":1.6659999999999999},"min":{"1min":1.3080000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.929,"5min":4.798,"15min":4.798},"last":1.272}]}]},{"osd":2,"up_from":14,"seq":60129542179,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":577536,"data_stored":229872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.88100000000000001,"15min":0.90100000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1970000000000001,"5min":1.371,"15min":1.371},"last":0.745},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.97699999999999998,"15min":0.96899999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":1.2210000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.96199999999999997,"15min":0.97599999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.2310000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.028,"5min":0.98199999999999998,"15min":0.97399999999999998},"min":{"1min":0.46000000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4870000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.98199999999999998}]},{"osd":3,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.1120000000000001,"15min":1.117},"min":{"1min":0.55300000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.746,"5min":1.98,"15min":1.98},"last":1.456},{"interface":"front","average":{"1min":1.087,"5min":1.091,"15min":1.0920000000000001},"min":{"1min":0.47499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.3939999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.345,"15min":1.347},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.756,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.538},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3440000000000001,"15min":1.3380000000000001},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":1.8200000000000001,"5min":1.871,"15min":1.871},"last":1.103}]},{"osd":5,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.4219999999999999,"15min":1.4359999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.0379999999999998,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.81399999999999995,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.0510000000000002,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.6919999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3720000000000001,"15min":1.3680000000000001},"min":{"1min":0.83599999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.8939999999999999,"5min":2.758,"15min":2.758},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5249999999999999,"15min":1.5389999999999999},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.169,"5min":2.8580000000000001,"15min":2.8580000000000001},"last":1.258}]},{"osd":7,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.411,"15min":1.389},"min":{"1min":0.80400000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.246,"5min":2.246,"15min":2.246},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.6080000000000001,"15min":1.6180000000000001},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.1120000000000001,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.621}]}]},{"osd":1,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":561152,"data_stored":227687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.876,"15min":0.85799999999999998},"min":{"1min":0.42499999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.2959999999999998,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.97099999999999997,"15min":0.98399999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.75,"5min":1.99,"15min":1.99},"last":0.64700000000000002}]},{"osd":2,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.97499999999999998,"15min":0.98199999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.696,"5min":1.738,"15min":1.738},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.085,"5min":1.1220000000000001,"15min":1.1279999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.2849999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.2050000000000001,"15min":1.22},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.1190000000000002,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1779999999999999},"min":{"1min":0.61499999999999999,"5min":0.434,"15min":0.434},"max":{"1min":1.736,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.415}]},{"osd":4,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.2450000000000001,"15min":1.2390000000000001},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.8959999999999999,"5min":2.153,"15min":2.153},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.337,"15min":1.347},"min":{"1min":0.93600000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.9730000000000001,"5min":2.758,"15min":2.758},"last":1.4790000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.4299999999999999,"15min":1.4530000000000001},"min":{"1min":1.075,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.514,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.647},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.45,"15min":1.47},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.9279999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.46}]},{"osd":6,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4350000000000001,"15min":1.4390000000000001},"min":{"1min":0.94799999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.631},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.5640000000000001,"15min":1.5980000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.869,"5min":2.9630000000000001,"15min":2.9630000000000001},"last":1.5640000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:20:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.5680000000000001,"15min":1.595},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":2.0049999999999999,"5min":3.2749999999999999,"15min":3.2749999999999999},"last":1.369},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.587,"15min":1.601},"min":{"1min":1.0569999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.4710000000000001,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.581}]}]},{"osd":0,"up_from":14,"seq":60129542178,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28152,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343688,"statfs":{"total":96636764160,"available":96607936512,"internally_reserved":0,"allocated":1155072,"data_stored":818019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.85999999999999999,"15min":0.85499999999999998},"min":{"1min":0.46500000000000002,"5min":0.379,"15min":0.379},"max":{"1min":1.2729999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.376},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.93000000000000005,"15min":0.92000000000000004},"min":{"1min":0.61199999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.542,"5min":1.694,"15min":1.694},"last":0.30399999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.03,"15min":1.026},"min":{"1min":0.57399999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.615,"5min":1.635,"15min":1.635},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.077,"5min":1.014,"15min":1.004},"min":{"1min":0.77500000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.3919999999999999,"5min":1.802,"15min":1.802},"last":0.432}]},{"osd":3,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0820000000000001,"15min":1.0960000000000001},"min":{"1min":0.66500000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.302,"15min":1.3129999999999999},"min":{"1min":0.70499999999999996,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9670000000000001,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":0.57499999999999996}]},{"osd":4,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.3859999999999999,"15min":1.381},"min":{"1min":1.0640000000000001,"5min":0.874,"15min":0.874},"max":{"1min":2.4239999999999999,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.274,"15min":1.2709999999999999},"min":{"1min":1.0049999999999999,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.9039999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":0.39900000000000002}]},{"osd":5,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.444,"15min":1.4670000000000001},"min":{"1min":0.85699999999999998,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.7210000000000001,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.458,"5min":1.5469999999999999,"15min":1.5620000000000001},"min":{"1min":0.92400000000000004,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":2.335,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":0.52700000000000002}]},{"osd":6,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.5680000000000001,"15min":1.583},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.4399999999999999,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.393,"5min":1.4350000000000001,"15min":1.4419999999999999},"min":{"1min":0.94399999999999995,"5min":0.753,"15min":0.753},"max":{"1min":1.841,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":0.58399999999999996}]},{"osd":7,"last update":"Sat Mar 23 11:20:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.5589999999999999,"15min":1.5609999999999999},"min":{"1min":1.026,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.5419999999999998,"5min":2.617,"15min":2.617},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.508,"15min":1.5189999999999999},"min":{"1min":1.0269999999999999,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.0169999999999999,"5min":3.081,"15min":3.081},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:21:30.723 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T11:21:30.723 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T11:21:30.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.994+0000 7fce57200640 1 Processor -- start 2024-03-23T11:21:30.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.994+0000 7fce57200640 1 -- start start 2024-03-23T11:21:30.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.996+0000 7fce57200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fce5015cca0 0x7fce501611c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:30.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.996+0000 7fce57200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500b2490 0x7fce500d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:30.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.996+0000 7fce57200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce500cfe40 con 0x7fce5015c8c0 2024-03-23T11:21:30.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.996+0000 7fce57200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce500b2920 con 0x7fce5015cca0 2024-03-23T11:21:30.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.996+0000 7fce57200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce500cf5a0 con 0x7fce500b2490 2024-03-23T11:21:30.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.996+0000 7fce55400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fce5015cca0 0x7fce501611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:30.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.997+0000 7fce55400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fce5015cca0 0x7fce501611c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34634/0 (socket says 172.21.15.62:34634) 2024-03-23T11:21:30.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.997+0000 7fce55400640 1 -- 172.21.15.62:0/2830018282 learned_addr learned my addr 172.21.15.62:0/2830018282 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:30.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.997+0000 7fce56800640 1 --2- 172.21.15.62:0/2830018282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500b2490 0x7fce500d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:30.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.997+0000 7fce55400640 1 -- 172.21.15.62:0/2830018282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500b2490 msgr2=0x7fce500d0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:30.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.997+0000 7fce55400640 1 --2- 172.21.15.62:0/2830018282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500b2490 0x7fce500d0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:30.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.997+0000 7fce55400640 1 -- 172.21.15.62:0/2830018282 >> v1:172.21.15.120:6789/0 conn(0x7fce5015c8c0 legacy=0x7fce500ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:30.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.997+0000 7fce55400640 1 -- 172.21.15.62:0/2830018282 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce500cf880 con 0x7fce5015cca0 2024-03-23T11:21:30.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.999+0000 7fce55400640 1 --2- 172.21.15.62:0/2830018282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce5015cca0 0x7fce501611c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fce40009860 tx=0x7fce4002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715cac1530da3998 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:30.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.999+0000 7fce54a00640 1 -- 172.21.15.62:0/2830018282 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce40030840 con 0x7fce5015cca0 2024-03-23T11:21:30.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.999+0000 7fce54a00640 1 -- 172.21.15.62:0/2830018282 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce400309a0 con 0x7fce5015cca0 2024-03-23T11:21:30.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:30.999+0000 7fce54a00640 1 -- 172.21.15.62:0/2830018282 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce40030d30 con 0x7fce5015cca0 2024-03-23T11:21:31.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.000+0000 7fce57200640 1 -- 172.21.15.62:0/2830018282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce5015cca0 msgr2=0x7fce501611c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:31.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.000+0000 7fce57200640 1 --2- 172.21.15.62:0/2830018282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce5015cca0 0x7fce501611c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fce40009860 tx=0x7fce4002b980 comp rx=0 tx=0).stop 2024-03-23T11:21:31.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.000+0000 7fce57200640 1 -- 172.21.15.62:0/2830018282 shutdown_connections 2024-03-23T11:21:31.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.000+0000 7fce57200640 1 --2- 172.21.15.62:0/2830018282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500b2490 0x7fce500d0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.000+0000 7fce57200640 1 --2- 172.21.15.62:0/2830018282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce5015cca0 0x7fce501611c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.000+0000 7fce57200640 1 -- 172.21.15.62:0/2830018282 >> 172.21.15.62:0/2830018282 conn(0x7fce5009ba50 msgr2=0x7fce50089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:31.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.000+0000 7fce57200640 1 -- 172.21.15.62:0/2830018282 shutdown_connections 2024-03-23T11:21:31.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.000+0000 7fce57200640 1 -- 172.21.15.62:0/2830018282 wait complete. 2024-03-23T11:21:31.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.003+0000 7fce57200640 1 Processor -- start 2024-03-23T11:21:31.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.003+0000 7fce57200640 1 -- start start 2024-03-23T11:21:31.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.004+0000 7fce57200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce500b2490 0x7fce5015d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.004+0000 7fce57200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5015c8c0 0x7fce5014b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.004+0000 7fce57200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce5015cca0 0x7fce5014bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.004+0000 7fce57200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5014f880 con 0x7fce500b2490 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.004+0000 7fce57200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5015f5e0 con 0x7fce5015cca0 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.004+0000 7fce57200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5015d640 con 0x7fce5015c8c0 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.004+0000 7fce55e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce500b2490 0x7fce5015d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.005+0000 7fce55e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce500b2490 0x7fce5015d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34648/0 (socket says 172.21.15.62:34648) 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.005+0000 7fce55e00640 1 -- 172.21.15.62:0/450906923 learned_addr learned my addr 172.21.15.62:0/450906923 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.005+0000 7fce55e00640 1 -- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5015c8c0 msgr2=0x7fce5014b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.005+0000 7fce55e00640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5015c8c0 0x7fce5014b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.005+0000 7fce55e00640 1 -- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce5015cca0 msgr2=0x7fce5014bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.005+0000 7fce56800640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce5015cca0 0x7fce5014bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.005+0000 7fce55e00640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce5015cca0 0x7fce5014bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.005+0000 7fce55e00640 1 -- 172.21.15.62:0/450906923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce40009470 con 0x7fce500b2490 2024-03-23T11:21:31.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.005+0000 7fce56800640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce5015cca0 0x7fce5014bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:31.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.005+0000 7fce55e00640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce500b2490 0x7fce5015d100 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fce4400c450 tx=0x7fce4400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:31.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.006+0000 7fce4a000640 1 -- 172.21.15.62:0/450906923 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce440052d0 con 0x7fce500b2490 2024-03-23T11:21:31.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.006+0000 7fce4a000640 1 -- 172.21.15.62:0/450906923 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce44005450 con 0x7fce500b2490 2024-03-23T11:21:31.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.006+0000 7fce4a000640 1 -- 172.21.15.62:0/450906923 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44003880 con 0x7fce500b2490 2024-03-23T11:21:31.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.006+0000 7fce57200640 1 -- 172.21.15.62:0/450906923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce500c9c40 con 0x7fce500b2490 2024-03-23T11:21:31.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.006+0000 7fce57200640 1 -- 172.21.15.62:0/450906923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5015fb10 con 0x7fce500b2490 2024-03-23T11:21:31.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.007+0000 7fce57200640 1 -- 172.21.15.62:0/450906923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce500d2230 con 0x7fce500b2490 2024-03-23T11:21:31.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.008+0000 7fce4a000640 1 -- 172.21.15.62:0/450906923 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fce440039e0 con 0x7fce500b2490 2024-03-23T11:21:31.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.009+0000 7fce4a000640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce3406a7a0 0x7fce3406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:31.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.009+0000 7fce4a000640 1 -- 172.21.15.62:0/450906923 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce44088970 con 0x7fce500b2490 2024-03-23T11:21:31.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.010+0000 7fce55400640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce3406a7a0 0x7fce3406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:31.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.011+0000 7fce55400640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce3406a7a0 0x7fce3406cc60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fce40036000 tx=0x7fce4003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.014+0000 7fce4a000640 1 -- 172.21.15.62:0/450906923 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce44053d80 con 0x7fce500b2490 2024-03-23T11:21:31.183 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:21:31.183+0000 7ff6f8200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:21:31.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.284+0000 7fce57200640 1 -- 172.21.15.62:0/450906923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fce50066bb0 con 0x7fce500b2490 2024-03-23T11:21:31.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.293+0000 7fce4a000640 1 -- 172.21.15.62:0/450906923 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) v1 ==== 190+0+0 (secure 0 0 0) 0x7fce440573a0 con 0x7fce500b2490 2024-03-23T11:21:31.293 INFO:teuthology.orchestra.run.smithi062.stderr:updated caps for client.0 2024-03-23T11:21:31.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.296+0000 7fce57200640 1 -- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce3406a7a0 msgr2=0x7fce3406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:31.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.296+0000 7fce57200640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce3406a7a0 0x7fce3406cc60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fce40036000 tx=0x7fce4003b040 comp rx=0 tx=0).stop 2024-03-23T11:21:31.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.296+0000 7fce57200640 1 -- 172.21.15.62:0/450906923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce500b2490 msgr2=0x7fce5015d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:31.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.296+0000 7fce57200640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce500b2490 0x7fce5015d100 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fce4400c450 tx=0x7fce4400c910 comp rx=0 tx=0).stop 2024-03-23T11:21:31.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.296+0000 7fce57200640 1 -- 172.21.15.62:0/450906923 shutdown_connections 2024-03-23T11:21:31.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.296+0000 7fce57200640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce3406a7a0 0x7fce3406cc60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.296+0000 7fce57200640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce5015cca0 0x7fce5014bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.296+0000 7fce57200640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5015c8c0 0x7fce5014b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.296+0000 7fce57200640 1 --2- 172.21.15.62:0/450906923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce500b2490 0x7fce5015d100 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.296+0000 7fce57200640 1 -- 172.21.15.62:0/450906923 >> 172.21.15.62:0/450906923 conn(0x7fce5009ba50 msgr2=0x7fce5006c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:31.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.297+0000 7fce57200640 1 -- 172.21.15.62:0/450906923 shutdown_connections 2024-03-23T11:21:31.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.297+0000 7fce57200640 1 -- 172.21.15.62:0/450906923 wait complete. 2024-03-23T11:21:31.313 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:21:31.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.525+0000 7ff685200640 1 Processor -- start 2024-03-23T11:21:31.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.525+0000 7ff685200640 1 -- start start 2024-03-23T11:21:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.527+0000 7ff685200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 0x7ff68015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.527+0000 7ff685200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6800b2f30 0x7ff6800704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:31.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.527+0000 7ff685200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff680070a30 con 0x7ff680070090 2024-03-23T11:21:31.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.527+0000 7ff685200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6800cd140 con 0x7ff6800b2f30 2024-03-23T11:21:31.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.527+0000 7ff685200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6800cd3e0 con 0x7ff680143dd0 2024-03-23T11:21:31.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.527+0000 7ff67ea00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff680070090 0x7ff6800b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59610/0 (socket says 172.21.15.62:59610) 2024-03-23T11:21:31.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.527+0000 7ff67ea00640 1 -- 172.21.15.62:0/3640694576 learned_addr learned my addr 172.21.15.62:0/3640694576 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:31.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.527+0000 7ff67fe00640 1 --2- 172.21.15.62:0/3640694576 >> v2:172.21.15.62:3300/0 conn(0x7ff6800b2f30 0x7ff6800704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.527+0000 7ff67f400640 1 --2- 172.21.15.62:0/3640694576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 0x7ff68015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.528+0000 7ff67e000640 1 -- 172.21.15.62:0/3640694576 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599621890 0 0) 0x7ff680070a30 con 0x7ff680070090 2024-03-23T11:21:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.528+0000 7ff67e000640 1 -- 172.21.15.62:0/3640694576 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff664003520 con 0x7ff680070090 2024-03-23T11:21:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.528+0000 7ff67fe00640 1 -- 172.21.15.62:0/3640694576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 msgr2=0x7ff68015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.528+0000 7ff67fe00640 1 --2- 172.21.15.62:0/3640694576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 0x7ff68015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.528+0000 7ff67fe00640 1 -- 172.21.15.62:0/3640694576 >> v1:172.21.15.62:6789/0 conn(0x7ff680070090 legacy=0x7ff6800b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:31.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.528+0000 7ff67fe00640 1 -- 172.21.15.62:0/3640694576 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6800cd6c0 con 0x7ff6800b2f30 2024-03-23T11:21:31.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.528+0000 7ff67f400640 1 --2- 172.21.15.62:0/3640694576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 0x7ff68015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:21:31.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.529+0000 7ff67fe00640 1 --2- 172.21.15.62:0/3640694576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6800b2f30 0x7ff6800704f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff678009d90 tx=0x7ff67802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3410da4bacc77d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:31.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.530+0000 7ff67e000640 1 -- 172.21.15.62:0/3640694576 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67802cc60 con 0x7ff6800b2f30 2024-03-23T11:21:31.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.530+0000 7ff67e000640 1 -- 172.21.15.62:0/3640694576 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67802cdc0 con 0x7ff6800b2f30 2024-03-23T11:21:31.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.530+0000 7ff67e000640 1 -- 172.21.15.62:0/3640694576 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678034660 con 0x7ff6800b2f30 2024-03-23T11:21:31.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.530+0000 7ff685200640 1 -- 172.21.15.62:0/3640694576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6800b2f30 msgr2=0x7ff6800704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:31.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.530+0000 7ff685200640 1 --2- 172.21.15.62:0/3640694576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6800b2f30 0x7ff6800704f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff678009d90 tx=0x7ff67802beb0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.530+0000 7ff685200640 1 -- 172.21.15.62:0/3640694576 shutdown_connections 2024-03-23T11:21:31.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.530+0000 7ff685200640 1 --2- 172.21.15.62:0/3640694576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6800b2f30 0x7ff6800704f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.530+0000 7ff685200640 1 --2- 172.21.15.62:0/3640694576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 0x7ff68015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.530+0000 7ff685200640 1 -- 172.21.15.62:0/3640694576 >> 172.21.15.62:0/3640694576 conn(0x7ff680066050 msgr2=0x7ff680069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:31.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.531+0000 7ff685200640 1 -- 172.21.15.62:0/3640694576 shutdown_connections 2024-03-23T11:21:31.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.531+0000 7ff685200640 1 -- 172.21.15.62:0/3640694576 wait complete. 2024-03-23T11:21:31.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.532+0000 7ff685200640 1 Processor -- start 2024-03-23T11:21:31.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.533+0000 7ff685200640 1 -- start start 2024-03-23T11:21:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.533+0000 7ff685200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff680070090 0x7ff680138e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.533+0000 7ff685200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6800b2f30 0x7ff6800704f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff685200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 0x7ff680070a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff685200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68014c8b0 con 0x7ff6800b2f30 2024-03-23T11:21:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff685200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6801f31b0 con 0x7ff680070090 2024-03-23T11:21:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff685200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68014c5d0 con 0x7ff680143dd0 2024-03-23T11:21:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff67ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6800b2f30 0x7ff6800704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff67ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6800b2f30 0x7ff6800704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34672/0 (socket says 172.21.15.62:34672) 2024-03-23T11:21:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff67ea00640 1 -- 172.21.15.62:0/3866611615 learned_addr learned my addr 172.21.15.62:0/3866611615 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:31.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff67f400640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff680070090 0x7ff680138e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:31.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff67fe00640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 0x7ff680070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:31.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff67ea00640 1 -- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 msgr2=0x7ff680070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:31.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff67ea00640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 0x7ff680070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff67ea00640 1 -- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff680070090 msgr2=0x7ff680138e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:31.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff67ea00640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff680070090 0x7ff680138e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff67ea00640 1 -- 172.21.15.62:0/3866611615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff678009a70 con 0x7ff6800b2f30 2024-03-23T11:21:31.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.534+0000 7ff67fe00640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 0x7ff680070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:31.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.535+0000 7ff67ea00640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6800b2f30 0x7ff6800704f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff66800c500 tx=0x7ff66800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:31.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.535+0000 7ff677400640 1 -- 172.21.15.62:0/3866611615 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668005ce0 con 0x7ff6800b2f30 2024-03-23T11:21:31.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.535+0000 7ff677400640 1 -- 172.21.15.62:0/3866611615 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff668005e40 con 0x7ff6800b2f30 2024-03-23T11:21:31.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.535+0000 7ff677400640 1 -- 172.21.15.62:0/3866611615 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668007330 con 0x7ff6800b2f30 2024-03-23T11:21:31.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.535+0000 7ff685200640 1 -- 172.21.15.62:0/3866611615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6801f3450 con 0x7ff6800b2f30 2024-03-23T11:21:31.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.535+0000 7ff685200640 1 -- 172.21.15.62:0/3866611615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6801f3c50 con 0x7ff6800b2f30 2024-03-23T11:21:31.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.536+0000 7ff685200640 1 -- 172.21.15.62:0/3866611615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6801f3010 con 0x7ff6800b2f30 2024-03-23T11:21:31.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.537+0000 7ff677400640 1 -- 172.21.15.62:0/3866611615 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff668007490 con 0x7ff6800b2f30 2024-03-23T11:21:31.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.538+0000 7ff677400640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff65c06a7a0 0x7ff65c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:31.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.538+0000 7ff677400640 1 -- 172.21.15.62:0/3866611615 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff668088430 con 0x7ff6800b2f30 2024-03-23T11:21:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.539+0000 7ff67f400640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff65c06a7a0 0x7ff65c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:31.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.540+0000 7ff67f400640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff65c06a7a0 0x7ff65c06cc60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff670006fd0 tx=0x7ff670008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.543+0000 7ff677400640 1 -- 172.21.15.62:0/3866611615 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6801f3010 con 0x7ff6800b2f30 2024-03-23T11:21:31.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.779+0000 7ff685200640 1 -- 172.21.15.62:0/3866611615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6801f3010 con 0x7ff6800b2f30 2024-03-23T11:21:31.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.780+0000 7ff677400640 1 -- 172.21.15.62:0/3866611615 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4740 (secure 0 0 0) 0x7ff6801f3010 con 0x7ff6800b2f30 2024-03-23T11:21:31.780 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:31.780 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3787987198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3787987198},{"type":"v1","addr":"172.21.15.120:6835","nonce":3787987198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2749498408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2749498408},{"type":"v1","addr":"172.21.15.120:6837","nonce":2749498408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2325988870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2325988870},{"type":"v1","addr":"172.21.15.62:6833","nonce":2325988870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:21:25.687136+0000","modified":"2024-03-23T11:21:28.704192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6835/1159419089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1159419089},{"type":"v1","addr":"172.21.15.62:6835","nonce":1159419089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T11:21:31.780 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 -- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff65c06a7a0 msgr2=0x7ff65c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff65c06a7a0 0x7ff65c06cc60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff670006fd0 tx=0x7ff670008040 comp rx=0 tx=0).stop 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 -- 172.21.15.62:0/3866611615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6800b2f30 msgr2=0x7ff6800704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6800b2f30 0x7ff6800704f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff66800c500 tx=0x7ff66800c9c0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 -- 172.21.15.62:0/3866611615 shutdown_connections 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff65c06a7a0 0x7ff65c06cc60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6800b2f30 0x7ff6800704f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff680143dd0 0x7ff680070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 --2- 172.21.15.62:0/3866611615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff680070090 0x7ff680138e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 -- 172.21.15.62:0/3866611615 >> 172.21.15.62:0/3866611615 conn(0x7ff680066050 msgr2=0x7ff6800ae330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 -- 172.21.15.62:0/3866611615 shutdown_connections 2024-03-23T11:21:31.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:31.783+0000 7ff685200640 1 -- 172.21.15.62:0/3866611615 wait complete. 2024-03-23T11:21:32.201 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:21:25.687136+0000', 'modified': '2024-03-23T11:21:28.704192+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5652}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5652': {'gid': 5652, 'name': 'a', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.62:6835/1159419089', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6834', 'nonce': 1159419089}, {'type': 'v1', 'addr': '172.21.15.62:6835', 'nonce': 1159419089}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:21:32.202 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:21:32.202 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T11:21:32.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.427+0000 7fad4dc00640 1 Processor -- start 2024-03-23T11:21:32.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.428+0000 7fad4dc00640 1 -- start start 2024-03-23T11:21:32.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.429+0000 7fad4dc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fad480b15e0 0x7fad48161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.429+0000 7fad4dc00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad480ae0c0 con 0x7fad480b2f90 2024-03-23T11:21:32.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.429+0000 7fad4dc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad480b2740 con 0x7fad4806c300 2024-03-23T11:21:32.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.429+0000 7fad4dc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad481623b0 con 0x7fad480b15e0 2024-03-23T11:21:32.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.429+0000 7fad46a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fad480b2f90 0x7fad480b3370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59626/0 (socket says 172.21.15.62:59626) 2024-03-23T11:21:32.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.429+0000 7fad46a00640 1 -- 172.21.15.62:0/3477058020 learned_addr learned my addr 172.21.15.62:0/3477058020 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:32.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.430+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898659383 0 0) 0x7fad480ae0c0 con 0x7fad480b2f90 2024-03-23T11:21:32.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.430+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad2c003520 con 0x7fad480b2f90 2024-03-23T11:21:32.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.430+0000 7fad3fe00640 1 --2- 172.21.15.62:0/3477058020 >> v2:172.21.15.120:3300/0 conn(0x7fad480b15e0 0x7fad48161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.430+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2045916780 0 0) 0x7fad2c003520 con 0x7fad480b2f90 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.430+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 >> v2:172.21.15.120:3300/0 conn(0x7fad480b15e0 msgr2=0x7fad48161e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.430+0000 7fad46000640 1 --2- 172.21.15.62:0/3477058020 >> v2:172.21.15.120:3300/0 conn(0x7fad480b15e0 0x7fad48161e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.430+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 >> v1:172.21.15.120:6789/0 conn(0x7fad4806c300 legacy=0x7fad4806c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.430+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad480b1100 con 0x7fad480b2f90 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.430+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fad38001670 con 0x7fad480b2f90 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.430+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 >> v1:172.21.15.62:6789/0 conn(0x7fad480b2f90 legacy=0x7fad480b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad46000640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c003970 0x7fad2c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad46000640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c0047c0 0x7fad2c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad46000640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c005180 0x7fad2c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad481623b0 con 0x7fad2c005180 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c00c0b0 con 0x7fad2c003970 2024-03-23T11:21:32.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c00c270 con 0x7fad2c0047c0 2024-03-23T11:21:32.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad47400640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c005180 0x7fad2c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad46a00640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c003970 0x7fad2c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad3fe00640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c0047c0 0x7fad2c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad47400640 1 -- 172.21.15.62:0/3477058020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c0047c0 msgr2=0x7fad2c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad47400640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c0047c0 0x7fad2c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad47400640 1 -- 172.21.15.62:0/3477058020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c003970 msgr2=0x7fad2c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad47400640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c003970 0x7fad2c003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad47400640 1 -- 172.21.15.62:0/3477058020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c003520 con 0x7fad2c005180 2024-03-23T11:21:32.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad46a00640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c003970 0x7fad2c003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:32.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.431+0000 7fad3fe00640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c0047c0 0x7fad2c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:32.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.433+0000 7fad47400640 1 -- 172.21.15.62:0/3477058020 reap_dead start 2024-03-23T11:21:32.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.433+0000 7fad47400640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c005180 0x7fad2c00b980 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fad40004480 tx=0x7fad2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7a6df701a9df52 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:32.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.433+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40006d50 con 0x7fad2c005180 2024-03-23T11:21:32.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.433+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad40007020 con 0x7fad2c005180 2024-03-23T11:21:32.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.433+0000 7fad46000640 1 -- 172.21.15.62:0/3477058020 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad400073d0 con 0x7fad2c005180 2024-03-23T11:21:32.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.434+0000 7fad4dc00640 1 -- 172.21.15.62:0/3477058020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c005180 msgr2=0x7fad2c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.434+0000 7fad4dc00640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c005180 0x7fad2c00b980 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fad40004480 tx=0x7fad2c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.434+0000 7fad4dc00640 1 -- 172.21.15.62:0/3477058020 shutdown_connections 2024-03-23T11:21:32.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.434+0000 7fad4dc00640 1 --2- 172.21.15.62:0/3477058020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c005180 0x7fad2c00b980 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.434+0000 7fad4dc00640 1 -- 172.21.15.62:0/3477058020 >> 172.21.15.62:0/3477058020 conn(0x7fad480b02e0 msgr2=0x7fad480b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:32.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.434+0000 7fad4dc00640 1 -- 172.21.15.62:0/3477058020 shutdown_connections 2024-03-23T11:21:32.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.434+0000 7fad4dc00640 1 -- 172.21.15.62:0/3477058020 wait complete. 2024-03-23T11:21:32.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.436+0000 7fad4dc00640 1 Processor -- start 2024-03-23T11:21:32.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.436+0000 7fad4dc00640 1 -- start start 2024-03-23T11:21:32.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.437+0000 7fad4dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c005180 0x7fad481e8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.437+0000 7fad4dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad481e7260 0x7fad48146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.437+0000 7fad4dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad481eb050 0x7fad481eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.437+0000 7fad4dc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad48146d50 con 0x7fad481e7260 2024-03-23T11:21:32.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.437+0000 7fad4dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad48146880 con 0x7fad2c005180 2024-03-23T11:21:32.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.437+0000 7fad4dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad481eba10 con 0x7fad481eb050 2024-03-23T11:21:32.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.437+0000 7fad3fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad481e7260 0x7fad48146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.437+0000 7fad3fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad481e7260 0x7fad48146340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34684/0 (socket says 172.21.15.62:34684) 2024-03-23T11:21:32.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.437+0000 7fad3fe00640 1 -- 172.21.15.62:0/2633047573 learned_addr learned my addr 172.21.15.62:0/2633047573 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:32.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.438+0000 7fad46a00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c005180 0x7fad481e8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.438+0000 7fad47400640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad481eb050 0x7fad481eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.438+0000 7fad3fe00640 1 -- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad481eb050 msgr2=0x7fad481eb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.438+0000 7fad3fe00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad481eb050 0x7fad481eb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.438+0000 7fad3fe00640 1 -- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c005180 msgr2=0x7fad481e8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.438+0000 7fad3fe00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c005180 0x7fad481e8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.438+0000 7fad3fe00640 1 -- 172.21.15.62:0/2633047573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad481e8270 con 0x7fad481e7260 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.438+0000 7fad47400640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad481eb050 0x7fad481eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.438+0000 7fad3fe00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad481e7260 0x7fad48146340 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fad3000e7e0 tx=0x7fad3000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.438+0000 7fad3ea00640 1 -- 172.21.15.62:0/2633047573 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3000c670 con 0x7fad481e7260 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.439+0000 7fad3ea00640 1 -- 172.21.15.62:0/2633047573 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3000c7d0 con 0x7fad481e7260 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.439+0000 7fad4dc00640 1 -- 172.21.15.62:0/2633047573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad481e79d0 con 0x7fad481e7260 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.439+0000 7fad3ea00640 1 -- 172.21.15.62:0/2633047573 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3000cb40 con 0x7fad481e7260 2024-03-23T11:21:32.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.439+0000 7fad4dc00640 1 -- 172.21.15.62:0/2633047573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad481e7f90 con 0x7fad481e7260 2024-03-23T11:21:32.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.441+0000 7fad3ea00640 1 -- 172.21.15.62:0/2633047573 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fad3000cca0 con 0x7fad481e7260 2024-03-23T11:21:32.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.442+0000 7fad3ea00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad2406a7a0 0x7fad2406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.442+0000 7fad3ea00640 1 -- 172.21.15.62:0/2633047573 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad30088b10 con 0x7fad481e7260 2024-03-23T11:21:32.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.442+0000 7fad507bd740 1 Processor -- start 2024-03-23T11:21:32.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.442+0000 7fad46a00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad2406a7a0 0x7fad2406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.442+0000 7fad507bd740 1 -- start start 2024-03-23T11:21:32.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.443+0000 7fad507bd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 0x55d0536769c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.443+0000 7fad507bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d053676f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.443+0000 7fad507bd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d0536718b0 0x55d0536a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.443+0000 7fad507bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0535e2110 con 0x55d053675410 2024-03-23T11:21:32.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.443+0000 7fad507bd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0535df8a0 con 0x55d0536bf390 2024-03-23T11:21:32.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.443+0000 7fad507bd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d0535db190 con 0x55d0536718b0 2024-03-23T11:21:32.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad3fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d053676f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad3fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d053676f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34696/0 (socket says 172.21.15.62:34696) 2024-03-23T11:21:32.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad3fe00640 1 -- 172.21.15.62:0/25989065 learned_addr learned my addr 172.21.15.62:0/25989065 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:32.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad46a00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad2406a7a0 0x7fad2406cc60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fad38001200 tx=0x7fad38009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:32.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad46a00640 1 --2- 172.21.15.62:0/25989065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 0x55d0536769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad3fe00640 1 -- 172.21.15.62:0/25989065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d0536718b0 msgr2=0x55d0536a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad3fe00640 1 --2- 172.21.15.62:0/25989065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d0536718b0 0x55d0536a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad3fe00640 1 -- 172.21.15.62:0/25989065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 msgr2=0x55d0536769c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad3fe00640 1 --2- 172.21.15.62:0/25989065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 0x55d0536769c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad3fe00640 1 -- 172.21.15.62:0/25989065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0535a6230 con 0x55d053675410 2024-03-23T11:21:32.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad3fe00640 1 --2- 172.21.15.62:0/25989065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d053676f00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fad3000cfd0 tx=0x7fad300582e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6164db0b433f9e53 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:32.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad37400640 1 -- 172.21.15.62:0/25989065 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3006f070 con 0x55d053675410 2024-03-23T11:21:32.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad37400640 1 -- 172.21.15.62:0/25989065 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3005ba10 con 0x55d053675410 2024-03-23T11:21:32.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.444+0000 7fad37400640 1 -- 172.21.15.62:0/25989065 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3005ffd0 con 0x55d053675410 2024-03-23T11:21:32.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.445+0000 7fad507bd740 1 -- 172.21.15.62:0/25989065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 msgr2=0x55d053676f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.445+0000 7fad507bd740 1 --2- 172.21.15.62:0/25989065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d053676f00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fad3000cfd0 tx=0x7fad300582e0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.445+0000 7fad507bd740 1 -- 172.21.15.62:0/25989065 shutdown_connections 2024-03-23T11:21:32.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.445+0000 7fad507bd740 1 --2- 172.21.15.62:0/25989065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d053676f00 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.445+0000 7fad507bd740 1 --2- 172.21.15.62:0/25989065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d0536718b0 0x55d0536a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.445+0000 7fad507bd740 1 --2- 172.21.15.62:0/25989065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 0x55d0536769c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.445+0000 7fad507bd740 1 -- 172.21.15.62:0/25989065 >> 172.21.15.62:0/25989065 conn(0x55d053673b20 msgr2=0x55d0536bc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.445+0000 7fad507bd740 1 -- 172.21.15.62:0/25989065 shutdown_connections 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.445+0000 7fad507bd740 1 -- 172.21.15.62:0/25989065 wait complete. 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.446+0000 7fad507bd740 1 -- start start 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.446+0000 7fad507bd740 1 Processor -- start 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.447+0000 7fad507bd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d053675410 0x55d0536c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.447+0000 7fad507bd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 0x55d0536a5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad507bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d05369f580 0x55d05369e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad507bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05357c8c0 con 0x55d05369f580 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad507bd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05357c560 con 0x55d0536bf390 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad507bd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d053342f20 con 0x55d053675410 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad507bd740 10 client.0 _ll_register_callbacks cb 0x55d0535d4800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad47400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d05369f580 0x55d05369e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad47400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d05369f580 0x55d05369e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34706/0 (socket says 172.21.15.62:34706) 2024-03-23T11:21:32.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad47400640 1 -- 172.21.15.62:0/117781006 learned_addr learned my addr 172.21.15.62:0/117781006 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad3fe00640 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 0x55d0536a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad46a00640 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d053675410 0x55d0536c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad47400640 1 -- 172.21.15.62:0/117781006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d053675410 msgr2=0x55d0536c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad47400640 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d053675410 0x55d0536c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad47400640 1 -- 172.21.15.62:0/117781006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 msgr2=0x55d0536a5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad47400640 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 0x55d0536a5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.448+0000 7fad47400640 1 -- 172.21.15.62:0/117781006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad481e7f90 con 0x55d05369f580 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.449+0000 7fad47400640 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d05369f580 0x55d05369e130 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fad400045a0 tx=0x7fad40003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed8438bd516fb6b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.449+0000 7fad2be00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.449+0000 7fad2be00640 1 -- 172.21.15.62:0/117781006 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4003e070 con 0x55d05369f580 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.449+0000 7fad2be00640 1 -- 172.21.15.62:0/117781006 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad40007650 con 0x55d05369f580 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.449+0000 7fad507bd740 1 -- 172.21.15.62:0/117781006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d053605280 con 0x55d05369f580 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.449+0000 7fad2be00640 1 -- 172.21.15.62:0/117781006 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad40007df0 con 0x55d05369f580 2024-03-23T11:21:32.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.449+0000 7fad507bd740 10 client.5790 fetch_fsmap learned FSMap version 25 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.449+0000 7fad507bd740 1 -- 172.21.15.62:0/117781006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55d0535d9810 con 0x55d05369f580 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.450+0000 7fad2be00640 1 -- 172.21.15.62:0/117781006 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d053605280 con 0x55d05369f580 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.450+0000 7fad2be00640 1 -- 172.21.15.62:0/117781006 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fad40007810 con 0x55d05369f580 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.450+0000 7fad507bd740 10 client.5790 fetch_fsmap finished waiting for FSMap version 25 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.450+0000 7fad507bd740 10 client.5790 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.450+0000 7fad507bd740 20 client.5790 populate_metadata read hostname 'smithi062' 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.450+0000 7fad507bd740 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55d0536872c0 0x55d053684cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.450+0000 7fad507bd740 4 client.5790 mds_command: new command op to 5652 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.450+0000 7fad507bd740 1 -- 172.21.15.62:0/117781006 --> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0535cd9d0 con 0x55d0536872c0 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.450+0000 7fad46a00640 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55d0536872c0 0x55d053684cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.451+0000 7fad46a00640 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55d0536872c0 0x55d053684cd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=610222a3a2595983 server_cookie=cb0602d1c33165cc in_seq=0 out_seq=0 2024-03-23T11:21:32.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.451+0000 7fad2be00640 10 client.5790 ms_handle_connect on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:32.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.451+0000 7fad2be00640 1 -- 172.21.15.62:0/117781006 <== mds.0 v2:172.21.15.62:6834/1159419089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55d0535cd9d0 con 0x55d0536872c0 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.451+0000 7fad2be00640 10 client.5790 handle_command_reply: tid=0 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 1 client.5790 shutdown 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 1 -- 172.21.15.62:0/117781006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d05369f580 msgr2=0x55d05369e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d05369f580 0x55d05369e130 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fad400045a0 tx=0x7fad40003a70 comp rx=0 tx=0).stop 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 1 -- 172.21.15.62:0/117781006 shutdown_connections 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55d0536872c0 0x55d053684cd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d05369f580 0x55d05369e130 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 0x55d0536a5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 1 --2- 172.21.15.62:0/117781006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d053675410 0x55d0536c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad2be00640 0 client.5790 ms_handle_reset on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 1 -- 172.21.15.62:0/117781006 >> 172.21.15.62:0/117781006 conn(0x55d0536718b0 msgr2=0x55d053675ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 1 -- 172.21.15.62:0/117781006 shutdown_connections 2024-03-23T11:21:32.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 1 -- 172.21.15.62:0/117781006 wait complete. 2024-03-23T11:21:32.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.452+0000 7fad507bd740 20 client.5790 trim_cache size 0 max 16384 2024-03-23T11:21:32.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.468+0000 7fad507bd740 1 Processor -- start 2024-03-23T11:21:32.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.468+0000 7fad507bd740 1 -- start start 2024-03-23T11:21:32.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad507bd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d053675410 0x55d053697dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad507bd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d05369f580 0x55d0536776c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad507bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0536bf390 0x55d0536c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad507bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0536983d0 con 0x55d0536bf390 2024-03-23T11:21:32.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad507bd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0536750d0 con 0x55d05369f580 2024-03-23T11:21:32.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad507bd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d0536746c0 con 0x55d053675410 2024-03-23T11:21:32.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad47400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0536bf390 0x55d0536c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad47400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0536bf390 0x55d0536c3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34722/0 (socket says 172.21.15.62:34722) 2024-03-23T11:21:32.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad47400640 1 -- 172.21.15.62:0/1843773105 learned_addr learned my addr 172.21.15.62:0/1843773105 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad3fe00640 1 --2- 172.21.15.62:0/1843773105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d05369f580 0x55d0536776c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad47400640 1 -- 172.21.15.62:0/1843773105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d053675410 msgr2=0x55d053697dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad47400640 1 --2- 172.21.15.62:0/1843773105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d053675410 0x55d053697dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.469+0000 7fad47400640 1 -- 172.21.15.62:0/1843773105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d05369f580 msgr2=0x55d0536776c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.470+0000 7fad47400640 1 --2- 172.21.15.62:0/1843773105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d05369f580 0x55d0536776c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.470+0000 7fad47400640 1 -- 172.21.15.62:0/1843773105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d05357bad0 con 0x55d0536bf390 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.470+0000 7fad47400640 1 --2- 172.21.15.62:0/1843773105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0536bf390 0x55d0536c3ac0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fad40004570 tx=0x7fad4003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2bc2c99f53b7da3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.470+0000 7fad37e00640 1 -- 172.21.15.62:0/1843773105 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40007850 con 0x55d0536bf390 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.470+0000 7fad37e00640 1 -- 172.21.15.62:0/1843773105 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad40043650 con 0x55d0536bf390 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.470+0000 7fad37e00640 1 -- 172.21.15.62:0/1843773105 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad400439c0 con 0x55d0536bf390 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.471+0000 7fad507bd740 1 -- 172.21.15.62:0/1843773105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0536bf390 msgr2=0x55d0536c3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.471+0000 7fad507bd740 1 --2- 172.21.15.62:0/1843773105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0536bf390 0x55d0536c3ac0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fad40004570 tx=0x7fad4003e000 comp rx=0 tx=0).stop 2024-03-23T11:21:32.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.471+0000 7fad507bd740 1 -- 172.21.15.62:0/1843773105 shutdown_connections 2024-03-23T11:21:32.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.471+0000 7fad507bd740 1 --2- 172.21.15.62:0/1843773105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d0536bf390 0x55d0536c3ac0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.471+0000 7fad507bd740 1 --2- 172.21.15.62:0/1843773105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d05369f580 0x55d0536776c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.471+0000 7fad507bd740 1 --2- 172.21.15.62:0/1843773105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d053675410 0x55d053697dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.471+0000 7fad507bd740 1 -- 172.21.15.62:0/1843773105 >> 172.21.15.62:0/1843773105 conn(0x55d0536718b0 msgr2=0x55d0536a0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:32.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.471+0000 7fad507bd740 1 -- 172.21.15.62:0/1843773105 shutdown_connections 2024-03-23T11:21:32.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.471+0000 7fad507bd740 1 -- 172.21.15.62:0/1843773105 wait complete. 2024-03-23T11:21:32.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.471+0000 7fad507bd740 1 -- start start 2024-03-23T11:21:32.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.472+0000 7fad507bd740 1 Processor -- start 2024-03-23T11:21:32.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad507bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d0537082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad507bd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d05369f580 0x55d053671d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad507bd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 0x55d053697720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad507bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0536bc320 con 0x55d053675410 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad507bd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d053672470 con 0x55d0536bf390 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad507bd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d053722330 con 0x55d05369f580 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad507bd740 10 client.0 _ll_register_callbacks cb 0x7fad2c0036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad46a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d0537082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad46a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d0537082e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34736/0 (socket says 172.21.15.62:34736) 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad46a00640 1 -- 172.21.15.62:0/2658659997 learned_addr learned my addr 172.21.15.62:0/2658659997 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad46a00640 1 -- 172.21.15.62:0/2658659997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d05369f580 msgr2=0x55d053671d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad46a00640 1 --2- 172.21.15.62:0/2658659997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d05369f580 0x55d053671d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad46a00640 1 -- 172.21.15.62:0/2658659997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 msgr2=0x55d053697720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad46a00640 1 --2- 172.21.15.62:0/2658659997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 0x55d053697720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad46a00640 1 -- 172.21.15.62:0/2658659997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d0535d9810 con 0x55d053675410 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad46a00640 1 --2- 172.21.15.62:0/2658659997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d0537082e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x55d053677410 tx=0x7fad38025d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97af3b5c0409a3ba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.473+0000 7fad2be00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad2be00640 1 -- 172.21.15.62:0/2658659997 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad380250b0 con 0x55d053675410 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad2be00640 1 -- 172.21.15.62:0/2658659997 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad3801b040 con 0x55d053675410 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad507bd740 1 -- 172.21.15.62:0/2658659997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad3005ba10 con 0x55d053675410 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad2be00640 1 -- 172.21.15.62:0/2658659997 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad38025210 con 0x55d053675410 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad507bd740 10 client.5796 fetch_fsmap learned FSMap version 25 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad507bd740 1 -- 172.21.15.62:0/2658659997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55d0535a0fe0 con 0x55d053675410 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad2be00640 1 -- 172.21.15.62:0/2658659997 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3005ba10 con 0x55d053675410 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad2be00640 1 -- 172.21.15.62:0/2658659997 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fad38012070 con 0x55d053675410 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad507bd740 10 client.5796 fetch_fsmap finished waiting for FSMap version 25 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad507bd740 10 client.5796 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:21:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad507bd740 20 client.5796 populate_metadata read hostname 'smithi062' 2024-03-23T11:21:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad507bd740 1 --2- 172.21.15.62:0/2658659997 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55d0536d5b60 0x55d0536d5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad507bd740 4 client.5796 mds_command: new command op to 5652 tid=0[{"prefix": "status"}] 2024-03-23T11:21:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.474+0000 7fad507bd740 1 -- 172.21.15.62:0/2658659997 --> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d0532cf3c0 con 0x55d0536d5b60 2024-03-23T11:21:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.475+0000 7fad3fe00640 1 --2- 172.21.15.62:0/2658659997 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55d0536d5b60 0x55d0536d5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.475+0000 7fad3fe00640 1 --2- 172.21.15.62:0/2658659997 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55d0536d5b60 0x55d0536d5f40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e8a6719a6c74eb server_cookie=b2bddcaf52d25dc4 in_seq=0 out_seq=0 2024-03-23T11:21:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.475+0000 7fad2be00640 10 client.5796 ms_handle_connect on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.475+0000 7fad2be00640 1 -- 172.21.15.62:0/2658659997 <== mds.0 v2:172.21.15.62:6834/1159419089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55d0532cf3c0 con 0x55d0536d5b60 2024-03-23T11:21:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.475+0000 7fad2be00640 10 client.5796 handle_command_reply: tid=0 2024-03-23T11:21:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.475+0000 7fad507bd740 1 client.5796 shutdown 2024-03-23T11:21:32.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad507bd740 1 -- 172.21.15.62:0/2658659997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 msgr2=0x55d0537082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad507bd740 1 --2- 172.21.15.62:0/2658659997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d0537082e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x55d053677410 tx=0x7fad38025d20 comp rx=0 tx=0).stop 2024-03-23T11:21:32.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad507bd740 1 -- 172.21.15.62:0/2658659997 shutdown_connections 2024-03-23T11:21:32.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad507bd740 1 --2- 172.21.15.62:0/2658659997 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55d0536d5b60 0x55d0536d5f40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad507bd740 1 --2- 172.21.15.62:0/2658659997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d0536bf390 0x55d053697720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad507bd740 1 --2- 172.21.15.62:0/2658659997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d05369f580 0x55d053671d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad507bd740 1 --2- 172.21.15.62:0/2658659997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d053675410 0x55d0537082e0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad2be00640 0 client.5796 ms_handle_reset on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:32.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad507bd740 1 -- 172.21.15.62:0/2658659997 >> 172.21.15.62:0/2658659997 conn(0x55d0536718b0 msgr2=0x55d053675d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:32.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad507bd740 1 -- 172.21.15.62:0/2658659997 shutdown_connections 2024-03-23T11:21:32.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad507bd740 1 -- 172.21.15.62:0/2658659997 wait complete. 2024-03-23T11:21:32.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.476+0000 7fad507bd740 20 client.5796 trim_cache size 0 max 16384 2024-03-23T11:21:32.479 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:21:32.479 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:21:32.479 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:21:32.479 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 5652, 2024-03-23T11:21:32.480 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:21:32.480 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:21:32.480 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:21:32.480 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 6.7854874819999997, 2024-03-23T11:21:32.480 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 25, 2024-03-23T11:21:32.480 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 48, 2024-03-23T11:21:32.480 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 48, 2024-03-23T11:21:32.480 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 12.164179799999999 2024-03-23T11:21:32.480 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:21:32.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 -- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad2406a7a0 msgr2=0x7fad2406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad2406a7a0 0x7fad2406cc60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fad38001200 tx=0x7fad38009040 comp rx=0 tx=0).stop 2024-03-23T11:21:32.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 -- 172.21.15.62:0/2633047573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad481e7260 msgr2=0x7fad48146340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad481e7260 0x7fad48146340 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fad3000e7e0 tx=0x7fad3000eca0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 -- 172.21.15.62:0/2633047573 shutdown_connections 2024-03-23T11:21:32.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad2406a7a0 0x7fad2406cc60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad481e7260 0x7fad48146340 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad481eb050 0x7fad481eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 --2- 172.21.15.62:0/2633047573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c005180 0x7fad481e8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 -- 172.21.15.62:0/2633047573 >> 172.21.15.62:0/2633047573 conn(0x7fad480b02e0 msgr2=0x7fad480b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:32.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 -- 172.21.15.62:0/2633047573 shutdown_connections 2024-03-23T11:21:32.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.477+0000 7fad4dc00640 1 -- 172.21.15.62:0/2633047573 wait complete. 2024-03-23T11:21:32.495 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:21:32.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.711+0000 7f190be00640 1 Processor -- start 2024-03-23T11:21:32.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.711+0000 7f190be00640 1 -- start start 2024-03-23T11:21:32.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.713+0000 7f190be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f190c15fcf0 0x7f190c0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.713+0000 7f190be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f190c0b6dc0 0x7f190c1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.713+0000 7f190be00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190c0d3540 con 0x7f190c160420 2024-03-23T11:21:32.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.713+0000 7f190be00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190c160df0 con 0x7f190c15fcf0 2024-03-23T11:21:32.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.713+0000 7f190be00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0ce510 con 0x7f190c0b6dc0 2024-03-23T11:21:32.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.713+0000 7f1903e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f190c160420 0x7f190c0d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59640/0 (socket says 172.21.15.62:59640) 2024-03-23T11:21:32.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.713+0000 7f1903e00640 1 -- 172.21.15.62:0/1879038631 learned_addr learned my addr 172.21.15.62:0/1879038631 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:32.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190aa00640 1 --2- 172.21.15.62:0/1879038631 >> v2:172.21.15.120:3300/0 conn(0x7f190c15fcf0 0x7f190c0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190b400640 1 --2- 172.21.15.62:0/1879038631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f190c0b6dc0 0x7f190c1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199677534 0 0) 0x7f190c0d3540 con 0x7f190c160420 2024-03-23T11:21:32.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f0003520 con 0x7f190c160420 2024-03-23T11:21:32.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 203256805 0 0) 0x7f18f0003520 con 0x7f190c160420 2024-03-23T11:21:32.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f190c0b6dc0 msgr2=0x7f190c1608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 --2- 172.21.15.62:0/1879038631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f190c0b6dc0 0x7f190c1608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 >> v2:172.21.15.120:3300/0 conn(0x7f190c15fcf0 msgr2=0x7f190c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 --2- 172.21.15.62:0/1879038631 >> v2:172.21.15.120:3300/0 conn(0x7f190c15fcf0 0x7f190c0d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c0cdc70 con 0x7f190c160420 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f18f4001670 con 0x7f190c160420 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 >> v1:172.21.15.62:6789/0 conn(0x7f190c160420 legacy=0x7f190c0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 --2- 172.21.15.62:0/1879038631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 0x7f18f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 --2- 172.21.15.62:0/1879038631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18f00047c0 0x7f18f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 --2- 172.21.15.62:0/1879038631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f190c0b6dc0 0x7f18f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c160df0 con 0x7f18f0003970 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0ce510 con 0x7f18f00047c0 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.714+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f000ba00 con 0x7f190c0b6dc0 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.715+0000 7f190aa00640 1 --2- 172.21.15.62:0/1879038631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 0x7f18f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.715+0000 7f190aa00640 1 -- 172.21.15.62:0/1879038631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f190c0b6dc0 msgr2=0x7f18f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.715+0000 7f190aa00640 1 --2- 172.21.15.62:0/1879038631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f190c0b6dc0 0x7f18f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.715+0000 7f190aa00640 1 -- 172.21.15.62:0/1879038631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18f00047c0 msgr2=0x7f18f0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:32.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.715+0000 7f190aa00640 1 --2- 172.21.15.62:0/1879038631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18f00047c0 0x7f18f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.715+0000 7f190aa00640 1 -- 172.21.15.62:0/1879038631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f0003520 con 0x7f18f0003970 2024-03-23T11:21:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.716+0000 7f190aa00640 1 --2- 172.21.15.62:0/1879038631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 0x7f18f0003d90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009ac0 tx=0x7f18f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4687c43fdcac87cc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.717+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc02c710 con 0x7f18f0003970 2024-03-23T11:21:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.717+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc02c870 con 0x7f18f0003970 2024-03-23T11:21:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.717+0000 7f190a000640 1 -- 172.21.15.62:0/1879038631 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc02cc00 con 0x7f18f0003970 2024-03-23T11:21:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.717+0000 7f190be00640 1 -- 172.21.15.62:0/1879038631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 msgr2=0x7f18f0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.717+0000 7f190be00640 1 --2- 172.21.15.62:0/1879038631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 0x7f18f0003d90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009ac0 tx=0x7f18f00057f0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.717+0000 7f190b400640 1 -- 172.21.15.62:0/1879038631 reap_dead start 2024-03-23T11:21:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.717+0000 7f190be00640 1 -- 172.21.15.62:0/1879038631 shutdown_connections 2024-03-23T11:21:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.718+0000 7f190be00640 1 -- 172.21.15.62:0/1879038631 >> 172.21.15.62:0/1879038631 conn(0x7f190c089f60 msgr2=0x7f190c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.718+0000 7f190be00640 1 -- 172.21.15.62:0/1879038631 shutdown_connections 2024-03-23T11:21:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.718+0000 7f190be00640 1 -- 172.21.15.62:0/1879038631 wait complete. 2024-03-23T11:21:32.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.719+0000 7f190be00640 1 Processor -- start 2024-03-23T11:21:32.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.720+0000 7f190be00640 1 -- start start 2024-03-23T11:21:32.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.720+0000 7f190be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 0x7f190c09f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.720+0000 7f190be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f190c0c8fc0 0x7f190c15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.720+0000 7f190be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f190c09cce0 0x7f190c09d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.720+0000 7f190be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c09d6a0 con 0x7f18f0003970 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.721+0000 7f190be00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0c65c0 con 0x7f190c09cce0 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.721+0000 7f190be00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c06cc10 con 0x7f190c0c8fc0 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.721+0000 7f190aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 0x7f190c09f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.721+0000 7f190aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 0x7f190c09f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34756/0 (socket says 172.21.15.62:34756) 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.721+0000 7f190aa00640 1 -- 172.21.15.62:0/2082786649 learned_addr learned my addr 172.21.15.62:0/2082786649 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.721+0000 7f190aa00640 1 -- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f190c0c8fc0 msgr2=0x7f190c15bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.721+0000 7f190aa00640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f190c0c8fc0 0x7f190c15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.721+0000 7f190aa00640 1 -- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f190c09cce0 msgr2=0x7f190c09d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.721+0000 7f190aa00640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f190c09cce0 0x7f190c09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.721+0000 7f190aa00640 1 -- 172.21.15.62:0/2082786649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c09ff50 con 0x7f18f0003970 2024-03-23T11:21:32.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.721+0000 7f190aa00640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 0x7f190c09f850 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f18fc037040 tx=0x7f18fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:32.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.722+0000 7f1903400640 1 -- 172.21.15.62:0/2082786649 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc02de90 con 0x7f18f0003970 2024-03-23T11:21:32.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.722+0000 7f1903400640 1 -- 172.21.15.62:0/2082786649 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc003610 con 0x7f18f0003970 2024-03-23T11:21:32.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.722+0000 7f1903400640 1 -- 172.21.15.62:0/2082786649 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc0039c0 con 0x7f18f0003970 2024-03-23T11:21:32.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.722+0000 7f190be00640 1 -- 172.21.15.62:0/2082786649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190c0a0230 con 0x7f18f0003970 2024-03-23T11:21:32.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.722+0000 7f190be00640 1 -- 172.21.15.62:0/2082786649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190c0c6250 con 0x7f18f0003970 2024-03-23T11:21:32.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.723+0000 7f190be00640 1 -- 172.21.15.62:0/2082786649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190c066bb0 con 0x7f18f0003970 2024-03-23T11:21:32.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.724+0000 7f1903400640 1 -- 172.21.15.62:0/2082786649 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f18fc003b20 con 0x7f18f0003970 2024-03-23T11:21:32.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.725+0000 7f1903400640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f18e806a7d0 0x7f18e806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:32.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.725+0000 7f1903400640 1 -- 172.21.15.62:0/2082786649 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18fc0aabb0 con 0x7f18f0003970 2024-03-23T11:21:32.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.726+0000 7f1903e00640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f18e806a7d0 0x7f18e806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:32.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.727+0000 7f1903e00640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f18e806a7d0 0x7f18e806cc90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f190c0cc1f0 tx=0x7f18f40016f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.729+0000 7f1903400640 1 -- 172.21.15.62:0/2082786649 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18fc0760e0 con 0x7f18f0003970 2024-03-23T11:21:32.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.958+0000 7f190be00640 1 -- 172.21.15.62:0/2082786649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f190c0cea40 con 0x7f18f0003970 2024-03-23T11:21:32.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.958+0000 7f1903400640 1 -- 172.21.15.62:0/2082786649 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7f18fc079700 con 0x7f18f0003970 2024-03-23T11:21:32.958 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:21:32.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 -- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f18e806a7d0 msgr2=0x7f18e806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f18e806a7d0 0x7f18e806cc90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f190c0cc1f0 tx=0x7f18f40016f0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 -- 172.21.15.62:0/2082786649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 msgr2=0x7f190c09f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:32.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 0x7f190c09f850 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f18fc037040 tx=0x7f18fc004730 comp rx=0 tx=0).stop 2024-03-23T11:21:32.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 -- 172.21.15.62:0/2082786649 shutdown_connections 2024-03-23T11:21:32.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f18e806a7d0 0x7f18e806cc90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f190c09cce0 0x7f190c09d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f190c0c8fc0 0x7f190c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 --2- 172.21.15.62:0/2082786649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f0003970 0x7f190c09f850 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:32.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 -- 172.21.15.62:0/2082786649 >> 172.21.15.62:0/2082786649 conn(0x7f190c089f60 msgr2=0x7f190c06b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:32.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 -- 172.21.15.62:0/2082786649 shutdown_connections 2024-03-23T11:21:32.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:32.961+0000 7f190be00640 1 -- 172.21.15.62:0/2082786649 wait complete. 2024-03-23T11:21:32.979 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:21:32.979 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:21:32.979 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:21:32.979 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:21:32.979 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:21:32.979 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:21:32.979 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:21:32.979 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:21:32.979 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:21:32.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:32.980 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:21:33.029 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:21:33.029 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:21:33.029 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:21:33.029 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:21:33.029 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:21:33.030 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42464sec preferred_lft 42464sec 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:21:33.031 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:21:33.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:33.032 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:21:33.121 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:21:33.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:33.126 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:21:33.181 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:21:33.181 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:21:33.181 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:21:33.181 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:21:33.181 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:21:33.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:33.182 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:21:33.183 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:21:33.183 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:21:33.183 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:21:33.183 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:21:33.183 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:21:33.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:33.380 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:21:33.432 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:21:33.432 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:21:33.433 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:21:33.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:33.433 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:21:33.531 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:21:33.532 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:21:33.532 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:21:33.588 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:21:33.589 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:21:33.589 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:21:33.641 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:21:33.750 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:21:33.750 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:21:33.840 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:21:33.845 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:21:33.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:33.846 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:21:33.904 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:21:33.904 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:21:33.911 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:21:33.989 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:21:33.989 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:21:34.068 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:21:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:21:34.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:34.072 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:21:34.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:21:34.107+0000 7f4cf483c480 -1 init, newargv = 0x55b766c8cd30 newargc=15 2024-03-23T11:21:34.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[53576]: starting ceph client 2024-03-23T11:21:34.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[53576]: starting fuse 2024-03-23T11:21:35.216 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:21:35.323 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:21:35.323 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:21:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:21:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:21:35.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:35.427 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:21:35.483 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:21:35.483 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:21:35.483 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:21:35.483 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:21:35.483 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:21:35.483 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:21:35.483 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:21:35.483 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:21:35.484 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:21:35.485 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:21:35.485 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:21:35.485 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:21:35.485 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:21:35.666 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.53576.asok 2024-03-23T11:21:35.845 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53576.asok 2024-03-23T11:21:35.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:35.846 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53576.asok status 2024-03-23T11:21:36.094 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:21:36.094 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:21:36.094 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:21:36.094 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:21:36.094 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:21:36.094 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:21:36.094 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:21:36.094 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "53576", 2024-03-23T11:21:36.094 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:21:36.094 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 5808, 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 5808 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1949657013 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:36.095 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1949657013 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.5808 192.168.0.1:0/1949657013", 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/1949657013", 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 25, 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 48, 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:21:36.096 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:21:36.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:36.110 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:21:36.110 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:21:36.162 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:21:36.162 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:21:36.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:36.162 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:21:36.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:36.261 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_a) 2024-03-23T11:21:36.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:21:36.327 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parent_a/alpha) 2024-03-23T11:21:36.390 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:21:36.390 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:21:36.390 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:21:36.390 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:21:36.390 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/parent_a/alpha").st_ino) 2024-03-23T11:21:36.390 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:21:36.537 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627777 2024-03-23T11:21:36.716 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:21:36.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.984+0000 7f9938e00640 1 Processor -- start 2024-03-23T11:21:36.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.984+0000 7f9938e00640 1 -- start start 2024-03-23T11:21:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.986+0000 7f9938e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9934159490 0x7f993406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.986+0000 7f9938e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f99340ce5f0 0x7f99340b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.986+0000 7f9938e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340ced90 con 0x7f993415da90 2024-03-23T11:21:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.986+0000 7f9938e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f993406e3e0 con 0x7f99340ce5f0 2024-03-23T11:21:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.986+0000 7f9938e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f993406c1c0 con 0x7f9934159490 2024-03-23T11:21:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.986+0000 7f9933400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9934159490 0x7f993406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.986+0000 7f9933e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f99340ce5f0 0x7f99340b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.986+0000 7f9933400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9934159490 0x7f993406cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:47170/0 (socket says 172.21.15.62:47170) 2024-03-23T11:21:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.986+0000 7f9933400640 1 -- 172.21.15.62:0/2409401142 learned_addr learned my addr 172.21.15.62:0/2409401142 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.987+0000 7f9933400640 1 -- 172.21.15.62:0/2409401142 >> v2:172.21.15.120:3300/0 conn(0x7f99340ce5f0 msgr2=0x7f99340b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.987+0000 7f9933400640 1 --2- 172.21.15.62:0/2409401142 >> v2:172.21.15.120:3300/0 conn(0x7f99340ce5f0 0x7f99340b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.987+0000 7f9933400640 1 -- 172.21.15.62:0/2409401142 >> v1:172.21.15.120:6789/0 conn(0x7f993415da90 legacy=0x7f99340cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.987+0000 7f9933400640 1 -- 172.21.15.62:0/2409401142 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993406e100 con 0x7f9934159490 2024-03-23T11:21:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.989+0000 7f9933400640 1 --2- 172.21.15.62:0/2409401142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9934159490 0x7f993406cc60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9924009870 tx=0x7f992402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3b9483823863d51 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.989+0000 7f9932000640 1 -- 172.21.15.62:0/2409401142 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924031640 con 0x7f9934159490 2024-03-23T11:21:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.989+0000 7f9932000640 1 -- 172.21.15.62:0/2409401142 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99240317a0 con 0x7f9934159490 2024-03-23T11:21:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.989+0000 7f9932000640 1 -- 172.21.15.62:0/2409401142 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924031b30 con 0x7f9934159490 2024-03-23T11:21:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.990+0000 7f9938e00640 1 -- 172.21.15.62:0/2409401142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9934159490 msgr2=0x7f993406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.990+0000 7f9938e00640 1 --2- 172.21.15.62:0/2409401142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9934159490 0x7f993406cc60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9924009870 tx=0x7f992402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:21:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.990+0000 7f9938e00640 1 -- 172.21.15.62:0/2409401142 shutdown_connections 2024-03-23T11:21:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.990+0000 7f9938e00640 1 --2- 172.21.15.62:0/2409401142 >> v2:172.21.15.120:3300/0 conn(0x7f99340ce5f0 0x7f99340b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.990+0000 7f9938e00640 1 --2- 172.21.15.62:0/2409401142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9934159490 0x7f993406cc60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.990+0000 7f9938e00640 1 -- 172.21.15.62:0/2409401142 >> 172.21.15.62:0/2409401142 conn(0x7f9934066050 msgr2=0x7f9934069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.990+0000 7f9938e00640 1 -- 172.21.15.62:0/2409401142 shutdown_connections 2024-03-23T11:21:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.990+0000 7f9938e00640 1 -- 172.21.15.62:0/2409401142 wait complete. 2024-03-23T11:21:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.992+0000 7f9938e00640 1 Processor -- start 2024-03-23T11:21:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.992+0000 7f9938e00640 1 -- start start 2024-03-23T11:21:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9938e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99340ce5f0 0x7f99340b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9938e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9934159490 0x7f99340ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9938e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f993415da90 0x7f99340c3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9938e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99341476d0 con 0x7f993415da90 2024-03-23T11:21:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9938e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9934147990 con 0x7f9934159490 2024-03-23T11:21:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9938e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9934147c50 con 0x7f99340ce5f0 2024-03-23T11:21:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9933e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f993415da90 0x7f99340c3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9933e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f993415da90 0x7f99340c3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34760/0 (socket says 172.21.15.62:34760) 2024-03-23T11:21:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9933e00640 1 -- 172.21.15.62:0/3622333243 learned_addr learned my addr 172.21.15.62:0/3622333243 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:36.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9933400640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99340ce5f0 0x7f99340b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:36.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9932a00640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9934159490 0x7f99340ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:36.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9933e00640 1 -- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99340ce5f0 msgr2=0x7f99340b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:36.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9933e00640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99340ce5f0 0x7f99340b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:36.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9933e00640 1 -- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9934159490 msgr2=0x7f99340ac130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:36.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.993+0000 7f9933e00640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9934159490 0x7f99340ac130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:36.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.994+0000 7f9933e00640 1 -- 172.21.15.62:0/3622333243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99240094d0 con 0x7f993415da90 2024-03-23T11:21:36.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.994+0000 7f9933e00640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f993415da90 0x7f99340c3420 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f992c00efc0 tx=0x7f992c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.994+0000 7f992b400640 1 -- 172.21.15.62:0/3622333243 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992c009b20 con 0x7f993415da90 2024-03-23T11:21:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.994+0000 7f992b400640 1 -- 172.21.15.62:0/3622333243 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992c00f040 con 0x7f993415da90 2024-03-23T11:21:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.994+0000 7f9938e00640 1 -- 172.21.15.62:0/3622333243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9934147f30 con 0x7f993415da90 2024-03-23T11:21:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.994+0000 7f992b400640 1 -- 172.21.15.62:0/3622333243 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992c0146c0 con 0x7f993415da90 2024-03-23T11:21:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.994+0000 7f9938e00640 1 -- 172.21.15.62:0/3622333243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99341484f0 con 0x7f993415da90 2024-03-23T11:21:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.995+0000 7f9938e00640 1 -- 172.21.15.62:0/3622333243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99340cb880 con 0x7f993415da90 2024-03-23T11:21:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.996+0000 7f992b400640 1 -- 172.21.15.62:0/3622333243 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f992c014820 con 0x7f993415da90 2024-03-23T11:21:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.997+0000 7f992b400640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f991406a7a0 0x7f991406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.997+0000 7f992b400640 1 -- 172.21.15.62:0/3622333243 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f992c089b90 con 0x7f993415da90 2024-03-23T11:21:36.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.998+0000 7f9933400640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f991406a7a0 0x7f991406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:36.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:36.999+0000 7f9933400640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f991406a7a0 0x7f991406cc60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9924036000 tx=0x7f992403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:37.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.002+0000 7f992b400640 1 -- 172.21.15.62:0/3622333243 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f992c054fa0 con 0x7f993415da90 2024-03-23T11:21:37.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.245+0000 7f9938e00640 1 -- 172.21.15.62:0/3622333243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f993415c400 con 0x7f993415da90 2024-03-23T11:21:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.247+0000 7f992b400640 1 -- 172.21.15.62:0/3622333243 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f992c018020 con 0x7f993415da90 2024-03-23T11:21:37.245 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:37.245 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3787987198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3787987198},{"type":"v1","addr":"172.21.15.120:6835","nonce":3787987198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2749498408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2749498408},{"type":"v1","addr":"172.21.15.120:6837","nonce":2749498408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2325988870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2325988870},{"type":"v1","addr":"172.21.15.62:6833","nonce":2325988870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:21:25.687136+0000","modified":"2024-03-23T11:21:28.704192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6835/1159419089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1159419089},{"type":"v1","addr":"172.21.15.62:6835","nonce":1159419089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T11:21:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-03-23T11:21:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 -- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f991406a7a0 msgr2=0x7f991406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f991406a7a0 0x7f991406cc60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9924036000 tx=0x7f992403c040 comp rx=0 tx=0).stop 2024-03-23T11:21:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 -- 172.21.15.62:0/3622333243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f993415da90 msgr2=0x7f99340c3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f993415da90 0x7f99340c3420 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f992c00efc0 tx=0x7f992c00c490 comp rx=0 tx=0).stop 2024-03-23T11:21:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 -- 172.21.15.62:0/3622333243 shutdown_connections 2024-03-23T11:21:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f991406a7a0 0x7f991406cc60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f993415da90 0x7f99340c3420 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9934159490 0x7f99340ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 --2- 172.21.15.62:0/3622333243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f99340ce5f0 0x7f99340b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 -- 172.21.15.62:0/3622333243 >> 172.21.15.62:0/3622333243 conn(0x7f9934066050 msgr2=0x7f99340cbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 -- 172.21.15.62:0/3622333243 shutdown_connections 2024-03-23T11:21:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.250+0000 7f9938e00640 1 -- 172.21.15.62:0/3622333243 wait complete. 2024-03-23T11:21:37.267 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-23T11:21:37.532 INFO:teuthology.orchestra.run.smithi062.stdout:{"message":"","return_code":0} 2024-03-23T11:21:37.533 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-23T11:21:37.533 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-23T11:21:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.643+0000 7f0a4440bf40 1 Processor -- start 2024-03-23T11:21:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.644+0000 7f0a4440bf40 1 -- start start 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.644+0000 7f0a4440bf40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4584261f0 0x55c4584cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.644+0000 7f0a4440bf40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c458427fd0 0x55c4584265d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a4440bf40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c4584422a0 0x55c458426b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a4440bf40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45833d800 con 0x55c458427fd0 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a4440bf40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45833d4a0 con 0x55c4584261f0 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a4440bf40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c458334110 con 0x55c4584422a0 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a40000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c458427fd0 0x55c4584265d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a40000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c458427fd0 0x55c4584265d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42612/0 (socket says 172.21.15.62:42612) 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a40000640 1 -- 172.21.15.62:0/412047997 learned_addr learned my addr 172.21.15.62:0/412047997 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a40000640 1 -- 172.21.15.62:0/412047997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c4584422a0 msgr2=0x55c458426b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a40000640 1 --2- 172.21.15.62:0/412047997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c4584422a0 0x55c458426b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a40000640 1 -- 172.21.15.62:0/412047997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4584261f0 msgr2=0x55c4584cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:37.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a40a00640 1 --2- 172.21.15.62:0/412047997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4584261f0 0x55c4584cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a40000640 1 --2- 172.21.15.62:0/412047997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4584261f0 0x55c4584cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.645+0000 7f0a40000640 1 -- 172.21.15.62:0/412047997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c45834c340 con 0x55c458427fd0 2024-03-23T11:21:37.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.646+0000 7f0a40000640 1 --2- 172.21.15.62:0/412047997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c458427fd0 0x55c4584265d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c00d4f0 tx=0x7f0a2c00d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269a31468fc9fcc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.646+0000 7f0a3ec00640 1 -- 172.21.15.62:0/412047997 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c00f7d0 con 0x55c458427fd0 2024-03-23T11:21:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.646+0000 7f0a3ec00640 1 -- 172.21.15.62:0/412047997 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2c00f950 con 0x55c458427fd0 2024-03-23T11:21:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.646+0000 7f0a3ec00640 1 -- 172.21.15.62:0/412047997 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c00fd00 con 0x55c458427fd0 2024-03-23T11:21:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.646+0000 7f0a4440bf40 1 -- 172.21.15.62:0/412047997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c458427fd0 msgr2=0x55c4584265d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.646+0000 7f0a4440bf40 1 --2- 172.21.15.62:0/412047997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c458427fd0 0x55c4584265d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c00d4f0 tx=0x7f0a2c00d9b0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.646+0000 7f0a4440bf40 1 -- 172.21.15.62:0/412047997 shutdown_connections 2024-03-23T11:21:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.646+0000 7f0a4440bf40 1 --2- 172.21.15.62:0/412047997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c458427fd0 0x55c4584265d0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.646+0000 7f0a4440bf40 1 --2- 172.21.15.62:0/412047997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c4584422a0 0x55c458426b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.646+0000 7f0a4440bf40 1 --2- 172.21.15.62:0/412047997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4584261f0 0x55c4584cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.646+0000 7f0a4440bf40 1 -- 172.21.15.62:0/412047997 >> 172.21.15.62:0/412047997 conn(0x55c458442ff0 msgr2=0x55c458441b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.647+0000 7f0a4440bf40 1 -- 172.21.15.62:0/412047997 shutdown_connections 2024-03-23T11:21:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.647+0000 7f0a4440bf40 1 -- 172.21.15.62:0/412047997 wait complete. 2024-03-23T11:21:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.647+0000 7f0a4440bf40 1 Processor -- start 2024-03-23T11:21:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.648+0000 7f0a4440bf40 1 -- start start 2024-03-23T11:21:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.648+0000 7f0a4440bf40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c4584261f0 0x55c4584aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.648+0000 7f0a4440bf40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c458427fd0 0x55c458563240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.648+0000 7f0a4440bf40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4584422a0 0x55c458563780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.648+0000 7f0a4440bf40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c458566590 con 0x55c4584422a0 2024-03-23T11:21:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.648+0000 7f0a4440bf40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4584cf190 con 0x55c458427fd0 2024-03-23T11:21:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.648+0000 7f0a4440bf40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c458563cc0 con 0x55c4584261f0 2024-03-23T11:21:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.649+0000 7f0a41400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4584422a0 0x55c458563780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.649+0000 7f0a41400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4584422a0 0x55c458563780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42628/0 (socket says 172.21.15.62:42628) 2024-03-23T11:21:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.649+0000 7f0a41400640 1 -- 172.21.15.62:0/1573150291 learned_addr learned my addr 172.21.15.62:0/1573150291 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.649+0000 7f0a40000640 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c458427fd0 0x55c458563240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.649+0000 7f0a41400640 1 -- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c4584261f0 msgr2=0x55c4584aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.649+0000 7f0a40a00640 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c4584261f0 0x55c4584aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.649+0000 7f0a41400640 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c4584261f0 0x55c4584aef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.649+0000 7f0a41400640 1 -- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c458427fd0 msgr2=0x55c458563240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.649+0000 7f0a41400640 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c458427fd0 0x55c458563240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.649+0000 7f0a41400640 1 -- 172.21.15.62:0/1573150291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4583664e0 con 0x55c4584422a0 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.649+0000 7f0a40000640 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c458427fd0 0x55c458563240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.650+0000 7f0a41400640 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4584422a0 0x55c458563780 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0a380448c0 tx=0x7f0a3801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.650+0000 7f0a3d800640 1 -- 172.21.15.62:0/1573150291 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a38061070 con 0x55c4584422a0 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.650+0000 7f0a3d800640 1 -- 172.21.15.62:0/1573150291 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a38052910 con 0x55c4584422a0 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.650+0000 7f0a3d800640 1 -- 172.21.15.62:0/1573150291 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a38052c80 con 0x55c4584422a0 2024-03-23T11:21:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.650+0000 7f0a4440bf40 1 -- 172.21.15.62:0/1573150291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c45833f780 con 0x55c4584422a0 2024-03-23T11:21:37.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.650+0000 7f0a4440bf40 1 -- 172.21.15.62:0/1573150291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c45833f1e0 con 0x55c4584422a0 2024-03-23T11:21:37.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.652+0000 7f0a3d800640 1 -- 172.21.15.62:0/1573150291 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0a38059040 con 0x55c4584422a0 2024-03-23T11:21:37.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.653+0000 7f0a3d800640 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0a20069180 0x7f0a2006b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.653+0000 7f0a3d800640 1 -- 172.21.15.62:0/1573150291 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a380c82c0 con 0x55c4584422a0 2024-03-23T11:21:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.653+0000 7f0a4440bf40 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55c458566040 0x55c4584c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.653+0000 7f0a4440bf40 1 -- 172.21.15.62:0/1573150291 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:1 7.7 7:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48) v8 -- 0x55c4584cc2a0 con 0x55c458566040 2024-03-23T11:21:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.654+0000 7f0a40a00640 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0a20069180 0x7f0a2006b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.654+0000 7f0a40000640 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55c458566040 0x55c4584c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.655+0000 7f0a40a00640 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0a20069180 0x7f0a2006b640 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55c4584c7530 tx=0x7f0a30007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:37.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.655+0000 7f0a40000640 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55c458566040 0x55c4584c4c10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:37.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a40000640 1 -- 172.21.15.62:0/1573150291 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7f0a2c00fac0 con 0x55c458566040 2024-03-23T11:21:37.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a4440bf40 1 -- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55c458566040 msgr2=0x55c4584c4c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a4440bf40 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55c458566040 0x55c4584c4c10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a4440bf40 1 -- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0a20069180 msgr2=0x7f0a2006b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a4440bf40 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0a20069180 0x7f0a2006b640 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55c4584c7530 tx=0x7f0a30007040 comp rx=0 tx=0).stop 2024-03-23T11:21:37.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a4440bf40 1 -- 172.21.15.62:0/1573150291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4584422a0 msgr2=0x55c458563780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a4440bf40 1 --2- 172.21.15.62:0/1573150291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4584422a0 0x55c458563780 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0a380448c0 tx=0x7f0a3801c970 comp rx=0 tx=0).stop 2024-03-23T11:21:37.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a41400640 1 -- 172.21.15.62:0/1573150291 reap_dead start 2024-03-23T11:21:37.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a4440bf40 1 -- 172.21.15.62:0/1573150291 shutdown_connections 2024-03-23T11:21:37.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a4440bf40 1 -- 172.21.15.62:0/1573150291 >> 172.21.15.62:0/1573150291 conn(0x55c458442ff0 msgr2=0x55c4584cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:37.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a4440bf40 1 -- 172.21.15.62:0/1573150291 shutdown_connections 2024-03-23T11:21:37.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.656+0000 7f0a4440bf40 1 -- 172.21.15.62:0/1573150291 wait complete. 2024-03-23T11:21:37.659 INFO:teuthology.orchestra.run.smithi062.stdout:Yalphaparent_a 2024-03-23T11:21:37.660 DEBUG:teuthology.orchestra.run.smithi062:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-23T11:21:37.840 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:21:37.840 INFO:teuthology.orchestra.run.smithi062.stdout: "ino": 1099511627777, 2024-03-23T11:21:37.840 INFO:teuthology.orchestra.run.smithi062.stdout: "ancestors": [ 2024-03-23T11:21:37.840 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:37.840 INFO:teuthology.orchestra.run.smithi062.stdout: "dirino": 1099511627776, 2024-03-23T11:21:37.840 INFO:teuthology.orchestra.run.smithi062.stdout: "dname": "alpha", 2024-03-23T11:21:37.840 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 2 2024-03-23T11:21:37.840 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:21:37.841 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:21:37.841 INFO:teuthology.orchestra.run.smithi062.stdout: "dirino": 1, 2024-03-23T11:21:37.841 INFO:teuthology.orchestra.run.smithi062.stdout: "dname": "parent_a", 2024-03-23T11:21:37.841 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 4 2024-03-23T11:21:37.841 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:37.841 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-03-23T11:21:37.841 INFO:teuthology.orchestra.run.smithi062.stdout: "pool": 7, 2024-03-23T11:21:37.841 INFO:teuthology.orchestra.run.smithi062.stdout: "old_pools": [] 2024-03-23T11:21:37.841 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:21:37.841 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:37.855 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'oh i'"'"'m sorry did i overwrite your xattr?' 2024-03-23T11:21:37.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.978+0000 7f82e773cf40 1 Processor -- start 2024-03-23T11:21:37.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.978+0000 7f82e773cf40 1 -- start start 2024-03-23T11:21:37.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.979+0000 7f82e773cf40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56169804d920 0x5616980e8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.979+0000 7f82e773cf40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616980f1c30 0x56169804d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.979+0000 7f82e773cf40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5616980f2550 0x5616980fbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.979+0000 7f82e773cf40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561697f62800 con 0x56169804d920 2024-03-23T11:21:37.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.979+0000 7f82e773cf40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561697f624a0 con 0x5616980f1c30 2024-03-23T11:21:37.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.979+0000 7f82e773cf40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561697f59110 con 0x5616980f2550 2024-03-23T11:21:37.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.979+0000 7f82e4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56169804d920 0x5616980e8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.979+0000 7f82e4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56169804d920 0x5616980e8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42630/0 (socket says 172.21.15.62:42630) 2024-03-23T11:21:37.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.979+0000 7f82e4a00640 1 -- 172.21.15.62:0/562794683 learned_addr learned my addr 172.21.15.62:0/562794683 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:37.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.980+0000 7f82e4000640 1 --2- 172.21.15.62:0/562794683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616980f1c30 0x56169804d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.980+0000 7f82e4a00640 1 -- 172.21.15.62:0/562794683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5616980f2550 msgr2=0x5616980fbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:37.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.980+0000 7f82e4a00640 1 --2- 172.21.15.62:0/562794683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5616980f2550 0x5616980fbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.980+0000 7f82e4a00640 1 -- 172.21.15.62:0/562794683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616980f1c30 msgr2=0x56169804d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:37.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.980+0000 7f82e4a00640 1 --2- 172.21.15.62:0/562794683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616980f1c30 0x56169804d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.980+0000 7f82e4a00640 1 -- 172.21.15.62:0/562794683 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d0005d20 con 0x56169804d920 2024-03-23T11:21:37.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.980+0000 7f82e4a00640 1 --2- 172.21.15.62:0/562794683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56169804d920 0x5616980e8fb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f82d40098e0 tx=0x7f82d4005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=213f4b507fd69c2c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:37.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.980+0000 7f82e2c00640 1 -- 172.21.15.62:0/562794683 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d400f040 con 0x56169804d920 2024-03-23T11:21:37.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.980+0000 7f82e2c00640 1 -- 172.21.15.62:0/562794683 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d4008950 con 0x56169804d920 2024-03-23T11:21:37.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.980+0000 7f82e2c00640 1 -- 172.21.15.62:0/562794683 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4008d00 con 0x56169804d920 2024-03-23T11:21:37.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.981+0000 7f82e773cf40 1 -- 172.21.15.62:0/562794683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56169804d920 msgr2=0x5616980e8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.981+0000 7f82e773cf40 1 --2- 172.21.15.62:0/562794683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56169804d920 0x5616980e8fb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f82d40098e0 tx=0x7f82d4005a00 comp rx=0 tx=0).stop 2024-03-23T11:21:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.981+0000 7f82e773cf40 1 -- 172.21.15.62:0/562794683 shutdown_connections 2024-03-23T11:21:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.981+0000 7f82e773cf40 1 --2- 172.21.15.62:0/562794683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5616980f2550 0x5616980fbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.981+0000 7f82e773cf40 1 --2- 172.21.15.62:0/562794683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616980f1c30 0x56169804d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.981+0000 7f82e773cf40 1 --2- 172.21.15.62:0/562794683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56169804d920 0x5616980e8fb0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.981+0000 7f82e773cf40 1 -- 172.21.15.62:0/562794683 >> 172.21.15.62:0/562794683 conn(0x561698068000 msgr2=0x5616980f5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.981+0000 7f82e773cf40 1 -- 172.21.15.62:0/562794683 shutdown_connections 2024-03-23T11:21:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.981+0000 7f82e773cf40 1 -- 172.21.15.62:0/562794683 wait complete. 2024-03-23T11:21:37.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.982+0000 7f82e773cf40 1 Processor -- start 2024-03-23T11:21:37.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.982+0000 7f82e773cf40 1 -- start start 2024-03-23T11:21:37.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.983+0000 7f82e773cf40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56169804d920 0x5616980fbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.983+0000 7f82e773cf40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616980f1c30 0x5616980ee0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.983+0000 7f82e773cf40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616980f2550 0x5616980e9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.983+0000 7f82e773cf40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616980f3730 con 0x5616980f2550 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.983+0000 7f82e773cf40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616980f3a10 con 0x5616980f1c30 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.983+0000 7f82e773cf40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5616980f3cf0 con 0x56169804d920 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.983+0000 7f82e5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616980f2550 0x5616980e9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.983+0000 7f82e5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616980f2550 0x5616980e9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42642/0 (socket says 172.21.15.62:42642) 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.983+0000 7f82e5400640 1 -- 172.21.15.62:0/2720122936 learned_addr learned my addr 172.21.15.62:0/2720122936 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.984+0000 7f82e4000640 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616980f1c30 0x5616980ee0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.984+0000 7f82e4a00640 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56169804d920 0x5616980fbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.984+0000 7f82e5400640 1 -- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56169804d920 msgr2=0x5616980fbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.984+0000 7f82e5400640 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56169804d920 0x5616980fbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.984+0000 7f82e5400640 1 -- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616980f1c30 msgr2=0x5616980ee0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.984+0000 7f82e5400640 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616980f1c30 0x5616980ee0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.984+0000 7f82e5400640 1 -- 172.21.15.62:0/2720122936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d4009510 con 0x5616980f2550 2024-03-23T11:21:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.984+0000 7f82e4a00640 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56169804d920 0x5616980fbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.984+0000 7f82e5400640 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616980f2550 0x5616980e9f60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f82dc006df0 tx=0x7f82dc00b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.984+0000 7f82e1800640 1 -- 172.21.15.62:0/2720122936 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0060d0 con 0x5616980f2550 2024-03-23T11:21:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.985+0000 7f82e1800640 1 -- 172.21.15.62:0/2720122936 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc006250 con 0x5616980f2550 2024-03-23T11:21:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.985+0000 7f82e1800640 1 -- 172.21.15.62:0/2720122936 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc006400 con 0x5616980f2550 2024-03-23T11:21:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.985+0000 7f82e773cf40 1 -- 172.21.15.62:0/2720122936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561697f71340 con 0x5616980f2550 2024-03-23T11:21:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.985+0000 7f82e773cf40 1 -- 172.21.15.62:0/2720122936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561697f64780 con 0x5616980f2550 2024-03-23T11:21:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.987+0000 7f82e1800640 1 -- 172.21.15.62:0/2720122936 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f82dc004720 con 0x5616980f2550 2024-03-23T11:21:37.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.988+0000 7f82e1800640 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f82c4069180 0x7f82c406b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.988+0000 7f82e1800640 1 -- 172.21.15.62:0/2720122936 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82dc086f00 con 0x5616980f2550 2024-03-23T11:21:37.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.988+0000 7f82e773cf40 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x5616980d4430 0x5616980d4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:37.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.988+0000 7f82e773cf40 1 -- 172.21.15.62:0/2720122936 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:1 7.7 7:e2e07930:::10000000001.00000000:head [setxattr parent (40) in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48) v8 -- 0x5616980ec8b0 con 0x5616980d4430 2024-03-23T11:21:37.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.988+0000 7f82e4a00640 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f82c4069180 0x7f82c406b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.989+0000 7f82e4000640 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x5616980d4430 0x5616980d4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:37.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.989+0000 7f82e4a00640 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f82c4069180 0x7f82c406b640 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f82d4000c00 tx=0x7f82d400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:37.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.990+0000 7f82e4000640 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x5616980d4430 0x5616980d4810 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e4000640 1 -- 172.21.15.62:0/2720122936 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (40)] v48'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f82d000a0c0 con 0x5616980d4430 2024-03-23T11:21:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e773cf40 1 -- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x5616980d4430 msgr2=0x5616980d4810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e773cf40 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x5616980d4430 0x5616980d4810 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e773cf40 1 -- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f82c4069180 msgr2=0x7f82c406b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e773cf40 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f82c4069180 0x7f82c406b640 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f82d4000c00 tx=0x7f82d400b040 comp rx=0 tx=0).stop 2024-03-23T11:21:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e773cf40 1 -- 172.21.15.62:0/2720122936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616980f2550 msgr2=0x5616980e9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e773cf40 1 --2- 172.21.15.62:0/2720122936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616980f2550 0x5616980e9f60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f82dc006df0 tx=0x7f82dc00b4b0 comp rx=0 tx=0).stop 2024-03-23T11:21:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e5400640 1 -- 172.21.15.62:0/2720122936 reap_dead start 2024-03-23T11:21:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e773cf40 1 -- 172.21.15.62:0/2720122936 shutdown_connections 2024-03-23T11:21:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e773cf40 1 -- 172.21.15.62:0/2720122936 >> 172.21.15.62:0/2720122936 conn(0x561698068000 msgr2=0x561698048f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e773cf40 1 -- 172.21.15.62:0/2720122936 shutdown_connections 2024-03-23T11:21:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:37.993+0000 7f82e773cf40 1 -- 172.21.15.62:0/2720122936 wait complete. 2024-03-23T11:21:37.996 DEBUG:teuthology.orchestra.run.smithi062:> sudo daemon-helper kill ceph --cluster ceph -w 2024-03-23T11:21:38.003 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start / repair,recursive 2024-03-23T11:21:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.202+0000 7f910a400640 1 Processor -- start 2024-03-23T11:21:38.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.202+0000 7f910a400640 1 -- start start 2024-03-23T11:21:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.204+0000 7f910a400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f910415cd00 0x7f91040b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.204+0000 7f910a400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91040c9d70 0x7f91040ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.204+0000 7f910a400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f91040ade20 0x7f91040cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.204+0000 7f910a400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91040b0250 con 0x7f910415cd00 2024-03-23T11:21:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.204+0000 7f910a400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91040ceec0 con 0x7f91040ade20 2024-03-23T11:21:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.204+0000 7f910a400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91040acc10 con 0x7f91040c9d70 2024-03-23T11:21:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.204+0000 7f9109000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f910415cd00 0x7f91040b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.204+0000 7f9109000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f910415cd00 0x7f91040b08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42644/0 (socket says 172.21.15.62:42644) 2024-03-23T11:21:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.204+0000 7f9109000640 1 -- 172.21.15.62:0/1279845910 learned_addr learned my addr 172.21.15.62:0/1279845910 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.205+0000 7f9109a00640 1 --2- 172.21.15.62:0/1279845910 >> v2:172.21.15.120:3300/0 conn(0x7f91040ade20 0x7f91040cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.205+0000 7f9103e00640 1 --2- 172.21.15.62:0/1279845910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91040c9d70 0x7f91040ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.205+0000 7f9109000640 1 -- 172.21.15.62:0/1279845910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91040c9d70 msgr2=0x7f91040ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.205+0000 7f9109000640 1 --2- 172.21.15.62:0/1279845910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91040c9d70 0x7f91040ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.205+0000 7f9109000640 1 -- 172.21.15.62:0/1279845910 >> v2:172.21.15.120:3300/0 conn(0x7f91040ade20 msgr2=0x7f91040cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.205+0000 7f9109000640 1 --2- 172.21.15.62:0/1279845910 >> v2:172.21.15.120:3300/0 conn(0x7f91040ade20 0x7f91040cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.205+0000 7f9109000640 1 -- 172.21.15.62:0/1279845910 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910415d0e0 con 0x7f910415cd00 2024-03-23T11:21:38.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.206+0000 7f9109000640 1 --2- 172.21.15.62:0/1279845910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f910415cd00 0x7f91040b08e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f91040669c0 tx=0x7f90f402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6ed22e219804d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.206+0000 7f9103400640 1 -- 172.21.15.62:0/1279845910 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f402c9e0 con 0x7f910415cd00 2024-03-23T11:21:38.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.207+0000 7f9103400640 1 -- 172.21.15.62:0/1279845910 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f402cb40 con 0x7f910415cd00 2024-03-23T11:21:38.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.207+0000 7f9103400640 1 -- 172.21.15.62:0/1279845910 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f40343f0 con 0x7f910415cd00 2024-03-23T11:21:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.207+0000 7f910a400640 1 -- 172.21.15.62:0/1279845910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f910415cd00 msgr2=0x7f91040b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.207+0000 7f910a400640 1 --2- 172.21.15.62:0/1279845910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f910415cd00 0x7f91040b08e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f91040669c0 tx=0x7f90f402be40 comp rx=0 tx=0).stop 2024-03-23T11:21:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.208+0000 7f910a400640 1 -- 172.21.15.62:0/1279845910 shutdown_connections 2024-03-23T11:21:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.208+0000 7f910a400640 1 --2- 172.21.15.62:0/1279845910 >> v2:172.21.15.120:3300/0 conn(0x7f91040ade20 0x7f91040cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.208+0000 7f910a400640 1 --2- 172.21.15.62:0/1279845910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91040c9d70 0x7f91040ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.208+0000 7f910a400640 1 --2- 172.21.15.62:0/1279845910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f910415cd00 0x7f91040b08e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.208+0000 7f910a400640 1 -- 172.21.15.62:0/1279845910 >> 172.21.15.62:0/1279845910 conn(0x7f91040989d0 msgr2=0x7f91040b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.208+0000 7f910a400640 1 -- 172.21.15.62:0/1279845910 shutdown_connections 2024-03-23T11:21:38.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.208+0000 7f910a400640 1 -- 172.21.15.62:0/1279845910 wait complete. 2024-03-23T11:21:38.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.209+0000 7f910a400640 1 Processor -- start 2024-03-23T11:21:38.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.210+0000 7f910a400640 1 -- start start 2024-03-23T11:21:38.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f910a400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91040ade20 0x7f91040b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f910a400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91040c9d70 0x7f91041f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f910a400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f910415cd00 0x7f910406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f910a400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91041f31c0 con 0x7f91040c9d70 2024-03-23T11:21:38.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f910a400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91041f2c00 con 0x7f91040ade20 2024-03-23T11:21:38.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f910a400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91041f2ee0 con 0x7f910415cd00 2024-03-23T11:21:38.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f9103e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91040c9d70 0x7f91041f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f9103e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91040c9d70 0x7f91041f2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42654/0 (socket says 172.21.15.62:42654) 2024-03-23T11:21:38.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f9103e00640 1 -- 172.21.15.62:0/4120440295 learned_addr learned my addr 172.21.15.62:0/4120440295 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f9109000640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91040ade20 0x7f91040b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f9103e00640 1 -- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f910415cd00 msgr2=0x7f910406d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f9103e00640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f910415cd00 0x7f910406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f9103e00640 1 -- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91040ade20 msgr2=0x7f91040b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f9103e00640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91040ade20 0x7f91040b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.211+0000 7f9103e00640 1 -- 172.21.15.62:0/4120440295 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f4009b30 con 0x7f91040c9d70 2024-03-23T11:21:38.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.212+0000 7f9103e00640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91040c9d70 0x7f91041f2220 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00d800 tx=0x7f90ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.212+0000 7f9100c00640 1 -- 172.21.15.62:0/4120440295 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec00b6c0 con 0x7f91040c9d70 2024-03-23T11:21:38.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.212+0000 7f9100c00640 1 -- 172.21.15.62:0/4120440295 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ec00b820 con 0x7f91040c9d70 2024-03-23T11:21:38.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.212+0000 7f9100c00640 1 -- 172.21.15.62:0/4120440295 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec00bbb0 con 0x7f91040c9d70 2024-03-23T11:21:38.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.212+0000 7f910a400640 1 -- 172.21.15.62:0/4120440295 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91040b14d0 con 0x7f91040c9d70 2024-03-23T11:21:38.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.212+0000 7f910a400640 1 -- 172.21.15.62:0/4120440295 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91040b2210 con 0x7f91040c9d70 2024-03-23T11:21:38.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.215+0000 7f9100c00640 1 -- 172.21.15.62:0/4120440295 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f90ec00bd10 con 0x7f91040c9d70 2024-03-23T11:21:38.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.216+0000 7f9100c00640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f90e006a7d0 0x7f90e006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.216+0000 7f9100c00640 1 -- 172.21.15.62:0/4120440295 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90ec089300 con 0x7f91040c9d70 2024-03-23T11:21:38.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.217+0000 7f9109000640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f90e006a7d0 0x7f90e006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.217+0000 7f910e7ce740 1 Processor -- start 2024-03-23T11:21:38.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.217+0000 7f910e7ce740 1 -- start start 2024-03-23T11:21:38.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.218+0000 7f9109000640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f90e006a7d0 0x7f90e006cc90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f90f4035000 tx=0x7f90f4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.218+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b404eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.218+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b404d570 0x5597b404f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.218+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b4049ca0 0x5597b4079800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.218+0000 7f910e7ce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597b3fb9f40 con 0x5597b40974f0 2024-03-23T11:21:38.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.218+0000 7f910e7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597b3fb76d0 con 0x5597b404d570 2024-03-23T11:21:38.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.218+0000 7f910e7ce740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5597b3fb2fc0 con 0x5597b4049ca0 2024-03-23T11:21:38.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9109000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b404eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9109000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b404eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42658/0 (socket says 172.21.15.62:42658) 2024-03-23T11:21:38.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9109000640 1 -- 172.21.15.62:0/1252971705 learned_addr learned my addr 172.21.15.62:0/1252971705 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9103e00640 1 --2- 172.21.15.62:0/1252971705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b404d570 0x5597b404f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9109a00640 1 --2- 172.21.15.62:0/1252971705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b4049ca0 0x5597b4079800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9109000640 1 -- 172.21.15.62:0/1252971705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b4049ca0 msgr2=0x5597b4079800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9109000640 1 --2- 172.21.15.62:0/1252971705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b4049ca0 0x5597b4079800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9109000640 1 -- 172.21.15.62:0/1252971705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b404d570 msgr2=0x5597b404f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9109000640 1 --2- 172.21.15.62:0/1252971705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b404d570 0x5597b404f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9109000640 1 -- 172.21.15.62:0/1252971705 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597b3f7a230 con 0x5597b40974f0 2024-03-23T11:21:38.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9103e00640 1 --2- 172.21.15.62:0/1252971705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b404d570 0x5597b404f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:38.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9109a00640 1 --2- 172.21.15.62:0/1252971705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b4049ca0 0x5597b4079800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.219+0000 7f9109000640 1 --2- 172.21.15.62:0/1252971705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b404eb20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f90f402c450 tx=0x7f90f402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e2aacc6eeecf73 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.220+0000 7f90f9600640 1 -- 172.21.15.62:0/1252971705 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f404e070 con 0x5597b40974f0 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.220+0000 7f90f9600640 1 -- 172.21.15.62:0/1252971705 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f403eb00 con 0x5597b40974f0 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.220+0000 7f90f9600640 1 -- 172.21.15.62:0/1252971705 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f403ee70 con 0x5597b40974f0 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.221+0000 7f910e7ce740 1 -- 172.21.15.62:0/1252971705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 msgr2=0x5597b404eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.221+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1252971705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b404eb20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f90f402c450 tx=0x7f90f402e000 comp rx=0 tx=0).stop 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.221+0000 7f910e7ce740 1 -- 172.21.15.62:0/1252971705 shutdown_connections 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.221+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1252971705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b4049ca0 0x5597b4079800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.221+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1252971705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b404d570 0x5597b404f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.221+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1252971705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b404eb20 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.221+0000 7f910e7ce740 1 -- 172.21.15.62:0/1252971705 >> 172.21.15.62:0/1252971705 conn(0x5597b404bf10 msgr2=0x5597b4094800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.221+0000 7f910e7ce740 1 -- 172.21.15.62:0/1252971705 shutdown_connections 2024-03-23T11:21:38.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.221+0000 7f910e7ce740 1 -- 172.21.15.62:0/1252971705 wait complete. 2024-03-23T11:21:38.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.222+0000 7f910e7ce740 1 -- start start 2024-03-23T11:21:38.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.222+0000 7f910e7ce740 1 Processor -- start 2024-03-23T11:21:38.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b404d570 0x5597b409c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b40974f0 0x5597b4074f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b40776e0 0x5597b40a7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f910e7ce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597b3f54320 con 0x5597b404d570 2024-03-23T11:21:38.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f910e7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597b3f53fc0 con 0x5597b40776e0 2024-03-23T11:21:38.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f910e7ce740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5597b3d1b0c0 con 0x5597b40974f0 2024-03-23T11:21:38.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f910e7ce740 10 client.0 _ll_register_callbacks cb 0x5597b3f79a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:21:38.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f9109000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b404d570 0x5597b409c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f9109000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b404d570 0x5597b409c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42660/0 (socket says 172.21.15.62:42660) 2024-03-23T11:21:38.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f9109000640 1 -- 172.21.15.62:0/2843597330 learned_addr learned my addr 172.21.15.62:0/2843597330 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f9109a00640 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b40776e0 0x5597b40a7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f9109000640 1 -- 172.21.15.62:0/2843597330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b40974f0 msgr2=0x5597b4074f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f9109000640 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b40974f0 0x5597b4074f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f9109000640 1 -- 172.21.15.62:0/2843597330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b40776e0 msgr2=0x5597b40a7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f9109000640 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b40776e0 0x5597b40a7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.223+0000 7f9109000640 1 -- 172.21.15.62:0/2843597330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910415d0e0 con 0x5597b404d570 2024-03-23T11:21:38.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.224+0000 7f9109000640 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b404d570 0x5597b409c8b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f90f4000f80 tx=0x7f90f4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade97bdfbfe3ecaf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.224+0000 7f90f2000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:21:38.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.224+0000 7f90f2000640 1 -- 172.21.15.62:0/2843597330 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f404e040 con 0x5597b404d570 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.224+0000 7f90f2000640 1 -- 172.21.15.62:0/2843597330 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90f404ade0 con 0x5597b404d570 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.224+0000 7f910e7ce740 1 -- 172.21.15.62:0/2843597330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5597b3fdcbb0 con 0x5597b404d570 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.224+0000 7f90f2000640 1 -- 172.21.15.62:0/2843597330 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90f4054410 con 0x5597b404d570 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.224+0000 7f910e7ce740 10 client.5838 fetch_fsmap learned FSMap version 25 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.224+0000 7f910e7ce740 1 -- 172.21.15.62:0/2843597330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5597b3fb1640 con 0x5597b404d570 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.225+0000 7f90f2000640 1 -- 172.21.15.62:0/2843597330 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5597b3fdcbb0 con 0x5597b404d570 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.225+0000 7f90f2000640 1 -- 172.21.15.62:0/2843597330 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f90f403c040 con 0x5597b404d570 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.225+0000 7f910e7ce740 10 client.5838 fetch_fsmap finished waiting for FSMap version 25 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.225+0000 7f910e7ce740 10 client.5838 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.225+0000 7f910e7ce740 20 client.5838 populate_metadata read hostname 'smithi062' 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.225+0000 7f910e7ce740 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x5597b40993a0 0x5597b4099780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.225+0000 7f910e7ce740 4 client.5838 mds_command: new command op to 5652 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:21:38.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.225+0000 7f910e7ce740 1 -- 172.21.15.62:0/2843597330 --> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5597b3ca7560 con 0x5597b40993a0 2024-03-23T11:21:38.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.225+0000 7f9103e00640 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x5597b40993a0 0x5597b4099780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.226+0000 7f9103e00640 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x5597b40993a0 0x5597b4099780 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed4fe428fadcb469 server_cookie=afd321b05d47d010 in_seq=0 out_seq=0 2024-03-23T11:21:38.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.226+0000 7f90f2000640 10 client.5838 ms_handle_connect on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:38.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.226+0000 7f90f2000640 1 -- 172.21.15.62:0/2843597330 <== mds.0 v2:172.21.15.62:6834/1159419089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5597b3ca7560 con 0x5597b40993a0 2024-03-23T11:21:38.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.226+0000 7f90f2000640 10 client.5838 handle_command_reply: tid=0 2024-03-23T11:21:38.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.226+0000 7fd8ff400640 1 Processor -- start 2024-03-23T11:21:38.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.226+0000 7f910e7ce740 1 client.5838 shutdown 2024-03-23T11:21:38.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7fd8ff400640 1 -- start start 2024-03-23T11:21:38.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f910e7ce740 1 -- 172.21.15.62:0/2843597330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b404d570 msgr2=0x5597b409c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f910e7ce740 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b404d570 0x5597b409c8b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f90f4000f80 tx=0x7f90f4004ab0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f910e7ce740 1 -- 172.21.15.62:0/2843597330 shutdown_connections 2024-03-23T11:21:38.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f910e7ce740 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x5597b40993a0 0x5597b4099780 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f910e7ce740 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b40776e0 0x5597b40a7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f910e7ce740 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b40974f0 0x5597b4074f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f910e7ce740 1 --2- 172.21.15.62:0/2843597330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b404d570 0x5597b409c8b0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f910e7ce740 1 -- 172.21.15.62:0/2843597330 >> 172.21.15.62:0/2843597330 conn(0x5597b4049ca0 msgr2=0x5597b404deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f90f2000640 0 client.5838 ms_handle_reset on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:38.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f910e7ce740 1 -- 172.21.15.62:0/2843597330 shutdown_connections 2024-03-23T11:21:38.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f910e7ce740 1 -- 172.21.15.62:0/2843597330 wait complete. 2024-03-23T11:21:38.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.227+0000 7f910e7ce740 20 client.5838 trim_cache size 0 max 16384 2024-03-23T11:21:38.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.228+0000 7fd8ff400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd8f815ab50 0x7fd8f80b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.228+0000 7fd8ff400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80cc690 con 0x7fd8f80ce310 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.228+0000 7fd8ff400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80cbdf0 con 0x7fd8f80cf430 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.228+0000 7fd8ff400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f80cc0d0 con 0x7fd8f815ab50 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.228+0000 7fd8f7e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd8f80ce310 0x7fd8f80ced20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47274/0 (socket says 172.21.15.62:47274) 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.228+0000 7fd8f7e00640 1 -- 172.21.15.62:0/4177533688 learned_addr learned my addr 172.21.15.62:0/4177533688 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.229+0000 7fd8fca00640 1 --2- 172.21.15.62:0/4177533688 >> v2:172.21.15.120:3300/0 conn(0x7fd8f815ab50 0x7fd8f80b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.229+0000 7fd8f7400640 1 -- 172.21.15.62:0/4177533688 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190103168 0 0) 0x7fd8f80cc690 con 0x7fd8f80ce310 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.229+0000 7fd8f7400640 1 -- 172.21.15.62:0/4177533688 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e0003520 con 0x7fd8f80ce310 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.229+0000 7fd8f7400640 1 -- 172.21.15.62:0/4177533688 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1571952038 0 0) 0x7fd8e0003520 con 0x7fd8f80ce310 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.229+0000 7fd8fca00640 1 -- 172.21.15.62:0/4177533688 >> v1:172.21.15.120:6789/0 conn(0x7fd8f80cf430 legacy=0x7fd8f8099270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.229+0000 7fd8fca00640 1 -- 172.21.15.62:0/4177533688 >> v1:172.21.15.62:6789/0 conn(0x7fd8f80ce310 legacy=0x7fd8f80ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.229+0000 7fd8fca00640 1 -- 172.21.15.62:0/4177533688 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f80cc3b0 con 0x7fd8f815ab50 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.231+0000 7fd8fca00640 1 --2- 172.21.15.62:0/4177533688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8f815ab50 0x7fd8f80b00a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0097a0 tx=0x7fd8ec02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a07655acf0346d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.231+0000 7fd8f7400640 1 -- 172.21.15.62:0/4177533688 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec02f050 con 0x7fd8f815ab50 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.231+0000 7fd8f7400640 1 -- 172.21.15.62:0/4177533688 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec031500 con 0x7fd8f815ab50 2024-03-23T11:21:38.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.231+0000 7fd8f7400640 1 -- 172.21.15.62:0/4177533688 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4001670 con 0x7fd8f815ab50 2024-03-23T11:21:38.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.232+0000 7fd8ff400640 1 -- 172.21.15.62:0/4177533688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8f815ab50 msgr2=0x7fd8f80b00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.232+0000 7fd8ff400640 1 --2- 172.21.15.62:0/4177533688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8f815ab50 0x7fd8f80b00a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0097a0 tx=0x7fd8ec02b820 comp rx=0 tx=0).stop 2024-03-23T11:21:38.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.232+0000 7fd8ff400640 1 -- 172.21.15.62:0/4177533688 shutdown_connections 2024-03-23T11:21:38.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.232+0000 7fd8ff400640 1 --2- 172.21.15.62:0/4177533688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8f815ab50 0x7fd8f80b00a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.232+0000 7fd8ff400640 1 -- 172.21.15.62:0/4177533688 >> 172.21.15.62:0/4177533688 conn(0x7fd8f80b2230 msgr2=0x7fd8f80b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.232+0000 7fd8ff400640 1 -- 172.21.15.62:0/4177533688 shutdown_connections 2024-03-23T11:21:38.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.232+0000 7fd8ff400640 1 -- 172.21.15.62:0/4177533688 wait complete. 2024-03-23T11:21:38.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.234+0000 7fd8ff400640 1 Processor -- start 2024-03-23T11:21:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.234+0000 7fd8ff400640 1 -- start start 2024-03-23T11:21:38.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.235+0000 7fd8ff400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8f80ce310 0x7fd8f81481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.235+0000 7fd8ff400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8f80cf430 0x7fd8f81f12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.235+0000 7fd8ff400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8f815ab50 0x7fd8f81f17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.235+0000 7fd8ff400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80ae9b0 con 0x7fd8f815ab50 2024-03-23T11:21:38.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.235+0000 7fd8ff400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80addc0 con 0x7fd8f80cf430 2024-03-23T11:21:38.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.235+0000 7fd8ff400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80aedd0 con 0x7fd8f80ce310 2024-03-23T11:21:38.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.235+0000 7fd8fd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8f815ab50 0x7fd8f81f17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.235+0000 7fd8fd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8f815ab50 0x7fd8f81f17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42666/0 (socket says 172.21.15.62:42666) 2024-03-23T11:21:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.235+0000 7fd8fd400640 1 -- 172.21.15.62:0/2192499017 learned_addr learned my addr 172.21.15.62:0/2192499017 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.235+0000 7fd8fca00640 1 --2- 172.21.15.62:0/2192499017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8f80ce310 0x7fd8f81481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.236+0000 7fd8f7e00640 1 --2- 172.21.15.62:0/2192499017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8f80cf430 0x7fd8f81f12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.236+0000 7fd8fd400640 1 -- 172.21.15.62:0/2192499017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8f80ce310 msgr2=0x7fd8f81481c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.236+0000 7fd8fd400640 1 --2- 172.21.15.62:0/2192499017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8f80ce310 0x7fd8f81481c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.236+0000 7fd8fd400640 1 -- 172.21.15.62:0/2192499017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8f80cf430 msgr2=0x7fd8f81f12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.236+0000 7fd8fd400640 1 --2- 172.21.15.62:0/2192499017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8f80cf430 0x7fd8f81f12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.236+0000 7fd8fd400640 1 -- 172.21.15.62:0/2192499017 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec009400 con 0x7fd8f815ab50 2024-03-23T11:21:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.236+0000 7fd8fd400640 1 --2- 172.21.15.62:0/2192499017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8f815ab50 0x7fd8f81f17e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000c6b0 tx=0x7fd8f000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.237+0000 7fd8f4c00640 1 -- 172.21.15.62:0/2192499017 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0003b70 con 0x7fd8f815ab50 2024-03-23T11:21:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.237+0000 7fd8f4c00640 1 -- 172.21.15.62:0/2192499017 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f0003cd0 con 0x7fd8f815ab50 2024-03-23T11:21:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.237+0000 7fd8f4c00640 1 -- 172.21.15.62:0/2192499017 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0005310 con 0x7fd8f815ab50 2024-03-23T11:21:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.237+0000 7fd8ff400640 1 -- 172.21.15.62:0/2192499017 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f80ae010 con 0x7fd8f815ab50 2024-03-23T11:21:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.237+0000 7fd8ff400640 1 -- 172.21.15.62:0/2192499017 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f80ae5d0 con 0x7fd8f815ab50 2024-03-23T11:21:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.238+0000 7fd8ff400640 1 -- 172.21.15.62:0/2192499017 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fd8f80672c0 con 0x7fd8f815ab50 2024-03-23T11:21:38.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.239+0000 7fd8f4c00640 1 -- 172.21.15.62:0/2192499017 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd8f0005470 con 0x7fd8f815ab50 2024-03-23T11:21:38.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.240+0000 7fd8f4c00640 1 --2- 172.21.15.62:0/2192499017 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8d806a7a0 0x7fd8d806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.240+0000 7fd8f4c00640 1 -- 172.21.15.62:0/2192499017 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8f0089510 con 0x7fd8f815ab50 2024-03-23T11:21:38.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.240+0000 7fd8f4c00640 1 -- 172.21.15.62:0/2192499017 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+501 (secure 0 0 0) 0x7fd8f00898a0 con 0x7fd8f815ab50 2024-03-23T11:21:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.240+0000 7fd8ff400640 1 -- 172.21.15.62:0/2192499017 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7fd8f80b30b0 con 0x7fd8f815ab50 2024-03-23T11:21:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.241+0000 7fd8fca00640 1 --2- 172.21.15.62:0/2192499017 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8d806a7a0 0x7fd8d806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.242+0000 7fd8fca00640 1 --2- 172.21.15.62:0/2192499017 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8d806a7a0 0x7fd8d806cc60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec035000 tx=0x7fd8ec008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.245+0000 7f910e7ce740 1 Processor -- start 2024-03-23T11:21:38.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.246+0000 7f910e7ce740 1 -- start start 2024-03-23T11:21:38.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.246+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b404d570 0x5597b409ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.246+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b40776e0 0x5597b40993a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.246+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40998e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.246+0000 7f910e7ce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597b4096dc0 con 0x5597b40974f0 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.246+0000 7f910e7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597b4075dc0 con 0x5597b40776e0 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.246+0000 7f910e7ce740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5597b3f6c4f0 con 0x5597b404d570 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.247+0000 7f9109a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.247+0000 7f9109a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42678/0 (socket says 172.21.15.62:42678) 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.247+0000 7f9109a00640 1 -- 172.21.15.62:0/1514330182 learned_addr learned my addr 172.21.15.62:0/1514330182 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.247+0000 7f9103e00640 1 --2- 172.21.15.62:0/1514330182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b40776e0 0x5597b40993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.247+0000 7f9109000640 1 --2- 172.21.15.62:0/1514330182 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b404d570 0x5597b409ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.247+0000 7f9109a00640 1 -- 172.21.15.62:0/1514330182 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b404d570 msgr2=0x5597b409ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.247+0000 7f9109a00640 1 --2- 172.21.15.62:0/1514330182 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b404d570 0x5597b409ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.247+0000 7f9109a00640 1 -- 172.21.15.62:0/1514330182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b40776e0 msgr2=0x5597b40993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.247+0000 7f9109a00640 1 --2- 172.21.15.62:0/1514330182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b40776e0 0x5597b40993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.247+0000 7f9109a00640 1 -- 172.21.15.62:0/1514330182 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597b3fe5f00 con 0x5597b40974f0 2024-03-23T11:21:38.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.247+0000 7f9109a00640 1 --2- 172.21.15.62:0/1514330182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40998e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f90fc006fd0 tx=0x7f90fc00daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cbf3dc0abe5931 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.248+0000 7f90fa000640 1 -- 172.21.15.62:0/1514330182 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc00f6e0 con 0x5597b40974f0 2024-03-23T11:21:38.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.248+0000 7f90fa000640 1 -- 172.21.15.62:0/1514330182 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90fc00c030 con 0x5597b40974f0 2024-03-23T11:21:38.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.248+0000 7f90fa000640 1 -- 172.21.15.62:0/1514330182 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc00faa0 con 0x5597b40974f0 2024-03-23T11:21:38.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.248+0000 7f910e7ce740 1 -- 172.21.15.62:0/1514330182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 msgr2=0x5597b40998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.248+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1514330182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40998e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f90fc006fd0 tx=0x7f90fc00daa0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.249+0000 7f910e7ce740 1 -- 172.21.15.62:0/1514330182 shutdown_connections 2024-03-23T11:21:38.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.249+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1514330182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40998e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.249+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1514330182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b40776e0 0x5597b40993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.249+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1514330182 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b404d570 0x5597b409ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.249+0000 7f910e7ce740 1 -- 172.21.15.62:0/1514330182 >> 172.21.15.62:0/1514330182 conn(0x5597b4049ca0 msgr2=0x5597b404d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.249+0000 7f910e7ce740 1 -- 172.21.15.62:0/1514330182 shutdown_connections 2024-03-23T11:21:38.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.249+0000 7f910e7ce740 1 -- 172.21.15.62:0/1514330182 wait complete. 2024-03-23T11:21:38.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.249+0000 7f910e7ce740 1 -- start start 2024-03-23T11:21:38.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.250+0000 7f910e7ce740 1 Processor -- start 2024-03-23T11:21:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b404d570 0x5597b4093e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b40776e0 0x5597b40e1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f910e7ce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f910e7ce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597b40e25e0 con 0x5597b40974f0 2024-03-23T11:21:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f910e7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597b40954b0 con 0x5597b404d570 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f910e7ce740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5597b4095700 con 0x5597b40776e0 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f910e7ce740 10 client.0 _ll_register_callbacks cb 0x7f90f4009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f9109a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f9109a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42692/0 (socket says 172.21.15.62:42692) 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f9109a00640 1 -- 172.21.15.62:0/1601364543 learned_addr learned my addr 172.21.15.62:0/1601364543 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f9109a00640 1 -- 172.21.15.62:0/1601364543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b40776e0 msgr2=0x5597b40e1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f9109a00640 1 --2- 172.21.15.62:0/1601364543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b40776e0 0x5597b40e1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f9109a00640 1 -- 172.21.15.62:0/1601364543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b404d570 msgr2=0x5597b4093e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f9109a00640 1 --2- 172.21.15.62:0/1601364543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b404d570 0x5597b4093e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.251+0000 7f9109a00640 1 -- 172.21.15.62:0/1601364543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc00e040 con 0x5597b40974f0 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.252+0000 7f9109a00640 1 --2- 172.21.15.62:0/1601364543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40984a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f90fc008010 tx=0x7f90fc006a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c54d58ab9cc4a4e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.252+0000 7f90f2000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.252+0000 7f90f2000640 1 -- 172.21.15.62:0/1601364543 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc022070 con 0x5597b40974f0 2024-03-23T11:21:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.252+0000 7f90f2000640 1 -- 172.21.15.62:0/1601364543 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90fc014480 con 0x5597b40974f0 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.252+0000 7f910e7ce740 1 -- 172.21.15.62:0/1601364543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5597b40a87a0 con 0x5597b40974f0 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.252+0000 7f90f2000640 1 -- 172.21.15.62:0/1601364543 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90fc014640 con 0x5597b40974f0 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.252+0000 7f910e7ce740 10 client.5850 fetch_fsmap learned FSMap version 25 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.252+0000 7f910e7ce740 1 -- 172.21.15.62:0/1601364543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5597b3f534c0 con 0x5597b40974f0 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.253+0000 7f90f2000640 1 -- 172.21.15.62:0/1601364543 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5597b40a87a0 con 0x5597b40974f0 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.253+0000 7f90f2000640 1 -- 172.21.15.62:0/1601364543 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f90fc0147a0 con 0x5597b40974f0 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.253+0000 7f910e7ce740 10 client.5850 fetch_fsmap finished waiting for FSMap version 25 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.253+0000 7f910e7ce740 10 client.5850 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.253+0000 7f910e7ce740 20 client.5850 populate_metadata read hostname 'smithi062' 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.253+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1601364543 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x5597b40e4650 0x5597b40993a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.253+0000 7f910e7ce740 4 client.5850 mds_command: new command op to 5652 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.253+0000 7f910e7ce740 1 -- 172.21.15.62:0/1601364543 --> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) v1 -- 0x7f90f403eb00 con 0x5597b40e4650 2024-03-23T11:21:38.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.253+0000 7f9109000640 1 --2- 172.21.15.62:0/1601364543 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x5597b40e4650 0x5597b40993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.253+0000 7f9109000640 1 --2- 172.21.15.62:0/1601364543 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x5597b40e4650 0x5597b40993a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f953a086dd554de8 server_cookie=f17efe894af9ba19 in_seq=0 out_seq=0 2024-03-23T11:21:38.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.253+0000 7f90f2000640 10 client.5850 ms_handle_connect on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:38.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.254+0000 7f90f2000640 1 -- 172.21.15.62:0/1601364543 <== mds.0 v2:172.21.15.62:6834/1159419089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7f90f403eb00 con 0x5597b40e4650 2024-03-23T11:21:38.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.254+0000 7f90f2000640 10 client.5850 handle_command_reply: tid=0 2024-03-23T11:21:38.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.254+0000 7f910e7ce740 1 client.5850 shutdown 2024-03-23T11:21:38.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f910e7ce740 1 -- 172.21.15.62:0/1601364543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 msgr2=0x5597b40984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1601364543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40984a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f90fc008010 tx=0x7f90fc006a20 comp rx=0 tx=0).stop 2024-03-23T11:21:38.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f910e7ce740 1 -- 172.21.15.62:0/1601364543 shutdown_connections 2024-03-23T11:21:38.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1601364543 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x5597b40e4650 0x5597b40993a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1601364543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5597b40974f0 0x5597b40984a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1601364543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597b40776e0 0x5597b40e1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f910e7ce740 1 --2- 172.21.15.62:0/1601364543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597b404d570 0x5597b4093e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f910e7ce740 1 -- 172.21.15.62:0/1601364543 >> 172.21.15.62:0/1601364543 conn(0x5597b4049ca0 msgr2=0x5597b404a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f90f2000640 0 client.5850 ms_handle_reset on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:38.255 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:21:38.255+0000 7ff6f3c00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5808=0-4194304@1} caps={5808=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5808 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x557d5f12e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-23T11:21:38.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f910e7ce740 1 -- 172.21.15.62:0/1601364543 shutdown_connections 2024-03-23T11:21:38.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f910e7ce740 1 -- 172.21.15.62:0/1601364543 wait complete. 2024-03-23T11:21:38.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.255+0000 7f910e7ce740 20 client.5850 trim_cache size 0 max 16384 2024-03-23T11:21:38.256 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:21:38.256 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-03-23T11:21:38.257 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "65e9eb4c-5561-4830-a003-faf678f23fd2", 2024-03-23T11:21:38.257 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-03-23T11:21:38.257 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:21:38.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 -- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f90e006a7d0 msgr2=0x7f90e006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f90e006a7d0 0x7f90e006cc90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f90f4035000 tx=0x7f90f4030000 comp rx=0 tx=0).stop 2024-03-23T11:21:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 -- 172.21.15.62:0/4120440295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91040c9d70 msgr2=0x7f91041f2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91040c9d70 0x7f91041f2220 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00d800 tx=0x7f90ec00dcc0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 -- 172.21.15.62:0/4120440295 shutdown_connections 2024-03-23T11:21:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f90e006a7d0 0x7f90e006cc90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91040c9d70 0x7f91041f2220 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f910415cd00 0x7f910406d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 --2- 172.21.15.62:0/4120440295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91040ade20 0x7f91040b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 -- 172.21.15.62:0/4120440295 >> 172.21.15.62:0/4120440295 conn(0x7f91040989d0 msgr2=0x7f91040cc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 -- 172.21.15.62:0/4120440295 shutdown_connections 2024-03-23T11:21:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.256+0000 7f910a400640 1 -- 172.21.15.62:0/4120440295 wait complete. 2024-03-23T11:21:38.272 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2024-03-23T11:21:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.344+0000 7fd8f4c00640 1 -- 172.21.15.62:0/2192499017 <== mon.0 v2:172.21.15.62:3300/0 7 ==== log(6 entries from seq 1 at 2024-03-23T11:21:38.255021+0000) v1 ==== 1078+0+0 (secure 0 0 0) 0x7fd8f0003e40 con 0x7fd8f815ab50 2024-03-23T11:21:38.343 INFO:teuthology.orchestra.run.smithi062.stdout: cluster: 2024-03-23T11:21:38.343 INFO:teuthology.orchestra.run.smithi062.stdout: id: e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:21:38.343 INFO:teuthology.orchestra.run.smithi062.stdout: health: HEALTH_OK 2024-03-23T11:21:38.343 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:38.343 INFO:teuthology.orchestra.run.smithi062.stdout: services: 2024-03-23T11:21:38.343 INFO:teuthology.orchestra.run.smithi062.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2024-03-23T11:21:38.343 INFO:teuthology.orchestra.run.smithi062.stdout: mgr: x(active, since 2m), standbys: y 2024-03-23T11:21:38.343 INFO:teuthology.orchestra.run.smithi062.stdout: mds: 1/1 daemons up, 3 standby 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: data: 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: volumes: 1/1 healthy 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: pools: 3 pools, 97 pgs 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: objects: 24 objects, 579 KiB 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: usage: 225 MiB used, 720 GiB / 720 GiB avail 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: pgs: 97 active+clean 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: io: 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: client: 1.3 KiB/s wr, 0 op/s rd, 4 op/s wr 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:21:38.255021+0000 mds.a [INF] scrub queued for path: / 2024-03-23T11:21:38.344 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:21:38.255025+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-23T11:21:38.345 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:21:38.255029+0000 mds.a [INF] scrub summary: active paths [/] 2024-03-23T11:21:38.345 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:21:38.256259+0000 mds.a [WRN] bad backtrace on inode 0x10000000001(/parent_a/alpha), rewriting it 2024-03-23T11:21:38.345 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:21:38.256268+0000 mds.a [INF] Scrub repaired inode 0x10000000001 (/parent_a/alpha) 2024-03-23T11:21:38.345 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:21:38.256558+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-23T11:21:38.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.473+0000 7fa162800640 1 Processor -- start 2024-03-23T11:21:38.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.473+0000 7fa162800640 1 -- start start 2024-03-23T11:21:38.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.474+0000 7fa162800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa15c15c9e0 0x7fa15c1607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.475+0000 7fa162800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c06bfc0 con 0x7fa15c0cf650 2024-03-23T11:21:38.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.475+0000 7fa162800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c0b5260 con 0x7fa15c15cdc0 2024-03-23T11:21:38.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.475+0000 7fa162800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa15c0d2630 con 0x7fa15c15c9e0 2024-03-23T11:21:38.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.475+0000 7fa15be00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa15c0cf650 0x7fa15c15d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47288/0 (socket says 172.21.15.62:47288) 2024-03-23T11:21:38.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.475+0000 7fa15be00640 1 -- 172.21.15.62:0/2176002225 learned_addr learned my addr 172.21.15.62:0/2176002225 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:21:38.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.475+0000 7fa15b400640 1 --2- 172.21.15.62:0/2176002225 >> v2:172.21.15.120:3300/0 conn(0x7fa15c15c9e0 0x7fa15c1607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.475+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771240533 0 0) 0x7fa15c06bfc0 con 0x7fa15c0cf650 2024-03-23T11:21:38.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.475+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa140003520 con 0x7fa15c0cf650 2024-03-23T11:21:38.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4044761050 0 0) 0x7fa140003520 con 0x7fa15c0cf650 2024-03-23T11:21:38.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 >> v2:172.21.15.120:3300/0 conn(0x7fa15c15c9e0 msgr2=0x7fa15c1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 --2- 172.21.15.62:0/2176002225 >> v2:172.21.15.120:3300/0 conn(0x7fa15c15c9e0 0x7fa15c1607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 >> v1:172.21.15.120:6789/0 conn(0x7fa15c15cdc0 legacy=0x7fa15c0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c0ce140 con 0x7fa15c0cf650 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fa154001670 con 0x7fa15c0cf650 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 >> v1:172.21.15.62:6789/0 conn(0x7fa15c0cf650 legacy=0x7fa15c15d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 0x7fa140003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1400047c0 0x7fa140004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa140005180 0x7fa14000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c0d2630 con 0x7fa1400039c0 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14000c080 con 0x7fa140005180 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14000c240 con 0x7fa1400047c0 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.476+0000 7fa15b400640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 0x7fa140003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.477+0000 7fa15be00640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa140005180 0x7fa14000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.477+0000 7fa15aa00640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1400047c0 0x7fa140004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.477+0000 7fa15b400640 1 -- 172.21.15.62:0/2176002225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1400047c0 msgr2=0x7fa140004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.477+0000 7fa15b400640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1400047c0 0x7fa140004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.477+0000 7fa15b400640 1 -- 172.21.15.62:0/2176002225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa140005180 msgr2=0x7fa14000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.477+0000 7fa15b400640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa140005180 0x7fa14000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.477+0000 7fa15b400640 1 -- 172.21.15.62:0/2176002225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa140003520 con 0x7fa1400039c0 2024-03-23T11:21:38.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.477+0000 7fa15be00640 1 -- 172.21.15.62:0/2176002225 reap_dead start 2024-03-23T11:21:38.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.477+0000 7fa15aa00640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1400047c0 0x7fa140004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:38.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.478+0000 7fa15b400640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 0x7fa140003de0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa14c009fc0 tx=0x7fa1400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1e496460568d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.478+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c02cc20 con 0x7fa1400039c0 2024-03-23T11:21:38.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.478+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14c00f040 con 0x7fa1400039c0 2024-03-23T11:21:38.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.478+0000 7fa15a000640 1 -- 172.21.15.62:0/2176002225 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c034440 con 0x7fa1400039c0 2024-03-23T11:21:38.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.479+0000 7fa162800640 1 -- 172.21.15.62:0/2176002225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 msgr2=0x7fa140003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.479+0000 7fa162800640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 0x7fa140003de0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa14c009fc0 tx=0x7fa1400057f0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.479+0000 7fa162800640 1 -- 172.21.15.62:0/2176002225 shutdown_connections 2024-03-23T11:21:38.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.479+0000 7fa162800640 1 --2- 172.21.15.62:0/2176002225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 0x7fa140003de0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.479+0000 7fa162800640 1 -- 172.21.15.62:0/2176002225 >> 172.21.15.62:0/2176002225 conn(0x7fa15c071570 msgr2=0x7fa15c071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.479+0000 7fa162800640 1 -- 172.21.15.62:0/2176002225 shutdown_connections 2024-03-23T11:21:38.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.479+0000 7fa162800640 1 -- 172.21.15.62:0/2176002225 wait complete. 2024-03-23T11:21:38.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.481+0000 7fa162800640 1 Processor -- start 2024-03-23T11:21:38.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.481+0000 7fa162800640 1 -- start start 2024-03-23T11:21:38.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.482+0000 7fa162800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 0x7fa15c0cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.482+0000 7fa162800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa15c15e400 0x7fa15c15e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.482+0000 7fa162800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa15c1f4d00 0x7fa15c1f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.482+0000 7fa162800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c0c9270 con 0x7fa1400039c0 2024-03-23T11:21:38.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.482+0000 7fa162800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c1f56c0 con 0x7fa15c1f4d00 2024-03-23T11:21:38.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.482+0000 7fa162800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c1f21c0 con 0x7fa15c15e400 2024-03-23T11:21:38.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.482+0000 7fa15b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 0x7fa15c0cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.482+0000 7fa15b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 0x7fa15c0cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42704/0 (socket says 172.21.15.62:42704) 2024-03-23T11:21:38.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.482+0000 7fa15b400640 1 -- 172.21.15.62:0/2041073677 learned_addr learned my addr 172.21.15.62:0/2041073677 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.483+0000 7fa15aa00640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa15c15e400 0x7fa15c15e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.483+0000 7fa15be00640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa15c1f4d00 0x7fa15c1f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.483+0000 7fa15b400640 1 -- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa15c15e400 msgr2=0x7fa15c15e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.483+0000 7fa15b400640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa15c15e400 0x7fa15c15e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.483+0000 7fa15b400640 1 -- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa15c1f4d00 msgr2=0x7fa15c1f5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.483+0000 7fa15b400640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa15c1f4d00 0x7fa15c1f5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.483+0000 7fa15b400640 1 -- 172.21.15.62:0/2041073677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c0c85f0 con 0x7fa1400039c0 2024-03-23T11:21:38.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.483+0000 7fa15b400640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 0x7fa15c0cbe00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa14c02c690 tx=0x7fa14c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.483+0000 7fa153400640 1 -- 172.21.15.62:0/2041073677 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c034c50 con 0x7fa1400039c0 2024-03-23T11:21:38.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.484+0000 7fa153400640 1 -- 172.21.15.62:0/2041073677 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14c034dd0 con 0x7fa1400039c0 2024-03-23T11:21:38.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.484+0000 7fa153400640 1 -- 172.21.15.62:0/2041073677 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c003bd0 con 0x7fa1400039c0 2024-03-23T11:21:38.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.484+0000 7fa162800640 1 -- 172.21.15.62:0/2041073677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15c0c88d0 con 0x7fa1400039c0 2024-03-23T11:21:38.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.484+0000 7fa162800640 1 -- 172.21.15.62:0/2041073677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15c0c8e90 con 0x7fa1400039c0 2024-03-23T11:21:38.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.486+0000 7fa153400640 1 -- 172.21.15.62:0/2041073677 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa14c040450 con 0x7fa1400039c0 2024-03-23T11:21:38.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.487+0000 7fa153400640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa13806a7a0 0x7fa13806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.487+0000 7fa153400640 1 -- 172.21.15.62:0/2041073677 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa14c0ab4d0 con 0x7fa1400039c0 2024-03-23T11:21:38.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.487+0000 7fa15aa00640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa13806a7a0 0x7fa13806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.488+0000 7fa1653be740 1 Processor -- start 2024-03-23T11:21:38.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.488+0000 7fa1653be740 1 -- start start 2024-03-23T11:21:38.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.489+0000 7fa15aa00640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa13806a7a0 0x7fa13806cc60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa15c152bc0 tx=0x7fa144009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.489+0000 7fa1653be740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234f7eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.489+0000 7fa1653be740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 0x564234f56a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.489+0000 7fa1653be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 0x564234f56f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.489+0000 7fa1653be740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564234f12cb0 con 0x564234f76910 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.489+0000 7fa1653be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x564234ec1500 con 0x564234f76cf0 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.489+0000 7fa1653be740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x564234ebec90 con 0x564234f7f280 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.489+0000 7fa15b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234f7eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.489+0000 7fa15b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234f7eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42714/0 (socket says 172.21.15.62:42714) 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.489+0000 7fa15b400640 1 -- 172.21.15.62:0/1026045495 learned_addr learned my addr 172.21.15.62:0/1026045495 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.489+0000 7fa15be00640 1 --2- 172.21.15.62:0/1026045495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 0x564234f56f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.490+0000 7fa15b400640 1 -- 172.21.15.62:0/1026045495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 msgr2=0x564234f56a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.490+0000 7fa15b400640 1 --2- 172.21.15.62:0/1026045495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 0x564234f56a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.490+0000 7fa15b400640 1 -- 172.21.15.62:0/1026045495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 msgr2=0x564234f56f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.490+0000 7fa15b400640 1 --2- 172.21.15.62:0/1026045495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 0x564234f56f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.490+0000 7fa15b400640 1 -- 172.21.15.62:0/1026045495 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564234e6c360 con 0x564234f76910 2024-03-23T11:21:38.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.490+0000 7fa15be00640 1 --2- 172.21.15.62:0/1026045495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 0x564234f56f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:38.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.490+0000 7fa15b400640 1 --2- 172.21.15.62:0/1026045495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234f7eb80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa14c076bc0 tx=0x7fa14c07a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8bfc1fc87b619c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.490+0000 7fa14be00640 1 -- 172.21.15.62:0/1026045495 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c082ab0 con 0x564234f76910 2024-03-23T11:21:38.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.490+0000 7fa14be00640 1 -- 172.21.15.62:0/1026045495 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14c078a60 con 0x564234f76910 2024-03-23T11:21:38.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.490+0000 7fa14be00640 1 -- 172.21.15.62:0/1026045495 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c076dd0 con 0x564234f76910 2024-03-23T11:21:38.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.491+0000 7fa1653be740 1 -- 172.21.15.62:0/1026045495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 msgr2=0x564234f7eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.491+0000 7fa1653be740 1 --2- 172.21.15.62:0/1026045495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234f7eb80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa14c076bc0 tx=0x7fa14c07a0e0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.491+0000 7fa1653be740 1 -- 172.21.15.62:0/1026045495 shutdown_connections 2024-03-23T11:21:38.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.491+0000 7fa1653be740 1 --2- 172.21.15.62:0/1026045495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 0x564234f56f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.491+0000 7fa1653be740 1 --2- 172.21.15.62:0/1026045495 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 0x564234f56a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.491+0000 7fa1653be740 1 --2- 172.21.15.62:0/1026045495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234f7eb80 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.491+0000 7fa1653be740 1 -- 172.21.15.62:0/1026045495 >> 172.21.15.62:0/1026045495 conn(0x564234f1eb40 msgr2=0x564234f9c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.491+0000 7fa1653be740 1 -- 172.21.15.62:0/1026045495 shutdown_connections 2024-03-23T11:21:38.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.491+0000 7fa1653be740 1 -- 172.21.15.62:0/1026045495 wait complete. 2024-03-23T11:21:38.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.492+0000 7fa1653be740 1 -- start start 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.492+0000 7fa1653be740 1 Processor -- start 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.493+0000 7fa1653be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76910 0x564234f80a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.493+0000 7fa1653be740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76cf0 0x564234f57520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.493+0000 7fa1653be740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 0x564234f57a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.493+0000 7fa1653be740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564234eba580 con 0x564234f76cf0 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.493+0000 7fa1653be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x564234e5ba50 con 0x564234f76910 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.493+0000 7fa1653be740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x564234e5b6f0 con 0x564234f7f280 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.493+0000 7fa1653be740 10 client.0 _ll_register_callbacks cb 0x564234eaaed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76cf0 0x564234f57520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76cf0 0x564234f57520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42716/0 (socket says 172.21.15.62:42716) 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15aa00640 1 -- 172.21.15.62:0/2580341474 learned_addr learned my addr 172.21.15.62:0/2580341474 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15b400640 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76910 0x564234f80a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15be00640 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 0x564234f57a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15aa00640 1 -- 172.21.15.62:0/2580341474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 msgr2=0x564234f57a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15aa00640 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 0x564234f57a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15aa00640 1 -- 172.21.15.62:0/2580341474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76910 msgr2=0x564234f80a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15aa00640 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76910 0x564234f80a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15aa00640 1 -- 172.21.15.62:0/2580341474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15c0c8e90 con 0x564234f76cf0 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15be00640 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 0x564234f57a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15b400640 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76910 0x564234f80a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa15aa00640 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76cf0 0x564234f57520 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa144003b70 tx=0x7fa144003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43488b86925f73fe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.494+0000 7fa148c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.495+0000 7fa148c00640 1 -- 172.21.15.62:0/2580341474 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14400fc10 con 0x564234f76cf0 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.495+0000 7fa148c00640 1 -- 172.21.15.62:0/2580341474 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa14400fd70 con 0x564234f76cf0 2024-03-23T11:21:38.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.495+0000 7fa1653be740 1 -- 172.21.15.62:0/2580341474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564234ee4fe0 con 0x564234f76cf0 2024-03-23T11:21:38.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.495+0000 7fa148c00640 1 -- 172.21.15.62:0/2580341474 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1440133f0 con 0x564234f76cf0 2024-03-23T11:21:38.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.495+0000 7fa1653be740 10 client.5862 fetch_fsmap learned FSMap version 25 2024-03-23T11:21:38.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.495+0000 7fa1653be740 1 -- 172.21.15.62:0/2580341474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x564234eb8c00 con 0x564234f76cf0 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.495+0000 7fa148c00640 1 -- 172.21.15.62:0/2580341474 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564234ee4fe0 con 0x564234f76cf0 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.495+0000 7fa148c00640 1 -- 172.21.15.62:0/2580341474 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa144012070 con 0x564234f76cf0 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.496+0000 7fa1653be740 10 client.5862 fetch_fsmap finished waiting for FSMap version 25 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.496+0000 7fa1653be740 10 client.5862 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.496+0000 7fa1653be740 20 client.5862 populate_metadata read hostname 'smithi062' 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.496+0000 7fa1653be740 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x564234f82300 0x564234fac000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.496+0000 7fa1653be740 4 client.5862 mds_command: new command op to 5652 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.496+0000 7fa1653be740 1 -- 172.21.15.62:0/2580341474 --> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564234ead240 con 0x564234f82300 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.496+0000 7fa15b400640 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x564234f82300 0x564234fac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.496+0000 7fa15b400640 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x564234f82300 0x564234fac000 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c478f40caff110b server_cookie=a07dc1ac3e77b188 in_seq=0 out_seq=0 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.496+0000 7fa148c00640 10 client.5862 ms_handle_connect on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa148c00640 1 -- 172.21.15.62:0/2580341474 <== mds.0 v2:172.21.15.62:6834/1159419089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x564234ead240 con 0x564234f82300 2024-03-23T11:21:38.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa148c00640 10 client.5862 handle_command_reply: tid=0 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa1653be740 1 client.5862 shutdown 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa1653be740 1 -- 172.21.15.62:0/2580341474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76cf0 msgr2=0x564234f57520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa1653be740 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76cf0 0x564234f57520 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa144003b70 tx=0x7fa144003f70 comp rx=0 tx=0).stop 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa1653be740 1 -- 172.21.15.62:0/2580341474 shutdown_connections 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa1653be740 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x564234f82300 0x564234fac000 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa1653be740 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76cf0 0x564234f57520 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa1653be740 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 0x564234f57a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa1653be740 1 --2- 172.21.15.62:0/2580341474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76910 0x564234f80a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa148c00640 0 client.5862 ms_handle_reset on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.497+0000 7fa1653be740 1 -- 172.21.15.62:0/2580341474 >> 172.21.15.62:0/2580341474 conn(0x564234f1eb40 msgr2=0x564234f75d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.498+0000 7fa1653be740 1 -- 172.21.15.62:0/2580341474 shutdown_connections 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.498+0000 7fa1653be740 1 -- 172.21.15.62:0/2580341474 wait complete. 2024-03-23T11:21:38.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.498+0000 7fa1653be740 20 client.5862 trim_cache size 0 max 16384 2024-03-23T11:21:38.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.514+0000 7fa1653be740 1 Processor -- start 2024-03-23T11:21:38.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.514+0000 7fa1653be740 1 -- start start 2024-03-23T11:21:38.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.515+0000 7fa1653be740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234faace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.515+0000 7fa1653be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 0x564234f73110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.515+0000 7fa1653be740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 0x564234f9fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.515+0000 7fa1653be740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564234bb7f90 con 0x564234f76910 2024-03-23T11:21:38.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.515+0000 7fa1653be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x564234f58c90 con 0x564234f76cf0 2024-03-23T11:21:38.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.515+0000 7fa1653be740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x564234f51300 con 0x564234f7f280 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.515+0000 7fa15b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234faace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.515+0000 7fa15b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234faace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42724/0 (socket says 172.21.15.62:42724) 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.515+0000 7fa15b400640 1 -- 172.21.15.62:0/3319450738 learned_addr learned my addr 172.21.15.62:0/3319450738 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.515+0000 7fa15aa00640 1 --2- 172.21.15.62:0/3319450738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 0x564234f73110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.516+0000 7fa15b400640 1 -- 172.21.15.62:0/3319450738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 msgr2=0x564234f9fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.516+0000 7fa15b400640 1 --2- 172.21.15.62:0/3319450738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 0x564234f9fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.516+0000 7fa15b400640 1 -- 172.21.15.62:0/3319450738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 msgr2=0x564234f73110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.516+0000 7fa15b400640 1 --2- 172.21.15.62:0/3319450738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 0x564234f73110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.516+0000 7fa15b400640 1 -- 172.21.15.62:0/3319450738 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564234e5ac60 con 0x564234f76910 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.516+0000 7fa15aa00640 1 --2- 172.21.15.62:0/3319450738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 0x564234f73110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.516+0000 7fa15b400640 1 --2- 172.21.15.62:0/3319450738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234faace0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa14c082580 tx=0x7fa14c07a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbad92bd3246830b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.516+0000 7fa150c00640 1 -- 172.21.15.62:0/3319450738 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c0a8040 con 0x564234f76910 2024-03-23T11:21:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.516+0000 7fa150c00640 1 -- 172.21.15.62:0/3319450738 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14c082b30 con 0x564234f76910 2024-03-23T11:21:38.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.516+0000 7fa150c00640 1 -- 172.21.15.62:0/3319450738 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c076dd0 con 0x564234f76910 2024-03-23T11:21:38.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.517+0000 7fa1653be740 1 -- 172.21.15.62:0/3319450738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 msgr2=0x564234faace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.517+0000 7fa1653be740 1 --2- 172.21.15.62:0/3319450738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234faace0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa14c082580 tx=0x7fa14c07a150 comp rx=0 tx=0).stop 2024-03-23T11:21:38.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.517+0000 7fa1653be740 1 -- 172.21.15.62:0/3319450738 shutdown_connections 2024-03-23T11:21:38.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.517+0000 7fa1653be740 1 --2- 172.21.15.62:0/3319450738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f7f280 0x564234f9fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.517+0000 7fa1653be740 1 --2- 172.21.15.62:0/3319450738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f76cf0 0x564234f73110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.517+0000 7fa1653be740 1 --2- 172.21.15.62:0/3319450738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234f76910 0x564234faace0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.517+0000 7fa1653be740 1 -- 172.21.15.62:0/3319450738 >> 172.21.15.62:0/3319450738 conn(0x564234f1eb40 msgr2=0x564234f9f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.517+0000 7fa1653be740 1 -- 172.21.15.62:0/3319450738 shutdown_connections 2024-03-23T11:21:38.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.517+0000 7fa1653be740 1 -- 172.21.15.62:0/3319450738 wait complete. 2024-03-23T11:21:38.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.518+0000 7fa1653be740 1 -- start start 2024-03-23T11:21:38.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.518+0000 7fa1653be740 1 Processor -- start 2024-03-23T11:21:38.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.519+0000 7fa1653be740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f76cf0 0x564234ffb640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.519+0000 7fa1653be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f7f280 0x564234f73110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.519+0000 7fa1653be740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234fac000 0x564234f73650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.519+0000 7fa1653be740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564234f55710 con 0x564234fac000 2024-03-23T11:21:38.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.519+0000 7fa1653be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x564234f9ec70 con 0x564234f7f280 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.519+0000 7fa1653be740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x564234f9eec0 con 0x564234f76cf0 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.519+0000 7fa1653be740 10 client.0 _ll_register_callbacks cb 0x7fa1440013b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.519+0000 7fa15be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234fac000 0x564234f73650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.519+0000 7fa15be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234fac000 0x564234f73650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42726/0 (socket says 172.21.15.62:42726) 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.519+0000 7fa15be00640 1 -- 172.21.15.62:0/135186229 learned_addr learned my addr 172.21.15.62:0/135186229 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa15b400640 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f76cf0 0x564234ffb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa15aa00640 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f7f280 0x564234f73110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa15be00640 1 -- 172.21.15.62:0/135186229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f76cf0 msgr2=0x564234ffb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa15be00640 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f76cf0 0x564234ffb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa15be00640 1 -- 172.21.15.62:0/135186229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f7f280 msgr2=0x564234f73110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa15be00640 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f7f280 0x564234f73110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa15be00640 1 -- 172.21.15.62:0/135186229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564234e6c360 con 0x564234fac000 2024-03-23T11:21:38.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa15b400640 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f76cf0 0x564234ffb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa15be00640 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234fac000 0x564234f73650 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa15c06bc00 tx=0x7fa154011680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f0ee5e6ba0610 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa148c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa148c00640 1 -- 172.21.15.62:0/135186229 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154014780 con 0x564234fac000 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa148c00640 1 -- 172.21.15.62:0/135186229 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1540148e0 con 0x564234fac000 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.520+0000 7fa1653be740 1 -- 172.21.15.62:0/135186229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa14c078a60 con 0x564234fac000 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa148c00640 1 -- 172.21.15.62:0/135186229 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa154011ec0 con 0x564234fac000 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa1653be740 10 client.5868 fetch_fsmap learned FSMap version 25 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa1653be740 1 -- 172.21.15.62:0/135186229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x564234e80170 con 0x564234fac000 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa148c00640 1 -- 172.21.15.62:0/135186229 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14c078a60 con 0x564234fac000 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa148c00640 1 -- 172.21.15.62:0/135186229 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa154014aa0 con 0x564234fac000 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa1653be740 10 client.5868 fetch_fsmap finished waiting for FSMap version 25 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa1653be740 10 client.5868 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-23T11:21:38.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa1653be740 20 client.5868 populate_metadata read hostname 'smithi062' 2024-03-23T11:21:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa1653be740 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x564235004b30 0x564235004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:21:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa1653be740 4 client.5868 mds_command: new command op to 5652 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:21:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa1653be740 1 -- 172.21.15.62:0/135186229 --> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564234bae4e0 con 0x564235004b30 2024-03-23T11:21:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.521+0000 7fa15b400640 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x564235004b30 0x564235004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:21:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa15b400640 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x564235004b30 0x564235004f40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8db286ea50a75c2d server_cookie=76d00bf54e43d795 in_seq=0 out_seq=0 2024-03-23T11:21:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa148c00640 10 client.5868 ms_handle_connect on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa148c00640 1 -- 172.21.15.62:0/135186229 <== mds.0 v2:172.21.15.62:6834/1159419089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x564234bae4e0 con 0x564235004b30 2024-03-23T11:21:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa148c00640 10 client.5868 handle_command_reply: tid=0 2024-03-23T11:21:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa1653be740 1 client.5868 shutdown 2024-03-23T11:21:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa1653be740 1 -- 172.21.15.62:0/135186229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234fac000 msgr2=0x564234f73650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa1653be740 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234fac000 0x564234f73650 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa15c06bc00 tx=0x7fa154011680 comp rx=0 tx=0).stop 2024-03-23T11:21:38.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa1653be740 1 -- 172.21.15.62:0/135186229 shutdown_connections 2024-03-23T11:21:38.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa1653be740 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x564235004b30 0x564235004f40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa1653be740 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564234fac000 0x564234f73650 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa1653be740 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564234f7f280 0x564234f73110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa1653be740 1 --2- 172.21.15.62:0/135186229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564234f76cf0 0x564234ffb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa148c00640 0 client.5868 ms_handle_reset on v2:172.21.15.62:6834/1159419089 2024-03-23T11:21:38.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.522+0000 7fa1653be740 1 -- 172.21.15.62:0/135186229 >> 172.21.15.62:0/135186229 conn(0x564234f1eb40 msgr2=0x564234f9fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa1653be740 1 -- 172.21.15.62:0/135186229 shutdown_connections 2024-03-23T11:21:38.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa1653be740 1 -- 172.21.15.62:0/135186229 wait complete. 2024-03-23T11:21:38.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa1653be740 20 client.5868 trim_cache size 0 max 16384 2024-03-23T11:21:38.525 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:21:38.525 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T11:21:38.525 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-03-23T11:21:38.526 INFO:teuthology.orchestra.run.smithi062.stdout: "65e9eb4c-5561-4830-a003-faf678f23fd2": { 2024-03-23T11:21:38.526 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-03-23T11:21:38.526 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "65e9eb4c-5561-4830-a003-faf678f23fd2", 2024-03-23T11:21:38.526 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,repair" 2024-03-23T11:21:38.526 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:38.526 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:21:38.526 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:21:38.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa162800640 1 -- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa13806a7a0 msgr2=0x7fa13806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa162800640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa13806a7a0 0x7fa13806cc60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa15c152bc0 tx=0x7fa144009040 comp rx=0 tx=0).stop 2024-03-23T11:21:38.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa162800640 1 -- 172.21.15.62:0/2041073677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 msgr2=0x7fa15c0cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:21:38.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa162800640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 0x7fa15c0cbe00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa14c02c690 tx=0x7fa14c009750 comp rx=0 tx=0).stop 2024-03-23T11:21:38.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa162800640 1 -- 172.21.15.62:0/2041073677 shutdown_connections 2024-03-23T11:21:38.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa162800640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa13806a7a0 0x7fa13806cc60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa162800640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa15c1f4d00 0x7fa15c1f5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa162800640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa15c15e400 0x7fa15c15e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.523+0000 7fa162800640 1 --2- 172.21.15.62:0/2041073677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1400039c0 0x7fa15c0cbe00 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:21:38.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.524+0000 7fa162800640 1 -- 172.21.15.62:0/2041073677 >> 172.21.15.62:0/2041073677 conn(0x7fa15c071570 msgr2=0x7fa15c0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:21:38.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.524+0000 7fa162800640 1 -- 172.21.15.62:0/2041073677 shutdown_connections 2024-03-23T11:21:38.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:38.524+0000 7fa162800640 1 -- 172.21.15.62:0/2041073677 wait complete. 2024-03-23T11:21:38.539 INFO:tasks.cephfs.filesystem:scrub status for tag:65e9eb4c-5561-4830-a003-faf678f23fd2 - {'path': '/', 'tag': '65e9eb4c-5561-4830-a003-faf678f23fd2', 'options': 'recursive,repair'} 2024-03-23T11:21:41.362 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:21:40.316603+0000 mds.a [INF] scrub summary: idle 2024-03-23T11:21:41.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:41.364+0000 7fd8f4c00640 1 -- 172.21.15.62:0/2192499017 <== mon.0 v2:172.21.15.62:3300/0 8 ==== log(1 entries from seq 7 at 2024-03-23T11:21:40.316603+0000) v1 ==== 189+0+0 (secure 0 0 0) 0x7fd8f00585e0 con 0x7fd8f815ab50 2024-03-23T11:21:42.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:42.366+0000 7fd8f4c00640 1 -- 172.21.15.62:0/2192499017 <== mon.0 v2:172.21.15.62:3300/0 9 ==== log(2 entries from seq 555 at 2024-03-23T11:21:41.887580+0000) v1 ==== 596+0+0 (secure 0 0 0) 0x7fd8f005c410 con 0x7fd8f815ab50 2024-03-23T11:21:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:21:43.386+0000 7fd8f4c00640 1 -- 172.21.15.62:0/2192499017 <== mon.0 v2:172.21.15.62:3300/0 10 ==== log(2 entries from seq 557 at 2024-03-23T11:21:42.379160+0000) v1 ==== 600+0+0 (secure 0 0 0) 0x7fd8f00609d0 con 0x7fd8f815ab50 2024-03-23T11:21:45.108 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:21:45.116 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:22:08.541 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2024-03-23T11:22:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.777+0000 7fe7e8a00640 1 Processor -- start 2024-03-23T11:22:08.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.777+0000 7fe7e8a00640 1 -- start start 2024-03-23T11:22:08.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.778+0000 7fe7e8a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe7e41431f0 0x7fe7e40b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.778+0000 7fe7e8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7e40cc9b0 0x7fe7e40704d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.778+0000 7fe7e8a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e40b20d0 con 0x7fe7e4070090 2024-03-23T11:22:08.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.778+0000 7fe7e8a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e4070a10 con 0x7fe7e41431f0 2024-03-23T11:22:08.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.778+0000 7fe7e8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e41435d0 con 0x7fe7e40cc9b0 2024-03-23T11:22:08.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.779+0000 7fe7e3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7e40cc9b0 0x7fe7e40704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.779+0000 7fe7e3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7e40cc9b0 0x7fe7e40704d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:44618/0 (socket says 172.21.15.62:44618) 2024-03-23T11:22:08.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.779+0000 7fe7e3e00640 1 -- 172.21.15.62:0/3662953027 learned_addr learned my addr 172.21.15.62:0/3662953027 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:08.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.779+0000 7fe7e3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe7e41431f0 0x7fe7e40b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.780+0000 7fe7e3400640 1 -- 172.21.15.62:0/3662953027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7e40cc9b0 msgr2=0x7fe7e40704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:08.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.780+0000 7fe7e3400640 1 --2- 172.21.15.62:0/3662953027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7e40cc9b0 0x7fe7e40704d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.780+0000 7fe7e3400640 1 -- 172.21.15.62:0/3662953027 >> v1:172.21.15.120:6789/0 conn(0x7fe7e4070090 legacy=0x7fe7e40cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:08.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.780+0000 7fe7e3400640 1 -- 172.21.15.62:0/3662953027 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e40b14c0 con 0x7fe7e41431f0 2024-03-23T11:22:08.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.781+0000 7fe7e3400640 1 --2- 172.21.15.62:0/3662953027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7e41431f0 0x7fe7e40b2560 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009d90 tx=0x7fe7d402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f7749d6e345218c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:08.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.782+0000 7fe7e2000640 1 -- 172.21.15.62:0/3662953027 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d402cbc0 con 0x7fe7e41431f0 2024-03-23T11:22:08.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.782+0000 7fe7e2000640 1 -- 172.21.15.62:0/3662953027 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d402cd20 con 0x7fe7e41431f0 2024-03-23T11:22:08.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.782+0000 7fe7e2000640 1 -- 172.21.15.62:0/3662953027 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4034660 con 0x7fe7e41431f0 2024-03-23T11:22:08.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.782+0000 7fe7e8a00640 1 -- 172.21.15.62:0/3662953027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7e41431f0 msgr2=0x7fe7e40b2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:08.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.782+0000 7fe7e8a00640 1 --2- 172.21.15.62:0/3662953027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7e41431f0 0x7fe7e40b2560 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009d90 tx=0x7fe7d402be10 comp rx=0 tx=0).stop 2024-03-23T11:22:08.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.783+0000 7fe7e8a00640 1 -- 172.21.15.62:0/3662953027 shutdown_connections 2024-03-23T11:22:08.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.783+0000 7fe7e8a00640 1 --2- 172.21.15.62:0/3662953027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7e40cc9b0 0x7fe7e40704d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.783+0000 7fe7e8a00640 1 --2- 172.21.15.62:0/3662953027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7e41431f0 0x7fe7e40b2560 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.783+0000 7fe7e8a00640 1 -- 172.21.15.62:0/3662953027 >> 172.21.15.62:0/3662953027 conn(0x7fe7e40b0090 msgr2=0x7fe7e40b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:08.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.783+0000 7fe7e8a00640 1 -- 172.21.15.62:0/3662953027 shutdown_connections 2024-03-23T11:22:08.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.783+0000 7fe7e8a00640 1 -- 172.21.15.62:0/3662953027 wait complete. 2024-03-23T11:22:08.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.784+0000 7fe7e8a00640 1 Processor -- start 2024-03-23T11:22:08.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.785+0000 7fe7e8a00640 1 -- start start 2024-03-23T11:22:08.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.785+0000 7fe7e8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7e4070090 0x7fe7e41591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.785+0000 7fe7e8a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e40cc9b0 0x7fe7e41f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e8a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7e41431f0 0x7fe7e413fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e8a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e41f1f90 con 0x7fe7e40cc9b0 2024-03-23T11:22:08.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e8a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e4140a30 con 0x7fe7e41431f0 2024-03-23T11:22:08.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e4140d10 con 0x7fe7e4070090 2024-03-23T11:22:08.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e40cc9b0 0x7fe7e41f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e40cc9b0 0x7fe7e41f2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48374/0 (socket says 172.21.15.62:48374) 2024-03-23T11:22:08.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e2a00640 1 -- 172.21.15.62:0/1327091884 learned_addr learned my addr 172.21.15.62:0/1327091884 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:08.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e3400640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7e4070090 0x7fe7e41591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e3e00640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7e41431f0 0x7fe7e413fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e2a00640 1 -- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7e4070090 msgr2=0x7fe7e41591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:08.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7e4070090 0x7fe7e41591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e2a00640 1 -- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7e41431f0 msgr2=0x7fe7e413fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:08.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7e41431f0 0x7fe7e413fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.786+0000 7fe7e2a00640 1 -- 172.21.15.62:0/1327091884 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d4009a70 con 0x7fe7e40cc9b0 2024-03-23T11:22:08.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.787+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e40cc9b0 0x7fe7e41f2750 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00c450 tx=0x7fe7cc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:08.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.787+0000 7fe7db400640 1 -- 172.21.15.62:0/1327091884 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0052d0 con 0x7fe7e40cc9b0 2024-03-23T11:22:08.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.787+0000 7fe7db400640 1 -- 172.21.15.62:0/1327091884 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc005430 con 0x7fe7e40cc9b0 2024-03-23T11:22:08.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.787+0000 7fe7e8a00640 1 -- 172.21.15.62:0/1327091884 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e41f2230 con 0x7fe7e40cc9b0 2024-03-23T11:22:08.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.787+0000 7fe7db400640 1 -- 172.21.15.62:0/1327091884 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc012870 con 0x7fe7e40cc9b0 2024-03-23T11:22:08.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.787+0000 7fe7e8a00640 1 -- 172.21.15.62:0/1327091884 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e4140ff0 con 0x7fe7e40cc9b0 2024-03-23T11:22:08.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.790+0000 7fe7db400640 1 -- 172.21.15.62:0/1327091884 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe7cc0129d0 con 0x7fe7e40cc9b0 2024-03-23T11:22:08.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.790+0000 7fe7db400640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7c006a7a0 0x7fe7c006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.791+0000 7fe7db400640 1 -- 172.21.15.62:0/1327091884 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe7cc088950 con 0x7fe7e40cc9b0 2024-03-23T11:22:08.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.791+0000 7fe7e3400640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7c006a7a0 0x7fe7c006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.791+0000 7fe7ece2f740 1 Processor -- start 2024-03-23T11:22:08.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.792+0000 7fe7ece2f740 1 -- start start 2024-03-23T11:22:08.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.792+0000 7fe7e3400640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7c006a7a0 0x7fe7c006cc60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4036000 tx=0x7fe7d4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.793+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b38bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.793+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b363c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.793+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b364160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.793+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b31fcb0 con 0x55ad3b383ed0 2024-03-23T11:22:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.793+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b2ce500 con 0x55ad3b383af0 2024-03-23T11:22:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.793+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b2cbc90 con 0x55ad3b38c460 2024-03-23T11:22:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.793+0000 7fe7e3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b364160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.793+0000 7fe7e3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b364160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48376/0 (socket says 172.21.15.62:48376) 2024-03-23T11:22:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.793+0000 7fe7e3e00640 1 -- 172.21.15.62:0/861711043 learned_addr learned my addr 172.21.15.62:0/861711043 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.793+0000 7fe7e3400640 1 --2- 172.21.15.62:0/861711043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b38bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.793+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/861711043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b363c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.794+0000 7fe7e3e00640 1 -- 172.21.15.62:0/861711043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 msgr2=0x55ad3b363c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.794+0000 7fe7e3e00640 1 --2- 172.21.15.62:0/861711043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b363c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.794+0000 7fe7e3e00640 1 -- 172.21.15.62:0/861711043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 msgr2=0x55ad3b38bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.794+0000 7fe7e3e00640 1 --2- 172.21.15.62:0/861711043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b38bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.794+0000 7fe7e3e00640 1 -- 172.21.15.62:0/861711043 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad3b279360 con 0x55ad3b383ed0 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.794+0000 7fe7e3e00640 1 --2- 172.21.15.62:0/861711043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b364160 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc009b60 tx=0x7fe7dc009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473fde3c47304106 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.794+0000 7fe7d3e00640 1 -- 172.21.15.62:0/861711043 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc009c50 con 0x55ad3b383ed0 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.794+0000 7fe7d3e00640 1 -- 172.21.15.62:0/861711043 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7dc009db0 con 0x55ad3b383ed0 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.794+0000 7fe7d3e00640 1 -- 172.21.15.62:0/861711043 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc015660 con 0x55ad3b383ed0 2024-03-23T11:22:08.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.795+0000 7fe7ece2f740 1 -- 172.21.15.62:0/861711043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 msgr2=0x55ad3b364160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.795+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/861711043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b364160 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc009b60 tx=0x7fe7dc009240 comp rx=0 tx=0).stop 2024-03-23T11:22:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.795+0000 7fe7ece2f740 1 -- 172.21.15.62:0/861711043 shutdown_connections 2024-03-23T11:22:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.795+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/861711043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b364160 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.795+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/861711043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b363c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.795+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/861711043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b38bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.795+0000 7fe7ece2f740 1 -- 172.21.15.62:0/861711043 >> 172.21.15.62:0/861711043 conn(0x55ad3b32bb40 msgr2=0x55ad3b3a9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.795+0000 7fe7ece2f740 1 -- 172.21.15.62:0/861711043 shutdown_connections 2024-03-23T11:22:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.795+0000 7fe7ece2f740 1 -- 172.21.15.62:0/861711043 wait complete. 2024-03-23T11:22:08.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.796+0000 7fe7ece2f740 1 -- start start 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.796+0000 7fe7ece2f740 1 Processor -- start 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.797+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b390ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.797+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b3ad280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.797+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b3824d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.797+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b2c7580 con 0x55ad3b383ed0 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.797+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b268a50 con 0x55ad3b383af0 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.797+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b2686f0 con 0x55ad3b38c460 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.797+0000 7fe7ece2f740 10 client.0 _ll_register_callbacks cb 0x55ad3b2b7ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.797+0000 7fe7e2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b3ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.797+0000 7fe7e2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b3ad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48392/0 (socket says 172.21.15.62:48392) 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.797+0000 7fe7e2a00640 1 -- 172.21.15.62:0/992489524 learned_addr learned my addr 172.21.15.62:0/992489524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7e3400640 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b390ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7e3e00640 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b3824d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7e2a00640 1 -- 172.21.15.62:0/992489524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 msgr2=0x55ad3b3824d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b3824d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7e2a00640 1 -- 172.21.15.62:0/992489524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 msgr2=0x55ad3b390ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b390ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7e2a00640 1 -- 172.21.15.62:0/992489524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7dc009820 con 0x55ad3b383ed0 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7e3e00640 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b3824d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b3ad280 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc007fd0 tx=0x7fe7cc058220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdba949351f93cf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7d0c00640 1 -- 172.21.15.62:0/992489524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc012c80 con 0x55ad3b383ed0 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7d0c00640 1 -- 172.21.15.62:0/992489524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe7cc012de0 con 0x55ad3b383ed0 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.798+0000 7fe7ece2f740 1 -- 172.21.15.62:0/992489524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad3b2f1fe0 con 0x55ad3b383ed0 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7d0c00640 1 -- 172.21.15.62:0/992489524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7cc06f210 con 0x55ad3b383ed0 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7ece2f740 10 client.5877 fetch_fsmap learned FSMap version 25 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7ece2f740 1 -- 172.21.15.62:0/992489524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55ad3b2c5c00 con 0x55ad3b383ed0 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7d0c00640 1 -- 172.21.15.62:0/992489524 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ad3b2f1fe0 con 0x55ad3b383ed0 2024-03-23T11:22:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7d0c00640 1 -- 172.21.15.62:0/992489524 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe7cc019020 con 0x55ad3b383ed0 2024-03-23T11:22:08.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7ece2f740 10 client.5877 fetch_fsmap finished waiting for FSMap version 25 2024-03-23T11:22:08.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7ece2f740 10 client.5877 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-23T11:22:08.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7ece2f740 20 client.5877 populate_metadata read hostname 'smithi062' 2024-03-23T11:22:08.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55ad3b3b9180 0x55ad3b3b9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7ece2f740 4 client.5877 mds_command: new command op to 5652 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:22:08.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7ece2f740 1 -- 172.21.15.62:0/992489524 --> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad3b2ba240 con 0x55ad3b3b9180 2024-03-23T11:22:08.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.799+0000 7fe7e3400640 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55ad3b3b9180 0x55ad3b3b9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.800+0000 7fe7e3400640 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55ad3b3b9180 0x55ad3b3b9590 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b1741b85b5b1855 server_cookie=66734c119d42a2e4 in_seq=0 out_seq=0 2024-03-23T11:22:08.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.800+0000 7fe7d0c00640 10 client.5877 ms_handle_connect on v2:172.21.15.62:6834/1159419089 2024-03-23T11:22:08.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7d0c00640 1 -- 172.21.15.62:0/992489524 <== mds.0 v2:172.21.15.62:6834/1159419089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55ad3b2ba240 con 0x55ad3b3b9180 2024-03-23T11:22:08.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7d0c00640 10 client.5877 handle_command_reply: tid=0 2024-03-23T11:22:08.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 1 client.5877 shutdown 2024-03-23T11:22:08.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 1 -- 172.21.15.62:0/992489524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 msgr2=0x55ad3b3ad280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:08.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b3ad280 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc007fd0 tx=0x7fe7cc058220 comp rx=0 tx=0).stop 2024-03-23T11:22:08.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 1 -- 172.21.15.62:0/992489524 shutdown_connections 2024-03-23T11:22:08.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55ad3b3b9180 0x55ad3b3b9590 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b3ad280 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b3824d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/992489524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b390ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7d0c00640 0 client.5877 ms_handle_reset on v2:172.21.15.62:6834/1159419089 2024-03-23T11:22:08.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 1 -- 172.21.15.62:0/992489524 >> 172.21.15.62:0/992489524 conn(0x55ad3b32bb40 msgr2=0x55ad3b38aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:08.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 1 -- 172.21.15.62:0/992489524 shutdown_connections 2024-03-23T11:22:08.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 1 -- 172.21.15.62:0/992489524 wait complete. 2024-03-23T11:22:08.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.801+0000 7fe7ece2f740 20 client.5877 trim_cache size 0 max 16384 2024-03-23T11:22:08.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.817+0000 7fe7ece2f740 1 Processor -- start 2024-03-23T11:22:08.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.817+0000 7fe7ece2f740 1 -- start start 2024-03-23T11:22:08.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.818+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b3c3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.818+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b362480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.818+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b3ac4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.818+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3afc4f90 con 0x55ad3b383ed0 2024-03-23T11:22:08.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.818+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b35a4f0 con 0x55ad3b383af0 2024-03-23T11:22:08.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.818+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b3cbdb0 con 0x55ad3b38c460 2024-03-23T11:22:08.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.818+0000 7fe7e2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b362480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.818+0000 7fe7e2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b362480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48408/0 (socket says 172.21.15.62:48408) 2024-03-23T11:22:08.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.818+0000 7fe7e2a00640 1 -- 172.21.15.62:0/2101829559 learned_addr learned my addr 172.21.15.62:0/2101829559 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:08.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.819+0000 7fe7e3400640 1 --2- 172.21.15.62:0/2101829559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b3c3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.819+0000 7fe7e2a00640 1 -- 172.21.15.62:0/2101829559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 msgr2=0x55ad3b3ac4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:08.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.819+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/2101829559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b3ac4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.819+0000 7fe7e2a00640 1 -- 172.21.15.62:0/2101829559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 msgr2=0x55ad3b3c3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:08.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.819+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/2101829559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b3c3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.819+0000 7fe7e2a00640 1 -- 172.21.15.62:0/2101829559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad3b267c60 con 0x55ad3b383ed0 2024-03-23T11:22:08.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.819+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/2101829559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b362480 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0576e0 tx=0x7fe7cc05a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536512e3d3e37e0c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:08.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.819+0000 7fe7d8c00640 1 -- 172.21.15.62:0/2101829559 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc06f210 con 0x55ad3b383ed0 2024-03-23T11:22:08.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.819+0000 7fe7d8c00640 1 -- 172.21.15.62:0/2101829559 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc00e040 con 0x55ad3b383ed0 2024-03-23T11:22:08.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.819+0000 7fe7d8c00640 1 -- 172.21.15.62:0/2101829559 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc05b810 con 0x55ad3b383ed0 2024-03-23T11:22:08.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.820+0000 7fe7ece2f740 1 -- 172.21.15.62:0/2101829559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 msgr2=0x55ad3b362480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:08.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.820+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/2101829559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b362480 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0576e0 tx=0x7fe7cc05a000 comp rx=0 tx=0).stop 2024-03-23T11:22:08.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.820+0000 7fe7ece2f740 1 -- 172.21.15.62:0/2101829559 shutdown_connections 2024-03-23T11:22:08.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.820+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/2101829559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b362480 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.820+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/2101829559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b38c460 0x55ad3b3ac4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.820+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/2101829559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b383af0 0x55ad3b3c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.820+0000 7fe7ece2f740 1 -- 172.21.15.62:0/2101829559 >> 172.21.15.62:0/2101829559 conn(0x55ad3b32bb40 msgr2=0x55ad3b3b9780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:08.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.820+0000 7fe7ece2f740 1 -- 172.21.15.62:0/2101829559 shutdown_connections 2024-03-23T11:22:08.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.820+0000 7fe7ece2f740 1 -- 172.21.15.62:0/2101829559 wait complete. 2024-03-23T11:22:08.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.821+0000 7fe7ece2f740 1 -- start start 2024-03-23T11:22:08.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.821+0000 7fe7ece2f740 1 Processor -- start 2024-03-23T11:22:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.822+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b383af0 0x55ad3b4178b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.822+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b38bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.822+0000 7fe7ece2f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b38c460 0x55ad3b38faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.822+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b3b8f20 con 0x55ad3b383ed0 2024-03-23T11:22:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.822+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b3909c0 con 0x55ad3b38c460 2024-03-23T11:22:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.822+0000 7fe7ece2f740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad3b39f510 con 0x55ad3b383af0 2024-03-23T11:22:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.822+0000 7fe7ece2f740 10 client.0 _ll_register_callbacks cb 0x7fe7dc009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.822+0000 7fe7e2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b38bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.822+0000 7fe7e2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b38bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48424/0 (socket says 172.21.15.62:48424) 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.822+0000 7fe7e2a00640 1 -- 172.21.15.62:0/3325947706 learned_addr learned my addr 172.21.15.62:0/3325947706 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7e3400640 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b383af0 0x55ad3b4178b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7e3e00640 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b38c460 0x55ad3b38faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7e2a00640 1 -- 172.21.15.62:0/3325947706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b383af0 msgr2=0x55ad3b4178b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b383af0 0x55ad3b4178b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7e2a00640 1 -- 172.21.15.62:0/3325947706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b38c460 msgr2=0x55ad3b38faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b38c460 0x55ad3b38faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7e2a00640 1 -- 172.21.15.62:0/3325947706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad3b2c5c00 con 0x55ad3b383ed0 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7e2a00640 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b38bb70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc000fb0 tx=0x7fe7cc086d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cfd1b0a2601f8fd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7d0c00640 1 -- 172.21.15.62:0/3325947706 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc05b810 con 0x55ad3b383ed0 2024-03-23T11:22:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7d0c00640 1 -- 172.21.15.62:0/3325947706 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe7cc05f940 con 0x55ad3b383ed0 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.823+0000 7fe7ece2f740 1 -- 172.21.15.62:0/3325947706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad3afbb4e0 con 0x55ad3b383ed0 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7d0c00640 1 -- 172.21.15.62:0/3325947706 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7cc012e80 con 0x55ad3b383ed0 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7ece2f740 10 client.5883 fetch_fsmap learned FSMap version 25 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7ece2f740 1 -- 172.21.15.62:0/3325947706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55ad3b28d170 con 0x55ad3b383ed0 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7d0c00640 1 -- 172.21.15.62:0/3325947706 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ad3afbb4e0 con 0x55ad3b383ed0 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7d0c00640 1 -- 172.21.15.62:0/3325947706 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe7cc019070 con 0x55ad3b383ed0 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7ece2f740 10 client.5883 fetch_fsmap finished waiting for FSMap version 25 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7ece2f740 10 client.5883 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7ece2f740 20 client.5883 populate_metadata read hostname 'smithi062' 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55ad3b38ffe0 0x55ad3b35dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7ece2f740 4 client.5883 mds_command: new command op to 5652 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7ece2f740 1 -- 172.21.15.62:0/3325947706 --> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ad3afcc7e0 con 0x55ad3b38ffe0 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.824+0000 7fe7e3400640 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55ad3b38ffe0 0x55ad3b35dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7e3400640 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55ad3b38ffe0 0x55ad3b35dca0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93091ef4ab7c728a server_cookie=1eb123460441655e in_seq=0 out_seq=0 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7d0c00640 10 client.5883 ms_handle_connect on v2:172.21.15.62:6834/1159419089 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7d0c00640 1 -- 172.21.15.62:0/3325947706 <== mds.0 v2:172.21.15.62:6834/1159419089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ad3afcc7e0 con 0x55ad3b38ffe0 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7d0c00640 10 client.5883 handle_command_reply: tid=0 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7ece2f740 1 client.5883 shutdown 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7ece2f740 1 -- 172.21.15.62:0/3325947706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 msgr2=0x55ad3b38bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b38bb70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc000fb0 tx=0x7fe7cc086d00 comp rx=0 tx=0).stop 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7ece2f740 1 -- 172.21.15.62:0/3325947706 shutdown_connections 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.62:6834/1159419089,v1:172.21.15.62:6835/1159419089] conn(0x55ad3b38ffe0 0x55ad3b35dca0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad3b383ed0 0x55ad3b38bb70 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad3b38c460 0x55ad3b38faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7ece2f740 1 --2- 172.21.15.62:0/3325947706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad3b383af0 0x55ad3b4178b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7d0c00640 0 client.5883 ms_handle_reset on v2:172.21.15.62:6834/1159419089 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.825+0000 7fe7ece2f740 1 -- 172.21.15.62:0/3325947706 >> 172.21.15.62:0/3325947706 conn(0x55ad3b32bb40 msgr2=0x55ad3b381ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.826+0000 7fe7ece2f740 1 -- 172.21.15.62:0/3325947706 shutdown_connections 2024-03-23T11:22:08.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.826+0000 7fe7ece2f740 1 -- 172.21.15.62:0/3325947706 wait complete. 2024-03-23T11:22:08.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.826+0000 7fe7ece2f740 20 client.5883 trim_cache size 0 max 16384 2024-03-23T11:22:08.827 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:22:08.827 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-03-23T11:22:08.827 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-03-23T11:22:08.827 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:22:08.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.826+0000 7fe7e8a00640 1 -- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7c006a7a0 msgr2=0x7fe7c006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:08.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.826+0000 7fe7e8a00640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7c006a7a0 0x7fe7c006cc60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4036000 tx=0x7fe7d4030000 comp rx=0 tx=0).stop 2024-03-23T11:22:08.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.826+0000 7fe7e8a00640 1 -- 172.21.15.62:0/1327091884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e40cc9b0 msgr2=0x7fe7e41f2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:08.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.826+0000 7fe7e8a00640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e40cc9b0 0x7fe7e41f2750 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00c450 tx=0x7fe7cc00c910 comp rx=0 tx=0).stop 2024-03-23T11:22:08.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.827+0000 7fe7e8a00640 1 -- 172.21.15.62:0/1327091884 shutdown_connections 2024-03-23T11:22:08.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.827+0000 7fe7e8a00640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7c006a7a0 0x7fe7c006cc60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.827+0000 7fe7e8a00640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7e40cc9b0 0x7fe7e41f2750 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.827+0000 7fe7e8a00640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7e41431f0 0x7fe7e413fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.827+0000 7fe7e8a00640 1 --2- 172.21.15.62:0/1327091884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7e4070090 0x7fe7e41591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:08.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.827+0000 7fe7e8a00640 1 -- 172.21.15.62:0/1327091884 >> 172.21.15.62:0/1327091884 conn(0x7fe7e40b0090 msgr2=0x7fe7e413e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:08.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.827+0000 7fe7e8a00640 1 -- 172.21.15.62:0/1327091884 shutdown_connections 2024-03-23T11:22:08.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:08.827+0000 7fe7e8a00640 1 -- 172.21.15.62:0/1327091884 wait complete. 2024-03-23T11:22:08.841 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T11:22:08.942 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:22:09.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.173+0000 7f381d600640 1 Processor -- start 2024-03-23T11:22:09.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.173+0000 7f381d600640 1 -- start start 2024-03-23T11:22:09.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.175+0000 7f381d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f381806c6e0 0x7f381806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.175+0000 7f381d600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38180cbc60 con 0x7f381806c300 2024-03-23T11:22:09.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.175+0000 7f381d600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38180cbf40 con 0x7f38180cafa0 2024-03-23T11:22:09.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.175+0000 7f381d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38180cd410 con 0x7f381806c6e0 2024-03-23T11:22:09.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.175+0000 7f3816a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f381806c300 0x7f38180ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:41214/0 (socket says 172.21.15.62:41214) 2024-03-23T11:22:09.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.175+0000 7f3816a00640 1 -- 172.21.15.62:0/1185032176 learned_addr learned my addr 172.21.15.62:0/1185032176 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:09.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.175+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954968224 0 0) 0x7f38180cbc60 con 0x7f381806c300 2024-03-23T11:22:09.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.175+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37fc003520 con 0x7f381806c300 2024-03-23T11:22:09.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.175+0000 7f3817400640 1 --2- 172.21.15.62:0/1185032176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f381806c6e0 0x7f381806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2932190530 0 0) 0x7f37fc003520 con 0x7f381806c300 2024-03-23T11:22:09.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f381806c6e0 msgr2=0x7f381806cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:09.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 --2- 172.21.15.62:0/1185032176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f381806c6e0 0x7f381806cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 >> v1:172.21.15.120:6789/0 conn(0x7f38180cafa0 legacy=0x7f38180cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38180cc220 con 0x7f381806c300 2024-03-23T11:22:09.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f3808001670 con 0x7f381806c300 2024-03-23T11:22:09.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 >> v1:172.21.15.62:6789/0 conn(0x7f381806c300 legacy=0x7f38180ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 --2- 172.21.15.62:0/1185032176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 0x7f37fc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 --2- 172.21.15.62:0/1185032176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37fc0047c0 0x7f37fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 --2- 172.21.15.62:0/1185032176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f381806c6e0 0x7f37fc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38180cd410 con 0x7f37fc0039a0 2024-03-23T11:22:09.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc00bb10 con 0x7f37fc0047c0 2024-03-23T11:22:09.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc00bcd0 con 0x7f381806c6e0 2024-03-23T11:22:09.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3816a00640 1 --2- 172.21.15.62:0/1185032176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 0x7f37fc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.176+0000 7f3816000640 1 --2- 172.21.15.62:0/1185032176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37fc0047c0 0x7f37fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.177+0000 7f3816a00640 1 -- 172.21.15.62:0/1185032176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f381806c6e0 msgr2=0x7f37fc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:09.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.177+0000 7f3816a00640 1 --2- 172.21.15.62:0/1185032176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f381806c6e0 0x7f37fc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.177+0000 7f3816a00640 1 -- 172.21.15.62:0/1185032176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37fc0047c0 msgr2=0x7f37fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:09.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.177+0000 7f3816a00640 1 --2- 172.21.15.62:0/1185032176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37fc0047c0 0x7f37fc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.177+0000 7f3816a00640 1 -- 172.21.15.62:0/1185032176 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc003520 con 0x7f37fc0039a0 2024-03-23T11:22:09.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.178+0000 7f3816a00640 1 --2- 172.21.15.62:0/1185032176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 0x7f37fc003dc0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f380800c0e0 tx=0x7f37fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43a4e596ef5386e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:09.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.178+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808005ad0 con 0x7f37fc0039a0 2024-03-23T11:22:09.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.178+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3808005c30 con 0x7f37fc0039a0 2024-03-23T11:22:09.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.178+0000 7f3815600640 1 -- 172.21.15.62:0/1185032176 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380802c810 con 0x7f37fc0039a0 2024-03-23T11:22:09.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.178+0000 7f381d600640 1 -- 172.21.15.62:0/1185032176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 msgr2=0x7f37fc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.178+0000 7f381d600640 1 --2- 172.21.15.62:0/1185032176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 0x7f37fc003dc0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f380800c0e0 tx=0x7f37fc0057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.178+0000 7f3817400640 1 -- 172.21.15.62:0/1185032176 reap_dead start 2024-03-23T11:22:09.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.179+0000 7f381d600640 1 -- 172.21.15.62:0/1185032176 shutdown_connections 2024-03-23T11:22:09.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.179+0000 7f381d600640 1 -- 172.21.15.62:0/1185032176 >> 172.21.15.62:0/1185032176 conn(0x7f38180b02e0 msgr2=0x7f38180b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:09.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.179+0000 7f381d600640 1 -- 172.21.15.62:0/1185032176 shutdown_connections 2024-03-23T11:22:09.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.179+0000 7f381d600640 1 -- 172.21.15.62:0/1185032176 wait complete. 2024-03-23T11:22:09.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.181+0000 7f381d600640 1 Processor -- start 2024-03-23T11:22:09.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.181+0000 7f381d600640 1 -- start start 2024-03-23T11:22:09.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f381d600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 0x7f38180b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f381d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38180ada00 0x7f3818134ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f381d600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3818132680 0x7f3818132ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f381d600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381815b440 con 0x7f37fc0039a0 2024-03-23T11:22:09.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f381d600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38180b2fd0 con 0x7f3818132680 2024-03-23T11:22:09.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f381d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38180b32b0 con 0x7f38180ada00 2024-03-23T11:22:09.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f3816a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 0x7f38180b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f3816a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 0x7f38180b26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48448/0 (socket says 172.21.15.62:48448) 2024-03-23T11:22:09.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f3816a00640 1 -- 172.21.15.62:0/969140514 learned_addr learned my addr 172.21.15.62:0/969140514 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:09.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f3816a00640 1 -- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38180ada00 msgr2=0x7f3818134ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f3816a00640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38180ada00 0x7f3818134ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f3816a00640 1 -- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3818132680 msgr2=0x7f3818132ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f3816a00640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3818132680 0x7f3818132ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.182+0000 7f3816a00640 1 -- 172.21.15.62:0/969140514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381815ad80 con 0x7f37fc0039a0 2024-03-23T11:22:09.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.183+0000 7f3816a00640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 0x7f38180b26f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f380800c200 tx=0x7f3808001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:09.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.183+0000 7f380ea00640 1 -- 172.21.15.62:0/969140514 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380803c040 con 0x7f37fc0039a0 2024-03-23T11:22:09.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.183+0000 7f380ea00640 1 -- 172.21.15.62:0/969140514 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380802dcb0 con 0x7f37fc0039a0 2024-03-23T11:22:09.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.183+0000 7f380ea00640 1 -- 172.21.15.62:0/969140514 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808007270 con 0x7f37fc0039a0 2024-03-23T11:22:09.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.183+0000 7f381d600640 1 -- 172.21.15.62:0/969140514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381815b060 con 0x7f37fc0039a0 2024-03-23T11:22:09.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.183+0000 7f381d600640 1 -- 172.21.15.62:0/969140514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3818135420 con 0x7f37fc0039a0 2024-03-23T11:22:09.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.184+0000 7f381d600640 1 -- 172.21.15.62:0/969140514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381815b6b0 con 0x7f37fc0039a0 2024-03-23T11:22:09.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.185+0000 7f380ea00640 1 -- 172.21.15.62:0/969140514 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3808032070 con 0x7f37fc0039a0 2024-03-23T11:22:09.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.186+0000 7f380ea00640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f806a7a0 0x7f37f806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.186+0000 7f380ea00640 1 -- 172.21.15.62:0/969140514 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f38080aa910 con 0x7f37fc0039a0 2024-03-23T11:22:09.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.187+0000 7f3816000640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f806a7a0 0x7f37f806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.188+0000 7f3816000640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f806a7a0 0x7f37f806cc60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3804007aa0 tx=0x7f3804009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:09.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.191+0000 7f380ea00640 1 -- 172.21.15.62:0/969140514 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3808075cd0 con 0x7f37fc0039a0 2024-03-23T11:22:09.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.432+0000 7f381d600640 1 -- 172.21.15.62:0/969140514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38181355b0 con 0x7f37fc0039a0 2024-03-23T11:22:09.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.434+0000 7f380ea00640 1 -- 172.21.15.62:0/969140514 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f38080792f0 con 0x7f37fc0039a0 2024-03-23T11:22:09.432 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:09.432 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3787987198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3787987198},{"type":"v1","addr":"172.21.15.120:6835","nonce":3787987198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2749498408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2749498408},{"type":"v1","addr":"172.21.15.120:6837","nonce":2749498408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2325988870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2325988870},{"type":"v1","addr":"172.21.15.62:6833","nonce":2325988870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:21:25.687136+0000","modified":"2024-03-23T11:21:28.704192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6835/1159419089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1159419089},{"type":"v1","addr":"172.21.15.62:6835","nonce":1159419089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T11:22:09.432 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-03-23T11:22:09.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.436+0000 7f381d600640 1 -- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f806a7a0 msgr2=0x7f37f806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.436+0000 7f381d600640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f806a7a0 0x7f37f806cc60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3804007aa0 tx=0x7f3804009040 comp rx=0 tx=0).stop 2024-03-23T11:22:09.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.436+0000 7f381d600640 1 -- 172.21.15.62:0/969140514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 msgr2=0x7f38180b26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.436+0000 7f381d600640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 0x7f38180b26f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f380800c200 tx=0x7f3808001930 comp rx=0 tx=0).stop 2024-03-23T11:22:09.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.436+0000 7f381d600640 1 -- 172.21.15.62:0/969140514 shutdown_connections 2024-03-23T11:22:09.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.436+0000 7f381d600640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f806a7a0 0x7f37f806cc60 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.436+0000 7f381d600640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3818132680 0x7f3818132ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.436+0000 7f381d600640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38180ada00 0x7f3818134ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.436+0000 7f381d600640 1 --2- 172.21.15.62:0/969140514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37fc0039a0 0x7f38180b26f0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.436+0000 7f381d600640 1 -- 172.21.15.62:0/969140514 >> 172.21.15.62:0/969140514 conn(0x7f38180b02e0 msgr2=0x7f381813f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:09.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.437+0000 7f381d600640 1 -- 172.21.15.62:0/969140514 shutdown_connections 2024-03-23T11:22:09.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.437+0000 7f381d600640 1 -- 172.21.15.62:0/969140514 wait complete. 2024-03-23T11:22:09.453 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-23T11:22:09.707 INFO:teuthology.orchestra.run.smithi062.stdout:{"message":"","return_code":0} 2024-03-23T11:22:09.708 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-23T11:22:09.708 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-23T11:22:09.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.812+0000 7f7ae6d7ff40 1 Processor -- start 2024-03-23T11:22:09.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.812+0000 7f7ae6d7ff40 1 -- start start 2024-03-23T11:22:09.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.813+0000 7f7ae6d7ff40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ea4641910 0x556ea46f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.813+0000 7f7ae6d7ff40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556ea46e5c20 0x556ea46e4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.813+0000 7f7ae6d7ff40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e6540 0x556ea46c7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.813+0000 7f7ae6d7ff40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ea45564a0 con 0x556ea46e6540 2024-03-23T11:22:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.813+0000 7f7ae6d7ff40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ea454d110 con 0x556ea4641910 2024-03-23T11:22:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.813+0000 7f7ae6d7ff40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x556ea46408c0 con 0x556ea46e5c20 2024-03-23T11:22:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.813+0000 7f7ae4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e6540 0x556ea46c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.813+0000 7f7ae4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e6540 0x556ea46c7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48464/0 (socket says 172.21.15.62:48464) 2024-03-23T11:22:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.813+0000 7f7ae4a00640 1 -- 172.21.15.62:0/3667629116 learned_addr learned my addr 172.21.15.62:0/3667629116 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.814+0000 7f7ae3600640 1 --2- 172.21.15.62:0/3667629116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556ea46e5c20 0x556ea46e4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.814+0000 7f7ae4a00640 1 -- 172.21.15.62:0/3667629116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556ea46e5c20 msgr2=0x556ea46e4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.814+0000 7f7ae4000640 1 --2- 172.21.15.62:0/3667629116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ea4641910 0x556ea46f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.814+0000 7f7ae4a00640 1 --2- 172.21.15.62:0/3667629116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556ea46e5c20 0x556ea46e4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.814+0000 7f7ae4a00640 1 -- 172.21.15.62:0/3667629116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ea4641910 msgr2=0x556ea46f02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:09.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.814+0000 7f7ae4a00640 1 --2- 172.21.15.62:0/3667629116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ea4641910 0x556ea46f02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.814+0000 7f7ae4a00640 1 -- 172.21.15.62:0/3667629116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad400a1c0 con 0x556ea46e6540 2024-03-23T11:22:09.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.814+0000 7f7ae4a00640 1 --2- 172.21.15.62:0/3667629116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e6540 0x556ea46c7820 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00a5c0 tx=0x7f7adc00a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab76aaab49d547 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.814+0000 7f7ae2200640 1 -- 172.21.15.62:0/3667629116 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc00f070 con 0x556ea46e6540 2024-03-23T11:22:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.814+0000 7f7ae2200640 1 -- 172.21.15.62:0/3667629116 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7adc007600 con 0x556ea46e6540 2024-03-23T11:22:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.814+0000 7f7ae2200640 1 -- 172.21.15.62:0/3667629116 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc007990 con 0x556ea46e6540 2024-03-23T11:22:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.815+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/3667629116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e6540 msgr2=0x556ea46c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.815+0000 7f7ae6d7ff40 1 --2- 172.21.15.62:0/3667629116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e6540 0x556ea46c7820 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00a5c0 tx=0x7f7adc00a6b0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.815+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/3667629116 shutdown_connections 2024-03-23T11:22:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.815+0000 7f7ae6d7ff40 1 --2- 172.21.15.62:0/3667629116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e6540 0x556ea46c7820 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.815+0000 7f7ae6d7ff40 1 --2- 172.21.15.62:0/3667629116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556ea46e5c20 0x556ea46e4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.815+0000 7f7ae6d7ff40 1 --2- 172.21.15.62:0/3667629116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ea4641910 0x556ea46f02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.815+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/3667629116 >> 172.21.15.62:0/3667629116 conn(0x556ea465bff0 msgr2=0x556ea465aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.815+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/3667629116 shutdown_connections 2024-03-23T11:22:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.815+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/3667629116 wait complete. 2024-03-23T11:22:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.816+0000 7f7ae6d7ff40 1 Processor -- start 2024-03-23T11:22:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.816+0000 7f7ae6d7ff40 1 -- start start 2024-03-23T11:22:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.817+0000 7f7ae6d7ff40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556ea4641910 0x556ea46c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.817+0000 7f7ae6d7ff40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e5c20 0x556ea463c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.817+0000 7f7ae6d7ff40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ea46e6540 0x556ea463ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.817+0000 7f7ae6d7ff40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ea46e3ef0 con 0x556ea46e5c20 2024-03-23T11:22:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.817+0000 7f7ae6d7ff40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ea46e41d0 con 0x556ea46e6540 2024-03-23T11:22:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.817+0000 7f7ae6d7ff40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x556ea46e44b0 con 0x556ea4641910 2024-03-23T11:22:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.817+0000 7f7ae3600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e5c20 0x556ea463c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.817+0000 7f7ae3600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e5c20 0x556ea463c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48478/0 (socket says 172.21.15.62:48478) 2024-03-23T11:22:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.817+0000 7f7ae3600640 1 -- 172.21.15.62:0/1831132212 learned_addr learned my addr 172.21.15.62:0/1831132212 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.817+0000 7f7ae4000640 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556ea4641910 0x556ea46c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae4a00640 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ea46e6540 0x556ea463ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae3600640 1 -- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556ea4641910 msgr2=0x556ea46c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae3600640 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556ea4641910 0x556ea46c6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae3600640 1 -- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ea46e6540 msgr2=0x556ea463ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae3600640 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ea46e6540 0x556ea463ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae3600640 1 -- 172.21.15.62:0/1831132212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc005e20 con 0x556ea46e5c20 2024-03-23T11:22:09.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae4a00640 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ea46e6540 0x556ea463ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:09.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae3600640 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e5c20 0x556ea463c970 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000be30 tx=0x7f7ad000bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:09.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae0e00640 1 -- 172.21.15.62:0/1831132212 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0010070 con 0x556ea46e5c20 2024-03-23T11:22:09.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae0e00640 1 -- 172.21.15.62:0/1831132212 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad0007b50 con 0x556ea46e5c20 2024-03-23T11:22:09.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae0e00640 1 -- 172.21.15.62:0/1831132212 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0014410 con 0x556ea46e5c20 2024-03-23T11:22:09.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.818+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/1831132212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556ea457f4e0 con 0x556ea46e5c20 2024-03-23T11:22:09.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.819+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/1831132212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556ea4565340 con 0x556ea46e5c20 2024-03-23T11:22:09.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.820+0000 7f7ae0e00640 1 -- 172.21.15.62:0/1831132212 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7ad0014570 con 0x556ea46e5c20 2024-03-23T11:22:09.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.821+0000 7f7ae0e00640 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7ac40691b0 0x7f7ac406b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.821+0000 7f7ae0e00640 1 -- 172.21.15.62:0/1831132212 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7ad0086b40 con 0x556ea46e5c20 2024-03-23T11:22:09.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.822+0000 7f7ae6d7ff40 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x556ea46de8b0 0x556ea46decc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:09.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.822+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/1831132212 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:1 7.7 7:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50) v8 -- 0x556ea46c6f80 con 0x556ea46de8b0 2024-03-23T11:22:09.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.822+0000 7f7ae4000640 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7ac40691b0 0x7f7ac406b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.822+0000 7f7ae4a00640 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x556ea46de8b0 0x556ea46decc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:09.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.823+0000 7f7ae4000640 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7ac40691b0 0x7f7ac406b670 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4052000 tx=0x7f7ad404d000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:09.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.823+0000 7f7ae4a00640 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x556ea46de8b0 0x556ea46decc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:09.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.824+0000 7f7ae4a00640 1 -- 172.21.15.62:0/1831132212 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7f7adc007a10 con 0x556ea46de8b0 2024-03-23T11:22:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.824+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x556ea46de8b0 msgr2=0x556ea46decc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.824+0000 7f7ae6d7ff40 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x556ea46de8b0 0x556ea46decc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.824+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7ac40691b0 msgr2=0x7f7ac406b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.824+0000 7f7ae6d7ff40 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7ac40691b0 0x7f7ac406b670 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4052000 tx=0x7f7ad404d000 comp rx=0 tx=0).stop 2024-03-23T11:22:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.824+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/1831132212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e5c20 msgr2=0x556ea463c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.824+0000 7f7ae6d7ff40 1 --2- 172.21.15.62:0/1831132212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556ea46e5c20 0x556ea463c970 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000be30 tx=0x7f7ad000bf20 comp rx=0 tx=0).stop 2024-03-23T11:22:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.824+0000 7f7ae4a00640 1 -- 172.21.15.62:0/1831132212 reap_dead start 2024-03-23T11:22:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.824+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/1831132212 shutdown_connections 2024-03-23T11:22:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.824+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/1831132212 >> 172.21.15.62:0/1831132212 conn(0x556ea465bff0 msgr2=0x556ea46c7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.824+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/1831132212 shutdown_connections 2024-03-23T11:22:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:09.825+0000 7f7ae6d7ff40 1 -- 172.21.15.62:0/1831132212 wait complete. 2024-03-23T11:22:09.827 INFO:teuthology.orchestra.run.smithi062.stdout:Yalphaparent_a 2024-03-23T11:22:09.827 DEBUG:teuthology.orchestra.run.smithi062:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout: "ino": 1099511627777, 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout: "ancestors": [ 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout: "dirino": 1099511627776, 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout: "dname": "alpha", 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 4 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout: "dirino": 1, 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout: "dname": "parent_a", 2024-03-23T11:22:09.936 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 7 2024-03-23T11:22:09.937 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:09.937 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-03-23T11:22:09.937 INFO:teuthology.orchestra.run.smithi062.stdout: "pool": 7, 2024-03-23T11:22:09.937 INFO:teuthology.orchestra.run.smithi062.stdout: "old_pools": [] 2024-03-23T11:22:09.937 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:22:09.937 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:09.943 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:22:09.943 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:22:10.059 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:22:10.176 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:22:10.176 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:22:10.176 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:22:10.237 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:22:10.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.462+0000 7f6974000640 1 Processor -- start 2024-03-23T11:22:10.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.463+0000 7f6974000640 1 -- start start 2024-03-23T11:22:10.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.464+0000 7f6974000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c0b84f0 0x7f696c0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.464+0000 7f6974000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c0b5ad0 con 0x7f696c0cdc40 2024-03-23T11:22:10.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.464+0000 7f6974000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c06b820 con 0x7f696c15cb90 2024-03-23T11:22:10.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.464+0000 7f6974000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c06c0c0 con 0x7f696c0b84f0 2024-03-23T11:22:10.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.465+0000 7f6972000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f696c0cdc40 0x7f696c0d1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:41216/0 (socket says 172.21.15.62:41216) 2024-03-23T11:22:10.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.465+0000 7f6972000640 1 -- 172.21.15.62:0/2391973817 learned_addr learned my addr 172.21.15.62:0/2391973817 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:10.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.465+0000 7f6970c00640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c0b84f0 0x7f696c0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.465+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024441109 0 0) 0x7f696c0b5ad0 con 0x7f696c0cdc40 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.465+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6954003520 con 0x7f696c0cdc40 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3137056402 0 0) 0x7f6954003520 con 0x7f696c0cdc40 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c0b84f0 msgr2=0x7f696c0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c0b84f0 0x7f696c0cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 >> v1:172.21.15.120:6789/0 conn(0x7f696c15cb90 legacy=0x7f696c06b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c06bb00 con 0x7f696c0cdc40 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f69640019f0 con 0x7f696c0cdc40 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 >> v1:172.21.15.62:6789/0 conn(0x7f696c0cdc40 legacy=0x7f696c0d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69540039a0 0x7f6954003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69540047c0 0x7f6954004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c0b84f0 0x7f6954005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c06c0c0 con 0x7f69540047c0 2024-03-23T11:22:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695400ba60 con 0x7f69540039a0 2024-03-23T11:22:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695400bc20 con 0x7f696c0b84f0 2024-03-23T11:22:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f6970c00640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69540047c0 0x7f6954004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.466+0000 7f6971600640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69540039a0 0x7f6954003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.467+0000 7f6972000640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c0b84f0 0x7f6954005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.467+0000 7f6970c00640 1 -- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c0b84f0 msgr2=0x7f6954005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.467+0000 7f6970c00640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c0b84f0 0x7f6954005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.467+0000 7f6970c00640 1 -- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69540039a0 msgr2=0x7f6954003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.467+0000 7f6970c00640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69540039a0 0x7f6954003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.467+0000 7f6970c00640 1 -- 172.21.15.62:0/2391973817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6954003520 con 0x7f69540047c0 2024-03-23T11:22:10.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.467+0000 7f6971600640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69540039a0 0x7f6954003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:22:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.468+0000 7f6970c00640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69540047c0 0x7f6954004c40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6958009a90 tx=0x7f69540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1d47c8b034023d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.468+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695802c680 con 0x7f69540047c0 2024-03-23T11:22:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.468+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695802c7e0 con 0x7f69540047c0 2024-03-23T11:22:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.468+0000 7f696be00640 1 -- 172.21.15.62:0/2391973817 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695802cb70 con 0x7f69540047c0 2024-03-23T11:22:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.469+0000 7f6974000640 1 -- 172.21.15.62:0/2391973817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69540047c0 msgr2=0x7f6954004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.469+0000 7f6974000640 1 --2- 172.21.15.62:0/2391973817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69540047c0 0x7f6954004c40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6958009a90 tx=0x7f69540057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.469+0000 7f6972000640 1 -- 172.21.15.62:0/2391973817 reap_dead start 2024-03-23T11:22:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.469+0000 7f6974000640 1 -- 172.21.15.62:0/2391973817 shutdown_connections 2024-03-23T11:22:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.469+0000 7f6974000640 1 -- 172.21.15.62:0/2391973817 >> 172.21.15.62:0/2391973817 conn(0x7f696c0cf7d0 msgr2=0x7f696c0cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.469+0000 7f6974000640 1 -- 172.21.15.62:0/2391973817 shutdown_connections 2024-03-23T11:22:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.469+0000 7f6974000640 1 -- 172.21.15.62:0/2391973817 wait complete. 2024-03-23T11:22:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.471+0000 7f6974000640 1 Processor -- start 2024-03-23T11:22:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.471+0000 7f6974000640 1 -- start start 2024-03-23T11:22:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.472+0000 7f6974000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69540047c0 0x7f696c1455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.472+0000 7f6974000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c1e4870 0x7f696c0c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.472+0000 7f6974000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c146db0 0x7f696c147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.472+0000 7f6974000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0d1010 con 0x7f696c1e4870 2024-03-23T11:22:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.472+0000 7f6974000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c158a50 con 0x7f69540047c0 2024-03-23T11:22:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.472+0000 7f6974000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c145ae0 con 0x7f696c146db0 2024-03-23T11:22:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.472+0000 7f6970c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c1e4870 0x7f696c0c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.472+0000 7f6970c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c1e4870 0x7f696c0c84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48494/0 (socket says 172.21.15.62:48494) 2024-03-23T11:22:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.472+0000 7f6970c00640 1 -- 172.21.15.62:0/3503675134 learned_addr learned my addr 172.21.15.62:0/3503675134 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.473+0000 7f6971600640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69540047c0 0x7f696c1455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.473+0000 7f6972000640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c146db0 0x7f696c147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.473+0000 7f6970c00640 1 -- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c146db0 msgr2=0x7f696c147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.473+0000 7f6970c00640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c146db0 0x7f696c147630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.473+0000 7f6970c00640 1 -- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69540047c0 msgr2=0x7f696c1455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.473+0000 7f6970c00640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69540047c0 0x7f696c1455a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.473+0000 7f6970c00640 1 -- 172.21.15.62:0/3503675134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c146920 con 0x7f696c1e4870 2024-03-23T11:22:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.473+0000 7f6971600640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69540047c0 0x7f696c1455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.473+0000 7f6972000640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c146db0 0x7f696c147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:10.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.473+0000 7f6970c00640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c1e4870 0x7f696c0c84b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6958009a90 tx=0x7f6958004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:10.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.474+0000 7f6969600640 1 -- 172.21.15.62:0/3503675134 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695802dea0 con 0x7f696c1e4870 2024-03-23T11:22:10.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.474+0000 7f6969600640 1 -- 172.21.15.62:0/3503675134 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6958003810 con 0x7f696c1e4870 2024-03-23T11:22:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.474+0000 7f6969600640 1 -- 172.21.15.62:0/3503675134 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6958003ba0 con 0x7f696c1e4870 2024-03-23T11:22:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.474+0000 7f6974000640 1 -- 172.21.15.62:0/3503675134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c0c89f0 con 0x7f696c1e4870 2024-03-23T11:22:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.474+0000 7f6974000640 1 -- 172.21.15.62:0/3503675134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c145ff0 con 0x7f696c1e4870 2024-03-23T11:22:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.475+0000 7f6974000640 1 -- 172.21.15.62:0/3503675134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c146ab0 con 0x7f696c1e4870 2024-03-23T11:22:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.476+0000 7f6969600640 1 -- 172.21.15.62:0/3503675134 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6958003d00 con 0x7f696c1e4870 2024-03-23T11:22:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.477+0000 7f6969600640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f694c06a7d0 0x7f694c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.477+0000 7f6969600640 1 -- 172.21.15.62:0/3503675134 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f69580ab1b0 con 0x7f696c1e4870 2024-03-23T11:22:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.478+0000 7f6971600640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f694c06a7d0 0x7f694c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.479+0000 7f6971600640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f694c06a7d0 0x7f694c06cc90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f69600016b0 tx=0x7f6960009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:10.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.482+0000 7f6969600640 1 -- 172.21.15.62:0/3503675134 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6958076460 con 0x7f696c1e4870 2024-03-23T11:22:10.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.679+0000 7f6974000640 1 -- 172.21.15.62:0/3503675134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f696c066550 con 0x7f696c1e4870 2024-03-23T11:22:10.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.682+0000 7f6969600640 1 -- 172.21.15.62:0/3503675134 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16197 (secure 0 0 0) 0x7f69580044e0 con 0x7f696c1e4870 2024-03-23T11:22:10.680 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:10.681 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":50,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:21:43.375126+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:21:23.756222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-23T11:21:24.715577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.2a","mappings":[{"from":0,"to":5}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:22:10.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.685+0000 7f6974000640 1 -- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f694c06a7d0 msgr2=0x7f694c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.685+0000 7f6974000640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f694c06a7d0 0x7f694c06cc90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f69600016b0 tx=0x7f6960009040 comp rx=0 tx=0).stop 2024-03-23T11:22:10.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.685+0000 7f6974000640 1 -- 172.21.15.62:0/3503675134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c1e4870 msgr2=0x7f696c0c84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.685+0000 7f6974000640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c1e4870 0x7f696c0c84b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6958009a90 tx=0x7f6958004730 comp rx=0 tx=0).stop 2024-03-23T11:22:10.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.685+0000 7f6974000640 1 -- 172.21.15.62:0/3503675134 shutdown_connections 2024-03-23T11:22:10.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.685+0000 7f6974000640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f694c06a7d0 0x7f694c06cc90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.685+0000 7f6974000640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c1e4870 0x7f696c0c84b0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.685+0000 7f6974000640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f696c146db0 0x7f696c147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.685+0000 7f6974000640 1 --2- 172.21.15.62:0/3503675134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69540047c0 0x7f696c1455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.685+0000 7f6974000640 1 -- 172.21.15.62:0/3503675134 >> 172.21.15.62:0/3503675134 conn(0x7f696c0cf7d0 msgr2=0x7f696c06b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:10.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.686+0000 7f6974000640 1 -- 172.21.15.62:0/3503675134 shutdown_connections 2024-03-23T11:22:10.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.686+0000 7f6974000640 1 -- 172.21.15.62:0/3503675134 wait complete. 2024-03-23T11:22:10.703 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:21:23.756222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:21:24.715577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:22:10.703 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:22:10.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.929+0000 7f7fd1800640 1 Processor -- start 2024-03-23T11:22:10.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.930+0000 7f7fd1800640 1 -- start start 2024-03-23T11:22:10.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.931+0000 7f7fd1800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fcc0989d0 0x7f7fcc098e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.931+0000 7f7fd1800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fcc0cf690 con 0x7f7fcc160240 2024-03-23T11:22:10.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.931+0000 7f7fd1800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fcc0cb730 con 0x7f7fcc0995b0 2024-03-23T11:22:10.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.931+0000 7f7fd1800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0cf490 con 0x7f7fcc0989d0 2024-03-23T11:22:10.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.931+0000 7f7fcaa00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7fcc160240 0x7f7fcc06ad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:41228/0 (socket says 172.21.15.62:41228) 2024-03-23T11:22:10.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.931+0000 7f7fcaa00640 1 -- 172.21.15.62:0/4143418570 learned_addr learned my addr 172.21.15.62:0/4143418570 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:10.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.932+0000 7f7fcb400640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fcc0989d0 0x7f7fcc098e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.932+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118909400 0 0) 0x7f7fcc0cf690 con 0x7f7fcc160240 2024-03-23T11:22:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.932+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb0003520 con 0x7f7fcc160240 2024-03-23T11:22:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.932+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 956072413 0 0) 0x7f7fb0003520 con 0x7f7fcc160240 2024-03-23T11:22:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.932+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fcc0989d0 msgr2=0x7f7fcc098e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.932+0000 7f7fc9600640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fcc0989d0 0x7f7fcc098e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.932+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 >> v1:172.21.15.120:6789/0 conn(0x7f7fcc0995b0 legacy=0x7f7fcc0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.932+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc0cba10 con 0x7f7fcc160240 2024-03-23T11:22:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.932+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7fbc001670 con 0x7f7fcc160240 2024-03-23T11:22:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.932+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 >> v1:172.21.15.62:6789/0 conn(0x7f7fcc160240 legacy=0x7f7fcc06ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fc9600640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fcc0989d0 0x7f7fb00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fc9600640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fb0003ee0 0x7f7fb00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fc9600640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb0004d00 0x7f7fb0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0cf490 con 0x7f7fb0003ee0 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb000ba90 con 0x7f7fb0004d00 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb000bc50 con 0x7f7fcc0989d0 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fca000640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fb0003ee0 0x7f7fb00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fcb400640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb0004d00 0x7f7fb0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fcaa00640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fcc0989d0 0x7f7fb00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fca000640 1 -- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fcc0989d0 msgr2=0x7f7fb00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fca000640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fcc0989d0 0x7f7fb00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fca000640 1 -- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb0004d00 msgr2=0x7f7fb0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fca000640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb0004d00 0x7f7fb0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fca000640 1 -- 172.21.15.62:0/4143418570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb0003520 con 0x7f7fb0003ee0 2024-03-23T11:22:10.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.933+0000 7f7fcb400640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb0004d00 0x7f7fb0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:10.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.934+0000 7f7fcaa00640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fcc0989d0 0x7f7fb00039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.935+0000 7f7fca000640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fb0003ee0 0x7f7fb00047c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4001750 tx=0x7f7fb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c456020ec52eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.935+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb40052d0 con 0x7f7fb0003ee0 2024-03-23T11:22:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.935+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb402e050 con 0x7f7fb0003ee0 2024-03-23T11:22:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.935+0000 7f7fc9600640 1 -- 172.21.15.62:0/4143418570 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4003830 con 0x7f7fb0003ee0 2024-03-23T11:22:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.935+0000 7f7fd1800640 1 -- 172.21.15.62:0/4143418570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fb0003ee0 msgr2=0x7f7fb00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.935+0000 7f7fd1800640 1 --2- 172.21.15.62:0/4143418570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fb0003ee0 0x7f7fb00047c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4001750 tx=0x7f7fb00057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.935+0000 7f7fcb400640 1 -- 172.21.15.62:0/4143418570 reap_dead start 2024-03-23T11:22:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.936+0000 7f7fd1800640 1 -- 172.21.15.62:0/4143418570 shutdown_connections 2024-03-23T11:22:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.936+0000 7f7fd1800640 1 -- 172.21.15.62:0/4143418570 >> 172.21.15.62:0/4143418570 conn(0x7f7fcc0b22c0 msgr2=0x7f7fcc0b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.936+0000 7f7fd1800640 1 -- 172.21.15.62:0/4143418570 shutdown_connections 2024-03-23T11:22:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.936+0000 7f7fd1800640 1 -- 172.21.15.62:0/4143418570 wait complete. 2024-03-23T11:22:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.937+0000 7f7fd1800640 1 Processor -- start 2024-03-23T11:22:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.938+0000 7f7fd1800640 1 -- start start 2024-03-23T11:22:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.938+0000 7f7fd1800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fb0003ee0 0x7f7fcc157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.938+0000 7f7fd1800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc1f6570 0x7f7fcc1607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.938+0000 7f7fd1800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fcc160ce0 0x7f7fcc0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.938+0000 7f7fd1800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc161160 con 0x7f7fcc1f6570 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.938+0000 7f7fd1800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc1f69d0 con 0x7f7fcc160ce0 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.938+0000 7f7fd1800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc157c50 con 0x7f7fb0003ee0 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.939+0000 7f7fca000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc1f6570 0x7f7fcc1607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.939+0000 7f7fca000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc1f6570 0x7f7fcc1607a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48520/0 (socket says 172.21.15.62:48520) 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.939+0000 7f7fca000640 1 -- 172.21.15.62:0/2662136705 learned_addr learned my addr 172.21.15.62:0/2662136705 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.939+0000 7f7fcb400640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fcc160ce0 0x7f7fcc0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.939+0000 7f7fcaa00640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fb0003ee0 0x7f7fcc157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.939+0000 7f7fca000640 1 -- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fb0003ee0 msgr2=0x7f7fcc157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.939+0000 7f7fca000640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fb0003ee0 0x7f7fcc157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.939+0000 7f7fca000640 1 -- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fcc160ce0 msgr2=0x7f7fcc0c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.939+0000 7f7fca000640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fcc160ce0 0x7f7fcc0c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:10.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.939+0000 7f7fca000640 1 -- 172.21.15.62:0/2662136705 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc157f30 con 0x7f7fcc1f6570 2024-03-23T11:22:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.939+0000 7f7fcb400640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fcc160ce0 0x7f7fcc0c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:22:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.940+0000 7f7fca000640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc1f6570 0x7f7fcc1607a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4008fd0 tx=0x7f7fb4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.940+0000 7f7fc2a00640 1 -- 172.21.15.62:0/2662136705 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb40076c0 con 0x7f7fcc1f6570 2024-03-23T11:22:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.940+0000 7f7fc2a00640 1 -- 172.21.15.62:0/2662136705 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb4007a30 con 0x7f7fcc1f6570 2024-03-23T11:22:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.940+0000 7f7fc2a00640 1 -- 172.21.15.62:0/2662136705 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4006be0 con 0x7f7fcc1f6570 2024-03-23T11:22:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.940+0000 7f7fd1800640 1 -- 172.21.15.62:0/2662136705 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fcc158210 con 0x7f7fcc1f6570 2024-03-23T11:22:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.940+0000 7f7fd1800640 1 -- 172.21.15.62:0/2662136705 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc1587d0 con 0x7f7fcc1f6570 2024-03-23T11:22:10.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.941+0000 7f7fd1800640 1 -- 172.21.15.62:0/2662136705 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc0738e0 con 0x7f7fcc1f6570 2024-03-23T11:22:10.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.942+0000 7f7fc2a00640 1 -- 172.21.15.62:0/2662136705 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7fb4006d40 con 0x7f7fcc1f6570 2024-03-23T11:22:10.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.943+0000 7f7fc2a00640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fa806a7a0 0x7f7fa806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:10.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.943+0000 7f7fc2a00640 1 -- 172.21.15.62:0/2662136705 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7fb40aae20 con 0x7f7fcc1f6570 2024-03-23T11:22:10.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.944+0000 7f7fcaa00640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fa806a7a0 0x7f7fa806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:10.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.945+0000 7f7fcaa00640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fa806a7a0 0x7f7fa806cc60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc000c00 tx=0x7f7fbc001c40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:10.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:10.948+0000 7f7fc2a00640 1 -- 172.21.15.62:0/2662136705 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7fb40762b0 con 0x7f7fcc1f6570 2024-03-23T11:22:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.143+0000 7f7fd1800640 1 -- 172.21.15.62:0/2662136705 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7fcc1610c0 con 0x7f7fcc1f6570 2024-03-23T11:22:11.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.144+0000 7f7fc2a00640 1 -- 172.21.15.62:0/2662136705 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7fb40798d0 con 0x7f7fcc1f6570 2024-03-23T11:22:11.142 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:22:11.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 -- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fa806a7a0 msgr2=0x7f7fa806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fa806a7a0 0x7f7fa806cc60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc000c00 tx=0x7f7fbc001c40 comp rx=0 tx=0).stop 2024-03-23T11:22:11.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 -- 172.21.15.62:0/2662136705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc1f6570 msgr2=0x7f7fcc1607a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc1f6570 0x7f7fcc1607a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4008fd0 tx=0x7f7fb4001120 comp rx=0 tx=0).stop 2024-03-23T11:22:11.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 -- 172.21.15.62:0/2662136705 shutdown_connections 2024-03-23T11:22:11.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7fa806a7a0 0x7f7fa806cc60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fcc1f6570 0x7f7fcc1607a0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fcc160ce0 0x7f7fcc0c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 --2- 172.21.15.62:0/2662136705 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fb0003ee0 0x7f7fcc157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 -- 172.21.15.62:0/2662136705 >> 172.21.15.62:0/2662136705 conn(0x7f7fcc0b22c0 msgr2=0x7f7fcc15a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 -- 172.21.15.62:0/2662136705 shutdown_connections 2024-03-23T11:22:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.147+0000 7f7fd1800640 1 -- 172.21.15.62:0/2662136705 wait complete. 2024-03-23T11:22:11.166 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:22:11.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.395+0000 7f0985800640 1 Processor -- start 2024-03-23T11:22:11.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.395+0000 7f0985800640 1 -- start start 2024-03-23T11:22:11.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.397+0000 7f0985800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0980070090 0x7f0980070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.397+0000 7f0985800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09800b2820 0x7f09800b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.397+0000 7f0985800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09800b31b0 con 0x7f0980143dd0 2024-03-23T11:22:11.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.397+0000 7f0985800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0980070a60 con 0x7f0980070090 2024-03-23T11:22:11.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.397+0000 7f0985800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09800cd140 con 0x7f09800b2820 2024-03-23T11:22:11.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.397+0000 7f097fe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0980143dd0 0x7f098015abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:41240/0 (socket says 172.21.15.62:41240) 2024-03-23T11:22:11.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.397+0000 7f097fe00640 1 -- 172.21.15.62:0/1773860373 learned_addr learned my addr 172.21.15.62:0/1773860373 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:11.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.397+0000 7f097f400640 1 --2- 172.21.15.62:0/1773860373 >> v2:172.21.15.62:3300/0 conn(0x7f0980070090 0x7f0980070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.397+0000 7f0984e00640 1 --2- 172.21.15.62:0/1773860373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09800b2820 0x7f09800b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.398+0000 7f097ea00640 1 -- 172.21.15.62:0/1773860373 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033534852 0 0) 0x7f09800b31b0 con 0x7f0980143dd0 2024-03-23T11:22:11.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.398+0000 7f097ea00640 1 -- 172.21.15.62:0/1773860373 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0964003520 con 0x7f0980143dd0 2024-03-23T11:22:11.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.398+0000 7f097f400640 1 -- 172.21.15.62:0/1773860373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09800b2820 msgr2=0x7f09800b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:11.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.398+0000 7f097f400640 1 --2- 172.21.15.62:0/1773860373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09800b2820 0x7f09800b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.398+0000 7f097f400640 1 -- 172.21.15.62:0/1773860373 >> v1:172.21.15.62:6789/0 conn(0x7f0980143dd0 legacy=0x7f098015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.398+0000 7f097f400640 1 -- 172.21.15.62:0/1773860373 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09800cd3e0 con 0x7f0980070090 2024-03-23T11:22:11.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.399+0000 7f097f400640 1 --2- 172.21.15.62:0/1773860373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0980070090 0x7f0980070520 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f096c0097a0 tx=0x7f096c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b61c50c92190ae9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:11.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.399+0000 7f097ea00640 1 -- 172.21.15.62:0/1773860373 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c02d040 con 0x7f0980070090 2024-03-23T11:22:11.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.399+0000 7f097ea00640 1 -- 172.21.15.62:0/1773860373 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096c031810 con 0x7f0980070090 2024-03-23T11:22:11.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.399+0000 7f097ea00640 1 -- 172.21.15.62:0/1773860373 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c031b80 con 0x7f0980070090 2024-03-23T11:22:11.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.400+0000 7f0985800640 1 -- 172.21.15.62:0/1773860373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0980070090 msgr2=0x7f0980070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.400+0000 7f0985800640 1 --2- 172.21.15.62:0/1773860373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0980070090 0x7f0980070520 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f096c0097a0 tx=0x7f096c02b8b0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.400+0000 7f0985800640 1 -- 172.21.15.62:0/1773860373 shutdown_connections 2024-03-23T11:22:11.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.400+0000 7f0985800640 1 --2- 172.21.15.62:0/1773860373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09800b2820 0x7f09800b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.400+0000 7f0985800640 1 --2- 172.21.15.62:0/1773860373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0980070090 0x7f0980070520 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.400+0000 7f0985800640 1 -- 172.21.15.62:0/1773860373 >> 172.21.15.62:0/1773860373 conn(0x7f0980066050 msgr2=0x7f0980069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:11.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.400+0000 7f0985800640 1 -- 172.21.15.62:0/1773860373 shutdown_connections 2024-03-23T11:22:11.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.400+0000 7f0985800640 1 -- 172.21.15.62:0/1773860373 wait complete. 2024-03-23T11:22:11.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.402+0000 7f0985800640 1 Processor -- start 2024-03-23T11:22:11.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.402+0000 7f0985800640 1 -- start start 2024-03-23T11:22:11.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.403+0000 7f0985800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0980070090 0x7f0980159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.403+0000 7f0985800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09800b2820 0x7f098015c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.403+0000 7f0985800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0980143dd0 0x7f098015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.403+0000 7f0985800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098013c090 con 0x7f0980143dd0 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.403+0000 7f0985800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098013b1a0 con 0x7f09800b2820 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.403+0000 7f0985800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098013b480 con 0x7f0980070090 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.403+0000 7f0984e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0980143dd0 0x7f098015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.403+0000 7f0984e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0980143dd0 0x7f098015ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48534/0 (socket says 172.21.15.62:48534) 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.403+0000 7f0984e00640 1 -- 172.21.15.62:0/195834005 learned_addr learned my addr 172.21.15.62:0/195834005 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f097fe00640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0980070090 0x7f0980159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f097f400640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09800b2820 0x7f098015c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f0984e00640 1 -- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0980070090 msgr2=0x7f0980159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f0984e00640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0980070090 0x7f0980159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f0984e00640 1 -- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09800b2820 msgr2=0x7f098015c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f0984e00640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09800b2820 0x7f098015c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f0984e00640 1 -- 172.21.15.62:0/195834005 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c009400 con 0x7f0980143dd0 2024-03-23T11:22:11.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f0984e00640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0980143dd0 0x7f098015ca10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0978007e40 tx=0x7f097800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:11.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f0977e00640 1 -- 172.21.15.62:0/195834005 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978014620 con 0x7f0980143dd0 2024-03-23T11:22:11.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f0977e00640 1 -- 172.21.15.62:0/195834005 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097800ee90 con 0x7f0980143dd0 2024-03-23T11:22:11.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f0977e00640 1 -- 172.21.15.62:0/195834005 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09780149e0 con 0x7f0980143dd0 2024-03-23T11:22:11.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.404+0000 7f0985800640 1 -- 172.21.15.62:0/195834005 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098013b760 con 0x7f0980143dd0 2024-03-23T11:22:11.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.405+0000 7f0985800640 1 -- 172.21.15.62:0/195834005 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098013bd20 con 0x7f0980143dd0 2024-03-23T11:22:11.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.406+0000 7f0985800640 1 -- 172.21.15.62:0/195834005 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098013beb0 con 0x7f0980143dd0 2024-03-23T11:22:11.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.406+0000 7f0977e00640 1 -- 172.21.15.62:0/195834005 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0978014b40 con 0x7f0980143dd0 2024-03-23T11:22:11.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.407+0000 7f0977e00640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f096006a7a0 0x7f096006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.407+0000 7f0977e00640 1 -- 172.21.15.62:0/195834005 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0978089850 con 0x7f0980143dd0 2024-03-23T11:22:11.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.408+0000 7f097fe00640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f096006a7a0 0x7f096006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.409+0000 7f097fe00640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f096006a7a0 0x7f096006cc60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0970005880 tx=0x7f0970009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:11.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.412+0000 7f0977e00640 1 -- 172.21.15.62:0/195834005 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0978054b90 con 0x7f0980143dd0 2024-03-23T11:22:11.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.610+0000 7f0985800640 1 -- 172.21.15.62:0/195834005 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f09800cbec0 con 0x7f0980143dd0 2024-03-23T11:22:11.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.611+0000 7f0977e00640 1 -- 172.21.15.62:0/195834005 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7f09780581b0 con 0x7f0980143dd0 2024-03-23T11:22:11.609 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:22:11.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 -- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f096006a7a0 msgr2=0x7f096006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f096006a7a0 0x7f096006cc60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0970005880 tx=0x7f0970009040 comp rx=0 tx=0).stop 2024-03-23T11:22:11.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 -- 172.21.15.62:0/195834005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0980143dd0 msgr2=0x7f098015ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0980143dd0 0x7f098015ca10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0978007e40 tx=0x7f097800e4b0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 -- 172.21.15.62:0/195834005 shutdown_connections 2024-03-23T11:22:11.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f096006a7a0 0x7f096006cc60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0980143dd0 0x7f098015ca10 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09800b2820 0x7f098015c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 --2- 172.21.15.62:0/195834005 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0980070090 0x7f0980159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 -- 172.21.15.62:0/195834005 >> 172.21.15.62:0/195834005 conn(0x7f0980066050 msgr2=0x7f0980143020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:11.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 -- 172.21.15.62:0/195834005 shutdown_connections 2024-03-23T11:22:11.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.614+0000 7f0985800640 1 -- 172.21.15.62:0/195834005 wait complete. 2024-03-23T11:22:11.632 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:22:11.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.861+0000 7fb545800640 1 Processor -- start 2024-03-23T11:22:11.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.861+0000 7fb545800640 1 -- start start 2024-03-23T11:22:11.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.862+0000 7fb545800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb54006c300 0x7fb5400ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.862+0000 7fb545800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb54006c6e0 0x7fb54006cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.862+0000 7fb545800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5400cbc60 con 0x7fb5400cafa0 2024-03-23T11:22:11.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.862+0000 7fb545800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5400cbf40 con 0x7fb54006c300 2024-03-23T11:22:11.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.862+0000 7fb545800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5400cc220 con 0x7fb54006c6e0 2024-03-23T11:22:11.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.863+0000 7fb53ea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb54006c300 0x7fb5400ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.863+0000 7fb53e000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb5400cafa0 0x7fb5400cb430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:41252/0 (socket says 172.21.15.62:41252) 2024-03-23T11:22:11.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.863+0000 7fb53e000640 1 -- 172.21.15.62:0/443152978 learned_addr learned my addr 172.21.15.62:0/443152978 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:11.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.863+0000 7fb53f400640 1 --2- 172.21.15.62:0/443152978 >> v2:172.21.15.120:3300/0 conn(0x7fb54006c6e0 0x7fb54006cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.863+0000 7fb53ea00640 1 -- 172.21.15.62:0/443152978 >> v2:172.21.15.120:3300/0 conn(0x7fb54006c6e0 msgr2=0x7fb54006cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:11.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.863+0000 7fb53ea00640 1 --2- 172.21.15.62:0/443152978 >> v2:172.21.15.120:3300/0 conn(0x7fb54006c6e0 0x7fb54006cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.863+0000 7fb53ea00640 1 -- 172.21.15.62:0/443152978 >> v1:172.21.15.62:6789/0 conn(0x7fb5400cafa0 legacy=0x7fb5400cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.863+0000 7fb53ea00640 1 -- 172.21.15.62:0/443152978 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5400cc500 con 0x7fb54006c300 2024-03-23T11:22:11.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.863+0000 7fb53d600640 1 -- 172.21.15.62:0/443152978 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067229778 0 0) 0x7fb5400cbc60 con 0x7fb5400cafa0 2024-03-23T11:22:11.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.865+0000 7fb53ea00640 1 --2- 172.21.15.62:0/443152978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb54006c300 0x7fb5400ccd00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb540070270 tx=0x7fb53002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9c0c4f736c14c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:11.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.865+0000 7fb53d600640 1 -- 172.21.15.62:0/443152978 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530031830 con 0x7fb54006c300 2024-03-23T11:22:11.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.865+0000 7fb53d600640 1 -- 172.21.15.62:0/443152978 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb530031990 con 0x7fb54006c300 2024-03-23T11:22:11.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.865+0000 7fb53d600640 1 -- 172.21.15.62:0/443152978 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530031d20 con 0x7fb54006c300 2024-03-23T11:22:11.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.865+0000 7fb545800640 1 -- 172.21.15.62:0/443152978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb54006c300 msgr2=0x7fb5400ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.865+0000 7fb545800640 1 --2- 172.21.15.62:0/443152978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb54006c300 0x7fb5400ccd00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb540070270 tx=0x7fb53002b8d0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.866+0000 7fb545800640 1 -- 172.21.15.62:0/443152978 shutdown_connections 2024-03-23T11:22:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.866+0000 7fb545800640 1 --2- 172.21.15.62:0/443152978 >> v2:172.21.15.120:3300/0 conn(0x7fb54006c6e0 0x7fb54006cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.866+0000 7fb545800640 1 --2- 172.21.15.62:0/443152978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb54006c300 0x7fb5400ccd00 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.866+0000 7fb545800640 1 -- 172.21.15.62:0/443152978 >> 172.21.15.62:0/443152978 conn(0x7fb5400b02e0 msgr2=0x7fb5400b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.866+0000 7fb545800640 1 -- 172.21.15.62:0/443152978 shutdown_connections 2024-03-23T11:22:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.866+0000 7fb545800640 1 -- 172.21.15.62:0/443152978 wait complete. 2024-03-23T11:22:11.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.867+0000 7fb545800640 1 Processor -- start 2024-03-23T11:22:11.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.868+0000 7fb545800640 1 -- start start 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb545800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb54006c300 0x7fb540155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb545800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb54006c6e0 0x7fb54013e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb545800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5400cafa0 0x7fb54013ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb545800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54015c840 con 0x7fb5400cafa0 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb545800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54013f530 con 0x7fb54006c300 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb545800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54015bf30 con 0x7fb54006c6e0 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb53f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5400cafa0 0x7fb54013ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb53f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5400cafa0 0x7fb54013ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48548/0 (socket says 172.21.15.62:48548) 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb53f400640 1 -- 172.21.15.62:0/1933828962 learned_addr learned my addr 172.21.15.62:0/1933828962 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb53ea00640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb54006c300 0x7fb540155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb53e000640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb54006c6e0 0x7fb54013e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb53f400640 1 -- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb54006c6e0 msgr2=0x7fb54013e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb53f400640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb54006c6e0 0x7fb54013e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb53f400640 1 -- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb54006c300 msgr2=0x7fb540155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb53f400640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb54006c300 0x7fb540155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb53f400640 1 -- 172.21.15.62:0/1933828962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb530009400 con 0x7fb5400cafa0 2024-03-23T11:22:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.869+0000 7fb53e000640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb54006c6e0 0x7fb54013e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.870+0000 7fb53f400640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5400cafa0 0x7fb54013ebf0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb538007e40 tx=0x7fb53800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.870+0000 7fb536a00640 1 -- 172.21.15.62:0/1933828962 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb538014620 con 0x7fb5400cafa0 2024-03-23T11:22:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.870+0000 7fb536a00640 1 -- 172.21.15.62:0/1933828962 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53800ee90 con 0x7fb5400cafa0 2024-03-23T11:22:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.870+0000 7fb545800640 1 -- 172.21.15.62:0/1933828962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb540156500 con 0x7fb5400cafa0 2024-03-23T11:22:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.870+0000 7fb536a00640 1 -- 172.21.15.62:0/1933828962 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5380149e0 con 0x7fb5400cafa0 2024-03-23T11:22:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.870+0000 7fb545800640 1 -- 172.21.15.62:0/1933828962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54013f250 con 0x7fb5400cafa0 2024-03-23T11:22:11.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.871+0000 7fb545800640 1 -- 172.21.15.62:0/1933828962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54015c980 con 0x7fb5400cafa0 2024-03-23T11:22:11.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.872+0000 7fb536a00640 1 -- 172.21.15.62:0/1933828962 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb538014b40 con 0x7fb5400cafa0 2024-03-23T11:22:11.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.873+0000 7fb536a00640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb52006a7a0 0x7fb52006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:11.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.874+0000 7fb536a00640 1 -- 172.21.15.62:0/1933828962 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb5380897b0 con 0x7fb5400cafa0 2024-03-23T11:22:11.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.874+0000 7fb53ea00640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb52006a7a0 0x7fb52006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:11.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.875+0000 7fb53ea00640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb52006a7a0 0x7fb52006cc60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb530036000 tx=0x7fb530008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:11.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:11.878+0000 7fb536a00640 1 -- 172.21.15.62:0/1933828962 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb54015c980 con 0x7fb5400cafa0 2024-03-23T11:22:12.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.078+0000 7fb545800640 1 -- 172.21.15.62:0/1933828962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb54015c980 con 0x7fb5400cafa0 2024-03-23T11:22:12.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.079+0000 7fb536a00640 1 -- 172.21.15.62:0/1933828962 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb54015c980 con 0x7fb5400cafa0 2024-03-23T11:22:12.077 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:22:12.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 -- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb52006a7a0 msgr2=0x7fb52006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:12.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb52006a7a0 0x7fb52006cc60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb530036000 tx=0x7fb530008030 comp rx=0 tx=0).stop 2024-03-23T11:22:12.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 -- 172.21.15.62:0/1933828962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5400cafa0 msgr2=0x7fb54013ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:12.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5400cafa0 0x7fb54013ebf0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb538007e40 tx=0x7fb53800e4b0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 -- 172.21.15.62:0/1933828962 shutdown_connections 2024-03-23T11:22:12.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb52006a7a0 0x7fb52006cc60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5400cafa0 0x7fb54013ebf0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb54006c6e0 0x7fb54013e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 --2- 172.21.15.62:0/1933828962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb54006c300 0x7fb540155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 -- 172.21.15.62:0/1933828962 >> 172.21.15.62:0/1933828962 conn(0x7fb5400b02e0 msgr2=0x7fb5400cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:12.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 -- 172.21.15.62:0/1933828962 shutdown_connections 2024-03-23T11:22:12.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.082+0000 7fb545800640 1 -- 172.21.15.62:0/1933828962 wait complete. 2024-03-23T11:22:12.098 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:22:12.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.325+0000 7f3812a00640 1 Processor -- start 2024-03-23T11:22:12.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.325+0000 7f3812a00640 1 -- start start 2024-03-23T11:22:12.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.326+0000 7f3812a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f380c15c960 0x7f380c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:12.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.326+0000 7f3812a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f380c0d13b0 0x7f380c0b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:12.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.326+0000 7f3812a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c0d0810 con 0x7f380c0cd260 2024-03-23T11:22:12.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.326+0000 7f3812a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380c0cd9d0 con 0x7f380c15c960 2024-03-23T11:22:12.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.326+0000 7f3812a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380c0cdcb0 con 0x7f380c0d13b0 2024-03-23T11:22:12.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.327+0000 7f3811600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f380c15c960 0x7f380c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:12.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.327+0000 7f3811600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f380c15c960 0x7f380c0cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48562/0 (socket says 172.21.15.62:48562) 2024-03-23T11:22:12.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.327+0000 7f3811600640 1 -- 172.21.15.62:0/3998628030 learned_addr learned my addr 172.21.15.62:0/3998628030 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:12.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.327+0000 7f3811600640 1 -- 172.21.15.62:0/3998628030 >> v2:172.21.15.120:3300/0 conn(0x7f380c0d13b0 msgr2=0x7f380c0b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:12.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.327+0000 7f3811600640 1 --2- 172.21.15.62:0/3998628030 >> v2:172.21.15.120:3300/0 conn(0x7f380c0d13b0 0x7f380c0b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.327+0000 7f3811600640 1 -- 172.21.15.62:0/3998628030 >> v1:172.21.15.120:6789/0 conn(0x7f380c0cd260 legacy=0x7f380c0d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:12.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.327+0000 7f3811600640 1 -- 172.21.15.62:0/3998628030 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c0cdf90 con 0x7f380c15c960 2024-03-23T11:22:12.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.328+0000 7f3811600640 1 --2- 172.21.15.62:0/3998628030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c15c960 0x7f380c0cf790 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f380c0b8740 tx=0x7f380002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd7b046b590bc8b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:12.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.329+0000 7f380be00640 1 -- 172.21.15.62:0/3998628030 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800030860 con 0x7f380c15c960 2024-03-23T11:22:12.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.329+0000 7f380be00640 1 -- 172.21.15.62:0/3998628030 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38000309c0 con 0x7f380c15c960 2024-03-23T11:22:12.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.329+0000 7f380be00640 1 -- 172.21.15.62:0/3998628030 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800030d50 con 0x7f380c15c960 2024-03-23T11:22:12.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.329+0000 7f3812a00640 1 -- 172.21.15.62:0/3998628030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c15c960 msgr2=0x7f380c0cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:12.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.329+0000 7f3812a00640 1 --2- 172.21.15.62:0/3998628030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c15c960 0x7f380c0cf790 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f380c0b8740 tx=0x7f380002b9a0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.330+0000 7f3812a00640 1 -- 172.21.15.62:0/3998628030 shutdown_connections 2024-03-23T11:22:12.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.330+0000 7f3812a00640 1 --2- 172.21.15.62:0/3998628030 >> v2:172.21.15.120:3300/0 conn(0x7f380c0d13b0 0x7f380c0b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.330+0000 7f3812a00640 1 --2- 172.21.15.62:0/3998628030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c15c960 0x7f380c0cf790 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.330+0000 7f3812a00640 1 -- 172.21.15.62:0/3998628030 >> 172.21.15.62:0/3998628030 conn(0x7f380c089f60 msgr2=0x7f380c0d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:12.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.330+0000 7f3812a00640 1 -- 172.21.15.62:0/3998628030 shutdown_connections 2024-03-23T11:22:12.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.330+0000 7f3812a00640 1 -- 172.21.15.62:0/3998628030 wait complete. 2024-03-23T11:22:12.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.332+0000 7f3812a00640 1 Processor -- start 2024-03-23T11:22:12.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.332+0000 7f3812a00640 1 -- start start 2024-03-23T11:22:12.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3812a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f380c0cd260 0x7f380c0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:12.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3812a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c0d13b0 0x7f380c1579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:12.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3812a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f380c15c960 0x7f380c15f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:12.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3812a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c15f4e0 con 0x7f380c0d13b0 2024-03-23T11:22:12.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3812a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c157f10 con 0x7f380c0cd260 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3812a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c15e8d0 con 0x7f380c15c960 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3810c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c0d13b0 0x7f380c1579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3810c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c0d13b0 0x7f380c1579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48572/0 (socket says 172.21.15.62:48572) 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3810c00640 1 -- 172.21.15.62:0/1887589004 learned_addr learned my addr 172.21.15.62:0/1887589004 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3811600640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f380c0cd260 0x7f380c0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3812000640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f380c15c960 0x7f380c15f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.333+0000 7f3810c00640 1 -- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f380c15c960 msgr2=0x7f380c15f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3810c00640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f380c15c960 0x7f380c15f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3810c00640 1 -- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f380c0cd260 msgr2=0x7f380c0cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3810c00640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f380c0cd260 0x7f380c0cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3810c00640 1 -- 172.21.15.62:0/1887589004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38000094d0 con 0x7f380c0d13b0 2024-03-23T11:22:12.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3812000640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f380c15c960 0x7f380c15f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:12.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3811600640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f380c0cd260 0x7f380c0cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:12.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3810c00640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c0d13b0 0x7f380c1579d0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00c7a0 tx=0x7f37fc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:12.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3809600640 1 -- 172.21.15.62:0/1887589004 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc00aa50 con 0x7f380c0d13b0 2024-03-23T11:22:12.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3809600640 1 -- 172.21.15.62:0/1887589004 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37fc00abb0 con 0x7f380c0d13b0 2024-03-23T11:22:12.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3809600640 1 -- 172.21.15.62:0/1887589004 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc012420 con 0x7f380c0d13b0 2024-03-23T11:22:12.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3812a00640 1 -- 172.21.15.62:0/1887589004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c15ebb0 con 0x7f380c0d13b0 2024-03-23T11:22:12.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.334+0000 7f3812a00640 1 -- 172.21.15.62:0/1887589004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c15f170 con 0x7f380c0d13b0 2024-03-23T11:22:12.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.335+0000 7f3812a00640 1 -- 172.21.15.62:0/1887589004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c066bb0 con 0x7f380c0d13b0 2024-03-23T11:22:12.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.337+0000 7f3809600640 1 -- 172.21.15.62:0/1887589004 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f37fc012580 con 0x7f380c0d13b0 2024-03-23T11:22:12.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.337+0000 7f3809600640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f006a7a0 0x7f37f006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:12.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.337+0000 7f3809600640 1 -- 172.21.15.62:0/1887589004 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f37fc089330 con 0x7f380c0d13b0 2024-03-23T11:22:12.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.338+0000 7f3811600640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f006a7a0 0x7f37f006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:12.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.339+0000 7f3811600640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f006a7a0 0x7f37f006cc60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3800036000 tx=0x7f380003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:12.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.344+0000 7f3809600640 1 -- 172.21.15.62:0/1887589004 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f37fc054710 con 0x7f380c0d13b0 2024-03-23T11:22:12.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.558+0000 7f3812a00640 1 -- 172.21.15.62:0/1887589004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f380c0d0a80 con 0x7f380c0d13b0 2024-03-23T11:22:12.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.559+0000 7f3809600640 1 -- 172.21.15.62:0/1887589004 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7f37fc057d30 con 0x7f380c0d13b0 2024-03-23T11:22:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:22:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.561+0000 7f3812a00640 1 -- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f006a7a0 msgr2=0x7f37f006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.561+0000 7f3812a00640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f006a7a0 0x7f37f006cc60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3800036000 tx=0x7f380003b040 comp rx=0 tx=0).stop 2024-03-23T11:22:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.561+0000 7f3812a00640 1 -- 172.21.15.62:0/1887589004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c0d13b0 msgr2=0x7f380c1579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.561+0000 7f3812a00640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c0d13b0 0x7f380c1579d0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00c7a0 tx=0x7f37fc00cc60 comp rx=0 tx=0).stop 2024-03-23T11:22:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.562+0000 7f3812a00640 1 -- 172.21.15.62:0/1887589004 shutdown_connections 2024-03-23T11:22:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.562+0000 7f3812a00640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f37f006a7a0 0x7f37f006cc60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.562+0000 7f3812a00640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c0d13b0 0x7f380c1579d0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.562+0000 7f3812a00640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f380c15c960 0x7f380c15f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.562+0000 7f3812a00640 1 --2- 172.21.15.62:0/1887589004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f380c0cd260 0x7f380c0cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.562+0000 7f3812a00640 1 -- 172.21.15.62:0/1887589004 >> 172.21.15.62:0/1887589004 conn(0x7f380c089f60 msgr2=0x7f380c0afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.562+0000 7f3812a00640 1 -- 172.21.15.62:0/1887589004 shutdown_connections 2024-03-23T11:22:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:12.562+0000 7f3812a00640 1 -- 172.21.15.62:0/1887589004 wait complete. 2024-03-23T11:22:12.579 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:22:12.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:12.579 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:22:12.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[53576]: fuse finished with error 0 and tester_r 0 2024-03-23T11:22:12.679 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:22:12.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:12.679 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:22:12.884 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:22:12.884 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:22:12.937 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:22:12.937 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:22:12.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:12.937 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T11:22:12.988 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-23T11:22:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:22:12.989 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:22:12.989 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:22:13.047 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:22:13.047 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:22:13.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:13.047 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T11:22:13.103 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-23T11:22:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:22:13.104 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:22:13.105 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:22:13.160 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:22:13.160 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:22:13.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:13.161 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T11:22:13.214 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-23T11:22:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:22:13.215 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:22:13.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.446+0000 7f8ffbe00640 1 Processor -- start 2024-03-23T11:22:13.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.447+0000 7f8ffbe00640 1 -- start start 2024-03-23T11:22:13.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.448+0000 7f8ffbe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8ffc15fcf0 0x7f8ffc0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.448+0000 7f8ffbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b6dc0 0x7f8ffc1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.448+0000 7f8ffbe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc0d3540 con 0x7f8ffc160420 2024-03-23T11:22:13.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.448+0000 7f8ffbe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ffc160df0 con 0x7f8ffc15fcf0 2024-03-23T11:22:13.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.448+0000 7f8ffbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0ce510 con 0x7f8ffc0b6dc0 2024-03-23T11:22:13.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.448+0000 7f8ff3e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8ffc160420 0x7f8ffc0d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:41264/0 (socket says 172.21.15.62:41264) 2024-03-23T11:22:13.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.448+0000 7f8ff3e00640 1 -- 172.21.15.62:0/1069348342 learned_addr learned my addr 172.21.15.62:0/1069348342 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:13.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffb400640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b6dc0 0x7f8ffc1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffaa00640 1 --2- 172.21.15.62:0/1069348342 >> v2:172.21.15.120:3300/0 conn(0x7f8ffc15fcf0 0x7f8ffc0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584456502 0 0) 0x7f8ffc0d3540 con 0x7f8ffc160420 2024-03-23T11:22:13.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe0003520 con 0x7f8ffc160420 2024-03-23T11:22:13.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2707030059 0 0) 0x7f8fe0003520 con 0x7f8ffc160420 2024-03-23T11:22:13.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b6dc0 msgr2=0x7f8ffc1608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:13.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b6dc0 0x7f8ffc1608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 >> v2:172.21.15.120:3300/0 conn(0x7f8ffc15fcf0 msgr2=0x7f8ffc0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:13.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 --2- 172.21.15.62:0/1069348342 >> v2:172.21.15.120:3300/0 conn(0x7f8ffc15fcf0 0x7f8ffc0d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc0cdc70 con 0x7f8ffc160420 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f8fe4001670 con 0x7f8ffc160420 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 >> v1:172.21.15.62:6789/0 conn(0x7f8ffc160420 legacy=0x7f8ffc0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0003970 0x7f8fe0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00047c0 0x7f8fe0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b6dc0 0x7f8fe0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc160df0 con 0x7f8fe00047c0 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0ce510 con 0x7f8fe0003970 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.449+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe000ba00 con 0x7f8ffc0b6dc0 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.450+0000 7f8ff3e00640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00047c0 0x7f8fe0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.450+0000 7f8ffb400640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b6dc0 0x7f8fe0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.450+0000 7f8ffaa00640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0003970 0x7f8fe0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.450+0000 7f8ff3e00640 1 -- 172.21.15.62:0/1069348342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b6dc0 msgr2=0x7f8fe0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.450+0000 7f8ff3e00640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0b6dc0 0x7f8fe0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.450+0000 7f8ff3e00640 1 -- 172.21.15.62:0/1069348342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0003970 msgr2=0x7f8fe0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:13.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.450+0000 7f8ff3e00640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0003970 0x7f8fe0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.450+0000 7f8ff3e00640 1 -- 172.21.15.62:0/1069348342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe0003520 con 0x7f8fe00047c0 2024-03-23T11:22:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.451+0000 7f8ff3e00640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00047c0 0x7f8fe0004c40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c290 tx=0x7f8fe00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec67cc6973d6bf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.452+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4005ad0 con 0x7f8fe00047c0 2024-03-23T11:22:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.452+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4005c30 con 0x7f8fe00047c0 2024-03-23T11:22:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.452+0000 7f8ffa000640 1 -- 172.21.15.62:0/1069348342 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe402c8d0 con 0x7f8fe00047c0 2024-03-23T11:22:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.452+0000 7f8ffbe00640 1 -- 172.21.15.62:0/1069348342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00047c0 msgr2=0x7f8fe0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.452+0000 7f8ffbe00640 1 --2- 172.21.15.62:0/1069348342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00047c0 0x7f8fe0004c40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c290 tx=0x7f8fe00057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.452+0000 7f8ffb400640 1 -- 172.21.15.62:0/1069348342 reap_dead start 2024-03-23T11:22:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.452+0000 7f8ffbe00640 1 -- 172.21.15.62:0/1069348342 shutdown_connections 2024-03-23T11:22:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.452+0000 7f8ffbe00640 1 -- 172.21.15.62:0/1069348342 >> 172.21.15.62:0/1069348342 conn(0x7f8ffc089f60 msgr2=0x7f8ffc0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.452+0000 7f8ffbe00640 1 -- 172.21.15.62:0/1069348342 shutdown_connections 2024-03-23T11:22:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.452+0000 7f8ffbe00640 1 -- 172.21.15.62:0/1069348342 wait complete. 2024-03-23T11:22:13.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.454+0000 7f8ffbe00640 1 Processor -- start 2024-03-23T11:22:13.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.454+0000 7f8ffbe00640 1 -- start start 2024-03-23T11:22:13.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.455+0000 7f8ffbe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe00047c0 0x7f8ffc09fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.455+0000 7f8ffbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ffc09ed40 0x7f8ffc15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.455+0000 7f8ffbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0c5ed0 0x7f8ffc0c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.455+0000 7f8ffbe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc09f1a0 con 0x7f8ffc09ed40 2024-03-23T11:22:13.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.455+0000 7f8ffbe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc15c470 con 0x7f8fe00047c0 2024-03-23T11:22:13.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.455+0000 7f8ffbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc06cc40 con 0x7f8ffc0c5ed0 2024-03-23T11:22:13.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.456+0000 7f8ff3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ffc09ed40 0x7f8ffc15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.456+0000 7f8ff3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ffc09ed40 0x7f8ffc15bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48584/0 (socket says 172.21.15.62:48584) 2024-03-23T11:22:13.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.456+0000 7f8ff3e00640 1 -- 172.21.15.62:0/3869808704 learned_addr learned my addr 172.21.15.62:0/3869808704 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:13.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.456+0000 7f8ffaa00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe00047c0 0x7f8ffc09fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.456+0000 7f8ffb400640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0c5ed0 0x7f8ffc0c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.456+0000 7f8ff3e00640 1 -- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0c5ed0 msgr2=0x7f8ffc0c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:13.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.456+0000 7f8ff3e00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0c5ed0 0x7f8ffc0c6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.456+0000 7f8ff3e00640 1 -- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe00047c0 msgr2=0x7f8ffc09fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:13.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.456+0000 7f8ff3e00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe00047c0 0x7f8ffc09fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.456+0000 7f8ff3e00640 1 -- 172.21.15.62:0/3869808704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc0a03b0 con 0x7f8ffc09ed40 2024-03-23T11:22:13.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.457+0000 7f8ff3e00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ffc09ed40 0x7f8ffc15bf30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c3b0 tx=0x7f8fe4000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:13.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.457+0000 7f8ff2a00640 1 -- 172.21.15.62:0/3869808704 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe402cce0 con 0x7f8ffc09ed40 2024-03-23T11:22:13.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.457+0000 7f8ff2a00640 1 -- 172.21.15.62:0/3869808704 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe402ce60 con 0x7f8ffc09ed40 2024-03-23T11:22:13.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.457+0000 7f8ff2a00640 1 -- 172.21.15.62:0/3869808704 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40072b0 con 0x7f8ffc09ed40 2024-03-23T11:22:13.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.457+0000 7f8ffbe00640 1 -- 172.21.15.62:0/3869808704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc0c7e80 con 0x7f8ffc09ed40 2024-03-23T11:22:13.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.457+0000 7f8ffbe00640 1 -- 172.21.15.62:0/3869808704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc0c8440 con 0x7f8ffc09ed40 2024-03-23T11:22:13.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.458+0000 7f8ffbe00640 1 -- 172.21.15.62:0/3869808704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc066bb0 con 0x7f8ffc09ed40 2024-03-23T11:22:13.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.459+0000 7f8ff2a00640 1 -- 172.21.15.62:0/3869808704 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8fe4032070 con 0x7f8ffc09ed40 2024-03-23T11:22:13.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.460+0000 7f8ff2a00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fd806a7d0 0x7f8fd806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.460+0000 7f8ff2a00640 1 -- 172.21.15.62:0/3869808704 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8fe403c040 con 0x7f8ffc09ed40 2024-03-23T11:22:13.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.461+0000 7f8ffaa00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fd806a7d0 0x7f8fd806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.462+0000 7f8ffaa00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fd806a7d0 0x7f8fd806cc90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009a10 tx=0x7f8fec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:13.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.464+0000 7f8ff2a00640 1 -- 172.21.15.62:0/3869808704 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8fe4075600 con 0x7f8ffc09ed40 2024-03-23T11:22:13.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.701+0000 7f8ffbe00640 1 -- 172.21.15.62:0/3869808704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ffc0cea40 con 0x7f8ffc09ed40 2024-03-23T11:22:13.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.702+0000 7f8ff2a00640 1 -- 172.21.15.62:0/3869808704 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f8fe4001c80 con 0x7f8ffc09ed40 2024-03-23T11:22:13.700 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:13.700 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3787987198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3787987198},{"type":"v1","addr":"172.21.15.120:6835","nonce":3787987198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2749498408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2749498408},{"type":"v1","addr":"172.21.15.120:6837","nonce":2749498408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2325988870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2325988870},{"type":"v1","addr":"172.21.15.62:6833","nonce":2325988870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:21:25.687136+0000","modified":"2024-03-23T11:21:28.704192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6835/1159419089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1159419089},{"type":"v1","addr":"172.21.15.62:6835","nonce":1159419089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T11:22:13.700 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-03-23T11:22:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.705+0000 7f8ffbe00640 1 -- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fd806a7d0 msgr2=0x7f8fd806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.705+0000 7f8ffbe00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fd806a7d0 0x7f8fd806cc90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009a10 tx=0x7f8fec008040 comp rx=0 tx=0).stop 2024-03-23T11:22:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.705+0000 7f8ffbe00640 1 -- 172.21.15.62:0/3869808704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ffc09ed40 msgr2=0x7f8ffc15bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.705+0000 7f8ffbe00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ffc09ed40 0x7f8ffc15bf30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c3b0 tx=0x7f8fe4000fe0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.705+0000 7f8ffbe00640 1 -- 172.21.15.62:0/3869808704 shutdown_connections 2024-03-23T11:22:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.705+0000 7f8ffbe00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fd806a7d0 0x7f8fd806cc90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.705+0000 7f8ffbe00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ffc09ed40 0x7f8ffc15bf30 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.705+0000 7f8ffbe00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ffc0c5ed0 0x7f8ffc0c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.705+0000 7f8ffbe00640 1 --2- 172.21.15.62:0/3869808704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe00047c0 0x7f8ffc09fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.705+0000 7f8ffbe00640 1 -- 172.21.15.62:0/3869808704 >> 172.21.15.62:0/3869808704 conn(0x7f8ffc089f60 msgr2=0x7f8ffc0cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.706+0000 7f8ffbe00640 1 -- 172.21.15.62:0/3869808704 shutdown_connections 2024-03-23T11:22:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.706+0000 7f8ffbe00640 1 -- 172.21.15.62:0/3869808704 wait complete. 2024-03-23T11:22:13.725 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:22:13.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.945+0000 7f6495a00640 1 Processor -- start 2024-03-23T11:22:13.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.946+0000 7f6495a00640 1 -- start start 2024-03-23T11:22:13.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.947+0000 7f6495a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f649015cce0 0x7f64900b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.947+0000 7f6495a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900cd690 0x7f64900cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.947+0000 7f6495a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64900d1100 con 0x7f649015c900 2024-03-23T11:22:13.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.947+0000 7f6495a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64900ce060 con 0x7f649015cce0 2024-03-23T11:22:13.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.947+0000 7f6495a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64900d13e0 con 0x7f64900cd690 2024-03-23T11:22:13.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.948+0000 7f648ea00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f649015c900 0x7f64900d21a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:41276/0 (socket says 172.21.15.62:41276) 2024-03-23T11:22:13.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.948+0000 7f648ea00640 1 -- 172.21.15.62:0/649155993 learned_addr learned my addr 172.21.15.62:0/649155993 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:13.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.948+0000 7f648e000640 1 --2- 172.21.15.62:0/649155993 >> v2:172.21.15.120:3300/0 conn(0x7f649015cce0 0x7f64900b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.948+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921721058 0 0) 0x7f64900d1100 con 0x7f649015c900 2024-03-23T11:22:13.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.948+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6474003520 con 0x7f649015c900 2024-03-23T11:22:13.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.948+0000 7f648f400640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900cd690 0x7f64900cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2517017004 0 0) 0x7f6474003520 con 0x7f649015c900 2024-03-23T11:22:13.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900cd690 msgr2=0x7f64900cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900cd690 0x7f64900cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 >> v2:172.21.15.120:3300/0 conn(0x7f649015cce0 msgr2=0x7f64900b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 --2- 172.21.15.62:0/649155993 >> v2:172.21.15.120:3300/0 conn(0x7f649015cce0 0x7f64900b82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64900d16c0 con 0x7f649015c900 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f6480001670 con 0x7f649015c900 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 >> v1:172.21.15.62:6789/0 conn(0x7f649015c900 legacy=0x7f64900d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900cd690 0x7f6474003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6474003ef0 0x7f64740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 0x7f6474005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64900ce060 con 0x7f6474004d00 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64900d13e0 con 0x7f6474003ef0 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647400b9d0 con 0x7f64900cd690 2024-03-23T11:22:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648f400640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 0x7f6474005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648e000640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6474003ef0 0x7f64740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.949+0000 7f648ea00640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900cd690 0x7f6474003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.950+0000 7f648f400640 1 -- 172.21.15.62:0/649155993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900cd690 msgr2=0x7f6474003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:13.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.950+0000 7f648f400640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900cd690 0x7f6474003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.950+0000 7f648f400640 1 -- 172.21.15.62:0/649155993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6474003ef0 msgr2=0x7f64740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:13.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.950+0000 7f648f400640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6474003ef0 0x7f64740047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.950+0000 7f648f400640 1 -- 172.21.15.62:0/649155993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6474003520 con 0x7f6474004d00 2024-03-23T11:22:13.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.951+0000 7f648f400640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 0x7f6474005180 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6488007360 tx=0x7f64740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fbac54fbf0aa43 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:13.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.951+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648802ccd0 con 0x7f6474004d00 2024-03-23T11:22:13.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.951+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648802ce30 con 0x7f6474004d00 2024-03-23T11:22:13.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.951+0000 7f648d600640 1 -- 172.21.15.62:0/649155993 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6488034660 con 0x7f6474004d00 2024-03-23T11:22:13.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.952+0000 7f6495a00640 1 -- 172.21.15.62:0/649155993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 msgr2=0x7f6474005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:13.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.952+0000 7f6495a00640 1 --2- 172.21.15.62:0/649155993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 0x7f6474005180 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6488007360 tx=0x7f64740057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.952+0000 7f648f400640 1 -- 172.21.15.62:0/649155993 reap_dead start 2024-03-23T11:22:13.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.952+0000 7f6495a00640 1 -- 172.21.15.62:0/649155993 shutdown_connections 2024-03-23T11:22:13.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.952+0000 7f6495a00640 1 -- 172.21.15.62:0/649155993 >> 172.21.15.62:0/649155993 conn(0x7f6490071390 msgr2=0x7f6490071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:13.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.952+0000 7f6495a00640 1 -- 172.21.15.62:0/649155993 shutdown_connections 2024-03-23T11:22:13.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.952+0000 7f6495a00640 1 -- 172.21.15.62:0/649155993 wait complete. 2024-03-23T11:22:13.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.954+0000 7f6495a00640 1 Processor -- start 2024-03-23T11:22:13.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.954+0000 7f6495a00640 1 -- start start 2024-03-23T11:22:13.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f6495a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 0x7f64901560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f6495a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900aa550 0x7f64900aa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f6495a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649015b2e0 0x7f649015b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f6495a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64901f2480 con 0x7f6474004d00 2024-03-23T11:22:13.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f6495a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64901f22b0 con 0x7f649015b2e0 2024-03-23T11:22:13.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f6495a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649015bca0 con 0x7f64900aa550 2024-03-23T11:22:13.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f648ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 0x7f64901560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f648ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 0x7f64901560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48602/0 (socket says 172.21.15.62:48602) 2024-03-23T11:22:13.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f648ea00640 1 -- 172.21.15.62:0/3724576511 learned_addr learned my addr 172.21.15.62:0/3724576511 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:13.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f648f400640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649015b2e0 0x7f649015b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f648ea00640 1 -- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900aa550 msgr2=0x7f64900aa9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:13.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.955+0000 7f648ea00640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900aa550 0x7f64900aa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.956+0000 7f648ea00640 1 -- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649015b2e0 msgr2=0x7f649015b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:13.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.956+0000 7f648ea00640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649015b2e0 0x7f649015b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:13.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.956+0000 7f648ea00640 1 -- 172.21.15.62:0/3724576511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64900cc020 con 0x7f6474004d00 2024-03-23T11:22:13.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.956+0000 7f648ea00640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 0x7f64901560b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f648000e8c0 tx=0x7f648000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:13.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.956+0000 7f6486a00640 1 -- 172.21.15.62:0/3724576511 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480005320 con 0x7f6474004d00 2024-03-23T11:22:13.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.956+0000 7f6486a00640 1 -- 172.21.15.62:0/3724576511 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64800054a0 con 0x7f6474004d00 2024-03-23T11:22:13.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.956+0000 7f6486a00640 1 -- 172.21.15.62:0/3724576511 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64800047e0 con 0x7f6474004d00 2024-03-23T11:22:13.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.956+0000 7f6495a00640 1 -- 172.21.15.62:0/3724576511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64900cc300 con 0x7f6474004d00 2024-03-23T11:22:13.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.956+0000 7f6495a00640 1 -- 172.21.15.62:0/3724576511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64900cc8c0 con 0x7f6474004d00 2024-03-23T11:22:13.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.957+0000 7f6495a00640 1 -- 172.21.15.62:0/3724576511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64901f25c0 con 0x7f6474004d00 2024-03-23T11:22:13.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.958+0000 7f6486a00640 1 -- 172.21.15.62:0/3724576511 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6480004960 con 0x7f6474004d00 2024-03-23T11:22:13.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.959+0000 7f6486a00640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f647006a7a0 0x7f647006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:13.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.959+0000 7f6486a00640 1 -- 172.21.15.62:0/3724576511 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6480088a50 con 0x7f6474004d00 2024-03-23T11:22:13.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.960+0000 7f648e000640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f647006a7a0 0x7f647006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:13.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.961+0000 7f648e000640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f647006a7a0 0x7f647006cc60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f64900cafb0 tx=0x7f647c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:13.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:13.964+0000 7f6486a00640 1 -- 172.21.15.62:0/3724576511 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64901f25c0 con 0x7f6474004d00 2024-03-23T11:22:14.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.161+0000 7f6495a00640 1 -- 172.21.15.62:0/3724576511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64901f25c0 con 0x7f6474004d00 2024-03-23T11:22:14.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.164+0000 7f6486a00640 1 -- 172.21.15.62:0/3724576511 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16197 (secure 0 0 0) 0x7f64901f25c0 con 0x7f6474004d00 2024-03-23T11:22:14.162 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:14.162 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":50,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:21:43.375126+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:21:23.756222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-23T11:21:24.715577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.2a","mappings":[{"from":0,"to":5}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:22:14.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 -- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f647006a7a0 msgr2=0x7f647006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f647006a7a0 0x7f647006cc60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f64900cafb0 tx=0x7f647c008040 comp rx=0 tx=0).stop 2024-03-23T11:22:14.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 -- 172.21.15.62:0/3724576511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 msgr2=0x7f64901560b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 0x7f64901560b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f648000e8c0 tx=0x7f648000ed80 comp rx=0 tx=0).stop 2024-03-23T11:22:14.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 -- 172.21.15.62:0/3724576511 shutdown_connections 2024-03-23T11:22:14.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f647006a7a0 0x7f647006cc60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f649015b2e0 0x7f649015b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64900aa550 0x7f64900aa9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 --2- 172.21.15.62:0/3724576511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6474004d00 0x7f64901560b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 -- 172.21.15.62:0/3724576511 >> 172.21.15.62:0/3724576511 conn(0x7f6490071390 msgr2=0x7f649015ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:14.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 -- 172.21.15.62:0/3724576511 shutdown_connections 2024-03-23T11:22:14.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.167+0000 7f6495a00640 1 -- 172.21.15.62:0/3724576511 wait complete. 2024-03-23T11:22:14.184 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:21:23.756222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:21:24.715577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:22:14.184 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:22:14.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.413+0000 7f59e8c00640 1 Processor -- start 2024-03-23T11:22:14.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.413+0000 7f59e8c00640 1 -- start start 2024-03-23T11:22:14.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.415+0000 7f59e8c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f59e41571e0 0x7f59e40ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.415+0000 7f59e8c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f59e40ade70 0x7f59e406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.415+0000 7f59e8c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e41613a0 0x7f59e415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.415+0000 7f59e8c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e406cf40 con 0x7f59e41571e0 2024-03-23T11:22:14.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.415+0000 7f59e8c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e41575c0 con 0x7f59e40ade70 2024-03-23T11:22:14.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.415+0000 7f59e8c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e406c3a0 con 0x7f59e41613a0 2024-03-23T11:22:14.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.416+0000 7f59e1e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f59e41571e0 0x7f59e40ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.416+0000 7f59e1e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f59e41571e0 0x7f59e40ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48614/0 (socket says 172.21.15.62:48614) 2024-03-23T11:22:14.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.416+0000 7f59e1e00640 1 -- 172.21.15.62:0/688745389 learned_addr learned my addr 172.21.15.62:0/688745389 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:14.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.416+0000 7f59e2800640 1 --2- 172.21.15.62:0/688745389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e41613a0 0x7f59e415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.416+0000 7f59dbe00640 1 --2- 172.21.15.62:0/688745389 >> v2:172.21.15.120:3300/0 conn(0x7f59e40ade70 0x7f59e406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.416+0000 7f59e1e00640 1 -- 172.21.15.62:0/688745389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e41613a0 msgr2=0x7f59e415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:14.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.416+0000 7f59e1e00640 1 --2- 172.21.15.62:0/688745389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e41613a0 0x7f59e415e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.416+0000 7f59e1e00640 1 -- 172.21.15.62:0/688745389 >> v2:172.21.15.120:3300/0 conn(0x7f59e40ade70 msgr2=0x7f59e406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:14.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.416+0000 7f59e1e00640 1 --2- 172.21.15.62:0/688745389 >> v2:172.21.15.120:3300/0 conn(0x7f59e40ade70 0x7f59e406d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.416+0000 7f59e1e00640 1 -- 172.21.15.62:0/688745389 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e406c610 con 0x7f59e41571e0 2024-03-23T11:22:14.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.416+0000 7f59dbe00640 1 --2- 172.21.15.62:0/688745389 >> v2:172.21.15.120:3300/0 conn(0x7f59e40ade70 0x7f59e406d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:14.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.417+0000 7f59e1e00640 1 --2- 172.21.15.62:0/688745389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e41571e0 0x7f59e40ad930 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f59e40b1280 tx=0x7f59d402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6987e8ee8d7d14ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:14.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.418+0000 7f59e1400640 1 -- 172.21.15.62:0/688745389 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d40317e0 con 0x7f59e41571e0 2024-03-23T11:22:14.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.418+0000 7f59e1400640 1 -- 172.21.15.62:0/688745389 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d4031940 con 0x7f59e41571e0 2024-03-23T11:22:14.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.418+0000 7f59e1400640 1 -- 172.21.15.62:0/688745389 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4031cd0 con 0x7f59e41571e0 2024-03-23T11:22:14.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.419+0000 7f59e8c00640 1 -- 172.21.15.62:0/688745389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e41571e0 msgr2=0x7f59e40ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.419+0000 7f59e8c00640 1 --2- 172.21.15.62:0/688745389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e41571e0 0x7f59e40ad930 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f59e40b1280 tx=0x7f59d402b880 comp rx=0 tx=0).stop 2024-03-23T11:22:14.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.419+0000 7f59e8c00640 1 -- 172.21.15.62:0/688745389 shutdown_connections 2024-03-23T11:22:14.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.419+0000 7f59e8c00640 1 --2- 172.21.15.62:0/688745389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e41613a0 0x7f59e415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.419+0000 7f59e8c00640 1 --2- 172.21.15.62:0/688745389 >> v2:172.21.15.120:3300/0 conn(0x7f59e40ade70 0x7f59e406d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.419+0000 7f59e8c00640 1 --2- 172.21.15.62:0/688745389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e41571e0 0x7f59e40ad930 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.419+0000 7f59e8c00640 1 -- 172.21.15.62:0/688745389 >> 172.21.15.62:0/688745389 conn(0x7f59e40b02e0 msgr2=0x7f59e40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:14.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.419+0000 7f59e8c00640 1 -- 172.21.15.62:0/688745389 shutdown_connections 2024-03-23T11:22:14.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.419+0000 7f59e8c00640 1 -- 172.21.15.62:0/688745389 wait complete. 2024-03-23T11:22:14.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.421+0000 7f59e8c00640 1 Processor -- start 2024-03-23T11:22:14.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.421+0000 7f59e8c00640 1 -- start start 2024-03-23T11:22:14.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e8c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e40ade70 0x7f59e4142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e8c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e41571e0 0x7f59e4159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e8c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e41613a0 0x7f59e4159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e8c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e415c730 con 0x7f59e41613a0 2024-03-23T11:22:14.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e8c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4143430 con 0x7f59e40ade70 2024-03-23T11:22:14.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e8c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4158400 con 0x7f59e41571e0 2024-03-23T11:22:14.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e41613a0 0x7f59e4159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e41613a0 0x7f59e4159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48618/0 (socket says 172.21.15.62:48618) 2024-03-23T11:22:14.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e2800640 1 -- 172.21.15.62:0/2789307523 learned_addr learned my addr 172.21.15.62:0/2789307523 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:14.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e1e00640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e40ade70 0x7f59e4142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59dbe00640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e41571e0 0x7f59e4159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e2800640 1 -- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e41571e0 msgr2=0x7f59e4159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.422+0000 7f59e2800640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e41571e0 0x7f59e4159250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.423+0000 7f59e2800640 1 -- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e40ade70 msgr2=0x7f59e4142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.423+0000 7f59e2800640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e40ade70 0x7f59e4142ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.423+0000 7f59e2800640 1 -- 172.21.15.62:0/2789307523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d40094d0 con 0x7f59e41613a0 2024-03-23T11:22:14.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.423+0000 7f59e2800640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e41613a0 0x7f59e4159790 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00cd20 tx=0x7f59dc00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:14.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.423+0000 7f59da000640 1 -- 172.21.15.62:0/2789307523 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc003b90 con 0x7f59e41613a0 2024-03-23T11:22:14.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.423+0000 7f59da000640 1 -- 172.21.15.62:0/2789307523 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc003cf0 con 0x7f59e41613a0 2024-03-23T11:22:14.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.423+0000 7f59da000640 1 -- 172.21.15.62:0/2789307523 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc004240 con 0x7f59e41613a0 2024-03-23T11:22:14.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.423+0000 7f59e8c00640 1 -- 172.21.15.62:0/2789307523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e41586e0 con 0x7f59e41613a0 2024-03-23T11:22:14.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.423+0000 7f59e8c00640 1 -- 172.21.15.62:0/2789307523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e4158ca0 con 0x7f59e41613a0 2024-03-23T11:22:14.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.425+0000 7f59e8c00640 1 -- 172.21.15.62:0/2789307523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e40cae30 con 0x7f59e41613a0 2024-03-23T11:22:14.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.425+0000 7f59da000640 1 -- 172.21.15.62:0/2789307523 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f59dc0043a0 con 0x7f59e41613a0 2024-03-23T11:22:14.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.427+0000 7f59da000640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f59c006a7d0 0x7f59c006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.427+0000 7f59da000640 1 -- 172.21.15.62:0/2789307523 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f59dc012070 con 0x7f59e41613a0 2024-03-23T11:22:14.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.427+0000 7f59e1e00640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f59c006a7d0 0x7f59c006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.429+0000 7f59e1e00640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f59c006a7d0 0x7f59c006cc90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f59d4036000 tx=0x7f59d403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:14.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.431+0000 7f59da000640 1 -- 172.21.15.62:0/2789307523 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59e40cae30 con 0x7f59e41613a0 2024-03-23T11:22:14.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.631+0000 7f59e8c00640 1 -- 172.21.15.62:0/2789307523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59e40cae30 con 0x7f59e41613a0 2024-03-23T11:22:14.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.632+0000 7f59da000640 1 -- 172.21.15.62:0/2789307523 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f59e40cae30 con 0x7f59e41613a0 2024-03-23T11:22:14.630 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 -- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f59c006a7d0 msgr2=0x7f59c006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f59c006a7d0 0x7f59c006cc90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f59d4036000 tx=0x7f59d403d040 comp rx=0 tx=0).stop 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 -- 172.21.15.62:0/2789307523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e41613a0 msgr2=0x7f59e4159790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e41613a0 0x7f59e4159790 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00cd20 tx=0x7f59dc00a660 comp rx=0 tx=0).stop 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 -- 172.21.15.62:0/2789307523 shutdown_connections 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f59c006a7d0 0x7f59c006cc90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e41613a0 0x7f59e4159790 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e41571e0 0x7f59e4159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 --2- 172.21.15.62:0/2789307523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e40ade70 0x7f59e4142ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 -- 172.21.15.62:0/2789307523 >> 172.21.15.62:0/2789307523 conn(0x7f59e40b02e0 msgr2=0x7f59e415ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 -- 172.21.15.62:0/2789307523 shutdown_connections 2024-03-23T11:22:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.635+0000 7f59e8c00640 1 -- 172.21.15.62:0/2789307523 wait complete. 2024-03-23T11:22:14.653 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:22:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.885+0000 7f2bed400640 1 Processor -- start 2024-03-23T11:22:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.885+0000 7f2bed400640 1 -- start start 2024-03-23T11:22:14.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.887+0000 7f2bed400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2be815c830 0x7f2be80ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.887+0000 7f2bed400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be81608d0 con 0x7f2be8160310 2024-03-23T11:22:14.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.887+0000 7f2bed400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be8160bb0 con 0x7f2be80ce250 2024-03-23T11:22:14.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.887+0000 7f2bed400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be80addb0 con 0x7f2be815c830 2024-03-23T11:22:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.887+0000 7f2be7400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2be8160310 0x7f2be80ab980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:41288/0 (socket says 172.21.15.62:41288) 2024-03-23T11:22:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.887+0000 7f2be7400640 1 -- 172.21.15.62:0/3880343156 learned_addr learned my addr 172.21.15.62:0/3880343156 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.887+0000 7f2be7e00640 1 --2- 172.21.15.62:0/3880343156 >> v2:172.21.15.120:3300/0 conn(0x7f2be815c830 0x7f2be80ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866244446 0 0) 0x7f2be81608d0 con 0x7f2be8160310 2024-03-23T11:22:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bcc003520 con 0x7f2be8160310 2024-03-23T11:22:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 457571534 0 0) 0x7f2bcc003520 con 0x7f2be8160310 2024-03-23T11:22:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 >> v2:172.21.15.120:3300/0 conn(0x7f2be815c830 msgr2=0x7f2be80ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 --2- 172.21.15.62:0/3880343156 >> v2:172.21.15.120:3300/0 conn(0x7f2be815c830 0x7f2be80ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 >> v1:172.21.15.120:6789/0 conn(0x7f2be80ce250 legacy=0x7f2be815d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be8160e90 con 0x7f2be8160310 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f2bd4001670 con 0x7f2be8160310 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 >> v1:172.21.15.62:6789/0 conn(0x7f2be8160310 legacy=0x7f2be80ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 --2- 172.21.15.62:0/3880343156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bcc0039c0 0x7f2bcc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 --2- 172.21.15.62:0/3880343156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 0x7f2bcc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 --2- 172.21.15.62:0/3880343156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bcc005180 0x7f2bcc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc00c050 con 0x7f2bcc0047c0 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc00c1c0 con 0x7f2bcc0039c0 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.888+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc00c380 con 0x7f2bcc005180 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.889+0000 7f2be7400640 1 --2- 172.21.15.62:0/3880343156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 0x7f2bcc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.889+0000 7f2be7e00640 1 --2- 172.21.15.62:0/3880343156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bcc0039c0 0x7f2bcc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.889+0000 7f2beca00640 1 --2- 172.21.15.62:0/3880343156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bcc005180 0x7f2bcc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.889+0000 7f2be7400640 1 -- 172.21.15.62:0/3880343156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bcc005180 msgr2=0x7f2bcc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.889+0000 7f2be7400640 1 --2- 172.21.15.62:0/3880343156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bcc005180 0x7f2bcc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.889+0000 7f2be7400640 1 -- 172.21.15.62:0/3880343156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bcc0039c0 msgr2=0x7f2bcc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:14.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.889+0000 7f2be7400640 1 --2- 172.21.15.62:0/3880343156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bcc0039c0 0x7f2bcc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.889+0000 7f2beca00640 1 -- 172.21.15.62:0/3880343156 reap_dead start 2024-03-23T11:22:14.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.889+0000 7f2be7400640 1 -- 172.21.15.62:0/3880343156 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc003520 con 0x7f2bcc0047c0 2024-03-23T11:22:14.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.890+0000 7f2be7400640 1 --2- 172.21.15.62:0/3880343156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 0x7f2bcc004c40 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400c0e0 tx=0x7f2bcc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e38774fa195e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:14.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.890+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4005ad0 con 0x7f2bcc0047c0 2024-03-23T11:22:14.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.890+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd4005c50 con 0x7f2bcc0047c0 2024-03-23T11:22:14.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.890+0000 7f2be6a00640 1 -- 172.21.15.62:0/3880343156 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd402c810 con 0x7f2bcc0047c0 2024-03-23T11:22:14.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.891+0000 7f2bed400640 1 -- 172.21.15.62:0/3880343156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 msgr2=0x7f2bcc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.891+0000 7f2bed400640 1 --2- 172.21.15.62:0/3880343156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 0x7f2bcc004c40 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400c0e0 tx=0x7f2bcc0057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.891+0000 7f2bed400640 1 -- 172.21.15.62:0/3880343156 shutdown_connections 2024-03-23T11:22:14.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.891+0000 7f2bed400640 1 --2- 172.21.15.62:0/3880343156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 0x7f2bcc004c40 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.891+0000 7f2bed400640 1 -- 172.21.15.62:0/3880343156 >> 172.21.15.62:0/3880343156 conn(0x7f2be8066050 msgr2=0x7f2be8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:14.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.891+0000 7f2bed400640 1 -- 172.21.15.62:0/3880343156 shutdown_connections 2024-03-23T11:22:14.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.891+0000 7f2bed400640 1 -- 172.21.15.62:0/3880343156 wait complete. 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.893+0000 7f2bed400640 1 Processor -- start 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.893+0000 7f2bed400640 1 -- start start 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.894+0000 7f2bed400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 0x7f2be80c4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.894+0000 7f2bed400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2be80c49c0 0x7f2be80c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.894+0000 7f2bed400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2be80c8f80 0x7f2be81495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.894+0000 7f2bed400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80c9400 con 0x7f2bcc0047c0 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.894+0000 7f2bed400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80c4f40 con 0x7f2be80c49c0 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.894+0000 7f2bed400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80c5220 con 0x7f2be80c8f80 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.894+0000 7f2be7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 0x7f2be80c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.894+0000 7f2be7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 0x7f2be80c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48638/0 (socket says 172.21.15.62:48638) 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.894+0000 7f2be7e00640 1 -- 172.21.15.62:0/86180975 learned_addr learned my addr 172.21.15.62:0/86180975 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.895+0000 7f2be7400640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2be80c49c0 0x7f2be80c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.895+0000 7f2beca00640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2be80c8f80 0x7f2be81495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.895+0000 7f2be7e00640 1 -- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2be80c8f80 msgr2=0x7f2be81495f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.895+0000 7f2be7e00640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2be80c8f80 0x7f2be81495f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.895+0000 7f2be7e00640 1 -- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2be80c49c0 msgr2=0x7f2be80c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:14.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.895+0000 7f2be7e00640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2be80c49c0 0x7f2be80c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:14.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.895+0000 7f2be7e00640 1 -- 172.21.15.62:0/86180975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80c5500 con 0x7f2bcc0047c0 2024-03-23T11:22:14.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.895+0000 7f2be7e00640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 0x7f2be80c4480 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8011510 tx=0x7f2bd80119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:14.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.895+0000 7f2bdfe00640 1 -- 172.21.15.62:0/86180975 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8015870 con 0x7f2bcc0047c0 2024-03-23T11:22:14.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.895+0000 7f2bdfe00640 1 -- 172.21.15.62:0/86180975 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd80159d0 con 0x7f2bcc0047c0 2024-03-23T11:22:14.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.895+0000 7f2bed400640 1 -- 172.21.15.62:0/86180975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be80c57e0 con 0x7f2bcc0047c0 2024-03-23T11:22:14.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.896+0000 7f2bdfe00640 1 -- 172.21.15.62:0/86180975 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8015d40 con 0x7f2bcc0047c0 2024-03-23T11:22:14.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.896+0000 7f2bed400640 1 -- 172.21.15.62:0/86180975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be814f8b0 con 0x7f2bcc0047c0 2024-03-23T11:22:14.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.897+0000 7f2bed400640 1 -- 172.21.15.62:0/86180975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be8066bb0 con 0x7f2bcc0047c0 2024-03-23T11:22:14.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.898+0000 7f2bdfe00640 1 -- 172.21.15.62:0/86180975 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2bd8013400 con 0x7f2bcc0047c0 2024-03-23T11:22:14.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.899+0000 7f2bdfe00640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bc806a7a0 0x7f2bc806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:14.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.899+0000 7f2bdfe00640 1 -- 172.21.15.62:0/86180975 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2bd80892c0 con 0x7f2bcc0047c0 2024-03-23T11:22:14.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.900+0000 7f2be7400640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bc806a7a0 0x7f2bc806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:14.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.901+0000 7f2be7400640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bc806a7a0 0x7f2bc806cc60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4034000 tx=0x7f2bd4009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:14.904+0000 7f2bdfe00640 1 -- 172.21.15.62:0/86180975 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2bd8054600 con 0x7f2bcc0047c0 2024-03-23T11:22:15.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.103+0000 7f2bed400640 1 -- 172.21.15.62:0/86180975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2be8069f20 con 0x7f2bcc0047c0 2024-03-23T11:22:15.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.104+0000 7f2bdfe00640 1 -- 172.21.15.62:0/86180975 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2bd8057c20 con 0x7f2bcc0047c0 2024-03-23T11:22:15.101 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:22:15.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 -- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bc806a7a0 msgr2=0x7f2bc806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:15.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bc806a7a0 0x7f2bc806cc60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4034000 tx=0x7f2bd4009030 comp rx=0 tx=0).stop 2024-03-23T11:22:15.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 -- 172.21.15.62:0/86180975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 msgr2=0x7f2be80c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:15.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 0x7f2be80c4480 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8011510 tx=0x7f2bd80119d0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 -- 172.21.15.62:0/86180975 shutdown_connections 2024-03-23T11:22:15.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bc806a7a0 0x7f2bc806cc60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2be80c8f80 0x7f2be81495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2be80c49c0 0x7f2be80c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 --2- 172.21.15.62:0/86180975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bcc0047c0 0x7f2be80c4480 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 -- 172.21.15.62:0/86180975 >> 172.21.15.62:0/86180975 conn(0x7f2be8066050 msgr2=0x7f2be806d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:15.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 -- 172.21.15.62:0/86180975 shutdown_connections 2024-03-23T11:22:15.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.107+0000 7f2bed400640 1 -- 172.21.15.62:0/86180975 wait complete. 2024-03-23T11:22:15.124 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:22:15.251 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:22:15.259 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:22:15.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.350+0000 7fafad400640 1 Processor -- start 2024-03-23T11:22:15.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.350+0000 7fafad400640 1 -- start start 2024-03-23T11:22:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.351+0000 7fafad400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fafa815c8e0 0x7fafa80cc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafad400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fafa8065c40 0x7fafa80c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafad400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa80660a0 con 0x7fafa806c760 2024-03-23T11:22:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafad400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa815ccc0 con 0x7fafa815c8e0 2024-03-23T11:22:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafad400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa80ccfc0 con 0x7fafa8065c40 2024-03-23T11:22:15.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafa7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fafa815c8e0 0x7fafa80cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafa7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fafa815c8e0 0x7fafa80cc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48648/0 (socket says 172.21.15.62:48648) 2024-03-23T11:22:15.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafa7e00640 1 -- 172.21.15.62:0/2923745326 learned_addr learned my addr 172.21.15.62:0/2923745326 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:15.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafa7400640 1 --2- 172.21.15.62:0/2923745326 >> v2:172.21.15.120:3300/0 conn(0x7fafa8065c40 0x7fafa80c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafa7e00640 1 -- 172.21.15.62:0/2923745326 >> v2:172.21.15.120:3300/0 conn(0x7fafa8065c40 msgr2=0x7fafa80c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:15.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafa7e00640 1 --2- 172.21.15.62:0/2923745326 >> v2:172.21.15.120:3300/0 conn(0x7fafa8065c40 0x7fafa80c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafa7e00640 1 -- 172.21.15.62:0/2923745326 >> v1:172.21.15.120:6789/0 conn(0x7fafa806c760 legacy=0x7fafa806aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:15.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.352+0000 7fafa7e00640 1 -- 172.21.15.62:0/2923745326 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa8069c60 con 0x7fafa815c8e0 2024-03-23T11:22:15.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.354+0000 7fafa7e00640 1 --2- 172.21.15.62:0/2923745326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa815c8e0 0x7fafa80cc930 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fafa806ea70 tx=0x7faf9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba04b4212d8c022f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:15.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.354+0000 7fafa6a00640 1 -- 172.21.15.62:0/2923745326 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98031870 con 0x7fafa815c8e0 2024-03-23T11:22:15.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.354+0000 7fafa6a00640 1 -- 172.21.15.62:0/2923745326 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf980319d0 con 0x7fafa815c8e0 2024-03-23T11:22:15.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.354+0000 7fafa6a00640 1 -- 172.21.15.62:0/2923745326 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98031d60 con 0x7fafa815c8e0 2024-03-23T11:22:15.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.355+0000 7fafad400640 1 -- 172.21.15.62:0/2923745326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa815c8e0 msgr2=0x7fafa80cc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:15.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.355+0000 7fafad400640 1 --2- 172.21.15.62:0/2923745326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa815c8e0 0x7fafa80cc930 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fafa806ea70 tx=0x7faf9802b910 comp rx=0 tx=0).stop 2024-03-23T11:22:15.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.355+0000 7fafad400640 1 -- 172.21.15.62:0/2923745326 shutdown_connections 2024-03-23T11:22:15.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.355+0000 7fafad400640 1 --2- 172.21.15.62:0/2923745326 >> v2:172.21.15.120:3300/0 conn(0x7fafa8065c40 0x7fafa80c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.355+0000 7fafad400640 1 --2- 172.21.15.62:0/2923745326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa815c8e0 0x7fafa80cc930 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.355+0000 7fafad400640 1 -- 172.21.15.62:0/2923745326 >> 172.21.15.62:0/2923745326 conn(0x7fafa80cb7e0 msgr2=0x7fafa80cbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:15.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.355+0000 7fafad400640 1 -- 172.21.15.62:0/2923745326 shutdown_connections 2024-03-23T11:22:15.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.355+0000 7fafad400640 1 -- 172.21.15.62:0/2923745326 wait complete. 2024-03-23T11:22:15.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.356+0000 7fafad400640 1 Processor -- start 2024-03-23T11:22:15.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.357+0000 7fafad400640 1 -- start start 2024-03-23T11:22:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.357+0000 7fafad400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafa8065c40 0x7fafa8156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafad400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa806c760 0x7fafa80b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafad400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa815c8e0 0x7fafa80b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafad400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa80b2910 con 0x7fafa806c760 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafad400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa80b4740 con 0x7fafa815c8e0 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafad400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa80ac1f0 con 0x7fafa8065c40 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafa7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa806c760 0x7fafa80b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafa7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa806c760 0x7fafa80b3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48662/0 (socket says 172.21.15.62:48662) 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafa7400640 1 -- 172.21.15.62:0/1048616117 learned_addr learned my addr 172.21.15.62:0/1048616117 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafaca00640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa815c8e0 0x7fafa80b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafa7e00640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafa8065c40 0x7fafa8156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafa7400640 1 -- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafa8065c40 msgr2=0x7fafa8156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafa7400640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafa8065c40 0x7fafa8156e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafa7400640 1 -- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa815c8e0 msgr2=0x7fafa80b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafa7400640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa815c8e0 0x7fafa80b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafa7400640 1 -- 172.21.15.62:0/1048616117 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf980094d0 con 0x7fafa806c760 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafaca00640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa815c8e0 0x7fafa80b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:22:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.358+0000 7fafa7e00640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafa8065c40 0x7fafa8156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:15.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.359+0000 7fafa7400640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa806c760 0x7fafa80b3cd0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7faf94007ef0 tx=0x7faf9400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:15.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.359+0000 7faf9fe00640 1 -- 172.21.15.62:0/1048616117 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf94014620 con 0x7fafa806c760 2024-03-23T11:22:15.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.359+0000 7faf9fe00640 1 -- 172.21.15.62:0/1048616117 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf94014780 con 0x7fafa806c760 2024-03-23T11:22:15.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.359+0000 7faf9fe00640 1 -- 172.21.15.62:0/1048616117 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf94014b30 con 0x7fafa806c760 2024-03-23T11:22:15.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.359+0000 7fafad400640 1 -- 172.21.15.62:0/1048616117 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa80ac4d0 con 0x7fafa806c760 2024-03-23T11:22:15.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.359+0000 7fafad400640 1 -- 172.21.15.62:0/1048616117 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa80ac860 con 0x7fafa806c760 2024-03-23T11:22:15.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.360+0000 7fafad400640 1 -- 172.21.15.62:0/1048616117 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa80672c0 con 0x7fafa806c760 2024-03-23T11:22:15.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.361+0000 7faf9fe00640 1 -- 172.21.15.62:0/1048616117 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7faf94014c90 con 0x7fafa806c760 2024-03-23T11:22:15.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.362+0000 7faf9fe00640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faf8806a7a0 0x7faf8806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.362+0000 7faf9fe00640 1 -- 172.21.15.62:0/1048616117 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faf940897e0 con 0x7fafa806c760 2024-03-23T11:22:15.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.363+0000 7fafa7e00640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faf8806a7a0 0x7faf8806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.364+0000 7fafa7e00640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faf8806a7a0 0x7faf8806cc60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7faf98036000 tx=0x7faf9803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:15.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.367+0000 7faf9fe00640 1 -- 172.21.15.62:0/1048616117 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf94054bb0 con 0x7fafa806c760 2024-03-23T11:22:15.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.567+0000 7fafad400640 1 -- 172.21.15.62:0/1048616117 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fafa8065a20 con 0x7fafa806c760 2024-03-23T11:22:15.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.568+0000 7faf9fe00640 1 -- 172.21.15.62:0/1048616117 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7faf940581d0 con 0x7fafa806c760 2024-03-23T11:22:15.565 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:22:15.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.570+0000 7fafad400640 1 -- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faf8806a7a0 msgr2=0x7faf8806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:15.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.570+0000 7fafad400640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faf8806a7a0 0x7faf8806cc60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7faf98036000 tx=0x7faf9803c040 comp rx=0 tx=0).stop 2024-03-23T11:22:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.570+0000 7fafad400640 1 -- 172.21.15.62:0/1048616117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa806c760 msgr2=0x7fafa80b3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.570+0000 7fafad400640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa806c760 0x7fafa80b3cd0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7faf94007ef0 tx=0x7faf9400e4b0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.571+0000 7fafad400640 1 -- 172.21.15.62:0/1048616117 shutdown_connections 2024-03-23T11:22:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.571+0000 7fafad400640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faf8806a7a0 0x7faf8806cc60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.571+0000 7fafad400640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafa806c760 0x7fafa80b3cd0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.571+0000 7fafad400640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa815c8e0 0x7fafa80b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.571+0000 7fafad400640 1 --2- 172.21.15.62:0/1048616117 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafa8065c40 0x7fafa8156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.571+0000 7fafad400640 1 -- 172.21.15.62:0/1048616117 >> 172.21.15.62:0/1048616117 conn(0x7fafa80cb7e0 msgr2=0x7fafa815bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.571+0000 7fafad400640 1 -- 172.21.15.62:0/1048616117 shutdown_connections 2024-03-23T11:22:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.571+0000 7fafad400640 1 -- 172.21.15.62:0/1048616117 wait complete. 2024-03-23T11:22:15.587 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:22:15.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.803+0000 7f6647e00640 1 Processor -- start 2024-03-23T11:22:15.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.803+0000 7f6647e00640 1 -- start start 2024-03-23T11:22:15.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.805+0000 7f6647e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f664806cc60 0x7f66480cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.805+0000 7f6647e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66480ce420 0x7f66480b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.805+0000 7f6647e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66480ce890 con 0x7f6648159490 2024-03-23T11:22:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.805+0000 7f6647e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66480ced90 con 0x7f664806cc60 2024-03-23T11:22:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.805+0000 7f6647e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664806e3e0 con 0x7f66480ce420 2024-03-23T11:22:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.806+0000 7f6647400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66480ce420 0x7f66480b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.806+0000 7f6647400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66480ce420 0x7f66480b26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:44910/0 (socket says 172.21.15.62:44910) 2024-03-23T11:22:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.806+0000 7f6647400640 1 -- 172.21.15.62:0/3027162954 learned_addr learned my addr 172.21.15.62:0/3027162954 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.806+0000 7f663fe00640 1 --2- 172.21.15.62:0/3027162954 >> v2:172.21.15.120:3300/0 conn(0x7f664806cc60 0x7f66480cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.806+0000 7f663fe00640 1 -- 172.21.15.62:0/3027162954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66480ce420 msgr2=0x7f66480b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.807+0000 7f663fe00640 1 --2- 172.21.15.62:0/3027162954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66480ce420 0x7f66480b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.807+0000 7f663fe00640 1 -- 172.21.15.62:0/3027162954 >> v1:172.21.15.120:6789/0 conn(0x7f6648159490 legacy=0x7f664815da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.807+0000 7f663fe00640 1 -- 172.21.15.62:0/3027162954 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664806c1c0 con 0x7f664806cc60 2024-03-23T11:22:15.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.807+0000 7f6647400640 1 --2- 172.21.15.62:0/3027162954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66480ce420 0x7f66480b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:22:15.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.808+0000 7f663fe00640 1 --2- 172.21.15.62:0/3027162954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664806cc60 0x7f66480cdee0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f66300097a0 tx=0x7f663002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f353c5a5345e9e12 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:15.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.808+0000 7f6646000640 1 -- 172.21.15.62:0/3027162954 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630031780 con 0x7f664806cc60 2024-03-23T11:22:15.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.808+0000 7f6646000640 1 -- 172.21.15.62:0/3027162954 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66300318e0 con 0x7f664806cc60 2024-03-23T11:22:15.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.808+0000 7f6646000640 1 -- 172.21.15.62:0/3027162954 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630031c70 con 0x7f664806cc60 2024-03-23T11:22:15.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.809+0000 7f6647e00640 1 -- 172.21.15.62:0/3027162954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664806cc60 msgr2=0x7f66480cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:15.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.809+0000 7f6647e00640 1 --2- 172.21.15.62:0/3027162954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664806cc60 0x7f66480cdee0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f66300097a0 tx=0x7f663002b820 comp rx=0 tx=0).stop 2024-03-23T11:22:15.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.809+0000 7f6647e00640 1 -- 172.21.15.62:0/3027162954 shutdown_connections 2024-03-23T11:22:15.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.809+0000 7f6647e00640 1 --2- 172.21.15.62:0/3027162954 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66480ce420 0x7f66480b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.809+0000 7f6647e00640 1 --2- 172.21.15.62:0/3027162954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664806cc60 0x7f66480cdee0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.809+0000 7f6647e00640 1 -- 172.21.15.62:0/3027162954 >> 172.21.15.62:0/3027162954 conn(0x7f6648066050 msgr2=0x7f6648069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:15.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.809+0000 7f6647e00640 1 -- 172.21.15.62:0/3027162954 shutdown_connections 2024-03-23T11:22:15.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.810+0000 7f6647e00640 1 -- 172.21.15.62:0/3027162954 wait complete. 2024-03-23T11:22:15.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.812+0000 7f6647e00640 1 Processor -- start 2024-03-23T11:22:15.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.812+0000 7f6647e00640 1 -- start start 2024-03-23T11:22:15.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.813+0000 7f6647e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f664806cc60 0x7f66480ac250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.813+0000 7f6647e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480ce420 0x7f664814ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.813+0000 7f6647e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6648159490 0x7f66481f5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.813+0000 7f6647e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66480caa30 con 0x7f66480ce420 2024-03-23T11:22:15.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.813+0000 7f6647e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66480ac790 con 0x7f6648159490 2024-03-23T11:22:15.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.813+0000 7f6647e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664814b790 con 0x7f664806cc60 2024-03-23T11:22:15.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.814+0000 7f663fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480ce420 0x7f664814ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.814+0000 7f663fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480ce420 0x7f664814ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48670/0 (socket says 172.21.15.62:48670) 2024-03-23T11:22:15.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.814+0000 7f663fe00640 1 -- 172.21.15.62:0/1846747293 learned_addr learned my addr 172.21.15.62:0/1846747293 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:15.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.814+0000 7f6646a00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f664806cc60 0x7f66480ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.814+0000 7f6647400640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6648159490 0x7f66481f5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.814+0000 7f663fe00640 1 -- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f664806cc60 msgr2=0x7f66480ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:15.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.814+0000 7f663fe00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f664806cc60 0x7f66480ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.814+0000 7f663fe00640 1 -- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6648159490 msgr2=0x7f66481f5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:15.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.814+0000 7f663fe00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6648159490 0x7f66481f5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:15.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.814+0000 7f663fe00640 1 -- 172.21.15.62:0/1846747293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640009b30 con 0x7f66480ce420 2024-03-23T11:22:15.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.814+0000 7f6646a00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f664806cc60 0x7f66480ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:22:15.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.815+0000 7f663fe00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480ce420 0x7f664814ae60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f66300098c0 tx=0x7f66300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:15.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.815+0000 7f663ea00640 1 -- 172.21.15.62:0/1846747293 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630032820 con 0x7f66480ce420 2024-03-23T11:22:15.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.815+0000 7f663ea00640 1 -- 172.21.15.62:0/1846747293 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66300329a0 con 0x7f66480ce420 2024-03-23T11:22:15.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.815+0000 7f6647e00640 1 -- 172.21.15.62:0/1846747293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6630009400 con 0x7f66480ce420 2024-03-23T11:22:15.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.815+0000 7f663ea00640 1 -- 172.21.15.62:0/1846747293 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630032d50 con 0x7f66480ce420 2024-03-23T11:22:15.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.815+0000 7f6647e00640 1 -- 172.21.15.62:0/1846747293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66480acca0 con 0x7f66480ce420 2024-03-23T11:22:15.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.817+0000 7f6647e00640 1 -- 172.21.15.62:0/1846747293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6648066bb0 con 0x7f66480ce420 2024-03-23T11:22:15.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.817+0000 7f663ea00640 1 -- 172.21.15.62:0/1846747293 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f663004e440 con 0x7f66480ce420 2024-03-23T11:22:15.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.818+0000 7f663ea00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 0x7f662406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:15.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.818+0000 7f663ea00640 1 -- 172.21.15.62:0/1846747293 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f66300aae20 con 0x7f66480ce420 2024-03-23T11:22:15.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.819+0000 7f6646a00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 0x7f662406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:15.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.820+0000 7f6646a00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 0x7f662406cc60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6638007fd0 tx=0x7f6638009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:15.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:15.823+0000 7f663ea00640 1 -- 172.21.15.62:0/1846747293 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6630076340 con 0x7f66480ce420 2024-03-23T11:22:16.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.061+0000 7f6647e00640 1 -- 172.21.15.62:0/1846747293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f664806d560 con 0x7f66480ce420 2024-03-23T11:22:16.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.063+0000 7f663ea00640 1 -- 172.21.15.62:0/1846747293 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f6630079960 con 0x7f66480ce420 2024-03-23T11:22:16.060 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:16.061 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3787987198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3787987198},{"type":"v1","addr":"172.21.15.120:6835","nonce":3787987198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2749498408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2749498408},{"type":"v1","addr":"172.21.15.120:6837","nonce":2749498408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2325988870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2325988870},{"type":"v1","addr":"172.21.15.62:6833","nonce":2325988870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:21:25.687136+0000","modified":"2024-03-23T11:21:28.704192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6835/1159419089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1159419089},{"type":"v1","addr":"172.21.15.62:6835","nonce":1159419089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T11:22:16.061 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-03-23T11:22:16.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.065+0000 7f6647e00640 1 -- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 msgr2=0x7f662406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:16.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.065+0000 7f6647e00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 0x7f662406cc60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6638007fd0 tx=0x7f6638009040 comp rx=0 tx=0).stop 2024-03-23T11:22:16.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.065+0000 7f6647e00640 1 -- 172.21.15.62:0/1846747293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480ce420 msgr2=0x7f664814ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:16.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.065+0000 7f6647e00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480ce420 0x7f664814ae60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f66300098c0 tx=0x7f66300046c0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.065+0000 7f6647e00640 1 -- 172.21.15.62:0/1846747293 shutdown_connections 2024-03-23T11:22:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.065+0000 7f6647e00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f662406a7a0 0x7f662406cc60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.065+0000 7f6647e00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66480ce420 0x7f664814ae60 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.065+0000 7f6647e00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6648159490 0x7f66481f5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.065+0000 7f6647e00640 1 --2- 172.21.15.62:0/1846747293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f664806cc60 0x7f66480ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.066+0000 7f6647e00640 1 -- 172.21.15.62:0/1846747293 >> 172.21.15.62:0/1846747293 conn(0x7f6648066050 msgr2=0x7f66480b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.066+0000 7f6647e00640 1 -- 172.21.15.62:0/1846747293 shutdown_connections 2024-03-23T11:22:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.066+0000 7f6647e00640 1 -- 172.21.15.62:0/1846747293 wait complete. 2024-03-23T11:22:16.083 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:22:16.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.309+0000 7fd1d6600640 1 Processor -- start 2024-03-23T11:22:16.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.309+0000 7fd1d6600640 1 -- start start 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.310+0000 7fd1d6600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd1d006c300 0x7fd1d00ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.311+0000 7fd1d6600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd1d006c6e0 0x7fd1d006cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.311+0000 7fd1d6600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d00cbc60 con 0x7fd1d00cafa0 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.311+0000 7fd1d6600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d00cbf40 con 0x7fd1d006c300 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.311+0000 7fd1d6600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d00cc220 con 0x7fd1d006c6e0 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.311+0000 7fd1cf400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd1d006c300 0x7fd1d00ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.311+0000 7fd1cf400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd1d006c300 0x7fd1d00ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48676/0 (socket says 172.21.15.62:48676) 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.311+0000 7fd1cf400640 1 -- 172.21.15.62:0/3579918002 learned_addr learned my addr 172.21.15.62:0/3579918002 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.311+0000 7fd1cf400640 1 -- 172.21.15.62:0/3579918002 >> v2:172.21.15.120:3300/0 conn(0x7fd1d006c6e0 msgr2=0x7fd1d006cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.311+0000 7fd1cf400640 1 --2- 172.21.15.62:0/3579918002 >> v2:172.21.15.120:3300/0 conn(0x7fd1d006c6e0 0x7fd1d006cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.311+0000 7fd1cf400640 1 -- 172.21.15.62:0/3579918002 >> v1:172.21.15.120:6789/0 conn(0x7fd1d00cafa0 legacy=0x7fd1d00cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:16.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.311+0000 7fd1cf400640 1 -- 172.21.15.62:0/3579918002 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d00cc500 con 0x7fd1d006c300 2024-03-23T11:22:16.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.312+0000 7fd1cf400640 1 --2- 172.21.15.62:0/3579918002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1d006c300 0x7fd1d00ccd00 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0070270 tx=0x7fd1c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424a46252b5e804a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:16.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.313+0000 7fd1cea00640 1 -- 172.21.15.62:0/3579918002 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0031870 con 0x7fd1d006c300 2024-03-23T11:22:16.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.313+0000 7fd1cea00640 1 -- 172.21.15.62:0/3579918002 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c00319d0 con 0x7fd1d006c300 2024-03-23T11:22:16.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.313+0000 7fd1cea00640 1 -- 172.21.15.62:0/3579918002 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0031d60 con 0x7fd1d006c300 2024-03-23T11:22:16.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.313+0000 7fd1d6600640 1 -- 172.21.15.62:0/3579918002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1d006c300 msgr2=0x7fd1d00ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:16.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.313+0000 7fd1d6600640 1 --2- 172.21.15.62:0/3579918002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1d006c300 0x7fd1d00ccd00 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0070270 tx=0x7fd1c002b910 comp rx=0 tx=0).stop 2024-03-23T11:22:16.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.314+0000 7fd1d6600640 1 -- 172.21.15.62:0/3579918002 shutdown_connections 2024-03-23T11:22:16.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.314+0000 7fd1d6600640 1 --2- 172.21.15.62:0/3579918002 >> v2:172.21.15.120:3300/0 conn(0x7fd1d006c6e0 0x7fd1d006cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.314+0000 7fd1d6600640 1 --2- 172.21.15.62:0/3579918002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1d006c300 0x7fd1d00ccd00 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.314+0000 7fd1d6600640 1 -- 172.21.15.62:0/3579918002 >> 172.21.15.62:0/3579918002 conn(0x7fd1d00b02e0 msgr2=0x7fd1d00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:16.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.314+0000 7fd1d6600640 1 -- 172.21.15.62:0/3579918002 shutdown_connections 2024-03-23T11:22:16.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.314+0000 7fd1d6600640 1 -- 172.21.15.62:0/3579918002 wait complete. 2024-03-23T11:22:16.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.316+0000 7fd1d6600640 1 Processor -- start 2024-03-23T11:22:16.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.316+0000 7fd1d6600640 1 -- start start 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1d6600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d006c300 0x7fd1d0156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1d6600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1d006c6e0 0x7fd1d013d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1d6600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1d00cafa0 0x7fd1d015c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1d6600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d013ebf0 con 0x7fd1d00cafa0 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1d6600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d013dfe0 con 0x7fd1d006c300 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1d6600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d013e2c0 con 0x7fd1d006c6e0 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1cfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1d00cafa0 0x7fd1d015c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1cfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1d00cafa0 0x7fd1d015c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48688/0 (socket says 172.21.15.62:48688) 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1cfe00640 1 -- 172.21.15.62:0/45851068 learned_addr learned my addr 172.21.15.62:0/45851068 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1cf400640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d006c300 0x7fd1d0156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1c7e00640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1d006c6e0 0x7fd1d013d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:16.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.317+0000 7fd1cfe00640 1 -- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1d006c6e0 msgr2=0x7fd1d013d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1cfe00640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1d006c6e0 0x7fd1d013d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1cfe00640 1 -- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d006c300 msgr2=0x7fd1d0156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1cfe00640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d006c300 0x7fd1d0156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1cfe00640 1 -- 172.21.15.62:0/45851068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c00094d0 con 0x7fd1d00cafa0 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1cf400640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d006c300 0x7fd1d0156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1c7e00640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1d006c6e0 0x7fd1d013d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1cfe00640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1d00cafa0 0x7fd1d015c790 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800b970 tx=0x7fd1c800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1c7400640 1 -- 172.21.15.62:0/45851068 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c800cb40 con 0x7fd1d00cafa0 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1c7400640 1 -- 172.21.15.62:0/45851068 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c800ccc0 con 0x7fd1d00cafa0 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1c7400640 1 -- 172.21.15.62:0/45851068 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8014680 con 0x7fd1d00cafa0 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1d6600640 1 -- 172.21.15.62:0/45851068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d013e5a0 con 0x7fd1d00cafa0 2024-03-23T11:22:16.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.318+0000 7fd1d6600640 1 -- 172.21.15.62:0/45851068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d01565b0 con 0x7fd1d00cafa0 2024-03-23T11:22:16.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.319+0000 7fd1d6600640 1 -- 172.21.15.62:0/45851068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d013eda0 con 0x7fd1d00cafa0 2024-03-23T11:22:16.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.320+0000 7fd1c7400640 1 -- 172.21.15.62:0/45851068 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd1c801d3b0 con 0x7fd1d00cafa0 2024-03-23T11:22:16.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.321+0000 7fd1c7400640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1ac06a7d0 0x7fd1ac06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.321+0000 7fd1c7400640 1 -- 172.21.15.62:0/45851068 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd1c8088d00 con 0x7fd1d00cafa0 2024-03-23T11:22:16.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.322+0000 7fd1cf400640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1ac06a7d0 0x7fd1ac06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:16.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.323+0000 7fd1cf400640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1ac06a7d0 0x7fd1ac06cc90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0036000 tx=0x7fd1c003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.326+0000 7fd1c7400640 1 -- 172.21.15.62:0/45851068 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1d013eda0 con 0x7fd1d00cafa0 2024-03-23T11:22:16.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.524+0000 7fd1d6600640 1 -- 172.21.15.62:0/45851068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1d0066c90 con 0x7fd1d00cafa0 2024-03-23T11:22:16.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.526+0000 7fd1c7400640 1 -- 172.21.15.62:0/45851068 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16197 (secure 0 0 0) 0x7fd1c8019020 con 0x7fd1d00cafa0 2024-03-23T11:22:16.524 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:16.524 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":50,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:21:43.375126+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:21:23.756222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-23T11:21:24.715577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.2a","mappings":[{"from":0,"to":5}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 -- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1ac06a7d0 msgr2=0x7fd1ac06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1ac06a7d0 0x7fd1ac06cc90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0036000 tx=0x7fd1c003c040 comp rx=0 tx=0).stop 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 -- 172.21.15.62:0/45851068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1d00cafa0 msgr2=0x7fd1d015c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1d00cafa0 0x7fd1d015c790 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800b970 tx=0x7fd1c800be30 comp rx=0 tx=0).stop 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 -- 172.21.15.62:0/45851068 shutdown_connections 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1ac06a7d0 0x7fd1ac06cc90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1d00cafa0 0x7fd1d015c790 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1d006c6e0 0x7fd1d013d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 --2- 172.21.15.62:0/45851068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d006c300 0x7fd1d0156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 -- 172.21.15.62:0/45851068 >> 172.21.15.62:0/45851068 conn(0x7fd1d00b02e0 msgr2=0x7fd1d00cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 -- 172.21.15.62:0/45851068 shutdown_connections 2024-03-23T11:22:16.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.529+0000 7fd1d6600640 1 -- 172.21.15.62:0/45851068 wait complete. 2024-03-23T11:22:16.546 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T11:22:16.547 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:22:16.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.771+0000 7f4a49000640 1 Processor -- start 2024-03-23T11:22:16.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.771+0000 7f4a49000640 1 -- start start 2024-03-23T11:22:16.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.772+0000 7f4a49000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a440cd7c0 0x7f4a440b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.772+0000 7f4a49000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a4406dca0 0x7f4a4406c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.772+0000 7f4a49000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4a440cdba0 0x7f4a440cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.772+0000 7f4a49000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a440cad20 con 0x7f4a440cdba0 2024-03-23T11:22:16.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.772+0000 7f4a49000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a4406e130 con 0x7f4a4406dca0 2024-03-23T11:22:16.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.772+0000 7f4a49000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4415a8a0 con 0x7f4a440cd7c0 2024-03-23T11:22:16.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.773+0000 7f4a42800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4a440cdba0 0x7f4a440cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:16.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.773+0000 7f4a42800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4a440cdba0 0x7f4a440cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48690/0 (socket says 172.21.15.62:48690) 2024-03-23T11:22:16.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.773+0000 7f4a42800640 1 -- 172.21.15.62:0/4006780335 learned_addr learned my addr 172.21.15.62:0/4006780335 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:16.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.773+0000 7f4a41400640 1 --2- 172.21.15.62:0/4006780335 >> v2:172.21.15.120:3300/0 conn(0x7f4a4406dca0 0x7f4a4406c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:16.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.773+0000 7f4a41e00640 1 --2- 172.21.15.62:0/4006780335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a440cd7c0 0x7f4a440b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:16.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.774+0000 7f4a42800640 1 -- 172.21.15.62:0/4006780335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a440cd7c0 msgr2=0x7f4a440b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:16.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.774+0000 7f4a42800640 1 --2- 172.21.15.62:0/4006780335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a440cd7c0 0x7f4a440b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.774+0000 7f4a42800640 1 -- 172.21.15.62:0/4006780335 >> v2:172.21.15.120:3300/0 conn(0x7f4a4406dca0 msgr2=0x7f4a4406c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:16.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.774+0000 7f4a42800640 1 --2- 172.21.15.62:0/4006780335 >> v2:172.21.15.120:3300/0 conn(0x7f4a4406dca0 0x7f4a4406c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.774+0000 7f4a42800640 1 -- 172.21.15.62:0/4006780335 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4415ab80 con 0x7f4a440cdba0 2024-03-23T11:22:16.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.774+0000 7f4a41e00640 1 --2- 172.21.15.62:0/4006780335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a440cd7c0 0x7f4a440b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:22:16.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.775+0000 7f4a42800640 1 --2- 172.21.15.62:0/4006780335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a440cdba0 0x7f4a440cb490 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4a440738b0 tx=0x7f4a3c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b7d0ed6d0f05c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:16.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.776+0000 7f4a40a00640 1 -- 172.21.15.62:0/4006780335 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c02c9e0 con 0x7f4a440cdba0 2024-03-23T11:22:16.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.776+0000 7f4a40a00640 1 -- 172.21.15.62:0/4006780335 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3c02cb40 con 0x7f4a440cdba0 2024-03-23T11:22:16.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.776+0000 7f4a40a00640 1 -- 172.21.15.62:0/4006780335 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c0343f0 con 0x7f4a440cdba0 2024-03-23T11:22:16.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.776+0000 7f4a49000640 1 -- 172.21.15.62:0/4006780335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a440cdba0 msgr2=0x7f4a440cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:16.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.776+0000 7f4a49000640 1 --2- 172.21.15.62:0/4006780335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a440cdba0 0x7f4a440cb490 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4a440738b0 tx=0x7f4a3c02be40 comp rx=0 tx=0).stop 2024-03-23T11:22:16.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.777+0000 7f4a49000640 1 -- 172.21.15.62:0/4006780335 shutdown_connections 2024-03-23T11:22:16.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.777+0000 7f4a49000640 1 --2- 172.21.15.62:0/4006780335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a440cdba0 0x7f4a440cb490 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.777+0000 7f4a49000640 1 --2- 172.21.15.62:0/4006780335 >> v2:172.21.15.120:3300/0 conn(0x7f4a4406dca0 0x7f4a4406c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.777+0000 7f4a49000640 1 --2- 172.21.15.62:0/4006780335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a440cd7c0 0x7f4a440b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.777+0000 7f4a49000640 1 -- 172.21.15.62:0/4006780335 >> 172.21.15.62:0/4006780335 conn(0x7f4a440b02e0 msgr2=0x7f4a440b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:16.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.777+0000 7f4a49000640 1 -- 172.21.15.62:0/4006780335 shutdown_connections 2024-03-23T11:22:16.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.777+0000 7f4a49000640 1 -- 172.21.15.62:0/4006780335 wait complete. 2024-03-23T11:22:16.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.778+0000 7f4a49000640 1 Processor -- start 2024-03-23T11:22:16.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.779+0000 7f4a49000640 1 -- start start 2024-03-23T11:22:16.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.779+0000 7f4a49000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a440cd7c0 0x7f4a44148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.779+0000 7f4a49000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a440cdba0 0x7f4a440af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a49000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a440afd10 0x7f4a44149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a49000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440ae730 con 0x7f4a440cdba0 2024-03-23T11:22:16.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a49000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440aea10 con 0x7f4a440cd7c0 2024-03-23T11:22:16.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a49000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440aecf0 con 0x7f4a440afd10 2024-03-23T11:22:16.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a41400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a440cdba0 0x7f4a440af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:16.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a41400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a440cdba0 0x7f4a440af7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48698/0 (socket says 172.21.15.62:48698) 2024-03-23T11:22:16.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a41400640 1 -- 172.21.15.62:0/2718682647 learned_addr learned my addr 172.21.15.62:0/2718682647 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a41e00640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a440cd7c0 0x7f4a44148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a41400640 1 -- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a440afd10 msgr2=0x7f4a44149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a41400640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a440afd10 0x7f4a44149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a41400640 1 -- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a440cd7c0 msgr2=0x7f4a44148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a41400640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a440cd7c0 0x7f4a44148d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.780+0000 7f4a41400640 1 -- 172.21.15.62:0/2718682647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c009b30 con 0x7f4a440cdba0 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.781+0000 7f4a41400640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a440cdba0 0x7f4a440af7d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00e870 tx=0x7f4a2c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.781+0000 7f4a3a000640 1 -- 172.21.15.62:0/2718682647 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c00ee60 con 0x7f4a440cdba0 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.781+0000 7f4a3a000640 1 -- 172.21.15.62:0/2718682647 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c00c940 con 0x7f4a440cdba0 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.781+0000 7f4a3a000640 1 -- 172.21.15.62:0/2718682647 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c00ccf0 con 0x7f4a440cdba0 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.781+0000 7f4a49000640 1 -- 172.21.15.62:0/2718682647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a440aefd0 con 0x7f4a440cdba0 2024-03-23T11:22:16.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.781+0000 7f4a49000640 1 -- 172.21.15.62:0/2718682647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a441492d0 con 0x7f4a440cdba0 2024-03-23T11:22:16.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.782+0000 7f4a49000640 1 -- 172.21.15.62:0/2718682647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a44066c90 con 0x7f4a440cdba0 2024-03-23T11:22:16.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.784+0000 7f4a3a000640 1 -- 172.21.15.62:0/2718682647 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4a2c013470 con 0x7f4a440cdba0 2024-03-23T11:22:16.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.784+0000 7f4a3a000640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4a2006a7a0 0x7f4a2006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:16.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.785+0000 7f4a3a000640 1 -- 172.21.15.62:0/2718682647 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4a2c088dd0 con 0x7f4a440cdba0 2024-03-23T11:22:16.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.785+0000 7f4a41e00640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4a2006a7a0 0x7f4a2006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:16.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.786+0000 7f4a41e00640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4a2006a7a0 0x7f4a2006cc60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400b540 tx=0x7f4a3400d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:16.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:16.789+0000 7f4a3a000640 1 -- 172.21.15.62:0/2718682647 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4a2c054240 con 0x7f4a440cdba0 2024-03-23T11:22:17.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.026+0000 7f4a49000640 1 -- 172.21.15.62:0/2718682647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a440ad9e0 con 0x7f4a440cdba0 2024-03-23T11:22:17.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.027+0000 7f4a3a000640 1 -- 172.21.15.62:0/2718682647 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f4a2c057860 con 0x7f4a440cdba0 2024-03-23T11:22:17.025 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:17.025 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3787987198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3787987198},{"type":"v1","addr":"172.21.15.120:6835","nonce":3787987198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2749498408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2749498408},{"type":"v1","addr":"172.21.15.120:6837","nonce":2749498408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2325988870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2325988870},{"type":"v1","addr":"172.21.15.62:6833","nonce":2325988870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:21:25.687136+0000","modified":"2024-03-23T11:21:28.704192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6835/1159419089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1159419089},{"type":"v1","addr":"172.21.15.62:6835","nonce":1159419089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T11:22:17.025 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-03-23T11:22:17.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 -- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4a2006a7a0 msgr2=0x7f4a2006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4a2006a7a0 0x7f4a2006cc60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400b540 tx=0x7f4a3400d040 comp rx=0 tx=0).stop 2024-03-23T11:22:17.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 -- 172.21.15.62:0/2718682647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a440cdba0 msgr2=0x7f4a440af7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a440cdba0 0x7f4a440af7d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00e870 tx=0x7f4a2c00ed30 comp rx=0 tx=0).stop 2024-03-23T11:22:17.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 -- 172.21.15.62:0/2718682647 shutdown_connections 2024-03-23T11:22:17.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4a2006a7a0 0x7f4a2006cc60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a440cdba0 0x7f4a440af7d0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a440afd10 0x7f4a44149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 --2- 172.21.15.62:0/2718682647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a440cd7c0 0x7f4a44148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 -- 172.21.15.62:0/2718682647 >> 172.21.15.62:0/2718682647 conn(0x7f4a440b02e0 msgr2=0x7f4a4406d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:17.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 -- 172.21.15.62:0/2718682647 shutdown_connections 2024-03-23T11:22:17.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.030+0000 7f4a49000640 1 -- 172.21.15.62:0/2718682647 wait complete. 2024-03-23T11:22:17.047 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:22:17.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.273+0000 7fb34b200640 1 Processor -- start 2024-03-23T11:22:17.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.273+0000 7fb34b200640 1 -- start start 2024-03-23T11:22:17.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.275+0000 7fb34b200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb34c15c830 0x7fb34c0ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.275+0000 7fb34b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb34c0ce250 0x7fb34c0ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.275+0000 7fb34b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34c1608d0 con 0x7fb34c160310 2024-03-23T11:22:17.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.275+0000 7fb34b200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb34c0addb0 con 0x7fb34c15c830 2024-03-23T11:22:17.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.275+0000 7fb34b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb34c15cc10 con 0x7fb34c0ce250 2024-03-23T11:22:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.275+0000 7fb349e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb34c15c830 0x7fb34c0ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.275+0000 7fb349e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb34c15c830 0x7fb34c0ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48704/0 (socket says 172.21.15.62:48704) 2024-03-23T11:22:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.275+0000 7fb349e00640 1 -- 172.21.15.62:0/3901932170 learned_addr learned my addr 172.21.15.62:0/3901932170 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.276+0000 7fb34a800640 1 --2- 172.21.15.62:0/3901932170 >> v2:172.21.15.120:3300/0 conn(0x7fb34c0ce250 0x7fb34c0ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.276+0000 7fb349e00640 1 -- 172.21.15.62:0/3901932170 >> v2:172.21.15.120:3300/0 conn(0x7fb34c0ce250 msgr2=0x7fb34c0ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.276+0000 7fb349e00640 1 --2- 172.21.15.62:0/3901932170 >> v2:172.21.15.120:3300/0 conn(0x7fb34c0ce250 0x7fb34c0ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.276+0000 7fb349e00640 1 -- 172.21.15.62:0/3901932170 >> v1:172.21.15.120:6789/0 conn(0x7fb34c160310 legacy=0x7fb34c0ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.276+0000 7fb349e00640 1 -- 172.21.15.62:0/3901932170 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c160bb0 con 0x7fb34c15c830 2024-03-23T11:22:17.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.277+0000 7fb349e00640 1 --2- 172.21.15.62:0/3901932170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb34c15c830 0x7fb34c0ad310 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb34c06c190 tx=0x7fb33c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb2e6bfb5cd62db server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:17.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.277+0000 7fb348a00640 1 -- 172.21.15.62:0/3901932170 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c031860 con 0x7fb34c15c830 2024-03-23T11:22:17.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.277+0000 7fb348a00640 1 -- 172.21.15.62:0/3901932170 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c0319c0 con 0x7fb34c15c830 2024-03-23T11:22:17.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.277+0000 7fb348a00640 1 -- 172.21.15.62:0/3901932170 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c031d50 con 0x7fb34c15c830 2024-03-23T11:22:17.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.278+0000 7fb34b200640 1 -- 172.21.15.62:0/3901932170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb34c15c830 msgr2=0x7fb34c0ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.278+0000 7fb34b200640 1 --2- 172.21.15.62:0/3901932170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb34c15c830 0x7fb34c0ad310 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb34c06c190 tx=0x7fb33c02b900 comp rx=0 tx=0).stop 2024-03-23T11:22:17.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.278+0000 7fb34b200640 1 -- 172.21.15.62:0/3901932170 shutdown_connections 2024-03-23T11:22:17.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.278+0000 7fb34b200640 1 --2- 172.21.15.62:0/3901932170 >> v2:172.21.15.120:3300/0 conn(0x7fb34c0ce250 0x7fb34c0ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.278+0000 7fb34b200640 1 --2- 172.21.15.62:0/3901932170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb34c15c830 0x7fb34c0ad310 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.278+0000 7fb34b200640 1 -- 172.21.15.62:0/3901932170 >> 172.21.15.62:0/3901932170 conn(0x7fb34c066050 msgr2=0x7fb34c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:17.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.278+0000 7fb34b200640 1 -- 172.21.15.62:0/3901932170 shutdown_connections 2024-03-23T11:22:17.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.278+0000 7fb34b200640 1 -- 172.21.15.62:0/3901932170 wait complete. 2024-03-23T11:22:17.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.280+0000 7fb34b200640 1 Processor -- start 2024-03-23T11:22:17.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.280+0000 7fb34b200640 1 -- start start 2024-03-23T11:22:17.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.281+0000 7fb34b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb34c0ce250 0x7fb34c145420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.281+0000 7fb34b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb34c15c830 0x7fb34c13fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.281+0000 7fb34b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb34c160310 0x7fb34c1439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.281+0000 7fb34b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c145000 con 0x7fb34c0ce250 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.281+0000 7fb34b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c144380 con 0x7fb34c15c830 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.281+0000 7fb34b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c144660 con 0x7fb34c160310 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.282+0000 7fb349e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb34c0ce250 0x7fb34c145420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.282+0000 7fb349e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb34c0ce250 0x7fb34c145420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53306/0 (socket says 172.21.15.62:53306) 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.282+0000 7fb349e00640 1 -- 172.21.15.62:0/492201537 learned_addr learned my addr 172.21.15.62:0/492201537 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.282+0000 7fb349e00640 1 -- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb34c160310 msgr2=0x7fb34c1439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.282+0000 7fb349e00640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb34c160310 0x7fb34c1439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.282+0000 7fb349e00640 1 -- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb34c15c830 msgr2=0x7fb34c13fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.282+0000 7fb349e00640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb34c15c830 0x7fb34c13fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.282+0000 7fb349e00640 1 -- 172.21.15.62:0/492201537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c0094d0 con 0x7fb34c0ce250 2024-03-23T11:22:17.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.282+0000 7fb349e00640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb34c0ce250 0x7fb34c145420 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb33c000c00 tx=0x7fb33c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:17.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.283+0000 7fb342000640 1 -- 172.21.15.62:0/492201537 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c03ac60 con 0x7fb34c0ce250 2024-03-23T11:22:17.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.283+0000 7fb342000640 1 -- 172.21.15.62:0/492201537 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c03adc0 con 0x7fb34c0ce250 2024-03-23T11:22:17.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.283+0000 7fb342000640 1 -- 172.21.15.62:0/492201537 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c044630 con 0x7fb34c0ce250 2024-03-23T11:22:17.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.283+0000 7fb34b200640 1 -- 172.21.15.62:0/492201537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34c144940 con 0x7fb34c0ce250 2024-03-23T11:22:17.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.283+0000 7fb34b200640 1 -- 172.21.15.62:0/492201537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34c140110 con 0x7fb34c0ce250 2024-03-23T11:22:17.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.284+0000 7fb34b200640 1 -- 172.21.15.62:0/492201537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34c066bb0 con 0x7fb34c0ce250 2024-03-23T11:22:17.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.285+0000 7fb342000640 1 -- 172.21.15.62:0/492201537 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb33c04d450 con 0x7fb34c0ce250 2024-03-23T11:22:17.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.286+0000 7fb342000640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb32c06a7a0 0x7fb32c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.286+0000 7fb342000640 1 -- 172.21.15.62:0/492201537 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb33c0ab330 con 0x7fb34c0ce250 2024-03-23T11:22:17.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.287+0000 7fb349400640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb32c06a7a0 0x7fb32c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:17.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.288+0000 7fb349400640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb32c06a7a0 0x7fb32c06cc60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0afff0 tx=0x7fb3380079b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:17.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.291+0000 7fb342000640 1 -- 172.21.15.62:0/492201537 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb33c0767a0 con 0x7fb34c0ce250 2024-03-23T11:22:17.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.533+0000 7fb34b200640 1 -- 172.21.15.62:0/492201537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb34c15af20 con 0x7fb34c0ce250 2024-03-23T11:22:17.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.534+0000 7fb342000640 1 -- 172.21.15.62:0/492201537 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fb33c079dc0 con 0x7fb34c0ce250 2024-03-23T11:22:17.531 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:17.531 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3787987198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3787987198},{"type":"v1","addr":"172.21.15.120:6835","nonce":3787987198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2749498408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2749498408},{"type":"v1","addr":"172.21.15.120:6837","nonce":2749498408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2325988870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2325988870},{"type":"v1","addr":"172.21.15.62:6833","nonce":2325988870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:21:25.687136+0000","modified":"2024-03-23T11:21:28.704192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6835/1159419089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1159419089},{"type":"v1","addr":"172.21.15.62:6835","nonce":1159419089}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T11:22:17.532 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-03-23T11:22:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.536+0000 7fb34b200640 1 -- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb32c06a7a0 msgr2=0x7fb32c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.536+0000 7fb34b200640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb32c06a7a0 0x7fb32c06cc60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0afff0 tx=0x7fb3380079b0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.536+0000 7fb34b200640 1 -- 172.21.15.62:0/492201537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb34c0ce250 msgr2=0x7fb34c145420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.536+0000 7fb34b200640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb34c0ce250 0x7fb34c145420 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb33c000c00 tx=0x7fb33c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.536+0000 7fb34b200640 1 -- 172.21.15.62:0/492201537 shutdown_connections 2024-03-23T11:22:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.536+0000 7fb34b200640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb32c06a7a0 0x7fb32c06cc60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.536+0000 7fb34b200640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb34c160310 0x7fb34c1439c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.536+0000 7fb34b200640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb34c15c830 0x7fb34c13fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.536+0000 7fb34b200640 1 --2- 172.21.15.62:0/492201537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb34c0ce250 0x7fb34c145420 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.536+0000 7fb34b200640 1 -- 172.21.15.62:0/492201537 >> 172.21.15.62:0/492201537 conn(0x7fb34c066050 msgr2=0x7fb34c15b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.537+0000 7fb34b200640 1 -- 172.21.15.62:0/492201537 shutdown_connections 2024-03-23T11:22:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.537+0000 7fb34b200640 1 -- 172.21.15.62:0/492201537 wait complete. 2024-03-23T11:22:17.554 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:22:17.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.774+0000 7f75f7a00640 1 Processor -- start 2024-03-23T11:22:17.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.775+0000 7f75f7a00640 1 -- start start 2024-03-23T11:22:17.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f7a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f75f015c9e0 0x7f75f01607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f7a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75f015cdc0 0x7f75f00d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f7a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f006bfc0 con 0x7f75f00cf650 2024-03-23T11:22:17.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f7a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f00ce140 con 0x7f75f015c9e0 2024-03-23T11:22:17.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f7a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f00b4370 con 0x7f75f015cdc0 2024-03-23T11:22:17.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f5000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f75f015c9e0 0x7f75f01607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:17.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f5000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f75f015c9e0 0x7f75f01607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53314/0 (socket says 172.21.15.62:53314) 2024-03-23T11:22:17.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f5000640 1 -- 172.21.15.62:0/3484736118 learned_addr learned my addr 172.21.15.62:0/3484736118 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:17.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f5000640 1 -- 172.21.15.62:0/3484736118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75f015cdc0 msgr2=0x7f75f00d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:17.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f5000640 1 --2- 172.21.15.62:0/3484736118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75f015cdc0 0x7f75f00d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f5000640 1 -- 172.21.15.62:0/3484736118 >> v1:172.21.15.120:6789/0 conn(0x7f75f00cf650 legacy=0x7f75f015d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:17.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.777+0000 7f75f5000640 1 -- 172.21.15.62:0/3484736118 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f00b5260 con 0x7f75f015c9e0 2024-03-23T11:22:17.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.779+0000 7f75f5000640 1 --2- 172.21.15.62:0/3484736118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75f015c9e0 0x7f75f01607c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f75e0009870 tx=0x7f75e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9463b0abe87218a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:17.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.779+0000 7f75ef400640 1 -- 172.21.15.62:0/3484736118 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0031640 con 0x7f75f015c9e0 2024-03-23T11:22:17.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.779+0000 7f75ef400640 1 -- 172.21.15.62:0/3484736118 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e00317a0 con 0x7f75f015c9e0 2024-03-23T11:22:17.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.779+0000 7f75ef400640 1 -- 172.21.15.62:0/3484736118 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0031b30 con 0x7f75f015c9e0 2024-03-23T11:22:17.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.780+0000 7f75f7a00640 1 -- 172.21.15.62:0/3484736118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75f015c9e0 msgr2=0x7f75f01607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.780+0000 7f75f7a00640 1 --2- 172.21.15.62:0/3484736118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75f015c9e0 0x7f75f01607c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f75e0009870 tx=0x7f75e002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.780+0000 7f75f7a00640 1 -- 172.21.15.62:0/3484736118 shutdown_connections 2024-03-23T11:22:17.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.780+0000 7f75f7a00640 1 --2- 172.21.15.62:0/3484736118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75f015cdc0 0x7f75f00d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.780+0000 7f75f7a00640 1 --2- 172.21.15.62:0/3484736118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75f015c9e0 0x7f75f01607c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.780+0000 7f75f7a00640 1 -- 172.21.15.62:0/3484736118 >> 172.21.15.62:0/3484736118 conn(0x7f75f0071570 msgr2=0x7f75f0071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:17.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.780+0000 7f75f7a00640 1 -- 172.21.15.62:0/3484736118 shutdown_connections 2024-03-23T11:22:17.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.780+0000 7f75f7a00640 1 -- 172.21.15.62:0/3484736118 wait complete. 2024-03-23T11:22:17.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.782+0000 7f75f7a00640 1 Processor -- start 2024-03-23T11:22:17.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.782+0000 7f75f7a00640 1 -- start start 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.783+0000 7f75f7a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75f00cf650 0x7f75f00abd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.783+0000 7f75f7a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75f015c9e0 0x7f75f0157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.783+0000 7f75f7a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75f015cdc0 0x7f75f0160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.783+0000 7f75f7a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f00ca9d0 con 0x7f75f015c9e0 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.783+0000 7f75f7a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f00cacb0 con 0x7f75f00cf650 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.783+0000 7f75f7a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f00caf90 con 0x7f75f015cdc0 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.783+0000 7f75efe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75f015c9e0 0x7f75f0157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.783+0000 7f75efe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75f015c9e0 0x7f75f0157c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53316/0 (socket says 172.21.15.62:53316) 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.783+0000 7f75efe00640 1 -- 172.21.15.62:0/1968860416 learned_addr learned my addr 172.21.15.62:0/1968860416 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75f5000640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75f00cf650 0x7f75f00abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75efe00640 1 -- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75f015cdc0 msgr2=0x7f75f0160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75f5a00640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75f015cdc0 0x7f75f0160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:17.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75efe00640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75f015cdc0 0x7f75f0160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75efe00640 1 -- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75f00cf650 msgr2=0x7f75f00abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75efe00640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75f00cf650 0x7f75f00abd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75efe00640 1 -- 172.21.15.62:0/1968860416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e00094d0 con 0x7f75f015c9e0 2024-03-23T11:22:17.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75f5000640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75f00cf650 0x7f75f00abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:17.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75efe00640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75f015c9e0 0x7f75f0157c70 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f75d800b410 tx=0x7f75d800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:17.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75ecc00640 1 -- 172.21.15.62:0/1968860416 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d8011830 con 0x7f75f015c9e0 2024-03-23T11:22:17.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75ecc00640 1 -- 172.21.15.62:0/1968860416 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d80119b0 con 0x7f75f015c9e0 2024-03-23T11:22:17.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75ecc00640 1 -- 172.21.15.62:0/1968860416 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d8011d60 con 0x7f75f015c9e0 2024-03-23T11:22:17.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75f7a00640 1 -- 172.21.15.62:0/1968860416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f00cb270 con 0x7f75f015c9e0 2024-03-23T11:22:17.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.784+0000 7f75f7a00640 1 -- 172.21.15.62:0/1968860416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f01581b0 con 0x7f75f015c9e0 2024-03-23T11:22:17.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.786+0000 7f75f7a00640 1 -- 172.21.15.62:0/1968860416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f00cb790 con 0x7f75f015c9e0 2024-03-23T11:22:17.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.787+0000 7f75ecc00640 1 -- 172.21.15.62:0/1968860416 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f75d801a400 con 0x7f75f015c9e0 2024-03-23T11:22:17.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.788+0000 7f75ecc00640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f75cc06a7a0 0x7f75cc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:17.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.788+0000 7f75ecc00640 1 -- 172.21.15.62:0/1968860416 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f75d8088ea0 con 0x7f75f015c9e0 2024-03-23T11:22:17.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.788+0000 7f75f5000640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f75cc06a7a0 0x7f75cc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:17.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.789+0000 7f75f5000640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f75cc06a7a0 0x7f75cc06cc60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f75e0036000 tx=0x7f75e003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:17.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.792+0000 7f75ecc00640 1 -- 172.21.15.62:0/1968860416 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f75f00cb790 con 0x7f75f015c9e0 2024-03-23T11:22:17.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.989+0000 7f75f7a00640 1 -- 172.21.15.62:0/1968860416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75f00cb790 con 0x7f75f015c9e0 2024-03-23T11:22:17.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.991+0000 7f75ecc00640 1 -- 172.21.15.62:0/1968860416 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16197 (secure 0 0 0) 0x7f75f00cb790 con 0x7f75f015c9e0 2024-03-23T11:22:17.989 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:17.989 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":50,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:21:43.375126+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:21:23.756222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-23T11:21:24.715577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.2a","mappings":[{"from":0,"to":5}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:22:17.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.994+0000 7f75f7a00640 1 -- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f75cc06a7a0 msgr2=0x7f75cc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.994+0000 7f75f7a00640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f75cc06a7a0 0x7f75cc06cc60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f75e0036000 tx=0x7f75e003c040 comp rx=0 tx=0).stop 2024-03-23T11:22:17.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.994+0000 7f75f7a00640 1 -- 172.21.15.62:0/1968860416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75f015c9e0 msgr2=0x7f75f0157c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:17.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.994+0000 7f75f7a00640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75f015c9e0 0x7f75f0157c70 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f75d800b410 tx=0x7f75d800b8d0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.994+0000 7f75f7a00640 1 -- 172.21.15.62:0/1968860416 shutdown_connections 2024-03-23T11:22:17.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.995+0000 7f75f7a00640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f75cc06a7a0 0x7f75cc06cc60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.995+0000 7f75f7a00640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75f015c9e0 0x7f75f0157c70 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.995+0000 7f75f7a00640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75f015cdc0 0x7f75f0160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.995+0000 7f75f7a00640 1 --2- 172.21.15.62:0/1968860416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75f00cf650 0x7f75f00abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:17.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.995+0000 7f75f7a00640 1 -- 172.21.15.62:0/1968860416 >> 172.21.15.62:0/1968860416 conn(0x7f75f0071570 msgr2=0x7f75f00cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:17.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.995+0000 7f75f7a00640 1 -- 172.21.15.62:0/1968860416 shutdown_connections 2024-03-23T11:22:17.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:17.995+0000 7f75f7a00640 1 -- 172.21.15.62:0/1968860416 wait complete. 2024-03-23T11:22:18.010 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:22:18.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.235+0000 7f99a2c00640 1 Processor -- start 2024-03-23T11:22:18.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.235+0000 7f99a2c00640 1 -- start start 2024-03-23T11:22:18.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.237+0000 7f99a2c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f999c069c60 0x7f999c06a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.237+0000 7f99a2c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f999c0caff0 0x7f999c0cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.237+0000 7f99a2c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f999c143450 0x7f999c1438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.237+0000 7f99a2c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f999c0ce1a0 con 0x7f999c143450 2024-03-23T11:22:18.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.237+0000 7f99a2c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f999c0cbcb0 con 0x7f999c069c60 2024-03-23T11:22:18.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.237+0000 7f99a2c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999c15ab40 con 0x7f999c0caff0 2024-03-23T11:22:18.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.237+0000 7f99a2200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f999c143450 0x7f999c1438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.237+0000 7f99a2200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f999c143450 0x7f999c1438e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53322/0 (socket says 172.21.15.62:53322) 2024-03-23T11:22:18.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.238+0000 7f99a2200640 1 -- 172.21.15.62:0/1909589945 learned_addr learned my addr 172.21.15.62:0/1909589945 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:18.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.238+0000 7f99a1800640 1 --2- 172.21.15.62:0/1909589945 >> v2:172.21.15.120:3300/0 conn(0x7f999c069c60 0x7f999c06a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.238+0000 7f99a0e00640 1 --2- 172.21.15.62:0/1909589945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f999c0caff0 0x7f999c0cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.238+0000 7f99a2200640 1 -- 172.21.15.62:0/1909589945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f999c0caff0 msgr2=0x7f999c0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:18.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.238+0000 7f99a2200640 1 --2- 172.21.15.62:0/1909589945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f999c0caff0 0x7f999c0cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.238+0000 7f99a2200640 1 -- 172.21.15.62:0/1909589945 >> v2:172.21.15.120:3300/0 conn(0x7f999c069c60 msgr2=0x7f999c06a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:18.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.238+0000 7f99a2200640 1 --2- 172.21.15.62:0/1909589945 >> v2:172.21.15.120:3300/0 conn(0x7f999c069c60 0x7f999c06a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.238+0000 7f99a2200640 1 -- 172.21.15.62:0/1909589945 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c15bb10 con 0x7f999c143450 2024-03-23T11:22:18.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.239+0000 7f99a2200640 1 --2- 172.21.15.62:0/1909589945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f999c143450 0x7f999c1438e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f999c06ad90 tx=0x7f999402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0de35331577e07a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:18.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.239+0000 7f999be00640 1 -- 172.21.15.62:0/1909589945 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999402ca80 con 0x7f999c143450 2024-03-23T11:22:18.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.239+0000 7f999be00640 1 -- 172.21.15.62:0/1909589945 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999402cbe0 con 0x7f999c143450 2024-03-23T11:22:18.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.239+0000 7f999be00640 1 -- 172.21.15.62:0/1909589945 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99940343f0 con 0x7f999c143450 2024-03-23T11:22:18.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.240+0000 7f99a2c00640 1 -- 172.21.15.62:0/1909589945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f999c143450 msgr2=0x7f999c1438e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:18.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.240+0000 7f99a2c00640 1 --2- 172.21.15.62:0/1909589945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f999c143450 0x7f999c1438e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f999c06ad90 tx=0x7f999402bee0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.240+0000 7f99a2c00640 1 -- 172.21.15.62:0/1909589945 shutdown_connections 2024-03-23T11:22:18.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.240+0000 7f99a2c00640 1 --2- 172.21.15.62:0/1909589945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f999c143450 0x7f999c1438e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.240+0000 7f99a2c00640 1 --2- 172.21.15.62:0/1909589945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f999c0caff0 0x7f999c0cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.240+0000 7f99a2c00640 1 --2- 172.21.15.62:0/1909589945 >> v2:172.21.15.120:3300/0 conn(0x7f999c069c60 0x7f999c06a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.240+0000 7f99a2c00640 1 -- 172.21.15.62:0/1909589945 >> 172.21.15.62:0/1909589945 conn(0x7f999c0b0090 msgr2=0x7f999c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:18.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.240+0000 7f99a2c00640 1 -- 172.21.15.62:0/1909589945 shutdown_connections 2024-03-23T11:22:18.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.240+0000 7f99a2c00640 1 -- 172.21.15.62:0/1909589945 wait complete. 2024-03-23T11:22:18.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.242+0000 7f99a2c00640 1 Processor -- start 2024-03-23T11:22:18.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.242+0000 7f99a2c00640 1 -- start start 2024-03-23T11:22:18.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.243+0000 7f99a2c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f999c069c60 0x7f999c13a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.243+0000 7f99a2c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f999c0caff0 0x7f999c1f1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.243+0000 7f99a2c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f999c143450 0x7f999c0c3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.243+0000 7f99a2c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c0b2c10 con 0x7f999c0caff0 2024-03-23T11:22:18.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.243+0000 7f99a2c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c13b080 con 0x7f999c143450 2024-03-23T11:22:18.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.243+0000 7f99a2c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999c0b2090 con 0x7f999c069c60 2024-03-23T11:22:18.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f999c0caff0 0x7f999c1f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f999c0caff0 0x7f999c1f1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53330/0 (socket says 172.21.15.62:53330) 2024-03-23T11:22:18.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a0e00640 1 -- 172.21.15.62:0/4285255172 learned_addr learned my addr 172.21.15.62:0/4285255172 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:18.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a2200640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f999c143450 0x7f999c0c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a1800640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f999c069c60 0x7f999c13a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a0e00640 1 -- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f999c069c60 msgr2=0x7f999c13a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:18.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a0e00640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f999c069c60 0x7f999c13a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a0e00640 1 -- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f999c143450 msgr2=0x7f999c0c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:18.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a0e00640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f999c143450 0x7f999c0c3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a0e00640 1 -- 172.21.15.62:0/4285255172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9994009b30 con 0x7f999c0caff0 2024-03-23T11:22:18.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a1800640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f999c069c60 0x7f999c13a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:18.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.244+0000 7f99a0e00640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f999c0caff0 0x7f999c1f1470 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f998800d800 tx=0x7f998800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:18.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.245+0000 7f9999600640 1 -- 172.21.15.62:0/4285255172 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998800b6c0 con 0x7f999c0caff0 2024-03-23T11:22:18.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.245+0000 7f9999600640 1 -- 172.21.15.62:0/4285255172 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998800b820 con 0x7f999c0caff0 2024-03-23T11:22:18.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.245+0000 7f9999600640 1 -- 172.21.15.62:0/4285255172 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998800bbb0 con 0x7f999c0caff0 2024-03-23T11:22:18.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.245+0000 7f99a2c00640 1 -- 172.21.15.62:0/4285255172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999c0b2370 con 0x7f999c0caff0 2024-03-23T11:22:18.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.245+0000 7f99a2c00640 1 -- 172.21.15.62:0/4285255172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999c0b2930 con 0x7f999c0caff0 2024-03-23T11:22:18.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.246+0000 7f99a2c00640 1 -- 172.21.15.62:0/4285255172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999c066bb0 con 0x7f999c0caff0 2024-03-23T11:22:18.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.247+0000 7f9999600640 1 -- 172.21.15.62:0/4285255172 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f998800bd10 con 0x7f999c0caff0 2024-03-23T11:22:18.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.248+0000 7f9999600640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997c06a7a0 0x7f997c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.248+0000 7f9999600640 1 -- 172.21.15.62:0/4285255172 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9988089460 con 0x7f999c0caff0 2024-03-23T11:22:18.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.249+0000 7f99a1800640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997c06a7a0 0x7f997c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.250+0000 7f99a1800640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997c06a7a0 0x7f997c06cc60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9990006fd0 tx=0x7f9990008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:18.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.253+0000 7f9999600640 1 -- 172.21.15.62:0/4285255172 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f99880547a0 con 0x7f999c0caff0 2024-03-23T11:22:18.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.500+0000 7f99a2c00640 1 -- 172.21.15.62:0/4285255172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f999c157640 con 0x7f999c0caff0 2024-03-23T11:22:18.500 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:22:18.503+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:22:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.519+0000 7f9999600640 1 -- 172.21.15.62:0/4285255172 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9988057dc0 con 0x7f999c0caff0 2024-03-23T11:22:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:22:18.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.521+0000 7f99a2c00640 1 -- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997c06a7a0 msgr2=0x7f997c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:18.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.521+0000 7f99a2c00640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997c06a7a0 0x7f997c06cc60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9990006fd0 tx=0x7f9990008040 comp rx=0 tx=0).stop 2024-03-23T11:22:18.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.522+0000 7f99a2c00640 1 -- 172.21.15.62:0/4285255172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f999c0caff0 msgr2=0x7f999c1f1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:18.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.522+0000 7f99a2c00640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f999c0caff0 0x7f999c1f1470 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f998800d800 tx=0x7f998800dcc0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.522+0000 7f99a2c00640 1 -- 172.21.15.62:0/4285255172 shutdown_connections 2024-03-23T11:22:18.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.522+0000 7f99a2c00640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f997c06a7a0 0x7f997c06cc60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.522+0000 7f99a2c00640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f999c0caff0 0x7f999c1f1470 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.522+0000 7f99a2c00640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f999c143450 0x7f999c0c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.522+0000 7f99a2c00640 1 --2- 172.21.15.62:0/4285255172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f999c069c60 0x7f999c13a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.522+0000 7f99a2c00640 1 -- 172.21.15.62:0/4285255172 >> 172.21.15.62:0/4285255172 conn(0x7f999c0b0090 msgr2=0x7f999c06ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:18.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.522+0000 7f99a2c00640 1 -- 172.21.15.62:0/4285255172 shutdown_connections 2024-03-23T11:22:18.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.522+0000 7f99a2c00640 1 -- 172.21.15.62:0/4285255172 wait complete. 2024-03-23T11:22:18.534 INFO:tasks.ceph.mds.a.smithi062.stderr: -4793> 2024-03-23T11:21:31.183+0000 7ff6f8200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:22:18.540 INFO:tasks.ceph.mds.a.smithi062.stderr: -3117> 2024-03-23T11:21:38.255+0000 7ff6f3c00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5808=0-4194304@1} caps={5808=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5808 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x557d5f12e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-23T11:22:18.545 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T11:22:18.593 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:22:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.737+0000 7fb9dc800640 1 Processor -- start 2024-03-23T11:22:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.737+0000 7fb9dc800640 1 -- start start 2024-03-23T11:22:18.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9dc800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb9d40cf780 0x7fb9d415b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9dc800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9d406fd20 0x7fb9d40701b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9dc800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d40b5120 con 0x7fb9d40cde00 2024-03-23T11:22:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9dc800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d40cd110 con 0x7fb9d40cf780 2024-03-23T11:22:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9dc800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d415bd80 con 0x7fb9d406fd20 2024-03-23T11:22:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9d9e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb9d40cf780 0x7fb9d415b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9d9e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb9d40cf780 0x7fb9d415b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53356/0 (socket says 172.21.15.62:53356) 2024-03-23T11:22:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9d9e00640 1 -- 172.21.15.62:0/935548900 learned_addr learned my addr 172.21.15.62:0/935548900 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9d9e00640 1 -- 172.21.15.62:0/935548900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9d406fd20 msgr2=0x7fb9d40701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9d9e00640 1 --2- 172.21.15.62:0/935548900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9d406fd20 0x7fb9d40701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9d9e00640 1 -- 172.21.15.62:0/935548900 >> v1:172.21.15.120:6789/0 conn(0x7fb9d40cde00 legacy=0x7fb9d40cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.739+0000 7fb9d9e00640 1 -- 172.21.15.62:0/935548900 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d40b57c0 con 0x7fb9d40cf780 2024-03-23T11:22:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.740+0000 7fb9d9e00640 1 --2- 172.21.15.62:0/935548900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9d40cf780 0x7fb9d415b840 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009e50 tx=0x7fb9cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d03d79a7a892ea server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.741+0000 7fb9d8a00640 1 -- 172.21.15.62:0/935548900 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc02ca70 con 0x7fb9d40cf780 2024-03-23T11:22:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.741+0000 7fb9d8a00640 1 -- 172.21.15.62:0/935548900 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9cc02cbd0 con 0x7fb9d40cf780 2024-03-23T11:22:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.741+0000 7fb9d8a00640 1 -- 172.21.15.62:0/935548900 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc0343f0 con 0x7fb9d40cf780 2024-03-23T11:22:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.741+0000 7fb9dc800640 1 -- 172.21.15.62:0/935548900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9d40cf780 msgr2=0x7fb9d415b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.741+0000 7fb9dc800640 1 --2- 172.21.15.62:0/935548900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9d40cf780 0x7fb9d415b840 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009e50 tx=0x7fb9cc02bed0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.741+0000 7fb9dc800640 1 -- 172.21.15.62:0/935548900 shutdown_connections 2024-03-23T11:22:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.741+0000 7fb9dc800640 1 --2- 172.21.15.62:0/935548900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9d406fd20 0x7fb9d40701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.741+0000 7fb9dc800640 1 --2- 172.21.15.62:0/935548900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9d40cf780 0x7fb9d415b840 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.741+0000 7fb9dc800640 1 -- 172.21.15.62:0/935548900 >> 172.21.15.62:0/935548900 conn(0x7fb9d406e580 msgr2=0x7fb9d406e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.741+0000 7fb9dc800640 1 -- 172.21.15.62:0/935548900 shutdown_connections 2024-03-23T11:22:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.741+0000 7fb9dc800640 1 -- 172.21.15.62:0/935548900 wait complete. 2024-03-23T11:22:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.743+0000 7fb9dc800640 1 Processor -- start 2024-03-23T11:22:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.743+0000 7fb9dc800640 1 -- start start 2024-03-23T11:22:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.744+0000 7fb9dc800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9d406fd20 0x7fb9d40c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.744+0000 7fb9dc800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9d40cde00 0x7fb9d41f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.744+0000 7fb9dc800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9d40cf780 0x7fb9d414d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.744+0000 7fb9dc800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d41f7450 con 0x7fb9d40cf780 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.744+0000 7fb9dc800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d40cb1f0 con 0x7fb9d40cde00 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.744+0000 7fb9dc800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d40ca670 con 0x7fb9d406fd20 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.744+0000 7fb9da800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9d40cf780 0x7fb9d414d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.744+0000 7fb9da800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9d40cf780 0x7fb9d414d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53358/0 (socket says 172.21.15.62:53358) 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.745+0000 7fb9da800640 1 -- 172.21.15.62:0/506123921 learned_addr learned my addr 172.21.15.62:0/506123921 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.745+0000 7fb9d9e00640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9d406fd20 0x7fb9d40c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.745+0000 7fb9d9400640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9d40cde00 0x7fb9d41f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.745+0000 7fb9da800640 1 -- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9d406fd20 msgr2=0x7fb9d40c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.745+0000 7fb9da800640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9d406fd20 0x7fb9d40c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.745+0000 7fb9da800640 1 -- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9d40cde00 msgr2=0x7fb9d41f6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.745+0000 7fb9da800640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9d40cde00 0x7fb9d41f6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.745+0000 7fb9da800640 1 -- 172.21.15.62:0/506123921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc009b30 con 0x7fb9d40cf780 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.745+0000 7fb9da800640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9d40cf780 0x7fb9d414d180 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000b410 tx=0x7fb9d000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.745+0000 7fb9ca000640 1 -- 172.21.15.62:0/506123921 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0011830 con 0x7fb9d40cf780 2024-03-23T11:22:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.746+0000 7fb9ca000640 1 -- 172.21.15.62:0/506123921 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d00119b0 con 0x7fb9d40cf780 2024-03-23T11:22:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.746+0000 7fb9dc800640 1 -- 172.21.15.62:0/506123921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d40ca950 con 0x7fb9d40cf780 2024-03-23T11:22:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.746+0000 7fb9ca000640 1 -- 172.21.15.62:0/506123921 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0011d60 con 0x7fb9d40cf780 2024-03-23T11:22:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.746+0000 7fb9dc800640 1 -- 172.21.15.62:0/506123921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d40caf10 con 0x7fb9d40cf780 2024-03-23T11:22:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.747+0000 7fb9dc800640 1 -- 172.21.15.62:0/506123921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d40672c0 con 0x7fb9d40cf780 2024-03-23T11:22:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.748+0000 7fb9ca000640 1 -- 172.21.15.62:0/506123921 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb9d001a400 con 0x7fb9d40cf780 2024-03-23T11:22:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.748+0000 7fb9ca000640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb9b806a7a0 0x7fb9b806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.749+0000 7fb9ca000640 1 -- 172.21.15.62:0/506123921 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb9d0088fd0 con 0x7fb9d40cf780 2024-03-23T11:22:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.749+0000 7fb9d9e00640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb9b806a7a0 0x7fb9b806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:18.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.750+0000 7fb9d9e00640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb9b806a7a0 0x7fb9b806cc60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc035000 tx=0x7fb9cc030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:18.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.753+0000 7fb9ca000640 1 -- 172.21.15.62:0/506123921 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb9d00542c0 con 0x7fb9d40cf780 2024-03-23T11:22:18.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:18.989+0000 7fb9dc800640 1 -- 172.21.15.62:0/506123921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb9d406f170 con 0x7fb9d40cf780 2024-03-23T11:22:19.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.522+0000 7fb9ca000640 1 -- 172.21.15.62:0/506123921 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb9d00578e0 con 0x7fb9d40cf780 2024-03-23T11:22:19.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.526+0000 7fb9dc800640 1 -- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb9b806a7a0 msgr2=0x7fb9b806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:19.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.526+0000 7fb9dc800640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb9b806a7a0 0x7fb9b806cc60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc035000 tx=0x7fb9cc030000 comp rx=0 tx=0).stop 2024-03-23T11:22:19.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.526+0000 7fb9dc800640 1 -- 172.21.15.62:0/506123921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9d40cf780 msgr2=0x7fb9d414d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:19.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.526+0000 7fb9dc800640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9d40cf780 0x7fb9d414d180 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000b410 tx=0x7fb9d000b8d0 comp rx=0 tx=0).stop 2024-03-23T11:22:19.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.526+0000 7fb9dc800640 1 -- 172.21.15.62:0/506123921 shutdown_connections 2024-03-23T11:22:19.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.526+0000 7fb9dc800640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb9b806a7a0 0x7fb9b806cc60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:19.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.526+0000 7fb9dc800640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9d40cf780 0x7fb9d414d180 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:19.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.526+0000 7fb9dc800640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9d40cde00 0x7fb9d41f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:19.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.526+0000 7fb9dc800640 1 --2- 172.21.15.62:0/506123921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9d406fd20 0x7fb9d40c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:19.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.526+0000 7fb9dc800640 1 -- 172.21.15.62:0/506123921 >> 172.21.15.62:0/506123921 conn(0x7fb9d406e580 msgr2=0x7fb9d40b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:19.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.527+0000 7fb9dc800640 1 -- 172.21.15.62:0/506123921 shutdown_connections 2024-03-23T11:22:19.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.527+0000 7fb9dc800640 1 -- 172.21.15.62:0/506123921 wait complete. 2024-03-23T11:22:19.544 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T11:22:19.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.769+0000 7f586ca00640 1 Processor -- start 2024-03-23T11:22:19.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.769+0000 7f586ca00640 1 -- start start 2024-03-23T11:22:19.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.771+0000 7f586ca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806d950 0x7f586806c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:19.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.771+0000 7f586ca00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586815cfd0 con 0x7f58680ce690 2024-03-23T11:22:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.771+0000 7f586ca00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586815d220 con 0x7f586806dd30 2024-03-23T11:22:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.771+0000 7f586ca00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586815d500 con 0x7f586806d950 2024-03-23T11:22:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.771+0000 7f5867200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f58680ce690 0x7f58680ceab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55958/0 (socket says 172.21.15.62:55958) 2024-03-23T11:22:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.771+0000 7f5867200640 1 -- 172.21.15.62:0/4097851755 learned_addr learned my addr 172.21.15.62:0/4097851755 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5866800640 1 --2- 172.21.15.62:0/4097851755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806d950 0x7f586806c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028360202 0 0) 0x7f586815cfd0 con 0x7f58680ce690 2024-03-23T11:22:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f584c003520 con 0x7f58680ce690 2024-03-23T11:22:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3233140312 0 0) 0x7f584c003520 con 0x7f58680ce690 2024-03-23T11:22:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806d950 msgr2=0x7f586806c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5865400640 1 --2- 172.21.15.62:0/4097851755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806d950 0x7f586806c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 >> v1:172.21.15.120:6789/0 conn(0x7f586806dd30 legacy=0x7f5868070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586815d7e0 con 0x7f58680ce690 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f58600019f0 con 0x7f58680ce690 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 >> v1:172.21.15.62:6789/0 conn(0x7f58680ce690 legacy=0x7f58680ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5865400640 1 --2- 172.21.15.62:0/4097851755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806d950 0x7f584c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.772+0000 7f5865400640 1 --2- 172.21.15.62:0/4097851755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f584c003f20 0x7f584c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.773+0000 7f5865400640 1 --2- 172.21.15.62:0/4097851755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f584c004d00 0x7f584c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.773+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586815d500 con 0x7f584c003f20 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.773+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c00bb10 con 0x7f584c004d00 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.773+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584c00bcd0 con 0x7f586806d950 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.773+0000 7f5865e00640 1 --2- 172.21.15.62:0/4097851755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f584c003f20 0x7f584c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.773+0000 7f5866800640 1 --2- 172.21.15.62:0/4097851755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806d950 0x7f584c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.773+0000 7f5865e00640 1 -- 172.21.15.62:0/4097851755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806d950 msgr2=0x7f584c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.773+0000 7f5865e00640 1 --2- 172.21.15.62:0/4097851755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806d950 0x7f584c0039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.773+0000 7f5865e00640 1 -- 172.21.15.62:0/4097851755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f584c004d00 msgr2=0x7f584c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.773+0000 7f5865e00640 1 --2- 172.21.15.62:0/4097851755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f584c004d00 0x7f584c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.773+0000 7f5865e00640 1 -- 172.21.15.62:0/4097851755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584c003520 con 0x7f584c003f20 2024-03-23T11:22:19.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.774+0000 7f5865e00640 1 --2- 172.21.15.62:0/4097851755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f584c003f20 0x7f584c0047c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5854001750 tx=0x7f584c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68179e5dacdf528b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:19.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.775+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58540052d0 con 0x7f584c003f20 2024-03-23T11:22:19.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.775+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f585402e050 con 0x7f584c003f20 2024-03-23T11:22:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.775+0000 7f5865400640 1 -- 172.21.15.62:0/4097851755 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58540035f0 con 0x7f584c003f20 2024-03-23T11:22:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.775+0000 7f586ca00640 1 -- 172.21.15.62:0/4097851755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f584c003f20 msgr2=0x7f584c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.775+0000 7f586ca00640 1 --2- 172.21.15.62:0/4097851755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f584c003f20 0x7f584c0047c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5854001750 tx=0x7f584c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.775+0000 7f5867200640 1 -- 172.21.15.62:0/4097851755 reap_dead start 2024-03-23T11:22:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.775+0000 7f586ca00640 1 -- 172.21.15.62:0/4097851755 shutdown_connections 2024-03-23T11:22:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.775+0000 7f586ca00640 1 -- 172.21.15.62:0/4097851755 >> 172.21.15.62:0/4097851755 conn(0x7f58680b0280 msgr2=0x7f58680b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.775+0000 7f586ca00640 1 -- 172.21.15.62:0/4097851755 shutdown_connections 2024-03-23T11:22:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.775+0000 7f586ca00640 1 -- 172.21.15.62:0/4097851755 wait complete. 2024-03-23T11:22:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.777+0000 7f586ca00640 1 Processor -- start 2024-03-23T11:22:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.777+0000 7f586ca00640 1 -- start start 2024-03-23T11:22:19.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.778+0000 7f586ca00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f584c003f20 0x7f5868148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:19.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.778+0000 7f586ca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58681f3b00 0x7f58681f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:19.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.778+0000 7f586ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5868148c30 0x7f58681490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:19.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.778+0000 7f586ca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5868149eb0 con 0x7f5868148c30 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.778+0000 7f586ca00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58681f44c0 con 0x7f584c003f20 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.778+0000 7f586ca00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5868157980 con 0x7f58681f3b00 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.778+0000 7f5867200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5868148c30 0x7f58681490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5867200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5868148c30 0x7f58681490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53386/0 (socket says 172.21.15.62:53386) 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5867200640 1 -- 172.21.15.62:0/3482866454 learned_addr learned my addr 172.21.15.62:0/3482866454 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5866800640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f584c003f20 0x7f5868148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5867200640 1 -- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58681f3b00 msgr2=0x7f58681f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5865e00640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58681f3b00 0x7f58681f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5867200640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58681f3b00 0x7f58681f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5867200640 1 -- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f584c003f20 msgr2=0x7f5868148620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5867200640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f584c003f20 0x7f5868148620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5867200640 1 -- 172.21.15.62:0/3482866454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5868135ca0 con 0x7f5868148c30 2024-03-23T11:22:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5866800640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f584c003f20 0x7f5868148620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:19.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5865e00640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58681f3b00 0x7f58681f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:19.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.779+0000 7f5867200640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5868148c30 0x7f58681490b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f58681599e0 tx=0x7f586000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:19.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.780+0000 7f585ea00640 1 -- 172.21.15.62:0/3482866454 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586000a610 con 0x7f5868148c30 2024-03-23T11:22:19.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.780+0000 7f585ea00640 1 -- 172.21.15.62:0/3482866454 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586000a770 con 0x7f5868148c30 2024-03-23T11:22:19.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.780+0000 7f585ea00640 1 -- 172.21.15.62:0/3482866454 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586000ab20 con 0x7f5868148c30 2024-03-23T11:22:19.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.780+0000 7f586ca00640 1 -- 172.21.15.62:0/3482866454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58681498f0 con 0x7f5868148c30 2024-03-23T11:22:19.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.780+0000 7f586ca00640 1 -- 172.21.15.62:0/3482866454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5868157610 con 0x7f5868148c30 2024-03-23T11:22:19.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.781+0000 7f586ca00640 1 -- 172.21.15.62:0/3482866454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5868157470 con 0x7f5868148c30 2024-03-23T11:22:19.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.782+0000 7f585ea00640 1 -- 172.21.15.62:0/3482866454 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f586000ac80 con 0x7f5868148c30 2024-03-23T11:22:19.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.783+0000 7f585ea00640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f584806a7a0 0x7f584806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:19.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.783+0000 7f585ea00640 1 -- 172.21.15.62:0/3482866454 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5860088810 con 0x7f5868148c30 2024-03-23T11:22:19.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.784+0000 7f5866800640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f584806a7a0 0x7f584806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:19.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.785+0000 7f5866800640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f584806a7a0 0x7f584806cc60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f58580099e0 tx=0x7f5858008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:19.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.788+0000 7f585ea00640 1 -- 172.21.15.62:0/3482866454 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5868157470 con 0x7f5868148c30 2024-03-23T11:22:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:19.989+0000 7f586ca00640 1 -- 172.21.15.62:0/3482866454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5868157470 con 0x7f5868148c30 2024-03-23T11:22:20.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.535+0000 7f585ea00640 1 -- 172.21.15.62:0/3482866454 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v52) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5868157470 con 0x7f5868148c30 2024-03-23T11:22:20.533 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' removed 2024-03-23T11:22:20.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.540+0000 7f586ca00640 1 -- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f584806a7a0 msgr2=0x7f584806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:20.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.540+0000 7f586ca00640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f584806a7a0 0x7f584806cc60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f58580099e0 tx=0x7f5858008040 comp rx=0 tx=0).stop 2024-03-23T11:22:20.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.540+0000 7f586ca00640 1 -- 172.21.15.62:0/3482866454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5868148c30 msgr2=0x7f58681490b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:20.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.540+0000 7f586ca00640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5868148c30 0x7f58681490b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f58681599e0 tx=0x7f586000cf00 comp rx=0 tx=0).stop 2024-03-23T11:22:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.541+0000 7f586ca00640 1 -- 172.21.15.62:0/3482866454 shutdown_connections 2024-03-23T11:22:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.541+0000 7f586ca00640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f584806a7a0 0x7f584806cc60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.541+0000 7f586ca00640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5868148c30 0x7f58681490b0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.541+0000 7f586ca00640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58681f3b00 0x7f58681f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.541+0000 7f586ca00640 1 --2- 172.21.15.62:0/3482866454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f584c003f20 0x7f5868148620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.541+0000 7f586ca00640 1 -- 172.21.15.62:0/3482866454 >> 172.21.15.62:0/3482866454 conn(0x7f58680b0280 msgr2=0x7f5868072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.541+0000 7f586ca00640 1 -- 172.21.15.62:0/3482866454 shutdown_connections 2024-03-23T11:22:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.541+0000 7f586ca00640 1 -- 172.21.15.62:0/3482866454 wait complete. 2024-03-23T11:22:20.560 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T11:22:20.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.782+0000 7f5fb0e00640 1 Processor -- start 2024-03-23T11:22:20.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.782+0000 7f5fb0e00640 1 -- start start 2024-03-23T11:22:20.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.784+0000 7f5fb0e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5fac0cf2c0 0x7f5fac0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:20.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.784+0000 7f5fb0e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fac0d1320 0x7f5fac0d1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.784+0000 7f5fb0e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac0b58b0 con 0x7f5fac15ca70 2024-03-23T11:22:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.784+0000 7f5fb0e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fac0cfc90 con 0x7f5fac0cf2c0 2024-03-23T11:22:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.784+0000 7f5fb0e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac0d0910 con 0x7f5fac0d1320 2024-03-23T11:22:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.784+0000 7f5fa9e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5fac15ca70 0x7f5fac1606a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55974/0 (socket says 172.21.15.62:55974) 2024-03-23T11:22:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.784+0000 7f5fa9e00640 1 -- 172.21.15.62:0/56045056 learned_addr learned my addr 172.21.15.62:0/56045056 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:20.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.784+0000 7f5fa9400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5fac0cf2c0 0x7f5fac0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:20.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.785+0000 7f5faa800640 1 --2- 172.21.15.62:0/56045056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fac0d1320 0x7f5fac0d1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:20.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.785+0000 7f5fa9400640 1 -- 172.21.15.62:0/56045056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fac0d1320 msgr2=0x7f5fac0d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:20.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.785+0000 7f5fa9400640 1 --2- 172.21.15.62:0/56045056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fac0d1320 0x7f5fac0d1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:20.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.785+0000 7f5fa9400640 1 -- 172.21.15.62:0/56045056 >> v1:172.21.15.62:6789/0 conn(0x7f5fac15ca70 legacy=0x7f5fac1606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:20.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.785+0000 7f5fa8a00640 1 -- 172.21.15.62:0/56045056 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587076763 0 0) 0x7f5fac0b58b0 con 0x7f5fac15ca70 2024-03-23T11:22:20.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.785+0000 7f5fa9400640 1 -- 172.21.15.62:0/56045056 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac161160 con 0x7f5fac0cf2c0 2024-03-23T11:22:20.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.786+0000 7f5fa9400640 1 --2- 172.21.15.62:0/56045056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fac0cf2c0 0x7f5fac0cf750 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5f980097a0 tx=0x7f5f9802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc3716bbd8b8b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:20.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.786+0000 7f5fa8a00640 1 -- 172.21.15.62:0/56045056 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98030800 con 0x7f5fac0cf2c0 2024-03-23T11:22:20.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.786+0000 7f5fa8a00640 1 -- 172.21.15.62:0/56045056 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f98030960 con 0x7f5fac0cf2c0 2024-03-23T11:22:20.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.786+0000 7f5fa8a00640 1 -- 172.21.15.62:0/56045056 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98030cf0 con 0x7f5fac0cf2c0 2024-03-23T11:22:20.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.787+0000 7f5fb0e00640 1 -- 172.21.15.62:0/56045056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fac0cf2c0 msgr2=0x7f5fac0cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:20.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.787+0000 7f5fb0e00640 1 --2- 172.21.15.62:0/56045056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fac0cf2c0 0x7f5fac0cf750 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5f980097a0 tx=0x7f5f9802b940 comp rx=0 tx=0).stop 2024-03-23T11:22:20.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.787+0000 7f5fb0e00640 1 -- 172.21.15.62:0/56045056 shutdown_connections 2024-03-23T11:22:20.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.787+0000 7f5fb0e00640 1 --2- 172.21.15.62:0/56045056 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fac0d1320 0x7f5fac0d1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:20.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.787+0000 7f5fb0e00640 1 --2- 172.21.15.62:0/56045056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fac0cf2c0 0x7f5fac0cf750 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:20.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.787+0000 7f5fb0e00640 1 -- 172.21.15.62:0/56045056 >> 172.21.15.62:0/56045056 conn(0x7f5fac071390 msgr2=0x7f5fac071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:20.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.787+0000 7f5fb0e00640 1 -- 172.21.15.62:0/56045056 shutdown_connections 2024-03-23T11:22:20.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.787+0000 7f5fb0e00640 1 -- 172.21.15.62:0/56045056 wait complete. 2024-03-23T11:22:20.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.789+0000 7f5fb0e00640 1 Processor -- start 2024-03-23T11:22:20.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.789+0000 7f5fb0e00640 1 -- start start 2024-03-23T11:22:20.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.790+0000 7f5fb0e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fac0cf2c0 0x7f5fac15d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:20.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.790+0000 7f5fb0e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fac0d1320 0x7f5fac156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:20.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.790+0000 7f5fb0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fac15ca70 0x7f5fac15f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:20.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.790+0000 7f5fb0e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac15e5a0 con 0x7f5fac15ca70 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.790+0000 7f5fb0e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac15e880 con 0x7f5fac0cf2c0 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.790+0000 7f5fb0e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac15eb60 con 0x7f5fac0d1320 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.790+0000 7f5faa800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fac15ca70 0x7f5fac15f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.790+0000 7f5faa800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fac15ca70 0x7f5fac15f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53410/0 (socket says 172.21.15.62:53410) 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.790+0000 7f5faa800640 1 -- 172.21.15.62:0/38320063 learned_addr learned my addr 172.21.15.62:0/38320063 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.791+0000 7f5fa9e00640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fac0cf2c0 0x7f5fac15d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.791+0000 7f5fa9400640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fac0d1320 0x7f5fac156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.791+0000 7f5faa800640 1 -- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fac0d1320 msgr2=0x7f5fac156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.791+0000 7f5faa800640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fac0d1320 0x7f5fac156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.791+0000 7f5faa800640 1 -- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fac0cf2c0 msgr2=0x7f5fac15d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.791+0000 7f5faa800640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fac0cf2c0 0x7f5fac15d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:20.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.791+0000 7f5faa800640 1 -- 172.21.15.62:0/38320063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98009400 con 0x7f5fac15ca70 2024-03-23T11:22:20.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.791+0000 7f5faa800640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fac15ca70 0x7f5fac15f640 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400d800 tx=0x7f5fa400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:20.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.792+0000 7f5fa2000640 1 -- 172.21.15.62:0/38320063 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa400b6c0 con 0x7f5fac15ca70 2024-03-23T11:22:20.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.792+0000 7f5fa2000640 1 -- 172.21.15.62:0/38320063 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa400b840 con 0x7f5fac15ca70 2024-03-23T11:22:20.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.792+0000 7f5fa2000640 1 -- 172.21.15.62:0/38320063 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa400bbf0 con 0x7f5fac15ca70 2024-03-23T11:22:20.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.792+0000 7f5fb0e00640 1 -- 172.21.15.62:0/38320063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fac15ee40 con 0x7f5fac15ca70 2024-03-23T11:22:20.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.792+0000 7f5fb0e00640 1 -- 172.21.15.62:0/38320063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fac15dda0 con 0x7f5fac15ca70 2024-03-23T11:22:20.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.793+0000 7f5fb0e00640 1 -- 172.21.15.62:0/38320063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fac15f360 con 0x7f5fac15ca70 2024-03-23T11:22:20.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.794+0000 7f5fa2000640 1 -- 172.21.15.62:0/38320063 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5fa400bd50 con 0x7f5fac15ca70 2024-03-23T11:22:20.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.795+0000 7f5fa2000640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f8c06a7d0 0x7f5f8c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:20.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.795+0000 7f5fa2000640 1 -- 172.21.15.62:0/38320063 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f5fa4089180 con 0x7f5fac15ca70 2024-03-23T11:22:20.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.795+0000 7f5fa9e00640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f8c06a7d0 0x7f5f8c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:20.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.797+0000 7f5fa9e00640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f8c06a7d0 0x7f5f8c06cc90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0012e0 tx=0x7f5f9c005eb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:20.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:20.800+0000 7f5fa2000640 1 -- 172.21.15.62:0/38320063 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5fac15f360 con 0x7f5fac15ca70 2024-03-23T11:22:21.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.013+0000 7f5fb0e00640 1 -- 172.21.15.62:0/38320063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5fac15f360 con 0x7f5fac15ca70 2024-03-23T11:22:21.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.553+0000 7f5fa2000640 1 -- 172.21.15.62:0/38320063 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v53) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5fac15f360 con 0x7f5fac15ca70 2024-03-23T11:22:21.551 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' removed 2024-03-23T11:22:21.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.557+0000 7f5fb0e00640 1 -- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f8c06a7d0 msgr2=0x7f5f8c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:21.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.557+0000 7f5fb0e00640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f8c06a7d0 0x7f5f8c06cc90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0012e0 tx=0x7f5f9c005eb0 comp rx=0 tx=0).stop 2024-03-23T11:22:21.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.557+0000 7f5fb0e00640 1 -- 172.21.15.62:0/38320063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fac15ca70 msgr2=0x7f5fac15f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:21.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.557+0000 7f5fb0e00640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fac15ca70 0x7f5fac15f640 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400d800 tx=0x7f5fa400dcc0 comp rx=0 tx=0).stop 2024-03-23T11:22:21.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.557+0000 7f5fb0e00640 1 -- 172.21.15.62:0/38320063 shutdown_connections 2024-03-23T11:22:21.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.557+0000 7f5fb0e00640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5f8c06a7d0 0x7f5f8c06cc90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:21.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.557+0000 7f5fb0e00640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fac15ca70 0x7f5fac15f640 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:21.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.557+0000 7f5fb0e00640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5fac0d1320 0x7f5fac156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:21.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.557+0000 7f5fb0e00640 1 --2- 172.21.15.62:0/38320063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5fac0cf2c0 0x7f5fac15d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:21.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.557+0000 7f5fb0e00640 1 -- 172.21.15.62:0/38320063 >> 172.21.15.62:0/38320063 conn(0x7f5fac071390 msgr2=0x7f5fac15bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:21.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.558+0000 7f5fb0e00640 1 -- 172.21.15.62:0/38320063 shutdown_connections 2024-03-23T11:22:21.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.558+0000 7f5fb0e00640 1 -- 172.21.15.62:0/38320063 wait complete. 2024-03-23T11:22:21.576 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T11:22:21.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.798+0000 7ff18bc00640 1 Processor -- start 2024-03-23T11:22:21.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.798+0000 7ff18bc00640 1 -- start start 2024-03-23T11:22:21.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.800+0000 7ff18bc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18406c300 0x7ff1840ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:21.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.800+0000 7ff18bc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff18406c6e0 0x7ff18406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:21.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.800+0000 7ff18bc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840cbc60 con 0x7ff1840cafa0 2024-03-23T11:22:21.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.800+0000 7ff18bc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1840cbf40 con 0x7ff18406c6e0 2024-03-23T11:22:21.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.800+0000 7ff18bc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840cc220 con 0x7ff18406c300 2024-03-23T11:22:21.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.800+0000 7ff189200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18406c300 0x7ff1840ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:21.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.801+0000 7ff189c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff18406c6e0 0x7ff18406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:21.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.801+0000 7ff189200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18406c300 0x7ff1840ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:45096/0 (socket says 172.21.15.62:45096) 2024-03-23T11:22:21.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.801+0000 7ff189200640 1 -- 172.21.15.62:0/3191266846 learned_addr learned my addr 172.21.15.62:0/3191266846 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:21.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.801+0000 7ff189200640 1 -- 172.21.15.62:0/3191266846 >> v2:172.21.15.120:3300/0 conn(0x7ff18406c6e0 msgr2=0x7ff18406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:21.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.801+0000 7ff189200640 1 --2- 172.21.15.62:0/3191266846 >> v2:172.21.15.120:3300/0 conn(0x7ff18406c6e0 0x7ff18406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:21.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.801+0000 7ff189200640 1 -- 172.21.15.62:0/3191266846 >> v1:172.21.15.120:6789/0 conn(0x7ff1840cafa0 legacy=0x7ff1840cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:21.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.801+0000 7ff189200640 1 -- 172.21.15.62:0/3191266846 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840cc500 con 0x7ff18406c300 2024-03-23T11:22:21.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.803+0000 7ff189200640 1 --2- 172.21.15.62:0/3191266846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18406c300 0x7ff1840ccd00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff174009e50 tx=0x7ff17402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b50b1eed060cfe67 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:21.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.804+0000 7ff183400640 1 -- 172.21.15.62:0/3191266846 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17402cb00 con 0x7ff18406c300 2024-03-23T11:22:21.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.804+0000 7ff183400640 1 -- 172.21.15.62:0/3191266846 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17402cc60 con 0x7ff18406c300 2024-03-23T11:22:21.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.804+0000 7ff183400640 1 -- 172.21.15.62:0/3191266846 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1740343f0 con 0x7ff18406c300 2024-03-23T11:22:21.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.804+0000 7ff18bc00640 1 -- 172.21.15.62:0/3191266846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18406c300 msgr2=0x7ff1840ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:21.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.804+0000 7ff18bc00640 1 --2- 172.21.15.62:0/3191266846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18406c300 0x7ff1840ccd00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff174009e50 tx=0x7ff17402bf60 comp rx=0 tx=0).stop 2024-03-23T11:22:21.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.804+0000 7ff18bc00640 1 -- 172.21.15.62:0/3191266846 shutdown_connections 2024-03-23T11:22:21.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.804+0000 7ff18bc00640 1 --2- 172.21.15.62:0/3191266846 >> v2:172.21.15.120:3300/0 conn(0x7ff18406c6e0 0x7ff18406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:21.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.804+0000 7ff18bc00640 1 --2- 172.21.15.62:0/3191266846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18406c300 0x7ff1840ccd00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:21.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.804+0000 7ff18bc00640 1 -- 172.21.15.62:0/3191266846 >> 172.21.15.62:0/3191266846 conn(0x7ff1840b02e0 msgr2=0x7ff1840b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:21.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.805+0000 7ff18bc00640 1 -- 172.21.15.62:0/3191266846 shutdown_connections 2024-03-23T11:22:21.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.805+0000 7ff18bc00640 1 -- 172.21.15.62:0/3191266846 wait complete. 2024-03-23T11:22:21.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.806+0000 7ff18bc00640 1 Processor -- start 2024-03-23T11:22:21.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.807+0000 7ff18bc00640 1 -- start start 2024-03-23T11:22:21.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.807+0000 7ff18bc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18406c300 0x7ff1840ac820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:21.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.807+0000 7ff18bc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18406c6e0 0x7ff1840aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:21.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.807+0000 7ff18bc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1840cafa0 0x7ff18415d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:21.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.807+0000 7ff18bc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840ace00 con 0x7ff18406c6e0 2024-03-23T11:22:21.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.807+0000 7ff18bc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840ad0e0 con 0x7ff1840cafa0 2024-03-23T11:22:21.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.807+0000 7ff18bc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840ad3c0 con 0x7ff18406c300 2024-03-23T11:22:21.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.808+0000 7ff183e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18406c6e0 0x7ff1840aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:21.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.808+0000 7ff183e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18406c6e0 0x7ff1840aa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53424/0 (socket says 172.21.15.62:53424) 2024-03-23T11:22:21.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.808+0000 7ff183e00640 1 -- 172.21.15.62:0/247694023 learned_addr learned my addr 172.21.15.62:0/247694023 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.808+0000 7ff189c00640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1840cafa0 0x7ff18415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.808+0000 7ff183e00640 1 -- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18406c300 msgr2=0x7ff1840ac820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.808+0000 7ff183e00640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18406c300 0x7ff1840ac820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.808+0000 7ff183e00640 1 -- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1840cafa0 msgr2=0x7ff18415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.808+0000 7ff183e00640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1840cafa0 0x7ff18415d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.808+0000 7ff183e00640 1 -- 172.21.15.62:0/247694023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff174009b30 con 0x7ff18406c6e0 2024-03-23T11:22:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.809+0000 7ff183e00640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18406c6e0 0x7ff1840aa1d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00c400 tx=0x7ff16c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.809+0000 7ff180c00640 1 -- 172.21.15.62:0/247694023 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c0052d0 con 0x7ff18406c6e0 2024-03-23T11:22:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.809+0000 7ff180c00640 1 -- 172.21.15.62:0/247694023 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16c005430 con 0x7ff18406c6e0 2024-03-23T11:22:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.809+0000 7ff180c00640 1 -- 172.21.15.62:0/247694023 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c003830 con 0x7ff18406c6e0 2024-03-23T11:22:21.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.809+0000 7ff18bc00640 1 -- 172.21.15.62:0/247694023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1840ad6a0 con 0x7ff18406c6e0 2024-03-23T11:22:21.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.809+0000 7ff18bc00640 1 -- 172.21.15.62:0/247694023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1840aa710 con 0x7ff18406c6e0 2024-03-23T11:22:21.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.810+0000 7ff18bc00640 1 -- 172.21.15.62:0/247694023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1840adbc0 con 0x7ff18406c6e0 2024-03-23T11:22:21.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.811+0000 7ff180c00640 1 -- 172.21.15.62:0/247694023 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff16c003990 con 0x7ff18406c6e0 2024-03-23T11:22:21.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.812+0000 7ff180c00640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff16006a7a0 0x7ff16006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:21.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.812+0000 7ff180c00640 1 -- 172.21.15.62:0/247694023 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff16c0880c0 con 0x7ff18406c6e0 2024-03-23T11:22:21.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.813+0000 7ff189200640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff16006a7a0 0x7ff16006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.814+0000 7ff189200640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff16006a7a0 0x7ff16006cc60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff174035000 tx=0x7ff17402e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:21.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:21.817+0000 7ff180c00640 1 -- 172.21.15.62:0/247694023 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff1840adbc0 con 0x7ff18406c6e0 2024-03-23T11:22:22.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.017+0000 7ff18bc00640 1 -- 172.21.15.62:0/247694023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff1840adbc0 con 0x7ff18406c6e0 2024-03-23T11:22:22.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.026+0000 7ff180c00640 1 -- 172.21.15.62:0/247694023 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff1840adbc0 con 0x7ff18406c6e0 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 -- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff16006a7a0 msgr2=0x7ff16006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff16006a7a0 0x7ff16006cc60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff174035000 tx=0x7ff17402e000 comp rx=0 tx=0).stop 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 -- 172.21.15.62:0/247694023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18406c6e0 msgr2=0x7ff1840aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18406c6e0 0x7ff1840aa1d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00c400 tx=0x7ff16c00c8c0 comp rx=0 tx=0).stop 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 -- 172.21.15.62:0/247694023 shutdown_connections 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff16006a7a0 0x7ff16006cc60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff18406c6e0 0x7ff1840aa1d0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1840cafa0 0x7ff18415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 --2- 172.21.15.62:0/247694023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff18406c300 0x7ff1840ac820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 -- 172.21.15.62:0/247694023 >> 172.21.15.62:0/247694023 conn(0x7ff1840b02e0 msgr2=0x7ff184072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 -- 172.21.15.62:0/247694023 shutdown_connections 2024-03-23T11:22:22.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.029+0000 7ff18bc00640 1 -- 172.21.15.62:0/247694023 wait complete. 2024-03-23T11:22:22.044 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-03-23T11:22:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.262+0000 7fb1a0800640 1 Processor -- start 2024-03-23T11:22:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.262+0000 7fb1a0800640 1 -- start start 2024-03-23T11:22:22.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.264+0000 7fb1a0800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb19806c300 0x7fb1980ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:22.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.264+0000 7fb1a0800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb1980cafa0 0x7fb1980cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:22.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.264+0000 7fb1a0800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1980cbc60 con 0x7fb19806c6e0 2024-03-23T11:22:22.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.264+0000 7fb1a0800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1980cbf40 con 0x7fb19806c300 2024-03-23T11:22:22.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.264+0000 7fb1a0800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1980cc220 con 0x7fb1980cafa0 2024-03-23T11:22:22.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.264+0000 7fb19de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb19806c300 0x7fb1980ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:22.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.264+0000 7fb19e800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb19806c6e0 0x7fb19806cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55976/0 (socket says 172.21.15.62:55976) 2024-03-23T11:22:22.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.264+0000 7fb19e800640 1 -- 172.21.15.62:0/787782263 learned_addr learned my addr 172.21.15.62:0/787782263 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.265+0000 7fb19d400640 1 --2- 172.21.15.62:0/787782263 >> v2:172.21.15.120:3300/0 conn(0x7fb1980cafa0 0x7fb1980cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.265+0000 7fb19de00640 1 -- 172.21.15.62:0/787782263 >> v2:172.21.15.120:3300/0 conn(0x7fb1980cafa0 msgr2=0x7fb1980cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.265+0000 7fb19de00640 1 --2- 172.21.15.62:0/787782263 >> v2:172.21.15.120:3300/0 conn(0x7fb1980cafa0 0x7fb1980cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.265+0000 7fb19de00640 1 -- 172.21.15.62:0/787782263 >> v1:172.21.15.62:6789/0 conn(0x7fb19806c6e0 legacy=0x7fb19806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.265+0000 7fb19de00640 1 -- 172.21.15.62:0/787782263 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1980cc500 con 0x7fb19806c300 2024-03-23T11:22:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.265+0000 7fb19d400640 1 --2- 172.21.15.62:0/787782263 >> v2:172.21.15.120:3300/0 conn(0x7fb1980cafa0 0x7fb1980cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.266+0000 7fb19de00640 1 --2- 172.21.15.62:0/787782263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb19806c300 0x7fb1980ccd00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb19806e370 tx=0x7fb18c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da1582f40c54348 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.267+0000 7fb19ca00640 1 -- 172.21.15.62:0/787782263 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c031830 con 0x7fb19806c300 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.267+0000 7fb19ca00640 1 -- 172.21.15.62:0/787782263 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18c031990 con 0x7fb19806c300 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.267+0000 7fb19ca00640 1 -- 172.21.15.62:0/787782263 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c031d20 con 0x7fb19806c300 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.267+0000 7fb1a0800640 1 -- 172.21.15.62:0/787782263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb19806c300 msgr2=0x7fb1980ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.267+0000 7fb1a0800640 1 --2- 172.21.15.62:0/787782263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb19806c300 0x7fb1980ccd00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb19806e370 tx=0x7fb18c02b8d0 comp rx=0 tx=0).stop 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.267+0000 7fb1a0800640 1 -- 172.21.15.62:0/787782263 shutdown_connections 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.267+0000 7fb1a0800640 1 --2- 172.21.15.62:0/787782263 >> v2:172.21.15.120:3300/0 conn(0x7fb1980cafa0 0x7fb1980cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.267+0000 7fb1a0800640 1 --2- 172.21.15.62:0/787782263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb19806c300 0x7fb1980ccd00 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.267+0000 7fb1a0800640 1 -- 172.21.15.62:0/787782263 >> 172.21.15.62:0/787782263 conn(0x7fb1980b02e0 msgr2=0x7fb1980b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.267+0000 7fb1a0800640 1 -- 172.21.15.62:0/787782263 shutdown_connections 2024-03-23T11:22:22.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.267+0000 7fb1a0800640 1 -- 172.21.15.62:0/787782263 wait complete. 2024-03-23T11:22:22.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.269+0000 7fb1a0800640 1 Processor -- start 2024-03-23T11:22:22.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.269+0000 7fb1a0800640 1 -- start start 2024-03-23T11:22:22.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.270+0000 7fb1a0800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb19806c300 0x7fb19813feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:22.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.270+0000 7fb1a0800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb19806c6e0 0x7fb1980b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:22.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.270+0000 7fb1a0800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1980cafa0 0x7fb19813bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:22.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.270+0000 7fb1a0800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19813f180 con 0x7fb1980cafa0 2024-03-23T11:22:22.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.270+0000 7fb1a0800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1980af030 con 0x7fb19806c300 2024-03-23T11:22:22.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.270+0000 7fb1a0800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb19813f3d0 con 0x7fb19806c6e0 2024-03-23T11:22:22.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.271+0000 7fb19e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1980cafa0 0x7fb19813bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:22.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.271+0000 7fb19e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1980cafa0 0x7fb19813bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53440/0 (socket says 172.21.15.62:53440) 2024-03-23T11:22:22.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.271+0000 7fb19e800640 1 -- 172.21.15.62:0/2179358427 learned_addr learned my addr 172.21.15.62:0/2179358427 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:22.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.271+0000 7fb19d400640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb19806c6e0 0x7fb1980b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:22.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.271+0000 7fb19de00640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb19806c300 0x7fb19813feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:22.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.271+0000 7fb19e800640 1 -- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb19806c6e0 msgr2=0x7fb1980b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:22.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.271+0000 7fb19e800640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb19806c6e0 0x7fb1980b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:22.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.271+0000 7fb19e800640 1 -- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb19806c300 msgr2=0x7fb19813feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:22.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.271+0000 7fb19e800640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb19806c300 0x7fb19813feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:22.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.271+0000 7fb19e800640 1 -- 172.21.15.62:0/2179358427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c009400 con 0x7fb1980cafa0 2024-03-23T11:22:22.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.272+0000 7fb19e800640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1980cafa0 0x7fb19813bfa0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb19400cbc0 tx=0x7fb19400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:22.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.272+0000 7fb192000640 1 -- 172.21.15.62:0/2179358427 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19400ad40 con 0x7fb1980cafa0 2024-03-23T11:22:22.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.272+0000 7fb192000640 1 -- 172.21.15.62:0/2179358427 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb194012400 con 0x7fb1980cafa0 2024-03-23T11:22:22.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.272+0000 7fb1a0800640 1 -- 172.21.15.62:0/2179358427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19813f6b0 con 0x7fb1980cafa0 2024-03-23T11:22:22.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.272+0000 7fb192000640 1 -- 172.21.15.62:0/2179358427 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19400aea0 con 0x7fb1980cafa0 2024-03-23T11:22:22.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.272+0000 7fb1a0800640 1 -- 172.21.15.62:0/2179358427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1980b2ee0 con 0x7fb1980cafa0 2024-03-23T11:22:22.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.273+0000 7fb1a0800640 1 -- 172.21.15.62:0/2179358427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19813fbd0 con 0x7fb1980cafa0 2024-03-23T11:22:22.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.274+0000 7fb192000640 1 -- 172.21.15.62:0/2179358427 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb194007070 con 0x7fb1980cafa0 2024-03-23T11:22:22.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.275+0000 7fb192000640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb17c06a7a0 0x7fb17c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:22.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.275+0000 7fb192000640 1 -- 172.21.15.62:0/2179358427 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb194088980 con 0x7fb1980cafa0 2024-03-23T11:22:22.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.276+0000 7fb19de00640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb17c06a7a0 0x7fb17c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:22.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.277+0000 7fb19de00640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb17c06a7a0 0x7fb17c06cc60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb18c036000 tx=0x7fb18c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:22.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.280+0000 7fb192000640 1 -- 172.21.15.62:0/2179358427 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb19813fbd0 con 0x7fb1980cafa0 2024-03-23T11:22:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:22.554+0000 7fb1a0800640 1 -- 172.21.15.62:0/2179358427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) v1 -- 0x7fb19813fbd0 con 0x7fb1980cafa0 2024-03-23T11:22:23.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.560+0000 7fb192000640 1 -- 172.21.15.62:0/2179358427 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v191) v1 ==== 149+0+0 (secure 0 0 0) 0x7fb19813fbd0 con 0x7fb1980cafa0 2024-03-23T11:22:23.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.563+0000 7fb1a0800640 1 -- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb17c06a7a0 msgr2=0x7fb17c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:23.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.563+0000 7fb1a0800640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb17c06a7a0 0x7fb17c06cc60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb18c036000 tx=0x7fb18c03c040 comp rx=0 tx=0).stop 2024-03-23T11:22:23.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.563+0000 7fb1a0800640 1 -- 172.21.15.62:0/2179358427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1980cafa0 msgr2=0x7fb19813bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:23.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.563+0000 7fb1a0800640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1980cafa0 0x7fb19813bfa0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb19400cbc0 tx=0x7fb19400a430 comp rx=0 tx=0).stop 2024-03-23T11:22:23.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.564+0000 7fb1a0800640 1 -- 172.21.15.62:0/2179358427 shutdown_connections 2024-03-23T11:22:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.564+0000 7fb1a0800640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb17c06a7a0 0x7fb17c06cc60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.564+0000 7fb1a0800640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1980cafa0 0x7fb19813bfa0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.564+0000 7fb1a0800640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb19806c6e0 0x7fb1980b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.564+0000 7fb1a0800640 1 --2- 172.21.15.62:0/2179358427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb19806c300 0x7fb19813feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.564+0000 7fb1a0800640 1 -- 172.21.15.62:0/2179358427 >> 172.21.15.62:0/2179358427 conn(0x7fb1980b02e0 msgr2=0x7fb19815a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.564+0000 7fb1a0800640 1 -- 172.21.15.62:0/2179358427 shutdown_connections 2024-03-23T11:22:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.564+0000 7fb1a0800640 1 -- 172.21.15.62:0/2179358427 wait complete. 2024-03-23T11:22:23.582 INFO:tasks.cephfs_test_runner:That the MDS can repair an inodes backtrace in the data pool ... ok 2024-03-23T11:22:23.583 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-03-23T11:22:23.583 INFO:tasks.cephfs_test_runner:test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-23T11:22:23.583 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-03-23T11:22:23.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.805+0000 7f436b000640 1 Processor -- start 2024-03-23T11:22:23.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.806+0000 7f436b000640 1 -- start start 2024-03-23T11:22:23.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.807+0000 7f436b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4364160310 0x7f43640ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:23.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.807+0000 7f436b000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f43640ce250 0x7f43640ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:23.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.807+0000 7f436b000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43641608d0 con 0x7f436415c830 2024-03-23T11:22:23.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.807+0000 7f436b000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43640addb0 con 0x7f43640ce250 2024-03-23T11:22:23.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.807+0000 7f436b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436415cc10 con 0x7f4364160310 2024-03-23T11:22:23.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.808+0000 7f436a600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f43640ce250 0x7f43640ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:23.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.808+0000 7f436a600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f43640ce250 0x7f43640ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53454/0 (socket says 172.21.15.62:53454) 2024-03-23T11:22:23.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.808+0000 7f436a600640 1 -- 172.21.15.62:0/2800920157 learned_addr learned my addr 172.21.15.62:0/2800920157 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:23.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.808+0000 7f4369200640 1 --2- 172.21.15.62:0/2800920157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4364160310 0x7f43640ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:23.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.808+0000 7f436a600640 1 -- 172.21.15.62:0/2800920157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4364160310 msgr2=0x7f43640ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:23.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.808+0000 7f436a600640 1 --2- 172.21.15.62:0/2800920157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4364160310 0x7f43640ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:23.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.808+0000 7f436a600640 1 -- 172.21.15.62:0/2800920157 >> v1:172.21.15.120:6789/0 conn(0x7f436415c830 legacy=0x7f43640ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:23.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.809+0000 7f436a600640 1 -- 172.21.15.62:0/2800920157 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4364160bb0 con 0x7f43640ce250 2024-03-23T11:22:23.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.809+0000 7f4369200640 1 --2- 172.21.15.62:0/2800920157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4364160310 0x7f43640ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:22:23.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.810+0000 7f436a600640 1 --2- 172.21.15.62:0/2800920157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43640ce250 0x7f43640ce6e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f435c009e80 tx=0x7f435c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfea677decba3678 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:23.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.810+0000 7f4363e00640 1 -- 172.21.15.62:0/2800920157 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c02cae0 con 0x7f43640ce250 2024-03-23T11:22:23.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.810+0000 7f4363e00640 1 -- 172.21.15.62:0/2800920157 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435c02cc40 con 0x7f43640ce250 2024-03-23T11:22:23.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.810+0000 7f4363e00640 1 -- 172.21.15.62:0/2800920157 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c0343f0 con 0x7f43640ce250 2024-03-23T11:22:23.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.811+0000 7f436b000640 1 -- 172.21.15.62:0/2800920157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43640ce250 msgr2=0x7f43640ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:23.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.811+0000 7f436b000640 1 --2- 172.21.15.62:0/2800920157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43640ce250 0x7f43640ce6e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f435c009e80 tx=0x7f435c02bf40 comp rx=0 tx=0).stop 2024-03-23T11:22:23.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.811+0000 7f436b000640 1 -- 172.21.15.62:0/2800920157 shutdown_connections 2024-03-23T11:22:23.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.811+0000 7f436b000640 1 --2- 172.21.15.62:0/2800920157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43640ce250 0x7f43640ce6e0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:23.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.811+0000 7f436b000640 1 --2- 172.21.15.62:0/2800920157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4364160310 0x7f43640ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:23.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.811+0000 7f436b000640 1 -- 172.21.15.62:0/2800920157 >> 172.21.15.62:0/2800920157 conn(0x7f4364066050 msgr2=0x7f4364069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:23.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.811+0000 7f436b000640 1 -- 172.21.15.62:0/2800920157 shutdown_connections 2024-03-23T11:22:23.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.811+0000 7f436b000640 1 -- 172.21.15.62:0/2800920157 wait complete. 2024-03-23T11:22:23.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.813+0000 7f436b000640 1 Processor -- start 2024-03-23T11:22:23.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.813+0000 7f436b000640 1 -- start start 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f436b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43640ce250 0x7f43641f4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f436b000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436415c830 0x7f43640c3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f436b000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4364160310 0x7f436415f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f436b000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43640c4560 con 0x7f436415c830 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f436b000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43640c4840 con 0x7f4364160310 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f436b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43640c4b20 con 0x7f43640ce250 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f4369200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436415c830 0x7f43640c3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f4369200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436415c830 0x7f43640c3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53462/0 (socket says 172.21.15.62:53462) 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f4369200640 1 -- 172.21.15.62:0/250170754 learned_addr learned my addr 172.21.15.62:0/250170754 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f4369c00640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43640ce250 0x7f43641f4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f436a600640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4364160310 0x7f436415f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f4369200640 1 -- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43640ce250 msgr2=0x7f43641f4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f4369200640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43640ce250 0x7f43641f4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f4369200640 1 -- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4364160310 msgr2=0x7f436415f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f4369200640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4364160310 0x7f436415f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f4369200640 1 -- 172.21.15.62:0/250170754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c009b30 con 0x7f436415c830 2024-03-23T11:22:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.814+0000 7f4369c00640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43640ce250 0x7f43641f4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.815+0000 7f4369200640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436415c830 0x7f43640c3ce0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f435000e730 tx=0x7f435000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.815+0000 7f4361600640 1 -- 172.21.15.62:0/250170754 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4350004230 con 0x7f436415c830 2024-03-23T11:22:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.815+0000 7f4361600640 1 -- 172.21.15.62:0/250170754 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4350004390 con 0x7f436415c830 2024-03-23T11:22:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.815+0000 7f4361600640 1 -- 172.21.15.62:0/250170754 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435000c8f0 con 0x7f436415c830 2024-03-23T11:22:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.815+0000 7f436b000640 1 -- 172.21.15.62:0/250170754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43640c4e00 con 0x7f436415c830 2024-03-23T11:22:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.815+0000 7f436b000640 1 -- 172.21.15.62:0/250170754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43640c4220 con 0x7f436415c830 2024-03-23T11:22:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.816+0000 7f436b000640 1 -- 172.21.15.62:0/250170754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4364066bb0 con 0x7f436415c830 2024-03-23T11:22:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.817+0000 7f4361600640 1 -- 172.21.15.62:0/250170754 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f435000ca50 con 0x7f436415c830 2024-03-23T11:22:23.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.818+0000 7f4361600640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f434406a7a0 0x7f434406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:23.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.818+0000 7f4361600640 1 -- 172.21.15.62:0/250170754 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4350088600 con 0x7f436415c830 2024-03-23T11:22:23.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.819+0000 7f4369c00640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f434406a7a0 0x7f434406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:23.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.820+0000 7f4369c00640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f434406a7a0 0x7f434406cc60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4358000f60 tx=0x7f4358009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:23.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:23.822+0000 7f4361600640 1 -- 172.21.15.62:0/250170754 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4350053db0 con 0x7f436415c830 2024-03-23T11:22:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.097+0000 7f436b000640 1 -- 172.21.15.62:0/250170754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) v1 -- 0x7f43641f1be0 con 0x7f436415c830 2024-03-23T11:22:24.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.564+0000 7f4361600640 1 -- 172.21.15.62:0/250170754 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v192) v1 ==== 172+0+0 (secure 0 0 0) 0x7f43500573d0 con 0x7f436415c830 2024-03-23T11:22:24.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.567+0000 7f436b000640 1 -- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f434406a7a0 msgr2=0x7f434406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:24.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.567+0000 7f436b000640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f434406a7a0 0x7f434406cc60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4358000f60 tx=0x7f4358009040 comp rx=0 tx=0).stop 2024-03-23T11:22:24.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.567+0000 7f436b000640 1 -- 172.21.15.62:0/250170754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436415c830 msgr2=0x7f43640c3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:24.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.567+0000 7f436b000640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436415c830 0x7f43640c3ce0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f435000e730 tx=0x7f435000ebf0 comp rx=0 tx=0).stop 2024-03-23T11:22:24.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.567+0000 7f436b000640 1 -- 172.21.15.62:0/250170754 shutdown_connections 2024-03-23T11:22:24.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.567+0000 7f436b000640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f434406a7a0 0x7f434406cc60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:24.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.567+0000 7f436b000640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436415c830 0x7f43640c3ce0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:24.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.567+0000 7f436b000640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4364160310 0x7f436415f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:24.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.567+0000 7f436b000640 1 --2- 172.21.15.62:0/250170754 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43640ce250 0x7f43641f4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:24.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.568+0000 7f436b000640 1 -- 172.21.15.62:0/250170754 >> 172.21.15.62:0/250170754 conn(0x7f4364066050 msgr2=0x7f436415c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:24.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.568+0000 7f436b000640 1 -- 172.21.15.62:0/250170754 shutdown_connections 2024-03-23T11:22:24.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.568+0000 7f436b000640 1 -- 172.21.15.62:0/250170754 wait complete. 2024-03-23T11:22:24.585 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T11:22:24.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.812+0000 7f0c77e00640 1 Processor -- start 2024-03-23T11:22:24.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.812+0000 7f0c77e00640 1 -- start start 2024-03-23T11:22:24.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.813+0000 7f0c77e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c7815fcf0 0x7f0c780d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:24.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.813+0000 7f0c77e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0c78160420 0x7f0c78160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:24.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.813+0000 7f0c77e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c78160dc0 con 0x7f0c780d06d0 2024-03-23T11:22:24.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.814+0000 7f0c77e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c780d3540 con 0x7f0c78160420 2024-03-23T11:22:24.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.814+0000 7f0c77e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780ce510 con 0x7f0c7815fcf0 2024-03-23T11:22:24.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.814+0000 7f0c76a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c7815fcf0 0x7f0c780d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:24.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.814+0000 7f0c76a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c7815fcf0 0x7f0c780d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:45164/0 (socket says 172.21.15.62:45164) 2024-03-23T11:22:24.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.814+0000 7f0c76a00640 1 -- 172.21.15.62:0/2047141902 learned_addr learned my addr 172.21.15.62:0/2047141902 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:24.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.814+0000 7f0c76000640 1 --2- 172.21.15.62:0/2047141902 >> v2:172.21.15.120:3300/0 conn(0x7f0c78160420 0x7f0c78160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:24.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.815+0000 7f0c76a00640 1 -- 172.21.15.62:0/2047141902 >> v2:172.21.15.120:3300/0 conn(0x7f0c78160420 msgr2=0x7f0c78160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:24.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.815+0000 7f0c76a00640 1 --2- 172.21.15.62:0/2047141902 >> v2:172.21.15.120:3300/0 conn(0x7f0c78160420 0x7f0c78160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:24.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.815+0000 7f0c75600640 1 -- 172.21.15.62:0/2047141902 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875275673 0 0) 0x7f0c78160dc0 con 0x7f0c780d06d0 2024-03-23T11:22:24.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.815+0000 7f0c76a00640 1 -- 172.21.15.62:0/2047141902 >> v1:172.21.15.120:6789/0 conn(0x7f0c780d06d0 legacy=0x7f0c780d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:24.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.815+0000 7f0c76a00640 1 -- 172.21.15.62:0/2047141902 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c780cdc70 con 0x7f0c7815fcf0 2024-03-23T11:22:24.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.817+0000 7f0c76a00640 1 --2- 172.21.15.62:0/2047141902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c7815fcf0 0x7f0c780d3000 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009870 tx=0x7f0c6802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfded1431b812c14 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:24.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.818+0000 7f0c75600640 1 -- 172.21.15.62:0/2047141902 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68031640 con 0x7f0c7815fcf0 2024-03-23T11:22:24.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.818+0000 7f0c75600640 1 -- 172.21.15.62:0/2047141902 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c680317a0 con 0x7f0c7815fcf0 2024-03-23T11:22:24.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.818+0000 7f0c75600640 1 -- 172.21.15.62:0/2047141902 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68031b30 con 0x7f0c7815fcf0 2024-03-23T11:22:24.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.818+0000 7f0c77e00640 1 -- 172.21.15.62:0/2047141902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c7815fcf0 msgr2=0x7f0c780d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:24.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.818+0000 7f0c77e00640 1 --2- 172.21.15.62:0/2047141902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c7815fcf0 0x7f0c780d3000 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009870 tx=0x7f0c6802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:22:24.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.818+0000 7f0c77e00640 1 -- 172.21.15.62:0/2047141902 shutdown_connections 2024-03-23T11:22:24.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.818+0000 7f0c77e00640 1 --2- 172.21.15.62:0/2047141902 >> v2:172.21.15.120:3300/0 conn(0x7f0c78160420 0x7f0c78160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:24.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.818+0000 7f0c77e00640 1 --2- 172.21.15.62:0/2047141902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c7815fcf0 0x7f0c780d3000 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:24.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.818+0000 7f0c77e00640 1 -- 172.21.15.62:0/2047141902 >> 172.21.15.62:0/2047141902 conn(0x7f0c78089f60 msgr2=0x7f0c780d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:24.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.819+0000 7f0c77e00640 1 -- 172.21.15.62:0/2047141902 shutdown_connections 2024-03-23T11:22:24.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.819+0000 7f0c77e00640 1 -- 172.21.15.62:0/2047141902 wait complete. 2024-03-23T11:22:24.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.820+0000 7f0c77e00640 1 Processor -- start 2024-03-23T11:22:24.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.820+0000 7f0c77e00640 1 -- start start 2024-03-23T11:22:24.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.821+0000 7f0c77e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c780d06d0 0x7f0c781591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:24.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.821+0000 7f0c77e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c7815fcf0 0x7f0c780ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:24.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.821+0000 7f0c77e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c78160420 0x7f0c780cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:24.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.821+0000 7f0c77e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7815b3f0 con 0x7f0c7815fcf0 2024-03-23T11:22:24.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.821+0000 7f0c77e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780cb2e0 con 0x7f0c780d06d0 2024-03-23T11:22:24.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.821+0000 7f0c77e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780cb5c0 con 0x7f0c78160420 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.822+0000 7f0c76000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c7815fcf0 0x7f0c780ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.822+0000 7f0c76000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c7815fcf0 0x7f0c780ca770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53464/0 (socket says 172.21.15.62:53464) 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.822+0000 7f0c76000640 1 -- 172.21.15.62:0/371186484 learned_addr learned my addr 172.21.15.62:0/371186484 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.822+0000 7f0c76a00640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c780d06d0 0x7f0c781591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.822+0000 7f0c77400640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c78160420 0x7f0c780cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.822+0000 7f0c76000640 1 -- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c78160420 msgr2=0x7f0c780cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.822+0000 7f0c76000640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c78160420 0x7f0c780cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.822+0000 7f0c76000640 1 -- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c780d06d0 msgr2=0x7f0c781591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.822+0000 7f0c76000640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c780d06d0 0x7f0c781591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.822+0000 7f0c76000640 1 -- 172.21.15.62:0/371186484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c680094d0 con 0x7f0c7815fcf0 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.822+0000 7f0c76a00640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c780d06d0 0x7f0c781591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:22:24.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.823+0000 7f0c76000640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c7815fcf0 0x7f0c780ca770 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000e870 tx=0x7f0c6000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:24.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.823+0000 7f0c6ea00640 1 -- 172.21.15.62:0/371186484 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c600094f0 con 0x7f0c7815fcf0 2024-03-23T11:22:24.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.823+0000 7f0c6ea00640 1 -- 172.21.15.62:0/371186484 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6000c940 con 0x7f0c7815fcf0 2024-03-23T11:22:24.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.823+0000 7f0c77e00640 1 -- 172.21.15.62:0/371186484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c780cb8a0 con 0x7f0c7815fcf0 2024-03-23T11:22:24.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.823+0000 7f0c6ea00640 1 -- 172.21.15.62:0/371186484 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6000ccf0 con 0x7f0c7815fcf0 2024-03-23T11:22:24.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.823+0000 7f0c77e00640 1 -- 172.21.15.62:0/371186484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c780cbe60 con 0x7f0c7815fcf0 2024-03-23T11:22:24.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.824+0000 7f0c77e00640 1 -- 172.21.15.62:0/371186484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c781600d0 con 0x7f0c7815fcf0 2024-03-23T11:22:24.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.826+0000 7f0c6ea00640 1 -- 172.21.15.62:0/371186484 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0c60013470 con 0x7f0c7815fcf0 2024-03-23T11:22:24.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.826+0000 7f0c6ea00640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c5406a7a0 0x7f0c5406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:24.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.827+0000 7f0c6ea00640 1 -- 172.21.15.62:0/371186484 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0c60088ca0 con 0x7f0c7815fcf0 2024-03-23T11:22:24.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.827+0000 7f0c76a00640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c5406a7a0 0x7f0c5406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:24.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.828+0000 7f0c76a00640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c5406a7a0 0x7f0c5406cc60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0c68036000 tx=0x7f0c6803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:24.831+0000 7f0c6ea00640 1 -- 172.21.15.62:0/371186484 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c60054290 con 0x7f0c7815fcf0 2024-03-23T11:22:25.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.029+0000 7f0c77e00640 1 -- 172.21.15.62:0/371186484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0c78066bb0 con 0x7f0c7815fcf0 2024-03-23T11:22:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.037+0000 7f0c6ea00640 1 -- 172.21.15.62:0/371186484 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0c600210c0 con 0x7f0c7815fcf0 2024-03-23T11:22:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.040+0000 7f0c77e00640 1 -- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c5406a7a0 msgr2=0x7f0c5406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.040+0000 7f0c77e00640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c5406a7a0 0x7f0c5406cc60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0c68036000 tx=0x7f0c6803c040 comp rx=0 tx=0).stop 2024-03-23T11:22:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.040+0000 7f0c77e00640 1 -- 172.21.15.62:0/371186484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c7815fcf0 msgr2=0x7f0c780ca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.040+0000 7f0c77e00640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c7815fcf0 0x7f0c780ca770 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000e870 tx=0x7f0c6000ed30 comp rx=0 tx=0).stop 2024-03-23T11:22:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.041+0000 7f0c77e00640 1 -- 172.21.15.62:0/371186484 shutdown_connections 2024-03-23T11:22:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.041+0000 7f0c77e00640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c5406a7a0 0x7f0c5406cc60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.041+0000 7f0c77e00640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c7815fcf0 0x7f0c780ca770 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.041+0000 7f0c77e00640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0c78160420 0x7f0c780cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.041+0000 7f0c77e00640 1 --2- 172.21.15.62:0/371186484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c780d06d0 0x7f0c781591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.041+0000 7f0c77e00640 1 -- 172.21.15.62:0/371186484 >> 172.21.15.62:0/371186484 conn(0x7f0c78089f60 msgr2=0x7f0c780b6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.041+0000 7f0c77e00640 1 -- 172.21.15.62:0/371186484 shutdown_connections 2024-03-23T11:22:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.041+0000 7f0c77e00640 1 -- 172.21.15.62:0/371186484 wait complete. 2024-03-23T11:22:25.059 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:22:25.059 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:22:25.176 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:22:25.294 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:22:25.294 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:22:25.294 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:22:25.354 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:22:25.355 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:22:25.355 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:22:25.409 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:22:25.409 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:22:25.409 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:22:25.469 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:22:25.469 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:22:25.469 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:22:25.524 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:22:25.524 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T11:22:25.525 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:22:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.756+0000 7f6997e00640 1 Processor -- start 2024-03-23T11:22:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.756+0000 7f6997e00640 1 -- start start 2024-03-23T11:22:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.758+0000 7f6997e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6998159490 0x7f699806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.758+0000 7f6997e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f699815da90 0x7f69980cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.758+0000 7f6997e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69980ce8b0 con 0x7f69980ce420 2024-03-23T11:22:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.758+0000 7f6997e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69980ced90 con 0x7f699815da90 2024-03-23T11:22:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.758+0000 7f6997e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699806e3e0 con 0x7f6998159490 2024-03-23T11:22:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.758+0000 7f6996000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f699815da90 0x7f69980cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.758+0000 7f6997400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f69980ce420 0x7f69980b26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55984/0 (socket says 172.21.15.62:55984) 2024-03-23T11:22:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.758+0000 7f6996000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f699815da90 0x7f69980cdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53476/0 (socket says 172.21.15.62:53476) 2024-03-23T11:22:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.758+0000 7f6997400640 1 -- 172.21.15.62:0/4025218509 learned_addr learned my addr 172.21.15.62:0/4025218509 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:25.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.759+0000 7f6996a00640 1 --2- 172.21.15.62:0/4025218509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6998159490 0x7f699806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:25.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.759+0000 7f6996000640 1 -- 172.21.15.62:0/4025218509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6998159490 msgr2=0x7f699806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:25.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.759+0000 7f6996000640 1 --2- 172.21.15.62:0/4025218509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6998159490 0x7f699806cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:25.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.759+0000 7f6996000640 1 -- 172.21.15.62:0/4025218509 >> v1:172.21.15.62:6789/0 conn(0x7f69980ce420 legacy=0x7f69980b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:25.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.759+0000 7f6996000640 1 -- 172.21.15.62:0/4025218509 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699806c1c0 con 0x7f699815da90 2024-03-23T11:22:25.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.759+0000 7f6996a00640 1 --2- 172.21.15.62:0/4025218509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6998159490 0x7f699806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:25.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.760+0000 7f6996000640 1 --2- 172.21.15.62:0/4025218509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f699815da90 0x7f69980cdee0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6988009d90 tx=0x7f698802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f557a4acc4a1c79 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:25.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.760+0000 7f6995600640 1 -- 172.21.15.62:0/4025218509 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698802cbc0 con 0x7f699815da90 2024-03-23T11:22:25.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.760+0000 7f6995600640 1 -- 172.21.15.62:0/4025218509 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698802cd20 con 0x7f699815da90 2024-03-23T11:22:25.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.760+0000 7f6995600640 1 -- 172.21.15.62:0/4025218509 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6988034660 con 0x7f699815da90 2024-03-23T11:22:25.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.761+0000 7f6997e00640 1 -- 172.21.15.62:0/4025218509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f699815da90 msgr2=0x7f69980cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:25.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.761+0000 7f6997e00640 1 --2- 172.21.15.62:0/4025218509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f699815da90 0x7f69980cdee0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6988009d90 tx=0x7f698802be10 comp rx=0 tx=0).stop 2024-03-23T11:22:25.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.761+0000 7f6997e00640 1 -- 172.21.15.62:0/4025218509 shutdown_connections 2024-03-23T11:22:25.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.761+0000 7f6997e00640 1 --2- 172.21.15.62:0/4025218509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f699815da90 0x7f69980cdee0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:25.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.761+0000 7f6997e00640 1 --2- 172.21.15.62:0/4025218509 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6998159490 0x7f699806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:25.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.761+0000 7f6997e00640 1 -- 172.21.15.62:0/4025218509 >> 172.21.15.62:0/4025218509 conn(0x7f6998066050 msgr2=0x7f6998069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:25.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.761+0000 7f6997e00640 1 -- 172.21.15.62:0/4025218509 shutdown_connections 2024-03-23T11:22:25.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.761+0000 7f6997e00640 1 -- 172.21.15.62:0/4025218509 wait complete. 2024-03-23T11:22:25.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.763+0000 7f6997e00640 1 Processor -- start 2024-03-23T11:22:25.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.763+0000 7f6997e00640 1 -- start start 2024-03-23T11:22:25.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.764+0000 7f6997e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69980ce420 0x7f6998151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:25.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.764+0000 7f6997e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6998159490 0x7f6998151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:25.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.764+0000 7f6997e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f699815da90 0x7f699815ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:25.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.764+0000 7f6997e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69980ca210 con 0x7f699815da90 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.764+0000 7f6997e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699815acd0 con 0x7f6998159490 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.764+0000 7f6997e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699815afb0 con 0x7f69980ce420 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.764+0000 7f6997400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f699815da90 0x7f699815ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.764+0000 7f6997400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f699815da90 0x7f699815ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53490/0 (socket says 172.21.15.62:53490) 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.764+0000 7f6997400640 1 -- 172.21.15.62:0/282480627 learned_addr learned my addr 172.21.15.62:0/282480627 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.765+0000 7f6996a00640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69980ce420 0x7f6998151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.765+0000 7f6996000640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6998159490 0x7f6998151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.765+0000 7f6997400640 1 -- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69980ce420 msgr2=0x7f6998151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.765+0000 7f6997400640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69980ce420 0x7f6998151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.765+0000 7f6997400640 1 -- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6998159490 msgr2=0x7f6998151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.765+0000 7f6997400640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6998159490 0x7f6998151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.765+0000 7f6997400640 1 -- 172.21.15.62:0/282480627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6988009a70 con 0x7f699815da90 2024-03-23T11:22:25.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.765+0000 7f6997400640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f699815da90 0x7f699815ba90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f699000c370 tx=0x7f699000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:25.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.765+0000 7f698f400640 1 -- 172.21.15.62:0/282480627 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6990005ce0 con 0x7f699815da90 2024-03-23T11:22:25.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.766+0000 7f698f400640 1 -- 172.21.15.62:0/282480627 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6990005e40 con 0x7f699815da90 2024-03-23T11:22:25.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.766+0000 7f698f400640 1 -- 172.21.15.62:0/282480627 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69900072d0 con 0x7f699815da90 2024-03-23T11:22:25.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.766+0000 7f6997e00640 1 -- 172.21.15.62:0/282480627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699815b290 con 0x7f699815da90 2024-03-23T11:22:25.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.766+0000 7f6997e00640 1 -- 172.21.15.62:0/282480627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69980c9e30 con 0x7f699815da90 2024-03-23T11:22:25.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.767+0000 7f6997e00640 1 -- 172.21.15.62:0/282480627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699815b7b0 con 0x7f699815da90 2024-03-23T11:22:25.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.768+0000 7f698f400640 1 -- 172.21.15.62:0/282480627 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f69900074f0 con 0x7f699815da90 2024-03-23T11:22:25.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.769+0000 7f698f400640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f697406a7a0 0x7f697406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:25.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.769+0000 7f698f400640 1 -- 172.21.15.62:0/282480627 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6990087ee0 con 0x7f699815da90 2024-03-23T11:22:25.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.770+0000 7f6996a00640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f697406a7a0 0x7f697406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:25.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.771+0000 7f6996a00640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f697406a7a0 0x7f697406cc60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f698000a990 tx=0x7f6980008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:25.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:25.775+0000 7f698f400640 1 -- 172.21.15.62:0/282480627 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f699815b7b0 con 0x7f699815da90 2024-03-23T11:22:26.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.015+0000 7f6997e00640 1 -- 172.21.15.62:0/282480627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f699815b7b0 con 0x7f699815da90 2024-03-23T11:22:26.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.016+0000 7f698f400640 1 -- 172.21.15.62:0/282480627 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f6990017de0 con 0x7f699815da90 2024-03-23T11:22:26.015 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:26.015 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3787987198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3787987198},{"type":"v1","addr":"172.21.15.120:6835","nonce":3787987198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5215,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2749498408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2749498408},{"type":"v1","addr":"172.21.15.120:6837","nonce":2749498408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2325988870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2325988870},{"type":"v1","addr":"172.21.15.62:6833","nonce":2325988870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6003,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/1567562724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":1567562724},{"type":"v1","addr":"172.21.15.62:6835","nonce":1567562724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[]} 2024-03-23T11:22:26.015 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-03-23T11:22:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.019+0000 7f6997e00640 1 -- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f697406a7a0 msgr2=0x7f697406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.019+0000 7f6997e00640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f697406a7a0 0x7f697406cc60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f698000a990 tx=0x7f6980008040 comp rx=0 tx=0).stop 2024-03-23T11:22:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.019+0000 7f6997e00640 1 -- 172.21.15.62:0/282480627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f699815da90 msgr2=0x7f699815ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.019+0000 7f6997e00640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f699815da90 0x7f699815ba90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f699000c370 tx=0x7f699000c830 comp rx=0 tx=0).stop 2024-03-23T11:22:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.019+0000 7f6997e00640 1 -- 172.21.15.62:0/282480627 shutdown_connections 2024-03-23T11:22:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.019+0000 7f6997e00640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f697406a7a0 0x7f697406cc60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.019+0000 7f6997e00640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f699815da90 0x7f699815ba90 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.019+0000 7f6997e00640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6998159490 0x7f6998151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.019+0000 7f6997e00640 1 --2- 172.21.15.62:0/282480627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69980ce420 0x7f6998151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:26.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.020+0000 7f6997e00640 1 -- 172.21.15.62:0/282480627 >> 172.21.15.62:0/282480627 conn(0x7f6998066050 msgr2=0x7f69980ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:26.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.020+0000 7f6997e00640 1 -- 172.21.15.62:0/282480627 shutdown_connections 2024-03-23T11:22:26.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.020+0000 7f6997e00640 1 -- 172.21.15.62:0/282480627 wait complete. 2024-03-23T11:22:26.039 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T11:22:26.039 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T11:22:26.039 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T11:22:26.039 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:22:26.040 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T11:22:26.040 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T11:22:26.040 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T11:22:26.040 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:22:26.040 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T11:22:26.040 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T11:22:26.040 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T11:22:26.041 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:22:26.041 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T11:22:26.041 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T11:22:26.041 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T11:22:26.041 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:22:26.056 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T11:22:26.056 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T11:22:26.057 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T11:22:26.057 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T11:22:26.067 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T11:22:26.067 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T11:22:26.069 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T11:22:26.069 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T11:22:26.104 INFO:tasks.ceph.mds.c:Started 2024-03-23T11:22:26.105 DEBUG:teuthology.parallel:result is None 2024-03-23T11:22:26.111 INFO:tasks.ceph.mds.a:Started 2024-03-23T11:22:26.111 DEBUG:teuthology.parallel:result is None 2024-03-23T11:22:26.115 INFO:tasks.ceph.mds.b:Started 2024-03-23T11:22:26.115 DEBUG:teuthology.parallel:result is None 2024-03-23T11:22:26.121 INFO:tasks.ceph.mds.d:Started 2024-03-23T11:22:26.121 DEBUG:teuthology.parallel:result is None 2024-03-23T11:22:26.122 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T11:22:26.261 INFO:tasks.ceph.mds.c.smithi062.stdout:starting mds.c at 2024-03-23T11:22:26.262 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:22:26.277 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-23T11:22:26.282 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-23T11:22:26.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.317+0000 7fbd12600640 1 Processor -- start 2024-03-23T11:22:26.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.317+0000 7fbd12600640 1 -- start start 2024-03-23T11:22:26.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.318+0000 7fbd12600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbd0c15c960 0x7fbd0c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:26.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.318+0000 7fbd12600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0c0d13b0 0x7fbd0c0b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:26.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.318+0000 7fbd12600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c0d0810 con 0x7fbd0c0cd260 2024-03-23T11:22:26.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.318+0000 7fbd12600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd0c0cd9d0 con 0x7fbd0c15c960 2024-03-23T11:22:26.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.318+0000 7fbd12600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c0cdcb0 con 0x7fbd0c0d13b0 2024-03-23T11:22:26.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.318+0000 7fbd11200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbd0c15c960 0x7fbd0c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:26.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.318+0000 7fbd11c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0c0d13b0 0x7fbd0c0b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:26.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.319+0000 7fbd11200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbd0c15c960 0x7fbd0c0cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:53216/0 (socket says 172.21.15.62:53216) 2024-03-23T11:22:26.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.319+0000 7fbd11200640 1 -- 172.21.15.62:0/3774618378 learned_addr learned my addr 172.21.15.62:0/3774618378 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:26.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.319+0000 7fbd0be00640 1 --1- 172.21.15.62:0/3774618378 >> v1:172.21.15.120:6789/0 conn(0x7fbd0c0cd260 0x7fbd0c0d0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:53410/0 (socket says 172.21.15.62:53410) 2024-03-23T11:22:26.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.319+0000 7fbd11200640 1 -- 172.21.15.62:0/3774618378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0c0d13b0 msgr2=0x7fbd0c0b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:26.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.319+0000 7fbd11200640 1 --2- 172.21.15.62:0/3774618378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0c0d13b0 0x7fbd0c0b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:26.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.319+0000 7fbd11200640 1 -- 172.21.15.62:0/3774618378 >> v1:172.21.15.120:6789/0 conn(0x7fbd0c0cd260 legacy=0x7fbd0c0d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:26.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.319+0000 7fbd11200640 1 -- 172.21.15.62:0/3774618378 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c0cdf90 con 0x7fbd0c15c960 2024-03-23T11:22:26.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.319+0000 7fbd11c00640 1 --2- 172.21.15.62:0/3774618378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0c0d13b0 0x7fbd0c0b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:22:26.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.320+0000 7fbd11200640 1 --2- 172.21.15.62:0/3774618378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd0c15c960 0x7fbd0c0cf790 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009870 tx=0x7fbd0002ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9321fbea6bbe33c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:26.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.320+0000 7fbd0b400640 1 -- 172.21.15.62:0/3774618378 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0002f830 con 0x7fbd0c15c960 2024-03-23T11:22:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.320+0000 7fbd0b400640 1 -- 172.21.15.62:0/3774618378 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0002f990 con 0x7fbd0c15c960 2024-03-23T11:22:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.320+0000 7fbd0b400640 1 -- 172.21.15.62:0/3774618378 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0002fd20 con 0x7fbd0c15c960 2024-03-23T11:22:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.321+0000 7fbd12600640 1 -- 172.21.15.62:0/3774618378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd0c15c960 msgr2=0x7fbd0c0cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.321+0000 7fbd12600640 1 --2- 172.21.15.62:0/3774618378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd0c15c960 0x7fbd0c0cf790 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009870 tx=0x7fbd0002ba10 comp rx=0 tx=0).stop 2024-03-23T11:22:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.321+0000 7fbd12600640 1 -- 172.21.15.62:0/3774618378 shutdown_connections 2024-03-23T11:22:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.321+0000 7fbd12600640 1 --2- 172.21.15.62:0/3774618378 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0c0d13b0 0x7fbd0c0b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.321+0000 7fbd12600640 1 --2- 172.21.15.62:0/3774618378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd0c15c960 0x7fbd0c0cf790 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.321+0000 7fbd12600640 1 -- 172.21.15.62:0/3774618378 >> 172.21.15.62:0/3774618378 conn(0x7fbd0c089f60 msgr2=0x7fbd0c0d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.321+0000 7fbd12600640 1 -- 172.21.15.62:0/3774618378 shutdown_connections 2024-03-23T11:22:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.321+0000 7fbd12600640 1 -- 172.21.15.62:0/3774618378 wait complete. 2024-03-23T11:22:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.323+0000 7fbd12600640 1 Processor -- start 2024-03-23T11:22:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.323+0000 7fbd12600640 1 -- start start 2024-03-23T11:22:26.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.324+0000 7fbd12600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd0c0cd260 0x7fbd0c1f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:26.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.324+0000 7fbd12600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0c0d13b0 0x7fbd0c15e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:26.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.324+0000 7fbd12600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd0c15c960 0x7fbd0c15e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:26.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.324+0000 7fbd12600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c14d720 con 0x7fbd0c0cd260 2024-03-23T11:22:26.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.324+0000 7fbd12600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c1f2260 con 0x7fbd0c15c960 2024-03-23T11:22:26.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.324+0000 7fbd12600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c09da70 con 0x7fbd0c0d13b0 2024-03-23T11:22:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.324+0000 7fbd11200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd0c0cd260 0x7fbd0c1f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.324+0000 7fbd11200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd0c0cd260 0x7fbd0c1f1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53530/0 (socket says 172.21.15.62:53530) 2024-03-23T11:22:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.324+0000 7fbd11200640 1 -- 172.21.15.62:0/3494259000 learned_addr learned my addr 172.21.15.62:0/3494259000 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.325+0000 7fbd0be00640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0c0d13b0 0x7fbd0c15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.325+0000 7fbd11c00640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd0c15c960 0x7fbd0c15e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.325+0000 7fbd11200640 1 -- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0c0d13b0 msgr2=0x7fbd0c15e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.325+0000 7fbd11200640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0c0d13b0 0x7fbd0c15e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.325+0000 7fbd11200640 1 -- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd0c15c960 msgr2=0x7fbd0c15e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.325+0000 7fbd11200640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd0c15c960 0x7fbd0c15e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.325+0000 7fbd11200640 1 -- 172.21.15.62:0/3494259000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd04009a70 con 0x7fbd0c0cd260 2024-03-23T11:22:26.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.325+0000 7fbd11200640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd0c0cd260 0x7fbd0c1f1d20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fbd00000c00 tx=0x7fbd00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:26.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.325+0000 7fbd08c00640 1 -- 172.21.15.62:0/3494259000 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00037c40 con 0x7fbd0c0cd260 2024-03-23T11:22:26.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.326+0000 7fbd08c00640 1 -- 172.21.15.62:0/3494259000 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd00037dc0 con 0x7fbd0c0cd260 2024-03-23T11:22:26.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.326+0000 7fbd12600640 1 -- 172.21.15.62:0/3494259000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd000094d0 con 0x7fbd0c0cd260 2024-03-23T11:22:26.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.326+0000 7fbd08c00640 1 -- 172.21.15.62:0/3494259000 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00041640 con 0x7fbd0c0cd260 2024-03-23T11:22:26.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.326+0000 7fbd12600640 1 -- 172.21.15.62:0/3494259000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0c14cd80 con 0x7fbd0c0cd260 2024-03-23T11:22:26.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.327+0000 7fbd12600640 1 -- 172.21.15.62:0/3494259000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0c066bb0 con 0x7fbd0c0cd260 2024-03-23T11:22:26.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.328+0000 7fbd08c00640 1 -- 172.21.15.62:0/3494259000 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbd000417a0 con 0x7fbd0c0cd260 2024-03-23T11:22:26.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.328+0000 7fbd08c00640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcec06a7a0 0x7fbcec06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:26.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.329+0000 7fbd08c00640 1 -- 172.21.15.62:0/3494259000 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbd0003c070 con 0x7fbd0c0cd260 2024-03-23T11:22:26.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.329+0000 7fbd0be00640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcec06a7a0 0x7fbcec06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:26.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.330+0000 7fbd0be00640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcec06a7a0 0x7fbcec06cc60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0b03a0 tx=0x7fbcf8007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:26.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.333+0000 7fbd08c00640 1 -- 172.21.15.62:0/3494259000 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd00076d90 con 0x7fbd0c0cd260 2024-03-23T11:22:26.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:26.545+0000 7fbd12600640 1 -- 172.21.15.62:0/3494259000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbd0c0d0a80 con 0x7fbd0c0cd260 2024-03-23T11:22:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.310+0000 7fbd08c00640 1 -- 172.21.15.62:0/3494259000 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v58) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbd0007a3b0 con 0x7fbd0c0cd260 2024-03-23T11:22:27.309 INFO:teuthology.orchestra.run.smithi062.stderr: removed all blocklist entries 2024-03-23T11:22:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.313+0000 7fbd12600640 1 -- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcec06a7a0 msgr2=0x7fbcec06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.313+0000 7fbd12600640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcec06a7a0 0x7fbcec06cc60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0b03a0 tx=0x7fbcf8007e30 comp rx=0 tx=0).stop 2024-03-23T11:22:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.313+0000 7fbd12600640 1 -- 172.21.15.62:0/3494259000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd0c0cd260 msgr2=0x7fbd0c1f1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.313+0000 7fbd12600640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd0c0cd260 0x7fbd0c1f1d20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fbd00000c00 tx=0x7fbd00004730 comp rx=0 tx=0).stop 2024-03-23T11:22:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.314+0000 7fbd12600640 1 -- 172.21.15.62:0/3494259000 shutdown_connections 2024-03-23T11:22:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.314+0000 7fbd12600640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcec06a7a0 0x7fbcec06cc60 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.314+0000 7fbd12600640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd0c15c960 0x7fbd0c15e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.314+0000 7fbd12600640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd0c0d13b0 0x7fbd0c15e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.314+0000 7fbd12600640 1 --2- 172.21.15.62:0/3494259000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd0c0cd260 0x7fbd0c1f1d20 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.314+0000 7fbd12600640 1 -- 172.21.15.62:0/3494259000 >> 172.21.15.62:0/3494259000 conn(0x7fbd0c089f60 msgr2=0x7fbd0c0af270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.314+0000 7fbd12600640 1 -- 172.21.15.62:0/3494259000 shutdown_connections 2024-03-23T11:22:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.314+0000 7fbd12600640 1 -- 172.21.15.62:0/3494259000 wait complete. 2024-03-23T11:22:27.334 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T11:22:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.557+0000 7f729a000640 1 Processor -- start 2024-03-23T11:22:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.558+0000 7f729a000640 1 -- start start 2024-03-23T11:22:27.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.559+0000 7f729a000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f72940cf650 0x7f72940cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:27.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.559+0000 7f729a000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729406bfc0 con 0x7f729415cdc0 2024-03-23T11:22:27.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.559+0000 7f729a000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72940b5260 con 0x7f729415c9e0 2024-03-23T11:22:27.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.559+0000 7f729a000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72940d0000 con 0x7f72940cf650 2024-03-23T11:22:27.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.560+0000 7f7293e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f72940cf650 0x7f72940cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:27.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.560+0000 7f7293e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f72940cf650 0x7f72940cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44880/0 (socket says 172.21.15.62:44880) 2024-03-23T11:22:27.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.560+0000 7f7293e00640 1 -- 172.21.15.62:0/2388793207 learned_addr learned my addr 172.21.15.62:0/2388793207 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:27.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.560+0000 7f7293e00640 1 -- 172.21.15.62:0/2388793207 >> v1:172.21.15.120:6789/0 conn(0x7f729415c9e0 legacy=0x7f72941607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:27.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.560+0000 7f7293e00640 1 -- 172.21.15.62:0/2388793207 >> v1:172.21.15.62:6789/0 conn(0x7f729415cdc0 legacy=0x7f72940d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:27.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.560+0000 7f7293e00640 1 -- 172.21.15.62:0/2388793207 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72940d2630 con 0x7f72940cf650 2024-03-23T11:22:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.561+0000 7f7293e00640 1 --2- 172.21.15.62:0/2388793207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72940cf650 0x7f72940cfac0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f728c009870 tx=0x7f728c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f9ba03c46246f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.562+0000 7f7292000640 1 -- 172.21.15.62:0/2388793207 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728c030860 con 0x7f72940cf650 2024-03-23T11:22:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.562+0000 7f7292000640 1 -- 172.21.15.62:0/2388793207 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728c0309c0 con 0x7f72940cf650 2024-03-23T11:22:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.562+0000 7f7292000640 1 -- 172.21.15.62:0/2388793207 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728c030d50 con 0x7f72940cf650 2024-03-23T11:22:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.562+0000 7f729a000640 1 -- 172.21.15.62:0/2388793207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72940cf650 msgr2=0x7f72940cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.562+0000 7f729a000640 1 --2- 172.21.15.62:0/2388793207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72940cf650 0x7f72940cfac0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f728c009870 tx=0x7f728c02b9a0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.562+0000 7f729a000640 1 -- 172.21.15.62:0/2388793207 shutdown_connections 2024-03-23T11:22:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.562+0000 7f729a000640 1 --2- 172.21.15.62:0/2388793207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72940cf650 0x7f72940cfac0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.562+0000 7f729a000640 1 -- 172.21.15.62:0/2388793207 >> 172.21.15.62:0/2388793207 conn(0x7f7294071570 msgr2=0x7f7294071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.562+0000 7f729a000640 1 -- 172.21.15.62:0/2388793207 shutdown_connections 2024-03-23T11:22:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.562+0000 7f729a000640 1 -- 172.21.15.62:0/2388793207 wait complete. 2024-03-23T11:22:27.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.564+0000 7f729a000640 1 Processor -- start 2024-03-23T11:22:27.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.564+0000 7f729a000640 1 -- start start 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.565+0000 7f729a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72940cf650 0x7f72941f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.565+0000 7f729a000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f729415c9e0 0x7f72940c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.565+0000 7f729a000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f729415cdc0 0x7f7294158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.565+0000 7f729a000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72940ca7e0 con 0x7f72940cf650 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.565+0000 7f729a000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72940c9b60 con 0x7f729415c9e0 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.565+0000 7f729a000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72940c9e40 con 0x7f729415cdc0 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.565+0000 7f7293400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72940cf650 0x7f72941f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.565+0000 7f7293400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72940cf650 0x7f72941f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44896/0 (socket says 172.21.15.62:44896) 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.565+0000 7f7293400640 1 -- 172.21.15.62:0/1166737418 learned_addr learned my addr 172.21.15.62:0/1166737418 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.566+0000 7f7292a00640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f729415c9e0 0x7f72940c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.566+0000 7f7293e00640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f729415cdc0 0x7f7294158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.566+0000 7f7293400640 1 -- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f729415cdc0 msgr2=0x7f7294158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.566+0000 7f7293400640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f729415cdc0 0x7f7294158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.566+0000 7f7293400640 1 -- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f729415c9e0 msgr2=0x7f72940c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.566+0000 7f7293400640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f729415c9e0 0x7f72940c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.566+0000 7f7293400640 1 -- 172.21.15.62:0/1166737418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c0094d0 con 0x7f72940cf650 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.566+0000 7f7293e00640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f729415cdc0 0x7f7294158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.566+0000 7f7292a00640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f729415c9e0 0x7f72940c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.566+0000 7f7293400640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72940cf650 0x7f72941f3210 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f728400c7c0 tx=0x7f728400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.566+0000 7f728b400640 1 -- 172.21.15.62:0/1166737418 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728400aa50 con 0x7f72940cf650 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.567+0000 7f728b400640 1 -- 172.21.15.62:0/1166737418 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728400abb0 con 0x7f72940cf650 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.567+0000 7f728b400640 1 -- 172.21.15.62:0/1166737418 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7284012420 con 0x7f72940cf650 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.567+0000 7f729a000640 1 -- 172.21.15.62:0/1166737418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72940ca120 con 0x7f72940cf650 2024-03-23T11:22:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.567+0000 7f729a000640 1 -- 172.21.15.62:0/1166737418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72940c72b0 con 0x7f72940cf650 2024-03-23T11:22:27.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.568+0000 7f729a000640 1 -- 172.21.15.62:0/1166737418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7294066c90 con 0x7f72940cf650 2024-03-23T11:22:27.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.569+0000 7f728b400640 1 -- 172.21.15.62:0/1166737418 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7284012580 con 0x7f72940cf650 2024-03-23T11:22:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.570+0000 7f728b400640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f727006a7a0 0x7f727006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.570+0000 7f728b400640 1 -- 172.21.15.62:0/1166737418 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7284088d00 con 0x7f72940cf650 2024-03-23T11:22:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.571+0000 7f7292a00640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f727006a7a0 0x7f727006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:27.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.572+0000 7f7292a00640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f727006a7a0 0x7f727006cc60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f727c000f60 tx=0x7f727c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.575+0000 7f728b400640 1 -- 172.21.15.62:0/1166737418 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7284054460 con 0x7f72940cf650 2024-03-23T11:22:27.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.850+0000 7f729a000640 1 -- 172.21.15.62:0/1166737418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7294071c10 con 0x7f72940cf650 2024-03-23T11:22:27.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.851+0000 7f728b400640 1 -- 172.21.15.62:0/1166737418 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7284057a80 con 0x7f72940cf650 2024-03-23T11:22:27.850 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:27.850 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:22:27.850 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_dump": [ 2024-03-23T11:22:27.850 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.850 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.a", 2024-03-23T11:22:27.850 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lYNxYERAAPVHWwRHM5V7OejL294b2zg==", 2024-03-23T11:22:27.850 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.850 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:22:27.850 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:22:27.851 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:22:27.851 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.851 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.851 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.851 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.851 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.b", 2024-03-23T11:22:27.851 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lmOUANBAAifuZ0DUPD9az/mturTndSA==", 2024-03-23T11:22:27.851 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.851 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:22:27.851 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:22:27.851 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.c", 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5ltqrjIhAA9+f82fvlrFlIEXqswPE0ig==", 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.852 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.853 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.853 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.d", 2024-03-23T11:22:27.853 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5l7ly4CBAA8+oim0z+bbljfP0VH7XSaw==", 2024-03-23T11:22:27.853 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.853 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:22:27.853 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:22:27.853 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:22:27.853 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.853 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.853 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.853 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.854 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.0", 2024-03-23T11:22:27.854 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBiuv5lRfaQBRAAvrtCZXwt00PUQfCqqKBzMQ==", 2024-03-23T11:22:27.854 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.854 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:22:27.854 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:22:27.854 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.854 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.854 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.854 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.854 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.1", 2024-03-23T11:22:27.854 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBluv5lzwS/IBAAmYU8yMLtCUJNr/AyE0MJDA==", 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.2", 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBouv5lyAy1OhAAJoUsg6DFpEPxAcH5oI2exQ==", 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:22:27.855 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.3", 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBsuv5lb0YcGBAA220D0uRhAZpG4w97wqsB5A==", 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.856 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.4", 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBzuv5lC8RIGhAAH+5HaeCgQhrQMrYm9iWaYA==", 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.5", 2024-03-23T11:22:27.857 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB2uv5lYi7ZIhAAMNIm/jPckCAbX9oEx5Li/Q==", 2024-03-23T11:22:27.858 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.858 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:22:27.858 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:22:27.858 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.858 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.858 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.858 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.858 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.6", 2024-03-23T11:22:27.858 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB5uv5lgoRtLBAAFT7uJtyqHtHAegOYyDH/WQ==", 2024-03-23T11:22:27.858 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.858 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.7", 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB9uv5l/dzrCBAA3jsua0LvBPI7I551/rRHrg==", 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.859 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.0", 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5ltavQFhAAvV2RWuHOPK2T9Q7gWkYA4w==", 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow r", 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.860 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.1", 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5luQ/zLRAAwjDtV64TCKnQliPq4iUAYg==", 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.2", 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lDRG1IhAAjVaXRR30G1985oMou7Ambg==", 2024-03-23T11:22:27.861 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.3", 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lomZyOBAA8BMIBD+Zhtp6Fa2RcEXCJg==", 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:22:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.admin", 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lNtZHBhAAhiAC1rdmANClOSwzrX9iSQ==", 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.863 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.864 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.864 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.864 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.x", 2024-03-23T11:22:27.864 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lT6+KBRAAl5TZsJGNr4gAB2cEDZo/hQ==", 2024-03-23T11:22:27.864 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.864 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:22:27.864 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:22:27.864 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.864 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.864 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:27.864 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:27.865 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.y", 2024-03-23T11:22:27.865 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lqnG8NRAAvyswMz5kpW0rGjzWYRTAAg==", 2024-03-23T11:22:27.865 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:22:27.865 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:22:27.865 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:22:27.865 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:22:27.865 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.865 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:27.865 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-03-23T11:22:27.865 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:22:27.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.854+0000 7f729a000640 1 -- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f727006a7a0 msgr2=0x7f727006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:27.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.854+0000 7f729a000640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f727006a7a0 0x7f727006cc60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f727c000f60 tx=0x7f727c009040 comp rx=0 tx=0).stop 2024-03-23T11:22:27.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.854+0000 7f729a000640 1 -- 172.21.15.62:0/1166737418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72940cf650 msgr2=0x7f72941f3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:27.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.854+0000 7f729a000640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72940cf650 0x7f72941f3210 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f728400c7c0 tx=0x7f728400cc80 comp rx=0 tx=0).stop 2024-03-23T11:22:27.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.854+0000 7f729a000640 1 -- 172.21.15.62:0/1166737418 shutdown_connections 2024-03-23T11:22:27.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.854+0000 7f729a000640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f727006a7a0 0x7f727006cc60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.854+0000 7f729a000640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f729415cdc0 0x7f7294158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.854+0000 7f729a000640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f729415c9e0 0x7f72940c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.854+0000 7f729a000640 1 --2- 172.21.15.62:0/1166737418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72940cf650 0x7f72941f3210 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:27.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.854+0000 7f729a000640 1 -- 172.21.15.62:0/1166737418 >> 172.21.15.62:0/1166737418 conn(0x7f7294071570 msgr2=0x7f729415ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:27.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.855+0000 7f729a000640 1 -- 172.21.15.62:0/1166737418 shutdown_connections 2024-03-23T11:22:27.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:27.855+0000 7f729a000640 1 -- 172.21.15.62:0/1166737418 wait complete. 2024-03-23T11:22:27.874 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:22:28.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.100+0000 7f60e4e00640 1 Processor -- start 2024-03-23T11:22:28.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.100+0000 7f60e4e00640 1 -- start start 2024-03-23T11:22:28.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.101+0000 7f60e4e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f60e015c9e0 0x7f60e01607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.101+0000 7f60e4e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00cf650 0x7f60e00cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.101+0000 7f60e4e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e006bfc0 con 0x7f60e015cdc0 2024-03-23T11:22:28.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.101+0000 7f60e4e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e00d0020 con 0x7f60e015c9e0 2024-03-23T11:22:28.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.102+0000 7f60e4e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e00d2630 con 0x7f60e00cf650 2024-03-23T11:22:28.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.102+0000 7f60d5400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f60e015cdc0 0x7f60e00d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40032/0 (socket says 172.21.15.62:40032) 2024-03-23T11:22:28.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.102+0000 7f60d5400640 1 -- 172.21.15.62:0/1502228254 learned_addr learned my addr 172.21.15.62:0/1502228254 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:28.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.102+0000 7f60dde00640 1 --2- 172.21.15.62:0/1502228254 >> v2:172.21.15.120:3300/0 conn(0x7f60e015c9e0 0x7f60e01607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.102+0000 7f60de800640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00cf650 0x7f60e00cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.102+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96640564 0 0) 0x7f60e006bfc0 con 0x7f60e015cdc0 2024-03-23T11:22:28.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.102+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c4003520 con 0x7f60e015cdc0 2024-03-23T11:22:28.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2941864683 0 0) 0x7f60c4003520 con 0x7f60e015cdc0 2024-03-23T11:22:28.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00cf650 msgr2=0x7f60e00cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:28.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00cf650 0x7f60e00cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 >> v2:172.21.15.120:3300/0 conn(0x7f60e015c9e0 msgr2=0x7f60e01607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:28.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 --2- 172.21.15.62:0/1502228254 >> v2:172.21.15.120:3300/0 conn(0x7f60e015c9e0 0x7f60e01607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e00b5260 con 0x7f60e015cdc0 2024-03-23T11:22:28.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f60c8001670 con 0x7f60e015cdc0 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 >> v1:172.21.15.62:6789/0 conn(0x7f60e015cdc0 legacy=0x7f60e00d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00cf650 0x7f60c4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4003ef0 0x7f60c40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c4004d00 0x7f60c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e00d0020 con 0x7f60c4003ef0 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e00d2630 con 0x7f60c4004d00 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c400b980 con 0x7f60e00cf650 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.103+0000 7f60d5400640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4003ef0 0x7f60c40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.104+0000 7f60dde00640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00cf650 0x7f60c4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.104+0000 7f60d5400640 1 -- 172.21.15.62:0/1502228254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00cf650 msgr2=0x7f60c4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.104+0000 7f60d5400640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00cf650 0x7f60c4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.104+0000 7f60d5400640 1 -- 172.21.15.62:0/1502228254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c4004d00 msgr2=0x7f60c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:28.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.104+0000 7f60d5400640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c4004d00 0x7f60c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.104+0000 7f60d5400640 1 -- 172.21.15.62:0/1502228254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c4003520 con 0x7f60c4003ef0 2024-03-23T11:22:28.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.104+0000 7f60dde00640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00cf650 0x7f60c4003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:28.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.105+0000 7f60d5400640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4003ef0 0x7f60c40047c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f60c800c240 tx=0x7f60c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5759d7183ab693 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:28.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.105+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c802cae0 con 0x7f60c4003ef0 2024-03-23T11:22:28.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.105+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c802cc40 con 0x7f60c4003ef0 2024-03-23T11:22:28.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.105+0000 7f60dd400640 1 -- 172.21.15.62:0/1502228254 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c8005af0 con 0x7f60c4003ef0 2024-03-23T11:22:28.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.106+0000 7f60e4e00640 1 -- 172.21.15.62:0/1502228254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4003ef0 msgr2=0x7f60c40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:28.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.106+0000 7f60e4e00640 1 --2- 172.21.15.62:0/1502228254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4003ef0 0x7f60c40047c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f60c800c240 tx=0x7f60c40057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.106+0000 7f60de800640 1 -- 172.21.15.62:0/1502228254 reap_dead start 2024-03-23T11:22:28.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.106+0000 7f60e4e00640 1 -- 172.21.15.62:0/1502228254 shutdown_connections 2024-03-23T11:22:28.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.106+0000 7f60e4e00640 1 -- 172.21.15.62:0/1502228254 >> 172.21.15.62:0/1502228254 conn(0x7f60e0071570 msgr2=0x7f60e0071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:28.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.106+0000 7f60e4e00640 1 -- 172.21.15.62:0/1502228254 shutdown_connections 2024-03-23T11:22:28.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.106+0000 7f60e4e00640 1 -- 172.21.15.62:0/1502228254 wait complete. 2024-03-23T11:22:28.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.108+0000 7f60e4e00640 1 Processor -- start 2024-03-23T11:22:28.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.108+0000 7f60e4e00640 1 -- start start 2024-03-23T11:22:28.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60e4e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c4003ef0 0x7f60e00ab690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60e4e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00c8570 0x7f60e00b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60e4e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e00b7070 0x7f60e00cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60e4e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e00ab200 con 0x7f60e00b7070 2024-03-23T11:22:28.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60e4e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e00aa5f0 con 0x7f60c4003ef0 2024-03-23T11:22:28.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60e4e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e00aa8d0 con 0x7f60e00c8570 2024-03-23T11:22:28.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60de800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e00b7070 0x7f60e00cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60de800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e00b7070 0x7f60e00cb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44914/0 (socket says 172.21.15.62:44914) 2024-03-23T11:22:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60de800640 1 -- 172.21.15.62:0/2184770317 learned_addr learned my addr 172.21.15.62:0/2184770317 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60dde00640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c4003ef0 0x7f60e00ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60de800640 1 -- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00c8570 msgr2=0x7f60e00b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60d5400640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00c8570 0x7f60e00b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60de800640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00c8570 0x7f60e00b6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60de800640 1 -- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c4003ef0 msgr2=0x7f60e00ab690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60de800640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c4003ef0 0x7f60e00ab690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.109+0000 7f60de800640 1 -- 172.21.15.62:0/2184770317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e00aabb0 con 0x7f60e00b7070 2024-03-23T11:22:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.110+0000 7f60dde00640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c4003ef0 0x7f60e00ab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.110+0000 7f60de800640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e00b7070 0x7f60e00cb9d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f60e006bc00 tx=0x7f60d800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:28.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.110+0000 7f60d6a00640 1 -- 172.21.15.62:0/2184770317 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d8010040 con 0x7f60e00b7070 2024-03-23T11:22:28.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.110+0000 7f60d6a00640 1 -- 172.21.15.62:0/2184770317 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d800bae0 con 0x7f60e00b7070 2024-03-23T11:22:28.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.110+0000 7f60e4e00640 1 -- 172.21.15.62:0/2184770317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e00aae90 con 0x7f60e00b7070 2024-03-23T11:22:28.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.110+0000 7f60d6a00640 1 -- 172.21.15.62:0/2184770317 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d800be70 con 0x7f60e00b7070 2024-03-23T11:22:28.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.110+0000 7f60e4e00640 1 -- 172.21.15.62:0/2184770317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e00c8bb0 con 0x7f60e00b7070 2024-03-23T11:22:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.111+0000 7f60e4e00640 1 -- 172.21.15.62:0/2184770317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e00ab370 con 0x7f60e00b7070 2024-03-23T11:22:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.112+0000 7f60d6a00640 1 -- 172.21.15.62:0/2184770317 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f60d801e410 con 0x7f60e00b7070 2024-03-23T11:22:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.113+0000 7f60d6a00640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60bc06a7a0 0x7f60bc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.113+0000 7f60d6a00640 1 -- 172.21.15.62:0/2184770317 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f60d8088d30 con 0x7f60e00b7070 2024-03-23T11:22:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.114+0000 7f60dde00640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60bc06a7a0 0x7f60bc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.115+0000 7f60dde00640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60bc06a7a0 0x7f60bc06cc60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009aa0 tx=0x7f60d0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:28.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.118+0000 7f60d6a00640 1 -- 172.21.15.62:0/2184770317 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f60d8054370 con 0x7f60e00b7070 2024-03-23T11:22:28.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.319+0000 7f60e4e00640 1 -- 172.21.15.62:0/2184770317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60e0066c90 con 0x7f60e00b7070 2024-03-23T11:22:28.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.321+0000 7f60d6a00640 1 -- 172.21.15.62:0/2184770317 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+12389 (secure 0 0 0) 0x7f60d8057990 con 0x7f60e00b7070 2024-03-23T11:22:28.320 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:28.320 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":58,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:22:27.302476+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.324+0000 7f60e4e00640 1 -- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60bc06a7a0 msgr2=0x7f60bc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.324+0000 7f60e4e00640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60bc06a7a0 0x7f60bc06cc60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009aa0 tx=0x7f60d0008040 comp rx=0 tx=0).stop 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.324+0000 7f60e4e00640 1 -- 172.21.15.62:0/2184770317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e00b7070 msgr2=0x7f60e00cb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.324+0000 7f60e4e00640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e00b7070 0x7f60e00cb9d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f60e006bc00 tx=0x7f60d800b480 comp rx=0 tx=0).stop 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.324+0000 7f60e4e00640 1 -- 172.21.15.62:0/2184770317 shutdown_connections 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.324+0000 7f60e4e00640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60bc06a7a0 0x7f60bc06cc60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.324+0000 7f60e4e00640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e00b7070 0x7f60e00cb9d0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.324+0000 7f60e4e00640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e00c8570 0x7f60e00b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.324+0000 7f60e4e00640 1 --2- 172.21.15.62:0/2184770317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60c4003ef0 0x7f60e00ab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.324+0000 7f60e4e00640 1 -- 172.21.15.62:0/2184770317 >> 172.21.15.62:0/2184770317 conn(0x7f60e0071570 msgr2=0x7f60e015d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.324+0000 7f60e4e00640 1 -- 172.21.15.62:0/2184770317 shutdown_connections 2024-03-23T11:22:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.325+0000 7f60e4e00640 1 -- 172.21.15.62:0/2184770317 wait complete. 2024-03-23T11:22:28.342 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:22:28.342 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:22:28.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.571+0000 7f9cb7200640 1 Processor -- start 2024-03-23T11:22:28.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.571+0000 7f9cb7200640 1 -- start start 2024-03-23T11:22:28.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.573+0000 7f9cb7200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9cb0143dd0 0x7f9cb015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.573+0000 7f9cb7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cb00b2f30 0x7f9cb0070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.573+0000 7f9cb7200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb00cd140 con 0x7f9cb0070090 2024-03-23T11:22:28.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.573+0000 7f9cb7200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb0070a60 con 0x7f9cb0143dd0 2024-03-23T11:22:28.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.573+0000 7f9cb7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb00cd3e0 con 0x7f9cb00b2f30 2024-03-23T11:22:28.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.573+0000 7f9cb5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9cb0143dd0 0x7f9cb015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.573+0000 7f9cb5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9cb0143dd0 0x7f9cb015abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44930/0 (socket says 172.21.15.62:44930) 2024-03-23T11:22:28.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.573+0000 7f9cb5e00640 1 -- 172.21.15.62:0/1576338278 learned_addr learned my addr 172.21.15.62:0/1576338278 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:28.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.574+0000 7f9cb6800640 1 --2- 172.21.15.62:0/1576338278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cb00b2f30 0x7f9cb0070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.574+0000 7f9cb5e00640 1 -- 172.21.15.62:0/1576338278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cb00b2f30 msgr2=0x7f9cb0070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:28.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.574+0000 7f9cb5e00640 1 --2- 172.21.15.62:0/1576338278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cb00b2f30 0x7f9cb0070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.574+0000 7f9cb5e00640 1 -- 172.21.15.62:0/1576338278 >> v1:172.21.15.120:6789/0 conn(0x7f9cb0070090 legacy=0x7f9cb00b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:28.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.574+0000 7f9cb5e00640 1 -- 172.21.15.62:0/1576338278 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb00cd6c0 con 0x7f9cb0143dd0 2024-03-23T11:22:28.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.575+0000 7f9cb5e00640 1 --2- 172.21.15.62:0/1576338278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cb0143dd0 0x7f9cb015abe0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4009870 tx=0x7f9ca402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41510a775a1ac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:28.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.576+0000 7f9cb4a00640 1 -- 172.21.15.62:0/1576338278 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4031640 con 0x7f9cb0143dd0 2024-03-23T11:22:28.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.576+0000 7f9cb4a00640 1 -- 172.21.15.62:0/1576338278 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca40317a0 con 0x7f9cb0143dd0 2024-03-23T11:22:28.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.576+0000 7f9cb4a00640 1 -- 172.21.15.62:0/1576338278 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4031b30 con 0x7f9cb0143dd0 2024-03-23T11:22:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.576+0000 7f9cb7200640 1 -- 172.21.15.62:0/1576338278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cb0143dd0 msgr2=0x7f9cb015abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.576+0000 7f9cb7200640 1 --2- 172.21.15.62:0/1576338278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cb0143dd0 0x7f9cb015abe0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4009870 tx=0x7f9ca402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.577+0000 7f9cb7200640 1 -- 172.21.15.62:0/1576338278 shutdown_connections 2024-03-23T11:22:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.577+0000 7f9cb7200640 1 --2- 172.21.15.62:0/1576338278 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cb00b2f30 0x7f9cb0070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.577+0000 7f9cb7200640 1 --2- 172.21.15.62:0/1576338278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cb0143dd0 0x7f9cb015abe0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.577+0000 7f9cb7200640 1 -- 172.21.15.62:0/1576338278 >> 172.21.15.62:0/1576338278 conn(0x7f9cb0066050 msgr2=0x7f9cb0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:28.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.577+0000 7f9cb7200640 1 -- 172.21.15.62:0/1576338278 shutdown_connections 2024-03-23T11:22:28.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.577+0000 7f9cb7200640 1 -- 172.21.15.62:0/1576338278 wait complete. 2024-03-23T11:22:28.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.578+0000 7f9cb7200640 1 Processor -- start 2024-03-23T11:22:28.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.579+0000 7f9cb7200640 1 -- start start 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cb0070090 0x7f9cb0141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cb00b2f30 0x7f9cb013f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cb0143dd0 0x7f9cb0141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb0140760 con 0x7f9cb0143dd0 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb01416b0 con 0x7f9cb00b2f30 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb01424b0 con 0x7f9cb0070090 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cb0143dd0 0x7f9cb0141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cb0143dd0 0x7f9cb0141a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44940/0 (socket says 172.21.15.62:44940) 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb6800640 1 -- 172.21.15.62:0/316359999 learned_addr learned my addr 172.21.15.62:0/316359999 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb5e00640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cb0070090 0x7f9cb0141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb6800640 1 -- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cb0070090 msgr2=0x7f9cb0141170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb6800640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cb0070090 0x7f9cb0141170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb5400640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cb00b2f30 0x7f9cb013f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb6800640 1 -- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cb00b2f30 msgr2=0x7f9cb013f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:28.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb6800640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cb00b2f30 0x7f9cb013f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb6800640 1 -- 172.21.15.62:0/316359999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca40094d0 con 0x7f9cb0143dd0 2024-03-23T11:22:28.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.580+0000 7f9cb5400640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cb00b2f30 0x7f9cb013f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:28.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.581+0000 7f9cb6800640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cb0143dd0 0x7f9cb0141a60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f9cac007e70 tx=0x7f9cac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:28.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.581+0000 7f9caa000640 1 -- 172.21.15.62:0/316359999 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac014620 con 0x7f9cb0143dd0 2024-03-23T11:22:28.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.581+0000 7f9caa000640 1 -- 172.21.15.62:0/316359999 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cac0147a0 con 0x7f9cb0143dd0 2024-03-23T11:22:28.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.581+0000 7f9caa000640 1 -- 172.21.15.62:0/316359999 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac014b50 con 0x7f9cb0143dd0 2024-03-23T11:22:28.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.581+0000 7f9cb7200640 1 -- 172.21.15.62:0/316359999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb0142790 con 0x7f9cb0143dd0 2024-03-23T11:22:28.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.581+0000 7f9cb7200640 1 -- 172.21.15.62:0/316359999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb0140c50 con 0x7f9cb0143dd0 2024-03-23T11:22:28.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.582+0000 7f9cb7200640 1 -- 172.21.15.62:0/316359999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb0066bb0 con 0x7f9cb0143dd0 2024-03-23T11:22:28.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.583+0000 7f9caa000640 1 -- 172.21.15.62:0/316359999 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9cac014cb0 con 0x7f9cb0143dd0 2024-03-23T11:22:28.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.584+0000 7f9caa000640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9c9406a7a0 0x7f9c9406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:28.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.584+0000 7f9caa000640 1 -- 172.21.15.62:0/316359999 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9cac0894b0 con 0x7f9cb0143dd0 2024-03-23T11:22:28.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.585+0000 7f9cb5e00640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9c9406a7a0 0x7f9c9406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:28.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.586+0000 7f9cb5e00640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9c9406a7a0 0x7f9c9406cc60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4036000 tx=0x7f9ca403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:28.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.589+0000 7f9caa000640 1 -- 172.21.15.62:0/316359999 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9cac054b80 con 0x7f9cb0143dd0 2024-03-23T11:22:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.784+0000 7f9cb7200640 1 -- 172.21.15.62:0/316359999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9cb015acc0 con 0x7f9cb0143dd0 2024-03-23T11:22:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.785+0000 7f9caa000640 1 -- 172.21.15.62:0/316359999 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9cac0581a0 con 0x7f9cb0143dd0 2024-03-23T11:22:28.783 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:22:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 -- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9c9406a7a0 msgr2=0x7f9c9406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9c9406a7a0 0x7f9c9406cc60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4036000 tx=0x7f9ca403c040 comp rx=0 tx=0).stop 2024-03-23T11:22:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 -- 172.21.15.62:0/316359999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cb0143dd0 msgr2=0x7f9cb0141a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cb0143dd0 0x7f9cb0141a60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f9cac007e70 tx=0x7f9cac00e4b0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 -- 172.21.15.62:0/316359999 shutdown_connections 2024-03-23T11:22:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9c9406a7a0 0x7f9c9406cc60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cb0143dd0 0x7f9cb0141a60 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cb00b2f30 0x7f9cb013f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 --2- 172.21.15.62:0/316359999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cb0070090 0x7f9cb0141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 -- 172.21.15.62:0/316359999 >> 172.21.15.62:0/316359999 conn(0x7f9cb0066050 msgr2=0x7f9cb00ade70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 -- 172.21.15.62:0/316359999 shutdown_connections 2024-03-23T11:22:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:28.788+0000 7f9cb7200640 1 -- 172.21.15.62:0/316359999 wait complete. 2024-03-23T11:22:28.806 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T11:22:29.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.025+0000 7fe56aa00640 1 Processor -- start 2024-03-23T11:22:29.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.026+0000 7fe56aa00640 1 -- start start 2024-03-23T11:22:29.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.027+0000 7fe56aa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe564140f80 0x7fe56415ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.027+0000 7fe56aa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe56415b170 0x7fe56406c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.027+0000 7fe56aa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe56406ce10 0x7fe56415c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.027+0000 7fe56aa00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5640a7540 con 0x7fe56406ce10 2024-03-23T11:22:29.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.027+0000 7fe56aa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56415be80 con 0x7fe564140f80 2024-03-23T11:22:29.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.027+0000 7fe56aa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56415b5d0 con 0x7fe56415b170 2024-03-23T11:22:29.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.028+0000 7fe56a000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe56406ce10 0x7fe56415c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:29.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.028+0000 7fe56a000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe56406ce10 0x7fe56415c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44942/0 (socket says 172.21.15.62:44942) 2024-03-23T11:22:29.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.028+0000 7fe56a000640 1 -- 172.21.15.62:0/3989248983 learned_addr learned my addr 172.21.15.62:0/3989248983 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:29.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.028+0000 7fe568c00640 1 --2- 172.21.15.62:0/3989248983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe56415b170 0x7fe56406c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:29.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.028+0000 7fe569600640 1 --2- 172.21.15.62:0/3989248983 >> v2:172.21.15.120:3300/0 conn(0x7fe564140f80 0x7fe56415ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:29.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.028+0000 7fe56a000640 1 -- 172.21.15.62:0/3989248983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe56415b170 msgr2=0x7fe56406c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.028+0000 7fe56a000640 1 --2- 172.21.15.62:0/3989248983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe56415b170 0x7fe56406c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.028+0000 7fe56a000640 1 -- 172.21.15.62:0/3989248983 >> v2:172.21.15.120:3300/0 conn(0x7fe564140f80 msgr2=0x7fe56415ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.028+0000 7fe56a000640 1 --2- 172.21.15.62:0/3989248983 >> v2:172.21.15.120:3300/0 conn(0x7fe564140f80 0x7fe56415ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.028+0000 7fe56a000640 1 -- 172.21.15.62:0/3989248983 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56406d5c0 con 0x7fe56406ce10 2024-03-23T11:22:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.029+0000 7fe56a000640 1 --2- 172.21.15.62:0/3989248983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe56406ce10 0x7fe56415c360 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe56404f0f0 tx=0x7fe55c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32fff7a1b9b63271 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.030+0000 7fe563e00640 1 -- 172.21.15.62:0/3989248983 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c02cb00 con 0x7fe56406ce10 2024-03-23T11:22:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.030+0000 7fe563e00640 1 -- 172.21.15.62:0/3989248983 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55c02cc60 con 0x7fe56406ce10 2024-03-23T11:22:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.030+0000 7fe563e00640 1 -- 172.21.15.62:0/3989248983 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c0343f0 con 0x7fe56406ce10 2024-03-23T11:22:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.030+0000 7fe56aa00640 1 -- 172.21.15.62:0/3989248983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe56406ce10 msgr2=0x7fe56415c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.030+0000 7fe56aa00640 1 --2- 172.21.15.62:0/3989248983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe56406ce10 0x7fe56415c360 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe56404f0f0 tx=0x7fe55c02bf60 comp rx=0 tx=0).stop 2024-03-23T11:22:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.030+0000 7fe56aa00640 1 -- 172.21.15.62:0/3989248983 shutdown_connections 2024-03-23T11:22:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.030+0000 7fe56aa00640 1 --2- 172.21.15.62:0/3989248983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe56406ce10 0x7fe56415c360 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.030+0000 7fe56aa00640 1 --2- 172.21.15.62:0/3989248983 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe56415b170 0x7fe56406c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.030+0000 7fe56aa00640 1 --2- 172.21.15.62:0/3989248983 >> v2:172.21.15.120:3300/0 conn(0x7fe564140f80 0x7fe56415ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.030+0000 7fe56aa00640 1 -- 172.21.15.62:0/3989248983 >> 172.21.15.62:0/3989248983 conn(0x7fe5640b0100 msgr2=0x7fe5640b0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.031+0000 7fe56aa00640 1 -- 172.21.15.62:0/3989248983 shutdown_connections 2024-03-23T11:22:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.031+0000 7fe56aa00640 1 -- 172.21.15.62:0/3989248983 wait complete. 2024-03-23T11:22:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.032+0000 7fe56aa00640 1 Processor -- start 2024-03-23T11:22:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.032+0000 7fe56aa00640 1 -- start start 2024-03-23T11:22:29.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.033+0000 7fe56aa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe56406ce10 0x7fe56413e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.033+0000 7fe56aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe564140f80 0x7fe5640ac550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.033+0000 7fe56aa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56415b170 0x7fe5641f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.033+0000 7fe56aa00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5641f3f20 con 0x7fe564140f80 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.033+0000 7fe56aa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56413c5a0 con 0x7fe56415b170 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.033+0000 7fe56aa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5641f3310 con 0x7fe56406ce10 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.034+0000 7fe568c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe564140f80 0x7fe5640ac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.034+0000 7fe568c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe564140f80 0x7fe5640ac550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44944/0 (socket says 172.21.15.62:44944) 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.034+0000 7fe568c00640 1 -- 172.21.15.62:0/1678996777 learned_addr learned my addr 172.21.15.62:0/1678996777 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.034+0000 7fe569600640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe56406ce10 0x7fe56413e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.034+0000 7fe56a000640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56415b170 0x7fe5641f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.034+0000 7fe568c00640 1 -- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe56406ce10 msgr2=0x7fe56413e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.034+0000 7fe568c00640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe56406ce10 0x7fe56413e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.034+0000 7fe568c00640 1 -- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56415b170 msgr2=0x7fe5641f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:29.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.034+0000 7fe568c00640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56415b170 0x7fe5641f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.034+0000 7fe568c00640 1 -- 172.21.15.62:0/1678996777 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c009b30 con 0x7fe564140f80 2024-03-23T11:22:29.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.034+0000 7fe568c00640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe564140f80 0x7fe5640ac550 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe55000d800 tx=0x7fe55000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:29.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.035+0000 7fe561600640 1 -- 172.21.15.62:0/1678996777 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55000b6c0 con 0x7fe564140f80 2024-03-23T11:22:29.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.035+0000 7fe561600640 1 -- 172.21.15.62:0/1678996777 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55000b820 con 0x7fe564140f80 2024-03-23T11:22:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.035+0000 7fe561600640 1 -- 172.21.15.62:0/1678996777 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55000bbb0 con 0x7fe564140f80 2024-03-23T11:22:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.035+0000 7fe56aa00640 1 -- 172.21.15.62:0/1678996777 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5641f35f0 con 0x7fe564140f80 2024-03-23T11:22:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.035+0000 7fe56aa00640 1 -- 172.21.15.62:0/1678996777 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5641f3bb0 con 0x7fe564140f80 2024-03-23T11:22:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.036+0000 7fe56aa00640 1 -- 172.21.15.62:0/1678996777 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5641f3d70 con 0x7fe564140f80 2024-03-23T11:22:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.037+0000 7fe561600640 1 -- 172.21.15.62:0/1678996777 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe55000bd10 con 0x7fe564140f80 2024-03-23T11:22:29.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.038+0000 7fe561600640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe54406a7d0 0x7fe54406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.038+0000 7fe561600640 1 -- 172.21.15.62:0/1678996777 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe550088ff0 con 0x7fe564140f80 2024-03-23T11:22:29.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.039+0000 7fe569600640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe54406a7d0 0x7fe54406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:29.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.040+0000 7fe569600640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe54406a7d0 0x7fe54406cc90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe558006fd0 tx=0x7fe558008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:29.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.042+0000 7fe561600640 1 -- 172.21.15.62:0/1678996777 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe5500bc580 con 0x7fe564140f80 2024-03-23T11:22:29.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.247+0000 7fe56aa00640 1 -- 172.21.15.62:0/1678996777 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe564066bb0 con 0x7fe564140f80 2024-03-23T11:22:29.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.248+0000 7fe561600640 1 -- 172.21.15.62:0/1678996777 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v58) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe550054700 con 0x7fe564140f80 2024-03-23T11:22:29.246 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:29.246 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:22:29.246 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:29.246 INFO:teuthology.orchestra.run.smithi062.stdout: "poolnum": 1, 2024-03-23T11:22:29.246 INFO:teuthology.orchestra.run.smithi062.stdout: "poolname": ".mgr" 2024-03-23T11:22:29.247 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:29.247 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:22:29.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.250+0000 7fe56aa00640 1 -- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe54406a7d0 msgr2=0x7fe54406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:29.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.250+0000 7fe56aa00640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe54406a7d0 0x7fe54406cc90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe558006fd0 tx=0x7fe558008040 comp rx=0 tx=0).stop 2024-03-23T11:22:29.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.250+0000 7fe56aa00640 1 -- 172.21.15.62:0/1678996777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe564140f80 msgr2=0x7fe5640ac550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:29.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.250+0000 7fe56aa00640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe564140f80 0x7fe5640ac550 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe55000d800 tx=0x7fe55000dcc0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.251+0000 7fe56aa00640 1 -- 172.21.15.62:0/1678996777 shutdown_connections 2024-03-23T11:22:29.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.251+0000 7fe56aa00640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe54406a7d0 0x7fe54406cc90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.251+0000 7fe56aa00640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe564140f80 0x7fe5640ac550 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.251+0000 7fe56aa00640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56415b170 0x7fe5641f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.251+0000 7fe56aa00640 1 --2- 172.21.15.62:0/1678996777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe56406ce10 0x7fe56413e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.251+0000 7fe56aa00640 1 -- 172.21.15.62:0/1678996777 >> 172.21.15.62:0/1678996777 conn(0x7fe5640b0100 msgr2=0x7fe5640b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:29.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.251+0000 7fe56aa00640 1 -- 172.21.15.62:0/1678996777 shutdown_connections 2024-03-23T11:22:29.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.251+0000 7fe56aa00640 1 -- 172.21.15.62:0/1678996777 wait complete. 2024-03-23T11:22:29.269 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T11:22:29.269 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T11:22:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.492+0000 7f0e61600640 1 Processor -- start 2024-03-23T11:22:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.493+0000 7f0e61600640 1 -- start start 2024-03-23T11:22:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.494+0000 7f0e61600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0e5c06db90 0x7f0e5c0cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.494+0000 7f0e61600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c0c3dd0 con 0x7f0e5c0cd6e0 2024-03-23T11:22:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.494+0000 7f0e61600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c0c3850 con 0x7f0e5c0b0d40 2024-03-23T11:22:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.494+0000 7f0e61600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5c159450 con 0x7f0e5c06db90 2024-03-23T11:22:29.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.495+0000 7f0e5aa00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f0e5c0b0d40 0x7f0e5c158d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:38126/0 (socket says 172.21.15.62:38126) 2024-03-23T11:22:29.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.495+0000 7f0e5aa00640 1 -- 172.21.15.62:0/752077879 learned_addr learned my addr 172.21.15.62:0/752077879 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:29.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.495+0000 7f0e5a000640 1 --2- 172.21.15.62:0/752077879 >> v2:172.21.15.120:3300/0 conn(0x7f0e5c06db90 0x7f0e5c0cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:29.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.496+0000 7f0e59600640 1 -- 172.21.15.62:0/752077879 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988082526 0 0) 0x7f0e5c0c3dd0 con 0x7f0e5c0cd6e0 2024-03-23T11:22:29.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.496+0000 7f0e59600640 1 -- 172.21.15.62:0/752077879 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e40005730 con 0x7f0e5c0cd6e0 2024-03-23T11:22:29.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.496+0000 7f0e59600640 1 -- 172.21.15.62:0/752077879 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997580354 0 0) 0x7f0e5c0c3850 con 0x7f0e5c0b0d40 2024-03-23T11:22:29.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.496+0000 7f0e59600640 1 -- 172.21.15.62:0/752077879 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e5c0c3dd0 con 0x7f0e5c0b0d40 2024-03-23T11:22:29.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.496+0000 7f0e5a000640 1 -- 172.21.15.62:0/752077879 >> v1:172.21.15.120:6789/0 conn(0x7f0e5c0b0d40 legacy=0x7f0e5c158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:29.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.496+0000 7f0e5a000640 1 -- 172.21.15.62:0/752077879 >> v1:172.21.15.62:6789/0 conn(0x7f0e5c0cd6e0 legacy=0x7f0e5c0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:29.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.496+0000 7f0e5a000640 1 -- 172.21.15.62:0/752077879 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c15d720 con 0x7f0e5c06db90 2024-03-23T11:22:29.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.497+0000 7f0e5a000640 1 --2- 172.21.15.62:0/752077879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e5c06db90 0x7f0e5c0cd1a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0e480097a0 tx=0x7f0e4802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d35c726f674e5e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:29.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.498+0000 7f0e59600640 1 -- 172.21.15.62:0/752077879 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48031830 con 0x7f0e5c06db90 2024-03-23T11:22:29.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.498+0000 7f0e59600640 1 -- 172.21.15.62:0/752077879 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e48031990 con 0x7f0e5c06db90 2024-03-23T11:22:29.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.498+0000 7f0e59600640 1 -- 172.21.15.62:0/752077879 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48031d20 con 0x7f0e5c06db90 2024-03-23T11:22:29.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.498+0000 7f0e61600640 1 -- 172.21.15.62:0/752077879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e5c06db90 msgr2=0x7f0e5c0cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:29.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.498+0000 7f0e61600640 1 --2- 172.21.15.62:0/752077879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e5c06db90 0x7f0e5c0cd1a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0e480097a0 tx=0x7f0e4802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.498+0000 7f0e61600640 1 -- 172.21.15.62:0/752077879 shutdown_connections 2024-03-23T11:22:29.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.498+0000 7f0e61600640 1 --2- 172.21.15.62:0/752077879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e5c06db90 0x7f0e5c0cd1a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.498+0000 7f0e61600640 1 -- 172.21.15.62:0/752077879 >> 172.21.15.62:0/752077879 conn(0x7f0e5c0b17d0 msgr2=0x7f0e5c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:29.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.498+0000 7f0e61600640 1 -- 172.21.15.62:0/752077879 shutdown_connections 2024-03-23T11:22:29.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.498+0000 7f0e61600640 1 -- 172.21.15.62:0/752077879 wait complete. 2024-03-23T11:22:29.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.500+0000 7f0e61600640 1 Processor -- start 2024-03-23T11:22:29.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.500+0000 7f0e61600640 1 -- start start 2024-03-23T11:22:29.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.501+0000 7f0e61600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e5c06db90 0x7f0e5c1482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.501+0000 7f0e61600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e5c0b0d40 0x7f0e5c155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.501+0000 7f0e61600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e5c0cd6e0 0x7f0e5c1556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.501+0000 7f0e61600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c150c60 con 0x7f0e5c06db90 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.501+0000 7f0e61600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c150340 con 0x7f0e5c0b0d40 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.501+0000 7f0e61600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c1505e0 con 0x7f0e5c0cd6e0 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.502+0000 7f0e5aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e5c06db90 0x7f0e5c1482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.502+0000 7f0e5aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e5c06db90 0x7f0e5c1482a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44950/0 (socket says 172.21.15.62:44950) 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.502+0000 7f0e5aa00640 1 -- 172.21.15.62:0/1071761209 learned_addr learned my addr 172.21.15.62:0/1071761209 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.502+0000 7f0e5aa00640 1 -- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e5c0cd6e0 msgr2=0x7f0e5c1556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.502+0000 7f0e5aa00640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e5c0cd6e0 0x7f0e5c1556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.502+0000 7f0e5aa00640 1 -- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e5c0b0d40 msgr2=0x7f0e5c155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.502+0000 7f0e5aa00640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e5c0b0d40 0x7f0e5c155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.502+0000 7f0e5aa00640 1 -- 172.21.15.62:0/1071761209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e48009400 con 0x7f0e5c06db90 2024-03-23T11:22:29.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.502+0000 7f0e5aa00640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e5c06db90 0x7f0e5c1482a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00c500 tx=0x7f0e4c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:29.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.503+0000 7f0e52a00640 1 -- 172.21.15.62:0/1071761209 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c005ce0 con 0x7f0e5c06db90 2024-03-23T11:22:29.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.503+0000 7f0e52a00640 1 -- 172.21.15.62:0/1071761209 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c005e40 con 0x7f0e5c06db90 2024-03-23T11:22:29.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.503+0000 7f0e61600640 1 -- 172.21.15.62:0/1071761209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5c15bd60 con 0x7f0e5c06db90 2024-03-23T11:22:29.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.503+0000 7f0e52a00640 1 -- 172.21.15.62:0/1071761209 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c007330 con 0x7f0e5c06db90 2024-03-23T11:22:29.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.503+0000 7f0e61600640 1 -- 172.21.15.62:0/1071761209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5c150880 con 0x7f0e5c06db90 2024-03-23T11:22:29.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.504+0000 7f0e61600640 1 -- 172.21.15.62:0/1071761209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5c066c90 con 0x7f0e5c06db90 2024-03-23T11:22:29.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.505+0000 7f0e52a00640 1 -- 172.21.15.62:0/1071761209 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0e4c007550 con 0x7f0e5c06db90 2024-03-23T11:22:29.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.506+0000 7f0e52a00640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0e3c06a7a0 0x7f0e3c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:29.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.506+0000 7f0e52a00640 1 -- 172.21.15.62:0/1071761209 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0e4c088180 con 0x7f0e5c06db90 2024-03-23T11:22:29.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.506+0000 7f0e5a000640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0e3c06a7a0 0x7f0e3c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:29.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.508+0000 7f0e5a000640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0e3c06a7a0 0x7f0e3c06cc60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0e48036000 tx=0x7f0e48030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:29.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.510+0000 7f0e52a00640 1 -- 172.21.15.62:0/1071761209 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0e4c053920 con 0x7f0e5c06db90 2024-03-23T11:22:29.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:29.709+0000 7f0e61600640 1 -- 172.21.15.62:0/1071761209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0e5c0b3430 con 0x7f0e5c06db90 2024-03-23T11:22:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.225+0000 7f0e52a00640 1 -- 172.21.15.62:0/1071761209 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0e4c056f40 con 0x7f0e5c06db90 2024-03-23T11:22:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' created 2024-03-23T11:22:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.229+0000 7f0e61600640 1 -- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0e3c06a7a0 msgr2=0x7f0e3c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.229+0000 7f0e61600640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0e3c06a7a0 0x7f0e3c06cc60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0e48036000 tx=0x7f0e48030000 comp rx=0 tx=0).stop 2024-03-23T11:22:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.229+0000 7f0e61600640 1 -- 172.21.15.62:0/1071761209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e5c06db90 msgr2=0x7f0e5c1482a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.229+0000 7f0e61600640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e5c06db90 0x7f0e5c1482a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00c500 tx=0x7f0e4c00c9c0 comp rx=0 tx=0).stop 2024-03-23T11:22:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.229+0000 7f0e61600640 1 -- 172.21.15.62:0/1071761209 shutdown_connections 2024-03-23T11:22:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.229+0000 7f0e61600640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0e3c06a7a0 0x7f0e3c06cc60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.229+0000 7f0e61600640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e5c0cd6e0 0x7f0e5c1556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.229+0000 7f0e61600640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e5c0b0d40 0x7f0e5c155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.229+0000 7f0e61600640 1 --2- 172.21.15.62:0/1071761209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e5c06db90 0x7f0e5c1482a0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.229+0000 7f0e61600640 1 -- 172.21.15.62:0/1071761209 >> 172.21.15.62:0/1071761209 conn(0x7f0e5c0b17d0 msgr2=0x7f0e5c158d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.229+0000 7f0e61600640 1 -- 172.21.15.62:0/1071761209 shutdown_connections 2024-03-23T11:22:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.230+0000 7f0e61600640 1 -- 172.21.15.62:0/1071761209 wait complete. 2024-03-23T11:22:30.251 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T11:22:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.461+0000 7f9069a00640 1 Processor -- start 2024-03-23T11:22:30.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.461+0000 7f9069a00640 1 -- start start 2024-03-23T11:22:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.462+0000 7f9069a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9064143fe0 0x7f90640ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.462+0000 7f9069a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90640ce780 0x7f906415e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.463+0000 7f9069a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f906415eec0 con 0x7f9064143c00 2024-03-23T11:22:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.463+0000 7f9069a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90640cec10 con 0x7f9064143fe0 2024-03-23T11:22:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.463+0000 7f9069a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90640b1a50 con 0x7f90640ce780 2024-03-23T11:22:30.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.463+0000 7f9063400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9064143fe0 0x7f90640ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:30.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.463+0000 7f9069000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90640ce780 0x7f906415e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:30.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.464+0000 7f9063400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9064143fe0 0x7f90640ce240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:58434/0 (socket says 172.21.15.62:58434) 2024-03-23T11:22:30.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.464+0000 7f9063400640 1 -- 172.21.15.62:0/893060734 learned_addr learned my addr 172.21.15.62:0/893060734 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:30.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.464+0000 7f9069000640 1 -- 172.21.15.62:0/893060734 >> v2:172.21.15.120:3300/0 conn(0x7f9064143fe0 msgr2=0x7f90640ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:30.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.464+0000 7f9069000640 1 --2- 172.21.15.62:0/893060734 >> v2:172.21.15.120:3300/0 conn(0x7f9064143fe0 0x7f90640ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.464+0000 7f9069000640 1 -- 172.21.15.62:0/893060734 >> v1:172.21.15.120:6789/0 conn(0x7f9064143c00 legacy=0x7f906415abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.464+0000 7f9069000640 1 -- 172.21.15.62:0/893060734 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90640b1d30 con 0x7f90640ce780 2024-03-23T11:22:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.464+0000 7f9063400640 1 --2- 172.21.15.62:0/893060734 >> v2:172.21.15.120:3300/0 conn(0x7f9064143fe0 0x7f90640ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:22:30.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.466+0000 7f9069000640 1 --2- 172.21.15.62:0/893060734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90640ce780 0x7f906415e190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f905c0097a0 tx=0x7f905c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c43ad7bcd59476 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.466+0000 7f9062a00640 1 -- 172.21.15.62:0/893060734 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c02d040 con 0x7f90640ce780 2024-03-23T11:22:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.467+0000 7f9062a00640 1 -- 172.21.15.62:0/893060734 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905c031810 con 0x7f90640ce780 2024-03-23T11:22:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.467+0000 7f9062a00640 1 -- 172.21.15.62:0/893060734 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c031ba0 con 0x7f90640ce780 2024-03-23T11:22:30.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.467+0000 7f9069a00640 1 -- 172.21.15.62:0/893060734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90640ce780 msgr2=0x7f906415e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:30.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.467+0000 7f9069a00640 1 --2- 172.21.15.62:0/893060734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90640ce780 0x7f906415e190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f905c0097a0 tx=0x7f905c02b8b0 comp rx=0 tx=0).stop 2024-03-23T11:22:30.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.467+0000 7f9069a00640 1 -- 172.21.15.62:0/893060734 shutdown_connections 2024-03-23T11:22:30.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.467+0000 7f9069a00640 1 --2- 172.21.15.62:0/893060734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90640ce780 0x7f906415e190 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:30.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.467+0000 7f9069a00640 1 --2- 172.21.15.62:0/893060734 >> v2:172.21.15.120:3300/0 conn(0x7f9064143fe0 0x7f90640ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:30.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.467+0000 7f9069a00640 1 -- 172.21.15.62:0/893060734 >> 172.21.15.62:0/893060734 conn(0x7f9064066050 msgr2=0x7f9064069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:30.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.468+0000 7f9069a00640 1 -- 172.21.15.62:0/893060734 shutdown_connections 2024-03-23T11:22:30.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.468+0000 7f9069a00640 1 -- 172.21.15.62:0/893060734 wait complete. 2024-03-23T11:22:30.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.470+0000 7f9069a00640 1 Processor -- start 2024-03-23T11:22:30.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.470+0000 7f9069a00640 1 -- start start 2024-03-23T11:22:30.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.471+0000 7f9069a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90640ce780 0x7f90640cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:30.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.471+0000 7f9069a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9064143c00 0x7f90641f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:30.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.471+0000 7f9069a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9064143fe0 0x7f906413ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:30.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.471+0000 7f9069a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90640adaf0 con 0x7f9064143c00 2024-03-23T11:22:30.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.471+0000 7f9069a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90641f3570 con 0x7f9064143fe0 2024-03-23T11:22:30.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.471+0000 7f9069a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90641f23e0 con 0x7f90640ce780 2024-03-23T11:22:30.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.471+0000 7f9063400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9064143c00 0x7f90641f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:30.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.472+0000 7f9063400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9064143c00 0x7f90641f3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44960/0 (socket says 172.21.15.62:44960) 2024-03-23T11:22:30.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.472+0000 7f9063400640 1 -- 172.21.15.62:0/3931900132 learned_addr learned my addr 172.21.15.62:0/3931900132 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:30.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.472+0000 7f9063e00640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90640ce780 0x7f90640cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:30.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.472+0000 7f9069000640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9064143fe0 0x7f906413ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:30.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.472+0000 7f9063400640 1 -- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90640ce780 msgr2=0x7f90640cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:30.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.472+0000 7f9063400640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90640ce780 0x7f90640cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:30.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.472+0000 7f9063400640 1 -- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9064143fe0 msgr2=0x7f906413ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:30.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.472+0000 7f9063400640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9064143fe0 0x7f906413ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:30.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.472+0000 7f9063400640 1 -- 172.21.15.62:0/3931900132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c0094d0 con 0x7f9064143c00 2024-03-23T11:22:30.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.472+0000 7f9069000640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9064143fe0 0x7f906413ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:30.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.472+0000 7f9063400640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9064143c00 0x7f90641f3030 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e9b0 tx=0x7f904c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:30.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.473+0000 7f905be00640 1 -- 172.21.15.62:0/3931900132 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c00c9a0 con 0x7f9064143c00 2024-03-23T11:22:30.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.473+0000 7f905be00640 1 -- 172.21.15.62:0/3931900132 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904c00cb00 con 0x7f9064143c00 2024-03-23T11:22:30.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.473+0000 7f9069a00640 1 -- 172.21.15.62:0/3931900132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905c009400 con 0x7f9064143c00 2024-03-23T11:22:30.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.473+0000 7f905be00640 1 -- 172.21.15.62:0/3931900132 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c00ce70 con 0x7f9064143c00 2024-03-23T11:22:30.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.473+0000 7f9069a00640 1 -- 172.21.15.62:0/3931900132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90641f29a0 con 0x7f9064143c00 2024-03-23T11:22:30.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.474+0000 7f9069a00640 1 -- 172.21.15.62:0/3931900132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9064066bb0 con 0x7f9064143c00 2024-03-23T11:22:30.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.476+0000 7f905be00640 1 -- 172.21.15.62:0/3931900132 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f904c026480 con 0x7f9064143c00 2024-03-23T11:22:30.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.477+0000 7f905be00640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f904006a7a0 0x7f904006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:30.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.477+0000 7f905be00640 1 -- 172.21.15.62:0/3931900132 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f904c0895f0 con 0x7f9064143c00 2024-03-23T11:22:30.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.478+0000 7f9063e00640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f904006a7a0 0x7f904006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:30.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.479+0000 7f9063e00640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f904006a7a0 0x7f904006cc60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9054001d40 tx=0x7f9054009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:30.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.481+0000 7f905be00640 1 -- 172.21.15.62:0/3931900132 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f904c054cd0 con 0x7f9064143c00 2024-03-23T11:22:30.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:30.684+0000 7f9069a00640 1 -- 172.21.15.62:0/3931900132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f90640cc620 con 0x7f9064143c00 2024-03-23T11:22:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.234+0000 7f905be00640 1 -- 172.21.15.62:0/3931900132 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v60) v1 ==== 170+0+0 (secure 0 0 0) 0x7f904c0582f0 con 0x7f9064143c00 2024-03-23T11:22:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' created 2024-03-23T11:22:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.242+0000 7f9069a00640 1 -- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f904006a7a0 msgr2=0x7f904006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.242+0000 7f9069a00640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f904006a7a0 0x7f904006cc60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9054001d40 tx=0x7f9054009040 comp rx=0 tx=0).stop 2024-03-23T11:22:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.242+0000 7f9069a00640 1 -- 172.21.15.62:0/3931900132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9064143c00 msgr2=0x7f90641f3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:31.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.242+0000 7f9069a00640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9064143c00 0x7f90641f3030 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e9b0 tx=0x7f904c00ee70 comp rx=0 tx=0).stop 2024-03-23T11:22:31.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.246+0000 7f9069a00640 1 -- 172.21.15.62:0/3931900132 shutdown_connections 2024-03-23T11:22:31.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.246+0000 7f9069a00640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f904006a7a0 0x7f904006cc60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.246+0000 7f9069a00640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9064143c00 0x7f90641f3030 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.246+0000 7f9069a00640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9064143fe0 0x7f906413ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.246+0000 7f9069a00640 1 --2- 172.21.15.62:0/3931900132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90640ce780 0x7f90640cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.246+0000 7f9069a00640 1 -- 172.21.15.62:0/3931900132 >> 172.21.15.62:0/3931900132 conn(0x7f9064066050 msgr2=0x7f90640b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.250+0000 7f9069a00640 1 -- 172.21.15.62:0/3931900132 shutdown_connections 2024-03-23T11:22:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.250+0000 7f9069a00640 1 -- 172.21.15.62:0/3931900132 wait complete. 2024-03-23T11:22:31.276 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T11:22:31.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.465+0000 7fda49a00640 1 Processor -- start 2024-03-23T11:22:31.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.466+0000 7fda49a00640 1 -- start start 2024-03-23T11:22:31.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.467+0000 7fda49a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4415c9e0 0x7fda441607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:31.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.467+0000 7fda49a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fda440cf650 0x7fda440cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:31.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.467+0000 7fda49a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4406bfc0 con 0x7fda4415cdc0 2024-03-23T11:22:31.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.467+0000 7fda49a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda440d0020 con 0x7fda440cf650 2024-03-23T11:22:31.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.467+0000 7fda49a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda440d2630 con 0x7fda4415c9e0 2024-03-23T11:22:31.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.468+0000 7fda43400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fda440cf650 0x7fda440cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:31.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.468+0000 7fda43400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fda440cf650 0x7fda440cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44966/0 (socket says 172.21.15.62:44966) 2024-03-23T11:22:31.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.468+0000 7fda43400640 1 -- 172.21.15.62:0/1451647534 learned_addr learned my addr 172.21.15.62:0/1451647534 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:31.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.468+0000 7fda42a00640 1 --2- 172.21.15.62:0/1451647534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4415c9e0 0x7fda441607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:31.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.468+0000 7fda43400640 1 -- 172.21.15.62:0/1451647534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4415c9e0 msgr2=0x7fda441607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:31.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.468+0000 7fda43400640 1 --2- 172.21.15.62:0/1451647534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4415c9e0 0x7fda441607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.468+0000 7fda43400640 1 -- 172.21.15.62:0/1451647534 >> v1:172.21.15.120:6789/0 conn(0x7fda4415cdc0 legacy=0x7fda440d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:31.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.468+0000 7fda43400640 1 -- 172.21.15.62:0/1451647534 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda440b5260 con 0x7fda440cf650 2024-03-23T11:22:31.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.469+0000 7fda43400640 1 --2- 172.21.15.62:0/1451647534 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda440cf650 0x7fda440cfae0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009870 tx=0x7fda3c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae36148c084f4ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:31.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.470+0000 7fda41600640 1 -- 172.21.15.62:0/1451647534 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c031640 con 0x7fda440cf650 2024-03-23T11:22:31.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.470+0000 7fda41600640 1 -- 172.21.15.62:0/1451647534 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c0317a0 con 0x7fda440cf650 2024-03-23T11:22:31.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.470+0000 7fda41600640 1 -- 172.21.15.62:0/1451647534 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c031b30 con 0x7fda440cf650 2024-03-23T11:22:31.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.470+0000 7fda49a00640 1 -- 172.21.15.62:0/1451647534 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda440cf650 msgr2=0x7fda440cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:31.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.470+0000 7fda49a00640 1 --2- 172.21.15.62:0/1451647534 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda440cf650 0x7fda440cfae0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009870 tx=0x7fda3c02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.470+0000 7fda49a00640 1 -- 172.21.15.62:0/1451647534 shutdown_connections 2024-03-23T11:22:31.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.470+0000 7fda49a00640 1 --2- 172.21.15.62:0/1451647534 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda440cf650 0x7fda440cfae0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.470+0000 7fda49a00640 1 --2- 172.21.15.62:0/1451647534 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4415c9e0 0x7fda441607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.470+0000 7fda49a00640 1 -- 172.21.15.62:0/1451647534 >> 172.21.15.62:0/1451647534 conn(0x7fda44071570 msgr2=0x7fda44071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:31.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.471+0000 7fda49a00640 1 -- 172.21.15.62:0/1451647534 shutdown_connections 2024-03-23T11:22:31.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.471+0000 7fda49a00640 1 -- 172.21.15.62:0/1451647534 wait complete. 2024-03-23T11:22:31.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.472+0000 7fda49a00640 1 Processor -- start 2024-03-23T11:22:31.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.473+0000 7fda49a00640 1 -- start start 2024-03-23T11:22:31.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.473+0000 7fda49a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda440cf650 0x7fda44159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:31.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.473+0000 7fda49a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4415c9e0 0x7fda440cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:31.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.473+0000 7fda49a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda4415cdc0 0x7fda440cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:31.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda49a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda441f3430 con 0x7fda4415cdc0 2024-03-23T11:22:31.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda49a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda440aacc0 con 0x7fda440cf650 2024-03-23T11:22:31.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda49a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda440aafa0 con 0x7fda4415c9e0 2024-03-23T11:22:31.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda43400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda4415cdc0 0x7fda440cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda43400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda4415cdc0 0x7fda440cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44974/0 (socket says 172.21.15.62:44974) 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda43400640 1 -- 172.21.15.62:0/1690211975 learned_addr learned my addr 172.21.15.62:0/1690211975 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda42a00640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda440cf650 0x7fda44159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda43400640 1 -- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4415c9e0 msgr2=0x7fda440cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda43400640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4415c9e0 0x7fda440cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda43400640 1 -- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda440cf650 msgr2=0x7fda44159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda43400640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda440cf650 0x7fda44159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.474+0000 7fda43400640 1 -- 172.21.15.62:0/1690211975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c0094d0 con 0x7fda4415cdc0 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.475+0000 7fda43400640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda4415cdc0 0x7fda440cbf10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009870 tx=0x7fda3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.475+0000 7fda3aa00640 1 -- 172.21.15.62:0/1690211975 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c03ac60 con 0x7fda4415cdc0 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.475+0000 7fda3aa00640 1 -- 172.21.15.62:0/1690211975 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c03ade0 con 0x7fda4415cdc0 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.475+0000 7fda49a00640 1 -- 172.21.15.62:0/1690211975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda440ab280 con 0x7fda4415cdc0 2024-03-23T11:22:31.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.475+0000 7fda3aa00640 1 -- 172.21.15.62:0/1690211975 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c044630 con 0x7fda4415cdc0 2024-03-23T11:22:31.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.475+0000 7fda49a00640 1 -- 172.21.15.62:0/1690211975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4415a320 con 0x7fda4415cdc0 2024-03-23T11:22:31.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.476+0000 7fda49a00640 1 -- 172.21.15.62:0/1690211975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda440ab7a0 con 0x7fda4415cdc0 2024-03-23T11:22:31.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.477+0000 7fda3aa00640 1 -- 172.21.15.62:0/1690211975 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fda3c02f050 con 0x7fda4415cdc0 2024-03-23T11:22:31.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.478+0000 7fda3aa00640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fda2406a7a0 0x7fda2406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:31.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.478+0000 7fda3aa00640 1 -- 172.21.15.62:0/1690211975 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fda3c0aabd0 con 0x7fda4415cdc0 2024-03-23T11:22:31.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.479+0000 7fda42a00640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fda2406a7a0 0x7fda2406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:31.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.480+0000 7fda42a00640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fda2406a7a0 0x7fda2406cc60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fda34006fd0 tx=0x7fda34008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:31.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.483+0000 7fda3aa00640 1 -- 172.21.15.62:0/1690211975 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fda440ab7a0 con 0x7fda4415cdc0 2024-03-23T11:22:31.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.722+0000 7fda49a00640 1 -- 172.21.15.62:0/1690211975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fda440ab7a0 con 0x7fda4415cdc0 2024-03-23T11:22:31.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.737+0000 7fda3aa00640 1 -- 172.21.15.62:0/1690211975 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v29) v1 ==== 172+0+0 (secure 0 0 0) 0x7fda440ab7a0 con 0x7fda4415cdc0 2024-03-23T11:22:31.737 INFO:teuthology.orchestra.run.smithi062.stderr:new fs with metadata pool 8 and data pool 9 2024-03-23T11:22:31.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.742+0000 7fda49a00640 1 -- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fda2406a7a0 msgr2=0x7fda2406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:31.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.742+0000 7fda49a00640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fda2406a7a0 0x7fda2406cc60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fda34006fd0 tx=0x7fda34008040 comp rx=0 tx=0).stop 2024-03-23T11:22:31.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.742+0000 7fda49a00640 1 -- 172.21.15.62:0/1690211975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda4415cdc0 msgr2=0x7fda440cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:31.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.742+0000 7fda49a00640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda4415cdc0 0x7fda440cbf10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009870 tx=0x7fda3c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.742+0000 7fda49a00640 1 -- 172.21.15.62:0/1690211975 shutdown_connections 2024-03-23T11:22:31.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.742+0000 7fda49a00640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fda2406a7a0 0x7fda2406cc60 secure :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fda34006fd0 tx=0x7fda34008040 comp rx=0 tx=0).stop 2024-03-23T11:22:31.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.742+0000 7fda49a00640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda4415cdc0 0x7fda440cbf10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.742+0000 7fda49a00640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4415c9e0 0x7fda440cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.742+0000 7fda49a00640 1 --2- 172.21.15.62:0/1690211975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda440cf650 0x7fda44159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.742+0000 7fda49a00640 1 -- 172.21.15.62:0/1690211975 >> 172.21.15.62:0/1690211975 conn(0x7fda44071570 msgr2=0x7fda4415fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:31.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.746+0000 7fda49a00640 1 -- 172.21.15.62:0/1690211975 shutdown_connections 2024-03-23T11:22:31.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.747+0000 7fda49a00640 1 -- 172.21.15.62:0/1690211975 wait complete. 2024-03-23T11:22:31.777 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:22:31.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.953+0000 7fbd50c00640 1 Processor -- start 2024-03-23T11:22:31.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.953+0000 7fbd50c00640 1 -- start start 2024-03-23T11:22:31.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.954+0000 7fbd50c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbd4c15e190 0x7fbd4c0c4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:31.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.954+0000 7fbd50c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4c0990c0 con 0x7fbd4c0cf3f0 2024-03-23T11:22:31.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.954+0000 7fbd50c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4c099c60 con 0x7fbd4c15ddb0 2024-03-23T11:22:31.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.954+0000 7fbd50c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4c098a70 con 0x7fbd4c15e190 2024-03-23T11:22:31.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.955+0000 7fbd41400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbd4c15e190 0x7fbd4c0c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:31.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.955+0000 7fbd4a800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbd4c0cf3f0 0x7fbd4c0ce3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40050/0 (socket says 172.21.15.62:40050) 2024-03-23T11:22:31.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.955+0000 7fbd4a800640 1 -- 172.21.15.62:0/2681899233 learned_addr learned my addr 172.21.15.62:0/2681899233 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:31.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.955+0000 7fbd41400640 1 -- 172.21.15.62:0/2681899233 >> v1:172.21.15.120:6789/0 conn(0x7fbd4c15ddb0 legacy=0x7fbd4c099550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:31.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.955+0000 7fbd41400640 1 -- 172.21.15.62:0/2681899233 >> v1:172.21.15.62:6789/0 conn(0x7fbd4c0cf3f0 legacy=0x7fbd4c0ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:31.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.955+0000 7fbd41400640 1 -- 172.21.15.62:0/2681899233 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c098d50 con 0x7fbd4c15e190 2024-03-23T11:22:31.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.955+0000 7fbd49400640 1 -- 172.21.15.62:0/2681899233 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437474187 0 0) 0x7fbd4c0990c0 con 0x7fbd4c0cf3f0 2024-03-23T11:22:31.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.956+0000 7fbd41400640 1 --2- 172.21.15.62:0/2681899233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd4c15e190 0x7fbd4c0c4320 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009d90 tx=0x7fbd3402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5060ec6554fd9ee3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:31.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.957+0000 7fbd49400640 1 -- 172.21.15.62:0/2681899233 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3402ccf0 con 0x7fbd4c15e190 2024-03-23T11:22:31.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.957+0000 7fbd49400640 1 -- 172.21.15.62:0/2681899233 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3402ce50 con 0x7fbd4c15e190 2024-03-23T11:22:31.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.957+0000 7fbd49400640 1 -- 172.21.15.62:0/2681899233 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd34034660 con 0x7fbd4c15e190 2024-03-23T11:22:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.957+0000 7fbd50c00640 1 -- 172.21.15.62:0/2681899233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd4c15e190 msgr2=0x7fbd4c0c4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.957+0000 7fbd50c00640 1 --2- 172.21.15.62:0/2681899233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd4c15e190 0x7fbd4c0c4320 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009d90 tx=0x7fbd3402bf40 comp rx=0 tx=0).stop 2024-03-23T11:22:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.957+0000 7fbd50c00640 1 -- 172.21.15.62:0/2681899233 shutdown_connections 2024-03-23T11:22:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.957+0000 7fbd50c00640 1 --2- 172.21.15.62:0/2681899233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd4c15e190 0x7fbd4c0c4320 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.957+0000 7fbd50c00640 1 -- 172.21.15.62:0/2681899233 >> 172.21.15.62:0/2681899233 conn(0x7fbd4c0b22c0 msgr2=0x7fbd4c0b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.957+0000 7fbd50c00640 1 -- 172.21.15.62:0/2681899233 shutdown_connections 2024-03-23T11:22:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.957+0000 7fbd50c00640 1 -- 172.21.15.62:0/2681899233 wait complete. 2024-03-23T11:22:31.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.959+0000 7fbd50c00640 1 Processor -- start 2024-03-23T11:22:31.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.959+0000 7fbd50c00640 1 -- start start 2024-03-23T11:22:31.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.960+0000 7fbd50c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd4c0cf3f0 0x7fbd4c159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:31.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.960+0000 7fbd50c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd4c15ddb0 0x7fbd4c158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:31.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.960+0000 7fbd50c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd4c15e190 0x7fbd4c15b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:31.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.960+0000 7fbd50c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c15ba30 con 0x7fbd4c15ddb0 2024-03-23T11:22:31.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.960+0000 7fbd50c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c15a0b0 con 0x7fbd4c15e190 2024-03-23T11:22:31.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.960+0000 7fbd50c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c1f0930 con 0x7fbd4c0cf3f0 2024-03-23T11:22:31.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.960+0000 7fbd41400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd4c15ddb0 0x7fbd4c158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:31.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.961+0000 7fbd41400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd4c15ddb0 0x7fbd4c158090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44994/0 (socket says 172.21.15.62:44994) 2024-03-23T11:22:31.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.961+0000 7fbd41400640 1 -- 172.21.15.62:0/937893004 learned_addr learned my addr 172.21.15.62:0/937893004 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:31.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.961+0000 7fbd4a800640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd4c15e190 0x7fbd4c15b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:31.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.961+0000 7fbd49e00640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd4c0cf3f0 0x7fbd4c159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:31.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.961+0000 7fbd41400640 1 -- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd4c0cf3f0 msgr2=0x7fbd4c159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:31.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.961+0000 7fbd41400640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd4c0cf3f0 0x7fbd4c159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.961+0000 7fbd41400640 1 -- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd4c15e190 msgr2=0x7fbd4c15b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:31.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.961+0000 7fbd41400640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd4c15e190 0x7fbd4c15b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:31.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.961+0000 7fbd41400640 1 -- 172.21.15.62:0/937893004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd34009a70 con 0x7fbd4c15ddb0 2024-03-23T11:22:31.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.961+0000 7fbd49e00640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd4c0cf3f0 0x7fbd4c159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:31.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.961+0000 7fbd41400640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd4c15ddb0 0x7fbd4c158090 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fbd34037040 tx=0x7fbd34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:31.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.962+0000 7fbd42a00640 1 -- 172.21.15.62:0/937893004 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3403e670 con 0x7fbd4c15ddb0 2024-03-23T11:22:31.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.962+0000 7fbd42a00640 1 -- 172.21.15.62:0/937893004 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3403e7d0 con 0x7fbd4c15ddb0 2024-03-23T11:22:31.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.962+0000 7fbd42a00640 1 -- 172.21.15.62:0/937893004 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3403eb80 con 0x7fbd4c15ddb0 2024-03-23T11:22:31.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.962+0000 7fbd50c00640 1 -- 172.21.15.62:0/937893004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4c159810 con 0x7fbd4c15ddb0 2024-03-23T11:22:31.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.962+0000 7fbd50c00640 1 -- 172.21.15.62:0/937893004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4c159dd0 con 0x7fbd4c15ddb0 2024-03-23T11:22:31.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.963+0000 7fbd50c00640 1 -- 172.21.15.62:0/937893004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4c1f0700 con 0x7fbd4c15ddb0 2024-03-23T11:22:31.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.964+0000 7fbd42a00640 1 -- 172.21.15.62:0/937893004 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbd3403b040 con 0x7fbd4c15ddb0 2024-03-23T11:22:31.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.965+0000 7fbd42a00640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbd2806a7a0 0x7fbd2806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:31.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.965+0000 7fbd42a00640 1 -- 172.21.15.62:0/937893004 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd340ab1e0 con 0x7fbd4c15ddb0 2024-03-23T11:22:31.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.965+0000 7fbd49e00640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbd2806a7a0 0x7fbd2806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:31.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.966+0000 7fbd49e00640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbd2806a7a0 0x7fbd2806cc60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c003eb0 tx=0x7fbd3c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:31.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:31.970+0000 7fbd42a00640 1 -- 172.21.15.62:0/937893004 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd4c1f0700 con 0x7fbd4c15ddb0 2024-03-23T11:22:32.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.160+0000 7fbd50c00640 1 -- 172.21.15.62:0/937893004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd4c1f0700 con 0x7fbd4c15ddb0 2024-03-23T11:22:32.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.163+0000 7fbd42a00640 1 -- 172.21.15.62:0/937893004 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16110 (secure 0 0 0) 0x7fbd4c1f0700 con 0x7fbd4c15ddb0 2024-03-23T11:22:32.162 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:32.162 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":61,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:22:31.724998+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:22:29.767222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-23T11:22:30.738596+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:22:32.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.165+0000 7fbd50c00640 1 -- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbd2806a7a0 msgr2=0x7fbd2806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.165+0000 7fbd50c00640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbd2806a7a0 0x7fbd2806cc60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c003eb0 tx=0x7fbd3c009040 comp rx=0 tx=0).stop 2024-03-23T11:22:32.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.165+0000 7fbd50c00640 1 -- 172.21.15.62:0/937893004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd4c15ddb0 msgr2=0x7fbd4c158090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.166+0000 7fbd50c00640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd4c15ddb0 0x7fbd4c158090 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fbd34037040 tx=0x7fbd34004a40 comp rx=0 tx=0).stop 2024-03-23T11:22:32.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.166+0000 7fbd50c00640 1 -- 172.21.15.62:0/937893004 shutdown_connections 2024-03-23T11:22:32.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.166+0000 7fbd50c00640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbd2806a7a0 0x7fbd2806cc60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.166+0000 7fbd50c00640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd4c15ddb0 0x7fbd4c158090 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.166+0000 7fbd50c00640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd4c15e190 0x7fbd4c15b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.166+0000 7fbd50c00640 1 --2- 172.21.15.62:0/937893004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd4c0cf3f0 0x7fbd4c159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.166+0000 7fbd50c00640 1 -- 172.21.15.62:0/937893004 >> 172.21.15.62:0/937893004 conn(0x7fbd4c0b22c0 msgr2=0x7fbd4c0c6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:32.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.166+0000 7fbd50c00640 1 -- 172.21.15.62:0/937893004 shutdown_connections 2024-03-23T11:22:32.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.166+0000 7fbd50c00640 1 -- 172.21.15.62:0/937893004 wait complete. 2024-03-23T11:22:32.181 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:22:32.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.396+0000 7f9379000640 1 Processor -- start 2024-03-23T11:22:32.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.396+0000 7f9379000640 1 -- start start 2024-03-23T11:22:32.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.398+0000 7f9379000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9374153360 0x7f93740b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.398+0000 7f9379000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f937415d1e0 0x7f9374070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.398+0000 7f9379000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9374071300 con 0x7f9374070ed0 2024-03-23T11:22:32.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.398+0000 7f9379000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9374159f70 con 0x7f937415d1e0 2024-03-23T11:22:32.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.398+0000 7f9379000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9374153740 con 0x7f9374153360 2024-03-23T11:22:32.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.398+0000 7f9373e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9374070ed0 0x7f93740b1400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40056/0 (socket says 172.21.15.62:40056) 2024-03-23T11:22:32.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.398+0000 7f9373e00640 1 -- 172.21.15.62:0/3883792799 learned_addr learned my addr 172.21.15.62:0/3883792799 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:32.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.398+0000 7f9372a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f937415d1e0 0x7f9374070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.398+0000 7f9373400640 1 --2- 172.21.15.62:0/3883792799 >> v2:172.21.15.120:3300/0 conn(0x7f9374153360 0x7f93740b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.399+0000 7f9372a00640 1 -- 172.21.15.62:0/3883792799 >> v2:172.21.15.120:3300/0 conn(0x7f9374153360 msgr2=0x7f93740b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:32.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.399+0000 7f9372a00640 1 --2- 172.21.15.62:0/3883792799 >> v2:172.21.15.120:3300/0 conn(0x7f9374153360 0x7f93740b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.399+0000 7f9372a00640 1 -- 172.21.15.62:0/3883792799 >> v1:172.21.15.62:6789/0 conn(0x7f9374070ed0 legacy=0x7f93740b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.399+0000 7f9372a00640 1 -- 172.21.15.62:0/3883792799 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937406d130 con 0x7f937415d1e0 2024-03-23T11:22:32.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.399+0000 7f9372000640 1 -- 172.21.15.62:0/3883792799 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161726983 0 0) 0x7f9374071300 con 0x7f9374070ed0 2024-03-23T11:22:32.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.400+0000 7f9372a00640 1 --2- 172.21.15.62:0/3883792799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f937415d1e0 0x7f9374070990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f937406d330 tx=0x7f936002c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfdd5089563309f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:32.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.400+0000 7f9372000640 1 -- 172.21.15.62:0/3883792799 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9360030810 con 0x7f937415d1e0 2024-03-23T11:22:32.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.400+0000 7f9372000640 1 -- 172.21.15.62:0/3883792799 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9360030970 con 0x7f937415d1e0 2024-03-23T11:22:32.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.400+0000 7f9372000640 1 -- 172.21.15.62:0/3883792799 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9360030d00 con 0x7f937415d1e0 2024-03-23T11:22:32.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.401+0000 7f9379000640 1 -- 172.21.15.62:0/3883792799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f937415d1e0 msgr2=0x7f9374070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.401+0000 7f9379000640 1 --2- 172.21.15.62:0/3883792799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f937415d1e0 0x7f9374070990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f937406d330 tx=0x7f936002c9f0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.401+0000 7f9379000640 1 -- 172.21.15.62:0/3883792799 shutdown_connections 2024-03-23T11:22:32.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.401+0000 7f9379000640 1 --2- 172.21.15.62:0/3883792799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f937415d1e0 0x7f9374070990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.401+0000 7f9379000640 1 --2- 172.21.15.62:0/3883792799 >> v2:172.21.15.120:3300/0 conn(0x7f9374153360 0x7f93740b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.401+0000 7f9379000640 1 -- 172.21.15.62:0/3883792799 >> 172.21.15.62:0/3883792799 conn(0x7f9374066050 msgr2=0x7f9374069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:32.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.401+0000 7f9379000640 1 -- 172.21.15.62:0/3883792799 shutdown_connections 2024-03-23T11:22:32.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.401+0000 7f9379000640 1 -- 172.21.15.62:0/3883792799 wait complete. 2024-03-23T11:22:32.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.403+0000 7f9379000640 1 Processor -- start 2024-03-23T11:22:32.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.403+0000 7f9379000640 1 -- start start 2024-03-23T11:22:32.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9379000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9374070ed0 0x7f9374148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9379000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9374153360 0x7f93741e26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9379000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f937415d1e0 0x7f93741e2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9379000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93740afa50 con 0x7f9374070ed0 2024-03-23T11:22:32.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9379000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93740afcf0 con 0x7f9374153360 2024-03-23T11:22:32.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9379000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93740affd0 con 0x7f937415d1e0 2024-03-23T11:22:32.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9373400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9374070ed0 0x7f9374148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9373400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9374070ed0 0x7f9374148540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45002/0 (socket says 172.21.15.62:45002) 2024-03-23T11:22:32.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9373400640 1 -- 172.21.15.62:0/2014141382 learned_addr learned my addr 172.21.15.62:0/2014141382 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:32.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9373400640 1 -- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f937415d1e0 msgr2=0x7f93741e2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9373400640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f937415d1e0 0x7f93741e2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9373400640 1 -- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9374153360 msgr2=0x7f93741e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9373400640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9374153360 0x7f93741e26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.404+0000 7f9373400640 1 -- 172.21.15.62:0/2014141382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936000a520 con 0x7f9374070ed0 2024-03-23T11:22:32.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.405+0000 7f9373400640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9374070ed0 0x7f9374148540 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f936400d800 tx=0x7f936400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:32.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.405+0000 7f936b400640 1 -- 172.21.15.62:0/2014141382 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936400b6c0 con 0x7f9374070ed0 2024-03-23T11:22:32.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.405+0000 7f936b400640 1 -- 172.21.15.62:0/2014141382 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936400b820 con 0x7f9374070ed0 2024-03-23T11:22:32.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.405+0000 7f936b400640 1 -- 172.21.15.62:0/2014141382 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936400bbb0 con 0x7f9374070ed0 2024-03-23T11:22:32.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.405+0000 7f9379000640 1 -- 172.21.15.62:0/2014141382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93740b02b0 con 0x7f9374070ed0 2024-03-23T11:22:32.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.405+0000 7f9379000640 1 -- 172.21.15.62:0/2014141382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9374148bf0 con 0x7f9374070ed0 2024-03-23T11:22:32.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.406+0000 7f9379000640 1 -- 172.21.15.62:0/2014141382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937404e030 con 0x7f9374070ed0 2024-03-23T11:22:32.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.407+0000 7f936b400640 1 -- 172.21.15.62:0/2014141382 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f936400bd10 con 0x7f9374070ed0 2024-03-23T11:22:32.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.408+0000 7f936b400640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f935406a7a0 0x7f935406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.408+0000 7f936b400640 1 -- 172.21.15.62:0/2014141382 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93640893e0 con 0x7f9374070ed0 2024-03-23T11:22:32.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.409+0000 7f9372a00640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f935406a7a0 0x7f935406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.410+0000 7f9372a00640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f935406a7a0 0x7f935406cc60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9360037000 tx=0x7f9360031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:32.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.412+0000 7f936b400640 1 -- 172.21.15.62:0/2014141382 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9364054760 con 0x7f9374070ed0 2024-03-23T11:22:32.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.615+0000 7f9379000640 1 -- 172.21.15.62:0/2014141382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93740b1550 con 0x7f9374070ed0 2024-03-23T11:22:32.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.617+0000 7f936b400640 1 -- 172.21.15.62:0/2014141382 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16110 (secure 0 0 0) 0x7f936405e030 con 0x7f9374070ed0 2024-03-23T11:22:32.616 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:32.616 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":61,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:22:31.724998+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:22:29.767222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-23T11:22:30.738596+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:22:32.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.620+0000 7f9379000640 1 -- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f935406a7a0 msgr2=0x7f935406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.620+0000 7f9379000640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f935406a7a0 0x7f935406cc60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9360037000 tx=0x7f9360031000 comp rx=0 tx=0).stop 2024-03-23T11:22:32.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.620+0000 7f9379000640 1 -- 172.21.15.62:0/2014141382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9374070ed0 msgr2=0x7f9374148540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.620+0000 7f9379000640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9374070ed0 0x7f9374148540 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f936400d800 tx=0x7f936400dcc0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.620+0000 7f9379000640 1 -- 172.21.15.62:0/2014141382 shutdown_connections 2024-03-23T11:22:32.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.620+0000 7f9379000640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f935406a7a0 0x7f935406cc60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.620+0000 7f9379000640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f937415d1e0 0x7f93741e2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.620+0000 7f9379000640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9374153360 0x7f93741e26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.620+0000 7f9379000640 1 --2- 172.21.15.62:0/2014141382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9374070ed0 0x7f9374148540 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.620+0000 7f9379000640 1 -- 172.21.15.62:0/2014141382 >> 172.21.15.62:0/2014141382 conn(0x7f9374066050 msgr2=0x7f93740b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:32.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.621+0000 7f9379000640 1 -- 172.21.15.62:0/2014141382 shutdown_connections 2024-03-23T11:22:32.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.621+0000 7f9379000640 1 -- 172.21.15.62:0/2014141382 wait complete. 2024-03-23T11:22:32.639 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T11:22:32.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.870+0000 7ff91f600640 1 Processor -- start 2024-03-23T11:22:32.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.870+0000 7ff91f600640 1 -- start start 2024-03-23T11:22:32.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.871+0000 7ff91f600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff91815cb90 0x7ff9180b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.871+0000 7ff91f600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9180b5ad0 con 0x7ff9180cde10 2024-03-23T11:22:32.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.872+0000 7ff91f600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91806b820 con 0x7ff91806b040 2024-03-23T11:22:32.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.872+0000 7ff91f600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff91806c0c0 con 0x7ff91815cb90 2024-03-23T11:22:32.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.872+0000 7ff91d600640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff9180cde10 0x7ff9180d1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40068/0 (socket says 172.21.15.62:40068) 2024-03-23T11:22:32.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.872+0000 7ff91d600640 1 -- 172.21.15.62:0/2703581757 learned_addr learned my addr 172.21.15.62:0/2703581757 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:32.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.872+0000 7ff91cc00640 1 --2- 172.21.15.62:0/2703581757 >> v2:172.21.15.120:3300/0 conn(0x7ff91815cb90 0x7ff9180b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.872+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891756161 0 0) 0x7ff9180b5ad0 con 0x7ff9180cde10 2024-03-23T11:22:32.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8fc003520 con 0x7ff9180cde10 2024-03-23T11:22:32.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 337365429 0 0) 0x7ff8fc003520 con 0x7ff9180cde10 2024-03-23T11:22:32.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 >> v2:172.21.15.120:3300/0 conn(0x7ff91815cb90 msgr2=0x7ff9180b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:32.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 --2- 172.21.15.62:0/2703581757 >> v2:172.21.15.120:3300/0 conn(0x7ff91815cb90 0x7ff9180b84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 >> v1:172.21.15.120:6789/0 conn(0x7ff91806b040 legacy=0x7ff9180cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91806bb00 con 0x7ff9180cde10 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7ff910001670 con 0x7ff9180cde10 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 >> v1:172.21.15.62:6789/0 conn(0x7ff9180cde10 legacy=0x7ff9180d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 --2- 172.21.15.62:0/2703581757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8fc003970 0x7ff8fc003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 --2- 172.21.15.62:0/2703581757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 0x7ff8fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 --2- 172.21.15.62:0/2703581757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8fc005180 0x7ff8fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91806c0c0 con 0x7ff8fc0047c0 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc00bfd0 con 0x7ff8fc003970 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc00c190 con 0x7ff8fc005180 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.873+0000 7ff917e00640 1 --2- 172.21.15.62:0/2703581757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 0x7ff8fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.874+0000 7ff91cc00640 1 --2- 172.21.15.62:0/2703581757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8fc003970 0x7ff8fc003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.874+0000 7ff91d600640 1 --2- 172.21.15.62:0/2703581757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8fc005180 0x7ff8fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.874+0000 7ff917e00640 1 -- 172.21.15.62:0/2703581757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8fc005180 msgr2=0x7ff8fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:32.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.874+0000 7ff917e00640 1 --2- 172.21.15.62:0/2703581757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8fc005180 0x7ff8fc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.874+0000 7ff917e00640 1 -- 172.21.15.62:0/2703581757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8fc003970 msgr2=0x7ff8fc003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:32.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.874+0000 7ff917e00640 1 --2- 172.21.15.62:0/2703581757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8fc003970 0x7ff8fc003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.874+0000 7ff917e00640 1 -- 172.21.15.62:0/2703581757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc003520 con 0x7ff8fc0047c0 2024-03-23T11:22:32.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.874+0000 7ff91d600640 1 -- 172.21.15.62:0/2703581757 reap_dead start 2024-03-23T11:22:32.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.875+0000 7ff917e00640 1 --2- 172.21.15.62:0/2703581757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 0x7ff8fc004c40 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff900001860 tx=0x7ff8fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb0331b6918365b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:32.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.875+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90002f050 con 0x7ff8fc0047c0 2024-03-23T11:22:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.875+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff900033610 con 0x7ff8fc0047c0 2024-03-23T11:22:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.875+0000 7ff917400640 1 -- 172.21.15.62:0/2703581757 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9000339c0 con 0x7ff8fc0047c0 2024-03-23T11:22:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.876+0000 7ff91f600640 1 -- 172.21.15.62:0/2703581757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 msgr2=0x7ff8fc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.876+0000 7ff91f600640 1 --2- 172.21.15.62:0/2703581757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 0x7ff8fc004c40 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff900001860 tx=0x7ff8fc0057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.876+0000 7ff91f600640 1 -- 172.21.15.62:0/2703581757 shutdown_connections 2024-03-23T11:22:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.876+0000 7ff91f600640 1 --2- 172.21.15.62:0/2703581757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 0x7ff8fc004c40 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.876+0000 7ff91f600640 1 -- 172.21.15.62:0/2703581757 >> 172.21.15.62:0/2703581757 conn(0x7ff9180cf7d0 msgr2=0x7ff9180cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.876+0000 7ff91f600640 1 -- 172.21.15.62:0/2703581757 shutdown_connections 2024-03-23T11:22:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.876+0000 7ff91f600640 1 -- 172.21.15.62:0/2703581757 wait complete. 2024-03-23T11:22:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.878+0000 7ff91f600640 1 Processor -- start 2024-03-23T11:22:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.878+0000 7ff91f600640 1 -- start start 2024-03-23T11:22:32.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.879+0000 7ff91f600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 0x7ff9180b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.879+0000 7ff91f600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9181f22b0 0x7ff9180c8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.879+0000 7ff91f600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9180c9a00 0x7ff918158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.879+0000 7ff91f600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9180d1010 con 0x7ff8fc0047c0 2024-03-23T11:22:32.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.879+0000 7ff91f600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9180c89c0 con 0x7ff9181f22b0 2024-03-23T11:22:32.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.879+0000 7ff91f600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9180ca180 con 0x7ff9180c9a00 2024-03-23T11:22:32.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.879+0000 7ff91cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 0x7ff9180b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.879+0000 7ff91cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 0x7ff9180b4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45028/0 (socket says 172.21.15.62:45028) 2024-03-23T11:22:32.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.879+0000 7ff91cc00640 1 -- 172.21.15.62:0/2817032677 learned_addr learned my addr 172.21.15.62:0/2817032677 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.880+0000 7ff917e00640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9181f22b0 0x7ff9180c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.880+0000 7ff91d600640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9180c9a00 0x7ff918158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.880+0000 7ff91cc00640 1 -- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9180c9a00 msgr2=0x7ff918158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.880+0000 7ff91cc00640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9180c9a00 0x7ff918158180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.880+0000 7ff91cc00640 1 -- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9181f22b0 msgr2=0x7ff9180c8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.880+0000 7ff91cc00640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9181f22b0 0x7ff9180c8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.880+0000 7ff91cc00640 1 -- 172.21.15.62:0/2817032677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9180ca460 con 0x7ff8fc0047c0 2024-03-23T11:22:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.880+0000 7ff917e00640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9181f22b0 0x7ff9180c8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.880+0000 7ff91cc00640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 0x7ff9180b4d30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff908009f30 tx=0x7ff9080115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.881+0000 7ff914c00640 1 -- 172.21.15.62:0/2817032677 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90800e030 con 0x7ff8fc0047c0 2024-03-23T11:22:32.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.881+0000 7ff914c00640 1 -- 172.21.15.62:0/2817032677 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff908016470 con 0x7ff8fc0047c0 2024-03-23T11:22:32.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.881+0000 7ff914c00640 1 -- 172.21.15.62:0/2817032677 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff908016820 con 0x7ff8fc0047c0 2024-03-23T11:22:32.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.881+0000 7ff91f600640 1 -- 172.21.15.62:0/2817032677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9180ca740 con 0x7ff8fc0047c0 2024-03-23T11:22:32.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.881+0000 7ff91f600640 1 -- 172.21.15.62:0/2817032677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9180cad00 con 0x7ff8fc0047c0 2024-03-23T11:22:32.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.882+0000 7ff91f600640 1 -- 172.21.15.62:0/2817032677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9180cae90 con 0x7ff8fc0047c0 2024-03-23T11:22:32.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.883+0000 7ff914c00640 1 -- 172.21.15.62:0/2817032677 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff908016980 con 0x7ff8fc0047c0 2024-03-23T11:22:32.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.884+0000 7ff914c00640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8f406a7a0 0x7ff8f406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:32.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.884+0000 7ff914c00640 1 -- 172.21.15.62:0/2817032677 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff90800f070 con 0x7ff8fc0047c0 2024-03-23T11:22:32.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.884+0000 7ff917e00640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8f406a7a0 0x7ff8f406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:32.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.886+0000 7ff917e00640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8f406a7a0 0x7ff8f406cc60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff900036000 tx=0x7ff900031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:32.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:32.888+0000 7ff914c00640 1 -- 172.21.15.62:0/2817032677 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff908054de0 con 0x7ff8fc0047c0 2024-03-23T11:22:33.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.128+0000 7ff91f600640 1 -- 172.21.15.62:0/2817032677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff918066c90 con 0x7ff8fc0047c0 2024-03-23T11:22:33.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.746+0000 7ff914c00640 1 -- 172.21.15.62:0/2817032677 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff908058400 con 0x7ff8fc0047c0 2024-03-23T11:22:33.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.749+0000 7ff91f600640 1 -- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8f406a7a0 msgr2=0x7ff8f406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:33.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.749+0000 7ff91f600640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8f406a7a0 0x7ff8f406cc60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff900036000 tx=0x7ff900031000 comp rx=0 tx=0).stop 2024-03-23T11:22:33.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.749+0000 7ff91f600640 1 -- 172.21.15.62:0/2817032677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 msgr2=0x7ff9180b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:33.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.749+0000 7ff91f600640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 0x7ff9180b4d30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff908009f30 tx=0x7ff9080115a0 comp rx=0 tx=0).stop 2024-03-23T11:22:33.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.750+0000 7ff91f600640 1 -- 172.21.15.62:0/2817032677 shutdown_connections 2024-03-23T11:22:33.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.750+0000 7ff91f600640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8f406a7a0 0x7ff8f406cc60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:33.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.750+0000 7ff91f600640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9180c9a00 0x7ff918158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:33.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.750+0000 7ff91f600640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9181f22b0 0x7ff9180c8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:33.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.750+0000 7ff91f600640 1 --2- 172.21.15.62:0/2817032677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8fc0047c0 0x7ff9180b4d30 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:33.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.750+0000 7ff91f600640 1 -- 172.21.15.62:0/2817032677 >> 172.21.15.62:0/2817032677 conn(0x7ff9180cf7d0 msgr2=0x7ff9180d1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:33.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.750+0000 7ff91f600640 1 -- 172.21.15.62:0/2817032677 shutdown_connections 2024-03-23T11:22:33.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:33.750+0000 7ff91f600640 1 -- 172.21.15.62:0/2817032677 wait complete. 2024-03-23T11:22:33.769 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T11:22:33.769 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T11:22:33.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.000+0000 7fe8f5800640 1 Processor -- start 2024-03-23T11:22:33.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.000+0000 7fe8f5800640 1 -- start start 2024-03-23T11:22:33.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.001+0000 7fe8f5800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe8f015c9e0 0x7fe8f01607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.001+0000 7fe8f5800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe8f015cdc0 0x7fe8f00d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.001+0000 7fe8f5800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f006bfc0 con 0x7fe8f00cf650 2024-03-23T11:22:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.001+0000 7fe8f5800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f00ce140 con 0x7fe8f015c9e0 2024-03-23T11:22:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.001+0000 7fe8f5800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f00b4370 con 0x7fe8f015cdc0 2024-03-23T11:22:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.002+0000 7fe8ef400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe8f00cf650 0x7fe8f015d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40070/0 (socket says 172.21.15.62:40070) 2024-03-23T11:22:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.002+0000 7fe8ef400640 1 -- 172.21.15.62:0/2996537891 learned_addr learned my addr 172.21.15.62:0/2996537891 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.002+0000 7fe8eea00640 1 --2- 172.21.15.62:0/2996537891 >> v2:172.21.15.62:3300/0 conn(0x7fe8f015c9e0 0x7fe8f01607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.002+0000 7fe8ee000640 1 --2- 172.21.15.62:0/2996537891 >> v2:172.21.15.120:3300/0 conn(0x7fe8f015cdc0 0x7fe8f00d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.002+0000 7fe8ed600640 1 -- 172.21.15.62:0/2996537891 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470258836 0 0) 0x7fe8f006bfc0 con 0x7fe8f00cf650 2024-03-23T11:22:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.002+0000 7fe8ed600640 1 -- 172.21.15.62:0/2996537891 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d0003520 con 0x7fe8f00cf650 2024-03-23T11:22:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.002+0000 7fe8eea00640 1 -- 172.21.15.62:0/2996537891 >> v2:172.21.15.120:3300/0 conn(0x7fe8f015cdc0 msgr2=0x7fe8f00d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.002+0000 7fe8eea00640 1 --2- 172.21.15.62:0/2996537891 >> v2:172.21.15.120:3300/0 conn(0x7fe8f015cdc0 0x7fe8f00d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.002+0000 7fe8eea00640 1 -- 172.21.15.62:0/2996537891 >> v1:172.21.15.62:6789/0 conn(0x7fe8f00cf650 legacy=0x7fe8f015d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.002+0000 7fe8eea00640 1 -- 172.21.15.62:0/2996537891 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f00b5260 con 0x7fe8f015c9e0 2024-03-23T11:22:34.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.004+0000 7fe8eea00640 1 --2- 172.21.15.62:0/2996537891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8f015c9e0 0x7fe8f01607c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe8f00b89a0 tx=0x7fe8e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a618dfaebdcd50b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:34.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.004+0000 7fe8ed600640 1 -- 172.21.15.62:0/2996537891 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0031830 con 0x7fe8f015c9e0 2024-03-23T11:22:34.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.004+0000 7fe8ed600640 1 -- 172.21.15.62:0/2996537891 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e0031990 con 0x7fe8f015c9e0 2024-03-23T11:22:34.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.004+0000 7fe8ed600640 1 -- 172.21.15.62:0/2996537891 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0031d20 con 0x7fe8f015c9e0 2024-03-23T11:22:34.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.005+0000 7fe8f5800640 1 -- 172.21.15.62:0/2996537891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8f015c9e0 msgr2=0x7fe8f01607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:34.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.005+0000 7fe8f5800640 1 --2- 172.21.15.62:0/2996537891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8f015c9e0 0x7fe8f01607c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe8f00b89a0 tx=0x7fe8e002b8d0 comp rx=0 tx=0).stop 2024-03-23T11:22:34.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.005+0000 7fe8f5800640 1 -- 172.21.15.62:0/2996537891 shutdown_connections 2024-03-23T11:22:34.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.005+0000 7fe8f5800640 1 --2- 172.21.15.62:0/2996537891 >> v2:172.21.15.120:3300/0 conn(0x7fe8f015cdc0 0x7fe8f00d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:34.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.005+0000 7fe8f5800640 1 --2- 172.21.15.62:0/2996537891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8f015c9e0 0x7fe8f01607c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:34.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.005+0000 7fe8f5800640 1 -- 172.21.15.62:0/2996537891 >> 172.21.15.62:0/2996537891 conn(0x7fe8f0071570 msgr2=0x7fe8f0071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:34.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.005+0000 7fe8f5800640 1 -- 172.21.15.62:0/2996537891 shutdown_connections 2024-03-23T11:22:34.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.005+0000 7fe8f5800640 1 -- 172.21.15.62:0/2996537891 wait complete. 2024-03-23T11:22:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.007+0000 7fe8f5800640 1 Processor -- start 2024-03-23T11:22:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.007+0000 7fe8f5800640 1 -- start start 2024-03-23T11:22:34.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8f5800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8f00cf650 0x7fe8f00d1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:34.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8f5800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8f015c9e0 0x7fe8f00a9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:34.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8f5800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8f015cdc0 0x7fe8f015d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:34.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8f5800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f00d2510 con 0x7fe8f015c9e0 2024-03-23T11:22:34.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8f5800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f00a9c50 con 0x7fe8f015cdc0 2024-03-23T11:22:34.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8f5800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f00a9f30 con 0x7fe8f00cf650 2024-03-23T11:22:34.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8ee000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8f015c9e0 0x7fe8f00a9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:34.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8ee000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8f015c9e0 0x7fe8f00a9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45056/0 (socket says 172.21.15.62:45056) 2024-03-23T11:22:34.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8ee000640 1 -- 172.21.15.62:0/2423474473 learned_addr learned my addr 172.21.15.62:0/2423474473 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:34.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8ef400640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8f015cdc0 0x7fe8f015d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:34.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8ee000640 1 -- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8f00cf650 msgr2=0x7fe8f00d1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:34.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8ee000640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8f00cf650 0x7fe8f00d1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:34.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8ee000640 1 -- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8f015cdc0 msgr2=0x7fe8f015d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:34.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8ee000640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8f015cdc0 0x7fe8f015d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:34.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8ee000640 1 -- 172.21.15.62:0/2423474473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e0009400 con 0x7fe8f015c9e0 2024-03-23T11:22:34.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.008+0000 7fe8ef400640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8f015cdc0 0x7fe8f015d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:34.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.009+0000 7fe8ee000640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8f015c9e0 0x7fe8f00a9300 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8007e40 tx=0x7fe8d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:34.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.009+0000 7fe8e6a00640 1 -- 172.21.15.62:0/2423474473 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8014620 con 0x7fe8f015c9e0 2024-03-23T11:22:34.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.009+0000 7fe8e6a00640 1 -- 172.21.15.62:0/2423474473 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d800ee90 con 0x7fe8f015c9e0 2024-03-23T11:22:34.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.009+0000 7fe8e6a00640 1 -- 172.21.15.62:0/2423474473 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d80149c0 con 0x7fe8f015c9e0 2024-03-23T11:22:34.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.009+0000 7fe8f5800640 1 -- 172.21.15.62:0/2423474473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f00aa210 con 0x7fe8f015c9e0 2024-03-23T11:22:34.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.009+0000 7fe8f5800640 1 -- 172.21.15.62:0/2423474473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f00aa7d0 con 0x7fe8f015c9e0 2024-03-23T11:22:34.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.010+0000 7fe8f5800640 1 -- 172.21.15.62:0/2423474473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f00aa960 con 0x7fe8f015c9e0 2024-03-23T11:22:34.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.011+0000 7fe8e6a00640 1 -- 172.21.15.62:0/2423474473 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe8d8014b20 con 0x7fe8f015c9e0 2024-03-23T11:22:34.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.012+0000 7fe8e6a00640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8cc06a7a0 0x7fe8cc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:34.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.012+0000 7fe8e6a00640 1 -- 172.21.15.62:0/2423474473 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8d8089710 con 0x7fe8f015c9e0 2024-03-23T11:22:34.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.013+0000 7fe8eea00640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8cc06a7a0 0x7fe8cc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:34.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.014+0000 7fe8eea00640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8cc06a7a0 0x7fe8cc06cc60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0036000 tx=0x7fe8e0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:34.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.017+0000 7fe8e6a00640 1 -- 172.21.15.62:0/2423474473 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe8d8054b20 con 0x7fe8f015c9e0 2024-03-23T11:22:34.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:34.261+0000 7fe8f5800640 1 -- 172.21.15.62:0/2423474473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe8f0066c90 con 0x7fe8f015c9e0 2024-03-23T11:22:35.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.265+0000 7fe8e6a00640 1 -- 172.21.15.62:0/2423474473 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe8d8058140 con 0x7fe8f015c9e0 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.268+0000 7fe8f5800640 1 -- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8cc06a7a0 msgr2=0x7fe8cc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.268+0000 7fe8f5800640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8cc06a7a0 0x7fe8cc06cc60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0036000 tx=0x7fe8e0030000 comp rx=0 tx=0).stop 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.268+0000 7fe8f5800640 1 -- 172.21.15.62:0/2423474473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8f015c9e0 msgr2=0x7fe8f00a9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.268+0000 7fe8f5800640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8f015c9e0 0x7fe8f00a9300 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8007e40 tx=0x7fe8d800e4b0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.268+0000 7fe8f5800640 1 -- 172.21.15.62:0/2423474473 shutdown_connections 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.268+0000 7fe8f5800640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8cc06a7a0 0x7fe8cc06cc60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.268+0000 7fe8f5800640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8f015c9e0 0x7fe8f00a9300 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.268+0000 7fe8f5800640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8f015cdc0 0x7fe8f015d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.268+0000 7fe8f5800640 1 --2- 172.21.15.62:0/2423474473 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe8f00cf650 0x7fe8f00d1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.268+0000 7fe8f5800640 1 -- 172.21.15.62:0/2423474473 >> 172.21.15.62:0/2423474473 conn(0x7fe8f0071570 msgr2=0x7fe8f00c91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.268+0000 7fe8f5800640 1 -- 172.21.15.62:0/2423474473 shutdown_connections 2024-03-23T11:22:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.269+0000 7fe8f5800640 1 -- 172.21.15.62:0/2423474473 wait complete. 2024-03-23T11:22:35.292 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T11:22:35.292 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:22:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.519+0000 7f1372000640 1 Processor -- start 2024-03-23T11:22:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.520+0000 7f1372000640 1 -- start start 2024-03-23T11:22:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.521+0000 7f1372000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c15fcf0 0x7f136c0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:35.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.521+0000 7f1372000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c0d14f0 con 0x7f136c160420 2024-03-23T11:22:35.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.521+0000 7f1372000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c0d3540 con 0x7f136c0b6dc0 2024-03-23T11:22:35.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.521+0000 7f1372000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136c0ce510 con 0x7f136c15fcf0 2024-03-23T11:22:35.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.521+0000 7f136be00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f136c160420 0x7f136c0d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40076/0 (socket says 172.21.15.62:40076) 2024-03-23T11:22:35.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.521+0000 7f136be00640 1 -- 172.21.15.62:0/887248645 learned_addr learned my addr 172.21.15.62:0/887248645 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.522+0000 7f1370c00640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c15fcf0 0x7f136c0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.522+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932268580 0 0) 0x7f136c0d14f0 con 0x7f136c160420 2024-03-23T11:22:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.522+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1350003520 con 0x7f136c160420 2024-03-23T11:22:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.522+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1528838344 0 0) 0x7f1350003520 con 0x7f136c160420 2024-03-23T11:22:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.522+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c15fcf0 msgr2=0x7f136c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.522+0000 7f136b400640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c15fcf0 0x7f136c0d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.522+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 >> v1:172.21.15.120:6789/0 conn(0x7f136c0b6dc0 legacy=0x7f136c0d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.522+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c0cdc70 con 0x7f136c160420 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.522+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f135c0019f0 con 0x7f136c160420 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.522+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 >> v1:172.21.15.62:6789/0 conn(0x7f136c160420 legacy=0x7f136c0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.522+0000 7f136b400640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c15fcf0 0x7f13500039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f136b400640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1350004000 0x7f13500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f136b400640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1350004d70 0x7f13500051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c0ce510 con 0x7f1350004d70 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135000bc80 con 0x7f1350004000 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135000be40 con 0x7f136c15fcf0 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f1371600640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1350004d70 0x7f13500051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f1370c00640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c15fcf0 0x7f13500039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f136be00640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1350004000 0x7f13500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f1371600640 1 -- 172.21.15.62:0/887248645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c15fcf0 msgr2=0x7f13500039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f1371600640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c15fcf0 0x7f13500039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f1371600640 1 -- 172.21.15.62:0/887248645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1350004000 msgr2=0x7f13500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:35.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f1371600640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1350004000 0x7f13500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.523+0000 7f1371600640 1 -- 172.21.15.62:0/887248645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1350003520 con 0x7f1350004d70 2024-03-23T11:22:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.525+0000 7f1371600640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1350004d70 0x7f13500051f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1364003db0 tx=0x7f13500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2136ba809b45a58 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.525+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1364032600 con 0x7f1350004d70 2024-03-23T11:22:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.525+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136402e050 con 0x7f1350004d70 2024-03-23T11:22:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.525+0000 7f136b400640 1 -- 172.21.15.62:0/887248645 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13640329e0 con 0x7f1350004d70 2024-03-23T11:22:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.525+0000 7f1372000640 1 -- 172.21.15.62:0/887248645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1350004d70 msgr2=0x7f13500051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.525+0000 7f1372000640 1 --2- 172.21.15.62:0/887248645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1350004d70 0x7f13500051f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1364003db0 tx=0x7f13500057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.525+0000 7f1371600640 1 -- 172.21.15.62:0/887248645 reap_dead start 2024-03-23T11:22:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.525+0000 7f1372000640 1 -- 172.21.15.62:0/887248645 shutdown_connections 2024-03-23T11:22:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.525+0000 7f1372000640 1 -- 172.21.15.62:0/887248645 >> 172.21.15.62:0/887248645 conn(0x7f136c089f60 msgr2=0x7f136c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.526+0000 7f1372000640 1 -- 172.21.15.62:0/887248645 shutdown_connections 2024-03-23T11:22:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.526+0000 7f1372000640 1 -- 172.21.15.62:0/887248645 wait complete. 2024-03-23T11:22:35.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.527+0000 7f1372000640 1 Processor -- start 2024-03-23T11:22:35.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.528+0000 7f1372000640 1 -- start start 2024-03-23T11:22:35.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.528+0000 7f1372000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1350004d70 0x7f136c0b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:35.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1372000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c0cb9f0 0x7f136c157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:35.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1372000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f136c1597f0 0x7f136c159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:35.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1372000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c15a1b0 con 0x7f136c1597f0 2024-03-23T11:22:35.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1372000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c0c78e0 con 0x7f1350004d70 2024-03-23T11:22:35.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1372000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136c0c7ba0 con 0x7f136c0cb9f0 2024-03-23T11:22:35.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1371600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f136c1597f0 0x7f136c159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:35.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1371600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f136c1597f0 0x7f136c159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45070/0 (socket says 172.21.15.62:45070) 2024-03-23T11:22:35.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1371600640 1 -- 172.21.15.62:0/1634494941 learned_addr learned my addr 172.21.15.62:0/1634494941 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:35.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1370c00640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1350004d70 0x7f136c0b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:35.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f136be00640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c0cb9f0 0x7f136c157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:35.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1371600640 1 -- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c0cb9f0 msgr2=0x7f136c157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:35.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1371600640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c0cb9f0 0x7f136c157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1371600640 1 -- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1350004d70 msgr2=0x7f136c0b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:35.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1371600640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1350004d70 0x7f136c0b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.529+0000 7f1371600640 1 -- 172.21.15.62:0/1634494941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c0c7e80 con 0x7f136c1597f0 2024-03-23T11:22:35.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.530+0000 7f1371600640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f136c1597f0 0x7f136c159c70 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1364003db0 tx=0x7f13640036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:35.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.530+0000 7f1368c00640 1 -- 172.21.15.62:0/1634494941 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136403d630 con 0x7f136c1597f0 2024-03-23T11:22:35.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.530+0000 7f1368c00640 1 -- 172.21.15.62:0/1634494941 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136403d790 con 0x7f136c1597f0 2024-03-23T11:22:35.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.530+0000 7f1368c00640 1 -- 172.21.15.62:0/1634494941 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136403db40 con 0x7f136c1597f0 2024-03-23T11:22:35.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.530+0000 7f1372000640 1 -- 172.21.15.62:0/1634494941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136c0c8160 con 0x7f136c1597f0 2024-03-23T11:22:35.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.530+0000 7f1372000640 1 -- 172.21.15.62:0/1634494941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136c15b7c0 con 0x7f136c1597f0 2024-03-23T11:22:35.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.532+0000 7f1372000640 1 -- 172.21.15.62:0/1634494941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136c0b55b0 con 0x7f136c1597f0 2024-03-23T11:22:35.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.532+0000 7f1368c00640 1 -- 172.21.15.62:0/1634494941 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f136403dca0 con 0x7f136c1597f0 2024-03-23T11:22:35.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.533+0000 7f1368c00640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f134806a7d0 0x7f134806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:35.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.533+0000 7f1368c00640 1 -- 172.21.15.62:0/1634494941 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1364000e40 con 0x7f136c1597f0 2024-03-23T11:22:35.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.534+0000 7f1370c00640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f134806a7d0 0x7f134806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:35.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.535+0000 7f1370c00640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f134806a7d0 0x7f134806cc90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1354006fd0 tx=0x7f1354008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:35.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.538+0000 7f1368c00640 1 -- 172.21.15.62:0/1634494941 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f136c0b55b0 con 0x7f136c1597f0 2024-03-23T11:22:35.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.773+0000 7f1372000640 1 -- 172.21.15.62:0/1634494941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f136c0b55b0 con 0x7f136c1597f0 2024-03-23T11:22:35.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.775+0000 7f1368c00640 1 -- 172.21.15.62:0/1634494941 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f136c0b55b0 con 0x7f136c1597f0 2024-03-23T11:22:35.773 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:35.773 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3287045135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3287045135},{"type":"v1","addr":"172.21.15.120:6835","nonce":3287045135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6066,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/2859430143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2859430143},{"type":"v1","addr":"172.21.15.62:6834","nonce":2859430143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2005768060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2005768060},{"type":"v1","addr":"172.21.15.62:6835","nonce":2005768060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:22:31.725865+0000","modified":"2024-03-23T11:22:35.259834+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6081},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6081":{"gid":6081,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/4158450746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":4158450746},{"type":"v1","addr":"172.21.15.120:6837","nonce":4158450746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T11:22:35.773 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 33 2024-03-23T11:22:35.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.777+0000 7f1372000640 1 -- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f134806a7d0 msgr2=0x7f134806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:35.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.777+0000 7f1372000640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f134806a7d0 0x7f134806cc90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1354006fd0 tx=0x7f1354008040 comp rx=0 tx=0).stop 2024-03-23T11:22:35.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.777+0000 7f1372000640 1 -- 172.21.15.62:0/1634494941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f136c1597f0 msgr2=0x7f136c159c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:35.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.777+0000 7f1372000640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f136c1597f0 0x7f136c159c70 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1364003db0 tx=0x7f13640036f0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.778+0000 7f1372000640 1 -- 172.21.15.62:0/1634494941 shutdown_connections 2024-03-23T11:22:35.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.778+0000 7f1372000640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f134806a7d0 0x7f134806cc90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.778+0000 7f1372000640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f136c1597f0 0x7f136c159c70 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.778+0000 7f1372000640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f136c0cb9f0 0x7f136c157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.778+0000 7f1372000640 1 --2- 172.21.15.62:0/1634494941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1350004d70 0x7f136c0b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:35.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.778+0000 7f1372000640 1 -- 172.21.15.62:0/1634494941 >> 172.21.15.62:0/1634494941 conn(0x7f136c089f60 msgr2=0x7f136c06bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:35.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.778+0000 7f1372000640 1 -- 172.21.15.62:0/1634494941 shutdown_connections 2024-03-23T11:22:35.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:35.778+0000 7f1372000640 1 -- 172.21.15.62:0/1634494941 wait complete. 2024-03-23T11:22:35.796 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:22:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.020+0000 7f0fbfe00640 1 Processor -- start 2024-03-23T11:22:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.020+0000 7f0fbfe00640 1 -- start start 2024-03-23T11:22:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.021+0000 7f0fbfe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0fc015fcf0 0x7f0fc00d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.021+0000 7f0fbfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc0160420 0x7f0fc0160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.021+0000 7f0fbfe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0fc00d06d0 0x7f0fc00d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.021+0000 7f0fbfe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc0160dc0 con 0x7f0fc015fcf0 2024-03-23T11:22:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.021+0000 7f0fbfe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc00d3540 con 0x7f0fc00d06d0 2024-03-23T11:22:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.021+0000 7f0fbfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00ce510 con 0x7f0fc0160420 2024-03-23T11:22:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.022+0000 7f0fbea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0fc015fcf0 0x7f0fc00d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.022+0000 7f0fbea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0fc015fcf0 0x7f0fc00d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45074/0 (socket says 172.21.15.62:45074) 2024-03-23T11:22:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.022+0000 7f0fbea00640 1 -- 172.21.15.62:0/1350291787 learned_addr learned my addr 172.21.15.62:0/1350291787 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.022+0000 7f0fbe000640 1 --2- 172.21.15.62:0/1350291787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc0160420 0x7f0fc0160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.022+0000 7f0fbf400640 1 --2- 172.21.15.62:0/1350291787 >> v2:172.21.15.120:3300/0 conn(0x7f0fc00d06d0 0x7f0fc00d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.022+0000 7f0fbea00640 1 -- 172.21.15.62:0/1350291787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc0160420 msgr2=0x7f0fc0160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.022+0000 7f0fbea00640 1 --2- 172.21.15.62:0/1350291787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc0160420 0x7f0fc0160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.022+0000 7f0fbea00640 1 -- 172.21.15.62:0/1350291787 >> v2:172.21.15.120:3300/0 conn(0x7f0fc00d06d0 msgr2=0x7f0fc00d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:36.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.022+0000 7f0fbea00640 1 --2- 172.21.15.62:0/1350291787 >> v2:172.21.15.120:3300/0 conn(0x7f0fc00d06d0 0x7f0fc00d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.022+0000 7f0fbea00640 1 -- 172.21.15.62:0/1350291787 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc00cdc70 con 0x7f0fc015fcf0 2024-03-23T11:22:36.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.023+0000 7f0fbea00640 1 --2- 172.21.15.62:0/1350291787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc015fcf0 0x7f0fc00d3000 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f0fc008c3e0 tx=0x7f0fb002b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0967eff8e8036ae server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.024+0000 7f0fbd600640 1 -- 172.21.15.62:0/1350291787 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb00317f0 con 0x7f0fc015fcf0 2024-03-23T11:22:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.024+0000 7f0fbd600640 1 -- 172.21.15.62:0/1350291787 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb0031950 con 0x7f0fc015fcf0 2024-03-23T11:22:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.024+0000 7f0fbd600640 1 -- 172.21.15.62:0/1350291787 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0031ce0 con 0x7f0fc015fcf0 2024-03-23T11:22:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.024+0000 7f0fbfe00640 1 -- 172.21.15.62:0/1350291787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc015fcf0 msgr2=0x7f0fc00d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.024+0000 7f0fbfe00640 1 --2- 172.21.15.62:0/1350291787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc015fcf0 0x7f0fc00d3000 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f0fc008c3e0 tx=0x7f0fb002b890 comp rx=0 tx=0).stop 2024-03-23T11:22:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.024+0000 7f0fbfe00640 1 -- 172.21.15.62:0/1350291787 shutdown_connections 2024-03-23T11:22:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.025+0000 7f0fbfe00640 1 --2- 172.21.15.62:0/1350291787 >> v2:172.21.15.120:3300/0 conn(0x7f0fc00d06d0 0x7f0fc00d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.025+0000 7f0fbfe00640 1 --2- 172.21.15.62:0/1350291787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc0160420 0x7f0fc0160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.025+0000 7f0fbfe00640 1 --2- 172.21.15.62:0/1350291787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc015fcf0 0x7f0fc00d3000 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.025+0000 7f0fbfe00640 1 -- 172.21.15.62:0/1350291787 >> 172.21.15.62:0/1350291787 conn(0x7f0fc0089f60 msgr2=0x7f0fc00d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:36.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.025+0000 7f0fbfe00640 1 -- 172.21.15.62:0/1350291787 shutdown_connections 2024-03-23T11:22:36.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.025+0000 7f0fbfe00640 1 -- 172.21.15.62:0/1350291787 wait complete. 2024-03-23T11:22:36.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.027+0000 7f0fbfe00640 1 Processor -- start 2024-03-23T11:22:36.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.027+0000 7f0fbfe00640 1 -- start start 2024-03-23T11:22:36.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.028+0000 7f0fbfe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fc00d06d0 0x7f0fc01f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.028+0000 7f0fbfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc015fcf0 0x7f0fc00c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.028+0000 7f0fbfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc0160420 0x7f0fc01f5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.028+0000 7f0fbfe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc01f3180 con 0x7f0fc0160420 2024-03-23T11:22:36.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.028+0000 7f0fbfe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc0158a60 con 0x7f0fc00d06d0 2024-03-23T11:22:36.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.028+0000 7f0fbfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00c8500 con 0x7f0fc015fcf0 2024-03-23T11:22:36.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.028+0000 7f0fbf400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc0160420 0x7f0fc01f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.028+0000 7f0fbf400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc0160420 0x7f0fc01f5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45076/0 (socket says 172.21.15.62:45076) 2024-03-23T11:22:36.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.028+0000 7f0fbf400640 1 -- 172.21.15.62:0/418689885 learned_addr learned my addr 172.21.15.62:0/418689885 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:36.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.028+0000 7f0fbe000640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc015fcf0 0x7f0fc00c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.028+0000 7f0fbea00640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fc00d06d0 0x7f0fc01f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.029+0000 7f0fbf400640 1 -- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc015fcf0 msgr2=0x7f0fc00c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.029+0000 7f0fbf400640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc015fcf0 0x7f0fc00c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.029+0000 7f0fbf400640 1 -- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fc00d06d0 msgr2=0x7f0fc01f2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.029+0000 7f0fbf400640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fc00d06d0 0x7f0fc01f2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.029+0000 7f0fbf400640 1 -- 172.21.15.62:0/418689885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb00094d0 con 0x7f0fc0160420 2024-03-23T11:22:36.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.029+0000 7f0fbf400640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc0160420 0x7f0fc01f5720 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800cb80 tx=0x7f0fb800a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:36.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.029+0000 7f0fb6a00640 1 -- 172.21.15.62:0/418689885 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb800ce70 con 0x7f0fc0160420 2024-03-23T11:22:36.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.029+0000 7f0fb6a00640 1 -- 172.21.15.62:0/418689885 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb800abd0 con 0x7f0fc0160420 2024-03-23T11:22:36.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.029+0000 7f0fb6a00640 1 -- 172.21.15.62:0/418689885 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8011410 con 0x7f0fc0160420 2024-03-23T11:22:36.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.029+0000 7f0fbfe00640 1 -- 172.21.15.62:0/418689885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc00c87e0 con 0x7f0fc0160420 2024-03-23T11:22:36.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.029+0000 7f0fbfe00640 1 -- 172.21.15.62:0/418689885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc00c8da0 con 0x7f0fc0160420 2024-03-23T11:22:36.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.030+0000 7f0fbfe00640 1 -- 172.21.15.62:0/418689885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc0066bb0 con 0x7f0fc0160420 2024-03-23T11:22:36.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.032+0000 7f0fb6a00640 1 -- 172.21.15.62:0/418689885 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0fb8011570 con 0x7f0fc0160420 2024-03-23T11:22:36.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.032+0000 7f0fb6a00640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f9c06a7a0 0x7f0f9c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.033+0000 7f0fb6a00640 1 -- 172.21.15.62:0/418689885 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fb8088b80 con 0x7f0fc0160420 2024-03-23T11:22:36.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.033+0000 7f0fbea00640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f9c06a7a0 0x7f0f9c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.034+0000 7f0fbea00640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f9c06a7a0 0x7f0f9c06cc60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0036000 tx=0x7f0fb003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:36.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.037+0000 7f0fb6a00640 1 -- 172.21.15.62:0/418689885 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0fb8054030 con 0x7f0fc0160420 2024-03-23T11:22:36.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.234+0000 7f0fbfe00640 1 -- 172.21.15.62:0/418689885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fc00cea40 con 0x7f0fc0160420 2024-03-23T11:22:36.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.237+0000 7f0fb6a00640 1 -- 172.21.15.62:0/418689885 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16097 (secure 0 0 0) 0x7f0fb801f090 con 0x7f0fc0160420 2024-03-23T11:22:36.236 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:36.236 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":62,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:22:32.731093+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:22:29.767222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-23T11:22:30.738596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:22:36.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.240+0000 7f0fbfe00640 1 -- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f9c06a7a0 msgr2=0x7f0f9c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.240+0000 7f0fbfe00640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f9c06a7a0 0x7f0f9c06cc60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0036000 tx=0x7f0fb003d040 comp rx=0 tx=0).stop 2024-03-23T11:22:36.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.240+0000 7f0fbfe00640 1 -- 172.21.15.62:0/418689885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc0160420 msgr2=0x7f0fc01f5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.240+0000 7f0fbfe00640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc0160420 0x7f0fc01f5720 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800cb80 tx=0x7f0fb800a4c0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.240+0000 7f0fbfe00640 1 -- 172.21.15.62:0/418689885 shutdown_connections 2024-03-23T11:22:36.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.240+0000 7f0fbfe00640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f9c06a7a0 0x7f0f9c06cc60 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.240+0000 7f0fbfe00640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc0160420 0x7f0fc01f5720 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.240+0000 7f0fbfe00640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fc015fcf0 0x7f0fc00c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.240+0000 7f0fbfe00640 1 --2- 172.21.15.62:0/418689885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fc00d06d0 0x7f0fc01f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.240+0000 7f0fbfe00640 1 -- 172.21.15.62:0/418689885 >> 172.21.15.62:0/418689885 conn(0x7f0fc0089f60 msgr2=0x7f0fc015f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:36.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.241+0000 7f0fbfe00640 1 -- 172.21.15.62:0/418689885 shutdown_connections 2024-03-23T11:22:36.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.241+0000 7f0fbfe00640 1 -- 172.21.15.62:0/418689885 wait complete. 2024-03-23T11:22:36.256 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T11:22:36.257 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:22:36.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.465+0000 7f81faa00640 1 Processor -- start 2024-03-23T11:22:36.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.465+0000 7f81faa00640 1 -- start start 2024-03-23T11:22:36.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.466+0000 7f81faa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f81f40cd7c0 0x7f81f40b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.466+0000 7f81faa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81f406c300 0x7f81f40cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.466+0000 7f81faa00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f40cad20 con 0x7f81f406dca0 2024-03-23T11:22:36.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.466+0000 7f81faa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f406c750 con 0x7f81f40cd7c0 2024-03-23T11:22:36.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.466+0000 7f81faa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f406e130 con 0x7f81f406c300 2024-03-23T11:22:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.466+0000 7f81f3400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f81f406dca0 0x7f81f40cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40078/0 (socket says 172.21.15.62:40078) 2024-03-23T11:22:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.466+0000 7f81f3400640 1 -- 172.21.15.62:0/2520055267 learned_addr learned my addr 172.21.15.62:0/2520055267 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.467+0000 7f81f3e00640 1 --2- 172.21.15.62:0/2520055267 >> v2:172.21.15.120:3300/0 conn(0x7f81f40cd7c0 0x7f81f40b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.467+0000 7f81f8a00640 1 --2- 172.21.15.62:0/2520055267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81f406c300 0x7f81f40cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.467+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672263224 0 0) 0x7f81f40cad20 con 0x7f81f406dca0 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.467+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d8003520 con 0x7f81f406dca0 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.467+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3848909709 0 0) 0x7f81d8003520 con 0x7f81f406dca0 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.467+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81f406c300 msgr2=0x7f81f40cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.467+0000 7f81f2a00640 1 --2- 172.21.15.62:0/2520055267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81f406c300 0x7f81f40cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.467+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 >> v2:172.21.15.120:3300/0 conn(0x7f81f40cd7c0 msgr2=0x7f81f40b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.467+0000 7f81f2a00640 1 --2- 172.21.15.62:0/2520055267 >> v2:172.21.15.120:3300/0 conn(0x7f81f40cd7c0 0x7f81f40b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.467+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f415a8a0 con 0x7f81f406dca0 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f81e0001670 con 0x7f81f406dca0 2024-03-23T11:22:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 >> v1:172.21.15.62:6789/0 conn(0x7f81f406dca0 legacy=0x7f81f40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f2a00640 1 --2- 172.21.15.62:0/2520055267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8003970 0x7f81d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f2a00640 1 --2- 172.21.15.62:0/2520055267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81f406c300 0x7f81d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f2a00640 1 --2- 172.21.15.62:0/2520055267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81d8004d00 0x7f81d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f406c750 con 0x7f81d8003970 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f406e130 con 0x7f81d8004d00 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d800b9b0 con 0x7f81f406c300 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f3e00640 1 --2- 172.21.15.62:0/2520055267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8003970 0x7f81d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f3400640 1 --2- 172.21.15.62:0/2520055267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81f406c300 0x7f81d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f3e00640 1 -- 172.21.15.62:0/2520055267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81f406c300 msgr2=0x7f81d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f3e00640 1 --2- 172.21.15.62:0/2520055267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81f406c300 0x7f81d80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f3e00640 1 -- 172.21.15.62:0/2520055267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81d8004d00 msgr2=0x7f81d8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f3e00640 1 --2- 172.21.15.62:0/2520055267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81d8004d00 0x7f81d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.468+0000 7f81f3e00640 1 -- 172.21.15.62:0/2520055267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d8003520 con 0x7f81d8003970 2024-03-23T11:22:36.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.470+0000 7f81f3e00640 1 --2- 172.21.15.62:0/2520055267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8003970 0x7f81d8003d90 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009fc0 tx=0x7f81d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e022c7d716548131 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:36.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.470+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e402cc60 con 0x7f81d8003970 2024-03-23T11:22:36.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.470+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e400f040 con 0x7f81d8003970 2024-03-23T11:22:36.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.470+0000 7f81f2a00640 1 -- 172.21.15.62:0/2520055267 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4034440 con 0x7f81d8003970 2024-03-23T11:22:36.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.470+0000 7f81faa00640 1 -- 172.21.15.62:0/2520055267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8003970 msgr2=0x7f81d8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.471+0000 7f81faa00640 1 --2- 172.21.15.62:0/2520055267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8003970 0x7f81d8003d90 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009fc0 tx=0x7f81d80057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.471+0000 7f81f8a00640 1 -- 172.21.15.62:0/2520055267 reap_dead start 2024-03-23T11:22:36.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.471+0000 7f81faa00640 1 -- 172.21.15.62:0/2520055267 shutdown_connections 2024-03-23T11:22:36.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.471+0000 7f81faa00640 1 -- 172.21.15.62:0/2520055267 >> 172.21.15.62:0/2520055267 conn(0x7f81f40b02e0 msgr2=0x7f81f40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:36.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.471+0000 7f81faa00640 1 -- 172.21.15.62:0/2520055267 shutdown_connections 2024-03-23T11:22:36.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.471+0000 7f81faa00640 1 -- 172.21.15.62:0/2520055267 wait complete. 2024-03-23T11:22:36.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.473+0000 7f81faa00640 1 Processor -- start 2024-03-23T11:22:36.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.473+0000 7f81faa00640 1 -- start start 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81faa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81d8003970 0x7f81f4147a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81faa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4148a70 0x7f81f414bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81faa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f415aef0 0x7f81f415b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81faa00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f4156e70 con 0x7f81f4148a70 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81faa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f415b8b0 con 0x7f81f415aef0 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81faa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f40a7b20 con 0x7f81d8003970 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81f3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4148a70 0x7f81f414bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81f3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4148a70 0x7f81f414bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45094/0 (socket says 172.21.15.62:45094) 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81f3400640 1 -- 172.21.15.62:0/1463699592 learned_addr learned my addr 172.21.15.62:0/1463699592 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81f8a00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f415aef0 0x7f81f415b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81f3400640 1 -- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81d8003970 msgr2=0x7f81f4147a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81f3e00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81d8003970 0x7f81f4147a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.474+0000 7f81f3400640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81d8003970 0x7f81f4147a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.475+0000 7f81f3400640 1 -- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f415aef0 msgr2=0x7f81f415b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.475+0000 7f81f3400640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f415aef0 0x7f81f415b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.475+0000 7f81f3400640 1 -- 172.21.15.62:0/1463699592 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f4149030 con 0x7f81f4148a70 2024-03-23T11:22:36.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.475+0000 7f81f8a00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f415aef0 0x7f81f415b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:36.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.475+0000 7f81f3400640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4148a70 0x7f81f414bce0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f81e000d7a0 tx=0x7f81e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:36.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.475+0000 7f81ebe00640 1 -- 172.21.15.62:0/1463699592 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e00074e0 con 0x7f81f4148a70 2024-03-23T11:22:36.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.475+0000 7f81ebe00640 1 -- 172.21.15.62:0/1463699592 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e0007640 con 0x7f81f4148a70 2024-03-23T11:22:36.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.475+0000 7f81ebe00640 1 -- 172.21.15.62:0/1463699592 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e00049e0 con 0x7f81f4148a70 2024-03-23T11:22:36.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.475+0000 7f81faa00640 1 -- 172.21.15.62:0/1463699592 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f4149310 con 0x7f81f4148a70 2024-03-23T11:22:36.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.475+0000 7f81faa00640 1 -- 172.21.15.62:0/1463699592 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f40a77b0 con 0x7f81f4148a70 2024-03-23T11:22:36.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.477+0000 7f81faa00640 1 -- 172.21.15.62:0/1463699592 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f40b3870 con 0x7f81f4148a70 2024-03-23T11:22:36.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.477+0000 7f81ebe00640 1 -- 172.21.15.62:0/1463699592 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f81e0004b40 con 0x7f81f4148a70 2024-03-23T11:22:36.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.478+0000 7f81ebe00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81d406a7a0 0x7f81d406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.478+0000 7f81ebe00640 1 -- 172.21.15.62:0/1463699592 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81e0088740 con 0x7f81f4148a70 2024-03-23T11:22:36.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.479+0000 7f81f3e00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81d406a7a0 0x7f81d406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.480+0000 7f81f3e00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81d406a7a0 0x7f81d406cc60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f81e4036000 tx=0x7f81e402e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:36.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.483+0000 7f81ebe00640 1 -- 172.21.15.62:0/1463699592 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f81e0053ac0 con 0x7f81f4148a70 2024-03-23T11:22:36.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.677+0000 7f81faa00640 1 -- 172.21.15.62:0/1463699592 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f81f40b1060 con 0x7f81d406a7a0 2024-03-23T11:22:36.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.703+0000 7f81ebe00640 1 -- 172.21.15.62:0/1463699592 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286864 (secure 0 0 0) 0x7f81e402cdd0 con 0x7f81d406a7a0 2024-03-23T11:22:36.706 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:36.708 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:22:36.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.710+0000 7f81faa00640 1 -- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81d406a7a0 msgr2=0x7f81d406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.711+0000 7f81faa00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81d406a7a0 0x7f81d406cc60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f81e4036000 tx=0x7f81e402e000 comp rx=0 tx=0).stop 2024-03-23T11:22:36.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.711+0000 7f81faa00640 1 -- 172.21.15.62:0/1463699592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4148a70 msgr2=0x7f81f414bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.711+0000 7f81faa00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4148a70 0x7f81f414bce0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f81e000d7a0 tx=0x7f81e000dc60 comp rx=0 tx=0).stop 2024-03-23T11:22:36.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.711+0000 7f81faa00640 1 -- 172.21.15.62:0/1463699592 shutdown_connections 2024-03-23T11:22:36.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.711+0000 7f81faa00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81d406a7a0 0x7f81d406cc60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.711+0000 7f81faa00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4148a70 0x7f81f414bce0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.711+0000 7f81faa00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f415aef0 0x7f81f415b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.711+0000 7f81faa00640 1 --2- 172.21.15.62:0/1463699592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81d8003970 0x7f81f4147a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.711+0000 7f81faa00640 1 -- 172.21.15.62:0/1463699592 >> 172.21.15.62:0/1463699592 conn(0x7f81f40b02e0 msgr2=0x7f81f40b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:36.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.711+0000 7f81faa00640 1 -- 172.21.15.62:0/1463699592 shutdown_connections 2024-03-23T11:22:36.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.711+0000 7f81faa00640 1 -- 172.21.15.62:0/1463699592 wait complete. 2024-03-23T11:22:36.729 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-03-23T11:22:35.307450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231316,"kb_used_data":6932,"kb_used_omap":13,"kb_used_meta":224242,"kb_avail":754743404,"statfs":{"total":773094113280,"available":772857245696,"internally_reserved":0,"allocated":7098368,"data_stored":3617197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":229623896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.085831"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745756+0000","last_change":"2024-03-23T11:22:31.755822+0000","last_active":"2024-03-23T11:22:32.745756+0000","last_peered":"2024-03-23T11:22:32.745756+0000","last_clean":"2024-03-23T11:22:32.745756+0000","last_became_active":"2024-03-23T11:22:31.754982+0000","last_became_peered":"2024-03-23T11:22:31.754982+0000","last_unstale":"2024-03-23T11:22:32.745756+0000","last_undegraded":"2024-03-23T11:22:32.745756+0000","last_fullsized":"2024-03-23T11:22:32.745756+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:48:12.358869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939162+0000","last_change":"2024-03-23T11:22:31.757391+0000","last_active":"2024-03-23T11:22:32.939162+0000","last_peered":"2024-03-23T11:22:32.939162+0000","last_clean":"2024-03-23T11:22:32.939162+0000","last_became_active":"2024-03-23T11:22:31.757140+0000","last_became_peered":"2024-03-23T11:22:31.757140+0000","last_unstale":"2024-03-23T11:22:32.939162+0000","last_undegraded":"2024-03-23T11:22:32.939162+0000","last_fullsized":"2024-03-23T11:22:32.939162+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:28:10.126290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748782+0000","last_change":"2024-03-23T11:22:31.759403+0000","last_active":"2024-03-23T11:22:32.748782+0000","last_peered":"2024-03-23T11:22:32.748782+0000","last_clean":"2024-03-23T11:22:32.748782+0000","last_became_active":"2024-03-23T11:22:31.759095+0000","last_became_peered":"2024-03-23T11:22:31.759095+0000","last_unstale":"2024-03-23T11:22:32.748782+0000","last_undegraded":"2024-03-23T11:22:32.748782+0000","last_fullsized":"2024-03-23T11:22:32.748782+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:21:39.030934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.762803+0000","last_change":"2024-03-23T11:22:31.757112+0000","last_active":"2024-03-23T11:22:31.762803+0000","last_peered":"2024-03-23T11:22:31.762803+0000","last_clean":"2024-03-23T11:22:31.762803+0000","last_became_active":"2024-03-23T11:22:31.756746+0000","last_became_peered":"2024-03-23T11:22:31.756746+0000","last_unstale":"2024-03-23T11:22:31.762803+0000","last_undegraded":"2024-03-23T11:22:31.762803+0000","last_fullsized":"2024-03-23T11:22:31.762803+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:46:36.179754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745535+0000","last_change":"2024-03-23T11:22:31.756871+0000","last_active":"2024-03-23T11:22:32.745535+0000","last_peered":"2024-03-23T11:22:32.745535+0000","last_clean":"2024-03-23T11:22:32.745535+0000","last_became_active":"2024-03-23T11:22:31.756597+0000","last_became_peered":"2024-03-23T11:22:31.756597+0000","last_unstale":"2024-03-23T11:22:32.745535+0000","last_undegraded":"2024-03-23T11:22:32.745535+0000","last_fullsized":"2024-03-23T11:22:32.745535+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:16:37.145910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778135+0000","last_change":"2024-03-23T11:22:31.774723+0000","last_active":"2024-03-23T11:22:31.778135+0000","last_peered":"2024-03-23T11:22:31.778135+0000","last_clean":"2024-03-23T11:22:31.778135+0000","last_became_active":"2024-03-23T11:22:31.771125+0000","last_became_peered":"2024-03-23T11:22:31.771125+0000","last_unstale":"2024-03-23T11:22:31.778135+0000","last_undegraded":"2024-03-23T11:22:31.778135+0000","last_fullsized":"2024-03-23T11:22:31.778135+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:46:36.179754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746660+0000","last_change":"2024-03-23T11:22:31.752247+0000","last_active":"2024-03-23T11:22:32.746660+0000","last_peered":"2024-03-23T11:22:32.746660+0000","last_clean":"2024-03-23T11:22:32.746660+0000","last_became_active":"2024-03-23T11:22:31.751333+0000","last_became_peered":"2024-03-23T11:22:31.751333+0000","last_unstale":"2024-03-23T11:22:32.746660+0000","last_undegraded":"2024-03-23T11:22:32.746660+0000","last_fullsized":"2024-03-23T11:22:32.746660+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:50:48.239290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938391+0000","last_change":"2024-03-23T11:22:31.747632+0000","last_active":"2024-03-23T11:22:32.938391+0000","last_peered":"2024-03-23T11:22:32.938391+0000","last_clean":"2024-03-23T11:22:32.938391+0000","last_became_active":"2024-03-23T11:22:31.747352+0000","last_became_peered":"2024-03-23T11:22:31.747352+0000","last_unstale":"2024-03-23T11:22:32.938391+0000","last_undegraded":"2024-03-23T11:22:32.938391+0000","last_fullsized":"2024-03-23T11:22:32.938391+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:44:33.088453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746805+0000","last_change":"2024-03-23T11:22:31.769768+0000","last_active":"2024-03-23T11:22:32.746805+0000","last_peered":"2024-03-23T11:22:32.746805+0000","last_clean":"2024-03-23T11:22:32.746805+0000","last_became_active":"2024-03-23T11:22:31.769448+0000","last_became_peered":"2024-03-23T11:22:31.769448+0000","last_unstale":"2024-03-23T11:22:32.746805+0000","last_undegraded":"2024-03-23T11:22:32.746805+0000","last_fullsized":"2024-03-23T11:22:32.746805+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:50:11.154778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.766880+0000","last_change":"2024-03-23T11:22:31.756698+0000","last_active":"2024-03-23T11:22:31.766880+0000","last_peered":"2024-03-23T11:22:31.766880+0000","last_clean":"2024-03-23T11:22:31.766880+0000","last_became_active":"2024-03-23T11:22:31.756407+0000","last_became_peered":"2024-03-23T11:22:31.756407+0000","last_unstale":"2024-03-23T11:22:31.766880+0000","last_undegraded":"2024-03-23T11:22:31.766880+0000","last_fullsized":"2024-03-23T11:22:31.766880+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:30:13.644040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768754+0000","last_change":"2024-03-23T11:22:31.765943+0000","last_active":"2024-03-23T11:22:31.768754+0000","last_peered":"2024-03-23T11:22:31.768754+0000","last_clean":"2024-03-23T11:22:31.768754+0000","last_became_active":"2024-03-23T11:22:31.765536+0000","last_became_peered":"2024-03-23T11:22:31.765536+0000","last_unstale":"2024-03-23T11:22:31.768754+0000","last_undegraded":"2024-03-23T11:22:31.768754+0000","last_fullsized":"2024-03-23T11:22:31.768754+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:30:55.729393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.766729+0000","last_change":"2024-03-23T11:22:31.762805+0000","last_active":"2024-03-23T11:22:31.766729+0000","last_peered":"2024-03-23T11:22:31.766729+0000","last_clean":"2024-03-23T11:22:31.766729+0000","last_became_active":"2024-03-23T11:22:31.762532+0000","last_became_peered":"2024-03-23T11:22:31.762532+0000","last_unstale":"2024-03-23T11:22:31.766729+0000","last_undegraded":"2024-03-23T11:22:31.766729+0000","last_fullsized":"2024-03-23T11:22:31.766729+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:22:24.908706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768490+0000","last_change":"2024-03-23T11:22:31.766751+0000","last_active":"2024-03-23T11:22:31.768490+0000","last_peered":"2024-03-23T11:22:31.768490+0000","last_clean":"2024-03-23T11:22:31.768490+0000","last_became_active":"2024-03-23T11:22:31.766162+0000","last_became_peered":"2024-03-23T11:22:31.766162+0000","last_unstale":"2024-03-23T11:22:31.768490+0000","last_undegraded":"2024-03-23T11:22:31.768490+0000","last_fullsized":"2024-03-23T11:22:31.768490+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:45:00.120978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746532+0000","last_change":"2024-03-23T11:22:31.753003+0000","last_active":"2024-03-23T11:22:32.746532+0000","last_peered":"2024-03-23T11:22:32.746532+0000","last_clean":"2024-03-23T11:22:32.746532+0000","last_became_active":"2024-03-23T11:22:31.752661+0000","last_became_peered":"2024-03-23T11:22:31.752661+0000","last_unstale":"2024-03-23T11:22:32.746532+0000","last_undegraded":"2024-03-23T11:22:32.746532+0000","last_fullsized":"2024-03-23T11:22:32.746532+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:49:52.269908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746690+0000","last_change":"2024-03-23T11:22:31.753832+0000","last_active":"2024-03-23T11:22:32.746690+0000","last_peered":"2024-03-23T11:22:32.746690+0000","last_clean":"2024-03-23T11:22:32.746690+0000","last_became_active":"2024-03-23T11:22:31.753590+0000","last_became_peered":"2024-03-23T11:22:31.753590+0000","last_unstale":"2024-03-23T11:22:32.746690+0000","last_undegraded":"2024-03-23T11:22:32.746690+0000","last_fullsized":"2024-03-23T11:22:32.746690+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:54:11.055811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778282+0000","last_change":"2024-03-23T11:22:31.776363+0000","last_active":"2024-03-23T11:22:31.778282+0000","last_peered":"2024-03-23T11:22:31.778282+0000","last_clean":"2024-03-23T11:22:31.778282+0000","last_became_active":"2024-03-23T11:22:31.771638+0000","last_became_peered":"2024-03-23T11:22:31.771638+0000","last_unstale":"2024-03-23T11:22:31.778282+0000","last_undegraded":"2024-03-23T11:22:31.778282+0000","last_fullsized":"2024-03-23T11:22:31.778282+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:00:51.026955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939299+0000","last_change":"2024-03-23T11:22:31.761897+0000","last_active":"2024-03-23T11:22:32.939299+0000","last_peered":"2024-03-23T11:22:32.939299+0000","last_clean":"2024-03-23T11:22:32.939299+0000","last_became_active":"2024-03-23T11:22:31.761195+0000","last_became_peered":"2024-03-23T11:22:31.761195+0000","last_unstale":"2024-03-23T11:22:32.939299+0000","last_undegraded":"2024-03-23T11:22:32.939299+0000","last_fullsized":"2024-03-23T11:22:32.939299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:47:58.423989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749186+0000","last_change":"2024-03-23T11:22:31.776125+0000","last_active":"2024-03-23T11:22:32.749186+0000","last_peered":"2024-03-23T11:22:32.749186+0000","last_clean":"2024-03-23T11:22:32.749186+0000","last_became_active":"2024-03-23T11:22:31.775617+0000","last_became_peered":"2024-03-23T11:22:31.775617+0000","last_unstale":"2024-03-23T11:22:32.749186+0000","last_undegraded":"2024-03-23T11:22:32.749186+0000","last_fullsized":"2024-03-23T11:22:32.749186+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:06.149914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745601+0000","last_change":"2024-03-23T11:22:31.755309+0000","last_active":"2024-03-23T11:22:32.745601+0000","last_peered":"2024-03-23T11:22:32.745601+0000","last_clean":"2024-03-23T11:22:32.745601+0000","last_became_active":"2024-03-23T11:22:31.755048+0000","last_became_peered":"2024-03-23T11:22:31.755048+0000","last_unstale":"2024-03-23T11:22:32.745601+0000","last_undegraded":"2024-03-23T11:22:32.745601+0000","last_fullsized":"2024-03-23T11:22:32.745601+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:03:24.515541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939090+0000","last_change":"2024-03-23T11:22:31.774011+0000","last_active":"2024-03-23T11:22:32.939090+0000","last_peered":"2024-03-23T11:22:32.939090+0000","last_clean":"2024-03-23T11:22:32.939090+0000","last_became_active":"2024-03-23T11:22:31.773707+0000","last_became_peered":"2024-03-23T11:22:31.773707+0000","last_unstale":"2024-03-23T11:22:32.939090+0000","last_undegraded":"2024-03-23T11:22:32.939090+0000","last_fullsized":"2024-03-23T11:22:32.939090+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:56.760297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748730+0000","last_change":"2024-03-23T11:22:31.758066+0000","last_active":"2024-03-23T11:22:32.748730+0000","last_peered":"2024-03-23T11:22:32.748730+0000","last_clean":"2024-03-23T11:22:32.748730+0000","last_became_active":"2024-03-23T11:22:31.757851+0000","last_became_peered":"2024-03-23T11:22:31.757851+0000","last_unstale":"2024-03-23T11:22:32.748730+0000","last_undegraded":"2024-03-23T11:22:32.748730+0000","last_fullsized":"2024-03-23T11:22:32.748730+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:31:50.946464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938633+0000","last_change":"2024-03-23T11:22:31.750381+0000","last_active":"2024-03-23T11:22:32.938633+0000","last_peered":"2024-03-23T11:22:32.938633+0000","last_clean":"2024-03-23T11:22:32.938633+0000","last_became_active":"2024-03-23T11:22:31.749517+0000","last_became_peered":"2024-03-23T11:22:31.749517+0000","last_unstale":"2024-03-23T11:22:32.938633+0000","last_undegraded":"2024-03-23T11:22:32.938633+0000","last_fullsized":"2024-03-23T11:22:32.938633+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:16:43.753148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748444+0000","last_change":"2024-03-23T11:22:31.756649+0000","last_active":"2024-03-23T11:22:32.748444+0000","last_peered":"2024-03-23T11:22:32.748444+0000","last_clean":"2024-03-23T11:22:32.748444+0000","last_became_active":"2024-03-23T11:22:31.754412+0000","last_became_peered":"2024-03-23T11:22:31.754412+0000","last_unstale":"2024-03-23T11:22:32.748444+0000","last_undegraded":"2024-03-23T11:22:32.748444+0000","last_fullsized":"2024-03-23T11:22:32.748444+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:55:42.979301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778024+0000","last_change":"2024-03-23T11:22:31.777173+0000","last_active":"2024-03-23T11:22:31.778024+0000","last_peered":"2024-03-23T11:22:31.778024+0000","last_clean":"2024-03-23T11:22:31.778024+0000","last_became_active":"2024-03-23T11:22:31.776866+0000","last_became_peered":"2024-03-23T11:22:31.776866+0000","last_unstale":"2024-03-23T11:22:31.778024+0000","last_undegraded":"2024-03-23T11:22:31.778024+0000","last_fullsized":"2024-03-23T11:22:31.778024+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:52:26.665790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745923+0000","last_change":"2024-03-23T11:22:31.764643+0000","last_active":"2024-03-23T11:22:32.745923+0000","last_peered":"2024-03-23T11:22:32.745923+0000","last_clean":"2024-03-23T11:22:32.745923+0000","last_became_active":"2024-03-23T11:22:31.763457+0000","last_became_peered":"2024-03-23T11:22:31.763457+0000","last_unstale":"2024-03-23T11:22:32.745923+0000","last_undegraded":"2024-03-23T11:22:32.745923+0000","last_fullsized":"2024-03-23T11:22:32.745923+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:01:29.163243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.762687+0000","last_change":"2024-03-23T11:22:31.755850+0000","last_active":"2024-03-23T11:22:31.762687+0000","last_peered":"2024-03-23T11:22:31.762687+0000","last_clean":"2024-03-23T11:22:31.762687+0000","last_became_active":"2024-03-23T11:22:31.755584+0000","last_became_peered":"2024-03-23T11:22:31.755584+0000","last_unstale":"2024-03-23T11:22:31.762687+0000","last_undegraded":"2024-03-23T11:22:31.762687+0000","last_fullsized":"2024-03-23T11:22:31.762687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:53.586433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778197+0000","last_change":"2024-03-23T11:22:31.774994+0000","last_active":"2024-03-23T11:22:31.778197+0000","last_peered":"2024-03-23T11:22:31.778197+0000","last_clean":"2024-03-23T11:22:31.778197+0000","last_became_active":"2024-03-23T11:22:31.772573+0000","last_became_peered":"2024-03-23T11:22:31.772573+0000","last_unstale":"2024-03-23T11:22:31.778197+0000","last_undegraded":"2024-03-23T11:22:31.778197+0000","last_fullsized":"2024-03-23T11:22:31.778197+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:14.173559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745441+0000","last_change":"2024-03-23T11:22:31.755440+0000","last_active":"2024-03-23T11:22:32.745441+0000","last_peered":"2024-03-23T11:22:32.745441+0000","last_clean":"2024-03-23T11:22:32.745441+0000","last_became_active":"2024-03-23T11:22:31.752956+0000","last_became_peered":"2024-03-23T11:22:31.752956+0000","last_unstale":"2024-03-23T11:22:32.745441+0000","last_undegraded":"2024-03-23T11:22:32.745441+0000","last_fullsized":"2024-03-23T11:22:32.745441+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:48:45.151345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938921+0000","last_change":"2024-03-23T11:22:31.763702+0000","last_active":"2024-03-23T11:22:32.938921+0000","last_peered":"2024-03-23T11:22:32.938921+0000","last_clean":"2024-03-23T11:22:32.938921+0000","last_became_active":"2024-03-23T11:22:31.763445+0000","last_became_peered":"2024-03-23T11:22:31.763445+0000","last_unstale":"2024-03-23T11:22:32.938921+0000","last_undegraded":"2024-03-23T11:22:32.938921+0000","last_fullsized":"2024-03-23T11:22:32.938921+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:06:06.972232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748561+0000","last_change":"2024-03-23T11:22:31.751995+0000","last_active":"2024-03-23T11:22:32.748561+0000","last_peered":"2024-03-23T11:22:32.748561+0000","last_clean":"2024-03-23T11:22:32.748561+0000","last_became_active":"2024-03-23T11:22:31.751709+0000","last_became_peered":"2024-03-23T11:22:31.751709+0000","last_unstale":"2024-03-23T11:22:32.748561+0000","last_undegraded":"2024-03-23T11:22:32.748561+0000","last_fullsized":"2024-03-23T11:22:32.748561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:10:08.399314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745338+0000","last_change":"2024-03-23T11:22:31.754634+0000","last_active":"2024-03-23T11:22:32.745338+0000","last_peered":"2024-03-23T11:22:32.745338+0000","last_clean":"2024-03-23T11:22:32.745338+0000","last_became_active":"2024-03-23T11:22:31.753393+0000","last_became_peered":"2024-03-23T11:22:31.753393+0000","last_unstale":"2024-03-23T11:22:32.745338+0000","last_undegraded":"2024-03-23T11:22:32.745338+0000","last_fullsized":"2024-03-23T11:22:32.745338+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:02:24.548514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778260+0000","last_change":"2024-03-23T11:22:31.775630+0000","last_active":"2024-03-23T11:22:31.778260+0000","last_peered":"2024-03-23T11:22:31.778260+0000","last_clean":"2024-03-23T11:22:31.778260+0000","last_became_active":"2024-03-23T11:22:31.771294+0000","last_became_peered":"2024-03-23T11:22:31.771294+0000","last_unstale":"2024-03-23T11:22:31.778260+0000","last_undegraded":"2024-03-23T11:22:31.778260+0000","last_fullsized":"2024-03-23T11:22:31.778260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:15:02.678591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778086+0000","last_change":"2024-03-23T11:22:31.774039+0000","last_active":"2024-03-23T11:22:31.778086+0000","last_peered":"2024-03-23T11:22:31.778086+0000","last_clean":"2024-03-23T11:22:31.778086+0000","last_became_active":"2024-03-23T11:22:31.773700+0000","last_became_peered":"2024-03-23T11:22:31.773700+0000","last_unstale":"2024-03-23T11:22:31.778086+0000","last_undegraded":"2024-03-23T11:22:31.778086+0000","last_fullsized":"2024-03-23T11:22:31.778086+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:09:01.658097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"61'4","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778343+0000","last_change":"2024-03-23T11:22:31.262916+0000","last_active":"2024-03-23T11:22:31.778343+0000","last_peered":"2024-03-23T11:22:31.778343+0000","last_clean":"2024-03-23T11:22:31.778343+0000","last_became_active":"2024-03-23T11:22:31.262651+0000","last_became_peered":"2024-03-23T11:22:31.262651+0000","last_unstale":"2024-03-23T11:22:31.778343+0000","last_undegraded":"2024-03-23T11:22:31.778343+0000","last_fullsized":"2024-03-23T11:22:31.778343+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:59:16.844989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768771+0000","last_change":"2024-03-23T11:22:31.765924+0000","last_active":"2024-03-23T11:22:31.768771+0000","last_peered":"2024-03-23T11:22:31.768771+0000","last_clean":"2024-03-23T11:22:31.768771+0000","last_became_active":"2024-03-23T11:22:31.765600+0000","last_became_peered":"2024-03-23T11:22:31.765600+0000","last_unstale":"2024-03-23T11:22:31.768771+0000","last_undegraded":"2024-03-23T11:22:31.768771+0000","last_fullsized":"2024-03-23T11:22:31.768771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:01:05.157583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"61'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939451+0000","last_change":"2024-03-23T11:22:31.264605+0000","last_active":"2024-03-23T11:22:32.939451+0000","last_peered":"2024-03-23T11:22:32.939451+0000","last_clean":"2024-03-23T11:22:32.939451+0000","last_became_active":"2024-03-23T11:22:31.264341+0000","last_became_peered":"2024-03-23T11:22:31.264341+0000","last_unstale":"2024-03-23T11:22:32.939451+0000","last_undegraded":"2024-03-23T11:22:32.939451+0000","last_fullsized":"2024-03-23T11:22:32.939451+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:35:51.264384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778045+0000","last_change":"2024-03-23T11:22:31.773721+0000","last_active":"2024-03-23T11:22:31.778045+0000","last_peered":"2024-03-23T11:22:31.778045+0000","last_clean":"2024-03-23T11:22:31.778045+0000","last_became_active":"2024-03-23T11:22:31.773443+0000","last_became_peered":"2024-03-23T11:22:31.773443+0000","last_unstale":"2024-03-23T11:22:31.778045+0000","last_undegraded":"2024-03-23T11:22:31.778045+0000","last_fullsized":"2024-03-23T11:22:31.778045+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:27:14.723335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"61'2","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.775185+0000","last_change":"2024-03-23T11:22:31.262398+0000","last_active":"2024-03-23T11:22:31.775185+0000","last_peered":"2024-03-23T11:22:31.775185+0000","last_clean":"2024-03-23T11:22:31.775185+0000","last_became_active":"2024-03-23T11:22:31.258009+0000","last_became_peered":"2024-03-23T11:22:31.258009+0000","last_unstale":"2024-03-23T11:22:31.775185+0000","last_undegraded":"2024-03-23T11:22:31.775185+0000","last_fullsized":"2024-03-23T11:22:31.775185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:32:47.093788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778022+0000","last_change":"2024-03-23T11:22:31.773674+0000","last_active":"2024-03-23T11:22:31.778022+0000","last_peered":"2024-03-23T11:22:31.778022+0000","last_clean":"2024-03-23T11:22:31.778022+0000","last_became_active":"2024-03-23T11:22:31.768814+0000","last_became_peered":"2024-03-23T11:22:31.768814+0000","last_unstale":"2024-03-23T11:22:31.778022+0000","last_undegraded":"2024-03-23T11:22:31.778022+0000","last_fullsized":"2024-03-23T11:22:31.778022+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:34:23.644768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"61'2","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.762263+0000","last_change":"2024-03-23T11:22:31.264826+0000","last_active":"2024-03-23T11:22:31.762263+0000","last_peered":"2024-03-23T11:22:31.762263+0000","last_clean":"2024-03-23T11:22:31.762263+0000","last_became_active":"2024-03-23T11:22:31.264459+0000","last_became_peered":"2024-03-23T11:22:31.264459+0000","last_unstale":"2024-03-23T11:22:31.762263+0000","last_undegraded":"2024-03-23T11:22:31.762263+0000","last_fullsized":"2024-03-23T11:22:31.762263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:35:51.264384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938644+0000","last_change":"2024-03-23T11:22:31.750384+0000","last_active":"2024-03-23T11:22:32.938644+0000","last_peered":"2024-03-23T11:22:32.938644+0000","last_clean":"2024-03-23T11:22:32.938644+0000","last_became_active":"2024-03-23T11:22:31.749914+0000","last_became_peered":"2024-03-23T11:22:31.749914+0000","last_unstale":"2024-03-23T11:22:32.938644+0000","last_undegraded":"2024-03-23T11:22:32.938644+0000","last_fullsized":"2024-03-23T11:22:32.938644+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:20:37.373989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"61'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746839+0000","last_change":"2024-03-23T11:22:31.263340+0000","last_active":"2024-03-23T11:22:32.746839+0000","last_peered":"2024-03-23T11:22:32.746839+0000","last_clean":"2024-03-23T11:22:32.746839+0000","last_became_active":"2024-03-23T11:22:31.263066+0000","last_became_peered":"2024-03-23T11:22:31.263066+0000","last_unstale":"2024-03-23T11:22:32.746839+0000","last_undegraded":"2024-03-23T11:22:32.746839+0000","last_fullsized":"2024-03-23T11:22:32.746839+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:57.593149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938836+0000","last_change":"2024-03-23T11:22:31.770892+0000","last_active":"2024-03-23T11:22:32.938836+0000","last_peered":"2024-03-23T11:22:32.938836+0000","last_clean":"2024-03-23T11:22:32.938836+0000","last_became_active":"2024-03-23T11:22:31.770567+0000","last_became_peered":"2024-03-23T11:22:31.770567+0000","last_unstale":"2024-03-23T11:22:32.938836+0000","last_undegraded":"2024-03-23T11:22:32.938836+0000","last_fullsized":"2024-03-23T11:22:32.938836+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:19.482944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"61'1","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.777775+0000","last_change":"2024-03-23T11:22:31.264847+0000","last_active":"2024-03-23T11:22:31.777775+0000","last_peered":"2024-03-23T11:22:31.777775+0000","last_clean":"2024-03-23T11:22:31.777775+0000","last_became_active":"2024-03-23T11:22:31.263443+0000","last_became_peered":"2024-03-23T11:22:31.263443+0000","last_unstale":"2024-03-23T11:22:31.777775+0000","last_undegraded":"2024-03-23T11:22:31.777775+0000","last_fullsized":"2024-03-23T11:22:31.777775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:44:59.112031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.761564+0000","last_change":"2024-03-23T11:22:31.759332+0000","last_active":"2024-03-23T11:22:31.761564+0000","last_peered":"2024-03-23T11:22:31.761564+0000","last_clean":"2024-03-23T11:22:31.761564+0000","last_became_active":"2024-03-23T11:22:31.759078+0000","last_became_peered":"2024-03-23T11:22:31.759078+0000","last_unstale":"2024-03-23T11:22:31.761564+0000","last_undegraded":"2024-03-23T11:22:31.761564+0000","last_fullsized":"2024-03-23T11:22:31.761564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:44:48.626587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.747194+0000","last_change":"2024-03-23T11:22:31.260619+0000","last_active":"2024-03-23T11:22:31.747194+0000","last_peered":"2024-03-23T11:22:31.747194+0000","last_clean":"2024-03-23T11:22:31.747194+0000","last_became_active":"2024-03-23T11:22:31.260211+0000","last_became_peered":"2024-03-23T11:22:31.260211+0000","last_unstale":"2024-03-23T11:22:31.747194+0000","last_undegraded":"2024-03-23T11:22:31.747194+0000","last_fullsized":"2024-03-23T11:22:31.747194+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:21:06.454365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938363+0000","last_change":"2024-03-23T11:22:31.770238+0000","last_active":"2024-03-23T11:22:32.938363+0000","last_peered":"2024-03-23T11:22:32.938363+0000","last_clean":"2024-03-23T11:22:32.938363+0000","last_became_active":"2024-03-23T11:22:31.766595+0000","last_became_peered":"2024-03-23T11:22:31.766595+0000","last_unstale":"2024-03-23T11:22:32.938363+0000","last_undegraded":"2024-03-23T11:22:32.938363+0000","last_fullsized":"2024-03-23T11:22:32.938363+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:32:36.695167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746846+0000","last_change":"2024-03-23T11:22:31.257165+0000","last_active":"2024-03-23T11:22:32.746846+0000","last_peered":"2024-03-23T11:22:32.746846+0000","last_clean":"2024-03-23T11:22:32.746846+0000","last_became_active":"2024-03-23T11:22:31.256771+0000","last_became_peered":"2024-03-23T11:22:31.256771+0000","last_unstale":"2024-03-23T11:22:32.746846+0000","last_undegraded":"2024-03-23T11:22:32.746846+0000","last_fullsized":"2024-03-23T11:22:32.746846+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:05:40.831430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746657+0000","last_change":"2024-03-23T11:22:31.754431+0000","last_active":"2024-03-23T11:22:32.746657+0000","last_peered":"2024-03-23T11:22:32.746657+0000","last_clean":"2024-03-23T11:22:32.746657+0000","last_became_active":"2024-03-23T11:22:31.753838+0000","last_became_peered":"2024-03-23T11:22:31.753838+0000","last_unstale":"2024-03-23T11:22:32.746657+0000","last_undegraded":"2024-03-23T11:22:32.746657+0000","last_fullsized":"2024-03-23T11:22:32.746657+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:29:43.839637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749315+0000","last_change":"2024-03-23T11:22:31.266527+0000","last_active":"2024-03-23T11:22:32.749315+0000","last_peered":"2024-03-23T11:22:32.749315+0000","last_clean":"2024-03-23T11:22:32.749315+0000","last_became_active":"2024-03-23T11:22:31.265979+0000","last_became_peered":"2024-03-23T11:22:31.265979+0000","last_unstale":"2024-03-23T11:22:32.749315+0000","last_undegraded":"2024-03-23T11:22:32.749315+0000","last_fullsized":"2024-03-23T11:22:32.749315+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:34:07.566641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749164+0000","last_change":"2024-03-23T11:22:31.764829+0000","last_active":"2024-03-23T11:22:32.749164+0000","last_peered":"2024-03-23T11:22:32.749164+0000","last_clean":"2024-03-23T11:22:32.749164+0000","last_became_active":"2024-03-23T11:22:31.764551+0000","last_became_peered":"2024-03-23T11:22:31.764551+0000","last_unstale":"2024-03-23T11:22:32.749164+0000","last_undegraded":"2024-03-23T11:22:32.749164+0000","last_fullsized":"2024-03-23T11:22:32.749164+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:00.036989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939444+0000","last_change":"2024-03-23T11:22:31.269216+0000","last_active":"2024-03-23T11:22:32.939444+0000","last_peered":"2024-03-23T11:22:32.939444+0000","last_clean":"2024-03-23T11:22:32.939444+0000","last_became_active":"2024-03-23T11:22:31.268653+0000","last_became_peered":"2024-03-23T11:22:31.268653+0000","last_unstale":"2024-03-23T11:22:32.939444+0000","last_undegraded":"2024-03-23T11:22:32.939444+0000","last_fullsized":"2024-03-23T11:22:32.939444+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:26.604228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768732+0000","last_change":"2024-03-23T11:22:31.766577+0000","last_active":"2024-03-23T11:22:31.768732+0000","last_peered":"2024-03-23T11:22:31.768732+0000","last_clean":"2024-03-23T11:22:31.768732+0000","last_became_active":"2024-03-23T11:22:31.762373+0000","last_became_peered":"2024-03-23T11:22:31.762373+0000","last_unstale":"2024-03-23T11:22:31.768732+0000","last_undegraded":"2024-03-23T11:22:31.768732+0000","last_fullsized":"2024-03-23T11:22:31.768732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:17:54.390500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"61'3","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749017+0000","last_change":"2024-03-23T11:22:31.266377+0000","last_active":"2024-03-23T11:22:32.749017+0000","last_peered":"2024-03-23T11:22:32.749017+0000","last_clean":"2024-03-23T11:22:32.749017+0000","last_became_active":"2024-03-23T11:22:31.264260+0000","last_became_peered":"2024-03-23T11:22:31.264260+0000","last_unstale":"2024-03-23T11:22:32.749017+0000","last_undegraded":"2024-03-23T11:22:32.749017+0000","last_fullsized":"2024-03-23T11:22:32.749017+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:56:04.434278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748911+0000","last_change":"2024-03-23T11:22:31.765120+0000","last_active":"2024-03-23T11:22:32.748911+0000","last_peered":"2024-03-23T11:22:32.748911+0000","last_clean":"2024-03-23T11:22:32.748911+0000","last_became_active":"2024-03-23T11:22:31.764698+0000","last_became_peered":"2024-03-23T11:22:31.764698+0000","last_unstale":"2024-03-23T11:22:32.748911+0000","last_undegraded":"2024-03-23T11:22:32.748911+0000","last_fullsized":"2024-03-23T11:22:32.748911+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:29:21.455065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"61'6","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.747133+0000","last_change":"2024-03-23T11:22:31.263750+0000","last_active":"2024-03-23T11:22:32.747133+0000","last_peered":"2024-03-23T11:22:32.747133+0000","last_clean":"2024-03-23T11:22:32.747133+0000","last_became_active":"2024-03-23T11:22:31.263456+0000","last_became_peered":"2024-03-23T11:22:31.263456+0000","last_unstale":"2024-03-23T11:22:32.747133+0000","last_undegraded":"2024-03-23T11:22:32.747133+0000","last_fullsized":"2024-03-23T11:22:32.747133+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:50:42.726444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.766708+0000","last_change":"2024-03-23T11:22:31.761908+0000","last_active":"2024-03-23T11:22:31.766708+0000","last_peered":"2024-03-23T11:22:31.766708+0000","last_clean":"2024-03-23T11:22:31.766708+0000","last_became_active":"2024-03-23T11:22:31.760305+0000","last_became_peered":"2024-03-23T11:22:31.760305+0000","last_unstale":"2024-03-23T11:22:31.766708+0000","last_undegraded":"2024-03-23T11:22:31.766708+0000","last_fullsized":"2024-03-23T11:22:31.766708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:00:51.026955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"61'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938812+0000","last_change":"2024-03-23T11:22:31.265890+0000","last_active":"2024-03-23T11:22:32.938812+0000","last_peered":"2024-03-23T11:22:32.938812+0000","last_clean":"2024-03-23T11:22:32.938812+0000","last_became_active":"2024-03-23T11:22:31.265618+0000","last_became_peered":"2024-03-23T11:22:31.265618+0000","last_unstale":"2024-03-23T11:22:32.938812+0000","last_undegraded":"2024-03-23T11:22:32.938812+0000","last_fullsized":"2024-03-23T11:22:32.938812+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:21:06.454365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748829+0000","last_change":"2024-03-23T11:22:31.753217+0000","last_active":"2024-03-23T11:22:32.748829+0000","last_peered":"2024-03-23T11:22:32.748829+0000","last_clean":"2024-03-23T11:22:32.748829+0000","last_became_active":"2024-03-23T11:22:31.752981+0000","last_became_peered":"2024-03-23T11:22:31.752981+0000","last_unstale":"2024-03-23T11:22:32.748829+0000","last_undegraded":"2024-03-23T11:22:32.748829+0000","last_fullsized":"2024-03-23T11:22:32.748829+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:55:49.707028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.746503+0000","last_change":"2024-03-23T11:22:31.267806+0000","last_active":"2024-03-23T11:22:31.746503+0000","last_peered":"2024-03-23T11:22:31.746503+0000","last_clean":"2024-03-23T11:22:31.746503+0000","last_became_active":"2024-03-23T11:22:31.267381+0000","last_became_peered":"2024-03-23T11:22:31.267381+0000","last_unstale":"2024-03-23T11:22:31.746503+0000","last_undegraded":"2024-03-23T11:22:31.746503+0000","last_fullsized":"2024-03-23T11:22:31.746503+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:41:33.525902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746455+0000","last_change":"2024-03-23T11:22:31.765499+0000","last_active":"2024-03-23T11:22:32.746455+0000","last_peered":"2024-03-23T11:22:32.746455+0000","last_clean":"2024-03-23T11:22:32.746455+0000","last_became_active":"2024-03-23T11:22:31.764902+0000","last_became_peered":"2024-03-23T11:22:31.764902+0000","last_unstale":"2024-03-23T11:22:32.746455+0000","last_undegraded":"2024-03-23T11:22:32.746455+0000","last_fullsized":"2024-03-23T11:22:32.746455+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:59:17.853936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.747302+0000","last_change":"2024-03-23T11:22:31.264671+0000","last_active":"2024-03-23T11:22:31.747302+0000","last_peered":"2024-03-23T11:22:31.747302+0000","last_clean":"2024-03-23T11:22:31.747302+0000","last_became_active":"2024-03-23T11:22:31.264346+0000","last_became_peered":"2024-03-23T11:22:31.264346+0000","last_unstale":"2024-03-23T11:22:31.747302+0000","last_undegraded":"2024-03-23T11:22:31.747302+0000","last_fullsized":"2024-03-23T11:22:31.747302+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:26.604228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748590+0000","last_change":"2024-03-23T11:22:31.761174+0000","last_active":"2024-03-23T11:22:32.748590+0000","last_peered":"2024-03-23T11:22:32.748590+0000","last_clean":"2024-03-23T11:22:32.748590+0000","last_became_active":"2024-03-23T11:22:31.754742+0000","last_became_peered":"2024-03-23T11:22:31.754742+0000","last_unstale":"2024-03-23T11:22:32.748590+0000","last_undegraded":"2024-03-23T11:22:32.748590+0000","last_fullsized":"2024-03-23T11:22:32.748590+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:10:07.723126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"61'8","reported_seq":19,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748242+0000","last_change":"2024-03-23T11:22:31.263806+0000","last_active":"2024-03-23T11:22:32.748242+0000","last_peered":"2024-03-23T11:22:32.748242+0000","last_clean":"2024-03-23T11:22:32.748242+0000","last_became_active":"2024-03-23T11:22:31.262960+0000","last_became_peered":"2024-03-23T11:22:31.262960+0000","last_unstale":"2024-03-23T11:22:32.748242+0000","last_undegraded":"2024-03-23T11:22:32.748242+0000","last_fullsized":"2024-03-23T11:22:32.748242+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:45:24.822418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749260+0000","last_change":"2024-03-23T11:22:31.758856+0000","last_active":"2024-03-23T11:22:32.749260+0000","last_peered":"2024-03-23T11:22:32.749260+0000","last_clean":"2024-03-23T11:22:32.749260+0000","last_became_active":"2024-03-23T11:22:31.758622+0000","last_became_peered":"2024-03-23T11:22:31.758622+0000","last_unstale":"2024-03-23T11:22:32.749260+0000","last_undegraded":"2024-03-23T11:22:32.749260+0000","last_fullsized":"2024-03-23T11:22:32.749260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:12:02.848172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"61'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939570+0000","last_change":"2024-03-23T11:22:31.265939+0000","last_active":"2024-03-23T11:22:32.939570+0000","last_peered":"2024-03-23T11:22:32.939570+0000","last_clean":"2024-03-23T11:22:32.939570+0000","last_became_active":"2024-03-23T11:22:31.263837+0000","last_became_peered":"2024-03-23T11:22:31.263837+0000","last_unstale":"2024-03-23T11:22:32.939570+0000","last_undegraded":"2024-03-23T11:22:32.939570+0000","last_fullsized":"2024-03-23T11:22:32.939570+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:53.321049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.769974+0000","last_change":"2024-03-23T11:22:31.766386+0000","last_active":"2024-03-23T11:22:31.769974+0000","last_peered":"2024-03-23T11:22:31.769974+0000","last_clean":"2024-03-23T11:22:31.769974+0000","last_became_active":"2024-03-23T11:22:31.766120+0000","last_became_peered":"2024-03-23T11:22:31.766120+0000","last_unstale":"2024-03-23T11:22:31.769974+0000","last_undegraded":"2024-03-23T11:22:31.769974+0000","last_fullsized":"2024-03-23T11:22:31.769974+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:22:35.016176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"61'2","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.773525+0000","last_change":"2024-03-23T11:22:31.268954+0000","last_active":"2024-03-23T11:22:31.773525+0000","last_peered":"2024-03-23T11:22:31.773525+0000","last_clean":"2024-03-23T11:22:31.773525+0000","last_became_active":"2024-03-23T11:22:31.268661+0000","last_became_peered":"2024-03-23T11:22:31.268661+0000","last_unstale":"2024-03-23T11:22:31.773525+0000","last_undegraded":"2024-03-23T11:22:31.773525+0000","last_fullsized":"2024-03-23T11:22:31.773525+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:50:47.230344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939350+0000","last_change":"2024-03-23T11:22:31.760846+0000","last_active":"2024-03-23T11:22:32.939350+0000","last_peered":"2024-03-23T11:22:32.939350+0000","last_clean":"2024-03-23T11:22:32.939350+0000","last_became_active":"2024-03-23T11:22:31.759124+0000","last_became_peered":"2024-03-23T11:22:31.759124+0000","last_unstale":"2024-03-23T11:22:32.939350+0000","last_undegraded":"2024-03-23T11:22:32.939350+0000","last_fullsized":"2024-03-23T11:22:32.939350+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:45:03.917246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"61'1","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.765614+0000","last_change":"2024-03-23T11:22:31.264775+0000","last_active":"2024-03-23T11:22:31.765614+0000","last_peered":"2024-03-23T11:22:31.765614+0000","last_clean":"2024-03-23T11:22:31.765614+0000","last_became_active":"2024-03-23T11:22:31.264424+0000","last_became_peered":"2024-03-23T11:22:31.264424+0000","last_unstale":"2024-03-23T11:22:31.765614+0000","last_undegraded":"2024-03-23T11:22:31.765614+0000","last_fullsized":"2024-03-23T11:22:31.765614+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:50:10.145832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748878+0000","last_change":"2024-03-23T11:22:31.760738+0000","last_active":"2024-03-23T11:22:32.748878+0000","last_peered":"2024-03-23T11:22:32.748878+0000","last_clean":"2024-03-23T11:22:32.748878+0000","last_became_active":"2024-03-23T11:22:31.759962+0000","last_became_peered":"2024-03-23T11:22:31.759962+0000","last_unstale":"2024-03-23T11:22:32.748878+0000","last_undegraded":"2024-03-23T11:22:32.748878+0000","last_fullsized":"2024-03-23T11:22:32.748878+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:40:12.835110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.763470+0000","last_change":"2024-03-23T11:22:31.258135+0000","last_active":"2024-03-23T11:22:31.763470+0000","last_peered":"2024-03-23T11:22:31.763470+0000","last_clean":"2024-03-23T11:22:31.763470+0000","last_became_active":"2024-03-23T11:22:31.257592+0000","last_became_peered":"2024-03-23T11:22:31.257592+0000","last_unstale":"2024-03-23T11:22:31.763470+0000","last_undegraded":"2024-03-23T11:22:31.763470+0000","last_fullsized":"2024-03-23T11:22:31.763470+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:54:10.046864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.762654+0000","last_change":"2024-03-23T11:22:31.758644+0000","last_active":"2024-03-23T11:22:31.762654+0000","last_peered":"2024-03-23T11:22:31.762654+0000","last_clean":"2024-03-23T11:22:31.762654+0000","last_became_active":"2024-03-23T11:22:31.755074+0000","last_became_peered":"2024-03-23T11:22:31.755074+0000","last_unstale":"2024-03-23T11:22:31.762654+0000","last_undegraded":"2024-03-23T11:22:31.762654+0000","last_fullsized":"2024-03-23T11:22:31.762654+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:14.173559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"61'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939101+0000","last_change":"2024-03-23T11:22:31.259688+0000","last_active":"2024-03-23T11:22:32.939101+0000","last_peered":"2024-03-23T11:22:32.939101+0000","last_clean":"2024-03-23T11:22:32.939101+0000","last_became_active":"2024-03-23T11:22:31.259460+0000","last_became_peered":"2024-03-23T11:22:31.259460+0000","last_unstale":"2024-03-23T11:22:32.939101+0000","last_undegraded":"2024-03-23T11:22:32.939101+0000","last_fullsized":"2024-03-23T11:22:32.939101+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:32.530250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.773250+0000","last_change":"2024-03-23T11:22:31.765060+0000","last_active":"2024-03-23T11:22:31.773250+0000","last_peered":"2024-03-23T11:22:31.773250+0000","last_clean":"2024-03-23T11:22:31.773250+0000","last_became_active":"2024-03-23T11:22:31.764769+0000","last_became_peered":"2024-03-23T11:22:31.764769+0000","last_unstale":"2024-03-23T11:22:31.773250+0000","last_undegraded":"2024-03-23T11:22:31.773250+0000","last_fullsized":"2024-03-23T11:22:31.773250+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:12:08.025880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.745835+0000","last_change":"2024-03-23T11:22:31.259575+0000","last_active":"2024-03-23T11:22:31.745835+0000","last_peered":"2024-03-23T11:22:31.745835+0000","last_clean":"2024-03-23T11:22:31.745835+0000","last_became_active":"2024-03-23T11:22:31.259282+0000","last_became_peered":"2024-03-23T11:22:31.259282+0000","last_unstale":"2024-03-23T11:22:31.745835+0000","last_undegraded":"2024-03-23T11:22:31.745835+0000","last_fullsized":"2024-03-23T11:22:31.745835+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:01.516022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.777773+0000","last_change":"2024-03-23T11:22:31.773034+0000","last_active":"2024-03-23T11:22:31.777773+0000","last_peered":"2024-03-23T11:22:31.777773+0000","last_clean":"2024-03-23T11:22:31.777773+0000","last_became_active":"2024-03-23T11:22:31.769007+0000","last_became_peered":"2024-03-23T11:22:31.769007+0000","last_unstale":"2024-03-23T11:22:31.777773+0000","last_undegraded":"2024-03-23T11:22:31.777773+0000","last_fullsized":"2024-03-23T11:22:31.777773+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:57.405432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.747859+0000","last_change":"2024-03-23T11:22:31.260154+0000","last_active":"2024-03-23T11:22:31.747859+0000","last_peered":"2024-03-23T11:22:31.747859+0000","last_clean":"2024-03-23T11:22:31.747859+0000","last_became_active":"2024-03-23T11:22:31.259709+0000","last_became_peered":"2024-03-23T11:22:31.259709+0000","last_unstale":"2024-03-23T11:22:31.747859+0000","last_undegraded":"2024-03-23T11:22:31.747859+0000","last_fullsized":"2024-03-23T11:22:31.747859+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:53.321049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.762708+0000","last_change":"2024-03-23T11:22:31.756415+0000","last_active":"2024-03-23T11:22:31.762708+0000","last_peered":"2024-03-23T11:22:31.762708+0000","last_clean":"2024-03-23T11:22:31.762708+0000","last_became_active":"2024-03-23T11:22:31.756127+0000","last_became_peered":"2024-03-23T11:22:31.756127+0000","last_unstale":"2024-03-23T11:22:31.762708+0000","last_undegraded":"2024-03-23T11:22:31.762708+0000","last_fullsized":"2024-03-23T11:22:31.762708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:15:02.678591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"61'2","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.767822+0000","last_change":"2024-03-23T11:22:31.254999+0000","last_active":"2024-03-23T11:22:31.767822+0000","last_peered":"2024-03-23T11:22:31.767822+0000","last_clean":"2024-03-23T11:22:31.767822+0000","last_became_active":"2024-03-23T11:22:31.254666+0000","last_became_peered":"2024-03-23T11:22:31.254666+0000","last_unstale":"2024-03-23T11:22:31.767822+0000","last_undegraded":"2024-03-23T11:22:31.767822+0000","last_fullsized":"2024-03-23T11:22:31.767822+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:09:50.262931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.766714+0000","last_change":"2024-03-23T11:22:31.763592+0000","last_active":"2024-03-23T11:22:31.766714+0000","last_peered":"2024-03-23T11:22:31.766714+0000","last_clean":"2024-03-23T11:22:31.766714+0000","last_became_active":"2024-03-23T11:22:31.763322+0000","last_became_peered":"2024-03-23T11:22:31.763322+0000","last_unstale":"2024-03-23T11:22:31.766714+0000","last_undegraded":"2024-03-23T11:22:31.766714+0000","last_fullsized":"2024-03-23T11:22:31.766714+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:12:08.025880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.761977+0000","last_change":"2024-03-23T11:22:31.265429+0000","last_active":"2024-03-23T11:22:31.761977+0000","last_peered":"2024-03-23T11:22:31.761977+0000","last_clean":"2024-03-23T11:22:31.761977+0000","last_became_active":"2024-03-23T11:22:31.264989+0000","last_became_peered":"2024-03-23T11:22:31.264989+0000","last_unstale":"2024-03-23T11:22:31.761977+0000","last_undegraded":"2024-03-23T11:22:31.761977+0000","last_fullsized":"2024-03-23T11:22:31.761977+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:41:46.750819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768711+0000","last_change":"2024-03-23T11:22:31.766413+0000","last_active":"2024-03-23T11:22:31.768711+0000","last_peered":"2024-03-23T11:22:31.768711+0000","last_clean":"2024-03-23T11:22:31.768711+0000","last_became_active":"2024-03-23T11:22:31.764791+0000","last_became_peered":"2024-03-23T11:22:31.764791+0000","last_unstale":"2024-03-23T11:22:31.768711+0000","last_undegraded":"2024-03-23T11:22:31.768711+0000","last_fullsized":"2024-03-23T11:22:31.768711+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:02.527716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.746622+0000","last_change":"2024-03-23T11:22:31.259321+0000","last_active":"2024-03-23T11:22:31.746622+0000","last_peered":"2024-03-23T11:22:31.746622+0000","last_clean":"2024-03-23T11:22:31.746622+0000","last_became_active":"2024-03-23T11:22:31.258609+0000","last_became_peered":"2024-03-23T11:22:31.258609+0000","last_unstale":"2024-03-23T11:22:31.746622+0000","last_undegraded":"2024-03-23T11:22:31.746622+0000","last_fullsized":"2024-03-23T11:22:31.746622+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:50:42.726444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778077+0000","last_change":"2024-03-23T11:22:31.773897+0000","last_active":"2024-03-23T11:22:31.778077+0000","last_peered":"2024-03-23T11:22:31.778077+0000","last_clean":"2024-03-23T11:22:31.778077+0000","last_became_active":"2024-03-23T11:22:31.767497+0000","last_became_peered":"2024-03-23T11:22:31.767497+0000","last_unstale":"2024-03-23T11:22:31.778077+0000","last_undegraded":"2024-03-23T11:22:31.778077+0000","last_fullsized":"2024-03-23T11:22:31.778077+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:09:54.090512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.746374+0000","last_change":"2024-03-23T11:22:31.260220+0000","last_active":"2024-03-23T11:22:31.746374+0000","last_peered":"2024-03-23T11:22:31.746374+0000","last_clean":"2024-03-23T11:22:31.746374+0000","last_became_active":"2024-03-23T11:22:31.259912+0000","last_became_peered":"2024-03-23T11:22:31.259912+0000","last_unstale":"2024-03-23T11:22:31.746374+0000","last_undegraded":"2024-03-23T11:22:31.746374+0000","last_fullsized":"2024-03-23T11:22:31.746374+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:02.515842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938953+0000","last_change":"2024-03-23T11:22:31.753748+0000","last_active":"2024-03-23T11:22:32.938953+0000","last_peered":"2024-03-23T11:22:32.938953+0000","last_clean":"2024-03-23T11:22:32.938953+0000","last_became_active":"2024-03-23T11:22:31.753494+0000","last_became_peered":"2024-03-23T11:22:31.753494+0000","last_unstale":"2024-03-23T11:22:32.938953+0000","last_undegraded":"2024-03-23T11:22:32.938953+0000","last_fullsized":"2024-03-23T11:22:32.938953+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:52:20.354587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.749975+0000","last_change":"2024-03-23T11:22:31.262108+0000","last_active":"2024-03-23T11:22:31.749975+0000","last_peered":"2024-03-23T11:22:31.749975+0000","last_clean":"2024-03-23T11:22:31.749975+0000","last_became_active":"2024-03-23T11:22:31.261843+0000","last_became_peered":"2024-03-23T11:22:31.261843+0000","last_unstale":"2024-03-23T11:22:31.749975+0000","last_undegraded":"2024-03-23T11:22:31.749975+0000","last_fullsized":"2024-03-23T11:22:31.749975+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:13:41.293233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749032+0000","last_change":"2024-03-23T11:22:31.753894+0000","last_active":"2024-03-23T11:22:32.749032+0000","last_peered":"2024-03-23T11:22:32.749032+0000","last_clean":"2024-03-23T11:22:32.749032+0000","last_became_active":"2024-03-23T11:22:31.753553+0000","last_became_peered":"2024-03-23T11:22:31.753553+0000","last_unstale":"2024-03-23T11:22:32.749032+0000","last_undegraded":"2024-03-23T11:22:32.749032+0000","last_fullsized":"2024-03-23T11:22:32.749032+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:23:26.437000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746657+0000","last_change":"2024-03-23T11:22:31.265577+0000","last_active":"2024-03-23T11:22:32.746657+0000","last_peered":"2024-03-23T11:22:32.746657+0000","last_clean":"2024-03-23T11:22:32.746657+0000","last_became_active":"2024-03-23T11:22:31.265317+0000","last_became_peered":"2024-03-23T11:22:31.265317+0000","last_unstale":"2024-03-23T11:22:32.746657+0000","last_undegraded":"2024-03-23T11:22:32.746657+0000","last_fullsized":"2024-03-23T11:22:32.746657+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:41:33.525902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768238+0000","last_change":"2024-03-23T11:22:31.759129+0000","last_active":"2024-03-23T11:22:31.768238+0000","last_peered":"2024-03-23T11:22:31.768238+0000","last_clean":"2024-03-23T11:22:31.768238+0000","last_became_active":"2024-03-23T11:22:31.758887+0000","last_became_peered":"2024-03-23T11:22:31.758887+0000","last_unstale":"2024-03-23T11:22:31.768238+0000","last_undegraded":"2024-03-23T11:22:31.768238+0000","last_fullsized":"2024-03-23T11:22:31.768238+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:46:29.980472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.746919+0000","last_change":"2024-03-23T11:22:31.265637+0000","last_active":"2024-03-23T11:22:31.746919+0000","last_peered":"2024-03-23T11:22:31.746919+0000","last_clean":"2024-03-23T11:22:31.746919+0000","last_became_active":"2024-03-23T11:22:31.265323+0000","last_became_peered":"2024-03-23T11:22:31.265323+0000","last_unstale":"2024-03-23T11:22:31.746919+0000","last_undegraded":"2024-03-23T11:22:31.746919+0000","last_fullsized":"2024-03-23T11:22:31.746919+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:54.898960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938475+0000","last_change":"2024-03-23T11:22:31.769993+0000","last_active":"2024-03-23T11:22:32.938475+0000","last_peered":"2024-03-23T11:22:32.938475+0000","last_clean":"2024-03-23T11:22:32.938475+0000","last_became_active":"2024-03-23T11:22:31.766568+0000","last_became_peered":"2024-03-23T11:22:31.766568+0000","last_unstale":"2024-03-23T11:22:32.938475+0000","last_undegraded":"2024-03-23T11:22:32.938475+0000","last_fullsized":"2024-03-23T11:22:32.938475+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:23:56.372856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938712+0000","last_change":"2024-03-23T11:22:31.268143+0000","last_active":"2024-03-23T11:22:32.938712+0000","last_peered":"2024-03-23T11:22:32.938712+0000","last_clean":"2024-03-23T11:22:32.938712+0000","last_became_active":"2024-03-23T11:22:31.267125+0000","last_became_peered":"2024-03-23T11:22:31.267125+0000","last_unstale":"2024-03-23T11:22:32.938712+0000","last_undegraded":"2024-03-23T11:22:32.938712+0000","last_fullsized":"2024-03-23T11:22:32.938712+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:02.515842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938475+0000","last_change":"2024-03-23T11:22:31.777352+0000","last_active":"2024-03-23T11:22:32.938475+0000","last_peered":"2024-03-23T11:22:32.938475+0000","last_clean":"2024-03-23T11:22:32.938475+0000","last_became_active":"2024-03-23T11:22:31.776865+0000","last_became_peered":"2024-03-23T11:22:31.776865+0000","last_unstale":"2024-03-23T11:22:32.938475+0000","last_undegraded":"2024-03-23T11:22:32.938475+0000","last_fullsized":"2024-03-23T11:22:32.938475+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:30.728849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.746040+0000","last_change":"2024-03-23T11:22:31.261261+0000","last_active":"2024-03-23T11:22:31.746040+0000","last_peered":"2024-03-23T11:22:31.746040+0000","last_clean":"2024-03-23T11:22:31.746040+0000","last_became_active":"2024-03-23T11:22:31.260936+0000","last_became_peered":"2024-03-23T11:22:31.260936+0000","last_unstale":"2024-03-23T11:22:31.746040+0000","last_undegraded":"2024-03-23T11:22:31.746040+0000","last_fullsized":"2024-03-23T11:22:31.746040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:57.593149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":205,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.737206+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:22:31.737206+0000","last_peered":"2024-03-23T11:22:31.737206+0000","last_clean":"2024-03-23T11:22:31.737206+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:22:31.737206+0000","last_undegraded":"2024-03-23T11:22:31.737206+0000","last_fullsized":"2024-03-23T11:22:31.737206+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542192,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1331200,"data_stored":893171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.93500000000000005,"15min":0.90300000000000002},"min":{"1min":0.68400000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5309999999999999,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1,"15min":0.97999999999999998},"min":{"1min":0.67300000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.86499999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.1819999999999999,"15min":1.1799999999999999},"min":{"1min":0.75600000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.7430000000000001,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":1.494},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.234,"15min":1.246},"min":{"1min":0.82999999999999996,"5min":0.753,"15min":0.753},"max":{"1min":1.6870000000000001,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.091}]},{"osd":2,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.052,"15min":1.0509999999999999},"min":{"1min":0.78000000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.262,"5min":1.1539999999999999,"15min":1.1319999999999999},"min":{"1min":0.90200000000000002,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.4079999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.153,"15min":1.127},"min":{"1min":1.0509999999999999,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.86199999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.6479999999999999,"5min":1.778,"15min":1.778},"last":1.3240000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.208,"15min":1.212},"min":{"1min":0.90200000000000002,"5min":0.627,"15min":0.627},"max":{"1min":1.681,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.383,"15min":1.357},"min":{"1min":1.2390000000000001,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.323,"15min":1.3320000000000001},"min":{"1min":0.82999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.7470000000000001,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.244},{"interface":"front","average":{"1min":1.45,"5min":1.357,"15min":1.347},"min":{"1min":1.1200000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.8839999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.169}]},{"osd":6,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.4019999999999999,"15min":1.4139999999999999},"min":{"1min":1.0409999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.9139999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.3620000000000001,"15min":1.339},"min":{"1min":1.01,"5min":0.747,"15min":0.747},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.0349999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542192,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":301974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.107,"15min":1.073},"min":{"1min":0.69999999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.8169999999999999,"5min":3.9140000000000001,"15min":3.9140000000000001},"last":1.097},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1459999999999999,"15min":1.1140000000000001},"min":{"1min":0.746,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.48,"5min":6.6539999999999999,"15min":6.6539999999999999},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.21,"15min":1.22},"min":{"1min":0.78000000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6819999999999999,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.2509999999999999,"15min":1.2729999999999999},"min":{"1min":0.83299999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.575,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.032}]},{"osd":2,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.254,"15min":1.252},"min":{"1min":0.92000000000000004,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.5369999999999999,"5min":3.73,"15min":3.73},"last":1.127},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.272,"15min":1.296},"min":{"1min":0.81999999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.546,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.355}]},{"osd":3,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.335,"15min":1.262},"min":{"1min":0.91900000000000004,"5min":0.749,"15min":0.749},"max":{"1min":1.782,"5min":12.914,"15min":12.914},"last":1.161},{"interface":"front","average":{"1min":1.262,"5min":1.361,"15min":1.383},"min":{"1min":0.89300000000000002,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.7350000000000001,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":1.2190000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.371,"15min":1.4139999999999999},"min":{"1min":0.89700000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.2200000000000002,"5min":2.702,"15min":2.702},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.4339999999999999,"15min":1.4379999999999999},"min":{"1min":0.86599999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.6919999999999999,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":1.5609999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.528,"15min":1.5720000000000001},"min":{"1min":1.0149999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.794,"5min":3.097,"15min":3.097},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.4830000000000001,"15min":1.5149999999999999},"min":{"1min":0.97399999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.282,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.593}]},{"osd":7,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.6759999999999999,"15min":1.708},"min":{"1min":1.2689999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.8720000000000001,"5min":4.9470000000000001,"15min":4.9470000000000001},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.5129999999999999,"15min":1.5329999999999999},"min":{"1min":1.054,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.806,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":1.478}]}]},{"osd":5,"up_from":14,"seq":60129542192,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":737280,"data_stored":303295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1619999999999999,"15min":1.2130000000000001},"min":{"1min":0.72099999999999997,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.9350000000000001,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.127,"5min":1.1499999999999999,"15min":1.157},"min":{"1min":0.65600000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.5229999999999999,"5min":2.355,"15min":2.355},"last":0.77100000000000002}]},{"osd":1,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1890000000000001,"15min":1.2010000000000001},"min":{"1min":0.78600000000000003,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.6020000000000001,"5min":2.2559999999999998,"15min":2.2559999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.1950000000000001,"15min":1.228},"min":{"1min":0.69799999999999995,"5min":0.495,"15min":0.495},"max":{"1min":1.6299999999999999,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.82199999999999995}]},{"osd":2,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2749999999999999,"15min":1.2729999999999999},"min":{"1min":1.006,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.7390000000000001,"5min":2.012,"15min":2.012},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.226,"15min":1.24},"min":{"1min":0.77000000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.6299999999999999,"5min":1.946,"15min":1.946},"last":0.92300000000000004}]},{"osd":3,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.304,"15min":1.3140000000000001},"min":{"1min":0.998,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.831,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.879},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.391,"15min":1.421},"min":{"1min":0.92300000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.8939999999999999,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.0129999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.405,"15min":1.4410000000000001},"min":{"1min":0.84099999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.7470000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.54,"5min":1.532,"15min":1.5620000000000001},"min":{"1min":1.323,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.048,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.1899999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.456,"15min":1.504},"min":{"1min":0.94699999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.7370000000000001,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.105},{"interface":"front","average":{"1min":1.391,"5min":1.5089999999999999,"15min":1.5649999999999999},"min":{"1min":1.012,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.02,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.2669999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.54,"15min":1.587},"min":{"1min":1.0780000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.9419999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.6060000000000001,"15min":1.6399999999999999},"min":{"1min":1.252,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.7949999999999999,"5min":2.516,"15min":2.516},"last":1.2310000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542192,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":741376,"data_stored":304207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.111,"15min":1.099},"min":{"1min":0.79000000000000004,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.252,"5min":1.2050000000000001,"15min":1.2010000000000001},"min":{"1min":0.77100000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.5589999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.3009999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.091,"15min":1.097},"min":{"1min":0.80800000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":1.069},{"interface":"front","average":{"1min":1.179,"5min":1.1140000000000001,"15min":1.099},"min":{"1min":0.875,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.4930000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.4339999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2070000000000001,"15min":1.2270000000000001},"min":{"1min":0.82499999999999996,"5min":0.746,"15min":0.746},"max":{"1min":1.5289999999999999,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.131},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.22,"15min":1.212},"min":{"1min":1.0600000000000001,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.5309999999999999,"5min":1.696,"15min":1.696},"last":1.242}]},{"osd":3,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.2110000000000001,"15min":1.2150000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.5900000000000001,"5min":1.635,"15min":1.635},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2430000000000001,"15min":1.2370000000000001},"min":{"1min":0.99299999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.6890000000000001,"5min":1.909,"15min":1.909},"last":1.6679999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.2649999999999999,"15min":1.2929999999999999},"min":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.4450000000000001,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.476},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.327,"15min":1.3460000000000001},"min":{"1min":0.875,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.6319999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.7569999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.365,"15min":1.373},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.73,"5min":2.319,"15min":2.319},"last":1.391},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4710000000000001,"15min":1.4790000000000001},"min":{"1min":1.218,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.79,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.5089999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4570000000000001,"15min":1.4650000000000001},"min":{"1min":1.109,"5min":0.754,"15min":0.754},"max":{"1min":1.7929999999999999,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.599},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3899999999999999,"15min":1.421},"min":{"1min":0.84899999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.7290000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.6359999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542192,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27756,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344084,"statfs":{"total":96636764160,"available":96608342016,"internally_reserved":0,"allocated":749568,"data_stored":308999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95499999999999996,"15min":0.95599999999999996},"min":{"1min":0.53800000000000003,"5min":0.434,"15min":0.434},"max":{"1min":1.3440000000000001,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.075,"5min":1.093,"15min":1.101},"min":{"1min":0.58999999999999997,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4850000000000001,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.2190000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.99099999999999999,"15min":1.0009999999999999},"min":{"1min":0.68300000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.6499999999999999,"5min":1.976,"15min":1.976},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.224,"5min":1.244,"15min":1.262},"min":{"1min":0.65100000000000002,"5min":0.5,"15min":0.5},"max":{"1min":1.5640000000000001,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.5580000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.1359999999999999,"15min":1.1479999999999999},"min":{"1min":0.61899999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.514,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.2170000000000001,"15min":1.2689999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4790000000000001,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.863}]},{"osd":4,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.28,"15min":1.2929999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.5049999999999999,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.506,"15min":1.546},"min":{"1min":0.998,"5min":0.998,"15min":0.998},"max":{"1min":1.788,"5min":4.8970000000000002,"15min":4.8970000000000002},"last":1.8}]},{"osd":5,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.5069999999999999,"15min":1.5369999999999999},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":1.702,"5min":3.4910000000000001,"15min":3.4910000000000001},"last":1.135},{"interface":"front","average":{"1min":1.379,"5min":1.4490000000000001,"15min":1.4850000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.716,"5min":3.125,"15min":3.125},"last":1.48}]},{"osd":6,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.494,"15min":1.55},"min":{"1min":0.94499999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.7410000000000001,"5min":4.5499999999999998,"15min":4.5499999999999998},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.532,"15min":1.5740000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.915,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":1.7410000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.49,"15min":1.534},"min":{"1min":0.90600000000000003,"5min":0.877,"15min":0.877},"max":{"1min":1.6759999999999999,"5min":2.952,"15min":2.952},"last":1.6759999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.6000000000000001,"15min":1.649},"min":{"1min":1.0049999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.847,"5min":4.798,"15min":4.798},"last":1.9530000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542192,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":749568,"data_stored":308999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.86799999999999999,"15min":0.89700000000000002},"min":{"1min":0.39400000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.0109999999999999,"15min":0.97999999999999998},"min":{"1min":0.46000000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.669,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.77600000000000002}]},{"osd":1,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.95299999999999996,"15min":0.97299999999999998},"min":{"1min":0.41499999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.429,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0009999999999999,"15min":0.98099999999999998},"min":{"1min":0.38400000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.468,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.90800000000000003}]},{"osd":3,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.119,"15min":1.119},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5369999999999999,"5min":1.98,"15min":1.98},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0840000000000001,"15min":1.0900000000000001},"min":{"1min":0.47399999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4430000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":0.98199999999999998}]},{"osd":4,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.357,"15min":1.351},"min":{"1min":0.93100000000000005,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.379,"15min":1.349},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.3740000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4159999999999999,"15min":1.4339999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.97,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.8149999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.3700000000000001,"15min":1.3720000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.8979999999999999,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.401}]},{"osd":6,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3779999999999999,"15min":1.3700000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9370000000000001,"5min":2.758,"15min":2.758},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.5029999999999999,"15min":1.5309999999999999},"min":{"1min":0.98799999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.0310000000000001,"5min":2.8580000000000001,"15min":2.8580000000000001},"last":1.8879999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.419,"15min":1.391},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.2069999999999999,"5min":2.246,"15min":2.246},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5820000000000001,"15min":1.6100000000000001},"min":{"1min":1.03,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.8799999999999999,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.9510000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":729088,"data_stored":302064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.90200000000000002,"15min":0.86599999999999999},"min":{"1min":0.64300000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.6040000000000001,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.97699999999999998,"15min":0.98599999999999999},"min":{"1min":0.53700000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.7070000000000001,"5min":1.99,"15min":1.99},"last":0.89400000000000002}]},{"osd":2,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.96699999999999997,"15min":0.97899999999999998},"min":{"1min":0.64700000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.3540000000000001,"5min":1.738,"15min":1.738},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.1240000000000001,"15min":1.129},"min":{"1min":0.55500000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.8720000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.84599999999999997}]},{"osd":3,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2090000000000001,"15min":1.2210000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.6519999999999999,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.125,"5min":1.143,"15min":1.1739999999999999},"min":{"1min":0.72899999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.919,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.0980000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.262,"15min":1.2450000000000001},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.0720000000000001,"5min":2.153,"15min":2.153},"last":1.23},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.323,"15min":1.343},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6759999999999999,"5min":2.758,"15min":2.758},"last":1.663}]},{"osd":5,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.407,"15min":1.4450000000000001},"min":{"1min":0.96799999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.0009999999999999,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.9570000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4430000000000001,"15min":1.4670000000000001},"min":{"1min":0.80400000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.165,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.524}]},{"osd":6,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.429,"15min":1.4370000000000001},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.95,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.516,"15min":1.5820000000000001},"min":{"1min":0.96199999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.649,"5min":2.9630000000000001,"15min":2.9630000000000001},"last":1.841}]},{"osd":7,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.5529999999999999,"15min":1.5900000000000001},"min":{"1min":0.89700000000000002,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":2.2120000000000002,"5min":3.2749999999999999,"15min":3.2749999999999999},"last":1.8049999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.5649999999999999,"15min":1.5940000000000001},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.0840000000000001,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.768}]}]},{"osd":0,"up_from":14,"seq":60129542191,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28332,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343508,"statfs":{"total":96636764160,"available":96607752192,"internally_reserved":0,"allocated":1335296,"data_stored":894488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.86499999999999999,"15min":0.85699999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.94299999999999995,"15min":0.92500000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.601,"5min":1.694,"15min":1.694},"last":0.72399999999999998}]},{"osd":2,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.0489999999999999,"15min":1.0329999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.03,"15min":1.0089999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.5009999999999999,"5min":1.802,"15min":1.802},"last":1.294}]},{"osd":3,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.101,"15min":1.1020000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.306,"15min":1.3140000000000001},"min":{"1min":0.57499999999999996,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9450000000000001,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":0.89400000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.379,"15min":1.379},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.8080000000000001,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.266,"15min":1.268},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.73,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.546}]},{"osd":5,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.389,"15min":1.4490000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.597,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.339,"5min":1.5009999999999999,"15min":1.5469999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.77,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.252}]},{"osd":6,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.5229999999999999,"15min":1.5680000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7609999999999999,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.7210000000000001,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.468}]},{"osd":7,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.538,"15min":1.554},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.722,"5min":2.617,"15min":2.617},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4970000000000001,"15min":1.5149999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":2.0379999999999998,"5min":3.081,"15min":3.081},"last":1.6879999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:22:36.734 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:22:36.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.981+0000 7fd0e9c00640 1 Processor -- start 2024-03-23T11:22:36.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.981+0000 7fd0e9c00640 1 -- start start 2024-03-23T11:22:36.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.983+0000 7fd0e9c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd0e406c300 0x7fd0e40ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.983+0000 7fd0e9c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd0e406c6e0 0x7fd0e406cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.983+0000 7fd0e9c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e40cbc60 con 0x7fd0e40cafa0 2024-03-23T11:22:36.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.983+0000 7fd0e9c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e40cbf40 con 0x7fd0e406c6e0 2024-03-23T11:22:36.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.983+0000 7fd0e9c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e40cc220 con 0x7fd0e406c300 2024-03-23T11:22:36.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.983+0000 7fd0e3400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd0e406c6e0 0x7fd0e406cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.983+0000 7fd0dbe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd0e40cafa0 0x7fd0e40cb430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40092/0 (socket says 172.21.15.62:40092) 2024-03-23T11:22:36.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.983+0000 7fd0dbe00640 1 -- 172.21.15.62:0/2384241999 learned_addr learned my addr 172.21.15.62:0/2384241999 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:36.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.984+0000 7fd0e2a00640 1 --2- 172.21.15.62:0/2384241999 >> v2:172.21.15.120:3300/0 conn(0x7fd0e406c300 0x7fd0e40ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.984+0000 7fd0e3400640 1 -- 172.21.15.62:0/2384241999 >> v2:172.21.15.120:3300/0 conn(0x7fd0e406c300 msgr2=0x7fd0e40ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:36.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.984+0000 7fd0e3400640 1 --2- 172.21.15.62:0/2384241999 >> v2:172.21.15.120:3300/0 conn(0x7fd0e406c300 0x7fd0e40ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.984+0000 7fd0e3400640 1 -- 172.21.15.62:0/2384241999 >> v1:172.21.15.62:6789/0 conn(0x7fd0e40cafa0 legacy=0x7fd0e40cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.984+0000 7fd0e3400640 1 -- 172.21.15.62:0/2384241999 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e40cc500 con 0x7fd0e406c6e0 2024-03-23T11:22:36.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.984+0000 7fd0e2000640 1 -- 172.21.15.62:0/2384241999 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562907988 0 0) 0x7fd0e40cbc60 con 0x7fd0e40cafa0 2024-03-23T11:22:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.985+0000 7fd0e3400640 1 --2- 172.21.15.62:0/2384241999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0e406c6e0 0x7fd0e406cb30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd0e406c1b0 tx=0x7fd0dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0d69346eb7068c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.985+0000 7fd0e2000640 1 -- 172.21.15.62:0/2384241999 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc02cc80 con 0x7fd0e406c6e0 2024-03-23T11:22:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.985+0000 7fd0e2000640 1 -- 172.21.15.62:0/2384241999 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc02cde0 con 0x7fd0e406c6e0 2024-03-23T11:22:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.985+0000 7fd0e2000640 1 -- 172.21.15.62:0/2384241999 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc034660 con 0x7fd0e406c6e0 2024-03-23T11:22:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.986+0000 7fd0e9c00640 1 -- 172.21.15.62:0/2384241999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0e406c6e0 msgr2=0x7fd0e406cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.986+0000 7fd0e9c00640 1 --2- 172.21.15.62:0/2384241999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0e406c6e0 0x7fd0e406cb30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd0e406c1b0 tx=0x7fd0dc02bed0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.986+0000 7fd0e9c00640 1 -- 172.21.15.62:0/2384241999 shutdown_connections 2024-03-23T11:22:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.986+0000 7fd0e9c00640 1 --2- 172.21.15.62:0/2384241999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0e406c6e0 0x7fd0e406cb30 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.986+0000 7fd0e9c00640 1 --2- 172.21.15.62:0/2384241999 >> v2:172.21.15.120:3300/0 conn(0x7fd0e406c300 0x7fd0e40ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.986+0000 7fd0e9c00640 1 -- 172.21.15.62:0/2384241999 >> 172.21.15.62:0/2384241999 conn(0x7fd0e40b02e0 msgr2=0x7fd0e40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.986+0000 7fd0e9c00640 1 -- 172.21.15.62:0/2384241999 shutdown_connections 2024-03-23T11:22:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.986+0000 7fd0e9c00640 1 -- 172.21.15.62:0/2384241999 wait complete. 2024-03-23T11:22:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.988+0000 7fd0e9c00640 1 Processor -- start 2024-03-23T11:22:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.988+0000 7fd0e9c00640 1 -- start start 2024-03-23T11:22:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.989+0000 7fd0e9c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0e406c300 0x7fd0e4156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.989+0000 7fd0e9c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0e406c6e0 0x7fd0e413f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.989+0000 7fd0e9c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0e40cafa0 0x7fd0e413fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.989+0000 7fd0e9c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e41567a0 con 0x7fd0e406c300 2024-03-23T11:22:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.989+0000 7fd0e9c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e40abcc0 con 0x7fd0e406c6e0 2024-03-23T11:22:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.989+0000 7fd0e9c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e415d6f0 con 0x7fd0e40cafa0 2024-03-23T11:22:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.989+0000 7fd0e2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0e406c300 0x7fd0e4156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.989+0000 7fd0e2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0e406c300 0x7fd0e4156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45106/0 (socket says 172.21.15.62:45106) 2024-03-23T11:22:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.989+0000 7fd0e2a00640 1 -- 172.21.15.62:0/573057260 learned_addr learned my addr 172.21.15.62:0/573057260 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0dbe00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0e406c6e0 0x7fd0e413f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0e3400640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0e40cafa0 0x7fd0e413fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0e2a00640 1 -- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0e40cafa0 msgr2=0x7fd0e413fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0e2a00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0e40cafa0 0x7fd0e413fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0e2a00640 1 -- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0e406c6e0 msgr2=0x7fd0e413f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0e2a00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0e406c6e0 0x7fd0e413f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0e2a00640 1 -- 172.21.15.62:0/573057260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc009a70 con 0x7fd0e406c300 2024-03-23T11:22:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0e3400640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0e40cafa0 0x7fd0e413fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0e2a00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0e406c300 0x7fd0e4156260 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400d830 tx=0x7fd0d400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0daa00640 1 -- 172.21.15.62:0/573057260 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d400b6c0 con 0x7fd0e406c300 2024-03-23T11:22:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0daa00640 1 -- 172.21.15.62:0/573057260 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d400b820 con 0x7fd0e406c300 2024-03-23T11:22:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.990+0000 7fd0daa00640 1 -- 172.21.15.62:0/573057260 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d400bbb0 con 0x7fd0e406c300 2024-03-23T11:22:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.991+0000 7fd0e9c00640 1 -- 172.21.15.62:0/573057260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e415ce50 con 0x7fd0e406c300 2024-03-23T11:22:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.991+0000 7fd0e9c00640 1 -- 172.21.15.62:0/573057260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e415d410 con 0x7fd0e406c300 2024-03-23T11:22:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.992+0000 7fd0e9c00640 1 -- 172.21.15.62:0/573057260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e40ce100 con 0x7fd0e406c300 2024-03-23T11:22:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.993+0000 7fd0daa00640 1 -- 172.21.15.62:0/573057260 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd0d401e460 con 0x7fd0e406c300 2024-03-23T11:22:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.994+0000 7fd0daa00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd0c006a7a0 0x7fd0c006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.994+0000 7fd0daa00640 1 -- 172.21.15.62:0/573057260 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0d4089550 con 0x7fd0e406c300 2024-03-23T11:22:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.995+0000 7fd0dbe00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd0c006a7a0 0x7fd0c006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:36.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.995+0000 7fd0dbe00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd0c006a7a0 0x7fd0c006cc60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40cb810 tx=0x7fd0cc007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:36.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:36.998+0000 7fd0daa00640 1 -- 172.21.15.62:0/573057260 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0d4054970 con 0x7fd0e406c300 2024-03-23T11:22:37.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.188+0000 7fd0e9c00640 1 -- 172.21.15.62:0/573057260 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0e406d070 con 0x7fd0c006a7a0 2024-03-23T11:22:37.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.211+0000 7fd0daa00640 1 -- 172.21.15.62:0/573057260 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286864 (secure 0 0 0) 0x7fd0cc007270 con 0x7fd0c006a7a0 2024-03-23T11:22:37.214 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:37.215 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:22:37.217 INFO:tasks.ceph.mds.b.smithi120.stderr:2024-03-23T11:22:37.216+0000 7f3f3bc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:22:37.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 -- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd0c006a7a0 msgr2=0x7fd0c006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:37.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd0c006a7a0 0x7fd0c006cc60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40cb810 tx=0x7fd0cc007c00 comp rx=0 tx=0).stop 2024-03-23T11:22:37.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 -- 172.21.15.62:0/573057260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0e406c300 msgr2=0x7fd0e4156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:37.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0e406c300 0x7fd0e4156260 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400d830 tx=0x7fd0d400dcf0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 -- 172.21.15.62:0/573057260 shutdown_connections 2024-03-23T11:22:37.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd0c006a7a0 0x7fd0c006cc60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0e40cafa0 0x7fd0e413fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0e406c6e0 0x7fd0e413f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 --2- 172.21.15.62:0/573057260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0e406c300 0x7fd0e4156260 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 -- 172.21.15.62:0/573057260 >> 172.21.15.62:0/573057260 conn(0x7fd0e40b02e0 msgr2=0x7fd0e4157010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:37.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 -- 172.21.15.62:0/573057260 shutdown_connections 2024-03-23T11:22:37.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.218+0000 7fd0e9c00640 1 -- 172.21.15.62:0/573057260 wait complete. 2024-03-23T11:22:37.235 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-03-23T11:22:35.307450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231316,"kb_used_data":6932,"kb_used_omap":13,"kb_used_meta":224242,"kb_avail":754743404,"statfs":{"total":773094113280,"available":772857245696,"internally_reserved":0,"allocated":7098368,"data_stored":3617197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":229623896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.085831"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745756+0000","last_change":"2024-03-23T11:22:31.755822+0000","last_active":"2024-03-23T11:22:32.745756+0000","last_peered":"2024-03-23T11:22:32.745756+0000","last_clean":"2024-03-23T11:22:32.745756+0000","last_became_active":"2024-03-23T11:22:31.754982+0000","last_became_peered":"2024-03-23T11:22:31.754982+0000","last_unstale":"2024-03-23T11:22:32.745756+0000","last_undegraded":"2024-03-23T11:22:32.745756+0000","last_fullsized":"2024-03-23T11:22:32.745756+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:48:12.358869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939162+0000","last_change":"2024-03-23T11:22:31.757391+0000","last_active":"2024-03-23T11:22:32.939162+0000","last_peered":"2024-03-23T11:22:32.939162+0000","last_clean":"2024-03-23T11:22:32.939162+0000","last_became_active":"2024-03-23T11:22:31.757140+0000","last_became_peered":"2024-03-23T11:22:31.757140+0000","last_unstale":"2024-03-23T11:22:32.939162+0000","last_undegraded":"2024-03-23T11:22:32.939162+0000","last_fullsized":"2024-03-23T11:22:32.939162+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:28:10.126290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748782+0000","last_change":"2024-03-23T11:22:31.759403+0000","last_active":"2024-03-23T11:22:32.748782+0000","last_peered":"2024-03-23T11:22:32.748782+0000","last_clean":"2024-03-23T11:22:32.748782+0000","last_became_active":"2024-03-23T11:22:31.759095+0000","last_became_peered":"2024-03-23T11:22:31.759095+0000","last_unstale":"2024-03-23T11:22:32.748782+0000","last_undegraded":"2024-03-23T11:22:32.748782+0000","last_fullsized":"2024-03-23T11:22:32.748782+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:21:39.030934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.762803+0000","last_change":"2024-03-23T11:22:31.757112+0000","last_active":"2024-03-23T11:22:31.762803+0000","last_peered":"2024-03-23T11:22:31.762803+0000","last_clean":"2024-03-23T11:22:31.762803+0000","last_became_active":"2024-03-23T11:22:31.756746+0000","last_became_peered":"2024-03-23T11:22:31.756746+0000","last_unstale":"2024-03-23T11:22:31.762803+0000","last_undegraded":"2024-03-23T11:22:31.762803+0000","last_fullsized":"2024-03-23T11:22:31.762803+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:46:36.179754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745535+0000","last_change":"2024-03-23T11:22:31.756871+0000","last_active":"2024-03-23T11:22:32.745535+0000","last_peered":"2024-03-23T11:22:32.745535+0000","last_clean":"2024-03-23T11:22:32.745535+0000","last_became_active":"2024-03-23T11:22:31.756597+0000","last_became_peered":"2024-03-23T11:22:31.756597+0000","last_unstale":"2024-03-23T11:22:32.745535+0000","last_undegraded":"2024-03-23T11:22:32.745535+0000","last_fullsized":"2024-03-23T11:22:32.745535+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:16:37.145910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778135+0000","last_change":"2024-03-23T11:22:31.774723+0000","last_active":"2024-03-23T11:22:31.778135+0000","last_peered":"2024-03-23T11:22:31.778135+0000","last_clean":"2024-03-23T11:22:31.778135+0000","last_became_active":"2024-03-23T11:22:31.771125+0000","last_became_peered":"2024-03-23T11:22:31.771125+0000","last_unstale":"2024-03-23T11:22:31.778135+0000","last_undegraded":"2024-03-23T11:22:31.778135+0000","last_fullsized":"2024-03-23T11:22:31.778135+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:46:36.179754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746660+0000","last_change":"2024-03-23T11:22:31.752247+0000","last_active":"2024-03-23T11:22:32.746660+0000","last_peered":"2024-03-23T11:22:32.746660+0000","last_clean":"2024-03-23T11:22:32.746660+0000","last_became_active":"2024-03-23T11:22:31.751333+0000","last_became_peered":"2024-03-23T11:22:31.751333+0000","last_unstale":"2024-03-23T11:22:32.746660+0000","last_undegraded":"2024-03-23T11:22:32.746660+0000","last_fullsized":"2024-03-23T11:22:32.746660+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:50:48.239290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938391+0000","last_change":"2024-03-23T11:22:31.747632+0000","last_active":"2024-03-23T11:22:32.938391+0000","last_peered":"2024-03-23T11:22:32.938391+0000","last_clean":"2024-03-23T11:22:32.938391+0000","last_became_active":"2024-03-23T11:22:31.747352+0000","last_became_peered":"2024-03-23T11:22:31.747352+0000","last_unstale":"2024-03-23T11:22:32.938391+0000","last_undegraded":"2024-03-23T11:22:32.938391+0000","last_fullsized":"2024-03-23T11:22:32.938391+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:44:33.088453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746805+0000","last_change":"2024-03-23T11:22:31.769768+0000","last_active":"2024-03-23T11:22:32.746805+0000","last_peered":"2024-03-23T11:22:32.746805+0000","last_clean":"2024-03-23T11:22:32.746805+0000","last_became_active":"2024-03-23T11:22:31.769448+0000","last_became_peered":"2024-03-23T11:22:31.769448+0000","last_unstale":"2024-03-23T11:22:32.746805+0000","last_undegraded":"2024-03-23T11:22:32.746805+0000","last_fullsized":"2024-03-23T11:22:32.746805+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:50:11.154778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.766880+0000","last_change":"2024-03-23T11:22:31.756698+0000","last_active":"2024-03-23T11:22:31.766880+0000","last_peered":"2024-03-23T11:22:31.766880+0000","last_clean":"2024-03-23T11:22:31.766880+0000","last_became_active":"2024-03-23T11:22:31.756407+0000","last_became_peered":"2024-03-23T11:22:31.756407+0000","last_unstale":"2024-03-23T11:22:31.766880+0000","last_undegraded":"2024-03-23T11:22:31.766880+0000","last_fullsized":"2024-03-23T11:22:31.766880+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:30:13.644040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768754+0000","last_change":"2024-03-23T11:22:31.765943+0000","last_active":"2024-03-23T11:22:31.768754+0000","last_peered":"2024-03-23T11:22:31.768754+0000","last_clean":"2024-03-23T11:22:31.768754+0000","last_became_active":"2024-03-23T11:22:31.765536+0000","last_became_peered":"2024-03-23T11:22:31.765536+0000","last_unstale":"2024-03-23T11:22:31.768754+0000","last_undegraded":"2024-03-23T11:22:31.768754+0000","last_fullsized":"2024-03-23T11:22:31.768754+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:30:55.729393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.766729+0000","last_change":"2024-03-23T11:22:31.762805+0000","last_active":"2024-03-23T11:22:31.766729+0000","last_peered":"2024-03-23T11:22:31.766729+0000","last_clean":"2024-03-23T11:22:31.766729+0000","last_became_active":"2024-03-23T11:22:31.762532+0000","last_became_peered":"2024-03-23T11:22:31.762532+0000","last_unstale":"2024-03-23T11:22:31.766729+0000","last_undegraded":"2024-03-23T11:22:31.766729+0000","last_fullsized":"2024-03-23T11:22:31.766729+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:22:24.908706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768490+0000","last_change":"2024-03-23T11:22:31.766751+0000","last_active":"2024-03-23T11:22:31.768490+0000","last_peered":"2024-03-23T11:22:31.768490+0000","last_clean":"2024-03-23T11:22:31.768490+0000","last_became_active":"2024-03-23T11:22:31.766162+0000","last_became_peered":"2024-03-23T11:22:31.766162+0000","last_unstale":"2024-03-23T11:22:31.768490+0000","last_undegraded":"2024-03-23T11:22:31.768490+0000","last_fullsized":"2024-03-23T11:22:31.768490+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:45:00.120978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746532+0000","last_change":"2024-03-23T11:22:31.753003+0000","last_active":"2024-03-23T11:22:32.746532+0000","last_peered":"2024-03-23T11:22:32.746532+0000","last_clean":"2024-03-23T11:22:32.746532+0000","last_became_active":"2024-03-23T11:22:31.752661+0000","last_became_peered":"2024-03-23T11:22:31.752661+0000","last_unstale":"2024-03-23T11:22:32.746532+0000","last_undegraded":"2024-03-23T11:22:32.746532+0000","last_fullsized":"2024-03-23T11:22:32.746532+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:49:52.269908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746690+0000","last_change":"2024-03-23T11:22:31.753832+0000","last_active":"2024-03-23T11:22:32.746690+0000","last_peered":"2024-03-23T11:22:32.746690+0000","last_clean":"2024-03-23T11:22:32.746690+0000","last_became_active":"2024-03-23T11:22:31.753590+0000","last_became_peered":"2024-03-23T11:22:31.753590+0000","last_unstale":"2024-03-23T11:22:32.746690+0000","last_undegraded":"2024-03-23T11:22:32.746690+0000","last_fullsized":"2024-03-23T11:22:32.746690+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:54:11.055811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778282+0000","last_change":"2024-03-23T11:22:31.776363+0000","last_active":"2024-03-23T11:22:31.778282+0000","last_peered":"2024-03-23T11:22:31.778282+0000","last_clean":"2024-03-23T11:22:31.778282+0000","last_became_active":"2024-03-23T11:22:31.771638+0000","last_became_peered":"2024-03-23T11:22:31.771638+0000","last_unstale":"2024-03-23T11:22:31.778282+0000","last_undegraded":"2024-03-23T11:22:31.778282+0000","last_fullsized":"2024-03-23T11:22:31.778282+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:00:51.026955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939299+0000","last_change":"2024-03-23T11:22:31.761897+0000","last_active":"2024-03-23T11:22:32.939299+0000","last_peered":"2024-03-23T11:22:32.939299+0000","last_clean":"2024-03-23T11:22:32.939299+0000","last_became_active":"2024-03-23T11:22:31.761195+0000","last_became_peered":"2024-03-23T11:22:31.761195+0000","last_unstale":"2024-03-23T11:22:32.939299+0000","last_undegraded":"2024-03-23T11:22:32.939299+0000","last_fullsized":"2024-03-23T11:22:32.939299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:47:58.423989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749186+0000","last_change":"2024-03-23T11:22:31.776125+0000","last_active":"2024-03-23T11:22:32.749186+0000","last_peered":"2024-03-23T11:22:32.749186+0000","last_clean":"2024-03-23T11:22:32.749186+0000","last_became_active":"2024-03-23T11:22:31.775617+0000","last_became_peered":"2024-03-23T11:22:31.775617+0000","last_unstale":"2024-03-23T11:22:32.749186+0000","last_undegraded":"2024-03-23T11:22:32.749186+0000","last_fullsized":"2024-03-23T11:22:32.749186+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:06.149914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745601+0000","last_change":"2024-03-23T11:22:31.755309+0000","last_active":"2024-03-23T11:22:32.745601+0000","last_peered":"2024-03-23T11:22:32.745601+0000","last_clean":"2024-03-23T11:22:32.745601+0000","last_became_active":"2024-03-23T11:22:31.755048+0000","last_became_peered":"2024-03-23T11:22:31.755048+0000","last_unstale":"2024-03-23T11:22:32.745601+0000","last_undegraded":"2024-03-23T11:22:32.745601+0000","last_fullsized":"2024-03-23T11:22:32.745601+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:03:24.515541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939090+0000","last_change":"2024-03-23T11:22:31.774011+0000","last_active":"2024-03-23T11:22:32.939090+0000","last_peered":"2024-03-23T11:22:32.939090+0000","last_clean":"2024-03-23T11:22:32.939090+0000","last_became_active":"2024-03-23T11:22:31.773707+0000","last_became_peered":"2024-03-23T11:22:31.773707+0000","last_unstale":"2024-03-23T11:22:32.939090+0000","last_undegraded":"2024-03-23T11:22:32.939090+0000","last_fullsized":"2024-03-23T11:22:32.939090+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:56.760297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748730+0000","last_change":"2024-03-23T11:22:31.758066+0000","last_active":"2024-03-23T11:22:32.748730+0000","last_peered":"2024-03-23T11:22:32.748730+0000","last_clean":"2024-03-23T11:22:32.748730+0000","last_became_active":"2024-03-23T11:22:31.757851+0000","last_became_peered":"2024-03-23T11:22:31.757851+0000","last_unstale":"2024-03-23T11:22:32.748730+0000","last_undegraded":"2024-03-23T11:22:32.748730+0000","last_fullsized":"2024-03-23T11:22:32.748730+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:31:50.946464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938633+0000","last_change":"2024-03-23T11:22:31.750381+0000","last_active":"2024-03-23T11:22:32.938633+0000","last_peered":"2024-03-23T11:22:32.938633+0000","last_clean":"2024-03-23T11:22:32.938633+0000","last_became_active":"2024-03-23T11:22:31.749517+0000","last_became_peered":"2024-03-23T11:22:31.749517+0000","last_unstale":"2024-03-23T11:22:32.938633+0000","last_undegraded":"2024-03-23T11:22:32.938633+0000","last_fullsized":"2024-03-23T11:22:32.938633+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:16:43.753148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748444+0000","last_change":"2024-03-23T11:22:31.756649+0000","last_active":"2024-03-23T11:22:32.748444+0000","last_peered":"2024-03-23T11:22:32.748444+0000","last_clean":"2024-03-23T11:22:32.748444+0000","last_became_active":"2024-03-23T11:22:31.754412+0000","last_became_peered":"2024-03-23T11:22:31.754412+0000","last_unstale":"2024-03-23T11:22:32.748444+0000","last_undegraded":"2024-03-23T11:22:32.748444+0000","last_fullsized":"2024-03-23T11:22:32.748444+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:55:42.979301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778024+0000","last_change":"2024-03-23T11:22:31.777173+0000","last_active":"2024-03-23T11:22:31.778024+0000","last_peered":"2024-03-23T11:22:31.778024+0000","last_clean":"2024-03-23T11:22:31.778024+0000","last_became_active":"2024-03-23T11:22:31.776866+0000","last_became_peered":"2024-03-23T11:22:31.776866+0000","last_unstale":"2024-03-23T11:22:31.778024+0000","last_undegraded":"2024-03-23T11:22:31.778024+0000","last_fullsized":"2024-03-23T11:22:31.778024+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:52:26.665790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745923+0000","last_change":"2024-03-23T11:22:31.764643+0000","last_active":"2024-03-23T11:22:32.745923+0000","last_peered":"2024-03-23T11:22:32.745923+0000","last_clean":"2024-03-23T11:22:32.745923+0000","last_became_active":"2024-03-23T11:22:31.763457+0000","last_became_peered":"2024-03-23T11:22:31.763457+0000","last_unstale":"2024-03-23T11:22:32.745923+0000","last_undegraded":"2024-03-23T11:22:32.745923+0000","last_fullsized":"2024-03-23T11:22:32.745923+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:01:29.163243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.762687+0000","last_change":"2024-03-23T11:22:31.755850+0000","last_active":"2024-03-23T11:22:31.762687+0000","last_peered":"2024-03-23T11:22:31.762687+0000","last_clean":"2024-03-23T11:22:31.762687+0000","last_became_active":"2024-03-23T11:22:31.755584+0000","last_became_peered":"2024-03-23T11:22:31.755584+0000","last_unstale":"2024-03-23T11:22:31.762687+0000","last_undegraded":"2024-03-23T11:22:31.762687+0000","last_fullsized":"2024-03-23T11:22:31.762687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:53.586433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778197+0000","last_change":"2024-03-23T11:22:31.774994+0000","last_active":"2024-03-23T11:22:31.778197+0000","last_peered":"2024-03-23T11:22:31.778197+0000","last_clean":"2024-03-23T11:22:31.778197+0000","last_became_active":"2024-03-23T11:22:31.772573+0000","last_became_peered":"2024-03-23T11:22:31.772573+0000","last_unstale":"2024-03-23T11:22:31.778197+0000","last_undegraded":"2024-03-23T11:22:31.778197+0000","last_fullsized":"2024-03-23T11:22:31.778197+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:14.173559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745441+0000","last_change":"2024-03-23T11:22:31.755440+0000","last_active":"2024-03-23T11:22:32.745441+0000","last_peered":"2024-03-23T11:22:32.745441+0000","last_clean":"2024-03-23T11:22:32.745441+0000","last_became_active":"2024-03-23T11:22:31.752956+0000","last_became_peered":"2024-03-23T11:22:31.752956+0000","last_unstale":"2024-03-23T11:22:32.745441+0000","last_undegraded":"2024-03-23T11:22:32.745441+0000","last_fullsized":"2024-03-23T11:22:32.745441+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:48:45.151345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938921+0000","last_change":"2024-03-23T11:22:31.763702+0000","last_active":"2024-03-23T11:22:32.938921+0000","last_peered":"2024-03-23T11:22:32.938921+0000","last_clean":"2024-03-23T11:22:32.938921+0000","last_became_active":"2024-03-23T11:22:31.763445+0000","last_became_peered":"2024-03-23T11:22:31.763445+0000","last_unstale":"2024-03-23T11:22:32.938921+0000","last_undegraded":"2024-03-23T11:22:32.938921+0000","last_fullsized":"2024-03-23T11:22:32.938921+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:06:06.972232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748561+0000","last_change":"2024-03-23T11:22:31.751995+0000","last_active":"2024-03-23T11:22:32.748561+0000","last_peered":"2024-03-23T11:22:32.748561+0000","last_clean":"2024-03-23T11:22:32.748561+0000","last_became_active":"2024-03-23T11:22:31.751709+0000","last_became_peered":"2024-03-23T11:22:31.751709+0000","last_unstale":"2024-03-23T11:22:32.748561+0000","last_undegraded":"2024-03-23T11:22:32.748561+0000","last_fullsized":"2024-03-23T11:22:32.748561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:10:08.399314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.745338+0000","last_change":"2024-03-23T11:22:31.754634+0000","last_active":"2024-03-23T11:22:32.745338+0000","last_peered":"2024-03-23T11:22:32.745338+0000","last_clean":"2024-03-23T11:22:32.745338+0000","last_became_active":"2024-03-23T11:22:31.753393+0000","last_became_peered":"2024-03-23T11:22:31.753393+0000","last_unstale":"2024-03-23T11:22:32.745338+0000","last_undegraded":"2024-03-23T11:22:32.745338+0000","last_fullsized":"2024-03-23T11:22:32.745338+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:02:24.548514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778260+0000","last_change":"2024-03-23T11:22:31.775630+0000","last_active":"2024-03-23T11:22:31.778260+0000","last_peered":"2024-03-23T11:22:31.778260+0000","last_clean":"2024-03-23T11:22:31.778260+0000","last_became_active":"2024-03-23T11:22:31.771294+0000","last_became_peered":"2024-03-23T11:22:31.771294+0000","last_unstale":"2024-03-23T11:22:31.778260+0000","last_undegraded":"2024-03-23T11:22:31.778260+0000","last_fullsized":"2024-03-23T11:22:31.778260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:15:02.678591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778086+0000","last_change":"2024-03-23T11:22:31.774039+0000","last_active":"2024-03-23T11:22:31.778086+0000","last_peered":"2024-03-23T11:22:31.778086+0000","last_clean":"2024-03-23T11:22:31.778086+0000","last_became_active":"2024-03-23T11:22:31.773700+0000","last_became_peered":"2024-03-23T11:22:31.773700+0000","last_unstale":"2024-03-23T11:22:31.778086+0000","last_undegraded":"2024-03-23T11:22:31.778086+0000","last_fullsized":"2024-03-23T11:22:31.778086+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:09:01.658097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"61'4","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778343+0000","last_change":"2024-03-23T11:22:31.262916+0000","last_active":"2024-03-23T11:22:31.778343+0000","last_peered":"2024-03-23T11:22:31.778343+0000","last_clean":"2024-03-23T11:22:31.778343+0000","last_became_active":"2024-03-23T11:22:31.262651+0000","last_became_peered":"2024-03-23T11:22:31.262651+0000","last_unstale":"2024-03-23T11:22:31.778343+0000","last_undegraded":"2024-03-23T11:22:31.778343+0000","last_fullsized":"2024-03-23T11:22:31.778343+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:59:16.844989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768771+0000","last_change":"2024-03-23T11:22:31.765924+0000","last_active":"2024-03-23T11:22:31.768771+0000","last_peered":"2024-03-23T11:22:31.768771+0000","last_clean":"2024-03-23T11:22:31.768771+0000","last_became_active":"2024-03-23T11:22:31.765600+0000","last_became_peered":"2024-03-23T11:22:31.765600+0000","last_unstale":"2024-03-23T11:22:31.768771+0000","last_undegraded":"2024-03-23T11:22:31.768771+0000","last_fullsized":"2024-03-23T11:22:31.768771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:01:05.157583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"61'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939451+0000","last_change":"2024-03-23T11:22:31.264605+0000","last_active":"2024-03-23T11:22:32.939451+0000","last_peered":"2024-03-23T11:22:32.939451+0000","last_clean":"2024-03-23T11:22:32.939451+0000","last_became_active":"2024-03-23T11:22:31.264341+0000","last_became_peered":"2024-03-23T11:22:31.264341+0000","last_unstale":"2024-03-23T11:22:32.939451+0000","last_undegraded":"2024-03-23T11:22:32.939451+0000","last_fullsized":"2024-03-23T11:22:32.939451+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:35:51.264384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778045+0000","last_change":"2024-03-23T11:22:31.773721+0000","last_active":"2024-03-23T11:22:31.778045+0000","last_peered":"2024-03-23T11:22:31.778045+0000","last_clean":"2024-03-23T11:22:31.778045+0000","last_became_active":"2024-03-23T11:22:31.773443+0000","last_became_peered":"2024-03-23T11:22:31.773443+0000","last_unstale":"2024-03-23T11:22:31.778045+0000","last_undegraded":"2024-03-23T11:22:31.778045+0000","last_fullsized":"2024-03-23T11:22:31.778045+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:27:14.723335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"61'2","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.775185+0000","last_change":"2024-03-23T11:22:31.262398+0000","last_active":"2024-03-23T11:22:31.775185+0000","last_peered":"2024-03-23T11:22:31.775185+0000","last_clean":"2024-03-23T11:22:31.775185+0000","last_became_active":"2024-03-23T11:22:31.258009+0000","last_became_peered":"2024-03-23T11:22:31.258009+0000","last_unstale":"2024-03-23T11:22:31.775185+0000","last_undegraded":"2024-03-23T11:22:31.775185+0000","last_fullsized":"2024-03-23T11:22:31.775185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:32:47.093788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778022+0000","last_change":"2024-03-23T11:22:31.773674+0000","last_active":"2024-03-23T11:22:31.778022+0000","last_peered":"2024-03-23T11:22:31.778022+0000","last_clean":"2024-03-23T11:22:31.778022+0000","last_became_active":"2024-03-23T11:22:31.768814+0000","last_became_peered":"2024-03-23T11:22:31.768814+0000","last_unstale":"2024-03-23T11:22:31.778022+0000","last_undegraded":"2024-03-23T11:22:31.778022+0000","last_fullsized":"2024-03-23T11:22:31.778022+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:34:23.644768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"61'2","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.762263+0000","last_change":"2024-03-23T11:22:31.264826+0000","last_active":"2024-03-23T11:22:31.762263+0000","last_peered":"2024-03-23T11:22:31.762263+0000","last_clean":"2024-03-23T11:22:31.762263+0000","last_became_active":"2024-03-23T11:22:31.264459+0000","last_became_peered":"2024-03-23T11:22:31.264459+0000","last_unstale":"2024-03-23T11:22:31.762263+0000","last_undegraded":"2024-03-23T11:22:31.762263+0000","last_fullsized":"2024-03-23T11:22:31.762263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:35:51.264384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938644+0000","last_change":"2024-03-23T11:22:31.750384+0000","last_active":"2024-03-23T11:22:32.938644+0000","last_peered":"2024-03-23T11:22:32.938644+0000","last_clean":"2024-03-23T11:22:32.938644+0000","last_became_active":"2024-03-23T11:22:31.749914+0000","last_became_peered":"2024-03-23T11:22:31.749914+0000","last_unstale":"2024-03-23T11:22:32.938644+0000","last_undegraded":"2024-03-23T11:22:32.938644+0000","last_fullsized":"2024-03-23T11:22:32.938644+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:20:37.373989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"61'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746839+0000","last_change":"2024-03-23T11:22:31.263340+0000","last_active":"2024-03-23T11:22:32.746839+0000","last_peered":"2024-03-23T11:22:32.746839+0000","last_clean":"2024-03-23T11:22:32.746839+0000","last_became_active":"2024-03-23T11:22:31.263066+0000","last_became_peered":"2024-03-23T11:22:31.263066+0000","last_unstale":"2024-03-23T11:22:32.746839+0000","last_undegraded":"2024-03-23T11:22:32.746839+0000","last_fullsized":"2024-03-23T11:22:32.746839+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:57.593149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938836+0000","last_change":"2024-03-23T11:22:31.770892+0000","last_active":"2024-03-23T11:22:32.938836+0000","last_peered":"2024-03-23T11:22:32.938836+0000","last_clean":"2024-03-23T11:22:32.938836+0000","last_became_active":"2024-03-23T11:22:31.770567+0000","last_became_peered":"2024-03-23T11:22:31.770567+0000","last_unstale":"2024-03-23T11:22:32.938836+0000","last_undegraded":"2024-03-23T11:22:32.938836+0000","last_fullsized":"2024-03-23T11:22:32.938836+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:19.482944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"61'1","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.777775+0000","last_change":"2024-03-23T11:22:31.264847+0000","last_active":"2024-03-23T11:22:31.777775+0000","last_peered":"2024-03-23T11:22:31.777775+0000","last_clean":"2024-03-23T11:22:31.777775+0000","last_became_active":"2024-03-23T11:22:31.263443+0000","last_became_peered":"2024-03-23T11:22:31.263443+0000","last_unstale":"2024-03-23T11:22:31.777775+0000","last_undegraded":"2024-03-23T11:22:31.777775+0000","last_fullsized":"2024-03-23T11:22:31.777775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:44:59.112031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.761564+0000","last_change":"2024-03-23T11:22:31.759332+0000","last_active":"2024-03-23T11:22:31.761564+0000","last_peered":"2024-03-23T11:22:31.761564+0000","last_clean":"2024-03-23T11:22:31.761564+0000","last_became_active":"2024-03-23T11:22:31.759078+0000","last_became_peered":"2024-03-23T11:22:31.759078+0000","last_unstale":"2024-03-23T11:22:31.761564+0000","last_undegraded":"2024-03-23T11:22:31.761564+0000","last_fullsized":"2024-03-23T11:22:31.761564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:44:48.626587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.747194+0000","last_change":"2024-03-23T11:22:31.260619+0000","last_active":"2024-03-23T11:22:31.747194+0000","last_peered":"2024-03-23T11:22:31.747194+0000","last_clean":"2024-03-23T11:22:31.747194+0000","last_became_active":"2024-03-23T11:22:31.260211+0000","last_became_peered":"2024-03-23T11:22:31.260211+0000","last_unstale":"2024-03-23T11:22:31.747194+0000","last_undegraded":"2024-03-23T11:22:31.747194+0000","last_fullsized":"2024-03-23T11:22:31.747194+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:21:06.454365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938363+0000","last_change":"2024-03-23T11:22:31.770238+0000","last_active":"2024-03-23T11:22:32.938363+0000","last_peered":"2024-03-23T11:22:32.938363+0000","last_clean":"2024-03-23T11:22:32.938363+0000","last_became_active":"2024-03-23T11:22:31.766595+0000","last_became_peered":"2024-03-23T11:22:31.766595+0000","last_unstale":"2024-03-23T11:22:32.938363+0000","last_undegraded":"2024-03-23T11:22:32.938363+0000","last_fullsized":"2024-03-23T11:22:32.938363+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:32:36.695167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746846+0000","last_change":"2024-03-23T11:22:31.257165+0000","last_active":"2024-03-23T11:22:32.746846+0000","last_peered":"2024-03-23T11:22:32.746846+0000","last_clean":"2024-03-23T11:22:32.746846+0000","last_became_active":"2024-03-23T11:22:31.256771+0000","last_became_peered":"2024-03-23T11:22:31.256771+0000","last_unstale":"2024-03-23T11:22:32.746846+0000","last_undegraded":"2024-03-23T11:22:32.746846+0000","last_fullsized":"2024-03-23T11:22:32.746846+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:05:40.831430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746657+0000","last_change":"2024-03-23T11:22:31.754431+0000","last_active":"2024-03-23T11:22:32.746657+0000","last_peered":"2024-03-23T11:22:32.746657+0000","last_clean":"2024-03-23T11:22:32.746657+0000","last_became_active":"2024-03-23T11:22:31.753838+0000","last_became_peered":"2024-03-23T11:22:31.753838+0000","last_unstale":"2024-03-23T11:22:32.746657+0000","last_undegraded":"2024-03-23T11:22:32.746657+0000","last_fullsized":"2024-03-23T11:22:32.746657+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:29:43.839637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749315+0000","last_change":"2024-03-23T11:22:31.266527+0000","last_active":"2024-03-23T11:22:32.749315+0000","last_peered":"2024-03-23T11:22:32.749315+0000","last_clean":"2024-03-23T11:22:32.749315+0000","last_became_active":"2024-03-23T11:22:31.265979+0000","last_became_peered":"2024-03-23T11:22:31.265979+0000","last_unstale":"2024-03-23T11:22:32.749315+0000","last_undegraded":"2024-03-23T11:22:32.749315+0000","last_fullsized":"2024-03-23T11:22:32.749315+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:34:07.566641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749164+0000","last_change":"2024-03-23T11:22:31.764829+0000","last_active":"2024-03-23T11:22:32.749164+0000","last_peered":"2024-03-23T11:22:32.749164+0000","last_clean":"2024-03-23T11:22:32.749164+0000","last_became_active":"2024-03-23T11:22:31.764551+0000","last_became_peered":"2024-03-23T11:22:31.764551+0000","last_unstale":"2024-03-23T11:22:32.749164+0000","last_undegraded":"2024-03-23T11:22:32.749164+0000","last_fullsized":"2024-03-23T11:22:32.749164+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:00.036989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939444+0000","last_change":"2024-03-23T11:22:31.269216+0000","last_active":"2024-03-23T11:22:32.939444+0000","last_peered":"2024-03-23T11:22:32.939444+0000","last_clean":"2024-03-23T11:22:32.939444+0000","last_became_active":"2024-03-23T11:22:31.268653+0000","last_became_peered":"2024-03-23T11:22:31.268653+0000","last_unstale":"2024-03-23T11:22:32.939444+0000","last_undegraded":"2024-03-23T11:22:32.939444+0000","last_fullsized":"2024-03-23T11:22:32.939444+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:26.604228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768732+0000","last_change":"2024-03-23T11:22:31.766577+0000","last_active":"2024-03-23T11:22:31.768732+0000","last_peered":"2024-03-23T11:22:31.768732+0000","last_clean":"2024-03-23T11:22:31.768732+0000","last_became_active":"2024-03-23T11:22:31.762373+0000","last_became_peered":"2024-03-23T11:22:31.762373+0000","last_unstale":"2024-03-23T11:22:31.768732+0000","last_undegraded":"2024-03-23T11:22:31.768732+0000","last_fullsized":"2024-03-23T11:22:31.768732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:17:54.390500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"61'3","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749017+0000","last_change":"2024-03-23T11:22:31.266377+0000","last_active":"2024-03-23T11:22:32.749017+0000","last_peered":"2024-03-23T11:22:32.749017+0000","last_clean":"2024-03-23T11:22:32.749017+0000","last_became_active":"2024-03-23T11:22:31.264260+0000","last_became_peered":"2024-03-23T11:22:31.264260+0000","last_unstale":"2024-03-23T11:22:32.749017+0000","last_undegraded":"2024-03-23T11:22:32.749017+0000","last_fullsized":"2024-03-23T11:22:32.749017+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:56:04.434278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748911+0000","last_change":"2024-03-23T11:22:31.765120+0000","last_active":"2024-03-23T11:22:32.748911+0000","last_peered":"2024-03-23T11:22:32.748911+0000","last_clean":"2024-03-23T11:22:32.748911+0000","last_became_active":"2024-03-23T11:22:31.764698+0000","last_became_peered":"2024-03-23T11:22:31.764698+0000","last_unstale":"2024-03-23T11:22:32.748911+0000","last_undegraded":"2024-03-23T11:22:32.748911+0000","last_fullsized":"2024-03-23T11:22:32.748911+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:29:21.455065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"61'6","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.747133+0000","last_change":"2024-03-23T11:22:31.263750+0000","last_active":"2024-03-23T11:22:32.747133+0000","last_peered":"2024-03-23T11:22:32.747133+0000","last_clean":"2024-03-23T11:22:32.747133+0000","last_became_active":"2024-03-23T11:22:31.263456+0000","last_became_peered":"2024-03-23T11:22:31.263456+0000","last_unstale":"2024-03-23T11:22:32.747133+0000","last_undegraded":"2024-03-23T11:22:32.747133+0000","last_fullsized":"2024-03-23T11:22:32.747133+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:50:42.726444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.766708+0000","last_change":"2024-03-23T11:22:31.761908+0000","last_active":"2024-03-23T11:22:31.766708+0000","last_peered":"2024-03-23T11:22:31.766708+0000","last_clean":"2024-03-23T11:22:31.766708+0000","last_became_active":"2024-03-23T11:22:31.760305+0000","last_became_peered":"2024-03-23T11:22:31.760305+0000","last_unstale":"2024-03-23T11:22:31.766708+0000","last_undegraded":"2024-03-23T11:22:31.766708+0000","last_fullsized":"2024-03-23T11:22:31.766708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:00:51.026955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"61'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938812+0000","last_change":"2024-03-23T11:22:31.265890+0000","last_active":"2024-03-23T11:22:32.938812+0000","last_peered":"2024-03-23T11:22:32.938812+0000","last_clean":"2024-03-23T11:22:32.938812+0000","last_became_active":"2024-03-23T11:22:31.265618+0000","last_became_peered":"2024-03-23T11:22:31.265618+0000","last_unstale":"2024-03-23T11:22:32.938812+0000","last_undegraded":"2024-03-23T11:22:32.938812+0000","last_fullsized":"2024-03-23T11:22:32.938812+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:21:06.454365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748829+0000","last_change":"2024-03-23T11:22:31.753217+0000","last_active":"2024-03-23T11:22:32.748829+0000","last_peered":"2024-03-23T11:22:32.748829+0000","last_clean":"2024-03-23T11:22:32.748829+0000","last_became_active":"2024-03-23T11:22:31.752981+0000","last_became_peered":"2024-03-23T11:22:31.752981+0000","last_unstale":"2024-03-23T11:22:32.748829+0000","last_undegraded":"2024-03-23T11:22:32.748829+0000","last_fullsized":"2024-03-23T11:22:32.748829+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:55:49.707028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.746503+0000","last_change":"2024-03-23T11:22:31.267806+0000","last_active":"2024-03-23T11:22:31.746503+0000","last_peered":"2024-03-23T11:22:31.746503+0000","last_clean":"2024-03-23T11:22:31.746503+0000","last_became_active":"2024-03-23T11:22:31.267381+0000","last_became_peered":"2024-03-23T11:22:31.267381+0000","last_unstale":"2024-03-23T11:22:31.746503+0000","last_undegraded":"2024-03-23T11:22:31.746503+0000","last_fullsized":"2024-03-23T11:22:31.746503+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:41:33.525902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746455+0000","last_change":"2024-03-23T11:22:31.765499+0000","last_active":"2024-03-23T11:22:32.746455+0000","last_peered":"2024-03-23T11:22:32.746455+0000","last_clean":"2024-03-23T11:22:32.746455+0000","last_became_active":"2024-03-23T11:22:31.764902+0000","last_became_peered":"2024-03-23T11:22:31.764902+0000","last_unstale":"2024-03-23T11:22:32.746455+0000","last_undegraded":"2024-03-23T11:22:32.746455+0000","last_fullsized":"2024-03-23T11:22:32.746455+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:59:17.853936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.747302+0000","last_change":"2024-03-23T11:22:31.264671+0000","last_active":"2024-03-23T11:22:31.747302+0000","last_peered":"2024-03-23T11:22:31.747302+0000","last_clean":"2024-03-23T11:22:31.747302+0000","last_became_active":"2024-03-23T11:22:31.264346+0000","last_became_peered":"2024-03-23T11:22:31.264346+0000","last_unstale":"2024-03-23T11:22:31.747302+0000","last_undegraded":"2024-03-23T11:22:31.747302+0000","last_fullsized":"2024-03-23T11:22:31.747302+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:26.604228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748590+0000","last_change":"2024-03-23T11:22:31.761174+0000","last_active":"2024-03-23T11:22:32.748590+0000","last_peered":"2024-03-23T11:22:32.748590+0000","last_clean":"2024-03-23T11:22:32.748590+0000","last_became_active":"2024-03-23T11:22:31.754742+0000","last_became_peered":"2024-03-23T11:22:31.754742+0000","last_unstale":"2024-03-23T11:22:32.748590+0000","last_undegraded":"2024-03-23T11:22:32.748590+0000","last_fullsized":"2024-03-23T11:22:32.748590+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:10:07.723126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"61'8","reported_seq":19,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748242+0000","last_change":"2024-03-23T11:22:31.263806+0000","last_active":"2024-03-23T11:22:32.748242+0000","last_peered":"2024-03-23T11:22:32.748242+0000","last_clean":"2024-03-23T11:22:32.748242+0000","last_became_active":"2024-03-23T11:22:31.262960+0000","last_became_peered":"2024-03-23T11:22:31.262960+0000","last_unstale":"2024-03-23T11:22:32.748242+0000","last_undegraded":"2024-03-23T11:22:32.748242+0000","last_fullsized":"2024-03-23T11:22:32.748242+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:45:24.822418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749260+0000","last_change":"2024-03-23T11:22:31.758856+0000","last_active":"2024-03-23T11:22:32.749260+0000","last_peered":"2024-03-23T11:22:32.749260+0000","last_clean":"2024-03-23T11:22:32.749260+0000","last_became_active":"2024-03-23T11:22:31.758622+0000","last_became_peered":"2024-03-23T11:22:31.758622+0000","last_unstale":"2024-03-23T11:22:32.749260+0000","last_undegraded":"2024-03-23T11:22:32.749260+0000","last_fullsized":"2024-03-23T11:22:32.749260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:12:02.848172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"61'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939570+0000","last_change":"2024-03-23T11:22:31.265939+0000","last_active":"2024-03-23T11:22:32.939570+0000","last_peered":"2024-03-23T11:22:32.939570+0000","last_clean":"2024-03-23T11:22:32.939570+0000","last_became_active":"2024-03-23T11:22:31.263837+0000","last_became_peered":"2024-03-23T11:22:31.263837+0000","last_unstale":"2024-03-23T11:22:32.939570+0000","last_undegraded":"2024-03-23T11:22:32.939570+0000","last_fullsized":"2024-03-23T11:22:32.939570+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:53.321049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.769974+0000","last_change":"2024-03-23T11:22:31.766386+0000","last_active":"2024-03-23T11:22:31.769974+0000","last_peered":"2024-03-23T11:22:31.769974+0000","last_clean":"2024-03-23T11:22:31.769974+0000","last_became_active":"2024-03-23T11:22:31.766120+0000","last_became_peered":"2024-03-23T11:22:31.766120+0000","last_unstale":"2024-03-23T11:22:31.769974+0000","last_undegraded":"2024-03-23T11:22:31.769974+0000","last_fullsized":"2024-03-23T11:22:31.769974+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:22:35.016176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"61'2","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.773525+0000","last_change":"2024-03-23T11:22:31.268954+0000","last_active":"2024-03-23T11:22:31.773525+0000","last_peered":"2024-03-23T11:22:31.773525+0000","last_clean":"2024-03-23T11:22:31.773525+0000","last_became_active":"2024-03-23T11:22:31.268661+0000","last_became_peered":"2024-03-23T11:22:31.268661+0000","last_unstale":"2024-03-23T11:22:31.773525+0000","last_undegraded":"2024-03-23T11:22:31.773525+0000","last_fullsized":"2024-03-23T11:22:31.773525+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:50:47.230344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939350+0000","last_change":"2024-03-23T11:22:31.760846+0000","last_active":"2024-03-23T11:22:32.939350+0000","last_peered":"2024-03-23T11:22:32.939350+0000","last_clean":"2024-03-23T11:22:32.939350+0000","last_became_active":"2024-03-23T11:22:31.759124+0000","last_became_peered":"2024-03-23T11:22:31.759124+0000","last_unstale":"2024-03-23T11:22:32.939350+0000","last_undegraded":"2024-03-23T11:22:32.939350+0000","last_fullsized":"2024-03-23T11:22:32.939350+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:45:03.917246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"61'1","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.765614+0000","last_change":"2024-03-23T11:22:31.264775+0000","last_active":"2024-03-23T11:22:31.765614+0000","last_peered":"2024-03-23T11:22:31.765614+0000","last_clean":"2024-03-23T11:22:31.765614+0000","last_became_active":"2024-03-23T11:22:31.264424+0000","last_became_peered":"2024-03-23T11:22:31.264424+0000","last_unstale":"2024-03-23T11:22:31.765614+0000","last_undegraded":"2024-03-23T11:22:31.765614+0000","last_fullsized":"2024-03-23T11:22:31.765614+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:50:10.145832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.748878+0000","last_change":"2024-03-23T11:22:31.760738+0000","last_active":"2024-03-23T11:22:32.748878+0000","last_peered":"2024-03-23T11:22:32.748878+0000","last_clean":"2024-03-23T11:22:32.748878+0000","last_became_active":"2024-03-23T11:22:31.759962+0000","last_became_peered":"2024-03-23T11:22:31.759962+0000","last_unstale":"2024-03-23T11:22:32.748878+0000","last_undegraded":"2024-03-23T11:22:32.748878+0000","last_fullsized":"2024-03-23T11:22:32.748878+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:40:12.835110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.763470+0000","last_change":"2024-03-23T11:22:31.258135+0000","last_active":"2024-03-23T11:22:31.763470+0000","last_peered":"2024-03-23T11:22:31.763470+0000","last_clean":"2024-03-23T11:22:31.763470+0000","last_became_active":"2024-03-23T11:22:31.257592+0000","last_became_peered":"2024-03-23T11:22:31.257592+0000","last_unstale":"2024-03-23T11:22:31.763470+0000","last_undegraded":"2024-03-23T11:22:31.763470+0000","last_fullsized":"2024-03-23T11:22:31.763470+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:54:10.046864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.762654+0000","last_change":"2024-03-23T11:22:31.758644+0000","last_active":"2024-03-23T11:22:31.762654+0000","last_peered":"2024-03-23T11:22:31.762654+0000","last_clean":"2024-03-23T11:22:31.762654+0000","last_became_active":"2024-03-23T11:22:31.755074+0000","last_became_peered":"2024-03-23T11:22:31.755074+0000","last_unstale":"2024-03-23T11:22:31.762654+0000","last_undegraded":"2024-03-23T11:22:31.762654+0000","last_fullsized":"2024-03-23T11:22:31.762654+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:14.173559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"61'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.939101+0000","last_change":"2024-03-23T11:22:31.259688+0000","last_active":"2024-03-23T11:22:32.939101+0000","last_peered":"2024-03-23T11:22:32.939101+0000","last_clean":"2024-03-23T11:22:32.939101+0000","last_became_active":"2024-03-23T11:22:31.259460+0000","last_became_peered":"2024-03-23T11:22:31.259460+0000","last_unstale":"2024-03-23T11:22:32.939101+0000","last_undegraded":"2024-03-23T11:22:32.939101+0000","last_fullsized":"2024-03-23T11:22:32.939101+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:32.530250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.773250+0000","last_change":"2024-03-23T11:22:31.765060+0000","last_active":"2024-03-23T11:22:31.773250+0000","last_peered":"2024-03-23T11:22:31.773250+0000","last_clean":"2024-03-23T11:22:31.773250+0000","last_became_active":"2024-03-23T11:22:31.764769+0000","last_became_peered":"2024-03-23T11:22:31.764769+0000","last_unstale":"2024-03-23T11:22:31.773250+0000","last_undegraded":"2024-03-23T11:22:31.773250+0000","last_fullsized":"2024-03-23T11:22:31.773250+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:12:08.025880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.745835+0000","last_change":"2024-03-23T11:22:31.259575+0000","last_active":"2024-03-23T11:22:31.745835+0000","last_peered":"2024-03-23T11:22:31.745835+0000","last_clean":"2024-03-23T11:22:31.745835+0000","last_became_active":"2024-03-23T11:22:31.259282+0000","last_became_peered":"2024-03-23T11:22:31.259282+0000","last_unstale":"2024-03-23T11:22:31.745835+0000","last_undegraded":"2024-03-23T11:22:31.745835+0000","last_fullsized":"2024-03-23T11:22:31.745835+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:01.516022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.777773+0000","last_change":"2024-03-23T11:22:31.773034+0000","last_active":"2024-03-23T11:22:31.777773+0000","last_peered":"2024-03-23T11:22:31.777773+0000","last_clean":"2024-03-23T11:22:31.777773+0000","last_became_active":"2024-03-23T11:22:31.769007+0000","last_became_peered":"2024-03-23T11:22:31.769007+0000","last_unstale":"2024-03-23T11:22:31.777773+0000","last_undegraded":"2024-03-23T11:22:31.777773+0000","last_fullsized":"2024-03-23T11:22:31.777773+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:57.405432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.747859+0000","last_change":"2024-03-23T11:22:31.260154+0000","last_active":"2024-03-23T11:22:31.747859+0000","last_peered":"2024-03-23T11:22:31.747859+0000","last_clean":"2024-03-23T11:22:31.747859+0000","last_became_active":"2024-03-23T11:22:31.259709+0000","last_became_peered":"2024-03-23T11:22:31.259709+0000","last_unstale":"2024-03-23T11:22:31.747859+0000","last_undegraded":"2024-03-23T11:22:31.747859+0000","last_fullsized":"2024-03-23T11:22:31.747859+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:53.321049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.762708+0000","last_change":"2024-03-23T11:22:31.756415+0000","last_active":"2024-03-23T11:22:31.762708+0000","last_peered":"2024-03-23T11:22:31.762708+0000","last_clean":"2024-03-23T11:22:31.762708+0000","last_became_active":"2024-03-23T11:22:31.756127+0000","last_became_peered":"2024-03-23T11:22:31.756127+0000","last_unstale":"2024-03-23T11:22:31.762708+0000","last_undegraded":"2024-03-23T11:22:31.762708+0000","last_fullsized":"2024-03-23T11:22:31.762708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:15:02.678591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"61'2","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.767822+0000","last_change":"2024-03-23T11:22:31.254999+0000","last_active":"2024-03-23T11:22:31.767822+0000","last_peered":"2024-03-23T11:22:31.767822+0000","last_clean":"2024-03-23T11:22:31.767822+0000","last_became_active":"2024-03-23T11:22:31.254666+0000","last_became_peered":"2024-03-23T11:22:31.254666+0000","last_unstale":"2024-03-23T11:22:31.767822+0000","last_undegraded":"2024-03-23T11:22:31.767822+0000","last_fullsized":"2024-03-23T11:22:31.767822+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:09:50.262931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.766714+0000","last_change":"2024-03-23T11:22:31.763592+0000","last_active":"2024-03-23T11:22:31.766714+0000","last_peered":"2024-03-23T11:22:31.766714+0000","last_clean":"2024-03-23T11:22:31.766714+0000","last_became_active":"2024-03-23T11:22:31.763322+0000","last_became_peered":"2024-03-23T11:22:31.763322+0000","last_unstale":"2024-03-23T11:22:31.766714+0000","last_undegraded":"2024-03-23T11:22:31.766714+0000","last_fullsized":"2024-03-23T11:22:31.766714+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:12:08.025880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.761977+0000","last_change":"2024-03-23T11:22:31.265429+0000","last_active":"2024-03-23T11:22:31.761977+0000","last_peered":"2024-03-23T11:22:31.761977+0000","last_clean":"2024-03-23T11:22:31.761977+0000","last_became_active":"2024-03-23T11:22:31.264989+0000","last_became_peered":"2024-03-23T11:22:31.264989+0000","last_unstale":"2024-03-23T11:22:31.761977+0000","last_undegraded":"2024-03-23T11:22:31.761977+0000","last_fullsized":"2024-03-23T11:22:31.761977+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:41:46.750819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768711+0000","last_change":"2024-03-23T11:22:31.766413+0000","last_active":"2024-03-23T11:22:31.768711+0000","last_peered":"2024-03-23T11:22:31.768711+0000","last_clean":"2024-03-23T11:22:31.768711+0000","last_became_active":"2024-03-23T11:22:31.764791+0000","last_became_peered":"2024-03-23T11:22:31.764791+0000","last_unstale":"2024-03-23T11:22:31.768711+0000","last_undegraded":"2024-03-23T11:22:31.768711+0000","last_fullsized":"2024-03-23T11:22:31.768711+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:02.527716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.746622+0000","last_change":"2024-03-23T11:22:31.259321+0000","last_active":"2024-03-23T11:22:31.746622+0000","last_peered":"2024-03-23T11:22:31.746622+0000","last_clean":"2024-03-23T11:22:31.746622+0000","last_became_active":"2024-03-23T11:22:31.258609+0000","last_became_peered":"2024-03-23T11:22:31.258609+0000","last_unstale":"2024-03-23T11:22:31.746622+0000","last_undegraded":"2024-03-23T11:22:31.746622+0000","last_fullsized":"2024-03-23T11:22:31.746622+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:50:42.726444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.778077+0000","last_change":"2024-03-23T11:22:31.773897+0000","last_active":"2024-03-23T11:22:31.778077+0000","last_peered":"2024-03-23T11:22:31.778077+0000","last_clean":"2024-03-23T11:22:31.778077+0000","last_became_active":"2024-03-23T11:22:31.767497+0000","last_became_peered":"2024-03-23T11:22:31.767497+0000","last_unstale":"2024-03-23T11:22:31.778077+0000","last_undegraded":"2024-03-23T11:22:31.778077+0000","last_fullsized":"2024-03-23T11:22:31.778077+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:09:54.090512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.746374+0000","last_change":"2024-03-23T11:22:31.260220+0000","last_active":"2024-03-23T11:22:31.746374+0000","last_peered":"2024-03-23T11:22:31.746374+0000","last_clean":"2024-03-23T11:22:31.746374+0000","last_became_active":"2024-03-23T11:22:31.259912+0000","last_became_peered":"2024-03-23T11:22:31.259912+0000","last_unstale":"2024-03-23T11:22:31.746374+0000","last_undegraded":"2024-03-23T11:22:31.746374+0000","last_fullsized":"2024-03-23T11:22:31.746374+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:02.515842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938953+0000","last_change":"2024-03-23T11:22:31.753748+0000","last_active":"2024-03-23T11:22:32.938953+0000","last_peered":"2024-03-23T11:22:32.938953+0000","last_clean":"2024-03-23T11:22:32.938953+0000","last_became_active":"2024-03-23T11:22:31.753494+0000","last_became_peered":"2024-03-23T11:22:31.753494+0000","last_unstale":"2024-03-23T11:22:32.938953+0000","last_undegraded":"2024-03-23T11:22:32.938953+0000","last_fullsized":"2024-03-23T11:22:32.938953+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:52:20.354587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.749975+0000","last_change":"2024-03-23T11:22:31.262108+0000","last_active":"2024-03-23T11:22:31.749975+0000","last_peered":"2024-03-23T11:22:31.749975+0000","last_clean":"2024-03-23T11:22:31.749975+0000","last_became_active":"2024-03-23T11:22:31.261843+0000","last_became_peered":"2024-03-23T11:22:31.261843+0000","last_unstale":"2024-03-23T11:22:31.749975+0000","last_undegraded":"2024-03-23T11:22:31.749975+0000","last_fullsized":"2024-03-23T11:22:31.749975+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:13:41.293233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.749032+0000","last_change":"2024-03-23T11:22:31.753894+0000","last_active":"2024-03-23T11:22:32.749032+0000","last_peered":"2024-03-23T11:22:32.749032+0000","last_clean":"2024-03-23T11:22:32.749032+0000","last_became_active":"2024-03-23T11:22:31.753553+0000","last_became_peered":"2024-03-23T11:22:31.753553+0000","last_unstale":"2024-03-23T11:22:32.749032+0000","last_undegraded":"2024-03-23T11:22:32.749032+0000","last_fullsized":"2024-03-23T11:22:32.749032+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:23:26.437000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.746657+0000","last_change":"2024-03-23T11:22:31.265577+0000","last_active":"2024-03-23T11:22:32.746657+0000","last_peered":"2024-03-23T11:22:32.746657+0000","last_clean":"2024-03-23T11:22:32.746657+0000","last_became_active":"2024-03-23T11:22:31.265317+0000","last_became_peered":"2024-03-23T11:22:31.265317+0000","last_unstale":"2024-03-23T11:22:32.746657+0000","last_undegraded":"2024-03-23T11:22:32.746657+0000","last_fullsized":"2024-03-23T11:22:32.746657+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:41:33.525902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.768238+0000","last_change":"2024-03-23T11:22:31.759129+0000","last_active":"2024-03-23T11:22:31.768238+0000","last_peered":"2024-03-23T11:22:31.768238+0000","last_clean":"2024-03-23T11:22:31.768238+0000","last_became_active":"2024-03-23T11:22:31.758887+0000","last_became_peered":"2024-03-23T11:22:31.758887+0000","last_unstale":"2024-03-23T11:22:31.768238+0000","last_undegraded":"2024-03-23T11:22:31.768238+0000","last_fullsized":"2024-03-23T11:22:31.768238+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:46:29.980472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.746919+0000","last_change":"2024-03-23T11:22:31.265637+0000","last_active":"2024-03-23T11:22:31.746919+0000","last_peered":"2024-03-23T11:22:31.746919+0000","last_clean":"2024-03-23T11:22:31.746919+0000","last_became_active":"2024-03-23T11:22:31.265323+0000","last_became_peered":"2024-03-23T11:22:31.265323+0000","last_unstale":"2024-03-23T11:22:31.746919+0000","last_undegraded":"2024-03-23T11:22:31.746919+0000","last_fullsized":"2024-03-23T11:22:31.746919+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:52:54.898960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938475+0000","last_change":"2024-03-23T11:22:31.769993+0000","last_active":"2024-03-23T11:22:32.938475+0000","last_peered":"2024-03-23T11:22:32.938475+0000","last_clean":"2024-03-23T11:22:32.938475+0000","last_became_active":"2024-03-23T11:22:31.766568+0000","last_became_peered":"2024-03-23T11:22:31.766568+0000","last_unstale":"2024-03-23T11:22:32.938475+0000","last_undegraded":"2024-03-23T11:22:32.938475+0000","last_fullsized":"2024-03-23T11:22:32.938475+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:23:56.372856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938712+0000","last_change":"2024-03-23T11:22:31.268143+0000","last_active":"2024-03-23T11:22:32.938712+0000","last_peered":"2024-03-23T11:22:32.938712+0000","last_clean":"2024-03-23T11:22:32.938712+0000","last_became_active":"2024-03-23T11:22:31.267125+0000","last_became_peered":"2024-03-23T11:22:31.267125+0000","last_unstale":"2024-03-23T11:22:32.938712+0000","last_undegraded":"2024-03-23T11:22:32.938712+0000","last_fullsized":"2024-03-23T11:22:32.938712+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:02.515842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-23T11:22:32.938475+0000","last_change":"2024-03-23T11:22:31.777352+0000","last_active":"2024-03-23T11:22:32.938475+0000","last_peered":"2024-03-23T11:22:32.938475+0000","last_clean":"2024-03-23T11:22:32.938475+0000","last_became_active":"2024-03-23T11:22:31.776865+0000","last_became_peered":"2024-03-23T11:22:31.776865+0000","last_unstale":"2024-03-23T11:22:32.938475+0000","last_undegraded":"2024-03-23T11:22:32.938475+0000","last_fullsized":"2024-03-23T11:22:32.938475+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:31.219908+0000","last_clean_scrub_stamp":"2024-03-23T11:22:31.219908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:30.728849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.746040+0000","last_change":"2024-03-23T11:22:31.261261+0000","last_active":"2024-03-23T11:22:31.746040+0000","last_peered":"2024-03-23T11:22:31.746040+0000","last_clean":"2024-03-23T11:22:31.746040+0000","last_became_active":"2024-03-23T11:22:31.260936+0000","last_became_peered":"2024-03-23T11:22:31.260936+0000","last_unstale":"2024-03-23T11:22:31.746040+0000","last_undegraded":"2024-03-23T11:22:31.746040+0000","last_fullsized":"2024-03-23T11:22:31.746040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:22:30.210962+0000","last_clean_scrub_stamp":"2024-03-23T11:22:30.210962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:48:57.593149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":205,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-23T11:22:31.737206+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:22:31.737206+0000","last_peered":"2024-03-23T11:22:31.737206+0000","last_clean":"2024-03-23T11:22:31.737206+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:22:31.737206+0000","last_undegraded":"2024-03-23T11:22:31.737206+0000","last_fullsized":"2024-03-23T11:22:31.737206+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542192,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1331200,"data_stored":893171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.93500000000000005,"15min":0.90300000000000002},"min":{"1min":0.68400000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5309999999999999,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1,"15min":0.97999999999999998},"min":{"1min":0.67300000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.86499999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.1819999999999999,"15min":1.1799999999999999},"min":{"1min":0.75600000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.7430000000000001,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":1.494},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.234,"15min":1.246},"min":{"1min":0.82999999999999996,"5min":0.753,"15min":0.753},"max":{"1min":1.6870000000000001,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.091}]},{"osd":2,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.052,"15min":1.0509999999999999},"min":{"1min":0.78000000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.262,"5min":1.1539999999999999,"15min":1.1319999999999999},"min":{"1min":0.90200000000000002,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.4079999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.153,"15min":1.127},"min":{"1min":1.0509999999999999,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.86199999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.6479999999999999,"5min":1.778,"15min":1.778},"last":1.3240000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.208,"15min":1.212},"min":{"1min":0.90200000000000002,"5min":0.627,"15min":0.627},"max":{"1min":1.681,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.383,"15min":1.357},"min":{"1min":1.2390000000000001,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.323,"15min":1.3320000000000001},"min":{"1min":0.82999999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.7470000000000001,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.244},{"interface":"front","average":{"1min":1.45,"5min":1.357,"15min":1.347},"min":{"1min":1.1200000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.8839999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.169}]},{"osd":6,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.4019999999999999,"15min":1.4139999999999999},"min":{"1min":1.0409999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.9139999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.3620000000000001,"15min":1.339},"min":{"1min":1.01,"5min":0.747,"15min":0.747},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.0349999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542192,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":301974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.107,"15min":1.073},"min":{"1min":0.69999999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.8169999999999999,"5min":3.9140000000000001,"15min":3.9140000000000001},"last":1.097},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1459999999999999,"15min":1.1140000000000001},"min":{"1min":0.746,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.48,"5min":6.6539999999999999,"15min":6.6539999999999999},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.21,"15min":1.22},"min":{"1min":0.78000000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6819999999999999,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.2509999999999999,"15min":1.2729999999999999},"min":{"1min":0.83299999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.575,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.032}]},{"osd":2,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.254,"15min":1.252},"min":{"1min":0.92000000000000004,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.5369999999999999,"5min":3.73,"15min":3.73},"last":1.127},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.272,"15min":1.296},"min":{"1min":0.81999999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.546,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.355}]},{"osd":3,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.335,"15min":1.262},"min":{"1min":0.91900000000000004,"5min":0.749,"15min":0.749},"max":{"1min":1.782,"5min":12.914,"15min":12.914},"last":1.161},{"interface":"front","average":{"1min":1.262,"5min":1.361,"15min":1.383},"min":{"1min":0.89300000000000002,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.7350000000000001,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":1.2190000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.371,"15min":1.4139999999999999},"min":{"1min":0.89700000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.2200000000000002,"5min":2.702,"15min":2.702},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.4339999999999999,"15min":1.4379999999999999},"min":{"1min":0.86599999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.6919999999999999,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":1.5609999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.528,"15min":1.5720000000000001},"min":{"1min":1.0149999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.794,"5min":3.097,"15min":3.097},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.4830000000000001,"15min":1.5149999999999999},"min":{"1min":0.97399999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.282,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.593}]},{"osd":7,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.6759999999999999,"15min":1.708},"min":{"1min":1.2689999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.8720000000000001,"5min":4.9470000000000001,"15min":4.9470000000000001},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.5129999999999999,"15min":1.5329999999999999},"min":{"1min":1.054,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.806,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":1.478}]}]},{"osd":5,"up_from":14,"seq":60129542192,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":737280,"data_stored":303295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1619999999999999,"15min":1.2130000000000001},"min":{"1min":0.72099999999999997,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.9350000000000001,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.127,"5min":1.1499999999999999,"15min":1.157},"min":{"1min":0.65600000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.5229999999999999,"5min":2.355,"15min":2.355},"last":0.77100000000000002}]},{"osd":1,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1890000000000001,"15min":1.2010000000000001},"min":{"1min":0.78600000000000003,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.6020000000000001,"5min":2.2559999999999998,"15min":2.2559999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.1950000000000001,"15min":1.228},"min":{"1min":0.69799999999999995,"5min":0.495,"15min":0.495},"max":{"1min":1.6299999999999999,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":0.82199999999999995}]},{"osd":2,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2749999999999999,"15min":1.2729999999999999},"min":{"1min":1.006,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.7390000000000001,"5min":2.012,"15min":2.012},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.226,"15min":1.24},"min":{"1min":0.77000000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.6299999999999999,"5min":1.946,"15min":1.946},"last":0.92300000000000004}]},{"osd":3,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.304,"15min":1.3140000000000001},"min":{"1min":0.998,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.831,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.879},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.391,"15min":1.421},"min":{"1min":0.92300000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.8939999999999999,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.0129999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.405,"15min":1.4410000000000001},"min":{"1min":0.84099999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.7470000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.54,"5min":1.532,"15min":1.5620000000000001},"min":{"1min":1.323,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.048,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.1899999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.456,"15min":1.504},"min":{"1min":0.94699999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.7370000000000001,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.105},{"interface":"front","average":{"1min":1.391,"5min":1.5089999999999999,"15min":1.5649999999999999},"min":{"1min":1.012,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.02,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.2669999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.54,"15min":1.587},"min":{"1min":1.0780000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.9419999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.6060000000000001,"15min":1.6399999999999999},"min":{"1min":1.252,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.7949999999999999,"5min":2.516,"15min":2.516},"last":1.2310000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542192,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":741376,"data_stored":304207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.111,"15min":1.099},"min":{"1min":0.79000000000000004,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.252,"5min":1.2050000000000001,"15min":1.2010000000000001},"min":{"1min":0.77100000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.5589999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.3009999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.091,"15min":1.097},"min":{"1min":0.80800000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":1.069},{"interface":"front","average":{"1min":1.179,"5min":1.1140000000000001,"15min":1.099},"min":{"1min":0.875,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.4930000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.4339999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2070000000000001,"15min":1.2270000000000001},"min":{"1min":0.82499999999999996,"5min":0.746,"15min":0.746},"max":{"1min":1.5289999999999999,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.131},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.22,"15min":1.212},"min":{"1min":1.0600000000000001,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.5309999999999999,"5min":1.696,"15min":1.696},"last":1.242}]},{"osd":3,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.2110000000000001,"15min":1.2150000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.5900000000000001,"5min":1.635,"15min":1.635},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2430000000000001,"15min":1.2370000000000001},"min":{"1min":0.99299999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.6890000000000001,"5min":1.909,"15min":1.909},"last":1.6679999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.2649999999999999,"15min":1.2929999999999999},"min":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.4450000000000001,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":1.476},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.327,"15min":1.3460000000000001},"min":{"1min":0.875,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.6319999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.7569999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.365,"15min":1.373},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.73,"5min":2.319,"15min":2.319},"last":1.391},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4710000000000001,"15min":1.4790000000000001},"min":{"1min":1.218,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.79,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.5089999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4570000000000001,"15min":1.4650000000000001},"min":{"1min":1.109,"5min":0.754,"15min":0.754},"max":{"1min":1.7929999999999999,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.599},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3899999999999999,"15min":1.421},"min":{"1min":0.84899999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.7290000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.6359999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542192,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27756,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344084,"statfs":{"total":96636764160,"available":96608342016,"internally_reserved":0,"allocated":749568,"data_stored":308999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95499999999999996,"15min":0.95599999999999996},"min":{"1min":0.53800000000000003,"5min":0.434,"15min":0.434},"max":{"1min":1.3440000000000001,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.075,"5min":1.093,"15min":1.101},"min":{"1min":0.58999999999999997,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4850000000000001,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.2190000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.99099999999999999,"15min":1.0009999999999999},"min":{"1min":0.68300000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.6499999999999999,"5min":1.976,"15min":1.976},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.224,"5min":1.244,"15min":1.262},"min":{"1min":0.65100000000000002,"5min":0.5,"15min":0.5},"max":{"1min":1.5640000000000001,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.5580000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.1359999999999999,"15min":1.1479999999999999},"min":{"1min":0.61899999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.514,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.2170000000000001,"15min":1.2689999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4790000000000001,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.863}]},{"osd":4,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.28,"15min":1.2929999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.5049999999999999,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.506,"15min":1.546},"min":{"1min":0.998,"5min":0.998,"15min":0.998},"max":{"1min":1.788,"5min":4.8970000000000002,"15min":4.8970000000000002},"last":1.8}]},{"osd":5,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.5069999999999999,"15min":1.5369999999999999},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":1.702,"5min":3.4910000000000001,"15min":3.4910000000000001},"last":1.135},{"interface":"front","average":{"1min":1.379,"5min":1.4490000000000001,"15min":1.4850000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.716,"5min":3.125,"15min":3.125},"last":1.48}]},{"osd":6,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.494,"15min":1.55},"min":{"1min":0.94499999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.7410000000000001,"5min":4.5499999999999998,"15min":4.5499999999999998},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.532,"15min":1.5740000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.915,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":1.7410000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:21:55 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.49,"15min":1.534},"min":{"1min":0.90600000000000003,"5min":0.877,"15min":0.877},"max":{"1min":1.6759999999999999,"5min":2.952,"15min":2.952},"last":1.6759999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.6000000000000001,"15min":1.649},"min":{"1min":1.0049999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.847,"5min":4.798,"15min":4.798},"last":1.9530000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542192,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":749568,"data_stored":308999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.86799999999999999,"15min":0.89700000000000002},"min":{"1min":0.39400000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.0109999999999999,"15min":0.97999999999999998},"min":{"1min":0.46000000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.669,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.77600000000000002}]},{"osd":1,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.95299999999999996,"15min":0.97299999999999998},"min":{"1min":0.41499999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.429,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0009999999999999,"15min":0.98099999999999998},"min":{"1min":0.38400000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.468,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.90800000000000003}]},{"osd":3,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.119,"15min":1.119},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5369999999999999,"5min":1.98,"15min":1.98},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0840000000000001,"15min":1.0900000000000001},"min":{"1min":0.47399999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4430000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":0.98199999999999998}]},{"osd":4,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.357,"15min":1.351},"min":{"1min":0.93100000000000005,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.379,"15min":1.349},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.3740000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4159999999999999,"15min":1.4339999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.97,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":1.8149999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.3700000000000001,"15min":1.3720000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.8979999999999999,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.401}]},{"osd":6,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3779999999999999,"15min":1.3700000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.9370000000000001,"5min":2.758,"15min":2.758},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.5029999999999999,"15min":1.5309999999999999},"min":{"1min":0.98799999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.0310000000000001,"5min":2.8580000000000001,"15min":2.8580000000000001},"last":1.8879999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.419,"15min":1.391},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.2069999999999999,"5min":2.246,"15min":2.246},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5820000000000001,"15min":1.6100000000000001},"min":{"1min":1.03,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.8799999999999999,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.9510000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":729088,"data_stored":302064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.90200000000000002,"15min":0.86599999999999999},"min":{"1min":0.64300000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.6040000000000001,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.97699999999999998,"15min":0.98599999999999999},"min":{"1min":0.53700000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.7070000000000001,"5min":1.99,"15min":1.99},"last":0.89400000000000002}]},{"osd":2,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.96699999999999997,"15min":0.97899999999999998},"min":{"1min":0.64700000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.3540000000000001,"5min":1.738,"15min":1.738},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.1240000000000001,"15min":1.129},"min":{"1min":0.55500000000000005,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.8720000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.84599999999999997}]},{"osd":3,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2090000000000001,"15min":1.2210000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.6519999999999999,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.125,"5min":1.143,"15min":1.1739999999999999},"min":{"1min":0.72899999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.919,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.0980000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.262,"15min":1.2450000000000001},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.0720000000000001,"5min":2.153,"15min":2.153},"last":1.23},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.323,"15min":1.343},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6759999999999999,"5min":2.758,"15min":2.758},"last":1.663}]},{"osd":5,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.407,"15min":1.4450000000000001},"min":{"1min":0.96799999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.0009999999999999,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.9570000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4430000000000001,"15min":1.4670000000000001},"min":{"1min":0.80400000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.165,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.524}]},{"osd":6,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.429,"15min":1.4370000000000001},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.95,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.516,"15min":1.5820000000000001},"min":{"1min":0.96199999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.649,"5min":2.9630000000000001,"15min":2.9630000000000001},"last":1.841}]},{"osd":7,"last update":"Sat Mar 23 11:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.5529999999999999,"15min":1.5900000000000001},"min":{"1min":0.89700000000000002,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":2.2120000000000002,"5min":3.2749999999999999,"15min":3.2749999999999999},"last":1.8049999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.5649999999999999,"15min":1.5940000000000001},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.0840000000000001,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.768}]}]},{"osd":0,"up_from":14,"seq":60129542191,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28332,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343508,"statfs":{"total":96636764160,"available":96607752192,"internally_reserved":0,"allocated":1335296,"data_stored":894488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.86499999999999999,"15min":0.85699999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.94299999999999995,"15min":0.92500000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.601,"5min":1.694,"15min":1.694},"last":0.72399999999999998}]},{"osd":2,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.0489999999999999,"15min":1.0329999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.03,"15min":1.0089999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.5009999999999999,"5min":1.802,"15min":1.802},"last":1.294}]},{"osd":3,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.101,"15min":1.1020000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.306,"15min":1.3140000000000001},"min":{"1min":0.57499999999999996,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9450000000000001,"5min":2.1819999999999999,"15min":2.1819999999999999},"last":0.89400000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.379,"15min":1.379},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.8080000000000001,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.266,"15min":1.268},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.73,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.546}]},{"osd":5,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.389,"15min":1.4490000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.597,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.339,"5min":1.5009999999999999,"15min":1.5469999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.77,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.252}]},{"osd":6,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.5229999999999999,"15min":1.5680000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7609999999999999,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.4099999999999999,"15min":1.4339999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.7210000000000001,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.468}]},{"osd":7,"last update":"Sat Mar 23 11:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.538,"15min":1.554},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.722,"5min":2.617,"15min":2.617},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4970000000000001,"15min":1.5149999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":2.0379999999999998,"5min":3.081,"15min":3.081},"last":1.6879999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:22:37.239 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T11:22:37.240 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T11:22:37.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.496+0000 7f88bd000640 1 Processor -- start 2024-03-23T11:22:37.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.496+0000 7f88bd000640 1 -- start start 2024-03-23T11:22:37.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.498+0000 7f88bd000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88b80ce6c0 0x7f88b80b3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:37.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.498+0000 7f88bd000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88b80d2e60 0x7f88b80d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:37.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.498+0000 7f88bd000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b80d0070 con 0x7f88b80b3590 2024-03-23T11:22:37.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.498+0000 7f88bd000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b80b1cc0 con 0x7f88b80ce6c0 2024-03-23T11:22:37.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.498+0000 7f88bd000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b80cfd90 con 0x7f88b80d2e60 2024-03-23T11:22:37.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.498+0000 7f88b7400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88b80ce6c0 0x7f88b80b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:37.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.498+0000 7f88b7400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88b80ce6c0 0x7f88b80b3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55878/0 (socket says 172.21.15.62:55878) 2024-03-23T11:22:37.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.498+0000 7f88b7400640 1 -- 172.21.15.62:0/1148401487 learned_addr learned my addr 172.21.15.62:0/1148401487 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:37.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.498+0000 7f88b7400640 1 -- 172.21.15.62:0/1148401487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88b80d2e60 msgr2=0x7f88b80d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:37.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.498+0000 7f88b7e00640 1 --2- 172.21.15.62:0/1148401487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88b80d2e60 0x7f88b80d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:37.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.499+0000 7f88b7400640 1 --2- 172.21.15.62:0/1148401487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88b80d2e60 0x7f88b80d0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.499+0000 7f88b7400640 1 -- 172.21.15.62:0/1148401487 >> v1:172.21.15.120:6789/0 conn(0x7f88b80b3590 legacy=0x7f88b80d2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:37.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.499+0000 7f88b7400640 1 -- 172.21.15.62:0/1148401487 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b80b2570 con 0x7f88b80ce6c0 2024-03-23T11:22:37.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.499+0000 7f88b7e00640 1 --2- 172.21.15.62:0/1148401487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88b80d2e60 0x7f88b80d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:37.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.500+0000 7f88b7400640 1 --2- 172.21.15.62:0/1148401487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88b80ce6c0 0x7f88b80b3050 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f88b0009ea0 tx=0x7f88b002be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b36f2b65c52e1c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:37.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.500+0000 7f88b6000640 1 -- 172.21.15.62:0/1148401487 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b002ca20 con 0x7f88b80ce6c0 2024-03-23T11:22:37.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.500+0000 7f88b6000640 1 -- 172.21.15.62:0/1148401487 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b002cb80 con 0x7f88b80ce6c0 2024-03-23T11:22:37.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.500+0000 7f88b6000640 1 -- 172.21.15.62:0/1148401487 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b00343f0 con 0x7f88b80ce6c0 2024-03-23T11:22:37.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.501+0000 7f88bd000640 1 -- 172.21.15.62:0/1148401487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88b80ce6c0 msgr2=0x7f88b80b3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:37.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.501+0000 7f88bd000640 1 --2- 172.21.15.62:0/1148401487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88b80ce6c0 0x7f88b80b3050 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f88b0009ea0 tx=0x7f88b002be80 comp rx=0 tx=0).stop 2024-03-23T11:22:37.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.501+0000 7f88bd000640 1 -- 172.21.15.62:0/1148401487 shutdown_connections 2024-03-23T11:22:37.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.501+0000 7f88bd000640 1 --2- 172.21.15.62:0/1148401487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88b80d2e60 0x7f88b80d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.501+0000 7f88bd000640 1 --2- 172.21.15.62:0/1148401487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88b80ce6c0 0x7f88b80b3050 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.501+0000 7f88bd000640 1 -- 172.21.15.62:0/1148401487 >> 172.21.15.62:0/1148401487 conn(0x7f88b809ba70 msgr2=0x7f88b8089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:37.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.501+0000 7f88bd000640 1 -- 172.21.15.62:0/1148401487 shutdown_connections 2024-03-23T11:22:37.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.501+0000 7f88bd000640 1 -- 172.21.15.62:0/1148401487 wait complete. 2024-03-23T11:22:37.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.503+0000 7f88bd000640 1 Processor -- start 2024-03-23T11:22:37.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.503+0000 7f88bd000640 1 -- start start 2024-03-23T11:22:37.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.504+0000 7f88bd000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88b80b3590 0x7f88b81f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:37.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.504+0000 7f88bd000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88b80ce6c0 0x7f88b8154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:37.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.504+0000 7f88bd000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88b80d2e60 0x7f88b8159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:37.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.504+0000 7f88bd000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b80c6890 con 0x7f88b80d2e60 2024-03-23T11:22:37.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.504+0000 7f88bd000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b80ca540 con 0x7f88b80ce6c0 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.504+0000 7f88bd000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b8154d70 con 0x7f88b80b3590 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.504+0000 7f88b7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88b80d2e60 0x7f88b8159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.505+0000 7f88b7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88b80d2e60 0x7f88b8159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55880/0 (socket says 172.21.15.62:55880) 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.505+0000 7f88b7e00640 1 -- 172.21.15.62:0/557938661 learned_addr learned my addr 172.21.15.62:0/557938661 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.505+0000 7f88b7400640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88b80b3590 0x7f88b81f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.505+0000 7f88b6a00640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88b80ce6c0 0x7f88b8154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.505+0000 7f88b7e00640 1 -- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88b80b3590 msgr2=0x7f88b81f63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.505+0000 7f88b7e00640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88b80b3590 0x7f88b81f63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.505+0000 7f88b7e00640 1 -- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88b80ce6c0 msgr2=0x7f88b8154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.505+0000 7f88b7e00640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88b80ce6c0 0x7f88b8154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.505+0000 7f88b7e00640 1 -- 172.21.15.62:0/557938661 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b0009b30 con 0x7f88b80d2e60 2024-03-23T11:22:37.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.505+0000 7f88b7e00640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88b80d2e60 0x7f88b8159050 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f88a800c7f0 tx=0x7f88a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.506+0000 7f88af400640 1 -- 172.21.15.62:0/557938661 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8004220 con 0x7f88b80d2e60 2024-03-23T11:22:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.506+0000 7f88af400640 1 -- 172.21.15.62:0/557938661 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a80043a0 con 0x7f88b80d2e60 2024-03-23T11:22:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.506+0000 7f88af400640 1 -- 172.21.15.62:0/557938661 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8007860 con 0x7f88b80d2e60 2024-03-23T11:22:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.506+0000 7f88bd000640 1 -- 172.21.15.62:0/557938661 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b80c9930 con 0x7f88b80d2e60 2024-03-23T11:22:37.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.506+0000 7f88bd000640 1 -- 172.21.15.62:0/557938661 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b80c9ef0 con 0x7f88b80d2e60 2024-03-23T11:22:37.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.507+0000 7f88bd000640 1 -- 172.21.15.62:0/557938661 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b80ca6b0 con 0x7f88b80d2e60 2024-03-23T11:22:37.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.508+0000 7f88af400640 1 -- 172.21.15.62:0/557938661 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f88a8007bd0 con 0x7f88b80d2e60 2024-03-23T11:22:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.509+0000 7f88af400640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f889806a7a0 0x7f889806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.509+0000 7f88af400640 1 -- 172.21.15.62:0/557938661 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88a8088ce0 con 0x7f88b80d2e60 2024-03-23T11:22:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.510+0000 7f88b7400640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f889806a7a0 0x7f889806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.511+0000 7f88b7400640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f889806a7a0 0x7f889806cc60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f88b0035000 tx=0x7f88b0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:37.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.514+0000 7f88af400640 1 -- 172.21.15.62:0/557938661 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f88a8054250 con 0x7f88b80d2e60 2024-03-23T11:22:37.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.785+0000 7f88bd000640 1 -- 172.21.15.62:0/557938661 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f88b80672c0 con 0x7f88b80d2e60 2024-03-23T11:22:37.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.794+0000 7f88af400640 1 -- 172.21.15.62:0/557938661 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+0 (secure 0 0 0) 0x7f88a801e0e0 con 0x7f88b80d2e60 2024-03-23T11:22:37.792 INFO:teuthology.orchestra.run.smithi062.stderr:updated caps for client.0 2024-03-23T11:22:37.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 -- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f889806a7a0 msgr2=0x7f889806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:37.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f889806a7a0 0x7f889806cc60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f88b0035000 tx=0x7f88b0030000 comp rx=0 tx=0).stop 2024-03-23T11:22:37.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 -- 172.21.15.62:0/557938661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88b80d2e60 msgr2=0x7f88b8159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:37.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88b80d2e60 0x7f88b8159050 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f88a800c7f0 tx=0x7f88a800ccb0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 -- 172.21.15.62:0/557938661 shutdown_connections 2024-03-23T11:22:37.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f889806a7a0 0x7f889806cc60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88b80d2e60 0x7f88b8159050 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88b80ce6c0 0x7f88b8154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 --2- 172.21.15.62:0/557938661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88b80b3590 0x7f88b81f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:37.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 -- 172.21.15.62:0/557938661 >> 172.21.15.62:0/557938661 conn(0x7f88b809ba70 msgr2=0x7f88b80cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:37.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 -- 172.21.15.62:0/557938661 shutdown_connections 2024-03-23T11:22:37.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:37.797+0000 7f88bd000640 1 -- 172.21.15.62:0/557938661 wait complete. 2024-03-23T11:22:37.811 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:22:38.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.025+0000 7ff05dc00640 1 Processor -- start 2024-03-23T11:22:38.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.026+0000 7ff05dc00640 1 -- start start 2024-03-23T11:22:38.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.027+0000 7ff05dc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff058143dd0 0x7ff05815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.027+0000 7ff05dc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff0580b2f30 0x7ff058070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.027+0000 7ff05dc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0580cd140 con 0x7ff058070090 2024-03-23T11:22:38.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.027+0000 7ff05dc00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff058070a60 con 0x7ff058143dd0 2024-03-23T11:22:38.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.027+0000 7ff05dc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0580cd3e0 con 0x7ff0580b2f30 2024-03-23T11:22:38.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.027+0000 7ff057e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff058143dd0 0x7ff05815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.027+0000 7ff057e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff058143dd0 0x7ff05815abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55886/0 (socket says 172.21.15.62:55886) 2024-03-23T11:22:38.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.027+0000 7ff057e00640 1 -- 172.21.15.62:0/1016558064 learned_addr learned my addr 172.21.15.62:0/1016558064 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:38.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.028+0000 7ff05d200640 1 --2- 172.21.15.62:0/1016558064 >> v2:172.21.15.120:3300/0 conn(0x7ff0580b2f30 0x7ff058070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.028+0000 7ff057e00640 1 -- 172.21.15.62:0/1016558064 >> v2:172.21.15.120:3300/0 conn(0x7ff0580b2f30 msgr2=0x7ff058070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.028+0000 7ff057e00640 1 --2- 172.21.15.62:0/1016558064 >> v2:172.21.15.120:3300/0 conn(0x7ff0580b2f30 0x7ff058070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.028+0000 7ff057e00640 1 -- 172.21.15.62:0/1016558064 >> v1:172.21.15.120:6789/0 conn(0x7ff058070090 legacy=0x7ff0580b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.028+0000 7ff057e00640 1 -- 172.21.15.62:0/1016558064 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0580cd6c0 con 0x7ff058143dd0 2024-03-23T11:22:38.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.028+0000 7ff05d200640 1 --2- 172.21.15.62:0/1016558064 >> v2:172.21.15.120:3300/0 conn(0x7ff0580b2f30 0x7ff058070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:38.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.029+0000 7ff057e00640 1 --2- 172.21.15.62:0/1016558064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff058143dd0 0x7ff05815abe0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff0580cbe50 tx=0x7ff04802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4318994cfbb367 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:38.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.029+0000 7ff057400640 1 -- 172.21.15.62:0/1016558064 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048031870 con 0x7ff058143dd0 2024-03-23T11:22:38.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.029+0000 7ff057400640 1 -- 172.21.15.62:0/1016558064 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0480319d0 con 0x7ff058143dd0 2024-03-23T11:22:38.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.029+0000 7ff057400640 1 -- 172.21.15.62:0/1016558064 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048031d60 con 0x7ff058143dd0 2024-03-23T11:22:38.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.031+0000 7ff05dc00640 1 -- 172.21.15.62:0/1016558064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff058143dd0 msgr2=0x7ff05815abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.031+0000 7ff05dc00640 1 --2- 172.21.15.62:0/1016558064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff058143dd0 0x7ff05815abe0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff0580cbe50 tx=0x7ff04802b910 comp rx=0 tx=0).stop 2024-03-23T11:22:38.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.031+0000 7ff05dc00640 1 -- 172.21.15.62:0/1016558064 shutdown_connections 2024-03-23T11:22:38.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.031+0000 7ff05dc00640 1 --2- 172.21.15.62:0/1016558064 >> v2:172.21.15.120:3300/0 conn(0x7ff0580b2f30 0x7ff058070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.031+0000 7ff05dc00640 1 --2- 172.21.15.62:0/1016558064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff058143dd0 0x7ff05815abe0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.031+0000 7ff05dc00640 1 -- 172.21.15.62:0/1016558064 >> 172.21.15.62:0/1016558064 conn(0x7ff058066050 msgr2=0x7ff058069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:38.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.031+0000 7ff05dc00640 1 -- 172.21.15.62:0/1016558064 shutdown_connections 2024-03-23T11:22:38.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.031+0000 7ff05dc00640 1 -- 172.21.15.62:0/1016558064 wait complete. 2024-03-23T11:22:38.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.032+0000 7ff05dc00640 1 Processor -- start 2024-03-23T11:22:38.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.032+0000 7ff05dc00640 1 -- start start 2024-03-23T11:22:38.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.033+0000 7ff05dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff058070090 0x7ff0580cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.033+0000 7ff05dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0580b2f30 0x7ff0580c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.033+0000 7ff05dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff058143dd0 0x7ff05813fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.033+0000 7ff05dc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0580c4490 con 0x7ff058143dd0 2024-03-23T11:22:38.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.033+0000 7ff05dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05806c6a0 con 0x7ff0580b2f30 2024-03-23T11:22:38.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.033+0000 7ff05dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05806c090 con 0x7ff058070090 2024-03-23T11:22:38.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.033+0000 7ff05d200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff058143dd0 0x7ff05813fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.033+0000 7ff05d200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff058143dd0 0x7ff05813fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55898/0 (socket says 172.21.15.62:55898) 2024-03-23T11:22:38.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.033+0000 7ff05d200640 1 -- 172.21.15.62:0/2077909025 learned_addr learned my addr 172.21.15.62:0/2077909025 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.034+0000 7ff05d200640 1 -- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff058070090 msgr2=0x7ff0580cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.034+0000 7ff05d200640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff058070090 0x7ff0580cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.034+0000 7ff05d200640 1 -- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0580b2f30 msgr2=0x7ff0580c3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.034+0000 7ff05d200640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0580b2f30 0x7ff0580c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.034+0000 7ff05d200640 1 -- 172.21.15.62:0/2077909025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0480094d0 con 0x7ff058143dd0 2024-03-23T11:22:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.034+0000 7ff05d200640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff058143dd0 0x7ff05813fcb0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff05000cef0 tx=0x7ff05000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.034+0000 7ff054c00640 1 -- 172.21.15.62:0/2077909025 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05000d840 con 0x7ff058143dd0 2024-03-23T11:22:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.034+0000 7ff054c00640 1 -- 172.21.15.62:0/2077909025 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05000d9c0 con 0x7ff058143dd0 2024-03-23T11:22:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.034+0000 7ff054c00640 1 -- 172.21.15.62:0/2077909025 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05000dd70 con 0x7ff058143dd0 2024-03-23T11:22:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.034+0000 7ff05dc00640 1 -- 172.21.15.62:0/2077909025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0580c46c0 con 0x7ff058143dd0 2024-03-23T11:22:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.034+0000 7ff05dc00640 1 -- 172.21.15.62:0/2077909025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05806c330 con 0x7ff058143dd0 2024-03-23T11:22:38.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.035+0000 7ff05dc00640 1 -- 172.21.15.62:0/2077909025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff058066bb0 con 0x7ff058143dd0 2024-03-23T11:22:38.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.036+0000 7ff054c00640 1 -- 172.21.15.62:0/2077909025 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff05001e3b0 con 0x7ff058143dd0 2024-03-23T11:22:38.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.037+0000 7ff054c00640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff03406a7a0 0x7ff03406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.037+0000 7ff054c00640 1 -- 172.21.15.62:0/2077909025 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff050089850 con 0x7ff058143dd0 2024-03-23T11:22:38.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.038+0000 7ff057e00640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff03406a7a0 0x7ff03406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.039+0000 7ff057e00640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff03406a7a0 0x7ff03406cc60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff048036000 tx=0x7ff048030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:38.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.041+0000 7ff054c00640 1 -- 172.21.15.62:0/2077909025 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff050054c60 con 0x7ff058143dd0 2024-03-23T11:22:38.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.279+0000 7ff05dc00640 1 -- 172.21.15.62:0/2077909025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff05806c810 con 0x7ff058143dd0 2024-03-23T11:22:38.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.284+0000 7ff054c00640 1 -- 172.21.15.62:0/2077909025 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4740 (secure 0 0 0) 0x7ff05001a090 con 0x7ff058143dd0 2024-03-23T11:22:38.281 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:38.281 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3287045135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3287045135},{"type":"v1","addr":"172.21.15.120:6835","nonce":3287045135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6066,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/2859430143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2859430143},{"type":"v1","addr":"172.21.15.62:6834","nonce":2859430143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2005768060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2005768060},{"type":"v1","addr":"172.21.15.62:6835","nonce":2005768060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:22:31.725865+0000","modified":"2024-03-23T11:22:35.259834+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6081},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6081":{"gid":6081,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/4158450746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":4158450746},{"type":"v1","addr":"172.21.15.120:6837","nonce":4158450746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T11:22:38.281 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 33 2024-03-23T11:22:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.286+0000 7ff05dc00640 1 -- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff03406a7a0 msgr2=0x7ff03406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.286+0000 7ff05dc00640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff03406a7a0 0x7ff03406cc60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff048036000 tx=0x7ff048030000 comp rx=0 tx=0).stop 2024-03-23T11:22:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.286+0000 7ff05dc00640 1 -- 172.21.15.62:0/2077909025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff058143dd0 msgr2=0x7ff05813fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.286+0000 7ff05dc00640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff058143dd0 0x7ff05813fcb0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff05000cef0 tx=0x7ff05000a710 comp rx=0 tx=0).stop 2024-03-23T11:22:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.287+0000 7ff05dc00640 1 -- 172.21.15.62:0/2077909025 shutdown_connections 2024-03-23T11:22:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.287+0000 7ff05dc00640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff03406a7a0 0x7ff03406cc60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.287+0000 7ff05dc00640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff058143dd0 0x7ff05813fcb0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.287+0000 7ff05dc00640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0580b2f30 0x7ff0580c3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.287+0000 7ff05dc00640 1 --2- 172.21.15.62:0/2077909025 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff058070090 0x7ff0580cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.287+0000 7ff05dc00640 1 -- 172.21.15.62:0/2077909025 >> 172.21.15.62:0/2077909025 conn(0x7ff058066050 msgr2=0x7ff05815b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.287+0000 7ff05dc00640 1 -- 172.21.15.62:0/2077909025 shutdown_connections 2024-03-23T11:22:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.287+0000 7ff05dc00640 1 -- 172.21.15.62:0/2077909025 wait complete. 2024-03-23T11:22:38.706 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:22:31.725865+0000', 'modified': '2024-03-23T11:22:35.259834+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6081}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6081': {'gid': 6081, 'name': 'b', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6837/4158450746', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 4158450746}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 4158450746}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:22:38.706 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:22:38.707 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-23T11:22:38.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.938+0000 7ff33be00640 1 Processor -- start 2024-03-23T11:22:38.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.938+0000 7ff33be00640 1 -- start start 2024-03-23T11:22:38.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.940+0000 7ff33be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff33c069c60 0x7ff33c06a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.940+0000 7ff33be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff33c0caff0 0x7ff33c0cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.940+0000 7ff33be00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff33c15bb10 con 0x7ff33c143450 2024-03-23T11:22:38.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.940+0000 7ff33be00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33c0ce1a0 con 0x7ff33c069c60 2024-03-23T11:22:38.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.940+0000 7ff33be00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33c0cbcb0 con 0x7ff33c0caff0 2024-03-23T11:22:38.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.941+0000 7ff33aa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff33c069c60 0x7ff33c06a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.941+0000 7ff33aa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff33c069c60 0x7ff33c06a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55900/0 (socket says 172.21.15.62:55900) 2024-03-23T11:22:38.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.941+0000 7ff33aa00640 1 -- 172.21.15.62:0/4103528469 learned_addr learned my addr 172.21.15.62:0/4103528469 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:38.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.941+0000 7ff332000640 1 --2- 172.21.15.62:0/4103528469 >> v2:172.21.15.120:3300/0 conn(0x7ff33c0caff0 0x7ff33c0cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.941+0000 7ff33aa00640 1 -- 172.21.15.62:0/4103528469 >> v2:172.21.15.120:3300/0 conn(0x7ff33c0caff0 msgr2=0x7ff33c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.941+0000 7ff33aa00640 1 --2- 172.21.15.62:0/4103528469 >> v2:172.21.15.120:3300/0 conn(0x7ff33c0caff0 0x7ff33c0cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.941+0000 7ff33aa00640 1 -- 172.21.15.62:0/4103528469 >> v1:172.21.15.120:6789/0 conn(0x7ff33c143450 legacy=0x7ff33c1438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.941+0000 7ff33aa00640 1 -- 172.21.15.62:0/4103528469 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33c06a580 con 0x7ff33c069c60 2024-03-23T11:22:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.942+0000 7ff33aa00640 1 --2- 172.21.15.62:0/4103528469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff33c069c60 0x7ff33c06a040 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff33c071110 tx=0x7ff32c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933d340b61555066 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:38.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.943+0000 7ff33a000640 1 -- 172.21.15.62:0/4103528469 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c031870 con 0x7ff33c069c60 2024-03-23T11:22:38.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.943+0000 7ff33a000640 1 -- 172.21.15.62:0/4103528469 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c0319d0 con 0x7ff33c069c60 2024-03-23T11:22:38.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.943+0000 7ff33a000640 1 -- 172.21.15.62:0/4103528469 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c031d60 con 0x7ff33c069c60 2024-03-23T11:22:38.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.943+0000 7ff33be00640 1 -- 172.21.15.62:0/4103528469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff33c069c60 msgr2=0x7ff33c06a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.943+0000 7ff33be00640 1 --2- 172.21.15.62:0/4103528469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff33c069c60 0x7ff33c06a040 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff33c071110 tx=0x7ff32c02b910 comp rx=0 tx=0).stop 2024-03-23T11:22:38.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.943+0000 7ff33be00640 1 -- 172.21.15.62:0/4103528469 shutdown_connections 2024-03-23T11:22:38.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.943+0000 7ff33be00640 1 --2- 172.21.15.62:0/4103528469 >> v2:172.21.15.120:3300/0 conn(0x7ff33c0caff0 0x7ff33c0cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.943+0000 7ff33be00640 1 --2- 172.21.15.62:0/4103528469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff33c069c60 0x7ff33c06a040 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.943+0000 7ff33be00640 1 -- 172.21.15.62:0/4103528469 >> 172.21.15.62:0/4103528469 conn(0x7ff33c0b0090 msgr2=0x7ff33c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:38.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.943+0000 7ff33be00640 1 -- 172.21.15.62:0/4103528469 shutdown_connections 2024-03-23T11:22:38.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.944+0000 7ff33be00640 1 -- 172.21.15.62:0/4103528469 wait complete. 2024-03-23T11:22:38.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.945+0000 7ff33be00640 1 Processor -- start 2024-03-23T11:22:38.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.945+0000 7ff33be00640 1 -- start start 2024-03-23T11:22:38.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.946+0000 7ff33be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff33c069c60 0x7ff33c0c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.946+0000 7ff33be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff33c0caff0 0x7ff33c158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.946+0000 7ff33be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff33c143450 0x7ff33c1f5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.946+0000 7ff33be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c1592d0 con 0x7ff33c143450 2024-03-23T11:22:38.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.946+0000 7ff33be00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c1f2fe0 con 0x7ff33c0caff0 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.946+0000 7ff33be00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c1594c0 con 0x7ff33c069c60 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.947+0000 7ff33b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff33c143450 0x7ff33c1f5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.947+0000 7ff33b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff33c143450 0x7ff33c1f5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55902/0 (socket says 172.21.15.62:55902) 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.947+0000 7ff33b400640 1 -- 172.21.15.62:0/3577705974 learned_addr learned my addr 172.21.15.62:0/3577705974 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.947+0000 7ff33aa00640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff33c069c60 0x7ff33c0c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.947+0000 7ff332000640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff33c0caff0 0x7ff33c158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.947+0000 7ff33b400640 1 -- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff33c069c60 msgr2=0x7ff33c0c3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.947+0000 7ff33b400640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff33c069c60 0x7ff33c0c3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.947+0000 7ff33b400640 1 -- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff33c0caff0 msgr2=0x7ff33c158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.947+0000 7ff33b400640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff33c0caff0 0x7ff33c158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.947+0000 7ff33b400640 1 -- 172.21.15.62:0/3577705974 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c0094d0 con 0x7ff33c143450 2024-03-23T11:22:38.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.947+0000 7ff33b400640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff33c143450 0x7ff33c1f5410 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff33400b970 tx=0x7ff33400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:38.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.948+0000 7ff333e00640 1 -- 172.21.15.62:0/3577705974 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33400cb40 con 0x7ff33c143450 2024-03-23T11:22:38.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.948+0000 7ff333e00640 1 -- 172.21.15.62:0/3577705974 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33400cca0 con 0x7ff33c143450 2024-03-23T11:22:38.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.948+0000 7ff33be00640 1 -- 172.21.15.62:0/3577705974 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33c1597a0 con 0x7ff33c143450 2024-03-23T11:22:38.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.948+0000 7ff333e00640 1 -- 172.21.15.62:0/3577705974 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334014680 con 0x7ff33c143450 2024-03-23T11:22:38.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.948+0000 7ff33be00640 1 -- 172.21.15.62:0/3577705974 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33c159d60 con 0x7ff33c143450 2024-03-23T11:22:38.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.951+0000 7ff333e00640 1 -- 172.21.15.62:0/3577705974 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff33401d3b0 con 0x7ff33c143450 2024-03-23T11:22:38.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.952+0000 7ff333e00640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff31806a7a0 0x7ff31806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.952+0000 7ff333e00640 1 -- 172.21.15.62:0/3577705974 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff334088c00 con 0x7ff33c143450 2024-03-23T11:22:38.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.952+0000 7ff344b90740 1 Processor -- start 2024-03-23T11:22:38.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.952+0000 7ff33aa00640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff31806a7a0 0x7ff31806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.952+0000 7ff344b90740 1 -- start start 2024-03-23T11:22:38.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.953+0000 7ff344b90740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4acb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.953+0000 7ff344b90740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.953+0000 7ff344b90740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4a7a40 0x555f0b4d7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.953+0000 7ff344b90740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0b418110 con 0x555f0b4ab5a0 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.953+0000 7ff344b90740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0b4158a0 con 0x555f0b4a7a40 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.953+0000 7ff344b90740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f0b411190 con 0x555f0b4f5520 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.953+0000 7ff33aa00640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff31806a7a0 0x7ff31806cc60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff32c036000 tx=0x7ff32c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff332000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff332000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4ad090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55914/0 (socket says 172.21.15.62:55914) 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff332000640 1 -- 172.21.15.62:0/991892825 learned_addr learned my addr 172.21.15.62:0/991892825 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff33aa00640 1 --2- 172.21.15.62:0/991892825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4acb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff33b400640 1 --2- 172.21.15.62:0/991892825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4a7a40 0x555f0b4d7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff332000640 1 -- 172.21.15.62:0/991892825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 msgr2=0x555f0b4acb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff332000640 1 --2- 172.21.15.62:0/991892825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4acb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff332000640 1 -- 172.21.15.62:0/991892825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4a7a40 msgr2=0x555f0b4d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff332000640 1 --2- 172.21.15.62:0/991892825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4a7a40 0x555f0b4d7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff332000640 1 -- 172.21.15.62:0/991892825 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f0b3dc230 con 0x555f0b4ab5a0 2024-03-23T11:22:38.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff33b400640 1 --2- 172.21.15.62:0/991892825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4a7a40 0x555f0b4d7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:38.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.954+0000 7ff332000640 1 --2- 172.21.15.62:0/991892825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4ad090 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff32400b870 tx=0x7ff32400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa42161628a630b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:38.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff32be00640 1 -- 172.21.15.62:0/991892825 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff324004020 con 0x555f0b4ab5a0 2024-03-23T11:22:38.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff32be00640 1 -- 172.21.15.62:0/991892825 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff324004180 con 0x555f0b4ab5a0 2024-03-23T11:22:38.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff32be00640 1 -- 172.21.15.62:0/991892825 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3240044f0 con 0x555f0b4ab5a0 2024-03-23T11:22:38.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff344b90740 1 -- 172.21.15.62:0/991892825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 msgr2=0x555f0b4ad090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff344b90740 1 --2- 172.21.15.62:0/991892825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4ad090 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff32400b870 tx=0x7ff32400bd30 comp rx=0 tx=0).stop 2024-03-23T11:22:38.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff344b90740 1 -- 172.21.15.62:0/991892825 shutdown_connections 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff344b90740 1 --2- 172.21.15.62:0/991892825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4ad090 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff344b90740 1 --2- 172.21.15.62:0/991892825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4a7a40 0x555f0b4d7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff344b90740 1 --2- 172.21.15.62:0/991892825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4acb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff344b90740 1 -- 172.21.15.62:0/991892825 >> 172.21.15.62:0/991892825 conn(0x555f0b4a9cb0 msgr2=0x555f0b4f2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff344b90740 1 -- 172.21.15.62:0/991892825 shutdown_connections 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.955+0000 7ff344b90740 1 -- 172.21.15.62:0/991892825 wait complete. 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.956+0000 7ff344b90740 1 -- start start 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.956+0000 7ff344b90740 1 Processor -- start 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.957+0000 7ff344b90740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4c6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.957+0000 7ff344b90740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4d3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.957+0000 7ff344b90740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 0x555f0b4c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.957+0000 7ff344b90740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0b3b28c0 con 0x555f0b4ab5a0 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.957+0000 7ff344b90740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0b3b2560 con 0x555f0b4f4ee0 2024-03-23T11:22:38.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.957+0000 7ff344b90740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f0b178f20 con 0x555f0b4f5520 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.957+0000 7ff344b90740 10 client.0 _ll_register_callbacks cb 0x555f0b40a800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff33aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff33aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4c6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55916/0 (socket says 172.21.15.62:55916) 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff33aa00640 1 -- 172.21.15.62:0/2899641390 learned_addr learned my addr 172.21.15.62:0/2899641390 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff332000640 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4d3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff33b400640 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 0x555f0b4c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff33aa00640 1 -- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 msgr2=0x555f0b4d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff33aa00640 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4d3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff33aa00640 1 -- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 msgr2=0x555f0b4c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff33aa00640 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 0x555f0b4c9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff33aa00640 1 -- 172.21.15.62:0/2899641390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32400e040 con 0x555f0b4ab5a0 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff332000640 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4d3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:38.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff33aa00640 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4c6910 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff32c02bf20 tx=0x7ff32c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bcb7a31c86250c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.958+0000 7ff328c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff328c00640 1 -- 172.21.15.62:0/2899641390 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c044a10 con 0x555f0b4ab5a0 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff328c00640 1 -- 172.21.15.62:0/2899641390 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff32c03f070 con 0x555f0b4ab5a0 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff344b90740 1 -- 172.21.15.62:0/2899641390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555f0b43b280 con 0x555f0b4ab5a0 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff328c00640 1 -- 172.21.15.62:0/2899641390 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff32c044b70 con 0x555f0b4ab5a0 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff344b90740 10 client.6210 fetch_fsmap learned FSMap version 33 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff344b90740 1 -- 172.21.15.62:0/2899641390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x555f0b40f810 con 0x555f0b4ab5a0 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff328c00640 1 -- 172.21.15.62:0/2899641390 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555f0b43b280 con 0x555f0b4ab5a0 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff328c00640 1 -- 172.21.15.62:0/2899641390 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff32c04e070 con 0x555f0b4ab5a0 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff344b90740 10 client.6210 fetch_fsmap finished waiting for FSMap version 33 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff344b90740 10 client.6210 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff344b90740 20 client.6210 populate_metadata read hostname 'smithi062' 2024-03-23T11:22:38.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff344b90740 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x555f0b4f3aa0 0x555f0b4f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff344b90740 4 client.6210 mds_command: new command op to 6081 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:22:38.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.959+0000 7ff344b90740 1 -- 172.21.15.62:0/2899641390 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f0b4039d0 con 0x555f0b4f3aa0 2024-03-23T11:22:38.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.960+0000 7ff332000640 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x555f0b4f3aa0 0x555f0b4f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.961+0000 7ff332000640 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x555f0b4f3aa0 0x555f0b4f3ea0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a5c7337adb0bbf2 server_cookie=d89fce4864c42926 in_seq=0 out_seq=0 2024-03-23T11:22:38.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.961+0000 7ff328c00640 10 client.6210 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:38.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.963+0000 7ff328c00640 1 -- 172.21.15.62:0/2899641390 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x555f0b4039d0 con 0x555f0b4f3aa0 2024-03-23T11:22:38.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.963+0000 7ff328c00640 10 client.6210 handle_command_reply: tid=0 2024-03-23T11:22:38.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.963+0000 7ff344b90740 1 client.6210 shutdown 2024-03-23T11:22:38.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.963+0000 7ff344b90740 1 -- 172.21.15.62:0/2899641390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 msgr2=0x555f0b4c6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.963+0000 7ff344b90740 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4c6910 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff32c02bf20 tx=0x7ff32c004730 comp rx=0 tx=0).stop 2024-03-23T11:22:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.964+0000 7ff344b90740 1 -- 172.21.15.62:0/2899641390 shutdown_connections 2024-03-23T11:22:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.964+0000 7ff344b90740 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x555f0b4f3aa0 0x555f0b4f3ea0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.964+0000 7ff344b90740 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 0x555f0b4c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.964+0000 7ff344b90740 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4d3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.964+0000 7ff344b90740 1 --2- 172.21.15.62:0/2899641390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b4c6910 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.964+0000 7ff344b90740 1 -- 172.21.15.62:0/2899641390 >> 172.21.15.62:0/2899641390 conn(0x555f0b4a7a40 msgr2=0x555f0b4abd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.964+0000 7ff328c00640 0 client.6210 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.964+0000 7ff344b90740 1 -- 172.21.15.62:0/2899641390 shutdown_connections 2024-03-23T11:22:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.964+0000 7ff344b90740 1 -- 172.21.15.62:0/2899641390 wait complete. 2024-03-23T11:22:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.964+0000 7ff344b90740 20 client.6210 trim_cache size 0 max 16384 2024-03-23T11:22:38.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.980+0000 7ff344b90740 1 Processor -- start 2024-03-23T11:22:38.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.980+0000 7ff344b90740 1 -- start start 2024-03-23T11:22:38.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.981+0000 7ff344b90740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4ab5a0 0x555f0b4d0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.981+0000 7ff344b90740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 0x555f0b4c9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.981+0000 7ff344b90740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4f5520 0x555f0b4c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.981+0000 7ff344b90740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0b4a9110 con 0x555f0b4f5520 2024-03-23T11:22:38.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.981+0000 7ff344b90740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0b4a84a0 con 0x555f0b4f4ee0 2024-03-23T11:22:38.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.981+0000 7ff344b90740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f0b4d0b20 con 0x555f0b4ab5a0 2024-03-23T11:22:38.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.981+0000 7ff33b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4f5520 0x555f0b4c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.981+0000 7ff33b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4f5520 0x555f0b4c9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55920/0 (socket says 172.21.15.62:55920) 2024-03-23T11:22:38.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.981+0000 7ff33b400640 1 -- 172.21.15.62:0/2701748881 learned_addr learned my addr 172.21.15.62:0/2701748881 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:38.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff33aa00640 1 --2- 172.21.15.62:0/2701748881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4ab5a0 0x555f0b4d0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff33b400640 1 -- 172.21.15.62:0/2701748881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4ab5a0 msgr2=0x555f0b4d0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff332000640 1 --2- 172.21.15.62:0/2701748881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 0x555f0b4c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff33b400640 1 --2- 172.21.15.62:0/2701748881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4ab5a0 0x555f0b4d0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff33b400640 1 -- 172.21.15.62:0/2701748881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 msgr2=0x555f0b4c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff33b400640 1 --2- 172.21.15.62:0/2701748881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 0x555f0b4c9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff33b400640 1 -- 172.21.15.62:0/2701748881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f0b4d5250 con 0x555f0b4f5520 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff33aa00640 1 --2- 172.21.15.62:0/2701748881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4ab5a0 0x555f0b4d0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff33b400640 1 --2- 172.21.15.62:0/2701748881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4f5520 0x555f0b4c9cc0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff33405fb90 tx=0x7ff334054140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5abba2083e5ca25 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff330c00640 1 -- 172.21.15.62:0/2701748881 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3340600c0 con 0x555f0b4f5520 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff330c00640 1 -- 172.21.15.62:0/2701748881 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff334056070 con 0x555f0b4f5520 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.982+0000 7ff330c00640 1 -- 172.21.15.62:0/2701748881 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3340571c0 con 0x555f0b4f5520 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.983+0000 7ff344b90740 1 -- 172.21.15.62:0/2701748881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4f5520 msgr2=0x555f0b4c9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.983+0000 7ff344b90740 1 --2- 172.21.15.62:0/2701748881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4f5520 0x555f0b4c9cc0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff33405fb90 tx=0x7ff334054140 comp rx=0 tx=0).stop 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.983+0000 7ff344b90740 1 -- 172.21.15.62:0/2701748881 shutdown_connections 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.983+0000 7ff344b90740 1 --2- 172.21.15.62:0/2701748881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4f5520 0x555f0b4c9cc0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.983+0000 7ff344b90740 1 --2- 172.21.15.62:0/2701748881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 0x555f0b4c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.983+0000 7ff344b90740 1 --2- 172.21.15.62:0/2701748881 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4ab5a0 0x555f0b4d0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.983+0000 7ff344b90740 1 -- 172.21.15.62:0/2701748881 >> 172.21.15.62:0/2701748881 conn(0x555f0b4a7a40 msgr2=0x555f0b4a6d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:38.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.983+0000 7ff344b90740 1 -- 172.21.15.62:0/2701748881 shutdown_connections 2024-03-23T11:22:38.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.983+0000 7ff344b90740 1 -- 172.21.15.62:0/2701748881 wait complete. 2024-03-23T11:22:38.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.984+0000 7ff344b90740 1 -- start start 2024-03-23T11:22:38.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.984+0000 7ff344b90740 1 Processor -- start 2024-03-23T11:22:38.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff344b90740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b557f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff344b90740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 0x555f0b4c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff344b90740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff344b90740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0b4f35e0 con 0x555f0b4ab5a0 2024-03-23T11:22:38.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff344b90740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0b4c9af0 con 0x555f0b4f4ee0 2024-03-23T11:22:38.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff344b90740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f0b4c9d40 con 0x555f0b4f5520 2024-03-23T11:22:38.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff344b90740 10 client.0 _ll_register_callbacks cb 0x555f0b4f3830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:22:38.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff33aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b557f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff33aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b557f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55922/0 (socket says 172.21.15.62:55922) 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff33aa00640 1 -- 172.21.15.62:0/2447274464 learned_addr learned my addr 172.21.15.62:0/2447274464 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff33aa00640 1 -- 172.21.15.62:0/2447274464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 msgr2=0x555f0b4c7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff33aa00640 1 --2- 172.21.15.62:0/2447274464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff33aa00640 1 -- 172.21.15.62:0/2447274464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 msgr2=0x555f0b4c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff33aa00640 1 --2- 172.21.15.62:0/2447274464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 0x555f0b4c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.985+0000 7ff33aa00640 1 -- 172.21.15.62:0/2447274464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33c159d60 con 0x555f0b4ab5a0 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.986+0000 7ff33aa00640 1 --2- 172.21.15.62:0/2447274464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b557f60 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff32c03c010 tx=0x7ff32c031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cefbec6abed03b16 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.986+0000 7ff328c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.986+0000 7ff328c00640 1 -- 172.21.15.62:0/2447274464 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c05e620 con 0x555f0b4ab5a0 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.986+0000 7ff328c00640 1 -- 172.21.15.62:0/2447274464 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff32c03f050 con 0x555f0b4ab5a0 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.986+0000 7ff344b90740 1 -- 172.21.15.62:0/2447274464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff324004180 con 0x555f0b4ab5a0 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.986+0000 7ff328c00640 1 -- 172.21.15.62:0/2447274464 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff32c05e780 con 0x555f0b4ab5a0 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.986+0000 7ff344b90740 10 client.6216 fetch_fsmap learned FSMap version 33 2024-03-23T11:22:38.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.986+0000 7ff344b90740 1 -- 172.21.15.62:0/2447274464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x555f0b3b1ad0 con 0x555f0b4ab5a0 2024-03-23T11:22:38.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.987+0000 7ff328c00640 1 -- 172.21.15.62:0/2447274464 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff324004180 con 0x555f0b4ab5a0 2024-03-23T11:22:38.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.987+0000 7ff328c00640 1 -- 172.21.15.62:0/2447274464 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff32c04e040 con 0x555f0b4ab5a0 2024-03-23T11:22:38.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.987+0000 7ff344b90740 10 client.6216 fetch_fsmap finished waiting for FSMap version 33 2024-03-23T11:22:38.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.987+0000 7ff344b90740 10 client.6216 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:22:38.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.987+0000 7ff344b90740 20 client.6216 populate_metadata read hostname 'smithi062' 2024-03-23T11:22:38.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.987+0000 7ff344b90740 1 --2- 172.21.15.62:0/2447274464 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x555f0b4ce6b0 0x555f0b4ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:38.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.987+0000 7ff344b90740 4 client.6216 mds_command: new command op to 6081 tid=0[{"prefix": "status"}] 2024-03-23T11:22:38.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.987+0000 7ff344b90740 1 -- 172.21.15.62:0/2447274464 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555f0b1053c0 con 0x555f0b4ce6b0 2024-03-23T11:22:38.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.988+0000 7ff332000640 1 --2- 172.21.15.62:0/2447274464 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x555f0b4ce6b0 0x555f0b4ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:38.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.988+0000 7ff332000640 1 --2- 172.21.15.62:0/2447274464 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x555f0b4ce6b0 0x555f0b4ceab0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c10f66652fdc9eb7 server_cookie=2b27256a8176ffd9 in_seq=0 out_seq=0 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.988+0000 7ff328c00640 10 client.6216 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff328c00640 1 -- 172.21.15.62:0/2447274464 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x555f0b1053c0 con 0x555f0b4ce6b0 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff328c00640 10 client.6216 handle_command_reply: tid=0 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff344b90740 1 client.6216 shutdown 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff344b90740 1 -- 172.21.15.62:0/2447274464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 msgr2=0x555f0b557f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff344b90740 1 --2- 172.21.15.62:0/2447274464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b557f60 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff32c03c010 tx=0x7ff32c031000 comp rx=0 tx=0).stop 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff344b90740 1 -- 172.21.15.62:0/2447274464 shutdown_connections 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff344b90740 1 --2- 172.21.15.62:0/2447274464 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x555f0b4ce6b0 0x555f0b4ceab0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff344b90740 1 --2- 172.21.15.62:0/2447274464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555f0b4f5520 0x555f0b4c7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff344b90740 1 --2- 172.21.15.62:0/2447274464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f0b4f4ee0 0x555f0b4c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff344b90740 1 --2- 172.21.15.62:0/2447274464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555f0b4ab5a0 0x555f0b557f60 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff328c00640 0 client.6216 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.989+0000 7ff344b90740 1 -- 172.21.15.62:0/2447274464 >> 172.21.15.62:0/2447274464 conn(0x555f0b4a7a40 msgr2=0x555f0b4abbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:38.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff344b90740 1 -- 172.21.15.62:0/2447274464 shutdown_connections 2024-03-23T11:22:38.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff344b90740 1 -- 172.21.15.62:0/2447274464 wait complete. 2024-03-23T11:22:38.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff344b90740 20 client.6216 trim_cache size 0 max 16384 2024-03-23T11:22:38.990 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:22:38.990 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:22:38.990 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:22:38.990 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 6081, 2024-03-23T11:22:38.990 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:22:38.990 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:22:38.990 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:22:38.990 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 7.257554743, 2024-03-23T11:22:38.991 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 33, 2024-03-23T11:22:38.991 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 62, 2024-03-23T11:22:38.991 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 62, 2024-03-23T11:22:38.991 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 12.699109721999999 2024-03-23T11:22:38.991 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff33be00640 1 -- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff31806a7a0 msgr2=0x7ff31806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff33be00640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff31806a7a0 0x7ff31806cc60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff32c036000 tx=0x7ff32c03c040 comp rx=0 tx=0).stop 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff33be00640 1 -- 172.21.15.62:0/3577705974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff33c143450 msgr2=0x7ff33c1f5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff33be00640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff33c143450 0x7ff33c1f5410 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff33400b970 tx=0x7ff33400be30 comp rx=0 tx=0).stop 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff33be00640 1 -- 172.21.15.62:0/3577705974 shutdown_connections 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff33be00640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff31806a7a0 0x7ff31806cc60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff33be00640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff33c143450 0x7ff33c1f5410 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff33be00640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff33c0caff0 0x7ff33c158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff33be00640 1 --2- 172.21.15.62:0/3577705974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff33c069c60 0x7ff33c0c3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff33be00640 1 -- 172.21.15.62:0/3577705974 >> 172.21.15.62:0/3577705974 conn(0x7ff33c0b0090 msgr2=0x7ff33c0b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.990+0000 7ff33be00640 1 -- 172.21.15.62:0/3577705974 shutdown_connections 2024-03-23T11:22:38.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:38.991+0000 7ff33be00640 1 -- 172.21.15.62:0/3577705974 wait complete. 2024-03-23T11:22:39.006 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:22:39.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.220+0000 7f60e9a00640 1 Processor -- start 2024-03-23T11:22:39.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.221+0000 7f60e9a00640 1 -- start start 2024-03-23T11:22:39.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.222+0000 7f60e9a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f60e4070090 0x7f60e4070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:39.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.222+0000 7f60e9a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60e40b2820 0x7f60e40b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:39.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.222+0000 7f60e9a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e40b31b0 con 0x7f60e4143dd0 2024-03-23T11:22:39.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.222+0000 7f60e9a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e4070a60 con 0x7f60e40b2820 2024-03-23T11:22:39.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.222+0000 7f60e9a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e40cd140 con 0x7f60e4070090 2024-03-23T11:22:39.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.222+0000 7f60e3e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f60e4143dd0 0x7f60e415abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59654/0 (socket says 172.21.15.62:59654) 2024-03-23T11:22:39.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.222+0000 7f60e3e00640 1 -- 172.21.15.62:0/3057151506 learned_addr learned my addr 172.21.15.62:0/3057151506 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:39.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.222+0000 7f60e9000640 1 --2- 172.21.15.62:0/3057151506 >> v2:172.21.15.62:3300/0 conn(0x7f60e40b2820 0x7f60e40b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:39.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.223+0000 7f60db400640 1 --2- 172.21.15.62:0/3057151506 >> v2:172.21.15.120:3300/0 conn(0x7f60e4070090 0x7f60e4070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:39.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.223+0000 7f60e9000640 1 -- 172.21.15.62:0/3057151506 >> v2:172.21.15.120:3300/0 conn(0x7f60e4070090 msgr2=0x7f60e4070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.223+0000 7f60e9000640 1 --2- 172.21.15.62:0/3057151506 >> v2:172.21.15.120:3300/0 conn(0x7f60e4070090 0x7f60e4070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.223+0000 7f60e9000640 1 -- 172.21.15.62:0/3057151506 >> v1:172.21.15.62:6789/0 conn(0x7f60e4143dd0 legacy=0x7f60e415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.223+0000 7f60e9000640 1 -- 172.21.15.62:0/3057151506 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e40cd3e0 con 0x7f60e40b2820 2024-03-23T11:22:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.223+0000 7f60e3400640 1 -- 172.21.15.62:0/3057151506 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494812893 0 0) 0x7f60e40b31b0 con 0x7f60e4143dd0 2024-03-23T11:22:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.224+0000 7f60e9000640 1 --2- 172.21.15.62:0/3057151506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e40b2820 0x7f60e40b2c70 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f60e4071530 tx=0x7f60dc02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b98e75318c1509 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.224+0000 7f60e3400640 1 -- 172.21.15.62:0/3057151506 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc02cc70 con 0x7f60e40b2820 2024-03-23T11:22:39.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.224+0000 7f60e3400640 1 -- 172.21.15.62:0/3057151506 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60dc02cdd0 con 0x7f60e40b2820 2024-03-23T11:22:39.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.224+0000 7f60e3400640 1 -- 172.21.15.62:0/3057151506 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc034660 con 0x7f60e40b2820 2024-03-23T11:22:39.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.225+0000 7f60e9a00640 1 -- 172.21.15.62:0/3057151506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e40b2820 msgr2=0x7f60e40b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.225+0000 7f60e9a00640 1 --2- 172.21.15.62:0/3057151506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e40b2820 0x7f60e40b2c70 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f60e4071530 tx=0x7f60dc02bec0 comp rx=0 tx=0).stop 2024-03-23T11:22:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.225+0000 7f60e9a00640 1 -- 172.21.15.62:0/3057151506 shutdown_connections 2024-03-23T11:22:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.225+0000 7f60e9a00640 1 --2- 172.21.15.62:0/3057151506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e40b2820 0x7f60e40b2c70 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.225+0000 7f60e9a00640 1 --2- 172.21.15.62:0/3057151506 >> v2:172.21.15.120:3300/0 conn(0x7f60e4070090 0x7f60e4070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.225+0000 7f60e9a00640 1 -- 172.21.15.62:0/3057151506 >> 172.21.15.62:0/3057151506 conn(0x7f60e4066050 msgr2=0x7f60e4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.225+0000 7f60e9a00640 1 -- 172.21.15.62:0/3057151506 shutdown_connections 2024-03-23T11:22:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.225+0000 7f60e9a00640 1 -- 172.21.15.62:0/3057151506 wait complete. 2024-03-23T11:22:39.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.227+0000 7f60e9a00640 1 Processor -- start 2024-03-23T11:22:39.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.227+0000 7f60e9a00640 1 -- start start 2024-03-23T11:22:39.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.228+0000 7f60e9a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60e4070090 0x7f60e415a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:39.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.228+0000 7f60e9a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e40b2820 0x7f60e40c3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:39.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.228+0000 7f60e9a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e4143dd0 0x7f60e40b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:39.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.228+0000 7f60e9a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e415ae10 con 0x7f60e4143dd0 2024-03-23T11:22:39.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.228+0000 7f60e9a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4132cb0 con 0x7f60e4070090 2024-03-23T11:22:39.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.228+0000 7f60e9a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4132f90 con 0x7f60e40b2820 2024-03-23T11:22:39.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.228+0000 7f60e9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e4143dd0 0x7f60e40b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:39.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.228+0000 7f60e9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e4143dd0 0x7f60e40b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55942/0 (socket says 172.21.15.62:55942) 2024-03-23T11:22:39.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.228+0000 7f60e9000640 1 -- 172.21.15.62:0/812827970 learned_addr learned my addr 172.21.15.62:0/812827970 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:39.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.229+0000 7f60e3e00640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60e4070090 0x7f60e415a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:39.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.229+0000 7f60db400640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e40b2820 0x7f60e40c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:39.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.229+0000 7f60e9000640 1 -- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e40b2820 msgr2=0x7f60e40c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:39.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.229+0000 7f60e9000640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e40b2820 0x7f60e40c3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:39.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.229+0000 7f60e9000640 1 -- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60e4070090 msgr2=0x7f60e415a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:39.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.229+0000 7f60e9000640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60e4070090 0x7f60e415a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:39.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.229+0000 7f60e9000640 1 -- 172.21.15.62:0/812827970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc009a70 con 0x7f60e4143dd0 2024-03-23T11:22:39.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.229+0000 7f60e9000640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e4143dd0 0x7f60e40b0000 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f60dc009e80 tx=0x7f60dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:39.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.229+0000 7f60e0c00640 1 -- 172.21.15.62:0/812827970 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc040650 con 0x7f60e4143dd0 2024-03-23T11:22:39.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.229+0000 7f60e0c00640 1 -- 172.21.15.62:0/812827970 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60dc0407d0 con 0x7f60e4143dd0 2024-03-23T11:22:39.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.229+0000 7f60e0c00640 1 -- 172.21.15.62:0/812827970 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc040b80 con 0x7f60e4143dd0 2024-03-23T11:22:39.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.230+0000 7f60e9a00640 1 -- 172.21.15.62:0/812827970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e4133270 con 0x7f60e4143dd0 2024-03-23T11:22:39.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.230+0000 7f60e9a00640 1 -- 172.21.15.62:0/812827970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e4133830 con 0x7f60e4143dd0 2024-03-23T11:22:39.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.231+0000 7f60e9a00640 1 -- 172.21.15.62:0/812827970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e4066bb0 con 0x7f60e4143dd0 2024-03-23T11:22:39.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.232+0000 7f60e0c00640 1 -- 172.21.15.62:0/812827970 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f60dc040ce0 con 0x7f60e4143dd0 2024-03-23T11:22:39.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.233+0000 7f60e0c00640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60c006a7a0 0x7f60c006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:39.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.233+0000 7f60e0c00640 1 -- 172.21.15.62:0/812827970 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60dc0abab0 con 0x7f60e4143dd0 2024-03-23T11:22:39.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.233+0000 7f60e3e00640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60c006a7a0 0x7f60c006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:39.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.234+0000 7f60e3e00640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60c006a7a0 0x7f60c006cc60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f60d4000f60 tx=0x7f60d4007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:39.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.237+0000 7f60e0c00640 1 -- 172.21.15.62:0/812827970 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f60dc077080 con 0x7f60e4143dd0 2024-03-23T11:22:39.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.478+0000 7f60e9a00640 1 -- 172.21.15.62:0/812827970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f60e41339f0 con 0x7f60e4143dd0 2024-03-23T11:22:39.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.479+0000 7f60e0c00640 1 -- 172.21.15.62:0/812827970 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7f60dc039060 con 0x7f60e4143dd0 2024-03-23T11:22:39.476 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:22:39.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.482+0000 7f60e9a00640 1 -- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60c006a7a0 msgr2=0x7f60c006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:39.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.482+0000 7f60e9a00640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60c006a7a0 0x7f60c006cc60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f60d4000f60 tx=0x7f60d4007c00 comp rx=0 tx=0).stop 2024-03-23T11:22:39.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.482+0000 7f60e9a00640 1 -- 172.21.15.62:0/812827970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e4143dd0 msgr2=0x7f60e40b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:39.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.482+0000 7f60e9a00640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e4143dd0 0x7f60e40b0000 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f60dc009e80 tx=0x7f60dc004a40 comp rx=0 tx=0).stop 2024-03-23T11:22:39.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.482+0000 7f60e9a00640 1 -- 172.21.15.62:0/812827970 shutdown_connections 2024-03-23T11:22:39.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.482+0000 7f60e9a00640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60c006a7a0 0x7f60c006cc60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:39.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.482+0000 7f60e9a00640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60e4143dd0 0x7f60e40b0000 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:39.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.482+0000 7f60e9a00640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e40b2820 0x7f60e40c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:39.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.482+0000 7f60e9a00640 1 --2- 172.21.15.62:0/812827970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60e4070090 0x7f60e415a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:39.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.482+0000 7f60e9a00640 1 -- 172.21.15.62:0/812827970 >> 172.21.15.62:0/812827970 conn(0x7f60e4066050 msgr2=0x7f60e406ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:39.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.482+0000 7f60e9a00640 1 -- 172.21.15.62:0/812827970 shutdown_connections 2024-03-23T11:22:39.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:39.483+0000 7f60e9a00640 1 -- 172.21.15.62:0/812827970 wait complete. 2024-03-23T11:22:39.499 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:22:39.499 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:22:39.499 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:22:39.499 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:22:39.499 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:22:39.499 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:22:39.499 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:22:39.500 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:22:39.500 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:22:39.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:39.500 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:22:39.554 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:22:39.554 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:22:39.554 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:22:39.554 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:22:39.554 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:22:39.554 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:22:39.554 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:22:39.554 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:22:39.554 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:22:39.554 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:22:39.554 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42397sec preferred_lft 42397sec 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:22:39.555 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:22:39.556 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:22:39.556 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:22:39.556 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:22:39.556 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:22:39.556 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:22:39.556 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:22:39.556 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:22:39.556 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:22:39.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:39.556 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:22:39.646 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:22:39.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:39.651 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:22:39.707 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:22:39.707 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:22:39.707 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:22:39.707 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:22:39.707 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:22:39.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:39.708 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:22:39.708 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:22:39.709 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:22:39.709 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:22:39.709 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:22:39.709 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:22:39.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:39.898 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:22:39.952 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:22:39.952 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:22:39.953 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:22:39.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:39.953 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:22:40.051 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:22:40.051 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:22:40.051 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:22:40.106 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:22:40.107 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:22:40.107 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:22:40.158 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:22:40.261 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:22:40.261 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:22:40.352 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:22:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:22:40.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:40.357 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:22:40.411 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:22:40.411 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:22:40.416 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:22:40.493 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:22:40.494 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:22:40.564 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:22:40.567 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:22:40.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:40.568 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:22:40.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:22:40.605+0000 7fcaa424b480 -1 init, newargv = 0x55a6f1760e20 newargc=15 2024-03-23T11:22:40.603 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[56728]: starting ceph client 2024-03-23T11:22:40.614 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[56728]: starting fuse 2024-03-23T11:22:41.714 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:22:41.820 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:22:41.821 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:22:41.921 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:22:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:22:41.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:41.926 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:22:41.982 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:22:41.982 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:22:41.983 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:22:41.984 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:22:42.174 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.56728.asok 2024-03-23T11:22:42.353 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56728.asok 2024-03-23T11:22:42.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:42.353 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56728.asok status 2024-03-23T11:22:42.601 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:22:42.601 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:22:42.601 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:22:42.601 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "56728", 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 6228, 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 6228 2024-03-23T11:22:42.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2464005726 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2464005726 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.6228 192.168.0.1:0/2464005726", 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/2464005726", 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:22:42.603 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 33, 2024-03-23T11:22:42.604 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 62, 2024-03-23T11:22:42.604 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:22:42.604 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:22:42.604 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:22:42.604 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:22:42.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:42.617 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:22:42.617 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:22:42.674 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:22:42.675 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:22:42.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:42.675 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:22:42.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:42.774 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_test) 2024-03-23T11:22:42.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:22:42.848 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir_test/file) 2024-03-23T11:22:42.914 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:22:42.914 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:22:42.915 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:22:42.915 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:22:42.915 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_test/file").st_ino) 2024-03-23T11:22:42.915 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:22:43.061 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627777 2024-03-23T11:22:43.242 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:22:43.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.520+0000 7fd7b6e00640 1 Processor -- start 2024-03-23T11:22:43.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.520+0000 7fd7b6e00640 1 -- start start 2024-03-23T11:22:43.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.522+0000 7fd7b6e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd7b0143dd0 0x7fd7b015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:43.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.522+0000 7fd7b6e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7b0070090 0x7fd7b00704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:43.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.522+0000 7fd7b6e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b0070a30 con 0x7fd7b00b2820 2024-03-23T11:22:43.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.522+0000 7fd7b6e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b00cd140 con 0x7fd7b0143dd0 2024-03-23T11:22:43.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.522+0000 7fd7b6e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b00cd3e0 con 0x7fd7b0070090 2024-03-23T11:22:43.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.523+0000 7fd7b5a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd7b0143dd0 0x7fd7b015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:43.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.523+0000 7fd7b6400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd7b00b2820 0x7fd7b00b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59664/0 (socket says 172.21.15.62:59664) 2024-03-23T11:22:43.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.523+0000 7fd7b6400640 1 -- 172.21.15.62:0/4206010165 learned_addr learned my addr 172.21.15.62:0/4206010165 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:43.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.523+0000 7fd7b5000640 1 --2- 172.21.15.62:0/4206010165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7b0070090 0x7fd7b00704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:43.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.523+0000 7fd7b5a00640 1 -- 172.21.15.62:0/4206010165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7b0070090 msgr2=0x7fd7b00704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:43.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.523+0000 7fd7b5a00640 1 --2- 172.21.15.62:0/4206010165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7b0070090 0x7fd7b00704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:43.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.523+0000 7fd7b5a00640 1 -- 172.21.15.62:0/4206010165 >> v1:172.21.15.62:6789/0 conn(0x7fd7b00b2820 legacy=0x7fd7b00b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:43.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.523+0000 7fd7b5a00640 1 -- 172.21.15.62:0/4206010165 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b00cd6c0 con 0x7fd7b0143dd0 2024-03-23T11:22:43.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.523+0000 7fd7afe00640 1 -- 172.21.15.62:0/4206010165 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622085390 0 0) 0x7fd7b0070a30 con 0x7fd7b00b2820 2024-03-23T11:22:43.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.524+0000 7fd7b5a00640 1 --2- 172.21.15.62:0/4206010165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7b0143dd0 0x7fd7b015abe0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40097a0 tx=0x7fd7a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9ff88fec6fe9aa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:43.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.525+0000 7fd7afe00640 1 -- 172.21.15.62:0/4206010165 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4031830 con 0x7fd7b0143dd0 2024-03-23T11:22:43.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.525+0000 7fd7afe00640 1 -- 172.21.15.62:0/4206010165 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a4031990 con 0x7fd7b0143dd0 2024-03-23T11:22:43.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.525+0000 7fd7afe00640 1 -- 172.21.15.62:0/4206010165 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4031d20 con 0x7fd7b0143dd0 2024-03-23T11:22:43.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.525+0000 7fd7b6e00640 1 -- 172.21.15.62:0/4206010165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7b0143dd0 msgr2=0x7fd7b015abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:43.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.525+0000 7fd7b6e00640 1 --2- 172.21.15.62:0/4206010165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7b0143dd0 0x7fd7b015abe0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40097a0 tx=0x7fd7a402b8d0 comp rx=0 tx=0).stop 2024-03-23T11:22:43.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.526+0000 7fd7b6e00640 1 -- 172.21.15.62:0/4206010165 shutdown_connections 2024-03-23T11:22:43.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.526+0000 7fd7b6e00640 1 --2- 172.21.15.62:0/4206010165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7b0070090 0x7fd7b00704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:43.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.526+0000 7fd7b6e00640 1 --2- 172.21.15.62:0/4206010165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7b0143dd0 0x7fd7b015abe0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:43.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.526+0000 7fd7b6e00640 1 -- 172.21.15.62:0/4206010165 >> 172.21.15.62:0/4206010165 conn(0x7fd7b0066050 msgr2=0x7fd7b0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:43.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.526+0000 7fd7b6e00640 1 -- 172.21.15.62:0/4206010165 shutdown_connections 2024-03-23T11:22:43.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.526+0000 7fd7b6e00640 1 -- 172.21.15.62:0/4206010165 wait complete. 2024-03-23T11:22:43.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.527+0000 7fd7b6e00640 1 Processor -- start 2024-03-23T11:22:43.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.527+0000 7fd7b6e00640 1 -- start start 2024-03-23T11:22:43.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.528+0000 7fd7b6e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7b0070090 0x7fd7b006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:43.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.528+0000 7fd7b6e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7b00b2820 0x7fd7b00c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:43.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.528+0000 7fd7b6e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7b0143dd0 0x7fd7b00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:43.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.528+0000 7fd7b6e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b01f2c30 con 0x7fd7b0070090 2024-03-23T11:22:43.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.528+0000 7fd7b6e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b006c230 con 0x7fd7b0143dd0 2024-03-23T11:22:43.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.528+0000 7fd7b6e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b006c510 con 0x7fd7b00b2820 2024-03-23T11:22:43.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.529+0000 7fd7b5a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7b0070090 0x7fd7b006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:43.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.529+0000 7fd7b5a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7b0070090 0x7fd7b006cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55964/0 (socket says 172.21.15.62:55964) 2024-03-23T11:22:43.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.529+0000 7fd7b5a00640 1 -- 172.21.15.62:0/3368867824 learned_addr learned my addr 172.21.15.62:0/3368867824 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:43.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.529+0000 7fd7b5000640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7b00b2820 0x7fd7b00c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:43.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.529+0000 7fd7b6400640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7b0143dd0 0x7fd7b00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:43.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.529+0000 7fd7b5a00640 1 -- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7b00b2820 msgr2=0x7fd7b00c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.529+0000 7fd7b5a00640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7b00b2820 0x7fd7b00c5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.529+0000 7fd7b5a00640 1 -- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7b0143dd0 msgr2=0x7fd7b00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.529+0000 7fd7b5a00640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7b0143dd0 0x7fd7b00ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.529+0000 7fd7b5a00640 1 -- 172.21.15.62:0/3368867824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a4009400 con 0x7fd7b0070090 2024-03-23T11:22:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.530+0000 7fd7b5a00640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7b0070090 0x7fd7b006cff0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4000c00 tx=0x7fd7a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.530+0000 7fd7ad600640 1 -- 172.21.15.62:0/3368867824 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a403aa60 con 0x7fd7b0070090 2024-03-23T11:22:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.530+0000 7fd7ad600640 1 -- 172.21.15.62:0/3368867824 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a403abc0 con 0x7fd7b0070090 2024-03-23T11:22:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.530+0000 7fd7ad600640 1 -- 172.21.15.62:0/3368867824 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a40443f0 con 0x7fd7b0070090 2024-03-23T11:22:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.530+0000 7fd7b6e00640 1 -- 172.21.15.62:0/3368867824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b006c7f0 con 0x7fd7b0070090 2024-03-23T11:22:43.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.530+0000 7fd7b6e00640 1 -- 172.21.15.62:0/3368867824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b00acdd0 con 0x7fd7b0070090 2024-03-23T11:22:43.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.531+0000 7fd7b6e00640 1 -- 172.21.15.62:0/3368867824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b0066bb0 con 0x7fd7b0070090 2024-03-23T11:22:43.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.532+0000 7fd7ad600640 1 -- 172.21.15.62:0/3368867824 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd7a404d450 con 0x7fd7b0070090 2024-03-23T11:22:43.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.534+0000 7fd7ad600640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd79006a7a0 0x7fd79006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:43.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.534+0000 7fd7ad600640 1 -- 172.21.15.62:0/3368867824 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7a40aaff0 con 0x7fd7b0070090 2024-03-23T11:22:43.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.534+0000 7fd7b5000640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd79006a7a0 0x7fd79006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:43.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.535+0000 7fd7b5000640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd79006a7a0 0x7fd79006cc60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0141c60 tx=0x7fd79c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:43.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.539+0000 7fd7ad600640 1 -- 172.21.15.62:0/3368867824 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd7a4076520 con 0x7fd7b0070090 2024-03-23T11:22:43.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.777+0000 7fd7b6e00640 1 -- 172.21.15.62:0/3368867824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7b0141a20 con 0x7fd7b0070090 2024-03-23T11:22:43.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.778+0000 7fd7ad600640 1 -- 172.21.15.62:0/3368867824 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fd7a4035020 con 0x7fd7b0070090 2024-03-23T11:22:43.775 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:43.775 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3287045135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3287045135},{"type":"v1","addr":"172.21.15.120:6835","nonce":3287045135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6066,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/2859430143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2859430143},{"type":"v1","addr":"172.21.15.62:6834","nonce":2859430143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2005768060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2005768060},{"type":"v1","addr":"172.21.15.62:6835","nonce":2005768060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:22:31.725865+0000","modified":"2024-03-23T11:22:35.259834+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6081},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6081":{"gid":6081,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/4158450746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":4158450746},{"type":"v1","addr":"172.21.15.120:6837","nonce":4158450746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T11:22:43.776 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 33 2024-03-23T11:22:43.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 -- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd79006a7a0 msgr2=0x7fd79006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:43.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd79006a7a0 0x7fd79006cc60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0141c60 tx=0x7fd79c008040 comp rx=0 tx=0).stop 2024-03-23T11:22:43.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 -- 172.21.15.62:0/3368867824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7b0070090 msgr2=0x7fd7b006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:43.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7b0070090 0x7fd7b006cff0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4000c00 tx=0x7fd7a40046c0 comp rx=0 tx=0).stop 2024-03-23T11:22:43.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 -- 172.21.15.62:0/3368867824 shutdown_connections 2024-03-23T11:22:43.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd79006a7a0 0x7fd79006cc60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:43.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7b0143dd0 0x7fd7b00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:43.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7b00b2820 0x7fd7b00c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:43.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 --2- 172.21.15.62:0/3368867824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7b0070090 0x7fd7b006cff0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:43.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 -- 172.21.15.62:0/3368867824 >> 172.21.15.62:0/3368867824 conn(0x7fd7b0066050 msgr2=0x7fd7b00704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:43.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 -- 172.21.15.62:0/3368867824 shutdown_connections 2024-03-23T11:22:43.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:43.781+0000 7fd7b6e00640 1 -- 172.21.15.62:0/3368867824 wait complete. 2024-03-23T11:22:43.797 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-23T11:22:44.049 INFO:teuthology.orchestra.run.smithi120.stdout:{"message":"","return_code":0} 2024-03-23T11:22:44.050 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-23T11:22:44.050 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-23T11:22:44.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.171+0000 7f133bbfef40 1 Processor -- start 2024-03-23T11:22:44.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.171+0000 7f133bbfef40 1 -- start start 2024-03-23T11:22:44.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.172+0000 7f133bbfef40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 0x56053e77bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.172+0000 7f133bbfef40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053e780c20 0x56053e77c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.172+0000 7f133bbfef40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56053e781540 0x56053e6d6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.172+0000 7f133bbfef40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053e5f1800 con 0x56053e6dc910 2024-03-23T11:22:44.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.172+0000 7f133bbfef40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053e5f14a0 con 0x56053e780c20 2024-03-23T11:22:44.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.172+0000 7f133bbfef40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56053e5e8110 con 0x56053e781540 2024-03-23T11:22:44.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.172+0000 7f1338e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 0x56053e77bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.172+0000 7f1338e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 0x56053e77bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55970/0 (socket says 172.21.15.62:55970) 2024-03-23T11:22:44.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.172+0000 7f1338e00640 1 -- 172.21.15.62:0/4116928709 learned_addr learned my addr 172.21.15.62:0/4116928709 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.173+0000 7f1330400640 1 --2- 172.21.15.62:0/4116928709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053e780c20 0x56053e77c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.173+0000 7f1338e00640 1 -- 172.21.15.62:0/4116928709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56053e781540 msgr2=0x56053e6d6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.173+0000 7f1339800640 1 --2- 172.21.15.62:0/4116928709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56053e781540 0x56053e6d6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.173+0000 7f1338e00640 1 --2- 172.21.15.62:0/4116928709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56053e781540 0x56053e6d6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.173+0000 7f1338e00640 1 -- 172.21.15.62:0/4116928709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053e780c20 msgr2=0x56053e77c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.173+0000 7f1338e00640 1 --2- 172.21.15.62:0/4116928709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053e780c20 0x56053e77c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.173+0000 7f1338e00640 1 -- 172.21.15.62:0/4116928709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132800a1c0 con 0x56053e6dc910 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.173+0000 7f1330400640 1 --2- 172.21.15.62:0/4116928709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053e780c20 0x56053e77c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.173+0000 7f1339800640 1 --2- 172.21.15.62:0/4116928709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56053e781540 0x56053e6d6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.173+0000 7f1338e00640 1 --2- 172.21.15.62:0/4116928709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 0x56053e77bd90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f132800a9d0 tx=0x7f132800ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27203c484e2af03e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.173+0000 7f1333400640 1 -- 172.21.15.62:0/4116928709 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132800a4a0 con 0x56053e6dc910 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.174+0000 7f1333400640 1 -- 172.21.15.62:0/4116928709 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132800dae0 con 0x56053e6dc910 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.174+0000 7f1333400640 1 -- 172.21.15.62:0/4116928709 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132800de90 con 0x56053e6dc910 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.174+0000 7f133bbfef40 1 -- 172.21.15.62:0/4116928709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 msgr2=0x56053e77bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.174+0000 7f133bbfef40 1 --2- 172.21.15.62:0/4116928709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 0x56053e77bd90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f132800a9d0 tx=0x7f132800ae90 comp rx=0 tx=0).stop 2024-03-23T11:22:44.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.174+0000 7f133bbfef40 1 -- 172.21.15.62:0/4116928709 shutdown_connections 2024-03-23T11:22:44.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.174+0000 7f133bbfef40 1 --2- 172.21.15.62:0/4116928709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56053e781540 0x56053e6d6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.174+0000 7f133bbfef40 1 --2- 172.21.15.62:0/4116928709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053e780c20 0x56053e77c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.174+0000 7f133bbfef40 1 --2- 172.21.15.62:0/4116928709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 0x56053e77bd90 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.174+0000 7f133bbfef40 1 -- 172.21.15.62:0/4116928709 >> 172.21.15.62:0/4116928709 conn(0x56053e6f6ff0 msgr2=0x56053e6d81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.174+0000 7f133bbfef40 1 -- 172.21.15.62:0/4116928709 shutdown_connections 2024-03-23T11:22:44.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.174+0000 7f133bbfef40 1 -- 172.21.15.62:0/4116928709 wait complete. 2024-03-23T11:22:44.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.175+0000 7f133bbfef40 1 Processor -- start 2024-03-23T11:22:44.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.175+0000 7f133bbfef40 1 -- start start 2024-03-23T11:22:44.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.176+0000 7f133bbfef40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 0x56053e75f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.176+0000 7f133bbfef40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053e780c20 0x56053e75c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.176+0000 7f133bbfef40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56053e781540 0x56053e75c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.176+0000 7f133bbfef40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053e764030 con 0x56053e6dc910 2024-03-23T11:22:44.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.176+0000 7f133bbfef40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053e75e060 con 0x56053e780c20 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.176+0000 7f133bbfef40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56053e75cd30 con 0x56053e781540 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.176+0000 7f1338e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 0x56053e75f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.176+0000 7f1338e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 0x56053e75f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55986/0 (socket says 172.21.15.62:55986) 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.176+0000 7f1338e00640 1 -- 172.21.15.62:0/1230689836 learned_addr learned my addr 172.21.15.62:0/1230689836 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.177+0000 7f1330400640 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053e780c20 0x56053e75c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.177+0000 7f1338e00640 1 -- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56053e781540 msgr2=0x56053e75c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.177+0000 7f1339800640 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56053e781540 0x56053e75c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.177+0000 7f1338e00640 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56053e781540 0x56053e75c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.177+0000 7f1338e00640 1 -- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053e780c20 msgr2=0x56053e75c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.177+0000 7f1338e00640 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053e780c20 0x56053e75c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.177+0000 7f1338e00640 1 -- 172.21.15.62:0/1230689836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132800a600 con 0x56053e6dc910 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.177+0000 7f1330400640 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053e780c20 0x56053e75c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.177+0000 7f1338e00640 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 0x56053e75f550 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1328000f80 tx=0x7f13280033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.178+0000 7f1332000640 1 -- 172.21.15.62:0/1230689836 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132801b420 con 0x56053e6dc910 2024-03-23T11:22:44.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.178+0000 7f1332000640 1 -- 172.21.15.62:0/1230689836 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132801f070 con 0x56053e6dc910 2024-03-23T11:22:44.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.178+0000 7f133bbfef40 1 -- 172.21.15.62:0/1230689836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1320005d20 con 0x56053e6dc910 2024-03-23T11:22:44.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.178+0000 7f1332000640 1 -- 172.21.15.62:0/1230689836 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132801b7e0 con 0x56053e6dc910 2024-03-23T11:22:44.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.178+0000 7f133bbfef40 1 -- 172.21.15.62:0/1230689836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56053e600340 con 0x56053e6dc910 2024-03-23T11:22:44.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.180+0000 7f1332000640 1 -- 172.21.15.62:0/1230689836 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f132801b940 con 0x56053e6dc910 2024-03-23T11:22:44.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.181+0000 7f1332000640 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1314069180 0x7f131406b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.181+0000 7f1332000640 1 -- 172.21.15.62:0/1230689836 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13280897d0 con 0x56053e6dc910 2024-03-23T11:22:44.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.181+0000 7f133bbfef40 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x56053e75fa90 0x56053e77e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.181+0000 7f133bbfef40 1 -- 172.21.15.62:0/1230689836 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:1 9.7 9:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62) v8 -- 0x56053e760340 con 0x56053e75fa90 2024-03-23T11:22:44.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.182+0000 7f1330400640 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1314069180 0x7f131406b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.182+0000 7f1339800640 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x56053e75fa90 0x56053e77e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.183+0000 7f1339800640 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x56053e75fa90 0x56053e77e090 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.183+0000 7f1330400640 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1314069180 0x7f131406b640 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1320054000 tx=0x7f132004e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f1339800640 1 -- 172.21.15.62:0/1230689836 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (94) out=94b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+94 (crc 0 0 0) 0x7f133400f3a0 con 0x56053e75fa90 2024-03-23T11:22:44.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f133bbfef40 1 -- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x56053e75fa90 msgr2=0x56053e77e090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f133bbfef40 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x56053e75fa90 0x56053e77e090 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f133bbfef40 1 -- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1314069180 msgr2=0x7f131406b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f133bbfef40 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1314069180 0x7f131406b640 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1320054000 tx=0x7f132004e000 comp rx=0 tx=0).stop 2024-03-23T11:22:44.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f133bbfef40 1 -- 172.21.15.62:0/1230689836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 msgr2=0x56053e75f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f133bbfef40 1 --2- 172.21.15.62:0/1230689836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56053e6dc910 0x56053e75f550 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1328000f80 tx=0x7f13280033a0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f1339800640 1 -- 172.21.15.62:0/1230689836 reap_dead start 2024-03-23T11:22:44.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f133bbfef40 1 -- 172.21.15.62:0/1230689836 shutdown_connections 2024-03-23T11:22:44.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f133bbfef40 1 -- 172.21.15.62:0/1230689836 >> 172.21.15.62:0/1230689836 conn(0x56053e6f6ff0 msgr2=0x56053e6d8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f133bbfef40 1 -- 172.21.15.62:0/1230689836 shutdown_connections 2024-03-23T11:22:44.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.184+0000 7f133bbfef40 1 -- 172.21.15.62:0/1230689836 wait complete. 2024-03-23T11:22:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:Xfiledir_test 2024-03-23T11:22:44.186 DEBUG:teuthology.orchestra.run.smithi062:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-23T11:22:44.298 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:22:44.298 INFO:teuthology.orchestra.run.smithi062.stdout: "ino": 1099511627777, 2024-03-23T11:22:44.298 INFO:teuthology.orchestra.run.smithi062.stdout: "ancestors": [ 2024-03-23T11:22:44.298 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:44.298 INFO:teuthology.orchestra.run.smithi062.stdout: "dirino": 1099511627776, 2024-03-23T11:22:44.298 INFO:teuthology.orchestra.run.smithi062.stdout: "dname": "file", 2024-03-23T11:22:44.298 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 2 2024-03-23T11:22:44.298 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:22:44.298 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:22:44.298 INFO:teuthology.orchestra.run.smithi062.stdout: "dirino": 1, 2024-03-23T11:22:44.298 INFO:teuthology.orchestra.run.smithi062.stdout: "dname": "dir_test", 2024-03-23T11:22:44.299 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 4 2024-03-23T11:22:44.299 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:44.299 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-03-23T11:22:44.299 INFO:teuthology.orchestra.run.smithi062.stdout: "pool": 9, 2024-03-23T11:22:44.299 INFO:teuthology.orchestra.run.smithi062.stdout: "old_pools": [] 2024-03-23T11:22:44.299 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:22:44.299 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:22:44.303 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'The backtrace is corrupted' 2024-03-23T11:22:44.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.409+0000 7fc67e340f40 1 Processor -- start 2024-03-23T11:22:44.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.409+0000 7fc67e340f40 1 -- start start 2024-03-23T11:22:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.410+0000 7fc67e340f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561d24e4aa30 0x561d24e91270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.410+0000 7fc67e340f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d24e772c0 0x561d24e36db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.410+0000 7fc67e340f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24f20f30 0x561d24f1c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.410+0000 7fc67e340f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d24d90800 con 0x561d24f20f30 2024-03-23T11:22:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.410+0000 7fc67e340f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d24d904a0 con 0x561d24e772c0 2024-03-23T11:22:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.410+0000 7fc67e340f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d24d87110 con 0x561d24e4aa30 2024-03-23T11:22:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.411+0000 7fc67c000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24f20f30 0x561d24f1c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.411+0000 7fc67c000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24f20f30 0x561d24f1c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56002/0 (socket says 172.21.15.62:56002) 2024-03-23T11:22:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.411+0000 7fc67c000640 1 -- 172.21.15.62:0/2582135440 learned_addr learned my addr 172.21.15.62:0/2582135440 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.411+0000 7fc67ac00640 1 --2- 172.21.15.62:0/2582135440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d24e772c0 0x561d24e36db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.411+0000 7fc67c000640 1 -- 172.21.15.62:0/2582135440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561d24e4aa30 msgr2=0x561d24e91270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.411+0000 7fc67c000640 1 --2- 172.21.15.62:0/2582135440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561d24e4aa30 0x561d24e91270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.411+0000 7fc67c000640 1 -- 172.21.15.62:0/2582135440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d24e772c0 msgr2=0x561d24e36db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.411+0000 7fc67c000640 1 --2- 172.21.15.62:0/2582135440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d24e772c0 0x561d24e36db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.411+0000 7fc67c000640 1 -- 172.21.15.62:0/2582135440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc668005de0 con 0x561d24f20f30 2024-03-23T11:22:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.411+0000 7fc67c000640 1 --2- 172.21.15.62:0/2582135440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24f20f30 0x561d24f1c670 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc67400a7d0 tx=0x7fc67400a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a0fc8041bf26e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc679800640 1 -- 172.21.15.62:0/2582135440 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67400f070 con 0x561d24f20f30 2024-03-23T11:22:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc679800640 1 -- 172.21.15.62:0/2582135440 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc674007600 con 0x561d24f20f30 2024-03-23T11:22:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc679800640 1 -- 172.21.15.62:0/2582135440 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc674007990 con 0x561d24f20f30 2024-03-23T11:22:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc67e340f40 1 -- 172.21.15.62:0/2582135440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24f20f30 msgr2=0x561d24f1c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc67e340f40 1 --2- 172.21.15.62:0/2582135440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24f20f30 0x561d24f1c670 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc67400a7d0 tx=0x7fc67400a8c0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc67e340f40 1 -- 172.21.15.62:0/2582135440 shutdown_connections 2024-03-23T11:22:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc67e340f40 1 --2- 172.21.15.62:0/2582135440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24f20f30 0x561d24f1c670 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc67e340f40 1 --2- 172.21.15.62:0/2582135440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d24e772c0 0x561d24e36db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc67e340f40 1 --2- 172.21.15.62:0/2582135440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561d24e4aa30 0x561d24e91270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc67e340f40 1 -- 172.21.15.62:0/2582135440 >> 172.21.15.62:0/2582135440 conn(0x561d24e46700 msgr2=0x561d24e92280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc67e340f40 1 -- 172.21.15.62:0/2582135440 shutdown_connections 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.412+0000 7fc67e340f40 1 -- 172.21.15.62:0/2582135440 wait complete. 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.413+0000 7fc67e340f40 1 Processor -- start 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.413+0000 7fc67e340f40 1 -- start start 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.414+0000 7fc67e340f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24e4aa30 0x561d24f1caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.414+0000 7fc67e340f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d24e772c0 0x561d24f1fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.414+0000 7fc67e340f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561d24f20f30 0x561d24f20120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.414+0000 7fc67e340f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d24f1a6d0 con 0x561d24e4aa30 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.414+0000 7fc67e340f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d24efca50 con 0x561d24e772c0 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.414+0000 7fc67e340f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d24efc770 con 0x561d24f20f30 2024-03-23T11:22:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.414+0000 7fc67b600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24e4aa30 0x561d24f1caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.414+0000 7fc67b600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24e4aa30 0x561d24f1caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56014/0 (socket says 172.21.15.62:56014) 2024-03-23T11:22:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.414+0000 7fc67b600640 1 -- 172.21.15.62:0/1103676413 learned_addr learned my addr 172.21.15.62:0/1103676413 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.415+0000 7fc67ac00640 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d24e772c0 0x561d24f1fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.415+0000 7fc67c000640 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561d24f20f30 0x561d24f20120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.415+0000 7fc67b600640 1 -- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561d24f20f30 msgr2=0x561d24f20120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.415+0000 7fc67b600640 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561d24f20f30 0x561d24f20120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.415+0000 7fc67b600640 1 -- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d24e772c0 msgr2=0x561d24f1fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.415+0000 7fc67b600640 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d24e772c0 0x561d24f1fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.415+0000 7fc67b600640 1 -- 172.21.15.62:0/1103676413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67400a420 con 0x561d24e4aa30 2024-03-23T11:22:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.415+0000 7fc67c000640 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561d24f20f30 0x561d24f20120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.415+0000 7fc67b600640 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24e4aa30 0x561d24f1caa0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc66c00afa0 tx=0x7fc66c0084f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.416+0000 7fc673e00640 1 -- 172.21.15.62:0/1103676413 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c010640 con 0x561d24e4aa30 2024-03-23T11:22:44.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.416+0000 7fc673e00640 1 -- 172.21.15.62:0/1103676413 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66c0107a0 con 0x561d24e4aa30 2024-03-23T11:22:44.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.416+0000 7fc673e00640 1 -- 172.21.15.62:0/1103676413 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c010b50 con 0x561d24e4aa30 2024-03-23T11:22:44.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.416+0000 7fc67e340f40 1 -- 172.21.15.62:0/1103676413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561d24d9f340 con 0x561d24e4aa30 2024-03-23T11:22:44.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.416+0000 7fc67e340f40 1 -- 172.21.15.62:0/1103676413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d24d92780 con 0x561d24e4aa30 2024-03-23T11:22:44.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.418+0000 7fc673e00640 1 -- 172.21.15.62:0/1103676413 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc66c010cb0 con 0x561d24e4aa30 2024-03-23T11:22:44.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.419+0000 7fc673e00640 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc65c0691b0 0x7fc65c06b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.419+0000 7fc673e00640 1 -- 172.21.15.62:0/1103676413 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc66c087a10 con 0x561d24e4aa30 2024-03-23T11:22:44.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.419+0000 7fc67e340f40 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x561d24efd810 0x561d24efdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.419+0000 7fc67e340f40 1 -- 172.21.15.62:0/1103676413 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:1 9.7 9:e2e07930:::10000000001.00000000:head [setxattr parent (26) in=32b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62) v8 -- 0x561d24efe190 con 0x561d24efd810 2024-03-23T11:22:44.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.419+0000 7fc67ac00640 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc65c0691b0 0x7fc65c06b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.420+0000 7fc67c000640 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x561d24efd810 0x561d24efdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.421+0000 7fc67ac00640 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc65c0691b0 0x7fc65c06b670 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc668054000 tx=0x7fc66804e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.421+0000 7fc67c000640 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x561d24efd810 0x561d24efdc50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.424+0000 7fc67c000640 1 -- 172.21.15.62:0/1103676413 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (26)] v62'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fc67401a040 con 0x561d24efd810 2024-03-23T11:22:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.424+0000 7fc67e340f40 1 -- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x561d24efd810 msgr2=0x561d24efdc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.424+0000 7fc67e340f40 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x561d24efd810 0x561d24efdc50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.425+0000 7fc67e340f40 1 -- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc65c0691b0 msgr2=0x7fc65c06b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.425+0000 7fc67e340f40 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc65c0691b0 0x7fc65c06b670 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc668054000 tx=0x7fc66804e000 comp rx=0 tx=0).stop 2024-03-23T11:22:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.425+0000 7fc67e340f40 1 -- 172.21.15.62:0/1103676413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24e4aa30 msgr2=0x561d24f1caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.425+0000 7fc67e340f40 1 --2- 172.21.15.62:0/1103676413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d24e4aa30 0x561d24f1caa0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc66c00afa0 tx=0x7fc66c0084f0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.425+0000 7fc67c000640 1 -- 172.21.15.62:0/1103676413 reap_dead start 2024-03-23T11:22:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.425+0000 7fc67e340f40 1 -- 172.21.15.62:0/1103676413 shutdown_connections 2024-03-23T11:22:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.425+0000 7fc67e340f40 1 -- 172.21.15.62:0/1103676413 >> 172.21.15.62:0/1103676413 conn(0x561d24e46700 msgr2=0x561d24e90640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.425+0000 7fc67e340f40 1 -- 172.21.15.62:0/1103676413 shutdown_connections 2024-03-23T11:22:44.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.425+0000 7fc67e340f40 1 -- 172.21.15.62:0/1103676413 wait complete. 2024-03-23T11:22:44.427 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub start / recursive 2024-03-23T11:22:44.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.654+0000 7faa33600640 1 Processor -- start 2024-03-23T11:22:44.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.655+0000 7faa33600640 1 -- start start 2024-03-23T11:22:44.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.656+0000 7faa33600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa2c15ca70 0x7faa2c1606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.656+0000 7faa33600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faa2c160be0 0x7faa2c0cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.656+0000 7faa33600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2c0b58b0 con 0x7faa2c0cf800 2024-03-23T11:22:44.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.656+0000 7faa33600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2c0d0910 con 0x7faa2c160be0 2024-03-23T11:22:44.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.656+0000 7faa33600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c0cfc50 con 0x7faa2c15ca70 2024-03-23T11:22:44.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.657+0000 7faa23e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faa2c160be0 0x7faa2c0cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.657+0000 7faa30c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa2c15ca70 0x7faa2c1606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.657+0000 7faa23e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faa2c160be0 0x7faa2c0cf2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:37208/0 (socket says 172.21.15.62:37208) 2024-03-23T11:22:44.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.657+0000 7faa23e00640 1 -- 172.21.15.62:0/3322589962 learned_addr learned my addr 172.21.15.62:0/3322589962 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.658+0000 7faa23e00640 1 -- 172.21.15.62:0/3322589962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa2c15ca70 msgr2=0x7faa2c1606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.658+0000 7faa23e00640 1 --2- 172.21.15.62:0/3322589962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa2c15ca70 0x7faa2c1606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.658+0000 7faa23e00640 1 -- 172.21.15.62:0/3322589962 >> v1:172.21.15.120:6789/0 conn(0x7faa2c0cf800 legacy=0x7faa2c0b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.658+0000 7faa23e00640 1 -- 172.21.15.62:0/3322589962 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2c161160 con 0x7faa2c160be0 2024-03-23T11:22:44.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.658+0000 7faa2be00640 1 -- 172.21.15.62:0/3322589962 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019152687 0 0) 0x7faa2c0b58b0 con 0x7faa2c0cf800 2024-03-23T11:22:44.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.659+0000 7faa23e00640 1 --2- 172.21.15.62:0/3322589962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa2c160be0 0x7faa2c0cf2c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7faa14009870 tx=0x7faa1402ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f639ec22d097f9d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.660+0000 7faa2be00640 1 -- 172.21.15.62:0/3322589962 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1402f830 con 0x7faa2c160be0 2024-03-23T11:22:44.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.660+0000 7faa2be00640 1 -- 172.21.15.62:0/3322589962 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1402f990 con 0x7faa2c160be0 2024-03-23T11:22:44.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.660+0000 7faa2be00640 1 -- 172.21.15.62:0/3322589962 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1402fd20 con 0x7faa2c160be0 2024-03-23T11:22:44.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.660+0000 7faa33600640 1 -- 172.21.15.62:0/3322589962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa2c160be0 msgr2=0x7faa2c0cf2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.660+0000 7faa33600640 1 --2- 172.21.15.62:0/3322589962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa2c160be0 0x7faa2c0cf2c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7faa14009870 tx=0x7faa1402ba10 comp rx=0 tx=0).stop 2024-03-23T11:22:44.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.660+0000 7faa33600640 1 -- 172.21.15.62:0/3322589962 shutdown_connections 2024-03-23T11:22:44.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.660+0000 7faa33600640 1 --2- 172.21.15.62:0/3322589962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa2c160be0 0x7faa2c0cf2c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.660+0000 7faa33600640 1 --2- 172.21.15.62:0/3322589962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa2c15ca70 0x7faa2c1606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.660+0000 7faa33600640 1 -- 172.21.15.62:0/3322589962 >> 172.21.15.62:0/3322589962 conn(0x7faa2c071390 msgr2=0x7faa2c071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.660+0000 7faa33600640 1 -- 172.21.15.62:0/3322589962 shutdown_connections 2024-03-23T11:22:44.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.660+0000 7faa33600640 1 -- 172.21.15.62:0/3322589962 wait complete. 2024-03-23T11:22:44.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.662+0000 7faa33600640 1 Processor -- start 2024-03-23T11:22:44.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.662+0000 7faa33600640 1 -- start start 2024-03-23T11:22:44.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.663+0000 7faa33600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa2c0cf800 0x7faa2c0b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.663+0000 7faa33600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa2c15ca70 0x7faa2c0c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.663+0000 7faa33600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa2c160be0 0x7faa2c0c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.663+0000 7faa33600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c0c9e60 con 0x7faa2c160be0 2024-03-23T11:22:44.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.663+0000 7faa33600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c150f30 con 0x7faa2c0cf800 2024-03-23T11:22:44.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.663+0000 7faa33600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c0b2fe0 con 0x7faa2c15ca70 2024-03-23T11:22:44.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.664+0000 7faa31600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa2c160be0 0x7faa2c0c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.664+0000 7faa31600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa2c160be0 0x7faa2c0c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56022/0 (socket says 172.21.15.62:56022) 2024-03-23T11:22:44.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.664+0000 7faa31600640 1 -- 172.21.15.62:0/3156276875 learned_addr learned my addr 172.21.15.62:0/3156276875 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.664+0000 7faa30c00640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa2c0cf800 0x7faa2c0b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.664+0000 7faa23e00640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa2c15ca70 0x7faa2c0c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.664+0000 7faa31600640 1 -- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa2c15ca70 msgr2=0x7faa2c0c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.664+0000 7faa31600640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa2c15ca70 0x7faa2c0c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.664+0000 7faa31600640 1 -- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa2c0cf800 msgr2=0x7faa2c0b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.664+0000 7faa31600640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa2c0cf800 0x7faa2c0b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.664+0000 7faa31600640 1 -- 172.21.15.62:0/3156276875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa140094d0 con 0x7faa2c160be0 2024-03-23T11:22:44.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.664+0000 7faa30c00640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa2c0cf800 0x7faa2c0b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:22:44.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.665+0000 7faa31600640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa2c160be0 0x7faa2c0c85a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7faa2400c7b0 tx=0x7faa2400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.665+0000 7faa29600640 1 -- 172.21.15.62:0/3156276875 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24003ba0 con 0x7faa2c160be0 2024-03-23T11:22:44.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.665+0000 7faa29600640 1 -- 172.21.15.62:0/3156276875 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa24003d00 con 0x7faa2c160be0 2024-03-23T11:22:44.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.665+0000 7faa33600640 1 -- 172.21.15.62:0/3156276875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2c0b32c0 con 0x7faa2c160be0 2024-03-23T11:22:44.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.665+0000 7faa29600640 1 -- 172.21.15.62:0/3156276875 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24005270 con 0x7faa2c160be0 2024-03-23T11:22:44.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.665+0000 7faa33600640 1 -- 172.21.15.62:0/3156276875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2c0b35a0 con 0x7faa2c160be0 2024-03-23T11:22:44.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.668+0000 7faa29600640 1 -- 172.21.15.62:0/3156276875 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7faa240053d0 con 0x7faa2c160be0 2024-03-23T11:22:44.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.669+0000 7faa29600640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa0806a7a0 0x7faa0806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.669+0000 7faa29600640 1 -- 172.21.15.62:0/3156276875 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa24088ac0 con 0x7faa2c160be0 2024-03-23T11:22:44.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.670+0000 7faa30c00640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa0806a7a0 0x7faa0806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.670+0000 7faa36238740 1 Processor -- start 2024-03-23T11:22:44.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.670+0000 7faa36238740 1 -- start start 2024-03-23T11:22:44.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.671+0000 7faa30c00640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa0806a7a0 0x7faa0806cc60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009b90 tx=0x7faa1c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.671+0000 7faa36238740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982a260 0x5652d982a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.671+0000 7faa36238740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fa280 0x5652d98274d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.671+0000 7faa36238740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d97fdde0 0x5652d9827a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.671+0000 7faa36238740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d976a280 con 0x5652d982a260 2024-03-23T11:22:44.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.671+0000 7faa36238740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d9767a10 con 0x5652d97fdde0 2024-03-23T11:22:44.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.671+0000 7faa36238740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652d9763300 con 0x5652d97fa280 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.671+0000 7faa30c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982a260 0x5652d982a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.671+0000 7faa30c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982a260 0x5652d982a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56028/0 (socket says 172.21.15.62:56028) 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.671+0000 7faa30c00640 1 -- 172.21.15.62:0/3937621745 learned_addr learned my addr 172.21.15.62:0/3937621745 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.672+0000 7faa31600640 1 --2- 172.21.15.62:0/3937621745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d97fdde0 0x5652d9827a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.672+0000 7faa23e00640 1 --2- 172.21.15.62:0/3937621745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fa280 0x5652d98274d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.672+0000 7faa30c00640 1 -- 172.21.15.62:0/3937621745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fa280 msgr2=0x5652d98274d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.672+0000 7faa30c00640 1 --2- 172.21.15.62:0/3937621745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fa280 0x5652d98274d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.672+0000 7faa30c00640 1 -- 172.21.15.62:0/3937621745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d97fdde0 msgr2=0x5652d9827a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.672+0000 7faa30c00640 1 --2- 172.21.15.62:0/3937621745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d97fdde0 0x5652d9827a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.672+0000 7faa30c00640 1 -- 172.21.15.62:0/3937621745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652d9704f60 con 0x5652d982a260 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.672+0000 7faa31600640 1 --2- 172.21.15.62:0/3937621745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d97fdde0 0x5652d9827a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:22:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.672+0000 7faa30c00640 1 --2- 172.21.15.62:0/3937621745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982a260 0x5652d982a640 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009510 tx=0x7faa1c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14378244bfb54d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa21600640 1 -- 172.21.15.62:0/3937621745 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c011e60 con 0x5652d982a260 2024-03-23T11:22:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa21600640 1 -- 172.21.15.62:0/3937621745 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1c021440 con 0x5652d982a260 2024-03-23T11:22:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa21600640 1 -- 172.21.15.62:0/3937621745 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c0217b0 con 0x5652d982a260 2024-03-23T11:22:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa36238740 1 -- 172.21.15.62:0/3937621745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982a260 msgr2=0x5652d982a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa36238740 1 --2- 172.21.15.62:0/3937621745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982a260 0x5652d982a640 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009510 tx=0x7faa1c014000 comp rx=0 tx=0).stop 2024-03-23T11:22:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa36238740 1 -- 172.21.15.62:0/3937621745 shutdown_connections 2024-03-23T11:22:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa36238740 1 --2- 172.21.15.62:0/3937621745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d97fdde0 0x5652d9827a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa36238740 1 --2- 172.21.15.62:0/3937621745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fa280 0x5652d98274d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa36238740 1 --2- 172.21.15.62:0/3937621745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982a260 0x5652d982a640 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa36238740 1 -- 172.21.15.62:0/3937621745 >> 172.21.15.62:0/3937621745 conn(0x5652d97fc4f0 msgr2=0x5652d9845070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa36238740 1 -- 172.21.15.62:0/3937621745 shutdown_connections 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.673+0000 7faa36238740 1 -- 172.21.15.62:0/3937621745 wait complete. 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.674+0000 7faa36238740 1 -- start start 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.674+0000 7faa36238740 1 Processor -- start 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.675+0000 7faa36238740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d97fdde0 0x5652d981f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.675+0000 7faa36238740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d982a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.675+0000 7faa36238740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d982ab80 0x5652d9820790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.675+0000 7faa36238740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d9721730 con 0x5652d97fdde0 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.675+0000 7faa36238740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d97213d0 con 0x5652d982a260 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.675+0000 7faa36238740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652d94caff0 con 0x5652d982ab80 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.675+0000 7faa36238740 10 client.0 _ll_register_callbacks cb 0x5652d975c970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:22:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa30c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d97fdde0 0x5652d981f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa30c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d97fdde0 0x5652d981f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56038/0 (socket says 172.21.15.62:56038) 2024-03-23T11:22:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa30c00640 1 -- 172.21.15.62:0/479337282 learned_addr learned my addr 172.21.15.62:0/479337282 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa23e00640 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d982a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa31600640 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d982ab80 0x5652d9820790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa30c00640 1 -- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d982ab80 msgr2=0x5652d9820790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa30c00640 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d982ab80 0x5652d9820790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa30c00640 1 -- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 msgr2=0x5652d982a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa30c00640 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d982a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa30c00640 1 -- 172.21.15.62:0/479337282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c011b40 con 0x5652d97fdde0 2024-03-23T11:22:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa31600640 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d982ab80 0x5652d9820790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:44.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa30c00640 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d97fdde0 0x5652d981f430 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7faa1c023040 tx=0x7faa1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc5b2eed9b622932 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.676+0000 7faa1a000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:22:44.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.677+0000 7faa1a000640 1 -- 172.21.15.62:0/479337282 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c021e50 con 0x5652d97fdde0 2024-03-23T11:22:44.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.677+0000 7faa1a000640 1 -- 172.21.15.62:0/479337282 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa1c031070 con 0x5652d97fdde0 2024-03-23T11:22:44.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.677+0000 7faa36238740 1 -- 172.21.15.62:0/479337282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5652d978d4a0 con 0x5652d97fdde0 2024-03-23T11:22:44.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.677+0000 7faa1a000640 1 -- 172.21.15.62:0/479337282 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa1c011e60 con 0x5652d97fdde0 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.677+0000 7faa36238740 10 client.6261 fetch_fsmap learned FSMap version 33 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.677+0000 7faa36238740 1 -- 172.21.15.62:0/479337282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5652d9761980 con 0x5652d97fdde0 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.677+0000 7faa1a000640 1 -- 172.21.15.62:0/479337282 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5652d978d4a0 con 0x5652d97fdde0 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.678+0000 7faa1a000640 1 -- 172.21.15.62:0/479337282 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7faa1c036480 con 0x5652d97fdde0 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.678+0000 7faa36238740 10 client.6261 fetch_fsmap finished waiting for FSMap version 33 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.678+0000 7faa36238740 10 client.6261 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.678+0000 7faa36238740 20 client.6261 populate_metadata read hostname 'smithi062' 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.678+0000 7faa36238740 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x5652d9824b20 0x5652d98622b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.678+0000 7faa36238740 4 client.6261 mds_command: new command op to 6081 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.678+0000 7faa36238740 1 -- 172.21.15.62:0/479337282 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652d9755e30 con 0x5652d9824b20 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.678+0000 7faa23e00640 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x5652d9824b20 0x5652d98622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.679+0000 7faa23e00640 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x5652d9824b20 0x5652d98622b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea29c8ad91c88cfb server_cookie=e3dad954da7ccbc8 in_seq=0 out_seq=0 2024-03-23T11:22:44.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.679+0000 7faa1a000640 10 client.6261 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa1a000640 1 -- 172.21.15.62:0/479337282 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5652d9755e30 con 0x5652d9824b20 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa1a000640 10 client.6261 handle_command_reply: tid=0 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa36238740 1 client.6261 shutdown 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa36238740 1 -- 172.21.15.62:0/479337282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d97fdde0 msgr2=0x5652d981f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa36238740 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d97fdde0 0x5652d981f430 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7faa1c023040 tx=0x7faa1c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa36238740 1 -- 172.21.15.62:0/479337282 shutdown_connections 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa36238740 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x5652d9824b20 0x5652d98622b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa36238740 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d982ab80 0x5652d9820790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa36238740 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d982a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa36238740 1 --2- 172.21.15.62:0/479337282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d97fdde0 0x5652d981f430 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa1a000640 0 client.6261 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.681+0000 7faa36238740 1 -- 172.21.15.62:0/479337282 >> 172.21.15.62:0/479337282 conn(0x5652d97fa280 msgr2=0x5652d97fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.682+0000 7faa36238740 1 -- 172.21.15.62:0/479337282 shutdown_connections 2024-03-23T11:22:44.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.682+0000 7faa36238740 1 -- 172.21.15.62:0/479337282 wait complete. 2024-03-23T11:22:44.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.682+0000 7faa36238740 20 client.6261 trim_cache size 0 max 16384 2024-03-23T11:22:44.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.697+0000 7faa36238740 1 Processor -- start 2024-03-23T11:22:44.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.698+0000 7faa36238740 1 -- start start 2024-03-23T11:22:44.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.698+0000 7faa36238740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fdde0 0x5652d9821060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.698+0000 7faa36238740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d97f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.698+0000 7faa36238740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d9827810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.698+0000 7faa36238740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d982a770 con 0x5652d982ab80 2024-03-23T11:22:44.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.698+0000 7faa36238740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d9820010 con 0x5652d982a260 2024-03-23T11:22:44.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.698+0000 7faa36238740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652d98209d0 con 0x5652d97fdde0 2024-03-23T11:22:44.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.699+0000 7faa31600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d9827810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.699+0000 7faa31600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d9827810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56054/0 (socket says 172.21.15.62:56054) 2024-03-23T11:22:44.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.699+0000 7faa31600640 1 -- 172.21.15.62:0/3972089443 learned_addr learned my addr 172.21.15.62:0/3972089443 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.699+0000 7faa23e00640 1 --2- 172.21.15.62:0/3972089443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d97f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.699+0000 7faa31600640 1 -- 172.21.15.62:0/3972089443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fdde0 msgr2=0x5652d9821060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.699+0000 7faa31600640 1 --2- 172.21.15.62:0/3972089443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fdde0 0x5652d9821060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.699+0000 7faa31600640 1 -- 172.21.15.62:0/3972089443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 msgr2=0x5652d97f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.699+0000 7faa31600640 1 --2- 172.21.15.62:0/3972089443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d97f5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.699+0000 7faa31600640 1 -- 172.21.15.62:0/3972089443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652d97037a0 con 0x5652d982ab80 2024-03-23T11:22:44.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.699+0000 7faa31600640 1 --2- 172.21.15.62:0/3972089443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d9827810 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7faa24087020 tx=0x7faa24057650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603e2acf21e1f7cd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.700+0000 7faa22000640 1 -- 172.21.15.62:0/3972089443 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2405ba60 con 0x5652d982ab80 2024-03-23T11:22:44.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.700+0000 7faa22000640 1 -- 172.21.15.62:0/3972089443 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa24060020 con 0x5652d982ab80 2024-03-23T11:22:44.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.700+0000 7faa22000640 1 -- 172.21.15.62:0/3972089443 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24005680 con 0x5652d982ab80 2024-03-23T11:22:44.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.700+0000 7faa36238740 1 -- 172.21.15.62:0/3972089443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 msgr2=0x5652d9827810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.700+0000 7faa36238740 1 --2- 172.21.15.62:0/3972089443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d9827810 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7faa24087020 tx=0x7faa24057650 comp rx=0 tx=0).stop 2024-03-23T11:22:44.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.700+0000 7faa36238740 1 -- 172.21.15.62:0/3972089443 shutdown_connections 2024-03-23T11:22:44.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.700+0000 7faa36238740 1 --2- 172.21.15.62:0/3972089443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d9827810 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.700+0000 7faa36238740 1 --2- 172.21.15.62:0/3972089443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d97f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.700+0000 7faa36238740 1 --2- 172.21.15.62:0/3972089443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fdde0 0x5652d9821060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.700+0000 7faa36238740 1 -- 172.21.15.62:0/3972089443 >> 172.21.15.62:0/3972089443 conn(0x5652d97fa280 msgr2=0x5652d981f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.701+0000 7faa36238740 1 -- 172.21.15.62:0/3972089443 shutdown_connections 2024-03-23T11:22:44.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.701+0000 7faa36238740 1 -- 172.21.15.62:0/3972089443 wait complete. 2024-03-23T11:22:44.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.701+0000 7faa36238740 1 -- start start 2024-03-23T11:22:44.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.701+0000 7faa36238740 1 Processor -- start 2024-03-23T11:22:44.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.702+0000 7faa36238740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fdde0 0x5652d98002a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.702+0000 7faa36238740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d9860b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.702+0000 7faa36238740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d98610d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.702+0000 7faa36238740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d9862b00 con 0x5652d982ab80 2024-03-23T11:22:44.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.702+0000 7faa36238740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652d9864290 con 0x5652d982a260 2024-03-23T11:22:44.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.702+0000 7faa36238740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652d9800950 con 0x5652d97fdde0 2024-03-23T11:22:44.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.702+0000 7faa36238740 10 client.0 _ll_register_callbacks cb 0x7faa1c011d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa31600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d98610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa31600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d98610d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56068/0 (socket says 172.21.15.62:56068) 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa31600640 1 -- 172.21.15.62:0/115634537 learned_addr learned my addr 172.21.15.62:0/115634537 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa30c00640 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fdde0 0x5652d98002a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa23e00640 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d9860b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa31600640 1 -- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fdde0 msgr2=0x5652d98002a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa31600640 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fdde0 0x5652d98002a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa31600640 1 -- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 msgr2=0x5652d9860b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa31600640 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d9860b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa31600640 1 -- 172.21.15.62:0/115634537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2c0b35a0 con 0x5652d982ab80 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa23e00640 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d9860b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa31600640 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d98610d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7faa2400a430 tx=0x7faa24001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1039b6f63976d6f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.703+0000 7faa1a000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:22:44.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa1a000640 1 -- 172.21.15.62:0/115634537 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24057320 con 0x5652d982ab80 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa1a000640 1 -- 172.21.15.62:0/115634537 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa2405fb90 con 0x5652d982ab80 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa36238740 1 -- 172.21.15.62:0/115634537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5652d97fa110 con 0x5652d982ab80 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa1a000640 1 -- 172.21.15.62:0/115634537 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa2405fd50 con 0x5652d982ab80 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa36238740 10 client.6267 fetch_fsmap learned FSMap version 33 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa36238740 1 -- 172.21.15.62:0/115634537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5652d9728d70 con 0x5652d982ab80 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa1a000640 1 -- 172.21.15.62:0/115634537 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5652d97fa110 con 0x5652d982ab80 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa1a000640 1 -- 172.21.15.62:0/115634537 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7faa24084030 con 0x5652d982ab80 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa36238740 10 client.6267 fetch_fsmap finished waiting for FSMap version 33 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa36238740 10 client.6267 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa36238740 20 client.6267 populate_metadata read hostname 'smithi062' 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa36238740 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x5652d9822d10 0x5652d9846860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa36238740 4 client.6267 mds_command: new command op to 6081 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}] 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.704+0000 7faa36238740 1 -- 172.21.15.62:0/115634537 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}) v1 -- 0x7faa1c021440 con 0x5652d9822d10 2024-03-23T11:22:44.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.705+0000 7faa30c00640 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x5652d9822d10 0x5652d9846860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.706+0000 7faa30c00640 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x5652d9822d10 0x5652d9846860 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca260f08e8b55372 server_cookie=3a0bcee052476d3c in_seq=0 out_seq=0 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.706+0000 7faa1a000640 10 client.6267 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.707+0000 7faa1a000640 1 -- 172.21.15.62:0/115634537 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7faa1c021440 con 0x5652d9822d10 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.707+0000 7faa1a000640 10 client.6267 handle_command_reply: tid=0 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 1 client.6267 shutdown 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 1 -- 172.21.15.62:0/115634537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 msgr2=0x5652d98610d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d98610d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7faa2400a430 tx=0x7faa24001250 comp rx=0 tx=0).stop 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 1 -- 172.21.15.62:0/115634537 shutdown_connections 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x5652d9822d10 0x5652d9846860 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d982ab80 0x5652d98610d0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652d982a260 0x5652d9860b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 1 --2- 172.21.15.62:0/115634537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652d97fdde0 0x5652d98002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 1 -- 172.21.15.62:0/115634537 >> 172.21.15.62:0/115634537 conn(0x5652d97fa280 msgr2=0x5652d982c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa1a000640 0 client.6267 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:44.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 1 -- 172.21.15.62:0/115634537 shutdown_connections 2024-03-23T11:22:44.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 1 -- 172.21.15.62:0/115634537 wait complete. 2024-03-23T11:22:44.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.708+0000 7faa36238740 20 client.6267 trim_cache size 0 max 16384 2024-03-23T11:22:44.707 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:22:44.707 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-03-23T11:22:44.707 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "8a0962cb-5fb5-4f03-8a3c-96b6cd806ade", 2024-03-23T11:22:44.707 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-03-23T11:22:44.707 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:22:44.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 -- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa0806a7a0 msgr2=0x7faa0806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa0806a7a0 0x7faa0806cc60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009b90 tx=0x7faa1c008040 comp rx=0 tx=0).stop 2024-03-23T11:22:44.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 -- 172.21.15.62:0/3156276875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa2c160be0 msgr2=0x7faa2c0c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa2c160be0 0x7faa2c0c85a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7faa2400c7b0 tx=0x7faa2400cbb0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 -- 172.21.15.62:0/3156276875 shutdown_connections 2024-03-23T11:22:44.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa0806a7a0 0x7faa0806cc60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa2c160be0 0x7faa2c0c85a0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa2c15ca70 0x7faa2c0c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 --2- 172.21.15.62:0/3156276875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa2c0cf800 0x7faa2c0b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 -- 172.21.15.62:0/3156276875 >> 172.21.15.62:0/3156276875 conn(0x7faa2c071390 msgr2=0x7faa2c0b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 -- 172.21.15.62:0/3156276875 shutdown_connections 2024-03-23T11:22:44.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.709+0000 7faa33600640 1 -- 172.21.15.62:0/3156276875 wait complete. 2024-03-23T11:22:44.709 INFO:tasks.ceph.mds.b.smithi120.stderr:2024-03-23T11:22:44.709+0000 7f3f37600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={6228=0-4194304@1} caps={6228=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6228 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x5565aeb39b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-23T11:22:44.725 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-23T11:22:44.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.942+0000 7f7ca6600640 1 Processor -- start 2024-03-23T11:22:44.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.943+0000 7f7ca6600640 1 -- start start 2024-03-23T11:22:44.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.944+0000 7f7ca6600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7ca00707a0 0x7f7ca00b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.944+0000 7f7ca6600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca00cc2a0 0x7f7ca00cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.944+0000 7f7ca6600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca00b20d0 con 0x7f7ca01431f0 2024-03-23T11:22:44.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.944+0000 7f7ca6600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca01435d0 con 0x7f7ca00707a0 2024-03-23T11:22:44.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.944+0000 7f7ca6600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca00b14c0 con 0x7f7ca00cc2a0 2024-03-23T11:22:44.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.944+0000 7f7ca5200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7ca01431f0 0x7f7ca0070090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59678/0 (socket says 172.21.15.62:59678) 2024-03-23T11:22:44.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.944+0000 7f7ca5200640 1 -- 172.21.15.62:0/2384107873 learned_addr learned my addr 172.21.15.62:0/2384107873 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:22:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/2384107873 >> v2:172.21.15.120:3300/0 conn(0x7f7ca00707a0 0x7f7ca00b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7ca5c00640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca00cc2a0 0x7f7ca00cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628060180 0 0) 0x7f7ca00b20d0 con 0x7f7ca01431f0 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c88003520 con 0x7f7ca01431f0 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2175202888 0 0) 0x7f7c88003520 con 0x7f7ca01431f0 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca00cc2a0 msgr2=0x7f7ca00cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca00cc2a0 0x7f7ca00cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 >> v2:172.21.15.120:3300/0 conn(0x7f7ca00707a0 msgr2=0x7f7ca00b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 --2- 172.21.15.62:0/2384107873 >> v2:172.21.15.120:3300/0 conn(0x7f7ca00707a0 0x7f7ca00b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca00b17a0 con 0x7f7ca01431f0 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7c94001670 con 0x7f7ca01431f0 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/2384107873 >> v2:172.21.15.120:3300/0 conn(0x7f7ca00707a0 0x7f7ca00b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 >> v1:172.21.15.62:6789/0 conn(0x7f7ca01431f0 legacy=0x7f7ca0070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca00cc2a0 0x7f7c88003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c88003ff0 0x7f7c880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c88004d00 0x7f7c88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca01435d0 con 0x7f7c88004d00 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca00b14c0 con 0x7f7c88003ff0 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.945+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8800bb90 con 0x7f7ca00cc2a0 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.946+0000 7f7ca5c00640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c88004d00 0x7f7c88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.946+0000 7f7ca5200640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca00cc2a0 0x7f7c88003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.946+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c88003ff0 0x7f7c880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.946+0000 7f7ca5c00640 1 -- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca00cc2a0 msgr2=0x7f7c88003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.946+0000 7f7ca5c00640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ca00cc2a0 0x7f7c88003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.946+0000 7f7ca5c00640 1 -- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c88003ff0 msgr2=0x7f7c880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.946+0000 7f7ca5c00640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c88003ff0 0x7f7c880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.946+0000 7f7ca5c00640 1 -- 172.21.15.62:0/2384107873 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c88003520 con 0x7f7c88004d00 2024-03-23T11:22:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.946+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c88003ff0 0x7f7c880047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:22:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.947+0000 7f7ca5c00640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c88004d00 0x7f7c88005180 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7c98007360 tx=0x7f7c880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537aa56b641b2fcd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.948+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9802ccb0 con 0x7f7c88004d00 2024-03-23T11:22:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.948+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9802ce10 con 0x7f7c88004d00 2024-03-23T11:22:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.948+0000 7f7c9f400640 1 -- 172.21.15.62:0/2384107873 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98034660 con 0x7f7c88004d00 2024-03-23T11:22:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.948+0000 7f7ca6600640 1 -- 172.21.15.62:0/2384107873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c88004d00 msgr2=0x7f7c88005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.948+0000 7f7ca6600640 1 --2- 172.21.15.62:0/2384107873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c88004d00 0x7f7c88005180 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7c98007360 tx=0x7f7c880057f0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.948+0000 7f7ca5c00640 1 -- 172.21.15.62:0/2384107873 reap_dead start 2024-03-23T11:22:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.948+0000 7f7ca6600640 1 -- 172.21.15.62:0/2384107873 shutdown_connections 2024-03-23T11:22:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.948+0000 7f7ca6600640 1 -- 172.21.15.62:0/2384107873 >> 172.21.15.62:0/2384107873 conn(0x7f7ca00b0090 msgr2=0x7f7ca00b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.948+0000 7f7ca6600640 1 -- 172.21.15.62:0/2384107873 shutdown_connections 2024-03-23T11:22:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.948+0000 7f7ca6600640 1 -- 172.21.15.62:0/2384107873 wait complete. 2024-03-23T11:22:44.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.950+0000 7f7ca6600640 1 Processor -- start 2024-03-23T11:22:44.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.950+0000 7f7ca6600640 1 -- start start 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.951+0000 7f7ca6600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c88004d00 0x7f7ca01f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.951+0000 7f7ca6600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ca015d260 0x7f7ca00c3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.951+0000 7f7ca6600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ca013d700 0x7f7ca013db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.951+0000 7f7ca6600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca01f3360 con 0x7f7ca013d700 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.951+0000 7f7ca6600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca01f35b0 con 0x7f7ca015d260 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.951+0000 7f7ca6600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca01f3890 con 0x7f7c88004d00 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.951+0000 7f7ca5c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ca013d700 0x7f7ca013db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.952+0000 7f7ca5c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ca013d700 0x7f7ca013db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56086/0 (socket says 172.21.15.62:56086) 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.952+0000 7f7ca5c00640 1 -- 172.21.15.62:0/2607521500 learned_addr learned my addr 172.21.15.62:0/2607521500 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.952+0000 7f7ca5200640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c88004d00 0x7f7ca01f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.952+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ca015d260 0x7f7ca00c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.952+0000 7f7ca5c00640 1 -- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c88004d00 msgr2=0x7f7ca01f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.952+0000 7f7ca5c00640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c88004d00 0x7f7ca01f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.952+0000 7f7ca5c00640 1 -- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ca015d260 msgr2=0x7f7ca00c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.952+0000 7f7ca5c00640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ca015d260 0x7f7ca00c3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.952+0000 7f7ca5c00640 1 -- 172.21.15.62:0/2607521500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca01f3b70 con 0x7f7ca013d700 2024-03-23T11:22:44.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.952+0000 7f7ca5c00640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ca013d700 0x7f7ca013db90 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7c98007360 tx=0x7f7c98004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.953+0000 7f7c9cc00640 1 -- 172.21.15.62:0/2607521500 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9803e600 con 0x7f7ca013d700 2024-03-23T11:22:44.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.953+0000 7f7c9cc00640 1 -- 172.21.15.62:0/2607521500 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9803e760 con 0x7f7ca013d700 2024-03-23T11:22:44.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.953+0000 7f7c9cc00640 1 -- 172.21.15.62:0/2607521500 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9803eb10 con 0x7f7ca013d700 2024-03-23T11:22:44.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.953+0000 7f7ca6600640 1 -- 172.21.15.62:0/2607521500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca01f3e50 con 0x7f7ca013d700 2024-03-23T11:22:44.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.953+0000 7f7ca6600640 1 -- 172.21.15.62:0/2607521500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca013e0d0 con 0x7f7ca013d700 2024-03-23T11:22:44.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.955+0000 7f7c9cc00640 1 -- 172.21.15.62:0/2607521500 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7c9803ec70 con 0x7f7ca013d700 2024-03-23T11:22:44.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.956+0000 7f7c9cc00640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c8006a7a0 0x7f7c8006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.956+0000 7f7c9cc00640 1 -- 172.21.15.62:0/2607521500 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c980ab260 con 0x7f7ca013d700 2024-03-23T11:22:44.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.957+0000 7f7ca5200640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c8006a7a0 0x7f7c8006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.957+0000 7f7caa973740 1 Processor -- start 2024-03-23T11:22:44.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.957+0000 7f7caa973740 1 -- start start 2024-03-23T11:22:44.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.958+0000 7f7ca5200640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c8006a7a0 0x7f7c8006cc60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7c94001050 tx=0x7f7c94009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.958+0000 7f7caa973740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959558d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.958+0000 7f7caa973740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x564959530c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.958+0000 7f7caa973740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550ed0 0x564959531160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.958+0000 7f7caa973740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649594eccb0 con 0x564959550af0 2024-03-23T11:22:44.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.958+0000 7f7caa973740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56495949b500 con 0x564959559460 2024-03-23T11:22:44.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.958+0000 7f7caa973740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x564959498c90 con 0x564959550ed0 2024-03-23T11:22:44.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.959+0000 7f7ca5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959558d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.959+0000 7f7ca5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959558d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56092/0 (socket says 172.21.15.62:56092) 2024-03-23T11:22:44.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.959+0000 7f7ca5200640 1 -- 172.21.15.62:0/3722746112 learned_addr learned my addr 172.21.15.62:0/3722746112 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.959+0000 7f7ca5c00640 1 --2- 172.21.15.62:0/3722746112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550ed0 0x564959531160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.959+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/3722746112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x564959530c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.959+0000 7f7ca5200640 1 -- 172.21.15.62:0/3722746112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550ed0 msgr2=0x564959531160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.959+0000 7f7ca5200640 1 --2- 172.21.15.62:0/3722746112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550ed0 0x564959531160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.959+0000 7f7ca5200640 1 -- 172.21.15.62:0/3722746112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 msgr2=0x564959530c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.959+0000 7f7ca5200640 1 --2- 172.21.15.62:0/3722746112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x564959530c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.959+0000 7f7ca5200640 1 -- 172.21.15.62:0/3722746112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564959446360 con 0x564959550af0 2024-03-23T11:22:44.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.959+0000 7f7ca5200640 1 --2- 172.21.15.62:0/3722746112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959558d60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f7c94010b90 tx=0x7f7c9400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=defc937d347f4fff server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.960+0000 7f7c91600640 1 -- 172.21.15.62:0/3722746112 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94005250 con 0x564959550af0 2024-03-23T11:22:44.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.960+0000 7f7c91600640 1 -- 172.21.15.62:0/3722746112 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c940053b0 con 0x564959550af0 2024-03-23T11:22:44.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.960+0000 7f7c91600640 1 -- 172.21.15.62:0/3722746112 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94005720 con 0x564959550af0 2024-03-23T11:22:44.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.960+0000 7f7caa973740 1 -- 172.21.15.62:0/3722746112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 msgr2=0x564959558d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.960+0000 7f7caa973740 1 --2- 172.21.15.62:0/3722746112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959558d60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f7c94010b90 tx=0x7f7c9400e000 comp rx=0 tx=0).stop 2024-03-23T11:22:44.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.960+0000 7f7caa973740 1 -- 172.21.15.62:0/3722746112 shutdown_connections 2024-03-23T11:22:44.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.960+0000 7f7caa973740 1 --2- 172.21.15.62:0/3722746112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550ed0 0x564959531160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.960+0000 7f7caa973740 1 --2- 172.21.15.62:0/3722746112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x564959530c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.960+0000 7f7caa973740 1 --2- 172.21.15.62:0/3722746112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959558d60 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.960+0000 7f7caa973740 1 -- 172.21.15.62:0/3722746112 >> 172.21.15.62:0/3722746112 conn(0x5649594f8b40 msgr2=0x564959576900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.960+0000 7f7caa973740 1 -- 172.21.15.62:0/3722746112 shutdown_connections 2024-03-23T11:22:44.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.961+0000 7f7caa973740 1 -- 172.21.15.62:0/3722746112 wait complete. 2024-03-23T11:22:44.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.961+0000 7f7caa973740 1 -- start start 2024-03-23T11:22:44.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.961+0000 7f7caa973740 1 Processor -- start 2024-03-23T11:22:44.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.962+0000 7f7caa973740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959586960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.962+0000 7f7caa973740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550ed0 0x56495955a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.962+0000 7f7caa973740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x564959575730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.962+0000 7f7caa973740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564959494580 con 0x564959550af0 2024-03-23T11:22:44.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.962+0000 7f7caa973740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x564959435a50 con 0x564959559460 2024-03-23T11:22:44.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.962+0000 7f7caa973740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649594356f0 con 0x564959550ed0 2024-03-23T11:22:44.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7caa973740 10 client.0 _ll_register_callbacks cb 0x564959484ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:22:44.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7ca5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959586960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7ca5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959586960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56098/0 (socket says 172.21.15.62:56098) 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7ca5200640 1 -- 172.21.15.62:0/2354519781 learned_addr learned my addr 172.21.15.62:0/2354519781 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550ed0 0x56495955a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7ca5200640 1 -- 172.21.15.62:0/2354519781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550ed0 msgr2=0x56495955a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7ca5200640 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550ed0 0x56495955a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7ca5200640 1 -- 172.21.15.62:0/2354519781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 msgr2=0x564959575730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7ca5200640 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x564959575730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7ca5200640 1 -- 172.21.15.62:0/2354519781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9400f040 con 0x564959550af0 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7ca5200640 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959586960 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7c94004bb0 tx=0x7f7c94001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdccf478be5c6c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.963+0000 7f7c86000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.964+0000 7f7c86000640 1 -- 172.21.15.62:0/2354519781 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94031070 con 0x564959550af0 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.964+0000 7f7c86000640 1 -- 172.21.15.62:0/2354519781 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c9401c050 con 0x564959550af0 2024-03-23T11:22:44.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.964+0000 7f7caa973740 1 -- 172.21.15.62:0/2354519781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5649594befe0 con 0x564959550af0 2024-03-23T11:22:44.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.964+0000 7f7c86000640 1 -- 172.21.15.62:0/2354519781 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c94021650 con 0x564959550af0 2024-03-23T11:22:44.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.964+0000 7f7caa973740 10 client.6279 fetch_fsmap learned FSMap version 33 2024-03-23T11:22:44.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.964+0000 7f7caa973740 1 -- 172.21.15.62:0/2354519781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x564959492c00 con 0x564959550af0 2024-03-23T11:22:44.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.964+0000 7f7c86000640 1 -- 172.21.15.62:0/2354519781 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5649594befe0 con 0x564959550af0 2024-03-23T11:22:44.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.965+0000 7f7c86000640 1 -- 172.21.15.62:0/2354519781 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7c94034020 con 0x564959550af0 2024-03-23T11:22:44.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.965+0000 7f7caa973740 10 client.6279 fetch_fsmap finished waiting for FSMap version 33 2024-03-23T11:22:44.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.965+0000 7f7caa973740 10 client.6279 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:22:44.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.965+0000 7f7caa973740 20 client.6279 populate_metadata read hostname 'smithi062' 2024-03-23T11:22:44.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.965+0000 7f7caa973740 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x56495955deb0 0x56495955e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.965+0000 7f7caa973740 4 client.6279 mds_command: new command op to 6081 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:22:44.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.965+0000 7f7caa973740 1 -- 172.21.15.62:0/2354519781 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564959487240 con 0x56495955deb0 2024-03-23T11:22:44.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.965+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x56495955deb0 0x56495955e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.966+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x56495955deb0 0x56495955e290 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7a3867043143e2c server_cookie=8210858e8d20d7a6 in_seq=0 out_seq=0 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.966+0000 7f7c86000640 10 client.6279 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7c86000640 1 -- 172.21.15.62:0/2354519781 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x564959487240 con 0x56495955deb0 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7c86000640 10 client.6279 handle_command_reply: tid=0 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7caa973740 1 client.6279 shutdown 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7caa973740 1 -- 172.21.15.62:0/2354519781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 msgr2=0x564959586960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7caa973740 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959586960 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7c94004bb0 tx=0x7f7c94001930 comp rx=0 tx=0).stop 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7caa973740 1 -- 172.21.15.62:0/2354519781 shutdown_connections 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7caa973740 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x56495955deb0 0x56495955e290 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7caa973740 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x564959575730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7caa973740 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550ed0 0x56495955a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7caa973740 1 --2- 172.21.15.62:0/2354519781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550af0 0x564959586960 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7caa973740 1 -- 172.21.15.62:0/2354519781 >> 172.21.15.62:0/2354519781 conn(0x5649594f8b40 msgr2=0x56495952aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7c86000640 0 client.6279 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:44.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7caa973740 1 -- 172.21.15.62:0/2354519781 shutdown_connections 2024-03-23T11:22:44.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.968+0000 7f7caa973740 1 -- 172.21.15.62:0/2354519781 wait complete. 2024-03-23T11:22:44.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.969+0000 7f7caa973740 20 client.6279 trim_cache size 0 max 16384 2024-03-23T11:22:44.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.984+0000 7f7caa973740 1 Processor -- start 2024-03-23T11:22:44.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.984+0000 7f7caa973740 1 -- start start 2024-03-23T11:22:44.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.985+0000 7f7caa973740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550af0 0x5649595792c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.985+0000 7f7caa973740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x564959579800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.985+0000 7f7caa973740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x56495952aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.985+0000 7f7caa973740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564959191f90 con 0x564959550ed0 2024-03-23T11:22:44.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.985+0000 7f7caa973740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x564959493720 con 0x564959559460 2024-03-23T11:22:44.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.985+0000 7f7caa973740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56495952f150 con 0x564959550af0 2024-03-23T11:22:44.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.986+0000 7f7c9fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x564959579800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.986+0000 7f7c9fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x564959579800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56108/0 (socket says 172.21.15.62:56108) 2024-03-23T11:22:44.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.986+0000 7f7c9fe00640 1 -- 172.21.15.62:0/388760806 learned_addr learned my addr 172.21.15.62:0/388760806 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.986+0000 7f7ca5200640 1 --2- 172.21.15.62:0/388760806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550af0 0x5649595792c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.986+0000 7f7ca5c00640 1 --2- 172.21.15.62:0/388760806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x56495952aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.986+0000 7f7c9fe00640 1 -- 172.21.15.62:0/388760806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550af0 msgr2=0x5649595792c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.986+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/388760806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550af0 0x5649595792c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.986+0000 7f7c9fe00640 1 -- 172.21.15.62:0/388760806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 msgr2=0x56495952aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.986+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/388760806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x56495952aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.986+0000 7f7c9fe00640 1 -- 172.21.15.62:0/388760806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564959434c60 con 0x564959550ed0 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.987+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/388760806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x564959579800 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c000c00 tx=0x7f7c8c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a48b4d68da3cfd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.987+0000 7f7c92000640 1 -- 172.21.15.62:0/388760806 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c01c070 con 0x564959550ed0 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.987+0000 7f7c92000640 1 -- 172.21.15.62:0/388760806 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8c020020 con 0x564959550ed0 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.987+0000 7f7c92000640 1 -- 172.21.15.62:0/388760806 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c01c1d0 con 0x564959550ed0 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.988+0000 7f7caa973740 1 -- 172.21.15.62:0/388760806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 msgr2=0x564959579800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.988+0000 7f7caa973740 1 --2- 172.21.15.62:0/388760806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x564959579800 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c000c00 tx=0x7f7c8c008040 comp rx=0 tx=0).stop 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.988+0000 7f7caa973740 1 -- 172.21.15.62:0/388760806 shutdown_connections 2024-03-23T11:22:44.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.988+0000 7f7caa973740 1 --2- 172.21.15.62:0/388760806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x564959579800 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.988+0000 7f7caa973740 1 --2- 172.21.15.62:0/388760806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x56495952aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.988+0000 7f7caa973740 1 --2- 172.21.15.62:0/388760806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550af0 0x5649595792c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.988+0000 7f7caa973740 1 -- 172.21.15.62:0/388760806 >> 172.21.15.62:0/388760806 conn(0x5649594f8b40 msgr2=0x564959573990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.988+0000 7f7caa973740 1 -- 172.21.15.62:0/388760806 shutdown_connections 2024-03-23T11:22:44.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.988+0000 7f7caa973740 1 -- 172.21.15.62:0/388760806 wait complete. 2024-03-23T11:22:44.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.988+0000 7f7caa973740 1 -- start start 2024-03-23T11:22:44.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.989+0000 7f7caa973740 1 Processor -- start 2024-03-23T11:22:44.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7caa973740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550af0 0x564959574150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7caa973740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x5649595dbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7caa973740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x56495952f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7caa973740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56495955a2f0 con 0x564959550ed0 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7caa973740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56495955bd00 con 0x564959559460 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7caa973740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649595749e0 con 0x564959550af0 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7caa973740 10 client.0 _ll_register_callbacks cb 0x7f7c940108f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7c9fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x5649595dbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7c9fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x5649595dbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56114/0 (socket says 172.21.15.62:56114) 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7c9fe00640 1 -- 172.21.15.62:0/540224464 learned_addr learned my addr 172.21.15.62:0/540224464 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7ca5200640 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550af0 0x564959574150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7c9fe00640 1 -- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550af0 msgr2=0x564959574150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7ca5c00640 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x56495952f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550af0 0x564959574150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7c9fe00640 1 -- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 msgr2=0x56495952f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x56495952f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.990+0000 7f7c9fe00640 1 -- 172.21.15.62:0/540224464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8c010760 con 0x564959550ed0 2024-03-23T11:22:44.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.991+0000 7f7c9fe00640 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x5649595dbc80 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x564959532830 tx=0x7f7c8c01c5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcbb5db92f227707 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:22:44.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.991+0000 7f7c86000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:22:44.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.991+0000 7f7c86000640 1 -- 172.21.15.62:0/540224464 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c004230 con 0x564959550ed0 2024-03-23T11:22:44.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.991+0000 7f7c86000640 1 -- 172.21.15.62:0/540224464 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c8c021030 con 0x564959550ed0 2024-03-23T11:22:44.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.991+0000 7f7caa973740 1 -- 172.21.15.62:0/540224464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c940053b0 con 0x564959550ed0 2024-03-23T11:22:44.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.991+0000 7f7c86000640 1 -- 172.21.15.62:0/540224464 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c8c01c400 con 0x564959550ed0 2024-03-23T11:22:44.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.991+0000 7f7caa973740 10 client.6285 fetch_fsmap learned FSMap version 33 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.991+0000 7f7caa973740 1 -- 172.21.15.62:0/540224464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x56495945a170 con 0x564959550ed0 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.992+0000 7f7c86000640 1 -- 172.21.15.62:0/540224464 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c940053b0 con 0x564959550ed0 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.992+0000 7f7c86000640 1 -- 172.21.15.62:0/540224464 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7c8c004390 con 0x564959550ed0 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.992+0000 7f7caa973740 10 client.6285 fetch_fsmap finished waiting for FSMap version 33 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.992+0000 7f7caa973740 10 client.6285 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.992+0000 7f7caa973740 20 client.6285 populate_metadata read hostname 'smithi062' 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.992+0000 7f7caa973740 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x564959586960 0x564959531220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.992+0000 7f7caa973740 4 client.6285 mds_command: new command op to 6081 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.992+0000 7f7caa973740 1 -- 172.21.15.62:0/540224464 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5649591884e0 con 0x564959586960 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.993+0000 7f7ca5200640 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x564959586960 0x564959531220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.993+0000 7f7ca5200640 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x564959586960 0x564959531220 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d33a659bf0e3190 server_cookie=376458305c230ba3 in_seq=0 out_seq=0 2024-03-23T11:22:44.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.993+0000 7f7c86000640 10 client.6285 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7c86000640 1 -- 172.21.15.62:0/540224464 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+253 (crc 0 0 0) 0x5649591884e0 con 0x564959586960 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7c86000640 10 client.6285 handle_command_reply: tid=0 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7caa973740 1 client.6285 shutdown 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7caa973740 1 -- 172.21.15.62:0/540224464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 msgr2=0x5649595dbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7caa973740 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x5649595dbc80 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x564959532830 tx=0x7f7c8c01c5b0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7caa973740 1 -- 172.21.15.62:0/540224464 shutdown_connections 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7caa973740 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x564959586960 0x564959531220 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7caa973740 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564959550ed0 0x5649595dbc80 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7caa973740 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564959559460 0x56495952f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7c86000640 0 client.6285 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7caa973740 1 --2- 172.21.15.62:0/540224464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x564959550af0 0x564959574150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7caa973740 1 -- 172.21.15.62:0/540224464 >> 172.21.15.62:0/540224464 conn(0x5649594f8b40 msgr2=0x56495955e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7caa973740 1 -- 172.21.15.62:0/540224464 shutdown_connections 2024-03-23T11:22:44.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.994+0000 7f7caa973740 1 -- 172.21.15.62:0/540224464 wait complete. 2024-03-23T11:22:44.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7caa973740 20 client.6285 trim_cache size 0 max 16384 2024-03-23T11:22:44.994 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:22:44.994 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T11:22:44.994 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-03-23T11:22:44.994 INFO:teuthology.orchestra.run.smithi062.stdout: "8a0962cb-5fb5-4f03-8a3c-96b6cd806ade": { 2024-03-23T11:22:44.994 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-03-23T11:22:44.994 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "8a0962cb-5fb5-4f03-8a3c-96b6cd806ade", 2024-03-23T11:22:44.994 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive" 2024-03-23T11:22:44.994 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:44.994 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:22:44.995 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:22:44.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 -- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c8006a7a0 msgr2=0x7f7c8006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c8006a7a0 0x7f7c8006cc60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7c94001050 tx=0x7f7c94009040 comp rx=0 tx=0).stop 2024-03-23T11:22:44.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 -- 172.21.15.62:0/2607521500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ca013d700 msgr2=0x7f7ca013db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:22:44.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ca013d700 0x7f7ca013db90 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7c98007360 tx=0x7f7c98004ab0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 -- 172.21.15.62:0/2607521500 shutdown_connections 2024-03-23T11:22:44.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c8006a7a0 0x7f7c8006cc60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ca013d700 0x7f7ca013db90 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ca015d260 0x7f7ca00c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 --2- 172.21.15.62:0/2607521500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c88004d00 0x7f7ca01f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:22:44.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 -- 172.21.15.62:0/2607521500 >> 172.21.15.62:0/2607521500 conn(0x7f7ca00b0090 msgr2=0x7f7ca015b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:22:44.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 -- 172.21.15.62:0/2607521500 shutdown_connections 2024-03-23T11:22:44.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:22:44.995+0000 7f7ca6600640 1 -- 172.21.15.62:0/2607521500 wait complete. 2024-03-23T11:22:45.009 INFO:tasks.cephfs.filesystem:scrub status for tag:8a0962cb-5fb5-4f03-8a3c-96b6cd806ade - {'path': '/', 'tag': '8a0962cb-5fb5-4f03-8a3c-96b6cd806ade', 'options': 'recursive'} 2024-03-23T11:22:45.388 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:22:45.396 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:22:48.364 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:22:48.368+0000 7f9201200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-23T11:23:15.010 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-23T11:23:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.250+0000 7fd6e4a00640 1 Processor -- start 2024-03-23T11:23:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.250+0000 7fd6e4a00640 1 -- start start 2024-03-23T11:23:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.252+0000 7fd6e4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6e006c6e0 0x7fd6e006cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.252+0000 7fd6e4a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e00cbc60 con 0x7fd6e006c300 2024-03-23T11:23:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.252+0000 7fd6e4a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e00cbf40 con 0x7fd6e00cafa0 2024-03-23T11:23:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.252+0000 7fd6e4a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e00cd410 con 0x7fd6e006c6e0 2024-03-23T11:23:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.252+0000 7fd6dde00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd6e006c300 0x7fd6e00ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37122/0 (socket says 172.21.15.62:37122) 2024-03-23T11:23:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.252+0000 7fd6dde00640 1 -- 172.21.15.62:0/2396517998 learned_addr learned my addr 172.21.15.62:0/2396517998 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.252+0000 7fd6de800640 1 --2- 172.21.15.62:0/2396517998 >> v2:172.21.15.62:3300/0 conn(0x7fd6e006c6e0 0x7fd6e006cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.253+0000 7fd6de800640 1 -- 172.21.15.62:0/2396517998 >> v1:172.21.15.120:6789/0 conn(0x7fd6e00cafa0 legacy=0x7fd6e00cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.253+0000 7fd6de800640 1 -- 172.21.15.62:0/2396517998 >> v1:172.21.15.62:6789/0 conn(0x7fd6e006c300 legacy=0x7fd6e00ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.253+0000 7fd6de800640 1 -- 172.21.15.62:0/2396517998 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e00cc220 con 0x7fd6e006c6e0 2024-03-23T11:23:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.253+0000 7fd6dca00640 1 -- 172.21.15.62:0/2396517998 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440000483 0 0) 0x7fd6e00cbc60 con 0x7fd6e006c300 2024-03-23T11:23:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.254+0000 7fd6de800640 1 --2- 172.21.15.62:0/2396517998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e006c6e0 0x7fd6e006cb30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009d90 tx=0x7fd6d802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97de23ca92895f5b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.255+0000 7fd6dca00640 1 -- 172.21.15.62:0/2396517998 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d802ccf0 con 0x7fd6e006c6e0 2024-03-23T11:23:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.255+0000 7fd6dca00640 1 -- 172.21.15.62:0/2396517998 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d802ce50 con 0x7fd6e006c6e0 2024-03-23T11:23:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.255+0000 7fd6dca00640 1 -- 172.21.15.62:0/2396517998 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8034660 con 0x7fd6e006c6e0 2024-03-23T11:23:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.255+0000 7fd6e4a00640 1 -- 172.21.15.62:0/2396517998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e006c6e0 msgr2=0x7fd6e006cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.255+0000 7fd6e4a00640 1 --2- 172.21.15.62:0/2396517998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e006c6e0 0x7fd6e006cb30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009d90 tx=0x7fd6d802bf40 comp rx=0 tx=0).stop 2024-03-23T11:23:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.256+0000 7fd6e4a00640 1 -- 172.21.15.62:0/2396517998 shutdown_connections 2024-03-23T11:23:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.256+0000 7fd6e4a00640 1 --2- 172.21.15.62:0/2396517998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e006c6e0 0x7fd6e006cb30 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.256+0000 7fd6e4a00640 1 -- 172.21.15.62:0/2396517998 >> 172.21.15.62:0/2396517998 conn(0x7fd6e00b02e0 msgr2=0x7fd6e00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.256+0000 7fd6e4a00640 1 -- 172.21.15.62:0/2396517998 shutdown_connections 2024-03-23T11:23:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.256+0000 7fd6e4a00640 1 -- 172.21.15.62:0/2396517998 wait complete. 2024-03-23T11:23:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.257+0000 7fd6e4a00640 1 Processor -- start 2024-03-23T11:23:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.258+0000 7fd6e4a00640 1 -- start start 2024-03-23T11:23:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.258+0000 7fd6e4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e006c300 0x7fd6e013a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.258+0000 7fd6e4a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6e006c6e0 0x7fd6e015db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.258+0000 7fd6e4a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6e00cafa0 0x7fd6e013c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.258+0000 7fd6e4a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e0151380 con 0x7fd6e006c300 2024-03-23T11:23:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.258+0000 7fd6e4a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e0154080 con 0x7fd6e00cafa0 2024-03-23T11:23:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.258+0000 7fd6e4a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e0151630 con 0x7fd6e006c6e0 2024-03-23T11:23:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.259+0000 7fd6dde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e006c300 0x7fd6e013a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.259+0000 7fd6dde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e006c300 0x7fd6e013a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41432/0 (socket says 172.21.15.62:41432) 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.259+0000 7fd6dde00640 1 -- 172.21.15.62:0/3727492038 learned_addr learned my addr 172.21.15.62:0/3727492038 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.259+0000 7fd6dd400640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6e006c6e0 0x7fd6e015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.259+0000 7fd6dde00640 1 -- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6e006c6e0 msgr2=0x7fd6e015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.259+0000 7fd6dde00640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6e006c6e0 0x7fd6e015db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.259+0000 7fd6dde00640 1 -- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6e00cafa0 msgr2=0x7fd6e013c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.259+0000 7fd6dde00640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6e00cafa0 0x7fd6e013c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.259+0000 7fd6dde00640 1 -- 172.21.15.62:0/3727492038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d8009a70 con 0x7fd6e006c300 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.260+0000 7fd6dde00640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e006c300 0x7fd6e013a990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000c4a0 tx=0x7fd6d000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.260+0000 7fd6d6000640 1 -- 172.21.15.62:0/3727492038 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00074e0 con 0x7fd6e006c300 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.260+0000 7fd6d6000640 1 -- 172.21.15.62:0/3727492038 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d0007640 con 0x7fd6e006c300 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.260+0000 7fd6e4a00640 1 -- 172.21.15.62:0/3727492038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e0151910 con 0x7fd6e006c300 2024-03-23T11:23:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.260+0000 7fd6d6000640 1 -- 172.21.15.62:0/3727492038 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0004a00 con 0x7fd6e006c300 2024-03-23T11:23:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.260+0000 7fd6e4a00640 1 -- 172.21.15.62:0/3727492038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e01510a0 con 0x7fd6e006c300 2024-03-23T11:23:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.262+0000 7fd6d6000640 1 -- 172.21.15.62:0/3727492038 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd6d0004b60 con 0x7fd6e006c300 2024-03-23T11:23:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.263+0000 7fd6d6000640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6c006a7a0 0x7fd6c006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.263+0000 7fd6d6000640 1 -- 172.21.15.62:0/3727492038 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6d00882f0 con 0x7fd6e006c300 2024-03-23T11:23:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.264+0000 7fd6dd400640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6c006a7a0 0x7fd6c006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.264+0000 7fd6e7789740 1 Processor -- start 2024-03-23T11:23:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.264+0000 7fd6e7789740 1 -- start start 2024-03-23T11:23:15.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.265+0000 7fd6dd400640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6c006a7a0 0x7fd6c006cc60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd6e01552d0 tx=0x7fd6cc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.265+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339910 0x55ed08341b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.265+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08319a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.265+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339cf0 0x55ed08319f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.265+0000 7fd6e7789740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed082d5cb0 con 0x55ed08342280 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.265+0000 7fd6e7789740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed08284500 con 0x55ed08339cf0 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.265+0000 7fd6e7789740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed08281c90 con 0x55ed08339910 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.265+0000 7fd6dd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08319a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.265+0000 7fd6dd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08319a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41444/0 (socket says 172.21.15.62:41444) 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.265+0000 7fd6dd400640 1 -- 172.21.15.62:0/3108099745 learned_addr learned my addr 172.21.15.62:0/3108099745 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6dde00640 1 --2- 172.21.15.62:0/3108099745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339910 0x55ed08341b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6de800640 1 --2- 172.21.15.62:0/3108099745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339cf0 0x55ed08319f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6dd400640 1 -- 172.21.15.62:0/3108099745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339910 msgr2=0x55ed08341b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:15.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6dd400640 1 --2- 172.21.15.62:0/3108099745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339910 0x55ed08341b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6dd400640 1 -- 172.21.15.62:0/3108099745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339cf0 msgr2=0x55ed08319f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:15.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6dd400640 1 --2- 172.21.15.62:0/3108099745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339cf0 0x55ed08319f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6dd400640 1 -- 172.21.15.62:0/3108099745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed0822f360 con 0x55ed08342280 2024-03-23T11:23:15.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6de800640 1 --2- 172.21.15.62:0/3108099745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339cf0 0x55ed08319f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:15.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6dd400640 1 --2- 172.21.15.62:0/3108099745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08319a40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0052d0 tx=0x7fd6cc010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68e5cd9bead7f87 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:15.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6caa00640 1 -- 172.21.15.62:0/3108099745 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc01c070 con 0x55ed08342280 2024-03-23T11:23:15.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6caa00640 1 -- 172.21.15.62:0/3108099745 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc021410 con 0x55ed08342280 2024-03-23T11:23:15.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.266+0000 7fd6caa00640 1 -- 172.21.15.62:0/3108099745 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc021780 con 0x55ed08342280 2024-03-23T11:23:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.267+0000 7fd6e7789740 1 -- 172.21.15.62:0/3108099745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 msgr2=0x55ed08319a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.267+0000 7fd6e7789740 1 --2- 172.21.15.62:0/3108099745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08319a40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0052d0 tx=0x7fd6cc010000 comp rx=0 tx=0).stop 2024-03-23T11:23:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.267+0000 7fd6e7789740 1 -- 172.21.15.62:0/3108099745 shutdown_connections 2024-03-23T11:23:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.267+0000 7fd6e7789740 1 --2- 172.21.15.62:0/3108099745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08319a40 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.267+0000 7fd6e7789740 1 --2- 172.21.15.62:0/3108099745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339cf0 0x55ed08319f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.267+0000 7fd6e7789740 1 --2- 172.21.15.62:0/3108099745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339910 0x55ed08341b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.267+0000 7fd6e7789740 1 -- 172.21.15.62:0/3108099745 >> 172.21.15.62:0/3108099745 conn(0x55ed082e1b40 msgr2=0x55ed0835f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.267+0000 7fd6e7789740 1 -- 172.21.15.62:0/3108099745 shutdown_connections 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.267+0000 7fd6e7789740 1 -- 172.21.15.62:0/3108099745 wait complete. 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.268+0000 7fd6e7789740 1 -- start start 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.268+0000 7fd6e7789740 1 Processor -- start 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.269+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339910 0x55ed08318600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.269+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339cf0 0x55ed08318b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.269+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08373360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.269+0000 7fd6e7789740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0827d580 con 0x55ed08342280 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.269+0000 7fd6e7789740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0821ea50 con 0x55ed08339910 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.269+0000 7fd6e7789740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed0821e6f0 con 0x55ed08339cf0 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.269+0000 7fd6e7789740 10 client.0 _ll_register_callbacks cb 0x55ed0826ded0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.270+0000 7fd6de800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08373360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.270+0000 7fd6de800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08373360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41458/0 (socket says 172.21.15.62:41458) 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.270+0000 7fd6de800640 1 -- 172.21.15.62:0/2181299981 learned_addr learned my addr 172.21.15.62:0/2181299981 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.270+0000 7fd6dde00640 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339910 0x55ed08318600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.270+0000 7fd6dd400640 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339cf0 0x55ed08318b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.270+0000 7fd6de800640 1 -- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339cf0 msgr2=0x55ed08318b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.270+0000 7fd6de800640 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339cf0 0x55ed08318b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.270+0000 7fd6de800640 1 -- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339910 msgr2=0x55ed08318600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.270+0000 7fd6de800640 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339910 0x55ed08318600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.270+0000 7fd6de800640 1 -- 172.21.15.62:0/2181299981 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e00cbf40 con 0x55ed08342280 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.271+0000 7fd6de800640 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08373360 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009d90 tx=0x7fd6d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501e1ec012727c47 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.271+0000 7fd6bf400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.271+0000 7fd6bf400640 1 -- 172.21.15.62:0/2181299981 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8034d40 con 0x55ed08342280 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.271+0000 7fd6bf400640 1 -- 172.21.15.62:0/2181299981 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6d803f480 con 0x55ed08342280 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.271+0000 7fd6e7789740 1 -- 172.21.15.62:0/2181299981 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ed082a7fe0 con 0x55ed08342280 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.271+0000 7fd6bf400640 1 -- 172.21.15.62:0/2181299981 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6d8034ea0 con 0x55ed08342280 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.271+0000 7fd6e7789740 10 client.6300 fetch_fsmap learned FSMap version 34 2024-03-23T11:23:15.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.271+0000 7fd6e7789740 1 -- 172.21.15.62:0/2181299981 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55ed0827bc00 con 0x55ed08342280 2024-03-23T11:23:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.272+0000 7fd6bf400640 1 -- 172.21.15.62:0/2181299981 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ed082a7fe0 con 0x55ed08342280 2024-03-23T11:23:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.272+0000 7fd6bf400640 1 -- 172.21.15.62:0/2181299981 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd6d803f640 con 0x55ed08342280 2024-03-23T11:23:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.272+0000 7fd6e7789740 10 client.6300 fetch_fsmap finished waiting for FSMap version 34 2024-03-23T11:23:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.272+0000 7fd6e7789740 10 client.6300 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:23:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.272+0000 7fd6e7789740 20 client.6300 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.272+0000 7fd6e7789740 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55ed0833aae0 0x55ed083738a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.272+0000 7fd6e7789740 4 client.6300 mds_command: new command op to 6081 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:23:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.272+0000 7fd6e7789740 1 -- 172.21.15.62:0/2181299981 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed08270240 con 0x55ed0833aae0 2024-03-23T11:23:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.273+0000 7fd6dde00640 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55ed0833aae0 0x55ed083738a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.274+0000 7fd6dde00640 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55ed0833aae0 0x55ed083738a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0df1457a7e86097 server_cookie=a287aa22674a4e6 in_seq=0 out_seq=0 2024-03-23T11:23:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.274+0000 7fd6bf400640 10 client.6300 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:15.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.275+0000 7fd6bf400640 1 -- 172.21.15.62:0/2181299981 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55ed08270240 con 0x55ed0833aae0 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.275+0000 7fd6bf400640 10 client.6300 handle_command_reply: tid=0 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.275+0000 7fd6e7789740 1 client.6300 shutdown 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6e7789740 1 -- 172.21.15.62:0/2181299981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 msgr2=0x55ed08373360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6e7789740 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08373360 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009d90 tx=0x7fd6d8004a40 comp rx=0 tx=0).stop 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6e7789740 1 -- 172.21.15.62:0/2181299981 shutdown_connections 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6e7789740 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55ed0833aae0 0x55ed083738a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6e7789740 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08342280 0x55ed08373360 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6e7789740 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339cf0 0x55ed08318b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6e7789740 1 --2- 172.21.15.62:0/2181299981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339910 0x55ed08318600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6e7789740 1 -- 172.21.15.62:0/2181299981 >> 172.21.15.62:0/2181299981 conn(0x55ed082e1b40 msgr2=0x55ed08362ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6bf400640 0 client.6300 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6e7789740 1 -- 172.21.15.62:0/2181299981 shutdown_connections 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6e7789740 1 -- 172.21.15.62:0/2181299981 wait complete. 2024-03-23T11:23:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.276+0000 7fd6e7789740 20 client.6300 trim_cache size 0 max 16384 2024-03-23T11:23:15.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.292+0000 7fd6e7789740 1 Processor -- start 2024-03-23T11:23:15.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.292+0000 7fd6e7789740 1 -- start start 2024-03-23T11:23:15.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339910 0x55ed08318490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339cf0 0x55ed083189d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08342280 0x55ed08313bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6e7789740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed07f7af90 con 0x55ed08339910 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6e7789740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed08316c30 con 0x55ed08339cf0 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6e7789740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed08337fc0 con 0x55ed08342280 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6dde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339910 0x55ed08318490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6dde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339910 0x55ed08318490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41462/0 (socket says 172.21.15.62:41462) 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6dde00640 1 -- 172.21.15.62:0/3512996171 learned_addr learned my addr 172.21.15.62:0/3512996171 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6dde00640 1 -- 172.21.15.62:0/3512996171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08342280 msgr2=0x55ed08313bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6dde00640 1 --2- 172.21.15.62:0/3512996171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08342280 0x55ed08313bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6dde00640 1 -- 172.21.15.62:0/3512996171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339cf0 msgr2=0x55ed083189d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6dde00640 1 --2- 172.21.15.62:0/3512996171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339cf0 0x55ed083189d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.293+0000 7fd6dde00640 1 -- 172.21.15.62:0/3512996171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed0821dc60 con 0x55ed08339910 2024-03-23T11:23:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.294+0000 7fd6dde00640 1 --2- 172.21.15.62:0/3512996171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339910 0x55ed08318490 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0000fb0 tx=0x7fd6d00866e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e6bb9be8b418ab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.294+0000 7fd6cb400640 1 -- 172.21.15.62:0/3512996171 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0055500 con 0x55ed08339910 2024-03-23T11:23:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.294+0000 7fd6cb400640 1 -- 172.21.15.62:0/3512996171 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d0086890 con 0x55ed08339910 2024-03-23T11:23:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.294+0000 7fd6cb400640 1 -- 172.21.15.62:0/3512996171 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0053ad0 con 0x55ed08339910 2024-03-23T11:23:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.295+0000 7fd6e7789740 1 -- 172.21.15.62:0/3512996171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339910 msgr2=0x55ed08318490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.295+0000 7fd6e7789740 1 --2- 172.21.15.62:0/3512996171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339910 0x55ed08318490 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0000fb0 tx=0x7fd6d00866e0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.295+0000 7fd6e7789740 1 -- 172.21.15.62:0/3512996171 shutdown_connections 2024-03-23T11:23:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.295+0000 7fd6e7789740 1 --2- 172.21.15.62:0/3512996171 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08342280 0x55ed08313bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.295+0000 7fd6e7789740 1 --2- 172.21.15.62:0/3512996171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08339cf0 0x55ed083189d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.295+0000 7fd6e7789740 1 --2- 172.21.15.62:0/3512996171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339910 0x55ed08318490 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.295+0000 7fd6e7789740 1 -- 172.21.15.62:0/3512996171 >> 172.21.15.62:0/3512996171 conn(0x55ed082e1b40 msgr2=0x55ed08341b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.295+0000 7fd6e7789740 1 -- 172.21.15.62:0/3512996171 shutdown_connections 2024-03-23T11:23:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.295+0000 7fd6e7789740 1 -- 172.21.15.62:0/3512996171 wait complete. 2024-03-23T11:23:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.295+0000 7fd6e7789740 1 -- start start 2024-03-23T11:23:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.296+0000 7fd6e7789740 1 Processor -- start 2024-03-23T11:23:15.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339910 0x55ed083c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339cf0 0x55ed08318490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6e7789740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08342280 0x55ed083189d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6e7789740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed083c8d60 con 0x55ed08339cf0 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6e7789740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0837a8a0 con 0x55ed08342280 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6e7789740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed0837aaf0 con 0x55ed08339910 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6e7789740 10 client.0 _ll_register_callbacks cb 0x7fd6cc001280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6dd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339cf0 0x55ed08318490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6dd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339cf0 0x55ed08318490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41474/0 (socket says 172.21.15.62:41474) 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6dd400640 1 -- 172.21.15.62:0/1157327524 learned_addr learned my addr 172.21.15.62:0/1157327524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6dde00640 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339910 0x55ed083c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6de800640 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08342280 0x55ed083189d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6dd400640 1 -- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339910 msgr2=0x55ed083c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6dd400640 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339910 0x55ed083c84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6dd400640 1 -- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08342280 msgr2=0x55ed083189d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6dd400640 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08342280 0x55ed083189d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.297+0000 7fd6dd400640 1 -- 172.21.15.62:0/1157327524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e01510a0 con 0x55ed08339cf0 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.298+0000 7fd6de800640 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08342280 0x55ed083189d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.298+0000 7fd6dd400640 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339cf0 0x55ed08318490 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc005270 tx=0x7fd6cc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88fecd2d7ca7efb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.298+0000 7fd6bf400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.298+0000 7fd6bf400640 1 -- 172.21.15.62:0/1157327524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc027040 con 0x55ed08339cf0 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.298+0000 7fd6bf400640 1 -- 172.21.15.62:0/1157327524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6cc0124f0 con 0x55ed08339cf0 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.298+0000 7fd6e7789740 1 -- 172.21.15.62:0/1157327524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6cc021410 con 0x55ed08339cf0 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.299+0000 7fd6bf400640 1 -- 172.21.15.62:0/1157327524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6cc022ea0 con 0x55ed08339cf0 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.299+0000 7fd6e7789740 10 client.6306 fetch_fsmap learned FSMap version 34 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.299+0000 7fd6e7789740 1 -- 172.21.15.62:0/1157327524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55ed08243170 con 0x55ed08339cf0 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.299+0000 7fd6bf400640 1 -- 172.21.15.62:0/1157327524 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc021410 con 0x55ed08339cf0 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.299+0000 7fd6bf400640 1 -- 172.21.15.62:0/1157327524 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd6cc0126b0 con 0x55ed08339cf0 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.299+0000 7fd6e7789740 10 client.6306 fetch_fsmap finished waiting for FSMap version 34 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.299+0000 7fd6e7789740 10 client.6306 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:23:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.299+0000 7fd6e7789740 20 client.6306 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:15.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.299+0000 7fd6e7789740 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55ed0837c060 0x55ed08319a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.299+0000 7fd6e7789740 4 client.6306 mds_command: new command op to 6081 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:23:15.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.299+0000 7fd6e7789740 1 -- 172.21.15.62:0/1157327524 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ed07f714e0 con 0x55ed0837c060 2024-03-23T11:23:15.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.300+0000 7fd6dde00640 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55ed0837c060 0x55ed08319a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.300+0000 7fd6dde00640 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55ed0837c060 0x55ed08319a70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccf8e2551bc64e4f server_cookie=f08b9147f3b39fb0 in_seq=0 out_seq=0 2024-03-23T11:23:15.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.300+0000 7fd6bf400640 10 client.6306 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:15.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6bf400640 1 -- 172.21.15.62:0/1157327524 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ed07f714e0 con 0x55ed0837c060 2024-03-23T11:23:15.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6bf400640 10 client.6306 handle_command_reply: tid=0 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6e7789740 1 client.6306 shutdown 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6e7789740 1 -- 172.21.15.62:0/1157327524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339cf0 msgr2=0x55ed08318490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6e7789740 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339cf0 0x55ed08318490 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc005270 tx=0x7fd6cc000ed0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6e7789740 1 -- 172.21.15.62:0/1157327524 shutdown_connections 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6e7789740 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55ed0837c060 0x55ed08319a70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6e7789740 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed08339cf0 0x55ed08318490 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6e7789740 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed08342280 0x55ed083189d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6e7789740 1 --2- 172.21.15.62:0/1157327524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ed08339910 0x55ed083c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6bf400640 0 client.6306 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6e7789740 1 -- 172.21.15.62:0/1157327524 >> 172.21.15.62:0/1157327524 conn(0x55ed082e1b40 msgr2=0x55ed0833bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6e7789740 1 -- 172.21.15.62:0/1157327524 shutdown_connections 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.301+0000 7fd6e7789740 1 -- 172.21.15.62:0/1157327524 wait complete. 2024-03-23T11:23:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e7789740 20 client.6306 trim_cache size 0 max 16384 2024-03-23T11:23:15.301 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:23:15.301 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-03-23T11:23:15.301 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-03-23T11:23:15.301 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:23:15.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 -- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6c006a7a0 msgr2=0x7fd6c006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6c006a7a0 0x7fd6c006cc60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd6e01552d0 tx=0x7fd6cc009040 comp rx=0 tx=0).stop 2024-03-23T11:23:15.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 -- 172.21.15.62:0/3727492038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e006c300 msgr2=0x7fd6e013a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e006c300 0x7fd6e013a990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000c4a0 tx=0x7fd6d000c960 comp rx=0 tx=0).stop 2024-03-23T11:23:15.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 -- 172.21.15.62:0/3727492038 shutdown_connections 2024-03-23T11:23:15.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6c006a7a0 0x7fd6c006cc60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6e00cafa0 0x7fd6e013c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6e006c6e0 0x7fd6e015db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 --2- 172.21.15.62:0/3727492038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6e006c300 0x7fd6e013a990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 -- 172.21.15.62:0/3727492038 >> 172.21.15.62:0/3727492038 conn(0x7fd6e00b02e0 msgr2=0x7fd6e013d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:15.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 -- 172.21.15.62:0/3727492038 shutdown_connections 2024-03-23T11:23:15.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.302+0000 7fd6e4a00640 1 -- 172.21.15.62:0/3727492038 wait complete. 2024-03-23T11:23:15.317 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T11:23:15.317 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:23:15.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.530+0000 7f329fe00640 1 Processor -- start 2024-03-23T11:23:15.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.530+0000 7f329fe00640 1 -- start start 2024-03-23T11:23:15.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.532+0000 7f329fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f329806c300 0x7f32980ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.532+0000 7f329fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32980cafa0 0x7f32980cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.532+0000 7f329fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f329806c6e0 0x7f329806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.532+0000 7f329fe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32980cbc60 con 0x7f329806c300 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.532+0000 7f329fe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32980cbf40 con 0x7f329806c6e0 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.532+0000 7f329fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32980cc220 con 0x7f32980cafa0 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.532+0000 7f329d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f329806c300 0x7f32980ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.533+0000 7f329d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f329806c300 0x7f32980ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41476/0 (socket says 172.21.15.62:41476) 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.533+0000 7f329d400640 1 -- 172.21.15.62:0/1649864443 learned_addr learned my addr 172.21.15.62:0/1649864443 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.533+0000 7f329de00640 1 --2- 172.21.15.62:0/1649864443 >> v2:172.21.15.120:3300/0 conn(0x7f329806c6e0 0x7f329806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.533+0000 7f329ca00640 1 --2- 172.21.15.62:0/1649864443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32980cafa0 0x7f32980cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.533+0000 7f329d400640 1 -- 172.21.15.62:0/1649864443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32980cafa0 msgr2=0x7f32980cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.533+0000 7f329d400640 1 --2- 172.21.15.62:0/1649864443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32980cafa0 0x7f32980cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.533+0000 7f329d400640 1 -- 172.21.15.62:0/1649864443 >> v2:172.21.15.120:3300/0 conn(0x7f329806c6e0 msgr2=0x7f329806cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.533+0000 7f329d400640 1 --2- 172.21.15.62:0/1649864443 >> v2:172.21.15.120:3300/0 conn(0x7f329806c6e0 0x7f329806cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.533+0000 7f329d400640 1 -- 172.21.15.62:0/1649864443 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32980cc500 con 0x7f329806c300 2024-03-23T11:23:15.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.534+0000 7f329d400640 1 --2- 172.21.15.62:0/1649864443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329806c300 0x7f32980ccd00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f32980cac70 tx=0x7f328c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209532b34412e576 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:15.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.535+0000 7f3297e00640 1 -- 172.21.15.62:0/1649864443 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c031870 con 0x7f329806c300 2024-03-23T11:23:15.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.535+0000 7f3297e00640 1 -- 172.21.15.62:0/1649864443 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c0319d0 con 0x7f329806c300 2024-03-23T11:23:15.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.535+0000 7f3297e00640 1 -- 172.21.15.62:0/1649864443 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c031d60 con 0x7f329806c300 2024-03-23T11:23:15.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.535+0000 7f329fe00640 1 -- 172.21.15.62:0/1649864443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329806c300 msgr2=0x7f32980ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.535+0000 7f329fe00640 1 --2- 172.21.15.62:0/1649864443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329806c300 0x7f32980ccd00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f32980cac70 tx=0x7f328c02b910 comp rx=0 tx=0).stop 2024-03-23T11:23:15.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.536+0000 7f329fe00640 1 -- 172.21.15.62:0/1649864443 shutdown_connections 2024-03-23T11:23:15.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.536+0000 7f329fe00640 1 --2- 172.21.15.62:0/1649864443 >> v2:172.21.15.120:3300/0 conn(0x7f329806c6e0 0x7f329806cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.536+0000 7f329fe00640 1 --2- 172.21.15.62:0/1649864443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32980cafa0 0x7f32980cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.536+0000 7f329fe00640 1 --2- 172.21.15.62:0/1649864443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329806c300 0x7f32980ccd00 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.536+0000 7f329fe00640 1 -- 172.21.15.62:0/1649864443 >> 172.21.15.62:0/1649864443 conn(0x7f32980b02e0 msgr2=0x7f32980b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:15.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.536+0000 7f329fe00640 1 -- 172.21.15.62:0/1649864443 shutdown_connections 2024-03-23T11:23:15.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.536+0000 7f329fe00640 1 -- 172.21.15.62:0/1649864443 wait complete. 2024-03-23T11:23:15.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.537+0000 7f329fe00640 1 Processor -- start 2024-03-23T11:23:15.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.538+0000 7f329fe00640 1 -- start start 2024-03-23T11:23:15.534 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:23:15.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.538+0000 7f329fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329806c300 0x7f329813d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.538+0000 7f329fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f329806c6e0 0x7f329813bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32980cafa0 0x7f329813eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329fe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329813ea60 con 0x7f329806c300 2024-03-23T11:23:15.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329813de50 con 0x7f329806c6e0 2024-03-23T11:23:15.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329813e130 con 0x7f32980cafa0 2024-03-23T11:23:15.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329806c300 0x7f329813d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329806c300 0x7f329813d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41486/0 (socket says 172.21.15.62:41486) 2024-03-23T11:23:15.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329d400640 1 -- 172.21.15.62:0/3264032104 learned_addr learned my addr 172.21.15.62:0/3264032104 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:15.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329d400640 1 -- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32980cafa0 msgr2=0x7f329813eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329d400640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32980cafa0 0x7f329813eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329d400640 1 -- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f329806c6e0 msgr2=0x7f329813bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329ca00640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f329806c6e0 0x7f329813bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329d400640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f329806c6e0 0x7f329813bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329d400640 1 -- 172.21.15.62:0/3264032104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c0094d0 con 0x7f329806c300 2024-03-23T11:23:15.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.539+0000 7f329ca00640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f329806c6e0 0x7f329813bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:15.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.540+0000 7f329d400640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329806c300 0x7f329813d870 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f328c000c00 tx=0x7f328c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:15.539 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:23:15.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.540+0000 7f3295600640 1 -- 172.21.15.62:0/3264032104 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c03ad30 con 0x7f329806c300 2024-03-23T11:23:15.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.540+0000 7f3295600640 1 -- 172.21.15.62:0/3264032104 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c03ae90 con 0x7f329806c300 2024-03-23T11:23:15.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.540+0000 7f3295600640 1 -- 172.21.15.62:0/3264032104 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c044630 con 0x7f329806c300 2024-03-23T11:23:15.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.540+0000 7f329fe00640 1 -- 172.21.15.62:0/3264032104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329813e410 con 0x7f329806c300 2024-03-23T11:23:15.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.540+0000 7f329fe00640 1 -- 172.21.15.62:0/3264032104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f329815dc50 con 0x7f329806c300 2024-03-23T11:23:15.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.542+0000 7f329fe00640 1 -- 172.21.15.62:0/3264032104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3298066c90 con 0x7f329806c300 2024-03-23T11:23:15.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.542+0000 7f3295600640 1 -- 172.21.15.62:0/3264032104 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f328c04d450 con 0x7f329806c300 2024-03-23T11:23:15.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.543+0000 7f3295600640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f327806a7d0 0x7f327806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:15.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.543+0000 7f3295600640 1 -- 172.21.15.62:0/3264032104 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f328c02d040 con 0x7f329806c300 2024-03-23T11:23:15.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.544+0000 7f329ca00640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f327806a7d0 0x7f327806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:15.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.545+0000 7f329ca00640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f327806a7d0 0x7f327806cc90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3284006fd0 tx=0x7f3284008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:15.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.549+0000 7f3295600640 1 -- 172.21.15.62:0/3264032104 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f328c076830 con 0x7f329806c300 2024-03-23T11:23:15.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.786+0000 7f329fe00640 1 -- 172.21.15.62:0/3264032104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f329806d360 con 0x7f329806c300 2024-03-23T11:23:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.788+0000 7f3295600640 1 -- 172.21.15.62:0/3264032104 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f328c079e50 con 0x7f329806c300 2024-03-23T11:23:15.785 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:15.785 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3287045135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3287045135},{"type":"v1","addr":"172.21.15.120:6835","nonce":3287045135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6066,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/2859430143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2859430143},{"type":"v1","addr":"172.21.15.62:6834","nonce":2859430143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2005768060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2005768060},{"type":"v1","addr":"172.21.15.62:6835","nonce":2005768060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:22:31.725865+0000","modified":"2024-03-23T11:22:48.368641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6081},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6081":{"gid":6081,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":7,"addr":"172.21.15.120:6837/4158450746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":4158450746},{"type":"v1","addr":"172.21.15.120:6837","nonce":4158450746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T11:23:15.785 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 34 2024-03-23T11:23:15.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 -- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f327806a7d0 msgr2=0x7f327806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f327806a7d0 0x7f327806cc90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3284006fd0 tx=0x7f3284008040 comp rx=0 tx=0).stop 2024-03-23T11:23:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 -- 172.21.15.62:0/3264032104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329806c300 msgr2=0x7f329813d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329806c300 0x7f329813d870 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f328c000c00 tx=0x7f328c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 -- 172.21.15.62:0/3264032104 shutdown_connections 2024-03-23T11:23:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f327806a7d0 0x7f327806cc90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32980cafa0 0x7f329813eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f329806c6e0 0x7f329813bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 --2- 172.21.15.62:0/3264032104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329806c300 0x7f329813d870 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 -- 172.21.15.62:0/3264032104 >> 172.21.15.62:0/3264032104 conn(0x7f32980b02e0 msgr2=0x7f329815a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 -- 172.21.15.62:0/3264032104 shutdown_connections 2024-03-23T11:23:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:15.791+0000 7f329fe00640 1 -- 172.21.15.62:0/3264032104 wait complete. 2024-03-23T11:23:15.807 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b damage ls --format=json-pretty 2024-03-23T11:23:16.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.043+0000 7fe7a0e00640 1 Processor -- start 2024-03-23T11:23:16.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.043+0000 7fe7a0e00640 1 -- start start 2024-03-23T11:23:16.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.045+0000 7fe7a0e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe79c0cc9b0 0x7fe79c0b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.045+0000 7fe7a0e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79c0b20d0 con 0x7fe79c0707a0 2024-03-23T11:23:16.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.045+0000 7fe7a0e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79c1435d0 con 0x7fe79c1431f0 2024-03-23T11:23:16.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.045+0000 7fe7a0e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe79c0b14c0 con 0x7fe79c0cc9b0 2024-03-23T11:23:16.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.045+0000 7fe79aa00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe79c0707a0 0x7fe79c0cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37132/0 (socket says 172.21.15.62:37132) 2024-03-23T11:23:16.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.045+0000 7fe79aa00640 1 -- 172.21.15.62:0/3564126610 learned_addr learned my addr 172.21.15.62:0/3564126610 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:16.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.045+0000 7fe79be00640 1 --2- 172.21.15.62:0/3564126610 >> v2:172.21.15.120:3300/0 conn(0x7fe79c0cc9b0 0x7fe79c0b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.045+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071753958 0 0) 0x7fe79c0b20d0 con 0x7fe79c0707a0 2024-03-23T11:23:16.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.045+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe780003520 con 0x7fe79c0707a0 2024-03-23T11:23:16.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2193258979 0 0) 0x7fe780003520 con 0x7fe79c0707a0 2024-03-23T11:23:16.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 >> v2:172.21.15.120:3300/0 conn(0x7fe79c0cc9b0 msgr2=0x7fe79c0b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 --2- 172.21.15.62:0/3564126610 >> v2:172.21.15.120:3300/0 conn(0x7fe79c0cc9b0 0x7fe79c0b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 >> v1:172.21.15.120:6789/0 conn(0x7fe79c1431f0 legacy=0x7fe79c070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79c0b17a0 con 0x7fe79c0707a0 2024-03-23T11:23:16.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fe788001670 con 0x7fe79c0707a0 2024-03-23T11:23:16.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 >> v1:172.21.15.62:6789/0 conn(0x7fe79c0707a0 legacy=0x7fe79c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 --2- 172.21.15.62:0/3564126610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe780003970 0x7fe780003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 --2- 172.21.15.62:0/3564126610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7800047c0 0x7fe780004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 --2- 172.21.15.62:0/3564126610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 0x7fe78000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c0b14c0 con 0x7fe780005180 2024-03-23T11:23:16.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78000c120 con 0x7fe780003970 2024-03-23T11:23:16.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78000c2e0 con 0x7fe7800047c0 2024-03-23T11:23:16.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.046+0000 7fe79be00640 1 --2- 172.21.15.62:0/3564126610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 0x7fe78000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.047+0000 7fe79b400640 1 --2- 172.21.15.62:0/3564126610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe780003970 0x7fe780003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.047+0000 7fe79be00640 1 -- 172.21.15.62:0/3564126610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7800047c0 msgr2=0x7fe780004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.047+0000 7fe79be00640 1 --2- 172.21.15.62:0/3564126610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7800047c0 0x7fe780004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.047+0000 7fe79be00640 1 -- 172.21.15.62:0/3564126610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe780003970 msgr2=0x7fe780003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.047+0000 7fe79be00640 1 --2- 172.21.15.62:0/3564126610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe780003970 0x7fe780003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.047+0000 7fe79be00640 1 -- 172.21.15.62:0/3564126610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe780003520 con 0x7fe780005180 2024-03-23T11:23:16.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.048+0000 7fe79be00640 1 -- 172.21.15.62:0/3564126610 reap_dead start 2024-03-23T11:23:16.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.048+0000 7fe79be00640 1 --2- 172.21.15.62:0/3564126610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 0x7fe78000b980 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fe794009fc0 tx=0x7fe7800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17288efbf2e4a240 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.048+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79402cc60 con 0x7fe780005180 2024-03-23T11:23:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.048+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79400f040 con 0x7fe780005180 2024-03-23T11:23:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.048+0000 7fe79a000640 1 -- 172.21.15.62:0/3564126610 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe794003990 con 0x7fe780005180 2024-03-23T11:23:16.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.049+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3564126610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 msgr2=0x7fe78000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.049+0000 7fe7a0e00640 1 --2- 172.21.15.62:0/3564126610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 0x7fe78000b980 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fe794009fc0 tx=0x7fe7800057f0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.049+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3564126610 shutdown_connections 2024-03-23T11:23:16.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.049+0000 7fe7a0e00640 1 --2- 172.21.15.62:0/3564126610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 0x7fe78000b980 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.049+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3564126610 >> 172.21.15.62:0/3564126610 conn(0x7fe79c0b0090 msgr2=0x7fe79c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.049+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3564126610 shutdown_connections 2024-03-23T11:23:16.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.049+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3564126610 wait complete. 2024-03-23T11:23:16.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.051+0000 7fe7a0e00640 1 Processor -- start 2024-03-23T11:23:16.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.051+0000 7fe7a0e00640 1 -- start start 2024-03-23T11:23:16.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe7a0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 0x7fe79c0af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe7a0e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe79c15c0e0 0x7fe79c159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe7a0e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe79c0ae5e0 0x7fe79c0aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe7a0e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c0aefa0 con 0x7fe780005180 2024-03-23T11:23:16.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe7a0e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c0a8520 con 0x7fe79c15c0e0 2024-03-23T11:23:16.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe7a0e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c0a8800 con 0x7fe79c0ae5e0 2024-03-23T11:23:16.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe79b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 0x7fe79c0af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe79b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 0x7fe79c0af220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41496/0 (socket says 172.21.15.62:41496) 2024-03-23T11:23:16.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe79b400640 1 -- 172.21.15.62:0/3061619486 learned_addr learned my addr 172.21.15.62:0/3061619486 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe79aa00640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe79c15c0e0 0x7fe79c159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe79be00640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe79c0ae5e0 0x7fe79c0aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.052+0000 7fe79b400640 1 -- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe79c0ae5e0 msgr2=0x7fe79c0aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.053+0000 7fe79b400640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe79c0ae5e0 0x7fe79c0aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.053+0000 7fe79b400640 1 -- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe79c15c0e0 msgr2=0x7fe79c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.053+0000 7fe79b400640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe79c15c0e0 0x7fe79c159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.053+0000 7fe79b400640 1 -- 172.21.15.62:0/3061619486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79c0a8ae0 con 0x7fe780005180 2024-03-23T11:23:16.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.053+0000 7fe79aa00640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe79c15c0e0 0x7fe79c159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:16.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.053+0000 7fe79b400640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 0x7fe79c0af220 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00d4c0 tx=0x7fe78c00d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.053+0000 7fe793400640 1 -- 172.21.15.62:0/3061619486 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c005480 con 0x7fe780005180 2024-03-23T11:23:16.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.053+0000 7fe793400640 1 -- 172.21.15.62:0/3061619486 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78c003610 con 0x7fe780005180 2024-03-23T11:23:16.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.053+0000 7fe793400640 1 -- 172.21.15.62:0/3061619486 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c0039e0 con 0x7fe780005180 2024-03-23T11:23:16.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.053+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3061619486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe79c0a8dc0 con 0x7fe780005180 2024-03-23T11:23:16.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.053+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3061619486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79c0cb170 con 0x7fe780005180 2024-03-23T11:23:16.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.056+0000 7fe793400640 1 -- 172.21.15.62:0/3061619486 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe78c003b40 con 0x7fe780005180 2024-03-23T11:23:16.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.057+0000 7fe793400640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe77c06a7d0 0x7fe77c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.057+0000 7fe793400640 1 -- 172.21.15.62:0/3061619486 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe78c088cf0 con 0x7fe780005180 2024-03-23T11:23:16.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.058+0000 7fe7a4faa740 1 Processor -- start 2024-03-23T11:23:16.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.058+0000 7fe79aa00640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe77c06a7d0 0x7fe77c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.058+0000 7fe7a4faa740 1 -- start start 2024-03-23T11:23:16.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.059+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e612f70 0x55da8e613350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.059+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e616fc0 0x55da8e618a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.059+0000 7fe79aa00640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe77c06a7d0 0x7fe77c06cc90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe79c159c50 tx=0x7fe788009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.059+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e642d00 0x55da8e643180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.059+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8e57ed80 con 0x55da8e612f70 2024-03-23T11:23:16.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.059+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8e57c350 con 0x55da8e642d00 2024-03-23T11:23:16.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.059+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da8e577c40 con 0x55da8e616fc0 2024-03-23T11:23:16.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.059+0000 7fe79b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e612f70 0x55da8e613350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.059+0000 7fe79b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e612f70 0x55da8e613350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41504/0 (socket says 172.21.15.62:41504) 2024-03-23T11:23:16.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.059+0000 7fe79b400640 1 -- 172.21.15.62:0/2885733286 learned_addr learned my addr 172.21.15.62:0/2885733286 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.060+0000 7fe79b400640 1 -- 172.21.15.62:0/2885733286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e616fc0 msgr2=0x55da8e618a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.060+0000 7fe79b400640 1 --2- 172.21.15.62:0/2885733286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e616fc0 0x55da8e618a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.060+0000 7fe79b400640 1 -- 172.21.15.62:0/2885733286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e642d00 msgr2=0x55da8e643180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.060+0000 7fe79b400640 1 --2- 172.21.15.62:0/2885733286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e642d00 0x55da8e643180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.060+0000 7fe79b400640 1 -- 172.21.15.62:0/2885733286 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da8e5423e0 con 0x55da8e612f70 2024-03-23T11:23:16.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.060+0000 7fe79b400640 1 --2- 172.21.15.62:0/2885733286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e612f70 0x55da8e613350 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00d490 tx=0x7fe78c058500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a06f569b5989b67 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.060+0000 7fe787e00640 1 -- 172.21.15.62:0/2885733286 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c0601c0 con 0x55da8e612f70 2024-03-23T11:23:16.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.060+0000 7fe787e00640 1 -- 172.21.15.62:0/2885733286 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78c0560e0 con 0x55da8e612f70 2024-03-23T11:23:16.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.060+0000 7fe787e00640 1 -- 172.21.15.62:0/2885733286 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c05fd30 con 0x55da8e612f70 2024-03-23T11:23:16.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.061+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2885733286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e612f70 msgr2=0x55da8e613350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.061+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/2885733286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e612f70 0x55da8e613350 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00d490 tx=0x7fe78c058500 comp rx=0 tx=0).stop 2024-03-23T11:23:16.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.061+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2885733286 shutdown_connections 2024-03-23T11:23:16.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.061+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/2885733286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e642d00 0x55da8e643180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.061+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/2885733286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e616fc0 0x55da8e618a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.061+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/2885733286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e612f70 0x55da8e613350 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.061+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2885733286 >> 172.21.15.62:0/2885733286 conn(0x55da8e614ff0 msgr2=0x55da8e6168b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.061+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2885733286 shutdown_connections 2024-03-23T11:23:16.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.061+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2885733286 wait complete. 2024-03-23T11:23:16.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.062+0000 7fe7a4faa740 1 -- start start 2024-03-23T11:23:16.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.062+0000 7fe7a4faa740 1 Processor -- start 2024-03-23T11:23:16.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.063+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e642d00 0x55da8e6700f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.063+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e633d80 0x55da8e631c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.063+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e65fd20 0x55da8e634700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.063+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8e518ce0 con 0x55da8e633d80 2024-03-23T11:23:16.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.063+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8e518980 con 0x55da8e642d00 2024-03-23T11:23:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.063+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da8e2e0000 con 0x55da8e65fd20 2024-03-23T11:23:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.063+0000 7fe7a4faa740 10 client.0 _ll_register_callbacks cb 0x55da8e571410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe79aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e633d80 0x55da8e631c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe79aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e633d80 0x55da8e631c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41512/0 (socket says 172.21.15.62:41512) 2024-03-23T11:23:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe79aa00640 1 -- 172.21.15.62:0/2249971692 learned_addr learned my addr 172.21.15.62:0/2249971692 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe79be00640 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e65fd20 0x55da8e634700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe79b400640 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e642d00 0x55da8e6700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe79aa00640 1 -- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e65fd20 msgr2=0x55da8e634700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe79aa00640 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e65fd20 0x55da8e634700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe79aa00640 1 -- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e642d00 msgr2=0x55da8e6700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe79aa00640 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e642d00 0x55da8e6700f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe79aa00640 1 -- 172.21.15.62:0/2249971692 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79c0cb170 con 0x55da8e633d80 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe79aa00640 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e633d80 0x55da8e631c70 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe788005fd0 tx=0x7fe788007770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3e693e0de7835f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.064+0000 7fe784c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.065+0000 7fe784c00640 1 -- 172.21.15.62:0/2249971692 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788021620 con 0x55da8e633d80 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.065+0000 7fe784c00640 1 -- 172.21.15.62:0/2249971692 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe788022e30 con 0x55da8e633d80 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.065+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2249971692 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da8e2298d0 con 0x55da8e633d80 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.065+0000 7fe784c00640 1 -- 172.21.15.62:0/2249971692 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe788012410 con 0x55da8e633d80 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.065+0000 7fe7a4faa740 10 client.6324 fetch_fsmap learned FSMap version 34 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.065+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2249971692 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55da8e5762c0 con 0x55da8e633d80 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.065+0000 7fe784c00640 1 -- 172.21.15.62:0/2249971692 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55da8e2298d0 con 0x55da8e633d80 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.066+0000 7fe784c00640 1 -- 172.21.15.62:0/2249971692 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe788021780 con 0x55da8e633d80 2024-03-23T11:23:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.066+0000 7fe7a4faa740 10 client.6324 fetch_fsmap finished waiting for FSMap version 34 2024-03-23T11:23:16.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.066+0000 7fe7a4faa740 10 client.6324 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:23:16.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.066+0000 7fe7a4faa740 20 client.6324 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:16.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.066+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55da8e640cc0 0x55da8e6410a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.066+0000 7fe7a4faa740 4 client.6324 mds_command: new command op to 6081 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:23:16.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.066+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2249971692 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da8e26c4a0 con 0x55da8e640cc0 2024-03-23T11:23:16.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.066+0000 7fe79b400640 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55da8e640cc0 0x55da8e6410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.067+0000 7fe79b400640 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55da8e640cc0 0x55da8e6410a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eabe46d9394548a6 server_cookie=10169609b54e8d66 in_seq=0 out_seq=0 2024-03-23T11:23:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.067+0000 7fe784c00640 10 client.6324 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe784c00640 1 -- 172.21.15.62:0/2249971692 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55da8e26c4a0 con 0x55da8e640cc0 2024-03-23T11:23:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe784c00640 10 client.6324 handle_command_reply: tid=0 2024-03-23T11:23:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe7a4faa740 1 client.6324 shutdown 2024-03-23T11:23:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2249971692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e633d80 msgr2=0x55da8e631c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e633d80 0x55da8e631c70 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe788005fd0 tx=0x7fe788007770 comp rx=0 tx=0).stop 2024-03-23T11:23:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2249971692 shutdown_connections 2024-03-23T11:23:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55da8e640cc0 0x55da8e6410a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e633d80 0x55da8e631c70 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe784c00640 0 client.6324 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e65fd20 0x55da8e634700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/2249971692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e642d00 0x55da8e6700f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2249971692 >> 172.21.15.62:0/2249971692 conn(0x55da8e612f70 msgr2=0x55da8e618c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2249971692 shutdown_connections 2024-03-23T11:23:16.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.069+0000 7fe7a4faa740 1 -- 172.21.15.62:0/2249971692 wait complete. 2024-03-23T11:23:16.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.070+0000 7fe7a4faa740 20 client.6324 trim_cache size 0 max 16384 2024-03-23T11:23:16.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.082+0000 7fe7a4faa740 1 Processor -- start 2024-03-23T11:23:16.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.082+0000 7fe7a4faa740 1 -- start start 2024-03-23T11:23:16.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.083+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e633d80 0x55da8e617bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.083+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e640cc0 0x55da8e618130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.083+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.083+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8e640650 con 0x55da8e642d00 2024-03-23T11:23:16.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.083+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8e65dac0 con 0x55da8e633d80 2024-03-23T11:23:16.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.083+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da8e65fab0 con 0x55da8e640cc0 2024-03-23T11:23:16.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.083+0000 7fe79be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.083+0000 7fe79be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41526/0 (socket says 172.21.15.62:41526) 2024-03-23T11:23:16.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.083+0000 7fe79be00640 1 -- 172.21.15.62:0/3018976885 learned_addr learned my addr 172.21.15.62:0/3018976885 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.084+0000 7fe79aa00640 1 --2- 172.21.15.62:0/3018976885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e640cc0 0x55da8e618130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.084+0000 7fe79b400640 1 --2- 172.21.15.62:0/3018976885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e633d80 0x55da8e617bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.084+0000 7fe79be00640 1 -- 172.21.15.62:0/3018976885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e640cc0 msgr2=0x55da8e618130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.084+0000 7fe79be00640 1 --2- 172.21.15.62:0/3018976885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e640cc0 0x55da8e618130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.084+0000 7fe79be00640 1 -- 172.21.15.62:0/3018976885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e633d80 msgr2=0x55da8e617bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.084+0000 7fe79be00640 1 --2- 172.21.15.62:0/3018976885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e633d80 0x55da8e617bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.084+0000 7fe79be00640 1 -- 172.21.15.62:0/3018976885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da8e518170 con 0x55da8e642d00 2024-03-23T11:23:16.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.084+0000 7fe79be00640 1 --2- 172.21.15.62:0/3018976885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646100 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe794009f90 tx=0x7fe794004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31dbef75f51d508d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.084+0000 7fe790c00640 1 -- 172.21.15.62:0/3018976885 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79402cb20 con 0x55da8e642d00 2024-03-23T11:23:16.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.084+0000 7fe790c00640 1 -- 172.21.15.62:0/3018976885 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79402cc80 con 0x55da8e642d00 2024-03-23T11:23:16.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.084+0000 7fe790c00640 1 -- 172.21.15.62:0/3018976885 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe794003990 con 0x55da8e642d00 2024-03-23T11:23:16.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.085+0000 7fe7a4faa740 1 -- 172.21.15.62:0/3018976885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 msgr2=0x55da8e646100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.085+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/3018976885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646100 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe794009f90 tx=0x7fe794004a40 comp rx=0 tx=0).stop 2024-03-23T11:23:16.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.085+0000 7fe7a4faa740 1 -- 172.21.15.62:0/3018976885 shutdown_connections 2024-03-23T11:23:16.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.085+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/3018976885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646100 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.085+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/3018976885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e640cc0 0x55da8e618130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.085+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/3018976885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e633d80 0x55da8e617bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.085+0000 7fe7a4faa740 1 -- 172.21.15.62:0/3018976885 >> 172.21.15.62:0/3018976885 conn(0x55da8e612f70 msgr2=0x55da8e6424e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.085+0000 7fe7a4faa740 1 -- 172.21.15.62:0/3018976885 shutdown_connections 2024-03-23T11:23:16.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.085+0000 7fe7a4faa740 1 -- 172.21.15.62:0/3018976885 wait complete. 2024-03-23T11:23:16.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.086+0000 7fe7a4faa740 1 -- start start 2024-03-23T11:23:16.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.086+0000 7fe7a4faa740 1 Processor -- start 2024-03-23T11:23:16.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.087+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e633d80 0x55da8e6c3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.087+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e640cc0 0x55da8e6462c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.087+0000 7fe7a4faa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.087+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8e67feb0 con 0x55da8e642d00 2024-03-23T11:23:16.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.087+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8e618580 con 0x55da8e633d80 2024-03-23T11:23:16.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.087+0000 7fe7a4faa740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da8e6c3860 con 0x55da8e640cc0 2024-03-23T11:23:16.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.087+0000 7fe7a4faa740 10 client.0 _ll_register_callbacks cb 0x7fe7880057f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:16.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.087+0000 7fe79be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.087+0000 7fe79be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41536/0 (socket says 172.21.15.62:41536) 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.087+0000 7fe79be00640 1 -- 172.21.15.62:0/1241512346 learned_addr learned my addr 172.21.15.62:0/1241512346 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe79b400640 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e633d80 0x55da8e6c3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe79aa00640 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e640cc0 0x55da8e6462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe79be00640 1 -- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e640cc0 msgr2=0x55da8e6462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe79be00640 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e640cc0 0x55da8e6462c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe79be00640 1 -- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e633d80 msgr2=0x55da8e6c3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe79be00640 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e633d80 0x55da8e6c3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe79be00640 1 -- 172.21.15.62:0/1241512346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe780003520 con 0x55da8e642d00 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe79aa00640 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e640cc0 0x55da8e6462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe79be00640 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646800 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe794041040 tx=0x7fe794004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697332de6e56443a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe784c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe784c00640 1 -- 172.21.15.62:0/1241512346 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe794004240 con 0x55da8e642d00 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe784c00640 1 -- 172.21.15.62:0/1241512346 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7940043a0 con 0x55da8e642d00 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.088+0000 7fe7a4faa740 1 -- 172.21.15.62:0/1241512346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe78c0560e0 con 0x55da8e642d00 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.089+0000 7fe784c00640 1 -- 172.21.15.62:0/1241512346 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe794004560 con 0x55da8e642d00 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.089+0000 7fe7a4faa740 10 client.6330 fetch_fsmap learned FSMap version 34 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.089+0000 7fe7a4faa740 1 -- 172.21.15.62:0/1241512346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55da8e53da50 con 0x55da8e642d00 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.089+0000 7fe784c00640 1 -- 172.21.15.62:0/1241512346 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78c0560e0 con 0x55da8e642d00 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.089+0000 7fe784c00640 1 -- 172.21.15.62:0/1241512346 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe7940046c0 con 0x55da8e642d00 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.089+0000 7fe7a4faa740 10 client.6330 fetch_fsmap finished waiting for FSMap version 34 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.089+0000 7fe7a4faa740 10 client.6330 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.089+0000 7fe7a4faa740 20 client.6330 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:16.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.089+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55da8e65fd20 0x55da8e6179c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.089+0000 7fe7a4faa740 4 client.6330 mds_command: new command op to 6081 tid=0[{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-23T11:23:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.089+0000 7fe7a4faa740 1 -- 172.21.15.62:0/1241512346 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x55da8e27d7a0 con 0x55da8e65fd20 2024-03-23T11:23:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.090+0000 7fe79b400640 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55da8e65fd20 0x55da8e6179c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.091+0000 7fe79b400640 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55da8e65fd20 0x55da8e6179c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d549ec4c7b720df server_cookie=5d41ed02b1587220 in_seq=0 out_seq=0 2024-03-23T11:23:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.091+0000 7fe784c00640 10 client.6330 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:16.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.091+0000 7fe784c00640 1 -- 172.21.15.62:0/1241512346 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+140 (crc 0 0 0) 0x55da8e27d7a0 con 0x55da8e65fd20 2024-03-23T11:23:16.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.091+0000 7fe784c00640 10 client.6330 handle_command_reply: tid=0 2024-03-23T11:23:16.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.091+0000 7fe7a4faa740 1 client.6330 shutdown 2024-03-23T11:23:16.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.091+0000 7fe7a4faa740 1 -- 172.21.15.62:0/1241512346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 msgr2=0x55da8e646800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.091+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646800 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe794041040 tx=0x7fe794004ab0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe7a4faa740 1 -- 172.21.15.62:0/1241512346 shutdown_connections 2024-03-23T11:23:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55da8e65fd20 0x55da8e6179c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da8e642d00 0x55da8e646800 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55da8e640cc0 0x55da8e6462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe7a4faa740 1 --2- 172.21.15.62:0/1241512346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55da8e633d80 0x55da8e6c3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe784c00640 0 client.6330 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe7a4faa740 1 -- 172.21.15.62:0/1241512346 >> 172.21.15.62:0/1241512346 conn(0x55da8e612f70 msgr2=0x55da8e630f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe7a4faa740 1 -- 172.21.15.62:0/1241512346 shutdown_connections 2024-03-23T11:23:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe7a4faa740 1 -- 172.21.15.62:0/1241512346 wait complete. 2024-03-23T11:23:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe7a4faa740 20 client.6330 trim_cache size 0 max 16384 2024-03-23T11:23:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:16.093 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:23:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:23:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "damage_type": "backtrace", 2024-03-23T11:23:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 205169123, 2024-03-23T11:23:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "ino": 1099511627777, 2024-03-23T11:23:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/dir_test/file" 2024-03-23T11:23:16.093 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:23:16.094 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe77c06a7d0 msgr2=0x7fe77c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.092+0000 7fe7a0e00640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe77c06a7d0 0x7fe77c06cc90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe79c159c50 tx=0x7fe788009040 comp rx=0 tx=0).stop 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.093+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3061619486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 msgr2=0x7fe79c0af220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.093+0000 7fe7a0e00640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 0x7fe79c0af220 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00d4c0 tx=0x7fe78c00d980 comp rx=0 tx=0).stop 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.093+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3061619486 shutdown_connections 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.093+0000 7fe7a0e00640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe77c06a7d0 0x7fe77c06cc90 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.093+0000 7fe7a0e00640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe79c0ae5e0 0x7fe79c0aea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.093+0000 7fe7a0e00640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe79c15c0e0 0x7fe79c159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.093+0000 7fe7a0e00640 1 --2- 172.21.15.62:0/3061619486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe780005180 0x7fe79c0af220 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.093+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3061619486 >> 172.21.15.62:0/3061619486 conn(0x7fe79c0b0090 msgr2=0x7fe79c13eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.093+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3061619486 shutdown_connections 2024-03-23T11:23:16.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.093+0000 7fe7a0e00640 1 -- 172.21.15.62:0/3061619486 wait complete. 2024-03-23T11:23:16.106 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-23T11:23:16.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.321+0000 7f419b800640 1 Processor -- start 2024-03-23T11:23:16.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.321+0000 7f419b800640 1 -- start start 2024-03-23T11:23:16.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.323+0000 7f419b800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419406c300 0x7f41940ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.323+0000 7f419b800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f419406c6e0 0x7f419406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.323+0000 7f419b800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41940cbc60 con 0x7f41940cafa0 2024-03-23T11:23:16.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.323+0000 7f419b800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41940cbf40 con 0x7f419406c6e0 2024-03-23T11:23:16.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.323+0000 7f419b800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41940cc220 con 0x7f419406c300 2024-03-23T11:23:16.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.323+0000 7f418be00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f41940cafa0 0x7f41940cb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37138/0 (socket says 172.21.15.62:37138) 2024-03-23T11:23:16.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.323+0000 7f418be00640 1 -- 172.21.15.62:0/2041570711 learned_addr learned my addr 172.21.15.62:0/2041570711 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.323+0000 7f4198e00640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419406c300 0x7f41940ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980681488 0 0) 0x7f41940cbc60 con 0x7f41940cafa0 2024-03-23T11:23:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4199800640 1 --2- 172.21.15.62:0/2041570711 >> v2:172.21.15.120:3300/0 conn(0x7f419406c6e0 0x7f419406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4178003520 con 0x7f41940cafa0 2024-03-23T11:23:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2063912409 0 0) 0x7f4178003520 con 0x7f41940cafa0 2024-03-23T11:23:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419406c300 msgr2=0x7f41940ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419406c300 0x7f41940ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 >> v2:172.21.15.120:3300/0 conn(0x7f419406c6e0 msgr2=0x7f419406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 --2- 172.21.15.62:0/2041570711 >> v2:172.21.15.120:3300/0 conn(0x7f419406c6e0 0x7f419406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940cc500 con 0x7f41940cafa0 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f417c001670 con 0x7f41940cafa0 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 >> v1:172.21.15.62:6789/0 conn(0x7f41940cafa0 legacy=0x7f41940cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419406c300 0x7f4178003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4178003f40 0x7f41780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4178004d00 0x7f4178005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41940cbf40 con 0x7f4178004d00 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41940cc220 con 0x7f4178003f40 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.324+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417800b980 con 0x7f419406c300 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.325+0000 7f4199800640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4178004d00 0x7f4178005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.325+0000 7f4198e00640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419406c300 0x7f4178003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.325+0000 7f418be00640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4178003f40 0x7f41780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.325+0000 7f4199800640 1 -- 172.21.15.62:0/2041570711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419406c300 msgr2=0x7f4178003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.325+0000 7f4199800640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f419406c300 0x7f4178003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.325+0000 7f4199800640 1 -- 172.21.15.62:0/2041570711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4178003f40 msgr2=0x7f41780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.325+0000 7f4199800640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4178003f40 0x7f41780047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.325+0000 7f4199800640 1 -- 172.21.15.62:0/2041570711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4178003520 con 0x7f4178004d00 2024-03-23T11:23:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.326+0000 7f4199800640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4178004d00 0x7f4178005180 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f418c00e170 tx=0x7f41780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58cdbd4af38a40d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.326+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c02cc20 con 0x7f4178004d00 2024-03-23T11:23:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.326+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c00f040 con 0x7f4178004d00 2024-03-23T11:23:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.326+0000 7f4193e00640 1 -- 172.21.15.62:0/2041570711 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c034440 con 0x7f4178004d00 2024-03-23T11:23:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.327+0000 7f419b800640 1 -- 172.21.15.62:0/2041570711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4178004d00 msgr2=0x7f4178005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.327+0000 7f419b800640 1 --2- 172.21.15.62:0/2041570711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4178004d00 0x7f4178005180 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f418c00e170 tx=0x7f41780057f0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.327+0000 7f4199800640 1 -- 172.21.15.62:0/2041570711 reap_dead start 2024-03-23T11:23:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.327+0000 7f419b800640 1 -- 172.21.15.62:0/2041570711 shutdown_connections 2024-03-23T11:23:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.327+0000 7f419b800640 1 -- 172.21.15.62:0/2041570711 >> 172.21.15.62:0/2041570711 conn(0x7f41940b02e0 msgr2=0x7f41940b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.327+0000 7f419b800640 1 -- 172.21.15.62:0/2041570711 shutdown_connections 2024-03-23T11:23:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.327+0000 7f419b800640 1 -- 172.21.15.62:0/2041570711 wait complete. 2024-03-23T11:23:16.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.329+0000 7f419b800640 1 Processor -- start 2024-03-23T11:23:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.329+0000 7f419b800640 1 -- start start 2024-03-23T11:23:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.330+0000 7f419b800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4178004d00 0x7f4194159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.330+0000 7f419b800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4194157120 0x7f419413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.330+0000 7f419b800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41940ac8f0 0x7f41940acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.330+0000 7f419b800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4194157bf0 con 0x7f41940ac8f0 2024-03-23T11:23:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.330+0000 7f419b800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419415c110 con 0x7f4194157120 2024-03-23T11:23:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.330+0000 7f419b800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41941406f0 con 0x7f4178004d00 2024-03-23T11:23:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.330+0000 7f4199800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41940ac8f0 0x7f41940acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.330+0000 7f4199800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41940ac8f0 0x7f41940acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41556/0 (socket says 172.21.15.62:41556) 2024-03-23T11:23:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.330+0000 7f4199800640 1 -- 172.21.15.62:0/2043754730 learned_addr learned my addr 172.21.15.62:0/2043754730 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.331+0000 7f418be00640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4194157120 0x7f419413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.331+0000 7f4198e00640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4178004d00 0x7f4194159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.331+0000 7f4199800640 1 -- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4178004d00 msgr2=0x7f4194159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.331+0000 7f4199800640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4178004d00 0x7f4194159500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.331+0000 7f4199800640 1 -- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4194157120 msgr2=0x7f419413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.331+0000 7f4199800640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4194157120 0x7f419413c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.331+0000 7f4199800640 1 -- 172.21.15.62:0/2043754730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940ad290 con 0x7f41940ac8f0 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.331+0000 7f4199800640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41940ac8f0 0x7f41940acd50 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f418c00e170 tx=0x7f418c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.332+0000 7f4191600640 1 -- 172.21.15.62:0/2043754730 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c043070 con 0x7f41940ac8f0 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.332+0000 7f4191600640 1 -- 172.21.15.62:0/2043754730 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c034b50 con 0x7f41940ac8f0 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.332+0000 7f4191600640 1 -- 172.21.15.62:0/2043754730 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c003990 con 0x7f41940ac8f0 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.332+0000 7f419b800640 1 -- 172.21.15.62:0/2043754730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419415c970 con 0x7f41940ac8f0 2024-03-23T11:23:16.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.332+0000 7f419b800640 1 -- 172.21.15.62:0/2043754730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419415c320 con 0x7f41940ac8f0 2024-03-23T11:23:16.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.333+0000 7f419b800640 1 -- 172.21.15.62:0/2043754730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419415cb00 con 0x7f41940ac8f0 2024-03-23T11:23:16.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.334+0000 7f4191600640 1 -- 172.21.15.62:0/2043754730 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f418c03e440 con 0x7f41940ac8f0 2024-03-23T11:23:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.335+0000 7f4191600640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f417006a7d0 0x7f417006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.335+0000 7f4191600640 1 -- 172.21.15.62:0/2043754730 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f418c0aa730 con 0x7f41940ac8f0 2024-03-23T11:23:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.336+0000 7f4198e00640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f417006a7d0 0x7f417006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.337+0000 7f4198e00640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f417006a7d0 0x7f417006cc90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4184009950 tx=0x7f4184008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.340+0000 7f4191600640 1 -- 172.21.15.62:0/2043754730 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f418c075b40 con 0x7f41940ac8f0 2024-03-23T11:23:16.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.581+0000 7f419b800640 1 -- 172.21.15.62:0/2043754730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4194066c90 con 0x7f41940ac8f0 2024-03-23T11:23:16.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.581+0000 7f4191600640 1 -- 172.21.15.62:0/2043754730 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f418c079160 con 0x7f41940ac8f0 2024-03-23T11:23:16.578 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:16.578 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-03-23T11:23:16.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 -- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f417006a7d0 msgr2=0x7f417006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f417006a7d0 0x7f417006cc90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4184009950 tx=0x7f4184008040 comp rx=0 tx=0).stop 2024-03-23T11:23:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 -- 172.21.15.62:0/2043754730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41940ac8f0 msgr2=0x7f41940acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41940ac8f0 0x7f41940acd50 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f418c00e170 tx=0x7f418c009750 comp rx=0 tx=0).stop 2024-03-23T11:23:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 -- 172.21.15.62:0/2043754730 shutdown_connections 2024-03-23T11:23:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f417006a7d0 0x7f417006cc90 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41940ac8f0 0x7f41940acd50 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4194157120 0x7f419413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 --2- 172.21.15.62:0/2043754730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4178004d00 0x7f4194159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 -- 172.21.15.62:0/2043754730 >> 172.21.15.62:0/2043754730 conn(0x7f41940b02e0 msgr2=0x7f41940abbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 -- 172.21.15.62:0/2043754730 shutdown_connections 2024-03-23T11:23:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.584+0000 7f419b800640 1 -- 172.21.15.62:0/2043754730 wait complete. 2024-03-23T11:23:16.600 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-23T11:23:16.600 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub start / repair,recursive,force 2024-03-23T11:23:16.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.829+0000 7fc676a00640 1 Processor -- start 2024-03-23T11:23:16.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.829+0000 7fc676a00640 1 -- start start 2024-03-23T11:23:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.831+0000 7fc676a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc67006c890 0x7fc6700b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.831+0000 7fc676a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc67015cd50 0x7fc67015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.831+0000 7fc676a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc6700ce690 0x7fc6700ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.831+0000 7fc676a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6700b20f0 con 0x7fc6700ce690 2024-03-23T11:23:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.831+0000 7fc676a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6700cf040 con 0x7fc67006c890 2024-03-23T11:23:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.831+0000 7fc676a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6700b1160 con 0x7fc67015cd50 2024-03-23T11:23:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.831+0000 7fc676000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc6700ce690 0x7fc6700ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.831+0000 7fc676000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc6700ce690 0x7fc6700ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41570/0 (socket says 172.21.15.62:41570) 2024-03-23T11:23:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.831+0000 7fc676000640 1 -- 172.21.15.62:0/1490150829 learned_addr learned my addr 172.21.15.62:0/1490150829 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.832+0000 7fc675600640 1 --2- 172.21.15.62:0/1490150829 >> v2:172.21.15.120:3300/0 conn(0x7fc67006c890 0x7fc6700b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.832+0000 7fc674c00640 1 --2- 172.21.15.62:0/1490150829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc67015cd50 0x7fc67015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.832+0000 7fc676000640 1 -- 172.21.15.62:0/1490150829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc67015cd50 msgr2=0x7fc67015d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.832+0000 7fc676000640 1 --2- 172.21.15.62:0/1490150829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc67015cd50 0x7fc67015d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.832+0000 7fc676000640 1 -- 172.21.15.62:0/1490150829 >> v2:172.21.15.120:3300/0 conn(0x7fc67006c890 msgr2=0x7fc6700b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.832+0000 7fc676000640 1 --2- 172.21.15.62:0/1490150829 >> v2:172.21.15.120:3300/0 conn(0x7fc67006c890 0x7fc6700b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.832+0000 7fc676000640 1 -- 172.21.15.62:0/1490150829 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67015d840 con 0x7fc6700ce690 2024-03-23T11:23:16.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.833+0000 7fc676000640 1 --2- 172.21.15.62:0/1490150829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6700ce690 0x7fc6700ceb00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc6700cf250 tx=0x7fc66802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab7ea2c95ff778a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.833+0000 7fc66fe00640 1 -- 172.21.15.62:0/1490150829 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66802ca80 con 0x7fc6700ce690 2024-03-23T11:23:16.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.833+0000 7fc66fe00640 1 -- 172.21.15.62:0/1490150829 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66802cbe0 con 0x7fc6700ce690 2024-03-23T11:23:16.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.833+0000 7fc66fe00640 1 -- 172.21.15.62:0/1490150829 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680343f0 con 0x7fc6700ce690 2024-03-23T11:23:16.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.834+0000 7fc676a00640 1 -- 172.21.15.62:0/1490150829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6700ce690 msgr2=0x7fc6700ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.834+0000 7fc676a00640 1 --2- 172.21.15.62:0/1490150829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6700ce690 0x7fc6700ceb00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc6700cf250 tx=0x7fc66802bee0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.834+0000 7fc676a00640 1 -- 172.21.15.62:0/1490150829 shutdown_connections 2024-03-23T11:23:16.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.834+0000 7fc676a00640 1 --2- 172.21.15.62:0/1490150829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6700ce690 0x7fc6700ceb00 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.834+0000 7fc676a00640 1 --2- 172.21.15.62:0/1490150829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc67015cd50 0x7fc67015d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.834+0000 7fc676a00640 1 --2- 172.21.15.62:0/1490150829 >> v2:172.21.15.120:3300/0 conn(0x7fc67006c890 0x7fc6700b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.834+0000 7fc676a00640 1 -- 172.21.15.62:0/1490150829 >> 172.21.15.62:0/1490150829 conn(0x7fc6700b0280 msgr2=0x7fc6700b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.834+0000 7fc676a00640 1 -- 172.21.15.62:0/1490150829 shutdown_connections 2024-03-23T11:23:16.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.834+0000 7fc676a00640 1 -- 172.21.15.62:0/1490150829 wait complete. 2024-03-23T11:23:16.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.836+0000 7fc676a00640 1 Processor -- start 2024-03-23T11:23:16.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.836+0000 7fc676a00640 1 -- start start 2024-03-23T11:23:16.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.837+0000 7fc676a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc67006c890 0x7fc670147570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.837+0000 7fc676a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6700ce690 0x7fc670070910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.837+0000 7fc676a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc67015cd50 0x7fc670070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.837+0000 7fc676a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc67015fd30 con 0x7fc6700ce690 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.837+0000 7fc676a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc670147ab0 con 0x7fc67006c890 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.837+0000 7fc676a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc67015ef70 con 0x7fc67015cd50 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.837+0000 7fc674c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6700ce690 0x7fc670070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.838+0000 7fc674c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6700ce690 0x7fc670070910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41582/0 (socket says 172.21.15.62:41582) 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.838+0000 7fc674c00640 1 -- 172.21.15.62:0/2048876193 learned_addr learned my addr 172.21.15.62:0/2048876193 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.838+0000 7fc676000640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc67015cd50 0x7fc670070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.838+0000 7fc675600640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc67006c890 0x7fc670147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.838+0000 7fc674c00640 1 -- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc67015cd50 msgr2=0x7fc670070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.838+0000 7fc674c00640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc67015cd50 0x7fc670070e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.838+0000 7fc674c00640 1 -- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc67006c890 msgr2=0x7fc670147570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.838+0000 7fc674c00640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc67006c890 0x7fc670147570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.838+0000 7fc674c00640 1 -- 172.21.15.62:0/2048876193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc668009b30 con 0x7fc6700ce690 2024-03-23T11:23:16.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.838+0000 7fc674c00640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6700ce690 0x7fc670070910 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00d5f0 tx=0x7fc65c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.839+0000 7fc66d600640 1 -- 172.21.15.62:0/2048876193 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c0125f0 con 0x7fc6700ce690 2024-03-23T11:23:16.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.839+0000 7fc66d600640 1 -- 172.21.15.62:0/2048876193 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c012750 con 0x7fc6700ce690 2024-03-23T11:23:16.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.839+0000 7fc66d600640 1 -- 172.21.15.62:0/2048876193 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c012b00 con 0x7fc6700ce690 2024-03-23T11:23:16.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.839+0000 7fc676a00640 1 -- 172.21.15.62:0/2048876193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc67015f250 con 0x7fc6700ce690 2024-03-23T11:23:16.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.839+0000 7fc676a00640 1 -- 172.21.15.62:0/2048876193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc67015f810 con 0x7fc6700ce690 2024-03-23T11:23:16.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.841+0000 7fc66d600640 1 -- 172.21.15.62:0/2048876193 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc65c012c60 con 0x7fc6700ce690 2024-03-23T11:23:16.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.842+0000 7fc66d600640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc65006a7a0 0x7fc65006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.842+0000 7fc66d600640 1 -- 172.21.15.62:0/2048876193 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc65c089250 con 0x7fc6700ce690 2024-03-23T11:23:16.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.843+0000 7fc675600640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc65006a7a0 0x7fc65006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.843+0000 7fc67addf740 1 Processor -- start 2024-03-23T11:23:16.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.843+0000 7fc67addf740 1 -- start start 2024-03-23T11:23:16.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.844+0000 7fc675600640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc65006a7a0 0x7fc65006cc60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc6701522b0 tx=0x7fc664008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.844+0000 7fc67addf740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.844+0000 7fc67addf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f296500 0x55976f2c3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.844+0000 7fc67addf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f2c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.844+0000 7fc67addf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976f206340 con 0x55976f296500 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.844+0000 7fc67addf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976f203ad0 con 0x55976f29a060 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.844+0000 7fc67addf740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55976f1ff3c0 con 0x55976f2c64e0 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.844+0000 7fc674c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f296500 0x55976f2c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.844+0000 7fc674c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f296500 0x55976f2c3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41584/0 (socket says 172.21.15.62:41584) 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.844+0000 7fc674c00640 1 -- 172.21.15.62:0/133238857 learned_addr learned my addr 172.21.15.62:0/133238857 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.844+0000 7fc675600640 1 --2- 172.21.15.62:0/133238857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.845+0000 7fc676000640 1 --2- 172.21.15.62:0/133238857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f2c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.845+0000 7fc674c00640 1 -- 172.21.15.62:0/133238857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 msgr2=0x55976f2c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.845+0000 7fc674c00640 1 --2- 172.21.15.62:0/133238857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2c68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.845+0000 7fc674c00640 1 -- 172.21.15.62:0/133238857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 msgr2=0x55976f2c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.845+0000 7fc674c00640 1 --2- 172.21.15.62:0/133238857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f2c3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.845+0000 7fc674c00640 1 -- 172.21.15.62:0/133238857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55976f1a1020 con 0x55976f296500 2024-03-23T11:23:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.845+0000 7fc676000640 1 --2- 172.21.15.62:0/133238857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f2c3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.845+0000 7fc674c00640 1 --2- 172.21.15.62:0/133238857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f296500 0x55976f2c3750 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc65c057f90 tx=0x7fc65c057f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d8e7e7836138c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.845+0000 7fc662000640 1 -- 172.21.15.62:0/133238857 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c06e070 con 0x55976f296500 2024-03-23T11:23:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.845+0000 7fc662000640 1 -- 172.21.15.62:0/133238857 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c058290 con 0x55976f296500 2024-03-23T11:23:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.845+0000 7fc662000640 1 -- 172.21.15.62:0/133238857 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c05c0c0 con 0x55976f296500 2024-03-23T11:23:16.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.846+0000 7fc67addf740 1 -- 172.21.15.62:0/133238857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f296500 msgr2=0x55976f2c3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.846+0000 7fc67addf740 1 --2- 172.21.15.62:0/133238857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f296500 0x55976f2c3750 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc65c057f90 tx=0x7fc65c057f00 comp rx=0 tx=0).stop 2024-03-23T11:23:16.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.846+0000 7fc67addf740 1 -- 172.21.15.62:0/133238857 shutdown_connections 2024-03-23T11:23:16.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.846+0000 7fc67addf740 1 --2- 172.21.15.62:0/133238857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f296500 0x55976f2c3750 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.846+0000 7fc67addf740 1 --2- 172.21.15.62:0/133238857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f2c3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.846+0000 7fc67addf740 1 --2- 172.21.15.62:0/133238857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2c68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.846+0000 7fc67addf740 1 -- 172.21.15.62:0/133238857 >> 172.21.15.62:0/133238857 conn(0x55976f298770 msgr2=0x55976f2e12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.846+0000 7fc67addf740 1 -- 172.21.15.62:0/133238857 shutdown_connections 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.846+0000 7fc67addf740 1 -- 172.21.15.62:0/133238857 wait complete. 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.847+0000 7fc67addf740 1 -- start start 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.847+0000 7fc67addf740 1 Processor -- start 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.848+0000 7fc67addf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f29a060 0x55976f2bae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.848+0000 7fc67addf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f2c64e0 0x55976f2c8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.848+0000 7fc67addf740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2bb970 0x55976f2bf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.848+0000 7fc67addf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976f1bd7f0 con 0x55976f29a060 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.848+0000 7fc67addf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976f1bd490 con 0x55976f2c64e0 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.848+0000 7fc67addf740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55976ef67180 con 0x55976f2bb970 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.848+0000 7fc67addf740 10 client.0 _ll_register_callbacks cb 0x55976f1f8a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.848+0000 7fc675600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f29a060 0x55976f2bae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.848+0000 7fc675600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f29a060 0x55976f2bae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41586/0 (socket says 172.21.15.62:41586) 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.848+0000 7fc675600640 1 -- 172.21.15.62:0/2064101836 learned_addr learned my addr 172.21.15.62:0/2064101836 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.849+0000 7fc674c00640 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f2c64e0 0x55976f2c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.849+0000 7fc676000640 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2bb970 0x55976f2bf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.849+0000 7fc675600640 1 -- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2bb970 msgr2=0x55976f2bf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.849+0000 7fc675600640 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2bb970 0x55976f2bf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.849+0000 7fc675600640 1 -- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f2c64e0 msgr2=0x55976f2c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.849+0000 7fc675600640 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f2c64e0 0x55976f2c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.849+0000 7fc675600640 1 -- 172.21.15.62:0/2064101836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc67015f810 con 0x55976f29a060 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.849+0000 7fc674c00640 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f2c64e0 0x55976f2c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.849+0000 7fc675600640 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f29a060 0x55976f2bae60 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc664015840 tx=0x7fc664004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7e6c134ddc58bd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.849+0000 7fc656a00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc656a00640 1 -- 172.21.15.62:0/2064101836 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664016ba0 con 0x55976f29a060 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc656a00640 1 -- 172.21.15.62:0/2064101836 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc664016d00 con 0x55976f29a060 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc67addf740 1 -- 172.21.15.62:0/2064101836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55976f229560 con 0x55976f29a060 2024-03-23T11:23:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc656a00640 1 -- 172.21.15.62:0/2064101836 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc664022420 con 0x55976f29a060 2024-03-23T11:23:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc67addf740 10 client.6348 fetch_fsmap learned FSMap version 34 2024-03-23T11:23:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc67addf740 1 -- 172.21.15.62:0/2064101836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55976f1fda40 con 0x55976f29a060 2024-03-23T11:23:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc656a00640 1 -- 172.21.15.62:0/2064101836 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55976f229560 con 0x55976f29a060 2024-03-23T11:23:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc656a00640 1 -- 172.21.15.62:0/2064101836 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc664022580 con 0x55976f29a060 2024-03-23T11:23:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc67addf740 10 client.6348 fetch_fsmap finished waiting for FSMap version 34 2024-03-23T11:23:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc67addf740 10 client.6348 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:23:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc67addf740 20 client.6348 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc67addf740 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55976f2c9290 0x55976f2c9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc67addf740 4 client.6348 mds_command: new command op to 6081 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:23:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.850+0000 7fc67addf740 1 -- 172.21.15.62:0/2064101836 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55976f1f1ef0 con 0x55976f2c9290 2024-03-23T11:23:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.851+0000 7fc674c00640 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55976f2c9290 0x55976f2c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.852+0000 7fc674c00640 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55976f2c9290 0x55976f2c9670 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5adad951c0792b7f server_cookie=1f8d2e63dd8cf6ec in_seq=0 out_seq=0 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.852+0000 7fc656a00640 10 client.6348 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc656a00640 1 -- 172.21.15.62:0/2064101836 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55976f1f1ef0 con 0x55976f2c9290 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc656a00640 10 client.6348 handle_command_reply: tid=0 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc67addf740 1 client.6348 shutdown 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc67addf740 1 -- 172.21.15.62:0/2064101836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f29a060 msgr2=0x55976f2bae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc67addf740 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f29a060 0x55976f2bae60 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc664015840 tx=0x7fc664004730 comp rx=0 tx=0).stop 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc67addf740 1 -- 172.21.15.62:0/2064101836 shutdown_connections 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc67addf740 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55976f2c9290 0x55976f2c9670 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc67addf740 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2bb970 0x55976f2bf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc67addf740 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f2c64e0 0x55976f2c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc67addf740 1 --2- 172.21.15.62:0/2064101836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f29a060 0x55976f2bae60 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc67addf740 1 -- 172.21.15.62:0/2064101836 >> 172.21.15.62:0/2064101836 conn(0x55976f296500 msgr2=0x55976f29a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc656a00640 0 client.6348 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc67addf740 1 -- 172.21.15.62:0/2064101836 shutdown_connections 2024-03-23T11:23:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.854+0000 7fc67addf740 1 -- 172.21.15.62:0/2064101836 wait complete. 2024-03-23T11:23:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.855+0000 7fc67addf740 20 client.6348 trim_cache size 0 max 16384 2024-03-23T11:23:16.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.870+0000 7fc67addf740 1 Processor -- start 2024-03-23T11:23:16.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.871+0000 7fc67addf740 1 -- start start 2024-03-23T11:23:16.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.871+0000 7fc67addf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f2c7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.871+0000 7fc67addf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f29bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.871+0000 7fc67addf740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2954a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.871+0000 7fc67addf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976f2b9260 con 0x55976f2bb970 2024-03-23T11:23:16.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.871+0000 7fc67addf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976f2c7780 con 0x55976f29a060 2024-03-23T11:23:16.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.871+0000 7fc67addf740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55976f2c6af0 con 0x55976f2c64e0 2024-03-23T11:23:16.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc674c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f29bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc674c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f29bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41602/0 (socket says 172.21.15.62:41602) 2024-03-23T11:23:16.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc674c00640 1 -- 172.21.15.62:0/2744669681 learned_addr learned my addr 172.21.15.62:0/2744669681 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc675600640 1 --2- 172.21.15.62:0/2744669681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f2c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc676000640 1 --2- 172.21.15.62:0/2744669681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2954a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc674c00640 1 -- 172.21.15.62:0/2744669681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 msgr2=0x55976f2954a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc674c00640 1 --2- 172.21.15.62:0/2744669681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2954a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc674c00640 1 -- 172.21.15.62:0/2744669681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 msgr2=0x55976f2c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc674c00640 1 --2- 172.21.15.62:0/2744669681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f2c7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc674c00640 1 -- 172.21.15.62:0/2744669681 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55976f19f860 con 0x55976f2bb970 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc675600640 1 --2- 172.21.15.62:0/2744669681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f2c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.872+0000 7fc676000640 1 --2- 172.21.15.62:0/2744669681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.873+0000 7fc674c00640 1 --2- 172.21.15.62:0/2744669681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f29bbf0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc65c060150 tx=0x7fc65c05e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e80eafc09c1b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.873+0000 7fc662a00640 1 -- 172.21.15.62:0/2744669681 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c06e070 con 0x55976f2bb970 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.873+0000 7fc662a00640 1 -- 172.21.15.62:0/2744669681 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c060680 con 0x55976f2bb970 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.873+0000 7fc662a00640 1 -- 172.21.15.62:0/2744669681 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c06fea0 con 0x55976f2bb970 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.873+0000 7fc67addf740 1 -- 172.21.15.62:0/2744669681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 msgr2=0x55976f29bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.874+0000 7fc67addf740 1 --2- 172.21.15.62:0/2744669681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f29bbf0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc65c060150 tx=0x7fc65c05e050 comp rx=0 tx=0).stop 2024-03-23T11:23:16.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.874+0000 7fc67addf740 1 -- 172.21.15.62:0/2744669681 shutdown_connections 2024-03-23T11:23:16.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.874+0000 7fc67addf740 1 --2- 172.21.15.62:0/2744669681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f29bbf0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.874+0000 7fc67addf740 1 --2- 172.21.15.62:0/2744669681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.874+0000 7fc67addf740 1 --2- 172.21.15.62:0/2744669681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f2c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.874+0000 7fc67addf740 1 -- 172.21.15.62:0/2744669681 >> 172.21.15.62:0/2744669681 conn(0x55976f296500 msgr2=0x55976f2df120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.874+0000 7fc67addf740 1 -- 172.21.15.62:0/2744669681 shutdown_connections 2024-03-23T11:23:16.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.874+0000 7fc67addf740 1 -- 172.21.15.62:0/2744669681 wait complete. 2024-03-23T11:23:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.874+0000 7fc67addf740 1 -- start start 2024-03-23T11:23:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.875+0000 7fc67addf740 1 Processor -- start 2024-03-23T11:23:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc67addf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f295a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc67addf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f330520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc67addf740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc67addf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976f2c8d00 con 0x55976f2bb970 2024-03-23T11:23:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc67addf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55976f330a60 con 0x55976f29a060 2024-03-23T11:23:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc67addf740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55976f2f3d30 con 0x55976f2c64e0 2024-03-23T11:23:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc67addf740 10 client.0 _ll_register_callbacks cb 0x7fc664015490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc674c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f330520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc674c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f330520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41608/0 (socket says 172.21.15.62:41608) 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc674c00640 1 -- 172.21.15.62:0/2833144093 learned_addr learned my addr 172.21.15.62:0/2833144093 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc674c00640 1 -- 172.21.15.62:0/2833144093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 msgr2=0x55976f2c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc674c00640 1 --2- 172.21.15.62:0/2833144093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc674c00640 1 -- 172.21.15.62:0/2833144093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 msgr2=0x55976f295a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc674c00640 1 --2- 172.21.15.62:0/2833144093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f295a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.876+0000 7fc674c00640 1 -- 172.21.15.62:0/2833144093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55976f1a1020 con 0x55976f2bb970 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc674c00640 1 --2- 172.21.15.62:0/2833144093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f330520 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0581d0 tx=0x7fc65c089040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb2d3a74c02a655 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc656a00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc656a00640 1 -- 172.21.15.62:0/2833144093 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c06e030 con 0x55976f2bb970 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc656a00640 1 -- 172.21.15.62:0/2833144093 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc65c06f070 con 0x55976f2bb970 2024-03-23T11:23:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc67addf740 1 -- 172.21.15.62:0/2833144093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55976f2e01d0 con 0x55976f2bb970 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc656a00640 1 -- 172.21.15.62:0/2833144093 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc65c06fea0 con 0x55976f2bb970 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc67addf740 10 client.6354 fetch_fsmap learned FSMap version 34 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc67addf740 1 -- 172.21.15.62:0/2833144093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55976f1c4e30 con 0x55976f2bb970 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc656a00640 1 -- 172.21.15.62:0/2833144093 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55976f2e01d0 con 0x55976f2bb970 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc656a00640 1 -- 172.21.15.62:0/2833144093 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc65c0549a0 con 0x55976f2bb970 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc67addf740 10 client.6354 fetch_fsmap finished waiting for FSMap version 34 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc67addf740 10 client.6354 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc67addf740 20 client.6354 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc67addf740 1 --2- 172.21.15.62:0/2833144093 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55976f330bd0 0x55976f2ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc67addf740 4 client.6354 mds_command: new command op to 6081 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}] 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.877+0000 7fc67addf740 1 -- 172.21.15.62:0/2833144093 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}) v1 -- 0x7fc65c058290 con 0x55976f330bd0 2024-03-23T11:23:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.878+0000 7fc675600640 1 --2- 172.21.15.62:0/2833144093 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55976f330bd0 0x55976f2ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.879+0000 7fc675600640 1 --2- 172.21.15.62:0/2833144093 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55976f330bd0 0x55976f2ca360 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a77602494b4abd10 server_cookie=b86826f29a0ce8f in_seq=0 out_seq=0 2024-03-23T11:23:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.879+0000 7fc656a00640 10 client.6354 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc656a00640 1 -- 172.21.15.62:0/2833144093 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7fc65c058290 con 0x55976f330bd0 2024-03-23T11:23:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc656a00640 10 client.6354 handle_command_reply: tid=0 2024-03-23T11:23:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 1 client.6354 shutdown 2024-03-23T11:23:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 1 -- 172.21.15.62:0/2833144093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 msgr2=0x55976f330520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 1 --2- 172.21.15.62:0/2833144093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f330520 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0581d0 tx=0x7fc65c089040 comp rx=0 tx=0).stop 2024-03-23T11:23:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 1 -- 172.21.15.62:0/2833144093 shutdown_connections 2024-03-23T11:23:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 1 --2- 172.21.15.62:0/2833144093 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55976f330bd0 0x55976f2ca360 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 1 --2- 172.21.15.62:0/2833144093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55976f2bb970 0x55976f330520 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 1 --2- 172.21.15.62:0/2833144093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55976f2c64e0 0x55976f2c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 1 --2- 172.21.15.62:0/2833144093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55976f29a060 0x55976f295a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 1 -- 172.21.15.62:0/2833144093 >> 172.21.15.62:0/2833144093 conn(0x55976f296500 msgr2=0x55976f2c0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc656a00640 0 client.6354 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 1 -- 172.21.15.62:0/2833144093 shutdown_connections 2024-03-23T11:23:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 1 -- 172.21.15.62:0/2833144093 wait complete. 2024-03-23T11:23:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.881+0000 7fc67addf740 20 client.6354 trim_cache size 0 max 16384 2024-03-23T11:23:16.880 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:23:16.880 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-03-23T11:23:16.880 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "54dc16e5-78ff-421d-a74e-2f271da3b50d", 2024-03-23T11:23:16.880 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-03-23T11:23:16.881 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:23:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 -- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc65006a7a0 msgr2=0x7fc65006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc65006a7a0 0x7fc65006cc60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc6701522b0 tx=0x7fc664008040 comp rx=0 tx=0).stop 2024-03-23T11:23:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 -- 172.21.15.62:0/2048876193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6700ce690 msgr2=0x7fc670070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6700ce690 0x7fc670070910 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00d5f0 tx=0x7fc65c00dab0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 -- 172.21.15.62:0/2048876193 shutdown_connections 2024-03-23T11:23:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc65006a7a0 0x7fc65006cc60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6700ce690 0x7fc670070910 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc67015cd50 0x7fc670070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 --2- 172.21.15.62:0/2048876193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc67006c890 0x7fc670147570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 -- 172.21.15.62:0/2048876193 >> 172.21.15.62:0/2048876193 conn(0x7fc6700b0280 msgr2=0x7fc6700aeec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 -- 172.21.15.62:0/2048876193 shutdown_connections 2024-03-23T11:23:16.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:16.882+0000 7fc676a00640 1 -- 172.21.15.62:0/2048876193 wait complete. 2024-03-23T11:23:16.883 INFO:tasks.ceph.mds.b.smithi120.stderr:2024-03-23T11:23:16.881+0000 7f3f37600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-03-23T11:22:42.917205+0000 1=1+0) (iversion lock) caps={6228=pAsLsXsFs/-@2},l=6228(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x5565aeb39b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-23T11:23:16.896 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-23T11:23:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.122+0000 7f693e200640 1 Processor -- start 2024-03-23T11:23:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.122+0000 7f693e200640 1 -- start start 2024-03-23T11:23:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.124+0000 7f693e200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f69380b25a0 0x7f69380b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.124+0000 7f693e200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f69380cb700 0x7f693806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.124+0000 7f693e200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69380cbb10 con 0x7f6938142f10 2024-03-23T11:23:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.124+0000 7f693e200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69380cbd40 con 0x7f69380cb700 2024-03-23T11:23:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.124+0000 7f693e200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69380b2ec0 con 0x7f69380b25a0 2024-03-23T11:23:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.124+0000 7f6937e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6938142f10 0x7f69380caff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37150/0 (socket says 172.21.15.62:37150) 2024-03-23T11:23:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.124+0000 7f6937e00640 1 -- 172.21.15.62:0/1807636380 learned_addr learned my addr 172.21.15.62:0/1807636380 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.124+0000 7f693d800640 1 --2- 172.21.15.62:0/1807636380 >> v2:172.21.15.62:3300/0 conn(0x7f69380cb700 0x7f693806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.124+0000 7f693ce00640 1 --2- 172.21.15.62:0/1807636380 >> v2:172.21.15.120:3300/0 conn(0x7f69380b25a0 0x7f69380b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.125+0000 7f6937400640 1 -- 172.21.15.62:0/1807636380 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509857552 0 0) 0x7f69380cbb10 con 0x7f6938142f10 2024-03-23T11:23:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.125+0000 7f6937400640 1 -- 172.21.15.62:0/1807636380 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6920003520 con 0x7f6938142f10 2024-03-23T11:23:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.125+0000 7f693d800640 1 -- 172.21.15.62:0/1807636380 >> v2:172.21.15.120:3300/0 conn(0x7f69380b25a0 msgr2=0x7f69380b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.125+0000 7f693d800640 1 --2- 172.21.15.62:0/1807636380 >> v2:172.21.15.120:3300/0 conn(0x7f69380b25a0 0x7f69380b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.125+0000 7f693d800640 1 -- 172.21.15.62:0/1807636380 >> v1:172.21.15.62:6789/0 conn(0x7f6938142f10 legacy=0x7f69380caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.125+0000 7f693d800640 1 -- 172.21.15.62:0/1807636380 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6938159d00 con 0x7f69380cb700 2024-03-23T11:23:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.125+0000 7f693ce00640 1 --2- 172.21.15.62:0/1807636380 >> v2:172.21.15.120:3300/0 conn(0x7f69380b25a0 0x7f69380b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:23:17.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.126+0000 7f693d800640 1 --2- 172.21.15.62:0/1807636380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69380cb700 0x7f693806c7e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f693806ad90 tx=0x7f693002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2952411f09b95c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.126+0000 7f6937400640 1 -- 172.21.15.62:0/1807636380 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693002cc70 con 0x7f69380cb700 2024-03-23T11:23:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.126+0000 7f6937400640 1 -- 172.21.15.62:0/1807636380 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693002cdd0 con 0x7f69380cb700 2024-03-23T11:23:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.126+0000 7f6937400640 1 -- 172.21.15.62:0/1807636380 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6930034660 con 0x7f69380cb700 2024-03-23T11:23:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.127+0000 7f693e200640 1 -- 172.21.15.62:0/1807636380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69380cb700 msgr2=0x7f693806c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.127+0000 7f693e200640 1 --2- 172.21.15.62:0/1807636380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69380cb700 0x7f693806c7e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f693806ad90 tx=0x7f693002bec0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.127+0000 7f693e200640 1 -- 172.21.15.62:0/1807636380 shutdown_connections 2024-03-23T11:23:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.127+0000 7f693e200640 1 --2- 172.21.15.62:0/1807636380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69380cb700 0x7f693806c7e0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.127+0000 7f693e200640 1 --2- 172.21.15.62:0/1807636380 >> v2:172.21.15.120:3300/0 conn(0x7f69380b25a0 0x7f69380b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.127+0000 7f693e200640 1 -- 172.21.15.62:0/1807636380 >> 172.21.15.62:0/1807636380 conn(0x7f69380b0090 msgr2=0x7f69380b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.127+0000 7f693e200640 1 -- 172.21.15.62:0/1807636380 shutdown_connections 2024-03-23T11:23:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.127+0000 7f693e200640 1 -- 172.21.15.62:0/1807636380 wait complete. 2024-03-23T11:23:17.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.129+0000 7f693e200640 1 Processor -- start 2024-03-23T11:23:17.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.129+0000 7f693e200640 1 -- start start 2024-03-23T11:23:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.130+0000 7f693e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69380b25a0 0x7f693815bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.130+0000 7f693e200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69380cb700 0x7f69380af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.130+0000 7f693e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f693815c460 0x7f69380af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.130+0000 7f693e200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69380cb2b0 con 0x7f693815c460 2024-03-23T11:23:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.130+0000 7f693e200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693815c280 con 0x7f69380cb700 2024-03-23T11:23:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.130+0000 7f693e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693815c940 con 0x7f69380b25a0 2024-03-23T11:23:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.130+0000 7f693d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f693815c460 0x7f69380af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.130+0000 7f693d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f693815c460 0x7f69380af590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41622/0 (socket says 172.21.15.62:41622) 2024-03-23T11:23:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.130+0000 7f693d800640 1 -- 172.21.15.62:0/3266141419 learned_addr learned my addr 172.21.15.62:0/3266141419 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.131+0000 7f693ce00640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69380b25a0 0x7f693815bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.131+0000 7f6937e00640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69380cb700 0x7f69380af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.131+0000 7f693d800640 1 -- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69380b25a0 msgr2=0x7f693815bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.131+0000 7f693d800640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69380b25a0 0x7f693815bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.131+0000 7f693d800640 1 -- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69380cb700 msgr2=0x7f69380af050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:17.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.131+0000 7f693d800640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69380cb700 0x7f69380af050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.131+0000 7f693d800640 1 -- 172.21.15.62:0/3266141419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6930009a70 con 0x7f693815c460 2024-03-23T11:23:17.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.131+0000 7f6937e00640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69380cb700 0x7f69380af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:17.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.131+0000 7f693d800640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f693815c460 0x7f69380af590 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6930009e80 tx=0x7f6930004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:17.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.131+0000 7f6934c00640 1 -- 172.21.15.62:0/3266141419 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6930040650 con 0x7f693815c460 2024-03-23T11:23:17.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.132+0000 7f6934c00640 1 -- 172.21.15.62:0/3266141419 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69300407b0 con 0x7f693815c460 2024-03-23T11:23:17.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.132+0000 7f693e200640 1 -- 172.21.15.62:0/3266141419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693814f650 con 0x7f693815c460 2024-03-23T11:23:17.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.132+0000 7f6934c00640 1 -- 172.21.15.62:0/3266141419 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6930040b60 con 0x7f693815c460 2024-03-23T11:23:17.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.132+0000 7f693e200640 1 -- 172.21.15.62:0/3266141419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693814fc10 con 0x7f693815c460 2024-03-23T11:23:17.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.134+0000 7f6934c00640 1 -- 172.21.15.62:0/3266141419 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6930040cc0 con 0x7f693815c460 2024-03-23T11:23:17.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.135+0000 7f6934c00640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f691806a7a0 0x7f691806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.135+0000 7f6934c00640 1 -- 172.21.15.62:0/3266141419 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69300abb70 con 0x7f693815c460 2024-03-23T11:23:17.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.135+0000 7f693ce00640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f691806a7a0 0x7f691806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.135+0000 7f6942409740 1 Processor -- start 2024-03-23T11:23:17.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.136+0000 7f6942409740 1 -- start start 2024-03-23T11:23:17.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.136+0000 7f693ce00640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f691806a7a0 0x7f691806cc60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6938159410 tx=0x7f692c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6942409740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347af0 0x55dba634fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6942409740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6350460 0x55dba6327c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6942409740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6347ed0 0x55dba6328160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6942409740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba62e3cb0 con 0x55dba6350460 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6942409740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba6292500 con 0x55dba6347af0 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6942409740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dba628fc90 con 0x55dba6347ed0 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6937e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6350460 0x55dba6327c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6937e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6350460 0x55dba6327c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41638/0 (socket says 172.21.15.62:41638) 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6937e00640 1 -- 172.21.15.62:0/3100233002 learned_addr learned my addr 172.21.15.62:0/3100233002 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6937e00640 1 -- 172.21.15.62:0/3100233002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6347ed0 msgr2=0x55dba6328160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6937e00640 1 --2- 172.21.15.62:0/3100233002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6347ed0 0x55dba6328160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f693ce00640 1 --2- 172.21.15.62:0/3100233002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347af0 0x55dba634fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6937e00640 1 -- 172.21.15.62:0/3100233002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347af0 msgr2=0x55dba634fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.137+0000 7f6937e00640 1 --2- 172.21.15.62:0/3100233002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347af0 0x55dba634fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.138+0000 7f6937e00640 1 -- 172.21.15.62:0/3100233002 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dba623d360 con 0x55dba6350460 2024-03-23T11:23:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.138+0000 7f6937e00640 1 --2- 172.21.15.62:0/3100233002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6350460 0x55dba6327c20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6924000f60 tx=0x7f69240019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb324197f41b1edb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.138+0000 7f6929600640 1 -- 172.21.15.62:0/3100233002 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692400b070 con 0x55dba6350460 2024-03-23T11:23:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.138+0000 7f6929600640 1 -- 172.21.15.62:0/3100233002 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6924005ce0 con 0x55dba6350460 2024-03-23T11:23:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.138+0000 7f6929600640 1 -- 172.21.15.62:0/3100233002 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69240072d0 con 0x55dba6350460 2024-03-23T11:23:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.139+0000 7f6942409740 1 -- 172.21.15.62:0/3100233002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6350460 msgr2=0x55dba6327c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.139+0000 7f6942409740 1 --2- 172.21.15.62:0/3100233002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6350460 0x55dba6327c20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6924000f60 tx=0x7f69240019e0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.139+0000 7f6942409740 1 -- 172.21.15.62:0/3100233002 shutdown_connections 2024-03-23T11:23:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.139+0000 7f6942409740 1 --2- 172.21.15.62:0/3100233002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6350460 0x55dba6327c20 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.139+0000 7f6942409740 1 --2- 172.21.15.62:0/3100233002 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6347ed0 0x55dba6328160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.139+0000 7f6942409740 1 --2- 172.21.15.62:0/3100233002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347af0 0x55dba634fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.139+0000 7f6942409740 1 -- 172.21.15.62:0/3100233002 >> 172.21.15.62:0/3100233002 conn(0x55dba62efb40 msgr2=0x55dba636d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.139+0000 7f6942409740 1 -- 172.21.15.62:0/3100233002 shutdown_connections 2024-03-23T11:23:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.139+0000 7f6942409740 1 -- 172.21.15.62:0/3100233002 wait complete. 2024-03-23T11:23:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.140+0000 7f6942409740 1 -- start start 2024-03-23T11:23:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.140+0000 7f6942409740 1 Processor -- start 2024-03-23T11:23:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.141+0000 7f6942409740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba6352d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.141+0000 7f6942409740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347ed0 0x55dba636c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.141+0000 7f6942409740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6350460 0x55dba636cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.141+0000 7f6942409740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba628b580 con 0x55dba6347af0 2024-03-23T11:23:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.141+0000 7f6942409740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba622ca50 con 0x55dba6347ed0 2024-03-23T11:23:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.141+0000 7f6942409740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dba622c6f0 con 0x55dba6350460 2024-03-23T11:23:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.141+0000 7f6942409740 10 client.0 _ll_register_callbacks cb 0x55dba627bed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.141+0000 7f693ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba6352d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.141+0000 7f693ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba6352d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41650/0 (socket says 172.21.15.62:41650) 2024-03-23T11:23:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.141+0000 7f693ce00640 1 -- 172.21.15.62:0/1669476515 learned_addr learned my addr 172.21.15.62:0/1669476515 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.142+0000 7f693d800640 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6350460 0x55dba636cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.142+0000 7f693ce00640 1 -- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6350460 msgr2=0x55dba636cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.142+0000 7f693ce00640 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6350460 0x55dba636cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.142+0000 7f693ce00640 1 -- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347ed0 msgr2=0x55dba636c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.142+0000 7f6937e00640 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347ed0 0x55dba636c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.142+0000 7f693ce00640 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347ed0 0x55dba636c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.142+0000 7f693ce00640 1 -- 172.21.15.62:0/1669476515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6920003520 con 0x55dba6347af0 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.142+0000 7f693ce00640 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba6352d30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f692c0156f0 tx=0x7f692c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b3ee30ece85451 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.142+0000 7f691e000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f691e000640 1 -- 172.21.15.62:0/1669476515 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c016970 con 0x55dba6347af0 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f691e000640 1 -- 172.21.15.62:0/1669476515 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f692c016ad0 con 0x55dba6347af0 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f6942409740 1 -- 172.21.15.62:0/1669476515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dba62b5fe0 con 0x55dba6347af0 2024-03-23T11:23:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f691e000640 1 -- 172.21.15.62:0/1669476515 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f692c016c90 con 0x55dba6347af0 2024-03-23T11:23:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f6942409740 10 client.6369 fetch_fsmap learned FSMap version 34 2024-03-23T11:23:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f6942409740 1 -- 172.21.15.62:0/1669476515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55dba6289c00 con 0x55dba6347af0 2024-03-23T11:23:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f691e000640 1 -- 172.21.15.62:0/1669476515 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55dba62b5fe0 con 0x55dba6347af0 2024-03-23T11:23:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f691e000640 1 -- 172.21.15.62:0/1669476515 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f692c022410 con 0x55dba6347af0 2024-03-23T11:23:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f6942409740 10 client.6369 fetch_fsmap finished waiting for FSMap version 34 2024-03-23T11:23:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f6942409740 10 client.6369 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:23:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f6942409740 20 client.6369 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f6942409740 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55dba6350d90 0x55dba6351170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f6942409740 4 client.6369 mds_command: new command op to 6081 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:23:17.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.143+0000 7f6942409740 1 -- 172.21.15.62:0/1669476515 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dba627e240 con 0x55dba6350d90 2024-03-23T11:23:17.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.144+0000 7f6937e00640 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55dba6350d90 0x55dba6351170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.145+0000 7f6937e00640 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55dba6350d90 0x55dba6351170 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2735f827f3d1da4 server_cookie=7b1166327a552894 in_seq=0 out_seq=0 2024-03-23T11:23:17.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.145+0000 7f691e000640 10 client.6369 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.146+0000 7f691e000640 1 -- 172.21.15.62:0/1669476515 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55dba627e240 con 0x55dba6350d90 2024-03-23T11:23:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.146+0000 7f691e000640 10 client.6369 handle_command_reply: tid=0 2024-03-23T11:23:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.146+0000 7f6942409740 1 client.6369 shutdown 2024-03-23T11:23:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f6942409740 1 -- 172.21.15.62:0/1669476515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 msgr2=0x55dba6352d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f6942409740 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba6352d30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f692c0156f0 tx=0x7f692c004730 comp rx=0 tx=0).stop 2024-03-23T11:23:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f6942409740 1 -- 172.21.15.62:0/1669476515 shutdown_connections 2024-03-23T11:23:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f6942409740 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55dba6350d90 0x55dba6351170 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f6942409740 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6350460 0x55dba636cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f6942409740 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347ed0 0x55dba636c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f6942409740 1 --2- 172.21.15.62:0/1669476515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba6352d30 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f6942409740 1 -- 172.21.15.62:0/1669476515 >> 172.21.15.62:0/1669476515 conn(0x55dba62efb40 msgr2=0x55dba6321ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:17.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f691e000640 0 client.6369 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:17.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f6942409740 1 -- 172.21.15.62:0/1669476515 shutdown_connections 2024-03-23T11:23:17.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f6942409740 1 -- 172.21.15.62:0/1669476515 wait complete. 2024-03-23T11:23:17.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.147+0000 7f6942409740 20 client.6369 trim_cache size 0 max 16384 2024-03-23T11:23:17.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.163+0000 7f6942409740 1 Processor -- start 2024-03-23T11:23:17.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.163+0000 7f6942409740 1 -- start start 2024-03-23T11:23:17.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.164+0000 7f6942409740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba62efb40 0x55dba636bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.164+0000 7f6942409740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6347af0 0x55dba636c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.164+0000 7f6942409740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347ed0 0x55dba636ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.164+0000 7f6942409740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba5f88f90 con 0x55dba6347ed0 2024-03-23T11:23:17.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.164+0000 7f6942409740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba63476e0 con 0x55dba62efb40 2024-03-23T11:23:17.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.164+0000 7f6942409740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dba63644d0 con 0x55dba6347af0 2024-03-23T11:23:17.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.164+0000 7f693d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347ed0 0x55dba636ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.164+0000 7f693d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347ed0 0x55dba636ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41660/0 (socket says 172.21.15.62:41660) 2024-03-23T11:23:17.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.164+0000 7f693d800640 1 -- 172.21.15.62:0/1030758414 learned_addr learned my addr 172.21.15.62:0/1030758414 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.165+0000 7f693d800640 1 -- 172.21.15.62:0/1030758414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6347af0 msgr2=0x55dba636c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.165+0000 7f6937e00640 1 --2- 172.21.15.62:0/1030758414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6347af0 0x55dba636c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.165+0000 7f693ce00640 1 --2- 172.21.15.62:0/1030758414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba62efb40 0x55dba636bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.165+0000 7f693d800640 1 --2- 172.21.15.62:0/1030758414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6347af0 0x55dba636c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.165+0000 7f693d800640 1 -- 172.21.15.62:0/1030758414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba62efb40 msgr2=0x55dba636bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.165+0000 7f693d800640 1 --2- 172.21.15.62:0/1030758414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba62efb40 0x55dba636bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.165+0000 7f693d800640 1 -- 172.21.15.62:0/1030758414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dba622bc60 con 0x55dba6347ed0 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.165+0000 7f693ce00640 1 --2- 172.21.15.62:0/1030758414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba62efb40 0x55dba636bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.165+0000 7f693d800640 1 --2- 172.21.15.62:0/1030758414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347ed0 0x55dba636ce30 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6930009e50 tx=0x7f693007a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe268e6d32359ca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.166+0000 7f692a000640 1 -- 172.21.15.62:0/1030758414 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693007abb0 con 0x55dba6347ed0 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.166+0000 7f692a000640 1 -- 172.21.15.62:0/1030758414 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69300a5070 con 0x55dba6347ed0 2024-03-23T11:23:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.166+0000 7f692a000640 1 -- 172.21.15.62:0/1030758414 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693007e9e0 con 0x55dba6347ed0 2024-03-23T11:23:17.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.166+0000 7f6942409740 1 -- 172.21.15.62:0/1030758414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347ed0 msgr2=0x55dba636ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:17.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.166+0000 7f6942409740 1 --2- 172.21.15.62:0/1030758414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347ed0 0x55dba636ce30 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6930009e50 tx=0x7f693007a5d0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.166+0000 7f6942409740 1 -- 172.21.15.62:0/1030758414 shutdown_connections 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.166+0000 7f6942409740 1 --2- 172.21.15.62:0/1030758414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347ed0 0x55dba636ce30 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.166+0000 7f6942409740 1 --2- 172.21.15.62:0/1030758414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba6347af0 0x55dba636c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.166+0000 7f6942409740 1 --2- 172.21.15.62:0/1030758414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba62efb40 0x55dba636bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.166+0000 7f6942409740 1 -- 172.21.15.62:0/1030758414 >> 172.21.15.62:0/1030758414 conn(0x55dba6352e90 msgr2=0x55dba63564b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.167+0000 7f6942409740 1 -- 172.21.15.62:0/1030758414 shutdown_connections 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.167+0000 7f6942409740 1 -- 172.21.15.62:0/1030758414 wait complete. 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.167+0000 7f6942409740 1 -- start start 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.167+0000 7f6942409740 1 Processor -- start 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.168+0000 7f6942409740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba62efb40 0x55dba63de5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.168+0000 7f6942409740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba63442f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.168+0000 7f6942409740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347ed0 0x55dba6344830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.168+0000 7f6942409740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba634a900 con 0x55dba6347af0 2024-03-23T11:23:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.168+0000 7f6942409740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dba6322750 con 0x55dba6347ed0 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.168+0000 7f6942409740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dba631e9c0 con 0x55dba62efb40 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.168+0000 7f6942409740 10 client.0 _ll_register_callbacks cb 0x7f6924007d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.169+0000 7f6937e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba63442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.169+0000 7f6937e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba63442f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41672/0 (socket says 172.21.15.62:41672) 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.169+0000 7f6937e00640 1 -- 172.21.15.62:0/9665412 learned_addr learned my addr 172.21.15.62:0/9665412 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.169+0000 7f6937e00640 1 -- 172.21.15.62:0/9665412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba62efb40 msgr2=0x55dba63de5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.169+0000 7f6937e00640 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba62efb40 0x55dba63de5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.169+0000 7f6937e00640 1 -- 172.21.15.62:0/9665412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347ed0 msgr2=0x55dba6344830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.169+0000 7f693d800640 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347ed0 0x55dba6344830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.169+0000 7f6937e00640 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347ed0 0x55dba6344830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.169+0000 7f6937e00640 1 -- 172.21.15.62:0/9665412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693814fc10 con 0x55dba6347af0 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.169+0000 7f6937e00640 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba63442f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6924016040 tx=0x7f6924000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7841cf9970435278 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.169+0000 7f691e000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:17.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f691e000640 1 -- 172.21.15.62:0/9665412 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6924024070 con 0x55dba6347af0 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f691e000640 1 -- 172.21.15.62:0/9665412 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69240070d0 con 0x55dba6347af0 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f6942409740 1 -- 172.21.15.62:0/9665412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6924005ce0 con 0x55dba6347af0 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f691e000640 1 -- 172.21.15.62:0/9665412 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6924001210 con 0x55dba6347af0 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f6942409740 10 client.6375 fetch_fsmap learned FSMap version 34 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f6942409740 1 -- 172.21.15.62:0/9665412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55dba6251170 con 0x55dba6347af0 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f691e000640 1 -- 172.21.15.62:0/9665412 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6924005ce0 con 0x55dba6347af0 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f691e000640 1 -- 172.21.15.62:0/9665412 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6924007290 con 0x55dba6347af0 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f6942409740 10 client.6375 fetch_fsmap finished waiting for FSMap version 34 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f6942409740 10 client.6375 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f6942409740 20 client.6375 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f6942409740 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55dba6350460 0x55dba6354a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f6942409740 4 client.6375 mds_command: new command op to 6081 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.170+0000 7f6942409740 1 -- 172.21.15.62:0/9665412 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55dba5f7f4e0 con 0x55dba6350460 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.171+0000 7f693ce00640 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55dba6350460 0x55dba6354a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.172+0000 7f693ce00640 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55dba6350460 0x55dba6354a90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65b8533f39e2fdcb server_cookie=8f4be4a141f0d811 in_seq=0 out_seq=0 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.172+0000 7f691e000640 10 client.6375 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f691e000640 1 -- 172.21.15.62:0/9665412 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x55dba5f7f4e0 con 0x55dba6350460 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f691e000640 10 client.6375 handle_command_reply: tid=0 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 1 client.6375 shutdown 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 1 -- 172.21.15.62:0/9665412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 msgr2=0x55dba63442f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba63442f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6924016040 tx=0x7f6924000ed0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 1 -- 172.21.15.62:0/9665412 shutdown_connections 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55dba6350460 0x55dba6354a90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dba6347af0 0x55dba63442f0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba6347ed0 0x55dba6344830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 1 --2- 172.21.15.62:0/9665412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55dba62efb40 0x55dba63de5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 1 -- 172.21.15.62:0/9665412 >> 172.21.15.62:0/9665412 conn(0x55dba6352e90 msgr2=0x55dba636bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:17.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f691e000640 0 client.6375 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:17.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 1 -- 172.21.15.62:0/9665412 shutdown_connections 2024-03-23T11:23:17.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 1 -- 172.21.15.62:0/9665412 wait complete. 2024-03-23T11:23:17.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.173+0000 7f6942409740 20 client.6375 trim_cache size 0 max 16384 2024-03-23T11:23:17.172 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:23:17.172 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T11:23:17.172 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-03-23T11:23:17.172 INFO:teuthology.orchestra.run.smithi062.stdout: "54dc16e5-78ff-421d-a74e-2f271da3b50d": { 2024-03-23T11:23:17.173 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-03-23T11:23:17.173 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "54dc16e5-78ff-421d-a74e-2f271da3b50d", 2024-03-23T11:23:17.173 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,repair,force" 2024-03-23T11:23:17.173 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:23:17.173 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:23:17.173 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:23:17.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 -- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f691806a7a0 msgr2=0x7f691806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:17.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f691806a7a0 0x7f691806cc60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6938159410 tx=0x7f692c008040 comp rx=0 tx=0).stop 2024-03-23T11:23:17.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 -- 172.21.15.62:0/3266141419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f693815c460 msgr2=0x7f69380af590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:17.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f693815c460 0x7f69380af590 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6930009e80 tx=0x7f6930004a40 comp rx=0 tx=0).stop 2024-03-23T11:23:17.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 -- 172.21.15.62:0/3266141419 shutdown_connections 2024-03-23T11:23:17.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f691806a7a0 0x7f691806cc60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f693815c460 0x7f69380af590 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69380cb700 0x7f69380af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 --2- 172.21.15.62:0/3266141419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69380b25a0 0x7f693815bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:17.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 -- 172.21.15.62:0/3266141419 >> 172.21.15.62:0/3266141419 conn(0x7f69380b0090 msgr2=0x7f6938151aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:17.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 -- 172.21.15.62:0/3266141419 shutdown_connections 2024-03-23T11:23:17.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:17.174+0000 7f693e200640 1 -- 172.21.15.62:0/3266141419 wait complete. 2024-03-23T11:23:17.188 INFO:tasks.cephfs.filesystem:scrub status for tag:54dc16e5-78ff-421d-a74e-2f271da3b50d - {'path': '/', 'tag': '54dc16e5-78ff-421d-a74e-2f271da3b50d', 'options': 'recursive,repair,force'} 2024-03-23T11:23:45.664 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:23:45.672 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:23:47.188 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub status 2024-03-23T11:23:47.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.417+0000 7f532be00640 1 Processor -- start 2024-03-23T11:23:47.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.417+0000 7f532be00640 1 -- start start 2024-03-23T11:23:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.419+0000 7f532be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f532c1431f0 0x7f532c0b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.419+0000 7f532be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f532c070090 0x7f532c0704d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.419+0000 7f532be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f532c0cc2a0 0x7f532c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.419+0000 7f532be00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532c0b20d0 con 0x7f532c070090 2024-03-23T11:23:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.419+0000 7f532be00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532c070a10 con 0x7f532c0cc2a0 2024-03-23T11:23:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.419+0000 7f532be00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1435d0 con 0x7f532c1431f0 2024-03-23T11:23:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.419+0000 7f532a000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f532c070090 0x7f532c0704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.419+0000 7f532a000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f532c070090 0x7f532c0704d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34702/0 (socket says 172.21.15.62:34702) 2024-03-23T11:23:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.419+0000 7f532a000640 1 -- 172.21.15.62:0/2701195297 learned_addr learned my addr 172.21.15.62:0/2701195297 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.419+0000 7f532aa00640 1 --2- 172.21.15.62:0/2701195297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f532c1431f0 0x7f532c0b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.419+0000 7f532b400640 1 --2- 172.21.15.62:0/2701195297 >> v2:172.21.15.120:3300/0 conn(0x7f532c0cc2a0 0x7f532c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.420+0000 7f532a000640 1 -- 172.21.15.62:0/2701195297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f532c1431f0 msgr2=0x7f532c0b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.420+0000 7f532a000640 1 --2- 172.21.15.62:0/2701195297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f532c1431f0 0x7f532c0b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.420+0000 7f532a000640 1 -- 172.21.15.62:0/2701195297 >> v2:172.21.15.120:3300/0 conn(0x7f532c0cc2a0 msgr2=0x7f532c0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.420+0000 7f532a000640 1 --2- 172.21.15.62:0/2701195297 >> v2:172.21.15.120:3300/0 conn(0x7f532c0cc2a0 0x7f532c0cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.420+0000 7f532a000640 1 -- 172.21.15.62:0/2701195297 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c0b14c0 con 0x7f532c070090 2024-03-23T11:23:47.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.421+0000 7f532a000640 1 --2- 172.21.15.62:0/2701195297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f532c070090 0x7f532c0704d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f532c0b3840 tx=0x7f531802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c5ec5c492d12fb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:47.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.421+0000 7f5329600640 1 -- 172.21.15.62:0/2701195297 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53180317e0 con 0x7f532c070090 2024-03-23T11:23:47.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.421+0000 7f5329600640 1 -- 172.21.15.62:0/2701195297 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5318031940 con 0x7f532c070090 2024-03-23T11:23:47.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.421+0000 7f5329600640 1 -- 172.21.15.62:0/2701195297 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318031cd0 con 0x7f532c070090 2024-03-23T11:23:47.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.422+0000 7f532be00640 1 -- 172.21.15.62:0/2701195297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f532c070090 msgr2=0x7f532c0704d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.422+0000 7f532be00640 1 --2- 172.21.15.62:0/2701195297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f532c070090 0x7f532c0704d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f532c0b3840 tx=0x7f531802b880 comp rx=0 tx=0).stop 2024-03-23T11:23:47.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.422+0000 7f532be00640 1 -- 172.21.15.62:0/2701195297 shutdown_connections 2024-03-23T11:23:47.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.422+0000 7f532be00640 1 --2- 172.21.15.62:0/2701195297 >> v2:172.21.15.120:3300/0 conn(0x7f532c0cc2a0 0x7f532c0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.422+0000 7f532be00640 1 --2- 172.21.15.62:0/2701195297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f532c070090 0x7f532c0704d0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.422+0000 7f532be00640 1 --2- 172.21.15.62:0/2701195297 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f532c1431f0 0x7f532c0b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.422+0000 7f532be00640 1 -- 172.21.15.62:0/2701195297 >> 172.21.15.62:0/2701195297 conn(0x7f532c0b0090 msgr2=0x7f532c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:47.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.422+0000 7f532be00640 1 -- 172.21.15.62:0/2701195297 shutdown_connections 2024-03-23T11:23:47.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.422+0000 7f532be00640 1 -- 172.21.15.62:0/2701195297 wait complete. 2024-03-23T11:23:47.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.424+0000 7f532be00640 1 Processor -- start 2024-03-23T11:23:47.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.424+0000 7f532be00640 1 -- start start 2024-03-23T11:23:47.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.425+0000 7f532be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f532c070090 0x7f532c13dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.425+0000 7f532be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f532c0cc2a0 0x7f532c0acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.425+0000 7f532be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f532c1431f0 0x7f532c0ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.425+0000 7f532be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1f2c00 con 0x7f532c0cc2a0 2024-03-23T11:23:47.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.425+0000 7f532be00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1f1ff0 con 0x7f532c1431f0 2024-03-23T11:23:47.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.425+0000 7f532be00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1f22d0 con 0x7f532c070090 2024-03-23T11:23:47.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.425+0000 7f532a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f532c0cc2a0 0x7f532c0acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.425+0000 7f532a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f532c0cc2a0 0x7f532c0acaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34704/0 (socket says 172.21.15.62:34704) 2024-03-23T11:23:47.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.425+0000 7f532a000640 1 -- 172.21.15.62:0/1063288388 learned_addr learned my addr 172.21.15.62:0/1063288388 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:47.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.426+0000 7f532aa00640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f532c070090 0x7f532c13dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.426+0000 7f532b400640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f532c1431f0 0x7f532c0ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.426+0000 7f532a000640 1 -- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f532c070090 msgr2=0x7f532c13dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.426+0000 7f532a000640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f532c070090 0x7f532c13dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.426+0000 7f532a000640 1 -- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f532c1431f0 msgr2=0x7f532c0ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.426+0000 7f532a000640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f532c1431f0 0x7f532c0ad030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.426+0000 7f532a000640 1 -- 172.21.15.62:0/1063288388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53180094d0 con 0x7f532c0cc2a0 2024-03-23T11:23:47.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.426+0000 7f532b400640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f532c1431f0 0x7f532c0ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:47.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.426+0000 7f532a000640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f532c0cc2a0 0x7f532c0acaf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5318000c00 tx=0x7f53180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:47.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.427+0000 7f5322a00640 1 -- 172.21.15.62:0/1063288388 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318032880 con 0x7f532c0cc2a0 2024-03-23T11:23:47.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.427+0000 7f5322a00640 1 -- 172.21.15.62:0/1063288388 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53180329e0 con 0x7f532c0cc2a0 2024-03-23T11:23:47.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.427+0000 7f5322a00640 1 -- 172.21.15.62:0/1063288388 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318032d70 con 0x7f532c0cc2a0 2024-03-23T11:23:47.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.427+0000 7f532be00640 1 -- 172.21.15.62:0/1063288388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532c1f25b0 con 0x7f532c0cc2a0 2024-03-23T11:23:47.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.427+0000 7f532be00640 1 -- 172.21.15.62:0/1063288388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532c13e150 con 0x7f532c0cc2a0 2024-03-23T11:23:47.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.429+0000 7f5322a00640 1 -- 172.21.15.62:0/1063288388 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f53180453f0 con 0x7f532c0cc2a0 2024-03-23T11:23:47.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.430+0000 7f5322a00640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f530c06a7a0 0x7f530c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.430+0000 7f5322a00640 1 -- 172.21.15.62:0/1063288388 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53180ab340 con 0x7f532c0cc2a0 2024-03-23T11:23:47.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.430+0000 7f532aa00640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f530c06a7a0 0x7f530c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.431+0000 7f53343db740 1 Processor -- start 2024-03-23T11:23:47.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.431+0000 7f53343db740 1 -- start start 2024-03-23T11:23:47.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.432+0000 7f532aa00640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f530c06a7a0 0x7f530c06cc60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f531c006fd0 tx=0x7f531c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:47.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.432+0000 7f53343db740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458718af0 0x557458720d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.432+0000 7f53343db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x5574586f8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.432+0000 7f53343db740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.432+0000 7f53343db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574586b4cb0 con 0x557458721460 2024-03-23T11:23:47.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.432+0000 7f53343db740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557458663500 con 0x557458718ed0 2024-03-23T11:23:47.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.432+0000 7f53343db740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557458660c90 con 0x557458718af0 2024-03-23T11:23:47.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.432+0000 7f532a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x5574586f8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.433+0000 7f532a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x5574586f8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34716/0 (socket says 172.21.15.62:34716) 2024-03-23T11:23:47.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.433+0000 7f532a000640 1 -- 172.21.15.62:0/750548401 learned_addr learned my addr 172.21.15.62:0/750548401 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:47.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.433+0000 7f532aa00640 1 --2- 172.21.15.62:0/750548401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458718af0 0x557458720d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.433+0000 7f532b400640 1 --2- 172.21.15.62:0/750548401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.433+0000 7f532a000640 1 -- 172.21.15.62:0/750548401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458718af0 msgr2=0x557458720d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.433+0000 7f532a000640 1 --2- 172.21.15.62:0/750548401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458718af0 0x557458720d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.433+0000 7f532a000640 1 -- 172.21.15.62:0/750548401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 msgr2=0x5574586f9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.433+0000 7f532a000640 1 --2- 172.21.15.62:0/750548401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.433+0000 7f532a000640 1 -- 172.21.15.62:0/750548401 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55745860e360 con 0x557458721460 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.433+0000 7f532b400640 1 --2- 172.21.15.62:0/750548401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.433+0000 7f532a000640 1 --2- 172.21.15.62:0/750548401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x5574586f8c20 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5318076a60 tx=0x7f531807ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55892091e578865 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f5317400640 1 -- 172.21.15.62:0/750548401 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531807e170 con 0x557458721460 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f5317400640 1 -- 172.21.15.62:0/750548401 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5318082950 con 0x557458721460 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f5317400640 1 -- 172.21.15.62:0/750548401 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318078900 con 0x557458721460 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f53343db740 1 -- 172.21.15.62:0/750548401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 msgr2=0x5574586f8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f53343db740 1 --2- 172.21.15.62:0/750548401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x5574586f8c20 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5318076a60 tx=0x7f531807ad20 comp rx=0 tx=0).stop 2024-03-23T11:23:47.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f53343db740 1 -- 172.21.15.62:0/750548401 shutdown_connections 2024-03-23T11:23:47.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f53343db740 1 --2- 172.21.15.62:0/750548401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x5574586f8c20 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f53343db740 1 --2- 172.21.15.62:0/750548401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f53343db740 1 --2- 172.21.15.62:0/750548401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458718af0 0x557458720d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f53343db740 1 -- 172.21.15.62:0/750548401 >> 172.21.15.62:0/750548401 conn(0x5574586c0b40 msgr2=0x55745873e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:47.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f53343db740 1 -- 172.21.15.62:0/750548401 shutdown_connections 2024-03-23T11:23:47.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.434+0000 7f53343db740 1 -- 172.21.15.62:0/750548401 wait complete. 2024-03-23T11:23:47.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.435+0000 7f53343db740 1 -- start start 2024-03-23T11:23:47.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.435+0000 7f53343db740 1 Processor -- start 2024-03-23T11:23:47.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.436+0000 7f53343db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x557458716c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.436+0000 7f53343db740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x557458726a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.436+0000 7f53343db740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458721460 0x557458723c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.436+0000 7f53343db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55745865c580 con 0x557458718af0 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.436+0000 7f53343db740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574585fda50 con 0x557458718ed0 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.436+0000 7f53343db740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5574585fd6f0 con 0x557458721460 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.437+0000 7f53343db740 10 client.0 _ll_register_callbacks cb 0x55745864ced0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.437+0000 7f532aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x557458716c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.437+0000 7f532aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x557458716c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34728/0 (socket says 172.21.15.62:34728) 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.437+0000 7f532aa00640 1 -- 172.21.15.62:0/56891502 learned_addr learned my addr 172.21.15.62:0/56891502 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.437+0000 7f532a000640 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x557458726a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.437+0000 7f532aa00640 1 -- 172.21.15.62:0/56891502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458721460 msgr2=0x557458723c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.437+0000 7f532aa00640 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458721460 0x557458723c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.437+0000 7f532aa00640 1 -- 172.21.15.62:0/56891502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 msgr2=0x557458726a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.437+0000 7f532aa00640 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x557458726a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.437+0000 7f532aa00640 1 -- 172.21.15.62:0/56891502 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532c13e150 con 0x557458718af0 2024-03-23T11:23:47.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.438+0000 7f532aa00640 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x557458716c30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f531c0144e0 tx=0x7f531c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef39483cf689e027 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:47.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.438+0000 7f530be00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:47.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.438+0000 7f530be00640 1 -- 172.21.15.62:0/56891502 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c021830 con 0x557458718af0 2024-03-23T11:23:47.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.438+0000 7f530be00640 1 -- 172.21.15.62:0/56891502 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f531c021990 con 0x557458718af0 2024-03-23T11:23:47.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.438+0000 7f53343db740 1 -- 172.21.15.62:0/56891502 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557458686fe0 con 0x557458718af0 2024-03-23T11:23:47.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.438+0000 7f530be00640 1 -- 172.21.15.62:0/56891502 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f531c021b50 con 0x557458718af0 2024-03-23T11:23:47.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.438+0000 7f53343db740 10 client.6387 fetch_fsmap learned FSMap version 35 2024-03-23T11:23:47.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.438+0000 7f53343db740 1 -- 172.21.15.62:0/56891502 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55745865ac00 con 0x557458718af0 2024-03-23T11:23:47.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.438+0000 7f530be00640 1 -- 172.21.15.62:0/56891502 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557458686fe0 con 0x557458718af0 2024-03-23T11:23:47.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.439+0000 7f530be00640 1 -- 172.21.15.62:0/56891502 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f531c021cb0 con 0x557458718af0 2024-03-23T11:23:47.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.439+0000 7f53343db740 10 client.6387 fetch_fsmap finished waiting for FSMap version 35 2024-03-23T11:23:47.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.439+0000 7f53343db740 10 client.6387 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:23:47.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.439+0000 7f53343db740 20 client.6387 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:47.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.439+0000 7f53343db740 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x557458724aa0 0x557458724e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.439+0000 7f53343db740 4 client.6387 mds_command: new command op to 6081 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:23:47.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.439+0000 7f53343db740 1 -- 172.21.15.62:0/56891502 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55745864f240 con 0x557458724aa0 2024-03-23T11:23:47.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.439+0000 7f532a000640 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x557458724aa0 0x557458724e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.440+0000 7f532a000640 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x557458724aa0 0x557458724e80 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f5bc258c4fdd28 server_cookie=42d160ebf520135b in_seq=0 out_seq=0 2024-03-23T11:23:47.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.440+0000 7f530be00640 10 client.6387 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:47.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f530be00640 1 -- 172.21.15.62:0/56891502 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55745864f240 con 0x557458724aa0 2024-03-23T11:23:47.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f530be00640 10 client.6387 handle_command_reply: tid=0 2024-03-23T11:23:47.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f53343db740 1 client.6387 shutdown 2024-03-23T11:23:47.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f53343db740 1 -- 172.21.15.62:0/56891502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 msgr2=0x557458716c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f53343db740 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x557458716c30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f531c0144e0 tx=0x7f531c004730 comp rx=0 tx=0).stop 2024-03-23T11:23:47.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f53343db740 1 -- 172.21.15.62:0/56891502 shutdown_connections 2024-03-23T11:23:47.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f53343db740 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x557458724aa0 0x557458724e80 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f53343db740 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458721460 0x557458723c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f53343db740 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x557458726a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f53343db740 1 --2- 172.21.15.62:0/56891502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x557458716c30 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f53343db740 1 -- 172.21.15.62:0/56891502 >> 172.21.15.62:0/56891502 conn(0x5574586c0b40 msgr2=0x5574587180d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:47.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.442+0000 7f530be00640 0 client.6387 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:47.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.443+0000 7f53343db740 1 -- 172.21.15.62:0/56891502 shutdown_connections 2024-03-23T11:23:47.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.443+0000 7f53343db740 1 -- 172.21.15.62:0/56891502 wait complete. 2024-03-23T11:23:47.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.443+0000 7f53343db740 20 client.6387 trim_cache size 0 max 16384 2024-03-23T11:23:47.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.458+0000 7f53343db740 1 Processor -- start 2024-03-23T11:23:47.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.459+0000 7f53343db740 1 -- start start 2024-03-23T11:23:47.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.459+0000 7f53343db740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458718af0 0x557458727790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.459+0000 7f53343db740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.459+0000 7f53343db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x557458742300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.460+0000 7f53343db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557458359f90 con 0x557458721460 2024-03-23T11:23:47.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.460+0000 7f53343db740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557458726420 con 0x557458718ed0 2024-03-23T11:23:47.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.460+0000 7f53343db740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55745871b3c0 con 0x557458718af0 2024-03-23T11:23:47.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.460+0000 7f532b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x557458742300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.460+0000 7f532b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x557458742300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34740/0 (socket says 172.21.15.62:34740) 2024-03-23T11:23:47.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.460+0000 7f532b400640 1 -- 172.21.15.62:0/3345486960 learned_addr learned my addr 172.21.15.62:0/3345486960 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:47.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.460+0000 7f532aa00640 1 --2- 172.21.15.62:0/3345486960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458718af0 0x557458727790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.460+0000 7f532a000640 1 --2- 172.21.15.62:0/3345486960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.461+0000 7f532b400640 1 -- 172.21.15.62:0/3345486960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458718af0 msgr2=0x557458727790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.461+0000 7f532b400640 1 --2- 172.21.15.62:0/3345486960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458718af0 0x557458727790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.461+0000 7f532b400640 1 -- 172.21.15.62:0/3345486960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 msgr2=0x5574586f8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.461+0000 7f532b400640 1 --2- 172.21.15.62:0/3345486960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.461+0000 7f532b400640 1 -- 172.21.15.62:0/3345486960 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574585fcc60 con 0x557458721460 2024-03-23T11:23:47.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.461+0000 7f532b400640 1 --2- 172.21.15.62:0/3345486960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x557458742300 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5324009e30 tx=0x7f5324007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524be31d9f291a0a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:47.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.461+0000 7f5317e00640 1 -- 172.21.15.62:0/3345486960 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53240147f0 con 0x557458721460 2024-03-23T11:23:47.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.461+0000 7f5317e00640 1 -- 172.21.15.62:0/3345486960 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532400f040 con 0x557458721460 2024-03-23T11:23:47.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.461+0000 7f5317e00640 1 -- 172.21.15.62:0/3345486960 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324014b50 con 0x557458721460 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.462+0000 7f53343db740 1 -- 172.21.15.62:0/3345486960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 msgr2=0x557458742300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.462+0000 7f53343db740 1 --2- 172.21.15.62:0/3345486960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x557458742300 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5324009e30 tx=0x7f5324007700 comp rx=0 tx=0).stop 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.462+0000 7f53343db740 1 -- 172.21.15.62:0/3345486960 shutdown_connections 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.462+0000 7f53343db740 1 --2- 172.21.15.62:0/3345486960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458721460 0x557458742300 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.462+0000 7f53343db740 1 --2- 172.21.15.62:0/3345486960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.462+0000 7f53343db740 1 --2- 172.21.15.62:0/3345486960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458718af0 0x557458727790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.462+0000 7f53343db740 1 -- 172.21.15.62:0/3345486960 >> 172.21.15.62:0/3345486960 conn(0x5574586c0b40 msgr2=0x5574587180d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.462+0000 7f53343db740 1 -- 172.21.15.62:0/3345486960 shutdown_connections 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.462+0000 7f53343db740 1 -- 172.21.15.62:0/3345486960 wait complete. 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.463+0000 7f53343db740 1 -- start start 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.463+0000 7f53343db740 1 Processor -- start 2024-03-23T11:23:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.464+0000 7f53343db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x5574587a4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.464+0000 7f53343db740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.464+0000 7f53343db740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458721460 0x5574587a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.464+0000 7f53343db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574586f80f0 con 0x557458718af0 2024-03-23T11:23:47.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.464+0000 7f53343db740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574587155b0 con 0x557458718ed0 2024-03-23T11:23:47.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.464+0000 7f53343db740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5574587a4b30 con 0x557458721460 2024-03-23T11:23:47.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.464+0000 7f53343db740 10 client.0 _ll_register_callbacks cb 0x7f5318032f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:47.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.464+0000 7f532aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x5574587a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.464+0000 7f532aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x5574587a4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34744/0 (socket says 172.21.15.62:34744) 2024-03-23T11:23:47.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.464+0000 7f532aa00640 1 -- 172.21.15.62:0/617747190 learned_addr learned my addr 172.21.15.62:0/617747190 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.465+0000 7f532a000640 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.465+0000 7f532b400640 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458721460 0x5574587a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.465+0000 7f532aa00640 1 -- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458721460 msgr2=0x5574587a4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.465+0000 7f532aa00640 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458721460 0x5574587a4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.465+0000 7f532aa00640 1 -- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 msgr2=0x5574586f7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.465+0000 7f532aa00640 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.465+0000 7f532aa00640 1 -- 172.21.15.62:0/617747190 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5324009a30 con 0x557458718af0 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.465+0000 7f532aa00640 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x5574587a4300 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f531c00c010 tx=0x7f531c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8df8c28155a6a82 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.465+0000 7f530be00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f530be00640 1 -- 172.21.15.62:0/617747190 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c015990 con 0x557458718af0 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f530be00640 1 -- 172.21.15.62:0/617747190 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f531c006d70 con 0x557458718af0 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f53343db740 1 -- 172.21.15.62:0/617747190 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5318082950 con 0x557458718af0 2024-03-23T11:23:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f530be00640 1 -- 172.21.15.62:0/617747190 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f531c015af0 con 0x557458718af0 2024-03-23T11:23:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f53343db740 10 client.6393 fetch_fsmap learned FSMap version 35 2024-03-23T11:23:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f53343db740 1 -- 172.21.15.62:0/617747190 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x557458622170 con 0x557458718af0 2024-03-23T11:23:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f530be00640 1 -- 172.21.15.62:0/617747190 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5318082950 con 0x557458718af0 2024-03-23T11:23:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f530be00640 1 -- 172.21.15.62:0/617747190 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f531c021630 con 0x557458718af0 2024-03-23T11:23:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f53343db740 10 client.6393 fetch_fsmap finished waiting for FSMap version 35 2024-03-23T11:23:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f53343db740 10 client.6393 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.b 2024-03-23T11:23:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f53343db740 20 client.6393 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f53343db740 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x557458760640 0x5574586f9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f53343db740 4 client.6393 mds_command: new command op to 6081 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:23:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.466+0000 7f53343db740 1 -- 172.21.15.62:0/617747190 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5574583504e0 con 0x557458760640 2024-03-23T11:23:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.467+0000 7f532a000640 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x557458760640 0x5574586f9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.468+0000 7f532a000640 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x557458760640 0x5574586f9660 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cd587f57a4d42e0 server_cookie=d39fc037eb8f2bb in_seq=0 out_seq=0 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.468+0000 7f530be00640 10 client.6393 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.468+0000 7f530be00640 1 -- 172.21.15.62:0/617747190 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5574583504e0 con 0x557458760640 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.468+0000 7f530be00640 10 client.6393 handle_command_reply: tid=0 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.468+0000 7f53343db740 1 client.6393 shutdown 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f53343db740 1 -- 172.21.15.62:0/617747190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 msgr2=0x5574587a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f53343db740 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x5574587a4300 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f531c00c010 tx=0x7f531c016000 comp rx=0 tx=0).stop 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f53343db740 1 -- 172.21.15.62:0/617747190 shutdown_connections 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f53343db740 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x557458760640 0x5574586f9660 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f53343db740 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557458721460 0x5574587a4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f53343db740 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557458718ed0 0x5574586f7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f53343db740 1 --2- 172.21.15.62:0/617747190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557458718af0 0x5574587a4300 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f53343db740 1 -- 172.21.15.62:0/617747190 >> 172.21.15.62:0/617747190 conn(0x5574586c0b40 msgr2=0x5574587238d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:47.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f530be00640 0 client.6393 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:47.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f53343db740 1 -- 172.21.15.62:0/617747190 shutdown_connections 2024-03-23T11:23:47.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f53343db740 1 -- 172.21.15.62:0/617747190 wait complete. 2024-03-23T11:23:47.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.469+0000 7f53343db740 20 client.6393 trim_cache size 0 max 16384 2024-03-23T11:23:47.469 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:23:47.469 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-03-23T11:23:47.469 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-03-23T11:23:47.469 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:23:47.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 -- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f530c06a7a0 msgr2=0x7f530c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f530c06a7a0 0x7f530c06cc60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f531c006fd0 tx=0x7f531c008040 comp rx=0 tx=0).stop 2024-03-23T11:23:47.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 -- 172.21.15.62:0/1063288388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f532c0cc2a0 msgr2=0x7f532c0acaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f532c0cc2a0 0x7f532c0acaf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5318000c00 tx=0x7f53180046c0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 -- 172.21.15.62:0/1063288388 shutdown_connections 2024-03-23T11:23:47.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f530c06a7a0 0x7f530c06cc60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f532c0cc2a0 0x7f532c0acaf0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f532c1431f0 0x7f532c0ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 --2- 172.21.15.62:0/1063288388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f532c070090 0x7f532c13dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 -- 172.21.15.62:0/1063288388 >> 172.21.15.62:0/1063288388 conn(0x7f532c0b0090 msgr2=0x7f532c15bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:47.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 -- 172.21.15.62:0/1063288388 shutdown_connections 2024-03-23T11:23:47.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.470+0000 7f532be00640 1 -- 172.21.15.62:0/1063288388 wait complete. 2024-03-23T11:23:47.484 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T11:23:47.484 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:23:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.702+0000 7fef59800640 1 Processor -- start 2024-03-23T11:23:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.703+0000 7fef59800640 1 -- start start 2024-03-23T11:23:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.704+0000 7fef59800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef5415c9e0 0x7fef541607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.704+0000 7fef59800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fef5415cdc0 0x7fef540d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.704+0000 7fef59800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5406bfc0 con 0x7fef540cf650 2024-03-23T11:23:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.704+0000 7fef59800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef540ce140 con 0x7fef5415cdc0 2024-03-23T11:23:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.704+0000 7fef59800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef540b4370 con 0x7fef5415c9e0 2024-03-23T11:23:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.705+0000 7fef52000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fef5415cdc0 0x7fef540d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.705+0000 7fef52000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fef5415cdc0 0x7fef540d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34750/0 (socket says 172.21.15.62:34750) 2024-03-23T11:23:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.705+0000 7fef52000640 1 -- 172.21.15.62:0/454474259 learned_addr learned my addr 172.21.15.62:0/454474259 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.705+0000 7fef52a00640 1 --2- 172.21.15.62:0/454474259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef5415c9e0 0x7fef541607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.705+0000 7fef52000640 1 -- 172.21.15.62:0/454474259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef5415c9e0 msgr2=0x7fef541607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.705+0000 7fef52000640 1 --2- 172.21.15.62:0/454474259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef5415c9e0 0x7fef541607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.705+0000 7fef52000640 1 -- 172.21.15.62:0/454474259 >> v1:172.21.15.120:6789/0 conn(0x7fef540cf650 legacy=0x7fef5415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:47.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.705+0000 7fef52000640 1 -- 172.21.15.62:0/454474259 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef540b5260 con 0x7fef5415cdc0 2024-03-23T11:23:47.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.706+0000 7fef52000640 1 --2- 172.21.15.62:0/454474259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef5415cdc0 0x7fef540d1f20 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009870 tx=0x7fef3c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2123c584e297eee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:47.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.707+0000 7fef51600640 1 -- 172.21.15.62:0/454474259 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c031640 con 0x7fef5415cdc0 2024-03-23T11:23:47.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.707+0000 7fef51600640 1 -- 172.21.15.62:0/454474259 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3c0317a0 con 0x7fef5415cdc0 2024-03-23T11:23:47.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.707+0000 7fef51600640 1 -- 172.21.15.62:0/454474259 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c031b30 con 0x7fef5415cdc0 2024-03-23T11:23:47.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.707+0000 7fef59800640 1 -- 172.21.15.62:0/454474259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef5415cdc0 msgr2=0x7fef540d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.707+0000 7fef59800640 1 --2- 172.21.15.62:0/454474259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef5415cdc0 0x7fef540d1f20 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009870 tx=0x7fef3c02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.708+0000 7fef59800640 1 -- 172.21.15.62:0/454474259 shutdown_connections 2024-03-23T11:23:47.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.708+0000 7fef59800640 1 --2- 172.21.15.62:0/454474259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef5415cdc0 0x7fef540d1f20 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.708+0000 7fef59800640 1 --2- 172.21.15.62:0/454474259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef5415c9e0 0x7fef541607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.708+0000 7fef59800640 1 -- 172.21.15.62:0/454474259 >> 172.21.15.62:0/454474259 conn(0x7fef54071570 msgr2=0x7fef54071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:47.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.708+0000 7fef59800640 1 -- 172.21.15.62:0/454474259 shutdown_connections 2024-03-23T11:23:47.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.708+0000 7fef59800640 1 -- 172.21.15.62:0/454474259 wait complete. 2024-03-23T11:23:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.709+0000 7fef59800640 1 Processor -- start 2024-03-23T11:23:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.710+0000 7fef59800640 1 -- start start 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.710+0000 7fef59800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540cf650 0x7fef541f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.710+0000 7fef59800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef5415c9e0 0x7fef5415ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.710+0000 7fef59800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5415cdc0 0x7fef541f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.710+0000 7fef59800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef541f6640 con 0x7fef5415c9e0 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.710+0000 7fef59800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef541f5da0 con 0x7fef5415cdc0 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.710+0000 7fef59800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef541f6080 con 0x7fef540cf650 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.711+0000 7fef52000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef5415c9e0 0x7fef5415ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.711+0000 7fef52000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef5415c9e0 0x7fef5415ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34764/0 (socket says 172.21.15.62:34764) 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.711+0000 7fef52000640 1 -- 172.21.15.62:0/1953362837 learned_addr learned my addr 172.21.15.62:0/1953362837 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.711+0000 7fef52000640 1 -- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540cf650 msgr2=0x7fef541f5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.711+0000 7fef52000640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540cf650 0x7fef541f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.711+0000 7fef52000640 1 -- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5415cdc0 msgr2=0x7fef541f21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.711+0000 7fef53400640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5415cdc0 0x7fef541f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.711+0000 7fef52000640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5415cdc0 0x7fef541f21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.711+0000 7fef52000640 1 -- 172.21.15.62:0/1953362837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c0094d0 con 0x7fef5415c9e0 2024-03-23T11:23:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.712+0000 7fef52000640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef5415c9e0 0x7fef5415ee10 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fef3c000c00 tx=0x7fef3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.712+0000 7fef4aa00640 1 -- 172.21.15.62:0/1953362837 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c03ac60 con 0x7fef5415c9e0 2024-03-23T11:23:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.712+0000 7fef4aa00640 1 -- 172.21.15.62:0/1953362837 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3c03adc0 con 0x7fef5415c9e0 2024-03-23T11:23:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.712+0000 7fef4aa00640 1 -- 172.21.15.62:0/1953362837 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c044630 con 0x7fef5415c9e0 2024-03-23T11:23:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.712+0000 7fef59800640 1 -- 172.21.15.62:0/1953362837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef541f6360 con 0x7fef5415c9e0 2024-03-23T11:23:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.712+0000 7fef59800640 1 -- 172.21.15.62:0/1953362837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef541f4f50 con 0x7fef5415c9e0 2024-03-23T11:23:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.713+0000 7fef59800640 1 -- 172.21.15.62:0/1953362837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef540aaa00 con 0x7fef5415c9e0 2024-03-23T11:23:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.714+0000 7fef4aa00640 1 -- 172.21.15.62:0/1953362837 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fef3c02f050 con 0x7fef5415c9e0 2024-03-23T11:23:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.715+0000 7fef4aa00640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3006a7a0 0x7fef3006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.715+0000 7fef4aa00640 1 -- 172.21.15.62:0/1953362837 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef3c0aaca0 con 0x7fef5415c9e0 2024-03-23T11:23:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.716+0000 7fef52a00640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3006a7a0 0x7fef3006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:47.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.717+0000 7fef52a00640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3006a7a0 0x7fef3006cc60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fef440098f0 tx=0x7fef44008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:47.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.720+0000 7fef4aa00640 1 -- 172.21.15.62:0/1953362837 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef540aaa00 con 0x7fef5415c9e0 2024-03-23T11:23:47.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.955+0000 7fef59800640 1 -- 172.21.15.62:0/1953362837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef540aaa00 con 0x7fef5415c9e0 2024-03-23T11:23:47.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.956+0000 7fef4aa00640 1 -- 172.21.15.62:0/1953362837 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fef540aaa00 con 0x7fef5415c9e0 2024-03-23T11:23:47.952 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:47.953 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3287045135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3287045135},{"type":"v1","addr":"172.21.15.120:6835","nonce":3287045135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6066,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/2859430143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2859430143},{"type":"v1","addr":"172.21.15.62:6834","nonce":2859430143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2005768060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2005768060},{"type":"v1","addr":"172.21.15.62:6835","nonce":2005768060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:22:31.725865+0000","modified":"2024-03-23T11:23:24.399625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6081},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6081":{"gid":6081,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":16,"addr":"172.21.15.120:6837/4158450746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":4158450746},{"type":"v1","addr":"172.21.15.120:6837","nonce":4158450746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T11:23:47.953 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-03-23T11:23:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 -- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3006a7a0 msgr2=0x7fef3006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3006a7a0 0x7fef3006cc60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fef440098f0 tx=0x7fef44008040 comp rx=0 tx=0).stop 2024-03-23T11:23:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 -- 172.21.15.62:0/1953362837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef5415c9e0 msgr2=0x7fef5415ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef5415c9e0 0x7fef5415ee10 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fef3c000c00 tx=0x7fef3c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 -- 172.21.15.62:0/1953362837 shutdown_connections 2024-03-23T11:23:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3006a7a0 0x7fef3006cc60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef5415c9e0 0x7fef5415ee10 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5415cdc0 0x7fef541f21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 --2- 172.21.15.62:0/1953362837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef540cf650 0x7fef541f5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:47.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 -- 172.21.15.62:0/1953362837 >> 172.21.15.62:0/1953362837 conn(0x7fef54071570 msgr2=0x7fef5415fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:47.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 -- 172.21.15.62:0/1953362837 shutdown_connections 2024-03-23T11:23:47.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:47.959+0000 7fef59800640 1 -- 172.21.15.62:0/1953362837 wait complete. 2024-03-23T11:23:47.973 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b damage ls --format=json-pretty 2024-03-23T11:23:48.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.192+0000 7fb340e00640 1 Processor -- start 2024-03-23T11:23:48.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.192+0000 7fb340e00640 1 -- start start 2024-03-23T11:23:48.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.194+0000 7fb340e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb33c0cc9b0 0x7fb33c0b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.194+0000 7fb340e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33c0b20d0 con 0x7fb33c1431f0 2024-03-23T11:23:48.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.194+0000 7fb340e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33c1435d0 con 0x7fb33c0707a0 2024-03-23T11:23:48.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.194+0000 7fb340e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33c0b14c0 con 0x7fb33c0cc9b0 2024-03-23T11:23:48.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.194+0000 7fb33b400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb33c1431f0 0x7fb33c070090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39296/0 (socket says 172.21.15.62:39296) 2024-03-23T11:23:48.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.194+0000 7fb33b400640 1 -- 172.21.15.62:0/1898438698 learned_addr learned my addr 172.21.15.62:0/1898438698 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:48.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.194+0000 7fb33be00640 1 --2- 172.21.15.62:0/1898438698 >> v2:172.21.15.62:3300/0 conn(0x7fb33c0cc9b0 0x7fb33c0b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.195+0000 7fb33aa00640 1 -- 172.21.15.62:0/1898438698 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398503984 0 0) 0x7fb33c0b20d0 con 0x7fb33c1431f0 2024-03-23T11:23:48.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.195+0000 7fb33aa00640 1 -- 172.21.15.62:0/1898438698 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb320003520 con 0x7fb33c1431f0 2024-03-23T11:23:48.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.195+0000 7fb33be00640 1 -- 172.21.15.62:0/1898438698 >> v1:172.21.15.120:6789/0 conn(0x7fb33c0707a0 legacy=0x7fb33c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.195+0000 7fb33be00640 1 -- 172.21.15.62:0/1898438698 >> v1:172.21.15.62:6789/0 conn(0x7fb33c1431f0 legacy=0x7fb33c070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.195+0000 7fb33be00640 1 -- 172.21.15.62:0/1898438698 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c0b17a0 con 0x7fb33c0cc9b0 2024-03-23T11:23:48.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.196+0000 7fb33be00640 1 --2- 172.21.15.62:0/1898438698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb33c0cc9b0 0x7fb33c0b2560 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb334009d90 tx=0x7fb33402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75cbc50dd2bee11f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:48.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.197+0000 7fb33aa00640 1 -- 172.21.15.62:0/1898438698 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33402cc60 con 0x7fb33c0cc9b0 2024-03-23T11:23:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.197+0000 7fb33aa00640 1 -- 172.21.15.62:0/1898438698 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33402cdc0 con 0x7fb33c0cc9b0 2024-03-23T11:23:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.197+0000 7fb33aa00640 1 -- 172.21.15.62:0/1898438698 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334034660 con 0x7fb33c0cc9b0 2024-03-23T11:23:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.197+0000 7fb340e00640 1 -- 172.21.15.62:0/1898438698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb33c0cc9b0 msgr2=0x7fb33c0b2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.197+0000 7fb340e00640 1 --2- 172.21.15.62:0/1898438698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb33c0cc9b0 0x7fb33c0b2560 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb334009d90 tx=0x7fb33402beb0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.197+0000 7fb340e00640 1 -- 172.21.15.62:0/1898438698 shutdown_connections 2024-03-23T11:23:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.197+0000 7fb340e00640 1 --2- 172.21.15.62:0/1898438698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb33c0cc9b0 0x7fb33c0b2560 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.197+0000 7fb340e00640 1 -- 172.21.15.62:0/1898438698 >> 172.21.15.62:0/1898438698 conn(0x7fb33c0b0090 msgr2=0x7fb33c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.197+0000 7fb340e00640 1 -- 172.21.15.62:0/1898438698 shutdown_connections 2024-03-23T11:23:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.197+0000 7fb340e00640 1 -- 172.21.15.62:0/1898438698 wait complete. 2024-03-23T11:23:48.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.199+0000 7fb340e00640 1 Processor -- start 2024-03-23T11:23:48.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.200+0000 7fb340e00640 1 -- start start 2024-03-23T11:23:48.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.201+0000 7fb340e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb33c0707a0 0x7fb33c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.201+0000 7fb340e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb33c0cc9b0 0x7fb33c154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.201+0000 7fb340e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb33c1431f0 0x7fb33c1367b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.201+0000 7fb340e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c155d30 con 0x7fb33c1431f0 2024-03-23T11:23:48.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.201+0000 7fb340e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c15e0a0 con 0x7fb33c0cc9b0 2024-03-23T11:23:48.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.201+0000 7fb340e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c139b80 con 0x7fb33c0707a0 2024-03-23T11:23:48.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.201+0000 7fb33be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb33c1431f0 0x7fb33c1367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.201+0000 7fb33be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb33c1431f0 0x7fb33c1367b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34792/0 (socket says 172.21.15.62:34792) 2024-03-23T11:23:48.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.202+0000 7fb33be00640 1 -- 172.21.15.62:0/3823749527 learned_addr learned my addr 172.21.15.62:0/3823749527 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:48.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.202+0000 7fb33b400640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb33c0707a0 0x7fb33c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.202+0000 7fb332a00640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb33c0cc9b0 0x7fb33c154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.202+0000 7fb33be00640 1 -- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb33c0707a0 msgr2=0x7fb33c06d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.202+0000 7fb33be00640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb33c0707a0 0x7fb33c06d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.202+0000 7fb33be00640 1 -- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb33c0cc9b0 msgr2=0x7fb33c154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.202+0000 7fb33be00640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb33c0cc9b0 0x7fb33c154ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.202+0000 7fb33be00640 1 -- 172.21.15.62:0/3823749527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334009a70 con 0x7fb33c1431f0 2024-03-23T11:23:48.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.202+0000 7fb33b400640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb33c0707a0 0x7fb33c06d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:48.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.202+0000 7fb33be00640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb33c1431f0 0x7fb33c1367b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb334009d90 tx=0x7fb334004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:48.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.203+0000 7fb333e00640 1 -- 172.21.15.62:0/3823749527 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334040650 con 0x7fb33c1431f0 2024-03-23T11:23:48.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.203+0000 7fb333e00640 1 -- 172.21.15.62:0/3823749527 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3340407b0 con 0x7fb33c1431f0 2024-03-23T11:23:48.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.203+0000 7fb333e00640 1 -- 172.21.15.62:0/3823749527 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334040b60 con 0x7fb33c1431f0 2024-03-23T11:23:48.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.203+0000 7fb340e00640 1 -- 172.21.15.62:0/3823749527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33c1588c0 con 0x7fb33c1431f0 2024-03-23T11:23:48.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.203+0000 7fb340e00640 1 -- 172.21.15.62:0/3823749527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33c158e40 con 0x7fb33c1431f0 2024-03-23T11:23:48.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.205+0000 7fb333e00640 1 -- 172.21.15.62:0/3823749527 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb334040cc0 con 0x7fb33c1431f0 2024-03-23T11:23:48.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.206+0000 7fb333e00640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb31806a7a0 0x7fb31806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.207+0000 7fb333e00640 1 -- 172.21.15.62:0/3823749527 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3340abbe0 con 0x7fb33c1431f0 2024-03-23T11:23:48.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.207+0000 7fb345016740 1 Processor -- start 2024-03-23T11:23:48.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.207+0000 7fb345016740 1 -- start start 2024-03-23T11:23:48.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.207+0000 7fb33b400640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb31806a7a0 0x7fb31806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.208+0000 7fb345016740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1adb60 0x55a32a1adf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.208+0000 7fb345016740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a1b1bb0 0x55a32a1b3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.208+0000 7fb345016740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1ddd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.208+0000 7fb345016740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32a11dd80 con 0x55a32a1dd8f0 2024-03-23T11:23:48.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.208+0000 7fb345016740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32a11b350 con 0x55a32a1b1bb0 2024-03-23T11:23:48.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.208+0000 7fb345016740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a32a116c40 con 0x55a32a1adb60 2024-03-23T11:23:48.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.208+0000 7fb33be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1ddd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.208+0000 7fb33be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1ddd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34806/0 (socket says 172.21.15.62:34806) 2024-03-23T11:23:48.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.208+0000 7fb33be00640 1 -- 172.21.15.62:0/567159616 learned_addr learned my addr 172.21.15.62:0/567159616 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:48.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb33b400640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb31806a7a0 0x7fb31806cc60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb32c007fd0 tx=0x7fb32c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:48.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb332a00640 1 --2- 172.21.15.62:0/567159616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a1b1bb0 0x55a32a1b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb33b400640 1 --2- 172.21.15.62:0/567159616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1adb60 0x55a32a1adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb33be00640 1 -- 172.21.15.62:0/567159616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1adb60 msgr2=0x55a32a1adf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:48.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb33be00640 1 --2- 172.21.15.62:0/567159616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1adb60 0x55a32a1adf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb33be00640 1 -- 172.21.15.62:0/567159616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a1b1bb0 msgr2=0x55a32a1b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:48.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb33be00640 1 --2- 172.21.15.62:0/567159616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a1b1bb0 0x55a32a1b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb33be00640 1 -- 172.21.15.62:0/567159616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a32a0e13e0 con 0x55a32a1dd8f0 2024-03-23T11:23:48.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb332a00640 1 --2- 172.21.15.62:0/567159616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a1b1bb0 0x55a32a1b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:48.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb33be00640 1 --2- 172.21.15.62:0/567159616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1ddd70 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fb334082ae0 tx=0x7fb334009450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e730cf88ce52963c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb32be00640 1 -- 172.21.15.62:0/567159616 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33407ea50 con 0x55a32a1dd8f0 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb32be00640 1 -- 172.21.15.62:0/567159616 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb334083010 con 0x55a32a1dd8f0 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.209+0000 7fb32be00640 1 -- 172.21.15.62:0/567159616 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33403cde0 con 0x55a32a1dd8f0 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.210+0000 7fb345016740 1 -- 172.21.15.62:0/567159616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 msgr2=0x55a32a1ddd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.210+0000 7fb345016740 1 --2- 172.21.15.62:0/567159616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1ddd70 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fb334082ae0 tx=0x7fb334009450 comp rx=0 tx=0).stop 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.210+0000 7fb345016740 1 -- 172.21.15.62:0/567159616 shutdown_connections 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.210+0000 7fb345016740 1 --2- 172.21.15.62:0/567159616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1ddd70 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.210+0000 7fb345016740 1 --2- 172.21.15.62:0/567159616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a1b1bb0 0x55a32a1b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.210+0000 7fb345016740 1 --2- 172.21.15.62:0/567159616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1adb60 0x55a32a1adf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.210+0000 7fb345016740 1 -- 172.21.15.62:0/567159616 >> 172.21.15.62:0/567159616 conn(0x55a32a1afbe0 msgr2=0x55a32a1b14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.210+0000 7fb345016740 1 -- 172.21.15.62:0/567159616 shutdown_connections 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.210+0000 7fb345016740 1 -- 172.21.15.62:0/567159616 wait complete. 2024-03-23T11:23:48.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.211+0000 7fb345016740 1 -- start start 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.211+0000 7fb345016740 1 Processor -- start 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.212+0000 7fb345016740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1db8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.212+0000 7fb345016740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1cc320 0x55a32a1dbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.212+0000 7fb345016740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 0x55a32a208170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.212+0000 7fb345016740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32a0b7ce0 con 0x55a32a1dd8f0 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.212+0000 7fb345016740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32a0b7980 con 0x55a32a207cf0 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.212+0000 7fb345016740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a329e7f000 con 0x55a32a1cc320 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.212+0000 7fb345016740 10 client.0 _ll_register_callbacks cb 0x55a32a110410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.212+0000 7fb33b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1db8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.212+0000 7fb33b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1db8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34822/0 (socket says 172.21.15.62:34822) 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.212+0000 7fb33b400640 1 -- 172.21.15.62:0/2302172867 learned_addr learned my addr 172.21.15.62:0/2302172867 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb33be00640 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 0x55a32a208170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb33b400640 1 -- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1cc320 msgr2=0x55a32a1dbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb332a00640 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1cc320 0x55a32a1dbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb33b400640 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1cc320 0x55a32a1dbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb33b400640 1 -- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 msgr2=0x55a32a208170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb33b400640 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 0x55a32a208170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb33b400640 1 -- 172.21.15.62:0/2302172867 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33c158e40 con 0x55a32a1dd8f0 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb33be00640 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 0x55a32a208170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb33b400640 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1db8b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb32c013750 tx=0x7fb32c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c556f5f48d7590 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb328c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb328c00640 1 -- 172.21.15.62:0/2302172867 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c005ce0 con 0x55a32a1dd8f0 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb328c00640 1 -- 172.21.15.62:0/2302172867 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb32c005e40 con 0x55a32a1dd8f0 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.213+0000 7fb345016740 1 -- 172.21.15.62:0/2302172867 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a329dc88d0 con 0x55a32a1dd8f0 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.214+0000 7fb328c00640 1 -- 172.21.15.62:0/2302172867 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb32c007110 con 0x55a32a1dd8f0 2024-03-23T11:23:48.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.214+0000 7fb345016740 10 client.6414 fetch_fsmap learned FSMap version 35 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.214+0000 7fb345016740 1 -- 172.21.15.62:0/2302172867 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55a32a1152c0 con 0x55a32a1dd8f0 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.214+0000 7fb328c00640 1 -- 172.21.15.62:0/2302172867 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a329dc88d0 con 0x55a32a1dd8f0 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.214+0000 7fb328c00640 1 -- 172.21.15.62:0/2302172867 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb32c007270 con 0x55a32a1dd8f0 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.214+0000 7fb345016740 10 client.6414 fetch_fsmap finished waiting for FSMap version 35 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.214+0000 7fb345016740 10 client.6414 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.214+0000 7fb345016740 20 client.6414 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.214+0000 7fb345016740 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55a32a1d0bf0 0x55a32a215660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.214+0000 7fb345016740 4 client.6414 mds_command: new command op to 6081 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.214+0000 7fb345016740 1 -- 172.21.15.62:0/2302172867 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a329e0b4a0 con 0x55a32a1d0bf0 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.215+0000 7fb332a00640 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55a32a1d0bf0 0x55a32a215660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.216+0000 7fb332a00640 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55a32a1d0bf0 0x55a32a215660 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c0a5e1fe929f3aa server_cookie=1c53c7288f21eb90 in_seq=0 out_seq=0 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.216+0000 7fb328c00640 10 client.6414 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:48.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.217+0000 7fb328c00640 1 -- 172.21.15.62:0/2302172867 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55a329e0b4a0 con 0x55a32a1d0bf0 2024-03-23T11:23:48.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.217+0000 7fb328c00640 10 client.6414 handle_command_reply: tid=0 2024-03-23T11:23:48.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 1 client.6414 shutdown 2024-03-23T11:23:48.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 1 -- 172.21.15.62:0/2302172867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 msgr2=0x55a32a1db8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1db8b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb32c013750 tx=0x7fb32c000ed0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 1 -- 172.21.15.62:0/2302172867 shutdown_connections 2024-03-23T11:23:48.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55a32a1d0bf0 0x55a32a215660 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 0x55a32a208170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1cc320 0x55a32a1dbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb328c00640 0 client.6414 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:48.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 1 --2- 172.21.15.62:0/2302172867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1db8b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 1 -- 172.21.15.62:0/2302172867 >> 172.21.15.62:0/2302172867 conn(0x55a32a1adb60 msgr2=0x55a32a1f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:48.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 1 -- 172.21.15.62:0/2302172867 shutdown_connections 2024-03-23T11:23:48.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 1 -- 172.21.15.62:0/2302172867 wait complete. 2024-03-23T11:23:48.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.218+0000 7fb345016740 20 client.6414 trim_cache size 0 max 16384 2024-03-23T11:23:48.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.231+0000 7fb345016740 1 Processor -- start 2024-03-23T11:23:48.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.231+0000 7fb345016740 1 -- start start 2024-03-23T11:23:48.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.232+0000 7fb345016740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a1cc320 0x55a32a1b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.232+0000 7fb345016740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1dd8f0 0x55a32a1cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.232+0000 7fb345016740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a207cf0 0x55a32a1cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.232+0000 7fb345016740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32a219b00 con 0x55a32a207cf0 2024-03-23T11:23:48.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.232+0000 7fb345016740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32a1ae3a0 con 0x55a32a1cc320 2024-03-23T11:23:48.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.232+0000 7fb345016740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a32a1dbcd0 con 0x55a32a1dd8f0 2024-03-23T11:23:48.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.232+0000 7fb33be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a207cf0 0x55a32a1cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.232+0000 7fb33be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a207cf0 0x55a32a1cfeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34826/0 (socket says 172.21.15.62:34826) 2024-03-23T11:23:48.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.232+0000 7fb33be00640 1 -- 172.21.15.62:0/3498167387 learned_addr learned my addr 172.21.15.62:0/3498167387 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:48.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.233+0000 7fb33be00640 1 -- 172.21.15.62:0/3498167387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1dd8f0 msgr2=0x55a32a1cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:48.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.233+0000 7fb33be00640 1 --2- 172.21.15.62:0/3498167387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1dd8f0 0x55a32a1cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.233+0000 7fb33be00640 1 -- 172.21.15.62:0/3498167387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a1cc320 msgr2=0x55a32a1b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:48.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.233+0000 7fb33b400640 1 --2- 172.21.15.62:0/3498167387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a1cc320 0x55a32a1b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.233+0000 7fb33be00640 1 --2- 172.21.15.62:0/3498167387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a1cc320 0x55a32a1b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.233+0000 7fb33be00640 1 -- 172.21.15.62:0/3498167387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a32a0b7170 con 0x55a32a207cf0 2024-03-23T11:23:48.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.233+0000 7fb33be00640 1 --2- 172.21.15.62:0/3498167387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a207cf0 0x55a32a1cfeb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fb33407abb0 tx=0x7fb3340ab000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b267b546a53871 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:48.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.233+0000 7fb330c00640 1 -- 172.21.15.62:0/3498167387 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33407ea50 con 0x55a32a207cf0 2024-03-23T11:23:48.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.233+0000 7fb330c00640 1 -- 172.21.15.62:0/3498167387 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33403cde0 con 0x55a32a207cf0 2024-03-23T11:23:48.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.233+0000 7fb330c00640 1 -- 172.21.15.62:0/3498167387 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334077330 con 0x55a32a207cf0 2024-03-23T11:23:48.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.234+0000 7fb345016740 1 -- 172.21.15.62:0/3498167387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a207cf0 msgr2=0x55a32a1cfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.234+0000 7fb345016740 1 --2- 172.21.15.62:0/3498167387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a207cf0 0x55a32a1cfeb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fb33407abb0 tx=0x7fb3340ab000 comp rx=0 tx=0).stop 2024-03-23T11:23:48.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.234+0000 7fb345016740 1 -- 172.21.15.62:0/3498167387 shutdown_connections 2024-03-23T11:23:48.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.234+0000 7fb345016740 1 --2- 172.21.15.62:0/3498167387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a207cf0 0x55a32a1cfeb0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.234+0000 7fb345016740 1 --2- 172.21.15.62:0/3498167387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1dd8f0 0x55a32a1cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.234+0000 7fb345016740 1 --2- 172.21.15.62:0/3498167387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a1cc320 0x55a32a1b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.234+0000 7fb345016740 1 -- 172.21.15.62:0/3498167387 >> 172.21.15.62:0/3498167387 conn(0x55a32a1adb60 msgr2=0x55a32a1ac910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:48.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.234+0000 7fb345016740 1 -- 172.21.15.62:0/3498167387 shutdown_connections 2024-03-23T11:23:48.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.234+0000 7fb345016740 1 -- 172.21.15.62:0/3498167387 wait complete. 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.235+0000 7fb345016740 1 -- start start 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.235+0000 7fb345016740 1 Processor -- start 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb345016740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1cc320 0x55a32a1d88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb345016740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1fbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb345016740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 0x55a32a1fc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb345016740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32a1d9150 con 0x55a32a1dd8f0 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb345016740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32a1dc1d0 con 0x55a32a207cf0 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb345016740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a32a26b830 con 0x55a32a1cc320 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb345016740 10 client.0 _ll_register_callbacks cb 0x7fb32c001d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb332a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1fbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb332a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1fbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34834/0 (socket says 172.21.15.62:34834) 2024-03-23T11:23:48.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb332a00640 1 -- 172.21.15.62:0/455723213 learned_addr learned my addr 172.21.15.62:0/455723213 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb33b400640 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1cc320 0x55a32a1d88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.236+0000 7fb33be00640 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 0x55a32a1fc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.237+0000 7fb332a00640 1 -- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1cc320 msgr2=0x55a32a1d88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.237+0000 7fb332a00640 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1cc320 0x55a32a1d88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.237+0000 7fb332a00640 1 -- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 msgr2=0x55a32a1fc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.237+0000 7fb332a00640 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 0x55a32a1fc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.237+0000 7fb332a00640 1 -- 172.21.15.62:0/455723213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a32a0e13e0 con 0x55a32a1dd8f0 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.237+0000 7fb33be00640 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 0x55a32a1fc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.237+0000 7fb332a00640 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1fbb20 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fb3240015e0 tx=0x7fb324018d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b42cb8a6f7fbad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.237+0000 7fb328c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.237+0000 7fb328c00640 1 -- 172.21.15.62:0/455723213 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3240052d0 con 0x55a32a1dd8f0 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.237+0000 7fb328c00640 1 -- 172.21.15.62:0/455723213 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb324005430 con 0x55a32a1dd8f0 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.237+0000 7fb345016740 1 -- 172.21.15.62:0/455723213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb334083010 con 0x55a32a1dd8f0 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.238+0000 7fb328c00640 1 -- 172.21.15.62:0/455723213 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3240035f0 con 0x55a32a1dd8f0 2024-03-23T11:23:48.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.238+0000 7fb345016740 10 client.6420 fetch_fsmap learned FSMap version 35 2024-03-23T11:23:48.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.238+0000 7fb345016740 1 -- 172.21.15.62:0/455723213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55a32a0dca50 con 0x55a32a1dd8f0 2024-03-23T11:23:48.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.238+0000 7fb328c00640 1 -- 172.21.15.62:0/455723213 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb334083010 con 0x55a32a1dd8f0 2024-03-23T11:23:48.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.238+0000 7fb328c00640 1 -- 172.21.15.62:0/455723213 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb324020070 con 0x55a32a1dd8f0 2024-03-23T11:23:48.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.238+0000 7fb345016740 10 client.6420 fetch_fsmap finished waiting for FSMap version 35 2024-03-23T11:23:48.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.238+0000 7fb345016740 10 client.6420 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:23:48.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.238+0000 7fb345016740 20 client.6420 populate_metadata read hostname 'smithi062' 2024-03-23T11:23:48.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.238+0000 7fb345016740 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55a32a1accf0 0x55a32a266ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.238+0000 7fb345016740 4 client.6420 mds_command: new command op to 6081 tid=0[{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-23T11:23:48.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.238+0000 7fb345016740 1 -- 172.21.15.62:0/455723213 --> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x55a329e1c7a0 con 0x55a32a1accf0 2024-03-23T11:23:48.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.239+0000 7fb33b400640 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55a32a1accf0 0x55a32a266ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.240+0000 7fb33b400640 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55a32a1accf0 0x55a32a266ea0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d38b8fef11025191 server_cookie=79f2d00c80a84e72 in_seq=0 out_seq=0 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.240+0000 7fb328c00640 10 client.6420 ms_handle_connect on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.240+0000 7fb328c00640 1 -- 172.21.15.62:0/455723213 <== mds.0 v2:172.21.15.120:6836/4158450746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a329e1c7a0 con 0x55a32a1accf0 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.240+0000 7fb328c00640 10 client.6420 handle_command_reply: tid=0 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.240+0000 7fb345016740 1 client.6420 shutdown 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb345016740 1 -- 172.21.15.62:0/455723213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 msgr2=0x55a32a1fbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb345016740 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1fbb20 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fb3240015e0 tx=0x7fb324018d40 comp rx=0 tx=0).stop 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb345016740 1 -- 172.21.15.62:0/455723213 shutdown_connections 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb345016740 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:6836/4158450746,v1:172.21.15.120:6837/4158450746] conn(0x55a32a1accf0 0x55a32a266ea0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb345016740 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a32a1dd8f0 0x55a32a1fbb20 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb345016740 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a32a207cf0 0x55a32a1fc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb345016740 1 --2- 172.21.15.62:0/455723213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a32a1cc320 0x55a32a1d88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb328c00640 0 client.6420 ms_handle_reset on v2:172.21.15.120:6836/4158450746 2024-03-23T11:23:48.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb345016740 1 -- 172.21.15.62:0/455723213 >> 172.21.15.62:0/455723213 conn(0x55a32a1adb60 msgr2=0x55a32a1dec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:48.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb345016740 1 -- 172.21.15.62:0/455723213 shutdown_connections 2024-03-23T11:23:48.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb345016740 1 -- 172.21.15.62:0/455723213 wait complete. 2024-03-23T11:23:48.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.241+0000 7fb345016740 20 client.6420 trim_cache size 0 max 16384 2024-03-23T11:23:48.239 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:48.239 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-03-23T11:23:48.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 -- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb31806a7a0 msgr2=0x7fb31806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb31806a7a0 0x7fb31806cc60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb32c007fd0 tx=0x7fb32c009040 comp rx=0 tx=0).stop 2024-03-23T11:23:48.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 -- 172.21.15.62:0/3823749527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb33c1431f0 msgr2=0x7fb33c1367b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb33c1431f0 0x7fb33c1367b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb334009d90 tx=0x7fb334004a40 comp rx=0 tx=0).stop 2024-03-23T11:23:48.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 -- 172.21.15.62:0/3823749527 shutdown_connections 2024-03-23T11:23:48.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb31806a7a0 0x7fb31806cc60 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb33c1431f0 0x7fb33c1367b0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb33c0cc9b0 0x7fb33c154ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 --2- 172.21.15.62:0/3823749527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb33c0707a0 0x7fb33c06d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 -- 172.21.15.62:0/3823749527 >> 172.21.15.62:0/3823749527 conn(0x7fb33c0b0090 msgr2=0x7fb33c15d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:48.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 -- 172.21.15.62:0/3823749527 shutdown_connections 2024-03-23T11:23:48.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.242+0000 7fb340e00640 1 -- 172.21.15.62:0/3823749527 wait complete. 2024-03-23T11:23:48.255 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-23T11:23:48.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.469+0000 7fe2ca400640 1 Processor -- start 2024-03-23T11:23:48.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.470+0000 7fe2ca400640 1 -- start start 2024-03-23T11:23:48.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.471+0000 7fe2ca400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40cf650 0x7fe2c40cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.471+0000 7fe2ca400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c406bfc0 con 0x7fe2c415cdc0 2024-03-23T11:23:48.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.471+0000 7fe2ca400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c40b5260 con 0x7fe2c415c9e0 2024-03-23T11:23:48.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.471+0000 7fe2ca400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40d0020 con 0x7fe2c40cf650 2024-03-23T11:23:48.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.472+0000 7fe2c2a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe2c415cdc0 0x7fe2c40d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39304/0 (socket says 172.21.15.62:39304) 2024-03-23T11:23:48.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.472+0000 7fe2c2a00640 1 -- 172.21.15.62:0/1814905583 learned_addr learned my addr 172.21.15.62:0/1814905583 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:48.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.472+0000 7fe2c3e00640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40cf650 0x7fe2c40cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.472+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627927382 0 0) 0x7fe2c406bfc0 con 0x7fe2c415cdc0 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.472+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a8003520 con 0x7fe2c415cdc0 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 810140066 0 0) 0x7fe2a8003520 con 0x7fe2c415cdc0 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40cf650 msgr2=0x7fe2c40cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40cf650 0x7fe2c40cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 >> v1:172.21.15.120:6789/0 conn(0x7fe2c415c9e0 legacy=0x7fe2c41607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c40d2630 con 0x7fe2c415cdc0 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fe2ac001670 con 0x7fe2c415cdc0 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 >> v1:172.21.15.62:6789/0 conn(0x7fe2c415cdc0 legacy=0x7fe2c40d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 0x7fe2a8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40cf650 0x7fe2a80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a8004d00 0x7fe2a8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40d0020 con 0x7fe2a80039a0 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a800ba60 con 0x7fe2a8004d00 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a800bc20 con 0x7fe2c40cf650 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c3400640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 0x7fe2a8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.473+0000 7fe2c2a00640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40cf650 0x7fe2a80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.474+0000 7fe2c3400640 1 -- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40cf650 msgr2=0x7fe2a80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.474+0000 7fe2c3400640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40cf650 0x7fe2a80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.474+0000 7fe2c3400640 1 -- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a8004d00 msgr2=0x7fe2a8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.474+0000 7fe2c3e00640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a8004d00 0x7fe2a8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.474+0000 7fe2c3400640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a8004d00 0x7fe2a8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.474+0000 7fe2c3400640 1 -- 172.21.15.62:0/1814905583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a8003520 con 0x7fe2a80039a0 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.474+0000 7fe2c2a00640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40cf650 0x7fe2a80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:48.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.474+0000 7fe2c3e00640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2a8004d00 0x7fe2a8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:48.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.475+0000 7fe2c3400640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 0x7fe2a8003dc0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4001e20 tx=0x7fe2a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924c11fab733052a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:48.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.475+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4003b70 con 0x7fe2a80039a0 2024-03-23T11:23:48.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.475+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b4003cd0 con 0x7fe2a80039a0 2024-03-23T11:23:48.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.475+0000 7fe2c2000640 1 -- 172.21.15.62:0/1814905583 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4005310 con 0x7fe2a80039a0 2024-03-23T11:23:48.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.476+0000 7fe2ca400640 1 -- 172.21.15.62:0/1814905583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 msgr2=0x7fe2a8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.476+0000 7fe2ca400640 1 --2- 172.21.15.62:0/1814905583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 0x7fe2a8003dc0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4001e20 tx=0x7fe2a80057f0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.476+0000 7fe2c3e00640 1 -- 172.21.15.62:0/1814905583 reap_dead start 2024-03-23T11:23:48.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.476+0000 7fe2ca400640 1 -- 172.21.15.62:0/1814905583 shutdown_connections 2024-03-23T11:23:48.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.476+0000 7fe2ca400640 1 -- 172.21.15.62:0/1814905583 >> 172.21.15.62:0/1814905583 conn(0x7fe2c4071570 msgr2=0x7fe2c4071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:48.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.476+0000 7fe2ca400640 1 -- 172.21.15.62:0/1814905583 shutdown_connections 2024-03-23T11:23:48.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.476+0000 7fe2ca400640 1 -- 172.21.15.62:0/1814905583 wait complete. 2024-03-23T11:23:48.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.478+0000 7fe2ca400640 1 Processor -- start 2024-03-23T11:23:48.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.478+0000 7fe2ca400640 1 -- start start 2024-03-23T11:23:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.479+0000 7fe2ca400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 0x7fe2c4156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.479+0000 7fe2ca400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40aa490 0x7fe2c40cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.479+0000 7fe2ca400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c40aa8f0 0x7fe2c40cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.479+0000 7fe2ca400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40a9430 con 0x7fe2a80039a0 2024-03-23T11:23:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.479+0000 7fe2ca400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40a96d0 con 0x7fe2c40aa8f0 2024-03-23T11:23:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.479+0000 7fe2ca400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40a99b0 con 0x7fe2c40aa490 2024-03-23T11:23:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.479+0000 7fe2c3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 0x7fe2c4156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.479+0000 7fe2c3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 0x7fe2c4156750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34860/0 (socket says 172.21.15.62:34860) 2024-03-23T11:23:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.479+0000 7fe2c3400640 1 -- 172.21.15.62:0/612180972 learned_addr learned my addr 172.21.15.62:0/612180972 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.480+0000 7fe2c3e00640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c40aa8f0 0x7fe2c40cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.480+0000 7fe2c2a00640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40aa490 0x7fe2c40cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.480+0000 7fe2c3400640 1 -- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40aa490 msgr2=0x7fe2c40cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.480+0000 7fe2c3400640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40aa490 0x7fe2c40cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.480+0000 7fe2c3400640 1 -- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c40aa8f0 msgr2=0x7fe2c40cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.480+0000 7fe2c3400640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c40aa8f0 0x7fe2c40cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.480+0000 7fe2c3400640 1 -- 172.21.15.62:0/612180972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c40a9c90 con 0x7fe2a80039a0 2024-03-23T11:23:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.480+0000 7fe2c2a00640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40aa490 0x7fe2c40cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:48.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.480+0000 7fe2c3400640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 0x7fe2c4156750 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe2b402bf70 tx=0x7fe2b40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:48.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.480+0000 7fe2bb400640 1 -- 172.21.15.62:0/612180972 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b403e070 con 0x7fe2a80039a0 2024-03-23T11:23:48.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.481+0000 7fe2bb400640 1 -- 172.21.15.62:0/612180972 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b4006b50 con 0x7fe2a80039a0 2024-03-23T11:23:48.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.481+0000 7fe2bb400640 1 -- 172.21.15.62:0/612180972 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4007030 con 0x7fe2a80039a0 2024-03-23T11:23:48.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.481+0000 7fe2ca400640 1 -- 172.21.15.62:0/612180972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c40a9430 con 0x7fe2a80039a0 2024-03-23T11:23:48.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.481+0000 7fe2ca400640 1 -- 172.21.15.62:0/612180972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c40ab030 con 0x7fe2a80039a0 2024-03-23T11:23:48.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.482+0000 7fe2ca400640 1 -- 172.21.15.62:0/612180972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c40a9810 con 0x7fe2a80039a0 2024-03-23T11:23:48.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.483+0000 7fe2bb400640 1 -- 172.21.15.62:0/612180972 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe2b4007190 con 0x7fe2a80039a0 2024-03-23T11:23:48.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.484+0000 7fe2bb400640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2a006a7a0 0x7fe2a006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:48.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.484+0000 7fe2bb400640 1 -- 172.21.15.62:0/612180972 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2b40aa930 con 0x7fe2a80039a0 2024-03-23T11:23:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.485+0000 7fe2c2a00640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2a006a7a0 0x7fe2a006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:48.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.486+0000 7fe2c2a00640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2a006a7a0 0x7fe2a006cc60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac0017a0 tx=0x7fe2ac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:48.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.489+0000 7fe2bb400640 1 -- 172.21.15.62:0/612180972 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2c40a9810 con 0x7fe2a80039a0 2024-03-23T11:23:48.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.732+0000 7fe2ca400640 1 -- 172.21.15.62:0/612180972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe2c40a9810 con 0x7fe2a80039a0 2024-03-23T11:23:48.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.732+0000 7fe2bb400640 1 -- 172.21.15.62:0/612180972 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe2c40a9810 con 0x7fe2a80039a0 2024-03-23T11:23:48.729 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:48.729 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-23T11:23:48.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.735+0000 7fe2ca400640 1 -- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2a006a7a0 msgr2=0x7fe2a006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.735+0000 7fe2ca400640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2a006a7a0 0x7fe2a006cc60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac0017a0 tx=0x7fe2ac009040 comp rx=0 tx=0).stop 2024-03-23T11:23:48.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.735+0000 7fe2ca400640 1 -- 172.21.15.62:0/612180972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 msgr2=0x7fe2c4156750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:48.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.735+0000 7fe2ca400640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 0x7fe2c4156750 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe2b402bf70 tx=0x7fe2b40014a0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.735+0000 7fe2ca400640 1 -- 172.21.15.62:0/612180972 shutdown_connections 2024-03-23T11:23:48.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.735+0000 7fe2ca400640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe2a006a7a0 0x7fe2a006cc60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.735+0000 7fe2ca400640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2c40aa8f0 0x7fe2c40cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.735+0000 7fe2ca400640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe2c40aa490 0x7fe2c40cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.735+0000 7fe2ca400640 1 --2- 172.21.15.62:0/612180972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2a80039a0 0x7fe2c4156750 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:48.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.735+0000 7fe2ca400640 1 -- 172.21.15.62:0/612180972 >> 172.21.15.62:0/612180972 conn(0x7fe2c4071570 msgr2=0x7fe2c415bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:48.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.735+0000 7fe2ca400640 1 -- 172.21.15.62:0/612180972 shutdown_connections 2024-03-23T11:23:48.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:48.736+0000 7fe2ca400640 1 -- 172.21.15.62:0/612180972 wait complete. 2024-03-23T11:23:48.751 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-23T11:23:48.751 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:23:48.751 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:23:48.864 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:23:48.983 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:23:48.983 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:23:48.983 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:23:49.042 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:23:49.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.267+0000 7f5136a00640 1 Processor -- start 2024-03-23T11:23:49.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.268+0000 7f5136a00640 1 -- start start 2024-03-23T11:23:49.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.269+0000 7f5136a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cd690 0x7f51300cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.269+0000 7f5136a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51300d1100 con 0x7f513015cce0 2024-03-23T11:23:49.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.269+0000 7f5136a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51300d13e0 con 0x7f513015c900 2024-03-23T11:23:49.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.269+0000 7f5136a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51300ce060 con 0x7f51300cd690 2024-03-23T11:23:49.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.270+0000 7f512f400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f513015cce0 0x7f51300b82e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39318/0 (socket says 172.21.15.62:39318) 2024-03-23T11:23:49.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.270+0000 7f512f400640 1 -- 172.21.15.62:0/4206118190 learned_addr learned my addr 172.21.15.62:0/4206118190 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.270+0000 7f5134a00640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cd690 0x7f51300cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.270+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486503081 0 0) 0x7f51300d1100 con 0x7f513015cce0 2024-03-23T11:23:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.270+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5114003520 con 0x7f513015cce0 2024-03-23T11:23:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2641200919 0 0) 0x7f5114003520 con 0x7f513015cce0 2024-03-23T11:23:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cd690 msgr2=0x7f51300cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cd690 0x7f51300cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 >> v1:172.21.15.120:6789/0 conn(0x7f513015c900 legacy=0x7f51300d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51300d16c0 con 0x7f513015cce0 2024-03-23T11:23:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f5118001670 con 0x7f513015cce0 2024-03-23T11:23:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 >> v1:172.21.15.62:6789/0 conn(0x7f513015cce0 legacy=0x7f51300b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 0x7f5114003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51140047c0 0x7f5114004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cd690 0x7f5114005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51300ce060 con 0x7f51140039a0 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511400ba90 con 0x7f51140047c0 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511400bc50 con 0x7f51300cd690 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512fe00640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 0x7f5114003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.271+0000 7f512f400640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51140047c0 0x7f5114004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.272+0000 7f5134a00640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cd690 0x7f5114005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.272+0000 7f512fe00640 1 -- 172.21.15.62:0/4206118190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cd690 msgr2=0x7f5114005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.272+0000 7f512fe00640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cd690 0x7f5114005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.272+0000 7f512fe00640 1 -- 172.21.15.62:0/4206118190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51140047c0 msgr2=0x7f5114004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.272+0000 7f512fe00640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51140047c0 0x7f5114004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.272+0000 7f512fe00640 1 -- 172.21.15.62:0/4206118190 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5114003520 con 0x7f51140039a0 2024-03-23T11:23:49.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.273+0000 7f512fe00640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 0x7f5114003dc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f5120001720 tx=0x7f51140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972d5edaeaa9e2eb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.273+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51200052d0 con 0x7f51140039a0 2024-03-23T11:23:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.273+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512002e050 con 0x7f51140039a0 2024-03-23T11:23:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.273+0000 7f512ea00640 1 -- 172.21.15.62:0/4206118190 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51200035f0 con 0x7f51140039a0 2024-03-23T11:23:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.274+0000 7f5136a00640 1 -- 172.21.15.62:0/4206118190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 msgr2=0x7f5114003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.274+0000 7f5136a00640 1 --2- 172.21.15.62:0/4206118190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 0x7f5114003dc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f5120001720 tx=0x7f51140057f0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.274+0000 7f5134a00640 1 -- 172.21.15.62:0/4206118190 reap_dead start 2024-03-23T11:23:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.274+0000 7f5136a00640 1 -- 172.21.15.62:0/4206118190 shutdown_connections 2024-03-23T11:23:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.274+0000 7f5136a00640 1 -- 172.21.15.62:0/4206118190 >> 172.21.15.62:0/4206118190 conn(0x7f5130071390 msgr2=0x7f5130071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.274+0000 7f5136a00640 1 -- 172.21.15.62:0/4206118190 shutdown_connections 2024-03-23T11:23:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.274+0000 7f5136a00640 1 -- 172.21.15.62:0/4206118190 wait complete. 2024-03-23T11:23:49.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.276+0000 7f5136a00640 1 Processor -- start 2024-03-23T11:23:49.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.276+0000 7f5136a00640 1 -- start start 2024-03-23T11:23:49.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.277+0000 7f5136a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 0x7f513014fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.277+0000 7f5136a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51300cc470 0x7f5130159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.277+0000 7f5136a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cc8f0 0x7f5130159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.277+0000 7f5136a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5130151380 con 0x7f51140039a0 2024-03-23T11:23:49.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.277+0000 7f5136a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5130151560 con 0x7f51300cc470 2024-03-23T11:23:49.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.277+0000 7f5136a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5130151840 con 0x7f51300cc8f0 2024-03-23T11:23:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.277+0000 7f512fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 0x7f513014fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.277+0000 7f512fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 0x7f513014fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34866/0 (socket says 172.21.15.62:34866) 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.277+0000 7f512fe00640 1 -- 172.21.15.62:0/1980337783 learned_addr learned my addr 172.21.15.62:0/1980337783 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.278+0000 7f512f400640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51300cc470 0x7f5130159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.278+0000 7f5134a00640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cc8f0 0x7f5130159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.278+0000 7f512fe00640 1 -- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cc8f0 msgr2=0x7f5130159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.278+0000 7f512fe00640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cc8f0 0x7f5130159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.278+0000 7f512fe00640 1 -- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51300cc470 msgr2=0x7f5130159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.278+0000 7f512fe00640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51300cc470 0x7f5130159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.278+0000 7f512fe00640 1 -- 172.21.15.62:0/1980337783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5130151b20 con 0x7f51140039a0 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.278+0000 7f512fe00640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 0x7f513014fea0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f5120008fd0 tx=0x7f5120001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.279+0000 7f5127e00640 1 -- 172.21.15.62:0/1980337783 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51200076c0 con 0x7f51140039a0 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.279+0000 7f5127e00640 1 -- 172.21.15.62:0/1980337783 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5120007a30 con 0x7f51140039a0 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.279+0000 7f5127e00640 1 -- 172.21.15.62:0/1980337783 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5120006be0 con 0x7f51140039a0 2024-03-23T11:23:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.279+0000 7f5136a00640 1 -- 172.21.15.62:0/1980337783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5130151e00 con 0x7f51140039a0 2024-03-23T11:23:49.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.279+0000 7f5136a00640 1 -- 172.21.15.62:0/1980337783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51301503e0 con 0x7f51140039a0 2024-03-23T11:23:49.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.280+0000 7f5136a00640 1 -- 172.21.15.62:0/1980337783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51300672c0 con 0x7f51140039a0 2024-03-23T11:23:49.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.281+0000 7f5127e00640 1 -- 172.21.15.62:0/1980337783 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5120006d40 con 0x7f51140039a0 2024-03-23T11:23:49.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.282+0000 7f5127e00640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f510c06a7d0 0x7f510c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.282+0000 7f5127e00640 1 -- 172.21.15.62:0/1980337783 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51200aae00 con 0x7f51140039a0 2024-03-23T11:23:49.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.283+0000 7f512f400640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f510c06a7d0 0x7f510c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.284+0000 7f512f400640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f510c06a7d0 0x7f510c06cc90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5130071360 tx=0x7f5118009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:49.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.286+0000 7f5127e00640 1 -- 172.21.15.62:0/1980337783 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51200763d0 con 0x7f51140039a0 2024-03-23T11:23:49.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.492+0000 7f5136a00640 1 -- 172.21.15.62:0/1980337783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f513015b030 con 0x7f51140039a0 2024-03-23T11:23:49.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.495+0000 7f5127e00640 1 -- 172.21.15.62:0/1980337783 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16097 (secure 0 0 0) 0x7f51200799f0 con 0x7f51140039a0 2024-03-23T11:23:49.491 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:49.491 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":62,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:22:32.731093+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:22:29.767222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-23T11:22:30.738596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:23:49.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.498+0000 7f5136a00640 1 -- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f510c06a7d0 msgr2=0x7f510c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.498+0000 7f5136a00640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f510c06a7d0 0x7f510c06cc90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5130071360 tx=0x7f5118009040 comp rx=0 tx=0).stop 2024-03-23T11:23:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.498+0000 7f5136a00640 1 -- 172.21.15.62:0/1980337783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 msgr2=0x7f513014fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.498+0000 7f5136a00640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 0x7f513014fea0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f5120008fd0 tx=0x7f5120001120 comp rx=0 tx=0).stop 2024-03-23T11:23:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.498+0000 7f5136a00640 1 -- 172.21.15.62:0/1980337783 shutdown_connections 2024-03-23T11:23:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.498+0000 7f5136a00640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f510c06a7d0 0x7f510c06cc90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.498+0000 7f5136a00640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300cc8f0 0x7f5130159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.498+0000 7f5136a00640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51300cc470 0x7f5130159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.499+0000 7f5136a00640 1 --2- 172.21.15.62:0/1980337783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51140039a0 0x7f513014fea0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.499+0000 7f5136a00640 1 -- 172.21.15.62:0/1980337783 >> 172.21.15.62:0/1980337783 conn(0x7f5130071390 msgr2=0x7f5130158c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.499+0000 7f5136a00640 1 -- 172.21.15.62:0/1980337783 shutdown_connections 2024-03-23T11:23:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.499+0000 7f5136a00640 1 -- 172.21.15.62:0/1980337783 wait complete. 2024-03-23T11:23:49.515 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:22:29.767222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:22:30.738596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:23:49.515 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:23:49.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.739+0000 7f988ce00640 1 Processor -- start 2024-03-23T11:23:49.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.739+0000 7f988ce00640 1 -- start start 2024-03-23T11:23:49.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.741+0000 7f988ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9888070090 0x7f9888070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.741+0000 7f988ce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98880b2820 0x7f98880b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.741+0000 7f988ce00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98880b31b0 con 0x7f9888143dd0 2024-03-23T11:23:49.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.741+0000 7f988ce00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9888070a60 con 0x7f9888070090 2024-03-23T11:23:49.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.741+0000 7f988ce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98880cd140 con 0x7f98880b2820 2024-03-23T11:23:49.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.741+0000 7f9887400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9888143dd0 0x7f988815abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39322/0 (socket says 172.21.15.62:39322) 2024-03-23T11:23:49.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.741+0000 7f9887400640 1 -- 172.21.15.62:0/3639770979 learned_addr learned my addr 172.21.15.62:0/3639770979 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:49.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.741+0000 7f9886a00640 1 --2- 172.21.15.62:0/3639770979 >> v2:172.21.15.62:3300/0 conn(0x7f9888070090 0x7f9888070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.742+0000 7f9887e00640 1 --2- 172.21.15.62:0/3639770979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98880b2820 0x7f98880b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.742+0000 7f9886000640 1 -- 172.21.15.62:0/3639770979 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182070190 0 0) 0x7f98880b31b0 con 0x7f9888143dd0 2024-03-23T11:23:49.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.742+0000 7f9886000640 1 -- 172.21.15.62:0/3639770979 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f986c003520 con 0x7f9888143dd0 2024-03-23T11:23:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.742+0000 7f9886a00640 1 -- 172.21.15.62:0/3639770979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98880b2820 msgr2=0x7f98880b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.742+0000 7f9886a00640 1 --2- 172.21.15.62:0/3639770979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98880b2820 0x7f98880b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.742+0000 7f9886a00640 1 -- 172.21.15.62:0/3639770979 >> v1:172.21.15.62:6789/0 conn(0x7f9888143dd0 legacy=0x7f988815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.742+0000 7f9886a00640 1 -- 172.21.15.62:0/3639770979 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98880cd3e0 con 0x7f9888070090 2024-03-23T11:23:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.743+0000 7f9886a00640 1 --2- 172.21.15.62:0/3639770979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9888070090 0x7f9888070520 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f98700097a0 tx=0x7f987002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf6b06cab96cfb48 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:49.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.744+0000 7f9886000640 1 -- 172.21.15.62:0/3639770979 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9870031830 con 0x7f9888070090 2024-03-23T11:23:49.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.744+0000 7f9886000640 1 -- 172.21.15.62:0/3639770979 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9870031990 con 0x7f9888070090 2024-03-23T11:23:49.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.744+0000 7f9886000640 1 -- 172.21.15.62:0/3639770979 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9870031d00 con 0x7f9888070090 2024-03-23T11:23:49.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.744+0000 7f988ce00640 1 -- 172.21.15.62:0/3639770979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9888070090 msgr2=0x7f9888070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.744+0000 7f988ce00640 1 --2- 172.21.15.62:0/3639770979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9888070090 0x7f9888070520 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f98700097a0 tx=0x7f987002b8d0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.744+0000 7f988ce00640 1 -- 172.21.15.62:0/3639770979 shutdown_connections 2024-03-23T11:23:49.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.745+0000 7f988ce00640 1 --2- 172.21.15.62:0/3639770979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f98880b2820 0x7f98880b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.745+0000 7f988ce00640 1 --2- 172.21.15.62:0/3639770979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9888070090 0x7f9888070520 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.745+0000 7f988ce00640 1 -- 172.21.15.62:0/3639770979 >> 172.21.15.62:0/3639770979 conn(0x7f9888066050 msgr2=0x7f9888069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:49.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.745+0000 7f988ce00640 1 -- 172.21.15.62:0/3639770979 shutdown_connections 2024-03-23T11:23:49.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.745+0000 7f988ce00640 1 -- 172.21.15.62:0/3639770979 wait complete. 2024-03-23T11:23:49.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.746+0000 7f988ce00640 1 Processor -- start 2024-03-23T11:23:49.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.746+0000 7f988ce00640 1 -- start start 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.747+0000 7f988ce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9888070090 0x7f98881373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.747+0000 7f988ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98880b2820 0x7f98880ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.747+0000 7f988ce00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9888143dd0 0x7f988813dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.747+0000 7f988ce00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988813d240 con 0x7f98880b2820 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.747+0000 7f988ce00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98880ac290 con 0x7f9888143dd0 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.747+0000 7f988ce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98880ac570 con 0x7f9888070090 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9886a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98880b2820 0x7f98880ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9886a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98880b2820 0x7f98880ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34870/0 (socket says 172.21.15.62:34870) 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9886a00640 1 -- 172.21.15.62:0/2177954868 learned_addr learned my addr 172.21.15.62:0/2177954868 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9887e00640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9888143dd0 0x7f988813dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9887400640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9888070090 0x7f98881373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9886a00640 1 -- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9888070090 msgr2=0x7f98881373b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9886a00640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9888070090 0x7f98881373b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9886a00640 1 -- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9888143dd0 msgr2=0x7f988813dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9886a00640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9888143dd0 0x7f988813dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9886a00640 1 -- 172.21.15.62:0/2177954868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9870009400 con 0x7f98880b2820 2024-03-23T11:23:49.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9887400640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9888070090 0x7f98881373b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:49.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9887e00640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9888143dd0 0x7f988813dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:49.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.748+0000 7f9886a00640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98880b2820 0x7f98880ab840 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f987002b910 tx=0x7f98700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:49.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.749+0000 7f987f400640 1 -- 172.21.15.62:0/2177954868 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987003a8f0 con 0x7f98880b2820 2024-03-23T11:23:49.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.749+0000 7f987f400640 1 -- 172.21.15.62:0/2177954868 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987003aa70 con 0x7f98880b2820 2024-03-23T11:23:49.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.749+0000 7f987f400640 1 -- 172.21.15.62:0/2177954868 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987003ae20 con 0x7f98880b2820 2024-03-23T11:23:49.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.749+0000 7f988ce00640 1 -- 172.21.15.62:0/2177954868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988813d4e0 con 0x7f98880b2820 2024-03-23T11:23:49.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.749+0000 7f988ce00640 1 -- 172.21.15.62:0/2177954868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98881378f0 con 0x7f98880b2820 2024-03-23T11:23:49.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.750+0000 7f988ce00640 1 -- 172.21.15.62:0/2177954868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98880cbec0 con 0x7f98880b2820 2024-03-23T11:23:49.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.751+0000 7f987f400640 1 -- 172.21.15.62:0/2177954868 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f987004d450 con 0x7f98880b2820 2024-03-23T11:23:49.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.752+0000 7f987f400640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f986406a7a0 0x7f986406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:49.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.752+0000 7f987f400640 1 -- 172.21.15.62:0/2177954868 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98700aae80 con 0x7f98880b2820 2024-03-23T11:23:49.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.752+0000 7f9887400640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f986406a7a0 0x7f986406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:49.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.754+0000 7f9887400640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f986406a7a0 0x7f986406cc60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9878005880 tx=0x7f9878009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:49.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.756+0000 7f987f400640 1 -- 172.21.15.62:0/2177954868 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9870076210 con 0x7f98880b2820 2024-03-23T11:23:49.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.961+0000 7f988ce00640 1 -- 172.21.15.62:0/2177954868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9888066bb0 con 0x7f98880b2820 2024-03-23T11:23:49.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.961+0000 7f987f400640 1 -- 172.21.15.62:0/2177954868 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9870079830 con 0x7f98880b2820 2024-03-23T11:23:49.957 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:23:49.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.964+0000 7f988ce00640 1 -- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f986406a7a0 msgr2=0x7f986406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.964+0000 7f988ce00640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f986406a7a0 0x7f986406cc60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9878005880 tx=0x7f9878009040 comp rx=0 tx=0).stop 2024-03-23T11:23:49.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.964+0000 7f988ce00640 1 -- 172.21.15.62:0/2177954868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98880b2820 msgr2=0x7f98880ab840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:49.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.964+0000 7f988ce00640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98880b2820 0x7f98880ab840 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f987002b910 tx=0x7f98700046c0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.964+0000 7f988ce00640 1 -- 172.21.15.62:0/2177954868 shutdown_connections 2024-03-23T11:23:49.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.964+0000 7f988ce00640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f986406a7a0 0x7f986406cc60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.964+0000 7f988ce00640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98880b2820 0x7f98880ab840 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.964+0000 7f988ce00640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9888143dd0 0x7f988813dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.964+0000 7f988ce00640 1 --2- 172.21.15.62:0/2177954868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9888070090 0x7f98881373b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:49.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.964+0000 7f988ce00640 1 -- 172.21.15.62:0/2177954868 >> 172.21.15.62:0/2177954868 conn(0x7f9888066050 msgr2=0x7f9888143020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:49.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.965+0000 7f988ce00640 1 -- 172.21.15.62:0/2177954868 shutdown_connections 2024-03-23T11:23:49.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:49.965+0000 7f988ce00640 1 -- 172.21.15.62:0/2177954868 wait complete. 2024-03-23T11:23:49.978 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:23:50.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.195+0000 7f3931600640 1 Processor -- start 2024-03-23T11:23:50.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.195+0000 7f3931600640 1 -- start start 2024-03-23T11:23:50.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.197+0000 7f3931600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f392c15fcf0 0x7f392c0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:50.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.197+0000 7f3931600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f392c160420 0x7f392c1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:50.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.197+0000 7f3931600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f392c0d3540 con 0x7f392c0d06d0 2024-03-23T11:23:50.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.197+0000 7f3931600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f392c160df0 con 0x7f392c15fcf0 2024-03-23T11:23:50.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.197+0000 7f3931600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f392c0ce510 con 0x7f392c160420 2024-03-23T11:23:50.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.197+0000 7f392be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f392c15fcf0 0x7f392c0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:50.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.197+0000 7f392be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f392c15fcf0 0x7f392c0d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34882/0 (socket says 172.21.15.62:34882) 2024-03-23T11:23:50.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.197+0000 7f392be00640 1 -- 172.21.15.62:0/2148340543 learned_addr learned my addr 172.21.15.62:0/2148340543 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:50.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.198+0000 7f392b400640 1 --2- 172.21.15.62:0/2148340543 >> v2:172.21.15.120:3300/0 conn(0x7f392c160420 0x7f392c1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:50.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.198+0000 7f392be00640 1 -- 172.21.15.62:0/2148340543 >> v2:172.21.15.120:3300/0 conn(0x7f392c160420 msgr2=0x7f392c1608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:50.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.198+0000 7f392be00640 1 --2- 172.21.15.62:0/2148340543 >> v2:172.21.15.120:3300/0 conn(0x7f392c160420 0x7f392c1608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.198+0000 7f392be00640 1 -- 172.21.15.62:0/2148340543 >> v1:172.21.15.120:6789/0 conn(0x7f392c0d06d0 legacy=0x7f392c0d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:50.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.198+0000 7f392be00640 1 -- 172.21.15.62:0/2148340543 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c0cdc70 con 0x7f392c15fcf0 2024-03-23T11:23:50.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.200+0000 7f392be00640 1 --2- 172.21.15.62:0/2148340543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c15fcf0 0x7f392c0d3000 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f392c161540 tx=0x7f391c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e13606b5bf2ab24 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:50.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.200+0000 7f392aa00640 1 -- 172.21.15.62:0/2148340543 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c0317e0 con 0x7f392c15fcf0 2024-03-23T11:23:50.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.200+0000 7f392aa00640 1 -- 172.21.15.62:0/2148340543 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391c031940 con 0x7f392c15fcf0 2024-03-23T11:23:50.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.200+0000 7f392aa00640 1 -- 172.21.15.62:0/2148340543 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c031cd0 con 0x7f392c15fcf0 2024-03-23T11:23:50.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.201+0000 7f3931600640 1 -- 172.21.15.62:0/2148340543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c15fcf0 msgr2=0x7f392c0d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.201+0000 7f3931600640 1 --2- 172.21.15.62:0/2148340543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c15fcf0 0x7f392c0d3000 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f392c161540 tx=0x7f391c02b880 comp rx=0 tx=0).stop 2024-03-23T11:23:50.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.201+0000 7f3931600640 1 -- 172.21.15.62:0/2148340543 shutdown_connections 2024-03-23T11:23:50.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.201+0000 7f3931600640 1 --2- 172.21.15.62:0/2148340543 >> v2:172.21.15.120:3300/0 conn(0x7f392c160420 0x7f392c1608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.201+0000 7f3931600640 1 --2- 172.21.15.62:0/2148340543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c15fcf0 0x7f392c0d3000 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.201+0000 7f3931600640 1 -- 172.21.15.62:0/2148340543 >> 172.21.15.62:0/2148340543 conn(0x7f392c089f60 msgr2=0x7f392c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:50.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.201+0000 7f3931600640 1 -- 172.21.15.62:0/2148340543 shutdown_connections 2024-03-23T11:23:50.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.201+0000 7f3931600640 1 -- 172.21.15.62:0/2148340543 wait complete. 2024-03-23T11:23:50.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.203+0000 7f3931600640 1 Processor -- start 2024-03-23T11:23:50.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.203+0000 7f3931600640 1 -- start start 2024-03-23T11:23:50.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.204+0000 7f3931600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f392c0d06d0 0x7f392c0c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:50.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.204+0000 7f3931600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c15fcf0 0x7f392c1e62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:50.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.204+0000 7f3931600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f392c160420 0x7f392c06bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:50.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.204+0000 7f3931600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c1e6800 con 0x7f392c15fcf0 2024-03-23T11:23:50.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.204+0000 7f3931600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c147630 con 0x7f392c160420 2024-03-23T11:23:50.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.204+0000 7f3931600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c147910 con 0x7f392c0d06d0 2024-03-23T11:23:50.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.204+0000 7f392b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c15fcf0 0x7f392c1e62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:50.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.204+0000 7f392b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c15fcf0 0x7f392c1e62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34886/0 (socket says 172.21.15.62:34886) 2024-03-23T11:23:50.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.204+0000 7f392b400640 1 -- 172.21.15.62:0/1335980540 learned_addr learned my addr 172.21.15.62:0/1335980540 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f3930c00640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f392c160420 0x7f392c06bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f392be00640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f392c0d06d0 0x7f392c0c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f392b400640 1 -- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f392c0d06d0 msgr2=0x7f392c0c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f392b400640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f392c0d06d0 0x7f392c0c5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f392b400640 1 -- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f392c160420 msgr2=0x7f392c06bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f392b400640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f392c160420 0x7f392c06bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f392b400640 1 -- 172.21.15.62:0/1335980540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c0094d0 con 0x7f392c15fcf0 2024-03-23T11:23:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f3930c00640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f392c160420 0x7f392c06bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f392b400640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c15fcf0 0x7f392c1e62c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f391400d5f0 tx=0x7f391400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f3923e00640 1 -- 172.21.15.62:0/1335980540 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39140125f0 con 0x7f392c15fcf0 2024-03-23T11:23:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f3923e00640 1 -- 172.21.15.62:0/1335980540 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3914012750 con 0x7f392c15fcf0 2024-03-23T11:23:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f3923e00640 1 -- 172.21.15.62:0/1335980540 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914012b00 con 0x7f392c15fcf0 2024-03-23T11:23:50.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f3931600640 1 -- 172.21.15.62:0/1335980540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392c147bf0 con 0x7f392c15fcf0 2024-03-23T11:23:50.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.205+0000 7f3931600640 1 -- 172.21.15.62:0/1335980540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c1481b0 con 0x7f392c15fcf0 2024-03-23T11:23:50.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.207+0000 7f3931600640 1 -- 172.21.15.62:0/1335980540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392c148370 con 0x7f392c15fcf0 2024-03-23T11:23:50.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.208+0000 7f3923e00640 1 -- 172.21.15.62:0/1335980540 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3914012c60 con 0x7f392c15fcf0 2024-03-23T11:23:50.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.208+0000 7f3923e00640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f390806a7a0 0x7f390806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:50.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.209+0000 7f3923e00640 1 -- 172.21.15.62:0/1335980540 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39140892c0 con 0x7f392c15fcf0 2024-03-23T11:23:50.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.209+0000 7f392be00640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f390806a7a0 0x7f390806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:50.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.210+0000 7f392be00640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f390806a7a0 0x7f390806cc60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f391c036000 tx=0x7f391c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:50.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.213+0000 7f3923e00640 1 -- 172.21.15.62:0/1335980540 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3914054640 con 0x7f392c15fcf0 2024-03-23T11:23:50.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.412+0000 7f3931600640 1 -- 172.21.15.62:0/1335980540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f392c066bb0 con 0x7f392c15fcf0 2024-03-23T11:23:50.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.412+0000 7f3923e00640 1 -- 172.21.15.62:0/1335980540 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v62) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3914057c60 con 0x7f392c15fcf0 2024-03-23T11:23:50.408 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:23:50.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 -- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f390806a7a0 msgr2=0x7f390806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f390806a7a0 0x7f390806cc60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f391c036000 tx=0x7f391c03d040 comp rx=0 tx=0).stop 2024-03-23T11:23:50.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 -- 172.21.15.62:0/1335980540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c15fcf0 msgr2=0x7f392c1e62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c15fcf0 0x7f392c1e62c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f391400d5f0 tx=0x7f391400dab0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 -- 172.21.15.62:0/1335980540 shutdown_connections 2024-03-23T11:23:50.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f390806a7a0 0x7f390806cc60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c15fcf0 0x7f392c1e62c0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f392c160420 0x7f392c06bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 --2- 172.21.15.62:0/1335980540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f392c0d06d0 0x7f392c0c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 -- 172.21.15.62:0/1335980540 >> 172.21.15.62:0/1335980540 conn(0x7f392c089f60 msgr2=0x7f392c0d0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:50.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 -- 172.21.15.62:0/1335980540 shutdown_connections 2024-03-23T11:23:50.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.416+0000 7f3931600640 1 -- 172.21.15.62:0/1335980540 wait complete. 2024-03-23T11:23:50.431 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:23:50.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.652+0000 7f5cf6400640 1 Processor -- start 2024-03-23T11:23:50.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.653+0000 7f5cf6400640 1 -- start start 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.654+0000 7f5cf6400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5cf01608b0 0x7f5cf00b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.654+0000 7f5cf6400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf00ce510 con 0x7f5cf015fcf0 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.654+0000 7f5cf6400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf00cdc70 con 0x7f5cf0160420 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.654+0000 7f5cf6400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf00cdf50 con 0x7f5cf01608b0 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.654+0000 7f5cf5000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5cf015fcf0 0x7f5cf00d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39332/0 (socket says 172.21.15.62:39332) 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.654+0000 7f5cf5000640 1 -- 172.21.15.62:0/484887573 learned_addr learned my addr 172.21.15.62:0/484887573 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.655+0000 7f5cf5a00640 1 --2- 172.21.15.62:0/484887573 >> v2:172.21.15.62:3300/0 conn(0x7f5cf01608b0 0x7f5cf00b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.655+0000 7f5cefe00640 1 -- 172.21.15.62:0/484887573 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102467323 0 0) 0x7f5cf00ce510 con 0x7f5cf015fcf0 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.655+0000 7f5cefe00640 1 -- 172.21.15.62:0/484887573 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd4003520 con 0x7f5cf015fcf0 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.655+0000 7f5cf5a00640 1 -- 172.21.15.62:0/484887573 >> v1:172.21.15.120:6789/0 conn(0x7f5cf0160420 legacy=0x7f5cf00b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.655+0000 7f5cf5a00640 1 -- 172.21.15.62:0/484887573 >> v1:172.21.15.62:6789/0 conn(0x7f5cf015fcf0 legacy=0x7f5cf00d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.655+0000 7f5cf5a00640 1 -- 172.21.15.62:0/484887573 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf00ce230 con 0x7f5cf01608b0 2024-03-23T11:23:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.656+0000 7f5cf5a00640 1 --2- 172.21.15.62:0/484887573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cf01608b0 0x7f5cf00b64c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80097a0 tx=0x7f5ce802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570f263d6af3e62d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.657+0000 7f5cefe00640 1 -- 172.21.15.62:0/484887573 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8031830 con 0x7f5cf01608b0 2024-03-23T11:23:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.657+0000 7f5cefe00640 1 -- 172.21.15.62:0/484887573 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce8031990 con 0x7f5cf01608b0 2024-03-23T11:23:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.657+0000 7f5cefe00640 1 -- 172.21.15.62:0/484887573 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8031d20 con 0x7f5cf01608b0 2024-03-23T11:23:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.657+0000 7f5cf6400640 1 -- 172.21.15.62:0/484887573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cf01608b0 msgr2=0x7f5cf00b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.657+0000 7f5cf6400640 1 --2- 172.21.15.62:0/484887573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cf01608b0 0x7f5cf00b64c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80097a0 tx=0x7f5ce802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.657+0000 7f5cf6400640 1 -- 172.21.15.62:0/484887573 shutdown_connections 2024-03-23T11:23:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.657+0000 7f5cf6400640 1 --2- 172.21.15.62:0/484887573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cf01608b0 0x7f5cf00b64c0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.657+0000 7f5cf6400640 1 -- 172.21.15.62:0/484887573 >> 172.21.15.62:0/484887573 conn(0x7f5cf0089f60 msgr2=0x7f5cf00d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:50.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.658+0000 7f5cf6400640 1 -- 172.21.15.62:0/484887573 shutdown_connections 2024-03-23T11:23:50.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.658+0000 7f5cf6400640 1 -- 172.21.15.62:0/484887573 wait complete. 2024-03-23T11:23:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.659+0000 7f5cf6400640 1 Processor -- start 2024-03-23T11:23:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.659+0000 7f5cf6400640 1 -- start start 2024-03-23T11:23:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.660+0000 7f5cf6400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf015fcf0 0x7f5cf015ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.660+0000 7f5cf6400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cf0160420 0x7f5cf009d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.660+0000 7f5cf6400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cf01608b0 0x7f5cf00cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.660+0000 7f5cf6400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf01f2c40 con 0x7f5cf01608b0 2024-03-23T11:23:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.660+0000 7f5cf6400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf01f22a0 con 0x7f5cf015fcf0 2024-03-23T11:23:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.660+0000 7f5cf6400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf01f2580 con 0x7f5cf0160420 2024-03-23T11:23:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.661+0000 7f5cf5a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cf01608b0 0x7f5cf00cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.661+0000 7f5cf5a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cf01608b0 0x7f5cf00cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34904/0 (socket says 172.21.15.62:34904) 2024-03-23T11:23:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.661+0000 7f5cf5a00640 1 -- 172.21.15.62:0/2773262026 learned_addr learned my addr 172.21.15.62:0/2773262026 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.661+0000 7f5cf5a00640 1 -- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cf0160420 msgr2=0x7f5cf009d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.661+0000 7f5cf5a00640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cf0160420 0x7f5cf009d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.661+0000 7f5cf5a00640 1 -- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf015fcf0 msgr2=0x7f5cf015ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.661+0000 7f5cf5a00640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf015fcf0 0x7f5cf015ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.661+0000 7f5cf5a00640 1 -- 172.21.15.62:0/2773262026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce8009400 con 0x7f5cf01608b0 2024-03-23T11:23:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.662+0000 7f5cf5a00640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cf01608b0 0x7f5cf00cbe00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8034040 tx=0x7f5ce80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.662+0000 7f5ced600640 1 -- 172.21.15.62:0/2773262026 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce803a830 con 0x7f5cf01608b0 2024-03-23T11:23:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.662+0000 7f5ced600640 1 -- 172.21.15.62:0/2773262026 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce803a9b0 con 0x7f5cf01608b0 2024-03-23T11:23:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.662+0000 7f5cf6400640 1 -- 172.21.15.62:0/2773262026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf01f2860 con 0x7f5cf01608b0 2024-03-23T11:23:50.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.662+0000 7f5ced600640 1 -- 172.21.15.62:0/2773262026 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce803ad60 con 0x7f5cf01608b0 2024-03-23T11:23:50.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.662+0000 7f5cf6400640 1 -- 172.21.15.62:0/2773262026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf00b3f90 con 0x7f5cf01608b0 2024-03-23T11:23:50.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.663+0000 7f5cf6400640 1 -- 172.21.15.62:0/2773262026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf0066bb0 con 0x7f5cf01608b0 2024-03-23T11:23:50.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.664+0000 7f5ced600640 1 -- 172.21.15.62:0/2773262026 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5ce80443f0 con 0x7f5cf01608b0 2024-03-23T11:23:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.665+0000 7f5ced600640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ccc06a7a0 0x7f5ccc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.665+0000 7f5ced600640 1 -- 172.21.15.62:0/2773262026 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ce80aad80 con 0x7f5cf01608b0 2024-03-23T11:23:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.666+0000 7f5cf5000640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ccc06a7a0 0x7f5ccc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.667+0000 7f5cf5000640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ccc06a7a0 0x7f5ccc06cc60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5ce00020a0 tx=0x7f5ce0001690 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.670+0000 7f5ced600640 1 -- 172.21.15.62:0/2773262026 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5ce8076230 con 0x7f5cf01608b0 2024-03-23T11:23:50.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.872+0000 7f5cf6400640 1 -- 172.21.15.62:0/2773262026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5cf00b4d90 con 0x7f5cf01608b0 2024-03-23T11:23:50.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.873+0000 7f5ced600640 1 -- 172.21.15.62:0/2773262026 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v62) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ce8035020 con 0x7f5cf01608b0 2024-03-23T11:23:50.869 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:23:50.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.875+0000 7f5cf6400640 1 -- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ccc06a7a0 msgr2=0x7f5ccc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.875+0000 7f5cf6400640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ccc06a7a0 0x7f5ccc06cc60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5ce00020a0 tx=0x7f5ce0001690 comp rx=0 tx=0).stop 2024-03-23T11:23:50.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.875+0000 7f5cf6400640 1 -- 172.21.15.62:0/2773262026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cf01608b0 msgr2=0x7f5cf00cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:50.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.875+0000 7f5cf6400640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cf01608b0 0x7f5cf00cbe00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8034040 tx=0x7f5ce80046c0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.876+0000 7f5cf6400640 1 -- 172.21.15.62:0/2773262026 shutdown_connections 2024-03-23T11:23:50.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.876+0000 7f5cf6400640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ccc06a7a0 0x7f5ccc06cc60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.876+0000 7f5cf6400640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cf01608b0 0x7f5cf00cbe00 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.876+0000 7f5cf6400640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5cf0160420 0x7f5cf009d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.876+0000 7f5cf6400640 1 --2- 172.21.15.62:0/2773262026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5cf015fcf0 0x7f5cf015ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:50.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.876+0000 7f5cf6400640 1 -- 172.21.15.62:0/2773262026 >> 172.21.15.62:0/2773262026 conn(0x7f5cf0089f60 msgr2=0x7f5cf00b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:50.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.876+0000 7f5cf6400640 1 -- 172.21.15.62:0/2773262026 shutdown_connections 2024-03-23T11:23:50.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:50.876+0000 7f5cf6400640 1 -- 172.21.15.62:0/2773262026 wait complete. 2024-03-23T11:23:50.889 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:23:51.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.098+0000 7f5d58c00640 1 Processor -- start 2024-03-23T11:23:51.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.098+0000 7f5d58c00640 1 -- start start 2024-03-23T11:23:51.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.099+0000 7f5d58c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d5415c9e0 0x7f5d541607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:51.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.099+0000 7f5d58c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d5415cdc0 0x7f5d540d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:51.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.099+0000 7f5d58c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5406bfc0 con 0x7f5d540cf650 2024-03-23T11:23:51.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.099+0000 7f5d58c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d540ce140 con 0x7f5d5415cdc0 2024-03-23T11:23:51.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.099+0000 7f5d58c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d540b4370 con 0x7f5d5415c9e0 2024-03-23T11:23:51.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.100+0000 7f5d51400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d5415cdc0 0x7f5d540d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:51.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.100+0000 7f5d51400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d5415cdc0 0x7f5d540d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34908/0 (socket says 172.21.15.62:34908) 2024-03-23T11:23:51.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.100+0000 7f5d51400640 1 -- 172.21.15.62:0/1761334503 learned_addr learned my addr 172.21.15.62:0/1761334503 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:51.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.100+0000 7f5d52800640 1 --1- 172.21.15.62:0/1761334503 >> v1:172.21.15.62:6789/0 conn(0x7f5d540cf650 0x7f5d5415d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39348/0 (socket says 172.21.15.62:39348) 2024-03-23T11:23:51.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.100+0000 7f5d51e00640 1 --2- 172.21.15.62:0/1761334503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d5415c9e0 0x7f5d541607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:51.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.100+0000 7f5d51400640 1 -- 172.21.15.62:0/1761334503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d5415c9e0 msgr2=0x7f5d541607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:51.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.100+0000 7f5d51400640 1 --2- 172.21.15.62:0/1761334503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d5415c9e0 0x7f5d541607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:51.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.100+0000 7f5d51400640 1 -- 172.21.15.62:0/1761334503 >> v1:172.21.15.62:6789/0 conn(0x7f5d540cf650 legacy=0x7f5d5415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:51.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.100+0000 7f5d51400640 1 -- 172.21.15.62:0/1761334503 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d540b5260 con 0x7f5d5415cdc0 2024-03-23T11:23:51.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.101+0000 7f5d51400640 1 --2- 172.21.15.62:0/1761334503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d5415cdc0 0x7f5d540d1f20 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009e50 tx=0x7f5d3c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9683522234ffcb68 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:51.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.102+0000 7f5d50a00640 1 -- 172.21.15.62:0/1761334503 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c02cb00 con 0x7f5d5415cdc0 2024-03-23T11:23:51.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.102+0000 7f5d50a00640 1 -- 172.21.15.62:0/1761334503 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c02cc60 con 0x7f5d5415cdc0 2024-03-23T11:23:51.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.102+0000 7f5d50a00640 1 -- 172.21.15.62:0/1761334503 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c0343f0 con 0x7f5d5415cdc0 2024-03-23T11:23:51.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.102+0000 7f5d58c00640 1 -- 172.21.15.62:0/1761334503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d5415cdc0 msgr2=0x7f5d540d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:51.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.102+0000 7f5d58c00640 1 --2- 172.21.15.62:0/1761334503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d5415cdc0 0x7f5d540d1f20 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009e50 tx=0x7f5d3c02bf60 comp rx=0 tx=0).stop 2024-03-23T11:23:51.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.102+0000 7f5d58c00640 1 -- 172.21.15.62:0/1761334503 shutdown_connections 2024-03-23T11:23:51.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.102+0000 7f5d58c00640 1 --2- 172.21.15.62:0/1761334503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d5415cdc0 0x7f5d540d1f20 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:51.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.102+0000 7f5d58c00640 1 --2- 172.21.15.62:0/1761334503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d5415c9e0 0x7f5d541607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:51.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.102+0000 7f5d58c00640 1 -- 172.21.15.62:0/1761334503 >> 172.21.15.62:0/1761334503 conn(0x7f5d54071570 msgr2=0x7f5d54071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:51.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.102+0000 7f5d58c00640 1 -- 172.21.15.62:0/1761334503 shutdown_connections 2024-03-23T11:23:51.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.102+0000 7f5d58c00640 1 -- 172.21.15.62:0/1761334503 wait complete. 2024-03-23T11:23:51.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.104+0000 7f5d58c00640 1 Processor -- start 2024-03-23T11:23:51.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.104+0000 7f5d58c00640 1 -- start start 2024-03-23T11:23:51.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.105+0000 7f5d58c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d540cf650 0x7f5d5415b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:51.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.105+0000 7f5d58c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d5415c9e0 0x7f5d540c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:51.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.105+0000 7f5d58c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d5415cdc0 0x7f5d540c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:51.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.105+0000 7f5d58c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5415bce0 con 0x7f5d540cf650 2024-03-23T11:23:51.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.105+0000 7f5d58c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d540a9f60 con 0x7f5d5415cdc0 2024-03-23T11:23:51.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.105+0000 7f5d58c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d541f2130 con 0x7f5d5415c9e0 2024-03-23T11:23:51.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.105+0000 7f5d51e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d540cf650 0x7f5d5415b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:51.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.105+0000 7f5d51e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d540cf650 0x7f5d5415b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34910/0 (socket says 172.21.15.62:34910) 2024-03-23T11:23:51.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.105+0000 7f5d51e00640 1 -- 172.21.15.62:0/640946315 learned_addr learned my addr 172.21.15.62:0/640946315 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:51.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.106+0000 7f5d51400640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d5415c9e0 0x7f5d540c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:51.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.106+0000 7f5d52800640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d5415cdc0 0x7f5d540c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:51.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.106+0000 7f5d51e00640 1 -- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d5415c9e0 msgr2=0x7f5d540c7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:51.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.106+0000 7f5d51e00640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d5415c9e0 0x7f5d540c7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:51.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.106+0000 7f5d51e00640 1 -- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d5415cdc0 msgr2=0x7f5d540c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:51.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.106+0000 7f5d51e00640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d5415cdc0 0x7f5d540c8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:51.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.106+0000 7f5d51e00640 1 -- 172.21.15.62:0/640946315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c009b30 con 0x7f5d540cf650 2024-03-23T11:23:51.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.106+0000 7f5d51e00640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d540cf650 0x7f5d5415b7a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400c7f0 tx=0x7f5d4400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:51.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.106+0000 7f5d4a000640 1 -- 172.21.15.62:0/640946315 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44004220 con 0x7f5d540cf650 2024-03-23T11:23:51.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.107+0000 7f5d4a000640 1 -- 172.21.15.62:0/640946315 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d440043a0 con 0x7f5d540cf650 2024-03-23T11:23:51.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.107+0000 7f5d4a000640 1 -- 172.21.15.62:0/640946315 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44007860 con 0x7f5d540cf650 2024-03-23T11:23:51.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.107+0000 7f5d58c00640 1 -- 172.21.15.62:0/640946315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d540c8a80 con 0x7f5d540cf650 2024-03-23T11:23:51.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.107+0000 7f5d58c00640 1 -- 172.21.15.62:0/640946315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d540c9040 con 0x7f5d540cf650 2024-03-23T11:23:51.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.108+0000 7f5d58c00640 1 -- 172.21.15.62:0/640946315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d541f22f0 con 0x7f5d540cf650 2024-03-23T11:23:51.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.109+0000 7f5d4a000640 1 -- 172.21.15.62:0/640946315 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5d44007bd0 con 0x7f5d540cf650 2024-03-23T11:23:51.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.109+0000 7f5d4a000640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5d3006a7a0 0x7f5d3006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:51.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.110+0000 7f5d4a000640 1 -- 172.21.15.62:0/640946315 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d44088cf0 con 0x7f5d540cf650 2024-03-23T11:23:51.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.110+0000 7f5d51400640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5d3006a7a0 0x7f5d3006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:51.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.111+0000 7f5d51400640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5d3006a7a0 0x7f5d3006cc60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c035000 tx=0x7f5d3c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:51.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.114+0000 7f5d4a000640 1 -- 172.21.15.62:0/640946315 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5d4408d070 con 0x7f5d540cf650 2024-03-23T11:23:51.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.332+0000 7f5d58c00640 1 -- 172.21.15.62:0/640946315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5d54066c90 con 0x7f5d540cf650 2024-03-23T11:23:51.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.333+0000 7f5d4a000640 1 -- 172.21.15.62:0/640946315 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v62) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5d4401e080 con 0x7f5d540cf650 2024-03-23T11:23:51.328 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:23:51.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.335+0000 7f5d58c00640 1 -- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5d3006a7a0 msgr2=0x7f5d3006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:51.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.335+0000 7f5d58c00640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5d3006a7a0 0x7f5d3006cc60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c035000 tx=0x7f5d3c030000 comp rx=0 tx=0).stop 2024-03-23T11:23:51.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.335+0000 7f5d58c00640 1 -- 172.21.15.62:0/640946315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d540cf650 msgr2=0x7f5d5415b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:51.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.335+0000 7f5d58c00640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d540cf650 0x7f5d5415b7a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400c7f0 tx=0x7f5d4400ccb0 comp rx=0 tx=0).stop 2024-03-23T11:23:51.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.335+0000 7f5d58c00640 1 -- 172.21.15.62:0/640946315 shutdown_connections 2024-03-23T11:23:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.335+0000 7f5d58c00640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5d3006a7a0 0x7f5d3006cc60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.335+0000 7f5d58c00640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d5415cdc0 0x7f5d540c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.335+0000 7f5d58c00640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d5415c9e0 0x7f5d540c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.335+0000 7f5d58c00640 1 --2- 172.21.15.62:0/640946315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d540cf650 0x7f5d5415b7a0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.336+0000 7f5d58c00640 1 -- 172.21.15.62:0/640946315 >> 172.21.15.62:0/640946315 conn(0x7f5d54071570 msgr2=0x7f5d540b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.336+0000 7f5d58c00640 1 -- 172.21.15.62:0/640946315 shutdown_connections 2024-03-23T11:23:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:51.336+0000 7f5d58c00640 1 -- 172.21.15.62:0/640946315 wait complete. 2024-03-23T11:23:51.350 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:23:51.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:23:51.350 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:23:51.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[56728]: fuse finished with error 0 and tester_r 0 2024-03-23T11:23:51.448 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:23:51.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:23:51.448 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:23:51.647 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:23:51.647 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:23:51.698 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:23:51.698 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:23:51.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:23:51.698 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T11:23:51.794 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-23T11:23:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:23:51.796 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:23:51.796 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:23:51.854 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:23:51.854 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:23:51.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:23:51.854 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T11:23:51.908 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-23T11:23:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:23:51.910 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:23:51.910 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:23:51.967 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:23:51.967 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:23:51.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:23:51.967 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T11:23:52.017 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-23T11:23:52.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:23:52.018 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:23:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.251+0000 7fce34e00640 1 Processor -- start 2024-03-23T11:23:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.252+0000 7fce34e00640 1 -- start start 2024-03-23T11:23:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.253+0000 7fce34e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fce30160420 0x7fce301608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.253+0000 7fce34e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce300ce510 con 0x7fce300b5db0 2024-03-23T11:23:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.253+0000 7fce34e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce300cdc70 con 0x7fce3015fcf0 2024-03-23T11:23:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.253+0000 7fce34e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce30160df0 con 0x7fce30160420 2024-03-23T11:23:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.254+0000 7fce2ea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fce30160420 0x7fce301608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.254+0000 7fce2ea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fce30160420 0x7fce301608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34922/0 (socket says 172.21.15.62:34922) 2024-03-23T11:23:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.254+0000 7fce2ea00640 1 -- 172.21.15.62:0/843766689 learned_addr learned my addr 172.21.15.62:0/843766689 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.254+0000 7fce2fe00640 1 --1- 172.21.15.62:0/843766689 >> v1:172.21.15.62:6789/0 conn(0x7fce300b5db0 0x7fce300b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39358/0 (socket says 172.21.15.62:39358) 2024-03-23T11:23:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.254+0000 7fce2ea00640 1 -- 172.21.15.62:0/843766689 >> v1:172.21.15.120:6789/0 conn(0x7fce3015fcf0 legacy=0x7fce300d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.254+0000 7fce2ea00640 1 -- 172.21.15.62:0/843766689 >> v1:172.21.15.62:6789/0 conn(0x7fce300b5db0 legacy=0x7fce300b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.254+0000 7fce2ea00640 1 -- 172.21.15.62:0/843766689 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce300cdf50 con 0x7fce30160420 2024-03-23T11:23:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.254+0000 7fce2e000640 1 -- 172.21.15.62:0/843766689 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564627104 0 0) 0x7fce300ce510 con 0x7fce300b5db0 2024-03-23T11:23:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.255+0000 7fce2ea00640 1 --2- 172.21.15.62:0/843766689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce30160420 0x7fce301608b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fce1c009e50 tx=0x7fce1c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233f50c816a45944 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.255+0000 7fce2e000640 1 -- 172.21.15.62:0/843766689 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c02ca80 con 0x7fce30160420 2024-03-23T11:23:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.256+0000 7fce2e000640 1 -- 172.21.15.62:0/843766689 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1c02cbe0 con 0x7fce30160420 2024-03-23T11:23:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.256+0000 7fce2e000640 1 -- 172.21.15.62:0/843766689 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c0343f0 con 0x7fce30160420 2024-03-23T11:23:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.256+0000 7fce34e00640 1 -- 172.21.15.62:0/843766689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce30160420 msgr2=0x7fce301608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.256+0000 7fce34e00640 1 --2- 172.21.15.62:0/843766689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce30160420 0x7fce301608b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fce1c009e50 tx=0x7fce1c02bee0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.256+0000 7fce34e00640 1 -- 172.21.15.62:0/843766689 shutdown_connections 2024-03-23T11:23:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.256+0000 7fce34e00640 1 --2- 172.21.15.62:0/843766689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce30160420 0x7fce301608b0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.256+0000 7fce34e00640 1 -- 172.21.15.62:0/843766689 >> 172.21.15.62:0/843766689 conn(0x7fce30089f60 msgr2=0x7fce300d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.256+0000 7fce34e00640 1 -- 172.21.15.62:0/843766689 shutdown_connections 2024-03-23T11:23:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.256+0000 7fce34e00640 1 -- 172.21.15.62:0/843766689 wait complete. 2024-03-23T11:23:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.258+0000 7fce34e00640 1 Processor -- start 2024-03-23T11:23:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.258+0000 7fce34e00640 1 -- start start 2024-03-23T11:23:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.259+0000 7fce34e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300b5db0 0x7fce30151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.259+0000 7fce34e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce3015fcf0 0x7fce300c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.259+0000 7fce34e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce30160420 0x7fce300ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.259+0000 7fce34e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce301520b0 con 0x7fce300b5db0 2024-03-23T11:23:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.259+0000 7fce34e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce301f6640 con 0x7fce3015fcf0 2024-03-23T11:23:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.259+0000 7fce34e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3009ec20 con 0x7fce30160420 2024-03-23T11:23:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.259+0000 7fce2f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300b5db0 0x7fce30151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.259+0000 7fce2f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300b5db0 0x7fce30151b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34928/0 (socket says 172.21.15.62:34928) 2024-03-23T11:23:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.259+0000 7fce2f400640 1 -- 172.21.15.62:0/1310059832 learned_addr learned my addr 172.21.15.62:0/1310059832 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.260+0000 7fce2ea00640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce3015fcf0 0x7fce300c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.260+0000 7fce2f400640 1 -- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce30160420 msgr2=0x7fce300ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.260+0000 7fce2f400640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce30160420 0x7fce300ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.260+0000 7fce2f400640 1 -- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce3015fcf0 msgr2=0x7fce300c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.260+0000 7fce2f400640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce3015fcf0 0x7fce300c77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.260+0000 7fce2f400640 1 -- 172.21.15.62:0/1310059832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1c009b30 con 0x7fce300b5db0 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.260+0000 7fce2f400640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300b5db0 0x7fce30151b70 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fce2000c430 tx=0x7fce2000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.260+0000 7fce27400640 1 -- 172.21.15.62:0/1310059832 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce20011790 con 0x7fce300b5db0 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.260+0000 7fce27400640 1 -- 172.21.15.62:0/1310059832 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce200118f0 con 0x7fce300b5db0 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.260+0000 7fce27400640 1 -- 172.21.15.62:0/1310059832 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce20011ce0 con 0x7fce300b5db0 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.260+0000 7fce34e00640 1 -- 172.21.15.62:0/1310059832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3009ef00 con 0x7fce300b5db0 2024-03-23T11:23:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.261+0000 7fce34e00640 1 -- 172.21.15.62:0/1310059832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce300c7d30 con 0x7fce300b5db0 2024-03-23T11:23:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.262+0000 7fce34e00640 1 -- 172.21.15.62:0/1310059832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3009f420 con 0x7fce300b5db0 2024-03-23T11:23:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.263+0000 7fce27400640 1 -- 172.21.15.62:0/1310059832 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fce2001c460 con 0x7fce300b5db0 2024-03-23T11:23:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.263+0000 7fce27400640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce1006a7f0 0x7fce1006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.264+0000 7fce27400640 1 -- 172.21.15.62:0/1310059832 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce20088d40 con 0x7fce300b5db0 2024-03-23T11:23:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.264+0000 7fce2ea00640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce1006a7f0 0x7fce1006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.265+0000 7fce2ea00640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce1006a7f0 0x7fce1006ccb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fce1c035000 tx=0x7fce1c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:52.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.268+0000 7fce27400640 1 -- 172.21.15.62:0/1310059832 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce3009f420 con 0x7fce300b5db0 2024-03-23T11:23:52.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.509+0000 7fce34e00640 1 -- 172.21.15.62:0/1310059832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce3009f420 con 0x7fce300b5db0 2024-03-23T11:23:52.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.510+0000 7fce27400640 1 -- 172.21.15.62:0/1310059832 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fce20054140 con 0x7fce300b5db0 2024-03-23T11:23:52.506 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:52.506 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3287045135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3287045135},{"type":"v1","addr":"172.21.15.120:6835","nonce":3287045135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6066,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/2859430143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2859430143},{"type":"v1","addr":"172.21.15.62:6834","nonce":2859430143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2005768060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2005768060},{"type":"v1","addr":"172.21.15.62:6835","nonce":2005768060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:22:31.725865+0000","modified":"2024-03-23T11:23:24.399625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6081},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6081":{"gid":6081,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":16,"addr":"172.21.15.120:6837/4158450746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":4158450746},{"type":"v1","addr":"172.21.15.120:6837","nonce":4158450746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T11:23:52.506 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-03-23T11:23:52.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 -- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce1006a7f0 msgr2=0x7fce1006ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:52.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce1006a7f0 0x7fce1006ccb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fce1c035000 tx=0x7fce1c030000 comp rx=0 tx=0).stop 2024-03-23T11:23:52.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 -- 172.21.15.62:0/1310059832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300b5db0 msgr2=0x7fce30151b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:52.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300b5db0 0x7fce30151b70 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fce2000c430 tx=0x7fce2000c8f0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 -- 172.21.15.62:0/1310059832 shutdown_connections 2024-03-23T11:23:52.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce1006a7f0 0x7fce1006ccb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce30160420 0x7fce300ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce3015fcf0 0x7fce300c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 --2- 172.21.15.62:0/1310059832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300b5db0 0x7fce30151b70 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 -- 172.21.15.62:0/1310059832 >> 172.21.15.62:0/1310059832 conn(0x7fce30089f60 msgr2=0x7fce300cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:52.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 -- 172.21.15.62:0/1310059832 shutdown_connections 2024-03-23T11:23:52.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.513+0000 7fce34e00640 1 -- 172.21.15.62:0/1310059832 wait complete. 2024-03-23T11:23:52.527 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:23:52.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.746+0000 7f12fac00640 1 Processor -- start 2024-03-23T11:23:52.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.747+0000 7f12fac00640 1 -- start start 2024-03-23T11:23:52.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.748+0000 7f12fac00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f12f415cbc0 0x7f12f40ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:52.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.748+0000 7f12fac00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f12f40b4880 0x7f12f40d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:52.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.748+0000 7f12fac00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f40cd3a0 con 0x7f12f40d0e20 2024-03-23T11:23:52.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.748+0000 7f12fac00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f40d17d0 con 0x7f12f415cbc0 2024-03-23T11:23:52.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.748+0000 7f12fac00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f40cd680 con 0x7f12f40b4880 2024-03-23T11:23:52.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.749+0000 7f12f9800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f12f415cbc0 0x7f12f40ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:52.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.749+0000 7f12f9800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f12f415cbc0 0x7f12f40ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34942/0 (socket says 172.21.15.62:34942) 2024-03-23T11:23:52.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.749+0000 7f12f9800640 1 -- 172.21.15.62:0/3655895914 learned_addr learned my addr 172.21.15.62:0/3655895914 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:52.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.749+0000 7f12fa200640 1 --2- 172.21.15.62:0/3655895914 >> v2:172.21.15.120:3300/0 conn(0x7f12f40b4880 0x7f12f40d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:52.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.749+0000 7f12f9800640 1 -- 172.21.15.62:0/3655895914 >> v2:172.21.15.120:3300/0 conn(0x7f12f40b4880 msgr2=0x7f12f40d1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:52.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.749+0000 7f12f9800640 1 --2- 172.21.15.62:0/3655895914 >> v2:172.21.15.120:3300/0 conn(0x7f12f40b4880 0x7f12f40d1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.749+0000 7f12f9800640 1 -- 172.21.15.62:0/3655895914 >> v1:172.21.15.120:6789/0 conn(0x7f12f40d0e20 legacy=0x7f12f40b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:52.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.749+0000 7f12f9800640 1 -- 172.21.15.62:0/3655895914 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f40cd960 con 0x7f12f415cbc0 2024-03-23T11:23:52.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.750+0000 7f12f9800640 1 --2- 172.21.15.62:0/3655895914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12f415cbc0 0x7f12f40ce440 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f12f408c2c0 tx=0x7f12e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2b3a6f196660aa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:52.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.751+0000 7f12f3e00640 1 -- 172.21.15.62:0/3655895914 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8030860 con 0x7f12f415cbc0 2024-03-23T11:23:52.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.751+0000 7f12f3e00640 1 -- 172.21.15.62:0/3655895914 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e80309c0 con 0x7f12f415cbc0 2024-03-23T11:23:52.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.751+0000 7f12f3e00640 1 -- 172.21.15.62:0/3655895914 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8030d50 con 0x7f12f415cbc0 2024-03-23T11:23:52.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.751+0000 7f12fac00640 1 -- 172.21.15.62:0/3655895914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12f415cbc0 msgr2=0x7f12f40ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:52.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.751+0000 7f12fac00640 1 --2- 172.21.15.62:0/3655895914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12f415cbc0 0x7f12f40ce440 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f12f408c2c0 tx=0x7f12e802b9a0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.751+0000 7f12fac00640 1 -- 172.21.15.62:0/3655895914 shutdown_connections 2024-03-23T11:23:52.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.751+0000 7f12fac00640 1 --2- 172.21.15.62:0/3655895914 >> v2:172.21.15.120:3300/0 conn(0x7f12f40b4880 0x7f12f40d1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.751+0000 7f12fac00640 1 --2- 172.21.15.62:0/3655895914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12f415cbc0 0x7f12f40ce440 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.751+0000 7f12fac00640 1 -- 172.21.15.62:0/3655895914 >> 172.21.15.62:0/3655895914 conn(0x7f12f4089f60 msgr2=0x7f12f40d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.752+0000 7f12fac00640 1 -- 172.21.15.62:0/3655895914 shutdown_connections 2024-03-23T11:23:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.752+0000 7f12fac00640 1 -- 172.21.15.62:0/3655895914 wait complete. 2024-03-23T11:23:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.753+0000 7f12fac00640 1 Processor -- start 2024-03-23T11:23:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.753+0000 7f12fac00640 1 -- start start 2024-03-23T11:23:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.754+0000 7f12fac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12f40b4880 0x7f12f415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.754+0000 7f12fac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12f40d0e20 0x7f12f40cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.754+0000 7f12fac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12f415cbc0 0x7f12f40cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.754+0000 7f12fac00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f40ca540 con 0x7f12f415cbc0 2024-03-23T11:23:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.754+0000 7f12fac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f40ca820 con 0x7f12f40b4880 2024-03-23T11:23:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.754+0000 7f12fac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f40cab00 con 0x7f12f40d0e20 2024-03-23T11:23:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12fa200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12f415cbc0 0x7f12f40cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12fa200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12f415cbc0 0x7f12f40cb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34954/0 (socket says 172.21.15.62:34954) 2024-03-23T11:23:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12fa200640 1 -- 172.21.15.62:0/2071533407 learned_addr learned my addr 172.21.15.62:0/2071533407 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12f8e00640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12f40d0e20 0x7f12f40cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12f9800640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12f40b4880 0x7f12f415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12fa200640 1 -- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12f40d0e20 msgr2=0x7f12f40cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12fa200640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12f40d0e20 0x7f12f40cb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12fa200640 1 -- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12f40b4880 msgr2=0x7f12f415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12fa200640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12f40b4880 0x7f12f415b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12fa200640 1 -- 172.21.15.62:0/2071533407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e80094d0 con 0x7f12f415cbc0 2024-03-23T11:23:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12f9800640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12f40b4880 0x7f12f415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.755+0000 7f12fa200640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12f415cbc0 0x7f12f40cb590 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009fd0 tx=0x7f12ec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:52.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.756+0000 7f12f1600640 1 -- 172.21.15.62:0/2071533407 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec014420 con 0x7f12f415cbc0 2024-03-23T11:23:52.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.756+0000 7f12f1600640 1 -- 172.21.15.62:0/2071533407 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec014580 con 0x7f12f415cbc0 2024-03-23T11:23:52.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.756+0000 7f12f1600640 1 -- 172.21.15.62:0/2071533407 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec014930 con 0x7f12f415cbc0 2024-03-23T11:23:52.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.756+0000 7f12fac00640 1 -- 172.21.15.62:0/2071533407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f415d2c0 con 0x7f12f415cbc0 2024-03-23T11:23:52.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.756+0000 7f12fac00640 1 -- 172.21.15.62:0/2071533407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f40c99f0 con 0x7f12f415cbc0 2024-03-23T11:23:52.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.757+0000 7f12fac00640 1 -- 172.21.15.62:0/2071533407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f415d4e0 con 0x7f12f415cbc0 2024-03-23T11:23:52.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.758+0000 7f12f1600640 1 -- 172.21.15.62:0/2071533407 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f12ec014a90 con 0x7f12f415cbc0 2024-03-23T11:23:52.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.759+0000 7f12f1600640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12d406a7a0 0x7f12d406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:52.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.759+0000 7f12f1600640 1 -- 172.21.15.62:0/2071533407 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12ec0896c0 con 0x7f12f415cbc0 2024-03-23T11:23:52.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.759+0000 7f12f9800640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12d406a7a0 0x7f12d406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:52.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.760+0000 7f12f9800640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12d406a7a0 0x7f12d406cc60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f12e8036000 tx=0x7f12e803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:52.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.763+0000 7f12f1600640 1 -- 172.21.15.62:0/2071533407 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f12ec054ae0 con 0x7f12f415cbc0 2024-03-23T11:23:52.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.966+0000 7f12fac00640 1 -- 172.21.15.62:0/2071533407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12f40d1270 con 0x7f12f415cbc0 2024-03-23T11:23:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.967+0000 7f12f1600640 1 -- 172.21.15.62:0/2071533407 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16097 (secure 0 0 0) 0x7f12ec058100 con 0x7f12f415cbc0 2024-03-23T11:23:52.964 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:52.964 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":62,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:22:32.731093+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:22:29.767222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-23T11:22:30.738596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:23:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.970+0000 7f12fac00640 1 -- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12d406a7a0 msgr2=0x7f12d406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.970+0000 7f12fac00640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12d406a7a0 0x7f12d406cc60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f12e8036000 tx=0x7f12e803b040 comp rx=0 tx=0).stop 2024-03-23T11:23:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.970+0000 7f12fac00640 1 -- 172.21.15.62:0/2071533407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12f415cbc0 msgr2=0x7f12f40cb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.970+0000 7f12fac00640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12f415cbc0 0x7f12f40cb590 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009fd0 tx=0x7f12ec00e4b0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.970+0000 7f12fac00640 1 -- 172.21.15.62:0/2071533407 shutdown_connections 2024-03-23T11:23:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.970+0000 7f12fac00640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f12d406a7a0 0x7f12d406cc60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.970+0000 7f12fac00640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12f415cbc0 0x7f12f40cb590 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.970+0000 7f12fac00640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12f40d0e20 0x7f12f40cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.970+0000 7f12fac00640 1 --2- 172.21.15.62:0/2071533407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12f40b4880 0x7f12f415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.970+0000 7f12fac00640 1 -- 172.21.15.62:0/2071533407 >> 172.21.15.62:0/2071533407 conn(0x7f12f4089f60 msgr2=0x7f12f40cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.970+0000 7f12fac00640 1 -- 172.21.15.62:0/2071533407 shutdown_connections 2024-03-23T11:23:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:52.971+0000 7f12fac00640 1 -- 172.21.15.62:0/2071533407 wait complete. 2024-03-23T11:23:52.986 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:22:29.767222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:22:30.738596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:23:52.986 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:23:53.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.210+0000 7f76ac200640 1 Processor -- start 2024-03-23T11:23:53.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.210+0000 7f76ac200640 1 -- start start 2024-03-23T11:23:53.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.212+0000 7f76ac200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76a406dca0 0x7f76a40b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.212+0000 7f76ac200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f76a406c300 0x7f76a40cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.212+0000 7f76ac200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a40cad20 con 0x7f76a40cd7c0 2024-03-23T11:23:53.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.212+0000 7f76ac200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a406c770 con 0x7f76a406c300 2024-03-23T11:23:53.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.212+0000 7f76ac200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a406e130 con 0x7f76a406dca0 2024-03-23T11:23:53.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.212+0000 7f76a9800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f76a40cd7c0 0x7f76a40cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39362/0 (socket says 172.21.15.62:39362) 2024-03-23T11:23:53.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.212+0000 7f76a9800640 1 -- 172.21.15.62:0/2149161219 learned_addr learned my addr 172.21.15.62:0/2149161219 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:53.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.213+0000 7f76aa200640 1 --2- 172.21.15.62:0/2149161219 >> v2:172.21.15.120:3300/0 conn(0x7f76a406c300 0x7f76a40cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.213+0000 7f76a8e00640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76a406dca0 0x7f76a40b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.213+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194851599 0 0) 0x7f76a40cad20 con 0x7f76a40cd7c0 2024-03-23T11:23:53.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.213+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768c005730 con 0x7f76a40cd7c0 2024-03-23T11:23:53.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.213+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1385467828 0 0) 0x7f768c005730 con 0x7f76a40cd7c0 2024-03-23T11:23:53.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.213+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76a406dca0 msgr2=0x7f76a40b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.213+0000 7f76a3e00640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76a406dca0 0x7f76a40b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.213+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 >> v2:172.21.15.120:3300/0 conn(0x7f76a406c300 msgr2=0x7f76a40cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.213+0000 7f76a3e00640 1 --2- 172.21.15.62:0/2149161219 >> v2:172.21.15.120:3300/0 conn(0x7f76a406c300 0x7f76a40cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.213+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a415a8a0 con 0x7f76a40cd7c0 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.213+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7698001670 con 0x7f76a40cd7c0 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 >> v1:172.21.15.62:6789/0 conn(0x7f76a40cd7c0 legacy=0x7f76a40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a3e00640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f768c005b80 0x7f768c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a3e00640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c0069d0 0x7f768c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a3e00640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f768c007390 0x7f768c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a406c770 con 0x7f768c0069d0 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a406e130 con 0x7f768c005b80 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c007880 con 0x7f768c007390 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a8e00640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c0069d0 0x7f768c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a9800640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f768c005b80 0x7f768c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76aa200640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f768c007390 0x7f768c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a8e00640 1 -- 172.21.15.62:0/2149161219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f768c007390 msgr2=0x7f768c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:53.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a8e00640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f768c007390 0x7f768c00db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.214+0000 7f76a8e00640 1 -- 172.21.15.62:0/2149161219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f768c005b80 msgr2=0x7f768c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:53.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.215+0000 7f76a8e00640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f768c005b80 0x7f768c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.215+0000 7f76a8e00640 1 -- 172.21.15.62:0/2149161219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c005730 con 0x7f768c0069d0 2024-03-23T11:23:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.216+0000 7f76a8e00640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c0069d0 0x7f768c006e50 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f7690009ac0 tx=0x7f768c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c9f30eb7c14964 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.216+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769002c6b0 con 0x7f768c0069d0 2024-03-23T11:23:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.216+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769002c810 con 0x7f768c0069d0 2024-03-23T11:23:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.216+0000 7f76a3e00640 1 -- 172.21.15.62:0/2149161219 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769002cba0 con 0x7f768c0069d0 2024-03-23T11:23:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.216+0000 7f76ac200640 1 -- 172.21.15.62:0/2149161219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c0069d0 msgr2=0x7f768c006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.216+0000 7f76ac200640 1 --2- 172.21.15.62:0/2149161219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c0069d0 0x7f768c006e50 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f7690009ac0 tx=0x7f768c007a00 comp rx=0 tx=0).stop 2024-03-23T11:23:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.217+0000 7f76aa200640 1 -- 172.21.15.62:0/2149161219 reap_dead start 2024-03-23T11:23:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.217+0000 7f76ac200640 1 -- 172.21.15.62:0/2149161219 shutdown_connections 2024-03-23T11:23:53.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.217+0000 7f76ac200640 1 -- 172.21.15.62:0/2149161219 >> 172.21.15.62:0/2149161219 conn(0x7f76a40b02e0 msgr2=0x7f76a40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:53.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.217+0000 7f76ac200640 1 -- 172.21.15.62:0/2149161219 shutdown_connections 2024-03-23T11:23:53.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.217+0000 7f76ac200640 1 -- 172.21.15.62:0/2149161219 wait complete. 2024-03-23T11:23:53.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.219+0000 7f76ac200640 1 Processor -- start 2024-03-23T11:23:53.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.219+0000 7f76ac200640 1 -- start start 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.220+0000 7f76ac200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f768c0069d0 0x7f76a40c3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.220+0000 7f76ac200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76a4156aa0 0x7f76a41455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.220+0000 7f76ac200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76a415c150 0x7f76a415c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.220+0000 7f76ac200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a415cb20 con 0x7f76a415c150 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.220+0000 7f76ac200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a4153ec0 con 0x7f768c0069d0 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.220+0000 7f76ac200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a4148f60 con 0x7f76a4156aa0 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.220+0000 7f76aa200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76a415c150 0x7f76a415c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.220+0000 7f76aa200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76a415c150 0x7f76a415c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34960/0 (socket says 172.21.15.62:34960) 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.220+0000 7f76aa200640 1 -- 172.21.15.62:0/3909182012 learned_addr learned my addr 172.21.15.62:0/3909182012 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.220+0000 7f76a9800640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f768c0069d0 0x7f76a40c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.220+0000 7f76a8e00640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76a4156aa0 0x7f76a41455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76aa200640 1 -- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76a4156aa0 msgr2=0x7f76a41455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76aa200640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76a4156aa0 0x7f76a41455a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76aa200640 1 -- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f768c0069d0 msgr2=0x7f76a40c3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76aa200640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f768c0069d0 0x7f76a40c3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76aa200640 1 -- 172.21.15.62:0/3909182012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a4146730 con 0x7f76a415c150 2024-03-23T11:23:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76a8e00640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76a4156aa0 0x7f76a41455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76aa200640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76a415c150 0x7f76a415c5e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f76a40b3a20 tx=0x7f769c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76a1600640 1 -- 172.21.15.62:0/3909182012 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c00e070 con 0x7f76a415c150 2024-03-23T11:23:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76a1600640 1 -- 172.21.15.62:0/3909182012 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769c011de0 con 0x7f76a415c150 2024-03-23T11:23:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76a1600640 1 -- 172.21.15.62:0/3909182012 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c003990 con 0x7f76a415c150 2024-03-23T11:23:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76ac200640 1 -- 172.21.15.62:0/3909182012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a40c3a90 con 0x7f76a415c150 2024-03-23T11:23:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.221+0000 7f76ac200640 1 -- 172.21.15.62:0/3909182012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a4148c80 con 0x7f76a415c150 2024-03-23T11:23:53.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.222+0000 7f76ac200640 1 -- 172.21.15.62:0/3909182012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a41468c0 con 0x7f76a415c150 2024-03-23T11:23:53.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.223+0000 7f76a1600640 1 -- 172.21.15.62:0/3909182012 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f769c003af0 con 0x7f76a415c150 2024-03-23T11:23:53.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.224+0000 7f76a1600640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f768406a7a0 0x7f768406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.224+0000 7f76a1600640 1 -- 172.21.15.62:0/3909182012 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f769c0894f0 con 0x7f76a415c150 2024-03-23T11:23:53.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.225+0000 7f76a9800640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f768406a7a0 0x7f768406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.226+0000 7f76a9800640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f768406a7a0 0x7f768406cc60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7698001050 tx=0x7f7698009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:53.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.229+0000 7f76a1600640 1 -- 172.21.15.62:0/3909182012 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f769c054870 con 0x7f76a415c150 2024-03-23T11:23:53.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.427+0000 7f76ac200640 1 -- 172.21.15.62:0/3909182012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f76a40b08f0 con 0x7f76a415c150 2024-03-23T11:23:53.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.428+0000 7f76a1600640 1 -- 172.21.15.62:0/3909182012 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f769c057e90 con 0x7f76a415c150 2024-03-23T11:23:53.424 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:23:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.431+0000 7f76ac200640 1 -- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f768406a7a0 msgr2=0x7f768406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.431+0000 7f76ac200640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f768406a7a0 0x7f768406cc60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7698001050 tx=0x7f7698009040 comp rx=0 tx=0).stop 2024-03-23T11:23:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.431+0000 7f76ac200640 1 -- 172.21.15.62:0/3909182012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76a415c150 msgr2=0x7f76a415c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.431+0000 7f76ac200640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76a415c150 0x7f76a415c5e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f76a40b3a20 tx=0x7f769c0115a0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.431+0000 7f76ac200640 1 -- 172.21.15.62:0/3909182012 shutdown_connections 2024-03-23T11:23:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.431+0000 7f76ac200640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f768406a7a0 0x7f768406cc60 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.431+0000 7f76ac200640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76a415c150 0x7f76a415c5e0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.431+0000 7f76ac200640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76a4156aa0 0x7f76a41455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.431+0000 7f76ac200640 1 --2- 172.21.15.62:0/3909182012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f768c0069d0 0x7f76a40c3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.432+0000 7f76ac200640 1 -- 172.21.15.62:0/3909182012 >> 172.21.15.62:0/3909182012 conn(0x7f76a40b02e0 msgr2=0x7f76a40b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.432+0000 7f76ac200640 1 -- 172.21.15.62:0/3909182012 shutdown_connections 2024-03-23T11:23:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.432+0000 7f76ac200640 1 -- 172.21.15.62:0/3909182012 wait complete. 2024-03-23T11:23:53.447 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:23:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.665+0000 7f7d8ec00640 1 Processor -- start 2024-03-23T11:23:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.666+0000 7f7d8ec00640 1 -- start start 2024-03-23T11:23:53.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.667+0000 7f7d8ec00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7d881608b0 0x7f7d880b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.667+0000 7f7d8ec00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d880ce510 con 0x7f7d8815fcf0 2024-03-23T11:23:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.667+0000 7f7d8ec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d880cdc70 con 0x7f7d88160420 2024-03-23T11:23:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.667+0000 7f7d8ec00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d880cdf50 con 0x7f7d881608b0 2024-03-23T11:23:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.667+0000 7f7d8d800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7d8815fcf0 0x7f7d880d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39372/0 (socket says 172.21.15.62:39372) 2024-03-23T11:23:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.667+0000 7f7d8d800640 1 -- 172.21.15.62:0/1140563111 learned_addr learned my addr 172.21.15.62:0/1140563111 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.668+0000 7f7d8e200640 1 --2- 172.21.15.62:0/1140563111 >> v2:172.21.15.120:3300/0 conn(0x7f7d881608b0 0x7f7d880b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.668+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421620964 0 0) 0x7f7d880ce510 con 0x7f7d8815fcf0 2024-03-23T11:23:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.668+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d70003520 con 0x7f7d8815fcf0 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.668+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 458805202 0 0) 0x7f7d70003520 con 0x7f7d8815fcf0 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.668+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 >> v2:172.21.15.120:3300/0 conn(0x7f7d881608b0 msgr2=0x7f7d880b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.668+0000 7f7d87e00640 1 --2- 172.21.15.62:0/1140563111 >> v2:172.21.15.120:3300/0 conn(0x7f7d881608b0 0x7f7d880b64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.668+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 >> v1:172.21.15.120:6789/0 conn(0x7f7d88160420 legacy=0x7f7d880b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.668+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d880ce230 con 0x7f7d8815fcf0 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.668+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7d7c001670 con 0x7f7d8815fcf0 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 >> v1:172.21.15.62:6789/0 conn(0x7f7d8815fcf0 legacy=0x7f7d880d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d87e00640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d70003970 0x7f7d70003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d87e00640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d700047c0 0x7f7d70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d87e00640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d70005180 0x7f7d7000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880cdf50 con 0x7f7d70005180 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7000c120 con 0x7f7d700047c0 2024-03-23T11:23:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7000c2e0 con 0x7f7d70003970 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d8e200640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d70005180 0x7f7d7000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d8d800640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d70003970 0x7f7d70003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d8ce00640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d700047c0 0x7f7d70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d8e200640 1 -- 172.21.15.62:0/1140563111 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d70003970 msgr2=0x7f7d70003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d8e200640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d70003970 0x7f7d70003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d8e200640 1 -- 172.21.15.62:0/1140563111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d700047c0 msgr2=0x7f7d70004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d8e200640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d700047c0 0x7f7d70004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d8e200640 1 -- 172.21.15.62:0/1140563111 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d70003520 con 0x7f7d70005180 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d8d800640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d70003970 0x7f7d70003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.669+0000 7f7d8ce00640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d700047c0 0x7f7d70004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.670+0000 7f7d8e200640 1 -- 172.21.15.62:0/1140563111 reap_dead start 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.671+0000 7f7d8e200640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d70005180 0x7f7d7000b980 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7d80009fc0 tx=0x7f7d700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e888e6554573d43a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.671+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8002cc60 con 0x7f7d70005180 2024-03-23T11:23:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.671+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8000f040 con 0x7f7d70005180 2024-03-23T11:23:53.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.671+0000 7f7d87e00640 1 -- 172.21.15.62:0/1140563111 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80003990 con 0x7f7d70005180 2024-03-23T11:23:53.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.671+0000 7f7d8ec00640 1 -- 172.21.15.62:0/1140563111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d70005180 msgr2=0x7f7d7000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.671+0000 7f7d8ec00640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d70005180 0x7f7d7000b980 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7d80009fc0 tx=0x7f7d700057f0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.671+0000 7f7d8ec00640 1 -- 172.21.15.62:0/1140563111 shutdown_connections 2024-03-23T11:23:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.671+0000 7f7d8ec00640 1 --2- 172.21.15.62:0/1140563111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d70005180 0x7f7d7000b980 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.671+0000 7f7d8ec00640 1 -- 172.21.15.62:0/1140563111 >> 172.21.15.62:0/1140563111 conn(0x7f7d88089f60 msgr2=0x7f7d880d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.672+0000 7f7d8ec00640 1 -- 172.21.15.62:0/1140563111 shutdown_connections 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.672+0000 7f7d8ec00640 1 -- 172.21.15.62:0/1140563111 wait complete. 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.673+0000 7f7d8ec00640 1 Processor -- start 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.674+0000 7f7d8ec00640 1 -- start start 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.674+0000 7f7d8ec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d70005180 0x7f7d8815dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.674+0000 7f7d8ec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d881f23a0 0x7f7d880cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d881f2820 0x7f7d880cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8809c650 con 0x7f7d881f23a0 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8809c8f0 con 0x7f7d70005180 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8809cbd0 con 0x7f7d881f2820 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d881f23a0 0x7f7d880cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d881f23a0 0x7f7d880cb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34980/0 (socket says 172.21.15.62:34980) 2024-03-23T11:23:53.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ce00640 1 -- 172.21.15.62:0/3659545100 learned_addr learned my addr 172.21.15.62:0/3659545100 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8d800640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d70005180 0x7f7d8815dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ce00640 1 -- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d881f2820 msgr2=0x7f7d880cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8e200640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d881f2820 0x7f7d880cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ce00640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d881f2820 0x7f7d880cbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ce00640 1 -- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d70005180 msgr2=0x7f7d8815dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ce00640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d70005180 0x7f7d8815dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.675+0000 7f7d8ce00640 1 -- 172.21.15.62:0/3659545100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8809ceb0 con 0x7f7d881f23a0 2024-03-23T11:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.676+0000 7f7d8ce00640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d881f23a0 0x7f7d880cb780 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800d770 tx=0x7f7d7800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.676+0000 7f7d85600640 1 -- 172.21.15.62:0/3659545100 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78005270 con 0x7f7d881f23a0 2024-03-23T11:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.676+0000 7f7d85600640 1 -- 172.21.15.62:0/3659545100 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d780053d0 con 0x7f7d881f23a0 2024-03-23T11:23:53.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.676+0000 7f7d85600640 1 -- 172.21.15.62:0/3659545100 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7800b8f0 con 0x7f7d881f23a0 2024-03-23T11:23:53.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.676+0000 7f7d8ec00640 1 -- 172.21.15.62:0/3659545100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8809d190 con 0x7f7d881f23a0 2024-03-23T11:23:53.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.676+0000 7f7d8ec00640 1 -- 172.21.15.62:0/3659545100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8809f000 con 0x7f7d881f23a0 2024-03-23T11:23:53.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.677+0000 7f7d8ec00640 1 -- 172.21.15.62:0/3659545100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d88066bb0 con 0x7f7d881f23a0 2024-03-23T11:23:53.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.678+0000 7f7d85600640 1 -- 172.21.15.62:0/3659545100 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7d7800ba50 con 0x7f7d881f23a0 2024-03-23T11:23:53.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.679+0000 7f7d85600640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7d6c06a7d0 0x7f7d6c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:53.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.679+0000 7f7d85600640 1 -- 172.21.15.62:0/3659545100 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d78088e20 con 0x7f7d881f23a0 2024-03-23T11:23:53.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.680+0000 7f7d8d800640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7d6c06a7d0 0x7f7d6c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:53.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.681+0000 7f7d8d800640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7d6c06a7d0 0x7f7d6c06cc90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c001200 tx=0x7f7d7c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:53.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.683+0000 7f7d85600640 1 -- 172.21.15.62:0/3659545100 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7d780541a0 con 0x7f7d881f23a0 2024-03-23T11:23:53.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.881+0000 7f7d8ec00640 1 -- 172.21.15.62:0/3659545100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7d880c9230 con 0x7f7d881f23a0 2024-03-23T11:23:53.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.882+0000 7f7d85600640 1 -- 172.21.15.62:0/3659545100 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v62) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d780577c0 con 0x7f7d881f23a0 2024-03-23T11:23:53.878 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:23:53.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.885+0000 7f7d8ec00640 1 -- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7d6c06a7d0 msgr2=0x7f7d6c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.885+0000 7f7d8ec00640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7d6c06a7d0 0x7f7d6c06cc90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c001200 tx=0x7f7d7c009040 comp rx=0 tx=0).stop 2024-03-23T11:23:53.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.885+0000 7f7d8ec00640 1 -- 172.21.15.62:0/3659545100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d881f23a0 msgr2=0x7f7d880cb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:53.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.885+0000 7f7d8ec00640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d881f23a0 0x7f7d880cb780 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800d770 tx=0x7f7d7800dc30 comp rx=0 tx=0).stop 2024-03-23T11:23:53.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.885+0000 7f7d8ec00640 1 -- 172.21.15.62:0/3659545100 shutdown_connections 2024-03-23T11:23:53.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.885+0000 7f7d8ec00640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7d6c06a7d0 0x7f7d6c06cc90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.885+0000 7f7d8ec00640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d881f23a0 0x7f7d880cb780 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.885+0000 7f7d8ec00640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d881f2820 0x7f7d880cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.885+0000 7f7d8ec00640 1 --2- 172.21.15.62:0/3659545100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d70005180 0x7f7d8815dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:53.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.885+0000 7f7d8ec00640 1 -- 172.21.15.62:0/3659545100 >> 172.21.15.62:0/3659545100 conn(0x7f7d88089f60 msgr2=0x7f7d8806ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:53.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.886+0000 7f7d8ec00640 1 -- 172.21.15.62:0/3659545100 shutdown_connections 2024-03-23T11:23:53.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:53.886+0000 7f7d8ec00640 1 -- 172.21.15.62:0/3659545100 wait complete. 2024-03-23T11:23:53.900 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:23:54.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.137+0000 7f63cbe00640 1 Processor -- start 2024-03-23T11:23:54.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.138+0000 7f63cbe00640 1 -- start start 2024-03-23T11:23:54.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.139+0000 7f63cbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63cc15fcf0 0x7f63cc0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:54.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.139+0000 7f63cbe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f63cc0b6dc0 0x7f63cc160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:54.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.139+0000 7f63cbe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc160dc0 con 0x7f63cc160420 2024-03-23T11:23:54.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.139+0000 7f63cbe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc0d3540 con 0x7f63cc0b6dc0 2024-03-23T11:23:54.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.139+0000 7f63cbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0ce510 con 0x7f63cc15fcf0 2024-03-23T11:23:54.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.140+0000 7f63cb400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f63cc0b6dc0 0x7f63cc160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:54.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.140+0000 7f63cb400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f63cc0b6dc0 0x7f63cc160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34994/0 (socket says 172.21.15.62:34994) 2024-03-23T11:23:54.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.140+0000 7f63cb400640 1 -- 172.21.15.62:0/1428591064 learned_addr learned my addr 172.21.15.62:0/1428591064 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:54.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.140+0000 7f63caa00640 1 --2- 172.21.15.62:0/1428591064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63cc15fcf0 0x7f63cc0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:54.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.140+0000 7f63cb400640 1 -- 172.21.15.62:0/1428591064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63cc15fcf0 msgr2=0x7f63cc0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:54.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.140+0000 7f63cb400640 1 --2- 172.21.15.62:0/1428591064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63cc15fcf0 0x7f63cc0d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.140+0000 7f63cb400640 1 -- 172.21.15.62:0/1428591064 >> v1:172.21.15.120:6789/0 conn(0x7f63cc160420 legacy=0x7f63cc0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:54.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.140+0000 7f63cb400640 1 -- 172.21.15.62:0/1428591064 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0cdc70 con 0x7f63cc0b6dc0 2024-03-23T11:23:54.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.141+0000 7f63cb400640 1 --2- 172.21.15.62:0/1428591064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63cc0b6dc0 0x7f63cc160880 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009870 tx=0x7f63c402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d287bcd550e13ea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:54.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.142+0000 7f63c9600640 1 -- 172.21.15.62:0/1428591064 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4031740 con 0x7f63cc0b6dc0 2024-03-23T11:23:54.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.142+0000 7f63c9600640 1 -- 172.21.15.62:0/1428591064 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c40318a0 con 0x7f63cc0b6dc0 2024-03-23T11:23:54.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.142+0000 7f63c9600640 1 -- 172.21.15.62:0/1428591064 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4031c30 con 0x7f63cc0b6dc0 2024-03-23T11:23:54.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.142+0000 7f63cbe00640 1 -- 172.21.15.62:0/1428591064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63cc0b6dc0 msgr2=0x7f63cc160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.142+0000 7f63cbe00640 1 --2- 172.21.15.62:0/1428591064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63cc0b6dc0 0x7f63cc160880 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009870 tx=0x7f63c402b7e0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.142+0000 7f63cbe00640 1 -- 172.21.15.62:0/1428591064 shutdown_connections 2024-03-23T11:23:54.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.142+0000 7f63cbe00640 1 --2- 172.21.15.62:0/1428591064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63cc0b6dc0 0x7f63cc160880 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.142+0000 7f63cbe00640 1 --2- 172.21.15.62:0/1428591064 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63cc15fcf0 0x7f63cc0d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.142+0000 7f63cbe00640 1 -- 172.21.15.62:0/1428591064 >> 172.21.15.62:0/1428591064 conn(0x7f63cc089f60 msgr2=0x7f63cc0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:54.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.143+0000 7f63cbe00640 1 -- 172.21.15.62:0/1428591064 shutdown_connections 2024-03-23T11:23:54.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.143+0000 7f63cbe00640 1 -- 172.21.15.62:0/1428591064 wait complete. 2024-03-23T11:23:54.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.144+0000 7f63cbe00640 1 Processor -- start 2024-03-23T11:23:54.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.145+0000 7f63cbe00640 1 -- start start 2024-03-23T11:23:54.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.145+0000 7f63cbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63cc0b6dc0 0x7f63cc1e7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:54.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cbe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63cc15fcf0 0x7f63cc157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:54.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63cc160420 0x7f63cc0c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:54.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cbe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1e7a50 con 0x7f63cc160420 2024-03-23T11:23:54.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cbe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1584c0 con 0x7f63cc15fcf0 2024-03-23T11:23:54.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0c7430 con 0x7f63cc0b6dc0 2024-03-23T11:23:54.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63cc160420 0x7f63cc0c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:54.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63cc160420 0x7f63cc0c84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35008/0 (socket says 172.21.15.62:35008) 2024-03-23T11:23:54.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cb400640 1 -- 172.21.15.62:0/3050798435 learned_addr learned my addr 172.21.15.62:0/3050798435 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:54.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63ca000640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63cc15fcf0 0x7f63cc157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:54.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63caa00640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63cc0b6dc0 0x7f63cc1e7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:54.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cb400640 1 -- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63cc0b6dc0 msgr2=0x7f63cc1e7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cb400640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63cc0b6dc0 0x7f63cc1e7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cb400640 1 -- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63cc15fcf0 msgr2=0x7f63cc157f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cb400640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63cc15fcf0 0x7f63cc157f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63cb400640 1 -- 172.21.15.62:0/3050798435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c40094d0 con 0x7f63cc160420 2024-03-23T11:23:54.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.146+0000 7f63ca000640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63cc15fcf0 0x7f63cc157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:54.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.147+0000 7f63cb400640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63cc160420 0x7f63cc0c84d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f63c402b820 tx=0x7f63c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:54.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.147+0000 7f63c3400640 1 -- 172.21.15.62:0/3050798435 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4033c70 con 0x7f63cc160420 2024-03-23T11:23:54.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.147+0000 7f63c3400640 1 -- 172.21.15.62:0/3050798435 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c4033dd0 con 0x7f63cc160420 2024-03-23T11:23:54.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.147+0000 7f63c3400640 1 -- 172.21.15.62:0/3050798435 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c403c640 con 0x7f63cc160420 2024-03-23T11:23:54.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.147+0000 7f63cbe00640 1 -- 172.21.15.62:0/3050798435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc0c7710 con 0x7f63cc160420 2024-03-23T11:23:54.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.147+0000 7f63cbe00640 1 -- 172.21.15.62:0/3050798435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc0c7cd0 con 0x7f63cc160420 2024-03-23T11:23:54.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.149+0000 7f63cbe00640 1 -- 172.21.15.62:0/3050798435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc066bb0 con 0x7f63cc160420 2024-03-23T11:23:54.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.150+0000 7f63c3400640 1 -- 172.21.15.62:0/3050798435 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f63c403a040 con 0x7f63cc160420 2024-03-23T11:23:54.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.150+0000 7f63c3400640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63ac06a7a0 0x7f63ac06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:54.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.151+0000 7f63c3400640 1 -- 172.21.15.62:0/3050798435 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63c40aa7a0 con 0x7f63cc160420 2024-03-23T11:23:54.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.151+0000 7f63caa00640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63ac06a7a0 0x7f63ac06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:54.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.152+0000 7f63caa00640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63ac06a7a0 0x7f63ac06cc60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009a60 tx=0x7f63bc003990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:54.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.155+0000 7f63c3400640 1 -- 172.21.15.62:0/3050798435 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f63c4075cd0 con 0x7f63cc160420 2024-03-23T11:23:54.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.354+0000 7f63cbe00640 1 -- 172.21.15.62:0/3050798435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63cc0cc040 con 0x7f63cc160420 2024-03-23T11:23:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.355+0000 7f63c3400640 1 -- 172.21.15.62:0/3050798435 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v62) v1 ==== 100+0+11 (secure 0 0 0) 0x7f63c402f0a0 con 0x7f63cc160420 2024-03-23T11:23:54.351 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:23:54.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.357+0000 7f63cbe00640 1 -- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63ac06a7a0 msgr2=0x7f63ac06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.357+0000 7f63cbe00640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63ac06a7a0 0x7f63ac06cc60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009a60 tx=0x7f63bc003990 comp rx=0 tx=0).stop 2024-03-23T11:23:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.357+0000 7f63cbe00640 1 -- 172.21.15.62:0/3050798435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63cc160420 msgr2=0x7f63cc0c84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.357+0000 7f63cbe00640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63cc160420 0x7f63cc0c84d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f63c402b820 tx=0x7f63c40046c0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.357+0000 7f63cbe00640 1 -- 172.21.15.62:0/3050798435 shutdown_connections 2024-03-23T11:23:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.357+0000 7f63cbe00640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f63ac06a7a0 0x7f63ac06cc60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.357+0000 7f63cbe00640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63cc160420 0x7f63cc0c84d0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.357+0000 7f63cbe00640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63cc15fcf0 0x7f63cc157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.357+0000 7f63cbe00640 1 --2- 172.21.15.62:0/3050798435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63cc0b6dc0 0x7f63cc1e7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.357+0000 7f63cbe00640 1 -- 172.21.15.62:0/3050798435 >> 172.21.15.62:0/3050798435 conn(0x7f63cc089f60 msgr2=0x7f63cc06c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.358+0000 7f63cbe00640 1 -- 172.21.15.62:0/3050798435 shutdown_connections 2024-03-23T11:23:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.358+0000 7f63cbe00640 1 -- 172.21.15.62:0/3050798435 wait complete. 2024-03-23T11:23:54.374 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:23:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.602+0000 7f0dce000640 1 Processor -- start 2024-03-23T11:23:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.602+0000 7f0dce000640 1 -- start start 2024-03-23T11:23:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.604+0000 7f0dce000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0dc815cdc0 0x7f0dc80d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.604+0000 7f0dce000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc806bfc0 con 0x7f0dc80cf650 2024-03-23T11:23:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.604+0000 7f0dce000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc80b5260 con 0x7f0dc815c9e0 2024-03-23T11:23:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.604+0000 7f0dce000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc80ce140 con 0x7f0dc815cdc0 2024-03-23T11:23:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.604+0000 7f0dc7e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0dc80cf650 0x7f0dc815d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39378/0 (socket says 172.21.15.62:39378) 2024-03-23T11:23:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.604+0000 7f0dc7e00640 1 -- 172.21.15.62:0/900206453 learned_addr learned my addr 172.21.15.62:0/900206453 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.604+0000 7f0dbea00640 1 --2- 172.21.15.62:0/900206453 >> v2:172.21.15.62:3300/0 conn(0x7f0dc815cdc0 0x7f0dc80d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.605+0000 7f0dbea00640 1 -- 172.21.15.62:0/900206453 >> v1:172.21.15.120:6789/0 conn(0x7f0dc815c9e0 legacy=0x7f0dc81607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.605+0000 7f0dc6a00640 1 -- 172.21.15.62:0/900206453 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277286317 0 0) 0x7f0dc806bfc0 con 0x7f0dc80cf650 2024-03-23T11:23:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.605+0000 7f0dbea00640 1 -- 172.21.15.62:0/900206453 >> v1:172.21.15.62:6789/0 conn(0x7f0dc80cf650 legacy=0x7f0dc815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.605+0000 7f0dbea00640 1 -- 172.21.15.62:0/900206453 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc80b4370 con 0x7f0dc815cdc0 2024-03-23T11:23:54.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.606+0000 7f0dbea00640 1 --2- 172.21.15.62:0/900206453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc815cdc0 0x7f0dc80d1f20 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fb1dd0d2db031c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:54.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.606+0000 7f0dc6a00640 1 -- 172.21.15.62:0/900206453 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0030610 con 0x7f0dc815cdc0 2024-03-23T11:23:54.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.606+0000 7f0dc6a00640 1 -- 172.21.15.62:0/900206453 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db0030770 con 0x7f0dc815cdc0 2024-03-23T11:23:54.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.606+0000 7f0dc6a00640 1 -- 172.21.15.62:0/900206453 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0030b00 con 0x7f0dc815cdc0 2024-03-23T11:23:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.607+0000 7f0dce000640 1 -- 172.21.15.62:0/900206453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc815cdc0 msgr2=0x7f0dc80d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.607+0000 7f0dce000640 1 --2- 172.21.15.62:0/900206453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc815cdc0 0x7f0dc80d1f20 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002b960 comp rx=0 tx=0).stop 2024-03-23T11:23:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.607+0000 7f0dce000640 1 -- 172.21.15.62:0/900206453 shutdown_connections 2024-03-23T11:23:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.607+0000 7f0dce000640 1 --2- 172.21.15.62:0/900206453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc815cdc0 0x7f0dc80d1f20 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.607+0000 7f0dce000640 1 -- 172.21.15.62:0/900206453 >> 172.21.15.62:0/900206453 conn(0x7f0dc8071570 msgr2=0x7f0dc8071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.607+0000 7f0dce000640 1 -- 172.21.15.62:0/900206453 shutdown_connections 2024-03-23T11:23:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.607+0000 7f0dce000640 1 -- 172.21.15.62:0/900206453 wait complete. 2024-03-23T11:23:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.609+0000 7f0dce000640 1 Processor -- start 2024-03-23T11:23:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.609+0000 7f0dce000640 1 -- start start 2024-03-23T11:23:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.610+0000 7f0dce000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc80cf650 0x7f0dc814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.610+0000 7f0dce000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dc815c9e0 0x7f0dc80cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.610+0000 7f0dce000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dc815cdc0 0x7f0dc80cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.610+0000 7f0dce000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc81f65f0 con 0x7f0dc80cf650 2024-03-23T11:23:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.610+0000 7f0dce000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc81f5720 con 0x7f0dc815c9e0 2024-03-23T11:23:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.610+0000 7f0dce000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc81f5970 con 0x7f0dc815cdc0 2024-03-23T11:23:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.610+0000 7f0dc7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc80cf650 0x7f0dc814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.610+0000 7f0dc7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc80cf650 0x7f0dc814fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35024/0 (socket says 172.21.15.62:35024) 2024-03-23T11:23:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.610+0000 7f0dc7400640 1 -- 172.21.15.62:0/3150607709 learned_addr learned my addr 172.21.15.62:0/3150607709 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.611+0000 7f0dc7e00640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dc815cdc0 0x7f0dc80cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.611+0000 7f0dbea00640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dc815c9e0 0x7f0dc80cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.611+0000 7f0dc7400640 1 -- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dc815cdc0 msgr2=0x7f0dc80cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.611+0000 7f0dc7400640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dc815cdc0 0x7f0dc80cc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.611+0000 7f0dc7400640 1 -- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dc815c9e0 msgr2=0x7f0dc80cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.611+0000 7f0dc7400640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dc815c9e0 0x7f0dc80cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.611+0000 7f0dc7400640 1 -- 172.21.15.62:0/3150607709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0009400 con 0x7f0dc80cf650 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.611+0000 7f0dc7400640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc80cf650 0x7f0dc814fff0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0db800c870 tx=0x7f0db800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.612+0000 7f0dbfe00640 1 -- 172.21.15.62:0/3150607709 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db8003b70 con 0x7f0dc80cf650 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.612+0000 7f0dbfe00640 1 -- 172.21.15.62:0/3150607709 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db8003cf0 con 0x7f0dc80cf650 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.612+0000 7f0dbfe00640 1 -- 172.21.15.62:0/3150607709 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db800aa90 con 0x7f0dc80cf650 2024-03-23T11:23:54.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.612+0000 7f0dce000640 1 -- 172.21.15.62:0/3150607709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc81f5c50 con 0x7f0dc80cf650 2024-03-23T11:23:54.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.612+0000 7f0dce000640 1 -- 172.21.15.62:0/3150607709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc81f6210 con 0x7f0dc80cf650 2024-03-23T11:23:54.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.613+0000 7f0dce000640 1 -- 172.21.15.62:0/3150607709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc8071aa0 con 0x7f0dc80cf650 2024-03-23T11:23:54.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.614+0000 7f0dbfe00640 1 -- 172.21.15.62:0/3150607709 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0db800abf0 con 0x7f0dc80cf650 2024-03-23T11:23:54.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.615+0000 7f0dbfe00640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0da406a7a0 0x7f0da406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:54.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.615+0000 7f0dbfe00640 1 -- 172.21.15.62:0/3150607709 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0db8088c50 con 0x7f0dc80cf650 2024-03-23T11:23:54.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.615+0000 7f0dbea00640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0da406a7a0 0x7f0da406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:54.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.617+0000 7f0dbea00640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0da406a7a0 0x7f0da406cc60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0db0036000 tx=0x7f0db0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:54.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.620+0000 7f0dbfe00640 1 -- 172.21.15.62:0/3150607709 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0db8054060 con 0x7f0dc80cf650 2024-03-23T11:23:54.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.859+0000 7f0dce000640 1 -- 172.21.15.62:0/3150607709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dc8066c90 con 0x7f0dc80cf650 2024-03-23T11:23:54.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.860+0000 7f0dbfe00640 1 -- 172.21.15.62:0/3150607709 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f0db8003e60 con 0x7f0dc80cf650 2024-03-23T11:23:54.856 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:54.856 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3287045135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3287045135},{"type":"v1","addr":"172.21.15.120:6835","nonce":3287045135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6066,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/2859430143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2859430143},{"type":"v1","addr":"172.21.15.62:6834","nonce":2859430143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2005768060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2005768060},{"type":"v1","addr":"172.21.15.62:6835","nonce":2005768060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:22:31.725865+0000","modified":"2024-03-23T11:23:24.399625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6081},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6081":{"gid":6081,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":16,"addr":"172.21.15.120:6837/4158450746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":4158450746},{"type":"v1","addr":"172.21.15.120:6837","nonce":4158450746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T11:23:54.856 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 -- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0da406a7a0 msgr2=0x7f0da406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0da406a7a0 0x7f0da406cc60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0db0036000 tx=0x7f0db0008030 comp rx=0 tx=0).stop 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 -- 172.21.15.62:0/3150607709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc80cf650 msgr2=0x7f0dc814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc80cf650 0x7f0dc814fff0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0db800c870 tx=0x7f0db800cd30 comp rx=0 tx=0).stop 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 -- 172.21.15.62:0/3150607709 shutdown_connections 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0da406a7a0 0x7f0da406cc60 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dc815cdc0 0x7f0dc80cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dc815c9e0 0x7f0dc80cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 --2- 172.21.15.62:0/3150607709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc80cf650 0x7f0dc814fff0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 -- 172.21.15.62:0/3150607709 >> 172.21.15.62:0/3150607709 conn(0x7f0dc8071570 msgr2=0x7f0dc8160f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 -- 172.21.15.62:0/3150607709 shutdown_connections 2024-03-23T11:23:54.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:54.863+0000 7f0dce000640 1 -- 172.21.15.62:0/3150607709 wait complete. 2024-03-23T11:23:54.879 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:23:55.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.108+0000 7f8f15800640 1 Processor -- start 2024-03-23T11:23:55.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.108+0000 7f8f15800640 1 -- start start 2024-03-23T11:23:55.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.110+0000 7f8f15800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f1006cc60 0x7f8f100cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.110+0000 7f8f15800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8f100ce420 0x7f8f100b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.110+0000 7f8f15800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f100ce8b0 con 0x7f8f10159490 2024-03-23T11:23:55.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.110+0000 7f8f15800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f100ced90 con 0x7f8f100ce420 2024-03-23T11:23:55.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.110+0000 7f8f15800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1006e3e0 con 0x7f8f1006cc60 2024-03-23T11:23:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.111+0000 7f8f07400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f1006cc60 0x7f8f100cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.111+0000 7f8f14e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8f100ce420 0x7f8f100b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.111+0000 7f8f0fe00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f8f10159490 0x7f8f1015da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:40074/0 (socket says 172.21.15.62:40074) 2024-03-23T11:23:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.111+0000 7f8f0fe00640 1 -- 172.21.15.62:0/1703213154 learned_addr learned my addr 172.21.15.62:0/1703213154 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:55.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.112+0000 7f8f07400640 1 -- 172.21.15.62:0/1703213154 >> v2:172.21.15.120:3300/0 conn(0x7f8f100ce420 msgr2=0x7f8f100b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.112+0000 7f8f07400640 1 --2- 172.21.15.62:0/1703213154 >> v2:172.21.15.120:3300/0 conn(0x7f8f100ce420 0x7f8f100b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.112+0000 7f8f07400640 1 -- 172.21.15.62:0/1703213154 >> v1:172.21.15.120:6789/0 conn(0x7f8f10159490 legacy=0x7f8f1015da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.112+0000 7f8f07400640 1 -- 172.21.15.62:0/1703213154 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1006c1c0 con 0x7f8f1006cc60 2024-03-23T11:23:55.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.113+0000 7f8f07400640 1 --2- 172.21.15.62:0/1703213154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f1006cc60 0x7f8f100cdee0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80097a0 tx=0x7f8ef802da50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5007ebd92ffb906a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:55.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.114+0000 7f8f0f400640 1 -- 172.21.15.62:0/1703213154 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8031870 con 0x7f8f1006cc60 2024-03-23T11:23:55.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.114+0000 7f8f0f400640 1 -- 172.21.15.62:0/1703213154 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef80319d0 con 0x7f8f1006cc60 2024-03-23T11:23:55.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.114+0000 7f8f0f400640 1 -- 172.21.15.62:0/1703213154 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8031d60 con 0x7f8f1006cc60 2024-03-23T11:23:55.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.115+0000 7f8f15800640 1 -- 172.21.15.62:0/1703213154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f1006cc60 msgr2=0x7f8f100cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.115+0000 7f8f15800640 1 --2- 172.21.15.62:0/1703213154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f1006cc60 0x7f8f100cdee0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80097a0 tx=0x7f8ef802da50 comp rx=0 tx=0).stop 2024-03-23T11:23:55.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.115+0000 7f8f15800640 1 -- 172.21.15.62:0/1703213154 shutdown_connections 2024-03-23T11:23:55.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.115+0000 7f8f15800640 1 --2- 172.21.15.62:0/1703213154 >> v2:172.21.15.120:3300/0 conn(0x7f8f100ce420 0x7f8f100b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.115+0000 7f8f15800640 1 --2- 172.21.15.62:0/1703213154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f1006cc60 0x7f8f100cdee0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.115+0000 7f8f15800640 1 -- 172.21.15.62:0/1703213154 >> 172.21.15.62:0/1703213154 conn(0x7f8f10066050 msgr2=0x7f8f10069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:55.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.115+0000 7f8f15800640 1 -- 172.21.15.62:0/1703213154 shutdown_connections 2024-03-23T11:23:55.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.115+0000 7f8f15800640 1 -- 172.21.15.62:0/1703213154 wait complete. 2024-03-23T11:23:55.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.117+0000 7f8f15800640 1 Processor -- start 2024-03-23T11:23:55.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.117+0000 7f8f15800640 1 -- start start 2024-03-23T11:23:55.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f15800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f1006cc60 0x7f8f1015c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f15800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f100ce420 0x7f8f100b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f15800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f10159490 0x7f8f1015a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f15800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1015a8b0 con 0x7f8f1006cc60 2024-03-23T11:23:55.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f15800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f100b1120 con 0x7f8f10159490 2024-03-23T11:23:55.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f15800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f100b1400 con 0x7f8f100ce420 2024-03-23T11:23:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f0fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f1006cc60 0x7f8f1015c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f0fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f1006cc60 0x7f8f1015c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35034/0 (socket says 172.21.15.62:35034) 2024-03-23T11:23:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f0fe00640 1 -- 172.21.15.62:0/1254225186 learned_addr learned my addr 172.21.15.62:0/1254225186 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f07400640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f100ce420 0x7f8f100b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f14e00640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f10159490 0x7f8f1015a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f0fe00640 1 -- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f100ce420 msgr2=0x7f8f100b1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f0fe00640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f100ce420 0x7f8f100b1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f0fe00640 1 -- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f10159490 msgr2=0x7f8f1015a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.118+0000 7f8f0fe00640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f10159490 0x7f8f1015a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.119+0000 7f8f0fe00640 1 -- 172.21.15.62:0/1254225186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef8009400 con 0x7f8f1006cc60 2024-03-23T11:23:55.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.119+0000 7f8f0fe00640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f1006cc60 0x7f8f1015c590 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000c4a0 tx=0x7f8f0000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:55.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.119+0000 7f8f0cc00640 1 -- 172.21.15.62:0/1254225186 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00005ce0 con 0x7f8f1006cc60 2024-03-23T11:23:55.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.119+0000 7f8f0cc00640 1 -- 172.21.15.62:0/1254225186 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f00005e40 con 0x7f8f1006cc60 2024-03-23T11:23:55.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.119+0000 7f8f0cc00640 1 -- 172.21.15.62:0/1254225186 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00007270 con 0x7f8f1006cc60 2024-03-23T11:23:55.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.119+0000 7f8f15800640 1 -- 172.21.15.62:0/1254225186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f100b16e0 con 0x7f8f1006cc60 2024-03-23T11:23:55.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.119+0000 7f8f15800640 1 -- 172.21.15.62:0/1254225186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1015a5a0 con 0x7f8f1006cc60 2024-03-23T11:23:55.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.121+0000 7f8f15800640 1 -- 172.21.15.62:0/1254225186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f100b1920 con 0x7f8f1006cc60 2024-03-23T11:23:55.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.122+0000 7f8f0cc00640 1 -- 172.21.15.62:0/1254225186 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8f00007490 con 0x7f8f1006cc60 2024-03-23T11:23:55.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.122+0000 7f8f0cc00640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8eec06a7a0 0x7f8eec06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.123+0000 7f8f0cc00640 1 -- 172.21.15.62:0/1254225186 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f000881e0 con 0x7f8f1006cc60 2024-03-23T11:23:55.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.123+0000 7f8f07400640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8eec06a7a0 0x7f8eec06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.124+0000 7f8f07400640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8eec06a7a0 0x7f8eec06cc60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8038000 tx=0x7f8ef803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:55.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.127+0000 7f8f0cc00640 1 -- 172.21.15.62:0/1254225186 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8f100b1920 con 0x7f8f1006cc60 2024-03-23T11:23:55.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.330+0000 7f8f15800640 1 -- 172.21.15.62:0/1254225186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f100b18d0 con 0x7f8f1006cc60 2024-03-23T11:23:55.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.334+0000 7f8f0cc00640 1 -- 172.21.15.62:0/1254225186 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16097 (secure 0 0 0) 0x7f8f000536b0 con 0x7f8f1006cc60 2024-03-23T11:23:55.329 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:55.330 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":62,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:22:32.731093+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:22:29.767222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-23T11:22:30.738596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.336+0000 7f8f15800640 1 -- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8eec06a7a0 msgr2=0x7f8eec06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.336+0000 7f8f15800640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8eec06a7a0 0x7f8eec06cc60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8038000 tx=0x7f8ef803b040 comp rx=0 tx=0).stop 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.336+0000 7f8f15800640 1 -- 172.21.15.62:0/1254225186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f1006cc60 msgr2=0x7f8f1015c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.336+0000 7f8f15800640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f1006cc60 0x7f8f1015c590 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000c4a0 tx=0x7f8f0000c960 comp rx=0 tx=0).stop 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.336+0000 7f8f15800640 1 -- 172.21.15.62:0/1254225186 shutdown_connections 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.336+0000 7f8f15800640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8eec06a7a0 0x7f8eec06cc60 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.336+0000 7f8f15800640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f10159490 0x7f8f1015a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.336+0000 7f8f15800640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f100ce420 0x7f8f100b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.336+0000 7f8f15800640 1 --2- 172.21.15.62:0/1254225186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f1006cc60 0x7f8f1015c590 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.336+0000 7f8f15800640 1 -- 172.21.15.62:0/1254225186 >> 172.21.15.62:0/1254225186 conn(0x7f8f10066050 msgr2=0x7f8f101e3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.337+0000 7f8f15800640 1 -- 172.21.15.62:0/1254225186 shutdown_connections 2024-03-23T11:23:55.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.337+0000 7f8f15800640 1 -- 172.21.15.62:0/1254225186 wait complete. 2024-03-23T11:23:55.351 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T11:23:55.351 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:23:55.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.580+0000 7fe328600640 1 Processor -- start 2024-03-23T11:23:55.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.580+0000 7fe328600640 1 -- start start 2024-03-23T11:23:55.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.582+0000 7fe328600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe3201613a0 0x7fe32006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.582+0000 7fe328600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe32006d910 0x7fe32015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.582+0000 7fe328600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32006cf40 con 0x7fe3201571e0 2024-03-23T11:23:55.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.582+0000 7fe328600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3201575c0 con 0x7fe3201613a0 2024-03-23T11:23:55.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.582+0000 7fe328600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32006c3a0 con 0x7fe32006d910 2024-03-23T11:23:55.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.582+0000 7fe325c00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe3201571e0 0x7fe3200ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39384/0 (socket says 172.21.15.62:39384) 2024-03-23T11:23:55.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.582+0000 7fe325c00640 1 -- 172.21.15.62:0/178108275 learned_addr learned my addr 172.21.15.62:0/178108275 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:55.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.582+0000 7fe325200640 1 --2- 172.21.15.62:0/178108275 >> v2:172.21.15.62:3300/0 conn(0x7fe3201613a0 0x7fe32006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.582+0000 7fe326600640 1 --2- 172.21.15.62:0/178108275 >> v2:172.21.15.120:3300/0 conn(0x7fe32006d910 0x7fe32015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.582+0000 7fe31fe00640 1 -- 172.21.15.62:0/178108275 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211837988 0 0) 0x7fe32006cf40 con 0x7fe3201571e0 2024-03-23T11:23:55.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.583+0000 7fe31fe00640 1 -- 172.21.15.62:0/178108275 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe308003520 con 0x7fe3201571e0 2024-03-23T11:23:55.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.583+0000 7fe325200640 1 -- 172.21.15.62:0/178108275 >> v2:172.21.15.120:3300/0 conn(0x7fe32006d910 msgr2=0x7fe32015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:55.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.583+0000 7fe325200640 1 --2- 172.21.15.62:0/178108275 >> v2:172.21.15.120:3300/0 conn(0x7fe32006d910 0x7fe32015e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.583+0000 7fe325200640 1 -- 172.21.15.62:0/178108275 >> v1:172.21.15.62:6789/0 conn(0x7fe3201571e0 legacy=0x7fe3200ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.583+0000 7fe325200640 1 -- 172.21.15.62:0/178108275 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3200ae040 con 0x7fe3201613a0 2024-03-23T11:23:55.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.584+0000 7fe325200640 1 --2- 172.21.15.62:0/178108275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3201613a0 0x7fe32006d3d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe3200b1280 tx=0x7fe30c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e6087d54da19b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:55.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.584+0000 7fe31fe00640 1 -- 172.21.15.62:0/178108275 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c02e070 con 0x7fe3201613a0 2024-03-23T11:23:55.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.584+0000 7fe31fe00640 1 -- 172.21.15.62:0/178108275 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c0315a0 con 0x7fe3201613a0 2024-03-23T11:23:55.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.585+0000 7fe31fe00640 1 -- 172.21.15.62:0/178108275 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c031930 con 0x7fe3201613a0 2024-03-23T11:23:55.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.585+0000 7fe328600640 1 -- 172.21.15.62:0/178108275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3201613a0 msgr2=0x7fe32006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.585+0000 7fe328600640 1 --2- 172.21.15.62:0/178108275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3201613a0 0x7fe32006d3d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe3200b1280 tx=0x7fe30c02b840 comp rx=0 tx=0).stop 2024-03-23T11:23:55.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.585+0000 7fe328600640 1 -- 172.21.15.62:0/178108275 shutdown_connections 2024-03-23T11:23:55.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.585+0000 7fe328600640 1 --2- 172.21.15.62:0/178108275 >> v2:172.21.15.120:3300/0 conn(0x7fe32006d910 0x7fe32015e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.585+0000 7fe328600640 1 --2- 172.21.15.62:0/178108275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3201613a0 0x7fe32006d3d0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.585+0000 7fe328600640 1 -- 172.21.15.62:0/178108275 >> 172.21.15.62:0/178108275 conn(0x7fe3200b02e0 msgr2=0x7fe3200b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:55.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.585+0000 7fe328600640 1 -- 172.21.15.62:0/178108275 shutdown_connections 2024-03-23T11:23:55.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.585+0000 7fe328600640 1 -- 172.21.15.62:0/178108275 wait complete. 2024-03-23T11:23:55.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.587+0000 7fe328600640 1 Processor -- start 2024-03-23T11:23:55.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.587+0000 7fe328600640 1 -- start start 2024-03-23T11:23:55.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.588+0000 7fe328600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe32006d910 0x7fe3200cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.588+0000 7fe328600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3201571e0 0x7fe320142a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.588+0000 7fe328600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3201613a0 0x7fe320142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.588+0000 7fe328600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3201443e0 con 0x7fe3201613a0 2024-03-23T11:23:55.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.588+0000 7fe328600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3201434b0 con 0x7fe3201571e0 2024-03-23T11:23:55.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.588+0000 7fe328600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe320143760 con 0x7fe32006d910 2024-03-23T11:23:55.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe326600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3201613a0 0x7fe320142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe326600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3201613a0 0x7fe320142f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35052/0 (socket says 172.21.15.62:35052) 2024-03-23T11:23:55.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe326600640 1 -- 172.21.15.62:0/4259200206 learned_addr learned my addr 172.21.15.62:0/4259200206 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:55.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe325c00640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe32006d910 0x7fe3200cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe325200640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3201571e0 0x7fe320142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe326600640 1 -- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe32006d910 msgr2=0x7fe3200cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe326600640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe32006d910 0x7fe3200cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe326600640 1 -- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3201571e0 msgr2=0x7fe320142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe326600640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3201571e0 0x7fe320142a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe326600640 1 -- 172.21.15.62:0/4259200206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c009400 con 0x7fe3201613a0 2024-03-23T11:23:55.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe325c00640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe32006d910 0x7fe3200cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:55.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.589+0000 7fe326600640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3201613a0 0x7fe320142f70 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe318007e40 tx=0x7fe31800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:55.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.590+0000 7fe31d600640 1 -- 172.21.15.62:0/4259200206 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318014620 con 0x7fe3201613a0 2024-03-23T11:23:55.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.590+0000 7fe31d600640 1 -- 172.21.15.62:0/4259200206 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31800ee90 con 0x7fe3201613a0 2024-03-23T11:23:55.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.590+0000 7fe328600640 1 -- 172.21.15.62:0/4259200206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe320143a40 con 0x7fe3201613a0 2024-03-23T11:23:55.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.590+0000 7fe31d600640 1 -- 172.21.15.62:0/4259200206 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3180149e0 con 0x7fe3201613a0 2024-03-23T11:23:55.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.590+0000 7fe328600640 1 -- 172.21.15.62:0/4259200206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe320144000 con 0x7fe3201613a0 2024-03-23T11:23:55.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.591+0000 7fe328600640 1 -- 172.21.15.62:0/4259200206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe320144520 con 0x7fe3201613a0 2024-03-23T11:23:55.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.592+0000 7fe31d600640 1 -- 172.21.15.62:0/4259200206 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe318014b40 con 0x7fe3201613a0 2024-03-23T11:23:55.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.593+0000 7fe31d600640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe30006a7d0 0x7fe30006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:55.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.593+0000 7fe31d600640 1 -- 172.21.15.62:0/4259200206 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe318089720 con 0x7fe3201613a0 2024-03-23T11:23:55.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.594+0000 7fe325c00640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe30006a7d0 0x7fe30006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:55.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.595+0000 7fe325c00640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe30006a7d0 0x7fe30006cc90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe3140011e0 tx=0x7fe314009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:55.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.598+0000 7fe31d600640 1 -- 172.21.15.62:0/4259200206 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe320144520 con 0x7fe3201613a0 2024-03-23T11:23:55.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.840+0000 7fe328600640 1 -- 172.21.15.62:0/4259200206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe320144520 con 0x7fe3201613a0 2024-03-23T11:23:55.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.841+0000 7fe31d600640 1 -- 172.21.15.62:0/4259200206 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fe320144520 con 0x7fe3201613a0 2024-03-23T11:23:55.837 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:55.837 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3287045135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3287045135},{"type":"v1","addr":"172.21.15.120:6835","nonce":3287045135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6066,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/2859430143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2859430143},{"type":"v1","addr":"172.21.15.62:6834","nonce":2859430143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2005768060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2005768060},{"type":"v1","addr":"172.21.15.62:6835","nonce":2005768060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:22:31.725865+0000","modified":"2024-03-23T11:23:24.399625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6081},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6081":{"gid":6081,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":16,"addr":"172.21.15.120:6837/4158450746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":4158450746},{"type":"v1","addr":"172.21.15.120:6837","nonce":4158450746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T11:23:55.837 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-03-23T11:23:55.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 -- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe30006a7d0 msgr2=0x7fe30006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe30006a7d0 0x7fe30006cc90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe3140011e0 tx=0x7fe314009040 comp rx=0 tx=0).stop 2024-03-23T11:23:55.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 -- 172.21.15.62:0/4259200206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3201613a0 msgr2=0x7fe320142f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:55.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3201613a0 0x7fe320142f70 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe318007e40 tx=0x7fe31800e4b0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 -- 172.21.15.62:0/4259200206 shutdown_connections 2024-03-23T11:23:55.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe30006a7d0 0x7fe30006cc90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3201613a0 0x7fe320142f70 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3201571e0 0x7fe320142a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 --2- 172.21.15.62:0/4259200206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe32006d910 0x7fe3200cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:55.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 -- 172.21.15.62:0/4259200206 >> 172.21.15.62:0/4259200206 conn(0x7fe3200b02e0 msgr2=0x7fe3200af240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:55.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 -- 172.21.15.62:0/4259200206 shutdown_connections 2024-03-23T11:23:55.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:55.844+0000 7fe328600640 1 -- 172.21.15.62:0/4259200206 wait complete. 2024-03-23T11:23:55.860 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:23:56.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.089+0000 7fc638e00640 1 Processor -- start 2024-03-23T11:23:56.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.089+0000 7fc638e00640 1 -- start start 2024-03-23T11:23:56.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.090+0000 7fc638e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc63415cbb0 0x7fc6340996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.090+0000 7fc638e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc6340990e0 0x7fc6340b1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.090+0000 7fc638e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6340cab10 con 0x7fc63406bfe0 2024-03-23T11:23:56.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.090+0000 7fc638e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6340cb3b0 con 0x7fc6340990e0 2024-03-23T11:23:56.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.090+0000 7fc638e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6340afe20 con 0x7fc63415cbb0 2024-03-23T11:23:56.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.091+0000 7fc633e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc6340990e0 0x7fc6340b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.091+0000 7fc632a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc63406bfe0 0x7fc6340989d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39392/0 (socket says 172.21.15.62:39392) 2024-03-23T11:23:56.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.091+0000 7fc632a00640 1 -- 172.21.15.62:0/4279630137 learned_addr learned my addr 172.21.15.62:0/4279630137 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:56.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.091+0000 7fc633400640 1 --2- 172.21.15.62:0/4279630137 >> v2:172.21.15.120:3300/0 conn(0x7fc63415cbb0 0x7fc6340996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.091+0000 7fc633e00640 1 -- 172.21.15.62:0/4279630137 >> v2:172.21.15.120:3300/0 conn(0x7fc63415cbb0 msgr2=0x7fc6340996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:56.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.091+0000 7fc633e00640 1 --2- 172.21.15.62:0/4279630137 >> v2:172.21.15.120:3300/0 conn(0x7fc63415cbb0 0x7fc6340996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.092+0000 7fc633e00640 1 -- 172.21.15.62:0/4279630137 >> v1:172.21.15.62:6789/0 conn(0x7fc63406bfe0 legacy=0x7fc6340989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.092+0000 7fc633e00640 1 -- 172.21.15.62:0/4279630137 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63406c750 con 0x7fc6340990e0 2024-03-23T11:23:56.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.092+0000 7fc632000640 1 -- 172.21.15.62:0/4279630137 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404939996 0 0) 0x7fc6340cab10 con 0x7fc63406bfe0 2024-03-23T11:23:56.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.093+0000 7fc633e00640 1 --2- 172.21.15.62:0/4279630137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6340990e0 0x7fc6340b1010 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc63406d410 tx=0x7fc62c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239b5605c7ec55e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:56.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.093+0000 7fc632000640 1 -- 172.21.15.62:0/4279630137 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c02cc70 con 0x7fc6340990e0 2024-03-23T11:23:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.093+0000 7fc632000640 1 -- 172.21.15.62:0/4279630137 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c02cdd0 con 0x7fc6340990e0 2024-03-23T11:23:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.093+0000 7fc632000640 1 -- 172.21.15.62:0/4279630137 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c034660 con 0x7fc6340990e0 2024-03-23T11:23:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.094+0000 7fc638e00640 1 -- 172.21.15.62:0/4279630137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6340990e0 msgr2=0x7fc6340b1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.094+0000 7fc638e00640 1 --2- 172.21.15.62:0/4279630137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6340990e0 0x7fc6340b1010 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc63406d410 tx=0x7fc62c02bec0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.094+0000 7fc638e00640 1 -- 172.21.15.62:0/4279630137 shutdown_connections 2024-03-23T11:23:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.094+0000 7fc638e00640 1 --2- 172.21.15.62:0/4279630137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6340990e0 0x7fc6340b1010 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.094+0000 7fc638e00640 1 --2- 172.21.15.62:0/4279630137 >> v2:172.21.15.120:3300/0 conn(0x7fc63415cbb0 0x7fc6340996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.094+0000 7fc638e00640 1 -- 172.21.15.62:0/4279630137 >> 172.21.15.62:0/4279630137 conn(0x7fc6340b1fb0 msgr2=0x7fc6340b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.094+0000 7fc638e00640 1 -- 172.21.15.62:0/4279630137 shutdown_connections 2024-03-23T11:23:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.094+0000 7fc638e00640 1 -- 172.21.15.62:0/4279630137 wait complete. 2024-03-23T11:23:56.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.096+0000 7fc638e00640 1 Processor -- start 2024-03-23T11:23:56.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.096+0000 7fc638e00640 1 -- start start 2024-03-23T11:23:56.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.097+0000 7fc638e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc63406bfe0 0x7fc63415eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.097+0000 7fc638e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6340990e0 0x7fc6340b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.097+0000 7fc638e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc63415cbb0 0x7fc6341590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.097+0000 7fc638e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6340b3ac0 con 0x7fc63415cbb0 2024-03-23T11:23:56.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.097+0000 7fc638e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6340b3da0 con 0x7fc63406bfe0 2024-03-23T11:23:56.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.097+0000 7fc638e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6340b4080 con 0x7fc6340990e0 2024-03-23T11:23:56.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.097+0000 7fc633e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc63415cbb0 0x7fc6341590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.097+0000 7fc633e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc63415cbb0 0x7fc6341590e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35076/0 (socket says 172.21.15.62:35076) 2024-03-23T11:23:56.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.097+0000 7fc633e00640 1 -- 172.21.15.62:0/3218254678 learned_addr learned my addr 172.21.15.62:0/3218254678 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:56.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.097+0000 7fc633400640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc63406bfe0 0x7fc63415eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.097+0000 7fc632a00640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6340990e0 0x7fc6340b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.098+0000 7fc633e00640 1 -- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6340990e0 msgr2=0x7fc6340b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.098+0000 7fc633e00640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6340990e0 0x7fc6340b30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.098+0000 7fc633e00640 1 -- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc63406bfe0 msgr2=0x7fc63415eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.098+0000 7fc633e00640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc63406bfe0 0x7fc63415eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.098+0000 7fc633e00640 1 -- 172.21.15.62:0/3218254678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62c009a70 con 0x7fc63415cbb0 2024-03-23T11:23:56.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.098+0000 7fc633e00640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc63415cbb0 0x7fc6341590e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc62c02c4d0 tx=0x7fc62c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:56.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.098+0000 7fc62b400640 1 -- 172.21.15.62:0/3218254678 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c03e610 con 0x7fc63415cbb0 2024-03-23T11:23:56.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.098+0000 7fc62b400640 1 -- 172.21.15.62:0/3218254678 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c03e790 con 0x7fc63415cbb0 2024-03-23T11:23:56.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.098+0000 7fc62b400640 1 -- 172.21.15.62:0/3218254678 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c03eb40 con 0x7fc63415cbb0 2024-03-23T11:23:56.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.098+0000 7fc638e00640 1 -- 172.21.15.62:0/3218254678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6340b4360 con 0x7fc63415cbb0 2024-03-23T11:23:56.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.099+0000 7fc638e00640 1 -- 172.21.15.62:0/3218254678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc634159ee0 con 0x7fc63415cbb0 2024-03-23T11:23:56.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.100+0000 7fc638e00640 1 -- 172.21.15.62:0/3218254678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6340b4880 con 0x7fc63415cbb0 2024-03-23T11:23:56.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.100+0000 7fc62b400640 1 -- 172.21.15.62:0/3218254678 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc62c03b050 con 0x7fc63415cbb0 2024-03-23T11:23:56.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.102+0000 7fc62b400640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc61406a7a0 0x7fc61406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.102+0000 7fc62b400640 1 -- 172.21.15.62:0/3218254678 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc62c0ab170 con 0x7fc63415cbb0 2024-03-23T11:23:56.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.102+0000 7fc633400640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc61406a7a0 0x7fc61406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.104+0000 7fc633400640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc61406a7a0 0x7fc61406cc60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc624006fd0 tx=0x7fc624008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:56.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.107+0000 7fc62b400640 1 -- 172.21.15.62:0/3218254678 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc6340b4880 con 0x7fc63415cbb0 2024-03-23T11:23:56.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.354+0000 7fc638e00640 1 -- 172.21.15.62:0/3218254678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6340b4880 con 0x7fc63415cbb0 2024-03-23T11:23:56.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.355+0000 7fc62b400640 1 -- 172.21.15.62:0/3218254678 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fc6340b4880 con 0x7fc63415cbb0 2024-03-23T11:23:56.351 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:56.351 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3287045135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3287045135},{"type":"v1","addr":"172.21.15.120:6835","nonce":3287045135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6066,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/2859430143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2859430143},{"type":"v1","addr":"172.21.15.62:6834","nonce":2859430143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2005768060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2005768060},{"type":"v1","addr":"172.21.15.62:6835","nonce":2005768060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:22:31.725865+0000","modified":"2024-03-23T11:23:24.399625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6081},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6081":{"gid":6081,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":16,"addr":"172.21.15.120:6837/4158450746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":4158450746},{"type":"v1","addr":"172.21.15.120:6837","nonce":4158450746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T11:23:56.351 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-03-23T11:23:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 -- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc61406a7a0 msgr2=0x7fc61406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc61406a7a0 0x7fc61406cc60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc624006fd0 tx=0x7fc624008040 comp rx=0 tx=0).stop 2024-03-23T11:23:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 -- 172.21.15.62:0/3218254678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc63415cbb0 msgr2=0x7fc6341590e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc63415cbb0 0x7fc6341590e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc62c02c4d0 tx=0x7fc62c004a40 comp rx=0 tx=0).stop 2024-03-23T11:23:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 -- 172.21.15.62:0/3218254678 shutdown_connections 2024-03-23T11:23:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc61406a7a0 0x7fc61406cc60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc63415cbb0 0x7fc6341590e0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6340990e0 0x7fc6340b30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 --2- 172.21.15.62:0/3218254678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc63406bfe0 0x7fc63415eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 -- 172.21.15.62:0/3218254678 >> 172.21.15.62:0/3218254678 conn(0x7fc6340b1fb0 msgr2=0x7fc6340af1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 -- 172.21.15.62:0/3218254678 shutdown_connections 2024-03-23T11:23:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.358+0000 7fc638e00640 1 -- 172.21.15.62:0/3218254678 wait complete. 2024-03-23T11:23:56.372 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:23:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.595+0000 7fe703000640 1 Processor -- start 2024-03-23T11:23:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.596+0000 7fe703000640 1 -- start start 2024-03-23T11:23:56.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.597+0000 7fe703000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc06c300 0x7fe6fc0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.597+0000 7fe703000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe6fc06c6e0 0x7fe6fc06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.597+0000 7fe703000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6fc0cbc60 con 0x7fe6fc0cafa0 2024-03-23T11:23:56.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.597+0000 7fe703000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6fc0cbf40 con 0x7fe6fc06c6e0 2024-03-23T11:23:56.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.597+0000 7fe703000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0cc220 con 0x7fe6fc06c300 2024-03-23T11:23:56.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.598+0000 7fe6fbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc06c300 0x7fe6fc0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.598+0000 7fe6fbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc06c300 0x7fe6fc0ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:46276/0 (socket says 172.21.15.62:46276) 2024-03-23T11:23:56.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.598+0000 7fe6fbe00640 1 -- 172.21.15.62:0/2461120454 learned_addr learned my addr 172.21.15.62:0/2461120454 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:56.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.598+0000 7fe701000640 1 --2- 172.21.15.62:0/2461120454 >> v2:172.21.15.120:3300/0 conn(0x7fe6fc06c6e0 0x7fe6fc06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.599+0000 7fe6fbe00640 1 -- 172.21.15.62:0/2461120454 >> v2:172.21.15.120:3300/0 conn(0x7fe6fc06c6e0 msgr2=0x7fe6fc06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:56.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.599+0000 7fe6fbe00640 1 --2- 172.21.15.62:0/2461120454 >> v2:172.21.15.120:3300/0 conn(0x7fe6fc06c6e0 0x7fe6fc06cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.599+0000 7fe6fbe00640 1 -- 172.21.15.62:0/2461120454 >> v1:172.21.15.120:6789/0 conn(0x7fe6fc0cafa0 legacy=0x7fe6fc0cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.599+0000 7fe6fbe00640 1 -- 172.21.15.62:0/2461120454 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc0cc500 con 0x7fe6fc06c300 2024-03-23T11:23:56.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.600+0000 7fe6fbe00640 1 --2- 172.21.15.62:0/2461120454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc06c300 0x7fe6fc0ccd00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec009870 tx=0x7fe6ec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb057266592158c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:56.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.601+0000 7fe6fb400640 1 -- 172.21.15.62:0/2461120454 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec030840 con 0x7fe6fc06c300 2024-03-23T11:23:56.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.601+0000 7fe6fb400640 1 -- 172.21.15.62:0/2461120454 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec0309a0 con 0x7fe6fc06c300 2024-03-23T11:23:56.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.601+0000 7fe6fb400640 1 -- 172.21.15.62:0/2461120454 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec030d30 con 0x7fe6fc06c300 2024-03-23T11:23:56.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.601+0000 7fe703000640 1 -- 172.21.15.62:0/2461120454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc06c300 msgr2=0x7fe6fc0ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.601+0000 7fe703000640 1 --2- 172.21.15.62:0/2461120454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc06c300 0x7fe6fc0ccd00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec009870 tx=0x7fe6ec02b980 comp rx=0 tx=0).stop 2024-03-23T11:23:56.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.601+0000 7fe703000640 1 -- 172.21.15.62:0/2461120454 shutdown_connections 2024-03-23T11:23:56.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.601+0000 7fe703000640 1 --2- 172.21.15.62:0/2461120454 >> v2:172.21.15.120:3300/0 conn(0x7fe6fc06c6e0 0x7fe6fc06cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.601+0000 7fe703000640 1 --2- 172.21.15.62:0/2461120454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc06c300 0x7fe6fc0ccd00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.601+0000 7fe703000640 1 -- 172.21.15.62:0/2461120454 >> 172.21.15.62:0/2461120454 conn(0x7fe6fc0b02e0 msgr2=0x7fe6fc0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:56.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.601+0000 7fe703000640 1 -- 172.21.15.62:0/2461120454 shutdown_connections 2024-03-23T11:23:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.602+0000 7fe703000640 1 -- 172.21.15.62:0/2461120454 wait complete. 2024-03-23T11:23:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.603+0000 7fe703000640 1 Processor -- start 2024-03-23T11:23:56.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.603+0000 7fe703000640 1 -- start start 2024-03-23T11:23:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.604+0000 7fe703000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6fc06c300 0x7fe6fc13cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.604+0000 7fe703000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6fc06c6e0 0x7fe6fc13f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.604+0000 7fe703000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc0cafa0 0x7fe6fc158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.604+0000 7fe703000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc15af10 con 0x7fe6fc06c300 2024-03-23T11:23:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.604+0000 7fe703000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc15b7b0 con 0x7fe6fc06c6e0 2024-03-23T11:23:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.604+0000 7fe703000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc140310 con 0x7fe6fc0cafa0 2024-03-23T11:23:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.605+0000 7fe6fbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6fc06c300 0x7fe6fc13cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.605+0000 7fe6fbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6fc06c300 0x7fe6fc13cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35084/0 (socket says 172.21.15.62:35084) 2024-03-23T11:23:56.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.605+0000 7fe6fbe00640 1 -- 172.21.15.62:0/2301725630 learned_addr learned my addr 172.21.15.62:0/2301725630 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:56.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.605+0000 7fe701000640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc0cafa0 0x7fe6fc158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.605+0000 7fe6f3400640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6fc06c6e0 0x7fe6fc13f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.605+0000 7fe6fbe00640 1 -- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc0cafa0 msgr2=0x7fe6fc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.605+0000 7fe6fbe00640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc0cafa0 0x7fe6fc158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.605+0000 7fe6fbe00640 1 -- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6fc06c6e0 msgr2=0x7fe6fc13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.605+0000 7fe6fbe00640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6fc06c6e0 0x7fe6fc13f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.605+0000 7fe6fbe00640 1 -- 172.21.15.62:0/2301725630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ec0094d0 con 0x7fe6fc06c300 2024-03-23T11:23:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.606+0000 7fe6fbe00640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6fc06c300 0x7fe6fc13cf90 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec000c00 tx=0x7fe6ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.606+0000 7fe6f8c00640 1 -- 172.21.15.62:0/2301725630 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec039c90 con 0x7fe6fc06c300 2024-03-23T11:23:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.606+0000 7fe6f8c00640 1 -- 172.21.15.62:0/2301725630 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec039df0 con 0x7fe6fc06c300 2024-03-23T11:23:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.606+0000 7fe703000640 1 -- 172.21.15.62:0/2301725630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6fc15b160 con 0x7fe6fc06c300 2024-03-23T11:23:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.606+0000 7fe6f8c00640 1 -- 172.21.15.62:0/2301725630 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec043640 con 0x7fe6fc06c300 2024-03-23T11:23:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.606+0000 7fe703000640 1 -- 172.21.15.62:0/2301725630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6fc13d4d0 con 0x7fe6fc06c300 2024-03-23T11:23:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.607+0000 7fe703000640 1 -- 172.21.15.62:0/2301725630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6fc066c90 con 0x7fe6fc06c300 2024-03-23T11:23:56.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.608+0000 7fe6f8c00640 1 -- 172.21.15.62:0/2301725630 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe6ec0437a0 con 0x7fe6fc06c300 2024-03-23T11:23:56.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.609+0000 7fe6f8c00640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe6d806a7d0 0x7fe6d806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:56.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.609+0000 7fe6f8c00640 1 -- 172.21.15.62:0/2301725630 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6ec03e070 con 0x7fe6fc06c300 2024-03-23T11:23:56.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.610+0000 7fe6f3400640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe6d806a7d0 0x7fe6d806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.611+0000 7fe6f3400640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe6d806a7d0 0x7fe6d806cc90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc140a90 tx=0x7fe6e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:56.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.614+0000 7fe6f8c00640 1 -- 172.21.15.62:0/2301725630 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6ec077150 con 0x7fe6fc06c300 2024-03-23T11:23:56.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.816+0000 7fe703000640 1 -- 172.21.15.62:0/2301725630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6fc0cb510 con 0x7fe6fc06c300 2024-03-23T11:23:56.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.819+0000 7fe6f8c00640 1 -- 172.21.15.62:0/2301725630 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16097 (secure 0 0 0) 0x7fe6ec07a770 con 0x7fe6fc06c300 2024-03-23T11:23:56.815 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:23:56.815 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":62,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:22:32.731093+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:22:29.767222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-23T11:22:30.738596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:23:56.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 -- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe6d806a7d0 msgr2=0x7fe6d806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe6d806a7d0 0x7fe6d806cc90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc140a90 tx=0x7fe6e4009040 comp rx=0 tx=0).stop 2024-03-23T11:23:56.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 -- 172.21.15.62:0/2301725630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6fc06c300 msgr2=0x7fe6fc13cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:56.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6fc06c300 0x7fe6fc13cf90 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec000c00 tx=0x7fe6ec004730 comp rx=0 tx=0).stop 2024-03-23T11:23:56.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 -- 172.21.15.62:0/2301725630 shutdown_connections 2024-03-23T11:23:56.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe6d806a7d0 0x7fe6d806cc90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6fc0cafa0 0x7fe6fc158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6fc06c6e0 0x7fe6fc13f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 --2- 172.21.15.62:0/2301725630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6fc06c300 0x7fe6fc13cf90 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 -- 172.21.15.62:0/2301725630 >> 172.21.15.62:0/2301725630 conn(0x7fe6fc0b02e0 msgr2=0x7fe6fc072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 -- 172.21.15.62:0/2301725630 shutdown_connections 2024-03-23T11:23:56.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:56.822+0000 7fe703000640 1 -- 172.21.15.62:0/2301725630 wait complete. 2024-03-23T11:23:56.838 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:23:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.066+0000 7fb392800640 1 Processor -- start 2024-03-23T11:23:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.067+0000 7fb392800640 1 -- start start 2024-03-23T11:23:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.068+0000 7fb392800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb38c0d13b0 0x7fb38c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.068+0000 7fb392800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38c0d0810 con 0x7fb38c0cd260 2024-03-23T11:23:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.068+0000 7fb392800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38c0cd9d0 con 0x7fb38c15c960 2024-03-23T11:23:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.068+0000 7fb392800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38c0cdcb0 con 0x7fb38c0d13b0 2024-03-23T11:23:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.068+0000 7fb391e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb38c0cd260 0x7fb38c0b3310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39398/0 (socket says 172.21.15.62:39398) 2024-03-23T11:23:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.068+0000 7fb391e00640 1 -- 172.21.15.62:0/445203975 learned_addr learned my addr 172.21.15.62:0/445203975 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb390a00640 1 --2- 172.21.15.62:0/445203975 >> v2:172.21.15.120:3300/0 conn(0x7fb38c0d13b0 0x7fb38c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088440160 0 0) 0x7fb384001490 con 0x7fb38c0cd260 2024-03-23T11:23:57.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb374003520 con 0x7fb38c0cd260 2024-03-23T11:23:57.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 736949159 0 0) 0x7fb374003520 con 0x7fb38c0cd260 2024-03-23T11:23:57.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 >> v2:172.21.15.120:3300/0 conn(0x7fb38c0d13b0 msgr2=0x7fb38c0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:57.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 --2- 172.21.15.62:0/445203975 >> v2:172.21.15.120:3300/0 conn(0x7fb38c0d13b0 0x7fb38c0cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 >> v1:172.21.15.120:6789/0 conn(0x7fb38c15c960 legacy=0x7fb38c0d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:57.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c0cdf90 con 0x7fb38c0cd260 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fb384001b80 con 0x7fb38c0cd260 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 >> v1:172.21.15.62:6789/0 conn(0x7fb38c0cd260 legacy=0x7fb38c0b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 --2- 172.21.15.62:0/445203975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3740039c0 0x7fb374003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 --2- 172.21.15.62:0/445203975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3740047c0 0x7fb374004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 --2- 172.21.15.62:0/445203975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb374005180 0x7fb37400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0cdcb0 con 0x7fb374005180 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37400c060 con 0x7fb3740047c0 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.069+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37400c220 con 0x7fb3740039c0 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.070+0000 7fb391e00640 1 --2- 172.21.15.62:0/445203975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb374005180 0x7fb37400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.070+0000 7fb390a00640 1 --2- 172.21.15.62:0/445203975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3740047c0 0x7fb374004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.070+0000 7fb391400640 1 --2- 172.21.15.62:0/445203975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3740039c0 0x7fb374003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.070+0000 7fb391e00640 1 -- 172.21.15.62:0/445203975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3740039c0 msgr2=0x7fb374003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:57.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.070+0000 7fb391e00640 1 --2- 172.21.15.62:0/445203975 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3740039c0 0x7fb374003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.070+0000 7fb391e00640 1 -- 172.21.15.62:0/445203975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3740047c0 msgr2=0x7fb374004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:57.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.070+0000 7fb391e00640 1 --2- 172.21.15.62:0/445203975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3740047c0 0x7fb374004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.070+0000 7fb391e00640 1 -- 172.21.15.62:0/445203975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb374003520 con 0x7fb374005180 2024-03-23T11:23:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.071+0000 7fb391e00640 1 -- 172.21.15.62:0/445203975 reap_dead start 2024-03-23T11:23:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.071+0000 7fb391e00640 1 --2- 172.21.15.62:0/445203975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb374005180 0x7fb37400b980 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb38400a730 tx=0x7fb3740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52fa3fca0b359b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.072+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3840072d0 con 0x7fb374005180 2024-03-23T11:23:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.072+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38402f070 con 0x7fb374005180 2024-03-23T11:23:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.072+0000 7fb38be00640 1 -- 172.21.15.62:0/445203975 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3840076b0 con 0x7fb374005180 2024-03-23T11:23:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.072+0000 7fb392800640 1 -- 172.21.15.62:0/445203975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb374005180 msgr2=0x7fb37400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.072+0000 7fb392800640 1 --2- 172.21.15.62:0/445203975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb374005180 0x7fb37400b980 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb38400a730 tx=0x7fb3740057f0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.072+0000 7fb392800640 1 -- 172.21.15.62:0/445203975 shutdown_connections 2024-03-23T11:23:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.072+0000 7fb392800640 1 --2- 172.21.15.62:0/445203975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb374005180 0x7fb37400b980 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.072+0000 7fb392800640 1 -- 172.21.15.62:0/445203975 >> 172.21.15.62:0/445203975 conn(0x7fb38c089f60 msgr2=0x7fb38c0d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.072+0000 7fb392800640 1 -- 172.21.15.62:0/445203975 shutdown_connections 2024-03-23T11:23:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.072+0000 7fb392800640 1 -- 172.21.15.62:0/445203975 wait complete. 2024-03-23T11:23:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.074+0000 7fb392800640 1 Processor -- start 2024-03-23T11:23:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.074+0000 7fb392800640 1 -- start start 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.075+0000 7fb392800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb374005180 0x7fb38c0b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.075+0000 7fb392800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb38c15dd80 0x7fb38c0b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.075+0000 7fb392800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb38c0c8200 0x7fb38c151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.075+0000 7fb392800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0b2500 con 0x7fb38c0c8200 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.075+0000 7fb392800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0b27e0 con 0x7fb38c15dd80 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.075+0000 7fb392800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0b2ac0 con 0x7fb374005180 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb391e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb38c0c8200 0x7fb38c151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb391e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb38c0c8200 0x7fb38c151740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35096/0 (socket says 172.21.15.62:35096) 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb391e00640 1 -- 172.21.15.62:0/894064663 learned_addr learned my addr 172.21.15.62:0/894064663 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb391400640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb374005180 0x7fb38c0b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb390a00640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb38c15dd80 0x7fb38c0b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb391e00640 1 -- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb374005180 msgr2=0x7fb38c0b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb391e00640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb374005180 0x7fb38c0b35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb391e00640 1 -- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb38c15dd80 msgr2=0x7fb38c0b1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb391e00640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb38c15dd80 0x7fb38c0b1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb391e00640 1 -- 172.21.15.62:0/894064663 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c0b2da0 con 0x7fb38c0c8200 2024-03-23T11:23:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb390a00640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb38c15dd80 0x7fb38c0b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.076+0000 7fb391e00640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb38c0c8200 0x7fb38c151740 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb38400a730 tx=0x7fb384001e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.077+0000 7fb389600640 1 -- 172.21.15.62:0/894064663 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384004b60 con 0x7fb38c0c8200 2024-03-23T11:23:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.077+0000 7fb389600640 1 -- 172.21.15.62:0/894064663 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb384004cc0 con 0x7fb38c0c8200 2024-03-23T11:23:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.077+0000 7fb389600640 1 -- 172.21.15.62:0/894064663 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384001ac0 con 0x7fb38c0c8200 2024-03-23T11:23:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.077+0000 7fb392800640 1 -- 172.21.15.62:0/894064663 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38c0b3080 con 0x7fb38c0c8200 2024-03-23T11:23:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.077+0000 7fb392800640 1 -- 172.21.15.62:0/894064663 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38c0b2140 con 0x7fb38c0c8200 2024-03-23T11:23:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.078+0000 7fb392800640 1 -- 172.21.15.62:0/894064663 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38c066bb0 con 0x7fb38c0c8200 2024-03-23T11:23:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.079+0000 7fb389600640 1 -- 172.21.15.62:0/894064663 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb384031030 con 0x7fb38c0c8200 2024-03-23T11:23:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.080+0000 7fb389600640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb36c06a7a0 0x7fb36c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.080+0000 7fb389600640 1 -- 172.21.15.62:0/894064663 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3840aa730 con 0x7fb38c0c8200 2024-03-23T11:23:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.081+0000 7fb391400640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb36c06a7a0 0x7fb36c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.082+0000 7fb391400640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb36c06a7a0 0x7fb36c06cc60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb38c15ead0 tx=0x7fb380009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:57.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.085+0000 7fb389600640 1 -- 172.21.15.62:0/894064663 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb384075b40 con 0x7fb38c0c8200 2024-03-23T11:23:57.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.323+0000 7fb392800640 1 -- 172.21.15.62:0/894064663 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb38c066bb0 con 0x7fb38c0c8200 2024-03-23T11:23:57.320 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:23:57.325+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:23:57.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.343+0000 7fb389600640 1 -- 172.21.15.62:0/894064663 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb384079160 con 0x7fb38c0c8200 2024-03-23T11:23:57.339 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:23:57.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 -- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb36c06a7a0 msgr2=0x7fb36c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:57.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb36c06a7a0 0x7fb36c06cc60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb38c15ead0 tx=0x7fb380009040 comp rx=0 tx=0).stop 2024-03-23T11:23:57.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 -- 172.21.15.62:0/894064663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb38c0c8200 msgr2=0x7fb38c151740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb38c0c8200 0x7fb38c151740 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb38400a730 tx=0x7fb384001e40 comp rx=0 tx=0).stop 2024-03-23T11:23:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 -- 172.21.15.62:0/894064663 shutdown_connections 2024-03-23T11:23:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb36c06a7a0 0x7fb36c06cc60 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb38c0c8200 0x7fb38c151740 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb38c15dd80 0x7fb38c0b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 --2- 172.21.15.62:0/894064663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb374005180 0x7fb38c0b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 -- 172.21.15.62:0/894064663 >> 172.21.15.62:0/894064663 conn(0x7fb38c089f60 msgr2=0x7fb38c0afb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 -- 172.21.15.62:0/894064663 shutdown_connections 2024-03-23T11:23:57.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.346+0000 7fb392800640 1 -- 172.21.15.62:0/894064663 wait complete. 2024-03-23T11:23:57.360 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T11:23:57.373 INFO:tasks.ceph.mds.b.smithi120.stderr: -6319> 2024-03-23T11:22:37.216+0000 7f3f3bc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:23:57.380 INFO:tasks.ceph.mds.b.smithi120.stderr: -4648> 2024-03-23T11:22:44.709+0000 7f3f37600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={6228=0-4194304@1} caps={6228=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6228 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x5565aeb39b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-23T11:23:57.384 INFO:tasks.ceph.mds.b.smithi120.stderr: -2808> 2024-03-23T11:23:16.881+0000 7f3f37600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-03-23T11:22:42.917205+0000 1=1+0) (iversion lock) caps={6228=pAsLsXsFs/-@2},l=6228(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x5565aeb39b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-23T11:23:57.457 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-23T11:23:57.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.590+0000 7f5f11e00640 1 Processor -- start 2024-03-23T11:23:57.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.590+0000 7f5f11e00640 1 -- start start 2024-03-23T11:23:57.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.592+0000 7f5f11e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5f0c1571e0 0x7f5f0c0ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.592+0000 7f5f11e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0c1613a0 0x7f5f0c15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.592+0000 7f5f11e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f0c06cf40 con 0x7f5f0c0ade70 2024-03-23T11:23:57.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.592+0000 7f5f11e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f0c1575c0 con 0x7f5f0c1571e0 2024-03-23T11:23:57.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.592+0000 7f5f11e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c06c3a0 con 0x7f5f0c1613a0 2024-03-23T11:23:57.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.592+0000 7f5f0aa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5f0c1571e0 0x7f5f0c0ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.592+0000 7f5f0aa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5f0c1571e0 0x7f5f0c0ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48618/0 (socket says 172.21.15.62:48618) 2024-03-23T11:23:57.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.592+0000 7f5f0aa00640 1 -- 172.21.15.62:0/186060554 learned_addr learned my addr 172.21.15.62:0/186060554 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:57.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.592+0000 7f5f0aa00640 1 -- 172.21.15.62:0/186060554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0c1613a0 msgr2=0x7f5f0c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:57.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.592+0000 7f5f0b400640 1 --2- 172.21.15.62:0/186060554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0c1613a0 0x7f5f0c15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.593+0000 7f5f0aa00640 1 --2- 172.21.15.62:0/186060554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0c1613a0 0x7f5f0c15e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.593+0000 7f5f0aa00640 1 -- 172.21.15.62:0/186060554 >> v1:172.21.15.120:6789/0 conn(0x7f5f0c0ade70 legacy=0x7f5f0c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:57.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.593+0000 7f5f0aa00640 1 -- 172.21.15.62:0/186060554 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c06c610 con 0x7f5f0c1571e0 2024-03-23T11:23:57.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.594+0000 7f5f0aa00640 1 --2- 172.21.15.62:0/186060554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f0c1571e0 0x7f5f0c0ad930 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009870 tx=0x7f5efc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2dbfdbdc6725794 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:57.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.594+0000 7f5f0a000640 1 -- 172.21.15.62:0/186060554 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc0317c0 con 0x7f5f0c1571e0 2024-03-23T11:23:57.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.594+0000 7f5f0a000640 1 -- 172.21.15.62:0/186060554 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc031920 con 0x7f5f0c1571e0 2024-03-23T11:23:57.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.594+0000 7f5f0a000640 1 -- 172.21.15.62:0/186060554 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc031cb0 con 0x7f5f0c1571e0 2024-03-23T11:23:57.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.595+0000 7f5f11e00640 1 -- 172.21.15.62:0/186060554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f0c1571e0 msgr2=0x7f5f0c0ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.595+0000 7f5f11e00640 1 --2- 172.21.15.62:0/186060554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f0c1571e0 0x7f5f0c0ad930 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009870 tx=0x7f5efc02b860 comp rx=0 tx=0).stop 2024-03-23T11:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.595+0000 7f5f11e00640 1 -- 172.21.15.62:0/186060554 shutdown_connections 2024-03-23T11:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.595+0000 7f5f11e00640 1 --2- 172.21.15.62:0/186060554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0c1613a0 0x7f5f0c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.595+0000 7f5f11e00640 1 --2- 172.21.15.62:0/186060554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f0c1571e0 0x7f5f0c0ad930 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.595+0000 7f5f11e00640 1 -- 172.21.15.62:0/186060554 >> 172.21.15.62:0/186060554 conn(0x7f5f0c0b02e0 msgr2=0x7f5f0c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.595+0000 7f5f11e00640 1 -- 172.21.15.62:0/186060554 shutdown_connections 2024-03-23T11:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.595+0000 7f5f11e00640 1 -- 172.21.15.62:0/186060554 wait complete. 2024-03-23T11:23:57.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.597+0000 7f5f11e00640 1 Processor -- start 2024-03-23T11:23:57.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.597+0000 7f5f11e00640 1 -- start start 2024-03-23T11:23:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.598+0000 7f5f11e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f0c0ade70 0x7f5f0c146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.598+0000 7f5f11e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0c1571e0 0x7f5f0c15a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.598+0000 7f5f11e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f0c1613a0 0x7f5f0c15acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.598+0000 7f5f11e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c0cc1d0 con 0x7f5f0c0ade70 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.598+0000 7f5f11e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c157f00 con 0x7f5f0c1613a0 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.598+0000 7f5f11e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c158150 con 0x7f5f0c1571e0 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.598+0000 7f5f0aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f0c0ade70 0x7f5f0c146d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.598+0000 7f5f0aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f0c0ade70 0x7f5f0c146d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48630/0 (socket says 172.21.15.62:48630) 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.598+0000 7f5f0aa00640 1 -- 172.21.15.62:0/2736928981 learned_addr learned my addr 172.21.15.62:0/2736928981 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.598+0000 7f5f02000640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0c1571e0 0x7f5f0c15a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f0b400640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f0c1613a0 0x7f5f0c15acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f0aa00640 1 -- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0c1571e0 msgr2=0x7f5f0c15a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f0aa00640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0c1571e0 0x7f5f0c15a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f0aa00640 1 -- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f0c1613a0 msgr2=0x7f5f0c15acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f0aa00640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f0c1613a0 0x7f5f0c15acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f0aa00640 1 -- 172.21.15.62:0/2736928981 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc0094d0 con 0x7f5f0c0ade70 2024-03-23T11:23:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f02000640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0c1571e0 0x7f5f0c15a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f0b400640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f0c1613a0 0x7f5f0c15acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f0aa00640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f0c0ade70 0x7f5f0c146d70 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5efc000c00 tx=0x7f5efc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f03e00640 1 -- 172.21.15.62:0/2736928981 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc0329a0 con 0x7f5f0c0ade70 2024-03-23T11:23:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f03e00640 1 -- 172.21.15.62:0/2736928981 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc032b20 con 0x7f5f0c0ade70 2024-03-23T11:23:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f11e00640 1 -- 172.21.15.62:0/2736928981 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0c158430 con 0x7f5f0c0ade70 2024-03-23T11:23:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.599+0000 7f5f03e00640 1 -- 172.21.15.62:0/2736928981 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc0453f0 con 0x7f5f0c0ade70 2024-03-23T11:23:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.600+0000 7f5f11e00640 1 -- 172.21.15.62:0/2736928981 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0c1589f0 con 0x7f5f0c0ade70 2024-03-23T11:23:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.601+0000 7f5f11e00640 1 -- 172.21.15.62:0/2736928981 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0c066c90 con 0x7f5f0c0ade70 2024-03-23T11:23:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.602+0000 7f5f03e00640 1 -- 172.21.15.62:0/2736928981 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5efc04d3f0 con 0x7f5f0c0ade70 2024-03-23T11:23:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.602+0000 7f5f03e00640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee806a7a0 0x7f5ee806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.603+0000 7f5f03e00640 1 -- 172.21.15.62:0/2736928981 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5efc0aada0 con 0x7f5f0c0ade70 2024-03-23T11:23:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.603+0000 7f5f02000640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee806a7a0 0x7f5ee806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.605+0000 7f5f02000640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee806a7a0 0x7f5ee806cc60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c0cace0 tx=0x7f5ef40073a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.608+0000 7f5f03e00640 1 -- 172.21.15.62:0/2736928981 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5efc076150 con 0x7f5f0c0ade70 2024-03-23T11:23:57.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:57.854+0000 7f5f11e00640 1 -- 172.21.15.62:0/2736928981 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5f0c066c90 con 0x7f5f0c0ade70 2024-03-23T11:23:58.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.449+0000 7f5f03e00640 1 -- 172.21.15.62:0/2736928981 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5efc079770 con 0x7f5f0c0ade70 2024-03-23T11:23:58.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.452+0000 7f5f11e00640 1 -- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee806a7a0 msgr2=0x7f5ee806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:58.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.452+0000 7f5f11e00640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee806a7a0 0x7f5ee806cc60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c0cace0 tx=0x7f5ef40073a0 comp rx=0 tx=0).stop 2024-03-23T11:23:58.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.452+0000 7f5f11e00640 1 -- 172.21.15.62:0/2736928981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f0c0ade70 msgr2=0x7f5f0c146d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:58.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.452+0000 7f5f11e00640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f0c0ade70 0x7f5f0c146d70 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5efc000c00 tx=0x7f5efc0046c0 comp rx=0 tx=0).stop 2024-03-23T11:23:58.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.453+0000 7f5f11e00640 1 -- 172.21.15.62:0/2736928981 shutdown_connections 2024-03-23T11:23:58.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.453+0000 7f5f11e00640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee806a7a0 0x7f5ee806cc60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:58.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.453+0000 7f5f11e00640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f0c1613a0 0x7f5f0c15acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:58.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.453+0000 7f5f11e00640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0c1571e0 0x7f5f0c15a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:58.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.453+0000 7f5f11e00640 1 --2- 172.21.15.62:0/2736928981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f0c0ade70 0x7f5f0c146d70 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:58.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.453+0000 7f5f11e00640 1 -- 172.21.15.62:0/2736928981 >> 172.21.15.62:0/2736928981 conn(0x7f5f0c0b02e0 msgr2=0x7f5f0c0acde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:58.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.453+0000 7f5f11e00640 1 -- 172.21.15.62:0/2736928981 shutdown_connections 2024-03-23T11:23:58.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.453+0000 7f5f11e00640 1 -- 172.21.15.62:0/2736928981 wait complete. 2024-03-23T11:23:58.469 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T11:23:58.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.689+0000 7fe099e00640 1 Processor -- start 2024-03-23T11:23:58.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.689+0000 7fe099e00640 1 -- start start 2024-03-23T11:23:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.691+0000 7fe099e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe0940b0f00 0x7fe094070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.691+0000 7fe099e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09406d610 con 0x7fe0940cd1a0 2024-03-23T11:23:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.691+0000 7fe099e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09406d8f0 con 0x7fe09415a8a0 2024-03-23T11:23:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.691+0000 7fe099e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe09406dbd0 con 0x7fe0940b0f00 2024-03-23T11:23:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.691+0000 7fe08a000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe0940b0f00 0x7fe094070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.691+0000 7fe08a000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe0940b0f00 0x7fe094070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48634/0 (socket says 172.21.15.62:48634) 2024-03-23T11:23:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.691+0000 7fe08a000640 1 -- 172.21.15.62:0/3377631028 learned_addr learned my addr 172.21.15.62:0/3377631028 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:58.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.692+0000 7fe08a000640 1 -- 172.21.15.62:0/3377631028 >> v1:172.21.15.120:6789/0 conn(0x7fe09415a8a0 legacy=0x7fe0940b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:58.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.692+0000 7fe08a000640 1 -- 172.21.15.62:0/3377631028 >> v1:172.21.15.62:6789/0 conn(0x7fe0940cd1a0 legacy=0x7fe0940cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:58.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.692+0000 7fe08a000640 1 -- 172.21.15.62:0/3377631028 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09406df40 con 0x7fe0940b0f00 2024-03-23T11:23:58.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.692+0000 7fe092000640 1 -- 172.21.15.62:0/3377631028 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219102684 0 0) 0x7fe09406d610 con 0x7fe0940cd1a0 2024-03-23T11:23:58.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.693+0000 7fe08a000640 1 --2- 172.21.15.62:0/3377631028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0940b0f00 0x7fe094070270 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009870 tx=0x7fe07c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99977a8ee983036d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:58.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.693+0000 7fe092000640 1 -- 172.21.15.62:0/3377631028 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c030860 con 0x7fe0940b0f00 2024-03-23T11:23:58.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.694+0000 7fe092000640 1 -- 172.21.15.62:0/3377631028 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c0309c0 con 0x7fe0940b0f00 2024-03-23T11:23:58.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.694+0000 7fe092000640 1 -- 172.21.15.62:0/3377631028 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c030d50 con 0x7fe0940b0f00 2024-03-23T11:23:58.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.694+0000 7fe099e00640 1 -- 172.21.15.62:0/3377631028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0940b0f00 msgr2=0x7fe094070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:58.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.694+0000 7fe099e00640 1 --2- 172.21.15.62:0/3377631028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0940b0f00 0x7fe094070270 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009870 tx=0x7fe07c02b9a0 comp rx=0 tx=0).stop 2024-03-23T11:23:58.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.694+0000 7fe099e00640 1 -- 172.21.15.62:0/3377631028 shutdown_connections 2024-03-23T11:23:58.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.694+0000 7fe099e00640 1 --2- 172.21.15.62:0/3377631028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0940b0f00 0x7fe094070270 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:58.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.694+0000 7fe099e00640 1 -- 172.21.15.62:0/3377631028 >> 172.21.15.62:0/3377631028 conn(0x7fe0940b17d0 msgr2=0x7fe0940b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:58.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.694+0000 7fe099e00640 1 -- 172.21.15.62:0/3377631028 shutdown_connections 2024-03-23T11:23:58.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.694+0000 7fe099e00640 1 -- 172.21.15.62:0/3377631028 wait complete. 2024-03-23T11:23:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.696+0000 7fe099e00640 1 Processor -- start 2024-03-23T11:23:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.696+0000 7fe099e00640 1 -- start start 2024-03-23T11:23:58.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.697+0000 7fe099e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0940b0f00 0x7fe09414d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:58.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.697+0000 7fe099e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0940cd1a0 0x7fe0941f3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:58.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.697+0000 7fe099e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415a8a0 0x7fe0941f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:58.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.697+0000 7fe099e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09414c540 con 0x7fe0940b0f00 2024-03-23T11:23:58.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.697+0000 7fe099e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09414c820 con 0x7fe0940cd1a0 2024-03-23T11:23:58.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.697+0000 7fe099e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09414cb00 con 0x7fe09415a8a0 2024-03-23T11:23:58.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.697+0000 7fe092a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0940b0f00 0x7fe09414d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:58.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.697+0000 7fe092a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0940b0f00 0x7fe09414d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48646/0 (socket says 172.21.15.62:48646) 2024-03-23T11:23:58.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.697+0000 7fe092a00640 1 -- 172.21.15.62:0/1265741437 learned_addr learned my addr 172.21.15.62:0/1265741437 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:58.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.698+0000 7fe092a00640 1 -- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415a8a0 msgr2=0x7fe0941f41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:58.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.698+0000 7fe092a00640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415a8a0 0x7fe0941f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:58.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.698+0000 7fe092a00640 1 -- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0940cd1a0 msgr2=0x7fe0941f3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:58.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.698+0000 7fe092a00640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0940cd1a0 0x7fe0941f3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:58.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.698+0000 7fe092a00640 1 -- 172.21.15.62:0/1265741437 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c0094d0 con 0x7fe0940b0f00 2024-03-23T11:23:58.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.698+0000 7fe092a00640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0940b0f00 0x7fe09414d5c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe08400c450 tx=0x7fe08400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.698+0000 7fe08be00640 1 -- 172.21.15.62:0/1265741437 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0840052d0 con 0x7fe0940b0f00 2024-03-23T11:23:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.698+0000 7fe08be00640 1 -- 172.21.15.62:0/1265741437 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe084005430 con 0x7fe0940b0f00 2024-03-23T11:23:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.698+0000 7fe08be00640 1 -- 172.21.15.62:0/1265741437 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084012870 con 0x7fe0940b0f00 2024-03-23T11:23:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.699+0000 7fe099e00640 1 -- 172.21.15.62:0/1265741437 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09414cde0 con 0x7fe0940b0f00 2024-03-23T11:23:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.699+0000 7fe099e00640 1 -- 172.21.15.62:0/1265741437 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09414dd60 con 0x7fe0940b0f00 2024-03-23T11:23:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.700+0000 7fe099e00640 1 -- 172.21.15.62:0/1265741437 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe094066c90 con 0x7fe0940b0f00 2024-03-23T11:23:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.701+0000 7fe08be00640 1 -- 172.21.15.62:0/1265741437 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe0840129d0 con 0x7fe0940b0f00 2024-03-23T11:23:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.701+0000 7fe08be00640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe07006a7a0 0x7fe07006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.702+0000 7fe08be00640 1 -- 172.21.15.62:0/1265741437 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe084088a00 con 0x7fe0940b0f00 2024-03-23T11:23:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.702+0000 7fe08a000640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe07006a7a0 0x7fe07006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.703+0000 7fe08a000640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe07006a7a0 0x7fe07006cc60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe07c036000 tx=0x7fe07c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:58.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.706+0000 7fe08be00640 1 -- 172.21.15.62:0/1265741437 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe084053d20 con 0x7fe0940b0f00 2024-03-23T11:23:58.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:58.903+0000 7fe099e00640 1 -- 172.21.15.62:0/1265741437 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe0940b1d50 con 0x7fe0940b0f00 2024-03-23T11:23:59.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.472+0000 7fe08be00640 1 -- 172.21.15.62:0/1265741437 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v64) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe084057340 con 0x7fe0940b0f00 2024-03-23T11:23:59.469 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' removed 2024-03-23T11:23:59.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.477+0000 7fe099e00640 1 -- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe07006a7a0 msgr2=0x7fe07006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:59.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.477+0000 7fe099e00640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe07006a7a0 0x7fe07006cc60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe07c036000 tx=0x7fe07c03b040 comp rx=0 tx=0).stop 2024-03-23T11:23:59.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.477+0000 7fe099e00640 1 -- 172.21.15.62:0/1265741437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0940b0f00 msgr2=0x7fe09414d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:59.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.477+0000 7fe099e00640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0940b0f00 0x7fe09414d5c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe08400c450 tx=0x7fe08400c910 comp rx=0 tx=0).stop 2024-03-23T11:23:59.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.477+0000 7fe099e00640 1 -- 172.21.15.62:0/1265741437 shutdown_connections 2024-03-23T11:23:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.477+0000 7fe099e00640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe07006a7a0 0x7fe07006cc60 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.477+0000 7fe099e00640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09415a8a0 0x7fe0941f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.477+0000 7fe099e00640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0940cd1a0 0x7fe0941f3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.477+0000 7fe099e00640 1 --2- 172.21.15.62:0/1265741437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0940b0f00 0x7fe09414d5c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.477+0000 7fe099e00640 1 -- 172.21.15.62:0/1265741437 >> 172.21.15.62:0/1265741437 conn(0x7fe0940b17d0 msgr2=0x7fe0940cb170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.478+0000 7fe099e00640 1 -- 172.21.15.62:0/1265741437 shutdown_connections 2024-03-23T11:23:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.478+0000 7fe099e00640 1 -- 172.21.15.62:0/1265741437 wait complete. 2024-03-23T11:23:59.494 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T11:23:59.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.718+0000 7efe53e00640 1 Processor -- start 2024-03-23T11:23:59.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.718+0000 7efe53e00640 1 -- start start 2024-03-23T11:23:59.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.719+0000 7efe53e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efe5406dd30 0x7efe5406c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:59.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.719+0000 7efe53e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5415cfd0 con 0x7efe540ce690 2024-03-23T11:23:59.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.719+0000 7efe53e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5415d220 con 0x7efe5406d950 2024-03-23T11:23:59.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.719+0000 7efe53e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe5415d500 con 0x7efe5406dd30 2024-03-23T11:23:59.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.720+0000 7efe52000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efe5406dd30 0x7efe5406c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:59.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.720+0000 7efe53400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7efe540ce690 0x7efe54159d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40890/0 (socket says 172.21.15.62:40890) 2024-03-23T11:23:59.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.720+0000 7efe53400640 1 -- 172.21.15.62:0/2413643412 learned_addr learned my addr 172.21.15.62:0/2413643412 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:23:59.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.720+0000 7efe52000640 1 -- 172.21.15.62:0/2413643412 >> v1:172.21.15.120:6789/0 conn(0x7efe5406d950 legacy=0x7efe54070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:23:59.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.720+0000 7efe51600640 1 -- 172.21.15.62:0/2413643412 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539291615 0 0) 0x7efe5415cfd0 con 0x7efe540ce690 2024-03-23T11:23:59.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.720+0000 7efe52000640 1 -- 172.21.15.62:0/2413643412 >> v1:172.21.15.62:6789/0 conn(0x7efe540ce690 legacy=0x7efe54159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:59.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.720+0000 7efe52000640 1 -- 172.21.15.62:0/2413643412 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5415d7e0 con 0x7efe5406dd30 2024-03-23T11:23:59.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.721+0000 7efe52000640 1 --2- 172.21.15.62:0/2413643412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe5406dd30 0x7efe5406c010 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efe400097a0 tx=0x7efe4002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3bf607583d6078 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:59.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.722+0000 7efe51600640 1 -- 172.21.15.62:0/2413643412 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4002e070 con 0x7efe5406dd30 2024-03-23T11:23:59.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.722+0000 7efe51600640 1 -- 172.21.15.62:0/2413643412 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe400315a0 con 0x7efe5406dd30 2024-03-23T11:23:59.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.722+0000 7efe51600640 1 -- 172.21.15.62:0/2413643412 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe40031930 con 0x7efe5406dd30 2024-03-23T11:23:59.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.722+0000 7efe53e00640 1 -- 172.21.15.62:0/2413643412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe5406dd30 msgr2=0x7efe5406c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:59.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.722+0000 7efe53e00640 1 --2- 172.21.15.62:0/2413643412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe5406dd30 0x7efe5406c010 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efe400097a0 tx=0x7efe4002b840 comp rx=0 tx=0).stop 2024-03-23T11:23:59.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.722+0000 7efe53e00640 1 -- 172.21.15.62:0/2413643412 shutdown_connections 2024-03-23T11:23:59.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.722+0000 7efe53e00640 1 --2- 172.21.15.62:0/2413643412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe5406dd30 0x7efe5406c010 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:59.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.722+0000 7efe53e00640 1 -- 172.21.15.62:0/2413643412 >> 172.21.15.62:0/2413643412 conn(0x7efe540b0280 msgr2=0x7efe540b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:23:59.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.723+0000 7efe53e00640 1 -- 172.21.15.62:0/2413643412 shutdown_connections 2024-03-23T11:23:59.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.723+0000 7efe53e00640 1 -- 172.21.15.62:0/2413643412 wait complete. 2024-03-23T11:23:59.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.724+0000 7efe53e00640 1 Processor -- start 2024-03-23T11:23:59.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.724+0000 7efe53e00640 1 -- start start 2024-03-23T11:23:59.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.725+0000 7efe53e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe5406d950 0x7efe541490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:59.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.725+0000 7efe53e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe5406dd30 0x7efe540b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:59.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.725+0000 7efe53e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe540ce690 0x7efe5414d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:59.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.725+0000 7efe53e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe541583d0 con 0x7efe5406dd30 2024-03-23T11:23:59.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.725+0000 7efe53e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe54149620 con 0x7efe5406d950 2024-03-23T11:23:59.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.725+0000 7efe53e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe54157470 con 0x7efe540ce690 2024-03-23T11:23:59.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.726+0000 7efe52000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe5406dd30 0x7efe540b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:59.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.726+0000 7efe52000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe5406dd30 0x7efe540b2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48664/0 (socket says 172.21.15.62:48664) 2024-03-23T11:23:59.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.726+0000 7efe52000640 1 -- 172.21.15.62:0/3600300970 learned_addr learned my addr 172.21.15.62:0/3600300970 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:23:59.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.726+0000 7efe53400640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe540ce690 0x7efe5414d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:59.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.726+0000 7efe52a00640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe5406d950 0x7efe541490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:59.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.726+0000 7efe52000640 1 -- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe540ce690 msgr2=0x7efe5414d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:59.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.726+0000 7efe52000640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe540ce690 0x7efe5414d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:59.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.726+0000 7efe52000640 1 -- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe5406d950 msgr2=0x7efe541490e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:23:59.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.726+0000 7efe52000640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe5406d950 0x7efe541490e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:23:59.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.726+0000 7efe52000640 1 -- 172.21.15.62:0/3600300970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe40009400 con 0x7efe5406dd30 2024-03-23T11:23:59.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.726+0000 7efe52a00640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe5406d950 0x7efe541490e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:23:59.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.727+0000 7efe52000640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe5406dd30 0x7efe540b2c10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efe40034040 tx=0x7efe400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:59.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.727+0000 7efe4aa00640 1 -- 172.21.15.62:0/3600300970 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe40032900 con 0x7efe5406dd30 2024-03-23T11:23:59.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.727+0000 7efe4aa00640 1 -- 172.21.15.62:0/3600300970 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe40032a60 con 0x7efe5406dd30 2024-03-23T11:23:59.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.727+0000 7efe4aa00640 1 -- 172.21.15.62:0/3600300970 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe40032e10 con 0x7efe5406dd30 2024-03-23T11:23:59.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.727+0000 7efe53e00640 1 -- 172.21.15.62:0/3600300970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe54157750 con 0x7efe5406dd30 2024-03-23T11:23:59.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.727+0000 7efe53e00640 1 -- 172.21.15.62:0/3600300970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe54157d10 con 0x7efe5406dd30 2024-03-23T11:23:59.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.728+0000 7efe53e00640 1 -- 172.21.15.62:0/3600300970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe54158510 con 0x7efe5406dd30 2024-03-23T11:23:59.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.729+0000 7efe4aa00640 1 -- 172.21.15.62:0/3600300970 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7efe4003b3f0 con 0x7efe5406dd30 2024-03-23T11:23:59.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.730+0000 7efe4aa00640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe3406a7a0 0x7efe3406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:23:59.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.730+0000 7efe4aa00640 1 -- 172.21.15.62:0/3600300970 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5357+0+0 (secure 0 0 0) 0x7efe400aadd0 con 0x7efe5406dd30 2024-03-23T11:23:59.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.731+0000 7efe52a00640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe3406a7a0 0x7efe3406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:23:59.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.732+0000 7efe52a00640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe3406a7a0 0x7efe3406cc60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efe541f14c0 tx=0x7efe44009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:23:59.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.736+0000 7efe4aa00640 1 -- 172.21.15.62:0/3600300970 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efe54158510 con 0x7efe5406dd30 2024-03-23T11:23:59.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:23:59.941+0000 7efe53e00640 1 -- 172.21.15.62:0/3600300970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efe54158510 con 0x7efe5406dd30 2024-03-23T11:24:00.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.483+0000 7efe4aa00640 1 -- 172.21.15.62:0/3600300970 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v65) v1 ==== 169+0+0 (secure 0 0 0) 0x7efe54158510 con 0x7efe5406dd30 2024-03-23T11:24:00.480 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' removed 2024-03-23T11:24:00.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.492+0000 7efe53e00640 1 -- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe3406a7a0 msgr2=0x7efe3406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:00.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.492+0000 7efe53e00640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe3406a7a0 0x7efe3406cc60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efe541f14c0 tx=0x7efe44009040 comp rx=0 tx=0).stop 2024-03-23T11:24:00.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.492+0000 7efe53e00640 1 -- 172.21.15.62:0/3600300970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe5406dd30 msgr2=0x7efe540b2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:00.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.492+0000 7efe53e00640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe5406dd30 0x7efe540b2c10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efe40034040 tx=0x7efe400046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.492+0000 7efe53e00640 1 -- 172.21.15.62:0/3600300970 shutdown_connections 2024-03-23T11:24:00.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.492+0000 7efe53e00640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efe3406a7a0 0x7efe3406cc60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.492+0000 7efe53e00640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe5406dd30 0x7efe540b2c10 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.492+0000 7efe53e00640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe540ce690 0x7efe5414d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.492+0000 7efe53e00640 1 --2- 172.21.15.62:0/3600300970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe5406d950 0x7efe541490e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.492+0000 7efe53e00640 1 -- 172.21.15.62:0/3600300970 >> 172.21.15.62:0/3600300970 conn(0x7efe540b0280 msgr2=0x7efe540ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:00.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.494+0000 7efe53e00640 1 -- 172.21.15.62:0/3600300970 shutdown_connections 2024-03-23T11:24:00.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.494+0000 7efe53e00640 1 -- 172.21.15.62:0/3600300970 wait complete. 2024-03-23T11:24:00.516 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T11:24:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.732+0000 7fbcb0800640 1 Processor -- start 2024-03-23T11:24:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.732+0000 7fbcb0800640 1 -- start start 2024-03-23T11:24:00.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.734+0000 7fbcb0800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbca815e280 0x7fbca815e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:00.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.734+0000 7fbcb0800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca806cf40 con 0x7fbca81571e0 2024-03-23T11:24:00.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.734+0000 7fbcb0800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca81575c0 con 0x7fbca81613a0 2024-03-23T11:24:00.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.734+0000 7fbcb0800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca806c3a0 con 0x7fbca815e280 2024-03-23T11:24:00.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.734+0000 7fbcade00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbca81571e0 0x7fbca80ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40898/0 (socket says 172.21.15.62:40898) 2024-03-23T11:24:00.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.734+0000 7fbcade00640 1 -- 172.21.15.62:0/2157425207 learned_addr learned my addr 172.21.15.62:0/2157425207 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:00.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.735+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321378019 0 0) 0x7fbca806cf40 con 0x7fbca81571e0 2024-03-23T11:24:00.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.735+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc90003520 con 0x7fbca81571e0 2024-03-23T11:24:00.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.735+0000 7fbcae800640 1 --2- 172.21.15.62:0/2157425207 >> v2:172.21.15.120:3300/0 conn(0x7fbca815e280 0x7fbca815e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:00.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.735+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1837304347 0 0) 0x7fbc90003520 con 0x7fbca81571e0 2024-03-23T11:24:00.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.735+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 >> v2:172.21.15.120:3300/0 conn(0x7fbca815e280 msgr2=0x7fbca815e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:00.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.735+0000 7fbcaca00640 1 --2- 172.21.15.62:0/2157425207 >> v2:172.21.15.120:3300/0 conn(0x7fbca815e280 0x7fbca815e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.735+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 >> v1:172.21.15.120:6789/0 conn(0x7fbca81613a0 legacy=0x7fbca806d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:00.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.735+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca80ae040 con 0x7fbca81571e0 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.735+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fbc9c001670 con 0x7fbca81571e0 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.735+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 >> v1:172.21.15.62:6789/0 conn(0x7fbca81571e0 legacy=0x7fbca80ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.735+0000 7fbcaca00640 1 --2- 172.21.15.62:0/2157425207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc900039c0 0x7fbc90003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcaca00640 1 --2- 172.21.15.62:0/2157425207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc900047c0 0x7fbc90004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcaca00640 1 --2- 172.21.15.62:0/2157425207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc900052a0 0x7fbc9000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca806c3a0 con 0x7fbc900047c0 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9000c1c0 con 0x7fbc900052a0 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9000c380 con 0x7fbc900039c0 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcad400640 1 --2- 172.21.15.62:0/2157425207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc900047c0 0x7fbc90004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcad400640 1 -- 172.21.15.62:0/2157425207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc900039c0 msgr2=0x7fbc90003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcad400640 1 --2- 172.21.15.62:0/2157425207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc900039c0 0x7fbc90003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcad400640 1 -- 172.21.15.62:0/2157425207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc900052a0 msgr2=0x7fbc9000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:00.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcae800640 1 --2- 172.21.15.62:0/2157425207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc900052a0 0x7fbc9000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:00.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcad400640 1 --2- 172.21.15.62:0/2157425207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc900052a0 0x7fbc9000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcae800640 1 -- 172.21.15.62:0/2157425207 reap_dead start 2024-03-23T11:24:00.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.736+0000 7fbcad400640 1 -- 172.21.15.62:0/2157425207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc90003520 con 0x7fbc900047c0 2024-03-23T11:24:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.737+0000 7fbcad400640 1 --2- 172.21.15.62:0/2157425207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc900047c0 0x7fbc90004c40 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fbca81617d0 tx=0x7fbc900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c0b8ccca47099e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.738+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc980052d0 con 0x7fbc900047c0 2024-03-23T11:24:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.738+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9802e050 con 0x7fbc900047c0 2024-03-23T11:24:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.738+0000 7fbcaca00640 1 -- 172.21.15.62:0/2157425207 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98003830 con 0x7fbc900047c0 2024-03-23T11:24:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.738+0000 7fbcb0800640 1 -- 172.21.15.62:0/2157425207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc900047c0 msgr2=0x7fbc90004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.738+0000 7fbcb0800640 1 --2- 172.21.15.62:0/2157425207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc900047c0 0x7fbc90004c40 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fbca81617d0 tx=0x7fbc900057f0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.738+0000 7fbcb0800640 1 -- 172.21.15.62:0/2157425207 shutdown_connections 2024-03-23T11:24:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.738+0000 7fbcb0800640 1 --2- 172.21.15.62:0/2157425207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc900047c0 0x7fbc90004c40 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.738+0000 7fbcb0800640 1 -- 172.21.15.62:0/2157425207 >> 172.21.15.62:0/2157425207 conn(0x7fbca80b02e0 msgr2=0x7fbca80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:00.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.738+0000 7fbcb0800640 1 -- 172.21.15.62:0/2157425207 shutdown_connections 2024-03-23T11:24:00.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.738+0000 7fbcb0800640 1 -- 172.21.15.62:0/2157425207 wait complete. 2024-03-23T11:24:00.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.740+0000 7fbcb0800640 1 Processor -- start 2024-03-23T11:24:00.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.740+0000 7fbcb0800640 1 -- start start 2024-03-23T11:24:00.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.741+0000 7fbcb0800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc900047c0 0x7fbca80cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:00.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.741+0000 7fbcb0800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbca81f6450 0x7fbca8142ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:00.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.741+0000 7fbcb0800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbca815fa10 0x7fbca815fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:00.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.741+0000 7fbcb0800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca8160400 con 0x7fbca81f6450 2024-03-23T11:24:00.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.741+0000 7fbcb0800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca815acc0 con 0x7fbca815fa10 2024-03-23T11:24:00.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.741+0000 7fbcb0800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca80cc070 con 0x7fbc900047c0 2024-03-23T11:24:00.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.741+0000 7fbcad400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbca81f6450 0x7fbca8142ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:00.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.741+0000 7fbcad400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbca81f6450 0x7fbca8142ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48676/0 (socket says 172.21.15.62:48676) 2024-03-23T11:24:00.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.741+0000 7fbcad400640 1 -- 172.21.15.62:0/2963753540 learned_addr learned my addr 172.21.15.62:0/2963753540 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.742+0000 7fbcade00640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc900047c0 0x7fbca80cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.742+0000 7fbcae800640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbca815fa10 0x7fbca815fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.742+0000 7fbcad400640 1 -- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc900047c0 msgr2=0x7fbca80cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.742+0000 7fbcad400640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc900047c0 0x7fbca80cae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.742+0000 7fbcad400640 1 -- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbca815fa10 msgr2=0x7fbca815fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.742+0000 7fbcad400640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbca815fa10 0x7fbca815fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.742+0000 7fbcad400640 1 -- 172.21.15.62:0/2963753540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca80cb4f0 con 0x7fbca81f6450 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.742+0000 7fbcad400640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbca81f6450 0x7fbca8142ff0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbc98000c00 tx=0x7fbc98001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.743+0000 7fbca2000640 1 -- 172.21.15.62:0/2963753540 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc980076c0 con 0x7fbca81f6450 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.743+0000 7fbca2000640 1 -- 172.21.15.62:0/2963753540 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc98007a30 con 0x7fbca81f6450 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.743+0000 7fbca2000640 1 -- 172.21.15.62:0/2963753540 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98006be0 con 0x7fbca81f6450 2024-03-23T11:24:00.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.743+0000 7fbcb0800640 1 -- 172.21.15.62:0/2963753540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca80cb7d0 con 0x7fbca81f6450 2024-03-23T11:24:00.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.743+0000 7fbcb0800640 1 -- 172.21.15.62:0/2963753540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca80cbd90 con 0x7fbca81f6450 2024-03-23T11:24:00.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.744+0000 7fbcb0800640 1 -- 172.21.15.62:0/2963753540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca8160400 con 0x7fbca81f6450 2024-03-23T11:24:00.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.745+0000 7fbca2000640 1 -- 172.21.15.62:0/2963753540 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbc98006d40 con 0x7fbca81f6450 2024-03-23T11:24:00.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.746+0000 7fbca2000640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc8c06a7a0 0x7fbc8c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:00.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.746+0000 7fbca2000640 1 -- 172.21.15.62:0/2963753540 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbc980aa700 con 0x7fbca81f6450 2024-03-23T11:24:00.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.746+0000 7fbcade00640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc8c06a7a0 0x7fbc8c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:00.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.748+0000 7fbcade00640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc8c06a7a0 0x7fbc8c06cc60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c005a90 tx=0x7fbc9c001680 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:00.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.751+0000 7fbca2000640 1 -- 172.21.15.62:0/2963753540 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbca8160400 con 0x7fbca81f6450 2024-03-23T11:24:00.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.954+0000 7fbcb0800640 1 -- 172.21.15.62:0/2963753540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbca8160400 con 0x7fbca81f6450 2024-03-23T11:24:00.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.960+0000 7fbca2000640 1 -- 172.21.15.62:0/2963753540 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbca8160400 con 0x7fbca81f6450 2024-03-23T11:24:00.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.962+0000 7fbcb0800640 1 -- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc8c06a7a0 msgr2=0x7fbc8c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:00.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.962+0000 7fbcb0800640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc8c06a7a0 0x7fbc8c06cc60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c005a90 tx=0x7fbc9c001680 comp rx=0 tx=0).stop 2024-03-23T11:24:00.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.963+0000 7fbcb0800640 1 -- 172.21.15.62:0/2963753540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbca81f6450 msgr2=0x7fbca8142ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:00.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.963+0000 7fbcb0800640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbca81f6450 0x7fbca8142ff0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbc98000c00 tx=0x7fbc98001120 comp rx=0 tx=0).stop 2024-03-23T11:24:00.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.963+0000 7fbcb0800640 1 -- 172.21.15.62:0/2963753540 shutdown_connections 2024-03-23T11:24:00.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.963+0000 7fbcb0800640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc8c06a7a0 0x7fbc8c06cc60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.963+0000 7fbcb0800640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbca81f6450 0x7fbca8142ff0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.963+0000 7fbcb0800640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbca815fa10 0x7fbca815fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.963+0000 7fbcb0800640 1 --2- 172.21.15.62:0/2963753540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc900047c0 0x7fbca80cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:00.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.963+0000 7fbcb0800640 1 -- 172.21.15.62:0/2963753540 >> 172.21.15.62:0/2963753540 conn(0x7fbca80b02e0 msgr2=0x7fbca80ae250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:00.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.963+0000 7fbcb0800640 1 -- 172.21.15.62:0/2963753540 shutdown_connections 2024-03-23T11:24:00.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:00.963+0000 7fbcb0800640 1 -- 172.21.15.62:0/2963753540 wait complete. 2024-03-23T11:24:00.977 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-03-23T11:24:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.197+0000 7fc0bb200640 1 Processor -- start 2024-03-23T11:24:01.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.198+0000 7fc0bb200640 1 -- start start 2024-03-23T11:24:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.199+0000 7fc0bb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0d0d40 0x7fc0bc0d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.199+0000 7fc0bb200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc0bc0b24e0 0x7fc0bc0b2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.199+0000 7fc0bb200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0bc0ce870 con 0x7fc0bc15ca00 2024-03-23T11:24:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.199+0000 7fc0bb200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0bc0d1700 con 0x7fc0bc0b24e0 2024-03-23T11:24:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.199+0000 7fc0bb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc0b2eb0 con 0x7fc0bc0d0d40 2024-03-23T11:24:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.200+0000 7fc0b9400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0d0d40 0x7fc0bc0d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.200+0000 7fc0b9400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0d0d40 0x7fc0bc0d11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:45042/0 (socket says 172.21.15.62:45042) 2024-03-23T11:24:01.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.200+0000 7fc0b9400640 1 -- 172.21.15.62:0/952009133 learned_addr learned my addr 172.21.15.62:0/952009133 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:01.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.201+0000 7fc0ba800640 1 --2- 172.21.15.62:0/952009133 >> v2:172.21.15.120:3300/0 conn(0x7fc0bc0b24e0 0x7fc0bc0b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:01.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.201+0000 7fc0b9400640 1 -- 172.21.15.62:0/952009133 >> v2:172.21.15.120:3300/0 conn(0x7fc0bc0b24e0 msgr2=0x7fc0bc0b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:01.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.201+0000 7fc0b9400640 1 --2- 172.21.15.62:0/952009133 >> v2:172.21.15.120:3300/0 conn(0x7fc0bc0b24e0 0x7fc0bc0b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:01.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.201+0000 7fc0b9400640 1 -- 172.21.15.62:0/952009133 >> v1:172.21.15.120:6789/0 conn(0x7fc0bc15ca00 legacy=0x7fc0bc0ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.201+0000 7fc0b9400640 1 -- 172.21.15.62:0/952009133 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc0cfe40 con 0x7fc0bc0d0d40 2024-03-23T11:24:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.202+0000 7fc0b9400640 1 --2- 172.21.15.62:0/952009133 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0d0d40 0x7fc0bc0d11c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8009870 tx=0x7fc0a802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84fe680120c4e740 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:01.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.203+0000 7fc0b8a00640 1 -- 172.21.15.62:0/952009133 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8030840 con 0x7fc0bc0d0d40 2024-03-23T11:24:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.203+0000 7fc0b8a00640 1 -- 172.21.15.62:0/952009133 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a80309a0 con 0x7fc0bc0d0d40 2024-03-23T11:24:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.203+0000 7fc0b8a00640 1 -- 172.21.15.62:0/952009133 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8030d30 con 0x7fc0bc0d0d40 2024-03-23T11:24:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.204+0000 7fc0bb200640 1 -- 172.21.15.62:0/952009133 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0d0d40 msgr2=0x7fc0bc0d11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.204+0000 7fc0bb200640 1 --2- 172.21.15.62:0/952009133 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0d0d40 0x7fc0bc0d11c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8009870 tx=0x7fc0a802b980 comp rx=0 tx=0).stop 2024-03-23T11:24:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.204+0000 7fc0bb200640 1 -- 172.21.15.62:0/952009133 shutdown_connections 2024-03-23T11:24:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.204+0000 7fc0bb200640 1 --2- 172.21.15.62:0/952009133 >> v2:172.21.15.120:3300/0 conn(0x7fc0bc0b24e0 0x7fc0bc0b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.204+0000 7fc0bb200640 1 --2- 172.21.15.62:0/952009133 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0d0d40 0x7fc0bc0d11c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.204+0000 7fc0bb200640 1 -- 172.21.15.62:0/952009133 >> 172.21.15.62:0/952009133 conn(0x7fc0bc09ba10 msgr2=0x7fc0bc089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.204+0000 7fc0bb200640 1 -- 172.21.15.62:0/952009133 shutdown_connections 2024-03-23T11:24:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.204+0000 7fc0bb200640 1 -- 172.21.15.62:0/952009133 wait complete. 2024-03-23T11:24:01.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.206+0000 7fc0bb200640 1 Processor -- start 2024-03-23T11:24:01.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.206+0000 7fc0bb200640 1 -- start start 2024-03-23T11:24:01.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.207+0000 7fc0bb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0b24e0 0x7fc0bc0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:01.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.207+0000 7fc0bb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0bc0d0d40 0x7fc0bc1f1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:01.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.207+0000 7fc0bb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0bc15ca00 0x7fc0bc0b7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.207+0000 7fc0bb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc0b16b0 con 0x7fc0bc15ca00 2024-03-23T11:24:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.207+0000 7fc0bb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc157d10 con 0x7fc0bc0d0d40 2024-03-23T11:24:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.207+0000 7fc0bb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc0b0820 con 0x7fc0bc0b24e0 2024-03-23T11:24:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.207+0000 7fc0ba800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0bc15ca00 0x7fc0bc0b7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.207+0000 7fc0ba800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0bc15ca00 0x7fc0bc0b7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48678/0 (socket says 172.21.15.62:48678) 2024-03-23T11:24:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.207+0000 7fc0ba800640 1 -- 172.21.15.62:0/2836445923 learned_addr learned my addr 172.21.15.62:0/2836445923 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.207+0000 7fc0b9e00640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0b24e0 0x7fc0bc0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.208+0000 7fc0b9400640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0bc0d0d40 0x7fc0bc1f1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.208+0000 7fc0ba800640 1 -- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0b24e0 msgr2=0x7fc0bc0b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.208+0000 7fc0ba800640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0b24e0 0x7fc0bc0b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.208+0000 7fc0ba800640 1 -- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0bc0d0d40 msgr2=0x7fc0bc1f1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.208+0000 7fc0ba800640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0bc0d0d40 0x7fc0bc1f1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.208+0000 7fc0ba800640 1 -- 172.21.15.62:0/2836445923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a80094d0 con 0x7fc0bc15ca00 2024-03-23T11:24:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.208+0000 7fc0ba800640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0bc15ca00 0x7fc0bc0b7840 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4007e60 tx=0x7fc0b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.208+0000 7fc0b2000640 1 -- 172.21.15.62:0/2836445923 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4014420 con 0x7fc0bc15ca00 2024-03-23T11:24:01.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.208+0000 7fc0b2000640 1 -- 172.21.15.62:0/2836445923 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b40145a0 con 0x7fc0bc15ca00 2024-03-23T11:24:01.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.208+0000 7fc0b2000640 1 -- 172.21.15.62:0/2836445923 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4014950 con 0x7fc0bc15ca00 2024-03-23T11:24:01.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.208+0000 7fc0bb200640 1 -- 172.21.15.62:0/2836445923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0bc0b0a30 con 0x7fc0bc15ca00 2024-03-23T11:24:01.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.209+0000 7fc0bb200640 1 -- 172.21.15.62:0/2836445923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0bc0b0ff0 con 0x7fc0bc15ca00 2024-03-23T11:24:01.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.210+0000 7fc0bb200640 1 -- 172.21.15.62:0/2836445923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0bc0b17f0 con 0x7fc0bc15ca00 2024-03-23T11:24:01.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.211+0000 7fc0b2000640 1 -- 172.21.15.62:0/2836445923 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc0b4014ab0 con 0x7fc0bc15ca00 2024-03-23T11:24:01.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.211+0000 7fc0b2000640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc09c06a7d0 0x7fc09c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:01.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.211+0000 7fc0b2000640 1 -- 172.21.15.62:0/2836445923 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc0b40893b0 con 0x7fc0bc15ca00 2024-03-23T11:24:01.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.212+0000 7fc0b9e00640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc09c06a7d0 0x7fc09c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:01.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.213+0000 7fc0b9e00640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc09c06a7d0 0x7fc09c06cc90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac000fb0 tx=0x7fc0ac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:01.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.216+0000 7fc0b2000640 1 -- 172.21.15.62:0/2836445923 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0bc0b17f0 con 0x7fc0bc15ca00 2024-03-23T11:24:01.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:01.484+0000 7fc0bb200640 1 -- 172.21.15.62:0/2836445923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) v1 -- 0x7fc0bc0b17f0 con 0x7fc0bc15ca00 2024-03-23T11:24:02.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.489+0000 7fc0b2000640 1 -- 172.21.15.62:0/2836445923 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v259) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc0bc0b17f0 con 0x7fc0bc15ca00 2024-03-23T11:24:02.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.492+0000 7fc0bb200640 1 -- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc09c06a7d0 msgr2=0x7fc09c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:02.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.492+0000 7fc0bb200640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc09c06a7d0 0x7fc09c06cc90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac000fb0 tx=0x7fc0ac009040 comp rx=0 tx=0).stop 2024-03-23T11:24:02.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.492+0000 7fc0bb200640 1 -- 172.21.15.62:0/2836445923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0bc15ca00 msgr2=0x7fc0bc0b7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:02.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.492+0000 7fc0bb200640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0bc15ca00 0x7fc0bc0b7840 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4007e60 tx=0x7fc0b400e4b0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.492+0000 7fc0bb200640 1 -- 172.21.15.62:0/2836445923 shutdown_connections 2024-03-23T11:24:02.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.493+0000 7fc0bb200640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc09c06a7d0 0x7fc09c06cc90 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.493+0000 7fc0bb200640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0bc15ca00 0x7fc0bc0b7840 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.493+0000 7fc0bb200640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0bc0d0d40 0x7fc0bc1f1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.493+0000 7fc0bb200640 1 --2- 172.21.15.62:0/2836445923 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0bc0b24e0 0x7fc0bc0b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.493+0000 7fc0bb200640 1 -- 172.21.15.62:0/2836445923 >> 172.21.15.62:0/2836445923 conn(0x7fc0bc09ba10 msgr2=0x7fc0bc072c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:02.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.493+0000 7fc0bb200640 1 -- 172.21.15.62:0/2836445923 shutdown_connections 2024-03-23T11:24:02.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.493+0000 7fc0bb200640 1 -- 172.21.15.62:0/2836445923 wait complete. 2024-03-23T11:24:02.513 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-03-23T11:24:02.514 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-03-23T11:24:02.514 INFO:tasks.cephfs_test_runner:test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-23T11:24:02.514 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-03-23T11:24:02.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.741+0000 7fd37de00640 1 Processor -- start 2024-03-23T11:24:02.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.741+0000 7fd37de00640 1 -- start start 2024-03-23T11:24:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.742+0000 7fd37de00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd378156920 0x7fd3781594c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.742+0000 7fd37de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd378156d00 0x7fd37806cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.742+0000 7fd37de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd37806de90 0x7fd37806d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.742+0000 7fd37de00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3780cd3d0 con 0x7fd37806de90 2024-03-23T11:24:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.742+0000 7fd37de00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3780cd6b0 con 0x7fd378156920 2024-03-23T11:24:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.742+0000 7fd37de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780cd990 con 0x7fd378156d00 2024-03-23T11:24:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.743+0000 7fd377400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd37806de90 0x7fd37806d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.743+0000 7fd377400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd37806de90 0x7fd37806d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48684/0 (socket says 172.21.15.62:48684) 2024-03-23T11:24:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.743+0000 7fd377400640 1 -- 172.21.15.62:0/2743137996 learned_addr learned my addr 172.21.15.62:0/2743137996 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.743+0000 7fd376000640 1 --2- 172.21.15.62:0/2743137996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd378156d00 0x7fd37806cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.743+0000 7fd376a00640 1 --2- 172.21.15.62:0/2743137996 >> v2:172.21.15.120:3300/0 conn(0x7fd378156920 0x7fd3781594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:02.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.743+0000 7fd377400640 1 -- 172.21.15.62:0/2743137996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd378156d00 msgr2=0x7fd37806cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:02.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.743+0000 7fd377400640 1 --2- 172.21.15.62:0/2743137996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd378156d00 0x7fd37806cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.743+0000 7fd377400640 1 -- 172.21.15.62:0/2743137996 >> v2:172.21.15.120:3300/0 conn(0x7fd378156920 msgr2=0x7fd3781594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:02.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.743+0000 7fd377400640 1 --2- 172.21.15.62:0/2743137996 >> v2:172.21.15.120:3300/0 conn(0x7fd378156920 0x7fd3781594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.743+0000 7fd377400640 1 -- 172.21.15.62:0/2743137996 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3780cdc70 con 0x7fd37806de90 2024-03-23T11:24:02.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.745+0000 7fd377400640 1 --2- 172.21.15.62:0/2743137996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd37806de90 0x7fd37806d0d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd3780ae220 tx=0x7fd37002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62245120f42e4ce server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:02.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.745+0000 7fd375600640 1 -- 172.21.15.62:0/2743137996 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37002c9e0 con 0x7fd37806de90 2024-03-23T11:24:02.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.745+0000 7fd375600640 1 -- 172.21.15.62:0/2743137996 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37002cb40 con 0x7fd37806de90 2024-03-23T11:24:02.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.745+0000 7fd375600640 1 -- 172.21.15.62:0/2743137996 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3700343f0 con 0x7fd37806de90 2024-03-23T11:24:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.746+0000 7fd37de00640 1 -- 172.21.15.62:0/2743137996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd37806de90 msgr2=0x7fd37806d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.746+0000 7fd37de00640 1 --2- 172.21.15.62:0/2743137996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd37806de90 0x7fd37806d0d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd3780ae220 tx=0x7fd37002be40 comp rx=0 tx=0).stop 2024-03-23T11:24:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.746+0000 7fd37de00640 1 -- 172.21.15.62:0/2743137996 shutdown_connections 2024-03-23T11:24:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.746+0000 7fd37de00640 1 --2- 172.21.15.62:0/2743137996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd37806de90 0x7fd37806d0d0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.746+0000 7fd37de00640 1 --2- 172.21.15.62:0/2743137996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd378156d00 0x7fd37806cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.746+0000 7fd37de00640 1 --2- 172.21.15.62:0/2743137996 >> v2:172.21.15.120:3300/0 conn(0x7fd378156920 0x7fd3781594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.746+0000 7fd37de00640 1 -- 172.21.15.62:0/2743137996 >> 172.21.15.62:0/2743137996 conn(0x7fd3780b02e0 msgr2=0x7fd3780b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.746+0000 7fd37de00640 1 -- 172.21.15.62:0/2743137996 shutdown_connections 2024-03-23T11:24:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.746+0000 7fd37de00640 1 -- 172.21.15.62:0/2743137996 wait complete. 2024-03-23T11:24:02.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.747+0000 7fd37de00640 1 Processor -- start 2024-03-23T11:24:02.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.748+0000 7fd37de00640 1 -- start start 2024-03-23T11:24:02.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd37de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd37806de90 0x7fd378145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:02.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd37de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd378156920 0x7fd3781433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:02.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd37de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd378156d00 0x7fd378144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:02.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd37de00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd378144840 con 0x7fd378156d00 2024-03-23T11:24:02.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd37de00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd378143930 con 0x7fd37806de90 2024-03-23T11:24:02.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd37de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd378143bc0 con 0x7fd378156920 2024-03-23T11:24:02.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd377400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd378156d00 0x7fd378144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:02.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd377400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd378156d00 0x7fd378144c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48700/0 (socket says 172.21.15.62:48700) 2024-03-23T11:24:02.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd377400640 1 -- 172.21.15.62:0/3806781298 learned_addr learned my addr 172.21.15.62:0/3806781298 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:02.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd376a00640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd37806de90 0x7fd378145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:02.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd377400640 1 -- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd378156920 msgr2=0x7fd3781433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:02.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd377400640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd378156920 0x7fd3781433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd377400640 1 -- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd37806de90 msgr2=0x7fd378145fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:02.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd377400640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd37806de90 0x7fd378145fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:02.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.749+0000 7fd377400640 1 -- 172.21.15.62:0/3806781298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd370009b30 con 0x7fd378156d00 2024-03-23T11:24:02.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.750+0000 7fd377400640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd378156d00 0x7fd378144c60 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fd370009e50 tx=0x7fd370004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:02.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.750+0000 7fd36f400640 1 -- 172.21.15.62:0/3806781298 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd370009510 con 0x7fd378156d00 2024-03-23T11:24:02.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.750+0000 7fd36f400640 1 -- 172.21.15.62:0/3806781298 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37003e450 con 0x7fd378156d00 2024-03-23T11:24:02.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.750+0000 7fd37de00640 1 -- 172.21.15.62:0/3806781298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd378143ea0 con 0x7fd378156d00 2024-03-23T11:24:02.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.750+0000 7fd36f400640 1 -- 172.21.15.62:0/3806781298 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37003e800 con 0x7fd378156d00 2024-03-23T11:24:02.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.750+0000 7fd37de00640 1 -- 172.21.15.62:0/3806781298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd378144460 con 0x7fd378156d00 2024-03-23T11:24:02.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.751+0000 7fd37de00640 1 -- 172.21.15.62:0/3806781298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd378144980 con 0x7fd378156d00 2024-03-23T11:24:02.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.752+0000 7fd36f400640 1 -- 172.21.15.62:0/3806781298 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd37003e960 con 0x7fd378156d00 2024-03-23T11:24:02.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.753+0000 7fd36f400640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35806a7a0 0x7fd35806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:02.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.753+0000 7fd36f400640 1 -- 172.21.15.62:0/3806781298 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd3700ab840 con 0x7fd378156d00 2024-03-23T11:24:02.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.754+0000 7fd376a00640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35806a7a0 0x7fd35806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:02.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.755+0000 7fd376a00640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35806a7a0 0x7fd35806cc60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd368006fd0 tx=0x7fd368008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:02.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:02.757+0000 7fd36f400640 1 -- 172.21.15.62:0/3806781298 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd378144980 con 0x7fd378156d00 2024-03-23T11:24:03.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.026+0000 7fd37de00640 1 -- 172.21.15.62:0/3806781298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) v1 -- 0x7fd378144980 con 0x7fd378156d00 2024-03-23T11:24:03.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.499+0000 7fd36f400640 1 -- 172.21.15.62:0/3806781298 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v260) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd378144980 con 0x7fd378156d00 2024-03-23T11:24:03.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 -- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35806a7a0 msgr2=0x7fd35806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:03.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35806a7a0 0x7fd35806cc60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd368006fd0 tx=0x7fd368008040 comp rx=0 tx=0).stop 2024-03-23T11:24:03.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 -- 172.21.15.62:0/3806781298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd378156d00 msgr2=0x7fd378144c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:03.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd378156d00 0x7fd378144c60 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fd370009e50 tx=0x7fd370004a40 comp rx=0 tx=0).stop 2024-03-23T11:24:03.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 -- 172.21.15.62:0/3806781298 shutdown_connections 2024-03-23T11:24:03.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35806a7a0 0x7fd35806cc60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd378156d00 0x7fd378144c60 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd378156920 0x7fd3781433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 --2- 172.21.15.62:0/3806781298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd37806de90 0x7fd378145fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 -- 172.21.15.62:0/3806781298 >> 172.21.15.62:0/3806781298 conn(0x7fd3780b02e0 msgr2=0x7fd3781559c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 -- 172.21.15.62:0/3806781298 shutdown_connections 2024-03-23T11:24:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.503+0000 7fd37de00640 1 -- 172.21.15.62:0/3806781298 wait complete. 2024-03-23T11:24:03.522 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T11:24:03.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.754+0000 7fcebd800640 1 Processor -- start 2024-03-23T11:24:03.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.755+0000 7fcebd800640 1 -- start start 2024-03-23T11:24:03.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.756+0000 7fcebd800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb815caf0 0x7fceb80b2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:03.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.756+0000 7fcebd800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fceb80cc990 0x7fceb8069c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:03.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.756+0000 7fcebd800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb80c99b0 con 0x7fceb80ce5f0 2024-03-23T11:24:03.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.756+0000 7fcebd800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb80b2790 con 0x7fceb80cc990 2024-03-23T11:24:03.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.756+0000 7fcebd800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb806a560 con 0x7fceb815caf0 2024-03-23T11:24:03.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.757+0000 7fceb7e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb815caf0 0x7fceb80b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:03.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.757+0000 7fceb7e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb815caf0 0x7fceb80b2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:45064/0 (socket says 172.21.15.62:45064) 2024-03-23T11:24:03.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.757+0000 7fceb7e00640 1 -- 172.21.15.62:0/3989681387 learned_addr learned my addr 172.21.15.62:0/3989681387 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:03.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.757+0000 7fceafe00640 1 --2- 172.21.15.62:0/3989681387 >> v2:172.21.15.120:3300/0 conn(0x7fceb80cc990 0x7fceb8069c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:03.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.758+0000 7fceb7e00640 1 -- 172.21.15.62:0/3989681387 >> v2:172.21.15.120:3300/0 conn(0x7fceb80cc990 msgr2=0x7fceb8069c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:03.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.758+0000 7fceb7e00640 1 --2- 172.21.15.62:0/3989681387 >> v2:172.21.15.120:3300/0 conn(0x7fceb80cc990 0x7fceb8069c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.758+0000 7fceb7e00640 1 -- 172.21.15.62:0/3989681387 >> v1:172.21.15.120:6789/0 conn(0x7fceb80ce5f0 legacy=0x7fceb80cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:03.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.758+0000 7fceb7e00640 1 -- 172.21.15.62:0/3989681387 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb80c9c00 con 0x7fceb815caf0 2024-03-23T11:24:03.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.759+0000 7fceb7e00640 1 --2- 172.21.15.62:0/3989681387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb815caf0 0x7fceb80b2250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009870 tx=0x7fcea802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=563dd7b79344a09a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:03.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.760+0000 7fceb7400640 1 -- 172.21.15.62:0/3989681387 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8031640 con 0x7fceb815caf0 2024-03-23T11:24:03.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.760+0000 7fceb7400640 1 -- 172.21.15.62:0/3989681387 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea80317a0 con 0x7fceb815caf0 2024-03-23T11:24:03.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.760+0000 7fceb7400640 1 -- 172.21.15.62:0/3989681387 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8031b30 con 0x7fceb815caf0 2024-03-23T11:24:03.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.760+0000 7fcebd800640 1 -- 172.21.15.62:0/3989681387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb815caf0 msgr2=0x7fceb80b2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:03.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.760+0000 7fcebd800640 1 --2- 172.21.15.62:0/3989681387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb815caf0 0x7fceb80b2250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009870 tx=0x7fcea802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.761+0000 7fcebd800640 1 -- 172.21.15.62:0/3989681387 shutdown_connections 2024-03-23T11:24:03.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.761+0000 7fcebd800640 1 --2- 172.21.15.62:0/3989681387 >> v2:172.21.15.120:3300/0 conn(0x7fceb80cc990 0x7fceb8069c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.761+0000 7fcebd800640 1 --2- 172.21.15.62:0/3989681387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb815caf0 0x7fceb80b2250 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.761+0000 7fcebd800640 1 -- 172.21.15.62:0/3989681387 >> 172.21.15.62:0/3989681387 conn(0x7fceb80b0f50 msgr2=0x7fceb80b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.761+0000 7fcebd800640 1 -- 172.21.15.62:0/3989681387 shutdown_connections 2024-03-23T11:24:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.761+0000 7fcebd800640 1 -- 172.21.15.62:0/3989681387 wait complete. 2024-03-23T11:24:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.762+0000 7fcebd800640 1 Processor -- start 2024-03-23T11:24:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.763+0000 7fcebd800640 1 -- start start 2024-03-23T11:24:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.763+0000 7fcebd800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fceb80cc990 0x7fceb80c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fcebd800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb80ce5f0 0x7fceb80b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fcebd800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb815caf0 0x7fceb80c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fcebd800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb80c3ac0 con 0x7fceb80cc990 2024-03-23T11:24:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fcebd800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb80c3da0 con 0x7fceb815caf0 2024-03-23T11:24:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fcebd800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb80c4080 con 0x7fceb80ce5f0 2024-03-23T11:24:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fceb7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fceb80cc990 0x7fceb80c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fceb7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fceb80cc990 0x7fceb80c4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48704/0 (socket says 172.21.15.62:48704) 2024-03-23T11:24:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fceb7e00640 1 -- 172.21.15.62:0/1316076945 learned_addr learned my addr 172.21.15.62:0/1316076945 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fcebce00640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb815caf0 0x7fceb80c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fceafe00640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb80ce5f0 0x7fceb80b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fceb7e00640 1 -- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb80ce5f0 msgr2=0x7fceb80b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fceb7e00640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb80ce5f0 0x7fceb80b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fceb7e00640 1 -- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb815caf0 msgr2=0x7fceb80c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fceb7e00640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb815caf0 0x7fceb80c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.764+0000 7fceb7e00640 1 -- 172.21.15.62:0/1316076945 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea80094d0 con 0x7fceb80cc990 2024-03-23T11:24:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.765+0000 7fcebce00640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb815caf0 0x7fceb80c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.765+0000 7fceb7e00640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fceb80cc990 0x7fceb80c4b60 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fcea80090b0 tx=0x7fcea80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.765+0000 7fceb4c00640 1 -- 172.21.15.62:0/1316076945 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea803ac60 con 0x7fceb80cc990 2024-03-23T11:24:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.765+0000 7fceb4c00640 1 -- 172.21.15.62:0/1316076945 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea803ade0 con 0x7fceb80cc990 2024-03-23T11:24:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.765+0000 7fceb4c00640 1 -- 172.21.15.62:0/1316076945 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8044630 con 0x7fceb80cc990 2024-03-23T11:24:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.765+0000 7fcebd800640 1 -- 172.21.15.62:0/1316076945 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb80c4360 con 0x7fceb80cc990 2024-03-23T11:24:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.765+0000 7fcebd800640 1 -- 172.21.15.62:0/1316076945 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb80c50a0 con 0x7fceb80cc990 2024-03-23T11:24:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.767+0000 7fcebd800640 1 -- 172.21.15.62:0/1316076945 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb80c41c0 con 0x7fceb80cc990 2024-03-23T11:24:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.768+0000 7fceb4c00640 1 -- 172.21.15.62:0/1316076945 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fcea802f050 con 0x7fceb80cc990 2024-03-23T11:24:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.769+0000 7fceb4c00640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9406a7a0 0x7fce9406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.769+0000 7fceb4c00640 1 -- 172.21.15.62:0/1316076945 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcea80aa9a0 con 0x7fceb80cc990 2024-03-23T11:24:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.770+0000 7fceafe00640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9406a7a0 0x7fce9406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:03.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.771+0000 7fceafe00640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9406a7a0 0x7fce9406cc60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fceb80c5f90 tx=0x7fcea0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:03.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.774+0000 7fceb4c00640 1 -- 172.21.15.62:0/1316076945 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fceb80c41c0 con 0x7fceb80cc990 2024-03-23T11:24:03.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.971+0000 7fcebd800640 1 -- 172.21.15.62:0/1316076945 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fceb80c41c0 con 0x7fceb80cc990 2024-03-23T11:24:03.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.979+0000 7fceb4c00640 1 -- 172.21.15.62:0/1316076945 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fceb80c41c0 con 0x7fceb80cc990 2024-03-23T11:24:03.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.982+0000 7fcebd800640 1 -- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9406a7a0 msgr2=0x7fce9406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:03.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.982+0000 7fcebd800640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9406a7a0 0x7fce9406cc60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fceb80c5f90 tx=0x7fcea0008040 comp rx=0 tx=0).stop 2024-03-23T11:24:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.982+0000 7fcebd800640 1 -- 172.21.15.62:0/1316076945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fceb80cc990 msgr2=0x7fceb80c4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.982+0000 7fcebd800640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fceb80cc990 0x7fceb80c4b60 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fcea80090b0 tx=0x7fcea80046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.982+0000 7fcebd800640 1 -- 172.21.15.62:0/1316076945 shutdown_connections 2024-03-23T11:24:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.982+0000 7fcebd800640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fce9406a7a0 0x7fce9406cc60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.982+0000 7fcebd800640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fceb815caf0 0x7fceb80c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.982+0000 7fcebd800640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fceb80ce5f0 0x7fceb80b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.982+0000 7fcebd800640 1 --2- 172.21.15.62:0/1316076945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fceb80cc990 0x7fceb80c4b60 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.982+0000 7fcebd800640 1 -- 172.21.15.62:0/1316076945 >> 172.21.15.62:0/1316076945 conn(0x7fceb80b0f50 msgr2=0x7fceb806d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.983+0000 7fcebd800640 1 -- 172.21.15.62:0/1316076945 shutdown_connections 2024-03-23T11:24:03.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:03.983+0000 7fcebd800640 1 -- 172.21.15.62:0/1316076945 wait complete. 2024-03-23T11:24:03.997 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:24:03.997 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:24:04.103 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:24:04.223 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:24:04.223 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:24:04.224 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:24:04.284 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:24:04.284 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:24:04.284 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:24:04.342 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:24:04.342 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:24:04.342 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:24:04.401 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:24:04.401 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:24:04.401 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:24:04.458 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:24:04.458 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T11:24:04.458 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:24:04.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.675+0000 7fcbb9400640 1 Processor -- start 2024-03-23T11:24:04.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.676+0000 7fcbb9400640 1 -- start start 2024-03-23T11:24:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.677+0000 7fcbb9400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbb415cdc0 0x7fcbb40d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.677+0000 7fcbb9400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcbb40cf650 0x7fcbb40cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.677+0000 7fcbb9400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb406bfc0 con 0x7fcbb415c9e0 2024-03-23T11:24:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.677+0000 7fcbb9400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbb40d0020 con 0x7fcbb40cf650 2024-03-23T11:24:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.677+0000 7fcbb9400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40ce140 con 0x7fcbb415cdc0 2024-03-23T11:24:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.677+0000 7fcbb3200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcbb40cf650 0x7fcbb40cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.677+0000 7fcbb3200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcbb40cf650 0x7fcbb40cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48712/0 (socket says 172.21.15.62:48712) 2024-03-23T11:24:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.678+0000 7fcbb3200640 1 -- 172.21.15.62:0/3983532662 learned_addr learned my addr 172.21.15.62:0/3983532662 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.678+0000 7fcbb1e00640 1 --2- 172.21.15.62:0/3983532662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbb415cdc0 0x7fcbb40d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.678+0000 7fcbb3200640 1 -- 172.21.15.62:0/3983532662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbb415cdc0 msgr2=0x7fcbb40d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.678+0000 7fcbb3200640 1 --2- 172.21.15.62:0/3983532662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbb415cdc0 0x7fcbb40d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.678+0000 7fcbb3200640 1 -- 172.21.15.62:0/3983532662 >> v1:172.21.15.120:6789/0 conn(0x7fcbb415c9e0 legacy=0x7fcbb41607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.678+0000 7fcbb3200640 1 -- 172.21.15.62:0/3983532662 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb40b5260 con 0x7fcbb40cf650 2024-03-23T11:24:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.678+0000 7fcbb1e00640 1 --2- 172.21.15.62:0/3983532662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbb415cdc0 0x7fcbb40d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.679+0000 7fcbb3200640 1 --2- 172.21.15.62:0/3983532662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb40cf650 0x7fcbb40cfae0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009870 tx=0x7fcbac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e9f743b92b59f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.679+0000 7fcbb1400640 1 -- 172.21.15.62:0/3983532662 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac031640 con 0x7fcbb40cf650 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.679+0000 7fcbb1400640 1 -- 172.21.15.62:0/3983532662 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbac0317a0 con 0x7fcbb40cf650 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.679+0000 7fcbb1400640 1 -- 172.21.15.62:0/3983532662 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac031b30 con 0x7fcbb40cf650 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.680+0000 7fcbb9400640 1 -- 172.21.15.62:0/3983532662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb40cf650 msgr2=0x7fcbb40cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.680+0000 7fcbb9400640 1 --2- 172.21.15.62:0/3983532662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb40cf650 0x7fcbb40cfae0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009870 tx=0x7fcbac02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.680+0000 7fcbb9400640 1 -- 172.21.15.62:0/3983532662 shutdown_connections 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.680+0000 7fcbb9400640 1 --2- 172.21.15.62:0/3983532662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb40cf650 0x7fcbb40cfae0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.680+0000 7fcbb9400640 1 --2- 172.21.15.62:0/3983532662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbb415cdc0 0x7fcbb40d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.680+0000 7fcbb9400640 1 -- 172.21.15.62:0/3983532662 >> 172.21.15.62:0/3983532662 conn(0x7fcbb4071570 msgr2=0x7fcbb4071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.680+0000 7fcbb9400640 1 -- 172.21.15.62:0/3983532662 shutdown_connections 2024-03-23T11:24:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.680+0000 7fcbb9400640 1 -- 172.21.15.62:0/3983532662 wait complete. 2024-03-23T11:24:04.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.682+0000 7fcbb9400640 1 Processor -- start 2024-03-23T11:24:04.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.682+0000 7fcbb9400640 1 -- start start 2024-03-23T11:24:04.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.683+0000 7fcbb9400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbb40cf650 0x7fcbb40b7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:04.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.683+0000 7fcbb9400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbb415c9e0 0x7fcbb4153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:04.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.683+0000 7fcbb9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb415cdc0 0x7fcbb4155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:04.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.683+0000 7fcbb9400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40cc830 con 0x7fcbb415cdc0 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.683+0000 7fcbb9400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb41541b0 con 0x7fcbb40cf650 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.683+0000 7fcbb9400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40c9570 con 0x7fcbb415c9e0 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.683+0000 7fcbb3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb415cdc0 0x7fcbb4155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.683+0000 7fcbb3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb415cdc0 0x7fcbb4155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48728/0 (socket says 172.21.15.62:48728) 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.683+0000 7fcbb3200640 1 -- 172.21.15.62:0/1073969070 learned_addr learned my addr 172.21.15.62:0/1073969070 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbb2800640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbb40cf650 0x7fcbb40b7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbb1e00640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbb415c9e0 0x7fcbb4153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbb3200640 1 -- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbb415c9e0 msgr2=0x7fcbb4153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbb3200640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbb415c9e0 0x7fcbb4153c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbb3200640 1 -- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbb40cf650 msgr2=0x7fcbb40b7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbb3200640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbb40cf650 0x7fcbb40b7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbb3200640 1 -- 172.21.15.62:0/1073969070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0094d0 con 0x7fcbb415cdc0 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbb2800640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbb40cf650 0x7fcbb40b7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbb3200640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb415cdc0 0x7fcbb4155e40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009870 tx=0x7fcbac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbaaa00640 1 -- 172.21.15.62:0/1073969070 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac03ac60 con 0x7fcbb415cdc0 2024-03-23T11:24:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbaaa00640 1 -- 172.21.15.62:0/1073969070 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbac03ade0 con 0x7fcbb415cdc0 2024-03-23T11:24:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbaaa00640 1 -- 172.21.15.62:0/1073969070 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac044630 con 0x7fcbb415cdc0 2024-03-23T11:24:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.684+0000 7fcbb9400640 1 -- 172.21.15.62:0/1073969070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb40cc270 con 0x7fcbb415cdc0 2024-03-23T11:24:04.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.685+0000 7fcbb9400640 1 -- 172.21.15.62:0/1073969070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb40b7790 con 0x7fcbb415cdc0 2024-03-23T11:24:04.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.686+0000 7fcbb9400640 1 -- 172.21.15.62:0/1073969070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb40b7600 con 0x7fcbb415cdc0 2024-03-23T11:24:04.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.687+0000 7fcbaaa00640 1 -- 172.21.15.62:0/1073969070 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fcbac02f050 con 0x7fcbb415cdc0 2024-03-23T11:24:04.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.688+0000 7fcbaaa00640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcb9006a7a0 0x7fcb9006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:04.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.688+0000 7fcbaaa00640 1 -- 172.21.15.62:0/1073969070 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcbac0aaae0 con 0x7fcbb415cdc0 2024-03-23T11:24:04.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.689+0000 7fcbb2800640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcb9006a7a0 0x7fcb9006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:04.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.690+0000 7fcbb2800640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcb9006a7a0 0x7fcb9006cc60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcbb415bf20 tx=0x7fcb9c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:04.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.693+0000 7fcbaaa00640 1 -- 172.21.15.62:0/1073969070 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcbac0761f0 con 0x7fcbb415cdc0 2024-03-23T11:24:04.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.933+0000 7fcbb9400640 1 -- 172.21.15.62:0/1073969070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbb40c96e0 con 0x7fcbb415cdc0 2024-03-23T11:24:04.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.934+0000 7fcbaaa00640 1 -- 172.21.15.62:0/1073969070 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fcbac035de0 con 0x7fcbb415cdc0 2024-03-23T11:24:04.930 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:04.930 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":37,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/3287045135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":3287045135},{"type":"v1","addr":"172.21.15.120:6835","nonce":3287045135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5689,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1446309136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":1446309136},{"type":"v1","addr":"172.21.15.120:6837","nonce":1446309136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6066,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/2859430143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2859430143},{"type":"v1","addr":"172.21.15.62:6834","nonce":2859430143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6069,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2005768060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2005768060},{"type":"v1","addr":"172.21.15.62:6835","nonce":2005768060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[]} 2024-03-23T11:24:04.930 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 37 2024-03-23T11:24:04.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 -- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcb9006a7a0 msgr2=0x7fcb9006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:04.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcb9006a7a0 0x7fcb9006cc60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcbb415bf20 tx=0x7fcb9c009040 comp rx=0 tx=0).stop 2024-03-23T11:24:04.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 -- 172.21.15.62:0/1073969070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb415cdc0 msgr2=0x7fcbb4155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:04.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb415cdc0 0x7fcbb4155e40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009870 tx=0x7fcbac0046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:04.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 -- 172.21.15.62:0/1073969070 shutdown_connections 2024-03-23T11:24:04.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcb9006a7a0 0x7fcb9006cc60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:04.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb415cdc0 0x7fcbb4155e40 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:04.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbb415c9e0 0x7fcbb4153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:04.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 --2- 172.21.15.62:0/1073969070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbb40cf650 0x7fcbb40b7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:04.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 -- 172.21.15.62:0/1073969070 >> 172.21.15.62:0/1073969070 conn(0x7fcbb4071570 msgr2=0x7fcbb415d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:04.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 -- 172.21.15.62:0/1073969070 shutdown_connections 2024-03-23T11:24:04.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:04.937+0000 7fcbb9400640 1 -- 172.21.15.62:0/1073969070 wait complete. 2024-03-23T11:24:04.952 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T11:24:04.952 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T11:24:04.952 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T11:24:04.952 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:24:04.952 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T11:24:04.953 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T11:24:04.953 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T11:24:04.953 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:24:04.953 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T11:24:04.953 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T11:24:04.953 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T11:24:04.953 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:24:04.953 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T11:24:04.954 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T11:24:04.954 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T11:24:04.954 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:24:04.971 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T11:24:04.971 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T11:24:04.975 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T11:24:04.975 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T11:24:04.978 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T11:24:04.978 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T11:24:04.979 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T11:24:04.979 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T11:24:05.022 INFO:tasks.ceph.mds.a:Started 2024-03-23T11:24:05.023 DEBUG:teuthology.parallel:result is None 2024-03-23T11:24:05.026 INFO:tasks.ceph.mds.d:Started 2024-03-23T11:24:05.027 DEBUG:teuthology.parallel:result is None 2024-03-23T11:24:05.029 INFO:tasks.ceph.mds.c:Started 2024-03-23T11:24:05.029 DEBUG:teuthology.parallel:result is None 2024-03-23T11:24:05.032 INFO:tasks.ceph.mds.b:Started 2024-03-23T11:24:05.032 DEBUG:teuthology.parallel:result is None 2024-03-23T11:24:05.033 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T11:24:05.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-23T11:24:05.183 INFO:tasks.ceph.mds.c.smithi062.stdout:starting mds.c at 2024-03-23T11:24:05.186 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:24:05.199 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-23T11:24:05.202 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-23T11:24:05.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.226+0000 7f9d4b000640 1 Processor -- start 2024-03-23T11:24:05.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.226+0000 7f9d4b000640 1 -- start start 2024-03-23T11:24:05.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.228+0000 7f9d4b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d440995b0 0x7f9d440ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:05.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.228+0000 7f9d4b000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9d440989d0 0x7f9d44098e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:05.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.228+0000 7f9d4b000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d440cf690 con 0x7f9d44160240 2024-03-23T11:24:05.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.228+0000 7f9d4b000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d440993a0 con 0x7f9d440989d0 2024-03-23T11:24:05.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.228+0000 7f9d4b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d440cb730 con 0x7f9d440995b0 2024-03-23T11:24:05.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.228+0000 7f9d43400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d440995b0 0x7f9d440ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:05.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.228+0000 7f9d49000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9d440989d0 0x7f9d44098e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:05.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.228+0000 7f9d49000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9d440989d0 0x7f9d44098e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:41990/0 (socket says 172.21.15.62:41990) 2024-03-23T11:24:05.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.228+0000 7f9d43400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d440995b0 0x7f9d440ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:45148/0 (socket says 172.21.15.62:45148) 2024-03-23T11:24:05.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.228+0000 7f9d49000640 1 -- 172.21.15.62:0/2341102069 learned_addr learned my addr 172.21.15.62:0/2341102069 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:05.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.229+0000 7f9d43400640 1 -- 172.21.15.62:0/2341102069 >> v2:172.21.15.120:3300/0 conn(0x7f9d440989d0 msgr2=0x7f9d44098e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:05.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.229+0000 7f9d43400640 1 --2- 172.21.15.62:0/2341102069 >> v2:172.21.15.120:3300/0 conn(0x7f9d440989d0 0x7f9d44098e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:05.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.229+0000 7f9d43400640 1 -- 172.21.15.62:0/2341102069 >> v1:172.21.15.120:6789/0 conn(0x7f9d44160240 legacy=0x7f9d4406ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:05.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.229+0000 7f9d43400640 1 -- 172.21.15.62:0/2341102069 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d440cba10 con 0x7f9d440995b0 2024-03-23T11:24:05.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.230+0000 7f9d43400640 1 --2- 172.21.15.62:0/2341102069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d440995b0 0x7f9d440ced80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9d300097a0 tx=0x7f9d3002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8adcda8fa7b0f5f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:05.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.232+0000 7f9d42a00640 1 -- 172.21.15.62:0/2341102069 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d30031780 con 0x7f9d440995b0 2024-03-23T11:24:05.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.232+0000 7f9d42a00640 1 -- 172.21.15.62:0/2341102069 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d300318e0 con 0x7f9d440995b0 2024-03-23T11:24:05.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.232+0000 7f9d42a00640 1 -- 172.21.15.62:0/2341102069 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d30031c70 con 0x7f9d440995b0 2024-03-23T11:24:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.233+0000 7f9d4b000640 1 -- 172.21.15.62:0/2341102069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d440995b0 msgr2=0x7f9d440ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.233+0000 7f9d4b000640 1 --2- 172.21.15.62:0/2341102069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d440995b0 0x7f9d440ced80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9d300097a0 tx=0x7f9d3002b820 comp rx=0 tx=0).stop 2024-03-23T11:24:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.233+0000 7f9d4b000640 1 -- 172.21.15.62:0/2341102069 shutdown_connections 2024-03-23T11:24:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.233+0000 7f9d4b000640 1 --2- 172.21.15.62:0/2341102069 >> v2:172.21.15.120:3300/0 conn(0x7f9d440989d0 0x7f9d44098e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.233+0000 7f9d4b000640 1 --2- 172.21.15.62:0/2341102069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d440995b0 0x7f9d440ced80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.233+0000 7f9d4b000640 1 -- 172.21.15.62:0/2341102069 >> 172.21.15.62:0/2341102069 conn(0x7f9d440b22c0 msgr2=0x7f9d440b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.233+0000 7f9d4b000640 1 -- 172.21.15.62:0/2341102069 shutdown_connections 2024-03-23T11:24:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.233+0000 7f9d4b000640 1 -- 172.21.15.62:0/2341102069 wait complete. 2024-03-23T11:24:05.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.235+0000 7f9d4b000640 1 Processor -- start 2024-03-23T11:24:05.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.236+0000 7f9d4b000640 1 -- start start 2024-03-23T11:24:05.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.236+0000 7f9d4b000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d440989d0 0x7f9d441f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:05.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.236+0000 7f9d4b000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d440995b0 0x7f9d440c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:05.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.236+0000 7f9d4b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d44160240 0x7f9d440ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:05.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.236+0000 7f9d4b000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d440c4430 con 0x7f9d440995b0 2024-03-23T11:24:05.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d4b000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d440ca700 con 0x7f9d440989d0 2024-03-23T11:24:05.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d4b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d440c9e60 con 0x7f9d44160240 2024-03-23T11:24:05.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d43400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d440995b0 0x7f9d440c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:05.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d49000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d44160240 0x7f9d440ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:05.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d43400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d440995b0 0x7f9d440c9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48742/0 (socket says 172.21.15.62:48742) 2024-03-23T11:24:05.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d43400640 1 -- 172.21.15.62:0/1738463191 learned_addr learned my addr 172.21.15.62:0/1738463191 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:05.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d43e00640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d440989d0 0x7f9d441f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:05.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d43400640 1 -- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d44160240 msgr2=0x7f9d440ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:05.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d43400640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d44160240 0x7f9d440ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:05.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d43400640 1 -- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d440989d0 msgr2=0x7f9d441f1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:05.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d43400640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d440989d0 0x7f9d441f1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:05.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.237+0000 7f9d43400640 1 -- 172.21.15.62:0/1738463191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d30009400 con 0x7f9d440995b0 2024-03-23T11:24:05.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.238+0000 7f9d43400640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d440995b0 0x7f9d440c9430 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f9d30034040 tx=0x7f9d300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:05.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.242+0000 7f9d3be00640 1 -- 172.21.15.62:0/1738463191 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d30032820 con 0x7f9d440995b0 2024-03-23T11:24:05.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.243+0000 7f9d3be00640 1 -- 172.21.15.62:0/1738463191 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d300329a0 con 0x7f9d440995b0 2024-03-23T11:24:05.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.243+0000 7f9d3be00640 1 -- 172.21.15.62:0/1738463191 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d30032d50 con 0x7f9d440995b0 2024-03-23T11:24:05.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.243+0000 7f9d4b000640 1 -- 172.21.15.62:0/1738463191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d440ca140 con 0x7f9d440995b0 2024-03-23T11:24:05.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.243+0000 7f9d4b000640 1 -- 172.21.15.62:0/1738463191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4414f540 con 0x7f9d440995b0 2024-03-23T11:24:05.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.244+0000 7f9d4b000640 1 -- 172.21.15.62:0/1738463191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d440c45e0 con 0x7f9d440995b0 2024-03-23T11:24:05.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.245+0000 7f9d3be00640 1 -- 172.21.15.62:0/1738463191 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9d3004d3f0 con 0x7f9d440995b0 2024-03-23T11:24:05.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.246+0000 7f9d3be00640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d2406a7f0 0x7f9d2406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:05.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.246+0000 7f9d3be00640 1 -- 172.21.15.62:0/1738463191 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9d300aa4b0 con 0x7f9d440995b0 2024-03-23T11:24:05.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.246+0000 7f9d43e00640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d2406a7f0 0x7f9d2406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:05.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.247+0000 7f9d43e00640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d2406a7f0 0x7f9d2406ccb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9d34000f60 tx=0x7f9d34009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:05.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.250+0000 7f9d3be00640 1 -- 172.21.15.62:0/1738463191 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9d440c45e0 con 0x7f9d440995b0 2024-03-23T11:24:05.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:05.466+0000 7f9d4b000640 1 -- 172.21.15.62:0/1738463191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f9d440738e0 con 0x7f9d440995b0 2024-03-23T11:24:06.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.257+0000 7f9d3be00640 1 -- 172.21.15.62:0/1738463191 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v70) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9d30075d60 con 0x7f9d440995b0 2024-03-23T11:24:06.253 INFO:teuthology.orchestra.run.smithi062.stderr: removed all blocklist entries 2024-03-23T11:24:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.261+0000 7f9d4b000640 1 -- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d2406a7f0 msgr2=0x7f9d2406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.261+0000 7f9d4b000640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d2406a7f0 0x7f9d2406ccb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9d34000f60 tx=0x7f9d34009040 comp rx=0 tx=0).stop 2024-03-23T11:24:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.261+0000 7f9d4b000640 1 -- 172.21.15.62:0/1738463191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d440995b0 msgr2=0x7f9d440c9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.261+0000 7f9d4b000640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d440995b0 0x7f9d440c9430 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f9d30034040 tx=0x7f9d300046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.261+0000 7f9d4b000640 1 -- 172.21.15.62:0/1738463191 shutdown_connections 2024-03-23T11:24:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.261+0000 7f9d4b000640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d2406a7f0 0x7f9d2406ccb0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.261+0000 7f9d4b000640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d440995b0 0x7f9d440c9430 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.261+0000 7f9d4b000640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d44160240 0x7f9d440ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.261+0000 7f9d4b000640 1 --2- 172.21.15.62:0/1738463191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d440989d0 0x7f9d441f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.261+0000 7f9d4b000640 1 -- 172.21.15.62:0/1738463191 >> 172.21.15.62:0/1738463191 conn(0x7f9d440b22c0 msgr2=0x7f9d4406e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:06.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.261+0000 7f9d4b000640 1 -- 172.21.15.62:0/1738463191 shutdown_connections 2024-03-23T11:24:06.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.262+0000 7f9d4b000640 1 -- 172.21.15.62:0/1738463191 wait complete. 2024-03-23T11:24:06.277 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T11:24:06.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.499+0000 7f71c9000640 1 Processor -- start 2024-03-23T11:24:06.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.499+0000 7f71c9000640 1 -- start start 2024-03-23T11:24:06.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.501+0000 7f71c9000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c4159490 0x7f71c406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:06.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.501+0000 7f71c9000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71c415da90 0x7f71c40cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:06.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.501+0000 7f71c9000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c40ce8b0 con 0x7f71c40ce420 2024-03-23T11:24:06.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.501+0000 7f71c9000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c40ced90 con 0x7f71c415da90 2024-03-23T11:24:06.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.501+0000 7f71c9000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c406e3e0 con 0x7f71c4159490 2024-03-23T11:24:06.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.501+0000 7f71c2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71c415da90 0x7f71c40cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:06.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.502+0000 7f71c3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c4159490 0x7f71c406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:06.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.502+0000 7f71c2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71c415da90 0x7f71c40cdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:42012/0 (socket says 172.21.15.62:42012) 2024-03-23T11:24:06.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.502+0000 7f71c2a00640 1 -- 172.21.15.62:0/13653301 learned_addr learned my addr 172.21.15.62:0/13653301 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:06.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.502+0000 7f71c3e00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f71c40ce420 0x7f71c40b26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:36990/0 (socket says 172.21.15.62:36990) 2024-03-23T11:24:06.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.502+0000 7f71c2a00640 1 -- 172.21.15.62:0/13653301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c4159490 msgr2=0x7f71c406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:06.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.502+0000 7f71c2a00640 1 --2- 172.21.15.62:0/13653301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c4159490 0x7f71c406cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.502+0000 7f71c2a00640 1 -- 172.21.15.62:0/13653301 >> v1:172.21.15.120:6789/0 conn(0x7f71c40ce420 legacy=0x7f71c40b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:06.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.502+0000 7f71c2a00640 1 -- 172.21.15.62:0/13653301 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c406c1c0 con 0x7f71c415da90 2024-03-23T11:24:06.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.504+0000 7f71c2a00640 1 --2- 172.21.15.62:0/13653301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c415da90 0x7f71c40cdee0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009870 tx=0x7f71b002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cf3f8d156dad910 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:06.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.504+0000 7f71c2000640 1 -- 172.21.15.62:0/13653301 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b0031640 con 0x7f71c415da90 2024-03-23T11:24:06.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.504+0000 7f71c2000640 1 -- 172.21.15.62:0/13653301 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b00317a0 con 0x7f71c415da90 2024-03-23T11:24:06.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.504+0000 7f71c2000640 1 -- 172.21.15.62:0/13653301 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b0031b30 con 0x7f71c415da90 2024-03-23T11:24:06.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.505+0000 7f71c9000640 1 -- 172.21.15.62:0/13653301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c415da90 msgr2=0x7f71c40cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:06.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.505+0000 7f71c9000640 1 --2- 172.21.15.62:0/13653301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c415da90 0x7f71c40cdee0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009870 tx=0x7f71b002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.505+0000 7f71c9000640 1 -- 172.21.15.62:0/13653301 shutdown_connections 2024-03-23T11:24:06.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.505+0000 7f71c9000640 1 --2- 172.21.15.62:0/13653301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c415da90 0x7f71c40cdee0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.505+0000 7f71c9000640 1 --2- 172.21.15.62:0/13653301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c4159490 0x7f71c406cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.505+0000 7f71c9000640 1 -- 172.21.15.62:0/13653301 >> 172.21.15.62:0/13653301 conn(0x7f71c4066050 msgr2=0x7f71c4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:06.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.505+0000 7f71c9000640 1 -- 172.21.15.62:0/13653301 shutdown_connections 2024-03-23T11:24:06.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.505+0000 7f71c9000640 1 -- 172.21.15.62:0/13653301 wait complete. 2024-03-23T11:24:06.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.507+0000 7f71c9000640 1 Processor -- start 2024-03-23T11:24:06.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.507+0000 7f71c9000640 1 -- start start 2024-03-23T11:24:06.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.508+0000 7f71c9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40ce420 0x7f71c4153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:06.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.508+0000 7f71c9000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c4159490 0x7f71c4146f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:06.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.508+0000 7f71c9000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415da90 0x7f71c41f1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:06.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.508+0000 7f71c9000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c41474a0 con 0x7f71c40ce420 2024-03-23T11:24:06.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.508+0000 7f71c9000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c4146510 con 0x7f71c4159490 2024-03-23T11:24:06.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.508+0000 7f71c9000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c41f26d0 con 0x7f71c415da90 2024-03-23T11:24:06.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.508+0000 7f71c3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40ce420 0x7f71c4153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:06.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.508+0000 7f71c3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40ce420 0x7f71c4153cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48754/0 (socket says 172.21.15.62:48754) 2024-03-23T11:24:06.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.508+0000 7f71c3400640 1 -- 172.21.15.62:0/789197755 learned_addr learned my addr 172.21.15.62:0/789197755 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.508+0000 7f71c2a00640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c4159490 0x7f71c4146f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71c3400640 1 -- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415da90 msgr2=0x7f71c41f1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71c3e00640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415da90 0x7f71c41f1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71c3400640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415da90 0x7f71c41f1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71c3400640 1 -- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c4159490 msgr2=0x7f71c4146f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71c3400640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c4159490 0x7f71c4146f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71c3400640 1 -- 172.21.15.62:0/789197755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b00094d0 con 0x7f71c40ce420 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71c3e00640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415da90 0x7f71c41f1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71c3400640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40ce420 0x7f71c4153cc0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f71b400e9b0 tx=0x7f71b400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71bb400640 1 -- 172.21.15.62:0/789197755 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b4009670 con 0x7f71c40ce420 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71bb400640 1 -- 172.21.15.62:0/789197755 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b400c9a0 con 0x7f71c40ce420 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71c9000640 1 -- 172.21.15.62:0/789197755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c41f29b0 con 0x7f71c40ce420 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.509+0000 7f71bb400640 1 -- 172.21.15.62:0/789197755 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b400cd50 con 0x7f71c40ce420 2024-03-23T11:24:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.510+0000 7f71c9000640 1 -- 172.21.15.62:0/789197755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c4146a40 con 0x7f71c40ce420 2024-03-23T11:24:06.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.511+0000 7f71c9000640 1 -- 172.21.15.62:0/789197755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c4066bb0 con 0x7f71c40ce420 2024-03-23T11:24:06.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.512+0000 7f71bb400640 1 -- 172.21.15.62:0/789197755 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f71b4013470 con 0x7f71c40ce420 2024-03-23T11:24:06.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.513+0000 7f71bb400640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a406a7a0 0x7f71a406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:06.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.513+0000 7f71bb400640 1 -- 172.21.15.62:0/789197755 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f71b4088cd0 con 0x7f71c40ce420 2024-03-23T11:24:06.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.513+0000 7f71c2a00640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a406a7a0 0x7f71a406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:06.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.515+0000 7f71c2a00640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a406a7a0 0x7f71a406cc60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f71b0036000 tx=0x7f71b003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:06.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.518+0000 7f71bb400640 1 -- 172.21.15.62:0/789197755 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f71b4054440 con 0x7f71c40ce420 2024-03-23T11:24:06.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.786+0000 7f71c9000640 1 -- 172.21.15.62:0/789197755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f71c406d560 con 0x7f71c40ce420 2024-03-23T11:24:06.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.787+0000 7f71bb400640 1 -- 172.21.15.62:0/789197755 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f71b4057a60 con 0x7f71c40ce420 2024-03-23T11:24:06.783 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:06.783 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:24:06.783 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_dump": [ 2024-03-23T11:24:06.783 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.783 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.a", 2024-03-23T11:24:06.783 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lYNxYERAAPVHWwRHM5V7OejL294b2zg==", 2024-03-23T11:24:06.783 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.783 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:24:06.783 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.b", 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lmOUANBAAifuZ0DUPD9az/mturTndSA==", 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.784 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.785 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.785 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.785 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.c", 2024-03-23T11:24:06.785 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5ltqrjIhAA9+f82fvlrFlIEXqswPE0ig==", 2024-03-23T11:24:06.785 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.785 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:24:06.785 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:24:06.785 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:24:06.785 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.785 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.d", 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5l7ly4CBAA8+oim0z+bbljfP0VH7XSaw==", 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.0", 2024-03-23T11:24:06.786 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBiuv5lRfaQBRAAvrtCZXwt00PUQfCqqKBzMQ==", 2024-03-23T11:24:06.787 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.787 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:24:06.787 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:24:06.787 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.787 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.787 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.787 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.787 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.1", 2024-03-23T11:24:06.787 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBluv5lzwS/IBAAmYU8yMLtCUJNr/AyE0MJDA==", 2024-03-23T11:24:06.787 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.2", 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBouv5lyAy1OhAAJoUsg6DFpEPxAcH5oI2exQ==", 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.788 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.3", 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBsuv5lb0YcGBAA220D0uRhAZpG4w97wqsB5A==", 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.4", 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBzuv5lC8RIGhAAH+5HaeCgQhrQMrYm9iWaYA==", 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.789 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.5", 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB2uv5lYi7ZIhAAMNIm/jPckCAbX9oEx5Li/Q==", 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.790 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.791 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.6", 2024-03-23T11:24:06.791 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB5uv5lgoRtLBAAFT7uJtyqHtHAegOYyDH/WQ==", 2024-03-23T11:24:06.791 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.791 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:24:06.791 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:24:06.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.791 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.791 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.791 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.7", 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB9uv5l/dzrCBAA3jsua0LvBPI7I551/rRHrg==", 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.0", 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5ltavQFhAAvV2RWuHOPK2T9Q7gWkYA4w==", 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow r", 2024-03-23T11:24:06.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.1", 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5luQ/zLRAAwjDtV64TCKnQliPq4iUAYg==", 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.793 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.2", 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lDRG1IhAAjVaXRR30G1985oMou7Ambg==", 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.3", 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lomZyOBAA8BMIBD+Zhtp6Fa2RcEXCJg==", 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:24:06.794 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.admin", 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lNtZHBhAAhiAC1rdmANClOSwzrX9iSQ==", 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.795 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.x", 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lT6+KBRAAl5TZsJGNr4gAB2cEDZo/hQ==", 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.y", 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lqnG8NRAAvyswMz5kpW0rGjzWYRTAAg==", 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:24:06.796 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:24:06.797 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:24:06.797 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.797 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:06.797 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-03-23T11:24:06.797 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:24:06.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 -- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a406a7a0 msgr2=0x7f71a406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:06.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a406a7a0 0x7f71a406cc60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f71b0036000 tx=0x7f71b003c040 comp rx=0 tx=0).stop 2024-03-23T11:24:06.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 -- 172.21.15.62:0/789197755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40ce420 msgr2=0x7f71c4153cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:06.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40ce420 0x7f71c4153cc0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f71b400e9b0 tx=0x7f71b400ee70 comp rx=0 tx=0).stop 2024-03-23T11:24:06.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 -- 172.21.15.62:0/789197755 shutdown_connections 2024-03-23T11:24:06.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71a406a7a0 0x7f71a406cc60 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c415da90 0x7f71c41f1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c4159490 0x7f71c4146f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 --2- 172.21.15.62:0/789197755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71c40ce420 0x7f71c4153cc0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:06.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 -- 172.21.15.62:0/789197755 >> 172.21.15.62:0/789197755 conn(0x7f71c4066050 msgr2=0x7f71c40cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:06.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 -- 172.21.15.62:0/789197755 shutdown_connections 2024-03-23T11:24:06.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:06.790+0000 7f71c9000640 1 -- 172.21.15.62:0/789197755 wait complete. 2024-03-23T11:24:06.805 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:24:07.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.032+0000 7fa0aa600640 1 Processor -- start 2024-03-23T11:24:07.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.032+0000 7fa0aa600640 1 -- start start 2024-03-23T11:24:07.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.034+0000 7fa0aa600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa0a4070090 0x7fa0a4070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.034+0000 7fa0aa600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0a40b2820 0x7fa0a40b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.034+0000 7fa0aa600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a40b31b0 con 0x7fa0a4143dd0 2024-03-23T11:24:07.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.034+0000 7fa0aa600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a4070a60 con 0x7fa0a40b2820 2024-03-23T11:24:07.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.034+0000 7fa0aa600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a40cd140 con 0x7fa0a4070090 2024-03-23T11:24:07.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.034+0000 7fa0a9c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0a40b2820 0x7fa0a40b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.034+0000 7fa0a9200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa0a4143dd0 0x7fa0a415abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40916/0 (socket says 172.21.15.62:40916) 2024-03-23T11:24:07.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.034+0000 7fa0a9200640 1 -- 172.21.15.62:0/2375045804 learned_addr learned my addr 172.21.15.62:0/2375045804 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:07.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.035+0000 7fa0a3e00640 1 --2- 172.21.15.62:0/2375045804 >> v2:172.21.15.120:3300/0 conn(0x7fa0a4070090 0x7fa0a4070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.035+0000 7fa0a9c00640 1 -- 172.21.15.62:0/2375045804 >> v2:172.21.15.120:3300/0 conn(0x7fa0a4070090 msgr2=0x7fa0a4070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:07.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.035+0000 7fa0a9c00640 1 --2- 172.21.15.62:0/2375045804 >> v2:172.21.15.120:3300/0 conn(0x7fa0a4070090 0x7fa0a4070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.035+0000 7fa0a9c00640 1 -- 172.21.15.62:0/2375045804 >> v1:172.21.15.62:6789/0 conn(0x7fa0a4143dd0 legacy=0x7fa0a415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.035+0000 7fa0a9c00640 1 -- 172.21.15.62:0/2375045804 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a40cd3e0 con 0x7fa0a40b2820 2024-03-23T11:24:07.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.036+0000 7fa0a9c00640 1 --2- 172.21.15.62:0/2375045804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0a40b2820 0x7fa0a40b2c70 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4071530 tx=0x7fa09c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b41833267152ee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:07.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.036+0000 7fa0a3400640 1 -- 172.21.15.62:0/2375045804 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09c02cc70 con 0x7fa0a40b2820 2024-03-23T11:24:07.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.036+0000 7fa0a3400640 1 -- 172.21.15.62:0/2375045804 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09c02cdd0 con 0x7fa0a40b2820 2024-03-23T11:24:07.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.037+0000 7fa0a3400640 1 -- 172.21.15.62:0/2375045804 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09c034660 con 0x7fa0a40b2820 2024-03-23T11:24:07.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.037+0000 7fa0aa600640 1 -- 172.21.15.62:0/2375045804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0a40b2820 msgr2=0x7fa0a40b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.037+0000 7fa0aa600640 1 --2- 172.21.15.62:0/2375045804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0a40b2820 0x7fa0a40b2c70 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4071530 tx=0x7fa09c02bec0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.037+0000 7fa0aa600640 1 -- 172.21.15.62:0/2375045804 shutdown_connections 2024-03-23T11:24:07.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.037+0000 7fa0aa600640 1 --2- 172.21.15.62:0/2375045804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0a40b2820 0x7fa0a40b2c70 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.037+0000 7fa0aa600640 1 --2- 172.21.15.62:0/2375045804 >> v2:172.21.15.120:3300/0 conn(0x7fa0a4070090 0x7fa0a4070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.037+0000 7fa0aa600640 1 -- 172.21.15.62:0/2375045804 >> 172.21.15.62:0/2375045804 conn(0x7fa0a4066050 msgr2=0x7fa0a4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:07.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.037+0000 7fa0aa600640 1 -- 172.21.15.62:0/2375045804 shutdown_connections 2024-03-23T11:24:07.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.037+0000 7fa0aa600640 1 -- 172.21.15.62:0/2375045804 wait complete. 2024-03-23T11:24:07.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.039+0000 7fa0aa600640 1 Processor -- start 2024-03-23T11:24:07.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.039+0000 7fa0aa600640 1 -- start start 2024-03-23T11:24:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.040+0000 7fa0aa600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0a4070090 0x7fa0a40c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.040+0000 7fa0aa600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0a40b2820 0x7fa0a40aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.040+0000 7fa0aa600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0a4143dd0 0x7fa0a413fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.040+0000 7fa0aa600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a40b07b0 con 0x7fa0a4070090 2024-03-23T11:24:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.040+0000 7fa0aa600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a40c33a0 con 0x7fa0a40b2820 2024-03-23T11:24:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.040+0000 7fa0aa600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a40af4f0 con 0x7fa0a4143dd0 2024-03-23T11:24:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.040+0000 7fa0a9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0a4070090 0x7fa0a40c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.040+0000 7fa0a9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0a4070090 0x7fa0a40c58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48774/0 (socket says 172.21.15.62:48774) 2024-03-23T11:24:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.040+0000 7fa0a9200640 1 -- 172.21.15.62:0/3562481788 learned_addr learned my addr 172.21.15.62:0/3562481788 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0a3e00640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0a40b2820 0x7fa0a40aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0a9200640 1 -- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0a4143dd0 msgr2=0x7fa0a413fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0a9200640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0a4143dd0 0x7fa0a413fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0a9200640 1 -- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0a40b2820 msgr2=0x7fa0a40aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0a9200640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0a40b2820 0x7fa0a40aee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0a9200640 1 -- 172.21.15.62:0/3562481788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c009a70 con 0x7fa0a4070090 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0a3e00640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0a40b2820 0x7fa0a40aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0a9200640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0a4070090 0x7fa0a40c58a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa09800c480 tx=0x7fa09800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0a0c00640 1 -- 172.21.15.62:0/3562481788 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098005ce0 con 0x7fa0a4070090 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0a0c00640 1 -- 172.21.15.62:0/3562481788 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa098005e40 con 0x7fa0a4070090 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0a0c00640 1 -- 172.21.15.62:0/3562481788 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0980072f0 con 0x7fa0a4070090 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.041+0000 7fa0aa600640 1 -- 172.21.15.62:0/3562481788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a40af7d0 con 0x7fa0a4070090 2024-03-23T11:24:07.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.042+0000 7fa0aa600640 1 -- 172.21.15.62:0/3562481788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a40afd90 con 0x7fa0a4070090 2024-03-23T11:24:07.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.043+0000 7fa0aa600640 1 -- 172.21.15.62:0/3562481788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a40cd960 con 0x7fa0a4070090 2024-03-23T11:24:07.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.044+0000 7fa0a0c00640 1 -- 172.21.15.62:0/3562481788 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa098007510 con 0x7fa0a4070090 2024-03-23T11:24:07.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.045+0000 7fa0a0c00640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa08406a7a0 0x7fa08406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.045+0000 7fa0a0c00640 1 -- 172.21.15.62:0/3562481788 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa098087f50 con 0x7fa0a4070090 2024-03-23T11:24:07.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.045+0000 7fa0a3e00640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa08406a7a0 0x7fa08406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.046+0000 7fa0a3e00640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa08406a7a0 0x7fa08406cc60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40c6f00 tx=0x7fa090008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:07.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.049+0000 7fa0a0c00640 1 -- 172.21.15.62:0/3562481788 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa098053760 con 0x7fa0a4070090 2024-03-23T11:24:07.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.249+0000 7fa0aa600640 1 -- 172.21.15.62:0/3562481788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0a415d8f0 con 0x7fa0a4070090 2024-03-23T11:24:07.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.250+0000 7fa0a0c00640 1 -- 172.21.15.62:0/3562481788 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+12389 (secure 0 0 0) 0x7fa098017020 con 0x7fa0a4070090 2024-03-23T11:24:07.246 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:07.246 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":70,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:06.242553+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:24:07.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.253+0000 7fa0aa600640 1 -- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa08406a7a0 msgr2=0x7fa08406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.253+0000 7fa0aa600640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa08406a7a0 0x7fa08406cc60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40c6f00 tx=0x7fa090008040 comp rx=0 tx=0).stop 2024-03-23T11:24:07.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.253+0000 7fa0aa600640 1 -- 172.21.15.62:0/3562481788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0a4070090 msgr2=0x7fa0a40c58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.253+0000 7fa0aa600640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0a4070090 0x7fa0a40c58a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa09800c480 tx=0x7fa09800c940 comp rx=0 tx=0).stop 2024-03-23T11:24:07.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.254+0000 7fa0aa600640 1 -- 172.21.15.62:0/3562481788 shutdown_connections 2024-03-23T11:24:07.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.254+0000 7fa0aa600640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa08406a7a0 0x7fa08406cc60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.254+0000 7fa0aa600640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0a4143dd0 0x7fa0a413fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.254+0000 7fa0aa600640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0a40b2820 0x7fa0a40aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.254+0000 7fa0aa600640 1 --2- 172.21.15.62:0/3562481788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0a4070090 0x7fa0a40c58a0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.254+0000 7fa0aa600640 1 -- 172.21.15.62:0/3562481788 >> 172.21.15.62:0/3562481788 conn(0x7fa0a4066050 msgr2=0x7fa0a40cb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:07.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.254+0000 7fa0aa600640 1 -- 172.21.15.62:0/3562481788 shutdown_connections 2024-03-23T11:24:07.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.254+0000 7fa0aa600640 1 -- 172.21.15.62:0/3562481788 wait complete. 2024-03-23T11:24:07.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:24:07.267 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:24:07.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.499+0000 7f54af200640 1 Processor -- start 2024-03-23T11:24:07.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.499+0000 7f54af200640 1 -- start start 2024-03-23T11:24:07.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.501+0000 7f54af200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b0143dd0 0x7f54b015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.501+0000 7f54af200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f54b0070090 0x7f54b00704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.501+0000 7f54af200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b0070a30 con 0x7f54b00b2820 2024-03-23T11:24:07.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.501+0000 7f54af200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b00cd140 con 0x7f54b0070090 2024-03-23T11:24:07.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.501+0000 7f54af200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b00cd3e0 con 0x7f54b0143dd0 2024-03-23T11:24:07.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.501+0000 7f54a7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f54b0070090 0x7f54b00704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.501+0000 7f54a7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f54b0070090 0x7f54b00704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59210/0 (socket says 172.21.15.62:59210) 2024-03-23T11:24:07.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.501+0000 7f54a7e00640 1 -- 172.21.15.62:0/3574960139 learned_addr learned my addr 172.21.15.62:0/3574960139 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:07.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.502+0000 7f54ade00640 1 --2- 172.21.15.62:0/3574960139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b0143dd0 0x7f54b015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.502+0000 7f54a7e00640 1 -- 172.21.15.62:0/3574960139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b0143dd0 msgr2=0x7f54b015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:07.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.502+0000 7f54a7e00640 1 --2- 172.21.15.62:0/3574960139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b0143dd0 0x7f54b015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.502+0000 7f54a7e00640 1 -- 172.21.15.62:0/3574960139 >> v1:172.21.15.120:6789/0 conn(0x7f54b00b2820 legacy=0x7f54b00b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:07.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.502+0000 7f54a7e00640 1 -- 172.21.15.62:0/3574960139 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b00cd6c0 con 0x7f54b0070090 2024-03-23T11:24:07.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.502+0000 7f54ade00640 1 --2- 172.21.15.62:0/3574960139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b0143dd0 0x7f54b015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:07.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.503+0000 7f54a7e00640 1 --2- 172.21.15.62:0/3574960139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54b0070090 0x7f54b00704f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f5498009870 tx=0x7f549802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7147fd5895754186 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:07.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.503+0000 7f54ad400640 1 -- 172.21.15.62:0/3574960139 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498031640 con 0x7f54b0070090 2024-03-23T11:24:07.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.504+0000 7f54ad400640 1 -- 172.21.15.62:0/3574960139 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54980317a0 con 0x7f54b0070090 2024-03-23T11:24:07.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.504+0000 7f54ad400640 1 -- 172.21.15.62:0/3574960139 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498031b30 con 0x7f54b0070090 2024-03-23T11:24:07.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.504+0000 7f54af200640 1 -- 172.21.15.62:0/3574960139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54b0070090 msgr2=0x7f54b00704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.504+0000 7f54af200640 1 --2- 172.21.15.62:0/3574960139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54b0070090 0x7f54b00704f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f5498009870 tx=0x7f549802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.504+0000 7f54af200640 1 -- 172.21.15.62:0/3574960139 shutdown_connections 2024-03-23T11:24:07.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.504+0000 7f54af200640 1 --2- 172.21.15.62:0/3574960139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54b0070090 0x7f54b00704f0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.504+0000 7f54af200640 1 --2- 172.21.15.62:0/3574960139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b0143dd0 0x7f54b015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.504+0000 7f54af200640 1 -- 172.21.15.62:0/3574960139 >> 172.21.15.62:0/3574960139 conn(0x7f54b0066050 msgr2=0x7f54b0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.505+0000 7f54af200640 1 -- 172.21.15.62:0/3574960139 shutdown_connections 2024-03-23T11:24:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.505+0000 7f54af200640 1 -- 172.21.15.62:0/3574960139 wait complete. 2024-03-23T11:24:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.506+0000 7f54af200640 1 Processor -- start 2024-03-23T11:24:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.506+0000 7f54af200640 1 -- start start 2024-03-23T11:24:07.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.507+0000 7f54af200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b0070090 0x7f54b00c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.507+0000 7f54af200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54b00b2820 0x7f54b0141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.507+0000 7f54af200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b0143dd0 0x7f54b0141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.507+0000 7f54af200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b00a9390 con 0x7f54b00b2820 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.507+0000 7f54af200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b00a90b0 con 0x7f54b0070090 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.507+0000 7f54af200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b0130b80 con 0x7f54b0143dd0 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.508+0000 7f54a7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54b00b2820 0x7f54b0141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.508+0000 7f54a7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54b00b2820 0x7f54b0141a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59222/0 (socket says 172.21.15.62:59222) 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.508+0000 7f54a7e00640 1 -- 172.21.15.62:0/3976101575 learned_addr learned my addr 172.21.15.62:0/3976101575 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.508+0000 7f54ade00640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b0070090 0x7f54b00c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.508+0000 7f54ae800640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b0143dd0 0x7f54b0141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.508+0000 7f54a7e00640 1 -- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b0143dd0 msgr2=0x7f54b0141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.508+0000 7f54a7e00640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b0143dd0 0x7f54b0141f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.508+0000 7f54a7e00640 1 -- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b0070090 msgr2=0x7f54b00c9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.508+0000 7f54a7e00640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b0070090 0x7f54b00c9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.508+0000 7f54a7e00640 1 -- 172.21.15.62:0/3976101575 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980094d0 con 0x7f54b00b2820 2024-03-23T11:24:07.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.508+0000 7f54a7e00640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54b00b2820 0x7f54b0141a10 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f5498009870 tx=0x7f54980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:07.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.509+0000 7f54a6000640 1 -- 172.21.15.62:0/3976101575 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549803ac60 con 0x7f54b00b2820 2024-03-23T11:24:07.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.509+0000 7f54a6000640 1 -- 172.21.15.62:0/3976101575 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549803ade0 con 0x7f54b00b2820 2024-03-23T11:24:07.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.509+0000 7f54af200640 1 -- 172.21.15.62:0/3976101575 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b0130e60 con 0x7f54b00b2820 2024-03-23T11:24:07.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.509+0000 7f54a6000640 1 -- 172.21.15.62:0/3976101575 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498044630 con 0x7f54b00b2820 2024-03-23T11:24:07.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.509+0000 7f54af200640 1 -- 172.21.15.62:0/3976101575 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b00ca090 con 0x7f54b00b2820 2024-03-23T11:24:07.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.510+0000 7f54af200640 1 -- 172.21.15.62:0/3976101575 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b0070f10 con 0x7f54b00b2820 2024-03-23T11:24:07.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.511+0000 7f54a6000640 1 -- 172.21.15.62:0/3976101575 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f549802f050 con 0x7f54b00b2820 2024-03-23T11:24:07.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.512+0000 7f54a6000640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548c06a7f0 0x7f548c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.512+0000 7f54a6000640 1 -- 172.21.15.62:0/3976101575 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54980aaae0 con 0x7f54b00b2820 2024-03-23T11:24:07.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.513+0000 7f54ade00640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548c06a7f0 0x7f548c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.514+0000 7f54ade00640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548c06a7f0 0x7f548c06ccb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f54a000a950 tx=0x7f54a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:07.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.517+0000 7f54a6000640 1 -- 172.21.15.62:0/3976101575 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5498076330 con 0x7f54b00b2820 2024-03-23T11:24:07.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.714+0000 7f54af200640 1 -- 172.21.15.62:0/3976101575 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54b0131380 con 0x7f54b00b2820 2024-03-23T11:24:07.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.714+0000 7f54a6000640 1 -- 172.21.15.62:0/3976101575 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f54b0131380 con 0x7f54b00b2820 2024-03-23T11:24:07.710 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:24:07.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.717+0000 7f54af200640 1 -- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548c06a7f0 msgr2=0x7f548c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.717+0000 7f54af200640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548c06a7f0 0x7f548c06ccb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f54a000a950 tx=0x7f54a0008040 comp rx=0 tx=0).stop 2024-03-23T11:24:07.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.717+0000 7f54af200640 1 -- 172.21.15.62:0/3976101575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54b00b2820 msgr2=0x7f54b0141a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.717+0000 7f54af200640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54b00b2820 0x7f54b0141a10 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f5498009870 tx=0x7f54980046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.717+0000 7f54af200640 1 -- 172.21.15.62:0/3976101575 shutdown_connections 2024-03-23T11:24:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.717+0000 7f54af200640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f548c06a7f0 0x7f548c06ccb0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.717+0000 7f54af200640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54b00b2820 0x7f54b0141a10 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.717+0000 7f54af200640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54b0143dd0 0x7f54b0141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.717+0000 7f54af200640 1 --2- 172.21.15.62:0/3976101575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54b0070090 0x7f54b00c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.717+0000 7f54af200640 1 -- 172.21.15.62:0/3976101575 >> 172.21.15.62:0/3976101575 conn(0x7f54b0066050 msgr2=0x7f54b015a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.718+0000 7f54af200640 1 -- 172.21.15.62:0/3976101575 shutdown_connections 2024-03-23T11:24:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.718+0000 7f54af200640 1 -- 172.21.15.62:0/3976101575 wait complete. 2024-03-23T11:24:07.733 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T11:24:07.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.957+0000 7f3372e00640 1 Processor -- start 2024-03-23T11:24:07.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.957+0000 7f3372e00640 1 -- start start 2024-03-23T11:24:07.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.959+0000 7f3372e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f336c15a7c0 0x7f336c15b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.959+0000 7f3372e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c15be90 con 0x7f336c0cf920 2024-03-23T11:24:07.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.959+0000 7f3372e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c0d0100 con 0x7f336c070480 2024-03-23T11:24:07.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.959+0000 7f3372e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336c15bcf0 con 0x7f336c15a7c0 2024-03-23T11:24:07.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.959+0000 7f336b400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f336c0cf920 0x7f336c06fd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:38574/0 (socket says 172.21.15.62:38574) 2024-03-23T11:24:07.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.959+0000 7f336b400640 1 -- 172.21.15.62:0/226445826 learned_addr learned my addr 172.21.15.62:0/226445826 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:07.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.959+0000 7f336be00640 1 --2- 172.21.15.62:0/226445826 >> v2:172.21.15.120:3300/0 conn(0x7f336c15a7c0 0x7f336c15b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.960+0000 7f336aa00640 1 -- 172.21.15.62:0/226445826 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243892330 0 0) 0x7f336c15be90 con 0x7f336c0cf920 2024-03-23T11:24:07.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.960+0000 7f336aa00640 1 -- 172.21.15.62:0/226445826 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3350003520 con 0x7f336c0cf920 2024-03-23T11:24:07.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.960+0000 7f336aa00640 1 -- 172.21.15.62:0/226445826 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3792395233 0 0) 0x7f3350003520 con 0x7f336c0cf920 2024-03-23T11:24:07.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.960+0000 7f336be00640 1 -- 172.21.15.62:0/226445826 >> v1:172.21.15.120:6789/0 conn(0x7f336c070480 legacy=0x7f336c15c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.960+0000 7f336be00640 1 -- 172.21.15.62:0/226445826 >> v1:172.21.15.62:6789/0 conn(0x7f336c0cf920 legacy=0x7f336c06fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.960+0000 7f336be00640 1 -- 172.21.15.62:0/226445826 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c0d03e0 con 0x7f336c15a7c0 2024-03-23T11:24:07.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.962+0000 7f336be00640 1 --2- 172.21.15.62:0/226445826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f336c15a7c0 0x7f336c15b7b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f335c0097a0 tx=0x7f335c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97ef7e2b78cd2277 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:07.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.962+0000 7f336aa00640 1 -- 172.21.15.62:0/226445826 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358001670 con 0x7f336c15a7c0 2024-03-23T11:24:07.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.962+0000 7f336aa00640 1 -- 172.21.15.62:0/226445826 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335c02e050 con 0x7f336c15a7c0 2024-03-23T11:24:07.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.962+0000 7f336aa00640 1 -- 172.21.15.62:0/226445826 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c030730 con 0x7f336c15a7c0 2024-03-23T11:24:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.963+0000 7f3372e00640 1 -- 172.21.15.62:0/226445826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f336c15a7c0 msgr2=0x7f336c15b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.963+0000 7f3372e00640 1 --2- 172.21.15.62:0/226445826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f336c15a7c0 0x7f336c15b7b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f335c0097a0 tx=0x7f335c02b790 comp rx=0 tx=0).stop 2024-03-23T11:24:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.963+0000 7f3372e00640 1 -- 172.21.15.62:0/226445826 shutdown_connections 2024-03-23T11:24:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.963+0000 7f3372e00640 1 --2- 172.21.15.62:0/226445826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f336c15a7c0 0x7f336c15b7b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.963+0000 7f3372e00640 1 -- 172.21.15.62:0/226445826 >> 172.21.15.62:0/226445826 conn(0x7f336c0b23e0 msgr2=0x7f336c0b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.963+0000 7f3372e00640 1 -- 172.21.15.62:0/226445826 shutdown_connections 2024-03-23T11:24:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.963+0000 7f3372e00640 1 -- 172.21.15.62:0/226445826 wait complete. 2024-03-23T11:24:07.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.965+0000 7f3372e00640 1 Processor -- start 2024-03-23T11:24:07.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.965+0000 7f3372e00640 1 -- start start 2024-03-23T11:24:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.966+0000 7f3372e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f336c070480 0x7f336c1e6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.966+0000 7f3372e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0cf920 0x7f336c155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.966+0000 7f3372e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f336c15a7c0 0x7f336c155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.966+0000 7f3372e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c145b30 con 0x7f336c0cf920 2024-03-23T11:24:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.966+0000 7f3372e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c144f20 con 0x7f336c070480 2024-03-23T11:24:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.966+0000 7f3372e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c145200 con 0x7f336c15a7c0 2024-03-23T11:24:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.966+0000 7f336b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0cf920 0x7f336c155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.967+0000 7f336b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0cf920 0x7f336c155150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59232/0 (socket says 172.21.15.62:59232) 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.967+0000 7f336b400640 1 -- 172.21.15.62:0/1103812044 learned_addr learned my addr 172.21.15.62:0/1103812044 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.967+0000 7f336be00640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f336c070480 0x7f336c1e6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.967+0000 7f336b400640 1 -- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f336c15a7c0 msgr2=0x7f336c155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.967+0000 7f336b400640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f336c15a7c0 0x7f336c155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.967+0000 7f336b400640 1 -- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f336c070480 msgr2=0x7f336c1e6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.967+0000 7f336b400640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f336c070480 0x7f336c1e6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.967+0000 7f336b400640 1 -- 172.21.15.62:0/1103812044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c009400 con 0x7f336c0cf920 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.967+0000 7f336be00640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f336c070480 0x7f336c1e6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.967+0000 7f336b400640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0cf920 0x7f336c155150 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f335800c760 tx=0x7f335800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.968+0000 7f3363e00640 1 -- 172.21.15.62:0/1103812044 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358005ad0 con 0x7f336c0cf920 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.968+0000 7f3363e00640 1 -- 172.21.15.62:0/1103812044 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3358005c30 con 0x7f336c0cf920 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.968+0000 7f3372e00640 1 -- 172.21.15.62:0/1103812044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c1454e0 con 0x7f336c0cf920 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.968+0000 7f3363e00640 1 -- 172.21.15.62:0/1103812044 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358007070 con 0x7f336c0cf920 2024-03-23T11:24:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.968+0000 7f3372e00640 1 -- 172.21.15.62:0/1103812044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c1e6ff0 con 0x7f336c0cf920 2024-03-23T11:24:07.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.969+0000 7f3372e00640 1 -- 172.21.15.62:0/1103812044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336c145ca0 con 0x7f336c0cf920 2024-03-23T11:24:07.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.970+0000 7f3363e00640 1 -- 172.21.15.62:0/1103812044 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f33580071d0 con 0x7f336c0cf920 2024-03-23T11:24:07.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.971+0000 7f3363e00640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f334c06a7a0 0x7f334c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:07.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.971+0000 7f3363e00640 1 -- 172.21.15.62:0/1103812044 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f33580887e0 con 0x7f336c0cf920 2024-03-23T11:24:07.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.972+0000 7f336be00640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f334c06a7a0 0x7f334c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.973+0000 7f336be00640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f334c06a7a0 0x7f334c06cc60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f335c034000 tx=0x7f335c02f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:07.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:07.975+0000 7f3363e00640 1 -- 172.21.15.62:0/1103812044 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3358053f40 con 0x7f336c0cf920 2024-03-23T11:24:08.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.175+0000 7f3372e00640 1 -- 172.21.15.62:0/1103812044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f336c0672c0 con 0x7f336c0cf920 2024-03-23T11:24:08.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.176+0000 7f3363e00640 1 -- 172.21.15.62:0/1103812044 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v70) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3358057560 con 0x7f336c0cf920 2024-03-23T11:24:08.171 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:08.171 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:24:08.172 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:24:08.172 INFO:teuthology.orchestra.run.smithi062.stdout: "poolnum": 1, 2024-03-23T11:24:08.172 INFO:teuthology.orchestra.run.smithi062.stdout: "poolname": ".mgr" 2024-03-23T11:24:08.172 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:08.172 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:24:08.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.178+0000 7f3372e00640 1 -- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f334c06a7a0 msgr2=0x7f334c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:08.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.178+0000 7f3372e00640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f334c06a7a0 0x7f334c06cc60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f335c034000 tx=0x7f335c02f000 comp rx=0 tx=0).stop 2024-03-23T11:24:08.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.178+0000 7f3372e00640 1 -- 172.21.15.62:0/1103812044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0cf920 msgr2=0x7f336c155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:08.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.178+0000 7f3372e00640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0cf920 0x7f336c155150 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f335800c760 tx=0x7f335800cc20 comp rx=0 tx=0).stop 2024-03-23T11:24:08.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.179+0000 7f3372e00640 1 -- 172.21.15.62:0/1103812044 shutdown_connections 2024-03-23T11:24:08.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.179+0000 7f3372e00640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f334c06a7a0 0x7f334c06cc60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:08.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.179+0000 7f3372e00640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0cf920 0x7f336c155150 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:08.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.179+0000 7f3372e00640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f336c15a7c0 0x7f336c155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:08.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.179+0000 7f3372e00640 1 --2- 172.21.15.62:0/1103812044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f336c070480 0x7f336c1e6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:08.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.179+0000 7f3372e00640 1 -- 172.21.15.62:0/1103812044 >> 172.21.15.62:0/1103812044 conn(0x7f336c0b23e0 msgr2=0x7f336c072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:08.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.179+0000 7f3372e00640 1 -- 172.21.15.62:0/1103812044 shutdown_connections 2024-03-23T11:24:08.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.179+0000 7f3372e00640 1 -- 172.21.15.62:0/1103812044 wait complete. 2024-03-23T11:24:08.194 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T11:24:08.194 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T11:24:08.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.422+0000 7f09d6400640 1 Processor -- start 2024-03-23T11:24:08.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.422+0000 7f09d6400640 1 -- start start 2024-03-23T11:24:08.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.424+0000 7f09d6400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f09d0154d30 0x7f09d00ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:08.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.424+0000 7f09d6400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f09d00ab850 0x7f09d006c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:08.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.424+0000 7f09d6400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d015bcb0 con 0x7f09d00cebd0 2024-03-23T11:24:08.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.424+0000 7f09d6400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d00cf080 con 0x7f09d0154d30 2024-03-23T11:24:08.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.424+0000 7f09d6400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d00abca0 con 0x7f09d00ab850 2024-03-23T11:24:08.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.424+0000 7f09d5000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f09d0154d30 0x7f09d00ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:08.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.425+0000 7f09d5000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f09d0154d30 0x7f09d00ce690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59248/0 (socket says 172.21.15.62:59248) 2024-03-23T11:24:08.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.425+0000 7f09d5000640 1 -- 172.21.15.62:0/1920994641 learned_addr learned my addr 172.21.15.62:0/1920994641 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:08.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.425+0000 7f09d5a00640 1 --2- 172.21.15.62:0/1920994641 >> v2:172.21.15.120:3300/0 conn(0x7f09d00ab850 0x7f09d006c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:08.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.425+0000 7f09d5000640 1 -- 172.21.15.62:0/1920994641 >> v2:172.21.15.120:3300/0 conn(0x7f09d00ab850 msgr2=0x7f09d006c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:08.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.425+0000 7f09d5000640 1 --2- 172.21.15.62:0/1920994641 >> v2:172.21.15.120:3300/0 conn(0x7f09d00ab850 0x7f09d006c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:08.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.425+0000 7f09d5000640 1 -- 172.21.15.62:0/1920994641 >> v1:172.21.15.120:6789/0 conn(0x7f09d00cebd0 legacy=0x7f09d00ab140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:08.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.425+0000 7f09d5000640 1 -- 172.21.15.62:0/1920994641 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d015ac90 con 0x7f09d0154d30 2024-03-23T11:24:08.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.426+0000 7f09d5000640 1 --2- 172.21.15.62:0/1920994641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09d0154d30 0x7f09d00ce690 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f09d00b3560 tx=0x7f09c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6a33a0f9210e9c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:08.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.427+0000 7f09cf400640 1 -- 172.21.15.62:0/1920994641 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0031870 con 0x7f09d0154d30 2024-03-23T11:24:08.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.427+0000 7f09cf400640 1 -- 172.21.15.62:0/1920994641 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c00319d0 con 0x7f09d0154d30 2024-03-23T11:24:08.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.427+0000 7f09cf400640 1 -- 172.21.15.62:0/1920994641 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0031d60 con 0x7f09d0154d30 2024-03-23T11:24:08.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.427+0000 7f09d6400640 1 -- 172.21.15.62:0/1920994641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09d0154d30 msgr2=0x7f09d00ce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:08.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.427+0000 7f09d6400640 1 --2- 172.21.15.62:0/1920994641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09d0154d30 0x7f09d00ce690 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f09d00b3560 tx=0x7f09c002b910 comp rx=0 tx=0).stop 2024-03-23T11:24:08.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.428+0000 7f09d6400640 1 -- 172.21.15.62:0/1920994641 shutdown_connections 2024-03-23T11:24:08.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.428+0000 7f09d6400640 1 --2- 172.21.15.62:0/1920994641 >> v2:172.21.15.120:3300/0 conn(0x7f09d00ab850 0x7f09d006c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:08.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.428+0000 7f09d6400640 1 --2- 172.21.15.62:0/1920994641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09d0154d30 0x7f09d00ce690 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:08.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.428+0000 7f09d6400640 1 -- 172.21.15.62:0/1920994641 >> 172.21.15.62:0/1920994641 conn(0x7f09d00b0280 msgr2=0x7f09d00b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:08.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.428+0000 7f09d6400640 1 -- 172.21.15.62:0/1920994641 shutdown_connections 2024-03-23T11:24:08.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.428+0000 7f09d6400640 1 -- 172.21.15.62:0/1920994641 wait complete. 2024-03-23T11:24:08.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.430+0000 7f09d6400640 1 Processor -- start 2024-03-23T11:24:08.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.430+0000 7f09d6400640 1 -- start start 2024-03-23T11:24:08.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.431+0000 7f09d6400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09d00ab850 0x7f09d014e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:08.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.431+0000 7f09d6400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09d00cebd0 0x7f09d0157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:08.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.431+0000 7f09d6400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09d0154d30 0x7f09d014db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:08.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.431+0000 7f09d6400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00ccc80 con 0x7f09d0154d30 2024-03-23T11:24:08.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.431+0000 7f09d6400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d01581d0 con 0x7f09d00ab850 2024-03-23T11:24:08.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.431+0000 7f09d6400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d014e8c0 con 0x7f09d00cebd0 2024-03-23T11:24:08.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.431+0000 7f09d5a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09d0154d30 0x7f09d014db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:08.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.431+0000 7f09d5a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09d0154d30 0x7f09d014db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59258/0 (socket says 172.21.15.62:59258) 2024-03-23T11:24:08.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.431+0000 7f09d5a00640 1 -- 172.21.15.62:0/2917356396 learned_addr learned my addr 172.21.15.62:0/2917356396 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:08.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.431+0000 7f09d5000640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09d00ab850 0x7f09d014e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:08.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.432+0000 7f09cfe00640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09d00cebd0 0x7f09d0157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.432+0000 7f09d5a00640 1 -- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09d00cebd0 msgr2=0x7f09d0157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.432+0000 7f09d5a00640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09d00cebd0 0x7f09d0157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.432+0000 7f09d5a00640 1 -- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09d00ab850 msgr2=0x7f09d014e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.432+0000 7f09d5a00640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09d00ab850 0x7f09d014e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.432+0000 7f09d5a00640 1 -- 172.21.15.62:0/2917356396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c00094d0 con 0x7f09d0154d30 2024-03-23T11:24:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.432+0000 7f09d5a00640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09d0154d30 0x7f09d014db40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f09c8007e40 tx=0x7f09c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.432+0000 7f09ccc00640 1 -- 172.21.15.62:0/2917356396 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8014620 con 0x7f09d0154d30 2024-03-23T11:24:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.432+0000 7f09ccc00640 1 -- 172.21.15.62:0/2917356396 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c800ee90 con 0x7f09d0154d30 2024-03-23T11:24:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.432+0000 7f09ccc00640 1 -- 172.21.15.62:0/2917356396 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c80149e0 con 0x7f09d0154d30 2024-03-23T11:24:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.432+0000 7f09d6400640 1 -- 172.21.15.62:0/2917356396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d00cced0 con 0x7f09d0154d30 2024-03-23T11:24:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.433+0000 7f09d6400640 1 -- 172.21.15.62:0/2917356396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d00cd490 con 0x7f09d0154d30 2024-03-23T11:24:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.434+0000 7f09d6400640 1 -- 172.21.15.62:0/2917356396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d00cd6b0 con 0x7f09d0154d30 2024-03-23T11:24:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.435+0000 7f09ccc00640 1 -- 172.21.15.62:0/2917356396 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f09c8014b40 con 0x7f09d0154d30 2024-03-23T11:24:08.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.436+0000 7f09ccc00640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f09b006a7a0 0x7f09b006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:08.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.436+0000 7f09ccc00640 1 -- 172.21.15.62:0/2917356396 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f09c8089470 con 0x7f09d0154d30 2024-03-23T11:24:08.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.436+0000 7f09d5000640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f09b006a7a0 0x7f09b006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:08.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.437+0000 7f09d5000640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f09b006a7a0 0x7f09b006cc60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f09c0036000 tx=0x7f09c003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:08.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.440+0000 7f09ccc00640 1 -- 172.21.15.62:0/2917356396 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09c8054b40 con 0x7f09d0154d30 2024-03-23T11:24:08.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:08.635+0000 7f09d6400640 1 -- 172.21.15.62:0/2917356396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f09d0066bb0 con 0x7f09d0154d30 2024-03-23T11:24:09.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.546+0000 7f09ccc00640 1 -- 172.21.15.62:0/2917356396 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f09c8058160 con 0x7f09d0154d30 2024-03-23T11:24:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' created 2024-03-23T11:24:09.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.551+0000 7f09d6400640 1 -- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f09b006a7a0 msgr2=0x7f09b006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:09.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.551+0000 7f09d6400640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f09b006a7a0 0x7f09b006cc60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f09c0036000 tx=0x7f09c003c040 comp rx=0 tx=0).stop 2024-03-23T11:24:09.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.551+0000 7f09d6400640 1 -- 172.21.15.62:0/2917356396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09d0154d30 msgr2=0x7f09d014db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:09.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.551+0000 7f09d6400640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09d0154d30 0x7f09d014db40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f09c8007e40 tx=0x7f09c800e4b0 comp rx=0 tx=0).stop 2024-03-23T11:24:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.551+0000 7f09d6400640 1 -- 172.21.15.62:0/2917356396 shutdown_connections 2024-03-23T11:24:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.551+0000 7f09d6400640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f09b006a7a0 0x7f09b006cc60 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.551+0000 7f09d6400640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09d0154d30 0x7f09d014db40 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.551+0000 7f09d6400640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09d00cebd0 0x7f09d0157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.551+0000 7f09d6400640 1 --2- 172.21.15.62:0/2917356396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09d00ab850 0x7f09d014e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.551+0000 7f09d6400640 1 -- 172.21.15.62:0/2917356396 >> 172.21.15.62:0/2917356396 conn(0x7f09d00b0280 msgr2=0x7f09d006d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.552+0000 7f09d6400640 1 -- 172.21.15.62:0/2917356396 shutdown_connections 2024-03-23T11:24:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.552+0000 7f09d6400640 1 -- 172.21.15.62:0/2917356396 wait complete. 2024-03-23T11:24:09.572 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T11:24:09.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.800+0000 7ff0eaa00640 1 Processor -- start 2024-03-23T11:24:09.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.800+0000 7ff0eaa00640 1 -- start start 2024-03-23T11:24:09.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.801+0000 7ff0eaa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff0e415cb40 0x7ff0e40d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:09.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.802+0000 7ff0eaa00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e40b8830 con 0x7ff0e40b83e0 2024-03-23T11:24:09.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.802+0000 7ff0eaa00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e40d1350 con 0x7ff0e40cdda0 2024-03-23T11:24:09.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.802+0000 7ff0eaa00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e40d1630 con 0x7ff0e415cb40 2024-03-23T11:24:09.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.802+0000 7ff0e3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff0e415cb40 0x7ff0e40d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:09.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.802+0000 7ff0e3400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff0e40b83e0 0x7ff0e40cd690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:38576/0 (socket says 172.21.15.62:38576) 2024-03-23T11:24:09.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.802+0000 7ff0e3400640 1 -- 172.21.15.62:0/2279204660 learned_addr learned my addr 172.21.15.62:0/2279204660 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:09.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.802+0000 7ff0e3e00640 1 -- 172.21.15.62:0/2279204660 >> v1:172.21.15.120:6789/0 conn(0x7ff0e40cdda0 legacy=0x7ff0e40b59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:09.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.802+0000 7ff0e3e00640 1 -- 172.21.15.62:0/2279204660 >> v1:172.21.15.62:6789/0 conn(0x7ff0e40b83e0 legacy=0x7ff0e40cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:09.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.802+0000 7ff0e3e00640 1 -- 172.21.15.62:0/2279204660 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e40d1910 con 0x7ff0e415cb40 2024-03-23T11:24:09.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.802+0000 7ff0e2a00640 1 -- 172.21.15.62:0/2279204660 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160500060 0 0) 0x7ff0e40b8830 con 0x7ff0e40b83e0 2024-03-23T11:24:09.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.804+0000 7ff0e3e00640 1 --2- 172.21.15.62:0/2279204660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0e415cb40 0x7ff0e40d0c90 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40097a0 tx=0x7ff0d402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=830594fcbadc986d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:09.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.804+0000 7ff0e2a00640 1 -- 172.21.15.62:0/2279204660 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4030610 con 0x7ff0e415cb40 2024-03-23T11:24:09.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.804+0000 7ff0e2a00640 1 -- 172.21.15.62:0/2279204660 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d4030770 con 0x7ff0e415cb40 2024-03-23T11:24:09.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.804+0000 7ff0e2a00640 1 -- 172.21.15.62:0/2279204660 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4030b00 con 0x7ff0e415cb40 2024-03-23T11:24:09.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.805+0000 7ff0eaa00640 1 -- 172.21.15.62:0/2279204660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0e415cb40 msgr2=0x7ff0e40d0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:09.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.805+0000 7ff0eaa00640 1 --2- 172.21.15.62:0/2279204660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0e415cb40 0x7ff0e40d0c90 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40097a0 tx=0x7ff0d402b960 comp rx=0 tx=0).stop 2024-03-23T11:24:09.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.805+0000 7ff0eaa00640 1 -- 172.21.15.62:0/2279204660 shutdown_connections 2024-03-23T11:24:09.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.805+0000 7ff0eaa00640 1 --2- 172.21.15.62:0/2279204660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0e415cb40 0x7ff0e40d0c90 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:09.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.805+0000 7ff0eaa00640 1 -- 172.21.15.62:0/2279204660 >> 172.21.15.62:0/2279204660 conn(0x7ff0e40717b0 msgr2=0x7ff0e4071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:09.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.805+0000 7ff0eaa00640 1 -- 172.21.15.62:0/2279204660 shutdown_connections 2024-03-23T11:24:09.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.805+0000 7ff0eaa00640 1 -- 172.21.15.62:0/2279204660 wait complete. 2024-03-23T11:24:09.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.807+0000 7ff0eaa00640 1 Processor -- start 2024-03-23T11:24:09.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.807+0000 7ff0eaa00640 1 -- start start 2024-03-23T11:24:09.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0eaa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e40b83e0 0x7ff0e415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:09.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0eaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0e40cdda0 0x7ff0e41f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:09.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0eaa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e415cb40 0x7ff0e415a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:09.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0eaa00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40cb740 con 0x7ff0e40cdda0 2024-03-23T11:24:09.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0eaa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e415d5e0 con 0x7ff0e40b83e0 2024-03-23T11:24:09.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0eaa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40ca610 con 0x7ff0e415cb40 2024-03-23T11:24:09.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0e3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0e40cdda0 0x7ff0e41f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:09.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0e3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0e40cdda0 0x7ff0e41f1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59274/0 (socket says 172.21.15.62:59274) 2024-03-23T11:24:09.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0e3400640 1 -- 172.21.15.62:0/1909642644 learned_addr learned my addr 172.21.15.62:0/1909642644 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0e3400640 1 -- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e415cb40 msgr2=0x7ff0e415a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0e3400640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e415cb40 0x7ff0e415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:09.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0e3400640 1 -- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e40b83e0 msgr2=0x7ff0e415d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:09.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0e3400640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e40b83e0 0x7ff0e415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:09.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.808+0000 7ff0e3400640 1 -- 172.21.15.62:0/1909642644 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d4009400 con 0x7ff0e40cdda0 2024-03-23T11:24:09.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.809+0000 7ff0e3400640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0e40cdda0 0x7ff0e41f1e30 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc001460 tx=0x7ff0cc00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:09.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.809+0000 7ff0dbe00640 1 -- 172.21.15.62:0/1909642644 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc00a610 con 0x7ff0e40cdda0 2024-03-23T11:24:09.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.809+0000 7ff0dbe00640 1 -- 172.21.15.62:0/1909642644 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0cc00a770 con 0x7ff0e40cdda0 2024-03-23T11:24:09.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.809+0000 7ff0dbe00640 1 -- 172.21.15.62:0/1909642644 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc00aba0 con 0x7ff0e40cdda0 2024-03-23T11:24:09.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.809+0000 7ff0eaa00640 1 -- 172.21.15.62:0/1909642644 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e40ca8f0 con 0x7ff0e40cdda0 2024-03-23T11:24:09.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.809+0000 7ff0eaa00640 1 -- 172.21.15.62:0/1909642644 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e40cb360 con 0x7ff0e40cdda0 2024-03-23T11:24:09.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.810+0000 7ff0eaa00640 1 -- 172.21.15.62:0/1909642644 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e40672c0 con 0x7ff0e40cdda0 2024-03-23T11:24:09.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.811+0000 7ff0dbe00640 1 -- 172.21.15.62:0/1909642644 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff0cc00ad00 con 0x7ff0e40cdda0 2024-03-23T11:24:09.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.812+0000 7ff0dbe00640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0c006a7d0 0x7ff0c006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:09.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.812+0000 7ff0dbe00640 1 -- 172.21.15.62:0/1909642644 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ff0cc088610 con 0x7ff0e40cdda0 2024-03-23T11:24:09.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.813+0000 7ff0e3e00640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0c006a7d0 0x7ff0c006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:09.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.814+0000 7ff0e3e00640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0c006a7d0 0x7ff0c006cc90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4036000 tx=0x7ff0d4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:09.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:09.818+0000 7ff0dbe00640 1 -- 172.21.15.62:0/1909642644 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff0cc053ae0 con 0x7ff0e40cdda0 2024-03-23T11:24:10.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.018+0000 7ff0eaa00640 1 -- 172.21.15.62:0/1909642644 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff0e40ce540 con 0x7ff0e40cdda0 2024-03-23T11:24:10.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.552+0000 7ff0dbe00640 1 -- 172.21.15.62:0/1909642644 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v72) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff0cc017020 con 0x7ff0e40cdda0 2024-03-23T11:24:10.549 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' created 2024-03-23T11:24:10.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.559+0000 7ff0eaa00640 1 -- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0c006a7d0 msgr2=0x7ff0c006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:10.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.559+0000 7ff0eaa00640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0c006a7d0 0x7ff0c006cc90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4036000 tx=0x7ff0d4030000 comp rx=0 tx=0).stop 2024-03-23T11:24:10.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.560+0000 7ff0eaa00640 1 -- 172.21.15.62:0/1909642644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0e40cdda0 msgr2=0x7ff0e41f1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:10.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.560+0000 7ff0eaa00640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0e40cdda0 0x7ff0e41f1e30 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc001460 tx=0x7ff0cc00cd20 comp rx=0 tx=0).stop 2024-03-23T11:24:10.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.560+0000 7ff0eaa00640 1 -- 172.21.15.62:0/1909642644 shutdown_connections 2024-03-23T11:24:10.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.560+0000 7ff0eaa00640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0c006a7d0 0x7ff0c006cc90 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:10.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.560+0000 7ff0eaa00640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0e40cdda0 0x7ff0e41f1e30 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:10.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.560+0000 7ff0eaa00640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0e415cb40 0x7ff0e415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:10.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.560+0000 7ff0eaa00640 1 --2- 172.21.15.62:0/1909642644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0e40b83e0 0x7ff0e415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:10.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.560+0000 7ff0eaa00640 1 -- 172.21.15.62:0/1909642644 >> 172.21.15.62:0/1909642644 conn(0x7ff0e40717b0 msgr2=0x7ff0e40d2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:10.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.560+0000 7ff0eaa00640 1 -- 172.21.15.62:0/1909642644 shutdown_connections 2024-03-23T11:24:10.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.560+0000 7ff0eaa00640 1 -- 172.21.15.62:0/1909642644 wait complete. 2024-03-23T11:24:10.587 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T11:24:10.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.785+0000 7f568a000640 1 Processor -- start 2024-03-23T11:24:10.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.785+0000 7f568a000640 1 -- start start 2024-03-23T11:24:10.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.787+0000 7f568a000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684143dd0 0x7f568415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:10.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.787+0000 7f568a000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f56840b2f30 0x7f56840704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:10.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.787+0000 7f568a000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5684070a30 con 0x7f5684070090 2024-03-23T11:24:10.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.787+0000 7f568a000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56840cd140 con 0x7f56840b2f30 2024-03-23T11:24:10.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.787+0000 7f568a000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56840cd3e0 con 0x7f5684143dd0 2024-03-23T11:24:10.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.788+0000 7f5688c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684143dd0 0x7f568415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:10.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.788+0000 7f5688c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684143dd0 0x7f568415abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:44026/0 (socket says 172.21.15.62:44026) 2024-03-23T11:24:10.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.788+0000 7f5688c00640 1 -- 172.21.15.62:0/429438282 learned_addr learned my addr 172.21.15.62:0/429438282 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:10.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.788+0000 7f5689600640 1 --2- 172.21.15.62:0/429438282 >> v2:172.21.15.120:3300/0 conn(0x7f56840b2f30 0x7f56840704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:10.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.789+0000 7f5688c00640 1 -- 172.21.15.62:0/429438282 >> v2:172.21.15.120:3300/0 conn(0x7f56840b2f30 msgr2=0x7f56840704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:10.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.789+0000 7f5688c00640 1 --2- 172.21.15.62:0/429438282 >> v2:172.21.15.120:3300/0 conn(0x7f56840b2f30 0x7f56840704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:10.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.789+0000 7f5688c00640 1 -- 172.21.15.62:0/429438282 >> v1:172.21.15.120:6789/0 conn(0x7f5684070090 legacy=0x7f56840b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:10.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.789+0000 7f5688c00640 1 -- 172.21.15.62:0/429438282 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56840cd6c0 con 0x7f5684143dd0 2024-03-23T11:24:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.791+0000 7f5688c00640 1 --2- 172.21.15.62:0/429438282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684143dd0 0x7f568415abe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5674009e50 tx=0x7f567402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cb8b88753d30cc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:10.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.791+0000 7f5683400640 1 -- 172.21.15.62:0/429438282 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567402cb00 con 0x7f5684143dd0 2024-03-23T11:24:10.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.792+0000 7f5683400640 1 -- 172.21.15.62:0/429438282 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567402cc60 con 0x7f5684143dd0 2024-03-23T11:24:10.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.792+0000 7f5683400640 1 -- 172.21.15.62:0/429438282 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56740343f0 con 0x7f5684143dd0 2024-03-23T11:24:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.792+0000 7f568a000640 1 -- 172.21.15.62:0/429438282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684143dd0 msgr2=0x7f568415abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.792+0000 7f568a000640 1 --2- 172.21.15.62:0/429438282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684143dd0 0x7f568415abe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5674009e50 tx=0x7f567402bf60 comp rx=0 tx=0).stop 2024-03-23T11:24:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.792+0000 7f568a000640 1 -- 172.21.15.62:0/429438282 shutdown_connections 2024-03-23T11:24:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.792+0000 7f568a000640 1 --2- 172.21.15.62:0/429438282 >> v2:172.21.15.120:3300/0 conn(0x7f56840b2f30 0x7f56840704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.792+0000 7f568a000640 1 --2- 172.21.15.62:0/429438282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684143dd0 0x7f568415abe0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.792+0000 7f568a000640 1 -- 172.21.15.62:0/429438282 >> 172.21.15.62:0/429438282 conn(0x7f5684066050 msgr2=0x7f5684069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.792+0000 7f568a000640 1 -- 172.21.15.62:0/429438282 shutdown_connections 2024-03-23T11:24:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.792+0000 7f568a000640 1 -- 172.21.15.62:0/429438282 wait complete. 2024-03-23T11:24:10.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.794+0000 7f568a000640 1 Processor -- start 2024-03-23T11:24:10.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.794+0000 7f568a000640 1 -- start start 2024-03-23T11:24:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.795+0000 7f568a000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684070090 0x7f56840c3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.795+0000 7f568a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56840b2f30 0x7f568413fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.795+0000 7f568a000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5684143dd0 0x7f56840c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.795+0000 7f568a000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56840c6120 con 0x7f56840b2f30 2024-03-23T11:24:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.795+0000 7f568a000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56840c4a20 con 0x7f5684143dd0 2024-03-23T11:24:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.795+0000 7f568a000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56840c4220 con 0x7f5684070090 2024-03-23T11:24:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.796+0000 7f5683e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56840b2f30 0x7f568413fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.796+0000 7f5683e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56840b2f30 0x7f568413fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59290/0 (socket says 172.21.15.62:59290) 2024-03-23T11:24:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.796+0000 7f5683e00640 1 -- 172.21.15.62:0/4079917443 learned_addr learned my addr 172.21.15.62:0/4079917443 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:10.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.796+0000 7f5688c00640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684070090 0x7f56840c3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:10.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.796+0000 7f5689600640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5684143dd0 0x7f56840c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:10.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.796+0000 7f5683e00640 1 -- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684070090 msgr2=0x7f56840c3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:10.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.796+0000 7f5683e00640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684070090 0x7f56840c3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:10.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.796+0000 7f5683e00640 1 -- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5684143dd0 msgr2=0x7f56840c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:10.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.796+0000 7f5683e00640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5684143dd0 0x7f56840c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:10.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.796+0000 7f5683e00640 1 -- 172.21.15.62:0/4079917443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5674009b30 con 0x7f56840b2f30 2024-03-23T11:24:10.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.796+0000 7f5683e00640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56840b2f30 0x7f568413fa60 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f567000c480 tx=0x7f567000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:10.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.797+0000 7f5680c00640 1 -- 172.21.15.62:0/4079917443 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56700052d0 con 0x7f56840b2f30 2024-03-23T11:24:10.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.797+0000 7f5680c00640 1 -- 172.21.15.62:0/4079917443 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5670005430 con 0x7f56840b2f30 2024-03-23T11:24:10.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.797+0000 7f568a000640 1 -- 172.21.15.62:0/4079917443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56840ae2f0 con 0x7f56840b2f30 2024-03-23T11:24:10.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.797+0000 7f5680c00640 1 -- 172.21.15.62:0/4079917443 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670012870 con 0x7f56840b2f30 2024-03-23T11:24:10.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.797+0000 7f568a000640 1 -- 172.21.15.62:0/4079917443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56840c4740 con 0x7f56840b2f30 2024-03-23T11:24:10.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.798+0000 7f568a000640 1 -- 172.21.15.62:0/4079917443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56840ae480 con 0x7f56840b2f30 2024-03-23T11:24:10.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.799+0000 7f5680c00640 1 -- 172.21.15.62:0/4079917443 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f56700129d0 con 0x7f56840b2f30 2024-03-23T11:24:10.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.800+0000 7f5680c00640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f566406a7a0 0x7f566406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:10.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.800+0000 7f5680c00640 1 -- 172.21.15.62:0/4079917443 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f56700888e0 con 0x7f56840b2f30 2024-03-23T11:24:10.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.801+0000 7f5688c00640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f566406a7a0 0x7f566406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:10.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.802+0000 7f5688c00640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f566406a7a0 0x7f566406cc60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5674035000 tx=0x7f5674030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:10.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:10.806+0000 7f5680c00640 1 -- 172.21.15.62:0/4079917443 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5670053d50 con 0x7f56840b2f30 2024-03-23T11:24:11.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.041+0000 7f568a000640 1 -- 172.21.15.62:0/4079917443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f56840cca40 con 0x7f56840b2f30 2024-03-23T11:24:11.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.053+0000 7f5680c00640 1 -- 172.21.15.62:0/4079917443 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v39) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5670057370 con 0x7f56840b2f30 2024-03-23T11:24:11.049 INFO:teuthology.orchestra.run.smithi062.stderr:new fs with metadata pool 10 and data pool 11 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.058+0000 7f568a000640 1 -- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f566406a7a0 msgr2=0x7f566406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.058+0000 7f568a000640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f566406a7a0 0x7f566406cc60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5674035000 tx=0x7f5674030000 comp rx=0 tx=0).stop 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.058+0000 7f568a000640 1 -- 172.21.15.62:0/4079917443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56840b2f30 msgr2=0x7f568413fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.058+0000 7f568a000640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56840b2f30 0x7f568413fa60 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f567000c480 tx=0x7f567000c940 comp rx=0 tx=0).stop 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.058+0000 7f568a000640 1 -- 172.21.15.62:0/4079917443 shutdown_connections 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.059+0000 7f568a000640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f566406a7a0 0x7f566406cc60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.059+0000 7f568a000640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56840b2f30 0x7f568413fa60 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.059+0000 7f568a000640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5684143dd0 0x7f56840c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.059+0000 7f568a000640 1 --2- 172.21.15.62:0/4079917443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5684070090 0x7f56840c3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.059+0000 7f568a000640 1 -- 172.21.15.62:0/4079917443 >> 172.21.15.62:0/4079917443 conn(0x7f5684066050 msgr2=0x7f568406c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.060+0000 7f568a000640 1 -- 172.21.15.62:0/4079917443 shutdown_connections 2024-03-23T11:24:11.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.060+0000 7f568a000640 1 -- 172.21.15.62:0/4079917443 wait complete. 2024-03-23T11:24:11.086 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:24:11.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.263+0000 7fe895200640 1 Processor -- start 2024-03-23T11:24:11.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.263+0000 7fe895200640 1 -- start start 2024-03-23T11:24:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.265+0000 7fe895200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe890143dd0 0x7fe89015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.265+0000 7fe895200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe8900b2f30 0x7fe8900704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.265+0000 7fe895200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe890070a30 con 0x7fe890070090 2024-03-23T11:24:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.265+0000 7fe895200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8900cd140 con 0x7fe8900b2f30 2024-03-23T11:24:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.265+0000 7fe895200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900cd3e0 con 0x7fe890143dd0 2024-03-23T11:24:11.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.265+0000 7fe88fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe8900b2f30 0x7fe8900704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.265+0000 7fe886a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe890070090 0x7fe8900b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:38592/0 (socket says 172.21.15.62:38592) 2024-03-23T11:24:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.265+0000 7fe886a00640 1 -- 172.21.15.62:0/1041979709 learned_addr learned my addr 172.21.15.62:0/1041979709 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.266+0000 7fe88f400640 1 --2- 172.21.15.62:0/1041979709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe890143dd0 0x7fe89015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.266+0000 7fe88fe00640 1 -- 172.21.15.62:0/1041979709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe890143dd0 msgr2=0x7fe89015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.266+0000 7fe88fe00640 1 --2- 172.21.15.62:0/1041979709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe890143dd0 0x7fe89015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.266+0000 7fe88fe00640 1 -- 172.21.15.62:0/1041979709 >> v1:172.21.15.62:6789/0 conn(0x7fe890070090 legacy=0x7fe8900b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.266+0000 7fe88fe00640 1 -- 172.21.15.62:0/1041979709 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900cd6c0 con 0x7fe8900b2f30 2024-03-23T11:24:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.266+0000 7fe88ea00640 1 -- 172.21.15.62:0/1041979709 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911604319 0 0) 0x7fe890070a30 con 0x7fe890070090 2024-03-23T11:24:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.266+0000 7fe88f400640 1 --2- 172.21.15.62:0/1041979709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe890143dd0 0x7fe89015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:11.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.267+0000 7fe88fe00640 1 --2- 172.21.15.62:0/1041979709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8900b2f30 0x7fe8900704f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe888009d90 tx=0x7fe88802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb76825c75209b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:11.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.267+0000 7fe88ea00640 1 -- 172.21.15.62:0/1041979709 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88802cc60 con 0x7fe8900b2f30 2024-03-23T11:24:11.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.267+0000 7fe88ea00640 1 -- 172.21.15.62:0/1041979709 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88802cdc0 con 0x7fe8900b2f30 2024-03-23T11:24:11.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.267+0000 7fe88ea00640 1 -- 172.21.15.62:0/1041979709 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888034660 con 0x7fe8900b2f30 2024-03-23T11:24:11.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.268+0000 7fe895200640 1 -- 172.21.15.62:0/1041979709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8900b2f30 msgr2=0x7fe8900704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.268+0000 7fe895200640 1 --2- 172.21.15.62:0/1041979709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8900b2f30 0x7fe8900704f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe888009d90 tx=0x7fe88802beb0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.268+0000 7fe895200640 1 -- 172.21.15.62:0/1041979709 shutdown_connections 2024-03-23T11:24:11.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.268+0000 7fe895200640 1 --2- 172.21.15.62:0/1041979709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8900b2f30 0x7fe8900704f0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.268+0000 7fe895200640 1 --2- 172.21.15.62:0/1041979709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe890143dd0 0x7fe89015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.268+0000 7fe895200640 1 -- 172.21.15.62:0/1041979709 >> 172.21.15.62:0/1041979709 conn(0x7fe890066050 msgr2=0x7fe890069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:11.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.268+0000 7fe895200640 1 -- 172.21.15.62:0/1041979709 shutdown_connections 2024-03-23T11:24:11.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.268+0000 7fe895200640 1 -- 172.21.15.62:0/1041979709 wait complete. 2024-03-23T11:24:11.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.270+0000 7fe895200640 1 Processor -- start 2024-03-23T11:24:11.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.270+0000 7fe895200640 1 -- start start 2024-03-23T11:24:11.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe895200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe890070090 0x7fe8900ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe895200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8900b2f30 0x7fe890159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe895200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe890143dd0 0x7fe8900ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe895200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900abd90 con 0x7fe8900b2f30 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe895200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900ac070 con 0x7fe890143dd0 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe895200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900ac350 con 0x7fe890070090 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe886a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8900b2f30 0x7fe890159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe886a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8900b2f30 0x7fe890159c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59304/0 (socket says 172.21.15.62:59304) 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe886a00640 1 -- 172.21.15.62:0/2016882432 learned_addr learned my addr 172.21.15.62:0/2016882432 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe88f400640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe890070090 0x7fe8900ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe886a00640 1 -- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe890070090 msgr2=0x7fe8900ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe886a00640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe890070090 0x7fe8900ace30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe88fe00640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe890143dd0 0x7fe8900ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe886a00640 1 -- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe890143dd0 msgr2=0x7fe8900ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe886a00640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe890143dd0 0x7fe8900ab490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.271+0000 7fe886a00640 1 -- 172.21.15.62:0/2016882432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe888009a70 con 0x7fe8900b2f30 2024-03-23T11:24:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.272+0000 7fe886a00640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8900b2f30 0x7fe890159c60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe87800c4d0 tx=0x7fe87800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.272+0000 7fe887e00640 1 -- 172.21.15.62:0/2016882432 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8780074e0 con 0x7fe8900b2f30 2024-03-23T11:24:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.272+0000 7fe887e00640 1 -- 172.21.15.62:0/2016882432 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe878007660 con 0x7fe8900b2f30 2024-03-23T11:24:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.272+0000 7fe887e00640 1 -- 172.21.15.62:0/2016882432 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe878004a20 con 0x7fe8900b2f30 2024-03-23T11:24:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.272+0000 7fe895200640 1 -- 172.21.15.62:0/2016882432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8900ac630 con 0x7fe8900b2f30 2024-03-23T11:24:11.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.272+0000 7fe895200640 1 -- 172.21.15.62:0/2016882432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89015a1a0 con 0x7fe8900b2f30 2024-03-23T11:24:11.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.273+0000 7fe895200640 1 -- 172.21.15.62:0/2016882432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8900acb50 con 0x7fe8900b2f30 2024-03-23T11:24:11.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.274+0000 7fe887e00640 1 -- 172.21.15.62:0/2016882432 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe878004b80 con 0x7fe8900b2f30 2024-03-23T11:24:11.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.275+0000 7fe887e00640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe86c06a7a0 0x7fe86c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.275+0000 7fe887e00640 1 -- 172.21.15.62:0/2016882432 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe878087f90 con 0x7fe8900b2f30 2024-03-23T11:24:11.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.276+0000 7fe88f400640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe86c06a7a0 0x7fe86c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.277+0000 7fe88f400640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe86c06a7a0 0x7fe86c06cc60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe88000b840 tx=0x7fe880008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:11.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.280+0000 7fe887e00640 1 -- 172.21.15.62:0/2016882432 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe8900acb50 con 0x7fe8900b2f30 2024-03-23T11:24:11.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.473+0000 7fe895200640 1 -- 172.21.15.62:0/2016882432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8900acb50 con 0x7fe8900b2f30 2024-03-23T11:24:11.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.475+0000 7fe887e00640 1 -- 172.21.15.62:0/2016882432 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16113 (secure 0 0 0) 0x7fe8900acb50 con 0x7fe8900b2f30 2024-03-23T11:24:11.470 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:11.470 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":73,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:11.043501+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:24:08.688706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-23T11:24:10.072774+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:24:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.477+0000 7fe895200640 1 -- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe86c06a7a0 msgr2=0x7fe86c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.477+0000 7fe895200640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe86c06a7a0 0x7fe86c06cc60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe88000b840 tx=0x7fe880008040 comp rx=0 tx=0).stop 2024-03-23T11:24:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.477+0000 7fe895200640 1 -- 172.21.15.62:0/2016882432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8900b2f30 msgr2=0x7fe890159c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.477+0000 7fe895200640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8900b2f30 0x7fe890159c60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe87800c4d0 tx=0x7fe87800c990 comp rx=0 tx=0).stop 2024-03-23T11:24:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.477+0000 7fe895200640 1 -- 172.21.15.62:0/2016882432 shutdown_connections 2024-03-23T11:24:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.477+0000 7fe895200640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe86c06a7a0 0x7fe86c06cc60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.477+0000 7fe895200640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8900b2f30 0x7fe890159c60 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.477+0000 7fe895200640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe890143dd0 0x7fe8900ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.477+0000 7fe895200640 1 --2- 172.21.15.62:0/2016882432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe890070090 0x7fe8900ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.477+0000 7fe895200640 1 -- 172.21.15.62:0/2016882432 >> 172.21.15.62:0/2016882432 conn(0x7fe890066050 msgr2=0x7fe89015cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:11.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.478+0000 7fe895200640 1 -- 172.21.15.62:0/2016882432 shutdown_connections 2024-03-23T11:24:11.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.478+0000 7fe895200640 1 -- 172.21.15.62:0/2016882432 wait complete. 2024-03-23T11:24:11.490 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:24:11.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.685+0000 7f5bcce00640 1 Processor -- start 2024-03-23T11:24:11.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.685+0000 7f5bcce00640 1 -- start start 2024-03-23T11:24:11.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.687+0000 7f5bcce00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5bc8143dd0 0x7f5bc815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.687+0000 7f5bcce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bc80b2f30 0x7f5bc8070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.687+0000 7f5bcce00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc80cd140 con 0x7f5bc8070090 2024-03-23T11:24:11.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.687+0000 7f5bcce00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc8070a60 con 0x7f5bc8143dd0 2024-03-23T11:24:11.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.687+0000 7f5bcce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80cd3e0 con 0x7f5bc80b2f30 2024-03-23T11:24:11.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.687+0000 7f5bc6a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5bc8070090 0x7f5bc80b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:38608/0 (socket says 172.21.15.62:38608) 2024-03-23T11:24:11.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.687+0000 7f5bc6a00640 1 -- 172.21.15.62:0/1928911036 learned_addr learned my addr 172.21.15.62:0/1928911036 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:11.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc7400640 1 --2- 172.21.15.62:0/1928911036 >> v2:172.21.15.120:3300/0 conn(0x7f5bc8143dd0 0x7f5bc815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093305160 0 0) 0x7f5bc80cd140 con 0x7f5bc8070090 2024-03-23T11:24:11.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bac003520 con 0x7f5bc8070090 2024-03-23T11:24:11.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc7e00640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bc80b2f30 0x7f5bc8070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2228923106 0 0) 0x7f5bac003520 con 0x7f5bc8070090 2024-03-23T11:24:11.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bc80b2f30 msgr2=0x7f5bc8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:11.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bc80b2f30 0x7f5bc8070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 >> v2:172.21.15.120:3300/0 conn(0x7f5bc8143dd0 msgr2=0x7f5bc815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:11.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 --2- 172.21.15.62:0/1928911036 >> v2:172.21.15.120:3300/0 conn(0x7f5bc8143dd0 0x7f5bc815abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc80cd6c0 con 0x7f5bc8070090 2024-03-23T11:24:11.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f5bb4001670 con 0x7f5bc8070090 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 >> v1:172.21.15.62:6789/0 conn(0x7f5bc8070090 legacy=0x7f5bc80b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bac003970 0x7f5bac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bac0047c0 0x7f5bac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bc80b2f30 0x7f5bac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc8070a60 con 0x7f5bac003970 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80cd3e0 con 0x7f5bac0047c0 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.688+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac00b980 con 0x7f5bc80b2f30 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.689+0000 7f5bc7400640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bac003970 0x7f5bac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.689+0000 7f5bc6a00640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bac0047c0 0x7f5bac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.689+0000 7f5bc7e00640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bc80b2f30 0x7f5bac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.689+0000 7f5bc7400640 1 -- 172.21.15.62:0/1928911036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bc80b2f30 msgr2=0x7f5bac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.689+0000 7f5bc7400640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bc80b2f30 0x7f5bac005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.689+0000 7f5bc7400640 1 -- 172.21.15.62:0/1928911036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bac0047c0 msgr2=0x7f5bac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:11.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.689+0000 7f5bc7400640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bac0047c0 0x7f5bac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.689+0000 7f5bc7400640 1 -- 172.21.15.62:0/1928911036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bac003520 con 0x7f5bac003970 2024-03-23T11:24:11.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.690+0000 7f5bc7400640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bac003970 0x7f5bac003d90 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009ac0 tx=0x7f5bac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6779690289d6549b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:11.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.691+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb802c710 con 0x7f5bac003970 2024-03-23T11:24:11.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.691+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb802c870 con 0x7f5bac003970 2024-03-23T11:24:11.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.691+0000 7f5bc6000640 1 -- 172.21.15.62:0/1928911036 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb802cc00 con 0x7f5bac003970 2024-03-23T11:24:11.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.691+0000 7f5bcce00640 1 -- 172.21.15.62:0/1928911036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bac003970 msgr2=0x7f5bac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.691+0000 7f5bcce00640 1 --2- 172.21.15.62:0/1928911036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bac003970 0x7f5bac003d90 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009ac0 tx=0x7f5bac0057f0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.691+0000 7f5bc7e00640 1 -- 172.21.15.62:0/1928911036 reap_dead start 2024-03-23T11:24:11.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.691+0000 7f5bcce00640 1 -- 172.21.15.62:0/1928911036 shutdown_connections 2024-03-23T11:24:11.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.691+0000 7f5bcce00640 1 -- 172.21.15.62:0/1928911036 >> 172.21.15.62:0/1928911036 conn(0x7f5bc8066050 msgr2=0x7f5bc8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:11.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.691+0000 7f5bcce00640 1 -- 172.21.15.62:0/1928911036 shutdown_connections 2024-03-23T11:24:11.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.691+0000 7f5bcce00640 1 -- 172.21.15.62:0/1928911036 wait complete. 2024-03-23T11:24:11.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.693+0000 7f5bcce00640 1 Processor -- start 2024-03-23T11:24:11.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.693+0000 7f5bcce00640 1 -- start start 2024-03-23T11:24:11.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.694+0000 7f5bcce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bac003970 0x7f5bc813f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.694+0000 7f5bcce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc80af090 0x7f5bc8159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.694+0000 7f5bcce00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bc80af510 0x7f5bc8159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.694+0000 7f5bcce00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc81f1120 con 0x7f5bc80af090 2024-03-23T11:24:11.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.694+0000 7f5bcce00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc81f13c0 con 0x7f5bc80af510 2024-03-23T11:24:11.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.694+0000 7f5bcce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc81f16a0 con 0x7f5bac003970 2024-03-23T11:24:11.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.694+0000 7f5bc6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc80af090 0x7f5bc8159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.695+0000 7f5bc6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc80af090 0x7f5bc8159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59330/0 (socket says 172.21.15.62:59330) 2024-03-23T11:24:11.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.695+0000 7f5bc6a00640 1 -- 172.21.15.62:0/2762269808 learned_addr learned my addr 172.21.15.62:0/2762269808 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.695+0000 7f5bc7e00640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bc80af510 0x7f5bc8159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.695+0000 7f5bc7400640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bac003970 0x7f5bc813f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.695+0000 7f5bc6a00640 1 -- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bac003970 msgr2=0x7f5bc813f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.695+0000 7f5bc6a00640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bac003970 0x7f5bc813f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.695+0000 7f5bc6a00640 1 -- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bc80af510 msgr2=0x7f5bc8159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.695+0000 7f5bc6a00640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bc80af510 0x7f5bc8159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.695+0000 7f5bc6a00640 1 -- 172.21.15.62:0/2762269808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc81f1980 con 0x7f5bc80af090 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.695+0000 7f5bc6a00640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc80af090 0x7f5bc8159850 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400e8f0 tx=0x7f5bb400edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.696+0000 7f5bbf400640 1 -- 172.21.15.62:0/2762269808 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb400c8d0 con 0x7f5bc80af090 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.696+0000 7f5bbf400640 1 -- 172.21.15.62:0/2762269808 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb400ca50 con 0x7f5bc80af090 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.696+0000 7f5bbf400640 1 -- 172.21.15.62:0/2762269808 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb400ce00 con 0x7f5bc80af090 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.696+0000 7f5bcce00640 1 -- 172.21.15.62:0/2762269808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc81f1c60 con 0x7f5bc80af090 2024-03-23T11:24:11.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.696+0000 7f5bcce00640 1 -- 172.21.15.62:0/2762269808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc813fd90 con 0x7f5bc80af090 2024-03-23T11:24:11.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.697+0000 7f5bcce00640 1 -- 172.21.15.62:0/2762269808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc81f1ea0 con 0x7f5bc80af090 2024-03-23T11:24:11.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.698+0000 7f5bbf400640 1 -- 172.21.15.62:0/2762269808 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5bb401d460 con 0x7f5bc80af090 2024-03-23T11:24:11.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.699+0000 7f5bbf400640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ba806a7d0 0x7f5ba806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:11.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.699+0000 7f5bbf400640 1 -- 172.21.15.62:0/2762269808 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5bb4088de0 con 0x7f5bc80af090 2024-03-23T11:24:11.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.700+0000 7f5bc7400640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ba806a7d0 0x7f5ba806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:11.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.701+0000 7f5bc7400640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ba806a7d0 0x7f5ba806cc90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8036000 tx=0x7f5bb8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:11.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.704+0000 7f5bbf400640 1 -- 172.21.15.62:0/2762269808 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5bc81f1ea0 con 0x7f5bc80af090 2024-03-23T11:24:11.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-03-23T11:24:11.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.906+0000 7f5bcce00640 1 -- 172.21.15.62:0/2762269808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bc81f1ea0 con 0x7f5bc80af090 2024-03-23T11:24:11.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.909+0000 7f5bbf400640 1 -- 172.21.15.62:0/2762269808 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16113 (secure 0 0 0) 0x7f5bc81f1ea0 con 0x7f5bc80af090 2024-03-23T11:24:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:11.905 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":73,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:11.043501+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:24:08.688706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-23T11:24:10.072774+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:24:11.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 -- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ba806a7d0 msgr2=0x7f5ba806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ba806a7d0 0x7f5ba806cc90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8036000 tx=0x7f5bb8030000 comp rx=0 tx=0).stop 2024-03-23T11:24:11.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 -- 172.21.15.62:0/2762269808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc80af090 msgr2=0x7f5bc8159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:11.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc80af090 0x7f5bc8159850 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400e8f0 tx=0x7f5bb400edb0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 -- 172.21.15.62:0/2762269808 shutdown_connections 2024-03-23T11:24:11.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ba806a7d0 0x7f5ba806cc90 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc80af090 0x7f5bc8159850 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bc80af510 0x7f5bc8159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 --2- 172.21.15.62:0/2762269808 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bac003970 0x7f5bc813f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:11.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 -- 172.21.15.62:0/2762269808 >> 172.21.15.62:0/2762269808 conn(0x7f5bc8066050 msgr2=0x7f5bc8141b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:11.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 -- 172.21.15.62:0/2762269808 shutdown_connections 2024-03-23T11:24:11.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:11.912+0000 7f5bcce00640 1 -- 172.21.15.62:0/2762269808 wait complete. 2024-03-23T11:24:11.927 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T11:24:12.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.160+0000 7f4070e00640 1 Processor -- start 2024-03-23T11:24:12.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.160+0000 7f4070e00640 1 -- start start 2024-03-23T11:24:12.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.162+0000 7f4070e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f406c159490 0x7f406c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:12.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.162+0000 7f4070e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f406c0ce5f0 0x7f406c0b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:12.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.162+0000 7f4070e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c0ced90 con 0x7f406c15da90 2024-03-23T11:24:12.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.162+0000 7f4070e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406c06e3e0 con 0x7f406c159490 2024-03-23T11:24:12.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.162+0000 7f4070e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c06c1c0 con 0x7f406c0ce5f0 2024-03-23T11:24:12.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.163+0000 7f406b400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f406c159490 0x7f406c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:12.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.163+0000 7f406b400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f406c159490 0x7f406c06cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:59292/0 (socket says 172.21.15.62:59292) 2024-03-23T11:24:12.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.163+0000 7f406b400640 1 -- 172.21.15.62:0/3968177437 learned_addr learned my addr 172.21.15.62:0/3968177437 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:12.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.163+0000 7f406be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f406c0ce5f0 0x7f406c0b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:12.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.163+0000 7f406b400640 1 -- 172.21.15.62:0/3968177437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f406c0ce5f0 msgr2=0x7f406c0b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:12.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.163+0000 7f406b400640 1 --2- 172.21.15.62:0/3968177437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f406c0ce5f0 0x7f406c0b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:12.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.163+0000 7f406b400640 1 -- 172.21.15.62:0/3968177437 >> v1:172.21.15.120:6789/0 conn(0x7f406c15da90 legacy=0x7f406c0cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:12.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.163+0000 7f406b400640 1 -- 172.21.15.62:0/3968177437 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c06e100 con 0x7f406c159490 2024-03-23T11:24:12.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.165+0000 7f406b400640 1 --2- 172.21.15.62:0/3968177437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f406c159490 0x7f406c06cc60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f405c009870 tx=0x7f405c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f48ccb563cc3d77d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:12.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.165+0000 7f406a000640 1 -- 172.21.15.62:0/3968177437 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c031640 con 0x7f406c159490 2024-03-23T11:24:12.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.165+0000 7f406a000640 1 -- 172.21.15.62:0/3968177437 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405c0317a0 con 0x7f406c159490 2024-03-23T11:24:12.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.165+0000 7f406a000640 1 -- 172.21.15.62:0/3968177437 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c031b30 con 0x7f406c159490 2024-03-23T11:24:12.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.166+0000 7f4070e00640 1 -- 172.21.15.62:0/3968177437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f406c159490 msgr2=0x7f406c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:12.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.166+0000 7f4070e00640 1 --2- 172.21.15.62:0/3968177437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f406c159490 0x7f406c06cc60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f405c009870 tx=0x7f405c02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:12.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.166+0000 7f4070e00640 1 -- 172.21.15.62:0/3968177437 shutdown_connections 2024-03-23T11:24:12.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.166+0000 7f4070e00640 1 --2- 172.21.15.62:0/3968177437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f406c0ce5f0 0x7f406c0b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:12.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.166+0000 7f4070e00640 1 --2- 172.21.15.62:0/3968177437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f406c159490 0x7f406c06cc60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:12.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.166+0000 7f4070e00640 1 -- 172.21.15.62:0/3968177437 >> 172.21.15.62:0/3968177437 conn(0x7f406c066050 msgr2=0x7f406c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:12.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.166+0000 7f4070e00640 1 -- 172.21.15.62:0/3968177437 shutdown_connections 2024-03-23T11:24:12.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.166+0000 7f4070e00640 1 -- 172.21.15.62:0/3968177437 wait complete. 2024-03-23T11:24:12.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.168+0000 7f4070e00640 1 Processor -- start 2024-03-23T11:24:12.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.168+0000 7f4070e00640 1 -- start start 2024-03-23T11:24:12.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.169+0000 7f4070e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f406c0ce5f0 0x7f406c151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:12.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.169+0000 7f4070e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f406c159490 0x7f406c0c3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:12.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.169+0000 7f4070e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f406c15da90 0x7f406c157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:12.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.169+0000 7f4070e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c151ea0 con 0x7f406c15da90 2024-03-23T11:24:12.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.169+0000 7f4070e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c152a40 con 0x7f406c159490 2024-03-23T11:24:12.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.169+0000 7f4070e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c15b960 con 0x7f406c0ce5f0 2024-03-23T11:24:12.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.169+0000 7f406be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f406c15da90 0x7f406c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:12.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.169+0000 7f406be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f406c15da90 0x7f406c157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59334/0 (socket says 172.21.15.62:59334) 2024-03-23T11:24:12.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.169+0000 7f406be00640 1 -- 172.21.15.62:0/3268918753 learned_addr learned my addr 172.21.15.62:0/3268918753 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:12.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.169+0000 7f406b400640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f406c0ce5f0 0x7f406c151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:12.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.169+0000 7f406aa00640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f406c159490 0x7f406c0c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:12.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.170+0000 7f406be00640 1 -- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f406c0ce5f0 msgr2=0x7f406c151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:12.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.170+0000 7f406be00640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f406c0ce5f0 0x7f406c151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:12.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.170+0000 7f406be00640 1 -- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f406c159490 msgr2=0x7f406c0c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:12.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.170+0000 7f406be00640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f406c159490 0x7f406c0c3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:12.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.170+0000 7f406be00640 1 -- 172.21.15.62:0/3268918753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c0094d0 con 0x7f406c15da90 2024-03-23T11:24:12.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.170+0000 7f406be00640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f406c15da90 0x7f406c157010 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f4064009e30 tx=0x7f406400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:12.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.170+0000 7f4063400640 1 -- 172.21.15.62:0/3268918753 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064009ca0 con 0x7f406c15da90 2024-03-23T11:24:12.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.170+0000 7f4063400640 1 -- 172.21.15.62:0/3268918753 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406400f040 con 0x7f406c15da90 2024-03-23T11:24:12.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.170+0000 7f4070e00640 1 -- 172.21.15.62:0/3268918753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c15bc40 con 0x7f406c15da90 2024-03-23T11:24:12.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.170+0000 7f4063400640 1 -- 172.21.15.62:0/3268918753 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40640146c0 con 0x7f406c15da90 2024-03-23T11:24:12.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.170+0000 7f4070e00640 1 -- 172.21.15.62:0/3268918753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c152f70 con 0x7f406c15da90 2024-03-23T11:24:12.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.171+0000 7f4070e00640 1 -- 172.21.15.62:0/3268918753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c153190 con 0x7f406c15da90 2024-03-23T11:24:12.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.173+0000 7f4063400640 1 -- 172.21.15.62:0/3268918753 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4064014820 con 0x7f406c15da90 2024-03-23T11:24:12.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.173+0000 7f4063400640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f404c06a7d0 0x7f404c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:12.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.174+0000 7f4063400640 1 -- 172.21.15.62:0/3268918753 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4064089c60 con 0x7f406c15da90 2024-03-23T11:24:12.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.174+0000 7f406b400640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f404c06a7d0 0x7f404c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:12.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.176+0000 7f406b400640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f404c06a7d0 0x7f404c06cc90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f405c036000 tx=0x7f405c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:12.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.178+0000 7f4063400640 1 -- 172.21.15.62:0/3268918753 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4064055080 con 0x7f406c15da90 2024-03-23T11:24:12.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:12.417+0000 7f4070e00640 1 -- 172.21.15.62:0/3268918753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f406c06d320 con 0x7f406c15da90 2024-03-23T11:24:13.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.066+0000 7f4063400640 1 -- 172.21.15.62:0/3268918753 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v42) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4064018020 con 0x7f406c15da90 2024-03-23T11:24:13.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 -- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f404c06a7d0 msgr2=0x7f404c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:13.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f404c06a7d0 0x7f404c06cc90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f405c036000 tx=0x7f405c03c040 comp rx=0 tx=0).stop 2024-03-23T11:24:13.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 -- 172.21.15.62:0/3268918753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f406c15da90 msgr2=0x7f406c157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:13.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f406c15da90 0x7f406c157010 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f4064009e30 tx=0x7f406400c490 comp rx=0 tx=0).stop 2024-03-23T11:24:13.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 -- 172.21.15.62:0/3268918753 shutdown_connections 2024-03-23T11:24:13.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f404c06a7d0 0x7f404c06cc90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:13.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f406c15da90 0x7f406c157010 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:13.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f406c159490 0x7f406c0c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:13.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 --2- 172.21.15.62:0/3268918753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f406c0ce5f0 0x7f406c151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:13.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 -- 172.21.15.62:0/3268918753 >> 172.21.15.62:0/3268918753 conn(0x7f406c066050 msgr2=0x7f406c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:13.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 -- 172.21.15.62:0/3268918753 shutdown_connections 2024-03-23T11:24:13.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.070+0000 7f4070e00640 1 -- 172.21.15.62:0/3268918753 wait complete. 2024-03-23T11:24:13.086 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T11:24:13.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T11:24:13.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.320+0000 7fb09b000640 1 Processor -- start 2024-03-23T11:24:13.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.320+0000 7fb09b000640 1 -- start start 2024-03-23T11:24:13.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.321+0000 7fb09b000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb0940b15e0 0x7fb094161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:13.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.321+0000 7fb09b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb09406c300 0x7fb09406c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:13.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.321+0000 7fb09b000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0940ae0c0 con 0x7fb0940b2f90 2024-03-23T11:24:13.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.321+0000 7fb09b000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0940b2740 con 0x7fb0940b15e0 2024-03-23T11:24:13.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.321+0000 7fb09b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0941623b0 con 0x7fb09406c300 2024-03-23T11:24:13.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.322+0000 7fb08be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb0940b15e0 0x7fb094161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:13.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.322+0000 7fb08be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb0940b15e0 0x7fb094161e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59342/0 (socket says 172.21.15.62:59342) 2024-03-23T11:24:13.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.322+0000 7fb08be00640 1 -- 172.21.15.62:0/2036998304 learned_addr learned my addr 172.21.15.62:0/2036998304 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:13.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.322+0000 7fb08be00640 1 -- 172.21.15.62:0/2036998304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb09406c300 msgr2=0x7fb09406c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:13.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.322+0000 7fb099000640 1 --2- 172.21.15.62:0/2036998304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb09406c300 0x7fb09406c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:13.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.322+0000 7fb08be00640 1 --2- 172.21.15.62:0/2036998304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb09406c300 0x7fb09406c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:13.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.322+0000 7fb08be00640 1 -- 172.21.15.62:0/2036998304 >> v1:172.21.15.120:6789/0 conn(0x7fb0940b2f90 legacy=0x7fb0940b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:13.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.322+0000 7fb08be00640 1 -- 172.21.15.62:0/2036998304 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0940b1100 con 0x7fb0940b15e0 2024-03-23T11:24:13.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.324+0000 7fb08be00640 1 --2- 172.21.15.62:0/2036998304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0940b15e0 0x7fb094161e70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009870 tx=0x7fb07c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca4b8753204f5b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:13.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.324+0000 7fb093400640 1 -- 172.21.15.62:0/2036998304 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c031640 con 0x7fb0940b15e0 2024-03-23T11:24:13.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.324+0000 7fb093400640 1 -- 172.21.15.62:0/2036998304 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07c0317a0 con 0x7fb0940b15e0 2024-03-23T11:24:13.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.324+0000 7fb093400640 1 -- 172.21.15.62:0/2036998304 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c031b30 con 0x7fb0940b15e0 2024-03-23T11:24:13.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.324+0000 7fb09b000640 1 -- 172.21.15.62:0/2036998304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0940b15e0 msgr2=0x7fb094161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:13.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.324+0000 7fb09b000640 1 --2- 172.21.15.62:0/2036998304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0940b15e0 0x7fb094161e70 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009870 tx=0x7fb07c02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:13.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.325+0000 7fb09b000640 1 -- 172.21.15.62:0/2036998304 shutdown_connections 2024-03-23T11:24:13.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.325+0000 7fb09b000640 1 --2- 172.21.15.62:0/2036998304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb09406c300 0x7fb09406c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:13.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.325+0000 7fb09b000640 1 --2- 172.21.15.62:0/2036998304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0940b15e0 0x7fb094161e70 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:13.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.325+0000 7fb09b000640 1 -- 172.21.15.62:0/2036998304 >> 172.21.15.62:0/2036998304 conn(0x7fb0940b02e0 msgr2=0x7fb0940b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:13.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.325+0000 7fb09b000640 1 -- 172.21.15.62:0/2036998304 shutdown_connections 2024-03-23T11:24:13.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.325+0000 7fb09b000640 1 -- 172.21.15.62:0/2036998304 wait complete. 2024-03-23T11:24:13.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.326+0000 7fb09b000640 1 Processor -- start 2024-03-23T11:24:13.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.327+0000 7fb09b000640 1 -- start start 2024-03-23T11:24:13.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.327+0000 7fb09b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb09406c300 0x7fb094155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:13.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb09b000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0940b15e0 0x7fb0941486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:13.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb09b000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0940b2f90 0x7fb0941f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:13.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb09b000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09415b1e0 con 0x7fb0940b15e0 2024-03-23T11:24:13.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb09b000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094148bf0 con 0x7fb0940b2f90 2024-03-23T11:24:13.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb09b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09415e0c0 con 0x7fb09406c300 2024-03-23T11:24:13.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb08be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0940b15e0 0x7fb0941486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:13.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb08be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0940b15e0 0x7fb0941486b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59354/0 (socket says 172.21.15.62:59354) 2024-03-23T11:24:13.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb08be00640 1 -- 172.21.15.62:0/3158038796 learned_addr learned my addr 172.21.15.62:0/3158038796 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:13.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb08be00640 1 -- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb09406c300 msgr2=0x7fb094155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:13.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb093e00640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb09406c300 0x7fb094155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:13.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb099000640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0940b2f90 0x7fb0941f6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:13.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb08be00640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb09406c300 0x7fb094155ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:13.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb08be00640 1 -- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0940b2f90 msgr2=0x7fb0941f6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:13.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb08be00640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0940b2f90 0x7fb0941f6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:13.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb08be00640 1 -- 172.21.15.62:0/3158038796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c0094d0 con 0x7fb0940b15e0 2024-03-23T11:24:13.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.328+0000 7fb099000640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0940b2f90 0x7fb0941f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:13.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.329+0000 7fb08be00640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0940b15e0 0x7fb0941486b0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009870 tx=0x7fb07c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:13.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.329+0000 7fb090c00640 1 -- 172.21.15.62:0/3158038796 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c03ac60 con 0x7fb0940b15e0 2024-03-23T11:24:13.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.329+0000 7fb090c00640 1 -- 172.21.15.62:0/3158038796 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07c03ade0 con 0x7fb0940b15e0 2024-03-23T11:24:13.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.329+0000 7fb09b000640 1 -- 172.21.15.62:0/3158038796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09415af00 con 0x7fb0940b15e0 2024-03-23T11:24:13.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.329+0000 7fb090c00640 1 -- 172.21.15.62:0/3158038796 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c044630 con 0x7fb0940b15e0 2024-03-23T11:24:13.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.329+0000 7fb09b000640 1 -- 172.21.15.62:0/3158038796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb094159350 con 0x7fb0940b15e0 2024-03-23T11:24:13.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.331+0000 7fb09b000640 1 -- 172.21.15.62:0/3158038796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094073860 con 0x7fb0940b15e0 2024-03-23T11:24:13.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.332+0000 7fb090c00640 1 -- 172.21.15.62:0/3158038796 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb07c02f050 con 0x7fb0940b15e0 2024-03-23T11:24:13.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.333+0000 7fb090c00640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07006a7a0 0x7fb07006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:13.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.333+0000 7fb090c00640 1 -- 172.21.15.62:0/3158038796 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb07c0ab0c0 con 0x7fb0940b15e0 2024-03-23T11:24:13.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.334+0000 7fb093e00640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07006a7a0 0x7fb07006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:13.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.335+0000 7fb093e00640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07006a7a0 0x7fb07006cc60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb084006fd0 tx=0x7fb084008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:13.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.338+0000 7fb090c00640 1 -- 172.21.15.62:0/3158038796 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb07c076560 con 0x7fb0940b15e0 2024-03-23T11:24:13.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:13.579+0000 7fb09b000640 1 -- 172.21.15.62:0/3158038796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb094066c90 con 0x7fb0940b15e0 2024-03-23T11:24:14.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.069+0000 7fb090c00640 1 -- 172.21.15.62:0/3158038796 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v43) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb07c079b80 con 0x7fb0940b15e0 2024-03-23T11:24:14.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 -- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07006a7a0 msgr2=0x7fb07006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07006a7a0 0x7fb07006cc60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb084006fd0 tx=0x7fb084008040 comp rx=0 tx=0).stop 2024-03-23T11:24:14.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 -- 172.21.15.62:0/3158038796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0940b15e0 msgr2=0x7fb0941486b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0940b15e0 0x7fb0941486b0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009870 tx=0x7fb07c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 -- 172.21.15.62:0/3158038796 shutdown_connections 2024-03-23T11:24:14.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb07006a7a0 0x7fb07006cc60 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0940b15e0 0x7fb0941486b0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0940b2f90 0x7fb0941f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 --2- 172.21.15.62:0/3158038796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb09406c300 0x7fb094155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 -- 172.21.15.62:0/3158038796 >> 172.21.15.62:0/3158038796 conn(0x7fb0940b02e0 msgr2=0x7fb09406d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:14.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 -- 172.21.15.62:0/3158038796 shutdown_connections 2024-03-23T11:24:14.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.073+0000 7fb09b000640 1 -- 172.21.15.62:0/3158038796 wait complete. 2024-03-23T11:24:14.090 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T11:24:14.090 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:24:14.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.313+0000 7fbf54c00640 1 Processor -- start 2024-03-23T11:24:14.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.314+0000 7fbf54c00640 1 -- start start 2024-03-23T11:24:14.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.315+0000 7fbf54c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbf50160310 0x7fbf500ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.315+0000 7fbf54c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf500ce250 0x7fbf500ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.315+0000 7fbf54c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf501608d0 con 0x7fbf5015c830 2024-03-23T11:24:14.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.315+0000 7fbf54c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf500addb0 con 0x7fbf50160310 2024-03-23T11:24:14.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.315+0000 7fbf54c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5015cc10 con 0x7fbf500ce250 2024-03-23T11:24:14.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.316+0000 7fbf46a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbf50160310 0x7fbf500ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.316+0000 7fbf4f400640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fbf5015c830 0x7fbf500ab980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:35948/0 (socket says 172.21.15.62:35948) 2024-03-23T11:24:14.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.316+0000 7fbf4f400640 1 -- 172.21.15.62:0/1482029239 learned_addr learned my addr 172.21.15.62:0/1482029239 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:14.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.316+0000 7fbf4fe00640 1 --2- 172.21.15.62:0/1482029239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf500ce250 0x7fbf500ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.317+0000 7fbf4ea00640 1 -- 172.21.15.62:0/1482029239 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708000328 0 0) 0x7fbf501608d0 con 0x7fbf5015c830 2024-03-23T11:24:14.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.317+0000 7fbf4ea00640 1 -- 172.21.15.62:0/1482029239 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf34003520 con 0x7fbf5015c830 2024-03-23T11:24:14.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.317+0000 7fbf4fe00640 1 -- 172.21.15.62:0/1482029239 >> v2:172.21.15.120:3300/0 conn(0x7fbf50160310 msgr2=0x7fbf500ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:14.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.317+0000 7fbf4fe00640 1 --2- 172.21.15.62:0/1482029239 >> v2:172.21.15.120:3300/0 conn(0x7fbf50160310 0x7fbf500ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.317+0000 7fbf4fe00640 1 -- 172.21.15.62:0/1482029239 >> v1:172.21.15.120:6789/0 conn(0x7fbf5015c830 legacy=0x7fbf500ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.317+0000 7fbf4fe00640 1 -- 172.21.15.62:0/1482029239 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf50160bb0 con 0x7fbf500ce250 2024-03-23T11:24:14.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.318+0000 7fbf4fe00640 1 --2- 172.21.15.62:0/1482029239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf500ce250 0x7fbf500ce6e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009d90 tx=0x7fbf4802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39b9e2604e938fe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:14.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.318+0000 7fbf4ea00640 1 -- 172.21.15.62:0/1482029239 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4802cc60 con 0x7fbf500ce250 2024-03-23T11:24:14.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.319+0000 7fbf4ea00640 1 -- 172.21.15.62:0/1482029239 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4802cdc0 con 0x7fbf500ce250 2024-03-23T11:24:14.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.319+0000 7fbf4ea00640 1 -- 172.21.15.62:0/1482029239 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48034640 con 0x7fbf500ce250 2024-03-23T11:24:14.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.319+0000 7fbf54c00640 1 -- 172.21.15.62:0/1482029239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf500ce250 msgr2=0x7fbf500ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.319+0000 7fbf54c00640 1 --2- 172.21.15.62:0/1482029239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf500ce250 0x7fbf500ce6e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009d90 tx=0x7fbf4802beb0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.319+0000 7fbf54c00640 1 -- 172.21.15.62:0/1482029239 shutdown_connections 2024-03-23T11:24:14.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.319+0000 7fbf54c00640 1 --2- 172.21.15.62:0/1482029239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf500ce250 0x7fbf500ce6e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.319+0000 7fbf54c00640 1 --2- 172.21.15.62:0/1482029239 >> v2:172.21.15.120:3300/0 conn(0x7fbf50160310 0x7fbf500ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.319+0000 7fbf54c00640 1 -- 172.21.15.62:0/1482029239 >> 172.21.15.62:0/1482029239 conn(0x7fbf50066050 msgr2=0x7fbf50069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:14.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.319+0000 7fbf54c00640 1 -- 172.21.15.62:0/1482029239 shutdown_connections 2024-03-23T11:24:14.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.319+0000 7fbf54c00640 1 -- 172.21.15.62:0/1482029239 wait complete. 2024-03-23T11:24:14.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.321+0000 7fbf54c00640 1 Processor -- start 2024-03-23T11:24:14.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.321+0000 7fbf54c00640 1 -- start start 2024-03-23T11:24:14.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.322+0000 7fbf54c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf500ce250 0x7fbf50159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.322+0000 7fbf54c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5015c830 0x7fbf501f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.322+0000 7fbf54c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf50160310 0x7fbf500c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.322+0000 7fbf54c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501598b0 con 0x7fbf5015c830 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.322+0000 7fbf54c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500c5640 con 0x7fbf50160310 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.322+0000 7fbf54c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500c58e0 con 0x7fbf500ce250 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.323+0000 7fbf46a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5015c830 0x7fbf501f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.323+0000 7fbf46a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5015c830 0x7fbf501f2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59370/0 (socket says 172.21.15.62:59370) 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.323+0000 7fbf46a00640 1 -- 172.21.15.62:0/2077364348 learned_addr learned my addr 172.21.15.62:0/2077364348 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.323+0000 7fbf4f400640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf500ce250 0x7fbf50159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.323+0000 7fbf4fe00640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf50160310 0x7fbf500c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.323+0000 7fbf46a00640 1 -- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf500ce250 msgr2=0x7fbf50159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.323+0000 7fbf46a00640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf500ce250 0x7fbf50159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.323+0000 7fbf46a00640 1 -- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf50160310 msgr2=0x7fbf500c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.323+0000 7fbf46a00640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf50160310 0x7fbf500c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.323+0000 7fbf46a00640 1 -- 172.21.15.62:0/2077364348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf48009a70 con 0x7fbf5015c830 2024-03-23T11:24:14.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.324+0000 7fbf46a00640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5015c830 0x7fbf501f2f70 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800e9b0 tx=0x7fbf3800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:14.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.324+0000 7fbf47e00640 1 -- 172.21.15.62:0/2077364348 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38009670 con 0x7fbf5015c830 2024-03-23T11:24:14.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.324+0000 7fbf47e00640 1 -- 172.21.15.62:0/2077364348 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3800c9a0 con 0x7fbf5015c830 2024-03-23T11:24:14.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.324+0000 7fbf47e00640 1 -- 172.21.15.62:0/2077364348 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3800cd50 con 0x7fbf5015c830 2024-03-23T11:24:14.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.324+0000 7fbf54c00640 1 -- 172.21.15.62:0/2077364348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf500c5bc0 con 0x7fbf5015c830 2024-03-23T11:24:14.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.324+0000 7fbf54c00640 1 -- 172.21.15.62:0/2077364348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf500c6180 con 0x7fbf5015c830 2024-03-23T11:24:14.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.325+0000 7fbf54c00640 1 -- 172.21.15.62:0/2077364348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf500b34c0 con 0x7fbf5015c830 2024-03-23T11:24:14.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.326+0000 7fbf47e00640 1 -- 172.21.15.62:0/2077364348 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbf38013470 con 0x7fbf5015c830 2024-03-23T11:24:14.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.327+0000 7fbf47e00640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 0x7fbf2c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.327+0000 7fbf47e00640 1 -- 172.21.15.62:0/2077364348 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf38088fa0 con 0x7fbf5015c830 2024-03-23T11:24:14.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.328+0000 7fbf4f400640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 0x7fbf2c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.329+0000 7fbf4f400640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 0x7fbf2c06cc60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbf400011e0 tx=0x7fbf40009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:14.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.331+0000 7fbf47e00640 1 -- 172.21.15.62:0/2077364348 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf380bc580 con 0x7fbf5015c830 2024-03-23T11:24:14.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.566+0000 7fbf54c00640 1 -- 172.21.15.62:0/2077364348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf50066bb0 con 0x7fbf5015c830 2024-03-23T11:24:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.568+0000 7fbf47e00640 1 -- 172.21.15.62:0/2077364348 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fbf3805ce60 con 0x7fbf5015c830 2024-03-23T11:24:14.563 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:14.564 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6618,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:24:14.062901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6621":{"gid":6621,"name":"d","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/3833782560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3833782560},{"type":"v1","addr":"172.21.15.120:6837","nonce":3833782560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:24:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 43 2024-03-23T11:24:14.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.570+0000 7fbf54c00640 1 -- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 msgr2=0x7fbf2c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.570+0000 7fbf54c00640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 0x7fbf2c06cc60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbf400011e0 tx=0x7fbf40009040 comp rx=0 tx=0).stop 2024-03-23T11:24:14.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.570+0000 7fbf54c00640 1 -- 172.21.15.62:0/2077364348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5015c830 msgr2=0x7fbf501f2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.570+0000 7fbf54c00640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5015c830 0x7fbf501f2f70 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800e9b0 tx=0x7fbf3800ee70 comp rx=0 tx=0).stop 2024-03-23T11:24:14.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.571+0000 7fbf54c00640 1 -- 172.21.15.62:0/2077364348 shutdown_connections 2024-03-23T11:24:14.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.571+0000 7fbf54c00640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 0x7fbf2c06cc60 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.571+0000 7fbf54c00640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5015c830 0x7fbf501f2f70 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.571+0000 7fbf54c00640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf50160310 0x7fbf500c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.571+0000 7fbf54c00640 1 --2- 172.21.15.62:0/2077364348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf500ce250 0x7fbf50159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.571+0000 7fbf54c00640 1 -- 172.21.15.62:0/2077364348 >> 172.21.15.62:0/2077364348 conn(0x7fbf50066050 msgr2=0x7fbf500ae000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:14.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.571+0000 7fbf54c00640 1 -- 172.21.15.62:0/2077364348 shutdown_connections 2024-03-23T11:24:14.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.571+0000 7fbf54c00640 1 -- 172.21.15.62:0/2077364348 wait complete. 2024-03-23T11:24:14.583 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:24:14.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.796+0000 7fad9f200640 1 Processor -- start 2024-03-23T11:24:14.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.796+0000 7fad9f200640 1 -- start start 2024-03-23T11:24:14.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.798+0000 7fad9f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 0x7fada015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.798+0000 7fad9f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fada0070090 0x7fada00704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.798+0000 7fad9f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada0070a30 con 0x7fada00b2820 2024-03-23T11:24:14.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.798+0000 7fad9f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada00cd140 con 0x7fada0070090 2024-03-23T11:24:14.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.798+0000 7fad9f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada00cd3e0 con 0x7fada0143dd0 2024-03-23T11:24:14.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.798+0000 7fad9de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 0x7fada015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.798+0000 7fad97e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fada0070090 0x7fada00704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.799+0000 7fad97e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fada0070090 0x7fada00704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:59344/0 (socket says 172.21.15.62:59344) 2024-03-23T11:24:14.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.799+0000 7fad97e00640 1 -- 172.21.15.62:0/2388036234 learned_addr learned my addr 172.21.15.62:0/2388036234 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:14.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.799+0000 7fad9de00640 1 --2- 172.21.15.62:0/2388036234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 0x7fada015abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:44130/0 (socket says 172.21.15.62:44130) 2024-03-23T11:24:14.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.799+0000 7fad97e00640 1 -- 172.21.15.62:0/2388036234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 msgr2=0x7fada015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:14.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.799+0000 7fad97e00640 1 --2- 172.21.15.62:0/2388036234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 0x7fada015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.799+0000 7fad97e00640 1 -- 172.21.15.62:0/2388036234 >> v1:172.21.15.120:6789/0 conn(0x7fada00b2820 legacy=0x7fada00b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.799+0000 7fad97e00640 1 -- 172.21.15.62:0/2388036234 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada00cd6c0 con 0x7fada0070090 2024-03-23T11:24:14.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.801+0000 7fad97e00640 1 --2- 172.21.15.62:0/2388036234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada0070090 0x7fada00704f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fad88009870 tx=0x7fad8802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1f39fbc7b2d5be5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:14.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.801+0000 7fad9d400640 1 -- 172.21.15.62:0/2388036234 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88030840 con 0x7fada0070090 2024-03-23T11:24:14.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.801+0000 7fad9d400640 1 -- 172.21.15.62:0/2388036234 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad880309a0 con 0x7fada0070090 2024-03-23T11:24:14.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.801+0000 7fad9d400640 1 -- 172.21.15.62:0/2388036234 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88030d30 con 0x7fada0070090 2024-03-23T11:24:14.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.802+0000 7fad9f200640 1 -- 172.21.15.62:0/2388036234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada0070090 msgr2=0x7fada00704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.802+0000 7fad9f200640 1 --2- 172.21.15.62:0/2388036234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada0070090 0x7fada00704f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fad88009870 tx=0x7fad8802b980 comp rx=0 tx=0).stop 2024-03-23T11:24:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.802+0000 7fad9f200640 1 -- 172.21.15.62:0/2388036234 shutdown_connections 2024-03-23T11:24:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.802+0000 7fad9f200640 1 --2- 172.21.15.62:0/2388036234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada0070090 0x7fada00704f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.802+0000 7fad9f200640 1 --2- 172.21.15.62:0/2388036234 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 0x7fada015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.802+0000 7fad9f200640 1 -- 172.21.15.62:0/2388036234 >> 172.21.15.62:0/2388036234 conn(0x7fada0066050 msgr2=0x7fada0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.802+0000 7fad9f200640 1 -- 172.21.15.62:0/2388036234 shutdown_connections 2024-03-23T11:24:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.802+0000 7fad9f200640 1 -- 172.21.15.62:0/2388036234 wait complete. 2024-03-23T11:24:14.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.804+0000 7fad9f200640 1 Processor -- start 2024-03-23T11:24:14.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.804+0000 7fad9f200640 1 -- start start 2024-03-23T11:24:14.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.805+0000 7fad9f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada0070090 0x7fada01f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.805+0000 7fad9f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fada00b2820 0x7fada0142d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.805+0000 7fad9f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 0x7fada00c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.805+0000 7fad9f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada00c5200 con 0x7fada00b2820 2024-03-23T11:24:14.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.805+0000 7fad9f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada00c4350 con 0x7fada0070090 2024-03-23T11:24:14.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.805+0000 7fad9f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada01f2ff0 con 0x7fada0143dd0 2024-03-23T11:24:14.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.805+0000 7fad97e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fada00b2820 0x7fada0142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.805+0000 7fad97e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fada00b2820 0x7fada0142d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59376/0 (socket says 172.21.15.62:59376) 2024-03-23T11:24:14.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.805+0000 7fad97e00640 1 -- 172.21.15.62:0/1585203282 learned_addr learned my addr 172.21.15.62:0/1585203282 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:14.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad9de00640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada0070090 0x7fada01f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad9e800640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 0x7fada00c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad97e00640 1 -- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 msgr2=0x7fada00c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad97e00640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 0x7fada00c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad97e00640 1 -- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada0070090 msgr2=0x7fada01f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad97e00640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada0070090 0x7fada01f2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad97e00640 1 -- 172.21.15.62:0/1585203282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad880094d0 con 0x7fada00b2820 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad9e800640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 0x7fada00c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad9de00640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada0070090 0x7fada01f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad97e00640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fada00b2820 0x7fada0142d50 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fad88000c00 tx=0x7fad88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad96000640 1 -- 172.21.15.62:0/1585203282 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88039c90 con 0x7fada00b2820 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad96000640 1 -- 172.21.15.62:0/1585203282 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad88039df0 con 0x7fada00b2820 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad9f200640 1 -- 172.21.15.62:0/1585203282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada00c45f0 con 0x7fada00b2820 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.806+0000 7fad96000640 1 -- 172.21.15.62:0/1585203282 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88043640 con 0x7fada00b2820 2024-03-23T11:24:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.807+0000 7fad9f200640 1 -- 172.21.15.62:0/1585203282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada00c4bb0 con 0x7fada00b2820 2024-03-23T11:24:14.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.808+0000 7fad9f200640 1 -- 172.21.15.62:0/1585203282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada00c53b0 con 0x7fada00b2820 2024-03-23T11:24:14.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.809+0000 7fad96000640 1 -- 172.21.15.62:0/1585203282 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fad880437a0 con 0x7fada00b2820 2024-03-23T11:24:14.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.809+0000 7fad96000640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad7c06a7a0 0x7fad7c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:14.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.810+0000 7fad96000640 1 -- 172.21.15.62:0/1585203282 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad8803e070 con 0x7fada00b2820 2024-03-23T11:24:14.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.810+0000 7fad9de00640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad7c06a7a0 0x7fad7c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.811+0000 7fad9de00640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad7c06a7a0 0x7fad7c06cc60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fad90009a20 tx=0x7fad90008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:14.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:14.814+0000 7fad96000640 1 -- 172.21.15.62:0/1585203282 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fada00c53b0 con 0x7fada00b2820 2024-03-23T11:24:15.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.015+0000 7fad9f200640 1 -- 172.21.15.62:0/1585203282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fada00cca40 con 0x7fada00b2820 2024-03-23T11:24:15.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.017+0000 7fad96000640 1 -- 172.21.15.62:0/1585203282 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fad88077170 con 0x7fada00b2820 2024-03-23T11:24:15.012 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:15.013 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":74,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:12.048899+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:24:08.688706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-23T11:24:10.072774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:24:15.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.019+0000 7fad9f200640 1 -- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad7c06a7a0 msgr2=0x7fad7c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:15.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.019+0000 7fad9f200640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad7c06a7a0 0x7fad7c06cc60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fad90009a20 tx=0x7fad90008040 comp rx=0 tx=0).stop 2024-03-23T11:24:15.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.019+0000 7fad9f200640 1 -- 172.21.15.62:0/1585203282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fada00b2820 msgr2=0x7fada0142d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:15.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.019+0000 7fad9f200640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fada00b2820 0x7fada0142d50 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fad88000c00 tx=0x7fad88004730 comp rx=0 tx=0).stop 2024-03-23T11:24:15.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.020+0000 7fad9f200640 1 -- 172.21.15.62:0/1585203282 shutdown_connections 2024-03-23T11:24:15.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.020+0000 7fad9f200640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad7c06a7a0 0x7fad7c06cc60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.020+0000 7fad9f200640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fada00b2820 0x7fada0142d50 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.020+0000 7fad9f200640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fada0143dd0 0x7fada00c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.020+0000 7fad9f200640 1 --2- 172.21.15.62:0/1585203282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fada0070090 0x7fada01f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.020+0000 7fad9f200640 1 -- 172.21.15.62:0/1585203282 >> 172.21.15.62:0/1585203282 conn(0x7fada0066050 msgr2=0x7fada00afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:15.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.020+0000 7fad9f200640 1 -- 172.21.15.62:0/1585203282 shutdown_connections 2024-03-23T11:24:15.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.020+0000 7fad9f200640 1 -- 172.21.15.62:0/1585203282 wait complete. 2024-03-23T11:24:15.035 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T11:24:15.035 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:24:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.249+0000 7f65ffe00640 1 Processor -- start 2024-03-23T11:24:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.250+0000 7f65ffe00640 1 -- start start 2024-03-23T11:24:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.251+0000 7f65ffe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660015c830 0x7f66000ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.251+0000 7f65ffe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f66000ce250 0x7f66000ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.251+0000 7f65ffe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66001608d0 con 0x7f6600160310 2024-03-23T11:24:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.251+0000 7f65ffe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66000addb0 con 0x7f66000ce250 2024-03-23T11:24:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.251+0000 7f65ffe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f660015cc10 con 0x7f660015c830 2024-03-23T11:24:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.252+0000 7f65ff400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f66000ce250 0x7f66000ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.252+0000 7f65ff400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f66000ce250 0x7f66000ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59390/0 (socket says 172.21.15.62:59390) 2024-03-23T11:24:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.252+0000 7f65ff400640 1 -- 172.21.15.62:0/101576294 learned_addr learned my addr 172.21.15.62:0/101576294 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.252+0000 7f65fea00640 1 --2- 172.21.15.62:0/101576294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660015c830 0x7f66000ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.252+0000 7f65ff400640 1 -- 172.21.15.62:0/101576294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660015c830 msgr2=0x7f66000ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.252+0000 7f65ff400640 1 --2- 172.21.15.62:0/101576294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660015c830 0x7f66000ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.252+0000 7f65ff400640 1 -- 172.21.15.62:0/101576294 >> v1:172.21.15.120:6789/0 conn(0x7f6600160310 legacy=0x7f66000ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.252+0000 7f65ff400640 1 -- 172.21.15.62:0/101576294 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6600160bb0 con 0x7f66000ce250 2024-03-23T11:24:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.253+0000 7f65ff400640 1 --2- 172.21.15.62:0/101576294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66000ce250 0x7f66000ce6e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f65f8009e80 tx=0x7f65f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c151e194c81d8936 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.254+0000 7f65fe000640 1 -- 172.21.15.62:0/101576294 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f802ca50 con 0x7f66000ce250 2024-03-23T11:24:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.254+0000 7f65fe000640 1 -- 172.21.15.62:0/101576294 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f802cbb0 con 0x7f66000ce250 2024-03-23T11:24:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.254+0000 7f65fe000640 1 -- 172.21.15.62:0/101576294 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f80343f0 con 0x7f66000ce250 2024-03-23T11:24:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.254+0000 7f65ffe00640 1 -- 172.21.15.62:0/101576294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66000ce250 msgr2=0x7f66000ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.254+0000 7f65ffe00640 1 --2- 172.21.15.62:0/101576294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66000ce250 0x7f66000ce6e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f65f8009e80 tx=0x7f65f802beb0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.255+0000 7f65ffe00640 1 -- 172.21.15.62:0/101576294 shutdown_connections 2024-03-23T11:24:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.255+0000 7f65ffe00640 1 --2- 172.21.15.62:0/101576294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66000ce250 0x7f66000ce6e0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.255+0000 7f65ffe00640 1 --2- 172.21.15.62:0/101576294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660015c830 0x7f66000ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.255+0000 7f65ffe00640 1 -- 172.21.15.62:0/101576294 >> 172.21.15.62:0/101576294 conn(0x7f6600066050 msgr2=0x7f6600069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.255+0000 7f65ffe00640 1 -- 172.21.15.62:0/101576294 shutdown_connections 2024-03-23T11:24:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.255+0000 7f65ffe00640 1 -- 172.21.15.62:0/101576294 wait complete. 2024-03-23T11:24:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.257+0000 7f65ffe00640 1 Processor -- start 2024-03-23T11:24:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.257+0000 7f65ffe00640 1 -- start start 2024-03-23T11:24:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.258+0000 7f65ffe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66000ce250 0x7f66001f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.258+0000 7f65ffe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f660015c830 0x7f66000b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.258+0000 7f65ffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6600160310 0x7f66000b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.258+0000 7f65ffe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66001f2f80 con 0x7f6600160310 2024-03-23T11:24:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.258+0000 7f65ffe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660014c690 con 0x7f660015c830 2024-03-23T11:24:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.258+0000 7f65ffe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66001f5700 con 0x7f66000ce250 2024-03-23T11:24:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.258+0000 7f65ff400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6600160310 0x7f66000b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.259+0000 7f65ff400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6600160310 0x7f66000b1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59402/0 (socket says 172.21.15.62:59402) 2024-03-23T11:24:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.259+0000 7f65ff400640 1 -- 172.21.15.62:0/3168422782 learned_addr learned my addr 172.21.15.62:0/3168422782 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.259+0000 7f65f7e00640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f660015c830 0x7f66000b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.259+0000 7f65fea00640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66000ce250 0x7f66001f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.259+0000 7f65ff400640 1 -- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66000ce250 msgr2=0x7f66001f1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.259+0000 7f65ff400640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66000ce250 0x7f66001f1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.259+0000 7f65ff400640 1 -- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f660015c830 msgr2=0x7f66000b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.259+0000 7f65ff400640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f660015c830 0x7f66000b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.259+0000 7f65ff400640 1 -- 172.21.15.62:0/3168422782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f8009b30 con 0x7f6600160310 2024-03-23T11:24:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.259+0000 7f65ff400640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6600160310 0x7f66000b1db0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f65f8036040 tx=0x7f65f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.260+0000 7f65f6a00640 1 -- 172.21.15.62:0/3168422782 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f8009510 con 0x7f6600160310 2024-03-23T11:24:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.260+0000 7f65f6a00640 1 -- 172.21.15.62:0/3168422782 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f803e450 con 0x7f6600160310 2024-03-23T11:24:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.260+0000 7f65ffe00640 1 -- 172.21.15.62:0/3168422782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66001f59e0 con 0x7f6600160310 2024-03-23T11:24:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.260+0000 7f65f6a00640 1 -- 172.21.15.62:0/3168422782 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f803e800 con 0x7f6600160310 2024-03-23T11:24:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.260+0000 7f65ffe00640 1 -- 172.21.15.62:0/3168422782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66001f5fa0 con 0x7f6600160310 2024-03-23T11:24:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.261+0000 7f65ffe00640 1 -- 172.21.15.62:0/3168422782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6600066bb0 con 0x7f6600160310 2024-03-23T11:24:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.262+0000 7f65f6a00640 1 -- 172.21.15.62:0/3168422782 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f65f803e960 con 0x7f6600160310 2024-03-23T11:24:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.263+0000 7f65f6a00640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f65dc06a7a0 0x7f65dc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.263+0000 7f65f6a00640 1 -- 172.21.15.62:0/3168422782 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65f80aba50 con 0x7f6600160310 2024-03-23T11:24:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.264+0000 7f65fea00640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f65dc06a7a0 0x7f65dc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.265+0000 7f65fea00640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f65dc06a7a0 0x7f65dc06cc60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f65f0006fd0 tx=0x7f65f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:15.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.267+0000 7f65f6a00640 1 -- 172.21.15.62:0/3168422782 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65f8076f00 con 0x7f6600160310 2024-03-23T11:24:15.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.458+0000 7f65ffe00640 1 -- 172.21.15.62:0/3168422782 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f66000cec20 con 0x7f65dc06a7a0 2024-03-23T11:24:15.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.484+0000 7f65f6a00640 1 -- 172.21.15.62:0/3168422782 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286867 (secure 0 0 0) 0x7f65f000f830 con 0x7f65dc06a7a0 2024-03-23T11:24:15.483 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:15.485 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:24:15.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.490+0000 7f65ffe00640 1 -- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f65dc06a7a0 msgr2=0x7f65dc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:15.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.490+0000 7f65ffe00640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f65dc06a7a0 0x7f65dc06cc60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f65f0006fd0 tx=0x7f65f0008040 comp rx=0 tx=0).stop 2024-03-23T11:24:15.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.490+0000 7f65ffe00640 1 -- 172.21.15.62:0/3168422782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6600160310 msgr2=0x7f66000b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:15.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.490+0000 7f65ffe00640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6600160310 0x7f66000b1db0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f65f8036040 tx=0x7f65f8004a40 comp rx=0 tx=0).stop 2024-03-23T11:24:15.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.490+0000 7f65ffe00640 1 -- 172.21.15.62:0/3168422782 shutdown_connections 2024-03-23T11:24:15.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.490+0000 7f65ffe00640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f65dc06a7a0 0x7f65dc06cc60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.490+0000 7f65ffe00640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6600160310 0x7f66000b1db0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.490+0000 7f65ffe00640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f660015c830 0x7f66000b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.491+0000 7f65ffe00640 1 --2- 172.21.15.62:0/3168422782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66000ce250 0x7f66001f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.491+0000 7f65ffe00640 1 -- 172.21.15.62:0/3168422782 >> 172.21.15.62:0/3168422782 conn(0x7f6600066050 msgr2=0x7f660015c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:15.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.491+0000 7f65ffe00640 1 -- 172.21.15.62:0/3168422782 shutdown_connections 2024-03-23T11:24:15.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.491+0000 7f65ffe00640 1 -- 172.21.15.62:0/3168422782 wait complete. 2024-03-23T11:24:15.508 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-03-23T11:24:15.346067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240756,"kb_used_data":8172,"kb_used_omap":13,"kb_used_meta":232434,"kb_avail":754733964,"statfs":{"total":773094113280,"available":772847579136,"internally_reserved":0,"allocated":8368128,"data_stored":4177917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":238012504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.804304"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070614+0000","last_change":"2024-03-23T11:24:11.078223+0000","last_active":"2024-03-23T11:24:12.070614+0000","last_peered":"2024-03-23T11:24:12.070614+0000","last_clean":"2024-03-23T11:24:12.070614+0000","last_became_active":"2024-03-23T11:24:11.077962+0000","last_became_peered":"2024-03-23T11:24:11.077962+0000","last_unstale":"2024-03-23T11:24:12.070614+0000","last_undegraded":"2024-03-23T11:24:12.070614+0000","last_fullsized":"2024-03-23T11:24:12.070614+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:35:42.435823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071168+0000","last_change":"2024-03-23T11:24:11.087210+0000","last_active":"2024-03-23T11:24:12.071168+0000","last_peered":"2024-03-23T11:24:12.071168+0000","last_clean":"2024-03-23T11:24:12.071168+0000","last_became_active":"2024-03-23T11:24:11.086983+0000","last_became_peered":"2024-03-23T11:24:11.086983+0000","last_unstale":"2024-03-23T11:24:12.071168+0000","last_undegraded":"2024-03-23T11:24:12.071168+0000","last_fullsized":"2024-03-23T11:24:12.071168+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:20:59.165481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262303+0000","last_change":"2024-03-23T11:24:11.079994+0000","last_active":"2024-03-23T11:24:13.262303+0000","last_peered":"2024-03-23T11:24:13.262303+0000","last_clean":"2024-03-23T11:24:13.262303+0000","last_became_active":"2024-03-23T11:24:11.079643+0000","last_became_peered":"2024-03-23T11:24:11.079643+0000","last_unstale":"2024-03-23T11:24:13.262303+0000","last_undegraded":"2024-03-23T11:24:13.262303+0000","last_fullsized":"2024-03-23T11:24:13.262303+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:01:22.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066724+0000","last_change":"2024-03-23T11:24:11.086417+0000","last_active":"2024-03-23T11:24:12.066724+0000","last_peered":"2024-03-23T11:24:12.066724+0000","last_clean":"2024-03-23T11:24:12.066724+0000","last_became_active":"2024-03-23T11:24:11.086149+0000","last_became_peered":"2024-03-23T11:24:11.086149+0000","last_unstale":"2024-03-23T11:24:12.066724+0000","last_undegraded":"2024-03-23T11:24:12.066724+0000","last_fullsized":"2024-03-23T11:24:12.066724+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:42:46.273199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.083970+0000","last_change":"2024-03-23T11:24:11.075931+0000","last_active":"2024-03-23T11:24:11.083970+0000","last_peered":"2024-03-23T11:24:11.083970+0000","last_clean":"2024-03-23T11:24:11.083970+0000","last_became_active":"2024-03-23T11:24:11.075678+0000","last_became_peered":"2024-03-23T11:24:11.075678+0000","last_unstale":"2024-03-23T11:24:11.083970+0000","last_undegraded":"2024-03-23T11:24:11.083970+0000","last_fullsized":"2024-03-23T11:24:11.083970+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:35.558343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069929+0000","last_change":"2024-03-23T11:24:11.087973+0000","last_active":"2024-03-23T11:24:12.069929+0000","last_peered":"2024-03-23T11:24:12.069929+0000","last_clean":"2024-03-23T11:24:12.069929+0000","last_became_active":"2024-03-23T11:24:11.087702+0000","last_became_peered":"2024-03-23T11:24:11.087702+0000","last_unstale":"2024-03-23T11:24:12.069929+0000","last_undegraded":"2024-03-23T11:24:12.069929+0000","last_fullsized":"2024-03-23T11:24:12.069929+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:55:45.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194032+0000","last_change":"2024-03-23T11:24:11.072450+0000","last_active":"2024-03-23T11:24:13.194032+0000","last_peered":"2024-03-23T11:24:13.194032+0000","last_clean":"2024-03-23T11:24:13.194032+0000","last_became_active":"2024-03-23T11:24:11.071363+0000","last_became_peered":"2024-03-23T11:24:11.071363+0000","last_unstale":"2024-03-23T11:24:13.194032+0000","last_undegraded":"2024-03-23T11:24:13.194032+0000","last_fullsized":"2024-03-23T11:24:13.194032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:35:26.679939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070168+0000","last_change":"2024-03-23T11:24:11.080137+0000","last_active":"2024-03-23T11:24:12.070168+0000","last_peered":"2024-03-23T11:24:12.070168+0000","last_clean":"2024-03-23T11:24:12.070168+0000","last_became_active":"2024-03-23T11:24:11.078526+0000","last_became_peered":"2024-03-23T11:24:11.078526+0000","last_unstale":"2024-03-23T11:24:12.070168+0000","last_undegraded":"2024-03-23T11:24:12.070168+0000","last_fullsized":"2024-03-23T11:24:12.070168+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:11:59.579321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.082588+0000","last_change":"2024-03-23T11:24:11.075934+0000","last_active":"2024-03-23T11:24:11.082588+0000","last_peered":"2024-03-23T11:24:11.082588+0000","last_clean":"2024-03-23T11:24:11.082588+0000","last_became_active":"2024-03-23T11:24:11.075617+0000","last_became_peered":"2024-03-23T11:24:11.075617+0000","last_unstale":"2024-03-23T11:24:11.082588+0000","last_undegraded":"2024-03-23T11:24:11.082588+0000","last_fullsized":"2024-03-23T11:24:11.082588+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:42:31.187510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072004+0000","last_change":"2024-03-23T11:24:11.084006+0000","last_active":"2024-03-23T11:24:13.072004+0000","last_peered":"2024-03-23T11:24:13.072004+0000","last_clean":"2024-03-23T11:24:13.072004+0000","last_became_active":"2024-03-23T11:24:11.082341+0000","last_became_peered":"2024-03-23T11:24:11.082341+0000","last_unstale":"2024-03-23T11:24:13.072004+0000","last_undegraded":"2024-03-23T11:24:13.072004+0000","last_fullsized":"2024-03-23T11:24:13.072004+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:45.962293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069194+0000","last_change":"2024-03-23T11:24:11.077569+0000","last_active":"2024-03-23T11:24:12.069194+0000","last_peered":"2024-03-23T11:24:12.069194+0000","last_clean":"2024-03-23T11:24:12.069194+0000","last_became_active":"2024-03-23T11:24:11.077238+0000","last_became_peered":"2024-03-23T11:24:11.077238+0000","last_unstale":"2024-03-23T11:24:12.069194+0000","last_undegraded":"2024-03-23T11:24:12.069194+0000","last_fullsized":"2024-03-23T11:24:12.069194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:39:26.172241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066855+0000","last_change":"2024-03-23T11:24:11.077251+0000","last_active":"2024-03-23T11:24:12.066855+0000","last_peered":"2024-03-23T11:24:12.066855+0000","last_clean":"2024-03-23T11:24:12.066855+0000","last_became_active":"2024-03-23T11:24:11.076703+0000","last_became_peered":"2024-03-23T11:24:11.076703+0000","last_unstale":"2024-03-23T11:24:12.066855+0000","last_undegraded":"2024-03-23T11:24:12.066855+0000","last_fullsized":"2024-03-23T11:24:12.066855+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:02:15.494810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.091483+0000","last_change":"2024-03-23T11:24:11.085605+0000","last_active":"2024-03-23T11:24:11.091483+0000","last_peered":"2024-03-23T11:24:11.091483+0000","last_clean":"2024-03-23T11:24:11.091483+0000","last_became_active":"2024-03-23T11:24:11.085317+0000","last_became_peered":"2024-03-23T11:24:11.085317+0000","last_unstale":"2024-03-23T11:24:11.091483+0000","last_undegraded":"2024-03-23T11:24:11.091483+0000","last_fullsized":"2024-03-23T11:24:11.091483+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:01:21.503319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.073558+0000","last_change":"2024-03-23T11:24:11.069770+0000","last_active":"2024-03-23T11:24:11.073558+0000","last_peered":"2024-03-23T11:24:11.073558+0000","last_clean":"2024-03-23T11:24:11.073558+0000","last_became_active":"2024-03-23T11:24:11.069414+0000","last_became_peered":"2024-03-23T11:24:11.069414+0000","last_unstale":"2024-03-23T11:24:11.073558+0000","last_undegraded":"2024-03-23T11:24:11.073558+0000","last_fullsized":"2024-03-23T11:24:11.073558+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:17.196321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.083600+0000","last_change":"2024-03-23T11:24:11.078141+0000","last_active":"2024-03-23T11:24:11.083600+0000","last_peered":"2024-03-23T11:24:11.083600+0000","last_clean":"2024-03-23T11:24:11.083600+0000","last_became_active":"2024-03-23T11:24:11.077876+0000","last_became_peered":"2024-03-23T11:24:11.077876+0000","last_unstale":"2024-03-23T11:24:11.083600+0000","last_undegraded":"2024-03-23T11:24:11.083600+0000","last_fullsized":"2024-03-23T11:24:11.083600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:46:58.997122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.071506+0000","last_change":"2024-03-23T11:24:11.087752+0000","last_active":"2024-03-23T11:24:13.071506+0000","last_peered":"2024-03-23T11:24:13.071506+0000","last_clean":"2024-03-23T11:24:13.071506+0000","last_became_active":"2024-03-23T11:24:11.087451+0000","last_became_peered":"2024-03-23T11:24:11.087451+0000","last_unstale":"2024-03-23T11:24:13.071506+0000","last_undegraded":"2024-03-23T11:24:13.071506+0000","last_fullsized":"2024-03-23T11:24:13.071506+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:09:42.953821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.074145+0000","last_change":"2024-03-23T11:24:11.070297+0000","last_active":"2024-03-23T11:24:11.074145+0000","last_peered":"2024-03-23T11:24:11.074145+0000","last_clean":"2024-03-23T11:24:11.074145+0000","last_became_active":"2024-03-23T11:24:11.070047+0000","last_became_peered":"2024-03-23T11:24:11.070047+0000","last_unstale":"2024-03-23T11:24:11.074145+0000","last_undegraded":"2024-03-23T11:24:11.074145+0000","last_fullsized":"2024-03-23T11:24:11.074145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:58:45.931096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066380+0000","last_change":"2024-03-23T11:24:11.088685+0000","last_active":"2024-03-23T11:24:12.066380+0000","last_peered":"2024-03-23T11:24:12.066380+0000","last_clean":"2024-03-23T11:24:12.066380+0000","last_became_active":"2024-03-23T11:24:11.088406+0000","last_became_peered":"2024-03-23T11:24:11.088406+0000","last_unstale":"2024-03-23T11:24:12.066380+0000","last_undegraded":"2024-03-23T11:24:12.066380+0000","last_fullsized":"2024-03-23T11:24:12.066380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:37.925686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070484+0000","last_change":"2024-03-23T11:24:11.084614+0000","last_active":"2024-03-23T11:24:12.070484+0000","last_peered":"2024-03-23T11:24:12.070484+0000","last_clean":"2024-03-23T11:24:12.070484+0000","last_became_active":"2024-03-23T11:24:11.084336+0000","last_became_peered":"2024-03-23T11:24:11.084336+0000","last_unstale":"2024-03-23T11:24:12.070484+0000","last_undegraded":"2024-03-23T11:24:12.070484+0000","last_fullsized":"2024-03-23T11:24:12.070484+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:46:58.997122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.083225+0000","last_change":"2024-03-23T11:24:11.075275+0000","last_active":"2024-03-23T11:24:11.083225+0000","last_peered":"2024-03-23T11:24:11.083225+0000","last_clean":"2024-03-23T11:24:11.083225+0000","last_became_active":"2024-03-23T11:24:11.075032+0000","last_became_peered":"2024-03-23T11:24:11.075032+0000","last_unstale":"2024-03-23T11:24:11.083225+0000","last_undegraded":"2024-03-23T11:24:11.083225+0000","last_fullsized":"2024-03-23T11:24:11.083225+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:42.150300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072013+0000","last_change":"2024-03-23T11:24:11.084697+0000","last_active":"2024-03-23T11:24:13.072013+0000","last_peered":"2024-03-23T11:24:13.072013+0000","last_clean":"2024-03-23T11:24:13.072013+0000","last_became_active":"2024-03-23T11:24:11.083709+0000","last_became_peered":"2024-03-23T11:24:11.083709+0000","last_unstale":"2024-03-23T11:24:13.072013+0000","last_undegraded":"2024-03-23T11:24:13.072013+0000","last_fullsized":"2024-03-23T11:24:13.072013+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:25:35.284033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.071657+0000","last_change":"2024-03-23T11:24:11.071917+0000","last_active":"2024-03-23T11:24:13.071657+0000","last_peered":"2024-03-23T11:24:13.071657+0000","last_clean":"2024-03-23T11:24:13.071657+0000","last_became_active":"2024-03-23T11:24:11.068485+0000","last_became_peered":"2024-03-23T11:24:11.068485+0000","last_unstale":"2024-03-23T11:24:13.071657+0000","last_undegraded":"2024-03-23T11:24:13.071657+0000","last_fullsized":"2024-03-23T11:24:13.071657+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:51:23.153567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.068567+0000","last_change":"2024-03-23T11:24:11.070113+0000","last_active":"2024-03-23T11:24:12.068567+0000","last_peered":"2024-03-23T11:24:12.068567+0000","last_clean":"2024-03-23T11:24:12.068567+0000","last_became_active":"2024-03-23T11:24:11.069833+0000","last_became_peered":"2024-03-23T11:24:11.069833+0000","last_unstale":"2024-03-23T11:24:12.068567+0000","last_undegraded":"2024-03-23T11:24:12.068567+0000","last_fullsized":"2024-03-23T11:24:12.068567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:37:01.697458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.068080+0000","last_change":"2024-03-23T11:24:11.087980+0000","last_active":"2024-03-23T11:24:12.068080+0000","last_peered":"2024-03-23T11:24:12.068080+0000","last_clean":"2024-03-23T11:24:12.068080+0000","last_became_active":"2024-03-23T11:24:11.087729+0000","last_became_peered":"2024-03-23T11:24:11.087729+0000","last_unstale":"2024-03-23T11:24:12.068080+0000","last_undegraded":"2024-03-23T11:24:12.068080+0000","last_fullsized":"2024-03-23T11:24:12.068080+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:14:34.648258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072248+0000","last_change":"2024-03-23T11:24:11.077763+0000","last_active":"2024-03-23T11:24:13.072248+0000","last_peered":"2024-03-23T11:24:13.072248+0000","last_clean":"2024-03-23T11:24:13.072248+0000","last_became_active":"2024-03-23T11:24:11.076383+0000","last_became_peered":"2024-03-23T11:24:11.076383+0000","last_unstale":"2024-03-23T11:24:13.072248+0000","last_undegraded":"2024-03-23T11:24:13.072248+0000","last_fullsized":"2024-03-23T11:24:13.072248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:11:59.579321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194661+0000","last_change":"2024-03-23T11:24:11.090977+0000","last_active":"2024-03-23T11:24:13.194661+0000","last_peered":"2024-03-23T11:24:13.194661+0000","last_clean":"2024-03-23T11:24:13.194661+0000","last_became_active":"2024-03-23T11:24:11.090679+0000","last_became_peered":"2024-03-23T11:24:11.090679+0000","last_unstale":"2024-03-23T11:24:13.194661+0000","last_undegraded":"2024-03-23T11:24:13.194661+0000","last_fullsized":"2024-03-23T11:24:13.194661+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:18:04.159007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262156+0000","last_change":"2024-03-23T11:24:11.075234+0000","last_active":"2024-03-23T11:24:13.262156+0000","last_peered":"2024-03-23T11:24:13.262156+0000","last_clean":"2024-03-23T11:24:13.262156+0000","last_became_active":"2024-03-23T11:24:11.074971+0000","last_became_peered":"2024-03-23T11:24:11.074971+0000","last_unstale":"2024-03-23T11:24:13.262156+0000","last_undegraded":"2024-03-23T11:24:13.262156+0000","last_fullsized":"2024-03-23T11:24:13.262156+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:59:57.629892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072255+0000","last_change":"2024-03-23T11:24:11.077975+0000","last_active":"2024-03-23T11:24:13.072255+0000","last_peered":"2024-03-23T11:24:13.072255+0000","last_clean":"2024-03-23T11:24:13.072255+0000","last_became_active":"2024-03-23T11:24:11.077700+0000","last_became_peered":"2024-03-23T11:24:11.077700+0000","last_unstale":"2024-03-23T11:24:13.072255+0000","last_undegraded":"2024-03-23T11:24:13.072255+0000","last_fullsized":"2024-03-23T11:24:13.072255+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:59:01.118041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.261835+0000","last_change":"2024-03-23T11:24:11.069184+0000","last_active":"2024-03-23T11:24:13.261835+0000","last_peered":"2024-03-23T11:24:13.261835+0000","last_clean":"2024-03-23T11:24:13.261835+0000","last_became_active":"2024-03-23T11:24:11.068839+0000","last_became_peered":"2024-03-23T11:24:11.068839+0000","last_unstale":"2024-03-23T11:24:13.261835+0000","last_undegraded":"2024-03-23T11:24:13.261835+0000","last_fullsized":"2024-03-23T11:24:13.261835+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:55:00.318209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194044+0000","last_change":"2024-03-23T11:24:11.088591+0000","last_active":"2024-03-23T11:24:13.194044+0000","last_peered":"2024-03-23T11:24:13.194044+0000","last_clean":"2024-03-23T11:24:13.194044+0000","last_became_active":"2024-03-23T11:24:11.086063+0000","last_became_peered":"2024-03-23T11:24:11.086063+0000","last_unstale":"2024-03-23T11:24:13.194044+0000","last_undegraded":"2024-03-23T11:24:13.194044+0000","last_fullsized":"2024-03-23T11:24:13.194044+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:55:54.448139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194496+0000","last_change":"2024-03-23T11:24:11.071424+0000","last_active":"2024-03-23T11:24:13.194496+0000","last_peered":"2024-03-23T11:24:13.194496+0000","last_clean":"2024-03-23T11:24:13.194496+0000","last_became_active":"2024-03-23T11:24:11.071105+0000","last_became_peered":"2024-03-23T11:24:11.071105+0000","last_unstale":"2024-03-23T11:24:13.194496+0000","last_undegraded":"2024-03-23T11:24:13.194496+0000","last_fullsized":"2024-03-23T11:24:13.194496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:29:46.001149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070466+0000","last_change":"2024-03-23T11:24:11.084226+0000","last_active":"2024-03-23T11:24:12.070466+0000","last_peered":"2024-03-23T11:24:12.070466+0000","last_clean":"2024-03-23T11:24:12.070466+0000","last_became_active":"2024-03-23T11:24:11.082119+0000","last_became_peered":"2024-03-23T11:24:11.082119+0000","last_unstale":"2024-03-23T11:24:12.070466+0000","last_undegraded":"2024-03-23T11:24:12.070466+0000","last_fullsized":"2024-03-23T11:24:12.070466+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:18:53.314479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069301+0000","last_change":"2024-03-23T11:24:11.092837+0000","last_active":"2024-03-23T11:24:12.069301+0000","last_peered":"2024-03-23T11:24:12.069301+0000","last_clean":"2024-03-23T11:24:12.069301+0000","last_became_active":"2024-03-23T11:24:11.092582+0000","last_became_peered":"2024-03-23T11:24:11.092582+0000","last_unstale":"2024-03-23T11:24:12.069301+0000","last_undegraded":"2024-03-23T11:24:12.069301+0000","last_fullsized":"2024-03-23T11:24:12.069301+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:49:21.313471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"73'4","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070815+0000","last_change":"2024-03-23T11:24:10.588379+0000","last_active":"2024-03-23T11:24:12.070815+0000","last_peered":"2024-03-23T11:24:12.070815+0000","last_clean":"2024-03-23T11:24:12.070815+0000","last_became_active":"2024-03-23T11:24:10.588040+0000","last_became_peered":"2024-03-23T11:24:10.588040+0000","last_unstale":"2024-03-23T11:24:12.070815+0000","last_undegraded":"2024-03-23T11:24:12.070815+0000","last_fullsized":"2024-03-23T11:24:12.070815+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:02:50.767178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.078368+0000","last_change":"2024-03-23T11:24:11.074259+0000","last_active":"2024-03-23T11:24:11.078368+0000","last_peered":"2024-03-23T11:24:11.078368+0000","last_clean":"2024-03-23T11:24:11.078368+0000","last_became_active":"2024-03-23T11:24:11.073957+0000","last_became_peered":"2024-03-23T11:24:11.073957+0000","last_unstale":"2024-03-23T11:24:11.078368+0000","last_undegraded":"2024-03-23T11:24:11.078368+0000","last_fullsized":"2024-03-23T11:24:11.078368+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:54:34.282504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"73'1","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.090876+0000","last_change":"2024-03-23T11:24:10.581333+0000","last_active":"2024-03-23T11:24:11.090876+0000","last_peered":"2024-03-23T11:24:11.090876+0000","last_clean":"2024-03-23T11:24:11.090876+0000","last_became_active":"2024-03-23T11:24:10.581019+0000","last_became_peered":"2024-03-23T11:24:10.581019+0000","last_unstale":"2024-03-23T11:24:11.090876+0000","last_undegraded":"2024-03-23T11:24:11.090876+0000","last_fullsized":"2024-03-23T11:24:11.090876+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:59:00.105792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071090+0000","last_change":"2024-03-23T11:24:11.085453+0000","last_active":"2024-03-23T11:24:12.071090+0000","last_peered":"2024-03-23T11:24:12.071090+0000","last_clean":"2024-03-23T11:24:12.071090+0000","last_became_active":"2024-03-23T11:24:11.083291+0000","last_became_peered":"2024-03-23T11:24:11.083291+0000","last_unstale":"2024-03-23T11:24:12.071090+0000","last_undegraded":"2024-03-23T11:24:12.071090+0000","last_fullsized":"2024-03-23T11:24:12.071090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:01:21.503319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072208+0000","last_change":"2024-03-23T11:24:10.582900+0000","last_active":"2024-03-23T11:24:13.072208+0000","last_peered":"2024-03-23T11:24:13.072208+0000","last_clean":"2024-03-23T11:24:13.072208+0000","last_became_active":"2024-03-23T11:24:10.582186+0000","last_became_peered":"2024-03-23T11:24:10.582186+0000","last_unstale":"2024-03-23T11:24:13.072208+0000","last_undegraded":"2024-03-23T11:24:13.072208+0000","last_fullsized":"2024-03-23T11:24:13.072208+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:46:08.860795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069481+0000","last_change":"2024-03-23T11:24:11.079508+0000","last_active":"2024-03-23T11:24:12.069481+0000","last_peered":"2024-03-23T11:24:12.069481+0000","last_clean":"2024-03-23T11:24:12.069481+0000","last_became_active":"2024-03-23T11:24:11.077841+0000","last_became_peered":"2024-03-23T11:24:11.077841+0000","last_unstale":"2024-03-23T11:24:12.069481+0000","last_undegraded":"2024-03-23T11:24:12.069481+0000","last_fullsized":"2024-03-23T11:24:12.069481+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:33:10.380948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071513+0000","last_change":"2024-03-23T11:24:10.583681+0000","last_active":"2024-03-23T11:24:12.071513+0000","last_peered":"2024-03-23T11:24:12.071513+0000","last_clean":"2024-03-23T11:24:12.071513+0000","last_became_active":"2024-03-23T11:24:10.582035+0000","last_became_peered":"2024-03-23T11:24:10.582035+0000","last_unstale":"2024-03-23T11:24:12.071513+0000","last_undegraded":"2024-03-23T11:24:12.071513+0000","last_fullsized":"2024-03-23T11:24:12.071513+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:42:30.175261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194203+0000","last_change":"2024-03-23T11:24:11.080481+0000","last_active":"2024-03-23T11:24:13.194203+0000","last_peered":"2024-03-23T11:24:13.194203+0000","last_clean":"2024-03-23T11:24:13.194203+0000","last_became_active":"2024-03-23T11:24:11.080209+0000","last_became_peered":"2024-03-23T11:24:11.080209+0000","last_unstale":"2024-03-23T11:24:13.194203+0000","last_undegraded":"2024-03-23T11:24:13.194203+0000","last_fullsized":"2024-03-23T11:24:13.194203+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:44:00.853141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"73'1","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262219+0000","last_change":"2024-03-23T11:24:10.586960+0000","last_active":"2024-03-23T11:24:13.262219+0000","last_peered":"2024-03-23T11:24:13.262219+0000","last_clean":"2024-03-23T11:24:13.262219+0000","last_became_active":"2024-03-23T11:24:10.585902+0000","last_became_peered":"2024-03-23T11:24:10.585902+0000","last_unstale":"2024-03-23T11:24:13.262219+0000","last_undegraded":"2024-03-23T11:24:13.262219+0000","last_fullsized":"2024-03-23T11:24:13.262219+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:07.177476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072002+0000","last_change":"2024-03-23T11:24:11.075018+0000","last_active":"2024-03-23T11:24:13.072002+0000","last_peered":"2024-03-23T11:24:13.072002+0000","last_clean":"2024-03-23T11:24:13.072002+0000","last_became_active":"2024-03-23T11:24:11.073200+0000","last_became_peered":"2024-03-23T11:24:11.073200+0000","last_unstale":"2024-03-23T11:24:13.072002+0000","last_undegraded":"2024-03-23T11:24:13.072002+0000","last_fullsized":"2024-03-23T11:24:13.072002+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:45:36.911552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"73'1","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070101+0000","last_change":"2024-03-23T11:24:10.576442+0000","last_active":"2024-03-23T11:24:12.070101+0000","last_peered":"2024-03-23T11:24:12.070101+0000","last_clean":"2024-03-23T11:24:12.070101+0000","last_became_active":"2024-03-23T11:24:10.576087+0000","last_became_peered":"2024-03-23T11:24:10.576087+0000","last_unstale":"2024-03-23T11:24:12.070101+0000","last_undegraded":"2024-03-23T11:24:12.070101+0000","last_fullsized":"2024-03-23T11:24:12.070101+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:42:01.153953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.091675+0000","last_change":"2024-03-23T11:24:11.087533+0000","last_active":"2024-03-23T11:24:11.091675+0000","last_peered":"2024-03-23T11:24:11.091675+0000","last_clean":"2024-03-23T11:24:11.091675+0000","last_became_active":"2024-03-23T11:24:11.087292+0000","last_became_peered":"2024-03-23T11:24:11.087292+0000","last_unstale":"2024-03-23T11:24:11.091675+0000","last_undegraded":"2024-03-23T11:24:11.091675+0000","last_fullsized":"2024-03-23T11:24:11.091675+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:06:45.108816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066299+0000","last_change":"2024-03-23T11:24:10.573973+0000","last_active":"2024-03-23T11:24:12.066299+0000","last_peered":"2024-03-23T11:24:12.066299+0000","last_clean":"2024-03-23T11:24:12.066299+0000","last_became_active":"2024-03-23T11:24:10.573666+0000","last_became_peered":"2024-03-23T11:24:10.573666+0000","last_unstale":"2024-03-23T11:24:12.066299+0000","last_undegraded":"2024-03-23T11:24:12.066299+0000","last_fullsized":"2024-03-23T11:24:12.066299+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:46:08.498563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.082621+0000","last_change":"2024-03-23T11:24:11.074392+0000","last_active":"2024-03-23T11:24:11.082621+0000","last_peered":"2024-03-23T11:24:11.082621+0000","last_clean":"2024-03-23T11:24:11.082621+0000","last_became_active":"2024-03-23T11:24:11.074120+0000","last_became_peered":"2024-03-23T11:24:11.074120+0000","last_unstale":"2024-03-23T11:24:11.082621+0000","last_undegraded":"2024-03-23T11:24:11.082621+0000","last_fullsized":"2024-03-23T11:24:11.082621+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:05.652914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.065553+0000","last_change":"2024-03-23T11:24:10.567379+0000","last_active":"2024-03-23T11:24:11.065553+0000","last_peered":"2024-03-23T11:24:11.065553+0000","last_clean":"2024-03-23T11:24:11.065553+0000","last_became_active":"2024-03-23T11:24:10.567204+0000","last_became_peered":"2024-03-23T11:24:10.567204+0000","last_unstale":"2024-03-23T11:24:11.065553+0000","last_undegraded":"2024-03-23T11:24:11.065553+0000","last_fullsized":"2024-03-23T11:24:11.065553+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:52.099376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194659+0000","last_change":"2024-03-23T11:24:11.073905+0000","last_active":"2024-03-23T11:24:13.194659+0000","last_peered":"2024-03-23T11:24:13.194659+0000","last_clean":"2024-03-23T11:24:13.194659+0000","last_became_active":"2024-03-23T11:24:11.073603+0000","last_became_peered":"2024-03-23T11:24:11.073603+0000","last_unstale":"2024-03-23T11:24:13.194659+0000","last_undegraded":"2024-03-23T11:24:13.194659+0000","last_fullsized":"2024-03-23T11:24:13.194659+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:41:41.482855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072298+0000","last_change":"2024-03-23T11:24:10.578499+0000","last_active":"2024-03-23T11:24:13.072298+0000","last_peered":"2024-03-23T11:24:13.072298+0000","last_clean":"2024-03-23T11:24:13.072298+0000","last_became_active":"2024-03-23T11:24:10.578229+0000","last_became_peered":"2024-03-23T11:24:10.578229+0000","last_unstale":"2024-03-23T11:24:13.072298+0000","last_undegraded":"2024-03-23T11:24:13.072298+0000","last_fullsized":"2024-03-23T11:24:13.072298+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:13:49.012961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070843+0000","last_change":"2024-03-23T11:24:11.087301+0000","last_active":"2024-03-23T11:24:12.070843+0000","last_peered":"2024-03-23T11:24:12.070843+0000","last_clean":"2024-03-23T11:24:12.070843+0000","last_became_active":"2024-03-23T11:24:11.087031+0000","last_became_peered":"2024-03-23T11:24:11.087031+0000","last_unstale":"2024-03-23T11:24:12.070843+0000","last_undegraded":"2024-03-23T11:24:12.070843+0000","last_fullsized":"2024-03-23T11:24:12.070843+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:26:52.158972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071415+0000","last_change":"2024-03-23T11:24:10.583641+0000","last_active":"2024-03-23T11:24:12.071415+0000","last_peered":"2024-03-23T11:24:12.071415+0000","last_clean":"2024-03-23T11:24:12.071415+0000","last_became_active":"2024-03-23T11:24:10.583296+0000","last_became_peered":"2024-03-23T11:24:10.583296+0000","last_unstale":"2024-03-23T11:24:12.071415+0000","last_undegraded":"2024-03-23T11:24:12.071415+0000","last_fullsized":"2024-03-23T11:24:12.071415+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:44:37.296461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071326+0000","last_change":"2024-03-23T11:24:11.086027+0000","last_active":"2024-03-23T11:24:12.071326+0000","last_peered":"2024-03-23T11:24:12.071326+0000","last_clean":"2024-03-23T11:24:12.071326+0000","last_became_active":"2024-03-23T11:24:11.085718+0000","last_became_peered":"2024-03-23T11:24:11.085718+0000","last_unstale":"2024-03-23T11:24:12.071326+0000","last_undegraded":"2024-03-23T11:24:12.071326+0000","last_fullsized":"2024-03-23T11:24:12.071326+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:21:46.895562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"73'3","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262286+0000","last_change":"2024-03-23T11:24:10.588288+0000","last_active":"2024-03-23T11:24:13.262286+0000","last_peered":"2024-03-23T11:24:13.262286+0000","last_clean":"2024-03-23T11:24:13.262286+0000","last_became_active":"2024-03-23T11:24:10.587901+0000","last_became_peered":"2024-03-23T11:24:10.587901+0000","last_unstale":"2024-03-23T11:24:13.262286+0000","last_undegraded":"2024-03-23T11:24:13.262286+0000","last_fullsized":"2024-03-23T11:24:13.262286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:41:40.470606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.090467+0000","last_change":"2024-03-23T11:24:11.080963+0000","last_active":"2024-03-23T11:24:11.090467+0000","last_peered":"2024-03-23T11:24:11.090467+0000","last_clean":"2024-03-23T11:24:11.090467+0000","last_became_active":"2024-03-23T11:24:11.079943+0000","last_became_peered":"2024-03-23T11:24:11.079943+0000","last_unstale":"2024-03-23T11:24:11.090467+0000","last_undegraded":"2024-03-23T11:24:11.090467+0000","last_fullsized":"2024-03-23T11:24:11.090467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:37:10.652496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"73'6","reported_seq":17,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.067100+0000","last_change":"2024-03-23T11:24:10.575152+0000","last_active":"2024-03-23T11:24:12.067100+0000","last_peered":"2024-03-23T11:24:12.067100+0000","last_clean":"2024-03-23T11:24:12.067100+0000","last_became_active":"2024-03-23T11:24:10.573893+0000","last_became_peered":"2024-03-23T11:24:10.573893+0000","last_unstale":"2024-03-23T11:24:12.067100+0000","last_undegraded":"2024-03-23T11:24:12.067100+0000","last_fullsized":"2024-03-23T11:24:12.067100+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:56:49.479990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.068467+0000","last_change":"2024-03-23T11:24:11.079877+0000","last_active":"2024-03-23T11:24:12.068467+0000","last_peered":"2024-03-23T11:24:12.068467+0000","last_clean":"2024-03-23T11:24:12.068467+0000","last_became_active":"2024-03-23T11:24:11.079567+0000","last_became_peered":"2024-03-23T11:24:11.079567+0000","last_unstale":"2024-03-23T11:24:12.068467+0000","last_undegraded":"2024-03-23T11:24:12.068467+0000","last_fullsized":"2024-03-23T11:24:12.068467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:44:38.284660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066565+0000","last_change":"2024-03-23T11:24:10.574701+0000","last_active":"2024-03-23T11:24:12.066565+0000","last_peered":"2024-03-23T11:24:12.066565+0000","last_clean":"2024-03-23T11:24:12.066565+0000","last_became_active":"2024-03-23T11:24:10.574451+0000","last_became_peered":"2024-03-23T11:24:10.574451+0000","last_unstale":"2024-03-23T11:24:12.066565+0000","last_undegraded":"2024-03-23T11:24:12.066565+0000","last_fullsized":"2024-03-23T11:24:12.066565+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:14:46.178001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194245+0000","last_change":"2024-03-23T11:24:11.079284+0000","last_active":"2024-03-23T11:24:13.194245+0000","last_peered":"2024-03-23T11:24:13.194245+0000","last_clean":"2024-03-23T11:24:13.194245+0000","last_became_active":"2024-03-23T11:24:11.078977+0000","last_became_peered":"2024-03-23T11:24:11.078977+0000","last_unstale":"2024-03-23T11:24:13.194245+0000","last_undegraded":"2024-03-23T11:24:13.194245+0000","last_fullsized":"2024-03-23T11:24:13.194245+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:26:09.227469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070235+0000","last_change":"2024-03-23T11:24:10.583159+0000","last_active":"2024-03-23T11:24:12.070235+0000","last_peered":"2024-03-23T11:24:12.070235+0000","last_clean":"2024-03-23T11:24:12.070235+0000","last_became_active":"2024-03-23T11:24:10.582665+0000","last_became_peered":"2024-03-23T11:24:10.582665+0000","last_unstale":"2024-03-23T11:24:12.070235+0000","last_undegraded":"2024-03-23T11:24:12.070235+0000","last_fullsized":"2024-03-23T11:24:12.070235+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:17:54.725364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.091362+0000","last_change":"2024-03-23T11:24:11.085719+0000","last_active":"2024-03-23T11:24:11.091362+0000","last_peered":"2024-03-23T11:24:11.091362+0000","last_clean":"2024-03-23T11:24:11.091362+0000","last_became_active":"2024-03-23T11:24:11.085484+0000","last_became_peered":"2024-03-23T11:24:11.085484+0000","last_unstale":"2024-03-23T11:24:11.091362+0000","last_undegraded":"2024-03-23T11:24:11.091362+0000","last_fullsized":"2024-03-23T11:24:11.091362+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:18:53.314479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194265+0000","last_change":"2024-03-23T11:24:10.585405+0000","last_active":"2024-03-23T11:24:13.194265+0000","last_peered":"2024-03-23T11:24:13.194265+0000","last_clean":"2024-03-23T11:24:13.194265+0000","last_became_active":"2024-03-23T11:24:10.584986+0000","last_became_peered":"2024-03-23T11:24:10.584986+0000","last_unstale":"2024-03-23T11:24:13.194265+0000","last_undegraded":"2024-03-23T11:24:13.194265+0000","last_fullsized":"2024-03-23T11:24:13.194265+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:21:45.883312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.090534+0000","last_change":"2024-03-23T11:24:11.081623+0000","last_active":"2024-03-23T11:24:11.090534+0000","last_peered":"2024-03-23T11:24:11.090534+0000","last_clean":"2024-03-23T11:24:11.090534+0000","last_became_active":"2024-03-23T11:24:11.078518+0000","last_became_peered":"2024-03-23T11:24:11.078518+0000","last_unstale":"2024-03-23T11:24:11.090534+0000","last_undegraded":"2024-03-23T11:24:11.090534+0000","last_fullsized":"2024-03-23T11:24:11.090534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:21:16.959427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"73'8","reported_seq":19,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.071767+0000","last_change":"2024-03-23T11:24:10.583814+0000","last_active":"2024-03-23T11:24:13.071767+0000","last_peered":"2024-03-23T11:24:13.071767+0000","last_clean":"2024-03-23T11:24:13.071767+0000","last_became_active":"2024-03-23T11:24:10.581693+0000","last_became_peered":"2024-03-23T11:24:10.581693+0000","last_unstale":"2024-03-23T11:24:13.071767+0000","last_undegraded":"2024-03-23T11:24:13.071767+0000","last_fullsized":"2024-03-23T11:24:13.071767+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:16.184072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072280+0000","last_change":"2024-03-23T11:24:11.080112+0000","last_active":"2024-03-23T11:24:13.072280+0000","last_peered":"2024-03-23T11:24:13.072280+0000","last_clean":"2024-03-23T11:24:13.072280+0000","last_became_active":"2024-03-23T11:24:11.077841+0000","last_became_peered":"2024-03-23T11:24:11.077841+0000","last_unstale":"2024-03-23T11:24:13.072280+0000","last_undegraded":"2024-03-23T11:24:13.072280+0000","last_fullsized":"2024-03-23T11:24:13.072280+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:39:35.570836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"73'1","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071396+0000","last_change":"2024-03-23T11:24:10.578691+0000","last_active":"2024-03-23T11:24:12.071396+0000","last_peered":"2024-03-23T11:24:12.071396+0000","last_clean":"2024-03-23T11:24:12.071396+0000","last_became_active":"2024-03-23T11:24:10.578420+0000","last_became_peered":"2024-03-23T11:24:10.578420+0000","last_unstale":"2024-03-23T11:24:12.071396+0000","last_undegraded":"2024-03-23T11:24:12.071396+0000","last_fullsized":"2024-03-23T11:24:12.071396+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:49:47.385865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070720+0000","last_change":"2024-03-23T11:24:11.081542+0000","last_active":"2024-03-23T11:24:12.070720+0000","last_peered":"2024-03-23T11:24:12.070720+0000","last_clean":"2024-03-23T11:24:12.070720+0000","last_became_active":"2024-03-23T11:24:11.081219+0000","last_became_peered":"2024-03-23T11:24:11.081219+0000","last_unstale":"2024-03-23T11:24:12.070720+0000","last_undegraded":"2024-03-23T11:24:12.070720+0000","last_fullsized":"2024-03-23T11:24:12.070720+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:30:15.190912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072346+0000","last_change":"2024-03-23T11:24:10.584541+0000","last_active":"2024-03-23T11:24:13.072346+0000","last_peered":"2024-03-23T11:24:13.072346+0000","last_clean":"2024-03-23T11:24:13.072346+0000","last_became_active":"2024-03-23T11:24:10.582141+0000","last_became_peered":"2024-03-23T11:24:10.582141+0000","last_unstale":"2024-03-23T11:24:13.072346+0000","last_undegraded":"2024-03-23T11:24:13.072346+0000","last_fullsized":"2024-03-23T11:24:13.072346+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:22:13.102795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.067006+0000","last_change":"2024-03-23T11:24:11.068894+0000","last_active":"2024-03-23T11:24:12.067006+0000","last_peered":"2024-03-23T11:24:12.067006+0000","last_clean":"2024-03-23T11:24:12.067006+0000","last_became_active":"2024-03-23T11:24:11.067737+0000","last_became_peered":"2024-03-23T11:24:11.067737+0000","last_unstale":"2024-03-23T11:24:12.067006+0000","last_undegraded":"2024-03-23T11:24:12.067006+0000","last_fullsized":"2024-03-23T11:24:12.067006+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:54:06.904127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"73'1","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071142+0000","last_change":"2024-03-23T11:24:10.578459+0000","last_active":"2024-03-23T11:24:12.071142+0000","last_peered":"2024-03-23T11:24:12.071142+0000","last_clean":"2024-03-23T11:24:12.071142+0000","last_became_active":"2024-03-23T11:24:10.578148+0000","last_became_peered":"2024-03-23T11:24:10.578148+0000","last_unstale":"2024-03-23T11:24:12.071142+0000","last_undegraded":"2024-03-23T11:24:12.071142+0000","last_fullsized":"2024-03-23T11:24:12.071142+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:46:08.860795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.091557+0000","last_change":"2024-03-23T11:24:11.084557+0000","last_active":"2024-03-23T11:24:11.091557+0000","last_peered":"2024-03-23T11:24:11.091557+0000","last_clean":"2024-03-23T11:24:11.091557+0000","last_became_active":"2024-03-23T11:24:11.084272+0000","last_became_peered":"2024-03-23T11:24:11.084272+0000","last_unstale":"2024-03-23T11:24:11.091557+0000","last_undegraded":"2024-03-23T11:24:11.091557+0000","last_fullsized":"2024-03-23T11:24:11.091557+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.959333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069164+0000","last_change":"2024-03-23T11:24:10.589923+0000","last_active":"2024-03-23T11:24:12.069164+0000","last_peered":"2024-03-23T11:24:12.069164+0000","last_clean":"2024-03-23T11:24:12.069164+0000","last_became_active":"2024-03-23T11:24:10.589535+0000","last_became_peered":"2024-03-23T11:24:10.589535+0000","last_unstale":"2024-03-23T11:24:12.069164+0000","last_undegraded":"2024-03-23T11:24:12.069164+0000","last_fullsized":"2024-03-23T11:24:12.069164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:39:01.813788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262505+0000","last_change":"2024-03-23T11:24:11.070257+0000","last_active":"2024-03-23T11:24:13.262505+0000","last_peered":"2024-03-23T11:24:13.262505+0000","last_clean":"2024-03-23T11:24:13.262505+0000","last_became_active":"2024-03-23T11:24:11.069988+0000","last_became_peered":"2024-03-23T11:24:11.069988+0000","last_unstale":"2024-03-23T11:24:13.262505+0000","last_undegraded":"2024-03-23T11:24:13.262505+0000","last_fullsized":"2024-03-23T11:24:13.262505+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:50:54.488598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070267+0000","last_change":"2024-03-23T11:24:10.576941+0000","last_active":"2024-03-23T11:24:12.070267+0000","last_peered":"2024-03-23T11:24:12.070267+0000","last_clean":"2024-03-23T11:24:12.070267+0000","last_became_active":"2024-03-23T11:24:10.576679+0000","last_became_peered":"2024-03-23T11:24:10.576679+0000","last_unstale":"2024-03-23T11:24:12.070267+0000","last_undegraded":"2024-03-23T11:24:12.070267+0000","last_fullsized":"2024-03-23T11:24:12.070267+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:45:09.482079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262502+0000","last_change":"2024-03-23T11:24:11.085777+0000","last_active":"2024-03-23T11:24:13.262502+0000","last_peered":"2024-03-23T11:24:13.262502+0000","last_clean":"2024-03-23T11:24:13.262502+0000","last_became_active":"2024-03-23T11:24:11.085510+0000","last_became_peered":"2024-03-23T11:24:11.085510+0000","last_unstale":"2024-03-23T11:24:13.262502+0000","last_undegraded":"2024-03-23T11:24:13.262502+0000","last_fullsized":"2024-03-23T11:24:13.262502+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:20:09.740677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194565+0000","last_change":"2024-03-23T11:24:10.586830+0000","last_active":"2024-03-23T11:24:13.194565+0000","last_peered":"2024-03-23T11:24:13.194565+0000","last_clean":"2024-03-23T11:24:13.194565+0000","last_became_active":"2024-03-23T11:24:10.586435+0000","last_became_peered":"2024-03-23T11:24:10.586435+0000","last_unstale":"2024-03-23T11:24:13.194565+0000","last_undegraded":"2024-03-23T11:24:13.194565+0000","last_fullsized":"2024-03-23T11:24:13.194565+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:54:33.135399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262156+0000","last_change":"2024-03-23T11:24:11.087515+0000","last_active":"2024-03-23T11:24:13.262156+0000","last_peered":"2024-03-23T11:24:13.262156+0000","last_clean":"2024-03-23T11:24:13.262156+0000","last_became_active":"2024-03-23T11:24:11.087204+0000","last_became_peered":"2024-03-23T11:24:11.087204+0000","last_unstale":"2024-03-23T11:24:13.262156+0000","last_undegraded":"2024-03-23T11:24:13.262156+0000","last_fullsized":"2024-03-23T11:24:13.262156+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:58:19.066834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.071480+0000","last_change":"2024-03-23T11:24:10.581764+0000","last_active":"2024-03-23T11:24:11.071480+0000","last_peered":"2024-03-23T11:24:11.071480+0000","last_clean":"2024-03-23T11:24:11.071480+0000","last_became_active":"2024-03-23T11:24:10.581448+0000","last_became_peered":"2024-03-23T11:24:10.581448+0000","last_unstale":"2024-03-23T11:24:11.071480+0000","last_undegraded":"2024-03-23T11:24:11.071480+0000","last_fullsized":"2024-03-23T11:24:11.071480+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:44.950044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.088781+0000","last_change":"2024-03-23T11:24:11.081596+0000","last_active":"2024-03-23T11:24:11.088781+0000","last_peered":"2024-03-23T11:24:11.088781+0000","last_clean":"2024-03-23T11:24:11.088781+0000","last_became_active":"2024-03-23T11:24:11.081321+0000","last_became_peered":"2024-03-23T11:24:11.081321+0000","last_unstale":"2024-03-23T11:24:11.088781+0000","last_undegraded":"2024-03-23T11:24:11.088781+0000","last_fullsized":"2024-03-23T11:24:11.088781+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:28.705045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071493+0000","last_change":"2024-03-23T11:24:10.579383+0000","last_active":"2024-03-23T11:24:12.071493+0000","last_peered":"2024-03-23T11:24:12.071493+0000","last_clean":"2024-03-23T11:24:12.071493+0000","last_became_active":"2024-03-23T11:24:10.579077+0000","last_became_peered":"2024-03-23T11:24:10.579077+0000","last_unstale":"2024-03-23T11:24:12.071493+0000","last_undegraded":"2024-03-23T11:24:12.071493+0000","last_fullsized":"2024-03-23T11:24:12.071493+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:22:13.102795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194200+0000","last_change":"2024-03-23T11:24:11.070880+0000","last_active":"2024-03-23T11:24:13.194200+0000","last_peered":"2024-03-23T11:24:13.194200+0000","last_clean":"2024-03-23T11:24:13.194200+0000","last_became_active":"2024-03-23T11:24:11.070414+0000","last_became_peered":"2024-03-23T11:24:11.070414+0000","last_unstale":"2024-03-23T11:24:13.194200+0000","last_undegraded":"2024-03-23T11:24:13.194200+0000","last_fullsized":"2024-03-23T11:24:13.194200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:04:08.143544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194913+0000","last_change":"2024-03-23T11:24:10.581413+0000","last_active":"2024-03-23T11:24:13.194913+0000","last_peered":"2024-03-23T11:24:13.194913+0000","last_clean":"2024-03-23T11:24:13.194913+0000","last_became_active":"2024-03-23T11:24:10.581077+0000","last_became_peered":"2024-03-23T11:24:10.581077+0000","last_unstale":"2024-03-23T11:24:13.194913+0000","last_undegraded":"2024-03-23T11:24:13.194913+0000","last_fullsized":"2024-03-23T11:24:13.194913+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:42:43.960260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069019+0000","last_change":"2024-03-23T11:24:11.092747+0000","last_active":"2024-03-23T11:24:12.069019+0000","last_peered":"2024-03-23T11:24:12.069019+0000","last_clean":"2024-03-23T11:24:12.069019+0000","last_became_active":"2024-03-23T11:24:11.092456+0000","last_became_peered":"2024-03-23T11:24:11.092456+0000","last_unstale":"2024-03-23T11:24:12.069019+0000","last_undegraded":"2024-03-23T11:24:12.069019+0000","last_fullsized":"2024-03-23T11:24:12.069019+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:32.792323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.065772+0000","last_change":"2024-03-23T11:24:10.579459+0000","last_active":"2024-03-23T11:24:11.065772+0000","last_peered":"2024-03-23T11:24:11.065772+0000","last_clean":"2024-03-23T11:24:11.065772+0000","last_became_active":"2024-03-23T11:24:10.578558+0000","last_became_peered":"2024-03-23T11:24:10.578558+0000","last_unstale":"2024-03-23T11:24:11.065772+0000","last_undegraded":"2024-03-23T11:24:11.065772+0000","last_fullsized":"2024-03-23T11:24:11.065772+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:21.208414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069799+0000","last_change":"2024-03-23T11:24:11.072559+0000","last_active":"2024-03-23T11:24:12.069799+0000","last_peered":"2024-03-23T11:24:12.069799+0000","last_clean":"2024-03-23T11:24:12.069799+0000","last_became_active":"2024-03-23T11:24:11.072292+0000","last_became_peered":"2024-03-23T11:24:11.072292+0000","last_unstale":"2024-03-23T11:24:12.069799+0000","last_undegraded":"2024-03-23T11:24:12.069799+0000","last_fullsized":"2024-03-23T11:24:12.069799+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:59:27.736931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.064776+0000","last_change":"2024-03-23T11:24:10.578417+0000","last_active":"2024-03-23T11:24:11.064776+0000","last_peered":"2024-03-23T11:24:11.064776+0000","last_clean":"2024-03-23T11:24:11.064776+0000","last_became_active":"2024-03-23T11:24:10.577492+0000","last_became_peered":"2024-03-23T11:24:10.577492+0000","last_unstale":"2024-03-23T11:24:11.064776+0000","last_undegraded":"2024-03-23T11:24:11.064776+0000","last_fullsized":"2024-03-23T11:24:11.064776+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:25:34.271784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.090519+0000","last_change":"2024-03-23T11:24:11.081149+0000","last_active":"2024-03-23T11:24:11.090519+0000","last_peered":"2024-03-23T11:24:11.090519+0000","last_clean":"2024-03-23T11:24:11.090519+0000","last_became_active":"2024-03-23T11:24:11.080908+0000","last_became_peered":"2024-03-23T11:24:11.080908+0000","last_unstale":"2024-03-23T11:24:11.090519+0000","last_undegraded":"2024-03-23T11:24:11.090519+0000","last_fullsized":"2024-03-23T11:24:11.090519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:50:10.372150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.065693+0000","last_change":"2024-03-23T11:24:10.583872+0000","last_active":"2024-03-23T11:24:11.065693+0000","last_peered":"2024-03-23T11:24:11.065693+0000","last_clean":"2024-03-23T11:24:11.065693+0000","last_became_active":"2024-03-23T11:24:10.582786+0000","last_became_peered":"2024-03-23T11:24:10.582786+0000","last_unstale":"2024-03-23T11:24:11.065693+0000","last_undegraded":"2024-03-23T11:24:11.065693+0000","last_fullsized":"2024-03-23T11:24:11.065693+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:10.886106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070126+0000","last_change":"2024-03-23T11:24:11.084432+0000","last_active":"2024-03-23T11:24:12.070126+0000","last_peered":"2024-03-23T11:24:12.070126+0000","last_clean":"2024-03-23T11:24:12.070126+0000","last_became_active":"2024-03-23T11:24:11.084137+0000","last_became_peered":"2024-03-23T11:24:11.084137+0000","last_unstale":"2024-03-23T11:24:12.070126+0000","last_undegraded":"2024-03-23T11:24:12.070126+0000","last_fullsized":"2024-03-23T11:24:12.070126+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:54:34.147648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.065982+0000","last_change":"2024-03-23T11:24:10.568748+0000","last_active":"2024-03-23T11:24:12.065982+0000","last_peered":"2024-03-23T11:24:12.065982+0000","last_clean":"2024-03-23T11:24:12.065982+0000","last_became_active":"2024-03-23T11:24:10.568383+0000","last_became_peered":"2024-03-23T11:24:10.568383+0000","last_unstale":"2024-03-23T11:24:12.065982+0000","last_undegraded":"2024-03-23T11:24:12.065982+0000","last_fullsized":"2024-03-23T11:24:12.065982+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:44:31.486382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066356+0000","last_change":"2024-03-23T11:24:11.067303+0000","last_active":"2024-03-23T11:24:12.066356+0000","last_peered":"2024-03-23T11:24:12.066356+0000","last_clean":"2024-03-23T11:24:12.066356+0000","last_became_active":"2024-03-23T11:24:11.066956+0000","last_became_peered":"2024-03-23T11:24:11.066956+0000","last_unstale":"2024-03-23T11:24:12.066356+0000","last_undegraded":"2024-03-23T11:24:12.066356+0000","last_fullsized":"2024-03-23T11:24:12.066356+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:07:28.883218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.071474+0000","last_change":"2024-03-23T11:24:10.584454+0000","last_active":"2024-03-23T11:24:13.071474+0000","last_peered":"2024-03-23T11:24:13.071474+0000","last_clean":"2024-03-23T11:24:13.071474+0000","last_became_active":"2024-03-23T11:24:10.582586+0000","last_became_peered":"2024-03-23T11:24:10.582586+0000","last_unstale":"2024-03-23T11:24:13.071474+0000","last_undegraded":"2024-03-23T11:24:13.071474+0000","last_fullsized":"2024-03-23T11:24:13.071474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:42:30.175261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070158+0000","last_change":"2024-03-23T11:24:11.083928+0000","last_active":"2024-03-23T11:24:12.070158+0000","last_peered":"2024-03-23T11:24:12.070158+0000","last_clean":"2024-03-23T11:24:12.070158+0000","last_became_active":"2024-03-23T11:24:11.083679+0000","last_became_peered":"2024-03-23T11:24:11.083679+0000","last_unstale":"2024-03-23T11:24:12.070158+0000","last_undegraded":"2024-03-23T11:24:12.070158+0000","last_fullsized":"2024-03-23T11:24:12.070158+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:42:44.972509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070490+0000","last_change":"2024-03-23T11:24:10.575445+0000","last_active":"2024-03-23T11:24:12.070490+0000","last_peered":"2024-03-23T11:24:12.070490+0000","last_clean":"2024-03-23T11:24:12.070490+0000","last_became_active":"2024-03-23T11:24:10.575115+0000","last_became_peered":"2024-03-23T11:24:10.575115+0000","last_unstale":"2024-03-23T11:24:12.070490+0000","last_undegraded":"2024-03-23T11:24:12.070490+0000","last_fullsized":"2024-03-23T11:24:12.070490+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:13:49.012961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.261879+0000","last_change":"2024-03-23T11:24:11.086227+0000","last_active":"2024-03-23T11:24:13.261879+0000","last_peered":"2024-03-23T11:24:13.261879+0000","last_clean":"2024-03-23T11:24:13.261879+0000","last_became_active":"2024-03-23T11:24:11.085990+0000","last_became_peered":"2024-03-23T11:24:11.085990+0000","last_unstale":"2024-03-23T11:24:13.261879+0000","last_undegraded":"2024-03-23T11:24:13.261879+0000","last_fullsized":"2024-03-23T11:24:13.261879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:04:44.710371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070286+0000","last_change":"2024-03-23T11:24:10.577230+0000","last_active":"2024-03-23T11:24:12.070286+0000","last_peered":"2024-03-23T11:24:12.070286+0000","last_clean":"2024-03-23T11:24:12.070286+0000","last_became_active":"2024-03-23T11:24:10.576144+0000","last_became_peered":"2024-03-23T11:24:10.576144+0000","last_unstale":"2024-03-23T11:24:12.070286+0000","last_undegraded":"2024-03-23T11:24:12.070286+0000","last_fullsized":"2024-03-23T11:24:12.070286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:50:09.359901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":218,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.058541+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:24:11.058541+0000","last_peered":"2024-03-23T11:24:11.058541+0000","last_clean":"2024-03-23T11:24:11.058541+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:24:11.058541+0000","last_undegraded":"2024-03-23T11:24:11.058541+0000","last_fullsized":"2024-03-23T11:24:11.058541+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542212,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1499136,"data_stored":968442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.999,"15min":0.92500000000000004},"min":{"1min":0.68700000000000006,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.3580000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.244},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0229999999999999,"15min":0.98799999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":1.155}]},{"osd":1,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.155,"15min":1.171},"min":{"1min":0.73999999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.617,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":1.331},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2330000000000001,"15min":1.246},"min":{"1min":0.90400000000000003,"5min":0.753,"15min":0.753},"max":{"1min":1.5609999999999999,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.2969999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0409999999999999,"15min":1.0469999999999999},"min":{"1min":0.72299999999999998,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.3460000000000001,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1779999999999999,"15min":1.1399999999999999},"min":{"1min":0.84599999999999997,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.548,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.091}]},{"osd":3,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.1859999999999999,"15min":1.1379999999999999},"min":{"1min":0.91800000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.6359999999999999,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2350000000000001,"15min":1.2490000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.4490000000000001,"5min":1.778,"15min":1.778},"last":1.381}]},{"osd":4,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2330000000000001,"15min":1.2210000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.591,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.488},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.399,"15min":1.363},"min":{"1min":0.65100000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.76,"5min":1.845,"15min":1.845},"last":1.2070000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.321,"15min":1.331},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.677,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.397,"15min":1.3600000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.7969999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.5860000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4119999999999999,"15min":1.4179999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.73,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.395},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.357,"15min":1.337},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.056}]}]},{"osd":6,"up_from":14,"seq":60129542212,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":901120,"data_stored":376881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1439999999999999,"15min":1.0860000000000001},"min":{"1min":0.63600000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.5660000000000001,"5min":3.9140000000000001,"15min":3.9140000000000001},"last":1.399},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.1399999999999999,"15min":1.1120000000000001},"min":{"1min":0.70199999999999996,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6639999999999999,"5min":6.6539999999999999,"15min":6.6539999999999999},"last":1.1499999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2170000000000001,"15min":1.2230000000000001},"min":{"1min":0.89600000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.7410000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.224,"5min":1.2130000000000001,"15min":1.26},"min":{"1min":0.90100000000000002,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.5660000000000001,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.0469999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2470000000000001,"15min":1.25},"min":{"1min":0.92200000000000004,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.4810000000000001,"5min":3.73,"15min":3.73},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.27,"15min":1.2949999999999999},"min":{"1min":0.94799999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.8340000000000001,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.0980000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.3440000000000001,"15min":1.2649999999999999},"min":{"1min":0.83099999999999996,"5min":0.749,"15min":0.749},"max":{"1min":1.5149999999999999,"5min":12.914,"15min":12.914},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3169999999999999,"15min":1.369},"min":{"1min":1.1539999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.458,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":1.2549999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3009999999999999,"15min":1.391},"min":{"1min":0.91500000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.665,"5min":2.702,"15min":2.702},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.365,"15min":1.415},"min":{"1min":0.86099999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.712,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":1.5029999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.458,"15min":1.5489999999999999},"min":{"1min":1.155,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.6899999999999999,"5min":3.097,"15min":3.097},"last":1.6299999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4419999999999999,"15min":1.502},"min":{"1min":1.1850000000000001,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.907,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.6839999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.5980000000000001,"15min":1.6819999999999999},"min":{"1min":1.127,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.7450000000000001,"5min":4.9470000000000001,"15min":4.9470000000000001},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.4370000000000001,"15min":1.5069999999999999},"min":{"1min":1.0169999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.601,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":1.3540000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542212,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":888832,"data_stored":371248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0860000000000001,"15min":1.1879999999999999},"min":{"1min":0.67400000000000004,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.623,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.85899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.643,"5min":2.355,"15min":2.355},"last":0.81299999999999994}]},{"osd":1,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1379999999999999,"15min":1.1839999999999999},"min":{"1min":0.75600000000000001,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.5209999999999999,"5min":2.2559999999999998,"15min":2.2559999999999998},"last":0.997},{"interface":"front","average":{"1min":1.169,"5min":1.1739999999999999,"15min":1.2210000000000001},"min":{"1min":0.72299999999999998,"5min":0.495,"15min":0.495},"max":{"1min":1.5760000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.4410000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.238,"15min":1.2609999999999999},"min":{"1min":0.71399999999999997,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6040000000000001,"5min":2.012,"15min":2.012},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.216,"15min":1.2370000000000001},"min":{"1min":0.754,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.7090000000000001,"5min":1.946,"15min":1.946},"last":0.877}]},{"osd":3,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.304,"15min":1.3140000000000001},"min":{"1min":0.76600000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.548,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3680000000000001,"15min":1.413},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.7709999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.337}]},{"osd":4,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.3220000000000001,"15min":1.4139999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.681,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.6040000000000001},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4830000000000001,"15min":1.5449999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.748,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.5640000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.381,"15min":1.4790000000000001},"min":{"1min":0.59199999999999997,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6259999999999999,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.4059999999999999,"15min":1.53},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.754,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.835}]},{"osd":7,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.454,"15min":1.5580000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7390000000000001,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.548,"15min":1.621},"min":{"1min":0.69099999999999995,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.819,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":1.5009999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542212,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":888832,"data_stored":371228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.131,"15min":1.1060000000000001},"min":{"1min":0.84799999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.49,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.244,"15min":1.214},"min":{"1min":1.01,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.679,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.288}]},{"osd":1,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.161,"15min":1.1200000000000001},"min":{"1min":0.91300000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6879999999999999,"5min":4.327,"15min":4.327},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.151,"15min":1.111},"min":{"1min":0.81699999999999995,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.468,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.2190000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2030000000000001,"15min":1.226},"min":{"1min":0.94299999999999995,"5min":0.624,"15min":0.624},"max":{"1min":1.6830000000000001,"5min":2.6080000000000001,"15min":2.6080000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.292,"15min":1.236},"min":{"1min":0.81799999999999995,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.6679999999999999,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":1.246}]},{"osd":3,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.222,"15min":1.2190000000000001},"min":{"1min":0.89800000000000002,"5min":0.622,"15min":0.622},"max":{"1min":1.605,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":1.135},{"interface":"front","average":{"1min":1.357,"5min":1.3240000000000001,"15min":1.264},"min":{"1min":0.92900000000000005,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.742,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":1.4239999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.238,"15min":1.284},"min":{"1min":0.80200000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.5740000000000001,"5min":2.508,"15min":2.508},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.381,"15min":1.3640000000000001},"min":{"1min":0.83299999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.0249999999999999,"5min":5.859,"15min":5.859},"last":1.3440000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.403,"15min":1.385},"min":{"1min":0.78000000000000003,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.857,"5min":5.3339999999999996,"15min":5.3339999999999996},"last":2.222},{"interface":"front","average":{"1min":1.448,"5min":1.4970000000000001,"15min":1.4870000000000001},"min":{"1min":0.995,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.8720000000000001,"5min":5.3940000000000001,"15min":5.3940000000000001},"last":1.5760000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4770000000000001,"15min":1.4710000000000001},"min":{"1min":1.0760000000000001,"5min":0.754,"15min":0.754},"max":{"1min":2.3399999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":2.1549999999999998},{"interface":"front","average":{"1min":1.319,"5min":1.367,"15min":1.413},"min":{"1min":0.92600000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.8169999999999999,"5min":2.641,"15min":2.641},"last":2.028}]}]},{"osd":3,"up_from":14,"seq":60129542212,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":905216,"data_stored":377570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.96599999999999997,"15min":0.95999999999999996},"min":{"1min":0.71799999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.4059999999999999,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.147,"5min":1.1100000000000001,"15min":1.1060000000000001},"min":{"1min":0.73199999999999998,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.4039999999999999,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.2589999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":0.98799999999999999,"15min":1},"min":{"1min":0.70299999999999996,"5min":0.442,"15min":0.442},"max":{"1min":1.8180000000000001,"5min":1.976,"15min":1.976},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.25,"15min":1.264},"min":{"1min":0.91300000000000003,"5min":0.5,"15min":0.5},"max":{"1min":1.7649999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.8100000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.121,"15min":1.143},"min":{"1min":0.71199999999999997,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0779999999999998,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":1.333},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1759999999999999,"15min":1.2549999999999999},"min":{"1min":0.877,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.6060000000000001,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":2.2440000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2669999999999999,"15min":1.2889999999999999},"min":{"1min":0.93799999999999994,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.506,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4650000000000001,"15min":1.5329999999999999},"min":{"1min":1.26,"5min":0.998,"15min":0.998},"max":{"1min":1.966,"5min":4.8970000000000002,"15min":4.8970000000000002},"last":1.7490000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.4730000000000001,"15min":1.5249999999999999},"min":{"1min":1.151,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":1.8080000000000001,"5min":3.4910000000000001,"15min":3.4910000000000001},"last":2.1520000000000001},{"interface":"front","average":{"1min":1.387,"5min":1.399,"15min":1.4690000000000001},"min":{"1min":1.079,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.637,"5min":3.125,"15min":3.125},"last":1.6120000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.4179999999999999,"15min":1.5249999999999999},"min":{"1min":1.0780000000000001,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.024,"5min":4.5499999999999998,"15min":4.5499999999999998},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5029999999999999,"15min":1.5640000000000001},"min":{"1min":1.3180000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.9199999999999999,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":1.974}]},{"osd":7,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.46,"15min":1.524},"min":{"1min":1.264,"5min":0.877,"15min":0.877},"max":{"1min":1.7669999999999999,"5min":2.952,"15min":2.952},"last":1.718},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5469999999999999,"15min":1.631},"min":{"1min":1.1200000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.9850000000000001,"5min":4.798,"15min":4.798},"last":1.4670000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":897024,"data_stored":372722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.878,"15min":0.90000000000000002},"min":{"1min":0.64300000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.405,"5min":1.462,"15min":1.462},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.145,"5min":1.0549999999999999,"15min":0.995},"min":{"1min":0.47299999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.5880000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.66200000000000003}]},{"osd":1,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.90700000000000003,"15min":0.95799999999999996},"min":{"1min":0.436,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.351,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.03,"15min":0.98999999999999999},"min":{"1min":0.61799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4710000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.60599999999999998}]},{"osd":3,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1180000000000001,"15min":1.119},"min":{"1min":0.59399999999999997,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.782,"5min":1.98,"15min":1.98},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.0600000000000001,"15min":1.0820000000000001},"min":{"1min":0.72099999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.0579999999999998,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":0.64900000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.361,"15min":1.353},"min":{"1min":1.04,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.702,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.476,"5min":1.4139999999999999,"15min":1.361},"min":{"1min":0.92900000000000005,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.869,"5min":2.085,"15min":2.085},"last":0.92200000000000004}]},{"osd":5,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.3759999999999999,"15min":1.4199999999999999},"min":{"1min":1.1120000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.2589999999999999,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.478,"5min":1.377,"15min":1.3740000000000001},"min":{"1min":0.94099999999999995,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.9119999999999999,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.038}]},{"osd":6,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.383,"15min":1.3720000000000001},"min":{"1min":1.02,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6859999999999999,"5min":2.758,"15min":2.758},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.4670000000000001,"15min":1.5189999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.9710000000000001,"5min":2.8580000000000001,"15min":2.8580000000000001},"last":0.80200000000000005}]},{"osd":7,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.454,"15min":1.403},"min":{"1min":1.004,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.752,"5min":2.246,"15min":2.246},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.524,"15min":1.5900000000000001},"min":{"1min":1.1739999999999999,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.9710000000000001,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.89200000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542213,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":913408,"data_stored":378969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.92100000000000004,"15min":0.873},"min":{"1min":0.64600000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.6459999999999999,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.006,"15min":0.996},"min":{"1min":0.77500000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.508,"5min":1.99,"15min":1.99},"last":0.75600000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.95499999999999996,"15min":0.97499999999999998},"min":{"1min":0.60699999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.6479999999999999,"5min":1.738,"15min":1.738},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.143,"5min":1.115,"15min":1.1259999999999999},"min":{"1min":0.90200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.6619999999999999,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.55100000000000005}]},{"osd":3,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.236,"15min":1.23},"min":{"1min":0.94799999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.4750000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.147,"5min":1.113,"15min":1.1639999999999999},"min":{"1min":0.83499999999999996,"5min":0.434,"15min":0.434},"max":{"1min":1.4990000000000001,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":0.90600000000000003}]},{"osd":4,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.3080000000000001,"15min":1.26},"min":{"1min":0.99099999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.8029999999999999,"5min":2.153,"15min":2.153},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3129999999999999,"15min":1.339},"min":{"1min":0.91700000000000004,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.8240000000000001,"5min":2.758,"15min":2.758},"last":0.84699999999999998}]},{"osd":5,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.359,"15min":1.429},"min":{"1min":1.1100000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.855,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.218},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4259999999999999,"15min":1.462},"min":{"1min":1.1399999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.893,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.2749999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4279999999999999,"15min":1.4370000000000001},"min":{"1min":1.0209999999999999,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.73,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.325},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4770000000000001,"15min":1.569},"min":{"1min":1.123,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9430000000000001,"5min":2.9630000000000001,"15min":2.9630000000000001},"last":1.109}]},{"osd":7,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.5329999999999999,"15min":1.583},"min":{"1min":1.2250000000000001,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.9890000000000001,"5min":3.2749999999999999,"15min":3.2749999999999999},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5289999999999999,"15min":1.5820000000000001},"min":{"1min":1.179,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.887,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.244}]}]},{"osd":0,"up_from":14,"seq":60129542211,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1474560,"data_stored":960857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.86099999999999999,"15min":0.85499999999999998},"min":{"1min":0.43099999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.115,"5min":2.3769999999999998,"15min":2.3769999999999998},"last":1.107},{"interface":"front","average":{"1min":1.04,"5min":0.98299999999999998,"15min":0.93799999999999994},"min":{"1min":0.63,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.5349999999999999,"5min":1.694,"15min":1.694},"last":0.996}]},{"osd":2,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.071,"15min":1.04},"min":{"1min":0.497,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.538,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.05,"15min":1.016},"min":{"1min":0.63200000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.5009999999999999,"5min":1.802,"15min":1.802},"last":0.77600000000000002}]},{"osd":3,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0880000000000001,"15min":1.0980000000000001},"min":{"1min":0.53700000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.5940000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3009999999999999,"15min":1.3129999999999999},"min":{"1min":0.59199999999999997,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.609,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.1519999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.353,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5569999999999999,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.087},{"interface":"front","average":{"1min":1.331,"5min":1.2749999999999999,"15min":1.2709999999999999},"min":{"1min":0.754,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.6990000000000001,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.675}]},{"osd":5,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.3440000000000001,"15min":1.4339999999999999},"min":{"1min":0.96099999999999997,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6879999999999999,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.444,"15min":1.528},"min":{"1min":0.77100000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.915,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.7789999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.454,"15min":1.5449999999999999},"min":{"1min":0.98099999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.768,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":1.712},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.399,"15min":1.4299999999999999},"min":{"1min":0.90100000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.1150000000000002,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.1279999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.514,"15min":1.546},"min":{"1min":1.246,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0489999999999999,"5min":2.617,"15min":2.617},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.4830000000000001,"15min":1.5109999999999999},"min":{"1min":1.0049999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":2.266,"5min":3.081,"15min":3.081},"last":1.762}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:24:15.514 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:24:15.514 INFO:tasks.ceph.mds.d.smithi120.stderr:2024-03-23T11:24:15.519+0000 7f6c14000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:24:15.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.778+0000 7ffb3d800640 1 Processor -- start 2024-03-23T11:24:15.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.779+0000 7ffb3d800640 1 -- start start 2024-03-23T11:24:15.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.780+0000 7ffb3d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffb38160420 0x7ffb381608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.780+0000 7ffb3d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb380b5db0 0x7ffb380b6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.780+0000 7ffb3d800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb380ce510 con 0x7ffb3815fcf0 2024-03-23T11:24:15.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.780+0000 7ffb3d800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb38160df0 con 0x7ffb38160420 2024-03-23T11:24:15.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.780+0000 7ffb3d800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb380cdc70 con 0x7ffb380b5db0 2024-03-23T11:24:15.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.780+0000 7ffb37400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffb38160420 0x7ffb381608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.780+0000 7ffb37400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffb38160420 0x7ffb381608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59404/0 (socket says 172.21.15.62:59404) 2024-03-23T11:24:15.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.780+0000 7ffb37400640 1 -- 172.21.15.62:0/974627568 learned_addr learned my addr 172.21.15.62:0/974627568 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:15.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.781+0000 7ffb3ce00640 1 --2- 172.21.15.62:0/974627568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb380b5db0 0x7ffb380b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.781+0000 7ffb37400640 1 -- 172.21.15.62:0/974627568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb380b5db0 msgr2=0x7ffb380b6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:15.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.781+0000 7ffb37400640 1 --2- 172.21.15.62:0/974627568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb380b5db0 0x7ffb380b6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.781+0000 7ffb37400640 1 -- 172.21.15.62:0/974627568 >> v1:172.21.15.120:6789/0 conn(0x7ffb3815fcf0 legacy=0x7ffb380d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:15.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.781+0000 7ffb37400640 1 -- 172.21.15.62:0/974627568 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb380cdf50 con 0x7ffb38160420 2024-03-23T11:24:15.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.782+0000 7ffb37400640 1 --2- 172.21.15.62:0/974627568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb38160420 0x7ffb381608b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7ffb20009870 tx=0x7ffb2002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45cebb36f7c66b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:15.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.782+0000 7ffb36a00640 1 -- 172.21.15.62:0/974627568 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb200317c0 con 0x7ffb38160420 2024-03-23T11:24:15.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.782+0000 7ffb36a00640 1 -- 172.21.15.62:0/974627568 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb20031920 con 0x7ffb38160420 2024-03-23T11:24:15.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.782+0000 7ffb36a00640 1 -- 172.21.15.62:0/974627568 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb20031cb0 con 0x7ffb38160420 2024-03-23T11:24:15.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.783+0000 7ffb3d800640 1 -- 172.21.15.62:0/974627568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb38160420 msgr2=0x7ffb381608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:15.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.783+0000 7ffb3d800640 1 --2- 172.21.15.62:0/974627568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb38160420 0x7ffb381608b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7ffb20009870 tx=0x7ffb2002b860 comp rx=0 tx=0).stop 2024-03-23T11:24:15.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.783+0000 7ffb3d800640 1 -- 172.21.15.62:0/974627568 shutdown_connections 2024-03-23T11:24:15.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.783+0000 7ffb3d800640 1 --2- 172.21.15.62:0/974627568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb380b5db0 0x7ffb380b6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.783+0000 7ffb3d800640 1 --2- 172.21.15.62:0/974627568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb38160420 0x7ffb381608b0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.783+0000 7ffb3d800640 1 -- 172.21.15.62:0/974627568 >> 172.21.15.62:0/974627568 conn(0x7ffb38089f60 msgr2=0x7ffb380d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:15.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.783+0000 7ffb3d800640 1 -- 172.21.15.62:0/974627568 shutdown_connections 2024-03-23T11:24:15.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.783+0000 7ffb3d800640 1 -- 172.21.15.62:0/974627568 wait complete. 2024-03-23T11:24:15.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.785+0000 7ffb3d800640 1 Processor -- start 2024-03-23T11:24:15.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.785+0000 7ffb3d800640 1 -- start start 2024-03-23T11:24:15.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.786+0000 7ffb3d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb380b5db0 0x7ffb381f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.786+0000 7ffb3d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb3815fcf0 0x7ffb380b79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.786+0000 7ffb3d800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb38160420 0x7ffb380b7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.786+0000 7ffb3d800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3814ee50 con 0x7ffb380b5db0 2024-03-23T11:24:15.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.786+0000 7ffb3d800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb380a0360 con 0x7ffb38160420 2024-03-23T11:24:15.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.786+0000 7ffb3d800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3809fda0 con 0x7ffb3815fcf0 2024-03-23T11:24:15.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.787+0000 7ffb37e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb380b5db0 0x7ffb381f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.787+0000 7ffb37e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb380b5db0 0x7ffb381f1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59414/0 (socket says 172.21.15.62:59414) 2024-03-23T11:24:15.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.787+0000 7ffb37e00640 1 -- 172.21.15.62:0/1633751969 learned_addr learned my addr 172.21.15.62:0/1633751969 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:15.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.787+0000 7ffb37400640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb3815fcf0 0x7ffb380b79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.787+0000 7ffb3ce00640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb38160420 0x7ffb380b7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.787+0000 7ffb37e00640 1 -- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb3815fcf0 msgr2=0x7ffb380b79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.787+0000 7ffb37e00640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb3815fcf0 0x7ffb380b79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.787+0000 7ffb37e00640 1 -- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb38160420 msgr2=0x7ffb380b7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.787+0000 7ffb37e00640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb38160420 0x7ffb380b7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.787+0000 7ffb37e00640 1 -- 172.21.15.62:0/1633751969 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb200094d0 con 0x7ffb380b5db0 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.787+0000 7ffb37400640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb3815fcf0 0x7ffb380b79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.788+0000 7ffb37e00640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb380b5db0 0x7ffb381f1ad0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800ba80 tx=0x7ffb2800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.788+0000 7ffb2fe00640 1 -- 172.21.15.62:0/1633751969 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2800cc00 con 0x7ffb380b5db0 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.788+0000 7ffb2fe00640 1 -- 172.21.15.62:0/1633751969 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2800cd60 con 0x7ffb380b5db0 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.788+0000 7ffb2fe00640 1 -- 172.21.15.62:0/1633751969 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb28003bb0 con 0x7ffb380b5db0 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.788+0000 7ffb3d800640 1 -- 172.21.15.62:0/1633751969 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb380a0080 con 0x7ffb380b5db0 2024-03-23T11:24:15.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.788+0000 7ffb3d800640 1 -- 172.21.15.62:0/1633751969 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb381f2010 con 0x7ffb380b5db0 2024-03-23T11:24:15.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.790+0000 7ffb3d800640 1 -- 172.21.15.62:0/1633751969 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb38066bb0 con 0x7ffb380b5db0 2024-03-23T11:24:15.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.790+0000 7ffb2fe00640 1 -- 172.21.15.62:0/1633751969 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ffb28014440 con 0x7ffb380b5db0 2024-03-23T11:24:15.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.791+0000 7ffb2fe00640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffb1406a7a0 0x7ffb1406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:15.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.791+0000 7ffb2fe00640 1 -- 172.21.15.62:0/1633751969 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb28088be0 con 0x7ffb380b5db0 2024-03-23T11:24:15.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.792+0000 7ffb37400640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffb1406a7a0 0x7ffb1406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.793+0000 7ffb37400640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffb1406a7a0 0x7ffb1406cc60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ffb20036000 tx=0x7ffb2003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:15.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.796+0000 7ffb2fe00640 1 -- 172.21.15.62:0/1633751969 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb28054000 con 0x7ffb380b5db0 2024-03-23T11:24:15.803 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:24:15.808 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:24:15.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:15.990+0000 7ffb3d800640 1 -- 172.21.15.62:0/1633751969 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb380b58b0 con 0x7ffb1406a7a0 2024-03-23T11:24:16.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.015+0000 7ffb2fe00640 1 -- 172.21.15.62:0/1633751969 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286867 (secure 0 0 0) 0x7ffb20031b90 con 0x7ffb1406a7a0 2024-03-23T11:24:16.012 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:16.016 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:24:16.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 -- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffb1406a7a0 msgr2=0x7ffb1406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:16.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffb1406a7a0 0x7ffb1406cc60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ffb20036000 tx=0x7ffb2003d040 comp rx=0 tx=0).stop 2024-03-23T11:24:16.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 -- 172.21.15.62:0/1633751969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb380b5db0 msgr2=0x7ffb381f1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:16.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb380b5db0 0x7ffb381f1ad0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800ba80 tx=0x7ffb2800bf40 comp rx=0 tx=0).stop 2024-03-23T11:24:16.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 -- 172.21.15.62:0/1633751969 shutdown_connections 2024-03-23T11:24:16.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffb1406a7a0 0x7ffb1406cc60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb38160420 0x7ffb380b7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffb3815fcf0 0x7ffb380b79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 --2- 172.21.15.62:0/1633751969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb380b5db0 0x7ffb381f1ad0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 -- 172.21.15.62:0/1633751969 >> 172.21.15.62:0/1633751969 conn(0x7ffb38089f60 msgr2=0x7ffb380b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:16.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 -- 172.21.15.62:0/1633751969 shutdown_connections 2024-03-23T11:24:16.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.022+0000 7ffb3d800640 1 -- 172.21.15.62:0/1633751969 wait complete. 2024-03-23T11:24:16.038 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-03-23T11:24:15.346067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240756,"kb_used_data":8172,"kb_used_omap":13,"kb_used_meta":232434,"kb_avail":754733964,"statfs":{"total":773094113280,"available":772847579136,"internally_reserved":0,"allocated":8368128,"data_stored":4177917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":238012504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.804304"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070614+0000","last_change":"2024-03-23T11:24:11.078223+0000","last_active":"2024-03-23T11:24:12.070614+0000","last_peered":"2024-03-23T11:24:12.070614+0000","last_clean":"2024-03-23T11:24:12.070614+0000","last_became_active":"2024-03-23T11:24:11.077962+0000","last_became_peered":"2024-03-23T11:24:11.077962+0000","last_unstale":"2024-03-23T11:24:12.070614+0000","last_undegraded":"2024-03-23T11:24:12.070614+0000","last_fullsized":"2024-03-23T11:24:12.070614+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:35:42.435823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071168+0000","last_change":"2024-03-23T11:24:11.087210+0000","last_active":"2024-03-23T11:24:12.071168+0000","last_peered":"2024-03-23T11:24:12.071168+0000","last_clean":"2024-03-23T11:24:12.071168+0000","last_became_active":"2024-03-23T11:24:11.086983+0000","last_became_peered":"2024-03-23T11:24:11.086983+0000","last_unstale":"2024-03-23T11:24:12.071168+0000","last_undegraded":"2024-03-23T11:24:12.071168+0000","last_fullsized":"2024-03-23T11:24:12.071168+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:20:59.165481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262303+0000","last_change":"2024-03-23T11:24:11.079994+0000","last_active":"2024-03-23T11:24:13.262303+0000","last_peered":"2024-03-23T11:24:13.262303+0000","last_clean":"2024-03-23T11:24:13.262303+0000","last_became_active":"2024-03-23T11:24:11.079643+0000","last_became_peered":"2024-03-23T11:24:11.079643+0000","last_unstale":"2024-03-23T11:24:13.262303+0000","last_undegraded":"2024-03-23T11:24:13.262303+0000","last_fullsized":"2024-03-23T11:24:13.262303+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:01:22.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066724+0000","last_change":"2024-03-23T11:24:11.086417+0000","last_active":"2024-03-23T11:24:12.066724+0000","last_peered":"2024-03-23T11:24:12.066724+0000","last_clean":"2024-03-23T11:24:12.066724+0000","last_became_active":"2024-03-23T11:24:11.086149+0000","last_became_peered":"2024-03-23T11:24:11.086149+0000","last_unstale":"2024-03-23T11:24:12.066724+0000","last_undegraded":"2024-03-23T11:24:12.066724+0000","last_fullsized":"2024-03-23T11:24:12.066724+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:42:46.273199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.083970+0000","last_change":"2024-03-23T11:24:11.075931+0000","last_active":"2024-03-23T11:24:11.083970+0000","last_peered":"2024-03-23T11:24:11.083970+0000","last_clean":"2024-03-23T11:24:11.083970+0000","last_became_active":"2024-03-23T11:24:11.075678+0000","last_became_peered":"2024-03-23T11:24:11.075678+0000","last_unstale":"2024-03-23T11:24:11.083970+0000","last_undegraded":"2024-03-23T11:24:11.083970+0000","last_fullsized":"2024-03-23T11:24:11.083970+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:35.558343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069929+0000","last_change":"2024-03-23T11:24:11.087973+0000","last_active":"2024-03-23T11:24:12.069929+0000","last_peered":"2024-03-23T11:24:12.069929+0000","last_clean":"2024-03-23T11:24:12.069929+0000","last_became_active":"2024-03-23T11:24:11.087702+0000","last_became_peered":"2024-03-23T11:24:11.087702+0000","last_unstale":"2024-03-23T11:24:12.069929+0000","last_undegraded":"2024-03-23T11:24:12.069929+0000","last_fullsized":"2024-03-23T11:24:12.069929+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:55:45.084475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194032+0000","last_change":"2024-03-23T11:24:11.072450+0000","last_active":"2024-03-23T11:24:13.194032+0000","last_peered":"2024-03-23T11:24:13.194032+0000","last_clean":"2024-03-23T11:24:13.194032+0000","last_became_active":"2024-03-23T11:24:11.071363+0000","last_became_peered":"2024-03-23T11:24:11.071363+0000","last_unstale":"2024-03-23T11:24:13.194032+0000","last_undegraded":"2024-03-23T11:24:13.194032+0000","last_fullsized":"2024-03-23T11:24:13.194032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:35:26.679939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070168+0000","last_change":"2024-03-23T11:24:11.080137+0000","last_active":"2024-03-23T11:24:12.070168+0000","last_peered":"2024-03-23T11:24:12.070168+0000","last_clean":"2024-03-23T11:24:12.070168+0000","last_became_active":"2024-03-23T11:24:11.078526+0000","last_became_peered":"2024-03-23T11:24:11.078526+0000","last_unstale":"2024-03-23T11:24:12.070168+0000","last_undegraded":"2024-03-23T11:24:12.070168+0000","last_fullsized":"2024-03-23T11:24:12.070168+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:11:59.579321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.082588+0000","last_change":"2024-03-23T11:24:11.075934+0000","last_active":"2024-03-23T11:24:11.082588+0000","last_peered":"2024-03-23T11:24:11.082588+0000","last_clean":"2024-03-23T11:24:11.082588+0000","last_became_active":"2024-03-23T11:24:11.075617+0000","last_became_peered":"2024-03-23T11:24:11.075617+0000","last_unstale":"2024-03-23T11:24:11.082588+0000","last_undegraded":"2024-03-23T11:24:11.082588+0000","last_fullsized":"2024-03-23T11:24:11.082588+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:42:31.187510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072004+0000","last_change":"2024-03-23T11:24:11.084006+0000","last_active":"2024-03-23T11:24:13.072004+0000","last_peered":"2024-03-23T11:24:13.072004+0000","last_clean":"2024-03-23T11:24:13.072004+0000","last_became_active":"2024-03-23T11:24:11.082341+0000","last_became_peered":"2024-03-23T11:24:11.082341+0000","last_unstale":"2024-03-23T11:24:13.072004+0000","last_undegraded":"2024-03-23T11:24:13.072004+0000","last_fullsized":"2024-03-23T11:24:13.072004+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:45.962293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069194+0000","last_change":"2024-03-23T11:24:11.077569+0000","last_active":"2024-03-23T11:24:12.069194+0000","last_peered":"2024-03-23T11:24:12.069194+0000","last_clean":"2024-03-23T11:24:12.069194+0000","last_became_active":"2024-03-23T11:24:11.077238+0000","last_became_peered":"2024-03-23T11:24:11.077238+0000","last_unstale":"2024-03-23T11:24:12.069194+0000","last_undegraded":"2024-03-23T11:24:12.069194+0000","last_fullsized":"2024-03-23T11:24:12.069194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:39:26.172241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066855+0000","last_change":"2024-03-23T11:24:11.077251+0000","last_active":"2024-03-23T11:24:12.066855+0000","last_peered":"2024-03-23T11:24:12.066855+0000","last_clean":"2024-03-23T11:24:12.066855+0000","last_became_active":"2024-03-23T11:24:11.076703+0000","last_became_peered":"2024-03-23T11:24:11.076703+0000","last_unstale":"2024-03-23T11:24:12.066855+0000","last_undegraded":"2024-03-23T11:24:12.066855+0000","last_fullsized":"2024-03-23T11:24:12.066855+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:02:15.494810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.091483+0000","last_change":"2024-03-23T11:24:11.085605+0000","last_active":"2024-03-23T11:24:11.091483+0000","last_peered":"2024-03-23T11:24:11.091483+0000","last_clean":"2024-03-23T11:24:11.091483+0000","last_became_active":"2024-03-23T11:24:11.085317+0000","last_became_peered":"2024-03-23T11:24:11.085317+0000","last_unstale":"2024-03-23T11:24:11.091483+0000","last_undegraded":"2024-03-23T11:24:11.091483+0000","last_fullsized":"2024-03-23T11:24:11.091483+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:01:21.503319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.073558+0000","last_change":"2024-03-23T11:24:11.069770+0000","last_active":"2024-03-23T11:24:11.073558+0000","last_peered":"2024-03-23T11:24:11.073558+0000","last_clean":"2024-03-23T11:24:11.073558+0000","last_became_active":"2024-03-23T11:24:11.069414+0000","last_became_peered":"2024-03-23T11:24:11.069414+0000","last_unstale":"2024-03-23T11:24:11.073558+0000","last_undegraded":"2024-03-23T11:24:11.073558+0000","last_fullsized":"2024-03-23T11:24:11.073558+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:17.196321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.083600+0000","last_change":"2024-03-23T11:24:11.078141+0000","last_active":"2024-03-23T11:24:11.083600+0000","last_peered":"2024-03-23T11:24:11.083600+0000","last_clean":"2024-03-23T11:24:11.083600+0000","last_became_active":"2024-03-23T11:24:11.077876+0000","last_became_peered":"2024-03-23T11:24:11.077876+0000","last_unstale":"2024-03-23T11:24:11.083600+0000","last_undegraded":"2024-03-23T11:24:11.083600+0000","last_fullsized":"2024-03-23T11:24:11.083600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:46:58.997122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.071506+0000","last_change":"2024-03-23T11:24:11.087752+0000","last_active":"2024-03-23T11:24:13.071506+0000","last_peered":"2024-03-23T11:24:13.071506+0000","last_clean":"2024-03-23T11:24:13.071506+0000","last_became_active":"2024-03-23T11:24:11.087451+0000","last_became_peered":"2024-03-23T11:24:11.087451+0000","last_unstale":"2024-03-23T11:24:13.071506+0000","last_undegraded":"2024-03-23T11:24:13.071506+0000","last_fullsized":"2024-03-23T11:24:13.071506+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:09:42.953821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.074145+0000","last_change":"2024-03-23T11:24:11.070297+0000","last_active":"2024-03-23T11:24:11.074145+0000","last_peered":"2024-03-23T11:24:11.074145+0000","last_clean":"2024-03-23T11:24:11.074145+0000","last_became_active":"2024-03-23T11:24:11.070047+0000","last_became_peered":"2024-03-23T11:24:11.070047+0000","last_unstale":"2024-03-23T11:24:11.074145+0000","last_undegraded":"2024-03-23T11:24:11.074145+0000","last_fullsized":"2024-03-23T11:24:11.074145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:58:45.931096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066380+0000","last_change":"2024-03-23T11:24:11.088685+0000","last_active":"2024-03-23T11:24:12.066380+0000","last_peered":"2024-03-23T11:24:12.066380+0000","last_clean":"2024-03-23T11:24:12.066380+0000","last_became_active":"2024-03-23T11:24:11.088406+0000","last_became_peered":"2024-03-23T11:24:11.088406+0000","last_unstale":"2024-03-23T11:24:12.066380+0000","last_undegraded":"2024-03-23T11:24:12.066380+0000","last_fullsized":"2024-03-23T11:24:12.066380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:37.925686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070484+0000","last_change":"2024-03-23T11:24:11.084614+0000","last_active":"2024-03-23T11:24:12.070484+0000","last_peered":"2024-03-23T11:24:12.070484+0000","last_clean":"2024-03-23T11:24:12.070484+0000","last_became_active":"2024-03-23T11:24:11.084336+0000","last_became_peered":"2024-03-23T11:24:11.084336+0000","last_unstale":"2024-03-23T11:24:12.070484+0000","last_undegraded":"2024-03-23T11:24:12.070484+0000","last_fullsized":"2024-03-23T11:24:12.070484+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:46:58.997122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.083225+0000","last_change":"2024-03-23T11:24:11.075275+0000","last_active":"2024-03-23T11:24:11.083225+0000","last_peered":"2024-03-23T11:24:11.083225+0000","last_clean":"2024-03-23T11:24:11.083225+0000","last_became_active":"2024-03-23T11:24:11.075032+0000","last_became_peered":"2024-03-23T11:24:11.075032+0000","last_unstale":"2024-03-23T11:24:11.083225+0000","last_undegraded":"2024-03-23T11:24:11.083225+0000","last_fullsized":"2024-03-23T11:24:11.083225+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:42.150300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072013+0000","last_change":"2024-03-23T11:24:11.084697+0000","last_active":"2024-03-23T11:24:13.072013+0000","last_peered":"2024-03-23T11:24:13.072013+0000","last_clean":"2024-03-23T11:24:13.072013+0000","last_became_active":"2024-03-23T11:24:11.083709+0000","last_became_peered":"2024-03-23T11:24:11.083709+0000","last_unstale":"2024-03-23T11:24:13.072013+0000","last_undegraded":"2024-03-23T11:24:13.072013+0000","last_fullsized":"2024-03-23T11:24:13.072013+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:25:35.284033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.071657+0000","last_change":"2024-03-23T11:24:11.071917+0000","last_active":"2024-03-23T11:24:13.071657+0000","last_peered":"2024-03-23T11:24:13.071657+0000","last_clean":"2024-03-23T11:24:13.071657+0000","last_became_active":"2024-03-23T11:24:11.068485+0000","last_became_peered":"2024-03-23T11:24:11.068485+0000","last_unstale":"2024-03-23T11:24:13.071657+0000","last_undegraded":"2024-03-23T11:24:13.071657+0000","last_fullsized":"2024-03-23T11:24:13.071657+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:51:23.153567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.068567+0000","last_change":"2024-03-23T11:24:11.070113+0000","last_active":"2024-03-23T11:24:12.068567+0000","last_peered":"2024-03-23T11:24:12.068567+0000","last_clean":"2024-03-23T11:24:12.068567+0000","last_became_active":"2024-03-23T11:24:11.069833+0000","last_became_peered":"2024-03-23T11:24:11.069833+0000","last_unstale":"2024-03-23T11:24:12.068567+0000","last_undegraded":"2024-03-23T11:24:12.068567+0000","last_fullsized":"2024-03-23T11:24:12.068567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:37:01.697458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.068080+0000","last_change":"2024-03-23T11:24:11.087980+0000","last_active":"2024-03-23T11:24:12.068080+0000","last_peered":"2024-03-23T11:24:12.068080+0000","last_clean":"2024-03-23T11:24:12.068080+0000","last_became_active":"2024-03-23T11:24:11.087729+0000","last_became_peered":"2024-03-23T11:24:11.087729+0000","last_unstale":"2024-03-23T11:24:12.068080+0000","last_undegraded":"2024-03-23T11:24:12.068080+0000","last_fullsized":"2024-03-23T11:24:12.068080+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:14:34.648258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072248+0000","last_change":"2024-03-23T11:24:11.077763+0000","last_active":"2024-03-23T11:24:13.072248+0000","last_peered":"2024-03-23T11:24:13.072248+0000","last_clean":"2024-03-23T11:24:13.072248+0000","last_became_active":"2024-03-23T11:24:11.076383+0000","last_became_peered":"2024-03-23T11:24:11.076383+0000","last_unstale":"2024-03-23T11:24:13.072248+0000","last_undegraded":"2024-03-23T11:24:13.072248+0000","last_fullsized":"2024-03-23T11:24:13.072248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:11:59.579321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194661+0000","last_change":"2024-03-23T11:24:11.090977+0000","last_active":"2024-03-23T11:24:13.194661+0000","last_peered":"2024-03-23T11:24:13.194661+0000","last_clean":"2024-03-23T11:24:13.194661+0000","last_became_active":"2024-03-23T11:24:11.090679+0000","last_became_peered":"2024-03-23T11:24:11.090679+0000","last_unstale":"2024-03-23T11:24:13.194661+0000","last_undegraded":"2024-03-23T11:24:13.194661+0000","last_fullsized":"2024-03-23T11:24:13.194661+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:18:04.159007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262156+0000","last_change":"2024-03-23T11:24:11.075234+0000","last_active":"2024-03-23T11:24:13.262156+0000","last_peered":"2024-03-23T11:24:13.262156+0000","last_clean":"2024-03-23T11:24:13.262156+0000","last_became_active":"2024-03-23T11:24:11.074971+0000","last_became_peered":"2024-03-23T11:24:11.074971+0000","last_unstale":"2024-03-23T11:24:13.262156+0000","last_undegraded":"2024-03-23T11:24:13.262156+0000","last_fullsized":"2024-03-23T11:24:13.262156+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:59:57.629892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072255+0000","last_change":"2024-03-23T11:24:11.077975+0000","last_active":"2024-03-23T11:24:13.072255+0000","last_peered":"2024-03-23T11:24:13.072255+0000","last_clean":"2024-03-23T11:24:13.072255+0000","last_became_active":"2024-03-23T11:24:11.077700+0000","last_became_peered":"2024-03-23T11:24:11.077700+0000","last_unstale":"2024-03-23T11:24:13.072255+0000","last_undegraded":"2024-03-23T11:24:13.072255+0000","last_fullsized":"2024-03-23T11:24:13.072255+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:59:01.118041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.261835+0000","last_change":"2024-03-23T11:24:11.069184+0000","last_active":"2024-03-23T11:24:13.261835+0000","last_peered":"2024-03-23T11:24:13.261835+0000","last_clean":"2024-03-23T11:24:13.261835+0000","last_became_active":"2024-03-23T11:24:11.068839+0000","last_became_peered":"2024-03-23T11:24:11.068839+0000","last_unstale":"2024-03-23T11:24:13.261835+0000","last_undegraded":"2024-03-23T11:24:13.261835+0000","last_fullsized":"2024-03-23T11:24:13.261835+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:55:00.318209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194044+0000","last_change":"2024-03-23T11:24:11.088591+0000","last_active":"2024-03-23T11:24:13.194044+0000","last_peered":"2024-03-23T11:24:13.194044+0000","last_clean":"2024-03-23T11:24:13.194044+0000","last_became_active":"2024-03-23T11:24:11.086063+0000","last_became_peered":"2024-03-23T11:24:11.086063+0000","last_unstale":"2024-03-23T11:24:13.194044+0000","last_undegraded":"2024-03-23T11:24:13.194044+0000","last_fullsized":"2024-03-23T11:24:13.194044+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:55:54.448139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194496+0000","last_change":"2024-03-23T11:24:11.071424+0000","last_active":"2024-03-23T11:24:13.194496+0000","last_peered":"2024-03-23T11:24:13.194496+0000","last_clean":"2024-03-23T11:24:13.194496+0000","last_became_active":"2024-03-23T11:24:11.071105+0000","last_became_peered":"2024-03-23T11:24:11.071105+0000","last_unstale":"2024-03-23T11:24:13.194496+0000","last_undegraded":"2024-03-23T11:24:13.194496+0000","last_fullsized":"2024-03-23T11:24:13.194496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:29:46.001149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070466+0000","last_change":"2024-03-23T11:24:11.084226+0000","last_active":"2024-03-23T11:24:12.070466+0000","last_peered":"2024-03-23T11:24:12.070466+0000","last_clean":"2024-03-23T11:24:12.070466+0000","last_became_active":"2024-03-23T11:24:11.082119+0000","last_became_peered":"2024-03-23T11:24:11.082119+0000","last_unstale":"2024-03-23T11:24:12.070466+0000","last_undegraded":"2024-03-23T11:24:12.070466+0000","last_fullsized":"2024-03-23T11:24:12.070466+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:18:53.314479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069301+0000","last_change":"2024-03-23T11:24:11.092837+0000","last_active":"2024-03-23T11:24:12.069301+0000","last_peered":"2024-03-23T11:24:12.069301+0000","last_clean":"2024-03-23T11:24:12.069301+0000","last_became_active":"2024-03-23T11:24:11.092582+0000","last_became_peered":"2024-03-23T11:24:11.092582+0000","last_unstale":"2024-03-23T11:24:12.069301+0000","last_undegraded":"2024-03-23T11:24:12.069301+0000","last_fullsized":"2024-03-23T11:24:12.069301+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:49:21.313471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"73'4","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070815+0000","last_change":"2024-03-23T11:24:10.588379+0000","last_active":"2024-03-23T11:24:12.070815+0000","last_peered":"2024-03-23T11:24:12.070815+0000","last_clean":"2024-03-23T11:24:12.070815+0000","last_became_active":"2024-03-23T11:24:10.588040+0000","last_became_peered":"2024-03-23T11:24:10.588040+0000","last_unstale":"2024-03-23T11:24:12.070815+0000","last_undegraded":"2024-03-23T11:24:12.070815+0000","last_fullsized":"2024-03-23T11:24:12.070815+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:02:50.767178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.078368+0000","last_change":"2024-03-23T11:24:11.074259+0000","last_active":"2024-03-23T11:24:11.078368+0000","last_peered":"2024-03-23T11:24:11.078368+0000","last_clean":"2024-03-23T11:24:11.078368+0000","last_became_active":"2024-03-23T11:24:11.073957+0000","last_became_peered":"2024-03-23T11:24:11.073957+0000","last_unstale":"2024-03-23T11:24:11.078368+0000","last_undegraded":"2024-03-23T11:24:11.078368+0000","last_fullsized":"2024-03-23T11:24:11.078368+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:54:34.282504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"73'1","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.090876+0000","last_change":"2024-03-23T11:24:10.581333+0000","last_active":"2024-03-23T11:24:11.090876+0000","last_peered":"2024-03-23T11:24:11.090876+0000","last_clean":"2024-03-23T11:24:11.090876+0000","last_became_active":"2024-03-23T11:24:10.581019+0000","last_became_peered":"2024-03-23T11:24:10.581019+0000","last_unstale":"2024-03-23T11:24:11.090876+0000","last_undegraded":"2024-03-23T11:24:11.090876+0000","last_fullsized":"2024-03-23T11:24:11.090876+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:59:00.105792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071090+0000","last_change":"2024-03-23T11:24:11.085453+0000","last_active":"2024-03-23T11:24:12.071090+0000","last_peered":"2024-03-23T11:24:12.071090+0000","last_clean":"2024-03-23T11:24:12.071090+0000","last_became_active":"2024-03-23T11:24:11.083291+0000","last_became_peered":"2024-03-23T11:24:11.083291+0000","last_unstale":"2024-03-23T11:24:12.071090+0000","last_undegraded":"2024-03-23T11:24:12.071090+0000","last_fullsized":"2024-03-23T11:24:12.071090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:01:21.503319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072208+0000","last_change":"2024-03-23T11:24:10.582900+0000","last_active":"2024-03-23T11:24:13.072208+0000","last_peered":"2024-03-23T11:24:13.072208+0000","last_clean":"2024-03-23T11:24:13.072208+0000","last_became_active":"2024-03-23T11:24:10.582186+0000","last_became_peered":"2024-03-23T11:24:10.582186+0000","last_unstale":"2024-03-23T11:24:13.072208+0000","last_undegraded":"2024-03-23T11:24:13.072208+0000","last_fullsized":"2024-03-23T11:24:13.072208+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:46:08.860795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069481+0000","last_change":"2024-03-23T11:24:11.079508+0000","last_active":"2024-03-23T11:24:12.069481+0000","last_peered":"2024-03-23T11:24:12.069481+0000","last_clean":"2024-03-23T11:24:12.069481+0000","last_became_active":"2024-03-23T11:24:11.077841+0000","last_became_peered":"2024-03-23T11:24:11.077841+0000","last_unstale":"2024-03-23T11:24:12.069481+0000","last_undegraded":"2024-03-23T11:24:12.069481+0000","last_fullsized":"2024-03-23T11:24:12.069481+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:33:10.380948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071513+0000","last_change":"2024-03-23T11:24:10.583681+0000","last_active":"2024-03-23T11:24:12.071513+0000","last_peered":"2024-03-23T11:24:12.071513+0000","last_clean":"2024-03-23T11:24:12.071513+0000","last_became_active":"2024-03-23T11:24:10.582035+0000","last_became_peered":"2024-03-23T11:24:10.582035+0000","last_unstale":"2024-03-23T11:24:12.071513+0000","last_undegraded":"2024-03-23T11:24:12.071513+0000","last_fullsized":"2024-03-23T11:24:12.071513+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:42:30.175261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194203+0000","last_change":"2024-03-23T11:24:11.080481+0000","last_active":"2024-03-23T11:24:13.194203+0000","last_peered":"2024-03-23T11:24:13.194203+0000","last_clean":"2024-03-23T11:24:13.194203+0000","last_became_active":"2024-03-23T11:24:11.080209+0000","last_became_peered":"2024-03-23T11:24:11.080209+0000","last_unstale":"2024-03-23T11:24:13.194203+0000","last_undegraded":"2024-03-23T11:24:13.194203+0000","last_fullsized":"2024-03-23T11:24:13.194203+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:44:00.853141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"73'1","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262219+0000","last_change":"2024-03-23T11:24:10.586960+0000","last_active":"2024-03-23T11:24:13.262219+0000","last_peered":"2024-03-23T11:24:13.262219+0000","last_clean":"2024-03-23T11:24:13.262219+0000","last_became_active":"2024-03-23T11:24:10.585902+0000","last_became_peered":"2024-03-23T11:24:10.585902+0000","last_unstale":"2024-03-23T11:24:13.262219+0000","last_undegraded":"2024-03-23T11:24:13.262219+0000","last_fullsized":"2024-03-23T11:24:13.262219+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:07.177476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072002+0000","last_change":"2024-03-23T11:24:11.075018+0000","last_active":"2024-03-23T11:24:13.072002+0000","last_peered":"2024-03-23T11:24:13.072002+0000","last_clean":"2024-03-23T11:24:13.072002+0000","last_became_active":"2024-03-23T11:24:11.073200+0000","last_became_peered":"2024-03-23T11:24:11.073200+0000","last_unstale":"2024-03-23T11:24:13.072002+0000","last_undegraded":"2024-03-23T11:24:13.072002+0000","last_fullsized":"2024-03-23T11:24:13.072002+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:45:36.911552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"73'1","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070101+0000","last_change":"2024-03-23T11:24:10.576442+0000","last_active":"2024-03-23T11:24:12.070101+0000","last_peered":"2024-03-23T11:24:12.070101+0000","last_clean":"2024-03-23T11:24:12.070101+0000","last_became_active":"2024-03-23T11:24:10.576087+0000","last_became_peered":"2024-03-23T11:24:10.576087+0000","last_unstale":"2024-03-23T11:24:12.070101+0000","last_undegraded":"2024-03-23T11:24:12.070101+0000","last_fullsized":"2024-03-23T11:24:12.070101+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:42:01.153953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.091675+0000","last_change":"2024-03-23T11:24:11.087533+0000","last_active":"2024-03-23T11:24:11.091675+0000","last_peered":"2024-03-23T11:24:11.091675+0000","last_clean":"2024-03-23T11:24:11.091675+0000","last_became_active":"2024-03-23T11:24:11.087292+0000","last_became_peered":"2024-03-23T11:24:11.087292+0000","last_unstale":"2024-03-23T11:24:11.091675+0000","last_undegraded":"2024-03-23T11:24:11.091675+0000","last_fullsized":"2024-03-23T11:24:11.091675+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:06:45.108816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066299+0000","last_change":"2024-03-23T11:24:10.573973+0000","last_active":"2024-03-23T11:24:12.066299+0000","last_peered":"2024-03-23T11:24:12.066299+0000","last_clean":"2024-03-23T11:24:12.066299+0000","last_became_active":"2024-03-23T11:24:10.573666+0000","last_became_peered":"2024-03-23T11:24:10.573666+0000","last_unstale":"2024-03-23T11:24:12.066299+0000","last_undegraded":"2024-03-23T11:24:12.066299+0000","last_fullsized":"2024-03-23T11:24:12.066299+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:46:08.498563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.082621+0000","last_change":"2024-03-23T11:24:11.074392+0000","last_active":"2024-03-23T11:24:11.082621+0000","last_peered":"2024-03-23T11:24:11.082621+0000","last_clean":"2024-03-23T11:24:11.082621+0000","last_became_active":"2024-03-23T11:24:11.074120+0000","last_became_peered":"2024-03-23T11:24:11.074120+0000","last_unstale":"2024-03-23T11:24:11.082621+0000","last_undegraded":"2024-03-23T11:24:11.082621+0000","last_fullsized":"2024-03-23T11:24:11.082621+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:05.652914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.065553+0000","last_change":"2024-03-23T11:24:10.567379+0000","last_active":"2024-03-23T11:24:11.065553+0000","last_peered":"2024-03-23T11:24:11.065553+0000","last_clean":"2024-03-23T11:24:11.065553+0000","last_became_active":"2024-03-23T11:24:10.567204+0000","last_became_peered":"2024-03-23T11:24:10.567204+0000","last_unstale":"2024-03-23T11:24:11.065553+0000","last_undegraded":"2024-03-23T11:24:11.065553+0000","last_fullsized":"2024-03-23T11:24:11.065553+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:52.099376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194659+0000","last_change":"2024-03-23T11:24:11.073905+0000","last_active":"2024-03-23T11:24:13.194659+0000","last_peered":"2024-03-23T11:24:13.194659+0000","last_clean":"2024-03-23T11:24:13.194659+0000","last_became_active":"2024-03-23T11:24:11.073603+0000","last_became_peered":"2024-03-23T11:24:11.073603+0000","last_unstale":"2024-03-23T11:24:13.194659+0000","last_undegraded":"2024-03-23T11:24:13.194659+0000","last_fullsized":"2024-03-23T11:24:13.194659+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:41:41.482855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072298+0000","last_change":"2024-03-23T11:24:10.578499+0000","last_active":"2024-03-23T11:24:13.072298+0000","last_peered":"2024-03-23T11:24:13.072298+0000","last_clean":"2024-03-23T11:24:13.072298+0000","last_became_active":"2024-03-23T11:24:10.578229+0000","last_became_peered":"2024-03-23T11:24:10.578229+0000","last_unstale":"2024-03-23T11:24:13.072298+0000","last_undegraded":"2024-03-23T11:24:13.072298+0000","last_fullsized":"2024-03-23T11:24:13.072298+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:13:49.012961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070843+0000","last_change":"2024-03-23T11:24:11.087301+0000","last_active":"2024-03-23T11:24:12.070843+0000","last_peered":"2024-03-23T11:24:12.070843+0000","last_clean":"2024-03-23T11:24:12.070843+0000","last_became_active":"2024-03-23T11:24:11.087031+0000","last_became_peered":"2024-03-23T11:24:11.087031+0000","last_unstale":"2024-03-23T11:24:12.070843+0000","last_undegraded":"2024-03-23T11:24:12.070843+0000","last_fullsized":"2024-03-23T11:24:12.070843+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:26:52.158972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071415+0000","last_change":"2024-03-23T11:24:10.583641+0000","last_active":"2024-03-23T11:24:12.071415+0000","last_peered":"2024-03-23T11:24:12.071415+0000","last_clean":"2024-03-23T11:24:12.071415+0000","last_became_active":"2024-03-23T11:24:10.583296+0000","last_became_peered":"2024-03-23T11:24:10.583296+0000","last_unstale":"2024-03-23T11:24:12.071415+0000","last_undegraded":"2024-03-23T11:24:12.071415+0000","last_fullsized":"2024-03-23T11:24:12.071415+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:44:37.296461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071326+0000","last_change":"2024-03-23T11:24:11.086027+0000","last_active":"2024-03-23T11:24:12.071326+0000","last_peered":"2024-03-23T11:24:12.071326+0000","last_clean":"2024-03-23T11:24:12.071326+0000","last_became_active":"2024-03-23T11:24:11.085718+0000","last_became_peered":"2024-03-23T11:24:11.085718+0000","last_unstale":"2024-03-23T11:24:12.071326+0000","last_undegraded":"2024-03-23T11:24:12.071326+0000","last_fullsized":"2024-03-23T11:24:12.071326+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:21:46.895562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"73'3","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262286+0000","last_change":"2024-03-23T11:24:10.588288+0000","last_active":"2024-03-23T11:24:13.262286+0000","last_peered":"2024-03-23T11:24:13.262286+0000","last_clean":"2024-03-23T11:24:13.262286+0000","last_became_active":"2024-03-23T11:24:10.587901+0000","last_became_peered":"2024-03-23T11:24:10.587901+0000","last_unstale":"2024-03-23T11:24:13.262286+0000","last_undegraded":"2024-03-23T11:24:13.262286+0000","last_fullsized":"2024-03-23T11:24:13.262286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:41:40.470606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.090467+0000","last_change":"2024-03-23T11:24:11.080963+0000","last_active":"2024-03-23T11:24:11.090467+0000","last_peered":"2024-03-23T11:24:11.090467+0000","last_clean":"2024-03-23T11:24:11.090467+0000","last_became_active":"2024-03-23T11:24:11.079943+0000","last_became_peered":"2024-03-23T11:24:11.079943+0000","last_unstale":"2024-03-23T11:24:11.090467+0000","last_undegraded":"2024-03-23T11:24:11.090467+0000","last_fullsized":"2024-03-23T11:24:11.090467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:37:10.652496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"73'6","reported_seq":17,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.067100+0000","last_change":"2024-03-23T11:24:10.575152+0000","last_active":"2024-03-23T11:24:12.067100+0000","last_peered":"2024-03-23T11:24:12.067100+0000","last_clean":"2024-03-23T11:24:12.067100+0000","last_became_active":"2024-03-23T11:24:10.573893+0000","last_became_peered":"2024-03-23T11:24:10.573893+0000","last_unstale":"2024-03-23T11:24:12.067100+0000","last_undegraded":"2024-03-23T11:24:12.067100+0000","last_fullsized":"2024-03-23T11:24:12.067100+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:56:49.479990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.068467+0000","last_change":"2024-03-23T11:24:11.079877+0000","last_active":"2024-03-23T11:24:12.068467+0000","last_peered":"2024-03-23T11:24:12.068467+0000","last_clean":"2024-03-23T11:24:12.068467+0000","last_became_active":"2024-03-23T11:24:11.079567+0000","last_became_peered":"2024-03-23T11:24:11.079567+0000","last_unstale":"2024-03-23T11:24:12.068467+0000","last_undegraded":"2024-03-23T11:24:12.068467+0000","last_fullsized":"2024-03-23T11:24:12.068467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:44:38.284660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066565+0000","last_change":"2024-03-23T11:24:10.574701+0000","last_active":"2024-03-23T11:24:12.066565+0000","last_peered":"2024-03-23T11:24:12.066565+0000","last_clean":"2024-03-23T11:24:12.066565+0000","last_became_active":"2024-03-23T11:24:10.574451+0000","last_became_peered":"2024-03-23T11:24:10.574451+0000","last_unstale":"2024-03-23T11:24:12.066565+0000","last_undegraded":"2024-03-23T11:24:12.066565+0000","last_fullsized":"2024-03-23T11:24:12.066565+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:14:46.178001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194245+0000","last_change":"2024-03-23T11:24:11.079284+0000","last_active":"2024-03-23T11:24:13.194245+0000","last_peered":"2024-03-23T11:24:13.194245+0000","last_clean":"2024-03-23T11:24:13.194245+0000","last_became_active":"2024-03-23T11:24:11.078977+0000","last_became_peered":"2024-03-23T11:24:11.078977+0000","last_unstale":"2024-03-23T11:24:13.194245+0000","last_undegraded":"2024-03-23T11:24:13.194245+0000","last_fullsized":"2024-03-23T11:24:13.194245+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:26:09.227469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070235+0000","last_change":"2024-03-23T11:24:10.583159+0000","last_active":"2024-03-23T11:24:12.070235+0000","last_peered":"2024-03-23T11:24:12.070235+0000","last_clean":"2024-03-23T11:24:12.070235+0000","last_became_active":"2024-03-23T11:24:10.582665+0000","last_became_peered":"2024-03-23T11:24:10.582665+0000","last_unstale":"2024-03-23T11:24:12.070235+0000","last_undegraded":"2024-03-23T11:24:12.070235+0000","last_fullsized":"2024-03-23T11:24:12.070235+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:17:54.725364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.091362+0000","last_change":"2024-03-23T11:24:11.085719+0000","last_active":"2024-03-23T11:24:11.091362+0000","last_peered":"2024-03-23T11:24:11.091362+0000","last_clean":"2024-03-23T11:24:11.091362+0000","last_became_active":"2024-03-23T11:24:11.085484+0000","last_became_peered":"2024-03-23T11:24:11.085484+0000","last_unstale":"2024-03-23T11:24:11.091362+0000","last_undegraded":"2024-03-23T11:24:11.091362+0000","last_fullsized":"2024-03-23T11:24:11.091362+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:18:53.314479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194265+0000","last_change":"2024-03-23T11:24:10.585405+0000","last_active":"2024-03-23T11:24:13.194265+0000","last_peered":"2024-03-23T11:24:13.194265+0000","last_clean":"2024-03-23T11:24:13.194265+0000","last_became_active":"2024-03-23T11:24:10.584986+0000","last_became_peered":"2024-03-23T11:24:10.584986+0000","last_unstale":"2024-03-23T11:24:13.194265+0000","last_undegraded":"2024-03-23T11:24:13.194265+0000","last_fullsized":"2024-03-23T11:24:13.194265+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:21:45.883312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.090534+0000","last_change":"2024-03-23T11:24:11.081623+0000","last_active":"2024-03-23T11:24:11.090534+0000","last_peered":"2024-03-23T11:24:11.090534+0000","last_clean":"2024-03-23T11:24:11.090534+0000","last_became_active":"2024-03-23T11:24:11.078518+0000","last_became_peered":"2024-03-23T11:24:11.078518+0000","last_unstale":"2024-03-23T11:24:11.090534+0000","last_undegraded":"2024-03-23T11:24:11.090534+0000","last_fullsized":"2024-03-23T11:24:11.090534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:21:16.959427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"73'8","reported_seq":19,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.071767+0000","last_change":"2024-03-23T11:24:10.583814+0000","last_active":"2024-03-23T11:24:13.071767+0000","last_peered":"2024-03-23T11:24:13.071767+0000","last_clean":"2024-03-23T11:24:13.071767+0000","last_became_active":"2024-03-23T11:24:10.581693+0000","last_became_peered":"2024-03-23T11:24:10.581693+0000","last_unstale":"2024-03-23T11:24:13.071767+0000","last_undegraded":"2024-03-23T11:24:13.071767+0000","last_fullsized":"2024-03-23T11:24:13.071767+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:16.184072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072280+0000","last_change":"2024-03-23T11:24:11.080112+0000","last_active":"2024-03-23T11:24:13.072280+0000","last_peered":"2024-03-23T11:24:13.072280+0000","last_clean":"2024-03-23T11:24:13.072280+0000","last_became_active":"2024-03-23T11:24:11.077841+0000","last_became_peered":"2024-03-23T11:24:11.077841+0000","last_unstale":"2024-03-23T11:24:13.072280+0000","last_undegraded":"2024-03-23T11:24:13.072280+0000","last_fullsized":"2024-03-23T11:24:13.072280+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:39:35.570836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"73'1","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071396+0000","last_change":"2024-03-23T11:24:10.578691+0000","last_active":"2024-03-23T11:24:12.071396+0000","last_peered":"2024-03-23T11:24:12.071396+0000","last_clean":"2024-03-23T11:24:12.071396+0000","last_became_active":"2024-03-23T11:24:10.578420+0000","last_became_peered":"2024-03-23T11:24:10.578420+0000","last_unstale":"2024-03-23T11:24:12.071396+0000","last_undegraded":"2024-03-23T11:24:12.071396+0000","last_fullsized":"2024-03-23T11:24:12.071396+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:49:47.385865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070720+0000","last_change":"2024-03-23T11:24:11.081542+0000","last_active":"2024-03-23T11:24:12.070720+0000","last_peered":"2024-03-23T11:24:12.070720+0000","last_clean":"2024-03-23T11:24:12.070720+0000","last_became_active":"2024-03-23T11:24:11.081219+0000","last_became_peered":"2024-03-23T11:24:11.081219+0000","last_unstale":"2024-03-23T11:24:12.070720+0000","last_undegraded":"2024-03-23T11:24:12.070720+0000","last_fullsized":"2024-03-23T11:24:12.070720+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:30:15.190912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.072346+0000","last_change":"2024-03-23T11:24:10.584541+0000","last_active":"2024-03-23T11:24:13.072346+0000","last_peered":"2024-03-23T11:24:13.072346+0000","last_clean":"2024-03-23T11:24:13.072346+0000","last_became_active":"2024-03-23T11:24:10.582141+0000","last_became_peered":"2024-03-23T11:24:10.582141+0000","last_unstale":"2024-03-23T11:24:13.072346+0000","last_undegraded":"2024-03-23T11:24:13.072346+0000","last_fullsized":"2024-03-23T11:24:13.072346+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:22:13.102795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.067006+0000","last_change":"2024-03-23T11:24:11.068894+0000","last_active":"2024-03-23T11:24:12.067006+0000","last_peered":"2024-03-23T11:24:12.067006+0000","last_clean":"2024-03-23T11:24:12.067006+0000","last_became_active":"2024-03-23T11:24:11.067737+0000","last_became_peered":"2024-03-23T11:24:11.067737+0000","last_unstale":"2024-03-23T11:24:12.067006+0000","last_undegraded":"2024-03-23T11:24:12.067006+0000","last_fullsized":"2024-03-23T11:24:12.067006+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:54:06.904127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"73'1","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071142+0000","last_change":"2024-03-23T11:24:10.578459+0000","last_active":"2024-03-23T11:24:12.071142+0000","last_peered":"2024-03-23T11:24:12.071142+0000","last_clean":"2024-03-23T11:24:12.071142+0000","last_became_active":"2024-03-23T11:24:10.578148+0000","last_became_peered":"2024-03-23T11:24:10.578148+0000","last_unstale":"2024-03-23T11:24:12.071142+0000","last_undegraded":"2024-03-23T11:24:12.071142+0000","last_fullsized":"2024-03-23T11:24:12.071142+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:46:08.860795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.091557+0000","last_change":"2024-03-23T11:24:11.084557+0000","last_active":"2024-03-23T11:24:11.091557+0000","last_peered":"2024-03-23T11:24:11.091557+0000","last_clean":"2024-03-23T11:24:11.091557+0000","last_became_active":"2024-03-23T11:24:11.084272+0000","last_became_peered":"2024-03-23T11:24:11.084272+0000","last_unstale":"2024-03-23T11:24:11.091557+0000","last_undegraded":"2024-03-23T11:24:11.091557+0000","last_fullsized":"2024-03-23T11:24:11.091557+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:46.959333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069164+0000","last_change":"2024-03-23T11:24:10.589923+0000","last_active":"2024-03-23T11:24:12.069164+0000","last_peered":"2024-03-23T11:24:12.069164+0000","last_clean":"2024-03-23T11:24:12.069164+0000","last_became_active":"2024-03-23T11:24:10.589535+0000","last_became_peered":"2024-03-23T11:24:10.589535+0000","last_unstale":"2024-03-23T11:24:12.069164+0000","last_undegraded":"2024-03-23T11:24:12.069164+0000","last_fullsized":"2024-03-23T11:24:12.069164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:39:01.813788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262505+0000","last_change":"2024-03-23T11:24:11.070257+0000","last_active":"2024-03-23T11:24:13.262505+0000","last_peered":"2024-03-23T11:24:13.262505+0000","last_clean":"2024-03-23T11:24:13.262505+0000","last_became_active":"2024-03-23T11:24:11.069988+0000","last_became_peered":"2024-03-23T11:24:11.069988+0000","last_unstale":"2024-03-23T11:24:13.262505+0000","last_undegraded":"2024-03-23T11:24:13.262505+0000","last_fullsized":"2024-03-23T11:24:13.262505+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:50:54.488598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070267+0000","last_change":"2024-03-23T11:24:10.576941+0000","last_active":"2024-03-23T11:24:12.070267+0000","last_peered":"2024-03-23T11:24:12.070267+0000","last_clean":"2024-03-23T11:24:12.070267+0000","last_became_active":"2024-03-23T11:24:10.576679+0000","last_became_peered":"2024-03-23T11:24:10.576679+0000","last_unstale":"2024-03-23T11:24:12.070267+0000","last_undegraded":"2024-03-23T11:24:12.070267+0000","last_fullsized":"2024-03-23T11:24:12.070267+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:45:09.482079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262502+0000","last_change":"2024-03-23T11:24:11.085777+0000","last_active":"2024-03-23T11:24:13.262502+0000","last_peered":"2024-03-23T11:24:13.262502+0000","last_clean":"2024-03-23T11:24:13.262502+0000","last_became_active":"2024-03-23T11:24:11.085510+0000","last_became_peered":"2024-03-23T11:24:11.085510+0000","last_unstale":"2024-03-23T11:24:13.262502+0000","last_undegraded":"2024-03-23T11:24:13.262502+0000","last_fullsized":"2024-03-23T11:24:13.262502+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:20:09.740677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194565+0000","last_change":"2024-03-23T11:24:10.586830+0000","last_active":"2024-03-23T11:24:13.194565+0000","last_peered":"2024-03-23T11:24:13.194565+0000","last_clean":"2024-03-23T11:24:13.194565+0000","last_became_active":"2024-03-23T11:24:10.586435+0000","last_became_peered":"2024-03-23T11:24:10.586435+0000","last_unstale":"2024-03-23T11:24:13.194565+0000","last_undegraded":"2024-03-23T11:24:13.194565+0000","last_fullsized":"2024-03-23T11:24:13.194565+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:54:33.135399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.262156+0000","last_change":"2024-03-23T11:24:11.087515+0000","last_active":"2024-03-23T11:24:13.262156+0000","last_peered":"2024-03-23T11:24:13.262156+0000","last_clean":"2024-03-23T11:24:13.262156+0000","last_became_active":"2024-03-23T11:24:11.087204+0000","last_became_peered":"2024-03-23T11:24:11.087204+0000","last_unstale":"2024-03-23T11:24:13.262156+0000","last_undegraded":"2024-03-23T11:24:13.262156+0000","last_fullsized":"2024-03-23T11:24:13.262156+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:58:19.066834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.071480+0000","last_change":"2024-03-23T11:24:10.581764+0000","last_active":"2024-03-23T11:24:11.071480+0000","last_peered":"2024-03-23T11:24:11.071480+0000","last_clean":"2024-03-23T11:24:11.071480+0000","last_became_active":"2024-03-23T11:24:10.581448+0000","last_became_peered":"2024-03-23T11:24:10.581448+0000","last_unstale":"2024-03-23T11:24:11.071480+0000","last_undegraded":"2024-03-23T11:24:11.071480+0000","last_fullsized":"2024-03-23T11:24:11.071480+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:44.950044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.088781+0000","last_change":"2024-03-23T11:24:11.081596+0000","last_active":"2024-03-23T11:24:11.088781+0000","last_peered":"2024-03-23T11:24:11.088781+0000","last_clean":"2024-03-23T11:24:11.088781+0000","last_became_active":"2024-03-23T11:24:11.081321+0000","last_became_peered":"2024-03-23T11:24:11.081321+0000","last_unstale":"2024-03-23T11:24:11.088781+0000","last_undegraded":"2024-03-23T11:24:11.088781+0000","last_fullsized":"2024-03-23T11:24:11.088781+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:28.705045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.071493+0000","last_change":"2024-03-23T11:24:10.579383+0000","last_active":"2024-03-23T11:24:12.071493+0000","last_peered":"2024-03-23T11:24:12.071493+0000","last_clean":"2024-03-23T11:24:12.071493+0000","last_became_active":"2024-03-23T11:24:10.579077+0000","last_became_peered":"2024-03-23T11:24:10.579077+0000","last_unstale":"2024-03-23T11:24:12.071493+0000","last_undegraded":"2024-03-23T11:24:12.071493+0000","last_fullsized":"2024-03-23T11:24:12.071493+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:22:13.102795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194200+0000","last_change":"2024-03-23T11:24:11.070880+0000","last_active":"2024-03-23T11:24:13.194200+0000","last_peered":"2024-03-23T11:24:13.194200+0000","last_clean":"2024-03-23T11:24:13.194200+0000","last_became_active":"2024-03-23T11:24:11.070414+0000","last_became_peered":"2024-03-23T11:24:11.070414+0000","last_unstale":"2024-03-23T11:24:13.194200+0000","last_undegraded":"2024-03-23T11:24:13.194200+0000","last_fullsized":"2024-03-23T11:24:13.194200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:04:08.143544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.194913+0000","last_change":"2024-03-23T11:24:10.581413+0000","last_active":"2024-03-23T11:24:13.194913+0000","last_peered":"2024-03-23T11:24:13.194913+0000","last_clean":"2024-03-23T11:24:13.194913+0000","last_became_active":"2024-03-23T11:24:10.581077+0000","last_became_peered":"2024-03-23T11:24:10.581077+0000","last_unstale":"2024-03-23T11:24:13.194913+0000","last_undegraded":"2024-03-23T11:24:13.194913+0000","last_fullsized":"2024-03-23T11:24:13.194913+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:42:43.960260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069019+0000","last_change":"2024-03-23T11:24:11.092747+0000","last_active":"2024-03-23T11:24:12.069019+0000","last_peered":"2024-03-23T11:24:12.069019+0000","last_clean":"2024-03-23T11:24:12.069019+0000","last_became_active":"2024-03-23T11:24:11.092456+0000","last_became_peered":"2024-03-23T11:24:11.092456+0000","last_unstale":"2024-03-23T11:24:12.069019+0000","last_undegraded":"2024-03-23T11:24:12.069019+0000","last_fullsized":"2024-03-23T11:24:12.069019+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:32.792323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.065772+0000","last_change":"2024-03-23T11:24:10.579459+0000","last_active":"2024-03-23T11:24:11.065772+0000","last_peered":"2024-03-23T11:24:11.065772+0000","last_clean":"2024-03-23T11:24:11.065772+0000","last_became_active":"2024-03-23T11:24:10.578558+0000","last_became_peered":"2024-03-23T11:24:10.578558+0000","last_unstale":"2024-03-23T11:24:11.065772+0000","last_undegraded":"2024-03-23T11:24:11.065772+0000","last_fullsized":"2024-03-23T11:24:11.065772+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:21.208414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.069799+0000","last_change":"2024-03-23T11:24:11.072559+0000","last_active":"2024-03-23T11:24:12.069799+0000","last_peered":"2024-03-23T11:24:12.069799+0000","last_clean":"2024-03-23T11:24:12.069799+0000","last_became_active":"2024-03-23T11:24:11.072292+0000","last_became_peered":"2024-03-23T11:24:11.072292+0000","last_unstale":"2024-03-23T11:24:12.069799+0000","last_undegraded":"2024-03-23T11:24:12.069799+0000","last_fullsized":"2024-03-23T11:24:12.069799+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:59:27.736931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.064776+0000","last_change":"2024-03-23T11:24:10.578417+0000","last_active":"2024-03-23T11:24:11.064776+0000","last_peered":"2024-03-23T11:24:11.064776+0000","last_clean":"2024-03-23T11:24:11.064776+0000","last_became_active":"2024-03-23T11:24:10.577492+0000","last_became_peered":"2024-03-23T11:24:10.577492+0000","last_unstale":"2024-03-23T11:24:11.064776+0000","last_undegraded":"2024-03-23T11:24:11.064776+0000","last_fullsized":"2024-03-23T11:24:11.064776+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:25:34.271784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.090519+0000","last_change":"2024-03-23T11:24:11.081149+0000","last_active":"2024-03-23T11:24:11.090519+0000","last_peered":"2024-03-23T11:24:11.090519+0000","last_clean":"2024-03-23T11:24:11.090519+0000","last_became_active":"2024-03-23T11:24:11.080908+0000","last_became_peered":"2024-03-23T11:24:11.080908+0000","last_unstale":"2024-03-23T11:24:11.090519+0000","last_undegraded":"2024-03-23T11:24:11.090519+0000","last_fullsized":"2024-03-23T11:24:11.090519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:50:10.372150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.065693+0000","last_change":"2024-03-23T11:24:10.583872+0000","last_active":"2024-03-23T11:24:11.065693+0000","last_peered":"2024-03-23T11:24:11.065693+0000","last_clean":"2024-03-23T11:24:11.065693+0000","last_became_active":"2024-03-23T11:24:10.582786+0000","last_became_peered":"2024-03-23T11:24:10.582786+0000","last_unstale":"2024-03-23T11:24:11.065693+0000","last_undegraded":"2024-03-23T11:24:11.065693+0000","last_fullsized":"2024-03-23T11:24:11.065693+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:10.886106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070126+0000","last_change":"2024-03-23T11:24:11.084432+0000","last_active":"2024-03-23T11:24:12.070126+0000","last_peered":"2024-03-23T11:24:12.070126+0000","last_clean":"2024-03-23T11:24:12.070126+0000","last_became_active":"2024-03-23T11:24:11.084137+0000","last_became_peered":"2024-03-23T11:24:11.084137+0000","last_unstale":"2024-03-23T11:24:12.070126+0000","last_undegraded":"2024-03-23T11:24:12.070126+0000","last_fullsized":"2024-03-23T11:24:12.070126+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:54:34.147648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.065982+0000","last_change":"2024-03-23T11:24:10.568748+0000","last_active":"2024-03-23T11:24:12.065982+0000","last_peered":"2024-03-23T11:24:12.065982+0000","last_clean":"2024-03-23T11:24:12.065982+0000","last_became_active":"2024-03-23T11:24:10.568383+0000","last_became_peered":"2024-03-23T11:24:10.568383+0000","last_unstale":"2024-03-23T11:24:12.065982+0000","last_undegraded":"2024-03-23T11:24:12.065982+0000","last_fullsized":"2024-03-23T11:24:12.065982+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:44:31.486382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.066356+0000","last_change":"2024-03-23T11:24:11.067303+0000","last_active":"2024-03-23T11:24:12.066356+0000","last_peered":"2024-03-23T11:24:12.066356+0000","last_clean":"2024-03-23T11:24:12.066356+0000","last_became_active":"2024-03-23T11:24:11.066956+0000","last_became_peered":"2024-03-23T11:24:11.066956+0000","last_unstale":"2024-03-23T11:24:12.066356+0000","last_undegraded":"2024-03-23T11:24:12.066356+0000","last_fullsized":"2024-03-23T11:24:12.066356+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:07:28.883218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.071474+0000","last_change":"2024-03-23T11:24:10.584454+0000","last_active":"2024-03-23T11:24:13.071474+0000","last_peered":"2024-03-23T11:24:13.071474+0000","last_clean":"2024-03-23T11:24:13.071474+0000","last_became_active":"2024-03-23T11:24:10.582586+0000","last_became_peered":"2024-03-23T11:24:10.582586+0000","last_unstale":"2024-03-23T11:24:13.071474+0000","last_undegraded":"2024-03-23T11:24:13.071474+0000","last_fullsized":"2024-03-23T11:24:13.071474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:42:30.175261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070158+0000","last_change":"2024-03-23T11:24:11.083928+0000","last_active":"2024-03-23T11:24:12.070158+0000","last_peered":"2024-03-23T11:24:12.070158+0000","last_clean":"2024-03-23T11:24:12.070158+0000","last_became_active":"2024-03-23T11:24:11.083679+0000","last_became_peered":"2024-03-23T11:24:11.083679+0000","last_unstale":"2024-03-23T11:24:12.070158+0000","last_undegraded":"2024-03-23T11:24:12.070158+0000","last_fullsized":"2024-03-23T11:24:12.070158+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:42:44.972509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070490+0000","last_change":"2024-03-23T11:24:10.575445+0000","last_active":"2024-03-23T11:24:12.070490+0000","last_peered":"2024-03-23T11:24:12.070490+0000","last_clean":"2024-03-23T11:24:12.070490+0000","last_became_active":"2024-03-23T11:24:10.575115+0000","last_became_peered":"2024-03-23T11:24:10.575115+0000","last_unstale":"2024-03-23T11:24:12.070490+0000","last_undegraded":"2024-03-23T11:24:12.070490+0000","last_fullsized":"2024-03-23T11:24:12.070490+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:13:49.012961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:13.261879+0000","last_change":"2024-03-23T11:24:11.086227+0000","last_active":"2024-03-23T11:24:13.261879+0000","last_peered":"2024-03-23T11:24:13.261879+0000","last_clean":"2024-03-23T11:24:13.261879+0000","last_became_active":"2024-03-23T11:24:11.085990+0000","last_became_peered":"2024-03-23T11:24:11.085990+0000","last_unstale":"2024-03-23T11:24:13.261879+0000","last_undegraded":"2024-03-23T11:24:13.261879+0000","last_fullsized":"2024-03-23T11:24:13.261879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:10.540509+0000","last_clean_scrub_stamp":"2024-03-23T11:24:10.540509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:04:44.710371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"73'2","reported_seq":13,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-23T11:24:12.070286+0000","last_change":"2024-03-23T11:24:10.577230+0000","last_active":"2024-03-23T11:24:12.070286+0000","last_peered":"2024-03-23T11:24:12.070286+0000","last_clean":"2024-03-23T11:24:12.070286+0000","last_became_active":"2024-03-23T11:24:10.576144+0000","last_became_peered":"2024-03-23T11:24:10.576144+0000","last_unstale":"2024-03-23T11:24:12.070286+0000","last_undegraded":"2024-03-23T11:24:12.070286+0000","last_fullsized":"2024-03-23T11:24:12.070286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:24:09.528260+0000","last_clean_scrub_stamp":"2024-03-23T11:24:09.528260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:50:09.359901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":218,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-23T11:24:11.058541+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:24:11.058541+0000","last_peered":"2024-03-23T11:24:11.058541+0000","last_clean":"2024-03-23T11:24:11.058541+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:24:11.058541+0000","last_undegraded":"2024-03-23T11:24:11.058541+0000","last_fullsized":"2024-03-23T11:24:11.058541+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542212,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1499136,"data_stored":968442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.999,"15min":0.92500000000000004},"min":{"1min":0.68700000000000006,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.3580000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.244},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0229999999999999,"15min":0.98799999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":1.155}]},{"osd":1,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.155,"15min":1.171},"min":{"1min":0.73999999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.617,"5min":2.3820000000000001,"15min":2.3820000000000001},"last":1.331},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2330000000000001,"15min":1.246},"min":{"1min":0.90400000000000003,"5min":0.753,"15min":0.753},"max":{"1min":1.5609999999999999,"5min":2.4350000000000001,"15min":2.4350000000000001},"last":1.2969999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0409999999999999,"15min":1.0469999999999999},"min":{"1min":0.72299999999999998,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.3460000000000001,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1779999999999999,"15min":1.1399999999999999},"min":{"1min":0.84599999999999997,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.548,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.091}]},{"osd":3,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.1859999999999999,"15min":1.1379999999999999},"min":{"1min":0.91800000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.6359999999999999,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2350000000000001,"15min":1.2490000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.4490000000000001,"5min":1.778,"15min":1.778},"last":1.381}]},{"osd":4,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2330000000000001,"15min":1.2210000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.591,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.488},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.399,"15min":1.363},"min":{"1min":0.65100000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.76,"5min":1.845,"15min":1.845},"last":1.2070000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.321,"15min":1.331},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.677,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.397,"15min":1.3600000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.7969999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.5860000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4119999999999999,"15min":1.4179999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.73,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.395},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.357,"15min":1.337},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.056}]}]},{"osd":6,"up_from":14,"seq":60129542212,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":901120,"data_stored":376881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1439999999999999,"15min":1.0860000000000001},"min":{"1min":0.63600000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.5660000000000001,"5min":3.9140000000000001,"15min":3.9140000000000001},"last":1.399},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.1399999999999999,"15min":1.1120000000000001},"min":{"1min":0.70199999999999996,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6639999999999999,"5min":6.6539999999999999,"15min":6.6539999999999999},"last":1.1499999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2170000000000001,"15min":1.2230000000000001},"min":{"1min":0.89600000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.7410000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.224,"5min":1.2130000000000001,"15min":1.26},"min":{"1min":0.90100000000000002,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.5660000000000001,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.0469999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2470000000000001,"15min":1.25},"min":{"1min":0.92200000000000004,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.4810000000000001,"5min":3.73,"15min":3.73},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.27,"15min":1.2949999999999999},"min":{"1min":0.94799999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.8340000000000001,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.0980000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.3440000000000001,"15min":1.2649999999999999},"min":{"1min":0.83099999999999996,"5min":0.749,"15min":0.749},"max":{"1min":1.5149999999999999,"5min":12.914,"15min":12.914},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3169999999999999,"15min":1.369},"min":{"1min":1.1539999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.458,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":1.2549999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3009999999999999,"15min":1.391},"min":{"1min":0.91500000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.665,"5min":2.702,"15min":2.702},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.365,"15min":1.415},"min":{"1min":0.86099999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.712,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":1.5029999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.458,"15min":1.5489999999999999},"min":{"1min":1.155,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.6899999999999999,"5min":3.097,"15min":3.097},"last":1.6299999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4419999999999999,"15min":1.502},"min":{"1min":1.1850000000000001,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.907,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.6839999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.5980000000000001,"15min":1.6819999999999999},"min":{"1min":1.127,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.7450000000000001,"5min":4.9470000000000001,"15min":4.9470000000000001},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.4370000000000001,"15min":1.5069999999999999},"min":{"1min":1.0169999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.601,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":1.3540000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542212,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":888832,"data_stored":371248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0860000000000001,"15min":1.1879999999999999},"min":{"1min":0.67400000000000004,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.623,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.85899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.643,"5min":2.355,"15min":2.355},"last":0.81299999999999994}]},{"osd":1,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1379999999999999,"15min":1.1839999999999999},"min":{"1min":0.75600000000000001,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.5209999999999999,"5min":2.2559999999999998,"15min":2.2559999999999998},"last":0.997},{"interface":"front","average":{"1min":1.169,"5min":1.1739999999999999,"15min":1.2210000000000001},"min":{"1min":0.72299999999999998,"5min":0.495,"15min":0.495},"max":{"1min":1.5760000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.4410000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.238,"15min":1.2609999999999999},"min":{"1min":0.71399999999999997,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6040000000000001,"5min":2.012,"15min":2.012},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.216,"15min":1.2370000000000001},"min":{"1min":0.754,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.7090000000000001,"5min":1.946,"15min":1.946},"last":0.877}]},{"osd":3,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.304,"15min":1.3140000000000001},"min":{"1min":0.76600000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.548,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.3680000000000001,"15min":1.413},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.7709999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.337}]},{"osd":4,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.3220000000000001,"15min":1.4139999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.681,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.6040000000000001},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4830000000000001,"15min":1.5449999999999999},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.748,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.5640000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.381,"15min":1.4790000000000001},"min":{"1min":0.59199999999999997,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6259999999999999,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.4059999999999999,"15min":1.53},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.754,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.835}]},{"osd":7,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.454,"15min":1.5580000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7390000000000001,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.548,"15min":1.621},"min":{"1min":0.69099999999999995,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.819,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":1.5009999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542212,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":888832,"data_stored":371228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.131,"15min":1.1060000000000001},"min":{"1min":0.84799999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.49,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.244,"15min":1.214},"min":{"1min":1.01,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.679,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.288}]},{"osd":1,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.161,"15min":1.1200000000000001},"min":{"1min":0.91300000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6879999999999999,"5min":4.327,"15min":4.327},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.151,"15min":1.111},"min":{"1min":0.81699999999999995,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.468,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.2190000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2030000000000001,"15min":1.226},"min":{"1min":0.94299999999999995,"5min":0.624,"15min":0.624},"max":{"1min":1.6830000000000001,"5min":2.6080000000000001,"15min":2.6080000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.292,"15min":1.236},"min":{"1min":0.81799999999999995,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.6679999999999999,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":1.246}]},{"osd":3,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.222,"15min":1.2190000000000001},"min":{"1min":0.89800000000000002,"5min":0.622,"15min":0.622},"max":{"1min":1.605,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":1.135},{"interface":"front","average":{"1min":1.357,"5min":1.3240000000000001,"15min":1.264},"min":{"1min":0.92900000000000005,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.742,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":1.4239999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.238,"15min":1.284},"min":{"1min":0.80200000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.5740000000000001,"5min":2.508,"15min":2.508},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.381,"15min":1.3640000000000001},"min":{"1min":0.83299999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.0249999999999999,"5min":5.859,"15min":5.859},"last":1.3440000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.403,"15min":1.385},"min":{"1min":0.78000000000000003,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.857,"5min":5.3339999999999996,"15min":5.3339999999999996},"last":2.222},{"interface":"front","average":{"1min":1.448,"5min":1.4970000000000001,"15min":1.4870000000000001},"min":{"1min":0.995,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.8720000000000001,"5min":5.3940000000000001,"15min":5.3940000000000001},"last":1.5760000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.4770000000000001,"15min":1.4710000000000001},"min":{"1min":1.0760000000000001,"5min":0.754,"15min":0.754},"max":{"1min":2.3399999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":2.1549999999999998},{"interface":"front","average":{"1min":1.319,"5min":1.367,"15min":1.413},"min":{"1min":0.92600000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.8169999999999999,"5min":2.641,"15min":2.641},"last":2.028}]}]},{"osd":3,"up_from":14,"seq":60129542212,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":905216,"data_stored":377570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.96599999999999997,"15min":0.95999999999999996},"min":{"1min":0.71799999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.4059999999999999,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.147,"5min":1.1100000000000001,"15min":1.1060000000000001},"min":{"1min":0.73199999999999998,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.4039999999999999,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.2589999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":0.98799999999999999,"15min":1},"min":{"1min":0.70299999999999996,"5min":0.442,"15min":0.442},"max":{"1min":1.8180000000000001,"5min":1.976,"15min":1.976},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.25,"15min":1.264},"min":{"1min":0.91300000000000003,"5min":0.5,"15min":0.5},"max":{"1min":1.7649999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.8100000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.121,"15min":1.143},"min":{"1min":0.71199999999999997,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0779999999999998,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":1.333},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1759999999999999,"15min":1.2549999999999999},"min":{"1min":0.877,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.6060000000000001,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":2.2440000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2669999999999999,"15min":1.2889999999999999},"min":{"1min":0.93799999999999994,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.506,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4650000000000001,"15min":1.5329999999999999},"min":{"1min":1.26,"5min":0.998,"15min":0.998},"max":{"1min":1.966,"5min":4.8970000000000002,"15min":4.8970000000000002},"last":1.7490000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.4730000000000001,"15min":1.5249999999999999},"min":{"1min":1.151,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":1.8080000000000001,"5min":3.4910000000000001,"15min":3.4910000000000001},"last":2.1520000000000001},{"interface":"front","average":{"1min":1.387,"5min":1.399,"15min":1.4690000000000001},"min":{"1min":1.079,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.637,"5min":3.125,"15min":3.125},"last":1.6120000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.4179999999999999,"15min":1.5249999999999999},"min":{"1min":1.0780000000000001,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.024,"5min":4.5499999999999998,"15min":4.5499999999999998},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5029999999999999,"15min":1.5640000000000001},"min":{"1min":1.3180000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.9199999999999999,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":1.974}]},{"osd":7,"last update":"Sat Mar 23 11:24:00 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.46,"15min":1.524},"min":{"1min":1.264,"5min":0.877,"15min":0.877},"max":{"1min":1.7669999999999999,"5min":2.952,"15min":2.952},"last":1.718},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5469999999999999,"15min":1.631},"min":{"1min":1.1200000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.9850000000000001,"5min":4.798,"15min":4.798},"last":1.4670000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":897024,"data_stored":372722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.878,"15min":0.90000000000000002},"min":{"1min":0.64300000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.405,"5min":1.462,"15min":1.462},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.145,"5min":1.0549999999999999,"15min":0.995},"min":{"1min":0.47299999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.5880000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.66200000000000003}]},{"osd":1,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.90700000000000003,"15min":0.95799999999999996},"min":{"1min":0.436,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.351,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.03,"15min":0.98999999999999999},"min":{"1min":0.61799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4710000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.60599999999999998}]},{"osd":3,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1180000000000001,"15min":1.119},"min":{"1min":0.59399999999999997,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.782,"5min":1.98,"15min":1.98},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.0600000000000001,"15min":1.0820000000000001},"min":{"1min":0.72099999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.0579999999999998,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":0.64900000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.361,"15min":1.353},"min":{"1min":1.04,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.702,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.476,"5min":1.4139999999999999,"15min":1.361},"min":{"1min":0.92900000000000005,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.869,"5min":2.085,"15min":2.085},"last":0.92200000000000004}]},{"osd":5,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.3759999999999999,"15min":1.4199999999999999},"min":{"1min":1.1120000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.2589999999999999,"5min":2.5830000000000002,"15min":2.5830000000000002},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.478,"5min":1.377,"15min":1.3740000000000001},"min":{"1min":0.94099999999999995,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.9119999999999999,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.038}]},{"osd":6,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.383,"15min":1.3720000000000001},"min":{"1min":1.02,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6859999999999999,"5min":2.758,"15min":2.758},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.4670000000000001,"15min":1.5189999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.9710000000000001,"5min":2.8580000000000001,"15min":2.8580000000000001},"last":0.80200000000000005}]},{"osd":7,"last update":"Sat Mar 23 11:23:59 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.454,"15min":1.403},"min":{"1min":1.004,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.752,"5min":2.246,"15min":2.246},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.524,"15min":1.5900000000000001},"min":{"1min":1.1739999999999999,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.9710000000000001,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.89200000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542213,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":913408,"data_stored":378969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.92100000000000004,"15min":0.873},"min":{"1min":0.64600000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.6459999999999999,"5min":3.2959999999999998,"15min":3.2959999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.006,"15min":0.996},"min":{"1min":0.77500000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.508,"5min":1.99,"15min":1.99},"last":0.75600000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.95499999999999996,"15min":0.97499999999999998},"min":{"1min":0.60699999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.6479999999999999,"5min":1.738,"15min":1.738},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.143,"5min":1.115,"15min":1.1259999999999999},"min":{"1min":0.90200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.6619999999999999,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.55100000000000005}]},{"osd":3,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.236,"15min":1.23},"min":{"1min":0.94799999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.4750000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.147,"5min":1.113,"15min":1.1639999999999999},"min":{"1min":0.83499999999999996,"5min":0.434,"15min":0.434},"max":{"1min":1.4990000000000001,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":0.90600000000000003}]},{"osd":4,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.3080000000000001,"15min":1.26},"min":{"1min":0.99099999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.8029999999999999,"5min":2.153,"15min":2.153},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3129999999999999,"15min":1.339},"min":{"1min":0.91700000000000004,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.8240000000000001,"5min":2.758,"15min":2.758},"last":0.84699999999999998}]},{"osd":5,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.359,"15min":1.429},"min":{"1min":1.1100000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.855,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.218},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4259999999999999,"15min":1.462},"min":{"1min":1.1399999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.893,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.2749999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4279999999999999,"15min":1.4370000000000001},"min":{"1min":1.0209999999999999,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.73,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.325},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.4770000000000001,"15min":1.569},"min":{"1min":1.123,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9430000000000001,"5min":2.9630000000000001,"15min":2.9630000000000001},"last":1.109}]},{"osd":7,"last update":"Sat Mar 23 11:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.5329999999999999,"15min":1.583},"min":{"1min":1.2250000000000001,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.9890000000000001,"5min":3.2749999999999999,"15min":3.2749999999999999},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5289999999999999,"15min":1.5820000000000001},"min":{"1min":1.179,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.887,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.244}]}]},{"osd":0,"up_from":14,"seq":60129542211,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1474560,"data_stored":960857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.86099999999999999,"15min":0.85499999999999998},"min":{"1min":0.43099999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.115,"5min":2.3769999999999998,"15min":2.3769999999999998},"last":1.107},{"interface":"front","average":{"1min":1.04,"5min":0.98299999999999998,"15min":0.93799999999999994},"min":{"1min":0.63,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.5349999999999999,"5min":1.694,"15min":1.694},"last":0.996}]},{"osd":2,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.071,"15min":1.04},"min":{"1min":0.497,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.538,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.05,"15min":1.016},"min":{"1min":0.63200000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.5009999999999999,"5min":1.802,"15min":1.802},"last":0.77600000000000002}]},{"osd":3,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0880000000000001,"15min":1.0980000000000001},"min":{"1min":0.53700000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.5940000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3009999999999999,"15min":1.3129999999999999},"min":{"1min":0.59199999999999997,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.609,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.1519999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.353,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5569999999999999,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.087},{"interface":"front","average":{"1min":1.331,"5min":1.2749999999999999,"15min":1.2709999999999999},"min":{"1min":0.754,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.6990000000000001,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.675}]},{"osd":5,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.3440000000000001,"15min":1.4339999999999999},"min":{"1min":0.96099999999999997,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6879999999999999,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.444,"15min":1.528},"min":{"1min":0.77100000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.915,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.7789999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.454,"15min":1.5449999999999999},"min":{"1min":0.98099999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.768,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":1.712},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.399,"15min":1.4299999999999999},"min":{"1min":0.90100000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.1150000000000002,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.1279999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.514,"15min":1.546},"min":{"1min":1.246,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0489999999999999,"5min":2.617,"15min":2.617},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.4830000000000001,"15min":1.5109999999999999},"min":{"1min":1.0049999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":2.266,"5min":3.081,"15min":3.081},"last":1.762}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:24:16.043 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T11:24:16.043 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T11:24:16.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.324+0000 7f17a6a00640 1 Processor -- start 2024-03-23T11:24:16.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.325+0000 7f17a6a00640 1 -- start start 2024-03-23T11:24:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.326+0000 7f17a6a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f17a00d1bd0 0x7f17a00b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.326+0000 7f17a6a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f17a00b4880 0x7f17a00b5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:16.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.326+0000 7f17a6a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a00b7d30 con 0x7f17a015c9b0 2024-03-23T11:24:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.326+0000 7f17a6a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a00b4cf0 con 0x7f17a00b4880 2024-03-23T11:24:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.326+0000 7f17a6a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a00b6ed0 con 0x7f17a00d1bd0 2024-03-23T11:24:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.327+0000 7f17a4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f17a00b4880 0x7f17a00b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.327+0000 7f17a4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f17a00b4880 0x7f17a00b5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59416/0 (socket says 172.21.15.62:59416) 2024-03-23T11:24:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.327+0000 7f17a4a00640 1 -- 172.21.15.62:0/1716566677 learned_addr learned my addr 172.21.15.62:0/1716566677 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.327+0000 7f179f400640 1 --2- 172.21.15.62:0/1716566677 >> v2:172.21.15.120:3300/0 conn(0x7f17a00d1bd0 0x7f17a00b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.327+0000 7f17a4a00640 1 -- 172.21.15.62:0/1716566677 >> v2:172.21.15.120:3300/0 conn(0x7f17a00d1bd0 msgr2=0x7f17a00b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.327+0000 7f17a4a00640 1 --2- 172.21.15.62:0/1716566677 >> v2:172.21.15.120:3300/0 conn(0x7f17a00d1bd0 0x7f17a00b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.327+0000 7f17a4a00640 1 -- 172.21.15.62:0/1716566677 >> v1:172.21.15.120:6789/0 conn(0x7f17a015c9b0 legacy=0x7f17a00b8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:16.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.327+0000 7f17a4a00640 1 -- 172.21.15.62:0/1716566677 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a00b7120 con 0x7f17a00b4880 2024-03-23T11:24:16.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.329+0000 7f17a4a00640 1 --2- 172.21.15.62:0/1716566677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17a00b4880 0x7f17a00b5980 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f17a00cd230 tx=0x7f179802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693e3dc4c59b1c7f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:16.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.329+0000 7f179ea00640 1 -- 172.21.15.62:0/1716566677 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179802ca80 con 0x7f17a00b4880 2024-03-23T11:24:16.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.329+0000 7f179ea00640 1 -- 172.21.15.62:0/1716566677 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179802cbe0 con 0x7f17a00b4880 2024-03-23T11:24:16.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.329+0000 7f179ea00640 1 -- 172.21.15.62:0/1716566677 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17980343f0 con 0x7f17a00b4880 2024-03-23T11:24:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.330+0000 7f17a6a00640 1 -- 172.21.15.62:0/1716566677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17a00b4880 msgr2=0x7f17a00b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.330+0000 7f17a6a00640 1 --2- 172.21.15.62:0/1716566677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17a00b4880 0x7f17a00b5980 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f17a00cd230 tx=0x7f179802bee0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.330+0000 7f17a6a00640 1 -- 172.21.15.62:0/1716566677 shutdown_connections 2024-03-23T11:24:16.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.330+0000 7f17a6a00640 1 --2- 172.21.15.62:0/1716566677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17a00b4880 0x7f17a00b5980 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.330+0000 7f17a6a00640 1 --2- 172.21.15.62:0/1716566677 >> v2:172.21.15.120:3300/0 conn(0x7f17a00d1bd0 0x7f17a00b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.330+0000 7f17a6a00640 1 -- 172.21.15.62:0/1716566677 >> 172.21.15.62:0/1716566677 conn(0x7f17a00717b0 msgr2=0x7f17a0071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:16.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.330+0000 7f17a6a00640 1 -- 172.21.15.62:0/1716566677 shutdown_connections 2024-03-23T11:24:16.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.330+0000 7f17a6a00640 1 -- 172.21.15.62:0/1716566677 wait complete. 2024-03-23T11:24:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.332+0000 7f17a6a00640 1 Processor -- start 2024-03-23T11:24:16.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.332+0000 7f17a6a00640 1 -- start start 2024-03-23T11:24:16.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.333+0000 7f17a6a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17a00b4880 0x7f17a0159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.333+0000 7f17a6a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17a00d1bd0 0x7f17a01f2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.333+0000 7f17a6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17a015c9b0 0x7f17a00ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f17a6a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a01f3310 con 0x7f17a015c9b0 2024-03-23T11:24:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f17a6a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00cc010 con 0x7f17a00d1bd0 2024-03-23T11:24:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f17a6a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00caaf0 con 0x7f17a00b4880 2024-03-23T11:24:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f17a4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17a015c9b0 0x7f17a00ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f17a4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17a015c9b0 0x7f17a00ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59418/0 (socket says 172.21.15.62:59418) 2024-03-23T11:24:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f17a4a00640 1 -- 172.21.15.62:0/1668651857 learned_addr learned my addr 172.21.15.62:0/1668651857 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f179f400640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17a00d1bd0 0x7f17a01f2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f179fe00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17a00b4880 0x7f17a0159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:16.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f17a4a00640 1 -- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17a00b4880 msgr2=0x7f17a0159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:16.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f17a4a00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17a00b4880 0x7f17a0159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f17a4a00640 1 -- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17a00d1bd0 msgr2=0x7f17a01f2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:16.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f17a4a00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17a00d1bd0 0x7f17a01f2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.334+0000 7f17a4a00640 1 -- 172.21.15.62:0/1668651857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1798009b30 con 0x7f17a015c9b0 2024-03-23T11:24:16.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.335+0000 7f17a4a00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17a015c9b0 0x7f17a00ca340 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f1798009f40 tx=0x7f1798004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:16.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.335+0000 7f1797e00640 1 -- 172.21.15.62:0/1668651857 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1798009510 con 0x7f17a015c9b0 2024-03-23T11:24:16.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.335+0000 7f1797e00640 1 -- 172.21.15.62:0/1668651857 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179803e450 con 0x7f17a015c9b0 2024-03-23T11:24:16.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.335+0000 7f1797e00640 1 -- 172.21.15.62:0/1668651857 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179803e7e0 con 0x7f17a015c9b0 2024-03-23T11:24:16.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.335+0000 7f17a6a00640 1 -- 172.21.15.62:0/1668651857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a00ca880 con 0x7f17a015c9b0 2024-03-23T11:24:16.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.335+0000 7f17a6a00640 1 -- 172.21.15.62:0/1668651857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a00cc500 con 0x7f17a015c9b0 2024-03-23T11:24:16.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.337+0000 7f17a6a00640 1 -- 172.21.15.62:0/1668651857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a0066bb0 con 0x7f17a015c9b0 2024-03-23T11:24:16.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.338+0000 7f1797e00640 1 -- 172.21.15.62:0/1668651857 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f179802cd50 con 0x7f17a015c9b0 2024-03-23T11:24:16.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.339+0000 7f1797e00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f177c06a7a0 0x7f177c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:16.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.339+0000 7f1797e00640 1 -- 172.21.15.62:0/1668651857 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17980ab990 con 0x7f17a015c9b0 2024-03-23T11:24:16.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.339+0000 7f179fe00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f177c06a7a0 0x7f177c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:16.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.341+0000 7f179fe00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f177c06a7a0 0x7f177c06cc60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1788006fd0 tx=0x7f1788008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:16.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.344+0000 7f1797e00640 1 -- 172.21.15.62:0/1668651857 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1798076d10 con 0x7f17a015c9b0 2024-03-23T11:24:16.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.618+0000 7f17a6a00640 1 -- 172.21.15.62:0/1668651857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f17a00c61d0 con 0x7f17a015c9b0 2024-03-23T11:24:16.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.624+0000 7f1797e00640 1 -- 172.21.15.62:0/1668651857 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+0 (secure 0 0 0) 0x7f17980430e0 con 0x7f17a015c9b0 2024-03-23T11:24:16.619 INFO:teuthology.orchestra.run.smithi062.stderr:updated caps for client.0 2024-03-23T11:24:16.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 -- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f177c06a7a0 msgr2=0x7f177c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:16.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f177c06a7a0 0x7f177c06cc60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1788006fd0 tx=0x7f1788008040 comp rx=0 tx=0).stop 2024-03-23T11:24:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 -- 172.21.15.62:0/1668651857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17a015c9b0 msgr2=0x7f17a00ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17a015c9b0 0x7f17a00ca340 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f1798009f40 tx=0x7f1798004a40 comp rx=0 tx=0).stop 2024-03-23T11:24:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 -- 172.21.15.62:0/1668651857 shutdown_connections 2024-03-23T11:24:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f177c06a7a0 0x7f177c06cc60 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17a015c9b0 0x7f17a00ca340 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17a00d1bd0 0x7f17a01f2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 --2- 172.21.15.62:0/1668651857 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17a00b4880 0x7f17a0159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 -- 172.21.15.62:0/1668651857 >> 172.21.15.62:0/1668651857 conn(0x7f17a00717b0 msgr2=0x7f17a00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 -- 172.21.15.62:0/1668651857 shutdown_connections 2024-03-23T11:24:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.627+0000 7f17a6a00640 1 -- 172.21.15.62:0/1668651857 wait complete. 2024-03-23T11:24:16.639 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:24:16.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.823+0000 7fa997200640 1 Processor -- start 2024-03-23T11:24:16.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.823+0000 7fa997200640 1 -- start start 2024-03-23T11:24:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.824+0000 7fa997200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa9981608b0 0x7fa9980b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.825+0000 7fa997200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980ce510 con 0x7fa99815fcf0 2024-03-23T11:24:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.825+0000 7fa997200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980cdc70 con 0x7fa998160420 2024-03-23T11:24:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.825+0000 7fa997200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9980cdf50 con 0x7fa9981608b0 2024-03-23T11:24:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.825+0000 7fa996800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa9981608b0 0x7fa9980b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.825+0000 7fa995e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa99815fcf0 0x7fa9980d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:38610/0 (socket says 172.21.15.62:38610) 2024-03-23T11:24:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.825+0000 7fa995e00640 1 -- 172.21.15.62:0/2208646823 learned_addr learned my addr 172.21.15.62:0/2208646823 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:16.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.825+0000 7fa996800640 1 -- 172.21.15.62:0/2208646823 >> v1:172.21.15.120:6789/0 conn(0x7fa998160420 legacy=0x7fa9980b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:16.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.825+0000 7fa996800640 1 -- 172.21.15.62:0/2208646823 >> v1:172.21.15.62:6789/0 conn(0x7fa99815fcf0 legacy=0x7fa9980d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:16.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.825+0000 7fa996800640 1 -- 172.21.15.62:0/2208646823 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9980ce230 con 0x7fa9981608b0 2024-03-23T11:24:16.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.826+0000 7fa996800640 1 --2- 172.21.15.62:0/2208646823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9981608b0 0x7fa9980b64c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa990009d90 tx=0x7fa99002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a20861412033132f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:16.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.827+0000 7fa994a00640 1 -- 172.21.15.62:0/2208646823 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99002cbd0 con 0x7fa9981608b0 2024-03-23T11:24:16.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.827+0000 7fa994a00640 1 -- 172.21.15.62:0/2208646823 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99002cd30 con 0x7fa9981608b0 2024-03-23T11:24:16.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.827+0000 7fa994a00640 1 -- 172.21.15.62:0/2208646823 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99002cea0 con 0x7fa9981608b0 2024-03-23T11:24:16.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.827+0000 7fa997200640 1 -- 172.21.15.62:0/2208646823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9981608b0 msgr2=0x7fa9980b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:16.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.827+0000 7fa997200640 1 --2- 172.21.15.62:0/2208646823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9981608b0 0x7fa9980b64c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa990009d90 tx=0x7fa99002be20 comp rx=0 tx=0).stop 2024-03-23T11:24:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.827+0000 7fa997200640 1 -- 172.21.15.62:0/2208646823 shutdown_connections 2024-03-23T11:24:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.827+0000 7fa997200640 1 --2- 172.21.15.62:0/2208646823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9981608b0 0x7fa9980b64c0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.827+0000 7fa997200640 1 -- 172.21.15.62:0/2208646823 >> 172.21.15.62:0/2208646823 conn(0x7fa998089f60 msgr2=0x7fa9980d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.827+0000 7fa997200640 1 -- 172.21.15.62:0/2208646823 shutdown_connections 2024-03-23T11:24:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.827+0000 7fa997200640 1 -- 172.21.15.62:0/2208646823 wait complete. 2024-03-23T11:24:16.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.829+0000 7fa997200640 1 Processor -- start 2024-03-23T11:24:16.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.829+0000 7fa997200640 1 -- start start 2024-03-23T11:24:16.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.830+0000 7fa997200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa99815fcf0 0x7fa998147880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:16.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.830+0000 7fa997200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa998160420 0x7fa9980c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:16.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.830+0000 7fa997200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9981608b0 0x7fa9980c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:16.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.830+0000 7fa997200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9980c8e00 con 0x7fa9981608b0 2024-03-23T11:24:16.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.830+0000 7fa997200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9980c8b20 con 0x7fa998160420 2024-03-23T11:24:16.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.830+0000 7fa997200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa998148100 con 0x7fa99815fcf0 2024-03-23T11:24:16.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.830+0000 7fa996800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9981608b0 0x7fa9980c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:16.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.830+0000 7fa996800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9981608b0 0x7fa9980c9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59428/0 (socket says 172.21.15.62:59428) 2024-03-23T11:24:16.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.830+0000 7fa996800640 1 -- 172.21.15.62:0/40188586 learned_addr learned my addr 172.21.15.62:0/40188586 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:16.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa995400640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa998160420 0x7fa9980c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:16.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa995e00640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa99815fcf0 0x7fa998147880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:16.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa996800640 1 -- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa99815fcf0 msgr2=0x7fa998147880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:16.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa996800640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa99815fcf0 0x7fa998147880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa996800640 1 -- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa998160420 msgr2=0x7fa9980c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa996800640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa998160420 0x7fa9980c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa996800640 1 -- 172.21.15.62:0/40188586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa990009a70 con 0x7fa9981608b0 2024-03-23T11:24:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa995400640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa998160420 0x7fa9980c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa996800640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9981608b0 0x7fa9980c9c60 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa990009d90 tx=0x7fa990004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa98e000640 1 -- 172.21.15.62:0/40188586 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99002cea0 con 0x7fa9981608b0 2024-03-23T11:24:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa98e000640 1 -- 172.21.15.62:0/40188586 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99003e630 con 0x7fa9981608b0 2024-03-23T11:24:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa98e000640 1 -- 172.21.15.62:0/40188586 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99003e9e0 con 0x7fa9981608b0 2024-03-23T11:24:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.831+0000 7fa997200640 1 -- 172.21.15.62:0/40188586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9981483e0 con 0x7fa9981608b0 2024-03-23T11:24:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.832+0000 7fa997200640 1 -- 172.21.15.62:0/40188586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9980c8780 con 0x7fa9981608b0 2024-03-23T11:24:16.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.833+0000 7fa997200640 1 -- 172.21.15.62:0/40188586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9980c6610 con 0x7fa9981608b0 2024-03-23T11:24:16.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.833+0000 7fa98e000640 1 -- 172.21.15.62:0/40188586 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa99003b050 con 0x7fa9981608b0 2024-03-23T11:24:16.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.834+0000 7fa98e000640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa97806a7a0 0x7fa97806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:16.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.834+0000 7fa98e000640 1 -- 172.21.15.62:0/40188586 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9900aafd0 con 0x7fa9981608b0 2024-03-23T11:24:16.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.835+0000 7fa995e00640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa97806a7a0 0x7fa97806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:16.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.836+0000 7fa995e00640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa97806a7a0 0x7fa97806cc60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa988005880 tx=0x7fa988009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:16.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:16.839+0000 7fa98e000640 1 -- 172.21.15.62:0/40188586 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa9900765c0 con 0x7fa9981608b0 2024-03-23T11:24:17.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.068+0000 7fa997200640 1 -- 172.21.15.62:0/40188586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa998066bb0 con 0x7fa9981608b0 2024-03-23T11:24:17.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.069+0000 7fa98e000640 1 -- 172.21.15.62:0/40188586 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fa990079be0 con 0x7fa9981608b0 2024-03-23T11:24:17.064 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:17.064 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6618,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:24:14.062901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6621":{"gid":6621,"name":"d","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/3833782560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3833782560},{"type":"v1","addr":"172.21.15.120:6837","nonce":3833782560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:24:17.064 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 43 2024-03-23T11:24:17.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.071+0000 7fa997200640 1 -- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa97806a7a0 msgr2=0x7fa97806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.071+0000 7fa997200640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa97806a7a0 0x7fa97806cc60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa988005880 tx=0x7fa988009040 comp rx=0 tx=0).stop 2024-03-23T11:24:17.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.072+0000 7fa997200640 1 -- 172.21.15.62:0/40188586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9981608b0 msgr2=0x7fa9980c9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.072+0000 7fa997200640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9981608b0 0x7fa9980c9c60 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa990009d90 tx=0x7fa990004a40 comp rx=0 tx=0).stop 2024-03-23T11:24:17.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.072+0000 7fa997200640 1 -- 172.21.15.62:0/40188586 shutdown_connections 2024-03-23T11:24:17.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.072+0000 7fa997200640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa97806a7a0 0x7fa97806cc60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.072+0000 7fa997200640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9981608b0 0x7fa9980c9c60 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.072+0000 7fa997200640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa998160420 0x7fa9980c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.072+0000 7fa997200640 1 --2- 172.21.15.62:0/40188586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa99815fcf0 0x7fa998147880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.072+0000 7fa997200640 1 -- 172.21.15.62:0/40188586 >> 172.21.15.62:0/40188586 conn(0x7fa998089f60 msgr2=0x7fa9980cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:17.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.072+0000 7fa997200640 1 -- 172.21.15.62:0/40188586 shutdown_connections 2024-03-23T11:24:17.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.072+0000 7fa997200640 1 -- 172.21.15.62:0/40188586 wait complete. 2024-03-23T11:24:17.488 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:24:11.044296+0000', 'modified': '2024-03-23T11:24:14.062901+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6621}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6621': {'gid': 6621, 'name': 'd', 'rank': 0, 'incarnation': 39, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6837/3833782560', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 3833782560}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 3833782560}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:24:17.488 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:24:17.489 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-23T11:24:17.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.703+0000 7f27a4800640 1 Processor -- start 2024-03-23T11:24:17.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.703+0000 7f27a4800640 1 -- start start 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.705+0000 7f27a4800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f279c15c900 0x7f279c0d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.705+0000 7f27a4800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f279c0cd690 0x7f279c0cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.705+0000 7f27a4800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c0ce020 con 0x7f279c15cce0 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.705+0000 7f27a4800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f279c0b89f0 con 0x7f279c0cd690 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.705+0000 7f27a4800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f279c0d1100 con 0x7f279c15c900 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.705+0000 7f27a2800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f279c0cd690 0x7f279c0cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.705+0000 7f27a2800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f279c0cd690 0x7f279c0cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53016/0 (socket says 172.21.15.62:53016) 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.705+0000 7f27a2800640 1 -- 172.21.15.62:0/1515804383 learned_addr learned my addr 172.21.15.62:0/1515804383 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.706+0000 7f27a2800640 1 -- 172.21.15.62:0/1515804383 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f279c15c900 msgr2=0x7f279c0d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.706+0000 7f27a2800640 1 --2- 172.21.15.62:0/1515804383 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f279c15c900 0x7f279c0d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.706+0000 7f27a2800640 1 -- 172.21.15.62:0/1515804383 >> v1:172.21.15.120:6789/0 conn(0x7f279c15cce0 legacy=0x7f279c0b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:17.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.706+0000 7f27a2800640 1 -- 172.21.15.62:0/1515804383 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c0d13e0 con 0x7f279c0cd690 2024-03-23T11:24:17.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.707+0000 7f27a2800640 1 --2- 172.21.15.62:0/1515804383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f279c0cd690 0x7f279c0cdae0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2798009e50 tx=0x7f279802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499eb0797c2e81bc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:17.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.707+0000 7f27a0a00640 1 -- 172.21.15.62:0/1515804383 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279802cb00 con 0x7f279c0cd690 2024-03-23T11:24:17.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.707+0000 7f27a0a00640 1 -- 172.21.15.62:0/1515804383 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279802cc60 con 0x7f279c0cd690 2024-03-23T11:24:17.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.707+0000 7f27a0a00640 1 -- 172.21.15.62:0/1515804383 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27980343f0 con 0x7f279c0cd690 2024-03-23T11:24:17.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.708+0000 7f27a4800640 1 -- 172.21.15.62:0/1515804383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f279c0cd690 msgr2=0x7f279c0cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.708+0000 7f27a4800640 1 --2- 172.21.15.62:0/1515804383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f279c0cd690 0x7f279c0cdae0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2798009e50 tx=0x7f279802bf60 comp rx=0 tx=0).stop 2024-03-23T11:24:17.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.708+0000 7f27a4800640 1 -- 172.21.15.62:0/1515804383 shutdown_connections 2024-03-23T11:24:17.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.708+0000 7f27a4800640 1 --2- 172.21.15.62:0/1515804383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f279c0cd690 0x7f279c0cdae0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.708+0000 7f27a4800640 1 --2- 172.21.15.62:0/1515804383 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f279c15c900 0x7f279c0d21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.708+0000 7f27a4800640 1 -- 172.21.15.62:0/1515804383 >> 172.21.15.62:0/1515804383 conn(0x7f279c071390 msgr2=0x7f279c071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:17.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.708+0000 7f27a4800640 1 -- 172.21.15.62:0/1515804383 shutdown_connections 2024-03-23T11:24:17.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.708+0000 7f27a4800640 1 -- 172.21.15.62:0/1515804383 wait complete. 2024-03-23T11:24:17.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.710+0000 7f27a4800640 1 Processor -- start 2024-03-23T11:24:17.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.710+0000 7f27a4800640 1 -- start start 2024-03-23T11:24:17.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a4800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f279c0cd690 0x7f279c0cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a4800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f279c15c900 0x7f279c15f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a4800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f279c15cce0 0x7f279c0ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a4800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c15fc90 con 0x7f279c0cd690 2024-03-23T11:24:17.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a4800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c0cb900 con 0x7f279c15cce0 2024-03-23T11:24:17.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a4800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f279c0cac80 con 0x7f279c15c900 2024-03-23T11:24:17.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f279c0cd690 0x7f279c0cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f279c0cd690 0x7f279c0cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53030/0 (socket says 172.21.15.62:53030) 2024-03-23T11:24:17.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a1e00640 1 -- 172.21.15.62:0/4058099860 learned_addr learned my addr 172.21.15.62:0/4058099860 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a2800640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f279c15cce0 0x7f279c0ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a1e00640 1 -- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f279c15c900 msgr2=0x7f279c15f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a1e00640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f279c15c900 0x7f279c15f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a1e00640 1 -- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f279c15cce0 msgr2=0x7f279c0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.711+0000 7f27a1e00640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f279c15cce0 0x7f279c0ca440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.712+0000 7f27a1e00640 1 -- 172.21.15.62:0/4058099860 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2798009b30 con 0x7f279c0cd690 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.712+0000 7f27a1e00640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f279c0cd690 0x7f279c0cd0f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f279000b4d0 tx=0x7f279000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.712+0000 7f2796000640 1 -- 172.21.15.62:0/4058099860 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790010870 con 0x7f279c0cd690 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.712+0000 7f2796000640 1 -- 172.21.15.62:0/4058099860 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27900109d0 con 0x7f279c0cd690 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.712+0000 7f2796000640 1 -- 172.21.15.62:0/4058099860 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790010d80 con 0x7f279c0cd690 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.712+0000 7f27a4800640 1 -- 172.21.15.62:0/4058099860 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f279c0caf60 con 0x7f279c0cd690 2024-03-23T11:24:17.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.712+0000 7f27a4800640 1 -- 172.21.15.62:0/4058099860 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279c0cb520 con 0x7f279c0cd690 2024-03-23T11:24:17.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.714+0000 7f2796000640 1 -- 172.21.15.62:0/4058099860 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f279000f3c0 con 0x7f279c0cd690 2024-03-23T11:24:17.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.715+0000 7f2796000640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f278006a7a0 0x7f278006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.715+0000 7f2796000640 1 -- 172.21.15.62:0/4058099860 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2790088d60 con 0x7f279c0cd690 2024-03-23T11:24:17.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.716+0000 7f27a1400640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f278006a7a0 0x7f278006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.716+0000 7f27a762a740 1 Processor -- start 2024-03-23T11:24:17.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.717+0000 7f27a762a740 1 -- start start 2024-03-23T11:24:17.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.717+0000 7f27a1400640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f278006a7a0 0x7f278006cc60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f279c156720 tx=0x7f278c006ba0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:17.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.718+0000 7f27a762a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7eddb8390 0x55a7edd6f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.718+0000 7f27a762a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7edd6e410 0x55a7edd6ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.718+0000 7f27a762a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6a8b0 0x55a7edd9a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.718+0000 7f27a762a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7edcdb110 con 0x55a7edd6a8b0 2024-03-23T11:24:17.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.718+0000 7f27a762a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7edcd88a0 con 0x55a7edd6e410 2024-03-23T11:24:17.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.718+0000 7f27a762a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a7edcd4190 con 0x55a7eddb8390 2024-03-23T11:24:17.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.718+0000 7f27a2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6a8b0 0x55a7edd9a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.718+0000 7f27a2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6a8b0 0x55a7edd9a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53034/0 (socket says 172.21.15.62:53034) 2024-03-23T11:24:17.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.718+0000 7f27a2800640 1 -- 172.21.15.62:0/1990543072 learned_addr learned my addr 172.21.15.62:0/1990543072 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:17.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.719+0000 7f27a1400640 1 --2- 172.21.15.62:0/1990543072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7edd6e410 0x55a7edd6ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.719+0000 7f27a2800640 1 -- 172.21.15.62:0/1990543072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7eddb8390 msgr2=0x55a7edd6f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:17.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.719+0000 7f27a2800640 1 --2- 172.21.15.62:0/1990543072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7eddb8390 0x55a7edd6f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.719+0000 7f27a2800640 1 -- 172.21.15.62:0/1990543072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7edd6e410 msgr2=0x55a7edd6ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:17.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.719+0000 7f27a2800640 1 --2- 172.21.15.62:0/1990543072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7edd6e410 0x55a7edd6ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.719+0000 7f27a2800640 1 -- 172.21.15.62:0/1990543072 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7edc9f230 con 0x55a7edd6a8b0 2024-03-23T11:24:17.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.719+0000 7f27a1400640 1 --2- 172.21.15.62:0/1990543072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7edd6e410 0x55a7edd6ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:17.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.719+0000 7f27a2800640 1 --2- 172.21.15.62:0/1990543072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6a8b0 0x55a7edd9a6a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2798009f70 tx=0x7f279802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b46b35a10384a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:17.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.719+0000 7f278aa00640 1 -- 172.21.15.62:0/1990543072 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2798040470 con 0x55a7edd6a8b0 2024-03-23T11:24:17.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.719+0000 7f278aa00640 1 -- 172.21.15.62:0/1990543072 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27980405d0 con 0x55a7edd6a8b0 2024-03-23T11:24:17.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.720+0000 7f278aa00640 1 -- 172.21.15.62:0/1990543072 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2798040960 con 0x55a7edd6a8b0 2024-03-23T11:24:17.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.720+0000 7f27a762a740 1 -- 172.21.15.62:0/1990543072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6a8b0 msgr2=0x55a7edd9a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.720+0000 7f27a762a740 1 --2- 172.21.15.62:0/1990543072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6a8b0 0x55a7edd9a6a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2798009f70 tx=0x7f279802e000 comp rx=0 tx=0).stop 2024-03-23T11:24:17.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.720+0000 7f27a762a740 1 -- 172.21.15.62:0/1990543072 shutdown_connections 2024-03-23T11:24:17.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.720+0000 7f27a762a740 1 --2- 172.21.15.62:0/1990543072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6a8b0 0x55a7edd9a6a0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.720+0000 7f27a762a740 1 --2- 172.21.15.62:0/1990543072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7edd6e410 0x55a7edd6ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.720+0000 7f27a762a740 1 --2- 172.21.15.62:0/1990543072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7eddb8390 0x55a7edd6f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.720+0000 7f27a762a740 1 -- 172.21.15.62:0/1990543072 >> 172.21.15.62:0/1990543072 conn(0x55a7edd6cb20 msgr2=0x55a7eddb56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:17.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.720+0000 7f27a762a740 1 -- 172.21.15.62:0/1990543072 shutdown_connections 2024-03-23T11:24:17.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.720+0000 7f27a762a740 1 -- 172.21.15.62:0/1990543072 wait complete. 2024-03-23T11:24:17.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.721+0000 7f27a762a740 1 -- start start 2024-03-23T11:24:17.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.722+0000 7f27a762a740 1 Processor -- start 2024-03-23T11:24:17.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.722+0000 7f27a762a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7edd83910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.722+0000 7f27a762a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7eddb8390 0x55a7edd922a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.722+0000 7f27a762a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7edd97130 0x55a7edd975b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.722+0000 7f27a762a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7edc758c0 con 0x55a7edd6e410 2024-03-23T11:24:17.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.722+0000 7f27a762a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7edc75560 con 0x55a7edd97130 2024-03-23T11:24:17.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.722+0000 7f27a762a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a7eda3bf20 con 0x55a7eddb8390 2024-03-23T11:24:17.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.723+0000 7f27a762a740 10 client.0 _ll_register_callbacks cb 0x55a7edccd800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:24:17.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.723+0000 7f27a1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7edd83910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.723+0000 7f27a1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7edd83910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53050/0 (socket says 172.21.15.62:53050) 2024-03-23T11:24:17.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.723+0000 7f27a1e00640 1 -- 172.21.15.62:0/2088243108 learned_addr learned my addr 172.21.15.62:0/2088243108 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:17.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.723+0000 7f27a1e00640 1 -- 172.21.15.62:0/2088243108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7eddb8390 msgr2=0x55a7edd922a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:17.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.723+0000 7f27a1e00640 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7eddb8390 0x55a7edd922a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.723+0000 7f27a2800640 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7edd97130 0x55a7edd975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.723+0000 7f27a1e00640 1 -- 172.21.15.62:0/2088243108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7edd97130 msgr2=0x55a7edd975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:17.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.723+0000 7f27a1e00640 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7edd97130 0x55a7edd975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.723+0000 7f27a1e00640 1 -- 172.21.15.62:0/2088243108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279c0d13e0 con 0x55a7edd6e410 2024-03-23T11:24:17.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f27a1e00640 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7edd83910 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2790054330 tx=0x7f279005bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4707ff2ba011b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:17.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f277f400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:24:17.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f277f400640 1 -- 172.21.15.62:0/2088243108 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279005ff90 con 0x55a7edd6e410 2024-03-23T11:24:17.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f277f400640 1 -- 172.21.15.62:0/2088243108 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2790058e40 con 0x55a7edd6e410 2024-03-23T11:24:17.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f27a762a740 1 -- 172.21.15.62:0/2088243108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a7edcfe280 con 0x55a7edd6e410 2024-03-23T11:24:17.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f277f400640 1 -- 172.21.15.62:0/2088243108 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f279005cb80 con 0x55a7edd6e410 2024-03-23T11:24:17.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f27a762a740 10 client.6735 fetch_fsmap learned FSMap version 43 2024-03-23T11:24:17.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f27a762a740 1 -- 172.21.15.62:0/2088243108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x55a7edcd2810 con 0x55a7edd6e410 2024-03-23T11:24:17.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f277f400640 1 -- 172.21.15.62:0/2088243108 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a7edcfe280 con 0x55a7edd6e410 2024-03-23T11:24:17.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f277f400640 1 -- 172.21.15.62:0/2088243108 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 43) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f279005cce0 con 0x55a7edd6e410 2024-03-23T11:24:17.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f27a762a740 10 client.6735 fetch_fsmap finished waiting for FSMap version 43 2024-03-23T11:24:17.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.724+0000 7f27a762a740 10 client.6735 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T11:24:17.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.725+0000 7f27a762a740 20 client.6735 populate_metadata read hostname 'smithi062' 2024-03-23T11:24:17.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.725+0000 7f27a762a740 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.120:6836/3833782560,v1:172.21.15.120:6837/3833782560] conn(0x55a7edd80c70 0x55a7eddbd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.725+0000 7f27a762a740 4 client.6735 mds_command: new command op to 6621 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:24:17.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.725+0000 7f27a762a740 1 -- 172.21.15.62:0/2088243108 --> [v2:172.21.15.120:6836/3833782560,v1:172.21.15.120:6837/3833782560] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a7edcc69d0 con 0x55a7edd80c70 2024-03-23T11:24:17.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.725+0000 7f27a1400640 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.120:6836/3833782560,v1:172.21.15.120:6837/3833782560] conn(0x55a7edd80c70 0x55a7eddbd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.726+0000 7f27a1400640 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.120:6836/3833782560,v1:172.21.15.120:6837/3833782560] conn(0x55a7edd80c70 0x55a7eddbd270 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=378637babb51894f server_cookie=a36e9e1e58e5c380 in_seq=0 out_seq=0 2024-03-23T11:24:17.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.726+0000 7f277f400640 10 client.6735 ms_handle_connect on v2:172.21.15.120:6836/3833782560 2024-03-23T11:24:17.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f277f400640 1 -- 172.21.15.62:0/2088243108 <== mds.0 v2:172.21.15.120:6836/3833782560 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55a7edcc69d0 con 0x55a7edd80c70 2024-03-23T11:24:17.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f277f400640 10 client.6735 handle_command_reply: tid=0 2024-03-23T11:24:17.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f27a762a740 1 client.6735 shutdown 2024-03-23T11:24:17.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f27a762a740 1 -- 172.21.15.62:0/2088243108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 msgr2=0x55a7edd83910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f27a762a740 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7edd83910 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2790054330 tx=0x7f279005bc60 comp rx=0 tx=0).stop 2024-03-23T11:24:17.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f27a762a740 1 -- 172.21.15.62:0/2088243108 shutdown_connections 2024-03-23T11:24:17.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f27a762a740 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.120:6836/3833782560,v1:172.21.15.120:6837/3833782560] conn(0x55a7edd80c70 0x55a7eddbd270 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f27a762a740 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7edd97130 0x55a7edd975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f27a762a740 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7eddb8390 0x55a7edd922a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f27a762a740 1 --2- 172.21.15.62:0/2088243108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7edd83910 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f277f400640 0 client.6735 ms_handle_reset on v2:172.21.15.120:6836/3833782560 2024-03-23T11:24:17.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.728+0000 7f27a762a740 1 -- 172.21.15.62:0/2088243108 >> 172.21.15.62:0/2088243108 conn(0x55a7edd6a8b0 msgr2=0x55a7edd6eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:17.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.729+0000 7f27a762a740 1 -- 172.21.15.62:0/2088243108 shutdown_connections 2024-03-23T11:24:17.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.729+0000 7f27a762a740 1 -- 172.21.15.62:0/2088243108 wait complete. 2024-03-23T11:24:17.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.729+0000 7f27a762a740 20 client.6735 trim_cache size 0 max 16384 2024-03-23T11:24:17.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.744+0000 7f27a762a740 1 Processor -- start 2024-03-23T11:24:17.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.745+0000 7f27a762a740 1 -- start start 2024-03-23T11:24:17.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.745+0000 7f27a762a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7edd6e410 0x55a7eddb4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.745+0000 7f27a762a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd97130 0x55a7eddbd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.745+0000 7f27a762a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7eddb8390 0x55a7eddbd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.745+0000 7f27a762a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7edd7f530 con 0x55a7edd97130 2024-03-23T11:24:17.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.745+0000 7f27a762a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7edd9e740 con 0x55a7eddb8390 2024-03-23T11:24:17.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.745+0000 7f27a762a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a7edd70d50 con 0x55a7edd6e410 2024-03-23T11:24:17.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.746+0000 7f27a1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd97130 0x55a7eddbd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.746+0000 7f27a1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd97130 0x55a7eddbd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53056/0 (socket says 172.21.15.62:53056) 2024-03-23T11:24:17.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.746+0000 7f27a1400640 1 -- 172.21.15.62:0/1854186903 learned_addr learned my addr 172.21.15.62:0/1854186903 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:17.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.746+0000 7f27a1400640 1 -- 172.21.15.62:0/1854186903 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7edd6e410 msgr2=0x55a7eddb4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:17.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.746+0000 7f27a1400640 1 --2- 172.21.15.62:0/1854186903 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7edd6e410 0x55a7eddb4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.746+0000 7f27a1400640 1 -- 172.21.15.62:0/1854186903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7eddb8390 msgr2=0x55a7eddbd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:17.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.746+0000 7f27a1400640 1 --2- 172.21.15.62:0/1854186903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7eddb8390 0x55a7eddbd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.746+0000 7f27a1400640 1 -- 172.21.15.62:0/1854186903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7edc74ad0 con 0x55a7edd97130 2024-03-23T11:24:17.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.747+0000 7f27a1400640 1 --2- 172.21.15.62:0/1854186903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd97130 0x55a7eddbd270 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f279c157810 tx=0x7f278c00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e1f83e3f5c23e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:17.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.747+0000 7f278b400640 1 -- 172.21.15.62:0/1854186903 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c018040 con 0x55a7edd97130 2024-03-23T11:24:17.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.747+0000 7f278b400640 1 -- 172.21.15.62:0/1854186903 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f278c025070 con 0x55a7edd97130 2024-03-23T11:24:17.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.747+0000 7f278b400640 1 -- 172.21.15.62:0/1854186903 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c013e30 con 0x55a7edd97130 2024-03-23T11:24:17.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.748+0000 7f27a762a740 1 -- 172.21.15.62:0/1854186903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd97130 msgr2=0x55a7eddbd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.748+0000 7f27a762a740 1 --2- 172.21.15.62:0/1854186903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd97130 0x55a7eddbd270 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f279c157810 tx=0x7f278c00e000 comp rx=0 tx=0).stop 2024-03-23T11:24:17.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.748+0000 7f27a762a740 1 -- 172.21.15.62:0/1854186903 shutdown_connections 2024-03-23T11:24:17.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.748+0000 7f27a762a740 1 --2- 172.21.15.62:0/1854186903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd97130 0x55a7eddbd270 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.748+0000 7f27a762a740 1 --2- 172.21.15.62:0/1854186903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7eddb8390 0x55a7eddbd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.748+0000 7f27a762a740 1 --2- 172.21.15.62:0/1854186903 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7edd6e410 0x55a7eddb4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.748+0000 7f27a762a740 1 -- 172.21.15.62:0/1854186903 >> 172.21.15.62:0/1854186903 conn(0x55a7edd6a8b0 msgr2=0x55a7edd66000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:17.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.748+0000 7f27a762a740 1 -- 172.21.15.62:0/1854186903 shutdown_connections 2024-03-23T11:24:17.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.748+0000 7f27a762a740 1 -- 172.21.15.62:0/1854186903 wait complete. 2024-03-23T11:24:17.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.748+0000 7f27a762a740 1 -- start start 2024-03-23T11:24:17.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.749+0000 7f27a762a740 1 Processor -- start 2024-03-23T11:24:17.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a762a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7eddbd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a762a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7edd97130 0x55a7eddbd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a762a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7eddb8390 0x55a7edd95aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a762a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7edd7f780 con 0x55a7edd6e410 2024-03-23T11:24:17.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a762a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7edd7fbe0 con 0x55a7eddb8390 2024-03-23T11:24:17.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a762a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a7eddb46b0 con 0x55a7edd97130 2024-03-23T11:24:17.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a762a740 10 client.0 _ll_register_callbacks cb 0x7f27900585c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:24:17.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7eddbd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7eddbd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53058/0 (socket says 172.21.15.62:53058) 2024-03-23T11:24:17.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a1e00640 1 -- 172.21.15.62:0/1540806401 learned_addr learned my addr 172.21.15.62:0/1540806401 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:17.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a1e00640 1 -- 172.21.15.62:0/1540806401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7edd97130 msgr2=0x55a7eddbd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:17.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a1e00640 1 --2- 172.21.15.62:0/1540806401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7edd97130 0x55a7eddbd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a1e00640 1 -- 172.21.15.62:0/1540806401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7eddb8390 msgr2=0x55a7edd95aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:17.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a1e00640 1 --2- 172.21.15.62:0/1540806401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7eddb8390 0x55a7edd95aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.750+0000 7f27a1e00640 1 -- 172.21.15.62:0/1540806401 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c0218e0 con 0x55a7edd6e410 2024-03-23T11:24:17.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f27a1e00640 1 --2- 172.21.15.62:0/1540806401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7eddbd270 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2790060190 tx=0x7f27900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302894a4afcbd9bf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:17.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f277f400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:24:17.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f277f400640 1 -- 172.21.15.62:0/1540806401 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27900092d0 con 0x55a7edd6e410 2024-03-23T11:24:17.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f277f400640 1 -- 172.21.15.62:0/1540806401 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2790058e40 con 0x55a7edd6e410 2024-03-23T11:24:17.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f27a762a740 1 -- 172.21.15.62:0/1540806401 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f27980405d0 con 0x55a7edd6e410 2024-03-23T11:24:17.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f277f400640 1 -- 172.21.15.62:0/1540806401 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2790076b80 con 0x55a7edd6e410 2024-03-23T11:24:17.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f27a762a740 10 client.6741 fetch_fsmap learned FSMap version 43 2024-03-23T11:24:17.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f27a762a740 1 -- 172.21.15.62:0/1540806401 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x55a7edc99fe0 con 0x55a7edd6e410 2024-03-23T11:24:17.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f277f400640 1 -- 172.21.15.62:0/1540806401 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27980405d0 con 0x55a7edd6e410 2024-03-23T11:24:17.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f277f400640 1 -- 172.21.15.62:0/1540806401 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 43) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2790076070 con 0x55a7edd6e410 2024-03-23T11:24:17.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f27a762a740 10 client.6741 fetch_fsmap finished waiting for FSMap version 43 2024-03-23T11:24:17.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f27a762a740 10 client.6741 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T11:24:17.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f27a762a740 20 client.6741 populate_metadata read hostname 'smithi062' 2024-03-23T11:24:17.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.751+0000 7f27a762a740 1 --2- 172.21.15.62:0/1540806401 >> [v2:172.21.15.120:6836/3833782560,v1:172.21.15.120:6837/3833782560] conn(0x55a7eddb5600 0x55a7edd95fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.752+0000 7f27a762a740 4 client.6741 mds_command: new command op to 6621 tid=0[{"prefix": "status"}] 2024-03-23T11:24:17.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.752+0000 7f27a762a740 1 -- 172.21.15.62:0/1540806401 --> [v2:172.21.15.120:6836/3833782560,v1:172.21.15.120:6837/3833782560] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a7ed9c83c0 con 0x55a7eddb5600 2024-03-23T11:24:17.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.752+0000 7f27a1400640 1 --2- 172.21.15.62:0/1540806401 >> [v2:172.21.15.120:6836/3833782560,v1:172.21.15.120:6837/3833782560] conn(0x55a7eddb5600 0x55a7edd95fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.753+0000 7f27a1400640 1 --2- 172.21.15.62:0/1540806401 >> [v2:172.21.15.120:6836/3833782560,v1:172.21.15.120:6837/3833782560] conn(0x55a7eddb5600 0x55a7edd95fe0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=443be636e01fbbd7 server_cookie=60ce5e4115e42d55 in_seq=0 out_seq=0 2024-03-23T11:24:17.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.753+0000 7f277f400640 10 client.6741 ms_handle_connect on v2:172.21.15.120:6836/3833782560 2024-03-23T11:24:17.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.753+0000 7f277f400640 1 -- 172.21.15.62:0/1540806401 <== mds.0 v2:172.21.15.120:6836/3833782560 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55a7ed9c83c0 con 0x55a7eddb5600 2024-03-23T11:24:17.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.753+0000 7f277f400640 10 client.6741 handle_command_reply: tid=0 2024-03-23T11:24:17.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 1 client.6741 shutdown 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 1 -- 172.21.15.62:0/1540806401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 msgr2=0x55a7eddbd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 1 --2- 172.21.15.62:0/1540806401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7eddbd270 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2790060190 tx=0x7f27900046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 1 -- 172.21.15.62:0/1540806401 shutdown_connections 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 1 --2- 172.21.15.62:0/1540806401 >> [v2:172.21.15.120:6836/3833782560,v1:172.21.15.120:6837/3833782560] conn(0x55a7eddb5600 0x55a7edd95fe0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 1 --2- 172.21.15.62:0/1540806401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7eddb8390 0x55a7edd95aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 1 --2- 172.21.15.62:0/1540806401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a7edd97130 0x55a7eddbd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 1 --2- 172.21.15.62:0/1540806401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a7edd6e410 0x55a7eddbd270 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f277f400640 0 client.6741 ms_handle_reset on v2:172.21.15.120:6836/3833782560 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 1 -- 172.21.15.62:0/1540806401 >> 172.21.15.62:0/1540806401 conn(0x55a7edd6a8b0 msgr2=0x55a7edd99140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 1 -- 172.21.15.62:0/1540806401 shutdown_connections 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 1 -- 172.21.15.62:0/1540806401 wait complete. 2024-03-23T11:24:17.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.754+0000 7f27a762a740 20 client.6741 trim_cache size 0 max 16384 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 6621, 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 6.7038614670000003, 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 43, 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 74, 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 74, 2024-03-23T11:24:17.753 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 12.542425941999999 2024-03-23T11:24:17.754 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:24:17.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 -- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f278006a7a0 msgr2=0x7f278006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f278006a7a0 0x7f278006cc60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f279c156720 tx=0x7f278c006ba0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 -- 172.21.15.62:0/4058099860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f279c0cd690 msgr2=0x7f279c0cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f279c0cd690 0x7f279c0cd0f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f279000b4d0 tx=0x7f279000b990 comp rx=0 tx=0).stop 2024-03-23T11:24:17.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 -- 172.21.15.62:0/4058099860 shutdown_connections 2024-03-23T11:24:17.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f278006a7a0 0x7f278006cc60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f279c15cce0 0x7f279c0ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f279c15c900 0x7f279c15f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 --2- 172.21.15.62:0/4058099860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f279c0cd690 0x7f279c0cd0f0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 -- 172.21.15.62:0/4058099860 >> 172.21.15.62:0/4058099860 conn(0x7f279c071390 msgr2=0x7f279c15af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:17.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 -- 172.21.15.62:0/4058099860 shutdown_connections 2024-03-23T11:24:17.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.755+0000 7f27a4800640 1 -- 172.21.15.62:0/4058099860 wait complete. 2024-03-23T11:24:17.765 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:24:17.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.977+0000 7fb7b1000640 1 Processor -- start 2024-03-23T11:24:17.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.977+0000 7fb7b1000640 1 -- start start 2024-03-23T11:24:17.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.978+0000 7fb7b1000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb7ac160310 0x7fb7ac0ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.978+0000 7fb7b1000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac1608d0 con 0x7fb7ac0ce250 2024-03-23T11:24:17.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.979+0000 7fb7b1000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac160bb0 con 0x7fb7ac15c830 2024-03-23T11:24:17.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.979+0000 7fb7b1000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ac0addb0 con 0x7fb7ac160310 2024-03-23T11:24:17.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.979+0000 7fb7aaa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb7ac160310 0x7fb7ac0ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.979+0000 7fb7aaa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb7ac160310 0x7fb7ac0ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53064/0 (socket says 172.21.15.62:53064) 2024-03-23T11:24:17.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.979+0000 7fb7aaa00640 1 -- 172.21.15.62:0/2745444804 learned_addr learned my addr 172.21.15.62:0/2745444804 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:17.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.979+0000 7fb7aaa00640 1 -- 172.21.15.62:0/2745444804 >> v1:172.21.15.120:6789/0 conn(0x7fb7ac15c830 legacy=0x7fb7ac0ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:17.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.979+0000 7fb7aaa00640 1 -- 172.21.15.62:0/2745444804 >> v1:172.21.15.62:6789/0 conn(0x7fb7ac0ce250 legacy=0x7fb7ac15d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.979+0000 7fb7aaa00640 1 -- 172.21.15.62:0/2745444804 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac160e90 con 0x7fb7ac160310 2024-03-23T11:24:17.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.979+0000 7fb7aa000640 1 -- 172.21.15.62:0/2745444804 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101851195 0 0) 0x7fb7ac1608d0 con 0x7fb7ac0ce250 2024-03-23T11:24:17.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.981+0000 7fb7aaa00640 1 --2- 172.21.15.62:0/2745444804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7ac160310 0x7fb7ac0ad310 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb798009870 tx=0x7fb79802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7bd87791937ff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:17.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.981+0000 7fb7aa000640 1 -- 172.21.15.62:0/2745444804 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798030860 con 0x7fb7ac160310 2024-03-23T11:24:17.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.981+0000 7fb7aa000640 1 -- 172.21.15.62:0/2745444804 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7980309c0 con 0x7fb7ac160310 2024-03-23T11:24:17.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.981+0000 7fb7aa000640 1 -- 172.21.15.62:0/2745444804 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798030d50 con 0x7fb7ac160310 2024-03-23T11:24:17.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.982+0000 7fb7b1000640 1 -- 172.21.15.62:0/2745444804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7ac160310 msgr2=0x7fb7ac0ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.982+0000 7fb7b1000640 1 --2- 172.21.15.62:0/2745444804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7ac160310 0x7fb7ac0ad310 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb798009870 tx=0x7fb79802b9a0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.982+0000 7fb7b1000640 1 -- 172.21.15.62:0/2745444804 shutdown_connections 2024-03-23T11:24:17.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.982+0000 7fb7b1000640 1 --2- 172.21.15.62:0/2745444804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7ac160310 0x7fb7ac0ad310 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.982+0000 7fb7b1000640 1 -- 172.21.15.62:0/2745444804 >> 172.21.15.62:0/2745444804 conn(0x7fb7ac066050 msgr2=0x7fb7ac069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:17.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.982+0000 7fb7b1000640 1 -- 172.21.15.62:0/2745444804 shutdown_connections 2024-03-23T11:24:17.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.982+0000 7fb7b1000640 1 -- 172.21.15.62:0/2745444804 wait complete. 2024-03-23T11:24:17.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.984+0000 7fb7b1000640 1 Processor -- start 2024-03-23T11:24:17.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.984+0000 7fb7b1000640 1 -- start start 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7b1000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7ac0ce250 0x7fb7ac159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7b1000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7ac15c830 0x7fb7ac0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7b1000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7ac160310 0x7fb7ac0b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7b1000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac15a2b0 con 0x7fb7ac15c830 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7b1000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac1f5ce0 con 0x7fb7ac160310 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7b1000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac1f5fc0 con 0x7fb7ac0ce250 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7aaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7ac15c830 0x7fb7ac0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7aaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7ac15c830 0x7fb7ac0cd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53080/0 (socket says 172.21.15.62:53080) 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7aaa00640 1 -- 172.21.15.62:0/2446084006 learned_addr learned my addr 172.21.15.62:0/2446084006 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7abe00640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7ac160310 0x7fb7ac0b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7aaa00640 1 -- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7ac0ce250 msgr2=0x7fb7ac159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7aaa00640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7ac0ce250 0x7fb7ac159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7aaa00640 1 -- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7ac160310 msgr2=0x7fb7ac0b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:17.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.985+0000 7fb7aaa00640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7ac160310 0x7fb7ac0b0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:17.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.986+0000 7fb7aaa00640 1 -- 172.21.15.62:0/2446084006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7980094d0 con 0x7fb7ac15c830 2024-03-23T11:24:17.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.986+0000 7fb7aaa00640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7ac15c830 0x7fb7ac0cd300 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fb798009870 tx=0x7fb798004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:17.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.986+0000 7fb7a3400640 1 -- 172.21.15.62:0/2446084006 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798030810 con 0x7fb7ac15c830 2024-03-23T11:24:17.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.986+0000 7fb7a3400640 1 -- 172.21.15.62:0/2446084006 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb798043400 con 0x7fb7ac15c830 2024-03-23T11:24:17.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.986+0000 7fb7a3400640 1 -- 172.21.15.62:0/2446084006 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7980437b0 con 0x7fb7ac15c830 2024-03-23T11:24:17.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.986+0000 7fb7b1000640 1 -- 172.21.15.62:0/2446084006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ac1f62a0 con 0x7fb7ac15c830 2024-03-23T11:24:17.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.986+0000 7fb7b1000640 1 -- 172.21.15.62:0/2446084006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac1f5900 con 0x7fb7ac15c830 2024-03-23T11:24:17.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.988+0000 7fb7b1000640 1 -- 172.21.15.62:0/2446084006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ac1f5a90 con 0x7fb7ac15c830 2024-03-23T11:24:17.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.988+0000 7fb7a3400640 1 -- 172.21.15.62:0/2446084006 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb798043910 con 0x7fb7ac15c830 2024-03-23T11:24:17.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.989+0000 7fb7a3400640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb78c06a7a0 0x7fb78c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:17.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.990+0000 7fb7a3400640 1 -- 172.21.15.62:0/2446084006 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb79803e070 con 0x7fb7ac15c830 2024-03-23T11:24:17.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.990+0000 7fb7ab400640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb78c06a7a0 0x7fb78c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:17.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.991+0000 7fb7ab400640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb78c06a7a0 0x7fb78c06cc60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb79c003cc0 tx=0x7fb79c0079b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:17.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:17.994+0000 7fb7a3400640 1 -- 172.21.15.62:0/2446084006 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb798077610 con 0x7fb7ac15c830 2024-03-23T11:24:18.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.233+0000 7fb7b1000640 1 -- 172.21.15.62:0/2446084006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb7ac066bb0 con 0x7fb7ac15c830 2024-03-23T11:24:18.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.233+0000 7fb7a3400640 1 -- 172.21.15.62:0/2446084006 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v43) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb798034070 con 0x7fb7ac15c830 2024-03-23T11:24:18.229 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:24:18.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 -- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb78c06a7a0 msgr2=0x7fb78c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:18.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb78c06a7a0 0x7fb78c06cc60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb79c003cc0 tx=0x7fb79c0079b0 comp rx=0 tx=0).stop 2024-03-23T11:24:18.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 -- 172.21.15.62:0/2446084006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7ac15c830 msgr2=0x7fb7ac0cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:18.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7ac15c830 0x7fb7ac0cd300 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fb798009870 tx=0x7fb798004730 comp rx=0 tx=0).stop 2024-03-23T11:24:18.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 -- 172.21.15.62:0/2446084006 shutdown_connections 2024-03-23T11:24:18.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb78c06a7a0 0x7fb78c06cc60 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:18.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7ac15c830 0x7fb7ac0cd300 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:18.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7ac160310 0x7fb7ac0b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:18.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 --2- 172.21.15.62:0/2446084006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7ac0ce250 0x7fb7ac159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:18.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 -- 172.21.15.62:0/2446084006 >> 172.21.15.62:0/2446084006 conn(0x7fb7ac066050 msgr2=0x7fb7ac15f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:18.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 -- 172.21.15.62:0/2446084006 shutdown_connections 2024-03-23T11:24:18.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:18.236+0000 7fb7b1000640 1 -- 172.21.15.62:0/2446084006 wait complete. 2024-03-23T11:24:18.250 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:24:18.250 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:24:18.250 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:24:18.250 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:24:18.250 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:18.250 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:24:18.250 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:24:18.250 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:24:18.250 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:24:18.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:18.251 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:24:18.305 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:24:18.305 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:24:18.306 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42299sec preferred_lft 42299sec 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:24:18.307 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:18.308 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:24:18.308 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:24:18.308 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:24:18.308 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:18.308 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:24:18.308 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:24:18.308 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:24:18.308 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:18.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:18.309 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:24:18.401 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:24:18.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:18.406 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:24:18.460 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:24:18.460 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:24:18.460 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:24:18.460 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:24:18.460 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:24:18.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:18.461 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:24:18.461 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:24:18.462 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:24:18.462 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:24:18.462 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:24:18.462 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:24:18.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:18.654 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:24:18.706 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:24:18.707 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:24:18.707 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:24:18.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:18.708 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:24:18.802 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:24:18.802 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:24:18.803 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:18.859 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:24:18.860 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:24:18.860 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:18.917 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:24:19.018 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:24:19.018 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:24:19.112 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:24:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:24:19.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:19.117 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:24:19.174 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:24:19.174 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:24:19.179 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:24:19.254 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:24:19.254 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:24:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:24:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:24:19.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:19.328 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:24:19.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:24:19.375+0000 7fd7dce47480 -1 init, newargv = 0x56139dd0e800 newargc=15 2024-03-23T11:24:19.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[60236]: starting ceph client 2024-03-23T11:24:19.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[60236]: starting fuse 2024-03-23T11:24:20.469 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:24:20.575 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:24:20.575 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:24:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:24:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:24:20.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:20.682 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:24:20.739 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:24:20.740 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:24:20.740 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:24:20.740 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:24:20.740 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:24:20.740 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:24:20.740 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:24:20.740 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:24:20.740 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:24:20.740 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:24:20.740 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:24:20.742 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:24:20.742 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:24:20.742 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:24:20.742 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:24:20.934 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.60236.asok 2024-03-23T11:24:21.113 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.60236.asok 2024-03-23T11:24:21.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:21.113 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.60236.asok status 2024-03-23T11:24:21.366 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:24:21.366 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:24:21.366 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:24:21.366 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:24:21.366 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:24:21.366 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:24:21.366 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "60236", 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 6756, 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 6756 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:21.367 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 3389682392 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 3389682392 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.6756 192.168.0.1:0/3389682392", 2024-03-23T11:24:21.368 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/3389682392", 2024-03-23T11:24:21.369 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:24:21.369 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 43, 2024-03-23T11:24:21.369 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 74, 2024-03-23T11:24:21.369 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:24:21.369 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:24:21.369 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:24:21.369 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:24:21.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:21.382 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:24:21.382 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:21.439 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:24:21.439 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:21.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:21.439 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:21.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:21.533 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir/) 2024-03-23T11:24:21.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:21.603 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_undamaged) 2024-03-23T11:24:21.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:21.663 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_to_be_damaged) 2024-03-23T11:24:21.721 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:24:21.721 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:24:21.721 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:24:21.721 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:24:21.721 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-03-23T11:24:21.721 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:24:21.855 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627776 2024-03-23T11:24:22.034 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:24:22.034 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:24:22.133 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:24:22.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.363+0000 7f0646400640 1 Processor -- start 2024-03-23T11:24:22.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.363+0000 7f0646400640 1 -- start start 2024-03-23T11:24:22.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.365+0000 7f0646400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0640143dd0 0x7f064015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.365+0000 7f0646400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f06400b2f30 0x7f06400704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.365+0000 7f0646400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0640070a30 con 0x7f0640070090 2024-03-23T11:24:22.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.365+0000 7f0646400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06400cd140 con 0x7f06400b2f30 2024-03-23T11:24:22.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.365+0000 7f0646400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06400cd3e0 con 0x7f0640143dd0 2024-03-23T11:24:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.365+0000 7f0645a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f06400b2f30 0x7f06400704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.366+0000 7f0645a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f06400b2f30 0x7f06400704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53088/0 (socket says 172.21.15.62:53088) 2024-03-23T11:24:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.366+0000 7f0645a00640 1 -- 172.21.15.62:0/2106291809 learned_addr learned my addr 172.21.15.62:0/2106291809 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.366+0000 7f0645000640 1 --2- 172.21.15.62:0/2106291809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0640143dd0 0x7f064015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.366+0000 7f0645a00640 1 -- 172.21.15.62:0/2106291809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0640143dd0 msgr2=0x7f064015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.366+0000 7f0645a00640 1 --2- 172.21.15.62:0/2106291809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0640143dd0 0x7f064015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.366+0000 7f0645a00640 1 -- 172.21.15.62:0/2106291809 >> v1:172.21.15.120:6789/0 conn(0x7f0640070090 legacy=0x7f06400b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.366+0000 7f0645a00640 1 -- 172.21.15.62:0/2106291809 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06400cd6c0 con 0x7f06400b2f30 2024-03-23T11:24:22.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.367+0000 7f0645a00640 1 --2- 172.21.15.62:0/2106291809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06400b2f30 0x7f06400704f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f0638009e50 tx=0x7f063802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86aba652c3bca107 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:22.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.368+0000 7f063fe00640 1 -- 172.21.15.62:0/2106291809 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063802ca70 con 0x7f06400b2f30 2024-03-23T11:24:22.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.368+0000 7f063fe00640 1 -- 172.21.15.62:0/2106291809 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063802cbd0 con 0x7f06400b2f30 2024-03-23T11:24:22.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.368+0000 7f063fe00640 1 -- 172.21.15.62:0/2106291809 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06380343f0 con 0x7f06400b2f30 2024-03-23T11:24:22.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.368+0000 7f0646400640 1 -- 172.21.15.62:0/2106291809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06400b2f30 msgr2=0x7f06400704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:22.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.368+0000 7f0646400640 1 --2- 172.21.15.62:0/2106291809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06400b2f30 0x7f06400704f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f0638009e50 tx=0x7f063802bed0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.368+0000 7f0646400640 1 -- 172.21.15.62:0/2106291809 shutdown_connections 2024-03-23T11:24:22.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.368+0000 7f0646400640 1 --2- 172.21.15.62:0/2106291809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06400b2f30 0x7f06400704f0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.368+0000 7f0646400640 1 --2- 172.21.15.62:0/2106291809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0640143dd0 0x7f064015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.369+0000 7f0646400640 1 -- 172.21.15.62:0/2106291809 >> 172.21.15.62:0/2106291809 conn(0x7f0640066050 msgr2=0x7f0640069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:22.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.369+0000 7f0646400640 1 -- 172.21.15.62:0/2106291809 shutdown_connections 2024-03-23T11:24:22.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.369+0000 7f0646400640 1 -- 172.21.15.62:0/2106291809 wait complete. 2024-03-23T11:24:22.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.370+0000 7f0646400640 1 Processor -- start 2024-03-23T11:24:22.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.371+0000 7f0646400640 1 -- start start 2024-03-23T11:24:22.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0646400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0640070090 0x7f06400c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0646400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06400b2f30 0x7f064015a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0646400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0640143dd0 0x7f064015aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0646400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06400cb320 con 0x7f0640070090 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0646400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06400c4030 con 0x7f06400b2f30 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0646400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06400c4310 con 0x7f0640143dd0 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0645000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0640070090 0x7f06400c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0645000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0640070090 0x7f06400c8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53102/0 (socket says 172.21.15.62:53102) 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0645000640 1 -- 172.21.15.62:0/3574584985 learned_addr learned my addr 172.21.15.62:0/3574584985 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0645000640 1 -- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0640143dd0 msgr2=0x7f064015aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0645000640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0640143dd0 0x7f064015aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0645000640 1 -- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06400b2f30 msgr2=0x7f064015a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0645000640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06400b2f30 0x7f064015a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.372+0000 7f0645000640 1 -- 172.21.15.62:0/3574584985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0638009b30 con 0x7f0640070090 2024-03-23T11:24:22.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.373+0000 7f0645000640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0640070090 0x7f06400c8c30 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f062800d5f0 tx=0x7f062800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:22.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.373+0000 7f063d600640 1 -- 172.21.15.62:0/3574584985 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06280125f0 con 0x7f0640070090 2024-03-23T11:24:22.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.373+0000 7f063d600640 1 -- 172.21.15.62:0/3574584985 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0628012750 con 0x7f0640070090 2024-03-23T11:24:22.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.373+0000 7f063d600640 1 -- 172.21.15.62:0/3574584985 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0628012b00 con 0x7f0640070090 2024-03-23T11:24:22.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.373+0000 7f0646400640 1 -- 172.21.15.62:0/3574584985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06400c45f0 con 0x7f0640070090 2024-03-23T11:24:22.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.373+0000 7f0646400640 1 -- 172.21.15.62:0/3574584985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06400c4bb0 con 0x7f0640070090 2024-03-23T11:24:22.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.374+0000 7f0646400640 1 -- 172.21.15.62:0/3574584985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06400cca40 con 0x7f0640070090 2024-03-23T11:24:22.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.375+0000 7f063d600640 1 -- 172.21.15.62:0/3574584985 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0628012c60 con 0x7f0640070090 2024-03-23T11:24:22.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.376+0000 7f063d600640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f061c06a7a0 0x7f061c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.376+0000 7f063d600640 1 -- 172.21.15.62:0/3574584985 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0628089310 con 0x7f0640070090 2024-03-23T11:24:22.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.377+0000 7f0637e00640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f061c06a7a0 0x7f061c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:22.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.378+0000 7f0637e00640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f061c06a7a0 0x7f061c06cc60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0630003cc0 tx=0x7f06300079b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:22.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.381+0000 7f063d600640 1 -- 172.21.15.62:0/3574584985 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06280546a0 con 0x7f0640070090 2024-03-23T11:24:22.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.582+0000 7f0646400640 1 -- 172.21.15.62:0/3574584985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0640066bb0 con 0x7f0640070090 2024-03-23T11:24:22.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.584+0000 7f063d600640 1 -- 172.21.15.62:0/3574584985 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+16100 (secure 0 0 0) 0x7f0628057cc0 con 0x7f0640070090 2024-03-23T11:24:22.580 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:22.580 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":74,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:12.048899+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:24:08.688706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-23T11:24:10.072774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:24:22.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.587+0000 7f0646400640 1 -- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f061c06a7a0 msgr2=0x7f061c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:22.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.587+0000 7f0646400640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f061c06a7a0 0x7f061c06cc60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0630003cc0 tx=0x7f06300079b0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.587+0000 7f0646400640 1 -- 172.21.15.62:0/3574584985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0640070090 msgr2=0x7f06400c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:22.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.587+0000 7f0646400640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0640070090 0x7f06400c8c30 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f062800d5f0 tx=0x7f062800dab0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.588+0000 7f0646400640 1 -- 172.21.15.62:0/3574584985 shutdown_connections 2024-03-23T11:24:22.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.588+0000 7f0646400640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f061c06a7a0 0x7f061c06cc60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.588+0000 7f0646400640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0640143dd0 0x7f064015aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.588+0000 7f0646400640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06400b2f30 0x7f064015a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.588+0000 7f0646400640 1 --2- 172.21.15.62:0/3574584985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0640070090 0x7f06400c8c30 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.588+0000 7f0646400640 1 -- 172.21.15.62:0/3574584985 >> 172.21.15.62:0/3574584985 conn(0x7f0640066050 msgr2=0x7f064006d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:22.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.588+0000 7f0646400640 1 -- 172.21.15.62:0/3574584985 shutdown_connections 2024-03-23T11:24:22.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.588+0000 7f0646400640 1 -- 172.21.15.62:0/3574584985 wait complete. 2024-03-23T11:24:22.603 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:24:08.688706+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:24:10.072774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:24:22.603 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:24:22.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.831+0000 7f4bad200640 1 Processor -- start 2024-03-23T11:24:22.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.832+0000 7f4bad200640 1 -- start start 2024-03-23T11:24:22.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.833+0000 7f4bad200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ba815c830 0x7f4ba80ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.833+0000 7f4bad200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4ba80ce250 0x7f4ba80ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.833+0000 7f4bad200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba81608d0 con 0x7f4ba8160310 2024-03-23T11:24:22.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.833+0000 7f4bad200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba80addb0 con 0x7f4ba815c830 2024-03-23T11:24:22.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.833+0000 7f4bad200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba815cc10 con 0x7f4ba80ce250 2024-03-23T11:24:22.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.834+0000 7f4ba7400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ba815c830 0x7f4ba80ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:22.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.834+0000 7f4ba7400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ba815c830 0x7f4ba80ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53106/0 (socket says 172.21.15.62:53106) 2024-03-23T11:24:22.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.834+0000 7f4ba7400640 1 -- 172.21.15.62:0/3636809549 learned_addr learned my addr 172.21.15.62:0/3636809549 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:22.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.834+0000 7f4ba7e00640 1 --2- 172.21.15.62:0/3636809549 >> v2:172.21.15.120:3300/0 conn(0x7f4ba80ce250 0x7f4ba80ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:22.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.834+0000 7f4ba7400640 1 -- 172.21.15.62:0/3636809549 >> v2:172.21.15.120:3300/0 conn(0x7f4ba80ce250 msgr2=0x7f4ba80ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:22.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.834+0000 7f4ba7400640 1 --2- 172.21.15.62:0/3636809549 >> v2:172.21.15.120:3300/0 conn(0x7f4ba80ce250 0x7f4ba80ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.834+0000 7f4ba7400640 1 -- 172.21.15.62:0/3636809549 >> v1:172.21.15.120:6789/0 conn(0x7f4ba8160310 legacy=0x7f4ba80ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:22.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.835+0000 7f4ba7400640 1 -- 172.21.15.62:0/3636809549 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba8160bb0 con 0x7f4ba815c830 2024-03-23T11:24:22.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.836+0000 7f4ba7400640 1 --2- 172.21.15.62:0/3636809549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba815c830 0x7f4ba80ad310 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f4ba806c190 tx=0x7f4b9802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6336891a95eddf2c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:22.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.837+0000 7f4ba6a00640 1 -- 172.21.15.62:0/3636809549 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98030860 con 0x7f4ba815c830 2024-03-23T11:24:22.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.837+0000 7f4ba6a00640 1 -- 172.21.15.62:0/3636809549 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b980309c0 con 0x7f4ba815c830 2024-03-23T11:24:22.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.837+0000 7f4ba6a00640 1 -- 172.21.15.62:0/3636809549 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98030d50 con 0x7f4ba815c830 2024-03-23T11:24:22.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.837+0000 7f4bad200640 1 -- 172.21.15.62:0/3636809549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba815c830 msgr2=0x7f4ba80ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:22.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.837+0000 7f4bad200640 1 --2- 172.21.15.62:0/3636809549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba815c830 0x7f4ba80ad310 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f4ba806c190 tx=0x7f4b9802b9a0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.837+0000 7f4bad200640 1 -- 172.21.15.62:0/3636809549 shutdown_connections 2024-03-23T11:24:22.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.837+0000 7f4bad200640 1 --2- 172.21.15.62:0/3636809549 >> v2:172.21.15.120:3300/0 conn(0x7f4ba80ce250 0x7f4ba80ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.837+0000 7f4bad200640 1 --2- 172.21.15.62:0/3636809549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba815c830 0x7f4ba80ad310 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.837+0000 7f4bad200640 1 -- 172.21.15.62:0/3636809549 >> 172.21.15.62:0/3636809549 conn(0x7f4ba8066050 msgr2=0x7f4ba8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:22.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.838+0000 7f4bad200640 1 -- 172.21.15.62:0/3636809549 shutdown_connections 2024-03-23T11:24:22.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.838+0000 7f4bad200640 1 -- 172.21.15.62:0/3636809549 wait complete. 2024-03-23T11:24:22.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.839+0000 7f4bad200640 1 Processor -- start 2024-03-23T11:24:22.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.840+0000 7f4bad200640 1 -- start start 2024-03-23T11:24:22.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.840+0000 7f4bad200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ba80ce250 0x7f4ba8151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.840+0000 7f4bad200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ba815c830 0x7f4ba80af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.840+0000 7f4bad200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8160310 0x7f4ba80af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4bad200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8140750 con 0x7f4ba8160310 2024-03-23T11:24:22.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4bad200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba81f20b0 con 0x7f4ba815c830 2024-03-23T11:24:22.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4bad200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba81f2370 con 0x7f4ba80ce250 2024-03-23T11:24:22.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4ba7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8160310 0x7f4ba80af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:22.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4ba7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8160310 0x7f4ba80af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53110/0 (socket says 172.21.15.62:53110) 2024-03-23T11:24:22.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4ba7e00640 1 -- 172.21.15.62:0/2076304484 learned_addr learned my addr 172.21.15.62:0/2076304484 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:22.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4b9ea00640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ba815c830 0x7f4ba80af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:22.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4ba7e00640 1 -- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ba80ce250 msgr2=0x7f4ba8151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:22.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4ba7e00640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ba80ce250 0x7f4ba8151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4ba7e00640 1 -- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ba815c830 msgr2=0x7f4ba80af0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:22.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4ba7e00640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ba815c830 0x7f4ba80af0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:22.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.841+0000 7f4ba7e00640 1 -- 172.21.15.62:0/2076304484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b980094d0 con 0x7f4ba8160310 2024-03-23T11:24:22.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.842+0000 7f4ba7e00640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8160310 0x7f4ba80af610 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0007e60 tx=0x7f4ba000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:22.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.842+0000 7f4b9fe00640 1 -- 172.21.15.62:0/2076304484 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0014420 con 0x7f4ba8160310 2024-03-23T11:24:22.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.842+0000 7f4b9fe00640 1 -- 172.21.15.62:0/2076304484 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba00145a0 con 0x7f4ba8160310 2024-03-23T11:24:22.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.842+0000 7f4b9fe00640 1 -- 172.21.15.62:0/2076304484 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0014950 con 0x7f4ba8160310 2024-03-23T11:24:22.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.842+0000 7f4bad200640 1 -- 172.21.15.62:0/2076304484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba81f2650 con 0x7f4ba8160310 2024-03-23T11:24:22.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.842+0000 7f4bad200640 1 -- 172.21.15.62:0/2076304484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba81f2c10 con 0x7f4ba8160310 2024-03-23T11:24:22.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.843+0000 7f4bad200640 1 -- 172.21.15.62:0/2076304484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba8066bb0 con 0x7f4ba8160310 2024-03-23T11:24:22.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.844+0000 7f4b9fe00640 1 -- 172.21.15.62:0/2076304484 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4ba0014ab0 con 0x7f4ba8160310 2024-03-23T11:24:22.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.845+0000 7f4b9fe00640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b8406a7a0 0x7f4b8406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:22.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.845+0000 7f4b9fe00640 1 -- 172.21.15.62:0/2076304484 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ba0089610 con 0x7f4ba8160310 2024-03-23T11:24:22.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.846+0000 7f4ba7400640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b8406a7a0 0x7f4b8406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:22.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.847+0000 7f4ba7400640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b8406a7a0 0x7f4b8406cc60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4b98036000 tx=0x7f4b9803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:22.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:22.850+0000 7f4b9fe00640 1 -- 172.21.15.62:0/2076304484 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ba0054ac0 con 0x7f4ba8160310 2024-03-23T11:24:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.049+0000 7f4bad200640 1 -- 172.21.15.62:0/2076304484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ba8155c90 con 0x7f4ba8160310 2024-03-23T11:24:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.050+0000 7f4b9fe00640 1 -- 172.21.15.62:0/2076304484 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ba00580e0 con 0x7f4ba8160310 2024-03-23T11:24:23.046 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:24:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.053+0000 7f4bad200640 1 -- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b8406a7a0 msgr2=0x7f4b8406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.053+0000 7f4bad200640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b8406a7a0 0x7f4b8406cc60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4b98036000 tx=0x7f4b9803b040 comp rx=0 tx=0).stop 2024-03-23T11:24:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.053+0000 7f4bad200640 1 -- 172.21.15.62:0/2076304484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8160310 msgr2=0x7f4ba80af610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.053+0000 7f4bad200640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8160310 0x7f4ba80af610 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0007e60 tx=0x7f4ba000e4b0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.054+0000 7f4bad200640 1 -- 172.21.15.62:0/2076304484 shutdown_connections 2024-03-23T11:24:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.054+0000 7f4bad200640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4b8406a7a0 0x7f4b8406cc60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.054+0000 7f4bad200640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8160310 0x7f4ba80af610 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.054+0000 7f4bad200640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ba815c830 0x7f4ba80af0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.054+0000 7f4bad200640 1 --2- 172.21.15.62:0/2076304484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ba80ce250 0x7f4ba8151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.054+0000 7f4bad200640 1 -- 172.21.15.62:0/2076304484 >> 172.21.15.62:0/2076304484 conn(0x7f4ba8066050 msgr2=0x7f4ba8070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.054+0000 7f4bad200640 1 -- 172.21.15.62:0/2076304484 shutdown_connections 2024-03-23T11:24:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.054+0000 7f4bad200640 1 -- 172.21.15.62:0/2076304484 wait complete. 2024-03-23T11:24:23.066 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:24:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.293+0000 7f95f3200640 1 Processor -- start 2024-03-23T11:24:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.294+0000 7f95f3200640 1 -- start start 2024-03-23T11:24:23.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.295+0000 7f95f3200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f95ec070090 0x7f95ec070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.295+0000 7f95f3200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0cd140 con 0x7f95ec143dd0 2024-03-23T11:24:23.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.295+0000 7f95f3200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0cd3e0 con 0x7f95ec0b2820 2024-03-23T11:24:23.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.295+0000 7f95f3200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec070a60 con 0x7f95ec070090 2024-03-23T11:24:23.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.296+0000 7f95f1e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f95ec143dd0 0x7f95ec15abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:56666/0 (socket says 172.21.15.62:56666) 2024-03-23T11:24:23.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.296+0000 7f95f1e00640 1 -- 172.21.15.62:0/378131849 learned_addr learned my addr 172.21.15.62:0/378131849 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.296+0000 7f95f1400640 1 --2- 172.21.15.62:0/378131849 >> v2:172.21.15.120:3300/0 conn(0x7f95ec070090 0x7f95ec070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.296+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290753282 0 0) 0x7f95ec0cd140 con 0x7f95ec143dd0 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.296+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d4003520 con 0x7f95ec143dd0 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1810805727 0 0) 0x7f95d4003520 con 0x7f95ec143dd0 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 >> v2:172.21.15.120:3300/0 conn(0x7f95ec070090 msgr2=0x7f95ec070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 --2- 172.21.15.62:0/378131849 >> v2:172.21.15.120:3300/0 conn(0x7f95ec070090 0x7f95ec070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 >> v1:172.21.15.120:6789/0 conn(0x7f95ec0b2820 legacy=0x7f95ec0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0cd6c0 con 0x7f95ec143dd0 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f95e0001670 con 0x7f95ec143dd0 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 >> v1:172.21.15.62:6789/0 conn(0x7f95ec143dd0 legacy=0x7f95ec15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 --2- 172.21.15.62:0/378131849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95d4003970 0x7f95d4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 --2- 172.21.15.62:0/378131849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95d40047c0 0x7f95d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 --2- 172.21.15.62:0/378131849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95d4005180 0x7f95d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec070a60 con 0x7f95d4005180 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d400c020 con 0x7f95d4003970 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d400c1e0 con 0x7f95d40047c0 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.297+0000 7f95f2800640 1 --2- 172.21.15.62:0/378131849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95d4005180 0x7f95d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.298+0000 7f95f1e00640 1 --2- 172.21.15.62:0/378131849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95d4003970 0x7f95d4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.298+0000 7f95f1400640 1 --2- 172.21.15.62:0/378131849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95d40047c0 0x7f95d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.298+0000 7f95f2800640 1 -- 172.21.15.62:0/378131849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95d40047c0 msgr2=0x7f95d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.298+0000 7f95f2800640 1 --2- 172.21.15.62:0/378131849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95d40047c0 0x7f95d4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.298+0000 7f95f2800640 1 -- 172.21.15.62:0/378131849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95d4003970 msgr2=0x7f95d4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.298+0000 7f95f2800640 1 --2- 172.21.15.62:0/378131849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95d4003970 0x7f95d4003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.298+0000 7f95f2800640 1 -- 172.21.15.62:0/378131849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d4003520 con 0x7f95d4005180 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.299+0000 7f95f2800640 1 -- 172.21.15.62:0/378131849 reap_dead start 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.299+0000 7f95f2800640 1 --2- 172.21.15.62:0/378131849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95d4005180 0x7f95d400b980 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f95e8001e50 tx=0x7f95d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=795cc80a0f30b59f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.299+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e8003b70 con 0x7f95d4005180 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.299+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e8003d10 con 0x7f95d4005180 2024-03-23T11:24:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.299+0000 7f95f0a00640 1 -- 172.21.15.62:0/378131849 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e8005310 con 0x7f95d4005180 2024-03-23T11:24:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.300+0000 7f95f3200640 1 -- 172.21.15.62:0/378131849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95d4005180 msgr2=0x7f95d400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.300+0000 7f95f3200640 1 --2- 172.21.15.62:0/378131849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95d4005180 0x7f95d400b980 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f95e8001e50 tx=0x7f95d40057f0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.300+0000 7f95f3200640 1 -- 172.21.15.62:0/378131849 shutdown_connections 2024-03-23T11:24:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.300+0000 7f95f3200640 1 --2- 172.21.15.62:0/378131849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95d4005180 0x7f95d400b980 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.300+0000 7f95f3200640 1 -- 172.21.15.62:0/378131849 >> 172.21.15.62:0/378131849 conn(0x7f95ec066050 msgr2=0x7f95ec069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.300+0000 7f95f3200640 1 -- 172.21.15.62:0/378131849 shutdown_connections 2024-03-23T11:24:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.300+0000 7f95f3200640 1 -- 172.21.15.62:0/378131849 wait complete. 2024-03-23T11:24:23.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.301+0000 7f95f3200640 1 Processor -- start 2024-03-23T11:24:23.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.302+0000 7f95f3200640 1 -- start start 2024-03-23T11:24:23.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f3200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95d4005180 0x7f95ec0aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f3200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95ec0c3c80 0x7f95ec1415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0c58e0 0x7f95ec0cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f3200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0ab3e0 con 0x7f95ec0c58e0 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f3200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0ab6f0 con 0x7f95ec0c3c80 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f3200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0ab9d0 con 0x7f95d4005180 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0c58e0 0x7f95ec0cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0c58e0 0x7f95ec0cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53140/0 (socket says 172.21.15.62:53140) 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f2800640 1 -- 172.21.15.62:0/3810452870 learned_addr learned my addr 172.21.15.62:0/3810452870 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f2800640 1 -- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95d4005180 msgr2=0x7f95ec0aaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f1e00640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95d4005180 0x7f95ec0aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f1400640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95ec0c3c80 0x7f95ec1415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f2800640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95d4005180 0x7f95ec0aaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f2800640 1 -- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95ec0c3c80 msgr2=0x7f95ec1415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f2800640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95ec0c3c80 0x7f95ec1415a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.303+0000 7f95f2800640 1 -- 172.21.15.62:0/3810452870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0abcb0 con 0x7f95ec0c58e0 2024-03-23T11:24:23.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.304+0000 7f95f2800640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0c58e0 0x7f95ec0cad00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f95e8001f70 tx=0x7f95e80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:23.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.304+0000 7f95e6000640 1 -- 172.21.15.62:0/3810452870 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e8006d50 con 0x7f95ec0c58e0 2024-03-23T11:24:23.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.304+0000 7f95e6000640 1 -- 172.21.15.62:0/3810452870 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e8007030 con 0x7f95ec0c58e0 2024-03-23T11:24:23.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.304+0000 7f95e6000640 1 -- 172.21.15.62:0/3810452870 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e8007420 con 0x7f95ec0c58e0 2024-03-23T11:24:23.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.304+0000 7f95f3200640 1 -- 172.21.15.62:0/3810452870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec0ab3e0 con 0x7f95ec0c58e0 2024-03-23T11:24:23.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.304+0000 7f95f3200640 1 -- 172.21.15.62:0/3810452870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec0ac270 con 0x7f95ec0c58e0 2024-03-23T11:24:23.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.306+0000 7f95f3200640 1 -- 172.21.15.62:0/3810452870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec066bb0 con 0x7f95ec0c58e0 2024-03-23T11:24:23.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.306+0000 7f95e6000640 1 -- 172.21.15.62:0/3810452870 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f95e8007580 con 0x7f95ec0c58e0 2024-03-23T11:24:23.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.307+0000 7f95e6000640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f95d006a7a0 0x7f95d006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.307+0000 7f95e6000640 1 -- 172.21.15.62:0/3810452870 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95e80aaa60 con 0x7f95ec0c58e0 2024-03-23T11:24:23.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.308+0000 7f95f1e00640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f95d006a7a0 0x7f95d006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.309+0000 7f95f1e00640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f95d006a7a0 0x7f95d006cc60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0b1860 tx=0x7f95e0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:23.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.312+0000 7f95e6000640 1 -- 172.21.15.62:0/3810452870 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f95e8075f10 con 0x7f95ec0c58e0 2024-03-23T11:24:23.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.516+0000 7f95f3200640 1 -- 172.21.15.62:0/3810452870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f95ec0b0ed0 con 0x7f95ec0c58e0 2024-03-23T11:24:23.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.517+0000 7f95e6000640 1 -- 172.21.15.62:0/3810452870 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v74) v1 ==== 104+0+11 (secure 0 0 0) 0x7f95e8079530 con 0x7f95ec0c58e0 2024-03-23T11:24:23.512 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:24:23.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.519+0000 7f95f3200640 1 -- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f95d006a7a0 msgr2=0x7f95d006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.519+0000 7f95f3200640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f95d006a7a0 0x7f95d006cc60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0b1860 tx=0x7f95e0009040 comp rx=0 tx=0).stop 2024-03-23T11:24:23.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.519+0000 7f95f3200640 1 -- 172.21.15.62:0/3810452870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0c58e0 msgr2=0x7f95ec0cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.519+0000 7f95f3200640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0c58e0 0x7f95ec0cad00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f95e8001f70 tx=0x7f95e80014a0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.519+0000 7f95f3200640 1 -- 172.21.15.62:0/3810452870 shutdown_connections 2024-03-23T11:24:23.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.519+0000 7f95f3200640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f95d006a7a0 0x7f95d006cc60 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.519+0000 7f95f3200640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95ec0c58e0 0x7f95ec0cad00 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.519+0000 7f95f3200640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95ec0c3c80 0x7f95ec1415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.519+0000 7f95f3200640 1 --2- 172.21.15.62:0/3810452870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95d4005180 0x7f95ec0aaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.519+0000 7f95f3200640 1 -- 172.21.15.62:0/3810452870 >> 172.21.15.62:0/3810452870 conn(0x7f95ec066050 msgr2=0x7f95ec15c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.520+0000 7f95f3200640 1 -- 172.21.15.62:0/3810452870 shutdown_connections 2024-03-23T11:24:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.520+0000 7f95f3200640 1 -- 172.21.15.62:0/3810452870 wait complete. 2024-03-23T11:24:23.535 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:24:23.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.763+0000 7f1eb6a00640 1 Processor -- start 2024-03-23T11:24:23.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.763+0000 7f1eb6a00640 1 -- start start 2024-03-23T11:24:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.765+0000 7f1eb6a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb0143dd0 0x7f1eb015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.765+0000 7f1eb6a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1eb00b2f30 0x7f1eb00704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.765+0000 7f1eb6a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb0070a30 con 0x7f1eb0070090 2024-03-23T11:24:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.765+0000 7f1eb6a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb00cd140 con 0x7f1eb00b2f30 2024-03-23T11:24:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.765+0000 7f1eb6a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00cd3e0 con 0x7f1eb0143dd0 2024-03-23T11:24:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.765+0000 7f1eb6000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1eb00b2f30 0x7f1eb00704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.765+0000 7f1eb6000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1eb00b2f30 0x7f1eb00704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53156/0 (socket says 172.21.15.62:53156) 2024-03-23T11:24:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.765+0000 7f1eb6000640 1 -- 172.21.15.62:0/1804577674 learned_addr learned my addr 172.21.15.62:0/1804577674 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.766+0000 7f1eb5600640 1 --2- 172.21.15.62:0/1804577674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb0143dd0 0x7f1eb015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.766+0000 7f1eb6000640 1 -- 172.21.15.62:0/1804577674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb0143dd0 msgr2=0x7f1eb015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.766+0000 7f1eb6000640 1 --2- 172.21.15.62:0/1804577674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb0143dd0 0x7f1eb015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.766+0000 7f1eb6000640 1 -- 172.21.15.62:0/1804577674 >> v1:172.21.15.62:6789/0 conn(0x7f1eb0070090 legacy=0x7f1eb00b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.766+0000 7f1eb6000640 1 -- 172.21.15.62:0/1804577674 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00cd6c0 con 0x7f1eb00b2f30 2024-03-23T11:24:23.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.767+0000 7f1eb6000640 1 --2- 172.21.15.62:0/1804577674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb00b2f30 0x7f1eb00704f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8009e50 tx=0x7f1ea802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2967a3d601036c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:23.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.767+0000 7f1eafe00640 1 -- 172.21.15.62:0/1804577674 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea802cb00 con 0x7f1eb00b2f30 2024-03-23T11:24:23.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.767+0000 7f1eafe00640 1 -- 172.21.15.62:0/1804577674 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea802cc60 con 0x7f1eb00b2f30 2024-03-23T11:24:23.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.767+0000 7f1eafe00640 1 -- 172.21.15.62:0/1804577674 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea80343f0 con 0x7f1eb00b2f30 2024-03-23T11:24:23.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.768+0000 7f1eb6a00640 1 -- 172.21.15.62:0/1804577674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb00b2f30 msgr2=0x7f1eb00704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.768+0000 7f1eb6a00640 1 --2- 172.21.15.62:0/1804577674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb00b2f30 0x7f1eb00704f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8009e50 tx=0x7f1ea802bf60 comp rx=0 tx=0).stop 2024-03-23T11:24:23.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.768+0000 7f1eb6a00640 1 -- 172.21.15.62:0/1804577674 shutdown_connections 2024-03-23T11:24:23.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.768+0000 7f1eb6a00640 1 --2- 172.21.15.62:0/1804577674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb00b2f30 0x7f1eb00704f0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.768+0000 7f1eb6a00640 1 --2- 172.21.15.62:0/1804577674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb0143dd0 0x7f1eb015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.768+0000 7f1eb6a00640 1 -- 172.21.15.62:0/1804577674 >> 172.21.15.62:0/1804577674 conn(0x7f1eb0066050 msgr2=0x7f1eb0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:23.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.768+0000 7f1eb6a00640 1 -- 172.21.15.62:0/1804577674 shutdown_connections 2024-03-23T11:24:23.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.768+0000 7f1eb6a00640 1 -- 172.21.15.62:0/1804577674 wait complete. 2024-03-23T11:24:23.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.770+0000 7f1eb6a00640 1 Processor -- start 2024-03-23T11:24:23.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.770+0000 7f1eb6a00640 1 -- start start 2024-03-23T11:24:23.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.771+0000 7f1eb6a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb0070090 0x7f1eb015a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.771+0000 7f1eb6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb00b2f30 0x7f1eb00c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.771+0000 7f1eb6a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb0143dd0 0x7f1eb00b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.771+0000 7f1eb6a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00cb210 con 0x7f1eb00b2f30 2024-03-23T11:24:23.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.771+0000 7f1eb6a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb015a610 con 0x7f1eb0143dd0 2024-03-23T11:24:23.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.771+0000 7f1eb6a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00ca600 con 0x7f1eb0070090 2024-03-23T11:24:23.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.771+0000 7f1eb4c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb00b2f30 0x7f1eb00c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.771+0000 7f1eb4c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb00b2f30 0x7f1eb00c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53158/0 (socket says 172.21.15.62:53158) 2024-03-23T11:24:23.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.771+0000 7f1eb4c00640 1 -- 172.21.15.62:0/499677389 learned_addr learned my addr 172.21.15.62:0/499677389 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:23.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1eb5600640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb0070090 0x7f1eb015a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1eb6000640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb0143dd0 0x7f1eb00b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1eb4c00640 1 -- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb0070090 msgr2=0x7f1eb015a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1eb4c00640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb0070090 0x7f1eb015a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1eb4c00640 1 -- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb0143dd0 msgr2=0x7f1eb00b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1eb4c00640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb0143dd0 0x7f1eb00b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1eb4c00640 1 -- 172.21.15.62:0/499677389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8009b30 con 0x7f1eb00b2f30 2024-03-23T11:24:23.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1eb6000640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb0143dd0 0x7f1eb00b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:23.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1eb4c00640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb00b2f30 0x7f1eb00c9bb0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00c450 tx=0x7f1e9c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:23.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1ead600640 1 -- 172.21.15.62:0/499677389 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c0052d0 con 0x7f1eb00b2f30 2024-03-23T11:24:23.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1ead600640 1 -- 172.21.15.62:0/499677389 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9c005430 con 0x7f1eb00b2f30 2024-03-23T11:24:23.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1eb6a00640 1 -- 172.21.15.62:0/499677389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb00ca8e0 con 0x7f1eb00b2f30 2024-03-23T11:24:23.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.772+0000 7f1ead600640 1 -- 172.21.15.62:0/499677389 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c012870 con 0x7f1eb00b2f30 2024-03-23T11:24:23.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.773+0000 7f1eb6a00640 1 -- 172.21.15.62:0/499677389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb00caea0 con 0x7f1eb00b2f30 2024-03-23T11:24:23.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.774+0000 7f1eb6a00640 1 -- 172.21.15.62:0/499677389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb0071320 con 0x7f1eb00b2f30 2024-03-23T11:24:23.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.775+0000 7f1ead600640 1 -- 172.21.15.62:0/499677389 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f1e9c0129d0 con 0x7f1eb00b2f30 2024-03-23T11:24:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.775+0000 7f1ead600640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1e9006a7d0 0x7f1e9006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.776+0000 7f1ead600640 1 -- 172.21.15.62:0/499677389 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e9c088920 con 0x7f1eb00b2f30 2024-03-23T11:24:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.776+0000 7f1eb5600640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1e9006a7d0 0x7f1e9006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.777+0000 7f1eb5600640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1e9006a7d0 0x7f1e9006cc90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4006fd0 tx=0x7f1ea4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.780+0000 7f1ead600640 1 -- 172.21.15.62:0/499677389 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e9c053ca0 con 0x7f1eb00b2f30 2024-03-23T11:24:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.979+0000 7f1eb6a00640 1 -- 172.21.15.62:0/499677389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1eb0066bb0 con 0x7f1eb00b2f30 2024-03-23T11:24:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.980+0000 7f1ead600640 1 -- 172.21.15.62:0/499677389 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v74) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1e9c0572c0 con 0x7f1eb00b2f30 2024-03-23T11:24:23.976 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.983+0000 7f1eb6a00640 1 -- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1e9006a7d0 msgr2=0x7f1e9006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.983+0000 7f1eb6a00640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1e9006a7d0 0x7f1e9006cc90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4006fd0 tx=0x7f1ea4008040 comp rx=0 tx=0).stop 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.983+0000 7f1eb6a00640 1 -- 172.21.15.62:0/499677389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb00b2f30 msgr2=0x7f1eb00c9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.983+0000 7f1eb6a00640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb00b2f30 0x7f1eb00c9bb0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00c450 tx=0x7f1e9c00c910 comp rx=0 tx=0).stop 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.983+0000 7f1eb6a00640 1 -- 172.21.15.62:0/499677389 shutdown_connections 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.983+0000 7f1eb6a00640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1e9006a7d0 0x7f1e9006cc90 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.983+0000 7f1eb6a00640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb00b2f30 0x7f1eb00c9bb0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.983+0000 7f1eb6a00640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1eb0143dd0 0x7f1eb00b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.983+0000 7f1eb6a00640 1 --2- 172.21.15.62:0/499677389 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1eb0070090 0x7f1eb015a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.983+0000 7f1eb6a00640 1 -- 172.21.15.62:0/499677389 >> 172.21.15.62:0/499677389 conn(0x7f1eb0066050 msgr2=0x7f1eb015b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.984+0000 7f1eb6a00640 1 -- 172.21.15.62:0/499677389 shutdown_connections 2024-03-23T11:24:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:23.984+0000 7f1eb6a00640 1 -- 172.21.15.62:0/499677389 wait complete. 2024-03-23T11:24:23.998 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:24:24.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.226+0000 7ffa53e00640 1 Processor -- start 2024-03-23T11:24:24.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.226+0000 7ffa53e00640 1 -- start start 2024-03-23T11:24:24.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.227+0000 7ffa53e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffa54069c60 0x7ffa5406a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.227+0000 7ffa53e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5415bb10 con 0x7ffa54143450 2024-03-23T11:24:24.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.227+0000 7ffa53e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5406a580 con 0x7ffa540caff0 2024-03-23T11:24:24.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.228+0000 7ffa53e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa540ce1a0 con 0x7ffa54069c60 2024-03-23T11:24:24.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.228+0000 7ffa53400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ffa54143450 0x7ffa541438a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:56694/0 (socket says 172.21.15.62:56694) 2024-03-23T11:24:24.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.228+0000 7ffa53400640 1 -- 172.21.15.62:0/3723807268 learned_addr learned my addr 172.21.15.62:0/3723807268 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:24.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.228+0000 7ffa52a00640 1 --2- 172.21.15.62:0/3723807268 >> v2:172.21.15.120:3300/0 conn(0x7ffa54069c60 0x7ffa5406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.228+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612148407 0 0) 0x7ffa5415bb10 con 0x7ffa54143450 2024-03-23T11:24:24.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.228+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa38003520 con 0x7ffa54143450 2024-03-23T11:24:24.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2387826575 0 0) 0x7ffa38003520 con 0x7ffa54143450 2024-03-23T11:24:24.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 >> v2:172.21.15.120:3300/0 conn(0x7ffa54069c60 msgr2=0x7ffa5406a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:24.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 --2- 172.21.15.62:0/3723807268 >> v2:172.21.15.120:3300/0 conn(0x7ffa54069c60 0x7ffa5406a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 >> v1:172.21.15.120:6789/0 conn(0x7ffa540caff0 legacy=0x7ffa540cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5415ad50 con 0x7ffa54143450 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7ffa4c0019f0 con 0x7ffa54143450 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 >> v1:172.21.15.62:6789/0 conn(0x7ffa54143450 legacy=0x7ffa541438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa38003970 0x7ffa38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa380047c0 0x7ffa38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa38005180 0x7ffa3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa540ce1a0 con 0x7ffa38005180 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3800c020 con 0x7ffa38003970 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3800c1e0 con 0x7ffa380047c0 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.229+0000 7ffa53400640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa38005180 0x7ffa3800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.230+0000 7ffa52a00640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa38003970 0x7ffa38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.230+0000 7ffa4be00640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa380047c0 0x7ffa38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.230+0000 7ffa53400640 1 -- 172.21.15.62:0/3723807268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa380047c0 msgr2=0x7ffa38004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:24.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.230+0000 7ffa53400640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa380047c0 0x7ffa38004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.230+0000 7ffa53400640 1 -- 172.21.15.62:0/3723807268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa38003970 msgr2=0x7ffa38003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:24.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.230+0000 7ffa53400640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa38003970 0x7ffa38003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.230+0000 7ffa53400640 1 -- 172.21.15.62:0/3723807268 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa38003520 con 0x7ffa38005180 2024-03-23T11:24:24.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.230+0000 7ffa52a00640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa38003970 0x7ffa38003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:24.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.230+0000 7ffa4be00640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa380047c0 0x7ffa38004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:24.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.231+0000 7ffa53400640 1 -- 172.21.15.62:0/3723807268 reap_dead start 2024-03-23T11:24:24.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.231+0000 7ffa53400640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa38005180 0x7ffa3800b980 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00a830 tx=0x7ffa380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec267f77f697b16 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:24.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.231+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c0072d0 con 0x7ffa38005180 2024-03-23T11:24:24.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.231+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4c02f070 con 0x7ffa38005180 2024-03-23T11:24:24.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.231+0000 7ffa52000640 1 -- 172.21.15.62:0/3723807268 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c0076b0 con 0x7ffa38005180 2024-03-23T11:24:24.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.232+0000 7ffa53e00640 1 -- 172.21.15.62:0/3723807268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa38005180 msgr2=0x7ffa3800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.232+0000 7ffa53e00640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa38005180 0x7ffa3800b980 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00a830 tx=0x7ffa380057f0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.232+0000 7ffa53e00640 1 -- 172.21.15.62:0/3723807268 shutdown_connections 2024-03-23T11:24:24.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.232+0000 7ffa53e00640 1 --2- 172.21.15.62:0/3723807268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa38005180 0x7ffa3800b980 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.232+0000 7ffa53e00640 1 -- 172.21.15.62:0/3723807268 >> 172.21.15.62:0/3723807268 conn(0x7ffa540b0090 msgr2=0x7ffa540b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:24.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.232+0000 7ffa53e00640 1 -- 172.21.15.62:0/3723807268 shutdown_connections 2024-03-23T11:24:24.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.232+0000 7ffa53e00640 1 -- 172.21.15.62:0/3723807268 wait complete. 2024-03-23T11:24:24.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.234+0000 7ffa53e00640 1 Processor -- start 2024-03-23T11:24:24.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.234+0000 7ffa53e00640 1 -- start start 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.235+0000 7ffa53e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa38005180 0x7ffa5413eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.235+0000 7ffa53e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa540af700 0x7ffa540ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.235+0000 7ffa53e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa540afb60 0x7ffa540acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.235+0000 7ffa53e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa540cbcb0 con 0x7ffa540af700 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.235+0000 7ffa53e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa540b2280 con 0x7ffa38005180 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.235+0000 7ffa53e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5413f980 con 0x7ffa540afb60 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.235+0000 7ffa4be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa540af700 0x7ffa540ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.235+0000 7ffa4be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa540af700 0x7ffa540ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53188/0 (socket says 172.21.15.62:53188) 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.235+0000 7ffa4be00640 1 -- 172.21.15.62:0/2799119229 learned_addr learned my addr 172.21.15.62:0/2799119229 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa52a00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa38005180 0x7ffa5413eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa53400640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa540afb60 0x7ffa540acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa4be00640 1 -- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa540afb60 msgr2=0x7ffa540acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa4be00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa540afb60 0x7ffa540acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa4be00640 1 -- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa38005180 msgr2=0x7ffa5413eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa4be00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa38005180 0x7ffa5413eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa4be00640 1 -- 172.21.15.62:0/2799119229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5413fc60 con 0x7ffa540af700 2024-03-23T11:24:24.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa4be00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa540af700 0x7ffa540ac890 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00d560 tx=0x7ffa3c00da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:24.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa4aa00640 1 -- 172.21.15.62:0/2799119229 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c005480 con 0x7ffa540af700 2024-03-23T11:24:24.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa4aa00640 1 -- 172.21.15.62:0/2799119229 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3c003610 con 0x7ffa540af700 2024-03-23T11:24:24.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa53e00640 1 -- 172.21.15.62:0/2799119229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa540b1c30 con 0x7ffa540af700 2024-03-23T11:24:24.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.236+0000 7ffa4aa00640 1 -- 172.21.15.62:0/2799119229 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c003980 con 0x7ffa540af700 2024-03-23T11:24:24.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.237+0000 7ffa53e00640 1 -- 172.21.15.62:0/2799119229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5413f0e0 con 0x7ffa540af700 2024-03-23T11:24:24.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.238+0000 7ffa53e00640 1 -- 172.21.15.62:0/2799119229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa54066bb0 con 0x7ffa540af700 2024-03-23T11:24:24.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.238+0000 7ffa4aa00640 1 -- 172.21.15.62:0/2799119229 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ffa3c003ae0 con 0x7ffa540af700 2024-03-23T11:24:24.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.239+0000 7ffa4aa00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa3006a7a0 0x7ffa3006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.239+0000 7ffa4aa00640 1 -- 172.21.15.62:0/2799119229 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa3c088d60 con 0x7ffa540af700 2024-03-23T11:24:24.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.240+0000 7ffa52a00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa3006a7a0 0x7ffa3006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.241+0000 7ffa52a00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa3006a7a0 0x7ffa3006cc60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ffa44009aa0 tx=0x7ffa44008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:24.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.244+0000 7ffa4aa00640 1 -- 172.21.15.62:0/2799119229 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa3c054050 con 0x7ffa540af700 2024-03-23T11:24:24.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.450+0000 7ffa53e00640 1 -- 172.21.15.62:0/2799119229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffa540b23c0 con 0x7ffa540af700 2024-03-23T11:24:24.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.451+0000 7ffa4aa00640 1 -- 172.21.15.62:0/2799119229 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v74) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffa3c001230 con 0x7ffa540af700 2024-03-23T11:24:24.446 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:24:24.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 -- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa3006a7a0 msgr2=0x7ffa3006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa3006a7a0 0x7ffa3006cc60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ffa44009aa0 tx=0x7ffa44008040 comp rx=0 tx=0).stop 2024-03-23T11:24:24.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 -- 172.21.15.62:0/2799119229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa540af700 msgr2=0x7ffa540ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa540af700 0x7ffa540ac890 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00d560 tx=0x7ffa3c00da20 comp rx=0 tx=0).stop 2024-03-23T11:24:24.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 -- 172.21.15.62:0/2799119229 shutdown_connections 2024-03-23T11:24:24.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa3006a7a0 0x7ffa3006cc60 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa540af700 0x7ffa540ac890 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa540afb60 0x7ffa540acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 --2- 172.21.15.62:0/2799119229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa38005180 0x7ffa5413eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 -- 172.21.15.62:0/2799119229 >> 172.21.15.62:0/2799119229 conn(0x7ffa540b0090 msgr2=0x7ffa540b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:24.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 -- 172.21.15.62:0/2799119229 shutdown_connections 2024-03-23T11:24:24.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.454+0000 7ffa53e00640 1 -- 172.21.15.62:0/2799119229 wait complete. 2024-03-23T11:24:24.465 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:24:24.465 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:24:24.514 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:24:24.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.734+0000 7fe471600640 1 Processor -- start 2024-03-23T11:24:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.735+0000 7fe471600640 1 -- start start 2024-03-23T11:24:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.736+0000 7fe471600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe46c15fcf0 0x7fe46c0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.736+0000 7fe471600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe46c160420 0x7fe46c1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.736+0000 7fe471600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46c0d06d0 0x7fe46c0d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.736+0000 7fe471600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe46c160df0 con 0x7fe46c160420 2024-03-23T11:24:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.736+0000 7fe471600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe46c0d3540 con 0x7fe46c15fcf0 2024-03-23T11:24:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.736+0000 7fe471600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c0ce510 con 0x7fe46c0d06d0 2024-03-23T11:24:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.737+0000 7fe46b400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe46c160420 0x7fe46c1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.737+0000 7fe46b400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe46c160420 0x7fe46c1608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53200/0 (socket says 172.21.15.62:53200) 2024-03-23T11:24:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.737+0000 7fe46b400640 1 -- 172.21.15.62:0/2747871843 learned_addr learned my addr 172.21.15.62:0/2747871843 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:24.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.737+0000 7fe470c00640 1 --2- 172.21.15.62:0/2747871843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46c0d06d0 0x7fe46c0d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.737+0000 7fe46be00640 1 --2- 172.21.15.62:0/2747871843 >> v2:172.21.15.120:3300/0 conn(0x7fe46c15fcf0 0x7fe46c0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.737+0000 7fe46b400640 1 -- 172.21.15.62:0/2747871843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46c0d06d0 msgr2=0x7fe46c0d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:24.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.737+0000 7fe46b400640 1 --2- 172.21.15.62:0/2747871843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46c0d06d0 0x7fe46c0d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.737+0000 7fe46b400640 1 -- 172.21.15.62:0/2747871843 >> v2:172.21.15.120:3300/0 conn(0x7fe46c15fcf0 msgr2=0x7fe46c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:24.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.737+0000 7fe46b400640 1 --2- 172.21.15.62:0/2747871843 >> v2:172.21.15.120:3300/0 conn(0x7fe46c15fcf0 0x7fe46c0d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.737+0000 7fe46b400640 1 -- 172.21.15.62:0/2747871843 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46c0cdc70 con 0x7fe46c160420 2024-03-23T11:24:24.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.737+0000 7fe470c00640 1 --2- 172.21.15.62:0/2747871843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46c0d06d0 0x7fe46c0d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.738+0000 7fe46b400640 1 --2- 172.21.15.62:0/2747871843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe46c160420 0x7fe46c1608b0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fe46c066480 tx=0x7fe45402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1e5636a6e36f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.739+0000 7fe46aa00640 1 -- 172.21.15.62:0/2747871843 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4540317d0 con 0x7fe46c160420 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.739+0000 7fe46aa00640 1 -- 172.21.15.62:0/2747871843 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe454031930 con 0x7fe46c160420 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.739+0000 7fe46aa00640 1 -- 172.21.15.62:0/2747871843 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454031cc0 con 0x7fe46c160420 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.739+0000 7fe471600640 1 -- 172.21.15.62:0/2747871843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe46c160420 msgr2=0x7fe46c1608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.739+0000 7fe471600640 1 --2- 172.21.15.62:0/2747871843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe46c160420 0x7fe46c1608b0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fe46c066480 tx=0x7fe45402b870 comp rx=0 tx=0).stop 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.739+0000 7fe471600640 1 -- 172.21.15.62:0/2747871843 shutdown_connections 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.740+0000 7fe471600640 1 --2- 172.21.15.62:0/2747871843 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46c0d06d0 0x7fe46c0d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.740+0000 7fe471600640 1 --2- 172.21.15.62:0/2747871843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe46c160420 0x7fe46c1608b0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.740+0000 7fe471600640 1 --2- 172.21.15.62:0/2747871843 >> v2:172.21.15.120:3300/0 conn(0x7fe46c15fcf0 0x7fe46c0d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.740+0000 7fe471600640 1 -- 172.21.15.62:0/2747871843 >> 172.21.15.62:0/2747871843 conn(0x7fe46c089f60 msgr2=0x7fe46c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.740+0000 7fe471600640 1 -- 172.21.15.62:0/2747871843 shutdown_connections 2024-03-23T11:24:24.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.740+0000 7fe471600640 1 -- 172.21.15.62:0/2747871843 wait complete. 2024-03-23T11:24:24.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.741+0000 7fe471600640 1 Processor -- start 2024-03-23T11:24:24.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.741+0000 7fe471600640 1 -- start start 2024-03-23T11:24:24.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.742+0000 7fe471600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe46c0d06d0 0x7fe46c09f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.742+0000 7fe471600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe46c15fcf0 0x7fe46c1f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.742+0000 7fe471600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46c160420 0x7fe46c09ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.742+0000 7fe471600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c15c4c0 con 0x7fe46c0d06d0 2024-03-23T11:24:24.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.742+0000 7fe471600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c09f620 con 0x7fe46c15fcf0 2024-03-23T11:24:24.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.742+0000 7fe471600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c09f900 con 0x7fe46c160420 2024-03-23T11:24:24.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.743+0000 7fe46be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe46c0d06d0 0x7fe46c09f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.743+0000 7fe46be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe46c0d06d0 0x7fe46c09f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53204/0 (socket says 172.21.15.62:53204) 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.743+0000 7fe46be00640 1 -- 172.21.15.62:0/29761544 learned_addr learned my addr 172.21.15.62:0/29761544 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.743+0000 7fe46b400640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe46c15fcf0 0x7fe46c1f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.743+0000 7fe470c00640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46c160420 0x7fe46c09ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.743+0000 7fe46be00640 1 -- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46c160420 msgr2=0x7fe46c09ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.743+0000 7fe46be00640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46c160420 0x7fe46c09ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.743+0000 7fe46be00640 1 -- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe46c15fcf0 msgr2=0x7fe46c1f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.743+0000 7fe46be00640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe46c15fcf0 0x7fe46c1f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.743+0000 7fe46be00640 1 -- 172.21.15.62:0/29761544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4540094d0 con 0x7fe46c0d06d0 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.744+0000 7fe46be00640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe46c0d06d0 0x7fe46c09f040 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fe45c007ef0 tx=0x7fe45c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.744+0000 7fe463e00640 1 -- 172.21.15.62:0/29761544 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c014620 con 0x7fe46c0d06d0 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.744+0000 7fe463e00640 1 -- 172.21.15.62:0/29761544 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45c014780 con 0x7fe46c0d06d0 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.744+0000 7fe463e00640 1 -- 172.21.15.62:0/29761544 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c014b30 con 0x7fe46c0d06d0 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.744+0000 7fe471600640 1 -- 172.21.15.62:0/29761544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe46c09fbe0 con 0x7fe46c0d06d0 2024-03-23T11:24:24.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.744+0000 7fe471600640 1 -- 172.21.15.62:0/29761544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46c0a01a0 con 0x7fe46c0d06d0 2024-03-23T11:24:24.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.745+0000 7fe471600640 1 -- 172.21.15.62:0/29761544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46c0a0330 con 0x7fe46c0d06d0 2024-03-23T11:24:24.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.746+0000 7fe463e00640 1 -- 172.21.15.62:0/29761544 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe45c014c90 con 0x7fe46c0d06d0 2024-03-23T11:24:24.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.747+0000 7fe463e00640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe44806a7d0 0x7fe44806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:24.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.747+0000 7fe463e00640 1 -- 172.21.15.62:0/29761544 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe45c089830 con 0x7fe46c0d06d0 2024-03-23T11:24:24.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.748+0000 7fe46b400640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe44806a7d0 0x7fe44806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:24.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.749+0000 7fe46b400640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe44806a7d0 0x7fe44806cc90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe454036000 tx=0x7fe45403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:24.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.751+0000 7fe463e00640 1 -- 172.21.15.62:0/29761544 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe45c054bb0 con 0x7fe46c0d06d0 2024-03-23T11:24:24.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.943+0000 7fe471600640 1 -- 172.21.15.62:0/29761544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe46c066bb0 con 0x7fe46c0d06d0 2024-03-23T11:24:24.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.946+0000 7fe463e00640 1 -- 172.21.15.62:0/29761544 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fe45c0581d0 con 0x7fe46c0d06d0 2024-03-23T11:24:24.942 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:24.942 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":74,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:12.048899+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:24:08.688706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-23T11:24:10.072774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:24:24.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 -- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe44806a7d0 msgr2=0x7fe44806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe44806a7d0 0x7fe44806cc90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe454036000 tx=0x7fe45403d040 comp rx=0 tx=0).stop 2024-03-23T11:24:24.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 -- 172.21.15.62:0/29761544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe46c0d06d0 msgr2=0x7fe46c09f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:24.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe46c0d06d0 0x7fe46c09f040 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fe45c007ef0 tx=0x7fe45c00e4b0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 -- 172.21.15.62:0/29761544 shutdown_connections 2024-03-23T11:24:24.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe44806a7d0 0x7fe44806cc90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46c160420 0x7fe46c09ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe46c15fcf0 0x7fe46c1f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 --2- 172.21.15.62:0/29761544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe46c0d06d0 0x7fe46c09f040 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:24.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 -- 172.21.15.62:0/29761544 >> 172.21.15.62:0/29761544 conn(0x7fe46c089f60 msgr2=0x7fe46c0b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:24.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 -- 172.21.15.62:0/29761544 shutdown_connections 2024-03-23T11:24:24.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:24.949+0000 7fe471600640 1 -- 172.21.15.62:0/29761544 wait complete. 2024-03-23T11:24:24.963 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:24:08.688706+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:24:10.072774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:24:24.963 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:24:25.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.187+0000 7fd3aa400640 1 Processor -- start 2024-03-23T11:24:25.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.188+0000 7fd3aa400640 1 -- start start 2024-03-23T11:24:25.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.189+0000 7fd3aa400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4160420 0x7fd3a4160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.189+0000 7fd3aa400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a4160dc0 con 0x7fd3a40b5db0 2024-03-23T11:24:25.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.189+0000 7fd3aa400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a40ce510 con 0x7fd3a415fcf0 2024-03-23T11:24:25.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.189+0000 7fd3aa400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a40cdc70 con 0x7fd3a4160420 2024-03-23T11:24:25.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.190+0000 7fd3a9a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd3a40b5db0 0x7fd3a40b6220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:56700/0 (socket says 172.21.15.62:56700) 2024-03-23T11:24:25.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.190+0000 7fd3a9a00640 1 -- 172.21.15.62:0/1868454126 learned_addr learned my addr 172.21.15.62:0/1868454126 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:25.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.190+0000 7fd3a3e00640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4160420 0x7fd3a4160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.190+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596654337 0 0) 0x7fd3a4160dc0 con 0x7fd3a40b5db0 2024-03-23T11:24:25.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.190+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd388003520 con 0x7fd3a40b5db0 2024-03-23T11:24:25.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.190+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3720104690 0 0) 0x7fd388003520 con 0x7fd3a40b5db0 2024-03-23T11:24:25.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.190+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4160420 msgr2=0x7fd3a4160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:25.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.190+0000 7fd3a3400640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4160420 0x7fd3a4160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.190+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 >> v1:172.21.15.120:6789/0 conn(0x7fd3a415fcf0 legacy=0x7fd3a40d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.190+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a40cdf50 con 0x7fd3a40b5db0 2024-03-23T11:24:25.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.190+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fd39c0019f0 con 0x7fd3a40b5db0 2024-03-23T11:24:25.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 >> v1:172.21.15.62:6789/0 conn(0x7fd3a40b5db0 legacy=0x7fd3a40b6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3400640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4160420 0x7fd3880039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3400640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd388003f20 0x7fd3880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3400640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd388004d00 0x7fd388005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a40cdc70 con 0x7fd388003f20 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38800bb10 con 0x7fd388004d00 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd38800bcd0 con 0x7fd3a4160420 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3e00640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd388003f20 0x7fd3880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a9000640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4160420 0x7fd3880039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3e00640 1 -- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4160420 msgr2=0x7fd3880039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a9a00640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd388004d00 0x7fd388005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3e00640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4160420 0x7fd3880039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3e00640 1 -- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd388004d00 msgr2=0x7fd388005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:25.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3e00640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd388004d00 0x7fd388005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a3e00640 1 -- 172.21.15.62:0/1868454126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd388003520 con 0x7fd388003f20 2024-03-23T11:24:25.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a9000640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4160420 0x7fd3880039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:25.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.191+0000 7fd3a9a00640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd388004d00 0x7fd388005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:25.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.192+0000 7fd3a3e00640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd388003f20 0x7fd3880047c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009ac0 tx=0x7fd3880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1361cc4025ca2008 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:25.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.193+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c02c6b0 con 0x7fd388003f20 2024-03-23T11:24:25.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.193+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd38c02c810 con 0x7fd388003f20 2024-03-23T11:24:25.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.193+0000 7fd3a3400640 1 -- 172.21.15.62:0/1868454126 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c02cba0 con 0x7fd388003f20 2024-03-23T11:24:25.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.193+0000 7fd3aa400640 1 -- 172.21.15.62:0/1868454126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd388003f20 msgr2=0x7fd3880047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.193+0000 7fd3aa400640 1 --2- 172.21.15.62:0/1868454126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd388003f20 0x7fd3880047c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009ac0 tx=0x7fd3880057f0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.193+0000 7fd3a9a00640 1 -- 172.21.15.62:0/1868454126 reap_dead start 2024-03-23T11:24:25.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.193+0000 7fd3aa400640 1 -- 172.21.15.62:0/1868454126 shutdown_connections 2024-03-23T11:24:25.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.193+0000 7fd3aa400640 1 -- 172.21.15.62:0/1868454126 >> 172.21.15.62:0/1868454126 conn(0x7fd3a4089f60 msgr2=0x7fd3a40d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:25.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.193+0000 7fd3aa400640 1 -- 172.21.15.62:0/1868454126 shutdown_connections 2024-03-23T11:24:25.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.194+0000 7fd3aa400640 1 -- 172.21.15.62:0/1868454126 wait complete. 2024-03-23T11:24:25.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.195+0000 7fd3aa400640 1 Processor -- start 2024-03-23T11:24:25.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.195+0000 7fd3aa400640 1 -- start start 2024-03-23T11:24:25.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.196+0000 7fd3aa400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd388003f20 0x7fd3a41478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.196+0000 7fd3aa400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4155bf0 0x7fd3a41584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.196+0000 7fd3aa400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a4156070 0x7fd3a4158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.196+0000 7fd3aa400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a4156640 con 0x7fd3a4156070 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.196+0000 7fd3aa400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a4156920 con 0x7fd388003f20 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.196+0000 7fd3aa400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a4156c00 con 0x7fd3a4155bf0 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.197+0000 7fd3a9a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a4156070 0x7fd3a4158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.197+0000 7fd3a9a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a4156070 0x7fd3a4158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53220/0 (socket says 172.21.15.62:53220) 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.197+0000 7fd3a9a00640 1 -- 172.21.15.62:0/3325999875 learned_addr learned my addr 172.21.15.62:0/3325999875 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.197+0000 7fd3a3e00640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4155bf0 0x7fd3a41584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.197+0000 7fd3a9000640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd388003f20 0x7fd3a41478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.197+0000 7fd3a9a00640 1 -- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4155bf0 msgr2=0x7fd3a41584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.197+0000 7fd3a9a00640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4155bf0 0x7fd3a41584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.197+0000 7fd3a9a00640 1 -- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd388003f20 msgr2=0x7fd3a41478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.197+0000 7fd3a9a00640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd388003f20 0x7fd3a41478a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.197+0000 7fd3a9a00640 1 -- 172.21.15.62:0/3325999875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a4156ee0 con 0x7fd3a4156070 2024-03-23T11:24:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.197+0000 7fd3a9a00640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a4156070 0x7fd3a4158a10 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd3a408aa80 tx=0x7fd39c00b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.198+0000 7fd3a0c00640 1 -- 172.21.15.62:0/3325999875 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c00b8d0 con 0x7fd3a4156070 2024-03-23T11:24:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.198+0000 7fd3a0c00640 1 -- 172.21.15.62:0/3325999875 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c00ba50 con 0x7fd3a4156070 2024-03-23T11:24:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.198+0000 7fd3a0c00640 1 -- 172.21.15.62:0/3325999875 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c00be00 con 0x7fd3a4156070 2024-03-23T11:24:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.198+0000 7fd3aa400640 1 -- 172.21.15.62:0/3325999875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a41571c0 con 0x7fd3a4156070 2024-03-23T11:24:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.198+0000 7fd3aa400640 1 -- 172.21.15.62:0/3325999875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a4147de0 con 0x7fd3a4156070 2024-03-23T11:24:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.199+0000 7fd3aa400640 1 -- 172.21.15.62:0/3325999875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a4066bb0 con 0x7fd3a4156070 2024-03-23T11:24:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.200+0000 7fd3a0c00640 1 -- 172.21.15.62:0/3325999875 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd39c007070 con 0x7fd3a4156070 2024-03-23T11:24:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.201+0000 7fd3a0c00640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd38006a7a0 0x7fd38006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.201+0000 7fd3a0c00640 1 -- 172.21.15.62:0/3325999875 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd39c088b80 con 0x7fd3a4156070 2024-03-23T11:24:25.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.202+0000 7fd3a9000640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd38006a7a0 0x7fd38006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.203+0000 7fd3a9000640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd38006a7a0 0x7fd38006cc60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd394001630 tx=0x7fd394009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:25.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.206+0000 7fd3a0c00640 1 -- 172.21.15.62:0/3325999875 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd39c053f00 con 0x7fd3a4156070 2024-03-23T11:24:25.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.445+0000 7fd3aa400640 1 -- 172.21.15.62:0/3325999875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd3a40cc570 con 0x7fd3a4156070 2024-03-23T11:24:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.446+0000 7fd3a0c00640 1 -- 172.21.15.62:0/3325999875 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd39c088e60 con 0x7fd3a4156070 2024-03-23T11:24:25.441 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:24:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 -- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd38006a7a0 msgr2=0x7fd38006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd38006a7a0 0x7fd38006cc60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd394001630 tx=0x7fd394009040 comp rx=0 tx=0).stop 2024-03-23T11:24:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 -- 172.21.15.62:0/3325999875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a4156070 msgr2=0x7fd3a4158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a4156070 0x7fd3a4158a10 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd3a408aa80 tx=0x7fd39c00b0d0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 -- 172.21.15.62:0/3325999875 shutdown_connections 2024-03-23T11:24:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd38006a7a0 0x7fd38006cc60 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3a4156070 0x7fd3a4158a10 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a4155bf0 0x7fd3a41584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 --2- 172.21.15.62:0/3325999875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd388003f20 0x7fd3a41478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 -- 172.21.15.62:0/3325999875 >> 172.21.15.62:0/3325999875 conn(0x7fd3a4089f60 msgr2=0x7fd3a40cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 -- 172.21.15.62:0/3325999875 shutdown_connections 2024-03-23T11:24:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.449+0000 7fd3aa400640 1 -- 172.21.15.62:0/3325999875 wait complete. 2024-03-23T11:24:25.468 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:24:25.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.687+0000 7f0fa4a00640 1 Processor -- start 2024-03-23T11:24:25.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.687+0000 7f0fa4a00640 1 -- start start 2024-03-23T11:24:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.689+0000 7f0fa4a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fa006c300 0x7f0fa00ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.689+0000 7f0fa4a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0fa00cafa0 0x7f0fa00cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.689+0000 7f0fa4a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa00cbc60 con 0x7f0fa006c6e0 2024-03-23T11:24:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.689+0000 7f0fa4a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa00cbf40 con 0x7f0fa00cafa0 2024-03-23T11:24:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.689+0000 7f0fa4a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00cc220 con 0x7f0fa006c300 2024-03-23T11:24:25.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.690+0000 7f0f95400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0fa00cafa0 0x7f0fa00cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.690+0000 7f0f9de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fa006c300 0x7f0fa00ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.690+0000 7f0f95400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0fa00cafa0 0x7f0fa00cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:56112/0 (socket says 172.21.15.62:56112) 2024-03-23T11:24:25.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.690+0000 7f0f95400640 1 -- 172.21.15.62:0/2994631308 learned_addr learned my addr 172.21.15.62:0/2994631308 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:25.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.691+0000 7f0f95400640 1 -- 172.21.15.62:0/2994631308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fa006c300 msgr2=0x7f0fa00ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:25.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.691+0000 7f0f95400640 1 --2- 172.21.15.62:0/2994631308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fa006c300 0x7f0fa00ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.691+0000 7f0f95400640 1 -- 172.21.15.62:0/2994631308 >> v1:172.21.15.120:6789/0 conn(0x7f0fa006c6e0 legacy=0x7f0fa006cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.691+0000 7f0f95400640 1 -- 172.21.15.62:0/2994631308 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa00cc500 con 0x7f0fa00cafa0 2024-03-23T11:24:25.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.692+0000 7f0f95400640 1 --2- 172.21.15.62:0/2994631308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fa00cafa0 0x7f0fa00cb400 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009870 tx=0x7f0f8802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abc2a3713fe8cf09 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:25.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.692+0000 7f0f9d400640 1 -- 172.21.15.62:0/2994631308 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88031640 con 0x7f0fa00cafa0 2024-03-23T11:24:25.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.692+0000 7f0f9d400640 1 -- 172.21.15.62:0/2994631308 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f880317a0 con 0x7f0fa00cafa0 2024-03-23T11:24:25.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.692+0000 7f0f9d400640 1 -- 172.21.15.62:0/2994631308 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88031b30 con 0x7f0fa00cafa0 2024-03-23T11:24:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.693+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2994631308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fa00cafa0 msgr2=0x7f0fa00cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.693+0000 7f0fa4a00640 1 --2- 172.21.15.62:0/2994631308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fa00cafa0 0x7f0fa00cb400 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009870 tx=0x7f0f8802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.693+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2994631308 shutdown_connections 2024-03-23T11:24:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.693+0000 7f0fa4a00640 1 --2- 172.21.15.62:0/2994631308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fa00cafa0 0x7f0fa00cb400 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.693+0000 7f0fa4a00640 1 --2- 172.21.15.62:0/2994631308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fa006c300 0x7f0fa00ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.693+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2994631308 >> 172.21.15.62:0/2994631308 conn(0x7f0fa00b02e0 msgr2=0x7f0fa00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.693+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2994631308 shutdown_connections 2024-03-23T11:24:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.693+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2994631308 wait complete. 2024-03-23T11:24:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.695+0000 7f0fa4a00640 1 Processor -- start 2024-03-23T11:24:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.695+0000 7f0fa4a00640 1 -- start start 2024-03-23T11:24:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.696+0000 7f0fa4a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fa006c300 0x7f0fa015bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.696+0000 7f0fa4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa006c6e0 0x7f0fa00ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.696+0000 7f0fa4a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fa00cafa0 0x7f0fa013ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.696+0000 7f0fa4a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00ac080 con 0x7f0fa006c6e0 2024-03-23T11:24:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.696+0000 7f0fa4a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa015d200 con 0x7f0fa00cafa0 2024-03-23T11:24:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.696+0000 7f0fa4a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00ade80 con 0x7f0fa006c300 2024-03-23T11:24:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.696+0000 7f0f95400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa006c6e0 0x7f0fa00ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.696+0000 7f0f95400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa006c6e0 0x7f0fa00ad940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53232/0 (socket says 172.21.15.62:53232) 2024-03-23T11:24:25.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.696+0000 7f0f95400640 1 -- 172.21.15.62:0/2201203468 learned_addr learned my addr 172.21.15.62:0/2201203468 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f9de00640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fa006c300 0x7f0fa015bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f9e800640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fa00cafa0 0x7f0fa013ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f95400640 1 -- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fa006c300 msgr2=0x7f0fa015bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f95400640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fa006c300 0x7f0fa015bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f95400640 1 -- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fa00cafa0 msgr2=0x7f0fa013ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f95400640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fa00cafa0 0x7f0fa013ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f95400640 1 -- 172.21.15.62:0/2201203468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f880094d0 con 0x7f0fa006c6e0 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f9e800640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fa00cafa0 0x7f0fa013ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f95400640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa006c6e0 0x7f0fa00ad940 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f0f88000c00 tx=0x7f0f880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f96a00640 1 -- 172.21.15.62:0/2201203468 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8803a930 con 0x7f0fa006c6e0 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f96a00640 1 -- 172.21.15.62:0/2201203468 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8803aa90 con 0x7f0fa006c6e0 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0f96a00640 1 -- 172.21.15.62:0/2201203468 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8803ae40 con 0x7f0fa006c6e0 2024-03-23T11:24:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.697+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2201203468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa015c5f0 con 0x7f0fa006c6e0 2024-03-23T11:24:25.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.698+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2201203468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa015cbb0 con 0x7f0fa006c6e0 2024-03-23T11:24:25.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.699+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2201203468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa0066c90 con 0x7f0fa006c6e0 2024-03-23T11:24:25.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.699+0000 7f0f96a00640 1 -- 172.21.15.62:0/2201203468 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0f8802f050 con 0x7f0fa006c6e0 2024-03-23T11:24:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.700+0000 7f0f96a00640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7c06a7a0 0x7f0f7c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.700+0000 7f0f96a00640 1 -- 172.21.15.62:0/2201203468 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f880aab40 con 0x7f0fa006c6e0 2024-03-23T11:24:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.701+0000 7f0f9de00640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7c06a7a0 0x7f0f7c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.702+0000 7f0f9de00640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7c06a7a0 0x7f0f7c06cc60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009ad0 tx=0x7f0f90008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:25.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.705+0000 7f0f96a00640 1 -- 172.21.15.62:0/2201203468 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f88075f50 con 0x7f0fa006c6e0 2024-03-23T11:24:25.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.901+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2201203468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0fa015a870 con 0x7f0fa006c6e0 2024-03-23T11:24:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.902+0000 7f0f96a00640 1 -- 172.21.15.62:0/2201203468 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v74) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0f88035020 con 0x7f0fa006c6e0 2024-03-23T11:24:25.897 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:24:25.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.904+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7c06a7a0 msgr2=0x7f0f7c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.904+0000 7f0fa4a00640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7c06a7a0 0x7f0f7c06cc60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009ad0 tx=0x7f0f90008040 comp rx=0 tx=0).stop 2024-03-23T11:24:25.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.904+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2201203468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa006c6e0 msgr2=0x7f0fa00ad940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:25.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.904+0000 7f0fa4a00640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa006c6e0 0x7f0fa00ad940 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f0f88000c00 tx=0x7f0f880046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.904+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2201203468 shutdown_connections 2024-03-23T11:24:25.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.904+0000 7f0fa4a00640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7c06a7a0 0x7f0f7c06cc60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.904+0000 7f0fa4a00640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fa006c6e0 0x7f0fa00ad940 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.904+0000 7f0fa4a00640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0fa00cafa0 0x7f0fa013ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.904+0000 7f0fa4a00640 1 --2- 172.21.15.62:0/2201203468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0fa006c300 0x7f0fa015bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:25.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.904+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2201203468 >> 172.21.15.62:0/2201203468 conn(0x7f0fa00b02e0 msgr2=0x7f0fa013eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:25.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.905+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2201203468 shutdown_connections 2024-03-23T11:24:25.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:25.905+0000 7f0fa4a00640 1 -- 172.21.15.62:0/2201203468 wait complete. 2024-03-23T11:24:25.920 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:24:26.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.150+0000 7fc139800640 1 Processor -- start 2024-03-23T11:24:26.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.150+0000 7fc139800640 1 -- start start 2024-03-23T11:24:26.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.151+0000 7fc139800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc13415fcf0 0x7fc1340d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.151+0000 7fc139800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc134160420 0x7fc1341608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.151+0000 7fc139800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1340d3540 con 0x7fc1340d06d0 2024-03-23T11:24:26.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.151+0000 7fc139800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc134160df0 con 0x7fc134160420 2024-03-23T11:24:26.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.151+0000 7fc139800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1340ce510 con 0x7fc13415fcf0 2024-03-23T11:24:26.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.151+0000 7fc133400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc134160420 0x7fc1341608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:26.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.151+0000 7fc133400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc134160420 0x7fc1341608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53242/0 (socket says 172.21.15.62:53242) 2024-03-23T11:24:26.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.151+0000 7fc133400640 1 -- 172.21.15.62:0/891731051 learned_addr learned my addr 172.21.15.62:0/891731051 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:26.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.152+0000 7fc133e00640 1 --2- 172.21.15.62:0/891731051 >> v2:172.21.15.120:3300/0 conn(0x7fc13415fcf0 0x7fc1340d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:26.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.152+0000 7fc133400640 1 -- 172.21.15.62:0/891731051 >> v2:172.21.15.120:3300/0 conn(0x7fc13415fcf0 msgr2=0x7fc1340d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:26.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.152+0000 7fc133400640 1 --2- 172.21.15.62:0/891731051 >> v2:172.21.15.120:3300/0 conn(0x7fc13415fcf0 0x7fc1340d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.152+0000 7fc133400640 1 -- 172.21.15.62:0/891731051 >> v1:172.21.15.120:6789/0 conn(0x7fc1340d06d0 legacy=0x7fc1340d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:26.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.152+0000 7fc133400640 1 -- 172.21.15.62:0/891731051 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1340cdc70 con 0x7fc134160420 2024-03-23T11:24:26.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.153+0000 7fc133400640 1 --2- 172.21.15.62:0/891731051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc134160420 0x7fc1341608b0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fc134066480 tx=0x7fc11c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b4cffc02b37054 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:26.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.153+0000 7fc132a00640 1 -- 172.21.15.62:0/891731051 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c02c9f0 con 0x7fc134160420 2024-03-23T11:24:26.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.153+0000 7fc132a00640 1 -- 172.21.15.62:0/891731051 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c02cb50 con 0x7fc134160420 2024-03-23T11:24:26.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.153+0000 7fc132a00640 1 -- 172.21.15.62:0/891731051 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c0343f0 con 0x7fc134160420 2024-03-23T11:24:26.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.154+0000 7fc139800640 1 -- 172.21.15.62:0/891731051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc134160420 msgr2=0x7fc1341608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:26.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.154+0000 7fc139800640 1 --2- 172.21.15.62:0/891731051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc134160420 0x7fc1341608b0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fc134066480 tx=0x7fc11c02be50 comp rx=0 tx=0).stop 2024-03-23T11:24:26.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.154+0000 7fc139800640 1 -- 172.21.15.62:0/891731051 shutdown_connections 2024-03-23T11:24:26.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.154+0000 7fc139800640 1 --2- 172.21.15.62:0/891731051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc134160420 0x7fc1341608b0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.154+0000 7fc139800640 1 --2- 172.21.15.62:0/891731051 >> v2:172.21.15.120:3300/0 conn(0x7fc13415fcf0 0x7fc1340d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.154+0000 7fc139800640 1 -- 172.21.15.62:0/891731051 >> 172.21.15.62:0/891731051 conn(0x7fc134089f60 msgr2=0x7fc1340d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:26.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.154+0000 7fc139800640 1 -- 172.21.15.62:0/891731051 shutdown_connections 2024-03-23T11:24:26.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.154+0000 7fc139800640 1 -- 172.21.15.62:0/891731051 wait complete. 2024-03-23T11:24:26.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.156+0000 7fc139800640 1 Processor -- start 2024-03-23T11:24:26.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.156+0000 7fc139800640 1 -- start start 2024-03-23T11:24:26.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc139800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1340d06d0 0x7fc1341f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc139800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13415fcf0 0x7fc1340b7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc139800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc134160420 0x7fc1340c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc139800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13409f9a0 con 0x7fc1340d06d0 2024-03-23T11:24:26.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc139800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13409c900 con 0x7fc134160420 2024-03-23T11:24:26.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc139800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13409f3e0 con 0x7fc13415fcf0 2024-03-23T11:24:26.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc133e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1340d06d0 0x7fc1341f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:26.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc133e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1340d06d0 0x7fc1341f1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53246/0 (socket says 172.21.15.62:53246) 2024-03-23T11:24:26.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc133e00640 1 -- 172.21.15.62:0/2795428236 learned_addr learned my addr 172.21.15.62:0/2795428236 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:26.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc133400640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13415fcf0 0x7fc1340b7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:26.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc138e00640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc134160420 0x7fc1340c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:26.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc133e00640 1 -- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13415fcf0 msgr2=0x7fc1340b7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:26.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc133e00640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13415fcf0 0x7fc1340b7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc133e00640 1 -- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc134160420 msgr2=0x7fc1340c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:26.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc133e00640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc134160420 0x7fc1340c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.157+0000 7fc133e00640 1 -- 172.21.15.62:0/2795428236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c009b30 con 0x7fc1340d06d0 2024-03-23T11:24:26.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.158+0000 7fc138e00640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc134160420 0x7fc1340c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:26.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.158+0000 7fc133e00640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1340d06d0 0x7fc1341f1d80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc12400d830 tx=0x7fc12400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:26.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.158+0000 7fc12be00640 1 -- 172.21.15.62:0/2795428236 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12400b660 con 0x7fc1340d06d0 2024-03-23T11:24:26.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.158+0000 7fc12be00640 1 -- 172.21.15.62:0/2795428236 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12400b7c0 con 0x7fc1340d06d0 2024-03-23T11:24:26.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.158+0000 7fc12be00640 1 -- 172.21.15.62:0/2795428236 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12400bb50 con 0x7fc1340d06d0 2024-03-23T11:24:26.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.158+0000 7fc139800640 1 -- 172.21.15.62:0/2795428236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13409f6c0 con 0x7fc1340d06d0 2024-03-23T11:24:26.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.158+0000 7fc139800640 1 -- 172.21.15.62:0/2795428236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1340b8510 con 0x7fc1340d06d0 2024-03-23T11:24:26.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.159+0000 7fc139800640 1 -- 172.21.15.62:0/2795428236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1340b5590 con 0x7fc1340d06d0 2024-03-23T11:24:26.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.160+0000 7fc12be00640 1 -- 172.21.15.62:0/2795428236 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc12401e460 con 0x7fc1340d06d0 2024-03-23T11:24:26.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.161+0000 7fc12be00640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc11006a7a0 0x7fc11006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.161+0000 7fc12be00640 1 -- 172.21.15.62:0/2795428236 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc124089570 con 0x7fc1340d06d0 2024-03-23T11:24:26.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.162+0000 7fc133400640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc11006a7a0 0x7fc11006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:26.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.163+0000 7fc133400640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc11006a7a0 0x7fc11006cc60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc11c035000 tx=0x7fc11c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:26.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.166+0000 7fc12be00640 1 -- 172.21.15.62:0/2795428236 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc124054980 con 0x7fc1340d06d0 2024-03-23T11:24:26.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.356+0000 7fc139800640 1 -- 172.21.15.62:0/2795428236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc134066bb0 con 0x7fc1340d06d0 2024-03-23T11:24:26.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.357+0000 7fc12be00640 1 -- 172.21.15.62:0/2795428236 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v74) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc124057fa0 con 0x7fc1340d06d0 2024-03-23T11:24:26.353 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:24:26.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 -- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc11006a7a0 msgr2=0x7fc11006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:26.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc11006a7a0 0x7fc11006cc60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc11c035000 tx=0x7fc11c030000 comp rx=0 tx=0).stop 2024-03-23T11:24:26.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 -- 172.21.15.62:0/2795428236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1340d06d0 msgr2=0x7fc1341f1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:26.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1340d06d0 0x7fc1341f1d80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc12400d830 tx=0x7fc12400dcf0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 -- 172.21.15.62:0/2795428236 shutdown_connections 2024-03-23T11:24:26.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc11006a7a0 0x7fc11006cc60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc134160420 0x7fc1340c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc13415fcf0 0x7fc1340b7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 --2- 172.21.15.62:0/2795428236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1340d06d0 0x7fc1341f1d80 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 -- 172.21.15.62:0/2795428236 >> 172.21.15.62:0/2795428236 conn(0x7fc134089f60 msgr2=0x7fc1340b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:26.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 -- 172.21.15.62:0/2795428236 shutdown_connections 2024-03-23T11:24:26.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.360+0000 7fc139800640 1 -- 172.21.15.62:0/2795428236 wait complete. 2024-03-23T11:24:26.372 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:24:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.592+0000 7f6da0e00640 1 Processor -- start 2024-03-23T11:24:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.592+0000 7f6da0e00640 1 -- start start 2024-03-23T11:24:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.594+0000 7f6da0e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d9c15ac50 0x7f6d9c0989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.594+0000 7f6da0e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6d9c098f10 0x7f6d9c0b1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.594+0000 7f6da0e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c099360 con 0x7f6d9c15a870 2024-03-23T11:24:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.594+0000 7f6da0e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d9c0cf320 con 0x7f6d9c15ac50 2024-03-23T11:24:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.594+0000 7f6da0e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d9c0cf7b0 con 0x7f6d9c098f10 2024-03-23T11:24:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.594+0000 7f6d91400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d9c15ac50 0x7f6d9c0989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.594+0000 7f6d91400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d9c15ac50 0x7f6d9c0989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53260/0 (socket says 172.21.15.62:53260) 2024-03-23T11:24:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.594+0000 7f6d91400640 1 -- 172.21.15.62:0/2062008446 learned_addr learned my addr 172.21.15.62:0/2062008446 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.595+0000 7f6d91400640 1 -- 172.21.15.62:0/2062008446 >> v2:172.21.15.120:3300/0 conn(0x7f6d9c098f10 msgr2=0x7f6d9c0b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.595+0000 7f6d9a800640 1 --2- 172.21.15.62:0/2062008446 >> v2:172.21.15.120:3300/0 conn(0x7f6d9c098f10 0x7f6d9c0b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:26.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.595+0000 7f6d91400640 1 --2- 172.21.15.62:0/2062008446 >> v2:172.21.15.120:3300/0 conn(0x7f6d9c098f10 0x7f6d9c0b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.595+0000 7f6d91400640 1 -- 172.21.15.62:0/2062008446 >> v1:172.21.15.62:6789/0 conn(0x7f6d9c15a870 legacy=0x7f6d9c0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:26.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.595+0000 7f6d91400640 1 -- 172.21.15.62:0/2062008446 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c0ce710 con 0x7f6d9c15ac50 2024-03-23T11:24:26.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.595+0000 7f6d9a800640 1 --2- 172.21.15.62:0/2062008446 >> v2:172.21.15.120:3300/0 conn(0x7f6d9c098f10 0x7f6d9c0b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:26.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.595+0000 7f6d99400640 1 -- 172.21.15.62:0/2062008446 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098550432 0 0) 0x7f6d8c001490 con 0x7f6d9c15a870 2024-03-23T11:24:26.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.596+0000 7f6d91400640 1 --2- 172.21.15.62:0/2062008446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d9c15ac50 0x7f6d9c0989d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c071ae0 tx=0x7f6d8402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5324084f7b31e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:26.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.596+0000 7f6d99400640 1 -- 172.21.15.62:0/2062008446 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84030870 con 0x7f6d9c15ac50 2024-03-23T11:24:26.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.596+0000 7f6d99400640 1 -- 172.21.15.62:0/2062008446 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d840309d0 con 0x7f6d9c15ac50 2024-03-23T11:24:26.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.596+0000 7f6d99400640 1 -- 172.21.15.62:0/2062008446 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84030d60 con 0x7f6d9c15ac50 2024-03-23T11:24:26.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.597+0000 7f6da0e00640 1 -- 172.21.15.62:0/2062008446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d9c15ac50 msgr2=0x7f6d9c0989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:26.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.597+0000 7f6da0e00640 1 --2- 172.21.15.62:0/2062008446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d9c15ac50 0x7f6d9c0989d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c071ae0 tx=0x7f6d8402b9b0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.597+0000 7f6da0e00640 1 -- 172.21.15.62:0/2062008446 shutdown_connections 2024-03-23T11:24:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.597+0000 7f6da0e00640 1 --2- 172.21.15.62:0/2062008446 >> v2:172.21.15.120:3300/0 conn(0x7f6d9c098f10 0x7f6d9c0b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.597+0000 7f6da0e00640 1 --2- 172.21.15.62:0/2062008446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d9c15ac50 0x7f6d9c0989d0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.597+0000 7f6da0e00640 1 -- 172.21.15.62:0/2062008446 >> 172.21.15.62:0/2062008446 conn(0x7f6d9c0b2270 msgr2=0x7f6d9c0b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.597+0000 7f6da0e00640 1 -- 172.21.15.62:0/2062008446 shutdown_connections 2024-03-23T11:24:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.597+0000 7f6da0e00640 1 -- 172.21.15.62:0/2062008446 wait complete. 2024-03-23T11:24:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.598+0000 7f6da0e00640 1 Processor -- start 2024-03-23T11:24:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.599+0000 7f6da0e00640 1 -- start start 2024-03-23T11:24:26.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.599+0000 7f6da0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d9c098f10 0x7f6d9c0c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.599+0000 7f6da0e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d9c15a870 0x7f6d9c09d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6da0e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d9c15ac50 0x7f6d9c09a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6da0e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0c6440 con 0x7f6d9c098f10 2024-03-23T11:24:26.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6da0e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c09c050 con 0x7f6d9c15a870 2024-03-23T11:24:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6da0e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0c57c0 con 0x7f6d9c15ac50 2024-03-23T11:24:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6d99e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d9c098f10 0x7f6d9c0c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6d99e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d9c098f10 0x7f6d9c0c6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53266/0 (socket says 172.21.15.62:53266) 2024-03-23T11:24:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6d99e00640 1 -- 172.21.15.62:0/1449718630 learned_addr learned my addr 172.21.15.62:0/1449718630 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6d99e00640 1 -- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d9c15ac50 msgr2=0x7f6d9c09a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6d99e00640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d9c15ac50 0x7f6d9c09a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6d99e00640 1 -- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d9c15a870 msgr2=0x7f6d9c09d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6d99e00640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d9c15a870 0x7f6d9c09d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6d99e00640 1 -- 172.21.15.62:0/1449718630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d840094d0 con 0x7f6d9c098f10 2024-03-23T11:24:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.600+0000 7f6d99e00640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d9c098f10 0x7f6d9c0c6860 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00c890 tx=0x7f6d8c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.601+0000 7f6d92a00640 1 -- 172.21.15.62:0/1449718630 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c003da0 con 0x7f6d9c098f10 2024-03-23T11:24:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.601+0000 7f6d92a00640 1 -- 172.21.15.62:0/1449718630 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8c005070 con 0x7f6d9c098f10 2024-03-23T11:24:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.601+0000 7f6d92a00640 1 -- 172.21.15.62:0/1449718630 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c0054a0 con 0x7f6d9c098f10 2024-03-23T11:24:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.601+0000 7f6da0e00640 1 -- 172.21.15.62:0/1449718630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c0c5aa0 con 0x7f6d9c098f10 2024-03-23T11:24:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.601+0000 7f6da0e00640 1 -- 172.21.15.62:0/1449718630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c0c6060 con 0x7f6d9c098f10 2024-03-23T11:24:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.602+0000 7f6da0e00640 1 -- 172.21.15.62:0/1449718630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c0c6580 con 0x7f6d9c098f10 2024-03-23T11:24:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.603+0000 7f6d92a00640 1 -- 172.21.15.62:0/1449718630 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6d8c005600 con 0x7f6d9c098f10 2024-03-23T11:24:26.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.604+0000 7f6d92a00640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d7806a7a0 0x7f6d7806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:26.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.604+0000 7f6d92a00640 1 -- 172.21.15.62:0/1449718630 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d8c0885a0 con 0x7f6d9c098f10 2024-03-23T11:24:26.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.605+0000 7f6d91400640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d7806a7a0 0x7f6d7806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.606+0000 7f6d91400640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d7806a7a0 0x7f6d7806cc60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6d84036000 tx=0x7f6d8403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:26.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.608+0000 7f6d92a00640 1 -- 172.21.15.62:0/1449718630 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6d9c0c6580 con 0x7f6d9c098f10 2024-03-23T11:24:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.819+0000 7f6da0e00640 1 -- 172.21.15.62:0/1449718630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6d9c099360 con 0x7f6d9c098f10 2024-03-23T11:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.820+0000 7f6d92a00640 1 -- 172.21.15.62:0/1449718630 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v74) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6d9c099360 con 0x7f6d9c098f10 2024-03-23T11:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:24:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.822+0000 7f6da0e00640 1 -- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d7806a7a0 msgr2=0x7f6d7806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.822+0000 7f6da0e00640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d7806a7a0 0x7f6d7806cc60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6d84036000 tx=0x7f6d8403b040 comp rx=0 tx=0).stop 2024-03-23T11:24:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.822+0000 7f6da0e00640 1 -- 172.21.15.62:0/1449718630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d9c098f10 msgr2=0x7f6d9c0c6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.822+0000 7f6da0e00640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d9c098f10 0x7f6d9c0c6860 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00c890 tx=0x7f6d8c00cd50 comp rx=0 tx=0).stop 2024-03-23T11:24:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.823+0000 7f6da0e00640 1 -- 172.21.15.62:0/1449718630 shutdown_connections 2024-03-23T11:24:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.823+0000 7f6da0e00640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d7806a7a0 0x7f6d7806cc60 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.823+0000 7f6da0e00640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d9c15ac50 0x7f6d9c09a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.823+0000 7f6da0e00640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d9c15a870 0x7f6d9c09d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.823+0000 7f6da0e00640 1 --2- 172.21.15.62:0/1449718630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d9c098f10 0x7f6d9c0c6860 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.823+0000 7f6da0e00640 1 -- 172.21.15.62:0/1449718630 >> 172.21.15.62:0/1449718630 conn(0x7f6d9c0b2270 msgr2=0x7f6d9c06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:26.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.823+0000 7f6da0e00640 1 -- 172.21.15.62:0/1449718630 shutdown_connections 2024-03-23T11:24:26.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:26.823+0000 7f6da0e00640 1 -- 172.21.15.62:0/1449718630 wait complete. 2024-03-23T11:24:26.837 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:24:26.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:26.837 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:26.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[60236]: fuse finished with error 0 and tester_r 0 2024-03-23T11:24:26.932 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:24:33.034 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:24:33.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:33.034 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:24:33.095 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:24:33.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.328+0000 7f4deae00640 1 Processor -- start 2024-03-23T11:24:33.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.328+0000 7f4deae00640 1 -- start start 2024-03-23T11:24:33.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.330+0000 7f4deae00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4de4070090 0x7f4de4070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:33.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.330+0000 7f4deae00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4de40b2820 0x7f4de40b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:33.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.330+0000 7f4deae00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de40b31b0 con 0x7f4de4143dd0 2024-03-23T11:24:33.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.330+0000 7f4deae00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de4070a60 con 0x7f4de40b2820 2024-03-23T11:24:33.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.330+0000 7f4deae00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de40cd140 con 0x7f4de4070090 2024-03-23T11:24:33.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.330+0000 7f4dea400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4de40b2820 0x7f4de40b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:33.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.330+0000 7f4dea400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4de40b2820 0x7f4de40b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51036/0 (socket says 172.21.15.62:51036) 2024-03-23T11:24:33.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.330+0000 7f4dea400640 1 -- 172.21.15.62:0/4292226561 learned_addr learned my addr 172.21.15.62:0/4292226561 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:33.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.331+0000 7f4de9000640 1 --2- 172.21.15.62:0/4292226561 >> v2:172.21.15.120:3300/0 conn(0x7f4de4070090 0x7f4de4070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:33.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.331+0000 7f4dea400640 1 -- 172.21.15.62:0/4292226561 >> v2:172.21.15.120:3300/0 conn(0x7f4de4070090 msgr2=0x7f4de4070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:33.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.331+0000 7f4dea400640 1 --2- 172.21.15.62:0/4292226561 >> v2:172.21.15.120:3300/0 conn(0x7f4de4070090 0x7f4de4070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:33.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.331+0000 7f4dea400640 1 -- 172.21.15.62:0/4292226561 >> v1:172.21.15.120:6789/0 conn(0x7f4de4143dd0 legacy=0x7f4de415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:33.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.331+0000 7f4dea400640 1 -- 172.21.15.62:0/4292226561 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40cd3e0 con 0x7f4de40b2820 2024-03-23T11:24:33.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.332+0000 7f4dea400640 1 --2- 172.21.15.62:0/4292226561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de40b2820 0x7f4de40b2c70 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f4de4071530 tx=0x7f4ddc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913831f7bbd2ba31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:33.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.333+0000 7f4de3e00640 1 -- 172.21.15.62:0/4292226561 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc02cb20 con 0x7f4de40b2820 2024-03-23T11:24:33.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.333+0000 7f4de3e00640 1 -- 172.21.15.62:0/4292226561 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc02cc80 con 0x7f4de40b2820 2024-03-23T11:24:33.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.333+0000 7f4de3e00640 1 -- 172.21.15.62:0/4292226561 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc0343f0 con 0x7f4de40b2820 2024-03-23T11:24:33.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.333+0000 7f4deae00640 1 -- 172.21.15.62:0/4292226561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de40b2820 msgr2=0x7f4de40b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:33.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.333+0000 7f4deae00640 1 --2- 172.21.15.62:0/4292226561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de40b2820 0x7f4de40b2c70 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f4de4071530 tx=0x7f4ddc02bf80 comp rx=0 tx=0).stop 2024-03-23T11:24:33.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.333+0000 7f4deae00640 1 -- 172.21.15.62:0/4292226561 shutdown_connections 2024-03-23T11:24:33.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.333+0000 7f4deae00640 1 --2- 172.21.15.62:0/4292226561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de40b2820 0x7f4de40b2c70 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:33.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.333+0000 7f4deae00640 1 --2- 172.21.15.62:0/4292226561 >> v2:172.21.15.120:3300/0 conn(0x7f4de4070090 0x7f4de4070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:33.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.333+0000 7f4deae00640 1 -- 172.21.15.62:0/4292226561 >> 172.21.15.62:0/4292226561 conn(0x7f4de4066050 msgr2=0x7f4de4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:33.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.333+0000 7f4deae00640 1 -- 172.21.15.62:0/4292226561 shutdown_connections 2024-03-23T11:24:33.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.333+0000 7f4deae00640 1 -- 172.21.15.62:0/4292226561 wait complete. 2024-03-23T11:24:33.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.335+0000 7f4deae00640 1 Processor -- start 2024-03-23T11:24:33.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.335+0000 7f4deae00640 1 -- start start 2024-03-23T11:24:33.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.336+0000 7f4deae00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de4070090 0x7f4de41e9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:33.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.336+0000 7f4deae00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de40b2820 0x7f4de412eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:33.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.336+0000 7f4deae00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de4143dd0 0x7f4de4130320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:33.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.336+0000 7f4deae00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de412ff00 con 0x7f4de4143dd0 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.336+0000 7f4deae00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de412efe0 con 0x7f4de40b2820 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.336+0000 7f4deae00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de412f280 con 0x7f4de4070090 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.337+0000 7f4dea400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de4143dd0 0x7f4de4130320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.337+0000 7f4dea400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de4143dd0 0x7f4de4130320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51038/0 (socket says 172.21.15.62:51038) 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.337+0000 7f4dea400640 1 -- 172.21.15.62:0/530054590 learned_addr learned my addr 172.21.15.62:0/530054590 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.337+0000 7f4de9a00640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de4070090 0x7f4de41e9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.337+0000 7f4de9000640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de40b2820 0x7f4de412eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.337+0000 7f4dea400640 1 -- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de4070090 msgr2=0x7f4de41e9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.337+0000 7f4dea400640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de4070090 0x7f4de41e9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.337+0000 7f4dea400640 1 -- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de40b2820 msgr2=0x7f4de412eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.337+0000 7f4dea400640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de40b2820 0x7f4de412eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.337+0000 7f4dea400640 1 -- 172.21.15.62:0/530054590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc009b30 con 0x7f4de4143dd0 2024-03-23T11:24:33.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.337+0000 7f4de9000640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de40b2820 0x7f4de412eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:33.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.338+0000 7f4dea400640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de4143dd0 0x7f4de4130320 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc036040 tx=0x7f4ddc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:33.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.338+0000 7f4de1600640 1 -- 172.21.15.62:0/530054590 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc009510 con 0x7f4de4143dd0 2024-03-23T11:24:33.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.338+0000 7f4de1600640 1 -- 172.21.15.62:0/530054590 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc02c920 con 0x7f4de4143dd0 2024-03-23T11:24:33.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.338+0000 7f4de1600640 1 -- 172.21.15.62:0/530054590 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc03e690 con 0x7f4de4143dd0 2024-03-23T11:24:33.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.338+0000 7f4deae00640 1 -- 172.21.15.62:0/530054590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de412f560 con 0x7f4de4143dd0 2024-03-23T11:24:33.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.338+0000 7f4deae00640 1 -- 172.21.15.62:0/530054590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de412fb20 con 0x7f4de4143dd0 2024-03-23T11:24:33.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.339+0000 7f4deae00640 1 -- 172.21.15.62:0/530054590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de4130040 con 0x7f4de4143dd0 2024-03-23T11:24:33.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.340+0000 7f4de1600640 1 -- 172.21.15.62:0/530054590 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4ddc03e7f0 con 0x7f4de4143dd0 2024-03-23T11:24:33.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.341+0000 7f4de1600640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4dc806a7d0 0x7f4dc806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:33.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.341+0000 7f4de1600640 1 -- 172.21.15.62:0/530054590 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ddc0ab890 con 0x7f4de4143dd0 2024-03-23T11:24:33.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.342+0000 7f4de9a00640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4dc806a7d0 0x7f4dc806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:33.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.343+0000 7f4de9a00640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4dc806a7d0 0x7f4dc806cc90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8000f60 tx=0x7f4dd8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:33.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.349+0000 7f4de1600640 1 -- 172.21.15.62:0/530054590 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ddc076ca0 con 0x7f4de4143dd0 2024-03-23T11:24:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.587+0000 7f4deae00640 1 -- 172.21.15.62:0/530054590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4de415ca80 con 0x7f4de4143dd0 2024-03-23T11:24:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.588+0000 7f4de1600640 1 -- 172.21.15.62:0/530054590 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f4ddc07a2c0 con 0x7f4de4143dd0 2024-03-23T11:24:33.585 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:33.585 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6618,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:24:14.062901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6621":{"gid":6621,"name":"d","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/3833782560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3833782560},{"type":"v1","addr":"172.21.15.120:6837","nonce":3833782560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:24:33.585 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 43 2024-03-23T11:24:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 -- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4dc806a7d0 msgr2=0x7f4dc806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4dc806a7d0 0x7f4dc806cc90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8000f60 tx=0x7f4dd8009040 comp rx=0 tx=0).stop 2024-03-23T11:24:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 -- 172.21.15.62:0/530054590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de4143dd0 msgr2=0x7f4de4130320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:33.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de4143dd0 0x7f4de4130320 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc036040 tx=0x7f4ddc004a40 comp rx=0 tx=0).stop 2024-03-23T11:24:33.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 -- 172.21.15.62:0/530054590 shutdown_connections 2024-03-23T11:24:33.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4dc806a7d0 0x7f4dc806cc90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:33.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4de4143dd0 0x7f4de4130320 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:33.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4de40b2820 0x7f4de412eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:33.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 --2- 172.21.15.62:0/530054590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4de4070090 0x7f4de41e9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:33.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 -- 172.21.15.62:0/530054590 >> 172.21.15.62:0/530054590 conn(0x7f4de4066050 msgr2=0x7f4de406ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:33.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 -- 172.21.15.62:0/530054590 shutdown_connections 2024-03-23T11:24:33.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:33.591+0000 7f4deae00640 1 -- 172.21.15.62:0/530054590 wait complete. 2024-03-23T11:24:33.607 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-03-23T11:24:33.863 INFO:teuthology.orchestra.run.smithi120.stdout:{"message":"","return_code":0} 2024-03-23T11:24:33.863 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-23T11:24:33.864 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:24:34.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.093+0000 7f0f94800640 1 Processor -- start 2024-03-23T11:24:34.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.093+0000 7f0f94800640 1 -- start start 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.094+0000 7f0f94800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0f8c0b2f90 0x7f0f8c161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.094+0000 7f0f94800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0f8c06c300 0x7f0f8c06c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.094+0000 7f0f94800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c0ae0c0 con 0x7f0f8c0b15e0 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.094+0000 7f0f94800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8c0b2740 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.094+0000 7f0f94800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8c1623b0 con 0x7f0f8c06c300 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.095+0000 7f0f91e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0f8c0b2f90 0x7f0f8c161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.095+0000 7f0f91e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0f8c0b2f90 0x7f0f8c161e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51046/0 (socket says 172.21.15.62:51046) 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.095+0000 7f0f91e00640 1 -- 172.21.15.62:0/3749673095 learned_addr learned my addr 172.21.15.62:0/3749673095 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.095+0000 7f0f92800640 1 --2- 172.21.15.62:0/3749673095 >> v2:172.21.15.120:3300/0 conn(0x7f0f8c06c300 0x7f0f8c06c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.095+0000 7f0f91e00640 1 -- 172.21.15.62:0/3749673095 >> v2:172.21.15.120:3300/0 conn(0x7f0f8c06c300 msgr2=0x7f0f8c06c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.095+0000 7f0f91e00640 1 --2- 172.21.15.62:0/3749673095 >> v2:172.21.15.120:3300/0 conn(0x7f0f8c06c300 0x7f0f8c06c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.095+0000 7f0f91e00640 1 -- 172.21.15.62:0/3749673095 >> v1:172.21.15.120:6789/0 conn(0x7f0f8c0b15e0 legacy=0x7f0f8c0b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:34.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.095+0000 7f0f91e00640 1 -- 172.21.15.62:0/3749673095 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c0b1100 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.096+0000 7f0f91e00640 1 --2- 172.21.15.62:0/3749673095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c0b2f90 0x7f0f8c161e70 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0ce2a0 tx=0x7f0f8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3084d86201216765 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:34.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.096+0000 7f0f90a00640 1 -- 172.21.15.62:0/3749673095 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f80031870 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.096+0000 7f0f90a00640 1 -- 172.21.15.62:0/3749673095 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f800319d0 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.096+0000 7f0f90a00640 1 -- 172.21.15.62:0/3749673095 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f80031d60 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.097+0000 7f0f94800640 1 -- 172.21.15.62:0/3749673095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c0b2f90 msgr2=0x7f0f8c161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.097+0000 7f0f94800640 1 --2- 172.21.15.62:0/3749673095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c0b2f90 0x7f0f8c161e70 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0ce2a0 tx=0x7f0f8002b910 comp rx=0 tx=0).stop 2024-03-23T11:24:34.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.097+0000 7f0f94800640 1 -- 172.21.15.62:0/3749673095 shutdown_connections 2024-03-23T11:24:34.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.097+0000 7f0f94800640 1 --2- 172.21.15.62:0/3749673095 >> v2:172.21.15.120:3300/0 conn(0x7f0f8c06c300 0x7f0f8c06c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.097+0000 7f0f94800640 1 --2- 172.21.15.62:0/3749673095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c0b2f90 0x7f0f8c161e70 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.097+0000 7f0f94800640 1 -- 172.21.15.62:0/3749673095 >> 172.21.15.62:0/3749673095 conn(0x7f0f8c0b02e0 msgr2=0x7f0f8c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:34.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.097+0000 7f0f94800640 1 -- 172.21.15.62:0/3749673095 shutdown_connections 2024-03-23T11:24:34.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.097+0000 7f0f94800640 1 -- 172.21.15.62:0/3749673095 wait complete. 2024-03-23T11:24:34.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.099+0000 7f0f94800640 1 Processor -- start 2024-03-23T11:24:34.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.099+0000 7f0f94800640 1 -- start start 2024-03-23T11:24:34.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.100+0000 7f0f94800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f8c06c300 0x7f0f8c15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.100+0000 7f0f94800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f8c0b15e0 0x7f0f8c150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.100+0000 7f0f94800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c0b2f90 0x7f0f8c14d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.100+0000 7f0f94800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c151860 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.100+0000 7f0f94800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c15b180 con 0x7f0f8c06c300 2024-03-23T11:24:34.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.100+0000 7f0f94800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c15b460 con 0x7f0f8c0b15e0 2024-03-23T11:24:34.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.100+0000 7f0f92800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c0b2f90 0x7f0f8c14d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.100+0000 7f0f92800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c0b2f90 0x7f0f8c14d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51052/0 (socket says 172.21.15.62:51052) 2024-03-23T11:24:34.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.101+0000 7f0f92800640 1 -- 172.21.15.62:0/3783573905 learned_addr learned my addr 172.21.15.62:0/3783573905 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:34.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.101+0000 7f0f91e00640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f8c06c300 0x7f0f8c15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.101+0000 7f0f91400640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f8c0b15e0 0x7f0f8c150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.101+0000 7f0f92800640 1 -- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f8c0b15e0 msgr2=0x7f0f8c150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.101+0000 7f0f92800640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f8c0b15e0 0x7f0f8c150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.101+0000 7f0f92800640 1 -- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f8c06c300 msgr2=0x7f0f8c15aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.101+0000 7f0f92800640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f8c06c300 0x7f0f8c15aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.101+0000 7f0f92800640 1 -- 172.21.15.62:0/3783573905 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f800094d0 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.101+0000 7f0f92800640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c0b2f90 0x7f0f8c14d5b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0f88007e40 tx=0x7f0f8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:34.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.102+0000 7f0f86000640 1 -- 172.21.15.62:0/3783573905 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88014620 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.102+0000 7f0f86000640 1 -- 172.21.15.62:0/3783573905 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8800ee90 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.102+0000 7f0f94800640 1 -- 172.21.15.62:0/3783573905 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8c15b740 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.102+0000 7f0f86000640 1 -- 172.21.15.62:0/3783573905 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f880149e0 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.102+0000 7f0f94800640 1 -- 172.21.15.62:0/3783573905 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8c151330 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.103+0000 7f0f94800640 1 -- 172.21.15.62:0/3783573905 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8c071570 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.104+0000 7f0f86000640 1 -- 172.21.15.62:0/3783573905 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0f88014b40 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.105+0000 7f0f86000640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7006a7a0 0x7f0f7006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.105+0000 7f0f86000640 1 -- 172.21.15.62:0/3783573905 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f880897c0 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.106+0000 7f0f91e00640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7006a7a0 0x7f0f7006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.107+0000 7f0f91e00640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7006a7a0 0x7f0f7006cc60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0f80036000 tx=0x7f0f8003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:34.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.109+0000 7f0f86000640 1 -- 172.21.15.62:0/3783573905 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f88054b40 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.348+0000 7f0f94800640 1 -- 172.21.15.62:0/3783573905 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0f8c066c90 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.345 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:24:34.350+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:24:34.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.366+0000 7f0f86000640 1 -- 172.21.15.62:0/3783573905 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v44) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0f88058160 con 0x7f0f8c0b2f90 2024-03-23T11:24:34.362 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:24:34.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 -- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7006a7a0 msgr2=0x7f0f7006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7006a7a0 0x7f0f7006cc60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0f80036000 tx=0x7f0f8003c040 comp rx=0 tx=0).stop 2024-03-23T11:24:34.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 -- 172.21.15.62:0/3783573905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c0b2f90 msgr2=0x7f0f8c14d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c0b2f90 0x7f0f8c14d5b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0f88007e40 tx=0x7f0f8800e4b0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 -- 172.21.15.62:0/3783573905 shutdown_connections 2024-03-23T11:24:34.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f7006a7a0 0x7f0f7006cc60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c0b2f90 0x7f0f8c14d5b0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f8c0b15e0 0x7f0f8c150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 --2- 172.21.15.62:0/3783573905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f8c06c300 0x7f0f8c15aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 -- 172.21.15.62:0/3783573905 >> 172.21.15.62:0/3783573905 conn(0x7f0f8c0b02e0 msgr2=0x7f0f8c159000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:34.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 -- 172.21.15.62:0/3783573905 shutdown_connections 2024-03-23T11:24:34.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.369+0000 7f0f94800640 1 -- 172.21.15.62:0/3783573905 wait complete. 2024-03-23T11:24:34.384 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-03-23T11:24:34.386 INFO:tasks.ceph.mds.d.smithi120.stderr: -3526> 2024-03-23T11:24:15.519+0000 7f6c14000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:24:34.455 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-23T11:24:34.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.501+0000 7fd563e4bf40 1 Processor -- start 2024-03-23T11:24:34.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.501+0000 7fd563e4bf40 1 -- start start 2024-03-23T11:24:34.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.502+0000 7fd563e4bf40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad8e222310 0x55ad8e2b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.502+0000 7fd563e4bf40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e2adc30 0x55ad8e2b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.502+0000 7fd563e4bf40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad8e2ae010 0x55ad8e203930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.502+0000 7fd563e4bf40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad8e11e800 con 0x55ad8e2adc30 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.502+0000 7fd563e4bf40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad8e11e4a0 con 0x55ad8e2ae010 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.502+0000 7fd563e4bf40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad8e115110 con 0x55ad8e222310 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.502+0000 7fd55fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e2adc30 0x55ad8e2b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.502+0000 7fd55fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e2adc30 0x55ad8e2b1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51074/0 (socket says 172.21.15.62:51074) 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.502+0000 7fd55fe00640 1 -- 172.21.15.62:0/1037169871 learned_addr learned my addr 172.21.15.62:0/1037169871 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.503+0000 7fd560800640 1 --2- 172.21.15.62:0/1037169871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad8e222310 0x55ad8e2b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.503+0000 7fd561200640 1 --2- 172.21.15.62:0/1037169871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad8e2ae010 0x55ad8e203930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.503+0000 7fd55fe00640 1 -- 172.21.15.62:0/1037169871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad8e222310 msgr2=0x55ad8e2b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.503+0000 7fd55fe00640 1 --2- 172.21.15.62:0/1037169871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad8e222310 0x55ad8e2b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.503+0000 7fd55fe00640 1 -- 172.21.15.62:0/1037169871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad8e2ae010 msgr2=0x55ad8e203930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:34.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.503+0000 7fd55fe00640 1 --2- 172.21.15.62:0/1037169871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad8e2ae010 0x55ad8e203930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.503+0000 7fd55fe00640 1 -- 172.21.15.62:0/1037169871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c005de0 con 0x55ad8e2adc30 2024-03-23T11:24:34.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.503+0000 7fd55fe00640 1 --2- 172.21.15.62:0/1037169871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e2adc30 0x55ad8e2b1ba0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fd54c044680 tx=0x7fd54c04c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671d7fef67003bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:34.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.503+0000 7fd55ea00640 1 -- 172.21.15.62:0/1037169871 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c052870 con 0x55ad8e2adc30 2024-03-23T11:24:34.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.504+0000 7fd55ea00640 1 -- 172.21.15.62:0/1037169871 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54c0529d0 con 0x55ad8e2adc30 2024-03-23T11:24:34.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.504+0000 7fd55ea00640 1 -- 172.21.15.62:0/1037169871 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c052d60 con 0x55ad8e2adc30 2024-03-23T11:24:34.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.504+0000 7fd563e4bf40 1 -- 172.21.15.62:0/1037169871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e2adc30 msgr2=0x55ad8e2b1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.504+0000 7fd563e4bf40 1 --2- 172.21.15.62:0/1037169871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e2adc30 0x55ad8e2b1ba0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fd54c044680 tx=0x7fd54c04c000 comp rx=0 tx=0).stop 2024-03-23T11:24:34.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.504+0000 7fd563e4bf40 1 -- 172.21.15.62:0/1037169871 shutdown_connections 2024-03-23T11:24:34.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.504+0000 7fd563e4bf40 1 --2- 172.21.15.62:0/1037169871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e2adc30 0x55ad8e2b1ba0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.504+0000 7fd563e4bf40 1 --2- 172.21.15.62:0/1037169871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad8e2ae010 0x55ad8e203930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.504+0000 7fd563e4bf40 1 --2- 172.21.15.62:0/1037169871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad8e222310 0x55ad8e2b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.504+0000 7fd563e4bf40 1 -- 172.21.15.62:0/1037169871 >> 172.21.15.62:0/1037169871 conn(0x55ad8e224000 msgr2=0x55ad8e208c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:34.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.504+0000 7fd563e4bf40 1 -- 172.21.15.62:0/1037169871 shutdown_connections 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.504+0000 7fd563e4bf40 1 -- 172.21.15.62:0/1037169871 wait complete. 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.505+0000 7fd563e4bf40 1 Processor -- start 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.505+0000 7fd563e4bf40 1 -- start start 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.506+0000 7fd563e4bf40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e222310 0x55ad8e28c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.506+0000 7fd563e4bf40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad8e2adc30 0x55ad8e2ac2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.506+0000 7fd563e4bf40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad8e2ae010 0x55ad8e2ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.506+0000 7fd563e4bf40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad8e28e280 con 0x55ad8e222310 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.506+0000 7fd563e4bf40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad8e28dcc0 con 0x55ad8e2adc30 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.506+0000 7fd563e4bf40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad8e28dfa0 con 0x55ad8e2ae010 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd560800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e222310 0x55ad8e28c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd560800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e222310 0x55ad8e28c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51080/0 (socket says 172.21.15.62:51080) 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd560800640 1 -- 172.21.15.62:0/2606736388 learned_addr learned my addr 172.21.15.62:0/2606736388 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd55fe00640 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad8e2adc30 0x55ad8e2ac2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd561200640 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad8e2ae010 0x55ad8e2ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd560800640 1 -- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad8e2ae010 msgr2=0x55ad8e2ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd560800640 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ad8e2ae010 0x55ad8e2ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd560800640 1 -- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad8e2adc30 msgr2=0x55ad8e2ac2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd560800640 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad8e2adc30 0x55ad8e2ac2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd560800640 1 -- 172.21.15.62:0/2606736388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad8e1474e0 con 0x55ad8e222310 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd55fe00640 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad8e2adc30 0x55ad8e2ac2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.507+0000 7fd560800640 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e222310 0x55ad8e28c000 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fd55000d440 tx=0x7fd55000d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.508+0000 7fd55d600640 1 -- 172.21.15.62:0/2606736388 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550003320 con 0x55ad8e222310 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.508+0000 7fd55d600640 1 -- 172.21.15.62:0/2606736388 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd550006030 con 0x55ad8e222310 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.508+0000 7fd563e4bf40 1 -- 172.21.15.62:0/2606736388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ad8e12d340 con 0x55ad8e222310 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.508+0000 7fd55d600640 1 -- 172.21.15.62:0/2606736388 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550001540 con 0x55ad8e222310 2024-03-23T11:24:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.508+0000 7fd563e4bf40 1 -- 172.21.15.62:0/2606736388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad8e120780 con 0x55ad8e222310 2024-03-23T11:24:34.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.510+0000 7fd55d600640 1 -- 172.21.15.62:0/2606736388 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd5500016a0 con 0x55ad8e222310 2024-03-23T11:24:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.511+0000 7fd55d600640 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd540069180 0x7fd54006b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.511+0000 7fd55d600640 1 -- 172.21.15.62:0/2606736388 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd550087450 con 0x55ad8e222310 2024-03-23T11:24:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.511+0000 7fd563e4bf40 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55ad8e2b7a60 0x55ad8e2aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.511+0000 7fd563e4bf40 1 -- 172.21.15.62:0/2606736388 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:1 10.10 10:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) v8 -- 0x55ad8e2aba90 con 0x55ad8e2b7a60 2024-03-23T11:24:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.512+0000 7fd55fe00640 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd540069180 0x7fd54006b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.512+0000 7fd561200640 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55ad8e2b7a60 0x55ad8e2aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.513+0000 7fd55fe00640 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd540069180 0x7fd54006b640 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd54c05e040 tx=0x7fd54c054040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:34.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.513+0000 7fd561200640 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55ad8e2b7a60 0x55ad8e2aa5f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:34.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.516+0000 7fd561200640 1 -- 172.21.15.62:0/2606736388 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v75'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fd5580097f0 con 0x55ad8e2b7a60 2024-03-23T11:24:34.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.516+0000 7fd563e4bf40 1 -- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55ad8e2b7a60 msgr2=0x55ad8e2aa5f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.516+0000 7fd563e4bf40 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55ad8e2b7a60 0x55ad8e2aa5f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.517+0000 7fd563e4bf40 1 -- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd540069180 msgr2=0x7fd54006b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.517+0000 7fd563e4bf40 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd540069180 0x7fd54006b640 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd54c05e040 tx=0x7fd54c054040 comp rx=0 tx=0).stop 2024-03-23T11:24:34.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.517+0000 7fd563e4bf40 1 -- 172.21.15.62:0/2606736388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e222310 msgr2=0x55ad8e28c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.517+0000 7fd563e4bf40 1 --2- 172.21.15.62:0/2606736388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ad8e222310 0x55ad8e28c000 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fd55000d440 tx=0x7fd55000d900 comp rx=0 tx=0).stop 2024-03-23T11:24:34.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.517+0000 7fd561200640 1 -- 172.21.15.62:0/2606736388 reap_dead start 2024-03-23T11:24:34.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.517+0000 7fd563e4bf40 1 -- 172.21.15.62:0/2606736388 shutdown_connections 2024-03-23T11:24:34.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.517+0000 7fd563e4bf40 1 -- 172.21.15.62:0/2606736388 >> 172.21.15.62:0/2606736388 conn(0x55ad8e224000 msgr2=0x55ad8e1d44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:34.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.517+0000 7fd563e4bf40 1 -- 172.21.15.62:0/2606736388 shutdown_connections 2024-03-23T11:24:34.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.517+0000 7fd563e4bf40 1 -- 172.21.15.62:0/2606736388 wait complete. 2024-03-23T11:24:34.517 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-23T11:24:34.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.741+0000 7f7149600640 1 Processor -- start 2024-03-23T11:24:34.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.741+0000 7f7149600640 1 -- start start 2024-03-23T11:24:34.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.743+0000 7f7149600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7144159490 0x7f714406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.743+0000 7f7149600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71440ce5f0 0x7f71440b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.743+0000 7f7149600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71440ced90 con 0x7f714415da90 2024-03-23T11:24:34.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.743+0000 7f7149600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f714406e3e0 con 0x7f71440ce5f0 2024-03-23T11:24:34.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.743+0000 7f7149600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714406c1c0 con 0x7f7144159490 2024-03-23T11:24:34.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.744+0000 7f7148c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71440ce5f0 0x7f71440b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.744+0000 7f7148c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71440ce5f0 0x7f71440b26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51096/0 (socket says 172.21.15.62:51096) 2024-03-23T11:24:34.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.744+0000 7f7148c00640 1 -- 172.21.15.62:0/2660379077 learned_addr learned my addr 172.21.15.62:0/2660379077 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:34.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.744+0000 7f7143e00640 1 --2- 172.21.15.62:0/2660379077 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7144159490 0x7f714406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.744+0000 7f7148c00640 1 -- 172.21.15.62:0/2660379077 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7144159490 msgr2=0x7f714406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:34.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.744+0000 7f7148c00640 1 --2- 172.21.15.62:0/2660379077 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7144159490 0x7f714406cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.744+0000 7f7148c00640 1 -- 172.21.15.62:0/2660379077 >> v1:172.21.15.120:6789/0 conn(0x7f714415da90 legacy=0x7f71440cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:34.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.744+0000 7f7148c00640 1 -- 172.21.15.62:0/2660379077 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714406e100 con 0x7f71440ce5f0 2024-03-23T11:24:34.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.745+0000 7f7148c00640 1 --2- 172.21.15.62:0/2660379077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71440ce5f0 0x7f71440b26b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f713c009e50 tx=0x7f713c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a4858f7fea707c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:34.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.746+0000 7f7143400640 1 -- 172.21.15.62:0/2660379077 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c02c9e0 con 0x7f71440ce5f0 2024-03-23T11:24:34.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.746+0000 7f7143400640 1 -- 172.21.15.62:0/2660379077 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c02cb40 con 0x7f71440ce5f0 2024-03-23T11:24:34.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.746+0000 7f7143400640 1 -- 172.21.15.62:0/2660379077 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c0343f0 con 0x7f71440ce5f0 2024-03-23T11:24:34.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.746+0000 7f7149600640 1 -- 172.21.15.62:0/2660379077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71440ce5f0 msgr2=0x7f71440b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.746+0000 7f7149600640 1 --2- 172.21.15.62:0/2660379077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71440ce5f0 0x7f71440b26b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f713c009e50 tx=0x7f713c02be40 comp rx=0 tx=0).stop 2024-03-23T11:24:34.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.746+0000 7f7149600640 1 -- 172.21.15.62:0/2660379077 shutdown_connections 2024-03-23T11:24:34.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.746+0000 7f7149600640 1 --2- 172.21.15.62:0/2660379077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71440ce5f0 0x7f71440b26b0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.746+0000 7f7149600640 1 --2- 172.21.15.62:0/2660379077 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7144159490 0x7f714406cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.746+0000 7f7149600640 1 -- 172.21.15.62:0/2660379077 >> 172.21.15.62:0/2660379077 conn(0x7f7144066050 msgr2=0x7f7144069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:34.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.746+0000 7f7149600640 1 -- 172.21.15.62:0/2660379077 shutdown_connections 2024-03-23T11:24:34.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.747+0000 7f7149600640 1 -- 172.21.15.62:0/2660379077 wait complete. 2024-03-23T11:24:34.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.748+0000 7f7149600640 1 Processor -- start 2024-03-23T11:24:34.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.748+0000 7f7149600640 1 -- start start 2024-03-23T11:24:34.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.749+0000 7f7149600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71440ce5f0 0x7f71441ed910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.749+0000 7f7149600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7144159490 0x7f71440ad210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.749+0000 7f7149600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f714415da90 0x7f71441f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.749+0000 7f7149600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714414c8c0 con 0x7f71440ce5f0 2024-03-23T11:24:34.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.749+0000 7f7149600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71440a74a0 con 0x7f714415da90 2024-03-23T11:24:34.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.749+0000 7f7149600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714415bdf0 con 0x7f7144159490 2024-03-23T11:24:34.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f7143e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71440ce5f0 0x7f71441ed910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f7143e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71440ce5f0 0x7f71441ed910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51100/0 (socket says 172.21.15.62:51100) 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f7143e00640 1 -- 172.21.15.62:0/747910271 learned_addr learned my addr 172.21.15.62:0/747910271 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f7148c00640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f714415da90 0x7f71441f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f713be00640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7144159490 0x7f71440ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f7143e00640 1 -- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7144159490 msgr2=0x7f71440ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f7143e00640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7144159490 0x7f71440ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f7143e00640 1 -- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f714415da90 msgr2=0x7f71441f0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f7143e00640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f714415da90 0x7f71441f0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f7143e00640 1 -- 172.21.15.62:0/747910271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c009b30 con 0x7f71440ce5f0 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f7148c00640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f714415da90 0x7f71441f0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.750+0000 7f713be00640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7144159490 0x7f71440ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.751+0000 7f7143e00640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71440ce5f0 0x7f71441ed910 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f713400d5f0 tx=0x7f713400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.751+0000 7f7140c00640 1 -- 172.21.15.62:0/747910271 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71340125f0 con 0x7f71440ce5f0 2024-03-23T11:24:34.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.751+0000 7f7140c00640 1 -- 172.21.15.62:0/747910271 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7134012750 con 0x7f71440ce5f0 2024-03-23T11:24:34.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.751+0000 7f7140c00640 1 -- 172.21.15.62:0/747910271 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134012b00 con 0x7f71440ce5f0 2024-03-23T11:24:34.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.751+0000 7f7149600640 1 -- 172.21.15.62:0/747910271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71441f0760 con 0x7f71440ce5f0 2024-03-23T11:24:34.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.751+0000 7f7149600640 1 -- 172.21.15.62:0/747910271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714415b830 con 0x7f71440ce5f0 2024-03-23T11:24:34.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.752+0000 7f7149600640 1 -- 172.21.15.62:0/747910271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714415ded0 con 0x7f71440ce5f0 2024-03-23T11:24:34.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.753+0000 7f7140c00640 1 -- 172.21.15.62:0/747910271 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7134012c60 con 0x7f71440ce5f0 2024-03-23T11:24:34.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.754+0000 7f7140c00640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f712006a7d0 0x7f712006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:34.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.754+0000 7f7140c00640 1 -- 172.21.15.62:0/747910271 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f71340893b0 con 0x7f71440ce5f0 2024-03-23T11:24:34.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.755+0000 7f713be00640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f712006a7d0 0x7f712006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:34.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.756+0000 7f713be00640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f712006a7d0 0x7f712006cc90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f714415b340 tx=0x7f712c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:34.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.759+0000 7f7140c00640 1 -- 172.21.15.62:0/747910271 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f714415ded0 con 0x7f71440ce5f0 2024-03-23T11:24:34.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:34.998+0000 7f7149600640 1 -- 172.21.15.62:0/747910271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f714415ded0 con 0x7f71440ce5f0 2024-03-23T11:24:35.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.247+0000 7f7140c00640 1 -- 172.21.15.62:0/747910271 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v45) v1 ==== 162+0+0 (secure 0 0 0) 0x7f714415ded0 con 0x7f71440ce5f0 2024-03-23T11:24:35.244 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-23T11:24:35.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.251+0000 7f7149600640 1 -- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f712006a7d0 msgr2=0x7f712006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:35.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.251+0000 7f7149600640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f712006a7d0 0x7f712006cc90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f714415b340 tx=0x7f712c009040 comp rx=0 tx=0).stop 2024-03-23T11:24:35.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.251+0000 7f7149600640 1 -- 172.21.15.62:0/747910271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71440ce5f0 msgr2=0x7f71441ed910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:35.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.251+0000 7f7149600640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71440ce5f0 0x7f71441ed910 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f713400d5f0 tx=0x7f713400dab0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.251+0000 7f7149600640 1 -- 172.21.15.62:0/747910271 shutdown_connections 2024-03-23T11:24:35.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.251+0000 7f7149600640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f712006a7d0 0x7f712006cc90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.251+0000 7f7149600640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f714415da90 0x7f71441f0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.252+0000 7f7149600640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7144159490 0x7f71440ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.252+0000 7f7149600640 1 --2- 172.21.15.62:0/747910271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71440ce5f0 0x7f71441ed910 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.252+0000 7f7149600640 1 -- 172.21.15.62:0/747910271 >> 172.21.15.62:0/747910271 conn(0x7f7144066050 msgr2=0x7f71440ae100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:35.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.252+0000 7f7149600640 1 -- 172.21.15.62:0/747910271 shutdown_connections 2024-03-23T11:24:35.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.252+0000 7f7149600640 1 -- 172.21.15.62:0/747910271 wait complete. 2024-03-23T11:24:35.269 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:24:35.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.475+0000 7f744ec00640 1 Processor -- start 2024-03-23T11:24:35.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.475+0000 7f744ec00640 1 -- start start 2024-03-23T11:24:35.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.476+0000 7f744ec00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f744815c900 0x7f74480d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:35.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.477+0000 7f744ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 0x7f74480b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:35.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.477+0000 7f744ec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74480d1100 con 0x7f74480cd690 2024-03-23T11:24:35.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.477+0000 7f744ec00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74480d13e0 con 0x7f744815c900 2024-03-23T11:24:35.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.477+0000 7f744ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74480d16c0 con 0x7f744815cce0 2024-03-23T11:24:35.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.477+0000 7f7447400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 0x7f74480b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:35.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.477+0000 7f7447e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f744815c900 0x7f74480d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:35.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.478+0000 7f7447400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 0x7f74480b82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:58422/0 (socket says 172.21.15.62:58422) 2024-03-23T11:24:35.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.478+0000 7f7447400640 1 -- 172.21.15.62:0/1740977331 learned_addr learned my addr 172.21.15.62:0/1740977331 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:35.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.478+0000 7f7447e00640 1 -- 172.21.15.62:0/1740977331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 msgr2=0x7f74480b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:35.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.478+0000 7f7447e00640 1 --2- 172.21.15.62:0/1740977331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 0x7f74480b82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.478+0000 7f7447e00640 1 -- 172.21.15.62:0/1740977331 >> v1:172.21.15.120:6789/0 conn(0x7f74480cd690 legacy=0x7f74480b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:35.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.478+0000 7f7447e00640 1 -- 172.21.15.62:0/1740977331 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74480d19a0 con 0x7f744815c900 2024-03-23T11:24:35.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.480+0000 7f7447e00640 1 --2- 172.21.15.62:0/1740977331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f744815c900 0x7f74480d21a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7440009d90 tx=0x7f744002bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c13cb0f66a1c5fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:35.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.480+0000 7f7446a00640 1 -- 172.21.15.62:0/1740977331 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744002cce0 con 0x7f744815c900 2024-03-23T11:24:35.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.480+0000 7f7446a00640 1 -- 172.21.15.62:0/1740977331 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744002ce40 con 0x7f744815c900 2024-03-23T11:24:35.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.480+0000 7f7446a00640 1 -- 172.21.15.62:0/1740977331 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7440034660 con 0x7f744815c900 2024-03-23T11:24:35.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.481+0000 7f744ec00640 1 -- 172.21.15.62:0/1740977331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f744815c900 msgr2=0x7f74480d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:35.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.481+0000 7f744ec00640 1 --2- 172.21.15.62:0/1740977331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f744815c900 0x7f74480d21a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7440009d90 tx=0x7f744002bf30 comp rx=0 tx=0).stop 2024-03-23T11:24:35.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.481+0000 7f744ec00640 1 -- 172.21.15.62:0/1740977331 shutdown_connections 2024-03-23T11:24:35.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.481+0000 7f744ec00640 1 --2- 172.21.15.62:0/1740977331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 0x7f74480b82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.481+0000 7f744ec00640 1 --2- 172.21.15.62:0/1740977331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f744815c900 0x7f74480d21a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.481+0000 7f744ec00640 1 -- 172.21.15.62:0/1740977331 >> 172.21.15.62:0/1740977331 conn(0x7f7448071390 msgr2=0x7f7448071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:35.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.481+0000 7f744ec00640 1 -- 172.21.15.62:0/1740977331 shutdown_connections 2024-03-23T11:24:35.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.481+0000 7f744ec00640 1 -- 172.21.15.62:0/1740977331 wait complete. 2024-03-23T11:24:35.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.483+0000 7f744ec00640 1 Processor -- start 2024-03-23T11:24:35.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.483+0000 7f744ec00640 1 -- start start 2024-03-23T11:24:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.484+0000 7f744ec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74480cd690 0x7f74480cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.484+0000 7f744ec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f744815c900 0x7f7448156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.484+0000 7f744ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 0x7f74481570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.484+0000 7f744ec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7448157620 con 0x7f744815c900 2024-03-23T11:24:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.484+0000 7f744ec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74481f2180 con 0x7f74480cd690 2024-03-23T11:24:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.484+0000 7f744ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74481f2420 con 0x7f744815cce0 2024-03-23T11:24:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.484+0000 7f7447400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f744815c900 0x7f7448156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.484+0000 7f7447400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f744815c900 0x7f7448156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51104/0 (socket says 172.21.15.62:51104) 2024-03-23T11:24:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.484+0000 7f7447400640 1 -- 172.21.15.62:0/3989354696 learned_addr learned my addr 172.21.15.62:0/3989354696 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f7447e00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74480cd690 0x7f74480cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:35.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f744cc00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 0x7f74481570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:35.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f7447400640 1 -- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 msgr2=0x7f74481570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:35.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f7447400640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 0x7f74481570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f7447400640 1 -- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74480cd690 msgr2=0x7f74480cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:35.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f7447400640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74480cd690 0x7f74480cbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f7447400640 1 -- 172.21.15.62:0/3989354696 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7440009a70 con 0x7f744815c900 2024-03-23T11:24:35.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f7447e00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74480cd690 0x7f74480cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:35.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f744cc00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 0x7f74481570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:35.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f7447400640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f744815c900 0x7f7448156ba0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f743400e840 tx=0x7f743400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:35.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f743fe00640 1 -- 172.21.15.62:0/3989354696 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743400c8e0 con 0x7f744815c900 2024-03-23T11:24:35.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f743fe00640 1 -- 172.21.15.62:0/3989354696 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743400ca40 con 0x7f744815c900 2024-03-23T11:24:35.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f743fe00640 1 -- 172.21.15.62:0/3989354696 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743400cdf0 con 0x7f744815c900 2024-03-23T11:24:35.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.485+0000 7f744ec00640 1 -- 172.21.15.62:0/3989354696 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74481f2700 con 0x7f744815c900 2024-03-23T11:24:35.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.486+0000 7f744ec00640 1 -- 172.21.15.62:0/3989354696 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74480cc250 con 0x7f744815c900 2024-03-23T11:24:35.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.487+0000 7f744ec00640 1 -- 172.21.15.62:0/3989354696 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74480672c0 con 0x7f744815c900 2024-03-23T11:24:35.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.488+0000 7f743fe00640 1 -- 172.21.15.62:0/3989354696 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7434013470 con 0x7f744815c900 2024-03-23T11:24:35.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.488+0000 7f743fe00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f742806a7a0 0x7f742806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:35.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.489+0000 7f743fe00640 1 -- 172.21.15.62:0/3989354696 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7434088f10 con 0x7f744815c900 2024-03-23T11:24:35.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.489+0000 7f7447e00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f742806a7a0 0x7f742806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:35.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.490+0000 7f7447e00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f742806a7a0 0x7f742806cc60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7440036000 tx=0x7f7440030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:35.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.493+0000 7f743fe00640 1 -- 172.21.15.62:0/3989354696 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f74340542c0 con 0x7f744815c900 2024-03-23T11:24:35.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.734+0000 7f744ec00640 1 -- 172.21.15.62:0/3989354696 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f744815e2e0 con 0x7f744815c900 2024-03-23T11:24:35.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.736+0000 7f743fe00640 1 -- 172.21.15.62:0/3989354696 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f74340578e0 con 0x7f744815c900 2024-03-23T11:24:35.732 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:35.732 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/29171509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":29171509},{"type":"v1","addr":"172.21.15.120:6837","nonce":29171509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:24:35.240444+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6618":{"gid":6618,"name":"b","rank":0,"incarnation":45,"state":"up:replay","state_seq":1,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:24:35.732 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 45 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.738+0000 7f744ec00640 1 -- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f742806a7a0 msgr2=0x7f742806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.738+0000 7f744ec00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f742806a7a0 0x7f742806cc60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7440036000 tx=0x7f7440030000 comp rx=0 tx=0).stop 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.738+0000 7f744ec00640 1 -- 172.21.15.62:0/3989354696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f744815c900 msgr2=0x7f7448156ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.738+0000 7f744ec00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f744815c900 0x7f7448156ba0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f743400e840 tx=0x7f743400ed00 comp rx=0 tx=0).stop 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.738+0000 7f744ec00640 1 -- 172.21.15.62:0/3989354696 shutdown_connections 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.738+0000 7f744ec00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f742806a7a0 0x7f742806cc60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.738+0000 7f744ec00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f744815c900 0x7f7448156ba0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.738+0000 7f744ec00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f744815cce0 0x7f74481570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.738+0000 7f744ec00640 1 --2- 172.21.15.62:0/3989354696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74480cd690 0x7f74480cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.738+0000 7f744ec00640 1 -- 172.21.15.62:0/3989354696 >> 172.21.15.62:0/3989354696 conn(0x7f7448071390 msgr2=0x7f744815f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.739+0000 7f744ec00640 1 -- 172.21.15.62:0/3989354696 shutdown_connections 2024-03-23T11:24:35.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:35.739+0000 7f744ec00640 1 -- 172.21.15.62:0/3989354696 wait complete. 2024-03-23T11:24:36.157 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:24:11.044296+0000', 'modified': '2024-03-23T11:24:35.240444+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 75, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6618}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6618': {'gid': 6618, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.120:6835/1201988215', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1201988215}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1201988215}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:24:36.158 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6618:up:replay 2024-03-23T11:24:37.158 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:24:37.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.399+0000 7f3bbcc00640 1 Processor -- start 2024-03-23T11:24:37.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.399+0000 7f3bbcc00640 1 -- start start 2024-03-23T11:24:37.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.401+0000 7f3bbcc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3bb80b2f30 0x7f3bb8070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:37.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.401+0000 7f3bbcc00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb80cd140 con 0x7f3bb8143dd0 2024-03-23T11:24:37.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.401+0000 7f3bbcc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb80cd3e0 con 0x7f3bb8070090 2024-03-23T11:24:37.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.401+0000 7f3bbcc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb8070a60 con 0x7f3bb80b2f30 2024-03-23T11:24:37.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.401+0000 7f3bb7400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3bb8143dd0 0x7f3bb815abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43432/0 (socket says 172.21.15.62:43432) 2024-03-23T11:24:37.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.401+0000 7f3bb7400640 1 -- 172.21.15.62:0/2497500172 learned_addr learned my addr 172.21.15.62:0/2497500172 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:37.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb7e00640 1 --2- 172.21.15.62:0/2497500172 >> v2:172.21.15.120:3300/0 conn(0x7f3bb80b2f30 0x7f3bb8070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:37.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288558011 0 0) 0x7f3bb80cd140 con 0x7f3bb8143dd0 2024-03-23T11:24:37.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b9c003520 con 0x7f3bb8143dd0 2024-03-23T11:24:37.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4083688810 0 0) 0x7f3b9c003520 con 0x7f3bb8143dd0 2024-03-23T11:24:37.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 >> v2:172.21.15.120:3300/0 conn(0x7f3bb80b2f30 msgr2=0x7f3bb8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:37.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 --2- 172.21.15.62:0/2497500172 >> v2:172.21.15.120:3300/0 conn(0x7f3bb80b2f30 0x7f3bb8070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 >> v1:172.21.15.120:6789/0 conn(0x7f3bb8070090 legacy=0x7f3bb80b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb80cd6c0 con 0x7f3bb8143dd0 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f3ba8001670 con 0x7f3bb8143dd0 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 >> v1:172.21.15.62:6789/0 conn(0x7f3bb8143dd0 legacy=0x7f3bb815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 --2- 172.21.15.62:0/2497500172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c0039c0 0x7f3b9c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 --2- 172.21.15.62:0/2497500172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0047c0 0x7f3b9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 --2- 172.21.15.62:0/2497500172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b9c005180 0x7f3b9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb8070a60 con 0x7f3b9c005180 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c00c080 con 0x7f3b9c0039c0 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.402+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c00c240 con 0x7f3b9c0047c0 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.403+0000 7f3bb7e00640 1 --2- 172.21.15.62:0/2497500172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b9c005180 0x7f3b9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.403+0000 7f3bb7400640 1 --2- 172.21.15.62:0/2497500172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c0039c0 0x7f3b9c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.403+0000 7f3bafe00640 1 --2- 172.21.15.62:0/2497500172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0047c0 0x7f3b9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.403+0000 7f3bb7e00640 1 -- 172.21.15.62:0/2497500172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0047c0 msgr2=0x7f3b9c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.403+0000 7f3bb7e00640 1 --2- 172.21.15.62:0/2497500172 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b9c0047c0 0x7f3b9c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.403+0000 7f3bb7e00640 1 -- 172.21.15.62:0/2497500172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c0039c0 msgr2=0x7f3b9c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:37.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.403+0000 7f3bb7e00640 1 --2- 172.21.15.62:0/2497500172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c0039c0 0x7f3b9c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.403+0000 7f3bb7e00640 1 -- 172.21.15.62:0/2497500172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c003520 con 0x7f3b9c005180 2024-03-23T11:24:37.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.404+0000 7f3bb7e00640 1 -- 172.21.15.62:0/2497500172 reap_dead start 2024-03-23T11:24:37.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.404+0000 7f3bb7e00640 1 --2- 172.21.15.62:0/2497500172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b9c005180 0x7f3b9c00b980 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009fc0 tx=0x7f3b9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1002adb1ffe2d3c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:37.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.404+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb002cc60 con 0x7f3b9c005180 2024-03-23T11:24:37.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.405+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb000f040 con 0x7f3b9c005180 2024-03-23T11:24:37.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.405+0000 7f3bb6a00640 1 -- 172.21.15.62:0/2497500172 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0003990 con 0x7f3b9c005180 2024-03-23T11:24:37.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.405+0000 7f3bbcc00640 1 -- 172.21.15.62:0/2497500172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b9c005180 msgr2=0x7f3b9c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:37.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.405+0000 7f3bbcc00640 1 --2- 172.21.15.62:0/2497500172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b9c005180 0x7f3b9c00b980 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009fc0 tx=0x7f3b9c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.405+0000 7f3bbcc00640 1 -- 172.21.15.62:0/2497500172 shutdown_connections 2024-03-23T11:24:37.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.405+0000 7f3bbcc00640 1 --2- 172.21.15.62:0/2497500172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b9c005180 0x7f3b9c00b980 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.405+0000 7f3bbcc00640 1 -- 172.21.15.62:0/2497500172 >> 172.21.15.62:0/2497500172 conn(0x7f3bb8066050 msgr2=0x7f3bb8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:37.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.405+0000 7f3bbcc00640 1 -- 172.21.15.62:0/2497500172 shutdown_connections 2024-03-23T11:24:37.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.405+0000 7f3bbcc00640 1 -- 172.21.15.62:0/2497500172 wait complete. 2024-03-23T11:24:37.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.407+0000 7f3bbcc00640 1 Processor -- start 2024-03-23T11:24:37.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.407+0000 7f3bbcc00640 1 -- start start 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.408+0000 7f3bbcc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c005180 0x7f3bb8159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.408+0000 7f3bbcc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bb8134470 0x7f3bb81f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.408+0000 7f3bbcc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bb80a9dc0 0x7f3bb815bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.408+0000 7f3bbcc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb815a430 con 0x7f3bb8134470 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.408+0000 7f3bbcc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb80a9600 con 0x7f3b9c005180 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.408+0000 7f3bbcc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb81f38f0 con 0x7f3bb80a9dc0 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.408+0000 7f3bafe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bb8134470 0x7f3bb81f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.408+0000 7f3bafe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bb8134470 0x7f3bb81f30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50432/0 (socket says 172.21.15.62:50432) 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.408+0000 7f3bafe00640 1 -- 172.21.15.62:0/1315503325 learned_addr learned my addr 172.21.15.62:0/1315503325 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.409+0000 7f3bb7400640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c005180 0x7f3bb8159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.409+0000 7f3bb7e00640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bb80a9dc0 0x7f3bb815bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:37.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.409+0000 7f3bafe00640 1 -- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bb80a9dc0 msgr2=0x7f3bb815bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:37.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.409+0000 7f3bafe00640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bb80a9dc0 0x7f3bb815bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.409+0000 7f3bafe00640 1 -- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c005180 msgr2=0x7f3bb8159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:37.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.409+0000 7f3bafe00640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c005180 0x7f3bb8159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.409+0000 7f3bafe00640 1 -- 172.21.15.62:0/1315503325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb81f3bd0 con 0x7f3bb8134470 2024-03-23T11:24:37.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.409+0000 7f3bafe00640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bb8134470 0x7f3bb81f30a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0001eb0 tx=0x7f3ba000d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:37.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.410+0000 7f3baf400640 1 -- 172.21.15.62:0/1315503325 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0005480 con 0x7f3bb8134470 2024-03-23T11:24:37.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.410+0000 7f3baf400640 1 -- 172.21.15.62:0/1315503325 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba0003610 con 0x7f3bb8134470 2024-03-23T11:24:37.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.410+0000 7f3baf400640 1 -- 172.21.15.62:0/1315503325 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0003980 con 0x7f3bb8134470 2024-03-23T11:24:37.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.410+0000 7f3bbcc00640 1 -- 172.21.15.62:0/1315503325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb81f3eb0 con 0x7f3bb8134470 2024-03-23T11:24:37.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.410+0000 7f3bbcc00640 1 -- 172.21.15.62:0/1315503325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb81f35e0 con 0x7f3bb8134470 2024-03-23T11:24:37.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.411+0000 7f3bbcc00640 1 -- 172.21.15.62:0/1315503325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb8066bb0 con 0x7f3bb8134470 2024-03-23T11:24:37.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.412+0000 7f3baf400640 1 -- 172.21.15.62:0/1315503325 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3ba0003ae0 con 0x7f3bb8134470 2024-03-23T11:24:37.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.413+0000 7f3baf400640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b9406a7d0 0x7f3b9406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:37.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.413+0000 7f3baf400640 1 -- 172.21.15.62:0/1315503325 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3ba0088d90 con 0x7f3bb8134470 2024-03-23T11:24:37.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.414+0000 7f3bb7400640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b9406a7d0 0x7f3b9406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:37.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.415+0000 7f3bb7400640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b9406a7d0 0x7f3b9406cc90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8001050 tx=0x7f3ba8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:37.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.418+0000 7f3baf400640 1 -- 172.21.15.62:0/1315503325 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ba00540b0 con 0x7f3bb8134470 2024-03-23T11:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.652+0000 7f3bbcc00640 1 -- 172.21.15.62:0/1315503325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bb815ac50 con 0x7f3bb8134470 2024-03-23T11:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.653+0000 7f3baf400640 1 -- 172.21.15.62:0/1315503325 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f3ba00576d0 con 0x7f3bb8134470 2024-03-23T11:24:37.649 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:37.650 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/29171509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":29171509},{"type":"v1","addr":"172.21.15.120:6837","nonce":29171509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:24:37.265205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6618":{"gid":6618,"name":"b","rank":0,"incarnation":45,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:24:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 47 2024-03-23T11:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 -- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b9406a7d0 msgr2=0x7f3b9406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b9406a7d0 0x7f3b9406cc90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8001050 tx=0x7f3ba8009040 comp rx=0 tx=0).stop 2024-03-23T11:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 -- 172.21.15.62:0/1315503325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bb8134470 msgr2=0x7f3bb81f30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bb8134470 0x7f3bb81f30a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0001eb0 tx=0x7f3ba000d9d0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 -- 172.21.15.62:0/1315503325 shutdown_connections 2024-03-23T11:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b9406a7d0 0x7f3b9406cc90 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bb8134470 0x7f3bb81f30a0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bb80a9dc0 0x7f3bb815bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 --2- 172.21.15.62:0/1315503325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b9c005180 0x7f3bb8159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:37.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 -- 172.21.15.62:0/1315503325 >> 172.21.15.62:0/1315503325 conn(0x7f3bb8066050 msgr2=0x7f3bb8141ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:37.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 -- 172.21.15.62:0/1315503325 shutdown_connections 2024-03-23T11:24:37.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:37.656+0000 7f3bbcc00640 1 -- 172.21.15.62:0/1315503325 wait complete. 2024-03-23T11:24:38.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:24:11.044296+0000', 'modified': '2024-03-23T11:24:37.265205+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 75, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6618}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6618': {'gid': 6618, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:rejoin', 'state_seq': 10, 'addr': '172.21.15.120:6835/1201988215', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1201988215}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1201988215}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:24:38.076 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6618:up:rejoin 2024-03-23T11:24:39.077 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:24:39.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.311+0000 7f6a09c00640 1 Processor -- start 2024-03-23T11:24:39.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.311+0000 7f6a09c00640 1 -- start start 2024-03-23T11:24:39.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.313+0000 7f6a09c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6a040cafa0 0x7f6a040cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:39.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.313+0000 7f6a09c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a040cbc60 con 0x7f6a0406c300 2024-03-23T11:24:39.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.313+0000 7f6a09c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a040cbf40 con 0x7f6a0406c6e0 2024-03-23T11:24:39.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.313+0000 7f6a09c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a040cd410 con 0x7f6a040cafa0 2024-03-23T11:24:39.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.313+0000 7f6a02a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6a0406c300 0x7f6a040ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43442/0 (socket says 172.21.15.62:43442) 2024-03-23T11:24:39.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.313+0000 7f6a02a00640 1 -- 172.21.15.62:0/3716315486 learned_addr learned my addr 172.21.15.62:0/3716315486 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:39.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f69fa000640 1 --2- 172.21.15.62:0/3716315486 >> v2:172.21.15.120:3300/0 conn(0x7f6a040cafa0 0x7f6a040cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:39.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494591199 0 0) 0x7f6a040cbc60 con 0x7f6a0406c300 2024-03-23T11:24:39.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e4003520 con 0x7f6a0406c300 2024-03-23T11:24:39.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 919179978 0 0) 0x7f69e4003520 con 0x7f6a0406c300 2024-03-23T11:24:39.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 >> v2:172.21.15.120:3300/0 conn(0x7f6a040cafa0 msgr2=0x7f6a040cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:39.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 --2- 172.21.15.62:0/3716315486 >> v2:172.21.15.120:3300/0 conn(0x7f6a040cafa0 0x7f6a040cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 >> v1:172.21.15.120:6789/0 conn(0x7f6a0406c6e0 legacy=0x7f6a0406cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a040cc220 con 0x7f6a0406c300 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f69f4001670 con 0x7f6a0406c300 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 >> v1:172.21.15.62:6789/0 conn(0x7f6a0406c300 legacy=0x7f6a040ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 --2- 172.21.15.62:0/3716315486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e40039c0 0x7f69e4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 --2- 172.21.15.62:0/3716315486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69e40047c0 0x7f69e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 --2- 172.21.15.62:0/3716315486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e40051d0 0x7f69e400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040cd410 con 0x7f69e40051d0 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e400c100 con 0x7f69e40039c0 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.314+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e400c2c0 con 0x7f69e40047c0 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.315+0000 7f6a03400640 1 --2- 172.21.15.62:0/3716315486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e40051d0 0x7f69e400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.315+0000 7f6a02a00640 1 --2- 172.21.15.62:0/3716315486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e40039c0 0x7f69e4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:39.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.315+0000 7f69fa000640 1 --2- 172.21.15.62:0/3716315486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69e40047c0 0x7f69e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:39.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.315+0000 7f6a03400640 1 -- 172.21.15.62:0/3716315486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69e40047c0 msgr2=0x7f69e4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:39.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.315+0000 7f6a03400640 1 --2- 172.21.15.62:0/3716315486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69e40047c0 0x7f69e4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:39.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.315+0000 7f6a03400640 1 -- 172.21.15.62:0/3716315486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e40039c0 msgr2=0x7f69e4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:39.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.315+0000 7f6a03400640 1 --2- 172.21.15.62:0/3716315486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e40039c0 0x7f69e4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:39.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.315+0000 7f6a03400640 1 -- 172.21.15.62:0/3716315486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e4003520 con 0x7f69e40051d0 2024-03-23T11:24:39.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.316+0000 7f6a03400640 1 -- 172.21.15.62:0/3716315486 reap_dead start 2024-03-23T11:24:39.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.316+0000 7f6a03400640 1 --2- 172.21.15.62:0/3716315486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e40051d0 0x7f69e400b980 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f69fc004480 tx=0x7f69e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb75b05009533bc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:39.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.316+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc006b50 con 0x7f69e40051d0 2024-03-23T11:24:39.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.316+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc006cd0 con 0x7f69e40051d0 2024-03-23T11:24:39.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.316+0000 7f6a02000640 1 -- 172.21.15.62:0/3716315486 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc007260 con 0x7f69e40051d0 2024-03-23T11:24:39.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.317+0000 7f6a09c00640 1 -- 172.21.15.62:0/3716315486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e40051d0 msgr2=0x7f69e400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:39.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.317+0000 7f6a09c00640 1 --2- 172.21.15.62:0/3716315486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e40051d0 0x7f69e400b980 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f69fc004480 tx=0x7f69e40057f0 comp rx=0 tx=0).stop 2024-03-23T11:24:39.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.317+0000 7f6a09c00640 1 -- 172.21.15.62:0/3716315486 shutdown_connections 2024-03-23T11:24:39.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.317+0000 7f6a09c00640 1 --2- 172.21.15.62:0/3716315486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69e40051d0 0x7f69e400b980 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:39.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.317+0000 7f6a09c00640 1 -- 172.21.15.62:0/3716315486 >> 172.21.15.62:0/3716315486 conn(0x7f6a040b02e0 msgr2=0x7f6a040b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:39.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.317+0000 7f6a09c00640 1 -- 172.21.15.62:0/3716315486 shutdown_connections 2024-03-23T11:24:39.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.317+0000 7f6a09c00640 1 -- 172.21.15.62:0/3716315486 wait complete. 2024-03-23T11:24:39.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.319+0000 7f6a09c00640 1 Processor -- start 2024-03-23T11:24:39.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.319+0000 7f6a09c00640 1 -- start start 2024-03-23T11:24:39.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.320+0000 7f6a09c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e40051d0 0x7f6a0415ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:39.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.320+0000 7f6a09c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a040b30c0 0x7f6a04155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:39.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.320+0000 7f6a09c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a0413dc10 0x7f6a0413e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:39.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.320+0000 7f6a09c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0413e600 con 0x7f6a0413dc10 2024-03-23T11:24:39.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.320+0000 7f6a09c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040ad7a0 con 0x7f69e40051d0 2024-03-23T11:24:39.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.320+0000 7f6a09c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a041f5950 con 0x7f6a040b30c0 2024-03-23T11:24:39.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.320+0000 7f6a03400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a0413dc10 0x7f6a0413e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:39.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.320+0000 7f6a03400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a0413dc10 0x7f6a0413e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50454/0 (socket says 172.21.15.62:50454) 2024-03-23T11:24:39.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.320+0000 7f6a03400640 1 -- 172.21.15.62:0/2335896008 learned_addr learned my addr 172.21.15.62:0/2335896008 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:39.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.321+0000 7f6a02a00640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e40051d0 0x7f6a0415ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:39.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.321+0000 7f69fa000640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a040b30c0 0x7f6a04155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:39.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.321+0000 7f6a03400640 1 -- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a040b30c0 msgr2=0x7f6a04155b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:39.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.321+0000 7f6a03400640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a040b30c0 0x7f6a04155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:39.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.321+0000 7f6a03400640 1 -- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e40051d0 msgr2=0x7f6a0415ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:39.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.321+0000 7f6a03400640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e40051d0 0x7f6a0415ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:39.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.321+0000 7f6a03400640 1 -- 172.21.15.62:0/2335896008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a041f5c30 con 0x7f6a0413dc10 2024-03-23T11:24:39.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.321+0000 7f6a03400640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a0413dc10 0x7f6a0413e0c0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f69fc02bf60 tx=0x7f69fc003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:39.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.321+0000 7f69fb400640 1 -- 172.21.15.62:0/2335896008 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc03d070 con 0x7f6a0413dc10 2024-03-23T11:24:39.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.322+0000 7f69fb400640 1 -- 172.21.15.62:0/2335896008 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc007a20 con 0x7f6a0413dc10 2024-03-23T11:24:39.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.322+0000 7f69fb400640 1 -- 172.21.15.62:0/2335896008 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc043640 con 0x7f6a0413dc10 2024-03-23T11:24:39.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.322+0000 7f6a09c00640 1 -- 172.21.15.62:0/2335896008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a041f5f10 con 0x7f6a0413dc10 2024-03-23T11:24:39.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.322+0000 7f6a09c00640 1 -- 172.21.15.62:0/2335896008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a041560a0 con 0x7f6a0413dc10 2024-03-23T11:24:39.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.323+0000 7f6a09c00640 1 -- 172.21.15.62:0/2335896008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0413e600 con 0x7f6a0413dc10 2024-03-23T11:24:39.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.324+0000 7f69fb400640 1 -- 172.21.15.62:0/2335896008 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f69fc041070 con 0x7f6a0413dc10 2024-03-23T11:24:39.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.325+0000 7f69fb400640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69e006a7d0 0x7f69e006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:39.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.325+0000 7f69fb400640 1 -- 172.21.15.62:0/2335896008 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f69fc0aa1e0 con 0x7f6a0413dc10 2024-03-23T11:24:39.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.325+0000 7f6a02a00640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69e006a7d0 0x7f69e006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:39.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.327+0000 7f6a02a00640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69e006a7d0 0x7f69e006cc90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f69f4001050 tx=0x7f69f4001d10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:39.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.329+0000 7f69fb400640 1 -- 172.21.15.62:0/2335896008 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a0413e600 con 0x7f6a0413dc10 2024-03-23T11:24:39.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.566+0000 7f6a09c00640 1 -- 172.21.15.62:0/2335896008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a0413e600 con 0x7f6a0413dc10 2024-03-23T11:24:39.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.567+0000 7f69fb400640 1 -- 172.21.15.62:0/2335896008 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f6a0413e600 con 0x7f6a0413dc10 2024-03-23T11:24:39.563 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:39.563 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":48,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/29171509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":29171509},{"type":"v1","addr":"172.21.15.120:6837","nonce":29171509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:24:38.269513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6618":{"gid":6618,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:24:39.563 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 48 2024-03-23T11:24:39.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 -- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69e006a7d0 msgr2=0x7f69e006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:39.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69e006a7d0 0x7f69e006cc90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f69f4001050 tx=0x7f69f4001d10 comp rx=0 tx=0).stop 2024-03-23T11:24:39.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 -- 172.21.15.62:0/2335896008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a0413dc10 msgr2=0x7f6a0413e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:39.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a0413dc10 0x7f6a0413e0c0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f69fc02bf60 tx=0x7f69fc003a70 comp rx=0 tx=0).stop 2024-03-23T11:24:39.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 -- 172.21.15.62:0/2335896008 shutdown_connections 2024-03-23T11:24:39.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69e006a7d0 0x7f69e006cc90 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:39.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a0413dc10 0x7f6a0413e0c0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:39.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a040b30c0 0x7f6a04155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:39.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 --2- 172.21.15.62:0/2335896008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e40051d0 0x7f6a0415ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:39.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 -- 172.21.15.62:0/2335896008 >> 172.21.15.62:0/2335896008 conn(0x7f6a040b02e0 msgr2=0x7f6a04158ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:39.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 -- 172.21.15.62:0/2335896008 shutdown_connections 2024-03-23T11:24:39.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:39.570+0000 7f6a09c00640 1 -- 172.21.15.62:0/2335896008 wait complete. 2024-03-23T11:24:39.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:24:11.044296+0000', 'modified': '2024-03-23T11:24:38.269513+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 75, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6618}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6618': {'gid': 6618, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.120:6835/1201988215', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1201988215}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1201988215}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:24:39.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:24:39.987 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-23T11:24:40.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.223+0000 7f5541e00640 1 Processor -- start 2024-03-23T11:24:40.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.224+0000 7f5541e00640 1 -- start start 2024-03-23T11:24:40.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.225+0000 7f5541e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f553c1613a0 0x7f553c06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.225+0000 7f5541e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c06cf40 con 0x7f553c1571e0 2024-03-23T11:24:40.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.225+0000 7f5541e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c1575c0 con 0x7f553c06d910 2024-03-23T11:24:40.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.225+0000 7f5541e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553c06c3a0 con 0x7f553c1613a0 2024-03-23T11:24:40.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.225+0000 7f553aa00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f553c1571e0 0x7f553c0ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43444/0 (socket says 172.21.15.62:43444) 2024-03-23T11:24:40.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.225+0000 7f553aa00640 1 -- 172.21.15.62:0/3069569393 learned_addr learned my addr 172.21.15.62:0/3069569393 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.226+0000 7f5533e00640 1 --2- 172.21.15.62:0/3069569393 >> v2:172.21.15.120:3300/0 conn(0x7f553c1613a0 0x7f553c06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.226+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902260631 0 0) 0x7f553c06cf40 con 0x7f553c1571e0 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.226+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5520003520 con 0x7f553c1571e0 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.226+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2909631716 0 0) 0x7f5520003520 con 0x7f553c1571e0 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.226+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 >> v2:172.21.15.120:3300/0 conn(0x7f553c1613a0 msgr2=0x7f553c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.226+0000 7f553a000640 1 --2- 172.21.15.62:0/3069569393 >> v2:172.21.15.120:3300/0 conn(0x7f553c1613a0 0x7f553c06d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.226+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 >> v1:172.21.15.120:6789/0 conn(0x7f553c06d910 legacy=0x7f553c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.226+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c0ae040 con 0x7f553c1571e0 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f552c001670 con 0x7f553c1571e0 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 >> v1:172.21.15.62:6789/0 conn(0x7f553c1571e0 legacy=0x7f553c0ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f553a000640 1 --2- 172.21.15.62:0/3069569393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5520003970 0x7f5520003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f553a000640 1 --2- 172.21.15.62:0/3069569393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 0x7f5520004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f553a000640 1 --2- 172.21.15.62:0/3069569393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5520005180 0x7f552000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c06c3a0 con 0x7f55200047c0 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552000c120 con 0x7f5520005180 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552000c2e0 con 0x7f5520003970 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f5533e00640 1 --2- 172.21.15.62:0/3069569393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 0x7f5520004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f553aa00640 1 --2- 172.21.15.62:0/3069569393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5520003970 0x7f5520003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f5533e00640 1 -- 172.21.15.62:0/3069569393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5520003970 msgr2=0x7f5520003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f5533e00640 1 --2- 172.21.15.62:0/3069569393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5520003970 0x7f5520003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f5533e00640 1 -- 172.21.15.62:0/3069569393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5520005180 msgr2=0x7f552000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f5533e00640 1 --2- 172.21.15.62:0/3069569393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5520005180 0x7f552000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f5533e00640 1 -- 172.21.15.62:0/3069569393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5520003520 con 0x7f55200047c0 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f553b400640 1 -- 172.21.15.62:0/3069569393 reap_dead start 2024-03-23T11:24:40.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.227+0000 7f553aa00640 1 --2- 172.21.15.62:0/3069569393 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5520003970 0x7f5520003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:40.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.229+0000 7f5533e00640 1 --2- 172.21.15.62:0/3069569393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 0x7f5520004c40 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f553c1617d0 tx=0x7f55200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db9ca7c8a5458d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.229+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552402c660 con 0x7f55200047c0 2024-03-23T11:24:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.229+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552402c7c0 con 0x7f55200047c0 2024-03-23T11:24:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.229+0000 7f553a000640 1 -- 172.21.15.62:0/3069569393 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552402cb70 con 0x7f55200047c0 2024-03-23T11:24:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.229+0000 7f5541e00640 1 -- 172.21.15.62:0/3069569393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 msgr2=0x7f5520004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.229+0000 7f5541e00640 1 --2- 172.21.15.62:0/3069569393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 0x7f5520004c40 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f553c1617d0 tx=0x7f55200057f0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.230+0000 7f5541e00640 1 -- 172.21.15.62:0/3069569393 shutdown_connections 2024-03-23T11:24:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.230+0000 7f5541e00640 1 --2- 172.21.15.62:0/3069569393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 0x7f5520004c40 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.230+0000 7f5541e00640 1 -- 172.21.15.62:0/3069569393 >> 172.21.15.62:0/3069569393 conn(0x7f553c0b02e0 msgr2=0x7f553c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.230+0000 7f5541e00640 1 -- 172.21.15.62:0/3069569393 shutdown_connections 2024-03-23T11:24:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.230+0000 7f5541e00640 1 -- 172.21.15.62:0/3069569393 wait complete. 2024-03-23T11:24:40.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.231+0000 7f5541e00640 1 Processor -- start 2024-03-23T11:24:40.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.232+0000 7f5541e00640 1 -- start start 2024-03-23T11:24:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.232+0000 7f5541e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 0x7f553c137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f5541e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f553c15dae0 0x7f553c15be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f5541e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f553c158310 0x7f553c158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f5541e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c158cd0 con 0x7f55200047c0 2024-03-23T11:24:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f5541e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0cb460 con 0x7f553c158310 2024-03-23T11:24:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f5541e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0cae30 con 0x7f553c15dae0 2024-03-23T11:24:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f553aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 0x7f553c137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f553aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 0x7f553c137c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50480/0 (socket says 172.21.15.62:50480) 2024-03-23T11:24:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f553aa00640 1 -- 172.21.15.62:0/1474391254 learned_addr learned my addr 172.21.15.62:0/1474391254 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f553b400640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f553c158310 0x7f553c158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f5533e00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f553c15dae0 0x7f553c15be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f553aa00640 1 -- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f553c15dae0 msgr2=0x7f553c15be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f553aa00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f553c15dae0 0x7f553c15be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f553aa00640 1 -- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f553c158310 msgr2=0x7f553c158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f553aa00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f553c158310 0x7f553c158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f553aa00640 1 -- 172.21.15.62:0/1474391254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c15c740 con 0x7f55200047c0 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.233+0000 7f553b400640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f553c158310 0x7f553c158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.234+0000 7f553aa00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 0x7f553c137c80 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f552c00d7d0 tx=0x7f552c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.234+0000 7f5533400640 1 -- 172.21.15.62:0/1474391254 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c0074e0 con 0x7f55200047c0 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.234+0000 7f5533400640 1 -- 172.21.15.62:0/1474391254 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c007640 con 0x7f55200047c0 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.234+0000 7f5533400640 1 -- 172.21.15.62:0/1474391254 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c004a00 con 0x7f55200047c0 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.234+0000 7f5541e00640 1 -- 172.21.15.62:0/1474391254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553c15ca20 con 0x7f55200047c0 2024-03-23T11:24:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.234+0000 7f5541e00640 1 -- 172.21.15.62:0/1474391254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553c1381c0 con 0x7f55200047c0 2024-03-23T11:24:40.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.236+0000 7f5533400640 1 -- 172.21.15.62:0/1474391254 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f552c004b60 con 0x7f55200047c0 2024-03-23T11:24:40.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.237+0000 7f5533400640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f551806a7a0 0x7f551806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.237+0000 7f5533400640 1 -- 172.21.15.62:0/1474391254 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f552c088770 con 0x7f55200047c0 2024-03-23T11:24:40.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.238+0000 7f5533e00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f551806a7a0 0x7f551806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.238+0000 7f5544b6d740 1 Processor -- start 2024-03-23T11:24:40.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.238+0000 7f5544b6d740 1 -- start start 2024-03-23T11:24:40.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.239+0000 7f5533e00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f551806a7a0 0x7f551806cc60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5524036000 tx=0x7f5524030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.239+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 0x55a6bc8639c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.239+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc863f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.239+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc85e8b0 0x55a6bc88e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.239+0000 7f5544b6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc7cf110 con 0x55a6bc862410 2024-03-23T11:24:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.239+0000 7f5544b6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc7cc8a0 con 0x55a6bc85e8b0 2024-03-23T11:24:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.239+0000 7f5544b6d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc7c8190 con 0x55a6bc8ac390 2024-03-23T11:24:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.240+0000 7f5533e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc863f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.240+0000 7f5533e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc863f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50496/0 (socket says 172.21.15.62:50496) 2024-03-23T11:24:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.240+0000 7f5533e00640 1 -- 172.21.15.62:0/305098217 learned_addr learned my addr 172.21.15.62:0/305098217 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.240+0000 7f553aa00640 1 --2- 172.21.15.62:0/305098217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 0x55a6bc8639c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.240+0000 7f553b400640 1 --2- 172.21.15.62:0/305098217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc85e8b0 0x55a6bc88e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.240+0000 7f5533e00640 1 -- 172.21.15.62:0/305098217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 msgr2=0x55a6bc8639c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.240+0000 7f5533e00640 1 --2- 172.21.15.62:0/305098217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 0x55a6bc8639c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.240+0000 7f5533e00640 1 -- 172.21.15.62:0/305098217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc85e8b0 msgr2=0x55a6bc88e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.240+0000 7f5533e00640 1 --2- 172.21.15.62:0/305098217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc85e8b0 0x55a6bc88e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.240+0000 7f5533e00640 1 -- 172.21.15.62:0/305098217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6bc793230 con 0x55a6bc862410 2024-03-23T11:24:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.240+0000 7f553b400640 1 --2- 172.21.15.62:0/305098217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc85e8b0 0x55a6bc88e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:40.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.241+0000 7f5533e00640 1 --2- 172.21.15.62:0/305098217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc863f00 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5524037040 tx=0x7f552402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf9c01ae380bdbf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:40.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.242+0000 7f552be00640 1 -- 172.21.15.62:0/305098217 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5524034940 con 0x55a6bc862410 2024-03-23T11:24:40.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.242+0000 7f552be00640 1 -- 172.21.15.62:0/305098217 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5524045070 con 0x55a6bc862410 2024-03-23T11:24:40.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.242+0000 7f552be00640 1 -- 172.21.15.62:0/305098217 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5524034ca0 con 0x55a6bc862410 2024-03-23T11:24:40.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.243+0000 7f5544b6d740 1 -- 172.21.15.62:0/305098217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 msgr2=0x55a6bc863f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.243+0000 7f5544b6d740 1 --2- 172.21.15.62:0/305098217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc863f00 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5524037040 tx=0x7f552402e000 comp rx=0 tx=0).stop 2024-03-23T11:24:40.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.243+0000 7f5544b6d740 1 -- 172.21.15.62:0/305098217 shutdown_connections 2024-03-23T11:24:40.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.243+0000 7f5544b6d740 1 --2- 172.21.15.62:0/305098217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc863f00 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.243+0000 7f5544b6d740 1 --2- 172.21.15.62:0/305098217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc85e8b0 0x55a6bc88e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.243+0000 7f5544b6d740 1 --2- 172.21.15.62:0/305098217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 0x55a6bc8639c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.243+0000 7f5544b6d740 1 -- 172.21.15.62:0/305098217 >> 172.21.15.62:0/305098217 conn(0x55a6bc860b20 msgr2=0x55a6bc8a96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:40.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.243+0000 7f5544b6d740 1 -- 172.21.15.62:0/305098217 shutdown_connections 2024-03-23T11:24:40.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.243+0000 7f5544b6d740 1 -- 172.21.15.62:0/305098217 wait complete. 2024-03-23T11:24:40.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.244+0000 7f5544b6d740 1 -- start start 2024-03-23T11:24:40.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.244+0000 7f5544b6d740 1 Processor -- start 2024-03-23T11:24:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.245+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc8b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.245+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 0x55a6bc8899a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.245+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8925f0 0x55a6bc889ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.245+0000 7f5544b6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc7698c0 con 0x55a6bc862410 2024-03-23T11:24:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.245+0000 7f5544b6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc769560 con 0x55a6bc8925f0 2024-03-23T11:24:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.245+0000 7f5544b6d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc52ff20 con 0x55a6bc8ac390 2024-03-23T11:24:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.245+0000 7f5544b6d740 10 client.0 _ll_register_callbacks cb 0x55a6bc7c1800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:24:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.245+0000 7f553aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc8b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.245+0000 7f553aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc8b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50500/0 (socket says 172.21.15.62:50500) 2024-03-23T11:24:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.245+0000 7f553aa00640 1 -- 172.21.15.62:0/2458727161 learned_addr learned my addr 172.21.15.62:0/2458727161 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.246+0000 7f553aa00640 1 -- 172.21.15.62:0/2458727161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 msgr2=0x55a6bc8899a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.246+0000 7f553aa00640 1 --2- 172.21.15.62:0/2458727161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 0x55a6bc8899a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.246+0000 7f553aa00640 1 -- 172.21.15.62:0/2458727161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8925f0 msgr2=0x55a6bc889ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.246+0000 7f553aa00640 1 --2- 172.21.15.62:0/2458727161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8925f0 0x55a6bc889ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.246+0000 7f553aa00640 1 -- 172.21.15.62:0/2458727161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5520003520 con 0x55a6bc862410 2024-03-23T11:24:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.246+0000 7f553aa00640 1 --2- 172.21.15.62:0/2458727161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc8b1580 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f552c001a60 tx=0x7f552c057f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14fba9dd58abc9b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.246+0000 7f5528c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:24:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.246+0000 7f5528c00640 1 -- 172.21.15.62:0/2458727161 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c05b360 con 0x55a6bc862410 2024-03-23T11:24:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.246+0000 7f5528c00640 1 -- 172.21.15.62:0/2458727161 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(75..75 src has 1..75) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f552c05fb40 con 0x55a6bc862410 2024-03-23T11:24:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.246+0000 7f5544b6d740 1 -- 172.21.15.62:0/2458727161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a6bc7f2280 con 0x55a6bc862410 2024-03-23T11:24:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.247+0000 7f5528c00640 1 -- 172.21.15.62:0/2458727161 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f552c06f170 con 0x55a6bc862410 2024-03-23T11:24:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.247+0000 7f5544b6d740 10 client.6879 fetch_fsmap learned FSMap version 48 2024-03-23T11:24:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.247+0000 7f5544b6d740 1 -- 172.21.15.62:0/2458727161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x55a6bc7c6810 con 0x55a6bc862410 2024-03-23T11:24:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.247+0000 7f5528c00640 1 -- 172.21.15.62:0/2458727161 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a6bc7f2280 con 0x55a6bc862410 2024-03-23T11:24:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.247+0000 7f5528c00640 1 -- 172.21.15.62:0/2458727161 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f552c05f6b0 con 0x55a6bc862410 2024-03-23T11:24:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.247+0000 7f5544b6d740 10 client.6879 fetch_fsmap finished waiting for FSMap version 48 2024-03-23T11:24:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.247+0000 7f5544b6d740 10 client.6879 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:24:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.247+0000 7f5544b6d740 20 client.6879 populate_metadata read hostname 'smithi062' 2024-03-23T11:24:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.247+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2458727161 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55a6bc8c4860 0x55a6bc8c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.247+0000 7f5544b6d740 4 client.6879 mds_command: new command op to 6618 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:24:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.247+0000 7f5544b6d740 1 -- 172.21.15.62:0/2458727161 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6bc7ba9d0 con 0x55a6bc8c4860 2024-03-23T11:24:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.248+0000 7f5533e00640 1 --2- 172.21.15.62:0/2458727161 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55a6bc8c4860 0x55a6bc8c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.249+0000 7f5533e00640 1 --2- 172.21.15.62:0/2458727161 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55a6bc8c4860 0x55a6bc8c6c50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be019fe9c14fc7bb server_cookie=1351c85e6f2c9c49 in_seq=0 out_seq=0 2024-03-23T11:24:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.249+0000 7f5528c00640 10 client.6879 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.251+0000 7f5528c00640 1 -- 172.21.15.62:0/2458727161 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55a6bc7ba9d0 con 0x55a6bc8c4860 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.251+0000 7f5528c00640 10 client.6879 handle_command_reply: tid=0 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.251+0000 7f5544b6d740 1 client.6879 shutdown 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.251+0000 7f5544b6d740 1 -- 172.21.15.62:0/2458727161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 msgr2=0x55a6bc8b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.251+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2458727161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc8b1580 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f552c001a60 tx=0x7f552c057f90 comp rx=0 tx=0).stop 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.251+0000 7f5544b6d740 1 -- 172.21.15.62:0/2458727161 shutdown_connections 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.251+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2458727161 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55a6bc8c4860 0x55a6bc8c6c50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.252+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2458727161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8925f0 0x55a6bc889ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.252+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2458727161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 0x55a6bc8899a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.252+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2458727161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc8b1580 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.252+0000 7f5528c00640 0 client.6879 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.252+0000 7f5544b6d740 1 -- 172.21.15.62:0/2458727161 >> 172.21.15.62:0/2458727161 conn(0x55a6bc85e8b0 msgr2=0x55a6bc862b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.252+0000 7f5544b6d740 1 -- 172.21.15.62:0/2458727161 shutdown_connections 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.252+0000 7f5544b6d740 1 -- 172.21.15.62:0/2458727161 wait complete. 2024-03-23T11:24:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.252+0000 7f5544b6d740 20 client.6879 trim_cache size 0 max 16384 2024-03-23T11:24:40.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.269+0000 7f5544b6d740 1 Processor -- start 2024-03-23T11:24:40.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.269+0000 7f5544b6d740 1 -- start start 2024-03-23T11:24:40.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.269+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc85d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8925f0 0x55a6bc885820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 0x55a6bc885d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f5544b6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc864710 con 0x55a6bc862410 2024-03-23T11:24:40.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f5544b6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc862880 con 0x55a6bc8925f0 2024-03-23T11:24:40.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f5544b6d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc88cda0 con 0x55a6bc8ac390 2024-03-23T11:24:40.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f553aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc85d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f553aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc85d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50506/0 (socket says 172.21.15.62:50506) 2024-03-23T11:24:40.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f553aa00640 1 -- 172.21.15.62:0/736596568 learned_addr learned my addr 172.21.15.62:0/736596568 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:40.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f553aa00640 1 -- 172.21.15.62:0/736596568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 msgr2=0x55a6bc885d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f553aa00640 1 --2- 172.21.15.62:0/736596568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 0x55a6bc885d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f553aa00640 1 -- 172.21.15.62:0/736596568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8925f0 msgr2=0x55a6bc885820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f553aa00640 1 --2- 172.21.15.62:0/736596568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8925f0 0x55a6bc885820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.270+0000 7f553aa00640 1 -- 172.21.15.62:0/736596568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6bc768ad0 con 0x55a6bc862410 2024-03-23T11:24:40.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.271+0000 7f553aa00640 1 --2- 172.21.15.62:0/736596568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc85d870 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f552c00d7a0 tx=0x7f552c05a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0fee3e3986a00b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:40.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.271+0000 7f5530c00640 1 -- 172.21.15.62:0/736596568 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c0014c0 con 0x55a6bc862410 2024-03-23T11:24:40.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.271+0000 7f5530c00640 1 -- 172.21.15.62:0/736596568 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c06f170 con 0x55a6bc862410 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.271+0000 7f5530c00640 1 -- 172.21.15.62:0/736596568 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c05b360 con 0x55a6bc862410 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.271+0000 7f5544b6d740 1 -- 172.21.15.62:0/736596568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 msgr2=0x55a6bc85d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.271+0000 7f5544b6d740 1 --2- 172.21.15.62:0/736596568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc85d870 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f552c00d7a0 tx=0x7f552c05a000 comp rx=0 tx=0).stop 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.272+0000 7f5544b6d740 1 -- 172.21.15.62:0/736596568 shutdown_connections 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.272+0000 7f5544b6d740 1 --2- 172.21.15.62:0/736596568 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8ac390 0x55a6bc885d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.272+0000 7f5544b6d740 1 --2- 172.21.15.62:0/736596568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8925f0 0x55a6bc885820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.272+0000 7f5544b6d740 1 --2- 172.21.15.62:0/736596568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc85d870 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.272+0000 7f5544b6d740 1 -- 172.21.15.62:0/736596568 >> 172.21.15.62:0/736596568 conn(0x55a6bc85e8b0 msgr2=0x55a6bc889cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.272+0000 7f5544b6d740 1 -- 172.21.15.62:0/736596568 shutdown_connections 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.272+0000 7f5544b6d740 1 -- 172.21.15.62:0/736596568 wait complete. 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.272+0000 7f5544b6d740 1 -- start start 2024-03-23T11:24:40.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.273+0000 7f5544b6d740 1 Processor -- start 2024-03-23T11:24:40.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc885d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8925f0 0x55a6bc8845d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f5544b6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8ac390 0x55a6bc884b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f5544b6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc885950 con 0x55a6bc862410 2024-03-23T11:24:40.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f5544b6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc885360 con 0x55a6bc8ac390 2024-03-23T11:24:40.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f5544b6d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6bc85a0b0 con 0x55a6bc8925f0 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f5544b6d740 10 client.0 _ll_register_callbacks cb 0x7f552c053a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f553aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc885d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f553aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc885d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50512/0 (socket says 172.21.15.62:50512) 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f553aa00640 1 -- 172.21.15.62:0/2772289304 learned_addr learned my addr 172.21.15.62:0/2772289304 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f553b400640 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8ac390 0x55a6bc884b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f553aa00640 1 -- 172.21.15.62:0/2772289304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8925f0 msgr2=0x55a6bc8845d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f553aa00640 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8925f0 0x55a6bc8845d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f553aa00640 1 -- 172.21.15.62:0/2772289304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8ac390 msgr2=0x55a6bc884b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f553aa00640 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8ac390 0x55a6bc884b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.274+0000 7f553aa00640 1 -- 172.21.15.62:0/2772289304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6bc7c6810 con 0x55a6bc862410 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.275+0000 7f553aa00640 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc885d70 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f552c0576e0 tx=0x7f552c086a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd8ffcb61a17fd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:40.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.275+0000 7f5528c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:24:40.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.277+0000 7f5528c00640 1 -- 172.21.15.62:0/2772289304 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c057290 con 0x55a6bc862410 2024-03-23T11:24:40.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.277+0000 7f5528c00640 1 -- 172.21.15.62:0/2772289304 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(75..75 src has 1..75) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f552c05fb40 con 0x55a6bc862410 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.277+0000 7f5544b6d740 1 -- 172.21.15.62:0/2772289304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5524045070 con 0x55a6bc862410 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.277+0000 7f5528c00640 1 -- 172.21.15.62:0/2772289304 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f552c0014c0 con 0x55a6bc862410 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.277+0000 7f5544b6d740 10 client.6885 fetch_fsmap learned FSMap version 48 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.277+0000 7f5544b6d740 1 -- 172.21.15.62:0/2772289304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x55a6bc78dfe0 con 0x55a6bc862410 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.277+0000 7f5528c00640 1 -- 172.21.15.62:0/2772289304 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c063020 con 0x55a6bc862410 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.278+0000 7f5528c00640 1 -- 172.21.15.62:0/2772289304 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f552c053540 con 0x55a6bc862410 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.278+0000 7f5544b6d740 10 client.6885 fetch_fsmap finished waiting for FSMap version 48 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.278+0000 7f5544b6d740 10 client.6885 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.278+0000 7f5544b6d740 20 client.6885 populate_metadata read hostname 'smithi062' 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.278+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55a6bc8aa910 0x55a6bc8a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.278+0000 7f5544b6d740 4 client.6885 mds_command: new command op to 6618 tid=0[{"prefix": "status"}] 2024-03-23T11:24:40.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.278+0000 7f5544b6d740 1 -- 172.21.15.62:0/2772289304 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a6bc4bc3c0 con 0x55a6bc8aa910 2024-03-23T11:24:40.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.278+0000 7f5533e00640 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55a6bc8aa910 0x55a6bc8a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.279+0000 7f5533e00640 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55a6bc8aa910 0x55a6bc8a84c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9885792063c656a server_cookie=fd74bbc9ffda7c82 in_seq=0 out_seq=0 2024-03-23T11:24:40.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.279+0000 7f5528c00640 10 client.6885 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:40.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5528c00640 1 -- 172.21.15.62:0/2772289304 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55a6bc4bc3c0 con 0x55a6bc8aa910 2024-03-23T11:24:40.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5528c00640 10 client.6885 handle_command_reply: tid=0 2024-03-23T11:24:40.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5544b6d740 1 client.6885 shutdown 2024-03-23T11:24:40.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5544b6d740 1 -- 172.21.15.62:0/2772289304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 msgr2=0x55a6bc885d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc885d70 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f552c0576e0 tx=0x7f552c086a70 comp rx=0 tx=0).stop 2024-03-23T11:24:40.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5544b6d740 1 -- 172.21.15.62:0/2772289304 shutdown_connections 2024-03-23T11:24:40.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55a6bc8aa910 0x55a6bc8a84c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a6bc8ac390 0x55a6bc884b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a6bc8925f0 0x55a6bc8845d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5544b6d740 1 --2- 172.21.15.62:0/2772289304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6bc862410 0x55a6bc885d70 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5528c00640 0 client.6885 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:40.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5544b6d740 1 -- 172.21.15.62:0/2772289304 >> 172.21.15.62:0/2772289304 conn(0x55a6bc85e8b0 msgr2=0x55a6bc85d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:40.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5544b6d740 1 -- 172.21.15.62:0/2772289304 shutdown_connections 2024-03-23T11:24:40.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.280+0000 7f5544b6d740 1 -- 172.21.15.62:0/2772289304 wait complete. 2024-03-23T11:24:40.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.281+0000 7f5544b6d740 20 client.6885 trim_cache size 0 max 16384 2024-03-23T11:24:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:24:40.278 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:24:40.278 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:24:40.278 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 6618, 2024-03-23T11:24:40.278 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:24:40.278 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:24:40.278 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:24:40.278 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 5.0353284179999998, 2024-03-23T11:24:40.279 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 48, 2024-03-23T11:24:40.279 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 75, 2024-03-23T11:24:40.279 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 75, 2024-03-23T11:24:40.279 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 35.073501344 2024-03-23T11:24:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:24:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.281+0000 7f5541e00640 1 -- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f551806a7a0 msgr2=0x7f551806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.281+0000 7f5541e00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f551806a7a0 0x7f551806cc60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5524036000 tx=0x7f5524030000 comp rx=0 tx=0).stop 2024-03-23T11:24:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.281+0000 7f5541e00640 1 -- 172.21.15.62:0/1474391254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 msgr2=0x7f553c137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.281+0000 7f5541e00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 0x7f553c137c80 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f552c00d7d0 tx=0x7f552c00dc90 comp rx=0 tx=0).stop 2024-03-23T11:24:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.281+0000 7f5541e00640 1 -- 172.21.15.62:0/1474391254 shutdown_connections 2024-03-23T11:24:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.281+0000 7f5541e00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f551806a7a0 0x7f551806cc60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.281+0000 7f5541e00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f553c158310 0x7f553c158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.281+0000 7f5541e00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f553c15dae0 0x7f553c15be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.282+0000 7f5541e00640 1 --2- 172.21.15.62:0/1474391254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55200047c0 0x7f553c137c80 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.282+0000 7f5541e00640 1 -- 172.21.15.62:0/1474391254 >> 172.21.15.62:0/1474391254 conn(0x7f553c0b02e0 msgr2=0x7f553c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.282+0000 7f5541e00640 1 -- 172.21.15.62:0/1474391254 shutdown_connections 2024-03-23T11:24:40.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.282+0000 7f5541e00640 1 -- 172.21.15.62:0/1474391254 wait complete. 2024-03-23T11:24:40.295 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:24:40.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.513+0000 7f441f200640 1 Processor -- start 2024-03-23T11:24:40.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.513+0000 7f441f200640 1 -- start start 2024-03-23T11:24:40.509 INFO:tasks.ceph.mds.b.smithi120.stderr:2024-03-23T11:24:40.514+0000 7f4efae00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:24:40.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.515+0000 7f441f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4420143dd0 0x7f442015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.515+0000 7f441f200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4420070090 0x7f44200704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.515+0000 7f441f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4420070a30 con 0x7f44200b2820 2024-03-23T11:24:40.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.515+0000 7f441f200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44200cd140 con 0x7f4420070090 2024-03-23T11:24:40.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.515+0000 7f441f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44200cd3e0 con 0x7f4420143dd0 2024-03-23T11:24:40.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.515+0000 7f441d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4420070090 0x7f44200704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.515+0000 7f441d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4420070090 0x7f44200704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50522/0 (socket says 172.21.15.62:50522) 2024-03-23T11:24:40.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.515+0000 7f441d400640 1 -- 172.21.15.62:0/2496092589 learned_addr learned my addr 172.21.15.62:0/2496092589 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:40.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.516+0000 7f441de00640 1 --2- 172.21.15.62:0/2496092589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4420143dd0 0x7f442015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.516+0000 7f441d400640 1 -- 172.21.15.62:0/2496092589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4420143dd0 msgr2=0x7f442015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.516+0000 7f441d400640 1 --2- 172.21.15.62:0/2496092589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4420143dd0 0x7f442015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.516+0000 7f441d400640 1 -- 172.21.15.62:0/2496092589 >> v1:172.21.15.120:6789/0 conn(0x7f44200b2820 legacy=0x7f44200b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:40.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.516+0000 7f441d400640 1 -- 172.21.15.62:0/2496092589 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44200cd6c0 con 0x7f4420070090 2024-03-23T11:24:40.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.517+0000 7f441d400640 1 --2- 172.21.15.62:0/2496092589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4420070090 0x7f44200704f0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f440c009870 tx=0x7f440c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5518f7bddfff34cf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:40.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.517+0000 7f441ca00640 1 -- 172.21.15.62:0/2496092589 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c031640 con 0x7f4420070090 2024-03-23T11:24:40.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.517+0000 7f441ca00640 1 -- 172.21.15.62:0/2496092589 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c0317a0 con 0x7f4420070090 2024-03-23T11:24:40.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.517+0000 7f441ca00640 1 -- 172.21.15.62:0/2496092589 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c031b30 con 0x7f4420070090 2024-03-23T11:24:40.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.518+0000 7f441f200640 1 -- 172.21.15.62:0/2496092589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4420070090 msgr2=0x7f44200704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.518+0000 7f441f200640 1 --2- 172.21.15.62:0/2496092589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4420070090 0x7f44200704f0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f440c009870 tx=0x7f440c02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.518+0000 7f441f200640 1 -- 172.21.15.62:0/2496092589 shutdown_connections 2024-03-23T11:24:40.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.518+0000 7f441f200640 1 --2- 172.21.15.62:0/2496092589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4420070090 0x7f44200704f0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.518+0000 7f441f200640 1 --2- 172.21.15.62:0/2496092589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4420143dd0 0x7f442015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.518+0000 7f441f200640 1 -- 172.21.15.62:0/2496092589 >> 172.21.15.62:0/2496092589 conn(0x7f4420066050 msgr2=0x7f4420069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:40.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.518+0000 7f441f200640 1 -- 172.21.15.62:0/2496092589 shutdown_connections 2024-03-23T11:24:40.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.518+0000 7f441f200640 1 -- 172.21.15.62:0/2496092589 wait complete. 2024-03-23T11:24:40.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.521+0000 7f441f200640 1 Processor -- start 2024-03-23T11:24:40.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.521+0000 7f441f200640 1 -- start start 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.522+0000 7f441f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4420070090 0x7f44200c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.522+0000 7f441f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44200b2820 0x7f4420159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.522+0000 7f441f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4420143dd0 0x7f442015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.522+0000 7f441f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4420135c80 con 0x7f44200b2820 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.522+0000 7f441f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4420135970 con 0x7f4420143dd0 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.522+0000 7f441f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44200c4890 con 0x7f4420070090 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.522+0000 7f441d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44200b2820 0x7f4420159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.522+0000 7f441d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44200b2820 0x7f4420159cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50538/0 (socket says 172.21.15.62:50538) 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.522+0000 7f441d400640 1 -- 172.21.15.62:0/2923810040 learned_addr learned my addr 172.21.15.62:0/2923810040 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.523+0000 7f441de00640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4420070090 0x7f44200c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.523+0000 7f441e800640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4420143dd0 0x7f442015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.523+0000 7f441d400640 1 -- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4420070090 msgr2=0x7f44200c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.523+0000 7f441d400640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4420070090 0x7f44200c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.523+0000 7f441d400640 1 -- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4420143dd0 msgr2=0x7f442015b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.523+0000 7f441d400640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4420143dd0 0x7f442015b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.523+0000 7f441d400640 1 -- 172.21.15.62:0/2923810040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440c0094d0 con 0x7f44200b2820 2024-03-23T11:24:40.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.523+0000 7f441e800640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4420143dd0 0x7f442015b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:24:40.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.523+0000 7f441d400640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44200b2820 0x7f4420159cf0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f440c000c00 tx=0x7f440c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:40.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.524+0000 7f4416000640 1 -- 172.21.15.62:0/2923810040 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c03ac60 con 0x7f44200b2820 2024-03-23T11:24:40.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.524+0000 7f4416000640 1 -- 172.21.15.62:0/2923810040 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c03adc0 con 0x7f44200b2820 2024-03-23T11:24:40.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.524+0000 7f4416000640 1 -- 172.21.15.62:0/2923810040 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c044630 con 0x7f44200b2820 2024-03-23T11:24:40.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.524+0000 7f441f200640 1 -- 172.21.15.62:0/2923810040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44200c4b10 con 0x7f44200b2820 2024-03-23T11:24:40.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.524+0000 7f441f200640 1 -- 172.21.15.62:0/2923810040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44200c50d0 con 0x7f44200b2820 2024-03-23T11:24:40.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.525+0000 7f441f200640 1 -- 172.21.15.62:0/2923810040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44200c52c0 con 0x7f44200b2820 2024-03-23T11:24:40.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.526+0000 7f4416000640 1 -- 172.21.15.62:0/2923810040 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f440c02f050 con 0x7f44200b2820 2024-03-23T11:24:40.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.527+0000 7f4416000640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440006a7d0 0x7f440006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:40.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.527+0000 7f4416000640 1 -- 172.21.15.62:0/2923810040 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f440c0aacb0 con 0x7f44200b2820 2024-03-23T11:24:40.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.528+0000 7f441de00640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440006a7d0 0x7f440006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:40.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.529+0000 7f441de00640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440006a7d0 0x7f440006cc90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f44100079a0 tx=0x7f4410008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:40.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.534+0000 7f4416000640 1 -- 172.21.15.62:0/2923810040 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f440c075fd0 con 0x7f44200b2820 2024-03-23T11:24:40.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.768+0000 7f441f200640 1 -- 172.21.15.62:0/2923810040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f44200c52c0 con 0x7f44200b2820 2024-03-23T11:24:40.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.769+0000 7f4416000640 1 -- 172.21.15.62:0/2923810040 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v48) v1 ==== 53+0+73 (secure 0 0 0) 0x7f440c035020 con 0x7f44200b2820 2024-03-23T11:24:40.766 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:24:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.772+0000 7f441f200640 1 -- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440006a7d0 msgr2=0x7f440006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.772+0000 7f441f200640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440006a7d0 0x7f440006cc90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f44100079a0 tx=0x7f4410008040 comp rx=0 tx=0).stop 2024-03-23T11:24:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.772+0000 7f441f200640 1 -- 172.21.15.62:0/2923810040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44200b2820 msgr2=0x7f4420159cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.772+0000 7f441f200640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44200b2820 0x7f4420159cf0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f440c000c00 tx=0x7f440c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.772+0000 7f441f200640 1 -- 172.21.15.62:0/2923810040 shutdown_connections 2024-03-23T11:24:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.772+0000 7f441f200640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f440006a7d0 0x7f440006cc90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.772+0000 7f441f200640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44200b2820 0x7f4420159cf0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.772+0000 7f441f200640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4420143dd0 0x7f442015b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.772+0000 7f441f200640 1 --2- 172.21.15.62:0/2923810040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4420070090 0x7f44200c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.772+0000 7f441f200640 1 -- 172.21.15.62:0/2923810040 >> 172.21.15.62:0/2923810040 conn(0x7f4420066050 msgr2=0x7f442006d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.772+0000 7f441f200640 1 -- 172.21.15.62:0/2923810040 shutdown_connections 2024-03-23T11:24:40.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:40.773+0000 7f441f200640 1 -- 172.21.15.62:0/2923810040 wait complete. 2024-03-23T11:24:40.787 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:24:40.787 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:24:40.787 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:24:40.787 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:24:40.787 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:40.787 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:24:40.787 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:24:40.787 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:24:40.787 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:24:40.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:40.788 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:24:40.839 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:24:40.839 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:24:40.839 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:24:40.839 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:40.839 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:24:40.839 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:40.839 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42276sec preferred_lft 42276sec 2024-03-23T11:24:40.840 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:40.841 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:24:40.842 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:24:40.842 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:24:40.842 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:24:40.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:40.842 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:24:40.933 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:24:40.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:40.937 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:24:40.990 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:24:40.990 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:24:40.990 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:24:40.990 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:24:40.990 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:24:40.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:40.991 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:24:40.991 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:24:40.992 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:24:40.992 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:24:40.992 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:24:40.992 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:24:41.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:41.190 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:24:41.243 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:24:41.243 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:24:41.244 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:24:41.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:41.244 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:24:41.335 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:24:41.336 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:24:41.336 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:41.390 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:24:41.390 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:24:41.390 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:41.441 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:24:41.540 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:24:41.540 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:24:41.641 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:24:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:24:41.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:41.646 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:24:41.703 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:24:41.704 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:24:41.708 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:24:41.785 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:24:41.785 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:24:41.864 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:24:41.868 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:24:41.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:41.869 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:24:41.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:24:41.908+0000 7fd5eca47480 -1 init, newargv = 0x5607ace46640 newargc=15 2024-03-23T11:24:41.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[61619]: starting ceph client 2024-03-23T11:24:41.914 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[61619]: starting fuse 2024-03-23T11:24:43.015 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:24:43.120 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:24:43.120 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:24:43.207 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:24:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:24:43.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:43.211 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:24:43.267 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:24:43.268 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:24:43.268 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:24:43.268 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:24:43.268 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:24:43.268 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:24:43.268 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:24:43.268 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:24:43.268 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:24:43.268 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:24:43.269 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:24:43.270 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:24:43.270 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:24:43.270 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:24:43.270 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:24:43.454 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.61619.asok 2024-03-23T11:24:43.633 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.61619.asok 2024-03-23T11:24:43.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:43.634 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.61619.asok status 2024-03-23T11:24:43.884 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "61619", 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 6900, 2024-03-23T11:24:43.885 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 6900 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1888163194 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:24:43.886 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:24:43.887 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1888163194 2024-03-23T11:24:43.887 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:24:43.887 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.6900 192.168.0.1:0/1888163194", 2024-03-23T11:24:43.887 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/1888163194", 2024-03-23T11:24:43.887 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:24:43.887 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 48, 2024-03-23T11:24:43.887 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 75, 2024-03-23T11:24:43.887 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:24:43.887 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:24:43.887 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:24:43.887 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:24:43.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:43.901 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:24:43.901 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:43.958 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:24:43.958 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:43.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:43.958 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:44.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:44.059 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir/) 2024-03-23T11:24:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:file_undamaged 2024-03-23T11:24:44.129 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:24:44.129 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:24:44.129 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:24:44.130 DEBUG:teuthology.orchestra.run.smithi062:> import json 2024-03-23T11:24:44.130 DEBUG:teuthology.orchestra.run.smithi062:> import sys 2024-03-23T11:24:44.130 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:24:44.130 DEBUG:teuthology.orchestra.run.smithi062:> try: 2024-03-23T11:24:44.130 DEBUG:teuthology.orchestra.run.smithi062:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-03-23T11:24:44.130 DEBUG:teuthology.orchestra.run.smithi062:> except OSError as e: 2024-03-23T11:24:44.130 DEBUG:teuthology.orchestra.run.smithi062:> sys.exit(e.errno) 2024-03-23T11:24:44.130 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:24:44.130 DEBUG:teuthology.orchestra.run.smithi062:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-03-23T11:24:44.130 DEBUG:teuthology.orchestra.run.smithi062:> print(json.dumps( 2024-03-23T11:24:44.131 DEBUG:teuthology.orchestra.run.smithi062:> dict([(a, getattr(s, a)) for a in attrs]), 2024-03-23T11:24:44.131 DEBUG:teuthology.orchestra.run.smithi062:> indent=2)) 2024-03-23T11:24:44.131 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:24:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:daemon-helper: command failed with exit status 2 2024-03-23T11:24:44.458 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-23T11:24:44.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:44.459 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-03-23T11:24:44.565 INFO:teuthology.orchestra.run.smithi062.stdout:2 2024-03-23T11:24:44.565 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:24:44.565 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:24:44.619 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:24:44.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.848+0000 7f2f40e00640 1 Processor -- start 2024-03-23T11:24:44.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.848+0000 7f2f40e00640 1 -- start start 2024-03-23T11:24:44.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.849+0000 7f2f40e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2f3c0ceb00 0x7f2f3c0ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:44.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.849+0000 7f2f40e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f3c0ce0f0 0x7f2f3c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:44.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.849+0000 7f2f40e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3c0cf420 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.849+0000 7f2f40e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f3c0cbc40 con 0x7f2f3c0ceb00 2024-03-23T11:24:44.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.849+0000 7f2f40e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c15c3a0 con 0x7f2f3c0ce0f0 2024-03-23T11:24:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.850+0000 7f2f3b400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2f3c0ceb00 0x7f2f3c0ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.850+0000 7f2f3be00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f2f3c0b12b0 0x7f2f3c15e970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:43072/0 (socket says 172.21.15.62:43072) 2024-03-23T11:24:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.850+0000 7f2f3be00640 1 -- 172.21.15.62:0/3641439073 learned_addr learned my addr 172.21.15.62:0/3641439073 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.850+0000 7f2f3aa00640 1 --2- 172.21.15.62:0/3641439073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f3c0ce0f0 0x7f2f3c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.851+0000 7f2f3a000640 1 -- 172.21.15.62:0/3641439073 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552389686 0 0) 0x7f2f3c0cf420 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.851+0000 7f2f3a000640 1 -- 172.21.15.62:0/3641439073 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f20003520 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.851+0000 7f2f3aa00640 1 -- 172.21.15.62:0/3641439073 >> v2:172.21.15.120:3300/0 conn(0x7f2f3c0ceb00 msgr2=0x7f2f3c0ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.851+0000 7f2f3aa00640 1 --2- 172.21.15.62:0/3641439073 >> v2:172.21.15.120:3300/0 conn(0x7f2f3c0ceb00 0x7f2f3c0ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.851+0000 7f2f3aa00640 1 -- 172.21.15.62:0/3641439073 >> v1:172.21.15.120:6789/0 conn(0x7f2f3c0b12b0 legacy=0x7f2f3c15e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.851+0000 7f2f3aa00640 1 -- 172.21.15.62:0/3641439073 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c0ce660 con 0x7f2f3c0ce0f0 2024-03-23T11:24:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.851+0000 7f2f3b400640 1 --2- 172.21.15.62:0/3641439073 >> v2:172.21.15.120:3300/0 conn(0x7f2f3c0ceb00 0x7f2f3c0ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:24:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.853+0000 7f2f3aa00640 1 --2- 172.21.15.62:0/3641439073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f3c0ce0f0 0x7f2f3c15e430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2f280097a0 tx=0x7f2f2802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13fa3e2f54289e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.853+0000 7f2f3a000640 1 -- 172.21.15.62:0/3641439073 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28031830 con 0x7f2f3c0ce0f0 2024-03-23T11:24:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.853+0000 7f2f3a000640 1 -- 172.21.15.62:0/3641439073 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f28031990 con 0x7f2f3c0ce0f0 2024-03-23T11:24:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.853+0000 7f2f3a000640 1 -- 172.21.15.62:0/3641439073 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28031d20 con 0x7f2f3c0ce0f0 2024-03-23T11:24:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.854+0000 7f2f40e00640 1 -- 172.21.15.62:0/3641439073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f3c0ce0f0 msgr2=0x7f2f3c15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.854+0000 7f2f40e00640 1 --2- 172.21.15.62:0/3641439073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f3c0ce0f0 0x7f2f3c15e430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2f280097a0 tx=0x7f2f2802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:24:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.854+0000 7f2f40e00640 1 -- 172.21.15.62:0/3641439073 shutdown_connections 2024-03-23T11:24:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.854+0000 7f2f40e00640 1 --2- 172.21.15.62:0/3641439073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f3c0ce0f0 0x7f2f3c15e430 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.854+0000 7f2f40e00640 1 --2- 172.21.15.62:0/3641439073 >> v2:172.21.15.120:3300/0 conn(0x7f2f3c0ceb00 0x7f2f3c0ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.854+0000 7f2f40e00640 1 -- 172.21.15.62:0/3641439073 >> 172.21.15.62:0/3641439073 conn(0x7f2f3c0b2010 msgr2=0x7f2f3c0b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.854+0000 7f2f40e00640 1 -- 172.21.15.62:0/3641439073 shutdown_connections 2024-03-23T11:24:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.854+0000 7f2f40e00640 1 -- 172.21.15.62:0/3641439073 wait complete. 2024-03-23T11:24:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.856+0000 7f2f40e00640 1 Processor -- start 2024-03-23T11:24:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.856+0000 7f2f40e00640 1 -- start start 2024-03-23T11:24:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f40e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f3c0b12b0 0x7f2f3c0c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f40e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f3c0ce0f0 0x7f2f3c1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f40e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f3c0ceb00 0x7f2f3c0c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f40e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0c7a70 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f40e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0c6b60 con 0x7f2f3c0ceb00 2024-03-23T11:24:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f40e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0c6e40 con 0x7f2f3c0ce0f0 2024-03-23T11:24:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f3b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f3c0b12b0 0x7f2f3c0c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f3b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f3c0b12b0 0x7f2f3c0c91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50548/0 (socket says 172.21.15.62:50548) 2024-03-23T11:24:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f3b400640 1 -- 172.21.15.62:0/1446691637 learned_addr learned my addr 172.21.15.62:0/1446691637 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:44.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f3aa00640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f3c0ce0f0 0x7f2f3c1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:44.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f3b400640 1 -- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f3c0ce0f0 msgr2=0x7f2f3c1579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:44.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f3b400640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f3c0ce0f0 0x7f2f3c1579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:44.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f3be00640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f3c0ceb00 0x7f2f3c0c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:44.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f3b400640 1 -- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f3c0ceb00 msgr2=0x7f2f3c0c7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:44.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.857+0000 7f2f3b400640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f3c0ceb00 0x7f2f3c0c7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.858+0000 7f2f3b400640 1 -- 172.21.15.62:0/1446691637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c009400 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.858+0000 7f2f3be00640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f3c0ceb00 0x7f2f3c0c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.858+0000 7f2f3b400640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f3c0b12b0 0x7f2f3c0c91d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00ebe0 tx=0x7f2f2c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.858+0000 7f2f33400640 1 -- 172.21.15.62:0/1446691637 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c009760 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.858+0000 7f2f33400640 1 -- 172.21.15.62:0/1446691637 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c00cc30 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.858+0000 7f2f33400640 1 -- 172.21.15.62:0/1446691637 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c00eee0 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.858+0000 7f2f40e00640 1 -- 172.21.15.62:0/1446691637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f28009400 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.858+0000 7f2f40e00640 1 -- 172.21.15.62:0/1446691637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3c0c7120 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.859+0000 7f2f40e00640 1 -- 172.21.15.62:0/1446691637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3c0c7ce0 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.860+0000 7f2f33400640 1 -- 172.21.15.62:0/1446691637 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2f2c013470 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.861+0000 7f2f33400640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f1c06a7a0 0x7f2f1c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.861+0000 7f2f33400640 1 -- 172.21.15.62:0/1446691637 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2f2c089150 con 0x7f2f3c0b12b0 2024-03-23T11:24:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.862+0000 7f2f3aa00640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f1c06a7a0 0x7f2f1c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.863+0000 7f2f3aa00640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f1c06a7a0 0x7f2f1c06cc60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2f28036000 tx=0x7f2f2803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:44.866+0000 7f2f33400640 1 -- 172.21.15.62:0/1446691637 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f2c0544d0 con 0x7f2f3c0b12b0 2024-03-23T11:24:45.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.064+0000 7f2f40e00640 1 -- 172.21.15.62:0/1446691637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f3c066bb0 con 0x7f2f3c0b12b0 2024-03-23T11:24:45.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.067+0000 7f2f33400640 1 -- 172.21.15.62:0/1446691637 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16329 (secure 0 0 0) 0x7f2f2c057af0 con 0x7f2f3c0b12b0 2024-03-23T11:24:45.062 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:45.062 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":77,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:44.311392+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:24:08.688706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-23T11:24:10.072774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1c","mappings":[{"from":6,"to":0}]},{"pgid":"11.3c","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6837/3833782560":"2024-03-24T11:24:34.350690+0000","172.21.15.120:6836/3833782560":"2024-03-24T11:24:34.350690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:24:45.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.070+0000 7f2f40e00640 1 -- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f1c06a7a0 msgr2=0x7f2f1c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.070+0000 7f2f40e00640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f1c06a7a0 0x7f2f1c06cc60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2f28036000 tx=0x7f2f2803c040 comp rx=0 tx=0).stop 2024-03-23T11:24:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.070+0000 7f2f40e00640 1 -- 172.21.15.62:0/1446691637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f3c0b12b0 msgr2=0x7f2f3c0c91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.070+0000 7f2f40e00640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f3c0b12b0 0x7f2f3c0c91d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00ebe0 tx=0x7f2f2c00c520 comp rx=0 tx=0).stop 2024-03-23T11:24:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.070+0000 7f2f40e00640 1 -- 172.21.15.62:0/1446691637 shutdown_connections 2024-03-23T11:24:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.070+0000 7f2f40e00640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f1c06a7a0 0x7f2f1c06cc60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.070+0000 7f2f40e00640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f3c0ceb00 0x7f2f3c0c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.070+0000 7f2f40e00640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f3c0ce0f0 0x7f2f3c1579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.070+0000 7f2f40e00640 1 --2- 172.21.15.62:0/1446691637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f3c0b12b0 0x7f2f3c0c91d0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.070+0000 7f2f40e00640 1 -- 172.21.15.62:0/1446691637 >> 172.21.15.62:0/1446691637 conn(0x7f2f3c0b2010 msgr2=0x7f2f3c15c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.071+0000 7f2f40e00640 1 -- 172.21.15.62:0/1446691637 shutdown_connections 2024-03-23T11:24:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.071+0000 7f2f40e00640 1 -- 172.21.15.62:0/1446691637 wait complete. 2024-03-23T11:24:45.085 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:24:08.688706+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:24:10.072774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:24:45.086 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:24:45.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.320+0000 7fe63f800640 1 Processor -- start 2024-03-23T11:24:45.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.320+0000 7fe63f800640 1 -- start start 2024-03-23T11:24:45.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.322+0000 7fe63f800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe6381571e0 0x7fe6380ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:45.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.322+0000 7fe63f800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6381613a0 0x7fe63815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:45.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.322+0000 7fe63f800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63806cf40 con 0x7fe6380ade70 2024-03-23T11:24:45.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.322+0000 7fe63f800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6381575c0 con 0x7fe6381571e0 2024-03-23T11:24:45.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.322+0000 7fe63f800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63806c3a0 con 0x7fe6381613a0 2024-03-23T11:24:45.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.323+0000 7fe63ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe6381571e0 0x7fe6380ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:45.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.323+0000 7fe63ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe6381571e0 0x7fe6380ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50558/0 (socket says 172.21.15.62:50558) 2024-03-23T11:24:45.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.323+0000 7fe63ce00640 1 -- 172.21.15.62:0/1264093862 learned_addr learned my addr 172.21.15.62:0/1264093862 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:45.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.323+0000 7fe63d800640 1 --2- 172.21.15.62:0/1264093862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6381613a0 0x7fe63815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:45.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.323+0000 7fe63ce00640 1 -- 172.21.15.62:0/1264093862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6381613a0 msgr2=0x7fe63815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:45.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.323+0000 7fe63ce00640 1 --2- 172.21.15.62:0/1264093862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6381613a0 0x7fe63815e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.323+0000 7fe63ce00640 1 -- 172.21.15.62:0/1264093862 >> v1:172.21.15.120:6789/0 conn(0x7fe6380ade70 legacy=0x7fe63806d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.323+0000 7fe63ce00640 1 -- 172.21.15.62:0/1264093862 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63806c610 con 0x7fe6381571e0 2024-03-23T11:24:45.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.325+0000 7fe63ce00640 1 --2- 172.21.15.62:0/1264093862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6381571e0 0x7fe6380ad930 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe628009870 tx=0x7fe62802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c018b13129d2a0b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:45.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.328+0000 7fe637400640 1 -- 172.21.15.62:0/1264093862 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628031640 con 0x7fe6381571e0 2024-03-23T11:24:45.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.328+0000 7fe637400640 1 -- 172.21.15.62:0/1264093862 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6280317a0 con 0x7fe6381571e0 2024-03-23T11:24:45.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.328+0000 7fe637400640 1 -- 172.21.15.62:0/1264093862 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628031b30 con 0x7fe6381571e0 2024-03-23T11:24:45.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.329+0000 7fe63f800640 1 -- 172.21.15.62:0/1264093862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6381571e0 msgr2=0x7fe6380ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.329+0000 7fe63f800640 1 --2- 172.21.15.62:0/1264093862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6381571e0 0x7fe6380ad930 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe628009870 tx=0x7fe62802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.329+0000 7fe63f800640 1 -- 172.21.15.62:0/1264093862 shutdown_connections 2024-03-23T11:24:45.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.329+0000 7fe63f800640 1 --2- 172.21.15.62:0/1264093862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6381613a0 0x7fe63815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.329+0000 7fe63f800640 1 --2- 172.21.15.62:0/1264093862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6381571e0 0x7fe6380ad930 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.329+0000 7fe63f800640 1 -- 172.21.15.62:0/1264093862 >> 172.21.15.62:0/1264093862 conn(0x7fe6380b02e0 msgr2=0x7fe6380b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:45.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.329+0000 7fe63f800640 1 -- 172.21.15.62:0/1264093862 shutdown_connections 2024-03-23T11:24:45.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.329+0000 7fe63f800640 1 -- 172.21.15.62:0/1264093862 wait complete. 2024-03-23T11:24:45.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.331+0000 7fe63f800640 1 Processor -- start 2024-03-23T11:24:45.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.331+0000 7fe63f800640 1 -- start start 2024-03-23T11:24:45.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.332+0000 7fe63f800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6380ade70 0x7fe6380b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:45.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.332+0000 7fe63f800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6381571e0 0x7fe6380b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:45.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.332+0000 7fe63f800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6381613a0 0x7fe638143a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:45.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.332+0000 7fe63f800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6381f3390 con 0x7fe6381613a0 2024-03-23T11:24:45.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.332+0000 7fe63f800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe638157a60 con 0x7fe6381571e0 2024-03-23T11:24:45.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.332+0000 7fe63f800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6381f2810 con 0x7fe6380ade70 2024-03-23T11:24:45.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.332+0000 7fe63d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6381613a0 0x7fe638143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:45.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.332+0000 7fe63d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6381613a0 0x7fe638143a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50560/0 (socket says 172.21.15.62:50560) 2024-03-23T11:24:45.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.333+0000 7fe63d800640 1 -- 172.21.15.62:0/2894039035 learned_addr learned my addr 172.21.15.62:0/2894039035 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:45.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.333+0000 7fe637e00640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6381571e0 0x7fe6380b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:45.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.333+0000 7fe63ce00640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6380ade70 0x7fe6380b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:45.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.333+0000 7fe63d800640 1 -- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6380ade70 msgr2=0x7fe6380b27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.333+0000 7fe63d800640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6380ade70 0x7fe6380b27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.333+0000 7fe63d800640 1 -- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6381571e0 msgr2=0x7fe6380b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.333+0000 7fe63d800640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6381571e0 0x7fe6380b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.333+0000 7fe63d800640 1 -- 172.21.15.62:0/2894039035 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6280094d0 con 0x7fe6381613a0 2024-03-23T11:24:45.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.333+0000 7fe63d800640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6381613a0 0x7fe638143a20 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fe63000efc0 tx=0x7fe63000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.334+0000 7fe634c00640 1 -- 172.21.15.62:0/2894039035 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630014480 con 0x7fe6381613a0 2024-03-23T11:24:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.334+0000 7fe634c00640 1 -- 172.21.15.62:0/2894039035 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63000f040 con 0x7fe6381613a0 2024-03-23T11:24:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.334+0000 7fe63f800640 1 -- 172.21.15.62:0/2894039035 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6381f2af0 con 0x7fe6381613a0 2024-03-23T11:24:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.334+0000 7fe634c00640 1 -- 172.21.15.62:0/2894039035 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630014820 con 0x7fe6381613a0 2024-03-23T11:24:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.334+0000 7fe63f800640 1 -- 172.21.15.62:0/2894039035 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6381f30b0 con 0x7fe6381613a0 2024-03-23T11:24:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.335+0000 7fe63f800640 1 -- 172.21.15.62:0/2894039035 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe638066c90 con 0x7fe6381613a0 2024-03-23T11:24:45.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.336+0000 7fe634c00640 1 -- 172.21.15.62:0/2894039035 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe630014980 con 0x7fe6381613a0 2024-03-23T11:24:45.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.337+0000 7fe634c00640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61406a7a0 0x7fe61406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:45.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.337+0000 7fe634c00640 1 -- 172.21.15.62:0/2894039035 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe630089a90 con 0x7fe6381613a0 2024-03-23T11:24:45.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.337+0000 7fe63ce00640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61406a7a0 0x7fe61406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:45.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.338+0000 7fe63ce00640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61406a7a0 0x7fe61406cc60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe628036000 tx=0x7fe62803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:45.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.341+0000 7fe634c00640 1 -- 172.21.15.62:0/2894039035 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe630054d80 con 0x7fe6381613a0 2024-03-23T11:24:45.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.540+0000 7fe63f800640 1 -- 172.21.15.62:0/2894039035 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6380b38c0 con 0x7fe6381613a0 2024-03-23T11:24:45.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.541+0000 7fe634c00640 1 -- 172.21.15.62:0/2894039035 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe630018020 con 0x7fe6381613a0 2024-03-23T11:24:45.536 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.543+0000 7fe63f800640 1 -- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61406a7a0 msgr2=0x7fe61406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.543+0000 7fe63f800640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61406a7a0 0x7fe61406cc60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe628036000 tx=0x7fe62803c040 comp rx=0 tx=0).stop 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.544+0000 7fe63f800640 1 -- 172.21.15.62:0/2894039035 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6381613a0 msgr2=0x7fe638143a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.544+0000 7fe63f800640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6381613a0 0x7fe638143a20 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fe63000efc0 tx=0x7fe63000c490 comp rx=0 tx=0).stop 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.544+0000 7fe63f800640 1 -- 172.21.15.62:0/2894039035 shutdown_connections 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.544+0000 7fe63f800640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61406a7a0 0x7fe61406cc60 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.544+0000 7fe63f800640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6381613a0 0x7fe638143a20 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.544+0000 7fe63f800640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6381571e0 0x7fe6380b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.544+0000 7fe63f800640 1 --2- 172.21.15.62:0/2894039035 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6380ade70 0x7fe6380b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.544+0000 7fe63f800640 1 -- 172.21.15.62:0/2894039035 >> 172.21.15.62:0/2894039035 conn(0x7fe6380b02e0 msgr2=0x7fe6380b18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.544+0000 7fe63f800640 1 -- 172.21.15.62:0/2894039035 shutdown_connections 2024-03-23T11:24:45.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.544+0000 7fe63f800640 1 -- 172.21.15.62:0/2894039035 wait complete. 2024-03-23T11:24:45.560 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:24:45.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.794+0000 7f877f200640 1 Processor -- start 2024-03-23T11:24:45.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.794+0000 7f877f200640 1 -- start start 2024-03-23T11:24:45.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.795+0000 7f877f200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f878015fcf0 0x7f87800d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:45.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.795+0000 7f877f200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87800d14f0 con 0x7f87800b6dc0 2024-03-23T11:24:45.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.795+0000 7f877f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87800d3540 con 0x7f8780160420 2024-03-23T11:24:45.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.795+0000 7f877f200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87800ce510 con 0x7f878015fcf0 2024-03-23T11:24:45.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.796+0000 7f877de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f878015fcf0 0x7f87800d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:45.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.796+0000 7f877de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f878015fcf0 0x7f87800d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50576/0 (socket says 172.21.15.62:50576) 2024-03-23T11:24:45.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.796+0000 7f877e800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f87800b6dc0 0x7f87800d0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43458/0 (socket says 172.21.15.62:43458) 2024-03-23T11:24:45.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.796+0000 7f877de00640 1 -- 172.21.15.62:0/755977319 learned_addr learned my addr 172.21.15.62:0/755977319 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:45.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.796+0000 7f877de00640 1 -- 172.21.15.62:0/755977319 >> v1:172.21.15.120:6789/0 conn(0x7f8780160420 legacy=0x7f87800d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:45.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.796+0000 7f877de00640 1 -- 172.21.15.62:0/755977319 >> v1:172.21.15.62:6789/0 conn(0x7f87800b6dc0 legacy=0x7f87800d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.796+0000 7f877de00640 1 -- 172.21.15.62:0/755977319 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87800cdc70 con 0x7f878015fcf0 2024-03-23T11:24:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.798+0000 7f877de00640 1 --2- 172.21.15.62:0/755977319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f878015fcf0 0x7f87800d3000 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f8770009e50 tx=0x7f877002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf3a26cb68e8e0f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:45.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.798+0000 7f877d400640 1 -- 172.21.15.62:0/755977319 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877002ca80 con 0x7f878015fcf0 2024-03-23T11:24:45.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.798+0000 7f877d400640 1 -- 172.21.15.62:0/755977319 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877002cbe0 con 0x7f878015fcf0 2024-03-23T11:24:45.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.798+0000 7f877d400640 1 -- 172.21.15.62:0/755977319 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87700343f0 con 0x7f878015fcf0 2024-03-23T11:24:45.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.799+0000 7f877f200640 1 -- 172.21.15.62:0/755977319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f878015fcf0 msgr2=0x7f87800d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.799+0000 7f877f200640 1 --2- 172.21.15.62:0/755977319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f878015fcf0 0x7f87800d3000 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f8770009e50 tx=0x7f877002bee0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.799+0000 7f877f200640 1 -- 172.21.15.62:0/755977319 shutdown_connections 2024-03-23T11:24:45.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.799+0000 7f877f200640 1 --2- 172.21.15.62:0/755977319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f878015fcf0 0x7f87800d3000 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.799+0000 7f877f200640 1 -- 172.21.15.62:0/755977319 >> 172.21.15.62:0/755977319 conn(0x7f8780089f60 msgr2=0x7f87800d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:45.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.799+0000 7f877f200640 1 -- 172.21.15.62:0/755977319 shutdown_connections 2024-03-23T11:24:45.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.799+0000 7f877f200640 1 -- 172.21.15.62:0/755977319 wait complete. 2024-03-23T11:24:45.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.801+0000 7f877f200640 1 Processor -- start 2024-03-23T11:24:45.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.801+0000 7f877f200640 1 -- start start 2024-03-23T11:24:45.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f877f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87800b6dc0 0x7f87800c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:45.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f877f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f878015fcf0 0x7f87801e6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:45.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f877f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8780160420 0x7f87801e6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:45.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f877f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8780158ff0 con 0x7f878015fcf0 2024-03-23T11:24:45.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f877f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87801e7060 con 0x7f87800b6dc0 2024-03-23T11:24:45.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f877f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87801e6d80 con 0x7f8780160420 2024-03-23T11:24:45.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f8775400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f878015fcf0 0x7f87801e6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:45.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f8775400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f878015fcf0 0x7f87801e6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50592/0 (socket says 172.21.15.62:50592) 2024-03-23T11:24:45.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f8775400640 1 -- 172.21.15.62:0/3019136866 learned_addr learned my addr 172.21.15.62:0/3019136866 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:45.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f8775400640 1 -- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8780160420 msgr2=0x7f87801e6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f8775400640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8780160420 0x7f87801e6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f8775400640 1 -- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87800b6dc0 msgr2=0x7f87800c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:45.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f877de00640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87800b6dc0 0x7f87800c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:45.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f8775400640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87800b6dc0 0x7f87800c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:45.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f8775400640 1 -- 172.21.15.62:0/3019136866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8770009b30 con 0x7f878015fcf0 2024-03-23T11:24:45.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.803+0000 7f877de00640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87800b6dc0 0x7f87800c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:45.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.804+0000 7f8775400640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f878015fcf0 0x7f87801e6250 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f876800c430 tx=0x7f876800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:45.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.804+0000 7f8776a00640 1 -- 172.21.15.62:0/3019136866 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8768011790 con 0x7f878015fcf0 2024-03-23T11:24:45.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.804+0000 7f8776a00640 1 -- 172.21.15.62:0/3019136866 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87680118f0 con 0x7f878015fcf0 2024-03-23T11:24:45.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.804+0000 7f8776a00640 1 -- 172.21.15.62:0/3019136866 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8768011ce0 con 0x7f878015fcf0 2024-03-23T11:24:45.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.804+0000 7f877f200640 1 -- 172.21.15.62:0/3019136866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8780147dd0 con 0x7f878015fcf0 2024-03-23T11:24:45.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.804+0000 7f877f200640 1 -- 172.21.15.62:0/3019136866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8780148390 con 0x7f878015fcf0 2024-03-23T11:24:45.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.806+0000 7f877f200640 1 -- 172.21.15.62:0/3019136866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8780148580 con 0x7f878015fcf0 2024-03-23T11:24:45.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.806+0000 7f8776a00640 1 -- 172.21.15.62:0/3019136866 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f876801c460 con 0x7f878015fcf0 2024-03-23T11:24:45.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.807+0000 7f8776a00640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f875c06a7a0 0x7f875c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:45.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.807+0000 7f8776a00640 1 -- 172.21.15.62:0/3019136866 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8768088d50 con 0x7f878015fcf0 2024-03-23T11:24:45.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.808+0000 7f877de00640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f875c06a7a0 0x7f875c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:45.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.809+0000 7f877de00640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f875c06a7a0 0x7f875c06cc60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8770035000 tx=0x7f8770030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:45.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:45.813+0000 7f8776a00640 1 -- 172.21.15.62:0/3019136866 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f876808c030 con 0x7f878015fcf0 2024-03-23T11:24:45.976 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:24:45.984 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:24:46.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.023+0000 7f877f200640 1 -- 172.21.15.62:0/3019136866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8780066bb0 con 0x7f878015fcf0 2024-03-23T11:24:46.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.023+0000 7f8776a00640 1 -- 172.21.15.62:0/3019136866 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8768054170 con 0x7f878015fcf0 2024-03-23T11:24:46.018 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:24:46.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 -- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f875c06a7a0 msgr2=0x7f875c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f875c06a7a0 0x7f875c06cc60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8770035000 tx=0x7f8770030000 comp rx=0 tx=0).stop 2024-03-23T11:24:46.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 -- 172.21.15.62:0/3019136866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f878015fcf0 msgr2=0x7f87801e6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f878015fcf0 0x7f87801e6250 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f876800c430 tx=0x7f876800c8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 -- 172.21.15.62:0/3019136866 shutdown_connections 2024-03-23T11:24:46.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f875c06a7a0 0x7f875c06cc60 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f878015fcf0 0x7f87801e6250 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8780160420 0x7f87801e6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 --2- 172.21.15.62:0/3019136866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87800b6dc0 0x7f87800c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 -- 172.21.15.62:0/3019136866 >> 172.21.15.62:0/3019136866 conn(0x7f8780089f60 msgr2=0x7f87800d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:46.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 -- 172.21.15.62:0/3019136866 shutdown_connections 2024-03-23T11:24:46.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.026+0000 7f877f200640 1 -- 172.21.15.62:0/3019136866 wait complete. 2024-03-23T11:24:46.040 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:24:46.242 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:24:46.247+0000 7f9201200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-23T11:24:46.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.315+0000 7f72b8800640 1 Processor -- start 2024-03-23T11:24:46.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.316+0000 7f72b8800640 1 -- start start 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.317+0000 7f72b8800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f72b01613a0 0x7f72b006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.317+0000 7f72b8800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f72b006d910 0x7f72b015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.317+0000 7f72b8800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b006cf40 con 0x7f72b01571e0 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.317+0000 7f72b8800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b01575c0 con 0x7f72b006d910 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.317+0000 7f72b8800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b006c3a0 con 0x7f72b01613a0 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.318+0000 7f72b6800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f72b006d910 0x7f72b015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.318+0000 7f72b6800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f72b006d910 0x7f72b015e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50606/0 (socket says 172.21.15.62:50606) 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.318+0000 7f72b6800640 1 -- 172.21.15.62:0/841911411 learned_addr learned my addr 172.21.15.62:0/841911411 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.318+0000 7f72b5e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f72b01571e0 0x7f72b00ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43460/0 (socket says 172.21.15.62:43460) 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.318+0000 7f72b5400640 1 --2- 172.21.15.62:0/841911411 >> v2:172.21.15.120:3300/0 conn(0x7f72b01613a0 0x7f72b006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.318+0000 7f72b6800640 1 -- 172.21.15.62:0/841911411 >> v2:172.21.15.120:3300/0 conn(0x7f72b01613a0 msgr2=0x7f72b006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.318+0000 7f72b6800640 1 --2- 172.21.15.62:0/841911411 >> v2:172.21.15.120:3300/0 conn(0x7f72b01613a0 0x7f72b006d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.318+0000 7f72b6800640 1 -- 172.21.15.62:0/841911411 >> v1:172.21.15.62:6789/0 conn(0x7f72b01571e0 legacy=0x7f72b00ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.318+0000 7f72b6800640 1 -- 172.21.15.62:0/841911411 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b00ae040 con 0x7f72b006d910 2024-03-23T11:24:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.319+0000 7f72b6800640 1 --2- 172.21.15.62:0/841911411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b006d910 0x7f72b015e280 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f72b006e7f0 tx=0x7f72ac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8034cef4155439d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.320+0000 7f72b4a00640 1 -- 172.21.15.62:0/841911411 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac02ca80 con 0x7f72b006d910 2024-03-23T11:24:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.320+0000 7f72b4a00640 1 -- 172.21.15.62:0/841911411 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac02cbe0 con 0x7f72b006d910 2024-03-23T11:24:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.320+0000 7f72b4a00640 1 -- 172.21.15.62:0/841911411 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac0343f0 con 0x7f72b006d910 2024-03-23T11:24:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.320+0000 7f72b8800640 1 -- 172.21.15.62:0/841911411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b006d910 msgr2=0x7f72b015e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.320+0000 7f72b8800640 1 --2- 172.21.15.62:0/841911411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b006d910 0x7f72b015e280 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f72b006e7f0 tx=0x7f72ac02bee0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.320+0000 7f72b8800640 1 -- 172.21.15.62:0/841911411 shutdown_connections 2024-03-23T11:24:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.320+0000 7f72b8800640 1 --2- 172.21.15.62:0/841911411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b006d910 0x7f72b015e280 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.320+0000 7f72b8800640 1 --2- 172.21.15.62:0/841911411 >> v2:172.21.15.120:3300/0 conn(0x7f72b01613a0 0x7f72b006d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.320+0000 7f72b8800640 1 -- 172.21.15.62:0/841911411 >> 172.21.15.62:0/841911411 conn(0x7f72b00b02e0 msgr2=0x7f72b00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.320+0000 7f72b8800640 1 -- 172.21.15.62:0/841911411 shutdown_connections 2024-03-23T11:24:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.320+0000 7f72b8800640 1 -- 172.21.15.62:0/841911411 wait complete. 2024-03-23T11:24:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.322+0000 7f72b8800640 1 Processor -- start 2024-03-23T11:24:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.322+0000 7f72b8800640 1 -- start start 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.323+0000 7f72b8800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b006d910 0x7f72b0158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.323+0000 7f72b8800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b01571e0 0x7f72b0153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.323+0000 7f72b8800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b01613a0 0x7f72b015b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.323+0000 7f72b8800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b015aab0 con 0x7f72b01613a0 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.323+0000 7f72b8800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b01595b0 con 0x7f72b01571e0 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.323+0000 7f72b8800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b01592d0 con 0x7f72b006d910 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.324+0000 7f72b6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b01613a0 0x7f72b015b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.324+0000 7f72b6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b01613a0 0x7f72b015b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50612/0 (socket says 172.21.15.62:50612) 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.324+0000 7f72b6800640 1 -- 172.21.15.62:0/1458340659 learned_addr learned my addr 172.21.15.62:0/1458340659 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.324+0000 7f72b5e00640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b006d910 0x7f72b0158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.324+0000 7f72b5400640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b01571e0 0x7f72b0153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.324+0000 7f72b6800640 1 -- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b006d910 msgr2=0x7f72b0158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.324+0000 7f72b6800640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b006d910 0x7f72b0158aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.324+0000 7f72b6800640 1 -- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b01571e0 msgr2=0x7f72b0153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.324+0000 7f72b6800640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b01571e0 0x7f72b0153180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.324+0000 7f72b6800640 1 -- 172.21.15.62:0/1458340659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac009b30 con 0x7f72b01613a0 2024-03-23T11:24:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.324+0000 7f72b6800640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b01613a0 0x7f72b015b7a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f72ac036040 tx=0x7f72ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.325+0000 7f72aa000640 1 -- 172.21.15.62:0/1458340659 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac009510 con 0x7f72b01613a0 2024-03-23T11:24:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.325+0000 7f72aa000640 1 -- 172.21.15.62:0/1458340659 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac03e450 con 0x7f72b01613a0 2024-03-23T11:24:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.325+0000 7f72aa000640 1 -- 172.21.15.62:0/1458340659 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac03e800 con 0x7f72b01613a0 2024-03-23T11:24:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.325+0000 7f72b8800640 1 -- 172.21.15.62:0/1458340659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b015acc0 con 0x7f72b01613a0 2024-03-23T11:24:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.325+0000 7f72b8800640 1 -- 172.21.15.62:0/1458340659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b015b280 con 0x7f72b01613a0 2024-03-23T11:24:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.326+0000 7f72b8800640 1 -- 172.21.15.62:0/1458340659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b0066c90 con 0x7f72b01613a0 2024-03-23T11:24:46.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.327+0000 7f72aa000640 1 -- 172.21.15.62:0/1458340659 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f72ac02cd50 con 0x7f72b01613a0 2024-03-23T11:24:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.328+0000 7f72aa000640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729006a7d0 0x7f729006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.328+0000 7f72aa000640 1 -- 172.21.15.62:0/1458340659 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f72ac0ab940 con 0x7f72b01613a0 2024-03-23T11:24:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.329+0000 7f72b5e00640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729006a7d0 0x7f729006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.330+0000 7f72b5e00640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729006a7d0 0x7f729006cc90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f72a4000f60 tx=0x7f72a4007c80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.334+0000 7f72aa000640 1 -- 172.21.15.62:0/1458340659 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72ac076cc0 con 0x7f72b01613a0 2024-03-23T11:24:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.532+0000 7f72b8800640 1 -- 172.21.15.62:0/1458340659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f72b00b38c0 con 0x7f72b01613a0 2024-03-23T11:24:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.533+0000 7f72aa000640 1 -- 172.21.15.62:0/1458340659 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) v1 ==== 100+0+11 (secure 0 0 0) 0x7f72ac043020 con 0x7f72b01613a0 2024-03-23T11:24:46.527 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:24:46.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.535+0000 7f72b8800640 1 -- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729006a7d0 msgr2=0x7f729006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.535+0000 7f72b8800640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729006a7d0 0x7f729006cc90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f72a4000f60 tx=0x7f72a4007c80 comp rx=0 tx=0).stop 2024-03-23T11:24:46.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.535+0000 7f72b8800640 1 -- 172.21.15.62:0/1458340659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b01613a0 msgr2=0x7f72b015b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.535+0000 7f72b8800640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b01613a0 0x7f72b015b7a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f72ac036040 tx=0x7f72ac004a40 comp rx=0 tx=0).stop 2024-03-23T11:24:46.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.535+0000 7f72b8800640 1 -- 172.21.15.62:0/1458340659 shutdown_connections 2024-03-23T11:24:46.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.535+0000 7f72b8800640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729006a7d0 0x7f729006cc90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.535+0000 7f72b8800640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b01613a0 0x7f72b015b7a0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.535+0000 7f72b8800640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b01571e0 0x7f72b0153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.535+0000 7f72b8800640 1 --2- 172.21.15.62:0/1458340659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b006d910 0x7f72b0158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.535+0000 7f72b8800640 1 -- 172.21.15.62:0/1458340659 >> 172.21.15.62:0/1458340659 conn(0x7f72b00b02e0 msgr2=0x7f72b00ad4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:46.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.536+0000 7f72b8800640 1 -- 172.21.15.62:0/1458340659 shutdown_connections 2024-03-23T11:24:46.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.536+0000 7f72b8800640 1 -- 172.21.15.62:0/1458340659 wait complete. 2024-03-23T11:24:46.551 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:24:46.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.778+0000 7f02e2600640 1 Processor -- start 2024-03-23T11:24:46.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.778+0000 7f02e2600640 1 -- start start 2024-03-23T11:24:46.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.779+0000 7f02e2600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02dc15c9e0 0x7f02dc1607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.779+0000 7f02e2600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02dc15cdc0 0x7f02dc0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.779+0000 7f02e2600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02dc06bfc0 con 0x7f02dc0cf650 2024-03-23T11:24:46.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.779+0000 7f02e2600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02dc0ce140 con 0x7f02dc15c9e0 2024-03-23T11:24:46.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.779+0000 7f02e2600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc0b4370 con 0x7f02dc15cdc0 2024-03-23T11:24:46.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.780+0000 7f02db400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02dc15c9e0 0x7f02dc1607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.780+0000 7f02daa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02dc15cdc0 0x7f02dc0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.780+0000 7f02db400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02dc15c9e0 0x7f02dc1607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:57648/0 (socket says 172.21.15.62:57648) 2024-03-23T11:24:46.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.780+0000 7f02db400640 1 -- 172.21.15.62:0/3378967102 learned_addr learned my addr 172.21.15.62:0/3378967102 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:46.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.781+0000 7f02daa00640 1 -- 172.21.15.62:0/3378967102 >> v2:172.21.15.120:3300/0 conn(0x7f02dc15c9e0 msgr2=0x7f02dc1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:46.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.781+0000 7f02daa00640 1 --2- 172.21.15.62:0/3378967102 >> v2:172.21.15.120:3300/0 conn(0x7f02dc15c9e0 0x7f02dc1607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.781+0000 7f02daa00640 1 -- 172.21.15.62:0/3378967102 >> v1:172.21.15.120:6789/0 conn(0x7f02dc0cf650 legacy=0x7f02dc15d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.781+0000 7f02daa00640 1 -- 172.21.15.62:0/3378967102 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc0b5260 con 0x7f02dc15cdc0 2024-03-23T11:24:46.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.783+0000 7f02daa00640 1 --2- 172.21.15.62:0/3378967102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02dc15cdc0 0x7f02dc0d1f20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f02c40097a0 tx=0x7f02c402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cda6b533c2604 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:46.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.783+0000 7f02da000640 1 -- 172.21.15.62:0/3378967102 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4031830 con 0x7f02dc15cdc0 2024-03-23T11:24:46.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.783+0000 7f02da000640 1 -- 172.21.15.62:0/3378967102 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c4031990 con 0x7f02dc15cdc0 2024-03-23T11:24:46.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.783+0000 7f02da000640 1 -- 172.21.15.62:0/3378967102 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4031d20 con 0x7f02dc15cdc0 2024-03-23T11:24:46.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.784+0000 7f02e2600640 1 -- 172.21.15.62:0/3378967102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02dc15cdc0 msgr2=0x7f02dc0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.784+0000 7f02e2600640 1 --2- 172.21.15.62:0/3378967102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02dc15cdc0 0x7f02dc0d1f20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f02c40097a0 tx=0x7f02c402b8d0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.784+0000 7f02e2600640 1 -- 172.21.15.62:0/3378967102 shutdown_connections 2024-03-23T11:24:46.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.784+0000 7f02e2600640 1 --2- 172.21.15.62:0/3378967102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02dc15cdc0 0x7f02dc0d1f20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.784+0000 7f02e2600640 1 --2- 172.21.15.62:0/3378967102 >> v2:172.21.15.120:3300/0 conn(0x7f02dc15c9e0 0x7f02dc1607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.784+0000 7f02e2600640 1 -- 172.21.15.62:0/3378967102 >> 172.21.15.62:0/3378967102 conn(0x7f02dc071570 msgr2=0x7f02dc071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:46.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.784+0000 7f02e2600640 1 -- 172.21.15.62:0/3378967102 shutdown_connections 2024-03-23T11:24:46.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.784+0000 7f02e2600640 1 -- 172.21.15.62:0/3378967102 wait complete. 2024-03-23T11:24:46.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.786+0000 7f02e2600640 1 Processor -- start 2024-03-23T11:24:46.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.786+0000 7f02e2600640 1 -- start start 2024-03-23T11:24:46.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.787+0000 7f02e2600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02dc0cf650 0x7f02dc0cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.787+0000 7f02e2600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02dc15c9e0 0x7f02dc1f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.787+0000 7f02e2600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02dc15cdc0 0x7f02dc15d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.787+0000 7f02e2600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc15aa70 con 0x7f02dc15c9e0 2024-03-23T11:24:46.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.787+0000 7f02e2600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc0abbd0 con 0x7f02dc0cf650 2024-03-23T11:24:46.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.787+0000 7f02e2600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc1f5570 con 0x7f02dc15cdc0 2024-03-23T11:24:46.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.787+0000 7f02daa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02dc15c9e0 0x7f02dc1f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.787+0000 7f02daa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02dc15c9e0 0x7f02dc1f5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50614/0 (socket says 172.21.15.62:50614) 2024-03-23T11:24:46.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.787+0000 7f02daa00640 1 -- 172.21.15.62:0/3754413731 learned_addr learned my addr 172.21.15.62:0/3754413731 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.788+0000 7f02dbe00640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02dc15cdc0 0x7f02dc15d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.788+0000 7f02db400640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02dc0cf650 0x7f02dc0cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.788+0000 7f02daa00640 1 -- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02dc15cdc0 msgr2=0x7f02dc15d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.788+0000 7f02daa00640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02dc15cdc0 0x7f02dc15d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.788+0000 7f02daa00640 1 -- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02dc0cf650 msgr2=0x7f02dc0cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.788+0000 7f02daa00640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02dc0cf650 0x7f02dc0cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.788+0000 7f02daa00640 1 -- 172.21.15.62:0/3754413731 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c4009400 con 0x7f02dc15c9e0 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.788+0000 7f02daa00640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02dc15c9e0 0x7f02dc1f5030 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f02c4034040 tx=0x7f02c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.788+0000 7f02d3400640 1 -- 172.21.15.62:0/3754413731 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c403a8f0 con 0x7f02dc15c9e0 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.788+0000 7f02d3400640 1 -- 172.21.15.62:0/3754413731 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c403aa70 con 0x7f02dc15c9e0 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.788+0000 7f02d3400640 1 -- 172.21.15.62:0/3754413731 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c403ae20 con 0x7f02dc15c9e0 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.789+0000 7f02e2600640 1 -- 172.21.15.62:0/3754413731 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02dc0aafc0 con 0x7f02dc15c9e0 2024-03-23T11:24:46.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.789+0000 7f02e2600640 1 -- 172.21.15.62:0/3754413731 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02dc0ab580 con 0x7f02dc15c9e0 2024-03-23T11:24:46.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.790+0000 7f02e2600640 1 -- 172.21.15.62:0/3754413731 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02dc066c90 con 0x7f02dc15c9e0 2024-03-23T11:24:46.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.791+0000 7f02d3400640 1 -- 172.21.15.62:0/3754413731 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f02c404d440 con 0x7f02dc15c9e0 2024-03-23T11:24:46.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.792+0000 7f02d3400640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7d0 0x7f02b806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:46.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.793+0000 7f02d3400640 1 -- 172.21.15.62:0/3754413731 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f02c40aaf40 con 0x7f02dc15c9e0 2024-03-23T11:24:46.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.793+0000 7f02db400640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7d0 0x7f02b806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:46.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.794+0000 7f02db400640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7d0 0x7f02b806cc90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009d80 tx=0x7f02cc00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:46.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:46.798+0000 7f02d3400640 1 -- 172.21.15.62:0/3754413731 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f02c4076230 con 0x7f02dc15c9e0 2024-03-23T11:24:47.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.015+0000 7f02e2600640 1 -- 172.21.15.62:0/3754413731 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f02dc15b1d0 con 0x7f02dc15c9e0 2024-03-23T11:24:47.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.016+0000 7f02d3400640 1 -- 172.21.15.62:0/3754413731 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v77) v1 ==== 80+0+124 (secure 0 0 0) 0x7f02c4035080 con 0x7f02dc15c9e0 2024-03-23T11:24:47.011 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6837/3833782560 2024-03-24T11:24:34.350690+0000 2024-03-23T11:24:47.011 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6836/3833782560 2024-03-24T11:24:34.350690+0000 2024-03-23T11:24:47.011 INFO:teuthology.orchestra.run.smithi062.stderr:listed 2 entries 2024-03-23T11:24:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.018+0000 7f02e2600640 1 -- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7d0 msgr2=0x7f02b806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.018+0000 7f02e2600640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7d0 0x7f02b806cc90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009d80 tx=0x7f02cc00b040 comp rx=0 tx=0).stop 2024-03-23T11:24:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.019+0000 7f02e2600640 1 -- 172.21.15.62:0/3754413731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02dc15c9e0 msgr2=0x7f02dc1f5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.019+0000 7f02e2600640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02dc15c9e0 0x7f02dc1f5030 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f02c4034040 tx=0x7f02c40046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.019+0000 7f02e2600640 1 -- 172.21.15.62:0/3754413731 shutdown_connections 2024-03-23T11:24:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.019+0000 7f02e2600640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7d0 0x7f02b806cc90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.019+0000 7f02e2600640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02dc15c9e0 0x7f02dc1f5030 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.019+0000 7f02e2600640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02dc15cdc0 0x7f02dc15d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.019+0000 7f02e2600640 1 --2- 172.21.15.62:0/3754413731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02dc0cf650 0x7f02dc0cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.019+0000 7f02e2600640 1 -- 172.21.15.62:0/3754413731 >> 172.21.15.62:0/3754413731 conn(0x7f02dc071570 msgr2=0x7f02dc071950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.019+0000 7f02e2600640 1 -- 172.21.15.62:0/3754413731 shutdown_connections 2024-03-23T11:24:47.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.019+0000 7f02e2600640 1 -- 172.21.15.62:0/3754413731 wait complete. 2024-03-23T11:24:47.031 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:24:47.031 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:24:47.077 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:24:47.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.350+0000 7fe99b200640 1 Processor -- start 2024-03-23T11:24:47.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.351+0000 7fe99b200640 1 -- start start 2024-03-23T11:24:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.352+0000 7fe99b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe99c15fcf0 0x7fe99c0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.352+0000 7fe99b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0b6dc0 0x7fe99c1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.352+0000 7fe99b200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c0d3540 con 0x7fe99c160420 2024-03-23T11:24:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.352+0000 7fe99b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe99c160df0 con 0x7fe99c15fcf0 2024-03-23T11:24:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.352+0000 7fe99b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c0ce510 con 0x7fe99c0b6dc0 2024-03-23T11:24:47.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.353+0000 7fe993e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe99c160420 0x7fe99c0d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50284/0 (socket says 172.21.15.62:50284) 2024-03-23T11:24:47.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.353+0000 7fe993e00640 1 -- 172.21.15.62:0/2871686424 learned_addr learned my addr 172.21.15.62:0/2871686424 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:47.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.353+0000 7fe99a800640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0b6dc0 0x7fe99c1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.353+0000 7fe999e00640 1 --2- 172.21.15.62:0/2871686424 >> v2:172.21.15.120:3300/0 conn(0x7fe99c15fcf0 0x7fe99c0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.353+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744241354 0 0) 0x7fe99c0d3540 con 0x7fe99c160420 2024-03-23T11:24:47.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.353+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe980003520 con 0x7fe99c160420 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.353+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1466466708 0 0) 0x7fe980003520 con 0x7fe99c160420 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0b6dc0 msgr2=0x7fe99c1608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0b6dc0 0x7fe99c1608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 >> v2:172.21.15.120:3300/0 conn(0x7fe99c15fcf0 msgr2=0x7fe99c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 --2- 172.21.15.62:0/2871686424 >> v2:172.21.15.120:3300/0 conn(0x7fe99c15fcf0 0x7fe99c0d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c0cdc70 con 0x7fe99c160420 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fe984001670 con 0x7fe99c160420 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 >> v1:172.21.15.62:6789/0 conn(0x7fe99c160420 legacy=0x7fe99c0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0b6dc0 0x7fe980003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe980003ef0 0x7fe9800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 0x7fe980005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c160df0 con 0x7fe980004d00 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c0ce510 con 0x7fe980003ef0 2024-03-23T11:24:47.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98000ba60 con 0x7fe99c0b6dc0 2024-03-23T11:24:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe99a800640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 0x7fe980005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe999e00640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0b6dc0 0x7fe980003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe993e00640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe980003ef0 0x7fe9800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe99a800640 1 -- 172.21.15.62:0/2871686424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0b6dc0 msgr2=0x7fe980003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe99a800640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0b6dc0 0x7fe980003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe99a800640 1 -- 172.21.15.62:0/2871686424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe980003ef0 msgr2=0x7fe9800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.354+0000 7fe99a800640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe980003ef0 0x7fe9800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.355+0000 7fe99a800640 1 -- 172.21.15.62:0/2871686424 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe980003520 con 0x7fe980004d00 2024-03-23T11:24:47.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.356+0000 7fe99a800640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 0x7fe980005180 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe994009fc0 tx=0x7fe9800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1850ea706f90844 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.356+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99402cc20 con 0x7fe980004d00 2024-03-23T11:24:47.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.356+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99400f040 con 0x7fe980004d00 2024-03-23T11:24:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.356+0000 7fe999400640 1 -- 172.21.15.62:0/2871686424 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994034440 con 0x7fe980004d00 2024-03-23T11:24:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.356+0000 7fe99b200640 1 -- 172.21.15.62:0/2871686424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 msgr2=0x7fe980005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.356+0000 7fe99b200640 1 --2- 172.21.15.62:0/2871686424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 0x7fe980005180 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe994009fc0 tx=0x7fe9800057f0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.356+0000 7fe99a800640 1 -- 172.21.15.62:0/2871686424 reap_dead start 2024-03-23T11:24:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.357+0000 7fe99b200640 1 -- 172.21.15.62:0/2871686424 shutdown_connections 2024-03-23T11:24:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.357+0000 7fe99b200640 1 -- 172.21.15.62:0/2871686424 >> 172.21.15.62:0/2871686424 conn(0x7fe99c089f60 msgr2=0x7fe99c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.357+0000 7fe99b200640 1 -- 172.21.15.62:0/2871686424 shutdown_connections 2024-03-23T11:24:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.357+0000 7fe99b200640 1 -- 172.21.15.62:0/2871686424 wait complete. 2024-03-23T11:24:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.359+0000 7fe99b200640 1 Processor -- start 2024-03-23T11:24:47.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.359+0000 7fe99b200640 1 -- start start 2024-03-23T11:24:47.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe99b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 0x7fe99c15c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe99b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0c7560 0x7fe99c14e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe99b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99c0cb8f0 0x7fe99c0cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe99b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c0c8a20 con 0x7fe980004d00 2024-03-23T11:24:47.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe99b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c0c8e40 con 0x7fe99c0cb8f0 2024-03-23T11:24:47.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe99b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c0cc2b0 con 0x7fe99c0c7560 2024-03-23T11:24:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe999e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 0x7fe99c15c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe999e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 0x7fe99c15c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45996/0 (socket says 172.21.15.62:45996) 2024-03-23T11:24:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe999e00640 1 -- 172.21.15.62:0/4226401656 learned_addr learned my addr 172.21.15.62:0/4226401656 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe993e00640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0c7560 0x7fe99c14e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe99a800640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99c0cb8f0 0x7fe99c0cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe999e00640 1 -- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0c7560 msgr2=0x7fe99c14e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe999e00640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0c7560 0x7fe99c14e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe999e00640 1 -- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99c0cb8f0 msgr2=0x7fe99c0cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.360+0000 7fe999e00640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99c0cb8f0 0x7fe99c0cbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.361+0000 7fe999e00640 1 -- 172.21.15.62:0/4226401656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c15b030 con 0x7fe980004d00 2024-03-23T11:24:47.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.361+0000 7fe999e00640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 0x7fe99c15c450 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0116d0 tx=0x7fe98c011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:47.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.361+0000 7fe992000640 1 -- 172.21.15.62:0/4226401656 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c00f9c0 con 0x7fe980004d00 2024-03-23T11:24:47.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.361+0000 7fe992000640 1 -- 172.21.15.62:0/4226401656 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98c00fb20 con 0x7fe980004d00 2024-03-23T11:24:47.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.361+0000 7fe99b200640 1 -- 172.21.15.62:0/4226401656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99c0c7da0 con 0x7fe980004d00 2024-03-23T11:24:47.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.361+0000 7fe992000640 1 -- 172.21.15.62:0/4226401656 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c00fe90 con 0x7fe980004d00 2024-03-23T11:24:47.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.361+0000 7fe99b200640 1 -- 172.21.15.62:0/4226401656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99c0c8360 con 0x7fe980004d00 2024-03-23T11:24:47.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.362+0000 7fe99b200640 1 -- 172.21.15.62:0/4226401656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe99c066bb0 con 0x7fe980004d00 2024-03-23T11:24:47.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.363+0000 7fe992000640 1 -- 172.21.15.62:0/4226401656 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe98c015460 con 0x7fe980004d00 2024-03-23T11:24:47.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.364+0000 7fe992000640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe97806a7a0 0x7fe97806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.364+0000 7fe992000640 1 -- 172.21.15.62:0/4226401656 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe98c0896f0 con 0x7fe980004d00 2024-03-23T11:24:47.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.365+0000 7fe993e00640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe97806a7a0 0x7fe97806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.366+0000 7fe993e00640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe97806a7a0 0x7fe97806cc60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe984001eb0 tx=0x7fe984000fe0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.369+0000 7fe992000640 1 -- 172.21.15.62:0/4226401656 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe98c054b30 con 0x7fe980004d00 2024-03-23T11:24:47.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.568+0000 7fe99b200640 1 -- 172.21.15.62:0/4226401656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe99c15b030 con 0x7fe980004d00 2024-03-23T11:24:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.572+0000 7fe992000640 1 -- 172.21.15.62:0/4226401656 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16329 (secure 0 0 0) 0x7fe99c15b030 con 0x7fe980004d00 2024-03-23T11:24:47.567 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:24:47.567 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":77,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:44.311392+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:24:08.688706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-23T11:24:10.072774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1c","mappings":[{"from":6,"to":0}]},{"pgid":"11.3c","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6837/3833782560":"2024-03-24T11:24:34.350690+0000","172.21.15.120:6836/3833782560":"2024-03-24T11:24:34.350690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:24:47.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.574+0000 7fe99b200640 1 -- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe97806a7a0 msgr2=0x7fe97806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.574+0000 7fe99b200640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe97806a7a0 0x7fe97806cc60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe984001eb0 tx=0x7fe984000fe0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.574+0000 7fe99b200640 1 -- 172.21.15.62:0/4226401656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 msgr2=0x7fe99c15c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.574+0000 7fe99b200640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 0x7fe99c15c450 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0116d0 tx=0x7fe98c011b90 comp rx=0 tx=0).stop 2024-03-23T11:24:47.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.574+0000 7fe99b200640 1 -- 172.21.15.62:0/4226401656 shutdown_connections 2024-03-23T11:24:47.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.574+0000 7fe99b200640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe97806a7a0 0x7fe97806cc60 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.574+0000 7fe99b200640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99c0cb8f0 0x7fe99c0cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.574+0000 7fe99b200640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe99c0c7560 0x7fe99c14e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.574+0000 7fe99b200640 1 --2- 172.21.15.62:0/4226401656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe980004d00 0x7fe99c15c450 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.574+0000 7fe99b200640 1 -- 172.21.15.62:0/4226401656 >> 172.21.15.62:0/4226401656 conn(0x7fe99c089f60 msgr2=0x7fe99c06b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:47.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.575+0000 7fe99b200640 1 -- 172.21.15.62:0/4226401656 shutdown_connections 2024-03-23T11:24:47.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.575+0000 7fe99b200640 1 -- 172.21.15.62:0/4226401656 wait complete. 2024-03-23T11:24:47.588 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:24:08.688706+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:24:10.072774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:24:47.588 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:24:47.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.816+0000 7f53d0000640 1 Processor -- start 2024-03-23T11:24:47.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.816+0000 7f53d0000640 1 -- start start 2024-03-23T11:24:47.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.817+0000 7f53d0000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53c815c9e0 0x7f53c81607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.817+0000 7f53d0000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f53c815cdc0 0x7f53c80d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.817+0000 7f53d0000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c806bfc0 con 0x7f53c80cf650 2024-03-23T11:24:47.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.817+0000 7f53d0000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c80ce140 con 0x7f53c815cdc0 2024-03-23T11:24:47.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.817+0000 7f53d0000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c80b4370 con 0x7f53c815c9e0 2024-03-23T11:24:47.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.818+0000 7f53ccc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f53c815cdc0 0x7f53c80d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.818+0000 7f53ccc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f53c815cdc0 0x7f53c80d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46008/0 (socket says 172.21.15.62:46008) 2024-03-23T11:24:47.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.818+0000 7f53ccc00640 1 -- 172.21.15.62:0/2628095611 learned_addr learned my addr 172.21.15.62:0/2628095611 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:47.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.818+0000 7f53cd600640 1 --2- 172.21.15.62:0/2628095611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53c815c9e0 0x7f53c81607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.818+0000 7f53ccc00640 1 -- 172.21.15.62:0/2628095611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53c815c9e0 msgr2=0x7f53c81607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:47.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.818+0000 7f53ccc00640 1 --2- 172.21.15.62:0/2628095611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53c815c9e0 0x7f53c81607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.818+0000 7f53ccc00640 1 -- 172.21.15.62:0/2628095611 >> v1:172.21.15.120:6789/0 conn(0x7f53c80cf650 legacy=0x7f53c815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.818+0000 7f53ccc00640 1 -- 172.21.15.62:0/2628095611 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c80b5260 con 0x7f53c815cdc0 2024-03-23T11:24:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.818+0000 7f53cd600640 1 --2- 172.21.15.62:0/2628095611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53c815c9e0 0x7f53c81607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:24:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.820+0000 7f53ccc00640 1 --2- 172.21.15.62:0/2628095611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c815cdc0 0x7f53c80d1f20 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009870 tx=0x7f53b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad1745e2f1bd23b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.820+0000 7f53c7e00640 1 -- 172.21.15.62:0/2628095611 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4030840 con 0x7f53c815cdc0 2024-03-23T11:24:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.820+0000 7f53c7e00640 1 -- 172.21.15.62:0/2628095611 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b40309a0 con 0x7f53c815cdc0 2024-03-23T11:24:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.820+0000 7f53c7e00640 1 -- 172.21.15.62:0/2628095611 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4030d30 con 0x7f53c815cdc0 2024-03-23T11:24:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.821+0000 7f53d0000640 1 -- 172.21.15.62:0/2628095611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c815cdc0 msgr2=0x7f53c80d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.821+0000 7f53d0000640 1 --2- 172.21.15.62:0/2628095611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c815cdc0 0x7f53c80d1f20 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009870 tx=0x7f53b402b980 comp rx=0 tx=0).stop 2024-03-23T11:24:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.821+0000 7f53d0000640 1 -- 172.21.15.62:0/2628095611 shutdown_connections 2024-03-23T11:24:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.821+0000 7f53d0000640 1 --2- 172.21.15.62:0/2628095611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c815cdc0 0x7f53c80d1f20 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.821+0000 7f53d0000640 1 --2- 172.21.15.62:0/2628095611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53c815c9e0 0x7f53c81607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.821+0000 7f53d0000640 1 -- 172.21.15.62:0/2628095611 >> 172.21.15.62:0/2628095611 conn(0x7f53c8071570 msgr2=0x7f53c8071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.821+0000 7f53d0000640 1 -- 172.21.15.62:0/2628095611 shutdown_connections 2024-03-23T11:24:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.821+0000 7f53d0000640 1 -- 172.21.15.62:0/2628095611 wait complete. 2024-03-23T11:24:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.823+0000 7f53d0000640 1 Processor -- start 2024-03-23T11:24:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.823+0000 7f53d0000640 1 -- start start 2024-03-23T11:24:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.824+0000 7f53d0000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53c80cf650 0x7f53c81f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.824+0000 7f53d0000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c815c9e0 0x7f53c80ab210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53d0000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c815cdc0 0x7f53c81f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53d0000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c80ac290 con 0x7f53c815c9e0 2024-03-23T11:24:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53d0000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c80ab750 con 0x7f53c815cdc0 2024-03-23T11:24:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53d0000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c80ab9f0 con 0x7f53c80cf650 2024-03-23T11:24:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53ccc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c815c9e0 0x7f53c80ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53ccc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c815c9e0 0x7f53c80ab210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46024/0 (socket says 172.21.15.62:46024) 2024-03-23T11:24:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53ccc00640 1 -- 172.21.15.62:0/3713461398 learned_addr learned my addr 172.21.15.62:0/3713461398 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:47.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53cd600640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53c80cf650 0x7f53c81f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53ce000640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c815cdc0 0x7f53c81f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53ccc00640 1 -- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53c80cf650 msgr2=0x7f53c81f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53ccc00640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53c80cf650 0x7f53c81f6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53ccc00640 1 -- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c815cdc0 msgr2=0x7f53c81f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:47.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53ccc00640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c815cdc0 0x7f53c81f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:47.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.825+0000 7f53ccc00640 1 -- 172.21.15.62:0/3713461398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40094d0 con 0x7f53c815c9e0 2024-03-23T11:24:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.826+0000 7f53ccc00640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c815c9e0 0x7f53c80ab210 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f53b4000c00 tx=0x7f53b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.826+0000 7f53c5600640 1 -- 172.21.15.62:0/3713461398 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4039a90 con 0x7f53c815c9e0 2024-03-23T11:24:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.826+0000 7f53c5600640 1 -- 172.21.15.62:0/3713461398 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b4039bf0 con 0x7f53c815c9e0 2024-03-23T11:24:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.826+0000 7f53c5600640 1 -- 172.21.15.62:0/3713461398 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b4043410 con 0x7f53c815c9e0 2024-03-23T11:24:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.826+0000 7f53d0000640 1 -- 172.21.15.62:0/3713461398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c80abcd0 con 0x7f53c815c9e0 2024-03-23T11:24:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.826+0000 7f53d0000640 1 -- 172.21.15.62:0/3713461398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c80a94a0 con 0x7f53c815c9e0 2024-03-23T11:24:47.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.828+0000 7f53d0000640 1 -- 172.21.15.62:0/3713461398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c80a9300 con 0x7f53c815c9e0 2024-03-23T11:24:47.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.828+0000 7f53c5600640 1 -- 172.21.15.62:0/3713461398 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f53b402f390 con 0x7f53c815c9e0 2024-03-23T11:24:47.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.829+0000 7f53c5600640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f53a806a7a0 0x7f53a806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:47.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.829+0000 7f53c5600640 1 -- 172.21.15.62:0/3713461398 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f53b40aafe0 con 0x7f53c815c9e0 2024-03-23T11:24:47.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.830+0000 7f53cd600640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f53a806a7a0 0x7f53a806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:47.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.831+0000 7f53cd600640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f53a806a7a0 0x7f53a806cc60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f53b8006fd0 tx=0x7f53b8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:47.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:47.834+0000 7f53c5600640 1 -- 172.21.15.62:0/3713461398 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f53c80a9300 con 0x7f53c815c9e0 2024-03-23T11:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.035+0000 7f53d0000640 1 -- 172.21.15.62:0/3713461398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53c80a9300 con 0x7f53c815c9e0 2024-03-23T11:24:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.036+0000 7f53c5600640 1 -- 172.21.15.62:0/3713461398 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f53b4034020 con 0x7f53c815c9e0 2024-03-23T11:24:48.033 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:24:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.038+0000 7f53d0000640 1 -- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f53a806a7a0 msgr2=0x7f53a806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.038+0000 7f53d0000640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f53a806a7a0 0x7f53a806cc60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f53b8006fd0 tx=0x7f53b8008040 comp rx=0 tx=0).stop 2024-03-23T11:24:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.039+0000 7f53d0000640 1 -- 172.21.15.62:0/3713461398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c815c9e0 msgr2=0x7f53c80ab210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.039+0000 7f53d0000640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c815c9e0 0x7f53c80ab210 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f53b4000c00 tx=0x7f53b4004730 comp rx=0 tx=0).stop 2024-03-23T11:24:48.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.039+0000 7f53d0000640 1 -- 172.21.15.62:0/3713461398 shutdown_connections 2024-03-23T11:24:48.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.039+0000 7f53d0000640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f53a806a7a0 0x7f53a806cc60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.039+0000 7f53d0000640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53c815c9e0 0x7f53c80ab210 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.039+0000 7f53d0000640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c815cdc0 0x7f53c81f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.039+0000 7f53d0000640 1 --2- 172.21.15.62:0/3713461398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f53c80cf650 0x7f53c81f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.039+0000 7f53d0000640 1 -- 172.21.15.62:0/3713461398 >> 172.21.15.62:0/3713461398 conn(0x7f53c8071570 msgr2=0x7f53c815f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:48.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.039+0000 7f53d0000640 1 -- 172.21.15.62:0/3713461398 shutdown_connections 2024-03-23T11:24:48.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.039+0000 7f53d0000640 1 -- 172.21.15.62:0/3713461398 wait complete. 2024-03-23T11:24:48.053 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:24:48.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.286+0000 7fc962200640 1 Processor -- start 2024-03-23T11:24:48.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.286+0000 7fc962200640 1 -- start start 2024-03-23T11:24:48.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.287+0000 7fc962200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc95c160420 0x7fc95c1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.287+0000 7fc962200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c0ce510 con 0x7fc95c15fcf0 2024-03-23T11:24:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.287+0000 7fc962200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c0cdc70 con 0x7fc95c0b5db0 2024-03-23T11:24:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.287+0000 7fc962200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc95c160df0 con 0x7fc95c160420 2024-03-23T11:24:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.288+0000 7fc95be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc95c160420 0x7fc95c1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.288+0000 7fc960e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc95c15fcf0 0x7fc95c0d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50300/0 (socket says 172.21.15.62:50300) 2024-03-23T11:24:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.288+0000 7fc960e00640 1 -- 172.21.15.62:0/1052910324 learned_addr learned my addr 172.21.15.62:0/1052910324 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.288+0000 7fc95be00640 1 -- 172.21.15.62:0/1052910324 >> v1:172.21.15.120:6789/0 conn(0x7fc95c0b5db0 legacy=0x7fc95c0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.288+0000 7fc95be00640 1 -- 172.21.15.62:0/1052910324 >> v1:172.21.15.62:6789/0 conn(0x7fc95c15fcf0 legacy=0x7fc95c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.288+0000 7fc95be00640 1 -- 172.21.15.62:0/1052910324 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c0cdf50 con 0x7fc95c160420 2024-03-23T11:24:48.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.289+0000 7fc95be00640 1 --2- 172.21.15.62:0/1052910324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c160420 0x7fc95c1608b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fc9480097a0 tx=0x7fc94802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70907598f5f104ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:48.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.290+0000 7fc95b400640 1 -- 172.21.15.62:0/1052910324 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94802e070 con 0x7fc95c160420 2024-03-23T11:24:48.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.290+0000 7fc95b400640 1 -- 172.21.15.62:0/1052910324 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9480315a0 con 0x7fc95c160420 2024-03-23T11:24:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.290+0000 7fc95b400640 1 -- 172.21.15.62:0/1052910324 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948031930 con 0x7fc95c160420 2024-03-23T11:24:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.290+0000 7fc962200640 1 -- 172.21.15.62:0/1052910324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c160420 msgr2=0x7fc95c1608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.290+0000 7fc962200640 1 --2- 172.21.15.62:0/1052910324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c160420 0x7fc95c1608b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fc9480097a0 tx=0x7fc94802b840 comp rx=0 tx=0).stop 2024-03-23T11:24:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.290+0000 7fc962200640 1 -- 172.21.15.62:0/1052910324 shutdown_connections 2024-03-23T11:24:48.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.290+0000 7fc962200640 1 --2- 172.21.15.62:0/1052910324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c160420 0x7fc95c1608b0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.290+0000 7fc962200640 1 -- 172.21.15.62:0/1052910324 >> 172.21.15.62:0/1052910324 conn(0x7fc95c089f60 msgr2=0x7fc95c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:48.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.291+0000 7fc962200640 1 -- 172.21.15.62:0/1052910324 shutdown_connections 2024-03-23T11:24:48.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.291+0000 7fc962200640 1 -- 172.21.15.62:0/1052910324 wait complete. 2024-03-23T11:24:48.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.292+0000 7fc962200640 1 Processor -- start 2024-03-23T11:24:48.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.292+0000 7fc962200640 1 -- start start 2024-03-23T11:24:48.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.293+0000 7fc962200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc95c0b5db0 0x7fc95c14a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.293+0000 7fc962200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c15fcf0 0x7fc95c0c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.293+0000 7fc962200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc95c160420 0x7fc95c0c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.293+0000 7fc962200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c09dfd0 con 0x7fc95c15fcf0 2024-03-23T11:24:48.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.293+0000 7fc962200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c14ae80 con 0x7fc95c0b5db0 2024-03-23T11:24:48.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.293+0000 7fc962200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c09c740 con 0x7fc95c160420 2024-03-23T11:24:48.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.294+0000 7fc95be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c15fcf0 0x7fc95c0c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.294+0000 7fc95be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c15fcf0 0x7fc95c0c64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46036/0 (socket says 172.21.15.62:46036) 2024-03-23T11:24:48.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.294+0000 7fc95be00640 1 -- 172.21.15.62:0/685604722 learned_addr learned my addr 172.21.15.62:0/685604722 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:48.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.294+0000 7fc960e00640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc95c0b5db0 0x7fc95c14a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.294+0000 7fc961800640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc95c160420 0x7fc95c0c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.294+0000 7fc95be00640 1 -- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc95c160420 msgr2=0x7fc95c0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.294+0000 7fc95be00640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc95c160420 0x7fc95c0c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.294+0000 7fc95be00640 1 -- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc95c0b5db0 msgr2=0x7fc95c14a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.294+0000 7fc95be00640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc95c0b5db0 0x7fc95c14a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.294+0000 7fc95be00640 1 -- 172.21.15.62:0/685604722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc948009400 con 0x7fc95c15fcf0 2024-03-23T11:24:48.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.295+0000 7fc95be00640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c15fcf0 0x7fc95c0c64b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc948034040 tx=0x7fc9480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:48.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.295+0000 7fc958c00640 1 -- 172.21.15.62:0/685604722 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948032900 con 0x7fc95c15fcf0 2024-03-23T11:24:48.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.295+0000 7fc958c00640 1 -- 172.21.15.62:0/685604722 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc948032a80 con 0x7fc95c15fcf0 2024-03-23T11:24:48.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.295+0000 7fc958c00640 1 -- 172.21.15.62:0/685604722 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948032e30 con 0x7fc95c15fcf0 2024-03-23T11:24:48.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.295+0000 7fc962200640 1 -- 172.21.15.62:0/685604722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95c09ca20 con 0x7fc95c15fcf0 2024-03-23T11:24:48.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.295+0000 7fc962200640 1 -- 172.21.15.62:0/685604722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95c14b510 con 0x7fc95c15fcf0 2024-03-23T11:24:48.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.296+0000 7fc962200640 1 -- 172.21.15.62:0/685604722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95c066bb0 con 0x7fc95c15fcf0 2024-03-23T11:24:48.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.297+0000 7fc958c00640 1 -- 172.21.15.62:0/685604722 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc94803b3f0 con 0x7fc95c15fcf0 2024-03-23T11:24:48.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.298+0000 7fc958c00640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc93c06a7a0 0x7fc93c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.298+0000 7fc958c00640 1 -- 172.21.15.62:0/685604722 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc9480aaea0 con 0x7fc95c15fcf0 2024-03-23T11:24:48.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.299+0000 7fc960e00640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc93c06a7a0 0x7fc93c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.300+0000 7fc960e00640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc93c06a7a0 0x7fc93c06cc60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc94c005880 tx=0x7fc94c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:48.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.303+0000 7fc958c00640 1 -- 172.21.15.62:0/685604722 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc948076280 con 0x7fc95c15fcf0 2024-03-23T11:24:48.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.503+0000 7fc962200640 1 -- 172.21.15.62:0/685604722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc95c0cea40 con 0x7fc95c15fcf0 2024-03-23T11:24:48.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.504+0000 7fc958c00640 1 -- 172.21.15.62:0/685604722 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc9480798a0 con 0x7fc95c15fcf0 2024-03-23T11:24:48.499 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:24:48.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.506+0000 7fc962200640 1 -- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc93c06a7a0 msgr2=0x7fc93c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.506+0000 7fc962200640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc93c06a7a0 0x7fc93c06cc60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc94c005880 tx=0x7fc94c009040 comp rx=0 tx=0).stop 2024-03-23T11:24:48.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.506+0000 7fc962200640 1 -- 172.21.15.62:0/685604722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c15fcf0 msgr2=0x7fc95c0c64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.506+0000 7fc962200640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c15fcf0 0x7fc95c0c64b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc948034040 tx=0x7fc9480046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.506+0000 7fc962200640 1 -- 172.21.15.62:0/685604722 shutdown_connections 2024-03-23T11:24:48.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.506+0000 7fc962200640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc93c06a7a0 0x7fc93c06cc60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.506+0000 7fc962200640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc95c15fcf0 0x7fc95c0c64b0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.507+0000 7fc962200640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc95c160420 0x7fc95c0c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.507+0000 7fc962200640 1 --2- 172.21.15.62:0/685604722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc95c0b5db0 0x7fc95c14a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.507+0000 7fc962200640 1 -- 172.21.15.62:0/685604722 >> 172.21.15.62:0/685604722 conn(0x7fc95c089f60 msgr2=0x7fc95c0b6910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:48.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.507+0000 7fc962200640 1 -- 172.21.15.62:0/685604722 shutdown_connections 2024-03-23T11:24:48.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.507+0000 7fc962200640 1 -- 172.21.15.62:0/685604722 wait complete. 2024-03-23T11:24:48.522 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:24:48.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.755+0000 7f03c9800640 1 Processor -- start 2024-03-23T11:24:48.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.756+0000 7f03c9800640 1 -- start start 2024-03-23T11:24:48.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.757+0000 7f03c9800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c40b2f30 0x7f03c4070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.757+0000 7f03c9800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c40cd140 con 0x7f03c4070090 2024-03-23T11:24:48.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.757+0000 7f03c9800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c40cd3e0 con 0x7f03c4143dd0 2024-03-23T11:24:48.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.757+0000 7f03c9800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c4070a60 con 0x7f03c40b2f30 2024-03-23T11:24:48.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.758+0000 7f03bbe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f03c4070090 0x7f03c40b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50316/0 (socket says 172.21.15.62:50316) 2024-03-23T11:24:48.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.758+0000 7f03bbe00640 1 -- 172.21.15.62:0/1113661279 learned_addr learned my addr 172.21.15.62:0/1113661279 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.758+0000 7f03c8e00640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c40b2f30 0x7f03c4070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.758+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900432275 0 0) 0x7f03c40cd140 con 0x7f03c4070090 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.758+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03a8003520 con 0x7f03c4070090 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 153907294 0 0) 0x7f03a8003520 con 0x7f03c4070090 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c40b2f30 msgr2=0x7f03c4070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c40b2f30 0x7f03c4070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 >> v1:172.21.15.120:6789/0 conn(0x7f03c4143dd0 legacy=0x7f03c415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c40cd6c0 con 0x7f03c4070090 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f03ac001670 con 0x7f03c4070090 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 >> v1:172.21.15.62:6789/0 conn(0x7f03c4070090 legacy=0x7f03c40b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c40b2f30 0x7f03a80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03a8003f70 0x7f03a80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a8004d00 0x7f03a8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c4070a60 con 0x7f03a8003f70 2024-03-23T11:24:48.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a800bac0 con 0x7f03a8004d00 2024-03-23T11:24:48.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a800bc80 con 0x7f03c40b2f30 2024-03-23T11:24:48.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.759+0000 7f03bbe00640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03a8003f70 0x7f03a80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.760+0000 7f03c3e00640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c40b2f30 0x7f03a80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.760+0000 7f03c8e00640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a8004d00 0x7f03a8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.760+0000 7f03bbe00640 1 -- 172.21.15.62:0/1113661279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c40b2f30 msgr2=0x7f03a80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:48.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.760+0000 7f03bbe00640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c40b2f30 0x7f03a80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.760+0000 7f03bbe00640 1 -- 172.21.15.62:0/1113661279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a8004d00 msgr2=0x7f03a8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:48.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.760+0000 7f03bbe00640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a8004d00 0x7f03a8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.760+0000 7f03bbe00640 1 -- 172.21.15.62:0/1113661279 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a8003520 con 0x7f03a8003f70 2024-03-23T11:24:48.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.761+0000 7f03bbe00640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03a8003f70 0x7f03a80047c0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00c0e0 tx=0x7f03a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3047a5f89ea4d827 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:48.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.761+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac005ad0 con 0x7f03a8003f70 2024-03-23T11:24:48.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.761+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ac005c50 con 0x7f03a8003f70 2024-03-23T11:24:48.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.761+0000 7f03c3400640 1 -- 172.21.15.62:0/1113661279 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac007070 con 0x7f03a8003f70 2024-03-23T11:24:48.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.762+0000 7f03c9800640 1 -- 172.21.15.62:0/1113661279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03a8003f70 msgr2=0x7f03a80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.762+0000 7f03c9800640 1 --2- 172.21.15.62:0/1113661279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03a8003f70 0x7f03a80047c0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00c0e0 tx=0x7f03a80057f0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.762+0000 7f03c8e00640 1 -- 172.21.15.62:0/1113661279 reap_dead start 2024-03-23T11:24:48.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.762+0000 7f03c9800640 1 -- 172.21.15.62:0/1113661279 shutdown_connections 2024-03-23T11:24:48.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.762+0000 7f03c9800640 1 -- 172.21.15.62:0/1113661279 >> 172.21.15.62:0/1113661279 conn(0x7f03c4066050 msgr2=0x7f03c4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:48.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.762+0000 7f03c9800640 1 -- 172.21.15.62:0/1113661279 shutdown_connections 2024-03-23T11:24:48.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.762+0000 7f03c9800640 1 -- 172.21.15.62:0/1113661279 wait complete. 2024-03-23T11:24:48.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.764+0000 7f03c9800640 1 Processor -- start 2024-03-23T11:24:48.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.764+0000 7f03c9800640 1 -- start start 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.765+0000 7f03c9800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a8003f70 0x7f03c41f0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.765+0000 7f03c9800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c40a8480 0x7f03c41393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.765+0000 7f03c9800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c41417f0 0x7f03c4141c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.765+0000 7f03c9800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c41421c0 con 0x7f03c40a8480 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.765+0000 7f03c9800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c40acc70 con 0x7f03a8003f70 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.765+0000 7f03c9800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c41f0d60 con 0x7f03c41417f0 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.765+0000 7f03bbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c40a8480 0x7f03c41393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.766+0000 7f03bbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c40a8480 0x7f03c41393d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46062/0 (socket says 172.21.15.62:46062) 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.766+0000 7f03bbe00640 1 -- 172.21.15.62:0/4201451430 learned_addr learned my addr 172.21.15.62:0/4201451430 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.766+0000 7f03c3e00640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a8003f70 0x7f03c41f0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.766+0000 7f03c8e00640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c41417f0 0x7f03c4141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.766+0000 7f03bbe00640 1 -- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c41417f0 msgr2=0x7f03c4141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.766+0000 7f03bbe00640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c41417f0 0x7f03c4141c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.766+0000 7f03bbe00640 1 -- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a8003f70 msgr2=0x7f03c41f0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.766+0000 7f03bbe00640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a8003f70 0x7f03c41f0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.766+0000 7f03bbe00640 1 -- 172.21.15.62:0/4201451430 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c40a8940 con 0x7f03c40a8480 2024-03-23T11:24:48.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.766+0000 7f03bbe00640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c40a8480 0x7f03c41393d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00c140 tx=0x7f03ac001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:48.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.767+0000 7f03c0c00640 1 -- 172.21.15.62:0/4201451430 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac039070 con 0x7f03c40a8480 2024-03-23T11:24:48.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.767+0000 7f03c0c00640 1 -- 172.21.15.62:0/4201451430 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ac007580 con 0x7f03c40a8480 2024-03-23T11:24:48.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.767+0000 7f03c0c00640 1 -- 172.21.15.62:0/4201451430 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac03c040 con 0x7f03c40a8480 2024-03-23T11:24:48.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.767+0000 7f03c9800640 1 -- 172.21.15.62:0/4201451430 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c40a8c20 con 0x7f03c40a8480 2024-03-23T11:24:48.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.767+0000 7f03c9800640 1 -- 172.21.15.62:0/4201451430 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c40a91e0 con 0x7f03c40a8480 2024-03-23T11:24:48.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.768+0000 7f03c9800640 1 -- 172.21.15.62:0/4201451430 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c40a9400 con 0x7f03c40a8480 2024-03-23T11:24:48.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.769+0000 7f03c0c00640 1 -- 172.21.15.62:0/4201451430 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f03ac032070 con 0x7f03c40a8480 2024-03-23T11:24:48.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.770+0000 7f03c0c00640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03a006a7a0 0x7f03a006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:48.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.770+0000 7f03c0c00640 1 -- 172.21.15.62:0/4201451430 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f03ac0aa160 con 0x7f03c40a8480 2024-03-23T11:24:48.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.770+0000 7f03c3e00640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03a006a7a0 0x7f03a006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:48.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.772+0000 7f03c3e00640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03a006a7a0 0x7f03a006cc60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f03b4007fd0 tx=0x7f03b4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:48.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.774+0000 7f03c0c00640 1 -- 172.21.15.62:0/4201451430 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f03ac075450 con 0x7f03c40a8480 2024-03-23T11:24:48.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.971+0000 7f03c9800640 1 -- 172.21.15.62:0/4201451430 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f03c40cca40 con 0x7f03c40a8480 2024-03-23T11:24:48.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.972+0000 7f03c0c00640 1 -- 172.21.15.62:0/4201451430 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) v1 ==== 100+0+11 (secure 0 0 0) 0x7f03ac07de60 con 0x7f03c40a8480 2024-03-23T11:24:48.967 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:24:48.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.975+0000 7f03c9800640 1 -- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03a006a7a0 msgr2=0x7f03a006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.975+0000 7f03c9800640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03a006a7a0 0x7f03a006cc60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f03b4007fd0 tx=0x7f03b4009040 comp rx=0 tx=0).stop 2024-03-23T11:24:48.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.975+0000 7f03c9800640 1 -- 172.21.15.62:0/4201451430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c40a8480 msgr2=0x7f03c41393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.975+0000 7f03c9800640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c40a8480 0x7f03c41393d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00c140 tx=0x7f03ac001930 comp rx=0 tx=0).stop 2024-03-23T11:24:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.975+0000 7f03c9800640 1 -- 172.21.15.62:0/4201451430 shutdown_connections 2024-03-23T11:24:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.975+0000 7f03c9800640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03a006a7a0 0x7f03a006cc60 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.975+0000 7f03c9800640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c40a8480 0x7f03c41393d0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.975+0000 7f03c9800640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03c41417f0 0x7f03c4141c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.975+0000 7f03c9800640 1 --2- 172.21.15.62:0/4201451430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03a8003f70 0x7f03c41f0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.975+0000 7f03c9800640 1 -- 172.21.15.62:0/4201451430 >> 172.21.15.62:0/4201451430 conn(0x7f03c4066050 msgr2=0x7f03c40b09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.976+0000 7f03c9800640 1 -- 172.21.15.62:0/4201451430 shutdown_connections 2024-03-23T11:24:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:48.976+0000 7f03c9800640 1 -- 172.21.15.62:0/4201451430 wait complete. 2024-03-23T11:24:48.992 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:24:49.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.214+0000 7fc1d9600640 1 Processor -- start 2024-03-23T11:24:49.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.214+0000 7fc1d9600640 1 -- start start 2024-03-23T11:24:49.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.216+0000 7fc1d9600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc1d40b2d50 0x7fc1d40cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:49.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.216+0000 7fc1d9600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d40d0810 con 0x7fc1d415c960 2024-03-23T11:24:49.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.216+0000 7fc1d9600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d40cd9d0 con 0x7fc1d40d13b0 2024-03-23T11:24:49.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.216+0000 7fc1d9600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d40cdcb0 con 0x7fc1d40b2d50 2024-03-23T11:24:49.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.216+0000 7fc1d3e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc1d415c960 0x7fc1d40d0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50330/0 (socket says 172.21.15.62:50330) 2024-03-23T11:24:49.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.216+0000 7fc1d3e00640 1 -- 172.21.15.62:0/1363494873 learned_addr learned my addr 172.21.15.62:0/1363494873 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:24:49.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.216+0000 7fc1d8c00640 1 --2- 172.21.15.62:0/1363494873 >> v2:172.21.15.62:3300/0 conn(0x7fc1d40b2d50 0x7fc1d40cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:49.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.216+0000 7fc1d2a00640 1 -- 172.21.15.62:0/1363494873 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221398549 0 0) 0x7fc1c4001110 con 0x7fc1d415c960 2024-03-23T11:24:49.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.216+0000 7fc1d2a00640 1 -- 172.21.15.62:0/1363494873 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b8003520 con 0x7fc1d415c960 2024-03-23T11:24:49.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.216+0000 7fc1d8c00640 1 -- 172.21.15.62:0/1363494873 >> v1:172.21.15.120:6789/0 conn(0x7fc1d40d13b0 legacy=0x7fc1d40b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:49.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.216+0000 7fc1d8c00640 1 -- 172.21.15.62:0/1363494873 >> v1:172.21.15.62:6789/0 conn(0x7fc1d415c960 legacy=0x7fc1d40d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:49.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.217+0000 7fc1d8c00640 1 -- 172.21.15.62:0/1363494873 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d40cdf90 con 0x7fc1d40b2d50 2024-03-23T11:24:49.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.218+0000 7fc1d8c00640 1 --2- 172.21.15.62:0/1363494873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1d40b2d50 0x7fc1d40cf790 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009d90 tx=0x7fc1cc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875b18883de505b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:49.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.218+0000 7fc1d2a00640 1 -- 172.21.15.62:0/1363494873 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc02cd00 con 0x7fc1d40b2d50 2024-03-23T11:24:49.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.218+0000 7fc1d2a00640 1 -- 172.21.15.62:0/1363494873 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1cc02ce60 con 0x7fc1d40b2d50 2024-03-23T11:24:49.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.218+0000 7fc1d2a00640 1 -- 172.21.15.62:0/1363494873 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc034640 con 0x7fc1d40b2d50 2024-03-23T11:24:49.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.219+0000 7fc1d9600640 1 -- 172.21.15.62:0/1363494873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1d40b2d50 msgr2=0x7fc1d40cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.219+0000 7fc1d9600640 1 --2- 172.21.15.62:0/1363494873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1d40b2d50 0x7fc1d40cf790 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009d90 tx=0x7fc1cc02bf50 comp rx=0 tx=0).stop 2024-03-23T11:24:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.219+0000 7fc1d9600640 1 -- 172.21.15.62:0/1363494873 shutdown_connections 2024-03-23T11:24:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.219+0000 7fc1d9600640 1 --2- 172.21.15.62:0/1363494873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1d40b2d50 0x7fc1d40cf790 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.219+0000 7fc1d9600640 1 -- 172.21.15.62:0/1363494873 >> 172.21.15.62:0/1363494873 conn(0x7fc1d4089f60 msgr2=0x7fc1d40d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.219+0000 7fc1d9600640 1 -- 172.21.15.62:0/1363494873 shutdown_connections 2024-03-23T11:24:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.219+0000 7fc1d9600640 1 -- 172.21.15.62:0/1363494873 wait complete. 2024-03-23T11:24:49.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.221+0000 7fc1d9600640 1 Processor -- start 2024-03-23T11:24:49.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.221+0000 7fc1d9600640 1 -- start start 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d9600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1d40b2d50 0x7fc1d41f55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d9600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1d40d13b0 0x7fc1d41f5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d9600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1d415c960 0x7fc1d4159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d9600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40c6e40 con 0x7fc1d40d13b0 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d9600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d41477a0 con 0x7fc1d40b2d50 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d9600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d415a310 con 0x7fc1d415c960 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1d40d13b0 0x7fc1d41f5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1d40d13b0 0x7fc1d41f5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46082/0 (socket says 172.21.15.62:46082) 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d3400640 1 -- 172.21.15.62:0/2675692477 learned_addr learned my addr 172.21.15.62:0/2675692477 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d3e00640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1d40b2d50 0x7fc1d41f55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d8c00640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1d415c960 0x7fc1d4159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d3400640 1 -- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1d415c960 msgr2=0x7fc1d4159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d3400640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1d415c960 0x7fc1d4159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:49.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d3400640 1 -- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1d40b2d50 msgr2=0x7fc1d41f55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:49.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d3400640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1d40b2d50 0x7fc1d41f55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:49.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d3400640 1 -- 172.21.15.62:0/2675692477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc009a70 con 0x7fc1d40d13b0 2024-03-23T11:24:49.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.222+0000 7fc1d3e00640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1d40b2d50 0x7fc1d41f55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:49.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.223+0000 7fc1d3400640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1d40d13b0 0x7fc1d41f5b10 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0004640 tx=0x7fc1c000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:49.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.223+0000 7fc1cbe00640 1 -- 172.21.15.62:0/2675692477 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0011790 con 0x7fc1d40d13b0 2024-03-23T11:24:49.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.223+0000 7fc1cbe00640 1 -- 172.21.15.62:0/2675692477 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c00118f0 con 0x7fc1d40d13b0 2024-03-23T11:24:49.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.223+0000 7fc1cbe00640 1 -- 172.21.15.62:0/2675692477 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0011ca0 con 0x7fc1d40d13b0 2024-03-23T11:24:49.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.223+0000 7fc1d9600640 1 -- 172.21.15.62:0/2675692477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d415a5f0 con 0x7fc1d40d13b0 2024-03-23T11:24:49.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.223+0000 7fc1d9600640 1 -- 172.21.15.62:0/2675692477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d40c6a60 con 0x7fc1d40d13b0 2024-03-23T11:24:49.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.224+0000 7fc1d9600640 1 -- 172.21.15.62:0/2675692477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d40c6f80 con 0x7fc1d40d13b0 2024-03-23T11:24:49.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.225+0000 7fc1cbe00640 1 -- 172.21.15.62:0/2675692477 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc1c0012420 con 0x7fc1d40d13b0 2024-03-23T11:24:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.226+0000 7fc1cbe00640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc1b406a7a0 0x7fc1b406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.226+0000 7fc1cbe00640 1 -- 172.21.15.62:0/2675692477 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc1c0088a60 con 0x7fc1d40d13b0 2024-03-23T11:24:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.227+0000 7fc1d3e00640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc1b406a7a0 0x7fc1b406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:49.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.228+0000 7fc1d3e00640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc1b406a7a0 0x7fc1b406cc60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4001270 tx=0x7fc1c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:49.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.231+0000 7fc1cbe00640 1 -- 172.21.15.62:0/2675692477 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc1d40c6f80 con 0x7fc1d40d13b0 2024-03-23T11:24:49.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.444+0000 7fc1d9600640 1 -- 172.21.15.62:0/2675692477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc1d40c6f80 con 0x7fc1d40d13b0 2024-03-23T11:24:49.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.445+0000 7fc1cbe00640 1 -- 172.21.15.62:0/2675692477 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v77) v1 ==== 80+0+124 (secure 0 0 0) 0x7fc1d40c6f80 con 0x7fc1d40d13b0 2024-03-23T11:24:49.440 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6837/3833782560 2024-03-24T11:24:34.350690+0000 2024-03-23T11:24:49.440 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6836/3833782560 2024-03-24T11:24:34.350690+0000 2024-03-23T11:24:49.440 INFO:teuthology.orchestra.run.smithi062.stderr:listed 2 entries 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 -- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc1b406a7a0 msgr2=0x7fc1b406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc1b406a7a0 0x7fc1b406cc60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4001270 tx=0x7fc1c4009040 comp rx=0 tx=0).stop 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 -- 172.21.15.62:0/2675692477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1d40d13b0 msgr2=0x7fc1d41f5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1d40d13b0 0x7fc1d41f5b10 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0004640 tx=0x7fc1c000c8e0 comp rx=0 tx=0).stop 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 -- 172.21.15.62:0/2675692477 shutdown_connections 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc1b406a7a0 0x7fc1b406cc60 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1d40d13b0 0x7fc1d41f5b10 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1d415c960 0x7fc1d4159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 --2- 172.21.15.62:0/2675692477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1d40b2d50 0x7fc1d41f55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 -- 172.21.15.62:0/2675692477 >> 172.21.15.62:0/2675692477 conn(0x7fc1d4089f60 msgr2=0x7fc1d40d0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 -- 172.21.15.62:0/2675692477 shutdown_connections 2024-03-23T11:24:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:49.448+0000 7fc1d9600640 1 -- 172.21.15.62:0/2675692477 wait complete. 2024-03-23T11:24:49.463 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:24:49.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:49.463 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:24:49.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[61619]: fuse finished with error 0 and tester_r 0 2024-03-23T11:24:49.559 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:24:55.661 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:24:55.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:24:55.662 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:24:55.723 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub start /subdir recursive 2024-03-23T11:24:55.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.957+0000 7fc28d000640 1 Processor -- start 2024-03-23T11:24:55.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.957+0000 7fc28d000640 1 -- start start 2024-03-23T11:24:55.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.958+0000 7fc28d000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc28806d950 0x7fc28806c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.958+0000 7fc28d000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28806dd30 0x7fc288070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.959+0000 7fc28d000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc2880711f0 0x7fc2880ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.959+0000 7fc28d000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28815cfd0 con 0x7fc2880711f0 2024-03-23T11:24:55.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.959+0000 7fc28d000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28815d220 con 0x7fc28806d950 2024-03-23T11:24:55.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.959+0000 7fc28d000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28815d500 con 0x7fc28806dd30 2024-03-23T11:24:55.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.959+0000 7fc287e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc2880711f0 0x7fc2880ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.959+0000 7fc287e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc2880711f0 0x7fc2880ce690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46088/0 (socket says 172.21.15.62:46088) 2024-03-23T11:24:55.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.959+0000 7fc287e00640 1 -- 172.21.15.62:0/369759315 learned_addr learned my addr 172.21.15.62:0/369759315 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:55.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.959+0000 7fc287400640 1 --2- 172.21.15.62:0/369759315 >> v2:172.21.15.120:3300/0 conn(0x7fc28806d950 0x7fc28806c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.959+0000 7fc286a00640 1 --2- 172.21.15.62:0/369759315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28806dd30 0x7fc288070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.959+0000 7fc287e00640 1 -- 172.21.15.62:0/369759315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28806dd30 msgr2=0x7fc288070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:55.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.959+0000 7fc287e00640 1 --2- 172.21.15.62:0/369759315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28806dd30 0x7fc288070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.960+0000 7fc287e00640 1 -- 172.21.15.62:0/369759315 >> v2:172.21.15.120:3300/0 conn(0x7fc28806d950 msgr2=0x7fc28806c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:55.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.960+0000 7fc287e00640 1 --2- 172.21.15.62:0/369759315 >> v2:172.21.15.120:3300/0 conn(0x7fc28806d950 0x7fc28806c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.960+0000 7fc287e00640 1 -- 172.21.15.62:0/369759315 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28815d7e0 con 0x7fc2880711f0 2024-03-23T11:24:55.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.961+0000 7fc287e00640 1 --2- 172.21.15.62:0/369759315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2880711f0 0x7fc2880ce690 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc2880707c0 tx=0x7fc28002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c86f0fd2c917d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:55.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.961+0000 7fc286000640 1 -- 172.21.15.62:0/369759315 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28002c9e0 con 0x7fc2880711f0 2024-03-23T11:24:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.961+0000 7fc286000640 1 -- 172.21.15.62:0/369759315 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28002cb40 con 0x7fc2880711f0 2024-03-23T11:24:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.961+0000 7fc286000640 1 -- 172.21.15.62:0/369759315 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2800343f0 con 0x7fc2880711f0 2024-03-23T11:24:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.962+0000 7fc28d000640 1 -- 172.21.15.62:0/369759315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2880711f0 msgr2=0x7fc2880ce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.962+0000 7fc28d000640 1 --2- 172.21.15.62:0/369759315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2880711f0 0x7fc2880ce690 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc2880707c0 tx=0x7fc28002be40 comp rx=0 tx=0).stop 2024-03-23T11:24:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.962+0000 7fc28d000640 1 -- 172.21.15.62:0/369759315 shutdown_connections 2024-03-23T11:24:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.962+0000 7fc28d000640 1 --2- 172.21.15.62:0/369759315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2880711f0 0x7fc2880ce690 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.962+0000 7fc28d000640 1 --2- 172.21.15.62:0/369759315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28806dd30 0x7fc288070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.962+0000 7fc28d000640 1 --2- 172.21.15.62:0/369759315 >> v2:172.21.15.120:3300/0 conn(0x7fc28806d950 0x7fc28806c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.962+0000 7fc28d000640 1 -- 172.21.15.62:0/369759315 >> 172.21.15.62:0/369759315 conn(0x7fc2880b0280 msgr2=0x7fc2880b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.962+0000 7fc28d000640 1 -- 172.21.15.62:0/369759315 shutdown_connections 2024-03-23T11:24:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.962+0000 7fc28d000640 1 -- 172.21.15.62:0/369759315 wait complete. 2024-03-23T11:24:55.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.964+0000 7fc28d000640 1 Processor -- start 2024-03-23T11:24:55.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.964+0000 7fc28d000640 1 -- start start 2024-03-23T11:24:55.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.965+0000 7fc28d000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28806d950 0x7fc28814e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.965+0000 7fc28d000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc28806dd30 0x7fc2880aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.965+0000 7fc28d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2880711f0 0x7fc288158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.965+0000 7fc28d000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28814bbe0 con 0x7fc2880711f0 2024-03-23T11:24:55.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.965+0000 7fc28d000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28814bec0 con 0x7fc28806dd30 2024-03-23T11:24:55.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.965+0000 7fc28d000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28814c1a0 con 0x7fc28806d950 2024-03-23T11:24:55.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.965+0000 7fc287e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2880711f0 0x7fc288158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.965+0000 7fc287e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2880711f0 0x7fc288158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46102/0 (socket says 172.21.15.62:46102) 2024-03-23T11:24:55.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.965+0000 7fc287e00640 1 -- 172.21.15.62:0/3365457298 learned_addr learned my addr 172.21.15.62:0/3365457298 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:55.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.966+0000 7fc287400640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28806d950 0x7fc28814e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.966+0000 7fc286a00640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc28806dd30 0x7fc2880aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.966+0000 7fc287e00640 1 -- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28806d950 msgr2=0x7fc28814e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:55.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.966+0000 7fc287e00640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28806d950 0x7fc28814e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.966+0000 7fc287e00640 1 -- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc28806dd30 msgr2=0x7fc2880aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:55.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.966+0000 7fc287e00640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc28806dd30 0x7fc2880aec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.966+0000 7fc287e00640 1 -- 172.21.15.62:0/3365457298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc280009b30 con 0x7fc2880711f0 2024-03-23T11:24:55.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.966+0000 7fc286a00640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc28806dd30 0x7fc2880aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:55.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.966+0000 7fc287e00640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2880711f0 0x7fc288158270 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fc280036040 tx=0x7fc280004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:55.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.967+0000 7fc27f400640 1 -- 172.21.15.62:0/3365457298 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280009510 con 0x7fc2880711f0 2024-03-23T11:24:55.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.967+0000 7fc27f400640 1 -- 172.21.15.62:0/3365457298 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28003e450 con 0x7fc2880711f0 2024-03-23T11:24:55.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.967+0000 7fc28d000640 1 -- 172.21.15.62:0/3365457298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28814c480 con 0x7fc2880711f0 2024-03-23T11:24:55.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.967+0000 7fc27f400640 1 -- 172.21.15.62:0/3365457298 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28003e800 con 0x7fc2880711f0 2024-03-23T11:24:55.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.967+0000 7fc28d000640 1 -- 172.21.15.62:0/3365457298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc288152a60 con 0x7fc2880711f0 2024-03-23T11:24:55.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.969+0000 7fc27f400640 1 -- 172.21.15.62:0/3365457298 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc28003e960 con 0x7fc2880711f0 2024-03-23T11:24:55.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.970+0000 7fc27f400640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc26406a7d0 0x7fc26406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.970+0000 7fc27f400640 1 -- 172.21.15.62:0/3365457298 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc2800abbd0 con 0x7fc2880711f0 2024-03-23T11:24:55.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.971+0000 7fc287400640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc26406a7d0 0x7fc26406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.971+0000 7fc291224740 1 Processor -- start 2024-03-23T11:24:55.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.971+0000 7fc291224740 1 -- start start 2024-03-23T11:24:55.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.972+0000 7fc291224740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea5cdb0 0x555bdea143e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.972+0000 7fc291224740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea12e30 0x555bdea14920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.972+0000 7fc291224740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea0f2d0 0x555bdea3f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.972+0000 7fc291224740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bde9805f0 con 0x555bdea12e30 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.972+0000 7fc291224740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bde97dd80 con 0x555bdea0f2d0 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.972+0000 7fc291224740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555bde979670 con 0x555bdea5cdb0 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.972+0000 7fc287400640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc26406a7d0 0x7fc26406cc90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc278006fd0 tx=0x7fc278008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.972+0000 7fc286a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea12e30 0x555bdea14920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.972+0000 7fc286a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea12e30 0x555bdea14920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46110/0 (socket says 172.21.15.62:46110) 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.972+0000 7fc286a00640 1 -- 172.21.15.62:0/2232825016 learned_addr learned my addr 172.21.15.62:0/2232825016 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.972+0000 7fc287400640 1 --2- 172.21.15.62:0/2232825016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea5cdb0 0x555bdea143e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.973+0000 7fc287e00640 1 --2- 172.21.15.62:0/2232825016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea0f2d0 0x555bdea3f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.973+0000 7fc286a00640 1 -- 172.21.15.62:0/2232825016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea5cdb0 msgr2=0x555bdea143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.973+0000 7fc286a00640 1 --2- 172.21.15.62:0/2232825016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea5cdb0 0x555bdea143e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.973+0000 7fc286a00640 1 -- 172.21.15.62:0/2232825016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea0f2d0 msgr2=0x555bdea3f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:55.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.973+0000 7fc286a00640 1 --2- 172.21.15.62:0/2232825016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea0f2d0 0x555bdea3f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.973+0000 7fc286a00640 1 -- 172.21.15.62:0/2232825016 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bde92b140 con 0x555bdea12e30 2024-03-23T11:24:55.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.973+0000 7fc286a00640 1 --2- 172.21.15.62:0/2232825016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea12e30 0x555bdea14920 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc270009970 tx=0x7fc270009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bda092b5fb8a9c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:55.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.973+0000 7fc277e00640 1 -- 172.21.15.62:0/2232825016 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27000c070 con 0x555bdea12e30 2024-03-23T11:24:55.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.973+0000 7fc277e00640 1 -- 172.21.15.62:0/2232825016 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc270004020 con 0x555bdea12e30 2024-03-23T11:24:55.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.973+0000 7fc277e00640 1 -- 172.21.15.62:0/2232825016 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2700043d0 con 0x555bdea12e30 2024-03-23T11:24:55.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.974+0000 7fc291224740 1 -- 172.21.15.62:0/2232825016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea12e30 msgr2=0x555bdea14920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:55.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.974+0000 7fc291224740 1 --2- 172.21.15.62:0/2232825016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea12e30 0x555bdea14920 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc270009970 tx=0x7fc270009e30 comp rx=0 tx=0).stop 2024-03-23T11:24:55.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.974+0000 7fc291224740 1 -- 172.21.15.62:0/2232825016 shutdown_connections 2024-03-23T11:24:55.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.974+0000 7fc291224740 1 --2- 172.21.15.62:0/2232825016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea12e30 0x555bdea14920 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.974+0000 7fc291224740 1 --2- 172.21.15.62:0/2232825016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea0f2d0 0x555bdea3f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.974+0000 7fc291224740 1 --2- 172.21.15.62:0/2232825016 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea5cdb0 0x555bdea143e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.974+0000 7fc291224740 1 -- 172.21.15.62:0/2232825016 >> 172.21.15.62:0/2232825016 conn(0x555bdea11540 msgr2=0x555bdea5a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:55.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.974+0000 7fc291224740 1 -- 172.21.15.62:0/2232825016 shutdown_connections 2024-03-23T11:24:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.974+0000 7fc291224740 1 -- 172.21.15.62:0/2232825016 wait complete. 2024-03-23T11:24:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.975+0000 7fc291224740 1 -- start start 2024-03-23T11:24:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.975+0000 7fc291224740 1 Processor -- start 2024-03-23T11:24:55.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.976+0000 7fc291224740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea12e30 0x555bdea37300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.976+0000 7fc291224740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea5cdb0 0x555bdea5f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.976+0000 7fc291224740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea238e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.976+0000 7fc291224740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bde91a6e0 con 0x555bdea24d40 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.976+0000 7fc291224740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bde91a380 con 0x555bdea5cdb0 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.976+0000 7fc291224740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555bde6771b0 con 0x555bdea12e30 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.976+0000 7fc291224740 10 client.0 _ll_register_callbacks cb 0x555bde900ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.976+0000 7fc287e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea238e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.976+0000 7fc287e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea238e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46114/0 (socket says 172.21.15.62:46114) 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.976+0000 7fc287e00640 1 -- 172.21.15.62:0/2857383699 learned_addr learned my addr 172.21.15.62:0/2857383699 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc287400640 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea12e30 0x555bdea37300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc286a00640 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea5cdb0 0x555bdea5f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc287e00640 1 -- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea12e30 msgr2=0x555bdea37300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc287e00640 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea12e30 0x555bdea37300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc287e00640 1 -- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea5cdb0 msgr2=0x555bdea5f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc287e00640 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea5cdb0 0x555bdea5f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc287e00640 1 -- 172.21.15.62:0/2857383699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2700095d0 con 0x555bdea24d40 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc286a00640 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea5cdb0 0x555bdea5f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc287e00640 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea238e0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc2800094a0 tx=0x7fc28007a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8a3ea1680684fa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc274c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc274c00640 1 -- 172.21.15.62:0/2857383699 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28007ea40 con 0x555bdea24d40 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc274c00640 1 -- 172.21.15.62:0/2857383699 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc280083000 con 0x555bdea24d40 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.977+0000 7fc291224740 1 -- 172.21.15.62:0/2857383699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555bde66d700 con 0x555bdea24d40 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.978+0000 7fc274c00640 1 -- 172.21.15.62:0/2857383699 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc28007a100 con 0x555bdea24d40 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.978+0000 7fc291224740 10 client.6978 fetch_fsmap learned FSMap version 49 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.978+0000 7fc291224740 1 -- 172.21.15.62:0/2857383699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x555bde977cf0 con 0x555bdea24d40 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.978+0000 7fc274c00640 1 -- 172.21.15.62:0/2857383699 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc280096020 con 0x555bdea24d40 2024-03-23T11:24:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.978+0000 7fc274c00640 1 -- 172.21.15.62:0/2857383699 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc28004f020 con 0x555bdea24d40 2024-03-23T11:24:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.978+0000 7fc291224740 10 client.6978 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:24:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.978+0000 7fc291224740 10 client.6978 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:24:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.978+0000 7fc291224740 20 client.6978 populate_metadata read hostname 'smithi062' 2024-03-23T11:24:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.978+0000 7fc291224740 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x555bdea23e20 0x555bdea5b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.978+0000 7fc291224740 4 client.6978 mds_command: new command op to 6618 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:24:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.978+0000 7fc291224740 1 -- 172.21.15.62:0/2857383699 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555bde67ea00 con 0x555bdea23e20 2024-03-23T11:24:55.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.979+0000 7fc287400640 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x555bdea23e20 0x555bdea5b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.980+0000 7fc287400640 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x555bdea23e20 0x555bdea5b800 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59dc42082f76d2bb server_cookie=f61a182b7ded0da6 in_seq=0 out_seq=0 2024-03-23T11:24:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.980+0000 7fc274c00640 10 client.6978 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.981+0000 7fc274c00640 1 -- 172.21.15.62:0/2857383699 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x555bde67ea00 con 0x555bdea23e20 2024-03-23T11:24:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.981+0000 7fc274c00640 10 client.6978 handle_command_reply: tid=0 2024-03-23T11:24:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.981+0000 7fc291224740 1 client.6978 shutdown 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.981+0000 7fc291224740 1 -- 172.21.15.62:0/2857383699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 msgr2=0x555bdea238e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.981+0000 7fc291224740 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea238e0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc2800094a0 tx=0x7fc28007a630 comp rx=0 tx=0).stop 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.981+0000 7fc291224740 1 -- 172.21.15.62:0/2857383699 shutdown_connections 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.981+0000 7fc291224740 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x555bdea23e20 0x555bdea5b800 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.981+0000 7fc291224740 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea238e0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.981+0000 7fc291224740 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea5cdb0 0x555bdea5f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.982+0000 7fc291224740 1 --2- 172.21.15.62:0/2857383699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea12e30 0x555bdea37300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.982+0000 7fc274c00640 0 client.6978 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.982+0000 7fc291224740 1 -- 172.21.15.62:0/2857383699 >> 172.21.15.62:0/2857383699 conn(0x555bdea0f2d0 msgr2=0x555bdea13620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.982+0000 7fc291224740 1 -- 172.21.15.62:0/2857383699 shutdown_connections 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.982+0000 7fc291224740 1 -- 172.21.15.62:0/2857383699 wait complete. 2024-03-23T11:24:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.982+0000 7fc291224740 20 client.6978 trim_cache size 0 max 16384 2024-03-23T11:24:55.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.998+0000 7fc291224740 1 Processor -- start 2024-03-23T11:24:55.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.998+0000 7fc291224740 1 -- start start 2024-03-23T11:24:55.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc291224740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea12e30 0x555bdea13890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc291224740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea24d40 0x555bdea5be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc291224740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea5cdb0 0x555bdea14320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:55.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc291224740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bdea11220 con 0x555bdea5cdb0 2024-03-23T11:24:55.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc291224740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bdea239f0 con 0x555bdea12e30 2024-03-23T11:24:55.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc291224740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555bdea13620 con 0x555bdea24d40 2024-03-23T11:24:55.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc287e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea5cdb0 0x555bdea14320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc287e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea5cdb0 0x555bdea14320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46118/0 (socket says 172.21.15.62:46118) 2024-03-23T11:24:55.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc287e00640 1 -- 172.21.15.62:0/2541871202 learned_addr learned my addr 172.21.15.62:0/2541871202 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc287400640 1 --2- 172.21.15.62:0/2541871202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea12e30 0x555bdea13890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc287e00640 1 -- 172.21.15.62:0/2541871202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea24d40 msgr2=0x555bdea5be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:55.999+0000 7fc286a00640 1 --2- 172.21.15.62:0/2541871202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea24d40 0x555bdea5be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.000+0000 7fc287e00640 1 --2- 172.21.15.62:0/2541871202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea24d40 0x555bdea5be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.000+0000 7fc287e00640 1 -- 172.21.15.62:0/2541871202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea12e30 msgr2=0x555bdea13890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.000+0000 7fc287e00640 1 --2- 172.21.15.62:0/2541871202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea12e30 0x555bdea13890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.000+0000 7fc287e00640 1 -- 172.21.15.62:0/2541871202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bde919850 con 0x555bdea5cdb0 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.000+0000 7fc287400640 1 --2- 172.21.15.62:0/2541871202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea12e30 0x555bdea13890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.000+0000 7fc286a00640 1 --2- 172.21.15.62:0/2541871202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea24d40 0x555bdea5be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.000+0000 7fc287e00640 1 --2- 172.21.15.62:0/2541871202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea5cdb0 0x555bdea14320 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc280077630 tx=0x7fc280041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d481ea89e8a1a03 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.000+0000 7fc27cc00640 1 -- 172.21.15.62:0/2541871202 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28007ea40 con 0x555bdea5cdb0 2024-03-23T11:24:55.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.000+0000 7fc27cc00640 1 -- 172.21.15.62:0/2541871202 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc280096070 con 0x555bdea5cdb0 2024-03-23T11:24:55.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.000+0000 7fc27cc00640 1 -- 172.21.15.62:0/2541871202 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28007a9e0 con 0x555bdea5cdb0 2024-03-23T11:24:55.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.001+0000 7fc291224740 1 -- 172.21.15.62:0/2541871202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea5cdb0 msgr2=0x555bdea14320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:55.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.001+0000 7fc291224740 1 --2- 172.21.15.62:0/2541871202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea5cdb0 0x555bdea14320 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc280077630 tx=0x7fc280041000 comp rx=0 tx=0).stop 2024-03-23T11:24:55.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.001+0000 7fc291224740 1 -- 172.21.15.62:0/2541871202 shutdown_connections 2024-03-23T11:24:55.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.001+0000 7fc291224740 1 --2- 172.21.15.62:0/2541871202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea5cdb0 0x555bdea14320 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.001+0000 7fc291224740 1 --2- 172.21.15.62:0/2541871202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea24d40 0x555bdea5be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.001+0000 7fc291224740 1 --2- 172.21.15.62:0/2541871202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea12e30 0x555bdea13890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:55.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.001+0000 7fc291224740 1 -- 172.21.15.62:0/2541871202 >> 172.21.15.62:0/2541871202 conn(0x555bdea0f2d0 msgr2=0x555bdea0aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:55.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.001+0000 7fc291224740 1 -- 172.21.15.62:0/2541871202 shutdown_connections 2024-03-23T11:24:55.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.001+0000 7fc291224740 1 -- 172.21.15.62:0/2541871202 wait complete. 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.002+0000 7fc291224740 1 -- start start 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.002+0000 7fc291224740 1 Processor -- start 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc291224740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea12e30 0x555bdea0f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc291224740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea0fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc291224740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea5cdb0 0x555bdea5e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc291224740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bdea5f370 con 0x555bdea24d40 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc291224740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bdea73ca0 con 0x555bdea12e30 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc291224740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555bdea50500 con 0x555bdea5cdb0 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc291224740 10 client.0 _ll_register_callbacks cb 0x7fc2700091a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc286a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea0fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc286a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea0fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46130/0 (socket says 172.21.15.62:46130) 2024-03-23T11:24:56.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc286a00640 1 -- 172.21.15.62:0/3032312807 learned_addr learned my addr 172.21.15.62:0/3032312807 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc286a00640 1 -- 172.21.15.62:0/3032312807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea5cdb0 msgr2=0x555bdea5e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc286a00640 1 --2- 172.21.15.62:0/3032312807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea5cdb0 0x555bdea5e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc286a00640 1 -- 172.21.15.62:0/3032312807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea12e30 msgr2=0x555bdea0f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc286a00640 1 --2- 172.21.15.62:0/3032312807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea12e30 0x555bdea0f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.003+0000 7fc286a00640 1 -- 172.21.15.62:0/3032312807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555bde977cf0 con 0x555bdea24d40 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.004+0000 7fc286a00640 1 --2- 172.21.15.62:0/3032312807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea0fcf0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc270000c00 tx=0x7fc270004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30acda4785aa6465 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.004+0000 7fc274c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.004+0000 7fc274c00640 1 -- 172.21.15.62:0/3032312807 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc270006dd0 con 0x555bdea24d40 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.004+0000 7fc274c00640 1 -- 172.21.15.62:0/3032312807 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc27000ec50 con 0x555bdea24d40 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.004+0000 7fc291224740 1 -- 172.21.15.62:0/3032312807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc270004020 con 0x555bdea24d40 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.004+0000 7fc274c00640 1 -- 172.21.15.62:0/3032312807 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc27000ee10 con 0x555bdea24d40 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.004+0000 7fc291224740 10 client.6984 fetch_fsmap learned FSMap version 49 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.004+0000 7fc291224740 1 -- 172.21.15.62:0/3032312807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x555bde93f090 con 0x555bdea24d40 2024-03-23T11:24:56.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.005+0000 7fc274c00640 1 -- 172.21.15.62:0/3032312807 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc270004020 con 0x555bdea24d40 2024-03-23T11:24:56.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.005+0000 7fc274c00640 1 -- 172.21.15.62:0/3032312807 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc270003610 con 0x555bdea24d40 2024-03-23T11:24:56.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.005+0000 7fc291224740 10 client.6984 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:24:56.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.005+0000 7fc291224740 10 client.6984 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:24:56.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.005+0000 7fc291224740 20 client.6984 populate_metadata read hostname 'smithi062' 2024-03-23T11:24:56.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.005+0000 7fc291224740 1 --2- 172.21.15.62:0/3032312807 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x555bdea3bd10 0x555bdea5e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.005+0000 7fc291224740 4 client.6984 mds_command: new command op to 6618 tid=0[{"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}] 2024-03-23T11:24:56.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.005+0000 7fc291224740 1 -- 172.21.15.62:0/3032312807 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}) v1 -- 0x555bde6d2050 con 0x555bdea3bd10 2024-03-23T11:24:56.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.006+0000 7fc287400640 1 --2- 172.21.15.62:0/3032312807 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x555bdea3bd10 0x555bdea5e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.006+0000 7fc287400640 1 --2- 172.21.15.62:0/3032312807 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x555bdea3bd10 0x555bdea5e990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d17aaca6d3d2c4a3 server_cookie=95b2551d871c94e in_seq=0 out_seq=0 2024-03-23T11:24:56.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.006+0000 7fc274c00640 10 client.6984 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:56.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc274c00640 1 -- 172.21.15.62:0/3032312807 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555bde6d2050 con 0x555bdea3bd10 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc274c00640 10 client.6984 handle_command_reply: tid=0 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc291224740 1 client.6984 shutdown 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc291224740 1 -- 172.21.15.62:0/3032312807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 msgr2=0x555bdea0fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc291224740 1 --2- 172.21.15.62:0/3032312807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea0fcf0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc270000c00 tx=0x7fc270004730 comp rx=0 tx=0).stop 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc291224740 1 -- 172.21.15.62:0/3032312807 shutdown_connections 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc291224740 1 --2- 172.21.15.62:0/3032312807 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x555bdea3bd10 0x555bdea5e990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc291224740 1 --2- 172.21.15.62:0/3032312807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555bdea24d40 0x555bdea0fcf0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc291224740 1 --2- 172.21.15.62:0/3032312807 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555bdea5cdb0 0x555bdea5e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc291224740 1 --2- 172.21.15.62:0/3032312807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bdea12e30 0x555bdea0f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc274c00640 0 client.6984 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc291224740 1 -- 172.21.15.62:0/3032312807 >> 172.21.15.62:0/3032312807 conn(0x555bdea0f2d0 msgr2=0x555bdea365f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc291224740 1 -- 172.21.15.62:0/3032312807 shutdown_connections 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.008+0000 7fc291224740 1 -- 172.21.15.62:0/3032312807 wait complete. 2024-03-23T11:24:56.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc291224740 20 client.6984 trim_cache size 0 max 16384 2024-03-23T11:24:56.005 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:24:56.005 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-03-23T11:24:56.005 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "523edc8e-13da-4edc-8016-64ca72858217", 2024-03-23T11:24:56.005 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-03-23T11:24:56.005 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:24:56.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 -- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc26406a7d0 msgr2=0x7fc26406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc26406a7d0 0x7fc26406cc90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc278006fd0 tx=0x7fc278008040 comp rx=0 tx=0).stop 2024-03-23T11:24:56.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 -- 172.21.15.62:0/3365457298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2880711f0 msgr2=0x7fc288158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2880711f0 0x7fc288158270 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fc280036040 tx=0x7fc280004a40 comp rx=0 tx=0).stop 2024-03-23T11:24:56.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 -- 172.21.15.62:0/3365457298 shutdown_connections 2024-03-23T11:24:56.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc26406a7d0 0x7fc26406cc90 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2880711f0 0x7fc288158270 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc28806dd30 0x7fc2880aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 --2- 172.21.15.62:0/3365457298 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc28806d950 0x7fc28814e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 -- 172.21.15.62:0/3365457298 >> 172.21.15.62:0/3365457298 conn(0x7fc2880b0280 msgr2=0x7fc2880b0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:56.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 -- 172.21.15.62:0/3365457298 shutdown_connections 2024-03-23T11:24:56.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.009+0000 7fc28d000640 1 -- 172.21.15.62:0/3365457298 wait complete. 2024-03-23T11:24:56.021 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-23T11:24:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.243+0000 7fd8e1a00640 1 Processor -- start 2024-03-23T11:24:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.243+0000 7fd8e1a00640 1 -- start start 2024-03-23T11:24:56.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.245+0000 7fd8e1a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc1431f0 0x7fd8dc0b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.245+0000 7fd8e1a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd8dc070090 0x7fd8dc0704d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.245+0000 7fd8e1a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc0b20d0 con 0x7fd8dc0cc2a0 2024-03-23T11:24:56.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.245+0000 7fd8e1a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8dc070a10 con 0x7fd8dc070090 2024-03-23T11:24:56.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.245+0000 7fd8e1a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc1435d0 con 0x7fd8dc1431f0 2024-03-23T11:24:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.245+0000 7fd8d3e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd8dc070090 0x7fd8dc0704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.245+0000 7fd8dbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc1431f0 0x7fd8dc0b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.246+0000 7fd8d3e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd8dc070090 0x7fd8dc0704d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:55222/0 (socket says 172.21.15.62:55222) 2024-03-23T11:24:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.246+0000 7fd8d3e00640 1 -- 172.21.15.62:0/3097833001 learned_addr learned my addr 172.21.15.62:0/3097833001 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.246+0000 7fd8dbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc1431f0 0x7fd8dc0b2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:38156/0 (socket says 172.21.15.62:38156) 2024-03-23T11:24:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.246+0000 7fd8d3e00640 1 -- 172.21.15.62:0/3097833001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc1431f0 msgr2=0x7fd8dc0b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.246+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/3097833001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc1431f0 0x7fd8dc0b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.246+0000 7fd8d3e00640 1 -- 172.21.15.62:0/3097833001 >> v1:172.21.15.120:6789/0 conn(0x7fd8dc0cc2a0 legacy=0x7fd8dc0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.246+0000 7fd8d3e00640 1 -- 172.21.15.62:0/3097833001 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc0b14c0 con 0x7fd8dc070090 2024-03-23T11:24:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.246+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/3097833001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc1431f0 0x7fd8dc0b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:24:56.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.248+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/3097833001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8dc070090 0x7fd8dc0704d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009870 tx=0x7fd8c402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66efce41f7df16e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:56.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.248+0000 7fd8db400640 1 -- 172.21.15.62:0/3097833001 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4031640 con 0x7fd8dc070090 2024-03-23T11:24:56.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.248+0000 7fd8db400640 1 -- 172.21.15.62:0/3097833001 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c40317a0 con 0x7fd8dc070090 2024-03-23T11:24:56.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.248+0000 7fd8db400640 1 -- 172.21.15.62:0/3097833001 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4031b30 con 0x7fd8dc070090 2024-03-23T11:24:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.249+0000 7fd8e1a00640 1 -- 172.21.15.62:0/3097833001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8dc070090 msgr2=0x7fd8dc0704d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.249+0000 7fd8e1a00640 1 --2- 172.21.15.62:0/3097833001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8dc070090 0x7fd8dc0704d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009870 tx=0x7fd8c402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.249+0000 7fd8e1a00640 1 -- 172.21.15.62:0/3097833001 shutdown_connections 2024-03-23T11:24:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.249+0000 7fd8e1a00640 1 --2- 172.21.15.62:0/3097833001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8dc070090 0x7fd8dc0704d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.249+0000 7fd8e1a00640 1 --2- 172.21.15.62:0/3097833001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc1431f0 0x7fd8dc0b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.249+0000 7fd8e1a00640 1 -- 172.21.15.62:0/3097833001 >> 172.21.15.62:0/3097833001 conn(0x7fd8dc0b0090 msgr2=0x7fd8dc0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.249+0000 7fd8e1a00640 1 -- 172.21.15.62:0/3097833001 shutdown_connections 2024-03-23T11:24:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.249+0000 7fd8e1a00640 1 -- 172.21.15.62:0/3097833001 wait complete. 2024-03-23T11:24:56.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.251+0000 7fd8e1a00640 1 Processor -- start 2024-03-23T11:24:56.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.251+0000 7fd8e1a00640 1 -- start start 2024-03-23T11:24:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.252+0000 7fd8e1a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc070090 0x7fd8dc0cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.252+0000 7fd8e1a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8dc0cc2a0 0x7fd8dc1efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.252+0000 7fd8e1a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8dc1431f0 0x7fd8dc130430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.252+0000 7fd8e1a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc1f0760 con 0x7fd8dc0cc2a0 2024-03-23T11:24:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.252+0000 7fd8e1a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc1f0a40 con 0x7fd8dc1431f0 2024-03-23T11:24:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.252+0000 7fd8e1a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc1f0d20 con 0x7fd8dc070090 2024-03-23T11:24:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.252+0000 7fd8d3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8dc0cc2a0 0x7fd8dc1efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.252+0000 7fd8d3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8dc0cc2a0 0x7fd8dc1efe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46146/0 (socket says 172.21.15.62:46146) 2024-03-23T11:24:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.252+0000 7fd8d3e00640 1 -- 172.21.15.62:0/2729362539 learned_addr learned my addr 172.21.15.62:0/2729362539 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.253+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc070090 0x7fd8dc0cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.253+0000 7fd8e1000640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8dc1431f0 0x7fd8dc130430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.253+0000 7fd8d3e00640 1 -- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc070090 msgr2=0x7fd8dc0cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.253+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc070090 0x7fd8dc0cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.253+0000 7fd8d3e00640 1 -- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8dc1431f0 msgr2=0x7fd8dc130430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.253+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8dc1431f0 0x7fd8dc130430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.253+0000 7fd8d3e00640 1 -- 172.21.15.62:0/2729362539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc009400 con 0x7fd8dc0cc2a0 2024-03-23T11:24:56.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.253+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc070090 0x7fd8dc0cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:24:56.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.253+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8dc0cc2a0 0x7fd8dc1efe30 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009870 tx=0x7fd8c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:56.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.256+0000 7fd8d8c00640 1 -- 172.21.15.62:0/2729362539 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c403a930 con 0x7fd8dc0cc2a0 2024-03-23T11:24:56.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.256+0000 7fd8e1a00640 1 -- 172.21.15.62:0/2729362539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c40094d0 con 0x7fd8dc0cc2a0 2024-03-23T11:24:56.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.256+0000 7fd8d8c00640 1 -- 172.21.15.62:0/2729362539 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c403aa90 con 0x7fd8dc0cc2a0 2024-03-23T11:24:56.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.256+0000 7fd8d8c00640 1 -- 172.21.15.62:0/2729362539 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c403ae40 con 0x7fd8dc0cc2a0 2024-03-23T11:24:56.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.257+0000 7fd8e1a00640 1 -- 172.21.15.62:0/2729362539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc12ff10 con 0x7fd8dc0cc2a0 2024-03-23T11:24:56.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.259+0000 7fd8d8c00640 1 -- 172.21.15.62:0/2729362539 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd8c402f050 con 0x7fd8dc0cc2a0 2024-03-23T11:24:56.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.260+0000 7fd8d8c00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8b806a7a0 0x7fd8b806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.260+0000 7fd8d8c00640 1 -- 172.21.15.62:0/2729362539 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd8c40aad30 con 0x7fd8dc0cc2a0 2024-03-23T11:24:56.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.261+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8b806a7a0 0x7fd8b806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.262+0000 7fd8e5deb740 1 Processor -- start 2024-03-23T11:24:56.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.262+0000 7fd8e5deb740 1 -- start start 2024-03-23T11:24:56.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.262+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8b806a7a0 0x7fd8b806cc60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009bb0 tx=0x7fd8cc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:56.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806e8e50 0x55f5806e9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5806eccf0 0x55f5806c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5807084d0 0x55f580708950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f580680cb0 con 0x55f5806e8e50 2024-03-23T11:24:56.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f58062f500 con 0x55f5806eccf0 2024-03-23T11:24:56.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f58062cc90 con 0x55f5807084d0 2024-03-23T11:24:56.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806e8e50 0x55f5806e9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806e8e50 0x55f5806e9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46148/0 (socket says 172.21.15.62:46148) 2024-03-23T11:24:56.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8dbe00640 1 -- 172.21.15.62:0/3162162484 learned_addr learned my addr 172.21.15.62:0/3162162484 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:56.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/3162162484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5806eccf0 0x55f5806c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8dbe00640 1 -- 172.21.15.62:0/3162162484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5807084d0 msgr2=0x55f580708950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:56.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/3162162484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5807084d0 0x55f580708950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8dbe00640 1 -- 172.21.15.62:0/3162162484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5806eccf0 msgr2=0x55f5806c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:56.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/3162162484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5806eccf0 0x55f5806c5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.263+0000 7fd8dbe00640 1 -- 172.21.15.62:0/3162162484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5805da360 con 0x55f5806e8e50 2024-03-23T11:24:56.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.264+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/3162162484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806e8e50 0x55f5806e9230 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc006e80 tx=0x7fd8cc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7861844ab3be1a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:56.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.264+0000 7fd8d0c00640 1 -- 172.21.15.62:0/3162162484 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc021440 con 0x55f5806e8e50 2024-03-23T11:24:56.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.264+0000 7fd8d0c00640 1 -- 172.21.15.62:0/3162162484 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8cc01a070 con 0x55f5806e8e50 2024-03-23T11:24:56.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.265+0000 7fd8d0c00640 1 -- 172.21.15.62:0/3162162484 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc0217a0 con 0x55f5806e8e50 2024-03-23T11:24:56.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.265+0000 7fd8e5deb740 1 -- 172.21.15.62:0/3162162484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806e8e50 msgr2=0x55f5806e9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.265+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/3162162484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806e8e50 0x55f5806e9230 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc006e80 tx=0x7fd8cc013000 comp rx=0 tx=0).stop 2024-03-23T11:24:56.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.265+0000 7fd8e5deb740 1 -- 172.21.15.62:0/3162162484 shutdown_connections 2024-03-23T11:24:56.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.265+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/3162162484 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5807084d0 0x55f580708950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.265+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/3162162484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5806eccf0 0x55f5806c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.265+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/3162162484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806e8e50 0x55f5806e9230 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.265+0000 7fd8e5deb740 1 -- 172.21.15.62:0/3162162484 >> 172.21.15.62:0/3162162484 conn(0x55f58068cb40 msgr2=0x55f58070a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:56.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.265+0000 7fd8e5deb740 1 -- 172.21.15.62:0/3162162484 shutdown_connections 2024-03-23T11:24:56.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.265+0000 7fd8e5deb740 1 -- 172.21.15.62:0/3162162484 wait complete. 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.266+0000 7fd8e5deb740 1 -- start start 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.266+0000 7fd8e5deb740 1 Processor -- start 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.267+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 0x55f580709fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.267+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5806eccf0 0x55f58070b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.267+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5807084d0 0x55f5806e9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.267+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f580628580 con 0x55f5807084d0 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.267+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5805c9a50 con 0x55f5806eccf0 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.267+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f5805c96f0 con 0x55f5806e8e50 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.267+0000 7fd8e5deb740 10 client.0 _ll_register_callbacks cb 0x55f580618ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.268+0000 7fd8e1000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5807084d0 0x55f5806e9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.268+0000 7fd8e1000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5807084d0 0x55f5806e9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46152/0 (socket says 172.21.15.62:46152) 2024-03-23T11:24:56.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.268+0000 7fd8e1000640 1 -- 172.21.15.62:0/2098933544 learned_addr learned my addr 172.21.15.62:0/2098933544 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:56.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.268+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5806eccf0 0x55f58070b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.268+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 0x55f580709fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.268+0000 7fd8e1000640 1 -- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 msgr2=0x55f580709fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:56.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.268+0000 7fd8e1000640 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 0x55f580709fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.268+0000 7fd8e1000640 1 -- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5806eccf0 msgr2=0x55f58070b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:56.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.268+0000 7fd8e1000640 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5806eccf0 0x55f58070b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.268+0000 7fd8e1000640 1 -- 172.21.15.62:0/2098933544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8cc00e050 con 0x55f5807084d0 2024-03-23T11:24:56.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.268+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5806eccf0 0x55f58070b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:24:56.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.269+0000 7fd8e1000640 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5807084d0 0x55f5806e9230 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400d430 tx=0x7fd8d400d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27edfc2bd2d5d354 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:56.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.269+0000 7fd8c9600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:24:56.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.269+0000 7fd8c9600640 1 -- 172.21.15.62:0/2098933544 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d400b070 con 0x55f5807084d0 2024-03-23T11:24:56.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.269+0000 7fd8c9600640 1 -- 172.21.15.62:0/2098933544 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd8d4003d80 con 0x55f5807084d0 2024-03-23T11:24:56.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.269+0000 7fd8e5deb740 1 -- 172.21.15.62:0/2098933544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f580652fe0 con 0x55f5807084d0 2024-03-23T11:24:56.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.269+0000 7fd8c9600640 1 -- 172.21.15.62:0/2098933544 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8d4001720 con 0x55f5807084d0 2024-03-23T11:24:56.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.269+0000 7fd8e5deb740 10 client.6993 fetch_fsmap learned FSMap version 49 2024-03-23T11:24:56.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.269+0000 7fd8e5deb740 1 -- 172.21.15.62:0/2098933544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x55f580626c00 con 0x55f5807084d0 2024-03-23T11:24:56.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.269+0000 7fd8c9600640 1 -- 172.21.15.62:0/2098933544 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f580652fe0 con 0x55f5807084d0 2024-03-23T11:24:56.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.270+0000 7fd8c9600640 1 -- 172.21.15.62:0/2098933544 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd8d40050d0 con 0x55f5807084d0 2024-03-23T11:24:56.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.270+0000 7fd8e5deb740 10 client.6993 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:24:56.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.270+0000 7fd8e5deb740 10 client.6993 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:24:56.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.270+0000 7fd8e5deb740 20 client.6993 populate_metadata read hostname 'smithi062' 2024-03-23T11:24:56.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.270+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55f580703f50 0x55f5806e6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.270+0000 7fd8e5deb740 4 client.6993 mds_command: new command op to 6618 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:24:56.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.270+0000 7fd8e5deb740 1 -- 172.21.15.62:0/2098933544 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f58061b240 con 0x55f580703f50 2024-03-23T11:24:56.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.270+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55f580703f50 0x55f5806e6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.271+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55f580703f50 0x55f5806e6ed0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdee7bb0703ee920 server_cookie=2005837013daf285 in_seq=0 out_seq=0 2024-03-23T11:24:56.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.271+0000 7fd8c9600640 10 client.6993 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:56.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8c9600640 1 -- 172.21.15.62:0/2098933544 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55f58061b240 con 0x55f580703f50 2024-03-23T11:24:56.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8c9600640 10 client.6993 handle_command_reply: tid=0 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8e5deb740 1 client.6993 shutdown 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8e5deb740 1 -- 172.21.15.62:0/2098933544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5807084d0 msgr2=0x55f5806e9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5807084d0 0x55f5806e9230 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400d430 tx=0x7fd8d400d8f0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8e5deb740 1 -- 172.21.15.62:0/2098933544 shutdown_connections 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55f580703f50 0x55f5806e6ed0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5807084d0 0x55f5806e9230 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5806eccf0 0x55f58070b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/2098933544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 0x55f580709fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8c9600640 0 client.6993 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8e5deb740 1 -- 172.21.15.62:0/2098933544 >> 172.21.15.62:0/2098933544 conn(0x55f58068cb40 msgr2=0x55f5806beca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.273+0000 7fd8e5deb740 1 -- 172.21.15.62:0/2098933544 shutdown_connections 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.274+0000 7fd8e5deb740 1 -- 172.21.15.62:0/2098933544 wait complete. 2024-03-23T11:24:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.274+0000 7fd8e5deb740 20 client.6993 trim_cache size 0 max 16384 2024-03-23T11:24:56.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.289+0000 7fd8e5deb740 1 Processor -- start 2024-03-23T11:24:56.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.290+0000 7fd8e5deb740 1 -- start start 2024-03-23T11:24:56.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.290+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 0x55f58070b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.290+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f58070b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.290+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5807084d0 0x55f5806f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.290+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f580325f90 con 0x55f5806eccf0 2024-03-23T11:24:56.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.291+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5805e4570 con 0x55f5807084d0 2024-03-23T11:24:56.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.291+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f58070d260 con 0x55f5806e8e50 2024-03-23T11:24:56.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.291+0000 7fd8d3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f58070b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.291+0000 7fd8d3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f58070b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46154/0 (socket says 172.21.15.62:46154) 2024-03-23T11:24:56.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.291+0000 7fd8d3e00640 1 -- 172.21.15.62:0/1005933767 learned_addr learned my addr 172.21.15.62:0/1005933767 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:56.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.291+0000 7fd8d3e00640 1 -- 172.21.15.62:0/1005933767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 msgr2=0x55f58070b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:56.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.291+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/1005933767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 0x55f58070b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.291+0000 7fd8d3e00640 1 -- 172.21.15.62:0/1005933767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5807084d0 msgr2=0x55f5806f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.291+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/1005933767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5807084d0 0x55f5806f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.291+0000 7fd8d3e00640 1 -- 172.21.15.62:0/1005933767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5805c8c60 con 0x55f5806eccf0 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.292+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/1005933767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f58070b6b0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4076200 tx=0x7fd8c4079ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb81d8d5859bda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.292+0000 7fd8d1600640 1 -- 172.21.15.62:0/1005933767 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c407db30 con 0x55f5806eccf0 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.292+0000 7fd8d1600640 1 -- 172.21.15.62:0/1005933767 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c40820f0 con 0x55f5806eccf0 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.292+0000 7fd8d1600640 1 -- 172.21.15.62:0/1005933767 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4076410 con 0x55f5806eccf0 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.292+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1005933767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 msgr2=0x55f58070b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.292+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/1005933767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f58070b6b0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4076200 tx=0x7fd8c4079ad0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.292+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1005933767 shutdown_connections 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.292+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/1005933767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f58070b6b0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.292+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/1005933767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5807084d0 0x55f5806f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.292+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/1005933767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 0x55f58070b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.292+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1005933767 >> 172.21.15.62:0/1005933767 conn(0x55f58068cb40 msgr2=0x55f5807212b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:56.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.293+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1005933767 shutdown_connections 2024-03-23T11:24:56.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.293+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1005933767 wait complete. 2024-03-23T11:24:56.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.293+0000 7fd8e5deb740 1 -- start start 2024-03-23T11:24:56.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.294+0000 7fd8e5deb740 1 Processor -- start 2024-03-23T11:24:56.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 0x55f5807740c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f5806e7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8e5deb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5807084d0 0x55f5806e7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5806c3810 con 0x55f5806eccf0 2024-03-23T11:24:56.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f580709360 con 0x55f5807084d0 2024-03-23T11:24:56.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8e5deb740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f580727d80 con 0x55f5806e8e50 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8e5deb740 10 client.0 _ll_register_callbacks cb 0x55f5806f0970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8d3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f5806e7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8d3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f5806e7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46164/0 (socket says 172.21.15.62:46164) 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8d3e00640 1 -- 172.21.15.62:0/1164846616 learned_addr learned my addr 172.21.15.62:0/1164846616 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8d3e00640 1 -- 172.21.15.62:0/1164846616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 msgr2=0x55f5807740c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/1164846616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 0x55f5807740c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8d3e00640 1 -- 172.21.15.62:0/1164846616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5807084d0 msgr2=0x55f5806e7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/1164846616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5807084d0 0x55f5806e7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.295+0000 7fd8d3e00640 1 -- 172.21.15.62:0/1164846616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc12ff10 con 0x55f5806eccf0 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8d3e00640 1 --2- 172.21.15.62:0/1164846616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f5806e7a10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd8c40797d0 tx=0x7fd8c404a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9622c82bb27ec644 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8c9600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8c9600640 1 -- 172.21.15.62:0/1164846616 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c407a160 con 0x55f5806eccf0 2024-03-23T11:24:56.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8c9600640 1 -- 172.21.15.62:0/1164846616 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd8c407db30 con 0x55f5806eccf0 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1164846616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8cc01a070 con 0x55f5806eccf0 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8c9600640 1 -- 172.21.15.62:0/1164846616 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8c40791f0 con 0x55f5806eccf0 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8e5deb740 10 client.6999 fetch_fsmap learned FSMap version 49 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1164846616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x55f5805ee170 con 0x55f5806eccf0 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8c9600640 1 -- 172.21.15.62:0/1164846616 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8cc01a070 con 0x55f5806eccf0 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8c9600640 1 -- 172.21.15.62:0/1164846616 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd8c4081c60 con 0x55f5806eccf0 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8e5deb740 10 client.6999 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8e5deb740 10 client.6999 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8e5deb740 20 client.6999 populate_metadata read hostname 'smithi062' 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/1164846616 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55f5806e5960 0x55f5806e5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8e5deb740 4 client.6999 mds_command: new command op to 6618 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:24:56.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.296+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1164846616 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f58031c4e0 con 0x55f5806e5960 2024-03-23T11:24:56.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.297+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/1164846616 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55f5806e5960 0x55f5806e5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:24:56.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.298+0000 7fd8dbe00640 1 --2- 172.21.15.62:0/1164846616 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55f5806e5960 0x55f5806e5d70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=293e63ead92a8ee3 server_cookie=86f56c7919759bf4 in_seq=0 out_seq=0 2024-03-23T11:24:56.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.298+0000 7fd8c9600640 10 client.6999 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:56.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.299+0000 7fd8c9600640 1 -- 172.21.15.62:0/1164846616 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f58031c4e0 con 0x55f5806e5960 2024-03-23T11:24:56.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.299+0000 7fd8c9600640 10 client.6999 handle_command_reply: tid=0 2024-03-23T11:24:56.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.299+0000 7fd8e5deb740 1 client.6999 shutdown 2024-03-23T11:24:56.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.299+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1164846616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 msgr2=0x55f5806e7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.299+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/1164846616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f5806e7a10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd8c40797d0 tx=0x7fd8c404a000 comp rx=0 tx=0).stop 2024-03-23T11:24:56.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.300+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1164846616 shutdown_connections 2024-03-23T11:24:56.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.300+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/1164846616 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55f5806e5960 0x55f5806e5d70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.300+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/1164846616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5806eccf0 0x55f5806e7a10 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.300+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/1164846616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f5807084d0 0x55f5806e7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.300+0000 7fd8e5deb740 1 --2- 172.21.15.62:0/1164846616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f5806e8e50 0x55f5807740c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.300+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1164846616 >> 172.21.15.62:0/1164846616 conn(0x55f58068cb40 msgr2=0x55f5806c5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:56.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.300+0000 7fd8c9600640 0 client.6999 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:24:56.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.300+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1164846616 shutdown_connections 2024-03-23T11:24:56.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.300+0000 7fd8e5deb740 1 -- 172.21.15.62:0/1164846616 wait complete. 2024-03-23T11:24:56.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.300+0000 7fd8e5deb740 20 client.6999 trim_cache size 0 max 16384 2024-03-23T11:24:56.297 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:24:56.297 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T11:24:56.297 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-03-23T11:24:56.298 INFO:teuthology.orchestra.run.smithi062.stdout: "523edc8e-13da-4edc-8016-64ca72858217": { 2024-03-23T11:24:56.298 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/subdir", 2024-03-23T11:24:56.298 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "523edc8e-13da-4edc-8016-64ca72858217", 2024-03-23T11:24:56.298 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive" 2024-03-23T11:24:56.298 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:56.298 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:24:56.298 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:24:56.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 -- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8b806a7a0 msgr2=0x7fd8b806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8b806a7a0 0x7fd8b806cc60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009bb0 tx=0x7fd8cc008040 comp rx=0 tx=0).stop 2024-03-23T11:24:56.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 -- 172.21.15.62:0/2729362539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8dc0cc2a0 msgr2=0x7fd8dc1efe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:24:56.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8dc0cc2a0 0x7fd8dc1efe30 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009870 tx=0x7fd8c40046c0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 -- 172.21.15.62:0/2729362539 shutdown_connections 2024-03-23T11:24:56.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd8b806a7a0 0x7fd8b806cc60 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8dc0cc2a0 0x7fd8dc1efe30 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8dc1431f0 0x7fd8dc130430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 --2- 172.21.15.62:0/2729362539 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8dc070090 0x7fd8dc0cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:24:56.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 -- 172.21.15.62:0/2729362539 >> 172.21.15.62:0/2729362539 conn(0x7fd8dc0b0090 msgr2=0x7fd8dc13d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:24:56.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 -- 172.21.15.62:0/2729362539 shutdown_connections 2024-03-23T11:24:56.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:24:56.301+0000 7fd8e1a00640 1 -- 172.21.15.62:0/2729362539 wait complete. 2024-03-23T11:24:56.312 INFO:tasks.cephfs.filesystem:scrub status for tag:523edc8e-13da-4edc-8016-64ca72858217 - {'path': '/subdir', 'tag': '523edc8e-13da-4edc-8016-64ca72858217', 'options': 'recursive'} 2024-03-23T11:25:16.122 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:25:16.130 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:25:26.313 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-23T11:25:26.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.550+0000 7f9d0d400640 1 Processor -- start 2024-03-23T11:25:26.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.550+0000 7f9d0d400640 1 -- start start 2024-03-23T11:25:26.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.552+0000 7f9d0d400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9d081431f0 0x7f9d080b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.552+0000 7f9d0d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d08070090 0x7f9d080704d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.552+0000 7f9d0d400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d080b20d0 con 0x7f9d080cc2a0 2024-03-23T11:25:26.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.552+0000 7f9d0d400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d08070a10 con 0x7f9d081431f0 2024-03-23T11:25:26.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.552+0000 7f9d0d400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d081435d0 con 0x7f9d08070090 2024-03-23T11:25:26.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.552+0000 7f9d0ca00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9d080cc2a0 0x7f9d080cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40714/0 (socket says 172.21.15.62:40714) 2024-03-23T11:25:26.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.552+0000 7f9d0ca00640 1 -- 172.21.15.62:0/858231739 learned_addr learned my addr 172.21.15.62:0/858231739 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:25:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.552+0000 7f9cffe00640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d08070090 0x7f9d080704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.552+0000 7f9d07e00640 1 --2- 172.21.15.62:0/858231739 >> v2:172.21.15.120:3300/0 conn(0x7f9d081431f0 0x7f9d080b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756635793 0 0) 0x7f9d080b20d0 con 0x7f9d080cc2a0 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cec003520 con 0x7f9d080cc2a0 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2241237465 0 0) 0x7f9cec003520 con 0x7f9d080cc2a0 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d08070090 msgr2=0x7f9d080704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d08070090 0x7f9d080704d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 >> v2:172.21.15.120:3300/0 conn(0x7f9d081431f0 msgr2=0x7f9d080b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 --2- 172.21.15.62:0/858231739 >> v2:172.21.15.120:3300/0 conn(0x7f9d081431f0 0x7f9d080b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d080b14c0 con 0x7f9d080cc2a0 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9d000019f0 con 0x7f9d080cc2a0 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 >> v1:172.21.15.62:6789/0 conn(0x7f9d080cc2a0 legacy=0x7f9d080cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d08070090 0x7f9cec003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cec003ef0 0x7f9cec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec004d00 0x7f9cec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d08070a10 con 0x7f9cec004d00 2024-03-23T11:25:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d081435d0 con 0x7f9cec003ef0 2024-03-23T11:25:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.553+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec00b9d0 con 0x7f9d08070090 2024-03-23T11:25:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.554+0000 7f9d0ca00640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec004d00 0x7f9cec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.554+0000 7f9d07e00640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d08070090 0x7f9cec003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.554+0000 7f9d0ca00640 1 -- 172.21.15.62:0/858231739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d08070090 msgr2=0x7f9cec003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.554+0000 7f9cffe00640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cec003ef0 0x7f9cec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.554+0000 7f9d0ca00640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d08070090 0x7f9cec003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.554+0000 7f9d0ca00640 1 -- 172.21.15.62:0/858231739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cec003ef0 msgr2=0x7f9cec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.554+0000 7f9d0ca00640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cec003ef0 0x7f9cec0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.554+0000 7f9d0ca00640 1 -- 172.21.15.62:0/858231739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec003520 con 0x7f9cec004d00 2024-03-23T11:25:26.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.555+0000 7f9d0ca00640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec004d00 0x7f9cec005180 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000a860 tx=0x7f9cec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d691c714a1abb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:26.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.555+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00034630 con 0x7f9cec004d00 2024-03-23T11:25:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.555+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0002f070 con 0x7f9cec004d00 2024-03-23T11:25:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.555+0000 7f9d07400640 1 -- 172.21.15.62:0/858231739 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00034a30 con 0x7f9cec004d00 2024-03-23T11:25:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.556+0000 7f9d0d400640 1 -- 172.21.15.62:0/858231739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec004d00 msgr2=0x7f9cec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.556+0000 7f9d0d400640 1 --2- 172.21.15.62:0/858231739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec004d00 0x7f9cec005180 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000a860 tx=0x7f9cec0057f0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.556+0000 7f9d0ca00640 1 -- 172.21.15.62:0/858231739 reap_dead start 2024-03-23T11:25:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.556+0000 7f9d0d400640 1 -- 172.21.15.62:0/858231739 shutdown_connections 2024-03-23T11:25:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.556+0000 7f9d0d400640 1 -- 172.21.15.62:0/858231739 >> 172.21.15.62:0/858231739 conn(0x7f9d080b0090 msgr2=0x7f9d080b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.556+0000 7f9d0d400640 1 -- 172.21.15.62:0/858231739 shutdown_connections 2024-03-23T11:25:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.556+0000 7f9d0d400640 1 -- 172.21.15.62:0/858231739 wait complete. 2024-03-23T11:25:26.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.558+0000 7f9d0d400640 1 Processor -- start 2024-03-23T11:25:26.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.558+0000 7f9d0d400640 1 -- start start 2024-03-23T11:25:26.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.559+0000 7f9d0d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec004d00 0x7f9d08141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.559+0000 7f9d0d400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d080aca80 0x7f9d080ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.559+0000 7f9d0d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d080acf00 0x7f9d080abd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.559+0000 7f9d0d400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0812f350 con 0x7f9d080acf00 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.559+0000 7f9d0d400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d081422e0 con 0x7f9d080aca80 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.559+0000 7f9d0d400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0812ea20 con 0x7f9cec004d00 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.559+0000 7f9d0ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d080acf00 0x7f9d080abd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.559+0000 7f9d0ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d080acf00 0x7f9d080abd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48666/0 (socket says 172.21.15.62:48666) 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.559+0000 7f9d0ca00640 1 -- 172.21.15.62:0/234173305 learned_addr learned my addr 172.21.15.62:0/234173305 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.560+0000 7f9d07e00640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec004d00 0x7f9d08141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.560+0000 7f9cffe00640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d080aca80 0x7f9d080ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.560+0000 7f9d0ca00640 1 -- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec004d00 msgr2=0x7f9d08141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.560+0000 7f9d0ca00640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec004d00 0x7f9d08141bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.560+0000 7f9d0ca00640 1 -- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d080aca80 msgr2=0x7f9d080ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.560+0000 7f9d0ca00640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d080aca80 0x7f9d080ab840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.560+0000 7f9d0ca00640 1 -- 172.21.15.62:0/234173305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0812ed00 con 0x7f9d080acf00 2024-03-23T11:25:26.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.560+0000 7f9d0ca00640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d080acf00 0x7f9d080abd80 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f9d00002230 tx=0x7f9d080b2040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:26.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.560+0000 7f9d04c00640 1 -- 172.21.15.62:0/234173305 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00043070 con 0x7f9d080acf00 2024-03-23T11:25:26.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.560+0000 7f9d04c00640 1 -- 172.21.15.62:0/234173305 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d00001fc0 con 0x7f9d080acf00 2024-03-23T11:25:26.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.560+0000 7f9d04c00640 1 -- 172.21.15.62:0/234173305 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00034840 con 0x7f9d080acf00 2024-03-23T11:25:26.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.561+0000 7f9d0d400640 1 -- 172.21.15.62:0/234173305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0812efe0 con 0x7f9d080acf00 2024-03-23T11:25:26.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.561+0000 7f9d0d400640 1 -- 172.21.15.62:0/234173305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d080ac2c0 con 0x7f9d080acf00 2024-03-23T11:25:26.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.563+0000 7f9d04c00640 1 -- 172.21.15.62:0/234173305 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9d00031030 con 0x7f9d080acf00 2024-03-23T11:25:26.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.564+0000 7f9d04c00640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ce406a7a0 0x7f9ce406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.564+0000 7f9d04c00640 1 -- 172.21.15.62:0/234173305 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9d000aa740 con 0x7f9d080acf00 2024-03-23T11:25:26.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.564+0000 7f9d11651740 1 Processor -- start 2024-03-23T11:25:26.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.564+0000 7f9d07e00640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ce406a7a0 0x7f9ce406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.565+0000 7f9d11651740 1 -- start start 2024-03-23T11:25:26.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.565+0000 7f9d11651740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e150 0x56552ce02b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.565+0000 7f9d11651740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce454d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.565+0000 7f9d11651740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce45a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.565+0000 7f9d11651740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552cdbdcb0 con 0x56552ce2a0f0 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.565+0000 7f9d11651740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552cd6c500 con 0x56552ce3e150 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.565+0000 7f9d11651740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56552cd69c90 con 0x56552ce3e530 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9d07e00640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ce406a7a0 0x7f9ce406cc60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009a90 tx=0x7f9cf8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9cffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9cffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce454d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48676/0 (socket says 172.21.15.62:48676) 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9cffe00640 1 -- 172.21.15.62:0/686607296 learned_addr learned my addr 172.21.15.62:0/686607296 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9d0ca00640 1 --2- 172.21.15.62:0/686607296 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce45a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9d07e00640 1 --2- 172.21.15.62:0/686607296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e150 0x56552ce02b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9cffe00640 1 -- 172.21.15.62:0/686607296 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 msgr2=0x56552ce45a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9cffe00640 1 --2- 172.21.15.62:0/686607296 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce45a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9cffe00640 1 -- 172.21.15.62:0/686607296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e150 msgr2=0x56552ce02b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9cffe00640 1 --2- 172.21.15.62:0/686607296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e150 0x56552ce02b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9cffe00640 1 -- 172.21.15.62:0/686607296 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56552cd17360 con 0x56552ce2a0f0 2024-03-23T11:25:26.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.566+0000 7f9d07e00640 1 --2- 172.21.15.62:0/686607296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e150 0x56552ce02b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:26.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.567+0000 7f9cffe00640 1 --2- 172.21.15.62:0/686607296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce454d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009ae0 tx=0x7f9cf0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd80092c6bb7cfa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:26.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.567+0000 7f9cfcc00640 1 -- 172.21.15.62:0/686607296 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf00129d0 con 0x56552ce2a0f0 2024-03-23T11:25:26.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.567+0000 7f9cfcc00640 1 -- 172.21.15.62:0/686607296 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf0012b30 con 0x56552ce2a0f0 2024-03-23T11:25:26.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.567+0000 7f9cfcc00640 1 -- 172.21.15.62:0/686607296 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0012ee0 con 0x56552ce2a0f0 2024-03-23T11:25:26.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.567+0000 7f9d11651740 1 -- 172.21.15.62:0/686607296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 msgr2=0x56552ce454d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.567+0000 7f9d11651740 1 --2- 172.21.15.62:0/686607296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce454d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009ae0 tx=0x7f9cf0009190 comp rx=0 tx=0).stop 2024-03-23T11:25:26.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.568+0000 7f9d11651740 1 -- 172.21.15.62:0/686607296 shutdown_connections 2024-03-23T11:25:26.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.568+0000 7f9d11651740 1 --2- 172.21.15.62:0/686607296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce454d0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.568+0000 7f9d11651740 1 --2- 172.21.15.62:0/686607296 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce45a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.568+0000 7f9d11651740 1 --2- 172.21.15.62:0/686607296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e150 0x56552ce02b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.568+0000 7f9d11651740 1 -- 172.21.15.62:0/686607296 >> 172.21.15.62:0/686607296 conn(0x56552cdc9b40 msgr2=0x56552ce47900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:26.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.568+0000 7f9d11651740 1 -- 172.21.15.62:0/686607296 shutdown_connections 2024-03-23T11:25:26.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.568+0000 7f9d11651740 1 -- 172.21.15.62:0/686607296 wait complete. 2024-03-23T11:25:26.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.568+0000 7f9d11651740 1 -- start start 2024-03-23T11:25:26.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.569+0000 7f9d11651740 1 Processor -- start 2024-03-23T11:25:26.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9d11651740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce3cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9d11651740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e150 0x56552ce488f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9d11651740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e530 0x56552ce3b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9d11651740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552cd65580 con 0x56552ce2a0f0 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9d11651740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552cd06a50 con 0x56552ce3e530 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9d11651740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56552cd066f0 con 0x56552ce3e150 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9d11651740 10 client.0 _ll_register_callbacks cb 0x56552cd55ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9d07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce3cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9d07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce3cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48680/0 (socket says 172.21.15.62:48680) 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9d07e00640 1 -- 172.21.15.62:0/2775990622 learned_addr learned my addr 172.21.15.62:0/2775990622 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9d0ca00640 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e530 0x56552ce3b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.570+0000 7f9cffe00640 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e150 0x56552ce488f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.571+0000 7f9d07e00640 1 -- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e150 msgr2=0x56552ce488f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.571+0000 7f9d07e00640 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e150 0x56552ce488f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.571+0000 7f9d07e00640 1 -- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e530 msgr2=0x56552ce3b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.571+0000 7f9d07e00640 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e530 0x56552ce3b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.571+0000 7f9d07e00640 1 -- 172.21.15.62:0/2775990622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf0009710 con 0x56552ce2a0f0 2024-03-23T11:25:26.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.571+0000 7f9d07e00640 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce3cfa0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8013f70 tx=0x7f9cf8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9dc7211a81b4d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.571+0000 7f9cf5600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.571+0000 7f9cf5600640 1 -- 172.21.15.62:0/2775990622 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8004220 con 0x56552ce2a0f0 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.571+0000 7f9cf5600640 1 -- 172.21.15.62:0/2775990622 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9cf8004380 con 0x56552ce2a0f0 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.571+0000 7f9d11651740 1 -- 172.21.15.62:0/2775990622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56552cd8ffe0 con 0x56552ce2a0f0 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.572+0000 7f9cf5600640 1 -- 172.21.15.62:0/2775990622 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cf8023620 con 0x56552ce2a0f0 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.572+0000 7f9d11651740 10 client.7011 fetch_fsmap learned FSMap version 49 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.572+0000 7f9d11651740 1 -- 172.21.15.62:0/2775990622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x56552cd63c00 con 0x56552ce2a0f0 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.572+0000 7f9cf5600640 1 -- 172.21.15.62:0/2775990622 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56552cd8ffe0 con 0x56552ce2a0f0 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.572+0000 7f9cf5600640 1 -- 172.21.15.62:0/2775990622 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9cf8023780 con 0x56552ce2a0f0 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.572+0000 7f9d11651740 10 client.7011 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.572+0000 7f9d11651740 10 client.7011 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.572+0000 7f9d11651740 20 client.7011 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:26.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.572+0000 7f9d11651740 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x56552ce46580 0x56552ce46960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.572+0000 7f9d11651740 4 client.7011 mds_command: new command op to 6618 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:25:26.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.572+0000 7f9d11651740 1 -- 172.21.15.62:0/2775990622 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56552cd58240 con 0x56552ce46580 2024-03-23T11:25:26.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.573+0000 7f9cffe00640 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x56552ce46580 0x56552ce46960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.574+0000 7f9cffe00640 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x56552ce46580 0x56552ce46960 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b55e7663c3e62c23 server_cookie=ede5d5fee0f51edf in_seq=0 out_seq=0 2024-03-23T11:25:26.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.574+0000 7f9cf5600640 10 client.7011 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:26.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.575+0000 7f9cf5600640 1 -- 172.21.15.62:0/2775990622 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56552cd58240 con 0x56552ce46580 2024-03-23T11:25:26.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9cf5600640 10 client.7011 handle_command_reply: tid=0 2024-03-23T11:25:26.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 1 client.7011 shutdown 2024-03-23T11:25:26.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 1 -- 172.21.15.62:0/2775990622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 msgr2=0x56552ce3cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce3cfa0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8013f70 tx=0x7f9cf8004730 comp rx=0 tx=0).stop 2024-03-23T11:25:26.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 1 -- 172.21.15.62:0/2775990622 shutdown_connections 2024-03-23T11:25:26.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x56552ce46580 0x56552ce46960 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e530 0x56552ce3b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e150 0x56552ce488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 1 --2- 172.21.15.62:0/2775990622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce3cfa0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9cf5600640 0 client.7011 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:26.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 1 -- 172.21.15.62:0/2775990622 >> 172.21.15.62:0/2775990622 conn(0x56552cdc9b40 msgr2=0x56552cdfbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:26.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 1 -- 172.21.15.62:0/2775990622 shutdown_connections 2024-03-23T11:25:26.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 1 -- 172.21.15.62:0/2775990622 wait complete. 2024-03-23T11:25:26.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.576+0000 7f9d11651740 20 client.7011 trim_cache size 0 max 16384 2024-03-23T11:25:26.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.592+0000 7f9d11651740 1 Processor -- start 2024-03-23T11:25:26.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.592+0000 7f9d11651740 1 -- start start 2024-03-23T11:25:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.593+0000 7f9d11651740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce2a0f0 0x56552ce4b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.593+0000 7f9d11651740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce3e150 0x56552ce4b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.593+0000 7f9d11651740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce2c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.593+0000 7f9d11651740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552ca62f90 con 0x56552ce3e150 2024-03-23T11:25:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.593+0000 7f9d11651740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552ce3ca60 con 0x56552ce2a0f0 2024-03-23T11:25:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.593+0000 7f9d11651740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56552ce3c810 con 0x56552ce3e530 2024-03-23T11:25:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.593+0000 7f9cffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce3e150 0x56552ce4b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.593+0000 7f9cffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce3e150 0x56552ce4b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48688/0 (socket says 172.21.15.62:48688) 2024-03-23T11:25:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.593+0000 7f9cffe00640 1 -- 172.21.15.62:0/27103106 learned_addr learned my addr 172.21.15.62:0/27103106 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.594+0000 7f9d0ca00640 1 --2- 172.21.15.62:0/27103106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce2c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.594+0000 7f9d07e00640 1 --2- 172.21.15.62:0/27103106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce2a0f0 0x56552ce4b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.594+0000 7f9cffe00640 1 -- 172.21.15.62:0/27103106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 msgr2=0x56552ce2c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.594+0000 7f9cffe00640 1 --2- 172.21.15.62:0/27103106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce2c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.594+0000 7f9cffe00640 1 -- 172.21.15.62:0/27103106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce2a0f0 msgr2=0x56552ce4b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.594+0000 7f9cffe00640 1 --2- 172.21.15.62:0/27103106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce2a0f0 0x56552ce4b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.594+0000 7f9cffe00640 1 -- 172.21.15.62:0/27103106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56552cd05c60 con 0x56552ce3e150 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.594+0000 7f9cffe00640 1 --2- 172.21.15.62:0/27103106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce3e150 0x56552ce4b0b0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0008010 tx=0x7f9cf0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3697678b41e9d24 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.594+0000 7f9cfd600640 1 -- 172.21.15.62:0/27103106 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0023030 con 0x56552ce3e150 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.594+0000 7f9cfd600640 1 -- 172.21.15.62:0/27103106 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf0024040 con 0x56552ce3e150 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.594+0000 7f9cfd600640 1 -- 172.21.15.62:0/27103106 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf00275b0 con 0x56552ce3e150 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.595+0000 7f9d11651740 1 -- 172.21.15.62:0/27103106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce3e150 msgr2=0x56552ce4b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.595+0000 7f9d11651740 1 --2- 172.21.15.62:0/27103106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce3e150 0x56552ce4b0b0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0008010 tx=0x7f9cf0010000 comp rx=0 tx=0).stop 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.595+0000 7f9d11651740 1 -- 172.21.15.62:0/27103106 shutdown_connections 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.595+0000 7f9d11651740 1 --2- 172.21.15.62:0/27103106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce3e150 0x56552ce4b0b0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.595+0000 7f9d11651740 1 --2- 172.21.15.62:0/27103106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce2c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.595+0000 7f9d11651740 1 --2- 172.21.15.62:0/27103106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce2a0f0 0x56552ce4b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.595+0000 7f9d11651740 1 -- 172.21.15.62:0/27103106 >> 172.21.15.62:0/27103106 conn(0x56552cdc9b40 msgr2=0x56552cdfbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.595+0000 7f9d11651740 1 -- 172.21.15.62:0/27103106 shutdown_connections 2024-03-23T11:25:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.595+0000 7f9d11651740 1 -- 172.21.15.62:0/27103106 wait complete. 2024-03-23T11:25:26.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.596+0000 7f9d11651740 1 -- start start 2024-03-23T11:25:26.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.596+0000 7f9d11651740 1 Processor -- start 2024-03-23T11:25:26.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.597+0000 7f9d11651740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce98ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.597+0000 7f9d11651740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e150 0x56552ce01d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.597+0000 7f9d11651740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce4b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.597+0000 7f9d11651740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552ce9a360 con 0x56552ce2a0f0 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.597+0000 7f9d11651740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552ce02640 con 0x56552ce3e150 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.597+0000 7f9d11651740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56552ce3b1a0 con 0x56552ce3e530 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.597+0000 7f9d11651740 10 client.0 _ll_register_callbacks cb 0x7f9cf0009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.597+0000 7f9d07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce98ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.597+0000 7f9d07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce98ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48696/0 (socket says 172.21.15.62:48696) 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.597+0000 7f9d07e00640 1 -- 172.21.15.62:0/3356821097 learned_addr learned my addr 172.21.15.62:0/3356821097 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.598+0000 7f9d0ca00640 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce4b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.598+0000 7f9cffe00640 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e150 0x56552ce01d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.598+0000 7f9d07e00640 1 -- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 msgr2=0x56552ce4b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.598+0000 7f9d07e00640 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce4b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.598+0000 7f9d07e00640 1 -- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e150 msgr2=0x56552ce01d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.598+0000 7f9d07e00640 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e150 0x56552ce01d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.598+0000 7f9d07e00640 1 -- 172.21.15.62:0/3356821097 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56552cd17360 con 0x56552ce2a0f0 2024-03-23T11:25:26.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.598+0000 7f9d07e00640 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce98ca0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009760 tx=0x7f9cf8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1a52e1d80762df server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:26.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.598+0000 7f9cf5600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:26.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.598+0000 7f9cf5600640 1 -- 172.21.15.62:0/3356821097 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8035800 con 0x56552ce2a0f0 2024-03-23T11:25:26.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9cf5600640 1 -- 172.21.15.62:0/3356821097 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9cf8023ce0 con 0x56552ce2a0f0 2024-03-23T11:25:26.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9d11651740 1 -- 172.21.15.62:0/3356821097 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9cf0012b30 con 0x56552ce2a0f0 2024-03-23T11:25:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9cf5600640 1 -- 172.21.15.62:0/3356821097 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cf8009790 con 0x56552ce2a0f0 2024-03-23T11:25:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9d11651740 10 client.7017 fetch_fsmap learned FSMap version 49 2024-03-23T11:25:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9d11651740 1 -- 172.21.15.62:0/3356821097 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x56552cd2b170 con 0x56552ce2a0f0 2024-03-23T11:25:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9cf5600640 1 -- 172.21.15.62:0/3356821097 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf0012b30 con 0x56552ce2a0f0 2024-03-23T11:25:26.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9cf5600640 1 -- 172.21.15.62:0/3356821097 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9cf8035960 con 0x56552ce2a0f0 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9d11651740 10 client.7017 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9d11651740 10 client.7017 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9d11651740 20 client.7017 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9d11651740 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x56552ce4b5f0 0x56552ce99330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9d11651740 4 client.7017 mds_command: new command op to 6618 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.599+0000 7f9d11651740 1 -- 172.21.15.62:0/3356821097 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56552ca594e0 con 0x56552ce4b5f0 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.600+0000 7f9cffe00640 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x56552ce4b5f0 0x56552ce99330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.601+0000 7f9cffe00640 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x56552ce4b5f0 0x56552ce99330 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be7a2bd313c0b636 server_cookie=7496b029cfb629fa in_seq=0 out_seq=0 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.601+0000 7f9cf5600640 10 client.7017 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.601+0000 7f9cf5600640 1 -- 172.21.15.62:0/3356821097 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56552ca594e0 con 0x56552ce4b5f0 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.601+0000 7f9cf5600640 10 client.7017 handle_command_reply: tid=0 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.601+0000 7f9d11651740 1 client.7017 shutdown 2024-03-23T11:25:26.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9d11651740 1 -- 172.21.15.62:0/3356821097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 msgr2=0x56552ce98ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9d11651740 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce98ca0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009760 tx=0x7f9cf8018000 comp rx=0 tx=0).stop 2024-03-23T11:25:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9d11651740 1 -- 172.21.15.62:0/3356821097 shutdown_connections 2024-03-23T11:25:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9d11651740 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x56552ce4b5f0 0x56552ce99330 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9d11651740 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56552ce3e530 0x56552ce4b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9d11651740 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56552ce3e150 0x56552ce01d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9d11651740 1 --2- 172.21.15.62:0/3356821097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56552ce2a0f0 0x56552ce98ca0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9cf5600640 0 client.7017 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9d11651740 1 -- 172.21.15.62:0/3356821097 >> 172.21.15.62:0/3356821097 conn(0x56552cdc9b40 msgr2=0x56552ce44c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9d11651740 1 -- 172.21.15.62:0/3356821097 shutdown_connections 2024-03-23T11:25:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9d11651740 1 -- 172.21.15.62:0/3356821097 wait complete. 2024-03-23T11:25:26.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.602+0000 7f9d11651740 20 client.7017 trim_cache size 0 max 16384 2024-03-23T11:25:26.599 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:25:26.599 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-03-23T11:25:26.599 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-03-23T11:25:26.599 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 -- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ce406a7a0 msgr2=0x7f9ce406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ce406a7a0 0x7f9ce406cc60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009a90 tx=0x7f9cf8008040 comp rx=0 tx=0).stop 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 -- 172.21.15.62:0/234173305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d080acf00 msgr2=0x7f9d080abd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d080acf00 0x7f9d080abd80 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f9d00002230 tx=0x7f9d080b2040 comp rx=0 tx=0).stop 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 -- 172.21.15.62:0/234173305 shutdown_connections 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ce406a7a0 0x7f9ce406cc60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d080acf00 0x7f9d080abd80 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d080aca80 0x7f9d080ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 --2- 172.21.15.62:0/234173305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec004d00 0x7f9d08141bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 -- 172.21.15.62:0/234173305 >> 172.21.15.62:0/234173305 conn(0x7f9d080b0090 msgr2=0x7f9d0815d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 -- 172.21.15.62:0/234173305 shutdown_connections 2024-03-23T11:25:26.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.603+0000 7f9d0d400640 1 -- 172.21.15.62:0/234173305 wait complete. 2024-03-23T11:25:26.613 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T11:25:26.613 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:25:26.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.832+0000 7f6930200640 1 Processor -- start 2024-03-23T11:25:26.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.832+0000 7f6930200640 1 -- start start 2024-03-23T11:25:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.834+0000 7f6930200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692815cdc0 0x7f69280d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.834+0000 7f6930200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692806bfc0 con 0x7f69280cf650 2024-03-23T11:25:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.834+0000 7f6930200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69280b5260 con 0x7f692815c9e0 2024-03-23T11:25:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.834+0000 7f6930200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69280ce140 con 0x7f692815cdc0 2024-03-23T11:25:26.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.834+0000 7f692e200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f69280cf650 0x7f692815d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40718/0 (socket says 172.21.15.62:40718) 2024-03-23T11:25:26.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.834+0000 7f692e200640 1 -- 172.21.15.62:0/3709929075 learned_addr learned my addr 172.21.15.62:0/3709929075 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:25:26.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f692ce00640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692815cdc0 0x7f69280d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245525559 0 0) 0x7f692806bfc0 con 0x7f69280cf650 2024-03-23T11:25:26.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6910003520 con 0x7f69280cf650 2024-03-23T11:25:26.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3347679461 0 0) 0x7f6910003520 con 0x7f69280cf650 2024-03-23T11:25:26.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692815cdc0 msgr2=0x7f69280d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692815cdc0 0x7f69280d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 >> v1:172.21.15.120:6789/0 conn(0x7f692815c9e0 legacy=0x7f69281607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69280b4370 con 0x7f69280cf650 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f69200019f0 con 0x7f69280cf650 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 >> v1:172.21.15.62:6789/0 conn(0x7f69280cf650 legacy=0x7f692815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69100039a0 0x7f6910003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692815cdc0 0x7f69100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.835+0000 7f6927e00640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6910004d00 0x7f6910005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.836+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69280ce140 con 0x7f6910004d00 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.836+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691000ba30 con 0x7f69100039a0 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.836+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691000bbf0 con 0x7f692815cdc0 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.836+0000 7f692e200640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6910004d00 0x7f6910005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.836+0000 7f692d800640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69100039a0 0x7f6910003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.836+0000 7f692ce00640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692815cdc0 0x7f69100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.836+0000 7f692e200640 1 -- 172.21.15.62:0/3709929075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692815cdc0 msgr2=0x7f69100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.836+0000 7f692e200640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692815cdc0 0x7f69100047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.836+0000 7f692e200640 1 -- 172.21.15.62:0/3709929075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69100039a0 msgr2=0x7f6910003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:26.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.836+0000 7f692e200640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69100039a0 0x7f6910003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.836+0000 7f692e200640 1 -- 172.21.15.62:0/3709929075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6910003520 con 0x7f6910004d00 2024-03-23T11:25:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.837+0000 7f692e200640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6910004d00 0x7f6910005180 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f692000a800 tx=0x7f69100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f0c8156202f61 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.838+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69200072d0 con 0x7f6910004d00 2024-03-23T11:25:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.838+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692002f070 con 0x7f6910004d00 2024-03-23T11:25:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.838+0000 7f6927e00640 1 -- 172.21.15.62:0/3709929075 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69200076b0 con 0x7f6910004d00 2024-03-23T11:25:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.838+0000 7f6930200640 1 -- 172.21.15.62:0/3709929075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6910004d00 msgr2=0x7f6910005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.838+0000 7f6930200640 1 --2- 172.21.15.62:0/3709929075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6910004d00 0x7f6910005180 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f692000a800 tx=0x7f69100057f0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.838+0000 7f692e200640 1 -- 172.21.15.62:0/3709929075 reap_dead start 2024-03-23T11:25:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.838+0000 7f6930200640 1 -- 172.21.15.62:0/3709929075 shutdown_connections 2024-03-23T11:25:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.838+0000 7f6930200640 1 -- 172.21.15.62:0/3709929075 >> 172.21.15.62:0/3709929075 conn(0x7f6928071570 msgr2=0x7f6928071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.839+0000 7f6930200640 1 -- 172.21.15.62:0/3709929075 shutdown_connections 2024-03-23T11:25:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.839+0000 7f6930200640 1 -- 172.21.15.62:0/3709929075 wait complete. 2024-03-23T11:25:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.840+0000 7f6930200640 1 Processor -- start 2024-03-23T11:25:26.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.840+0000 7f6930200640 1 -- start start 2024-03-23T11:25:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.841+0000 7f6930200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6910004d00 0x7f692814c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.841+0000 7f6930200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69280a99a0 0x7f692814a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.841+0000 7f6930200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692814aa00 0x7f69280cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.841+0000 7f6930200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692814bf80 con 0x7f69280a99a0 2024-03-23T11:25:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.841+0000 7f6930200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692814b370 con 0x7f6910004d00 2024-03-23T11:25:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.841+0000 7f6930200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f692814b650 con 0x7f692814aa00 2024-03-23T11:25:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.842+0000 7f692ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69280a99a0 0x7f692814a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.842+0000 7f692ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69280a99a0 0x7f692814a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48718/0 (socket says 172.21.15.62:48718) 2024-03-23T11:25:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.842+0000 7f692ce00640 1 -- 172.21.15.62:0/1470022661 learned_addr learned my addr 172.21.15.62:0/1470022661 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:26.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.842+0000 7f692d800640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6910004d00 0x7f692814c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.842+0000 7f692ce00640 1 -- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692814aa00 msgr2=0x7f69280cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.842+0000 7f692ce00640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692814aa00 0x7f69280cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.842+0000 7f692ce00640 1 -- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6910004d00 msgr2=0x7f692814c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:26.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.842+0000 7f692ce00640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6910004d00 0x7f692814c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:26.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.842+0000 7f692ce00640 1 -- 172.21.15.62:0/1470022661 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692814b930 con 0x7f69280a99a0 2024-03-23T11:25:26.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.842+0000 7f692ce00640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69280a99a0 0x7f692814a350 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f6914010860 tx=0x7f6914010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:26.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.843+0000 7f6925600640 1 -- 172.21.15.62:0/1470022661 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914009610 con 0x7f69280a99a0 2024-03-23T11:25:26.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.843+0000 7f6925600640 1 -- 172.21.15.62:0/1470022661 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691400eae0 con 0x7f69280a99a0 2024-03-23T11:25:26.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.843+0000 7f6925600640 1 -- 172.21.15.62:0/1470022661 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691400ee70 con 0x7f69280a99a0 2024-03-23T11:25:26.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.843+0000 7f6930200640 1 -- 172.21.15.62:0/1470022661 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692814bc10 con 0x7f69280a99a0 2024-03-23T11:25:26.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.843+0000 7f6930200640 1 -- 172.21.15.62:0/1470022661 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692814b060 con 0x7f69280a99a0 2024-03-23T11:25:26.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.844+0000 7f6930200640 1 -- 172.21.15.62:0/1470022661 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6928066c90 con 0x7f69280a99a0 2024-03-23T11:25:26.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.845+0000 7f6925600640 1 -- 172.21.15.62:0/1470022661 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f691401c400 con 0x7f69280a99a0 2024-03-23T11:25:26.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.846+0000 7f6925600640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f690806a7a0 0x7f690806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:26.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.846+0000 7f6925600640 1 -- 172.21.15.62:0/1470022661 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f69140894b0 con 0x7f69280a99a0 2024-03-23T11:25:26.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.847+0000 7f692d800640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f690806a7a0 0x7f690806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:26.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.848+0000 7f692d800640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f690806a7a0 0x7f690806cc60 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f691c007fd0 tx=0x7f691c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:26.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:26.851+0000 7f6925600640 1 -- 172.21.15.62:0/1470022661 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69140547c0 con 0x7f69280a99a0 2024-03-23T11:25:27.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.088+0000 7f6930200640 1 -- 172.21.15.62:0/1470022661 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69280b5470 con 0x7f69280a99a0 2024-03-23T11:25:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.089+0000 7f6925600640 1 -- 172.21.15.62:0/1470022661 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f6914057de0 con 0x7f69280a99a0 2024-03-23T11:25:27.083 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:25:27.084 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":49,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/29171509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":29171509},{"type":"v1","addr":"172.21.15.120:6837","nonce":29171509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:24:46.247989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6618":{"gid":6618,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":13,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:25:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 49 2024-03-23T11:25:27.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 -- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f690806a7a0 msgr2=0x7f690806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f690806a7a0 0x7f690806cc60 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f691c007fd0 tx=0x7f691c009040 comp rx=0 tx=0).stop 2024-03-23T11:25:27.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 -- 172.21.15.62:0/1470022661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69280a99a0 msgr2=0x7f692814a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69280a99a0 0x7f692814a350 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f6914010860 tx=0x7f6914010d20 comp rx=0 tx=0).stop 2024-03-23T11:25:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 -- 172.21.15.62:0/1470022661 shutdown_connections 2024-03-23T11:25:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f690806a7a0 0x7f690806cc60 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69280a99a0 0x7f692814a350 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f692814aa00 0x7f69280cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 --2- 172.21.15.62:0/1470022661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6910004d00 0x7f692814c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 -- 172.21.15.62:0/1470022661 >> 172.21.15.62:0/1470022661 conn(0x7f6928071570 msgr2=0x7f69280b7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 -- 172.21.15.62:0/1470022661 shutdown_connections 2024-03-23T11:25:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.092+0000 7f6930200640 1 -- 172.21.15.62:0/1470022661 wait complete. 2024-03-23T11:25:27.106 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b damage ls --format=json-pretty 2024-03-23T11:25:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.334+0000 7fe759e00640 1 Processor -- start 2024-03-23T11:25:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.334+0000 7fe759e00640 1 -- start start 2024-03-23T11:25:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.336+0000 7fe759e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe7541431f0 0x7fe7540b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.336+0000 7fe759e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7540b20d0 con 0x7fe7540cc9b0 2024-03-23T11:25:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.336+0000 7fe759e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7541435d0 con 0x7fe754070090 2024-03-23T11:25:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.336+0000 7fe759e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7540b14c0 con 0x7fe7541431f0 2024-03-23T11:25:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.337+0000 7fe759400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe7540cc9b0 0x7fe75415aae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48418/0 (socket says 172.21.15.62:48418) 2024-03-23T11:25:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.337+0000 7fe759400640 1 -- 172.21.15.62:0/352893838 learned_addr learned my addr 172.21.15.62:0/352893838 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:25:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.337+0000 7fe758a00640 1 --2- 172.21.15.62:0/352893838 >> v2:172.21.15.120:3300/0 conn(0x7fe7541431f0 0x7fe7540b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.337+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055823666 0 0) 0x7fe7540b20d0 con 0x7fe7540cc9b0 2024-03-23T11:25:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.337+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe73c003520 con 0x7fe7540cc9b0 2024-03-23T11:25:27.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2736476160 0 0) 0x7fe73c003520 con 0x7fe7540cc9b0 2024-03-23T11:25:27.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 >> v2:172.21.15.120:3300/0 conn(0x7fe7541431f0 msgr2=0x7fe7540b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 --2- 172.21.15.62:0/352893838 >> v2:172.21.15.120:3300/0 conn(0x7fe7541431f0 0x7fe7540b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 >> v1:172.21.15.120:6789/0 conn(0x7fe754070090 legacy=0x7fe7540cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7540b17a0 con 0x7fe7540cc9b0 2024-03-23T11:25:27.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fe74c0019f0 con 0x7fe7540cc9b0 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 >> v1:172.21.15.62:6789/0 conn(0x7fe7540cc9b0 legacy=0x7fe75415aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe73c0039c0 0x7fe73c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe73c0047c0 0x7fe73c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe73c0051d0 0x7fe73c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7540b14c0 con 0x7fe73c0039c0 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c00c120 con 0x7fe73c0051d0 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c00c2e0 con 0x7fe73c0047c0 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe758a00640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe73c0039c0 0x7fe73c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.338+0000 7fe759400640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe73c0051d0 0x7fe73c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.339+0000 7fe753e00640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe73c0047c0 0x7fe73c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.339+0000 7fe758a00640 1 -- 172.21.15.62:0/352893838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe73c0047c0 msgr2=0x7fe73c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.339+0000 7fe758a00640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe73c0047c0 0x7fe73c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.339+0000 7fe758a00640 1 -- 172.21.15.62:0/352893838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe73c0051d0 msgr2=0x7fe73c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.339+0000 7fe758a00640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe73c0051d0 0x7fe73c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.339+0000 7fe758a00640 1 -- 172.21.15.62:0/352893838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73c003520 con 0x7fe73c0039c0 2024-03-23T11:25:27.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.339+0000 7fe753e00640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe73c0047c0 0x7fe73c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:27.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.339+0000 7fe759400640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe73c0051d0 0x7fe73c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:27.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.339+0000 7fe759400640 1 -- 172.21.15.62:0/352893838 reap_dead start 2024-03-23T11:25:27.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.340+0000 7fe758a00640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe73c0039c0 0x7fe73c003de0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fe748009a90 tx=0x7fe73c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2cdc108396cd18 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:27.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.340+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74802c680 con 0x7fe73c0039c0 2024-03-23T11:25:27.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.340+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74802c7e0 con 0x7fe73c0039c0 2024-03-23T11:25:27.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.340+0000 7fe753400640 1 -- 172.21.15.62:0/352893838 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74802cb70 con 0x7fe73c0039c0 2024-03-23T11:25:27.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.341+0000 7fe759e00640 1 -- 172.21.15.62:0/352893838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe73c0039c0 msgr2=0x7fe73c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.341+0000 7fe759e00640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe73c0039c0 0x7fe73c003de0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fe748009a90 tx=0x7fe73c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.341+0000 7fe759e00640 1 -- 172.21.15.62:0/352893838 shutdown_connections 2024-03-23T11:25:27.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.341+0000 7fe759e00640 1 --2- 172.21.15.62:0/352893838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe73c0039c0 0x7fe73c003de0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.341+0000 7fe759e00640 1 -- 172.21.15.62:0/352893838 >> 172.21.15.62:0/352893838 conn(0x7fe7540b0090 msgr2=0x7fe7540b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:27.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.341+0000 7fe759e00640 1 -- 172.21.15.62:0/352893838 shutdown_connections 2024-03-23T11:25:27.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.341+0000 7fe759e00640 1 -- 172.21.15.62:0/352893838 wait complete. 2024-03-23T11:25:27.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.343+0000 7fe759e00640 1 Processor -- start 2024-03-23T11:25:27.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.343+0000 7fe759e00640 1 -- start start 2024-03-23T11:25:27.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe759e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe73c0039c0 0x7fe7541f1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe759e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7540acaf0 0x7fe75406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe759e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7540acf70 0x7fe75406c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe759e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75415b9b0 con 0x7fe7540acaf0 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe759e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541f1c90 con 0x7fe73c0039c0 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe759e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75415ad30 con 0x7fe7540acf70 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe753e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7540acaf0 0x7fe75406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe753e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7540acaf0 0x7fe75406c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49570/0 (socket says 172.21.15.62:49570) 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe753e00640 1 -- 172.21.15.62:0/729634783 learned_addr learned my addr 172.21.15.62:0/729634783 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe753e00640 1 -- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7540acf70 msgr2=0x7fe75406c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe753e00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7540acf70 0x7fe75406c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe753e00640 1 -- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe73c0039c0 msgr2=0x7fe7541f1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe753e00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe73c0039c0 0x7fe7541f1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.344+0000 7fe753e00640 1 -- 172.21.15.62:0/729634783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75415b010 con 0x7fe7540acaf0 2024-03-23T11:25:27.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.345+0000 7fe753e00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7540acaf0 0x7fe75406c0d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fe74000d400 tx=0x7fe74000d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:27.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.345+0000 7fe750c00640 1 -- 172.21.15.62:0/729634783 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74000b070 con 0x7fe7540acaf0 2024-03-23T11:25:27.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.345+0000 7fe750c00640 1 -- 172.21.15.62:0/729634783 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe740003610 con 0x7fe7540acaf0 2024-03-23T11:25:27.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.345+0000 7fe759e00640 1 -- 172.21.15.62:0/729634783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75415b2f0 con 0x7fe7540acaf0 2024-03-23T11:25:27.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.345+0000 7fe750c00640 1 -- 172.21.15.62:0/729634783 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7400039c0 con 0x7fe7540acaf0 2024-03-23T11:25:27.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.345+0000 7fe759e00640 1 -- 172.21.15.62:0/729634783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75415a950 con 0x7fe7540acaf0 2024-03-23T11:25:27.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.347+0000 7fe750c00640 1 -- 172.21.15.62:0/729634783 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe740003b20 con 0x7fe7540acaf0 2024-03-23T11:25:27.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.348+0000 7fe750c00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe73406a7d0 0x7fe73406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.348+0000 7fe750c00640 1 -- 172.21.15.62:0/729634783 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe7400885f0 con 0x7fe7540acaf0 2024-03-23T11:25:27.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.349+0000 7fe758a00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe73406a7d0 0x7fe73406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.349+0000 7fe75e18a740 1 Processor -- start 2024-03-23T11:25:27.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.349+0000 7fe75e18a740 1 -- start start 2024-03-23T11:25:27.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.350+0000 7fe758a00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe73406a7d0 0x7fe73406cc90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe748036000 tx=0x7fe74802e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:27.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.350+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573d3b60 0x55b7573d3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.350+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573d7bb0 0x55b7573d9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.350+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7574038f0 0x55b757403d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.350+0000 7fe75e18a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b757343d80 con 0x55b7573d7bb0 2024-03-23T11:25:27.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.350+0000 7fe75e18a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b757341350 con 0x55b7574038f0 2024-03-23T11:25:27.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.350+0000 7fe75e18a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b75733cc40 con 0x55b7573d3b60 2024-03-23T11:25:27.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.350+0000 7fe753e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573d7bb0 0x55b7573d9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.350+0000 7fe753e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573d7bb0 0x55b7573d9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49578/0 (socket says 172.21.15.62:49578) 2024-03-23T11:25:27.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.350+0000 7fe753e00640 1 -- 172.21.15.62:0/3506473519 learned_addr learned my addr 172.21.15.62:0/3506473519 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:27.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.351+0000 7fe758a00640 1 --2- 172.21.15.62:0/3506473519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573d3b60 0x55b7573d3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.351+0000 7fe759400640 1 --2- 172.21.15.62:0/3506473519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7574038f0 0x55b757403d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.351+0000 7fe753e00640 1 -- 172.21.15.62:0/3506473519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573d3b60 msgr2=0x55b7573d3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.351+0000 7fe753e00640 1 --2- 172.21.15.62:0/3506473519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573d3b60 0x55b7573d3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.351+0000 7fe753e00640 1 -- 172.21.15.62:0/3506473519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7574038f0 msgr2=0x55b757403d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.351+0000 7fe753e00640 1 --2- 172.21.15.62:0/3506473519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7574038f0 0x55b757403d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.351+0000 7fe753e00640 1 -- 172.21.15.62:0/3506473519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7573073e0 con 0x55b7573d7bb0 2024-03-23T11:25:27.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.351+0000 7fe753e00640 1 --2- 172.21.15.62:0/3506473519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573d7bb0 0x55b7573d9650 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fe740086a20 tx=0x7fe740057170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c666720645f963bb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:27.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.351+0000 7fe745600640 1 -- 172.21.15.62:0/3506473519 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74005fa20 con 0x55b7573d7bb0 2024-03-23T11:25:27.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.352+0000 7fe745600640 1 -- 172.21.15.62:0/3506473519 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe740055820 con 0x55b7573d7bb0 2024-03-23T11:25:27.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.352+0000 7fe745600640 1 -- 172.21.15.62:0/3506473519 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74005b170 con 0x55b7573d7bb0 2024-03-23T11:25:27.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.352+0000 7fe75e18a740 1 -- 172.21.15.62:0/3506473519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573d7bb0 msgr2=0x55b7573d9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.352+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3506473519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573d7bb0 0x55b7573d9650 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fe740086a20 tx=0x7fe740057170 comp rx=0 tx=0).stop 2024-03-23T11:25:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.352+0000 7fe75e18a740 1 -- 172.21.15.62:0/3506473519 shutdown_connections 2024-03-23T11:25:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.352+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3506473519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573d7bb0 0x55b7573d9650 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.352+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3506473519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7574038f0 0x55b757403d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.352+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3506473519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573d3b60 0x55b7573d3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.352+0000 7fe75e18a740 1 -- 172.21.15.62:0/3506473519 >> 172.21.15.62:0/3506473519 conn(0x55b7573d5be0 msgr2=0x55b7573d74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.352+0000 7fe75e18a740 1 -- 172.21.15.62:0/3506473519 shutdown_connections 2024-03-23T11:25:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.352+0000 7fe75e18a740 1 -- 172.21.15.62:0/3506473519 wait complete. 2024-03-23T11:25:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.353+0000 7fe75e18a740 1 -- start start 2024-03-23T11:25:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.353+0000 7fe75e18a740 1 Processor -- start 2024-03-23T11:25:27.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.354+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7574038f0 0x55b757420910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.354+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7573f0370 0x55b7573d90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.354+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f1c10 0x55b7573f2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.354+0000 7fe75e18a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7572ddce0 con 0x55b7573f1c10 2024-03-23T11:25:27.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.354+0000 7fe75e18a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7572dd980 con 0x55b7573f0370 2024-03-23T11:25:27.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.354+0000 7fe75e18a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b7570a5000 con 0x55b7574038f0 2024-03-23T11:25:27.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.354+0000 7fe75e18a740 10 client.0 _ll_register_callbacks cb 0x55b757336410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:27.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe759400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f1c10 0x55b7573f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe759400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f1c10 0x55b7573f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49594/0 (socket says 172.21.15.62:49594) 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe759400640 1 -- 172.21.15.62:0/3920297973 learned_addr learned my addr 172.21.15.62:0/3920297973 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe758a00640 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7574038f0 0x55b757420910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe753e00640 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7573f0370 0x55b7573d90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe759400640 1 -- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7574038f0 msgr2=0x55b757420910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe759400640 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7574038f0 0x55b757420910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe759400640 1 -- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7573f0370 msgr2=0x55b7573d90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe759400640 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7573f0370 0x55b7573d90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe759400640 1 -- 172.21.15.62:0/3920297973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75415a950 con 0x55b7573f1c10 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe753e00640 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7573f0370 0x55b7573d90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe759400640 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f1c10 0x55b7573f2090 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fe754071490 tx=0x7fe74c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20d90fa291f3e37 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.355+0000 7fe73a000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:27.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.356+0000 7fe73a000640 1 -- 172.21.15.62:0/3920297973 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c0079d0 con 0x55b7573f1c10 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.356+0000 7fe73a000640 1 -- 172.21.15.62:0/3920297973 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe74c00f910 con 0x55b7573f1c10 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.356+0000 7fe75e18a740 1 -- 172.21.15.62:0/3920297973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b756fee8d0 con 0x55b7573f1c10 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.356+0000 7fe73a000640 1 -- 172.21.15.62:0/3920297973 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe74c00fad0 con 0x55b7573f1c10 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.356+0000 7fe75e18a740 10 client.7035 fetch_fsmap learned FSMap version 49 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.356+0000 7fe75e18a740 1 -- 172.21.15.62:0/3920297973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x55b75733b2c0 con 0x55b7573f1c10 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.356+0000 7fe73a000640 1 -- 172.21.15.62:0/3920297973 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b756fee8d0 con 0x55b7573f1c10 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.356+0000 7fe73a000640 1 -- 172.21.15.62:0/3920297973 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe74c012070 con 0x55b7573f1c10 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.356+0000 7fe75e18a740 10 client.7035 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.356+0000 7fe75e18a740 10 client.7035 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.356+0000 7fe75e18a740 20 client.7035 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.357+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55b757404a60 0x55b757404e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.357+0000 7fe75e18a740 4 client.7035 mds_command: new command op to 6618 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.357+0000 7fe75e18a740 1 -- 172.21.15.62:0/3920297973 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b7570314a0 con 0x55b757404a60 2024-03-23T11:25:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.357+0000 7fe758a00640 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55b757404a60 0x55b757404e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.358+0000 7fe758a00640 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55b757404a60 0x55b757404e40 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3465f86bd399cc44 server_cookie=a010cb9edd127ae in_seq=0 out_seq=0 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.358+0000 7fe73a000640 10 client.7035 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe73a000640 1 -- 172.21.15.62:0/3920297973 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55b7570314a0 con 0x55b757404a60 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe73a000640 10 client.7035 handle_command_reply: tid=0 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe75e18a740 1 client.7035 shutdown 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe75e18a740 1 -- 172.21.15.62:0/3920297973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f1c10 msgr2=0x55b7573f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f1c10 0x55b7573f2090 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fe754071490 tx=0x7fe74c007780 comp rx=0 tx=0).stop 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe75e18a740 1 -- 172.21.15.62:0/3920297973 shutdown_connections 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55b757404a60 0x55b757404e40 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f1c10 0x55b7573f2090 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7573f0370 0x55b7573d90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3920297973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7574038f0 0x55b757420910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe75e18a740 1 -- 172.21.15.62:0/3920297973 >> 172.21.15.62:0/3920297973 conn(0x55b7573d3b60 msgr2=0x55b75741fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:27.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe73a000640 0 client.7035 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:27.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe75e18a740 1 -- 172.21.15.62:0/3920297973 shutdown_connections 2024-03-23T11:25:27.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.360+0000 7fe75e18a740 1 -- 172.21.15.62:0/3920297973 wait complete. 2024-03-23T11:25:27.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.361+0000 7fe75e18a740 20 client.7035 trim_cache size 0 max 16384 2024-03-23T11:25:27.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.373+0000 7fe75e18a740 1 Processor -- start 2024-03-23T11:25:27.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.373+0000 7fe75e18a740 1 -- start start 2024-03-23T11:25:27.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.374+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7573f0370 0x55b7573f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.374+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573f1c10 0x55b7573f1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.374+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7574038f0 0x55b75741e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.374+0000 7fe75e18a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7573f1ab0 con 0x55b7574038f0 2024-03-23T11:25:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.374+0000 7fe75e18a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7574310e0 con 0x55b7573f0370 2024-03-23T11:25:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.374+0000 7fe75e18a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b7573d6110 con 0x55b7573f1c10 2024-03-23T11:25:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.375+0000 7fe759400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7574038f0 0x55b75741e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.375+0000 7fe759400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7574038f0 0x55b75741e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49600/0 (socket says 172.21.15.62:49600) 2024-03-23T11:25:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.375+0000 7fe759400640 1 -- 172.21.15.62:0/4175477218 learned_addr learned my addr 172.21.15.62:0/4175477218 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:27.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.375+0000 7fe753e00640 1 --2- 172.21.15.62:0/4175477218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573f1c10 0x55b7573f1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.375+0000 7fe759400640 1 -- 172.21.15.62:0/4175477218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573f1c10 msgr2=0x55b7573f1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.375+0000 7fe759400640 1 --2- 172.21.15.62:0/4175477218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573f1c10 0x55b7573f1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.375+0000 7fe759400640 1 -- 172.21.15.62:0/4175477218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7573f0370 msgr2=0x55b7573f0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.375+0000 7fe759400640 1 --2- 172.21.15.62:0/4175477218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7573f0370 0x55b7573f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.375+0000 7fe759400640 1 -- 172.21.15.62:0/4175477218 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7572dd170 con 0x55b7574038f0 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.375+0000 7fe753e00640 1 --2- 172.21.15.62:0/4175477218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573f1c10 0x55b7573f1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.375+0000 7fe759400640 1 --2- 172.21.15.62:0/4175477218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7574038f0 0x55b75741e4c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fe74c000f80 tx=0x7fe74c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200aa499354e8364 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe746000640 1 -- 172.21.15.62:0/4175477218 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c001770 con 0x55b7574038f0 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe746000640 1 -- 172.21.15.62:0/4175477218 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c00d050 con 0x55b7574038f0 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe746000640 1 -- 172.21.15.62:0/4175477218 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c00fd70 con 0x55b7574038f0 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe75e18a740 1 -- 172.21.15.62:0/4175477218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7574038f0 msgr2=0x55b75741e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe75e18a740 1 --2- 172.21.15.62:0/4175477218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7574038f0 0x55b75741e4c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fe74c000f80 tx=0x7fe74c013000 comp rx=0 tx=0).stop 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe75e18a740 1 -- 172.21.15.62:0/4175477218 shutdown_connections 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe75e18a740 1 --2- 172.21.15.62:0/4175477218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7574038f0 0x55b75741e4c0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe75e18a740 1 --2- 172.21.15.62:0/4175477218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573f1c10 0x55b7573f1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe75e18a740 1 --2- 172.21.15.62:0/4175477218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7573f0370 0x55b7573f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe75e18a740 1 -- 172.21.15.62:0/4175477218 >> 172.21.15.62:0/4175477218 conn(0x55b7573d3b60 msgr2=0x55b7573d85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:27.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe75e18a740 1 -- 172.21.15.62:0/4175477218 shutdown_connections 2024-03-23T11:25:27.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.376+0000 7fe75e18a740 1 -- 172.21.15.62:0/4175477218 wait complete. 2024-03-23T11:25:27.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.377+0000 7fe75e18a740 1 -- start start 2024-03-23T11:25:27.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.377+0000 7fe75e18a740 1 Processor -- start 2024-03-23T11:25:27.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.378+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f0370 0x55b757483d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.378+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573f1c10 0x55b7573fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.378+0000 7fe75e18a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7574038f0 0x55b75741e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.378+0000 7fe75e18a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b757484760 con 0x55b7573f0370 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.378+0000 7fe75e18a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7573fa640 con 0x55b7574038f0 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.378+0000 7fe75e18a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b757484290 con 0x55b7573f1c10 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.378+0000 7fe75e18a740 10 client.0 _ll_register_callbacks cb 0x7fe74c001990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe758a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f0370 0x55b757483d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe758a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f0370 0x55b757483d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49610/0 (socket says 172.21.15.62:49610) 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe758a00640 1 -- 172.21.15.62:0/3393171090 learned_addr learned my addr 172.21.15.62:0/3393171090 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe759400640 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7574038f0 0x55b75741e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe753e00640 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573f1c10 0x55b7573fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe758a00640 1 -- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573f1c10 msgr2=0x55b7573fbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe758a00640 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573f1c10 0x55b7573fbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe758a00640 1 -- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7574038f0 msgr2=0x55b75741e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe758a00640 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7574038f0 0x55b75741e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe758a00640 1 -- 172.21.15.62:0/3393171090 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b75733b2c0 con 0x55b7573f0370 2024-03-23T11:25:27.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe758a00640 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f0370 0x55b757483d50 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fe748000c00 tx=0x7fe748004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f3e03d2a3e430d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:27.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.379+0000 7fe73a000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.380+0000 7fe73a000640 1 -- 172.21.15.62:0/3393171090 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748047040 con 0x55b7573f0370 2024-03-23T11:25:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.380+0000 7fe75e18a740 1 -- 172.21.15.62:0/3393171090 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe740055820 con 0x55b7573f0370 2024-03-23T11:25:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.380+0000 7fe73a000640 1 -- 172.21.15.62:0/3393171090 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe748005b90 con 0x55b7573f0370 2024-03-23T11:25:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.380+0000 7fe73a000640 1 -- 172.21.15.62:0/3393171090 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe748005d50 con 0x55b7573f0370 2024-03-23T11:25:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.380+0000 7fe75e18a740 10 client.7041 fetch_fsmap learned FSMap version 49 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.380+0000 7fe75e18a740 1 -- 172.21.15.62:0/3393171090 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x55b757302a50 con 0x55b7573f0370 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.380+0000 7fe73a000640 1 -- 172.21.15.62:0/3393171090 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe740055820 con 0x55b7573f0370 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.381+0000 7fe73a000640 1 -- 172.21.15.62:0/3393171090 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe74802c950 con 0x55b7573f0370 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.381+0000 7fe75e18a740 10 client.7041 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.381+0000 7fe75e18a740 10 client.7041 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.381+0000 7fe75e18a740 20 client.7041 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.381+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55b75747ddb0 0x55b75741ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.381+0000 7fe75e18a740 4 client.7041 mds_command: new command op to 6618 tid=0[{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.381+0000 7fe75e18a740 1 -- 172.21.15.62:0/3393171090 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x55b7570427a0 con 0x55b75747ddb0 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.381+0000 7fe753e00640 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55b75747ddb0 0x55b75741ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.382+0000 7fe753e00640 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55b75747ddb0 0x55b75741ea00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd3c9737b4e82d7d server_cookie=e50f2c21f9fbe4ec in_seq=0 out_seq=0 2024-03-23T11:25:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.382+0000 7fe73a000640 10 client.7041 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:27.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe73a000640 1 -- 172.21.15.62:0/3393171090 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+237 (crc 0 0 0) 0x55b7570427a0 con 0x55b75747ddb0 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe73a000640 10 client.7041 handle_command_reply: tid=0 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 1 client.7041 shutdown 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 1 -- 172.21.15.62:0/3393171090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f0370 msgr2=0x55b757483d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f0370 0x55b757483d50 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fe748000c00 tx=0x7fe748004730 comp rx=0 tx=0).stop 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 1 -- 172.21.15.62:0/3393171090 shutdown_connections 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x55b75747ddb0 0x55b75741ea00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b7574038f0 0x55b75741e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b7573f1c10 0x55b7573fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe73a000640 0 client.7041 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 1 --2- 172.21.15.62:0/3393171090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7573f0370 0x55b757483d50 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 1 -- 172.21.15.62:0/3393171090 >> 172.21.15.62:0/3393171090 conn(0x55b7573d3b60 msgr2=0x55b75741f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 1 -- 172.21.15.62:0/3393171090 shutdown_connections 2024-03-23T11:25:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 1 -- 172.21.15.62:0/3393171090 wait complete. 2024-03-23T11:25:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.383+0000 7fe75e18a740 20 client.7041 trim_cache size 0 max 16384 2024-03-23T11:25:27.380 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:25:27.380 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:25:27.380 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:25:27.380 INFO:teuthology.orchestra.run.smithi062.stdout: "damage_type": "dentry", 2024-03-23T11:25:27.380 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 2687696748, 2024-03-23T11:25:27.380 INFO:teuthology.orchestra.run.smithi062.stdout: "ino": 1099511627776, 2024-03-23T11:25:27.380 INFO:teuthology.orchestra.run.smithi062.stdout: "frag": "*", 2024-03-23T11:25:27.380 INFO:teuthology.orchestra.run.smithi062.stdout: "dname": "file_to_be_damaged", 2024-03-23T11:25:27.380 INFO:teuthology.orchestra.run.smithi062.stdout: "snap_id": "head", 2024-03-23T11:25:27.381 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/subdir/file_to_be_damaged" 2024-03-23T11:25:27.381 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:25:27.381 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:25:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 -- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe73406a7d0 msgr2=0x7fe73406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe73406a7d0 0x7fe73406cc90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe748036000 tx=0x7fe74802e000 comp rx=0 tx=0).stop 2024-03-23T11:25:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 -- 172.21.15.62:0/729634783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7540acaf0 msgr2=0x7fe75406c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7540acaf0 0x7fe75406c0d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fe74000d400 tx=0x7fe74000d8c0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 -- 172.21.15.62:0/729634783 shutdown_connections 2024-03-23T11:25:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe73406a7d0 0x7fe73406cc90 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7540acaf0 0x7fe75406c0d0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7540acf70 0x7fe75406c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 --2- 172.21.15.62:0/729634783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe73c0039c0 0x7fe7541f1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 -- 172.21.15.62:0/729634783 >> 172.21.15.62:0/729634783 conn(0x7fe7540b0090 msgr2=0x7fe7540cb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 -- 172.21.15.62:0/729634783 shutdown_connections 2024-03-23T11:25:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.384+0000 7fe759e00640 1 -- 172.21.15.62:0/729634783 wait complete. 2024-03-23T11:25:27.396 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-23T11:25:27.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.611+0000 7f371ba00640 1 Processor -- start 2024-03-23T11:25:27.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.611+0000 7f371ba00640 1 -- start start 2024-03-23T11:25:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.612+0000 7f371ba00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f371415ddb0 0x7f3714099550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.612+0000 7f371ba00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f371415e190 0x7f37140ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.612+0000 7f371ba00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37140990c0 con 0x7f3714099a90 2024-03-23T11:25:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.612+0000 7f371ba00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3714098a70 con 0x7f371415ddb0 2024-03-23T11:25:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.612+0000 7f371ba00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3714098d50 con 0x7f371415e190 2024-03-23T11:25:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.613+0000 7f3713e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3714099a90 0x7f37140c4320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48424/0 (socket says 172.21.15.62:48424) 2024-03-23T11:25:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.613+0000 7f3713e00640 1 -- 172.21.15.62:0/336104589 learned_addr learned my addr 172.21.15.62:0/336104589 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:25:27.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.613+0000 7f3719000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f371415ddb0 0x7f3714099550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.613+0000 7f3719a00640 1 --2- 172.21.15.62:0/336104589 >> v2:172.21.15.120:3300/0 conn(0x7f371415e190 0x7f37140ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.613+0000 7f3713400640 1 -- 172.21.15.62:0/336104589 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949061916 0 0) 0x7f37140990c0 con 0x7f3714099a90 2024-03-23T11:25:27.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.613+0000 7f3713400640 1 -- 172.21.15.62:0/336104589 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f8003520 con 0x7f3714099a90 2024-03-23T11:25:27.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.613+0000 7f3719000640 1 -- 172.21.15.62:0/336104589 >> v2:172.21.15.120:3300/0 conn(0x7f371415e190 msgr2=0x7f37140ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:27.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.613+0000 7f3719000640 1 --2- 172.21.15.62:0/336104589 >> v2:172.21.15.120:3300/0 conn(0x7f371415e190 0x7f37140ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.613+0000 7f3719000640 1 -- 172.21.15.62:0/336104589 >> v1:172.21.15.62:6789/0 conn(0x7f3714099a90 legacy=0x7f37140c4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.613+0000 7f3719000640 1 -- 172.21.15.62:0/336104589 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37140cc230 con 0x7f371415ddb0 2024-03-23T11:25:27.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.615+0000 7f3719000640 1 --2- 172.21.15.62:0/336104589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371415ddb0 0x7f3714099550 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f37140cb190 tx=0x7f370402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fdb329ee044614a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:27.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.615+0000 7f3713400640 1 -- 172.21.15.62:0/336104589 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704030610 con 0x7f371415ddb0 2024-03-23T11:25:27.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.615+0000 7f3713400640 1 -- 172.21.15.62:0/336104589 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3704030770 con 0x7f371415ddb0 2024-03-23T11:25:27.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.615+0000 7f3713400640 1 -- 172.21.15.62:0/336104589 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704030ae0 con 0x7f371415ddb0 2024-03-23T11:25:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.615+0000 7f371ba00640 1 -- 172.21.15.62:0/336104589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371415ddb0 msgr2=0x7f3714099550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.615+0000 7f371ba00640 1 --2- 172.21.15.62:0/336104589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371415ddb0 0x7f3714099550 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f37140cb190 tx=0x7f370402b960 comp rx=0 tx=0).stop 2024-03-23T11:25:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.616+0000 7f371ba00640 1 -- 172.21.15.62:0/336104589 shutdown_connections 2024-03-23T11:25:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.616+0000 7f371ba00640 1 --2- 172.21.15.62:0/336104589 >> v2:172.21.15.120:3300/0 conn(0x7f371415e190 0x7f37140ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.616+0000 7f371ba00640 1 --2- 172.21.15.62:0/336104589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371415ddb0 0x7f3714099550 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.616+0000 7f371ba00640 1 -- 172.21.15.62:0/336104589 >> 172.21.15.62:0/336104589 conn(0x7f37140b22c0 msgr2=0x7f37140b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:27.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.616+0000 7f371ba00640 1 -- 172.21.15.62:0/336104589 shutdown_connections 2024-03-23T11:25:27.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.616+0000 7f371ba00640 1 -- 172.21.15.62:0/336104589 wait complete. 2024-03-23T11:25:27.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.617+0000 7f371ba00640 1 Processor -- start 2024-03-23T11:25:27.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.618+0000 7f371ba00640 1 -- start start 2024-03-23T11:25:27.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.618+0000 7f371ba00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f371415ddb0 0x7f371415ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.618+0000 7f371ba00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371415e190 0x7f37140b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.618+0000 7f371ba00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3714156bf0 0x7f3714157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.618+0000 7f371ba00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37140ca2f0 con 0x7f371415e190 2024-03-23T11:25:27.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.618+0000 7f371ba00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37140ae3f0 con 0x7f371415ddb0 2024-03-23T11:25:27.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.618+0000 7f371ba00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37141575c0 con 0x7f3714156bf0 2024-03-23T11:25:27.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.619+0000 7f3713e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371415e190 0x7f37140b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.619+0000 7f3713e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371415e190 0x7f37140b01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49628/0 (socket says 172.21.15.62:49628) 2024-03-23T11:25:27.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.619+0000 7f3713e00640 1 -- 172.21.15.62:0/3664725950 learned_addr learned my addr 172.21.15.62:0/3664725950 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:27.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.619+0000 7f3713e00640 1 -- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3714156bf0 msgr2=0x7f3714157080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.619+0000 7f3713e00640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3714156bf0 0x7f3714157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.619+0000 7f3713e00640 1 -- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f371415ddb0 msgr2=0x7f371415ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.619+0000 7f3713e00640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f371415ddb0 0x7f371415ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.619+0000 7f3713e00640 1 -- 172.21.15.62:0/3664725950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3704009400 con 0x7f371415e190 2024-03-23T11:25:27.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.619+0000 7f3713e00640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371415e190 0x7f37140b01b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f370000c430 tx=0x7f370000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:27.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.620+0000 7f3710c00640 1 -- 172.21.15.62:0/3664725950 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700005ce0 con 0x7f371415e190 2024-03-23T11:25:27.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.620+0000 7f3710c00640 1 -- 172.21.15.62:0/3664725950 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3700005e40 con 0x7f371415e190 2024-03-23T11:25:27.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.620+0000 7f3710c00640 1 -- 172.21.15.62:0/3664725950 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37000072f0 con 0x7f371415e190 2024-03-23T11:25:27.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.620+0000 7f371ba00640 1 -- 172.21.15.62:0/3664725950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37140adac0 con 0x7f371415e190 2024-03-23T11:25:27.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.620+0000 7f371ba00640 1 -- 172.21.15.62:0/3664725950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37140ae080 con 0x7f371415e190 2024-03-23T11:25:27.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.621+0000 7f371ba00640 1 -- 172.21.15.62:0/3664725950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37140738e0 con 0x7f371415e190 2024-03-23T11:25:27.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.622+0000 7f3710c00640 1 -- 172.21.15.62:0/3664725950 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3700007450 con 0x7f371415e190 2024-03-23T11:25:27.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.623+0000 7f3710c00640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36f406a7d0 0x7f36f406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:27.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.623+0000 7f3710c00640 1 -- 172.21.15.62:0/3664725950 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3700088570 con 0x7f371415e190 2024-03-23T11:25:27.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.623+0000 7f3719000640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36f406a7d0 0x7f36f406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:27.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.624+0000 7f3719000640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36f406a7d0 0x7f36f406cc90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3704036000 tx=0x7f370403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:27.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.628+0000 7f3710c00640 1 -- 172.21.15.62:0/3664725950 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3700053860 con 0x7f371415e190 2024-03-23T11:25:27.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.867+0000 7f371ba00640 1 -- 172.21.15.62:0/3664725950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f37140cf3c0 con 0x7f371415e190 2024-03-23T11:25:27.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.868+0000 7f3710c00640 1 -- 172.21.15.62:0/3664725950 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f3700056e80 con 0x7f371415e190 2024-03-23T11:25:27.862 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:25:27.862 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-03-23T11:25:27.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 -- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36f406a7d0 msgr2=0x7f36f406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36f406a7d0 0x7f36f406cc90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3704036000 tx=0x7f370403b040 comp rx=0 tx=0).stop 2024-03-23T11:25:27.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 -- 172.21.15.62:0/3664725950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371415e190 msgr2=0x7f37140b01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:27.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371415e190 0x7f37140b01b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f370000c430 tx=0x7f370000c8f0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 -- 172.21.15.62:0/3664725950 shutdown_connections 2024-03-23T11:25:27.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36f406a7d0 0x7f36f406cc90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f371415e190 0x7f37140b01b0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3714156bf0 0x7f3714157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 --2- 172.21.15.62:0/3664725950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f371415ddb0 0x7f371415ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:27.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 -- 172.21.15.62:0/3664725950 >> 172.21.15.62:0/3664725950 conn(0x7f37140b22c0 msgr2=0x7f37140ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:27.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 -- 172.21.15.62:0/3664725950 shutdown_connections 2024-03-23T11:25:27.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:27.871+0000 7f371ba00640 1 -- 172.21.15.62:0/3664725950 wait complete. 2024-03-23T11:25:27.884 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-23T11:25:27.884 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub start /subdir repair,recursive 2024-03-23T11:25:28.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.114+0000 7f4552a00640 1 Processor -- start 2024-03-23T11:25:28.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.114+0000 7f4552a00640 1 -- start start 2024-03-23T11:25:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.116+0000 7f4552a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f454c154d30 0x7f454c0ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.116+0000 7f4552a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f454c0cebd0 0x7f454c0ab140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.116+0000 7f4552a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c15bcb0 con 0x7f454c0ab680 2024-03-23T11:25:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.116+0000 7f4552a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f454c0cf080 con 0x7f454c0cebd0 2024-03-23T11:25:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.116+0000 7f4552a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f454c15ac90 con 0x7f454c154d30 2024-03-23T11:25:28.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.116+0000 7f4552000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f454c0ab680 0x7f454c0b1160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48440/0 (socket says 172.21.15.62:48440) 2024-03-23T11:25:28.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.116+0000 7f4552000640 1 -- 172.21.15.62:0/785646308 learned_addr learned my addr 172.21.15.62:0/785646308 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:25:28.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.116+0000 7f4550c00640 1 --2- 172.21.15.62:0/785646308 >> v2:172.21.15.62:3300/0 conn(0x7f454c0cebd0 0x7f454c0ab140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.116+0000 7f4551600640 1 --2- 172.21.15.62:0/785646308 >> v2:172.21.15.120:3300/0 conn(0x7f454c154d30 0x7f454c0ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.117+0000 7f454be00640 1 -- 172.21.15.62:0/785646308 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126660122 0 0) 0x7f454c15bcb0 con 0x7f454c0ab680 2024-03-23T11:25:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.117+0000 7f454be00640 1 -- 172.21.15.62:0/785646308 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534003520 con 0x7f454c0ab680 2024-03-23T11:25:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.117+0000 7f4550c00640 1 -- 172.21.15.62:0/785646308 >> v2:172.21.15.120:3300/0 conn(0x7f454c154d30 msgr2=0x7f454c0ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.117+0000 7f4550c00640 1 --2- 172.21.15.62:0/785646308 >> v2:172.21.15.120:3300/0 conn(0x7f454c154d30 0x7f454c0ce690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.117+0000 7f4550c00640 1 -- 172.21.15.62:0/785646308 >> v1:172.21.15.62:6789/0 conn(0x7f454c0ab680 legacy=0x7f454c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.117+0000 7f4550c00640 1 -- 172.21.15.62:0/785646308 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c15af30 con 0x7f454c0cebd0 2024-03-23T11:25:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.118+0000 7f4550c00640 1 --2- 172.21.15.62:0/785646308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f454c0cebd0 0x7f454c0ab140 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f454c0b2a50 tx=0x7f453802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a8d4b2efa7cff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.118+0000 7f454be00640 1 -- 172.21.15.62:0/785646308 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4538031830 con 0x7f454c0cebd0 2024-03-23T11:25:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.118+0000 7f454be00640 1 -- 172.21.15.62:0/785646308 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4538031990 con 0x7f454c0cebd0 2024-03-23T11:25:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.118+0000 7f454be00640 1 -- 172.21.15.62:0/785646308 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4538031d20 con 0x7f454c0cebd0 2024-03-23T11:25:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.119+0000 7f4552a00640 1 -- 172.21.15.62:0/785646308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f454c0cebd0 msgr2=0x7f454c0ab140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.119+0000 7f4552a00640 1 --2- 172.21.15.62:0/785646308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f454c0cebd0 0x7f454c0ab140 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f454c0b2a50 tx=0x7f453802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.119+0000 7f4552a00640 1 -- 172.21.15.62:0/785646308 shutdown_connections 2024-03-23T11:25:28.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.119+0000 7f4552a00640 1 --2- 172.21.15.62:0/785646308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f454c0cebd0 0x7f454c0ab140 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.119+0000 7f4552a00640 1 --2- 172.21.15.62:0/785646308 >> v2:172.21.15.120:3300/0 conn(0x7f454c154d30 0x7f454c0ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.119+0000 7f4552a00640 1 -- 172.21.15.62:0/785646308 >> 172.21.15.62:0/785646308 conn(0x7f454c0b0280 msgr2=0x7f454c0b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.119+0000 7f4552a00640 1 -- 172.21.15.62:0/785646308 shutdown_connections 2024-03-23T11:25:28.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.119+0000 7f4552a00640 1 -- 172.21.15.62:0/785646308 wait complete. 2024-03-23T11:25:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.121+0000 7f4552a00640 1 Processor -- start 2024-03-23T11:25:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.121+0000 7f4552a00640 1 -- start start 2024-03-23T11:25:28.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.122+0000 7f4552a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f454c0ab680 0x7f454c0c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.122+0000 7f4552a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f454c0cebd0 0x7f454c0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.122+0000 7f4552a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f454c154d30 0x7f454c06c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.122+0000 7f4552a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c070ef0 con 0x7f454c0cebd0 2024-03-23T11:25:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.122+0000 7f4552a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c071150 con 0x7f454c154d30 2024-03-23T11:25:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.122+0000 7f4552a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454c071430 con 0x7f454c0ab680 2024-03-23T11:25:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.122+0000 7f4550c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f454c0cebd0 0x7f454c0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.123+0000 7f4550c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f454c0cebd0 0x7f454c0709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49642/0 (socket says 172.21.15.62:49642) 2024-03-23T11:25:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.123+0000 7f4550c00640 1 -- 172.21.15.62:0/1857163462 learned_addr learned my addr 172.21.15.62:0/1857163462 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.123+0000 7f4550c00640 1 -- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f454c0ab680 msgr2=0x7f454c0c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.123+0000 7f4550c00640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f454c0ab680 0x7f454c0c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.123+0000 7f4550c00640 1 -- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f454c154d30 msgr2=0x7f454c06c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.123+0000 7f4552000640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f454c154d30 0x7f454c06c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.123+0000 7f4550c00640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f454c154d30 0x7f454c06c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.123+0000 7f4550c00640 1 -- 172.21.15.62:0/1857163462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4538009400 con 0x7f454c0cebd0 2024-03-23T11:25:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.123+0000 7f4552000640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f454c154d30 0x7f454c06c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.123+0000 7f4550c00640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f454c0cebd0 0x7f454c0709b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f4538009890 tx=0x7f45380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.124+0000 7f4549600640 1 -- 172.21.15.62:0/1857163462 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453803a6f0 con 0x7f454c0cebd0 2024-03-23T11:25:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.124+0000 7f4549600640 1 -- 172.21.15.62:0/1857163462 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453803a870 con 0x7f454c0cebd0 2024-03-23T11:25:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.124+0000 7f4549600640 1 -- 172.21.15.62:0/1857163462 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453803ac20 con 0x7f454c0cebd0 2024-03-23T11:25:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.124+0000 7f4552a00640 1 -- 172.21.15.62:0/1857163462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454c071710 con 0x7f454c0cebd0 2024-03-23T11:25:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.124+0000 7f4552a00640 1 -- 172.21.15.62:0/1857163462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454c071cd0 con 0x7f454c0cebd0 2024-03-23T11:25:28.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.126+0000 7f4549600640 1 -- 172.21.15.62:0/1857163462 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4538043410 con 0x7f454c0cebd0 2024-03-23T11:25:28.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.127+0000 7f4549600640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f452c06a7a0 0x7f452c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.127+0000 7f4549600640 1 -- 172.21.15.62:0/1857163462 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f45380aaa80 con 0x7f454c0cebd0 2024-03-23T11:25:28.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.128+0000 7f4551600640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f452c06a7a0 0x7f452c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.128+0000 7f4556e00740 1 Processor -- start 2024-03-23T11:25:28.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.128+0000 7f4556e00740 1 -- start start 2024-03-23T11:25:28.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.129+0000 7f4551600640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f452c06a7a0 0x7f452c06cc60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4540006fd0 tx=0x7f4540008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.129+0000 7f4556e00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7af37c0 0x562bc7af3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.129+0000 7f4556e00740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7af95e0 0x562bc7af7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.129+0000 7f4556e00740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7b41350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.129+0000 7f4556e00740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc7a631d0 con 0x562bc7b40ed0 2024-03-23T11:25:28.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.129+0000 7f4556e00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc7a60960 con 0x562bc7af37c0 2024-03-23T11:25:28.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.129+0000 7f4556e00740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562bc7a5c250 con 0x562bc7af95e0 2024-03-23T11:25:28.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.130+0000 7f4552000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7b41350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.130+0000 7f4552000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7b41350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49654/0 (socket says 172.21.15.62:49654) 2024-03-23T11:25:28.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.130+0000 7f4552000640 1 -- 172.21.15.62:0/3890825664 learned_addr learned my addr 172.21.15.62:0/3890825664 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.130+0000 7f4551600640 1 --2- 172.21.15.62:0/3890825664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7af37c0 0x562bc7af3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.130+0000 7f4550c00640 1 --2- 172.21.15.62:0/3890825664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7af95e0 0x562bc7af7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.130+0000 7f4552000640 1 -- 172.21.15.62:0/3890825664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7af95e0 msgr2=0x562bc7af7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.130+0000 7f4552000640 1 --2- 172.21.15.62:0/3890825664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7af95e0 0x562bc7af7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.130+0000 7f4552000640 1 -- 172.21.15.62:0/3890825664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7af37c0 msgr2=0x562bc7af3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.130+0000 7f4552000640 1 --2- 172.21.15.62:0/3890825664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7af37c0 0x562bc7af3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.130+0000 7f4552000640 1 -- 172.21.15.62:0/3890825664 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bc79fdeb0 con 0x562bc7b40ed0 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.130+0000 7f4552000640 1 --2- 172.21.15.62:0/3890825664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7b41350 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4544002050 tx=0x7f454400d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef934d71a13fea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.131+0000 7f453e000640 1 -- 172.21.15.62:0/3890825664 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454400b070 con 0x562bc7b40ed0 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.131+0000 7f453e000640 1 -- 172.21.15.62:0/3890825664 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4544013670 con 0x562bc7b40ed0 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.131+0000 7f453e000640 1 -- 172.21.15.62:0/3890825664 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544013a00 con 0x562bc7b40ed0 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.131+0000 7f4556e00740 1 -- 172.21.15.62:0/3890825664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 msgr2=0x562bc7b41350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.131+0000 7f4556e00740 1 --2- 172.21.15.62:0/3890825664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7b41350 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4544002050 tx=0x7f454400d520 comp rx=0 tx=0).stop 2024-03-23T11:25:28.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.131+0000 7f4556e00740 1 -- 172.21.15.62:0/3890825664 shutdown_connections 2024-03-23T11:25:28.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.131+0000 7f4556e00740 1 --2- 172.21.15.62:0/3890825664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7b41350 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.131+0000 7f4556e00740 1 --2- 172.21.15.62:0/3890825664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7af95e0 0x562bc7af7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.131+0000 7f4556e00740 1 --2- 172.21.15.62:0/3890825664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7af37c0 0x562bc7af3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.131+0000 7f4556e00740 1 -- 172.21.15.62:0/3890825664 >> 172.21.15.62:0/3890825664 conn(0x562bc7af5840 msgr2=0x562bc7af7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.132+0000 7f4556e00740 1 -- 172.21.15.62:0/3890825664 shutdown_connections 2024-03-23T11:25:28.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.132+0000 7f4556e00740 1 -- 172.21.15.62:0/3890825664 wait complete. 2024-03-23T11:25:28.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.132+0000 7f4556e00740 1 -- start start 2024-03-23T11:25:28.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.132+0000 7f4556e00740 1 Processor -- start 2024-03-23T11:25:28.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4556e00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7af95e0 0x562bc7b42240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4556e00740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7b40ed0 0x562bc7b378d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4556e00740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b26860 0x562bc7b37e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4556e00740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc7a1a680 con 0x562bc7b26860 2024-03-23T11:25:28.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4556e00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc7a1a320 con 0x562bc7af95e0 2024-03-23T11:25:28.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4556e00740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562bc77c4330 con 0x562bc7b40ed0 2024-03-23T11:25:28.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4556e00740 10 client.0 _ll_register_callbacks cb 0x562bc7a558c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:28.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4552000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b26860 0x562bc7b37e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4552000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b26860 0x562bc7b37e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49668/0 (socket says 172.21.15.62:49668) 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4552000640 1 -- 172.21.15.62:0/4166576669 learned_addr learned my addr 172.21.15.62:0/4166576669 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4552000640 1 -- 172.21.15.62:0/4166576669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7b40ed0 msgr2=0x562bc7b378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4552000640 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7b40ed0 0x562bc7b378d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4552000640 1 -- 172.21.15.62:0/4166576669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7af95e0 msgr2=0x562bc7b42240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4551600640 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7af95e0 0x562bc7b42240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4552000640 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7af95e0 0x562bc7b42240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.134+0000 7f4552000640 1 -- 172.21.15.62:0/4166576669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4534003520 con 0x562bc7b26860 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.135+0000 7f4552000640 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b26860 0x562bc7b37e10 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f4544001580 tx=0x7f4544001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510a40452bdb3cab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.135+0000 7f4532a00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.135+0000 7f4532a00640 1 -- 172.21.15.62:0/4166576669 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544013470 con 0x562bc7b26860 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.135+0000 7f4532a00640 1 -- 172.21.15.62:0/4166576669 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f454400de40 con 0x562bc7b26860 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.135+0000 7f4556e00740 1 -- 172.21.15.62:0/4166576669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562bc7a863f0 con 0x562bc7b26860 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.135+0000 7f4532a00640 1 -- 172.21.15.62:0/4166576669 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45440072e0 con 0x562bc7b26860 2024-03-23T11:25:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.135+0000 7f4556e00740 10 client.7065 fetch_fsmap learned FSMap version 49 2024-03-23T11:25:28.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.135+0000 7f4556e00740 1 -- 172.21.15.62:0/4166576669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x562bc7a5a8d0 con 0x562bc7b26860 2024-03-23T11:25:28.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.136+0000 7f4532a00640 1 -- 172.21.15.62:0/4166576669 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562bc7a863f0 con 0x562bc7b26860 2024-03-23T11:25:28.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.136+0000 7f4532a00640 1 -- 172.21.15.62:0/4166576669 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4544007440 con 0x562bc7b26860 2024-03-23T11:25:28.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.136+0000 7f4556e00740 10 client.7065 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:25:28.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.136+0000 7f4556e00740 10 client.7065 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:25:28.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.136+0000 7f4556e00740 20 client.7065 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:28.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.136+0000 7f4556e00740 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562bc7b05ab0 0x562bc7b05ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.136+0000 7f4556e00740 4 client.7065 mds_command: new command op to 6618 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:25:28.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.136+0000 7f4556e00740 1 -- 172.21.15.62:0/4166576669 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562bc7a4e970 con 0x562bc7b05ab0 2024-03-23T11:25:28.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.137+0000 7f4551600640 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562bc7b05ab0 0x562bc7b05ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.137+0000 7f4551600640 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562bc7b05ab0 0x562bc7b05ec0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4980d416abc623ff server_cookie=68d7899919b8c503 in_seq=0 out_seq=0 2024-03-23T11:25:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.137+0000 7f4532a00640 10 client.7065 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4532a00640 1 -- 172.21.15.62:0/4166576669 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x562bc7a4e970 con 0x562bc7b05ab0 2024-03-23T11:25:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4532a00640 10 client.7065 handle_command_reply: tid=0 2024-03-23T11:25:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4556e00740 1 client.7065 shutdown 2024-03-23T11:25:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4556e00740 1 -- 172.21.15.62:0/4166576669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b26860 msgr2=0x562bc7b37e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4556e00740 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b26860 0x562bc7b37e10 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f4544001580 tx=0x7f4544001250 comp rx=0 tx=0).stop 2024-03-23T11:25:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4556e00740 1 -- 172.21.15.62:0/4166576669 shutdown_connections 2024-03-23T11:25:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4556e00740 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562bc7b05ab0 0x562bc7b05ec0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4556e00740 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b26860 0x562bc7b37e10 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4556e00740 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7b40ed0 0x562bc7b378d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4556e00740 1 --2- 172.21.15.62:0/4166576669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7af95e0 0x562bc7b42240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4532a00640 0 client.7065 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:28.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4556e00740 1 -- 172.21.15.62:0/4166576669 >> 172.21.15.62:0/4166576669 conn(0x562bc7af37c0 msgr2=0x562bc7af2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.139+0000 7f4556e00740 1 -- 172.21.15.62:0/4166576669 shutdown_connections 2024-03-23T11:25:28.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.140+0000 7f4556e00740 1 -- 172.21.15.62:0/4166576669 wait complete. 2024-03-23T11:25:28.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.140+0000 7f4556e00740 20 client.7065 trim_cache size 0 max 16384 2024-03-23T11:25:28.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.156+0000 7f4556e00740 1 Processor -- start 2024-03-23T11:25:28.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.156+0000 7f4556e00740 1 -- start start 2024-03-23T11:25:28.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.156+0000 7f4556e00740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7af95e0 0x562bc7b60bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.156+0000 7f4556e00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7b26860 0x562bc7b24770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.156+0000 7f4556e00740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7af3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.156+0000 7f4556e00740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc7b18ca0 con 0x562bc7b40ed0 2024-03-23T11:25:28.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.156+0000 7f4556e00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc7b1f720 con 0x562bc7b26860 2024-03-23T11:25:28.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.156+0000 7f4556e00740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562bc7afa230 con 0x562bc7af95e0 2024-03-23T11:25:28.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.157+0000 7f4552000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7af3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.157+0000 7f4552000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7af3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49676/0 (socket says 172.21.15.62:49676) 2024-03-23T11:25:28.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.157+0000 7f4552000640 1 -- 172.21.15.62:0/206101491 learned_addr learned my addr 172.21.15.62:0/206101491 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:28.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.157+0000 7f4552000640 1 -- 172.21.15.62:0/206101491 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7af95e0 msgr2=0x562bc7b60bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.157+0000 7f4552000640 1 --2- 172.21.15.62:0/206101491 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7af95e0 0x562bc7b60bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.157+0000 7f4552000640 1 -- 172.21.15.62:0/206101491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7b26860 msgr2=0x562bc7b24770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.157+0000 7f4552000640 1 --2- 172.21.15.62:0/206101491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7b26860 0x562bc7b24770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.157+0000 7f4552000640 1 -- 172.21.15.62:0/206101491 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bc79fc6f0 con 0x562bc7b40ed0 2024-03-23T11:25:28.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.157+0000 7f4552000640 1 --2- 172.21.15.62:0/206101491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7af3df0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f4544000f80 tx=0x7f4544019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62274283f664316f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f453ea00640 1 -- 172.21.15.62:0/206101491 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544013e10 con 0x562bc7b40ed0 2024-03-23T11:25:28.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f453ea00640 1 -- 172.21.15.62:0/206101491 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4544007900 con 0x562bc7b40ed0 2024-03-23T11:25:28.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f453ea00640 1 -- 172.21.15.62:0/206101491 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45440072e0 con 0x562bc7b40ed0 2024-03-23T11:25:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f4556e00740 1 -- 172.21.15.62:0/206101491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 msgr2=0x562bc7af3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f4556e00740 1 --2- 172.21.15.62:0/206101491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7af3df0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f4544000f80 tx=0x7f4544019000 comp rx=0 tx=0).stop 2024-03-23T11:25:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f4556e00740 1 -- 172.21.15.62:0/206101491 shutdown_connections 2024-03-23T11:25:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f4556e00740 1 --2- 172.21.15.62:0/206101491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7b40ed0 0x562bc7af3df0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f4556e00740 1 --2- 172.21.15.62:0/206101491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7b26860 0x562bc7b24770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f4556e00740 1 --2- 172.21.15.62:0/206101491 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7af95e0 0x562bc7b60bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f4556e00740 1 -- 172.21.15.62:0/206101491 >> 172.21.15.62:0/206101491 conn(0x562bc7af37c0 msgr2=0x562bc7af7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f4556e00740 1 -- 172.21.15.62:0/206101491 shutdown_connections 2024-03-23T11:25:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.158+0000 7f4556e00740 1 -- 172.21.15.62:0/206101491 wait complete. 2024-03-23T11:25:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.159+0000 7f4556e00740 1 -- start start 2024-03-23T11:25:28.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.159+0000 7f4556e00740 1 Processor -- start 2024-03-23T11:25:28.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4556e00740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7af95e0 0x562bc7b8a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4556e00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7b26860 0x562bc7b17a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4556e00740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7b40ed0 0x562bc7b17f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4556e00740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc7ba4e80 con 0x562bc7af95e0 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4556e00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc7b448e0 con 0x562bc7b26860 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4556e00740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562bc7b4e1a0 con 0x562bc7b40ed0 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4556e00740 10 client.0 _ll_register_callbacks cb 0x7f4544001a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4551600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7af95e0 0x562bc7b8a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4551600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7af95e0 0x562bc7b8a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49690/0 (socket says 172.21.15.62:49690) 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4551600640 1 -- 172.21.15.62:0/4152067787 learned_addr learned my addr 172.21.15.62:0/4152067787 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4551600640 1 -- 172.21.15.62:0/4152067787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7b40ed0 msgr2=0x562bc7b17f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4551600640 1 --2- 172.21.15.62:0/4152067787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7b40ed0 0x562bc7b17f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4551600640 1 -- 172.21.15.62:0/4152067787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7b26860 msgr2=0x562bc7b17a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4551600640 1 --2- 172.21.15.62:0/4152067787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7b26860 0x562bc7b17a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.161+0000 7f4551600640 1 -- 172.21.15.62:0/4152067787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562bc7a5a8d0 con 0x562bc7af95e0 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.162+0000 7f4551600640 1 --2- 172.21.15.62:0/4152067787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7af95e0 0x562bc7b8a330 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f4540025b70 tx=0x7f4540025f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7edae2f0ad9d490 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.162+0000 7f4532a00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.162+0000 7f4532a00640 1 -- 172.21.15.62:0/4152067787 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540025300 con 0x562bc7af95e0 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.162+0000 7f4532a00640 1 -- 172.21.15.62:0/4152067787 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f454002a030 con 0x562bc7af95e0 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.162+0000 7f4556e00740 1 -- 172.21.15.62:0/4152067787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562bc7afa510 con 0x562bc7af95e0 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.162+0000 7f4532a00640 1 -- 172.21.15.62:0/4152067787 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4540023450 con 0x562bc7af95e0 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.162+0000 7f4556e00740 10 client.7071 fetch_fsmap learned FSMap version 49 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.162+0000 7f4556e00740 1 -- 172.21.15.62:0/4152067787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x562bc7a21cc0 con 0x562bc7af95e0 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.162+0000 7f4532a00640 1 -- 172.21.15.62:0/4152067787 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562bc7afa510 con 0x562bc7af95e0 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.163+0000 7f4532a00640 1 -- 172.21.15.62:0/4152067787 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f45400235b0 con 0x562bc7af95e0 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.163+0000 7f4556e00740 10 client.7071 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.163+0000 7f4556e00740 10 client.7071 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:25:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.163+0000 7f4556e00740 20 client.7071 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:28.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.163+0000 7f4556e00740 1 --2- 172.21.15.62:0/4152067787 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562bc7b06ed0 0x562bc7b23540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.163+0000 7f4556e00740 4 client.7071 mds_command: new command op to 6618 tid=0[{"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}] 2024-03-23T11:25:28.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.163+0000 7f4556e00740 1 -- 172.21.15.62:0/4152067787 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}) v1 -- 0x7f4544013670 con 0x562bc7b06ed0 2024-03-23T11:25:28.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.163+0000 7f4550c00640 1 --2- 172.21.15.62:0/4152067787 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562bc7b06ed0 0x562bc7b23540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.164+0000 7f4550c00640 1 --2- 172.21.15.62:0/4152067787 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562bc7b06ed0 0x562bc7b23540 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=367a096532166305 server_cookie=6c88b7bb558bc646 in_seq=0 out_seq=0 2024-03-23T11:25:28.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.164+0000 7f4532a00640 10 client.7071 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4532a00640 1 -- 172.21.15.62:0/4152067787 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7f4544013670 con 0x562bc7b06ed0 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4532a00640 10 client.7071 handle_command_reply: tid=0 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4556e00740 1 client.7071 shutdown 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4556e00740 1 -- 172.21.15.62:0/4152067787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7af95e0 msgr2=0x562bc7b8a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4556e00740 1 --2- 172.21.15.62:0/4152067787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7af95e0 0x562bc7b8a330 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f4540025b70 tx=0x7f4540025f70 comp rx=0 tx=0).stop 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4556e00740 1 -- 172.21.15.62:0/4152067787 shutdown_connections 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4556e00740 1 --2- 172.21.15.62:0/4152067787 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562bc7b06ed0 0x562bc7b23540 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4556e00740 1 --2- 172.21.15.62:0/4152067787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562bc7b40ed0 0x562bc7b17f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4556e00740 1 --2- 172.21.15.62:0/4152067787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bc7b26860 0x562bc7b17a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4556e00740 1 --2- 172.21.15.62:0/4152067787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562bc7af95e0 0x562bc7b8a330 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4556e00740 1 -- 172.21.15.62:0/4152067787 >> 172.21.15.62:0/4152067787 conn(0x562bc7af37c0 msgr2=0x562bc7b3ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.166+0000 7f4532a00640 0 client.7071 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4556e00740 1 -- 172.21.15.62:0/4152067787 shutdown_connections 2024-03-23T11:25:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4556e00740 1 -- 172.21.15.62:0/4152067787 wait complete. 2024-03-23T11:25:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4556e00740 20 client.7071 trim_cache size 0 max 16384 2024-03-23T11:25:28.162 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:25:28.162 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-03-23T11:25:28.162 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "56711550-dd1b-42c1-96c4-6d2c40bbc197", 2024-03-23T11:25:28.162 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-03-23T11:25:28.162 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:25:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4552a00640 1 -- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f452c06a7a0 msgr2=0x7f452c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4552a00640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f452c06a7a0 0x7f452c06cc60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4540006fd0 tx=0x7f4540008040 comp rx=0 tx=0).stop 2024-03-23T11:25:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4552a00640 1 -- 172.21.15.62:0/1857163462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f454c0cebd0 msgr2=0x7f454c0709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4552a00640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f454c0cebd0 0x7f454c0709b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f4538009890 tx=0x7f45380046c0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4552a00640 1 -- 172.21.15.62:0/1857163462 shutdown_connections 2024-03-23T11:25:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4552a00640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f452c06a7a0 0x7f452c06cc60 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4552a00640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f454c0cebd0 0x7f454c0709b0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4552a00640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f454c154d30 0x7f454c06c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4552a00640 1 --2- 172.21.15.62:0/1857163462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f454c0ab680 0x7f454c0c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.167+0000 7f4552a00640 1 -- 172.21.15.62:0/1857163462 >> 172.21.15.62:0/1857163462 conn(0x7f454c0b0280 msgr2=0x7f454c0addd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.168+0000 7f4552a00640 1 -- 172.21.15.62:0/1857163462 shutdown_connections 2024-03-23T11:25:28.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.168+0000 7f4552a00640 1 -- 172.21.15.62:0/1857163462 wait complete. 2024-03-23T11:25:28.181 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-23T11:25:28.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.401+0000 7f2fffe00640 1 Processor -- start 2024-03-23T11:25:28.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.401+0000 7f2fffe00640 1 -- start start 2024-03-23T11:25:28.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.403+0000 7f2fffe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f300015fd10 0x7f30000aba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.403+0000 7f2fffe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3000071680 0x7f30000b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.403+0000 7f2fffe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300006d320 con 0x7f30000b2530 2024-03-23T11:25:28.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.403+0000 7f2fffe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300006c710 con 0x7f3000071680 2024-03-23T11:25:28.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.403+0000 7f2fffe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300006c9f0 con 0x7f300015fd10 2024-03-23T11:25:28.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.404+0000 7f2fff400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f30000b2530 0x7f300006d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48452/0 (socket says 172.21.15.62:48452) 2024-03-23T11:25:28.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.404+0000 7f2fff400640 1 -- 172.21.15.62:0/2183287357 learned_addr learned my addr 172.21.15.62:0/2183287357 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:25:28.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.404+0000 7f2ffe000640 1 --2- 172.21.15.62:0/2183287357 >> v2:172.21.15.62:3300/0 conn(0x7f3000071680 0x7f30000b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.404+0000 7f2ffea00640 1 --2- 172.21.15.62:0/2183287357 >> v2:172.21.15.120:3300/0 conn(0x7f300015fd10 0x7f30000aba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.404+0000 7f2ffe000640 1 -- 172.21.15.62:0/2183287357 >> v2:172.21.15.120:3300/0 conn(0x7f300015fd10 msgr2=0x7f30000aba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.404+0000 7f2ffe000640 1 --2- 172.21.15.62:0/2183287357 >> v2:172.21.15.120:3300/0 conn(0x7f300015fd10 0x7f30000aba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.404+0000 7f2ffe000640 1 -- 172.21.15.62:0/2183287357 >> v1:172.21.15.62:6789/0 conn(0x7f30000b2530 legacy=0x7f300006d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.404+0000 7f2ffe000640 1 -- 172.21.15.62:0/2183287357 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300006ccd0 con 0x7f3000071680 2024-03-23T11:25:28.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.405+0000 7f2ffe000640 1 --2- 172.21.15.62:0/2183287357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3000071680 0x7f30000b1ab0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f30000b4d20 tx=0x7f2fe802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f96f49a6299c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.406+0000 7f2ffd600640 1 -- 172.21.15.62:0/2183287357 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8031830 con 0x7f3000071680 2024-03-23T11:25:28.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.406+0000 7f2ffd600640 1 -- 172.21.15.62:0/2183287357 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe8031990 con 0x7f3000071680 2024-03-23T11:25:28.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.406+0000 7f2ffd600640 1 -- 172.21.15.62:0/2183287357 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8031d20 con 0x7f3000071680 2024-03-23T11:25:28.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.406+0000 7f2fffe00640 1 -- 172.21.15.62:0/2183287357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3000071680 msgr2=0x7f30000b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.406+0000 7f2fffe00640 1 --2- 172.21.15.62:0/2183287357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3000071680 0x7f30000b1ab0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f30000b4d20 tx=0x7f2fe802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.406+0000 7f2fffe00640 1 -- 172.21.15.62:0/2183287357 shutdown_connections 2024-03-23T11:25:28.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.406+0000 7f2fffe00640 1 --2- 172.21.15.62:0/2183287357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3000071680 0x7f30000b1ab0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.406+0000 7f2fffe00640 1 --2- 172.21.15.62:0/2183287357 >> v2:172.21.15.120:3300/0 conn(0x7f300015fd10 0x7f30000aba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.406+0000 7f2fffe00640 1 -- 172.21.15.62:0/2183287357 >> 172.21.15.62:0/2183287357 conn(0x7f3000070810 msgr2=0x7f3000070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.407+0000 7f2fffe00640 1 -- 172.21.15.62:0/2183287357 shutdown_connections 2024-03-23T11:25:28.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.407+0000 7f2fffe00640 1 -- 172.21.15.62:0/2183287357 wait complete. 2024-03-23T11:25:28.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.408+0000 7f2fffe00640 1 Processor -- start 2024-03-23T11:25:28.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.408+0000 7f2fffe00640 1 -- start start 2024-03-23T11:25:28.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.409+0000 7f2fffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3000071680 0x7f300015dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.409+0000 7f2fffe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30000b2530 0x7f3000156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.409+0000 7f2fffe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f300015fd10 0x7f3000145180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.409+0000 7f2fffe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300015e850 con 0x7f3000071680 2024-03-23T11:25:28.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.409+0000 7f2fffe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30001541b0 con 0x7f300015fd10 2024-03-23T11:25:28.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.409+0000 7f2fffe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3000153ba0 con 0x7f30000b2530 2024-03-23T11:25:28.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2ffea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3000071680 0x7f300015dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2ffea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3000071680 0x7f300015dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49714/0 (socket says 172.21.15.62:49714) 2024-03-23T11:25:28.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2ffea00640 1 -- 172.21.15.62:0/540850248 learned_addr learned my addr 172.21.15.62:0/540850248 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2ffe000640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30000b2530 0x7f3000156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2fff400640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f300015fd10 0x7f3000145180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2ffea00640 1 -- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30000b2530 msgr2=0x7f3000156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2ffea00640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30000b2530 0x7f3000156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2ffea00640 1 -- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f300015fd10 msgr2=0x7f3000145180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2ffea00640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f300015fd10 0x7f3000145180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2ffea00640 1 -- 172.21.15.62:0/540850248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe8009400 con 0x7f3000071680 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2ffe000640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30000b2530 0x7f3000156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.410+0000 7f2fff400640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f300015fd10 0x7f3000145180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.411+0000 7f2ffea00640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3000071680 0x7f300015dc70 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000d5f0 tx=0x7f2ff000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.411+0000 7f2ff6a00640 1 -- 172.21.15.62:0/540850248 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff00125f0 con 0x7f3000071680 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.411+0000 7f2ff6a00640 1 -- 172.21.15.62:0/540850248 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff0012750 con 0x7f3000071680 2024-03-23T11:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.411+0000 7f2fffe00640 1 -- 172.21.15.62:0/540850248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300015e570 con 0x7f3000071680 2024-03-23T11:25:28.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.411+0000 7f2ff6a00640 1 -- 172.21.15.62:0/540850248 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0012b00 con 0x7f3000071680 2024-03-23T11:25:28.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.411+0000 7f2fffe00640 1 -- 172.21.15.62:0/540850248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300015e1b0 con 0x7f3000071680 2024-03-23T11:25:28.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.414+0000 7f2ff6a00640 1 -- 172.21.15.62:0/540850248 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2ff0012c60 con 0x7f3000071680 2024-03-23T11:25:28.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.415+0000 7f2ff6a00640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fdc06a7a0 0x7f2fdc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.415+0000 7f2ff6a00640 1 -- 172.21.15.62:0/540850248 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2ff0089380 con 0x7f3000071680 2024-03-23T11:25:28.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.416+0000 7f2ffe000640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fdc06a7a0 0x7f2fdc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.416+0000 7f300878b740 1 Processor -- start 2024-03-23T11:25:28.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.416+0000 7f300878b740 1 -- start start 2024-03-23T11:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.417+0000 7f300878b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead15d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.417+0000 7f300878b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562deacedc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.417+0000 7f2ffe000640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fdc06a7a0 0x7f2fdc06cc60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8036000 tx=0x7f2fe8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.417+0000 7f300878b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 0x562deacee160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.417+0000 7f300878b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562deaca9cb0 con 0x562dead0daf0 2024-03-23T11:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.417+0000 7f300878b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562deac58500 con 0x562dead16460 2024-03-23T11:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.417+0000 7f300878b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562deac55c90 con 0x562dead0ded0 2024-03-23T11:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.417+0000 7f2ffea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead15d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.417+0000 7f2ffea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead15d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49728/0 (socket says 172.21.15.62:49728) 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.417+0000 7f2ffea00640 1 -- 172.21.15.62:0/3043691334 learned_addr learned my addr 172.21.15.62:0/3043691334 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.418+0000 7f2ffea00640 1 -- 172.21.15.62:0/3043691334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 msgr2=0x562deacee160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.418+0000 7f2ffea00640 1 --2- 172.21.15.62:0/3043691334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 0x562deacee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.418+0000 7f2ffe000640 1 --2- 172.21.15.62:0/3043691334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562deacedc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.418+0000 7f2ffea00640 1 -- 172.21.15.62:0/3043691334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 msgr2=0x562deacedc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.418+0000 7f2ffea00640 1 --2- 172.21.15.62:0/3043691334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562deacedc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.418+0000 7f2ffea00640 1 -- 172.21.15.62:0/3043691334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562deac03360 con 0x562dead0daf0 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.418+0000 7f2ffea00640 1 --2- 172.21.15.62:0/3043691334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead15d60 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00582c0 tx=0x7f2ff0058230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca04058a7402d93e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.418+0000 7f2fef400640 1 -- 172.21.15.62:0/3043691334 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff005c160 con 0x562dead0daf0 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.418+0000 7f2fef400640 1 -- 172.21.15.62:0/3043691334 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff0060720 con 0x562dead0daf0 2024-03-23T11:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.418+0000 7f2fef400640 1 -- 172.21.15.62:0/3043691334 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0089070 con 0x562dead0daf0 2024-03-23T11:25:28.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.419+0000 7f300878b740 1 -- 172.21.15.62:0/3043691334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 msgr2=0x562dead15d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.419+0000 7f300878b740 1 --2- 172.21.15.62:0/3043691334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead15d60 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00582c0 tx=0x7f2ff0058230 comp rx=0 tx=0).stop 2024-03-23T11:25:28.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.419+0000 7f300878b740 1 -- 172.21.15.62:0/3043691334 shutdown_connections 2024-03-23T11:25:28.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.419+0000 7f300878b740 1 --2- 172.21.15.62:0/3043691334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 0x562deacee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.419+0000 7f300878b740 1 --2- 172.21.15.62:0/3043691334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562deacedc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.419+0000 7f300878b740 1 --2- 172.21.15.62:0/3043691334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead15d60 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.419+0000 7f300878b740 1 -- 172.21.15.62:0/3043691334 >> 172.21.15.62:0/3043691334 conn(0x562deacb5b40 msgr2=0x562dead33900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.419+0000 7f300878b740 1 -- 172.21.15.62:0/3043691334 shutdown_connections 2024-03-23T11:25:28.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.419+0000 7f300878b740 1 -- 172.21.15.62:0/3043691334 wait complete. 2024-03-23T11:25:28.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.420+0000 7f300878b740 1 -- start start 2024-03-23T11:25:28.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.420+0000 7f300878b740 1 Processor -- start 2024-03-23T11:25:28.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.421+0000 7f300878b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.421+0000 7f300878b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 0x562dead43130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.421+0000 7f300878b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562dead1b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.421+0000 7f300878b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562deac51580 con 0x562dead0daf0 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.421+0000 7f300878b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562deabf2a50 con 0x562dead16460 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.421+0000 7f300878b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562deabf26f0 con 0x562dead0ded0 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.421+0000 7f300878b740 10 client.0 _ll_register_callbacks cb 0x562deac41ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.421+0000 7f2ffea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.421+0000 7f2ffea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49734/0 (socket says 172.21.15.62:49734) 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.421+0000 7f2ffea00640 1 -- 172.21.15.62:0/4294141247 learned_addr learned my addr 172.21.15.62:0/4294141247 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f2ffe000640 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 0x562dead43130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f2fff400640 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562dead1b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f2ffea00640 1 -- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 msgr2=0x562dead43130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f2ffea00640 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 0x562dead43130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f2ffea00640 1 -- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 msgr2=0x562dead1b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f2ffea00640 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562dead1b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f2ffea00640 1 -- 172.21.15.62:0/4294141247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300015e1b0 con 0x562dead0daf0 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f2ffea00640 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0c4d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000d5c0 tx=0x7f2ff0006f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f793dcae3ac88ae server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f2fe3e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f2fe3e00640 1 -- 172.21.15.62:0/4294141247 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0089070 con 0x562dead0daf0 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f2fe3e00640 1 -- 172.21.15.62:0/4294141247 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2ff0054a40 con 0x562dead0daf0 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.422+0000 7f300878b740 1 -- 172.21.15.62:0/4294141247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562deac7bfe0 con 0x562dead0daf0 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.423+0000 7f2fe3e00640 1 -- 172.21.15.62:0/4294141247 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ff0054c00 con 0x562dead0daf0 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.423+0000 7f300878b740 10 client.7086 fetch_fsmap learned FSMap version 49 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.423+0000 7f300878b740 1 -- 172.21.15.62:0/4294141247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x562deac4fc00 con 0x562dead0daf0 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.423+0000 7f2fe3e00640 1 -- 172.21.15.62:0/4294141247 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562deac7bfe0 con 0x562dead0daf0 2024-03-23T11:25:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.423+0000 7f2fe3e00640 1 -- 172.21.15.62:0/4294141247 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2ff006f070 con 0x562dead0daf0 2024-03-23T11:25:28.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.423+0000 7f300878b740 10 client.7086 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:25:28.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.423+0000 7f300878b740 10 client.7086 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:25:28.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.423+0000 7f300878b740 20 client.7086 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:28.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.423+0000 7f300878b740 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562dead0d0d0 0x562dead327c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.423+0000 7f300878b740 4 client.7086 mds_command: new command op to 6618 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:25:28.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.423+0000 7f300878b740 1 -- 172.21.15.62:0/4294141247 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562deac44240 con 0x562dead0d0d0 2024-03-23T11:25:28.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.424+0000 7f2ffe000640 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562dead0d0d0 0x562dead327c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.425+0000 7f2ffe000640 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562dead0d0d0 0x562dead327c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb48d59c68909cf server_cookie=c62bafdd23a248a5 in_seq=0 out_seq=0 2024-03-23T11:25:28.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.425+0000 7f2fe3e00640 10 client.7086 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:28.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.426+0000 7f2fe3e00640 1 -- 172.21.15.62:0/4294141247 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x562deac44240 con 0x562dead0d0d0 2024-03-23T11:25:28.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.426+0000 7f2fe3e00640 10 client.7086 handle_command_reply: tid=0 2024-03-23T11:25:28.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.426+0000 7f300878b740 1 client.7086 shutdown 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.426+0000 7f300878b740 1 -- 172.21.15.62:0/4294141247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 msgr2=0x562dead0c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.426+0000 7f300878b740 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0c4d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000d5c0 tx=0x7f2ff0006f30 comp rx=0 tx=0).stop 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.427+0000 7f300878b740 1 -- 172.21.15.62:0/4294141247 shutdown_connections 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.427+0000 7f300878b740 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562dead0d0d0 0x562dead327c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.427+0000 7f300878b740 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562dead1b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.427+0000 7f300878b740 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 0x562dead43130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.427+0000 7f300878b740 1 --2- 172.21.15.62:0/4294141247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0c4d0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.427+0000 7f300878b740 1 -- 172.21.15.62:0/4294141247 >> 172.21.15.62:0/4294141247 conn(0x562deacb5b40 msgr2=0x562deaced9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.427+0000 7f2fe3e00640 0 client.7086 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.427+0000 7f300878b740 1 -- 172.21.15.62:0/4294141247 shutdown_connections 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.427+0000 7f300878b740 1 -- 172.21.15.62:0/4294141247 wait complete. 2024-03-23T11:25:28.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.427+0000 7f300878b740 20 client.7086 trim_cache size 0 max 16384 2024-03-23T11:25:28.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.443+0000 7f300878b740 1 Processor -- start 2024-03-23T11:25:28.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.443+0000 7f300878b740 1 -- start start 2024-03-23T11:25:28.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.443+0000 7f300878b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.443+0000 7f300878b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 0x562dead19120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.443+0000 7f300878b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562dead1b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f300878b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dea94ef90 con 0x562dead0daf0 2024-03-23T11:25:28.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f300878b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dead4d0c0 con 0x562dead16460 2024-03-23T11:25:28.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f300878b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562dead14d80 con 0x562dead0ded0 2024-03-23T11:25:28.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f2ffea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f2ffea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49740/0 (socket says 172.21.15.62:49740) 2024-03-23T11:25:28.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f2ffea00640 1 -- 172.21.15.62:0/908713919 learned_addr learned my addr 172.21.15.62:0/908713919 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:28.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f2ffea00640 1 -- 172.21.15.62:0/908713919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 msgr2=0x562dead19120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f2ffea00640 1 --2- 172.21.15.62:0/908713919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 0x562dead19120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f2ffea00640 1 -- 172.21.15.62:0/908713919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 msgr2=0x562dead1b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f2ffea00640 1 --2- 172.21.15.62:0/908713919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562dead1b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f2ffea00640 1 -- 172.21.15.62:0/908713919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562deabf1c60 con 0x562dead0daf0 2024-03-23T11:25:28.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.444+0000 7f2ffea00640 1 --2- 172.21.15.62:0/908713919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0bb40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0012f60 tx=0x7f2ff0086000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e13a09ebcdf320f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f2fefe00640 1 -- 172.21.15.62:0/908713919 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff005c160 con 0x562dead0daf0 2024-03-23T11:25:28.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f2fefe00640 1 -- 172.21.15.62:0/908713919 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff0058a30 con 0x562dead0daf0 2024-03-23T11:25:28.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f2fefe00640 1 -- 172.21.15.62:0/908713919 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0087330 con 0x562dead0daf0 2024-03-23T11:25:28.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f300878b740 1 -- 172.21.15.62:0/908713919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 msgr2=0x562dead0bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f300878b740 1 --2- 172.21.15.62:0/908713919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0bb40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0012f60 tx=0x7f2ff0086000 comp rx=0 tx=0).stop 2024-03-23T11:25:28.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f300878b740 1 -- 172.21.15.62:0/908713919 shutdown_connections 2024-03-23T11:25:28.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f300878b740 1 --2- 172.21.15.62:0/908713919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562dead1b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f300878b740 1 --2- 172.21.15.62:0/908713919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0ded0 0x562dead19120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f300878b740 1 --2- 172.21.15.62:0/908713919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0daf0 0x562dead0bb40 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f300878b740 1 -- 172.21.15.62:0/908713919 >> 172.21.15.62:0/908713919 conn(0x562deacb5b40 msgr2=0x562deacee690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f300878b740 1 -- 172.21.15.62:0/908713919 shutdown_connections 2024-03-23T11:25:28.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.445+0000 7f300878b740 1 -- 172.21.15.62:0/908713919 wait complete. 2024-03-23T11:25:28.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.446+0000 7f300878b740 1 -- start start 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.446+0000 7f300878b740 1 Processor -- start 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.447+0000 7f300878b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0daf0 0x562dead424c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.447+0000 7f300878b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0ded0 0x562dead98e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.447+0000 7f300878b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562dead9a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.447+0000 7f300878b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dead0bf50 con 0x562dead0ded0 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.447+0000 7f300878b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dead9ad20 con 0x562dead16460 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.447+0000 7f300878b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562dead4d5a0 con 0x562dead0daf0 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.447+0000 7f300878b740 10 client.0 _ll_register_callbacks cb 0x7f2ff005e040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.447+0000 7f2ffe000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0ded0 0x562dead98e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.447+0000 7f2ffe000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0ded0 0x562dead98e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49744/0 (socket says 172.21.15.62:49744) 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.448+0000 7f2ffe000640 1 -- 172.21.15.62:0/1704030444 learned_addr learned my addr 172.21.15.62:0/1704030444 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.448+0000 7f2ffe000640 1 -- 172.21.15.62:0/1704030444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0daf0 msgr2=0x562dead424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.448+0000 7f2ffe000640 1 --2- 172.21.15.62:0/1704030444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0daf0 0x562dead424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.448+0000 7f2ffe000640 1 -- 172.21.15.62:0/1704030444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 msgr2=0x562dead9a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:28.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.448+0000 7f2ffe000640 1 --2- 172.21.15.62:0/1704030444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562dead9a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.448+0000 7f2ffe000640 1 -- 172.21.15.62:0/1704030444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562deac4fc00 con 0x562dead0ded0 2024-03-23T11:25:28.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.448+0000 7f2ffe000640 1 --2- 172.21.15.62:0/1704030444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0ded0 0x562dead98e80 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2fe802bee0 tx=0x7f2fe80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78425261b355b9ee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:28.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.448+0000 7f2fe3e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.448+0000 7f2fe3e00640 1 -- 172.21.15.62:0/1704030444 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe803f070 con 0x562dead0ded0 2024-03-23T11:25:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.448+0000 7f2fe3e00640 1 -- 172.21.15.62:0/1704030444 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2fe8004410 con 0x562dead0ded0 2024-03-23T11:25:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.448+0000 7f300878b740 1 -- 172.21.15.62:0/1704030444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562dead14af0 con 0x562dead0ded0 2024-03-23T11:25:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.449+0000 7f2fe3e00640 1 -- 172.21.15.62:0/1704030444 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fe8050790 con 0x562dead0ded0 2024-03-23T11:25:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.449+0000 7f300878b740 10 client.7092 fetch_fsmap learned FSMap version 49 2024-03-23T11:25:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.449+0000 7f300878b740 1 -- 172.21.15.62:0/1704030444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x562deac17170 con 0x562dead0ded0 2024-03-23T11:25:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.449+0000 7f2fe3e00640 1 -- 172.21.15.62:0/1704030444 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562dead14af0 con 0x562dead0ded0 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.449+0000 7f2fe3e00640 1 -- 172.21.15.62:0/1704030444 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2fe80508f0 con 0x562dead0ded0 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.449+0000 7f300878b740 10 client.7092 fetch_fsmap finished waiting for FSMap version 49 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.449+0000 7f300878b740 10 client.7092 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.449+0000 7f300878b740 20 client.7092 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.449+0000 7f300878b740 1 --2- 172.21.15.62:0/1704030444 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562dead1b8a0 0x562dead16840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.449+0000 7f300878b740 4 client.7092 mds_command: new command op to 6618 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.449+0000 7f300878b740 1 -- 172.21.15.62:0/1704030444 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f2ff0060720 con 0x562dead1b8a0 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.450+0000 7f2ffea00640 1 --2- 172.21.15.62:0/1704030444 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562dead1b8a0 0x562dead16840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.451+0000 7f2ffea00640 1 --2- 172.21.15.62:0/1704030444 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562dead1b8a0 0x562dead16840 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8bc0c06808587a6 server_cookie=352869c90bd128ed in_seq=0 out_seq=0 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.451+0000 7f2fe3e00640 10 client.7092 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.451+0000 7f2fe3e00640 1 -- 172.21.15.62:0/1704030444 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x7f2ff0060720 con 0x562dead1b8a0 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.451+0000 7f2fe3e00640 10 client.7092 handle_command_reply: tid=0 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.451+0000 7f300878b740 1 client.7092 shutdown 2024-03-23T11:25:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.451+0000 7f300878b740 1 -- 172.21.15.62:0/1704030444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0ded0 msgr2=0x562dead98e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.451+0000 7f300878b740 1 --2- 172.21.15.62:0/1704030444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0ded0 0x562dead98e80 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2fe802bee0 tx=0x7f2fe80046c0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f300878b740 1 -- 172.21.15.62:0/1704030444 shutdown_connections 2024-03-23T11:25:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f300878b740 1 --2- 172.21.15.62:0/1704030444 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562dead1b8a0 0x562dead16840 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f300878b740 1 --2- 172.21.15.62:0/1704030444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562dead0ded0 0x562dead98e80 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f300878b740 1 --2- 172.21.15.62:0/1704030444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562dead16460 0x562dead9a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f300878b740 1 --2- 172.21.15.62:0/1704030444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562dead0daf0 0x562dead424c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f300878b740 1 -- 172.21.15.62:0/1704030444 >> 172.21.15.62:0/1704030444 conn(0x562deacb5b40 msgr2=0x562dead1b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f2fe3e00640 0 client.7092 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f300878b740 1 -- 172.21.15.62:0/1704030444 shutdown_connections 2024-03-23T11:25:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f300878b740 1 -- 172.21.15.62:0/1704030444 wait complete. 2024-03-23T11:25:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f300878b740 20 client.7092 trim_cache size 0 max 16384 2024-03-23T11:25:28.449 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:25:28.449 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T11:25:28.449 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-03-23T11:25:28.449 INFO:teuthology.orchestra.run.smithi062.stdout: "56711550-dd1b-42c1-96c4-6d2c40bbc197": { 2024-03-23T11:25:28.449 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/subdir", 2024-03-23T11:25:28.449 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "56711550-dd1b-42c1-96c4-6d2c40bbc197", 2024-03-23T11:25:28.449 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,repair" 2024-03-23T11:25:28.449 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:25:28.449 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:25:28.449 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:25:28.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f2fffe00640 1 -- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fdc06a7a0 msgr2=0x7f2fdc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f2fffe00640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fdc06a7a0 0x7f2fdc06cc60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8036000 tx=0x7f2fe8008030 comp rx=0 tx=0).stop 2024-03-23T11:25:28.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f2fffe00640 1 -- 172.21.15.62:0/540850248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3000071680 msgr2=0x7f300015dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:28.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.452+0000 7f2fffe00640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3000071680 0x7f300015dc70 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000d5f0 tx=0x7f2ff000dab0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.453+0000 7f2fffe00640 1 -- 172.21.15.62:0/540850248 shutdown_connections 2024-03-23T11:25:28.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.453+0000 7f2fffe00640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fdc06a7a0 0x7f2fdc06cc60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.453+0000 7f2fffe00640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f300015fd10 0x7f3000145180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.453+0000 7f2fffe00640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30000b2530 0x7f3000156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.453+0000 7f2fffe00640 1 --2- 172.21.15.62:0/540850248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3000071680 0x7f300015dc70 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:28.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.453+0000 7f2fffe00640 1 -- 172.21.15.62:0/540850248 >> 172.21.15.62:0/540850248 conn(0x7f3000070810 msgr2=0x7f300006af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:28.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.453+0000 7f2fffe00640 1 -- 172.21.15.62:0/540850248 shutdown_connections 2024-03-23T11:25:28.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:28.453+0000 7f2fffe00640 1 -- 172.21.15.62:0/540850248 wait complete. 2024-03-23T11:25:28.463 INFO:tasks.cephfs.filesystem:scrub status for tag:56711550-dd1b-42c1-96c4-6d2c40bbc197 - {'path': '/subdir', 'tag': '56711550-dd1b-42c1-96c4-6d2c40bbc197', 'options': 'recursive,repair'} 2024-03-23T11:25:46.265 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:25:46.275 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:25:58.465 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub status 2024-03-23T11:25:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.702+0000 7faaa4e00640 1 Processor -- start 2024-03-23T11:25:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.703+0000 7faaa4e00640 1 -- start start 2024-03-23T11:25:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.704+0000 7faaa4e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faaa015cc20 0x7faaa00b4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.704+0000 7faaa4e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faaa00b46c0 0x7faaa006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.704+0000 7faaa4e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa00b0000 con 0x7faaa0070260 2024-03-23T11:25:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.705+0000 7faaa4e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa00706b0 con 0x7faaa00b46c0 2024-03-23T11:25:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.705+0000 7faaa4e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa006afa0 con 0x7faaa015cc20 2024-03-23T11:25:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.705+0000 7faa9e800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7faaa0070260 0x7faaa006b430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53206/0 (socket says 172.21.15.62:53206) 2024-03-23T11:25:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.705+0000 7faa9e800640 1 -- 172.21.15.62:0/2233599994 learned_addr learned my addr 172.21.15.62:0/2233599994 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:25:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.705+0000 7faa9d400640 1 --2- 172.21.15.62:0/2233599994 >> v2:172.21.15.62:3300/0 conn(0x7faaa00b46c0 0x7faaa006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.705+0000 7faa9de00640 1 --2- 172.21.15.62:0/2233599994 >> v2:172.21.15.120:3300/0 conn(0x7faaa015cc20 0x7faaa00b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.705+0000 7faa9ca00640 1 -- 172.21.15.62:0/2233599994 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404382012 0 0) 0x7faa98001490 con 0x7faaa0070260 2024-03-23T11:25:58.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.705+0000 7faa9ca00640 1 -- 172.21.15.62:0/2233599994 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa84003520 con 0x7faaa0070260 2024-03-23T11:25:58.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.705+0000 7faa9d400640 1 -- 172.21.15.62:0/2233599994 >> v2:172.21.15.120:3300/0 conn(0x7faaa015cc20 msgr2=0x7faaa00b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:58.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.705+0000 7faa9d400640 1 --2- 172.21.15.62:0/2233599994 >> v2:172.21.15.120:3300/0 conn(0x7faaa015cc20 0x7faaa00b4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.705+0000 7faa9d400640 1 -- 172.21.15.62:0/2233599994 >> v1:172.21.15.62:6789/0 conn(0x7faaa0070260 legacy=0x7faaa006b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.706+0000 7faa9d400640 1 -- 172.21.15.62:0/2233599994 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa00af760 con 0x7faaa00b46c0 2024-03-23T11:25:58.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.707+0000 7faa9d400640 1 --2- 172.21.15.62:0/2233599994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa00b46c0 0x7faaa006fd20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7faaa00d1830 tx=0x7faa8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dfea424f99bc8b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:58.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.707+0000 7faa9ca00640 1 -- 172.21.15.62:0/2233599994 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c031830 con 0x7faaa00b46c0 2024-03-23T11:25:58.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.707+0000 7faa9ca00640 1 -- 172.21.15.62:0/2233599994 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8c031990 con 0x7faaa00b46c0 2024-03-23T11:25:58.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.707+0000 7faa9ca00640 1 -- 172.21.15.62:0/2233599994 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c031d20 con 0x7faaa00b46c0 2024-03-23T11:25:58.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.707+0000 7faaa4e00640 1 -- 172.21.15.62:0/2233599994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa00b46c0 msgr2=0x7faaa006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.707+0000 7faaa4e00640 1 --2- 172.21.15.62:0/2233599994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa00b46c0 0x7faaa006fd20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7faaa00d1830 tx=0x7faa8c02b8d0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.708+0000 7faaa4e00640 1 -- 172.21.15.62:0/2233599994 shutdown_connections 2024-03-23T11:25:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.708+0000 7faaa4e00640 1 --2- 172.21.15.62:0/2233599994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa00b46c0 0x7faaa006fd20 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.708+0000 7faaa4e00640 1 --2- 172.21.15.62:0/2233599994 >> v2:172.21.15.120:3300/0 conn(0x7faaa015cc20 0x7faaa00b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.708+0000 7faaa4e00640 1 -- 172.21.15.62:0/2233599994 >> 172.21.15.62:0/2233599994 conn(0x7faaa006e580 msgr2=0x7faaa006e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.708+0000 7faaa4e00640 1 -- 172.21.15.62:0/2233599994 shutdown_connections 2024-03-23T11:25:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.708+0000 7faaa4e00640 1 -- 172.21.15.62:0/2233599994 wait complete. 2024-03-23T11:25:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.710+0000 7faaa4e00640 1 Processor -- start 2024-03-23T11:25:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.710+0000 7faaa4e00640 1 -- start start 2024-03-23T11:25:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faaa4e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faaa0070260 0x7faaa015d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faaa4e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faaa00b46c0 0x7faaa01460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faaa4e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa015cc20 0x7faaa015b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faaa4e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa015a1e0 con 0x7faaa015cc20 2024-03-23T11:25:58.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faaa4e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa015ad60 con 0x7faaa00b46c0 2024-03-23T11:25:58.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faaa4e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa015ea30 con 0x7faaa0070260 2024-03-23T11:25:58.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faa9e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa015cc20 0x7faaa015b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faa9e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa015cc20 0x7faaa015b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52402/0 (socket says 172.21.15.62:52402) 2024-03-23T11:25:58.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faa9e800640 1 -- 172.21.15.62:0/635030641 learned_addr learned my addr 172.21.15.62:0/635030641 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:58.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faa9de00640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faaa0070260 0x7faaa015d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faa9e800640 1 -- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faaa0070260 msgr2=0x7faaa015d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faa9d400640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faaa00b46c0 0x7faaa01460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faa9e800640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faaa0070260 0x7faaa015d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faa9e800640 1 -- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faaa00b46c0 msgr2=0x7faaa01460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faa9e800640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faaa00b46c0 0x7faaa01460b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.711+0000 7faa9e800640 1 -- 172.21.15.62:0/635030641 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c009400 con 0x7faaa015cc20 2024-03-23T11:25:58.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.712+0000 7faa9e800640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa015cc20 0x7faaa015b1f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7faa9800cba0 tx=0x7faa9800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:58.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.712+0000 7faa96000640 1 -- 172.21.15.62:0/635030641 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9800ad40 con 0x7faaa015cc20 2024-03-23T11:25:58.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.712+0000 7faa96000640 1 -- 172.21.15.62:0/635030641 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa98012400 con 0x7faaa015cc20 2024-03-23T11:25:58.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.712+0000 7faa96000640 1 -- 172.21.15.62:0/635030641 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9800aea0 con 0x7faaa015cc20 2024-03-23T11:25:58.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.712+0000 7faaa4e00640 1 -- 172.21.15.62:0/635030641 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa015a430 con 0x7faaa015cc20 2024-03-23T11:25:58.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.712+0000 7faaa4e00640 1 -- 172.21.15.62:0/635030641 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa015a9f0 con 0x7faaa015cc20 2024-03-23T11:25:58.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.714+0000 7faa96000640 1 -- 172.21.15.62:0/635030641 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7faa98007070 con 0x7faaa015cc20 2024-03-23T11:25:58.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.715+0000 7faa96000640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa8006a7a0 0x7faa8006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.715+0000 7faa96000640 1 -- 172.21.15.62:0/635030641 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7faa98088df0 con 0x7faaa015cc20 2024-03-23T11:25:58.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.716+0000 7faa9de00640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa8006a7a0 0x7faa8006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.716+0000 7faaa7bec740 1 Processor -- start 2024-03-23T11:25:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.716+0000 7faaa7bec740 1 -- start start 2024-03-23T11:25:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.717+0000 7faa9de00640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa8006a7a0 0x7faa8006cc60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7faa90006fd0 tx=0x7faa90008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612beb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612bf280 0x562b61296a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6cf0 0x562b61296f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faaa7bec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b61252cb0 con 0x562b612b6910 2024-03-23T11:25:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faaa7bec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b61201500 con 0x562b612b6cf0 2024-03-23T11:25:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faaa7bec740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562b611fec90 con 0x562b612bf280 2024-03-23T11:25:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faa9de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612beb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faa9de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612beb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52410/0 (socket says 172.21.15.62:52410) 2024-03-23T11:25:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faa9de00640 1 -- 172.21.15.62:0/2308161566 learned_addr learned my addr 172.21.15.62:0/2308161566 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faa9de00640 1 -- 172.21.15.62:0/2308161566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612bf280 msgr2=0x562b61296a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faa9de00640 1 --2- 172.21.15.62:0/2308161566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612bf280 0x562b61296a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faa9de00640 1 -- 172.21.15.62:0/2308161566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6cf0 msgr2=0x562b61296f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faa9e800640 1 --2- 172.21.15.62:0/2308161566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6cf0 0x562b61296f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faa9de00640 1 --2- 172.21.15.62:0/2308161566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6cf0 0x562b61296f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.718+0000 7faa9de00640 1 -- 172.21.15.62:0/2308161566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b611ac360 con 0x562b612b6910 2024-03-23T11:25:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.719+0000 7faa9de00640 1 --2- 172.21.15.62:0/2308161566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612beb80 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7faa9000e840 tx=0x7faa90011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d161269abcd9e111 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.719+0000 7faa8aa00640 1 -- 172.21.15.62:0/2308161566 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90015640 con 0x562b612b6910 2024-03-23T11:25:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.719+0000 7faa8aa00640 1 -- 172.21.15.62:0/2308161566 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa900157a0 con 0x562b612b6910 2024-03-23T11:25:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.719+0000 7faa8aa00640 1 -- 172.21.15.62:0/2308161566 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90015b30 con 0x562b612b6910 2024-03-23T11:25:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.720+0000 7faaa7bec740 1 -- 172.21.15.62:0/2308161566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 msgr2=0x562b612beb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.720+0000 7faaa7bec740 1 --2- 172.21.15.62:0/2308161566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612beb80 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7faa9000e840 tx=0x7faa90011000 comp rx=0 tx=0).stop 2024-03-23T11:25:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.720+0000 7faaa7bec740 1 -- 172.21.15.62:0/2308161566 shutdown_connections 2024-03-23T11:25:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.720+0000 7faaa7bec740 1 --2- 172.21.15.62:0/2308161566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6cf0 0x562b61296f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.720+0000 7faaa7bec740 1 --2- 172.21.15.62:0/2308161566 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612bf280 0x562b61296a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.720+0000 7faaa7bec740 1 --2- 172.21.15.62:0/2308161566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612beb80 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.720+0000 7faaa7bec740 1 -- 172.21.15.62:0/2308161566 >> 172.21.15.62:0/2308161566 conn(0x562b6125eb40 msgr2=0x562b612dc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.720+0000 7faaa7bec740 1 -- 172.21.15.62:0/2308161566 shutdown_connections 2024-03-23T11:25:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.720+0000 7faaa7bec740 1 -- 172.21.15.62:0/2308161566 wait complete. 2024-03-23T11:25:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.721+0000 7faaa7bec740 1 -- start start 2024-03-23T11:25:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.721+0000 7faaa7bec740 1 Processor -- start 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.722+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.722+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6cf0 0x562b612bcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.722+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612bf280 0x562b612dbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.722+0000 7faaa7bec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b611fa580 con 0x562b612b6910 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.722+0000 7faaa7bec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b6119ba50 con 0x562b612bf280 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.722+0000 7faaa7bec740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562b6119b6f0 con 0x562b612b6cf0 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.722+0000 7faaa7bec740 10 client.0 _ll_register_callbacks cb 0x562b611eaed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.722+0000 7faa9de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.722+0000 7faa9de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612b5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52426/0 (socket says 172.21.15.62:52426) 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.722+0000 7faa9de00640 1 -- 172.21.15.62:0/2527905955 learned_addr learned my addr 172.21.15.62:0/2527905955 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa9e800640 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612bf280 0x562b612dbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa9d400640 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6cf0 0x562b612bcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa9de00640 1 -- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6cf0 msgr2=0x562b612bcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa9de00640 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6cf0 0x562b612bcc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa9de00640 1 -- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612bf280 msgr2=0x562b612dbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa9de00640 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612bf280 0x562b612dbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa9de00640 1 -- 172.21.15.62:0/2527905955 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9000e5b0 con 0x562b612b6910 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa9e800640 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612bf280 0x562b612dbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa9de00640 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612b5b10 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7faa90023040 tx=0x7faa90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a462e3dc9f32690c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa7f400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa7f400640 1 -- 172.21.15.62:0/2527905955 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa900208d0 con 0x562b612b6910 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faa7f400640 1 -- 172.21.15.62:0/2527905955 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7faa90020a30 con 0x562b612b6910 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.723+0000 7faaa7bec740 1 -- 172.21.15.62:0/2527905955 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562b61224fe0 con 0x562b612b6910 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.724+0000 7faa7f400640 1 -- 172.21.15.62:0/2527905955 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa90020bf0 con 0x562b612b6910 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.724+0000 7faaa7bec740 10 client.7107 fetch_fsmap learned FSMap version 50 2024-03-23T11:25:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.724+0000 7faaa7bec740 1 -- 172.21.15.62:0/2527905955 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x562b611f8c00 con 0x562b612b6910 2024-03-23T11:25:58.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.724+0000 7faa7f400640 1 -- 172.21.15.62:0/2527905955 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562b61224fe0 con 0x562b612b6910 2024-03-23T11:25:58.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.724+0000 7faa7f400640 1 -- 172.21.15.62:0/2527905955 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 50) v1 ==== 3114+0+0 (secure 0 0 0) 0x7faa90029430 con 0x562b612b6910 2024-03-23T11:25:58.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.724+0000 7faaa7bec740 10 client.7107 fetch_fsmap finished waiting for FSMap version 50 2024-03-23T11:25:58.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.724+0000 7faaa7bec740 10 client.7107 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:25:58.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.724+0000 7faaa7bec740 20 client.7107 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:58.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.724+0000 7faaa7bec740 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562b612c08b0 0x562b612c0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.724+0000 7faaa7bec740 4 client.7107 mds_command: new command op to 6618 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:25:58.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.724+0000 7faaa7bec740 1 -- 172.21.15.62:0/2527905955 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b611ed240 con 0x562b612c08b0 2024-03-23T11:25:58.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.725+0000 7faa9d400640 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562b612c08b0 0x562b612c0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.726+0000 7faa9d400640 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562b612c08b0 0x562b612c0c90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=820a3524d0275afb server_cookie=3de5fb1f93fcba0b in_seq=0 out_seq=0 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.726+0000 7faa7f400640 10 client.7107 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.727+0000 7faa7f400640 1 -- 172.21.15.62:0/2527905955 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x562b611ed240 con 0x562b612c08b0 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.727+0000 7faa7f400640 10 client.7107 handle_command_reply: tid=0 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.727+0000 7faaa7bec740 1 client.7107 shutdown 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faaa7bec740 1 -- 172.21.15.62:0/2527905955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 msgr2=0x562b612b5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faaa7bec740 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612b5b10 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7faa90023040 tx=0x7faa90004730 comp rx=0 tx=0).stop 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faaa7bec740 1 -- 172.21.15.62:0/2527905955 shutdown_connections 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faaa7bec740 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562b612c08b0 0x562b612c0c90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faaa7bec740 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612bf280 0x562b612dbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faaa7bec740 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6cf0 0x562b612bcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faaa7bec740 1 --2- 172.21.15.62:0/2527905955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612b6910 0x562b612b5b10 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faa7f400640 0 client.7107 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faaa7bec740 1 -- 172.21.15.62:0/2527905955 >> 172.21.15.62:0/2527905955 conn(0x562b6125eb40 msgr2=0x562b612dfca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faaa7bec740 1 -- 172.21.15.62:0/2527905955 shutdown_connections 2024-03-23T11:25:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faaa7bec740 1 -- 172.21.15.62:0/2527905955 wait complete. 2024-03-23T11:25:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.728+0000 7faaa7bec740 20 client.7107 trim_cache size 0 max 16384 2024-03-23T11:25:58.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.744+0000 7faaa7bec740 1 Processor -- start 2024-03-23T11:25:58.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.744+0000 7faaa7bec740 1 -- start start 2024-03-23T11:25:58.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.744+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6910 0x562b612b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.744+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6cf0 0x562b612df0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612df620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faaa7bec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b60ef7f90 con 0x562b612bf280 2024-03-23T11:25:58.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faaa7bec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b612c15a0 con 0x562b612b6910 2024-03-23T11:25:58.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faaa7bec740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562b612915f0 con 0x562b612b6cf0 2024-03-23T11:25:58.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faa9e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612df620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faa9e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612df620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52436/0 (socket says 172.21.15.62:52436) 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faa9e800640 1 -- 172.21.15.62:0/729504929 learned_addr learned my addr 172.21.15.62:0/729504929 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faa9de00640 1 --2- 172.21.15.62:0/729504929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6910 0x562b612b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faa9d400640 1 --2- 172.21.15.62:0/729504929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6cf0 0x562b612df0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faa9e800640 1 -- 172.21.15.62:0/729504929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6cf0 msgr2=0x562b612df0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faa9e800640 1 --2- 172.21.15.62:0/729504929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6cf0 0x562b612df0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faa9e800640 1 -- 172.21.15.62:0/729504929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6910 msgr2=0x562b612b61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faa9e800640 1 --2- 172.21.15.62:0/729504929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6910 0x562b612b61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.745+0000 7faa9e800640 1 -- 172.21.15.62:0/729504929 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b6119ac60 con 0x562b612bf280 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.746+0000 7faa9e800640 1 --2- 172.21.15.62:0/729504929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612df620 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7faa980012e0 tx=0x7faa98057d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64fc6694d9755441 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.746+0000 7faa8b400640 1 -- 172.21.15.62:0/729504929 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa980560b0 con 0x562b612bf280 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.746+0000 7faa8b400640 1 -- 172.21.15.62:0/729504929 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa98054420 con 0x562b612bf280 2024-03-23T11:25:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.746+0000 7faa8b400640 1 -- 172.21.15.62:0/729504929 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9806be00 con 0x562b612bf280 2024-03-23T11:25:58.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.746+0000 7faaa7bec740 1 -- 172.21.15.62:0/729504929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 msgr2=0x562b612df620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.746+0000 7faaa7bec740 1 --2- 172.21.15.62:0/729504929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612df620 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7faa980012e0 tx=0x7faa98057d10 comp rx=0 tx=0).stop 2024-03-23T11:25:58.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.746+0000 7faaa7bec740 1 -- 172.21.15.62:0/729504929 shutdown_connections 2024-03-23T11:25:58.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.746+0000 7faaa7bec740 1 --2- 172.21.15.62:0/729504929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612df620 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.746+0000 7faaa7bec740 1 --2- 172.21.15.62:0/729504929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6cf0 0x562b612df0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.746+0000 7faaa7bec740 1 --2- 172.21.15.62:0/729504929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6910 0x562b612b61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.746+0000 7faaa7bec740 1 -- 172.21.15.62:0/729504929 >> 172.21.15.62:0/729504929 conn(0x562b6125eb40 msgr2=0x562b612c0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.747+0000 7faaa7bec740 1 -- 172.21.15.62:0/729504929 shutdown_connections 2024-03-23T11:25:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.747+0000 7faaa7bec740 1 -- 172.21.15.62:0/729504929 wait complete. 2024-03-23T11:25:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.747+0000 7faaa7bec740 1 -- start start 2024-03-23T11:25:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.747+0000 7faaa7bec740 1 Processor -- start 2024-03-23T11:25:58.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.748+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6910 0x562b6132add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.748+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6cf0 0x562b612c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.748+0000 7faaa7bec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612c4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.748+0000 7faaa7bec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b612e0bc0 con 0x562b612bf280 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.748+0000 7faaa7bec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b612f4d00 con 0x562b612b6cf0 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.748+0000 7faaa7bec740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x562b612c1da0 con 0x562b612b6910 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.748+0000 7faaa7bec740 10 client.0 _ll_register_callbacks cb 0x7faa900094e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612c4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52442/0 (socket says 172.21.15.62:52442) 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9e800640 1 -- 172.21.15.62:0/781376136 learned_addr learned my addr 172.21.15.62:0/781376136 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9d400640 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6cf0 0x562b612c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9e800640 1 -- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6910 msgr2=0x562b6132add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9de00640 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6910 0x562b6132add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9e800640 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6910 0x562b6132add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9e800640 1 -- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6cf0 msgr2=0x562b612c16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9e800640 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6cf0 0x562b612c16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9e800640 1 -- 172.21.15.62:0/781376136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa015a9f0 con 0x562b612bf280 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9d400640 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6cf0 0x562b612c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa9e800640 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612c4070 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7faa98001c20 tx=0x7faa98001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6335ad2b255a633 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.749+0000 7faa7f400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faa7f400640 1 -- 172.21.15.62:0/781376136 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98057400 con 0x562b612bf280 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faa7f400640 1 -- 172.21.15.62:0/781376136 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7faa98073070 con 0x562b612bf280 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faaa7bec740 1 -- 172.21.15.62:0/781376136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faa900157a0 con 0x562b612bf280 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faa7f400640 1 -- 172.21.15.62:0/781376136 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa98054590 con 0x562b612bf280 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faaa7bec740 10 client.7113 fetch_fsmap learned FSMap version 50 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faaa7bec740 1 -- 172.21.15.62:0/781376136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x562b611c0170 con 0x562b612bf280 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faa7f400640 1 -- 172.21.15.62:0/781376136 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa900157a0 con 0x562b612bf280 2024-03-23T11:25:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faa7f400640 1 -- 172.21.15.62:0/781376136 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 50) v1 ==== 3114+0+0 (secure 0 0 0) 0x7faa98058d20 con 0x562b612bf280 2024-03-23T11:25:58.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faaa7bec740 10 client.7113 fetch_fsmap finished waiting for FSMap version 50 2024-03-23T11:25:58.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faaa7bec740 10 client.7113 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.b 2024-03-23T11:25:58.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faaa7bec740 20 client.7113 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:58.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faaa7bec740 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562b6132b3c0 0x562b612c45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faaa7bec740 4 client.7113 mds_command: new command op to 6618 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:25:58.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.750+0000 7faaa7bec740 1 -- 172.21.15.62:0/781376136 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562b60eee4e0 con 0x562b6132b3c0 2024-03-23T11:25:58.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.751+0000 7faa9de00640 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562b6132b3c0 0x562b612c45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.752+0000 7faa9de00640 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562b6132b3c0 0x562b612c45b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a9f8d92579140cf server_cookie=900859b068519dce in_seq=0 out_seq=0 2024-03-23T11:25:58.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.752+0000 7faa7f400640 10 client.7113 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:58.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faa7f400640 1 -- 172.21.15.62:0/781376136 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562b60eee4e0 con 0x562b6132b3c0 2024-03-23T11:25:58.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faa7f400640 10 client.7113 handle_command_reply: tid=0 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 1 client.7113 shutdown 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 1 -- 172.21.15.62:0/781376136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 msgr2=0x562b612c4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612c4070 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7faa98001c20 tx=0x7faa98001250 comp rx=0 tx=0).stop 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 1 -- 172.21.15.62:0/781376136 shutdown_connections 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x562b6132b3c0 0x562b612c45b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b612bf280 0x562b612c4070 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b612b6cf0 0x562b612c16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 1 --2- 172.21.15.62:0/781376136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x562b612b6910 0x562b6132add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faa7f400640 0 client.7113 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 1 -- 172.21.15.62:0/781376136 >> 172.21.15.62:0/781376136 conn(0x562b6125eb40 msgr2=0x562b61290b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 1 -- 172.21.15.62:0/781376136 shutdown_connections 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 1 -- 172.21.15.62:0/781376136 wait complete. 2024-03-23T11:25:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.753+0000 7faaa7bec740 20 client.7113 trim_cache size 0 max 16384 2024-03-23T11:25:58.750 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:25:58.750 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-03-23T11:25:58.750 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-03-23T11:25:58.750 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:25:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 -- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa8006a7a0 msgr2=0x7faa8006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa8006a7a0 0x7faa8006cc60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7faa90006fd0 tx=0x7faa90008040 comp rx=0 tx=0).stop 2024-03-23T11:25:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 -- 172.21.15.62:0/635030641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa015cc20 msgr2=0x7faaa015b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa015cc20 0x7faaa015b1f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7faa9800cba0 tx=0x7faa9800a430 comp rx=0 tx=0).stop 2024-03-23T11:25:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 -- 172.21.15.62:0/635030641 shutdown_connections 2024-03-23T11:25:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7faa8006a7a0 0x7faa8006cc60 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa015cc20 0x7faaa015b1f0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faaa00b46c0 0x7faaa01460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 --2- 172.21.15.62:0/635030641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faaa0070260 0x7faaa015d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 -- 172.21.15.62:0/635030641 >> 172.21.15.62:0/635030641 conn(0x7faaa006e580 msgr2=0x7faaa015bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 -- 172.21.15.62:0/635030641 shutdown_connections 2024-03-23T11:25:58.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.754+0000 7faaa4e00640 1 -- 172.21.15.62:0/635030641 wait complete. 2024-03-23T11:25:58.766 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T11:25:58.767 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:25:58.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.990+0000 7efd87200640 1 Processor -- start 2024-03-23T11:25:58.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.991+0000 7efd87200640 1 -- start start 2024-03-23T11:25:58.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.992+0000 7efd87200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efd8815fcf0 0x7efd880d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.992+0000 7efd87200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd88160420 0x7efd88160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.992+0000 7efd87200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efd880d06d0 0x7efd880d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.992+0000 7efd87200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd88160dc0 con 0x7efd880d06d0 2024-03-23T11:25:58.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.992+0000 7efd87200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd880d3540 con 0x7efd8815fcf0 2024-03-23T11:25:58.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.992+0000 7efd87200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd880ce510 con 0x7efd88160420 2024-03-23T11:25:58.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.993+0000 7efd86800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efd880d06d0 0x7efd880d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.993+0000 7efd86800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efd880d06d0 0x7efd880d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52456/0 (socket says 172.21.15.62:52456) 2024-03-23T11:25:58.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.993+0000 7efd86800640 1 -- 172.21.15.62:0/1768175573 learned_addr learned my addr 172.21.15.62:0/1768175573 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:58.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.993+0000 7efd85e00640 1 --2- 172.21.15.62:0/1768175573 >> v2:172.21.15.120:3300/0 conn(0x7efd8815fcf0 0x7efd880d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.993+0000 7efd85400640 1 --2- 172.21.15.62:0/1768175573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd88160420 0x7efd88160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.993+0000 7efd86800640 1 -- 172.21.15.62:0/1768175573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd88160420 msgr2=0x7efd88160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:58.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.993+0000 7efd86800640 1 --2- 172.21.15.62:0/1768175573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd88160420 0x7efd88160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.993+0000 7efd86800640 1 -- 172.21.15.62:0/1768175573 >> v2:172.21.15.120:3300/0 conn(0x7efd8815fcf0 msgr2=0x7efd880d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:58.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.993+0000 7efd86800640 1 --2- 172.21.15.62:0/1768175573 >> v2:172.21.15.120:3300/0 conn(0x7efd8815fcf0 0x7efd880d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.993+0000 7efd86800640 1 -- 172.21.15.62:0/1768175573 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd880cdc70 con 0x7efd880d06d0 2024-03-23T11:25:58.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.994+0000 7efd86800640 1 --2- 172.21.15.62:0/1768175573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd880d06d0 0x7efd880d0b20 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7efd88066480 tx=0x7efd8002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f2c7d73c737440 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:58.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.995+0000 7efd84a00640 1 -- 172.21.15.62:0/1768175573 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd800317e0 con 0x7efd880d06d0 2024-03-23T11:25:58.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.995+0000 7efd84a00640 1 -- 172.21.15.62:0/1768175573 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd80031940 con 0x7efd880d06d0 2024-03-23T11:25:58.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.995+0000 7efd84a00640 1 -- 172.21.15.62:0/1768175573 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80031cd0 con 0x7efd880d06d0 2024-03-23T11:25:58.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.995+0000 7efd87200640 1 -- 172.21.15.62:0/1768175573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd880d06d0 msgr2=0x7efd880d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.995+0000 7efd87200640 1 --2- 172.21.15.62:0/1768175573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd880d06d0 0x7efd880d0b20 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7efd88066480 tx=0x7efd8002b880 comp rx=0 tx=0).stop 2024-03-23T11:25:58.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.995+0000 7efd87200640 1 -- 172.21.15.62:0/1768175573 shutdown_connections 2024-03-23T11:25:58.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.995+0000 7efd87200640 1 --2- 172.21.15.62:0/1768175573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd880d06d0 0x7efd880d0b20 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.995+0000 7efd87200640 1 --2- 172.21.15.62:0/1768175573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd88160420 0x7efd88160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.995+0000 7efd87200640 1 --2- 172.21.15.62:0/1768175573 >> v2:172.21.15.120:3300/0 conn(0x7efd8815fcf0 0x7efd880d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.995+0000 7efd87200640 1 -- 172.21.15.62:0/1768175573 >> 172.21.15.62:0/1768175573 conn(0x7efd88089f60 msgr2=0x7efd880d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:58.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.996+0000 7efd87200640 1 -- 172.21.15.62:0/1768175573 shutdown_connections 2024-03-23T11:25:58.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.996+0000 7efd87200640 1 -- 172.21.15.62:0/1768175573 wait complete. 2024-03-23T11:25:58.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.997+0000 7efd87200640 1 Processor -- start 2024-03-23T11:25:58.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.997+0000 7efd87200640 1 -- start start 2024-03-23T11:25:58.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.998+0000 7efd87200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd880d06d0 0x7efd880c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.998+0000 7efd87200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd8815fcf0 0x7efd880c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.998+0000 7efd87200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd88160420 0x7efd880b6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.998+0000 7efd87200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd880c9e30 con 0x7efd8815fcf0 2024-03-23T11:25:58.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.998+0000 7efd87200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd881f63c0 con 0x7efd88160420 2024-03-23T11:25:58.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.998+0000 7efd87200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd881f5600 con 0x7efd880d06d0 2024-03-23T11:25:58.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd85400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd8815fcf0 0x7efd880c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd85400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd8815fcf0 0x7efd880c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52470/0 (socket says 172.21.15.62:52470) 2024-03-23T11:25:58.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd85400640 1 -- 172.21.15.62:0/4247721106 learned_addr learned my addr 172.21.15.62:0/4247721106 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:58.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd85e00640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd880d06d0 0x7efd880c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd86800640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd88160420 0x7efd880b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd85400640 1 -- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd880d06d0 msgr2=0x7efd880c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd85400640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd880d06d0 0x7efd880c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd85400640 1 -- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd88160420 msgr2=0x7efd880b6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:58.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd85400640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd88160420 0x7efd880b6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:58.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd85400640 1 -- 172.21.15.62:0/4247721106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd800094d0 con 0x7efd8815fcf0 2024-03-23T11:25:58.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd85e00640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd880d06d0 0x7efd880c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:58.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd86800640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd88160420 0x7efd880b6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:58.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:58.999+0000 7efd85400640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd8815fcf0 0x7efd880c5ed0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7efd7000d800 tx=0x7efd7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:58.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.000+0000 7efd7e000640 1 -- 172.21.15.62:0/4247721106 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7000b6c0 con 0x7efd8815fcf0 2024-03-23T11:25:58.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.000+0000 7efd7e000640 1 -- 172.21.15.62:0/4247721106 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7000b820 con 0x7efd8815fcf0 2024-03-23T11:25:58.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.000+0000 7efd87200640 1 -- 172.21.15.62:0/4247721106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd881f58e0 con 0x7efd8815fcf0 2024-03-23T11:25:58.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.000+0000 7efd7e000640 1 -- 172.21.15.62:0/4247721106 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7000bbb0 con 0x7efd8815fcf0 2024-03-23T11:25:58.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.000+0000 7efd87200640 1 -- 172.21.15.62:0/4247721106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd881f5ea0 con 0x7efd8815fcf0 2024-03-23T11:25:58.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.001+0000 7efd87200640 1 -- 172.21.15.62:0/4247721106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd881f6060 con 0x7efd8815fcf0 2024-03-23T11:25:58.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.002+0000 7efd7e000640 1 -- 172.21.15.62:0/4247721106 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7efd7000bd10 con 0x7efd8815fcf0 2024-03-23T11:25:58.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.003+0000 7efd7e000640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd6406a7a0 0x7efd6406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:58.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.003+0000 7efd7e000640 1 -- 172.21.15.62:0/4247721106 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7efd70089390 con 0x7efd8815fcf0 2024-03-23T11:25:58.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.003+0000 7efd85e00640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd6406a7a0 0x7efd6406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:58.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.004+0000 7efd85e00640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd6406a7a0 0x7efd6406cc60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7efd880cc7b0 tx=0x7efd7800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:59.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.007+0000 7efd7e000640 1 -- 172.21.15.62:0/4247721106 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efd70054710 con 0x7efd8815fcf0 2024-03-23T11:25:59.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.243+0000 7efd87200640 1 -- 172.21.15.62:0/4247721106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd88066bb0 con 0x7efd8815fcf0 2024-03-23T11:25:59.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.244+0000 7efd7e000640 1 -- 172.21.15.62:0/4247721106 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4738 (secure 0 0 0) 0x7efd70057d30 con 0x7efd8815fcf0 2024-03-23T11:25:59.239 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:25:59.239 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/29171509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":29171509},{"type":"v1","addr":"172.21.15.120:6837","nonce":29171509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:25:33.449465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6618":{"gid":6618,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":25,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:25:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 50 2024-03-23T11:25:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 -- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd6406a7a0 msgr2=0x7efd6406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd6406a7a0 0x7efd6406cc60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7efd880cc7b0 tx=0x7efd7800b040 comp rx=0 tx=0).stop 2024-03-23T11:25:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 -- 172.21.15.62:0/4247721106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd8815fcf0 msgr2=0x7efd880c5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd8815fcf0 0x7efd880c5ed0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7efd7000d800 tx=0x7efd7000dcc0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 -- 172.21.15.62:0/4247721106 shutdown_connections 2024-03-23T11:25:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efd6406a7a0 0x7efd6406cc60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd8815fcf0 0x7efd880c5ed0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd88160420 0x7efd880b6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 --2- 172.21.15.62:0/4247721106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efd880d06d0 0x7efd880c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 -- 172.21.15.62:0/4247721106 >> 172.21.15.62:0/4247721106 conn(0x7efd88089f60 msgr2=0x7efd880b7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 -- 172.21.15.62:0/4247721106 shutdown_connections 2024-03-23T11:25:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.247+0000 7efd87200640 1 -- 172.21.15.62:0/4247721106 wait complete. 2024-03-23T11:25:59.261 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b damage ls --format=json-pretty 2024-03-23T11:25:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.492+0000 7f9d6e800640 1 Processor -- start 2024-03-23T11:25:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.492+0000 7f9d6e800640 1 -- start start 2024-03-23T11:25:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.494+0000 7f9d6e800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9d681431f0 0x7f9d680b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.494+0000 7f9d6e800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d680b20d0 con 0x7f9d680cc9b0 2024-03-23T11:25:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.494+0000 7f9d6e800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d681435d0 con 0x7f9d68070090 2024-03-23T11:25:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.494+0000 7f9d6e800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d680b14c0 con 0x7f9d681431f0 2024-03-23T11:25:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.494+0000 7f9d6de00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9d680cc9b0 0x7f9d6815aae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53218/0 (socket says 172.21.15.62:53218) 2024-03-23T11:25:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.494+0000 7f9d6de00640 1 -- 172.21.15.62:0/1318236190 learned_addr learned my addr 172.21.15.62:0/1318236190 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:25:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.494+0000 7f9d6d400640 1 --2- 172.21.15.62:0/1318236190 >> v2:172.21.15.120:3300/0 conn(0x7f9d681431f0 0x7f9d680b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190674457 0 0) 0x7f9d680b20d0 con 0x7f9d680cc9b0 2024-03-23T11:25:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d50003520 con 0x7f9d680cc9b0 2024-03-23T11:25:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3633649297 0 0) 0x7f9d50003520 con 0x7f9d680cc9b0 2024-03-23T11:25:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 >> v2:172.21.15.120:3300/0 conn(0x7f9d681431f0 msgr2=0x7f9d680b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 --2- 172.21.15.62:0/1318236190 >> v2:172.21.15.120:3300/0 conn(0x7f9d681431f0 0x7f9d680b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 >> v1:172.21.15.120:6789/0 conn(0x7f9d68070090 legacy=0x7f9d680cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d680b17a0 con 0x7f9d680cc9b0 2024-03-23T11:25:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9d600019f0 con 0x7f9d680cc9b0 2024-03-23T11:25:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 >> v1:172.21.15.62:6789/0 conn(0x7f9d680cc9b0 legacy=0x7f9d6815aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 --2- 172.21.15.62:0/1318236190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 0x7f9d50003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 --2- 172.21.15.62:0/1318236190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d500047c0 0x7f9d50004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 --2- 172.21.15.62:0/1318236190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d50005180 0x7f9d5000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680b14c0 con 0x7f9d50003970 2024-03-23T11:25:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5000c120 con 0x7f9d500047c0 2024-03-23T11:25:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5000c2e0 con 0x7f9d50005180 2024-03-23T11:25:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.495+0000 7f9d6d400640 1 --2- 172.21.15.62:0/1318236190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 0x7f9d50003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.496+0000 7f9d6d400640 1 -- 172.21.15.62:0/1318236190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d50005180 msgr2=0x7f9d5000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.496+0000 7f9d6d400640 1 --2- 172.21.15.62:0/1318236190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d50005180 0x7f9d5000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.496+0000 7f9d6d400640 1 -- 172.21.15.62:0/1318236190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d500047c0 msgr2=0x7f9d50004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.496+0000 7f9d6d400640 1 --2- 172.21.15.62:0/1318236190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d500047c0 0x7f9d50004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.496+0000 7f9d6d400640 1 -- 172.21.15.62:0/1318236190 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d50003520 con 0x7f9d50003970 2024-03-23T11:25:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.496+0000 7f9d6de00640 1 -- 172.21.15.62:0/1318236190 reap_dead start 2024-03-23T11:25:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.497+0000 7f9d6d400640 1 --2- 172.21.15.62:0/1318236190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 0x7f9d50003d90 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c009ac0 tx=0x7f9d500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ddf628df2039487 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.497+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c02c6d0 con 0x7f9d50003970 2024-03-23T11:25:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.497+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c02c830 con 0x7f9d50003970 2024-03-23T11:25:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.498+0000 7f9d67e00640 1 -- 172.21.15.62:0/1318236190 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c02cbc0 con 0x7f9d50003970 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.498+0000 7f9d6e800640 1 -- 172.21.15.62:0/1318236190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 msgr2=0x7f9d50003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.498+0000 7f9d6e800640 1 --2- 172.21.15.62:0/1318236190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 0x7f9d50003d90 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c009ac0 tx=0x7f9d500057f0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.498+0000 7f9d6e800640 1 -- 172.21.15.62:0/1318236190 shutdown_connections 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.498+0000 7f9d6e800640 1 --2- 172.21.15.62:0/1318236190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 0x7f9d50003d90 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.498+0000 7f9d6e800640 1 -- 172.21.15.62:0/1318236190 >> 172.21.15.62:0/1318236190 conn(0x7f9d680b0090 msgr2=0x7f9d680b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.498+0000 7f9d6e800640 1 -- 172.21.15.62:0/1318236190 shutdown_connections 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.498+0000 7f9d6e800640 1 -- 172.21.15.62:0/1318236190 wait complete. 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.500+0000 7f9d6e800640 1 Processor -- start 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.500+0000 7f9d6e800640 1 -- start start 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.501+0000 7f9d6e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 0x7f9d680aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.501+0000 7f9d6e800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d68140a00 0x7f9d6806ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.501+0000 7f9d6e800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d68140e80 0x7f9d6806cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.501+0000 7f9d6e800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d68141370 con 0x7f9d50003970 2024-03-23T11:25:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.501+0000 7f9d6e800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d68141650 con 0x7f9d68140e80 2024-03-23T11:25:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.501+0000 7f9d6e800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d68141930 con 0x7f9d68140a00 2024-03-23T11:25:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.501+0000 7f9d6d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 0x7f9d680aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.501+0000 7f9d6d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 0x7f9d680aceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52484/0 (socket says 172.21.15.62:52484) 2024-03-23T11:25:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.501+0000 7f9d6d400640 1 -- 172.21.15.62:0/3745860227 learned_addr learned my addr 172.21.15.62:0/3745860227 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.502+0000 7f9d6d400640 1 -- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d68140a00 msgr2=0x7f9d6806ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.502+0000 7f9d6d400640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d68140a00 0x7f9d6806ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.502+0000 7f9d6d400640 1 -- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d68140e80 msgr2=0x7f9d6806cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.502+0000 7f9d6d400640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d68140e80 0x7f9d6806cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.502+0000 7f9d6d400640 1 -- 172.21.15.62:0/3745860227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d68157330 con 0x7f9d50003970 2024-03-23T11:25:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.502+0000 7f9d6d400640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 0x7f9d680aceb0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c02c270 tx=0x7f9d5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.502+0000 7f9d65600640 1 -- 172.21.15.62:0/3745860227 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c003810 con 0x7f9d50003970 2024-03-23T11:25:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.502+0000 7f9d65600640 1 -- 172.21.15.62:0/3745860227 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c003970 con 0x7f9d50003970 2024-03-23T11:25:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.502+0000 7f9d65600640 1 -- 172.21.15.62:0/3745860227 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c003ce0 con 0x7f9d50003970 2024-03-23T11:25:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.502+0000 7f9d6e800640 1 -- 172.21.15.62:0/3745860227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d68141c10 con 0x7f9d50003970 2024-03-23T11:25:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.503+0000 7f9d6e800640 1 -- 172.21.15.62:0/3745860227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6813f3a0 con 0x7f9d50003970 2024-03-23T11:25:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.505+0000 7f9d65600640 1 -- 172.21.15.62:0/3745860227 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9d5c040660 con 0x7f9d50003970 2024-03-23T11:25:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.506+0000 7f9d65600640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d4806a7a0 0x7f9d4806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.506+0000 7f9d65600640 1 -- 172.21.15.62:0/3745860227 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9d5c0aae80 con 0x7f9d50003970 2024-03-23T11:25:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.507+0000 7f9d6ca00640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d4806a7a0 0x7f9d4806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.507+0000 7f9d72a12740 1 Processor -- start 2024-03-23T11:25:59.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.507+0000 7f9d72a12740 1 -- start start 2024-03-23T11:25:59.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.508+0000 7f9d6ca00640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d4806a7a0 0x7f9d4806cc60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9d6806d740 tx=0x7f9d540079d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:59.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.508+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecbd8b60 0x5609ecbd8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.508+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecbdcbb0 0x5609ecbde650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.508+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc088f0 0x5609ecc08d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.508+0000 7f9d72a12740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ecb48d80 con 0x5609ecc088f0 2024-03-23T11:25:59.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.508+0000 7f9d72a12740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ecb46350 con 0x5609ecbd8b60 2024-03-23T11:25:59.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.508+0000 7f9d72a12740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5609ecb41c40 con 0x5609ecbdcbb0 2024-03-23T11:25:59.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.508+0000 7f9d6de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc088f0 0x5609ecc08d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.508+0000 7f9d6de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc088f0 0x5609ecc08d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52500/0 (socket says 172.21.15.62:52500) 2024-03-23T11:25:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.508+0000 7f9d6de00640 1 -- 172.21.15.62:0/1999975132 learned_addr learned my addr 172.21.15.62:0/1999975132 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.509+0000 7f9d6ca00640 1 --2- 172.21.15.62:0/1999975132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecbdcbb0 0x5609ecbde650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.509+0000 7f9d6d400640 1 --2- 172.21.15.62:0/1999975132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecbd8b60 0x5609ecbd8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.509+0000 7f9d6de00640 1 -- 172.21.15.62:0/1999975132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecbdcbb0 msgr2=0x5609ecbde650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.509+0000 7f9d6de00640 1 --2- 172.21.15.62:0/1999975132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecbdcbb0 0x5609ecbde650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.509+0000 7f9d6de00640 1 -- 172.21.15.62:0/1999975132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecbd8b60 msgr2=0x5609ecbd8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.509+0000 7f9d6de00640 1 --2- 172.21.15.62:0/1999975132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecbd8b60 0x5609ecbd8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.509+0000 7f9d6de00640 1 -- 172.21.15.62:0/1999975132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609ecb0c3e0 con 0x5609ecc088f0 2024-03-23T11:25:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.509+0000 7f9d6de00640 1 --2- 172.21.15.62:0/1999975132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc088f0 0x5609ecc08d70 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f9d680adfd0 tx=0x7f9d60002150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0699176e3f76814 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.509+0000 7f9d5a000640 1 -- 172.21.15.62:0/1999975132 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60007c50 con 0x5609ecc088f0 2024-03-23T11:25:59.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.509+0000 7f9d5a000640 1 -- 172.21.15.62:0/1999975132 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6000c810 con 0x5609ecc088f0 2024-03-23T11:25:59.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.509+0000 7f9d5a000640 1 -- 172.21.15.62:0/1999975132 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6000cb80 con 0x5609ecc088f0 2024-03-23T11:25:59.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.510+0000 7f9d72a12740 1 -- 172.21.15.62:0/1999975132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc088f0 msgr2=0x5609ecc08d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.510+0000 7f9d72a12740 1 --2- 172.21.15.62:0/1999975132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc088f0 0x5609ecc08d70 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f9d680adfd0 tx=0x7f9d60002150 comp rx=0 tx=0).stop 2024-03-23T11:25:59.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.510+0000 7f9d72a12740 1 -- 172.21.15.62:0/1999975132 shutdown_connections 2024-03-23T11:25:59.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.510+0000 7f9d72a12740 1 --2- 172.21.15.62:0/1999975132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc088f0 0x5609ecc08d70 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.510+0000 7f9d72a12740 1 --2- 172.21.15.62:0/1999975132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecbdcbb0 0x5609ecbde650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.510+0000 7f9d72a12740 1 --2- 172.21.15.62:0/1999975132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecbd8b60 0x5609ecbd8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.510+0000 7f9d72a12740 1 -- 172.21.15.62:0/1999975132 >> 172.21.15.62:0/1999975132 conn(0x5609ecbdabe0 msgr2=0x5609ecbdc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:59.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.510+0000 7f9d72a12740 1 -- 172.21.15.62:0/1999975132 shutdown_connections 2024-03-23T11:25:59.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.510+0000 7f9d72a12740 1 -- 172.21.15.62:0/1999975132 wait complete. 2024-03-23T11:25:59.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.511+0000 7f9d72a12740 1 -- start start 2024-03-23T11:25:59.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.511+0000 7f9d72a12740 1 Processor -- start 2024-03-23T11:25:59.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.512+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc088f0 0x5609ecc082d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.512+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc041e0 0x5609ecc02110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.512+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc045c0 0x5609ecc02650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.512+0000 7f9d72a12740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ecae2ce0 con 0x5609ecc045c0 2024-03-23T11:25:59.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.512+0000 7f9d72a12740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ecae2980 con 0x5609ecc041e0 2024-03-23T11:25:59.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.512+0000 7f9d72a12740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5609ec8aa000 con 0x5609ecc088f0 2024-03-23T11:25:59.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.512+0000 7f9d72a12740 10 client.0 _ll_register_callbacks cb 0x5609ecb3b410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:59.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.512+0000 7f9d6de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc045c0 0x5609ecc02650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.512+0000 7f9d6de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc045c0 0x5609ecc02650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52504/0 (socket says 172.21.15.62:52504) 2024-03-23T11:25:59.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.512+0000 7f9d6de00640 1 -- 172.21.15.62:0/795378043 learned_addr learned my addr 172.21.15.62:0/795378043 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:59.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d6d400640 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc088f0 0x5609ecc082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d6ca00640 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc041e0 0x5609ecc02110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d6de00640 1 -- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc088f0 msgr2=0x5609ecc082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d6de00640 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc088f0 0x5609ecc082d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d6de00640 1 -- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc041e0 msgr2=0x5609ecc02110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d6de00640 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc041e0 0x5609ecc02110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d6de00640 1 -- 172.21.15.62:0/795378043 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d680b17a0 con 0x5609ecc045c0 2024-03-23T11:25:59.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d6de00640 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc045c0 0x5609ecc02650 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x5609ecc065c0 tx=0x7f9d600017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2685d31d9deff2e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:59.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:59.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d4ea00640 1 -- 172.21.15.62:0/795378043 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60019040 con 0x5609ecc045c0 2024-03-23T11:25:59.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d4ea00640 1 -- 172.21.15.62:0/795378043 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9d60007440 con 0x5609ecc045c0 2024-03-23T11:25:59.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.513+0000 7f9d72a12740 1 -- 172.21.15.62:0/795378043 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5609ec7f38d0 con 0x5609ecc045c0 2024-03-23T11:25:59.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.514+0000 7f9d4ea00640 1 -- 172.21.15.62:0/795378043 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d60007600 con 0x5609ecc045c0 2024-03-23T11:25:59.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.514+0000 7f9d72a12740 10 client.7131 fetch_fsmap learned FSMap version 50 2024-03-23T11:25:59.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.514+0000 7f9d72a12740 1 -- 172.21.15.62:0/795378043 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x5609ecb402c0 con 0x5609ecc045c0 2024-03-23T11:25:59.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.514+0000 7f9d4ea00640 1 -- 172.21.15.62:0/795378043 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5609ec7f38d0 con 0x5609ecc045c0 2024-03-23T11:25:59.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.514+0000 7f9d4ea00640 1 -- 172.21.15.62:0/795378043 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 50) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9d6001c070 con 0x5609ecc045c0 2024-03-23T11:25:59.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.514+0000 7f9d72a12740 10 client.7131 fetch_fsmap finished waiting for FSMap version 50 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.514+0000 7f9d72a12740 10 client.7131 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.514+0000 7f9d72a12740 20 client.7131 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.514+0000 7f9d72a12740 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x5609ecc06630 0x5609ecc06a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.514+0000 7f9d72a12740 4 client.7131 mds_command: new command op to 6618 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.514+0000 7f9d72a12740 1 -- 172.21.15.62:0/795378043 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5609ec8364a0 con 0x5609ecc06630 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.515+0000 7f9d6d400640 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x5609ecc06630 0x5609ecc06a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.516+0000 7f9d6d400640 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x5609ecc06630 0x5609ecc06a10 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec6f3a84c31c7c75 server_cookie=3fd9eb49f865a62f in_seq=0 out_seq=0 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.516+0000 7f9d4ea00640 10 client.7131 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.517+0000 7f9d4ea00640 1 -- 172.21.15.62:0/795378043 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5609ec8364a0 con 0x5609ecc06630 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.517+0000 7f9d4ea00640 10 client.7131 handle_command_reply: tid=0 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.517+0000 7f9d72a12740 1 client.7131 shutdown 2024-03-23T11:25:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.517+0000 7f9d72a12740 1 -- 172.21.15.62:0/795378043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc045c0 msgr2=0x5609ecc02650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.517+0000 7f9d72a12740 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc045c0 0x5609ecc02650 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x5609ecc065c0 tx=0x7f9d600017b0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.518+0000 7f9d72a12740 1 -- 172.21.15.62:0/795378043 shutdown_connections 2024-03-23T11:25:59.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.518+0000 7f9d72a12740 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x5609ecc06630 0x5609ecc06a10 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.518+0000 7f9d72a12740 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc045c0 0x5609ecc02650 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.518+0000 7f9d72a12740 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc041e0 0x5609ecc02110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.518+0000 7f9d72a12740 1 --2- 172.21.15.62:0/795378043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc088f0 0x5609ecc082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.518+0000 7f9d4ea00640 0 client.7131 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:59.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.518+0000 7f9d72a12740 1 -- 172.21.15.62:0/795378043 >> 172.21.15.62:0/795378043 conn(0x5609ecbd8b60 msgr2=0x5609ecbdeab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:59.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.518+0000 7f9d72a12740 1 -- 172.21.15.62:0/795378043 shutdown_connections 2024-03-23T11:25:59.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.518+0000 7f9d72a12740 1 -- 172.21.15.62:0/795378043 wait complete. 2024-03-23T11:25:59.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.518+0000 7f9d72a12740 20 client.7131 trim_cache size 0 max 16384 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.530+0000 7f9d72a12740 1 Processor -- start 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.531+0000 7f9d72a12740 1 -- start start 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.531+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecbdc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.531+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc045c0 0x5609ecc00590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.531+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc088f0 0x5609ecc25350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.531+0000 7f9d72a12740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ecc45730 con 0x5609ecc041e0 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.531+0000 7f9d72a12740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ecc32820 con 0x5609ecc088f0 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.531+0000 7f9d72a12740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5609ecc24e00 con 0x5609ecc045c0 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.532+0000 7f9d6d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecbdc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.532+0000 7f9d6d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecbdc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52510/0 (socket says 172.21.15.62:52510) 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.532+0000 7f9d6d400640 1 -- 172.21.15.62:0/1486627074 learned_addr learned my addr 172.21.15.62:0/1486627074 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.532+0000 7f9d6d400640 1 -- 172.21.15.62:0/1486627074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc045c0 msgr2=0x5609ecc00590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.532+0000 7f9d6ca00640 1 --2- 172.21.15.62:0/1486627074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc045c0 0x5609ecc00590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.532+0000 7f9d6d400640 1 --2- 172.21.15.62:0/1486627074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc045c0 0x5609ecc00590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.532+0000 7f9d6de00640 1 --2- 172.21.15.62:0/1486627074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc088f0 0x5609ecc25350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.532+0000 7f9d6d400640 1 -- 172.21.15.62:0/1486627074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc088f0 msgr2=0x5609ecc25350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.532+0000 7f9d6d400640 1 --2- 172.21.15.62:0/1486627074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc088f0 0x5609ecc25350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.532+0000 7f9d6d400640 1 -- 172.21.15.62:0/1486627074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609ecae2170 con 0x5609ecc041e0 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.533+0000 7f9d6d400640 1 --2- 172.21.15.62:0/1486627074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecbdc650 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0a9320 tx=0x7f9d5c006f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995ebbb0cf703d07 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.533+0000 7f9d5aa00640 1 -- 172.21.15.62:0/1486627074 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c079ff0 con 0x5609ecc041e0 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.533+0000 7f9d5aa00640 1 -- 172.21.15.62:0/1486627074 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c07de20 con 0x5609ecc041e0 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.533+0000 7f9d5aa00640 1 -- 172.21.15.62:0/1486627074 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c0823e0 con 0x5609ecc041e0 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.533+0000 7f9d72a12740 1 -- 172.21.15.62:0/1486627074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 msgr2=0x5609ecbdc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.533+0000 7f9d72a12740 1 --2- 172.21.15.62:0/1486627074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecbdc650 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0a9320 tx=0x7f9d5c006f40 comp rx=0 tx=0).stop 2024-03-23T11:25:59.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.533+0000 7f9d72a12740 1 -- 172.21.15.62:0/1486627074 shutdown_connections 2024-03-23T11:25:59.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.533+0000 7f9d72a12740 1 --2- 172.21.15.62:0/1486627074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc088f0 0x5609ecc25350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.533+0000 7f9d72a12740 1 --2- 172.21.15.62:0/1486627074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc045c0 0x5609ecc00590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.533+0000 7f9d72a12740 1 --2- 172.21.15.62:0/1486627074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecbdc650 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.533+0000 7f9d72a12740 1 -- 172.21.15.62:0/1486627074 >> 172.21.15.62:0/1486627074 conn(0x5609ecbd8b60 msgr2=0x5609ecbdd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:59.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.534+0000 7f9d72a12740 1 -- 172.21.15.62:0/1486627074 shutdown_connections 2024-03-23T11:25:59.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.534+0000 7f9d72a12740 1 -- 172.21.15.62:0/1486627074 wait complete. 2024-03-23T11:25:59.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.534+0000 7f9d72a12740 1 -- start start 2024-03-23T11:25:59.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.534+0000 7f9d72a12740 1 Processor -- start 2024-03-23T11:25:59.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.535+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecc0ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.535+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc045c0 0x5609ecc91800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.535+0000 7f9d72a12740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc088f0 0x5609ecc0c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.535+0000 7f9d72a12740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ecc0bf80 con 0x5609ecc041e0 2024-03-23T11:25:59.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.535+0000 7f9d72a12740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ecc91d40 con 0x5609ecc088f0 2024-03-23T11:25:59.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.535+0000 7f9d72a12740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5609ecc23400 con 0x5609ecc045c0 2024-03-23T11:25:59.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.535+0000 7f9d72a12740 10 client.0 _ll_register_callbacks cb 0x7f9d6000cd60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:25:59.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.536+0000 7f9d6d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecc0ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.536+0000 7f9d6d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecc0ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52516/0 (socket says 172.21.15.62:52516) 2024-03-23T11:25:59.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.536+0000 7f9d6d400640 1 -- 172.21.15.62:0/4079251863 learned_addr learned my addr 172.21.15.62:0/4079251863 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:59.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.536+0000 7f9d6d400640 1 -- 172.21.15.62:0/4079251863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc045c0 msgr2=0x5609ecc91800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.536+0000 7f9d6d400640 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc045c0 0x5609ecc91800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.536+0000 7f9d6d400640 1 -- 172.21.15.62:0/4079251863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc088f0 msgr2=0x5609ecc0c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.536+0000 7f9d6de00640 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc088f0 0x5609ecc0c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.536+0000 7f9d6d400640 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc088f0 0x5609ecc0c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.536+0000 7f9d6d400640 1 -- 172.21.15.62:0/4079251863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6813f3a0 con 0x5609ecc041e0 2024-03-23T11:25:59.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.536+0000 7f9d6d400640 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecc0ba40 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c000c00 tx=0x7f9d5c080040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa49a932eb0b43 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:59.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.536+0000 7f9d4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:25:59.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d4ea00640 1 -- 172.21.15.62:0/4079251863 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c087070 con 0x5609ecc041e0 2024-03-23T11:25:59.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d4ea00640 1 -- 172.21.15.62:0/4079251863 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9d5c0800b0 con 0x5609ecc041e0 2024-03-23T11:25:59.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d72a12740 1 -- 172.21.15.62:0/4079251863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d6000c810 con 0x5609ecc041e0 2024-03-23T11:25:59.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d4ea00640 1 -- 172.21.15.62:0/4079251863 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d5c078390 con 0x5609ecc041e0 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d72a12740 10 client.7137 fetch_fsmap learned FSMap version 50 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d72a12740 1 -- 172.21.15.62:0/4079251863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x5609ecb07a50 con 0x5609ecc041e0 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d4ea00640 1 -- 172.21.15.62:0/4079251863 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6000c810 con 0x5609ecc041e0 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d4ea00640 1 -- 172.21.15.62:0/4079251863 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 50) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9d5c08a020 con 0x5609ecc041e0 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d72a12740 10 client.7137 fetch_fsmap finished waiting for FSMap version 50 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d72a12740 10 client.7137 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d72a12740 20 client.7137 populate_metadata read hostname 'smithi062' 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d72a12740 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x5609ecc06a10 0x5609ecc06df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d72a12740 4 client.7137 mds_command: new command op to 6618 tid=0[{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.537+0000 7f9d72a12740 1 -- 172.21.15.62:0/4079251863 --> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x5609ec8477a0 con 0x5609ecc06a10 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.538+0000 7f9d6ca00640 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x5609ecc06a10 0x5609ecc06df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.539+0000 7f9d6ca00640 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x5609ecc06a10 0x5609ecc06df0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a943d02aa129932 server_cookie=8caae6ce9d84bf06 in_seq=0 out_seq=0 2024-03-23T11:25:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.539+0000 7f9d4ea00640 10 client.7137 ms_handle_connect on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d4ea00640 1 -- 172.21.15.62:0/4079251863 <== mds.0 v2:172.21.15.120:6834/1201988215 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5609ec8477a0 con 0x5609ecc06a10 2024-03-23T11:25:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d4ea00640 10 client.7137 handle_command_reply: tid=0 2024-03-23T11:25:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 1 client.7137 shutdown 2024-03-23T11:25:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 1 -- 172.21.15.62:0/4079251863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 msgr2=0x5609ecc0ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecc0ba40 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c000c00 tx=0x7f9d5c080040 comp rx=0 tx=0).stop 2024-03-23T11:25:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 1 -- 172.21.15.62:0/4079251863 shutdown_connections 2024-03-23T11:25:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.120:6834/1201988215,v1:172.21.15.120:6835/1201988215] conn(0x5609ecc06a10 0x5609ecc06df0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5609ecc088f0 0x5609ecc0c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5609ecc045c0 0x5609ecc91800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d4ea00640 0 client.7137 ms_handle_reset on v2:172.21.15.120:6834/1201988215 2024-03-23T11:25:59.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 1 --2- 172.21.15.62:0/4079251863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609ecc041e0 0x5609ecc0ba40 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 1 -- 172.21.15.62:0/4079251863 >> 172.21.15.62:0/4079251863 conn(0x5609ecbd8b60 msgr2=0x5609ecbd9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:59.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 1 -- 172.21.15.62:0/4079251863 shutdown_connections 2024-03-23T11:25:59.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 1 -- 172.21.15.62:0/4079251863 wait complete. 2024-03-23T11:25:59.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.540+0000 7f9d72a12740 20 client.7137 trim_cache size 0 max 16384 2024-03-23T11:25:59.567 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:25:59.567 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-03-23T11:25:59.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 -- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d4806a7a0 msgr2=0x7f9d4806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d4806a7a0 0x7f9d4806cc60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9d6806d740 tx=0x7f9d540079d0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 -- 172.21.15.62:0/3745860227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 msgr2=0x7f9d680aceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 0x7f9d680aceb0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c02c270 tx=0x7f9d5c004730 comp rx=0 tx=0).stop 2024-03-23T11:25:59.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 -- 172.21.15.62:0/3745860227 shutdown_connections 2024-03-23T11:25:59.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d4806a7a0 0x7f9d4806cc60 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d68140e80 0x7f9d6806cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d68140a00 0x7f9d6806ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 --2- 172.21.15.62:0/3745860227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d50003970 0x7f9d680aceb0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 -- 172.21.15.62:0/3745860227 >> 172.21.15.62:0/3745860227 conn(0x7f9d680b0090 msgr2=0x7f9d681424a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 -- 172.21.15.62:0/3745860227 shutdown_connections 2024-03-23T11:25:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.541+0000 7f9d6e800640 1 -- 172.21.15.62:0/3745860227 wait complete. 2024-03-23T11:25:59.570 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-23T11:25:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.802+0000 7f95a9e00640 1 Processor -- start 2024-03-23T11:25:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.802+0000 7f95a9e00640 1 -- start start 2024-03-23T11:25:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a9e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f95a406d230 0x7f95a406d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a9e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f95a4159df0 0x7f95a415a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a9e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a40711a0 con 0x7f95a40cd1b0 2024-03-23T11:25:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a9e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a406db50 con 0x7f95a4159df0 2024-03-23T11:25:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a9e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a4070100 con 0x7f95a406d230 2024-03-23T11:25:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f95a4159df0 0x7f95a415a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f95a4159df0 0x7f95a415a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52530/0 (socket says 172.21.15.62:52530) 2024-03-23T11:25:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a3e00640 1 -- 172.21.15.62:0/4062976002 learned_addr learned my addr 172.21.15.62:0/4062976002 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a8a00640 1 --2- 172.21.15.62:0/4062976002 >> v2:172.21.15.120:3300/0 conn(0x7f95a406d230 0x7f95a406d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a3e00640 1 -- 172.21.15.62:0/4062976002 >> v2:172.21.15.120:3300/0 conn(0x7f95a406d230 msgr2=0x7f95a406d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a3e00640 1 --2- 172.21.15.62:0/4062976002 >> v2:172.21.15.120:3300/0 conn(0x7f95a406d230 0x7f95a406d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a3e00640 1 -- 172.21.15.62:0/4062976002 >> v1:172.21.15.120:6789/0 conn(0x7f95a40cd1b0 legacy=0x7f95a40cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:25:59.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.804+0000 7f95a3e00640 1 -- 172.21.15.62:0/4062976002 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a40703e0 con 0x7f95a4159df0 2024-03-23T11:25:59.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.805+0000 7f95a8a00640 1 --2- 172.21.15.62:0/4062976002 >> v2:172.21.15.120:3300/0 conn(0x7f95a406d230 0x7f95a406d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:59.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.806+0000 7f95a3e00640 1 --2- 172.21.15.62:0/4062976002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95a4159df0 0x7f95a415a2a0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f95a406a990 tx=0x7f959002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36c13dfab6ce8ee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:59.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.806+0000 7f95a3400640 1 -- 172.21.15.62:0/4062976002 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590031870 con 0x7f95a4159df0 2024-03-23T11:25:59.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.806+0000 7f95a3400640 1 -- 172.21.15.62:0/4062976002 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95900319d0 con 0x7f95a4159df0 2024-03-23T11:25:59.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.806+0000 7f95a3400640 1 -- 172.21.15.62:0/4062976002 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590031d60 con 0x7f95a4159df0 2024-03-23T11:25:59.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.807+0000 7f95a9e00640 1 -- 172.21.15.62:0/4062976002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95a4159df0 msgr2=0x7f95a415a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.807+0000 7f95a9e00640 1 --2- 172.21.15.62:0/4062976002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95a4159df0 0x7f95a415a2a0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f95a406a990 tx=0x7f959002b910 comp rx=0 tx=0).stop 2024-03-23T11:25:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.807+0000 7f95a9e00640 1 -- 172.21.15.62:0/4062976002 shutdown_connections 2024-03-23T11:25:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.807+0000 7f95a9e00640 1 --2- 172.21.15.62:0/4062976002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95a4159df0 0x7f95a415a2a0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.807+0000 7f95a9e00640 1 --2- 172.21.15.62:0/4062976002 >> v2:172.21.15.120:3300/0 conn(0x7f95a406d230 0x7f95a406d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.807+0000 7f95a9e00640 1 -- 172.21.15.62:0/4062976002 >> 172.21.15.62:0/4062976002 conn(0x7f95a4066050 msgr2=0x7f95a4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:25:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.807+0000 7f95a9e00640 1 -- 172.21.15.62:0/4062976002 shutdown_connections 2024-03-23T11:25:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.807+0000 7f95a9e00640 1 -- 172.21.15.62:0/4062976002 wait complete. 2024-03-23T11:25:59.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.809+0000 7f95a9e00640 1 Processor -- start 2024-03-23T11:25:59.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.810+0000 7f95a9e00640 1 -- start start 2024-03-23T11:25:59.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.811+0000 7f95a9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95a406d230 0x7f95a40b2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.811+0000 7f95a9e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95a40cd1b0 0x7f95a4154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.811+0000 7f95a9e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95a4159df0 0x7f95a4155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.811+0000 7f95a9e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41f5e90 con 0x7f95a406d230 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.811+0000 7f95a9e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41579e0 con 0x7f95a40cd1b0 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.811+0000 7f95a9e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41f62b0 con 0x7f95a4159df0 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.811+0000 7f95a8a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95a406d230 0x7f95a40b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.811+0000 7f95a8a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95a406d230 0x7f95a40b2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52536/0 (socket says 172.21.15.62:52536) 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.811+0000 7f95a8a00640 1 -- 172.21.15.62:0/3806759891 learned_addr learned my addr 172.21.15.62:0/3806759891 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a3e00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95a40cd1b0 0x7f95a4154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a9400640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95a4159df0 0x7f95a4155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a8a00640 1 -- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95a4159df0 msgr2=0x7f95a4155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a8a00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95a4159df0 0x7f95a4155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a8a00640 1 -- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95a40cd1b0 msgr2=0x7f95a4154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:25:59.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a8a00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95a40cd1b0 0x7f95a4154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:25:59.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a8a00640 1 -- 172.21.15.62:0/3806759891 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95900094d0 con 0x7f95a406d230 2024-03-23T11:25:59.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a9400640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95a4159df0 0x7f95a4155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:25:59.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a8a00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95a406d230 0x7f95a40b2730 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f959800c7f0 tx=0x7f959800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:59.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a0c00640 1 -- 172.21.15.62:0/3806759891 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95980077c0 con 0x7f95a406d230 2024-03-23T11:25:59.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a0c00640 1 -- 172.21.15.62:0/3806759891 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9598007920 con 0x7f95a406d230 2024-03-23T11:25:59.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.812+0000 7f95a0c00640 1 -- 172.21.15.62:0/3806759891 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598007c90 con 0x7f95a406d230 2024-03-23T11:25:59.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.813+0000 7f95a9e00640 1 -- 172.21.15.62:0/3806759891 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a41f54f0 con 0x7f95a406d230 2024-03-23T11:25:59.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.813+0000 7f95a9e00640 1 -- 172.21.15.62:0/3806759891 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a41f5ab0 con 0x7f95a406d230 2024-03-23T11:25:59.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.814+0000 7f95a9e00640 1 -- 172.21.15.62:0/3806759891 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a404e030 con 0x7f95a406d230 2024-03-23T11:25:59.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.815+0000 7f95a0c00640 1 -- 172.21.15.62:0/3806759891 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f95980113f0 con 0x7f95a406d230 2024-03-23T11:25:59.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.816+0000 7f95a0c00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f958406a7a0 0x7f958406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:25:59.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.816+0000 7f95a0c00640 1 -- 172.21.15.62:0/3806759891 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9598089130 con 0x7f95a406d230 2024-03-23T11:25:59.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.816+0000 7f95a3e00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f958406a7a0 0x7f958406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:25:59.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.818+0000 7f95a3e00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f958406a7a0 0x7f958406cc60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9590036000 tx=0x7f959003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:25:59.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:25:59.821+0000 7f95a0c00640 1 -- 172.21.15.62:0/3806759891 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f95980544b0 con 0x7f95a406d230 2024-03-23T11:26:00.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.063+0000 7f95a9e00640 1 -- 172.21.15.62:0/3806759891 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f95a40cbd10 con 0x7f95a406d230 2024-03-23T11:26:00.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.064+0000 7f95a0c00640 1 -- 172.21.15.62:0/3806759891 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f9598057ad0 con 0x7f95a406d230 2024-03-23T11:26:00.058 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:00.058 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-23T11:26:00.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.066+0000 7f95a9e00640 1 -- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f958406a7a0 msgr2=0x7f958406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:00.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.066+0000 7f95a9e00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f958406a7a0 0x7f958406cc60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9590036000 tx=0x7f959003c040 comp rx=0 tx=0).stop 2024-03-23T11:26:00.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.066+0000 7f95a9e00640 1 -- 172.21.15.62:0/3806759891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95a406d230 msgr2=0x7f95a40b2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:00.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.066+0000 7f95a9e00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95a406d230 0x7f95a40b2730 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f959800c7f0 tx=0x7f959800ccb0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.067+0000 7f95a9e00640 1 -- 172.21.15.62:0/3806759891 shutdown_connections 2024-03-23T11:26:00.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.067+0000 7f95a9e00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f958406a7a0 0x7f958406cc60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.067+0000 7f95a9e00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f95a4159df0 0x7f95a4155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.067+0000 7f95a9e00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95a40cd1b0 0x7f95a4154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.067+0000 7f95a9e00640 1 --2- 172.21.15.62:0/3806759891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95a406d230 0x7f95a40b2730 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.067+0000 7f95a9e00640 1 -- 172.21.15.62:0/3806759891 >> 172.21.15.62:0/3806759891 conn(0x7f95a4066050 msgr2=0x7f95a4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:00.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.067+0000 7f95a9e00640 1 -- 172.21.15.62:0/3806759891 shutdown_connections 2024-03-23T11:26:00.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.067+0000 7f95a9e00640 1 -- 172.21.15.62:0/3806759891 wait complete. 2024-03-23T11:26:00.080 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-23T11:26:00.081 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:26:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.307+0000 7f07ac800640 1 Processor -- start 2024-03-23T11:26:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.307+0000 7f07ac800640 1 -- start start 2024-03-23T11:26:00.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.309+0000 7f07ac800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f07a40cafa0 0x7f07a40cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:00.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.309+0000 7f07ac800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07a406c6e0 0x7f07a406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:00.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.309+0000 7f07ac800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40cbc60 con 0x7f07a406c300 2024-03-23T11:26:00.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.309+0000 7f07ac800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a40cd410 con 0x7f07a40cafa0 2024-03-23T11:26:00.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.309+0000 7f07ac800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40cbf40 con 0x7f07a406c6e0 2024-03-23T11:26:00.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.310+0000 7f07aa800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07a406c6e0 0x7f07a406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:00.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.310+0000 7f07a9400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f07a40cafa0 0x7f07a40cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:00.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.310+0000 7f07a9400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f07a40cafa0 0x7f07a40cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:57870/0 (socket says 172.21.15.62:57870) 2024-03-23T11:26:00.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.310+0000 7f07a9400640 1 -- 172.21.15.62:0/3203395621 learned_addr learned my addr 172.21.15.62:0/3203395621 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.310+0000 7f07a9400640 1 -- 172.21.15.62:0/3203395621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07a406c6e0 msgr2=0x7f07a406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.310+0000 7f07a9400640 1 --2- 172.21.15.62:0/3203395621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07a406c6e0 0x7f07a406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.311+0000 7f07a9400640 1 -- 172.21.15.62:0/3203395621 >> v1:172.21.15.120:6789/0 conn(0x7f07a406c300 legacy=0x7f07a40ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.311+0000 7f07a9400640 1 -- 172.21.15.62:0/3203395621 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40cc220 con 0x7f07a40cafa0 2024-03-23T11:26:00.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.312+0000 7f07a9400640 1 --2- 172.21.15.62:0/3203395621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07a40cafa0 0x7f07a40cb430 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0794009870 tx=0x7f079402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d2a8ddcd4e5ca83 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:00.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.312+0000 7f07a8a00640 1 -- 172.21.15.62:0/3203395621 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794030840 con 0x7f07a40cafa0 2024-03-23T11:26:00.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.312+0000 7f07a8a00640 1 -- 172.21.15.62:0/3203395621 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07940309a0 con 0x7f07a40cafa0 2024-03-23T11:26:00.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.312+0000 7f07a8a00640 1 -- 172.21.15.62:0/3203395621 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794030d30 con 0x7f07a40cafa0 2024-03-23T11:26:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.313+0000 7f07ac800640 1 -- 172.21.15.62:0/3203395621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07a40cafa0 msgr2=0x7f07a40cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.313+0000 7f07ac800640 1 --2- 172.21.15.62:0/3203395621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07a40cafa0 0x7f07a40cb430 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0794009870 tx=0x7f079402b980 comp rx=0 tx=0).stop 2024-03-23T11:26:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.313+0000 7f07ac800640 1 -- 172.21.15.62:0/3203395621 shutdown_connections 2024-03-23T11:26:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.313+0000 7f07ac800640 1 --2- 172.21.15.62:0/3203395621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07a406c6e0 0x7f07a406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.313+0000 7f07ac800640 1 --2- 172.21.15.62:0/3203395621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07a40cafa0 0x7f07a40cb430 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.313+0000 7f07ac800640 1 -- 172.21.15.62:0/3203395621 >> 172.21.15.62:0/3203395621 conn(0x7f07a40b02e0 msgr2=0x7f07a40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.313+0000 7f07ac800640 1 -- 172.21.15.62:0/3203395621 shutdown_connections 2024-03-23T11:26:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.313+0000 7f07ac800640 1 -- 172.21.15.62:0/3203395621 wait complete. 2024-03-23T11:26:00.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.316+0000 7f07ac800640 1 Processor -- start 2024-03-23T11:26:00.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.316+0000 7f07ac800640 1 -- start start 2024-03-23T11:26:00.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.317+0000 7f07ac800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a406c300 0x7f07a415d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:00.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.317+0000 7f07ac800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07a406c6e0 0x7f07a41591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.317+0000 7f07ac800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07a40cafa0 0x7f07a41f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.317+0000 7f07ac800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40b2750 con 0x7f07a406c300 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.317+0000 7f07ac800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a41596e0 con 0x7f07a406c6e0 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.317+0000 7f07ac800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40b2420 con 0x7f07a40cafa0 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.317+0000 7f07a9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a406c300 0x7f07a415d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.317+0000 7f07a9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a406c300 0x7f07a415d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52540/0 (socket says 172.21.15.62:52540) 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.317+0000 7f07a9e00640 1 -- 172.21.15.62:0/122508757 learned_addr learned my addr 172.21.15.62:0/122508757 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.318+0000 7f07a9400640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07a406c6e0 0x7f07a41591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.318+0000 7f07aa800640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07a40cafa0 0x7f07a41f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.318+0000 7f07a9e00640 1 -- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07a40cafa0 msgr2=0x7f07a41f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.318+0000 7f07a9e00640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07a40cafa0 0x7f07a41f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.318+0000 7f07a9e00640 1 -- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07a406c6e0 msgr2=0x7f07a41591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.318+0000 7f07a9e00640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07a406c6e0 0x7f07a41591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.318+0000 7f07a9e00640 1 -- 172.21.15.62:0/122508757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07940094d0 con 0x7f07a406c300 2024-03-23T11:26:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.318+0000 7f07a9400640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07a406c6e0 0x7f07a41591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:26:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.318+0000 7f07a9e00640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a406c300 0x7f07a415d3b0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f079800c450 tx=0x7f079800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:00.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.319+0000 7f079e000640 1 -- 172.21.15.62:0/122508757 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07980052d0 con 0x7f07a406c300 2024-03-23T11:26:00.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.319+0000 7f079e000640 1 -- 172.21.15.62:0/122508757 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798005430 con 0x7f07a406c300 2024-03-23T11:26:00.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.319+0000 7f079e000640 1 -- 172.21.15.62:0/122508757 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798012870 con 0x7f07a406c300 2024-03-23T11:26:00.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.319+0000 7f07ac800640 1 -- 172.21.15.62:0/122508757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a41f2910 con 0x7f07a406c300 2024-03-23T11:26:00.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.319+0000 7f07ac800640 1 -- 172.21.15.62:0/122508757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a41f2ed0 con 0x7f07a406c300 2024-03-23T11:26:00.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.320+0000 7f07ac800640 1 -- 172.21.15.62:0/122508757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a40ce100 con 0x7f07a406c300 2024-03-23T11:26:00.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.321+0000 7f079e000640 1 -- 172.21.15.62:0/122508757 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f07980129d0 con 0x7f07a406c300 2024-03-23T11:26:00.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.323+0000 7f079e000640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f078806a7a0 0x7f078806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:00.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.323+0000 7f079e000640 1 -- 172.21.15.62:0/122508757 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0798088a30 con 0x7f07a406c300 2024-03-23T11:26:00.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.323+0000 7f07a9400640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f078806a7a0 0x7f078806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:00.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.324+0000 7f07a9400640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f078806a7a0 0x7f078806cc60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0794036000 tx=0x7f079403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:00.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.329+0000 7f079e000640 1 -- 172.21.15.62:0/122508757 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0798053d20 con 0x7f07a406c300 2024-03-23T11:26:00.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.570+0000 7f07ac800640 1 -- 172.21.15.62:0/122508757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f07a40cb940 con 0x7f07a406c300 2024-03-23T11:26:00.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.571+0000 7f079e000640 1 -- 172.21.15.62:0/122508757 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0798057340 con 0x7f07a406c300 2024-03-23T11:26:00.565 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:26:00.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 -- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f078806a7a0 msgr2=0x7f078806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:00.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f078806a7a0 0x7f078806cc60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0794036000 tx=0x7f079403b040 comp rx=0 tx=0).stop 2024-03-23T11:26:00.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 -- 172.21.15.62:0/122508757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a406c300 msgr2=0x7f07a415d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:00.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a406c300 0x7f07a415d3b0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f079800c450 tx=0x7f079800c910 comp rx=0 tx=0).stop 2024-03-23T11:26:00.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 -- 172.21.15.62:0/122508757 shutdown_connections 2024-03-23T11:26:00.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f078806a7a0 0x7f078806cc60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07a40cafa0 0x7f07a41f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07a406c6e0 0x7f07a41591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 --2- 172.21.15.62:0/122508757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a406c300 0x7f07a415d3b0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:00.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 -- 172.21.15.62:0/122508757 >> 172.21.15.62:0/122508757 conn(0x7f07a40b02e0 msgr2=0x7f07a413eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:00.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 -- 172.21.15.62:0/122508757 shutdown_connections 2024-03-23T11:26:00.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:00.574+0000 7f07ac800640 1 -- 172.21.15.62:0/122508757 wait complete. 2024-03-23T11:26:00.586 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:26:00.586 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:26:00.586 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:26:00.586 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:26:00.586 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:00.586 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:26:00.586 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:26:00.587 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:26:00.587 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:26:00.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:00.587 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:26:00.643 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:26:00.644 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42196sec preferred_lft 42196sec 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:26:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:00.646 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:26:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:26:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:26:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:00.646 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:26:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:26:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:26:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:00.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:00.647 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:26:00.743 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:26:00.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:00.748 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:26:00.804 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:26:00.805 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:26:00.805 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:26:00.805 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:26:00.805 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:26:00.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:00.806 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:26:00.806 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:26:00.806 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:26:00.806 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:26:00.806 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:26:00.806 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:26:01.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:01.006 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:26:01.059 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:26:01.059 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:26:01.060 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:26:01.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:01.061 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:26:01.162 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:26:01.162 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:26:01.162 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:01.217 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:26:01.217 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:26:01.218 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:01.271 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:26:01.374 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:26:01.374 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:26:01.476 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:26:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:26:01.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:01.480 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:26:01.538 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:26:01.538 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:26:01.543 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:26:01.618 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:26:01.618 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:26:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:26:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:26:01.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:01.694 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:26:01.731 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:26:01.737+0000 7fbd91eb7480 -1 init, newargv = 0x558a4b6fa310 newargc=15 2024-03-23T11:26:01.731 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[63379]: starting ceph client 2024-03-23T11:26:01.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[63379]: starting fuse 2024-03-23T11:26:02.837 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:26:02.947 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:26:02.947 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:26:03.045 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:26:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:26:03.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:03.051 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:26:03.108 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:26:03.109 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:26:03.109 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:26:03.109 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:26:03.109 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:26:03.109 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:26:03.109 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:26:03.109 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:03.109 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:26:03.109 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:26:03.110 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:26:03.111 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:03.111 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:26:03.111 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:26:03.293 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.63379.asok 2024-03-23T11:26:03.473 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.63379.asok 2024-03-23T11:26:03.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:03.474 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.63379.asok status 2024-03-23T11:26:03.724 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:26:03.724 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:26:03.724 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "63379", 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 7152, 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:26:03.725 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 7152 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 359329794 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 359329794 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.7152 192.168.0.1:0/359329794", 2024-03-23T11:26:03.726 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/359329794", 2024-03-23T11:26:03.727 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:26:03.727 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 50, 2024-03-23T11:26:03.727 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 77, 2024-03-23T11:26:03.727 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:26:03.727 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:26:03.727 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:26:03.727 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:26:03.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:03.740 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:26:03.740 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:03.793 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:26:03.793 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:03.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:03.793 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:03.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:03.889 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-03-23T11:26:03.952 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:26:03.953 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-03-23T11:26:04.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.069+0000 7fb0655fef40 1 Processor -- start 2024-03-23T11:26:04.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.069+0000 7fb0655fef40 1 -- start start 2024-03-23T11:26:04.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.070+0000 7fb0655fef40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc1af71920 0x55fc1af8c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.070+0000 7fb0655fef40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 0x55fc1b0109c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.070+0000 7fb0655fef40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fc1b016720 0x55fc1b010f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.070+0000 7fb0655fef40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc1ae86800 con 0x55fc1b015c30 2024-03-23T11:26:04.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.070+0000 7fb0655fef40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc1ae864a0 con 0x55fc1af71920 2024-03-23T11:26:04.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.070+0000 7fb0655fef40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fc1ae7d110 con 0x55fc1b016720 2024-03-23T11:26:04.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.070+0000 7fb061e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 0x55fc1b0109c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.071+0000 7fb061e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 0x55fc1b0109c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52552/0 (socket says 172.21.15.62:52552) 2024-03-23T11:26:04.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.071+0000 7fb061e00640 1 -- 172.21.15.62:0/2822725071 learned_addr learned my addr 172.21.15.62:0/2822725071 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:04.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.071+0000 7fb062800640 1 --2- 172.21.15.62:0/2822725071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc1af71920 0x55fc1af8c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.071+0000 7fb063200640 1 --2- 172.21.15.62:0/2822725071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fc1b016720 0x55fc1b010f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.071+0000 7fb061e00640 1 -- 172.21.15.62:0/2822725071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fc1b016720 msgr2=0x55fc1b010f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:04.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.071+0000 7fb061e00640 1 --2- 172.21.15.62:0/2822725071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fc1b016720 0x55fc1b010f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.071+0000 7fb061e00640 1 -- 172.21.15.62:0/2822725071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc1af71920 msgr2=0x55fc1af8c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:04.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.071+0000 7fb061e00640 1 --2- 172.21.15.62:0/2822725071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc1af71920 0x55fc1af8c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.071+0000 7fb061e00640 1 -- 172.21.15.62:0/2822725071 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c005de0 con 0x55fc1b015c30 2024-03-23T11:26:04.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.071+0000 7fb061e00640 1 --2- 172.21.15.62:0/2822725071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 0x55fc1b0109c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb050001e30 tx=0x7fb05000b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47e3f0346c0ce70 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.072+0000 7fb060a00640 1 -- 172.21.15.62:0/2822725071 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050014070 con 0x55fc1b015c30 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.072+0000 7fb060a00640 1 -- 172.21.15.62:0/2822725071 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb050008030 con 0x55fc1b015c30 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.072+0000 7fb060a00640 1 -- 172.21.15.62:0/2822725071 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05000f400 con 0x55fc1b015c30 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.072+0000 7fb0655fef40 1 -- 172.21.15.62:0/2822725071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 msgr2=0x55fc1b0109c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.072+0000 7fb0655fef40 1 --2- 172.21.15.62:0/2822725071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 0x55fc1b0109c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb050001e30 tx=0x7fb05000b670 comp rx=0 tx=0).stop 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.072+0000 7fb0655fef40 1 -- 172.21.15.62:0/2822725071 shutdown_connections 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.072+0000 7fb0655fef40 1 --2- 172.21.15.62:0/2822725071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 0x55fc1b0109c0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.072+0000 7fb0655fef40 1 --2- 172.21.15.62:0/2822725071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fc1b016720 0x55fc1b010f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.072+0000 7fb0655fef40 1 --2- 172.21.15.62:0/2822725071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc1af71920 0x55fc1af8c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.072+0000 7fb0655fef40 1 -- 172.21.15.62:0/2822725071 >> 172.21.15.62:0/2822725071 conn(0x55fc1af8c000 msgr2=0x55fc1b00f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.072+0000 7fb0655fef40 1 -- 172.21.15.62:0/2822725071 shutdown_connections 2024-03-23T11:26:04.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.073+0000 7fb0655fef40 1 -- 172.21.15.62:0/2822725071 wait complete. 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.073+0000 7fb0655fef40 1 Processor -- start 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.074+0000 7fb0655fef40 1 -- start start 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.074+0000 7fb0655fef40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc1af71920 0x55fc1af3c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.074+0000 7fb0655fef40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 0x55fc1b00b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.074+0000 7fb0655fef40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fc1b016720 0x55fc1b00bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.074+0000 7fb0655fef40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc1b00c580 con 0x55fc1b015c30 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.074+0000 7fb0655fef40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc1b00a9c0 con 0x55fc1af71920 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.074+0000 7fb0655fef40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fc1b00aca0 con 0x55fc1b016720 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.075+0000 7fb061e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 0x55fc1b00b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.075+0000 7fb061e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 0x55fc1b00b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52566/0 (socket says 172.21.15.62:52566) 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.075+0000 7fb061e00640 1 -- 172.21.15.62:0/3117535915 learned_addr learned my addr 172.21.15.62:0/3117535915 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:04.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.075+0000 7fb063200640 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fc1b016720 0x55fc1b00bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.075+0000 7fb062800640 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc1af71920 0x55fc1af3c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.075+0000 7fb061e00640 1 -- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fc1b016720 msgr2=0x55fc1b00bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.075+0000 7fb061e00640 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55fc1b016720 0x55fc1b00bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.075+0000 7fb061e00640 1 -- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc1af71920 msgr2=0x55fc1af3c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.075+0000 7fb061e00640 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc1af71920 0x55fc1af3c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.075+0000 7fb061e00640 1 -- 172.21.15.62:0/3117535915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05000a040 con 0x55fc1b015c30 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.075+0000 7fb062800640 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc1af71920 0x55fc1af3c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.076+0000 7fb061e00640 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 0x55fc1b00b9f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x55fc1af8bdf0 tx=0x7fb050000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.076+0000 7fb05b400640 1 -- 172.21.15.62:0/3117535915 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050014040 con 0x55fc1b015c30 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.076+0000 7fb05b400640 1 -- 172.21.15.62:0/3117535915 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05001b430 con 0x55fc1b015c30 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.076+0000 7fb05b400640 1 -- 172.21.15.62:0/3117535915 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05001b7e0 con 0x55fc1b015c30 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.076+0000 7fb0655fef40 1 -- 172.21.15.62:0/3117535915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fc1ae95340 con 0x55fc1b015c30 2024-03-23T11:26:04.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.076+0000 7fb0655fef40 1 -- 172.21.15.62:0/3117535915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc1ae88780 con 0x55fc1b015c30 2024-03-23T11:26:04.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.078+0000 7fb05b400640 1 -- 172.21.15.62:0/3117535915 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb05001b940 con 0x55fc1b015c30 2024-03-23T11:26:04.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.079+0000 7fb05b400640 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0440691b0 0x7fb04406b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.079+0000 7fb05b400640 1 -- 172.21.15.62:0/3117535915 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb050089080 con 0x55fc1b015c30 2024-03-23T11:26:04.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.080+0000 7fb0655fef40 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55fc1af702c0 0x55fc1b00de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.080+0000 7fb0655fef40 1 -- 172.21.15.62:0/3117535915 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:1 10.10 10:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) v8 -- 0x55fc1b00e370 con 0x55fc1af702c0 2024-03-23T11:26:04.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.080+0000 7fb062800640 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0440691b0 0x7fb04406b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.080+0000 7fb063200640 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55fc1af702c0 0x55fc1b00de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.081+0000 7fb062800640 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0440691b0 0x7fb04406b670 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb054005fd0 tx=0x7fb054007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:04.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.081+0000 7fb063200640 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55fc1af702c0 0x55fc1b00de30 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:04.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb063200640 1 -- 172.21.15.62:0/3117535915 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v77'12 uv12 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fb05c0525b0 con 0x55fc1af702c0 2024-03-23T11:26:04.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb0655fef40 1 -- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55fc1af702c0 msgr2=0x55fc1b00de30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb0655fef40 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55fc1af702c0 0x55fc1b00de30 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb0655fef40 1 -- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0440691b0 msgr2=0x7fb04406b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb0655fef40 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0440691b0 0x7fb04406b670 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb054005fd0 tx=0x7fb054007040 comp rx=0 tx=0).stop 2024-03-23T11:26:04.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb0655fef40 1 -- 172.21.15.62:0/3117535915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 msgr2=0x55fc1b00b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb0655fef40 1 --2- 172.21.15.62:0/3117535915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fc1b015c30 0x55fc1b00b9f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x55fc1af8bdf0 tx=0x7fb050000ed0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb063200640 1 -- 172.21.15.62:0/3117535915 reap_dead start 2024-03-23T11:26:04.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb0655fef40 1 -- 172.21.15.62:0/3117535915 shutdown_connections 2024-03-23T11:26:04.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb0655fef40 1 -- 172.21.15.62:0/3117535915 >> 172.21.15.62:0/3117535915 conn(0x55fc1af8c000 msgr2=0x55fc1af8c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:04.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb0655fef40 1 -- 172.21.15.62:0/3117535915 shutdown_connections 2024-03-23T11:26:04.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.085+0000 7fb0655fef40 1 -- 172.21.15.62:0/3117535915 wait complete. 2024-03-23T11:26:04.084 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:26:04.085 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:26:04.203 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:26:04.321 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:26:04.321 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:26:04.321 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:26:04.382 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:26:04.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.611+0000 7f19b3e00640 1 Processor -- start 2024-03-23T11:26:04.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.611+0000 7f19b3e00640 1 -- start start 2024-03-23T11:26:04.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.612+0000 7f19b3e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f19b406cc60 0x7f19b40cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.612+0000 7f19b3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f19b40ce420 0x7f19b40b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b3e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b40ced90 con 0x7f19b4159490 2024-03-23T11:26:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b3e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b406e3e0 con 0x7f19b40ce420 2024-03-23T11:26:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b3e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b40ce850 con 0x7f19b406cc60 2024-03-23T11:26:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b2a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f19b4159490 0x7f19b415da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53230/0 (socket says 172.21.15.62:53230) 2024-03-23T11:26:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b2a00640 1 -- 172.21.15.62:0/1774364387 learned_addr learned my addr 172.21.15.62:0/1774364387 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b3400640 1 --2- 172.21.15.62:0/1774364387 >> v2:172.21.15.62:3300/0 conn(0x7f19b40ce420 0x7f19b40b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b2000640 1 --2- 172.21.15.62:0/1774364387 >> v2:172.21.15.120:3300/0 conn(0x7f19b406cc60 0x7f19b40cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b1600640 1 -- 172.21.15.62:0/1774364387 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590286243 0 0) 0x7f19b40ced90 con 0x7f19b4159490 2024-03-23T11:26:04.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b1600640 1 -- 172.21.15.62:0/1774364387 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1998005730 con 0x7f19b4159490 2024-03-23T11:26:04.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b3400640 1 -- 172.21.15.62:0/1774364387 >> v2:172.21.15.120:3300/0 conn(0x7f19b406cc60 msgr2=0x7f19b40cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:04.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b3400640 1 --2- 172.21.15.62:0/1774364387 >> v2:172.21.15.120:3300/0 conn(0x7f19b406cc60 0x7f19b40cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.613+0000 7f19b3400640 1 -- 172.21.15.62:0/1774364387 >> v1:172.21.15.62:6789/0 conn(0x7f19b4159490 legacy=0x7f19b415da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.614+0000 7f19b3400640 1 -- 172.21.15.62:0/1774364387 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b406c1c0 con 0x7f19b40ce420 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.615+0000 7f19b3400640 1 --2- 172.21.15.62:0/1774364387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b40ce420 0x7f19b40b26b0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f19b4071280 tx=0x7f19ac02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6a8aecf9e4c791 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.615+0000 7f19b1600640 1 -- 172.21.15.62:0/1774364387 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac02cbf0 con 0x7f19b40ce420 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.615+0000 7f19b1600640 1 -- 172.21.15.62:0/1774364387 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac02cd50 con 0x7f19b40ce420 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.615+0000 7f19b1600640 1 -- 172.21.15.62:0/1774364387 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac034640 con 0x7f19b40ce420 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.615+0000 7f19b3e00640 1 -- 172.21.15.62:0/1774364387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b40ce420 msgr2=0x7f19b40b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.616+0000 7f19b3e00640 1 --2- 172.21.15.62:0/1774364387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b40ce420 0x7f19b40b26b0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f19b4071280 tx=0x7f19ac02be40 comp rx=0 tx=0).stop 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.616+0000 7f19b3e00640 1 -- 172.21.15.62:0/1774364387 shutdown_connections 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.616+0000 7f19b3e00640 1 --2- 172.21.15.62:0/1774364387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b40ce420 0x7f19b40b26b0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.616+0000 7f19b3e00640 1 --2- 172.21.15.62:0/1774364387 >> v2:172.21.15.120:3300/0 conn(0x7f19b406cc60 0x7f19b40cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.616+0000 7f19b3e00640 1 -- 172.21.15.62:0/1774364387 >> 172.21.15.62:0/1774364387 conn(0x7f19b4066050 msgr2=0x7f19b4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.616+0000 7f19b3e00640 1 -- 172.21.15.62:0/1774364387 shutdown_connections 2024-03-23T11:26:04.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.616+0000 7f19b3e00640 1 -- 172.21.15.62:0/1774364387 wait complete. 2024-03-23T11:26:04.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.617+0000 7f19b3e00640 1 Processor -- start 2024-03-23T11:26:04.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.618+0000 7f19b3e00640 1 -- start start 2024-03-23T11:26:04.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19b406cc60 0x7f19b41edbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19b40ce420 0x7f19b40b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b4159490 0x7f19b4141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b415a2b0 con 0x7f19b4159490 2024-03-23T11:26:04.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b40b17c0 con 0x7f19b40ce420 2024-03-23T11:26:04.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b4159910 con 0x7f19b406cc60 2024-03-23T11:26:04.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b4159490 0x7f19b4141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b4159490 0x7f19b4141aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52582/0 (socket says 172.21.15.62:52582) 2024-03-23T11:26:04.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3400640 1 -- 172.21.15.62:0/2236154050 learned_addr learned my addr 172.21.15.62:0/2236154050 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:04.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b2000640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19b40ce420 0x7f19b40b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b2a00640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19b406cc60 0x7f19b41edbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3400640 1 -- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19b406cc60 msgr2=0x7f19b41edbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3400640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19b406cc60 0x7f19b41edbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3400640 1 -- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19b40ce420 msgr2=0x7f19b40b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3400640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19b40ce420 0x7f19b40b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b3400640 1 -- 172.21.15.62:0/2236154050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac009a70 con 0x7f19b4159490 2024-03-23T11:26:04.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.619+0000 7f19b2000640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19b40ce420 0x7f19b40b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:04.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.620+0000 7f19b2a00640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19b406cc60 0x7f19b41edbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:04.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.620+0000 7f19b3400640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b4159490 0x7f19b4141aa0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f19ac02c450 tx=0x7f19ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:04.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.620+0000 7f19aaa00640 1 -- 172.21.15.62:0/2236154050 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac03e610 con 0x7f19b4159490 2024-03-23T11:26:04.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.620+0000 7f19aaa00640 1 -- 172.21.15.62:0/2236154050 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac03e770 con 0x7f19b4159490 2024-03-23T11:26:04.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.620+0000 7f19aaa00640 1 -- 172.21.15.62:0/2236154050 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac03eb60 con 0x7f19b4159490 2024-03-23T11:26:04.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.620+0000 7f19b3e00640 1 -- 172.21.15.62:0/2236154050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b4159bf0 con 0x7f19b4159490 2024-03-23T11:26:04.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.620+0000 7f19b3e00640 1 -- 172.21.15.62:0/2236154050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b40b1490 con 0x7f19b4159490 2024-03-23T11:26:04.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.622+0000 7f19b3e00640 1 -- 172.21.15.62:0/2236154050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b415a3f0 con 0x7f19b4159490 2024-03-23T11:26:04.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.622+0000 7f19aaa00640 1 -- 172.21.15.62:0/2236154050 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f19ac03b050 con 0x7f19b4159490 2024-03-23T11:26:04.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.623+0000 7f19aaa00640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f199406a7a0 0x7f199406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:04.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.623+0000 7f19aaa00640 1 -- 172.21.15.62:0/2236154050 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f19ac0ab080 con 0x7f19b4159490 2024-03-23T11:26:04.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.624+0000 7f19b2a00640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f199406a7a0 0x7f199406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:04.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.625+0000 7f19b2a00640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f199406a7a0 0x7f199406cc60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f19a4005880 tx=0x7f19a4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.628+0000 7f19aaa00640 1 -- 172.21.15.62:0/2236154050 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f19ac0de580 con 0x7f19b4159490 2024-03-23T11:26:04.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.828+0000 7f19b3e00640 1 -- 172.21.15.62:0/2236154050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19b415c980 con 0x7f19b4159490 2024-03-23T11:26:04.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.831+0000 7f19aaa00640 1 -- 172.21.15.62:0/2236154050 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16329 (secure 0 0 0) 0x7f19b415a3f0 con 0x7f19b4159490 2024-03-23T11:26:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:04.825 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":77,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:44.311392+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:24:08.688706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-23T11:24:10.072774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1c","mappings":[{"from":6,"to":0}]},{"pgid":"11.3c","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6837/3833782560":"2024-03-24T11:24:34.350690+0000","172.21.15.120:6836/3833782560":"2024-03-24T11:24:34.350690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 -- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f199406a7a0 msgr2=0x7f199406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f199406a7a0 0x7f199406cc60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f19a4005880 tx=0x7f19a4009040 comp rx=0 tx=0).stop 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 -- 172.21.15.62:0/2236154050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b4159490 msgr2=0x7f19b4141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b4159490 0x7f19b4141aa0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f19ac02c450 tx=0x7f19ac004a40 comp rx=0 tx=0).stop 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 -- 172.21.15.62:0/2236154050 shutdown_connections 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f199406a7a0 0x7f199406cc60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b4159490 0x7f19b4141aa0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f19b40ce420 0x7f19b40b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 --2- 172.21.15.62:0/2236154050 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f19b406cc60 0x7f19b41edbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 -- 172.21.15.62:0/2236154050 >> 172.21.15.62:0/2236154050 conn(0x7f19b4066050 msgr2=0x7f19b40aec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 -- 172.21.15.62:0/2236154050 shutdown_connections 2024-03-23T11:26:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:04.834+0000 7f19b3e00640 1 -- 172.21.15.62:0/2236154050 wait complete. 2024-03-23T11:26:04.847 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:24:08.688706+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:24:10.072774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:26:04.848 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:26:05.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.079+0000 7fc0ef200640 1 Processor -- start 2024-03-23T11:26:05.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.079+0000 7fc0ef200640 1 -- start start 2024-03-23T11:26:05.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.081+0000 7fc0ef200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0e80cafa0 0x7fc0e80cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.081+0000 7fc0ef200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc0e806c6e0 0x7fc0e806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.081+0000 7fc0ef200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e80cbc60 con 0x7fc0e806c300 2024-03-23T11:26:05.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.081+0000 7fc0ef200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e80cd410 con 0x7fc0e806c6e0 2024-03-23T11:26:05.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.081+0000 7fc0ef200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80cbf40 con 0x7fc0e80cafa0 2024-03-23T11:26:05.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.082+0000 7fc0e7e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc0e806c300 0x7fc0e80ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53242/0 (socket says 172.21.15.62:53242) 2024-03-23T11:26:05.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.082+0000 7fc0e7e00640 1 -- 172.21.15.62:0/901258611 learned_addr learned my addr 172.21.15.62:0/901258611 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:05.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.082+0000 7fc0e7400640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0e80cafa0 0x7fc0e80cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.082+0000 7fc0ed200640 1 --2- 172.21.15.62:0/901258611 >> v2:172.21.15.120:3300/0 conn(0x7fc0e806c6e0 0x7fc0e806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.082+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826154521 0 0) 0x7fc0e80cbc60 con 0x7fc0e806c300 2024-03-23T11:26:05.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.082+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0cc003520 con 0x7fc0e806c300 2024-03-23T11:26:05.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1232012153 0 0) 0x7fc0cc003520 con 0x7fc0e806c300 2024-03-23T11:26:05.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0e80cafa0 msgr2=0x7fc0e80cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:05.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0e80cafa0 0x7fc0e80cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 >> v2:172.21.15.120:3300/0 conn(0x7fc0e806c6e0 msgr2=0x7fc0e806cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:05.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 --2- 172.21.15.62:0/901258611 >> v2:172.21.15.120:3300/0 conn(0x7fc0e806c6e0 0x7fc0e806cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e80cc220 con 0x7fc0e806c300 2024-03-23T11:26:05.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fc0d8001670 con 0x7fc0e806c300 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 >> v1:172.21.15.62:6789/0 conn(0x7fc0e806c300 legacy=0x7fc0e80ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0cc003970 0x7fc0cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0e80cafa0 0x7fc0cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0cc004d00 0x7fc0cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80cd410 con 0x7fc0cc003970 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80cbf40 con 0x7fc0cc004d00 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc00b980 con 0x7fc0e80cafa0 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e7e00640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0cc003970 0x7fc0cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e7e00640 1 -- 172.21.15.62:0/901258611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0e80cafa0 msgr2=0x7fc0cc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e7e00640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0e80cafa0 0x7fc0cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e7e00640 1 -- 172.21.15.62:0/901258611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0cc004d00 msgr2=0x7fc0cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0ed200640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0cc004d00 0x7fc0cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.083+0000 7fc0e7e00640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0cc004d00 0x7fc0cc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.084+0000 7fc0e7e00640 1 -- 172.21.15.62:0/901258611 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc003520 con 0x7fc0cc003970 2024-03-23T11:26:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.084+0000 7fc0ed200640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0cc004d00 0x7fc0cc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:05.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.085+0000 7fc0e7e00640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0cc003970 0x7fc0cc003d90 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800c240 tx=0x7fc0cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7924daeae0af4d70 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:05.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.085+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d802cb00 con 0x7fc0cc003970 2024-03-23T11:26:05.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.085+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d802cc60 con 0x7fc0cc003970 2024-03-23T11:26:05.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.085+0000 7fc0e6a00640 1 -- 172.21.15.62:0/901258611 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8005af0 con 0x7fc0cc003970 2024-03-23T11:26:05.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.086+0000 7fc0ef200640 1 -- 172.21.15.62:0/901258611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0cc003970 msgr2=0x7fc0cc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:05.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.086+0000 7fc0ef200640 1 --2- 172.21.15.62:0/901258611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0cc003970 0x7fc0cc003d90 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800c240 tx=0x7fc0cc0057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.086+0000 7fc0ed200640 1 -- 172.21.15.62:0/901258611 reap_dead start 2024-03-23T11:26:05.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.086+0000 7fc0ef200640 1 -- 172.21.15.62:0/901258611 shutdown_connections 2024-03-23T11:26:05.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.086+0000 7fc0ef200640 1 -- 172.21.15.62:0/901258611 >> 172.21.15.62:0/901258611 conn(0x7fc0e80b02e0 msgr2=0x7fc0e80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:05.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.086+0000 7fc0ef200640 1 -- 172.21.15.62:0/901258611 shutdown_connections 2024-03-23T11:26:05.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.086+0000 7fc0ef200640 1 -- 172.21.15.62:0/901258611 wait complete. 2024-03-23T11:26:05.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.088+0000 7fc0ef200640 1 Processor -- start 2024-03-23T11:26:05.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.088+0000 7fc0ef200640 1 -- start start 2024-03-23T11:26:05.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.089+0000 7fc0ef200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc003970 0x7fc0e814ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.089+0000 7fc0ef200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0e81f1810 0x7fc0e80b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.089+0000 7fc0ef200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0e81f1c70 0x7fc0e813e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.089+0000 7fc0ef200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80ac2d0 con 0x7fc0e81f1810 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.089+0000 7fc0ef200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80b3700 con 0x7fc0e81f1c70 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.089+0000 7fc0ef200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80abcc0 con 0x7fc0cc003970 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.089+0000 7fc0e7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0e81f1810 0x7fc0e80b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.089+0000 7fc0e7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0e81f1810 0x7fc0e80b2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52606/0 (socket says 172.21.15.62:52606) 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.089+0000 7fc0e7400640 1 -- 172.21.15.62:0/1218114167 learned_addr learned my addr 172.21.15.62:0/1218114167 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.090+0000 7fc0e7e00640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc003970 0x7fc0e814ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.090+0000 7fc0ed200640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0e81f1c70 0x7fc0e813e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.090+0000 7fc0e7400640 1 -- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc003970 msgr2=0x7fc0e814ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.090+0000 7fc0e7400640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc003970 0x7fc0e814ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.090+0000 7fc0e7400640 1 -- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0e81f1c70 msgr2=0x7fc0e813e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.090+0000 7fc0e7400640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0e81f1c70 0x7fc0e813e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.090+0000 7fc0e7400640 1 -- 172.21.15.62:0/1218114167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e80b3420 con 0x7fc0e81f1810 2024-03-23T11:26:05.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.090+0000 7fc0ed200640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0e81f1c70 0x7fc0e813e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:05.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.090+0000 7fc0e7400640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0e81f1810 0x7fc0e80b2960 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400e7c0 tx=0x7fc0d400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:05.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.090+0000 7fc0dfe00640 1 -- 172.21.15.62:0/1218114167 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d400eed0 con 0x7fc0e81f1810 2024-03-23T11:26:05.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.091+0000 7fc0dfe00640 1 -- 172.21.15.62:0/1218114167 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d400c940 con 0x7fc0e81f1810 2024-03-23T11:26:05.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.091+0000 7fc0dfe00640 1 -- 172.21.15.62:0/1218114167 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d400ccd0 con 0x7fc0e81f1810 2024-03-23T11:26:05.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.091+0000 7fc0ef200640 1 -- 172.21.15.62:0/1218114167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e80abf60 con 0x7fc0e81f1810 2024-03-23T11:26:05.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.091+0000 7fc0ef200640 1 -- 172.21.15.62:0/1218114167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e80b2ea0 con 0x7fc0e81f1810 2024-03-23T11:26:05.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.092+0000 7fc0ef200640 1 -- 172.21.15.62:0/1218114167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e8066c90 con 0x7fc0e81f1810 2024-03-23T11:26:05.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.093+0000 7fc0dfe00640 1 -- 172.21.15.62:0/1218114167 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc0d4013470 con 0x7fc0e81f1810 2024-03-23T11:26:05.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.094+0000 7fc0dfe00640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0c806a7a0 0x7fc0c806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.094+0000 7fc0dfe00640 1 -- 172.21.15.62:0/1218114167 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc0d4088f50 con 0x7fc0e81f1810 2024-03-23T11:26:05.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.095+0000 7fc0e7e00640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0c806a7a0 0x7fc0c806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.096+0000 7fc0e7e00640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0c806a7a0 0x7fc0c806cc60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8034000 tx=0x7fc0d8007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:05.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.098+0000 7fc0dfe00640 1 -- 172.21.15.62:0/1218114167 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0d40542d0 con 0x7fc0e81f1810 2024-03-23T11:26:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.299+0000 7fc0ef200640 1 -- 172.21.15.62:0/1218114167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0e80ade50 con 0x7fc0e81f1810 2024-03-23T11:26:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.300+0000 7fc0dfe00640 1 -- 172.21.15.62:0/1218114167 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0d40578f0 con 0x7fc0e81f1810 2024-03-23T11:26:05.294 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:26:05.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 -- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0c806a7a0 msgr2=0x7fc0c806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:05.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0c806a7a0 0x7fc0c806cc60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8034000 tx=0x7fc0d8007000 comp rx=0 tx=0).stop 2024-03-23T11:26:05.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 -- 172.21.15.62:0/1218114167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0e81f1810 msgr2=0x7fc0e80b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:05.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0e81f1810 0x7fc0e80b2960 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400e7c0 tx=0x7fc0d400ec80 comp rx=0 tx=0).stop 2024-03-23T11:26:05.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 -- 172.21.15.62:0/1218114167 shutdown_connections 2024-03-23T11:26:05.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0c806a7a0 0x7fc0c806cc60 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0e81f1810 0x7fc0e80b2960 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0e81f1c70 0x7fc0e813e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 --2- 172.21.15.62:0/1218114167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc003970 0x7fc0e814ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 -- 172.21.15.62:0/1218114167 >> 172.21.15.62:0/1218114167 conn(0x7fc0e80b02e0 msgr2=0x7fc0e80cd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:05.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 -- 172.21.15.62:0/1218114167 shutdown_connections 2024-03-23T11:26:05.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.304+0000 7fc0ef200640 1 -- 172.21.15.62:0/1218114167 wait complete. 2024-03-23T11:26:05.318 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:26:05.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.551+0000 7ffaf7000640 1 Processor -- start 2024-03-23T11:26:05.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.552+0000 7ffaf7000640 1 -- start start 2024-03-23T11:26:05.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.553+0000 7ffaf7000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf015c9e0 0x7ffaf01607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.553+0000 7ffaf7000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffaf015cdc0 0x7ffaf00d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.553+0000 7ffaf7000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffaf00cf650 0x7ffaf00cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.553+0000 7ffaf7000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf00d0020 con 0x7ffaf00cf650 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.553+0000 7ffaf7000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf006bfc0 con 0x7ffaf015cdc0 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.553+0000 7ffaf7000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf00b5260 con 0x7ffaf015c9e0 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.554+0000 7ffaf5000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffaf00cf650 0x7ffaf00cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.554+0000 7ffaf5000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffaf00cf650 0x7ffaf00cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52620/0 (socket says 172.21.15.62:52620) 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.554+0000 7ffaf5000640 1 -- 172.21.15.62:0/1216326355 learned_addr learned my addr 172.21.15.62:0/1216326355 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.554+0000 7ffaefe00640 1 --2- 172.21.15.62:0/1216326355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf015c9e0 0x7ffaf01607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.554+0000 7ffaef400640 1 --2- 172.21.15.62:0/1216326355 >> v2:172.21.15.120:3300/0 conn(0x7ffaf015cdc0 0x7ffaf00d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.554+0000 7ffaf5000640 1 -- 172.21.15.62:0/1216326355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf015c9e0 msgr2=0x7ffaf01607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:05.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.554+0000 7ffaf5000640 1 --2- 172.21.15.62:0/1216326355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf015c9e0 0x7ffaf01607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.554+0000 7ffaf5000640 1 -- 172.21.15.62:0/1216326355 >> v2:172.21.15.120:3300/0 conn(0x7ffaf015cdc0 msgr2=0x7ffaf00d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.554+0000 7ffaf5000640 1 --2- 172.21.15.62:0/1216326355 >> v2:172.21.15.120:3300/0 conn(0x7ffaf015cdc0 0x7ffaf00d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.554+0000 7ffaf5000640 1 -- 172.21.15.62:0/1216326355 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf00ce140 con 0x7ffaf00cf650 2024-03-23T11:26:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.555+0000 7ffaf5000640 1 --2- 172.21.15.62:0/1216326355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf00cf650 0x7ffaf00cfae0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00b53f0 tx=0x7ffae802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6b214f4705ae23 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.556+0000 7ffaeea00640 1 -- 172.21.15.62:0/1216326355 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae802ca70 con 0x7ffaf00cf650 2024-03-23T11:26:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.556+0000 7ffaeea00640 1 -- 172.21.15.62:0/1216326355 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae802cbd0 con 0x7ffaf00cf650 2024-03-23T11:26:05.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.556+0000 7ffaeea00640 1 -- 172.21.15.62:0/1216326355 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae80343f0 con 0x7ffaf00cf650 2024-03-23T11:26:05.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.557+0000 7ffaf7000640 1 -- 172.21.15.62:0/1216326355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf00cf650 msgr2=0x7ffaf00cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:05.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.557+0000 7ffaf7000640 1 --2- 172.21.15.62:0/1216326355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf00cf650 0x7ffaf00cfae0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00b53f0 tx=0x7ffae802bed0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.557+0000 7ffaf7000640 1 -- 172.21.15.62:0/1216326355 shutdown_connections 2024-03-23T11:26:05.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.557+0000 7ffaf7000640 1 --2- 172.21.15.62:0/1216326355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf00cf650 0x7ffaf00cfae0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.557+0000 7ffaf7000640 1 --2- 172.21.15.62:0/1216326355 >> v2:172.21.15.120:3300/0 conn(0x7ffaf015cdc0 0x7ffaf00d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.557+0000 7ffaf7000640 1 --2- 172.21.15.62:0/1216326355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf015c9e0 0x7ffaf01607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.557+0000 7ffaf7000640 1 -- 172.21.15.62:0/1216326355 >> 172.21.15.62:0/1216326355 conn(0x7ffaf0071570 msgr2=0x7ffaf0071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:05.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.557+0000 7ffaf7000640 1 -- 172.21.15.62:0/1216326355 shutdown_connections 2024-03-23T11:26:05.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.557+0000 7ffaf7000640 1 -- 172.21.15.62:0/1216326355 wait complete. 2024-03-23T11:26:05.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.559+0000 7ffaf7000640 1 Processor -- start 2024-03-23T11:26:05.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.559+0000 7ffaf7000640 1 -- start start 2024-03-23T11:26:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaf7000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf00cf650 0x7ffaf0147e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaf7000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf015c9e0 0x7ffaf00caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaf7000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf015cdc0 0x7ffaf015e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaf7000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf00cb8d0 con 0x7ffaf015c9e0 2024-03-23T11:26:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaf7000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf00cbbb0 con 0x7ffaf00cf650 2024-03-23T11:26:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaf7000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf00cbe90 con 0x7ffaf015cdc0 2024-03-23T11:26:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaef400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf015c9e0 0x7ffaf00caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaef400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf015c9e0 0x7ffaf00caea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52632/0 (socket says 172.21.15.62:52632) 2024-03-23T11:26:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaef400640 1 -- 172.21.15.62:0/2260069400 learned_addr learned my addr 172.21.15.62:0/2260069400 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaef400640 1 -- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf015cdc0 msgr2=0x7ffaf015e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaef400640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf015cdc0 0x7ffaf015e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaef400640 1 -- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf00cf650 msgr2=0x7ffaf0147e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaefe00640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf00cf650 0x7ffaf0147e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaef400640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf00cf650 0x7ffaf0147e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.561+0000 7ffaef400640 1 -- 172.21.15.62:0/2260069400 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae8009b30 con 0x7ffaf015c9e0 2024-03-23T11:26:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.562+0000 7ffaef400640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf015c9e0 0x7ffaf00caea0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7ffad800d5f0 tx=0x7ffad800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.562+0000 7ffae7e00640 1 -- 172.21.15.62:0/2260069400 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad80125f0 con 0x7ffaf015c9e0 2024-03-23T11:26:05.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.562+0000 7ffae7e00640 1 -- 172.21.15.62:0/2260069400 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad8012750 con 0x7ffaf015c9e0 2024-03-23T11:26:05.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.562+0000 7ffaf7000640 1 -- 172.21.15.62:0/2260069400 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf00cc170 con 0x7ffaf015c9e0 2024-03-23T11:26:05.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.562+0000 7ffae7e00640 1 -- 172.21.15.62:0/2260069400 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8012b00 con 0x7ffaf015c9e0 2024-03-23T11:26:05.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.562+0000 7ffaf7000640 1 -- 172.21.15.62:0/2260069400 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf00c6380 con 0x7ffaf015c9e0 2024-03-23T11:26:05.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.564+0000 7ffaf7000640 1 -- 172.21.15.62:0/2260069400 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf00c6510 con 0x7ffaf015c9e0 2024-03-23T11:26:05.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.565+0000 7ffae7e00640 1 -- 172.21.15.62:0/2260069400 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ffad8012c60 con 0x7ffaf015c9e0 2024-03-23T11:26:05.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.566+0000 7ffae7e00640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffacc06a7a0 0x7ffacc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:05.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.566+0000 7ffae7e00640 1 -- 172.21.15.62:0/2260069400 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ffad80893b0 con 0x7ffaf015c9e0 2024-03-23T11:26:05.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.567+0000 7ffaefe00640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffacc06a7a0 0x7ffacc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:05.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.568+0000 7ffaefe00640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffacc06a7a0 0x7ffacc06cc60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffae0006fd0 tx=0x7ffae0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:05.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.571+0000 7ffae7e00640 1 -- 172.21.15.62:0/2260069400 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffad80546a0 con 0x7ffaf015c9e0 2024-03-23T11:26:05.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.779+0000 7ffaf7000640 1 -- 172.21.15.62:0/2260069400 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffaf0066c90 con 0x7ffaf015c9e0 2024-03-23T11:26:05.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.780+0000 7ffae7e00640 1 -- 172.21.15.62:0/2260069400 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffad8057cc0 con 0x7ffaf015c9e0 2024-03-23T11:26:05.773 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:26:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.782+0000 7ffaf7000640 1 -- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffacc06a7a0 msgr2=0x7ffacc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.782+0000 7ffaf7000640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffacc06a7a0 0x7ffacc06cc60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffae0006fd0 tx=0x7ffae0008040 comp rx=0 tx=0).stop 2024-03-23T11:26:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.782+0000 7ffaf7000640 1 -- 172.21.15.62:0/2260069400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf015c9e0 msgr2=0x7ffaf00caea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.782+0000 7ffaf7000640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf015c9e0 0x7ffaf00caea0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7ffad800d5f0 tx=0x7ffad800dab0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.782+0000 7ffaf7000640 1 -- 172.21.15.62:0/2260069400 shutdown_connections 2024-03-23T11:26:05.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.782+0000 7ffaf7000640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffacc06a7a0 0x7ffacc06cc60 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.782+0000 7ffaf7000640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaf015c9e0 0x7ffaf00caea0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.782+0000 7ffaf7000640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf015cdc0 0x7ffaf015e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.782+0000 7ffaf7000640 1 --2- 172.21.15.62:0/2260069400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf00cf650 0x7ffaf0147e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:05.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.782+0000 7ffaf7000640 1 -- 172.21.15.62:0/2260069400 >> 172.21.15.62:0/2260069400 conn(0x7ffaf0071570 msgr2=0x7ffaf015be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:05.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.783+0000 7ffaf7000640 1 -- 172.21.15.62:0/2260069400 shutdown_connections 2024-03-23T11:26:05.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:05.783+0000 7ffaf7000640 1 -- 172.21.15.62:0/2260069400 wait complete. 2024-03-23T11:26:05.795 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:26:06.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.021+0000 7f9b96600640 1 Processor -- start 2024-03-23T11:26:06.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.022+0000 7f9b96600640 1 -- start start 2024-03-23T11:26:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.023+0000 7f9b96600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b901608b0 0x7f9b900b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.023+0000 7f9b96600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b900ce510 con 0x7f9b9015fcf0 2024-03-23T11:26:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.023+0000 7f9b96600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b90160d20 con 0x7f9b90160420 2024-03-23T11:26:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.023+0000 7f9b96600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b900cdc70 con 0x7f9b901608b0 2024-03-23T11:26:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.024+0000 7f9b95200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9b9015fcf0 0x7f9b900d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53256/0 (socket says 172.21.15.62:53256) 2024-03-23T11:26:06.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.024+0000 7f9b95200640 1 -- 172.21.15.62:0/2353913076 learned_addr learned my addr 172.21.15.62:0/2353913076 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:06.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.024+0000 7f9b95c00640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b901608b0 0x7f9b900b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.024+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381472923 0 0) 0x7f9b900ce510 con 0x7f9b9015fcf0 2024-03-23T11:26:06.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.024+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b74003520 con 0x7f9b9015fcf0 2024-03-23T11:26:06.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2042286296 0 0) 0x7f9b74003520 con 0x7f9b9015fcf0 2024-03-23T11:26:06.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b901608b0 msgr2=0x7f9b900b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:06.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b901608b0 0x7f9b900b64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 >> v1:172.21.15.120:6789/0 conn(0x7f9b90160420 legacy=0x7f9b900b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b900cdf50 con 0x7f9b9015fcf0 2024-03-23T11:26:06.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9b80001670 con 0x7f9b9015fcf0 2024-03-23T11:26:06.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 >> v1:172.21.15.62:6789/0 conn(0x7f9b9015fcf0 legacy=0x7f9b900d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b740039a0 0x7f9b74003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b901608b0 0x7f9b740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b74004d50 0x7f9b740051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b900cdc70 con 0x7f9b740039a0 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7400bbe0 con 0x7f9b74004d50 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7400bda0 con 0x7f9b901608b0 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b95200640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b740039a0 0x7f9b74003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b95c00640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b74004d50 0x7f9b740051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b8fe00640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b901608b0 0x7f9b740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b95200640 1 -- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b901608b0 msgr2=0x7f9b740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b95200640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b901608b0 0x7f9b740047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.025+0000 7f9b95200640 1 -- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b74004d50 msgr2=0x7f9b740051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:06.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.026+0000 7f9b95200640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b74004d50 0x7f9b740051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.026+0000 7f9b95200640 1 -- 172.21.15.62:0/2353913076 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b74003520 con 0x7f9b740039a0 2024-03-23T11:26:06.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.026+0000 7f9b95c00640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b74004d50 0x7f9b740051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.027+0000 7f9b95200640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b740039a0 0x7f9b74003de0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000c0e0 tx=0x7f9b740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b5f2fdb9f2986e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.027+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80005ad0 con 0x7f9b740039a0 2024-03-23T11:26:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.027+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b80005c30 con 0x7f9b740039a0 2024-03-23T11:26:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.027+0000 7f9b8f400640 1 -- 172.21.15.62:0/2353913076 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8002c810 con 0x7f9b740039a0 2024-03-23T11:26:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.028+0000 7f9b96600640 1 -- 172.21.15.62:0/2353913076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b740039a0 msgr2=0x7f9b74003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.028+0000 7f9b96600640 1 --2- 172.21.15.62:0/2353913076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b740039a0 0x7f9b74003de0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000c0e0 tx=0x7f9b740057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.028+0000 7f9b95c00640 1 -- 172.21.15.62:0/2353913076 reap_dead start 2024-03-23T11:26:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.028+0000 7f9b96600640 1 -- 172.21.15.62:0/2353913076 shutdown_connections 2024-03-23T11:26:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.028+0000 7f9b96600640 1 -- 172.21.15.62:0/2353913076 >> 172.21.15.62:0/2353913076 conn(0x7f9b90089f60 msgr2=0x7f9b900d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.028+0000 7f9b96600640 1 -- 172.21.15.62:0/2353913076 shutdown_connections 2024-03-23T11:26:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.028+0000 7f9b96600640 1 -- 172.21.15.62:0/2353913076 wait complete. 2024-03-23T11:26:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.030+0000 7f9b96600640 1 Processor -- start 2024-03-23T11:26:06.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.030+0000 7f9b96600640 1 -- start start 2024-03-23T11:26:06.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b96600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b740039a0 0x7f9b90151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b96600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9009cc70 0x7f9b9009d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b96600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b900b65f0 0x7f9b900b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b96600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b900b6fb0 con 0x7f9b9009cc70 2024-03-23T11:26:06.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b96600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9015c540 con 0x7f9b740039a0 2024-03-23T11:26:06.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b96600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9009f3c0 con 0x7f9b900b65f0 2024-03-23T11:26:06.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b8fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9009cc70 0x7f9b9009d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b8fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9009cc70 0x7f9b9009d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52658/0 (socket says 172.21.15.62:52658) 2024-03-23T11:26:06.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b8fe00640 1 -- 172.21.15.62:0/258549702 learned_addr learned my addr 172.21.15.62:0/258549702 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b95200640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b740039a0 0x7f9b90151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b95c00640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b900b65f0 0x7f9b900b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b8fe00640 1 -- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b900b65f0 msgr2=0x7f9b900b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b8fe00640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b900b65f0 0x7f9b900b6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b8fe00640 1 -- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b740039a0 msgr2=0x7f9b90151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b8fe00640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b740039a0 0x7f9b90151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.031+0000 7f9b8fe00640 1 -- 172.21.15.62:0/258549702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9009f660 con 0x7f9b9009cc70 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.032+0000 7f9b95200640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b740039a0 0x7f9b90151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.032+0000 7f9b8fe00640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9009cc70 0x7f9b9009d0f0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800d570 tx=0x7f9b7800da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.032+0000 7f9b8cc00640 1 -- 172.21.15.62:0/258549702 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b78005480 con 0x7f9b9009cc70 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.032+0000 7f9b8cc00640 1 -- 172.21.15.62:0/258549702 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b78003610 con 0x7f9b9009cc70 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.032+0000 7f9b8cc00640 1 -- 172.21.15.62:0/258549702 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b78003980 con 0x7f9b9009cc70 2024-03-23T11:26:06.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.032+0000 7f9b96600640 1 -- 172.21.15.62:0/258549702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9009f940 con 0x7f9b9009cc70 2024-03-23T11:26:06.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.032+0000 7f9b96600640 1 -- 172.21.15.62:0/258549702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9009ff00 con 0x7f9b9009cc70 2024-03-23T11:26:06.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.033+0000 7f9b96600640 1 -- 172.21.15.62:0/258549702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b90066bb0 con 0x7f9b9009cc70 2024-03-23T11:26:06.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.036+0000 7f9b8cc00640 1 -- 172.21.15.62:0/258549702 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9b78003ae0 con 0x7f9b9009cc70 2024-03-23T11:26:06.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.037+0000 7f9b8cc00640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b6c06a7d0 0x7f9b6c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.037+0000 7f9b8cc00640 1 -- 172.21.15.62:0/258549702 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9b780897d0 con 0x7f9b9009cc70 2024-03-23T11:26:06.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.037+0000 7f9b95200640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b6c06a7d0 0x7f9b6c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.039+0000 7f9b95200640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b6c06a7d0 0x7f9b6c06cc90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9b80034000 tx=0x7f9b80009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:06.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.040+0000 7f9b8cc00640 1 -- 172.21.15.62:0/258549702 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9b78054bd0 con 0x7f9b9009cc70 2024-03-23T11:26:06.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.237+0000 7f9b96600640 1 -- 172.21.15.62:0/258549702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9b900b4d90 con 0x7f9b9009cc70 2024-03-23T11:26:06.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.238+0000 7f9b8cc00640 1 -- 172.21.15.62:0/258549702 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9b78001230 con 0x7f9b9009cc70 2024-03-23T11:26:06.232 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:26:06.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.241+0000 7f9b96600640 1 -- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b6c06a7d0 msgr2=0x7f9b6c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.241+0000 7f9b96600640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b6c06a7d0 0x7f9b6c06cc90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9b80034000 tx=0x7f9b80009030 comp rx=0 tx=0).stop 2024-03-23T11:26:06.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.241+0000 7f9b96600640 1 -- 172.21.15.62:0/258549702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9009cc70 msgr2=0x7f9b9009d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.241+0000 7f9b96600640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9009cc70 0x7f9b9009d0f0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800d570 tx=0x7f9b7800da30 comp rx=0 tx=0).stop 2024-03-23T11:26:06.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.241+0000 7f9b96600640 1 -- 172.21.15.62:0/258549702 shutdown_connections 2024-03-23T11:26:06.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.241+0000 7f9b96600640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9b6c06a7d0 0x7f9b6c06cc90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.241+0000 7f9b96600640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b9009cc70 0x7f9b9009d0f0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.241+0000 7f9b96600640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9b900b65f0 0x7f9b900b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.241+0000 7f9b96600640 1 --2- 172.21.15.62:0/258549702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b740039a0 0x7f9b90151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.241+0000 7f9b96600640 1 -- 172.21.15.62:0/258549702 >> 172.21.15.62:0/258549702 conn(0x7f9b90089f60 msgr2=0x7f9b900cd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:06.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.241+0000 7f9b96600640 1 -- 172.21.15.62:0/258549702 shutdown_connections 2024-03-23T11:26:06.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.242+0000 7f9b96600640 1 -- 172.21.15.62:0/258549702 wait complete. 2024-03-23T11:26:06.253 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:26:06.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.482+0000 7f9cebe00640 1 Processor -- start 2024-03-23T11:26:06.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.482+0000 7f9cebe00640 1 -- start start 2024-03-23T11:26:06.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.484+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec0d13b0 0x7f9cec0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.484+0000 7f9cebe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9cec0cd260 0x7f9cec0b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.484+0000 7f9cebe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec0d0810 con 0x7f9cec15c960 2024-03-23T11:26:06.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.484+0000 7f9cebe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cec0cd9d0 con 0x7f9cec0cd260 2024-03-23T11:26:06.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.484+0000 7f9cebe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0cdcb0 con 0x7f9cec0d13b0 2024-03-23T11:26:06.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.484+0000 7f9ceaa00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9cec15c960 0x7f9cec0d0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53258/0 (socket says 172.21.15.62:53258) 2024-03-23T11:26:06.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.484+0000 7f9ceaa00640 1 -- 172.21.15.62:0/2270910590 learned_addr learned my addr 172.21.15.62:0/2270910590 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:06.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9cea000640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec0d13b0 0x7f9cec0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762364503 0 0) 0x7f9cdc001110 con 0x7f9cec15c960 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd0003520 con 0x7f9cec15c960 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ceb400640 1 --2- 172.21.15.62:0/2270910590 >> v2:172.21.15.120:3300/0 conn(0x7f9cec0cd260 0x7f9cec0b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1369138183 0 0) 0x7f9cd0003520 con 0x7f9cec15c960 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec0d13b0 msgr2=0x7f9cec0cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec0d13b0 0x7f9cec0cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 >> v2:172.21.15.120:3300/0 conn(0x7f9cec0cd260 msgr2=0x7f9cec0b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 --2- 172.21.15.62:0/2270910590 >> v2:172.21.15.120:3300/0 conn(0x7f9cec0cd260 0x7f9cec0b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec0cdf90 con 0x7f9cec15c960 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9cdc001820 con 0x7f9cec15c960 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 >> v1:172.21.15.62:6789/0 conn(0x7f9cec15c960 legacy=0x7f9cec0d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cd0003970 0x7f9cd0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.485+0000 7f9ce9600640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd00047c0 0x7f9cd0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ce9600640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd0005180 0x7f9cd000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0cd9d0 con 0x7f9cd0003970 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0cdcb0 con 0x7f9cd00047c0 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd000c000 con 0x7f9cd0005180 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9cea000640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec0d13b0 0x7f9cec0cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ceaa00640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cd0003970 0x7f9cd0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ceb400640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd0005180 0x7f9cd000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ceaa00640 1 -- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd0005180 msgr2=0x7f9cd000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ceaa00640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd0005180 0x7f9cd000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ceaa00640 1 -- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd00047c0 msgr2=0x7f9cd0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9cea000640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd00047c0 0x7f9cd0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ceaa00640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cd00047c0 0x7f9cd0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ceaa00640 1 -- 172.21.15.62:0/2270910590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd0003520 con 0x7f9cd0003970 2024-03-23T11:26:06.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.486+0000 7f9ceb400640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cd0005180 0x7f9cd000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:06.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.488+0000 7f9ceaa00640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cd0003970 0x7f9cd0003d90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c160 tx=0x7f9cd00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9dd5306db7a570c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:06.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.488+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc005ad0 con 0x7f9cd0003970 2024-03-23T11:26:06.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.488+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc005c30 con 0x7f9cd0003970 2024-03-23T11:26:06.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.488+0000 7f9ce9600640 1 -- 172.21.15.62:0/2270910590 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc02c810 con 0x7f9cd0003970 2024-03-23T11:26:06.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.489+0000 7f9cebe00640 1 -- 172.21.15.62:0/2270910590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cd0003970 msgr2=0x7f9cd0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.489+0000 7f9cebe00640 1 --2- 172.21.15.62:0/2270910590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cd0003970 0x7f9cd0003d90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c160 tx=0x7f9cd00057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.489+0000 7f9ceb400640 1 -- 172.21.15.62:0/2270910590 reap_dead start 2024-03-23T11:26:06.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.489+0000 7f9cebe00640 1 -- 172.21.15.62:0/2270910590 shutdown_connections 2024-03-23T11:26:06.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.489+0000 7f9cebe00640 1 -- 172.21.15.62:0/2270910590 >> 172.21.15.62:0/2270910590 conn(0x7f9cec089f60 msgr2=0x7f9cec0d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:06.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.489+0000 7f9cebe00640 1 -- 172.21.15.62:0/2270910590 shutdown_connections 2024-03-23T11:26:06.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.489+0000 7f9cebe00640 1 -- 172.21.15.62:0/2270910590 wait complete. 2024-03-23T11:26:06.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.491+0000 7f9cebe00640 1 Processor -- start 2024-03-23T11:26:06.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.491+0000 7f9cebe00640 1 -- start start 2024-03-23T11:26:06.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.492+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec155510 0x7f9cec0b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.492+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec1f2bd0 0x7f9cec0b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.492+0000 7f9cebe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cec1f3030 0x7f9cec0b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.492+0000 7f9cebe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0b4bf0 con 0x7f9cec155510 2024-03-23T11:26:06.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.492+0000 7f9cebe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0ca580 con 0x7f9cec1f3030 2024-03-23T11:26:06.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.492+0000 7f9cebe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0ca860 con 0x7f9cec1f2bd0 2024-03-23T11:26:06.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.492+0000 7f9ceaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec155510 0x7f9cec0b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.492+0000 7f9ceaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec155510 0x7f9cec0b25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52674/0 (socket says 172.21.15.62:52674) 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.492+0000 7f9ceaa00640 1 -- 172.21.15.62:0/1072520065 learned_addr learned my addr 172.21.15.62:0/1072520065 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.493+0000 7f9cea000640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec1f2bd0 0x7f9cec0b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.493+0000 7f9ceb400640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cec1f3030 0x7f9cec0b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.493+0000 7f9ceaa00640 1 -- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec1f2bd0 msgr2=0x7f9cec0b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.493+0000 7f9ceaa00640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec1f2bd0 0x7f9cec0b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.493+0000 7f9ceaa00640 1 -- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cec1f3030 msgr2=0x7f9cec0b46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.493+0000 7f9ceaa00640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cec1f3030 0x7f9cec0b46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.493+0000 7f9ceaa00640 1 -- 172.21.15.62:0/1072520065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec0cab40 con 0x7f9cec155510 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.493+0000 7f9ceaa00640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec155510 0x7f9cec0b25c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c280 tx=0x7f9cdc001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.493+0000 7f9ce3400640 1 -- 172.21.15.62:0/1072520065 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc03c070 con 0x7f9cec155510 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.494+0000 7f9ce3400640 1 -- 172.21.15.62:0/1072520065 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc0017f0 con 0x7f9cec155510 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.494+0000 7f9ce3400640 1 -- 172.21.15.62:0/1072520065 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc004aa0 con 0x7f9cec155510 2024-03-23T11:26:06.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.494+0000 7f9cebe00640 1 -- 172.21.15.62:0/1072520065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cec0cae20 con 0x7f9cec155510 2024-03-23T11:26:06.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.494+0000 7f9cebe00640 1 -- 172.21.15.62:0/1072520065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cec0b2b00 con 0x7f9cec155510 2024-03-23T11:26:06.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.495+0000 7f9cebe00640 1 -- 172.21.15.62:0/1072520065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cec066bb0 con 0x7f9cec155510 2024-03-23T11:26:06.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.496+0000 7f9ce3400640 1 -- 172.21.15.62:0/1072520065 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9cdc032070 con 0x7f9cec155510 2024-03-23T11:26:06.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.497+0000 7f9ce3400640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ccc06a7a0 0x7f9ccc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:06.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.497+0000 7f9ce3400640 1 -- 172.21.15.62:0/1072520065 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9cdc0a9970 con 0x7f9cec155510 2024-03-23T11:26:06.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.497+0000 7f9cea000640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ccc06a7a0 0x7f9ccc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:06.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.499+0000 7f9cea000640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ccc06a7a0 0x7f9ccc06cc60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0b8740 tx=0x7f9cd8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:06.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.501+0000 7f9ce3400640 1 -- 172.21.15.62:0/1072520065 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9cdc074c60 con 0x7f9cec155510 2024-03-23T11:26:06.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.712+0000 7f9cebe00640 1 -- 172.21.15.62:0/1072520065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9cec0d0810 con 0x7f9cec155510 2024-03-23T11:26:06.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.713+0000 7f9ce3400640 1 -- 172.21.15.62:0/1072520065 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v77) v1 ==== 80+0+124 (secure 0 0 0) 0x7f9cec0d0810 con 0x7f9cec155510 2024-03-23T11:26:06.707 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6837/3833782560 2024-03-24T11:24:34.350690+0000 2024-03-23T11:26:06.707 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6836/3833782560 2024-03-24T11:24:34.350690+0000 2024-03-23T11:26:06.707 INFO:teuthology.orchestra.run.smithi062.stderr:listed 2 entries 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 -- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ccc06a7a0 msgr2=0x7f9ccc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ccc06a7a0 0x7f9ccc06cc60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0b8740 tx=0x7f9cd8008040 comp rx=0 tx=0).stop 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 -- 172.21.15.62:0/1072520065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec155510 msgr2=0x7f9cec0b25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec155510 0x7f9cec0b25c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c280 tx=0x7f9cdc001ae0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 -- 172.21.15.62:0/1072520065 shutdown_connections 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9ccc06a7a0 0x7f9ccc06cc60 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9cec1f3030 0x7f9cec0b46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9cec1f2bd0 0x7f9cec0b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 --2- 172.21.15.62:0/1072520065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec155510 0x7f9cec0b25c0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 -- 172.21.15.62:0/1072520065 >> 172.21.15.62:0/1072520065 conn(0x7f9cec089f60 msgr2=0x7f9cec0af270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 -- 172.21.15.62:0/1072520065 shutdown_connections 2024-03-23T11:26:06.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:06.716+0000 7f9cebe00640 1 -- 172.21.15.62:0/1072520065 wait complete. 2024-03-23T11:26:06.728 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:26:06.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:06.729 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:06.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[63379]: fuse finished with error 0 and tester_r 0 2024-03-23T11:26:06.823 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:26:06.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:06.824 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:26:06.933 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:26:06.933 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:26:06.992 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:26:06.992 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:26:06.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:06.992 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T11:26:07.044 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-23T11:26:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:26:07.045 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:26:07.045 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:26:07.105 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:26:07.105 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:26:07.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:07.106 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T11:26:07.159 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-23T11:26:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:26:07.160 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:26:07.160 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:26:07.216 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:26:07.217 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:26:07.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:07.217 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T11:26:07.270 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-23T11:26:07.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:26:07.272 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:26:07.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.511+0000 7fe10fa00640 1 Processor -- start 2024-03-23T11:26:07.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.511+0000 7fe10fa00640 1 -- start start 2024-03-23T11:26:07.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.513+0000 7fe10fa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe10806c300 0x7fe1080ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:07.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.513+0000 7fe10fa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe10806c6e0 0x7fe10806cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:07.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.513+0000 7fe10fa00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1080cbc60 con 0x7fe1080cafa0 2024-03-23T11:26:07.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.513+0000 7fe10fa00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1080cbf40 con 0x7fe10806c300 2024-03-23T11:26:07.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.513+0000 7fe10fa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1080cc220 con 0x7fe10806c6e0 2024-03-23T11:26:07.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.513+0000 7fe10d000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe10806c300 0x7fe1080ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:07.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.513+0000 7fe10d000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe10806c300 0x7fe1080ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42704/0 (socket says 172.21.15.62:42704) 2024-03-23T11:26:07.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.513+0000 7fe10d000640 1 -- 172.21.15.62:0/1037321858 learned_addr learned my addr 172.21.15.62:0/1037321858 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:07.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.514+0000 7fe10da00640 1 --2- 172.21.15.62:0/1037321858 >> v2:172.21.15.120:3300/0 conn(0x7fe10806c6e0 0x7fe10806cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:07.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.514+0000 7fe10d000640 1 -- 172.21.15.62:0/1037321858 >> v2:172.21.15.120:3300/0 conn(0x7fe10806c6e0 msgr2=0x7fe10806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:07.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.514+0000 7fe10d000640 1 --2- 172.21.15.62:0/1037321858 >> v2:172.21.15.120:3300/0 conn(0x7fe10806c6e0 0x7fe10806cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:07.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.514+0000 7fe10d000640 1 -- 172.21.15.62:0/1037321858 >> v1:172.21.15.120:6789/0 conn(0x7fe1080cafa0 legacy=0x7fe1080cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:07.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.514+0000 7fe10d000640 1 -- 172.21.15.62:0/1037321858 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1080cc500 con 0x7fe10806c300 2024-03-23T11:26:07.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.515+0000 7fe10d000640 1 --2- 172.21.15.62:0/1037321858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe10806c300 0x7fe1080ccd00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe108070270 tx=0x7fe0fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3d921f959413e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:07.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.515+0000 7fe107400640 1 -- 172.21.15.62:0/1037321858 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc031870 con 0x7fe10806c300 2024-03-23T11:26:07.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.516+0000 7fe107400640 1 -- 172.21.15.62:0/1037321858 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc0319d0 con 0x7fe10806c300 2024-03-23T11:26:07.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.516+0000 7fe107400640 1 -- 172.21.15.62:0/1037321858 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc031d60 con 0x7fe10806c300 2024-03-23T11:26:07.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.516+0000 7fe10fa00640 1 -- 172.21.15.62:0/1037321858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe10806c300 msgr2=0x7fe1080ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:07.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.516+0000 7fe10fa00640 1 --2- 172.21.15.62:0/1037321858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe10806c300 0x7fe1080ccd00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe108070270 tx=0x7fe0fc02b910 comp rx=0 tx=0).stop 2024-03-23T11:26:07.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.516+0000 7fe10fa00640 1 -- 172.21.15.62:0/1037321858 shutdown_connections 2024-03-23T11:26:07.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.516+0000 7fe10fa00640 1 --2- 172.21.15.62:0/1037321858 >> v2:172.21.15.120:3300/0 conn(0x7fe10806c6e0 0x7fe10806cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:07.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.516+0000 7fe10fa00640 1 --2- 172.21.15.62:0/1037321858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe10806c300 0x7fe1080ccd00 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:07.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.516+0000 7fe10fa00640 1 -- 172.21.15.62:0/1037321858 >> 172.21.15.62:0/1037321858 conn(0x7fe1080b02e0 msgr2=0x7fe1080b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:07.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.517+0000 7fe10fa00640 1 -- 172.21.15.62:0/1037321858 shutdown_connections 2024-03-23T11:26:07.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.517+0000 7fe10fa00640 1 -- 172.21.15.62:0/1037321858 wait complete. 2024-03-23T11:26:07.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.519+0000 7fe10fa00640 1 Processor -- start 2024-03-23T11:26:07.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.519+0000 7fe10fa00640 1 -- start start 2024-03-23T11:26:07.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.520+0000 7fe10fa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe10806c300 0x7fe108157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:07.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.520+0000 7fe10fa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe10806c6e0 0x7fe1080ac5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:07.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.520+0000 7fe10fa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1080cafa0 0x7fe108132680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:07.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.520+0000 7fe10fa00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10813ca60 con 0x7fe10806c300 2024-03-23T11:26:07.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.520+0000 7fe10fa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10813f7b0 con 0x7fe1080cafa0 2024-03-23T11:26:07.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.520+0000 7fe10fa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1081575a0 con 0x7fe10806c6e0 2024-03-23T11:26:07.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.521+0000 7fe10d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe10806c300 0x7fe108157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:07.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.521+0000 7fe10d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe10806c300 0x7fe108157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42712/0 (socket says 172.21.15.62:42712) 2024-03-23T11:26:07.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.521+0000 7fe10d000640 1 -- 172.21.15.62:0/478523485 learned_addr learned my addr 172.21.15.62:0/478523485 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:07.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.521+0000 7fe10da00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1080cafa0 0x7fe108132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:07.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.521+0000 7fe107e00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe10806c6e0 0x7fe1080ac5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:07.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.521+0000 7fe10d000640 1 -- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe10806c6e0 msgr2=0x7fe1080ac5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:07.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.521+0000 7fe10d000640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe10806c6e0 0x7fe1080ac5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:07.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.521+0000 7fe10d000640 1 -- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1080cafa0 msgr2=0x7fe108132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:07.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.521+0000 7fe10d000640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1080cafa0 0x7fe108132680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:07.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.521+0000 7fe10d000640 1 -- 172.21.15.62:0/478523485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc0094d0 con 0x7fe10806c300 2024-03-23T11:26:07.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.521+0000 7fe107e00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe10806c6e0 0x7fe1080ac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:07.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.522+0000 7fe10d000640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe10806c300 0x7fe108157060 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc000c00 tx=0x7fe0fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:07.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.522+0000 7fe104c00640 1 -- 172.21.15.62:0/478523485 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc03abe0 con 0x7fe10806c300 2024-03-23T11:26:07.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.522+0000 7fe104c00640 1 -- 172.21.15.62:0/478523485 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc03ad40 con 0x7fe10806c300 2024-03-23T11:26:07.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.522+0000 7fe104c00640 1 -- 172.21.15.62:0/478523485 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc044610 con 0x7fe10806c300 2024-03-23T11:26:07.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.522+0000 7fe10fa00640 1 -- 172.21.15.62:0/478523485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10813eba0 con 0x7fe10806c300 2024-03-23T11:26:07.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.522+0000 7fe10fa00640 1 -- 172.21.15.62:0/478523485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10813f160 con 0x7fe10806c300 2024-03-23T11:26:07.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.524+0000 7fe10fa00640 1 -- 172.21.15.62:0/478523485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe108066c90 con 0x7fe10806c300 2024-03-23T11:26:07.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.524+0000 7fe104c00640 1 -- 172.21.15.62:0/478523485 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe0fc044770 con 0x7fe10806c300 2024-03-23T11:26:07.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.525+0000 7fe104c00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe0e806a7d0 0x7fe0e806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:07.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.525+0000 7fe104c00640 1 -- 172.21.15.62:0/478523485 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe0fc02d040 con 0x7fe10806c300 2024-03-23T11:26:07.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.526+0000 7fe107e00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe0e806a7d0 0x7fe0e806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:07.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.527+0000 7fe107e00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe0e806a7d0 0x7fe0e806cc90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe108135850 tx=0x7fe0f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:07.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.530+0000 7fe104c00640 1 -- 172.21.15.62:0/478523485 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe0fc076820 con 0x7fe10806c300 2024-03-23T11:26:07.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.770+0000 7fe10fa00640 1 -- 172.21.15.62:0/478523485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1080ade50 con 0x7fe10806c300 2024-03-23T11:26:07.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.771+0000 7fe104c00640 1 -- 172.21.15.62:0/478523485 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fe0fc079e40 con 0x7fe10806c300 2024-03-23T11:26:07.764 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:07.765 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/29171509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":29171509},{"type":"v1","addr":"172.21.15.120:6837","nonce":29171509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:25:33.449465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6618":{"gid":6618,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":25,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:26:07.765 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 50 2024-03-23T11:26:07.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.773+0000 7fe10fa00640 1 -- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe0e806a7d0 msgr2=0x7fe0e806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:07.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.773+0000 7fe10fa00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe0e806a7d0 0x7fe0e806cc90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe108135850 tx=0x7fe0f4008040 comp rx=0 tx=0).stop 2024-03-23T11:26:07.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.773+0000 7fe10fa00640 1 -- 172.21.15.62:0/478523485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe10806c300 msgr2=0x7fe108157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:07.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.773+0000 7fe10fa00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe10806c300 0x7fe108157060 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc000c00 tx=0x7fe0fc0046c0 comp rx=0 tx=0).stop 2024-03-23T11:26:07.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.773+0000 7fe10fa00640 1 -- 172.21.15.62:0/478523485 shutdown_connections 2024-03-23T11:26:07.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.773+0000 7fe10fa00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe0e806a7d0 0x7fe0e806cc90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:07.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.773+0000 7fe10fa00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1080cafa0 0x7fe108132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:07.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.773+0000 7fe10fa00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe10806c6e0 0x7fe1080ac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:07.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.773+0000 7fe10fa00640 1 --2- 172.21.15.62:0/478523485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe10806c300 0x7fe108157060 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:07.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.774+0000 7fe10fa00640 1 -- 172.21.15.62:0/478523485 >> 172.21.15.62:0/478523485 conn(0x7fe1080b02e0 msgr2=0x7fe10815b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:07.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.774+0000 7fe10fa00640 1 -- 172.21.15.62:0/478523485 shutdown_connections 2024-03-23T11:26:07.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:07.774+0000 7fe10fa00640 1 -- 172.21.15.62:0/478523485 wait complete. 2024-03-23T11:26:07.788 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:26:08.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.014+0000 7f1b44c00640 1 Processor -- start 2024-03-23T11:26:08.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.014+0000 7f1b44c00640 1 -- start start 2024-03-23T11:26:08.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.016+0000 7f1b44c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1b40159490 0x7f1b4006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.016+0000 7f1b44c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400ced90 con 0x7f1b400ce5f0 2024-03-23T11:26:08.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.016+0000 7f1b44c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4006e3e0 con 0x7f1b4015da90 2024-03-23T11:26:08.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.016+0000 7f1b44c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b4006c1c0 con 0x7f1b40159490 2024-03-23T11:26:08.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.016+0000 7f1b3fe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1b400ce5f0 0x7f1b400b26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33186/0 (socket says 172.21.15.62:33186) 2024-03-23T11:26:08.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.016+0000 7f1b3fe00640 1 -- 172.21.15.62:0/1091266989 learned_addr learned my addr 172.21.15.62:0/1091266989 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:08.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3f400640 1 --2- 172.21.15.62:0/1091266989 >> v2:172.21.15.120:3300/0 conn(0x7f1b40159490 0x7f1b4006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484452003 0 0) 0x7f1b400ced90 con 0x7f1b400ce5f0 2024-03-23T11:26:08.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24003520 con 0x7f1b400ce5f0 2024-03-23T11:26:08.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1795944562 0 0) 0x7f1b24003520 con 0x7f1b400ce5f0 2024-03-23T11:26:08.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 >> v2:172.21.15.120:3300/0 conn(0x7f1b40159490 msgr2=0x7f1b4006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:08.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 --2- 172.21.15.62:0/1091266989 >> v2:172.21.15.120:3300/0 conn(0x7f1b40159490 0x7f1b4006cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 >> v1:172.21.15.120:6789/0 conn(0x7f1b4015da90 legacy=0x7f1b400cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4006e100 con 0x7f1b400ce5f0 2024-03-23T11:26:08.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f1b38001670 con 0x7f1b400ce5f0 2024-03-23T11:26:08.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 >> v1:172.21.15.62:6789/0 conn(0x7f1b400ce5f0 legacy=0x7f1b400b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 --2- 172.21.15.62:0/1091266989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b24003970 0x7f1b24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 --2- 172.21.15.62:0/1091266989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b240047c0 0x7f1b24004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 --2- 172.21.15.62:0/1091266989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b24005180 0x7f1b2400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4006c1c0 con 0x7f1b240047c0 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2400c120 con 0x7f1b24003970 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.017+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2400c2e0 con 0x7f1b24005180 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.018+0000 7f1b3ea00640 1 --2- 172.21.15.62:0/1091266989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b240047c0 0x7f1b24004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.018+0000 7f1b3f400640 1 --2- 172.21.15.62:0/1091266989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b24003970 0x7f1b24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.018+0000 7f1b3fe00640 1 --2- 172.21.15.62:0/1091266989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b24005180 0x7f1b2400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.018+0000 7f1b3ea00640 1 -- 172.21.15.62:0/1091266989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b24005180 msgr2=0x7f1b2400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.018+0000 7f1b3ea00640 1 --2- 172.21.15.62:0/1091266989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b24005180 0x7f1b2400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.018+0000 7f1b3ea00640 1 -- 172.21.15.62:0/1091266989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b24003970 msgr2=0x7f1b24003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.018+0000 7f1b3ea00640 1 --2- 172.21.15.62:0/1091266989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b24003970 0x7f1b24003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.018+0000 7f1b3ea00640 1 -- 172.21.15.62:0/1091266989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b24003520 con 0x7f1b240047c0 2024-03-23T11:26:08.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.018+0000 7f1b3fe00640 1 -- 172.21.15.62:0/1091266989 reap_dead start 2024-03-23T11:26:08.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.019+0000 7f1b3ea00640 1 --2- 172.21.15.62:0/1091266989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b240047c0 0x7f1b24004c40 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c001330 tx=0x7f1b240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a670e8a1e3b09b19 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:08.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.020+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c0052d0 con 0x7f1b240047c0 2024-03-23T11:26:08.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.020+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2c005430 con 0x7f1b240047c0 2024-03-23T11:26:08.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.020+0000 7f1b3e000640 1 -- 172.21.15.62:0/1091266989 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c003830 con 0x7f1b240047c0 2024-03-23T11:26:08.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.020+0000 7f1b44c00640 1 -- 172.21.15.62:0/1091266989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b240047c0 msgr2=0x7f1b24004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.020+0000 7f1b44c00640 1 --2- 172.21.15.62:0/1091266989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b240047c0 0x7f1b24004c40 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c001330 tx=0x7f1b240057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.020+0000 7f1b44c00640 1 -- 172.21.15.62:0/1091266989 shutdown_connections 2024-03-23T11:26:08.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.020+0000 7f1b44c00640 1 --2- 172.21.15.62:0/1091266989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b240047c0 0x7f1b24004c40 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.020+0000 7f1b44c00640 1 -- 172.21.15.62:0/1091266989 >> 172.21.15.62:0/1091266989 conn(0x7f1b40066050 msgr2=0x7f1b40069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:08.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.021+0000 7f1b44c00640 1 -- 172.21.15.62:0/1091266989 shutdown_connections 2024-03-23T11:26:08.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.021+0000 7f1b44c00640 1 -- 172.21.15.62:0/1091266989 wait complete. 2024-03-23T11:26:08.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.022+0000 7f1b44c00640 1 Processor -- start 2024-03-23T11:26:08.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.022+0000 7f1b44c00640 1 -- start start 2024-03-23T11:26:08.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.023+0000 7f1b44c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b240047c0 0x7f1b4014f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.023+0000 7f1b44c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b400ac5a0 0x7f1b400aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.023+0000 7f1b44c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b40153d70 0x7f1b401541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.023+0000 7f1b44c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b40154730 con 0x7f1b40153d70 2024-03-23T11:26:08.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.023+0000 7f1b44c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b401f5730 con 0x7f1b240047c0 2024-03-23T11:26:08.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.023+0000 7f1b44c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b401f5a10 con 0x7f1b400ac5a0 2024-03-23T11:26:08.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.024+0000 7f1b3fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b40153d70 0x7f1b401541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.024+0000 7f1b3fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b40153d70 0x7f1b401541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42724/0 (socket says 172.21.15.62:42724) 2024-03-23T11:26:08.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.024+0000 7f1b3fe00640 1 -- 172.21.15.62:0/1100564417 learned_addr learned my addr 172.21.15.62:0/1100564417 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:08.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.024+0000 7f1b3f400640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b240047c0 0x7f1b4014f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.024+0000 7f1b3ea00640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b400ac5a0 0x7f1b400aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.024+0000 7f1b3fe00640 1 -- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b400ac5a0 msgr2=0x7f1b400aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.024+0000 7f1b3fe00640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b400ac5a0 0x7f1b400aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.024+0000 7f1b3fe00640 1 -- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b240047c0 msgr2=0x7f1b4014f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.024+0000 7f1b3fe00640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b240047c0 0x7f1b4014f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.024+0000 7f1b3fe00640 1 -- 172.21.15.62:0/1100564417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b401f5cf0 con 0x7f1b40153d70 2024-03-23T11:26:08.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.024+0000 7f1b3f400640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b240047c0 0x7f1b4014f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:08.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.025+0000 7f1b3fe00640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b40153d70 0x7f1b401541f0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f1b4015cfe0 tx=0x7f1b3800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:08.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.025+0000 7f1b37400640 1 -- 172.21.15.62:0/1100564417 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3800eed0 con 0x7f1b40153d70 2024-03-23T11:26:08.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.025+0000 7f1b37400640 1 -- 172.21.15.62:0/1100564417 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3800c610 con 0x7f1b40153d70 2024-03-23T11:26:08.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.025+0000 7f1b37400640 1 -- 172.21.15.62:0/1100564417 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3800c9a0 con 0x7f1b40153d70 2024-03-23T11:26:08.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.025+0000 7f1b44c00640 1 -- 172.21.15.62:0/1100564417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b401f5fd0 con 0x7f1b40153d70 2024-03-23T11:26:08.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.025+0000 7f1b44c00640 1 -- 172.21.15.62:0/1100564417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b401500d0 con 0x7f1b40153d70 2024-03-23T11:26:08.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.026+0000 7f1b44c00640 1 -- 172.21.15.62:0/1100564417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b40159870 con 0x7f1b40153d70 2024-03-23T11:26:08.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.027+0000 7f1b37400640 1 -- 172.21.15.62:0/1100564417 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f1b3800cb00 con 0x7f1b40153d70 2024-03-23T11:26:08.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.029+0000 7f1b37400640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1b2006a7a0 0x7f1b2006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.029+0000 7f1b37400640 1 -- 172.21.15.62:0/1100564417 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1b38088aa0 con 0x7f1b40153d70 2024-03-23T11:26:08.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.029+0000 7f1b3f400640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1b2006a7a0 0x7f1b2006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.030+0000 7f1b3f400640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1b2006a7a0 0x7f1b2006cc60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1b40067260 tx=0x7f1b3000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:08.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.034+0000 7f1b37400640 1 -- 172.21.15.62:0/1100564417 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b38053e20 con 0x7f1b40153d70 2024-03-23T11:26:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.231+0000 7f1b44c00640 1 -- 172.21.15.62:0/1100564417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b40066bb0 con 0x7f1b40153d70 2024-03-23T11:26:08.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.234+0000 7f1b37400640 1 -- 172.21.15.62:0/1100564417 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16329 (secure 0 0 0) 0x7f1b38057440 con 0x7f1b40153d70 2024-03-23T11:26:08.230 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:08.230 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":77,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:44.311392+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:24:08.688706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-23T11:24:10.072774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1c","mappings":[{"from":6,"to":0}]},{"pgid":"11.3c","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6837/3833782560":"2024-03-24T11:24:34.350690+0000","172.21.15.120:6836/3833782560":"2024-03-24T11:24:34.350690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:26:08.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.237+0000 7f1b44c00640 1 -- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1b2006a7a0 msgr2=0x7f1b2006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.237+0000 7f1b44c00640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1b2006a7a0 0x7f1b2006cc60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1b40067260 tx=0x7f1b3000b040 comp rx=0 tx=0).stop 2024-03-23T11:26:08.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.237+0000 7f1b44c00640 1 -- 172.21.15.62:0/1100564417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b40153d70 msgr2=0x7f1b401541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.237+0000 7f1b44c00640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b40153d70 0x7f1b401541f0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f1b4015cfe0 tx=0x7f1b3800eb90 comp rx=0 tx=0).stop 2024-03-23T11:26:08.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.237+0000 7f1b44c00640 1 -- 172.21.15.62:0/1100564417 shutdown_connections 2024-03-23T11:26:08.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.237+0000 7f1b44c00640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1b2006a7a0 0x7f1b2006cc60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.237+0000 7f1b44c00640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b40153d70 0x7f1b401541f0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.237+0000 7f1b44c00640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b400ac5a0 0x7f1b400aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.237+0000 7f1b44c00640 1 --2- 172.21.15.62:0/1100564417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b240047c0 0x7f1b4014f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.237+0000 7f1b44c00640 1 -- 172.21.15.62:0/1100564417 >> 172.21.15.62:0/1100564417 conn(0x7f1b40066050 msgr2=0x7f1b40158620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:08.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.238+0000 7f1b44c00640 1 -- 172.21.15.62:0/1100564417 shutdown_connections 2024-03-23T11:26:08.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.238+0000 7f1b44c00640 1 -- 172.21.15.62:0/1100564417 wait complete. 2024-03-23T11:26:08.252 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:24:08.688706+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:24:10.072774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:26:08.252 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:26:08.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.475+0000 7f8c1be00640 1 Processor -- start 2024-03-23T11:26:08.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.476+0000 7f8c1be00640 1 -- start start 2024-03-23T11:26:08.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.477+0000 7f8c1be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8c1c070090 0x7f8c1c070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.477+0000 7f8c1be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8c1c0b2820 0x7f8c1c0b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.477+0000 7f8c1be00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1c0b31b0 con 0x7f8c1c143dd0 2024-03-23T11:26:08.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.477+0000 7f8c1be00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c1c070a60 con 0x7f8c1c070090 2024-03-23T11:26:08.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.477+0000 7f8c1be00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c1c0cd140 con 0x7f8c1c0b2820 2024-03-23T11:26:08.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.478+0000 7f8c13e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8c1c070090 0x7f8c1c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.478+0000 7f8c13e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8c1c070090 0x7f8c1c070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42740/0 (socket says 172.21.15.62:42740) 2024-03-23T11:26:08.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.478+0000 7f8c13e00640 1 -- 172.21.15.62:0/4129310123 learned_addr learned my addr 172.21.15.62:0/4129310123 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:08.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.478+0000 7f8c1b400640 1 --2- 172.21.15.62:0/4129310123 >> v2:172.21.15.120:3300/0 conn(0x7f8c1c0b2820 0x7f8c1c0b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.479+0000 7f8c13e00640 1 -- 172.21.15.62:0/4129310123 >> v2:172.21.15.120:3300/0 conn(0x7f8c1c0b2820 msgr2=0x7f8c1c0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:08.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.479+0000 7f8c13e00640 1 --2- 172.21.15.62:0/4129310123 >> v2:172.21.15.120:3300/0 conn(0x7f8c1c0b2820 0x7f8c1c0b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.479+0000 7f8c13e00640 1 -- 172.21.15.62:0/4129310123 >> v1:172.21.15.120:6789/0 conn(0x7f8c1c143dd0 legacy=0x7f8c1c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.479+0000 7f8c13e00640 1 -- 172.21.15.62:0/4129310123 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c0cd3e0 con 0x7f8c1c070090 2024-03-23T11:26:08.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.480+0000 7f8c13e00640 1 --2- 172.21.15.62:0/4129310123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c1c070090 0x7f8c1c070520 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c0cbe50 tx=0x7f8c0402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4dcc9aac3079324 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:08.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.480+0000 7f8c1a000640 1 -- 172.21.15.62:0/4129310123 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04030860 con 0x7f8c1c070090 2024-03-23T11:26:08.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.480+0000 7f8c1a000640 1 -- 172.21.15.62:0/4129310123 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c040309c0 con 0x7f8c1c070090 2024-03-23T11:26:08.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.480+0000 7f8c1a000640 1 -- 172.21.15.62:0/4129310123 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04030d50 con 0x7f8c1c070090 2024-03-23T11:26:08.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.481+0000 7f8c1be00640 1 -- 172.21.15.62:0/4129310123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c1c070090 msgr2=0x7f8c1c070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.481+0000 7f8c1be00640 1 --2- 172.21.15.62:0/4129310123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c1c070090 0x7f8c1c070520 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c0cbe50 tx=0x7f8c0402b9a0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.481+0000 7f8c1be00640 1 -- 172.21.15.62:0/4129310123 shutdown_connections 2024-03-23T11:26:08.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.481+0000 7f8c1be00640 1 --2- 172.21.15.62:0/4129310123 >> v2:172.21.15.120:3300/0 conn(0x7f8c1c0b2820 0x7f8c1c0b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.481+0000 7f8c1be00640 1 --2- 172.21.15.62:0/4129310123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c1c070090 0x7f8c1c070520 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.481+0000 7f8c1be00640 1 -- 172.21.15.62:0/4129310123 >> 172.21.15.62:0/4129310123 conn(0x7f8c1c066050 msgr2=0x7f8c1c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:08.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.481+0000 7f8c1be00640 1 -- 172.21.15.62:0/4129310123 shutdown_connections 2024-03-23T11:26:08.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.481+0000 7f8c1be00640 1 -- 172.21.15.62:0/4129310123 wait complete. 2024-03-23T11:26:08.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.483+0000 7f8c1be00640 1 Processor -- start 2024-03-23T11:26:08.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.483+0000 7f8c1be00640 1 -- start start 2024-03-23T11:26:08.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.484+0000 7f8c1be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c1c070090 0x7f8c1c0c3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.484+0000 7f8c1be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c1c0b2820 0x7f8c1c0a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.484+0000 7f8c1be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c1c143dd0 0x7f8c1c0a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.484+0000 7f8c1be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c135e10 con 0x7f8c1c0b2820 2024-03-23T11:26:08.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.484+0000 7f8c1be00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c136320 con 0x7f8c1c070090 2024-03-23T11:26:08.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.484+0000 7f8c1be00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c0a90d0 con 0x7f8c1c143dd0 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.484+0000 7f8c13e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c1c0b2820 0x7f8c1c0a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.484+0000 7f8c13e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c1c0b2820 0x7f8c1c0a8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42748/0 (socket says 172.21.15.62:42748) 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.484+0000 7f8c13e00640 1 -- 172.21.15.62:0/4102136144 learned_addr learned my addr 172.21.15.62:0/4102136144 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.484+0000 7f8c1aa00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c1c070090 0x7f8c1c0c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c1b400640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c1c143dd0 0x7f8c1c0a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c13e00640 1 -- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c1c143dd0 msgr2=0x7f8c1c0a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c13e00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c1c143dd0 0x7f8c1c0a89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c13e00640 1 -- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c1c070090 msgr2=0x7f8c1c0c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c13e00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c1c070090 0x7f8c1c0c3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c13e00640 1 -- 172.21.15.62:0/4102136144 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c040094d0 con 0x7f8c1c0b2820 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c1aa00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c1c070090 0x7f8c1c0c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c13e00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c1c0b2820 0x7f8c1c0a8480 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f8c0402b9e0 tx=0x7f8c04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:08.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c12a00640 1 -- 172.21.15.62:0/4102136144 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04039d60 con 0x7f8c1c0b2820 2024-03-23T11:26:08.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c12a00640 1 -- 172.21.15.62:0/4102136144 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c04043400 con 0x7f8c1c0b2820 2024-03-23T11:26:08.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c12a00640 1 -- 172.21.15.62:0/4102136144 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040437b0 con 0x7f8c1c0b2820 2024-03-23T11:26:08.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.485+0000 7f8c1be00640 1 -- 172.21.15.62:0/4102136144 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1c0a93b0 con 0x7f8c1c0b2820 2024-03-23T11:26:08.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.486+0000 7f8c1be00640 1 -- 172.21.15.62:0/4102136144 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1c0a9970 con 0x7f8c1c0b2820 2024-03-23T11:26:08.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.487+0000 7f8c1be00640 1 -- 172.21.15.62:0/4102136144 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1c0ccd20 con 0x7f8c1c0b2820 2024-03-23T11:26:08.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.488+0000 7f8c12a00640 1 -- 172.21.15.62:0/4102136144 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8c04043910 con 0x7f8c1c0b2820 2024-03-23T11:26:08.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.489+0000 7f8c12a00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8bf806a7a0 0x7f8bf806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.489+0000 7f8c12a00640 1 -- 172.21.15.62:0/4102136144 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8c0403e070 con 0x7f8c1c0b2820 2024-03-23T11:26:08.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.490+0000 7f8c1aa00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8bf806a7a0 0x7f8bf806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.491+0000 7f8c1aa00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8bf806a7a0 0x7f8bf806cc60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c000f60 tx=0x7f8c0c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:08.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.494+0000 7f8c12a00640 1 -- 172.21.15.62:0/4102136144 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8c040b0070 con 0x7f8c1c0b2820 2024-03-23T11:26:08.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.692+0000 7f8c1be00640 1 -- 172.21.15.62:0/4102136144 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c1c066bb0 con 0x7f8c1c0b2820 2024-03-23T11:26:08.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.693+0000 7f8c12a00640 1 -- 172.21.15.62:0/4102136144 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c04034020 con 0x7f8c1c0b2820 2024-03-23T11:26:08.687 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:26:08.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 -- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8bf806a7a0 msgr2=0x7f8bf806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8bf806a7a0 0x7f8bf806cc60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c000f60 tx=0x7f8c0c009040 comp rx=0 tx=0).stop 2024-03-23T11:26:08.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 -- 172.21.15.62:0/4102136144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c1c0b2820 msgr2=0x7f8c1c0a8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c1c0b2820 0x7f8c1c0a8480 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f8c0402b9e0 tx=0x7f8c04004730 comp rx=0 tx=0).stop 2024-03-23T11:26:08.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 -- 172.21.15.62:0/4102136144 shutdown_connections 2024-03-23T11:26:08.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8bf806a7a0 0x7f8bf806cc60 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c1c0b2820 0x7f8c1c0a8480 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c1c143dd0 0x7f8c1c0a89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 --2- 172.21.15.62:0/4102136144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c1c070090 0x7f8c1c0c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 -- 172.21.15.62:0/4102136144 >> 172.21.15.62:0/4102136144 conn(0x7f8c1c066050 msgr2=0x7f8c1c0adde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:08.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 -- 172.21.15.62:0/4102136144 shutdown_connections 2024-03-23T11:26:08.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.696+0000 7f8c1be00640 1 -- 172.21.15.62:0/4102136144 wait complete. 2024-03-23T11:26:08.708 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:26:08.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.931+0000 7f6b17e00640 1 Processor -- start 2024-03-23T11:26:08.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.932+0000 7f6b17e00640 1 -- start start 2024-03-23T11:26:08.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.933+0000 7f6b17e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6b180b2f30 0x7f6b18070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.933+0000 7f6b17e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b180cd140 con 0x7f6b18070090 2024-03-23T11:26:08.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.933+0000 7f6b17e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b180cd3e0 con 0x7f6b18143dd0 2024-03-23T11:26:08.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.934+0000 7f6b17e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b18070a60 con 0x7f6b180b2f30 2024-03-23T11:26:08.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.934+0000 7f6b0fe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6b18070090 0x7f6b180b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33200/0 (socket says 172.21.15.62:33200) 2024-03-23T11:26:08.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.934+0000 7f6b0fe00640 1 -- 172.21.15.62:0/409691792 learned_addr learned my addr 172.21.15.62:0/409691792 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:08.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.934+0000 7f6b17400640 1 --2- 172.21.15.62:0/409691792 >> v2:172.21.15.120:3300/0 conn(0x7f6b180b2f30 0x7f6b18070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.934+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432497548 0 0) 0x7f6b180cd140 con 0x7f6b18070090 2024-03-23T11:26:08.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.934+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6af8003520 con 0x7f6b18070090 2024-03-23T11:26:08.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856141864 0 0) 0x7f6b180cd3e0 con 0x7f6b18143dd0 2024-03-23T11:26:08.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b180cd140 con 0x7f6b18143dd0 2024-03-23T11:26:08.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 808754132 0 0) 0x7f6af8003520 con 0x7f6b18070090 2024-03-23T11:26:08.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 >> v2:172.21.15.120:3300/0 conn(0x7f6b180b2f30 msgr2=0x7f6b18070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:08.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 --2- 172.21.15.62:0/409691792 >> v2:172.21.15.120:3300/0 conn(0x7f6b180b2f30 0x7f6b18070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 >> v1:172.21.15.120:6789/0 conn(0x7f6b18143dd0 legacy=0x7f6b1815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b180cd6c0 con 0x7f6b18070090 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f6b00001670 con 0x7f6b18070090 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b17400640 1 --2- 172.21.15.62:0/409691792 >> v2:172.21.15.120:3300/0 conn(0x7f6b180b2f30 0x7f6b18070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 >> v1:172.21.15.62:6789/0 conn(0x7f6b18070090 legacy=0x7f6b180b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af8003b00 0x7f6af8003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6af80047c0 0x7f6af8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6af8005180 0x7f6af800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b18070a60 con 0x7f6af8005180 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af800c000 con 0x7f6af80047c0 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af800c1c0 con 0x7f6af8003b00 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.935+0000 7f6b17400640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6af8005180 0x7f6af800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.936+0000 7f6b16a00640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af8003b00 0x7f6af8003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.936+0000 7f6b0fe00640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6af80047c0 0x7f6af8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.936+0000 7f6b17400640 1 -- 172.21.15.62:0/409691792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af8003b00 msgr2=0x7f6af8003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:08.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.936+0000 7f6b17400640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af8003b00 0x7f6af8003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.936+0000 7f6b17400640 1 -- 172.21.15.62:0/409691792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6af80047c0 msgr2=0x7f6af8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:08.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.936+0000 7f6b17400640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6af80047c0 0x7f6af8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.936+0000 7f6b17400640 1 -- 172.21.15.62:0/409691792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af8003520 con 0x7f6af8005180 2024-03-23T11:26:08.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.936+0000 7f6b16a00640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af8003b00 0x7f6af8003f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:08.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.937+0000 7f6b17400640 1 -- 172.21.15.62:0/409691792 reap_dead start 2024-03-23T11:26:08.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.937+0000 7f6b17400640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6af8005180 0x7f6af800b980 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000e350 tx=0x7f6af80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad4a448518c69e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.937+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1002ce00 con 0x7f6af8005180 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.937+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1000f040 con 0x7f6af8005180 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.937+0000 7f6b16000640 1 -- 172.21.15.62:0/409691792 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b10003bd0 con 0x7f6af8005180 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.938+0000 7f6b17e00640 1 -- 172.21.15.62:0/409691792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6af8005180 msgr2=0x7f6af800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.938+0000 7f6b17e00640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6af8005180 0x7f6af800b980 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000e350 tx=0x7f6af80057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.938+0000 7f6b17e00640 1 -- 172.21.15.62:0/409691792 shutdown_connections 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.938+0000 7f6b17e00640 1 --2- 172.21.15.62:0/409691792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6af8005180 0x7f6af800b980 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.938+0000 7f6b17e00640 1 -- 172.21.15.62:0/409691792 >> 172.21.15.62:0/409691792 conn(0x7f6b18066050 msgr2=0x7f6b18069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.938+0000 7f6b17e00640 1 -- 172.21.15.62:0/409691792 shutdown_connections 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.938+0000 7f6b17e00640 1 -- 172.21.15.62:0/409691792 wait complete. 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.940+0000 7f6b17e00640 1 Processor -- start 2024-03-23T11:26:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.940+0000 7f6b17e00640 1 -- start start 2024-03-23T11:26:08.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.941+0000 7f6b17e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af8005180 0x7f6b1813da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.941+0000 7f6b17e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b180ac690 0x7f6b180a98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.941+0000 7f6b17e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b1813f6d0 0x7f6b180aa240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.941+0000 7f6b17e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b180a7430 con 0x7f6b180ac690 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.941+0000 7f6b17e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b180a9df0 con 0x7f6b1813f6d0 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.941+0000 7f6b17e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b180acc30 con 0x7f6af8005180 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.941+0000 7f6b0fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b180ac690 0x7f6b180a98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.942+0000 7f6b0fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b180ac690 0x7f6b180a98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42772/0 (socket says 172.21.15.62:42772) 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.942+0000 7f6b0fe00640 1 -- 172.21.15.62:0/319430853 learned_addr learned my addr 172.21.15.62:0/319430853 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.942+0000 7f6b16a00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af8005180 0x7f6b1813da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.942+0000 7f6b17400640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b1813f6d0 0x7f6b180aa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.942+0000 7f6b0fe00640 1 -- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af8005180 msgr2=0x7f6b1813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.942+0000 7f6b0fe00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af8005180 0x7f6b1813da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.942+0000 7f6b0fe00640 1 -- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b1813f6d0 msgr2=0x7f6b180aa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.942+0000 7f6b0fe00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b1813f6d0 0x7f6b180aa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:08.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.942+0000 7f6b0fe00640 1 -- 172.21.15.62:0/319430853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b180a76f0 con 0x7f6b180ac690 2024-03-23T11:26:08.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.942+0000 7f6b17400640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b1813f6d0 0x7f6b180aa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:08.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.942+0000 7f6b0fe00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b180ac690 0x7f6b180a98b0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000d770 tx=0x7f6b0000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:08.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.943+0000 7f6b0ea00640 1 -- 172.21.15.62:0/319430853 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b000074e0 con 0x7f6b180ac690 2024-03-23T11:26:08.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.943+0000 7f6b0ea00640 1 -- 172.21.15.62:0/319430853 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b00007640 con 0x7f6b180ac690 2024-03-23T11:26:08.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.943+0000 7f6b17e00640 1 -- 172.21.15.62:0/319430853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b180a79d0 con 0x7f6b180ac690 2024-03-23T11:26:08.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.943+0000 7f6b0ea00640 1 -- 172.21.15.62:0/319430853 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b00004a00 con 0x7f6b180ac690 2024-03-23T11:26:08.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.943+0000 7f6b17e00640 1 -- 172.21.15.62:0/319430853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1813fb60 con 0x7f6b180ac690 2024-03-23T11:26:08.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.944+0000 7f6b17e00640 1 -- 172.21.15.62:0/319430853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b180a7ef0 con 0x7f6b180ac690 2024-03-23T11:26:08.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.945+0000 7f6b0ea00640 1 -- 172.21.15.62:0/319430853 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6b00004b60 con 0x7f6b180ac690 2024-03-23T11:26:08.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.945+0000 7f6b0ea00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6af406a7a0 0x7f6af406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:08.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.946+0000 7f6b0ea00640 1 -- 172.21.15.62:0/319430853 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f6b00088ae0 con 0x7f6b180ac690 2024-03-23T11:26:08.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.946+0000 7f6b16a00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6af406a7a0 0x7f6af406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:08.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.947+0000 7f6b16a00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6af406a7a0 0x7f6af406cc60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6b080011e0 tx=0x7f6b08009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:08.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:08.950+0000 7f6b0ea00640 1 -- 172.21.15.62:0/319430853 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6b180a7ef0 con 0x7f6b180ac690 2024-03-23T11:26:09.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.146+0000 7f6b17e00640 1 -- 172.21.15.62:0/319430853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6b180a7ef0 con 0x7f6b180ac690 2024-03-23T11:26:09.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.147+0000 7f6b0ea00640 1 -- 172.21.15.62:0/319430853 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6b180a7ef0 con 0x7f6b180ac690 2024-03-23T11:26:09.140 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:26:09.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.149+0000 7f6b17e00640 1 -- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6af406a7a0 msgr2=0x7f6af406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:09.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.149+0000 7f6b17e00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6af406a7a0 0x7f6af406cc60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6b080011e0 tx=0x7f6b08009040 comp rx=0 tx=0).stop 2024-03-23T11:26:09.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.149+0000 7f6b17e00640 1 -- 172.21.15.62:0/319430853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b180ac690 msgr2=0x7f6b180a98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:09.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.149+0000 7f6b17e00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b180ac690 0x7f6b180a98b0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000d770 tx=0x7f6b0000dc30 comp rx=0 tx=0).stop 2024-03-23T11:26:09.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.149+0000 7f6b17e00640 1 -- 172.21.15.62:0/319430853 shutdown_connections 2024-03-23T11:26:09.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.149+0000 7f6b17e00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6af406a7a0 0x7f6af406cc60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.149+0000 7f6b17e00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b180ac690 0x7f6b180a98b0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.149+0000 7f6b17e00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b1813f6d0 0x7f6b180aa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.150+0000 7f6b17e00640 1 --2- 172.21.15.62:0/319430853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6af8005180 0x7f6b1813da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.150+0000 7f6b17e00640 1 -- 172.21.15.62:0/319430853 >> 172.21.15.62:0/319430853 conn(0x7f6b18066050 msgr2=0x7f6b180b05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:09.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.150+0000 7f6b17e00640 1 -- 172.21.15.62:0/319430853 shutdown_connections 2024-03-23T11:26:09.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.150+0000 7f6b17e00640 1 -- 172.21.15.62:0/319430853 wait complete. 2024-03-23T11:26:09.163 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:26:09.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.383+0000 7f6d7c600640 1 Processor -- start 2024-03-23T11:26:09.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.384+0000 7f6d7c600640 1 -- start start 2024-03-23T11:26:09.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.385+0000 7f6d7c600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d7415cdc0 0x7f6d740d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.385+0000 7f6d7c600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7406bfc0 con 0x7f6d7415c9e0 2024-03-23T11:26:09.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.385+0000 7f6d7c600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740b5260 con 0x7f6d740cf650 2024-03-23T11:26:09.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.385+0000 7f6d7c600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d740ce140 con 0x7f6d7415cdc0 2024-03-23T11:26:09.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.386+0000 7f6d79c00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6d7415c9e0 0x7f6d741607c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33212/0 (socket says 172.21.15.62:33212) 2024-03-23T11:26:09.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.386+0000 7f6d79c00640 1 -- 172.21.15.62:0/3321232438 learned_addr learned my addr 172.21.15.62:0/3321232438 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:09.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.386+0000 7f6d79200640 1 --2- 172.21.15.62:0/3321232438 >> v2:172.21.15.62:3300/0 conn(0x7f6d7415cdc0 0x7f6d740d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:09.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.386+0000 7f6d73e00640 1 -- 172.21.15.62:0/3321232438 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185928263 0 0) 0x7f6d7406bfc0 con 0x7f6d7415c9e0 2024-03-23T11:26:09.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.386+0000 7f6d73e00640 1 -- 172.21.15.62:0/3321232438 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d5c003520 con 0x7f6d7415c9e0 2024-03-23T11:26:09.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.386+0000 7f6d79200640 1 -- 172.21.15.62:0/3321232438 >> v1:172.21.15.120:6789/0 conn(0x7f6d740cf650 legacy=0x7f6d7415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:09.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.386+0000 7f6d79200640 1 -- 172.21.15.62:0/3321232438 >> v1:172.21.15.62:6789/0 conn(0x7f6d7415c9e0 legacy=0x7f6d741607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:09.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.386+0000 7f6d79200640 1 -- 172.21.15.62:0/3321232438 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d740b4370 con 0x7f6d7415cdc0 2024-03-23T11:26:09.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.387+0000 7f6d79200640 1 --2- 172.21.15.62:0/3321232438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d7415cdc0 0x7f6d740d1f20 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097a0 tx=0x7f6d6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225d9e17d6f65fd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:09.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.388+0000 7f6d73e00640 1 -- 172.21.15.62:0/3321232438 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60031830 con 0x7f6d7415cdc0 2024-03-23T11:26:09.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.388+0000 7f6d73e00640 1 -- 172.21.15.62:0/3321232438 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d60031990 con 0x7f6d7415cdc0 2024-03-23T11:26:09.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.388+0000 7f6d73e00640 1 -- 172.21.15.62:0/3321232438 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60031d20 con 0x7f6d7415cdc0 2024-03-23T11:26:09.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.388+0000 7f6d7c600640 1 -- 172.21.15.62:0/3321232438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d7415cdc0 msgr2=0x7f6d740d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:09.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.388+0000 7f6d7c600640 1 --2- 172.21.15.62:0/3321232438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d7415cdc0 0x7f6d740d1f20 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097a0 tx=0x7f6d6002b8d0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.388+0000 7f6d7c600640 1 -- 172.21.15.62:0/3321232438 shutdown_connections 2024-03-23T11:26:09.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.388+0000 7f6d7c600640 1 --2- 172.21.15.62:0/3321232438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d7415cdc0 0x7f6d740d1f20 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.388+0000 7f6d7c600640 1 -- 172.21.15.62:0/3321232438 >> 172.21.15.62:0/3321232438 conn(0x7f6d74071570 msgr2=0x7f6d74071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:09.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.388+0000 7f6d7c600640 1 -- 172.21.15.62:0/3321232438 shutdown_connections 2024-03-23T11:26:09.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.388+0000 7f6d7c600640 1 -- 172.21.15.62:0/3321232438 wait complete. 2024-03-23T11:26:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.390+0000 7f6d7c600640 1 Processor -- start 2024-03-23T11:26:09.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.390+0000 7f6d7c600640 1 -- start start 2024-03-23T11:26:09.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.391+0000 7f6d7c600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d740cf650 0x7f6d741f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.391+0000 7f6d7c600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d7415c9e0 0x7f6d74152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.391+0000 7f6d7c600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d7415cdc0 0x7f6d7415d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.391+0000 7f6d7c600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d740cb530 con 0x7f6d7415cdc0 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.391+0000 7f6d7c600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d740cac00 con 0x7f6d740cf650 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.391+0000 7f6d7c600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d740caee0 con 0x7f6d7415c9e0 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d7a600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d7415cdc0 0x7f6d7415d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d7a600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d7415cdc0 0x7f6d7415d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42792/0 (socket says 172.21.15.62:42792) 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d7a600640 1 -- 172.21.15.62:0/188822167 learned_addr learned my addr 172.21.15.62:0/188822167 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d79200640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d7415c9e0 0x7f6d74152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d79c00640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d740cf650 0x7f6d741f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d7a600640 1 -- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d7415c9e0 msgr2=0x7f6d74152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:09.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d7a600640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d7415c9e0 0x7f6d74152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d7a600640 1 -- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d740cf650 msgr2=0x7f6d741f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:09.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d7a600640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d740cf650 0x7f6d741f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d7a600640 1 -- 172.21.15.62:0/188822167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d60009400 con 0x7f6d7415cdc0 2024-03-23T11:26:09.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d79200640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d7415c9e0 0x7f6d74152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:09.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.392+0000 7f6d7a600640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d7415cdc0 0x7f6d7415d440 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c00c780 tx=0x7f6d6c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:09.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.393+0000 7f6d71600640 1 -- 172.21.15.62:0/188822167 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c004b60 con 0x7f6d7415cdc0 2024-03-23T11:26:09.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.393+0000 7f6d71600640 1 -- 172.21.15.62:0/188822167 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6c00ce90 con 0x7f6d7415cdc0 2024-03-23T11:26:09.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.393+0000 7f6d71600640 1 -- 172.21.15.62:0/188822167 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c00a850 con 0x7f6d7415cdc0 2024-03-23T11:26:09.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.393+0000 7f6d7c600640 1 -- 172.21.15.62:0/188822167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d740cb1c0 con 0x7f6d7415cdc0 2024-03-23T11:26:09.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.393+0000 7f6d7c600640 1 -- 172.21.15.62:0/188822167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d74152550 con 0x7f6d7415cdc0 2024-03-23T11:26:09.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.394+0000 7f6d7c600640 1 -- 172.21.15.62:0/188822167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d740cb6a0 con 0x7f6d7415cdc0 2024-03-23T11:26:09.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.395+0000 7f6d71600640 1 -- 172.21.15.62:0/188822167 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6d6c00a9b0 con 0x7f6d7415cdc0 2024-03-23T11:26:09.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.396+0000 7f6d71600640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d5406a7a0 0x7f6d5406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.396+0000 7f6d71600640 1 -- 172.21.15.62:0/188822167 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f6d6c0890f0 con 0x7f6d7415cdc0 2024-03-23T11:26:09.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.396+0000 7f6d79c00640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d5406a7a0 0x7f6d5406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:09.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.398+0000 7f6d79c00640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d5406a7a0 0x7f6d5406cc60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6d680011e0 tx=0x7f6d68009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:09.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.400+0000 7f6d71600640 1 -- 172.21.15.62:0/188822167 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6d6c0543e0 con 0x7f6d7415cdc0 2024-03-23T11:26:09.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.596+0000 7f6d7c600640 1 -- 172.21.15.62:0/188822167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6d74066c90 con 0x7f6d7415cdc0 2024-03-23T11:26:09.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.597+0000 7f6d71600640 1 -- 172.21.15.62:0/188822167 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6d6c057a00 con 0x7f6d7415cdc0 2024-03-23T11:26:09.591 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:26:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 -- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d5406a7a0 msgr2=0x7f6d5406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d5406a7a0 0x7f6d5406cc60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6d680011e0 tx=0x7f6d68009040 comp rx=0 tx=0).stop 2024-03-23T11:26:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 -- 172.21.15.62:0/188822167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d7415cdc0 msgr2=0x7f6d7415d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d7415cdc0 0x7f6d7415d440 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c00c780 tx=0x7f6d6c00cc40 comp rx=0 tx=0).stop 2024-03-23T11:26:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 -- 172.21.15.62:0/188822167 shutdown_connections 2024-03-23T11:26:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d5406a7a0 0x7f6d5406cc60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d7415cdc0 0x7f6d7415d440 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d7415c9e0 0x7f6d74152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 --2- 172.21.15.62:0/188822167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d740cf650 0x7f6d741f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 -- 172.21.15.62:0/188822167 >> 172.21.15.62:0/188822167 conn(0x7f6d74071570 msgr2=0x7f6d7415be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 -- 172.21.15.62:0/188822167 shutdown_connections 2024-03-23T11:26:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.600+0000 7f6d7c600640 1 -- 172.21.15.62:0/188822167 wait complete. 2024-03-23T11:26:09.612 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:26:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.830+0000 7f04b4e00640 1 Processor -- start 2024-03-23T11:26:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.831+0000 7f04b4e00640 1 -- start start 2024-03-23T11:26:09.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.832+0000 7f04b4e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f04b015c9e0 0x7f04b01607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.832+0000 7f04b4e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04b015cdc0 0x7f04b00d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04b4e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f04b00cf650 0x7f04b00cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04b4e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b00d0020 con 0x7f04b015c9e0 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04b4e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b006bfc0 con 0x7f04b00cf650 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04b4e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00b5260 con 0x7f04b015cdc0 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04ade00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f04b015c9e0 0x7f04b01607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04ade00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f04b015c9e0 0x7f04b01607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42802/0 (socket says 172.21.15.62:42802) 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04ade00640 1 -- 172.21.15.62:0/2062728888 learned_addr learned my addr 172.21.15.62:0/2062728888 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04ade00640 1 -- 172.21.15.62:0/2062728888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04b015cdc0 msgr2=0x7f04b00d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04ade00640 1 --2- 172.21.15.62:0/2062728888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04b015cdc0 0x7f04b00d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04ade00640 1 -- 172.21.15.62:0/2062728888 >> v2:172.21.15.120:3300/0 conn(0x7f04b00cf650 msgr2=0x7f04b00cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04ade00640 1 --2- 172.21.15.62:0/2062728888 >> v2:172.21.15.120:3300/0 conn(0x7f04b00cf650 0x7f04b00cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.833+0000 7f04ade00640 1 -- 172.21.15.62:0/2062728888 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b00ce140 con 0x7f04b015c9e0 2024-03-23T11:26:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.834+0000 7f04ade00640 1 --2- 172.21.15.62:0/2062728888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04b015c9e0 0x7f04b01607c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f04b00b51d0 tx=0x7f04a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d5356d71fa0e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.835+0000 7f04aca00640 1 -- 172.21.15.62:0/2062728888 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0031640 con 0x7f04b015c9e0 2024-03-23T11:26:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.835+0000 7f04aca00640 1 -- 172.21.15.62:0/2062728888 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a00317a0 con 0x7f04b015c9e0 2024-03-23T11:26:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.835+0000 7f04aca00640 1 -- 172.21.15.62:0/2062728888 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0031b30 con 0x7f04b015c9e0 2024-03-23T11:26:09.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.835+0000 7f04b4e00640 1 -- 172.21.15.62:0/2062728888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04b015c9e0 msgr2=0x7f04b01607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:09.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.835+0000 7f04b4e00640 1 --2- 172.21.15.62:0/2062728888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04b015c9e0 0x7f04b01607c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f04b00b51d0 tx=0x7f04a002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.836+0000 7f04b4e00640 1 -- 172.21.15.62:0/2062728888 shutdown_connections 2024-03-23T11:26:09.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.836+0000 7f04b4e00640 1 --2- 172.21.15.62:0/2062728888 >> v2:172.21.15.120:3300/0 conn(0x7f04b00cf650 0x7f04b00cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.836+0000 7f04b4e00640 1 --2- 172.21.15.62:0/2062728888 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04b015cdc0 0x7f04b00d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.836+0000 7f04b4e00640 1 --2- 172.21.15.62:0/2062728888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04b015c9e0 0x7f04b01607c0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.836+0000 7f04b4e00640 1 -- 172.21.15.62:0/2062728888 >> 172.21.15.62:0/2062728888 conn(0x7f04b0071570 msgr2=0x7f04b0071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:09.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.836+0000 7f04b4e00640 1 -- 172.21.15.62:0/2062728888 shutdown_connections 2024-03-23T11:26:09.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.836+0000 7f04b4e00640 1 -- 172.21.15.62:0/2062728888 wait complete. 2024-03-23T11:26:09.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.838+0000 7f04b4e00640 1 Processor -- start 2024-03-23T11:26:09.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.838+0000 7f04b4e00640 1 -- start start 2024-03-23T11:26:09.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.839+0000 7f04b4e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b00cf650 0x7f04b00d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04b4e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04b015c9e0 0x7f04b00a9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04b4e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04b015cdc0 0x7f04b00a9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04b4e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00b5e80 con 0x7f04b015c9e0 2024-03-23T11:26:09.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04b4e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00ab4d0 con 0x7f04b00cf650 2024-03-23T11:26:09.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04b4e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0159ad0 con 0x7f04b015cdc0 2024-03-23T11:26:09.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04ad400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04b015c9e0 0x7f04b00a9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:09.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04ad400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04b015c9e0 0x7f04b00a9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42816/0 (socket says 172.21.15.62:42816) 2024-03-23T11:26:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04ad400640 1 -- 172.21.15.62:0/1565527086 learned_addr learned my addr 172.21.15.62:0/1565527086 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04ade00640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b00cf650 0x7f04b00d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04ad400640 1 -- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04b015cdc0 msgr2=0x7f04b00a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04ae800640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04b015cdc0 0x7f04b00a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04ad400640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04b015cdc0 0x7f04b00a9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04ad400640 1 -- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b00cf650 msgr2=0x7f04b00d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04ad400640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b00cf650 0x7f04b00d1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04ad400640 1 -- 172.21.15.62:0/1565527086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a00094d0 con 0x7f04b015c9e0 2024-03-23T11:26:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.840+0000 7f04ade00640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b00cf650 0x7f04b00d1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.841+0000 7f04ad400640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04b015c9e0 0x7f04b00a9300 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f049800b440 tx=0x7f049800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.841+0000 7f04a6000640 1 -- 172.21.15.62:0/1565527086 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0498011890 con 0x7f04b015c9e0 2024-03-23T11:26:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.841+0000 7f04a6000640 1 -- 172.21.15.62:0/1565527086 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04980119f0 con 0x7f04b015c9e0 2024-03-23T11:26:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.841+0000 7f04a6000640 1 -- 172.21.15.62:0/1565527086 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0498011da0 con 0x7f04b015c9e0 2024-03-23T11:26:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.841+0000 7f04b4e00640 1 -- 172.21.15.62:0/1565527086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b00ab720 con 0x7f04b015c9e0 2024-03-23T11:26:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.841+0000 7f04b4e00640 1 -- 172.21.15.62:0/1565527086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b00abce0 con 0x7f04b015c9e0 2024-03-23T11:26:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.842+0000 7f04b4e00640 1 -- 172.21.15.62:0/1565527086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b00b5fc0 con 0x7f04b015c9e0 2024-03-23T11:26:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.843+0000 7f04a6000640 1 -- 172.21.15.62:0/1565527086 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f049801a400 con 0x7f04b015c9e0 2024-03-23T11:26:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.844+0000 7f04a6000640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f048c06a7a0 0x7f048c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.844+0000 7f04a6000640 1 -- 172.21.15.62:0/1565527086 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f04980891e0 con 0x7f04b015c9e0 2024-03-23T11:26:09.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.845+0000 7f04ade00640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f048c06a7a0 0x7f048c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:09.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.846+0000 7f04ade00640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f048c06a7a0 0x7f048c06cc60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f04a0036000 tx=0x7f04a003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:09.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:09.848+0000 7f04a6000640 1 -- 172.21.15.62:0/1565527086 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f04b00b5fc0 con 0x7f04b015c9e0 2024-03-23T11:26:10.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.082+0000 7f04b4e00640 1 -- 172.21.15.62:0/1565527086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04b00b5fc0 con 0x7f04b015c9e0 2024-03-23T11:26:10.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.084+0000 7f04a6000640 1 -- 172.21.15.62:0/1565527086 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f04b00b5fc0 con 0x7f04b015c9e0 2024-03-23T11:26:10.078 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:10.078 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/29171509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":29171509},{"type":"v1","addr":"172.21.15.120:6837","nonce":29171509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:25:33.449465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6618":{"gid":6618,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":25,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:26:10.078 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 50 2024-03-23T11:26:10.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 -- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f048c06a7a0 msgr2=0x7f048c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f048c06a7a0 0x7f048c06cc60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f04a0036000 tx=0x7f04a003c040 comp rx=0 tx=0).stop 2024-03-23T11:26:10.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 -- 172.21.15.62:0/1565527086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04b015c9e0 msgr2=0x7f04b00a9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04b015c9e0 0x7f04b00a9300 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f049800b440 tx=0x7f049800b900 comp rx=0 tx=0).stop 2024-03-23T11:26:10.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 -- 172.21.15.62:0/1565527086 shutdown_connections 2024-03-23T11:26:10.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f048c06a7a0 0x7f048c06cc60 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04b015c9e0 0x7f04b00a9300 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04b015cdc0 0x7f04b00a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 --2- 172.21.15.62:0/1565527086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04b00cf650 0x7f04b00d1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 -- 172.21.15.62:0/1565527086 >> 172.21.15.62:0/1565527086 conn(0x7f04b0071570 msgr2=0x7f04b015eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:10.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 -- 172.21.15.62:0/1565527086 shutdown_connections 2024-03-23T11:26:10.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.086+0000 7f04b4e00640 1 -- 172.21.15.62:0/1565527086 wait complete. 2024-03-23T11:26:10.098 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:26:10.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.327+0000 7fd1c7c00640 1 Processor -- start 2024-03-23T11:26:10.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.327+0000 7fd1c7c00640 1 -- start start 2024-03-23T11:26:10.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.329+0000 7fd1c7c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd1c01613a0 0x7fd1c006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:10.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.329+0000 7fd1c7c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c006cf40 con 0x7fd1c01571e0 2024-03-23T11:26:10.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.329+0000 7fd1c7c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c01575c0 con 0x7fd1c006d910 2024-03-23T11:26:10.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.329+0000 7fd1c7c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1c006c3a0 con 0x7fd1c01613a0 2024-03-23T11:26:10.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.329+0000 7fd1c5200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd1c01571e0 0x7fd1c00ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33216/0 (socket says 172.21.15.62:33216) 2024-03-23T11:26:10.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.329+0000 7fd1c5200640 1 -- 172.21.15.62:0/1577329639 learned_addr learned my addr 172.21.15.62:0/1577329639 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:10.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.330+0000 7fd1bfe00640 1 --2- 172.21.15.62:0/1577329639 >> v2:172.21.15.62:3300/0 conn(0x7fd1c01613a0 0x7fd1c006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:10.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.330+0000 7fd1bfe00640 1 -- 172.21.15.62:0/1577329639 >> v1:172.21.15.120:6789/0 conn(0x7fd1c006d910 legacy=0x7fd1c015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:10.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.330+0000 7fd1bfe00640 1 -- 172.21.15.62:0/1577329639 >> v1:172.21.15.62:6789/0 conn(0x7fd1c01571e0 legacy=0x7fd1c00ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.330+0000 7fd1bfe00640 1 -- 172.21.15.62:0/1577329639 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c00ae040 con 0x7fd1c01613a0 2024-03-23T11:26:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.331+0000 7fd1bfe00640 1 --2- 172.21.15.62:0/1577329639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c01613a0 0x7fd1c006d3d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00097a0 tx=0x7fd1b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f61eea1af6c42ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.331+0000 7fd1bf400640 1 -- 172.21.15.62:0/1577329639 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b002e070 con 0x7fd1c01613a0 2024-03-23T11:26:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.332+0000 7fd1bf400640 1 -- 172.21.15.62:0/1577329639 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b00315a0 con 0x7fd1c01613a0 2024-03-23T11:26:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.332+0000 7fd1bf400640 1 -- 172.21.15.62:0/1577329639 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0031930 con 0x7fd1c01613a0 2024-03-23T11:26:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.332+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1577329639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c01613a0 msgr2=0x7fd1c006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.332+0000 7fd1c7c00640 1 --2- 172.21.15.62:0/1577329639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c01613a0 0x7fd1c006d3d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00097a0 tx=0x7fd1b002b840 comp rx=0 tx=0).stop 2024-03-23T11:26:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.332+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1577329639 shutdown_connections 2024-03-23T11:26:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.332+0000 7fd1c7c00640 1 --2- 172.21.15.62:0/1577329639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c01613a0 0x7fd1c006d3d0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.332+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1577329639 >> 172.21.15.62:0/1577329639 conn(0x7fd1c00b02e0 msgr2=0x7fd1c00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.332+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1577329639 shutdown_connections 2024-03-23T11:26:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.332+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1577329639 wait complete. 2024-03-23T11:26:10.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.334+0000 7fd1c7c00640 1 Processor -- start 2024-03-23T11:26:10.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.334+0000 7fd1c7c00640 1 -- start start 2024-03-23T11:26:10.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.335+0000 7fd1c7c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c006d910 0x7fd1c00cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:10.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.335+0000 7fd1c7c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1c01571e0 0x7fd1c0148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:10.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.335+0000 7fd1c7c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1c01613a0 0x7fd1c0158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:10.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.335+0000 7fd1c7c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c00ce0b0 con 0x7fd1c006d910 2024-03-23T11:26:10.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.335+0000 7fd1c7c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c0157c90 con 0x7fd1c01571e0 2024-03-23T11:26:10.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.335+0000 7fd1c7c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c00cdaf0 con 0x7fd1c01613a0 2024-03-23T11:26:10.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.336+0000 7fd1c5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c006d910 0x7fd1c00cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:10.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.336+0000 7fd1c5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c006d910 0x7fd1c00cd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42842/0 (socket says 172.21.15.62:42842) 2024-03-23T11:26:10.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.336+0000 7fd1c5200640 1 -- 172.21.15.62:0/1230851320 learned_addr learned my addr 172.21.15.62:0/1230851320 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:10.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.336+0000 7fd1bfe00640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1c01571e0 0x7fd1c0148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:10.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.336+0000 7fd1c5c00640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1c01613a0 0x7fd1c0158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:10.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.336+0000 7fd1c5200640 1 -- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1c01613a0 msgr2=0x7fd1c0158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.336+0000 7fd1c5200640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1c01613a0 0x7fd1c0158120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.336+0000 7fd1c5200640 1 -- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1c01571e0 msgr2=0x7fd1c0148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.336+0000 7fd1c5200640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1c01571e0 0x7fd1c0148d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.336+0000 7fd1c5200640 1 -- 172.21.15.62:0/1230851320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b0009400 con 0x7fd1c006d910 2024-03-23T11:26:10.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.337+0000 7fd1c5200640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c006d910 0x7fd1c00cd230 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400c4a0 tx=0x7fd1b400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:10.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.337+0000 7fd1bcc00640 1 -- 172.21.15.62:0/1230851320 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4005ce0 con 0x7fd1c006d910 2024-03-23T11:26:10.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.337+0000 7fd1bcc00640 1 -- 172.21.15.62:0/1230851320 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b4005e40 con 0x7fd1c006d910 2024-03-23T11:26:10.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.337+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1230851320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c00cddd0 con 0x7fd1c006d910 2024-03-23T11:26:10.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.337+0000 7fd1bcc00640 1 -- 172.21.15.62:0/1230851320 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4007270 con 0x7fd1c006d910 2024-03-23T11:26:10.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.337+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1230851320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c00cd770 con 0x7fd1c006d910 2024-03-23T11:26:10.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.338+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1230851320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c0157780 con 0x7fd1c006d910 2024-03-23T11:26:10.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.339+0000 7fd1bcc00640 1 -- 172.21.15.62:0/1230851320 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd1b4007490 con 0x7fd1c006d910 2024-03-23T11:26:10.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.340+0000 7fd1bcc00640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1a406a7a0 0x7fd1a406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:10.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.340+0000 7fd1bcc00640 1 -- 172.21.15.62:0/1230851320 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd1b4088310 con 0x7fd1c006d910 2024-03-23T11:26:10.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.341+0000 7fd1bfe00640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1a406a7a0 0x7fd1a406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:10.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.342+0000 7fd1bfe00640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1a406a7a0 0x7fd1a406cc60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0036000 tx=0x7fd1b003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:10.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.344+0000 7fd1bcc00640 1 -- 172.21.15.62:0/1230851320 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1c0157780 con 0x7fd1c006d910 2024-03-23T11:26:10.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.545+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1230851320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1c0157780 con 0x7fd1c006d910 2024-03-23T11:26:10.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.548+0000 7fd1bcc00640 1 -- 172.21.15.62:0/1230851320 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16329 (secure 0 0 0) 0x7fd1c0157780 con 0x7fd1c006d910 2024-03-23T11:26:10.541 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:10.542 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":77,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:44.311392+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:24:08.688706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-23T11:24:10.072774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1c","mappings":[{"from":6,"to":0}]},{"pgid":"11.3c","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6837/3833782560":"2024-03-24T11:24:34.350690+0000","172.21.15.120:6836/3833782560":"2024-03-24T11:24:34.350690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:26:10.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.550+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1a406a7a0 msgr2=0x7fd1a406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.550+0000 7fd1c7c00640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1a406a7a0 0x7fd1a406cc60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0036000 tx=0x7fd1b003d040 comp rx=0 tx=0).stop 2024-03-23T11:26:10.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.550+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1230851320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c006d910 msgr2=0x7fd1c00cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.550+0000 7fd1c7c00640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c006d910 0x7fd1c00cd230 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400c4a0 tx=0x7fd1b400c960 comp rx=0 tx=0).stop 2024-03-23T11:26:10.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.550+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1230851320 shutdown_connections 2024-03-23T11:26:10.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.550+0000 7fd1c7c00640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1a406a7a0 0x7fd1a406cc60 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.550+0000 7fd1c7c00640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1c01613a0 0x7fd1c0158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.550+0000 7fd1c7c00640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1c01571e0 0x7fd1c0148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.550+0000 7fd1c7c00640 1 --2- 172.21.15.62:0/1230851320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c006d910 0x7fd1c00cd230 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.550+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1230851320 >> 172.21.15.62:0/1230851320 conn(0x7fd1c00b02e0 msgr2=0x7fd1c0160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:10.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.551+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1230851320 shutdown_connections 2024-03-23T11:26:10.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.551+0000 7fd1c7c00640 1 -- 172.21.15.62:0/1230851320 wait complete. 2024-03-23T11:26:10.564 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T11:26:10.564 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:26:10.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.783+0000 7fe7f7200640 1 Processor -- start 2024-03-23T11:26:10.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.784+0000 7fe7f7200640 1 -- start start 2024-03-23T11:26:10.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.785+0000 7fe7f7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7f815fcf0 0x7fe7f80d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:10.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.785+0000 7fe7f7200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe7f80b6dc0 0x7fe7f8160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:10.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.785+0000 7fe7f7200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f8160dc0 con 0x7fe7f8160420 2024-03-23T11:26:10.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.785+0000 7fe7f7200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f80d3540 con 0x7fe7f80b6dc0 2024-03-23T11:26:10.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.785+0000 7fe7f7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f80ce510 con 0x7fe7f815fcf0 2024-03-23T11:26:10.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.785+0000 7fe7f5400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe7f8160420 0x7fe7f80d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33226/0 (socket says 172.21.15.62:33226) 2024-03-23T11:26:10.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.785+0000 7fe7f5400640 1 -- 172.21.15.62:0/2277329306 learned_addr learned my addr 172.21.15.62:0/2277329306 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:10.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.785+0000 7fe7f6800640 1 --2- 172.21.15.62:0/2277329306 >> v2:172.21.15.62:3300/0 conn(0x7fe7f80b6dc0 0x7fe7f8160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:10.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.786+0000 7fe7f5e00640 1 --2- 172.21.15.62:0/2277329306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7f815fcf0 0x7fe7f80d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:10.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.786+0000 7fe7f6800640 1 -- 172.21.15.62:0/2277329306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7f815fcf0 msgr2=0x7fe7f80d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:10.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.786+0000 7fe7f6800640 1 --2- 172.21.15.62:0/2277329306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7f815fcf0 0x7fe7f80d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.786+0000 7fe7f6800640 1 -- 172.21.15.62:0/2277329306 >> v1:172.21.15.62:6789/0 conn(0x7fe7f8160420 legacy=0x7fe7f80d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.786+0000 7fe7f6800640 1 -- 172.21.15.62:0/2277329306 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f80cdc70 con 0x7fe7f80b6dc0 2024-03-23T11:26:10.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.786+0000 7fe7f4a00640 1 -- 172.21.15.62:0/2277329306 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314993776 0 0) 0x7fe7f8160dc0 con 0x7fe7f8160420 2024-03-23T11:26:10.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.786+0000 7fe7f5e00640 1 --2- 172.21.15.62:0/2277329306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7f815fcf0 0x7fe7f80d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:26:10.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.787+0000 7fe7f6800640 1 --2- 172.21.15.62:0/2277329306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7f80b6dc0 0x7fe7f8160880 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00097a0 tx=0x7fe7f002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c8c9ec7760c6ff server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:10.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.787+0000 7fe7f4a00640 1 -- 172.21.15.62:0/2277329306 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00316f0 con 0x7fe7f80b6dc0 2024-03-23T11:26:10.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.787+0000 7fe7f4a00640 1 -- 172.21.15.62:0/2277329306 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f0031850 con 0x7fe7f80b6dc0 2024-03-23T11:26:10.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.787+0000 7fe7f4a00640 1 -- 172.21.15.62:0/2277329306 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0031be0 con 0x7fe7f80b6dc0 2024-03-23T11:26:10.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.788+0000 7fe7f7200640 1 -- 172.21.15.62:0/2277329306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7f80b6dc0 msgr2=0x7fe7f8160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.788+0000 7fe7f7200640 1 --2- 172.21.15.62:0/2277329306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7f80b6dc0 0x7fe7f8160880 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00097a0 tx=0x7fe7f002b790 comp rx=0 tx=0).stop 2024-03-23T11:26:10.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.788+0000 7fe7f7200640 1 -- 172.21.15.62:0/2277329306 shutdown_connections 2024-03-23T11:26:10.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.788+0000 7fe7f7200640 1 --2- 172.21.15.62:0/2277329306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7f80b6dc0 0x7fe7f8160880 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.788+0000 7fe7f7200640 1 --2- 172.21.15.62:0/2277329306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7f815fcf0 0x7fe7f80d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.788+0000 7fe7f7200640 1 -- 172.21.15.62:0/2277329306 >> 172.21.15.62:0/2277329306 conn(0x7fe7f8089f60 msgr2=0x7fe7f80d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:10.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.788+0000 7fe7f7200640 1 -- 172.21.15.62:0/2277329306 shutdown_connections 2024-03-23T11:26:10.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.788+0000 7fe7f7200640 1 -- 172.21.15.62:0/2277329306 wait complete. 2024-03-23T11:26:10.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.790+0000 7fe7f7200640 1 Processor -- start 2024-03-23T11:26:10.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.790+0000 7fe7f7200640 1 -- start start 2024-03-23T11:26:10.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.791+0000 7fe7f7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7f80b6dc0 0x7fe7f81f19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.791+0000 7fe7f7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f815fcf0 0x7fe7f806be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.791+0000 7fe7f7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7f8160420 0x7fe7f806c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.791+0000 7fe7f7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f80c74a0 con 0x7fe7f8160420 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.791+0000 7fe7f7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f8155b20 con 0x7fe7f815fcf0 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.791+0000 7fe7f7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f8155dc0 con 0x7fe7f80b6dc0 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.791+0000 7fe7f6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7f8160420 0x7fe7f806c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.792+0000 7fe7f6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7f8160420 0x7fe7f806c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42864/0 (socket says 172.21.15.62:42864) 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.792+0000 7fe7f6800640 1 -- 172.21.15.62:0/3634965840 learned_addr learned my addr 172.21.15.62:0/3634965840 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.792+0000 7fe7f5e00640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7f80b6dc0 0x7fe7f81f19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.792+0000 7fe7f5400640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f815fcf0 0x7fe7f806be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.792+0000 7fe7f6800640 1 -- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7f80b6dc0 msgr2=0x7fe7f81f19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.792+0000 7fe7f6800640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7f80b6dc0 0x7fe7f81f19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.792+0000 7fe7f6800640 1 -- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f815fcf0 msgr2=0x7fe7f806be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:10.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.792+0000 7fe7f6800640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f815fcf0 0x7fe7f806be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:10.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.792+0000 7fe7f6800640 1 -- 172.21.15.62:0/3634965840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f0009400 con 0x7fe7f8160420 2024-03-23T11:26:10.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.792+0000 7fe7f6800640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7f8160420 0x7fe7f806c390 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00097a0 tx=0x7fe7f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.793+0000 7fe7ee000640 1 -- 172.21.15.62:0/3634965840 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0034680 con 0x7fe7f8160420 2024-03-23T11:26:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.793+0000 7fe7ee000640 1 -- 172.21.15.62:0/3634965840 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f00347e0 con 0x7fe7f8160420 2024-03-23T11:26:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.793+0000 7fe7f7200640 1 -- 172.21.15.62:0/3634965840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f80c6c00 con 0x7fe7f8160420 2024-03-23T11:26:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.793+0000 7fe7ee000640 1 -- 172.21.15.62:0/3634965840 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0034b70 con 0x7fe7f8160420 2024-03-23T11:26:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.793+0000 7fe7f7200640 1 -- 172.21.15.62:0/3634965840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f80c71c0 con 0x7fe7f8160420 2024-03-23T11:26:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.794+0000 7fe7f7200640 1 -- 172.21.15.62:0/3634965840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f8155f80 con 0x7fe7f8160420 2024-03-23T11:26:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.795+0000 7fe7ee000640 1 -- 172.21.15.62:0/3634965840 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe7f003b040 con 0x7fe7f8160420 2024-03-23T11:26:10.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.796+0000 7fe7ee000640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7d406a7a0 0x7fe7d406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:10.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.796+0000 7fe7ee000640 1 -- 172.21.15.62:0/3634965840 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe7f00aa380 con 0x7fe7f8160420 2024-03-23T11:26:10.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.797+0000 7fe7f5e00640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7d406a7a0 0x7fe7d406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.798+0000 7fe7f5e00640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7d406a7a0 0x7fe7d406cc60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8006fd0 tx=0x7fe7e8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:10.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:10.801+0000 7fe7ee000640 1 -- 172.21.15.62:0/3634965840 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe7f0075820 con 0x7fe7f8160420 2024-03-23T11:26:11.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.042+0000 7fe7f7200640 1 -- 172.21.15.62:0/3634965840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7f8066bb0 con 0x7fe7f8160420 2024-03-23T11:26:11.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.044+0000 7fe7ee000640 1 -- 172.21.15.62:0/3634965840 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fe7f0078e40 con 0x7fe7f8160420 2024-03-23T11:26:11.037 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:11.037 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/29171509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":29171509},{"type":"v1","addr":"172.21.15.120:6837","nonce":29171509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:25:33.449465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6618":{"gid":6618,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":25,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:26:11.038 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 50 2024-03-23T11:26:11.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.046+0000 7fe7f7200640 1 -- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7d406a7a0 msgr2=0x7fe7d406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:11.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.046+0000 7fe7f7200640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7d406a7a0 0x7fe7d406cc60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8006fd0 tx=0x7fe7e8008040 comp rx=0 tx=0).stop 2024-03-23T11:26:11.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.046+0000 7fe7f7200640 1 -- 172.21.15.62:0/3634965840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7f8160420 msgr2=0x7fe7f806c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:11.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.046+0000 7fe7f7200640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7f8160420 0x7fe7f806c390 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00097a0 tx=0x7fe7f00046c0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.047+0000 7fe7f7200640 1 -- 172.21.15.62:0/3634965840 shutdown_connections 2024-03-23T11:26:11.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.047+0000 7fe7f7200640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe7d406a7a0 0x7fe7d406cc60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.047+0000 7fe7f7200640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7f8160420 0x7fe7f806c390 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.047+0000 7fe7f7200640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f815fcf0 0x7fe7f806be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.047+0000 7fe7f7200640 1 --2- 172.21.15.62:0/3634965840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7f80b6dc0 0x7fe7f81f19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.047+0000 7fe7f7200640 1 -- 172.21.15.62:0/3634965840 >> 172.21.15.62:0/3634965840 conn(0x7fe7f8089f60 msgr2=0x7fe7f8160880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:11.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.047+0000 7fe7f7200640 1 -- 172.21.15.62:0/3634965840 shutdown_connections 2024-03-23T11:26:11.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.047+0000 7fe7f7200640 1 -- 172.21.15.62:0/3634965840 wait complete. 2024-03-23T11:26:11.059 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:26:11.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.284+0000 7fa4be200640 1 Processor -- start 2024-03-23T11:26:11.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.285+0000 7fa4be200640 1 -- start start 2024-03-23T11:26:11.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.286+0000 7fa4be200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa4b80cd7c0 0x7fa4b80b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.286+0000 7fa4be200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa4b806c300 0x7fa4b80cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.286+0000 7fa4be200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b80cad20 con 0x7fa4b806dca0 2024-03-23T11:26:11.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.286+0000 7fa4be200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b806e130 con 0x7fa4b806c300 2024-03-23T11:26:11.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.286+0000 7fa4be200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b815a8a0 con 0x7fa4b80cd7c0 2024-03-23T11:26:11.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.287+0000 7fa4b7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa4b806c300 0x7fa4b80cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.287+0000 7fa4aea00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa4b806dca0 0x7fa4b80cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33242/0 (socket says 172.21.15.62:33242) 2024-03-23T11:26:11.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.287+0000 7fa4aea00640 1 -- 172.21.15.62:0/2583017935 learned_addr learned my addr 172.21.15.62:0/2583017935 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:11.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.287+0000 7fa4b7400640 1 --2- 172.21.15.62:0/2583017935 >> v2:172.21.15.120:3300/0 conn(0x7fa4b80cd7c0 0x7fa4b80b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.287+0000 7fa4b7e00640 1 -- 172.21.15.62:0/2583017935 >> v2:172.21.15.120:3300/0 conn(0x7fa4b80cd7c0 msgr2=0x7fa4b80b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:11.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.287+0000 7fa4b7e00640 1 --2- 172.21.15.62:0/2583017935 >> v2:172.21.15.120:3300/0 conn(0x7fa4b80cd7c0 0x7fa4b80b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.287+0000 7fa4b7e00640 1 -- 172.21.15.62:0/2583017935 >> v1:172.21.15.62:6789/0 conn(0x7fa4b806dca0 legacy=0x7fa4b80cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:11.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.287+0000 7fa4b7e00640 1 -- 172.21.15.62:0/2583017935 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b815ab80 con 0x7fa4b806c300 2024-03-23T11:26:11.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.288+0000 7fa4b7e00640 1 --2- 172.21.15.62:0/2583017935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4b806c300 0x7fa4b80cb490 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa4b806ac20 tx=0x7fa4b002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20696b5bd0697f7e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:11.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.289+0000 7fa4b6a00640 1 -- 172.21.15.62:0/2583017935 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b002cbe0 con 0x7fa4b806c300 2024-03-23T11:26:11.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.289+0000 7fa4b6a00640 1 -- 172.21.15.62:0/2583017935 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b002cd40 con 0x7fa4b806c300 2024-03-23T11:26:11.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.289+0000 7fa4b6a00640 1 -- 172.21.15.62:0/2583017935 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0034660 con 0x7fa4b806c300 2024-03-23T11:26:11.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.289+0000 7fa4be200640 1 -- 172.21.15.62:0/2583017935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4b806c300 msgr2=0x7fa4b80cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:11.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.289+0000 7fa4be200640 1 --2- 172.21.15.62:0/2583017935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4b806c300 0x7fa4b80cb490 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa4b806ac20 tx=0x7fa4b002be30 comp rx=0 tx=0).stop 2024-03-23T11:26:11.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.290+0000 7fa4be200640 1 -- 172.21.15.62:0/2583017935 shutdown_connections 2024-03-23T11:26:11.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.290+0000 7fa4be200640 1 --2- 172.21.15.62:0/2583017935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4b806c300 0x7fa4b80cb490 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.290+0000 7fa4be200640 1 --2- 172.21.15.62:0/2583017935 >> v2:172.21.15.120:3300/0 conn(0x7fa4b80cd7c0 0x7fa4b80b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.290+0000 7fa4be200640 1 -- 172.21.15.62:0/2583017935 >> 172.21.15.62:0/2583017935 conn(0x7fa4b80b02e0 msgr2=0x7fa4b80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:11.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.290+0000 7fa4be200640 1 -- 172.21.15.62:0/2583017935 shutdown_connections 2024-03-23T11:26:11.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.290+0000 7fa4be200640 1 -- 172.21.15.62:0/2583017935 wait complete. 2024-03-23T11:26:11.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.292+0000 7fa4be200640 1 Processor -- start 2024-03-23T11:26:11.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.292+0000 7fa4be200640 1 -- start start 2024-03-23T11:26:11.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.293+0000 7fa4be200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4b806c300 0x7fa4b8154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.293+0000 7fa4be200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4b80cd7c0 0x7fa4b8159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.293+0000 7fa4be200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4b8158d50 0x7fa4b806d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.293+0000 7fa4be200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b8159200 con 0x7fa4b806c300 2024-03-23T11:26:11.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.293+0000 7fa4be200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b8158870 con 0x7fa4b80cd7c0 2024-03-23T11:26:11.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.293+0000 7fa4be200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b8147f80 con 0x7fa4b8158d50 2024-03-23T11:26:11.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.293+0000 7fa4b7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4b806c300 0x7fa4b8154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.293+0000 7fa4b7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4b806c300 0x7fa4b8154be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42874/0 (socket says 172.21.15.62:42874) 2024-03-23T11:26:11.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.293+0000 7fa4b7400640 1 -- 172.21.15.62:0/2337243271 learned_addr learned my addr 172.21.15.62:0/2337243271 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:11.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4b7e00640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4b8158d50 0x7fa4b806d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4b7400640 1 -- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4b8158d50 msgr2=0x7fa4b806d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:11.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4aea00640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4b80cd7c0 0x7fa4b8159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4b7400640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4b8158d50 0x7fa4b806d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4b7400640 1 -- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4b80cd7c0 msgr2=0x7fa4b8159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:11.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4b7400640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4b80cd7c0 0x7fa4b8159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4b7400640 1 -- 172.21.15.62:0/2337243271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0009a70 con 0x7fa4b806c300 2024-03-23T11:26:11.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4b7400640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4b806c300 0x7fa4b8154be0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800d5f0 tx=0x7fa4a800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:11.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4afe00640 1 -- 172.21.15.62:0/2337243271 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a80125f0 con 0x7fa4b806c300 2024-03-23T11:26:11.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4afe00640 1 -- 172.21.15.62:0/2337243271 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a8012750 con 0x7fa4b806c300 2024-03-23T11:26:11.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4be200640 1 -- 172.21.15.62:0/2337243271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b81551c0 con 0x7fa4b806c300 2024-03-23T11:26:11.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.294+0000 7fa4afe00640 1 -- 172.21.15.62:0/2337243271 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8012b00 con 0x7fa4b806c300 2024-03-23T11:26:11.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.295+0000 7fa4be200640 1 -- 172.21.15.62:0/2337243271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b8155970 con 0x7fa4b806c300 2024-03-23T11:26:11.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.296+0000 7fa4be200640 1 -- 172.21.15.62:0/2337243271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b8066c90 con 0x7fa4b806c300 2024-03-23T11:26:11.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.297+0000 7fa4afe00640 1 -- 172.21.15.62:0/2337243271 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa4a8012c60 con 0x7fa4b806c300 2024-03-23T11:26:11.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.297+0000 7fa4afe00640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa49406a7a0 0x7fa49406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.298+0000 7fa4afe00640 1 -- 172.21.15.62:0/2337243271 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa4a8089350 con 0x7fa4b806c300 2024-03-23T11:26:11.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.298+0000 7fa4aea00640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa49406a7a0 0x7fa49406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.299+0000 7fa4aea00640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa49406a7a0 0x7fa49406cc60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa4b815a5e0 tx=0x7fa4a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:11.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.302+0000 7fa4afe00640 1 -- 172.21.15.62:0/2337243271 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4a8054640 con 0x7fa4b806c300 2024-03-23T11:26:11.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.542+0000 7fa4be200640 1 -- 172.21.15.62:0/2337243271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4b806c970 con 0x7fa4b806c300 2024-03-23T11:26:11.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.543+0000 7fa4afe00640 1 -- 172.21.15.62:0/2337243271 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fa4a8057c60 con 0x7fa4b806c300 2024-03-23T11:26:11.537 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:11.537 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/29171509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":29171509},{"type":"v1","addr":"172.21.15.120:6837","nonce":29171509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:24:11.044296+0000","modified":"2024-03-23T11:25:33.449465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6618":{"gid":6618,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":25,"addr":"172.21.15.120:6835/1201988215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1201988215},{"type":"v1","addr":"172.21.15.120:6835","nonce":1201988215}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T11:26:11.537 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 50 2024-03-23T11:26:11.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.546+0000 7fa4be200640 1 -- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa49406a7a0 msgr2=0x7fa49406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:11.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.546+0000 7fa4be200640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa49406a7a0 0x7fa49406cc60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa4b815a5e0 tx=0x7fa4a0009040 comp rx=0 tx=0).stop 2024-03-23T11:26:11.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.546+0000 7fa4be200640 1 -- 172.21.15.62:0/2337243271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4b806c300 msgr2=0x7fa4b8154be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:11.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.546+0000 7fa4be200640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4b806c300 0x7fa4b8154be0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800d5f0 tx=0x7fa4a800dab0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.546+0000 7fa4be200640 1 -- 172.21.15.62:0/2337243271 shutdown_connections 2024-03-23T11:26:11.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.546+0000 7fa4be200640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa49406a7a0 0x7fa49406cc60 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.546+0000 7fa4be200640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa4b8158d50 0x7fa4b806d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.546+0000 7fa4be200640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4b80cd7c0 0x7fa4b8159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.546+0000 7fa4be200640 1 --2- 172.21.15.62:0/2337243271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4b806c300 0x7fa4b8154be0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.546+0000 7fa4be200640 1 -- 172.21.15.62:0/2337243271 >> 172.21.15.62:0/2337243271 conn(0x7fa4b80b02e0 msgr2=0x7fa4b814e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.546+0000 7fa4be200640 1 -- 172.21.15.62:0/2337243271 shutdown_connections 2024-03-23T11:26:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.547+0000 7fa4be200640 1 -- 172.21.15.62:0/2337243271 wait complete. 2024-03-23T11:26:11.559 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:26:11.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.784+0000 7f5495000640 1 Processor -- start 2024-03-23T11:26:11.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.785+0000 7f5495000640 1 -- start start 2024-03-23T11:26:11.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.786+0000 7f5495000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f549015cdc0 0x7f54900d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.786+0000 7f5495000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f54900cf650 0x7f54900cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.786+0000 7f5495000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549006bfc0 con 0x7f549015c9e0 2024-03-23T11:26:11.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.786+0000 7f5495000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54900d0000 con 0x7f54900cf650 2024-03-23T11:26:11.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.786+0000 7f5495000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54900ce140 con 0x7f549015cdc0 2024-03-23T11:26:11.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.787+0000 7f548e800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f54900cf650 0x7f54900cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.787+0000 7f548e800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f54900cf650 0x7f54900cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42890/0 (socket says 172.21.15.62:42890) 2024-03-23T11:26:11.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.787+0000 7f548e800640 1 -- 172.21.15.62:0/1563695189 learned_addr learned my addr 172.21.15.62:0/1563695189 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:11.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.787+0000 7f5485400640 1 --2- 172.21.15.62:0/1563695189 >> v2:172.21.15.120:3300/0 conn(0x7f549015cdc0 0x7f54900d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.787+0000 7f548e800640 1 -- 172.21.15.62:0/1563695189 >> v2:172.21.15.120:3300/0 conn(0x7f549015cdc0 msgr2=0x7f54900d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:11.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.787+0000 7f548e800640 1 --2- 172.21.15.62:0/1563695189 >> v2:172.21.15.120:3300/0 conn(0x7f549015cdc0 0x7f54900d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.787+0000 7f548e800640 1 -- 172.21.15.62:0/1563695189 >> v1:172.21.15.120:6789/0 conn(0x7f549015c9e0 legacy=0x7f54901607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:11.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.787+0000 7f548e800640 1 -- 172.21.15.62:0/1563695189 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54900b5260 con 0x7f54900cf650 2024-03-23T11:26:11.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.788+0000 7f548e800640 1 --2- 172.21.15.62:0/1563695189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54900cf650 0x7f54900cfac0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f54900b89a0 tx=0x7f548802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727ba2ac5bfd090a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:11.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.789+0000 7f548d400640 1 -- 172.21.15.62:0/1563695189 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548802ca80 con 0x7f54900cf650 2024-03-23T11:26:11.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.789+0000 7f548d400640 1 -- 172.21.15.62:0/1563695189 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548802cbe0 con 0x7f54900cf650 2024-03-23T11:26:11.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.789+0000 7f548d400640 1 -- 172.21.15.62:0/1563695189 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54880343f0 con 0x7f54900cf650 2024-03-23T11:26:11.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.789+0000 7f5495000640 1 -- 172.21.15.62:0/1563695189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54900cf650 msgr2=0x7f54900cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:11.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.789+0000 7f5495000640 1 --2- 172.21.15.62:0/1563695189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54900cf650 0x7f54900cfac0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f54900b89a0 tx=0x7f548802bee0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.789+0000 7f5495000640 1 -- 172.21.15.62:0/1563695189 shutdown_connections 2024-03-23T11:26:11.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.789+0000 7f5495000640 1 --2- 172.21.15.62:0/1563695189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54900cf650 0x7f54900cfac0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.789+0000 7f5495000640 1 --2- 172.21.15.62:0/1563695189 >> v2:172.21.15.120:3300/0 conn(0x7f549015cdc0 0x7f54900d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.789+0000 7f5495000640 1 -- 172.21.15.62:0/1563695189 >> 172.21.15.62:0/1563695189 conn(0x7f5490071570 msgr2=0x7f5490071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:11.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.790+0000 7f5495000640 1 -- 172.21.15.62:0/1563695189 shutdown_connections 2024-03-23T11:26:11.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.790+0000 7f5495000640 1 -- 172.21.15.62:0/1563695189 wait complete. 2024-03-23T11:26:11.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.791+0000 7f5495000640 1 Processor -- start 2024-03-23T11:26:11.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.791+0000 7f5495000640 1 -- start start 2024-03-23T11:26:11.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.792+0000 7f5495000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54900cf650 0x7f5490151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.792+0000 7f5495000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f549015c9e0 0x7f549015e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.792+0000 7f5495000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f549015cdc0 0x7f549015ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.792+0000 7f5495000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5490152760 con 0x7f549015cdc0 2024-03-23T11:26:11.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.792+0000 7f5495000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54901523a0 con 0x7f54900cf650 2024-03-23T11:26:11.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.792+0000 7f5495000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5490155b90 con 0x7f549015c9e0 2024-03-23T11:26:11.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.793+0000 7f548e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f549015cdc0 0x7f549015ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.793+0000 7f548e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f549015cdc0 0x7f549015ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42904/0 (socket says 172.21.15.62:42904) 2024-03-23T11:26:11.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.793+0000 7f548e800640 1 -- 172.21.15.62:0/3790498795 learned_addr learned my addr 172.21.15.62:0/3790498795 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:11.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.793+0000 7f548de00640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54900cf650 0x7f5490151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.793+0000 7f5485400640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f549015c9e0 0x7f549015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.793+0000 7f548e800640 1 -- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f549015c9e0 msgr2=0x7f549015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:11.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.793+0000 7f548e800640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f549015c9e0 0x7f549015e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.793+0000 7f548e800640 1 -- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54900cf650 msgr2=0x7f5490151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:11.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.793+0000 7f548e800640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54900cf650 0x7f5490151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:11.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.793+0000 7f548e800640 1 -- 172.21.15.62:0/3790498795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5488009b30 con 0x7f549015cdc0 2024-03-23T11:26:11.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.793+0000 7f5485400640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f549015c9e0 0x7f549015e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:11.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.794+0000 7f548e800640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f549015cdc0 0x7f549015ead0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f5488036040 tx=0x7f5488004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:11.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.794+0000 7f5486a00640 1 -- 172.21.15.62:0/3790498795 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5488009510 con 0x7f549015cdc0 2024-03-23T11:26:11.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.794+0000 7f5486a00640 1 -- 172.21.15.62:0/3790498795 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548803e450 con 0x7f549015cdc0 2024-03-23T11:26:11.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.794+0000 7f5495000640 1 -- 172.21.15.62:0/3790498795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54901558b0 con 0x7f549015cdc0 2024-03-23T11:26:11.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.794+0000 7f5486a00640 1 -- 172.21.15.62:0/3790498795 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548803e7e0 con 0x7f549015cdc0 2024-03-23T11:26:11.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.794+0000 7f5495000640 1 -- 172.21.15.62:0/3790498795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5490152ce0 con 0x7f549015cdc0 2024-03-23T11:26:11.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.795+0000 7f5495000640 1 -- 172.21.15.62:0/3790498795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5490066c90 con 0x7f549015cdc0 2024-03-23T11:26:11.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.796+0000 7f5486a00640 1 -- 172.21.15.62:0/3790498795 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f548802cd50 con 0x7f549015cdc0 2024-03-23T11:26:11.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.798+0000 7f5486a00640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f546c06a7a0 0x7f546c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:11.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.798+0000 7f5486a00640 1 -- 172.21.15.62:0/3790498795 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f54880ab9e0 con 0x7f549015cdc0 2024-03-23T11:26:11.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.798+0000 7f548de00640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f546c06a7a0 0x7f546c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:11.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.799+0000 7f548de00640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f546c06a7a0 0x7f546c06cc60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5480004ea0 tx=0x7f5480009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:11.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:11.802+0000 7f5486a00640 1 -- 172.21.15.62:0/3790498795 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5488076cd0 con 0x7f549015cdc0 2024-03-23T11:26:12.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.012+0000 7f5495000640 1 -- 172.21.15.62:0/3790498795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5490071b60 con 0x7f549015cdc0 2024-03-23T11:26:12.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.015+0000 7f5486a00640 1 -- 172.21.15.62:0/3790498795 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16329 (secure 0 0 0) 0x7f5488043090 con 0x7f549015cdc0 2024-03-23T11:26:12.009 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:12.009 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":77,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:24:44.311392+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:24:08.688706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-23T11:24:10.072774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.1c","mappings":[{"from":6,"to":0}]},{"pgid":"11.3c","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6837/3833782560":"2024-03-24T11:24:34.350690+0000","172.21.15.120:6836/3833782560":"2024-03-24T11:24:34.350690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:26:12.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 -- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f546c06a7a0 msgr2=0x7f546c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:12.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f546c06a7a0 0x7f546c06cc60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5480004ea0 tx=0x7f5480009040 comp rx=0 tx=0).stop 2024-03-23T11:26:12.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 -- 172.21.15.62:0/3790498795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f549015cdc0 msgr2=0x7f549015ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:12.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f549015cdc0 0x7f549015ead0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f5488036040 tx=0x7f5488004a40 comp rx=0 tx=0).stop 2024-03-23T11:26:12.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 -- 172.21.15.62:0/3790498795 shutdown_connections 2024-03-23T11:26:12.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f546c06a7a0 0x7f546c06cc60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f549015cdc0 0x7f549015ead0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f549015c9e0 0x7f549015e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 --2- 172.21.15.62:0/3790498795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54900cf650 0x7f5490151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 -- 172.21.15.62:0/3790498795 >> 172.21.15.62:0/3790498795 conn(0x7f5490071570 msgr2=0x7f54900b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:12.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 -- 172.21.15.62:0/3790498795 shutdown_connections 2024-03-23T11:26:12.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.018+0000 7f5495000640 1 -- 172.21.15.62:0/3790498795 wait complete. 2024-03-23T11:26:12.030 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:26:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.259+0000 7f9fd9c00640 1 Processor -- start 2024-03-23T11:26:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.259+0000 7f9fd9c00640 1 -- start start 2024-03-23T11:26:12.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.261+0000 7f9fd9c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9fd41593e0 0x7f9fd415d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.261+0000 7f9fd9c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9fd406a290 0x7f9fd40cc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.261+0000 7f9fd9c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9fd40b2b30 0x7f9fd40b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.261+0000 7f9fd9c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd406a020 con 0x7f9fd40b2b30 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.261+0000 7f9fd9c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd40b34e0 con 0x7f9fd406a290 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.261+0000 7f9fd9c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd40b14c0 con 0x7f9fd41593e0 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.261+0000 7f9fd9200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9fd40b2b30 0x7f9fd40b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.262+0000 7f9fd9200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9fd40b2b30 0x7f9fd40b2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42906/0 (socket says 172.21.15.62:42906) 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.262+0000 7f9fd9200640 1 -- 172.21.15.62:0/2591242573 learned_addr learned my addr 172.21.15.62:0/2591242573 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.262+0000 7f9fd3e00640 1 --2- 172.21.15.62:0/2591242573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9fd41593e0 0x7f9fd415d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.262+0000 7f9fcbe00640 1 --2- 172.21.15.62:0/2591242573 >> v2:172.21.15.120:3300/0 conn(0x7f9fd406a290 0x7f9fd40cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.262+0000 7f9fd9200640 1 -- 172.21.15.62:0/2591242573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9fd41593e0 msgr2=0x7f9fd415d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.262+0000 7f9fd9200640 1 --2- 172.21.15.62:0/2591242573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9fd41593e0 0x7f9fd415d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.262+0000 7f9fd9200640 1 -- 172.21.15.62:0/2591242573 >> v2:172.21.15.120:3300/0 conn(0x7f9fd406a290 msgr2=0x7f9fd40cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:12.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.262+0000 7f9fd9200640 1 --2- 172.21.15.62:0/2591242573 >> v2:172.21.15.120:3300/0 conn(0x7f9fd406a290 0x7f9fd40cc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.262+0000 7f9fd9200640 1 -- 172.21.15.62:0/2591242573 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd406a6f0 con 0x7f9fd40b2b30 2024-03-23T11:26:12.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.263+0000 7f9fd9200640 1 --2- 172.21.15.62:0/2591242573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fd40b2b30 0x7f9fd40b2fa0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4071530 tx=0x7f9fcc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f06137db3bc94e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:12.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.264+0000 7f9fd3400640 1 -- 172.21.15.62:0/2591242573 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc02ca70 con 0x7f9fd40b2b30 2024-03-23T11:26:12.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.264+0000 7f9fd3400640 1 -- 172.21.15.62:0/2591242573 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fcc02cbd0 con 0x7f9fd40b2b30 2024-03-23T11:26:12.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.264+0000 7f9fd3400640 1 -- 172.21.15.62:0/2591242573 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc0343f0 con 0x7f9fd40b2b30 2024-03-23T11:26:12.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.265+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2591242573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fd40b2b30 msgr2=0x7f9fd40b2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:12.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.265+0000 7f9fd9c00640 1 --2- 172.21.15.62:0/2591242573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fd40b2b30 0x7f9fd40b2fa0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4071530 tx=0x7f9fcc02bed0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.265+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2591242573 shutdown_connections 2024-03-23T11:26:12.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.265+0000 7f9fd9c00640 1 --2- 172.21.15.62:0/2591242573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fd40b2b30 0x7f9fd40b2fa0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.265+0000 7f9fd9c00640 1 --2- 172.21.15.62:0/2591242573 >> v2:172.21.15.120:3300/0 conn(0x7f9fd406a290 0x7f9fd40cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.265+0000 7f9fd9c00640 1 --2- 172.21.15.62:0/2591242573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9fd41593e0 0x7f9fd415d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.265+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2591242573 >> 172.21.15.62:0/2591242573 conn(0x7f9fd40b0090 msgr2=0x7f9fd40b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:12.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.265+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2591242573 shutdown_connections 2024-03-23T11:26:12.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.265+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2591242573 wait complete. 2024-03-23T11:26:12.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.267+0000 7f9fd9c00640 1 Processor -- start 2024-03-23T11:26:12.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.267+0000 7f9fd9c00640 1 -- start start 2024-03-23T11:26:12.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.268+0000 7f9fd9c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fd406a290 0x7f9fd4152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.268+0000 7f9fd9c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fd40b2b30 0x7f9fd40aad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.268+0000 7f9fd9c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9fd41593e0 0x7f9fd4151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.268+0000 7f9fd9c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd40ab2d0 con 0x7f9fd40b2b30 2024-03-23T11:26:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.268+0000 7f9fd9c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd40a8600 con 0x7f9fd406a290 2024-03-23T11:26:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.268+0000 7f9fd9c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd40a79f0 con 0x7f9fd41593e0 2024-03-23T11:26:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.269+0000 7f9fcbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fd40b2b30 0x7f9fd40aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.269+0000 7f9fcbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fd40b2b30 0x7f9fd40aad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42920/0 (socket says 172.21.15.62:42920) 2024-03-23T11:26:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.269+0000 7f9fcbe00640 1 -- 172.21.15.62:0/2078580893 learned_addr learned my addr 172.21.15.62:0/2078580893 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.269+0000 7f9fd3e00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fd406a290 0x7f9fd4152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.269+0000 7f9fd9200640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9fd41593e0 0x7f9fd4151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.269+0000 7f9fcbe00640 1 -- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9fd41593e0 msgr2=0x7f9fd4151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:12.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.269+0000 7f9fcbe00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9fd41593e0 0x7f9fd4151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.269+0000 7f9fcbe00640 1 -- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fd406a290 msgr2=0x7f9fd4152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:12.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.269+0000 7f9fcbe00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fd406a290 0x7f9fd4152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.269+0000 7f9fcbe00640 1 -- 172.21.15.62:0/2078580893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc009b30 con 0x7f9fd40b2b30 2024-03-23T11:26:12.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.269+0000 7f9fd9200640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9fd41593e0 0x7f9fd4151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:26:12.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.270+0000 7f9fcbe00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fd40b2b30 0x7f9fd40aad90 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00d830 tx=0x7f9fbc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:12.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.270+0000 7f9fd0c00640 1 -- 172.21.15.62:0/2078580893 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc00b6c0 con 0x7f9fd40b2b30 2024-03-23T11:26:12.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.270+0000 7f9fd0c00640 1 -- 172.21.15.62:0/2078580893 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc00b820 con 0x7f9fd40b2b30 2024-03-23T11:26:12.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.270+0000 7f9fd0c00640 1 -- 172.21.15.62:0/2078580893 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc00bbb0 con 0x7f9fd40b2b30 2024-03-23T11:26:12.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.270+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2078580893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd40a7cd0 con 0x7f9fd40b2b30 2024-03-23T11:26:12.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.270+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2078580893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd40a8290 con 0x7f9fd40b2b30 2024-03-23T11:26:12.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.271+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2078580893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd41597c0 con 0x7f9fd40b2b30 2024-03-23T11:26:12.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.273+0000 7f9fd0c00640 1 -- 172.21.15.62:0/2078580893 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9fbc01e460 con 0x7f9fd40b2b30 2024-03-23T11:26:12.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.274+0000 7f9fd0c00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9fb006a7a0 0x7f9fb006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.274+0000 7f9fd0c00640 1 -- 172.21.15.62:0/2078580893 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9fbc0895e0 con 0x7f9fd40b2b30 2024-03-23T11:26:12.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.274+0000 7f9fd3e00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9fb006a7a0 0x7f9fb006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.275+0000 7f9fd3e00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9fb006a7a0 0x7f9fb006cc60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4006fd0 tx=0x7f9fc4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:12.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.278+0000 7f9fd0c00640 1 -- 172.21.15.62:0/2078580893 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9fbc054970 con 0x7f9fd40b2b30 2024-03-23T11:26:12.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.517+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2078580893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9fd4066bb0 con 0x7f9fd40b2b30 2024-03-23T11:26:12.512 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:26:12.519+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:26:12.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.535+0000 7f9fd0c00640 1 -- 172.21.15.62:0/2078580893 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v51) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9fbc057f90 con 0x7f9fd40b2b30 2024-03-23T11:26:12.529 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:26:12.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9fb006a7a0 msgr2=0x7f9fb006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:12.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9fb006a7a0 0x7f9fb006cc60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4006fd0 tx=0x7f9fc4008040 comp rx=0 tx=0).stop 2024-03-23T11:26:12.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2078580893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fd40b2b30 msgr2=0x7f9fd40aad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:12.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fd40b2b30 0x7f9fd40aad90 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00d830 tx=0x7f9fbc00dcf0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2078580893 shutdown_connections 2024-03-23T11:26:12.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9fb006a7a0 0x7f9fb006cc60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fd40b2b30 0x7f9fd40aad90 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9fd41593e0 0x7f9fd4151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 --2- 172.21.15.62:0/2078580893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fd406a290 0x7f9fd4152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2078580893 >> 172.21.15.62:0/2078580893 conn(0x7f9fd40b0090 msgr2=0x7f9fd4155aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:12.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2078580893 shutdown_connections 2024-03-23T11:26:12.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.538+0000 7f9fd9c00640 1 -- 172.21.15.62:0/2078580893 wait complete. 2024-03-23T11:26:12.551 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T11:26:12.558 INFO:tasks.ceph.mds.b.smithi120.stderr: -4382> 2024-03-23T11:24:40.514+0000 7f4efae00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:26:12.643 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-23T11:26:12.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.765+0000 7f174e800640 1 Processor -- start 2024-03-23T11:26:12.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.765+0000 7f174e800640 1 -- start start 2024-03-23T11:26:12.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.767+0000 7f174e800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1748069c60 0x7f174806a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.767+0000 7f174e800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17480caff0 0x7f17480cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.767+0000 7f174e800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1748143450 0x7f17481438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.767+0000 7f174e800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17480ce1a0 con 0x7f1748143450 2024-03-23T11:26:12.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.767+0000 7f174e800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17480cbcb0 con 0x7f1748069c60 2024-03-23T11:26:12.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.767+0000 7f174e800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174815ab40 con 0x7f17480caff0 2024-03-23T11:26:12.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.767+0000 7f174de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1748143450 0x7f17481438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.767+0000 7f174de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1748143450 0x7f17481438e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42934/0 (socket says 172.21.15.62:42934) 2024-03-23T11:26:12.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.767+0000 7f174de00640 1 -- 172.21.15.62:0/2343954614 learned_addr learned my addr 172.21.15.62:0/2343954614 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:12.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.767+0000 7f174d400640 1 --2- 172.21.15.62:0/2343954614 >> v2:172.21.15.120:3300/0 conn(0x7f1748069c60 0x7f174806a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.768+0000 7f174ca00640 1 --2- 172.21.15.62:0/2343954614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17480caff0 0x7f17480cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.768+0000 7f174de00640 1 -- 172.21.15.62:0/2343954614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17480caff0 msgr2=0x7f17480cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:12.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.768+0000 7f174de00640 1 --2- 172.21.15.62:0/2343954614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17480caff0 0x7f17480cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.768+0000 7f174de00640 1 -- 172.21.15.62:0/2343954614 >> v2:172.21.15.120:3300/0 conn(0x7f1748069c60 msgr2=0x7f174806a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:12.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.768+0000 7f174de00640 1 --2- 172.21.15.62:0/2343954614 >> v2:172.21.15.120:3300/0 conn(0x7f1748069c60 0x7f174806a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.768+0000 7f174de00640 1 -- 172.21.15.62:0/2343954614 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174815bb10 con 0x7f1748143450 2024-03-23T11:26:12.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.769+0000 7f174de00640 1 --2- 172.21.15.62:0/2343954614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1748143450 0x7f17481438e0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f174806ad90 tx=0x7f174002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c9bd6fc045b1f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:12.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.769+0000 7f1747e00640 1 -- 172.21.15.62:0/2343954614 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174002ca70 con 0x7f1748143450 2024-03-23T11:26:12.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.769+0000 7f1747e00640 1 -- 172.21.15.62:0/2343954614 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174002cbd0 con 0x7f1748143450 2024-03-23T11:26:12.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.769+0000 7f1747e00640 1 -- 172.21.15.62:0/2343954614 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17400343f0 con 0x7f1748143450 2024-03-23T11:26:12.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.770+0000 7f174e800640 1 -- 172.21.15.62:0/2343954614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1748143450 msgr2=0x7f17481438e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:12.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.770+0000 7f174e800640 1 --2- 172.21.15.62:0/2343954614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1748143450 0x7f17481438e0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f174806ad90 tx=0x7f174002bed0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.770+0000 7f174e800640 1 -- 172.21.15.62:0/2343954614 shutdown_connections 2024-03-23T11:26:12.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.770+0000 7f174e800640 1 --2- 172.21.15.62:0/2343954614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1748143450 0x7f17481438e0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.770+0000 7f174e800640 1 --2- 172.21.15.62:0/2343954614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17480caff0 0x7f17480cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.770+0000 7f174e800640 1 --2- 172.21.15.62:0/2343954614 >> v2:172.21.15.120:3300/0 conn(0x7f1748069c60 0x7f174806a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.770+0000 7f174e800640 1 -- 172.21.15.62:0/2343954614 >> 172.21.15.62:0/2343954614 conn(0x7f17480b0090 msgr2=0x7f17480b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:12.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.770+0000 7f174e800640 1 -- 172.21.15.62:0/2343954614 shutdown_connections 2024-03-23T11:26:12.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.770+0000 7f174e800640 1 -- 172.21.15.62:0/2343954614 wait complete. 2024-03-23T11:26:12.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.772+0000 7f174e800640 1 Processor -- start 2024-03-23T11:26:12.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.772+0000 7f174e800640 1 -- start start 2024-03-23T11:26:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.773+0000 7f174e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1748069c60 0x7f17480ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.773+0000 7f174e800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17480caff0 0x7f1748142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.773+0000 7f174e800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1748143450 0x7f17480acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.773+0000 7f174e800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1748143300 con 0x7f1748069c60 2024-03-23T11:26:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.773+0000 7f174e800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17480abf00 con 0x7f17480caff0 2024-03-23T11:26:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.773+0000 7f174e800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17480ac1e0 con 0x7f1748143450 2024-03-23T11:26:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.773+0000 7f174d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1748069c60 0x7f17480ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.773+0000 7f174d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1748069c60 0x7f17480ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42942/0 (socket says 172.21.15.62:42942) 2024-03-23T11:26:12.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.773+0000 7f174d400640 1 -- 172.21.15.62:0/118516996 learned_addr learned my addr 172.21.15.62:0/118516996 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:12.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f174ca00640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17480caff0 0x7f1748142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f174d400640 1 -- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1748143450 msgr2=0x7f17480acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:12.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f174de00640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1748143450 0x7f17480acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f174d400640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1748143450 0x7f17480acfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f174d400640 1 -- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17480caff0 msgr2=0x7f1748142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:12.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f174d400640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17480caff0 0x7f1748142dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:12.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f174d400640 1 -- 172.21.15.62:0/118516996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1740009b30 con 0x7f1748069c60 2024-03-23T11:26:12.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f174d400640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1748069c60 0x7f17480ab840 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f173c00d800 tx=0x7f173c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:12.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f1745600640 1 -- 172.21.15.62:0/118516996 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c0096a0 con 0x7f1748069c60 2024-03-23T11:26:12.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f1745600640 1 -- 172.21.15.62:0/118516996 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c009800 con 0x7f1748069c60 2024-03-23T11:26:12.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f174e800640 1 -- 172.21.15.62:0/118516996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17480ac4c0 con 0x7f1748069c60 2024-03-23T11:26:12.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f1745600640 1 -- 172.21.15.62:0/118516996 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c009b90 con 0x7f1748069c60 2024-03-23T11:26:12.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.774+0000 7f174e800640 1 -- 172.21.15.62:0/118516996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17480aca80 con 0x7f1748069c60 2024-03-23T11:26:12.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.775+0000 7f174e800640 1 -- 172.21.15.62:0/118516996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17480cc1a0 con 0x7f1748069c60 2024-03-23T11:26:12.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.776+0000 7f1745600640 1 -- 172.21.15.62:0/118516996 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f173c009cf0 con 0x7f1748069c60 2024-03-23T11:26:12.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.777+0000 7f1745600640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f172806a7a0 0x7f172806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:12.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.777+0000 7f1745600640 1 -- 172.21.15.62:0/118516996 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f173c0895a0 con 0x7f1748069c60 2024-03-23T11:26:12.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.778+0000 7f174ca00640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f172806a7a0 0x7f172806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:12.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.779+0000 7f174ca00640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f172806a7a0 0x7f172806cc60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f17480ca340 tx=0x7f1734008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:12.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:12.782+0000 7f1745600640 1 -- 172.21.15.62:0/118516996 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f173c0548c0 con 0x7f1748069c60 2024-03-23T11:26:13.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.021+0000 7f174e800640 1 -- 172.21.15.62:0/118516996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1748066bb0 con 0x7f1748069c60 2024-03-23T11:26:13.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.542+0000 7f1745600640 1 -- 172.21.15.62:0/118516996 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v52) v1 ==== 104+0+0 (secure 0 0 0) 0x7f173c00b440 con 0x7f1748069c60 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 -- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f172806a7a0 msgr2=0x7f172806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f172806a7a0 0x7f172806cc60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f17480ca340 tx=0x7f1734008040 comp rx=0 tx=0).stop 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 -- 172.21.15.62:0/118516996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1748069c60 msgr2=0x7f17480ab840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1748069c60 0x7f17480ab840 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f173c00d800 tx=0x7f173c00dcc0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 -- 172.21.15.62:0/118516996 shutdown_connections 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f172806a7a0 0x7f172806cc60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1748143450 0x7f17480acfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17480caff0 0x7f1748142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 --2- 172.21.15.62:0/118516996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1748069c60 0x7f17480ab840 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 -- 172.21.15.62:0/118516996 >> 172.21.15.62:0/118516996 conn(0x7f17480b0090 msgr2=0x7f17480cd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 -- 172.21.15.62:0/118516996 shutdown_connections 2024-03-23T11:26:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.546+0000 7f174e800640 1 -- 172.21.15.62:0/118516996 wait complete. 2024-03-23T11:26:13.561 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T11:26:13.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.785+0000 7f3228a00640 1 Processor -- start 2024-03-23T11:26:13.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.785+0000 7f3228a00640 1 -- start start 2024-03-23T11:26:13.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.787+0000 7f3228a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3224154d30 0x7f32240ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:13.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.787+0000 7f3228a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f32240cebd0 0x7f32240ab140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:13.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.787+0000 7f3228a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322415bcb0 con 0x7f32240ab680 2024-03-23T11:26:13.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.787+0000 7f3228a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32240cf050 con 0x7f32240cebd0 2024-03-23T11:26:13.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.787+0000 7f3228a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322415ac90 con 0x7f3224154d30 2024-03-23T11:26:13.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.787+0000 7f3223200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f32240ab680 0x7f32240b1160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33246/0 (socket says 172.21.15.62:33246) 2024-03-23T11:26:13.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.787+0000 7f3223200640 1 -- 172.21.15.62:0/148170753 learned_addr learned my addr 172.21.15.62:0/148170753 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:13.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.787+0000 7f3221e00640 1 --2- 172.21.15.62:0/148170753 >> v2:172.21.15.120:3300/0 conn(0x7f32240cebd0 0x7f32240ab140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.787+0000 7f3222800640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3224154d30 0x7f32240ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078263219 0 0) 0x7f322415bcb0 con 0x7f32240ab680 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3208003520 con 0x7f32240ab680 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3975295466 0 0) 0x7f3208003520 con 0x7f32240ab680 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3224154d30 msgr2=0x7f32240ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3224154d30 0x7f32240ce690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 >> v2:172.21.15.120:3300/0 conn(0x7f32240cebd0 msgr2=0x7f32240ab140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 --2- 172.21.15.62:0/148170753 >> v2:172.21.15.120:3300/0 conn(0x7f32240cebd0 0x7f32240ab140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322415af30 con 0x7f32240ab680 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f321c0019f0 con 0x7f32240ab680 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 >> v1:172.21.15.62:6789/0 conn(0x7f32240ab680 legacy=0x7f32240b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3208003970 0x7f3208003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:13.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3224154d30 0x7f32080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3208004d00 0x7f3208005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32240cf050 con 0x7f3208004d00 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322415ac90 con 0x7f3208003970 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.788+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320800b980 con 0x7f3224154d30 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.789+0000 7f3223200640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3208004d00 0x7f3208005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.789+0000 7f3222800640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3208003970 0x7f3208003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.789+0000 7f3221e00640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3224154d30 0x7f32080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.789+0000 7f3223200640 1 -- 172.21.15.62:0/148170753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3224154d30 msgr2=0x7f32080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.789+0000 7f3223200640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3224154d30 0x7f32080047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.789+0000 7f3223200640 1 -- 172.21.15.62:0/148170753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3208003970 msgr2=0x7f3208003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.789+0000 7f3223200640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3208003970 0x7f3208003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.789+0000 7f3223200640 1 -- 172.21.15.62:0/148170753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3208003520 con 0x7f3208004d00 2024-03-23T11:26:13.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.790+0000 7f3223200640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3208004d00 0x7f3208005180 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f321c00a790 tx=0x7f32080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c656b66c346d5b27 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:13.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.790+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c034630 con 0x7f3208004d00 2024-03-23T11:26:13.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.790+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c02f070 con 0x7f3208004d00 2024-03-23T11:26:13.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.790+0000 7f3221400640 1 -- 172.21.15.62:0/148170753 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c0349f0 con 0x7f3208004d00 2024-03-23T11:26:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.791+0000 7f3228a00640 1 -- 172.21.15.62:0/148170753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3208004d00 msgr2=0x7f3208005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.791+0000 7f3228a00640 1 --2- 172.21.15.62:0/148170753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3208004d00 0x7f3208005180 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f321c00a790 tx=0x7f32080057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.791+0000 7f3223200640 1 -- 172.21.15.62:0/148170753 reap_dead start 2024-03-23T11:26:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.791+0000 7f3228a00640 1 -- 172.21.15.62:0/148170753 shutdown_connections 2024-03-23T11:26:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.791+0000 7f3228a00640 1 -- 172.21.15.62:0/148170753 >> 172.21.15.62:0/148170753 conn(0x7f32240b0280 msgr2=0x7f32240b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.791+0000 7f3228a00640 1 -- 172.21.15.62:0/148170753 shutdown_connections 2024-03-23T11:26:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.791+0000 7f3228a00640 1 -- 172.21.15.62:0/148170753 wait complete. 2024-03-23T11:26:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.793+0000 7f3228a00640 1 Processor -- start 2024-03-23T11:26:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.793+0000 7f3228a00640 1 -- start start 2024-03-23T11:26:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.794+0000 7f3228a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3208004d00 0x7f32241f14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.794+0000 7f3228a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32241f1a20 0x7f3224159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.794+0000 7f3228a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3224159ca0 0x7f32240cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.794+0000 7f3228a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322415a130 con 0x7f3224159ca0 2024-03-23T11:26:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.794+0000 7f3228a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32241528c0 con 0x7f32241f1a20 2024-03-23T11:26:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.794+0000 7f3228a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3224151cb0 con 0x7f3208004d00 2024-03-23T11:26:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.794+0000 7f3223200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3224159ca0 0x7f32240cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.794+0000 7f3223200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3224159ca0 0x7f32240cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42970/0 (socket says 172.21.15.62:42970) 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.794+0000 7f3223200640 1 -- 172.21.15.62:0/57582110 learned_addr learned my addr 172.21.15.62:0/57582110 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.794+0000 7f3222800640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3208004d00 0x7f32241f14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.794+0000 7f3221e00640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32241f1a20 0x7f3224159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.795+0000 7f3223200640 1 -- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3208004d00 msgr2=0x7f32241f14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.795+0000 7f3223200640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3208004d00 0x7f32241f14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.795+0000 7f3223200640 1 -- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32241f1a20 msgr2=0x7f3224159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.795+0000 7f3223200640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32241f1a20 0x7f3224159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.795+0000 7f3223200640 1 -- 172.21.15.62:0/57582110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224151f90 con 0x7f3224159ca0 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.795+0000 7f3223200640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3224159ca0 0x7f32240cca00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f321c00a8b0 tx=0x7f321c001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.795+0000 7f321aa00640 1 -- 172.21.15.62:0/57582110 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c034840 con 0x7f3224159ca0 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.795+0000 7f321aa00640 1 -- 172.21.15.62:0/57582110 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c0349a0 con 0x7f3224159ca0 2024-03-23T11:26:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.795+0000 7f321aa00640 1 -- 172.21.15.62:0/57582110 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c034d10 con 0x7f3224159ca0 2024-03-23T11:26:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.795+0000 7f3228a00640 1 -- 172.21.15.62:0/57582110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3224152270 con 0x7f3224159ca0 2024-03-23T11:26:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.795+0000 7f3228a00640 1 -- 172.21.15.62:0/57582110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32240c7ea0 con 0x7f3224159ca0 2024-03-23T11:26:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.796+0000 7f3228a00640 1 -- 172.21.15.62:0/57582110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3224152a50 con 0x7f3224159ca0 2024-03-23T11:26:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.797+0000 7f321aa00640 1 -- 172.21.15.62:0/57582110 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f321c031030 con 0x7f3224159ca0 2024-03-23T11:26:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.798+0000 7f321aa00640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f320006a7d0 0x7f320006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.798+0000 7f321aa00640 1 -- 172.21.15.62:0/57582110 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f321c0ab1f0 con 0x7f3224159ca0 2024-03-23T11:26:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.799+0000 7f3222800640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f320006a7d0 0x7f320006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:13.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.800+0000 7f3222800640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f320006a7d0 0x7f320006cc90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3214009990 tx=0x7f3214008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:13.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:13.802+0000 7f321aa00640 1 -- 172.21.15.62:0/57582110 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f321c076640 con 0x7f3224159ca0 2024-03-23T11:26:13.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.001+0000 7f3228a00640 1 -- 172.21.15.62:0/57582110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3224152a00 con 0x7f3224159ca0 2024-03-23T11:26:14.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.548+0000 7f321aa00640 1 -- 172.21.15.62:0/57582110 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v79) v1 ==== 181+0+0 (secure 0 0 0) 0x7f321c001f00 con 0x7f3224159ca0 2024-03-23T11:26:14.542 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' removed 2024-03-23T11:26:14.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.554+0000 7f3228a00640 1 -- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f320006a7d0 msgr2=0x7f320006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:14.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.554+0000 7f3228a00640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f320006a7d0 0x7f320006cc90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3214009990 tx=0x7f3214008040 comp rx=0 tx=0).stop 2024-03-23T11:26:14.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.554+0000 7f3228a00640 1 -- 172.21.15.62:0/57582110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3224159ca0 msgr2=0x7f32240cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:14.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.554+0000 7f3228a00640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3224159ca0 0x7f32240cca00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f321c00a8b0 tx=0x7f321c001cb0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.554+0000 7f3228a00640 1 -- 172.21.15.62:0/57582110 shutdown_connections 2024-03-23T11:26:14.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.554+0000 7f3228a00640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f320006a7d0 0x7f320006cc90 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.554+0000 7f3228a00640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3224159ca0 0x7f32240cca00 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.554+0000 7f3228a00640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32241f1a20 0x7f3224159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.554+0000 7f3228a00640 1 --2- 172.21.15.62:0/57582110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3208004d00 0x7f32241f14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.554+0000 7f3228a00640 1 -- 172.21.15.62:0/57582110 >> 172.21.15.62:0/57582110 conn(0x7f32240b0280 msgr2=0x7f322406c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:14.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.555+0000 7f3228a00640 1 -- 172.21.15.62:0/57582110 shutdown_connections 2024-03-23T11:26:14.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.555+0000 7f3228a00640 1 -- 172.21.15.62:0/57582110 wait complete. 2024-03-23T11:26:14.573 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T11:26:14.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.807+0000 7f8a3d200640 1 Processor -- start 2024-03-23T11:26:14.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.807+0000 7f8a3d200640 1 -- start start 2024-03-23T11:26:14.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.809+0000 7f8a3d200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8a380ca3e0 0x7f8a38167920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.809+0000 7f8a3d200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a380ce1b0 con 0x7f8a38153810 2024-03-23T11:26:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.809+0000 7f8a3d200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a380b12d0 con 0x7f8a38153bf0 2024-03-23T11:26:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.809+0000 7f8a3d200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a3806d5b0 con 0x7f8a380ca3e0 2024-03-23T11:26:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.809+0000 7f8a35e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8a38153810 0x7f8a380b0760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33256/0 (socket says 172.21.15.62:33256) 2024-03-23T11:26:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.809+0000 7f8a35e00640 1 -- 172.21.15.62:0/1039965062 learned_addr learned my addr 172.21.15.62:0/1039965062 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:14.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.810+0000 7f8a36800640 1 --2- 172.21.15.62:0/1039965062 >> v2:172.21.15.120:3300/0 conn(0x7f8a380ca3e0 0x7f8a38167920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:14.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.810+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964077039 0 0) 0x7f8a380ce1b0 con 0x7f8a38153810 2024-03-23T11:26:14.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.810+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a1c003520 con 0x7f8a38153810 2024-03-23T11:26:14.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.810+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2092303112 0 0) 0x7f8a1c003520 con 0x7f8a38153810 2024-03-23T11:26:14.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.810+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 >> v2:172.21.15.120:3300/0 conn(0x7f8a380ca3e0 msgr2=0x7f8a38167920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:14.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.810+0000 7f8a34a00640 1 --2- 172.21.15.62:0/1039965062 >> v2:172.21.15.120:3300/0 conn(0x7f8a380ca3e0 0x7f8a38167920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.810+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 >> v1:172.21.15.120:6789/0 conn(0x7f8a38153bf0 legacy=0x7f8a380c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.810+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a38070cd0 con 0x7f8a38153810 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.810+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f8a28001670 con 0x7f8a38153810 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.810+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 >> v1:172.21.15.62:6789/0 conn(0x7f8a38153810 legacy=0x7f8a380b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a34a00640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a1c0039c0 0x7f8a1c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a34a00640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a1c0047c0 0x7f8a1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a34a00640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a1c0051d0 0x7f8a1c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c00c0f0 con 0x7f8a1c0051d0 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c00c260 con 0x7f8a1c0047c0 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c00c420 con 0x7f8a1c0039c0 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a36800640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a1c0051d0 0x7f8a1c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a35400640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a1c0047c0 0x7f8a1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a35e00640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a1c0039c0 0x7f8a1c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:14.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a36800640 1 -- 172.21.15.62:0/1039965062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a1c0039c0 msgr2=0x7f8a1c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:14.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a36800640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a1c0039c0 0x7f8a1c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a36800640 1 -- 172.21.15.62:0/1039965062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a1c0047c0 msgr2=0x7f8a1c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:14.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a36800640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a1c0047c0 0x7f8a1c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a36800640 1 -- 172.21.15.62:0/1039965062 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c003520 con 0x7f8a1c0051d0 2024-03-23T11:26:14.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.811+0000 7f8a35e00640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a1c0039c0 0x7f8a1c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:14.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.812+0000 7f8a36800640 1 -- 172.21.15.62:0/1039965062 reap_dead start 2024-03-23T11:26:14.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.813+0000 7f8a36800640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a1c0051d0 0x7f8a1c00b980 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8a3806d110 tx=0x7f8a1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1a6e47ba17b71 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:14.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.813+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3002cbd0 con 0x7f8a1c0051d0 2024-03-23T11:26:14.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.813+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3000f040 con 0x7f8a1c0051d0 2024-03-23T11:26:14.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.813+0000 7f8a34a00640 1 -- 172.21.15.62:0/1039965062 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a30003990 con 0x7f8a1c0051d0 2024-03-23T11:26:14.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.814+0000 7f8a3d200640 1 -- 172.21.15.62:0/1039965062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a1c0051d0 msgr2=0x7f8a1c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:14.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.814+0000 7f8a3d200640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a1c0051d0 0x7f8a1c00b980 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8a3806d110 tx=0x7f8a1c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.814+0000 7f8a3d200640 1 -- 172.21.15.62:0/1039965062 shutdown_connections 2024-03-23T11:26:14.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.814+0000 7f8a3d200640 1 --2- 172.21.15.62:0/1039965062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a1c0051d0 0x7f8a1c00b980 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.814+0000 7f8a3d200640 1 -- 172.21.15.62:0/1039965062 >> 172.21.15.62:0/1039965062 conn(0x7f8a380b17d0 msgr2=0x7f8a380b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:14.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.814+0000 7f8a3d200640 1 -- 172.21.15.62:0/1039965062 shutdown_connections 2024-03-23T11:26:14.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.814+0000 7f8a3d200640 1 -- 172.21.15.62:0/1039965062 wait complete. 2024-03-23T11:26:14.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.816+0000 7f8a3d200640 1 Processor -- start 2024-03-23T11:26:14.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.816+0000 7f8a3d200640 1 -- start start 2024-03-23T11:26:14.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.817+0000 7f8a3d200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a1c0051d0 0x7f8a380cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.817+0000 7f8a3d200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a38151ec0 0x7f8a38152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.817+0000 7f8a3d200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a3815e0d0 0x7f8a3815e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.817+0000 7f8a3d200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a38152880 con 0x7f8a3815e0d0 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.817+0000 7f8a3d200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3815ed70 con 0x7f8a1c0051d0 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.817+0000 7f8a3d200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3815f050 con 0x7f8a38151ec0 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.817+0000 7f8a36800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a3815e0d0 0x7f8a3815e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.818+0000 7f8a36800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a3815e0d0 0x7f8a3815e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42992/0 (socket says 172.21.15.62:42992) 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.818+0000 7f8a36800640 1 -- 172.21.15.62:0/3124267567 learned_addr learned my addr 172.21.15.62:0/3124267567 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.818+0000 7f8a35e00640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a1c0051d0 0x7f8a380cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.818+0000 7f8a35400640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a38151ec0 0x7f8a38152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.818+0000 7f8a36800640 1 -- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a38151ec0 msgr2=0x7f8a38152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.818+0000 7f8a36800640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a38151ec0 0x7f8a38152340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.818+0000 7f8a36800640 1 -- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a1c0051d0 msgr2=0x7f8a380cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.818+0000 7f8a36800640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a1c0051d0 0x7f8a380cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.818+0000 7f8a36800640 1 -- 172.21.15.62:0/3124267567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3815f330 con 0x7f8a3815e0d0 2024-03-23T11:26:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.818+0000 7f8a35e00640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a1c0051d0 0x7f8a380cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:26:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.818+0000 7f8a36800640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a3815e0d0 0x7f8a3815e550 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8a3002c640 tx=0x7f8a30009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:14.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.819+0000 7f8a2e000640 1 -- 172.21.15.62:0/3124267567 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3002ce50 con 0x7f8a3815e0d0 2024-03-23T11:26:14.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.819+0000 7f8a2e000640 1 -- 172.21.15.62:0/3124267567 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a30004230 con 0x7f8a3815e0d0 2024-03-23T11:26:14.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.819+0000 7f8a2e000640 1 -- 172.21.15.62:0/3124267567 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a300045c0 con 0x7f8a3815e0d0 2024-03-23T11:26:14.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.819+0000 7f8a3d200640 1 -- 172.21.15.62:0/3124267567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a38155950 con 0x7f8a3815e0d0 2024-03-23T11:26:14.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.819+0000 7f8a3d200640 1 -- 172.21.15.62:0/3124267567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a380cae20 con 0x7f8a3815e0d0 2024-03-23T11:26:14.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.820+0000 7f8a3d200640 1 -- 172.21.15.62:0/3124267567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a380cafb0 con 0x7f8a3815e0d0 2024-03-23T11:26:14.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.821+0000 7f8a2e000640 1 -- 172.21.15.62:0/3124267567 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8a30004720 con 0x7f8a3815e0d0 2024-03-23T11:26:14.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.822+0000 7f8a2e000640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8a1806a7d0 0x7f8a1806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:14.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.822+0000 7f8a2e000640 1 -- 172.21.15.62:0/3124267567 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5501+0+0 (secure 0 0 0) 0x7f8a300ab750 con 0x7f8a3815e0d0 2024-03-23T11:26:14.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.823+0000 7f8a35e00640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8a1806a7d0 0x7f8a1806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:14.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.824+0000 7f8a35e00640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8a1806a7d0 0x7f8a1806cc90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8a28001eb0 tx=0x7f8a28009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:14.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:14.826+0000 7f8a2e000640 1 -- 172.21.15.62:0/3124267567 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a380cafb0 con 0x7f8a3815e0d0 2024-03-23T11:26:15.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.025+0000 7f8a3d200640 1 -- 172.21.15.62:0/3124267567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8a380cafb0 con 0x7f8a3815e0d0 2024-03-23T11:26:15.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.551+0000 7f8a2e000640 1 -- 172.21.15.62:0/3124267567 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v80) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8a380cafb0 con 0x7f8a3815e0d0 2024-03-23T11:26:15.545 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' removed 2024-03-23T11:26:15.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.556+0000 7f8a3d200640 1 -- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8a1806a7d0 msgr2=0x7f8a1806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:15.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.556+0000 7f8a3d200640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8a1806a7d0 0x7f8a1806cc90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8a28001eb0 tx=0x7f8a28009040 comp rx=0 tx=0).stop 2024-03-23T11:26:15.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.556+0000 7f8a3d200640 1 -- 172.21.15.62:0/3124267567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a3815e0d0 msgr2=0x7f8a3815e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:15.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.556+0000 7f8a3d200640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a3815e0d0 0x7f8a3815e550 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8a3002c640 tx=0x7f8a30009750 comp rx=0 tx=0).stop 2024-03-23T11:26:15.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.557+0000 7f8a3d200640 1 -- 172.21.15.62:0/3124267567 shutdown_connections 2024-03-23T11:26:15.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.557+0000 7f8a3d200640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8a1806a7d0 0x7f8a1806cc90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:15.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.557+0000 7f8a3d200640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a3815e0d0 0x7f8a3815e550 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:15.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.557+0000 7f8a3d200640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a38151ec0 0x7f8a38152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:15.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.557+0000 7f8a3d200640 1 --2- 172.21.15.62:0/3124267567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a1c0051d0 0x7f8a380cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:15.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.557+0000 7f8a3d200640 1 -- 172.21.15.62:0/3124267567 >> 172.21.15.62:0/3124267567 conn(0x7f8a380b17d0 msgr2=0x7f8a380b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:15.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.557+0000 7f8a3d200640 1 -- 172.21.15.62:0/3124267567 shutdown_connections 2024-03-23T11:26:15.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.557+0000 7f8a3d200640 1 -- 172.21.15.62:0/3124267567 wait complete. 2024-03-23T11:26:15.575 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T11:26:15.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.792+0000 7fb39e200640 1 Processor -- start 2024-03-23T11:26:15.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.792+0000 7fb39e200640 1 -- start start 2024-03-23T11:26:15.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.793+0000 7fb39e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb39815c9e0 0x7fb3981607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:15.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.793+0000 7fb39e200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39806bfc0 con 0x7fb3980cf650 2024-03-23T11:26:15.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.793+0000 7fb39e200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3980b5260 con 0x7fb39815cdc0 2024-03-23T11:26:15.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.793+0000 7fb39e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3980d2630 con 0x7fb39815c9e0 2024-03-23T11:26:15.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.793+0000 7fb397e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb3980cf650 0x7fb39815d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33270/0 (socket says 172.21.15.62:33270) 2024-03-23T11:26:15.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.794+0000 7fb397e00640 1 -- 172.21.15.62:0/1781743556 learned_addr learned my addr 172.21.15.62:0/1781743556 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:15.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.794+0000 7fb397400640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb39815c9e0 0x7fb3981607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:15.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.794+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111380293 0 0) 0x7fb39806bfc0 con 0x7fb3980cf650 2024-03-23T11:26:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.794+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb37c003520 con 0x7fb3980cf650 2024-03-23T11:26:15.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.794+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2340744396 0 0) 0x7fb37c003520 con 0x7fb3980cf650 2024-03-23T11:26:15.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.794+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb39815c9e0 msgr2=0x7fb3981607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:15.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.794+0000 7fb396000640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb39815c9e0 0x7fb3981607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:15.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.794+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 >> v1:172.21.15.120:6789/0 conn(0x7fb39815cdc0 legacy=0x7fb3980d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:15.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.794+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3980ce140 con 0x7fb3980cf650 2024-03-23T11:26:15.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.794+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fb3900019f0 con 0x7fb3980cf650 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 >> v1:172.21.15.62:6789/0 conn(0x7fb3980cf650 legacy=0x7fb39815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396000640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb39815c9e0 0x7fb37c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396000640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb37c003f70 0x7fb37c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396000640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb37c004d00 0x7fb37c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3980d2630 con 0x7fb37c003f70 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c00bac0 con 0x7fb37c004d00 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c00bc80 con 0x7fb39815c9e0 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396a00640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb37c003f70 0x7fb37c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb397e00640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb37c004d00 0x7fb37c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb397400640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb39815c9e0 0x7fb37c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396a00640 1 -- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb39815c9e0 msgr2=0x7fb37c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396a00640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb39815c9e0 0x7fb37c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396a00640 1 -- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb37c004d00 msgr2=0x7fb37c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:15.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396a00640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb37c004d00 0x7fb37c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:15.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.795+0000 7fb396a00640 1 -- 172.21.15.62:0/1781743556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c003520 con 0x7fb37c003f70 2024-03-23T11:26:15.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.796+0000 7fb397400640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb39815c9e0 0x7fb37c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:15.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.796+0000 7fb397e00640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb37c004d00 0x7fb37c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:15.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.797+0000 7fb396a00640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb37c003f70 0x7fb37c0047c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb384001750 tx=0x7fb37c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e08ae6c7fc64e19 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:15.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.797+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3840052d0 con 0x7fb37c003f70 2024-03-23T11:26:15.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.797+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38402e050 con 0x7fb37c003f70 2024-03-23T11:26:15.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.797+0000 7fb396000640 1 -- 172.21.15.62:0/1781743556 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384003830 con 0x7fb37c003f70 2024-03-23T11:26:15.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.798+0000 7fb39e200640 1 -- 172.21.15.62:0/1781743556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb37c003f70 msgr2=0x7fb37c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:15.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.798+0000 7fb39e200640 1 --2- 172.21.15.62:0/1781743556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb37c003f70 0x7fb37c0047c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb384001750 tx=0x7fb37c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:15.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.798+0000 7fb397e00640 1 -- 172.21.15.62:0/1781743556 reap_dead start 2024-03-23T11:26:15.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.798+0000 7fb39e200640 1 -- 172.21.15.62:0/1781743556 shutdown_connections 2024-03-23T11:26:15.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.798+0000 7fb39e200640 1 -- 172.21.15.62:0/1781743556 >> 172.21.15.62:0/1781743556 conn(0x7fb398071570 msgr2=0x7fb398071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:15.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.798+0000 7fb39e200640 1 -- 172.21.15.62:0/1781743556 shutdown_connections 2024-03-23T11:26:15.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.798+0000 7fb39e200640 1 -- 172.21.15.62:0/1781743556 wait complete. 2024-03-23T11:26:15.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.800+0000 7fb39e200640 1 Processor -- start 2024-03-23T11:26:15.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.800+0000 7fb39e200640 1 -- start start 2024-03-23T11:26:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb39e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c003f70 0x7fb398152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb39e200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3980c9070 0x7fb3980cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb39e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39815a860 0x7fb39815ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb39e200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3980cc100 con 0x7fb39815a860 2024-03-23T11:26:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb39e200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39815b250 con 0x7fb3980c9070 2024-03-23T11:26:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb39e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3980cb480 con 0x7fb37c003f70 2024-03-23T11:26:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb397e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39815a860 0x7fb39815ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb397e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39815a860 0x7fb39815ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43018/0 (socket says 172.21.15.62:43018) 2024-03-23T11:26:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb397e00640 1 -- 172.21.15.62:0/786460913 learned_addr learned my addr 172.21.15.62:0/786460913 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb397e00640 1 -- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c003f70 msgr2=0x7fb398152430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb397e00640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c003f70 0x7fb398152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb396a00640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3980c9070 0x7fb3980cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb397e00640 1 -- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3980c9070 msgr2=0x7fb3980cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb397e00640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3980c9070 0x7fb3980cab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.801+0000 7fb397e00640 1 -- 172.21.15.62:0/786460913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3980cb760 con 0x7fb39815a860 2024-03-23T11:26:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.802+0000 7fb397e00640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39815a860 0x7fb39815ad10 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb3980b5910 tx=0x7fb39000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:15.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.802+0000 7fb38f400640 1 -- 172.21.15.62:0/786460913 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390004d60 con 0x7fb39815a860 2024-03-23T11:26:15.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.802+0000 7fb38f400640 1 -- 172.21.15.62:0/786460913 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39000bb80 con 0x7fb39815a860 2024-03-23T11:26:15.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.802+0000 7fb38f400640 1 -- 172.21.15.62:0/786460913 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390007070 con 0x7fb39815a860 2024-03-23T11:26:15.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.802+0000 7fb39e200640 1 -- 172.21.15.62:0/786460913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3980cba40 con 0x7fb39815a860 2024-03-23T11:26:15.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.802+0000 7fb39e200640 1 -- 172.21.15.62:0/786460913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb398152970 con 0x7fb39815a860 2024-03-23T11:26:15.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.803+0000 7fb39e200640 1 -- 172.21.15.62:0/786460913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb398152b00 con 0x7fb39815a860 2024-03-23T11:26:15.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.804+0000 7fb38f400640 1 -- 172.21.15.62:0/786460913 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb3900071d0 con 0x7fb39815a860 2024-03-23T11:26:15.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.805+0000 7fb38f400640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb37806a7a0 0x7fb37806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:15.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.805+0000 7fb38f400640 1 -- 172.21.15.62:0/786460913 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb390088ea0 con 0x7fb39815a860 2024-03-23T11:26:15.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.806+0000 7fb397400640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb37806a7a0 0x7fb37806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:15.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.807+0000 7fb397400640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb37806a7a0 0x7fb37806cc60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb3880099e0 tx=0x7fb388008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:15.810+0000 7fb38f400640 1 -- 172.21.15.62:0/786460913 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb3900544c0 con 0x7fb39815a860 2024-03-23T11:26:16.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.011+0000 7fb39e200640 1 -- 172.21.15.62:0/786460913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb398066c90 con 0x7fb39815a860 2024-03-23T11:26:16.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.022+0000 7fb38f400640 1 -- 172.21.15.62:0/786460913 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb390057ae0 con 0x7fb39815a860 2024-03-23T11:26:16.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.025+0000 7fb39e200640 1 -- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb37806a7a0 msgr2=0x7fb37806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.025+0000 7fb39e200640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb37806a7a0 0x7fb37806cc60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb3880099e0 tx=0x7fb388008040 comp rx=0 tx=0).stop 2024-03-23T11:26:16.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.025+0000 7fb39e200640 1 -- 172.21.15.62:0/786460913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39815a860 msgr2=0x7fb39815ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.025+0000 7fb39e200640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39815a860 0x7fb39815ad10 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb3980b5910 tx=0x7fb39000b4a0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.026+0000 7fb39e200640 1 -- 172.21.15.62:0/786460913 shutdown_connections 2024-03-23T11:26:16.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.026+0000 7fb39e200640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb37806a7a0 0x7fb37806cc60 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.026+0000 7fb39e200640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39815a860 0x7fb39815ad10 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.026+0000 7fb39e200640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3980c9070 0x7fb3980cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.026+0000 7fb39e200640 1 --2- 172.21.15.62:0/786460913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb37c003f70 0x7fb398152430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.026+0000 7fb39e200640 1 -- 172.21.15.62:0/786460913 >> 172.21.15.62:0/786460913 conn(0x7fb398071570 msgr2=0x7fb3980b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:16.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.026+0000 7fb39e200640 1 -- 172.21.15.62:0/786460913 shutdown_connections 2024-03-23T11:26:16.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.026+0000 7fb39e200640 1 -- 172.21.15.62:0/786460913 wait complete. 2024-03-23T11:26:16.037 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-03-23T11:26:16.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.264+0000 7ff403400640 1 Processor -- start 2024-03-23T11:26:16.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.264+0000 7ff403400640 1 -- start start 2024-03-23T11:26:16.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.265+0000 7ff403400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff3fc15cdc0 0x7ff3fc0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.265+0000 7ff403400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc06bfc0 con 0x7ff3fc15c9e0 2024-03-23T11:26:16.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.265+0000 7ff403400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0b5260 con 0x7ff3fc0cf650 2024-03-23T11:26:16.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.265+0000 7ff403400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3fc0ce140 con 0x7ff3fc15cdc0 2024-03-23T11:26:16.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.265+0000 7ff400a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff3fc15c9e0 0x7ff3fc1607c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33286/0 (socket says 172.21.15.62:33286) 2024-03-23T11:26:16.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.265+0000 7ff400a00640 1 -- 172.21.15.62:0/1801016116 learned_addr learned my addr 172.21.15.62:0/1801016116 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:16.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.266+0000 7ff3fbe00640 1 --2- 172.21.15.62:0/1801016116 >> v2:172.21.15.120:3300/0 conn(0x7ff3fc15cdc0 0x7ff3fc0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.266+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182614200 0 0) 0x7ff3fc06bfc0 con 0x7ff3fc15c9e0 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.266+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e0003520 con 0x7ff3fc15c9e0 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.266+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4052248718 0 0) 0x7ff3e0003520 con 0x7ff3fc15c9e0 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.266+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 >> v2:172.21.15.120:3300/0 conn(0x7ff3fc15cdc0 msgr2=0x7ff3fc0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.266+0000 7ff3fb400640 1 --2- 172.21.15.62:0/1801016116 >> v2:172.21.15.120:3300/0 conn(0x7ff3fc15cdc0 0x7ff3fc0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.266+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 >> v1:172.21.15.120:6789/0 conn(0x7ff3fc0cf650 legacy=0x7ff3fc15d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.266+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc0b4370 con 0x7ff3fc15c9e0 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.266+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7ff3ec001670 con 0x7ff3fc15c9e0 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.266+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 >> v1:172.21.15.62:6789/0 conn(0x7ff3fc15c9e0 legacy=0x7ff3fc1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.266+0000 7ff3fb400640 1 --2- 172.21.15.62:0/1801016116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 0x7ff3e0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff3fb400640 1 --2- 172.21.15.62:0/1801016116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3e00047c0 0x7ff3e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff3fb400640 1 --2- 172.21.15.62:0/1801016116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3e0005180 0x7ff3e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc0ce140 con 0x7ff3e0003970 2024-03-23T11:26:16.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e000c000 con 0x7ff3e00047c0 2024-03-23T11:26:16.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e000c1c0 con 0x7ff3e0005180 2024-03-23T11:26:16.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff400a00640 1 --2- 172.21.15.62:0/1801016116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 0x7ff3e0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:16.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff3fbe00640 1 --2- 172.21.15.62:0/1801016116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3e00047c0 0x7ff3e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:16.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff400a00640 1 -- 172.21.15.62:0/1801016116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3e0005180 msgr2=0x7ff3e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:16.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff400a00640 1 --2- 172.21.15.62:0/1801016116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3e0005180 0x7ff3e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff400a00640 1 -- 172.21.15.62:0/1801016116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3e00047c0 msgr2=0x7ff3e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:16.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff400a00640 1 --2- 172.21.15.62:0/1801016116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3e00047c0 0x7ff3e0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff400a00640 1 -- 172.21.15.62:0/1801016116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e0003520 con 0x7ff3e0003970 2024-03-23T11:26:16.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.267+0000 7ff401400640 1 -- 172.21.15.62:0/1801016116 reap_dead start 2024-03-23T11:26:16.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.268+0000 7ff400a00640 1 --2- 172.21.15.62:0/1801016116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 0x7ff3e0003d70 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00c210 tx=0x7ff3e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1260446912d5d5da server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:16.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.269+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec02cae0 con 0x7ff3e0003970 2024-03-23T11:26:16.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.269+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ec02cc60 con 0x7ff3e0003970 2024-03-23T11:26:16.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.269+0000 7ff3fb400640 1 -- 172.21.15.62:0/1801016116 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec005b10 con 0x7ff3e0003970 2024-03-23T11:26:16.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.269+0000 7ff403400640 1 -- 172.21.15.62:0/1801016116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 msgr2=0x7ff3e0003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.269+0000 7ff403400640 1 --2- 172.21.15.62:0/1801016116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 0x7ff3e0003d70 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00c210 tx=0x7ff3e00057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.269+0000 7ff403400640 1 -- 172.21.15.62:0/1801016116 shutdown_connections 2024-03-23T11:26:16.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.269+0000 7ff403400640 1 --2- 172.21.15.62:0/1801016116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 0x7ff3e0003d70 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.269+0000 7ff403400640 1 -- 172.21.15.62:0/1801016116 >> 172.21.15.62:0/1801016116 conn(0x7ff3fc071570 msgr2=0x7ff3fc071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:16.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.269+0000 7ff403400640 1 -- 172.21.15.62:0/1801016116 shutdown_connections 2024-03-23T11:26:16.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.269+0000 7ff403400640 1 -- 172.21.15.62:0/1801016116 wait complete. 2024-03-23T11:26:16.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.271+0000 7ff403400640 1 Processor -- start 2024-03-23T11:26:16.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.272+0000 7ff403400640 1 -- start start 2024-03-23T11:26:16.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff403400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 0x7ff3fc15d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff403400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3fc1478a0 0x7ff3fc1f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff403400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc0d28d0 0x7ff3fc0d2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff403400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc0d3290 con 0x7ff3e0003970 2024-03-23T11:26:16.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff403400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc1f5ba0 con 0x7ff3fc0d28d0 2024-03-23T11:26:16.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff403400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc1f6a00 con 0x7ff3fc1478a0 2024-03-23T11:26:16.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff400a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 0x7ff3fc15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:16.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff400a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 0x7ff3fc15d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43046/0 (socket says 172.21.15.62:43046) 2024-03-23T11:26:16.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff400a00640 1 -- 172.21.15.62:0/64752704 learned_addr learned my addr 172.21.15.62:0/64752704 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:16.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff400a00640 1 -- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3fc1478a0 msgr2=0x7ff3fc1f1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff400a00640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3fc1478a0 0x7ff3fc1f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff400a00640 1 -- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc0d28d0 msgr2=0x7ff3fc0d2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff400a00640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc0d28d0 0x7ff3fc0d2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.273+0000 7ff400a00640 1 -- 172.21.15.62:0/64752704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc1f2490 con 0x7ff3e0003970 2024-03-23T11:26:16.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.274+0000 7ff400a00640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 0x7ff3fc15d390 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00c210 tx=0x7ff3ec02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:16.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.274+0000 7ff3f8c00640 1 -- 172.21.15.62:0/64752704 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec03c040 con 0x7ff3e0003970 2024-03-23T11:26:16.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.274+0000 7ff3f8c00640 1 -- 172.21.15.62:0/64752704 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ec004bf0 con 0x7ff3e0003970 2024-03-23T11:26:16.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.274+0000 7ff403400640 1 -- 172.21.15.62:0/64752704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3fc1f5df0 con 0x7ff3e0003970 2024-03-23T11:26:16.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.274+0000 7ff3f8c00640 1 -- 172.21.15.62:0/64752704 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec007070 con 0x7ff3e0003970 2024-03-23T11:26:16.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.274+0000 7ff403400640 1 -- 172.21.15.62:0/64752704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3fc1f63b0 con 0x7ff3e0003970 2024-03-23T11:26:16.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.275+0000 7ff403400640 1 -- 172.21.15.62:0/64752704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3fc066c90 con 0x7ff3e0003970 2024-03-23T11:26:16.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.276+0000 7ff3f8c00640 1 -- 172.21.15.62:0/64752704 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff3ec032070 con 0x7ff3e0003970 2024-03-23T11:26:16.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.277+0000 7ff3f8c00640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff3dc06a7a0 0x7ff3dc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.277+0000 7ff3f8c00640 1 -- 172.21.15.62:0/64752704 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5029+0+0 (secure 0 0 0) 0x7ff3ec0aa200 con 0x7ff3e0003970 2024-03-23T11:26:16.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.278+0000 7ff3fbe00640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff3dc06a7a0 0x7ff3dc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:16.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.279+0000 7ff3fbe00640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff3dc06a7a0 0x7ff3dc06cc60 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0d3620 tx=0x7ff3e8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:16.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.282+0000 7ff3f8c00640 1 -- 172.21.15.62:0/64752704 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff3ec0758b0 con 0x7ff3e0003970 2024-03-23T11:26:16.407 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:26:16.414 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:26:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.555+0000 7ff403400640 1 -- 172.21.15.62:0/64752704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) v1 -- 0x7ff3fc0d3740 con 0x7ff3e0003970 2024-03-23T11:26:16.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.571+0000 7ff3f8c00640 1 -- 172.21.15.62:0/64752704 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v354) v1 ==== 166+0+0 (secure 0 0 0) 0x7ff3ec078ed0 con 0x7ff3e0003970 2024-03-23T11:26:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 -- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff3dc06a7a0 msgr2=0x7ff3dc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff3dc06a7a0 0x7ff3dc06cc60 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0d3620 tx=0x7ff3e8008040 comp rx=0 tx=0).stop 2024-03-23T11:26:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 -- 172.21.15.62:0/64752704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 msgr2=0x7ff3fc15d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 0x7ff3fc15d390 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00c210 tx=0x7ff3ec02e070 comp rx=0 tx=0).stop 2024-03-23T11:26:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 -- 172.21.15.62:0/64752704 shutdown_connections 2024-03-23T11:26:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff3dc06a7a0 0x7ff3dc06cc60 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc0d28d0 0x7ff3fc0d2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3fc1478a0 0x7ff3fc1f1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 --2- 172.21.15.62:0/64752704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3e0003970 0x7ff3fc15d390 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 -- 172.21.15.62:0/64752704 >> 172.21.15.62:0/64752704 conn(0x7ff3fc071570 msgr2=0x7ff3fc0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 -- 172.21.15.62:0/64752704 shutdown_connections 2024-03-23T11:26:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.575+0000 7ff403400640 1 -- 172.21.15.62:0/64752704 wait complete. 2024-03-23T11:26:16.590 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-03-23T11:26:16.590 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-03-23T11:26:16.590 INFO:tasks.cephfs_test_runner:test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-23T11:26:16.591 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-03-23T11:26:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.853+0000 7fbbf9c00640 1 Processor -- start 2024-03-23T11:26:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.854+0000 7fbbf9c00640 1 -- start start 2024-03-23T11:26:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.855+0000 7fbbf9c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbbf4070090 0x7fbbf4070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.855+0000 7fbbf9c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf40cd140 con 0x7fbbf4143dd0 2024-03-23T11:26:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.855+0000 7fbbf9c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf40cd3e0 con 0x7fbbf40b2820 2024-03-23T11:26:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.855+0000 7fbbf9c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf4070a60 con 0x7fbbf4070090 2024-03-23T11:26:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.856+0000 7fbbf3e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbbf4143dd0 0x7fbbf415abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33298/0 (socket says 172.21.15.62:33298) 2024-03-23T11:26:16.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.856+0000 7fbbf3e00640 1 -- 172.21.15.62:0/1330247676 learned_addr learned my addr 172.21.15.62:0/1330247676 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:16.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.856+0000 7fbbebe00640 1 --2- 172.21.15.62:0/1330247676 >> v2:172.21.15.120:3300/0 conn(0x7fbbf4070090 0x7fbbf4070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:16.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.857+0000 7fbbf3400640 1 -- 172.21.15.62:0/1330247676 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246452931 0 0) 0x7fbbf40cd140 con 0x7fbbf4143dd0 2024-03-23T11:26:16.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.857+0000 7fbbf3400640 1 -- 172.21.15.62:0/1330247676 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd8003520 con 0x7fbbf4143dd0 2024-03-23T11:26:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.857+0000 7fbbebe00640 1 -- 172.21.15.62:0/1330247676 >> v1:172.21.15.120:6789/0 conn(0x7fbbf40b2820 legacy=0x7fbbf40b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.857+0000 7fbbebe00640 1 -- 172.21.15.62:0/1330247676 >> v1:172.21.15.62:6789/0 conn(0x7fbbf4143dd0 legacy=0x7fbbf415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.857+0000 7fbbebe00640 1 -- 172.21.15.62:0/1330247676 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf40cd6c0 con 0x7fbbf4070090 2024-03-23T11:26:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.859+0000 7fbbebe00640 1 --2- 172.21.15.62:0/1330247676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbf4070090 0x7fbbf4070520 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0097a0 tx=0x7fbbdc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b75671cc3b56c44 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.859+0000 7fbbf3400640 1 -- 172.21.15.62:0/1330247676 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc030610 con 0x7fbbf4070090 2024-03-23T11:26:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.859+0000 7fbbf3400640 1 -- 172.21.15.62:0/1330247676 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbdc030770 con 0x7fbbf4070090 2024-03-23T11:26:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.859+0000 7fbbf3400640 1 -- 172.21.15.62:0/1330247676 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc030b00 con 0x7fbbf4070090 2024-03-23T11:26:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.860+0000 7fbbf9c00640 1 -- 172.21.15.62:0/1330247676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbf4070090 msgr2=0x7fbbf4070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.860+0000 7fbbf9c00640 1 --2- 172.21.15.62:0/1330247676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbf4070090 0x7fbbf4070520 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0097a0 tx=0x7fbbdc02b960 comp rx=0 tx=0).stop 2024-03-23T11:26:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.860+0000 7fbbf9c00640 1 -- 172.21.15.62:0/1330247676 shutdown_connections 2024-03-23T11:26:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.860+0000 7fbbf9c00640 1 --2- 172.21.15.62:0/1330247676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbf4070090 0x7fbbf4070520 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.860+0000 7fbbf9c00640 1 -- 172.21.15.62:0/1330247676 >> 172.21.15.62:0/1330247676 conn(0x7fbbf4066050 msgr2=0x7fbbf4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.860+0000 7fbbf9c00640 1 -- 172.21.15.62:0/1330247676 shutdown_connections 2024-03-23T11:26:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.860+0000 7fbbf9c00640 1 -- 172.21.15.62:0/1330247676 wait complete. 2024-03-23T11:26:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.862+0000 7fbbf9c00640 1 Processor -- start 2024-03-23T11:26:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.862+0000 7fbbf9c00640 1 -- start start 2024-03-23T11:26:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.862+0000 7fbbf9c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4070090 0x7fbbf415a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf9c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbf40b2820 0x7fbbf40c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf9c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbf4143dd0 0x7fbbf40c4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf9c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf40c52a0 con 0x7fbbf4070090 2024-03-23T11:26:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf9c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf40ca620 con 0x7fbbf4143dd0 2024-03-23T11:26:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf9c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf40ca900 con 0x7fbbf40b2820 2024-03-23T11:26:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4070090 0x7fbbf415a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4070090 0x7fbbf415a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43052/0 (socket says 172.21.15.62:43052) 2024-03-23T11:26:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf3e00640 1 -- 172.21.15.62:0/2769270879 learned_addr learned my addr 172.21.15.62:0/2769270879 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf3e00640 1 -- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbf40b2820 msgr2=0x7fbbf40c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf3e00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbf40b2820 0x7fbbf40c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf3e00640 1 -- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbf4143dd0 msgr2=0x7fbbf40c4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf3e00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbf4143dd0 0x7fbbf40c4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.863+0000 7fbbf3e00640 1 -- 172.21.15.62:0/2769270879 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbdc009400 con 0x7fbbf4070090 2024-03-23T11:26:16.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.864+0000 7fbbf3e00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4070090 0x7fbbf415a7e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400c410 tx=0x7fbbe400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:16.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.864+0000 7fbbf0c00640 1 -- 172.21.15.62:0/2769270879 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4005ce0 con 0x7fbbf4070090 2024-03-23T11:26:16.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.864+0000 7fbbf0c00640 1 -- 172.21.15.62:0/2769270879 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe4005e40 con 0x7fbbf4070090 2024-03-23T11:26:16.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.864+0000 7fbbf0c00640 1 -- 172.21.15.62:0/2769270879 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe40072f0 con 0x7fbbf4070090 2024-03-23T11:26:16.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.864+0000 7fbbf9c00640 1 -- 172.21.15.62:0/2769270879 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf40cabe0 con 0x7fbbf4070090 2024-03-23T11:26:16.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.864+0000 7fbbf9c00640 1 -- 172.21.15.62:0/2769270879 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf40cb1a0 con 0x7fbbf4070090 2024-03-23T11:26:16.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.865+0000 7fbbf9c00640 1 -- 172.21.15.62:0/2769270879 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf4066bb0 con 0x7fbbf4070090 2024-03-23T11:26:16.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.866+0000 7fbbf0c00640 1 -- 172.21.15.62:0/2769270879 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbbe4007450 con 0x7fbbf4070090 2024-03-23T11:26:16.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.867+0000 7fbbf0c00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbd006a7a0 0x7fbbd006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:16.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.867+0000 7fbbf0c00640 1 -- 172.21.15.62:0/2769270879 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fbbe40881c0 con 0x7fbbf4070090 2024-03-23T11:26:16.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.867+0000 7fbbebe00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbd006a7a0 0x7fbbd006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:16.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.869+0000 7fbbebe00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbd006a7a0 0x7fbbd006cc60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc036000 tx=0x7fbbdc03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:16.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:16.871+0000 7fbbf0c00640 1 -- 172.21.15.62:0/2769270879 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbbe40573c0 con 0x7fbbf4070090 2024-03-23T11:26:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.141+0000 7fbbf9c00640 1 -- 172.21.15.62:0/2769270879 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) v1 -- 0x7fbbf40cc440 con 0x7fbbf4070090 2024-03-23T11:26:17.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.581+0000 7fbbf0c00640 1 -- 172.21.15.62:0/2769270879 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v355) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbbe4056f50 con 0x7fbbf4070090 2024-03-23T11:26:17.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.584+0000 7fbbf9c00640 1 -- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbd006a7a0 msgr2=0x7fbbd006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:17.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.584+0000 7fbbf9c00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbd006a7a0 0x7fbbd006cc60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc036000 tx=0x7fbbdc03b040 comp rx=0 tx=0).stop 2024-03-23T11:26:17.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.585+0000 7fbbf9c00640 1 -- 172.21.15.62:0/2769270879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4070090 msgr2=0x7fbbf415a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:17.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.585+0000 7fbbf9c00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4070090 0x7fbbf415a7e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400c410 tx=0x7fbbe400c8d0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.585+0000 7fbbf9c00640 1 -- 172.21.15.62:0/2769270879 shutdown_connections 2024-03-23T11:26:17.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.585+0000 7fbbf9c00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbbd006a7a0 0x7fbbd006cc60 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.585+0000 7fbbf9c00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbf4143dd0 0x7fbbf40c4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.585+0000 7fbbf9c00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbbf40b2820 0x7fbbf40c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.585+0000 7fbbf9c00640 1 --2- 172.21.15.62:0/2769270879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4070090 0x7fbbf415a7e0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.585+0000 7fbbf9c00640 1 -- 172.21.15.62:0/2769270879 >> 172.21.15.62:0/2769270879 conn(0x7fbbf4066050 msgr2=0x7fbbf40afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:17.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.585+0000 7fbbf9c00640 1 -- 172.21.15.62:0/2769270879 shutdown_connections 2024-03-23T11:26:17.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.585+0000 7fbbf9c00640 1 -- 172.21.15.62:0/2769270879 wait complete. 2024-03-23T11:26:17.605 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T11:26:17.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.825+0000 7f5c19c00640 1 Processor -- start 2024-03-23T11:26:17.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.825+0000 7f5c19c00640 1 -- start start 2024-03-23T11:26:17.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.827+0000 7f5c19c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c14143dd0 0x7f5c1415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:17.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.827+0000 7f5c19c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5c14070090 0x7f5c140704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:17.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.827+0000 7f5c19c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5c140b2820 0x7f5c140b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:17.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.827+0000 7f5c19c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c14070a30 con 0x7f5c14070090 2024-03-23T11:26:17.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.827+0000 7f5c19c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c140cd140 con 0x7f5c140b2820 2024-03-23T11:26:17.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.827+0000 7f5c19c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c140b3180 con 0x7f5c14143dd0 2024-03-23T11:26:17.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.827+0000 7f5c0be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5c14070090 0x7f5c140704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:17.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.827+0000 7f5c0be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5c14070090 0x7f5c140704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32910/0 (socket says 172.21.15.62:32910) 2024-03-23T11:26:17.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.827+0000 7f5c0be00640 1 -- 172.21.15.62:0/1441596997 learned_addr learned my addr 172.21.15.62:0/1441596997 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:17.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.828+0000 7f5c0be00640 1 -- 172.21.15.62:0/1441596997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c14143dd0 msgr2=0x7f5c1415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:17.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.828+0000 7f5c19200640 1 --2- 172.21.15.62:0/1441596997 >> v2:172.21.15.120:3300/0 conn(0x7f5c140b2820 0x7f5c140b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:17.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.828+0000 7f5c13e00640 1 --2- 172.21.15.62:0/1441596997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c14143dd0 0x7f5c1415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:17.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.828+0000 7f5c0be00640 1 --2- 172.21.15.62:0/1441596997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c14143dd0 0x7f5c1415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.828+0000 7f5c0be00640 1 -- 172.21.15.62:0/1441596997 >> v2:172.21.15.120:3300/0 conn(0x7f5c140b2820 msgr2=0x7f5c140b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:17.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.828+0000 7f5c0be00640 1 --2- 172.21.15.62:0/1441596997 >> v2:172.21.15.120:3300/0 conn(0x7f5c140b2820 0x7f5c140b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.828+0000 7f5c0be00640 1 -- 172.21.15.62:0/1441596997 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c140cd3e0 con 0x7f5c14070090 2024-03-23T11:26:17.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.828+0000 7f5c19200640 1 --2- 172.21.15.62:0/1441596997 >> v2:172.21.15.120:3300/0 conn(0x7f5c140b2820 0x7f5c140b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:17.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.828+0000 7f5c13e00640 1 --2- 172.21.15.62:0/1441596997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c14143dd0 0x7f5c1415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:17.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.829+0000 7f5c0be00640 1 --2- 172.21.15.62:0/1441596997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14070090 0x7f5c140704f0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f5c14071500 tx=0x7f5bfc02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7184892da283ac56 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:17.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.830+0000 7f5c13400640 1 -- 172.21.15.62:0/1441596997 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc031880 con 0x7f5c14070090 2024-03-23T11:26:17.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.830+0000 7f5c13400640 1 -- 172.21.15.62:0/1441596997 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bfc0319e0 con 0x7f5c14070090 2024-03-23T11:26:17.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.830+0000 7f5c13400640 1 -- 172.21.15.62:0/1441596997 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc031d70 con 0x7f5c14070090 2024-03-23T11:26:17.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.830+0000 7f5c19c00640 1 -- 172.21.15.62:0/1441596997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14070090 msgr2=0x7f5c140704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:17.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.830+0000 7f5c19c00640 1 --2- 172.21.15.62:0/1441596997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14070090 0x7f5c140704f0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f5c14071500 tx=0x7f5bfc02b920 comp rx=0 tx=0).stop 2024-03-23T11:26:17.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.830+0000 7f5c19c00640 1 -- 172.21.15.62:0/1441596997 shutdown_connections 2024-03-23T11:26:17.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.830+0000 7f5c19c00640 1 --2- 172.21.15.62:0/1441596997 >> v2:172.21.15.120:3300/0 conn(0x7f5c140b2820 0x7f5c140b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.830+0000 7f5c19c00640 1 --2- 172.21.15.62:0/1441596997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14070090 0x7f5c140704f0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.830+0000 7f5c19c00640 1 --2- 172.21.15.62:0/1441596997 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c14143dd0 0x7f5c1415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.830+0000 7f5c19c00640 1 -- 172.21.15.62:0/1441596997 >> 172.21.15.62:0/1441596997 conn(0x7f5c14066050 msgr2=0x7f5c14069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:17.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.830+0000 7f5c19c00640 1 -- 172.21.15.62:0/1441596997 shutdown_connections 2024-03-23T11:26:17.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.831+0000 7f5c19c00640 1 -- 172.21.15.62:0/1441596997 wait complete. 2024-03-23T11:26:17.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.832+0000 7f5c19c00640 1 Processor -- start 2024-03-23T11:26:17.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.833+0000 7f5c19c00640 1 -- start start 2024-03-23T11:26:17.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.833+0000 7f5c19c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c14070090 0x7f5c14135870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:17.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.833+0000 7f5c19c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c140b2820 0x7f5c140c3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:17.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14143dd0 0x7f5c1415b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:17.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c14133dd0 con 0x7f5c14143dd0 2024-03-23T11:26:17.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c14135db0 con 0x7f5c140b2820 2024-03-23T11:26:17.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c14133150 con 0x7f5c14070090 2024-03-23T11:26:17.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14143dd0 0x7f5c1415b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:17.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14143dd0 0x7f5c1415b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32914/0 (socket says 172.21.15.62:32914) 2024-03-23T11:26:17.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19200640 1 -- 172.21.15.62:0/203211822 learned_addr learned my addr 172.21.15.62:0/203211822 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:17.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c13e00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c14070090 0x7f5c14135870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:17.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c0be00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c140b2820 0x7f5c140c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:17.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19200640 1 -- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c14070090 msgr2=0x7f5c14135870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:17.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19200640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c14070090 0x7f5c14135870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19200640 1 -- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c140b2820 msgr2=0x7f5c140c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:17.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19200640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c140b2820 0x7f5c140c3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:17.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c19200640 1 -- 172.21.15.62:0/203211822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc0094d0 con 0x7f5c14143dd0 2024-03-23T11:26:17.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.834+0000 7f5c0be00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c140b2820 0x7f5c140c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:17.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.835+0000 7f5c19200640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14143dd0 0x7f5c1415b160 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c00cef0 tx=0x7f5c0c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:17.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.835+0000 7f5c10c00640 1 -- 172.21.15.62:0/203211822 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c00d840 con 0x7f5c14143dd0 2024-03-23T11:26:17.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.835+0000 7f5c10c00640 1 -- 172.21.15.62:0/203211822 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c00d9c0 con 0x7f5c14143dd0 2024-03-23T11:26:17.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.835+0000 7f5c10c00640 1 -- 172.21.15.62:0/203211822 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c00dd70 con 0x7f5c14143dd0 2024-03-23T11:26:17.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.835+0000 7f5c19c00640 1 -- 172.21.15.62:0/203211822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c14133430 con 0x7f5c14143dd0 2024-03-23T11:26:17.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.835+0000 7f5c19c00640 1 -- 172.21.15.62:0/203211822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c141339f0 con 0x7f5c14143dd0 2024-03-23T11:26:17.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.836+0000 7f5c19c00640 1 -- 172.21.15.62:0/203211822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c14133f10 con 0x7f5c14143dd0 2024-03-23T11:26:17.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.837+0000 7f5c10c00640 1 -- 172.21.15.62:0/203211822 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5c0c01e3b0 con 0x7f5c14143dd0 2024-03-23T11:26:17.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.838+0000 7f5c10c00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 0x7f5bf006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:17.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.838+0000 7f5c10c00640 1 -- 172.21.15.62:0/203211822 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f5c0c089590 con 0x7f5c14143dd0 2024-03-23T11:26:17.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.839+0000 7f5c13e00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 0x7f5bf006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:17.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.840+0000 7f5c13e00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 0x7f5bf006cc60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5c140ae410 tx=0x7f5c04008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:17.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:17.843+0000 7f5c10c00640 1 -- 172.21.15.62:0/203211822 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c14133f10 con 0x7f5c14143dd0 2024-03-23T11:26:18.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.045+0000 7f5c19c00640 1 -- 172.21.15.62:0/203211822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5c14133f10 con 0x7f5c14143dd0 2024-03-23T11:26:18.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.050+0000 7f5c10c00640 1 -- 172.21.15.62:0/203211822 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5c14133f10 con 0x7f5c14143dd0 2024-03-23T11:26:18.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.053+0000 7f5c19c00640 1 -- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 msgr2=0x7f5bf006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:18.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.053+0000 7f5c19c00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 0x7f5bf006cc60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5c140ae410 tx=0x7f5c04008040 comp rx=0 tx=0).stop 2024-03-23T11:26:18.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.053+0000 7f5c19c00640 1 -- 172.21.15.62:0/203211822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14143dd0 msgr2=0x7f5c1415b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:18.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.053+0000 7f5c19c00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14143dd0 0x7f5c1415b160 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c00cef0 tx=0x7f5c0c00a710 comp rx=0 tx=0).stop 2024-03-23T11:26:18.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.053+0000 7f5c19c00640 1 -- 172.21.15.62:0/203211822 shutdown_connections 2024-03-23T11:26:18.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.053+0000 7f5c19c00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 0x7f5bf006cc60 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:18.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.053+0000 7f5c19c00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14143dd0 0x7f5c1415b160 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:18.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.053+0000 7f5c19c00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c140b2820 0x7f5c140c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:18.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.053+0000 7f5c19c00640 1 --2- 172.21.15.62:0/203211822 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c14070090 0x7f5c14135870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:18.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.053+0000 7f5c19c00640 1 -- 172.21.15.62:0/203211822 >> 172.21.15.62:0/203211822 conn(0x7f5c14066050 msgr2=0x7f5c140cb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:18.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.053+0000 7f5c19c00640 1 -- 172.21.15.62:0/203211822 shutdown_connections 2024-03-23T11:26:18.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.054+0000 7f5c19c00640 1 -- 172.21.15.62:0/203211822 wait complete. 2024-03-23T11:26:18.066 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:26:18.066 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:26:18.180 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:26:18.285 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:26:18.285 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:26:18.285 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:26:18.346 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:26:18.346 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:26:18.346 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:26:18.403 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:26:18.403 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:26:18.403 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:26:18.464 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:26:18.465 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:26:18.465 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:26:18.523 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:26:18.523 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T11:26:18.523 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:26:18.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.755+0000 7f349dc00640 1 Processor -- start 2024-03-23T11:26:18.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.756+0000 7f349dc00640 1 -- start start 2024-03-23T11:26:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.757+0000 7f349dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f349815fcf0 0x7f34980d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.757+0000 7f349dc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f34980b6dc0 0x7f3498160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.757+0000 7f349dc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3498160dc0 con 0x7f3498160420 2024-03-23T11:26:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.757+0000 7f349dc00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34980d3540 con 0x7f34980b6dc0 2024-03-23T11:26:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.757+0000 7f349dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34980ce510 con 0x7f349815fcf0 2024-03-23T11:26:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.757+0000 7f349d200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f34980b6dc0 0x7f3498160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.758+0000 7f349d200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f34980b6dc0 0x7f3498160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32916/0 (socket says 172.21.15.62:32916) 2024-03-23T11:26:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.758+0000 7f349d200640 1 -- 172.21.15.62:0/3491329472 learned_addr learned my addr 172.21.15.62:0/3491329472 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.758+0000 7f3497e00640 1 --2- 172.21.15.62:0/3491329472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f349815fcf0 0x7f34980d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.758+0000 7f349d200640 1 -- 172.21.15.62:0/3491329472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f349815fcf0 msgr2=0x7f34980d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.758+0000 7f349d200640 1 --2- 172.21.15.62:0/3491329472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f349815fcf0 0x7f34980d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.758+0000 7f349d200640 1 -- 172.21.15.62:0/3491329472 >> v1:172.21.15.120:6789/0 conn(0x7f3498160420 legacy=0x7f34980d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.758+0000 7f349d200640 1 -- 172.21.15.62:0/3491329472 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34980cdc70 con 0x7f34980b6dc0 2024-03-23T11:26:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.759+0000 7f349d200640 1 --2- 172.21.15.62:0/3491329472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34980b6dc0 0x7f3498160880 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f3490009e50 tx=0x7f349002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7615af8d3b586dd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:18.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.760+0000 7f3496a00640 1 -- 172.21.15.62:0/3491329472 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349002c950 con 0x7f34980b6dc0 2024-03-23T11:26:18.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.760+0000 7f3496a00640 1 -- 172.21.15.62:0/3491329472 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349002cab0 con 0x7f34980b6dc0 2024-03-23T11:26:18.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.760+0000 7f3496a00640 1 -- 172.21.15.62:0/3491329472 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349002ce40 con 0x7f34980b6dc0 2024-03-23T11:26:18.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.760+0000 7f349dc00640 1 -- 172.21.15.62:0/3491329472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34980b6dc0 msgr2=0x7f3498160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:18.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.760+0000 7f349dc00640 1 --2- 172.21.15.62:0/3491329472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34980b6dc0 0x7f3498160880 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f3490009e50 tx=0x7f349002bdb0 comp rx=0 tx=0).stop 2024-03-23T11:26:18.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.761+0000 7f349dc00640 1 -- 172.21.15.62:0/3491329472 shutdown_connections 2024-03-23T11:26:18.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.761+0000 7f349dc00640 1 --2- 172.21.15.62:0/3491329472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34980b6dc0 0x7f3498160880 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:18.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.761+0000 7f349dc00640 1 --2- 172.21.15.62:0/3491329472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f349815fcf0 0x7f34980d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:18.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.761+0000 7f349dc00640 1 -- 172.21.15.62:0/3491329472 >> 172.21.15.62:0/3491329472 conn(0x7f3498089f60 msgr2=0x7f34980d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:18.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.761+0000 7f349dc00640 1 -- 172.21.15.62:0/3491329472 shutdown_connections 2024-03-23T11:26:18.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.761+0000 7f349dc00640 1 -- 172.21.15.62:0/3491329472 wait complete. 2024-03-23T11:26:18.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.762+0000 7f349dc00640 1 Processor -- start 2024-03-23T11:26:18.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.763+0000 7f349dc00640 1 -- start start 2024-03-23T11:26:18.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.763+0000 7f349dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34980b6dc0 0x7f3498150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:18.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.763+0000 7f349dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f349815fcf0 0x7f349814d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:18.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.763+0000 7f349dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3498160420 0x7f349814fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:18.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f349dc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349814d7d0 con 0x7f34980b6dc0 2024-03-23T11:26:18.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f349dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349814da50 con 0x7f3498160420 2024-03-23T11:26:18.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f349dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349814dd30 con 0x7f349815fcf0 2024-03-23T11:26:18.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f3497e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34980b6dc0 0x7f3498150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:18.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f3497e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34980b6dc0 0x7f3498150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32928/0 (socket says 172.21.15.62:32928) 2024-03-23T11:26:18.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f3497e00640 1 -- 172.21.15.62:0/2197996799 learned_addr learned my addr 172.21.15.62:0/2197996799 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:18.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f3497e00640 1 -- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f349815fcf0 msgr2=0x7f349814d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:18.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f3497e00640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f349815fcf0 0x7f349814d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:18.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f3497e00640 1 -- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3498160420 msgr2=0x7f349814fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:18.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f3497e00640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3498160420 0x7f349814fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:18.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f3497e00640 1 -- 172.21.15.62:0/2197996799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3490009b30 con 0x7f34980b6dc0 2024-03-23T11:26:18.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.764+0000 7f3497e00640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34980b6dc0 0x7f3498150f40 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f348800d5f0 tx=0x7f348800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:18.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.765+0000 7f348fe00640 1 -- 172.21.15.62:0/2197996799 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34880125f0 con 0x7f34980b6dc0 2024-03-23T11:26:18.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.765+0000 7f348fe00640 1 -- 172.21.15.62:0/2197996799 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3488012750 con 0x7f34980b6dc0 2024-03-23T11:26:18.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.765+0000 7f349dc00640 1 -- 172.21.15.62:0/2197996799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349814e010 con 0x7f34980b6dc0 2024-03-23T11:26:18.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.765+0000 7f348fe00640 1 -- 172.21.15.62:0/2197996799 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488012b00 con 0x7f34980b6dc0 2024-03-23T11:26:18.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.765+0000 7f349dc00640 1 -- 172.21.15.62:0/2197996799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349814e5d0 con 0x7f34980b6dc0 2024-03-23T11:26:18.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.766+0000 7f349dc00640 1 -- 172.21.15.62:0/2197996799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3498066bb0 con 0x7f34980b6dc0 2024-03-23T11:26:18.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.767+0000 7f348fe00640 1 -- 172.21.15.62:0/2197996799 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3488012c60 con 0x7f34980b6dc0 2024-03-23T11:26:18.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.768+0000 7f348fe00640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347406a7a0 0x7f347406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:18.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.768+0000 7f348fe00640 1 -- 172.21.15.62:0/2197996799 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3488089010 con 0x7f34980b6dc0 2024-03-23T11:26:18.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.769+0000 7f3497400640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347406a7a0 0x7f347406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:18.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.770+0000 7f3497400640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347406a7a0 0x7f347406cc60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f34980a05c0 tx=0x7f34800079b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:18.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:18.772+0000 7f348fe00640 1 -- 172.21.15.62:0/2197996799 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f34880546d0 con 0x7f34980b6dc0 2024-03-23T11:26:19.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.013+0000 7f349dc00640 1 -- 172.21.15.62:0/2197996799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34980cea40 con 0x7f34980b6dc0 2024-03-23T11:26:19.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.014+0000 7f348fe00640 1 -- 172.21.15.62:0/2197996799 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f3488057cf0 con 0x7f34980b6dc0 2024-03-23T11:26:19.008 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:19.008 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":52,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/29171509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":29171509},{"type":"v1","addr":"172.21.15.120:6837","nonce":29171509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":5513,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2531968231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2531968231},{"type":"v1","addr":"172.21.15.120:6835","nonce":2531968231}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":6603,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3572189997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3572189997},{"type":"v1","addr":"172.21.15.62:6833","nonce":3572189997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38},{"gid":6609,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2482513142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2482513142},{"type":"v1","addr":"172.21.15.62:6835","nonce":2482513142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":38}],"filesystems":[]} 2024-03-23T11:26:19.008 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 52 2024-03-23T11:26:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 -- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347406a7a0 msgr2=0x7f347406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347406a7a0 0x7f347406cc60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f34980a05c0 tx=0x7f34800079b0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 -- 172.21.15.62:0/2197996799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34980b6dc0 msgr2=0x7f3498150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34980b6dc0 0x7f3498150f40 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f348800d5f0 tx=0x7f348800dab0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 -- 172.21.15.62:0/2197996799 shutdown_connections 2024-03-23T11:26:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f347406a7a0 0x7f347406cc60 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3498160420 0x7f349814fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f349815fcf0 0x7f349814d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 --2- 172.21.15.62:0/2197996799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34980b6dc0 0x7f3498150f40 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 -- 172.21.15.62:0/2197996799 >> 172.21.15.62:0/2197996799 conn(0x7f3498089f60 msgr2=0x7f34980cb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 -- 172.21.15.62:0/2197996799 shutdown_connections 2024-03-23T11:26:19.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.017+0000 7f349dc00640 1 -- 172.21.15.62:0/2197996799 wait complete. 2024-03-23T11:26:19.030 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T11:26:19.031 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T11:26:19.031 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T11:26:19.031 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:26:19.031 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T11:26:19.031 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T11:26:19.031 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T11:26:19.032 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:26:19.032 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T11:26:19.032 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T11:26:19.032 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T11:26:19.032 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:26:19.032 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T11:26:19.032 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T11:26:19.033 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T11:26:19.033 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:26:19.052 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T11:26:19.053 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T11:26:19.056 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T11:26:19.056 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T11:26:19.058 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T11:26:19.058 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T11:26:19.066 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T11:26:19.066 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T11:26:19.104 INFO:tasks.ceph.mds.b:Started 2024-03-23T11:26:19.105 DEBUG:teuthology.parallel:result is None 2024-03-23T11:26:19.111 INFO:tasks.ceph.mds.d:Started 2024-03-23T11:26:19.111 DEBUG:teuthology.parallel:result is None 2024-03-23T11:26:19.113 INFO:tasks.ceph.mds.a:Started 2024-03-23T11:26:19.113 DEBUG:teuthology.parallel:result is None 2024-03-23T11:26:19.120 INFO:tasks.ceph.mds.c:Started 2024-03-23T11:26:19.120 DEBUG:teuthology.parallel:result is None 2024-03-23T11:26:19.121 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T11:26:19.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-23T11:26:19.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-23T11:26:19.264 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-23T11:26:19.266 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-23T11:26:19.272 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:26:19.279 INFO:tasks.ceph.mds.c.smithi062.stdout:starting mds.c at 2024-03-23T11:26:19.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.322+0000 7f8cea400640 1 Processor -- start 2024-03-23T11:26:19.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.322+0000 7f8cea400640 1 -- start start 2024-03-23T11:26:19.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.323+0000 7f8cea400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8ce415ca50 0x7f8ce40b5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:19.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.323+0000 7f8cea400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ce415ce30 0x7f8ce40b8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:19.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.323+0000 7f8cea400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8ce40d0ba0 0x7f8ce40cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:19.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.323+0000 7f8cea400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce40b5e80 con 0x7f8ce40d0ba0 2024-03-23T11:26:19.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.323+0000 7f8cea400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce40b87c0 con 0x7f8ce415ca50 2024-03-23T11:26:19.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.323+0000 7f8cea400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce40b60b0 con 0x7f8ce415ce30 2024-03-23T11:26:19.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.323+0000 7f8ce3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8ce40d0ba0 0x7f8ce40cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:19.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.324+0000 7f8ce3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8ce40d0ba0 0x7f8ce40cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32960/0 (socket says 172.21.15.62:32960) 2024-03-23T11:26:19.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.324+0000 7f8ce3e00640 1 -- 172.21.15.62:0/4248744855 learned_addr learned my addr 172.21.15.62:0/4248744855 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:19.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.324+0000 7f8ce3400640 1 --2- 172.21.15.62:0/4248744855 >> v2:172.21.15.120:3300/0 conn(0x7f8ce415ca50 0x7f8ce40b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:19.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.324+0000 7f8cdaa00640 1 --2- 172.21.15.62:0/4248744855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ce415ce30 0x7f8ce40b8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:19.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.324+0000 7f8ce3e00640 1 -- 172.21.15.62:0/4248744855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ce415ce30 msgr2=0x7f8ce40b8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:19.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.324+0000 7f8ce3e00640 1 --2- 172.21.15.62:0/4248744855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ce415ce30 0x7f8ce40b8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.324+0000 7f8ce3e00640 1 -- 172.21.15.62:0/4248744855 >> v2:172.21.15.120:3300/0 conn(0x7f8ce415ca50 msgr2=0x7f8ce40b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:19.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.324+0000 7f8ce3e00640 1 --2- 172.21.15.62:0/4248744855 >> v2:172.21.15.120:3300/0 conn(0x7f8ce415ca50 0x7f8ce40b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.324+0000 7f8ce3e00640 1 -- 172.21.15.62:0/4248744855 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce40b6390 con 0x7f8ce40d0ba0 2024-03-23T11:26:19.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.324+0000 7f8ce3400640 1 --2- 172.21.15.62:0/4248744855 >> v2:172.21.15.120:3300/0 conn(0x7f8ce415ca50 0x7f8ce40b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:26:19.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.325+0000 7f8ce3e00640 1 --2- 172.21.15.62:0/4248744855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ce40d0ba0 0x7f8ce40cf7e0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40d38a0 tx=0x7f8cdc02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bee482c21d01114 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:19.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.325+0000 7f8ce2a00640 1 -- 172.21.15.62:0/4248744855 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc0317d0 con 0x7f8ce40d0ba0 2024-03-23T11:26:19.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.325+0000 7f8ce2a00640 1 -- 172.21.15.62:0/4248744855 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cdc031930 con 0x7f8ce40d0ba0 2024-03-23T11:26:19.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.325+0000 7f8ce2a00640 1 -- 172.21.15.62:0/4248744855 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc031cc0 con 0x7f8ce40d0ba0 2024-03-23T11:26:19.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.326+0000 7f8cea400640 1 -- 172.21.15.62:0/4248744855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ce40d0ba0 msgr2=0x7f8ce40cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:19.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.326+0000 7f8cea400640 1 --2- 172.21.15.62:0/4248744855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ce40d0ba0 0x7f8ce40cf7e0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40d38a0 tx=0x7f8cdc02b870 comp rx=0 tx=0).stop 2024-03-23T11:26:19.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.326+0000 7f8cea400640 1 -- 172.21.15.62:0/4248744855 shutdown_connections 2024-03-23T11:26:19.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.326+0000 7f8cea400640 1 --2- 172.21.15.62:0/4248744855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ce40d0ba0 0x7f8ce40cf7e0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.326+0000 7f8cea400640 1 --2- 172.21.15.62:0/4248744855 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ce415ce30 0x7f8ce40b8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.326+0000 7f8cea400640 1 --2- 172.21.15.62:0/4248744855 >> v2:172.21.15.120:3300/0 conn(0x7f8ce415ca50 0x7f8ce40b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.326+0000 7f8cea400640 1 -- 172.21.15.62:0/4248744855 >> 172.21.15.62:0/4248744855 conn(0x7f8ce40717b0 msgr2=0x7f8ce4071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:19.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.326+0000 7f8cea400640 1 -- 172.21.15.62:0/4248744855 shutdown_connections 2024-03-23T11:26:19.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.326+0000 7f8cea400640 1 -- 172.21.15.62:0/4248744855 wait complete. 2024-03-23T11:26:19.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.328+0000 7f8cea400640 1 Processor -- start 2024-03-23T11:26:19.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.328+0000 7f8cea400640 1 -- start start 2024-03-23T11:26:19.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.329+0000 7f8cea400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ce40d0ba0 0x7f8ce4152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.329+0000 7f8cea400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ce415ca50 0x7f8ce40c6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.329+0000 7f8cea400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ce415ce30 0x7f8ce40c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.329+0000 7f8cea400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce41533f0 con 0x7f8ce415ca50 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.329+0000 7f8cea400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce41536d0 con 0x7f8ce40d0ba0 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.329+0000 7f8cea400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce41539b0 con 0x7f8ce415ce30 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.329+0000 7f8cdaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ce415ca50 0x7f8ce40c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.329+0000 7f8cdaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ce415ca50 0x7f8ce40c6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32972/0 (socket says 172.21.15.62:32972) 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.329+0000 7f8cdaa00640 1 -- 172.21.15.62:0/1616378524 learned_addr learned my addr 172.21.15.62:0/1616378524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.329+0000 7f8ce3400640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ce40d0ba0 0x7f8ce4152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.329+0000 7f8ce3e00640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ce415ce30 0x7f8ce40c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8cdaa00640 1 -- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ce415ce30 msgr2=0x7f8ce40c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8cdaa00640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ce415ce30 0x7f8ce40c5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8cdaa00640 1 -- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ce40d0ba0 msgr2=0x7f8ce4152d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:19.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8cdaa00640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ce40d0ba0 0x7f8ce4152d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:19.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8cdaa00640 1 -- 172.21.15.62:0/1616378524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cdc0094d0 con 0x7f8ce415ca50 2024-03-23T11:26:19.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8ce3e00640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ce415ce30 0x7f8ce40c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:26:19.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8cdaa00640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ce415ca50 0x7f8ce40c6690 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00d5f0 tx=0x7f8ccc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:19.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8cdbe00640 1 -- 172.21.15.62:0/1616378524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc0125f0 con 0x7f8ce415ca50 2024-03-23T11:26:19.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8cdbe00640 1 -- 172.21.15.62:0/1616378524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc012750 con 0x7f8ce415ca50 2024-03-23T11:26:19.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8cdbe00640 1 -- 172.21.15.62:0/1616378524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc012b00 con 0x7f8ce415ca50 2024-03-23T11:26:19.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8cea400640 1 -- 172.21.15.62:0/1616378524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce4153c90 con 0x7f8ce415ca50 2024-03-23T11:26:19.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.330+0000 7f8cea400640 1 -- 172.21.15.62:0/1616378524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce40c6bd0 con 0x7f8ce415ca50 2024-03-23T11:26:19.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.331+0000 7f8cea400640 1 -- 172.21.15.62:0/1616378524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce4066bb0 con 0x7f8ce415ca50 2024-03-23T11:26:19.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.333+0000 7f8cdbe00640 1 -- 172.21.15.62:0/1616378524 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8ccc012c60 con 0x7f8ce415ca50 2024-03-23T11:26:19.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.333+0000 7f8cdbe00640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8cc006a7d0 0x7f8cc006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:19.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.333+0000 7f8cdbe00640 1 -- 172.21.15.62:0/1616378524 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8ccc088e60 con 0x7f8ce415ca50 2024-03-23T11:26:19.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.334+0000 7f8ce3400640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8cc006a7d0 0x7f8cc006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:19.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.335+0000 7f8ce3400640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8cc006a7d0 0x7f8cc006cc90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4003aa0 tx=0x7f8cd4009810 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:19.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.338+0000 7f8cdbe00640 1 -- 172.21.15.62:0/1616378524 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ccc0545b0 con 0x7f8ce415ca50 2024-03-23T11:26:19.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:19.548+0000 7f8cea400640 1 -- 172.21.15.62:0/1616378524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8ce40d0470 con 0x7f8ce415ca50 2024-03-23T11:26:20.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.325+0000 7f8cdbe00640 1 -- 172.21.15.62:0/1616378524 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v85) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8ccc057bd0 con 0x7f8ce415ca50 2024-03-23T11:26:20.319 INFO:teuthology.orchestra.run.smithi062.stderr: removed all blocklist entries 2024-03-23T11:26:20.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 -- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8cc006a7d0 msgr2=0x7f8cc006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:20.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8cc006a7d0 0x7f8cc006cc90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4003aa0 tx=0x7f8cd4009810 comp rx=0 tx=0).stop 2024-03-23T11:26:20.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 -- 172.21.15.62:0/1616378524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ce415ca50 msgr2=0x7f8ce40c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:20.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ce415ca50 0x7f8ce40c6690 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00d5f0 tx=0x7f8ccc00dab0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 -- 172.21.15.62:0/1616378524 shutdown_connections 2024-03-23T11:26:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8cc006a7d0 0x7f8cc006cc90 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ce415ca50 0x7f8ce40c6690 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ce415ce30 0x7f8ce40c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 --2- 172.21.15.62:0/1616378524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ce40d0ba0 0x7f8ce4152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 -- 172.21.15.62:0/1616378524 >> 172.21.15.62:0/1616378524 conn(0x7f8ce40717b0 msgr2=0x7f8ce415fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 -- 172.21.15.62:0/1616378524 shutdown_connections 2024-03-23T11:26:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.329+0000 7f8cea400640 1 -- 172.21.15.62:0/1616378524 wait complete. 2024-03-23T11:26:20.345 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T11:26:20.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.576+0000 7f91dec00640 1 Processor -- start 2024-03-23T11:26:20.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.576+0000 7f91dec00640 1 -- start start 2024-03-23T11:26:20.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.578+0000 7f91dec00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f91d80ce960 0x7f91d80ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:20.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.578+0000 7f91dec00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d81608d0 con 0x7f91d815c830 2024-03-23T11:26:20.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.578+0000 7f91dec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d8160bb0 con 0x7f91d8160310 2024-03-23T11:26:20.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.578+0000 7f91dec00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d80addb0 con 0x7f91d80ce960 2024-03-23T11:26:20.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.578+0000 7f91dd800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f91d815c830 0x7f91d80ab980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43528/0 (socket says 172.21.15.62:43528) 2024-03-23T11:26:20.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.578+0000 7f91dd800640 1 -- 172.21.15.62:0/844076303 learned_addr learned my addr 172.21.15.62:0/844076303 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:20.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.579+0000 7f91de200640 1 --2- 172.21.15.62:0/844076303 >> v2:172.21.15.120:3300/0 conn(0x7f91d80ce960 0x7f91d80ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:20.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.579+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155268075 0 0) 0x7f91d81608d0 con 0x7f91d815c830 2024-03-23T11:26:20.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.579+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c0003520 con 0x7f91d815c830 2024-03-23T11:26:20.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.579+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 523086157 0 0) 0x7f91c0003520 con 0x7f91d815c830 2024-03-23T11:26:20.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 >> v2:172.21.15.120:3300/0 conn(0x7f91d80ce960 msgr2=0x7f91d80ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:20.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 --2- 172.21.15.62:0/844076303 >> v2:172.21.15.120:3300/0 conn(0x7f91d80ce960 0x7f91d80ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 >> v1:172.21.15.120:6789/0 conn(0x7f91d8160310 legacy=0x7f91d80ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d8160e90 con 0x7f91d815c830 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f91c8001670 con 0x7f91d815c830 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 >> v1:172.21.15.62:6789/0 conn(0x7f91d815c830 legacy=0x7f91d80ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 --2- 172.21.15.62:0/844076303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 0x7f91c0003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 --2- 172.21.15.62:0/844076303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91c00047c0 0x7f91c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 --2- 172.21.15.62:0/844076303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91c0005180 0x7f91c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c000bfa0 con 0x7f91c0003970 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c000c110 con 0x7f91c00047c0 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c000c2d0 con 0x7f91c0005180 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91dd800640 1 --2- 172.21.15.62:0/844076303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 0x7f91c0003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.580+0000 7f91dce00640 1 --2- 172.21.15.62:0/844076303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91c00047c0 0x7f91c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.581+0000 7f91dd800640 1 -- 172.21.15.62:0/844076303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91c0005180 msgr2=0x7f91c000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:20.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.581+0000 7f91dd800640 1 --2- 172.21.15.62:0/844076303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91c0005180 0x7f91c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.581+0000 7f91dd800640 1 -- 172.21.15.62:0/844076303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91c00047c0 msgr2=0x7f91c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:20.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.581+0000 7f91dd800640 1 --2- 172.21.15.62:0/844076303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91c00047c0 0x7f91c0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.581+0000 7f91dd800640 1 -- 172.21.15.62:0/844076303 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c0003520 con 0x7f91c0003970 2024-03-23T11:26:20.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.581+0000 7f91de200640 1 -- 172.21.15.62:0/844076303 reap_dead start 2024-03-23T11:26:20.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.582+0000 7f91dd800640 1 --2- 172.21.15.62:0/844076303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 0x7f91c0003d50 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f91c800c190 tx=0x7f91c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d6344560ae0a10 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:20.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.582+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c8005ad0 con 0x7f91c0003970 2024-03-23T11:26:20.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.582+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c8005c30 con 0x7f91c0003970 2024-03-23T11:26:20.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.582+0000 7f91d7e00640 1 -- 172.21.15.62:0/844076303 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c802c7d0 con 0x7f91c0003970 2024-03-23T11:26:20.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.583+0000 7f91dec00640 1 -- 172.21.15.62:0/844076303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 msgr2=0x7f91c0003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:20.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.583+0000 7f91dec00640 1 --2- 172.21.15.62:0/844076303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 0x7f91c0003d50 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f91c800c190 tx=0x7f91c00057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.583+0000 7f91dec00640 1 -- 172.21.15.62:0/844076303 shutdown_connections 2024-03-23T11:26:20.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.583+0000 7f91dec00640 1 --2- 172.21.15.62:0/844076303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 0x7f91c0003d50 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.583+0000 7f91dec00640 1 -- 172.21.15.62:0/844076303 >> 172.21.15.62:0/844076303 conn(0x7f91d8066050 msgr2=0x7f91d8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:20.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.583+0000 7f91dec00640 1 -- 172.21.15.62:0/844076303 shutdown_connections 2024-03-23T11:26:20.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.584+0000 7f91dec00640 1 -- 172.21.15.62:0/844076303 wait complete. 2024-03-23T11:26:20.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.585+0000 7f91dec00640 1 Processor -- start 2024-03-23T11:26:20.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.585+0000 7f91dec00640 1 -- start start 2024-03-23T11:26:20.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.586+0000 7f91dec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 0x7f91d8149060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:20.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.586+0000 7f91dec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91d815bb20 0x7f91d815bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:20.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.586+0000 7f91dec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d8149800 0x7f91d8149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:20.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.586+0000 7f91dec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d81495a0 con 0x7f91c0003970 2024-03-23T11:26:20.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.586+0000 7f91dec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d815c4e0 con 0x7f91d815bb20 2024-03-23T11:26:20.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.586+0000 7f91dec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d814b140 con 0x7f91d8149800 2024-03-23T11:26:20.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.587+0000 7f91dd800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 0x7f91d8149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:20.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.587+0000 7f91dd800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 0x7f91d8149060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32980/0 (socket says 172.21.15.62:32980) 2024-03-23T11:26:20.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.587+0000 7f91dd800640 1 -- 172.21.15.62:0/2854981309 learned_addr learned my addr 172.21.15.62:0/2854981309 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:20.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.587+0000 7f91de200640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d8149800 0x7f91d8149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:20.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.587+0000 7f91dd800640 1 -- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d8149800 msgr2=0x7f91d8149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:20.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.587+0000 7f91dd800640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d8149800 0x7f91d8149c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.587+0000 7f91dd800640 1 -- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91d815bb20 msgr2=0x7f91d815bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:20.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.587+0000 7f91dd800640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91d815bb20 0x7f91d815bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.587+0000 7f91dd800640 1 -- 172.21.15.62:0/2854981309 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d814b5d0 con 0x7f91c0003970 2024-03-23T11:26:20.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.587+0000 7f91dd800640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 0x7f91d8149060 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f91c802c370 tx=0x7f91c802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:20.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.588+0000 7f91d5600640 1 -- 172.21.15.62:0/2854981309 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c803c040 con 0x7f91c0003970 2024-03-23T11:26:20.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.588+0000 7f91d5600640 1 -- 172.21.15.62:0/2854981309 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c802cce0 con 0x7f91c0003970 2024-03-23T11:26:20.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.588+0000 7f91d5600640 1 -- 172.21.15.62:0/2854981309 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c80072b0 con 0x7f91c0003970 2024-03-23T11:26:20.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.588+0000 7f91dec00640 1 -- 172.21.15.62:0/2854981309 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d814a530 con 0x7f91c0003970 2024-03-23T11:26:20.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.588+0000 7f91dec00640 1 -- 172.21.15.62:0/2854981309 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d814aaf0 con 0x7f91c0003970 2024-03-23T11:26:20.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.589+0000 7f91dec00640 1 -- 172.21.15.62:0/2854981309 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d8066bb0 con 0x7f91c0003970 2024-03-23T11:26:20.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.590+0000 7f91d5600640 1 -- 172.21.15.62:0/2854981309 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f91c8032070 con 0x7f91c0003970 2024-03-23T11:26:20.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.591+0000 7f91d5600640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f91bc06a7a0 0x7f91bc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:20.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.591+0000 7f91d5600640 1 -- 172.21.15.62:0/2854981309 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f91c80a9e30 con 0x7f91c0003970 2024-03-23T11:26:20.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.592+0000 7f91dce00640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f91bc06a7a0 0x7f91bc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:20.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.593+0000 7f91dce00640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f91bc06a7a0 0x7f91bc06cc60 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f91d80adad0 tx=0x7f91cc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:20.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.596+0000 7f91d5600640 1 -- 172.21.15.62:0/2854981309 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f91c8075590 con 0x7f91c0003970 2024-03-23T11:26:20.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.865+0000 7f91dec00640 1 -- 172.21.15.62:0/2854981309 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f91d80ce500 con 0x7f91c0003970 2024-03-23T11:26:20.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.866+0000 7f91d5600640 1 -- 172.21.15.62:0/2854981309 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f91c8078bb0 con 0x7f91c0003970 2024-03-23T11:26:20.860 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:20.860 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:26:20.860 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_dump": [ 2024-03-23T11:26:20.860 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.860 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.a", 2024-03-23T11:26:20.860 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lYNxYERAAPVHWwRHM5V7OejL294b2zg==", 2024-03-23T11:26:20.860 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.860 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:26:20.860 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:26:20.860 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.b", 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lmOUANBAAifuZ0DUPD9az/mturTndSA==", 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.861 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.c", 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5ltqrjIhAA9+f82fvlrFlIEXqswPE0ig==", 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.862 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.d", 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5l7ly4CBAA8+oim0z+bbljfP0VH7XSaw==", 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.0", 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBiuv5lRfaQBRAAvrtCZXwt00PUQfCqqKBzMQ==", 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:26:20.863 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.1", 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBluv5lzwS/IBAAmYU8yMLtCUJNr/AyE0MJDA==", 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.864 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.2", 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBouv5lyAy1OhAAJoUsg6DFpEPxAcH5oI2exQ==", 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.3", 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBsuv5lb0YcGBAA220D0uRhAZpG4w97wqsB5A==", 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:26:20.865 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.4", 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBzuv5lC8RIGhAAH+5HaeCgQhrQMrYm9iWaYA==", 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.866 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.5", 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB2uv5lYi7ZIhAAMNIm/jPckCAbX9oEx5Li/Q==", 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.6", 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB5uv5lgoRtLBAAFT7uJtyqHtHAegOYyDH/WQ==", 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:26:20.867 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.868 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.868 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.868 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.868 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.7", 2024-03-23T11:26:20.868 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB9uv5l/dzrCBAA3jsua0LvBPI7I551/rRHrg==", 2024-03-23T11:26:20.868 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.868 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:26:20.868 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:26:20.869 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.869 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.869 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.869 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.869 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.0", 2024-03-23T11:26:20.869 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5ltavQFhAAvV2RWuHOPK2T9Q7gWkYA4w==", 2024-03-23T11:26:20.869 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.869 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:26:20.869 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow r", 2024-03-23T11:26:20.869 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.1", 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5luQ/zLRAAwjDtV64TCKnQliPq4iUAYg==", 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.870 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.2", 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lDRG1IhAAjVaXRR30G1985oMou7Ambg==", 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.3", 2024-03-23T11:26:20.871 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lomZyOBAA8BMIBD+Zhtp6Fa2RcEXCJg==", 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.admin", 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lNtZHBhAAhiAC1rdmANClOSwzrX9iSQ==", 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:26:20.872 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.x", 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lT6+KBRAAl5TZsJGNr4gAB2cEDZo/hQ==", 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.873 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:20.874 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:20.874 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.y", 2024-03-23T11:26:20.874 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lqnG8NRAAvyswMz5kpW0rGjzWYRTAAg==", 2024-03-23T11:26:20.874 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:26:20.874 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:26:20.874 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:26:20.874 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:26:20.874 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.874 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:20.874 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-03-23T11:26:20.874 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:26:20.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 -- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f91bc06a7a0 msgr2=0x7f91bc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:20.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f91bc06a7a0 0x7f91bc06cc60 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f91d80adad0 tx=0x7f91cc009040 comp rx=0 tx=0).stop 2024-03-23T11:26:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 -- 172.21.15.62:0/2854981309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 msgr2=0x7f91d8149060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 0x7f91d8149060 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f91c802c370 tx=0x7f91c802e070 comp rx=0 tx=0).stop 2024-03-23T11:26:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 -- 172.21.15.62:0/2854981309 shutdown_connections 2024-03-23T11:26:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f91bc06a7a0 0x7f91bc06cc60 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d8149800 0x7f91d8149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91d815bb20 0x7f91d815bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 --2- 172.21.15.62:0/2854981309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c0003970 0x7f91d8149060 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 -- 172.21.15.62:0/2854981309 >> 172.21.15.62:0/2854981309 conn(0x7f91d8066050 msgr2=0x7f91d815b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 -- 172.21.15.62:0/2854981309 shutdown_connections 2024-03-23T11:26:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:20.869+0000 7f91dec00640 1 -- 172.21.15.62:0/2854981309 wait complete. 2024-03-23T11:26:20.882 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:26:21.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.109+0000 7fcf03a00640 1 Processor -- start 2024-03-23T11:26:21.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.109+0000 7fcf03a00640 1 -- start start 2024-03-23T11:26:21.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.111+0000 7fcf03a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0cafa0 0x7fcefc0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.111+0000 7fcf03a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcefc06c6e0 0x7fcefc06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.111+0000 7fcf03a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc0cbc60 con 0x7fcefc06c300 2024-03-23T11:26:21.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.111+0000 7fcf03a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc0cd410 con 0x7fcefc06c6e0 2024-03-23T11:26:21.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.111+0000 7fcf03a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0cbf40 con 0x7fcefc0cafa0 2024-03-23T11:26:21.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.111+0000 7fcf01000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fcefc06c300 0x7fcefc0ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43544/0 (socket says 172.21.15.62:43544) 2024-03-23T11:26:21.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.111+0000 7fcf01000640 1 -- 172.21.15.62:0/2860405362 learned_addr learned my addr 172.21.15.62:0/2860405362 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:21.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.111+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346243107 0 0) 0x7fcefc0cbc60 con 0x7fcefc06c300 2024-03-23T11:26:21.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.111+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee0003520 con 0x7fcefc06c300 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcf01a00640 1 --2- 172.21.15.62:0/2860405362 >> v2:172.21.15.120:3300/0 conn(0x7fcefc06c6e0 0x7fcefc06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefbe00640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0cafa0 0x7fcefc0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2825603370 0 0) 0x7fcee0003520 con 0x7fcefc06c300 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0cafa0 msgr2=0x7fcefc0cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0cafa0 0x7fcefc0cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 >> v2:172.21.15.120:3300/0 conn(0x7fcefc06c6e0 msgr2=0x7fcefc06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 --2- 172.21.15.62:0/2860405362 >> v2:172.21.15.120:3300/0 conn(0x7fcefc06c6e0 0x7fcefc06cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc0cc220 con 0x7fcefc06c300 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fcee4001670 con 0x7fcefc06c300 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 >> v1:172.21.15.62:6789/0 conn(0x7fcefc06c300 legacy=0x7fcefc0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 0x7fcee0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcee00047c0 0x7fcee0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0cafa0 0x7fcee0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0cd410 con 0x7fcee0003970 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0cbf40 con 0x7fcee00047c0 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee000ba80 con 0x7fcefc0cafa0 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.112+0000 7fcf01000640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 0x7fcee0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.113+0000 7fcefbe00640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcee00047c0 0x7fcee0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.113+0000 7fcf01a00640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0cafa0 0x7fcee0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.113+0000 7fcf01000640 1 -- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0cafa0 msgr2=0x7fcee0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.113+0000 7fcf01000640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc0cafa0 0x7fcee0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.113+0000 7fcf01000640 1 -- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcee00047c0 msgr2=0x7fcee0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.113+0000 7fcf01000640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcee00047c0 0x7fcee0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.113+0000 7fcf01000640 1 -- 172.21.15.62:0/2860405362 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee0003520 con 0x7fcee0003970 2024-03-23T11:26:21.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.113+0000 7fcefbe00640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcee00047c0 0x7fcee0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:21.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.114+0000 7fcf01000640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 0x7fcee0003d90 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fcee4001050 tx=0x7fcee00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306c076247023ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:21.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.114+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4005ad0 con 0x7fcee0003970 2024-03-23T11:26:21.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.114+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee4005c30 con 0x7fcee0003970 2024-03-23T11:26:21.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.114+0000 7fcefb400640 1 -- 172.21.15.62:0/2860405362 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee402c7c0 con 0x7fcee0003970 2024-03-23T11:26:21.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.115+0000 7fcf03a00640 1 -- 172.21.15.62:0/2860405362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 msgr2=0x7fcee0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.115+0000 7fcf03a00640 1 --2- 172.21.15.62:0/2860405362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 0x7fcee0003d90 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fcee4001050 tx=0x7fcee00057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.115+0000 7fcf01a00640 1 -- 172.21.15.62:0/2860405362 reap_dead start 2024-03-23T11:26:21.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.115+0000 7fcf03a00640 1 -- 172.21.15.62:0/2860405362 shutdown_connections 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.115+0000 7fcf03a00640 1 -- 172.21.15.62:0/2860405362 >> 172.21.15.62:0/2860405362 conn(0x7fcefc0b02e0 msgr2=0x7fcefc0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.115+0000 7fcf03a00640 1 -- 172.21.15.62:0/2860405362 shutdown_connections 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.115+0000 7fcf03a00640 1 -- 172.21.15.62:0/2860405362 wait complete. 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.117+0000 7fcf03a00640 1 Processor -- start 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.117+0000 7fcf03a00640 1 -- start start 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.118+0000 7fcf03a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 0x7fcefc1582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.118+0000 7fcf03a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc13d740 0x7fcefc15c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.118+0000 7fcf03a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcefc15c670 0x7fcefc13e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.118+0000 7fcf03a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0ca4b0 con 0x7fcee0003970 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.118+0000 7fcf03a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc15bca0 con 0x7fcefc15c670 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.118+0000 7fcf03a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc15b090 con 0x7fcefc13d740 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.118+0000 7fcf01000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 0x7fcefc1582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.118+0000 7fcf01000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 0x7fcefc1582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33000/0 (socket says 172.21.15.62:33000) 2024-03-23T11:26:21.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.118+0000 7fcf01000640 1 -- 172.21.15.62:0/1253993317 learned_addr learned my addr 172.21.15.62:0/1253993317 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcf01a00640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcefc15c670 0x7fcefc13e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcefbe00640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc13d740 0x7fcefc15c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcf01000640 1 -- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc13d740 msgr2=0x7fcefc15c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcf01000640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc13d740 0x7fcefc15c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcf01000640 1 -- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcefc15c670 msgr2=0x7fcefc13e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcf01000640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcefc15c670 0x7fcefc13e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcf01000640 1 -- 172.21.15.62:0/1253993317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc15b370 con 0x7fcee0003970 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcf01000640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 0x7fcefc1582e0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fcee402c360 tx=0x7fcee4000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcef8c00640 1 -- 172.21.15.62:0/1253993317 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4039070 con 0x7fcee0003970 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcef8c00640 1 -- 172.21.15.62:0/1253993317 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee402dc80 con 0x7fcee0003970 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcef8c00640 1 -- 172.21.15.62:0/1253993317 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4007070 con 0x7fcee0003970 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.119+0000 7fcf03a00640 1 -- 172.21.15.62:0/1253993317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc15b650 con 0x7fcee0003970 2024-03-23T11:26:21.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.120+0000 7fcf03a00640 1 -- 172.21.15.62:0/1253993317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc15ad80 con 0x7fcee0003970 2024-03-23T11:26:21.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.121+0000 7fcf03a00640 1 -- 172.21.15.62:0/1253993317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc066c90 con 0x7fcee0003970 2024-03-23T11:26:21.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.122+0000 7fcef8c00640 1 -- 172.21.15.62:0/1253993317 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fcee4032070 con 0x7fcee0003970 2024-03-23T11:26:21.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.122+0000 7fcef8c00640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fced806a7a0 0x7fced806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.123+0000 7fcef8c00640 1 -- 172.21.15.62:0/1253993317 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcee403c040 con 0x7fcee0003970 2024-03-23T11:26:21.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.123+0000 7fcefbe00640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fced806a7a0 0x7fced806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.124+0000 7fcefbe00640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fced806a7a0 0x7fced806cc60 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0ca070 tx=0x7fceec00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:21.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.127+0000 7fcef8c00640 1 -- 172.21.15.62:0/1253993317 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcee40752b0 con 0x7fcee0003970 2024-03-23T11:26:21.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.325+0000 7fcf03a00640 1 -- 172.21.15.62:0/1253993317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcefc13d220 con 0x7fcee0003970 2024-03-23T11:26:21.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.327+0000 7fcef8c00640 1 -- 172.21.15.62:0/1253993317 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+12390 (secure 0 0 0) 0x7fcee4045020 con 0x7fcee0003970 2024-03-23T11:26:21.320 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:21.321 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":85,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:26:20.310887+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:26:21.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 -- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fced806a7a0 msgr2=0x7fced806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fced806a7a0 0x7fced806cc60 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0ca070 tx=0x7fceec00b040 comp rx=0 tx=0).stop 2024-03-23T11:26:21.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 -- 172.21.15.62:0/1253993317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 msgr2=0x7fcefc1582e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 0x7fcefc1582e0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fcee402c360 tx=0x7fcee4000fe0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 -- 172.21.15.62:0/1253993317 shutdown_connections 2024-03-23T11:26:21.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fced806a7a0 0x7fced806cc60 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcefc15c670 0x7fcefc13e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcefc13d740 0x7fcefc15c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 --2- 172.21.15.62:0/1253993317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee0003970 0x7fcefc1582e0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 -- 172.21.15.62:0/1253993317 >> 172.21.15.62:0/1253993317 conn(0x7fcefc0b02e0 msgr2=0x7fcefc13c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:21.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 -- 172.21.15.62:0/1253993317 shutdown_connections 2024-03-23T11:26:21.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.330+0000 7fcf03a00640 1 -- 172.21.15.62:0/1253993317 wait complete. 2024-03-23T11:26:21.344 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:26:21.344 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:26:21.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.568+0000 7f466d600640 1 Processor -- start 2024-03-23T11:26:21.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.568+0000 7f466d600640 1 -- start start 2024-03-23T11:26:21.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.570+0000 7f466d600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f46680b15e0 0x7f466806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.570+0000 7f466d600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46680cad20 con 0x7f466806dca0 2024-03-23T11:26:21.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.570+0000 7f466d600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46680b19f0 con 0x7f46680cd7c0 2024-03-23T11:26:21.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.570+0000 7f466d600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466806e130 con 0x7f46680b15e0 2024-03-23T11:26:21.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.571+0000 7f4666000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f466806dca0 0x7f46680cb490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43558/0 (socket says 172.21.15.62:43558) 2024-03-23T11:26:21.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.571+0000 7f4666000640 1 -- 172.21.15.62:0/841854730 learned_addr learned my addr 172.21.15.62:0/841854730 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.571+0000 7f4667400640 1 --2- 172.21.15.62:0/841854730 >> v2:172.21.15.120:3300/0 conn(0x7f46680b15e0 0x7f466806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.571+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149308053 0 0) 0x7f46680cad20 con 0x7f466806dca0 2024-03-23T11:26:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.571+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f464c003520 con 0x7f466806dca0 2024-03-23T11:26:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3796300936 0 0) 0x7f464c003520 con 0x7f466806dca0 2024-03-23T11:26:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 >> v2:172.21.15.120:3300/0 conn(0x7f46680b15e0 msgr2=0x7f466806c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 --2- 172.21.15.62:0/841854730 >> v2:172.21.15.120:3300/0 conn(0x7f46680b15e0 0x7f466806c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 >> v1:172.21.15.120:6789/0 conn(0x7f46680cd7c0 legacy=0x7f46680cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466815a8a0 con 0x7f466806dca0 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f4654001670 con 0x7f466806dca0 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 >> v1:172.21.15.62:6789/0 conn(0x7f466806dca0 legacy=0x7f46680cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 --2- 172.21.15.62:0/841854730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464c003970 0x7f464c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 --2- 172.21.15.62:0/841854730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f464c0047c0 0x7f464c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 --2- 172.21.15.62:0/841854730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464c005180 0x7f464c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466806e130 con 0x7f464c005180 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c00e290 con 0x7f464c003970 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c00e450 con 0x7f464c0047c0 2024-03-23T11:26:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4667400640 1 --2- 172.21.15.62:0/841854730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464c005180 0x7f464c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4667400640 1 -- 172.21.15.62:0/841854730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f464c0047c0 msgr2=0x7f464c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:21.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4667400640 1 --2- 172.21.15.62:0/841854730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f464c0047c0 0x7f464c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4667400640 1 -- 172.21.15.62:0/841854730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464c003970 msgr2=0x7f464c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.572+0000 7f4666a00640 1 --2- 172.21.15.62:0/841854730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464c003970 0x7f464c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.573+0000 7f4667400640 1 --2- 172.21.15.62:0/841854730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464c003970 0x7f464c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.573+0000 7f4667400640 1 -- 172.21.15.62:0/841854730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466815a8a0 con 0x7f464c005180 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.573+0000 7f4666a00640 1 --2- 172.21.15.62:0/841854730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464c003970 0x7f464c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.573+0000 7f4667400640 1 -- 172.21.15.62:0/841854730 reap_dead start 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.574+0000 7f4667400640 1 --2- 172.21.15.62:0/841854730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464c005180 0x7f464c00db90 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f4660009fc0 tx=0x7f464c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5444d8fb96f0c83c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.574+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466002cc60 con 0x7f464c005180 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.574+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466000f040 con 0x7f464c005180 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.574+0000 7f4665600640 1 -- 172.21.15.62:0/841854730 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660003990 con 0x7f464c005180 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.574+0000 7f466d600640 1 -- 172.21.15.62:0/841854730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464c005180 msgr2=0x7f464c00db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.575+0000 7f466d600640 1 --2- 172.21.15.62:0/841854730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464c005180 0x7f464c00db90 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f4660009fc0 tx=0x7f464c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.575+0000 7f466d600640 1 -- 172.21.15.62:0/841854730 shutdown_connections 2024-03-23T11:26:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.575+0000 7f466d600640 1 --2- 172.21.15.62:0/841854730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464c005180 0x7f464c00db90 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.575+0000 7f466d600640 1 -- 172.21.15.62:0/841854730 >> 172.21.15.62:0/841854730 conn(0x7f46680b02e0 msgr2=0x7f46680b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:21.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.575+0000 7f466d600640 1 -- 172.21.15.62:0/841854730 shutdown_connections 2024-03-23T11:26:21.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.575+0000 7f466d600640 1 -- 172.21.15.62:0/841854730 wait complete. 2024-03-23T11:26:21.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.577+0000 7f466d600640 1 Processor -- start 2024-03-23T11:26:21.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.577+0000 7f466d600640 1 -- start start 2024-03-23T11:26:21.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.578+0000 7f466d600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464c005180 0x7f466814f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.578+0000 7f466d600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46680ad100 0x7f46680ad580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.578+0000 7f466d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46680af180 0x7f46680af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.578+0000 7f466d600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4668150750 con 0x7f46680ad100 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.578+0000 7f466d600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46680afb50 con 0x7f464c005180 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.578+0000 7f466d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466814f5f0 con 0x7f46680af180 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.578+0000 7f4666000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46680ad100 0x7f46680ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.579+0000 7f4666000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46680ad100 0x7f46680ad580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33014/0 (socket says 172.21.15.62:33014) 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.579+0000 7f4666000640 1 -- 172.21.15.62:0/2081836731 learned_addr learned my addr 172.21.15.62:0/2081836731 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.579+0000 7f4666a00640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464c005180 0x7f466814f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.579+0000 7f4667400640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46680af180 0x7f46680af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.579+0000 7f4666000640 1 -- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46680af180 msgr2=0x7f46680af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.579+0000 7f4666000640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46680af180 0x7f46680af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.579+0000 7f4666000640 1 -- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464c005180 msgr2=0x7f466814f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.579+0000 7f4666000640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464c005180 0x7f466814f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.579+0000 7f4666000640 1 -- 172.21.15.62:0/2081836731 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466814f860 con 0x7f46680ad100 2024-03-23T11:26:21.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.579+0000 7f4666000640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46680ad100 0x7f46680ad580 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f465400c7b0 tx=0x7f465400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:21.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.580+0000 7f465ea00640 1 -- 172.21.15.62:0/2081836731 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46540074e0 con 0x7f46680ad100 2024-03-23T11:26:21.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.580+0000 7f465ea00640 1 -- 172.21.15.62:0/2081836731 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4654007640 con 0x7f46680ad100 2024-03-23T11:26:21.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.580+0000 7f466d600640 1 -- 172.21.15.62:0/2081836731 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466814fb40 con 0x7f46680ad100 2024-03-23T11:26:21.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.580+0000 7f465ea00640 1 -- 172.21.15.62:0/2081836731 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654004a00 con 0x7f46680ad100 2024-03-23T11:26:21.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.580+0000 7f466d600640 1 -- 172.21.15.62:0/2081836731 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4668150100 con 0x7f46680ad100 2024-03-23T11:26:21.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.581+0000 7f466d600640 1 -- 172.21.15.62:0/2081836731 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4668066c90 con 0x7f46680ad100 2024-03-23T11:26:21.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.582+0000 7f465ea00640 1 -- 172.21.15.62:0/2081836731 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4654004b60 con 0x7f46680ad100 2024-03-23T11:26:21.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.583+0000 7f465ea00640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f464806a7d0 0x7f464806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:21.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.583+0000 7f465ea00640 1 -- 172.21.15.62:0/2081836731 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f46540882a0 con 0x7f46680ad100 2024-03-23T11:26:21.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.584+0000 7f4666a00640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f464806a7d0 0x7f464806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:21.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.585+0000 7f4666a00640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f464806a7d0 0x7f464806cc90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4658000f60 tx=0x7f4658009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:21.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.588+0000 7f465ea00640 1 -- 172.21.15.62:0/2081836731 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4654053970 con 0x7f46680ad100 2024-03-23T11:26:21.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.796+0000 7f466d600640 1 -- 172.21.15.62:0/2081836731 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f466806c970 con 0x7f46680ad100 2024-03-23T11:26:21.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.797+0000 7f465ea00640 1 -- 172.21.15.62:0/2081836731 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4654018020 con 0x7f46680ad100 2024-03-23T11:26:21.790 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:26:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 -- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f464806a7d0 msgr2=0x7f464806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f464806a7d0 0x7f464806cc90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4658000f60 tx=0x7f4658009040 comp rx=0 tx=0).stop 2024-03-23T11:26:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 -- 172.21.15.62:0/2081836731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46680ad100 msgr2=0x7f46680ad580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46680ad100 0x7f46680ad580 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f465400c7b0 tx=0x7f465400cc70 comp rx=0 tx=0).stop 2024-03-23T11:26:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 -- 172.21.15.62:0/2081836731 shutdown_connections 2024-03-23T11:26:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f464806a7d0 0x7f464806cc90 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46680ad100 0x7f46680ad580 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46680af180 0x7f46680af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 --2- 172.21.15.62:0/2081836731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f464c005180 0x7f466814f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 -- 172.21.15.62:0/2081836731 >> 172.21.15.62:0/2081836731 conn(0x7f46680b02e0 msgr2=0x7f466806cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 -- 172.21.15.62:0/2081836731 shutdown_connections 2024-03-23T11:26:21.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:21.800+0000 7f466d600640 1 -- 172.21.15.62:0/2081836731 wait complete. 2024-03-23T11:26:21.813 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T11:26:22.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.042+0000 7f886f400640 1 Processor -- start 2024-03-23T11:26:22.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.043+0000 7f886f400640 1 -- start start 2024-03-23T11:26:22.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.044+0000 7f886f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f886815cae0 0x7f88680d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.044+0000 7f886f400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f886815cec0 0x7f88680b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.044+0000 7f886f400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8868160820 0x7f88680cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.044+0000 7f886f400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8868160d50 con 0x7f886815cae0 2024-03-23T11:26:22.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.044+0000 7f886f400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8868161030 con 0x7f8868160820 2024-03-23T11:26:22.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.044+0000 7f886f400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8868161310 con 0x7f886815cec0 2024-03-23T11:26:22.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.044+0000 7f886ca00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f886815cae0 0x7f88680d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:22.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.045+0000 7f886ca00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f886815cae0 0x7f88680d2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33024/0 (socket says 172.21.15.62:33024) 2024-03-23T11:26:22.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.045+0000 7f886ca00640 1 -- 172.21.15.62:0/45035255 learned_addr learned my addr 172.21.15.62:0/45035255 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:22.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.045+0000 7f886ca00640 1 -- 172.21.15.62:0/45035255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f886815cec0 msgr2=0x7f88680b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:22.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.045+0000 7f886d400640 1 --2- 172.21.15.62:0/45035255 >> v2:172.21.15.120:3300/0 conn(0x7f8868160820 0x7f88680cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:22.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.045+0000 7f8867e00640 1 --2- 172.21.15.62:0/45035255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f886815cec0 0x7f88680b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:22.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.045+0000 7f886ca00640 1 --2- 172.21.15.62:0/45035255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f886815cec0 0x7f88680b5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.045+0000 7f886ca00640 1 -- 172.21.15.62:0/45035255 >> v2:172.21.15.120:3300/0 conn(0x7f8868160820 msgr2=0x7f88680cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:22.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.045+0000 7f886ca00640 1 --2- 172.21.15.62:0/45035255 >> v2:172.21.15.120:3300/0 conn(0x7f8868160820 0x7f88680cf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.045+0000 7f886ca00640 1 -- 172.21.15.62:0/45035255 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88681615f0 con 0x7f886815cae0 2024-03-23T11:26:22.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.045+0000 7f886d400640 1 --2- 172.21.15.62:0/45035255 >> v2:172.21.15.120:3300/0 conn(0x7f8868160820 0x7f88680cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:22.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.046+0000 7f886ca00640 1 --2- 172.21.15.62:0/45035255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f886815cae0 0x7f88680d2090 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f88680d0ca0 tx=0x7f885802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c09b1f250df976 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:22.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.047+0000 7f8867400640 1 -- 172.21.15.62:0/45035255 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858031870 con 0x7f886815cae0 2024-03-23T11:26:22.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.047+0000 7f8867400640 1 -- 172.21.15.62:0/45035255 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88580319d0 con 0x7f886815cae0 2024-03-23T11:26:22.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.047+0000 7f8867400640 1 -- 172.21.15.62:0/45035255 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858031d60 con 0x7f886815cae0 2024-03-23T11:26:22.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.047+0000 7f886f400640 1 -- 172.21.15.62:0/45035255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f886815cae0 msgr2=0x7f88680d2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:22.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.047+0000 7f886f400640 1 --2- 172.21.15.62:0/45035255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f886815cae0 0x7f88680d2090 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f88680d0ca0 tx=0x7f885802b910 comp rx=0 tx=0).stop 2024-03-23T11:26:22.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.047+0000 7f886f400640 1 -- 172.21.15.62:0/45035255 shutdown_connections 2024-03-23T11:26:22.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.047+0000 7f886f400640 1 --2- 172.21.15.62:0/45035255 >> v2:172.21.15.120:3300/0 conn(0x7f8868160820 0x7f88680cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.047+0000 7f886f400640 1 --2- 172.21.15.62:0/45035255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f886815cec0 0x7f88680b5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.047+0000 7f886f400640 1 --2- 172.21.15.62:0/45035255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f886815cae0 0x7f88680d2090 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.048+0000 7f886f400640 1 -- 172.21.15.62:0/45035255 >> 172.21.15.62:0/45035255 conn(0x7f8868071570 msgr2=0x7f8868071950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:22.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.048+0000 7f886f400640 1 -- 172.21.15.62:0/45035255 shutdown_connections 2024-03-23T11:26:22.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.048+0000 7f886f400640 1 -- 172.21.15.62:0/45035255 wait complete. 2024-03-23T11:26:22.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.049+0000 7f886f400640 1 Processor -- start 2024-03-23T11:26:22.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.050+0000 7f886f400640 1 -- start start 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f886815cae0 0x7f88680b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886f400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f886815cec0 0x7f88680b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886f400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8868160820 0x7f8868152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886f400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88680b3ff0 con 0x7f886815cae0 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886f400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88680b3240 con 0x7f886815cec0 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886f400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8868156c10 con 0x7f8868160820 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f886815cae0 0x7f88680b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f886815cae0 0x7f88680b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33036/0 (socket says 172.21.15.62:33036) 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886ca00640 1 -- 172.21.15.62:0/1687784645 learned_addr learned my addr 172.21.15.62:0/1687784645 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886ca00640 1 -- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8868160820 msgr2=0x7f8868152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886ca00640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8868160820 0x7f8868152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886ca00640 1 -- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f886815cec0 msgr2=0x7f88680b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:22.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886ca00640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f886815cec0 0x7f88680b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.051+0000 7f886ca00640 1 -- 172.21.15.62:0/1687784645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88580094d0 con 0x7f886815cae0 2024-03-23T11:26:22.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.052+0000 7f886ca00640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f886815cae0 0x7f88680b4e20 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f885802b950 tx=0x7f88580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:22.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.052+0000 7f8864c00640 1 -- 172.21.15.62:0/1687784645 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858031860 con 0x7f886815cae0 2024-03-23T11:26:22.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.052+0000 7f8864c00640 1 -- 172.21.15.62:0/1687784645 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88580443f0 con 0x7f886815cae0 2024-03-23T11:26:22.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.052+0000 7f8864c00640 1 -- 172.21.15.62:0/1687784645 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88580447a0 con 0x7f886815cae0 2024-03-23T11:26:22.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.052+0000 7f886f400640 1 -- 172.21.15.62:0/1687784645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88680b3470 con 0x7f886815cae0 2024-03-23T11:26:22.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.052+0000 7f886f400640 1 -- 172.21.15.62:0/1687784645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88680b3a30 con 0x7f886815cae0 2024-03-23T11:26:22.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.053+0000 7f886f400640 1 -- 172.21.15.62:0/1687784645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8868156dd0 con 0x7f886815cae0 2024-03-23T11:26:22.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.054+0000 7f8864c00640 1 -- 172.21.15.62:0/1687784645 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f885804d450 con 0x7f886815cae0 2024-03-23T11:26:22.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.056+0000 7f8864c00640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f884406a7a0 0x7f884406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.056+0000 7f8864c00640 1 -- 172.21.15.62:0/1687784645 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f885802d040 con 0x7f886815cae0 2024-03-23T11:26:22.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.056+0000 7f8867e00640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f884406a7a0 0x7f884406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:22.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.057+0000 7f8867e00640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f884406a7a0 0x7f884406cc60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8850007130 tx=0x7f8850007310 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:22.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.060+0000 7f8864c00640 1 -- 172.21.15.62:0/1687784645 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8858076a30 con 0x7f886815cae0 2024-03-23T11:26:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.263+0000 7f886f400640 1 -- 172.21.15.62:0/1687784645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8868066c20 con 0x7f886815cae0 2024-03-23T11:26:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.263+0000 7f8864c00640 1 -- 172.21.15.62:0/1687784645 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v85) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8858035060 con 0x7f886815cae0 2024-03-23T11:26:22.257 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:22.257 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:26:22.257 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:26:22.257 INFO:teuthology.orchestra.run.smithi062.stdout: "poolnum": 1, 2024-03-23T11:26:22.257 INFO:teuthology.orchestra.run.smithi062.stdout: "poolname": ".mgr" 2024-03-23T11:26:22.257 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:22.257 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 -- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f884406a7a0 msgr2=0x7f884406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f884406a7a0 0x7f884406cc60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8850007130 tx=0x7f8850007310 comp rx=0 tx=0).stop 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 -- 172.21.15.62:0/1687784645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f886815cae0 msgr2=0x7f88680b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f886815cae0 0x7f88680b4e20 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f885802b950 tx=0x7f88580046c0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 -- 172.21.15.62:0/1687784645 shutdown_connections 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f884406a7a0 0x7f884406cc60 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8868160820 0x7f8868152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f886815cec0 0x7f88680b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 --2- 172.21.15.62:0/1687784645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f886815cae0 0x7f88680b4e20 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 -- 172.21.15.62:0/1687784645 >> 172.21.15.62:0/1687784645 conn(0x7f8868071570 msgr2=0x7f886815d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 -- 172.21.15.62:0/1687784645 shutdown_connections 2024-03-23T11:26:22.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.266+0000 7f886f400640 1 -- 172.21.15.62:0/1687784645 wait complete. 2024-03-23T11:26:22.281 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T11:26:22.281 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T11:26:22.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.519+0000 7f5f0d200640 1 Processor -- start 2024-03-23T11:26:22.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.519+0000 7f5f0d200640 1 -- start start 2024-03-23T11:26:22.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.521+0000 7f5f0d200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0815a8a0 0x7f5f08070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.521+0000 7f5f0d200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5f080b07f0 0x7f5f080b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.521+0000 7f5f0d200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5f080cd1a0 0x7f5f080cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.521+0000 7f5f0d200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f080b1190 con 0x7f5f080cd1a0 2024-03-23T11:26:22.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.521+0000 7f5f0d200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f080cdb70 con 0x7f5f080b07f0 2024-03-23T11:26:22.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.521+0000 7f5f0d200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0806d610 con 0x7f5f0815a8a0 2024-03-23T11:26:22.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.522+0000 7f5f06800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5f080cd1a0 0x7f5f080cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:22.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.522+0000 7f5f06800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5f080cd1a0 0x7f5f080cd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33052/0 (socket says 172.21.15.62:33052) 2024-03-23T11:26:22.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.522+0000 7f5f06800640 1 -- 172.21.15.62:0/2974341365 learned_addr learned my addr 172.21.15.62:0/2974341365 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:22.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.522+0000 7f5f05e00640 1 --2- 172.21.15.62:0/2974341365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0815a8a0 0x7f5f08070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:22.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.522+0000 7f5f05400640 1 --2- 172.21.15.62:0/2974341365 >> v2:172.21.15.120:3300/0 conn(0x7f5f080b07f0 0x7f5f080b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:22.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.522+0000 7f5f06800640 1 -- 172.21.15.62:0/2974341365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0815a8a0 msgr2=0x7f5f08070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:22.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.522+0000 7f5f06800640 1 --2- 172.21.15.62:0/2974341365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0815a8a0 0x7f5f08070270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.522+0000 7f5f06800640 1 -- 172.21.15.62:0/2974341365 >> v2:172.21.15.120:3300/0 conn(0x7f5f080b07f0 msgr2=0x7f5f080b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:22.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.522+0000 7f5f06800640 1 --2- 172.21.15.62:0/2974341365 >> v2:172.21.15.120:3300/0 conn(0x7f5f080b07f0 0x7f5f080b0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.522+0000 7f5f06800640 1 -- 172.21.15.62:0/2974341365 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0806d8f0 con 0x7f5f080cd1a0 2024-03-23T11:26:22.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.523+0000 7f5f06800640 1 --2- 172.21.15.62:0/2974341365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f080cd1a0 0x7f5f080cd630 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f5f080b26f0 tx=0x7f5f0002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b466306c70544b19 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:22.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.524+0000 7f5f04a00640 1 -- 172.21.15.62:0/2974341365 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0002c9e0 con 0x7f5f080cd1a0 2024-03-23T11:26:22.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.524+0000 7f5f04a00640 1 -- 172.21.15.62:0/2974341365 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0002cb40 con 0x7f5f080cd1a0 2024-03-23T11:26:22.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.524+0000 7f5f04a00640 1 -- 172.21.15.62:0/2974341365 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f000343f0 con 0x7f5f080cd1a0 2024-03-23T11:26:22.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.524+0000 7f5f0d200640 1 -- 172.21.15.62:0/2974341365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f080cd1a0 msgr2=0x7f5f080cd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:22.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.524+0000 7f5f0d200640 1 --2- 172.21.15.62:0/2974341365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f080cd1a0 0x7f5f080cd630 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f5f080b26f0 tx=0x7f5f0002be40 comp rx=0 tx=0).stop 2024-03-23T11:26:22.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.524+0000 7f5f0d200640 1 -- 172.21.15.62:0/2974341365 shutdown_connections 2024-03-23T11:26:22.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.524+0000 7f5f0d200640 1 --2- 172.21.15.62:0/2974341365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f080cd1a0 0x7f5f080cd630 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.524+0000 7f5f0d200640 1 --2- 172.21.15.62:0/2974341365 >> v2:172.21.15.120:3300/0 conn(0x7f5f080b07f0 0x7f5f080b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.524+0000 7f5f0d200640 1 --2- 172.21.15.62:0/2974341365 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0815a8a0 0x7f5f08070270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.524+0000 7f5f0d200640 1 -- 172.21.15.62:0/2974341365 >> 172.21.15.62:0/2974341365 conn(0x7f5f080b17d0 msgr2=0x7f5f080b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:22.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.525+0000 7f5f0d200640 1 -- 172.21.15.62:0/2974341365 shutdown_connections 2024-03-23T11:26:22.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.525+0000 7f5f0d200640 1 -- 172.21.15.62:0/2974341365 wait complete. 2024-03-23T11:26:22.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.526+0000 7f5f0d200640 1 Processor -- start 2024-03-23T11:26:22.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.526+0000 7f5f0d200640 1 -- start start 2024-03-23T11:26:22.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.527+0000 7f5f0d200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f080b07f0 0x7f5f0815d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.527+0000 7f5f0d200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f080cd1a0 0x7f5f081edba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.527+0000 7f5f0d200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0815a8a0 0x7f5f081561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.527+0000 7f5f0d200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0815ae00 con 0x7f5f080b07f0 2024-03-23T11:26:22.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.527+0000 7f5f0d200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0815b610 con 0x7f5f080cd1a0 2024-03-23T11:26:22.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.527+0000 7f5f0d200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f08159360 con 0x7f5f0815a8a0 2024-03-23T11:26:22.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.528+0000 7f5f05e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f080b07f0 0x7f5f0815d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:22.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.528+0000 7f5f05e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f080b07f0 0x7f5f0815d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33056/0 (socket says 172.21.15.62:33056) 2024-03-23T11:26:22.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.528+0000 7f5f05e00640 1 -- 172.21.15.62:0/230170264 learned_addr learned my addr 172.21.15.62:0/230170264 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.528+0000 7f5f05e00640 1 -- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0815a8a0 msgr2=0x7f5f081561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.528+0000 7f5f05e00640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0815a8a0 0x7f5f081561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.528+0000 7f5f05e00640 1 -- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f080cd1a0 msgr2=0x7f5f081edba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.528+0000 7f5f05e00640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f080cd1a0 0x7f5f081edba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.528+0000 7f5f05e00640 1 -- 172.21.15.62:0/230170264 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f00009b30 con 0x7f5f080b07f0 2024-03-23T11:26:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.528+0000 7f5f05e00640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f080b07f0 0x7f5f0815d710 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800d5f0 tx=0x7f5ef800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.528+0000 7f5efe000640 1 -- 172.21.15.62:0/230170264 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef80125f0 con 0x7f5f080b07f0 2024-03-23T11:26:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.529+0000 7f5efe000640 1 -- 172.21.15.62:0/230170264 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef8012750 con 0x7f5f080b07f0 2024-03-23T11:26:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.529+0000 7f5f0d200640 1 -- 172.21.15.62:0/230170264 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0815b050 con 0x7f5f080b07f0 2024-03-23T11:26:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.529+0000 7f5efe000640 1 -- 172.21.15.62:0/230170264 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8012b00 con 0x7f5f080b07f0 2024-03-23T11:26:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.529+0000 7f5f0d200640 1 -- 172.21.15.62:0/230170264 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0815dc50 con 0x7f5f080b07f0 2024-03-23T11:26:22.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.530+0000 7f5f0d200640 1 -- 172.21.15.62:0/230170264 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f080b33c0 con 0x7f5f080b07f0 2024-03-23T11:26:22.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.531+0000 7f5efe000640 1 -- 172.21.15.62:0/230170264 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5ef8012c60 con 0x7f5f080b07f0 2024-03-23T11:26:22.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.531+0000 7f5efe000640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee406a7a0 0x7f5ee406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:22.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.532+0000 7f5efe000640 1 -- 172.21.15.62:0/230170264 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5ef8088e40 con 0x7f5f080b07f0 2024-03-23T11:26:22.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.532+0000 7f5f05400640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee406a7a0 0x7f5ee406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:22.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.533+0000 7f5f05400640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee406a7a0 0x7f5ee406cc60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5f080b1100 tx=0x7f5ef0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:22.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.536+0000 7f5efe000640 1 -- 172.21.15.62:0/230170264 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5ef8054640 con 0x7f5f080b07f0 2024-03-23T11:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:22.737+0000 7f5f0d200640 1 -- 172.21.15.62:0/230170264 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5f08066c90 con 0x7f5f080b07f0 2024-03-23T11:26:23.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.628+0000 7f5efe000640 1 -- 172.21.15.62:0/230170264 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v86) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5ef8057c60 con 0x7f5f080b07f0 2024-03-23T11:26:23.622 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' created 2024-03-23T11:26:23.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.634+0000 7f5f0d200640 1 -- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee406a7a0 msgr2=0x7f5ee406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:23.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.634+0000 7f5f0d200640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee406a7a0 0x7f5ee406cc60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5f080b1100 tx=0x7f5ef0008040 comp rx=0 tx=0).stop 2024-03-23T11:26:23.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.634+0000 7f5f0d200640 1 -- 172.21.15.62:0/230170264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f080b07f0 msgr2=0x7f5f0815d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:23.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.634+0000 7f5f0d200640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f080b07f0 0x7f5f0815d710 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800d5f0 tx=0x7f5ef800dab0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.635+0000 7f5f0d200640 1 -- 172.21.15.62:0/230170264 shutdown_connections 2024-03-23T11:26:23.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.635+0000 7f5f0d200640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5ee406a7a0 0x7f5ee406cc60 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.635+0000 7f5f0d200640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f0815a8a0 0x7f5f081561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.635+0000 7f5f0d200640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f080cd1a0 0x7f5f081edba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.635+0000 7f5f0d200640 1 --2- 172.21.15.62:0/230170264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f080b07f0 0x7f5f0815d710 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.635+0000 7f5f0d200640 1 -- 172.21.15.62:0/230170264 >> 172.21.15.62:0/230170264 conn(0x7f5f080b17d0 msgr2=0x7f5f080b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:23.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.635+0000 7f5f0d200640 1 -- 172.21.15.62:0/230170264 shutdown_connections 2024-03-23T11:26:23.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.635+0000 7f5f0d200640 1 -- 172.21.15.62:0/230170264 wait complete. 2024-03-23T11:26:23.654 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T11:26:23.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.878+0000 7fd203e00640 1 Processor -- start 2024-03-23T11:26:23.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.878+0000 7fd203e00640 1 -- start start 2024-03-23T11:26:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.880+0000 7fd203e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd2040af970 0x7fd2040cab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.880+0000 7fd203e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2040c9d80 con 0x7fd2040cef10 2024-03-23T11:26:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.880+0000 7fd203e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2040ca060 con 0x7fd2040cb080 2024-03-23T11:26:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.880+0000 7fd203e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2040ca340 con 0x7fd2040af970 2024-03-23T11:26:23.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.880+0000 7fd203400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd2040cef10 0x7fd2040b0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43562/0 (socket says 172.21.15.62:43562) 2024-03-23T11:26:23.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.880+0000 7fd203400640 1 -- 172.21.15.62:0/3088977267 learned_addr learned my addr 172.21.15.62:0/3088977267 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:23.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd202a00640 1 --2- 172.21.15.62:0/3088977267 >> v2:172.21.15.120:3300/0 conn(0x7fd2040af970 0x7fd2040cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:23.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777178886 0 0) 0x7fd2040c9d80 con 0x7fd2040cef10 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e8003520 con 0x7fd2040cef10 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1303218135 0 0) 0x7fd1e8003520 con 0x7fd2040cef10 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 >> v2:172.21.15.120:3300/0 conn(0x7fd2040af970 msgr2=0x7fd2040cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 --2- 172.21.15.62:0/3088977267 >> v2:172.21.15.120:3300/0 conn(0x7fd2040af970 0x7fd2040cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 >> v1:172.21.15.120:6789/0 conn(0x7fd2040cb080 legacy=0x7fd2040ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2040ca620 con 0x7fd2040cef10 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fd1fc0019f0 con 0x7fd2040cef10 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 >> v1:172.21.15.62:6789/0 conn(0x7fd2040cef10 legacy=0x7fd2040b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 --2- 172.21.15.62:0/3088977267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1e8003970 0x7fd1e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 --2- 172.21.15.62:0/3088977267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e80047c0 0x7fd1e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 --2- 172.21.15.62:0/3088977267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1e8005180 0x7fd1e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040ca340 con 0x7fd1e8005180 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e800c120 con 0x7fd1e8003970 2024-03-23T11:26:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.881+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e800c2e0 con 0x7fd1e80047c0 2024-03-23T11:26:23.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.882+0000 7fd203400640 1 --2- 172.21.15.62:0/3088977267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1e8005180 0x7fd1e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:23.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.882+0000 7fd202a00640 1 --2- 172.21.15.62:0/3088977267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1e8003970 0x7fd1e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:23.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.882+0000 7fd203400640 1 -- 172.21.15.62:0/3088977267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e80047c0 msgr2=0x7fd1e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:23.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.882+0000 7fd203400640 1 --2- 172.21.15.62:0/3088977267 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e80047c0 0x7fd1e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.882+0000 7fd203400640 1 -- 172.21.15.62:0/3088977267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1e8003970 msgr2=0x7fd1e8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:23.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.882+0000 7fd203400640 1 --2- 172.21.15.62:0/3088977267 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1e8003970 0x7fd1e8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.882+0000 7fd203400640 1 -- 172.21.15.62:0/3088977267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e8003520 con 0x7fd1e8005180 2024-03-23T11:26:23.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.883+0000 7fd203400640 1 -- 172.21.15.62:0/3088977267 reap_dead start 2024-03-23T11:26:23.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.883+0000 7fd203400640 1 --2- 172.21.15.62:0/3088977267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1e8005180 0x7fd1e800b980 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00a830 tx=0x7fd1e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3b8467dbf6d5ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:23.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.884+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0072d0 con 0x7fd1e8005180 2024-03-23T11:26:23.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.884+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1fc02f070 con 0x7fd1e8005180 2024-03-23T11:26:23.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.884+0000 7fd201600640 1 -- 172.21.15.62:0/3088977267 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0076b0 con 0x7fd1e8005180 2024-03-23T11:26:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.884+0000 7fd203e00640 1 -- 172.21.15.62:0/3088977267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1e8005180 msgr2=0x7fd1e800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.884+0000 7fd203e00640 1 --2- 172.21.15.62:0/3088977267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1e8005180 0x7fd1e800b980 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00a830 tx=0x7fd1e80057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.884+0000 7fd203e00640 1 -- 172.21.15.62:0/3088977267 shutdown_connections 2024-03-23T11:26:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.884+0000 7fd203e00640 1 --2- 172.21.15.62:0/3088977267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1e8005180 0x7fd1e800b980 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.884+0000 7fd203e00640 1 -- 172.21.15.62:0/3088977267 >> 172.21.15.62:0/3088977267 conn(0x7fd2040989d0 msgr2=0x7fd2040cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.885+0000 7fd203e00640 1 -- 172.21.15.62:0/3088977267 shutdown_connections 2024-03-23T11:26:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.885+0000 7fd203e00640 1 -- 172.21.15.62:0/3088977267 wait complete. 2024-03-23T11:26:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.886+0000 7fd203e00640 1 Processor -- start 2024-03-23T11:26:23.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.887+0000 7fd203e00640 1 -- start start 2024-03-23T11:26:23.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.887+0000 7fd203e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e8005180 0x7fd20415bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.887+0000 7fd203e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2040c9200 0x7fd2040c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2041f2560 0x7fd2041f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040c8d70 con 0x7fd2041f2560 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040c81d0 con 0x7fd2040c9200 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20415c0f0 con 0x7fd1e8005180 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2041f2560 0x7fd2041f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2041f2560 0x7fd2041f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33074/0 (socket says 172.21.15.62:33074) 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203400640 1 -- 172.21.15.62:0/1611876209 learned_addr learned my addr 172.21.15.62:0/1611876209 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd202a00640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e8005180 0x7fd20415bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd202000640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2040c9200 0x7fd2040c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203400640 1 -- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e8005180 msgr2=0x7fd20415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203400640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e8005180 0x7fd20415bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203400640 1 -- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2040c9200 msgr2=0x7fd2040c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:23.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203400640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2040c9200 0x7fd2040c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:23.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd203400640 1 -- 172.21.15.62:0/1611876209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2040c8440 con 0x7fd2041f2560 2024-03-23T11:26:23.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.888+0000 7fd202000640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2040c9200 0x7fd2040c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:23.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.889+0000 7fd203400640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2041f2560 0x7fd2041f2a10 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00a920 tx=0x7fd1fc001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:23.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.889+0000 7fd1faa00640 1 -- 172.21.15.62:0/1611876209 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc007400 con 0x7fd2041f2560 2024-03-23T11:26:23.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.889+0000 7fd1faa00640 1 -- 172.21.15.62:0/1611876209 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1fc007560 con 0x7fd2041f2560 2024-03-23T11:26:23.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.889+0000 7fd1faa00640 1 -- 172.21.15.62:0/1611876209 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0078d0 con 0x7fd2041f2560 2024-03-23T11:26:23.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.889+0000 7fd203e00640 1 -- 172.21.15.62:0/1611876209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2040c8720 con 0x7fd2041f2560 2024-03-23T11:26:23.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.889+0000 7fd203e00640 1 -- 172.21.15.62:0/1611876209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2040b0540 con 0x7fd2041f2560 2024-03-23T11:26:23.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.890+0000 7fd203e00640 1 -- 172.21.15.62:0/1611876209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2040ce440 con 0x7fd2041f2560 2024-03-23T11:26:23.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.892+0000 7fd1faa00640 1 -- 172.21.15.62:0/1611876209 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd1fc031030 con 0x7fd2041f2560 2024-03-23T11:26:23.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.893+0000 7fd1faa00640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1e406a7d0 0x7fd1e406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:23.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.893+0000 7fd1faa00640 1 -- 172.21.15.62:0/1611876209 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd1fc0aa760 con 0x7fd2041f2560 2024-03-23T11:26:23.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.893+0000 7fd202a00640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1e406a7d0 0x7fd1e406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:23.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.894+0000 7fd202a00640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1e406a7d0 0x7fd1e406cc90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4009990 tx=0x7fd1f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:23.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:23.898+0000 7fd1faa00640 1 -- 172.21.15.62:0/1611876209 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1fc075cc0 con 0x7fd2041f2560 2024-03-23T11:26:24.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.097+0000 7fd203e00640 1 -- 172.21.15.62:0/1611876209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd2040672c0 con 0x7fd2041f2560 2024-03-23T11:26:24.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.639+0000 7fd1faa00640 1 -- 172.21.15.62:0/1611876209 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v87) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd1fc0792e0 con 0x7fd2041f2560 2024-03-23T11:26:24.633 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' created 2024-03-23T11:26:24.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.647+0000 7fd203e00640 1 -- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1e406a7d0 msgr2=0x7fd1e406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:24.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.647+0000 7fd203e00640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1e406a7d0 0x7fd1e406cc90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4009990 tx=0x7fd1f4008040 comp rx=0 tx=0).stop 2024-03-23T11:26:24.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.647+0000 7fd203e00640 1 -- 172.21.15.62:0/1611876209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2041f2560 msgr2=0x7fd2041f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:24.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.647+0000 7fd203e00640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2041f2560 0x7fd2041f2a10 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00a920 tx=0x7fd1fc001d80 comp rx=0 tx=0).stop 2024-03-23T11:26:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.649+0000 7fd203e00640 1 -- 172.21.15.62:0/1611876209 shutdown_connections 2024-03-23T11:26:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.649+0000 7fd203e00640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd1e406a7d0 0x7fd1e406cc90 secure :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4009990 tx=0x7fd1f4008040 comp rx=0 tx=0).stop 2024-03-23T11:26:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.649+0000 7fd203e00640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2041f2560 0x7fd2041f2a10 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.649+0000 7fd203e00640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2040c9200 0x7fd2040c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.649+0000 7fd203e00640 1 --2- 172.21.15.62:0/1611876209 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1e8005180 0x7fd20415bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.649+0000 7fd203e00640 1 -- 172.21.15.62:0/1611876209 >> 172.21.15.62:0/1611876209 conn(0x7fd2040989d0 msgr2=0x7fd20406eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:24.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.651+0000 7fd203e00640 1 -- 172.21.15.62:0/1611876209 shutdown_connections 2024-03-23T11:26:24.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.652+0000 7fd203e00640 1 -- 172.21.15.62:0/1611876209 wait complete. 2024-03-23T11:26:24.675 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T11:26:24.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.866+0000 7f2700c00640 1 Processor -- start 2024-03-23T11:26:24.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.866+0000 7f2700c00640 1 -- start start 2024-03-23T11:26:24.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.868+0000 7f2700c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f26fc06b380 0x7f26fc15da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:24.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.868+0000 7f2700c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc0b1ef0 con 0x7f26fc15a620 2024-03-23T11:26:24.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.868+0000 7f2700c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc0ad680 con 0x7f26fc15a240 2024-03-23T11:26:24.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.868+0000 7f2700c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26fc0acde0 con 0x7f26fc06b380 2024-03-23T11:26:24.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.868+0000 7f26f9400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f26fc15a620 0x7f26fc06ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43570/0 (socket says 172.21.15.62:43570) 2024-03-23T11:26:24.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.868+0000 7f26f9400640 1 -- 172.21.15.62:0/1605570678 learned_addr learned my addr 172.21.15.62:0/1605570678 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:24.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.869+0000 7f26fa800640 1 --2- 172.21.15.62:0/1605570678 >> v2:172.21.15.62:3300/0 conn(0x7f26fc06b380 0x7f26fc15da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:24.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.869+0000 7f26f8a00640 1 -- 172.21.15.62:0/1605570678 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128729817 0 0) 0x7f26fc0b1ef0 con 0x7f26fc15a620 2024-03-23T11:26:24.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.869+0000 7f26f8a00640 1 -- 172.21.15.62:0/1605570678 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26dc003520 con 0x7f26fc15a620 2024-03-23T11:26:24.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.869+0000 7f26fa800640 1 -- 172.21.15.62:0/1605570678 >> v1:172.21.15.120:6789/0 conn(0x7f26fc15a240 legacy=0x7f26fc0ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:24.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.869+0000 7f26fa800640 1 -- 172.21.15.62:0/1605570678 >> v1:172.21.15.62:6789/0 conn(0x7f26fc15a620 legacy=0x7f26fc06ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:24.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.869+0000 7f26fa800640 1 -- 172.21.15.62:0/1605570678 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc0ad0c0 con 0x7f26fc06b380 2024-03-23T11:26:24.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.870+0000 7f26fa800640 1 --2- 172.21.15.62:0/1605570678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26fc06b380 0x7f26fc15da70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f26f40097a0 tx=0x7f26f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b9f90ede3607d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:24.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.871+0000 7f26f8a00640 1 -- 172.21.15.62:0/1605570678 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4031640 con 0x7f26fc06b380 2024-03-23T11:26:24.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.871+0000 7f26f8a00640 1 -- 172.21.15.62:0/1605570678 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f40317a0 con 0x7f26fc06b380 2024-03-23T11:26:24.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.871+0000 7f26f8a00640 1 -- 172.21.15.62:0/1605570678 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4031b30 con 0x7f26fc06b380 2024-03-23T11:26:24.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.871+0000 7f2700c00640 1 -- 172.21.15.62:0/1605570678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26fc06b380 msgr2=0x7f26fc15da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:24.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.871+0000 7f2700c00640 1 --2- 172.21.15.62:0/1605570678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26fc06b380 0x7f26fc15da70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f26f40097a0 tx=0x7f26f402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:26:24.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.871+0000 7f2700c00640 1 -- 172.21.15.62:0/1605570678 shutdown_connections 2024-03-23T11:26:24.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.871+0000 7f2700c00640 1 --2- 172.21.15.62:0/1605570678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26fc06b380 0x7f26fc15da70 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:24.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.871+0000 7f2700c00640 1 -- 172.21.15.62:0/1605570678 >> 172.21.15.62:0/1605570678 conn(0x7f26fc070b50 msgr2=0x7f26fc070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:24.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.871+0000 7f2700c00640 1 -- 172.21.15.62:0/1605570678 shutdown_connections 2024-03-23T11:26:24.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.872+0000 7f2700c00640 1 -- 172.21.15.62:0/1605570678 wait complete. 2024-03-23T11:26:24.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.873+0000 7f2700c00640 1 Processor -- start 2024-03-23T11:26:24.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.873+0000 7f2700c00640 1 -- start start 2024-03-23T11:26:24.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.874+0000 7f2700c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26fc06b380 0x7f26fc154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:24.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.874+0000 7f2700c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26fc15a240 0x7f26fc157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:24.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.874+0000 7f2700c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26fc15a620 0x7f26fc15efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:24.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.874+0000 7f2700c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc158400 con 0x7f26fc15a620 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.874+0000 7f2700c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc1586e0 con 0x7f26fc15a240 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.874+0000 7f2700c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc1589c0 con 0x7f26fc06b380 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.875+0000 7f26fa800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26fc15a620 0x7f26fc15efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.875+0000 7f26fa800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26fc15a620 0x7f26fc15efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33088/0 (socket says 172.21.15.62:33088) 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.875+0000 7f26fa800640 1 -- 172.21.15.62:0/4082440995 learned_addr learned my addr 172.21.15.62:0/4082440995 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.875+0000 7f26f9400640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26fc15a240 0x7f26fc157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.875+0000 7f26f9e00640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26fc06b380 0x7f26fc154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.875+0000 7f26fa800640 1 -- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26fc06b380 msgr2=0x7f26fc154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.875+0000 7f26fa800640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26fc06b380 0x7f26fc154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.875+0000 7f26fa800640 1 -- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26fc15a240 msgr2=0x7f26fc157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.875+0000 7f26fa800640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26fc15a240 0x7f26fc157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.875+0000 7f26fa800640 1 -- 172.21.15.62:0/4082440995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f4009400 con 0x7f26fc15a620 2024-03-23T11:26:24.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.875+0000 7f26f9e00640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26fc06b380 0x7f26fc154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:24.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.876+0000 7f26fa800640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26fc15a620 0x7f26fc15efd0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f26f4034040 tx=0x7f26f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:24.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.876+0000 7f26f2000640 1 -- 172.21.15.62:0/4082440995 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f403abe0 con 0x7f26fc15a620 2024-03-23T11:26:24.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.876+0000 7f26f2000640 1 -- 172.21.15.62:0/4082440995 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f403ad60 con 0x7f26fc15a620 2024-03-23T11:26:24.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.876+0000 7f26f2000640 1 -- 172.21.15.62:0/4082440995 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4044630 con 0x7f26fc15a620 2024-03-23T11:26:24.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.876+0000 7f2700c00640 1 -- 172.21.15.62:0/4082440995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc158ca0 con 0x7f26fc15a620 2024-03-23T11:26:24.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.876+0000 7f2700c00640 1 -- 172.21.15.62:0/4082440995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc154ec0 con 0x7f26fc15a620 2024-03-23T11:26:24.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.877+0000 7f2700c00640 1 -- 172.21.15.62:0/4082440995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc065b40 con 0x7f26fc15a620 2024-03-23T11:26:24.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.878+0000 7f26f2000640 1 -- 172.21.15.62:0/4082440995 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f26f402f050 con 0x7f26fc15a620 2024-03-23T11:26:24.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.879+0000 7f26f2000640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26d806a7a0 0x7f26d806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:24.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.879+0000 7f26f2000640 1 -- 172.21.15.62:0/4082440995 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f26f40aac50 con 0x7f26fc15a620 2024-03-23T11:26:24.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.880+0000 7f26f9e00640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26d806a7a0 0x7f26d806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:24.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.881+0000 7f26f9e00640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26d806a7a0 0x7f26d806cc60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f26ec000f60 tx=0x7f26ec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:24.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:24.884+0000 7f26f2000640 1 -- 172.21.15.62:0/4082440995 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26f4075fa0 con 0x7f26fc15a620 2024-03-23T11:26:25.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.128+0000 7f2700c00640 1 -- 172.21.15.62:0/4082440995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f26fc066c90 con 0x7f26fc15a620 2024-03-23T11:26:25.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.138+0000 7f26f2000640 1 -- 172.21.15.62:0/4082440995 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v54) v1 ==== 174+0+0 (secure 0 0 0) 0x7f26f40350c0 con 0x7f26fc15a620 2024-03-23T11:26:25.136 INFO:teuthology.orchestra.run.smithi062.stderr:new fs with metadata pool 12 and data pool 13 2024-03-23T11:26:25.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.145+0000 7f2700c00640 1 -- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26d806a7a0 msgr2=0x7f26d806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.145+0000 7f2700c00640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26d806a7a0 0x7f26d806cc60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f26ec000f60 tx=0x7f26ec009040 comp rx=0 tx=0).stop 2024-03-23T11:26:25.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.145+0000 7f2700c00640 1 -- 172.21.15.62:0/4082440995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26fc15a620 msgr2=0x7f26fc15efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.145+0000 7f2700c00640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26fc15a620 0x7f26fc15efd0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f26f4034040 tx=0x7f26f40046c0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.145+0000 7f2700c00640 1 -- 172.21.15.62:0/4082440995 shutdown_connections 2024-03-23T11:26:25.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.145+0000 7f2700c00640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26d806a7a0 0x7f26d806cc60 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.145+0000 7f2700c00640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26fc15a620 0x7f26fc15efd0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.145+0000 7f2700c00640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26fc15a240 0x7f26fc157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.145+0000 7f2700c00640 1 --2- 172.21.15.62:0/4082440995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26fc06b380 0x7f26fc154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.145+0000 7f2700c00640 1 -- 172.21.15.62:0/4082440995 >> 172.21.15.62:0/4082440995 conn(0x7f26fc070b50 msgr2=0x7f26fc15e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:25.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.145+0000 7f2700c00640 1 -- 172.21.15.62:0/4082440995 shutdown_connections 2024-03-23T11:26:25.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.146+0000 7f2700c00640 1 -- 172.21.15.62:0/4082440995 wait complete. 2024-03-23T11:26:25.168 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:26:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.359+0000 7fbe5f800640 1 Processor -- start 2024-03-23T11:26:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.360+0000 7fbe5f800640 1 -- start start 2024-03-23T11:26:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.361+0000 7fbe5f800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe5815cdc0 0x7fbe580d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.361+0000 7fbe5f800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe580cf650 0x7fbe580cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.361+0000 7fbe5f800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5806bfc0 con 0x7fbe5815c9e0 2024-03-23T11:26:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.361+0000 7fbe5f800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe580d0020 con 0x7fbe5815cdc0 2024-03-23T11:26:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.361+0000 7fbe5f800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe580ce140 con 0x7fbe580cf650 2024-03-23T11:26:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.361+0000 7fbe57e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe5815cdc0 0x7fbe580d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.361+0000 7fbe57e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe5815cdc0 0x7fbe580d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33104/0 (socket says 172.21.15.62:33104) 2024-03-23T11:26:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.361+0000 7fbe57e00640 1 -- 172.21.15.62:0/1380300859 learned_addr learned my addr 172.21.15.62:0/1380300859 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.362+0000 7fbe5d800640 1 --2- 172.21.15.62:0/1380300859 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe580cf650 0x7fbe580cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.362+0000 7fbe57e00640 1 -- 172.21.15.62:0/1380300859 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe580cf650 msgr2=0x7fbe580cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:25.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.362+0000 7fbe57e00640 1 --2- 172.21.15.62:0/1380300859 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe580cf650 0x7fbe580cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.362+0000 7fbe57e00640 1 -- 172.21.15.62:0/1380300859 >> v1:172.21.15.120:6789/0 conn(0x7fbe5815c9e0 legacy=0x7fbe581607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:25.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.362+0000 7fbe57e00640 1 -- 172.21.15.62:0/1380300859 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe580b5260 con 0x7fbe5815cdc0 2024-03-23T11:26:25.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.363+0000 7fbe57e00640 1 --2- 172.21.15.62:0/1380300859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5815cdc0 0x7fbe580d1f20 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009870 tx=0x7fbe4002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7643001878a125c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:25.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.363+0000 7fbe57400640 1 -- 172.21.15.62:0/1380300859 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40031640 con 0x7fbe5815cdc0 2024-03-23T11:26:25.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.363+0000 7fbe57400640 1 -- 172.21.15.62:0/1380300859 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe400317a0 con 0x7fbe5815cdc0 2024-03-23T11:26:25.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.363+0000 7fbe57400640 1 -- 172.21.15.62:0/1380300859 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40031b30 con 0x7fbe5815cdc0 2024-03-23T11:26:25.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.364+0000 7fbe5f800640 1 -- 172.21.15.62:0/1380300859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5815cdc0 msgr2=0x7fbe580d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.364+0000 7fbe5f800640 1 --2- 172.21.15.62:0/1380300859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5815cdc0 0x7fbe580d1f20 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009870 tx=0x7fbe4002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.364+0000 7fbe5f800640 1 -- 172.21.15.62:0/1380300859 shutdown_connections 2024-03-23T11:26:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.364+0000 7fbe5f800640 1 --2- 172.21.15.62:0/1380300859 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe580cf650 0x7fbe580cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.364+0000 7fbe5f800640 1 --2- 172.21.15.62:0/1380300859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5815cdc0 0x7fbe580d1f20 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.364+0000 7fbe5f800640 1 -- 172.21.15.62:0/1380300859 >> 172.21.15.62:0/1380300859 conn(0x7fbe58071570 msgr2=0x7fbe58071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.364+0000 7fbe5f800640 1 -- 172.21.15.62:0/1380300859 shutdown_connections 2024-03-23T11:26:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.364+0000 7fbe5f800640 1 -- 172.21.15.62:0/1380300859 wait complete. 2024-03-23T11:26:25.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.366+0000 7fbe5f800640 1 Processor -- start 2024-03-23T11:26:25.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.366+0000 7fbe5f800640 1 -- start start 2024-03-23T11:26:25.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.367+0000 7fbe5f800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe580cf650 0x7fbe580c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.367+0000 7fbe5f800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe5815c9e0 0x7fbe580cc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.367+0000 7fbe5f800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5815cdc0 0x7fbe580b7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.367+0000 7fbe5f800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe580b7bd0 con 0x7fbe5815cdc0 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.367+0000 7fbe5f800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe58149170 con 0x7fbe5815c9e0 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.367+0000 7fbe5f800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe580cc770 con 0x7fbe580cf650 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.367+0000 7fbe5d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5815cdc0 0x7fbe580b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.367+0000 7fbe5d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5815cdc0 0x7fbe580b7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33118/0 (socket says 172.21.15.62:33118) 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.367+0000 7fbe5d800640 1 -- 172.21.15.62:0/3821587784 learned_addr learned my addr 172.21.15.62:0/3821587784 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.367+0000 7fbe5ce00640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe580cf650 0x7fbe580c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe57e00640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe5815c9e0 0x7fbe580cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe5d800640 1 -- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe580cf650 msgr2=0x7fbe580c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe5d800640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe580cf650 0x7fbe580c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe5d800640 1 -- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe5815c9e0 msgr2=0x7fbe580cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe5d800640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe5815c9e0 0x7fbe580cc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe5d800640 1 -- 172.21.15.62:0/3821587784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe400094d0 con 0x7fbe5815cdc0 2024-03-23T11:26:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe5ce00640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe580cf650 0x7fbe580c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe57e00640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe5815c9e0 0x7fbe580cc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe5d800640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5815cdc0 0x7fbe580b7690 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fbe50007e40 tx=0x7fbe5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe54c00640 1 -- 172.21.15.62:0/3821587784 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe50014620 con 0x7fbe5815cdc0 2024-03-23T11:26:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe54c00640 1 -- 172.21.15.62:0/3821587784 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5000ee90 con 0x7fbe5815cdc0 2024-03-23T11:26:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe54c00640 1 -- 172.21.15.62:0/3821587784 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe500149e0 con 0x7fbe5815cdc0 2024-03-23T11:26:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe5f800640 1 -- 172.21.15.62:0/3821587784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe58148840 con 0x7fbe5815cdc0 2024-03-23T11:26:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.368+0000 7fbe5f800640 1 -- 172.21.15.62:0/3821587784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe58148e00 con 0x7fbe5815cdc0 2024-03-23T11:26:25.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.370+0000 7fbe5f800640 1 -- 172.21.15.62:0/3821587784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe581492e0 con 0x7fbe5815cdc0 2024-03-23T11:26:25.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.371+0000 7fbe54c00640 1 -- 172.21.15.62:0/3821587784 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbe50014b40 con 0x7fbe5815cdc0 2024-03-23T11:26:25.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.372+0000 7fbe54c00640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe3406a7a0 0x7fbe3406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.372+0000 7fbe54c00640 1 -- 172.21.15.62:0/3821587784 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe50089810 con 0x7fbe5815cdc0 2024-03-23T11:26:25.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.372+0000 7fbe5ce00640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe3406a7a0 0x7fbe3406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.374+0000 7fbe5ce00640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe3406a7a0 0x7fbe3406cc60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbe48003eb0 tx=0x7fbe48009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:25.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.377+0000 7fbe54c00640 1 -- 172.21.15.62:0/3821587784 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbe50054b90 con 0x7fbe5815cdc0 2024-03-23T11:26:25.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.574+0000 7fbe5f800640 1 -- 172.21.15.62:0/3821587784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe58066c90 con 0x7fbe5815cdc0 2024-03-23T11:26:25.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.577+0000 7fbe54c00640 1 -- 172.21.15.62:0/3821587784 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16105 (secure 0 0 0) 0x7fbe500581b0 con 0x7fbe5815cdc0 2024-03-23T11:26:25.570 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:25.570 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:26:25.130407+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:26:22.794149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-23T11:26:24.150997+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:26:25.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 -- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe3406a7a0 msgr2=0x7fbe3406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe3406a7a0 0x7fbe3406cc60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbe48003eb0 tx=0x7fbe48009040 comp rx=0 tx=0).stop 2024-03-23T11:26:25.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 -- 172.21.15.62:0/3821587784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5815cdc0 msgr2=0x7fbe580b7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5815cdc0 0x7fbe580b7690 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fbe50007e40 tx=0x7fbe5000e4b0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 -- 172.21.15.62:0/3821587784 shutdown_connections 2024-03-23T11:26:25.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbe3406a7a0 0x7fbe3406cc60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe5815cdc0 0x7fbe580b7690 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbe5815c9e0 0x7fbe580cc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 --2- 172.21.15.62:0/3821587784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbe580cf650 0x7fbe580c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 -- 172.21.15.62:0/3821587784 >> 172.21.15.62:0/3821587784 conn(0x7fbe58071570 msgr2=0x7fbe5815c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:25.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 -- 172.21.15.62:0/3821587784 shutdown_connections 2024-03-23T11:26:25.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.579+0000 7fbe5f800640 1 -- 172.21.15.62:0/3821587784 wait complete. 2024-03-23T11:26:25.590 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:26:25.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.815+0000 7fb57f200640 1 Processor -- start 2024-03-23T11:26:25.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.815+0000 7fb57f200640 1 -- start start 2024-03-23T11:26:25.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.817+0000 7fb57f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5800ce960 0x7fb5800ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.817+0000 7fb57f200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5801608d0 con 0x7fb580160310 2024-03-23T11:26:25.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.817+0000 7fb57f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb580160bb0 con 0x7fb58015c830 2024-03-23T11:26:25.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.817+0000 7fb57f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5800addb0 con 0x7fb5800ce960 2024-03-23T11:26:25.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.817+0000 7fb57d400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb580160310 0x7fb5800ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43578/0 (socket says 172.21.15.62:43578) 2024-03-23T11:26:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.817+0000 7fb57d400640 1 -- 172.21.15.62:0/613474481 learned_addr learned my addr 172.21.15.62:0/613474481 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57e800640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5800ce960 0x7fb5800ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058862541 0 0) 0x7fb5801608d0 con 0x7fb580160310 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb564003520 con 0x7fb580160310 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2149356373 0 0) 0x7fb564003520 con 0x7fb580160310 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5800ce960 msgr2=0x7fb5800ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5800ce960 0x7fb5800ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 >> v1:172.21.15.120:6789/0 conn(0x7fb58015c830 legacy=0x7fb5800ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb580160e90 con 0x7fb580160310 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fb56c001670 con 0x7fb580160310 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 >> v1:172.21.15.62:6789/0 conn(0x7fb580160310 legacy=0x7fb5800ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5640039a0 0x7fb564003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5640047c0 0x7fb564004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5800ce960 0x7fb564005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56400ba30 con 0x7fb5640047c0 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56400bb70 con 0x7fb5640039a0 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.818+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56400bd30 con 0x7fb5800ce960 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.819+0000 7fb57d400640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5640047c0 0x7fb564004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.819+0000 7fb57de00640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5640039a0 0x7fb564003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.819+0000 7fb57e800640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5800ce960 0x7fb564005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.819+0000 7fb57d400640 1 -- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5800ce960 msgr2=0x7fb564005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.819+0000 7fb57d400640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5800ce960 0x7fb564005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.819+0000 7fb57d400640 1 -- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5640039a0 msgr2=0x7fb564003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.819+0000 7fb57d400640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5640039a0 0x7fb564003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.819+0000 7fb57d400640 1 -- 172.21.15.62:0/613474481 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb564003520 con 0x7fb5640047c0 2024-03-23T11:26:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.819+0000 7fb57de00640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5640039a0 0x7fb564003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.820+0000 7fb57d400640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5640047c0 0x7fb564004c40 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00c210 tx=0x7fb5640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447db8a02202c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.820+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c005ad0 con 0x7fb5640047c0 2024-03-23T11:26:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.820+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c005c30 con 0x7fb5640047c0 2024-03-23T11:26:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.820+0000 7fb57ca00640 1 -- 172.21.15.62:0/613474481 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c02c870 con 0x7fb5640047c0 2024-03-23T11:26:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.821+0000 7fb57f200640 1 -- 172.21.15.62:0/613474481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5640047c0 msgr2=0x7fb564004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.821+0000 7fb57f200640 1 --2- 172.21.15.62:0/613474481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5640047c0 0x7fb564004c40 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00c210 tx=0x7fb5640057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.821+0000 7fb57e800640 1 -- 172.21.15.62:0/613474481 reap_dead start 2024-03-23T11:26:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.821+0000 7fb57f200640 1 -- 172.21.15.62:0/613474481 shutdown_connections 2024-03-23T11:26:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.821+0000 7fb57f200640 1 -- 172.21.15.62:0/613474481 >> 172.21.15.62:0/613474481 conn(0x7fb580066050 msgr2=0x7fb580069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.821+0000 7fb57f200640 1 -- 172.21.15.62:0/613474481 shutdown_connections 2024-03-23T11:26:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.821+0000 7fb57f200640 1 -- 172.21.15.62:0/613474481 wait complete. 2024-03-23T11:26:25.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.823+0000 7fb57f200640 1 Processor -- start 2024-03-23T11:26:25.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.823+0000 7fb57f200640 1 -- start start 2024-03-23T11:26:25.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.824+0000 7fb57f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5640047c0 0x7fb580153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.824+0000 7fb57f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5801f3320 0x7fb580156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.824+0000 7fb57f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5801f37a0 0x7fb580156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.824+0000 7fb57f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb580155c10 con 0x7fb5801f3320 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.824+0000 7fb57f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5800ade50 con 0x7fb5640047c0 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.824+0000 7fb57f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5800ae0f0 con 0x7fb5801f37a0 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.824+0000 7fb57d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5801f3320 0x7fb580156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.824+0000 7fb57d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5801f3320 0x7fb580156780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33128/0 (socket says 172.21.15.62:33128) 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.824+0000 7fb57d400640 1 -- 172.21.15.62:0/207037148 learned_addr learned my addr 172.21.15.62:0/207037148 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.825+0000 7fb57de00640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5640047c0 0x7fb580153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.825+0000 7fb57e800640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5801f37a0 0x7fb580156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.825+0000 7fb57d400640 1 -- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5801f37a0 msgr2=0x7fb580156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.825+0000 7fb57d400640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5801f37a0 0x7fb580156cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.825+0000 7fb57d400640 1 -- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5640047c0 msgr2=0x7fb580153dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.825+0000 7fb57d400640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5640047c0 0x7fb580153dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.825+0000 7fb57d400640 1 -- 172.21.15.62:0/207037148 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5800ae3d0 con 0x7fb5801f3320 2024-03-23T11:26:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.825+0000 7fb57d400640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5801f3320 0x7fb580156780 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00c210 tx=0x7fb56c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.826+0000 7fb576000640 1 -- 172.21.15.62:0/207037148 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c039070 con 0x7fb5801f3320 2024-03-23T11:26:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.826+0000 7fb576000640 1 -- 172.21.15.62:0/207037148 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c02cc80 con 0x7fb5801f3320 2024-03-23T11:26:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.826+0000 7fb576000640 1 -- 172.21.15.62:0/207037148 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c03c040 con 0x7fb5801f3320 2024-03-23T11:26:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.826+0000 7fb57f200640 1 -- 172.21.15.62:0/207037148 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5800ae6b0 con 0x7fb5801f3320 2024-03-23T11:26:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.826+0000 7fb57f200640 1 -- 172.21.15.62:0/207037148 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb580154f90 con 0x7fb5801f3320 2024-03-23T11:26:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.827+0000 7fb57f200640 1 -- 172.21.15.62:0/207037148 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb580066bb0 con 0x7fb5801f3320 2024-03-23T11:26:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.828+0000 7fb576000640 1 -- 172.21.15.62:0/207037148 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb56c032070 con 0x7fb5801f3320 2024-03-23T11:26:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.828+0000 7fb576000640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb56006a7a0 0x7fb56006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.829+0000 7fb576000640 1 -- 172.21.15.62:0/207037148 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb56c0a9ff0 con 0x7fb5801f3320 2024-03-23T11:26:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.829+0000 7fb57de00640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb56006a7a0 0x7fb56006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:25.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.830+0000 7fb57de00640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb56006a7a0 0x7fb56006cc60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb570000f60 tx=0x7fb570009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:25.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:25.833+0000 7fb576000640 1 -- 172.21.15.62:0/207037148 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb56c0dd580 con 0x7fb5801f3320 2024-03-23T11:26:25.972 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-23T11:26:25.972 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-03-23T11:26:26.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.031+0000 7fb57f200640 1 -- 172.21.15.62:0/207037148 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb580069e00 con 0x7fb5801f3320 2024-03-23T11:26:26.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.034+0000 7fb576000640 1 -- 172.21.15.62:0/207037148 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16105 (secure 0 0 0) 0x7fb56c075420 con 0x7fb5801f3320 2024-03-23T11:26:26.027 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:26.027 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:26:25.130407+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:26:22.794149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-23T11:26:24.150997+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:26:26.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 -- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb56006a7a0 msgr2=0x7fb56006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:26.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb56006a7a0 0x7fb56006cc60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb570000f60 tx=0x7fb570009040 comp rx=0 tx=0).stop 2024-03-23T11:26:26.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 -- 172.21.15.62:0/207037148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5801f3320 msgr2=0x7fb580156780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:26.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5801f3320 0x7fb580156780 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00c210 tx=0x7fb56c001a00 comp rx=0 tx=0).stop 2024-03-23T11:26:26.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 -- 172.21.15.62:0/207037148 shutdown_connections 2024-03-23T11:26:26.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb56006a7a0 0x7fb56006cc60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:26.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5801f3320 0x7fb580156780 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:26.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb5801f37a0 0x7fb580156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:26.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 --2- 172.21.15.62:0/207037148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5640047c0 0x7fb580153dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:26.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 -- 172.21.15.62:0/207037148 >> 172.21.15.62:0/207037148 conn(0x7fb580066050 msgr2=0x7fb5800b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:26.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 -- 172.21.15.62:0/207037148 shutdown_connections 2024-03-23T11:26:26.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.037+0000 7fb57f200640 1 -- 172.21.15.62:0/207037148 wait complete. 2024-03-23T11:26:26.051 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T11:26:26.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.285+0000 7fae50e00640 1 Processor -- start 2024-03-23T11:26:26.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.285+0000 7fae50e00640 1 -- start start 2024-03-23T11:26:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.287+0000 7fae50e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c143dd0 0x7fae4c15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.287+0000 7fae50e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fae4c070090 0x7fae4c0704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.287+0000 7fae50e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae4c070a30 con 0x7fae4c0b2820 2024-03-23T11:26:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.287+0000 7fae50e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0cd140 con 0x7fae4c070090 2024-03-23T11:26:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.287+0000 7fae50e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0cd3e0 con 0x7fae4c143dd0 2024-03-23T11:26:26.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.287+0000 7fae4be00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fae4c0b2820 0x7fae4c0b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43588/0 (socket says 172.21.15.62:43588) 2024-03-23T11:26:26.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.287+0000 7fae4be00640 1 -- 172.21.15.62:0/1095709388 learned_addr learned my addr 172.21.15.62:0/1095709388 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4b400640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c143dd0 0x7fae4c15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae43e00640 1 --2- 172.21.15.62:0/1095709388 >> v2:172.21.15.120:3300/0 conn(0x7fae4c070090 0x7fae4c0704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709539033 0 0) 0x7fae4c070a30 con 0x7fae4c0b2820 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae30003520 con 0x7fae4c0b2820 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1185778591 0 0) 0x7fae30003520 con 0x7fae4c0b2820 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c143dd0 msgr2=0x7fae4c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c143dd0 0x7fae4c15abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 >> v2:172.21.15.120:3300/0 conn(0x7fae4c070090 msgr2=0x7fae4c0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 --2- 172.21.15.62:0/1095709388 >> v2:172.21.15.120:3300/0 conn(0x7fae4c070090 0x7fae4c0704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c0cd6c0 con 0x7fae4c0b2820 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fae440019f0 con 0x7fae4c0b2820 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 >> v1:172.21.15.62:6789/0 conn(0x7fae4c0b2820 legacy=0x7fae4c0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c143dd0 0x7fae30003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:26.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae30003f40 0x7fae300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.288+0000 7fae4aa00640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae30004d00 0x7fae30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.289+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0cd140 con 0x7fae30004d00 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.289+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0cd3e0 con 0x7fae30003f40 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.289+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3000b980 con 0x7fae4c143dd0 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.289+0000 7fae4be00640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae30004d00 0x7fae30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.289+0000 7fae4b400640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c143dd0 0x7fae30003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.289+0000 7fae43e00640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae30003f40 0x7fae300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.289+0000 7fae4be00640 1 -- 172.21.15.62:0/1095709388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c143dd0 msgr2=0x7fae30003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.289+0000 7fae4be00640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c143dd0 0x7fae30003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.289+0000 7fae4be00640 1 -- 172.21.15.62:0/1095709388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae30003f40 msgr2=0x7fae300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.289+0000 7fae4be00640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae30003f40 0x7fae300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:26.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.289+0000 7fae4be00640 1 -- 172.21.15.62:0/1095709388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae30003520 con 0x7fae30004d00 2024-03-23T11:26:26.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.291+0000 7fae4be00640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae30004d00 0x7fae30005180 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fae4400a780 tx=0x7fae300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be8efec641d16cc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:26.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.291+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae44034630 con 0x7fae30004d00 2024-03-23T11:26:26.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.291+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4402f070 con 0x7fae30004d00 2024-03-23T11:26:26.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.291+0000 7fae4aa00640 1 -- 172.21.15.62:0/1095709388 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae44034a10 con 0x7fae30004d00 2024-03-23T11:26:26.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.292+0000 7fae50e00640 1 -- 172.21.15.62:0/1095709388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae30004d00 msgr2=0x7fae30005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:26.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.292+0000 7fae50e00640 1 --2- 172.21.15.62:0/1095709388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae30004d00 0x7fae30005180 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fae4400a780 tx=0x7fae300057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:26.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.292+0000 7fae4be00640 1 -- 172.21.15.62:0/1095709388 reap_dead start 2024-03-23T11:26:26.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.292+0000 7fae50e00640 1 -- 172.21.15.62:0/1095709388 shutdown_connections 2024-03-23T11:26:26.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.292+0000 7fae50e00640 1 -- 172.21.15.62:0/1095709388 >> 172.21.15.62:0/1095709388 conn(0x7fae4c066050 msgr2=0x7fae4c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:26.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.292+0000 7fae50e00640 1 -- 172.21.15.62:0/1095709388 shutdown_connections 2024-03-23T11:26:26.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.292+0000 7fae50e00640 1 -- 172.21.15.62:0/1095709388 wait complete. 2024-03-23T11:26:26.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.294+0000 7fae50e00640 1 Processor -- start 2024-03-23T11:26:26.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.294+0000 7fae50e00640 1 -- start start 2024-03-23T11:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.295+0000 7fae50e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c15bb90 0x7fae4c15bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.295+0000 7fae50e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae4c15c4b0 0x7fae4c1418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.295+0000 7fae50e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae4c141e20 0x7fae4c1422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.295+0000 7fae50e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c140bd0 con 0x7fae4c15c4b0 2024-03-23T11:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.295+0000 7fae50e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c13f280 con 0x7fae4c141e20 2024-03-23T11:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.295+0000 7fae50e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c141450 con 0x7fae4c15bb90 2024-03-23T11:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae43e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae4c15c4b0 0x7fae4c1418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae43e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae4c15c4b0 0x7fae4c1418e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33152/0 (socket says 172.21.15.62:33152) 2024-03-23T11:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae43e00640 1 -- 172.21.15.62:0/802806842 learned_addr learned my addr 172.21.15.62:0/802806842 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:26.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae4b400640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c15bb90 0x7fae4c15bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:26.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae4be00640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae4c141e20 0x7fae4c1422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:26.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae43e00640 1 -- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c15bb90 msgr2=0x7fae4c15bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:26.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae43e00640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c15bb90 0x7fae4c15bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:26.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae43e00640 1 -- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae4c141e20 msgr2=0x7fae4c1422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:26.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae43e00640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae4c141e20 0x7fae4c1422a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:26.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae43e00640 1 -- 172.21.15.62:0/802806842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c1427e0 con 0x7fae4c15c4b0 2024-03-23T11:26:26.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae4b400640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c15bb90 0x7fae4c15bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:26.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.296+0000 7fae4be00640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae4c141e20 0x7fae4c1422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:26.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.297+0000 7fae43e00640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae4c15c4b0 0x7fae4c1418e0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fae34011630 tx=0x7fae34011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:26.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.297+0000 7fae43400640 1 -- 172.21.15.62:0/802806842 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3400f920 con 0x7fae4c15c4b0 2024-03-23T11:26:26.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.297+0000 7fae43400640 1 -- 172.21.15.62:0/802806842 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3400fa80 con 0x7fae4c15c4b0 2024-03-23T11:26:26.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.297+0000 7fae43400640 1 -- 172.21.15.62:0/802806842 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3400fdf0 con 0x7fae4c15c4b0 2024-03-23T11:26:26.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.297+0000 7fae50e00640 1 -- 172.21.15.62:0/802806842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4c13efa0 con 0x7fae4c15c4b0 2024-03-23T11:26:26.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.297+0000 7fae50e00640 1 -- 172.21.15.62:0/802806842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4c1410e0 con 0x7fae4c15c4b0 2024-03-23T11:26:26.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.298+0000 7fae50e00640 1 -- 172.21.15.62:0/802806842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4c066bb0 con 0x7fae4c15c4b0 2024-03-23T11:26:26.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.300+0000 7fae43400640 1 -- 172.21.15.62:0/802806842 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fae34015460 con 0x7fae4c15c4b0 2024-03-23T11:26:26.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.301+0000 7fae43400640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae2806a7d0 0x7fae2806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:26.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.301+0000 7fae43400640 1 -- 172.21.15.62:0/802806842 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae34089310 con 0x7fae4c15c4b0 2024-03-23T11:26:26.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.302+0000 7fae4b400640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae2806a7d0 0x7fae2806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:26.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.303+0000 7fae4b400640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae2806a7d0 0x7fae2806cc90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009a00 tx=0x7fae3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:26.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.306+0000 7fae43400640 1 -- 172.21.15.62:0/802806842 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae340546a0 con 0x7fae4c15c4b0 2024-03-23T11:26:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:26.556+0000 7fae50e00640 1 -- 172.21.15.62:0/802806842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fae4c141590 con 0x7fae4c15c4b0 2024-03-23T11:26:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.307+0000 7fae43400640 1 -- 172.21.15.62:0/802806842 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v56) v1 ==== 118+0+0 (secure 0 0 0) 0x7fae34057cc0 con 0x7fae4c15c4b0 2024-03-23T11:26:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 -- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae2806a7d0 msgr2=0x7fae2806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae2806a7d0 0x7fae2806cc90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009a00 tx=0x7fae3c008040 comp rx=0 tx=0).stop 2024-03-23T11:26:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 -- 172.21.15.62:0/802806842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae4c15c4b0 msgr2=0x7fae4c1418e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae4c15c4b0 0x7fae4c1418e0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fae34011630 tx=0x7fae34011af0 comp rx=0 tx=0).stop 2024-03-23T11:26:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 -- 172.21.15.62:0/802806842 shutdown_connections 2024-03-23T11:26:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fae2806a7d0 0x7fae2806cc90 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae4c15c4b0 0x7fae4c1418e0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fae4c141e20 0x7fae4c1422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 --2- 172.21.15.62:0/802806842 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fae4c15bb90 0x7fae4c15bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 -- 172.21.15.62:0/802806842 >> 172.21.15.62:0/802806842 conn(0x7fae4c066050 msgr2=0x7fae4c0ab840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 -- 172.21.15.62:0/802806842 shutdown_connections 2024-03-23T11:26:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.311+0000 7fae50e00640 1 -- 172.21.15.62:0/802806842 wait complete. 2024-03-23T11:26:27.325 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T11:26:27.326 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T11:26:27.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.549+0000 7f7a55a00640 1 Processor -- start 2024-03-23T11:26:27.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.549+0000 7f7a55a00640 1 -- start start 2024-03-23T11:26:27.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.550+0000 7f7a55a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7a500d01b0 0x7f7a500cb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:27.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.550+0000 7f7a55a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7a500cbc80 0x7f7a500cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:27.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.550+0000 7f7a55a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a500cb2b0 con 0x7f7a5015ce90 2024-03-23T11:26:27.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.551+0000 7f7a55a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a500aeaf0 con 0x7f7a500cbc80 2024-03-23T11:26:27.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.551+0000 7f7a55a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a500cc980 con 0x7f7a500d01b0 2024-03-23T11:26:27.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.551+0000 7f7a55000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7a500cbc80 0x7f7a500cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:27.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.551+0000 7f7a55000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7a500cbc80 0x7f7a500cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53962/0 (socket says 172.21.15.62:53962) 2024-03-23T11:26:27.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.551+0000 7f7a55000640 1 -- 172.21.15.62:0/1028587530 learned_addr learned my addr 172.21.15.62:0/1028587530 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:27.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.551+0000 7f7a47400640 1 --2- 172.21.15.62:0/1028587530 >> v2:172.21.15.120:3300/0 conn(0x7f7a500d01b0 0x7f7a500cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:27.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.551+0000 7f7a55000640 1 -- 172.21.15.62:0/1028587530 >> v2:172.21.15.120:3300/0 conn(0x7f7a500d01b0 msgr2=0x7f7a500cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:27.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.551+0000 7f7a55000640 1 --2- 172.21.15.62:0/1028587530 >> v2:172.21.15.120:3300/0 conn(0x7f7a500d01b0 0x7f7a500cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:27.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.551+0000 7f7a55000640 1 -- 172.21.15.62:0/1028587530 >> v1:172.21.15.120:6789/0 conn(0x7f7a5015ce90 legacy=0x7f7a500ae3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:27.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.551+0000 7f7a55000640 1 -- 172.21.15.62:0/1028587530 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a500ccc60 con 0x7f7a500cbc80 2024-03-23T11:26:27.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.551+0000 7f7a47400640 1 --2- 172.21.15.62:0/1028587530 >> v2:172.21.15.120:3300/0 conn(0x7f7a500d01b0 0x7f7a500cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:27.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.552+0000 7f7a55000640 1 --2- 172.21.15.62:0/1028587530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a500cbc80 0x7f7a500cc0b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f7a500cd5a0 tx=0x7f7a4802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e19de86b05a2b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:27.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.553+0000 7f7a4f400640 1 -- 172.21.15.62:0/1028587530 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a480317e0 con 0x7f7a500cbc80 2024-03-23T11:26:27.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.553+0000 7f7a4f400640 1 -- 172.21.15.62:0/1028587530 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a48031940 con 0x7f7a500cbc80 2024-03-23T11:26:27.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.553+0000 7f7a4f400640 1 -- 172.21.15.62:0/1028587530 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48031cd0 con 0x7f7a500cbc80 2024-03-23T11:26:27.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.553+0000 7f7a55a00640 1 -- 172.21.15.62:0/1028587530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a500cbc80 msgr2=0x7f7a500cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:27.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.553+0000 7f7a55a00640 1 --2- 172.21.15.62:0/1028587530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a500cbc80 0x7f7a500cc0b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f7a500cd5a0 tx=0x7f7a4802b880 comp rx=0 tx=0).stop 2024-03-23T11:26:27.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.553+0000 7f7a55a00640 1 -- 172.21.15.62:0/1028587530 shutdown_connections 2024-03-23T11:26:27.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.553+0000 7f7a55a00640 1 --2- 172.21.15.62:0/1028587530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a500cbc80 0x7f7a500cc0b0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:27.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.553+0000 7f7a55a00640 1 --2- 172.21.15.62:0/1028587530 >> v2:172.21.15.120:3300/0 conn(0x7f7a500d01b0 0x7f7a500cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:27.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.554+0000 7f7a55a00640 1 -- 172.21.15.62:0/1028587530 >> 172.21.15.62:0/1028587530 conn(0x7f7a500989d0 msgr2=0x7f7a500cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:27.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.554+0000 7f7a55a00640 1 -- 172.21.15.62:0/1028587530 shutdown_connections 2024-03-23T11:26:27.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.554+0000 7f7a55a00640 1 -- 172.21.15.62:0/1028587530 wait complete. 2024-03-23T11:26:27.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.555+0000 7f7a55a00640 1 Processor -- start 2024-03-23T11:26:27.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.555+0000 7f7a55a00640 1 -- start start 2024-03-23T11:26:27.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.556+0000 7f7a55a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a500cbc80 0x7f7a500b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.556+0000 7f7a55a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a500d01b0 0x7f7a500ad140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.556+0000 7f7a55a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a5015ce90 0x7f7a5014f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.556+0000 7f7a55a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5006e6e0 con 0x7f7a500d01b0 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.556+0000 7f7a55a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500b1ad0 con 0x7f7a500cbc80 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.556+0000 7f7a55a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500c3530 con 0x7f7a5015ce90 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.557+0000 7f7a47400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a500d01b0 0x7f7a500ad140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.557+0000 7f7a47400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a500d01b0 0x7f7a500ad140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53978/0 (socket says 172.21.15.62:53978) 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.557+0000 7f7a47400640 1 -- 172.21.15.62:0/3125426629 learned_addr learned my addr 172.21.15.62:0/3125426629 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.557+0000 7f7a55000640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a5015ce90 0x7f7a5014f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.557+0000 7f7a4fe00640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a500cbc80 0x7f7a500b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.557+0000 7f7a47400640 1 -- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a5015ce90 msgr2=0x7f7a5014f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.557+0000 7f7a47400640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a5015ce90 0x7f7a5014f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.557+0000 7f7a47400640 1 -- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a500cbc80 msgr2=0x7f7a500b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:27.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.557+0000 7f7a47400640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a500cbc80 0x7f7a500b1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:27.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.557+0000 7f7a47400640 1 -- 172.21.15.62:0/3125426629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a480094d0 con 0x7f7a500d01b0 2024-03-23T11:26:27.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.557+0000 7f7a55000640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a5015ce90 0x7f7a5014f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:27.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.558+0000 7f7a47400640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a500d01b0 0x7f7a500ad140 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c7f0 tx=0x7f7a3800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:27.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.558+0000 7f7a4cc00640 1 -- 172.21.15.62:0/3125426629 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3800cea0 con 0x7f7a500d01b0 2024-03-23T11:26:27.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.558+0000 7f7a4cc00640 1 -- 172.21.15.62:0/3125426629 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a38007a30 con 0x7f7a500d01b0 2024-03-23T11:26:27.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.558+0000 7f7a4cc00640 1 -- 172.21.15.62:0/3125426629 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38007dc0 con 0x7f7a500d01b0 2024-03-23T11:26:27.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.558+0000 7f7a55a00640 1 -- 172.21.15.62:0/3125426629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a500c3810 con 0x7f7a500d01b0 2024-03-23T11:26:27.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.558+0000 7f7a55a00640 1 -- 172.21.15.62:0/3125426629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a500c3dd0 con 0x7f7a500d01b0 2024-03-23T11:26:27.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.559+0000 7f7a55a00640 1 -- 172.21.15.62:0/3125426629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a500ccfb0 con 0x7f7a500d01b0 2024-03-23T11:26:27.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.560+0000 7f7a4cc00640 1 -- 172.21.15.62:0/3125426629 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7a380113f0 con 0x7f7a500d01b0 2024-03-23T11:26:27.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.561+0000 7f7a4cc00640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a2c06a7a0 0x7f7a2c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:27.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.561+0000 7f7a4cc00640 1 -- 172.21.15.62:0/3125426629 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a38088fb0 con 0x7f7a500d01b0 2024-03-23T11:26:27.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.562+0000 7f7a4fe00640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a2c06a7a0 0x7f7a2c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.563+0000 7f7a4fe00640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a2c06a7a0 0x7f7a2c06cc60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7a40007bf0 tx=0x7f7a40007240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:27.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.565+0000 7f7a4cc00640 1 -- 172.21.15.62:0/3125426629 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7a38054330 con 0x7f7a500d01b0 2024-03-23T11:26:27.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:27.801+0000 7f7a55a00640 1 -- 172.21.15.62:0/3125426629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7a500672c0 con 0x7f7a500d01b0 2024-03-23T11:26:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.319+0000 7f7a4cc00640 1 -- 172.21.15.62:0/3125426629 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7a3801f0c0 con 0x7f7a500d01b0 2024-03-23T11:26:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.323+0000 7f7a55a00640 1 -- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a2c06a7a0 msgr2=0x7f7a2c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.323+0000 7f7a55a00640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a2c06a7a0 0x7f7a2c06cc60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7a40007bf0 tx=0x7f7a40007240 comp rx=0 tx=0).stop 2024-03-23T11:26:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.323+0000 7f7a55a00640 1 -- 172.21.15.62:0/3125426629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a500d01b0 msgr2=0x7f7a500ad140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.323+0000 7f7a55a00640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a500d01b0 0x7f7a500ad140 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c7f0 tx=0x7f7a3800ccb0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.323+0000 7f7a55a00640 1 -- 172.21.15.62:0/3125426629 shutdown_connections 2024-03-23T11:26:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.323+0000 7f7a55a00640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a2c06a7a0 0x7f7a2c06cc60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.323+0000 7f7a55a00640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a500d01b0 0x7f7a500ad140 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.323+0000 7f7a55a00640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a5015ce90 0x7f7a5014f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.323+0000 7f7a55a00640 1 --2- 172.21.15.62:0/3125426629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a500cbc80 0x7f7a500b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.323+0000 7f7a55a00640 1 -- 172.21.15.62:0/3125426629 >> 172.21.15.62:0/3125426629 conn(0x7f7a500989d0 msgr2=0x7f7a500c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:28.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.324+0000 7f7a55a00640 1 -- 172.21.15.62:0/3125426629 shutdown_connections 2024-03-23T11:26:28.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.324+0000 7f7a55a00640 1 -- 172.21.15.62:0/3125426629 wait complete. 2024-03-23T11:26:28.338 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T11:26:28.338 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:26:28.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.548+0000 7fa3d2000640 1 Processor -- start 2024-03-23T11:26:28.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.548+0000 7fa3d2000640 1 -- start start 2024-03-23T11:26:28.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.550+0000 7fa3d2000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc143dd0 0x7fa3cc15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:28.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.550+0000 7fa3d2000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc0cd140 con 0x7fa3cc0b2f30 2024-03-23T11:26:28.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.550+0000 7fa3d2000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc0cd3e0 con 0x7fa3cc070090 2024-03-23T11:26:28.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.550+0000 7fa3d2000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0cd6c0 con 0x7fa3cc143dd0 2024-03-23T11:26:28.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.550+0000 7fa3d1600640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa3cc0b2f30 0x7fa3cc0ce480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37304/0 (socket says 172.21.15.62:37304) 2024-03-23T11:26:28.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.550+0000 7fa3d1600640 1 -- 172.21.15.62:0/2694663844 learned_addr learned my addr 172.21.15.62:0/2694663844 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:28.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3d0c00640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc143dd0 0x7fa3cc15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:28.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708613890 0 0) 0x7fa3cc0cd140 con 0x7fa3cc0b2f30 2024-03-23T11:26:28.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b0003520 con 0x7fa3cc0b2f30 2024-03-23T11:26:28.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 523394752 0 0) 0x7fa3b0003520 con 0x7fa3cc0b2f30 2024-03-23T11:26:28.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc143dd0 msgr2=0x7fa3cc15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:28.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3cbe00640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc143dd0 0x7fa3cc15abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 >> v1:172.21.15.120:6789/0 conn(0x7fa3cc070090 legacy=0x7fa3cc0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc0cd9a0 con 0x7fa3cc0b2f30 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fa3c40019f0 con 0x7fa3cc0b2f30 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 >> v1:172.21.15.62:6789/0 conn(0x7fa3cc0b2f30 legacy=0x7fa3cc0ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3cbe00640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 0x7fa3b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.551+0000 7fa3cbe00640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3b00047c0 0x7fa3b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3cbe00640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc143dd0 0x7fa3b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0cd6c0 con 0x7fa3b00039a0 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b000ba60 con 0x7fa3b00047c0 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b000bc20 con 0x7fa3cc143dd0 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3d0c00640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 0x7fa3b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3c3e00640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3b00047c0 0x7fa3b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:28.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3d0c00640 1 -- 172.21.15.62:0/2694663844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc143dd0 msgr2=0x7fa3b0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:28.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3d1600640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc143dd0 0x7fa3b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:28.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3d0c00640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc143dd0 0x7fa3b0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3d0c00640 1 -- 172.21.15.62:0/2694663844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3b00047c0 msgr2=0x7fa3b0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:28.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3d0c00640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3b00047c0 0x7fa3b0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.552+0000 7fa3d0c00640 1 -- 172.21.15.62:0/2694663844 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b0003520 con 0x7fa3b00039a0 2024-03-23T11:26:28.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.553+0000 7fa3d0c00640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 0x7fa3b0003dc0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009a90 tx=0x7fa3b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eab161be9c72664 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:28.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.554+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc02c680 con 0x7fa3b00039a0 2024-03-23T11:26:28.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.554+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc02c7e0 con 0x7fa3b00039a0 2024-03-23T11:26:28.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.554+0000 7fa3cbe00640 1 -- 172.21.15.62:0/2694663844 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc02cb70 con 0x7fa3b00039a0 2024-03-23T11:26:28.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.554+0000 7fa3d2000640 1 -- 172.21.15.62:0/2694663844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 msgr2=0x7fa3b0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:28.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.554+0000 7fa3d2000640 1 --2- 172.21.15.62:0/2694663844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 0x7fa3b0003dc0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009a90 tx=0x7fa3b00057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.554+0000 7fa3d1600640 1 -- 172.21.15.62:0/2694663844 reap_dead start 2024-03-23T11:26:28.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.554+0000 7fa3d2000640 1 -- 172.21.15.62:0/2694663844 shutdown_connections 2024-03-23T11:26:28.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.554+0000 7fa3d2000640 1 -- 172.21.15.62:0/2694663844 >> 172.21.15.62:0/2694663844 conn(0x7fa3cc066050 msgr2=0x7fa3cc069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:28.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.555+0000 7fa3d2000640 1 -- 172.21.15.62:0/2694663844 shutdown_connections 2024-03-23T11:26:28.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.555+0000 7fa3d2000640 1 -- 172.21.15.62:0/2694663844 wait complete. 2024-03-23T11:26:28.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.556+0000 7fa3d2000640 1 Processor -- start 2024-03-23T11:26:28.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.556+0000 7fa3d2000640 1 -- start start 2024-03-23T11:26:28.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.557+0000 7fa3d2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 0x7fa3cc159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:28.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.557+0000 7fa3d2000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc13cc40 0x7fa3cc1587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:28.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.557+0000 7fa3d2000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc0af6a0 0x7fa3cc0afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:28.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.557+0000 7fa3d2000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0b0090 con 0x7fa3b00039a0 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.557+0000 7fa3d2000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc15aa90 con 0x7fa3cc13cc40 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.557+0000 7fa3d2000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc13d440 con 0x7fa3cc0af6a0 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.557+0000 7fa3d0c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 0x7fa3cc159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3d0c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 0x7fa3cc159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53984/0 (socket says 172.21.15.62:53984) 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3d0c00640 1 -- 172.21.15.62:0/1688160075 learned_addr learned my addr 172.21.15.62:0/1688160075 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3d1600640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc0af6a0 0x7fa3cc0afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3c3e00640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc13cc40 0x7fa3cc1587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3d0c00640 1 -- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc0af6a0 msgr2=0x7fa3cc0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3d0c00640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc0af6a0 0x7fa3cc0afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3d0c00640 1 -- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc13cc40 msgr2=0x7fa3cc1587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3d0c00640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc13cc40 0x7fa3cc1587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3d0c00640 1 -- 172.21.15.62:0/1688160075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc13d720 con 0x7fa3b00039a0 2024-03-23T11:26:28.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3d1600640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc0af6a0 0x7fa3cc0afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:28.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3c3e00640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc13cc40 0x7fa3cc1587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:28.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.558+0000 7fa3d0c00640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 0x7fa3cc159fa0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009a90 tx=0x7fa3bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:28.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.559+0000 7fa3c9600640 1 -- 172.21.15.62:0/1688160075 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc02dea0 con 0x7fa3b00039a0 2024-03-23T11:26:28.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.559+0000 7fa3c9600640 1 -- 172.21.15.62:0/1688160075 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc003810 con 0x7fa3b00039a0 2024-03-23T11:26:28.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.559+0000 7fa3c9600640 1 -- 172.21.15.62:0/1688160075 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc003ba0 con 0x7fa3b00039a0 2024-03-23T11:26:28.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.559+0000 7fa3d2000640 1 -- 172.21.15.62:0/1688160075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc15ad30 con 0x7fa3b00039a0 2024-03-23T11:26:28.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.559+0000 7fa3d2000640 1 -- 172.21.15.62:0/1688160075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc13d0c0 con 0x7fa3b00039a0 2024-03-23T11:26:28.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.560+0000 7fa3d2000640 1 -- 172.21.15.62:0/1688160075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3cc066bb0 con 0x7fa3b00039a0 2024-03-23T11:26:28.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.561+0000 7fa3c9600640 1 -- 172.21.15.62:0/1688160075 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa3bc003d00 con 0x7fa3b00039a0 2024-03-23T11:26:28.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.561+0000 7fa3c9600640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa3a806a7d0 0x7fa3a806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:28.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.562+0000 7fa3c9600640 1 -- 172.21.15.62:0/1688160075 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3bc0aac00 con 0x7fa3b00039a0 2024-03-23T11:26:28.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.562+0000 7fa3c3e00640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa3a806a7d0 0x7fa3a806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:28.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.563+0000 7fa3c3e00640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa3a806a7d0 0x7fa3a806cc90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4000c00 tx=0x7fa3b4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:28.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.566+0000 7fa3c9600640 1 -- 172.21.15.62:0/1688160075 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa3bc076010 con 0x7fa3b00039a0 2024-03-23T11:26:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.795+0000 7fa3d2000640 1 -- 172.21.15.62:0/1688160075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3cc0b2cf0 con 0x7fa3b00039a0 2024-03-23T11:26:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.796+0000 7fa3c9600640 1 -- 172.21.15.62:0/1688160075 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fa3bc0044e0 con 0x7fa3b00039a0 2024-03-23T11:26:28.790 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:28.790 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":57,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:26:28.303857+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6154},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6154":{"gid":6154,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/3056962984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3056962984},{"type":"v1","addr":"172.21.15.62:6833","nonce":3056962984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:26:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 57 2024-03-23T11:26:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 -- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa3a806a7d0 msgr2=0x7fa3a806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa3a806a7d0 0x7fa3a806cc90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4000c00 tx=0x7fa3b4009040 comp rx=0 tx=0).stop 2024-03-23T11:26:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 -- 172.21.15.62:0/1688160075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 msgr2=0x7fa3cc159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 0x7fa3cc159fa0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009a90 tx=0x7fa3bc004730 comp rx=0 tx=0).stop 2024-03-23T11:26:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 -- 172.21.15.62:0/1688160075 shutdown_connections 2024-03-23T11:26:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa3a806a7d0 0x7fa3a806cc90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3cc0af6a0 0x7fa3cc0afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc13cc40 0x7fa3cc1587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 --2- 172.21.15.62:0/1688160075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3b00039a0 0x7fa3cc159fa0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 -- 172.21.15.62:0/1688160075 >> 172.21.15.62:0/1688160075 conn(0x7fa3cc066050 msgr2=0x7fa3cc0704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 -- 172.21.15.62:0/1688160075 shutdown_connections 2024-03-23T11:26:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:28.799+0000 7fa3d2000640 1 -- 172.21.15.62:0/1688160075 wait complete. 2024-03-23T11:26:28.810 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:26:29.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.032+0000 7ff6f7200640 1 Processor -- start 2024-03-23T11:26:29.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.032+0000 7ff6f7200640 1 -- start start 2024-03-23T11:26:29.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.034+0000 7ff6f7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6f8160c00 0x7ff6f806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.034+0000 7ff6f7200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6f80cde90 0x7ff6f80ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.034+0000 7ff6f7200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f80ce820 con 0x7ff6f80cda20 2024-03-23T11:26:29.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.034+0000 7ff6f7200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f80b1ad0 con 0x7ff6f80cde90 2024-03-23T11:26:29.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.034+0000 7ff6f7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f8160fe0 con 0x7ff6f8160c00 2024-03-23T11:26:29.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.034+0000 7ff6f5400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff6f80cda20 0x7ff6f806de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37310/0 (socket says 172.21.15.62:37310) 2024-03-23T11:26:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.034+0000 7ff6f5400640 1 -- 172.21.15.62:0/2351951698 learned_addr learned my addr 172.21.15.62:0/2351951698 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.034+0000 7ff6f6800640 1 --2- 172.21.15.62:0/2351951698 >> v2:172.21.15.62:3300/0 conn(0x7ff6f80cde90 0x7ff6f80ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.035+0000 7ff6f5e00640 1 --2- 172.21.15.62:0/2351951698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6f8160c00 0x7ff6f806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.035+0000 7ff6f4a00640 1 -- 172.21.15.62:0/2351951698 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681125911 0 0) 0x7ff6e4001110 con 0x7ff6f80cda20 2024-03-23T11:26:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.035+0000 7ff6f4a00640 1 -- 172.21.15.62:0/2351951698 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6dc003520 con 0x7ff6f80cda20 2024-03-23T11:26:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.035+0000 7ff6f6800640 1 -- 172.21.15.62:0/2351951698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6f8160c00 msgr2=0x7ff6f806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.035+0000 7ff6f6800640 1 --2- 172.21.15.62:0/2351951698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6f8160c00 0x7ff6f806d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.035+0000 7ff6f6800640 1 -- 172.21.15.62:0/2351951698 >> v1:172.21.15.62:6789/0 conn(0x7ff6f80cda20 legacy=0x7ff6f806de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.035+0000 7ff6f6800640 1 -- 172.21.15.62:0/2351951698 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f80b1510 con 0x7ff6f80cde90 2024-03-23T11:26:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.036+0000 7ff6f6800640 1 --2- 172.21.15.62:0/2351951698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f80cde90 0x7ff6f80ce2e0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009d90 tx=0x7ff6f002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b66924ba883ee33 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.037+0000 7ff6f4a00640 1 -- 172.21.15.62:0/2351951698 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f002cc50 con 0x7ff6f80cde90 2024-03-23T11:26:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.037+0000 7ff6f4a00640 1 -- 172.21.15.62:0/2351951698 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f002cdb0 con 0x7ff6f80cde90 2024-03-23T11:26:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.037+0000 7ff6f4a00640 1 -- 172.21.15.62:0/2351951698 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0034660 con 0x7ff6f80cde90 2024-03-23T11:26:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.037+0000 7ff6f7200640 1 -- 172.21.15.62:0/2351951698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f80cde90 msgr2=0x7ff6f80ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.037+0000 7ff6f7200640 1 --2- 172.21.15.62:0/2351951698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f80cde90 0x7ff6f80ce2e0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009d90 tx=0x7ff6f002bea0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.037+0000 7ff6f7200640 1 -- 172.21.15.62:0/2351951698 shutdown_connections 2024-03-23T11:26:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.037+0000 7ff6f7200640 1 --2- 172.21.15.62:0/2351951698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f80cde90 0x7ff6f80ce2e0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.037+0000 7ff6f7200640 1 --2- 172.21.15.62:0/2351951698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6f8160c00 0x7ff6f806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.037+0000 7ff6f7200640 1 -- 172.21.15.62:0/2351951698 >> 172.21.15.62:0/2351951698 conn(0x7ff6f8066050 msgr2=0x7ff6f8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.037+0000 7ff6f7200640 1 -- 172.21.15.62:0/2351951698 shutdown_connections 2024-03-23T11:26:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.037+0000 7ff6f7200640 1 -- 172.21.15.62:0/2351951698 wait complete. 2024-03-23T11:26:29.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.039+0000 7ff6f7200640 1 Processor -- start 2024-03-23T11:26:29.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.039+0000 7ff6f7200640 1 -- start start 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.040+0000 7ff6f7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6f80cda20 0x7ff6f814bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.040+0000 7ff6f7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f80cde90 0x7ff6f815c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.040+0000 7ff6f7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6f8160c00 0x7ff6f815ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.040+0000 7ff6f7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81490a0 con 0x7ff6f80cde90 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.040+0000 7ff6f7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f8149330 con 0x7ff6f80cda20 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.040+0000 7ff6f7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f814c2c0 con 0x7ff6f8160c00 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.040+0000 7ff6f5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f80cde90 0x7ff6f815c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.040+0000 7ff6f5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f80cde90 0x7ff6f815c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54012/0 (socket says 172.21.15.62:54012) 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.040+0000 7ff6f5400640 1 -- 172.21.15.62:0/44696765 learned_addr learned my addr 172.21.15.62:0/44696765 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.041+0000 7ff6f5e00640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6f80cda20 0x7ff6f814bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.041+0000 7ff6f6800640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6f8160c00 0x7ff6f815ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.041+0000 7ff6f5400640 1 -- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6f8160c00 msgr2=0x7ff6f815ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.041+0000 7ff6f5400640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6f8160c00 0x7ff6f815ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.041+0000 7ff6f5400640 1 -- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6f80cda20 msgr2=0x7ff6f814bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.041+0000 7ff6f5400640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6f80cda20 0x7ff6f814bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.041+0000 7ff6f5400640 1 -- 172.21.15.62:0/44696765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f0009a70 con 0x7ff6f80cde90 2024-03-23T11:26:29.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.041+0000 7ff6f5400640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f80cde90 0x7ff6f815c8d0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400c700 tx=0x7ff6e400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:29.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.041+0000 7ff6ee000640 1 -- 172.21.15.62:0/44696765 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4005ce0 con 0x7ff6f80cde90 2024-03-23T11:26:29.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.042+0000 7ff6ee000640 1 -- 172.21.15.62:0/44696765 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e4005e40 con 0x7ff6f80cde90 2024-03-23T11:26:29.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.042+0000 7ff6ee000640 1 -- 172.21.15.62:0/44696765 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4007290 con 0x7ff6f80cde90 2024-03-23T11:26:29.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.042+0000 7ff6f7200640 1 -- 172.21.15.62:0/44696765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f815bd90 con 0x7ff6f80cde90 2024-03-23T11:26:29.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.042+0000 7ff6f7200640 1 -- 172.21.15.62:0/44696765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f8149580 con 0x7ff6f80cde90 2024-03-23T11:26:29.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.043+0000 7ff6f7200640 1 -- 172.21.15.62:0/44696765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f8066bb0 con 0x7ff6f80cde90 2024-03-23T11:26:29.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.044+0000 7ff6ee000640 1 -- 172.21.15.62:0/44696765 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff6e40074b0 con 0x7ff6f80cde90 2024-03-23T11:26:29.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.045+0000 7ff6ee000640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6d806a7a0 0x7ff6d806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.045+0000 7ff6ee000640 1 -- 172.21.15.62:0/44696765 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6e4088240 con 0x7ff6f80cde90 2024-03-23T11:26:29.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.045+0000 7ff6f5e00640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6d806a7a0 0x7ff6d806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.046+0000 7ff6f5e00640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6d806a7a0 0x7ff6d806cc60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009a30 tx=0x7ff6e8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:29.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.049+0000 7ff6ee000640 1 -- 172.21.15.62:0/44696765 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6e40570f0 con 0x7ff6f80cde90 2024-03-23T11:26:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.238+0000 7ff6f7200640 1 -- 172.21.15.62:0/44696765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6f80ce820 con 0x7ff6f80cde90 2024-03-23T11:26:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.239+0000 7ff6ee000640 1 -- 172.21.15.62:0/44696765 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16092 (secure 0 0 0) 0x7ff6f80ce820 con 0x7ff6f80cde90 2024-03-23T11:26:29.233 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:29.233 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":89,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:26:26.134957+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:26:22.794149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-23T11:26:24.150997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:26:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.242+0000 7ff6f7200640 1 -- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6d806a7a0 msgr2=0x7ff6d806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.242+0000 7ff6f7200640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6d806a7a0 0x7ff6d806cc60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009a30 tx=0x7ff6e8008040 comp rx=0 tx=0).stop 2024-03-23T11:26:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.242+0000 7ff6f7200640 1 -- 172.21.15.62:0/44696765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f80cde90 msgr2=0x7ff6f815c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.242+0000 7ff6f7200640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f80cde90 0x7ff6f815c8d0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400c700 tx=0x7ff6e400cb00 comp rx=0 tx=0).stop 2024-03-23T11:26:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.242+0000 7ff6f7200640 1 -- 172.21.15.62:0/44696765 shutdown_connections 2024-03-23T11:26:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.242+0000 7ff6f7200640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6d806a7a0 0x7ff6d806cc60 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.242+0000 7ff6f7200640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f80cde90 0x7ff6f815c8d0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.242+0000 7ff6f7200640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6f8160c00 0x7ff6f815ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.242+0000 7ff6f7200640 1 --2- 172.21.15.62:0/44696765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6f80cda20 0x7ff6f814bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.242+0000 7ff6f7200640 1 -- 172.21.15.62:0/44696765 >> 172.21.15.62:0/44696765 conn(0x7ff6f8066050 msgr2=0x7ff6f80ae2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:29.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.243+0000 7ff6f7200640 1 -- 172.21.15.62:0/44696765 shutdown_connections 2024-03-23T11:26:29.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.243+0000 7ff6f7200640 1 -- 172.21.15.62:0/44696765 wait complete. 2024-03-23T11:26:29.255 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T11:26:29.255 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:26:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.472+0000 7eff97a00640 1 Processor -- start 2024-03-23T11:26:29.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.473+0000 7eff97a00640 1 -- start start 2024-03-23T11:26:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.474+0000 7eff97a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff901613a0 0x7eff9006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.474+0000 7eff97a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff9006cf40 con 0x7eff901571e0 2024-03-23T11:26:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.474+0000 7eff97a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff901575c0 con 0x7eff9006d910 2024-03-23T11:26:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.474+0000 7eff97a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff9006c3a0 con 0x7eff901613a0 2024-03-23T11:26:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.475+0000 7eff8fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff901613a0 0x7eff9006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.475+0000 7eff8fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff901613a0 0x7eff9006d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54020/0 (socket says 172.21.15.62:54020) 2024-03-23T11:26:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.475+0000 7eff8fe00640 1 -- 172.21.15.62:0/756843627 learned_addr learned my addr 172.21.15.62:0/756843627 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.476+0000 7eff8fe00640 1 -- 172.21.15.62:0/756843627 >> v1:172.21.15.120:6789/0 conn(0x7eff9006d910 legacy=0x7eff9015e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.476+0000 7eff8fe00640 1 -- 172.21.15.62:0/756843627 >> v1:172.21.15.62:6789/0 conn(0x7eff901571e0 legacy=0x7eff900ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.476+0000 7eff8fe00640 1 -- 172.21.15.62:0/756843627 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff900ae040 con 0x7eff901613a0 2024-03-23T11:26:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.477+0000 7eff8fe00640 1 --2- 172.21.15.62:0/756843627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff901613a0 0x7eff9006d3d0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7eff80009870 tx=0x7eff8002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbfe83ac3532773 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.477+0000 7eff8f400640 1 -- 172.21.15.62:0/756843627 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff80030850 con 0x7eff901613a0 2024-03-23T11:26:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.477+0000 7eff8f400640 1 -- 172.21.15.62:0/756843627 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff800309b0 con 0x7eff901613a0 2024-03-23T11:26:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.477+0000 7eff8f400640 1 -- 172.21.15.62:0/756843627 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff80030d40 con 0x7eff901613a0 2024-03-23T11:26:29.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.478+0000 7eff97a00640 1 -- 172.21.15.62:0/756843627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff901613a0 msgr2=0x7eff9006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.478+0000 7eff97a00640 1 --2- 172.21.15.62:0/756843627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff901613a0 0x7eff9006d3d0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7eff80009870 tx=0x7eff8002b990 comp rx=0 tx=0).stop 2024-03-23T11:26:29.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.478+0000 7eff97a00640 1 -- 172.21.15.62:0/756843627 shutdown_connections 2024-03-23T11:26:29.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.478+0000 7eff97a00640 1 --2- 172.21.15.62:0/756843627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff901613a0 0x7eff9006d3d0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.478+0000 7eff97a00640 1 -- 172.21.15.62:0/756843627 >> 172.21.15.62:0/756843627 conn(0x7eff900b02e0 msgr2=0x7eff900b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.478+0000 7eff97a00640 1 -- 172.21.15.62:0/756843627 shutdown_connections 2024-03-23T11:26:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.478+0000 7eff97a00640 1 -- 172.21.15.62:0/756843627 wait complete. 2024-03-23T11:26:29.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.480+0000 7eff97a00640 1 Processor -- start 2024-03-23T11:26:29.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.480+0000 7eff97a00640 1 -- start start 2024-03-23T11:26:29.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.481+0000 7eff97a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff9006d910 0x7eff9014fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff97a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff901571e0 0x7eff900af350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff97a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff901613a0 0x7eff900af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff97a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff90152c20 con 0x7eff901613a0 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff97a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff900cbd60 con 0x7eff9006d910 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff97a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff900cba80 con 0x7eff901571e0 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff95a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff901613a0 0x7eff900af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff95a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff901613a0 0x7eff900af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54022/0 (socket says 172.21.15.62:54022) 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff95a00640 1 -- 172.21.15.62:0/2403854345 learned_addr learned my addr 172.21.15.62:0/2403854345 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff95000640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff9006d910 0x7eff9014fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff8fe00640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff901571e0 0x7eff900af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff95a00640 1 -- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff901571e0 msgr2=0x7eff900af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff95a00640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff901571e0 0x7eff900af350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff95a00640 1 -- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff9006d910 msgr2=0x7eff9014fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff95a00640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff9006d910 0x7eff9014fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.482+0000 7eff95a00640 1 -- 172.21.15.62:0/2403854345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff800094d0 con 0x7eff901613a0 2024-03-23T11:26:29.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.483+0000 7eff95a00640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff901613a0 0x7eff900af890 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7eff8800c7a0 tx=0x7eff8800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:29.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.483+0000 7eff8cc00640 1 -- 172.21.15.62:0/2403854345 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88003b70 con 0x7eff901613a0 2024-03-23T11:26:29.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.483+0000 7eff8cc00640 1 -- 172.21.15.62:0/2403854345 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff88003cf0 con 0x7eff901613a0 2024-03-23T11:26:29.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.483+0000 7eff97a00640 1 -- 172.21.15.62:0/2403854345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff901f5db0 con 0x7eff901613a0 2024-03-23T11:26:29.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.483+0000 7eff8cc00640 1 -- 172.21.15.62:0/2403854345 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88003ea0 con 0x7eff901613a0 2024-03-23T11:26:29.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.483+0000 7eff97a00640 1 -- 172.21.15.62:0/2403854345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff901f6370 con 0x7eff901613a0 2024-03-23T11:26:29.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.484+0000 7eff97a00640 1 -- 172.21.15.62:0/2403854345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff90066c90 con 0x7eff901613a0 2024-03-23T11:26:29.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.485+0000 7eff8cc00640 1 -- 172.21.15.62:0/2403854345 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7eff8800aa90 con 0x7eff901613a0 2024-03-23T11:26:29.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.486+0000 7eff8cc00640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff7406a7a0 0x7eff7406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.486+0000 7eff8cc00640 1 -- 172.21.15.62:0/2403854345 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff88088c10 con 0x7eff901613a0 2024-03-23T11:26:29.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.487+0000 7eff95000640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff7406a7a0 0x7eff7406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.488+0000 7eff95000640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff7406a7a0 0x7eff7406cc60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7eff84000f60 tx=0x7eff84009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.491+0000 7eff8cc00640 1 -- 172.21.15.62:0/2403854345 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff88054020 con 0x7eff901613a0 2024-03-23T11:26:29.590 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:26:29.597+0000 7fbffa600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:26:29.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.680+0000 7eff97a00640 1 -- 172.21.15.62:0/2403854345 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7eff9006cbd0 con 0x7eff7406a7a0 2024-03-23T11:26:29.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.703+0000 7eff8cc00640 1 -- 172.21.15.62:0/2403854345 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286643 (secure 0 0 0) 0x7eff84001c70 con 0x7eff7406a7a0 2024-03-23T11:26:29.701 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:29.702 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:26:29.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.709+0000 7eff97a00640 1 -- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff7406a7a0 msgr2=0x7eff7406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.709+0000 7eff97a00640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff7406a7a0 0x7eff7406cc60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7eff84000f60 tx=0x7eff84009040 comp rx=0 tx=0).stop 2024-03-23T11:26:29.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.709+0000 7eff97a00640 1 -- 172.21.15.62:0/2403854345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff901613a0 msgr2=0x7eff900af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.709+0000 7eff97a00640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff901613a0 0x7eff900af890 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7eff8800c7a0 tx=0x7eff8800cc60 comp rx=0 tx=0).stop 2024-03-23T11:26:29.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.709+0000 7eff97a00640 1 -- 172.21.15.62:0/2403854345 shutdown_connections 2024-03-23T11:26:29.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.709+0000 7eff97a00640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff7406a7a0 0x7eff7406cc60 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.709+0000 7eff97a00640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff901613a0 0x7eff900af890 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.709+0000 7eff97a00640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff901571e0 0x7eff900af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.709+0000 7eff97a00640 1 --2- 172.21.15.62:0/2403854345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff9006d910 0x7eff9014fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.709+0000 7eff97a00640 1 -- 172.21.15.62:0/2403854345 >> 172.21.15.62:0/2403854345 conn(0x7eff900b02e0 msgr2=0x7eff90160920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:29.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.710+0000 7eff97a00640 1 -- 172.21.15.62:0/2403854345 shutdown_connections 2024-03-23T11:26:29.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.710+0000 7eff97a00640 1 -- 172.21.15.62:0/2403854345 wait complete. 2024-03-23T11:26:29.721 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-03-23T11:26:29.396524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254624,"kb_used_data":9724,"kb_used_omap":13,"kb_used_meta":244722,"kb_avail":754720096,"statfs":{"total":773094113280,"available":772833378304,"internally_reserved":0,"allocated":9957376,"data_stored":4903901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":250595416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.772083"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491033+0000","last_change":"2024-03-23T11:26:25.180067+0000","last_active":"2024-03-23T11:26:26.491033+0000","last_peered":"2024-03-23T11:26:26.491033+0000","last_clean":"2024-03-23T11:26:26.491033+0000","last_became_active":"2024-03-23T11:26:25.179491+0000","last_became_peered":"2024-03-23T11:26:25.179491+0000","last_unstale":"2024-03-23T11:26:26.491033+0000","last_undegraded":"2024-03-23T11:26:26.491033+0000","last_fullsized":"2024-03-23T11:26:26.491033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:06:58.990977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488596+0000","last_change":"2024-03-23T11:26:25.179202+0000","last_active":"2024-03-23T11:26:26.488596+0000","last_peered":"2024-03-23T11:26:26.488596+0000","last_clean":"2024-03-23T11:26:26.488596+0000","last_became_active":"2024-03-23T11:26:25.177537+0000","last_became_peered":"2024-03-23T11:26:25.177537+0000","last_unstale":"2024-03-23T11:26:26.488596+0000","last_undegraded":"2024-03-23T11:26:26.488596+0000","last_fullsized":"2024-03-23T11:26:26.488596+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:49.433614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147082+0000","last_change":"2024-03-23T11:26:25.174759+0000","last_active":"2024-03-23T11:26:26.147082+0000","last_peered":"2024-03-23T11:26:26.147082+0000","last_clean":"2024-03-23T11:26:26.147082+0000","last_became_active":"2024-03-23T11:26:25.174468+0000","last_became_peered":"2024-03-23T11:26:25.174468+0000","last_unstale":"2024-03-23T11:26:26.147082+0000","last_undegraded":"2024-03-23T11:26:26.147082+0000","last_fullsized":"2024-03-23T11:26:26.147082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:28:17.495305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145530+0000","last_change":"2024-03-23T11:26:25.154383+0000","last_active":"2024-03-23T11:26:26.145530+0000","last_peered":"2024-03-23T11:26:26.145530+0000","last_clean":"2024-03-23T11:26:26.145530+0000","last_became_active":"2024-03-23T11:26:25.154093+0000","last_became_peered":"2024-03-23T11:26:25.154093+0000","last_unstale":"2024-03-23T11:26:26.145530+0000","last_undegraded":"2024-03-23T11:26:26.145530+0000","last_fullsized":"2024-03-23T11:26:26.145530+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:43:59.471046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146029+0000","last_change":"2024-03-23T11:26:25.173149+0000","last_active":"2024-03-23T11:26:26.146029+0000","last_peered":"2024-03-23T11:26:26.146029+0000","last_clean":"2024-03-23T11:26:26.146029+0000","last_became_active":"2024-03-23T11:26:25.169216+0000","last_became_peered":"2024-03-23T11:26:25.169216+0000","last_unstale":"2024-03-23T11:26:26.146029+0000","last_undegraded":"2024-03-23T11:26:26.146029+0000","last_fullsized":"2024-03-23T11:26:26.146029+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:33:22.112218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488745+0000","last_change":"2024-03-23T11:26:25.178395+0000","last_active":"2024-03-23T11:26:26.488745+0000","last_peered":"2024-03-23T11:26:26.488745+0000","last_clean":"2024-03-23T11:26:26.488745+0000","last_became_active":"2024-03-23T11:26:25.178072+0000","last_became_peered":"2024-03-23T11:26:25.178072+0000","last_unstale":"2024-03-23T11:26:26.488745+0000","last_undegraded":"2024-03-23T11:26:26.488745+0000","last_fullsized":"2024-03-23T11:26:26.488745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:21.703839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.144567+0000","last_change":"2024-03-23T11:26:25.158554+0000","last_active":"2024-03-23T11:26:26.144567+0000","last_peered":"2024-03-23T11:26:26.144567+0000","last_clean":"2024-03-23T11:26:26.144567+0000","last_became_active":"2024-03-23T11:26:25.158118+0000","last_became_peered":"2024-03-23T11:26:25.158118+0000","last_unstale":"2024-03-23T11:26:26.144567+0000","last_undegraded":"2024-03-23T11:26:26.144567+0000","last_fullsized":"2024-03-23T11:26:26.144567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:25:38.549422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488517+0000","last_change":"2024-03-23T11:26:25.176171+0000","last_active":"2024-03-23T11:26:26.488517+0000","last_peered":"2024-03-23T11:26:26.488517+0000","last_clean":"2024-03-23T11:26:26.488517+0000","last_became_active":"2024-03-23T11:26:25.174866+0000","last_became_peered":"2024-03-23T11:26:25.174866+0000","last_unstale":"2024-03-23T11:26:26.488517+0000","last_undegraded":"2024-03-23T11:26:26.488517+0000","last_fullsized":"2024-03-23T11:26:26.488517+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:30:28.522489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489817+0000","last_change":"2024-03-23T11:26:25.175666+0000","last_active":"2024-03-23T11:26:26.489817+0000","last_peered":"2024-03-23T11:26:26.489817+0000","last_clean":"2024-03-23T11:26:26.489817+0000","last_became_active":"2024-03-23T11:26:25.175368+0000","last_became_peered":"2024-03-23T11:26:25.175368+0000","last_unstale":"2024-03-23T11:26:26.489817+0000","last_undegraded":"2024-03-23T11:26:26.489817+0000","last_fullsized":"2024-03-23T11:26:26.489817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:21:03.591269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491167+0000","last_change":"2024-03-23T11:26:25.182826+0000","last_active":"2024-03-23T11:26:26.491167+0000","last_peered":"2024-03-23T11:26:26.491167+0000","last_clean":"2024-03-23T11:26:26.491167+0000","last_became_active":"2024-03-23T11:26:25.179154+0000","last_became_peered":"2024-03-23T11:26:25.179154+0000","last_unstale":"2024-03-23T11:26:26.491167+0000","last_undegraded":"2024-03-23T11:26:26.491167+0000","last_fullsized":"2024-03-23T11:26:26.491167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:54:20.316933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488251+0000","last_change":"2024-03-23T11:26:25.160264+0000","last_active":"2024-03-23T11:26:26.488251+0000","last_peered":"2024-03-23T11:26:26.488251+0000","last_clean":"2024-03-23T11:26:26.488251+0000","last_became_active":"2024-03-23T11:26:25.160001+0000","last_became_peered":"2024-03-23T11:26:25.160001+0000","last_unstale":"2024-03-23T11:26:26.488251+0000","last_undegraded":"2024-03-23T11:26:26.488251+0000","last_fullsized":"2024-03-23T11:26:26.488251+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:56:46.539720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490047+0000","last_change":"2024-03-23T11:26:25.180646+0000","last_active":"2024-03-23T11:26:26.490047+0000","last_peered":"2024-03-23T11:26:26.490047+0000","last_clean":"2024-03-23T11:26:26.490047+0000","last_became_active":"2024-03-23T11:26:25.180316+0000","last_became_peered":"2024-03-23T11:26:25.180316+0000","last_unstale":"2024-03-23T11:26:26.490047+0000","last_undegraded":"2024-03-23T11:26:26.490047+0000","last_fullsized":"2024-03-23T11:26:26.490047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:37.845104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147300+0000","last_change":"2024-03-23T11:26:25.162043+0000","last_active":"2024-03-23T11:26:26.147300+0000","last_peered":"2024-03-23T11:26:26.147300+0000","last_clean":"2024-03-23T11:26:26.147300+0000","last_became_active":"2024-03-23T11:26:25.161722+0000","last_became_peered":"2024-03-23T11:26:25.161722+0000","last_unstale":"2024-03-23T11:26:26.147300+0000","last_undegraded":"2024-03-23T11:26:26.147300+0000","last_fullsized":"2024-03-23T11:26:26.147300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:43:28.253054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.144690+0000","last_change":"2024-03-23T11:26:25.154154+0000","last_active":"2024-03-23T11:26:26.144690+0000","last_peered":"2024-03-23T11:26:26.144690+0000","last_clean":"2024-03-23T11:26:26.144690+0000","last_became_active":"2024-03-23T11:26:25.153823+0000","last_became_peered":"2024-03-23T11:26:25.153823+0000","last_unstale":"2024-03-23T11:26:26.144690+0000","last_undegraded":"2024-03-23T11:26:26.144690+0000","last_fullsized":"2024-03-23T11:26:26.144690+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:56:46.817396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.294004+0000","last_change":"2024-03-23T11:26:25.160147+0000","last_active":"2024-03-23T11:26:26.294004+0000","last_peered":"2024-03-23T11:26:26.294004+0000","last_clean":"2024-03-23T11:26:26.294004+0000","last_became_active":"2024-03-23T11:26:25.159887+0000","last_became_peered":"2024-03-23T11:26:25.159887+0000","last_unstale":"2024-03-23T11:26:26.294004+0000","last_undegraded":"2024-03-23T11:26:26.294004+0000","last_fullsized":"2024-03-23T11:26:26.294004+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:50:28.540065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146183+0000","last_change":"2024-03-23T11:26:25.175501+0000","last_active":"2024-03-23T11:26:26.146183+0000","last_peered":"2024-03-23T11:26:26.146183+0000","last_clean":"2024-03-23T11:26:26.146183+0000","last_became_active":"2024-03-23T11:26:25.175230+0000","last_became_peered":"2024-03-23T11:26:25.175230+0000","last_unstale":"2024-03-23T11:26:26.146183+0000","last_undegraded":"2024-03-23T11:26:26.146183+0000","last_fullsized":"2024-03-23T11:26:26.146183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:52.469820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147645+0000","last_change":"2024-03-23T11:26:25.161519+0000","last_active":"2024-03-23T11:26:26.147645+0000","last_peered":"2024-03-23T11:26:26.147645+0000","last_clean":"2024-03-23T11:26:26.147645+0000","last_became_active":"2024-03-23T11:26:25.161200+0000","last_became_peered":"2024-03-23T11:26:25.161200+0000","last_unstale":"2024-03-23T11:26:26.147645+0000","last_undegraded":"2024-03-23T11:26:26.147645+0000","last_fullsized":"2024-03-23T11:26:26.147645+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:10.886471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145079+0000","last_change":"2024-03-23T11:26:25.180169+0000","last_active":"2024-03-23T11:26:26.145079+0000","last_peered":"2024-03-23T11:26:26.145079+0000","last_clean":"2024-03-23T11:26:26.145079+0000","last_became_active":"2024-03-23T11:26:25.179906+0000","last_became_peered":"2024-03-23T11:26:25.179906+0000","last_unstale":"2024-03-23T11:26:26.145079+0000","last_undegraded":"2024-03-23T11:26:26.145079+0000","last_fullsized":"2024-03-23T11:26:26.145079+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:01:57.170178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490169+0000","last_change":"2024-03-23T11:26:25.178448+0000","last_active":"2024-03-23T11:26:26.490169+0000","last_peered":"2024-03-23T11:26:26.490169+0000","last_clean":"2024-03-23T11:26:26.490169+0000","last_became_active":"2024-03-23T11:26:25.177971+0000","last_became_peered":"2024-03-23T11:26:25.177971+0000","last_unstale":"2024-03-23T11:26:26.490169+0000","last_undegraded":"2024-03-23T11:26:26.490169+0000","last_fullsized":"2024-03-23T11:26:26.490169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:34.862798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145630+0000","last_change":"2024-03-23T11:26:25.158769+0000","last_active":"2024-03-23T11:26:26.145630+0000","last_peered":"2024-03-23T11:26:26.145630+0000","last_clean":"2024-03-23T11:26:26.145630+0000","last_became_active":"2024-03-23T11:26:25.158469+0000","last_became_peered":"2024-03-23T11:26:25.158469+0000","last_unstale":"2024-03-23T11:26:26.145630+0000","last_undegraded":"2024-03-23T11:26:26.145630+0000","last_fullsized":"2024-03-23T11:26:26.145630+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:07.951537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145160+0000","last_change":"2024-03-23T11:26:25.170810+0000","last_active":"2024-03-23T11:26:26.145160+0000","last_peered":"2024-03-23T11:26:26.145160+0000","last_clean":"2024-03-23T11:26:26.145160+0000","last_became_active":"2024-03-23T11:26:25.170500+0000","last_became_peered":"2024-03-23T11:26:25.170500+0000","last_unstale":"2024-03-23T11:26:26.145160+0000","last_undegraded":"2024-03-23T11:26:26.145160+0000","last_fullsized":"2024-03-23T11:26:26.145160+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:57:23.323902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.487857+0000","last_change":"2024-03-23T11:26:25.168338+0000","last_active":"2024-03-23T11:26:26.487857+0000","last_peered":"2024-03-23T11:26:26.487857+0000","last_clean":"2024-03-23T11:26:26.487857+0000","last_became_active":"2024-03-23T11:26:25.168110+0000","last_became_peered":"2024-03-23T11:26:25.168110+0000","last_unstale":"2024-03-23T11:26:26.487857+0000","last_undegraded":"2024-03-23T11:26:26.487857+0000","last_fullsized":"2024-03-23T11:26:26.487857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:53:19.310572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489505+0000","last_change":"2024-03-23T11:26:25.178671+0000","last_active":"2024-03-23T11:26:26.489505+0000","last_peered":"2024-03-23T11:26:26.489505+0000","last_clean":"2024-03-23T11:26:26.489505+0000","last_became_active":"2024-03-23T11:26:25.178202+0000","last_became_peered":"2024-03-23T11:26:25.178202+0000","last_unstale":"2024-03-23T11:26:26.489505+0000","last_undegraded":"2024-03-23T11:26:26.489505+0000","last_fullsized":"2024-03-23T11:26:26.489505+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:56:46.397432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.144714+0000","last_change":"2024-03-23T11:26:25.170834+0000","last_active":"2024-03-23T11:26:26.144714+0000","last_peered":"2024-03-23T11:26:26.144714+0000","last_clean":"2024-03-23T11:26:26.144714+0000","last_became_active":"2024-03-23T11:26:25.170480+0000","last_became_peered":"2024-03-23T11:26:25.170480+0000","last_unstale":"2024-03-23T11:26:26.144714+0000","last_undegraded":"2024-03-23T11:26:26.144714+0000","last_fullsized":"2024-03-23T11:26:26.144714+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:33:39.993333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488664+0000","last_change":"2024-03-23T11:26:25.178428+0000","last_active":"2024-03-23T11:26:26.488664+0000","last_peered":"2024-03-23T11:26:26.488664+0000","last_clean":"2024-03-23T11:26:26.488664+0000","last_became_active":"2024-03-23T11:26:25.178170+0000","last_became_peered":"2024-03-23T11:26:25.178170+0000","last_unstale":"2024-03-23T11:26:26.488664+0000","last_undegraded":"2024-03-23T11:26:26.488664+0000","last_fullsized":"2024-03-23T11:26:26.488664+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:11:13.865349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.148198+0000","last_change":"2024-03-23T11:26:25.168575+0000","last_active":"2024-03-23T11:26:26.148198+0000","last_peered":"2024-03-23T11:26:26.148198+0000","last_clean":"2024-03-23T11:26:26.148198+0000","last_became_active":"2024-03-23T11:26:25.168304+0000","last_became_peered":"2024-03-23T11:26:25.168304+0000","last_unstale":"2024-03-23T11:26:26.148198+0000","last_undegraded":"2024-03-23T11:26:26.148198+0000","last_fullsized":"2024-03-23T11:26:26.148198+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:21:25.998148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145944+0000","last_change":"2024-03-23T11:26:25.162817+0000","last_active":"2024-03-23T11:26:26.145944+0000","last_peered":"2024-03-23T11:26:26.145944+0000","last_clean":"2024-03-23T11:26:26.145944+0000","last_became_active":"2024-03-23T11:26:25.161854+0000","last_became_peered":"2024-03-23T11:26:25.161854+0000","last_unstale":"2024-03-23T11:26:26.145944+0000","last_undegraded":"2024-03-23T11:26:26.145944+0000","last_fullsized":"2024-03-23T11:26:26.145944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:13:09.998627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490305+0000","last_change":"2024-03-23T11:26:25.179703+0000","last_active":"2024-03-23T11:26:26.490305+0000","last_peered":"2024-03-23T11:26:26.490305+0000","last_clean":"2024-03-23T11:26:26.490305+0000","last_became_active":"2024-03-23T11:26:25.179152+0000","last_became_peered":"2024-03-23T11:26:25.179152+0000","last_unstale":"2024-03-23T11:26:26.490305+0000","last_undegraded":"2024-03-23T11:26:26.490305+0000","last_fullsized":"2024-03-23T11:26:26.490305+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:47.516318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488193+0000","last_change":"2024-03-23T11:26:25.151893+0000","last_active":"2024-03-23T11:26:26.488193+0000","last_peered":"2024-03-23T11:26:26.488193+0000","last_clean":"2024-03-23T11:26:26.488193+0000","last_became_active":"2024-03-23T11:26:25.151625+0000","last_became_peered":"2024-03-23T11:26:25.151625+0000","last_unstale":"2024-03-23T11:26:26.488193+0000","last_undegraded":"2024-03-23T11:26:26.488193+0000","last_fullsized":"2024-03-23T11:26:26.488193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:36:16.570415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488948+0000","last_change":"2024-03-23T11:26:25.151063+0000","last_active":"2024-03-23T11:26:26.488948+0000","last_peered":"2024-03-23T11:26:26.488948+0000","last_clean":"2024-03-23T11:26:26.488948+0000","last_became_active":"2024-03-23T11:26:25.150753+0000","last_became_peered":"2024-03-23T11:26:25.150753+0000","last_unstale":"2024-03-23T11:26:26.488948+0000","last_undegraded":"2024-03-23T11:26:26.488948+0000","last_fullsized":"2024-03-23T11:26:26.488948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:19:57.485482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146068+0000","last_change":"2024-03-23T11:26:25.158789+0000","last_active":"2024-03-23T11:26:26.146068+0000","last_peered":"2024-03-23T11:26:26.146068+0000","last_clean":"2024-03-23T11:26:26.146068+0000","last_became_active":"2024-03-23T11:26:25.158117+0000","last_became_peered":"2024-03-23T11:26:25.158117+0000","last_unstale":"2024-03-23T11:26:26.146068+0000","last_undegraded":"2024-03-23T11:26:26.146068+0000","last_fullsized":"2024-03-23T11:26:26.146068+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:09.005346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489210+0000","last_change":"2024-03-23T11:26:25.183462+0000","last_active":"2024-03-23T11:26:26.489210+0000","last_peered":"2024-03-23T11:26:26.489210+0000","last_clean":"2024-03-23T11:26:26.489210+0000","last_became_active":"2024-03-23T11:26:25.183152+0000","last_became_peered":"2024-03-23T11:26:25.183152+0000","last_unstale":"2024-03-23T11:26:26.489210+0000","last_undegraded":"2024-03-23T11:26:26.489210+0000","last_fullsized":"2024-03-23T11:26:26.489210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:08.692375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147123+0000","last_change":"2024-03-23T11:26:25.174889+0000","last_active":"2024-03-23T11:26:26.147123+0000","last_peered":"2024-03-23T11:26:26.147123+0000","last_clean":"2024-03-23T11:26:26.147123+0000","last_became_active":"2024-03-23T11:26:25.174629+0000","last_became_peered":"2024-03-23T11:26:25.174629+0000","last_unstale":"2024-03-23T11:26:26.147123+0000","last_undegraded":"2024-03-23T11:26:26.147123+0000","last_fullsized":"2024-03-23T11:26:26.147123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:32:36.038543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"88'4","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491218+0000","last_change":"2024-03-23T11:26:24.671290+0000","last_active":"2024-03-23T11:26:26.491218+0000","last_peered":"2024-03-23T11:26:26.491218+0000","last_clean":"2024-03-23T11:26:26.491218+0000","last_became_active":"2024-03-23T11:26:24.670843+0000","last_became_peered":"2024-03-23T11:26:24.670843+0000","last_unstale":"2024-03-23T11:26:26.491218+0000","last_undegraded":"2024-03-23T11:26:26.491218+0000","last_fullsized":"2024-03-23T11:26:26.491218+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:19:36.996545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491089+0000","last_change":"2024-03-23T11:26:25.180388+0000","last_active":"2024-03-23T11:26:26.491089+0000","last_peered":"2024-03-23T11:26:26.491089+0000","last_clean":"2024-03-23T11:26:26.491089+0000","last_became_active":"2024-03-23T11:26:25.179946+0000","last_became_peered":"2024-03-23T11:26:25.179946+0000","last_unstale":"2024-03-23T11:26:26.491089+0000","last_undegraded":"2024-03-23T11:26:26.491089+0000","last_fullsized":"2024-03-23T11:26:26.491089+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:17.879569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147342+0000","last_change":"2024-03-23T11:26:24.669998+0000","last_active":"2024-03-23T11:26:26.147342+0000","last_peered":"2024-03-23T11:26:26.147342+0000","last_clean":"2024-03-23T11:26:26.147342+0000","last_became_active":"2024-03-23T11:26:24.669690+0000","last_became_peered":"2024-03-23T11:26:24.669690+0000","last_unstale":"2024-03-23T11:26:26.147342+0000","last_undegraded":"2024-03-23T11:26:26.147342+0000","last_fullsized":"2024-03-23T11:26:26.147342+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:54:32.796874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490263+0000","last_change":"2024-03-23T11:26:25.181106+0000","last_active":"2024-03-23T11:26:26.490263+0000","last_peered":"2024-03-23T11:26:26.490263+0000","last_clean":"2024-03-23T11:26:26.490263+0000","last_became_active":"2024-03-23T11:26:25.180695+0000","last_became_peered":"2024-03-23T11:26:25.180695+0000","last_unstale":"2024-03-23T11:26:26.490263+0000","last_undegraded":"2024-03-23T11:26:26.490263+0000","last_fullsized":"2024-03-23T11:26:26.490263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:34.862798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490151+0000","last_change":"2024-03-23T11:26:24.670158+0000","last_active":"2024-03-23T11:26:26.490151+0000","last_peered":"2024-03-23T11:26:26.490151+0000","last_clean":"2024-03-23T11:26:26.490151+0000","last_became_active":"2024-03-23T11:26:24.666903+0000","last_became_peered":"2024-03-23T11:26:24.666903+0000","last_unstale":"2024-03-23T11:26:26.490151+0000","last_undegraded":"2024-03-23T11:26:26.490151+0000","last_fullsized":"2024-03-23T11:26:26.490151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:07.991929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.148010+0000","last_change":"2024-03-23T11:26:25.162193+0000","last_active":"2024-03-23T11:26:26.148010+0000","last_peered":"2024-03-23T11:26:26.148010+0000","last_clean":"2024-03-23T11:26:26.148010+0000","last_became_active":"2024-03-23T11:26:25.161332+0000","last_became_peered":"2024-03-23T11:26:25.161332+0000","last_unstale":"2024-03-23T11:26:26.148010+0000","last_undegraded":"2024-03-23T11:26:26.148010+0000","last_fullsized":"2024-03-23T11:26:26.148010+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:12:51.654663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490641+0000","last_change":"2024-03-23T11:26:24.669843+0000","last_active":"2024-03-23T11:26:26.490641+0000","last_peered":"2024-03-23T11:26:26.490641+0000","last_clean":"2024-03-23T11:26:26.490641+0000","last_became_active":"2024-03-23T11:26:24.669521+0000","last_became_peered":"2024-03-23T11:26:24.669521+0000","last_unstale":"2024-03-23T11:26:26.490641+0000","last_undegraded":"2024-03-23T11:26:26.490641+0000","last_fullsized":"2024-03-23T11:26:26.490641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:26:05.728586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488840+0000","last_change":"2024-03-23T11:26:25.158898+0000","last_active":"2024-03-23T11:26:26.488840+0000","last_peered":"2024-03-23T11:26:26.488840+0000","last_clean":"2024-03-23T11:26:26.488840+0000","last_became_active":"2024-03-23T11:26:25.155583+0000","last_became_peered":"2024-03-23T11:26:25.155583+0000","last_unstale":"2024-03-23T11:26:26.488840+0000","last_undegraded":"2024-03-23T11:26:26.488840+0000","last_fullsized":"2024-03-23T11:26:26.488840+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:58:03.691137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146556+0000","last_change":"2024-03-23T11:26:24.668582+0000","last_active":"2024-03-23T11:26:26.146556+0000","last_peered":"2024-03-23T11:26:26.146556+0000","last_clean":"2024-03-23T11:26:26.146556+0000","last_became_active":"2024-03-23T11:26:24.664916+0000","last_became_peered":"2024-03-23T11:26:24.664916+0000","last_unstale":"2024-03-23T11:26:26.146556+0000","last_undegraded":"2024-03-23T11:26:26.146556+0000","last_fullsized":"2024-03-23T11:26:26.146556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:43:01.127370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146263+0000","last_change":"2024-03-23T11:26:25.163953+0000","last_active":"2024-03-23T11:26:26.146263+0000","last_peered":"2024-03-23T11:26:26.146263+0000","last_clean":"2024-03-23T11:26:26.146263+0000","last_became_active":"2024-03-23T11:26:25.163684+0000","last_became_peered":"2024-03-23T11:26:25.163684+0000","last_unstale":"2024-03-23T11:26:26.146263+0000","last_undegraded":"2024-03-23T11:26:26.146263+0000","last_fullsized":"2024-03-23T11:26:26.146263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:09:31.199645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146523+0000","last_change":"2024-03-23T11:26:24.664701+0000","last_active":"2024-03-23T11:26:26.146523+0000","last_peered":"2024-03-23T11:26:26.146523+0000","last_clean":"2024-03-23T11:26:26.146523+0000","last_became_active":"2024-03-23T11:26:24.664325+0000","last_became_peered":"2024-03-23T11:26:24.664325+0000","last_unstale":"2024-03-23T11:26:26.146523+0000","last_undegraded":"2024-03-23T11:26:26.146523+0000","last_fullsized":"2024-03-23T11:26:26.146523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:47:04.633428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145444+0000","last_change":"2024-03-23T11:26:25.171512+0000","last_active":"2024-03-23T11:26:26.145444+0000","last_peered":"2024-03-23T11:26:26.145444+0000","last_clean":"2024-03-23T11:26:26.145444+0000","last_became_active":"2024-03-23T11:26:25.171180+0000","last_became_peered":"2024-03-23T11:26:25.171180+0000","last_unstale":"2024-03-23T11:26:26.145444+0000","last_undegraded":"2024-03-23T11:26:26.145444+0000","last_fullsized":"2024-03-23T11:26:26.145444+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:07:43.159151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489532+0000","last_change":"2024-03-23T11:26:24.663805+0000","last_active":"2024-03-23T11:26:26.489532+0000","last_peered":"2024-03-23T11:26:26.489532+0000","last_clean":"2024-03-23T11:26:26.489532+0000","last_became_active":"2024-03-23T11:26:24.663515+0000","last_became_peered":"2024-03-23T11:26:24.663515+0000","last_unstale":"2024-03-23T11:26:26.489532+0000","last_undegraded":"2024-03-23T11:26:26.489532+0000","last_fullsized":"2024-03-23T11:26:26.489532+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:32:35.025125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.294060+0000","last_change":"2024-03-23T11:26:25.159583+0000","last_active":"2024-03-23T11:26:26.294060+0000","last_peered":"2024-03-23T11:26:26.294060+0000","last_clean":"2024-03-23T11:26:26.294060+0000","last_became_active":"2024-03-23T11:26:25.158018+0000","last_became_peered":"2024-03-23T11:26:25.158018+0000","last_unstale":"2024-03-23T11:26:26.294060+0000","last_undegraded":"2024-03-23T11:26:26.294060+0000","last_fullsized":"2024-03-23T11:26:26.294060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:17.879569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488022+0000","last_change":"2024-03-23T11:26:24.673233+0000","last_active":"2024-03-23T11:26:26.488022+0000","last_peered":"2024-03-23T11:26:26.488022+0000","last_clean":"2024-03-23T11:26:26.488022+0000","last_became_active":"2024-03-23T11:26:24.672939+0000","last_became_peered":"2024-03-23T11:26:24.672939+0000","last_unstale":"2024-03-23T11:26:26.488022+0000","last_undegraded":"2024-03-23T11:26:26.488022+0000","last_fullsized":"2024-03-23T11:26:26.488022+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:04:43.824387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.294467+0000","last_change":"2024-03-23T11:26:25.148389+0000","last_active":"2024-03-23T11:26:26.294467+0000","last_peered":"2024-03-23T11:26:26.294467+0000","last_clean":"2024-03-23T11:26:26.294467+0000","last_became_active":"2024-03-23T11:26:25.148079+0000","last_became_peered":"2024-03-23T11:26:25.148079+0000","last_unstale":"2024-03-23T11:26:26.294467+0000","last_undegraded":"2024-03-23T11:26:26.294467+0000","last_fullsized":"2024-03-23T11:26:26.294467+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:21.703839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491320+0000","last_change":"2024-03-23T11:26:24.672251+0000","last_active":"2024-03-23T11:26:26.491320+0000","last_peered":"2024-03-23T11:26:26.491320+0000","last_clean":"2024-03-23T11:26:26.491320+0000","last_became_active":"2024-03-23T11:26:24.671880+0000","last_became_peered":"2024-03-23T11:26:24.671880+0000","last_unstale":"2024-03-23T11:26:26.491320+0000","last_undegraded":"2024-03-23T11:26:26.491320+0000","last_fullsized":"2024-03-23T11:26:26.491320+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:04:35.096903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489817+0000","last_change":"2024-03-23T11:26:25.172438+0000","last_active":"2024-03-23T11:26:26.489817+0000","last_peered":"2024-03-23T11:26:26.489817+0000","last_clean":"2024-03-23T11:26:26.489817+0000","last_became_active":"2024-03-23T11:26:25.172149+0000","last_became_peered":"2024-03-23T11:26:25.172149+0000","last_unstale":"2024-03-23T11:26:26.489817+0000","last_undegraded":"2024-03-23T11:26:26.489817+0000","last_fullsized":"2024-03-23T11:26:26.489817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:43:29.366812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491289+0000","last_change":"2024-03-23T11:26:24.670501+0000","last_active":"2024-03-23T11:26:26.491289+0000","last_peered":"2024-03-23T11:26:26.491289+0000","last_clean":"2024-03-23T11:26:26.491289+0000","last_became_active":"2024-03-23T11:26:24.670157+0000","last_became_peered":"2024-03-23T11:26:24.670157+0000","last_unstale":"2024-03-23T11:26:26.491289+0000","last_undegraded":"2024-03-23T11:26:26.491289+0000","last_fullsized":"2024-03-23T11:26:26.491289+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:46.322563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490365+0000","last_change":"2024-03-23T11:26:25.177116+0000","last_active":"2024-03-23T11:26:26.490365+0000","last_peered":"2024-03-23T11:26:26.490365+0000","last_clean":"2024-03-23T11:26:26.490365+0000","last_became_active":"2024-03-23T11:26:25.176582+0000","last_became_peered":"2024-03-23T11:26:25.176582+0000","last_unstale":"2024-03-23T11:26:26.490365+0000","last_undegraded":"2024-03-23T11:26:26.490365+0000","last_fullsized":"2024-03-23T11:26:26.490365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:47.516318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"88'3","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490695+0000","last_change":"2024-03-23T11:26:24.675259+0000","last_active":"2024-03-23T11:26:26.490695+0000","last_peered":"2024-03-23T11:26:26.490695+0000","last_clean":"2024-03-23T11:26:26.490695+0000","last_became_active":"2024-03-23T11:26:24.675020+0000","last_became_peered":"2024-03-23T11:26:24.675020+0000","last_unstale":"2024-03-23T11:26:26.490695+0000","last_undegraded":"2024-03-23T11:26:26.490695+0000","last_fullsized":"2024-03-23T11:26:26.490695+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:52:05.497089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146202+0000","last_change":"2024-03-23T11:26:25.172849+0000","last_active":"2024-03-23T11:26:26.146202+0000","last_peered":"2024-03-23T11:26:26.146202+0000","last_clean":"2024-03-23T11:26:26.146202+0000","last_became_active":"2024-03-23T11:26:25.172536+0000","last_became_peered":"2024-03-23T11:26:25.172536+0000","last_unstale":"2024-03-23T11:26:26.146202+0000","last_undegraded":"2024-03-23T11:26:26.146202+0000","last_fullsized":"2024-03-23T11:26:26.146202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:37.845104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"88'6","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489214+0000","last_change":"2024-03-23T11:26:24.664637+0000","last_active":"2024-03-23T11:26:26.489214+0000","last_peered":"2024-03-23T11:26:26.489214+0000","last_clean":"2024-03-23T11:26:26.489214+0000","last_became_active":"2024-03-23T11:26:24.663951+0000","last_became_peered":"2024-03-23T11:26:24.663951+0000","last_unstale":"2024-03-23T11:26:26.489214+0000","last_undegraded":"2024-03-23T11:26:26.489214+0000","last_fullsized":"2024-03-23T11:26:26.489214+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:54:19.303515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489208+0000","last_change":"2024-03-23T11:26:25.161708+0000","last_active":"2024-03-23T11:26:26.489208+0000","last_peered":"2024-03-23T11:26:26.489208+0000","last_clean":"2024-03-23T11:26:26.489208+0000","last_became_active":"2024-03-23T11:26:25.161428+0000","last_became_peered":"2024-03-23T11:26:25.161428+0000","last_unstale":"2024-03-23T11:26:26.489208+0000","last_undegraded":"2024-03-23T11:26:26.489208+0000","last_fullsized":"2024-03-23T11:26:26.489208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:13:09.998627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488445+0000","last_change":"2024-03-23T11:26:24.665210+0000","last_active":"2024-03-23T11:26:26.488445+0000","last_peered":"2024-03-23T11:26:26.488445+0000","last_clean":"2024-03-23T11:26:26.488445+0000","last_became_active":"2024-03-23T11:26:24.664838+0000","last_became_peered":"2024-03-23T11:26:24.664838+0000","last_unstale":"2024-03-23T11:26:26.488445+0000","last_undegraded":"2024-03-23T11:26:26.488445+0000","last_fullsized":"2024-03-23T11:26:26.488445+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:03:06.157933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147554+0000","last_change":"2024-03-23T11:26:25.168101+0000","last_active":"2024-03-23T11:26:26.147554+0000","last_peered":"2024-03-23T11:26:26.147554+0000","last_clean":"2024-03-23T11:26:26.147554+0000","last_became_active":"2024-03-23T11:26:25.167767+0000","last_became_peered":"2024-03-23T11:26:25.167767+0000","last_unstale":"2024-03-23T11:26:26.147554+0000","last_undegraded":"2024-03-23T11:26:26.147554+0000","last_fullsized":"2024-03-23T11:26:26.147554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:20:59.367745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.294071+0000","last_change":"2024-03-23T11:26:24.671390+0000","last_active":"2024-03-23T11:26:26.294071+0000","last_peered":"2024-03-23T11:26:26.294071+0000","last_clean":"2024-03-23T11:26:26.294071+0000","last_became_active":"2024-03-23T11:26:24.671039+0000","last_became_peered":"2024-03-23T11:26:24.671039+0000","last_unstale":"2024-03-23T11:26:26.294071+0000","last_undegraded":"2024-03-23T11:26:26.294071+0000","last_fullsized":"2024-03-23T11:26:26.294071+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:56:45.384014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.487985+0000","last_change":"2024-03-23T11:26:25.149881+0000","last_active":"2024-03-23T11:26:26.487985+0000","last_peered":"2024-03-23T11:26:26.487985+0000","last_clean":"2024-03-23T11:26:26.487985+0000","last_became_active":"2024-03-23T11:26:25.149580+0000","last_became_peered":"2024-03-23T11:26:25.149580+0000","last_unstale":"2024-03-23T11:26:26.487985+0000","last_undegraded":"2024-03-23T11:26:26.487985+0000","last_fullsized":"2024-03-23T11:26:26.487985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:57:48.324512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488702+0000","last_change":"2024-03-23T11:26:24.669332+0000","last_active":"2024-03-23T11:26:26.488702+0000","last_peered":"2024-03-23T11:26:26.488702+0000","last_clean":"2024-03-23T11:26:26.488702+0000","last_became_active":"2024-03-23T11:26:24.668966+0000","last_became_peered":"2024-03-23T11:26:24.668966+0000","last_unstale":"2024-03-23T11:26:26.488702+0000","last_undegraded":"2024-03-23T11:26:26.488702+0000","last_fullsized":"2024-03-23T11:26:26.488702+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:56:45.384014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.487551+0000","last_change":"2024-03-23T11:26:25.174460+0000","last_active":"2024-03-23T11:26:26.487551+0000","last_peered":"2024-03-23T11:26:26.487551+0000","last_clean":"2024-03-23T11:26:26.487551+0000","last_became_active":"2024-03-23T11:26:25.174148+0000","last_became_peered":"2024-03-23T11:26:25.174148+0000","last_unstale":"2024-03-23T11:26:26.487551+0000","last_undegraded":"2024-03-23T11:26:26.487551+0000","last_fullsized":"2024-03-23T11:26:26.487551+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:34.156067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"88'8","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489310+0000","last_change":"2024-03-23T11:26:24.666094+0000","last_active":"2024-03-23T11:26:26.489310+0000","last_peered":"2024-03-23T11:26:26.489310+0000","last_clean":"2024-03-23T11:26:26.489310+0000","last_became_active":"2024-03-23T11:26:24.665790+0000","last_became_peered":"2024-03-23T11:26:24.665790+0000","last_unstale":"2024-03-23T11:26:26.489310+0000","last_undegraded":"2024-03-23T11:26:26.489310+0000","last_fullsized":"2024-03-23T11:26:26.489310+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:33:21.098801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.148147+0000","last_change":"2024-03-23T11:26:25.168013+0000","last_active":"2024-03-23T11:26:26.148147+0000","last_peered":"2024-03-23T11:26:26.148147+0000","last_clean":"2024-03-23T11:26:26.148147+0000","last_became_active":"2024-03-23T11:26:25.167721+0000","last_became_peered":"2024-03-23T11:26:25.167721+0000","last_unstale":"2024-03-23T11:26:26.148147+0000","last_undegraded":"2024-03-23T11:26:26.148147+0000","last_fullsized":"2024-03-23T11:26:26.148147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:40:01.187632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491466+0000","last_change":"2024-03-23T11:26:24.670293+0000","last_active":"2024-03-23T11:26:26.491466+0000","last_peered":"2024-03-23T11:26:26.491466+0000","last_clean":"2024-03-23T11:26:26.491466+0000","last_became_active":"2024-03-23T11:26:24.669998+0000","last_became_peered":"2024-03-23T11:26:24.669998+0000","last_unstale":"2024-03-23T11:26:26.491466+0000","last_undegraded":"2024-03-23T11:26:26.491466+0000","last_fullsized":"2024-03-23T11:26:26.491466+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:08:41.590426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491193+0000","last_change":"2024-03-23T11:26:25.185110+0000","last_active":"2024-03-23T11:26:26.491193+0000","last_peered":"2024-03-23T11:26:26.491193+0000","last_clean":"2024-03-23T11:26:26.491193+0000","last_became_active":"2024-03-23T11:26:25.184733+0000","last_became_peered":"2024-03-23T11:26:25.184733+0000","last_unstale":"2024-03-23T11:26:26.491193+0000","last_undegraded":"2024-03-23T11:26:26.491193+0000","last_fullsized":"2024-03-23T11:26:26.491193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:13:09.998627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490075+0000","last_change":"2024-03-23T11:26:24.667647+0000","last_active":"2024-03-23T11:26:26.490075+0000","last_peered":"2024-03-23T11:26:26.490075+0000","last_clean":"2024-03-23T11:26:26.490075+0000","last_became_active":"2024-03-23T11:26:24.667301+0000","last_became_peered":"2024-03-23T11:26:24.667301+0000","last_unstale":"2024-03-23T11:26:26.490075+0000","last_undegraded":"2024-03-23T11:26:26.490075+0000","last_fullsized":"2024-03-23T11:26:26.490075+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:51.456402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146627+0000","last_change":"2024-03-23T11:26:25.171011+0000","last_active":"2024-03-23T11:26:26.146627+0000","last_peered":"2024-03-23T11:26:26.146627+0000","last_clean":"2024-03-23T11:26:26.146627+0000","last_became_active":"2024-03-23T11:26:25.170341+0000","last_became_peered":"2024-03-23T11:26:25.170341+0000","last_unstale":"2024-03-23T11:26:26.146627+0000","last_undegraded":"2024-03-23T11:26:26.146627+0000","last_fullsized":"2024-03-23T11:26:26.146627+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:54:20.316933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490633+0000","last_change":"2024-03-23T11:26:24.667917+0000","last_active":"2024-03-23T11:26:26.490633+0000","last_peered":"2024-03-23T11:26:26.490633+0000","last_clean":"2024-03-23T11:26:26.490633+0000","last_became_active":"2024-03-23T11:26:24.667626+0000","last_became_peered":"2024-03-23T11:26:24.667626+0000","last_unstale":"2024-03-23T11:26:26.490633+0000","last_undegraded":"2024-03-23T11:26:26.490633+0000","last_fullsized":"2024-03-23T11:26:26.490633+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:35:28.898907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.144924+0000","last_change":"2024-03-23T11:26:25.151407+0000","last_active":"2024-03-23T11:26:26.144924+0000","last_peered":"2024-03-23T11:26:26.144924+0000","last_clean":"2024-03-23T11:26:26.144924+0000","last_became_active":"2024-03-23T11:26:25.151060+0000","last_became_peered":"2024-03-23T11:26:25.151060+0000","last_unstale":"2024-03-23T11:26:26.144924+0000","last_undegraded":"2024-03-23T11:26:26.144924+0000","last_fullsized":"2024-03-23T11:26:26.144924+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:02.065602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.294312+0000","last_change":"2024-03-23T11:26:24.669604+0000","last_active":"2024-03-23T11:26:26.294312+0000","last_peered":"2024-03-23T11:26:26.294312+0000","last_clean":"2024-03-23T11:26:26.294312+0000","last_became_active":"2024-03-23T11:26:24.669320+0000","last_became_peered":"2024-03-23T11:26:24.669320+0000","last_unstale":"2024-03-23T11:26:26.294312+0000","last_undegraded":"2024-03-23T11:26:26.294312+0000","last_fullsized":"2024-03-23T11:26:26.294312+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:36.831686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488843+0000","last_change":"2024-03-23T11:26:25.175918+0000","last_active":"2024-03-23T11:26:26.488843+0000","last_peered":"2024-03-23T11:26:26.488843+0000","last_clean":"2024-03-23T11:26:26.488843+0000","last_became_active":"2024-03-23T11:26:25.174608+0000","last_became_peered":"2024-03-23T11:26:25.174608+0000","last_unstale":"2024-03-23T11:26:26.488843+0000","last_undegraded":"2024-03-23T11:26:26.488843+0000","last_fullsized":"2024-03-23T11:26:26.488843+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:36:42.753677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145104+0000","last_change":"2024-03-23T11:26:24.657738+0000","last_active":"2024-03-23T11:26:26.145104+0000","last_peered":"2024-03-23T11:26:26.145104+0000","last_clean":"2024-03-23T11:26:26.145104+0000","last_became_active":"2024-03-23T11:26:24.657363+0000","last_became_peered":"2024-03-23T11:26:24.657363+0000","last_unstale":"2024-03-23T11:26:26.145104+0000","last_undegraded":"2024-03-23T11:26:26.145104+0000","last_fullsized":"2024-03-23T11:26:26.145104+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:51:22.194384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145044+0000","last_change":"2024-03-23T11:26:25.166812+0000","last_active":"2024-03-23T11:26:26.145044+0000","last_peered":"2024-03-23T11:26:26.145044+0000","last_clean":"2024-03-23T11:26:26.145044+0000","last_became_active":"2024-03-23T11:26:25.166514+0000","last_became_peered":"2024-03-23T11:26:25.166514+0000","last_unstale":"2024-03-23T11:26:26.145044+0000","last_undegraded":"2024-03-23T11:26:26.145044+0000","last_fullsized":"2024-03-23T11:26:26.145044+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:17:42.796450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.144838+0000","last_change":"2024-03-23T11:26:24.673398+0000","last_active":"2024-03-23T11:26:26.144838+0000","last_peered":"2024-03-23T11:26:26.144838+0000","last_clean":"2024-03-23T11:26:26.144838+0000","last_became_active":"2024-03-23T11:26:24.671762+0000","last_became_peered":"2024-03-23T11:26:24.671762+0000","last_unstale":"2024-03-23T11:26:26.144838+0000","last_undegraded":"2024-03-23T11:26:26.144838+0000","last_fullsized":"2024-03-23T11:26:26.144838+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:25.909709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488525+0000","last_change":"2024-03-23T11:26:25.168717+0000","last_active":"2024-03-23T11:26:26.488525+0000","last_peered":"2024-03-23T11:26:26.488525+0000","last_clean":"2024-03-23T11:26:26.488525+0000","last_became_active":"2024-03-23T11:26:25.168438+0000","last_became_peered":"2024-03-23T11:26:25.168438+0000","last_unstale":"2024-03-23T11:26:26.488525+0000","last_undegraded":"2024-03-23T11:26:26.488525+0000","last_fullsized":"2024-03-23T11:26:26.488525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:17:45.197260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.293973+0000","last_change":"2024-03-23T11:26:24.666191+0000","last_active":"2024-03-23T11:26:26.293973+0000","last_peered":"2024-03-23T11:26:26.293973+0000","last_clean":"2024-03-23T11:26:26.293973+0000","last_became_active":"2024-03-23T11:26:24.665892+0000","last_became_peered":"2024-03-23T11:26:24.665892+0000","last_unstale":"2024-03-23T11:26:26.293973+0000","last_undegraded":"2024-03-23T11:26:26.293973+0000","last_fullsized":"2024-03-23T11:26:26.293973+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:13:08.985209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147245+0000","last_change":"2024-03-23T11:26:25.167577+0000","last_active":"2024-03-23T11:26:26.147245+0000","last_peered":"2024-03-23T11:26:26.147245+0000","last_clean":"2024-03-23T11:26:26.147245+0000","last_became_active":"2024-03-23T11:26:25.167235+0000","last_became_peered":"2024-03-23T11:26:25.167235+0000","last_unstale":"2024-03-23T11:26:26.147245+0000","last_undegraded":"2024-03-23T11:26:26.147245+0000","last_fullsized":"2024-03-23T11:26:26.147245+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:35:29.912325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488712+0000","last_change":"2024-03-23T11:26:24.671960+0000","last_active":"2024-03-23T11:26:26.488712+0000","last_peered":"2024-03-23T11:26:26.488712+0000","last_clean":"2024-03-23T11:26:26.488712+0000","last_became_active":"2024-03-23T11:26:24.671548+0000","last_became_peered":"2024-03-23T11:26:24.671548+0000","last_unstale":"2024-03-23T11:26:26.488712+0000","last_undegraded":"2024-03-23T11:26:26.488712+0000","last_fullsized":"2024-03-23T11:26:26.488712+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:47:44.295130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.148384+0000","last_change":"2024-03-23T11:26:25.179780+0000","last_active":"2024-03-23T11:26:26.148384+0000","last_peered":"2024-03-23T11:26:26.148384+0000","last_clean":"2024-03-23T11:26:26.148384+0000","last_became_active":"2024-03-23T11:26:25.179472+0000","last_became_peered":"2024-03-23T11:26:25.179472+0000","last_unstale":"2024-03-23T11:26:26.148384+0000","last_undegraded":"2024-03-23T11:26:26.148384+0000","last_fullsized":"2024-03-23T11:26:26.148384+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:41.196699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488755+0000","last_change":"2024-03-23T11:26:24.668371+0000","last_active":"2024-03-23T11:26:26.488755+0000","last_peered":"2024-03-23T11:26:26.488755+0000","last_clean":"2024-03-23T11:26:26.488755+0000","last_became_active":"2024-03-23T11:26:24.668094+0000","last_became_peered":"2024-03-23T11:26:24.668094+0000","last_unstale":"2024-03-23T11:26:26.488755+0000","last_undegraded":"2024-03-23T11:26:26.488755+0000","last_fullsized":"2024-03-23T11:26:26.488755+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:30:58.650839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490547+0000","last_change":"2024-03-23T11:26:25.179195+0000","last_active":"2024-03-23T11:26:26.490547+0000","last_peered":"2024-03-23T11:26:26.490547+0000","last_clean":"2024-03-23T11:26:26.490547+0000","last_became_active":"2024-03-23T11:26:25.178691+0000","last_became_peered":"2024-03-23T11:26:25.178691+0000","last_unstale":"2024-03-23T11:26:26.490547+0000","last_undegraded":"2024-03-23T11:26:26.490547+0000","last_fullsized":"2024-03-23T11:26:26.490547+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:10.886471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490899+0000","last_change":"2024-03-23T11:26:24.669717+0000","last_active":"2024-03-23T11:26:26.490899+0000","last_peered":"2024-03-23T11:26:26.490899+0000","last_clean":"2024-03-23T11:26:26.490899+0000","last_became_active":"2024-03-23T11:26:24.669011+0000","last_became_peered":"2024-03-23T11:26:24.669011+0000","last_unstale":"2024-03-23T11:26:26.490899+0000","last_undegraded":"2024-03-23T11:26:26.490899+0000","last_fullsized":"2024-03-23T11:26:26.490899+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:39:06.934929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146868+0000","last_change":"2024-03-23T11:26:25.171186+0000","last_active":"2024-03-23T11:26:26.146868+0000","last_peered":"2024-03-23T11:26:26.146868+0000","last_clean":"2024-03-23T11:26:26.146868+0000","last_became_active":"2024-03-23T11:26:25.170732+0000","last_became_peered":"2024-03-23T11:26:25.170732+0000","last_unstale":"2024-03-23T11:26:26.146868+0000","last_undegraded":"2024-03-23T11:26:26.146868+0000","last_fullsized":"2024-03-23T11:26:26.146868+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:56:46.397432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490673+0000","last_change":"2024-03-23T11:26:24.672425+0000","last_active":"2024-03-23T11:26:26.490673+0000","last_peered":"2024-03-23T11:26:26.490673+0000","last_clean":"2024-03-23T11:26:26.490673+0000","last_became_active":"2024-03-23T11:26:24.672075+0000","last_became_peered":"2024-03-23T11:26:24.672075+0000","last_unstale":"2024-03-23T11:26:26.490673+0000","last_undegraded":"2024-03-23T11:26:26.490673+0000","last_fullsized":"2024-03-23T11:26:26.490673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:40.266855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488869+0000","last_change":"2024-03-23T11:26:25.178223+0000","last_active":"2024-03-23T11:26:26.488869+0000","last_peered":"2024-03-23T11:26:26.488869+0000","last_clean":"2024-03-23T11:26:26.488869+0000","last_became_active":"2024-03-23T11:26:25.177291+0000","last_became_peered":"2024-03-23T11:26:25.177291+0000","last_unstale":"2024-03-23T11:26:26.488869+0000","last_undegraded":"2024-03-23T11:26:26.488869+0000","last_fullsized":"2024-03-23T11:26:26.488869+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:03:21.571996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489870+0000","last_change":"2024-03-23T11:26:24.665792+0000","last_active":"2024-03-23T11:26:26.489870+0000","last_peered":"2024-03-23T11:26:26.489870+0000","last_clean":"2024-03-23T11:26:26.489870+0000","last_became_active":"2024-03-23T11:26:24.665112+0000","last_became_peered":"2024-03-23T11:26:24.665112+0000","last_unstale":"2024-03-23T11:26:26.489870+0000","last_undegraded":"2024-03-23T11:26:26.489870+0000","last_fullsized":"2024-03-23T11:26:26.489870+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:18:24.663510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489593+0000","last_change":"2024-03-23T11:26:25.163589+0000","last_active":"2024-03-23T11:26:26.489593+0000","last_peered":"2024-03-23T11:26:26.489593+0000","last_clean":"2024-03-23T11:26:26.489593+0000","last_became_active":"2024-03-23T11:26:25.163275+0000","last_became_peered":"2024-03-23T11:26:25.163275+0000","last_unstale":"2024-03-23T11:26:26.489593+0000","last_undegraded":"2024-03-23T11:26:26.489593+0000","last_fullsized":"2024-03-23T11:26:26.489593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:08.692375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146563+0000","last_change":"2024-03-23T11:26:24.668434+0000","last_active":"2024-03-23T11:26:26.146563+0000","last_peered":"2024-03-23T11:26:26.146563+0000","last_clean":"2024-03-23T11:26:26.146563+0000","last_became_active":"2024-03-23T11:26:24.668116+0000","last_became_peered":"2024-03-23T11:26:24.668116+0000","last_unstale":"2024-03-23T11:26:26.146563+0000","last_undegraded":"2024-03-23T11:26:26.146563+0000","last_fullsized":"2024-03-23T11:26:26.146563+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:49:47.737859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145701+0000","last_change":"2024-03-23T11:26:25.173255+0000","last_active":"2024-03-23T11:26:26.145701+0000","last_peered":"2024-03-23T11:26:26.145701+0000","last_clean":"2024-03-23T11:26:26.145701+0000","last_became_active":"2024-03-23T11:26:25.172181+0000","last_became_peered":"2024-03-23T11:26:25.172181+0000","last_unstale":"2024-03-23T11:26:26.145701+0000","last_undegraded":"2024-03-23T11:26:26.145701+0000","last_fullsized":"2024-03-23T11:26:26.145701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:52:38.435785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490031+0000","last_change":"2024-03-23T11:26:24.672211+0000","last_active":"2024-03-23T11:26:26.490031+0000","last_peered":"2024-03-23T11:26:26.490031+0000","last_clean":"2024-03-23T11:26:26.490031+0000","last_became_active":"2024-03-23T11:26:24.671848+0000","last_became_peered":"2024-03-23T11:26:24.671848+0000","last_unstale":"2024-03-23T11:26:26.490031+0000","last_undegraded":"2024-03-23T11:26:26.490031+0000","last_fullsized":"2024-03-23T11:26:26.490031+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:11:54.456971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489763+0000","last_change":"2024-03-23T11:26:25.178859+0000","last_active":"2024-03-23T11:26:26.489763+0000","last_peered":"2024-03-23T11:26:26.489763+0000","last_clean":"2024-03-23T11:26:26.489763+0000","last_became_active":"2024-03-23T11:26:25.178584+0000","last_became_peered":"2024-03-23T11:26:25.178584+0000","last_unstale":"2024-03-23T11:26:26.489763+0000","last_undegraded":"2024-03-23T11:26:26.489763+0000","last_fullsized":"2024-03-23T11:26:26.489763+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:09:31.199645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146202+0000","last_change":"2024-03-23T11:26:24.669960+0000","last_active":"2024-03-23T11:26:26.146202+0000","last_peered":"2024-03-23T11:26:26.146202+0000","last_clean":"2024-03-23T11:26:26.146202+0000","last_became_active":"2024-03-23T11:26:24.669208+0000","last_became_peered":"2024-03-23T11:26:24.669208+0000","last_unstale":"2024-03-23T11:26:26.146202+0000","last_undegraded":"2024-03-23T11:26:26.146202+0000","last_fullsized":"2024-03-23T11:26:26.146202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:46.322563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488188+0000","last_change":"2024-03-23T11:26:25.172071+0000","last_active":"2024-03-23T11:26:26.488188+0000","last_peered":"2024-03-23T11:26:26.488188+0000","last_clean":"2024-03-23T11:26:26.488188+0000","last_became_active":"2024-03-23T11:26:25.171820+0000","last_became_peered":"2024-03-23T11:26:25.171820+0000","last_unstale":"2024-03-23T11:26:26.488188+0000","last_undegraded":"2024-03-23T11:26:26.488188+0000","last_fullsized":"2024-03-23T11:26:26.488188+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:39:40.831693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.148051+0000","last_change":"2024-03-23T11:26:24.675232+0000","last_active":"2024-03-23T11:26:26.148051+0000","last_peered":"2024-03-23T11:26:26.148051+0000","last_clean":"2024-03-23T11:26:26.148051+0000","last_became_active":"2024-03-23T11:26:24.674490+0000","last_became_peered":"2024-03-23T11:26:24.674490+0000","last_unstale":"2024-03-23T11:26:26.148051+0000","last_undegraded":"2024-03-23T11:26:26.148051+0000","last_fullsized":"2024-03-23T11:26:26.148051+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:37:35.845166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":234,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T11:26:25.147211+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:26:25.147211+0000","last_peered":"2024-03-23T11:26:25.147211+0000","last_clean":"2024-03-23T11:26:25.147211+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:26:25.147211+0000","last_undegraded":"2024-03-23T11:26:25.147211+0000","last_fullsized":"2024-03-23T11:26:25.147211+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542239,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32772,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1687552,"data_stored":1055494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0469999999999999,"15min":0.94699999999999995},"min":{"1min":0.76400000000000001,"5min":0.67600000000000005,"15min":0.55900000000000005},"max":{"1min":1.423,"5min":1.5309999999999999,"15min":1.7470000000000001},"last":2.8370000000000002},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.0680000000000001,"15min":1.0049999999999999},"min":{"1min":0.77800000000000002,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.599,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.38300000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.173,"15min":1.1739999999999999},"min":{"1min":0.88400000000000001,"5min":0.71599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6399999999999999,"5min":1.7430000000000001,"15min":2.3820000000000001},"last":3.569},{"interface":"front","average":{"1min":1.381,"5min":1.2649999999999999,"15min":1.252},"min":{"1min":1.1180000000000001,"5min":0.77300000000000002,"15min":0.753},"max":{"1min":1.736,"5min":1.736,"15min":2.4350000000000001},"last":0.98599999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.0980000000000001,"15min":1.0629999999999999},"min":{"1min":0.93200000000000005,"5min":0.67500000000000004,"15min":0.66200000000000003},"max":{"1min":1.5549999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":2.7440000000000002},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.218,"15min":1.155},"min":{"1min":0.84499999999999997,"5min":0.81200000000000006,"15min":0.74299999999999999},"max":{"1min":1.675,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":3.524}]},{"osd":3,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2250000000000001,"15min":1.151},"min":{"1min":0.88100000000000001,"5min":0.76000000000000001,"15min":0.72799999999999998},"max":{"1min":1.6799999999999999,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":2.6389999999999998},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2629999999999999,"15min":1.254},"min":{"1min":1.0549999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.5940000000000001,"5min":1.6479999999999999,"15min":1.778},"last":2.9470000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.2350000000000001,"15min":1.2210000000000001},"min":{"1min":0.84699999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.8799999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4239999999999999,"15min":1.3720000000000001},"min":{"1min":1.1870000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6990000000000001,"5min":1.845,"15min":1.845},"last":2.5569999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.357,"15min":1.335},"min":{"1min":0.95399999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.8049999999999999,"5min":1.847,"15min":2.0590000000000002},"last":2.8999999999999999},{"interface":"front","average":{"1min":1.552,"5min":1.4650000000000001,"15min":1.381},"min":{"1min":1.3340000000000001,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":2.0859999999999999},"last":0.90000000000000002}]},{"osd":6,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.444,"15min":1.4239999999999999},"min":{"1min":1.2410000000000001,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9890000000000001},"last":1.026},{"interface":"front","average":{"1min":1.403,"5min":1.383,"15min":1.349},"min":{"1min":0.94299999999999995,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7949999999999999,"5min":1.887,"15min":1.887},"last":3.218}]}]},{"osd":6,"up_from":14,"seq":60129542239,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1097728,"data_stored":465081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.143,"15min":1.097},"min":{"1min":0.82299999999999995,"5min":0.63600000000000001,"15min":0.56899999999999995},"max":{"1min":1.651,"5min":1.8169999999999999,"15min":3.9140000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.1319999999999999,"15min":1.1240000000000001},"min":{"1min":0.94399999999999995,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6000000000000001,"5min":1.774,"15min":6.6539999999999999},"last":0.96699999999999997}]},{"osd":1,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.226,"15min":1.2190000000000001},"min":{"1min":0.88800000000000001,"5min":0.78000000000000003,"15min":0.71499999999999997},"max":{"1min":1.5269999999999999,"5min":1.7410000000000001,"15min":1.9430000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.254,"5min":1.196,"15min":1.2470000000000001},"min":{"1min":0.90900000000000003,"5min":0.76200000000000001,"15min":0.70399999999999996},"max":{"1min":1.752,"5min":1.752,"15min":2.0859999999999999},"last":1.171}]},{"osd":2,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.212,"15min":1.244},"min":{"1min":0.873,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.5089999999999999,"5min":1.6419999999999999,"15min":3.73},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.2749999999999999,"15min":1.2909999999999999},"min":{"1min":0.90600000000000003,"5min":0.72899999999999998,"15min":0.68100000000000005},"max":{"1min":1.712,"5min":1.8340000000000001,"15min":2.2280000000000002},"last":0.90700000000000003}]},{"osd":3,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.2649999999999999,"15min":1.27},"min":{"1min":0.878,"5min":0.83099999999999996,"15min":0.749},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":12.914},"last":1.05},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.302,"15min":1.361},"min":{"1min":1.095,"5min":0.86599999999999999,"15min":0.72699999999999998},"max":{"1min":1.6899999999999999,"5min":1.7350000000000001,"15min":3.8100000000000001},"last":0.84499999999999997}]},{"osd":4,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.28,"15min":1.371},"min":{"1min":0.82599999999999996,"5min":0.79600000000000004,"15min":0.51100000000000001},"max":{"1min":1.603,"5min":2.2200000000000002,"15min":2.702},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3100000000000001,"15min":1.4039999999999999},"min":{"1min":0.96299999999999997,"5min":0.85399999999999998,"15min":0.82299999999999995},"max":{"1min":1.944,"5min":1.944,"15min":7.2140000000000004},"last":1.26}]},{"osd":5,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4179999999999999,"15min":1.522},"min":{"1min":1.008,"5min":0.93600000000000005,"15min":0.73899999999999999},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":3.097},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.468,"5min":1.4470000000000001,"15min":1.4930000000000001},"min":{"1min":1.238,"5min":0.90800000000000003,"15min":0.72499999999999998},"max":{"1min":1.7370000000000001,"5min":2.282,"15min":2.8109999999999999},"last":1.286}]},{"osd":7,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.524,"15min":1.651},"min":{"1min":1.2190000000000001,"5min":1.127,"15min":0.80400000000000005},"max":{"1min":2.028,"5min":2.028,"15min":4.9470000000000001},"last":1.395},{"interface":"front","average":{"1min":1.397,"5min":1.393,"15min":1.4910000000000001},"min":{"1min":1.075,"5min":1.014,"15min":0.57699999999999996},"max":{"1min":1.7250000000000001,"5min":1.903,"15min":3.7749999999999999},"last":1.238}]}]},{"osd":5,"up_from":14,"seq":60129542239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1105920,"data_stored":466642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.0620000000000001,"15min":1.165},"min":{"1min":0.76300000000000001,"5min":0.60399999999999998,"15min":0.55400000000000005},"max":{"1min":1.5009999999999999,"5min":1.9350000000000001,"15min":2.2999999999999998},"last":1.149},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1319999999999999,"15min":1.1499999999999999},"min":{"1min":0.72599999999999998,"5min":0.61499999999999999,"15min":0.47799999999999998},"max":{"1min":1.524,"5min":1.643,"15min":2.355},"last":0.81899999999999995}]},{"osd":1,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.119,"15min":1.1759999999999999},"min":{"1min":0.91000000000000003,"5min":0.66800000000000004,"15min":0.65300000000000002},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":2.2559999999999998},"last":1.206},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1419999999999999,"15min":1.2050000000000001},"min":{"1min":0.86599999999999999,"5min":0.66900000000000004,"15min":0.495},"max":{"1min":1.5429999999999999,"5min":1.841,"15min":1.8580000000000001},"last":1.45}]},{"osd":2,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2010000000000001,"15min":1.2490000000000001},"min":{"1min":0.879,"5min":0.71399999999999997,"15min":0.60199999999999998},"max":{"1min":1.6000000000000001,"5min":1.7390000000000001,"15min":2.012},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2290000000000001,"15min":1.2370000000000001},"min":{"1min":0.93600000000000005,"5min":0.72399999999999998,"15min":0.64400000000000002},"max":{"1min":1.4570000000000001,"5min":1.7649999999999999,"15min":1.946},"last":0.98399999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.2889999999999999,"15min":1.304},"min":{"1min":1.0069999999999999,"5min":0.76600000000000001,"15min":0.66800000000000004},"max":{"1min":1.6279999999999999,"5min":1.835,"15min":2.1539999999999999},"last":1.244},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3500000000000001,"15min":1.3959999999999999},"min":{"1min":1.016,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.577,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.296}]},{"osd":4,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2769999999999999,"15min":1.3899999999999999},"min":{"1min":0.83999999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.53,"5min":2.0230000000000001,"15min":2.2090000000000001},"last":1.337},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4530000000000001,"15min":1.5229999999999999},"min":{"1min":1.218,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.633,"5min":2.1720000000000002,"15min":2.5009999999999999},"last":1.3700000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.331,"15min":1.4510000000000001},"min":{"1min":0.85699999999999998,"5min":0.59199999999999997,"15min":0.48899999999999999},"max":{"1min":1.6060000000000001,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.339,"15min":1.494},"min":{"1min":0.92700000000000005,"5min":0.624,"15min":0.624},"max":{"1min":1.5980000000000001,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.55}]},{"osd":7,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.4059999999999999,"15min":1.5309999999999999},"min":{"1min":1.0249999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.6970000000000001,"5min":2.4449999999999998,"15min":2.7229999999999999},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.5049999999999999,"15min":1.595},"min":{"1min":1.1140000000000001,"5min":0.69099999999999995,"15min":0.60099999999999998},"max":{"1min":1.6850000000000001,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":1.103}]}]},{"osd":4,"up_from":14,"seq":60129542239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1089536,"data_stored":465056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.181,"15min":1.121},"min":{"1min":0.85499999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.613,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.391},{"interface":"front","average":{"1min":1.333,"5min":1.3,"15min":1.2310000000000001},"min":{"1min":1.0940000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.633,"5min":1.679,"15min":1.8009999999999999},"last":1.589}]},{"osd":1,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.2330000000000001,"15min":1.1379999999999999},"min":{"1min":0.86399999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6990000000000001,"5min":4.327,"15min":4.327},"last":1.45},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1699999999999999,"15min":1.119},"min":{"1min":0.83699999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.5149999999999999,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.98299999999999998}]},{"osd":2,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2130000000000001,"15min":1.2230000000000001},"min":{"1min":0.997,"5min":0.624,"15min":0.624},"max":{"1min":1.4750000000000001,"5min":2.6080000000000001,"15min":2.6080000000000001},"last":1.417},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.244},"min":{"1min":0.89500000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.927,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":1.321}]},{"osd":3,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2330000000000001,"15min":1.222},"min":{"1min":0.872,"5min":0.622,"15min":0.622},"max":{"1min":1.518,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":1.266},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3779999999999999,"15min":1.2809999999999999},"min":{"1min":0.93799999999999994,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.665,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":1.5049999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.2130000000000001,"15min":1.2709999999999999},"min":{"1min":0.88800000000000001,"5min":0.77500000000000002,"15min":0.74299999999999999},"max":{"1min":1.5740000000000001,"5min":2.508,"15min":2.508},"last":1.153},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.411,"15min":1.367},"min":{"1min":1.0660000000000001,"5min":0.79400000000000004,"15min":0.60499999999999998},"max":{"1min":1.843,"5min":5.859,"15min":5.859},"last":1.2050000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.427,"15min":1.393},"min":{"1min":1.27,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.607,"5min":5.3339999999999996,"15min":5.3339999999999996},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.514,"15min":1.4910000000000001},"min":{"1min":1.21,"5min":0.76300000000000001,"15min":0.72599999999999998},"max":{"1min":1.752,"5min":5.3940000000000001,"15min":5.3940000000000001},"last":1.375}]},{"osd":7,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.504,"15min":1.48},"min":{"1min":1.2210000000000001,"5min":0.93200000000000005,"15min":0.754},"max":{"1min":1.6870000000000001,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":1.528},{"interface":"front","average":{"1min":1.345,"5min":1.369,"15min":1.403},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.6799999999999999,"5min":2.641,"15min":2.641},"last":1.468}]}]},{"osd":3,"up_from":14,"seq":60129542239,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1101824,"data_stored":466620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.94799999999999995,"15min":0.95199999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.434},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.115,"15min":1.1060000000000001},"min":{"1min":0.503,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.3360000000000001,"5min":1.8879999999999999,"15min":2.0790000000000002},"last":0.55200000000000005}]},{"osd":1,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0089999999999999,"15min":1.004},"min":{"1min":0.499,"5min":0.499,"15min":0.442},"max":{"1min":1.8049999999999999,"5min":1.8180000000000001,"15min":1.976},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.274,"15min":1.266},"min":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.5},"max":{"1min":1.7589999999999999,"5min":1.8360000000000001,"15min":2.2309999999999999},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.105,"15min":1.141},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.53300000000000003},"max":{"1min":1.3420000000000001,"5min":2.0779999999999998,"15min":3.3109999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.1859999999999999,"15min":1.246},"min":{"1min":0.69799999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.9319999999999999,"5min":2.2440000000000002,"15min":2.7109999999999999},"last":0.51900000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2589999999999999,"15min":1.2869999999999999},"min":{"1min":0.90500000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.897,"5min":1.897,"15min":2.3490000000000002},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.452,"15min":1.516},"min":{"1min":1.095,"5min":0.998,"15min":0.998},"max":{"1min":1.8009999999999999,"5min":1.966,"15min":4.8970000000000002},"last":0.68400000000000005}]},{"osd":5,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.48,"15min":1.5209999999999999},"min":{"1min":1.054,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":1.9850000000000001,"5min":2.1520000000000001,"15min":3.4910000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.407,"15min":1.4610000000000001},"min":{"1min":1.0209999999999999,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":3.125},"last":1.1359999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.375,"15min":1.4990000000000001},"min":{"1min":0.89300000000000002,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.667,"5min":2.024,"15min":4.5499999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.546,"5min":1.504,"15min":1.5549999999999999},"min":{"1min":0.94699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.8450000000000002},"last":0.85099999999999998}]},{"osd":7,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4419999999999999,"15min":1.5109999999999999},"min":{"1min":0.92300000000000004,"5min":0.88800000000000001,"15min":0.877},"max":{"1min":1.8100000000000001,"5min":1.9279999999999999,"15min":2.952},"last":1.105},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.498,"15min":1.6080000000000001},"min":{"1min":1.05,"5min":0.91000000000000003,"15min":0.90900000000000003},"max":{"1min":1.8520000000000001,"5min":2,"15min":4.798},"last":1.181}]}]},{"osd":2,"up_from":14,"seq":60129542239,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1093632,"data_stored":465638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.93300000000000005,"15min":0.90800000000000003},"min":{"1min":0.72399999999999998,"5min":0.39400000000000002,"15min":0.318},"max":{"1min":1.3779999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.1259999999999999,"15min":1.022},"min":{"1min":0.92300000000000004,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.7310000000000001,"5min":1.7709999999999999,"15min":1.8220000000000001},"last":1.198}]},{"osd":1,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.88300000000000001,"15min":0.94299999999999995},"min":{"1min":0.76600000000000001,"5min":0.41499999999999998,"15min":0.34999999999999998},"max":{"1min":1.2689999999999999,"5min":1.429,"15min":1.9430000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.097,"5min":1.0489999999999999,"15min":1},"min":{"1min":0.79400000000000004,"5min":0.38400000000000001,"15min":0.35799999999999998},"max":{"1min":1.4870000000000001,"5min":1.542,"15min":1.7230000000000001},"last":1.075}]},{"osd":3,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.129,"15min":1.1200000000000001},"min":{"1min":0.878,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.905,"5min":1.905,"15min":1.98},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.0649999999999999,"15min":1.083},"min":{"1min":0.86299999999999999,"5min":0.47399999999999998,"15min":0.46200000000000002},"max":{"1min":1.52,"5min":2.0579999999999998,"15min":2.1560000000000001},"last":1.302}]},{"osd":4,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.373,"15min":1.3560000000000001},"min":{"1min":1.147,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.772,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.7},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4550000000000001,"15min":1.3779999999999999},"min":{"1min":1.1180000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.732,"5min":2.085,"15min":2.085},"last":1.5609999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.345,"15min":1.403},"min":{"1min":1.2609999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.6619999999999999,"5min":2.2589999999999999,"15min":2.5830000000000002},"last":1.254},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.393,"15min":1.379},"min":{"1min":1.2270000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.788,"5min":1.9119999999999999,"15min":2.6469999999999998},"last":1.452}]},{"osd":6,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.385,"15min":1.375},"min":{"1min":1.1859999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.875,"5min":1.9370000000000001,"15min":2.758},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.4199999999999999,"15min":1.4970000000000001},"min":{"1min":1.113,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.7090000000000001,"5min":2.0310000000000001,"15min":2.8580000000000001},"last":1.5169999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.4450000000000001,"15min":1.411},"min":{"1min":1.268,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.952,"5min":2.2069999999999999,"15min":2.246},"last":1.611},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.478,"15min":1.5700000000000001},"min":{"1min":1.1890000000000001,"5min":0.89200000000000002,"15min":0.75600000000000001},"max":{"1min":1.839,"5min":1.9710000000000001,"15min":2.7120000000000002},"last":1.649}]}]},{"osd":1,"up_from":14,"seq":60129542240,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32192,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339648,"statfs":{"total":96636764160,"available":96603799552,"internally_reserved":0,"allocated":1097728,"data_stored":464581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.91100000000000003,"15min":0.878},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.534,"5min":1.6459999999999999,"15min":3.2959999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0109999999999999,"15min":0.99099999999999999},"min":{"1min":0.71399999999999997,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5780000000000001,"5min":1.7070000000000001,"15min":1.99},"last":0.64500000000000002}]},{"osd":2,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.94299999999999995,"15min":0.96799999999999997},"min":{"1min":0.65800000000000003,"5min":0.505,"15min":0.40699999999999997},"max":{"1min":1.224,"5min":1.6479999999999999,"15min":1.738},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.0860000000000001,"15min":1.113},"min":{"1min":0.60299999999999998,"5min":0.55100000000000005,"15min":0.49399999999999999},"max":{"1min":1.536,"5min":1.8720000000000001,"15min":2.2160000000000002},"last":1.083}]},{"osd":3,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.24,"15min":1.224},"min":{"1min":0.69299999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.4930000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1040000000000001,"15min":1.1499999999999999},"min":{"1min":0.626,"5min":0.52000000000000002,"15min":0.434},"max":{"1min":1.5509999999999999,"5min":1.919,"15min":2.4169999999999998},"last":1.0549999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.345,"15min":1.276},"min":{"1min":1.077,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.825,"5min":2.0720000000000001,"15min":2.153},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.276,"15min":1.3220000000000001},"min":{"1min":0.81299999999999994,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.5409999999999999,"5min":1.8240000000000001,"15min":2.758},"last":1.4159999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.337,"15min":1.4099999999999999},"min":{"1min":0.91500000000000004,"5min":0.85299999999999998,"15min":0.81699999999999995},"max":{"1min":1.7230000000000001,"5min":2.0009999999999999,"15min":2.6589999999999998},"last":1.542},{"interface":"front","average":{"1min":1.494,"5min":1.419,"15min":1.45},"min":{"1min":1.038,"5min":0.80400000000000005,"15min":0.72199999999999998},"max":{"1min":1.7529999999999999,"5min":2.165,"15min":3.0329999999999999},"last":1.202}]},{"osd":6,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.413,"15min":1.4299999999999999},"min":{"1min":0.89300000000000002,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.8380000000000001,"5min":2.0800000000000001,"15min":2.2730000000000001},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.403,"5min":1.4259999999999999,"15min":1.5349999999999999},"min":{"1min":0.97799999999999998,"5min":0.88200000000000001,"15min":0.83599999999999997},"max":{"1min":1.595,"5min":2.0270000000000001,"15min":2.9630000000000001},"last":1.3380000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.5049999999999999,"15min":1.5609999999999999},"min":{"1min":0.874,"5min":0.874,"15min":0.82699999999999996},"max":{"1min":1.788,"5min":2.3260000000000001,"15min":3.2749999999999999},"last":1.365},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.4990000000000001,"15min":1.5649999999999999},"min":{"1min":1.012,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.9330000000000001,"5min":2.0840000000000001,"15min":2.7829999999999999},"last":1.3899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542238,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32772,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1683456,"data_stored":1054789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.876,"15min":0.86299999999999999},"min":{"1min":0.68400000000000005,"5min":0.376,"15min":0.376},"max":{"1min":1.413,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.99299999999999999,"15min":0.94599999999999995},"min":{"1min":0.71999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.52,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0660000000000001,"15min":1.04},"min":{"1min":0.85799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.361,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0629999999999999,"15min":1.0249999999999999},"min":{"1min":0.85799999999999998,"5min":0.432,"15min":0.432},"max":{"1min":1.429,"5min":2.5760000000000001,"15min":2.5760000000000001},"last":0.66400000000000003}]},{"osd":3,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.1479999999999999,"15min":1.111},"min":{"1min":0.84599999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.8440000000000001,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.349,"15min":1.323},"min":{"1min":1.155,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.7529999999999999,"5min":2.9390000000000001,"15min":2.9390000000000001},"last":0.73099999999999998}]},{"osd":4,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3200000000000001,"15min":1.3620000000000001},"min":{"1min":1.077,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6859999999999999,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.367,"5min":1.302,"15min":1.282},"min":{"1min":1.133,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.7230000000000001,"5min":3.125,"15min":3.125},"last":1.077}]},{"osd":5,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.341,"15min":1.421},"min":{"1min":1.212,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.621,"5min":3.0830000000000002,"15min":3.0830000000000002},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.385,"5min":1.399,"15min":1.5049999999999999},"min":{"1min":1.046,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.6850000000000001,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":1.0249999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4279999999999999,"15min":1.5289999999999999},"min":{"1min":1.2929999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7010000000000001,"5min":3.391,"15min":3.391},"last":1.284},{"interface":"front","average":{"1min":1.393,"5min":1.3859999999999999,"15min":1.4219999999999999},"min":{"1min":1.0369999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.667,"5min":3.153,"15min":3.153},"last":0.81999999999999995}]},{"osd":7,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.5,"15min":1.54},"min":{"1min":1.298,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.78,"5min":3.5350000000000001,"15min":3.5350000000000001},"last":1.147},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4890000000000001,"15min":1.5069999999999999},"min":{"1min":1.137,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.7589999999999999,"5min":3.3650000000000002,"15min":3.3650000000000002},"last":0.84899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:26:29.726 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:26:29.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.980+0000 7f1028e00640 1 Processor -- start 2024-03-23T11:26:29.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.980+0000 7f1028e00640 1 -- start start 2024-03-23T11:26:29.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.981+0000 7f1028e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f102415fcf0 0x7f10240d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.981+0000 7f1028e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1024160420 0x7f1024160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.982+0000 7f1028e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1024160dc0 con 0x7f10240d06d0 2024-03-23T11:26:29.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.982+0000 7f1028e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10240d3540 con 0x7f102415fcf0 2024-03-23T11:26:29.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.982+0000 7f1028e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10240ce510 con 0x7f1024160420 2024-03-23T11:26:29.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.982+0000 7f1023400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f102415fcf0 0x7f10240d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.982+0000 7f1022a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1024160420 0x7f1024160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.982+0000 7f1023400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f102415fcf0 0x7f10240d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:47380/0 (socket says 172.21.15.62:47380) 2024-03-23T11:26:29.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.982+0000 7f1023400640 1 -- 172.21.15.62:0/4150945894 learned_addr learned my addr 172.21.15.62:0/4150945894 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:29.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.983+0000 7f1022a00640 1 -- 172.21.15.62:0/4150945894 >> v2:172.21.15.120:3300/0 conn(0x7f102415fcf0 msgr2=0x7f10240d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:29.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.983+0000 7f1022a00640 1 --2- 172.21.15.62:0/4150945894 >> v2:172.21.15.120:3300/0 conn(0x7f102415fcf0 0x7f10240d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.983+0000 7f1022a00640 1 -- 172.21.15.62:0/4150945894 >> v1:172.21.15.120:6789/0 conn(0x7f10240d06d0 legacy=0x7f10240d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:29.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.983+0000 7f1022a00640 1 -- 172.21.15.62:0/4150945894 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10240cdc70 con 0x7f1024160420 2024-03-23T11:26:29.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.985+0000 7f1022a00640 1 --2- 172.21.15.62:0/4150945894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1024160420 0x7f1024160880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f10100097a0 tx=0x7f101002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2569ecbd04c3cf4f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:29.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.985+0000 7f1022000640 1 -- 172.21.15.62:0/4150945894 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101002f050 con 0x7f1024160420 2024-03-23T11:26:29.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.985+0000 7f1022000640 1 -- 172.21.15.62:0/4150945894 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1010031700 con 0x7f1024160420 2024-03-23T11:26:29.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.985+0000 7f1022000640 1 -- 172.21.15.62:0/4150945894 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010031a90 con 0x7f1024160420 2024-03-23T11:26:29.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.986+0000 7f1028e00640 1 -- 172.21.15.62:0/4150945894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1024160420 msgr2=0x7f1024160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.986+0000 7f1028e00640 1 --2- 172.21.15.62:0/4150945894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1024160420 0x7f1024160880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f10100097a0 tx=0x7f101002b7a0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.986+0000 7f1028e00640 1 -- 172.21.15.62:0/4150945894 shutdown_connections 2024-03-23T11:26:29.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.986+0000 7f1028e00640 1 --2- 172.21.15.62:0/4150945894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1024160420 0x7f1024160880 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.986+0000 7f1028e00640 1 --2- 172.21.15.62:0/4150945894 >> v2:172.21.15.120:3300/0 conn(0x7f102415fcf0 0x7f10240d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.986+0000 7f1028e00640 1 -- 172.21.15.62:0/4150945894 >> 172.21.15.62:0/4150945894 conn(0x7f1024089f60 msgr2=0x7f10240d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:29.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.986+0000 7f1028e00640 1 -- 172.21.15.62:0/4150945894 shutdown_connections 2024-03-23T11:26:29.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.987+0000 7f1028e00640 1 -- 172.21.15.62:0/4150945894 wait complete. 2024-03-23T11:26:29.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.989+0000 7f1028e00640 1 Processor -- start 2024-03-23T11:26:29.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.989+0000 7f1028e00640 1 -- start start 2024-03-23T11:26:29.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.990+0000 7f1028e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10240d06d0 0x7f102415a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.990+0000 7f1028e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f102415fcf0 0x7f102409da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.990+0000 7f1028e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1024160420 0x7f10240cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.990+0000 7f1028e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102409d270 con 0x7f10240d06d0 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.990+0000 7f1028e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102415ae80 con 0x7f102415fcf0 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.990+0000 7f1028e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102409dfb0 con 0x7f1024160420 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.991+0000 7f1023400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10240d06d0 0x7f102415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.991+0000 7f1023400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10240d06d0 0x7f102415a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54024/0 (socket says 172.21.15.62:54024) 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.991+0000 7f1023400640 1 -- 172.21.15.62:0/678476866 learned_addr learned my addr 172.21.15.62:0/678476866 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.991+0000 7f1023e00640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1024160420 0x7f10240cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.991+0000 7f1022a00640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f102415fcf0 0x7f102409da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.991+0000 7f1023400640 1 -- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1024160420 msgr2=0x7f10240cb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.991+0000 7f1023400640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1024160420 0x7f10240cb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.991+0000 7f1023400640 1 -- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f102415fcf0 msgr2=0x7f102409da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:29.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.991+0000 7f1023400640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f102415fcf0 0x7f102409da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:29.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.991+0000 7f1023400640 1 -- 172.21.15.62:0/678476866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1010009400 con 0x7f10240d06d0 2024-03-23T11:26:29.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.992+0000 7f1023400640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10240d06d0 0x7f102415a940 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f1014009fd0 tx=0x7f101400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:29.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.992+0000 7f101b400640 1 -- 172.21.15.62:0/678476866 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101400c940 con 0x7f10240d06d0 2024-03-23T11:26:29.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.992+0000 7f101b400640 1 -- 172.21.15.62:0/678476866 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101400caa0 con 0x7f10240d06d0 2024-03-23T11:26:29.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.992+0000 7f101b400640 1 -- 172.21.15.62:0/678476866 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101400ce30 con 0x7f10240d06d0 2024-03-23T11:26:29.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.992+0000 7f1028e00640 1 -- 172.21.15.62:0/678476866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10240c8950 con 0x7f10240d06d0 2024-03-23T11:26:29.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.992+0000 7f1028e00640 1 -- 172.21.15.62:0/678476866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10240c5ed0 con 0x7f10240d06d0 2024-03-23T11:26:29.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.993+0000 7f1028e00640 1 -- 172.21.15.62:0/678476866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1024066bb0 con 0x7f10240d06d0 2024-03-23T11:26:29.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.994+0000 7f101b400640 1 -- 172.21.15.62:0/678476866 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f1014013470 con 0x7f10240d06d0 2024-03-23T11:26:29.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.995+0000 7f101b400640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f100406a7a0 0x7f100406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:29.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.995+0000 7f101b400640 1 -- 172.21.15.62:0/678476866 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1014088f30 con 0x7f10240d06d0 2024-03-23T11:26:29.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.996+0000 7f1022a00640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f100406a7a0 0x7f100406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:29.997+0000 7f1022a00640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f100406a7a0 0x7f100406cc60 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1010035000 tx=0x7f1010008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:29.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.001+0000 7f101b400640 1 -- 172.21.15.62:0/678476866 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1014054340 con 0x7f10240d06d0 2024-03-23T11:26:30.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.198+0000 7f1028e00640 1 -- 172.21.15.62:0/678476866 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10240b55d0 con 0x7f100406a7a0 2024-03-23T11:26:30.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.221+0000 7f101b400640 1 -- 172.21.15.62:0/678476866 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286643 (secure 0 0 0) 0x7f1010044090 con 0x7f100406a7a0 2024-03-23T11:26:30.218 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:30.220 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:26:30.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.227+0000 7f1028e00640 1 -- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f100406a7a0 msgr2=0x7f100406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.227+0000 7f1028e00640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f100406a7a0 0x7f100406cc60 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1010035000 tx=0x7f1010008030 comp rx=0 tx=0).stop 2024-03-23T11:26:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.227+0000 7f1028e00640 1 -- 172.21.15.62:0/678476866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10240d06d0 msgr2=0x7f102415a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.227+0000 7f1028e00640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10240d06d0 0x7f102415a940 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f1014009fd0 tx=0x7f101400ed40 comp rx=0 tx=0).stop 2024-03-23T11:26:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.227+0000 7f1028e00640 1 -- 172.21.15.62:0/678476866 shutdown_connections 2024-03-23T11:26:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.227+0000 7f1028e00640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f100406a7a0 0x7f100406cc60 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.227+0000 7f1028e00640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1024160420 0x7f10240cb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.227+0000 7f1028e00640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f102415fcf0 0x7f102409da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.227+0000 7f1028e00640 1 --2- 172.21.15.62:0/678476866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10240d06d0 0x7f102415a940 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.227+0000 7f1028e00640 1 -- 172.21.15.62:0/678476866 >> 172.21.15.62:0/678476866 conn(0x7f1024089f60 msgr2=0x7f10240b5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.228+0000 7f1028e00640 1 -- 172.21.15.62:0/678476866 shutdown_connections 2024-03-23T11:26:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.228+0000 7f1028e00640 1 -- 172.21.15.62:0/678476866 wait complete. 2024-03-23T11:26:30.241 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-03-23T11:26:29.396524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254624,"kb_used_data":9724,"kb_used_omap":13,"kb_used_meta":244722,"kb_avail":754720096,"statfs":{"total":773094113280,"available":772833378304,"internally_reserved":0,"allocated":9957376,"data_stored":4903901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":250595416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.772083"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491033+0000","last_change":"2024-03-23T11:26:25.180067+0000","last_active":"2024-03-23T11:26:26.491033+0000","last_peered":"2024-03-23T11:26:26.491033+0000","last_clean":"2024-03-23T11:26:26.491033+0000","last_became_active":"2024-03-23T11:26:25.179491+0000","last_became_peered":"2024-03-23T11:26:25.179491+0000","last_unstale":"2024-03-23T11:26:26.491033+0000","last_undegraded":"2024-03-23T11:26:26.491033+0000","last_fullsized":"2024-03-23T11:26:26.491033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:06:58.990977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488596+0000","last_change":"2024-03-23T11:26:25.179202+0000","last_active":"2024-03-23T11:26:26.488596+0000","last_peered":"2024-03-23T11:26:26.488596+0000","last_clean":"2024-03-23T11:26:26.488596+0000","last_became_active":"2024-03-23T11:26:25.177537+0000","last_became_peered":"2024-03-23T11:26:25.177537+0000","last_unstale":"2024-03-23T11:26:26.488596+0000","last_undegraded":"2024-03-23T11:26:26.488596+0000","last_fullsized":"2024-03-23T11:26:26.488596+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:49.433614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147082+0000","last_change":"2024-03-23T11:26:25.174759+0000","last_active":"2024-03-23T11:26:26.147082+0000","last_peered":"2024-03-23T11:26:26.147082+0000","last_clean":"2024-03-23T11:26:26.147082+0000","last_became_active":"2024-03-23T11:26:25.174468+0000","last_became_peered":"2024-03-23T11:26:25.174468+0000","last_unstale":"2024-03-23T11:26:26.147082+0000","last_undegraded":"2024-03-23T11:26:26.147082+0000","last_fullsized":"2024-03-23T11:26:26.147082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:28:17.495305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145530+0000","last_change":"2024-03-23T11:26:25.154383+0000","last_active":"2024-03-23T11:26:26.145530+0000","last_peered":"2024-03-23T11:26:26.145530+0000","last_clean":"2024-03-23T11:26:26.145530+0000","last_became_active":"2024-03-23T11:26:25.154093+0000","last_became_peered":"2024-03-23T11:26:25.154093+0000","last_unstale":"2024-03-23T11:26:26.145530+0000","last_undegraded":"2024-03-23T11:26:26.145530+0000","last_fullsized":"2024-03-23T11:26:26.145530+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:43:59.471046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146029+0000","last_change":"2024-03-23T11:26:25.173149+0000","last_active":"2024-03-23T11:26:26.146029+0000","last_peered":"2024-03-23T11:26:26.146029+0000","last_clean":"2024-03-23T11:26:26.146029+0000","last_became_active":"2024-03-23T11:26:25.169216+0000","last_became_peered":"2024-03-23T11:26:25.169216+0000","last_unstale":"2024-03-23T11:26:26.146029+0000","last_undegraded":"2024-03-23T11:26:26.146029+0000","last_fullsized":"2024-03-23T11:26:26.146029+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:33:22.112218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488745+0000","last_change":"2024-03-23T11:26:25.178395+0000","last_active":"2024-03-23T11:26:26.488745+0000","last_peered":"2024-03-23T11:26:26.488745+0000","last_clean":"2024-03-23T11:26:26.488745+0000","last_became_active":"2024-03-23T11:26:25.178072+0000","last_became_peered":"2024-03-23T11:26:25.178072+0000","last_unstale":"2024-03-23T11:26:26.488745+0000","last_undegraded":"2024-03-23T11:26:26.488745+0000","last_fullsized":"2024-03-23T11:26:26.488745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:21.703839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.144567+0000","last_change":"2024-03-23T11:26:25.158554+0000","last_active":"2024-03-23T11:26:26.144567+0000","last_peered":"2024-03-23T11:26:26.144567+0000","last_clean":"2024-03-23T11:26:26.144567+0000","last_became_active":"2024-03-23T11:26:25.158118+0000","last_became_peered":"2024-03-23T11:26:25.158118+0000","last_unstale":"2024-03-23T11:26:26.144567+0000","last_undegraded":"2024-03-23T11:26:26.144567+0000","last_fullsized":"2024-03-23T11:26:26.144567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:25:38.549422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488517+0000","last_change":"2024-03-23T11:26:25.176171+0000","last_active":"2024-03-23T11:26:26.488517+0000","last_peered":"2024-03-23T11:26:26.488517+0000","last_clean":"2024-03-23T11:26:26.488517+0000","last_became_active":"2024-03-23T11:26:25.174866+0000","last_became_peered":"2024-03-23T11:26:25.174866+0000","last_unstale":"2024-03-23T11:26:26.488517+0000","last_undegraded":"2024-03-23T11:26:26.488517+0000","last_fullsized":"2024-03-23T11:26:26.488517+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:30:28.522489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489817+0000","last_change":"2024-03-23T11:26:25.175666+0000","last_active":"2024-03-23T11:26:26.489817+0000","last_peered":"2024-03-23T11:26:26.489817+0000","last_clean":"2024-03-23T11:26:26.489817+0000","last_became_active":"2024-03-23T11:26:25.175368+0000","last_became_peered":"2024-03-23T11:26:25.175368+0000","last_unstale":"2024-03-23T11:26:26.489817+0000","last_undegraded":"2024-03-23T11:26:26.489817+0000","last_fullsized":"2024-03-23T11:26:26.489817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:21:03.591269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491167+0000","last_change":"2024-03-23T11:26:25.182826+0000","last_active":"2024-03-23T11:26:26.491167+0000","last_peered":"2024-03-23T11:26:26.491167+0000","last_clean":"2024-03-23T11:26:26.491167+0000","last_became_active":"2024-03-23T11:26:25.179154+0000","last_became_peered":"2024-03-23T11:26:25.179154+0000","last_unstale":"2024-03-23T11:26:26.491167+0000","last_undegraded":"2024-03-23T11:26:26.491167+0000","last_fullsized":"2024-03-23T11:26:26.491167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:54:20.316933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488251+0000","last_change":"2024-03-23T11:26:25.160264+0000","last_active":"2024-03-23T11:26:26.488251+0000","last_peered":"2024-03-23T11:26:26.488251+0000","last_clean":"2024-03-23T11:26:26.488251+0000","last_became_active":"2024-03-23T11:26:25.160001+0000","last_became_peered":"2024-03-23T11:26:25.160001+0000","last_unstale":"2024-03-23T11:26:26.488251+0000","last_undegraded":"2024-03-23T11:26:26.488251+0000","last_fullsized":"2024-03-23T11:26:26.488251+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:56:46.539720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490047+0000","last_change":"2024-03-23T11:26:25.180646+0000","last_active":"2024-03-23T11:26:26.490047+0000","last_peered":"2024-03-23T11:26:26.490047+0000","last_clean":"2024-03-23T11:26:26.490047+0000","last_became_active":"2024-03-23T11:26:25.180316+0000","last_became_peered":"2024-03-23T11:26:25.180316+0000","last_unstale":"2024-03-23T11:26:26.490047+0000","last_undegraded":"2024-03-23T11:26:26.490047+0000","last_fullsized":"2024-03-23T11:26:26.490047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:37.845104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147300+0000","last_change":"2024-03-23T11:26:25.162043+0000","last_active":"2024-03-23T11:26:26.147300+0000","last_peered":"2024-03-23T11:26:26.147300+0000","last_clean":"2024-03-23T11:26:26.147300+0000","last_became_active":"2024-03-23T11:26:25.161722+0000","last_became_peered":"2024-03-23T11:26:25.161722+0000","last_unstale":"2024-03-23T11:26:26.147300+0000","last_undegraded":"2024-03-23T11:26:26.147300+0000","last_fullsized":"2024-03-23T11:26:26.147300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:43:28.253054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.144690+0000","last_change":"2024-03-23T11:26:25.154154+0000","last_active":"2024-03-23T11:26:26.144690+0000","last_peered":"2024-03-23T11:26:26.144690+0000","last_clean":"2024-03-23T11:26:26.144690+0000","last_became_active":"2024-03-23T11:26:25.153823+0000","last_became_peered":"2024-03-23T11:26:25.153823+0000","last_unstale":"2024-03-23T11:26:26.144690+0000","last_undegraded":"2024-03-23T11:26:26.144690+0000","last_fullsized":"2024-03-23T11:26:26.144690+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:56:46.817396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.294004+0000","last_change":"2024-03-23T11:26:25.160147+0000","last_active":"2024-03-23T11:26:26.294004+0000","last_peered":"2024-03-23T11:26:26.294004+0000","last_clean":"2024-03-23T11:26:26.294004+0000","last_became_active":"2024-03-23T11:26:25.159887+0000","last_became_peered":"2024-03-23T11:26:25.159887+0000","last_unstale":"2024-03-23T11:26:26.294004+0000","last_undegraded":"2024-03-23T11:26:26.294004+0000","last_fullsized":"2024-03-23T11:26:26.294004+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:50:28.540065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146183+0000","last_change":"2024-03-23T11:26:25.175501+0000","last_active":"2024-03-23T11:26:26.146183+0000","last_peered":"2024-03-23T11:26:26.146183+0000","last_clean":"2024-03-23T11:26:26.146183+0000","last_became_active":"2024-03-23T11:26:25.175230+0000","last_became_peered":"2024-03-23T11:26:25.175230+0000","last_unstale":"2024-03-23T11:26:26.146183+0000","last_undegraded":"2024-03-23T11:26:26.146183+0000","last_fullsized":"2024-03-23T11:26:26.146183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:52.469820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147645+0000","last_change":"2024-03-23T11:26:25.161519+0000","last_active":"2024-03-23T11:26:26.147645+0000","last_peered":"2024-03-23T11:26:26.147645+0000","last_clean":"2024-03-23T11:26:26.147645+0000","last_became_active":"2024-03-23T11:26:25.161200+0000","last_became_peered":"2024-03-23T11:26:25.161200+0000","last_unstale":"2024-03-23T11:26:26.147645+0000","last_undegraded":"2024-03-23T11:26:26.147645+0000","last_fullsized":"2024-03-23T11:26:26.147645+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:10.886471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145079+0000","last_change":"2024-03-23T11:26:25.180169+0000","last_active":"2024-03-23T11:26:26.145079+0000","last_peered":"2024-03-23T11:26:26.145079+0000","last_clean":"2024-03-23T11:26:26.145079+0000","last_became_active":"2024-03-23T11:26:25.179906+0000","last_became_peered":"2024-03-23T11:26:25.179906+0000","last_unstale":"2024-03-23T11:26:26.145079+0000","last_undegraded":"2024-03-23T11:26:26.145079+0000","last_fullsized":"2024-03-23T11:26:26.145079+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:01:57.170178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490169+0000","last_change":"2024-03-23T11:26:25.178448+0000","last_active":"2024-03-23T11:26:26.490169+0000","last_peered":"2024-03-23T11:26:26.490169+0000","last_clean":"2024-03-23T11:26:26.490169+0000","last_became_active":"2024-03-23T11:26:25.177971+0000","last_became_peered":"2024-03-23T11:26:25.177971+0000","last_unstale":"2024-03-23T11:26:26.490169+0000","last_undegraded":"2024-03-23T11:26:26.490169+0000","last_fullsized":"2024-03-23T11:26:26.490169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:34.862798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145630+0000","last_change":"2024-03-23T11:26:25.158769+0000","last_active":"2024-03-23T11:26:26.145630+0000","last_peered":"2024-03-23T11:26:26.145630+0000","last_clean":"2024-03-23T11:26:26.145630+0000","last_became_active":"2024-03-23T11:26:25.158469+0000","last_became_peered":"2024-03-23T11:26:25.158469+0000","last_unstale":"2024-03-23T11:26:26.145630+0000","last_undegraded":"2024-03-23T11:26:26.145630+0000","last_fullsized":"2024-03-23T11:26:26.145630+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:07.951537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145160+0000","last_change":"2024-03-23T11:26:25.170810+0000","last_active":"2024-03-23T11:26:26.145160+0000","last_peered":"2024-03-23T11:26:26.145160+0000","last_clean":"2024-03-23T11:26:26.145160+0000","last_became_active":"2024-03-23T11:26:25.170500+0000","last_became_peered":"2024-03-23T11:26:25.170500+0000","last_unstale":"2024-03-23T11:26:26.145160+0000","last_undegraded":"2024-03-23T11:26:26.145160+0000","last_fullsized":"2024-03-23T11:26:26.145160+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:57:23.323902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.487857+0000","last_change":"2024-03-23T11:26:25.168338+0000","last_active":"2024-03-23T11:26:26.487857+0000","last_peered":"2024-03-23T11:26:26.487857+0000","last_clean":"2024-03-23T11:26:26.487857+0000","last_became_active":"2024-03-23T11:26:25.168110+0000","last_became_peered":"2024-03-23T11:26:25.168110+0000","last_unstale":"2024-03-23T11:26:26.487857+0000","last_undegraded":"2024-03-23T11:26:26.487857+0000","last_fullsized":"2024-03-23T11:26:26.487857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:53:19.310572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489505+0000","last_change":"2024-03-23T11:26:25.178671+0000","last_active":"2024-03-23T11:26:26.489505+0000","last_peered":"2024-03-23T11:26:26.489505+0000","last_clean":"2024-03-23T11:26:26.489505+0000","last_became_active":"2024-03-23T11:26:25.178202+0000","last_became_peered":"2024-03-23T11:26:25.178202+0000","last_unstale":"2024-03-23T11:26:26.489505+0000","last_undegraded":"2024-03-23T11:26:26.489505+0000","last_fullsized":"2024-03-23T11:26:26.489505+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:56:46.397432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.144714+0000","last_change":"2024-03-23T11:26:25.170834+0000","last_active":"2024-03-23T11:26:26.144714+0000","last_peered":"2024-03-23T11:26:26.144714+0000","last_clean":"2024-03-23T11:26:26.144714+0000","last_became_active":"2024-03-23T11:26:25.170480+0000","last_became_peered":"2024-03-23T11:26:25.170480+0000","last_unstale":"2024-03-23T11:26:26.144714+0000","last_undegraded":"2024-03-23T11:26:26.144714+0000","last_fullsized":"2024-03-23T11:26:26.144714+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:33:39.993333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488664+0000","last_change":"2024-03-23T11:26:25.178428+0000","last_active":"2024-03-23T11:26:26.488664+0000","last_peered":"2024-03-23T11:26:26.488664+0000","last_clean":"2024-03-23T11:26:26.488664+0000","last_became_active":"2024-03-23T11:26:25.178170+0000","last_became_peered":"2024-03-23T11:26:25.178170+0000","last_unstale":"2024-03-23T11:26:26.488664+0000","last_undegraded":"2024-03-23T11:26:26.488664+0000","last_fullsized":"2024-03-23T11:26:26.488664+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:11:13.865349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.148198+0000","last_change":"2024-03-23T11:26:25.168575+0000","last_active":"2024-03-23T11:26:26.148198+0000","last_peered":"2024-03-23T11:26:26.148198+0000","last_clean":"2024-03-23T11:26:26.148198+0000","last_became_active":"2024-03-23T11:26:25.168304+0000","last_became_peered":"2024-03-23T11:26:25.168304+0000","last_unstale":"2024-03-23T11:26:26.148198+0000","last_undegraded":"2024-03-23T11:26:26.148198+0000","last_fullsized":"2024-03-23T11:26:26.148198+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:21:25.998148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145944+0000","last_change":"2024-03-23T11:26:25.162817+0000","last_active":"2024-03-23T11:26:26.145944+0000","last_peered":"2024-03-23T11:26:26.145944+0000","last_clean":"2024-03-23T11:26:26.145944+0000","last_became_active":"2024-03-23T11:26:25.161854+0000","last_became_peered":"2024-03-23T11:26:25.161854+0000","last_unstale":"2024-03-23T11:26:26.145944+0000","last_undegraded":"2024-03-23T11:26:26.145944+0000","last_fullsized":"2024-03-23T11:26:26.145944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:13:09.998627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490305+0000","last_change":"2024-03-23T11:26:25.179703+0000","last_active":"2024-03-23T11:26:26.490305+0000","last_peered":"2024-03-23T11:26:26.490305+0000","last_clean":"2024-03-23T11:26:26.490305+0000","last_became_active":"2024-03-23T11:26:25.179152+0000","last_became_peered":"2024-03-23T11:26:25.179152+0000","last_unstale":"2024-03-23T11:26:26.490305+0000","last_undegraded":"2024-03-23T11:26:26.490305+0000","last_fullsized":"2024-03-23T11:26:26.490305+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:47.516318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488193+0000","last_change":"2024-03-23T11:26:25.151893+0000","last_active":"2024-03-23T11:26:26.488193+0000","last_peered":"2024-03-23T11:26:26.488193+0000","last_clean":"2024-03-23T11:26:26.488193+0000","last_became_active":"2024-03-23T11:26:25.151625+0000","last_became_peered":"2024-03-23T11:26:25.151625+0000","last_unstale":"2024-03-23T11:26:26.488193+0000","last_undegraded":"2024-03-23T11:26:26.488193+0000","last_fullsized":"2024-03-23T11:26:26.488193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:36:16.570415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488948+0000","last_change":"2024-03-23T11:26:25.151063+0000","last_active":"2024-03-23T11:26:26.488948+0000","last_peered":"2024-03-23T11:26:26.488948+0000","last_clean":"2024-03-23T11:26:26.488948+0000","last_became_active":"2024-03-23T11:26:25.150753+0000","last_became_peered":"2024-03-23T11:26:25.150753+0000","last_unstale":"2024-03-23T11:26:26.488948+0000","last_undegraded":"2024-03-23T11:26:26.488948+0000","last_fullsized":"2024-03-23T11:26:26.488948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:19:57.485482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146068+0000","last_change":"2024-03-23T11:26:25.158789+0000","last_active":"2024-03-23T11:26:26.146068+0000","last_peered":"2024-03-23T11:26:26.146068+0000","last_clean":"2024-03-23T11:26:26.146068+0000","last_became_active":"2024-03-23T11:26:25.158117+0000","last_became_peered":"2024-03-23T11:26:25.158117+0000","last_unstale":"2024-03-23T11:26:26.146068+0000","last_undegraded":"2024-03-23T11:26:26.146068+0000","last_fullsized":"2024-03-23T11:26:26.146068+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:09.005346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489210+0000","last_change":"2024-03-23T11:26:25.183462+0000","last_active":"2024-03-23T11:26:26.489210+0000","last_peered":"2024-03-23T11:26:26.489210+0000","last_clean":"2024-03-23T11:26:26.489210+0000","last_became_active":"2024-03-23T11:26:25.183152+0000","last_became_peered":"2024-03-23T11:26:25.183152+0000","last_unstale":"2024-03-23T11:26:26.489210+0000","last_undegraded":"2024-03-23T11:26:26.489210+0000","last_fullsized":"2024-03-23T11:26:26.489210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:08.692375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147123+0000","last_change":"2024-03-23T11:26:25.174889+0000","last_active":"2024-03-23T11:26:26.147123+0000","last_peered":"2024-03-23T11:26:26.147123+0000","last_clean":"2024-03-23T11:26:26.147123+0000","last_became_active":"2024-03-23T11:26:25.174629+0000","last_became_peered":"2024-03-23T11:26:25.174629+0000","last_unstale":"2024-03-23T11:26:26.147123+0000","last_undegraded":"2024-03-23T11:26:26.147123+0000","last_fullsized":"2024-03-23T11:26:26.147123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:32:36.038543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"88'4","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491218+0000","last_change":"2024-03-23T11:26:24.671290+0000","last_active":"2024-03-23T11:26:26.491218+0000","last_peered":"2024-03-23T11:26:26.491218+0000","last_clean":"2024-03-23T11:26:26.491218+0000","last_became_active":"2024-03-23T11:26:24.670843+0000","last_became_peered":"2024-03-23T11:26:24.670843+0000","last_unstale":"2024-03-23T11:26:26.491218+0000","last_undegraded":"2024-03-23T11:26:26.491218+0000","last_fullsized":"2024-03-23T11:26:26.491218+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:19:36.996545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491089+0000","last_change":"2024-03-23T11:26:25.180388+0000","last_active":"2024-03-23T11:26:26.491089+0000","last_peered":"2024-03-23T11:26:26.491089+0000","last_clean":"2024-03-23T11:26:26.491089+0000","last_became_active":"2024-03-23T11:26:25.179946+0000","last_became_peered":"2024-03-23T11:26:25.179946+0000","last_unstale":"2024-03-23T11:26:26.491089+0000","last_undegraded":"2024-03-23T11:26:26.491089+0000","last_fullsized":"2024-03-23T11:26:26.491089+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:17.879569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147342+0000","last_change":"2024-03-23T11:26:24.669998+0000","last_active":"2024-03-23T11:26:26.147342+0000","last_peered":"2024-03-23T11:26:26.147342+0000","last_clean":"2024-03-23T11:26:26.147342+0000","last_became_active":"2024-03-23T11:26:24.669690+0000","last_became_peered":"2024-03-23T11:26:24.669690+0000","last_unstale":"2024-03-23T11:26:26.147342+0000","last_undegraded":"2024-03-23T11:26:26.147342+0000","last_fullsized":"2024-03-23T11:26:26.147342+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:54:32.796874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490263+0000","last_change":"2024-03-23T11:26:25.181106+0000","last_active":"2024-03-23T11:26:26.490263+0000","last_peered":"2024-03-23T11:26:26.490263+0000","last_clean":"2024-03-23T11:26:26.490263+0000","last_became_active":"2024-03-23T11:26:25.180695+0000","last_became_peered":"2024-03-23T11:26:25.180695+0000","last_unstale":"2024-03-23T11:26:26.490263+0000","last_undegraded":"2024-03-23T11:26:26.490263+0000","last_fullsized":"2024-03-23T11:26:26.490263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:34.862798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490151+0000","last_change":"2024-03-23T11:26:24.670158+0000","last_active":"2024-03-23T11:26:26.490151+0000","last_peered":"2024-03-23T11:26:26.490151+0000","last_clean":"2024-03-23T11:26:26.490151+0000","last_became_active":"2024-03-23T11:26:24.666903+0000","last_became_peered":"2024-03-23T11:26:24.666903+0000","last_unstale":"2024-03-23T11:26:26.490151+0000","last_undegraded":"2024-03-23T11:26:26.490151+0000","last_fullsized":"2024-03-23T11:26:26.490151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:07.991929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.148010+0000","last_change":"2024-03-23T11:26:25.162193+0000","last_active":"2024-03-23T11:26:26.148010+0000","last_peered":"2024-03-23T11:26:26.148010+0000","last_clean":"2024-03-23T11:26:26.148010+0000","last_became_active":"2024-03-23T11:26:25.161332+0000","last_became_peered":"2024-03-23T11:26:25.161332+0000","last_unstale":"2024-03-23T11:26:26.148010+0000","last_undegraded":"2024-03-23T11:26:26.148010+0000","last_fullsized":"2024-03-23T11:26:26.148010+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:12:51.654663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490641+0000","last_change":"2024-03-23T11:26:24.669843+0000","last_active":"2024-03-23T11:26:26.490641+0000","last_peered":"2024-03-23T11:26:26.490641+0000","last_clean":"2024-03-23T11:26:26.490641+0000","last_became_active":"2024-03-23T11:26:24.669521+0000","last_became_peered":"2024-03-23T11:26:24.669521+0000","last_unstale":"2024-03-23T11:26:26.490641+0000","last_undegraded":"2024-03-23T11:26:26.490641+0000","last_fullsized":"2024-03-23T11:26:26.490641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:26:05.728586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488840+0000","last_change":"2024-03-23T11:26:25.158898+0000","last_active":"2024-03-23T11:26:26.488840+0000","last_peered":"2024-03-23T11:26:26.488840+0000","last_clean":"2024-03-23T11:26:26.488840+0000","last_became_active":"2024-03-23T11:26:25.155583+0000","last_became_peered":"2024-03-23T11:26:25.155583+0000","last_unstale":"2024-03-23T11:26:26.488840+0000","last_undegraded":"2024-03-23T11:26:26.488840+0000","last_fullsized":"2024-03-23T11:26:26.488840+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:58:03.691137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146556+0000","last_change":"2024-03-23T11:26:24.668582+0000","last_active":"2024-03-23T11:26:26.146556+0000","last_peered":"2024-03-23T11:26:26.146556+0000","last_clean":"2024-03-23T11:26:26.146556+0000","last_became_active":"2024-03-23T11:26:24.664916+0000","last_became_peered":"2024-03-23T11:26:24.664916+0000","last_unstale":"2024-03-23T11:26:26.146556+0000","last_undegraded":"2024-03-23T11:26:26.146556+0000","last_fullsized":"2024-03-23T11:26:26.146556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:43:01.127370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146263+0000","last_change":"2024-03-23T11:26:25.163953+0000","last_active":"2024-03-23T11:26:26.146263+0000","last_peered":"2024-03-23T11:26:26.146263+0000","last_clean":"2024-03-23T11:26:26.146263+0000","last_became_active":"2024-03-23T11:26:25.163684+0000","last_became_peered":"2024-03-23T11:26:25.163684+0000","last_unstale":"2024-03-23T11:26:26.146263+0000","last_undegraded":"2024-03-23T11:26:26.146263+0000","last_fullsized":"2024-03-23T11:26:26.146263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:09:31.199645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146523+0000","last_change":"2024-03-23T11:26:24.664701+0000","last_active":"2024-03-23T11:26:26.146523+0000","last_peered":"2024-03-23T11:26:26.146523+0000","last_clean":"2024-03-23T11:26:26.146523+0000","last_became_active":"2024-03-23T11:26:24.664325+0000","last_became_peered":"2024-03-23T11:26:24.664325+0000","last_unstale":"2024-03-23T11:26:26.146523+0000","last_undegraded":"2024-03-23T11:26:26.146523+0000","last_fullsized":"2024-03-23T11:26:26.146523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:47:04.633428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145444+0000","last_change":"2024-03-23T11:26:25.171512+0000","last_active":"2024-03-23T11:26:26.145444+0000","last_peered":"2024-03-23T11:26:26.145444+0000","last_clean":"2024-03-23T11:26:26.145444+0000","last_became_active":"2024-03-23T11:26:25.171180+0000","last_became_peered":"2024-03-23T11:26:25.171180+0000","last_unstale":"2024-03-23T11:26:26.145444+0000","last_undegraded":"2024-03-23T11:26:26.145444+0000","last_fullsized":"2024-03-23T11:26:26.145444+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:07:43.159151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489532+0000","last_change":"2024-03-23T11:26:24.663805+0000","last_active":"2024-03-23T11:26:26.489532+0000","last_peered":"2024-03-23T11:26:26.489532+0000","last_clean":"2024-03-23T11:26:26.489532+0000","last_became_active":"2024-03-23T11:26:24.663515+0000","last_became_peered":"2024-03-23T11:26:24.663515+0000","last_unstale":"2024-03-23T11:26:26.489532+0000","last_undegraded":"2024-03-23T11:26:26.489532+0000","last_fullsized":"2024-03-23T11:26:26.489532+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:32:35.025125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.294060+0000","last_change":"2024-03-23T11:26:25.159583+0000","last_active":"2024-03-23T11:26:26.294060+0000","last_peered":"2024-03-23T11:26:26.294060+0000","last_clean":"2024-03-23T11:26:26.294060+0000","last_became_active":"2024-03-23T11:26:25.158018+0000","last_became_peered":"2024-03-23T11:26:25.158018+0000","last_unstale":"2024-03-23T11:26:26.294060+0000","last_undegraded":"2024-03-23T11:26:26.294060+0000","last_fullsized":"2024-03-23T11:26:26.294060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:17.879569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488022+0000","last_change":"2024-03-23T11:26:24.673233+0000","last_active":"2024-03-23T11:26:26.488022+0000","last_peered":"2024-03-23T11:26:26.488022+0000","last_clean":"2024-03-23T11:26:26.488022+0000","last_became_active":"2024-03-23T11:26:24.672939+0000","last_became_peered":"2024-03-23T11:26:24.672939+0000","last_unstale":"2024-03-23T11:26:26.488022+0000","last_undegraded":"2024-03-23T11:26:26.488022+0000","last_fullsized":"2024-03-23T11:26:26.488022+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:04:43.824387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.294467+0000","last_change":"2024-03-23T11:26:25.148389+0000","last_active":"2024-03-23T11:26:26.294467+0000","last_peered":"2024-03-23T11:26:26.294467+0000","last_clean":"2024-03-23T11:26:26.294467+0000","last_became_active":"2024-03-23T11:26:25.148079+0000","last_became_peered":"2024-03-23T11:26:25.148079+0000","last_unstale":"2024-03-23T11:26:26.294467+0000","last_undegraded":"2024-03-23T11:26:26.294467+0000","last_fullsized":"2024-03-23T11:26:26.294467+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:21.703839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491320+0000","last_change":"2024-03-23T11:26:24.672251+0000","last_active":"2024-03-23T11:26:26.491320+0000","last_peered":"2024-03-23T11:26:26.491320+0000","last_clean":"2024-03-23T11:26:26.491320+0000","last_became_active":"2024-03-23T11:26:24.671880+0000","last_became_peered":"2024-03-23T11:26:24.671880+0000","last_unstale":"2024-03-23T11:26:26.491320+0000","last_undegraded":"2024-03-23T11:26:26.491320+0000","last_fullsized":"2024-03-23T11:26:26.491320+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:04:35.096903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489817+0000","last_change":"2024-03-23T11:26:25.172438+0000","last_active":"2024-03-23T11:26:26.489817+0000","last_peered":"2024-03-23T11:26:26.489817+0000","last_clean":"2024-03-23T11:26:26.489817+0000","last_became_active":"2024-03-23T11:26:25.172149+0000","last_became_peered":"2024-03-23T11:26:25.172149+0000","last_unstale":"2024-03-23T11:26:26.489817+0000","last_undegraded":"2024-03-23T11:26:26.489817+0000","last_fullsized":"2024-03-23T11:26:26.489817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:43:29.366812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491289+0000","last_change":"2024-03-23T11:26:24.670501+0000","last_active":"2024-03-23T11:26:26.491289+0000","last_peered":"2024-03-23T11:26:26.491289+0000","last_clean":"2024-03-23T11:26:26.491289+0000","last_became_active":"2024-03-23T11:26:24.670157+0000","last_became_peered":"2024-03-23T11:26:24.670157+0000","last_unstale":"2024-03-23T11:26:26.491289+0000","last_undegraded":"2024-03-23T11:26:26.491289+0000","last_fullsized":"2024-03-23T11:26:26.491289+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:46.322563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490365+0000","last_change":"2024-03-23T11:26:25.177116+0000","last_active":"2024-03-23T11:26:26.490365+0000","last_peered":"2024-03-23T11:26:26.490365+0000","last_clean":"2024-03-23T11:26:26.490365+0000","last_became_active":"2024-03-23T11:26:25.176582+0000","last_became_peered":"2024-03-23T11:26:25.176582+0000","last_unstale":"2024-03-23T11:26:26.490365+0000","last_undegraded":"2024-03-23T11:26:26.490365+0000","last_fullsized":"2024-03-23T11:26:26.490365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:47.516318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"88'3","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490695+0000","last_change":"2024-03-23T11:26:24.675259+0000","last_active":"2024-03-23T11:26:26.490695+0000","last_peered":"2024-03-23T11:26:26.490695+0000","last_clean":"2024-03-23T11:26:26.490695+0000","last_became_active":"2024-03-23T11:26:24.675020+0000","last_became_peered":"2024-03-23T11:26:24.675020+0000","last_unstale":"2024-03-23T11:26:26.490695+0000","last_undegraded":"2024-03-23T11:26:26.490695+0000","last_fullsized":"2024-03-23T11:26:26.490695+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:52:05.497089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146202+0000","last_change":"2024-03-23T11:26:25.172849+0000","last_active":"2024-03-23T11:26:26.146202+0000","last_peered":"2024-03-23T11:26:26.146202+0000","last_clean":"2024-03-23T11:26:26.146202+0000","last_became_active":"2024-03-23T11:26:25.172536+0000","last_became_peered":"2024-03-23T11:26:25.172536+0000","last_unstale":"2024-03-23T11:26:26.146202+0000","last_undegraded":"2024-03-23T11:26:26.146202+0000","last_fullsized":"2024-03-23T11:26:26.146202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:37.845104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"88'6","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489214+0000","last_change":"2024-03-23T11:26:24.664637+0000","last_active":"2024-03-23T11:26:26.489214+0000","last_peered":"2024-03-23T11:26:26.489214+0000","last_clean":"2024-03-23T11:26:26.489214+0000","last_became_active":"2024-03-23T11:26:24.663951+0000","last_became_peered":"2024-03-23T11:26:24.663951+0000","last_unstale":"2024-03-23T11:26:26.489214+0000","last_undegraded":"2024-03-23T11:26:26.489214+0000","last_fullsized":"2024-03-23T11:26:26.489214+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:54:19.303515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489208+0000","last_change":"2024-03-23T11:26:25.161708+0000","last_active":"2024-03-23T11:26:26.489208+0000","last_peered":"2024-03-23T11:26:26.489208+0000","last_clean":"2024-03-23T11:26:26.489208+0000","last_became_active":"2024-03-23T11:26:25.161428+0000","last_became_peered":"2024-03-23T11:26:25.161428+0000","last_unstale":"2024-03-23T11:26:26.489208+0000","last_undegraded":"2024-03-23T11:26:26.489208+0000","last_fullsized":"2024-03-23T11:26:26.489208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:13:09.998627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488445+0000","last_change":"2024-03-23T11:26:24.665210+0000","last_active":"2024-03-23T11:26:26.488445+0000","last_peered":"2024-03-23T11:26:26.488445+0000","last_clean":"2024-03-23T11:26:26.488445+0000","last_became_active":"2024-03-23T11:26:24.664838+0000","last_became_peered":"2024-03-23T11:26:24.664838+0000","last_unstale":"2024-03-23T11:26:26.488445+0000","last_undegraded":"2024-03-23T11:26:26.488445+0000","last_fullsized":"2024-03-23T11:26:26.488445+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:03:06.157933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147554+0000","last_change":"2024-03-23T11:26:25.168101+0000","last_active":"2024-03-23T11:26:26.147554+0000","last_peered":"2024-03-23T11:26:26.147554+0000","last_clean":"2024-03-23T11:26:26.147554+0000","last_became_active":"2024-03-23T11:26:25.167767+0000","last_became_peered":"2024-03-23T11:26:25.167767+0000","last_unstale":"2024-03-23T11:26:26.147554+0000","last_undegraded":"2024-03-23T11:26:26.147554+0000","last_fullsized":"2024-03-23T11:26:26.147554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:20:59.367745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.294071+0000","last_change":"2024-03-23T11:26:24.671390+0000","last_active":"2024-03-23T11:26:26.294071+0000","last_peered":"2024-03-23T11:26:26.294071+0000","last_clean":"2024-03-23T11:26:26.294071+0000","last_became_active":"2024-03-23T11:26:24.671039+0000","last_became_peered":"2024-03-23T11:26:24.671039+0000","last_unstale":"2024-03-23T11:26:26.294071+0000","last_undegraded":"2024-03-23T11:26:26.294071+0000","last_fullsized":"2024-03-23T11:26:26.294071+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:56:45.384014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.487985+0000","last_change":"2024-03-23T11:26:25.149881+0000","last_active":"2024-03-23T11:26:26.487985+0000","last_peered":"2024-03-23T11:26:26.487985+0000","last_clean":"2024-03-23T11:26:26.487985+0000","last_became_active":"2024-03-23T11:26:25.149580+0000","last_became_peered":"2024-03-23T11:26:25.149580+0000","last_unstale":"2024-03-23T11:26:26.487985+0000","last_undegraded":"2024-03-23T11:26:26.487985+0000","last_fullsized":"2024-03-23T11:26:26.487985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:57:48.324512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488702+0000","last_change":"2024-03-23T11:26:24.669332+0000","last_active":"2024-03-23T11:26:26.488702+0000","last_peered":"2024-03-23T11:26:26.488702+0000","last_clean":"2024-03-23T11:26:26.488702+0000","last_became_active":"2024-03-23T11:26:24.668966+0000","last_became_peered":"2024-03-23T11:26:24.668966+0000","last_unstale":"2024-03-23T11:26:26.488702+0000","last_undegraded":"2024-03-23T11:26:26.488702+0000","last_fullsized":"2024-03-23T11:26:26.488702+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:56:45.384014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.487551+0000","last_change":"2024-03-23T11:26:25.174460+0000","last_active":"2024-03-23T11:26:26.487551+0000","last_peered":"2024-03-23T11:26:26.487551+0000","last_clean":"2024-03-23T11:26:26.487551+0000","last_became_active":"2024-03-23T11:26:25.174148+0000","last_became_peered":"2024-03-23T11:26:25.174148+0000","last_unstale":"2024-03-23T11:26:26.487551+0000","last_undegraded":"2024-03-23T11:26:26.487551+0000","last_fullsized":"2024-03-23T11:26:26.487551+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:53:34.156067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"88'8","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489310+0000","last_change":"2024-03-23T11:26:24.666094+0000","last_active":"2024-03-23T11:26:26.489310+0000","last_peered":"2024-03-23T11:26:26.489310+0000","last_clean":"2024-03-23T11:26:26.489310+0000","last_became_active":"2024-03-23T11:26:24.665790+0000","last_became_peered":"2024-03-23T11:26:24.665790+0000","last_unstale":"2024-03-23T11:26:26.489310+0000","last_undegraded":"2024-03-23T11:26:26.489310+0000","last_fullsized":"2024-03-23T11:26:26.489310+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:33:21.098801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.148147+0000","last_change":"2024-03-23T11:26:25.168013+0000","last_active":"2024-03-23T11:26:26.148147+0000","last_peered":"2024-03-23T11:26:26.148147+0000","last_clean":"2024-03-23T11:26:26.148147+0000","last_became_active":"2024-03-23T11:26:25.167721+0000","last_became_peered":"2024-03-23T11:26:25.167721+0000","last_unstale":"2024-03-23T11:26:26.148147+0000","last_undegraded":"2024-03-23T11:26:26.148147+0000","last_fullsized":"2024-03-23T11:26:26.148147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:40:01.187632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491466+0000","last_change":"2024-03-23T11:26:24.670293+0000","last_active":"2024-03-23T11:26:26.491466+0000","last_peered":"2024-03-23T11:26:26.491466+0000","last_clean":"2024-03-23T11:26:26.491466+0000","last_became_active":"2024-03-23T11:26:24.669998+0000","last_became_peered":"2024-03-23T11:26:24.669998+0000","last_unstale":"2024-03-23T11:26:26.491466+0000","last_undegraded":"2024-03-23T11:26:26.491466+0000","last_fullsized":"2024-03-23T11:26:26.491466+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:08:41.590426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.491193+0000","last_change":"2024-03-23T11:26:25.185110+0000","last_active":"2024-03-23T11:26:26.491193+0000","last_peered":"2024-03-23T11:26:26.491193+0000","last_clean":"2024-03-23T11:26:26.491193+0000","last_became_active":"2024-03-23T11:26:25.184733+0000","last_became_peered":"2024-03-23T11:26:25.184733+0000","last_unstale":"2024-03-23T11:26:26.491193+0000","last_undegraded":"2024-03-23T11:26:26.491193+0000","last_fullsized":"2024-03-23T11:26:26.491193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:13:09.998627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490075+0000","last_change":"2024-03-23T11:26:24.667647+0000","last_active":"2024-03-23T11:26:26.490075+0000","last_peered":"2024-03-23T11:26:26.490075+0000","last_clean":"2024-03-23T11:26:26.490075+0000","last_became_active":"2024-03-23T11:26:24.667301+0000","last_became_peered":"2024-03-23T11:26:24.667301+0000","last_unstale":"2024-03-23T11:26:26.490075+0000","last_undegraded":"2024-03-23T11:26:26.490075+0000","last_fullsized":"2024-03-23T11:26:26.490075+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:51.456402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146627+0000","last_change":"2024-03-23T11:26:25.171011+0000","last_active":"2024-03-23T11:26:26.146627+0000","last_peered":"2024-03-23T11:26:26.146627+0000","last_clean":"2024-03-23T11:26:26.146627+0000","last_became_active":"2024-03-23T11:26:25.170341+0000","last_became_peered":"2024-03-23T11:26:25.170341+0000","last_unstale":"2024-03-23T11:26:26.146627+0000","last_undegraded":"2024-03-23T11:26:26.146627+0000","last_fullsized":"2024-03-23T11:26:26.146627+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:54:20.316933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490633+0000","last_change":"2024-03-23T11:26:24.667917+0000","last_active":"2024-03-23T11:26:26.490633+0000","last_peered":"2024-03-23T11:26:26.490633+0000","last_clean":"2024-03-23T11:26:26.490633+0000","last_became_active":"2024-03-23T11:26:24.667626+0000","last_became_peered":"2024-03-23T11:26:24.667626+0000","last_unstale":"2024-03-23T11:26:26.490633+0000","last_undegraded":"2024-03-23T11:26:26.490633+0000","last_fullsized":"2024-03-23T11:26:26.490633+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:35:28.898907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.144924+0000","last_change":"2024-03-23T11:26:25.151407+0000","last_active":"2024-03-23T11:26:26.144924+0000","last_peered":"2024-03-23T11:26:26.144924+0000","last_clean":"2024-03-23T11:26:26.144924+0000","last_became_active":"2024-03-23T11:26:25.151060+0000","last_became_peered":"2024-03-23T11:26:25.151060+0000","last_unstale":"2024-03-23T11:26:26.144924+0000","last_undegraded":"2024-03-23T11:26:26.144924+0000","last_fullsized":"2024-03-23T11:26:26.144924+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:02.065602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.294312+0000","last_change":"2024-03-23T11:26:24.669604+0000","last_active":"2024-03-23T11:26:26.294312+0000","last_peered":"2024-03-23T11:26:26.294312+0000","last_clean":"2024-03-23T11:26:26.294312+0000","last_became_active":"2024-03-23T11:26:24.669320+0000","last_became_peered":"2024-03-23T11:26:24.669320+0000","last_unstale":"2024-03-23T11:26:26.294312+0000","last_undegraded":"2024-03-23T11:26:26.294312+0000","last_fullsized":"2024-03-23T11:26:26.294312+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:36.831686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488843+0000","last_change":"2024-03-23T11:26:25.175918+0000","last_active":"2024-03-23T11:26:26.488843+0000","last_peered":"2024-03-23T11:26:26.488843+0000","last_clean":"2024-03-23T11:26:26.488843+0000","last_became_active":"2024-03-23T11:26:25.174608+0000","last_became_peered":"2024-03-23T11:26:25.174608+0000","last_unstale":"2024-03-23T11:26:26.488843+0000","last_undegraded":"2024-03-23T11:26:26.488843+0000","last_fullsized":"2024-03-23T11:26:26.488843+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:36:42.753677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145104+0000","last_change":"2024-03-23T11:26:24.657738+0000","last_active":"2024-03-23T11:26:26.145104+0000","last_peered":"2024-03-23T11:26:26.145104+0000","last_clean":"2024-03-23T11:26:26.145104+0000","last_became_active":"2024-03-23T11:26:24.657363+0000","last_became_peered":"2024-03-23T11:26:24.657363+0000","last_unstale":"2024-03-23T11:26:26.145104+0000","last_undegraded":"2024-03-23T11:26:26.145104+0000","last_fullsized":"2024-03-23T11:26:26.145104+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:51:22.194384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145044+0000","last_change":"2024-03-23T11:26:25.166812+0000","last_active":"2024-03-23T11:26:26.145044+0000","last_peered":"2024-03-23T11:26:26.145044+0000","last_clean":"2024-03-23T11:26:26.145044+0000","last_became_active":"2024-03-23T11:26:25.166514+0000","last_became_peered":"2024-03-23T11:26:25.166514+0000","last_unstale":"2024-03-23T11:26:26.145044+0000","last_undegraded":"2024-03-23T11:26:26.145044+0000","last_fullsized":"2024-03-23T11:26:26.145044+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:17:42.796450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.144838+0000","last_change":"2024-03-23T11:26:24.673398+0000","last_active":"2024-03-23T11:26:26.144838+0000","last_peered":"2024-03-23T11:26:26.144838+0000","last_clean":"2024-03-23T11:26:26.144838+0000","last_became_active":"2024-03-23T11:26:24.671762+0000","last_became_peered":"2024-03-23T11:26:24.671762+0000","last_unstale":"2024-03-23T11:26:26.144838+0000","last_undegraded":"2024-03-23T11:26:26.144838+0000","last_fullsized":"2024-03-23T11:26:26.144838+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:25.909709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488525+0000","last_change":"2024-03-23T11:26:25.168717+0000","last_active":"2024-03-23T11:26:26.488525+0000","last_peered":"2024-03-23T11:26:26.488525+0000","last_clean":"2024-03-23T11:26:26.488525+0000","last_became_active":"2024-03-23T11:26:25.168438+0000","last_became_peered":"2024-03-23T11:26:25.168438+0000","last_unstale":"2024-03-23T11:26:26.488525+0000","last_undegraded":"2024-03-23T11:26:26.488525+0000","last_fullsized":"2024-03-23T11:26:26.488525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:17:45.197260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.293973+0000","last_change":"2024-03-23T11:26:24.666191+0000","last_active":"2024-03-23T11:26:26.293973+0000","last_peered":"2024-03-23T11:26:26.293973+0000","last_clean":"2024-03-23T11:26:26.293973+0000","last_became_active":"2024-03-23T11:26:24.665892+0000","last_became_peered":"2024-03-23T11:26:24.665892+0000","last_unstale":"2024-03-23T11:26:26.293973+0000","last_undegraded":"2024-03-23T11:26:26.293973+0000","last_fullsized":"2024-03-23T11:26:26.293973+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:13:08.985209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.147245+0000","last_change":"2024-03-23T11:26:25.167577+0000","last_active":"2024-03-23T11:26:26.147245+0000","last_peered":"2024-03-23T11:26:26.147245+0000","last_clean":"2024-03-23T11:26:26.147245+0000","last_became_active":"2024-03-23T11:26:25.167235+0000","last_became_peered":"2024-03-23T11:26:25.167235+0000","last_unstale":"2024-03-23T11:26:26.147245+0000","last_undegraded":"2024-03-23T11:26:26.147245+0000","last_fullsized":"2024-03-23T11:26:26.147245+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:35:29.912325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488712+0000","last_change":"2024-03-23T11:26:24.671960+0000","last_active":"2024-03-23T11:26:26.488712+0000","last_peered":"2024-03-23T11:26:26.488712+0000","last_clean":"2024-03-23T11:26:26.488712+0000","last_became_active":"2024-03-23T11:26:24.671548+0000","last_became_peered":"2024-03-23T11:26:24.671548+0000","last_unstale":"2024-03-23T11:26:26.488712+0000","last_undegraded":"2024-03-23T11:26:26.488712+0000","last_fullsized":"2024-03-23T11:26:26.488712+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:47:44.295130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.148384+0000","last_change":"2024-03-23T11:26:25.179780+0000","last_active":"2024-03-23T11:26:26.148384+0000","last_peered":"2024-03-23T11:26:26.148384+0000","last_clean":"2024-03-23T11:26:26.148384+0000","last_became_active":"2024-03-23T11:26:25.179472+0000","last_became_peered":"2024-03-23T11:26:25.179472+0000","last_unstale":"2024-03-23T11:26:26.148384+0000","last_undegraded":"2024-03-23T11:26:26.148384+0000","last_fullsized":"2024-03-23T11:26:26.148384+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:41.196699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488755+0000","last_change":"2024-03-23T11:26:24.668371+0000","last_active":"2024-03-23T11:26:26.488755+0000","last_peered":"2024-03-23T11:26:26.488755+0000","last_clean":"2024-03-23T11:26:26.488755+0000","last_became_active":"2024-03-23T11:26:24.668094+0000","last_became_peered":"2024-03-23T11:26:24.668094+0000","last_unstale":"2024-03-23T11:26:26.488755+0000","last_undegraded":"2024-03-23T11:26:26.488755+0000","last_fullsized":"2024-03-23T11:26:26.488755+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:30:58.650839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490547+0000","last_change":"2024-03-23T11:26:25.179195+0000","last_active":"2024-03-23T11:26:26.490547+0000","last_peered":"2024-03-23T11:26:26.490547+0000","last_clean":"2024-03-23T11:26:26.490547+0000","last_became_active":"2024-03-23T11:26:25.178691+0000","last_became_peered":"2024-03-23T11:26:25.178691+0000","last_unstale":"2024-03-23T11:26:26.490547+0000","last_undegraded":"2024-03-23T11:26:26.490547+0000","last_fullsized":"2024-03-23T11:26:26.490547+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:10.886471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490899+0000","last_change":"2024-03-23T11:26:24.669717+0000","last_active":"2024-03-23T11:26:26.490899+0000","last_peered":"2024-03-23T11:26:26.490899+0000","last_clean":"2024-03-23T11:26:26.490899+0000","last_became_active":"2024-03-23T11:26:24.669011+0000","last_became_peered":"2024-03-23T11:26:24.669011+0000","last_unstale":"2024-03-23T11:26:26.490899+0000","last_undegraded":"2024-03-23T11:26:26.490899+0000","last_fullsized":"2024-03-23T11:26:26.490899+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:39:06.934929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146868+0000","last_change":"2024-03-23T11:26:25.171186+0000","last_active":"2024-03-23T11:26:26.146868+0000","last_peered":"2024-03-23T11:26:26.146868+0000","last_clean":"2024-03-23T11:26:26.146868+0000","last_became_active":"2024-03-23T11:26:25.170732+0000","last_became_peered":"2024-03-23T11:26:25.170732+0000","last_unstale":"2024-03-23T11:26:26.146868+0000","last_undegraded":"2024-03-23T11:26:26.146868+0000","last_fullsized":"2024-03-23T11:26:26.146868+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:56:46.397432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490673+0000","last_change":"2024-03-23T11:26:24.672425+0000","last_active":"2024-03-23T11:26:26.490673+0000","last_peered":"2024-03-23T11:26:26.490673+0000","last_clean":"2024-03-23T11:26:26.490673+0000","last_became_active":"2024-03-23T11:26:24.672075+0000","last_became_peered":"2024-03-23T11:26:24.672075+0000","last_unstale":"2024-03-23T11:26:26.490673+0000","last_undegraded":"2024-03-23T11:26:26.490673+0000","last_fullsized":"2024-03-23T11:26:26.490673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:40.266855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488869+0000","last_change":"2024-03-23T11:26:25.178223+0000","last_active":"2024-03-23T11:26:26.488869+0000","last_peered":"2024-03-23T11:26:26.488869+0000","last_clean":"2024-03-23T11:26:26.488869+0000","last_became_active":"2024-03-23T11:26:25.177291+0000","last_became_peered":"2024-03-23T11:26:25.177291+0000","last_unstale":"2024-03-23T11:26:26.488869+0000","last_undegraded":"2024-03-23T11:26:26.488869+0000","last_fullsized":"2024-03-23T11:26:26.488869+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:03:21.571996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489870+0000","last_change":"2024-03-23T11:26:24.665792+0000","last_active":"2024-03-23T11:26:26.489870+0000","last_peered":"2024-03-23T11:26:26.489870+0000","last_clean":"2024-03-23T11:26:26.489870+0000","last_became_active":"2024-03-23T11:26:24.665112+0000","last_became_peered":"2024-03-23T11:26:24.665112+0000","last_unstale":"2024-03-23T11:26:26.489870+0000","last_undegraded":"2024-03-23T11:26:26.489870+0000","last_fullsized":"2024-03-23T11:26:26.489870+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:18:24.663510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489593+0000","last_change":"2024-03-23T11:26:25.163589+0000","last_active":"2024-03-23T11:26:26.489593+0000","last_peered":"2024-03-23T11:26:26.489593+0000","last_clean":"2024-03-23T11:26:26.489593+0000","last_became_active":"2024-03-23T11:26:25.163275+0000","last_became_peered":"2024-03-23T11:26:25.163275+0000","last_unstale":"2024-03-23T11:26:26.489593+0000","last_undegraded":"2024-03-23T11:26:26.489593+0000","last_fullsized":"2024-03-23T11:26:26.489593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:08.692375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146563+0000","last_change":"2024-03-23T11:26:24.668434+0000","last_active":"2024-03-23T11:26:26.146563+0000","last_peered":"2024-03-23T11:26:26.146563+0000","last_clean":"2024-03-23T11:26:26.146563+0000","last_became_active":"2024-03-23T11:26:24.668116+0000","last_became_peered":"2024-03-23T11:26:24.668116+0000","last_unstale":"2024-03-23T11:26:26.146563+0000","last_undegraded":"2024-03-23T11:26:26.146563+0000","last_fullsized":"2024-03-23T11:26:26.146563+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:49:47.737859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.145701+0000","last_change":"2024-03-23T11:26:25.173255+0000","last_active":"2024-03-23T11:26:26.145701+0000","last_peered":"2024-03-23T11:26:26.145701+0000","last_clean":"2024-03-23T11:26:26.145701+0000","last_became_active":"2024-03-23T11:26:25.172181+0000","last_became_peered":"2024-03-23T11:26:25.172181+0000","last_unstale":"2024-03-23T11:26:26.145701+0000","last_undegraded":"2024-03-23T11:26:26.145701+0000","last_fullsized":"2024-03-23T11:26:26.145701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:52:38.435785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.490031+0000","last_change":"2024-03-23T11:26:24.672211+0000","last_active":"2024-03-23T11:26:26.490031+0000","last_peered":"2024-03-23T11:26:26.490031+0000","last_clean":"2024-03-23T11:26:26.490031+0000","last_became_active":"2024-03-23T11:26:24.671848+0000","last_became_peered":"2024-03-23T11:26:24.671848+0000","last_unstale":"2024-03-23T11:26:26.490031+0000","last_undegraded":"2024-03-23T11:26:26.490031+0000","last_fullsized":"2024-03-23T11:26:26.490031+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:11:54.456971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.489763+0000","last_change":"2024-03-23T11:26:25.178859+0000","last_active":"2024-03-23T11:26:26.489763+0000","last_peered":"2024-03-23T11:26:26.489763+0000","last_clean":"2024-03-23T11:26:26.489763+0000","last_became_active":"2024-03-23T11:26:25.178584+0000","last_became_peered":"2024-03-23T11:26:25.178584+0000","last_unstale":"2024-03-23T11:26:26.489763+0000","last_undegraded":"2024-03-23T11:26:26.489763+0000","last_fullsized":"2024-03-23T11:26:26.489763+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:09:31.199645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.146202+0000","last_change":"2024-03-23T11:26:24.669960+0000","last_active":"2024-03-23T11:26:26.146202+0000","last_peered":"2024-03-23T11:26:26.146202+0000","last_clean":"2024-03-23T11:26:26.146202+0000","last_became_active":"2024-03-23T11:26:24.669208+0000","last_became_peered":"2024-03-23T11:26:24.669208+0000","last_unstale":"2024-03-23T11:26:26.146202+0000","last_undegraded":"2024-03-23T11:26:26.146202+0000","last_fullsized":"2024-03-23T11:26:26.146202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:14:46.322563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.488188+0000","last_change":"2024-03-23T11:26:25.172071+0000","last_active":"2024-03-23T11:26:26.488188+0000","last_peered":"2024-03-23T11:26:26.488188+0000","last_clean":"2024-03-23T11:26:26.488188+0000","last_became_active":"2024-03-23T11:26:25.171820+0000","last_became_peered":"2024-03-23T11:26:25.171820+0000","last_unstale":"2024-03-23T11:26:26.488188+0000","last_undegraded":"2024-03-23T11:26:26.488188+0000","last_fullsized":"2024-03-23T11:26:26.488188+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:24.623098+0000","last_clean_scrub_stamp":"2024-03-23T11:26:24.623098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:39:40.831693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-23T11:26:26.148051+0000","last_change":"2024-03-23T11:26:24.675232+0000","last_active":"2024-03-23T11:26:26.148051+0000","last_peered":"2024-03-23T11:26:26.148051+0000","last_clean":"2024-03-23T11:26:26.148051+0000","last_became_active":"2024-03-23T11:26:24.674490+0000","last_became_peered":"2024-03-23T11:26:24.674490+0000","last_unstale":"2024-03-23T11:26:26.148051+0000","last_undegraded":"2024-03-23T11:26:26.148051+0000","last_fullsized":"2024-03-23T11:26:26.148051+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:26:23.609680+0000","last_clean_scrub_stamp":"2024-03-23T11:26:23.609680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:37:35.845166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":234,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-23T11:26:25.147211+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:26:25.147211+0000","last_peered":"2024-03-23T11:26:25.147211+0000","last_clean":"2024-03-23T11:26:25.147211+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:26:25.147211+0000","last_undegraded":"2024-03-23T11:26:25.147211+0000","last_fullsized":"2024-03-23T11:26:25.147211+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542239,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32772,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1687552,"data_stored":1055494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0469999999999999,"15min":0.94699999999999995},"min":{"1min":0.76400000000000001,"5min":0.67600000000000005,"15min":0.55900000000000005},"max":{"1min":1.423,"5min":1.5309999999999999,"15min":1.7470000000000001},"last":2.8370000000000002},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.0680000000000001,"15min":1.0049999999999999},"min":{"1min":0.77800000000000002,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.599,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.38300000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.173,"15min":1.1739999999999999},"min":{"1min":0.88400000000000001,"5min":0.71599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6399999999999999,"5min":1.7430000000000001,"15min":2.3820000000000001},"last":3.569},{"interface":"front","average":{"1min":1.381,"5min":1.2649999999999999,"15min":1.252},"min":{"1min":1.1180000000000001,"5min":0.77300000000000002,"15min":0.753},"max":{"1min":1.736,"5min":1.736,"15min":2.4350000000000001},"last":0.98599999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.0980000000000001,"15min":1.0629999999999999},"min":{"1min":0.93200000000000005,"5min":0.67500000000000004,"15min":0.66200000000000003},"max":{"1min":1.5549999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":2.7440000000000002},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.218,"15min":1.155},"min":{"1min":0.84499999999999997,"5min":0.81200000000000006,"15min":0.74299999999999999},"max":{"1min":1.675,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":3.524}]},{"osd":3,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2250000000000001,"15min":1.151},"min":{"1min":0.88100000000000001,"5min":0.76000000000000001,"15min":0.72799999999999998},"max":{"1min":1.6799999999999999,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":2.6389999999999998},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2629999999999999,"15min":1.254},"min":{"1min":1.0549999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.5940000000000001,"5min":1.6479999999999999,"15min":1.778},"last":2.9470000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.2350000000000001,"15min":1.2210000000000001},"min":{"1min":0.84699999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.8799999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4239999999999999,"15min":1.3720000000000001},"min":{"1min":1.1870000000000001,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.6990000000000001,"5min":1.845,"15min":1.845},"last":2.5569999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.357,"15min":1.335},"min":{"1min":0.95399999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.8049999999999999,"5min":1.847,"15min":2.0590000000000002},"last":2.8999999999999999},{"interface":"front","average":{"1min":1.552,"5min":1.4650000000000001,"15min":1.381},"min":{"1min":1.3340000000000001,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":2.0859999999999999},"last":0.90000000000000002}]},{"osd":6,"last update":"Sat Mar 23 11:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.444,"15min":1.4239999999999999},"min":{"1min":1.2410000000000001,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9890000000000001},"last":1.026},{"interface":"front","average":{"1min":1.403,"5min":1.383,"15min":1.349},"min":{"1min":0.94299999999999995,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7949999999999999,"5min":1.887,"15min":1.887},"last":3.218}]}]},{"osd":6,"up_from":14,"seq":60129542239,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1097728,"data_stored":465081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.143,"15min":1.097},"min":{"1min":0.82299999999999995,"5min":0.63600000000000001,"15min":0.56899999999999995},"max":{"1min":1.651,"5min":1.8169999999999999,"15min":3.9140000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.1319999999999999,"15min":1.1240000000000001},"min":{"1min":0.94399999999999995,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6000000000000001,"5min":1.774,"15min":6.6539999999999999},"last":0.96699999999999997}]},{"osd":1,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.226,"15min":1.2190000000000001},"min":{"1min":0.88800000000000001,"5min":0.78000000000000003,"15min":0.71499999999999997},"max":{"1min":1.5269999999999999,"5min":1.7410000000000001,"15min":1.9430000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.254,"5min":1.196,"15min":1.2470000000000001},"min":{"1min":0.90900000000000003,"5min":0.76200000000000001,"15min":0.70399999999999996},"max":{"1min":1.752,"5min":1.752,"15min":2.0859999999999999},"last":1.171}]},{"osd":2,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.212,"15min":1.244},"min":{"1min":0.873,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.5089999999999999,"5min":1.6419999999999999,"15min":3.73},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.2749999999999999,"15min":1.2909999999999999},"min":{"1min":0.90600000000000003,"5min":0.72899999999999998,"15min":0.68100000000000005},"max":{"1min":1.712,"5min":1.8340000000000001,"15min":2.2280000000000002},"last":0.90700000000000003}]},{"osd":3,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.2649999999999999,"15min":1.27},"min":{"1min":0.878,"5min":0.83099999999999996,"15min":0.749},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":12.914},"last":1.05},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.302,"15min":1.361},"min":{"1min":1.095,"5min":0.86599999999999999,"15min":0.72699999999999998},"max":{"1min":1.6899999999999999,"5min":1.7350000000000001,"15min":3.8100000000000001},"last":0.84499999999999997}]},{"osd":4,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.28,"15min":1.371},"min":{"1min":0.82599999999999996,"5min":0.79600000000000004,"15min":0.51100000000000001},"max":{"1min":1.603,"5min":2.2200000000000002,"15min":2.702},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3100000000000001,"15min":1.4039999999999999},"min":{"1min":0.96299999999999997,"5min":0.85399999999999998,"15min":0.82299999999999995},"max":{"1min":1.944,"5min":1.944,"15min":7.2140000000000004},"last":1.26}]},{"osd":5,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4179999999999999,"15min":1.522},"min":{"1min":1.008,"5min":0.93600000000000005,"15min":0.73899999999999999},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":3.097},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.468,"5min":1.4470000000000001,"15min":1.4930000000000001},"min":{"1min":1.238,"5min":0.90800000000000003,"15min":0.72499999999999998},"max":{"1min":1.7370000000000001,"5min":2.282,"15min":2.8109999999999999},"last":1.286}]},{"osd":7,"last update":"Sat Mar 23 11:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.524,"15min":1.651},"min":{"1min":1.2190000000000001,"5min":1.127,"15min":0.80400000000000005},"max":{"1min":2.028,"5min":2.028,"15min":4.9470000000000001},"last":1.395},{"interface":"front","average":{"1min":1.397,"5min":1.393,"15min":1.4910000000000001},"min":{"1min":1.075,"5min":1.014,"15min":0.57699999999999996},"max":{"1min":1.7250000000000001,"5min":1.903,"15min":3.7749999999999999},"last":1.238}]}]},{"osd":5,"up_from":14,"seq":60129542239,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1105920,"data_stored":466642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.0620000000000001,"15min":1.165},"min":{"1min":0.76300000000000001,"5min":0.60399999999999998,"15min":0.55400000000000005},"max":{"1min":1.5009999999999999,"5min":1.9350000000000001,"15min":2.2999999999999998},"last":1.149},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1319999999999999,"15min":1.1499999999999999},"min":{"1min":0.72599999999999998,"5min":0.61499999999999999,"15min":0.47799999999999998},"max":{"1min":1.524,"5min":1.643,"15min":2.355},"last":0.81899999999999995}]},{"osd":1,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.119,"15min":1.1759999999999999},"min":{"1min":0.91000000000000003,"5min":0.66800000000000004,"15min":0.65300000000000002},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":2.2559999999999998},"last":1.206},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1419999999999999,"15min":1.2050000000000001},"min":{"1min":0.86599999999999999,"5min":0.66900000000000004,"15min":0.495},"max":{"1min":1.5429999999999999,"5min":1.841,"15min":1.8580000000000001},"last":1.45}]},{"osd":2,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2010000000000001,"15min":1.2490000000000001},"min":{"1min":0.879,"5min":0.71399999999999997,"15min":0.60199999999999998},"max":{"1min":1.6000000000000001,"5min":1.7390000000000001,"15min":2.012},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2290000000000001,"15min":1.2370000000000001},"min":{"1min":0.93600000000000005,"5min":0.72399999999999998,"15min":0.64400000000000002},"max":{"1min":1.4570000000000001,"5min":1.7649999999999999,"15min":1.946},"last":0.98399999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.2889999999999999,"15min":1.304},"min":{"1min":1.0069999999999999,"5min":0.76600000000000001,"15min":0.66800000000000004},"max":{"1min":1.6279999999999999,"5min":1.835,"15min":2.1539999999999999},"last":1.244},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3500000000000001,"15min":1.3959999999999999},"min":{"1min":1.016,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.577,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.296}]},{"osd":4,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2769999999999999,"15min":1.3899999999999999},"min":{"1min":0.83999999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.53,"5min":2.0230000000000001,"15min":2.2090000000000001},"last":1.337},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4530000000000001,"15min":1.5229999999999999},"min":{"1min":1.218,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.633,"5min":2.1720000000000002,"15min":2.5009999999999999},"last":1.3700000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.331,"15min":1.4510000000000001},"min":{"1min":0.85699999999999998,"5min":0.59199999999999997,"15min":0.48899999999999999},"max":{"1min":1.6060000000000001,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.339,"15min":1.494},"min":{"1min":0.92700000000000005,"5min":0.624,"15min":0.624},"max":{"1min":1.5980000000000001,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.55}]},{"osd":7,"last update":"Sat Mar 23 11:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.4059999999999999,"15min":1.5309999999999999},"min":{"1min":1.0249999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.6970000000000001,"5min":2.4449999999999998,"15min":2.7229999999999999},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.5049999999999999,"15min":1.595},"min":{"1min":1.1140000000000001,"5min":0.69099999999999995,"15min":0.60099999999999998},"max":{"1min":1.6850000000000001,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":1.103}]}]},{"osd":4,"up_from":14,"seq":60129542239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1089536,"data_stored":465056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.181,"15min":1.121},"min":{"1min":0.85499999999999998,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.613,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.391},{"interface":"front","average":{"1min":1.333,"5min":1.3,"15min":1.2310000000000001},"min":{"1min":1.0940000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.633,"5min":1.679,"15min":1.8009999999999999},"last":1.589}]},{"osd":1,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.2330000000000001,"15min":1.1379999999999999},"min":{"1min":0.86399999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6990000000000001,"5min":4.327,"15min":4.327},"last":1.45},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1699999999999999,"15min":1.119},"min":{"1min":0.83699999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.5149999999999999,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":0.98299999999999998}]},{"osd":2,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2130000000000001,"15min":1.2230000000000001},"min":{"1min":0.997,"5min":0.624,"15min":0.624},"max":{"1min":1.4750000000000001,"5min":2.6080000000000001,"15min":2.6080000000000001},"last":1.417},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.244},"min":{"1min":0.89500000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.927,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":1.321}]},{"osd":3,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2330000000000001,"15min":1.222},"min":{"1min":0.872,"5min":0.622,"15min":0.622},"max":{"1min":1.518,"5min":2.5579999999999998,"15min":2.5579999999999998},"last":1.266},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3779999999999999,"15min":1.2809999999999999},"min":{"1min":0.93799999999999994,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.665,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":1.5049999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.2130000000000001,"15min":1.2709999999999999},"min":{"1min":0.88800000000000001,"5min":0.77500000000000002,"15min":0.74299999999999999},"max":{"1min":1.5740000000000001,"5min":2.508,"15min":2.508},"last":1.153},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.411,"15min":1.367},"min":{"1min":1.0660000000000001,"5min":0.79400000000000004,"15min":0.60499999999999998},"max":{"1min":1.843,"5min":5.859,"15min":5.859},"last":1.2050000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.427,"15min":1.393},"min":{"1min":1.27,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.607,"5min":5.3339999999999996,"15min":5.3339999999999996},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.514,"15min":1.4910000000000001},"min":{"1min":1.21,"5min":0.76300000000000001,"15min":0.72599999999999998},"max":{"1min":1.752,"5min":5.3940000000000001,"15min":5.3940000000000001},"last":1.375}]},{"osd":7,"last update":"Sat Mar 23 11:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.504,"15min":1.48},"min":{"1min":1.2210000000000001,"5min":0.93200000000000005,"15min":0.754},"max":{"1min":1.6870000000000001,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":1.528},{"interface":"front","average":{"1min":1.345,"5min":1.369,"15min":1.403},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.6799999999999999,"5min":2.641,"15min":2.641},"last":1.468}]}]},{"osd":3,"up_from":14,"seq":60129542239,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1101824,"data_stored":466620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.94799999999999995,"15min":0.95199999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.434},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.115,"15min":1.1060000000000001},"min":{"1min":0.503,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.3360000000000001,"5min":1.8879999999999999,"15min":2.0790000000000002},"last":0.55200000000000005}]},{"osd":1,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0089999999999999,"15min":1.004},"min":{"1min":0.499,"5min":0.499,"15min":0.442},"max":{"1min":1.8049999999999999,"5min":1.8180000000000001,"15min":1.976},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.274,"15min":1.266},"min":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.5},"max":{"1min":1.7589999999999999,"5min":1.8360000000000001,"15min":2.2309999999999999},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.105,"15min":1.141},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.53300000000000003},"max":{"1min":1.3420000000000001,"5min":2.0779999999999998,"15min":3.3109999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.1859999999999999,"15min":1.246},"min":{"1min":0.69799999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.9319999999999999,"5min":2.2440000000000002,"15min":2.7109999999999999},"last":0.51900000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2589999999999999,"15min":1.2869999999999999},"min":{"1min":0.90500000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.897,"5min":1.897,"15min":2.3490000000000002},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.452,"15min":1.516},"min":{"1min":1.095,"5min":0.998,"15min":0.998},"max":{"1min":1.8009999999999999,"5min":1.966,"15min":4.8970000000000002},"last":0.68400000000000005}]},{"osd":5,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.48,"15min":1.5209999999999999},"min":{"1min":1.054,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":1.9850000000000001,"5min":2.1520000000000001,"15min":3.4910000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.407,"15min":1.4610000000000001},"min":{"1min":1.0209999999999999,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":3.125},"last":1.1359999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.375,"15min":1.4990000000000001},"min":{"1min":0.89300000000000002,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.667,"5min":2.024,"15min":4.5499999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.546,"5min":1.504,"15min":1.5549999999999999},"min":{"1min":0.94699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.8450000000000002},"last":0.85099999999999998}]},{"osd":7,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.4419999999999999,"15min":1.5109999999999999},"min":{"1min":0.92300000000000004,"5min":0.88800000000000001,"15min":0.877},"max":{"1min":1.8100000000000001,"5min":1.9279999999999999,"15min":2.952},"last":1.105},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.498,"15min":1.6080000000000001},"min":{"1min":1.05,"5min":0.91000000000000003,"15min":0.90900000000000003},"max":{"1min":1.8520000000000001,"5min":2,"15min":4.798},"last":1.181}]}]},{"osd":2,"up_from":14,"seq":60129542239,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1093632,"data_stored":465638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.93300000000000005,"15min":0.90800000000000003},"min":{"1min":0.72399999999999998,"5min":0.39400000000000002,"15min":0.318},"max":{"1min":1.3779999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.1259999999999999,"15min":1.022},"min":{"1min":0.92300000000000004,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.7310000000000001,"5min":1.7709999999999999,"15min":1.8220000000000001},"last":1.198}]},{"osd":1,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.88300000000000001,"15min":0.94299999999999995},"min":{"1min":0.76600000000000001,"5min":0.41499999999999998,"15min":0.34999999999999998},"max":{"1min":1.2689999999999999,"5min":1.429,"15min":1.9430000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.097,"5min":1.0489999999999999,"15min":1},"min":{"1min":0.79400000000000004,"5min":0.38400000000000001,"15min":0.35799999999999998},"max":{"1min":1.4870000000000001,"5min":1.542,"15min":1.7230000000000001},"last":1.075}]},{"osd":3,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.129,"15min":1.1200000000000001},"min":{"1min":0.878,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.905,"5min":1.905,"15min":1.98},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.0649999999999999,"15min":1.083},"min":{"1min":0.86299999999999999,"5min":0.47399999999999998,"15min":0.46200000000000002},"max":{"1min":1.52,"5min":2.0579999999999998,"15min":2.1560000000000001},"last":1.302}]},{"osd":4,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.373,"15min":1.3560000000000001},"min":{"1min":1.147,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.772,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.7},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4550000000000001,"15min":1.3779999999999999},"min":{"1min":1.1180000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.732,"5min":2.085,"15min":2.085},"last":1.5609999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.345,"15min":1.403},"min":{"1min":1.2609999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.6619999999999999,"5min":2.2589999999999999,"15min":2.5830000000000002},"last":1.254},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.393,"15min":1.379},"min":{"1min":1.2270000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.788,"5min":1.9119999999999999,"15min":2.6469999999999998},"last":1.452}]},{"osd":6,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.385,"15min":1.375},"min":{"1min":1.1859999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.875,"5min":1.9370000000000001,"15min":2.758},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.4199999999999999,"15min":1.4970000000000001},"min":{"1min":1.113,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.7090000000000001,"5min":2.0310000000000001,"15min":2.8580000000000001},"last":1.5169999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.4450000000000001,"15min":1.411},"min":{"1min":1.268,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.952,"5min":2.2069999999999999,"15min":2.246},"last":1.611},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.478,"15min":1.5700000000000001},"min":{"1min":1.1890000000000001,"5min":0.89200000000000002,"15min":0.75600000000000001},"max":{"1min":1.839,"5min":1.9710000000000001,"15min":2.7120000000000002},"last":1.649}]}]},{"osd":1,"up_from":14,"seq":60129542240,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32192,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339648,"statfs":{"total":96636764160,"available":96603799552,"internally_reserved":0,"allocated":1097728,"data_stored":464581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.91100000000000003,"15min":0.878},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.534,"5min":1.6459999999999999,"15min":3.2959999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0109999999999999,"15min":0.99099999999999999},"min":{"1min":0.71399999999999997,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5780000000000001,"5min":1.7070000000000001,"15min":1.99},"last":0.64500000000000002}]},{"osd":2,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.94299999999999995,"15min":0.96799999999999997},"min":{"1min":0.65800000000000003,"5min":0.505,"15min":0.40699999999999997},"max":{"1min":1.224,"5min":1.6479999999999999,"15min":1.738},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.0860000000000001,"15min":1.113},"min":{"1min":0.60299999999999998,"5min":0.55100000000000005,"15min":0.49399999999999999},"max":{"1min":1.536,"5min":1.8720000000000001,"15min":2.2160000000000002},"last":1.083}]},{"osd":3,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.24,"15min":1.224},"min":{"1min":0.69299999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.4930000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1040000000000001,"15min":1.1499999999999999},"min":{"1min":0.626,"5min":0.52000000000000002,"15min":0.434},"max":{"1min":1.5509999999999999,"5min":1.919,"15min":2.4169999999999998},"last":1.0549999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.345,"15min":1.276},"min":{"1min":1.077,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.825,"5min":2.0720000000000001,"15min":2.153},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.276,"15min":1.3220000000000001},"min":{"1min":0.81299999999999994,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.5409999999999999,"5min":1.8240000000000001,"15min":2.758},"last":1.4159999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.337,"15min":1.4099999999999999},"min":{"1min":0.91500000000000004,"5min":0.85299999999999998,"15min":0.81699999999999995},"max":{"1min":1.7230000000000001,"5min":2.0009999999999999,"15min":2.6589999999999998},"last":1.542},{"interface":"front","average":{"1min":1.494,"5min":1.419,"15min":1.45},"min":{"1min":1.038,"5min":0.80400000000000005,"15min":0.72199999999999998},"max":{"1min":1.7529999999999999,"5min":2.165,"15min":3.0329999999999999},"last":1.202}]},{"osd":6,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.413,"15min":1.4299999999999999},"min":{"1min":0.89300000000000002,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.8380000000000001,"5min":2.0800000000000001,"15min":2.2730000000000001},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.403,"5min":1.4259999999999999,"15min":1.5349999999999999},"min":{"1min":0.97799999999999998,"5min":0.88200000000000001,"15min":0.83599999999999997},"max":{"1min":1.595,"5min":2.0270000000000001,"15min":2.9630000000000001},"last":1.3380000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.5049999999999999,"15min":1.5609999999999999},"min":{"1min":0.874,"5min":0.874,"15min":0.82699999999999996},"max":{"1min":1.788,"5min":2.3260000000000001,"15min":3.2749999999999999},"last":1.365},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.4990000000000001,"15min":1.5649999999999999},"min":{"1min":1.012,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.9330000000000001,"5min":2.0840000000000001,"15min":2.7829999999999999},"last":1.3899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542238,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32772,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339068,"statfs":{"total":96636764160,"available":96603205632,"internally_reserved":0,"allocated":1683456,"data_stored":1054789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.876,"15min":0.86299999999999999},"min":{"1min":0.68400000000000005,"5min":0.376,"15min":0.376},"max":{"1min":1.413,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.99299999999999999,"15min":0.94599999999999995},"min":{"1min":0.71999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.52,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0660000000000001,"15min":1.04},"min":{"1min":0.85799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.361,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0629999999999999,"15min":1.0249999999999999},"min":{"1min":0.85799999999999998,"5min":0.432,"15min":0.432},"max":{"1min":1.429,"5min":2.5760000000000001,"15min":2.5760000000000001},"last":0.66400000000000003}]},{"osd":3,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.1479999999999999,"15min":1.111},"min":{"1min":0.84599999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.8440000000000001,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.349,"15min":1.323},"min":{"1min":1.155,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.7529999999999999,"5min":2.9390000000000001,"15min":2.9390000000000001},"last":0.73099999999999998}]},{"osd":4,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3200000000000001,"15min":1.3620000000000001},"min":{"1min":1.077,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6859999999999999,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.367,"5min":1.302,"15min":1.282},"min":{"1min":1.133,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.7230000000000001,"5min":3.125,"15min":3.125},"last":1.077}]},{"osd":5,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.341,"15min":1.421},"min":{"1min":1.212,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.621,"5min":3.0830000000000002,"15min":3.0830000000000002},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.385,"5min":1.399,"15min":1.5049999999999999},"min":{"1min":1.046,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.6850000000000001,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":1.0249999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4279999999999999,"15min":1.5289999999999999},"min":{"1min":1.2929999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7010000000000001,"5min":3.391,"15min":3.391},"last":1.284},{"interface":"front","average":{"1min":1.393,"5min":1.3859999999999999,"15min":1.4219999999999999},"min":{"1min":1.0369999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.667,"5min":3.153,"15min":3.153},"last":0.81999999999999995}]},{"osd":7,"last update":"Sat Mar 23 11:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.5,"15min":1.54},"min":{"1min":1.298,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.78,"5min":3.5350000000000001,"15min":3.5350000000000001},"last":1.147},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4890000000000001,"15min":1.5069999999999999},"min":{"1min":1.137,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.7589999999999999,"5min":3.3650000000000002,"15min":3.3650000000000002},"last":0.84899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:26:30.246 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T11:26:30.246 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T11:26:30.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.518+0000 7f3d5ec00640 1 Processor -- start 2024-03-23T11:26:30.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.519+0000 7f3d5ec00640 1 -- start start 2024-03-23T11:26:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.520+0000 7f3d5ec00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3d580b4a50 0x7f3d580b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:30.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.520+0000 7f3d5ec00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d580b7d30 con 0x7f3d580d1bd0 2024-03-23T11:26:30.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.520+0000 7f3d5ec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d580b6ed0 con 0x7f3d5815c9b0 2024-03-23T11:26:30.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.520+0000 7f3d5ec00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d580b7120 con 0x7f3d580b4a50 2024-03-23T11:26:30.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.521+0000 7f3d57400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3d580d1bd0 0x7f3d580b4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37326/0 (socket says 172.21.15.62:37326) 2024-03-23T11:26:30.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.521+0000 7f3d57400640 1 -- 172.21.15.62:0/2633268629 learned_addr learned my addr 172.21.15.62:0/2633268629 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:30.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.521+0000 7f3d5cc00640 1 --2- 172.21.15.62:0/2633268629 >> v2:172.21.15.120:3300/0 conn(0x7f3d580b4a50 0x7f3d580b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:30.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.521+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922405696 0 0) 0x7f3d580b7d30 con 0x7f3d580d1bd0 2024-03-23T11:26:30.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.521+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d3c003520 con 0x7f3d580d1bd0 2024-03-23T11:26:30.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3857960026 0 0) 0x7f3d3c003520 con 0x7f3d580d1bd0 2024-03-23T11:26:30.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 >> v2:172.21.15.120:3300/0 conn(0x7f3d580b4a50 msgr2=0x7f3d580b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:30.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 --2- 172.21.15.62:0/2633268629 >> v2:172.21.15.120:3300/0 conn(0x7f3d580b4a50 0x7f3d580b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 >> v1:172.21.15.120:6789/0 conn(0x7f3d5815c9b0 legacy=0x7f3d580b8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d580b7400 con 0x7f3d580d1bd0 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f3d44001670 con 0x7f3d580d1bd0 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 >> v1:172.21.15.62:6789/0 conn(0x7f3d580d1bd0 legacy=0x7f3d580b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 --2- 172.21.15.62:0/2633268629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3c0039e0 0x7f3d3c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 --2- 172.21.15.62:0/2633268629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d3c0047c0 0x7f3d3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 --2- 172.21.15.62:0/2633268629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3c005180 0x7f3d3c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d580b7120 con 0x7f3d3c0047c0 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c00c0b0 con 0x7f3d3c0039e0 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c00c270 con 0x7f3d3c005180 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d57400640 1 --2- 172.21.15.62:0/2633268629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d3c0047c0 0x7f3d3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.522+0000 7f3d57e00640 1 --2- 172.21.15.62:0/2633268629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3c0039e0 0x7f3d3c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.523+0000 7f3d5cc00640 1 --2- 172.21.15.62:0/2633268629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3c005180 0x7f3d3c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:30.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.523+0000 7f3d57400640 1 -- 172.21.15.62:0/2633268629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3c005180 msgr2=0x7f3d3c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:30.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.523+0000 7f3d57400640 1 --2- 172.21.15.62:0/2633268629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3c005180 0x7f3d3c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.523+0000 7f3d57400640 1 -- 172.21.15.62:0/2633268629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3c0039e0 msgr2=0x7f3d3c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:30.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.523+0000 7f3d57400640 1 --2- 172.21.15.62:0/2633268629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3c0039e0 0x7f3d3c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.523+0000 7f3d57400640 1 -- 172.21.15.62:0/2633268629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c003520 con 0x7f3d3c0047c0 2024-03-23T11:26:30.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.523+0000 7f3d5cc00640 1 -- 172.21.15.62:0/2633268629 reap_dead start 2024-03-23T11:26:30.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.524+0000 7f3d57400640 1 --2- 172.21.15.62:0/2633268629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d3c0047c0 0x7f3d3c004c40 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f3d44007fd0 tx=0x7f3d3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed9bf5577ddd65a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:30.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.524+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d44005ad0 con 0x7f3d3c0047c0 2024-03-23T11:26:30.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.524+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d44005c30 con 0x7f3d3c0047c0 2024-03-23T11:26:30.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.524+0000 7f3d56a00640 1 -- 172.21.15.62:0/2633268629 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4402c810 con 0x7f3d3c0047c0 2024-03-23T11:26:30.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.525+0000 7f3d5ec00640 1 -- 172.21.15.62:0/2633268629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d3c0047c0 msgr2=0x7f3d3c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:30.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.525+0000 7f3d5ec00640 1 --2- 172.21.15.62:0/2633268629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d3c0047c0 0x7f3d3c004c40 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f3d44007fd0 tx=0x7f3d3c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.525+0000 7f3d5ec00640 1 -- 172.21.15.62:0/2633268629 shutdown_connections 2024-03-23T11:26:30.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.525+0000 7f3d5ec00640 1 --2- 172.21.15.62:0/2633268629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d3c0047c0 0x7f3d3c004c40 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.525+0000 7f3d5ec00640 1 -- 172.21.15.62:0/2633268629 >> 172.21.15.62:0/2633268629 conn(0x7f3d580717b0 msgr2=0x7f3d58071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:30.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.525+0000 7f3d5ec00640 1 -- 172.21.15.62:0/2633268629 shutdown_connections 2024-03-23T11:26:30.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.525+0000 7f3d5ec00640 1 -- 172.21.15.62:0/2633268629 wait complete. 2024-03-23T11:26:30.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.527+0000 7f3d5ec00640 1 Processor -- start 2024-03-23T11:26:30.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.527+0000 7f3d5ec00640 1 -- start start 2024-03-23T11:26:30.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.528+0000 7f3d5ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3c0047c0 0x7f3d580c8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:30.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.528+0000 7f3d5ec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d58153810 0x7f3d581519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:30.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.528+0000 7f3d5ec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d58153c90 0x7f3d58151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:30.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.528+0000 7f3d5ec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d58152ae0 con 0x7f3d58153c90 2024-03-23T11:26:30.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.528+0000 7f3d5ec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5815fe00 con 0x7f3d58153810 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.528+0000 7f3d5ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d58152d30 con 0x7f3d3c0047c0 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.528+0000 7f3d5cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d58153c90 0x7f3d58151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.528+0000 7f3d5cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d58153c90 0x7f3d58151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54030/0 (socket says 172.21.15.62:54030) 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.528+0000 7f3d5cc00640 1 -- 172.21.15.62:0/1814520162 learned_addr learned my addr 172.21.15.62:0/1814520162 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.529+0000 7f3d57e00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3c0047c0 0x7f3d580c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.529+0000 7f3d57400640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d58153810 0x7f3d581519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.529+0000 7f3d5cc00640 1 -- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3c0047c0 msgr2=0x7f3d580c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.529+0000 7f3d5cc00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3c0047c0 0x7f3d580c8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.529+0000 7f3d5cc00640 1 -- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d58153810 msgr2=0x7f3d581519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.529+0000 7f3d5cc00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d58153810 0x7f3d581519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.529+0000 7f3d5cc00640 1 -- 172.21.15.62:0/1814520162 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d58153010 con 0x7f3d58153c90 2024-03-23T11:26:30.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.529+0000 7f3d5cc00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d58153c90 0x7f3d58151ef0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f3d580ce9c0 tx=0x7f3d50009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:30.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.530+0000 7f3d4fe00640 1 -- 172.21.15.62:0/1814520162 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50004500 con 0x7f3d58153c90 2024-03-23T11:26:30.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.530+0000 7f3d4fe00640 1 -- 172.21.15.62:0/1814520162 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d50003610 con 0x7f3d58153c90 2024-03-23T11:26:30.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.530+0000 7f3d4fe00640 1 -- 172.21.15.62:0/1814520162 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d500039e0 con 0x7f3d58153c90 2024-03-23T11:26:30.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.530+0000 7f3d5ec00640 1 -- 172.21.15.62:0/1814520162 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d581532f0 con 0x7f3d58153c90 2024-03-23T11:26:30.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.530+0000 7f3d5ec00640 1 -- 172.21.15.62:0/1814520162 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d580c8f50 con 0x7f3d58153c90 2024-03-23T11:26:30.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.531+0000 7f3d5ec00640 1 -- 172.21.15.62:0/1814520162 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d58066bb0 con 0x7f3d58153c90 2024-03-23T11:26:30.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.532+0000 7f3d4fe00640 1 -- 172.21.15.62:0/1814520162 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3d50003b40 con 0x7f3d58153c90 2024-03-23T11:26:30.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.533+0000 7f3d4fe00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3d3806a7a0 0x7f3d3806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:30.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.533+0000 7f3d4fe00640 1 -- 172.21.15.62:0/1814520162 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d50089030 con 0x7f3d58153c90 2024-03-23T11:26:30.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.533+0000 7f3d57e00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3d3806a7a0 0x7f3d3806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:30.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.535+0000 7f3d57e00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3d3806a7a0 0x7f3d3806cc60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3d48007fd0 tx=0x7f3d48009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:30.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.538+0000 7f3d4fe00640 1 -- 172.21.15.62:0/1814520162 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3d500544e0 con 0x7f3d58153c90 2024-03-23T11:26:30.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.809+0000 7f3d5ec00640 1 -- 172.21.15.62:0/1814520162 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3d580c8f50 con 0x7f3d58153c90 2024-03-23T11:26:30.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.823+0000 7f3d4fe00640 1 -- 172.21.15.62:0/1814520162 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7f3d580c8f50 con 0x7f3d58153c90 2024-03-23T11:26:30.816 INFO:teuthology.orchestra.run.smithi062.stderr:updated caps for client.0 2024-03-23T11:26:30.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 -- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3d3806a7a0 msgr2=0x7f3d3806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:30.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3d3806a7a0 0x7f3d3806cc60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3d48007fd0 tx=0x7f3d48009040 comp rx=0 tx=0).stop 2024-03-23T11:26:30.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 -- 172.21.15.62:0/1814520162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d58153c90 msgr2=0x7f3d58151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:30.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d58153c90 0x7f3d58151ef0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f3d580ce9c0 tx=0x7f3d50009f90 comp rx=0 tx=0).stop 2024-03-23T11:26:30.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 -- 172.21.15.62:0/1814520162 shutdown_connections 2024-03-23T11:26:30.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3d3806a7a0 0x7f3d3806cc60 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d58153c90 0x7f3d58151ef0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d58153810 0x7f3d581519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 --2- 172.21.15.62:0/1814520162 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d3c0047c0 0x7f3d580c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:30.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 -- 172.21.15.62:0/1814520162 >> 172.21.15.62:0/1814520162 conn(0x7f3d580717b0 msgr2=0x7f3d580b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:30.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 -- 172.21.15.62:0/1814520162 shutdown_connections 2024-03-23T11:26:30.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:30.826+0000 7f3d5ec00640 1 -- 172.21.15.62:0/1814520162 wait complete. 2024-03-23T11:26:30.838 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:26:31.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.065+0000 7fe032e00640 1 Processor -- start 2024-03-23T11:26:31.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.065+0000 7fe032e00640 1 -- start start 2024-03-23T11:26:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.067+0000 7fe032e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe02c160420 0x7fe02c1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.067+0000 7fe032e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c0b5db0 0x7fe02c0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.067+0000 7fe032e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c0ce510 con 0x7fe02c15fcf0 2024-03-23T11:26:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.067+0000 7fe032e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe02c160df0 con 0x7fe02c160420 2024-03-23T11:26:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.067+0000 7fe032e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0cdc70 con 0x7fe02c0b5db0 2024-03-23T11:26:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.067+0000 7fe031a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe02c15fcf0 0x7fe02c0d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37340/0 (socket says 172.21.15.62:37340) 2024-03-23T11:26:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.067+0000 7fe031a00640 1 -- 172.21.15.62:0/3020673411 learned_addr learned my addr 172.21.15.62:0/3020673411 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe032400640 1 --2- 172.21.15.62:0/3020673411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c0b5db0 0x7fe02c0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe031000640 1 --2- 172.21.15.62:0/3020673411 >> v2:172.21.15.120:3300/0 conn(0x7fe02c160420 0x7fe02c1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747345345 0 0) 0x7fe02c0ce510 con 0x7fe02c15fcf0 2024-03-23T11:26:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe014003520 con 0x7fe02c15fcf0 2024-03-23T11:26:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1554476892 0 0) 0x7fe014003520 con 0x7fe02c15fcf0 2024-03-23T11:26:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c0b5db0 msgr2=0x7fe02c0b6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe02be00640 1 --2- 172.21.15.62:0/3020673411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c0b5db0 0x7fe02c0b6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 >> v2:172.21.15.120:3300/0 conn(0x7fe02c160420 msgr2=0x7fe02c1608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe02be00640 1 --2- 172.21.15.62:0/3020673411 >> v2:172.21.15.120:3300/0 conn(0x7fe02c160420 0x7fe02c1608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c0cdf50 con 0x7fe02c15fcf0 2024-03-23T11:26:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fe020001670 con 0x7fe02c15fcf0 2024-03-23T11:26:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.068+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 >> v1:172.21.15.62:6789/0 conn(0x7fe02c15fcf0 legacy=0x7fe02c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe02be00640 1 --2- 172.21.15.62:0/3020673411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe014003970 0x7fe014003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe02be00640 1 --2- 172.21.15.62:0/3020673411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 0x7fe014004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe02be00640 1 --2- 172.21.15.62:0/3020673411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c0b5db0 0x7fe014005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c160df0 con 0x7fe0140047c0 2024-03-23T11:26:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0cdc70 con 0x7fe014003970 2024-03-23T11:26:31.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01400b980 con 0x7fe02c0b5db0 2024-03-23T11:26:31.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe031000640 1 --2- 172.21.15.62:0/3020673411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 0x7fe014004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe031000640 1 -- 172.21.15.62:0/3020673411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c0b5db0 msgr2=0x7fe014005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:31.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe031000640 1 --2- 172.21.15.62:0/3020673411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c0b5db0 0x7fe014005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe031000640 1 -- 172.21.15.62:0/3020673411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe014003970 msgr2=0x7fe014003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:31.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe031000640 1 --2- 172.21.15.62:0/3020673411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe014003970 0x7fe014003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.069+0000 7fe031000640 1 -- 172.21.15.62:0/3020673411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe014003520 con 0x7fe0140047c0 2024-03-23T11:26:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.070+0000 7fe031000640 1 --2- 172.21.15.62:0/3020673411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 0x7fe014004c40 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fe018009af0 tx=0x7fe0140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba2ed46dc85501e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.071+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01802c710 con 0x7fe0140047c0 2024-03-23T11:26:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.071+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01802c870 con 0x7fe0140047c0 2024-03-23T11:26:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.071+0000 7fe02be00640 1 -- 172.21.15.62:0/3020673411 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01802cc00 con 0x7fe0140047c0 2024-03-23T11:26:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.071+0000 7fe032e00640 1 -- 172.21.15.62:0/3020673411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 msgr2=0x7fe014004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.071+0000 7fe032e00640 1 --2- 172.21.15.62:0/3020673411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 0x7fe014004c40 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fe018009af0 tx=0x7fe0140057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.071+0000 7fe032400640 1 -- 172.21.15.62:0/3020673411 reap_dead start 2024-03-23T11:26:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.071+0000 7fe032e00640 1 -- 172.21.15.62:0/3020673411 shutdown_connections 2024-03-23T11:26:31.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.071+0000 7fe032e00640 1 -- 172.21.15.62:0/3020673411 >> 172.21.15.62:0/3020673411 conn(0x7fe02c089f60 msgr2=0x7fe02c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:31.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.071+0000 7fe032e00640 1 -- 172.21.15.62:0/3020673411 shutdown_connections 2024-03-23T11:26:31.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.071+0000 7fe032e00640 1 -- 172.21.15.62:0/3020673411 wait complete. 2024-03-23T11:26:31.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.073+0000 7fe032e00640 1 Processor -- start 2024-03-23T11:26:31.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.073+0000 7fe032e00640 1 -- start start 2024-03-23T11:26:31.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.074+0000 7fe032e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 0x7fe02c0ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.074+0000 7fe032e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c15b380 0x7fe02c0cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.074+0000 7fe032e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe02c1595c0 0x7fe02c159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.074+0000 7fe032e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c159fb0 con 0x7fe0140047c0 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.074+0000 7fe032e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c15b910 con 0x7fe02c1595c0 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.074+0000 7fe032e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c15bbf0 con 0x7fe02c15b380 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe031a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 0x7fe02c0ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe031a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 0x7fe02c0ca9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54046/0 (socket says 172.21.15.62:54046) 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe031a00640 1 -- 172.21.15.62:0/2816114390 learned_addr learned my addr 172.21.15.62:0/2816114390 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe031000640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c15b380 0x7fe02c0cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe032400640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe02c1595c0 0x7fe02c159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe031a00640 1 -- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c15b380 msgr2=0x7fe02c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe031a00640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c15b380 0x7fe02c0cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe031a00640 1 -- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe02c1595c0 msgr2=0x7fe02c159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe031a00640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe02c1595c0 0x7fe02c159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe031a00640 1 -- 172.21.15.62:0/2816114390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c15bed0 con 0x7fe0140047c0 2024-03-23T11:26:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe031000640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c15b380 0x7fe02c0cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.075+0000 7fe031a00640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 0x7fe02c0ca9e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fe02000c7f0 tx=0x7fe02000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.076+0000 7fe029600640 1 -- 172.21.15.62:0/2816114390 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0200074e0 con 0x7fe0140047c0 2024-03-23T11:26:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.076+0000 7fe029600640 1 -- 172.21.15.62:0/2816114390 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe020007640 con 0x7fe0140047c0 2024-03-23T11:26:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.076+0000 7fe032e00640 1 -- 172.21.15.62:0/2816114390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c15c1b0 con 0x7fe0140047c0 2024-03-23T11:26:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.076+0000 7fe029600640 1 -- 172.21.15.62:0/2816114390 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe020004a20 con 0x7fe0140047c0 2024-03-23T11:26:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.076+0000 7fe032e00640 1 -- 172.21.15.62:0/2816114390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c0caf20 con 0x7fe0140047c0 2024-03-23T11:26:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.077+0000 7fe032e00640 1 -- 172.21.15.62:0/2816114390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02c066bb0 con 0x7fe0140047c0 2024-03-23T11:26:31.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.078+0000 7fe029600640 1 -- 172.21.15.62:0/2816114390 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe020004b80 con 0x7fe0140047c0 2024-03-23T11:26:31.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.079+0000 7fe029600640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 0x7fe00c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.079+0000 7fe029600640 1 -- 172.21.15.62:0/2816114390 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe020088310 con 0x7fe0140047c0 2024-03-23T11:26:31.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.079+0000 7fe031000640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 0x7fe00c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.081+0000 7fe031000640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 0x7fe00c06cc60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe018036000 tx=0x7fe018030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:31.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.083+0000 7fe029600640 1 -- 172.21.15.62:0/2816114390 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe0200536a0 con 0x7fe0140047c0 2024-03-23T11:26:31.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.322+0000 7fe032e00640 1 -- 172.21.15.62:0/2816114390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe02c0cea40 con 0x7fe0140047c0 2024-03-23T11:26:31.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.323+0000 7fe029600640 1 -- 172.21.15.62:0/2816114390 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fe020056cc0 con 0x7fe0140047c0 2024-03-23T11:26:31.316 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:31.316 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":57,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:26:28.303857+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6154},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6154":{"gid":6154,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/3056962984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3056962984},{"type":"v1","addr":"172.21.15.62:6833","nonce":3056962984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:26:31.317 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 57 2024-03-23T11:26:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.325+0000 7fe032e00640 1 -- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 msgr2=0x7fe00c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.325+0000 7fe032e00640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 0x7fe00c06cc60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe018036000 tx=0x7fe018030000 comp rx=0 tx=0).stop 2024-03-23T11:26:31.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.325+0000 7fe032e00640 1 -- 172.21.15.62:0/2816114390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 msgr2=0x7fe02c0ca9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.325+0000 7fe032e00640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 0x7fe02c0ca9e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fe02000c7f0 tx=0x7fe02000ccb0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.326+0000 7fe032e00640 1 -- 172.21.15.62:0/2816114390 shutdown_connections 2024-03-23T11:26:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.326+0000 7fe032e00640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe00c06a7a0 0x7fe00c06cc60 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.326+0000 7fe032e00640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe02c1595c0 0x7fe02c159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.326+0000 7fe032e00640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe02c15b380 0x7fe02c0cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.326+0000 7fe032e00640 1 --2- 172.21.15.62:0/2816114390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0140047c0 0x7fe02c0ca9e0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.326+0000 7fe032e00640 1 -- 172.21.15.62:0/2816114390 >> 172.21.15.62:0/2816114390 conn(0x7fe02c089f60 msgr2=0x7fe02c0b7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.326+0000 7fe032e00640 1 -- 172.21.15.62:0/2816114390 shutdown_connections 2024-03-23T11:26:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.326+0000 7fe032e00640 1 -- 172.21.15.62:0/2816114390 wait complete. 2024-03-23T11:26:31.740 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:26:25.131291+0000', 'modified': '2024-03-23T11:26:28.303857+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6154}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6154': {'gid': 6154, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.62:6833/3056962984', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 3056962984}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 3056962984}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:26:31.741 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:26:31.741 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T11:26:31.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.979+0000 7fe414200640 1 Processor -- start 2024-03-23T11:26:31.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.979+0000 7fe414200640 1 -- start start 2024-03-23T11:26:31.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.981+0000 7fe414200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe40c15e120 0x7fe40c099910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.981+0000 7fe414200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe40c0b0970 0x7fe40c15e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.981+0000 7fe414200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40c099480 con 0x7fe40c0cee40 2024-03-23T11:26:31.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.981+0000 7fe414200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40c098bc0 con 0x7fe40c0b0970 2024-03-23T11:26:31.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.981+0000 7fe414200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0cf280 con 0x7fe40c15e120 2024-03-23T11:26:31.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.981+0000 7fe410e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe40c0cee40 0x7fe40c0b0260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37352/0 (socket says 172.21.15.62:37352) 2024-03-23T11:26:31.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.981+0000 7fe410e00640 1 -- 172.21.15.62:0/3008390031 learned_addr learned my addr 172.21.15.62:0/3008390031 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:31.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.981+0000 7fe412200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe40c0b0970 0x7fe40c15e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.982+0000 7fe411800640 1 --2- 172.21.15.62:0/3008390031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe40c15e120 0x7fe40c099910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.982+0000 7fe40be00640 1 -- 172.21.15.62:0/3008390031 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162186671 0 0) 0x7fe40c099480 con 0x7fe40c0cee40 2024-03-23T11:26:31.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.982+0000 7fe412200640 1 -- 172.21.15.62:0/3008390031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe40c15e120 msgr2=0x7fe40c099910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:31.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.982+0000 7fe412200640 1 --2- 172.21.15.62:0/3008390031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe40c15e120 0x7fe40c099910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.982+0000 7fe412200640 1 -- 172.21.15.62:0/3008390031 >> v1:172.21.15.62:6789/0 conn(0x7fe40c0cee40 legacy=0x7fe40c0b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.982+0000 7fe412200640 1 -- 172.21.15.62:0/3008390031 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c098e30 con 0x7fe40c0b0970 2024-03-23T11:26:31.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.983+0000 7fe412200640 1 --2- 172.21.15.62:0/3008390031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c0b0970 0x7fe40c15e600 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fe404009d90 tx=0x7fe40402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c180dfb161bb9323 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:31.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.983+0000 7fe40be00640 1 -- 172.21.15.62:0/3008390031 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40402cc50 con 0x7fe40c0b0970 2024-03-23T11:26:31.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.984+0000 7fe40be00640 1 -- 172.21.15.62:0/3008390031 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40402cdb0 con 0x7fe40c0b0970 2024-03-23T11:26:31.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.984+0000 7fe40be00640 1 -- 172.21.15.62:0/3008390031 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe404034660 con 0x7fe40c0b0970 2024-03-23T11:26:31.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.984+0000 7fe414200640 1 -- 172.21.15.62:0/3008390031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c0b0970 msgr2=0x7fe40c15e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.984+0000 7fe414200640 1 --2- 172.21.15.62:0/3008390031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c0b0970 0x7fe40c15e600 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fe404009d90 tx=0x7fe40402bea0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.984+0000 7fe414200640 1 -- 172.21.15.62:0/3008390031 shutdown_connections 2024-03-23T11:26:31.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.984+0000 7fe414200640 1 --2- 172.21.15.62:0/3008390031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c0b0970 0x7fe40c15e600 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.984+0000 7fe414200640 1 --2- 172.21.15.62:0/3008390031 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe40c15e120 0x7fe40c099910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.984+0000 7fe414200640 1 -- 172.21.15.62:0/3008390031 >> 172.21.15.62:0/3008390031 conn(0x7fe40c0b23f0 msgr2=0x7fe40c0b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:31.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.984+0000 7fe414200640 1 -- 172.21.15.62:0/3008390031 shutdown_connections 2024-03-23T11:26:31.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.984+0000 7fe414200640 1 -- 172.21.15.62:0/3008390031 wait complete. 2024-03-23T11:26:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.986+0000 7fe414200640 1 Processor -- start 2024-03-23T11:26:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.986+0000 7fe414200640 1 -- start start 2024-03-23T11:26:31.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.987+0000 7fe414200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe40c0b0970 0x7fe40c06c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.987+0000 7fe414200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe40c0cee40 0x7fe40c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.987+0000 7fe414200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c15e120 0x7fe40c152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.987+0000 7fe414200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c06bc90 con 0x7fe40c15e120 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.987+0000 7fe414200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0c9a10 con 0x7fe40c0b0970 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.987+0000 7fe414200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c1527d0 con 0x7fe40c0cee40 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.987+0000 7fe412200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c15e120 0x7fe40c152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.988+0000 7fe412200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c15e120 0x7fe40c152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54056/0 (socket says 172.21.15.62:54056) 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.988+0000 7fe412200640 1 -- 172.21.15.62:0/227692922 learned_addr learned my addr 172.21.15.62:0/227692922 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.988+0000 7fe411800640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe40c0b0970 0x7fe40c06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.988+0000 7fe410e00640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe40c0cee40 0x7fe40c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.988+0000 7fe412200640 1 -- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe40c0cee40 msgr2=0x7fe40c06b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.988+0000 7fe412200640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe40c0cee40 0x7fe40c06b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.988+0000 7fe412200640 1 -- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe40c0b0970 msgr2=0x7fe40c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.988+0000 7fe412200640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe40c0b0970 0x7fe40c06c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.988+0000 7fe412200640 1 -- 172.21.15.62:0/227692922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe404009a70 con 0x7fe40c15e120 2024-03-23T11:26:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.988+0000 7fe412200640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c15e120 0x7fe40c152290 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fe404009d90 tx=0x7fe404004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.989+0000 7fe409600640 1 -- 172.21.15.62:0/227692922 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40403e610 con 0x7fe40c15e120 2024-03-23T11:26:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.989+0000 7fe409600640 1 -- 172.21.15.62:0/227692922 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40403e790 con 0x7fe40c15e120 2024-03-23T11:26:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.989+0000 7fe409600640 1 -- 172.21.15.62:0/227692922 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40403eb40 con 0x7fe40c15e120 2024-03-23T11:26:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.989+0000 7fe414200640 1 -- 172.21.15.62:0/227692922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c0c90e0 con 0x7fe40c15e120 2024-03-23T11:26:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.989+0000 7fe414200640 1 -- 172.21.15.62:0/227692922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c0c96a0 con 0x7fe40c15e120 2024-03-23T11:26:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.991+0000 7fe409600640 1 -- 172.21.15.62:0/227692922 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe40403b050 con 0x7fe40c15e120 2024-03-23T11:26:31.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.992+0000 7fe409600640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe3f006a7a0 0x7fe3f006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.992+0000 7fe409600640 1 -- 172.21.15.62:0/227692922 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4040ab840 con 0x7fe40c15e120 2024-03-23T11:26:31.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.993+0000 7fe411800640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe3f006a7a0 0x7fe3f006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.993+0000 7fe416f64740 1 Processor -- start 2024-03-23T11:26:31.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.993+0000 7fe416f64740 1 -- start start 2024-03-23T11:26:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe416f64740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf3390 0x559450caa9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe416f64740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450caaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe416f64740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450ca58b0 0x559450cd56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe416f64740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559450c16110 con 0x559450ca9410 2024-03-23T11:26:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe416f64740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559450c138a0 con 0x559450ca58b0 2024-03-23T11:26:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe416f64740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559450c0f190 con 0x559450cf3390 2024-03-23T11:26:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe411800640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe3f006a7a0 0x7fe3f006cc60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc006fd0 tx=0x7fe3fc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe410e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450caaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe410e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450caaf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54064/0 (socket says 172.21.15.62:54064) 2024-03-23T11:26:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe410e00640 1 -- 172.21.15.62:0/106939498 learned_addr learned my addr 172.21.15.62:0/106939498 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe411800640 1 --2- 172.21.15.62:0/106939498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf3390 0x559450caa9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.994+0000 7fe412200640 1 --2- 172.21.15.62:0/106939498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450ca58b0 0x559450cd56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.995+0000 7fe410e00640 1 -- 172.21.15.62:0/106939498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf3390 msgr2=0x559450caa9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.995+0000 7fe410e00640 1 --2- 172.21.15.62:0/106939498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf3390 0x559450caa9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.995+0000 7fe410e00640 1 -- 172.21.15.62:0/106939498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450ca58b0 msgr2=0x559450cd56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.995+0000 7fe410e00640 1 --2- 172.21.15.62:0/106939498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450ca58b0 0x559450cd56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.995+0000 7fe410e00640 1 -- 172.21.15.62:0/106939498 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559450bda230 con 0x559450ca9410 2024-03-23T11:26:31.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.995+0000 7fe410e00640 1 --2- 172.21.15.62:0/106939498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450caaf00 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fe4000011e0 tx=0x7fe400005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb50dad5055e147e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:31.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.995+0000 7fe3fa000640 1 -- 172.21.15.62:0/106939498 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe400005e10 con 0x559450ca9410 2024-03-23T11:26:31.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.995+0000 7fe3fa000640 1 -- 172.21.15.62:0/106939498 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe400001890 con 0x559450ca9410 2024-03-23T11:26:31.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.995+0000 7fe3fa000640 1 -- 172.21.15.62:0/106939498 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe400001c40 con 0x559450ca9410 2024-03-23T11:26:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.996+0000 7fe416f64740 1 -- 172.21.15.62:0/106939498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 msgr2=0x559450caaf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.996+0000 7fe416f64740 1 --2- 172.21.15.62:0/106939498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450caaf00 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fe4000011e0 tx=0x7fe400005bc0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.996+0000 7fe416f64740 1 -- 172.21.15.62:0/106939498 shutdown_connections 2024-03-23T11:26:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.996+0000 7fe416f64740 1 --2- 172.21.15.62:0/106939498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450caaf00 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.996+0000 7fe416f64740 1 --2- 172.21.15.62:0/106939498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450ca58b0 0x559450cd56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.996+0000 7fe416f64740 1 --2- 172.21.15.62:0/106939498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf3390 0x559450caa9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.996+0000 7fe416f64740 1 -- 172.21.15.62:0/106939498 >> 172.21.15.62:0/106939498 conn(0x559450ca7b20 msgr2=0x559450cf06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.996+0000 7fe416f64740 1 -- 172.21.15.62:0/106939498 shutdown_connections 2024-03-23T11:26:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.996+0000 7fe416f64740 1 -- 172.21.15.62:0/106939498 wait complete. 2024-03-23T11:26:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.997+0000 7fe416f64740 1 -- start start 2024-03-23T11:26:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.997+0000 7fe416f64740 1 Processor -- start 2024-03-23T11:26:31.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.998+0000 7fe416f64740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450ccd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.998+0000 7fe416f64740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 0x559450cf5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.998+0000 7fe416f64740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf60e0 0x559450cd2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.998+0000 7fe416f64740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559450bb08c0 con 0x559450ca9410 2024-03-23T11:26:31.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.998+0000 7fe416f64740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559450bb0560 con 0x559450cf3390 2024-03-23T11:26:31.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.998+0000 7fe416f64740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559450976f20 con 0x559450cf60e0 2024-03-23T11:26:31.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.998+0000 7fe416f64740 10 client.0 _ll_register_callbacks cb 0x559450c08800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:26:31.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.998+0000 7fe411800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450ccd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.998+0000 7fe411800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450ccd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54074/0 (socket says 172.21.15.62:54074) 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.998+0000 7fe411800640 1 -- 172.21.15.62:0/88513299 learned_addr learned my addr 172.21.15.62:0/88513299 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.999+0000 7fe410e00640 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 0x559450cf5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.999+0000 7fe412200640 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf60e0 0x559450cd2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.999+0000 7fe411800640 1 -- 172.21.15.62:0/88513299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf60e0 msgr2=0x559450cd2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.999+0000 7fe411800640 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf60e0 0x559450cd2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.999+0000 7fe411800640 1 -- 172.21.15.62:0/88513299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 msgr2=0x559450cf5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.999+0000 7fe411800640 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 0x559450cf5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.999+0000 7fe411800640 1 -- 172.21.15.62:0/88513299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40000f040 con 0x559450ca9410 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.999+0000 7fe410e00640 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 0x559450cf5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.999+0000 7fe411800640 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450ccd9a0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc014740 tx=0x7fe3fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b12347a24419be server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:31.999+0000 7fe3eea00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.000+0000 7fe3eea00640 1 -- 172.21.15.62:0/88513299 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc0158e0 con 0x559450ca9410 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.000+0000 7fe3eea00640 1 -- 172.21.15.62:0/88513299 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3fc015a40 con 0x559450ca9410 2024-03-23T11:26:31.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.000+0000 7fe416f64740 1 -- 172.21.15.62:0/88513299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559450c39280 con 0x559450ca9410 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.000+0000 7fe3eea00640 1 -- 172.21.15.62:0/88513299 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3fc015c00 con 0x559450ca9410 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.000+0000 7fe416f64740 10 client.7473 fetch_fsmap learned FSMap version 57 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.000+0000 7fe416f64740 1 -- 172.21.15.62:0/88513299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x559450c0d810 con 0x559450ca9410 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.000+0000 7fe3eea00640 1 -- 172.21.15.62:0/88513299 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559450c39280 con 0x559450ca9410 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.000+0000 7fe3eea00640 1 -- 172.21.15.62:0/88513299 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 57) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe3fc021420 con 0x559450ca9410 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.001+0000 7fe416f64740 10 client.7473 fetch_fsmap finished waiting for FSMap version 57 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.001+0000 7fe416f64740 10 client.7473 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.001+0000 7fe416f64740 20 client.7473 populate_metadata read hostname 'smithi062' 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.001+0000 7fe416f64740 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.62:6832/3056962984,v1:172.21.15.62:6833/3056962984] conn(0x559450cba9d0 0x559450cbadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.001+0000 7fe416f64740 4 client.7473 mds_command: new command op to 6154 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.001+0000 7fe416f64740 1 -- 172.21.15.62:0/88513299 --> [v2:172.21.15.62:6832/3056962984,v1:172.21.15.62:6833/3056962984] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559450c019d0 con 0x559450cba9d0 2024-03-23T11:26:31.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.001+0000 7fe410e00640 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.62:6832/3056962984,v1:172.21.15.62:6833/3056962984] conn(0x559450cba9d0 0x559450cbadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:31.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.001+0000 7fe410e00640 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.62:6832/3056962984,v1:172.21.15.62:6833/3056962984] conn(0x559450cba9d0 0x559450cbadb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baab164e01e12936 server_cookie=546b9260f301503b in_seq=0 out_seq=0 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.001+0000 7fe3eea00640 10 client.7473 ms_handle_connect on v2:172.21.15.62:6832/3056962984 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.002+0000 7fe3eea00640 1 -- 172.21.15.62:0/88513299 <== mds.0 v2:172.21.15.62:6832/3056962984 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x559450c019d0 con 0x559450cba9d0 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.002+0000 7fe3eea00640 10 client.7473 handle_command_reply: tid=0 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.002+0000 7fe416f64740 1 client.7473 shutdown 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe416f64740 1 -- 172.21.15.62:0/88513299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 msgr2=0x559450ccd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe416f64740 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450ccd9a0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc014740 tx=0x7fe3fc004730 comp rx=0 tx=0).stop 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe416f64740 1 -- 172.21.15.62:0/88513299 shutdown_connections 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe416f64740 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.62:6832/3056962984,v1:172.21.15.62:6833/3056962984] conn(0x559450cba9d0 0x559450cbadb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe416f64740 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf60e0 0x559450cd2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe416f64740 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 0x559450cf5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe416f64740 1 --2- 172.21.15.62:0/88513299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450ccd9a0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:31.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe3eea00640 0 client.7473 ms_handle_reset on v2:172.21.15.62:6832/3056962984 2024-03-23T11:26:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe416f64740 1 -- 172.21.15.62:0/88513299 >> 172.21.15.62:0/88513299 conn(0x559450ca58b0 msgr2=0x559450ca4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe416f64740 1 -- 172.21.15.62:0/88513299 shutdown_connections 2024-03-23T11:26:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe416f64740 1 -- 172.21.15.62:0/88513299 wait complete. 2024-03-23T11:26:31.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.003+0000 7fe416f64740 20 client.7473 trim_cache size 0 max 16384 2024-03-23T11:26:32.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.019+0000 7fe416f64740 1 Processor -- start 2024-03-23T11:26:32.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.019+0000 7fe416f64740 1 -- start start 2024-03-23T11:26:32.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.020+0000 7fe416f64740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450cd39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.020+0000 7fe416f64740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf3390 0x559450cd1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.020+0000 7fe416f64740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf60e0 0x559450cb7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.020+0000 7fe416f64740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559450d13060 con 0x559450ca9410 2024-03-23T11:26:32.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.020+0000 7fe416f64740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559450cce130 con 0x559450cf60e0 2024-03-23T11:26:32.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.020+0000 7fe416f64740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559450cac600 con 0x559450cf3390 2024-03-23T11:26:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe411800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450cd39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe411800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450cd39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54078/0 (socket says 172.21.15.62:54078) 2024-03-23T11:26:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe411800640 1 -- 172.21.15.62:0/3216461869 learned_addr learned my addr 172.21.15.62:0/3216461869 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe410e00640 1 --2- 172.21.15.62:0/3216461869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf3390 0x559450cd1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe412200640 1 --2- 172.21.15.62:0/3216461869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf60e0 0x559450cb7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe411800640 1 -- 172.21.15.62:0/3216461869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf3390 msgr2=0x559450cd1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe411800640 1 --2- 172.21.15.62:0/3216461869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf3390 0x559450cd1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe411800640 1 -- 172.21.15.62:0/3216461869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf60e0 msgr2=0x559450cb7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe411800640 1 --2- 172.21.15.62:0/3216461869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf60e0 0x559450cb7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe411800640 1 -- 172.21.15.62:0/3216461869 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559450bafad0 con 0x559450ca9410 2024-03-23T11:26:32.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe412200640 1 --2- 172.21.15.62:0/3216461869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf60e0 0x559450cb7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.021+0000 7fe411800640 1 --2- 172.21.15.62:0/3216461869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450cd39a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00b680 tx=0x7fe3fc01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c420bc2371bfe66e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.022+0000 7fe3faa00640 1 -- 172.21.15.62:0/3216461869 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc037070 con 0x559450ca9410 2024-03-23T11:26:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.022+0000 7fe3faa00640 1 -- 172.21.15.62:0/3216461869 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc021520 con 0x559450ca9410 2024-03-23T11:26:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.022+0000 7fe3faa00640 1 -- 172.21.15.62:0/3216461869 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc00fcc0 con 0x559450ca9410 2024-03-23T11:26:32.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.022+0000 7fe416f64740 1 -- 172.21.15.62:0/3216461869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 msgr2=0x559450cd39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:32.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.022+0000 7fe416f64740 1 --2- 172.21.15.62:0/3216461869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450cd39a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00b680 tx=0x7fe3fc01b000 comp rx=0 tx=0).stop 2024-03-23T11:26:32.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.022+0000 7fe416f64740 1 -- 172.21.15.62:0/3216461869 shutdown_connections 2024-03-23T11:26:32.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.022+0000 7fe416f64740 1 --2- 172.21.15.62:0/3216461869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf60e0 0x559450cb7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.022+0000 7fe416f64740 1 --2- 172.21.15.62:0/3216461869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450cf3390 0x559450cd1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.022+0000 7fe416f64740 1 --2- 172.21.15.62:0/3216461869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450ca9410 0x559450cd39a0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.022+0000 7fe416f64740 1 -- 172.21.15.62:0/3216461869 >> 172.21.15.62:0/3216461869 conn(0x559450ca58b0 msgr2=0x559450cd43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:32.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.022+0000 7fe416f64740 1 -- 172.21.15.62:0/3216461869 shutdown_connections 2024-03-23T11:26:32.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.023+0000 7fe416f64740 1 -- 172.21.15.62:0/3216461869 wait complete. 2024-03-23T11:26:32.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.023+0000 7fe416f64740 1 -- start start 2024-03-23T11:26:32.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.023+0000 7fe416f64740 1 Processor -- start 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.024+0000 7fe416f64740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450ca9410 0x559450d02a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.024+0000 7fe416f64740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 0x559450d55b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.024+0000 7fe416f64740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450cf60e0 0x559450cee070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.024+0000 7fe416f64740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559450cf6840 con 0x559450cf60e0 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.024+0000 7fe416f64740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559450d57700 con 0x559450cf3390 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.024+0000 7fe416f64740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559450cb7720 con 0x559450ca9410 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.024+0000 7fe416f64740 10 client.0 _ll_register_callbacks cb 0x7fe3fc00bea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe412200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450cf60e0 0x559450cee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe412200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450cf60e0 0x559450cee070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54086/0 (socket says 172.21.15.62:54086) 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe412200640 1 -- 172.21.15.62:0/3263539339 learned_addr learned my addr 172.21.15.62:0/3263539339 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe411800640 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450ca9410 0x559450d02a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe410e00640 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 0x559450d55b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe412200640 1 -- 172.21.15.62:0/3263539339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450ca9410 msgr2=0x559450d02a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe412200640 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450ca9410 0x559450d02a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe412200640 1 -- 172.21.15.62:0/3263539339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 msgr2=0x559450d55b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe412200640 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 0x559450d55b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe412200640 1 -- 172.21.15.62:0/3263539339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559450c0d810 con 0x559450cf60e0 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe411800640 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450ca9410 0x559450d02a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe410e00640 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 0x559450d55b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe412200640 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450cf60e0 0x559450cee070 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe404082860 tx=0x7fe40407a3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4332628734fcdb8c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.025+0000 7fe3eea00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe3eea00640 1 -- 172.21.15.62:0/3263539339 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4040ab040 con 0x559450cf60e0 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe3eea00640 1 -- 172.21.15.62:0/3263539339 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe40407e740 con 0x559450cf60e0 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe416f64740 1 -- 172.21.15.62:0/3263539339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe400001890 con 0x559450cf60e0 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe3eea00640 1 -- 172.21.15.62:0/3263539339 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe40403eea0 con 0x559450cf60e0 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe416f64740 10 client.7479 fetch_fsmap learned FSMap version 57 2024-03-23T11:26:32.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe416f64740 1 -- 172.21.15.62:0/3263539339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x559450bd4fe0 con 0x559450cf60e0 2024-03-23T11:26:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe3eea00640 1 -- 172.21.15.62:0/3263539339 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe400001890 con 0x559450cf60e0 2024-03-23T11:26:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe3eea00640 1 -- 172.21.15.62:0/3263539339 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 57) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe404079e00 con 0x559450cf60e0 2024-03-23T11:26:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe416f64740 10 client.7479 fetch_fsmap finished waiting for FSMap version 57 2024-03-23T11:26:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe416f64740 10 client.7479 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:26:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe416f64740 20 client.7479 populate_metadata read hostname 'smithi062' 2024-03-23T11:26:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe416f64740 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.62:6832/3056962984,v1:172.21.15.62:6833/3056962984] conn(0x559450d56bd0 0x559450caa630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe416f64740 4 client.7479 mds_command: new command op to 6154 tid=0[{"prefix": "status"}] 2024-03-23T11:26:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.026+0000 7fe416f64740 1 -- 172.21.15.62:0/3263539339 --> [v2:172.21.15.62:6832/3056962984,v1:172.21.15.62:6833/3056962984] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5594509033c0 con 0x559450d56bd0 2024-03-23T11:26:32.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.027+0000 7fe411800640 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.62:6832/3056962984,v1:172.21.15.62:6833/3056962984] conn(0x559450d56bd0 0x559450caa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.027+0000 7fe411800640 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.62:6832/3056962984,v1:172.21.15.62:6833/3056962984] conn(0x559450d56bd0 0x559450caa630 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d5e91bc01ea062e server_cookie=da186b7a0a11ef77 in_seq=0 out_seq=0 2024-03-23T11:26:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.027+0000 7fe3eea00640 10 client.7479 ms_handle_connect on v2:172.21.15.62:6832/3056962984 2024-03-23T11:26:32.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.027+0000 7fe3eea00640 1 -- 172.21.15.62:0/3263539339 <== mds.0 v2:172.21.15.62:6832/3056962984 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5594509033c0 con 0x559450d56bd0 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.027+0000 7fe3eea00640 10 client.7479 handle_command_reply: tid=0 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.027+0000 7fe416f64740 1 client.7479 shutdown 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe416f64740 1 -- 172.21.15.62:0/3263539339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450cf60e0 msgr2=0x559450cee070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe416f64740 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450cf60e0 0x559450cee070 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe404082860 tx=0x7fe40407a3a0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe416f64740 1 -- 172.21.15.62:0/3263539339 shutdown_connections 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe416f64740 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.62:6832/3056962984,v1:172.21.15.62:6833/3056962984] conn(0x559450d56bd0 0x559450caa630 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe416f64740 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559450cf60e0 0x559450cee070 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe416f64740 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559450cf3390 0x559450d55b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe416f64740 1 --2- 172.21.15.62:0/3263539339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559450ca9410 0x559450d02a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe416f64740 1 -- 172.21.15.62:0/3263539339 >> 172.21.15.62:0/3263539339 conn(0x559450ca58b0 msgr2=0x559450cf5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe3eea00640 0 client.7479 ms_handle_reset on v2:172.21.15.62:6832/3056962984 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe416f64740 1 -- 172.21.15.62:0/3263539339 shutdown_connections 2024-03-23T11:26:32.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe416f64740 1 -- 172.21.15.62:0/3263539339 wait complete. 2024-03-23T11:26:32.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.028+0000 7fe416f64740 20 client.7479 trim_cache size 0 max 16384 2024-03-23T11:26:32.025 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:26:32.025 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:26:32.025 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:26:32.025 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 6154, 2024-03-23T11:26:32.025 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:26:32.025 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:26:32.025 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:26:32.025 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 6.8896879780000004, 2024-03-23T11:26:32.025 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 57, 2024-03-23T11:26:32.026 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 89, 2024-03-23T11:26:32.026 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 89, 2024-03-23T11:26:32.026 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 12.741676341 2024-03-23T11:26:32.026 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 -- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe3f006a7a0 msgr2=0x7fe3f006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe3f006a7a0 0x7fe3f006cc60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc006fd0 tx=0x7fe3fc008040 comp rx=0 tx=0).stop 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 -- 172.21.15.62:0/227692922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c15e120 msgr2=0x7fe40c152290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c15e120 0x7fe40c152290 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fe404009d90 tx=0x7fe404004a40 comp rx=0 tx=0).stop 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 -- 172.21.15.62:0/227692922 shutdown_connections 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe3f006a7a0 0x7fe3f006cc60 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe40c15e120 0x7fe40c152290 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe40c0cee40 0x7fe40c06b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 --2- 172.21.15.62:0/227692922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe40c0b0970 0x7fe40c06c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 -- 172.21.15.62:0/227692922 >> 172.21.15.62:0/227692922 conn(0x7fe40c0b23f0 msgr2=0x7fe40c0b0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 -- 172.21.15.62:0/227692922 shutdown_connections 2024-03-23T11:26:32.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.029+0000 7fe414200640 1 -- 172.21.15.62:0/227692922 wait complete. 2024-03-23T11:26:32.038 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:26:32.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.256+0000 7fb897200640 1 Processor -- start 2024-03-23T11:26:32.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.256+0000 7fb897200640 1 -- start start 2024-03-23T11:26:32.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.258+0000 7fb897200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb898159490 0x7fb89806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.258+0000 7fb897200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8980ce5f0 0x7fb8980b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.258+0000 7fb897200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8980ced90 con 0x7fb89815da90 2024-03-23T11:26:32.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.258+0000 7fb897200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89806e3e0 con 0x7fb898159490 2024-03-23T11:26:32.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.258+0000 7fb897200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89806c1c0 con 0x7fb8980ce5f0 2024-03-23T11:26:32.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.258+0000 7fb895e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb898159490 0x7fb89806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.258+0000 7fb895400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb89815da90 0x7fb8980cdee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37364/0 (socket says 172.21.15.62:37364) 2024-03-23T11:26:32.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.258+0000 7fb895400640 1 -- 172.21.15.62:0/2802246163 learned_addr learned my addr 172.21.15.62:0/2802246163 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:32.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.259+0000 7fb895e00640 1 -- 172.21.15.62:0/2802246163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8980ce5f0 msgr2=0x7fb8980b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:32.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.258+0000 7fb896800640 1 --2- 172.21.15.62:0/2802246163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8980ce5f0 0x7fb8980b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.259+0000 7fb895e00640 1 --2- 172.21.15.62:0/2802246163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8980ce5f0 0x7fb8980b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.259+0000 7fb895e00640 1 -- 172.21.15.62:0/2802246163 >> v1:172.21.15.62:6789/0 conn(0x7fb89815da90 legacy=0x7fb8980cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:32.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.259+0000 7fb894a00640 1 -- 172.21.15.62:0/2802246163 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334051418 0 0) 0x7fb8980ced90 con 0x7fb89815da90 2024-03-23T11:26:32.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.259+0000 7fb895e00640 1 -- 172.21.15.62:0/2802246163 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89806e100 con 0x7fb898159490 2024-03-23T11:26:32.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.259+0000 7fb896800640 1 --2- 172.21.15.62:0/2802246163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8980ce5f0 0x7fb8980b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:32.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.260+0000 7fb895e00640 1 --2- 172.21.15.62:0/2802246163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb898159490 0x7fb89806cc60 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb8880097a0 tx=0x7fb88802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3019d9cf248121 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:32.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.260+0000 7fb894a00640 1 -- 172.21.15.62:0/2802246163 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888031780 con 0x7fb898159490 2024-03-23T11:26:32.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.260+0000 7fb894a00640 1 -- 172.21.15.62:0/2802246163 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8880318e0 con 0x7fb898159490 2024-03-23T11:26:32.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.260+0000 7fb894a00640 1 -- 172.21.15.62:0/2802246163 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888031c70 con 0x7fb898159490 2024-03-23T11:26:32.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.261+0000 7fb897200640 1 -- 172.21.15.62:0/2802246163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb898159490 msgr2=0x7fb89806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:32.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.261+0000 7fb897200640 1 --2- 172.21.15.62:0/2802246163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb898159490 0x7fb89806cc60 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb8880097a0 tx=0x7fb88802b820 comp rx=0 tx=0).stop 2024-03-23T11:26:32.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.261+0000 7fb897200640 1 -- 172.21.15.62:0/2802246163 shutdown_connections 2024-03-23T11:26:32.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.261+0000 7fb897200640 1 --2- 172.21.15.62:0/2802246163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8980ce5f0 0x7fb8980b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.261+0000 7fb897200640 1 --2- 172.21.15.62:0/2802246163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb898159490 0x7fb89806cc60 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.261+0000 7fb897200640 1 -- 172.21.15.62:0/2802246163 >> 172.21.15.62:0/2802246163 conn(0x7fb898066050 msgr2=0x7fb898069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:32.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.261+0000 7fb897200640 1 -- 172.21.15.62:0/2802246163 shutdown_connections 2024-03-23T11:26:32.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.261+0000 7fb897200640 1 -- 172.21.15.62:0/2802246163 wait complete. 2024-03-23T11:26:32.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.263+0000 7fb897200640 1 Processor -- start 2024-03-23T11:26:32.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.263+0000 7fb897200640 1 -- start start 2024-03-23T11:26:32.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.264+0000 7fb897200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8980ce5f0 0x7fb8981ed760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.264+0000 7fb897200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb898159490 0x7fb8981f1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.264+0000 7fb897200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb89815da90 0x7fb898157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.264+0000 7fb897200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898141810 con 0x7fb8980ce5f0 2024-03-23T11:26:32.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.264+0000 7fb897200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898156220 con 0x7fb898159490 2024-03-23T11:26:32.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.264+0000 7fb897200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981f2200 con 0x7fb89815da90 2024-03-23T11:26:32.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.264+0000 7fb895e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8980ce5f0 0x7fb8981ed760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.264+0000 7fb895e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8980ce5f0 0x7fb8981ed760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54090/0 (socket says 172.21.15.62:54090) 2024-03-23T11:26:32.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.264+0000 7fb895e00640 1 -- 172.21.15.62:0/756343653 learned_addr learned my addr 172.21.15.62:0/756343653 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:32.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb895400640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb898159490 0x7fb8981f1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb896800640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb89815da90 0x7fb898157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb895e00640 1 -- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb89815da90 msgr2=0x7fb898157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:32.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb895e00640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb89815da90 0x7fb898157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb895e00640 1 -- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb898159490 msgr2=0x7fb8981f1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:32.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb895e00640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb898159490 0x7fb8981f1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb895e00640 1 -- 172.21.15.62:0/756343653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb888009400 con 0x7fb8980ce5f0 2024-03-23T11:26:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb895400640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb898159490 0x7fb8981f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb895e00640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8980ce5f0 0x7fb8981ed760 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb888034040 tx=0x7fb8880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb88e000640 1 -- 172.21.15.62:0/756343653 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8880328a0 con 0x7fb8980ce5f0 2024-03-23T11:26:32.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb88e000640 1 -- 172.21.15.62:0/756343653 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb888032a20 con 0x7fb8980ce5f0 2024-03-23T11:26:32.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb88e000640 1 -- 172.21.15.62:0/756343653 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888032dd0 con 0x7fb8980ce5f0 2024-03-23T11:26:32.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.265+0000 7fb897200640 1 -- 172.21.15.62:0/756343653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb898156470 con 0x7fb8980ce5f0 2024-03-23T11:26:32.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.266+0000 7fb897200640 1 -- 172.21.15.62:0/756343653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb898156a30 con 0x7fb8980ce5f0 2024-03-23T11:26:32.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.267+0000 7fb897200640 1 -- 172.21.15.62:0/756343653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb898066bb0 con 0x7fb8980ce5f0 2024-03-23T11:26:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.268+0000 7fb88e000640 1 -- 172.21.15.62:0/756343653 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb888045400 con 0x7fb8980ce5f0 2024-03-23T11:26:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.268+0000 7fb88e000640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb87806a7a0 0x7fb87806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:32.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.269+0000 7fb88e000640 1 -- 172.21.15.62:0/756343653 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8880ab120 con 0x7fb8980ce5f0 2024-03-23T11:26:32.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.269+0000 7fb895400640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb87806a7a0 0x7fb87806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:32.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.270+0000 7fb895400640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb87806a7a0 0x7fb87806cc60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb8980ad460 tx=0x7fb884009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:32.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.273+0000 7fb88e000640 1 -- 172.21.15.62:0/756343653 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb8880765d0 con 0x7fb8980ce5f0 2024-03-23T11:26:32.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.511+0000 7fb897200640 1 -- 172.21.15.62:0/756343653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb89806d560 con 0x7fb8980ce5f0 2024-03-23T11:26:32.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.512+0000 7fb88e000640 1 -- 172.21.15.62:0/756343653 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb888079bf0 con 0x7fb8980ce5f0 2024-03-23T11:26:32.506 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:26:32.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 -- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb87806a7a0 msgr2=0x7fb87806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:32.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb87806a7a0 0x7fb87806cc60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb8980ad460 tx=0x7fb884009040 comp rx=0 tx=0).stop 2024-03-23T11:26:32.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 -- 172.21.15.62:0/756343653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8980ce5f0 msgr2=0x7fb8981ed760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:32.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8980ce5f0 0x7fb8981ed760 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb888034040 tx=0x7fb8880046c0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 -- 172.21.15.62:0/756343653 shutdown_connections 2024-03-23T11:26:32.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb87806a7a0 0x7fb87806cc60 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb89815da90 0x7fb898157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb898159490 0x7fb8981f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 --2- 172.21.15.62:0/756343653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8980ce5f0 0x7fb8981ed760 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:32.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 -- 172.21.15.62:0/756343653 >> 172.21.15.62:0/756343653 conn(0x7fb898066050 msgr2=0x7fb8980cd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:32.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 -- 172.21.15.62:0/756343653 shutdown_connections 2024-03-23T11:26:32.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:32.515+0000 7fb897200640 1 -- 172.21.15.62:0/756343653 wait complete. 2024-03-23T11:26:32.528 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:26:32.528 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:26:32.528 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:26:32.528 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:26:32.528 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:32.529 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:26:32.529 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:26:32.529 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:26:32.529 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:26:32.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:32.529 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:26:32.583 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:26:32.583 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:26:32.583 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:26:32.583 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:32.583 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:26:32.583 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:32.583 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:26:32.583 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:32.583 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:26:32.583 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42164sec preferred_lft 42164sec 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:32.584 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:26:32.585 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:32.585 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:26:32.585 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:32.585 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:26:32.585 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:26:32.585 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:26:32.585 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:32.585 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:26:32.585 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:26:32.585 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:26:32.585 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:32.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:32.587 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:26:32.684 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:26:32.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:32.689 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:26:32.740 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:26:32.741 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:26:32.741 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:26:32.741 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:26:32.741 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:26:32.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:32.743 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:26:32.743 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:26:32.743 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:26:32.744 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:26:32.744 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:26:32.744 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:26:32.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:32.941 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:26:32.994 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:26:32.994 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:26:32.994 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:26:32.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:32.995 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:26:33.095 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:26:33.095 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:26:33.096 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:33.154 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:26:33.155 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:26:33.155 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:33.208 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:26:33.307 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:26:33.307 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:26:33.403 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:26:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:26:33.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:33.408 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:26:33.461 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:26:33.462 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:26:33.467 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:26:33.553 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:26:33.553 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:26:33.630 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:26:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:26:33.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:33.634 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:26:33.662 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:26:33.668+0000 7f2297c38480 -1 init, newargv = 0x55b5cdceded0 newargc=15 2024-03-23T11:26:33.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[66002]: starting ceph client 2024-03-23T11:26:33.671 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[66002]: starting fuse 2024-03-23T11:26:33.676 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:26:33.677 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:26:33.677 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:26:33.677 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:26:33.677 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:26:33.677 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:26:33.677 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:26:33.677 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:33.677 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:26:33.677 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:26:33.677 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:26:33.678 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:26:33.679 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:33.679 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:26:33.679 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:26:33.883 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.66002.asok 2024-03-23T11:26:34.062 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.66002.asok 2024-03-23T11:26:34.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:34.062 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.66002.asok status 2024-03-23T11:26:34.303 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "66002", 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:26:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 7497, 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 7497 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1058153915 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:26:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:26:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1058153915 2024-03-23T11:26:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.7497 192.168.0.1:0/1058153915", 2024-03-23T11:26:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/1058153915", 2024-03-23T11:26:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:26:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 57, 2024-03-23T11:26:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 89, 2024-03-23T11:26:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:26:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:26:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:26:34.306 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:26:34.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:34.319 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:26:34.319 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:34.375 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:26:34.375 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:34.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:34.376 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:34.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:34.473 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir) 2024-03-23T11:26:34.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:34.543 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir/file) 2024-03-23T11:26:34.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:34.595 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir) 2024-03-23T11:26:34.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:34.652 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir/file testdir/hardlink) 2024-03-23T11:26:34.720 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:26:34.720 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:26:34.720 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:26:34.720 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:34.720 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir").st_ino) 2024-03-23T11:26:34.720 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:26:34.869 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627776 2024-03-23T11:26:35.050 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:26:35.050 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:26:35.149 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:26:35.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.374+0000 7fb688c00640 1 Processor -- start 2024-03-23T11:26:35.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.375+0000 7fb688c00640 1 -- start start 2024-03-23T11:26:35.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.376+0000 7fb688c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb684143dd0 0x7fb68415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.376+0000 7fb688c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb684070090 0x7fb684070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.376+0000 7fb688c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6840b2820 0x7fb6840b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.376+0000 7fb688c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb684070a60 con 0x7fb684143dd0 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.376+0000 7fb688c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6840b31b0 con 0x7fb684070090 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.376+0000 7fb688c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840cd140 con 0x7fb6840b2820 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.377+0000 7fb683400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb684143dd0 0x7fb68415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.377+0000 7fb683400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb684143dd0 0x7fb68415abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54098/0 (socket says 172.21.15.62:54098) 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.377+0000 7fb683400640 1 -- 172.21.15.62:0/300871283 learned_addr learned my addr 172.21.15.62:0/300871283 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.377+0000 7fb67be00640 1 --2- 172.21.15.62:0/300871283 >> v2:172.21.15.120:3300/0 conn(0x7fb684070090 0x7fb684070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.377+0000 7fb683e00640 1 --2- 172.21.15.62:0/300871283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6840b2820 0x7fb6840b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.377+0000 7fb683400640 1 -- 172.21.15.62:0/300871283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6840b2820 msgr2=0x7fb6840b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.377+0000 7fb683400640 1 --2- 172.21.15.62:0/300871283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6840b2820 0x7fb6840b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.377+0000 7fb683400640 1 -- 172.21.15.62:0/300871283 >> v2:172.21.15.120:3300/0 conn(0x7fb684070090 msgr2=0x7fb684070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:35.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.377+0000 7fb683400640 1 --2- 172.21.15.62:0/300871283 >> v2:172.21.15.120:3300/0 conn(0x7fb684070090 0x7fb684070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.377+0000 7fb683400640 1 -- 172.21.15.62:0/300871283 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6840cd3e0 con 0x7fb684143dd0 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.377+0000 7fb67be00640 1 --2- 172.21.15.62:0/300871283 >> v2:172.21.15.120:3300/0 conn(0x7fb684070090 0x7fb684070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.379+0000 7fb683400640 1 --2- 172.21.15.62:0/300871283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb684143dd0 0x7fb68415abe0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fb684071530 tx=0x7fb67402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3ed54d31526aa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.379+0000 7fb682a00640 1 -- 172.21.15.62:0/300871283 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674031870 con 0x7fb684143dd0 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.379+0000 7fb682a00640 1 -- 172.21.15.62:0/300871283 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6740319d0 con 0x7fb684143dd0 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.379+0000 7fb682a00640 1 -- 172.21.15.62:0/300871283 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674031d60 con 0x7fb684143dd0 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.380+0000 7fb688c00640 1 -- 172.21.15.62:0/300871283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb684143dd0 msgr2=0x7fb68415abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.380+0000 7fb688c00640 1 --2- 172.21.15.62:0/300871283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb684143dd0 0x7fb68415abe0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fb684071530 tx=0x7fb67402b910 comp rx=0 tx=0).stop 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.380+0000 7fb688c00640 1 -- 172.21.15.62:0/300871283 shutdown_connections 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.380+0000 7fb688c00640 1 --2- 172.21.15.62:0/300871283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb6840b2820 0x7fb6840b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.380+0000 7fb688c00640 1 --2- 172.21.15.62:0/300871283 >> v2:172.21.15.120:3300/0 conn(0x7fb684070090 0x7fb684070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.380+0000 7fb688c00640 1 --2- 172.21.15.62:0/300871283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb684143dd0 0x7fb68415abe0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.380+0000 7fb688c00640 1 -- 172.21.15.62:0/300871283 >> 172.21.15.62:0/300871283 conn(0x7fb684066050 msgr2=0x7fb684069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:35.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.380+0000 7fb688c00640 1 -- 172.21.15.62:0/300871283 shutdown_connections 2024-03-23T11:26:35.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.380+0000 7fb688c00640 1 -- 172.21.15.62:0/300871283 wait complete. 2024-03-23T11:26:35.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.382+0000 7fb688c00640 1 Processor -- start 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.382+0000 7fb688c00640 1 -- start start 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.383+0000 7fb688c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb684070090 0x7fb68413ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.383+0000 7fb688c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6840b2820 0x7fb6840b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.383+0000 7fb688c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb684143dd0 0x7fb684138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.383+0000 7fb688c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840af2f0 con 0x7fb6840b2820 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.383+0000 7fb688c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840af5d0 con 0x7fb684143dd0 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.383+0000 7fb688c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840af8b0 con 0x7fb684070090 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.383+0000 7fb67be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6840b2820 0x7fb6840b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.383+0000 7fb67be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6840b2820 0x7fb6840b0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54104/0 (socket says 172.21.15.62:54104) 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.383+0000 7fb67be00640 1 -- 172.21.15.62:0/470808699 learned_addr learned my addr 172.21.15.62:0/470808699 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.384+0000 7fb67be00640 1 -- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb684070090 msgr2=0x7fb68413ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:35.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.384+0000 7fb683400640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb684070090 0x7fb68413ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.384+0000 7fb67be00640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb684070090 0x7fb68413ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.384+0000 7fb67be00640 1 -- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb684143dd0 msgr2=0x7fb684138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:35.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.384+0000 7fb67be00640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb684143dd0 0x7fb684138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.384+0000 7fb67be00640 1 -- 172.21.15.62:0/470808699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6740094d0 con 0x7fb6840b2820 2024-03-23T11:26:35.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.384+0000 7fb67be00640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6840b2820 0x7fb6840b0390 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00c7f0 tx=0x7fb66c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:35.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.384+0000 7fb67b400640 1 -- 172.21.15.62:0/470808699 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c007820 con 0x7fb6840b2820 2024-03-23T11:26:35.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.384+0000 7fb67b400640 1 -- 172.21.15.62:0/470808699 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c007980 con 0x7fb6840b2820 2024-03-23T11:26:35.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.384+0000 7fb67b400640 1 -- 172.21.15.62:0/470808699 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c007cf0 con 0x7fb6840b2820 2024-03-23T11:26:35.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.384+0000 7fb688c00640 1 -- 172.21.15.62:0/470808699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6840afb90 con 0x7fb6840b2820 2024-03-23T11:26:35.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.385+0000 7fb688c00640 1 -- 172.21.15.62:0/470808699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb684138f80 con 0x7fb6840b2820 2024-03-23T11:26:35.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.386+0000 7fb688c00640 1 -- 172.21.15.62:0/470808699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb684066bb0 con 0x7fb6840b2820 2024-03-23T11:26:35.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.387+0000 7fb67b400640 1 -- 172.21.15.62:0/470808699 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb66c0113f0 con 0x7fb6840b2820 2024-03-23T11:26:35.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.387+0000 7fb67b400640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb66006a7d0 0x7fb66006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.388+0000 7fb67b400640 1 -- 172.21.15.62:0/470808699 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb66c0891c0 con 0x7fb6840b2820 2024-03-23T11:26:35.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.388+0000 7fb683400640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb66006a7d0 0x7fb66006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.389+0000 7fb683400640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb66006a7d0 0x7fb66006cc90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb674036000 tx=0x7fb67403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:35.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.393+0000 7fb67b400640 1 -- 172.21.15.62:0/470808699 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb66c0544b0 con 0x7fb6840b2820 2024-03-23T11:26:35.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.591+0000 7fb688c00640 1 -- 172.21.15.62:0/470808699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb68415dc40 con 0x7fb6840b2820 2024-03-23T11:26:35.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.595+0000 7fb67b400640 1 -- 172.21.15.62:0/470808699 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16092 (secure 0 0 0) 0x7fb66c057ad0 con 0x7fb6840b2820 2024-03-23T11:26:35.589 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:35.590 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":89,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:26:26.134957+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:26:22.794149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-23T11:26:24.150997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:26:35.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 -- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb66006a7d0 msgr2=0x7fb66006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:35.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb66006a7d0 0x7fb66006cc90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb674036000 tx=0x7fb67403c040 comp rx=0 tx=0).stop 2024-03-23T11:26:35.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 -- 172.21.15.62:0/470808699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6840b2820 msgr2=0x7fb6840b0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:35.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6840b2820 0x7fb6840b0390 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00c7f0 tx=0x7fb66c00ccb0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 -- 172.21.15.62:0/470808699 shutdown_connections 2024-03-23T11:26:35.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb66006a7d0 0x7fb66006cc90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6840b2820 0x7fb6840b0390 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb684143dd0 0x7fb684138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 --2- 172.21.15.62:0/470808699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb684070090 0x7fb68413ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 -- 172.21.15.62:0/470808699 >> 172.21.15.62:0/470808699 conn(0x7fb684066050 msgr2=0x7fb68415b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:35.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 -- 172.21.15.62:0/470808699 shutdown_connections 2024-03-23T11:26:35.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.598+0000 7fb688c00640 1 -- 172.21.15.62:0/470808699 wait complete. 2024-03-23T11:26:35.612 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:26:22.794149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:26:24.150997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:26:35.613 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:26:35.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.857+0000 7fd25d800640 1 Processor -- start 2024-03-23T11:26:35.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.857+0000 7fd25d800640 1 -- start start 2024-03-23T11:26:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.859+0000 7fd25d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2580cc990 0x7fd258069c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.859+0000 7fd25d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2580ce5f0 0x7fd2580cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.859+0000 7fd25d800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2580c99b0 con 0x7fd25815caf0 2024-03-23T11:26:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.859+0000 7fd25d800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25806a560 con 0x7fd2580ce5f0 2024-03-23T11:26:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.859+0000 7fd25d800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2580c9c00 con 0x7fd2580cc990 2024-03-23T11:26:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.859+0000 7fd25ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2580ce5f0 0x7fd2580cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.859+0000 7fd25ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2580ce5f0 0x7fd2580cea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54108/0 (socket says 172.21.15.62:54108) 2024-03-23T11:26:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.859+0000 7fd25ce00640 1 -- 172.21.15.62:0/3172482055 learned_addr learned my addr 172.21.15.62:0/3172482055 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.859+0000 7fd257400640 1 --2- 172.21.15.62:0/3172482055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2580cc990 0x7fd258069c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.860+0000 7fd25ce00640 1 -- 172.21.15.62:0/3172482055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2580cc990 msgr2=0x7fd258069c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:35.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.860+0000 7fd25ce00640 1 --2- 172.21.15.62:0/3172482055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2580cc990 0x7fd258069c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.860+0000 7fd25ce00640 1 -- 172.21.15.62:0/3172482055 >> v1:172.21.15.120:6789/0 conn(0x7fd25815caf0 legacy=0x7fd2580b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:35.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.860+0000 7fd25ce00640 1 -- 172.21.15.62:0/3172482055 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2580c9ee0 con 0x7fd2580ce5f0 2024-03-23T11:26:35.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.860+0000 7fd257400640 1 --2- 172.21.15.62:0/3172482055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2580cc990 0x7fd258069c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:35.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.861+0000 7fd25ce00640 1 --2- 172.21.15.62:0/3172482055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580ce5f0 0x7fd2580cea80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fd250009e50 tx=0x7fd25002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69710777d1800371 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:35.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.861+0000 7fd256a00640 1 -- 172.21.15.62:0/3172482055 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25002ca70 con 0x7fd2580ce5f0 2024-03-23T11:26:35.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.861+0000 7fd256a00640 1 -- 172.21.15.62:0/3172482055 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25002cbd0 con 0x7fd2580ce5f0 2024-03-23T11:26:35.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.861+0000 7fd256a00640 1 -- 172.21.15.62:0/3172482055 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2500343f0 con 0x7fd2580ce5f0 2024-03-23T11:26:35.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.862+0000 7fd25d800640 1 -- 172.21.15.62:0/3172482055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580ce5f0 msgr2=0x7fd2580cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:35.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.862+0000 7fd25d800640 1 --2- 172.21.15.62:0/3172482055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580ce5f0 0x7fd2580cea80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fd250009e50 tx=0x7fd25002bed0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.862+0000 7fd25d800640 1 -- 172.21.15.62:0/3172482055 shutdown_connections 2024-03-23T11:26:35.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.862+0000 7fd25d800640 1 --2- 172.21.15.62:0/3172482055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580ce5f0 0x7fd2580cea80 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.862+0000 7fd25d800640 1 --2- 172.21.15.62:0/3172482055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2580cc990 0x7fd258069c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.862+0000 7fd25d800640 1 -- 172.21.15.62:0/3172482055 >> 172.21.15.62:0/3172482055 conn(0x7fd2580b0f50 msgr2=0x7fd2580b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:35.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.862+0000 7fd25d800640 1 -- 172.21.15.62:0/3172482055 shutdown_connections 2024-03-23T11:26:35.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.862+0000 7fd25d800640 1 -- 172.21.15.62:0/3172482055 wait complete. 2024-03-23T11:26:35.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.864+0000 7fd25d800640 1 Processor -- start 2024-03-23T11:26:35.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.864+0000 7fd25d800640 1 -- start start 2024-03-23T11:26:35.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd25d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cc990 0x7fd25806e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd25d800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2580ce5f0 0x7fd25806e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd25d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25815caf0 0x7fd2580c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd25d800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25815d7d0 con 0x7fd2580cc990 2024-03-23T11:26:35.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd25d800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2580c7e10 con 0x7fd2580ce5f0 2024-03-23T11:26:35.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd25d800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2580c80f0 con 0x7fd25815caf0 2024-03-23T11:26:35.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd257e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cc990 0x7fd25806e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd257e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cc990 0x7fd25806e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54122/0 (socket says 172.21.15.62:54122) 2024-03-23T11:26:35.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd257e00640 1 -- 172.21.15.62:0/31305114 learned_addr learned my addr 172.21.15.62:0/31305114 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:35.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd257400640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2580ce5f0 0x7fd25806e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd25ce00640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25815caf0 0x7fd2580c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd257e00640 1 -- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25815caf0 msgr2=0x7fd2580c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:35.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd257e00640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25815caf0 0x7fd2580c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd257e00640 1 -- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2580ce5f0 msgr2=0x7fd25806e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:35.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd257e00640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2580ce5f0 0x7fd25806e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:35.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.865+0000 7fd257e00640 1 -- 172.21.15.62:0/31305114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd250009b30 con 0x7fd2580cc990 2024-03-23T11:26:35.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.866+0000 7fd257e00640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cc990 0x7fd25806e360 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fd24800b3f0 tx=0x7fd24800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:35.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.866+0000 7fd24fe00640 1 -- 172.21.15.62:0/31305114 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd248011830 con 0x7fd2580cc990 2024-03-23T11:26:35.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.866+0000 7fd24fe00640 1 -- 172.21.15.62:0/31305114 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd248011990 con 0x7fd2580cc990 2024-03-23T11:26:35.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.866+0000 7fd24fe00640 1 -- 172.21.15.62:0/31305114 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd248011d40 con 0x7fd2580cc990 2024-03-23T11:26:35.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.866+0000 7fd25d800640 1 -- 172.21.15.62:0/31305114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2580c8730 con 0x7fd2580cc990 2024-03-23T11:26:35.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.866+0000 7fd25d800640 1 -- 172.21.15.62:0/31305114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25815db10 con 0x7fd2580cc990 2024-03-23T11:26:35.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.867+0000 7fd25d800640 1 -- 172.21.15.62:0/31305114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2580c82b0 con 0x7fd2580cc990 2024-03-23T11:26:35.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.868+0000 7fd24fe00640 1 -- 172.21.15.62:0/31305114 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd24801a400 con 0x7fd2580cc990 2024-03-23T11:26:35.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.869+0000 7fd24fe00640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 0x7fd23806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:35.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.869+0000 7fd24fe00640 1 -- 172.21.15.62:0/31305114 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd248088c80 con 0x7fd2580cc990 2024-03-23T11:26:35.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.870+0000 7fd257400640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 0x7fd23806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:35.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.871+0000 7fd257400640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 0x7fd23806cc60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd25815e090 tx=0x7fd244008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:35.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:35.873+0000 7fd24fe00640 1 -- 172.21.15.62:0/31305114 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd248054000 con 0x7fd2580cc990 2024-03-23T11:26:36.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.070+0000 7fd25d800640 1 -- 172.21.15.62:0/31305114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd2580c8c50 con 0x7fd2580cc990 2024-03-23T11:26:36.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.071+0000 7fd24fe00640 1 -- 172.21.15.62:0/31305114 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd2580c8c50 con 0x7fd2580cc990 2024-03-23T11:26:36.066 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:26:36.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 -- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 msgr2=0x7fd23806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 0x7fd23806cc60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd25815e090 tx=0x7fd244008040 comp rx=0 tx=0).stop 2024-03-23T11:26:36.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 -- 172.21.15.62:0/31305114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cc990 msgr2=0x7fd25806e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cc990 0x7fd25806e360 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fd24800b3f0 tx=0x7fd24800b8b0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 -- 172.21.15.62:0/31305114 shutdown_connections 2024-03-23T11:26:36.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd23806a7a0 0x7fd23806cc60 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd25815caf0 0x7fd2580c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2580ce5f0 0x7fd25806e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 --2- 172.21.15.62:0/31305114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2580cc990 0x7fd25806e360 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 -- 172.21.15.62:0/31305114 >> 172.21.15.62:0/31305114 conn(0x7fd2580b0f50 msgr2=0x7fd2580b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:36.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 -- 172.21.15.62:0/31305114 shutdown_connections 2024-03-23T11:26:36.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.074+0000 7fd25d800640 1 -- 172.21.15.62:0/31305114 wait complete. 2024-03-23T11:26:36.088 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:26:36.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.329+0000 7f4392200640 1 Processor -- start 2024-03-23T11:26:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.329+0000 7f4392200640 1 -- start start 2024-03-23T11:26:36.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.331+0000 7f4392200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f438c0b15e0 0x7f438c06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.331+0000 7f4392200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438c0cad20 con 0x7f438c0cd7c0 2024-03-23T11:26:36.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.331+0000 7f4392200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438c0b19f0 con 0x7f438c06dca0 2024-03-23T11:26:36.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.331+0000 7f4392200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438c06e130 con 0x7f438c0b15e0 2024-03-23T11:26:36.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.331+0000 7f438b400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f438c0cd7c0 0x7f438c0cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37368/0 (socket says 172.21.15.62:37368) 2024-03-23T11:26:36.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.331+0000 7f438b400640 1 -- 172.21.15.62:0/2293158946 learned_addr learned my addr 172.21.15.62:0/2293158946 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:36.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.331+0000 7f438be00640 1 --2- 172.21.15.62:0/2293158946 >> v2:172.21.15.120:3300/0 conn(0x7f438c0b15e0 0x7f438c06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202673318 0 0) 0x7f438c0cad20 con 0x7f438c0cd7c0 2024-03-23T11:26:36.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4370005730 con 0x7f438c0cd7c0 2024-03-23T11:26:36.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3930123537 0 0) 0x7f4370005730 con 0x7f438c0cd7c0 2024-03-23T11:26:36.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 >> v2:172.21.15.120:3300/0 conn(0x7f438c0b15e0 msgr2=0x7f438c06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:36.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 --2- 172.21.15.62:0/2293158946 >> v2:172.21.15.120:3300/0 conn(0x7f438c0b15e0 0x7f438c06c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 >> v1:172.21.15.120:6789/0 conn(0x7f438c06dca0 legacy=0x7f438c0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c15a8a0 con 0x7f438c0cd7c0 2024-03-23T11:26:36.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f437c001670 con 0x7f438c0cd7c0 2024-03-23T11:26:36.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 >> v1:172.21.15.62:6789/0 conn(0x7f438c0cd7c0 legacy=0x7f438c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 0x7f4370005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43700069d0 0x7f4370006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43700073e0 0x7f437000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c06e130 con 0x7f4370005b80 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437000e310 con 0x7f43700069d0 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437000e4d0 con 0x7f43700073e0 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.332+0000 7f438b400640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 0x7f4370005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.333+0000 7f438be00640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43700073e0 0x7f437000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.333+0000 7f438b400640 1 -- 172.21.15.62:0/2293158946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43700073e0 msgr2=0x7f437000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.333+0000 7f438b400640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43700073e0 0x7f437000db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.333+0000 7f438b400640 1 -- 172.21.15.62:0/2293158946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43700069d0 msgr2=0x7f4370006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.333+0000 7f438aa00640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43700069d0 0x7f4370006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.333+0000 7f438b400640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43700069d0 0x7f4370006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.333+0000 7f438b400640 1 -- 172.21.15.62:0/2293158946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370005730 con 0x7f4370005b80 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.333+0000 7f438be00640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43700073e0 0x7f437000db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.333+0000 7f438aa00640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43700069d0 0x7f4370006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:36.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.333+0000 7f438be00640 1 -- 172.21.15.62:0/2293158946 reap_dead start 2024-03-23T11:26:36.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.334+0000 7f438b400640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 0x7f4370005f60 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f437c00c190 tx=0x7f4370007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7bfcb7fa296d135 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:36.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.334+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c005ad0 con 0x7f4370005b80 2024-03-23T11:26:36.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.335+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437c005c50 con 0x7f4370005b80 2024-03-23T11:26:36.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.335+0000 7f438a000640 1 -- 172.21.15.62:0/2293158946 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c02c810 con 0x7f4370005b80 2024-03-23T11:26:36.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.335+0000 7f4392200640 1 -- 172.21.15.62:0/2293158946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 msgr2=0x7f4370005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.335+0000 7f4392200640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 0x7f4370005f60 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f437c00c190 tx=0x7f4370007a00 comp rx=0 tx=0).stop 2024-03-23T11:26:36.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.335+0000 7f4392200640 1 -- 172.21.15.62:0/2293158946 shutdown_connections 2024-03-23T11:26:36.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.335+0000 7f4392200640 1 --2- 172.21.15.62:0/2293158946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 0x7f4370005f60 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.335+0000 7f4392200640 1 -- 172.21.15.62:0/2293158946 >> 172.21.15.62:0/2293158946 conn(0x7f438c0b02e0 msgr2=0x7f438c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:36.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.335+0000 7f4392200640 1 -- 172.21.15.62:0/2293158946 shutdown_connections 2024-03-23T11:26:36.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.335+0000 7f4392200640 1 -- 172.21.15.62:0/2293158946 wait complete. 2024-03-23T11:26:36.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.338+0000 7f4392200640 1 Processor -- start 2024-03-23T11:26:36.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.338+0000 7f4392200640 1 -- start start 2024-03-23T11:26:36.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.339+0000 7f4392200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 0x7f438c0c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.339+0000 7f4392200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f438c153ee0 0x7f438c0ac530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.339+0000 7f4392200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f438c0aca70 0x7f438c148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.339+0000 7f4392200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c146c90 con 0x7f4370005b80 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.339+0000 7f4392200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c147810 con 0x7f438c0aca70 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.339+0000 7f4392200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438c146ee0 con 0x7f438c153ee0 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.339+0000 7f438b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 0x7f438c0c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.339+0000 7f438b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 0x7f438c0c38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54142/0 (socket says 172.21.15.62:54142) 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.339+0000 7f438b400640 1 -- 172.21.15.62:0/1938475454 learned_addr learned my addr 172.21.15.62:0/1938475454 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f438aa00640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f438c153ee0 0x7f438c0ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f438be00640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f438c0aca70 0x7f438c148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f438b400640 1 -- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f438c153ee0 msgr2=0x7f438c0ac530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f438b400640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f438c153ee0 0x7f438c0ac530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f438b400640 1 -- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f438c0aca70 msgr2=0x7f438c148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f438b400640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f438c0aca70 0x7f438c148d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f438b400640 1 -- 172.21.15.62:0/1938475454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c1471c0 con 0x7f4370005b80 2024-03-23T11:26:36.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f438b400640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 0x7f438c0c38a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f437c00c2b0 tx=0x7f437c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:36.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f4383400640 1 -- 172.21.15.62:0/1938475454 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c03c040 con 0x7f4370005b80 2024-03-23T11:26:36.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f4383400640 1 -- 172.21.15.62:0/1938475454 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437c02cc20 con 0x7f4370005b80 2024-03-23T11:26:36.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f4383400640 1 -- 172.21.15.62:0/1938475454 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c007070 con 0x7f4370005b80 2024-03-23T11:26:36.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.340+0000 7f4392200640 1 -- 172.21.15.62:0/1938475454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438c1474a0 con 0x7f4370005b80 2024-03-23T11:26:36.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.341+0000 7f4392200640 1 -- 172.21.15.62:0/1938475454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438c0c3de0 con 0x7f4370005b80 2024-03-23T11:26:36.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.342+0000 7f4392200640 1 -- 172.21.15.62:0/1938475454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438c06e270 con 0x7f4370005b80 2024-03-23T11:26:36.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.343+0000 7f4383400640 1 -- 172.21.15.62:0/1938475454 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f437c032070 con 0x7f4370005b80 2024-03-23T11:26:36.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.344+0000 7f4383400640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f436c06a7f0 0x7f436c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.344+0000 7f4383400640 1 -- 172.21.15.62:0/1938475454 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f437c0aa220 con 0x7f4370005b80 2024-03-23T11:26:36.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.344+0000 7f438aa00640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f436c06a7f0 0x7f436c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.346+0000 7f438aa00640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f436c06a7f0 0x7f436c06ccb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f438c0c3600 tx=0x7f4378009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:36.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.349+0000 7f4383400640 1 -- 172.21.15.62:0/1938475454 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f437c0757d0 con 0x7f4370005b80 2024-03-23T11:26:36.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.550+0000 7f4392200640 1 -- 172.21.15.62:0/1938475454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f438c066c90 con 0x7f4370005b80 2024-03-23T11:26:36.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.551+0000 7f4383400640 1 -- 172.21.15.62:0/1938475454 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f437c078df0 con 0x7f4370005b80 2024-03-23T11:26:36.545 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:26:36.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 -- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f436c06a7f0 msgr2=0x7f436c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f436c06a7f0 0x7f436c06ccb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f438c0c3600 tx=0x7f4378009040 comp rx=0 tx=0).stop 2024-03-23T11:26:36.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 -- 172.21.15.62:0/1938475454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 msgr2=0x7f438c0c38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 0x7f438c0c38a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f437c00c2b0 tx=0x7f437c02e070 comp rx=0 tx=0).stop 2024-03-23T11:26:36.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 -- 172.21.15.62:0/1938475454 shutdown_connections 2024-03-23T11:26:36.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f436c06a7f0 0x7f436c06ccb0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f438c0aca70 0x7f438c148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f438c153ee0 0x7f438c0ac530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 --2- 172.21.15.62:0/1938475454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4370005b80 0x7f438c0c38a0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 -- 172.21.15.62:0/1938475454 >> 172.21.15.62:0/1938475454 conn(0x7f438c0b02e0 msgr2=0x7f438c0afb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:36.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 -- 172.21.15.62:0/1938475454 shutdown_connections 2024-03-23T11:26:36.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.554+0000 7f4392200640 1 -- 172.21.15.62:0/1938475454 wait complete. 2024-03-23T11:26:36.568 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:26:36.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.799+0000 7f02bc800640 1 Processor -- start 2024-03-23T11:26:36.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.800+0000 7f02bc800640 1 -- start start 2024-03-23T11:26:36.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.801+0000 7f02bc800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b41613a0 0x7f02b406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.801+0000 7f02bc800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02b406d910 0x7f02b415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.801+0000 7f02bc800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b406cf40 con 0x7f02b41571e0 2024-03-23T11:26:36.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.801+0000 7f02bc800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b41575c0 con 0x7f02b406d910 2024-03-23T11:26:36.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.801+0000 7f02bc800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b406c3a0 con 0x7f02b41613a0 2024-03-23T11:26:36.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.802+0000 7f02b9e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f02b41571e0 0x7f02b40ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37378/0 (socket says 172.21.15.62:37378) 2024-03-23T11:26:36.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.802+0000 7f02b9e00640 1 -- 172.21.15.62:0/3504073562 learned_addr learned my addr 172.21.15.62:0/3504073562 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:36.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.802+0000 7f02b9400640 1 --2- 172.21.15.62:0/3504073562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b41613a0 0x7f02b406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.802+0000 7f02ba800640 1 --2- 172.21.15.62:0/3504073562 >> v2:172.21.15.120:3300/0 conn(0x7f02b406d910 0x7f02b415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.802+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499833204 0 0) 0x7f02b406cf40 con 0x7f02b41571e0 2024-03-23T11:26:36.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.802+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02a0003520 con 0x7f02b41571e0 2024-03-23T11:26:36.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.802+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3618590481 0 0) 0x7f02a0003520 con 0x7f02b41571e0 2024-03-23T11:26:36.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b41613a0 msgr2=0x7f02b406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:36.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 --2- 172.21.15.62:0/3504073562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b41613a0 0x7f02b406d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 >> v2:172.21.15.120:3300/0 conn(0x7f02b406d910 msgr2=0x7f02b415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:36.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 --2- 172.21.15.62:0/3504073562 >> v2:172.21.15.120:3300/0 conn(0x7f02b406d910 0x7f02b415e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b40ae040 con 0x7f02b41571e0 2024-03-23T11:26:36.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f02ac0019f0 con 0x7f02b41571e0 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 >> v1:172.21.15.62:6789/0 conn(0x7f02b41571e0 legacy=0x7f02b40ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 --2- 172.21.15.62:0/3504073562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02a0003970 0x7f02a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 --2- 172.21.15.62:0/3504073562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02a00047c0 0x7f02a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 --2- 172.21.15.62:0/3504073562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b41613a0 0x7f02a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b41575c0 con 0x7f02a0003970 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b406c3a0 con 0x7f02a00047c0 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a000ba80 con 0x7f02b41613a0 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02ba800640 1 --2- 172.21.15.62:0/3504073562 >> v2:172.21.15.120:3300/0 conn(0x7f02b406d910 0x7f02b415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b9e00640 1 --2- 172.21.15.62:0/3504073562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02a0003970 0x7f02a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b9e00640 1 -- 172.21.15.62:0/3504073562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b41613a0 msgr2=0x7f02a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b9e00640 1 --2- 172.21.15.62:0/3504073562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02b41613a0 0x7f02a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b9e00640 1 -- 172.21.15.62:0/3504073562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02a00047c0 msgr2=0x7f02a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b9e00640 1 --2- 172.21.15.62:0/3504073562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02a00047c0 0x7f02a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.803+0000 7f02b9e00640 1 -- 172.21.15.62:0/3504073562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a0003520 con 0x7f02a0003970 2024-03-23T11:26:36.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.805+0000 7f02b9e00640 1 --2- 172.21.15.62:0/3504073562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02a0003970 0x7f02a0003d90 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00a790 tx=0x7f02a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab70f7d55d7ca143 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:36.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.805+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac034630 con 0x7f02a0003970 2024-03-23T11:26:36.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.805+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ac02f070 con 0x7f02a0003970 2024-03-23T11:26:36.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.805+0000 7f02b8a00640 1 -- 172.21.15.62:0/3504073562 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0349f0 con 0x7f02a0003970 2024-03-23T11:26:36.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.805+0000 7f02bc800640 1 -- 172.21.15.62:0/3504073562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02a0003970 msgr2=0x7f02a0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.805+0000 7f02bc800640 1 --2- 172.21.15.62:0/3504073562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02a0003970 0x7f02a0003d90 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00a790 tx=0x7f02a00057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.806+0000 7f02ba800640 1 -- 172.21.15.62:0/3504073562 reap_dead start 2024-03-23T11:26:36.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.806+0000 7f02bc800640 1 -- 172.21.15.62:0/3504073562 shutdown_connections 2024-03-23T11:26:36.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.806+0000 7f02bc800640 1 -- 172.21.15.62:0/3504073562 >> 172.21.15.62:0/3504073562 conn(0x7f02b40b02e0 msgr2=0x7f02b40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:36.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.806+0000 7f02bc800640 1 -- 172.21.15.62:0/3504073562 shutdown_connections 2024-03-23T11:26:36.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.806+0000 7f02bc800640 1 -- 172.21.15.62:0/3504073562 wait complete. 2024-03-23T11:26:36.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.808+0000 7f02bc800640 1 Processor -- start 2024-03-23T11:26:36.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.808+0000 7f02bc800640 1 -- start start 2024-03-23T11:26:36.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02bc800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02a0003970 0x7f02b40cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02bc800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b4137b80 0x7f02b415e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02bc800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b4138000 0x7f02b415e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02bc800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b41421d0 con 0x7f02b4138000 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02bc800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b40cb370 con 0x7f02b4137b80 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02bc800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b415ace0 con 0x7f02a0003970 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02ba800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b4138000 0x7f02b415e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02ba800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b4138000 0x7f02b415e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54166/0 (socket says 172.21.15.62:54166) 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02ba800640 1 -- 172.21.15.62:0/3742728724 learned_addr learned my addr 172.21.15.62:0/3742728724 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02ba800640 1 -- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02a0003970 msgr2=0x7f02b40cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02ba800640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02a0003970 0x7f02b40cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02ba800640 1 -- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b4137b80 msgr2=0x7f02b415e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:36.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02b9400640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b4137b80 0x7f02b415e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02ba800640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b4137b80 0x7f02b415e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:36.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.809+0000 7f02ba800640 1 -- 172.21.15.62:0/3742728724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b415afc0 con 0x7f02b4138000 2024-03-23T11:26:36.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.810+0000 7f02ba800640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b4138000 0x7f02b415e5f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f02b0006fd0 tx=0x7f02b000e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:36.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.810+0000 7f02aa000640 1 -- 172.21.15.62:0/3742728724 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b000c070 con 0x7f02b4138000 2024-03-23T11:26:36.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.810+0000 7f02aa000640 1 -- 172.21.15.62:0/3742728724 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b0004230 con 0x7f02b4138000 2024-03-23T11:26:36.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.810+0000 7f02aa000640 1 -- 172.21.15.62:0/3742728724 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0003610 con 0x7f02b4138000 2024-03-23T11:26:36.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.810+0000 7f02bc800640 1 -- 172.21.15.62:0/3742728724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b415b2a0 con 0x7f02b4138000 2024-03-23T11:26:36.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.810+0000 7f02bc800640 1 -- 172.21.15.62:0/3742728724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b415b860 con 0x7f02b4138000 2024-03-23T11:26:36.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.811+0000 7f02bc800640 1 -- 172.21.15.62:0/3742728724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b415ba50 con 0x7f02b4138000 2024-03-23T11:26:36.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.812+0000 7f02aa000640 1 -- 172.21.15.62:0/3742728724 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f02b0003770 con 0x7f02b4138000 2024-03-23T11:26:36.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.814+0000 7f02aa000640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f029806a7a0 0x7f029806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:36.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.814+0000 7f02aa000640 1 -- 172.21.15.62:0/3742728724 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02b00883e0 con 0x7f02b4138000 2024-03-23T11:26:36.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.814+0000 7f02b9e00640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f029806a7a0 0x7f029806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:36.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.815+0000 7f02b9e00640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f029806a7a0 0x7f029806cc60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f02ac034000 tx=0x7f02ac007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:36.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:36.818+0000 7f02aa000640 1 -- 172.21.15.62:0/3742728724 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f02b00537f0 con 0x7f02b4138000 2024-03-23T11:26:37.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.018+0000 7f02bc800640 1 -- 172.21.15.62:0/3742728724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02b4066c90 con 0x7f02b4138000 2024-03-23T11:26:37.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.019+0000 7f02aa000640 1 -- 172.21.15.62:0/3742728724 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f02b0056e10 con 0x7f02b4138000 2024-03-23T11:26:37.014 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.022+0000 7f02bc800640 1 -- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f029806a7a0 msgr2=0x7f029806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.022+0000 7f02bc800640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f029806a7a0 0x7f029806cc60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f02ac034000 tx=0x7f02ac007000 comp rx=0 tx=0).stop 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.022+0000 7f02bc800640 1 -- 172.21.15.62:0/3742728724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b4138000 msgr2=0x7f02b415e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.022+0000 7f02bc800640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b4138000 0x7f02b415e5f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f02b0006fd0 tx=0x7f02b000e830 comp rx=0 tx=0).stop 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.022+0000 7f02bc800640 1 -- 172.21.15.62:0/3742728724 shutdown_connections 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.022+0000 7f02bc800640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f029806a7a0 0x7f029806cc60 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.022+0000 7f02bc800640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b4138000 0x7f02b415e5f0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.022+0000 7f02bc800640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02b4137b80 0x7f02b415e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.022+0000 7f02bc800640 1 --2- 172.21.15.62:0/3742728724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02a0003970 0x7f02b40cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.022+0000 7f02bc800640 1 -- 172.21.15.62:0/3742728724 >> 172.21.15.62:0/3742728724 conn(0x7f02b40b02e0 msgr2=0x7f02b40ae1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.023+0000 7f02bc800640 1 -- 172.21.15.62:0/3742728724 shutdown_connections 2024-03-23T11:26:37.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.023+0000 7f02bc800640 1 -- 172.21.15.62:0/3742728724 wait complete. 2024-03-23T11:26:37.035 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:26:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.252+0000 7ff11f600640 1 Processor -- start 2024-03-23T11:26:37.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.252+0000 7ff11f600640 1 -- start start 2024-03-23T11:26:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.254+0000 7ff11f600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff1180cafa0 0x7ff1180cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.254+0000 7ff11f600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1180cbc60 con 0x7ff11806c300 2024-03-23T11:26:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.254+0000 7ff11f600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1180cbf40 con 0x7ff11806c6e0 2024-03-23T11:26:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.254+0000 7ff11f600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1180cd410 con 0x7ff1180cafa0 2024-03-23T11:26:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.254+0000 7ff117e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff1180cafa0 0x7ff1180cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.255+0000 7ff117e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff1180cafa0 0x7ff1180cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54180/0 (socket says 172.21.15.62:54180) 2024-03-23T11:26:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.255+0000 7ff117e00640 1 -- 172.21.15.62:0/1343542939 learned_addr learned my addr 172.21.15.62:0/1343542939 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.255+0000 7ff117e00640 1 -- 172.21.15.62:0/1343542939 >> v1:172.21.15.120:6789/0 conn(0x7ff11806c6e0 legacy=0x7ff11806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.255+0000 7ff117e00640 1 -- 172.21.15.62:0/1343542939 >> v1:172.21.15.62:6789/0 conn(0x7ff11806c300 legacy=0x7ff1180ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.255+0000 7ff117e00640 1 -- 172.21.15.62:0/1343542939 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1180cc220 con 0x7ff1180cafa0 2024-03-23T11:26:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.256+0000 7ff117e00640 1 --2- 172.21.15.62:0/1343542939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1180cafa0 0x7ff1180cb430 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff104009870 tx=0x7ff10402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b582bbf25e011f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.256+0000 7ff117400640 1 -- 172.21.15.62:0/1343542939 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104030860 con 0x7ff1180cafa0 2024-03-23T11:26:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.257+0000 7ff117400640 1 -- 172.21.15.62:0/1343542939 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1040309c0 con 0x7ff1180cafa0 2024-03-23T11:26:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.257+0000 7ff117400640 1 -- 172.21.15.62:0/1343542939 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104030d50 con 0x7ff1180cafa0 2024-03-23T11:26:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.257+0000 7ff11f600640 1 -- 172.21.15.62:0/1343542939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1180cafa0 msgr2=0x7ff1180cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.257+0000 7ff11f600640 1 --2- 172.21.15.62:0/1343542939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1180cafa0 0x7ff1180cb430 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff104009870 tx=0x7ff10402b9a0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.257+0000 7ff11f600640 1 -- 172.21.15.62:0/1343542939 shutdown_connections 2024-03-23T11:26:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.257+0000 7ff11f600640 1 --2- 172.21.15.62:0/1343542939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1180cafa0 0x7ff1180cb430 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.257+0000 7ff11f600640 1 -- 172.21.15.62:0/1343542939 >> 172.21.15.62:0/1343542939 conn(0x7ff1180b02e0 msgr2=0x7ff1180b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.257+0000 7ff11f600640 1 -- 172.21.15.62:0/1343542939 shutdown_connections 2024-03-23T11:26:37.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.257+0000 7ff11f600640 1 -- 172.21.15.62:0/1343542939 wait complete. 2024-03-23T11:26:37.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.259+0000 7ff11f600640 1 Processor -- start 2024-03-23T11:26:37.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.259+0000 7ff11f600640 1 -- start start 2024-03-23T11:26:37.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.260+0000 7ff11f600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff11806c300 0x7ff1180b16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.260+0000 7ff11f600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff11806c6e0 0x7ff1180b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.260+0000 7ff11f600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1180cafa0 0x7ff1180abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.260+0000 7ff11f600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff118155690 con 0x7ff1180cafa0 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.260+0000 7ff11f600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180b1e10 con 0x7ff11806c6e0 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.260+0000 7ff11f600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180b23d0 con 0x7ff11806c300 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.260+0000 7ff11d600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1180cafa0 0x7ff1180abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff11d600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1180cafa0 0x7ff1180abb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54196/0 (socket says 172.21.15.62:54196) 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff11d600640 1 -- 172.21.15.62:0/1706704537 learned_addr learned my addr 172.21.15.62:0/1706704537 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff117e00640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff11806c6e0 0x7ff1180b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff11d600640 1 -- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff11806c300 msgr2=0x7ff1180b16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff11cc00640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff11806c300 0x7ff1180b16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff11d600640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff11806c300 0x7ff1180b16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff11d600640 1 -- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff11806c6e0 msgr2=0x7ff1180b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff11d600640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff11806c6e0 0x7ff1180b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff11d600640 1 -- 172.21.15.62:0/1706704537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1040094d0 con 0x7ff1180cafa0 2024-03-23T11:26:37.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff117e00640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff11806c6e0 0x7ff1180b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:37.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff11cc00640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff11806c300 0x7ff1180b16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:37.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.261+0000 7ff11d600640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1180cafa0 0x7ff1180abb90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7ff11000c780 tx=0x7ff11000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:37.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.262+0000 7ff114c00640 1 -- 172.21.15.62:0/1706704537 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff110004b60 con 0x7ff1180cafa0 2024-03-23T11:26:37.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.262+0000 7ff114c00640 1 -- 172.21.15.62:0/1706704537 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11000ce90 con 0x7ff1180cafa0 2024-03-23T11:26:37.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.262+0000 7ff114c00640 1 -- 172.21.15.62:0/1706704537 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11000a850 con 0x7ff1180cafa0 2024-03-23T11:26:37.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.262+0000 7ff11f600640 1 -- 172.21.15.62:0/1706704537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1181550d0 con 0x7ff1180cafa0 2024-03-23T11:26:37.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.262+0000 7ff11f600640 1 -- 172.21.15.62:0/1706704537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1180b2060 con 0x7ff1180cafa0 2024-03-23T11:26:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.263+0000 7ff11f600640 1 -- 172.21.15.62:0/1706704537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1180b2540 con 0x7ff1180cafa0 2024-03-23T11:26:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.264+0000 7ff114c00640 1 -- 172.21.15.62:0/1706704537 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff11000a9b0 con 0x7ff1180cafa0 2024-03-23T11:26:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.265+0000 7ff114c00640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0f806a7a0 0x7ff0f806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.265+0000 7ff114c00640 1 -- 172.21.15.62:0/1706704537 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff110089060 con 0x7ff1180cafa0 2024-03-23T11:26:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.266+0000 7ff11cc00640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0f806a7a0 0x7ff0f806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.267+0000 7ff11cc00640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0f806a7a0 0x7ff0f806cc60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff10c001680 tx=0x7ff10c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.269+0000 7ff114c00640 1 -- 172.21.15.62:0/1706704537 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff1100bc580 con 0x7ff1180cafa0 2024-03-23T11:26:37.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.482+0000 7ff11f600640 1 -- 172.21.15.62:0/1706704537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff1180b1c20 con 0x7ff1180cafa0 2024-03-23T11:26:37.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.483+0000 7ff114c00640 1 -- 172.21.15.62:0/1706704537 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v89) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff1100543e0 con 0x7ff1180cafa0 2024-03-23T11:26:37.477 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:26:37.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.485+0000 7ff11f600640 1 -- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0f806a7a0 msgr2=0x7ff0f806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.485+0000 7ff11f600640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0f806a7a0 0x7ff0f806cc60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff10c001680 tx=0x7ff10c009040 comp rx=0 tx=0).stop 2024-03-23T11:26:37.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.485+0000 7ff11f600640 1 -- 172.21.15.62:0/1706704537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1180cafa0 msgr2=0x7ff1180abb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.486+0000 7ff11f600640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1180cafa0 0x7ff1180abb90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7ff11000c780 tx=0x7ff11000cc40 comp rx=0 tx=0).stop 2024-03-23T11:26:37.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.486+0000 7ff11f600640 1 -- 172.21.15.62:0/1706704537 shutdown_connections 2024-03-23T11:26:37.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.486+0000 7ff11f600640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0f806a7a0 0x7ff0f806cc60 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.486+0000 7ff11f600640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1180cafa0 0x7ff1180abb90 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.486+0000 7ff11f600640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff11806c6e0 0x7ff1180b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.486+0000 7ff11f600640 1 --2- 172.21.15.62:0/1706704537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff11806c300 0x7ff1180b16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.486+0000 7ff11f600640 1 -- 172.21.15.62:0/1706704537 >> 172.21.15.62:0/1706704537 conn(0x7ff1180b02e0 msgr2=0x7ff11813d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:37.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.486+0000 7ff11f600640 1 -- 172.21.15.62:0/1706704537 shutdown_connections 2024-03-23T11:26:37.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.486+0000 7ff11f600640 1 -- 172.21.15.62:0/1706704537 wait complete. 2024-03-23T11:26:37.500 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:26:37.500 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:26:37.550 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:26:37.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.823+0000 7f3516e00640 1 Processor -- start 2024-03-23T11:26:37.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.823+0000 7f3516e00640 1 -- start start 2024-03-23T11:26:37.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.825+0000 7f3516e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f351006cc60 0x7f35100cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.825+0000 7f3516e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f35100ce420 0x7f35100b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.825+0000 7f3516e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35100ce8b0 con 0x7f3510159490 2024-03-23T11:26:37.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.825+0000 7f3516e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35100ced90 con 0x7f35100ce420 2024-03-23T11:26:37.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.825+0000 7f3516e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351006e3e0 con 0x7f351006cc60 2024-03-23T11:26:37.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.825+0000 7f3515a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3510159490 0x7f351015da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50916/0 (socket says 172.21.15.62:50916) 2024-03-23T11:26:37.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.825+0000 7f3515a00640 1 -- 172.21.15.62:0/1552958129 learned_addr learned my addr 172.21.15.62:0/1552958129 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:37.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f3515000640 1 --2- 172.21.15.62:0/1552958129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f351006cc60 0x7f35100cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f3516400640 1 --2- 172.21.15.62:0/1552958129 >> v2:172.21.15.120:3300/0 conn(0x7f35100ce420 0x7f35100b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142978206 0 0) 0x7f35100ce8b0 con 0x7f3510159490 2024-03-23T11:26:37.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f8003520 con 0x7f3510159490 2024-03-23T11:26:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3526128789 0 0) 0x7f34f8003520 con 0x7f3510159490 2024-03-23T11:26:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f351006cc60 msgr2=0x7f35100cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f350fe00640 1 --2- 172.21.15.62:0/1552958129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f351006cc60 0x7f35100cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 >> v2:172.21.15.120:3300/0 conn(0x7f35100ce420 msgr2=0x7f35100b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f350fe00640 1 --2- 172.21.15.62:0/1552958129 >> v2:172.21.15.120:3300/0 conn(0x7f35100ce420 0x7f35100b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351006c1c0 con 0x7f3510159490 2024-03-23T11:26:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f3504001670 con 0x7f3510159490 2024-03-23T11:26:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 >> v1:172.21.15.62:6789/0 conn(0x7f3510159490 legacy=0x7f351015da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.826+0000 7f350fe00640 1 --2- 172.21.15.62:0/1552958129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f8003970 0x7f34f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.827+0000 7f350fe00640 1 --2- 172.21.15.62:0/1552958129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f351006cc60 0x7f34f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.827+0000 7f350fe00640 1 --2- 172.21.15.62:0/1552958129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f8004d00 0x7f34f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.827+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35100ced90 con 0x7f34f8003970 2024-03-23T11:26:37.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.827+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351006e3e0 con 0x7f34f8004d00 2024-03-23T11:26:37.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.827+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f800ba60 con 0x7f351006cc60 2024-03-23T11:26:37.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.827+0000 7f3515a00640 1 --2- 172.21.15.62:0/1552958129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f8003970 0x7f34f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.827+0000 7f3515a00640 1 -- 172.21.15.62:0/1552958129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f351006cc60 msgr2=0x7f34f80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:37.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.827+0000 7f3515a00640 1 --2- 172.21.15.62:0/1552958129 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f351006cc60 0x7f34f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.827+0000 7f3515a00640 1 -- 172.21.15.62:0/1552958129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f8004d00 msgr2=0x7f34f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:37.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.827+0000 7f3515a00640 1 --2- 172.21.15.62:0/1552958129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f8004d00 0x7f34f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.827+0000 7f3515a00640 1 -- 172.21.15.62:0/1552958129 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f8003520 con 0x7f34f8003970 2024-03-23T11:26:37.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.828+0000 7f3515a00640 1 --2- 172.21.15.62:0/1552958129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f8003970 0x7f34f8003d90 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f350400c0e0 tx=0x7f34f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f5d26077b6ee18 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:37.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.829+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3504005ad0 con 0x7f34f8003970 2024-03-23T11:26:37.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.829+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3504005c50 con 0x7f34f8003970 2024-03-23T11:26:37.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.829+0000 7f350fe00640 1 -- 172.21.15.62:0/1552958129 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350402c810 con 0x7f34f8003970 2024-03-23T11:26:37.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.829+0000 7f3516e00640 1 -- 172.21.15.62:0/1552958129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f8003970 msgr2=0x7f34f8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.829+0000 7f3516e00640 1 --2- 172.21.15.62:0/1552958129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f8003970 0x7f34f8003d90 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f350400c0e0 tx=0x7f34f80057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.829+0000 7f3516400640 1 -- 172.21.15.62:0/1552958129 reap_dead start 2024-03-23T11:26:37.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.829+0000 7f3516e00640 1 -- 172.21.15.62:0/1552958129 shutdown_connections 2024-03-23T11:26:37.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.829+0000 7f3516e00640 1 -- 172.21.15.62:0/1552958129 >> 172.21.15.62:0/1552958129 conn(0x7f3510066050 msgr2=0x7f3510069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:37.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.829+0000 7f3516e00640 1 -- 172.21.15.62:0/1552958129 shutdown_connections 2024-03-23T11:26:37.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.829+0000 7f3516e00640 1 -- 172.21.15.62:0/1552958129 wait complete. 2024-03-23T11:26:37.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.831+0000 7f3516e00640 1 Processor -- start 2024-03-23T11:26:37.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.831+0000 7f3516e00640 1 -- start start 2024-03-23T11:26:37.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.832+0000 7f3516e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f8003970 0x7f35100a8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.832+0000 7f3516e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3510146a10 0x7f3510144920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.832+0000 7f3516e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3510145040 0x7f35100ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.832+0000 7f3516e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3510146e70 con 0x7f3510145040 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.832+0000 7f3516e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3510145970 con 0x7f34f8003970 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.832+0000 7f3516e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3510145c50 con 0x7f3510146a10 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.833+0000 7f3516400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3510145040 0x7f35100ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.833+0000 7f3516400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3510145040 0x7f35100ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44200/0 (socket says 172.21.15.62:44200) 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.833+0000 7f3516400640 1 -- 172.21.15.62:0/3999447244 learned_addr learned my addr 172.21.15.62:0/3999447244 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.833+0000 7f3515000640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3510146a10 0x7f3510144920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.833+0000 7f3515a00640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f8003970 0x7f35100a8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.833+0000 7f3516400640 1 -- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3510146a10 msgr2=0x7f3510144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.833+0000 7f3516400640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3510146a10 0x7f3510144920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.833+0000 7f3516400640 1 -- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f8003970 msgr2=0x7f35100a8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.833+0000 7f3516400640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f8003970 0x7f35100a8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.833+0000 7f3516400640 1 -- 172.21.15.62:0/3999447244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3510145f30 con 0x7f3510145040 2024-03-23T11:26:37.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.833+0000 7f3515a00640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f8003970 0x7f35100a8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:37.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.834+0000 7f3516400640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3510145040 0x7f35100ccba0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f351015cfe0 tx=0x7f350800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:37.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.834+0000 7f350d600640 1 -- 172.21.15.62:0/3999447244 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508010040 con 0x7f3510145040 2024-03-23T11:26:37.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.834+0000 7f350d600640 1 -- 172.21.15.62:0/3999447244 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350800bae0 con 0x7f3510145040 2024-03-23T11:26:37.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.834+0000 7f3516e00640 1 -- 172.21.15.62:0/3999447244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3510146210 con 0x7f3510145040 2024-03-23T11:26:37.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.834+0000 7f350d600640 1 -- 172.21.15.62:0/3999447244 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350800be70 con 0x7f3510145040 2024-03-23T11:26:37.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.834+0000 7f3516e00640 1 -- 172.21.15.62:0/3999447244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35101454c0 con 0x7f3510145040 2024-03-23T11:26:37.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.835+0000 7f3516e00640 1 -- 172.21.15.62:0/3999447244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3510146730 con 0x7f3510145040 2024-03-23T11:26:37.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.836+0000 7f350d600640 1 -- 172.21.15.62:0/3999447244 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f350801e410 con 0x7f3510145040 2024-03-23T11:26:37.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.837+0000 7f350d600640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34f006a7d0 0x7f34f006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:37.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.837+0000 7f350d600640 1 -- 172.21.15.62:0/3999447244 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3508089090 con 0x7f3510145040 2024-03-23T11:26:37.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.838+0000 7f3515a00640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34f006a7d0 0x7f34f006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:37.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.839+0000 7f3515a00640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34f006a7d0 0x7f34f006cc90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3504034000 tx=0x7f3504007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:37.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:37.841+0000 7f350d600640 1 -- 172.21.15.62:0/3999447244 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3510146730 con 0x7f3510145040 2024-03-23T11:26:38.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.040+0000 7f3516e00640 1 -- 172.21.15.62:0/3999447244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3510146730 con 0x7f3510145040 2024-03-23T11:26:38.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.043+0000 7f350d600640 1 -- 172.21.15.62:0/3999447244 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16092 (secure 0 0 0) 0x7f3510146730 con 0x7f3510145040 2024-03-23T11:26:38.038 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:38.038 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":89,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:26:26.134957+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:26:22.794149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-23T11:26:24.150997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:26:38.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.046+0000 7f3516e00640 1 -- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34f006a7d0 msgr2=0x7f34f006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.046+0000 7f3516e00640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34f006a7d0 0x7f34f006cc90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3504034000 tx=0x7f3504007000 comp rx=0 tx=0).stop 2024-03-23T11:26:38.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.046+0000 7f3516e00640 1 -- 172.21.15.62:0/3999447244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3510145040 msgr2=0x7f35100ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.046+0000 7f3516e00640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3510145040 0x7f35100ccba0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f351015cfe0 tx=0x7f350800b480 comp rx=0 tx=0).stop 2024-03-23T11:26:38.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.046+0000 7f3516e00640 1 -- 172.21.15.62:0/3999447244 shutdown_connections 2024-03-23T11:26:38.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.046+0000 7f3516e00640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f34f006a7d0 0x7f34f006cc90 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.046+0000 7f3516e00640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3510145040 0x7f35100ccba0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.046+0000 7f3516e00640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3510146a10 0x7f3510144920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.046+0000 7f3516e00640 1 --2- 172.21.15.62:0/3999447244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34f8003970 0x7f35100a8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.046+0000 7f3516e00640 1 -- 172.21.15.62:0/3999447244 >> 172.21.15.62:0/3999447244 conn(0x7f3510066050 msgr2=0x7f35100af850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:38.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.047+0000 7f3516e00640 1 -- 172.21.15.62:0/3999447244 shutdown_connections 2024-03-23T11:26:38.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.047+0000 7f3516e00640 1 -- 172.21.15.62:0/3999447244 wait complete. 2024-03-23T11:26:38.059 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:26:22.794149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:26:24.150997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:26:38.060 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:26:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.288+0000 7fc3bfe00640 1 Processor -- start 2024-03-23T11:26:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.288+0000 7fc3bfe00640 1 -- start start 2024-03-23T11:26:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.290+0000 7fc3bfe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc3c0159490 0x7fc3c006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.290+0000 7fc3bfe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc3c00ce5f0 0x7fc3c00b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.290+0000 7fc3bfe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c00ced90 con 0x7fc3c015da90 2024-03-23T11:26:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.290+0000 7fc3bfe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3c006e3e0 con 0x7fc3c0159490 2024-03-23T11:26:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.290+0000 7fc3bfe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3c006c1c0 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.290+0000 7fc3bea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc3c0159490 0x7fc3c006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.290+0000 7fc3be000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc3c015da90 0x7fc3c00cdee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50918/0 (socket says 172.21.15.62:50918) 2024-03-23T11:26:38.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.290+0000 7fc3be000640 1 -- 172.21.15.62:0/1367266383 learned_addr learned my addr 172.21.15.62:0/1367266383 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.290+0000 7fc3bf400640 1 --2- 172.21.15.62:0/1367266383 >> v2:172.21.15.120:3300/0 conn(0x7fc3c00ce5f0 0x7fc3c00b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.291+0000 7fc3bd600640 1 -- 172.21.15.62:0/1367266383 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399771904 0 0) 0x7fc3c00ced90 con 0x7fc3c015da90 2024-03-23T11:26:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.291+0000 7fc3bd600640 1 -- 172.21.15.62:0/1367266383 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a4003520 con 0x7fc3c015da90 2024-03-23T11:26:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.291+0000 7fc3bea00640 1 -- 172.21.15.62:0/1367266383 >> v2:172.21.15.120:3300/0 conn(0x7fc3c00ce5f0 msgr2=0x7fc3c00b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:38.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.291+0000 7fc3bea00640 1 --2- 172.21.15.62:0/1367266383 >> v2:172.21.15.120:3300/0 conn(0x7fc3c00ce5f0 0x7fc3c00b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.291+0000 7fc3bea00640 1 -- 172.21.15.62:0/1367266383 >> v1:172.21.15.62:6789/0 conn(0x7fc3c015da90 legacy=0x7fc3c00cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.291+0000 7fc3bea00640 1 -- 172.21.15.62:0/1367266383 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c006e100 con 0x7fc3c0159490 2024-03-23T11:26:38.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.292+0000 7fc3bea00640 1 --2- 172.21.15.62:0/1367266383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c0159490 0x7fc3c006cc60 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00ad990 tx=0x7fc3b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856f2bbaf585a978 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:38.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.292+0000 7fc3bd600640 1 -- 172.21.15.62:0/1367266383 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b002e070 con 0x7fc3c0159490 2024-03-23T11:26:38.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.292+0000 7fc3bd600640 1 -- 172.21.15.62:0/1367266383 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b00315a0 con 0x7fc3c0159490 2024-03-23T11:26:38.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.292+0000 7fc3bd600640 1 -- 172.21.15.62:0/1367266383 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0031930 con 0x7fc3c0159490 2024-03-23T11:26:38.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.293+0000 7fc3bfe00640 1 -- 172.21.15.62:0/1367266383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c0159490 msgr2=0x7fc3c006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.293+0000 7fc3bfe00640 1 --2- 172.21.15.62:0/1367266383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c0159490 0x7fc3c006cc60 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00ad990 tx=0x7fc3b002b840 comp rx=0 tx=0).stop 2024-03-23T11:26:38.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.293+0000 7fc3bfe00640 1 -- 172.21.15.62:0/1367266383 shutdown_connections 2024-03-23T11:26:38.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.293+0000 7fc3bfe00640 1 --2- 172.21.15.62:0/1367266383 >> v2:172.21.15.120:3300/0 conn(0x7fc3c00ce5f0 0x7fc3c00b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.293+0000 7fc3bfe00640 1 --2- 172.21.15.62:0/1367266383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c0159490 0x7fc3c006cc60 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.293+0000 7fc3bfe00640 1 -- 172.21.15.62:0/1367266383 >> 172.21.15.62:0/1367266383 conn(0x7fc3c0066050 msgr2=0x7fc3c0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:38.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.293+0000 7fc3bfe00640 1 -- 172.21.15.62:0/1367266383 shutdown_connections 2024-03-23T11:26:38.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.293+0000 7fc3bfe00640 1 -- 172.21.15.62:0/1367266383 wait complete. 2024-03-23T11:26:38.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.295+0000 7fc3bfe00640 1 Processor -- start 2024-03-23T11:26:38.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.295+0000 7fc3bfe00640 1 -- start start 2024-03-23T11:26:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.296+0000 7fc3bfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c00ce5f0 0x7fc3c0143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.296+0000 7fc3bfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c0159490 0x7fc3c0143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.296+0000 7fc3bfe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c015da90 0x7fc3c00a9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.296+0000 7fc3bfe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c0156290 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.296+0000 7fc3bfe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c0155980 con 0x7fc3c015da90 2024-03-23T11:26:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.296+0000 7fc3bfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c01533b0 con 0x7fc3c0159490 2024-03-23T11:26:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.296+0000 7fc3bea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c00ce5f0 0x7fc3c0143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.296+0000 7fc3bea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c00ce5f0 0x7fc3c0143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44208/0 (socket says 172.21.15.62:44208) 2024-03-23T11:26:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.296+0000 7fc3bea00640 1 -- 172.21.15.62:0/236372999 learned_addr learned my addr 172.21.15.62:0/236372999 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.297+0000 7fc3bf400640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c015da90 0x7fc3c00a9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.297+0000 7fc3bea00640 1 -- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c0159490 msgr2=0x7fc3c0143fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.297+0000 7fc3bea00640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c0159490 0x7fc3c0143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.297+0000 7fc3bea00640 1 -- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c015da90 msgr2=0x7fc3c00a9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.297+0000 7fc3bea00640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c015da90 0x7fc3c00a9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.297+0000 7fc3bea00640 1 -- 172.21.15.62:0/236372999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b0009400 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.297+0000 7fc3bf400640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c015da90 0x7fc3c00a9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.297+0000 7fc3bea00640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c00ce5f0 0x7fc3c0143910 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0009890 tx=0x7fc3b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.298+0000 7fc3b7400640 1 -- 172.21.15.62:0/236372999 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0032840 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.298+0000 7fc3b7400640 1 -- 172.21.15.62:0/236372999 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b0031f00 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.298+0000 7fc3b7400640 1 -- 172.21.15.62:0/236372999 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0032be0 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.298+0000 7fc3bfe00640 1 -- 172.21.15.62:0/236372999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c00ad300 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.298+0000 7fc3bfe00640 1 -- 172.21.15.62:0/236372999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c0155eb0 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.299+0000 7fc3bfe00640 1 -- 172.21.15.62:0/236372999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c0066bb0 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.300+0000 7fc3b7400640 1 -- 172.21.15.62:0/236372999 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc3b0032d40 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.301+0000 7fc3b7400640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc3a006a7a0 0x7fc3a006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.302+0000 7fc3b7400640 1 -- 172.21.15.62:0/236372999 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3b00aac30 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.302+0000 7fc3be000640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc3a006a7a0 0x7fc3a006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:38.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.303+0000 7fc3be000640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc3a006a7a0 0x7fc3a006cc60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00ad690 tx=0x7fc3ac007930 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:38.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.306+0000 7fc3b7400640 1 -- 172.21.15.62:0/236372999 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc3b00b1080 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.509+0000 7fc3bfe00640 1 -- 172.21.15.62:0/236372999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc3c0155370 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.510+0000 7fc3b7400640 1 -- 172.21.15.62:0/236372999 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc3b00761e0 con 0x7fc3c00ce5f0 2024-03-23T11:26:38.504 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:26:38.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 -- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc3a006a7a0 msgr2=0x7fc3a006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc3a006a7a0 0x7fc3a006cc60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00ad690 tx=0x7fc3ac007930 comp rx=0 tx=0).stop 2024-03-23T11:26:38.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 -- 172.21.15.62:0/236372999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c00ce5f0 msgr2=0x7fc3c0143910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c00ce5f0 0x7fc3c0143910 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0009890 tx=0x7fc3b00046c0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 -- 172.21.15.62:0/236372999 shutdown_connections 2024-03-23T11:26:38.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc3a006a7a0 0x7fc3a006cc60 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c015da90 0x7fc3c00a9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c0159490 0x7fc3c0143fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 --2- 172.21.15.62:0/236372999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3c00ce5f0 0x7fc3c0143910 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 -- 172.21.15.62:0/236372999 >> 172.21.15.62:0/236372999 conn(0x7fc3c0066050 msgr2=0x7fc3c0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:38.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 -- 172.21.15.62:0/236372999 shutdown_connections 2024-03-23T11:26:38.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.513+0000 7fc3bfe00640 1 -- 172.21.15.62:0/236372999 wait complete. 2024-03-23T11:26:38.526 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:26:38.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.754+0000 7f4173000640 1 Processor -- start 2024-03-23T11:26:38.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.755+0000 7f4173000640 1 -- start start 2024-03-23T11:26:38.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.757+0000 7f4173000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f416c0cda20 0x7f416c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.757+0000 7f4173000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f416c0cde90 0x7f416c0ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.757+0000 7f4173000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c0ce820 con 0x7f416c160c00 2024-03-23T11:26:38.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.757+0000 7f4173000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416c0b1ad0 con 0x7f416c0cda20 2024-03-23T11:26:38.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.757+0000 7f4173000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416c160fe0 con 0x7f416c0cde90 2024-03-23T11:26:38.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.757+0000 7f4171200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f416c0cda20 0x7f416c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:38.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.757+0000 7f4171200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f416c0cda20 0x7f416c06d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44210/0 (socket says 172.21.15.62:44210) 2024-03-23T11:26:38.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.757+0000 7f4171200640 1 -- 172.21.15.62:0/4030834700 learned_addr learned my addr 172.21.15.62:0/4030834700 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:38.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.757+0000 7f4172600640 1 --2- 172.21.15.62:0/4030834700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f416c0cde90 0x7f416c0ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:38.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.758+0000 7f4171200640 1 -- 172.21.15.62:0/4030834700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f416c0cde90 msgr2=0x7f416c0ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:38.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.758+0000 7f4171200640 1 --2- 172.21.15.62:0/4030834700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f416c0cde90 0x7f416c0ce2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.758+0000 7f4171200640 1 -- 172.21.15.62:0/4030834700 >> v1:172.21.15.120:6789/0 conn(0x7f416c160c00 legacy=0x7f416c06de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:38.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.758+0000 7f4171200640 1 -- 172.21.15.62:0/4030834700 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c0b1510 con 0x7f416c0cda20 2024-03-23T11:26:38.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.758+0000 7f4172600640 1 --2- 172.21.15.62:0/4030834700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f416c0cde90 0x7f416c0ce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:38.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.759+0000 7f4171200640 1 --2- 172.21.15.62:0/4030834700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f416c0cda20 0x7f416c06d530 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f4158009870 tx=0x7f415802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31cd0bb0524168e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:38.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.759+0000 7f416be00640 1 -- 172.21.15.62:0/4030834700 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158031640 con 0x7f416c0cda20 2024-03-23T11:26:38.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.759+0000 7f416be00640 1 -- 172.21.15.62:0/4030834700 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41580317a0 con 0x7f416c0cda20 2024-03-23T11:26:38.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.759+0000 7f416be00640 1 -- 172.21.15.62:0/4030834700 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158031b30 con 0x7f416c0cda20 2024-03-23T11:26:38.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.760+0000 7f4173000640 1 -- 172.21.15.62:0/4030834700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f416c0cda20 msgr2=0x7f416c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.760+0000 7f4173000640 1 --2- 172.21.15.62:0/4030834700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f416c0cda20 0x7f416c06d530 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f4158009870 tx=0x7f415802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.760+0000 7f4173000640 1 -- 172.21.15.62:0/4030834700 shutdown_connections 2024-03-23T11:26:38.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.760+0000 7f4173000640 1 --2- 172.21.15.62:0/4030834700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f416c0cde90 0x7f416c0ce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.760+0000 7f4173000640 1 --2- 172.21.15.62:0/4030834700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f416c0cda20 0x7f416c06d530 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.760+0000 7f4173000640 1 -- 172.21.15.62:0/4030834700 >> 172.21.15.62:0/4030834700 conn(0x7f416c066050 msgr2=0x7f416c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:38.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.760+0000 7f4173000640 1 -- 172.21.15.62:0/4030834700 shutdown_connections 2024-03-23T11:26:38.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.760+0000 7f4173000640 1 -- 172.21.15.62:0/4030834700 wait complete. 2024-03-23T11:26:38.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.762+0000 7f4173000640 1 Processor -- start 2024-03-23T11:26:38.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.762+0000 7f4173000640 1 -- start start 2024-03-23T11:26:38.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.763+0000 7f4173000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f416c0cda20 0x7f416c1526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.763+0000 7f4173000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f416c0cde90 0x7f416c15bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.763+0000 7f4173000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f416c160c00 0x7f416c0aa130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.763+0000 7f4173000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c0a87f0 con 0x7f416c0cda20 2024-03-23T11:26:38.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.763+0000 7f4173000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c0a7900 con 0x7f416c160c00 2024-03-23T11:26:38.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.763+0000 7f4173000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416c0a7be0 con 0x7f416c0cde90 2024-03-23T11:26:38.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.763+0000 7f4171c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f416c0cda20 0x7f416c1526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:38.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.763+0000 7f4171c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f416c0cda20 0x7f416c1526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44218/0 (socket says 172.21.15.62:44218) 2024-03-23T11:26:38.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.763+0000 7f4171c00640 1 -- 172.21.15.62:0/254981354 learned_addr learned my addr 172.21.15.62:0/254981354 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:38.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.763+0000 7f4171200640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f416c0cde90 0x7f416c15bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:38.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.763+0000 7f4172600640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f416c160c00 0x7f416c0aa130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:38.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.764+0000 7f4171c00640 1 -- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f416c0cde90 msgr2=0x7f416c15bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.764+0000 7f4171c00640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f416c0cde90 0x7f416c15bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.764+0000 7f4171c00640 1 -- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f416c160c00 msgr2=0x7f416c0aa130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.764+0000 7f4171c00640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f416c160c00 0x7f416c0aa130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.764+0000 7f4171c00640 1 -- 172.21.15.62:0/254981354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41580094d0 con 0x7f416c0cda20 2024-03-23T11:26:38.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.764+0000 7f4171c00640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f416c0cda20 0x7f416c1526d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f416000b3f0 tx=0x7f416000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:38.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.764+0000 7f4169600640 1 -- 172.21.15.62:0/254981354 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4160011830 con 0x7f416c0cda20 2024-03-23T11:26:38.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.764+0000 7f4169600640 1 -- 172.21.15.62:0/254981354 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4160011990 con 0x7f416c0cda20 2024-03-23T11:26:38.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.764+0000 7f4169600640 1 -- 172.21.15.62:0/254981354 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4160011d40 con 0x7f416c0cda20 2024-03-23T11:26:38.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.764+0000 7f4173000640 1 -- 172.21.15.62:0/254981354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c0a7ec0 con 0x7f416c0cda20 2024-03-23T11:26:38.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.764+0000 7f4173000640 1 -- 172.21.15.62:0/254981354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c0a8480 con 0x7f416c0cda20 2024-03-23T11:26:38.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.766+0000 7f4173000640 1 -- 172.21.15.62:0/254981354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416c066bb0 con 0x7f416c0cda20 2024-03-23T11:26:38.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.767+0000 7f4169600640 1 -- 172.21.15.62:0/254981354 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f416001a400 con 0x7f416c0cda20 2024-03-23T11:26:38.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.768+0000 7f4169600640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f414c06a7d0 0x7f414c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:38.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.768+0000 7f4169600640 1 -- 172.21.15.62:0/254981354 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4160088b50 con 0x7f416c0cda20 2024-03-23T11:26:38.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.769+0000 7f4171200640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f414c06a7d0 0x7f414c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:38.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.770+0000 7f4171200640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f414c06a7d0 0x7f414c06cc90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4158036000 tx=0x7f415803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:38.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.772+0000 7f4169600640 1 -- 172.21.15.62:0/254981354 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4160054000 con 0x7f416c0cda20 2024-03-23T11:26:38.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.969+0000 7f4173000640 1 -- 172.21.15.62:0/254981354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f416c0a76e0 con 0x7f416c0cda20 2024-03-23T11:26:38.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.970+0000 7f4169600640 1 -- 172.21.15.62:0/254981354 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4160015020 con 0x7f416c0cda20 2024-03-23T11:26:38.964 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:26:38.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 -- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f414c06a7d0 msgr2=0x7f414c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f414c06a7d0 0x7f414c06cc90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4158036000 tx=0x7f415803c040 comp rx=0 tx=0).stop 2024-03-23T11:26:38.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 -- 172.21.15.62:0/254981354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f416c0cda20 msgr2=0x7f416c1526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:38.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f416c0cda20 0x7f416c1526d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f416000b3f0 tx=0x7f416000b8b0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 -- 172.21.15.62:0/254981354 shutdown_connections 2024-03-23T11:26:38.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f414c06a7d0 0x7f414c06cc90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f416c160c00 0x7f416c0aa130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f416c0cde90 0x7f416c15bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 --2- 172.21.15.62:0/254981354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f416c0cda20 0x7f416c1526d0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:38.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 -- 172.21.15.62:0/254981354 >> 172.21.15.62:0/254981354 conn(0x7f416c066050 msgr2=0x7f416c070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:38.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 -- 172.21.15.62:0/254981354 shutdown_connections 2024-03-23T11:26:38.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:38.973+0000 7f4173000640 1 -- 172.21.15.62:0/254981354 wait complete. 2024-03-23T11:26:38.987 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:26:39.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.208+0000 7fa389200640 1 Processor -- start 2024-03-23T11:26:39.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.209+0000 7fa389200640 1 -- start start 2024-03-23T11:26:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.210+0000 7fa389200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa384143dd0 0x7fa38415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.210+0000 7fa389200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa384070090 0x7fa3840704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.210+0000 7fa389200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa3840b2820 0x7fa3840b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa389200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa384070a30 con 0x7fa384070090 2024-03-23T11:26:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa389200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3840cd140 con 0x7fa3840b2820 2024-03-23T11:26:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa389200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3840b3180 con 0x7fa384143dd0 2024-03-23T11:26:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa37be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa384070090 0x7fa3840704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa37be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa384070090 0x7fa3840704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44222/0 (socket says 172.21.15.62:44222) 2024-03-23T11:26:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa37be00640 1 -- 172.21.15.62:0/3393783636 learned_addr learned my addr 172.21.15.62:0/3393783636 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:39.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa383e00640 1 --2- 172.21.15.62:0/3393783636 >> v2:172.21.15.120:3300/0 conn(0x7fa3840b2820 0x7fa3840b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:39.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa383400640 1 --2- 172.21.15.62:0/3393783636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa384143dd0 0x7fa38415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:39.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa37be00640 1 -- 172.21.15.62:0/3393783636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa384143dd0 msgr2=0x7fa38415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:39.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa37be00640 1 --2- 172.21.15.62:0/3393783636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa384143dd0 0x7fa38415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa37be00640 1 -- 172.21.15.62:0/3393783636 >> v2:172.21.15.120:3300/0 conn(0x7fa3840b2820 msgr2=0x7fa3840b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:39.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa37be00640 1 --2- 172.21.15.62:0/3393783636 >> v2:172.21.15.120:3300/0 conn(0x7fa3840b2820 0x7fa3840b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa37be00640 1 -- 172.21.15.62:0/3393783636 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3840cd3e0 con 0x7fa384070090 2024-03-23T11:26:39.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa383400640 1 --2- 172.21.15.62:0/3393783636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa384143dd0 0x7fa38415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:39.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.211+0000 7fa383e00640 1 --2- 172.21.15.62:0/3393783636 >> v2:172.21.15.120:3300/0 conn(0x7fa3840b2820 0x7fa3840b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:39.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.213+0000 7fa37be00640 1 --2- 172.21.15.62:0/3393783636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa384070090 0x7fa3840704f0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fa384071500 tx=0x7fa36c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efef1daab7509b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.213+0000 7fa382a00640 1 -- 172.21.15.62:0/3393783636 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c031880 con 0x7fa384070090 2024-03-23T11:26:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.213+0000 7fa382a00640 1 -- 172.21.15.62:0/3393783636 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c0319e0 con 0x7fa384070090 2024-03-23T11:26:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.213+0000 7fa382a00640 1 -- 172.21.15.62:0/3393783636 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c031d70 con 0x7fa384070090 2024-03-23T11:26:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.214+0000 7fa389200640 1 -- 172.21.15.62:0/3393783636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa384070090 msgr2=0x7fa3840704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.214+0000 7fa389200640 1 --2- 172.21.15.62:0/3393783636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa384070090 0x7fa3840704f0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fa384071500 tx=0x7fa36c02b920 comp rx=0 tx=0).stop 2024-03-23T11:26:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.214+0000 7fa389200640 1 -- 172.21.15.62:0/3393783636 shutdown_connections 2024-03-23T11:26:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.214+0000 7fa389200640 1 --2- 172.21.15.62:0/3393783636 >> v2:172.21.15.120:3300/0 conn(0x7fa3840b2820 0x7fa3840b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.214+0000 7fa389200640 1 --2- 172.21.15.62:0/3393783636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa384070090 0x7fa3840704f0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.214+0000 7fa389200640 1 --2- 172.21.15.62:0/3393783636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa384143dd0 0x7fa38415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.214+0000 7fa389200640 1 -- 172.21.15.62:0/3393783636 >> 172.21.15.62:0/3393783636 conn(0x7fa384066050 msgr2=0x7fa384069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.214+0000 7fa389200640 1 -- 172.21.15.62:0/3393783636 shutdown_connections 2024-03-23T11:26:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.214+0000 7fa389200640 1 -- 172.21.15.62:0/3393783636 wait complete. 2024-03-23T11:26:39.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.216+0000 7fa389200640 1 Processor -- start 2024-03-23T11:26:39.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.216+0000 7fa389200640 1 -- start start 2024-03-23T11:26:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.217+0000 7fa389200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa384070090 0x7fa38413a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.217+0000 7fa389200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3840b2820 0x7fa3840ac7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.217+0000 7fa389200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa384143dd0 0x7fa3840ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.217+0000 7fa389200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3841f2760 con 0x7fa384143dd0 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.217+0000 7fa389200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3840ac1c0 con 0x7fa3840b2820 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.217+0000 7fa389200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38413dc00 con 0x7fa384070090 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.217+0000 7fa383e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa384143dd0 0x7fa3840ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.217+0000 7fa383e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa384143dd0 0x7fa3840ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44236/0 (socket says 172.21.15.62:44236) 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.217+0000 7fa383e00640 1 -- 172.21.15.62:0/1444714266 learned_addr learned my addr 172.21.15.62:0/1444714266 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.217+0000 7fa383400640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa384070090 0x7fa38413a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.217+0000 7fa37be00640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3840b2820 0x7fa3840ac7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.218+0000 7fa383e00640 1 -- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa384070090 msgr2=0x7fa38413a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.218+0000 7fa383e00640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa384070090 0x7fa38413a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.218+0000 7fa383e00640 1 -- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3840b2820 msgr2=0x7fa3840ac7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.218+0000 7fa383e00640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3840b2820 0x7fa3840ac7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.218+0000 7fa383e00640 1 -- 172.21.15.62:0/1444714266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c0094d0 con 0x7fa384143dd0 2024-03-23T11:26:39.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.218+0000 7fa383e00640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa384143dd0 0x7fa3840ab840 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fa37c00cef0 tx=0x7fa37c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:39.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.218+0000 7fa37b400640 1 -- 172.21.15.62:0/1444714266 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37c00d840 con 0x7fa384143dd0 2024-03-23T11:26:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.218+0000 7fa37b400640 1 -- 172.21.15.62:0/1444714266 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37c00d9c0 con 0x7fa384143dd0 2024-03-23T11:26:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.218+0000 7fa37b400640 1 -- 172.21.15.62:0/1444714266 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37c00dd70 con 0x7fa384143dd0 2024-03-23T11:26:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.218+0000 7fa389200640 1 -- 172.21.15.62:0/1444714266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38413dee0 con 0x7fa384143dd0 2024-03-23T11:26:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.219+0000 7fa389200640 1 -- 172.21.15.62:0/1444714266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3841f23e0 con 0x7fa384143dd0 2024-03-23T11:26:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.220+0000 7fa389200640 1 -- 172.21.15.62:0/1444714266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3841f28d0 con 0x7fa384143dd0 2024-03-23T11:26:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.220+0000 7fa37b400640 1 -- 172.21.15.62:0/1444714266 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa37c01e3b0 con 0x7fa384143dd0 2024-03-23T11:26:39.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.221+0000 7fa37b400640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36006a7a0 0x7fa36006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.222+0000 7fa37b400640 1 -- 172.21.15.62:0/1444714266 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa37c089880 con 0x7fa384143dd0 2024-03-23T11:26:39.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.222+0000 7fa383400640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36006a7a0 0x7fa36006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:39.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.223+0000 7fa383400640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36006a7a0 0x7fa36006cc60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa3841433d0 tx=0x7fa374008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:39.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.226+0000 7fa37b400640 1 -- 172.21.15.62:0/1444714266 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa37c054d50 con 0x7fa384143dd0 2024-03-23T11:26:39.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.427+0000 7fa389200640 1 -- 172.21.15.62:0/1444714266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa384066bb0 con 0x7fa384143dd0 2024-03-23T11:26:39.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.428+0000 7fa37b400640 1 -- 172.21.15.62:0/1444714266 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa37c01a0a0 con 0x7fa384143dd0 2024-03-23T11:26:39.423 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:26:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.431+0000 7fa389200640 1 -- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36006a7a0 msgr2=0x7fa36006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.431+0000 7fa389200640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36006a7a0 0x7fa36006cc60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa3841433d0 tx=0x7fa374008040 comp rx=0 tx=0).stop 2024-03-23T11:26:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.431+0000 7fa389200640 1 -- 172.21.15.62:0/1444714266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa384143dd0 msgr2=0x7fa3840ab840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.431+0000 7fa389200640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa384143dd0 0x7fa3840ab840 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fa37c00cef0 tx=0x7fa37c00a710 comp rx=0 tx=0).stop 2024-03-23T11:26:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.431+0000 7fa389200640 1 -- 172.21.15.62:0/1444714266 shutdown_connections 2024-03-23T11:26:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.431+0000 7fa389200640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa36006a7a0 0x7fa36006cc60 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.431+0000 7fa389200640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa384143dd0 0x7fa3840ab840 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.431+0000 7fa389200640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3840b2820 0x7fa3840ac7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.431+0000 7fa389200640 1 --2- 172.21.15.62:0/1444714266 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa384070090 0x7fa38413a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.431+0000 7fa389200640 1 -- 172.21.15.62:0/1444714266 >> 172.21.15.62:0/1444714266 conn(0x7fa384066050 msgr2=0x7fa3840cb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:39.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.432+0000 7fa389200640 1 -- 172.21.15.62:0/1444714266 shutdown_connections 2024-03-23T11:26:39.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.432+0000 7fa389200640 1 -- 172.21.15.62:0/1444714266 wait complete. 2024-03-23T11:26:39.445 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:26:39.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.677+0000 7fd62a200640 1 Processor -- start 2024-03-23T11:26:39.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.677+0000 7fd62a200640 1 -- start start 2024-03-23T11:26:39.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd62a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd62415c960 0x7fd6240cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd62a200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6240cd260 0x7fd6240d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd62a200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6240d11e0 0x7fd6240b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd62a200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6240d1650 con 0x7fd6240d11e0 2024-03-23T11:26:39.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd62a200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6240d0810 con 0x7fd6240cd260 2024-03-23T11:26:39.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd62a200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6240cd9d0 con 0x7fd62415c960 2024-03-23T11:26:39.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd629800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6240d11e0 0x7fd6240b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:39.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd629800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6240d11e0 0x7fd6240b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44242/0 (socket says 172.21.15.62:44242) 2024-03-23T11:26:39.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd629800640 1 -- 172.21.15.62:0/2772730786 learned_addr learned my addr 172.21.15.62:0/2772730786 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:39.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd629800640 1 -- 172.21.15.62:0/2772730786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd62415c960 msgr2=0x7fd6240cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:39.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd629800640 1 --2- 172.21.15.62:0/2772730786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd62415c960 0x7fd6240cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd629800640 1 -- 172.21.15.62:0/2772730786 >> v2:172.21.15.120:3300/0 conn(0x7fd6240cd260 msgr2=0x7fd6240d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:39.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd629800640 1 --2- 172.21.15.62:0/2772730786 >> v2:172.21.15.120:3300/0 conn(0x7fd6240cd260 0x7fd6240d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.679+0000 7fd629800640 1 -- 172.21.15.62:0/2772730786 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6240cdcb0 con 0x7fd6240d11e0 2024-03-23T11:26:39.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.681+0000 7fd629800640 1 --2- 172.21.15.62:0/2772730786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6240d11e0 0x7fd6240b3310 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fd624066520 tx=0x7fd61c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1e830269ddebbb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:39.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.681+0000 7fd623e00640 1 -- 172.21.15.62:0/2772730786 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c02cb00 con 0x7fd6240d11e0 2024-03-23T11:26:39.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.681+0000 7fd623e00640 1 -- 172.21.15.62:0/2772730786 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61c02cc60 con 0x7fd6240d11e0 2024-03-23T11:26:39.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.681+0000 7fd623e00640 1 -- 172.21.15.62:0/2772730786 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c0343f0 con 0x7fd6240d11e0 2024-03-23T11:26:39.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.681+0000 7fd62a200640 1 -- 172.21.15.62:0/2772730786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6240d11e0 msgr2=0x7fd6240b3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:39.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.681+0000 7fd62a200640 1 --2- 172.21.15.62:0/2772730786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6240d11e0 0x7fd6240b3310 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fd624066520 tx=0x7fd61c02bf60 comp rx=0 tx=0).stop 2024-03-23T11:26:39.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.682+0000 7fd62a200640 1 -- 172.21.15.62:0/2772730786 shutdown_connections 2024-03-23T11:26:39.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.682+0000 7fd62a200640 1 --2- 172.21.15.62:0/2772730786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6240d11e0 0x7fd6240b3310 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.682+0000 7fd62a200640 1 --2- 172.21.15.62:0/2772730786 >> v2:172.21.15.120:3300/0 conn(0x7fd6240cd260 0x7fd6240d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.682+0000 7fd62a200640 1 --2- 172.21.15.62:0/2772730786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd62415c960 0x7fd6240cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.682+0000 7fd62a200640 1 -- 172.21.15.62:0/2772730786 >> 172.21.15.62:0/2772730786 conn(0x7fd624089f60 msgr2=0x7fd6240d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:39.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.682+0000 7fd62a200640 1 -- 172.21.15.62:0/2772730786 shutdown_connections 2024-03-23T11:26:39.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.682+0000 7fd62a200640 1 -- 172.21.15.62:0/2772730786 wait complete. 2024-03-23T11:26:39.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.683+0000 7fd62a200640 1 Processor -- start 2024-03-23T11:26:39.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.683+0000 7fd62a200640 1 -- start start 2024-03-23T11:26:39.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.684+0000 7fd62a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6240cd260 0x7fd6240c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd62a200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6240d11e0 0x7fd62409c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd62a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd62415c960 0x7fd6240c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd62a200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6240ca930 con 0x7fd6240cd260 2024-03-23T11:26:39.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd62a200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd624151210 con 0x7fd6240d11e0 2024-03-23T11:26:39.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd62a200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6240cb590 con 0x7fd62415c960 2024-03-23T11:26:39.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd628e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6240cd260 0x7fd6240c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:39.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd628e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6240cd260 0x7fd6240c74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44258/0 (socket says 172.21.15.62:44258) 2024-03-23T11:26:39.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd628e00640 1 -- 172.21.15.62:0/3492625477 learned_addr learned my addr 172.21.15.62:0/3492625477 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:39.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd628e00640 1 -- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd62415c960 msgr2=0x7fd6240c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:39.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd628e00640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd62415c960 0x7fd6240c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd628e00640 1 -- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6240d11e0 msgr2=0x7fd62409c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:39.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd628e00640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6240d11e0 0x7fd62409c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.685+0000 7fd628e00640 1 -- 172.21.15.62:0/3492625477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c009b30 con 0x7fd6240cd260 2024-03-23T11:26:39.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.686+0000 7fd628e00640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6240cd260 0x7fd6240c74b0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fd61400b4a0 tx=0x7fd61400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:39.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.686+0000 7fd621600640 1 -- 172.21.15.62:0/3492625477 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd614010610 con 0x7fd6240cd260 2024-03-23T11:26:39.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.686+0000 7fd621600640 1 -- 172.21.15.62:0/3492625477 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd614010770 con 0x7fd6240cd260 2024-03-23T11:26:39.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.686+0000 7fd621600640 1 -- 172.21.15.62:0/3492625477 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd614010b20 con 0x7fd6240cd260 2024-03-23T11:26:39.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.686+0000 7fd62a200640 1 -- 172.21.15.62:0/3492625477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6240ca370 con 0x7fd6240cd260 2024-03-23T11:26:39.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.686+0000 7fd62a200640 1 -- 172.21.15.62:0/3492625477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6240cb830 con 0x7fd6240cd260 2024-03-23T11:26:39.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.687+0000 7fd62a200640 1 -- 172.21.15.62:0/3492625477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6240cb3f0 con 0x7fd6240cd260 2024-03-23T11:26:39.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.688+0000 7fd621600640 1 -- 172.21.15.62:0/3492625477 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd614010c80 con 0x7fd6240cd260 2024-03-23T11:26:39.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.690+0000 7fd621600640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd60006a7a0 0x7fd60006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:39.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.690+0000 7fd621600640 1 -- 172.21.15.62:0/3492625477 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd614088a30 con 0x7fd6240cd260 2024-03-23T11:26:39.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.690+0000 7fd61be00640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd60006a7a0 0x7fd60006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:39.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.691+0000 7fd61be00640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd60006a7a0 0x7fd60006cc60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd624065fc0 tx=0x7fd60c006be0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:39.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.694+0000 7fd621600640 1 -- 172.21.15.62:0/3492625477 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd6240cb3f0 con 0x7fd6240cd260 2024-03-23T11:26:39.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.905+0000 7fd62a200640 1 -- 172.21.15.62:0/3492625477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd6240cb3f0 con 0x7fd6240cd260 2024-03-23T11:26:39.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.906+0000 7fd621600640 1 -- 172.21.15.62:0/3492625477 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v89) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd6240cb3f0 con 0x7fd6240cd260 2024-03-23T11:26:39.900 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:26:39.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.909+0000 7fd62a200640 1 -- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd60006a7a0 msgr2=0x7fd60006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:39.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.909+0000 7fd62a200640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd60006a7a0 0x7fd60006cc60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd624065fc0 tx=0x7fd60c006be0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.909+0000 7fd62a200640 1 -- 172.21.15.62:0/3492625477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6240cd260 msgr2=0x7fd6240c74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.909+0000 7fd62a200640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6240cd260 0x7fd6240c74b0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fd61400b4a0 tx=0x7fd61400b960 comp rx=0 tx=0).stop 2024-03-23T11:26:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.909+0000 7fd62a200640 1 -- 172.21.15.62:0/3492625477 shutdown_connections 2024-03-23T11:26:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.909+0000 7fd62a200640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd60006a7a0 0x7fd60006cc60 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.909+0000 7fd62a200640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd62415c960 0x7fd6240c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.909+0000 7fd62a200640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6240d11e0 0x7fd62409c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.909+0000 7fd62a200640 1 --2- 172.21.15.62:0/3492625477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6240cd260 0x7fd6240c74b0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.909+0000 7fd62a200640 1 -- 172.21.15.62:0/3492625477 >> 172.21.15.62:0/3492625477 conn(0x7fd624089f60 msgr2=0x7fd6240b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.909+0000 7fd62a200640 1 -- 172.21.15.62:0/3492625477 shutdown_connections 2024-03-23T11:26:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:39.910+0000 7fd62a200640 1 -- 172.21.15.62:0/3492625477 wait complete. 2024-03-23T11:26:39.922 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:26:39.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:39.923 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:40.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[66002]: fuse finished with error 0 and tester_r 0 2024-03-23T11:26:40.023 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:26:46.125 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:26:46.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:46.125 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:26:46.189 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:26:46.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.422+0000 7f36ee400640 1 Processor -- start 2024-03-23T11:26:46.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.422+0000 7f36ee400640 1 -- start start 2024-03-23T11:26:46.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.423+0000 7f36ee400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f36e8160420 0x7f36e81608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:46.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.423+0000 7f36ee400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e80b5db0 0x7f36e80b6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:46.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.423+0000 7f36ee400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e80ce510 con 0x7f36e815fda0 2024-03-23T11:26:46.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.423+0000 7f36ee400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e8160df0 con 0x7f36e8160420 2024-03-23T11:26:46.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.423+0000 7f36ee400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e8160180 con 0x7f36e80b5db0 2024-03-23T11:26:46.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.424+0000 7f36e7e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f36e8160420 0x7f36e81608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:46.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.424+0000 7f36eda00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e80b5db0 0x7f36e80b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:46.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.424+0000 7f36eda00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e80b5db0 0x7f36e80b6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:51742/0 (socket says 172.21.15.62:51742) 2024-03-23T11:26:46.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.424+0000 7f36eda00640 1 -- 172.21.15.62:0/2534700121 learned_addr learned my addr 172.21.15.62:0/2534700121 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:46.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.425+0000 7f36e7e00640 1 -- 172.21.15.62:0/2534700121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e80b5db0 msgr2=0x7f36e80b6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:46.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.425+0000 7f36e7e00640 1 --2- 172.21.15.62:0/2534700121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e80b5db0 0x7f36e80b6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:46.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.425+0000 7f36e7e00640 1 -- 172.21.15.62:0/2534700121 >> v1:172.21.15.120:6789/0 conn(0x7f36e815fda0 legacy=0x7f36e80d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:46.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.425+0000 7f36e7e00640 1 -- 172.21.15.62:0/2534700121 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e80cdf50 con 0x7f36e8160420 2024-03-23T11:26:46.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.426+0000 7f36e7e00640 1 --2- 172.21.15.62:0/2534700121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e8160420 0x7f36e81608b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009d90 tx=0x7f36d802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcb0e1390e655c75 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:46.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.427+0000 7f36e7400640 1 -- 172.21.15.62:0/2534700121 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d802cbd0 con 0x7f36e8160420 2024-03-23T11:26:46.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.428+0000 7f36e7400640 1 -- 172.21.15.62:0/2534700121 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d802cd30 con 0x7f36e8160420 2024-03-23T11:26:46.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.428+0000 7f36e7400640 1 -- 172.21.15.62:0/2534700121 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d802cea0 con 0x7f36e8160420 2024-03-23T11:26:46.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.428+0000 7f36ee400640 1 -- 172.21.15.62:0/2534700121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e8160420 msgr2=0x7f36e81608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:46.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.428+0000 7f36ee400640 1 --2- 172.21.15.62:0/2534700121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e8160420 0x7f36e81608b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009d90 tx=0x7f36d802be20 comp rx=0 tx=0).stop 2024-03-23T11:26:46.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.429+0000 7f36ee400640 1 -- 172.21.15.62:0/2534700121 shutdown_connections 2024-03-23T11:26:46.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.429+0000 7f36ee400640 1 --2- 172.21.15.62:0/2534700121 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e80b5db0 0x7f36e80b6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:46.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.429+0000 7f36ee400640 1 --2- 172.21.15.62:0/2534700121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e8160420 0x7f36e81608b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:46.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.429+0000 7f36ee400640 1 -- 172.21.15.62:0/2534700121 >> 172.21.15.62:0/2534700121 conn(0x7f36e8089f60 msgr2=0x7f36e80d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:46.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.429+0000 7f36ee400640 1 -- 172.21.15.62:0/2534700121 shutdown_connections 2024-03-23T11:26:46.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.429+0000 7f36ee400640 1 -- 172.21.15.62:0/2534700121 wait complete. 2024-03-23T11:26:46.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.431+0000 7f36ee400640 1 Processor -- start 2024-03-23T11:26:46.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.431+0000 7f36ee400640 1 -- start start 2024-03-23T11:26:46.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.433+0000 7f36ee400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e80b5db0 0x7f36e814ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:46.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.433+0000 7f36ee400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e815fda0 0x7f36e80ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:46.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.433+0000 7f36ee400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e8160420 0x7f36e80cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:46.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.433+0000 7f36ee400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e81594f0 con 0x7f36e80b5db0 2024-03-23T11:26:46.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.433+0000 7f36ee400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e814f0f0 con 0x7f36e815fda0 2024-03-23T11:26:46.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.433+0000 7f36ee400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e81f5ca0 con 0x7f36e8160420 2024-03-23T11:26:46.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.433+0000 7f36ed000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e80b5db0 0x7f36e814ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:46.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.433+0000 7f36ed000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e80b5db0 0x7f36e814ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44260/0 (socket says 172.21.15.62:44260) 2024-03-23T11:26:46.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.433+0000 7f36ed000640 1 -- 172.21.15.62:0/586309088 learned_addr learned my addr 172.21.15.62:0/586309088 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:46.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.433+0000 7f36e7e00640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e815fda0 0x7f36e80ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:46.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.433+0000 7f36eda00640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e8160420 0x7f36e80cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:46.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.434+0000 7f36ed000640 1 -- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e8160420 msgr2=0x7f36e80cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:46.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.434+0000 7f36ed000640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e8160420 0x7f36e80cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:46.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.434+0000 7f36ed000640 1 -- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e815fda0 msgr2=0x7f36e80ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:46.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.434+0000 7f36ed000640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e815fda0 0x7f36e80ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:46.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.434+0000 7f36ed000640 1 -- 172.21.15.62:0/586309088 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8009a70 con 0x7f36e80b5db0 2024-03-23T11:26:46.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.434+0000 7f36ed000640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e80b5db0 0x7f36e814ebb0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f36d000c400 tx=0x7f36d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:46.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.434+0000 7f36e4c00640 1 -- 172.21.15.62:0/586309088 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d00052d0 con 0x7f36e80b5db0 2024-03-23T11:26:46.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.434+0000 7f36e4c00640 1 -- 172.21.15.62:0/586309088 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d0005430 con 0x7f36e80b5db0 2024-03-23T11:26:46.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.434+0000 7f36ee400640 1 -- 172.21.15.62:0/586309088 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e81f5f80 con 0x7f36e80b5db0 2024-03-23T11:26:46.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.435+0000 7f36e4c00640 1 -- 172.21.15.62:0/586309088 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d0003830 con 0x7f36e80b5db0 2024-03-23T11:26:46.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.435+0000 7f36ee400640 1 -- 172.21.15.62:0/586309088 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e81f6540 con 0x7f36e80b5db0 2024-03-23T11:26:46.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.436+0000 7f36ee400640 1 -- 172.21.15.62:0/586309088 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e804e030 con 0x7f36e80b5db0 2024-03-23T11:26:46.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.437+0000 7f36e4c00640 1 -- 172.21.15.62:0/586309088 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f36d0003990 con 0x7f36e80b5db0 2024-03-23T11:26:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.438+0000 7f36e4c00640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a7a0 0x7f36c406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.438+0000 7f36e4c00640 1 -- 172.21.15.62:0/586309088 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f36d00882f0 con 0x7f36e80b5db0 2024-03-23T11:26:46.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.439+0000 7f36e7e00640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a7a0 0x7f36c406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:46.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.440+0000 7f36e7e00640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a7a0 0x7f36c406cc60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f36d8036000 tx=0x7f36d8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:46.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.443+0000 7f36e4c00640 1 -- 172.21.15.62:0/586309088 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f36d00536e0 con 0x7f36e80b5db0 2024-03-23T11:26:46.554 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:26:46.562 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:26:46.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.681+0000 7f36ee400640 1 -- 172.21.15.62:0/586309088 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36e80cea40 con 0x7f36e80b5db0 2024-03-23T11:26:46.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.682+0000 7f36e4c00640 1 -- 172.21.15.62:0/586309088 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f36d0056d00 con 0x7f36e80b5db0 2024-03-23T11:26:46.676 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:46.676 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":57,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:26:28.303857+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6154},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6154":{"gid":6154,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/3056962984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3056962984},{"type":"v1","addr":"172.21.15.62:6833","nonce":3056962984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:26:46.676 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 57 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.685+0000 7f36ee400640 1 -- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a7a0 msgr2=0x7f36c406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.685+0000 7f36ee400640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a7a0 0x7f36c406cc60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f36d8036000 tx=0x7f36d8030000 comp rx=0 tx=0).stop 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.685+0000 7f36ee400640 1 -- 172.21.15.62:0/586309088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e80b5db0 msgr2=0x7f36e814ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.685+0000 7f36ee400640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e80b5db0 0x7f36e814ebb0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f36d000c400 tx=0x7f36d000c8c0 comp rx=0 tx=0).stop 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.685+0000 7f36ee400640 1 -- 172.21.15.62:0/586309088 shutdown_connections 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.685+0000 7f36ee400640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f36c406a7a0 0x7f36c406cc60 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.685+0000 7f36ee400640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e8160420 0x7f36e80cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.685+0000 7f36ee400640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e815fda0 0x7f36e80ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.685+0000 7f36ee400640 1 --2- 172.21.15.62:0/586309088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36e80b5db0 0x7f36e814ebb0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.685+0000 7f36ee400640 1 -- 172.21.15.62:0/586309088 >> 172.21.15.62:0/586309088 conn(0x7f36e8089f60 msgr2=0x7f36e80b7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.686+0000 7f36ee400640 1 -- 172.21.15.62:0/586309088 shutdown_connections 2024-03-23T11:26:46.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:46.686+0000 7f36ee400640 1 -- 172.21.15.62:0/586309088 wait complete. 2024-03-23T11:26:46.697 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-23T11:26:46.997 INFO:teuthology.orchestra.run.smithi062.stdout:{"message":"","return_code":0} 2024-03-23T11:26:46.997 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-23T11:26:46.997 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:26:47.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.225+0000 7fee2f200640 1 Processor -- start 2024-03-23T11:26:47.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.225+0000 7fee2f200640 1 -- start start 2024-03-23T11:26:47.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.226+0000 7fee2f200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fee3015c960 0x7fee300cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.226+0000 7fee2f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fee300cd260 0x7fee300d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.226+0000 7fee2f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee300d11e0 0x7fee300b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.226+0000 7fee2f200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee300d1650 con 0x7fee3015c960 2024-03-23T11:26:47.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.226+0000 7fee2f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee300d0810 con 0x7fee300cd260 2024-03-23T11:26:47.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.226+0000 7fee2f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee300cd9d0 con 0x7fee300d11e0 2024-03-23T11:26:47.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.227+0000 7fee2de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fee3015c960 0x7fee300cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.227+0000 7fee2de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fee3015c960 0x7fee300cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44276/0 (socket says 172.21.15.62:44276) 2024-03-23T11:26:47.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.227+0000 7fee2de00640 1 -- 172.21.15.62:0/2684542228 learned_addr learned my addr 172.21.15.62:0/2684542228 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.227+0000 7fee2de00640 1 -- 172.21.15.62:0/2684542228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee300d11e0 msgr2=0x7fee300b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.227+0000 7fee2e800640 1 --2- 172.21.15.62:0/2684542228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee300d11e0 0x7fee300b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.227+0000 7fee2d400640 1 --2- 172.21.15.62:0/2684542228 >> v2:172.21.15.120:3300/0 conn(0x7fee300cd260 0x7fee300d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.227+0000 7fee2de00640 1 --2- 172.21.15.62:0/2684542228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee300d11e0 0x7fee300b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.227+0000 7fee2de00640 1 -- 172.21.15.62:0/2684542228 >> v2:172.21.15.120:3300/0 conn(0x7fee300cd260 msgr2=0x7fee300d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.227+0000 7fee2de00640 1 --2- 172.21.15.62:0/2684542228 >> v2:172.21.15.120:3300/0 conn(0x7fee300cd260 0x7fee300d0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.227+0000 7fee2de00640 1 -- 172.21.15.62:0/2684542228 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee300cdcb0 con 0x7fee3015c960 2024-03-23T11:26:47.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.229+0000 7fee2de00640 1 --2- 172.21.15.62:0/2684542228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee3015c960 0x7fee300cf790 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fee300cfcd0 tx=0x7fee2002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591fefce1af31d90 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.229+0000 7fee2ca00640 1 -- 172.21.15.62:0/2684542228 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee20030860 con 0x7fee3015c960 2024-03-23T11:26:47.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.229+0000 7fee2ca00640 1 -- 172.21.15.62:0/2684542228 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee200309c0 con 0x7fee3015c960 2024-03-23T11:26:47.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.229+0000 7fee2ca00640 1 -- 172.21.15.62:0/2684542228 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee20030d50 con 0x7fee3015c960 2024-03-23T11:26:47.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.230+0000 7fee2f200640 1 -- 172.21.15.62:0/2684542228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee3015c960 msgr2=0x7fee300cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.230+0000 7fee2f200640 1 --2- 172.21.15.62:0/2684542228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee3015c960 0x7fee300cf790 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fee300cfcd0 tx=0x7fee2002b9a0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.230+0000 7fee2f200640 1 -- 172.21.15.62:0/2684542228 shutdown_connections 2024-03-23T11:26:47.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.230+0000 7fee2f200640 1 --2- 172.21.15.62:0/2684542228 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee300d11e0 0x7fee300b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.230+0000 7fee2f200640 1 --2- 172.21.15.62:0/2684542228 >> v2:172.21.15.120:3300/0 conn(0x7fee300cd260 0x7fee300d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.230+0000 7fee2f200640 1 --2- 172.21.15.62:0/2684542228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee3015c960 0x7fee300cf790 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.230+0000 7fee2f200640 1 -- 172.21.15.62:0/2684542228 >> 172.21.15.62:0/2684542228 conn(0x7fee30089f60 msgr2=0x7fee300d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:47.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.230+0000 7fee2f200640 1 -- 172.21.15.62:0/2684542228 shutdown_connections 2024-03-23T11:26:47.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.230+0000 7fee2f200640 1 -- 172.21.15.62:0/2684542228 wait complete. 2024-03-23T11:26:47.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.232+0000 7fee2f200640 1 Processor -- start 2024-03-23T11:26:47.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.232+0000 7fee2f200640 1 -- start start 2024-03-23T11:26:47.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.233+0000 7fee2f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee300cd260 0x7fee300c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.233+0000 7fee2f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee300d11e0 0x7fee301f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.233+0000 7fee2f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee3015c960 0x7fee300cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.233+0000 7fee2f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee30159500 con 0x7fee300d11e0 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.233+0000 7fee2f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3015ecf0 con 0x7fee300cd260 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.233+0000 7fee2f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3015efd0 con 0x7fee3015c960 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee300d11e0 0x7fee301f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee300d11e0 0x7fee301f1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44292/0 (socket says 172.21.15.62:44292) 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2d400640 1 -- 172.21.15.62:0/4009988464 learned_addr learned my addr 172.21.15.62:0/4009988464 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2de00640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee300cd260 0x7fee300c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2e800640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee3015c960 0x7fee300cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2d400640 1 -- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee3015c960 msgr2=0x7fee300cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2d400640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee3015c960 0x7fee300cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2d400640 1 -- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee300cd260 msgr2=0x7fee300c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2d400640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee300cd260 0x7fee300c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2d400640 1 -- 172.21.15.62:0/4009988464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee200094d0 con 0x7fee300d11e0 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2de00640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee300cd260 0x7fee300c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.234+0000 7fee2e800640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee3015c960 0x7fee300cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.235+0000 7fee2d400640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee300d11e0 0x7fee301f1df0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fee1800cd40 tx=0x7fee1800a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.235+0000 7fee26000640 1 -- 172.21.15.62:0/4009988464 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee18003b90 con 0x7fee300d11e0 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.235+0000 7fee26000640 1 -- 172.21.15.62:0/4009988464 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee18003cf0 con 0x7fee300d11e0 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.235+0000 7fee26000640 1 -- 172.21.15.62:0/4009988464 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee18004260 con 0x7fee300d11e0 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.235+0000 7fee2f200640 1 -- 172.21.15.62:0/4009988464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3015f2b0 con 0x7fee300d11e0 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.235+0000 7fee2f200640 1 -- 172.21.15.62:0/4009988464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee300c8460 con 0x7fee300d11e0 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.236+0000 7fee2f200640 1 -- 172.21.15.62:0/4009988464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee30066bb0 con 0x7fee300d11e0 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.237+0000 7fee26000640 1 -- 172.21.15.62:0/4009988464 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fee180043c0 con 0x7fee300d11e0 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.238+0000 7fee26000640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee0c06a7a0 0x7fee0c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.238+0000 7fee26000640 1 -- 172.21.15.62:0/4009988464 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fee18089690 con 0x7fee300d11e0 2024-03-23T11:26:47.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.238+0000 7fee2de00640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee0c06a7a0 0x7fee0c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.240+0000 7fee2de00640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee0c06a7a0 0x7fee0c06cc60 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fee20036000 tx=0x7fee2003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.242+0000 7fee26000640 1 -- 172.21.15.62:0/4009988464 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fee18054960 con 0x7fee300d11e0 2024-03-23T11:26:47.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.479+0000 7fee2f200640 1 -- 172.21.15.62:0/4009988464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fee3015c160 con 0x7fee300d11e0 2024-03-23T11:26:47.474 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:26:47.482+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:26:47.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.498+0000 7fee26000640 1 -- 172.21.15.62:0/4009988464 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v58) v1 ==== 161+0+0 (secure 0 0 0) 0x7fee1801d020 con 0x7fee300d11e0 2024-03-23T11:26:47.491 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:26:47.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 -- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee0c06a7a0 msgr2=0x7fee0c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee0c06a7a0 0x7fee0c06cc60 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fee20036000 tx=0x7fee2003b040 comp rx=0 tx=0).stop 2024-03-23T11:26:47.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 -- 172.21.15.62:0/4009988464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee300d11e0 msgr2=0x7fee301f1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee300d11e0 0x7fee301f1df0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fee1800cd40 tx=0x7fee1800a610 comp rx=0 tx=0).stop 2024-03-23T11:26:47.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 -- 172.21.15.62:0/4009988464 shutdown_connections 2024-03-23T11:26:47.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fee0c06a7a0 0x7fee0c06cc60 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee300d11e0 0x7fee301f1df0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee3015c960 0x7fee300cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 --2- 172.21.15.62:0/4009988464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee300cd260 0x7fee300c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 -- 172.21.15.62:0/4009988464 >> 172.21.15.62:0/4009988464 conn(0x7fee30089f60 msgr2=0x7fee300b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:47.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 -- 172.21.15.62:0/4009988464 shutdown_connections 2024-03-23T11:26:47.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.501+0000 7fee2f200640 1 -- 172.21.15.62:0/4009988464 wait complete. 2024-03-23T11:26:47.517 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 10000000000.00000000 2024-03-23T11:26:47.518 INFO:tasks.ceph.mds.a.smithi062.stderr: -3946> 2024-03-23T11:26:29.597+0000 7fbffa600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:26:47.599 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:26:47.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.625+0000 7fd94f235f40 1 Processor -- start 2024-03-23T11:26:47.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.625+0000 7fd94f235f40 1 -- start start 2024-03-23T11:26:47.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.626+0000 7fd94f235f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55895c6e4310 0x55895c76a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.626+0000 7fd94f235f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c76ec20 0x55895c6e38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.626+0000 7fd94f235f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895c76f000 0x55895c769490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.626+0000 7fd94f235f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895c643c20 con 0x55895c76ec20 2024-03-23T11:26:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.626+0000 7fd94f235f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895c5df800 con 0x55895c76f000 2024-03-23T11:26:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.626+0000 7fd94f235f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55895c5df4a0 con 0x55895c6e4310 2024-03-23T11:26:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c76ec20 0x55895c6e38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c76ec20 0x55895c6e38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36684/0 (socket says 172.21.15.62:36684) 2024-03-23T11:26:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94b200640 1 -- 172.21.15.62:0/107905029 learned_addr learned my addr 172.21.15.62:0/107905029 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94bc00640 1 --2- 172.21.15.62:0/107905029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55895c6e4310 0x55895c76a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94b200640 1 -- 172.21.15.62:0/107905029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55895c6e4310 msgr2=0x55895c76a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94c600640 1 --2- 172.21.15.62:0/107905029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895c76f000 0x55895c769490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94b200640 1 --2- 172.21.15.62:0/107905029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55895c6e4310 0x55895c76a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94b200640 1 -- 172.21.15.62:0/107905029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895c76f000 msgr2=0x55895c769490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94b200640 1 --2- 172.21.15.62:0/107905029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895c76f000 0x55895c769490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94b200640 1 -- 172.21.15.62:0/107905029 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94400a100 con 0x55895c76ec20 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94c600640 1 --2- 172.21.15.62:0/107905029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895c76f000 0x55895c769490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.627+0000 7fd94b200640 1 --2- 172.21.15.62:0/107905029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c76ec20 0x55895c6e38f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd938001e90 tx=0x7fd93800b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1e6a253b1da7ed server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd949e00640 1 -- 172.21.15.62:0/107905029 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93800e670 con 0x55895c76ec20 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd949e00640 1 -- 172.21.15.62:0/107905029 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93800e7d0 con 0x55895c76ec20 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd949e00640 1 -- 172.21.15.62:0/107905029 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93800eb80 con 0x55895c76ec20 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd94f235f40 1 -- 172.21.15.62:0/107905029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c76ec20 msgr2=0x55895c6e38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd94f235f40 1 --2- 172.21.15.62:0/107905029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c76ec20 0x55895c6e38f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd938001e90 tx=0x7fd93800b780 comp rx=0 tx=0).stop 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd94f235f40 1 -- 172.21.15.62:0/107905029 shutdown_connections 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd94f235f40 1 --2- 172.21.15.62:0/107905029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c76ec20 0x55895c6e38f0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd94f235f40 1 --2- 172.21.15.62:0/107905029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895c76f000 0x55895c769490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd94f235f40 1 --2- 172.21.15.62:0/107905029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55895c6e4310 0x55895c76a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd94f235f40 1 -- 172.21.15.62:0/107905029 >> 172.21.15.62:0/107905029 conn(0x55895c6e4ff0 msgr2=0x55895c6e31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:47.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd94f235f40 1 -- 172.21.15.62:0/107905029 shutdown_connections 2024-03-23T11:26:47.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.628+0000 7fd94f235f40 1 -- 172.21.15.62:0/107905029 wait complete. 2024-03-23T11:26:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.629+0000 7fd94f235f40 1 Processor -- start 2024-03-23T11:26:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.629+0000 7fd94f235f40 1 -- start start 2024-03-23T11:26:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.630+0000 7fd94f235f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c6e4310 0x55895c6c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.630+0000 7fd94f235f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55895c76ec20 0x55895c6956a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.630+0000 7fd94f235f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895c76f000 0x55895c6c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.630+0000 7fd94f235f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895c5d6110 con 0x55895c6e4310 2024-03-23T11:26:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.630+0000 7fd94f235f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895c750400 con 0x55895c76f000 2024-03-23T11:26:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.630+0000 7fd94f235f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55895c760ca0 con 0x55895c76ec20 2024-03-23T11:26:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.630+0000 7fd94bc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c6e4310 0x55895c6c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.631+0000 7fd94bc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c6e4310 0x55895c6c5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36686/0 (socket says 172.21.15.62:36686) 2024-03-23T11:26:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.631+0000 7fd94bc00640 1 -- 172.21.15.62:0/3490809613 learned_addr learned my addr 172.21.15.62:0/3490809613 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.631+0000 7fd94c600640 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895c76f000 0x55895c6c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.631+0000 7fd94b200640 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55895c76ec20 0x55895c6956a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.631+0000 7fd94bc00640 1 -- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55895c76ec20 msgr2=0x55895c6956a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.631+0000 7fd94bc00640 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55895c76ec20 0x55895c6956a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.631+0000 7fd94bc00640 1 -- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895c76f000 msgr2=0x55895c6c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.631+0000 7fd94bc00640 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895c76f000 0x55895c6c8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.631+0000 7fd94bc00640 1 -- 172.21.15.62:0/3490809613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd938001a90 con 0x55895c6e4310 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.631+0000 7fd94b200640 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55895c76ec20 0x55895c6956a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.631+0000 7fd94bc00640 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c6e4310 0x55895c6c5640 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00dfc0 tx=0x7fd93c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.632+0000 7fd948a00640 1 -- 172.21.15.62:0/3490809613 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c011640 con 0x55895c6e4310 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.632+0000 7fd948a00640 1 -- 172.21.15.62:0/3490809613 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c008040 con 0x55895c6e4310 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.632+0000 7fd948a00640 1 -- 172.21.15.62:0/3490809613 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c0119e0 con 0x55895c6e4310 2024-03-23T11:26:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.632+0000 7fd94f235f40 1 -- 172.21.15.62:0/3490809613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55895c5e1780 con 0x55895c6e4310 2024-03-23T11:26:47.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.632+0000 7fd94f235f40 1 -- 172.21.15.62:0/3490809613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55895c5e11e0 con 0x55895c6e4310 2024-03-23T11:26:47.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.634+0000 7fd948a00640 1 -- 172.21.15.62:0/3490809613 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd93c011b40 con 0x55895c6e4310 2024-03-23T11:26:47.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.634+0000 7fd948a00640 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd92c0691b0 0x7fd92c06b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.635+0000 7fd948a00640 1 -- 172.21.15.62:0/3490809613 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fd93c087890 con 0x55895c6e4310 2024-03-23T11:26:47.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.635+0000 7fd94f235f40 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55895c695be0 0x55895c770150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.635+0000 7fd94f235f40 1 -- 172.21.15.62:0/3490809613 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) v8 -- 0x55895c751960 con 0x55895c695be0 2024-03-23T11:26:47.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.635+0000 7fd94c600640 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55895c695be0 0x55895c770150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.635+0000 7fd94b200640 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd92c0691b0 0x7fd92c06b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.636+0000 7fd94c600640 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55895c695be0 0x55895c770150 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.636+0000 7fd94b200640 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd92c0691b0 0x7fd92c06b670 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x55895c75e360 tx=0x7fd938009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94c600640 1 -- 172.21.15.62:0/3490809613 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== osd_op_reply(1 10000000000.00000000 [delete] v92'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fd944052980 con 0x55895c695be0 2024-03-23T11:26:47.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94f235f40 1 -- 172.21.15.62:0/3490809613 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55895c695be0 msgr2=0x55895c770150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94f235f40 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55895c695be0 0x55895c770150 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94f235f40 1 -- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd92c0691b0 msgr2=0x7fd92c06b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94f235f40 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd92c0691b0 0x7fd92c06b670 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x55895c75e360 tx=0x7fd938009000 comp rx=0 tx=0).stop 2024-03-23T11:26:47.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94f235f40 1 -- 172.21.15.62:0/3490809613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c6e4310 msgr2=0x55895c6c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94f235f40 1 --2- 172.21.15.62:0/3490809613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55895c6e4310 0x55895c6c5640 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00dfc0 tx=0x7fd93c00b5b0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94c600640 1 -- 172.21.15.62:0/3490809613 reap_dead start 2024-03-23T11:26:47.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94f235f40 1 -- 172.21.15.62:0/3490809613 shutdown_connections 2024-03-23T11:26:47.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94f235f40 1 -- 172.21.15.62:0/3490809613 >> 172.21.15.62:0/3490809613 conn(0x55895c6e4ff0 msgr2=0x55895c76a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:47.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94f235f40 1 -- 172.21.15.62:0/3490809613 shutdown_connections 2024-03-23T11:26:47.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.639+0000 7fd94f235f40 1 -- 172.21.15.62:0/3490809613 wait complete. 2024-03-23T11:26:47.638 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset 2024-03-23T11:26:47.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.735+0000 7fee4064b280 1 -- start start 2024-03-23T11:26:47.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.735+0000 7fee4064b280 1 Processor -- start 2024-03-23T11:26:47.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee4064b280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfcca800 0x555cdef83b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee4064b280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9c00 0x555cdef82b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee4064b280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfccbc00 0x555cdef83600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee4064b280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdefcaa00 con 0x555cdfccbc00 2024-03-23T11:26:47.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee4064b280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdef43e00 con 0x555cdfcca800 2024-03-23T11:26:47.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee4064b280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555cdefca280 con 0x555cdeef9c00 2024-03-23T11:26:47.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee3ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfccbc00 0x555cdef83600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee3ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfccbc00 0x555cdef83600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36710/0 (socket says 172.21.15.62:36710) 2024-03-23T11:26:47.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee3ea00640 1 -- 172.21.15.62:0/1806442328 learned_addr learned my addr 172.21.15.62:0/1806442328 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:47.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee3d600640 1 --2- 172.21.15.62:0/1806442328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9c00 0x555cdef82b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee3e000640 1 --2- 172.21.15.62:0/1806442328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfcca800 0x555cdef83b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee3ea00640 1 -- 172.21.15.62:0/1806442328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9c00 msgr2=0x555cdef82b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:47.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.736+0000 7fee3ea00640 1 --2- 172.21.15.62:0/1806442328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9c00 0x555cdef82b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.737+0000 7fee3ea00640 1 -- 172.21.15.62:0/1806442328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfcca800 msgr2=0x555cdef83b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:47.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.737+0000 7fee3ea00640 1 --2- 172.21.15.62:0/1806442328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfcca800 0x555cdef83b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.737+0000 7fee3ea00640 1 -- 172.21.15.62:0/1806442328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cdef1ed00 con 0x555cdfccbc00 2024-03-23T11:26:47.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.737+0000 7fee3ea00640 1 --2- 172.21.15.62:0/1806442328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfccbc00 0x555cdef83600 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x555cdfc60210 tx=0x555cdfcfcc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9efae0a0570da5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.737+0000 7fee3ae00640 1 -- 172.21.15.62:0/1806442328 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555cdef41080 con 0x555cdfccbc00 2024-03-23T11:26:47.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.737+0000 7fee3ae00640 1 -- 172.21.15.62:0/1806442328 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555cdfdb6000 con 0x555cdfccbc00 2024-03-23T11:26:47.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.737+0000 7fee3ae00640 1 -- 172.21.15.62:0/1806442328 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555cdef41600 con 0x555cdfccbc00 2024-03-23T11:26:47.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.737+0000 7fee4064b280 1 -- 172.21.15.62:0/1806442328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555cdef1e820 con 0x555cdfccbc00 2024-03-23T11:26:47.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.737+0000 7fee4064b280 1 -- 172.21.15.62:0/1806442328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x555cdef1e4e0 con 0x555cdfccbc00 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.737+0000 7fee4064b280 4 waiting for MDS map... 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.737+0000 7fee3ae00640 1 -- 172.21.15.62:0/1806442328 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x555cdef24380 con 0x555cdfccbc00 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.738+0000 7fee3ae00640 1 -- 172.21.15.62:0/1806442328 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 58) v1 ==== 2573+0+0 (secure 0 0 0) 0x555cdef4af00 con 0x555cdfccbc00 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.738+0000 7fee4064b280 4 Got MDS map 58 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.738+0000 7fee4064b280 10 main: JournalTool::main 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.738+0000 7fee4064b280 4 main: JournalTool: connecting to RADOS... 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.738+0000 7fee4064b280 1 Processor -- start 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.739+0000 7fee4064b280 1 -- start start 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.739+0000 7fee4064b280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfcca000 0x555cdef83080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.739+0000 7fee4064b280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9800 0x555cdef84100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.739+0000 7fee4064b280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfe16000 0x555cdef84680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.739+0000 7fee4064b280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdefca3c0 con 0x555cdfcca000 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.739+0000 7fee4064b280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdefca500 con 0x555cdfe16000 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.739+0000 7fee4064b280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555cdefca640 con 0x555cdeef9800 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee3d600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfcca000 0x555cdef83080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee3d600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfcca000 0x555cdef83080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36716/0 (socket says 172.21.15.62:36716) 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee3d600640 1 -- 172.21.15.62:0/2521631863 learned_addr learned my addr 172.21.15.62:0/2521631863 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee3e000640 1 --2- 172.21.15.62:0/2521631863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9800 0x555cdef84100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee3d600640 1 -- 172.21.15.62:0/2521631863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9800 msgr2=0x555cdef84100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee3d600640 1 --2- 172.21.15.62:0/2521631863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9800 0x555cdef84100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee3d600640 1 -- 172.21.15.62:0/2521631863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfe16000 msgr2=0x555cdef84680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee3d600640 1 --2- 172.21.15.62:0/2521631863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfe16000 0x555cdef84680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee3d600640 1 -- 172.21.15.62:0/2521631863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cdef1e1a0 con 0x555cdfcca000 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee3e000640 1 --2- 172.21.15.62:0/2521631863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9800 0x555cdef84100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:47.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee3d600640 1 --2- 172.21.15.62:0/2521631863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfcca000 0x555cdef83080 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x555cdfe26000 tx=0x555cdfdcac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6324e0708786b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.740+0000 7fee38600640 1 -- 172.21.15.62:0/2521631863 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555cdef41080 con 0x555cdfcca000 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.741+0000 7fee38600640 1 -- 172.21.15.62:0/2521631863 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555cdfdb6480 con 0x555cdfcca000 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.741+0000 7fee38600640 1 -- 172.21.15.62:0/2521631863 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555cdef41600 con 0x555cdfcca000 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.741+0000 7fee4064b280 1 -- 172.21.15.62:0/2521631863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfcca000 msgr2=0x555cdef83080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.741+0000 7fee4064b280 1 --2- 172.21.15.62:0/2521631863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfcca000 0x555cdef83080 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x555cdfe26000 tx=0x555cdfdcac00 comp rx=0 tx=0).stop 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.741+0000 7fee4064b280 1 -- 172.21.15.62:0/2521631863 shutdown_connections 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.741+0000 7fee4064b280 1 --2- 172.21.15.62:0/2521631863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfe16000 0x555cdef84680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.741+0000 7fee4064b280 1 --2- 172.21.15.62:0/2521631863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9800 0x555cdef84100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.741+0000 7fee4064b280 1 --2- 172.21.15.62:0/2521631863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfcca000 0x555cdef83080 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.741+0000 7fee4064b280 1 -- 172.21.15.62:0/2521631863 >> 172.21.15.62:0/2521631863 conn(0x555cdfcca400 msgr2=0x555cdefdf800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.741+0000 7fee4064b280 1 -- 172.21.15.62:0/2521631863 shutdown_connections 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.741+0000 7fee4064b280 1 -- 172.21.15.62:0/2521631863 wait complete. 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.742+0000 7fee4064b280 1 Processor -- start 2024-03-23T11:26:47.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.742+0000 7fee4064b280 1 -- start start 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.743+0000 7fee4064b280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9800 0x555cdef85700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.743+0000 7fee4064b280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfcca000 0x555cdef85180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.743+0000 7fee4064b280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe17400 0x555cdef84c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.743+0000 7fee4064b280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdefca640 con 0x555cdfe17400 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.743+0000 7fee4064b280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdefca500 con 0x555cdfcca000 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.743+0000 7fee4064b280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555cdefca780 con 0x555cdeef9800 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.743+0000 7fee3d600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe17400 0x555cdef84c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.743+0000 7fee3d600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe17400 0x555cdef84c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36730/0 (socket says 172.21.15.62:36730) 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.743+0000 7fee3d600640 1 -- 172.21.15.62:0/2217407699 learned_addr learned my addr 172.21.15.62:0/2217407699 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.743+0000 7fee3d600640 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9800 0x555cdef85700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.743+0000 7fee3e000640 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfcca000 0x555cdef85180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee3d600640 1 -- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9800 msgr2=0x555cdef85700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee3d600640 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9800 0x555cdef85700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee3d600640 1 -- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfcca000 msgr2=0x555cdef85180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee3d600640 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfcca000 0x555cdef85180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee3d600640 1 -- 172.21.15.62:0/2217407699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cdef1eb60 con 0x555cdfe17400 2024-03-23T11:26:47.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee3d600640 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe17400 0x555cdef84c00 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x555cdfe26cc0 tx=0x555cdfdcac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee36800640 1 -- 172.21.15.62:0/2217407699 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555cdef40f20 con 0x555cdfe17400 2024-03-23T11:26:47.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee36800640 1 -- 172.21.15.62:0/2217407699 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555cdfdb6600 con 0x555cdfe17400 2024-03-23T11:26:47.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee36800640 1 -- 172.21.15.62:0/2217407699 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555cdef41080 con 0x555cdfe17400 2024-03-23T11:26:47.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee4064b280 1 -- 172.21.15.62:0/2217407699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555cdef1e680 con 0x555cdfe17400 2024-03-23T11:26:47.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee4064b280 1 -- 172.21.15.62:0/2217407699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555cdef1ed00 con 0x555cdfe17400 2024-03-23T11:26:47.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.744+0000 7fee4064b280 4 main: JournalTool: resolving pool 12 2024-03-23T11:26:47.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.746+0000 7fee36800640 1 -- 172.21.15.62:0/2217407699 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x555cdefc6b00 con 0x555cdfe17400 2024-03-23T11:26:47.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.747+0000 7fee36800640 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x555cdfe36c00 0x555cdf24e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.747+0000 7fee36800640 1 -- 172.21.15.62:0/2217407699 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x555cdef24540 con 0x555cdfe17400 2024-03-23T11:26:47.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.747+0000 7fee4064b280 4 main: JournalTool: creating IoCtx.. 2024-03-23T11:26:47.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.747+0000 7fee4064b280 4 main: Executing for rank 0 2024-03-23T11:26:47.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.747+0000 7fee3e000640 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x555cdfe36c00 0x555cdf24e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.748+0000 7fee4064b280 1 -- start start 2024-03-23T11:26:47.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.748+0000 7fee4064b280 1 Processor -- start 2024-03-23T11:26:47.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.748+0000 7fee3e000640 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x555cdfe36c00 0x555cdf24e000 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x555cdfc60240 tx=0x555cdf25acc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.749+0000 7fee4064b280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdfe16c00 0x555cdef84680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.749+0000 7fee4064b280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe16800 0x555cdef83080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.749+0000 7fee4064b280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfe16400 0x555cdef84100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.749+0000 7fee4064b280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdefca280 con 0x555cdfe16800 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.749+0000 7fee4064b280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdefcaa00 con 0x555cdfe16400 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.749+0000 7fee4064b280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555cdefca780 con 0x555cdfe16c00 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.749+0000 7fee3e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe16800 0x555cdef83080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.749+0000 7fee3e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe16800 0x555cdef83080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36746/0 (socket says 172.21.15.62:36746) 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.749+0000 7fee3e000640 1 -- 172.21.15.62:0/2758633949 learned_addr learned my addr 172.21.15.62:0/2758633949 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.750+0000 7fee3ea00640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdfe16c00 0x555cdef84680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.750+0000 7fee3d600640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfe16400 0x555cdef84100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.750+0000 7fee3e000640 1 -- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdfe16c00 msgr2=0x555cdef84680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.750+0000 7fee3e000640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdfe16c00 0x555cdef84680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.750+0000 7fee3e000640 1 -- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfe16400 msgr2=0x555cdef84100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.750+0000 7fee3e000640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfe16400 0x555cdef84100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.750+0000 7fee3e000640 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cdfe62680 con 0x555cdfe16800 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.750+0000 7fee3ea00640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdfe16c00 0x555cdef84680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.750+0000 7fee3e000640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe16800 0x555cdef83080 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x555cdf225380 tx=0x555cdf25ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07ed06e69549444 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.751+0000 7fee32c00640 1 -- 172.21.15.62:0/2758633949 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555cdef402c0 con 0x555cdfe16800 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.751+0000 7fee32c00640 1 -- 172.21.15.62:0/2758633949 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555cdef56480 con 0x555cdfe16800 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.751+0000 7fee32c00640 1 -- 172.21.15.62:0/2758633949 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555cdef406e0 con 0x555cdfe16800 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.751+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555cdfe624e0 con 0x555cdfe16800 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.751+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x555cdfe5fd40 con 0x555cdfe16800 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.751+0000 7fee4064b280 4 waiting for MDS map... 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.751+0000 7fee32c00640 1 -- 172.21.15.62:0/2758633949 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x555cdef24a80 con 0x555cdfe16800 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.751+0000 7fee32c00640 1 -- 172.21.15.62:0/2758633949 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 58) v1 ==== 2573+0+0 (secure 0 0 0) 0x555cdf2c4000 con 0x555cdfe16800 2024-03-23T11:26:47.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.751+0000 7fee4064b280 4 Got MDS map 58 2024-03-23T11:26:47.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.751+0000 7fee4064b280 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x555cdfe16000 0x555cdf24e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.751+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdf2b9400 con 0x555cdfe16000 2024-03-23T11:26:47.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.752+0000 7fee3ea00640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x555cdfe16000 0x555cdf24e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.753+0000 7fee3ea00640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x555cdfe16000 0x555cdf24e580 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=d19452ec7b526554 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.760+0000 7fee3ea00640 1 -- 172.21.15.62:0/2758633949 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x555cdfcf0b40 con 0x555cdfe16000 2024-03-23T11:26:47.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.760+0000 7fee4064b280 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x555cdf2b9800 0x555cdf24eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.761+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:2 12.14 12:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdf2b9c00 con 0x555cdf2b9800 2024-03-23T11:26:47.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.761+0000 7fee3d600640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x555cdf2b9800 0x555cdf24eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.762+0000 7fee3d600640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x555cdf2b9800 0x555cdf24eb00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=1547aedfd535843b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.769+0000 7fee3d600640 1 -- 172.21.15.62:0/2758633949 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv19 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x555cdfcf0d80 con 0x555cdf2b9800 2024-03-23T11:26:47.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.769+0000 7fee30e00640 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:3 12.14 12:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdf2d8000 con 0x555cdf2b9800 2024-03-23T11:26:47.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.769+0000 7fee30e00640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x555cdf2d8400 0x555cdf250c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.769+0000 7fee30e00640 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:4 12.1b 12:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdf2d8800 con 0x555cdf2d8400 2024-03-23T11:26:47.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.769+0000 7fee3d600640 1 -- 172.21.15.62:0/2758633949 <== osd.5 v2:172.21.15.120:6824/37642 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv16 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x555cdfcf0d80 con 0x555cdf2b9800 2024-03-23T11:26:47.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.770+0000 7fee3ea00640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x555cdf2d8400 0x555cdf250c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.771+0000 7fee3ea00640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x555cdf2d8400 0x555cdf250c00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=c7730bf68663f5d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.777+0000 7fee3ea00640 1 -- 172.21.15.62:0/2758633949 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x555cdfcf0b40 con 0x555cdf2d8400 2024-03-23T11:26:47.770 INFO:teuthology.orchestra.run.smithi062.stdout:old journal was 4209787~892 2024-03-23T11:26:47.770 INFO:teuthology.orchestra.run.smithi062.stdout:new journal start will be 8388608 (4177929 bytes past old end) 2024-03-23T11:26:47.770 INFO:teuthology.orchestra.run.smithi062.stdout:writing journal head 2024-03-23T11:26:47.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.777+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:5 12.14 12:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdf2d8c00 con 0x555cdf2b9800 2024-03-23T11:26:47.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.781+0000 7fee3d600640 1 -- 172.21.15.62:0/2758633949 <== osd.5 v2:172.21.15.120:6824/37642 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v92'20 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x555cdfcf0d80 con 0x555cdf2b9800 2024-03-23T11:26:47.774 INFO:teuthology.orchestra.run.smithi062.stdout:writing EResetJournal entry 2024-03-23T11:26:47.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.781+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:6 12.1b 12:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdf2d9000 con 0x555cdf2d8400 2024-03-23T11:26:47.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.781+0000 7fee4064b280 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x555cdf2d9400 0x555cdf24f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.781+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:7 12.b 12:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdf2d9800 con 0x555cdf2d9400 2024-03-23T11:26:47.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.781+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:8 12.1f 12:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdf2d9c00 con 0x555cdfe16000 2024-03-23T11:26:47.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.781+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:9 12.1a 12:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdf2d8000 con 0x555cdf2d8400 2024-03-23T11:26:47.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.781+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:10 12.1e 12:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdf2d8c00 con 0x555cdf2d8400 2024-03-23T11:26:47.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.781+0000 7fee4064b280 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x555cdf2b9c00 0x555cdf24f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.781+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:11 12.1d 12:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdfe36800 con 0x555cdf2b9c00 2024-03-23T11:26:47.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.781+0000 7fee3e000640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x555cdf2d9400 0x555cdf24f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.782+0000 7fee3d600640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x555cdf2b9c00 0x555cdf24f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.783+0000 7fee3e000640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x555cdf2d9400 0x555cdf24f080 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=830d8bc5d072465 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.784+0000 7fee3d600640 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x555cdf2b9c00 0x555cdf24f600 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=35d1deabf39d1c39 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.784+0000 7fee3ea00640 1 -- 172.21.15.62:0/2758633949 <== osd.4 v2:172.21.15.120:6803/37640 2 ==== osd_op_reply(6 200.00000002 [delete] v92'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x555cdfcf0b40 con 0x555cdf2d8400 2024-03-23T11:26:47.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.784+0000 7fee3ea00640 1 -- 172.21.15.62:0/2758633949 <== osd.4 v2:172.21.15.120:6803/37640 3 ==== osd_op_reply(9 200.00000005 [delete] v92'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x555cdfcf0b40 con 0x555cdf2d8400 2024-03-23T11:26:47.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.784+0000 7fee3ea00640 1 -- 172.21.15.62:0/2758633949 <== osd.6 v2:172.21.15.120:6800/37641 2 ==== osd_op_reply(8 200.00000004 [delete] v92'7 uv6 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x555cdfcf0b40 con 0x555cdfe16000 2024-03-23T11:26:47.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.785+0000 7fee3ea00640 1 -- 172.21.15.62:0/2758633949 <== osd.4 v2:172.21.15.120:6803/37640 4 ==== osd_op_reply(10 200.00000006 [delete] v92'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x555cdfcf0b40 con 0x555cdf2d8400 2024-03-23T11:26:47.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.785+0000 7fee3e000640 1 -- 172.21.15.62:0/2758633949 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== osd_op_reply(7 200.00000003 [delete] v92'4 uv3 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x555cdfcf1440 con 0x555cdf2d9400 2024-03-23T11:26:47.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.785+0000 7fee30e00640 1 -- 172.21.15.62:0/2758633949 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:12 12.1b 12:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) v8 -- 0x555cdf2d8800 con 0x555cdf2d8400 2024-03-23T11:26:47.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.786+0000 7fee3d600640 1 -- 172.21.15.62:0/2758633949 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== osd_op_reply(11 200.00000007 [delete] v92'3 uv2 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x555cdfcf0d80 con 0x555cdf2b9c00 2024-03-23T11:26:47.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee3ea00640 1 -- 172.21.15.62:0/2758633949 <== osd.4 v2:172.21.15.120:6803/37640 5 ==== osd_op_reply(12 200.00000002 [write 0~48] v92'3 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x555cdfcf0b40 con 0x555cdf2d8400 2024-03-23T11:26:47.782 INFO:teuthology.orchestra.run.smithi062.stdout:done 2024-03-23T11:26:47.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x555cdf2b9c00 msgr2=0x555cdf24f600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x555cdf2b9c00 0x555cdf24f600 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x555cdf2d9400 msgr2=0x555cdf24f080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x555cdf2d9400 0x555cdf24f080 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x555cdf2d8400 msgr2=0x555cdf250c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x555cdf2d8400 0x555cdf250c00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x555cdf2b9800 msgr2=0x555cdf24eb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x555cdf2b9800 0x555cdf24eb00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x555cdfe16000 msgr2=0x555cdf24e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x555cdfe16000 0x555cdf24e580 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee3d600640 1 -- 172.21.15.62:0/2758633949 reap_dead start 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe16800 msgr2=0x555cdef83080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee3d600640 1 -- 172.21.15.62:0/2758633949 reap_dead start 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 --2- 172.21.15.62:0/2758633949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe16800 0x555cdef83080 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x555cdf225380 tx=0x555cdf25ac60 comp rx=0 tx=0).stop 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee3d600640 1 -- 172.21.15.62:0/2758633949 reap_dead start 2024-03-23T11:26:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 shutdown_connections 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 >> 172.21.15.62:0/2758633949 conn(0x555cdfe17000 msgr2=0x555cdefdf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 shutdown_connections 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.789+0000 7fee4064b280 1 -- 172.21.15.62:0/2758633949 wait complete. 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 -- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x555cdfe36c00 msgr2=0x555cdf24e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x555cdfe36c00 0x555cdf24e000 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x555cdfc60240 tx=0x555cdf25acc0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 -- 172.21.15.62:0/2217407699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe17400 msgr2=0x555cdef84c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe17400 0x555cdef84c00 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x555cdfe26cc0 tx=0x555cdfdcac60 comp rx=0 tx=0).stop 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 -- 172.21.15.62:0/2217407699 shutdown_connections 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x555cdfe36c00 0x555cdf24e000 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfe17400 0x555cdef84c00 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfcca000 0x555cdef85180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 --2- 172.21.15.62:0/2217407699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9800 0x555cdef85700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 -- 172.21.15.62:0/2217407699 >> 172.21.15.62:0/2217407699 conn(0x555cdfcca400 msgr2=0x555cdefdf800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:47.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 -- 172.21.15.62:0/2217407699 shutdown_connections 2024-03-23T11:26:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.790+0000 7fee4064b280 1 -- 172.21.15.62:0/2217407699 wait complete. 2024-03-23T11:26:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.791+0000 7fee4064b280 1 -- 172.21.15.62:0/1806442328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfccbc00 msgr2=0x555cdef83600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.791+0000 7fee4064b280 1 --2- 172.21.15.62:0/1806442328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfccbc00 0x555cdef83600 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x555cdfc60210 tx=0x555cdfcfcc00 comp rx=0 tx=0).stop 2024-03-23T11:26:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.791+0000 7fee4064b280 1 -- 172.21.15.62:0/1806442328 shutdown_connections 2024-03-23T11:26:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.791+0000 7fee4064b280 1 --2- 172.21.15.62:0/1806442328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555cdfccbc00 0x555cdef83600 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.791+0000 7fee4064b280 1 --2- 172.21.15.62:0/1806442328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555cdeef9c00 0x555cdef82b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.791+0000 7fee4064b280 1 --2- 172.21.15.62:0/1806442328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdfcca800 0x555cdef83b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.791+0000 7fee4064b280 1 -- 172.21.15.62:0/1806442328 >> 172.21.15.62:0/1806442328 conn(0x555cdeef9000 msgr2=0x555cdefde000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:47.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.791+0000 7fee4064b280 1 -- 172.21.15.62:0/1806442328 shutdown_connections 2024-03-23T11:26:47.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.791+0000 7fee4064b280 1 -- 172.21.15.62:0/1806442328 wait complete. 2024-03-23T11:26:47.790 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset'] in time 0:00:00.151433, result: old journal was 4209787~892 new journal start will be 8388608 (4177929 bytes past old end) writing journal head writing EResetJournal entry done 2024-03-23T11:26:47.790 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-23T11:26:47.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.996+0000 7f5224c00640 1 Processor -- start 2024-03-23T11:26:47.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.996+0000 7f5224c00640 1 -- start start 2024-03-23T11:26:47.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.998+0000 7f5224c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f522015c830 0x7f52200ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.998+0000 7f5224c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52200ce250 0x7f52200ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.998+0000 7f5224c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52201608d0 con 0x7f5220160310 2024-03-23T11:26:47.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.998+0000 7f5224c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52200addb0 con 0x7f522015c830 2024-03-23T11:26:47.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.998+0000 7f5224c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522015cc10 con 0x7f52200ce250 2024-03-23T11:26:47.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.998+0000 7f521f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f522015c830 0x7f52200ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.998+0000 7f5217e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5220160310 0x7f52200ab980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40674/0 (socket says 172.21.15.62:40674) 2024-03-23T11:26:47.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.998+0000 7f5217e00640 1 -- 172.21.15.62:0/2773860464 learned_addr learned my addr 172.21.15.62:0/2773860464 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:47.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.998+0000 7f521f400640 1 --2- 172.21.15.62:0/2773860464 >> v2:172.21.15.62:3300/0 conn(0x7f522015c830 0x7f52200ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36754/0 (socket says 172.21.15.62:36754) 2024-03-23T11:26:47.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.999+0000 7f521fe00640 1 --2- 172.21.15.62:0/2773860464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52200ce250 0x7f52200ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.999+0000 7f521f400640 1 -- 172.21.15.62:0/2773860464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52200ce250 msgr2=0x7f52200ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:47.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.999+0000 7f521f400640 1 --2- 172.21.15.62:0/2773860464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52200ce250 0x7f52200ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.999+0000 7f521f400640 1 -- 172.21.15.62:0/2773860464 >> v1:172.21.15.62:6789/0 conn(0x7f5220160310 legacy=0x7f52200ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.999+0000 7f521f400640 1 -- 172.21.15.62:0/2773860464 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5220160bb0 con 0x7f522015c830 2024-03-23T11:26:47.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:47.999+0000 7f521ea00640 1 -- 172.21.15.62:0/2773860464 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684425877 0 0) 0x7f52201608d0 con 0x7f5220160310 2024-03-23T11:26:47.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.000+0000 7f521f400640 1 --2- 172.21.15.62:0/2773860464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522015c830 0x7f52200ad310 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f52100097d0 tx=0x7f521002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4a7f25169b5e3a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.001+0000 7f521ea00640 1 -- 172.21.15.62:0/2773860464 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5210031800 con 0x7f522015c830 2024-03-23T11:26:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.001+0000 7f521ea00640 1 -- 172.21.15.62:0/2773860464 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5210031960 con 0x7f522015c830 2024-03-23T11:26:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.001+0000 7f521ea00640 1 -- 172.21.15.62:0/2773860464 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5210031cf0 con 0x7f522015c830 2024-03-23T11:26:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.001+0000 7f5224c00640 1 -- 172.21.15.62:0/2773860464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522015c830 msgr2=0x7f52200ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.001+0000 7f5224c00640 1 --2- 172.21.15.62:0/2773860464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522015c830 0x7f52200ad310 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f52100097d0 tx=0x7f521002b8a0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.001+0000 7f5224c00640 1 -- 172.21.15.62:0/2773860464 shutdown_connections 2024-03-23T11:26:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.001+0000 7f5224c00640 1 --2- 172.21.15.62:0/2773860464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52200ce250 0x7f52200ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.001+0000 7f5224c00640 1 --2- 172.21.15.62:0/2773860464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522015c830 0x7f52200ad310 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.002+0000 7f5224c00640 1 -- 172.21.15.62:0/2773860464 >> 172.21.15.62:0/2773860464 conn(0x7f5220066050 msgr2=0x7f5220069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.002+0000 7f5224c00640 1 -- 172.21.15.62:0/2773860464 shutdown_connections 2024-03-23T11:26:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.002+0000 7f5224c00640 1 -- 172.21.15.62:0/2773860464 wait complete. 2024-03-23T11:26:47.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.003+0000 7f5224c00640 1 Processor -- start 2024-03-23T11:26:47.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.004+0000 7f5224c00640 1 -- start start 2024-03-23T11:26:47.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.004+0000 7f5224c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52200ce250 0x7f52201408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f5224c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f522015c830 0x7f5220141100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f5224c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5220160310 0x7f5220151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f5224c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522015d2c0 con 0x7f52200ce250 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f5224c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5220140e20 con 0x7f522015c830 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f5224c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522015d510 con 0x7f5220160310 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f521f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52200ce250 0x7f52201408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f521f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52200ce250 0x7f52201408e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36756/0 (socket says 172.21.15.62:36756) 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f521f400640 1 -- 172.21.15.62:0/99449666 learned_addr learned my addr 172.21.15.62:0/99449666 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f521fe00640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5220160310 0x7f5220151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f521f400640 1 -- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5220160310 msgr2=0x7f5220151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f521f400640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5220160310 0x7f5220151c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f521f400640 1 -- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f522015c830 msgr2=0x7f5220141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f521f400640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f522015c830 0x7f5220141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.005+0000 7f521f400640 1 -- 172.21.15.62:0/99449666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5210009400 con 0x7f52200ce250 2024-03-23T11:26:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.006+0000 7f521f400640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52200ce250 0x7f52201408e0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f52100098f0 tx=0x7f52100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:48.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.006+0000 7f5217400640 1 -- 172.21.15.62:0/99449666 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521003bd30 con 0x7f52200ce250 2024-03-23T11:26:48.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.006+0000 7f5217400640 1 -- 172.21.15.62:0/99449666 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52100453f0 con 0x7f52200ce250 2024-03-23T11:26:48.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.006+0000 7f5217400640 1 -- 172.21.15.62:0/99449666 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52100457a0 con 0x7f52200ce250 2024-03-23T11:26:48.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.006+0000 7f5224c00640 1 -- 172.21.15.62:0/99449666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522015d7f0 con 0x7f52200ce250 2024-03-23T11:26:48.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.006+0000 7f5224c00640 1 -- 172.21.15.62:0/99449666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522015ddb0 con 0x7f52200ce250 2024-03-23T11:26:48.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.008+0000 7f5224c00640 1 -- 172.21.15.62:0/99449666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522015dfa0 con 0x7f52200ce250 2024-03-23T11:26:48.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.008+0000 7f5217400640 1 -- 172.21.15.62:0/99449666 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f521004e450 con 0x7f52200ce250 2024-03-23T11:26:48.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.009+0000 7f5217400640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51fc06a7d0 0x7f51fc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:48.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.009+0000 7f5217400640 1 -- 172.21.15.62:0/99449666 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f521002e070 con 0x7f52200ce250 2024-03-23T11:26:48.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.010+0000 7f5217e00640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51fc06a7d0 0x7f51fc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:48.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.011+0000 7f5217e00640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51fc06a7d0 0x7f51fc06cc90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5208007fd0 tx=0x7f5208007920 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:48.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.014+0000 7f5217400640 1 -- 172.21.15.62:0/99449666 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f52100de580 con 0x7f52200ce250 2024-03-23T11:26:48.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.254+0000 7f5224c00640 1 -- 172.21.15.62:0/99449666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f5220066bb0 con 0x7f52200ce250 2024-03-23T11:26:48.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.517+0000 7f5217400640 1 -- 172.21.15.62:0/99449666 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v59) v1 ==== 162+0+0 (secure 0 0 0) 0x7f52100350b0 con 0x7f52200ce250 2024-03-23T11:26:48.511 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-23T11:26:48.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.521+0000 7f5224c00640 1 -- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51fc06a7d0 msgr2=0x7f51fc06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:48.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.521+0000 7f5224c00640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51fc06a7d0 0x7f51fc06cc90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5208007fd0 tx=0x7f5208007920 comp rx=0 tx=0).stop 2024-03-23T11:26:48.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.522+0000 7f5224c00640 1 -- 172.21.15.62:0/99449666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52200ce250 msgr2=0x7f52201408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:48.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.522+0000 7f5224c00640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52200ce250 0x7f52201408e0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f52100098f0 tx=0x7f52100046c0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.522+0000 7f5224c00640 1 -- 172.21.15.62:0/99449666 shutdown_connections 2024-03-23T11:26:48.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.522+0000 7f5224c00640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51fc06a7d0 0x7f51fc06cc90 secure :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5208007fd0 tx=0x7f5208007920 comp rx=0 tx=0).stop 2024-03-23T11:26:48.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.522+0000 7f5224c00640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5220160310 0x7f5220151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.522+0000 7f5224c00640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f522015c830 0x7f5220141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.522+0000 7f5224c00640 1 --2- 172.21.15.62:0/99449666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52200ce250 0x7f52201408e0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.522+0000 7f5224c00640 1 -- 172.21.15.62:0/99449666 >> 172.21.15.62:0/99449666 conn(0x7f5220066050 msgr2=0x7f52200adf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:48.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.522+0000 7f5224c00640 1 -- 172.21.15.62:0/99449666 shutdown_connections 2024-03-23T11:26:48.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.522+0000 7f5224c00640 1 -- 172.21.15.62:0/99449666 wait complete. 2024-03-23T11:26:48.538 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:26:48.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.742+0000 7fbf28a00640 1 Processor -- start 2024-03-23T11:26:48.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.742+0000 7fbf28a00640 1 -- start start 2024-03-23T11:26:48.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.743+0000 7fbf28a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbf24070090 0x7fbf24070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:48.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.743+0000 7fbf28a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf240b2820 0x7fbf240b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:48.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf28a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240b31b0 con 0x7fbf24143dd0 2024-03-23T11:26:48.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf28a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf24070a60 con 0x7fbf24070090 2024-03-23T11:26:48.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf28a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240cd140 con 0x7fbf240b2820 2024-03-23T11:26:48.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf19e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbf24070090 0x7fbf24070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:48.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf19e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbf24070090 0x7fbf24070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36762/0 (socket says 172.21.15.62:36762) 2024-03-23T11:26:48.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf19e00640 1 -- 172.21.15.62:0/3718741973 learned_addr learned my addr 172.21.15.62:0/3718741973 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:48.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf23200640 1 --2- 172.21.15.62:0/3718741973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf240b2820 0x7fbf240b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:48.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf19e00640 1 -- 172.21.15.62:0/3718741973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf240b2820 msgr2=0x7fbf240b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:48.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf19e00640 1 --2- 172.21.15.62:0/3718741973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf240b2820 0x7fbf240b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf19e00640 1 -- 172.21.15.62:0/3718741973 >> v1:172.21.15.120:6789/0 conn(0x7fbf24143dd0 legacy=0x7fbf2415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:48.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf19e00640 1 -- 172.21.15.62:0/3718741973 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240cd3e0 con 0x7fbf24070090 2024-03-23T11:26:48.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.744+0000 7fbf23200640 1 --2- 172.21.15.62:0/3718741973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf240b2820 0x7fbf240b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:48.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.745+0000 7fbf19e00640 1 --2- 172.21.15.62:0/3718741973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf24070090 0x7fbf24070520 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c009870 tx=0x7fbf0c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a84cffd8110f18 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:48.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.746+0000 7fbf21e00640 1 -- 172.21.15.62:0/3718741973 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c031640 con 0x7fbf24070090 2024-03-23T11:26:48.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.746+0000 7fbf21e00640 1 -- 172.21.15.62:0/3718741973 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0c0317a0 con 0x7fbf24070090 2024-03-23T11:26:48.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.746+0000 7fbf21e00640 1 -- 172.21.15.62:0/3718741973 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c031b30 con 0x7fbf24070090 2024-03-23T11:26:48.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.746+0000 7fbf28a00640 1 -- 172.21.15.62:0/3718741973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf24070090 msgr2=0x7fbf24070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:48.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.746+0000 7fbf28a00640 1 --2- 172.21.15.62:0/3718741973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf24070090 0x7fbf24070520 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c009870 tx=0x7fbf0c02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.747+0000 7fbf28a00640 1 -- 172.21.15.62:0/3718741973 shutdown_connections 2024-03-23T11:26:48.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.747+0000 7fbf28a00640 1 --2- 172.21.15.62:0/3718741973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf240b2820 0x7fbf240b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.747+0000 7fbf28a00640 1 --2- 172.21.15.62:0/3718741973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf24070090 0x7fbf24070520 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.747+0000 7fbf28a00640 1 -- 172.21.15.62:0/3718741973 >> 172.21.15.62:0/3718741973 conn(0x7fbf24066050 msgr2=0x7fbf24069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:48.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.747+0000 7fbf28a00640 1 -- 172.21.15.62:0/3718741973 shutdown_connections 2024-03-23T11:26:48.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.747+0000 7fbf28a00640 1 -- 172.21.15.62:0/3718741973 wait complete. 2024-03-23T11:26:48.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.748+0000 7fbf28a00640 1 Processor -- start 2024-03-23T11:26:48.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.749+0000 7fbf28a00640 1 -- start start 2024-03-23T11:26:48.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf28a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf24070090 0x7fbf2413ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:48.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf28a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf240b2820 0x7fbf2406c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:48.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf28a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf24143dd0 0x7fbf2406c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:48.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf28a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240b07e0 con 0x7fbf24143dd0 2024-03-23T11:26:48.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf28a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240afb60 con 0x7fbf24070090 2024-03-23T11:26:48.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf28a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240afe40 con 0x7fbf240b2820 2024-03-23T11:26:48.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf23200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf24143dd0 0x7fbf2406c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:48.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf23200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf24143dd0 0x7fbf2406c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36774/0 (socket says 172.21.15.62:36774) 2024-03-23T11:26:48.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf23200640 1 -- 172.21.15.62:0/3462561890 learned_addr learned my addr 172.21.15.62:0/3462561890 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:48.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf22800640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf24070090 0x7fbf2413ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf19e00640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf240b2820 0x7fbf2406c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf23200640 1 -- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf240b2820 msgr2=0x7fbf2406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf23200640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf240b2820 0x7fbf2406c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf23200640 1 -- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf24070090 msgr2=0x7fbf2413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf23200640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf24070090 0x7fbf2413ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf23200640 1 -- 172.21.15.62:0/3462561890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c0094d0 con 0x7fbf24143dd0 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.750+0000 7fbf22800640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf24070090 0x7fbf2413ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.751+0000 7fbf23200640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf24143dd0 0x7fbf2406c680 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c00c7f0 tx=0x7fbf1c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.751+0000 7fbf1b400640 1 -- 172.21.15.62:0/3462561890 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c007820 con 0x7fbf24143dd0 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.751+0000 7fbf1b400640 1 -- 172.21.15.62:0/3462561890 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1c007980 con 0x7fbf24143dd0 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.751+0000 7fbf28a00640 1 -- 172.21.15.62:0/3462561890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf240b0120 con 0x7fbf24143dd0 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.751+0000 7fbf1b400640 1 -- 172.21.15.62:0/3462561890 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c007d70 con 0x7fbf24143dd0 2024-03-23T11:26:48.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.751+0000 7fbf28a00640 1 -- 172.21.15.62:0/3462561890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf240b0c00 con 0x7fbf24143dd0 2024-03-23T11:26:48.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.752+0000 7fbf28a00640 1 -- 172.21.15.62:0/3462561890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf24066bb0 con 0x7fbf24143dd0 2024-03-23T11:26:48.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.753+0000 7fbf1b400640 1 -- 172.21.15.62:0/3462561890 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbf1c0113f0 con 0x7fbf24143dd0 2024-03-23T11:26:48.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.754+0000 7fbf1b400640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf0006a7d0 0x7fbf0006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:48.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.754+0000 7fbf1b400640 1 -- 172.21.15.62:0/3462561890 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fbf1c0891a0 con 0x7fbf24143dd0 2024-03-23T11:26:48.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.755+0000 7fbf22800640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf0006a7d0 0x7fbf0006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:48.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.756+0000 7fbf22800640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf0006a7d0 0x7fbf0006cc90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbf2406cf10 tx=0x7fbf14009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:48.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.759+0000 7fbf1b400640 1 -- 172.21.15.62:0/3462561890 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf1c054560 con 0x7fbf24143dd0 2024-03-23T11:26:48.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.996+0000 7fbf28a00640 1 -- 172.21.15.62:0/3462561890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf24070710 con 0x7fbf24143dd0 2024-03-23T11:26:48.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:48.998+0000 7fbf1b400640 1 -- 172.21.15.62:0/3462561890 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fbf1c057b80 con 0x7fbf24143dd0 2024-03-23T11:26:48.991 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:48.991 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":60,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:26:48.518614+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":60,"state":"up:replay","state_seq":1,"addr":"172.21.15.62:6833/2573443114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2573443114},{"type":"v1","addr":"172.21.15.62:6833","nonce":2573443114}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:26:48.991 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 60 2024-03-23T11:26:48.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.000+0000 7fbf28a00640 1 -- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf0006a7d0 msgr2=0x7fbf0006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:48.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.000+0000 7fbf28a00640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf0006a7d0 0x7fbf0006cc90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbf2406cf10 tx=0x7fbf14009040 comp rx=0 tx=0).stop 2024-03-23T11:26:48.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.000+0000 7fbf28a00640 1 -- 172.21.15.62:0/3462561890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf24143dd0 msgr2=0x7fbf2406c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:48.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.000+0000 7fbf28a00640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf24143dd0 0x7fbf2406c680 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c00c7f0 tx=0x7fbf1c00ccb0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.000+0000 7fbf28a00640 1 -- 172.21.15.62:0/3462561890 shutdown_connections 2024-03-23T11:26:48.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.000+0000 7fbf28a00640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf0006a7d0 0x7fbf0006cc90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.000+0000 7fbf28a00640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf24143dd0 0x7fbf2406c680 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.000+0000 7fbf28a00640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf240b2820 0x7fbf2406c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.000+0000 7fbf28a00640 1 --2- 172.21.15.62:0/3462561890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf24070090 0x7fbf2413ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:48.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.000+0000 7fbf28a00640 1 -- 172.21.15.62:0/3462561890 >> 172.21.15.62:0/3462561890 conn(0x7fbf24066050 msgr2=0x7fbf24069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:48.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.001+0000 7fbf28a00640 1 -- 172.21.15.62:0/3462561890 shutdown_connections 2024-03-23T11:26:48.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:49.001+0000 7fbf28a00640 1 -- 172.21.15.62:0/3462561890 wait complete. 2024-03-23T11:26:49.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:26:25.131291+0000', 'modified': '2024-03-23T11:26:48.518614+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 92, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7578': {'gid': 7578, 'name': 'a', 'rank': 0, 'incarnation': 60, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.62:6833/2573443114', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 2573443114}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 2573443114}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:26:49.415 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7578:up:replay 2024-03-23T11:26:50.417 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:26:50.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.653+0000 7fd1b8a00640 1 Processor -- start 2024-03-23T11:26:50.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.653+0000 7fd1b8a00640 1 -- start start 2024-03-23T11:26:50.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.655+0000 7fd1b8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b415fcf0 0x7fd1b40d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:50.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.655+0000 7fd1b8a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd1b4160420 0x7fd1b4160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:50.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.655+0000 7fd1b8a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b4160dc0 con 0x7fd1b40d06d0 2024-03-23T11:26:50.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.655+0000 7fd1b8a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b40d3540 con 0x7fd1b4160420 2024-03-23T11:26:50.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.655+0000 7fd1b8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b40ce510 con 0x7fd1b415fcf0 2024-03-23T11:26:50.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.655+0000 7fd1abe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd1b4160420 0x7fd1b4160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:50.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.655+0000 7fd1abe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd1b4160420 0x7fd1b4160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36788/0 (socket says 172.21.15.62:36788) 2024-03-23T11:26:50.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.655+0000 7fd1abe00640 1 -- 172.21.15.62:0/2121259033 learned_addr learned my addr 172.21.15.62:0/2121259033 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:50.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.656+0000 7fd1b3400640 1 --2- 172.21.15.62:0/2121259033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b415fcf0 0x7fd1b40d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:50.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.656+0000 7fd1abe00640 1 -- 172.21.15.62:0/2121259033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b415fcf0 msgr2=0x7fd1b40d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:50.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.656+0000 7fd1abe00640 1 --2- 172.21.15.62:0/2121259033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b415fcf0 0x7fd1b40d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:50.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.656+0000 7fd1abe00640 1 -- 172.21.15.62:0/2121259033 >> v1:172.21.15.120:6789/0 conn(0x7fd1b40d06d0 legacy=0x7fd1b40d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:50.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.656+0000 7fd1abe00640 1 -- 172.21.15.62:0/2121259033 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b40cdc70 con 0x7fd1b4160420 2024-03-23T11:26:50.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.656+0000 7fd1b3400640 1 --2- 172.21.15.62:0/2121259033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b415fcf0 0x7fd1b40d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.657+0000 7fd1abe00640 1 --2- 172.21.15.62:0/2121259033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1b4160420 0x7fd1b4160880 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009870 tx=0x7fd19c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864080055029e5b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.657+0000 7fd1b2a00640 1 -- 172.21.15.62:0/2121259033 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c0317c0 con 0x7fd1b4160420 2024-03-23T11:26:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.657+0000 7fd1b2a00640 1 -- 172.21.15.62:0/2121259033 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c031920 con 0x7fd1b4160420 2024-03-23T11:26:50.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.657+0000 7fd1b2a00640 1 -- 172.21.15.62:0/2121259033 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c031cb0 con 0x7fd1b4160420 2024-03-23T11:26:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.658+0000 7fd1b8a00640 1 -- 172.21.15.62:0/2121259033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1b4160420 msgr2=0x7fd1b4160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.658+0000 7fd1b8a00640 1 --2- 172.21.15.62:0/2121259033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1b4160420 0x7fd1b4160880 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009870 tx=0x7fd19c02b860 comp rx=0 tx=0).stop 2024-03-23T11:26:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.658+0000 7fd1b8a00640 1 -- 172.21.15.62:0/2121259033 shutdown_connections 2024-03-23T11:26:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.658+0000 7fd1b8a00640 1 --2- 172.21.15.62:0/2121259033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1b4160420 0x7fd1b4160880 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.658+0000 7fd1b8a00640 1 --2- 172.21.15.62:0/2121259033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b415fcf0 0x7fd1b40d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.658+0000 7fd1b8a00640 1 -- 172.21.15.62:0/2121259033 >> 172.21.15.62:0/2121259033 conn(0x7fd1b4089f60 msgr2=0x7fd1b40d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.658+0000 7fd1b8a00640 1 -- 172.21.15.62:0/2121259033 shutdown_connections 2024-03-23T11:26:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.658+0000 7fd1b8a00640 1 -- 172.21.15.62:0/2121259033 wait complete. 2024-03-23T11:26:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.660+0000 7fd1b8a00640 1 Processor -- start 2024-03-23T11:26:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.660+0000 7fd1b8a00640 1 -- start start 2024-03-23T11:26:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.661+0000 7fd1b8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b40d06d0 0x7fd1b4152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.661+0000 7fd1b8a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1b415fcf0 0x7fd1b41f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.661+0000 7fd1b8a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b4160420 0x7fd1b40cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.661+0000 7fd1b8a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b40c9850 con 0x7fd1b415fcf0 2024-03-23T11:26:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.661+0000 7fd1b8a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b40c9b30 con 0x7fd1b4160420 2024-03-23T11:26:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.661+0000 7fd1b8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b40c9e10 con 0x7fd1b40d06d0 2024-03-23T11:26:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.661+0000 7fd1abe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1b415fcf0 0x7fd1b41f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.661+0000 7fd1abe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1b415fcf0 0x7fd1b41f2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36792/0 (socket says 172.21.15.62:36792) 2024-03-23T11:26:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.661+0000 7fd1abe00640 1 -- 172.21.15.62:0/1717669890 learned_addr learned my addr 172.21.15.62:0/1717669890 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.662+0000 7fd1b3400640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b40d06d0 0x7fd1b4152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.662+0000 7fd1b3e00640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b4160420 0x7fd1b40cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.662+0000 7fd1abe00640 1 -- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b40d06d0 msgr2=0x7fd1b4152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.662+0000 7fd1abe00640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b40d06d0 0x7fd1b4152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.662+0000 7fd1abe00640 1 -- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b4160420 msgr2=0x7fd1b40cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.662+0000 7fd1abe00640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b4160420 0x7fd1b40cbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.662+0000 7fd1abe00640 1 -- 172.21.15.62:0/1717669890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c0094d0 con 0x7fd1b415fcf0 2024-03-23T11:26:50.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.662+0000 7fd1abe00640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1b415fcf0 0x7fd1b41f2810 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fd19c000c00 tx=0x7fd19c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.662+0000 7fd1ab400640 1 -- 172.21.15.62:0/1717669890 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c0329a0 con 0x7fd1b415fcf0 2024-03-23T11:26:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.663+0000 7fd1ab400640 1 -- 172.21.15.62:0/1717669890 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c032b20 con 0x7fd1b415fcf0 2024-03-23T11:26:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.663+0000 7fd1ab400640 1 -- 172.21.15.62:0/1717669890 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c0453f0 con 0x7fd1b415fcf0 2024-03-23T11:26:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.663+0000 7fd1b8a00640 1 -- 172.21.15.62:0/1717669890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b40ca0f0 con 0x7fd1b415fcf0 2024-03-23T11:26:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.663+0000 7fd1b8a00640 1 -- 172.21.15.62:0/1717669890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b4152e90 con 0x7fd1b415fcf0 2024-03-23T11:26:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.664+0000 7fd1b8a00640 1 -- 172.21.15.62:0/1717669890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b40c9f50 con 0x7fd1b415fcf0 2024-03-23T11:26:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.665+0000 7fd1ab400640 1 -- 172.21.15.62:0/1717669890 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd19c04d3f0 con 0x7fd1b415fcf0 2024-03-23T11:26:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.665+0000 7fd1ab400640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd19006a7d0 0x7fd19006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.666+0000 7fd1ab400640 1 -- 172.21.15.62:0/1717669890 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fd19c0aadf0 con 0x7fd1b415fcf0 2024-03-23T11:26:50.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.666+0000 7fd1b3400640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd19006a7d0 0x7fd19006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:50.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.667+0000 7fd1b3400640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd19006a7d0 0x7fd19006cc90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400a990 tx=0x7fd1a4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.670+0000 7fd1ab400640 1 -- 172.21.15.62:0/1717669890 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1b40c9f50 con 0x7fd1b415fcf0 2024-03-23T11:26:50.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.907+0000 7fd1b8a00640 1 -- 172.21.15.62:0/1717669890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1b40c9f50 con 0x7fd1b415fcf0 2024-03-23T11:26:50.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.908+0000 7fd1ab400640 1 -- 172.21.15.62:0/1717669890 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fd1b40c9f50 con 0x7fd1b415fcf0 2024-03-23T11:26:50.902 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:50.902 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":62,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:26:50.536082+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":60,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.62:6833/2573443114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2573443114},{"type":"v1","addr":"172.21.15.62:6833","nonce":2573443114}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:26:50.902 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 62 2024-03-23T11:26:50.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 -- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd19006a7d0 msgr2=0x7fd19006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:50.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd19006a7d0 0x7fd19006cc90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400a990 tx=0x7fd1a4008040 comp rx=0 tx=0).stop 2024-03-23T11:26:50.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 -- 172.21.15.62:0/1717669890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1b415fcf0 msgr2=0x7fd1b41f2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:50.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1b415fcf0 0x7fd1b41f2810 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fd19c000c00 tx=0x7fd19c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:26:50.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 -- 172.21.15.62:0/1717669890 shutdown_connections 2024-03-23T11:26:50.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd19006a7d0 0x7fd19006cc90 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:50.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1b415fcf0 0x7fd1b41f2810 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:50.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1b4160420 0x7fd1b40cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:50.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 --2- 172.21.15.62:0/1717669890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1b40d06d0 0x7fd1b4152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:50.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 -- 172.21.15.62:0/1717669890 >> 172.21.15.62:0/1717669890 conn(0x7fd1b4089f60 msgr2=0x7fd1b40d0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:50.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 -- 172.21.15.62:0/1717669890 shutdown_connections 2024-03-23T11:26:50.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:50.911+0000 7fd1b8a00640 1 -- 172.21.15.62:0/1717669890 wait complete. 2024-03-23T11:26:51.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:26:25.131291+0000', 'modified': '2024-03-23T11:26:50.536082+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 92, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7578': {'gid': 7578, 'name': 'a', 'rank': 0, 'incarnation': 60, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.62:6833/2573443114', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 2573443114}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 2573443114}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:26:51.331 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7578:up:rejoin 2024-03-23T11:26:52.332 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:26:52.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.562+0000 7f5ca5c00640 1 Processor -- start 2024-03-23T11:26:52.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.563+0000 7f5ca5c00640 1 -- start start 2024-03-23T11:26:52.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.564+0000 7f5ca5c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5ca00cd7c0 0x7f5ca00b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:52.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.564+0000 7f5ca5c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5ca006dca0 0x7f5ca006c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:52.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.564+0000 7f5ca5c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca00cad20 con 0x7f5ca00cdba0 2024-03-23T11:26:52.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.564+0000 7f5ca5c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca006e130 con 0x7f5ca006dca0 2024-03-23T11:26:52.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.564+0000 7f5ca5c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca015a8a0 con 0x7f5ca00cd7c0 2024-03-23T11:26:52.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.565+0000 7f5c97e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5ca006dca0 0x7f5ca006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:52.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.565+0000 7f5c97e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5ca006dca0 0x7f5ca006c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36802/0 (socket says 172.21.15.62:36802) 2024-03-23T11:26:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.565+0000 7f5c97e00640 1 -- 172.21.15.62:0/458952351 learned_addr learned my addr 172.21.15.62:0/458952351 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.565+0000 7f5c9ea00640 1 --2- 172.21.15.62:0/458952351 >> v2:172.21.15.120:3300/0 conn(0x7f5ca00cd7c0 0x7f5ca00b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.565+0000 7f5c97e00640 1 -- 172.21.15.62:0/458952351 >> v2:172.21.15.120:3300/0 conn(0x7f5ca00cd7c0 msgr2=0x7f5ca00b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.565+0000 7f5c97e00640 1 --2- 172.21.15.62:0/458952351 >> v2:172.21.15.120:3300/0 conn(0x7f5ca00cd7c0 0x7f5ca00b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.565+0000 7f5c97e00640 1 -- 172.21.15.62:0/458952351 >> v1:172.21.15.120:6789/0 conn(0x7f5ca00cdba0 legacy=0x7f5ca00cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.565+0000 7f5c97e00640 1 -- 172.21.15.62:0/458952351 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca015ab80 con 0x7f5ca006dca0 2024-03-23T11:26:52.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.566+0000 7f5c97e00640 1 --2- 172.21.15.62:0/458952351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ca006dca0 0x7f5ca006c300 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5ca006ac20 tx=0x7f5c8802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e406ed7cf58b592b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:52.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.567+0000 7f5c9e000640 1 -- 172.21.15.62:0/458952351 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c880317e0 con 0x7f5ca006dca0 2024-03-23T11:26:52.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.567+0000 7f5c9e000640 1 -- 172.21.15.62:0/458952351 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c88031940 con 0x7f5ca006dca0 2024-03-23T11:26:52.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.567+0000 7f5c9e000640 1 -- 172.21.15.62:0/458952351 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88031cd0 con 0x7f5ca006dca0 2024-03-23T11:26:52.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.567+0000 7f5ca5c00640 1 -- 172.21.15.62:0/458952351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ca006dca0 msgr2=0x7f5ca006c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:52.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.567+0000 7f5ca5c00640 1 --2- 172.21.15.62:0/458952351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ca006dca0 0x7f5ca006c300 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5ca006ac20 tx=0x7f5c8802b880 comp rx=0 tx=0).stop 2024-03-23T11:26:52.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.567+0000 7f5ca5c00640 1 -- 172.21.15.62:0/458952351 shutdown_connections 2024-03-23T11:26:52.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.567+0000 7f5ca5c00640 1 --2- 172.21.15.62:0/458952351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ca006dca0 0x7f5ca006c300 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:52.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.567+0000 7f5ca5c00640 1 --2- 172.21.15.62:0/458952351 >> v2:172.21.15.120:3300/0 conn(0x7f5ca00cd7c0 0x7f5ca00b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.567+0000 7f5ca5c00640 1 -- 172.21.15.62:0/458952351 >> 172.21.15.62:0/458952351 conn(0x7f5ca00b02e0 msgr2=0x7f5ca00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.567+0000 7f5ca5c00640 1 -- 172.21.15.62:0/458952351 shutdown_connections 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.567+0000 7f5ca5c00640 1 -- 172.21.15.62:0/458952351 wait complete. 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.569+0000 7f5ca5c00640 1 Processor -- start 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.569+0000 7f5ca5c00640 1 -- start start 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.570+0000 7f5ca5c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ca00cd7c0 0x7f5ca014ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.570+0000 7f5ca5c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ca00cdba0 0x7f5ca0154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.570+0000 7f5ca5c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0155120 0x7f5ca0151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.570+0000 7f5ca5c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca01556a0 con 0x7f5ca00cdba0 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.570+0000 7f5ca5c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca0155980 con 0x7f5ca0155120 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.570+0000 7f5ca5c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca0155c60 con 0x7f5ca00cd7c0 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.570+0000 7f5c97e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ca00cdba0 0x7f5ca0154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c97e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ca00cdba0 0x7f5ca0154be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36806/0 (socket says 172.21.15.62:36806) 2024-03-23T11:26:52.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c97e00640 1 -- 172.21.15.62:0/2682368321 learned_addr learned my addr 172.21.15.62:0/2682368321 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:52.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c9ea00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ca00cd7c0 0x7f5ca014ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:52.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c9f400640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0155120 0x7f5ca0151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:52.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c97e00640 1 -- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ca00cd7c0 msgr2=0x7f5ca014ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:52.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c97e00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ca00cd7c0 0x7f5ca014ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:52.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c97e00640 1 -- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0155120 msgr2=0x7f5ca0151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:52.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c97e00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0155120 0x7f5ca0151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:52.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c97e00640 1 -- 172.21.15.62:0/2682368321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c880094d0 con 0x7f5ca00cdba0 2024-03-23T11:26:52.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c9ea00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ca00cd7c0 0x7f5ca014ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:52.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c9f400640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0155120 0x7f5ca0151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:52.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.571+0000 7f5c97e00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ca00cdba0 0x7f5ca0154be0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f5c8802b8c0 tx=0x7f5c880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:52.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.572+0000 7f5c96a00640 1 -- 172.21.15.62:0/2682368321 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c880329c0 con 0x7f5ca00cdba0 2024-03-23T11:26:52.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.572+0000 7f5c96a00640 1 -- 172.21.15.62:0/2682368321 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c88032b40 con 0x7f5ca00cdba0 2024-03-23T11:26:52.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.572+0000 7f5c96a00640 1 -- 172.21.15.62:0/2682368321 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c880453f0 con 0x7f5ca00cdba0 2024-03-23T11:26:52.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.572+0000 7f5ca5c00640 1 -- 172.21.15.62:0/2682368321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca0155f40 con 0x7f5ca00cdba0 2024-03-23T11:26:52.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.572+0000 7f5ca5c00640 1 -- 172.21.15.62:0/2682368321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca014b640 con 0x7f5ca00cdba0 2024-03-23T11:26:52.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.573+0000 7f5ca5c00640 1 -- 172.21.15.62:0/2682368321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca0066c90 con 0x7f5ca00cdba0 2024-03-23T11:26:52.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.574+0000 7f5c96a00640 1 -- 172.21.15.62:0/2682368321 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5c8804d3f0 con 0x7f5ca00cdba0 2024-03-23T11:26:52.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.575+0000 7f5c96a00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c7c06a7a0 0x7f5c7c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:52.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.575+0000 7f5c96a00640 1 -- 172.21.15.62:0/2682368321 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f5c880aad70 con 0x7f5ca00cdba0 2024-03-23T11:26:52.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.576+0000 7f5c9ea00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c7c06a7a0 0x7f5c7c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:52.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.577+0000 7f5c9ea00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c7c06a7a0 0x7f5c7c06cc60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0154690 tx=0x7f5c90008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:52.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.579+0000 7f5c96a00640 1 -- 172.21.15.62:0/2682368321 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c88076150 con 0x7f5ca00cdba0 2024-03-23T11:26:52.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.817+0000 7f5ca5c00640 1 -- 172.21.15.62:0/2682368321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ca0151c20 con 0x7f5ca00cdba0 2024-03-23T11:26:52.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.818+0000 7f5c96a00640 1 -- 172.21.15.62:0/2682368321 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f5c88079770 con 0x7f5ca00cdba0 2024-03-23T11:26:52.811 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:26:52.811 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":63,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:26:51.549115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6833/2573443114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2573443114},{"type":"v1","addr":"172.21.15.62:6833","nonce":2573443114}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:26:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 63 2024-03-23T11:26:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.820+0000 7f5ca5c00640 1 -- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c7c06a7a0 msgr2=0x7f5c7c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.820+0000 7f5ca5c00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c7c06a7a0 0x7f5c7c06cc60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0154690 tx=0x7f5c90008040 comp rx=0 tx=0).stop 2024-03-23T11:26:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.820+0000 7f5ca5c00640 1 -- 172.21.15.62:0/2682368321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ca00cdba0 msgr2=0x7f5ca0154be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.820+0000 7f5ca5c00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ca00cdba0 0x7f5ca0154be0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f5c8802b8c0 tx=0x7f5c880046c0 comp rx=0 tx=0).stop 2024-03-23T11:26:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.820+0000 7f5ca5c00640 1 -- 172.21.15.62:0/2682368321 shutdown_connections 2024-03-23T11:26:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.820+0000 7f5ca5c00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5c7c06a7a0 0x7f5c7c06cc60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.820+0000 7f5ca5c00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ca00cdba0 0x7f5ca0154be0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.820+0000 7f5ca5c00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0155120 0x7f5ca0151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.820+0000 7f5ca5c00640 1 --2- 172.21.15.62:0/2682368321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5ca00cd7c0 0x7f5ca014ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.820+0000 7f5ca5c00640 1 -- 172.21.15.62:0/2682368321 >> 172.21.15.62:0/2682368321 conn(0x7f5ca00b02e0 msgr2=0x7f5ca015aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.821+0000 7f5ca5c00640 1 -- 172.21.15.62:0/2682368321 shutdown_connections 2024-03-23T11:26:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:52.821+0000 7f5ca5c00640 1 -- 172.21.15.62:0/2682368321 wait complete. 2024-03-23T11:26:53.236 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:26:25.131291+0000', 'modified': '2024-03-23T11:26:51.549115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 92, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7578': {'gid': 7578, 'name': 'a', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.62:6833/2573443114', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 2573443114}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 2573443114}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:26:53.236 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:26:53.236 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T11:26:53.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.473+0000 7f576fa00640 1 Processor -- start 2024-03-23T11:26:53.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.473+0000 7f576fa00640 1 -- start start 2024-03-23T11:26:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.475+0000 7f576fa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f576806c6e0 0x7f576806cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.475+0000 7f576fa00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57680cbc60 con 0x7f57680cafa0 2024-03-23T11:26:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.475+0000 7f576fa00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57680cbf40 con 0x7f576806c300 2024-03-23T11:26:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.475+0000 7f576fa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57680cd410 con 0x7f576806c6e0 2024-03-23T11:26:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.475+0000 7f5767e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f57680cafa0 0x7f57680cb430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40676/0 (socket says 172.21.15.62:40676) 2024-03-23T11:26:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.475+0000 7f5767e00640 1 -- 172.21.15.62:0/2325502326 learned_addr learned my addr 172.21.15.62:0/2325502326 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:53.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.475+0000 7f576da00640 1 --2- 172.21.15.62:0/2325502326 >> v2:172.21.15.120:3300/0 conn(0x7f576806c6e0 0x7f576806cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218993491 0 0) 0x7f57680cbc60 con 0x7f57680cafa0 2024-03-23T11:26:53.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5750003520 con 0x7f57680cafa0 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 902400921 0 0) 0x7f5750003520 con 0x7f57680cafa0 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 >> v2:172.21.15.120:3300/0 conn(0x7f576806c6e0 msgr2=0x7f576806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 --2- 172.21.15.62:0/2325502326 >> v2:172.21.15.120:3300/0 conn(0x7f576806c6e0 0x7f576806cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 >> v1:172.21.15.120:6789/0 conn(0x7f576806c300 legacy=0x7f57680ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57680cc220 con 0x7f57680cafa0 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f5754001670 con 0x7f57680cafa0 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 >> v1:172.21.15.62:6789/0 conn(0x7f57680cafa0 legacy=0x7f57680cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 --2- 172.21.15.62:0/2325502326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5750003970 0x7f5750003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 --2- 172.21.15.62:0/2325502326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57500047c0 0x7f5750004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 --2- 172.21.15.62:0/2325502326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5750005180 0x7f575000fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57680cd410 con 0x7f5750003970 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5750010430 con 0x7f57500047c0 2024-03-23T11:26:53.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57500105f0 con 0x7f5750005180 2024-03-23T11:26:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.476+0000 7f576d000640 1 --2- 172.21.15.62:0/2325502326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5750003970 0x7f5750003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.477+0000 7f5767e00640 1 --2- 172.21.15.62:0/2325502326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57500047c0 0x7f5750004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.477+0000 7f576da00640 1 --2- 172.21.15.62:0/2325502326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5750005180 0x7f575000fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.477+0000 7f576d000640 1 -- 172.21.15.62:0/2325502326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5750005180 msgr2=0x7f575000fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.477+0000 7f576d000640 1 --2- 172.21.15.62:0/2325502326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5750005180 0x7f575000fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.477+0000 7f576d000640 1 -- 172.21.15.62:0/2325502326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57500047c0 msgr2=0x7f5750004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.477+0000 7f576d000640 1 --2- 172.21.15.62:0/2325502326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57500047c0 0x7f5750004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.477+0000 7f576d000640 1 -- 172.21.15.62:0/2325502326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5750003520 con 0x7f5750003970 2024-03-23T11:26:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.477+0000 7f576da00640 1 -- 172.21.15.62:0/2325502326 reap_dead start 2024-03-23T11:26:53.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.479+0000 7f576d000640 1 --2- 172.21.15.62:0/2325502326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5750003970 0x7f5750003d90 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f575c001750 tx=0x7f57500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd837dc251816b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:53.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.479+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c0052d0 con 0x7f5750003970 2024-03-23T11:26:53.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.479+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c02e050 con 0x7f5750003970 2024-03-23T11:26:53.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.479+0000 7f5767400640 1 -- 172.21.15.62:0/2325502326 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c003830 con 0x7f5750003970 2024-03-23T11:26:53.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.479+0000 7f576fa00640 1 -- 172.21.15.62:0/2325502326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5750003970 msgr2=0x7f5750003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.479+0000 7f576fa00640 1 --2- 172.21.15.62:0/2325502326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5750003970 0x7f5750003d90 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f575c001750 tx=0x7f57500057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.480+0000 7f576fa00640 1 -- 172.21.15.62:0/2325502326 shutdown_connections 2024-03-23T11:26:53.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.480+0000 7f576fa00640 1 --2- 172.21.15.62:0/2325502326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5750003970 0x7f5750003d90 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.480+0000 7f576fa00640 1 -- 172.21.15.62:0/2325502326 >> 172.21.15.62:0/2325502326 conn(0x7f57680b02e0 msgr2=0x7f57680b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:53.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.480+0000 7f576fa00640 1 -- 172.21.15.62:0/2325502326 shutdown_connections 2024-03-23T11:26:53.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.480+0000 7f576fa00640 1 -- 172.21.15.62:0/2325502326 wait complete. 2024-03-23T11:26:53.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.482+0000 7f576fa00640 1 Processor -- start 2024-03-23T11:26:53.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.482+0000 7f576fa00640 1 -- start start 2024-03-23T11:26:53.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.483+0000 7f576fa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5750003970 0x7f576813ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.483+0000 7f576fa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57681ee860 0x7f57681f1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.483+0000 7f576fa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681f1980 0x7f576815cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.483+0000 7f576fa00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57681f2430 con 0x7f57681ee860 2024-03-23T11:26:53.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.483+0000 7f576fa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57681eddf0 con 0x7f57681f1980 2024-03-23T11:26:53.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.483+0000 7f576fa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576813d3b0 con 0x7f5750003970 2024-03-23T11:26:53.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.483+0000 7f5767e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57681ee860 0x7f57681f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f5767e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57681ee860 0x7f57681f1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36812/0 (socket says 172.21.15.62:36812) 2024-03-23T11:26:53.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f5767e00640 1 -- 172.21.15.62:0/3923635552 learned_addr learned my addr 172.21.15.62:0/3923635552 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:53.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f576d000640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5750003970 0x7f576813ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f576da00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681f1980 0x7f576815cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f5767e00640 1 -- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5750003970 msgr2=0x7f576813ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f5767e00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5750003970 0x7f576813ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f5767e00640 1 -- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681f1980 msgr2=0x7f576815cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f5767e00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681f1980 0x7f576815cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f5767e00640 1 -- 172.21.15.62:0/3923635552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57681f1e70 con 0x7f57681ee860 2024-03-23T11:26:53.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f576da00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681f1980 0x7f576815cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:53.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f576d000640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5750003970 0x7f576813ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:53.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.484+0000 7f5767e00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57681ee860 0x7f57681f1440 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f575400e700 tx=0x7f575400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:53.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.485+0000 7f5764c00640 1 -- 172.21.15.62:0/3923635552 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575400c6c0 con 0x7f57681ee860 2024-03-23T11:26:53.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.485+0000 7f5764c00640 1 -- 172.21.15.62:0/3923635552 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575400c820 con 0x7f57681ee860 2024-03-23T11:26:53.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.485+0000 7f576fa00640 1 -- 172.21.15.62:0/3923635552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57681f2150 con 0x7f57681ee860 2024-03-23T11:26:53.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.485+0000 7f5764c00640 1 -- 172.21.15.62:0/3923635552 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575400cbd0 con 0x7f57681ee860 2024-03-23T11:26:53.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.485+0000 7f576fa00640 1 -- 172.21.15.62:0/3923635552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57681ee340 con 0x7f57681ee860 2024-03-23T11:26:53.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.487+0000 7f5764c00640 1 -- 172.21.15.62:0/3923635552 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f575400cd30 con 0x7f57681ee860 2024-03-23T11:26:53.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.489+0000 7f5764c00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f574806a7a0 0x7f574806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.489+0000 7f5764c00640 1 -- 172.21.15.62:0/3923635552 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f5754088d00 con 0x7f57681ee860 2024-03-23T11:26:53.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.489+0000 7f576d000640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f574806a7a0 0x7f574806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.489+0000 7f57725e6740 1 Processor -- start 2024-03-23T11:26:53.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.490+0000 7f57725e6740 1 -- start start 2024-03-23T11:26:53.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.490+0000 7f576d000640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f574806a7a0 0x7f574806cc60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f575c033000 tx=0x7f575c009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:53.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.490+0000 7f57725e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.490+0000 7f57725e6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371a7410 0x55b6371a8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.490+0000 7f57725e6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a38b0 0x55b6371d36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f57725e6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b637114110 con 0x55b6371a38b0 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f57725e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6371118a0 con 0x55b6371f1390 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f57725e6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b63710d190 con 0x55b6371a7410 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f576da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a38b0 0x55b6371d36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f576da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a38b0 0x55b6371d36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36826/0 (socket says 172.21.15.62:36826) 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f576da00640 1 -- 172.21.15.62:0/1648721034 learned_addr learned my addr 172.21.15.62:0/1648721034 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f5767e00640 1 --2- 172.21.15.62:0/1648721034 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371a7410 0x55b6371a8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f576d000640 1 --2- 172.21.15.62:0/1648721034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f576da00640 1 -- 172.21.15.62:0/1648721034 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371a7410 msgr2=0x55b6371a8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f576da00640 1 --2- 172.21.15.62:0/1648721034 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371a7410 0x55b6371a8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f576da00640 1 -- 172.21.15.62:0/1648721034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 msgr2=0x55b6371a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f576da00640 1 --2- 172.21.15.62:0/1648721034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371a89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f576da00640 1 -- 172.21.15.62:0/1648721034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6370d8230 con 0x55b6371a38b0 2024-03-23T11:26:53.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.491+0000 7f576d000640 1 --2- 172.21.15.62:0/1648721034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371a89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:53.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.492+0000 7f576da00640 1 --2- 172.21.15.62:0/1648721034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a38b0 0x55b6371d36a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f576815e3a0 tx=0x7f5760009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99445e0a49e4f74b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.492+0000 7f5759600640 1 -- 172.21.15.62:0/1648721034 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5760004780 con 0x55b6371a38b0 2024-03-23T11:26:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.492+0000 7f5759600640 1 -- 172.21.15.62:0/1648721034 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5760015420 con 0x55b6371a38b0 2024-03-23T11:26:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.492+0000 7f5759600640 1 -- 172.21.15.62:0/1648721034 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57600157b0 con 0x55b6371a38b0 2024-03-23T11:26:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.493+0000 7f57725e6740 1 -- 172.21.15.62:0/1648721034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a38b0 msgr2=0x55b6371d36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.493+0000 7f57725e6740 1 --2- 172.21.15.62:0/1648721034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a38b0 0x55b6371d36a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f576815e3a0 tx=0x7f5760009750 comp rx=0 tx=0).stop 2024-03-23T11:26:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.493+0000 7f57725e6740 1 -- 172.21.15.62:0/1648721034 shutdown_connections 2024-03-23T11:26:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.493+0000 7f57725e6740 1 --2- 172.21.15.62:0/1648721034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a38b0 0x55b6371d36a0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.493+0000 7f57725e6740 1 --2- 172.21.15.62:0/1648721034 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371a7410 0x55b6371a8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.493+0000 7f57725e6740 1 --2- 172.21.15.62:0/1648721034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371a89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.493+0000 7f57725e6740 1 -- 172.21.15.62:0/1648721034 >> 172.21.15.62:0/1648721034 conn(0x55b6371a5b20 msgr2=0x55b6371ee6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:53.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.493+0000 7f57725e6740 1 -- 172.21.15.62:0/1648721034 shutdown_connections 2024-03-23T11:26:53.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.493+0000 7f57725e6740 1 -- 172.21.15.62:0/1648721034 wait complete. 2024-03-23T11:26:53.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.494+0000 7f57725e6740 1 -- start start 2024-03-23T11:26:53.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.494+0000 7f57725e6740 1 Processor -- start 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f57725e6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371a7410 0x55b6371d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f57725e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f57725e6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f57725e6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6370ae8c0 con 0x55b6371b9600 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f57725e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6370ae560 con 0x55b6371f1390 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f57725e6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b636e74f20 con 0x55b6371a7410 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f57725e6740 10 client.0 _ll_register_callbacks cb 0x55b637106800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f576da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f576da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36838/0 (socket says 172.21.15.62:36838) 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f576da00640 1 -- 172.21.15.62:0/654661328 learned_addr learned my addr 172.21.15.62:0/654661328 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f576d000640 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371a7410 0x55b6371d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f576da00640 1 -- 172.21.15.62:0/654661328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371a7410 msgr2=0x55b6371d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f5767e00640 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f576da00640 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371a7410 0x55b6371d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f576da00640 1 -- 172.21.15.62:0/654661328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 msgr2=0x55b6371ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f576da00640 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371ce9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.495+0000 7f576da00640 1 -- 172.21.15.62:0/654661328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5760011040 con 0x55b6371b9600 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.496+0000 7f576da00640 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ceee0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f5760009790 tx=0x7f5760004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b5ff55bc48a116 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.496+0000 7f574e000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.496+0000 7f574e000640 1 -- 172.21.15.62:0/654661328 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5760015b40 con 0x55b6371b9600 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.496+0000 7f574e000640 1 -- 172.21.15.62:0/654661328 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f5760004850 con 0x55b6371b9600 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.496+0000 7f57725e6740 1 -- 172.21.15.62:0/654661328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b637137280 con 0x55b6371b9600 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.496+0000 7f574e000640 1 -- 172.21.15.62:0/654661328 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5760015ca0 con 0x55b6371b9600 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.496+0000 7f57725e6740 10 client.7644 fetch_fsmap learned FSMap version 63 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.496+0000 7f57725e6740 1 -- 172.21.15.62:0/654661328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x55b63710b810 con 0x55b6371b9600 2024-03-23T11:26:53.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.497+0000 7f574e000640 1 -- 172.21.15.62:0/654661328 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b637137280 con 0x55b6371b9600 2024-03-23T11:26:53.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.497+0000 7f574e000640 1 -- 172.21.15.62:0/654661328 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5760033470 con 0x55b6371b9600 2024-03-23T11:26:53.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.497+0000 7f57725e6740 10 client.7644 fetch_fsmap finished waiting for FSMap version 63 2024-03-23T11:26:53.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.497+0000 7f57725e6740 10 client.7644 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:26:53.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.497+0000 7f57725e6740 20 client.7644 populate_metadata read hostname 'smithi062' 2024-03-23T11:26:53.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.497+0000 7f57725e6740 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b6371f2d70 0x55b6371f3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.497+0000 7f57725e6740 4 client.7644 mds_command: new command op to 7578 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:26:53.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.497+0000 7f57725e6740 1 -- 172.21.15.62:0/654661328 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b6370ff9d0 con 0x55b6371f2d70 2024-03-23T11:26:53.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.497+0000 7f576d000640 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b6371f2d70 0x55b6371f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.498+0000 7f576d000640 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b6371f2d70 0x55b6371f3150 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76dc3bad0e34892c server_cookie=e8907888a77471a5 in_seq=0 out_seq=0 2024-03-23T11:26:53.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.498+0000 7f574e000640 10 client.7644 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:53.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f574e000640 1 -- 172.21.15.62:0/654661328 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55b6370ff9d0 con 0x55b6371f2d70 2024-03-23T11:26:53.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f574e000640 10 client.7644 handle_command_reply: tid=0 2024-03-23T11:26:53.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 1 client.7644 shutdown 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 1 -- 172.21.15.62:0/654661328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 msgr2=0x55b6371ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ceee0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f5760009790 tx=0x7f5760004a40 comp rx=0 tx=0).stop 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 1 -- 172.21.15.62:0/654661328 shutdown_connections 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b6371f2d70 0x55b6371f3150 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ceee0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 1 --2- 172.21.15.62:0/654661328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371a7410 0x55b6371d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f574e000640 0 client.7644 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 1 -- 172.21.15.62:0/654661328 >> 172.21.15.62:0/654661328 conn(0x55b6371a38b0 msgr2=0x55b6371a2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 1 -- 172.21.15.62:0/654661328 shutdown_connections 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 1 -- 172.21.15.62:0/654661328 wait complete. 2024-03-23T11:26:53.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.499+0000 7f57725e6740 20 client.7644 trim_cache size 0 max 16384 2024-03-23T11:26:53.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.516+0000 7f57725e6740 1 Processor -- start 2024-03-23T11:26:53.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.517+0000 7f57725e6740 1 -- start start 2024-03-23T11:26:53.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.517+0000 7f57725e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371a7410 0x55b6371cad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.517+0000 7f57725e6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ed4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f57725e6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371f1390 0x55b6371a7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f57725e6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6371cf4b0 con 0x55b6371b9600 2024-03-23T11:26:53.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f57725e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6371f1210 con 0x55b6371a7410 2024-03-23T11:26:53.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f57725e6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6371f0b30 con 0x55b6371f1390 2024-03-23T11:26:53.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f5767e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ed4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f5767e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ed4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36846/0 (socket says 172.21.15.62:36846) 2024-03-23T11:26:53.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f5767e00640 1 -- 172.21.15.62:0/249140552 learned_addr learned my addr 172.21.15.62:0/249140552 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:53.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f576d000640 1 --2- 172.21.15.62:0/249140552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371a7410 0x55b6371cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f576da00640 1 --2- 172.21.15.62:0/249140552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371f1390 0x55b6371a7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f5767e00640 1 -- 172.21.15.62:0/249140552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371f1390 msgr2=0x55b6371a7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f5767e00640 1 --2- 172.21.15.62:0/249140552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371f1390 0x55b6371a7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f5767e00640 1 -- 172.21.15.62:0/249140552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371a7410 msgr2=0x55b6371cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f5767e00640 1 --2- 172.21.15.62:0/249140552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371a7410 0x55b6371cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f5767e00640 1 -- 172.21.15.62:0/249140552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6370adad0 con 0x55b6371b9600 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.518+0000 7f576d000640 1 --2- 172.21.15.62:0/249140552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371a7410 0x55b6371cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.519+0000 7f5767e00640 1 --2- 172.21.15.62:0/249140552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ed4c0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f5754001eb0 tx=0x7f57540584b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54cdf046657ce577 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.519+0000 7f575a000640 1 -- 172.21.15.62:0/249140552 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57540600e0 con 0x55b6371b9600 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.519+0000 7f575a000640 1 -- 172.21.15.62:0/249140552 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5754056090 con 0x55b6371b9600 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.519+0000 7f575a000640 1 -- 172.21.15.62:0/249140552 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57540571e0 con 0x55b6371b9600 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.520+0000 7f57725e6740 1 -- 172.21.15.62:0/249140552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 msgr2=0x55b6371ed4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.520+0000 7f57725e6740 1 --2- 172.21.15.62:0/249140552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ed4c0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f5754001eb0 tx=0x7f57540584b0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.520+0000 7f57725e6740 1 -- 172.21.15.62:0/249140552 shutdown_connections 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.520+0000 7f57725e6740 1 --2- 172.21.15.62:0/249140552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371b9600 0x55b6371ed4c0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.520+0000 7f57725e6740 1 --2- 172.21.15.62:0/249140552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371f1390 0x55b6371a7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.520+0000 7f57725e6740 1 --2- 172.21.15.62:0/249140552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371a7410 0x55b6371cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.520+0000 7f57725e6740 1 -- 172.21.15.62:0/249140552 >> 172.21.15.62:0/249140552 conn(0x55b6371a38b0 msgr2=0x55b6371d2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.520+0000 7f57725e6740 1 -- 172.21.15.62:0/249140552 shutdown_connections 2024-03-23T11:26:53.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.520+0000 7f57725e6740 1 -- 172.21.15.62:0/249140552 wait complete. 2024-03-23T11:26:53.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.520+0000 7f57725e6740 1 -- start start 2024-03-23T11:26:53.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.521+0000 7f57725e6740 1 Processor -- start 2024-03-23T11:26:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.521+0000 7f57725e6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a7410 0x55b637207a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.521+0000 7f57725e6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371b9600 0x55b6371b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f57725e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f57725e6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6371d0f70 con 0x55b6371a7410 2024-03-23T11:26:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f57725e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6371e43b0 con 0x55b6371f1390 2024-03-23T11:26:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f57725e6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6371e4690 con 0x55b6371b9600 2024-03-23T11:26:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f57725e6740 10 client.0 _ll_register_callbacks cb 0x7f5760009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:26:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f576d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a7410 0x55b637207a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f576d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a7410 0x55b637207a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36850/0 (socket says 172.21.15.62:36850) 2024-03-23T11:26:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f576d000640 1 -- 172.21.15.62:0/2538023023 learned_addr learned my addr 172.21.15.62:0/2538023023 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f576d000640 1 -- 172.21.15.62:0/2538023023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371b9600 msgr2=0x55b6371b63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f576d000640 1 --2- 172.21.15.62:0/2538023023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371b9600 0x55b6371b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f576d000640 1 -- 172.21.15.62:0/2538023023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 msgr2=0x55b6371d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f576d000640 1 --2- 172.21.15.62:0/2538023023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.522+0000 7f576d000640 1 -- 172.21.15.62:0/2538023023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57681ee340 con 0x55b6371a7410 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.523+0000 7f576d000640 1 --2- 172.21.15.62:0/2538023023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a7410 0x55b637207a50 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f575c001870 tx=0x7f575c001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69976c3304254 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.523+0000 7f574e000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.523+0000 7f574e000640 1 -- 172.21.15.62:0/2538023023 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c043040 con 0x55b6371a7410 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.523+0000 7f574e000640 1 -- 172.21.15.62:0/2538023023 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f575c04e0b0 con 0x55b6371a7410 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.523+0000 7f57725e6740 1 -- 172.21.15.62:0/2538023023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5760015420 con 0x55b6371a7410 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.523+0000 7f574e000640 1 -- 172.21.15.62:0/2538023023 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f575c007c00 con 0x55b6371a7410 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.523+0000 7f57725e6740 10 client.7650 fetch_fsmap learned FSMap version 63 2024-03-23T11:26:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.523+0000 7f57725e6740 1 -- 172.21.15.62:0/2538023023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x55b6370d2fe0 con 0x55b6371a7410 2024-03-23T11:26:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.523+0000 7f574e000640 1 -- 172.21.15.62:0/2538023023 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5760015420 con 0x55b6371a7410 2024-03-23T11:26:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.524+0000 7f574e000640 1 -- 172.21.15.62:0/2538023023 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f575c04e270 con 0x55b6371a7410 2024-03-23T11:26:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.524+0000 7f57725e6740 10 client.7650 fetch_fsmap finished waiting for FSMap version 63 2024-03-23T11:26:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.524+0000 7f57725e6740 10 client.7650 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:26:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.524+0000 7f57725e6740 20 client.7650 populate_metadata read hostname 'smithi062' 2024-03-23T11:26:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.524+0000 7f57725e6740 1 --2- 172.21.15.62:0/2538023023 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b6371ee710 0x55b6371eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.524+0000 7f57725e6740 4 client.7650 mds_command: new command op to 7578 tid=0[{"prefix": "status"}] 2024-03-23T11:26:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.524+0000 7f57725e6740 1 -- 172.21.15.62:0/2538023023 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b636e013c0 con 0x55b6371ee710 2024-03-23T11:26:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.524+0000 7f5767e00640 1 --2- 172.21.15.62:0/2538023023 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b6371ee710 0x55b6371eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.524+0000 7f5767e00640 1 --2- 172.21.15.62:0/2538023023 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b6371ee710 0x55b6371eeb10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3e90895c0c509b0 server_cookie=d0b1ab5bbe0f0327 in_seq=0 out_seq=0 2024-03-23T11:26:53.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.524+0000 7f574e000640 10 client.7650 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:53.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f574e000640 1 -- 172.21.15.62:0/2538023023 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55b636e013c0 con 0x55b6371ee710 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f574e000640 10 client.7650 handle_command_reply: tid=0 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 1 client.7650 shutdown 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 1 -- 172.21.15.62:0/2538023023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a7410 msgr2=0x55b637207a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 1 --2- 172.21.15.62:0/2538023023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a7410 0x55b637207a50 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f575c001870 tx=0x7f575c001190 comp rx=0 tx=0).stop 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 1 -- 172.21.15.62:0/2538023023 shutdown_connections 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 1 --2- 172.21.15.62:0/2538023023 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b6371ee710 0x55b6371eeb10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 1 --2- 172.21.15.62:0/2538023023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6371f1390 0x55b6371d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 1 --2- 172.21.15.62:0/2538023023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b6371b9600 0x55b6371b63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 1 --2- 172.21.15.62:0/2538023023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b6371a7410 0x55b637207a50 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f574e000640 0 client.7650 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 1 -- 172.21.15.62:0/2538023023 >> 172.21.15.62:0/2538023023 conn(0x55b6371a38b0 msgr2=0x55b6371d28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 1 -- 172.21.15.62:0/2538023023 shutdown_connections 2024-03-23T11:26:53.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 1 -- 172.21.15.62:0/2538023023 wait complete. 2024-03-23T11:26:53.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.525+0000 7f57725e6740 20 client.7650 trim_cache size 0 max 16384 2024-03-23T11:26:53.522 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:26:53.522 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:26:53.522 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:26:53.522 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 7578, 2024-03-23T11:26:53.522 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:26:53.522 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:26:53.522 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:26:53.522 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 4.9978638530000001, 2024-03-23T11:26:53.522 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 63, 2024-03-23T11:26:53.523 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 92, 2024-03-23T11:26:53.523 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 92, 2024-03-23T11:26:53.523 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 5.9151860579999997 2024-03-23T11:26:53.523 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:26:53.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 -- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f574806a7a0 msgr2=0x7f574806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f574806a7a0 0x7f574806cc60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f575c033000 tx=0x7f575c009030 comp rx=0 tx=0).stop 2024-03-23T11:26:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 -- 172.21.15.62:0/3923635552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57681ee860 msgr2=0x7f57681f1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57681ee860 0x7f57681f1440 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f575400e700 tx=0x7f575400ebc0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 -- 172.21.15.62:0/3923635552 shutdown_connections 2024-03-23T11:26:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f574806a7a0 0x7f574806cc60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57681ee860 0x7f57681f1440 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681f1980 0x7f576815cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 --2- 172.21.15.62:0/3923635552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5750003970 0x7f576813ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 -- 172.21.15.62:0/3923635552 >> 172.21.15.62:0/3923635552 conn(0x7f57680b02e0 msgr2=0x7f57680b13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 -- 172.21.15.62:0/3923635552 shutdown_connections 2024-03-23T11:26:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.526+0000 7f576fa00640 1 -- 172.21.15.62:0/3923635552 wait complete. 2024-03-23T11:26:53.536 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:26:53.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.748+0000 7fc8f8c00640 1 Processor -- start 2024-03-23T11:26:53.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.749+0000 7fc8f8c00640 1 -- start start 2024-03-23T11:26:53.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.750+0000 7fc8f8c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8f415fcf0 0x7fc8f40d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.750+0000 7fc8f8c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc8f40b6dc0 0x7fc8f41608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.750+0000 7fc8f8c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f40d3540 con 0x7fc8f4160420 2024-03-23T11:26:53.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.750+0000 7fc8f8c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f4160df0 con 0x7fc8f40b6dc0 2024-03-23T11:26:53.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.750+0000 7fc8f8c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f40ce510 con 0x7fc8f415fcf0 2024-03-23T11:26:53.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.750+0000 7fc8f3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc8f40b6dc0 0x7fc8f41608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.750+0000 7fc8f3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc8f40b6dc0 0x7fc8f41608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36864/0 (socket says 172.21.15.62:36864) 2024-03-23T11:26:53.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.750+0000 7fc8f3e00640 1 -- 172.21.15.62:0/4045259493 learned_addr learned my addr 172.21.15.62:0/4045259493 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:53.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.751+0000 7fc8f3400640 1 --2- 172.21.15.62:0/4045259493 >> v2:172.21.15.120:3300/0 conn(0x7fc8f415fcf0 0x7fc8f40d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.751+0000 7fc8f3e00640 1 -- 172.21.15.62:0/4045259493 >> v2:172.21.15.120:3300/0 conn(0x7fc8f415fcf0 msgr2=0x7fc8f40d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.751+0000 7fc8f3e00640 1 --2- 172.21.15.62:0/4045259493 >> v2:172.21.15.120:3300/0 conn(0x7fc8f415fcf0 0x7fc8f40d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.751+0000 7fc8f3e00640 1 -- 172.21.15.62:0/4045259493 >> v1:172.21.15.120:6789/0 conn(0x7fc8f4160420 legacy=0x7fc8f40d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.751+0000 7fc8f3e00640 1 -- 172.21.15.62:0/4045259493 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f40cdc70 con 0x7fc8f40b6dc0 2024-03-23T11:26:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.751+0000 7fc8f3400640 1 --2- 172.21.15.62:0/4045259493 >> v2:172.21.15.120:3300/0 conn(0x7fc8f415fcf0 0x7fc8f40d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.752+0000 7fc8f3e00640 1 --2- 172.21.15.62:0/4045259493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8f40b6dc0 0x7fc8f41608b0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4066480 tx=0x7fc8ec02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5363204cd4301964 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.752+0000 7fc8f2000640 1 -- 172.21.15.62:0/4045259493 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec02c9f0 con 0x7fc8f40b6dc0 2024-03-23T11:26:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.752+0000 7fc8f2000640 1 -- 172.21.15.62:0/4045259493 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec02cb50 con 0x7fc8f40b6dc0 2024-03-23T11:26:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.752+0000 7fc8f2000640 1 -- 172.21.15.62:0/4045259493 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec0343f0 con 0x7fc8f40b6dc0 2024-03-23T11:26:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.753+0000 7fc8f8c00640 1 -- 172.21.15.62:0/4045259493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8f40b6dc0 msgr2=0x7fc8f41608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.753+0000 7fc8f8c00640 1 --2- 172.21.15.62:0/4045259493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8f40b6dc0 0x7fc8f41608b0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4066480 tx=0x7fc8ec02be50 comp rx=0 tx=0).stop 2024-03-23T11:26:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.753+0000 7fc8f8c00640 1 -- 172.21.15.62:0/4045259493 shutdown_connections 2024-03-23T11:26:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.753+0000 7fc8f8c00640 1 --2- 172.21.15.62:0/4045259493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8f40b6dc0 0x7fc8f41608b0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.753+0000 7fc8f8c00640 1 --2- 172.21.15.62:0/4045259493 >> v2:172.21.15.120:3300/0 conn(0x7fc8f415fcf0 0x7fc8f40d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.753+0000 7fc8f8c00640 1 -- 172.21.15.62:0/4045259493 >> 172.21.15.62:0/4045259493 conn(0x7fc8f4089f60 msgr2=0x7fc8f40d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.753+0000 7fc8f8c00640 1 -- 172.21.15.62:0/4045259493 shutdown_connections 2024-03-23T11:26:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.753+0000 7fc8f8c00640 1 -- 172.21.15.62:0/4045259493 wait complete. 2024-03-23T11:26:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.755+0000 7fc8f8c00640 1 Processor -- start 2024-03-23T11:26:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.755+0000 7fc8f8c00640 1 -- start start 2024-03-23T11:26:53.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.756+0000 7fc8f8c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8f40b6dc0 0x7fc8f41f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.756+0000 7fc8f8c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8f415fcf0 0x7fc8f409fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.756+0000 7fc8f8c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8f4160420 0x7fc8f40c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.756+0000 7fc8f8c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40a0420 con 0x7fc8f4160420 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.756+0000 7fc8f8c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4150a10 con 0x7fc8f40b6dc0 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.756+0000 7fc8f8c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4150cf0 con 0x7fc8f415fcf0 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.756+0000 7fc8f3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8f4160420 0x7fc8f40c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.756+0000 7fc8f3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8f4160420 0x7fc8f40c80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36874/0 (socket says 172.21.15.62:36874) 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.756+0000 7fc8f3e00640 1 -- 172.21.15.62:0/2752095876 learned_addr learned my addr 172.21.15.62:0/2752095876 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.757+0000 7fc8f3400640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8f40b6dc0 0x7fc8f41f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.757+0000 7fc8f2a00640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8f415fcf0 0x7fc8f409fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.757+0000 7fc8f3e00640 1 -- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8f415fcf0 msgr2=0x7fc8f409fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.757+0000 7fc8f3e00640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8f415fcf0 0x7fc8f409fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.757+0000 7fc8f3e00640 1 -- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8f40b6dc0 msgr2=0x7fc8f41f2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.757+0000 7fc8f3e00640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8f40b6dc0 0x7fc8f41f2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.757+0000 7fc8f3e00640 1 -- 172.21.15.62:0/2752095876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec009b30 con 0x7fc8f4160420 2024-03-23T11:26:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.757+0000 7fc8f3e00640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8f4160420 0x7fc8f40c80a0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec036040 tx=0x7fc8ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.757+0000 7fc8eb400640 1 -- 172.21.15.62:0/2752095876 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec03e450 con 0x7fc8f4160420 2024-03-23T11:26:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.758+0000 7fc8eb400640 1 -- 172.21.15.62:0/2752095876 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec03e5d0 con 0x7fc8f4160420 2024-03-23T11:26:53.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.758+0000 7fc8f8c00640 1 -- 172.21.15.62:0/2752095876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f4150fd0 con 0x7fc8f4160420 2024-03-23T11:26:53.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.758+0000 7fc8eb400640 1 -- 172.21.15.62:0/2752095876 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec03e980 con 0x7fc8f4160420 2024-03-23T11:26:53.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.758+0000 7fc8f8c00640 1 -- 172.21.15.62:0/2752095876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f40c85e0 con 0x7fc8f4160420 2024-03-23T11:26:53.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.759+0000 7fc8f8c00640 1 -- 172.21.15.62:0/2752095876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f41514f0 con 0x7fc8f4160420 2024-03-23T11:26:53.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.760+0000 7fc8eb400640 1 -- 172.21.15.62:0/2752095876 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc8ec03eae0 con 0x7fc8f4160420 2024-03-23T11:26:53.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.761+0000 7fc8eb400640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc8d006a7a0 0x7fc8d006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:53.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.761+0000 7fc8eb400640 1 -- 172.21.15.62:0/2752095876 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fc8ec0abcf0 con 0x7fc8f4160420 2024-03-23T11:26:53.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.761+0000 7fc8f3400640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc8d006a7a0 0x7fc8d006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:53.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.762+0000 7fc8f3400640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc8d006a7a0 0x7fc8d006cc60 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400a990 tx=0x7fc8e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:53.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:53.765+0000 7fc8eb400640 1 -- 172.21.15.62:0/2752095876 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc8f41514f0 con 0x7fc8f4160420 2024-03-23T11:26:53.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.003+0000 7fc8f8c00640 1 -- 172.21.15.62:0/2752095876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc8f41514f0 con 0x7fc8f4160420 2024-03-23T11:26:53.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.004+0000 7fc8eb400640 1 -- 172.21.15.62:0/2752095876 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v63) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc8f41514f0 con 0x7fc8f4160420 2024-03-23T11:26:53.997 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:26:54.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 -- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc8d006a7a0 msgr2=0x7fc8d006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:54.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc8d006a7a0 0x7fc8d006cc60 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400a990 tx=0x7fc8e4008040 comp rx=0 tx=0).stop 2024-03-23T11:26:54.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 -- 172.21.15.62:0/2752095876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8f4160420 msgr2=0x7fc8f40c80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:54.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8f4160420 0x7fc8f40c80a0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec036040 tx=0x7fc8ec004a40 comp rx=0 tx=0).stop 2024-03-23T11:26:54.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 -- 172.21.15.62:0/2752095876 shutdown_connections 2024-03-23T11:26:54.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc8d006a7a0 0x7fc8d006cc60 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:54.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8f4160420 0x7fc8f40c80a0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:54.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8f415fcf0 0x7fc8f409fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:54.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 --2- 172.21.15.62:0/2752095876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8f40b6dc0 0x7fc8f41f2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:54.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 -- 172.21.15.62:0/2752095876 >> 172.21.15.62:0/2752095876 conn(0x7fc8f4089f60 msgr2=0x7fc8f40cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:54.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 -- 172.21.15.62:0/2752095876 shutdown_connections 2024-03-23T11:26:54.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:54.007+0000 7fc8f8c00640 1 -- 172.21.15.62:0/2752095876 wait complete. 2024-03-23T11:26:54.020 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:26:54.020 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:26:54.020 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:26:54.020 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:26:54.020 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:54.020 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:26:54.020 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:26:54.020 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:26:54.020 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:26:54.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:54.021 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:26:54.076 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:26:54.076 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:26:54.076 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:26:54.076 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:54.076 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42143sec preferred_lft 42143sec 2024-03-23T11:26:54.077 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:26:54.078 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:26:54.079 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:26:54.079 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:26:54.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:54.080 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:26:54.175 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:26:54.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:54.179 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:26:54.236 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:26:54.236 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:26:54.236 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:26:54.236 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:26:54.236 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:26:54.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:54.237 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:26:54.237 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:26:54.237 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:26:54.238 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:26:54.238 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:26:54.238 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:26:54.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:54.431 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:26:54.483 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:26:54.483 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:26:54.484 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:26:54.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:54.484 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:26:54.580 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:26:54.581 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:26:54.581 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:26:54.637 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:26:54.637 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:54.689 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:26:54.784 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:26:54.784 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:26:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:26:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:26:54.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:54.889 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:26:54.945 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:26:54.945 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:26:54.950 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:26:55.028 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:26:55.029 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:26:55.099 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:26:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:26:55.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:55.103 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:26:55.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:26:55.147+0000 7fa70f20b480 -1 init, newargv = 0x56334849ae40 newargc=15 2024-03-23T11:26:55.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[67465]: starting ceph client 2024-03-23T11:26:55.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[67465]: starting fuse 2024-03-23T11:26:56.247 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:26:56.364 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:26:56.365 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:26:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:26:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:26:56.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:56.473 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:26:56.535 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:26:56.535 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:56.536 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:26:56.537 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:26:56.715 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.67465.asok 2024-03-23T11:26:56.895 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.67465.asok 2024-03-23T11:26:56.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:56.895 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.67465.asok status 2024-03-23T11:26:57.148 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:26:57.148 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:26:57.148 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:26:57.148 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:26:57.148 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "67465", 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 7659, 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:26:57.149 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 7659 2024-03-23T11:26:57.150 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:57.150 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:26:57.150 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:26:57.150 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:26:57.150 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2295126839 2024-03-23T11:26:57.150 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:57.150 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:57.150 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:26:57.150 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:26:57.150 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:26:57.150 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2295126839 2024-03-23T11:26:57.151 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:26:57.151 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.7659 192.168.0.1:0/2295126839", 2024-03-23T11:26:57.151 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/2295126839", 2024-03-23T11:26:57.151 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:26:57.151 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 63, 2024-03-23T11:26:57.151 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 92, 2024-03-23T11:26:57.151 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:26:57.151 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:26:57.151 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:26:57.151 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:26:57.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:57.163 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:26:57.163 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:57.219 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:26:57.219 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:57.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:57.219 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:26:57.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:26:57.309 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat testdir/hardlink) 2024-03-23T11:26:57.369 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:26:57.376+0000 7fd996000640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-03-23T11:26:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:stat: cannot statx 'testdir/hardlink': Input/output error 2024-03-23T11:26:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:26:57.373 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive 2024-03-23T11:26:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.606+0000 7f7552c00640 1 Processor -- start 2024-03-23T11:26:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.606+0000 7f7552c00640 1 -- start start 2024-03-23T11:26:57.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.608+0000 7f7552c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c15c820 0x7f754c0b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.608+0000 7f7552c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c0d0200 con 0x7f754c0b28e0 2024-03-23T11:26:57.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.608+0000 7f7552c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c0d07c0 con 0x7f754c0cf710 2024-03-23T11:26:57.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.608+0000 7f7552c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754c0ce070 con 0x7f754c15c820 2024-03-23T11:26:57.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.608+0000 7f7550e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f754c0b28e0 0x7f754c0cf000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43150/0 (socket says 172.21.15.62:43150) 2024-03-23T11:26:57.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.608+0000 7f7550e00640 1 -- 172.21.15.62:0/2800481515 learned_addr learned my addr 172.21.15.62:0/2800481515 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:26:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.609+0000 7f7551800640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c15c820 0x7f754c0b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.609+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912652227 0 0) 0x7f754c0d0200 con 0x7f754c0b28e0 2024-03-23T11:26:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.609+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7534003520 con 0x7f754c0b28e0 2024-03-23T11:26:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.609+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1386196152 0 0) 0x7f7534003520 con 0x7f754c0b28e0 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.609+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c15c820 msgr2=0x7f754c0b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.609+0000 7f754be00640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c15c820 0x7f754c0b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.609+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 >> v1:172.21.15.120:6789/0 conn(0x7f754c0cf710 legacy=0x7f754c0d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.609+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c0ce350 con 0x7f754c0b28e0 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.609+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f753c0017f0 con 0x7f754c0b28e0 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.609+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 >> v1:172.21.15.62:6789/0 conn(0x7f754c0b28e0 legacy=0x7f754c0cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.609+0000 7f754be00640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c15c820 0x7f75340039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f754be00640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7534003f20 0x7f75340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f754be00640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7534004d00 0x7f7534005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c0ce070 con 0x7f7534004d00 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753400bb10 con 0x7f7534003f20 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f753400bcd0 con 0x7f754c15c820 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f7552200640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7534004d00 0x7f7534005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f7550e00640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7534003f20 0x7f75340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f7551800640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c15c820 0x7f75340039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f7552200640 1 -- 172.21.15.62:0/2800481515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c15c820 msgr2=0x7f75340039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f7552200640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c15c820 0x7f75340039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f7552200640 1 -- 172.21.15.62:0/2800481515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7534003f20 msgr2=0x7f75340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f7552200640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7534003f20 0x7f75340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.610+0000 7f7552200640 1 -- 172.21.15.62:0/2800481515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7534003520 con 0x7f7534004d00 2024-03-23T11:26:57.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.612+0000 7f7552200640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7534004d00 0x7f7534005180 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f7544001e50 tx=0x7f75340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91575482b37f95d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.612+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544003b70 con 0x7f7534004d00 2024-03-23T11:26:57.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.612+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7544003cd0 con 0x7f7534004d00 2024-03-23T11:26:57.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.612+0000 7f754be00640 1 -- 172.21.15.62:0/2800481515 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544005310 con 0x7f7534004d00 2024-03-23T11:26:57.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.613+0000 7f7552c00640 1 -- 172.21.15.62:0/2800481515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7534004d00 msgr2=0x7f7534005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.613+0000 7f7552c00640 1 --2- 172.21.15.62:0/2800481515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7534004d00 0x7f7534005180 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f7544001e50 tx=0x7f75340057f0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.613+0000 7f7552200640 1 -- 172.21.15.62:0/2800481515 reap_dead start 2024-03-23T11:26:57.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.613+0000 7f7552c00640 1 -- 172.21.15.62:0/2800481515 shutdown_connections 2024-03-23T11:26:57.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.613+0000 7f7552c00640 1 -- 172.21.15.62:0/2800481515 >> 172.21.15.62:0/2800481515 conn(0x7f754c09b9d0 msgr2=0x7f754c089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.613+0000 7f7552c00640 1 -- 172.21.15.62:0/2800481515 shutdown_connections 2024-03-23T11:26:57.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.613+0000 7f7552c00640 1 -- 172.21.15.62:0/2800481515 wait complete. 2024-03-23T11:26:57.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.615+0000 7f7552c00640 1 Processor -- start 2024-03-23T11:26:57.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.615+0000 7f7552c00640 1 -- start start 2024-03-23T11:26:57.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.616+0000 7f7552c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7534004d00 0x7f754c1f1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.616+0000 7f7552c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0c8aa0 0x7f754c15f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.616+0000 7f7552c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c0b5fc0 0x7f754c0b6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.616+0000 7f7552c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c0c6270 con 0x7f754c0c8aa0 2024-03-23T11:26:57.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.616+0000 7f7552c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c0c6550 con 0x7f7534004d00 2024-03-23T11:26:57.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.616+0000 7f7552c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754c0c6830 con 0x7f754c0b5fc0 2024-03-23T11:26:57.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.617+0000 7f7550e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0c8aa0 0x7f754c15f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.617+0000 7f7550e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0c8aa0 0x7f754c15f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37658/0 (socket says 172.21.15.62:37658) 2024-03-23T11:26:57.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.617+0000 7f7550e00640 1 -- 172.21.15.62:0/377943841 learned_addr learned my addr 172.21.15.62:0/377943841 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.617+0000 7f7551800640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7534004d00 0x7f754c1f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.617+0000 7f7552200640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c0b5fc0 0x7f754c0b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.617+0000 7f7550e00640 1 -- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c0b5fc0 msgr2=0x7f754c0b6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.617+0000 7f7550e00640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c0b5fc0 0x7f754c0b6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.617+0000 7f7550e00640 1 -- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7534004d00 msgr2=0x7f754c1f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.617+0000 7f7550e00640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7534004d00 0x7f754c1f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.617+0000 7f7550e00640 1 -- 172.21.15.62:0/377943841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c0c6b10 con 0x7f754c0c8aa0 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.617+0000 7f7550e00640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0c8aa0 0x7f754c15f210 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f753c00e760 tx=0x7f753c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.618+0000 7f7549600640 1 -- 172.21.15.62:0/377943841 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c00ee70 con 0x7f754c0c8aa0 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.618+0000 7f7549600640 1 -- 172.21.15.62:0/377943841 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c0072e0 con 0x7f754c0c8aa0 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.618+0000 7f7552c00640 1 -- 172.21.15.62:0/377943841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754c0c6df0 con 0x7f754c0c8aa0 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.618+0000 7f7549600640 1 -- 172.21.15.62:0/377943841 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c0057a0 con 0x7f754c0c8aa0 2024-03-23T11:26:57.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.618+0000 7f7552c00640 1 -- 172.21.15.62:0/377943841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754c15f750 con 0x7f754c0c8aa0 2024-03-23T11:26:57.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.620+0000 7f7549600640 1 -- 172.21.15.62:0/377943841 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f753c005900 con 0x7f754c0c8aa0 2024-03-23T11:26:57.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.621+0000 7f7549600640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f753006a7a0 0x7f753006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.621+0000 7f7549600640 1 -- 172.21.15.62:0/377943841 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f753c088f20 con 0x7f754c0c8aa0 2024-03-23T11:26:57.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.622+0000 7f7551800640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f753006a7a0 0x7f753006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.622+0000 7f7556ff4740 1 Processor -- start 2024-03-23T11:26:57.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.622+0000 7f7556ff4740 1 -- start start 2024-03-23T11:26:57.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.623+0000 7f7551800640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f753006a7a0 0x7f753006cc60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7540009a00 tx=0x7f7540008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.623+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0fd420 0x56411a0fd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.623+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0cd440 0x56411a0fa690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.623+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0fabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.623+0000 7f7556ff4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a03d280 con 0x56411a0d0fa0 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.623+0000 7f7556ff4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a03aa10 con 0x56411a0cd440 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.623+0000 7f7556ff4740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56411a036300 con 0x56411a0fd420 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.623+0000 7f7552200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0fabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.623+0000 7f7552200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0fabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37670/0 (socket says 172.21.15.62:37670) 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.623+0000 7f7552200640 1 -- 172.21.15.62:0/2119180686 learned_addr learned my addr 172.21.15.62:0/2119180686 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.624+0000 7f7551800640 1 --2- 172.21.15.62:0/2119180686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0fd420 0x56411a0fd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.624+0000 7f7550e00640 1 --2- 172.21.15.62:0/2119180686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0cd440 0x56411a0fa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.624+0000 7f7552200640 1 -- 172.21.15.62:0/2119180686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0fd420 msgr2=0x56411a0fd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.624+0000 7f7552200640 1 --2- 172.21.15.62:0/2119180686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0fd420 0x56411a0fd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.624+0000 7f7552200640 1 -- 172.21.15.62:0/2119180686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0cd440 msgr2=0x56411a0fa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.624+0000 7f7552200640 1 --2- 172.21.15.62:0/2119180686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0cd440 0x56411a0fa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.624+0000 7f7552200640 1 -- 172.21.15.62:0/2119180686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564119fd7f60 con 0x56411a0d0fa0 2024-03-23T11:26:57.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.624+0000 7f7552200640 1 --2- 172.21.15.62:0/2119180686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0fabd0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f7544001f70 tx=0x7f7544030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8927515043fbc95a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.624+0000 7f753a000640 1 -- 172.21.15.62:0/2119180686 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544003b70 con 0x56411a0d0fa0 2024-03-23T11:26:57.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.625+0000 7f753a000640 1 -- 172.21.15.62:0/2119180686 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7544003e40 con 0x56411a0d0fa0 2024-03-23T11:26:57.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.625+0000 7f753a000640 1 -- 172.21.15.62:0/2119180686 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544006d90 con 0x56411a0d0fa0 2024-03-23T11:26:57.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.625+0000 7f7556ff4740 1 -- 172.21.15.62:0/2119180686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 msgr2=0x56411a0fabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.625+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2119180686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0fabd0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f7544001f70 tx=0x7f7544030000 comp rx=0 tx=0).stop 2024-03-23T11:26:57.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.625+0000 7f7556ff4740 1 -- 172.21.15.62:0/2119180686 shutdown_connections 2024-03-23T11:26:57.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.625+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2119180686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0fabd0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.625+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2119180686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0cd440 0x56411a0fa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.625+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2119180686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0fd420 0x56411a0fd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.625+0000 7f7556ff4740 1 -- 172.21.15.62:0/2119180686 >> 172.21.15.62:0/2119180686 conn(0x56411a0cf6b0 msgr2=0x56411a118230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.625+0000 7f7556ff4740 1 -- 172.21.15.62:0/2119180686 shutdown_connections 2024-03-23T11:26:57.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.625+0000 7f7556ff4740 1 -- 172.21.15.62:0/2119180686 wait complete. 2024-03-23T11:26:57.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.626+0000 7f7556ff4740 1 -- start start 2024-03-23T11:26:57.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.626+0000 7f7556ff4740 1 Processor -- start 2024-03-23T11:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.627+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0ea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.627+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0fd420 0x56411a0eb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.627+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0f9310 0x56411a101b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.627+0000 7f7556ff4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119ff4730 con 0x56411a0d0fa0 2024-03-23T11:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.627+0000 7f7556ff4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119ff43d0 con 0x56411a0fd420 2024-03-23T11:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.627+0000 7f7556ff4740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x564119d9dff0 con 0x56411a0f9310 2024-03-23T11:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.627+0000 7f7556ff4740 10 client.0 _ll_register_callbacks cb 0x56411a02f970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f7551800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0ea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f7551800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0ea7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37672/0 (socket says 172.21.15.62:37672) 2024-03-23T11:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f7551800640 1 -- 172.21.15.62:0/1600485816 learned_addr learned my addr 172.21.15.62:0/1600485816 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f7550e00640 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0fd420 0x56411a0eb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f7552200640 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0f9310 0x56411a101b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f7551800640 1 -- 172.21.15.62:0/1600485816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0f9310 msgr2=0x56411a101b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f7551800640 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0f9310 0x56411a101b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f7551800640 1 -- 172.21.15.62:0/1600485816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0fd420 msgr2=0x56411a0eb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f7551800640 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0fd420 0x56411a0eb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f7551800640 1 -- 172.21.15.62:0/1600485816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7534003520 con 0x56411a0d0fa0 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f7551800640 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0ea7c0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f7540012f70 tx=0x7f7540004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591db48b75faf4d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.628+0000 7f752ea00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.629+0000 7f752ea00640 1 -- 172.21.15.62:0/1600485816 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540023830 con 0x56411a0d0fa0 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.629+0000 7f752ea00640 1 -- 172.21.15.62:0/1600485816 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f7540023990 con 0x56411a0d0fa0 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.629+0000 7f7556ff4740 1 -- 172.21.15.62:0/1600485816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56411a0604a0 con 0x56411a0d0fa0 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.629+0000 7f752ea00640 1 -- 172.21.15.62:0/1600485816 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7540023b90 con 0x56411a0d0fa0 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.629+0000 7f7556ff4740 10 client.7671 fetch_fsmap learned FSMap version 63 2024-03-23T11:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.629+0000 7f7556ff4740 1 -- 172.21.15.62:0/1600485816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x56411a034980 con 0x56411a0d0fa0 2024-03-23T11:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.629+0000 7f752ea00640 1 -- 172.21.15.62:0/1600485816 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56411a0604a0 con 0x56411a0d0fa0 2024-03-23T11:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.629+0000 7f752ea00640 1 -- 172.21.15.62:0/1600485816 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7540023cf0 con 0x56411a0d0fa0 2024-03-23T11:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.629+0000 7f7556ff4740 10 client.7671 fetch_fsmap finished waiting for FSMap version 63 2024-03-23T11:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.629+0000 7f7556ff4740 10 client.7671 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.630+0000 7f7556ff4740 20 client.7671 populate_metadata read hostname 'smithi062' 2024-03-23T11:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.630+0000 7f7556ff4740 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x56411a0fc940 0x56411a0fedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.630+0000 7f7556ff4740 4 client.7671 mds_command: new command op to 7578 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.630+0000 7f7556ff4740 1 -- 172.21.15.62:0/1600485816 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56411a028e30 con 0x56411a0fc940 2024-03-23T11:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.630+0000 7f7550e00640 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x56411a0fc940 0x56411a0fedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.631+0000 7f7550e00640 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x56411a0fc940 0x56411a0fedb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d9f451f706b766b server_cookie=21aaa1638be93718 in_seq=0 out_seq=0 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.631+0000 7f752ea00640 10 client.7671 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.631+0000 7f752ea00640 1 -- 172.21.15.62:0/1600485816 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56411a028e30 con 0x56411a0fc940 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.631+0000 7f752ea00640 10 client.7671 handle_command_reply: tid=0 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.631+0000 7f7556ff4740 1 client.7671 shutdown 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f7556ff4740 1 -- 172.21.15.62:0/1600485816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 msgr2=0x56411a0ea7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f7556ff4740 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0ea7c0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f7540012f70 tx=0x7f7540004730 comp rx=0 tx=0).stop 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f7556ff4740 1 -- 172.21.15.62:0/1600485816 shutdown_connections 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f7556ff4740 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x56411a0fc940 0x56411a0fedb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f7556ff4740 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0f9310 0x56411a101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f752ea00640 0 client.7671 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f7556ff4740 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0fd420 0x56411a0eb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f7556ff4740 1 --2- 172.21.15.62:0/1600485816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0ea7c0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f7556ff4740 1 -- 172.21.15.62:0/1600485816 >> 172.21.15.62:0/1600485816 conn(0x56411a0cd440 msgr2=0x56411a11bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f7556ff4740 1 -- 172.21.15.62:0/1600485816 shutdown_connections 2024-03-23T11:26:57.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f7556ff4740 1 -- 172.21.15.62:0/1600485816 wait complete. 2024-03-23T11:26:57.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.632+0000 7f7556ff4740 20 client.7671 trim_cache size 0 max 16384 2024-03-23T11:26:57.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.648+0000 7f7556ff4740 1 Processor -- start 2024-03-23T11:26:57.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.648+0000 7f7556ff4740 1 -- start start 2024-03-23T11:26:57.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.649+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.649+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0f9310 0x56411a100bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.649+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0fd420 0x56411a105320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.649+0000 7f7556ff4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a069810 con 0x56411a0d0fa0 2024-03-23T11:26:57.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.649+0000 7f7556ff4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a100000 con 0x56411a0f9310 2024-03-23T11:26:57.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.649+0000 7f7556ff4740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56411a0f8f60 con 0x56411a0fd420 2024-03-23T11:26:57.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.649+0000 7f7551800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.649+0000 7f7551800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37676/0 (socket says 172.21.15.62:37676) 2024-03-23T11:26:57.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.649+0000 7f7551800640 1 -- 172.21.15.62:0/2482598528 learned_addr learned my addr 172.21.15.62:0/2482598528 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:57.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.650+0000 7f7551800640 1 -- 172.21.15.62:0/2482598528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0fd420 msgr2=0x56411a105320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.650+0000 7f7551800640 1 --2- 172.21.15.62:0/2482598528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0fd420 0x56411a105320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.650+0000 7f7551800640 1 -- 172.21.15.62:0/2482598528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0f9310 msgr2=0x56411a100bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.650+0000 7f7551800640 1 --2- 172.21.15.62:0/2482598528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0f9310 0x56411a100bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.650+0000 7f7551800640 1 -- 172.21.15.62:0/2482598528 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564119fd67a0 con 0x56411a0d0fa0 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.650+0000 7f7551800640 1 --2- 172.21.15.62:0/2482598528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0cda00 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f7540009830 tx=0x7f7540023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c9d2bdbf2fe3f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.650+0000 7f753aa00640 1 -- 172.21.15.62:0/2482598528 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540015c50 con 0x56411a0d0fa0 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.650+0000 7f753aa00640 1 -- 172.21.15.62:0/2482598528 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754002c850 con 0x56411a0d0fa0 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.650+0000 7f753aa00640 1 -- 172.21.15.62:0/2482598528 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754002cbc0 con 0x56411a0d0fa0 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.651+0000 7f7556ff4740 1 -- 172.21.15.62:0/2482598528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 msgr2=0x56411a0cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.651+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2482598528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0cda00 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f7540009830 tx=0x7f7540023000 comp rx=0 tx=0).stop 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.651+0000 7f7556ff4740 1 -- 172.21.15.62:0/2482598528 shutdown_connections 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.651+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2482598528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0fd420 0x56411a105320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.651+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2482598528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0f9310 0x56411a100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.651+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2482598528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a0cda00 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.651+0000 7f7556ff4740 1 -- 172.21.15.62:0/2482598528 >> 172.21.15.62:0/2482598528 conn(0x56411a0cd440 msgr2=0x56411a0cc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.651+0000 7f7556ff4740 1 -- 172.21.15.62:0/2482598528 shutdown_connections 2024-03-23T11:26:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.651+0000 7f7556ff4740 1 -- 172.21.15.62:0/2482598528 wait complete. 2024-03-23T11:26:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.652+0000 7f7556ff4740 1 -- start start 2024-03-23T11:26:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.652+0000 7f7556ff4740 1 Processor -- start 2024-03-23T11:26:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.653+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a17ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.653+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0f9310 0x56411a1046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.653+0000 7f7556ff4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0fd420 0x56411a1738d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.653+0000 7f7556ff4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a1756e0 con 0x56411a0d0fa0 2024-03-23T11:26:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.653+0000 7f7556ff4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a173510 con 0x56411a0fd420 2024-03-23T11:26:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.653+0000 7f7556ff4740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56411a176500 con 0x56411a0f9310 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.653+0000 7f7556ff4740 10 client.0 _ll_register_callbacks cb 0x56411a0fdc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.653+0000 7f7551800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a17ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.653+0000 7f7551800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a17ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37682/0 (socket says 172.21.15.62:37682) 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.653+0000 7f7551800640 1 -- 172.21.15.62:0/2089224550 learned_addr learned my addr 172.21.15.62:0/2089224550 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f7550e00640 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0f9310 0x56411a1046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f7551800640 1 -- 172.21.15.62:0/2089224550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0f9310 msgr2=0x56411a1046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f7551800640 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0f9310 0x56411a1046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f7551800640 1 -- 172.21.15.62:0/2089224550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0fd420 msgr2=0x56411a1738d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f7552200640 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0fd420 0x56411a1738d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f7551800640 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0fd420 0x56411a1738d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f7551800640 1 -- 172.21.15.62:0/2089224550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56411a034980 con 0x56411a0d0fa0 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f7550e00640 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0f9310 0x56411a1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f7551800640 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a17ef20 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f754002c460 tx=0x7f7540017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a75293459f485f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f752ea00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f752ea00640 1 -- 172.21.15.62:0/2089224550 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754002c6f0 con 0x56411a0d0fa0 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.654+0000 7f752ea00640 1 -- 172.21.15.62:0/2089224550 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f754002cbc0 con 0x56411a0d0fa0 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f7556ff4740 1 -- 172.21.15.62:0/2089224550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56411a0f9a60 con 0x56411a0d0fa0 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f752ea00640 1 -- 172.21.15.62:0/2089224550 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7540015c50 con 0x56411a0d0fa0 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f7556ff4740 10 client.7677 fetch_fsmap learned FSMap version 63 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f7556ff4740 1 -- 172.21.15.62:0/2089224550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x564119ffbd70 con 0x56411a0d0fa0 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f752ea00640 1 -- 172.21.15.62:0/2089224550 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56411a0f9a60 con 0x56411a0d0fa0 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f752ea00640 1 -- 172.21.15.62:0/2089224550 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7540047030 con 0x56411a0d0fa0 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f7556ff4740 10 client.7677 fetch_fsmap finished waiting for FSMap version 63 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f7556ff4740 10 client.7677 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f7556ff4740 20 client.7677 populate_metadata read hostname 'smithi062' 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x56411a173e10 0x56411a181a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f7556ff4740 4 client.7677 mds_command: new command op to 7578 tid=0[{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}] 2024-03-23T11:26:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f7556ff4740 1 -- 172.21.15.62:0/2089224550 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}) v1 -- 0x7f7544003e40 con 0x56411a173e10 2024-03-23T11:26:57.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.655+0000 7f7550e00640 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x56411a173e10 0x56411a181a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.656+0000 7f7550e00640 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x56411a173e10 0x56411a181a90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1873bf824c757df5 server_cookie=565a1ca292b4cda5 in_seq=0 out_seq=0 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.656+0000 7f752ea00640 10 client.7677 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.656+0000 7f752ea00640 1 -- 172.21.15.62:0/2089224550 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7f7544003e40 con 0x56411a173e10 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.656+0000 7f752ea00640 10 client.7677 handle_command_reply: tid=0 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.656+0000 7f7556ff4740 1 client.7677 shutdown 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f7556ff4740 1 -- 172.21.15.62:0/2089224550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 msgr2=0x56411a17ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a17ef20 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f754002c460 tx=0x7f7540017000 comp rx=0 tx=0).stop 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f7556ff4740 1 -- 172.21.15.62:0/2089224550 shutdown_connections 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x56411a173e10 0x56411a181a90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56411a0fd420 0x56411a1738d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56411a0f9310 0x56411a1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f7556ff4740 1 --2- 172.21.15.62:0/2089224550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56411a0d0fa0 0x56411a17ef20 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f752ea00640 0 client.7677 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f7556ff4740 1 -- 172.21.15.62:0/2089224550 >> 172.21.15.62:0/2089224550 conn(0x56411a0cd440 msgr2=0x56411a0cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.653 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:26:57.657+0000 7fd996000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 ap=1 f(v0 m2024-03-23T11:26:34.599368+0000 1=1+0) n(v0 rc2024-03-23T11:26:34.723466+0000 2=1+1) (iversion lock) | dirfrag=0 authpin=1 scrubqueue=0 0x55c63f9e2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-23T11:26:34.599368+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-23T11:26:34.723466+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-03-23T11:26:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f7556ff4740 1 -- 172.21.15.62:0/2089224550 shutdown_connections 2024-03-23T11:26:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f7556ff4740 1 -- 172.21.15.62:0/2089224550 wait complete. 2024-03-23T11:26:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.657+0000 7f7556ff4740 20 client.7677 trim_cache size 0 max 16384 2024-03-23T11:26:57.654 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:26:57.654 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-03-23T11:26:57.654 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "32b18fe6-0fad-41ae-931a-f4f42171aa2c", 2024-03-23T11:26:57.654 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-03-23T11:26:57.654 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:26:57.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 -- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f753006a7a0 msgr2=0x7f753006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f753006a7a0 0x7f753006cc60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7540009a00 tx=0x7f7540008040 comp rx=0 tx=0).stop 2024-03-23T11:26:57.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 -- 172.21.15.62:0/377943841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0c8aa0 msgr2=0x7f754c15f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0c8aa0 0x7f754c15f210 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f753c00e760 tx=0x7f753c00ec20 comp rx=0 tx=0).stop 2024-03-23T11:26:57.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 -- 172.21.15.62:0/377943841 shutdown_connections 2024-03-23T11:26:57.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f753006a7a0 0x7f753006cc60 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0c8aa0 0x7f754c15f210 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754c0b5fc0 0x7f754c0b6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 --2- 172.21.15.62:0/377943841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7534004d00 0x7f754c1f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 -- 172.21.15.62:0/377943841 >> 172.21.15.62:0/377943841 conn(0x7f754c09b9d0 msgr2=0x7f754c0b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 -- 172.21.15.62:0/377943841 shutdown_connections 2024-03-23T11:26:57.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.658+0000 7f7552c00640 1 -- 172.21.15.62:0/377943841 wait complete. 2024-03-23T11:26:57.670 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-23T11:26:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.883+0000 7f2a48a00640 1 Processor -- start 2024-03-23T11:26:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.883+0000 7f2a48a00640 1 -- start start 2024-03-23T11:26:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.885+0000 7f2a48a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2a44142f10 0x7f2a440caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.885+0000 7f2a48a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a440cb530 0x7f2a4406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.885+0000 7f2a48a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a44159d00 con 0x7f2a440b25a0 2024-03-23T11:26:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.885+0000 7f2a48a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a440cbd40 con 0x7f2a44142f10 2024-03-23T11:26:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.885+0000 7f2a48a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a44159fe0 con 0x7f2a440cb530 2024-03-23T11:26:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.885+0000 7f2a41e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2a44142f10 0x7f2a440caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.885+0000 7f2a41e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2a44142f10 0x7f2a440caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37698/0 (socket says 172.21.15.62:37698) 2024-03-23T11:26:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.885+0000 7f2a41e00640 1 -- 172.21.15.62:0/2949197287 learned_addr learned my addr 172.21.15.62:0/2949197287 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:57.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.885+0000 7f2a43200640 1 --2- 172.21.15.62:0/2949197287 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a440cb530 0x7f2a4406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.885+0000 7f2a41e00640 1 -- 172.21.15.62:0/2949197287 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a440cb530 msgr2=0x7f2a4406c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.886+0000 7f2a41e00640 1 --2- 172.21.15.62:0/2949197287 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a440cb530 0x7f2a4406c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.886+0000 7f2a41e00640 1 -- 172.21.15.62:0/2949197287 >> v1:172.21.15.120:6789/0 conn(0x7f2a440b25a0 legacy=0x7f2a440b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.886+0000 7f2a41e00640 1 -- 172.21.15.62:0/2949197287 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4415a2c0 con 0x7f2a44142f10 2024-03-23T11:26:57.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.887+0000 7f2a41e00640 1 --2- 172.21.15.62:0/2949197287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a44142f10 0x7f2a440caff0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f2a30009870 tx=0x7f2a3002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eac802b2375bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.887+0000 7f2a41400640 1 -- 172.21.15.62:0/2949197287 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30030840 con 0x7f2a44142f10 2024-03-23T11:26:57.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.887+0000 7f2a41400640 1 -- 172.21.15.62:0/2949197287 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a300309a0 con 0x7f2a44142f10 2024-03-23T11:26:57.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.887+0000 7f2a41400640 1 -- 172.21.15.62:0/2949197287 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30030d30 con 0x7f2a44142f10 2024-03-23T11:26:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.888+0000 7f2a48a00640 1 -- 172.21.15.62:0/2949197287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a44142f10 msgr2=0x7f2a440caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.888+0000 7f2a48a00640 1 --2- 172.21.15.62:0/2949197287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a44142f10 0x7f2a440caff0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f2a30009870 tx=0x7f2a3002b980 comp rx=0 tx=0).stop 2024-03-23T11:26:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.888+0000 7f2a48a00640 1 -- 172.21.15.62:0/2949197287 shutdown_connections 2024-03-23T11:26:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.888+0000 7f2a48a00640 1 --2- 172.21.15.62:0/2949197287 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a440cb530 0x7f2a4406c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.888+0000 7f2a48a00640 1 --2- 172.21.15.62:0/2949197287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a44142f10 0x7f2a440caff0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.888+0000 7f2a48a00640 1 -- 172.21.15.62:0/2949197287 >> 172.21.15.62:0/2949197287 conn(0x7f2a440b0090 msgr2=0x7f2a440b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.888+0000 7f2a48a00640 1 -- 172.21.15.62:0/2949197287 shutdown_connections 2024-03-23T11:26:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.888+0000 7f2a48a00640 1 -- 172.21.15.62:0/2949197287 wait complete. 2024-03-23T11:26:57.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.889+0000 7f2a48a00640 1 Processor -- start 2024-03-23T11:26:57.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.890+0000 7f2a48a00640 1 -- start start 2024-03-23T11:26:57.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.890+0000 7f2a48a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a440b25a0 0x7f2a4413a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a48a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a440cb530 0x7f2a441f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a48a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a4415b430 0x7f2a4415b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a48a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4415bdf0 con 0x7f2a4415b430 2024-03-23T11:26:57.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a48a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a441f2f90 con 0x7f2a440cb530 2024-03-23T11:26:57.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a48a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4413bc30 con 0x7f2a440b25a0 2024-03-23T11:26:57.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a43200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a4415b430 0x7f2a4415b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a43200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a4415b430 0x7f2a4415b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37706/0 (socket says 172.21.15.62:37706) 2024-03-23T11:26:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a43200640 1 -- 172.21.15.62:0/1967774638 learned_addr learned my addr 172.21.15.62:0/1967774638 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a41e00640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a440cb530 0x7f2a441f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a43200640 1 -- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a440b25a0 msgr2=0x7f2a4413a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a42800640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a440b25a0 0x7f2a4413a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a43200640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a440b25a0 0x7f2a4413a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a43200640 1 -- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a440cb530 msgr2=0x7f2a441f2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a43200640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a440cb530 0x7f2a441f2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a43200640 1 -- 172.21.15.62:0/1967774638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a300094d0 con 0x7f2a4415b430 2024-03-23T11:26:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.891+0000 7f2a41e00640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a440cb530 0x7f2a441f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:57.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.892+0000 7f2a43200640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a4415b430 0x7f2a4415b8b0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00cd70 tx=0x7f2a3c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.892+0000 7f2a3aa00640 1 -- 172.21.15.62:0/1967774638 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c003b90 con 0x7f2a4415b430 2024-03-23T11:26:57.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.892+0000 7f2a3aa00640 1 -- 172.21.15.62:0/1967774638 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c003cf0 con 0x7f2a4415b430 2024-03-23T11:26:57.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.892+0000 7f2a3aa00640 1 -- 172.21.15.62:0/1967774638 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c004240 con 0x7f2a4415b430 2024-03-23T11:26:57.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.892+0000 7f2a48a00640 1 -- 172.21.15.62:0/1967774638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4413bf10 con 0x7f2a4415b430 2024-03-23T11:26:57.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.892+0000 7f2a48a00640 1 -- 172.21.15.62:0/1967774638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4413f9d0 con 0x7f2a4415b430 2024-03-23T11:26:57.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.894+0000 7f2a3aa00640 1 -- 172.21.15.62:0/1967774638 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2a3c0043a0 con 0x7f2a4415b430 2024-03-23T11:26:57.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.895+0000 7f2a3aa00640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2a2406a7a0 0x7f2a2406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.895+0000 7f2a3aa00640 1 -- 172.21.15.62:0/1967774638 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f2a3c012070 con 0x7f2a4415b430 2024-03-23T11:26:57.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.896+0000 7f2a42800640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2a2406a7a0 0x7f2a2406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.896+0000 7f2a4c1cd740 1 Processor -- start 2024-03-23T11:26:57.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.896+0000 7f2a4c1cd740 1 -- start start 2024-03-23T11:26:57.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.897+0000 7f2a42800640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2a2406a7a0 0x7f2a2406cc60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2a4413b790 tx=0x7f2a34009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.897+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b457d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.897+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b42fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.897+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44fed0 0x55b85b430160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.897+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b85b3ebcb0 con 0x55b85b44faf0 2024-03-23T11:26:57.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.897+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b85b39a500 con 0x55b85b458460 2024-03-23T11:26:57.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.897+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b85b397c90 con 0x55b85b44fed0 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a42800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b457d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a42800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b457d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37708/0 (socket says 172.21.15.62:37708) 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a42800640 1 -- 172.21.15.62:0/1829767875 learned_addr learned my addr 172.21.15.62:0/1829767875 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a41e00640 1 --2- 172.21.15.62:0/1829767875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b42fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a43200640 1 --2- 172.21.15.62:0/1829767875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44fed0 0x55b85b430160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a42800640 1 -- 172.21.15.62:0/1829767875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44fed0 msgr2=0x55b85b430160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a42800640 1 --2- 172.21.15.62:0/1829767875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44fed0 0x55b85b430160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a42800640 1 -- 172.21.15.62:0/1829767875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 msgr2=0x55b85b42fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a42800640 1 --2- 172.21.15.62:0/1829767875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b42fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a42800640 1 -- 172.21.15.62:0/1829767875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b85b345360 con 0x55b85b44faf0 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a41e00640 1 --2- 172.21.15.62:0/1829767875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b42fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.898+0000 7f2a42800640 1 --2- 172.21.15.62:0/1829767875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b457d60 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400d6e0 tx=0x7f2a34010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14910dbeabf6cb50 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.899+0000 7f2a2f400640 1 -- 172.21.15.62:0/1829767875 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a340035f0 con 0x55b85b44faf0 2024-03-23T11:26:57.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.899+0000 7f2a2f400640 1 -- 172.21.15.62:0/1829767875 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a34003750 con 0x55b85b44faf0 2024-03-23T11:26:57.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.899+0000 7f2a2f400640 1 -- 172.21.15.62:0/1829767875 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a34003ac0 con 0x55b85b44faf0 2024-03-23T11:26:57.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.899+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/1829767875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 msgr2=0x55b85b457d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.899+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/1829767875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b457d60 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400d6e0 tx=0x7f2a34010000 comp rx=0 tx=0).stop 2024-03-23T11:26:57.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.899+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/1829767875 shutdown_connections 2024-03-23T11:26:57.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.899+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/1829767875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44fed0 0x55b85b430160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.899+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/1829767875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b42fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.899+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/1829767875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b457d60 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.899+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/1829767875 >> 172.21.15.62:0/1829767875 conn(0x55b85b3f7b40 msgr2=0x55b85b475900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.899+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/1829767875 shutdown_connections 2024-03-23T11:26:57.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.900+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/1829767875 wait complete. 2024-03-23T11:26:57.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.900+0000 7f2a4c1cd740 1 -- start start 2024-03-23T11:26:57.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.900+0000 7f2a4c1cd740 1 Processor -- start 2024-03-23T11:26:57.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.901+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b487ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.901+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44fed0 0x55b85b474cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.901+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b45d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.901+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b85b393580 con 0x55b85b44faf0 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.901+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b85b334a50 con 0x55b85b458460 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.901+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b85b3346f0 con 0x55b85b44fed0 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.902+0000 7f2a4c1cd740 10 client.0 _ll_register_callbacks cb 0x55b85b383ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.902+0000 7f2a42800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b487ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.902+0000 7f2a42800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b487ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37720/0 (socket says 172.21.15.62:37720) 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.902+0000 7f2a42800640 1 -- 172.21.15.62:0/3310588718 learned_addr learned my addr 172.21.15.62:0/3310588718 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.902+0000 7f2a41e00640 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44fed0 0x55b85b474cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.902+0000 7f2a43200640 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b45d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.902+0000 7f2a42800640 1 -- 172.21.15.62:0/3310588718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44fed0 msgr2=0x55b85b474cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.902+0000 7f2a42800640 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44fed0 0x55b85b474cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.902+0000 7f2a42800640 1 -- 172.21.15.62:0/3310588718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 msgr2=0x55b85b45d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.902+0000 7f2a42800640 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b45d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.902+0000 7f2a42800640 1 -- 172.21.15.62:0/3310588718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a44159d00 con 0x55b85b44faf0 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.903+0000 7f2a42800640 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b487ea0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x55b85b431e60 tx=0x7f2a34000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1202382cd7639822 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.903+0000 7f2a23e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.903+0000 7f2a23e00640 1 -- 172.21.15.62:0/3310588718 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a34003dd0 con 0x55b85b44faf0 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.903+0000 7f2a23e00640 1 -- 172.21.15.62:0/3310588718 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f2a34027070 con 0x55b85b44faf0 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.903+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/3310588718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b85b3bdfe0 con 0x55b85b44faf0 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.903+0000 7f2a23e00640 1 -- 172.21.15.62:0/3310588718 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a340145e0 con 0x55b85b44faf0 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.903+0000 7f2a4c1cd740 10 client.7689 fetch_fsmap learned FSMap version 63 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.903+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/3310588718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x55b85b391c00 con 0x55b85b44faf0 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.903+0000 7f2a23e00640 1 -- 172.21.15.62:0/3310588718 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b85b3bdfe0 con 0x55b85b44faf0 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.904+0000 7f2a23e00640 1 -- 172.21.15.62:0/3310588718 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2a34014740 con 0x55b85b44faf0 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.904+0000 7f2a4c1cd740 10 client.7689 fetch_fsmap finished waiting for FSMap version 63 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.904+0000 7f2a4c1cd740 10 client.7689 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:26:57.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.904+0000 7f2a4c1cd740 20 client.7689 populate_metadata read hostname 'smithi062' 2024-03-23T11:26:57.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.904+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b85b4597b0 0x55b85b459b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.904+0000 7f2a4c1cd740 4 client.7689 mds_command: new command op to 7578 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:26:57.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.904+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/3310588718 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b85b386240 con 0x55b85b4597b0 2024-03-23T11:26:57.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.904+0000 7f2a41e00640 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b85b4597b0 0x55b85b459b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.905+0000 7f2a23e00640 10 client.7689 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:57.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.905+0000 7f2a41e00640 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b85b4597b0 0x55b85b459b90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9fd64c86dd4876d server_cookie=5fba0c8960336509 in_seq=0 out_seq=0 2024-03-23T11:26:57.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.905+0000 7f2a23e00640 1 -- 172.21.15.62:0/3310588718 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55b85b386240 con 0x55b85b4597b0 2024-03-23T11:26:57.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.905+0000 7f2a23e00640 10 client.7689 handle_command_reply: tid=0 2024-03-23T11:26:57.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.905+0000 7f2a4c1cd740 1 client.7689 shutdown 2024-03-23T11:26:57.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.905+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/3310588718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 msgr2=0x55b85b487ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.905+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b487ea0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x55b85b431e60 tx=0x7f2a34000ed0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.906+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/3310588718 shutdown_connections 2024-03-23T11:26:57.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.906+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b85b4597b0 0x55b85b459b90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.906+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b45d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.906+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44fed0 0x55b85b474cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.906+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/3310588718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44faf0 0x55b85b487ea0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.906+0000 7f2a23e00640 0 client.7689 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:57.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.906+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/3310588718 >> 172.21.15.62:0/3310588718 conn(0x55b85b3f7b40 msgr2=0x55b85b42f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.906+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/3310588718 shutdown_connections 2024-03-23T11:26:57.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.906+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/3310588718 wait complete. 2024-03-23T11:26:57.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.906+0000 7f2a4c1cd740 20 client.7689 trim_cache size 0 max 16384 2024-03-23T11:26:57.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.922+0000 7f2a4c1cd740 1 Processor -- start 2024-03-23T11:26:57.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.922+0000 7f2a4c1cd740 1 -- start start 2024-03-23T11:26:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.923+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44faf0 0x55b85b42ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.923+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b42a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.923+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b4782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.923+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b85b090f90 con 0x55b85b44fed0 2024-03-23T11:26:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.923+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b85b42b4e0 con 0x55b85b458460 2024-03-23T11:26:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.923+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b85b474470 con 0x55b85b44faf0 2024-03-23T11:26:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.923+0000 7f2a41e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b42a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.923+0000 7f2a41e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b42a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37734/0 (socket says 172.21.15.62:37734) 2024-03-23T11:26:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.923+0000 7f2a41e00640 1 -- 172.21.15.62:0/1366476764 learned_addr learned my addr 172.21.15.62:0/1366476764 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:57.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.923+0000 7f2a43200640 1 --2- 172.21.15.62:0/1366476764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b4782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.924+0000 7f2a41e00640 1 -- 172.21.15.62:0/1366476764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44faf0 msgr2=0x55b85b42ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.924+0000 7f2a42800640 1 --2- 172.21.15.62:0/1366476764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44faf0 0x55b85b42ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.924+0000 7f2a41e00640 1 --2- 172.21.15.62:0/1366476764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44faf0 0x55b85b42ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.924+0000 7f2a41e00640 1 -- 172.21.15.62:0/1366476764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 msgr2=0x55b85b4782c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.924+0000 7f2a41e00640 1 --2- 172.21.15.62:0/1366476764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b4782c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.924+0000 7f2a41e00640 1 -- 172.21.15.62:0/1366476764 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b85b333c60 con 0x55b85b44fed0 2024-03-23T11:26:57.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.924+0000 7f2a43200640 1 --2- 172.21.15.62:0/1366476764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b4782c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.924+0000 7f2a41e00640 1 --2- 172.21.15.62:0/1366476764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b42a0c0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f2a300090b0 tx=0x7f2a30030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7951489429a4bd1d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.924+0000 7f2a2fe00640 1 -- 172.21.15.62:0/1366476764 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3003e040 con 0x55b85b44fed0 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.924+0000 7f2a2fe00640 1 -- 172.21.15.62:0/1366476764 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a30031ec0 con 0x55b85b44fed0 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.924+0000 7f2a2fe00640 1 -- 172.21.15.62:0/1366476764 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30004280 con 0x55b85b44fed0 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.925+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/1366476764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 msgr2=0x55b85b42a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.925+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/1366476764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b42a0c0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f2a300090b0 tx=0x7f2a30030000 comp rx=0 tx=0).stop 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.925+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/1366476764 shutdown_connections 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.925+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/1366476764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b42a0c0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.925+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/1366476764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b4782c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.925+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/1366476764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44faf0 0x55b85b42ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.925+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/1366476764 >> 172.21.15.62:0/1366476764 conn(0x55b85b3f7b40 msgr2=0x55b85b456c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.925+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/1366476764 shutdown_connections 2024-03-23T11:26:57.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.925+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/1366476764 wait complete. 2024-03-23T11:26:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.926+0000 7f2a4c1cd740 1 -- start start 2024-03-23T11:26:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.926+0000 7f2a4c1cd740 1 Processor -- start 2024-03-23T11:26:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.927+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44faf0 0x55b85b46bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.927+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b4308e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.927+0000 7f2a4c1cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b4de650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.927+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b85b4c4670 con 0x55b85b44fed0 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.927+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b85b4df220 con 0x55b85b458460 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.927+0000 7f2a4c1cd740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b85b45dec0 con 0x55b85b44faf0 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.927+0000 7f2a4c1cd740 10 client.0 _ll_register_callbacks cb 0x7f2a34001340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.927+0000 7f2a41e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b4308e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.927+0000 7f2a41e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b4308e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37742/0 (socket says 172.21.15.62:37742) 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.927+0000 7f2a41e00640 1 -- 172.21.15.62:0/526153098 learned_addr learned my addr 172.21.15.62:0/526153098 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a42800640 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44faf0 0x55b85b46bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a43200640 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b4de650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a41e00640 1 -- 172.21.15.62:0/526153098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44faf0 msgr2=0x55b85b46bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a41e00640 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44faf0 0x55b85b46bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a41e00640 1 -- 172.21.15.62:0/526153098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 msgr2=0x55b85b4de650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a41e00640 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b4de650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a41e00640 1 -- 172.21.15.62:0/526153098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4415a2c0 con 0x55b85b44fed0 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a43200640 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b4de650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a41e00640 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b4308e0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f2a300090b0 tx=0x7f2a30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a20674ea278aea server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a23e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a23e00640 1 -- 172.21.15.62:0/526153098 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3003e070 con 0x55b85b44fed0 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a23e00640 1 -- 172.21.15.62:0/526153098 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f2a30004240 con 0x55b85b44fed0 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.928+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/526153098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b85b0874e0 con 0x55b85b44fed0 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a23e00640 1 -- 172.21.15.62:0/526153098 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a30004400 con 0x55b85b44fed0 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a4c1cd740 10 client.7695 fetch_fsmap learned FSMap version 63 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/526153098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x55b85b359170 con 0x55b85b44fed0 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a23e00640 1 -- 172.21.15.62:0/526153098 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b85b0874e0 con 0x55b85b44fed0 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a23e00640 1 -- 172.21.15.62:0/526153098 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2a30043780 con 0x55b85b44fed0 2024-03-23T11:26:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a4c1cd740 10 client.7695 fetch_fsmap finished waiting for FSMap version 63 2024-03-23T11:26:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a4c1cd740 10 client.7695 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:26:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a4c1cd740 20 client.7695 populate_metadata read hostname 'smithi062' 2024-03-23T11:26:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b85b4c11c0 0x55b85b4782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:26:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a4c1cd740 4 client.7695 mds_command: new command op to 7578 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:26:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/526153098 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b85b0987e0 con 0x55b85b4c11c0 2024-03-23T11:26:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.929+0000 7f2a42800640 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b85b4c11c0 0x55b85b4782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:26:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a42800640 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b85b4c11c0 0x55b85b4782c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c232311dd1dec6 server_cookie=352fc2e9508c8dda in_seq=0 out_seq=0 2024-03-23T11:26:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a23e00640 10 client.7695 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a23e00640 1 -- 172.21.15.62:0/526153098 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+256 (crc 0 0 0) 0x55b85b0987e0 con 0x55b85b4c11c0 2024-03-23T11:26:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a23e00640 10 client.7695 handle_command_reply: tid=0 2024-03-23T11:26:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a4c1cd740 1 client.7695 shutdown 2024-03-23T11:26:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/526153098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 msgr2=0x55b85b4308e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b4308e0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f2a300090b0 tx=0x7f2a30004730 comp rx=0 tx=0).stop 2024-03-23T11:26:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/526153098 shutdown_connections 2024-03-23T11:26:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b85b4c11c0 0x55b85b4782c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b85b44fed0 0x55b85b4308e0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b85b458460 0x55b85b4de650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a4c1cd740 1 --2- 172.21.15.62:0/526153098 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b85b44faf0 0x55b85b46bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a23e00640 0 client.7695 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:26:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/526153098 >> 172.21.15.62:0/526153098 conn(0x55b85b3f7b40 msgr2=0x55b85b45bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.930+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/526153098 shutdown_connections 2024-03-23T11:26:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a4c1cd740 1 -- 172.21.15.62:0/526153098 wait complete. 2024-03-23T11:26:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a4c1cd740 20 client.7695 trim_cache size 0 max 16384 2024-03-23T11:26:57.928 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:26:57.928 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T11:26:57.928 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-03-23T11:26:57.928 INFO:teuthology.orchestra.run.smithi062.stdout: "32b18fe6-0fad-41ae-931a-f4f42171aa2c": { 2024-03-23T11:26:57.928 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/dir", 2024-03-23T11:26:57.928 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "32b18fe6-0fad-41ae-931a-f4f42171aa2c", 2024-03-23T11:26:57.928 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive" 2024-03-23T11:26:57.928 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:57.928 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:26:57.928 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:26:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a48a00640 1 -- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2a2406a7a0 msgr2=0x7f2a2406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a48a00640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2a2406a7a0 0x7f2a2406cc60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2a4413b790 tx=0x7f2a34009040 comp rx=0 tx=0).stop 2024-03-23T11:26:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a48a00640 1 -- 172.21.15.62:0/1967774638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a4415b430 msgr2=0x7f2a4415b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:26:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a48a00640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a4415b430 0x7f2a4415b8b0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00cd70 tx=0x7f2a3c00a660 comp rx=0 tx=0).stop 2024-03-23T11:26:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a48a00640 1 -- 172.21.15.62:0/1967774638 shutdown_connections 2024-03-23T11:26:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a48a00640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2a2406a7a0 0x7f2a2406cc60 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a48a00640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a4415b430 0x7f2a4415b8b0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a48a00640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a440cb530 0x7f2a441f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a48a00640 1 --2- 172.21.15.62:0/1967774638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a440b25a0 0x7f2a4413a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:26:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.931+0000 7f2a48a00640 1 -- 172.21.15.62:0/1967774638 >> 172.21.15.62:0/1967774638 conn(0x7f2a440b0090 msgr2=0x7f2a440b2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:26:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.932+0000 7f2a48a00640 1 -- 172.21.15.62:0/1967774638 shutdown_connections 2024-03-23T11:26:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:26:57.932+0000 7f2a48a00640 1 -- 172.21.15.62:0/1967774638 wait complete. 2024-03-23T11:26:57.941 INFO:tasks.cephfs.filesystem:scrub status for tag:32b18fe6-0fad-41ae-931a-f4f42171aa2c - {'path': '/dir', 'tag': '32b18fe6-0fad-41ae-931a-f4f42171aa2c', 'options': 'recursive'} 2024-03-23T11:26:59.458 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:26:59.465+0000 7f9201200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-23T11:27:16.750 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:27:16.759 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:27:27.942 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-23T11:27:28.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.181+0000 7fa5af200640 1 Processor -- start 2024-03-23T11:27:28.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.181+0000 7fa5af200640 1 -- start start 2024-03-23T11:27:28.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.183+0000 7fa5af200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b00707a0 0x7fa5b00b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.183+0000 7fa5af200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b00b20d0 con 0x7fa5b00cc2a0 2024-03-23T11:27:28.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.183+0000 7fa5af200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b01435d0 con 0x7fa5b01431f0 2024-03-23T11:27:28.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.183+0000 7fa5af200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b00b14c0 con 0x7fa5b00707a0 2024-03-23T11:27:28.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.183+0000 7fa5ae800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa5b00cc2a0 0x7fa5b00cc750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:46186/0 (socket says 172.21.15.62:46186) 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.183+0000 7fa5ae800640 1 -- 172.21.15.62:0/1406268576 learned_addr learned my addr 172.21.15.62:0/1406268576 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5ad400640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b00707a0 0x7fa5b00b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248021641 0 0) 0x7fa5b00b20d0 con 0x7fa5b00cc2a0 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa594003520 con 0x7fa5b00cc2a0 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 747977334 0 0) 0x7fa594003520 con 0x7fa5b00cc2a0 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b00707a0 msgr2=0x7fa5b00b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b00707a0 0x7fa5b00b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 >> v1:172.21.15.120:6789/0 conn(0x7fa5b01431f0 legacy=0x7fa5b0070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b00b17a0 con 0x7fa5b00cc2a0 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fa5a80019f0 con 0x7fa5b00cc2a0 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 >> v1:172.21.15.62:6789/0 conn(0x7fa5b00cc2a0 legacy=0x7fa5b00cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5940039a0 0x7fa594003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5940047c0 0x7fa594004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b00707a0 0x7fa594005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b00b14c0 con 0x7fa5940039a0 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59400bb10 con 0x7fa5940047c0 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.184+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59400bcd0 con 0x7fa5b00707a0 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.185+0000 7fa5ade00640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5940039a0 0x7fa594003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.185+0000 7fa5ae800640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b00707a0 0x7fa594005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.185+0000 7fa5ad400640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5940047c0 0x7fa594004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.185+0000 7fa5ade00640 1 -- 172.21.15.62:0/1406268576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b00707a0 msgr2=0x7fa594005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.185+0000 7fa5ade00640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b00707a0 0x7fa594005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.185+0000 7fa5ade00640 1 -- 172.21.15.62:0/1406268576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5940047c0 msgr2=0x7fa594004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.185+0000 7fa5ade00640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5940047c0 0x7fa594004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.185+0000 7fa5ade00640 1 -- 172.21.15.62:0/1406268576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa594003520 con 0x7fa5940039a0 2024-03-23T11:27:28.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.187+0000 7fa5ade00640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5940039a0 0x7fa594003dc0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001750 tx=0x7fa5940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5352afd166aab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.187+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a00052d0 con 0x7fa5940039a0 2024-03-23T11:27:28.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.187+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a002e050 con 0x7fa5940039a0 2024-03-23T11:27:28.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.187+0000 7fa5aca00640 1 -- 172.21.15.62:0/1406268576 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a00035f0 con 0x7fa5940039a0 2024-03-23T11:27:28.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.187+0000 7fa5af200640 1 -- 172.21.15.62:0/1406268576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5940039a0 msgr2=0x7fa594003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.187+0000 7fa5af200640 1 --2- 172.21.15.62:0/1406268576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5940039a0 0x7fa594003dc0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001750 tx=0x7fa5940057f0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.188+0000 7fa5ae800640 1 -- 172.21.15.62:0/1406268576 reap_dead start 2024-03-23T11:27:28.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.188+0000 7fa5af200640 1 -- 172.21.15.62:0/1406268576 shutdown_connections 2024-03-23T11:27:28.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.188+0000 7fa5af200640 1 -- 172.21.15.62:0/1406268576 >> 172.21.15.62:0/1406268576 conn(0x7fa5b00b0090 msgr2=0x7fa5b00b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.188+0000 7fa5af200640 1 -- 172.21.15.62:0/1406268576 shutdown_connections 2024-03-23T11:27:28.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.188+0000 7fa5af200640 1 -- 172.21.15.62:0/1406268576 wait complete. 2024-03-23T11:27:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.190+0000 7fa5af200640 1 Processor -- start 2024-03-23T11:27:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.190+0000 7fa5af200640 1 -- start start 2024-03-23T11:27:28.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5af200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5940039a0 0x7fa5b006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5af200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b013d9f0 0x7fa5b00af2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5af200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b006d850 0x7fa5b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5af200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b006d3c0 con 0x7fa5b006d850 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5af200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b013de70 con 0x7fa5940039a0 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5af200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b006c7b0 con 0x7fa5b013d9f0 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5ae800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b006d850 0x7fa5b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5ae800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b006d850 0x7fa5b006dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43032/0 (socket says 172.21.15.62:43032) 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5ae800640 1 -- 172.21.15.62:0/2358928567 learned_addr learned my addr 172.21.15.62:0/2358928567 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5ade00640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5940039a0 0x7fa5b006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5ae800640 1 -- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b013d9f0 msgr2=0x7fa5b00af2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5ae800640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b013d9f0 0x7fa5b00af2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5ae800640 1 -- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5940039a0 msgr2=0x7fa5b006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5ae800640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5940039a0 0x7fa5b006c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5ae800640 1 -- 172.21.15.62:0/2358928567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b006ca90 con 0x7fa5b006d850 2024-03-23T11:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.191+0000 7fa5ade00640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5940039a0 0x7fa5b006c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.192+0000 7fa5ae800640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b006d850 0x7fa5b006dcd0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00adfd0 tx=0x7fa5a800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.192+0000 7fa5a6000640 1 -- 172.21.15.62:0/2358928567 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a800ec10 con 0x7fa5b006d850 2024-03-23T11:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.192+0000 7fa5a6000640 1 -- 172.21.15.62:0/2358928567 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a800ed70 con 0x7fa5b006d850 2024-03-23T11:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.192+0000 7fa5a6000640 1 -- 172.21.15.62:0/2358928567 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a800eee0 con 0x7fa5b006d850 2024-03-23T11:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.192+0000 7fa5af200640 1 -- 172.21.15.62:0/2358928567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b006cd70 con 0x7fa5b006d850 2024-03-23T11:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.192+0000 7fa5af200640 1 -- 172.21.15.62:0/2358928567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b00af810 con 0x7fa5b006d850 2024-03-23T11:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.194+0000 7fa5a6000640 1 -- 172.21.15.62:0/2358928567 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa5a8013660 con 0x7fa5b006d850 2024-03-23T11:27:28.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.195+0000 7fa5a6000640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa59006a7a0 0x7fa59006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.195+0000 7fa5a6000640 1 -- 172.21.15.62:0/2358928567 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fa5a80892a0 con 0x7fa5b006d850 2024-03-23T11:27:28.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.196+0000 7fa5ade00640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa59006a7a0 0x7fa59006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.196+0000 7fa5b7bce740 1 Processor -- start 2024-03-23T11:27:28.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.196+0000 7fa5b7bce740 1 -- start start 2024-03-23T11:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.197+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067373d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.197+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x55906734bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.197+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 0x55906734c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.197+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559067307cb0 con 0x55906736baf0 2024-03-23T11:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.197+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590672b6500 con 0x559067374460 2024-03-23T11:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.197+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5590672b3c90 con 0x55906736bed0 2024-03-23T11:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.197+0000 7fa5ade00640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa59006a7a0 0x7fa59006cc60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001750 tx=0x7fa5a0009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.197+0000 7fa5ade00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067373d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.197+0000 7fa5ade00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067373d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43042/0 (socket says 172.21.15.62:43042) 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.197+0000 7fa5ade00640 1 -- 172.21.15.62:0/3323175701 learned_addr learned my addr 172.21.15.62:0/3323175701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.198+0000 7fa5ae800640 1 --2- 172.21.15.62:0/3323175701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 0x55906734c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.198+0000 7fa5ad400640 1 --2- 172.21.15.62:0/3323175701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x55906734bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.198+0000 7fa5ade00640 1 -- 172.21.15.62:0/3323175701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 msgr2=0x55906734c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.198+0000 7fa5ade00640 1 --2- 172.21.15.62:0/3323175701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 0x55906734c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.198+0000 7fa5ade00640 1 -- 172.21.15.62:0/3323175701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 msgr2=0x55906734bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.198+0000 7fa5ade00640 1 --2- 172.21.15.62:0/3323175701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x55906734bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.198+0000 7fa5ade00640 1 -- 172.21.15.62:0/3323175701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559067261360 con 0x55906736baf0 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.198+0000 7fa5ade00640 1 --2- 172.21.15.62:0/3323175701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067373d60 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001870 tx=0x7fa5a002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406172a479d75eb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.198+0000 7fa59aa00640 1 -- 172.21.15.62:0/3323175701 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a00074c0 con 0x55906736baf0 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.198+0000 7fa59aa00640 1 -- 172.21.15.62:0/3323175701 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a0007620 con 0x55906736baf0 2024-03-23T11:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.198+0000 7fa59aa00640 1 -- 172.21.15.62:0/3323175701 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a00069d0 con 0x55906736baf0 2024-03-23T11:27:28.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.199+0000 7fa5b7bce740 1 -- 172.21.15.62:0/3323175701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 msgr2=0x559067373d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.199+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/3323175701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067373d60 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001870 tx=0x7fa5a002f000 comp rx=0 tx=0).stop 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.199+0000 7fa5b7bce740 1 -- 172.21.15.62:0/3323175701 shutdown_connections 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.199+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/3323175701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 0x55906734c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.199+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/3323175701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x55906734bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.199+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/3323175701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067373d60 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.199+0000 7fa5b7bce740 1 -- 172.21.15.62:0/3323175701 >> 172.21.15.62:0/3323175701 conn(0x559067313b40 msgr2=0x559067391900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.199+0000 7fa5b7bce740 1 -- 172.21.15.62:0/3323175701 shutdown_connections 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.199+0000 7fa5b7bce740 1 -- 172.21.15.62:0/3323175701 wait complete. 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.200+0000 7fa5b7bce740 1 -- start start 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.200+0000 7fa5b7bce740 1 Processor -- start 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.201+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067390200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.201+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 0x559067390740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.201+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x5590673a1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.201+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590672af580 con 0x55906736baf0 2024-03-23T11:27:28.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.201+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559067250a50 con 0x559067374460 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.201+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5590672506f0 con 0x55906736bed0 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.201+0000 7fa5b7bce740 10 client.0 _ll_register_callbacks cb 0x55906729fed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.202+0000 7fa5ade00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067390200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.202+0000 7fa5ade00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067390200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43046/0 (socket says 172.21.15.62:43046) 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.202+0000 7fa5ade00640 1 -- 172.21.15.62:0/4146212343 learned_addr learned my addr 172.21.15.62:0/4146212343 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.202+0000 7fa5ae800640 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x5590673a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.202+0000 7fa5ad400640 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 0x559067390740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.202+0000 7fa5ade00640 1 -- 172.21.15.62:0/4146212343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 msgr2=0x559067390740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.202+0000 7fa5ade00640 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 0x559067390740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.202+0000 7fa5ade00640 1 -- 172.21.15.62:0/4146212343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 msgr2=0x5590673a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.202+0000 7fa5ade00640 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x5590673a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.202+0000 7fa5ade00640 1 -- 172.21.15.62:0/4146212343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa594003520 con 0x55906736baf0 2024-03-23T11:27:28.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.203+0000 7fa5ade00640 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067390200 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0000c00 tx=0x7fa5a002d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865cd66ad890cfa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.203+0000 7fa58f400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:27:28.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.203+0000 7fa58f400640 1 -- 172.21.15.62:0/4146212343 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0053070 con 0x55906736baf0 2024-03-23T11:27:28.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.203+0000 7fa58f400640 1 -- 172.21.15.62:0/4146212343 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fa5a0006b20 con 0x55906736baf0 2024-03-23T11:27:28.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.203+0000 7fa5b7bce740 1 -- 172.21.15.62:0/4146212343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5590672d9fe0 con 0x55906736baf0 2024-03-23T11:27:28.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.203+0000 7fa58f400640 1 -- 172.21.15.62:0/4146212343 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5a0006ce0 con 0x55906736baf0 2024-03-23T11:27:28.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.203+0000 7fa5b7bce740 10 client.7707 fetch_fsmap learned FSMap version 64 2024-03-23T11:27:28.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.203+0000 7fa5b7bce740 1 -- 172.21.15.62:0/4146212343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x5590672adc00 con 0x55906736baf0 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.203+0000 7fa58f400640 1 -- 172.21.15.62:0/4146212343 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5590672d9fe0 con 0x55906736baf0 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.203+0000 7fa58f400640 1 -- 172.21.15.62:0/4146212343 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa5a00035f0 con 0x55906736baf0 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.204+0000 7fa5b7bce740 10 client.7707 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.204+0000 7fa5b7bce740 10 client.7707 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.204+0000 7fa5b7bce740 20 client.7707 populate_metadata read hostname 'smithi062' 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.204+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5590673773d0 0x5590673a1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.204+0000 7fa5b7bce740 4 client.7707 mds_command: new command op to 7578 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.204+0000 7fa5b7bce740 1 -- 172.21.15.62:0/4146212343 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5590672a2240 con 0x5590673773d0 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.204+0000 7fa5ad400640 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5590673773d0 0x5590673a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.205+0000 7fa5ad400640 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5590673773d0 0x5590673a1670 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39b512a2d6e18a9c server_cookie=1a436b1097470282 in_seq=0 out_seq=0 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.205+0000 7fa58f400640 10 client.7707 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.205+0000 7fa58f400640 1 -- 172.21.15.62:0/4146212343 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5590672a2240 con 0x5590673773d0 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.205+0000 7fa58f400640 10 client.7707 handle_command_reply: tid=0 2024-03-23T11:27:28.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 1 client.7707 shutdown 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 1 -- 172.21.15.62:0/4146212343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 msgr2=0x559067390200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067390200 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0000c00 tx=0x7fa5a002d000 comp rx=0 tx=0).stop 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 1 -- 172.21.15.62:0/4146212343 shutdown_connections 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5590673773d0 0x5590673a1670 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x5590673a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 0x559067390740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/4146212343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x559067390200 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 1 -- 172.21.15.62:0/4146212343 >> 172.21.15.62:0/4146212343 conn(0x559067313b40 msgr2=0x559067392230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa58f400640 0 client.7707 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 1 -- 172.21.15.62:0/4146212343 shutdown_connections 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 1 -- 172.21.15.62:0/4146212343 wait complete. 2024-03-23T11:27:28.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.206+0000 7fa5b7bce740 20 client.7707 trim_cache size 0 max 16384 2024-03-23T11:27:28.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.222+0000 7fa5b7bce740 1 Processor -- start 2024-03-23T11:27:28.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.222+0000 7fa5b7bce740 1 -- start start 2024-03-23T11:27:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.223+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x5590673468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.223+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 0x559067345e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.223+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x559067387290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.223+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559066facf90 con 0x55906736baf0 2024-03-23T11:27:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.223+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590673a19f0 con 0x559067374460 2024-03-23T11:27:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.223+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5590673a1570 con 0x55906736bed0 2024-03-23T11:27:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.223+0000 7fa5ade00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x5590673468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.223+0000 7fa5ade00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x5590673468f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43048/0 (socket says 172.21.15.62:43048) 2024-03-23T11:27:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.223+0000 7fa5ade00640 1 -- 172.21.15.62:0/497901527 learned_addr learned my addr 172.21.15.62:0/497901527 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.224+0000 7fa5ade00640 1 -- 172.21.15.62:0/497901527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 msgr2=0x559067345e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.224+0000 7fa5ade00640 1 --2- 172.21.15.62:0/497901527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 0x559067345e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.224+0000 7fa5ade00640 1 -- 172.21.15.62:0/497901527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 msgr2=0x559067387290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.224+0000 7fa5ade00640 1 --2- 172.21.15.62:0/497901527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x559067387290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.224+0000 7fa5ade00640 1 -- 172.21.15.62:0/497901527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55906724fc60 con 0x55906736baf0 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.224+0000 7fa5ade00640 1 --2- 172.21.15.62:0/497901527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x5590673468f0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fa5a002bda0 tx=0x7fa5a0045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3436bc3d1a63e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.224+0000 7fa59b400640 1 -- 172.21.15.62:0/497901527 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0003b00 con 0x55906736baf0 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.224+0000 7fa59b400640 1 -- 172.21.15.62:0/497901527 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a0006b20 con 0x55906736baf0 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.224+0000 7fa59b400640 1 -- 172.21.15.62:0/497901527 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0006ed0 con 0x55906736baf0 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.225+0000 7fa5b7bce740 1 -- 172.21.15.62:0/497901527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 msgr2=0x5590673468f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.225+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/497901527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x5590673468f0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fa5a002bda0 tx=0x7fa5a0045000 comp rx=0 tx=0).stop 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.225+0000 7fa5b7bce740 1 -- 172.21.15.62:0/497901527 shutdown_connections 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.225+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/497901527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x559067387290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.225+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/497901527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736bed0 0x559067345e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.225+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/497901527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736baf0 0x5590673468f0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.225+0000 7fa5b7bce740 1 -- 172.21.15.62:0/497901527 >> 172.21.15.62:0/497901527 conn(0x559067313b40 msgr2=0x559067379aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.225+0000 7fa5b7bce740 1 -- 172.21.15.62:0/497901527 shutdown_connections 2024-03-23T11:27:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.225+0000 7fa5b7bce740 1 -- 172.21.15.62:0/497901527 wait complete. 2024-03-23T11:27:28.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.226+0000 7fa5b7bce740 1 -- start start 2024-03-23T11:27:28.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.226+0000 7fa5b7bce740 1 Processor -- start 2024-03-23T11:27:28.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736baf0 0x559067379a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736bed0 0x559067379f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5b7bce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x559067387290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590673f6d10 con 0x55906736bed0 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590673f1070 con 0x559067374460 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5b7bce740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5590673721f0 con 0x55906736baf0 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5b7bce740 10 client.0 _ll_register_callbacks cb 0x7fa5940036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5ad400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736bed0 0x559067379f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5ad400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736bed0 0x559067379f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43060/0 (socket says 172.21.15.62:43060) 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5ad400640 1 -- 172.21.15.62:0/955176833 learned_addr learned my addr 172.21.15.62:0/955176833 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5ad400640 1 -- 172.21.15.62:0/955176833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736baf0 msgr2=0x559067379a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.227+0000 7fa5ad400640 1 --2- 172.21.15.62:0/955176833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736baf0 0x559067379a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.228+0000 7fa5ad400640 1 -- 172.21.15.62:0/955176833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 msgr2=0x559067387290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.228+0000 7fa5ad400640 1 --2- 172.21.15.62:0/955176833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x559067387290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.228+0000 7fa5ad400640 1 -- 172.21.15.62:0/955176833 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590672adc00 con 0x55906736bed0 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.228+0000 7fa5ad400640 1 --2- 172.21.15.62:0/955176833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736bed0 0x559067379f50 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x55906734d830 tx=0x7fa59c017810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b312db06699da3f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.228+0000 7fa58f400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.228+0000 7fa58f400640 1 -- 172.21.15.62:0/955176833 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c004230 con 0x55906736bed0 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.228+0000 7fa58f400640 1 -- 172.21.15.62:0/955176833 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fa59c004390 con 0x55906736bed0 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.228+0000 7fa5b7bce740 1 -- 172.21.15.62:0/955176833 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5a0007620 con 0x55906736bed0 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.228+0000 7fa58f400640 1 -- 172.21.15.62:0/955176833 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa59c005990 con 0x55906736bed0 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.229+0000 7fa5b7bce740 10 client.7713 fetch_fsmap learned FSMap version 64 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.229+0000 7fa5b7bce740 1 -- 172.21.15.62:0/955176833 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x559067275170 con 0x55906736bed0 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.229+0000 7fa58f400640 1 -- 172.21.15.62:0/955176833 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a0007620 con 0x55906736bed0 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.229+0000 7fa58f400640 1 -- 172.21.15.62:0/955176833 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa59c005af0 con 0x55906736bed0 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.229+0000 7fa5b7bce740 10 client.7713 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.229+0000 7fa5b7bce740 10 client.7713 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:27:28.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.229+0000 7fa5b7bce740 20 client.7713 populate_metadata read hostname 'smithi062' 2024-03-23T11:27:28.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.229+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/955176833 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x559067345f20 0x5590673877d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.229+0000 7fa5b7bce740 4 client.7713 mds_command: new command op to 7578 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:27:28.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.229+0000 7fa5b7bce740 1 -- 172.21.15.62:0/955176833 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559066fa34e0 con 0x559067345f20 2024-03-23T11:27:28.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.229+0000 7fa5ade00640 1 --2- 172.21.15.62:0/955176833 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x559067345f20 0x5590673877d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5ade00640 1 --2- 172.21.15.62:0/955176833 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x559067345f20 0x5590673877d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=956be384669ef69d server_cookie=bc7043f76b3e1a86 in_seq=0 out_seq=0 2024-03-23T11:27:28.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa58f400640 10 client.7713 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:28.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa58f400640 1 -- 172.21.15.62:0/955176833 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559066fa34e0 con 0x559067345f20 2024-03-23T11:27:28.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa58f400640 10 client.7713 handle_command_reply: tid=0 2024-03-23T11:27:28.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5b7bce740 1 client.7713 shutdown 2024-03-23T11:27:28.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5b7bce740 1 -- 172.21.15.62:0/955176833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736bed0 msgr2=0x559067379f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/955176833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736bed0 0x559067379f50 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x55906734d830 tx=0x7fa59c017810 comp rx=0 tx=0).stop 2024-03-23T11:27:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5b7bce740 1 -- 172.21.15.62:0/955176833 shutdown_connections 2024-03-23T11:27:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/955176833 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x559067345f20 0x5590673877d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/955176833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55906736bed0 0x559067379f50 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/955176833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559067374460 0x559067387290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa58f400640 0 client.7713 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5b7bce740 1 --2- 172.21.15.62:0/955176833 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55906736baf0 0x559067379a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5b7bce740 1 -- 172.21.15.62:0/955176833 >> 172.21.15.62:0/955176833 conn(0x559067313b40 msgr2=0x559067394210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5b7bce740 1 -- 172.21.15.62:0/955176833 shutdown_connections 2024-03-23T11:27:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.230+0000 7fa5b7bce740 1 -- 172.21.15.62:0/955176833 wait complete. 2024-03-23T11:27:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5b7bce740 20 client.7713 trim_cache size 0 max 16384 2024-03-23T11:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:27:28.227 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-03-23T11:27:28.227 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-03-23T11:27:28.227 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:27:28.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 -- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa59006a7a0 msgr2=0x7fa59006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa59006a7a0 0x7fa59006cc60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001750 tx=0x7fa5a0009030 comp rx=0 tx=0).stop 2024-03-23T11:27:28.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 -- 172.21.15.62:0/2358928567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b006d850 msgr2=0x7fa5b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b006d850 0x7fa5b006dcd0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00adfd0 tx=0x7fa5a800e530 comp rx=0 tx=0).stop 2024-03-23T11:27:28.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 -- 172.21.15.62:0/2358928567 shutdown_connections 2024-03-23T11:27:28.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa59006a7a0 0x7fa59006cc60 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b006d850 0x7fa5b006dcd0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5b013d9f0 0x7fa5b00af2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 --2- 172.21.15.62:0/2358928567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa5940039a0 0x7fa5b006c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 -- 172.21.15.62:0/2358928567 >> 172.21.15.62:0/2358928567 conn(0x7fa5b00b0090 msgr2=0x7fa5b00ae000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 -- 172.21.15.62:0/2358928567 shutdown_connections 2024-03-23T11:27:28.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.231+0000 7fa5af200640 1 -- 172.21.15.62:0/2358928567 wait complete. 2024-03-23T11:27:28.241 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T11:27:28.241 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:27:28.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.448+0000 7fab42e00640 1 Processor -- start 2024-03-23T11:27:28.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.449+0000 7fab42e00640 1 -- start start 2024-03-23T11:27:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.450+0000 7fab42e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fab3c06c6e0 0x7fab3c06cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.450+0000 7fab42e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3c0cbc60 con 0x7fab3c0cafa0 2024-03-23T11:27:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.450+0000 7fab42e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3c0cbf40 con 0x7fab3c06c300 2024-03-23T11:27:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.450+0000 7fab42e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3c0cd410 con 0x7fab3c06c6e0 2024-03-23T11:27:28.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.451+0000 7fab3b400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fab3c0cafa0 0x7fab3c0cb430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:46192/0 (socket says 172.21.15.62:46192) 2024-03-23T11:27:28.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.451+0000 7fab3b400640 1 -- 172.21.15.62:0/2690819961 learned_addr learned my addr 172.21.15.62:0/2690819961 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:27:28.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.451+0000 7fab40e00640 1 --2- 172.21.15.62:0/2690819961 >> v2:172.21.15.62:3300/0 conn(0x7fab3c06c6e0 0x7fab3c06cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.451+0000 7fab3aa00640 1 -- 172.21.15.62:0/2690819961 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031297812 0 0) 0x7fab3c0cbc60 con 0x7fab3c0cafa0 2024-03-23T11:27:28.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.451+0000 7fab40e00640 1 -- 172.21.15.62:0/2690819961 >> v1:172.21.15.120:6789/0 conn(0x7fab3c06c300 legacy=0x7fab3c0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.451+0000 7fab40e00640 1 -- 172.21.15.62:0/2690819961 >> v1:172.21.15.62:6789/0 conn(0x7fab3c0cafa0 legacy=0x7fab3c0cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.451+0000 7fab40e00640 1 -- 172.21.15.62:0/2690819961 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c0cc220 con 0x7fab3c06c6e0 2024-03-23T11:27:28.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.453+0000 7fab40e00640 1 --2- 172.21.15.62:0/2690819961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3c06c6e0 0x7fab3c06cb30 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fab34009d90 tx=0x7fab3402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abbb8cee9d70e90 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.453+0000 7fab3aa00640 1 -- 172.21.15.62:0/2690819961 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3402cd00 con 0x7fab3c06c6e0 2024-03-23T11:27:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.453+0000 7fab3aa00640 1 -- 172.21.15.62:0/2690819961 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3402ce60 con 0x7fab3c06c6e0 2024-03-23T11:27:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.453+0000 7fab3aa00640 1 -- 172.21.15.62:0/2690819961 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34034660 con 0x7fab3c06c6e0 2024-03-23T11:27:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.454+0000 7fab42e00640 1 -- 172.21.15.62:0/2690819961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3c06c6e0 msgr2=0x7fab3c06cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.454+0000 7fab42e00640 1 --2- 172.21.15.62:0/2690819961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3c06c6e0 0x7fab3c06cb30 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fab34009d90 tx=0x7fab3402bf50 comp rx=0 tx=0).stop 2024-03-23T11:27:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.454+0000 7fab42e00640 1 -- 172.21.15.62:0/2690819961 shutdown_connections 2024-03-23T11:27:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.454+0000 7fab42e00640 1 --2- 172.21.15.62:0/2690819961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3c06c6e0 0x7fab3c06cb30 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.454+0000 7fab42e00640 1 -- 172.21.15.62:0/2690819961 >> 172.21.15.62:0/2690819961 conn(0x7fab3c0b02e0 msgr2=0x7fab3c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.454+0000 7fab42e00640 1 -- 172.21.15.62:0/2690819961 shutdown_connections 2024-03-23T11:27:28.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.454+0000 7fab42e00640 1 -- 172.21.15.62:0/2690819961 wait complete. 2024-03-23T11:27:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.456+0000 7fab42e00640 1 Processor -- start 2024-03-23T11:27:28.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.456+0000 7fab42e00640 1 -- start start 2024-03-23T11:27:28.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.457+0000 7fab42e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3c06c300 0x7fab3c15adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.457+0000 7fab42e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3c06c6e0 0x7fab3c0b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.457+0000 7fab42e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab3c0cafa0 0x7fab3c13ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.457+0000 7fab42e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c15c670 con 0x7fab3c06c6e0 2024-03-23T11:27:28.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.457+0000 7fab42e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c0b3720 con 0x7fab3c0cafa0 2024-03-23T11:27:28.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.457+0000 7fab42e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c0b27b0 con 0x7fab3c06c300 2024-03-23T11:27:28.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.457+0000 7fab3b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3c06c6e0 0x7fab3c0b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.457+0000 7fab3b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3c06c6e0 0x7fab3c0b31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43080/0 (socket says 172.21.15.62:43080) 2024-03-23T11:27:28.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.457+0000 7fab3b400640 1 -- 172.21.15.62:0/716004940 learned_addr learned my addr 172.21.15.62:0/716004940 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab40e00640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab3c0cafa0 0x7fab3c13ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab3be00640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3c06c300 0x7fab3c15adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab3b400640 1 -- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3c06c300 msgr2=0x7fab3c15adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab3b400640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3c06c300 0x7fab3c15adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab3b400640 1 -- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab3c0cafa0 msgr2=0x7fab3c13ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab3b400640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab3c0cafa0 0x7fab3c13ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab3b400640 1 -- 172.21.15.62:0/716004940 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab34009a70 con 0x7fab3c06c6e0 2024-03-23T11:27:28.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab3b400640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3c06c6e0 0x7fab3c0b31e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fab2400c3f0 tx=0x7fab2400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab33e00640 1 -- 172.21.15.62:0/716004940 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab240074e0 con 0x7fab3c06c6e0 2024-03-23T11:27:28.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab33e00640 1 -- 172.21.15.62:0/716004940 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab24007640 con 0x7fab3c06c6e0 2024-03-23T11:27:28.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab42e00640 1 -- 172.21.15.62:0/716004940 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3c0b1f10 con 0x7fab3c06c6e0 2024-03-23T11:27:28.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab33e00640 1 -- 172.21.15.62:0/716004940 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24004a00 con 0x7fab3c06c6e0 2024-03-23T11:27:28.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.458+0000 7fab42e00640 1 -- 172.21.15.62:0/716004940 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c0b24d0 con 0x7fab3c06c6e0 2024-03-23T11:27:28.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.459+0000 7fab42e00640 1 -- 172.21.15.62:0/716004940 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3c066c90 con 0x7fab3c06c6e0 2024-03-23T11:27:28.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.460+0000 7fab33e00640 1 -- 172.21.15.62:0/716004940 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fab24004b60 con 0x7fab3c06c6e0 2024-03-23T11:27:28.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.461+0000 7fab33e00640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab1806a7a0 0x7fab1806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.461+0000 7fab33e00640 1 -- 172.21.15.62:0/716004940 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fab24088bf0 con 0x7fab3c06c6e0 2024-03-23T11:27:28.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.462+0000 7fab3be00640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab1806a7a0 0x7fab1806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.463+0000 7fab3be00640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab1806a7a0 0x7fab1806cc60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fab2c000f60 tx=0x7fab2c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.468+0000 7fab33e00640 1 -- 172.21.15.62:0/716004940 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fab24053f20 con 0x7fab3c06c6e0 2024-03-23T11:27:28.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.710+0000 7fab42e00640 1 -- 172.21.15.62:0/716004940 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab3c1409e0 con 0x7fab3c06c6e0 2024-03-23T11:27:28.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.712+0000 7fab33e00640 1 -- 172.21.15.62:0/716004940 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fab24057540 con 0x7fab3c06c6e0 2024-03-23T11:27:28.704 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:27:28.704 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:26:59.466628+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":6,"addr":"172.21.15.62:6833/2573443114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2573443114},{"type":"v1","addr":"172.21.15.62:6833","nonce":2573443114}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:27:28.704 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 64 2024-03-23T11:27:28.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.714+0000 7fab42e00640 1 -- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab1806a7a0 msgr2=0x7fab1806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.714+0000 7fab42e00640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab1806a7a0 0x7fab1806cc60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fab2c000f60 tx=0x7fab2c009040 comp rx=0 tx=0).stop 2024-03-23T11:27:28.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.714+0000 7fab42e00640 1 -- 172.21.15.62:0/716004940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3c06c6e0 msgr2=0x7fab3c0b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.714+0000 7fab42e00640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3c06c6e0 0x7fab3c0b31e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fab2400c3f0 tx=0x7fab2400c8b0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.715+0000 7fab42e00640 1 -- 172.21.15.62:0/716004940 shutdown_connections 2024-03-23T11:27:28.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.715+0000 7fab42e00640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab1806a7a0 0x7fab1806cc60 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.715+0000 7fab42e00640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3c06c6e0 0x7fab3c0b31e0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.715+0000 7fab42e00640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab3c0cafa0 0x7fab3c13ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.715+0000 7fab42e00640 1 --2- 172.21.15.62:0/716004940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab3c06c300 0x7fab3c15adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.715+0000 7fab42e00640 1 -- 172.21.15.62:0/716004940 >> 172.21.15.62:0/716004940 conn(0x7fab3c0b02e0 msgr2=0x7fab3c0cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.715+0000 7fab42e00640 1 -- 172.21.15.62:0/716004940 shutdown_connections 2024-03-23T11:27:28.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.715+0000 7fab42e00640 1 -- 172.21.15.62:0/716004940 wait complete. 2024-03-23T11:27:28.726 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-03-23T11:27:28.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.953+0000 7f71dda00640 1 Processor -- start 2024-03-23T11:27:28.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.953+0000 7f71dda00640 1 -- start start 2024-03-23T11:27:28.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.955+0000 7f71dda00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71d80cafa0 0x7f71d80cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.955+0000 7f71dda00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71d806c6e0 0x7f71d806cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.955+0000 7f71dda00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d80cbc60 con 0x7f71d806c300 2024-03-23T11:27:28.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.955+0000 7f71dda00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d80cd410 con 0x7f71d80cafa0 2024-03-23T11:27:28.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.955+0000 7f71dda00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d80cbf40 con 0x7f71d806c6e0 2024-03-23T11:27:28.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.955+0000 7f71ce000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71d80cafa0 0x7f71d80cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.956+0000 7f71ce000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71d80cafa0 0x7f71d80cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43086/0 (socket says 172.21.15.62:43086) 2024-03-23T11:27:28.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.956+0000 7f71ce000640 1 -- 172.21.15.62:0/3333087202 learned_addr learned my addr 172.21.15.62:0/3333087202 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.956+0000 7f71d7400640 1 --2- 172.21.15.62:0/3333087202 >> v2:172.21.15.120:3300/0 conn(0x7f71d806c6e0 0x7f71d806cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.956+0000 7f71ce000640 1 -- 172.21.15.62:0/3333087202 >> v2:172.21.15.120:3300/0 conn(0x7f71d806c6e0 msgr2=0x7f71d806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.956+0000 7f71ce000640 1 --2- 172.21.15.62:0/3333087202 >> v2:172.21.15.120:3300/0 conn(0x7f71d806c6e0 0x7f71d806cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.956+0000 7f71ce000640 1 -- 172.21.15.62:0/3333087202 >> v1:172.21.15.120:6789/0 conn(0x7f71d806c300 legacy=0x7f71d80ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.956+0000 7f71ce000640 1 -- 172.21.15.62:0/3333087202 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d80cc220 con 0x7f71d80cafa0 2024-03-23T11:27:28.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.957+0000 7f71ce000640 1 --2- 172.21.15.62:0/3333087202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71d80cafa0 0x7f71d80cb430 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f71d8070270 tx=0x7f71c002b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=620b687a2dc44e14 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.957+0000 7f71d6000640 1 -- 172.21.15.62:0/3333087202 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0031890 con 0x7f71d80cafa0 2024-03-23T11:27:28.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.958+0000 7f71d6000640 1 -- 172.21.15.62:0/3333087202 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c00319f0 con 0x7f71d80cafa0 2024-03-23T11:27:28.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.958+0000 7f71d6000640 1 -- 172.21.15.62:0/3333087202 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0031d80 con 0x7f71d80cafa0 2024-03-23T11:27:28.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.958+0000 7f71dda00640 1 -- 172.21.15.62:0/3333087202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71d80cafa0 msgr2=0x7f71d80cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.958+0000 7f71dda00640 1 --2- 172.21.15.62:0/3333087202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71d80cafa0 0x7f71d80cb430 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f71d8070270 tx=0x7f71c002b930 comp rx=0 tx=0).stop 2024-03-23T11:27:28.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.958+0000 7f71dda00640 1 -- 172.21.15.62:0/3333087202 shutdown_connections 2024-03-23T11:27:28.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.958+0000 7f71dda00640 1 --2- 172.21.15.62:0/3333087202 >> v2:172.21.15.120:3300/0 conn(0x7f71d806c6e0 0x7f71d806cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.958+0000 7f71dda00640 1 --2- 172.21.15.62:0/3333087202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71d80cafa0 0x7f71d80cb430 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.958+0000 7f71dda00640 1 -- 172.21.15.62:0/3333087202 >> 172.21.15.62:0/3333087202 conn(0x7f71d80b02e0 msgr2=0x7f71d80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.958+0000 7f71dda00640 1 -- 172.21.15.62:0/3333087202 shutdown_connections 2024-03-23T11:27:28.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.958+0000 7f71dda00640 1 -- 172.21.15.62:0/3333087202 wait complete. 2024-03-23T11:27:28.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.960+0000 7f71dda00640 1 Processor -- start 2024-03-23T11:27:28.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.960+0000 7f71dda00640 1 -- start start 2024-03-23T11:27:28.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.961+0000 7f71dda00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d806c300 0x7f71d815b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.961+0000 7f71dda00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71d806c6e0 0x7f71d8157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.961+0000 7f71dda00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d80cafa0 0x7f71d80abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.961+0000 7f71dda00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d815b3e0 con 0x7f71d806c6e0 2024-03-23T11:27:28.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.961+0000 7f71dda00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d815ae20 con 0x7f71d806c300 2024-03-23T11:27:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.961+0000 7f71dda00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d815b100 con 0x7f71d80cafa0 2024-03-23T11:27:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.961+0000 7f71ce000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71d806c6e0 0x7f71d8157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.962+0000 7f71ce000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71d806c6e0 0x7f71d8157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43092/0 (socket says 172.21.15.62:43092) 2024-03-23T11:27:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.962+0000 7f71ce000640 1 -- 172.21.15.62:0/3009180645 learned_addr learned my addr 172.21.15.62:0/3009180645 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.962+0000 7f71d6a00640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d806c300 0x7f71d815b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.962+0000 7f71d7400640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d80cafa0 0x7f71d80abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.962+0000 7f71ce000640 1 -- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d80cafa0 msgr2=0x7f71d80abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.962+0000 7f71ce000640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d80cafa0 0x7f71d80abb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.962+0000 7f71ce000640 1 -- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d806c300 msgr2=0x7f71d815b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.962+0000 7f71ce000640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d806c300 0x7f71d815b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.962+0000 7f71ce000640 1 -- 172.21.15.62:0/3009180645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c00094d0 con 0x7f71d806c6e0 2024-03-23T11:27:28.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.962+0000 7f71ce000640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71d806c6e0 0x7f71d8157060 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f71c002b970 tx=0x7f71c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.963+0000 7f71cf400640 1 -- 172.21.15.62:0/3009180645 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c003ac60 con 0x7f71d806c6e0 2024-03-23T11:27:28.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.963+0000 7f71cf400640 1 -- 172.21.15.62:0/3009180645 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c003adc0 con 0x7f71d806c6e0 2024-03-23T11:27:28.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.963+0000 7f71cf400640 1 -- 172.21.15.62:0/3009180645 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0044630 con 0x7f71d806c6e0 2024-03-23T11:27:28.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.963+0000 7f71dda00640 1 -- 172.21.15.62:0/3009180645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d80b2fc0 con 0x7f71d806c6e0 2024-03-23T11:27:28.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.963+0000 7f71dda00640 1 -- 172.21.15.62:0/3009180645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d815bc70 con 0x7f71d806c6e0 2024-03-23T11:27:28.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.966+0000 7f71cf400640 1 -- 172.21.15.62:0/3009180645 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f71c004d450 con 0x7f71d806c6e0 2024-03-23T11:27:28.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.967+0000 7f71cf400640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71b406a7a0 0x7f71b406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.967+0000 7f71cf400640 1 -- 172.21.15.62:0/3009180645 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f71c00ab550 con 0x7f71d806c6e0 2024-03-23T11:27:28.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.968+0000 7f71d6a00640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71b406a7a0 0x7f71b406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.968+0000 7f71e05a9740 1 Processor -- start 2024-03-23T11:27:28.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.968+0000 7f71e05a9740 1 -- start start 2024-03-23T11:27:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71d6a00640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71b406a7a0 0x7f71b406cc60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f71c8004ea0 tx=0x7f71c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469b8980 0x55b0469b8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469bc9d0 0x55b0469be470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71e05a9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b046928d80 con 0x55b0469e8710 2024-03-23T11:27:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71e05a9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b046926350 con 0x55b0469bc9d0 2024-03-23T11:27:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71e05a9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b046921c40 con 0x55b0469b8980 2024-03-23T11:27:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71d7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469e8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71d7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469e8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43102/0 (socket says 172.21.15.62:43102) 2024-03-23T11:27:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71d7400640 1 -- 172.21.15.62:0/785109967 learned_addr learned my addr 172.21.15.62:0/785109967 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71d6a00640 1 --2- 172.21.15.62:0/785109967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469b8980 0x55b0469b8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.969+0000 7f71ce000640 1 --2- 172.21.15.62:0/785109967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469bc9d0 0x55b0469be470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.970+0000 7f71d7400640 1 -- 172.21.15.62:0/785109967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469b8980 msgr2=0x55b0469b8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.970+0000 7f71d7400640 1 --2- 172.21.15.62:0/785109967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469b8980 0x55b0469b8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.970+0000 7f71d7400640 1 -- 172.21.15.62:0/785109967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469bc9d0 msgr2=0x55b0469be470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.970+0000 7f71d7400640 1 --2- 172.21.15.62:0/785109967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469bc9d0 0x55b0469be470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.970+0000 7f71d7400640 1 -- 172.21.15.62:0/785109967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0468ec3e0 con 0x55b0469e8710 2024-03-23T11:27:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.970+0000 7f71d7400640 1 --2- 172.21.15.62:0/785109967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469e8b90 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f71d0007360 tx=0x7f71d0007820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ffedf8dec3a8be server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.970+0000 7f71c7400640 1 -- 172.21.15.62:0/785109967 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0013780 con 0x55b0469e8710 2024-03-23T11:27:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.970+0000 7f71c7400640 1 -- 172.21.15.62:0/785109967 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d000f050 con 0x55b0469e8710 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.970+0000 7f71c7400640 1 -- 172.21.15.62:0/785109967 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0013b00 con 0x55b0469e8710 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.971+0000 7f71e05a9740 1 -- 172.21.15.62:0/785109967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 msgr2=0x55b0469e8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.971+0000 7f71e05a9740 1 --2- 172.21.15.62:0/785109967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469e8b90 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f71d0007360 tx=0x7f71d0007820 comp rx=0 tx=0).stop 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.971+0000 7f71e05a9740 1 -- 172.21.15.62:0/785109967 shutdown_connections 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.971+0000 7f71e05a9740 1 --2- 172.21.15.62:0/785109967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469e8b90 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.971+0000 7f71e05a9740 1 --2- 172.21.15.62:0/785109967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469bc9d0 0x55b0469be470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.971+0000 7f71e05a9740 1 --2- 172.21.15.62:0/785109967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469b8980 0x55b0469b8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.971+0000 7f71e05a9740 1 -- 172.21.15.62:0/785109967 >> 172.21.15.62:0/785109967 conn(0x55b0469baa00 msgr2=0x55b0469bc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.971+0000 7f71e05a9740 1 -- 172.21.15.62:0/785109967 shutdown_connections 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.971+0000 7f71e05a9740 1 -- 172.21.15.62:0/785109967 wait complete. 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.972+0000 7f71e05a9740 1 -- start start 2024-03-23T11:27:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.972+0000 7f71e05a9740 1 Processor -- start 2024-03-23T11:27:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469ec280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469cc500 0x55b0469cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8af0 0x55b0469e8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71e05a9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0468c2ce0 con 0x55b0469e8710 2024-03-23T11:27:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71e05a9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0468c2980 con 0x55b0469e8af0 2024-03-23T11:27:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71e05a9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b04668a000 con 0x55b0469cc500 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71e05a9740 10 client.0 _ll_register_callbacks cb 0x55b04691b410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71d6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469ec280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71d6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469ec280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43106/0 (socket says 172.21.15.62:43106) 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71d6a00640 1 -- 172.21.15.62:0/1612111080 learned_addr learned my addr 172.21.15.62:0/1612111080 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71ce000640 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469cc500 0x55b0469cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.973+0000 7f71d7400640 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8af0 0x55b0469e8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.974+0000 7f71d6a00640 1 -- 172.21.15.62:0/1612111080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469cc500 msgr2=0x55b0469cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.974+0000 7f71d6a00640 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469cc500 0x55b0469cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.974+0000 7f71d6a00640 1 -- 172.21.15.62:0/1612111080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8af0 msgr2=0x55b0469e8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.974+0000 7f71d6a00640 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8af0 0x55b0469e8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.974+0000 7f71d6a00640 1 -- 172.21.15.62:0/1612111080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d0009c40 con 0x55b0469e8710 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.974+0000 7f71d7400640 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8af0 0x55b0469e8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.974+0000 7f71d6a00640 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469ec280 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f71c800ee60 tx=0x7f71c80036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4692d8dd8bde6f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.974+0000 7f71bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.974+0000 7f71bfe00640 1 -- 172.21.15.62:0/1612111080 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8021420 con 0x55b0469e8710 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.974+0000 7f71bfe00640 1 -- 172.21.15.62:0/1612111080 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f71c8021580 con 0x55b0469e8710 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.974+0000 7f71e05a9740 1 -- 172.21.15.62:0/1612111080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b0465d38d0 con 0x55b0469e8710 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71bfe00640 1 -- 172.21.15.62:0/1612111080 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71c8003800 con 0x55b0469e8710 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71e05a9740 10 client.7734 fetch_fsmap learned FSMap version 64 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71e05a9740 1 -- 172.21.15.62:0/1612111080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55b0469202c0 con 0x55b0469e8710 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71bfe00640 1 -- 172.21.15.62:0/1612111080 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b0465d38d0 con 0x55b0469e8710 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71bfe00640 1 -- 172.21.15.62:0/1612111080 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f71c8021740 con 0x55b0469e8710 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71e05a9740 10 client.7734 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71e05a9740 10 client.7734 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71e05a9740 20 client.7734 populate_metadata read hostname 'smithi062' 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71e05a9740 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b0469e5a10 0x55b046a16420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71e05a9740 4 client.7734 mds_command: new command op to 7578 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:27:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71e05a9740 1 -- 172.21.15.62:0/1612111080 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b0466164a0 con 0x55b0469e5a10 2024-03-23T11:27:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.975+0000 7f71ce000640 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b0469e5a10 0x55b046a16420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.976+0000 7f71ce000640 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b0469e5a10 0x55b046a16420 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3aeba6499e36af21 server_cookie=6723a3740cf34d92 in_seq=0 out_seq=0 2024-03-23T11:27:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.976+0000 7f71bfe00640 10 client.7734 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.976+0000 7f71bfe00640 1 -- 172.21.15.62:0/1612111080 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55b0466164a0 con 0x55b0469e5a10 2024-03-23T11:27:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.976+0000 7f71bfe00640 10 client.7734 handle_command_reply: tid=0 2024-03-23T11:27:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 1 client.7734 shutdown 2024-03-23T11:27:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 1 -- 172.21.15.62:0/1612111080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 msgr2=0x55b0469ec280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469ec280 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f71c800ee60 tx=0x7f71c80036f0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 1 -- 172.21.15.62:0/1612111080 shutdown_connections 2024-03-23T11:27:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b0469e5a10 0x55b046a16420 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8af0 0x55b0469e8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469cc500 0x55b0469cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71bfe00640 0 client.7734 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 1 --2- 172.21.15.62:0/1612111080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469e8710 0x55b0469ec280 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 1 -- 172.21.15.62:0/1612111080 >> 172.21.15.62:0/1612111080 conn(0x55b0469b8980 msgr2=0x55b0469be8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 1 -- 172.21.15.62:0/1612111080 shutdown_connections 2024-03-23T11:27:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 1 -- 172.21.15.62:0/1612111080 wait complete. 2024-03-23T11:27:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.977+0000 7f71e05a9740 20 client.7734 trim_cache size 0 max 16384 2024-03-23T11:27:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.990+0000 7f71e05a9740 1 Processor -- start 2024-03-23T11:27:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.990+0000 7f71e05a9740 1 -- start start 2024-03-23T11:27:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a60030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8710 0x55b0469df2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469e8af0 0x55b0469df800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71e05a9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0469e4bd0 con 0x55b0469cc500 2024-03-23T11:27:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71e05a9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0469bd470 con 0x55b0469e8710 2024-03-23T11:27:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71e05a9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b046a60570 con 0x55b0469e8af0 2024-03-23T11:27:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71d6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a60030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71d6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a60030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43118/0 (socket says 172.21.15.62:43118) 2024-03-23T11:27:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71d6a00640 1 -- 172.21.15.62:0/1515961210 learned_addr learned my addr 172.21.15.62:0/1515961210 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71d6a00640 1 -- 172.21.15.62:0/1515961210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469e8af0 msgr2=0x55b0469df800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71ce000640 1 --2- 172.21.15.62:0/1515961210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8710 0x55b0469df2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71d6a00640 1 --2- 172.21.15.62:0/1515961210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469e8af0 0x55b0469df800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71d6a00640 1 -- 172.21.15.62:0/1515961210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8710 msgr2=0x55b0469df2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71d6a00640 1 --2- 172.21.15.62:0/1515961210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8710 0x55b0469df2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.991+0000 7f71d6a00640 1 -- 172.21.15.62:0/1515961210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0468c2170 con 0x55b0469cc500 2024-03-23T11:27:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.992+0000 7f71d6a00640 1 --2- 172.21.15.62:0/1515961210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a60030 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f71c800e5d0 tx=0x7f71c801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea0019cbb1577c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.992+0000 7f71c7e00640 1 -- 172.21.15.62:0/1515961210 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8036070 con 0x55b0469cc500 2024-03-23T11:27:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.992+0000 7f71c7e00640 1 -- 172.21.15.62:0/1515961210 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c8005420 con 0x55b0469cc500 2024-03-23T11:27:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.992+0000 7f71c7e00640 1 -- 172.21.15.62:0/1515961210 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8021c00 con 0x55b0469cc500 2024-03-23T11:27:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.993+0000 7f71e05a9740 1 -- 172.21.15.62:0/1515961210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 msgr2=0x55b046a60030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.993+0000 7f71e05a9740 1 --2- 172.21.15.62:0/1515961210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a60030 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f71c800e5d0 tx=0x7f71c801f000 comp rx=0 tx=0).stop 2024-03-23T11:27:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.993+0000 7f71e05a9740 1 -- 172.21.15.62:0/1515961210 shutdown_connections 2024-03-23T11:27:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.993+0000 7f71e05a9740 1 --2- 172.21.15.62:0/1515961210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469e8af0 0x55b0469df800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.993+0000 7f71e05a9740 1 --2- 172.21.15.62:0/1515961210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8710 0x55b0469df2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.993+0000 7f71e05a9740 1 --2- 172.21.15.62:0/1515961210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a60030 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.993+0000 7f71e05a9740 1 -- 172.21.15.62:0/1515961210 >> 172.21.15.62:0/1515961210 conn(0x55b0469b8980 msgr2=0x55b0469bc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.993+0000 7f71e05a9740 1 -- 172.21.15.62:0/1515961210 shutdown_connections 2024-03-23T11:27:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.993+0000 7f71e05a9740 1 -- 172.21.15.62:0/1515961210 wait complete. 2024-03-23T11:27:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.994+0000 7f71e05a9740 1 -- start start 2024-03-23T11:27:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.994+0000 7f71e05a9740 1 Processor -- start 2024-03-23T11:27:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.995+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a010e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.995+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469e8710 0x55b0469df6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.995+0000 7f71e05a9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8af0 0x55b046a56230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.995+0000 7f71e05a9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b046a1c910 con 0x55b0469cc500 2024-03-23T11:27:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.995+0000 7f71e05a9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b046a032b0 con 0x55b0469e8af0 2024-03-23T11:27:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.995+0000 7f71e05a9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0469dfc00 con 0x55b0469e8710 2024-03-23T11:27:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.995+0000 7f71e05a9740 10 client.0 _ll_register_callbacks cb 0x55b0469b91f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:27:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.995+0000 7f71d6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.995+0000 7f71d6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a010e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43128/0 (socket says 172.21.15.62:43128) 2024-03-23T11:27:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.995+0000 7f71d6a00640 1 -- 172.21.15.62:0/4163429450 learned_addr learned my addr 172.21.15.62:0/4163429450 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.996+0000 7f71ce000640 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469e8710 0x55b0469df6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.996+0000 7f71d6a00640 1 -- 172.21.15.62:0/4163429450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469e8710 msgr2=0x55b0469df6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.996+0000 7f71d6a00640 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469e8710 0x55b0469df6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.996+0000 7f71d6a00640 1 -- 172.21.15.62:0/4163429450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8af0 msgr2=0x55b046a56230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.996+0000 7f71d6a00640 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8af0 0x55b046a56230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.996+0000 7f71d6a00640 1 -- 172.21.15.62:0/4163429450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0469202c0 con 0x55b0469cc500 2024-03-23T11:27:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.996+0000 7f71d6a00640 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a010e0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f71c800ee60 tx=0x7f71c8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbed828c1c791432 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.996+0000 7f71bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.996+0000 7f71bfe00640 1 -- 172.21.15.62:0/4163429450 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8036040 con 0x55b0469cc500 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.996+0000 7f71bfe00640 1 -- 172.21.15.62:0/4163429450 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f71c80216f0 con 0x55b0469cc500 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.996+0000 7f71e05a9740 1 -- 172.21.15.62:0/4163429450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f71d000f050 con 0x55b0469cc500 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71bfe00640 1 -- 172.21.15.62:0/4163429450 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71c80218b0 con 0x55b0469cc500 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71e05a9740 10 client.7740 fetch_fsmap learned FSMap version 64 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71e05a9740 1 -- 172.21.15.62:0/4163429450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55b0468e7a50 con 0x55b0469cc500 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71bfe00640 1 -- 172.21.15.62:0/4163429450 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d000f050 con 0x55b0469cc500 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71bfe00640 1 -- 172.21.15.62:0/4163429450 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f71c8003800 con 0x55b0469cc500 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71e05a9740 10 client.7740 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71e05a9740 10 client.7740 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71e05a9740 20 client.7740 populate_metadata read hostname 'smithi062' 2024-03-23T11:27:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71e05a9740 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b046a56930 0x55b046a5db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71e05a9740 4 client.7740 mds_command: new command op to 7578 tid=0[{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71e05a9740 1 -- 172.21.15.62:0/4163429450 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x55b0466277a0 con 0x55b046a56930 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.997+0000 7f71ce000640 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b046a56930 0x55b046a5db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71ce000640 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b046a56930 0x55b046a5db00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=719ca2c07e9149ed server_cookie=36ad68450a7c2e54 in_seq=0 out_seq=0 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71bfe00640 10 client.7740 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71bfe00640 1 -- 172.21.15.62:0/4163429450 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+419 (crc 0 0 0) 0x55b0466277a0 con 0x55b046a56930 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71bfe00640 10 client.7740 handle_command_reply: tid=0 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71e05a9740 1 client.7740 shutdown 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71e05a9740 1 -- 172.21.15.62:0/4163429450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 msgr2=0x55b046a010e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71e05a9740 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a010e0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f71c800ee60 tx=0x7f71c8013000 comp rx=0 tx=0).stop 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71e05a9740 1 -- 172.21.15.62:0/4163429450 shutdown_connections 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71e05a9740 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b046a56930 0x55b046a5db00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71e05a9740 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0469e8af0 0x55b046a56230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71e05a9740 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b0469e8710 0x55b0469df6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71e05a9740 1 --2- 172.21.15.62:0/4163429450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0469cc500 0x55b046a010e0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71bfe00640 0 client.7740 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71e05a9740 1 -- 172.21.15.62:0/4163429450 >> 172.21.15.62:0/4163429450 conn(0x55b0469b8980 msgr2=0x55b0469be880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.998+0000 7f71e05a9740 1 -- 172.21.15.62:0/4163429450 shutdown_connections 2024-03-23T11:27:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71e05a9740 1 -- 172.21.15.62:0/4163429450 wait complete. 2024-03-23T11:27:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71e05a9740 20 client.7740 trim_cache size 0 max 16384 2024-03-23T11:27:28.994 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:27:28.994 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:27:28.994 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:27:28.994 INFO:teuthology.orchestra.run.smithi062.stdout: "damage_type": "backtrace", 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 846551848, 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: "ino": 1099511627777, 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/testdir/hardlink" 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: "damage_type": "dir_frag", 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 2716107859, 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: "ino": 1099511627776, 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: "frag": "*", 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/dir" 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: "damage_type": "backtrace", 2024-03-23T11:27:28.995 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 3792386667, 2024-03-23T11:27:28.996 INFO:teuthology.orchestra.run.smithi062.stdout: "ino": 1099511627776, 2024-03-23T11:27:28.996 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/dir" 2024-03-23T11:27:28.996 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:27:28.996 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:27:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71dda00640 1 -- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71b406a7a0 msgr2=0x7f71b406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71dda00640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71b406a7a0 0x7f71b406cc60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f71c8004ea0 tx=0x7f71c8009040 comp rx=0 tx=0).stop 2024-03-23T11:27:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71dda00640 1 -- 172.21.15.62:0/3009180645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71d806c6e0 msgr2=0x7f71d8157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71dda00640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71d806c6e0 0x7f71d8157060 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f71c002b970 tx=0x7f71c00046c0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71dda00640 1 -- 172.21.15.62:0/3009180645 shutdown_connections 2024-03-23T11:27:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71dda00640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71b406a7a0 0x7f71b406cc60 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71dda00640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71d806c6e0 0x7f71d8157060 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71dda00640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d80cafa0 0x7f71d80abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71dda00640 1 --2- 172.21.15.62:0/3009180645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d806c300 0x7f71d815b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71dda00640 1 -- 172.21.15.62:0/3009180645 >> 172.21.15.62:0/3009180645 conn(0x7f71d80b02e0 msgr2=0x7f71d813df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:28.999+0000 7f71dda00640 1 -- 172.21.15.62:0/3009180645 shutdown_connections 2024-03-23T11:27:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.000+0000 7f71dda00640 1 -- 172.21.15.62:0/3009180645 wait complete. 2024-03-23T11:27:29.008 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-23T11:27:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.235+0000 7fe925400640 1 Processor -- start 2024-03-23T11:27:29.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.235+0000 7fe925400640 1 -- start start 2024-03-23T11:27:29.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.237+0000 7fe925400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe920143580 0x7fe9201440b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.237+0000 7fe925400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe9200b2670 0x7fe920071b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.237+0000 7fe925400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9200b2e50 con 0x7fe9200720a0 2024-03-23T11:27:29.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.237+0000 7fe925400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9200b3130 con 0x7fe9200b2670 2024-03-23T11:27:29.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.237+0000 7fe925400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9200b3410 con 0x7fe920143580 2024-03-23T11:27:29.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.237+0000 7fe917e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe9200b2670 0x7fe920071b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.237+0000 7fe917e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe9200b2670 0x7fe920071b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43138/0 (socket says 172.21.15.62:43138) 2024-03-23T11:27:29.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.237+0000 7fe917e00640 1 -- 172.21.15.62:0/2990912923 learned_addr learned my addr 172.21.15.62:0/2990912923 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:29.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.237+0000 7fe91f200640 1 --1- 172.21.15.62:0/2990912923 >> v1:172.21.15.62:6789/0 conn(0x7fe9200720a0 0x7fe920069c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:46204/0 (socket says 172.21.15.62:46204) 2024-03-23T11:27:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.238+0000 7fe91e800640 1 --2- 172.21.15.62:0/2990912923 >> v2:172.21.15.120:3300/0 conn(0x7fe920143580 0x7fe9201440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.238+0000 7fe917e00640 1 -- 172.21.15.62:0/2990912923 >> v2:172.21.15.120:3300/0 conn(0x7fe920143580 msgr2=0x7fe9201440b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.238+0000 7fe917e00640 1 --2- 172.21.15.62:0/2990912923 >> v2:172.21.15.120:3300/0 conn(0x7fe920143580 0x7fe9201440b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.238+0000 7fe917e00640 1 -- 172.21.15.62:0/2990912923 >> v1:172.21.15.62:6789/0 conn(0x7fe9200720a0 legacy=0x7fe920069c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.238+0000 7fe917e00640 1 -- 172.21.15.62:0/2990912923 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe920071360 con 0x7fe9200b2670 2024-03-23T11:27:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.239+0000 7fe917e00640 1 --2- 172.21.15.62:0/2990912923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9200b2670 0x7fe920071b60 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fe92006e350 tx=0x7fe90802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9774e13d970dfb2d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.239+0000 7fe91de00640 1 -- 172.21.15.62:0/2990912923 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908030860 con 0x7fe9200b2670 2024-03-23T11:27:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.239+0000 7fe91de00640 1 -- 172.21.15.62:0/2990912923 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9080309c0 con 0x7fe9200b2670 2024-03-23T11:27:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.239+0000 7fe91de00640 1 -- 172.21.15.62:0/2990912923 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908030d50 con 0x7fe9200b2670 2024-03-23T11:27:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.240+0000 7fe925400640 1 -- 172.21.15.62:0/2990912923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9200b2670 msgr2=0x7fe920071b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.240+0000 7fe925400640 1 --2- 172.21.15.62:0/2990912923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9200b2670 0x7fe920071b60 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fe92006e350 tx=0x7fe90802b9a0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.240+0000 7fe925400640 1 -- 172.21.15.62:0/2990912923 shutdown_connections 2024-03-23T11:27:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.240+0000 7fe925400640 1 --2- 172.21.15.62:0/2990912923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9200b2670 0x7fe920071b60 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.240+0000 7fe925400640 1 --2- 172.21.15.62:0/2990912923 >> v2:172.21.15.120:3300/0 conn(0x7fe920143580 0x7fe9201440b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.240+0000 7fe925400640 1 -- 172.21.15.62:0/2990912923 >> 172.21.15.62:0/2990912923 conn(0x7fe9200b1370 msgr2=0x7fe9200b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.240+0000 7fe925400640 1 -- 172.21.15.62:0/2990912923 shutdown_connections 2024-03-23T11:27:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.240+0000 7fe925400640 1 -- 172.21.15.62:0/2990912923 wait complete. 2024-03-23T11:27:29.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.242+0000 7fe925400640 1 Processor -- start 2024-03-23T11:27:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.242+0000 7fe925400640 1 -- start start 2024-03-23T11:27:29.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.243+0000 7fe925400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9200720a0 0x7fe92015ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.243+0000 7fe925400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9200b2670 0x7fe920138d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.243+0000 7fe925400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe920143580 0x7fe920135dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.243+0000 7fe925400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92015bff0 con 0x7fe9200b2670 2024-03-23T11:27:29.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.243+0000 7fe925400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92015a550 con 0x7fe9200720a0 2024-03-23T11:27:29.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.243+0000 7fe925400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92015b370 con 0x7fe920143580 2024-03-23T11:27:29.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.244+0000 7fe917e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9200b2670 0x7fe920138d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.244+0000 7fe917e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9200b2670 0x7fe920138d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43140/0 (socket says 172.21.15.62:43140) 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.244+0000 7fe917e00640 1 -- 172.21.15.62:0/1709098548 learned_addr learned my addr 172.21.15.62:0/1709098548 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.244+0000 7fe91e800640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9200720a0 0x7fe92015ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.244+0000 7fe91f200640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe920143580 0x7fe920135dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.244+0000 7fe917e00640 1 -- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe920143580 msgr2=0x7fe920135dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.244+0000 7fe917e00640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe920143580 0x7fe920135dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.244+0000 7fe917e00640 1 -- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9200720a0 msgr2=0x7fe92015ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.244+0000 7fe917e00640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9200720a0 0x7fe92015ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.244+0000 7fe917e00640 1 -- 172.21.15.62:0/1709098548 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9080094d0 con 0x7fe9200b2670 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.244+0000 7fe917e00640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9200b2670 0x7fe920138d10 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fe90802b9e0 tx=0x7fe908004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.245+0000 7fe916a00640 1 -- 172.21.15.62:0/1709098548 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908039d60 con 0x7fe9200b2670 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.245+0000 7fe916a00640 1 -- 172.21.15.62:0/1709098548 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe908043400 con 0x7fe9200b2670 2024-03-23T11:27:29.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.245+0000 7fe916a00640 1 -- 172.21.15.62:0/1709098548 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9080437b0 con 0x7fe9200b2670 2024-03-23T11:27:29.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.245+0000 7fe925400640 1 -- 172.21.15.62:0/1709098548 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92015b650 con 0x7fe9200b2670 2024-03-23T11:27:29.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.245+0000 7fe925400640 1 -- 172.21.15.62:0/1709098548 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92015bc10 con 0x7fe9200b2670 2024-03-23T11:27:29.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.246+0000 7fe925400640 1 -- 172.21.15.62:0/1709098548 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9200b4340 con 0x7fe9200b2670 2024-03-23T11:27:29.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.247+0000 7fe916a00640 1 -- 172.21.15.62:0/1709098548 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe908043910 con 0x7fe9200b2670 2024-03-23T11:27:29.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.248+0000 7fe916a00640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8fc06a7a0 0x7fe8fc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.248+0000 7fe916a00640 1 -- 172.21.15.62:0/1709098548 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fe90803e070 con 0x7fe9200b2670 2024-03-23T11:27:29.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.248+0000 7fe91e800640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8fc06a7a0 0x7fe8fc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.249+0000 7fe91e800640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8fc06a7a0 0x7fe8fc06cc60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe910006fd0 tx=0x7fe910008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:29.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.253+0000 7fe916a00640 1 -- 172.21.15.62:0/1709098548 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe908077410 con 0x7fe9200b2670 2024-03-23T11:27:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.496+0000 7fe925400640 1 -- 172.21.15.62:0/1709098548 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe9200672c0 con 0x7fe9200b2670 2024-03-23T11:27:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.497+0000 7fe916a00640 1 -- 172.21.15.62:0/1709098548 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7fe9080340c0 con 0x7fe9200b2670 2024-03-23T11:27:29.490 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:27:29.490 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-03-23T11:27:29.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 -- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8fc06a7a0 msgr2=0x7fe8fc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8fc06a7a0 0x7fe8fc06cc60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe910006fd0 tx=0x7fe910008040 comp rx=0 tx=0).stop 2024-03-23T11:27:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 -- 172.21.15.62:0/1709098548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9200b2670 msgr2=0x7fe920138d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9200b2670 0x7fe920138d10 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fe90802b9e0 tx=0x7fe908004730 comp rx=0 tx=0).stop 2024-03-23T11:27:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 -- 172.21.15.62:0/1709098548 shutdown_connections 2024-03-23T11:27:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe8fc06a7a0 0x7fe8fc06cc60 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9200b2670 0x7fe920138d10 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe920143580 0x7fe920135dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 --2- 172.21.15.62:0/1709098548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9200720a0 0x7fe92015ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 -- 172.21.15.62:0/1709098548 >> 172.21.15.62:0/1709098548 conn(0x7fe9200b1370 msgr2=0x7fe920142460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 -- 172.21.15.62:0/1709098548 shutdown_connections 2024-03-23T11:27:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.500+0000 7fe925400640 1 -- 172.21.15.62:0/1709098548 wait complete. 2024-03-23T11:27:29.513 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-23T11:27:29.513 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-03-23T11:27:29.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.769+0000 7f82a0a00640 1 Processor -- start 2024-03-23T11:27:29.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.769+0000 7f82a0a00640 1 -- start start 2024-03-23T11:27:29.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.771+0000 7f82a0a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f829c154d30 0x7f829c0ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.771+0000 7f82a0a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f829c0cebd0 0x7f829c0ab140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.771+0000 7f82a0a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f829c0ab680 0x7f829c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.771+0000 7f82a0a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829c0cf080 con 0x7f829c154d30 2024-03-23T11:27:29.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.771+0000 7f82a0a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829c15bcb0 con 0x7f829c0cebd0 2024-03-23T11:27:29.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.771+0000 7f82a0a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c15ac90 con 0x7f829c0ab680 2024-03-23T11:27:29.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.771+0000 7f829b400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f829c154d30 0x7f829c0ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.771+0000 7f829b400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f829c154d30 0x7f829c0ce690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43144/0 (socket says 172.21.15.62:43144) 2024-03-23T11:27:29.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.771+0000 7f829b400640 1 -- 172.21.15.62:0/1674240169 learned_addr learned my addr 172.21.15.62:0/1674240169 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:29.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.772+0000 7f829be00640 1 --2- 172.21.15.62:0/1674240169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f829c0ab680 0x7f829c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.772+0000 7f829aa00640 1 --2- 172.21.15.62:0/1674240169 >> v2:172.21.15.120:3300/0 conn(0x7f829c0cebd0 0x7f829c0ab140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.772+0000 7f829b400640 1 -- 172.21.15.62:0/1674240169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f829c0ab680 msgr2=0x7f829c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:29.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.772+0000 7f829b400640 1 --2- 172.21.15.62:0/1674240169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f829c0ab680 0x7f829c0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.772+0000 7f829b400640 1 -- 172.21.15.62:0/1674240169 >> v2:172.21.15.120:3300/0 conn(0x7f829c0cebd0 msgr2=0x7f829c0ab140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:29.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.772+0000 7f829b400640 1 --2- 172.21.15.62:0/1674240169 >> v2:172.21.15.120:3300/0 conn(0x7f829c0cebd0 0x7f829c0ab140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.772+0000 7f829b400640 1 -- 172.21.15.62:0/1674240169 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c15af30 con 0x7f829c154d30 2024-03-23T11:27:29.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.773+0000 7f829b400640 1 --2- 172.21.15.62:0/1674240169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c154d30 0x7f829c0ce690 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f829c072330 tx=0x7f828c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99495f1adae2b795 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:29.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.773+0000 7f829a000640 1 -- 172.21.15.62:0/1674240169 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c031860 con 0x7f829c154d30 2024-03-23T11:27:29.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.773+0000 7f829a000640 1 -- 172.21.15.62:0/1674240169 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828c0319c0 con 0x7f829c154d30 2024-03-23T11:27:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.773+0000 7f829a000640 1 -- 172.21.15.62:0/1674240169 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c031d50 con 0x7f829c154d30 2024-03-23T11:27:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.774+0000 7f82a0a00640 1 -- 172.21.15.62:0/1674240169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c154d30 msgr2=0x7f829c0ce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.774+0000 7f82a0a00640 1 --2- 172.21.15.62:0/1674240169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c154d30 0x7f829c0ce690 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f829c072330 tx=0x7f828c02b900 comp rx=0 tx=0).stop 2024-03-23T11:27:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.774+0000 7f82a0a00640 1 -- 172.21.15.62:0/1674240169 shutdown_connections 2024-03-23T11:27:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.774+0000 7f82a0a00640 1 --2- 172.21.15.62:0/1674240169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f829c0ab680 0x7f829c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.774+0000 7f82a0a00640 1 --2- 172.21.15.62:0/1674240169 >> v2:172.21.15.120:3300/0 conn(0x7f829c0cebd0 0x7f829c0ab140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.774+0000 7f82a0a00640 1 --2- 172.21.15.62:0/1674240169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c154d30 0x7f829c0ce690 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.774+0000 7f82a0a00640 1 -- 172.21.15.62:0/1674240169 >> 172.21.15.62:0/1674240169 conn(0x7f829c0b0280 msgr2=0x7f829c0b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.774+0000 7f82a0a00640 1 -- 172.21.15.62:0/1674240169 shutdown_connections 2024-03-23T11:27:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.774+0000 7f82a0a00640 1 -- 172.21.15.62:0/1674240169 wait complete. 2024-03-23T11:27:29.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.776+0000 7f82a0a00640 1 Processor -- start 2024-03-23T11:27:29.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.776+0000 7f82a0a00640 1 -- start start 2024-03-23T11:27:29.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.777+0000 7f82a0a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f829c0ab680 0x7f829c1512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.777+0000 7f82a0a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f829c0cebd0 0x7f829c159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.777+0000 7f82a0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c154d30 0x7f829c155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.777+0000 7f82a0a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c151830 con 0x7f829c154d30 2024-03-23T11:27:29.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.777+0000 7f82a0a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c158d60 con 0x7f829c0ab680 2024-03-23T11:27:29.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.777+0000 7f82a0a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0982f0 con 0x7f829c0cebd0 2024-03-23T11:27:29.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.777+0000 7f829be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c154d30 0x7f829c155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.777+0000 7f829be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c154d30 0x7f829c155b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43154/0 (socket says 172.21.15.62:43154) 2024-03-23T11:27:29.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.777+0000 7f829be00640 1 -- 172.21.15.62:0/1312337483 learned_addr learned my addr 172.21.15.62:0/1312337483 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:29.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.778+0000 7f829b400640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f829c0ab680 0x7f829c1512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.778+0000 7f829aa00640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f829c0cebd0 0x7f829c159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.778+0000 7f829be00640 1 -- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f829c0cebd0 msgr2=0x7f829c159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.778+0000 7f829be00640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f829c0cebd0 0x7f829c159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.778+0000 7f829be00640 1 -- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f829c0ab680 msgr2=0x7f829c1512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.778+0000 7f829be00640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f829c0ab680 0x7f829c1512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.778+0000 7f829be00640 1 -- 172.21.15.62:0/1312337483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c0094d0 con 0x7f829c154d30 2024-03-23T11:27:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.778+0000 7f829be00640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c154d30 0x7f829c155b50 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f8294007e60 tx=0x7f829400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.779+0000 7f8293400640 1 -- 172.21.15.62:0/1312337483 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294014620 con 0x7f829c154d30 2024-03-23T11:27:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.779+0000 7f8293400640 1 -- 172.21.15.62:0/1312337483 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829400ee90 con 0x7f829c154d30 2024-03-23T11:27:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.779+0000 7f8293400640 1 -- 172.21.15.62:0/1312337483 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82940149e0 con 0x7f829c154d30 2024-03-23T11:27:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.779+0000 7f82a0a00640 1 -- 172.21.15.62:0/1312337483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829c0985d0 con 0x7f829c154d30 2024-03-23T11:27:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.779+0000 7f82a0a00640 1 -- 172.21.15.62:0/1312337483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c159040 con 0x7f829c154d30 2024-03-23T11:27:29.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.781+0000 7f8293400640 1 -- 172.21.15.62:0/1312337483 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8294014b40 con 0x7f829c154d30 2024-03-23T11:27:29.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.782+0000 7f8293400640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f827c06a7a0 0x7f827c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.782+0000 7f8293400640 1 -- 172.21.15.62:0/1312337483 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f8294089700 con 0x7f829c154d30 2024-03-23T11:27:29.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.782+0000 7f829b400640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f827c06a7a0 0x7f827c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.783+0000 7f82a4df9740 1 Processor -- start 2024-03-23T11:27:29.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.783+0000 7f82a4df9740 1 -- start start 2024-03-23T11:27:29.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.784+0000 7f829b400640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f827c06a7a0 0x7f827c06cc60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f828c036000 tx=0x7f828c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.784+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9aa050 0x5569fc9c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.784+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9c8240 0x5569fc982530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.784+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9acc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.784+0000 7f82a4df9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569fc8ecf40 con 0x5569fc9ac800 2024-03-23T11:27:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.784+0000 7f82a4df9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569fc8ea6d0 con 0x5569fc9aa050 2024-03-23T11:27:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.784+0000 7f82a4df9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5569fc8e5fc0 con 0x5569fc9c8240 2024-03-23T11:27:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.784+0000 7f829be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.784+0000 7f829be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9acc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43168/0 (socket says 172.21.15.62:43168) 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f829be00640 1 -- 172.21.15.62:0/1235538768 learned_addr learned my addr 172.21.15.62:0/1235538768 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f829b400640 1 --2- 172.21.15.62:0/1235538768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9aa050 0x5569fc9c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f829aa00640 1 --2- 172.21.15.62:0/1235538768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9c8240 0x5569fc982530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f829be00640 1 -- 172.21.15.62:0/1235538768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9c8240 msgr2=0x5569fc982530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f829be00640 1 --2- 172.21.15.62:0/1235538768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9c8240 0x5569fc982530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f829be00640 1 -- 172.21.15.62:0/1235538768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9aa050 msgr2=0x5569fc9c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f829be00640 1 --2- 172.21.15.62:0/1235538768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9aa050 0x5569fc9c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f829be00640 1 -- 172.21.15.62:0/1235538768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569fc8ad230 con 0x5569fc9ac800 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f829be00640 1 --2- 172.21.15.62:0/1235538768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9acc80 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f829400c040 tx=0x7f82940113a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75fa2e6d203c300 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f8287e00640 1 -- 172.21.15.62:0/1235538768 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294070170 con 0x5569fc9ac800 2024-03-23T11:27:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f8287e00640 1 -- 172.21.15.62:0/1235538768 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829405c5b0 con 0x5569fc9ac800 2024-03-23T11:27:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.785+0000 7f8287e00640 1 -- 172.21.15.62:0/1235538768 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294060b70 con 0x5569fc9ac800 2024-03-23T11:27:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.786+0000 7f82a4df9740 1 -- 172.21.15.62:0/1235538768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 msgr2=0x5569fc9acc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.786+0000 7f82a4df9740 1 --2- 172.21.15.62:0/1235538768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9acc80 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f829400c040 tx=0x7f82940113a0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.786+0000 7f82a4df9740 1 -- 172.21.15.62:0/1235538768 shutdown_connections 2024-03-23T11:27:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.786+0000 7f82a4df9740 1 --2- 172.21.15.62:0/1235538768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9acc80 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.786+0000 7f82a4df9740 1 --2- 172.21.15.62:0/1235538768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9c8240 0x5569fc982530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.786+0000 7f82a4df9740 1 --2- 172.21.15.62:0/1235538768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9aa050 0x5569fc9c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.786+0000 7f82a4df9740 1 -- 172.21.15.62:0/1235538768 >> 172.21.15.62:0/1235538768 conn(0x5569fc97ef10 msgr2=0x5569fc9c7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.786+0000 7f82a4df9740 1 -- 172.21.15.62:0/1235538768 shutdown_connections 2024-03-23T11:27:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.786+0000 7f82a4df9740 1 -- 172.21.15.62:0/1235538768 wait complete. 2024-03-23T11:27:29.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.787+0000 7f82a4df9740 1 -- start start 2024-03-23T11:27:29.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.787+0000 7f82a4df9740 1 Processor -- start 2024-03-23T11:27:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.788+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9ac800 0x5569fc9af940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.788+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fc9be8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.788+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9bedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.788+0000 7f82a4df9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569fc887320 con 0x5569fc9c9440 2024-03-23T11:27:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.788+0000 7f82a4df9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569fc886fc0 con 0x5569fc9c8240 2024-03-23T11:27:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.788+0000 7f82a4df9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5569fc64e0c0 con 0x5569fc9ac800 2024-03-23T11:27:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.788+0000 7f82a4df9740 10 client.0 _ll_register_callbacks cb 0x5569fc8aca10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:27:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9bedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9bedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43172/0 (socket says 172.21.15.62:43172) 2024-03-23T11:27:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829be00640 1 -- 172.21.15.62:0/2841209324 learned_addr learned my addr 172.21.15.62:0/2841209324 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829aa00640 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fc9be8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829b400640 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9ac800 0x5569fc9af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829be00640 1 -- 172.21.15.62:0/2841209324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9ac800 msgr2=0x5569fc9af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829be00640 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9ac800 0x5569fc9af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829be00640 1 -- 172.21.15.62:0/2841209324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 msgr2=0x5569fc9be8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829be00640 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fc9be8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829be00640 1 -- 172.21.15.62:0/2841209324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c159040 con 0x5569fc9c9440 2024-03-23T11:27:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829aa00640 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fc9be8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:27:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829b400640 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9ac800 0x5569fc9af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:27:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f829be00640 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9bedf0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f829400c040 tx=0x7f8294007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf97da6136a82877 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f8284c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:27:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f8284c00640 1 -- 172.21.15.62:0/2841209324 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294060b70 con 0x5569fc9c9440 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.789+0000 7f8284c00640 1 -- 172.21.15.62:0/2841209324 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f8294054e90 con 0x5569fc9c9440 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f82a4df9740 1 -- 172.21.15.62:0/2841209324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5569fc90fbb0 con 0x5569fc9c9440 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f8284c00640 1 -- 172.21.15.62:0/2841209324 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8294077250 con 0x5569fc9c9440 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f82a4df9740 10 client.7761 fetch_fsmap learned FSMap version 64 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f82a4df9740 1 -- 172.21.15.62:0/2841209324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x5569fc8e4640 con 0x5569fc9c9440 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f8284c00640 1 -- 172.21.15.62:0/2841209324 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829407c070 con 0x5569fc9c9440 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f8284c00640 1 -- 172.21.15.62:0/2841209324 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f829405c170 con 0x5569fc9c9440 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f82a4df9740 10 client.7761 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f82a4df9740 10 client.7761 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f82a4df9740 20 client.7761 populate_metadata read hostname 'smithi062' 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f82a4df9740 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5569fc99d6b0 0x5569fc99de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f82a4df9740 4 client.7761 mds_command: new command op to 7578 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f82a4df9740 1 -- 172.21.15.62:0/2841209324 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5569fc5da560 con 0x5569fc99d6b0 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.790+0000 7f829b400640 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5569fc99d6b0 0x5569fc99de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.791+0000 7f8284c00640 10 client.7761 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.791+0000 7f829b400640 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5569fc99d6b0 0x5569fc99de20 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddba9c0f95d0c08e server_cookie=4e66ef48d634dd1f in_seq=0 out_seq=0 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f8284c00640 1 -- 172.21.15.62:0/2841209324 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5569fc5da560 con 0x5569fc99d6b0 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f8284c00640 10 client.7761 handle_command_reply: tid=0 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 1 client.7761 shutdown 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 1 -- 172.21.15.62:0/2841209324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 msgr2=0x5569fc9bedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9bedf0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f829400c040 tx=0x7f8294007f50 comp rx=0 tx=0).stop 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 1 -- 172.21.15.62:0/2841209324 shutdown_connections 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5569fc99d6b0 0x5569fc99de20 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9bedf0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fc9be8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 1 --2- 172.21.15.62:0/2841209324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9ac800 0x5569fc9af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f8284c00640 0 client.7761 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:29.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 1 -- 172.21.15.62:0/2841209324 >> 172.21.15.62:0/2841209324 conn(0x5569fc9aa050 msgr2=0x5569fc97d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:29.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 1 -- 172.21.15.62:0/2841209324 shutdown_connections 2024-03-23T11:27:29.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 1 -- 172.21.15.62:0/2841209324 wait complete. 2024-03-23T11:27:29.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.792+0000 7f82a4df9740 20 client.7761 trim_cache size 0 max 16384 2024-03-23T11:27:29.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.808+0000 7f82a4df9740 1 Processor -- start 2024-03-23T11:27:29.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.808+0000 7f82a4df9740 1 -- start start 2024-03-23T11:27:29.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.809+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9ac800 0x5569fc9e5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.809+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fc9c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.809+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9c6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.809+0000 7f82a4df9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569fc9c5270 con 0x5569fc9c9440 2024-03-23T11:27:29.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.809+0000 7f82a4df9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569fc9afca0 con 0x5569fc9c8240 2024-03-23T11:27:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.809+0000 7f82a4df9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5569fc8deee0 con 0x5569fc9ac800 2024-03-23T11:27:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.809+0000 7f829be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.809+0000 7f829be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9c6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43188/0 (socket says 172.21.15.62:43188) 2024-03-23T11:27:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.809+0000 7f829be00640 1 -- 172.21.15.62:0/2425592339 learned_addr learned my addr 172.21.15.62:0/2425592339 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.810+0000 7f829be00640 1 -- 172.21.15.62:0/2425592339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9ac800 msgr2=0x5569fc9e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.810+0000 7f829be00640 1 --2- 172.21.15.62:0/2425592339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9ac800 0x5569fc9e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.810+0000 7f829be00640 1 -- 172.21.15.62:0/2425592339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 msgr2=0x5569fc9c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.810+0000 7f829aa00640 1 --2- 172.21.15.62:0/2425592339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fc9c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.810+0000 7f829be00640 1 --2- 172.21.15.62:0/2425592339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fc9c78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.810+0000 7f829be00640 1 -- 172.21.15.62:0/2425592339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569fc918f00 con 0x5569fc9c9440 2024-03-23T11:27:29.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.810+0000 7f829be00640 1 --2- 172.21.15.62:0/2425592339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9c6710 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f8294058710 tx=0x7f829406e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3275d1666036b230 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:29.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.810+0000 7f8290c00640 1 -- 172.21.15.62:0/2425592339 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294077250 con 0x5569fc9c9440 2024-03-23T11:27:29.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.810+0000 7f8290c00640 1 -- 172.21.15.62:0/2425592339 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829407c070 con 0x5569fc9c9440 2024-03-23T11:27:29.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.810+0000 7f8290c00640 1 -- 172.21.15.62:0/2425592339 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294054e90 con 0x5569fc9c9440 2024-03-23T11:27:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.811+0000 7f82a4df9740 1 -- 172.21.15.62:0/2425592339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 msgr2=0x5569fc9c6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.811+0000 7f82a4df9740 1 --2- 172.21.15.62:0/2425592339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9c6710 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f8294058710 tx=0x7f829406e040 comp rx=0 tx=0).stop 2024-03-23T11:27:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.811+0000 7f82a4df9740 1 -- 172.21.15.62:0/2425592339 shutdown_connections 2024-03-23T11:27:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.811+0000 7f82a4df9740 1 --2- 172.21.15.62:0/2425592339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9c9440 0x5569fc9c6710 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.811+0000 7f82a4df9740 1 --2- 172.21.15.62:0/2425592339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fc9c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.811+0000 7f82a4df9740 1 --2- 172.21.15.62:0/2425592339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9ac800 0x5569fc9e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.811+0000 7f82a4df9740 1 -- 172.21.15.62:0/2425592339 >> 172.21.15.62:0/2425592339 conn(0x5569fc9aa050 msgr2=0x5569fc9be8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.811+0000 7f82a4df9740 1 -- 172.21.15.62:0/2425592339 shutdown_connections 2024-03-23T11:27:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.811+0000 7f82a4df9740 1 -- 172.21.15.62:0/2425592339 wait complete. 2024-03-23T11:27:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.812+0000 7f82a4df9740 1 -- start start 2024-03-23T11:27:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.812+0000 7f82a4df9740 1 Processor -- start 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9aa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fca32430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f82a4df9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9c9440 0x5569fca32970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f82a4df9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569fc9ad840 con 0x5569fc9ac800 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f82a4df9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569fc9bf2b0 con 0x5569fc9c8240 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f82a4df9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5569fc9aac20 con 0x5569fc9c9440 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f82a4df9740 10 client.0 _ll_register_callbacks cb 0x7f8294054ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f829b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9aa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f829b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9aa6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43192/0 (socket says 172.21.15.62:43192) 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f829b400640 1 -- 172.21.15.62:0/1100028563 learned_addr learned my addr 172.21.15.62:0/1100028563 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f829be00640 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9c9440 0x5569fca32970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.813+0000 7f829b400640 1 -- 172.21.15.62:0/1100028563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9c9440 msgr2=0x5569fca32970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f829b400640 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9c9440 0x5569fca32970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f829aa00640 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fca32430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f829b400640 1 -- 172.21.15.62:0/1100028563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 msgr2=0x5569fca32430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f829b400640 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fca32430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f829b400640 1 -- 172.21.15.62:0/1100028563 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569fc8e4640 con 0x5569fc9ac800 2024-03-23T11:27:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f829aa00640 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fca32430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:27:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f829b400640 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9aa6e0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f828c000c00 tx=0x7f828c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7be84423ac35e74 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f8284c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:27:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f8284c00640 1 -- 172.21.15.62:0/1100028563 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c049040 con 0x5569fc9ac800 2024-03-23T11:27:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f8284c00640 1 -- 172.21.15.62:0/1100028563 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f828c03f070 con 0x5569fc9ac800 2024-03-23T11:27:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f82a4df9740 1 -- 172.21.15.62:0/1100028563 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f829405c5b0 con 0x5569fc9ac800 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f8284c00640 1 -- 172.21.15.62:0/1100028563 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f828c052110 con 0x5569fc9ac800 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f82a4df9740 10 client.7767 fetch_fsmap learned FSMap version 64 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.814+0000 7f82a4df9740 1 -- 172.21.15.62:0/1100028563 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x5569fc8864c0 con 0x5569fc9ac800 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.815+0000 7f8284c00640 1 -- 172.21.15.62:0/1100028563 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829405c5b0 con 0x5569fc9ac800 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.815+0000 7f8284c00640 1 -- 172.21.15.62:0/1100028563 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f828c052270 con 0x5569fc9ac800 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.815+0000 7f82a4df9740 10 client.7767 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.815+0000 7f82a4df9740 10 client.7767 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.815+0000 7f82a4df9740 20 client.7767 populate_metadata read hostname 'smithi062' 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.815+0000 7f82a4df9740 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5569fc9dd700 0x5569fc9ddae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.815+0000 7f82a4df9740 4 client.7767 mds_command: new command op to 7578 tid=0[{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.815+0000 7f82a4df9740 1 -- 172.21.15.62:0/1100028563 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) v1 -- 0x5569fc5eb860 con 0x5569fc9dd700 2024-03-23T11:27:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.815+0000 7f829aa00640 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5569fc9dd700 0x5569fc9ddae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.816+0000 7f829aa00640 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5569fc9dd700 0x5569fc9ddae0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f7515cb6201ce05 server_cookie=87e1ef1064a6e332 in_seq=0 out_seq=0 2024-03-23T11:27:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.816+0000 7f8284c00640 10 client.7767 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.816+0000 7f8284c00640 1 -- 172.21.15.62:0/1100028563 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5569fc5eb860 con 0x5569fc9dd700 2024-03-23T11:27:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.816+0000 7f8284c00640 10 client.7767 handle_command_reply: tid=0 2024-03-23T11:27:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.816+0000 7f82a4df9740 1 client.7767 shutdown 2024-03-23T11:27:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f82a4df9740 1 -- 172.21.15.62:0/1100028563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 msgr2=0x5569fc9aa6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f82a4df9740 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9aa6e0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f828c000c00 tx=0x7f828c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f82a4df9740 1 -- 172.21.15.62:0/1100028563 shutdown_connections 2024-03-23T11:27:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f82a4df9740 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x5569fc9dd700 0x5569fc9ddae0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f82a4df9740 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5569fc9c9440 0x5569fca32970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f82a4df9740 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569fc9c8240 0x5569fca32430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f82a4df9740 1 --2- 172.21.15.62:0/1100028563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5569fc9ac800 0x5569fc9aa6e0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f82a4df9740 1 -- 172.21.15.62:0/1100028563 >> 172.21.15.62:0/1100028563 conn(0x5569fc9aa050 msgr2=0x5569fc9c6570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:29.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f8284c00640 0 client.7767 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:29.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f82a4df9740 1 -- 172.21.15.62:0/1100028563 shutdown_connections 2024-03-23T11:27:29.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f82a4df9740 1 -- 172.21.15.62:0/1100028563 wait complete. 2024-03-23T11:27:29.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.817+0000 7f82a4df9740 20 client.7767 trim_cache size 0 max 16384 2024-03-23T11:27:29.814 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:27:29.817+0000 7fd996000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 pv17 ap=3 DIRTYPARENT f(v0 m2024-03-23T11:26:34.599368+0000 1=1+0)->f(v1 m2024-03-23T11:26:34.599368+0000) n(v0 rc2024-03-23T11:26:34.723466+0000 2=1+1)->n(v0 rc2024-03-23T11:26:34.723466+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55c63f9e2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-23T11:26:34.599368+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-23T11:26:34.723466+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-03-23T11:27:29.815 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:27:29.815 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-03-23T11:27:29.815 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "b9ff12a0-8143-4550-9d0e-d13720d8b3d7", 2024-03-23T11:27:29.815 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-03-23T11:27:29.815 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:27:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 -- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f827c06a7a0 msgr2=0x7f827c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f827c06a7a0 0x7f827c06cc60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f828c036000 tx=0x7f828c03c040 comp rx=0 tx=0).stop 2024-03-23T11:27:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 -- 172.21.15.62:0/1312337483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c154d30 msgr2=0x7f829c155b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c154d30 0x7f829c155b50 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f8294007e60 tx=0x7f829400e4b0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 -- 172.21.15.62:0/1312337483 shutdown_connections 2024-03-23T11:27:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f827c06a7a0 0x7f827c06cc60 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f829c154d30 0x7f829c155b50 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f829c0cebd0 0x7f829c159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 --2- 172.21.15.62:0/1312337483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f829c0ab680 0x7f829c1512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:29.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 -- 172.21.15.62:0/1312337483 >> 172.21.15.62:0/1312337483 conn(0x7f829c0b0280 msgr2=0x7f829c0b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:29.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 -- 172.21.15.62:0/1312337483 shutdown_connections 2024-03-23T11:27:29.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:29.818+0000 7f82a0a00640 1 -- 172.21.15.62:0/1312337483 wait complete. 2024-03-23T11:27:29.830 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-23T11:27:30.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.050+0000 7fcbdb200640 1 Processor -- start 2024-03-23T11:27:30.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.050+0000 7fcbdb200640 1 -- start start 2024-03-23T11:27:30.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.051+0000 7fcbdb200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcbdc1431f0 0x7fcbdc0b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.051+0000 7fcbdb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbdc070090 0x7fcbdc0704d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.051+0000 7fcbdb200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc0b20d0 con 0x7fcbdc0cc2a0 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.051+0000 7fcbdb200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbdc070a10 con 0x7fcbdc1431f0 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.051+0000 7fcbdb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc1435d0 con 0x7fcbdc070090 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.052+0000 7fcbd9e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcbdc1431f0 0x7fcbdc0b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.052+0000 7fcbd9e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcbdc1431f0 0x7fcbdc0b2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43196/0 (socket says 172.21.15.62:43196) 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.052+0000 7fcbd9e00640 1 -- 172.21.15.62:0/2451462891 learned_addr learned my addr 172.21.15.62:0/2451462891 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.052+0000 7fcbda800640 1 --1- 172.21.15.62:0/2451462891 >> v1:172.21.15.62:6789/0 conn(0x7fcbdc0cc2a0 0x7fcbdc0cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:46210/0 (socket says 172.21.15.62:46210) 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.052+0000 7fcbd1400640 1 --2- 172.21.15.62:0/2451462891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbdc070090 0x7fcbdc0704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.052+0000 7fcbd9e00640 1 -- 172.21.15.62:0/2451462891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbdc070090 msgr2=0x7fcbdc0704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.052+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/2451462891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbdc070090 0x7fcbdc0704d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.052+0000 7fcbd9e00640 1 -- 172.21.15.62:0/2451462891 >> v1:172.21.15.62:6789/0 conn(0x7fcbdc0cc2a0 legacy=0x7fcbdc0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:30.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.052+0000 7fcbd9e00640 1 -- 172.21.15.62:0/2451462891 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc0b14c0 con 0x7fcbdc1431f0 2024-03-23T11:27:30.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.052+0000 7fcbd9400640 1 -- 172.21.15.62:0/2451462891 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352203412 0 0) 0x7fcbdc0b20d0 con 0x7fcbdc0cc2a0 2024-03-23T11:27:30.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.054+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/2451462891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbdc1431f0 0x7fcbdc0b2560 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009870 tx=0x7fcbcc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df3354c7c95ead server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:30.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.054+0000 7fcbd9400640 1 -- 172.21.15.62:0/2451462891 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc031640 con 0x7fcbdc1431f0 2024-03-23T11:27:30.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.054+0000 7fcbd9400640 1 -- 172.21.15.62:0/2451462891 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc0317a0 con 0x7fcbdc1431f0 2024-03-23T11:27:30.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.054+0000 7fcbd9400640 1 -- 172.21.15.62:0/2451462891 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc031b30 con 0x7fcbdc1431f0 2024-03-23T11:27:30.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.054+0000 7fcbdb200640 1 -- 172.21.15.62:0/2451462891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbdc1431f0 msgr2=0x7fcbdc0b2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:30.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.054+0000 7fcbdb200640 1 --2- 172.21.15.62:0/2451462891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbdc1431f0 0x7fcbdc0b2560 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009870 tx=0x7fcbcc02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.055+0000 7fcbdb200640 1 -- 172.21.15.62:0/2451462891 shutdown_connections 2024-03-23T11:27:30.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.055+0000 7fcbdb200640 1 --2- 172.21.15.62:0/2451462891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbdc070090 0x7fcbdc0704d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.055+0000 7fcbdb200640 1 --2- 172.21.15.62:0/2451462891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbdc1431f0 0x7fcbdc0b2560 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.055+0000 7fcbdb200640 1 -- 172.21.15.62:0/2451462891 >> 172.21.15.62:0/2451462891 conn(0x7fcbdc0b0090 msgr2=0x7fcbdc0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:30.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.055+0000 7fcbdb200640 1 -- 172.21.15.62:0/2451462891 shutdown_connections 2024-03-23T11:27:30.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.055+0000 7fcbdb200640 1 -- 172.21.15.62:0/2451462891 wait complete. 2024-03-23T11:27:30.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.057+0000 7fcbdb200640 1 Processor -- start 2024-03-23T11:27:30.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.057+0000 7fcbdb200640 1 -- start start 2024-03-23T11:27:30.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbdb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbdc070090 0x7fcbdc159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbdb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbdc0cc2a0 0x7fcbdc0a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbdb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbdc1431f0 0x7fcbdc12d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbdb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc159690 con 0x7fcbdc1431f0 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbdb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc0a9940 con 0x7fcbdc070090 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbdb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc12fd40 con 0x7fcbdc0cc2a0 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbda800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbdc1431f0 0x7fcbdc12d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbda800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbdc1431f0 0x7fcbdc12d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43208/0 (socket says 172.21.15.62:43208) 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbda800640 1 -- 172.21.15.62:0/577979751 learned_addr learned my addr 172.21.15.62:0/577979751 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbd1400640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbdc0cc2a0 0x7fcbdc0a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbdc070090 0x7fcbdc159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbda800640 1 -- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbdc0cc2a0 msgr2=0x7fcbdc0a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbda800640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbdc0cc2a0 0x7fcbdc0a9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbda800640 1 -- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbdc070090 msgr2=0x7fcbdc159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbda800640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbdc070090 0x7fcbdc159150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.058+0000 7fcbda800640 1 -- 172.21.15.62:0/577979751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc0094d0 con 0x7fcbdc1431f0 2024-03-23T11:27:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.059+0000 7fcbda800640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbdc1431f0 0x7fcbdc12d7d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400c7e0 tx=0x7fcbd400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.059+0000 7fcbd2a00640 1 -- 172.21.15.62:0/577979751 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400a640 con 0x7fcbdc1431f0 2024-03-23T11:27:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.059+0000 7fcbd2a00640 1 -- 172.21.15.62:0/577979751 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd400a7c0 con 0x7fcbdc1431f0 2024-03-23T11:27:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.059+0000 7fcbdb200640 1 -- 172.21.15.62:0/577979751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbdc130020 con 0x7fcbdc1431f0 2024-03-23T11:27:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.059+0000 7fcbd2a00640 1 -- 172.21.15.62:0/577979751 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400ab70 con 0x7fcbdc1431f0 2024-03-23T11:27:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.059+0000 7fcbdb200640 1 -- 172.21.15.62:0/577979751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbdc1305e0 con 0x7fcbdc1431f0 2024-03-23T11:27:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.062+0000 7fcbd2a00640 1 -- 172.21.15.62:0/577979751 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fcbd400acd0 con 0x7fcbdc1431f0 2024-03-23T11:27:30.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.063+0000 7fcbd2a00640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbb806a7d0 0x7fcbb806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.063+0000 7fcbd2a00640 1 -- 172.21.15.62:0/577979751 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fcbd4088650 con 0x7fcbdc1431f0 2024-03-23T11:27:30.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.064+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbb806a7d0 0x7fcbb806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.064+0000 7fcbe3c83740 1 Processor -- start 2024-03-23T11:27:30.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.064+0000 7fcbe3c83740 1 -- start start 2024-03-23T11:27:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f8070b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f806e3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f806e3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f8069acb0 con 0x561f8070b870 2024-03-23T11:27:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f80649500 con 0x561f807032e0 2024-03-23T11:27:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561f80646c90 con 0x561f80702f00 2024-03-23T11:27:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbb806a7d0 0x7fcbb806cc90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc036000 tx=0x7fcbcc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbd1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f806e3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbd1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f806e3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43216/0 (socket says 172.21.15.62:43216) 2024-03-23T11:27:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbd1400640 1 -- 172.21.15.62:0/3439619915 learned_addr learned my addr 172.21.15.62:0/3439619915 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:30.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/3439619915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f8070b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.065+0000 7fcbd1400640 1 -- 172.21.15.62:0/3439619915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 msgr2=0x561f8070b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:30.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.066+0000 7fcbda800640 1 --2- 172.21.15.62:0/3439619915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f806e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.066+0000 7fcbd1400640 1 --2- 172.21.15.62:0/3439619915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f8070b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.066+0000 7fcbd1400640 1 -- 172.21.15.62:0/3439619915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 msgr2=0x561f806e3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:30.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.066+0000 7fcbd1400640 1 --2- 172.21.15.62:0/3439619915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f806e3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.066+0000 7fcbd1400640 1 -- 172.21.15.62:0/3439619915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f805f4360 con 0x561f8070b870 2024-03-23T11:27:30.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.066+0000 7fcbd1400640 1 --2- 172.21.15.62:0/3439619915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f806e3030 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4006fd0 tx=0x7fcbc4009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce4f3f843077118 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:30.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.066+0000 7fcbcaa00640 1 -- 172.21.15.62:0/3439619915 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc400c070 con 0x561f8070b870 2024-03-23T11:27:30.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.066+0000 7fcbcaa00640 1 -- 172.21.15.62:0/3439619915 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc400e950 con 0x561f8070b870 2024-03-23T11:27:30.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.066+0000 7fcbcaa00640 1 -- 172.21.15.62:0/3439619915 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc400ed00 con 0x561f8070b870 2024-03-23T11:27:30.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.067+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3439619915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 msgr2=0x561f806e3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:30.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.067+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3439619915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f806e3030 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4006fd0 tx=0x7fcbc4009d70 comp rx=0 tx=0).stop 2024-03-23T11:27:30.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.067+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3439619915 shutdown_connections 2024-03-23T11:27:30.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.067+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3439619915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f806e3030 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.067+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3439619915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f806e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.067+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3439619915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f8070b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.067+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3439619915 >> 172.21.15.62:0/3439619915 conn(0x561f806a6b40 msgr2=0x561f80728d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:30.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.067+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3439619915 shutdown_connections 2024-03-23T11:27:30.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.067+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3439619915 wait complete. 2024-03-23T11:27:30.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.068+0000 7fcbe3c83740 1 -- start start 2024-03-23T11:27:30.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.068+0000 7fcbe3c83740 1 Processor -- start 2024-03-23T11:27:30.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.069+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f807286b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.069+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f8072ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.069+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f8072b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.069+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f80642580 con 0x561f8070b870 2024-03-23T11:27:30.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.069+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f805e3a50 con 0x561f807032e0 2024-03-23T11:27:30.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.069+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561f805e36f0 con 0x561f80702f00 2024-03-23T11:27:30.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.069+0000 7fcbe3c83740 10 client.0 _ll_register_callbacks cb 0x561f80632ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:27:30.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.069+0000 7fcbda800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f8072b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.069+0000 7fcbda800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f8072b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43232/0 (socket says 172.21.15.62:43232) 2024-03-23T11:27:30.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.069+0000 7fcbda800640 1 -- 172.21.15.62:0/1870290237 learned_addr learned my addr 172.21.15.62:0/1870290237 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:30.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbd1400640 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f8072ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f807286b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbda800640 1 -- 172.21.15.62:0/1870290237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 msgr2=0x561f807286b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbda800640 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f807286b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbda800640 1 -- 172.21.15.62:0/1870290237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 msgr2=0x561f8072ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbda800640 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f8072ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbda800640 1 -- 172.21.15.62:0/1870290237 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc4009540 con 0x561f8070b870 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f807286b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbda800640 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f8072b380 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400a430 tx=0x7fcbd40576d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5bdb3ee1beb9468 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbbf400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbbf400640 1 -- 172.21.15.62:0/1870290237 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd405f630 con 0x561f8070b870 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbbf400640 1 -- 172.21.15.62:0/1870290237 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fcbd405f790 con 0x561f8070b870 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbe3c83740 1 -- 172.21.15.62:0/1870290237 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561f8066cfe0 con 0x561f8070b870 2024-03-23T11:27:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.070+0000 7fcbbf400640 1 -- 172.21.15.62:0/1870290237 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbd40534c0 con 0x561f8070b870 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.071+0000 7fcbe3c83740 10 client.7782 fetch_fsmap learned FSMap version 64 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.071+0000 7fcbe3c83740 1 -- 172.21.15.62:0/1870290237 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x561f80640c00 con 0x561f8070b870 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.071+0000 7fcbbf400640 1 -- 172.21.15.62:0/1870290237 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561f8066cfe0 con 0x561f8070b870 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.071+0000 7fcbbf400640 1 -- 172.21.15.62:0/1870290237 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fcbd4053620 con 0x561f8070b870 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.071+0000 7fcbe3c83740 10 client.7782 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.071+0000 7fcbe3c83740 10 client.7782 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.071+0000 7fcbe3c83740 20 client.7782 populate_metadata read hostname 'smithi062' 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.071+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x561f80728bf0 0x561f80726a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.071+0000 7fcbe3c83740 4 client.7782 mds_command: new command op to 7578 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.071+0000 7fcbe3c83740 1 -- 172.21.15.62:0/1870290237 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561f80635240 con 0x561f80728bf0 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.071+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x561f80728bf0 0x561f80726a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.072+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x561f80728bf0 0x561f80726a90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9426c09ea7c72cd3 server_cookie=91d87c53f0358f2 in_seq=0 out_seq=0 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.072+0000 7fcbbf400640 10 client.7782 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:30.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbbf400640 1 -- 172.21.15.62:0/1870290237 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x561f80635240 con 0x561f80728bf0 2024-03-23T11:27:30.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbbf400640 10 client.7782 handle_command_reply: tid=0 2024-03-23T11:27:30.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 1 client.7782 shutdown 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 1 -- 172.21.15.62:0/1870290237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 msgr2=0x561f8072b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f8072b380 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400a430 tx=0x7fcbd40576d0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 1 -- 172.21.15.62:0/1870290237 shutdown_connections 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x561f80728bf0 0x561f80726a90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f8072b380 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f8072ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/1870290237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f807286b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbbf400640 0 client.7782 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 1 -- 172.21.15.62:0/1870290237 >> 172.21.15.62:0/1870290237 conn(0x561f806a6b40 msgr2=0x561f807025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 1 -- 172.21.15.62:0/1870290237 shutdown_connections 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 1 -- 172.21.15.62:0/1870290237 wait complete. 2024-03-23T11:27:30.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.073+0000 7fcbe3c83740 20 client.7782 trim_cache size 0 max 16384 2024-03-23T11:27:30.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.089+0000 7fcbe3c83740 1 Processor -- start 2024-03-23T11:27:30.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.089+0000 7fcbe3c83740 1 -- start start 2024-03-23T11:27:30.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.090+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f8070ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.090+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f80725da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.090+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f80737e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.090+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f8033ff90 con 0x561f8070b870 2024-03-23T11:27:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.090+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f806e4ad0 con 0x561f807032e0 2024-03-23T11:27:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.090+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561f806dff70 con 0x561f80702f00 2024-03-23T11:27:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.090+0000 7fcbda800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f80737e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.090+0000 7fcbda800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f80737e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43248/0 (socket says 172.21.15.62:43248) 2024-03-23T11:27:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.090+0000 7fcbda800640 1 -- 172.21.15.62:0/3324595766 learned_addr learned my addr 172.21.15.62:0/3324595766 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:30.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.091+0000 7fcbda800640 1 -- 172.21.15.62:0/3324595766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 msgr2=0x561f8070ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:30.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.091+0000 7fcbda800640 1 --2- 172.21.15.62:0/3324595766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f8070ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.091+0000 7fcbda800640 1 -- 172.21.15.62:0/3324595766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 msgr2=0x561f80725da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:30.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.091+0000 7fcbd1400640 1 --2- 172.21.15.62:0/3324595766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f80725da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.091+0000 7fcbda800640 1 --2- 172.21.15.62:0/3324595766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f80725da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.091+0000 7fcbda800640 1 -- 172.21.15.62:0/3324595766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f805e2c60 con 0x561f8070b870 2024-03-23T11:27:30.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.091+0000 7fcbda800640 1 --2- 172.21.15.62:0/3324595766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f80737e40 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fcbd405b6c0 tx=0x7fcbd4088070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c66b4960419b69d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:30.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.091+0000 7fcbcb400640 1 -- 172.21.15.62:0/3324595766 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4001890 con 0x561f8070b870 2024-03-23T11:27:30.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.091+0000 7fcbcb400640 1 -- 172.21.15.62:0/3324595766 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd4057030 con 0x561f8070b870 2024-03-23T11:27:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.091+0000 7fcbcb400640 1 -- 172.21.15.62:0/3324595766 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd405f630 con 0x561f8070b870 2024-03-23T11:27:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.092+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3324595766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 msgr2=0x561f80737e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.092+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3324595766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f80737e40 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fcbd405b6c0 tx=0x7fcbd4088070 comp rx=0 tx=0).stop 2024-03-23T11:27:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.092+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3324595766 shutdown_connections 2024-03-23T11:27:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.092+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3324595766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f8070b870 0x561f80737e40 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.092+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3324595766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f80725da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.092+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3324595766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f80702f00 0x561f8070ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.092+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3324595766 >> 172.21.15.62:0/3324595766 conn(0x561f806a6b40 msgr2=0x561f80727090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.092+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3324595766 shutdown_connections 2024-03-23T11:27:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.092+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3324595766 wait complete. 2024-03-23T11:27:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.093+0000 7fcbe3c83740 1 -- start start 2024-03-23T11:27:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.093+0000 7fcbe3c83740 1 Processor -- start 2024-03-23T11:27:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.094+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f80702f00 0x561f8071e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.094+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f807866d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.094+0000 7fcbe3c83740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f8070b870 0x561f80700fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.094+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f8070cf80 con 0x561f80702f00 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.094+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f8070d4b0 con 0x561f807032e0 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.094+0000 7fcbe3c83740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561f80786c10 con 0x561f8070b870 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.094+0000 7fcbe3c83740 10 client.0 _ll_register_callbacks cb 0x7fcbc40091a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.094+0000 7fcbd9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f80702f00 0x561f8071e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.094+0000 7fcbd9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f80702f00 0x561f8071e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43262/0 (socket says 172.21.15.62:43262) 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.094+0000 7fcbd9e00640 1 -- 172.21.15.62:0/3061157738 learned_addr learned my addr 172.21.15.62:0/3061157738 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbd1400640 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f807866d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbda800640 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f8070b870 0x561f80700fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbd9e00640 1 -- 172.21.15.62:0/3061157738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f8070b870 msgr2=0x561f80700fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f8070b870 0x561f80700fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbd9e00640 1 -- 172.21.15.62:0/3061157738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 msgr2=0x561f807866d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f807866d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbd9e00640 1 -- 172.21.15.62:0/3061157738 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f80640c00 con 0x561f80702f00 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbd1400640 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f807866d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbd9e00640 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f80702f00 0x561f8071e850 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0090b0 tx=0x7fcbcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb5f50b28ff7ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbbf400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbbf400640 1 -- 172.21.15.62:0/3061157738 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc049040 con 0x561f80702f00 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbbf400640 1 -- 172.21.15.62:0/3061157738 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fcbcc050c60 con 0x561f80702f00 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.095+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3061157738 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbc400e950 con 0x561f80702f00 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbbf400640 1 -- 172.21.15.62:0/3061157738 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbcc050e20 con 0x561f80702f00 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbe3c83740 10 client.7788 fetch_fsmap learned FSMap version 64 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3061157738 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x561f80608170 con 0x561f80702f00 2024-03-23T11:27:30.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbbf400640 1 -- 172.21.15.62:0/3061157738 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc400e950 con 0x561f80702f00 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbbf400640 1 -- 172.21.15.62:0/3061157738 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fcbcc059450 con 0x561f80702f00 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbe3c83740 10 client.7788 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbe3c83740 10 client.7788 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbe3c83740 20 client.7788 populate_metadata read hostname 'smithi062' 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x561f80701760 0x561f806e1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbe3c83740 4 client.7788 mds_command: new command op to 7578 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3061157738 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561f803364e0 con 0x561f80701760 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.096+0000 7fcbd1400640 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x561f80701760 0x561f806e1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.097+0000 7fcbd1400640 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x561f80701760 0x561f806e1a80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=175c0907d93a0ded server_cookie=b65282bf6cc49210 in_seq=0 out_seq=0 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.097+0000 7fcbbf400640 10 client.7788 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.097+0000 7fcbbf400640 1 -- 172.21.15.62:0/3061157738 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+263 (crc 0 0 0) 0x561f803364e0 con 0x561f80701760 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.097+0000 7fcbbf400640 10 client.7788 handle_command_reply: tid=0 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.097+0000 7fcbe3c83740 1 client.7788 shutdown 2024-03-23T11:27:30.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.097+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3061157738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f80702f00 msgr2=0x561f8071e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:30.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.097+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f80702f00 0x561f8071e850 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0090b0 tx=0x7fcbcc0046c0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3061157738 shutdown_connections 2024-03-23T11:27:30.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x561f80701760 0x561f806e1a80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561f8070b870 0x561f80700fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f807032e0 0x561f807866d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbe3c83740 1 --2- 172.21.15.62:0/3061157738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f80702f00 0x561f8071e850 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3061157738 >> 172.21.15.62:0/3061157738 conn(0x561f806a6b40 msgr2=0x561f80737a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:30.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbbf400640 0 client.7788 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:27:30.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3061157738 shutdown_connections 2024-03-23T11:27:30.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbe3c83740 1 -- 172.21.15.62:0/3061157738 wait complete. 2024-03-23T11:27:30.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbe3c83740 20 client.7788 trim_cache size 0 max 16384 2024-03-23T11:27:30.095 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:27:30.095 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T11:27:30.095 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-03-23T11:27:30.095 INFO:teuthology.orchestra.run.smithi062.stdout: "b9ff12a0-8143-4550-9d0e-d13720d8b3d7": { 2024-03-23T11:27:30.095 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/dir", 2024-03-23T11:27:30.095 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "b9ff12a0-8143-4550-9d0e-d13720d8b3d7", 2024-03-23T11:27:30.095 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,repair" 2024-03-23T11:27:30.095 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:27:30.095 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:27:30.096 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:27:30.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbdb200640 1 -- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbb806a7d0 msgr2=0x7fcbb806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:30.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbdb200640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbb806a7d0 0x7fcbb806cc90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc036000 tx=0x7fcbcc03c040 comp rx=0 tx=0).stop 2024-03-23T11:27:30.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbdb200640 1 -- 172.21.15.62:0/577979751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbdc1431f0 msgr2=0x7fcbdc12d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:27:30.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.098+0000 7fcbdb200640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbdc1431f0 0x7fcbdc12d7d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400c7e0 tx=0x7fcbd400cca0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.099+0000 7fcbdb200640 1 -- 172.21.15.62:0/577979751 shutdown_connections 2024-03-23T11:27:30.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.099+0000 7fcbdb200640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbb806a7d0 0x7fcbb806cc90 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.099+0000 7fcbdb200640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbdc1431f0 0x7fcbdc12d7d0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.099+0000 7fcbdb200640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcbdc0cc2a0 0x7fcbdc0a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.099+0000 7fcbdb200640 1 --2- 172.21.15.62:0/577979751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbdc070090 0x7fcbdc159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:27:30.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.099+0000 7fcbdb200640 1 -- 172.21.15.62:0/577979751 >> 172.21.15.62:0/577979751 conn(0x7fcbdc0b0090 msgr2=0x7fcbdc0af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:27:30.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.099+0000 7fcbdb200640 1 -- 172.21.15.62:0/577979751 shutdown_connections 2024-03-23T11:27:30.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:27:30.099+0000 7fcbdb200640 1 -- 172.21.15.62:0/577979751 wait complete. 2024-03-23T11:27:30.108 INFO:tasks.cephfs.filesystem:scrub status for tag:b9ff12a0-8143-4550-9d0e-d13720d8b3d7 - {'path': '/dir', 'tag': 'b9ff12a0-8143-4550-9d0e-d13720d8b3d7', 'options': 'recursive,repair'} 2024-03-23T11:27:46.888 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:27:46.896 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:28:00.109 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-23T11:28:00.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.350+0000 7fe1f9a00640 1 Processor -- start 2024-03-23T11:28:00.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.350+0000 7fe1f9a00640 1 -- start start 2024-03-23T11:28:00.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.352+0000 7fe1f9a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1f41571e0 0x7fe1f40ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.352+0000 7fe1f9a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe1f40ade70 0x7fe1f406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.352+0000 7fe1f9a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe1f41613a0 0x7fe1f415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.352+0000 7fe1f9a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f406cf40 con 0x7fe1f41613a0 2024-03-23T11:28:00.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.352+0000 7fe1f9a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f41575c0 con 0x7fe1f40ade70 2024-03-23T11:28:00.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.352+0000 7fe1f9a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f406c3a0 con 0x7fe1f41571e0 2024-03-23T11:28:00.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.352+0000 7fe1f3400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe1f41613a0 0x7fe1f415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.353+0000 7fe1f3400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe1f41613a0 0x7fe1f415e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53932/0 (socket says 172.21.15.62:53932) 2024-03-23T11:28:00.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.353+0000 7fe1f3400640 1 -- 172.21.15.62:0/2337306869 learned_addr learned my addr 172.21.15.62:0/2337306869 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.353+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/2337306869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1f41571e0 0x7fe1f40ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.353+0000 7fe1ebe00640 1 --2- 172.21.15.62:0/2337306869 >> v2:172.21.15.120:3300/0 conn(0x7fe1f40ade70 0x7fe1f406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.353+0000 7fe1f3400640 1 -- 172.21.15.62:0/2337306869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1f41571e0 msgr2=0x7fe1f40ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.353+0000 7fe1f3400640 1 --2- 172.21.15.62:0/2337306869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1f41571e0 0x7fe1f40ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.353+0000 7fe1f3400640 1 -- 172.21.15.62:0/2337306869 >> v2:172.21.15.120:3300/0 conn(0x7fe1f40ade70 msgr2=0x7fe1f406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.353+0000 7fe1f3400640 1 --2- 172.21.15.62:0/2337306869 >> v2:172.21.15.120:3300/0 conn(0x7fe1f40ade70 0x7fe1f406d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.353+0000 7fe1f3400640 1 -- 172.21.15.62:0/2337306869 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f406c610 con 0x7fe1f41613a0 2024-03-23T11:28:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.353+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/2337306869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1f41571e0 0x7fe1f40ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.353+0000 7fe1ebe00640 1 --2- 172.21.15.62:0/2337306869 >> v2:172.21.15.120:3300/0 conn(0x7fe1f40ade70 0x7fe1f406d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:00.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.354+0000 7fe1f3400640 1 --2- 172.21.15.62:0/2337306869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1f41613a0 0x7fe1f415e280 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4071bf0 tx=0x7fe1ec02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce06976aed6c0ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:00.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.354+0000 7fe1f2000640 1 -- 172.21.15.62:0/2337306869 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec02c9e0 con 0x7fe1f41613a0 2024-03-23T11:28:00.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.355+0000 7fe1f2000640 1 -- 172.21.15.62:0/2337306869 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec02cb40 con 0x7fe1f41613a0 2024-03-23T11:28:00.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.355+0000 7fe1f2000640 1 -- 172.21.15.62:0/2337306869 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec0343f0 con 0x7fe1f41613a0 2024-03-23T11:28:00.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.355+0000 7fe1f9a00640 1 -- 172.21.15.62:0/2337306869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1f41613a0 msgr2=0x7fe1f415e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.355+0000 7fe1f9a00640 1 --2- 172.21.15.62:0/2337306869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1f41613a0 0x7fe1f415e280 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4071bf0 tx=0x7fe1ec02be40 comp rx=0 tx=0).stop 2024-03-23T11:28:00.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.356+0000 7fe1f9a00640 1 -- 172.21.15.62:0/2337306869 shutdown_connections 2024-03-23T11:28:00.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.356+0000 7fe1f9a00640 1 --2- 172.21.15.62:0/2337306869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1f41613a0 0x7fe1f415e280 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.356+0000 7fe1f9a00640 1 --2- 172.21.15.62:0/2337306869 >> v2:172.21.15.120:3300/0 conn(0x7fe1f40ade70 0x7fe1f406d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.356+0000 7fe1f9a00640 1 --2- 172.21.15.62:0/2337306869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1f41571e0 0x7fe1f40ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.356+0000 7fe1f9a00640 1 -- 172.21.15.62:0/2337306869 >> 172.21.15.62:0/2337306869 conn(0x7fe1f40b02e0 msgr2=0x7fe1f40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:00.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.356+0000 7fe1f9a00640 1 -- 172.21.15.62:0/2337306869 shutdown_connections 2024-03-23T11:28:00.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.356+0000 7fe1f9a00640 1 -- 172.21.15.62:0/2337306869 wait complete. 2024-03-23T11:28:00.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.358+0000 7fe1f9a00640 1 Processor -- start 2024-03-23T11:28:00.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.358+0000 7fe1f9a00640 1 -- start start 2024-03-23T11:28:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.359+0000 7fe1f9a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1f40ade70 0x7fe1f4150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.359+0000 7fe1f9a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1f41571e0 0x7fe1f4144ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.359+0000 7fe1f9a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1f41613a0 0x7fe1f4145400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.359+0000 7fe1f9a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f4151bf0 con 0x7fe1f41571e0 2024-03-23T11:28:00.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.359+0000 7fe1f9a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f4145940 con 0x7fe1f41613a0 2024-03-23T11:28:00.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.359+0000 7fe1f9a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f414e5e0 con 0x7fe1f40ade70 2024-03-23T11:28:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.360+0000 7fe1ebe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1f41571e0 0x7fe1f4144ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.360+0000 7fe1ebe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1f41571e0 0x7fe1f4144ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53944/0 (socket says 172.21.15.62:53944) 2024-03-23T11:28:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.360+0000 7fe1ebe00640 1 -- 172.21.15.62:0/3977668525 learned_addr learned my addr 172.21.15.62:0/3977668525 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.360+0000 7fe1f3400640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1f41613a0 0x7fe1f4145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.360+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1f40ade70 0x7fe1f4150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.360+0000 7fe1ebe00640 1 -- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1f40ade70 msgr2=0x7fe1f4150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.360+0000 7fe1ebe00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1f40ade70 0x7fe1f4150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.360+0000 7fe1ebe00640 1 -- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1f41613a0 msgr2=0x7fe1f4145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.360+0000 7fe1ebe00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1f41613a0 0x7fe1f4145400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.360+0000 7fe1ebe00640 1 -- 172.21.15.62:0/3977668525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec009b30 con 0x7fe1f41571e0 2024-03-23T11:28:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.360+0000 7fe1f3400640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1f41613a0 0x7fe1f4145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:28:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.361+0000 7fe1ebe00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1f41571e0 0x7fe1f4144ec0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00c7f0 tx=0x7fe1dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.361+0000 7fe1eaa00640 1 -- 172.21.15.62:0/3977668525 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc007820 con 0x7fe1f41571e0 2024-03-23T11:28:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.361+0000 7fe1eaa00640 1 -- 172.21.15.62:0/3977668525 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1dc007980 con 0x7fe1f41571e0 2024-03-23T11:28:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.361+0000 7fe1eaa00640 1 -- 172.21.15.62:0/3977668525 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc007d30 con 0x7fe1f41571e0 2024-03-23T11:28:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.361+0000 7fe1f9a00640 1 -- 172.21.15.62:0/3977668525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f40cbda0 con 0x7fe1f41571e0 2024-03-23T11:28:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.361+0000 7fe1f9a00640 1 -- 172.21.15.62:0/3977668525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f40cc360 con 0x7fe1f41571e0 2024-03-23T11:28:00.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.364+0000 7fe1eaa00640 1 -- 172.21.15.62:0/3977668525 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe1dc0113f0 con 0x7fe1f41571e0 2024-03-23T11:28:00.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.365+0000 7fe1eaa00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe1d006a7a0 0x7fe1d006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.365+0000 7fe1eaa00640 1 -- 172.21.15.62:0/3977668525 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fe1dc0890e0 con 0x7fe1f41571e0 2024-03-23T11:28:00.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.366+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe1d006a7a0 0x7fe1d006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.366+0000 7fe1fc5d3740 1 Processor -- start 2024-03-23T11:28:00.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.366+0000 7fe1fc5d3740 1 -- start start 2024-03-23T11:28:00.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.367+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d07b910 0x55ce6d083b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.367+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d084280 0x55ce6d05ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.367+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07bcf0 0x55ce6d05bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.367+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6d017cb0 con 0x55ce6d07b910 2024-03-23T11:28:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.367+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6cfc6500 con 0x55ce6d07bcf0 2024-03-23T11:28:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.367+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce6cfc3c90 con 0x55ce6d084280 2024-03-23T11:28:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.367+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe1d006a7a0 0x7fe1d006cc60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400a950 tx=0x7fe1e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.367+0000 7fe1f2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d07b910 0x55ce6d083b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.367+0000 7fe1f2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d07b910 0x55ce6d083b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53948/0 (socket says 172.21.15.62:53948) 2024-03-23T11:28:00.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.367+0000 7fe1f2a00640 1 -- 172.21.15.62:0/3246166253 learned_addr learned my addr 172.21.15.62:0/3246166253 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:00.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.368+0000 7fe1ebe00640 1 --2- 172.21.15.62:0/3246166253 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d084280 0x55ce6d05ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.368+0000 7fe1f3400640 1 --2- 172.21.15.62:0/3246166253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07bcf0 0x55ce6d05bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.368+0000 7fe1f2a00640 1 -- 172.21.15.62:0/3246166253 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d084280 msgr2=0x55ce6d05ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.368+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/3246166253 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d084280 0x55ce6d05ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.368+0000 7fe1f2a00640 1 -- 172.21.15.62:0/3246166253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07bcf0 msgr2=0x55ce6d05bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.368+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/3246166253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07bcf0 0x55ce6d05bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.368+0000 7fe1f2a00640 1 -- 172.21.15.62:0/3246166253 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce6cf71360 con 0x55ce6d07b910 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.368+0000 7fe1f3400640 1 --2- 172.21.15.62:0/3246166253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07bcf0 0x55ce6d05bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.368+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/3246166253 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d07b910 0x55ce6d083b80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400ebd0 tx=0x7fe1e4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b239da871f8f0407 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1e3400640 1 -- 172.21.15.62:0/3246166253 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4013d60 con 0x55ce6d07b910 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1e3400640 1 -- 172.21.15.62:0/3246166253 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e4018070 con 0x55ce6d07b910 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1e3400640 1 -- 172.21.15.62:0/3246166253 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e401f640 con 0x55ce6d07b910 2024-03-23T11:28:00.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/3246166253 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d07b910 msgr2=0x55ce6d083b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/3246166253 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d07b910 0x55ce6d083b80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400ebd0 tx=0x7fe1e4011000 comp rx=0 tx=0).stop 2024-03-23T11:28:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/3246166253 shutdown_connections 2024-03-23T11:28:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/3246166253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07bcf0 0x55ce6d05bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/3246166253 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d084280 0x55ce6d05ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/3246166253 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d07b910 0x55ce6d083b80 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/3246166253 >> 172.21.15.62:0/3246166253 conn(0x55ce6d023b40 msgr2=0x55ce6d0a1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/3246166253 shutdown_connections 2024-03-23T11:28:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.369+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/3246166253 wait complete. 2024-03-23T11:28:00.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.370+0000 7fe1fc5d3740 1 -- start start 2024-03-23T11:28:00.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.370+0000 7fe1fc5d3740 1 Processor -- start 2024-03-23T11:28:00.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.371+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07b910 0x55ce6d0a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.371+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07bcf0 0x55ce6d07c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.371+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d087010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6cfbf580 con 0x55ce6d084280 2024-03-23T11:28:00.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6cf60a50 con 0x55ce6d07b910 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce6cf606f0 con 0x55ce6d07bcf0 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1fc5d3740 10 client.0 _ll_register_callbacks cb 0x55ce6cfafed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1f3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d087010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1f3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d087010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53964/0 (socket says 172.21.15.62:53964) 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1f3400640 1 -- 172.21.15.62:0/2496726586 learned_addr learned my addr 172.21.15.62:0/2496726586 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1ebe00640 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07bcf0 0x55ce6d07c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1f3400640 1 -- 172.21.15.62:0/2496726586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07bcf0 msgr2=0x55ce6d07c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07b910 0x55ce6d0a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1f3400640 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07bcf0 0x55ce6d07c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1f3400640 1 -- 172.21.15.62:0/2496726586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07b910 msgr2=0x55ce6d0a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1f3400640 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07b910 0x55ce6d0a3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.372+0000 7fe1f3400640 1 -- 172.21.15.62:0/2496726586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e400e8b0 con 0x55ce6d084280 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.373+0000 7fe1f3400640 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d087010 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009f40 tx=0x7fe1ec004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a41cb4b020dea54 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.373+0000 7fe1d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:00.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.373+0000 7fe1d7e00640 1 -- 172.21.15.62:0/2496726586 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec034d00 con 0x55ce6d084280 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.373+0000 7fe1d7e00640 1 -- 172.21.15.62:0/2496726586 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fe1ec049b30 con 0x55ce6d084280 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.373+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2496726586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ce6cfe9fe0 con 0x55ce6d084280 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.373+0000 7fe1d7e00640 1 -- 172.21.15.62:0/2496726586 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1ec034e60 con 0x55ce6d084280 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.373+0000 7fe1fc5d3740 10 client.7800 fetch_fsmap learned FSMap version 64 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.373+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2496726586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55ce6cfbdc00 con 0x55ce6d084280 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.374+0000 7fe1d7e00640 1 -- 172.21.15.62:0/2496726586 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ce6cfe9fe0 con 0x55ce6d084280 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.374+0000 7fe1d7e00640 1 -- 172.21.15.62:0/2496726586 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe1ec049cf0 con 0x55ce6d084280 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.374+0000 7fe1fc5d3740 10 client.7800 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.374+0000 7fe1fc5d3740 10 client.7800 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.374+0000 7fe1fc5d3740 20 client.7800 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.374+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55ce6d07c6f0 0x55ce6d084bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.374+0000 7fe1fc5d3740 4 client.7800 mds_command: new command op to 7578 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.374+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2496726586 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce6cfb2240 con 0x55ce6d07c6f0 2024-03-23T11:28:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.374+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55ce6d07c6f0 0x55ce6d084bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.375+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55ce6d07c6f0 0x55ce6d084bb0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91e7c719c7e43cb9 server_cookie=a7ee4e337ce9aed5 in_seq=0 out_seq=0 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.375+0000 7fe1d7e00640 10 client.7800 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.375+0000 7fe1d7e00640 1 -- 172.21.15.62:0/2496726586 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55ce6cfb2240 con 0x55ce6d07c6f0 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.375+0000 7fe1d7e00640 10 client.7800 handle_command_reply: tid=0 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.375+0000 7fe1fc5d3740 1 client.7800 shutdown 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2496726586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 msgr2=0x55ce6d087010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d087010 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009f40 tx=0x7fe1ec004ab0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2496726586 shutdown_connections 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55ce6d07c6f0 0x55ce6d084bb0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d087010 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07bcf0 0x55ce6d07c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/2496726586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07b910 0x55ce6d0a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1d7e00640 0 client.7800 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:28:00.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2496726586 >> 172.21.15.62:0/2496726586 conn(0x55ce6d023b40 msgr2=0x55ce6d05ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:00.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2496726586 shutdown_connections 2024-03-23T11:28:00.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2496726586 wait complete. 2024-03-23T11:28:00.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.376+0000 7fe1fc5d3740 20 client.7800 trim_cache size 0 max 16384 2024-03-23T11:28:00.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.392+0000 7fe1fc5d3740 1 Processor -- start 2024-03-23T11:28:00.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.392+0000 7fe1fc5d3740 1 -- start start 2024-03-23T11:28:00.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.393+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07b910 0x55ce6d052450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.393+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07bcf0 0x55ce6d07c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.393+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.393+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6ccbcf90 con 0x55ce6d084280 2024-03-23T11:28:00.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.393+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6d05a200 con 0x55ce6d07bcf0 2024-03-23T11:28:00.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.393+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce6d0876c0 con 0x55ce6d07b910 2024-03-23T11:28:00.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.393+0000 7fe1f3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.393+0000 7fe1f3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0a50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53972/0 (socket says 172.21.15.62:53972) 2024-03-23T11:28:00.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.393+0000 7fe1f3400640 1 -- 172.21.15.62:0/2487211444 learned_addr learned my addr 172.21.15.62:0/2487211444 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.394+0000 7fe1ebe00640 1 --2- 172.21.15.62:0/2487211444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07bcf0 0x55ce6d07c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.394+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/2487211444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07b910 0x55ce6d052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.394+0000 7fe1f3400640 1 -- 172.21.15.62:0/2487211444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07b910 msgr2=0x55ce6d052450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.394+0000 7fe1f3400640 1 --2- 172.21.15.62:0/2487211444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07b910 0x55ce6d052450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.394+0000 7fe1f3400640 1 -- 172.21.15.62:0/2487211444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07bcf0 msgr2=0x55ce6d07c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.394+0000 7fe1f3400640 1 --2- 172.21.15.62:0/2487211444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07bcf0 0x55ce6d07c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.394+0000 7fe1f3400640 1 -- 172.21.15.62:0/2487211444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce6cf5fc60 con 0x55ce6d084280 2024-03-23T11:28:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.394+0000 7fe1f3400640 1 --2- 172.21.15.62:0/2487211444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0a50c0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec002230 tx=0x7fe1ec03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31cc40cdea5c59d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.394+0000 7fe1e3e00640 1 -- 172.21.15.62:0/2487211444 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec0346d0 con 0x55ce6d084280 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.395+0000 7fe1e3e00640 1 -- 172.21.15.62:0/2487211444 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec049250 con 0x55ce6d084280 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.395+0000 7fe1e3e00640 1 -- 172.21.15.62:0/2487211444 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec034d00 con 0x55ce6d084280 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.395+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2487211444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 msgr2=0x55ce6d0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.395+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/2487211444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0a50c0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec002230 tx=0x7fe1ec03f000 comp rx=0 tx=0).stop 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.395+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2487211444 shutdown_connections 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.395+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/2487211444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0a50c0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.395+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/2487211444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07bcf0 0x55ce6d07c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.395+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/2487211444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07b910 0x55ce6d052450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.395+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2487211444 >> 172.21.15.62:0/2487211444 conn(0x55ce6d023b40 msgr2=0x55ce6d083a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.395+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2487211444 shutdown_connections 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.395+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/2487211444 wait complete. 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.396+0000 7fe1fc5d3740 1 -- start start 2024-03-23T11:28:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.396+0000 7fe1fc5d3740 1 Processor -- start 2024-03-23T11:28:00.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.397+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07b910 0x55ce6d10a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.397+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07bcf0 0x55ce6d0a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.397+0000 7fe1fc5d3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0970b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.397+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6d0527d0 con 0x55ce6d084280 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.397+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce6d0c41d0 con 0x55ce6d07b910 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.397+0000 7fe1fc5d3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce6d0859f0 con 0x55ce6d07bcf0 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.397+0000 7fe1fc5d3740 10 client.0 _ll_register_callbacks cb 0x7fe1e400a1b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1f3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1f3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53980/0 (socket says 172.21.15.62:53980) 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1f3400640 1 -- 172.21.15.62:0/475789157 learned_addr learned my addr 172.21.15.62:0/475789157 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07b910 0x55ce6d10a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1ebe00640 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07bcf0 0x55ce6d0a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1f3400640 1 -- 172.21.15.62:0/475789157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07bcf0 msgr2=0x55ce6d0a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1f3400640 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07bcf0 0x55ce6d0a20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1f3400640 1 -- 172.21.15.62:0/475789157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07b910 msgr2=0x55ce6d10a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1f3400640 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07b910 0x55ce6d10a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1f3400640 1 -- 172.21.15.62:0/475789157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce6cfbdc00 con 0x55ce6d084280 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1f3400640 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0970b0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec004b30 tx=0x7fe1ec009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78940e749e602df server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.398+0000 7fe1d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.399+0000 7fe1d7e00640 1 -- 172.21.15.62:0/475789157 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec003ba0 con 0x55ce6d084280 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.399+0000 7fe1d7e00640 1 -- 172.21.15.62:0/475789157 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fe1ec003d00 con 0x55ce6d084280 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.399+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/475789157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe1e4018070 con 0x55ce6d084280 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.399+0000 7fe1d7e00640 1 -- 172.21.15.62:0/475789157 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1ec004040 con 0x55ce6d084280 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.399+0000 7fe1fc5d3740 10 client.7806 fetch_fsmap learned FSMap version 64 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.399+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/475789157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55ce6cf85170 con 0x55ce6d084280 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.399+0000 7fe1d7e00640 1 -- 172.21.15.62:0/475789157 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e4018070 con 0x55ce6d084280 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.399+0000 7fe1d7e00640 1 -- 172.21.15.62:0/475789157 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe1ec0041a0 con 0x55ce6d084280 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.399+0000 7fe1fc5d3740 10 client.7806 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.399+0000 7fe1fc5d3740 10 client.7806 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:28:00.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.399+0000 7fe1fc5d3740 20 client.7806 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.400+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55ce6d105f70 0x55ce6d0975f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.400+0000 7fe1fc5d3740 4 client.7806 mds_command: new command op to 7578 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.400+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/475789157 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ce6ccb34e0 con 0x55ce6d105f70 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.400+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55ce6d105f70 0x55ce6d0975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.400+0000 7fe1f2a00640 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55ce6d105f70 0x55ce6d0975f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76a812aa4e25f0b server_cookie=2e1dcb2d9395b641 in_seq=0 out_seq=0 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.400+0000 7fe1d7e00640 10 client.7806 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1d7e00640 1 -- 172.21.15.62:0/475789157 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ce6ccb34e0 con 0x55ce6d105f70 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1d7e00640 10 client.7806 handle_command_reply: tid=0 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 1 client.7806 shutdown 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/475789157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 msgr2=0x55ce6d0970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0970b0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec004b30 tx=0x7fe1ec009810 comp rx=0 tx=0).stop 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/475789157 shutdown_connections 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55ce6d105f70 0x55ce6d0975f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ce6d084280 0x55ce6d0970b0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ce6d07bcf0 0x55ce6d0a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 1 --2- 172.21.15.62:0/475789157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce6d07b910 0x55ce6d10a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1d7e00640 0 client.7806 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:28:00.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/475789157 >> 172.21.15.62:0/475789157 conn(0x55ce6d023b40 msgr2=0x55ce6d0885b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:00.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/475789157 shutdown_connections 2024-03-23T11:28:00.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 1 -- 172.21.15.62:0/475789157 wait complete. 2024-03-23T11:28:00.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.401+0000 7fe1fc5d3740 20 client.7806 trim_cache size 0 max 16384 2024-03-23T11:28:00.397 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:28:00.397 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-03-23T11:28:00.397 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-03-23T11:28:00.398 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:28:00.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 -- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe1d006a7a0 msgr2=0x7fe1d006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe1d006a7a0 0x7fe1d006cc60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400a950 tx=0x7fe1e4008040 comp rx=0 tx=0).stop 2024-03-23T11:28:00.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 -- 172.21.15.62:0/3977668525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1f41571e0 msgr2=0x7fe1f4144ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1f41571e0 0x7fe1f4144ec0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00c7f0 tx=0x7fe1dc00ccb0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 -- 172.21.15.62:0/3977668525 shutdown_connections 2024-03-23T11:28:00.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe1d006a7a0 0x7fe1d006cc60 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1f41571e0 0x7fe1f4144ec0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1f41613a0 0x7fe1f4145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 --2- 172.21.15.62:0/3977668525 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe1f40ade70 0x7fe1f4150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 -- 172.21.15.62:0/3977668525 >> 172.21.15.62:0/3977668525 conn(0x7fe1f40b02e0 msgr2=0x7fe1f40ae250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:00.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 -- 172.21.15.62:0/3977668525 shutdown_connections 2024-03-23T11:28:00.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.402+0000 7fe1f9a00640 1 -- 172.21.15.62:0/3977668525 wait complete. 2024-03-23T11:28:00.413 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T11:28:00.414 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:28:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.640+0000 7fd58c200640 1 Processor -- start 2024-03-23T11:28:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.640+0000 7fd58c200640 1 -- start start 2024-03-23T11:28:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.641+0000 7fd58c200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd5840d1170 0x7fd5840d28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.641+0000 7fd58c200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd5840d1620 0x7fd5840d2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.641+0000 7fd58c200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd58415cf00 con 0x7fd58415cb20 2024-03-23T11:28:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.641+0000 7fd58c200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5840cd040 con 0x7fd5840d1170 2024-03-23T11:28:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.641+0000 7fd58c200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5840d22f0 con 0x7fd5840d1620 2024-03-23T11:28:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.642+0000 7fd588e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd5840d1170 0x7fd5840d28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.642+0000 7fd588e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd5840d1170 0x7fd5840d28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53988/0 (socket says 172.21.15.62:53988) 2024-03-23T11:28:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.642+0000 7fd588e00640 1 -- 172.21.15.62:0/2152975696 learned_addr learned my addr 172.21.15.62:0/2152975696 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.642+0000 7fd58a200640 1 --2- 172.21.15.62:0/2152975696 >> v2:172.21.15.120:3300/0 conn(0x7fd5840d1620 0x7fd5840d2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.642+0000 7fd588e00640 1 -- 172.21.15.62:0/2152975696 >> v2:172.21.15.120:3300/0 conn(0x7fd5840d1620 msgr2=0x7fd5840d2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.642+0000 7fd588e00640 1 --2- 172.21.15.62:0/2152975696 >> v2:172.21.15.120:3300/0 conn(0x7fd5840d1620 0x7fd5840d2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.642+0000 7fd588e00640 1 -- 172.21.15.62:0/2152975696 >> v1:172.21.15.120:6789/0 conn(0x7fd58415cb20 legacy=0x7fd5840cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.642+0000 7fd588e00640 1 -- 172.21.15.62:0/2152975696 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5840cd2e0 con 0x7fd5840d1170 2024-03-23T11:28:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.643+0000 7fd588e00640 1 --2- 172.21.15.62:0/2152975696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5840d1170 0x7fd5840d28e0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fd5840cfb80 tx=0x7fd57402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c94adb137de23e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:00.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.644+0000 7fd583e00640 1 -- 172.21.15.62:0/2152975696 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574031870 con 0x7fd5840d1170 2024-03-23T11:28:00.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.644+0000 7fd583e00640 1 -- 172.21.15.62:0/2152975696 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5740319d0 con 0x7fd5840d1170 2024-03-23T11:28:00.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.644+0000 7fd583e00640 1 -- 172.21.15.62:0/2152975696 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574031d60 con 0x7fd5840d1170 2024-03-23T11:28:00.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.644+0000 7fd58c200640 1 -- 172.21.15.62:0/2152975696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5840d1170 msgr2=0x7fd5840d28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.644+0000 7fd58c200640 1 --2- 172.21.15.62:0/2152975696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5840d1170 0x7fd5840d28e0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fd5840cfb80 tx=0x7fd57402b910 comp rx=0 tx=0).stop 2024-03-23T11:28:00.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.644+0000 7fd58c200640 1 -- 172.21.15.62:0/2152975696 shutdown_connections 2024-03-23T11:28:00.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.644+0000 7fd58c200640 1 --2- 172.21.15.62:0/2152975696 >> v2:172.21.15.120:3300/0 conn(0x7fd5840d1620 0x7fd5840d2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.644+0000 7fd58c200640 1 --2- 172.21.15.62:0/2152975696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5840d1170 0x7fd5840d28e0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.644+0000 7fd58c200640 1 -- 172.21.15.62:0/2152975696 >> 172.21.15.62:0/2152975696 conn(0x7fd58406b540 msgr2=0x7fd58406b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:00.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.645+0000 7fd58c200640 1 -- 172.21.15.62:0/2152975696 shutdown_connections 2024-03-23T11:28:00.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.645+0000 7fd58c200640 1 -- 172.21.15.62:0/2152975696 wait complete. 2024-03-23T11:28:00.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.646+0000 7fd58c200640 1 Processor -- start 2024-03-23T11:28:00.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.647+0000 7fd58c200640 1 -- start start 2024-03-23T11:28:00.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd58c200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5840d1170 0x7fd5840c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd58c200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5840d1620 0x7fd5840ca430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd58c200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd58415cb20 0x7fd5840ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd58c200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840c9080 con 0x7fd5840d1620 2024-03-23T11:28:00.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd58c200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd584150fd0 con 0x7fd5840d1170 2024-03-23T11:28:00.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd58c200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58415ff70 con 0x7fd58415cb20 2024-03-23T11:28:00.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd588e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5840d1620 0x7fd5840ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd588e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5840d1620 0x7fd5840ca430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53994/0 (socket says 172.21.15.62:53994) 2024-03-23T11:28:00.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd588e00640 1 -- 172.21.15.62:0/2501643768 learned_addr learned my addr 172.21.15.62:0/2501643768 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:00.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd588e00640 1 -- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd58415cb20 msgr2=0x7fd5840ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd588e00640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd58415cb20 0x7fd5840ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd588e00640 1 -- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5840d1170 msgr2=0x7fd5840c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd588e00640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5840d1170 0x7fd5840c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.648+0000 7fd588e00640 1 -- 172.21.15.62:0/2501643768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5740094d0 con 0x7fd5840d1620 2024-03-23T11:28:00.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.649+0000 7fd588e00640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5840d1620 0x7fd5840ca430 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fd574000c00 tx=0x7fd5740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:00.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.649+0000 7fd581600640 1 -- 172.21.15.62:0/2501643768 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57403abe0 con 0x7fd5840d1620 2024-03-23T11:28:00.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.649+0000 7fd581600640 1 -- 172.21.15.62:0/2501643768 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57403ad40 con 0x7fd5840d1620 2024-03-23T11:28:00.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.649+0000 7fd58c200640 1 -- 172.21.15.62:0/2501643768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5840c60c0 con 0x7fd5840d1620 2024-03-23T11:28:00.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.649+0000 7fd581600640 1 -- 172.21.15.62:0/2501643768 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574044610 con 0x7fd5840d1620 2024-03-23T11:28:00.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.649+0000 7fd58c200640 1 -- 172.21.15.62:0/2501643768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58415fc90 con 0x7fd5840d1620 2024-03-23T11:28:00.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.650+0000 7fd58c200640 1 -- 172.21.15.62:0/2501643768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd584066c90 con 0x7fd5840d1620 2024-03-23T11:28:00.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.651+0000 7fd581600640 1 -- 172.21.15.62:0/2501643768 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd57404d450 con 0x7fd5840d1620 2024-03-23T11:28:00.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.652+0000 7fd581600640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd56806a7a0 0x7fd56806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:00.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.652+0000 7fd581600640 1 -- 172.21.15.62:0/2501643768 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fd57402d040 con 0x7fd5840d1620 2024-03-23T11:28:00.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.653+0000 7fd589800640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd56806a7a0 0x7fd56806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:00.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.654+0000 7fd589800640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd56806a7a0 0x7fd56806cc60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd578006fd0 tx=0x7fd578006bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:00.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.657+0000 7fd581600640 1 -- 172.21.15.62:0/2501643768 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd574076800 con 0x7fd5840d1620 2024-03-23T11:28:00.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.895+0000 7fd58c200640 1 -- 172.21.15.62:0/2501643768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd584154300 con 0x7fd5840d1620 2024-03-23T11:28:00.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.896+0000 7fd581600640 1 -- 172.21.15.62:0/2501643768 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fd574079e20 con 0x7fd5840d1620 2024-03-23T11:28:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:00.889 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:26:59.466628+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":6,"addr":"172.21.15.62:6833/2573443114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2573443114},{"type":"v1","addr":"172.21.15.62:6833","nonce":2573443114}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:28:00.889 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 64 2024-03-23T11:28:00.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 -- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd56806a7a0 msgr2=0x7fd56806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd56806a7a0 0x7fd56806cc60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd578006fd0 tx=0x7fd578006bc0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 -- 172.21.15.62:0/2501643768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5840d1620 msgr2=0x7fd5840ca430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:00.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5840d1620 0x7fd5840ca430 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fd574000c00 tx=0x7fd5740046c0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 -- 172.21.15.62:0/2501643768 shutdown_connections 2024-03-23T11:28:00.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd56806a7a0 0x7fd56806cc60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5840d1620 0x7fd5840ca430 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd58415cb20 0x7fd5840ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 --2- 172.21.15.62:0/2501643768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5840d1170 0x7fd5840c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:00.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 -- 172.21.15.62:0/2501643768 >> 172.21.15.62:0/2501643768 conn(0x7fd58406b540 msgr2=0x7fd584072c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:00.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 -- 172.21.15.62:0/2501643768 shutdown_connections 2024-03-23T11:28:00.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:00.899+0000 7fd58c200640 1 -- 172.21.15.62:0/2501643768 wait complete. 2024-03-23T11:28:00.911 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-03-23T11:28:01.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.144+0000 7f475ec00640 1 Processor -- start 2024-03-23T11:28:01.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.145+0000 7f475ec00640 1 -- start start 2024-03-23T11:28:01.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.146+0000 7f475ec00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f47580c5290 0x7f47580b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.146+0000 7f475ec00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475806c4a0 con 0x7f47580cd2c0 2024-03-23T11:28:01.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.146+0000 7f475ec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475815b3b0 con 0x7f475815a8c0 2024-03-23T11:28:01.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.146+0000 7f475ec00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47580c5720 con 0x7f47580c5290 2024-03-23T11:28:01.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.147+0000 7f475cc00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f47580cd2c0 0x7f475806d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:52038/0 (socket says 172.21.15.62:52038) 2024-03-23T11:28:01.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.147+0000 7f475cc00640 1 -- 172.21.15.62:0/3885086433 learned_addr learned my addr 172.21.15.62:0/3885086433 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:01.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.147+0000 7f474fe00640 1 --2- 172.21.15.62:0/3885086433 >> v2:172.21.15.120:3300/0 conn(0x7f47580c5290 0x7f47580b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.147+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878317885 0 0) 0x7f475806c4a0 con 0x7f47580cd2c0 2024-03-23T11:28:01.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.147+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f473c003520 con 0x7f47580cd2c0 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 624551910 0 0) 0x7f473c003520 con 0x7f47580cd2c0 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 >> v2:172.21.15.120:3300/0 conn(0x7f47580c5290 msgr2=0x7f47580b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 --2- 172.21.15.62:0/3885086433 >> v2:172.21.15.120:3300/0 conn(0x7f47580c5290 0x7f47580b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 >> v1:172.21.15.120:6789/0 conn(0x7f475815a8c0 legacy=0x7f475815aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475806c760 con 0x7f47580cd2c0 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f47500019f0 con 0x7f47580cd2c0 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 >> v1:172.21.15.62:6789/0 conn(0x7f47580cd2c0 legacy=0x7f475806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f473c003970 0x7f473c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f473c0047c0 0x7f473c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f473c005180 0x7f473c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47580c5720 con 0x7f473c005180 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c00c020 con 0x7f473c003970 2024-03-23T11:28:01.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c00c1e0 con 0x7f473c0047c0 2024-03-23T11:28:01.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f475cc00640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f473c005180 0x7f473c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.148+0000 7f474fe00640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f473c0047c0 0x7f473c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.149+0000 7f475cc00640 1 -- 172.21.15.62:0/3885086433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f473c0047c0 msgr2=0x7f473c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.149+0000 7f4757e00640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f473c003970 0x7f473c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.149+0000 7f475cc00640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f473c0047c0 0x7f473c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.149+0000 7f475cc00640 1 -- 172.21.15.62:0/3885086433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f473c003970 msgr2=0x7f473c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.149+0000 7f475cc00640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f473c003970 0x7f473c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.149+0000 7f475cc00640 1 -- 172.21.15.62:0/3885086433 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c003520 con 0x7f473c005180 2024-03-23T11:28:01.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.149+0000 7f474fe00640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f473c0047c0 0x7f473c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:01.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.149+0000 7f4757e00640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f473c003970 0x7f473c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:01.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.150+0000 7f475cc00640 1 -- 172.21.15.62:0/3885086433 reap_dead start 2024-03-23T11:28:01.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.150+0000 7f475cc00640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f473c005180 0x7f473c00b980 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f475000a830 tx=0x7f473c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b9b61785dcc2a7c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.150+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47500072d0 con 0x7f473c005180 2024-03-23T11:28:01.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.150+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475002f070 con 0x7f473c005180 2024-03-23T11:28:01.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.150+0000 7f4757400640 1 -- 172.21.15.62:0/3885086433 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750007650 con 0x7f473c005180 2024-03-23T11:28:01.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.151+0000 7f475ec00640 1 -- 172.21.15.62:0/3885086433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f473c005180 msgr2=0x7f473c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.151+0000 7f475ec00640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f473c005180 0x7f473c00b980 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f475000a830 tx=0x7f473c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.151+0000 7f475ec00640 1 -- 172.21.15.62:0/3885086433 shutdown_connections 2024-03-23T11:28:01.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.151+0000 7f475ec00640 1 --2- 172.21.15.62:0/3885086433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f473c005180 0x7f473c00b980 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.151+0000 7f475ec00640 1 -- 172.21.15.62:0/3885086433 >> 172.21.15.62:0/3885086433 conn(0x7f4758066050 msgr2=0x7f4758069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:01.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.151+0000 7f475ec00640 1 -- 172.21.15.62:0/3885086433 shutdown_connections 2024-03-23T11:28:01.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.151+0000 7f475ec00640 1 -- 172.21.15.62:0/3885086433 wait complete. 2024-03-23T11:28:01.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.153+0000 7f475ec00640 1 Processor -- start 2024-03-23T11:28:01.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.153+0000 7f475ec00640 1 -- start start 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.154+0000 7f475ec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f473c005180 0x7f47580c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.154+0000 7f475ec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47580a76f0 0x7f4758144f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.154+0000 7f475ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47580c35d0 0x7f47580c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.154+0000 7f475ec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47580c8260 con 0x7f47580a76f0 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.154+0000 7f475ec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47580c3f90 con 0x7f473c005180 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.154+0000 7f475ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47580c7dd0 con 0x7f47580c35d0 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.154+0000 7f474fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47580a76f0 0x7f4758144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.155+0000 7f474fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47580a76f0 0x7f4758144f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54004/0 (socket says 172.21.15.62:54004) 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.155+0000 7f474fe00640 1 -- 172.21.15.62:0/2043060524 learned_addr learned my addr 172.21.15.62:0/2043060524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.155+0000 7f4757e00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f473c005180 0x7f47580c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.155+0000 7f475cc00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47580c35d0 0x7f47580c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.155+0000 7f474fe00640 1 -- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47580c35d0 msgr2=0x7f47580c3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.155+0000 7f474fe00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47580c35d0 0x7f47580c3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.155+0000 7f474fe00640 1 -- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f473c005180 msgr2=0x7f47580c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.155+0000 7f474fe00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f473c005180 0x7f47580c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.155+0000 7f474fe00640 1 -- 172.21.15.62:0/2043060524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4758145440 con 0x7f47580a76f0 2024-03-23T11:28:01.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.155+0000 7f474fe00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47580a76f0 0x7f4758144f00 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f47400108b0 tx=0x7f4740010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.156+0000 7f4754c00640 1 -- 172.21.15.62:0/2043060524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474000eb60 con 0x7f47580a76f0 2024-03-23T11:28:01.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.156+0000 7f4754c00640 1 -- 172.21.15.62:0/2043060524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474000ecc0 con 0x7f47580a76f0 2024-03-23T11:28:01.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.156+0000 7f4754c00640 1 -- 172.21.15.62:0/2043060524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474001c400 con 0x7f47580a76f0 2024-03-23T11:28:01.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.156+0000 7f475ec00640 1 -- 172.21.15.62:0/2043060524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47580a7b70 con 0x7f47580a76f0 2024-03-23T11:28:01.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.156+0000 7f475ec00640 1 -- 172.21.15.62:0/2043060524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47580c74a0 con 0x7f47580a76f0 2024-03-23T11:28:01.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.158+0000 7f4754c00640 1 -- 172.21.15.62:0/2043060524 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f474001c560 con 0x7f47580a76f0 2024-03-23T11:28:01.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.160+0000 7f4754c00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f473406a7d0 0x7f473406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.160+0000 7f4754c00640 1 -- 172.21.15.62:0/2043060524 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f4740089480 con 0x7f47580a76f0 2024-03-23T11:28:01.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.161+0000 7f4757e00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f473406a7d0 0x7f473406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.161+0000 7f4761792740 1 Processor -- start 2024-03-23T11:28:01.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.161+0000 7f4761792740 1 -- start start 2024-03-23T11:28:01.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.162+0000 7f4757e00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f473406a7d0 0x7f473406cc90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4748000f60 tx=0x7f4748009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.162+0000 7f4761792740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d0e420 0x55b389cded90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.162+0000 7f4761792740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d0e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.162+0000 7f4761792740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0b0d0 0x55b389cda790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.162+0000 7f4761792740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b389c4ad80 con 0x55b389d0bae0 2024-03-23T11:28:01.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.162+0000 7f4761792740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b389c48350 con 0x55b389d0b0d0 2024-03-23T11:28:01.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.162+0000 7f4761792740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b389c43c40 con 0x55b389d0e420 2024-03-23T11:28:01.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.163+0000 7f474fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d0e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.163+0000 7f474fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d0e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54008/0 (socket says 172.21.15.62:54008) 2024-03-23T11:28:01.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.163+0000 7f474fe00640 1 -- 172.21.15.62:0/2099714699 learned_addr learned my addr 172.21.15.62:0/2099714699 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:01.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.163+0000 7f474fe00640 1 -- 172.21.15.62:0/2099714699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d0e420 msgr2=0x55b389cded90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.163+0000 7f474fe00640 1 --2- 172.21.15.62:0/2099714699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d0e420 0x55b389cded90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.163+0000 7f474fe00640 1 -- 172.21.15.62:0/2099714699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0b0d0 msgr2=0x55b389cda790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.163+0000 7f475cc00640 1 --2- 172.21.15.62:0/2099714699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0b0d0 0x55b389cda790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.163+0000 7f474fe00640 1 --2- 172.21.15.62:0/2099714699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0b0d0 0x55b389cda790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.163+0000 7f474fe00640 1 -- 172.21.15.62:0/2099714699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b389c0e3e0 con 0x55b389d0bae0 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.163+0000 7f475cc00640 1 --2- 172.21.15.62:0/2099714699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0b0d0 0x55b389cda790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.163+0000 7f474fe00640 1 --2- 172.21.15.62:0/2099714699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d0e800 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f4740087800 tx=0x7f4740058520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3bbc8992bf1a07d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.164+0000 7f474cc00640 1 -- 172.21.15.62:0/2099714699 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474005c070 con 0x55b389d0bae0 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.164+0000 7f474cc00640 1 -- 172.21.15.62:0/2099714699 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4740060860 con 0x55b389d0bae0 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.164+0000 7f474cc00640 1 -- 172.21.15.62:0/2099714699 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740054c30 con 0x55b389d0bae0 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.164+0000 7f4761792740 1 -- 172.21.15.62:0/2099714699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 msgr2=0x55b389d0e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.164+0000 7f4761792740 1 --2- 172.21.15.62:0/2099714699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d0e800 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f4740087800 tx=0x7f4740058520 comp rx=0 tx=0).stop 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.164+0000 7f4761792740 1 -- 172.21.15.62:0/2099714699 shutdown_connections 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.164+0000 7f4761792740 1 --2- 172.21.15.62:0/2099714699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d0e800 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.164+0000 7f4761792740 1 --2- 172.21.15.62:0/2099714699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0b0d0 0x55b389cda790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.164+0000 7f4761792740 1 --2- 172.21.15.62:0/2099714699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d0e420 0x55b389cded90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.164+0000 7f4761792740 1 -- 172.21.15.62:0/2099714699 >> 172.21.15.62:0/2099714699 conn(0x55b389cdca00 msgr2=0x55b389cde2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:01.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.165+0000 7f4761792740 1 -- 172.21.15.62:0/2099714699 shutdown_connections 2024-03-23T11:28:01.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.165+0000 7f4761792740 1 -- 172.21.15.62:0/2099714699 wait complete. 2024-03-23T11:28:01.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.165+0000 7f4761792740 1 -- start start 2024-03-23T11:28:01.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.166+0000 7f4761792740 1 Processor -- start 2024-03-23T11:28:01.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.166+0000 7f4761792740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d07440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4761792740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d0e420 0x55b389d04de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4761792740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d03370 0x55b389d05320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4761792740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b389be4ce0 con 0x55b389d0bae0 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4761792740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b389be4980 con 0x55b389d03370 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4761792740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b3899ac000 con 0x55b389d0e420 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4761792740 10 client.0 _ll_register_callbacks cb 0x55b389c3d410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4757e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d07440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4757e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d07440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54022/0 (socket says 172.21.15.62:54022) 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4757e00640 1 -- 172.21.15.62:0/786633906 learned_addr learned my addr 172.21.15.62:0/786633906 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f475cc00640 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d03370 0x55b389d05320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f474fe00640 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d0e420 0x55b389d04de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4757e00640 1 -- 172.21.15.62:0/786633906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d0e420 msgr2=0x55b389d04de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4757e00640 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d0e420 0x55b389d04de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4757e00640 1 -- 172.21.15.62:0/786633906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d03370 msgr2=0x55b389d05320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4757e00640 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d03370 0x55b389d05320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.167+0000 7f4757e00640 1 -- 172.21.15.62:0/786633906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47580c74a0 con 0x55b389d0bae0 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.168+0000 7f4757e00640 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d07440 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f474800fe60 tx=0x7f47480039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd436b8cd589a0cc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.168+0000 7f4745600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.168+0000 7f4745600640 1 -- 172.21.15.62:0/786633906 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474800fe90 con 0x55b389d0bae0 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.168+0000 7f4745600640 1 -- 172.21.15.62:0/786633906 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f4748023b40 con 0x55b389d0bae0 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.168+0000 7f4761792740 1 -- 172.21.15.62:0/786633906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b3898f58d0 con 0x55b389d0bae0 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.168+0000 7f4745600640 1 -- 172.21.15.62:0/786633906 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4748023d00 con 0x55b389d0bae0 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.168+0000 7f4761792740 10 client.7824 fetch_fsmap learned FSMap version 64 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.168+0000 7f4761792740 1 -- 172.21.15.62:0/786633906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55b389c422c0 con 0x55b389d0bae0 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.169+0000 7f4745600640 1 -- 172.21.15.62:0/786633906 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b3898f58d0 con 0x55b389d0bae0 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.169+0000 7f4745600640 1 -- 172.21.15.62:0/786633906 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4748012070 con 0x55b389d0bae0 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.169+0000 7f4761792740 10 client.7824 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.169+0000 7f4761792740 10 client.7824 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.169+0000 7f4761792740 20 client.7824 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:01.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.169+0000 7f4761792740 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b389d02e50 0x55b389d46980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.169+0000 7f4761792740 4 client.7824 mds_command: new command op to 7578 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:28:01.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.169+0000 7f4761792740 1 -- 172.21.15.62:0/786633906 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b3899384a0 con 0x55b389d02e50 2024-03-23T11:28:01.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.169+0000 7f474fe00640 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b389d02e50 0x55b389d46980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.170+0000 7f474fe00640 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b389d02e50 0x55b389d46980 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e84a9b49d30b4027 server_cookie=ecda7da5457785c8 in_seq=0 out_seq=0 2024-03-23T11:28:01.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.170+0000 7f4745600640 10 client.7824 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:28:01.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.170+0000 7f4745600640 1 -- 172.21.15.62:0/786633906 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55b3899384a0 con 0x55b389d02e50 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.170+0000 7f4745600640 10 client.7824 handle_command_reply: tid=0 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.170+0000 7f4761792740 1 client.7824 shutdown 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4761792740 1 -- 172.21.15.62:0/786633906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 msgr2=0x55b389d07440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4761792740 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d07440 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f474800fe60 tx=0x7f47480039c0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4761792740 1 -- 172.21.15.62:0/786633906 shutdown_connections 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4761792740 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b389d02e50 0x55b389d46980 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4761792740 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d03370 0x55b389d05320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4761792740 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d0e420 0x55b389d04de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4761792740 1 --2- 172.21.15.62:0/786633906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0bae0 0x55b389d07440 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4745600640 0 client.7824 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4761792740 1 -- 172.21.15.62:0/786633906 >> 172.21.15.62:0/786633906 conn(0x55b389d0b0d0 msgr2=0x55b389d270f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4761792740 1 -- 172.21.15.62:0/786633906 shutdown_connections 2024-03-23T11:28:01.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4761792740 1 -- 172.21.15.62:0/786633906 wait complete. 2024-03-23T11:28:01.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.171+0000 7f4761792740 20 client.7824 trim_cache size 0 max 16384 2024-03-23T11:28:01.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.185+0000 7f4761792740 1 Processor -- start 2024-03-23T11:28:01.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.186+0000 7f4761792740 1 -- start start 2024-03-23T11:28:01.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.186+0000 7f4761792740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d03370 0x55b389d292a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.186+0000 7f4761792740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 0x55b389d041b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.186+0000 7f4761792740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389cda7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.186+0000 7f4761792740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b389d0b710 con 0x55b389d0e420 2024-03-23T11:28:01.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.186+0000 7f4761792740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b389d48e00 con 0x55b389d0bae0 2024-03-23T11:28:01.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.186+0000 7f4761792740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b389d298a0 con 0x55b389d03370 2024-03-23T11:28:01.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.187+0000 7f475cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389cda7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.187+0000 7f4757e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d03370 0x55b389d292a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.187+0000 7f475cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389cda7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54038/0 (socket says 172.21.15.62:54038) 2024-03-23T11:28:01.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.187+0000 7f475cc00640 1 -- 172.21.15.62:0/3181404346 learned_addr learned my addr 172.21.15.62:0/3181404346 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:01.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.187+0000 7f474fe00640 1 --2- 172.21.15.62:0/3181404346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 0x55b389d041b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.188+0000 7f475cc00640 1 -- 172.21.15.62:0/3181404346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d03370 msgr2=0x55b389d292a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.188+0000 7f475cc00640 1 --2- 172.21.15.62:0/3181404346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d03370 0x55b389d292a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.188+0000 7f475cc00640 1 -- 172.21.15.62:0/3181404346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 msgr2=0x55b389d041b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.188+0000 7f475cc00640 1 --2- 172.21.15.62:0/3181404346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 0x55b389d041b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.188+0000 7f475cc00640 1 -- 172.21.15.62:0/3181404346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b389be4170 con 0x55b389d0e420 2024-03-23T11:28:01.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.188+0000 7f474fe00640 1 --2- 172.21.15.62:0/3181404346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 0x55b389d041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:28:01.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.188+0000 7f475cc00640 1 --2- 172.21.15.62:0/3181404346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389cda7c0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f475000a950 tx=0x7f475002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d3828702f07c9d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.188+0000 7f474d600640 1 -- 172.21.15.62:0/3181404346 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750001930 con 0x55b389d0e420 2024-03-23T11:28:01.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.188+0000 7f474d600640 1 -- 172.21.15.62:0/3181404346 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4750001a90 con 0x55b389d0e420 2024-03-23T11:28:01.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.188+0000 7f474d600640 1 -- 172.21.15.62:0/3181404346 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47500072d0 con 0x55b389d0e420 2024-03-23T11:28:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.189+0000 7f4761792740 1 -- 172.21.15.62:0/3181404346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 msgr2=0x55b389cda7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.189+0000 7f4761792740 1 --2- 172.21.15.62:0/3181404346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389cda7c0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f475000a950 tx=0x7f475002f000 comp rx=0 tx=0).stop 2024-03-23T11:28:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.189+0000 7f4761792740 1 -- 172.21.15.62:0/3181404346 shutdown_connections 2024-03-23T11:28:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.189+0000 7f4761792740 1 --2- 172.21.15.62:0/3181404346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389cda7c0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.189+0000 7f4761792740 1 --2- 172.21.15.62:0/3181404346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 0x55b389d041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.189+0000 7f4761792740 1 --2- 172.21.15.62:0/3181404346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d03370 0x55b389d292a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.189+0000 7f4761792740 1 -- 172.21.15.62:0/3181404346 >> 172.21.15.62:0/3181404346 conn(0x55b389d0b0d0 msgr2=0x55b389d068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:01.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.189+0000 7f4761792740 1 -- 172.21.15.62:0/3181404346 shutdown_connections 2024-03-23T11:28:01.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.189+0000 7f4761792740 1 -- 172.21.15.62:0/3181404346 wait complete. 2024-03-23T11:28:01.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.190+0000 7f4761792740 1 -- start start 2024-03-23T11:28:01.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.190+0000 7f4761792740 1 Processor -- start 2024-03-23T11:28:01.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.191+0000 7f4761792740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d03370 0x55b389d0a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.191+0000 7f4761792740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 0x55b389d48630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.191+0000 7f4761792740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389d083b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.191+0000 7f4761792740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b389d8fdb0 con 0x55b389d0e420 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.191+0000 7f4761792740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b389cdfbc0 con 0x55b389d0bae0 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.191+0000 7f4761792740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b389d09940 con 0x55b389d03370 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.191+0000 7f4761792740 10 client.0 _ll_register_callbacks cb 0x7f4748001470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.191+0000 7f475cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389d083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.191+0000 7f475cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389d083b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54044/0 (socket says 172.21.15.62:54044) 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.191+0000 7f475cc00640 1 -- 172.21.15.62:0/1198143701 learned_addr learned my addr 172.21.15.62:0/1198143701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f474fe00640 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 0x55b389d48630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f475cc00640 1 -- 172.21.15.62:0/1198143701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d03370 msgr2=0x55b389d0a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f475cc00640 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d03370 0x55b389d0a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f475cc00640 1 -- 172.21.15.62:0/1198143701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 msgr2=0x55b389d48630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f475cc00640 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 0x55b389d48630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f475cc00640 1 -- 172.21.15.62:0/1198143701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4750038040 con 0x55b389d0e420 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f474fe00640 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 0x55b389d48630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f475cc00640 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389d083b0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4750005c20 tx=0x7f4750009050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790b9415ea21d4c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f4745600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f4745600640 1 -- 172.21.15.62:0/1198143701 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750001930 con 0x55b389d0e420 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f4745600640 1 -- 172.21.15.62:0/1198143701 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f47500070d0 con 0x55b389d0e420 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.192+0000 7f4761792740 1 -- 172.21.15.62:0/1198143701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4740060860 con 0x55b389d0e420 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4745600640 1 -- 172.21.15.62:0/1198143701 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47500072d0 con 0x55b389d0e420 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4761792740 10 client.7830 fetch_fsmap learned FSMap version 64 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4761792740 1 -- 172.21.15.62:0/1198143701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55b389c09a50 con 0x55b389d0e420 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4745600640 1 -- 172.21.15.62:0/1198143701 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4740060860 con 0x55b389d0e420 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4745600640 1 -- 172.21.15.62:0/1198143701 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4750045020 con 0x55b389d0e420 2024-03-23T11:28:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4761792740 10 client.7830 fetch_fsmap finished waiting for FSMap version 64 2024-03-23T11:28:01.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4761792740 10 client.7830 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:28:01.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4761792740 20 client.7830 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:01.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4761792740 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b389d901d0 0x55b389d905b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4761792740 4 client.7830 mds_command: new command op to 7578 tid=0[{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-23T11:28:01.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4761792740 1 -- 172.21.15.62:0/1198143701 --> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x55b3899497a0 con 0x55b389d901d0 2024-03-23T11:28:01.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.193+0000 7f4757e00640 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b389d901d0 0x55b389d905b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.194+0000 7f4757e00640 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b389d901d0 0x55b389d905b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cae362fc81653317 server_cookie=497c51fbdbe4f8d2 in_seq=0 out_seq=0 2024-03-23T11:28:01.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.194+0000 7f4745600640 10 client.7830 ms_handle_connect on v2:172.21.15.62:6832/2573443114 2024-03-23T11:28:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.194+0000 7f4745600640 1 -- 172.21.15.62:0/1198143701 <== mds.0 v2:172.21.15.62:6832/2573443114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+143 (crc 0 0 0) 0x55b3899497a0 con 0x55b389d901d0 2024-03-23T11:28:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.194+0000 7f4745600640 10 client.7830 handle_command_reply: tid=0 2024-03-23T11:28:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.194+0000 7f4761792740 1 client.7830 shutdown 2024-03-23T11:28:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4761792740 1 -- 172.21.15.62:0/1198143701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 msgr2=0x55b389d083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4761792740 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389d083b0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4750005c20 tx=0x7f4750009050 comp rx=0 tx=0).stop 2024-03-23T11:28:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4761792740 1 -- 172.21.15.62:0/1198143701 shutdown_connections 2024-03-23T11:28:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4761792740 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.62:6832/2573443114,v1:172.21.15.62:6833/2573443114] conn(0x55b389d901d0 0x55b389d905b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4761792740 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b389d0e420 0x55b389d083b0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4761792740 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b389d0bae0 0x55b389d48630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4761792740 1 --2- 172.21.15.62:0/1198143701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b389d03370 0x55b389d0a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4745600640 0 client.7830 ms_handle_reset on v2:172.21.15.62:6832/2573443114 2024-03-23T11:28:01.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4761792740 1 -- 172.21.15.62:0/1198143701 >> 172.21.15.62:0/1198143701 conn(0x55b389d0b0d0 msgr2=0x55b389cda9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:01.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4761792740 1 -- 172.21.15.62:0/1198143701 shutdown_connections 2024-03-23T11:28:01.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4761792740 1 -- 172.21.15.62:0/1198143701 wait complete. 2024-03-23T11:28:01.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.195+0000 7f4761792740 20 client.7830 trim_cache size 0 max 16384 2024-03-23T11:28:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:01.191 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:28:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:28:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "damage_type": "backtrace", 2024-03-23T11:28:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 846551848, 2024-03-23T11:28:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "ino": 1099511627777, 2024-03-23T11:28:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/testdir/hardlink" 2024-03-23T11:28:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:28:01.192 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:28:01.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 -- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f473406a7d0 msgr2=0x7f473406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f473406a7d0 0x7f473406cc90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4748000f60 tx=0x7f4748009040 comp rx=0 tx=0).stop 2024-03-23T11:28:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 -- 172.21.15.62:0/2043060524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47580a76f0 msgr2=0x7f4758144f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47580a76f0 0x7f4758144f00 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f47400108b0 tx=0x7f4740010d70 comp rx=0 tx=0).stop 2024-03-23T11:28:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 -- 172.21.15.62:0/2043060524 shutdown_connections 2024-03-23T11:28:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f473406a7d0 0x7f473406cc90 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47580a76f0 0x7f4758144f00 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47580c35d0 0x7f47580c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 --2- 172.21.15.62:0/2043060524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f473c005180 0x7f47580c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 -- 172.21.15.62:0/2043060524 >> 172.21.15.62:0/2043060524 conn(0x7f4758066050 msgr2=0x7f475815d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 -- 172.21.15.62:0/2043060524 shutdown_connections 2024-03-23T11:28:01.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.196+0000 7f475ec00640 1 -- 172.21.15.62:0/2043060524 wait complete. 2024-03-23T11:28:01.207 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:28:01.208 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:28:01.259 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:28:01.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.474+0000 7ff02b200640 1 Processor -- start 2024-03-23T11:28:01.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.475+0000 7ff02b200640 1 -- start start 2024-03-23T11:28:01.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.476+0000 7ff02b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff0241608b0 0x7ff0240b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.476+0000 7ff02b200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0240ce510 con 0x7ff02415fcf0 2024-03-23T11:28:01.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.476+0000 7ff02b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0240cdc70 con 0x7ff024160420 2024-03-23T11:28:01.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.476+0000 7ff02b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0240cdf50 con 0x7ff0241608b0 2024-03-23T11:28:01.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.476+0000 7ff029e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff02415fcf0 0x7ff0240d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:52050/0 (socket says 172.21.15.62:52050) 2024-03-23T11:28:01.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.476+0000 7ff029e00640 1 -- 172.21.15.62:0/1794225672 learned_addr learned my addr 172.21.15.62:0/1794225672 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:01.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.477+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915576270 0 0) 0x7ff0240ce510 con 0x7ff02415fcf0 2024-03-23T11:28:01.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.477+0000 7ff02a800640 1 --2- 172.21.15.62:0/1794225672 >> v2:172.21.15.120:3300/0 conn(0x7ff0241608b0 0x7ff0240b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.477+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff00c003520 con 0x7ff02415fcf0 2024-03-23T11:28:01.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.477+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2134488369 0 0) 0x7ff00c003520 con 0x7ff02415fcf0 2024-03-23T11:28:01.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.477+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 >> v2:172.21.15.120:3300/0 conn(0x7ff0241608b0 msgr2=0x7ff0240b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.477+0000 7ff028a00640 1 --2- 172.21.15.62:0/1794225672 >> v2:172.21.15.120:3300/0 conn(0x7ff0241608b0 0x7ff0240b64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.477+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 >> v1:172.21.15.120:6789/0 conn(0x7ff024160420 legacy=0x7ff0240b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.477+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0240ce230 con 0x7ff02415fcf0 2024-03-23T11:28:01.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.477+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7ff018001670 con 0x7ff02415fcf0 2024-03-23T11:28:01.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.477+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 >> v1:172.21.15.62:6789/0 conn(0x7ff02415fcf0 legacy=0x7ff0240d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff028a00640 1 --2- 172.21.15.62:0/1794225672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 0x7ff00c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff028a00640 1 --2- 172.21.15.62:0/1794225672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff00c0047c0 0x7ff00c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff028a00640 1 --2- 172.21.15.62:0/1794225672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff00c0052a0 0x7ff00c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240cdf50 con 0x7ff00c0039c0 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c00c1c0 con 0x7ff00c0047c0 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c00c380 con 0x7ff00c0052a0 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff029e00640 1 --2- 172.21.15.62:0/1794225672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 0x7ff00c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff029e00640 1 -- 172.21.15.62:0/1794225672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff00c0052a0 msgr2=0x7ff00c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff029e00640 1 --2- 172.21.15.62:0/1794225672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff00c0052a0 0x7ff00c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff029e00640 1 -- 172.21.15.62:0/1794225672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff00c0047c0 msgr2=0x7ff00c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff029400640 1 --2- 172.21.15.62:0/1794225672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff00c0047c0 0x7ff00c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff029e00640 1 --2- 172.21.15.62:0/1794225672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff00c0047c0 0x7ff00c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff02a800640 1 -- 172.21.15.62:0/1794225672 reap_dead start 2024-03-23T11:28:01.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.478+0000 7ff029e00640 1 -- 172.21.15.62:0/1794225672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c003520 con 0x7ff00c0039c0 2024-03-23T11:28:01.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.479+0000 7ff029e00640 1 --2- 172.21.15.62:0/1794225672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 0x7ff00c003de0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c0e0 tx=0x7ff00c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ea3e1e1f80ab58 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.480+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff018005ad0 con 0x7ff00c0039c0 2024-03-23T11:28:01.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.480+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff018005c30 con 0x7ff00c0039c0 2024-03-23T11:28:01.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.480+0000 7ff028a00640 1 -- 172.21.15.62:0/1794225672 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01802ca50 con 0x7ff00c0039c0 2024-03-23T11:28:01.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.480+0000 7ff02b200640 1 -- 172.21.15.62:0/1794225672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 msgr2=0x7ff00c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.480+0000 7ff02b200640 1 --2- 172.21.15.62:0/1794225672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 0x7ff00c003de0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c0e0 tx=0x7ff00c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.480+0000 7ff02b200640 1 -- 172.21.15.62:0/1794225672 shutdown_connections 2024-03-23T11:28:01.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.480+0000 7ff02b200640 1 --2- 172.21.15.62:0/1794225672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 0x7ff00c003de0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.480+0000 7ff02b200640 1 -- 172.21.15.62:0/1794225672 >> 172.21.15.62:0/1794225672 conn(0x7ff024089f60 msgr2=0x7ff0240d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:01.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.481+0000 7ff02b200640 1 -- 172.21.15.62:0/1794225672 shutdown_connections 2024-03-23T11:28:01.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.481+0000 7ff02b200640 1 -- 172.21.15.62:0/1794225672 wait complete. 2024-03-23T11:28:01.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.482+0000 7ff02b200640 1 Processor -- start 2024-03-23T11:28:01.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.482+0000 7ff02b200640 1 -- start start 2024-03-23T11:28:01.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.483+0000 7ff02b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 0x7ff0240c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.483+0000 7ff02b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0240cb910 0x7ff0240cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.483+0000 7ff02b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff02415a660 0x7ff02415aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.483+0000 7ff02b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0241f30a0 con 0x7ff00c0039c0 2024-03-23T11:28:01.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.483+0000 7ff02b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240c7950 con 0x7ff02415a660 2024-03-23T11:28:01.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.483+0000 7ff02b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240c7c10 con 0x7ff0240cb910 2024-03-23T11:28:01.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.484+0000 7ff029e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 0x7ff0240c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.484+0000 7ff029e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 0x7ff0240c6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54052/0 (socket says 172.21.15.62:54052) 2024-03-23T11:28:01.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.484+0000 7ff029e00640 1 -- 172.21.15.62:0/915057364 learned_addr learned my addr 172.21.15.62:0/915057364 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:01.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.484+0000 7ff029400640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0240cb910 0x7ff0240cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.484+0000 7ff02a800640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff02415a660 0x7ff02415aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.484+0000 7ff029e00640 1 -- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0240cb910 msgr2=0x7ff0240cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.484+0000 7ff029e00640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0240cb910 0x7ff0240cbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.484+0000 7ff029e00640 1 -- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff02415a660 msgr2=0x7ff02415aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.484+0000 7ff029e00640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff02415a660 0x7ff02415aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.484+0000 7ff029e00640 1 -- 172.21.15.62:0/915057364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0240c7ef0 con 0x7ff00c0039c0 2024-03-23T11:28:01.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.484+0000 7ff02a800640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff02415a660 0x7ff02415aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:01.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.485+0000 7ff029e00640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 0x7ff0240c6140 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c0e0 tx=0x7ff01802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.485+0000 7ff01e000640 1 -- 172.21.15.62:0/915057364 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01803c040 con 0x7ff00c0039c0 2024-03-23T11:28:01.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.485+0000 7ff01e000640 1 -- 172.21.15.62:0/915057364 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01802cd60 con 0x7ff00c0039c0 2024-03-23T11:28:01.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.485+0000 7ff02b200640 1 -- 172.21.15.62:0/915057364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0240c81d0 con 0x7ff00c0039c0 2024-03-23T11:28:01.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.485+0000 7ff01e000640 1 -- 172.21.15.62:0/915057364 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0180072b0 con 0x7ff00c0039c0 2024-03-23T11:28:01.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.485+0000 7ff02b200640 1 -- 172.21.15.62:0/915057364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0240c89d0 con 0x7ff00c0039c0 2024-03-23T11:28:01.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.486+0000 7ff02b200640 1 -- 172.21.15.62:0/915057364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff024066bb0 con 0x7ff00c0039c0 2024-03-23T11:28:01.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.487+0000 7ff01e000640 1 -- 172.21.15.62:0/915057364 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff018032070 con 0x7ff00c0039c0 2024-03-23T11:28:01.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.488+0000 7ff01e000640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff00806a7a0 0x7ff00806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.489+0000 7ff01e000640 1 -- 172.21.15.62:0/915057364 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7ff0180aa370 con 0x7ff00c0039c0 2024-03-23T11:28:01.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.489+0000 7ff029400640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff00806a7a0 0x7ff00806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.490+0000 7ff029400640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff00806a7a0 0x7ff00806cc60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff014000c00 tx=0x7ff014009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.493+0000 7ff01e000640 1 -- 172.21.15.62:0/915057364 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff018075730 con 0x7ff00c0039c0 2024-03-23T11:28:01.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.694+0000 7ff02b200640 1 -- 172.21.15.62:0/915057364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0240b4d90 con 0x7ff00c0039c0 2024-03-23T11:28:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.697+0000 7ff01e000640 1 -- 172.21.15.62:0/915057364 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16271 (secure 0 0 0) 0x7ff018078d50 con 0x7ff00c0039c0 2024-03-23T11:28:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:01.689 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":92,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:26:47.481481+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:26:22.794149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-23T11:26:24.150997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6833/3056962984":"2024-03-24T11:26:47.481437+0000","172.21.15.62:6832/3056962984":"2024-03-24T11:26:47.481437+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:28:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.699+0000 7ff02b200640 1 -- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff00806a7a0 msgr2=0x7ff00806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.699+0000 7ff02b200640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff00806a7a0 0x7ff00806cc60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff014000c00 tx=0x7ff014009040 comp rx=0 tx=0).stop 2024-03-23T11:28:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.699+0000 7ff02b200640 1 -- 172.21.15.62:0/915057364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 msgr2=0x7ff0240c6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.699+0000 7ff02b200640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 0x7ff0240c6140 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c0e0 tx=0x7ff01802e070 comp rx=0 tx=0).stop 2024-03-23T11:28:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.700+0000 7ff02b200640 1 -- 172.21.15.62:0/915057364 shutdown_connections 2024-03-23T11:28:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.700+0000 7ff02b200640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff00806a7a0 0x7ff00806cc60 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.700+0000 7ff02b200640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff02415a660 0x7ff02415aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.700+0000 7ff02b200640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff0240cb910 0x7ff0240cbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.700+0000 7ff02b200640 1 --2- 172.21.15.62:0/915057364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff00c0039c0 0x7ff0240c6140 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.700+0000 7ff02b200640 1 -- 172.21.15.62:0/915057364 >> 172.21.15.62:0/915057364 conn(0x7ff024089f60 msgr2=0x7ff0240b6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:01.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.700+0000 7ff02b200640 1 -- 172.21.15.62:0/915057364 shutdown_connections 2024-03-23T11:28:01.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.700+0000 7ff02b200640 1 -- 172.21.15.62:0/915057364 wait complete. 2024-03-23T11:28:01.712 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:26:22.794149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:26:24.150997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:28:01.712 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:28:01.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.940+0000 7f3a0ee00640 1 Processor -- start 2024-03-23T11:28:01.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.940+0000 7f3a0ee00640 1 -- start start 2024-03-23T11:28:01.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.942+0000 7f3a0ee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a080cd7c0 0x7f3a080b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.942+0000 7f3a0ee00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3a0806c300 0x7f3a080cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.942+0000 7f3a0ee00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a080cad20 con 0x7f3a0806dca0 2024-03-23T11:28:01.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.942+0000 7f3a0ee00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0806c770 con 0x7f3a0806c300 2024-03-23T11:28:01.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.942+0000 7f3a0ee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0806e130 con 0x7f3a080cd7c0 2024-03-23T11:28:01.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.942+0000 7f3a0ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3a0806c300 0x7f3a080cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.943+0000 7f3a0ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3a0806c300 0x7f3a080cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54064/0 (socket says 172.21.15.62:54064) 2024-03-23T11:28:01.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.943+0000 7f3a0ce00640 1 -- 172.21.15.62:0/1053024439 learned_addr learned my addr 172.21.15.62:0/1053024439 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:01.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.943+0000 7f3a07e00640 1 --2- 172.21.15.62:0/1053024439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a080cd7c0 0x7f3a080b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.943+0000 7f3a0ce00640 1 -- 172.21.15.62:0/1053024439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a080cd7c0 msgr2=0x7f3a080b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:01.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.943+0000 7f3a0ce00640 1 --2- 172.21.15.62:0/1053024439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a080cd7c0 0x7f3a080b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.943+0000 7f3a07400640 1 -- 172.21.15.62:0/1053024439 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248784074 0 0) 0x7f3a080cad20 con 0x7f3a0806dca0 2024-03-23T11:28:01.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.943+0000 7f3a0ce00640 1 -- 172.21.15.62:0/1053024439 >> v1:172.21.15.62:6789/0 conn(0x7f3a0806dca0 legacy=0x7f3a080cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.943+0000 7f3a0ce00640 1 -- 172.21.15.62:0/1053024439 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0815a8a0 con 0x7f3a0806c300 2024-03-23T11:28:01.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.944+0000 7f3a0ce00640 1 --2- 172.21.15.62:0/1053024439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a0806c300 0x7f3a080cb490 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009e50 tx=0x7f3a0002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a5bc4e9346ebbf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.945+0000 7f3a07400640 1 -- 172.21.15.62:0/1053024439 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0002ca70 con 0x7f3a0806c300 2024-03-23T11:28:01.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.945+0000 7f3a07400640 1 -- 172.21.15.62:0/1053024439 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0002cbd0 con 0x7f3a0806c300 2024-03-23T11:28:01.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.945+0000 7f3a07400640 1 -- 172.21.15.62:0/1053024439 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a000343f0 con 0x7f3a0806c300 2024-03-23T11:28:01.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.945+0000 7f3a0ee00640 1 -- 172.21.15.62:0/1053024439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a0806c300 msgr2=0x7f3a080cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.945+0000 7f3a0ee00640 1 --2- 172.21.15.62:0/1053024439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a0806c300 0x7f3a080cb490 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009e50 tx=0x7f3a0002bed0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.945+0000 7f3a0ee00640 1 -- 172.21.15.62:0/1053024439 shutdown_connections 2024-03-23T11:28:01.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.945+0000 7f3a0ee00640 1 --2- 172.21.15.62:0/1053024439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a0806c300 0x7f3a080cb490 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.945+0000 7f3a0ee00640 1 --2- 172.21.15.62:0/1053024439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a080cd7c0 0x7f3a080b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.945+0000 7f3a0ee00640 1 -- 172.21.15.62:0/1053024439 >> 172.21.15.62:0/1053024439 conn(0x7f3a080b02e0 msgr2=0x7f3a080b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:01.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.946+0000 7f3a0ee00640 1 -- 172.21.15.62:0/1053024439 shutdown_connections 2024-03-23T11:28:01.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.946+0000 7f3a0ee00640 1 -- 172.21.15.62:0/1053024439 wait complete. 2024-03-23T11:28:01.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.947+0000 7f3a0ee00640 1 Processor -- start 2024-03-23T11:28:01.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.947+0000 7f3a0ee00640 1 -- start start 2024-03-23T11:28:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.948+0000 7f3a0ee00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a0806c300 0x7f3a08153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.948+0000 7f3a0ee00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a080cd7c0 0x7f3a080accf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.948+0000 7f3a0ee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a081567c0 0x7f3a080ad230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.948+0000 7f3a0ee00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08165b50 con 0x7f3a0806c300 2024-03-23T11:28:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.948+0000 7f3a0ee00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081665e0 con 0x7f3a080cd7c0 2024-03-23T11:28:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.948+0000 7f3a0ee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081651b0 con 0x7f3a081567c0 2024-03-23T11:28:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.949+0000 7f3a07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a0806c300 0x7f3a08153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.949+0000 7f3a07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a0806c300 0x7f3a08153ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54068/0 (socket says 172.21.15.62:54068) 2024-03-23T11:28:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.949+0000 7f3a07e00640 1 -- 172.21.15.62:0/2048239613 learned_addr learned my addr 172.21.15.62:0/2048239613 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:01.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.949+0000 7f3a07e00640 1 -- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a081567c0 msgr2=0x7f3a080ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.949+0000 7f3a07e00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a081567c0 0x7f3a080ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.949+0000 7f3a07e00640 1 -- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a080cd7c0 msgr2=0x7f3a080accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:01.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.949+0000 7f3a07e00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a080cd7c0 0x7f3a080accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:01.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.949+0000 7f3a07e00640 1 -- 172.21.15.62:0/2048239613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a00009b30 con 0x7f3a0806c300 2024-03-23T11:28:01.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.949+0000 7f3a07e00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a0806c300 0x7f3a08153ec0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f39f800e870 tx=0x7f39f800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.950+0000 7f3a04c00640 1 -- 172.21.15.62:0/2048239613 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f80094f0 con 0x7f3a0806c300 2024-03-23T11:28:01.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.950+0000 7f3a04c00640 1 -- 172.21.15.62:0/2048239613 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f800c940 con 0x7f3a0806c300 2024-03-23T11:28:01.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.950+0000 7f3a04c00640 1 -- 172.21.15.62:0/2048239613 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f800ccf0 con 0x7f3a0806c300 2024-03-23T11:28:01.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.950+0000 7f3a0ee00640 1 -- 172.21.15.62:0/2048239613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a08165490 con 0x7f3a0806c300 2024-03-23T11:28:01.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.950+0000 7f3a0ee00640 1 -- 172.21.15.62:0/2048239613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a081662f0 con 0x7f3a0806c300 2024-03-23T11:28:01.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.951+0000 7f3a0ee00640 1 -- 172.21.15.62:0/2048239613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a08066c90 con 0x7f3a0806c300 2024-03-23T11:28:01.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.952+0000 7f3a04c00640 1 -- 172.21.15.62:0/2048239613 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f39f8013470 con 0x7f3a0806c300 2024-03-23T11:28:01.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.953+0000 7f3a04c00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39e406a7a0 0x7f39e406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:01.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.953+0000 7f3a04c00640 1 -- 172.21.15.62:0/2048239613 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f39f8088ef0 con 0x7f3a0806c300 2024-03-23T11:28:01.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.954+0000 7f39ffe00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39e406a7a0 0x7f39e406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:01.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.955+0000 7f39ffe00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39e406a7a0 0x7f39e406cc60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3a0806d3c0 tx=0x7f39f0007910 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:01.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:01.957+0000 7f3a04c00640 1 -- 172.21.15.62:0/2048239613 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39f80542a0 con 0x7f3a0806c300 2024-03-23T11:28:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.154+0000 7f3a0ee00640 1 -- 172.21.15.62:0/2048239613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a081563f0 con 0x7f3a0806c300 2024-03-23T11:28:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.155+0000 7f3a04c00640 1 -- 172.21.15.62:0/2048239613 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f39f80578c0 con 0x7f3a0806c300 2024-03-23T11:28:02.147 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:28:02.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.157+0000 7f3a0ee00640 1 -- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39e406a7a0 msgr2=0x7f39e406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:02.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.157+0000 7f3a0ee00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39e406a7a0 0x7f39e406cc60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3a0806d3c0 tx=0x7f39f0007910 comp rx=0 tx=0).stop 2024-03-23T11:28:02.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.157+0000 7f3a0ee00640 1 -- 172.21.15.62:0/2048239613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a0806c300 msgr2=0x7f3a08153ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:02.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.157+0000 7f3a0ee00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a0806c300 0x7f3a08153ec0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f39f800e870 tx=0x7f39f800ed30 comp rx=0 tx=0).stop 2024-03-23T11:28:02.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.157+0000 7f3a0ee00640 1 -- 172.21.15.62:0/2048239613 shutdown_connections 2024-03-23T11:28:02.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.157+0000 7f3a0ee00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39e406a7a0 0x7f39e406cc60 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.157+0000 7f3a0ee00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a081567c0 0x7f3a080ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.157+0000 7f3a0ee00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a080cd7c0 0x7f3a080accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.157+0000 7f3a0ee00640 1 --2- 172.21.15.62:0/2048239613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a0806c300 0x7f3a08153ec0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.157+0000 7f3a0ee00640 1 -- 172.21.15.62:0/2048239613 >> 172.21.15.62:0/2048239613 conn(0x7f3a080b02e0 msgr2=0x7f3a0815b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:02.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.158+0000 7f3a0ee00640 1 -- 172.21.15.62:0/2048239613 shutdown_connections 2024-03-23T11:28:02.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.158+0000 7f3a0ee00640 1 -- 172.21.15.62:0/2048239613 wait complete. 2024-03-23T11:28:02.168 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:28:02.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.399+0000 7f400e200640 1 Processor -- start 2024-03-23T11:28:02.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.400+0000 7f400e200640 1 -- start start 2024-03-23T11:28:02.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.401+0000 7f400e200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f400815c9e0 0x7f40081607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.401+0000 7f400e200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f40080cf650 0x7f40080cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.401+0000 7f400e200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400806bfc0 con 0x7f400815cdc0 2024-03-23T11:28:02.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.401+0000 7f400e200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40080d0000 con 0x7f400815c9e0 2024-03-23T11:28:02.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.401+0000 7f400e200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40080d2630 con 0x7f40080cf650 2024-03-23T11:28:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.402+0000 7f4007400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f400815c9e0 0x7f40081607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.402+0000 7f4007400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f400815c9e0 0x7f40081607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54076/0 (socket says 172.21.15.62:54076) 2024-03-23T11:28:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.402+0000 7f4007400640 1 -- 172.21.15.62:0/868292533 learned_addr learned my addr 172.21.15.62:0/868292533 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.402+0000 7f4007e00640 1 --2- 172.21.15.62:0/868292533 >> v2:172.21.15.120:3300/0 conn(0x7f40080cf650 0x7f40080cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.402+0000 7f4007400640 1 -- 172.21.15.62:0/868292533 >> v2:172.21.15.120:3300/0 conn(0x7f40080cf650 msgr2=0x7f40080cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.402+0000 7f4007400640 1 --2- 172.21.15.62:0/868292533 >> v2:172.21.15.120:3300/0 conn(0x7f40080cf650 0x7f40080cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.402+0000 7f4007400640 1 -- 172.21.15.62:0/868292533 >> v1:172.21.15.120:6789/0 conn(0x7f400815cdc0 legacy=0x7f40080d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.402+0000 7f4007400640 1 -- 172.21.15.62:0/868292533 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40080b5260 con 0x7f400815c9e0 2024-03-23T11:28:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.404+0000 7f4007400640 1 --2- 172.21.15.62:0/868292533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f400815c9e0 0x7f40081607c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f400806b570 tx=0x7f3ff802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9bd79d926225116 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.404+0000 7f4006a00640 1 -- 172.21.15.62:0/868292533 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8031870 con 0x7f400815c9e0 2024-03-23T11:28:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.404+0000 7f4006a00640 1 -- 172.21.15.62:0/868292533 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff80319d0 con 0x7f400815c9e0 2024-03-23T11:28:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.404+0000 7f4006a00640 1 -- 172.21.15.62:0/868292533 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8031d60 con 0x7f400815c9e0 2024-03-23T11:28:02.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.405+0000 7f400e200640 1 -- 172.21.15.62:0/868292533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f400815c9e0 msgr2=0x7f40081607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:02.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.405+0000 7f400e200640 1 --2- 172.21.15.62:0/868292533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f400815c9e0 0x7f40081607c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f400806b570 tx=0x7f3ff802b910 comp rx=0 tx=0).stop 2024-03-23T11:28:02.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.405+0000 7f400e200640 1 -- 172.21.15.62:0/868292533 shutdown_connections 2024-03-23T11:28:02.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.405+0000 7f400e200640 1 --2- 172.21.15.62:0/868292533 >> v2:172.21.15.120:3300/0 conn(0x7f40080cf650 0x7f40080cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.405+0000 7f400e200640 1 --2- 172.21.15.62:0/868292533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f400815c9e0 0x7f40081607c0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.405+0000 7f400e200640 1 -- 172.21.15.62:0/868292533 >> 172.21.15.62:0/868292533 conn(0x7f4008071570 msgr2=0x7f4008071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:02.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.405+0000 7f400e200640 1 -- 172.21.15.62:0/868292533 shutdown_connections 2024-03-23T11:28:02.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.405+0000 7f400e200640 1 -- 172.21.15.62:0/868292533 wait complete. 2024-03-23T11:28:02.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.407+0000 7f400e200640 1 Processor -- start 2024-03-23T11:28:02.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.408+0000 7f400e200640 1 -- start start 2024-03-23T11:28:02.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.408+0000 7f400e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40080cf650 0x7f40080c6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f400e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f400815c9e0 0x7f4008157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f400e200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f400815cdc0 0x7f40080b6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f400e200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40080ca990 con 0x7f40080cf650 2024-03-23T11:28:02.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f400e200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40080c9d80 con 0x7f400815cdc0 2024-03-23T11:28:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f400e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40080ca060 con 0x7f400815c9e0 2024-03-23T11:28:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f4007400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40080cf650 0x7f40080c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f4007400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40080cf650 0x7f40080c6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54078/0 (socket says 172.21.15.62:54078) 2024-03-23T11:28:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f4007400640 1 -- 172.21.15.62:0/3534319746 learned_addr learned my addr 172.21.15.62:0/3534319746 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f4007400640 1 -- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f400815c9e0 msgr2=0x7f4008157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f4007e00640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f400815cdc0 0x7f40080b6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f3fffe00640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f400815c9e0 0x7f4008157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f4007400640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f400815c9e0 0x7f4008157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f4007400640 1 -- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f400815cdc0 msgr2=0x7f40080b6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f4007400640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f400815cdc0 0x7f40080b6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.409+0000 7f4007400640 1 -- 172.21.15.62:0/3534319746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff80094d0 con 0x7f40080cf650 2024-03-23T11:28:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.410+0000 7f4007400640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40080cf650 0x7f40080c6810 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8000c00 tx=0x7f3ff80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.410+0000 7f3fff400640 1 -- 172.21.15.62:0/3534319746 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff803abe0 con 0x7f40080cf650 2024-03-23T11:28:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.410+0000 7f3fff400640 1 -- 172.21.15.62:0/3534319746 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff803ad40 con 0x7f40080cf650 2024-03-23T11:28:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.410+0000 7f3fff400640 1 -- 172.21.15.62:0/3534319746 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8044610 con 0x7f40080cf650 2024-03-23T11:28:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.410+0000 7f400e200640 1 -- 172.21.15.62:0/3534319746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40080ca340 con 0x7f40080cf650 2024-03-23T11:28:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.410+0000 7f400e200640 1 -- 172.21.15.62:0/3534319746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40080caea0 con 0x7f40080cf650 2024-03-23T11:28:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.411+0000 7f400e200640 1 -- 172.21.15.62:0/3534319746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40080c8090 con 0x7f40080cf650 2024-03-23T11:28:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.412+0000 7f3fff400640 1 -- 172.21.15.62:0/3534319746 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3ff804d450 con 0x7f40080cf650 2024-03-23T11:28:02.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.413+0000 7f3fff400640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe406a7a0 0x7f3fe406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.413+0000 7f3fff400640 1 -- 172.21.15.62:0/3534319746 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f3ff802d040 con 0x7f40080cf650 2024-03-23T11:28:02.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.414+0000 7f3fffe00640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe406a7a0 0x7f3fe406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.415+0000 7f3fffe00640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe406a7a0 0x7f3fe406cc60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f400815b820 tx=0x7f3ff0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:02.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.418+0000 7f3fff400640 1 -- 172.21.15.62:0/3534319746 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ff8076870 con 0x7f40080cf650 2024-03-23T11:28:02.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.614+0000 7f400e200640 1 -- 172.21.15.62:0/3534319746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f40080b4320 con 0x7f40080cf650 2024-03-23T11:28:02.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.615+0000 7f3fff400640 1 -- 172.21.15.62:0/3534319746 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ff8079e90 con 0x7f40080cf650 2024-03-23T11:28:02.607 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:28:02.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.617+0000 7f400e200640 1 -- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe406a7a0 msgr2=0x7f3fe406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.617+0000 7f400e200640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe406a7a0 0x7f3fe406cc60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f400815b820 tx=0x7f3ff0008040 comp rx=0 tx=0).stop 2024-03-23T11:28:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.617+0000 7f400e200640 1 -- 172.21.15.62:0/3534319746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40080cf650 msgr2=0x7f40080c6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.617+0000 7f400e200640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40080cf650 0x7f40080c6810 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8000c00 tx=0x7f3ff80046c0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.617+0000 7f400e200640 1 -- 172.21.15.62:0/3534319746 shutdown_connections 2024-03-23T11:28:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.617+0000 7f400e200640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3fe406a7a0 0x7f3fe406cc60 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.617+0000 7f400e200640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f400815cdc0 0x7f40080b6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.617+0000 7f400e200640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f400815c9e0 0x7f4008157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.617+0000 7f400e200640 1 --2- 172.21.15.62:0/3534319746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40080cf650 0x7f40080c6810 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.617+0000 7f400e200640 1 -- 172.21.15.62:0/3534319746 >> 172.21.15.62:0/3534319746 conn(0x7f4008071570 msgr2=0x7f400815fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.617+0000 7f400e200640 1 -- 172.21.15.62:0/3534319746 shutdown_connections 2024-03-23T11:28:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.618+0000 7f400e200640 1 -- 172.21.15.62:0/3534319746 wait complete. 2024-03-23T11:28:02.629 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:28:02.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.865+0000 7f9db3e00640 1 Processor -- start 2024-03-23T11:28:02.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.866+0000 7f9db3e00640 1 -- start start 2024-03-23T11:28:02.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.867+0000 7f9db3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db4070090 0x7f9db40704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.867+0000 7f9db3e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9db40b2820 0x7f9db40b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.867+0000 7f9db3e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db4070a30 con 0x7f9db4143dd0 2024-03-23T11:28:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.867+0000 7f9db3e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db40cd140 con 0x7f9db40b2820 2024-03-23T11:28:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.867+0000 7f9db3e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db40b3180 con 0x7f9db4070090 2024-03-23T11:28:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.867+0000 7f9db2a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9db4143dd0 0x7f9db415abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:52058/0 (socket says 172.21.15.62:52058) 2024-03-23T11:28:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.867+0000 7f9db2a00640 1 -- 172.21.15.62:0/4064418405 learned_addr learned my addr 172.21.15.62:0/4064418405 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:02.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415796021 0 0) 0x7f9db4070a30 con 0x7f9db4143dd0 2024-03-23T11:28:02.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d98003520 con 0x7f9db4143dd0 2024-03-23T11:28:02.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9dabe00640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db4070090 0x7f9db40704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db3400640 1 --2- 172.21.15.62:0/4064418405 >> v2:172.21.15.120:3300/0 conn(0x7f9db40b2820 0x7f9db40b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2682662271 0 0) 0x7f9d98003520 con 0x7f9db4143dd0 2024-03-23T11:28:02.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db4070090 msgr2=0x7f9db40704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:02.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db4070090 0x7f9db40704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 >> v2:172.21.15.120:3300/0 conn(0x7f9db40b2820 msgr2=0x7f9db40b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:02.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 --2- 172.21.15.62:0/4064418405 >> v2:172.21.15.120:3300/0 conn(0x7f9db40b2820 0x7f9db40b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db40cd3e0 con 0x7f9db4143dd0 2024-03-23T11:28:02.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9da4001670 con 0x7f9db4143dd0 2024-03-23T11:28:02.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 >> v1:172.21.15.62:6789/0 conn(0x7f9db4143dd0 legacy=0x7f9db415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:02.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db4070090 0x7f9d98003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d98003fa0 0x7f9d980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 0x7f9d98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db40cd140 con 0x7f9d98004d00 2024-03-23T11:28:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db40b3180 con 0x7f9d98003fa0 2024-03-23T11:28:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9800ba80 con 0x7f9db4070090 2024-03-23T11:28:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.868+0000 7f9db3400640 1 --2- 172.21.15.62:0/4064418405 >> v2:172.21.15.120:3300/0 conn(0x7f9db40b2820 0x7f9db40b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:28:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.869+0000 7f9db3400640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 0x7f9d98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.869+0000 7f9db2a00640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db4070090 0x7f9d98003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.869+0000 7f9dabe00640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d98003fa0 0x7f9d980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.869+0000 7f9db3400640 1 -- 172.21.15.62:0/4064418405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db4070090 msgr2=0x7f9d98003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.869+0000 7f9db3400640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db4070090 0x7f9d98003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.869+0000 7f9db3400640 1 -- 172.21.15.62:0/4064418405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d98003fa0 msgr2=0x7f9d980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.869+0000 7f9db3400640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d98003fa0 0x7f9d980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.869+0000 7f9db3400640 1 -- 172.21.15.62:0/4064418405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d98003520 con 0x7f9d98004d00 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.870+0000 7f9db3400640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 0x7f9d98005180 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f9dac0073e0 tx=0x7f9d980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dec600a3b79934b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.870+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac02cdc0 con 0x7f9d98004d00 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.870+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dac034440 con 0x7f9d98004d00 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.870+0000 7f9db2000640 1 -- 172.21.15.62:0/4064418405 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac0347b0 con 0x7f9d98004d00 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.871+0000 7f9db3e00640 1 -- 172.21.15.62:0/4064418405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 msgr2=0x7f9d98005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.871+0000 7f9db3e00640 1 --2- 172.21.15.62:0/4064418405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 0x7f9d98005180 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f9dac0073e0 tx=0x7f9d980057f0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.871+0000 7f9db3400640 1 -- 172.21.15.62:0/4064418405 reap_dead start 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.871+0000 7f9db3e00640 1 -- 172.21.15.62:0/4064418405 shutdown_connections 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.871+0000 7f9db3e00640 1 -- 172.21.15.62:0/4064418405 >> 172.21.15.62:0/4064418405 conn(0x7f9db4066050 msgr2=0x7f9db4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.871+0000 7f9db3e00640 1 -- 172.21.15.62:0/4064418405 shutdown_connections 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.871+0000 7f9db3e00640 1 -- 172.21.15.62:0/4064418405 wait complete. 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.873+0000 7f9db3e00640 1 Processor -- start 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.873+0000 7f9db3e00640 1 -- start start 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.874+0000 7f9db3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 0x7f9db406dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.874+0000 7f9db3e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db406cdc0 0x7f9db4157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.874+0000 7f9db3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db40ae0c0 0x7f9db40ae540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.874+0000 7f9db3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db40aea80 con 0x7f9d98004d00 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.874+0000 7f9db3e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db406c090 con 0x7f9db406cdc0 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.874+0000 7f9db3e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db4139c90 con 0x7f9db40ae0c0 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.874+0000 7f9db2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 0x7f9db406dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.875+0000 7f9db2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 0x7f9db406dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54104/0 (socket says 172.21.15.62:54104) 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.875+0000 7f9db2a00640 1 -- 172.21.15.62:0/984804952 learned_addr learned my addr 172.21.15.62:0/984804952 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.875+0000 7f9db3400640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db40ae0c0 0x7f9db40ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.875+0000 7f9dabe00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db406cdc0 0x7f9db4157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.875+0000 7f9db2a00640 1 -- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db40ae0c0 msgr2=0x7f9db40ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.875+0000 7f9db2a00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db40ae0c0 0x7f9db40ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.875+0000 7f9db2a00640 1 -- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db406cdc0 msgr2=0x7f9db4157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.875+0000 7f9db2a00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db406cdc0 0x7f9db4157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.875+0000 7f9db2a00640 1 -- 172.21.15.62:0/984804952 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db406c2e0 con 0x7f9d98004d00 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.875+0000 7f9dabe00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db406cdc0 0x7f9db4157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.875+0000 7f9db2a00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 0x7f9db406dcc0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f9da400e8a0 tx=0x7f9da400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.876+0000 7f9daaa00640 1 -- 172.21.15.62:0/984804952 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da400c870 con 0x7f9d98004d00 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.876+0000 7f9daaa00640 1 -- 172.21.15.62:0/984804952 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da400c9f0 con 0x7f9d98004d00 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.876+0000 7f9daaa00640 1 -- 172.21.15.62:0/984804952 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da400cda0 con 0x7f9d98004d00 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.876+0000 7f9db3e00640 1 -- 172.21.15.62:0/984804952 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db406c5c0 con 0x7f9d98004d00 2024-03-23T11:28:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.876+0000 7f9db3e00640 1 -- 172.21.15.62:0/984804952 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db41576c0 con 0x7f9d98004d00 2024-03-23T11:28:02.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.877+0000 7f9db3e00640 1 -- 172.21.15.62:0/984804952 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db406cae0 con 0x7f9d98004d00 2024-03-23T11:28:02.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.878+0000 7f9daaa00640 1 -- 172.21.15.62:0/984804952 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9da401d460 con 0x7f9d98004d00 2024-03-23T11:28:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.879+0000 7f9daaa00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d9006a7d0 0x7f9d9006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.879+0000 7f9daaa00640 1 -- 172.21.15.62:0/984804952 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f9da4088f40 con 0x7f9d98004d00 2024-03-23T11:28:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.879+0000 7f9dabe00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d9006a7d0 0x7f9d9006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.881+0000 7f9dabe00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d9006a7d0 0x7f9d9006cc90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9db40ac2b0 tx=0x7f9d9c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:02.883+0000 7f9daaa00640 1 -- 172.21.15.62:0/984804952 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9db406cae0 con 0x7f9d98004d00 2024-03-23T11:28:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.080+0000 7f9db3e00640 1 -- 172.21.15.62:0/984804952 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9db4066bb0 con 0x7f9d98004d00 2024-03-23T11:28:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.081+0000 7f9daaa00640 1 -- 172.21.15.62:0/984804952 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9da40541e0 con 0x7f9d98004d00 2024-03-23T11:28:03.073 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:28:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 -- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d9006a7d0 msgr2=0x7f9d9006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d9006a7d0 0x7f9d9006cc90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9db40ac2b0 tx=0x7f9d9c008040 comp rx=0 tx=0).stop 2024-03-23T11:28:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 -- 172.21.15.62:0/984804952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 msgr2=0x7f9db406dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 0x7f9db406dcc0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f9da400e8a0 tx=0x7f9da400ed60 comp rx=0 tx=0).stop 2024-03-23T11:28:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 -- 172.21.15.62:0/984804952 shutdown_connections 2024-03-23T11:28:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9d9006a7d0 0x7f9d9006cc90 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9db40ae0c0 0x7f9db40ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db406cdc0 0x7f9db4157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 --2- 172.21.15.62:0/984804952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d98004d00 0x7f9db406dcc0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 -- 172.21.15.62:0/984804952 >> 172.21.15.62:0/984804952 conn(0x7f9db4066050 msgr2=0x7f9db40b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 -- 172.21.15.62:0/984804952 shutdown_connections 2024-03-23T11:28:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.084+0000 7f9db3e00640 1 -- 172.21.15.62:0/984804952 wait complete. 2024-03-23T11:28:03.094 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:28:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.318+0000 7ff353400640 1 Processor -- start 2024-03-23T11:28:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.319+0000 7ff353400640 1 -- start start 2024-03-23T11:28:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.320+0000 7ff353400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff34c06c300 0x7ff34c0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.320+0000 7ff353400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff34c06c6e0 0x7ff34c06cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.320+0000 7ff353400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34c0cbc60 con 0x7ff34c0cafa0 2024-03-23T11:28:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.320+0000 7ff353400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34c0cbf40 con 0x7ff34c06c6e0 2024-03-23T11:28:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.320+0000 7ff353400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34c0cc220 con 0x7ff34c06c300 2024-03-23T11:28:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.321+0000 7ff351400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff34c06c6e0 0x7ff34c06cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.321+0000 7ff351400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff34c06c6e0 0x7ff34c06cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54116/0 (socket says 172.21.15.62:54116) 2024-03-23T11:28:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.321+0000 7ff351400640 1 -- 172.21.15.62:0/1364766633 learned_addr learned my addr 172.21.15.62:0/1364766633 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.321+0000 7ff350a00640 1 --2- 172.21.15.62:0/1364766633 >> v2:172.21.15.120:3300/0 conn(0x7ff34c06c300 0x7ff34c0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.321+0000 7ff351400640 1 -- 172.21.15.62:0/1364766633 >> v2:172.21.15.120:3300/0 conn(0x7ff34c06c300 msgr2=0x7ff34c0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.321+0000 7ff351400640 1 --2- 172.21.15.62:0/1364766633 >> v2:172.21.15.120:3300/0 conn(0x7ff34c06c300 0x7ff34c0ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.321+0000 7ff351400640 1 -- 172.21.15.62:0/1364766633 >> v1:172.21.15.120:6789/0 conn(0x7ff34c0cafa0 legacy=0x7ff34c0cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.321+0000 7ff351400640 1 -- 172.21.15.62:0/1364766633 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c0cc500 con 0x7ff34c06c6e0 2024-03-23T11:28:03.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.322+0000 7ff351400640 1 --2- 172.21.15.62:0/1364766633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff34c06c6e0 0x7ff34c06cb30 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ff34c06c1b0 tx=0x7ff34402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267406d6fd0aa9e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:03.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.323+0000 7ff34b400640 1 -- 172.21.15.62:0/1364766633 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34402ca80 con 0x7ff34c06c6e0 2024-03-23T11:28:03.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.323+0000 7ff34b400640 1 -- 172.21.15.62:0/1364766633 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34402cbe0 con 0x7ff34c06c6e0 2024-03-23T11:28:03.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.323+0000 7ff34b400640 1 -- 172.21.15.62:0/1364766633 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3440343f0 con 0x7ff34c06c6e0 2024-03-23T11:28:03.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.323+0000 7ff353400640 1 -- 172.21.15.62:0/1364766633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff34c06c6e0 msgr2=0x7ff34c06cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:03.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.323+0000 7ff353400640 1 --2- 172.21.15.62:0/1364766633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff34c06c6e0 0x7ff34c06cb30 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ff34c06c1b0 tx=0x7ff34402bee0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.323+0000 7ff353400640 1 -- 172.21.15.62:0/1364766633 shutdown_connections 2024-03-23T11:28:03.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.323+0000 7ff353400640 1 --2- 172.21.15.62:0/1364766633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff34c06c6e0 0x7ff34c06cb30 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.323+0000 7ff353400640 1 --2- 172.21.15.62:0/1364766633 >> v2:172.21.15.120:3300/0 conn(0x7ff34c06c300 0x7ff34c0ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.323+0000 7ff353400640 1 -- 172.21.15.62:0/1364766633 >> 172.21.15.62:0/1364766633 conn(0x7ff34c0b02e0 msgr2=0x7ff34c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:03.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.324+0000 7ff353400640 1 -- 172.21.15.62:0/1364766633 shutdown_connections 2024-03-23T11:28:03.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.324+0000 7ff353400640 1 -- 172.21.15.62:0/1364766633 wait complete. 2024-03-23T11:28:03.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.325+0000 7ff353400640 1 Processor -- start 2024-03-23T11:28:03.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.326+0000 7ff353400640 1 -- start start 2024-03-23T11:28:03.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.326+0000 7ff353400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff34c06c300 0x7ff34c1f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.326+0000 7ff353400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff34c06c6e0 0x7ff34c13ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff353400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff34c0cafa0 0x7ff34c13e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff353400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c15c2f0 con 0x7ff34c0cafa0 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff353400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0acaa0 con 0x7ff34c06c6e0 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff353400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0acd80 con 0x7ff34c06c300 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff351400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff34c0cafa0 0x7ff34c13e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff351400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff34c0cafa0 0x7ff34c13e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54118/0 (socket says 172.21.15.62:54118) 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff351400640 1 -- 172.21.15.62:0/1917579307 learned_addr learned my addr 172.21.15.62:0/1917579307 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff350a00640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff34c06c300 0x7ff34c1f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff34be00640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff34c06c6e0 0x7ff34c13ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff351400640 1 -- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff34c06c300 msgr2=0x7ff34c1f1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff351400640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff34c06c300 0x7ff34c1f1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff351400640 1 -- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff34c06c6e0 msgr2=0x7ff34c13ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff351400640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff34c06c6e0 0x7ff34c13ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.327+0000 7ff351400640 1 -- 172.21.15.62:0/1917579307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff344009b30 con 0x7ff34c0cafa0 2024-03-23T11:28:03.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.328+0000 7ff351400640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff34c0cafa0 0x7ff34c13e2f0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff344036040 tx=0x7ff344004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:03.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.328+0000 7ff348c00640 1 -- 172.21.15.62:0/1917579307 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344009510 con 0x7ff34c0cafa0 2024-03-23T11:28:03.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.328+0000 7ff348c00640 1 -- 172.21.15.62:0/1917579307 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34403e450 con 0x7ff34c0cafa0 2024-03-23T11:28:03.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.328+0000 7ff348c00640 1 -- 172.21.15.62:0/1917579307 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34403e7e0 con 0x7ff34c0cafa0 2024-03-23T11:28:03.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.328+0000 7ff353400640 1 -- 172.21.15.62:0/1917579307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34c0ad060 con 0x7ff34c0cafa0 2024-03-23T11:28:03.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.328+0000 7ff353400640 1 -- 172.21.15.62:0/1917579307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34c1f1fa0 con 0x7ff34c0cafa0 2024-03-23T11:28:03.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.329+0000 7ff353400640 1 -- 172.21.15.62:0/1917579307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34c066c90 con 0x7ff34c0cafa0 2024-03-23T11:28:03.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.330+0000 7ff348c00640 1 -- 172.21.15.62:0/1917579307 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff34402cd50 con 0x7ff34c0cafa0 2024-03-23T11:28:03.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.332+0000 7ff348c00640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff32c06a7a0 0x7ff32c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.332+0000 7ff348c00640 1 -- 172.21.15.62:0/1917579307 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7ff3440ab9f0 con 0x7ff34c0cafa0 2024-03-23T11:28:03.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.332+0000 7ff350a00640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff32c06a7a0 0x7ff32c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.334+0000 7ff350a00640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff32c06a7a0 0x7ff32c06cc60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff340006fd0 tx=0x7ff340008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:03.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.336+0000 7ff348c00640 1 -- 172.21.15.62:0/1917579307 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff344076d20 con 0x7ff34c0cafa0 2024-03-23T11:28:03.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.552+0000 7ff353400640 1 -- 172.21.15.62:0/1917579307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff34c13f450 con 0x7ff34c0cafa0 2024-03-23T11:28:03.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.553+0000 7ff348c00640 1 -- 172.21.15.62:0/1917579307 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v92) v1 ==== 80+0+122 (secure 0 0 0) 0x7ff344043080 con 0x7ff34c0cafa0 2024-03-23T11:28:03.545 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6833/3056962984 2024-03-24T11:26:47.481437+0000 2024-03-23T11:28:03.545 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6832/3056962984 2024-03-24T11:26:47.481437+0000 2024-03-23T11:28:03.545 INFO:teuthology.orchestra.run.smithi062.stderr:listed 2 entries 2024-03-23T11:28:03.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.555+0000 7ff353400640 1 -- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff32c06a7a0 msgr2=0x7ff32c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:03.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.555+0000 7ff353400640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff32c06a7a0 0x7ff32c06cc60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff340006fd0 tx=0x7ff340008040 comp rx=0 tx=0).stop 2024-03-23T11:28:03.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.556+0000 7ff353400640 1 -- 172.21.15.62:0/1917579307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff34c0cafa0 msgr2=0x7ff34c13e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:03.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.556+0000 7ff353400640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff34c0cafa0 0x7ff34c13e2f0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff344036040 tx=0x7ff344004a40 comp rx=0 tx=0).stop 2024-03-23T11:28:03.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.556+0000 7ff353400640 1 -- 172.21.15.62:0/1917579307 shutdown_connections 2024-03-23T11:28:03.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.556+0000 7ff353400640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff32c06a7a0 0x7ff32c06cc60 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.556+0000 7ff353400640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff34c0cafa0 0x7ff34c13e2f0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.556+0000 7ff353400640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff34c06c6e0 0x7ff34c13ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.556+0000 7ff353400640 1 --2- 172.21.15.62:0/1917579307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff34c06c300 0x7ff34c1f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.556+0000 7ff353400640 1 -- 172.21.15.62:0/1917579307 >> 172.21.15.62:0/1917579307 conn(0x7ff34c0b02e0 msgr2=0x7ff34c140780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.556+0000 7ff353400640 1 -- 172.21.15.62:0/1917579307 shutdown_connections 2024-03-23T11:28:03.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.556+0000 7ff353400640 1 -- 172.21.15.62:0/1917579307 wait complete. 2024-03-23T11:28:03.566 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:28:03.567 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:28:03.614 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:28:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.879+0000 7fad2b200640 1 Processor -- start 2024-03-23T11:28:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.879+0000 7fad2b200640 1 -- start start 2024-03-23T11:28:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.881+0000 7fad2b200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fad2c143dd0 0x7fad2c15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.881+0000 7fad2b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c0b2f30 0x7fad2c070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.881+0000 7fad2b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2c0cd140 con 0x7fad2c070090 2024-03-23T11:28:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.881+0000 7fad2b200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2c070a60 con 0x7fad2c143dd0 2024-03-23T11:28:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.881+0000 7fad2b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0cd3e0 con 0x7fad2c0b2f30 2024-03-23T11:28:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.881+0000 7fad29e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fad2c143dd0 0x7fad2c15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.881+0000 7fad29e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fad2c143dd0 0x7fad2c15abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54124/0 (socket says 172.21.15.62:54124) 2024-03-23T11:28:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.881+0000 7fad29e00640 1 -- 172.21.15.62:0/1748124118 learned_addr learned my addr 172.21.15.62:0/1748124118 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.881+0000 7fad2a800640 1 --2- 172.21.15.62:0/1748124118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c0b2f30 0x7fad2c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.882+0000 7fad29e00640 1 -- 172.21.15.62:0/1748124118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c0b2f30 msgr2=0x7fad2c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.882+0000 7fad29e00640 1 --2- 172.21.15.62:0/1748124118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c0b2f30 0x7fad2c070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.882+0000 7fad29e00640 1 -- 172.21.15.62:0/1748124118 >> v1:172.21.15.120:6789/0 conn(0x7fad2c070090 legacy=0x7fad2c0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.882+0000 7fad29e00640 1 -- 172.21.15.62:0/1748124118 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c0cd6c0 con 0x7fad2c143dd0 2024-03-23T11:28:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.882+0000 7fad2a800640 1 --2- 172.21.15.62:0/1748124118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c0b2f30 0x7fad2c070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.883+0000 7fad29e00640 1 --2- 172.21.15.62:0/1748124118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c143dd0 0x7fad2c15abe0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009870 tx=0x7fad1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456d0b68de5d1384 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.883+0000 7fad28a00640 1 -- 172.21.15.62:0/1748124118 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c031640 con 0x7fad2c143dd0 2024-03-23T11:28:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.883+0000 7fad28a00640 1 -- 172.21.15.62:0/1748124118 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c0317a0 con 0x7fad2c143dd0 2024-03-23T11:28:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.883+0000 7fad28a00640 1 -- 172.21.15.62:0/1748124118 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c031b30 con 0x7fad2c143dd0 2024-03-23T11:28:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.884+0000 7fad2b200640 1 -- 172.21.15.62:0/1748124118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c143dd0 msgr2=0x7fad2c15abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.884+0000 7fad2b200640 1 --2- 172.21.15.62:0/1748124118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c143dd0 0x7fad2c15abe0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009870 tx=0x7fad1c02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.884+0000 7fad2b200640 1 -- 172.21.15.62:0/1748124118 shutdown_connections 2024-03-23T11:28:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.884+0000 7fad2b200640 1 --2- 172.21.15.62:0/1748124118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c0b2f30 0x7fad2c070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.884+0000 7fad2b200640 1 --2- 172.21.15.62:0/1748124118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c143dd0 0x7fad2c15abe0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.884+0000 7fad2b200640 1 -- 172.21.15.62:0/1748124118 >> 172.21.15.62:0/1748124118 conn(0x7fad2c066050 msgr2=0x7fad2c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.884+0000 7fad2b200640 1 -- 172.21.15.62:0/1748124118 shutdown_connections 2024-03-23T11:28:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.884+0000 7fad2b200640 1 -- 172.21.15.62:0/1748124118 wait complete. 2024-03-23T11:28:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.886+0000 7fad2b200640 1 Processor -- start 2024-03-23T11:28:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.886+0000 7fad2b200640 1 -- start start 2024-03-23T11:28:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad2b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c070090 0x7fad2c14cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad2b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c0b2f30 0x7fad2c0acce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad2b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c143dd0 0x7fad2c14ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad2b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c13dbe0 con 0x7fad2c070090 2024-03-23T11:28:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad2b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c13d5d0 con 0x7fad2c0b2f30 2024-03-23T11:28:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad2b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c14fc40 con 0x7fad2c143dd0 2024-03-23T11:28:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad29e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c070090 0x7fad2c14cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad29e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c070090 0x7fad2c14cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54138/0 (socket says 172.21.15.62:54138) 2024-03-23T11:28:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad29e00640 1 -- 172.21.15.62:0/4047290176 learned_addr learned my addr 172.21.15.62:0/4047290176 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad29e00640 1 -- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c143dd0 msgr2=0x7fad2c14ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad2a800640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c143dd0 0x7fad2c14ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.887+0000 7fad29400640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c0b2f30 0x7fad2c0acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.888+0000 7fad29e00640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c143dd0 0x7fad2c14ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.888+0000 7fad29e00640 1 -- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c0b2f30 msgr2=0x7fad2c0acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.888+0000 7fad29e00640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c0b2f30 0x7fad2c0acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.888+0000 7fad29e00640 1 -- 172.21.15.62:0/4047290176 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c0094d0 con 0x7fad2c070090 2024-03-23T11:28:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.888+0000 7fad29400640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c0b2f30 0x7fad2c0acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.888+0000 7fad29e00640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c070090 0x7fad2c14cc80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009870 tx=0x7fad1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.888+0000 7fad22000640 1 -- 172.21.15.62:0/4047290176 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c03ac60 con 0x7fad2c070090 2024-03-23T11:28:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.888+0000 7fad22000640 1 -- 172.21.15.62:0/4047290176 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c03ade0 con 0x7fad2c070090 2024-03-23T11:28:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.888+0000 7fad22000640 1 -- 172.21.15.62:0/4047290176 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c044630 con 0x7fad2c070090 2024-03-23T11:28:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.888+0000 7fad2b200640 1 -- 172.21.15.62:0/4047290176 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2c14f680 con 0x7fad2c070090 2024-03-23T11:28:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.888+0000 7fad2b200640 1 -- 172.21.15.62:0/4047290176 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c13d800 con 0x7fad2c070090 2024-03-23T11:28:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.890+0000 7fad2b200640 1 -- 172.21.15.62:0/4047290176 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c13dd20 con 0x7fad2c070090 2024-03-23T11:28:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.890+0000 7fad22000640 1 -- 172.21.15.62:0/4047290176 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fad1c02f050 con 0x7fad2c070090 2024-03-23T11:28:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.891+0000 7fad22000640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad0c06a7a0 0x7fad0c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.891+0000 7fad22000640 1 -- 172.21.15.62:0/4047290176 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fad1c0aae90 con 0x7fad2c070090 2024-03-23T11:28:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.892+0000 7fad29400640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad0c06a7a0 0x7fad0c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:03.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.893+0000 7fad29400640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad0c06a7a0 0x7fad0c06cc60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fad2c1f5a00 tx=0x7fad18009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:03.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:03.896+0000 7fad22000640 1 -- 172.21.15.62:0/4047290176 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fad2c13dd20 con 0x7fad2c070090 2024-03-23T11:28:04.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.094+0000 7fad2b200640 1 -- 172.21.15.62:0/4047290176 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad2c13dd20 con 0x7fad2c070090 2024-03-23T11:28:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.096+0000 7fad22000640 1 -- 172.21.15.62:0/4047290176 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16271 (secure 0 0 0) 0x7fad2c13dd20 con 0x7fad2c070090 2024-03-23T11:28:04.089 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:04.089 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":92,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:26:47.481481+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:26:22.794149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-23T11:26:24.150997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6833/3056962984":"2024-03-24T11:26:47.481437+0000","172.21.15.62:6832/3056962984":"2024-03-24T11:26:47.481437+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:28:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.099+0000 7fad2b200640 1 -- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad0c06a7a0 msgr2=0x7fad0c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.099+0000 7fad2b200640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad0c06a7a0 0x7fad0c06cc60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fad2c1f5a00 tx=0x7fad18009040 comp rx=0 tx=0).stop 2024-03-23T11:28:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.099+0000 7fad2b200640 1 -- 172.21.15.62:0/4047290176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c070090 msgr2=0x7fad2c14cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.099+0000 7fad2b200640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c070090 0x7fad2c14cc80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009870 tx=0x7fad1c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.099+0000 7fad2b200640 1 -- 172.21.15.62:0/4047290176 shutdown_connections 2024-03-23T11:28:04.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.099+0000 7fad2b200640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fad0c06a7a0 0x7fad0c06cc60 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.099+0000 7fad2b200640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad2c143dd0 0x7fad2c14ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.099+0000 7fad2b200640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad2c0b2f30 0x7fad2c0acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.099+0000 7fad2b200640 1 --2- 172.21.15.62:0/4047290176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad2c070090 0x7fad2c14cc80 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.099+0000 7fad2b200640 1 -- 172.21.15.62:0/4047290176 >> 172.21.15.62:0/4047290176 conn(0x7fad2c066050 msgr2=0x7fad2c15b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:04.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.100+0000 7fad2b200640 1 -- 172.21.15.62:0/4047290176 shutdown_connections 2024-03-23T11:28:04.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.100+0000 7fad2b200640 1 -- 172.21.15.62:0/4047290176 wait complete. 2024-03-23T11:28:04.111 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:26:22.794149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:26:24.150997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:28:04.112 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:28:04.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.340+0000 7f5e0f200640 1 Processor -- start 2024-03-23T11:28:04.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.340+0000 7f5e0f200640 1 -- start start 2024-03-23T11:28:04.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.341+0000 7f5e0f200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5e08160240 0x7f5e0806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.341+0000 7f5e0f200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0806b2b0 con 0x7f5e080995b0 2024-03-23T11:28:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.341+0000 7f5e0f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e080cf690 con 0x7f5e080989d0 2024-03-23T11:28:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.341+0000 7f5e0f200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e080cf490 con 0x7f5e08160240 2024-03-23T11:28:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.341+0000 7f5e07e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5e08160240 0x7f5e0806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.342+0000 7f5e07e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5e08160240 0x7f5e0806ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54146/0 (socket says 172.21.15.62:54146) 2024-03-23T11:28:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.342+0000 7f5e07400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5e080995b0 0x7f5e080ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:52070/0 (socket says 172.21.15.62:52070) 2024-03-23T11:28:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.342+0000 7f5e07e00640 1 -- 172.21.15.62:0/2469626740 learned_addr learned my addr 172.21.15.62:0/2469626740 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:04.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.342+0000 7f5e07e00640 1 -- 172.21.15.62:0/2469626740 >> v1:172.21.15.120:6789/0 conn(0x7f5e080989d0 legacy=0x7f5e080cc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:04.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.342+0000 7f5e07e00640 1 -- 172.21.15.62:0/2469626740 >> v1:172.21.15.62:6789/0 conn(0x7f5e080995b0 legacy=0x7f5e080ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.342+0000 7f5e06a00640 1 -- 172.21.15.62:0/2469626740 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275799861 0 0) 0x7f5e0806b2b0 con 0x7f5e080995b0 2024-03-23T11:28:04.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.342+0000 7f5e07e00640 1 -- 172.21.15.62:0/2469626740 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e080cb730 con 0x7f5e08160240 2024-03-23T11:28:04.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.343+0000 7f5e07e00640 1 --2- 172.21.15.62:0/2469626740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e08160240 0x7f5e0806ad70 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f5df8009870 tx=0x7f5df802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc43d1d01ffb2613 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:04.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.343+0000 7f5e06a00640 1 -- 172.21.15.62:0/2469626740 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8030860 con 0x7f5e08160240 2024-03-23T11:28:04.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.344+0000 7f5e06a00640 1 -- 172.21.15.62:0/2469626740 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df80309c0 con 0x7f5e08160240 2024-03-23T11:28:04.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.344+0000 7f5e06a00640 1 -- 172.21.15.62:0/2469626740 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8030d50 con 0x7f5e08160240 2024-03-23T11:28:04.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.344+0000 7f5e0f200640 1 -- 172.21.15.62:0/2469626740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e08160240 msgr2=0x7f5e0806ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.344+0000 7f5e0f200640 1 --2- 172.21.15.62:0/2469626740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e08160240 0x7f5e0806ad70 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f5df8009870 tx=0x7f5df802b9a0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.344+0000 7f5e0f200640 1 -- 172.21.15.62:0/2469626740 shutdown_connections 2024-03-23T11:28:04.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.344+0000 7f5e0f200640 1 --2- 172.21.15.62:0/2469626740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e08160240 0x7f5e0806ad70 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.344+0000 7f5e0f200640 1 -- 172.21.15.62:0/2469626740 >> 172.21.15.62:0/2469626740 conn(0x7f5e080b22c0 msgr2=0x7f5e080b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:04.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.344+0000 7f5e0f200640 1 -- 172.21.15.62:0/2469626740 shutdown_connections 2024-03-23T11:28:04.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.344+0000 7f5e0f200640 1 -- 172.21.15.62:0/2469626740 wait complete. 2024-03-23T11:28:04.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.346+0000 7f5e0f200640 1 Processor -- start 2024-03-23T11:28:04.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.346+0000 7f5e0f200640 1 -- start start 2024-03-23T11:28:04.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.347+0000 7f5e0f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e080989d0 0x7f5e080ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:04.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.347+0000 7f5e0f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e080995b0 0x7f5e081f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:04.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.347+0000 7f5e0f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08160240 0x7f5e080c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:04.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.347+0000 7f5e0f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e080c78a0 con 0x7f5e080989d0 2024-03-23T11:28:04.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.347+0000 7f5e0f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e080b5a70 con 0x7f5e08160240 2024-03-23T11:28:04.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.347+0000 7f5e0f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e080b5160 con 0x7f5e080995b0 2024-03-23T11:28:04.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.347+0000 7f5e07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e080989d0 0x7f5e080ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:04.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.347+0000 7f5e07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e080989d0 0x7f5e080ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54162/0 (socket says 172.21.15.62:54162) 2024-03-23T11:28:04.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.347+0000 7f5e07e00640 1 -- 172.21.15.62:0/2770108522 learned_addr learned my addr 172.21.15.62:0/2770108522 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:04.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5e07400640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e080995b0 0x7f5e081f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:04.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5e0d200640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08160240 0x7f5e080c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:04.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5e07e00640 1 -- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e080995b0 msgr2=0x7f5e081f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5e07e00640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e080995b0 0x7f5e081f1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5e07e00640 1 -- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08160240 msgr2=0x7f5e080c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5e07e00640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08160240 0x7f5e080c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5e07e00640 1 -- 172.21.15.62:0/2770108522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df80094d0 con 0x7f5e080989d0 2024-03-23T11:28:04.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5e07400640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e080995b0 0x7f5e081f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:04.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5e0d200640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08160240 0x7f5e080c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:04.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5e07e00640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e080989d0 0x7f5e080ca4f0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f5df8009870 tx=0x7f5df8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:04.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5dffe00640 1 -- 172.21.15.62:0/2770108522 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8030810 con 0x7f5e080989d0 2024-03-23T11:28:04.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5dffe00640 1 -- 172.21.15.62:0/2770108522 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df8043400 con 0x7f5e080989d0 2024-03-23T11:28:04.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.348+0000 7f5dffe00640 1 -- 172.21.15.62:0/2770108522 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df80437b0 con 0x7f5e080989d0 2024-03-23T11:28:04.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.349+0000 7f5e0f200640 1 -- 172.21.15.62:0/2770108522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e080b6150 con 0x7f5e080989d0 2024-03-23T11:28:04.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.349+0000 7f5e0f200640 1 -- 172.21.15.62:0/2770108522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e080b5420 con 0x7f5e080989d0 2024-03-23T11:28:04.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.350+0000 7f5e0f200640 1 -- 172.21.15.62:0/2770108522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e080738e0 con 0x7f5e080989d0 2024-03-23T11:28:04.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.351+0000 7f5dffe00640 1 -- 172.21.15.62:0/2770108522 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5df8043910 con 0x7f5e080989d0 2024-03-23T11:28:04.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.351+0000 7f5dffe00640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5de406a7d0 0x7f5de406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:04.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.352+0000 7f5dffe00640 1 -- 172.21.15.62:0/2770108522 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f5df803e070 con 0x7f5e080989d0 2024-03-23T11:28:04.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.352+0000 7f5e07400640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5de406a7d0 0x7f5de406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:04.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.353+0000 7f5e07400640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5de406a7d0 0x7f5de406cc90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5e0815f270 tx=0x7f5df0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:04.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.356+0000 7f5dffe00640 1 -- 172.21.15.62:0/2770108522 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5df8077060 con 0x7f5e080989d0 2024-03-23T11:28:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.554+0000 7f5e0f200640 1 -- 172.21.15.62:0/2770108522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5e08066c90 con 0x7f5e080989d0 2024-03-23T11:28:04.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.555+0000 7f5dffe00640 1 -- 172.21.15.62:0/2770108522 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5df807a680 con 0x7f5e080989d0 2024-03-23T11:28:04.547 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.558+0000 7f5e0f200640 1 -- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5de406a7d0 msgr2=0x7f5de406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.558+0000 7f5e0f200640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5de406a7d0 0x7f5de406cc90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5e0815f270 tx=0x7f5df0009040 comp rx=0 tx=0).stop 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.558+0000 7f5e0f200640 1 -- 172.21.15.62:0/2770108522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e080989d0 msgr2=0x7f5e080ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.558+0000 7f5e0f200640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e080989d0 0x7f5e080ca4f0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f5df8009870 tx=0x7f5df8004730 comp rx=0 tx=0).stop 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.558+0000 7f5e0f200640 1 -- 172.21.15.62:0/2770108522 shutdown_connections 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.558+0000 7f5e0f200640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5de406a7d0 0x7f5de406cc90 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.558+0000 7f5e0f200640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08160240 0x7f5e080c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.558+0000 7f5e0f200640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e080995b0 0x7f5e081f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.558+0000 7f5e0f200640 1 --2- 172.21.15.62:0/2770108522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e080989d0 0x7f5e080ca4f0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.558+0000 7f5e0f200640 1 -- 172.21.15.62:0/2770108522 >> 172.21.15.62:0/2770108522 conn(0x7f5e080b22c0 msgr2=0x7f5e0806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.559+0000 7f5e0f200640 1 -- 172.21.15.62:0/2770108522 shutdown_connections 2024-03-23T11:28:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.559+0000 7f5e0f200640 1 -- 172.21.15.62:0/2770108522 wait complete. 2024-03-23T11:28:04.569 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:28:04.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.798+0000 7fa66ac00640 1 Processor -- start 2024-03-23T11:28:04.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.799+0000 7fa66ac00640 1 -- start start 2024-03-23T11:28:04.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.800+0000 7fa66ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 0x7fa6641607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:04.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.800+0000 7fa66ac00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa66415cdc0 0x7fa6640d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:04.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.800+0000 7fa66ac00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66406bfc0 con 0x7fa6640cf650 2024-03-23T11:28:04.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.800+0000 7fa66ac00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6640ce140 con 0x7fa66415cdc0 2024-03-23T11:28:04.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.800+0000 7fa66ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640b4370 con 0x7fa66415c9e0 2024-03-23T11:28:04.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.801+0000 7fa668c00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa6640cf650 0x7fa66415d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:52078/0 (socket says 172.21.15.62:52078) 2024-03-23T11:28:04.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.801+0000 7fa668c00640 1 -- 172.21.15.62:0/571288188 learned_addr learned my addr 172.21.15.62:0/571288188 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:04.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.801+0000 7fa663400640 1 --2- 172.21.15.62:0/571288188 >> v2:172.21.15.62:3300/0 conn(0x7fa66415cdc0 0x7fa6640d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:04.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.801+0000 7fa663e00640 1 --2- 172.21.15.62:0/571288188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 0x7fa6641607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:04.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.801+0000 7fa662a00640 1 -- 172.21.15.62:0/571288188 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544899114 0 0) 0x7fa66406bfc0 con 0x7fa6640cf650 2024-03-23T11:28:04.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.801+0000 7fa662a00640 1 -- 172.21.15.62:0/571288188 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa648003520 con 0x7fa6640cf650 2024-03-23T11:28:04.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.801+0000 7fa663400640 1 -- 172.21.15.62:0/571288188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 msgr2=0x7fa6641607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:04.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.801+0000 7fa663400640 1 --2- 172.21.15.62:0/571288188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 0x7fa6641607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.801+0000 7fa663400640 1 -- 172.21.15.62:0/571288188 >> v1:172.21.15.62:6789/0 conn(0x7fa6640cf650 legacy=0x7fa66415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.801+0000 7fa663400640 1 -- 172.21.15.62:0/571288188 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6640b5260 con 0x7fa66415cdc0 2024-03-23T11:28:04.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.801+0000 7fa663e00640 1 --2- 172.21.15.62:0/571288188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 0x7fa6641607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:28:04.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.803+0000 7fa663400640 1 --2- 172.21.15.62:0/571288188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa66415cdc0 0x7fa6640d1f20 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fa650009d90 tx=0x7fa65002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619b8015c97fb314 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:04.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.803+0000 7fa662a00640 1 -- 172.21.15.62:0/571288188 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65002cc50 con 0x7fa66415cdc0 2024-03-23T11:28:04.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.803+0000 7fa662a00640 1 -- 172.21.15.62:0/571288188 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65002cdb0 con 0x7fa66415cdc0 2024-03-23T11:28:04.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.803+0000 7fa662a00640 1 -- 172.21.15.62:0/571288188 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650034640 con 0x7fa66415cdc0 2024-03-23T11:28:04.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.804+0000 7fa66ac00640 1 -- 172.21.15.62:0/571288188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa66415cdc0 msgr2=0x7fa6640d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.804+0000 7fa66ac00640 1 --2- 172.21.15.62:0/571288188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa66415cdc0 0x7fa6640d1f20 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fa650009d90 tx=0x7fa65002bea0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.804+0000 7fa66ac00640 1 -- 172.21.15.62:0/571288188 shutdown_connections 2024-03-23T11:28:04.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.804+0000 7fa66ac00640 1 --2- 172.21.15.62:0/571288188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa66415cdc0 0x7fa6640d1f20 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.804+0000 7fa66ac00640 1 --2- 172.21.15.62:0/571288188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 0x7fa6641607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.804+0000 7fa66ac00640 1 -- 172.21.15.62:0/571288188 >> 172.21.15.62:0/571288188 conn(0x7fa664071570 msgr2=0x7fa664071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:04.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.804+0000 7fa66ac00640 1 -- 172.21.15.62:0/571288188 shutdown_connections 2024-03-23T11:28:04.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.804+0000 7fa66ac00640 1 -- 172.21.15.62:0/571288188 wait complete. 2024-03-23T11:28:04.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.806+0000 7fa66ac00640 1 Processor -- start 2024-03-23T11:28:04.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.806+0000 7fa66ac00640 1 -- start start 2024-03-23T11:28:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.807+0000 7fa66ac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6640cf650 0x7fa66414e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.807+0000 7fa66ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 0x7fa6640b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.807+0000 7fa66ac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa66415cdc0 0x7fa6640cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.807+0000 7fa66ac00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640b5e50 con 0x7fa66415cdc0 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.807+0000 7fa66ac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640b6130 con 0x7fa6640cf650 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.807+0000 7fa66ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640b6410 con 0x7fa66415c9e0 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.807+0000 7fa668c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa66415cdc0 0x7fa6640cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.807+0000 7fa668c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa66415cdc0 0x7fa6640cb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54190/0 (socket says 172.21.15.62:54190) 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.807+0000 7fa668c00640 1 -- 172.21.15.62:0/1201536033 learned_addr learned my addr 172.21.15.62:0/1201536033 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.807+0000 7fa663400640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 0x7fa6640b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.807+0000 7fa663e00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6640cf650 0x7fa66414e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa668c00640 1 -- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 msgr2=0x7fa6640b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa668c00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 0x7fa6640b6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa668c00640 1 -- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6640cf650 msgr2=0x7fa66414e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa668c00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6640cf650 0x7fa66414e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa668c00640 1 -- 172.21.15.62:0/1201536033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa650009a70 con 0x7fa66415cdc0 2024-03-23T11:28:04.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa663400640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 0x7fa6640b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:04.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa663e00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6640cf650 0x7fa66414e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:04.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa668c00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa66415cdc0 0x7fa6640cb9d0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00c430 tx=0x7fa65c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:04.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa65be00640 1 -- 172.21.15.62:0/1201536033 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c005ce0 con 0x7fa66415cdc0 2024-03-23T11:28:04.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa65be00640 1 -- 172.21.15.62:0/1201536033 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c005e40 con 0x7fa66415cdc0 2024-03-23T11:28:04.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa65be00640 1 -- 172.21.15.62:0/1201536033 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c007310 con 0x7fa66415cdc0 2024-03-23T11:28:04.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa66ac00640 1 -- 172.21.15.62:0/1201536033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6640b66f0 con 0x7fa66415cdc0 2024-03-23T11:28:04.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.808+0000 7fa66ac00640 1 -- 172.21.15.62:0/1201536033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6640b7430 con 0x7fa66415cdc0 2024-03-23T11:28:04.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.810+0000 7fa66ac00640 1 -- 172.21.15.62:0/1201536033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6640b6c10 con 0x7fa66415cdc0 2024-03-23T11:28:04.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.810+0000 7fa65be00640 1 -- 172.21.15.62:0/1201536033 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa65c007470 con 0x7fa66415cdc0 2024-03-23T11:28:04.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.811+0000 7fa65be00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa64406a7a0 0x7fa64406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:04.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.811+0000 7fa65be00640 1 -- 172.21.15.62:0/1201536033 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fa65c088460 con 0x7fa66415cdc0 2024-03-23T11:28:04.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.812+0000 7fa663e00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa64406a7a0 0x7fa64406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:04.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.813+0000 7fa663e00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa64406a7a0 0x7fa64406cc60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa654006fd0 tx=0x7fa654008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:04.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:04.816+0000 7fa65be00640 1 -- 172.21.15.62:0/1201536033 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa6640b6c10 con 0x7fa66415cdc0 2024-03-23T11:28:05.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.009+0000 7fa66ac00640 1 -- 172.21.15.62:0/1201536033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa6640b6c10 con 0x7fa66415cdc0 2024-03-23T11:28:05.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.009+0000 7fa65be00640 1 -- 172.21.15.62:0/1201536033 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa6640b6c10 con 0x7fa66415cdc0 2024-03-23T11:28:05.001 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:28:05.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 -- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa64406a7a0 msgr2=0x7fa64406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa64406a7a0 0x7fa64406cc60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa654006fd0 tx=0x7fa654008040 comp rx=0 tx=0).stop 2024-03-23T11:28:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 -- 172.21.15.62:0/1201536033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa66415cdc0 msgr2=0x7fa6640cb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa66415cdc0 0x7fa6640cb9d0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00c430 tx=0x7fa65c00c8f0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 -- 172.21.15.62:0/1201536033 shutdown_connections 2024-03-23T11:28:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa64406a7a0 0x7fa64406cc60 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa66415cdc0 0x7fa6640cb9d0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa66415c9e0 0x7fa6640b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 --2- 172.21.15.62:0/1201536033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6640cf650 0x7fa66414e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 -- 172.21.15.62:0/1201536033 >> 172.21.15.62:0/1201536033 conn(0x7fa664071570 msgr2=0x7fa66415e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 -- 172.21.15.62:0/1201536033 shutdown_connections 2024-03-23T11:28:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.012+0000 7fa66ac00640 1 -- 172.21.15.62:0/1201536033 wait complete. 2024-03-23T11:28:05.025 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:28:05.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.253+0000 7f2f77200640 1 Processor -- start 2024-03-23T11:28:05.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.254+0000 7f2f77200640 1 -- start start 2024-03-23T11:28:05.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.255+0000 7f2f77200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2f7815fcf0 0x7f2f780d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.255+0000 7f2f77200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2f780b6dc0 0x7f2f781608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.255+0000 7f2f77200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f780d3540 con 0x7f2f78160420 2024-03-23T11:28:05.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.255+0000 7f2f77200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f78160df0 con 0x7f2f7815fcf0 2024-03-23T11:28:05.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.255+0000 7f2f77200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f780ce510 con 0x7f2f780b6dc0 2024-03-23T11:28:05.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.255+0000 7f2f75e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2f7815fcf0 0x7f2f780d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.255+0000 7f2f75e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2f7815fcf0 0x7f2f780d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54204/0 (socket says 172.21.15.62:54204) 2024-03-23T11:28:05.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.255+0000 7f2f75e00640 1 -- 172.21.15.62:0/1862494791 learned_addr learned my addr 172.21.15.62:0/1862494791 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:05.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.256+0000 7f2f76800640 1 --2- 172.21.15.62:0/1862494791 >> v2:172.21.15.120:3300/0 conn(0x7f2f780b6dc0 0x7f2f781608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.256+0000 7f2f75e00640 1 -- 172.21.15.62:0/1862494791 >> v2:172.21.15.120:3300/0 conn(0x7f2f780b6dc0 msgr2=0x7f2f781608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:05.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.256+0000 7f2f75e00640 1 --2- 172.21.15.62:0/1862494791 >> v2:172.21.15.120:3300/0 conn(0x7f2f780b6dc0 0x7f2f781608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.256+0000 7f2f75e00640 1 -- 172.21.15.62:0/1862494791 >> v1:172.21.15.62:6789/0 conn(0x7f2f78160420 legacy=0x7f2f780d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.256+0000 7f2f75e00640 1 -- 172.21.15.62:0/1862494791 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f780cdc70 con 0x7f2f7815fcf0 2024-03-23T11:28:05.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.257+0000 7f2f75e00640 1 --2- 172.21.15.62:0/1862494791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f7815fcf0 0x7f2f780d3000 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f2f7808bb20 tx=0x7f2f6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4050f8e506c669c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:05.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.258+0000 7f2f74a00640 1 -- 172.21.15.62:0/1862494791 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68031870 con 0x7f2f7815fcf0 2024-03-23T11:28:05.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.258+0000 7f2f74a00640 1 -- 172.21.15.62:0/1862494791 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f680319d0 con 0x7f2f7815fcf0 2024-03-23T11:28:05.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.258+0000 7f2f74a00640 1 -- 172.21.15.62:0/1862494791 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68031d60 con 0x7f2f7815fcf0 2024-03-23T11:28:05.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.258+0000 7f2f77200640 1 -- 172.21.15.62:0/1862494791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f7815fcf0 msgr2=0x7f2f780d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.258+0000 7f2f77200640 1 --2- 172.21.15.62:0/1862494791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f7815fcf0 0x7f2f780d3000 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f2f7808bb20 tx=0x7f2f6802b910 comp rx=0 tx=0).stop 2024-03-23T11:28:05.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.258+0000 7f2f77200640 1 -- 172.21.15.62:0/1862494791 shutdown_connections 2024-03-23T11:28:05.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.258+0000 7f2f77200640 1 --2- 172.21.15.62:0/1862494791 >> v2:172.21.15.120:3300/0 conn(0x7f2f780b6dc0 0x7f2f781608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.258+0000 7f2f77200640 1 --2- 172.21.15.62:0/1862494791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f7815fcf0 0x7f2f780d3000 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.258+0000 7f2f77200640 1 -- 172.21.15.62:0/1862494791 >> 172.21.15.62:0/1862494791 conn(0x7f2f78089f60 msgr2=0x7f2f780d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:05.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.258+0000 7f2f77200640 1 -- 172.21.15.62:0/1862494791 shutdown_connections 2024-03-23T11:28:05.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.258+0000 7f2f77200640 1 -- 172.21.15.62:0/1862494791 wait complete. 2024-03-23T11:28:05.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.260+0000 7f2f77200640 1 Processor -- start 2024-03-23T11:28:05.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.260+0000 7f2f77200640 1 -- start start 2024-03-23T11:28:05.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.261+0000 7f2f77200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f780b6dc0 0x7f2f78151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.261+0000 7f2f77200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f7815fcf0 0x7f2f78150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.261+0000 7f2f77200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f78160420 0x7f2f7809f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.261+0000 7f2f77200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7809d120 con 0x7f2f7815fcf0 2024-03-23T11:28:05.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.261+0000 7f2f77200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7809d400 con 0x7f2f78160420 2024-03-23T11:28:05.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.261+0000 7f2f77200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7809d6e0 con 0x7f2f780b6dc0 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f75400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f7815fcf0 0x7f2f78150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f75400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f7815fcf0 0x7f2f78150af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54208/0 (socket says 172.21.15.62:54208) 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f75400640 1 -- 172.21.15.62:0/753528004 learned_addr learned my addr 172.21.15.62:0/753528004 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f75e00640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f780b6dc0 0x7f2f78151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f76800640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f78160420 0x7f2f7809f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f75400640 1 -- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f780b6dc0 msgr2=0x7f2f78151380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f75400640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f780b6dc0 0x7f2f78151380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f75400640 1 -- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f78160420 msgr2=0x7f2f7809f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f75400640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f78160420 0x7f2f7809f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f75400640 1 -- 172.21.15.62:0/753528004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f680094d0 con 0x7f2f7815fcf0 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f75e00640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f780b6dc0 0x7f2f78151380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:05.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.262+0000 7f2f75400640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f7815fcf0 0x7f2f78150af0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000c730 tx=0x7f2f6000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:05.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.263+0000 7f2f6e000640 1 -- 172.21.15.62:0/753528004 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60003b70 con 0x7f2f7815fcf0 2024-03-23T11:28:05.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.263+0000 7f2f6e000640 1 -- 172.21.15.62:0/753528004 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f60003cf0 con 0x7f2f7815fcf0 2024-03-23T11:28:05.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.263+0000 7f2f6e000640 1 -- 172.21.15.62:0/753528004 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60003ea0 con 0x7f2f7815fcf0 2024-03-23T11:28:05.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.263+0000 7f2f77200640 1 -- 172.21.15.62:0/753528004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7809d9c0 con 0x7f2f7815fcf0 2024-03-23T11:28:05.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.263+0000 7f2f77200640 1 -- 172.21.15.62:0/753528004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f78151030 con 0x7f2f7815fcf0 2024-03-23T11:28:05.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.264+0000 7f2f77200640 1 -- 172.21.15.62:0/753528004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7809dee0 con 0x7f2f7815fcf0 2024-03-23T11:28:05.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.265+0000 7f2f6e000640 1 -- 172.21.15.62:0/753528004 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2f6000aa90 con 0x7f2f7815fcf0 2024-03-23T11:28:05.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.266+0000 7f2f6e000640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f5406a7a0 0x7f2f5406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.266+0000 7f2f6e000640 1 -- 172.21.15.62:0/753528004 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f2f60088c90 con 0x7f2f7815fcf0 2024-03-23T11:28:05.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.266+0000 7f2f75e00640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f5406a7a0 0x7f2f5406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.268+0000 7f2f75e00640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f5406a7a0 0x7f2f5406cc60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2f68036000 tx=0x7f2f6803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:05.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.270+0000 7f2f6e000640 1 -- 172.21.15.62:0/753528004 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f7809dee0 con 0x7f2f7815fcf0 2024-03-23T11:28:05.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.461+0000 7f2f77200640 1 -- 172.21.15.62:0/753528004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2f7809dee0 con 0x7f2f7815fcf0 2024-03-23T11:28:05.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.462+0000 7f2f6e000640 1 -- 172.21.15.62:0/753528004 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2f7809dee0 con 0x7f2f7815fcf0 2024-03-23T11:28:05.454 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:28:05.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 -- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f5406a7a0 msgr2=0x7f2f5406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f5406a7a0 0x7f2f5406cc60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2f68036000 tx=0x7f2f6803c040 comp rx=0 tx=0).stop 2024-03-23T11:28:05.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 -- 172.21.15.62:0/753528004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f7815fcf0 msgr2=0x7f2f78150af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f7815fcf0 0x7f2f78150af0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000c730 tx=0x7f2f6000cbf0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 -- 172.21.15.62:0/753528004 shutdown_connections 2024-03-23T11:28:05.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2f5406a7a0 0x7f2f5406cc60 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f7815fcf0 0x7f2f78150af0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f78160420 0x7f2f7809f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 --2- 172.21.15.62:0/753528004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f780b6dc0 0x7f2f78151380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 -- 172.21.15.62:0/753528004 >> 172.21.15.62:0/753528004 conn(0x7f2f78089f60 msgr2=0x7f2f780cb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:05.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 -- 172.21.15.62:0/753528004 shutdown_connections 2024-03-23T11:28:05.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.465+0000 7f2f77200640 1 -- 172.21.15.62:0/753528004 wait complete. 2024-03-23T11:28:05.476 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:28:05.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.701+0000 7f8ec2600640 1 Processor -- start 2024-03-23T11:28:05.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.701+0000 7f8ec2600640 1 -- start start 2024-03-23T11:28:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.703+0000 7f8ec2600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8ebc06c300 0x7f8ebc0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.703+0000 7f8ec2600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ebc0cafa0 0x7f8ebc0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.703+0000 7f8ec2600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ebc0cbc60 con 0x7f8ebc06c6e0 2024-03-23T11:28:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.703+0000 7f8ec2600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ebc0cbf40 con 0x7f8ebc06c300 2024-03-23T11:28:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.703+0000 7f8ec2600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0cc220 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.703+0000 7f8ebb400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8ebc06c300 0x7f8ebc0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.703+0000 7f8ebbe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8ebc06c6e0 0x7f8ebc06cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:52106/0 (socket says 172.21.15.62:52106) 2024-03-23T11:28:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.703+0000 7f8ebbe00640 1 -- 172.21.15.62:0/2380870503 learned_addr learned my addr 172.21.15.62:0/2380870503 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.704+0000 7f8ebaa00640 1 --2- 172.21.15.62:0/2380870503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ebc0cafa0 0x7f8ebc0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.704+0000 7f8ebb400640 1 -- 172.21.15.62:0/2380870503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ebc0cafa0 msgr2=0x7f8ebc0cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.704+0000 7f8ebb400640 1 --2- 172.21.15.62:0/2380870503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ebc0cafa0 0x7f8ebc0cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.704+0000 7f8ebb400640 1 -- 172.21.15.62:0/2380870503 >> v1:172.21.15.62:6789/0 conn(0x7f8ebc06c6e0 legacy=0x7f8ebc06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.704+0000 7f8ebb400640 1 -- 172.21.15.62:0/2380870503 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc0cc500 con 0x7f8ebc06c300 2024-03-23T11:28:05.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.705+0000 7f8ebb400640 1 --2- 172.21.15.62:0/2380870503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ebc06c300 0x7f8ebc0ccd00 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0097a0 tx=0x7f8eac02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57e93b13c2e53c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:05.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.705+0000 7f8eba000640 1 -- 172.21.15.62:0/2380870503 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac031780 con 0x7f8ebc06c300 2024-03-23T11:28:05.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.705+0000 7f8eba000640 1 -- 172.21.15.62:0/2380870503 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eac0318e0 con 0x7f8ebc06c300 2024-03-23T11:28:05.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.705+0000 7f8eba000640 1 -- 172.21.15.62:0/2380870503 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac031c70 con 0x7f8ebc06c300 2024-03-23T11:28:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.706+0000 7f8ec2600640 1 -- 172.21.15.62:0/2380870503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ebc06c300 msgr2=0x7f8ebc0ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.706+0000 7f8ec2600640 1 --2- 172.21.15.62:0/2380870503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ebc06c300 0x7f8ebc0ccd00 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0097a0 tx=0x7f8eac02b820 comp rx=0 tx=0).stop 2024-03-23T11:28:05.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.706+0000 7f8ec2600640 1 -- 172.21.15.62:0/2380870503 shutdown_connections 2024-03-23T11:28:05.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.706+0000 7f8ec2600640 1 --2- 172.21.15.62:0/2380870503 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ebc0cafa0 0x7f8ebc0cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.706+0000 7f8ec2600640 1 --2- 172.21.15.62:0/2380870503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ebc06c300 0x7f8ebc0ccd00 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.706+0000 7f8ec2600640 1 -- 172.21.15.62:0/2380870503 >> 172.21.15.62:0/2380870503 conn(0x7f8ebc0b02e0 msgr2=0x7f8ebc0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:05.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.706+0000 7f8ec2600640 1 -- 172.21.15.62:0/2380870503 shutdown_connections 2024-03-23T11:28:05.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.706+0000 7f8ec2600640 1 -- 172.21.15.62:0/2380870503 wait complete. 2024-03-23T11:28:05.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.708+0000 7f8ec2600640 1 Processor -- start 2024-03-23T11:28:05.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.708+0000 7f8ec2600640 1 -- start start 2024-03-23T11:28:05.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.709+0000 7f8ec2600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ebc06c300 0x7f8ebc13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.709+0000 7f8ec2600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc06c6e0 0x7f8ebc13fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.709+0000 7f8ec2600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ebc0cafa0 0x7f8ebc15c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.709+0000 7f8ec2600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc13e360 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.709+0000 7f8ec2600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc15d650 con 0x7f8ebc06c6e0 2024-03-23T11:28:05.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.709+0000 7f8ec2600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc13e600 con 0x7f8ebc06c300 2024-03-23T11:28:05.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.709+0000 7f8ebbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ebc0cafa0 0x7f8ebc15c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.709+0000 7f8ebbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ebc0cafa0 0x7f8ebc15c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54226/0 (socket says 172.21.15.62:54226) 2024-03-23T11:28:05.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.709+0000 7f8ebbe00640 1 -- 172.21.15.62:0/2030427144 learned_addr learned my addr 172.21.15.62:0/2030427144 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:05.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.710+0000 7f8ebb400640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ebc06c300 0x7f8ebc13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.710+0000 7f8ebbe00640 1 -- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ebc06c300 msgr2=0x7f8ebc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.710+0000 7f8ebaa00640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc06c6e0 0x7f8ebc13fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.710+0000 7f8ebbe00640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ebc06c300 0x7f8ebc13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.710+0000 7f8ebbe00640 1 -- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc06c6e0 msgr2=0x7f8ebc13fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.710+0000 7f8ebbe00640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc06c6e0 0x7f8ebc13fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.710+0000 7f8ebbe00640 1 -- 172.21.15.62:0/2030427144 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac009400 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.710+0000 7f8ebaa00640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc06c6e0 0x7f8ebc13fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.710+0000 7f8ebb400640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ebc06c300 0x7f8ebc13cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.710+0000 7f8ebbe00640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ebc0cafa0 0x7f8ebc15c810 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400cbc0 tx=0x7f8eb400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.711+0000 7f8eb3400640 1 -- 172.21.15.62:0/2030427144 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400ad40 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.711+0000 7f8eb3400640 1 -- 172.21.15.62:0/2030427144 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb4012400 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.711+0000 7f8eb3400640 1 -- 172.21.15.62:0/2030427144 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400aea0 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.711+0000 7f8ec2600640 1 -- 172.21.15.62:0/2030427144 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ebc13e8e0 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.711+0000 7f8ec2600640 1 -- 172.21.15.62:0/2030427144 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc13eea0 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.712+0000 7f8ec2600640 1 -- 172.21.15.62:0/2030427144 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ebc066c90 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.713+0000 7f8eb3400640 1 -- 172.21.15.62:0/2030427144 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8eb4007070 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.714+0000 7f8eb3400640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e9c06a7a0 0x7f8e9c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:05.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.714+0000 7f8eb3400640 1 -- 172.21.15.62:0/2030427144 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f8eb4088d70 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.714+0000 7f8ebb400640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e9c06a7a0 0x7f8e9c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:05.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.716+0000 7f8ebb400640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e9c06a7a0 0x7f8e9c06cc60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8eac036000 tx=0x7f8eac03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.719+0000 7f8eb3400640 1 -- 172.21.15.62:0/2030427144 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8eb4054150 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.928+0000 7f8ec2600640 1 -- 172.21.15.62:0/2030427144 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8ebc06d4b0 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.929+0000 7f8eb3400640 1 -- 172.21.15.62:0/2030427144 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v92) v1 ==== 80+0+122 (secure 0 0 0) 0x7f8eb4017020 con 0x7f8ebc0cafa0 2024-03-23T11:28:05.921 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6833/3056962984 2024-03-24T11:26:47.481437+0000 2024-03-23T11:28:05.922 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6832/3056962984 2024-03-24T11:26:47.481437+0000 2024-03-23T11:28:05.922 INFO:teuthology.orchestra.run.smithi062.stderr:listed 2 entries 2024-03-23T11:28:05.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.931+0000 7f8ec2600640 1 -- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e9c06a7a0 msgr2=0x7f8e9c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.931+0000 7f8ec2600640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e9c06a7a0 0x7f8e9c06cc60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8eac036000 tx=0x7f8eac03d040 comp rx=0 tx=0).stop 2024-03-23T11:28:05.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.931+0000 7f8ec2600640 1 -- 172.21.15.62:0/2030427144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ebc0cafa0 msgr2=0x7f8ebc15c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:05.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.931+0000 7f8ec2600640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ebc0cafa0 0x7f8ebc15c810 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400cbc0 tx=0x7f8eb400a430 comp rx=0 tx=0).stop 2024-03-23T11:28:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.932+0000 7f8ec2600640 1 -- 172.21.15.62:0/2030427144 shutdown_connections 2024-03-23T11:28:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.932+0000 7f8ec2600640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e9c06a7a0 0x7f8e9c06cc60 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.932+0000 7f8ec2600640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ebc0cafa0 0x7f8ebc15c810 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.932+0000 7f8ec2600640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ebc06c6e0 0x7f8ebc13fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.932+0000 7f8ec2600640 1 --2- 172.21.15.62:0/2030427144 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ebc06c300 0x7f8ebc13cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.932+0000 7f8ec2600640 1 -- 172.21.15.62:0/2030427144 >> 172.21.15.62:0/2030427144 conn(0x7f8ebc0b02e0 msgr2=0x7f8ebc0ac4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.932+0000 7f8ec2600640 1 -- 172.21.15.62:0/2030427144 shutdown_connections 2024-03-23T11:28:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:05.932+0000 7f8ec2600640 1 -- 172.21.15.62:0/2030427144 wait complete. 2024-03-23T11:28:05.943 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:28:05.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:05.944 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:28:06.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[67465]: fuse finished with error 0 and tester_r 0 2024-03-23T11:28:06.037 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:28:12.139 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:28:12.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:12.140 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:28:12.201 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:28:12.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.438+0000 7f4701200640 1 Processor -- start 2024-03-23T11:28:12.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.438+0000 7f4701200640 1 -- start start 2024-03-23T11:28:12.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.439+0000 7f4701200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46fc160c00 0x7f46fc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:12.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.440+0000 7f4701200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f46fc0cda20 0x7f46fc06de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:12.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.440+0000 7f4701200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc160fe0 con 0x7f46fc0cde90 2024-03-23T11:28:12.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.440+0000 7f4701200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc0b1ad0 con 0x7f46fc0cda20 2024-03-23T11:28:12.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.440+0000 7f4701200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0b1510 con 0x7f46fc160c00 2024-03-23T11:28:12.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.440+0000 7f46f2a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f46fc0cda20 0x7f46fc06de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:12.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.440+0000 7f46f2a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f46fc0cda20 0x7f46fc06de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50790/0 (socket says 172.21.15.62:50790) 2024-03-23T11:28:12.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.440+0000 7f46f2a00640 1 -- 172.21.15.62:0/3608492327 learned_addr learned my addr 172.21.15.62:0/3608492327 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:12.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.440+0000 7f46fb400640 1 --2- 172.21.15.62:0/3608492327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46fc160c00 0x7f46fc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:12.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.441+0000 7f46f2a00640 1 -- 172.21.15.62:0/3608492327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46fc160c00 msgr2=0x7f46fc06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:12.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.441+0000 7f46f2a00640 1 --2- 172.21.15.62:0/3608492327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46fc160c00 0x7f46fc06d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:12.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.441+0000 7f46f2a00640 1 -- 172.21.15.62:0/3608492327 >> v1:172.21.15.120:6789/0 conn(0x7f46fc0cde90 legacy=0x7f46fc0caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:12.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.441+0000 7f46f2a00640 1 -- 172.21.15.62:0/3608492327 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0b17f0 con 0x7f46fc0cda20 2024-03-23T11:28:12.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.441+0000 7f46fb400640 1 --2- 172.21.15.62:0/3608492327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46fc160c00 0x7f46fc06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:12.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.442+0000 7f46f2a00640 1 --2- 172.21.15.62:0/3608492327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46fc0cda20 0x7f46fc06de10 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009870 tx=0x7f46e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c4e1482832fc95 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:12.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.442+0000 7f46faa00640 1 -- 172.21.15.62:0/3608492327 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4031640 con 0x7f46fc0cda20 2024-03-23T11:28:12.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.442+0000 7f46faa00640 1 -- 172.21.15.62:0/3608492327 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e40317a0 con 0x7f46fc0cda20 2024-03-23T11:28:12.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.442+0000 7f46faa00640 1 -- 172.21.15.62:0/3608492327 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4031b30 con 0x7f46fc0cda20 2024-03-23T11:28:12.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.443+0000 7f4701200640 1 -- 172.21.15.62:0/3608492327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46fc0cda20 msgr2=0x7f46fc06de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:12.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.443+0000 7f4701200640 1 --2- 172.21.15.62:0/3608492327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46fc0cda20 0x7f46fc06de10 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009870 tx=0x7f46e402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:28:12.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.443+0000 7f4701200640 1 -- 172.21.15.62:0/3608492327 shutdown_connections 2024-03-23T11:28:12.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.443+0000 7f4701200640 1 --2- 172.21.15.62:0/3608492327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46fc0cda20 0x7f46fc06de10 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:12.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.443+0000 7f4701200640 1 --2- 172.21.15.62:0/3608492327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46fc160c00 0x7f46fc06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:12.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.443+0000 7f4701200640 1 -- 172.21.15.62:0/3608492327 >> 172.21.15.62:0/3608492327 conn(0x7f46fc066050 msgr2=0x7f46fc069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:12.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.443+0000 7f4701200640 1 -- 172.21.15.62:0/3608492327 shutdown_connections 2024-03-23T11:28:12.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.443+0000 7f4701200640 1 -- 172.21.15.62:0/3608492327 wait complete. 2024-03-23T11:28:12.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.445+0000 7f4701200640 1 Processor -- start 2024-03-23T11:28:12.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.445+0000 7f4701200640 1 -- start start 2024-03-23T11:28:12.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.446+0000 7f4701200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46fc0cda20 0x7f46fc163860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:12.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.446+0000 7f4701200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46fc0cde90 0x7f46fc1f97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:12.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.446+0000 7f4701200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46fc160c00 0x7f46fc0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:12.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.446+0000 7f4701200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc161190 con 0x7f46fc0cde90 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.446+0000 7f4701200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc1f9d20 con 0x7f46fc0cda20 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.446+0000 7f4701200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc1f9fa0 con 0x7f46fc160c00 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.446+0000 7f46f2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46fc0cde90 0x7f46fc1f97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.446+0000 7f46f2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46fc0cde90 0x7f46fc1f97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50806/0 (socket says 172.21.15.62:50806) 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.446+0000 7f46f2a00640 1 -- 172.21.15.62:0/149210728 learned_addr learned my addr 172.21.15.62:0/149210728 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f46fb400640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46fc0cda20 0x7f46fc163860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f46fbe00640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46fc160c00 0x7f46fc0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f46f2a00640 1 -- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46fc160c00 msgr2=0x7f46fc0b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f46f2a00640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46fc160c00 0x7f46fc0b02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f46f2a00640 1 -- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46fc0cda20 msgr2=0x7f46fc163860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f46f2a00640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46fc0cda20 0x7f46fc163860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f46f2a00640 1 -- 172.21.15.62:0/149210728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e40094d0 con 0x7f46fc0cde90 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f46f2a00640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46fc0cde90 0x7f46fc1f97e0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009870 tx=0x7f46e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f46f3e00640 1 -- 172.21.15.62:0/149210728 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e403ac60 con 0x7f46fc0cde90 2024-03-23T11:28:12.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f46f3e00640 1 -- 172.21.15.62:0/149210728 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e403ade0 con 0x7f46fc0cde90 2024-03-23T11:28:12.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f46f3e00640 1 -- 172.21.15.62:0/149210728 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4044630 con 0x7f46fc0cde90 2024-03-23T11:28:12.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.447+0000 7f4701200640 1 -- 172.21.15.62:0/149210728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc1fa280 con 0x7f46fc0cde90 2024-03-23T11:28:12.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.448+0000 7f4701200640 1 -- 172.21.15.62:0/149210728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc1613a0 con 0x7f46fc0cde90 2024-03-23T11:28:12.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.449+0000 7f4701200640 1 -- 172.21.15.62:0/149210728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc066bb0 con 0x7f46fc0cde90 2024-03-23T11:28:12.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.450+0000 7f46f3e00640 1 -- 172.21.15.62:0/149210728 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f46e402f050 con 0x7f46fc0cde90 2024-03-23T11:28:12.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.450+0000 7f46f3e00640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f46d806a7a0 0x7f46d806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:12.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.451+0000 7f46f3e00640 1 -- 172.21.15.62:0/149210728 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f46e40aaf70 con 0x7f46fc0cde90 2024-03-23T11:28:12.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.451+0000 7f46fb400640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f46d806a7a0 0x7f46d806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:12.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.453+0000 7f46fb400640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f46d806a7a0 0x7f46d806cc60 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0b30b0 tx=0x7f46ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:12.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.455+0000 7f46f3e00640 1 -- 172.21.15.62:0/149210728 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f46e4076330 con 0x7f46fc0cde90 2024-03-23T11:28:12.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.691+0000 7f4701200640 1 -- 172.21.15.62:0/149210728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46fc0ae7f0 con 0x7f46fc0cde90 2024-03-23T11:28:12.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.692+0000 7f46f3e00640 1 -- 172.21.15.62:0/149210728 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f46e4079950 con 0x7f46fc0cde90 2024-03-23T11:28:12.685 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:12.685 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:26:59.466628+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":6,"addr":"172.21.15.62:6833/2573443114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2573443114},{"type":"v1","addr":"172.21.15.62:6833","nonce":2573443114}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:28:12.685 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 64 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 -- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f46d806a7a0 msgr2=0x7f46d806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f46d806a7a0 0x7f46d806cc60 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0b30b0 tx=0x7f46ec008040 comp rx=0 tx=0).stop 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 -- 172.21.15.62:0/149210728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46fc0cde90 msgr2=0x7f46fc1f97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46fc0cde90 0x7f46fc1f97e0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009870 tx=0x7f46e40046c0 comp rx=0 tx=0).stop 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 -- 172.21.15.62:0/149210728 shutdown_connections 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f46d806a7a0 0x7f46d806cc60 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46fc0cde90 0x7f46fc1f97e0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46fc160c00 0x7f46fc0b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 --2- 172.21.15.62:0/149210728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46fc0cda20 0x7f46fc163860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 -- 172.21.15.62:0/149210728 >> 172.21.15.62:0/149210728 conn(0x7f46fc066050 msgr2=0x7f46fc0cced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 -- 172.21.15.62:0/149210728 shutdown_connections 2024-03-23T11:28:12.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:12.696+0000 7f4701200640 1 -- 172.21.15.62:0/149210728 wait complete. 2024-03-23T11:28:12.706 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-23T11:28:12.965 INFO:teuthology.orchestra.run.smithi062.stdout:{"message":"","return_code":0} 2024-03-23T11:28:12.965 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-23T11:28:12.965 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:28:13.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.173+0000 7fd02b000640 1 Processor -- start 2024-03-23T11:28:13.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.173+0000 7fd02b000640 1 -- start start 2024-03-23T11:28:13.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.175+0000 7fd02b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02406c6e0 0x7fd02406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.175+0000 7fd02b000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0240cbc60 con 0x7fd02406c300 2024-03-23T11:28:13.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.175+0000 7fd02b000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0240cbf40 con 0x7fd0240cafa0 2024-03-23T11:28:13.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.175+0000 7fd02b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0240cd410 con 0x7fd02406c6e0 2024-03-23T11:28:13.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.175+0000 7fd023e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd02406c300 0x7fd0240ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44004/0 (socket says 172.21.15.62:44004) 2024-03-23T11:28:13.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.175+0000 7fd023e00640 1 -- 172.21.15.62:0/2590605741 learned_addr learned my addr 172.21.15.62:0/2590605741 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:13.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd029000640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02406c6e0 0x7fd02406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510010611 0 0) 0x7fd0240cbc60 con 0x7fd02406c300 2024-03-23T11:28:13.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd008003520 con 0x7fd02406c300 2024-03-23T11:28:13.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981447925 0 0) 0x7fd008003520 con 0x7fd02406c300 2024-03-23T11:28:13.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02406c6e0 msgr2=0x7fd02406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02406c6e0 0x7fd02406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 >> v1:172.21.15.120:6789/0 conn(0x7fd0240cafa0 legacy=0x7fd0240cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0240cc220 con 0x7fd02406c300 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fd014001670 con 0x7fd02406c300 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 >> v1:172.21.15.62:6789/0 conn(0x7fd02406c300 legacy=0x7fd0240ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02406c6e0 0x7fd0080039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 0x7fd0080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd008004d00 0x7fd008005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0240cd410 con 0x7fd008003f70 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00800bac0 con 0x7fd008004d00 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.176+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00800bc80 con 0x7fd02406c6e0 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.177+0000 7fd023400640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 0x7fd0080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.177+0000 7fd029000640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd008004d00 0x7fd008005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.177+0000 7fd023e00640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02406c6e0 0x7fd0080039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.177+0000 7fd023400640 1 -- 172.21.15.62:0/2590605741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02406c6e0 msgr2=0x7fd0080039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.177+0000 7fd023400640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd02406c6e0 0x7fd0080039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.177+0000 7fd023400640 1 -- 172.21.15.62:0/2590605741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd008004d00 msgr2=0x7fd008005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.177+0000 7fd023400640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd008004d00 0x7fd008005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.177+0000 7fd023400640 1 -- 172.21.15.62:0/2590605741 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd008003520 con 0x7fd008003f70 2024-03-23T11:28:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.179+0000 7fd023400640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 0x7fd0080047c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fd010001750 tx=0x7fd0080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b579679998db90 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.179+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0100052d0 con 0x7fd008003f70 2024-03-23T11:28:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.179+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01002e050 con 0x7fd008003f70 2024-03-23T11:28:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.179+0000 7fd022a00640 1 -- 172.21.15.62:0/2590605741 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd010003830 con 0x7fd008003f70 2024-03-23T11:28:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.179+0000 7fd02b000640 1 -- 172.21.15.62:0/2590605741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 msgr2=0x7fd0080047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.179+0000 7fd02b000640 1 --2- 172.21.15.62:0/2590605741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 0x7fd0080047c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fd010001750 tx=0x7fd0080057f0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.179+0000 7fd029000640 1 -- 172.21.15.62:0/2590605741 reap_dead start 2024-03-23T11:28:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.180+0000 7fd02b000640 1 -- 172.21.15.62:0/2590605741 shutdown_connections 2024-03-23T11:28:13.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.180+0000 7fd02b000640 1 -- 172.21.15.62:0/2590605741 >> 172.21.15.62:0/2590605741 conn(0x7fd0240b02e0 msgr2=0x7fd0240b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:13.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.180+0000 7fd02b000640 1 -- 172.21.15.62:0/2590605741 shutdown_connections 2024-03-23T11:28:13.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.180+0000 7fd02b000640 1 -- 172.21.15.62:0/2590605741 wait complete. 2024-03-23T11:28:13.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.182+0000 7fd02b000640 1 Processor -- start 2024-03-23T11:28:13.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.182+0000 7fd02b000640 1 -- start start 2024-03-23T11:28:13.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd02b000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 0x7fd0240a8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd02b000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0240a7780 0x7fd02413bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd02b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0240abab0 0x7fd0240a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd02b000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0240a8bb0 con 0x7fd008003f70 2024-03-23T11:28:13.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd02b000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0240a8e60 con 0x7fd0240a7780 2024-03-23T11:28:13.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd02b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02415be70 con 0x7fd0240abab0 2024-03-23T11:28:13.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd023e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 0x7fd0240a8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd023e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 0x7fd0240a8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50822/0 (socket says 172.21.15.62:50822) 2024-03-23T11:28:13.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd023e00640 1 -- 172.21.15.62:0/112860102 learned_addr learned my addr 172.21.15.62:0/112860102 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd029000640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0240abab0 0x7fd0240a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd023e00640 1 -- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0240abab0 msgr2=0x7fd0240a9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd023e00640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0240abab0 0x7fd0240a9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd023e00640 1 -- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0240a7780 msgr2=0x7fd02413bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.183+0000 7fd023400640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0240a7780 0x7fd02413bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.184+0000 7fd023e00640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0240a7780 0x7fd02413bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.184+0000 7fd023e00640 1 -- 172.21.15.62:0/112860102 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02415c150 con 0x7fd008003f70 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.184+0000 7fd023e00640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 0x7fd0240a8670 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fd01400e730 tx=0x7fd01400eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.184+0000 7fd01be00640 1 -- 172.21.15.62:0/112860102 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01400c6c0 con 0x7fd008003f70 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.184+0000 7fd01be00640 1 -- 172.21.15.62:0/112860102 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01400c820 con 0x7fd008003f70 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.184+0000 7fd01be00640 1 -- 172.21.15.62:0/112860102 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01400cbd0 con 0x7fd008003f70 2024-03-23T11:28:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.184+0000 7fd02b000640 1 -- 172.21.15.62:0/112860102 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02415c430 con 0x7fd008003f70 2024-03-23T11:28:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.184+0000 7fd02b000640 1 -- 172.21.15.62:0/112860102 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02413c530 con 0x7fd008003f70 2024-03-23T11:28:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.186+0000 7fd02b000640 1 -- 172.21.15.62:0/112860102 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02415c950 con 0x7fd008003f70 2024-03-23T11:28:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.186+0000 7fd01be00640 1 -- 172.21.15.62:0/112860102 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd01400cd30 con 0x7fd008003f70 2024-03-23T11:28:13.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.188+0000 7fd01be00640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd00406a7d0 0x7fd00406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.188+0000 7fd01be00640 1 -- 172.21.15.62:0/112860102 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fd014088df0 con 0x7fd008003f70 2024-03-23T11:28:13.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.189+0000 7fd023400640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd00406a7d0 0x7fd00406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.190+0000 7fd023400640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd00406a7d0 0x7fd00406cc90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd010033000 tx=0x7fd010009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.193+0000 7fd01be00640 1 -- 172.21.15.62:0/112860102 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd02415c950 con 0x7fd008003f70 2024-03-23T11:28:13.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.434+0000 7fd02b000640 1 -- 172.21.15.62:0/112860102 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd02415c950 con 0x7fd008003f70 2024-03-23T11:28:13.428 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:28:13.436+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:28:13.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.455+0000 7fd01be00640 1 -- 172.21.15.62:0/112860102 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v65) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd02415c950 con 0x7fd008003f70 2024-03-23T11:28:13.447 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.458+0000 7fd02b000640 1 -- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd00406a7d0 msgr2=0x7fd00406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.458+0000 7fd02b000640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd00406a7d0 0x7fd00406cc90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd010033000 tx=0x7fd010009030 comp rx=0 tx=0).stop 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.458+0000 7fd02b000640 1 -- 172.21.15.62:0/112860102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 msgr2=0x7fd0240a8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.458+0000 7fd02b000640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 0x7fd0240a8670 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fd01400e730 tx=0x7fd01400eb30 comp rx=0 tx=0).stop 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.458+0000 7fd02b000640 1 -- 172.21.15.62:0/112860102 shutdown_connections 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.458+0000 7fd02b000640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd00406a7d0 0x7fd00406cc90 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.458+0000 7fd02b000640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0240abab0 0x7fd0240a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.458+0000 7fd02b000640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0240a7780 0x7fd02413bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.458+0000 7fd02b000640 1 --2- 172.21.15.62:0/112860102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008003f70 0x7fd0240a8670 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.458+0000 7fd02b000640 1 -- 172.21.15.62:0/112860102 >> 172.21.15.62:0/112860102 conn(0x7fd0240b02e0 msgr2=0x7fd02413ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.459+0000 7fd02b000640 1 -- 172.21.15.62:0/112860102 shutdown_connections 2024-03-23T11:28:13.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.459+0000 7fd02b000640 1 -- 172.21.15.62:0/112860102 wait complete. 2024-03-23T11:28:13.466 INFO:tasks.ceph.mds.a.smithi062.stderr: -4168> 2024-03-23T11:26:57.376+0000 7fd996000640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-03-23T11:28:13.468 INFO:tasks.ceph.mds.a.smithi062.stderr: -3961> 2024-03-23T11:26:57.657+0000 7fd996000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 ap=1 f(v0 m2024-03-23T11:26:34.599368+0000 1=1+0) n(v0 rc2024-03-23T11:26:34.723466+0000 2=1+1) (iversion lock) | dirfrag=0 authpin=1 scrubqueue=0 0x55c63f9e2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-23T11:26:34.599368+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-23T11:26:34.723466+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-03-23T11:28:13.469 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-03-23T11:28:13.480 INFO:tasks.ceph.mds.a.smithi062.stderr: -2374> 2024-03-23T11:27:29.817+0000 7fd996000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 pv17 ap=3 DIRTYPARENT f(v0 m2024-03-23T11:26:34.599368+0000 1=1+0)->f(v1 m2024-03-23T11:26:34.599368+0000) n(v0 rc2024-03-23T11:26:34.723466+0000 2=1+1)->n(v0 rc2024-03-23T11:26:34.723466+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55c63f9e2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-03-23T11:26:34.599368+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-23T11:26:34.723466+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-03-23T11:28:13.538 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:28:13.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.583+0000 7febef6b5740 1 -- start start 2024-03-23T11:28:13.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.583+0000 7febef6b5740 1 Processor -- start 2024-03-23T11:28:13.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.584+0000 7febef6b5740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd14d6c00 0x558bd155e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.584+0000 7febef6b5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd14d7800 0x558bd155eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.584+0000 7febef6b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd14d7c00 0x558bd155f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.584+0000 7febef6b5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd15a8780 con 0x558bd14d7800 2024-03-23T11:28:13.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.584+0000 7febef6b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd15a8a00 con 0x558bd14d7c00 2024-03-23T11:28:13.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.584+0000 7febef6b5740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558bd15a8640 con 0x558bd14d6c00 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.585+0000 7febece00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd14d7800 0x558bd155eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.585+0000 7febece00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd14d7800 0x558bd155eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50848/0 (socket says 172.21.15.62:50848) 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.585+0000 7febece00640 1 -- 172.21.15.62:0/2387813333 learned_addr learned my addr 172.21.15.62:0/2387813333 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.585+0000 7febed800640 1 --2- 172.21.15.62:0/2387813333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd14d6c00 0x558bd155e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.585+0000 7febee200640 1 --2- 172.21.15.62:0/2387813333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd14d7c00 0x558bd155f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.585+0000 7febece00640 1 -- 172.21.15.62:0/2387813333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd14d6c00 msgr2=0x558bd155e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.585+0000 7febece00640 1 --2- 172.21.15.62:0/2387813333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd14d6c00 0x558bd155e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.585+0000 7febece00640 1 -- 172.21.15.62:0/2387813333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd14d7c00 msgr2=0x558bd155f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.585+0000 7febece00640 1 --2- 172.21.15.62:0/2387813333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd14d7c00 0x558bd155f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.585+0000 7febece00640 1 -- 172.21.15.62:0/2387813333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558bd14fc9c0 con 0x558bd14d7800 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.586+0000 7febece00640 1 --2- 172.21.15.62:0/2387813333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd14d7800 0x558bd155eb00 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x558bd22ef290 tx=0x558bd23a2060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d9d0429e8ba277 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.586+0000 7febea600640 1 -- 172.21.15.62:0/2387813333 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558bd151d080 con 0x558bd14d7800 2024-03-23T11:28:13.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.586+0000 7febea600640 1 -- 172.21.15.62:0/2387813333 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558bd15ba300 con 0x558bd14d7800 2024-03-23T11:28:13.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.586+0000 7febea600640 1 -- 172.21.15.62:0/2387813333 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558bd151d600 con 0x558bd14d7800 2024-03-23T11:28:13.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.586+0000 7febef6b5740 1 -- 172.21.15.62:0/2387813333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558bd14fcd00 con 0x558bd14d7800 2024-03-23T11:28:13.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.586+0000 7febef6b5740 1 -- 172.21.15.62:0/2387813333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x558bd14fc9c0 con 0x558bd14d7800 2024-03-23T11:28:13.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.586+0000 7febef6b5740 4 waiting for MDS map... 2024-03-23T11:28:13.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.586+0000 7febea600640 1 -- 172.21.15.62:0/2387813333 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x558bd152cc40 con 0x558bd14d7800 2024-03-23T11:28:13.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.587+0000 7febea600640 1 -- 172.21.15.62:0/2387813333 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 65) v1 ==== 2573+0+0 (secure 0 0 0) 0x558bd1526f00 con 0x558bd14d7800 2024-03-23T11:28:13.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.587+0000 7febef6b5740 4 Got MDS map 65 2024-03-23T11:28:13.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.587+0000 7febef6b5740 4 datascan.main: Using metadata pool output 2024-03-23T11:28:13.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.587+0000 7febef6b5740 4 datascan.main: connecting to RADOS... 2024-03-23T11:28:13.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.587+0000 7febef6b5740 1 Processor -- start 2024-03-23T11:28:13.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.588+0000 7febef6b5740 1 -- start start 2024-03-23T11:28:13.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.588+0000 7febef6b5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd22a3c00 0x558bd155f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.588+0000 7febef6b5740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd2408000 0x558bd155fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.588+0000 7febef6b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd2408400 0x558bd1560100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.588+0000 7febef6b5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd151fe00 con 0x558bd22a3c00 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.588+0000 7febef6b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd15a8a00 con 0x558bd2408400 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.588+0000 7febef6b5740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558bd15a8780 con 0x558bd2408000 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.589+0000 7febee200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd22a3c00 0x558bd155f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.589+0000 7febee200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd22a3c00 0x558bd155f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50864/0 (socket says 172.21.15.62:50864) 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.589+0000 7febee200640 1 -- 172.21.15.62:0/2769028203 learned_addr learned my addr 172.21.15.62:0/2769028203 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.589+0000 7febed800640 1 --2- 172.21.15.62:0/2769028203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd2408000 0x558bd155fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.589+0000 7febee200640 1 -- 172.21.15.62:0/2769028203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd2408000 msgr2=0x558bd155fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.589+0000 7febee200640 1 --2- 172.21.15.62:0/2769028203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd2408000 0x558bd155fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.589+0000 7febee200640 1 -- 172.21.15.62:0/2769028203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd2408400 msgr2=0x558bd1560100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.589+0000 7febee200640 1 --2- 172.21.15.62:0/2769028203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd2408400 0x558bd1560100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.589+0000 7febee200640 1 -- 172.21.15.62:0/2769028203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558bd14fcb60 con 0x558bd22a3c00 2024-03-23T11:28:13.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.589+0000 7febee200640 1 --2- 172.21.15.62:0/2769028203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd22a3c00 0x558bd155f600 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x558bd239c8d0 tx=0x558bd23bec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f79b3e3403c80bc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.590+0000 7febe7e00640 1 -- 172.21.15.62:0/2769028203 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558bd151d080 con 0x558bd22a3c00 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.590+0000 7febe7e00640 1 -- 172.21.15.62:0/2769028203 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558bd15baa80 con 0x558bd22a3c00 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.590+0000 7febe7e00640 1 -- 172.21.15.62:0/2769028203 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558bd151d600 con 0x558bd22a3c00 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.590+0000 7febef6b5740 1 -- 172.21.15.62:0/2769028203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd22a3c00 msgr2=0x558bd155f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.590+0000 7febef6b5740 1 --2- 172.21.15.62:0/2769028203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd22a3c00 0x558bd155f600 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x558bd239c8d0 tx=0x558bd23bec00 comp rx=0 tx=0).stop 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.590+0000 7febef6b5740 1 -- 172.21.15.62:0/2769028203 shutdown_connections 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.590+0000 7febef6b5740 1 --2- 172.21.15.62:0/2769028203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd2408400 0x558bd1560100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.590+0000 7febef6b5740 1 --2- 172.21.15.62:0/2769028203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd2408000 0x558bd155fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.590+0000 7febef6b5740 1 --2- 172.21.15.62:0/2769028203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd22a3c00 0x558bd155f600 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.590+0000 7febef6b5740 1 -- 172.21.15.62:0/2769028203 >> 172.21.15.62:0/2769028203 conn(0x558bd22a3800 msgr2=0x558bd15dd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.590+0000 7febef6b5740 1 -- 172.21.15.62:0/2769028203 shutdown_connections 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.591+0000 7febef6b5740 1 -- 172.21.15.62:0/2769028203 wait complete. 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.591+0000 7febef6b5740 1 Processor -- start 2024-03-23T11:28:13.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.591+0000 7febef6b5740 1 -- start start 2024-03-23T11:28:13.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.592+0000 7febef6b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd22a3800 0x558bd1561180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.592+0000 7febef6b5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd2409000 0x558bd1560c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.592+0000 7febef6b5740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd2408c00 0x558bd1560680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.592+0000 7febef6b5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd15a8500 con 0x558bd2409000 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.592+0000 7febef6b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd15a83c0 con 0x558bd22a3800 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.592+0000 7febef6b5740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558bd15a8280 con 0x558bd2408c00 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.592+0000 7febed800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd2409000 0x558bd1560c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.592+0000 7febed800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd2409000 0x558bd1560c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50874/0 (socket says 172.21.15.62:50874) 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.592+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 learned_addr learned my addr 172.21.15.62:0/2541096646 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.593+0000 7febece00640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd2408c00 0x558bd1560680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.593+0000 7febee200640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd22a3800 0x558bd1561180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.593+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd2408c00 msgr2=0x558bd1560680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.593+0000 7febed800640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd2408c00 0x558bd1560680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.593+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd22a3800 msgr2=0x558bd1561180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.593+0000 7febed800640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd22a3800 0x558bd1561180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.593+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558bd14fc820 con 0x558bd2409000 2024-03-23T11:28:13.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.593+0000 7febed800640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd2409000 0x558bd1560c00 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x558bd239d5f0 tx=0x558bd23bed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.593+0000 7febe6000640 1 -- 172.21.15.62:0/2541096646 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558bd151d080 con 0x558bd2409000 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.593+0000 7febe6000640 1 -- 172.21.15.62:0/2541096646 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558bd1532180 con 0x558bd2409000 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.593+0000 7febe6000640 1 -- 172.21.15.62:0/2541096646 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558bd151d600 con 0x558bd2409000 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.594+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558bd14fc4e0 con 0x558bd2409000 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.594+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558bd14fc1a0 con 0x558bd2409000 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.594+0000 7febef6b5740 4 datascan.init: resolving metadata pool 12 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.595+0000 7febe6000640 1 -- 172.21.15.62:0/2541096646 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x558bd1508840 con 0x558bd2409000 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.596+0000 7febe6000640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558bd2420c00 0x558bd1561700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.596+0000 7febe6000640 1 -- 172.21.15.62:0/2541096646 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x558bd152ca80 con 0x558bd2409000 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.596+0000 7febef6b5740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.596+0000 7febef6b5740 4 datascan.main: data pool 'cephfs_data' has ID 13 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.596+0000 7febef6b5740 4 datascan.main: opening data pool 'cephfs_data' 2024-03-23T11:28:13.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.596+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x558bd2408800 0x558bd1560100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.596+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:1 13.0 13:00000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2408400 con 0x558bd2408800 2024-03-23T11:28:13.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.597+0000 7febee200640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558bd2420c00 0x558bd1561700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.597+0000 7febece00640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x558bd2408800 0x558bd1560100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.598+0000 7febee200640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558bd2420c00 0x558bd1561700 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x558bd239c8d0 tx=0x558bd226b140 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.598+0000 7febece00640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x558bd2408800 0x558bd1560100 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.599+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.7 v2:172.21.15.120:6816/37655 1 ==== osd_op_reply(1 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2408800 2024-03-23T11:28:13.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.599+0000 7febe7400640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x558bd2421000 0x558bd185a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.599+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:2 13.20 13:04000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2421400 con 0x558bd2421000 2024-03-23T11:28:13.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.600+0000 7febee200640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x558bd2421000 0x558bd185a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.600+0000 7febee200640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x558bd2421000 0x558bd185a000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.608+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== osd_op_reply(2 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd2421000 2024-03-23T11:28:13.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.608+0000 7febe7e00640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x558bd2421c00 0x558bd185a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.608+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:3 13.10 13:08000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d2000 con 0x558bd2421c00 2024-03-23T11:28:13.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.608+0000 7febed800640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x558bd2421c00 0x558bd185a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.609+0000 7febed800640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x558bd2421c00 0x558bd185a580 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.612+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(3 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd2421c00 2024-03-23T11:28:13.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.612+0000 7febe7400640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x558bd2421800 0x558bd185ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.612+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:4 13.30 13:0c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d2800 con 0x558bd2421800 2024-03-23T11:28:13.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.613+0000 7febece00640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x558bd2421800 0x558bd185ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.614+0000 7febece00640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x558bd2421800 0x558bd185ab00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.618+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== osd_op_reply(4 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2421800 2024-03-23T11:28:13.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.618+0000 7febe7e00640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x558bd17d2400 0x558bd185b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.618+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:5 13.8 13:10000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d3400 con 0x558bd17d2400 2024-03-23T11:28:13.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.618+0000 7febee200640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x558bd17d2400 0x558bd185b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.619+0000 7febee200640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x558bd17d2400 0x558bd185b600 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.619+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.2 v2:172.21.15.62:6800/36996 1 ==== osd_op_reply(5 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd17d2400 2024-03-23T11:28:13.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.619+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:6 13.28 13:14000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d2c00 con 0x558bd17d2400 2024-03-23T11:28:13.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.620+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.2 v2:172.21.15.62:6800/36996 2 ==== osd_op_reply(6 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd17d2400 2024-03-23T11:28:13.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.620+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:7 13.18 13:18000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d3800 con 0x558bd2408800 2024-03-23T11:28:13.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.621+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.7 v2:172.21.15.120:6816/37655 2 ==== osd_op_reply(7 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2408800 2024-03-23T11:28:13.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.621+0000 7febe7400640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x558bd17d3000 0x558bd185b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.621+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:8 13.38 13:1c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d3400 con 0x558bd17d3000 2024-03-23T11:28:13.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.621+0000 7febed800640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x558bd17d3000 0x558bd185b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.622+0000 7febed800640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x558bd17d3000 0x558bd185b080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.623+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== osd_op_reply(8 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd17d3000 2024-03-23T11:28:13.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.623+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:9 13.4 13:20000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d3c00 con 0x558bd2421000 2024-03-23T11:28:13.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.624+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.6 v2:172.21.15.120:6800/37641 2 ==== osd_op_reply(9 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd2421000 2024-03-23T11:28:13.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.624+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:10 13.24 13:24000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d2c00 con 0x558bd2421000 2024-03-23T11:28:13.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.625+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.6 v2:172.21.15.120:6800/37641 3 ==== osd_op_reply(10 [pgnls start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd2421000 2024-03-23T11:28:13.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.625+0000 7febe7e00640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x558bd1874000 0x558bd185bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.625+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:11 13.14 13:28000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874400 con 0x558bd1874000 2024-03-23T11:28:13.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.625+0000 7febece00640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x558bd1874000 0x558bd185bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.626+0000 7febece00640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x558bd1874000 0x558bd185bb80 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.627+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== osd_op_reply(11 [pgnls start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd1874000 2024-03-23T11:28:13.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.627+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:12 13.34 13:2c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2421400 con 0x558bd2421000 2024-03-23T11:28:13.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.628+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.6 v2:172.21.15.120:6800/37641 4 ==== osd_op_reply(12 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd2421000 2024-03-23T11:28:13.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.628+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:13 13.c 13:30000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874800 con 0x558bd2421c00 2024-03-23T11:28:13.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.628+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.1 v2:172.21.15.62:6824/37001 2 ==== osd_op_reply(13 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd2421c00 2024-03-23T11:28:13.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.628+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:14 13.2c 13:34000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2409c00 con 0x558bd17d2400 2024-03-23T11:28:13.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.629+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.2 v2:172.21.15.62:6800/36996 3 ==== osd_op_reply(14 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd17d2400 2024-03-23T11:28:13.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.629+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:15 13.1c 13:38000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874c00 con 0x558bd2421000 2024-03-23T11:28:13.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.630+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.6 v2:172.21.15.120:6800/37641 5 ==== osd_op_reply(15 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd2421000 2024-03-23T11:28:13.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.630+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:16 13.3c 13:3c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d3c00 con 0x558bd17d2400 2024-03-23T11:28:13.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.630+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.2 v2:172.21.15.62:6800/36996 4 ==== osd_op_reply(16 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd17d2400 2024-03-23T11:28:13.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.630+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:17 13.2 13:40000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874c00 con 0x558bd17d3000 2024-03-23T11:28:13.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.631+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.5 v2:172.21.15.120:6824/37642 2 ==== osd_op_reply(17 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd17d3000 2024-03-23T11:28:13.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.631+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:18 13.22 13:44000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2409c00 con 0x558bd2421c00 2024-03-23T11:28:13.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.632+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.1 v2:172.21.15.62:6824/37001 3 ==== osd_op_reply(18 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd2421c00 2024-03-23T11:28:13.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.632+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:19 13.12 13:48000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874400 con 0x558bd1874000 2024-03-23T11:28:13.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.632+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.0 v2:172.21.15.62:6811/36998 2 ==== osd_op_reply(19 [pgnls start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd1874000 2024-03-23T11:28:13.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.632+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:20 13.32 13:4c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2408400 con 0x558bd17d2400 2024-03-23T11:28:13.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.633+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.2 v2:172.21.15.62:6800/36996 5 ==== osd_op_reply(20 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd17d2400 2024-03-23T11:28:13.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.633+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:21 13.a 13:50000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d2c00 con 0x558bd17d2400 2024-03-23T11:28:13.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.633+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.2 v2:172.21.15.62:6800/36996 6 ==== osd_op_reply(21 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd17d2400 2024-03-23T11:28:13.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.633+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:22 13.2a 13:54000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd22a3400 con 0x558bd2421c00 2024-03-23T11:28:13.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.634+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.1 v2:172.21.15.62:6824/37001 4 ==== osd_op_reply(22 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd2421c00 2024-03-23T11:28:13.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.634+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:23 13.1a 13:58000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d2800 con 0x558bd2421c00 2024-03-23T11:28:13.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.634+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.1 v2:172.21.15.62:6824/37001 5 ==== osd_op_reply(23 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd2421c00 2024-03-23T11:28:13.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.634+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:24 13.3a 13:5c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2421400 con 0x558bd17d3000 2024-03-23T11:28:13.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.635+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.5 v2:172.21.15.120:6824/37642 3 ==== osd_op_reply(24 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd17d3000 2024-03-23T11:28:13.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.636+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:25 13.6 13:60000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d3800 con 0x558bd2421800 2024-03-23T11:28:13.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.636+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.4 v2:172.21.15.120:6803/37640 2 ==== osd_op_reply(25 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2421800 2024-03-23T11:28:13.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.637+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:26 13.26 13:64000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874800 con 0x558bd2408800 2024-03-23T11:28:13.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.638+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.7 v2:172.21.15.120:6816/37655 3 ==== osd_op_reply(26 [pgnls start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2408800 2024-03-23T11:28:13.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.638+0000 7febe7e00640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x558bd1875800 0x558bd185c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.638+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:27 13.16 13:68000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1875c00 con 0x558bd1875800 2024-03-23T11:28:13.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.638+0000 7febee200640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x558bd1875800 0x558bd185c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.639+0000 7febee200640 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x558bd1875800 0x558bd185c100 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.639+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== osd_op_reply(27 [pgnls start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd1875800 2024-03-23T11:28:13.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.639+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:28 13.36 13:6c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874c00 con 0x558bd2421000 2024-03-23T11:28:13.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.640+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.6 v2:172.21.15.120:6800/37641 6 ==== osd_op_reply(28 [pgnls start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd2421000 2024-03-23T11:28:13.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.640+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:29 13.e 13:70000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1886000 con 0x558bd2408800 2024-03-23T11:28:13.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.641+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.7 v2:172.21.15.120:6816/37655 4 ==== osd_op_reply(29 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2408800 2024-03-23T11:28:13.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.641+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:30 13.2e 13:74000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1875000 con 0x558bd17d2400 2024-03-23T11:28:13.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.642+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.2 v2:172.21.15.62:6800/36996 7 ==== osd_op_reply(30 [pgnls start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd17d2400 2024-03-23T11:28:13.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.642+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:31 13.1e 13:78000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1886400 con 0x558bd2421800 2024-03-23T11:28:13.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.643+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.4 v2:172.21.15.120:6803/37640 3 ==== osd_op_reply(31 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2421800 2024-03-23T11:28:13.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.643+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:32 13.3e 13:7c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1875400 con 0x558bd2421c00 2024-03-23T11:28:13.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.644+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.1 v2:172.21.15.62:6824/37001 6 ==== osd_op_reply(32 [pgnls start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd2421c00 2024-03-23T11:28:13.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.644+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:33 13.1 13:80000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1886800 con 0x558bd2421000 2024-03-23T11:28:13.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.645+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.6 v2:172.21.15.120:6800/37641 7 ==== osd_op_reply(33 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd2421000 2024-03-23T11:28:13.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.645+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:34 13.21 13:84000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d2000 con 0x558bd2421800 2024-03-23T11:28:13.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.646+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.4 v2:172.21.15.120:6803/37640 4 ==== osd_op_reply(34 [pgnls start_epoch 93 out=49b] v89'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2421800 2024-03-23T11:28:13.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.646+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:35 13.11 13:88000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1886c00 con 0x558bd1875800 2024-03-23T11:28:13.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.647+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.3 v2:172.21.15.62:6808/37000 2 ==== osd_op_reply(35 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd1875800 2024-03-23T11:28:13.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.647+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:36 13.31 13:8c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d3400 con 0x558bd1875800 2024-03-23T11:28:13.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.647+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.3 v2:172.21.15.62:6808/37000 3 ==== osd_op_reply(36 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd1875800 2024-03-23T11:28:13.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.647+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:37 13.9 13:90000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1887000 con 0x558bd17d3000 2024-03-23T11:28:13.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.648+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.5 v2:172.21.15.120:6824/37642 4 ==== osd_op_reply(37 [pgnls start_epoch 93 out=49b] v89'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd17d3000 2024-03-23T11:28:13.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.648+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:38 13.29 13:94000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d3800 con 0x558bd2421000 2024-03-23T11:28:13.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.649+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.6 v2:172.21.15.120:6800/37641 8 ==== osd_op_reply(38 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd2421000 2024-03-23T11:28:13.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.649+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:39 13.19 13:98000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1887400 con 0x558bd1875800 2024-03-23T11:28:13.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.650+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.3 v2:172.21.15.62:6808/37000 4 ==== osd_op_reply(39 [pgnls start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd1875800 2024-03-23T11:28:13.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.650+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:40 13.39 13:9c000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d2000 con 0x558bd17d2400 2024-03-23T11:28:13.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.651+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.2 v2:172.21.15.62:6800/36996 8 ==== osd_op_reply(40 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd17d2400 2024-03-23T11:28:13.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.651+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:41 13.5 13:a0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1887800 con 0x558bd2421800 2024-03-23T11:28:13.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.652+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.4 v2:172.21.15.120:6803/37640 5 ==== osd_op_reply(41 [pgnls start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2421800 2024-03-23T11:28:13.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.652+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:42 13.25 13:a4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d2800 con 0x558bd2421800 2024-03-23T11:28:13.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.653+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.4 v2:172.21.15.120:6803/37640 6 ==== osd_op_reply(42 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2421800 2024-03-23T11:28:13.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.653+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:43 13.15 13:a8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874400 con 0x558bd2421800 2024-03-23T11:28:13.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.654+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.4 v2:172.21.15.120:6803/37640 7 ==== osd_op_reply(43 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2421800 2024-03-23T11:28:13.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.654+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:44 13.35 13:ac000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2409c00 con 0x558bd2421c00 2024-03-23T11:28:13.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.654+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.1 v2:172.21.15.62:6824/37001 7 ==== osd_op_reply(44 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd2421c00 2024-03-23T11:28:13.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.654+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:45 13.d 13:b0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874800 con 0x558bd2421800 2024-03-23T11:28:13.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.656+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.4 v2:172.21.15.120:6803/37640 8 ==== osd_op_reply(45 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2421800 2024-03-23T11:28:13.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.656+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:46 13.2d 13:b4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2408400 con 0x558bd1874000 2024-03-23T11:28:13.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.656+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.0 v2:172.21.15.62:6811/36998 3 ==== osd_op_reply(46 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd1874000 2024-03-23T11:28:13.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.656+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:47 13.1d 13:b8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1875400 con 0x558bd2408800 2024-03-23T11:28:13.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.657+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.7 v2:172.21.15.120:6816/37655 5 ==== osd_op_reply(47 [pgnls start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2408800 2024-03-23T11:28:13.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.657+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:48 13.3d 13:bc000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2421400 con 0x558bd2408800 2024-03-23T11:28:13.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.659+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.7 v2:172.21.15.120:6816/37655 6 ==== osd_op_reply(48 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2408800 2024-03-23T11:28:13.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.659+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:49 13.3 13:c0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1875c00 con 0x558bd1874000 2024-03-23T11:28:13.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.659+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.0 v2:172.21.15.62:6811/36998 4 ==== osd_op_reply(49 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd1874000 2024-03-23T11:28:13.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.659+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:50 13.23 13:c4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd22a3400 con 0x558bd2421c00 2024-03-23T11:28:13.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.660+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.1 v2:172.21.15.62:6824/37001 8 ==== osd_op_reply(50 [pgnls start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd2421c00 2024-03-23T11:28:13.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.660+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:51 13.13 13:c8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874c00 con 0x558bd2408800 2024-03-23T11:28:13.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.661+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.7 v2:172.21.15.120:6816/37655 7 ==== osd_op_reply(51 [pgnls start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2408800 2024-03-23T11:28:13.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.661+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:52 13.33 13:cc000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1887800 con 0x558bd2408800 2024-03-23T11:28:13.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.662+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.7 v2:172.21.15.120:6816/37655 8 ==== osd_op_reply(52 [pgnls start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2408800 2024-03-23T11:28:13.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.662+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:53 13.b 13:d0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1875000 con 0x558bd17d3000 2024-03-23T11:28:13.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.664+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.5 v2:172.21.15.120:6824/37642 5 ==== osd_op_reply(53 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd17d3000 2024-03-23T11:28:13.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.664+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:54 13.2b 13:d4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1886000 con 0x558bd17d2400 2024-03-23T11:28:13.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.664+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.2 v2:172.21.15.62:6800/36996 9 ==== osd_op_reply(54 [pgnls start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd17d2400 2024-03-23T11:28:13.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.664+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:55 13.1b 13:d8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d3c00 con 0x558bd2421800 2024-03-23T11:28:13.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.666+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.4 v2:172.21.15.120:6803/37640 9 ==== osd_op_reply(55 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2421800 2024-03-23T11:28:13.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.666+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:56 13.3b 13:dc000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2421400 con 0x558bd2421800 2024-03-23T11:28:13.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.667+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.4 v2:172.21.15.120:6803/37640 10 ==== osd_op_reply(56 [pgnls start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2421800 2024-03-23T11:28:13.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.667+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:57 13.7 13:e0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd17d2c00 con 0x558bd2408800 2024-03-23T11:28:13.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.668+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.7 v2:172.21.15.120:6816/37655 9 ==== osd_op_reply(57 [pgnls start_epoch 93 out=81b] v91'6 uv6 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2408800 2024-03-23T11:28:13.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.668+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:58 13.7 13:e2e07930:::10000000001.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd2408000 con 0x558bd2408800 2024-03-23T11:28:13.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.670+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.7 v2:172.21.15.120:6816/37655 10 ==== osd_op_reply(58 10000000001.00000000 [stat out=16b] v0'0 uv6 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2408800 2024-03-23T11:28:13.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.670+0000 7febef6b5740 10 datascan.operator(): handling object 1099511627777.0 2024-03-23T11:28:13.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.670+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:59 13.7 13:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1886800 con 0x558bd2408800 2024-03-23T11:28:13.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.674+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.7 v2:172.21.15.120:6816/37655 11 ==== osd_op_reply(59 10000000001.00000000 [call] v93'7 uv7 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2408800 2024-03-23T11:28:13.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.674+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:60 13.27 13:e4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1886c00 con 0x558bd2421c00 2024-03-23T11:28:13.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.675+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.1 v2:172.21.15.62:6824/37001 9 ==== osd_op_reply(60 [pgnls start_epoch 93 out=49b] v89'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd2421c00 2024-03-23T11:28:13.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.675+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:61 13.17 13:e8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1887800 con 0x558bd17d3000 2024-03-23T11:28:13.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.676+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.5 v2:172.21.15.120:6824/37642 6 ==== osd_op_reply(61 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd17d3000 2024-03-23T11:28:13.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.676+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:62 13.37 13:ec000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874400 con 0x558bd17d3000 2024-03-23T11:28:13.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.677+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 <== osd.5 v2:172.21.15.120:6824/37642 7 ==== osd_op_reply(62 [pgnls start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0fc0 con 0x558bd17d3000 2024-03-23T11:28:13.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.677+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:63 13.f 13:f0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1886800 con 0x558bd2421000 2024-03-23T11:28:13.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.678+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.6 v2:172.21.15.120:6800/37641 9 ==== osd_op_reply(63 [pgnls start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd2421000 2024-03-23T11:28:13.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.678+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:64 13.2f 13:f4000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1874800 con 0x558bd2421800 2024-03-23T11:28:13.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.680+0000 7febece00640 1 -- 172.21.15.62:0/2541096646 <== osd.4 v2:172.21.15.120:6803/37640 11 ==== osd_op_reply(64 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0b40 con 0x558bd2421800 2024-03-23T11:28:13.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.680+0000 7febe7e00640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:65 13.1f 13:f8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1887c00 con 0x558bd2421000 2024-03-23T11:28:13.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.681+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.6 v2:172.21.15.120:6800/37641 10 ==== osd_op_reply(65 [pgnls start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd2421000 2024-03-23T11:28:13.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.681+0000 7febe7400640 1 -- 172.21.15.62:0/2541096646 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:66 13.3f 13:fc000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x558bd1875400 con 0x558bd2421000 2024-03-23T11:28:13.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febee200640 1 -- 172.21.15.62:0/2541096646 <== osd.6 v2:172.21.15.120:6800/37641 11 ==== osd_op_reply(66 [pgnls start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558bd22a0d80 con 0x558bd2421000 2024-03-23T11:28:13.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x558bd1874000 msgr2=0x558bd185bb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x558bd1874000 0x558bd185bb80 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x558bd2421c00 msgr2=0x558bd185a580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x558bd2421c00 0x558bd185a580 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x558bd17d2400 msgr2=0x558bd185b600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x558bd17d2400 0x558bd185b600 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x558bd1875800 msgr2=0x558bd185c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x558bd1875800 0x558bd185c100 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x558bd2421800 msgr2=0x558bd185ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x558bd2421800 0x558bd185ab00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x558bd17d3000 msgr2=0x558bd185b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x558bd17d3000 0x558bd185b080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x558bd2421000 msgr2=0x558bd185a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x558bd2421000 0x558bd185a000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x558bd2408800 msgr2=0x558bd1560100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x558bd2408800 0x558bd1560100 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 reap_dead start 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 reap_dead start 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.682+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 reap_dead start 2024-03-23T11:28:13.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.683+0000 7febed800640 1 -- 172.21.15.62:0/2541096646 reap_dead start 2024-03-23T11:28:13.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.683+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558bd2420c00 msgr2=0x558bd1561700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.683+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558bd2420c00 0x558bd1561700 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x558bd239c8d0 tx=0x558bd226b140 comp rx=0 tx=0).stop 2024-03-23T11:28:13.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.683+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd2409000 msgr2=0x558bd1560c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.683+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd2409000 0x558bd1560c00 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x558bd239d5f0 tx=0x558bd23bed20 comp rx=0 tx=0).stop 2024-03-23T11:28:13.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.683+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 shutdown_connections 2024-03-23T11:28:13.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.683+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558bd2420c00 0x558bd1561700 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.683+0000 7febef6b5740 1 --2- 172.21.15.62:0/2541096646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd2409000 0x558bd1560c00 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.683+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 >> 172.21.15.62:0/2541096646 conn(0x558bd22a3c00 msgr2=0x558bd15dd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:13.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.684+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 shutdown_connections 2024-03-23T11:28:13.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.684+0000 7febef6b5740 1 -- 172.21.15.62:0/2541096646 wait complete. 2024-03-23T11:28:13.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.684+0000 7febef6b5740 1 -- 172.21.15.62:0/2387813333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd14d7800 msgr2=0x558bd155eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.684+0000 7febef6b5740 1 --2- 172.21.15.62:0/2387813333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd14d7800 0x558bd155eb00 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x558bd22ef290 tx=0x558bd23a2060 comp rx=0 tx=0).stop 2024-03-23T11:28:13.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.685+0000 7febef6b5740 1 -- 172.21.15.62:0/2387813333 shutdown_connections 2024-03-23T11:28:13.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.685+0000 7febef6b5740 1 --2- 172.21.15.62:0/2387813333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558bd14d7800 0x558bd155eb00 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.685+0000 7febef6b5740 1 --2- 172.21.15.62:0/2387813333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558bd14d7c00 0x558bd155f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.685+0000 7febef6b5740 1 --2- 172.21.15.62:0/2387813333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558bd14d6c00 0x558bd155e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.685+0000 7febef6b5740 1 -- 172.21.15.62:0/2387813333 >> 172.21.15.62:0/2387813333 conn(0x558bd14d7000 msgr2=0x558bd15dc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:13.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.685+0000 7febef6b5740 1 -- 172.21.15.62:0/2387813333 shutdown_connections 2024-03-23T11:28:13.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.685+0000 7febef6b5740 1 -- 172.21.15.62:0/2387813333 wait complete. 2024-03-23T11:28:13.686 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.217563, result: 2024-03-23T11:28:13.686 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes cephfs_data 2024-03-23T11:28:13.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.847+0000 7f5cebe3d740 1 -- start start 2024-03-23T11:28:13.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.847+0000 7f5cebe3d740 1 Processor -- start 2024-03-23T11:28:13.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.848+0000 7f5cebe3d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebaf98c00 0x561ebb020580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.848+0000 7f5cebe3d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebaf99800 0x561ebb020b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.848+0000 7f5cebe3d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebaf99c00 0x561ebb021080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.848+0000 7f5cebe3d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ebb06aa00 con 0x561ebaf99c00 2024-03-23T11:28:13.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.848+0000 7f5cebe3d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ebb06a780 con 0x561ebaf98c00 2024-03-23T11:28:13.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.848+0000 7f5cebe3d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ebb06a640 con 0x561ebaf99800 2024-03-23T11:28:13.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.848+0000 7f5cea200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebaf99c00 0x561ebb021080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.848+0000 7f5cea200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebaf99c00 0x561ebb021080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50886/0 (socket says 172.21.15.62:50886) 2024-03-23T11:28:13.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.848+0000 7f5cea200640 1 -- 172.21.15.62:0/1212995627 learned_addr learned my addr 172.21.15.62:0/1212995627 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:13.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.849+0000 7f5ce9800640 1 --2- 172.21.15.62:0/1212995627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebaf98c00 0x561ebb020580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.849+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/1212995627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebaf99800 0x561ebb020b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.849+0000 7f5cea200640 1 -- 172.21.15.62:0/1212995627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebaf99800 msgr2=0x561ebb020b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:13.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.849+0000 7f5cea200640 1 --2- 172.21.15.62:0/1212995627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebaf99800 0x561ebb020b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.849+0000 7f5cea200640 1 -- 172.21.15.62:0/1212995627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebaf98c00 msgr2=0x561ebb020580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.849+0000 7f5cea200640 1 --2- 172.21.15.62:0/1212995627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebaf98c00 0x561ebb020580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.849+0000 7f5cea200640 1 -- 172.21.15.62:0/1212995627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ebafbe9c0 con 0x561ebaf99c00 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.849+0000 7f5cea200640 1 --2- 172.21.15.62:0/1212995627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebaf99c00 0x561ebb021080 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x561ebbdb5290 tx=0x561ebbdc00c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d72dce40a4a5c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.849+0000 7f5ce6600640 1 -- 172.21.15.62:0/1212995627 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561ebafdf080 con 0x561ebaf99c00 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.849+0000 7f5ce6600640 1 -- 172.21.15.62:0/1212995627 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561ebaff4180 con 0x561ebaf99c00 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.849+0000 7f5ce6600640 1 -- 172.21.15.62:0/1212995627 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561ebafdf600 con 0x561ebaf99c00 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.850+0000 7f5cebe3d740 1 -- 172.21.15.62:0/1212995627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ebafbed00 con 0x561ebaf99c00 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.850+0000 7f5cebe3d740 1 -- 172.21.15.62:0/1212995627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x561ebafbe9c0 con 0x561ebaf99c00 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.850+0000 7f5cebe3d740 4 waiting for MDS map... 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.850+0000 7f5ce6600640 1 -- 172.21.15.62:0/1212995627 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x561ebafeec40 con 0x561ebaf99c00 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.850+0000 7f5ce6600640 1 -- 172.21.15.62:0/1212995627 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 65) v1 ==== 2573+0+0 (secure 0 0 0) 0x561ebafe8f00 con 0x561ebaf99c00 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.850+0000 7f5cebe3d740 4 Got MDS map 65 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.850+0000 7f5cebe3d740 4 datascan.main: Using metadata pool output 2024-03-23T11:28:13.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.850+0000 7f5cebe3d740 4 datascan.main: connecting to RADOS... 2024-03-23T11:28:13.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.851+0000 7f5cebe3d740 1 Processor -- start 2024-03-23T11:28:13.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.851+0000 7f5cebe3d740 1 -- start start 2024-03-23T11:28:13.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.852+0000 7f5cebe3d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebb07dc00 0x561ebb021600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.852+0000 7f5cebe3d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebbec0000 0x561ebb021b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.852+0000 7f5cebe3d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebbec0400 0x561ebb022100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.852+0000 7f5cebe3d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ebafe1e00 con 0x561ebb07dc00 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.852+0000 7f5cebe3d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ebb06a780 con 0x561ebbec0000 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.852+0000 7f5cebe3d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ebb06aa00 con 0x561ebbec0400 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.852+0000 7f5ce8e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebb07dc00 0x561ebb021600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.852+0000 7f5ce8e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebb07dc00 0x561ebb021600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50892/0 (socket says 172.21.15.62:50892) 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.852+0000 7f5ce8e00640 1 -- 172.21.15.62:0/630684559 learned_addr learned my addr 172.21.15.62:0/630684559 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.853+0000 7f5ce9800640 1 --2- 172.21.15.62:0/630684559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebbec0000 0x561ebb021b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.853+0000 7f5ce8e00640 1 -- 172.21.15.62:0/630684559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebbec0400 msgr2=0x561ebb022100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.853+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/630684559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebbec0400 0x561ebb022100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.853+0000 7f5ce8e00640 1 -- 172.21.15.62:0/630684559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebbec0000 msgr2=0x561ebb021b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.853+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/630684559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebbec0000 0x561ebb021b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.853+0000 7f5ce8e00640 1 -- 172.21.15.62:0/630684559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ebafbeb60 con 0x561ebb07dc00 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.853+0000 7f5ce9800640 1 --2- 172.21.15.62:0/630684559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebbec0000 0x561ebb021b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.853+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/630684559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebb07dc00 0x561ebb021600 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x561ebbe64900 tx=0x561ebbe68c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3c98c401b43816 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.853+0000 7f5ce3e00640 1 -- 172.21.15.62:0/630684559 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561ebafdf080 con 0x561ebb07dc00 2024-03-23T11:28:13.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.853+0000 7f5ce3e00640 1 -- 172.21.15.62:0/630684559 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561ebbd74300 con 0x561ebb07dc00 2024-03-23T11:28:13.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.853+0000 7f5ce3e00640 1 -- 172.21.15.62:0/630684559 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561ebafdf600 con 0x561ebb07dc00 2024-03-23T11:28:13.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.854+0000 7f5cebe3d740 1 -- 172.21.15.62:0/630684559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebb07dc00 msgr2=0x561ebb021600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.854+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/630684559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebb07dc00 0x561ebb021600 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x561ebbe64900 tx=0x561ebbe68c00 comp rx=0 tx=0).stop 2024-03-23T11:28:13.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.854+0000 7f5cebe3d740 1 -- 172.21.15.62:0/630684559 shutdown_connections 2024-03-23T11:28:13.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.854+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/630684559 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebbec0400 0x561ebb022100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.854+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/630684559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebbec0000 0x561ebb021b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.854+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/630684559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebb07dc00 0x561ebb021600 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.854+0000 7f5cebe3d740 1 -- 172.21.15.62:0/630684559 >> 172.21.15.62:0/630684559 conn(0x561ebb07d800 msgr2=0x561ebb09f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:13.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.854+0000 7f5cebe3d740 1 -- 172.21.15.62:0/630684559 shutdown_connections 2024-03-23T11:28:13.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.854+0000 7f5cebe3d740 1 -- 172.21.15.62:0/630684559 wait complete. 2024-03-23T11:28:13.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.855+0000 7f5cebe3d740 1 Processor -- start 2024-03-23T11:28:13.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.855+0000 7f5cebe3d740 1 -- start start 2024-03-23T11:28:13.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.856+0000 7f5cebe3d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebb07d800 0x561ebb023180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.856+0000 7f5cebe3d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebbec1000 0x561ebb022c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.856+0000 7f5cebe3d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebbec0c00 0x561ebb022680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.856+0000 7f5cebe3d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ebb06a500 con 0x561ebbec1000 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.856+0000 7f5cebe3d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ebb06a3c0 con 0x561ebb07d800 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.856+0000 7f5cebe3d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ebb06a280 con 0x561ebbec0c00 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.856+0000 7f5ce9800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebbec1000 0x561ebb022c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.856+0000 7f5ce9800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebbec1000 0x561ebb022c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50908/0 (socket says 172.21.15.62:50908) 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.856+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 learned_addr learned my addr 172.21.15.62:0/3971276824 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.857+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebb07d800 0x561ebb023180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.857+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebbec0c00 0x561ebb022680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.857+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebbec0c00 msgr2=0x561ebb022680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.857+0000 7f5ce9800640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebbec0c00 0x561ebb022680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.857+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebb07d800 msgr2=0x561ebb023180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.857+0000 7f5ce9800640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebb07d800 0x561ebb023180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.857+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ebafbe820 con 0x561ebbec1000 2024-03-23T11:28:13.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.857+0000 7f5ce9800640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebbec1000 0x561ebb022c00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x561ebbe655f0 tx=0x561ebbe68d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.857+0000 7f5ce2000640 1 -- 172.21.15.62:0/3971276824 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561ebafdf080 con 0x561ebbec1000 2024-03-23T11:28:13.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.857+0000 7f5ce2000640 1 -- 172.21.15.62:0/3971276824 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561ebbd75500 con 0x561ebbec1000 2024-03-23T11:28:13.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.858+0000 7f5ce2000640 1 -- 172.21.15.62:0/3971276824 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561ebafdf600 con 0x561ebbec1000 2024-03-23T11:28:13.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.858+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561ebafbe4e0 con 0x561ebbec1000 2024-03-23T11:28:13.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.858+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ebafbe1a0 con 0x561ebbec1000 2024-03-23T11:28:13.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.858+0000 7f5cebe3d740 4 datascan.init: resolving metadata pool 12 2024-03-23T11:28:13.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.859+0000 7f5ce2000640 1 -- 172.21.15.62:0/3971276824 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x561ebafca840 con 0x561ebbec1000 2024-03-23T11:28:13.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.860+0000 7f5ce2000640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561ebbee4c00 0x561ebb023700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.860+0000 7f5ce2000640 1 -- 172.21.15.62:0/3971276824 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x561ebafeea80 con 0x561ebbec1000 2024-03-23T11:28:13.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.860+0000 7f5cebe3d740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-23T11:28:13.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.860+0000 7f5cebe3d740 4 datascan.main: data pool 'cephfs_data' has ID 13 2024-03-23T11:28:13.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.860+0000 7f5cebe3d740 4 datascan.main: opening data pool 'cephfs_data' 2024-03-23T11:28:13.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.860+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x561ebbec0800 0x561ebb022100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.860+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec0400 con 0x561ebbec0800 2024-03-23T11:28:13.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.861+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561ebbee4c00 0x561ebb023700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.861+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x561ebbec0800 0x561ebb022100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.862+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x561ebbec0800 0x561ebb022100 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.862+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561ebbee4c00 0x561ebb023700 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x561ebbe64a50 tx=0x561ebbe69ec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.863+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv24 ondisk = 0) v8 ==== 160+0+16 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.863+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x561ebbec0000 0x561ebb021b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.863+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:2 12.1f 12:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbee5000 con 0x561ebbec0000 2024-03-23T11:28:13.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.863+0000 7f5cea200640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x561ebbec0000 0x561ebb021b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.864+0000 7f5cea200640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x561ebbec0000 0x561ebb021b80 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.865+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+16 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.865+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x561ebbee5400 0x561ebb021600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.865+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:3 13.0 13:00000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbee5800 con 0x561ebbee5400 2024-03-23T11:28:13.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.866+0000 7f5ce9800640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x561ebbee5400 0x561ebb021600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.867+0000 7f5ce9800640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x561ebbee5400 0x561ebb021600 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.868+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.868+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:4 13.20 13:04000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328000 con 0x561ebbec0000 2024-03-23T11:28:13.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.869+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 2 ==== osd_op_reply(4 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.869+0000 7f5ce3e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x561ebb328400 0x561ebb32e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.869+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:5 13.10 13:08000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328800 con 0x561ebb328400 2024-03-23T11:28:13.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.869+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x561ebb328400 0x561ebb32e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.870+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x561ebb328400 0x561ebb32e000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.870+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(5 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.870+0000 7f5ce3400640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x561ebb329000 0x561ebb32e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.870+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:6 13.30 13:0c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329400 con 0x561ebb329000 2024-03-23T11:28:13.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.871+0000 7f5cea200640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x561ebb329000 0x561ebb32e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.872+0000 7f5cea200640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x561ebb329000 0x561ebb32e580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.873+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== osd_op_reply(6 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.873+0000 7f5ce3e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x561ebb328c00 0x561ebb32eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.873+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:7 13.8 13:10000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329800 con 0x561ebb328c00 2024-03-23T11:28:13.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.873+0000 7f5ce9800640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x561ebb328c00 0x561ebb32eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.874+0000 7f5ce9800640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x561ebb328c00 0x561ebb32eb00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.874+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 1 ==== osd_op_reply(7 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.874+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:8 13.28 13:14000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336000 con 0x561ebb328c00 2024-03-23T11:28:13.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.875+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 2 ==== osd_op_reply(8 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.875+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:9 13.18 13:18000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329c00 con 0x561ebbee5400 2024-03-23T11:28:13.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.876+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 2 ==== osd_op_reply(9 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.876+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:10 13.38 13:1c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336400 con 0x561ebbec0800 2024-03-23T11:28:13.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.877+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 2 ==== osd_op_reply(10 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.877+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:11 13.4 13:20000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328800 con 0x561ebbec0000 2024-03-23T11:28:13.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.877+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 3 ==== osd_op_reply(11 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.878+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:12 13.24 13:24000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336800 con 0x561ebbec0000 2024-03-23T11:28:13.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.879+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 4 ==== osd_op_reply(12 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.879+0000 7f5ce3e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x561ebbee5800 0x561ebb32f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.879+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:13 13.14 13:28000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec1c00 con 0x561ebbee5800 2024-03-23T11:28:13.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.879+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x561ebbee5800 0x561ebb32f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.880+0000 7f5ce8e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x561ebbee5800 0x561ebb32f080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.880+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== osd_op_reply(13 [pgnls-filter start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.880+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:14 13.34 13:2c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328000 con 0x561ebbec0000 2024-03-23T11:28:13.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.881+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 5 ==== osd_op_reply(14 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.881+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:15 13.c 13:30000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec0400 con 0x561ebb328400 2024-03-23T11:28:13.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.882+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 2 ==== osd_op_reply(15 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.882+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:16 13.2c 13:34000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329400 con 0x561ebb328c00 2024-03-23T11:28:13.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.882+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 3 ==== osd_op_reply(16 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.882+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:17 13.1c 13:38000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336400 con 0x561ebbec0000 2024-03-23T11:28:13.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.883+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 6 ==== osd_op_reply(17 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.883+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:18 13.3c 13:3c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbee5000 con 0x561ebb328c00 2024-03-23T11:28:13.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.884+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 4 ==== osd_op_reply(18 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.884+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:19 13.2 13:40000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336000 con 0x561ebbec0800 2024-03-23T11:28:13.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.885+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 3 ==== osd_op_reply(19 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.885+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:20 13.22 13:44000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb07d400 con 0x561ebb328400 2024-03-23T11:28:13.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.885+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 3 ==== osd_op_reply(20 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.885+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:21 13.12 13:48000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328800 con 0x561ebbee5800 2024-03-23T11:28:13.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.886+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 2 ==== osd_op_reply(21 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.886+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:22 13.32 13:4c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336000 con 0x561ebb328c00 2024-03-23T11:28:13.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.886+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 5 ==== osd_op_reply(22 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.887+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:23 13.a 13:50000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329800 con 0x561ebb328c00 2024-03-23T11:28:13.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.887+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 6 ==== osd_op_reply(23 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.887+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:24 13.2a 13:54000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328800 con 0x561ebb328400 2024-03-23T11:28:13.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.887+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 4 ==== osd_op_reply(24 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.887+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:25 13.1a 13:58000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329c00 con 0x561ebb328400 2024-03-23T11:28:13.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.888+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 5 ==== osd_op_reply(25 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.888+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:26 13.3a 13:5c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb07d400 con 0x561ebbec0800 2024-03-23T11:28:13.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.889+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 4 ==== osd_op_reply(26 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.889+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:27 13.6 13:60000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336c00 con 0x561ebb329000 2024-03-23T11:28:13.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.890+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 2 ==== osd_op_reply(27 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.890+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:28 13.26 13:64000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec1c00 con 0x561ebbee5400 2024-03-23T11:28:13.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.891+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 3 ==== osd_op_reply(28 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.891+0000 7f5ce3e00640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x561ebb337000 0x561ebb32f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:13.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.891+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:29 13.16 13:68000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337400 con 0x561ebb337000 2024-03-23T11:28:13.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.891+0000 7f5cea200640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x561ebb337000 0x561ebb32f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:13.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.892+0000 7f5cea200640 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x561ebb337000 0x561ebb32f600 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:13.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.893+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== osd_op_reply(29 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb337000 2024-03-23T11:28:13.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.893+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:30 13.36 13:6c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec0400 con 0x561ebbec0000 2024-03-23T11:28:13.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.894+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 7 ==== osd_op_reply(30 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.894+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:31 13.e 13:70000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337800 con 0x561ebbee5400 2024-03-23T11:28:13.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.895+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 4 ==== osd_op_reply(31 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.895+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:32 13.2e 13:74000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336800 con 0x561ebb328c00 2024-03-23T11:28:13.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.895+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 7 ==== osd_op_reply(32 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.895+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:33 13.1e 13:78000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337c00 con 0x561ebb329000 2024-03-23T11:28:13.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.896+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 3 ==== osd_op_reply(33 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.897+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:34 13.3e 13:7c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336400 con 0x561ebb328400 2024-03-23T11:28:13.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.897+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 6 ==== osd_op_reply(34 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.897+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:35 13.1 13:80000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec0400 con 0x561ebbec0000 2024-03-23T11:28:13.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.898+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 8 ==== osd_op_reply(35 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.898+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:36 13.21 13:84000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336c00 con 0x561ebb329000 2024-03-23T11:28:13.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.899+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 4 ==== osd_op_reply(36 [pgnls-filter start_epoch 93 out=49b] v89'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.899+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:37 13.11 13:88000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec1c00 con 0x561ebb337000 2024-03-23T11:28:13.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.900+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.3 v2:172.21.15.62:6808/37000 2 ==== osd_op_reply(37 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb337000 2024-03-23T11:28:13.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.900+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:38 13.31 13:8c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337400 con 0x561ebb337000 2024-03-23T11:28:13.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.901+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.3 v2:172.21.15.62:6808/37000 3 ==== osd_op_reply(38 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb337000 2024-03-23T11:28:13.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.901+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:39 13.9 13:90000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328000 con 0x561ebbec0800 2024-03-23T11:28:13.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.902+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 5 ==== osd_op_reply(39 [pgnls-filter start_epoch 93 out=49b] v89'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.902+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:40 13.29 13:94000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336000 con 0x561ebbec0000 2024-03-23T11:28:13.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.903+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 9 ==== osd_op_reply(40 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.903+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:41 13.19 13:98000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329400 con 0x561ebb337000 2024-03-23T11:28:13.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.903+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.3 v2:172.21.15.62:6808/37000 4 ==== osd_op_reply(41 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb337000 2024-03-23T11:28:13.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.904+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:42 13.39 13:9c000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbee5000 con 0x561ebb328c00 2024-03-23T11:28:13.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.904+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 8 ==== osd_op_reply(42 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.904+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:43 13.5 13:a0000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329800 con 0x561ebb329000 2024-03-23T11:28:13.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.905+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 5 ==== osd_op_reply(43 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.906+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:44 13.25 13:a4000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329400 con 0x561ebb329000 2024-03-23T11:28:13.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.907+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 6 ==== osd_op_reply(44 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.907+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:45 13.15 13:a8000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344000 con 0x561ebb329000 2024-03-23T11:28:13.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.908+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 7 ==== osd_op_reply(45 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.908+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:46 13.35 13:ac000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337c00 con 0x561ebb328400 2024-03-23T11:28:13.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.909+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 7 ==== osd_op_reply(46 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.909+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:47 13.d 13:b0000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344400 con 0x561ebb329000 2024-03-23T11:28:13.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.910+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 8 ==== osd_op_reply(47 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.910+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:48 13.2d 13:b4000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336c00 con 0x561ebbee5800 2024-03-23T11:28:13.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.911+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 3 ==== osd_op_reply(48 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.911+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:49 13.1d 13:b8000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344800 con 0x561ebbee5400 2024-03-23T11:28:13.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.912+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 5 ==== osd_op_reply(49 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.912+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:50 13.3d 13:bc000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337400 con 0x561ebbee5400 2024-03-23T11:28:13.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.913+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 6 ==== osd_op_reply(50 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.913+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:51 13.3 13:c0000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344c00 con 0x561ebbee5800 2024-03-23T11:28:13.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.914+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 4 ==== osd_op_reply(51 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.914+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:52 13.23 13:c4000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336000 con 0x561ebb328400 2024-03-23T11:28:13.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.915+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 8 ==== osd_op_reply(52 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.915+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:53 13.13 13:c8000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345000 con 0x561ebbee5400 2024-03-23T11:28:13.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.916+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 7 ==== osd_op_reply(53 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.916+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:54 13.33 13:cc000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec0400 con 0x561ebbee5400 2024-03-23T11:28:13.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.918+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 8 ==== osd_op_reply(54 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.918+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:55 13.b 13:d0000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345400 con 0x561ebbec0800 2024-03-23T11:28:13.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.919+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 6 ==== osd_op_reply(55 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.919+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:56 13.2b 13:d4000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec1c00 con 0x561ebb328c00 2024-03-23T11:28:13.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.920+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 9 ==== osd_op_reply(56 [pgnls-filter start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.920+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:57 13.1b 13:d8000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345800 con 0x561ebb329000 2024-03-23T11:28:13.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.921+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 9 ==== osd_op_reply(57 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.921+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:58 13.3b 13:dc000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337800 con 0x561ebb329000 2024-03-23T11:28:13.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.922+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 10 ==== osd_op_reply(58 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.923+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:59 13.7 13:e0000000::::head [pgnls-filter start_epoch 93 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345c00 con 0x561ebbee5400 2024-03-23T11:28:13.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.924+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 9 ==== osd_op_reply(59 [pgnls-filter start_epoch 93 out=81b] v93'7 uv7 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.924+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:60 13.0 13:00000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbee5c00 con 0x561ebbee5400 2024-03-23T11:28:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.925+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 10 ==== osd_op_reply(60 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.925+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:61 13.20 13:04000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336800 con 0x561ebbec0000 2024-03-23T11:28:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.926+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 10 ==== osd_op_reply(61 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.926+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:62 13.10 13:08000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328800 con 0x561ebb328400 2024-03-23T11:28:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.927+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 9 ==== osd_op_reply(62 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.928+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:63 13.30 13:0c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337400 con 0x561ebb329000 2024-03-23T11:28:13.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.929+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 11 ==== osd_op_reply(63 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.929+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:64 13.8 13:10000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329c00 con 0x561ebb328c00 2024-03-23T11:28:13.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.930+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 10 ==== osd_op_reply(64 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.930+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:65 13.28 13:14000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336400 con 0x561ebb328c00 2024-03-23T11:28:13.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.931+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 11 ==== osd_op_reply(65 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.931+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:66 13.18 13:18000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328000 con 0x561ebbee5400 2024-03-23T11:28:13.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.932+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 11 ==== osd_op_reply(66 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.932+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:67 13.38 13:1c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337c00 con 0x561ebbec0800 2024-03-23T11:28:13.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.933+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 7 ==== osd_op_reply(67 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.933+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:68 13.4 13:20000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb07d400 con 0x561ebbec0000 2024-03-23T11:28:13.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.934+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 11 ==== osd_op_reply(68 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.934+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:69 13.24 13:24000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345c00 con 0x561ebbec0000 2024-03-23T11:28:13.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.935+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 12 ==== osd_op_reply(69 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.935+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:70 13.14 13:28000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344800 con 0x561ebbee5800 2024-03-23T11:28:13.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.936+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 5 ==== osd_op_reply(70 [pgnls-filter start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.936+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:71 13.34 13:2c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344000 con 0x561ebbec0000 2024-03-23T11:28:13.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.938+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 13 ==== osd_op_reply(71 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.938+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:72 13.c 13:30000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345000 con 0x561ebb328400 2024-03-23T11:28:13.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.939+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 10 ==== osd_op_reply(72 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.939+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:73 13.2c 13:34000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344400 con 0x561ebb328c00 2024-03-23T11:28:13.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.940+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 12 ==== osd_op_reply(73 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.940+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:74 13.1c 13:38000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbee5000 con 0x561ebbec0000 2024-03-23T11:28:13.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.941+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 14 ==== osd_op_reply(74 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.941+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:75 13.3c 13:3c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329c00 con 0x561ebb328c00 2024-03-23T11:28:13.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.942+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 13 ==== osd_op_reply(75 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.942+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:76 13.2 13:40000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336c00 con 0x561ebbec0800 2024-03-23T11:28:13.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.943+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 8 ==== osd_op_reply(76 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.943+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:77 13.22 13:44000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328000 con 0x561ebb328400 2024-03-23T11:28:13.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.944+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 11 ==== osd_op_reply(77 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.944+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:78 13.12 13:48000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336000 con 0x561ebbee5800 2024-03-23T11:28:13.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.945+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 6 ==== osd_op_reply(78 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.945+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:79 13.32 13:4c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329800 con 0x561ebb328c00 2024-03-23T11:28:13.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.946+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 14 ==== osd_op_reply(79 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.946+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:80 13.a 13:50000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337c00 con 0x561ebb328c00 2024-03-23T11:28:13.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.947+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 15 ==== osd_op_reply(80 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.947+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:81 13.2a 13:54000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329400 con 0x561ebb328400 2024-03-23T11:28:13.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.948+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 12 ==== osd_op_reply(81 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.948+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:82 13.1a 13:58000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336400 con 0x561ebb328400 2024-03-23T11:28:13.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.949+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 13 ==== osd_op_reply(82 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.949+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:83 13.3a 13:5c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbee5c00 con 0x561ebbec0800 2024-03-23T11:28:13.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.950+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 9 ==== osd_op_reply(83 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.950+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:84 13.6 13:60000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344c00 con 0x561ebb329000 2024-03-23T11:28:13.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.952+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 12 ==== osd_op_reply(84 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.952+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:85 13.26 13:64000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec0400 con 0x561ebbee5400 2024-03-23T11:28:13.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.953+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 12 ==== osd_op_reply(85 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.953+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:86 13.16 13:68000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345400 con 0x561ebb337000 2024-03-23T11:28:13.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.954+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.3 v2:172.21.15.62:6808/37000 5 ==== osd_op_reply(86 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb337000 2024-03-23T11:28:13.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.954+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:87 13.36 13:6c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec1c00 con 0x561ebbec0000 2024-03-23T11:28:13.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.955+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 15 ==== osd_op_reply(87 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.955+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:88 13.e 13:70000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344800 con 0x561ebbee5400 2024-03-23T11:28:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.956+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 13 ==== osd_op_reply(88 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.956+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:89 13.2e 13:74000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337800 con 0x561ebb328c00 2024-03-23T11:28:13.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.957+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 16 ==== osd_op_reply(89 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.957+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:90 13.1e 13:78000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345800 con 0x561ebb329000 2024-03-23T11:28:13.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.958+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 13 ==== osd_op_reply(90 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.959+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:91 13.3e 13:7c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336800 con 0x561ebb328400 2024-03-23T11:28:13.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.959+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 14 ==== osd_op_reply(91 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.960+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:92 13.1 13:80000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328800 con 0x561ebbec0000 2024-03-23T11:28:13.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.961+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 16 ==== osd_op_reply(92 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.961+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:93 13.21 13:84000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337400 con 0x561ebb329000 2024-03-23T11:28:13.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.962+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 14 ==== osd_op_reply(93 [pgnls-filter start_epoch 93 out=49b] v89'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.962+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:94 13.11 13:88000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328800 con 0x561ebb337000 2024-03-23T11:28:13.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.963+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.3 v2:172.21.15.62:6808/37000 6 ==== osd_op_reply(94 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb337000 2024-03-23T11:28:13.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.963+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:95 13.31 13:8c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336c00 con 0x561ebb337000 2024-03-23T11:28:13.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.964+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.3 v2:172.21.15.62:6808/37000 7 ==== osd_op_reply(95 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb337000 2024-03-23T11:28:13.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.964+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:96 13.9 13:90000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb328000 con 0x561ebbec0800 2024-03-23T11:28:13.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.965+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 10 ==== osd_op_reply(96 [pgnls-filter start_epoch 93 out=49b] v89'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.965+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:97 13.29 13:94000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336000 con 0x561ebbec0000 2024-03-23T11:28:13.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.967+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 17 ==== osd_op_reply(97 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.967+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:98 13.19 13:98000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345c00 con 0x561ebb337000 2024-03-23T11:28:13.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.968+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.3 v2:172.21.15.62:6808/37000 8 ==== osd_op_reply(98 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb337000 2024-03-23T11:28:13.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.968+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:99 13.39 13:9c000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336400 con 0x561ebb328c00 2024-03-23T11:28:13.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.969+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 17 ==== osd_op_reply(99 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.969+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:100 13.5 13:a0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344000 con 0x561ebb329000 2024-03-23T11:28:13.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.970+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 15 ==== osd_op_reply(100 [pgnls-filter start_epoch 93 out=49b] v89'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.970+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:101 13.25 13:a4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336800 con 0x561ebb329000 2024-03-23T11:28:13.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.971+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 16 ==== osd_op_reply(101 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.971+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:102 13.15 13:a8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344c00 con 0x561ebb329000 2024-03-23T11:28:13.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.972+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 17 ==== osd_op_reply(102 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.972+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:103 13.35 13:ac000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbee5000 con 0x561ebb328400 2024-03-23T11:28:13.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.973+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 15 ==== osd_op_reply(103 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.973+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:104 13.d 13:b0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345400 con 0x561ebb329000 2024-03-23T11:28:13.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.974+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 18 ==== osd_op_reply(104 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.974+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:105 13.2d 13:b4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbee5c00 con 0x561ebbee5800 2024-03-23T11:28:13.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.975+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 7 ==== osd_op_reply(105 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.975+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:106 13.1d 13:b8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345800 con 0x561ebbee5400 2024-03-23T11:28:13.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.976+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 14 ==== osd_op_reply(106 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.976+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:107 13.3d 13:bc000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329400 con 0x561ebbee5400 2024-03-23T11:28:13.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.977+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 15 ==== osd_op_reply(107 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.977+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:108 13.3 13:c0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345000 con 0x561ebbee5800 2024-03-23T11:28:13.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.978+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 8 ==== osd_op_reply(108 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.978+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:109 13.23 13:c4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337c00 con 0x561ebb328400 2024-03-23T11:28:13.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.979+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 16 ==== osd_op_reply(109 [pgnls-filter start_epoch 93 out=49b] v89'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.980+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:110 13.13 13:c8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec1c00 con 0x561ebbee5400 2024-03-23T11:28:13.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.981+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 16 ==== osd_op_reply(110 [pgnls-filter start_epoch 93 out=49b] v89'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.981+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:111 13.33 13:cc000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337800 con 0x561ebbee5400 2024-03-23T11:28:13.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.982+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 17 ==== osd_op_reply(111 [pgnls-filter start_epoch 93 out=49b] v89'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.982+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:112 13.b 13:d0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb07d400 con 0x561ebbec0800 2024-03-23T11:28:13.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.983+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 11 ==== osd_op_reply(112 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.983+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:113 13.2b 13:d4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336400 con 0x561ebb328c00 2024-03-23T11:28:13.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.984+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.2 v2:172.21.15.62:6800/36996 18 ==== osd_op_reply(113 [pgnls-filter start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebb328c00 2024-03-23T11:28:13.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.984+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:114 13.1b 13:d8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb350000 con 0x561ebb329000 2024-03-23T11:28:13.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.985+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 19 ==== osd_op_reply(114 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.985+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:115 13.3b 13:dc000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344400 con 0x561ebb329000 2024-03-23T11:28:13.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.986+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 20 ==== osd_op_reply(115 [pgnls-filter start_epoch 93 out=49b] v89'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.986+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:116 13.7 13:e0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb350400 con 0x561ebbee5400 2024-03-23T11:28:13.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.988+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 18 ==== osd_op_reply(116 [pgnls-filter start_epoch 93 out=81b] v93'7 uv7 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.988+0000 7f5cebe3d740 20 datascan.forall_objects: OSD matched oid 10000000001.00000000 2024-03-23T11:28:13.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.988+0000 7f5cebe3d740 10 datascan.operator(): handling object 10000000001.0 2024-03-23T11:28:13.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.988+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:117 13.7 13:e2e07930:::10000000001.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345c00 con 0x561ebbee5400 2024-03-23T11:28:13.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.989+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 <== osd.7 v2:172.21.15.120:6816/37655 19 ==== osd_op_reply(117 10000000001.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (89) out=89b,getxattr (30) out=30b,getxattr] v0'0 uv7 ondisk = 0) v8 ==== 416+0+157 (crc 0 0 0) 0x561ebbd8cfc0 con 0x561ebbee5400 2024-03-23T11:28:13.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.989+0000 7f5cebe3d740 10 datascan.inject_with_backtrace: inode: 0x0x10000000001 2024-03-23T11:28:13.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.989+0000 7f5cebe3d740 10 datascan.inject_with_backtrace: backptr: 0x0x10000000000/file 2024-03-23T11:28:13.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.989+0000 7f5cebe3d740 20 datascan.get_frag_of: dirino=0x10000000000 target_dname=file 2024-03-23T11:28:13.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.989+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:118 12.10 12:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344000 con 0x561ebbee5800 2024-03-23T11:28:13.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.990+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 9 ==== osd_op_reply(118 10000000000.00000000 [getxattr (59) out=59b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+59 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.990+0000 7f5cebe3d740 20 datascan.get_frag_of: got backtrace parent 0x1/dir 2024-03-23T11:28:13.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.990+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344c00 con 0x561ebbec0000 2024-03-23T11:28:13.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.992+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 18 ==== osd_op_reply(119 1.00000000 [omap-get-vals-by-keys out=520b] v0'0 uv9 ondisk = 0) v8 ==== 154+0+520 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.992+0000 7f5cebe3d740 20 datascan.read_dentry: oid=1.00000000 dname=dir frag=*, r=0 2024-03-23T11:28:13.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.992+0000 7f5cebe3d740 20 datascan.get_frag_of: fast path, fragtree is fragtree_t() 2024-03-23T11:28:13.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.992+0000 7f5cebe3d740 20 datascan.get_frag_of: frag is * 2024-03-23T11:28:13.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.992+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:120 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb345400 con 0x561ebbee5800 2024-03-23T11:28:13.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.993+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 10 ==== osd_op_reply(120 10000000000.00000000 [omap-get-header out=274b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.993+0000 7f5cebe3d740 10 datascan.find_or_create_dirfrag: read_version = 17 2024-03-23T11:28:13.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.993+0000 7f5cebe3d740 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000000 * 2024-03-23T11:28:13.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.993+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:121 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb337400 con 0x561ebbee5800 2024-03-23T11:28:13.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.994+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 11 ==== osd_op_reply(121 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.994+0000 7f5cebe3d740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-03-23T11:28:13.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.994+0000 7f5cebe3d740 20 datascan.read_dentry: file_head not found in result 2024-03-23T11:28:13.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.994+0000 7f5cebe3d740 10 datascan.inject_with_backtrace: Linking inode 0x0x10000000001 at 0x0x10000000000/file with size=0 bytes 2024-03-23T11:28:13.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.994+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:122 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336c00 con 0x561ebbee5800 2024-03-23T11:28:13.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.997+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.0 v2:172.21.15.62:6811/36998 12 ==== osd_op_reply(122 10000000000.00000000 [omap-set-vals] v93'18 uv18 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbee5800 2024-03-23T11:28:13.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.997+0000 7f5cebe3d740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-03-23T11:28:13.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.997+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:123 13.27 13:e4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb336000 con 0x561ebb328400 2024-03-23T11:28:13.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.998+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.1 v2:172.21.15.62:6824/37001 17 ==== osd_op_reply(123 [pgnls-filter start_epoch 93 out=49b] v89'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebb328400 2024-03-23T11:28:13.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.998+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:124 13.17 13:e8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb350800 con 0x561ebbec0800 2024-03-23T11:28:13.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:13.999+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 12 ==== osd_op_reply(124 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.000+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:125 13.37 13:ec000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb344800 con 0x561ebbec0800 2024-03-23T11:28:13.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.001+0000 7f5ce8e00640 1 -- 172.21.15.62:0/3971276824 <== osd.5 v2:172.21.15.120:6824/37642 13 ==== osd_op_reply(125 [pgnls-filter start_epoch 93 out=49b] v89'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cb40 con 0x561ebbec0800 2024-03-23T11:28:13.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.001+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:126 13.f 13:f0000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb350c00 con 0x561ebbec0000 2024-03-23T11:28:13.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.002+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 19 ==== osd_op_reply(126 [pgnls-filter start_epoch 93 out=49b] v89'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.002+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:127 13.2f 13:f4000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebbec0400 con 0x561ebb329000 2024-03-23T11:28:13.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.003+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.4 v2:172.21.15.120:6803/37640 21 ==== osd_op_reply(127 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebb329000 2024-03-23T11:28:13.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.003+0000 7f5ce3400640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:128 13.1f 13:f8000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb351000 con 0x561ebbec0000 2024-03-23T11:28:13.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.004+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 20 ==== osd_op_reply(128 [pgnls-filter start_epoch 93 out=49b] v89'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.004+0000 7f5ce3e00640 1 -- 172.21.15.62:0/3971276824 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:129 13.3f 13:fc000000::::head [pgnls-filter start_epoch 93 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x561ebb329c00 con 0x561ebbec0000 2024-03-23T11:28:13.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cea200640 1 -- 172.21.15.62:0/3971276824 <== osd.6 v2:172.21.15.120:6800/37641 21 ==== osd_op_reply(129 [pgnls-filter start_epoch 93 out=49b] v89'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x561ebbd8cd80 con 0x561ebbec0000 2024-03-23T11:28:13.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x561ebbee5800 msgr2=0x561ebb32f080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x561ebbee5800 0x561ebb32f080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x561ebb328400 msgr2=0x561ebb32e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x561ebb328400 0x561ebb32e000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x561ebb328c00 msgr2=0x561ebb32eb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x561ebb328c00 0x561ebb32eb00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x561ebb337000 msgr2=0x561ebb32f600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x561ebb337000 0x561ebb32f600 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 reap_dead start 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x561ebb329000 msgr2=0x561ebb32e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x561ebb329000 0x561ebb32e580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x561ebbec0800 msgr2=0x561ebb022100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x561ebbec0800 0x561ebb022100 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x561ebbec0000 msgr2=0x561ebb021b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x561ebbec0000 0x561ebb021b80 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x561ebbee5400 msgr2=0x561ebb021600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x561ebbee5400 0x561ebb021600 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5ce9800640 1 -- 172.21.15.62:0/3971276824 reap_dead start 2024-03-23T11:28:14.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561ebbee4c00 msgr2=0x561ebb023700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561ebbee4c00 0x561ebb023700 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x561ebbe64a50 tx=0x561ebbe69ec0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.006+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebbec1000 msgr2=0x561ebb022c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.007+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebbec1000 0x561ebb022c00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x561ebbe655f0 tx=0x561ebbe68d20 comp rx=0 tx=0).stop 2024-03-23T11:28:14.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.007+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 shutdown_connections 2024-03-23T11:28:14.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.007+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561ebbee4c00 0x561ebb023700 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.007+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/3971276824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebbec1000 0x561ebb022c00 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.007+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 >> 172.21.15.62:0/3971276824 conn(0x561ebb07dc00 msgr2=0x561ebb09f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:14.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.007+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 shutdown_connections 2024-03-23T11:28:14.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.008+0000 7f5cebe3d740 1 -- 172.21.15.62:0/3971276824 wait complete. 2024-03-23T11:28:14.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.009+0000 7f5cebe3d740 1 -- 172.21.15.62:0/1212995627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebaf99c00 msgr2=0x561ebb021080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.009+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/1212995627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebaf99c00 0x561ebb021080 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x561ebbdb5290 tx=0x561ebbdc00c0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.009+0000 7f5cebe3d740 1 -- 172.21.15.62:0/1212995627 shutdown_connections 2024-03-23T11:28:14.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.009+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/1212995627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ebaf99c00 0x561ebb021080 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.009+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/1212995627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ebaf99800 0x561ebb020b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.009+0000 7f5cebe3d740 1 --2- 172.21.15.62:0/1212995627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ebaf98c00 0x561ebb020580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.009+0000 7f5cebe3d740 1 -- 172.21.15.62:0/1212995627 >> 172.21.15.62:0/1212995627 conn(0x561ebaf99000 msgr2=0x561ebb09e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:14.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.009+0000 7f5cebe3d740 1 -- 172.21.15.62:0/1212995627 shutdown_connections 2024-03-23T11:28:14.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.009+0000 7f5cebe3d740 1 -- 172.21.15.62:0/1212995627 wait complete. 2024-03-23T11:28:14.014 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', 'cephfs_data'] in time 0:00:00.328183, result: 2024-03-23T11:28:14.015 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-03-23T11:28:14.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.132+0000 7f4014a13740 1 -- start start 2024-03-23T11:28:14.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.132+0000 7f4014a13740 1 Processor -- start 2024-03-23T11:28:14.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4014a13740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ebf38c00 0x5562ebfc0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4014a13740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ebf39800 0x5562ebfc0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4014a13740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ebf39c00 0x5562ebfc1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4014a13740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562ec00aa00 con 0x5562ebf38c00 2024-03-23T11:28:14.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4014a13740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562ec00a780 con 0x5562ebf39c00 2024-03-23T11:28:14.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4014a13740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562ec00a640 con 0x5562ebf39800 2024-03-23T11:28:14.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4012000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ebf38c00 0x5562ebfc0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4012000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ebf38c00 0x5562ebfc0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50932/0 (socket says 172.21.15.62:50932) 2024-03-23T11:28:14.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4012000640 1 -- 172.21.15.62:0/701029444 learned_addr learned my addr 172.21.15.62:0/701029444 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:14.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4011600640 1 --2- 172.21.15.62:0/701029444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ebf39800 0x5562ebfc0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4012a00640 1 --2- 172.21.15.62:0/701029444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ebf39c00 0x5562ebfc1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4012000640 1 -- 172.21.15.62:0/701029444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ebf39800 msgr2=0x5562ebfc0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4012000640 1 --2- 172.21.15.62:0/701029444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ebf39800 0x5562ebfc0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4012000640 1 -- 172.21.15.62:0/701029444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ebf39c00 msgr2=0x5562ebfc1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.134+0000 7f4012000640 1 --2- 172.21.15.62:0/701029444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ebf39c00 0x5562ebfc1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.135+0000 7f4012000640 1 -- 172.21.15.62:0/701029444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562ebf5e9c0 con 0x5562ebf38c00 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.135+0000 7f4012a00640 1 --2- 172.21.15.62:0/701029444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ebf39c00 0x5562ebfc1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.135+0000 7f4012000640 1 --2- 172.21.15.62:0/701029444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ebf38c00 0x5562ebfc0580 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x5562ecd4f3b0 tx=0x5562ecd5ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4bea4d6ad792646 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.135+0000 7f400ee00640 1 -- 172.21.15.62:0/701029444 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5562ebf7f600 con 0x5562ebf38c00 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.135+0000 7f400ee00640 1 -- 172.21.15.62:0/701029444 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5562ebf94480 con 0x5562ebf38c00 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.135+0000 7f400ee00640 1 -- 172.21.15.62:0/701029444 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5562ebf7f1e0 con 0x5562ebf38c00 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.135+0000 7f4014a13740 1 -- 172.21.15.62:0/701029444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562ebf5eb60 con 0x5562ebf38c00 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.135+0000 7f4014a13740 1 -- 172.21.15.62:0/701029444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5562ebf5e820 con 0x5562ebf38c00 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.135+0000 7f4014a13740 4 waiting for MDS map... 2024-03-23T11:28:14.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.135+0000 7f400ee00640 1 -- 172.21.15.62:0/701029444 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x5562ebf8ec40 con 0x5562ebf38c00 2024-03-23T11:28:14.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.136+0000 7f400ee00640 1 -- 172.21.15.62:0/701029444 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 65) v1 ==== 2573+0+0 (secure 0 0 0) 0x5562ebf88f00 con 0x5562ebf38c00 2024-03-23T11:28:14.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.136+0000 7f4014a13740 4 Got MDS map 65 2024-03-23T11:28:14.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.136+0000 7f4014a13740 4 datascan.main: Using metadata pool output 2024-03-23T11:28:14.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.136+0000 7f4014a13740 4 datascan.main: connecting to RADOS... 2024-03-23T11:28:14.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.137+0000 7f4014a13740 1 Processor -- start 2024-03-23T11:28:14.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.137+0000 7f4014a13740 1 -- start start 2024-03-23T11:28:14.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4014a13740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05000 0x5562ebfc1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4014a13740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ece05400 0x5562ebfc1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4014a13740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ece05800 0x5562ebfc2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4014a13740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562ebf81e00 con 0x5562ece05000 2024-03-23T11:28:14.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4014a13740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562ec00a780 con 0x5562ece05800 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4014a13740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562ec00a500 con 0x5562ece05400 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4012a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05000 0x5562ebfc1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4012a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05000 0x5562ebfc1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50942/0 (socket says 172.21.15.62:50942) 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4012a00640 1 -- 172.21.15.62:0/790688606 learned_addr learned my addr 172.21.15.62:0/790688606 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4012000640 1 --2- 172.21.15.62:0/790688606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ece05400 0x5562ebfc1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4011600640 1 --2- 172.21.15.62:0/790688606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ece05800 0x5562ebfc2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4012a00640 1 -- 172.21.15.62:0/790688606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ece05400 msgr2=0x5562ebfc1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4012a00640 1 --2- 172.21.15.62:0/790688606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ece05400 0x5562ebfc1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4012a00640 1 -- 172.21.15.62:0/790688606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ece05800 msgr2=0x5562ebfc2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4012a00640 1 --2- 172.21.15.62:0/790688606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ece05800 0x5562ebfc2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.138+0000 7f4012a00640 1 -- 172.21.15.62:0/790688606 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562ebf5e4e0 con 0x5562ece05000 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.139+0000 7f4012a00640 1 --2- 172.21.15.62:0/790688606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05000 0x5562ebfc1600 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x5562ecd4ea80 tx=0x5562ecd5a0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d1eb95c43ea414 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.139+0000 7f400c600640 1 -- 172.21.15.62:0/790688606 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5562ebf7f080 con 0x5562ece05000 2024-03-23T11:28:14.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.139+0000 7f400c600640 1 -- 172.21.15.62:0/790688606 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5562ec01ca80 con 0x5562ece05000 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.139+0000 7f400c600640 1 -- 172.21.15.62:0/790688606 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5562ebf7f600 con 0x5562ece05000 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.139+0000 7f4014a13740 1 -- 172.21.15.62:0/790688606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05000 msgr2=0x5562ebfc1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.139+0000 7f4014a13740 1 --2- 172.21.15.62:0/790688606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05000 0x5562ebfc1600 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x5562ecd4ea80 tx=0x5562ecd5a0c0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.140+0000 7f4014a13740 1 -- 172.21.15.62:0/790688606 shutdown_connections 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.140+0000 7f4014a13740 1 --2- 172.21.15.62:0/790688606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ece05800 0x5562ebfc2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.140+0000 7f4014a13740 1 --2- 172.21.15.62:0/790688606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ece05400 0x5562ebfc1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.140+0000 7f4014a13740 1 --2- 172.21.15.62:0/790688606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05000 0x5562ebfc1600 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.140+0000 7f4014a13740 1 -- 172.21.15.62:0/790688606 >> 172.21.15.62:0/790688606 conn(0x5562ece04c00 msgr2=0x5562eccb7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.140+0000 7f4014a13740 1 -- 172.21.15.62:0/790688606 shutdown_connections 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.140+0000 7f4014a13740 1 -- 172.21.15.62:0/790688606 wait complete. 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.140+0000 7f4014a13740 1 Processor -- start 2024-03-23T11:28:14.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.141+0000 7f4014a13740 1 -- start start 2024-03-23T11:28:14.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.141+0000 7f4014a13740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ece05000 0x5562ebfc3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.141+0000 7f4014a13740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05400 0x5562ebfc2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.141+0000 7f4014a13740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ece05800 0x5562ebfc2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.141+0000 7f4014a13740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562ec00a3c0 con 0x5562ece05400 2024-03-23T11:28:14.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.141+0000 7f4014a13740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562ec00a280 con 0x5562ece05800 2024-03-23T11:28:14.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.141+0000 7f4014a13740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562ec00a140 con 0x5562ece05000 2024-03-23T11:28:14.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.142+0000 7f4012000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05400 0x5562ebfc2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.142+0000 7f4012000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05400 0x5562ebfc2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50952/0 (socket says 172.21.15.62:50952) 2024-03-23T11:28:14.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.142+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 learned_addr learned my addr 172.21.15.62:0/722667902 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.142+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ece05000 msgr2=0x5562ebfc3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.142+0000 7f4012000640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ece05000 0x5562ebfc3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.142+0000 7f4011600640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ece05800 0x5562ebfc2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.142+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ece05800 msgr2=0x5562ebfc2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.142+0000 7f4012000640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ece05800 0x5562ebfc2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.142+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562ebf5ed00 con 0x5562ece05400 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.142+0000 7f4012000640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05400 0x5562ebfc2c00 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x5562ece1b410 tx=0x5562ecccc300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.143+0000 7f400a800640 1 -- 172.21.15.62:0/722667902 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5562ebf7ef20 con 0x5562ece05400 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.143+0000 7f400a800640 1 -- 172.21.15.62:0/722667902 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5562ec01cc00 con 0x5562ece05400 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.143+0000 7f400a800640 1 -- 172.21.15.62:0/722667902 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5562ebf7f080 con 0x5562ece05400 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.143+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5562ebf5e4e0 con 0x5562ece05400 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.143+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562ebf5e9c0 con 0x5562ece05400 2024-03-23T11:28:14.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.143+0000 7f4014a13740 4 datascan.init: resolving metadata pool 12 2024-03-23T11:28:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.145+0000 7f400a800640 1 -- 172.21.15.62:0/722667902 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x5562ebf6b080 con 0x5562ece05400 2024-03-23T11:28:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.145+0000 7f400a800640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5562ec28a000 0x5562ebfc3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.145+0000 7f400a800640 1 -- 172.21.15.62:0/722667902 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x5562ebf8ea80 con 0x5562ece05400 2024-03-23T11:28:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.145+0000 7f4014a13740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-23T11:28:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.145+0000 7f4014a13740 4 datascan.main: resolving metadata pool 12 2024-03-23T11:28:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.145+0000 7f4014a13740 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x5562ec28a400 0x5562ebfc2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.145+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:1 12.0 12:00000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28a800 con 0x5562ec28a400 2024-03-23T11:28:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.146+0000 7f4012a00640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5562ec28a000 0x5562ebfc3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.146+0000 7f4011600640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x5562ec28a400 0x5562ebfc2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.147+0000 7f4011600640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x5562ec28a400 0x5562ebfc2100 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.147+0000 7f4012a00640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5562ec28a000 0x5562ebfc3700 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x5562ecd4ea80 tx=0x5562ecd5a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.148+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== osd_op_reply(1 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ec28a400 2024-03-23T11:28:14.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.148+0000 7f4011600640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x5562ec28a800 0x5562ecdf4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.148+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:2 12.10 12:08000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28b800 con 0x5562ec28a800 2024-03-23T11:28:14.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.149+0000 7f4012a00640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x5562ec28a800 0x5562ecdf4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.149+0000 7f4012a00640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x5562ec28a800 0x5562ecdf4000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.150+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== osd_op_reply(2 [pgnls start_epoch 93 out=177b] v93'18 uv18 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.150+0000 7f4012a00640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5562ec28b800 0x5562ecdf4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.150+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:3 12.8 12:10000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ecd15000 con 0x5562ec28b800 2024-03-23T11:28:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.150+0000 7f4012000640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5562ec28b800 0x5562ecdf4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.150+0000 7f4012000640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5562ec28b800 0x5562ecdf4580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.151+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== osd_op_reply(3 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec28b800 2024-03-23T11:28:14.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.151+0000 7f4012000640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5562ecd15000 0x5562ecdf4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.151+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:4 12.18 12:18000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28bc00 con 0x5562ecd15000 2024-03-23T11:28:14.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.151+0000 7f4011600640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5562ecd15000 0x5562ecdf4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.152+0000 7f4011600640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5562ecd15000 0x5562ecdf4b00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.152+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(4 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.152+0000 7f4011600640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x5562ec28bc00 0x5562ecdf5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.152+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:5 12.4 12:20000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6000 con 0x5562ec28bc00 2024-03-23T11:28:14.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.153+0000 7f4012a00640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x5562ec28bc00 0x5562ecdf5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.154+0000 7f4012a00640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x5562ec28bc00 0x5562ecdf5080 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.155+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== osd_op_reply(5 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.155+0000 7f4012a00640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x5562ec2b6000 0x5562ecdf5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.155+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:6 12.14 12:28000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ecd15400 con 0x5562ec2b6000 2024-03-23T11:28:14.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.155+0000 7f4012000640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x5562ec2b6000 0x5562ecdf5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.156+0000 7f4012000640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x5562ec2b6000 0x5562ecdf5b80 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.157+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== osd_op_reply(6 [pgnls start_epoch 93 out=149b] v92'27 uv27 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec2b6000 2024-03-23T11:28:14.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.157+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:7 12.c 12:30000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ecd15400 con 0x5562ec28a800 2024-03-23T11:28:14.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.158+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 2 ==== osd_op_reply(7 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.158+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:8 12.1c 12:38000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ecd15400 con 0x5562ec28a800 2024-03-23T11:28:14.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.158+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 3 ==== osd_op_reply(8 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.158+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:9 12.2 12:40000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ecd15400 con 0x5562ec28a400 2024-03-23T11:28:14.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.159+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.4 v2:172.21.15.120:6803/37640 2 ==== osd_op_reply(9 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ec28a400 2024-03-23T11:28:14.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.159+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:10 12.12 12:48000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ecd15400 con 0x5562ec28b800 2024-03-23T11:28:14.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.159+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.3 v2:172.21.15.62:6808/37000 2 ==== osd_op_reply(10 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec28b800 2024-03-23T11:28:14.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.159+0000 7f4012000640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x5562ecd15400 0x5562ecdf6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.159+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:11 12.a 12:50000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ecd15400 2024-03-23T11:28:14.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.159+0000 7f4011600640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x5562ecd15400 0x5562ecdf6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.160+0000 7f4011600640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x5562ecd15400 0x5562ecdf6680 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.160+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.2 v2:172.21.15.62:6800/36996 1 ==== osd_op_reply(11 [pgnls start_epoch 93 out=81b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15400 2024-03-23T11:28:14.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.160+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:12 12.1a 12:58000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ec28a400 2024-03-23T11:28:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.161+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.4 v2:172.21.15.120:6803/37640 3 ==== osd_op_reply(12 [pgnls start_epoch 93 out=49b] v92'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ec28a400 2024-03-23T11:28:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.161+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:13 12.6 12:60000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ecd15000 2024-03-23T11:28:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.162+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 2 ==== osd_op_reply(13 [pgnls start_epoch 93 out=77b] v92'8 uv8 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.162+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:14 12.16 12:68000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ec28bc00 2024-03-23T11:28:14.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.163+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 2 ==== osd_op_reply(14 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.163+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:15 12.e 12:70000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ec28a800 2024-03-23T11:28:14.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.163+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 4 ==== osd_op_reply(15 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.163+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:16 12.1e 12:78000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ec28a400 2024-03-23T11:28:14.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.164+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.4 v2:172.21.15.120:6803/37640 4 ==== osd_op_reply(16 [pgnls start_epoch 93 out=49b] v92'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ec28a400 2024-03-23T11:28:14.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.164+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:17 12.1 12:80000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ec28bc00 2024-03-23T11:28:14.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.164+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 3 ==== osd_op_reply(17 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.164+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:18 12.11 12:88000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ec2b6000 2024-03-23T11:28:14.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.165+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.5 v2:172.21.15.120:6824/37642 2 ==== osd_op_reply(18 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec2b6000 2024-03-23T11:28:14.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.165+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:19 12.9 12:90000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ecd15400 2024-03-23T11:28:14.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.166+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.2 v2:172.21.15.62:6800/36996 2 ==== osd_op_reply(19 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15400 2024-03-23T11:28:14.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.166+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:20 12.19 12:98000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ec28a800 2024-03-23T11:28:14.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.166+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 5 ==== osd_op_reply(20 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.166+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:21 12.5 12:a0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ec28bc00 2024-03-23T11:28:14.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.167+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 4 ==== osd_op_reply(21 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.167+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:22 12.15 12:a8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6800 con 0x5562ec28bc00 2024-03-23T11:28:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.168+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 5 ==== osd_op_reply(22 [pgnls start_epoch 93 out=97b] v92'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.168+0000 7f4012a00640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5562ec2b6800 0x5562ecdf6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.168+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:23 12.d 12:b0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6000 con 0x5562ec2b6800 2024-03-23T11:28:14.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.168+0000 7f4012a00640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5562ec2b6800 0x5562ecdf6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.169+0000 7f4012a00640 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5562ec2b6800 0x5562ecdf6100 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.170+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.7 v2:172.21.15.120:6816/37655 1 ==== osd_op_reply(23 [pgnls start_epoch 93 out=74b] v88'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec2b6800 2024-03-23T11:28:14.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.170+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:24 12.1d 12:b8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6000 con 0x5562ec28a800 2024-03-23T11:28:14.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.171+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 6 ==== osd_op_reply(24 [pgnls start_epoch 93 out=73b] v92'5 uv5 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.171+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:25 12.3 12:c0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6000 con 0x5562ec28a800 2024-03-23T11:28:14.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.171+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 7 ==== osd_op_reply(25 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.171+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:26 12.13 12:c8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6000 con 0x5562ecd15000 2024-03-23T11:28:14.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.172+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 3 ==== osd_op_reply(26 [pgnls start_epoch 93 out=98b] v92'4 uv4 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.172+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:27 12.b 12:d0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6000 con 0x5562ec28b800 2024-03-23T11:28:14.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.172+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.3 v2:172.21.15.62:6808/37000 3 ==== osd_op_reply(27 [pgnls start_epoch 93 out=76b] v92'6 uv6 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec28b800 2024-03-23T11:28:14.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.172+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:28 12.1b 12:d8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6000 con 0x5562ec28a400 2024-03-23T11:28:14.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.174+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.4 v2:172.21.15.120:6803/37640 5 ==== osd_op_reply(28 [pgnls start_epoch 93 out=73b] v92'10 uv10 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ec28a400 2024-03-23T11:28:14.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.174+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:29 12.7 12:e0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6000 con 0x5562ecd15000 2024-03-23T11:28:14.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.174+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 4 ==== osd_op_reply(29 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.174+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:30 12.17 12:e8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6000 con 0x5562ec28bc00 2024-03-23T11:28:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.175+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 6 ==== osd_op_reply(30 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.175+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:31 12.f 12:f0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6000 con 0x5562ec28bc00 2024-03-23T11:28:14.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.176+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 7 ==== osd_op_reply(31 [pgnls start_epoch 93 out=73b] v88'1 uv1 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.176+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:32 12.1f 12:f8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6000 con 0x5562ec28bc00 2024-03-23T11:28:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.177+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 8 ==== osd_op_reply(32 [pgnls start_epoch 93 out=101b] v92'9 uv9 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.177+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 601.00000000 2024-03-23T11:28:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.177+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:33 12.10 12:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28ac00 con 0x5562ec28a800 2024-03-23T11:28:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.178+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 8 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.178+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 602.00000000 2024-03-23T11:28:14.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.178+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:34 12.10 12:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28b000 con 0x5562ec28a800 2024-03-23T11:28:14.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.178+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 9 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.178+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-03-23T11:28:14.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.178+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:35 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28b400 con 0x5562ec28a800 2024-03-23T11:28:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.179+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 10 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) v8 ==== 164+0+506 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.179+0000 7f4014a13740 20 datascan.scan_links: injected ino detected 2024-03-23T11:28:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.179+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 600.00000000 2024-03-23T11:28:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.179+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:36 12.10 12:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28ac00 con 0x5562ec28a800 2024-03-23T11:28:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 11 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 603.00000000 2024-03-23T11:28:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:37 12.10 12:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28b000 con 0x5562ec28a800 2024-03-23T11:28:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 12 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-03-23T11:28:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-23T11:28:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 200.00000000 2024-03-23T11:28:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-23T11:28:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 200.00000001 2024-03-23T11:28:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-23T11:28:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 606.00000000 2024-03-23T11:28:14.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.180+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:38 12.14 12:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6000 con 0x5562ec2b6000 2024-03-23T11:28:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.182+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.5 v2:172.21.15.120:6824/37642 3 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv25 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec2b6000 2024-03-23T11:28:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.182+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 607.00000000 2024-03-23T11:28:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.182+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:39 12.1c 12:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6400 con 0x5562ec28a800 2024-03-23T11:28:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.182+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 13 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.182+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 100000001f6.00000000 2024-03-23T11:28:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.182+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:40 12.a 12:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6800 con 0x5562ecd15400 2024-03-23T11:28:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.183+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.2 v2:172.21.15.62:6800/36996 3 ==== osd_op_reply(40 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+54 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15400 2024-03-23T11:28:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.183+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-03-23T11:28:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.183+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-23T11:28:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.183+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 608.00000000 2024-03-23T11:28:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.183+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:41 12.e 12:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6c00 con 0x5562ec28a800 2024-03-23T11:28:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.184+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 14 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.184+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 604.00000000 2024-03-23T11:28:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.184+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:42 12.15 12:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28bc00 2024-03-23T11:28:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.185+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 9 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.185+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 500.00000000 2024-03-23T11:28:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.185+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-23T11:28:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.185+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-03-23T11:28:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.185+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-23T11:28:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.185+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 605.00000000 2024-03-23T11:28:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.185+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:43 12.1d 12:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7400 con 0x5562ec28a800 2024-03-23T11:28:14.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.186+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 15 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.186+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-03-23T11:28:14.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.186+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-23T11:28:14.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.186+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 100.00000000 2024-03-23T11:28:14.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.186+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:44 12.13 12:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7800 con 0x5562ecd15000 2024-03-23T11:28:14.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.187+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 5 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.187+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-03-23T11:28:14.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.187+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-23T11:28:14.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.187+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 200.00000002 2024-03-23T11:28:14.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.187+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-03-23T11:28:14.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.187+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 609.00000000 2024-03-23T11:28:14.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.187+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:45 12.7 12:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6400 con 0x5562ecd15000 2024-03-23T11:28:14.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.188+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 6 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.188+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 400.00000000 2024-03-23T11:28:14.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.188+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-23T11:28:14.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.188+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-03-23T11:28:14.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.188+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-23T11:28:14.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.188+0000 7f4014a13740 10 datascan.scan_links: step 1: handling object 1.00000000 2024-03-23T11:28:14.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.188+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:46 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6c00 con 0x5562ec28bc00 2024-03-23T11:28:14.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.189+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 10 ==== osd_op_reply(46 1.00000000 [omap-get-vals out=1041b] v0'0 uv9 ondisk = 0) v8 ==== 154+0+1041 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.189+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:47 12.0 12:00000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a400 2024-03-23T11:28:14.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.190+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.4 v2:172.21.15.120:6803/37640 6 ==== osd_op_reply(47 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ec28a400 2024-03-23T11:28:14.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.190+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:48 12.10 12:08000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a800 2024-03-23T11:28:14.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.191+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 16 ==== osd_op_reply(48 [pgnls start_epoch 93 out=177b] v93'18 uv18 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.191+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:49 12.8 12:10000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28b800 2024-03-23T11:28:14.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.192+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.3 v2:172.21.15.62:6808/37000 4 ==== osd_op_reply(49 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec28b800 2024-03-23T11:28:14.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.192+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:50 12.18 12:18000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ecd15000 2024-03-23T11:28:14.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.193+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 7 ==== osd_op_reply(50 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.193+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:51 12.4 12:20000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28bc00 2024-03-23T11:28:14.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.194+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 11 ==== osd_op_reply(51 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.194+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:52 12.14 12:28000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec2b6000 2024-03-23T11:28:14.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.195+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.5 v2:172.21.15.120:6824/37642 4 ==== osd_op_reply(52 [pgnls start_epoch 93 out=149b] v92'27 uv27 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec2b6000 2024-03-23T11:28:14.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.195+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:53 12.c 12:30000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a800 2024-03-23T11:28:14.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.196+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 17 ==== osd_op_reply(53 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.196+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:54 12.1c 12:38000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a800 2024-03-23T11:28:14.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.197+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 18 ==== osd_op_reply(54 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.197+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:55 12.2 12:40000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a400 2024-03-23T11:28:14.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.198+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.4 v2:172.21.15.120:6803/37640 7 ==== osd_op_reply(55 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ec28a400 2024-03-23T11:28:14.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.198+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:56 12.12 12:48000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28b800 2024-03-23T11:28:14.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.199+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.3 v2:172.21.15.62:6808/37000 5 ==== osd_op_reply(56 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec28b800 2024-03-23T11:28:14.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.199+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:57 12.a 12:50000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ecd15400 2024-03-23T11:28:14.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.200+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.2 v2:172.21.15.62:6800/36996 4 ==== osd_op_reply(57 [pgnls start_epoch 93 out=81b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15400 2024-03-23T11:28:14.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.200+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:58 12.1a 12:58000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a400 2024-03-23T11:28:14.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.201+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.4 v2:172.21.15.120:6803/37640 8 ==== osd_op_reply(58 [pgnls start_epoch 93 out=49b] v92'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ec28a400 2024-03-23T11:28:14.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.201+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:59 12.6 12:60000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ecd15000 2024-03-23T11:28:14.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.202+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 8 ==== osd_op_reply(59 [pgnls start_epoch 93 out=77b] v92'8 uv8 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.202+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:60 12.16 12:68000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28bc00 2024-03-23T11:28:14.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.203+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 12 ==== osd_op_reply(60 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.203+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:61 12.e 12:70000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a800 2024-03-23T11:28:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.204+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 19 ==== osd_op_reply(61 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.204+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:62 12.1e 12:78000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a400 2024-03-23T11:28:14.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.205+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.4 v2:172.21.15.120:6803/37640 9 ==== osd_op_reply(62 [pgnls start_epoch 93 out=49b] v92'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ec28a400 2024-03-23T11:28:14.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.205+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:63 12.1 12:80000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28bc00 2024-03-23T11:28:14.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.206+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 13 ==== osd_op_reply(63 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.206+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:64 12.11 12:88000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec2b6000 2024-03-23T11:28:14.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.207+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.5 v2:172.21.15.120:6824/37642 5 ==== osd_op_reply(64 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec2b6000 2024-03-23T11:28:14.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.207+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:65 12.9 12:90000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ecd15400 2024-03-23T11:28:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.208+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.2 v2:172.21.15.62:6800/36996 5 ==== osd_op_reply(65 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15400 2024-03-23T11:28:14.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.208+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:66 12.19 12:98000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a800 2024-03-23T11:28:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.209+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 20 ==== osd_op_reply(66 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.209+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:67 12.5 12:a0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28bc00 2024-03-23T11:28:14.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.210+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 14 ==== osd_op_reply(67 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.210+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:68 12.15 12:a8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28bc00 2024-03-23T11:28:14.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.211+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 15 ==== osd_op_reply(68 [pgnls start_epoch 93 out=97b] v92'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.211+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:69 12.d 12:b0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec2b6800 2024-03-23T11:28:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.212+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.7 v2:172.21.15.120:6816/37655 2 ==== osd_op_reply(69 [pgnls start_epoch 93 out=74b] v88'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec2b6800 2024-03-23T11:28:14.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.212+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:70 12.1d 12:b8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a800 2024-03-23T11:28:14.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.213+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 21 ==== osd_op_reply(70 [pgnls start_epoch 93 out=73b] v92'5 uv5 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.213+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:71 12.3 12:c0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a800 2024-03-23T11:28:14.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.214+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 22 ==== osd_op_reply(71 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.214+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:72 12.13 12:c8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ecd15000 2024-03-23T11:28:14.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.215+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 9 ==== osd_op_reply(72 [pgnls start_epoch 93 out=98b] v92'4 uv4 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.215+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:73 12.b 12:d0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28b800 2024-03-23T11:28:14.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.216+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.3 v2:172.21.15.62:6808/37000 6 ==== osd_op_reply(73 [pgnls start_epoch 93 out=76b] v92'6 uv6 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec28b800 2024-03-23T11:28:14.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.216+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:74 12.1b 12:d8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a400 2024-03-23T11:28:14.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.218+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.4 v2:172.21.15.120:6803/37640 10 ==== osd_op_reply(74 [pgnls start_epoch 93 out=73b] v92'10 uv10 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ec28a400 2024-03-23T11:28:14.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.218+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:75 12.7 12:e0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ecd15000 2024-03-23T11:28:14.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.219+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 10 ==== osd_op_reply(75 [pgnls start_epoch 93 out=73b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.219+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:76 12.17 12:e8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28bc00 2024-03-23T11:28:14.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.220+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 16 ==== osd_op_reply(76 [pgnls start_epoch 93 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.220+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:77 12.f 12:f0000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28bc00 2024-03-23T11:28:14.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.221+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 17 ==== osd_op_reply(77 [pgnls start_epoch 93 out=73b] v88'1 uv1 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.221+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:78 12.1f 12:f8000000::::head [pgnls start_epoch 93 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28bc00 2024-03-23T11:28:14.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.222+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 18 ==== osd_op_reply(78 [pgnls start_epoch 93 out=101b] v92'9 uv9 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.222+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 601.00000000 2024-03-23T11:28:14.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.222+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:79 12.10 12:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7400 con 0x5562ec28a800 2024-03-23T11:28:14.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.223+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 23 ==== osd_op_reply(79 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.223+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 602.00000000 2024-03-23T11:28:14.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.223+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:80 12.10 12:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7c00 con 0x5562ec28a800 2024-03-23T11:28:14.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.224+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 24 ==== osd_op_reply(80 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.224+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-03-23T11:28:14.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.224+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:81 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28b400 con 0x5562ec28a800 2024-03-23T11:28:14.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.225+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 25 ==== osd_op_reply(81 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) v8 ==== 164+0+506 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.225+0000 7f4014a13740 20 datascan.scan_links: injected ino detected 2024-03-23T11:28:14.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.225+0000 7f4014a13740 -1 datascan.scan_links: Bad nlink on 0x10000000001 expected 2 has 1 2024-03-23T11:28:14.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.225+0000 7f4014a13740 20 datascan.scan_links: adding 0x10000000001 for future processing to fix dnfirst 2024-03-23T11:28:14.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.225+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 600.00000000 2024-03-23T11:28:14.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.225+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:82 12.10 12:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28ac00 con 0x5562ec28a800 2024-03-23T11:28:14.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.226+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 26 ==== osd_op_reply(82 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.226+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 603.00000000 2024-03-23T11:28:14.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.226+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:83 12.10 12:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28b000 con 0x5562ec28a800 2024-03-23T11:28:14.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.227+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 27 ==== osd_op_reply(83 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.227+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-03-23T11:28:14.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.227+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-23T11:28:14.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.227+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 200.00000000 2024-03-23T11:28:14.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.227+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-23T11:28:14.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.227+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 200.00000001 2024-03-23T11:28:14.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.227+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-23T11:28:14.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.227+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 606.00000000 2024-03-23T11:28:14.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.227+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:84 12.14 12:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6800 con 0x5562ec2b6000 2024-03-23T11:28:14.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.228+0000 7f4012000640 1 -- 172.21.15.62:0/722667902 <== osd.5 v2:172.21.15.120:6824/37642 6 ==== osd_op_reply(84 606.00000000 [omap-get-vals out=5b] v0'0 uv25 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeefc0 con 0x5562ec2b6000 2024-03-23T11:28:14.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.228+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 607.00000000 2024-03-23T11:28:14.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.228+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:85 12.1c 12:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7800 con 0x5562ec28a800 2024-03-23T11:28:14.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.229+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 28 ==== osd_op_reply(85 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.229+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 100000001f6.00000000 2024-03-23T11:28:14.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.229+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:86 12.a 12:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6400 con 0x5562ecd15400 2024-03-23T11:28:14.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.230+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.2 v2:172.21.15.62:6800/36996 6 ==== osd_op_reply(86 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+54 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15400 2024-03-23T11:28:14.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.230+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-03-23T11:28:14.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.230+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-23T11:28:14.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.230+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 608.00000000 2024-03-23T11:28:14.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.230+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:87 12.e 12:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2b6400 con 0x5562ec28a800 2024-03-23T11:28:14.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.231+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 29 ==== osd_op_reply(87 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.231+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 604.00000000 2024-03-23T11:28:14.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.231+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:88 12.15 12:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c6c00 con 0x5562ec28bc00 2024-03-23T11:28:14.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.233+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 19 ==== osd_op_reply(88 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.233+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 500.00000000 2024-03-23T11:28:14.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.233+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-23T11:28:14.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.233+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-03-23T11:28:14.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.233+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-23T11:28:14.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.233+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 605.00000000 2024-03-23T11:28:14.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.233+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7000 con 0x5562ec28a800 2024-03-23T11:28:14.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.234+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 30 ==== osd_op_reply(89 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.234+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-03-23T11:28:14.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.234+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-23T11:28:14.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.234+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 100.00000000 2024-03-23T11:28:14.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.234+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:90 12.13 12:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7400 con 0x5562ecd15000 2024-03-23T11:28:14.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.236+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 11 ==== osd_op_reply(90 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.236+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-03-23T11:28:14.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.236+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-23T11:28:14.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.236+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 200.00000002 2024-03-23T11:28:14.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.236+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-03-23T11:28:14.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.236+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 609.00000000 2024-03-23T11:28:14.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.236+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:91 12.7 12:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7c00 con 0x5562ecd15000 2024-03-23T11:28:14.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.237+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 12 ==== osd_op_reply(91 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.237+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 400.00000000 2024-03-23T11:28:14.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.237+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-23T11:28:14.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.237+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-03-23T11:28:14.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.237+0000 7f4014a13740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-23T11:28:14.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.237+0000 7f4014a13740 10 datascan.scan_links: step 2: handling object 1.00000000 2024-03-23T11:28:14.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.237+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:92 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2c7800 con 0x5562ec28bc00 2024-03-23T11:28:14.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.238+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.6 v2:172.21.15.120:6800/37641 20 ==== osd_op_reply(92 1.00000000 [omap-get-vals out=1041b] v0'0 uv9 ondisk = 0) v8 ==== 154+0+1041 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28bc00 2024-03-23T11:28:14.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.238+0000 7f4014a13740 10 datascan.scan_links: processing 0 dup_primaries, 1 remote_links 2024-03-23T11:28:14.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.238+0000 7f4014a13740 10 datascan.scan_links: removing dup dentries from 0 objects 2024-03-23T11:28:14.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.238+0000 7f4014a13740 10 datascan.scan_links: processing 1 bad_nlink_inos 2024-03-23T11:28:14.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.238+0000 7f4014a13740 10 datascan.scan_links: handling bad_nlink_ino 0x10000000001 2024-03-23T11:28:14.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.238+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:93 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28b400 con 0x5562ec28a800 2024-03-23T11:28:14.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.239+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 31 ==== osd_op_reply(93 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv18 ondisk = 0) v8 ==== 164+0+505 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.239+0000 7f4014a13740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-03-23T11:28:14.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.239+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:94 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28ac00 con 0x5562ec28a800 2024-03-23T11:28:14.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.243+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 32 ==== osd_op_reply(94 10000000000.00000000 [omap-set-vals] v93'19 uv19 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.243+0000 7f4014a13740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-03-23T11:28:14.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.243+0000 7f4014a13740 10 datascan.scan_links: processing 1 injected_inos 2024-03-23T11:28:14.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.243+0000 7f4014a13740 10 datascan.scan_links: handling injected_ino 0x10000000001 2024-03-23T11:28:14.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.243+0000 7f4014a13740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000000:file 2024-03-23T11:28:14.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.243+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:95 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec28b000 con 0x5562ec28a800 2024-03-23T11:28:14.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.244+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 33 ==== osd_op_reply(95 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv19 ondisk = 0) v8 ==== 164+0+505 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.244+0000 7f4014a13740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-03-23T11:28:14.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.244+0000 7f4014a13740 20 datascan.scan_links: first is now 2 2024-03-23T11:28:14.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.244+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:96 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2d8000 con 0x5562ec28a800 2024-03-23T11:28:14.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.248+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.0 v2:172.21.15.62:6811/36998 34 ==== osd_op_reply(96 10000000000.00000000 [omap-set-vals] v93'20 uv20 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec28a800 2024-03-23T11:28:14.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.248+0000 7f4014a13740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-03-23T11:28:14.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.248+0000 7f4014a13740 10 datascan.scan_links: updating inotable 2024-03-23T11:28:14.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.248+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:97 12.13 12:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2d8400 con 0x5562ecd15000 2024-03-23T11:28:14.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.249+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 13 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv3 ondisk = 0) v8 ==== 157+0+50 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.249+0000 7f4014a13740 10 mds.0.inotable: skip_inos was [0x10000000002~0x1f4,0x100000001f7~0xfffffffe09] 2024-03-23T11:28:14.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.249+0000 7f4014a13740 10 mds.0.inotable: skip_inos now [0x100000001f7~0xfffffffe09] 2024-03-23T11:28:14.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.250+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:98 12.13 12:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2d8800 con 0x5562ecd15000 2024-03-23T11:28:14.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.254+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 <== osd.1 v2:172.21.15.62:6824/37001 14 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v93'5 uv5 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x5562ecdeeb40 con 0x5562ecd15000 2024-03-23T11:28:14.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.254+0000 7f4014a13740 10 datascan.scan_links: updating snaptable 2024-03-23T11:28:14.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.254+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:99 12.d 12:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93) v8 -- 0x5562ec2d8c00 con 0x5562ec2b6800 2024-03-23T11:28:14.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.255+0000 7f4012a00640 1 -- 172.21.15.62:0/722667902 <== osd.7 v2:172.21.15.120:6816/37655 3 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+70 (crc 0 0 0) 0x5562ecdeed80 con 0x5562ec2b6800 2024-03-23T11:28:14.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x5562ec28a800 msgr2=0x5562ecdf4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x5562ec28a800 0x5562ecdf4000 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5562ecd15000 msgr2=0x5562ecdf4b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5562ecd15000 0x5562ecdf4b00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x5562ecd15400 msgr2=0x5562ecdf6680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x5562ecd15400 0x5562ecdf6680 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5562ec28b800 msgr2=0x5562ecdf4580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5562ec28b800 0x5562ecdf4580 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x5562ec28a400 msgr2=0x5562ebfc2100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x5562ec28a400 0x5562ebfc2100 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x5562ec2b6000 msgr2=0x5562ecdf5b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x5562ec2b6000 0x5562ecdf5b80 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x5562ec28bc00 msgr2=0x5562ecdf5080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x5562ec28bc00 0x5562ecdf5080 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5562ec2b6800 msgr2=0x5562ecdf6100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5562ec2b6800 0x5562ecdf6100 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 reap_dead start 2024-03-23T11:28:14.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5562ec28a000 msgr2=0x5562ebfc3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.256+0000 7f4014a13740 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5562ec28a000 0x5562ebfc3700 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x5562ecd4ea80 tx=0x5562ecd5a000 comp rx=0 tx=0).stop 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.257+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05400 msgr2=0x5562ebfc2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.257+0000 7f4014a13740 1 --2- 172.21.15.62:0/722667902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ece05400 0x5562ebfc2c00 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x5562ece1b410 tx=0x5562ecccc300 comp rx=0 tx=0).stop 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.257+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 reap_dead start 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.257+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 reap_dead start 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.257+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 shutdown_connections 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.257+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 >> 172.21.15.62:0/722667902 conn(0x5562ece04c00 msgr2=0x5562eccb7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.257+0000 7f4011600640 1 -- 172.21.15.62:0/722667902 reap_dead start 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.257+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 shutdown_connections 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.257+0000 7f4014a13740 1 -- 172.21.15.62:0/722667902 wait complete. 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.258+0000 7f4014a13740 1 -- 172.21.15.62:0/701029444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ebf38c00 msgr2=0x5562ebfc0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.258+0000 7f4014a13740 1 --2- 172.21.15.62:0/701029444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ebf38c00 0x5562ebfc0580 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x5562ecd4f3b0 tx=0x5562ecd5ad80 comp rx=0 tx=0).stop 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.258+0000 7f4014a13740 1 -- 172.21.15.62:0/701029444 shutdown_connections 2024-03-23T11:28:14.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.258+0000 7f4014a13740 1 --2- 172.21.15.62:0/701029444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5562ebf39c00 0x5562ebfc1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.258+0000 7f4014a13740 1 --2- 172.21.15.62:0/701029444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5562ebf39800 0x5562ebfc0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.258+0000 7f4014a13740 1 --2- 172.21.15.62:0/701029444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5562ebf38c00 0x5562ebfc0580 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.258+0000 7f4014a13740 1 -- 172.21.15.62:0/701029444 >> 172.21.15.62:0/701029444 conn(0x5562ebf39000 msgr2=0x5562eccb6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:14.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.259+0000 7f4014a13740 1 -- 172.21.15.62:0/701029444 shutdown_connections 2024-03-23T11:28:14.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.259+0000 7f4014a13740 1 -- 172.21.15.62:0/701029444 wait complete. 2024-03-23T11:28:14.260 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.245030, result: 2024-03-23T11:28:14.260 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-23T11:28:14.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.497+0000 7fc110000640 1 Processor -- start 2024-03-23T11:28:14.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.498+0000 7fc110000640 1 -- start start 2024-03-23T11:28:14.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.499+0000 7fc110000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc1081571e0 0x7fc1080ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.499+0000 7fc110000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1080ade70 0x7fc10806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.499+0000 7fc110000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc1081613a0 0x7fc10815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.499+0000 7fc110000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc10806cf40 con 0x7fc1081571e0 2024-03-23T11:28:14.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.499+0000 7fc110000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1081575c0 con 0x7fc1081613a0 2024-03-23T11:28:14.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.499+0000 7fc110000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10806c3a0 con 0x7fc1080ade70 2024-03-23T11:28:14.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.500+0000 7fc10d600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc1081571e0 0x7fc1080ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.500+0000 7fc10d600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc1081571e0 0x7fc1080ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50954/0 (socket says 172.21.15.62:50954) 2024-03-23T11:28:14.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.500+0000 7fc10d600640 1 -- 172.21.15.62:0/2993091145 learned_addr learned my addr 172.21.15.62:0/2993091145 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:14.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.500+0000 7fc10d600640 1 -- 172.21.15.62:0/2993091145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1080ade70 msgr2=0x7fc10806d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:14.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.500+0000 7fc10d600640 1 --2- 172.21.15.62:0/2993091145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1080ade70 0x7fc10806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.500+0000 7fc10d600640 1 -- 172.21.15.62:0/2993091145 >> v2:172.21.15.120:3300/0 conn(0x7fc1081613a0 msgr2=0x7fc10815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:14.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.500+0000 7fc10d600640 1 --2- 172.21.15.62:0/2993091145 >> v2:172.21.15.120:3300/0 conn(0x7fc1081613a0 0x7fc10815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.500+0000 7fc10d600640 1 -- 172.21.15.62:0/2993091145 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10806c610 con 0x7fc1081571e0 2024-03-23T11:28:14.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.501+0000 7fc10d600640 1 --2- 172.21.15.62:0/2993091145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1081571e0 0x7fc1080ad930 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fc108073800 tx=0x7fc0fc02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f037aad9ec9964c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.502+0000 7fc107e00640 1 -- 172.21.15.62:0/2993091145 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc031800 con 0x7fc1081571e0 2024-03-23T11:28:14.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.502+0000 7fc107e00640 1 -- 172.21.15.62:0/2993091145 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0fc031960 con 0x7fc1081571e0 2024-03-23T11:28:14.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.502+0000 7fc107e00640 1 -- 172.21.15.62:0/2993091145 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc031cf0 con 0x7fc1081571e0 2024-03-23T11:28:14.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.502+0000 7fc110000640 1 -- 172.21.15.62:0/2993091145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1081571e0 msgr2=0x7fc1080ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.502+0000 7fc110000640 1 --2- 172.21.15.62:0/2993091145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1081571e0 0x7fc1080ad930 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fc108073800 tx=0x7fc0fc02b8a0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.502+0000 7fc110000640 1 -- 172.21.15.62:0/2993091145 shutdown_connections 2024-03-23T11:28:14.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.502+0000 7fc110000640 1 --2- 172.21.15.62:0/2993091145 >> v2:172.21.15.120:3300/0 conn(0x7fc1081613a0 0x7fc10815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.502+0000 7fc110000640 1 --2- 172.21.15.62:0/2993091145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1080ade70 0x7fc10806d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.502+0000 7fc110000640 1 --2- 172.21.15.62:0/2993091145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1081571e0 0x7fc1080ad930 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.502+0000 7fc110000640 1 -- 172.21.15.62:0/2993091145 >> 172.21.15.62:0/2993091145 conn(0x7fc1080b02e0 msgr2=0x7fc1080b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:14.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.503+0000 7fc110000640 1 -- 172.21.15.62:0/2993091145 shutdown_connections 2024-03-23T11:28:14.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.503+0000 7fc110000640 1 -- 172.21.15.62:0/2993091145 wait complete. 2024-03-23T11:28:14.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.504+0000 7fc110000640 1 Processor -- start 2024-03-23T11:28:14.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.504+0000 7fc110000640 1 -- start start 2024-03-23T11:28:14.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.505+0000 7fc110000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1080ade70 0x7fc10815ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.505+0000 7fc110000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1081571e0 0x7fc108151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.505+0000 7fc110000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1081613a0 0x7fc1081585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.505+0000 7fc110000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10815b130 con 0x7fc1081613a0 2024-03-23T11:28:14.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.505+0000 7fc110000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10815b410 con 0x7fc1080ade70 2024-03-23T11:28:14.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.505+0000 7fc110000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10815b6f0 con 0x7fc1081571e0 2024-03-23T11:28:14.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.506+0000 7fc10e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1081613a0 0x7fc1081585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.506+0000 7fc10e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1081613a0 0x7fc1081585a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50960/0 (socket says 172.21.15.62:50960) 2024-03-23T11:28:14.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.506+0000 7fc10e000640 1 -- 172.21.15.62:0/1476585405 learned_addr learned my addr 172.21.15.62:0/1476585405 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:14.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.506+0000 7fc10d600640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1080ade70 0x7fc10815ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.506+0000 7fc10cc00640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1081571e0 0x7fc108151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.506+0000 7fc10e000640 1 -- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1081571e0 msgr2=0x7fc108151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.506+0000 7fc10e000640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1081571e0 0x7fc108151ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.506+0000 7fc10e000640 1 -- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1080ade70 msgr2=0x7fc10815ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:14.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.506+0000 7fc10e000640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1080ade70 0x7fc10815ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:14.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.506+0000 7fc10e000640 1 -- 172.21.15.62:0/1476585405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0fc0094d0 con 0x7fc1081613a0 2024-03-23T11:28:14.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.507+0000 7fc10e000640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1081613a0 0x7fc1081585a0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fc10000b970 tx=0x7fc10000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.507+0000 7fc105600640 1 -- 172.21.15.62:0/1476585405 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10000cb40 con 0x7fc1081613a0 2024-03-23T11:28:14.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.507+0000 7fc105600640 1 -- 172.21.15.62:0/1476585405 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10000ccc0 con 0x7fc1081613a0 2024-03-23T11:28:14.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.507+0000 7fc105600640 1 -- 172.21.15.62:0/1476585405 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc100014680 con 0x7fc1081613a0 2024-03-23T11:28:14.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.507+0000 7fc110000640 1 -- 172.21.15.62:0/1476585405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10815b9d0 con 0x7fc1081613a0 2024-03-23T11:28:14.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.507+0000 7fc110000640 1 -- 172.21.15.62:0/1476585405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc108152400 con 0x7fc1081613a0 2024-03-23T11:28:14.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.508+0000 7fc110000640 1 -- 172.21.15.62:0/1476585405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1080ce3a0 con 0x7fc1081613a0 2024-03-23T11:28:14.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.509+0000 7fc105600640 1 -- 172.21.15.62:0/1476585405 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc10001d3b0 con 0x7fc1081613a0 2024-03-23T11:28:14.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.510+0000 7fc105600640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0ec06a7a0 0x7fc0ec06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:14.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.511+0000 7fc105600640 1 -- 172.21.15.62:0/1476585405 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fc100088e60 con 0x7fc1081613a0 2024-03-23T11:28:14.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.511+0000 7fc10d600640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0ec06a7a0 0x7fc0ec06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:14.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.512+0000 7fc10d600640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0ec06a7a0 0x7fc0ec06cc60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc036000 tx=0x7fc0fc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:14.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.514+0000 7fc105600640 1 -- 172.21.15.62:0/1476585405 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc100054110 con 0x7fc1081613a0 2024-03-23T11:28:14.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:14.753+0000 7fc110000640 1 -- 172.21.15.62:0/1476585405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7fc10815bef0 con 0x7fc1081613a0 2024-03-23T11:28:15.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.343+0000 7fc105600640 1 -- 172.21.15.62:0/1476585405 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v67) v1 ==== 162+0+0 (secure 0 0 0) 0x7fc10815bef0 con 0x7fc1081613a0 2024-03-23T11:28:15.335 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-23T11:28:15.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.346+0000 7fc110000640 1 -- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0ec06a7a0 msgr2=0x7fc0ec06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:15.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.346+0000 7fc110000640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0ec06a7a0 0x7fc0ec06cc60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc036000 tx=0x7fc0fc03c040 comp rx=0 tx=0).stop 2024-03-23T11:28:15.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.346+0000 7fc110000640 1 -- 172.21.15.62:0/1476585405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1081613a0 msgr2=0x7fc1081585a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:15.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.346+0000 7fc110000640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1081613a0 0x7fc1081585a0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fc10000b970 tx=0x7fc10000be30 comp rx=0 tx=0).stop 2024-03-23T11:28:15.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.347+0000 7fc110000640 1 -- 172.21.15.62:0/1476585405 shutdown_connections 2024-03-23T11:28:15.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.347+0000 7fc110000640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc0ec06a7a0 0x7fc0ec06cc60 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.347+0000 7fc110000640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1081613a0 0x7fc1081585a0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.347+0000 7fc110000640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1081571e0 0x7fc108151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.347+0000 7fc110000640 1 --2- 172.21.15.62:0/1476585405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1080ade70 0x7fc10815ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.347+0000 7fc110000640 1 -- 172.21.15.62:0/1476585405 >> 172.21.15.62:0/1476585405 conn(0x7fc1080b02e0 msgr2=0x7fc1080b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:15.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.347+0000 7fc110000640 1 -- 172.21.15.62:0/1476585405 shutdown_connections 2024-03-23T11:28:15.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.347+0000 7fc110000640 1 -- 172.21.15.62:0/1476585405 wait complete. 2024-03-23T11:28:15.360 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:28:15.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.566+0000 7f9020200640 1 Processor -- start 2024-03-23T11:28:15.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.566+0000 7f9020200640 1 -- start start 2024-03-23T11:28:15.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.568+0000 7f9020200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f90180cd7c0 0x7f90180b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:15.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.568+0000 7f9020200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90180cad20 con 0x7f901806c300 2024-03-23T11:28:15.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.568+0000 7f9020200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901806e130 con 0x7f901806dca0 2024-03-23T11:28:15.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.568+0000 7f9020200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901815a8a0 con 0x7f90180cd7c0 2024-03-23T11:28:15.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.568+0000 7f901d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f90180cd7c0 0x7f90180b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:15.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.568+0000 7f901d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f90180cd7c0 0x7f90180b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50974/0 (socket says 172.21.15.62:50974) 2024-03-23T11:28:15.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.568+0000 7f901d800640 1 -- 172.21.15.62:0/1074744111 learned_addr learned my addr 172.21.15.62:0/1074744111 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:15.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.568+0000 7f901e200640 1 --1- 172.21.15.62:0/1074744111 >> v1:172.21.15.62:6789/0 conn(0x7f901806c300 0x7f90180cb490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44058/0 (socket says 172.21.15.62:44058) 2024-03-23T11:28:15.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.569+0000 7f901d800640 1 -- 172.21.15.62:0/1074744111 >> v1:172.21.15.120:6789/0 conn(0x7f901806dca0 legacy=0x7f90180cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:15.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.569+0000 7f901d800640 1 -- 172.21.15.62:0/1074744111 >> v1:172.21.15.62:6789/0 conn(0x7f901806c300 legacy=0x7f90180cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:15.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.569+0000 7f901d800640 1 -- 172.21.15.62:0/1074744111 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901815ab80 con 0x7f90180cd7c0 2024-03-23T11:28:15.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.570+0000 7f901d800640 1 --2- 172.21.15.62:0/1074744111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90180cd7c0 0x7f90180b15e0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f900c009870 tx=0x7f900c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f99e97c44f98bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:15.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.571+0000 7f9017e00640 1 -- 172.21.15.62:0/1074744111 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c031870 con 0x7f90180cd7c0 2024-03-23T11:28:15.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.571+0000 7f9017e00640 1 -- 172.21.15.62:0/1074744111 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c0319d0 con 0x7f90180cd7c0 2024-03-23T11:28:15.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.571+0000 7f9017e00640 1 -- 172.21.15.62:0/1074744111 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c031d60 con 0x7f90180cd7c0 2024-03-23T11:28:15.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.571+0000 7f9020200640 1 -- 172.21.15.62:0/1074744111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90180cd7c0 msgr2=0x7f90180b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:15.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.571+0000 7f9020200640 1 --2- 172.21.15.62:0/1074744111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90180cd7c0 0x7f90180b15e0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f900c009870 tx=0x7f900c02b910 comp rx=0 tx=0).stop 2024-03-23T11:28:15.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.572+0000 7f9020200640 1 -- 172.21.15.62:0/1074744111 shutdown_connections 2024-03-23T11:28:15.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.572+0000 7f9020200640 1 --2- 172.21.15.62:0/1074744111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90180cd7c0 0x7f90180b15e0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.572+0000 7f9020200640 1 -- 172.21.15.62:0/1074744111 >> 172.21.15.62:0/1074744111 conn(0x7f90180b02e0 msgr2=0x7f90180b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:15.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.572+0000 7f9020200640 1 -- 172.21.15.62:0/1074744111 shutdown_connections 2024-03-23T11:28:15.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.572+0000 7f9020200640 1 -- 172.21.15.62:0/1074744111 wait complete. 2024-03-23T11:28:15.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.574+0000 7f9020200640 1 Processor -- start 2024-03-23T11:28:15.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.574+0000 7f9020200640 1 -- start start 2024-03-23T11:28:15.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f9020200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f901806c300 0x7f90180acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:15.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f9020200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90180cd7c0 0x7f90181582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:15.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f9020200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9018155100 0x7f9018155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:15.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f9020200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90181547d0 con 0x7f90180cd7c0 2024-03-23T11:28:15.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f9020200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90181538e0 con 0x7f9018155100 2024-03-23T11:28:15.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f9020200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9018153bc0 con 0x7f901806c300 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f901ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90180cd7c0 0x7f90181582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f901ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90180cd7c0 0x7f90181582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50982/0 (socket says 172.21.15.62:50982) 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f901ce00640 1 -- 172.21.15.62:0/316888304 learned_addr learned my addr 172.21.15.62:0/316888304 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f901d800640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f901806c300 0x7f90180acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f901ce00640 1 -- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f901806c300 msgr2=0x7f90180acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f901ce00640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f901806c300 0x7f90180acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f901ce00640 1 -- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9018155100 msgr2=0x7f9018155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.575+0000 7f901e200640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9018155100 0x7f9018155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.576+0000 7f901ce00640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9018155100 0x7f9018155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.576+0000 7f901ce00640 1 -- 172.21.15.62:0/316888304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c0094d0 con 0x7f90180cd7c0 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.576+0000 7f901e200640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9018155100 0x7f9018155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.576+0000 7f901ce00640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90180cd7c0 0x7f90181582a0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f900400c430 tx=0x7f900400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.576+0000 7f9015600640 1 -- 172.21.15.62:0/316888304 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004011790 con 0x7f90180cd7c0 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.576+0000 7f9015600640 1 -- 172.21.15.62:0/316888304 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90040118f0 con 0x7f90180cd7c0 2024-03-23T11:28:15.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.576+0000 7f9020200640 1 -- 172.21.15.62:0/316888304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9018153ea0 con 0x7f90180cd7c0 2024-03-23T11:28:15.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.576+0000 7f9015600640 1 -- 172.21.15.62:0/316888304 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004011ce0 con 0x7f90180cd7c0 2024-03-23T11:28:15.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.576+0000 7f9020200640 1 -- 172.21.15.62:0/316888304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9018154460 con 0x7f90180cd7c0 2024-03-23T11:28:15.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.578+0000 7f9020200640 1 -- 172.21.15.62:0/316888304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9018154980 con 0x7f90180cd7c0 2024-03-23T11:28:15.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.578+0000 7f9015600640 1 -- 172.21.15.62:0/316888304 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f900401c460 con 0x7f90180cd7c0 2024-03-23T11:28:15.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.579+0000 7f9015600640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ff806a7a0 0x7f8ff806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:15.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.579+0000 7f9015600640 1 -- 172.21.15.62:0/316888304 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f9004088ee0 con 0x7f90180cd7c0 2024-03-23T11:28:15.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.580+0000 7f901d800640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ff806a7a0 0x7f8ff806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:15.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.581+0000 7f901d800640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ff806a7a0 0x7f8ff806cc60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f900c036000 tx=0x7f900c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:15.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.584+0000 7f9015600640 1 -- 172.21.15.62:0/316888304 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9018154980 con 0x7f90180cd7c0 2024-03-23T11:28:15.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.823+0000 7f9020200640 1 -- 172.21.15.62:0/316888304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9018066c90 con 0x7f90180cd7c0 2024-03-23T11:28:15.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.824+0000 7f9015600640 1 -- 172.21.15.62:0/316888304 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f9004054100 con 0x7f90180cd7c0 2024-03-23T11:28:15.816 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:15.816 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":67,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:28:15.335439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"a","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.62:6833/1594260761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1594260761},{"type":"v1","addr":"172.21.15.62:6833","nonce":1594260761}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:28:15.817 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 67 2024-03-23T11:28:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.827+0000 7f9020200640 1 -- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ff806a7a0 msgr2=0x7f8ff806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.827+0000 7f9020200640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ff806a7a0 0x7f8ff806cc60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f900c036000 tx=0x7f900c03c040 comp rx=0 tx=0).stop 2024-03-23T11:28:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.827+0000 7f9020200640 1 -- 172.21.15.62:0/316888304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90180cd7c0 msgr2=0x7f90181582a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.827+0000 7f9020200640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90180cd7c0 0x7f90181582a0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f900400c430 tx=0x7f900400c8f0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.827+0000 7f9020200640 1 -- 172.21.15.62:0/316888304 shutdown_connections 2024-03-23T11:28:15.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.827+0000 7f9020200640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ff806a7a0 0x7f8ff806cc60 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.827+0000 7f9020200640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90180cd7c0 0x7f90181582a0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.827+0000 7f9020200640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9018155100 0x7f9018155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.827+0000 7f9020200640 1 --2- 172.21.15.62:0/316888304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f901806c300 0x7f90180acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:15.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.827+0000 7f9020200640 1 -- 172.21.15.62:0/316888304 >> 172.21.15.62:0/316888304 conn(0x7f90180b02e0 msgr2=0x7f901815c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:15.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.827+0000 7f9020200640 1 -- 172.21.15.62:0/316888304 shutdown_connections 2024-03-23T11:28:15.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:15.828+0000 7f9020200640 1 -- 172.21.15.62:0/316888304 wait complete. 2024-03-23T11:28:16.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:26:25.131291+0000', 'modified': '2024-03-23T11:28:15.335439+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 93, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7926}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7926': {'gid': 7926, 'name': 'a', 'rank': 0, 'incarnation': 67, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.62:6833/1594260761', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 1594260761}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 1594260761}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:28:16.240 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7926:up:replay 2024-03-23T11:28:17.026 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:28:17.034 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:28:17.241 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:28:17.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.477+0000 7fbb4f200640 1 Processor -- start 2024-03-23T11:28:17.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.478+0000 7fbb4f200640 1 -- start start 2024-03-23T11:28:17.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.479+0000 7fbb4f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb4815fcf0 0x7fbb480d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:17.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.479+0000 7fbb4f200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbb48160420 0x7fbb48160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:17.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.479+0000 7fbb4f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbb480d06d0 0x7fbb480d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:17.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.479+0000 7fbb4f200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb48160dc0 con 0x7fbb48160420 2024-03-23T11:28:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.479+0000 7fbb4f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb480d3540 con 0x7fbb480d06d0 2024-03-23T11:28:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.479+0000 7fbb4f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480ce510 con 0x7fbb4815fcf0 2024-03-23T11:28:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.479+0000 7fbb4d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbb48160420 0x7fbb48160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.480+0000 7fbb4d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbb48160420 0x7fbb48160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58810/0 (socket says 172.21.15.62:58810) 2024-03-23T11:28:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.480+0000 7fbb4d400640 1 -- 172.21.15.62:0/1442403119 learned_addr learned my addr 172.21.15.62:0/1442403119 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.480+0000 7fbb4de00640 1 --2- 172.21.15.62:0/1442403119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb4815fcf0 0x7fbb480d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.480+0000 7fbb4d400640 1 -- 172.21.15.62:0/1442403119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb4815fcf0 msgr2=0x7fbb480d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.480+0000 7fbb4e800640 1 --2- 172.21.15.62:0/1442403119 >> v2:172.21.15.120:3300/0 conn(0x7fbb480d06d0 0x7fbb480d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.480+0000 7fbb4d400640 1 --2- 172.21.15.62:0/1442403119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb4815fcf0 0x7fbb480d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.480+0000 7fbb4d400640 1 -- 172.21.15.62:0/1442403119 >> v2:172.21.15.120:3300/0 conn(0x7fbb480d06d0 msgr2=0x7fbb480d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.480+0000 7fbb4d400640 1 --2- 172.21.15.62:0/1442403119 >> v2:172.21.15.120:3300/0 conn(0x7fbb480d06d0 0x7fbb480d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.480+0000 7fbb4d400640 1 -- 172.21.15.62:0/1442403119 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb480cdc70 con 0x7fbb48160420 2024-03-23T11:28:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.480+0000 7fbb4de00640 1 --2- 172.21.15.62:0/1442403119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb4815fcf0 0x7fbb480d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.480+0000 7fbb4e800640 1 --2- 172.21.15.62:0/1442403119 >> v2:172.21.15.120:3300/0 conn(0x7fbb480d06d0 0x7fbb480d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.481+0000 7fbb4d400640 1 --2- 172.21.15.62:0/1442403119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb48160420 0x7fbb48160880 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fbb4808c3e0 tx=0x7fbb3402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379af9e6e0ae2e1c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:17.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.482+0000 7fbb4ca00640 1 -- 172.21.15.62:0/1442403119 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb340317d0 con 0x7fbb48160420 2024-03-23T11:28:17.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.482+0000 7fbb4ca00640 1 -- 172.21.15.62:0/1442403119 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb34031930 con 0x7fbb48160420 2024-03-23T11:28:17.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.482+0000 7fbb4ca00640 1 -- 172.21.15.62:0/1442403119 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb34031cc0 con 0x7fbb48160420 2024-03-23T11:28:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.482+0000 7fbb4f200640 1 -- 172.21.15.62:0/1442403119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb48160420 msgr2=0x7fbb48160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.482+0000 7fbb4f200640 1 --2- 172.21.15.62:0/1442403119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb48160420 0x7fbb48160880 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fbb4808c3e0 tx=0x7fbb3402b870 comp rx=0 tx=0).stop 2024-03-23T11:28:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.482+0000 7fbb4f200640 1 -- 172.21.15.62:0/1442403119 shutdown_connections 2024-03-23T11:28:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.482+0000 7fbb4f200640 1 --2- 172.21.15.62:0/1442403119 >> v2:172.21.15.120:3300/0 conn(0x7fbb480d06d0 0x7fbb480d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.482+0000 7fbb4f200640 1 --2- 172.21.15.62:0/1442403119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb48160420 0x7fbb48160880 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.482+0000 7fbb4f200640 1 --2- 172.21.15.62:0/1442403119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb4815fcf0 0x7fbb480d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.483+0000 7fbb4f200640 1 -- 172.21.15.62:0/1442403119 >> 172.21.15.62:0/1442403119 conn(0x7fbb48089f60 msgr2=0x7fbb480d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.483+0000 7fbb4f200640 1 -- 172.21.15.62:0/1442403119 shutdown_connections 2024-03-23T11:28:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.483+0000 7fbb4f200640 1 -- 172.21.15.62:0/1442403119 wait complete. 2024-03-23T11:28:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.484+0000 7fbb4f200640 1 Processor -- start 2024-03-23T11:28:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.485+0000 7fbb4f200640 1 -- start start 2024-03-23T11:28:17.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.485+0000 7fbb4f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb480d06d0 0x7fbb480c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:17.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.485+0000 7fbb4f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb4815fcf0 0x7fbb4809ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:17.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.485+0000 7fbb4f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb48160420 0x7fbb4815bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.485+0000 7fbb4f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480caaa0 con 0x7fbb48160420 2024-03-23T11:28:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.485+0000 7fbb4f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480a0440 con 0x7fbb480d06d0 2024-03-23T11:28:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.485+0000 7fbb4f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480c9e20 con 0x7fbb4815fcf0 2024-03-23T11:28:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.486+0000 7fbb4e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb48160420 0x7fbb4815bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.486+0000 7fbb4e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb48160420 0x7fbb4815bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58814/0 (socket says 172.21.15.62:58814) 2024-03-23T11:28:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.486+0000 7fbb4e800640 1 -- 172.21.15.62:0/2004641468 learned_addr learned my addr 172.21.15.62:0/2004641468 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:17.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.486+0000 7fbb4d400640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb4815fcf0 0x7fbb4809ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:17.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.486+0000 7fbb4e800640 1 -- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb4815fcf0 msgr2=0x7fbb4809ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:17.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.486+0000 7fbb4e800640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb4815fcf0 0x7fbb4809ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:17.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.486+0000 7fbb4e800640 1 -- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb480d06d0 msgr2=0x7fbb480c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:17.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.486+0000 7fbb4e800640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb480d06d0 0x7fbb480c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:17.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.486+0000 7fbb4e800640 1 -- 172.21.15.62:0/2004641468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb340094d0 con 0x7fbb48160420 2024-03-23T11:28:17.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.487+0000 7fbb4e800640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb48160420 0x7fbb4815bca0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400cd10 tx=0x7fbb4400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:17.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.487+0000 7fbb42000640 1 -- 172.21.15.62:0/2004641468 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4400f680 con 0x7fbb48160420 2024-03-23T11:28:17.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.487+0000 7fbb42000640 1 -- 172.21.15.62:0/2004641468 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4400f800 con 0x7fbb48160420 2024-03-23T11:28:17.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.487+0000 7fbb4f200640 1 -- 172.21.15.62:0/2004641468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb480ca100 con 0x7fbb48160420 2024-03-23T11:28:17.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.487+0000 7fbb42000640 1 -- 172.21.15.62:0/2004641468 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4400fbb0 con 0x7fbb48160420 2024-03-23T11:28:17.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.487+0000 7fbb4f200640 1 -- 172.21.15.62:0/2004641468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb480ca6c0 con 0x7fbb48160420 2024-03-23T11:28:17.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.488+0000 7fbb4f200640 1 -- 172.21.15.62:0/2004641468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb48066bb0 con 0x7fbb48160420 2024-03-23T11:28:17.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.489+0000 7fbb42000640 1 -- 172.21.15.62:0/2004641468 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbb4400b040 con 0x7fbb48160420 2024-03-23T11:28:17.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.490+0000 7fbb42000640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbb2806a7a0 0x7fbb2806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:17.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.490+0000 7fbb42000640 1 -- 172.21.15.62:0/2004641468 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fbb44089880 con 0x7fbb48160420 2024-03-23T11:28:17.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.491+0000 7fbb4de00640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbb2806a7a0 0x7fbb2806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:17.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.492+0000 7fbb4de00640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbb2806a7a0 0x7fbb2806cc60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c006fd0 tx=0x7fbb3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:17.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.495+0000 7fbb42000640 1 -- 172.21.15.62:0/2004641468 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbb44054b30 con 0x7fbb48160420 2024-03-23T11:28:17.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.741+0000 7fbb4f200640 1 -- 172.21.15.62:0/2004641468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb480cabe0 con 0x7fbb48160420 2024-03-23T11:28:17.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.742+0000 7fbb42000640 1 -- 172.21.15.62:0/2004641468 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fbb480cabe0 con 0x7fbb48160420 2024-03-23T11:28:17.734 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:17.734 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":69,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:28:17.358928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"a","rank":0,"incarnation":67,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.62:6833/1594260761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1594260761},{"type":"v1","addr":"172.21.15.62:6833","nonce":1594260761}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:28:17.734 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 69 2024-03-23T11:28:17.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 -- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbb2806a7a0 msgr2=0x7fbb2806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:17.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbb2806a7a0 0x7fbb2806cc60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c006fd0 tx=0x7fbb3c008040 comp rx=0 tx=0).stop 2024-03-23T11:28:17.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 -- 172.21.15.62:0/2004641468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb48160420 msgr2=0x7fbb4815bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:17.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb48160420 0x7fbb4815bca0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400cd10 tx=0x7fbb4400a650 comp rx=0 tx=0).stop 2024-03-23T11:28:17.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 -- 172.21.15.62:0/2004641468 shutdown_connections 2024-03-23T11:28:17.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbb2806a7a0 0x7fbb2806cc60 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:17.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb48160420 0x7fbb4815bca0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:17.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbb4815fcf0 0x7fbb4809ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:17.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 --2- 172.21.15.62:0/2004641468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb480d06d0 0x7fbb480c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:17.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 -- 172.21.15.62:0/2004641468 >> 172.21.15.62:0/2004641468 conn(0x7fbb48089f60 msgr2=0x7fbb480b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:17.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 -- 172.21.15.62:0/2004641468 shutdown_connections 2024-03-23T11:28:17.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:17.745+0000 7fbb4f200640 1 -- 172.21.15.62:0/2004641468 wait complete. 2024-03-23T11:28:18.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:26:25.131291+0000', 'modified': '2024-03-23T11:28:17.358928+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 93, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7926}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7926': {'gid': 7926, 'name': 'a', 'rank': 0, 'incarnation': 67, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.62:6833/1594260761', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 1594260761}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 1594260761}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:28:18.157 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7926:up:rejoin 2024-03-23T11:28:19.157 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:28:19.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.399+0000 7f8fcca00640 1 Processor -- start 2024-03-23T11:28:19.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.399+0000 7f8fcca00640 1 -- start start 2024-03-23T11:28:19.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.401+0000 7f8fcca00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fc81571e0 0x7f8fc80ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:19.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.401+0000 7f8fcca00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8fc81613a0 0x7f8fc815e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:19.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.401+0000 7f8fcca00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc806cf40 con 0x7f8fc80ade70 2024-03-23T11:28:19.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.401+0000 7f8fcca00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc81575c0 con 0x7f8fc81571e0 2024-03-23T11:28:19.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.401+0000 7f8fcca00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc806c3a0 con 0x7f8fc81613a0 2024-03-23T11:28:19.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.401+0000 7f8fc5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fc81571e0 0x7f8fc80ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:19.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.401+0000 7f8fc5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fc81571e0 0x7f8fc80ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58828/0 (socket says 172.21.15.62:58828) 2024-03-23T11:28:19.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.401+0000 7f8fc5e00640 1 -- 172.21.15.62:0/2855137230 learned_addr learned my addr 172.21.15.62:0/2855137230 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:19.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.402+0000 7f8fc5e00640 1 -- 172.21.15.62:0/2855137230 >> v2:172.21.15.120:3300/0 conn(0x7f8fc81613a0 msgr2=0x7f8fc815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:19.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.402+0000 7f8fc5e00640 1 --2- 172.21.15.62:0/2855137230 >> v2:172.21.15.120:3300/0 conn(0x7f8fc81613a0 0x7f8fc815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:19.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.402+0000 7f8fc5e00640 1 -- 172.21.15.62:0/2855137230 >> v1:172.21.15.120:6789/0 conn(0x7f8fc80ade70 legacy=0x7f8fc806d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:19.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.402+0000 7f8fc5e00640 1 -- 172.21.15.62:0/2855137230 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc806c610 con 0x7f8fc81571e0 2024-03-23T11:28:19.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.403+0000 7f8fc5e00640 1 --2- 172.21.15.62:0/2855137230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fc81571e0 0x7f8fc80ad930 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80ac760 tx=0x7f8fb802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c06c990331caaad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:19.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.403+0000 7f8fc4a00640 1 -- 172.21.15.62:0/2855137230 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb80317e0 con 0x7f8fc81571e0 2024-03-23T11:28:19.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.403+0000 7f8fc4a00640 1 -- 172.21.15.62:0/2855137230 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb8031940 con 0x7f8fc81571e0 2024-03-23T11:28:19.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.403+0000 7f8fc4a00640 1 -- 172.21.15.62:0/2855137230 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8031cd0 con 0x7f8fc81571e0 2024-03-23T11:28:19.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.404+0000 7f8fcca00640 1 -- 172.21.15.62:0/2855137230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fc81571e0 msgr2=0x7f8fc80ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:19.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.404+0000 7f8fcca00640 1 --2- 172.21.15.62:0/2855137230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fc81571e0 0x7f8fc80ad930 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80ac760 tx=0x7f8fb802b880 comp rx=0 tx=0).stop 2024-03-23T11:28:19.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.404+0000 7f8fcca00640 1 -- 172.21.15.62:0/2855137230 shutdown_connections 2024-03-23T11:28:19.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.404+0000 7f8fcca00640 1 --2- 172.21.15.62:0/2855137230 >> v2:172.21.15.120:3300/0 conn(0x7f8fc81613a0 0x7f8fc815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:19.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.404+0000 7f8fcca00640 1 --2- 172.21.15.62:0/2855137230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fc81571e0 0x7f8fc80ad930 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:19.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.404+0000 7f8fcca00640 1 -- 172.21.15.62:0/2855137230 >> 172.21.15.62:0/2855137230 conn(0x7f8fc80b02e0 msgr2=0x7f8fc80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:19.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.404+0000 7f8fcca00640 1 -- 172.21.15.62:0/2855137230 shutdown_connections 2024-03-23T11:28:19.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.404+0000 7f8fcca00640 1 -- 172.21.15.62:0/2855137230 wait complete. 2024-03-23T11:28:19.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.406+0000 7f8fcca00640 1 Processor -- start 2024-03-23T11:28:19.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.406+0000 7f8fcca00640 1 -- start start 2024-03-23T11:28:19.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.407+0000 7f8fcca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fc80ade70 0x7f8fc815c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:19.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.407+0000 7f8fcca00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc81571e0 0x7f8fc8159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:19.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.407+0000 7f8fcca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc81613a0 0x7f8fc815a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:19.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.407+0000 7f8fcca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc81f2380 con 0x7f8fc80ade70 2024-03-23T11:28:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.407+0000 7f8fcca00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc8157ad0 con 0x7f8fc81571e0 2024-03-23T11:28:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.407+0000 7f8fcca00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc8157db0 con 0x7f8fc81613a0 2024-03-23T11:28:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.407+0000 7f8fc5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fc80ade70 0x7f8fc815c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.407+0000 7f8fc5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fc80ade70 0x7f8fc815c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58842/0 (socket says 172.21.15.62:58842) 2024-03-23T11:28:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.407+0000 7f8fc5e00640 1 -- 172.21.15.62:0/2783025921 learned_addr learned my addr 172.21.15.62:0/2783025921 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:19.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fc5400640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc81571e0 0x7f8fc8159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:19.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fc5e00640 1 -- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc81613a0 msgr2=0x7f8fc815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:19.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fc6800640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc81613a0 0x7f8fc815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fc5e00640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc81613a0 0x7f8fc815a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fc5e00640 1 -- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc81571e0 msgr2=0x7f8fc8159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fc5e00640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc81571e0 0x7f8fc8159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fc5e00640 1 -- 172.21.15.62:0/2783025921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb80094d0 con 0x7f8fc80ade70 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fc5400640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc81571e0 0x7f8fc8159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fc5e00640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fc80ade70 0x7f8fc815c6e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f8fb802b8c0 tx=0x7f8fb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fbe000640 1 -- 172.21.15.62:0/2783025921 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8032940 con 0x7f8fc80ade70 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fbe000640 1 -- 172.21.15.62:0/2783025921 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb8032aa0 con 0x7f8fc80ade70 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.408+0000 7f8fcca00640 1 -- 172.21.15.62:0/2783025921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc8158090 con 0x7f8fc80ade70 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.409+0000 7f8fbe000640 1 -- 172.21.15.62:0/2783025921 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8032e50 con 0x7f8fc80ade70 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.409+0000 7f8fcca00640 1 -- 172.21.15.62:0/2783025921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc8158650 con 0x7f8fc80ade70 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.410+0000 7f8fcca00640 1 -- 172.21.15.62:0/2783025921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc8066c90 con 0x7f8fc80ade70 2024-03-23T11:28:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.411+0000 7f8fbe000640 1 -- 172.21.15.62:0/2783025921 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8fb804e450 con 0x7f8fc80ade70 2024-03-23T11:28:19.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.411+0000 7f8fbe000640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fa406a7f0 0x7f8fa406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:19.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.412+0000 7f8fbe000640 1 -- 172.21.15.62:0/2783025921 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f8fb80ab150 con 0x7f8fc80ade70 2024-03-23T11:28:19.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.412+0000 7f8fc5400640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fa406a7f0 0x7f8fa406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:19.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.413+0000 7f8fc5400640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fa406a7f0 0x7f8fa406ccb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80ad150 tx=0x7f8fb0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:19.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.416+0000 7f8fbe000640 1 -- 172.21.15.62:0/2783025921 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8fb8076530 con 0x7f8fc80ade70 2024-03-23T11:28:19.460 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:28:19.468+0000 7fa2d0200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:28:19.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.655+0000 7f8fcca00640 1 -- 172.21.15.62:0/2783025921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fc8154fa0 con 0x7f8fc80ade70 2024-03-23T11:28:19.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.656+0000 7f8fbe000640 1 -- 172.21.15.62:0/2783025921 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f8fb8079b50 con 0x7f8fc80ade70 2024-03-23T11:28:19.649 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:19.649 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":70,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:28:18.362214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6833/1594260761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1594260761},{"type":"v1","addr":"172.21.15.62:6833","nonce":1594260761}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:28:19.649 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 70 2024-03-23T11:28:19.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.659+0000 7f8fcca00640 1 -- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fa406a7f0 msgr2=0x7f8fa406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:19.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.659+0000 7f8fcca00640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fa406a7f0 0x7f8fa406ccb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80ad150 tx=0x7f8fb0008040 comp rx=0 tx=0).stop 2024-03-23T11:28:19.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.659+0000 7f8fcca00640 1 -- 172.21.15.62:0/2783025921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fc80ade70 msgr2=0x7f8fc815c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:19.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.659+0000 7f8fcca00640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fc80ade70 0x7f8fc815c6e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f8fb802b8c0 tx=0x7f8fb80046c0 comp rx=0 tx=0).stop 2024-03-23T11:28:19.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.659+0000 7f8fcca00640 1 -- 172.21.15.62:0/2783025921 shutdown_connections 2024-03-23T11:28:19.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.659+0000 7f8fcca00640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fa406a7f0 0x7f8fa406ccb0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:19.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.659+0000 7f8fcca00640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fc81613a0 0x7f8fc815a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:19.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.659+0000 7f8fcca00640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fc81571e0 0x7f8fc8159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:19.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.659+0000 7f8fcca00640 1 --2- 172.21.15.62:0/2783025921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fc80ade70 0x7f8fc815c6e0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:19.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.659+0000 7f8fcca00640 1 -- 172.21.15.62:0/2783025921 >> 172.21.15.62:0/2783025921 conn(0x7f8fc80b02e0 msgr2=0x7f8fc815ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:19.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.659+0000 7f8fcca00640 1 -- 172.21.15.62:0/2783025921 shutdown_connections 2024-03-23T11:28:19.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:19.660+0000 7f8fcca00640 1 -- 172.21.15.62:0/2783025921 wait complete. 2024-03-23T11:28:20.081 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:26:25.131291+0000', 'modified': '2024-03-23T11:28:18.362214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 93, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7926}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7926': {'gid': 7926, 'name': 'a', 'rank': 0, 'incarnation': 67, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.62:6833/1594260761', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 1594260761}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 1594260761}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:28:20.081 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:28:20.081 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T11:28:20.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.319+0000 7f833fe00640 1 Processor -- start 2024-03-23T11:28:20.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.319+0000 7f833fe00640 1 -- start start 2024-03-23T11:28:20.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.321+0000 7f833fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83400ad370 0x7f83400b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.321+0000 7f833fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8340069c60 0x7f834006a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.321+0000 7f833fe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400abe50 con 0x7f834015c730 2024-03-23T11:28:20.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.321+0000 7f833fe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f834006a630 con 0x7f8340069c60 2024-03-23T11:28:20.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.321+0000 7f833fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f834006cba0 con 0x7f83400ad370 2024-03-23T11:28:20.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.322+0000 7f833f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8340069c60 0x7f834006a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.322+0000 7f833ea00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f834015c730 0x7f8340160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54538/0 (socket says 172.21.15.62:54538) 2024-03-23T11:28:20.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.322+0000 7f833f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8340069c60 0x7f834006a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58856/0 (socket says 172.21.15.62:58856) 2024-03-23T11:28:20.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.322+0000 7f833ea00640 1 -- 172.21.15.62:0/4207820641 learned_addr learned my addr 172.21.15.62:0/4207820641 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:20.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.322+0000 7f8337e00640 1 --2- 172.21.15.62:0/4207820641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83400ad370 0x7f83400b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.322+0000 7f833f400640 1 -- 172.21.15.62:0/4207820641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83400ad370 msgr2=0x7f83400b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:20.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.322+0000 7f833f400640 1 --2- 172.21.15.62:0/4207820641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83400ad370 0x7f83400b1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.322+0000 7f833f400640 1 -- 172.21.15.62:0/4207820641 >> v1:172.21.15.62:6789/0 conn(0x7f834015c730 legacy=0x7f8340160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.322+0000 7f833f400640 1 -- 172.21.15.62:0/4207820641 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834006ce80 con 0x7f8340069c60 2024-03-23T11:28:20.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.323+0000 7f833f400640 1 --2- 172.21.15.62:0/4207820641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8340069c60 0x7f834006a0f0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f8338009d90 tx=0x7f833802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc96ce9baf556253 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:20.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.324+0000 7f833e000640 1 -- 172.21.15.62:0/4207820641 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833802cc60 con 0x7f8340069c60 2024-03-23T11:28:20.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.324+0000 7f833e000640 1 -- 172.21.15.62:0/4207820641 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833802cdc0 con 0x7f8340069c60 2024-03-23T11:28:20.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.324+0000 7f833e000640 1 -- 172.21.15.62:0/4207820641 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338034660 con 0x7f8340069c60 2024-03-23T11:28:20.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.324+0000 7f833fe00640 1 -- 172.21.15.62:0/4207820641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8340069c60 msgr2=0x7f834006a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.324+0000 7f833fe00640 1 --2- 172.21.15.62:0/4207820641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8340069c60 0x7f834006a0f0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f8338009d90 tx=0x7f833802beb0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.325+0000 7f833fe00640 1 -- 172.21.15.62:0/4207820641 shutdown_connections 2024-03-23T11:28:20.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.325+0000 7f833fe00640 1 --2- 172.21.15.62:0/4207820641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8340069c60 0x7f834006a0f0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.325+0000 7f833fe00640 1 --2- 172.21.15.62:0/4207820641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83400ad370 0x7f83400b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.325+0000 7f833fe00640 1 -- 172.21.15.62:0/4207820641 >> 172.21.15.62:0/4207820641 conn(0x7f83400b0090 msgr2=0x7f83400b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:20.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.325+0000 7f833fe00640 1 -- 172.21.15.62:0/4207820641 shutdown_connections 2024-03-23T11:28:20.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.325+0000 7f833fe00640 1 -- 172.21.15.62:0/4207820641 wait complete. 2024-03-23T11:28:20.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.327+0000 7f833fe00640 1 Processor -- start 2024-03-23T11:28:20.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.327+0000 7f833fe00640 1 -- start start 2024-03-23T11:28:20.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.328+0000 7f833fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8340069c60 0x7f83400c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.328+0000 7f833fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83400ad370 0x7f834015b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.328+0000 7f833fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834015c730 0x7f83401f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.328+0000 7f833fe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83400c3e70 con 0x7f834015c730 2024-03-23T11:28:20.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.328+0000 7f833fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83400c4150 con 0x7f83400ad370 2024-03-23T11:28:20.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.328+0000 7f833fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83400c4430 con 0x7f8340069c60 2024-03-23T11:28:20.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.329+0000 7f833f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834015c730 0x7f83401f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.329+0000 7f833f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834015c730 0x7f83401f1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58862/0 (socket says 172.21.15.62:58862) 2024-03-23T11:28:20.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.329+0000 7f833f400640 1 -- 172.21.15.62:0/1964733384 learned_addr learned my addr 172.21.15.62:0/1964733384 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:20.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.329+0000 7f833ea00640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8340069c60 0x7f83400c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.329+0000 7f8337e00640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83400ad370 0x7f834015b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.329+0000 7f833f400640 1 -- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8340069c60 msgr2=0x7f83400c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.329+0000 7f833f400640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8340069c60 0x7f83400c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.329+0000 7f833f400640 1 -- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83400ad370 msgr2=0x7f834015b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.329+0000 7f833f400640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83400ad370 0x7f834015b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.329+0000 7f833f400640 1 -- 172.21.15.62:0/1964733384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8338009a70 con 0x7f834015c730 2024-03-23T11:28:20.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.329+0000 7f833f400640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834015c730 0x7f83401f1d80 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f8338009d90 tx=0x7f8338004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:20.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.330+0000 7f8337400640 1 -- 172.21.15.62:0/1964733384 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338040650 con 0x7f834015c730 2024-03-23T11:28:20.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.330+0000 7f8337400640 1 -- 172.21.15.62:0/1964733384 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83380407d0 con 0x7f834015c730 2024-03-23T11:28:20.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.330+0000 7f8337400640 1 -- 172.21.15.62:0/1964733384 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338040b80 con 0x7f834015c730 2024-03-23T11:28:20.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.330+0000 7f833fe00640 1 -- 172.21.15.62:0/1964733384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83400c4710 con 0x7f834015c730 2024-03-23T11:28:20.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.330+0000 7f833fe00640 1 -- 172.21.15.62:0/1964733384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83400c3ab0 con 0x7f834015c730 2024-03-23T11:28:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.333+0000 7f8337400640 1 -- 172.21.15.62:0/1964733384 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8338040ce0 con 0x7f834015c730 2024-03-23T11:28:20.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.334+0000 7f8337400640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f831c06a7a0 0x7f831c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.334+0000 7f8337400640 1 -- 172.21.15.62:0/1964733384 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f83380abcb0 con 0x7f834015c730 2024-03-23T11:28:20.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.334+0000 7f8348b86740 1 Processor -- start 2024-03-23T11:28:20.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.335+0000 7f8348b86740 1 -- start start 2024-03-23T11:28:20.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.335+0000 7f833ea00640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f831c06a7a0 0x7f831c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.335+0000 7f8348b86740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f3eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.335+0000 7f8348b86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f3d5a0 0x55ea19f3f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.335+0000 7f8348b86740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f39a40 0x55ea19f69830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.335+0000 7f8348b86740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea19eaa110 con 0x55ea19f87520 2024-03-23T11:28:20.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.335+0000 7f8348b86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea19ea78a0 con 0x55ea19f3d5a0 2024-03-23T11:28:20.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.335+0000 7f8348b86740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea19ea3190 con 0x55ea19f39a40 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.336+0000 7f833ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f3eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.336+0000 7f833ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f3eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58868/0 (socket says 172.21.15.62:58868) 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.336+0000 7f833ea00640 1 -- 172.21.15.62:0/3420392524 learned_addr learned my addr 172.21.15.62:0/3420392524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.336+0000 7f833f400640 1 --2- 172.21.15.62:0/3420392524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f39a40 0x55ea19f69830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.336+0000 7f8337e00640 1 --2- 172.21.15.62:0/3420392524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f3d5a0 0x55ea19f3f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.336+0000 7f833ea00640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f831c06a7a0 0x7f831c06cc60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8330005880 tx=0x7f8330001a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.336+0000 7f833ea00640 1 -- 172.21.15.62:0/3420392524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f39a40 msgr2=0x55ea19f69830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.336+0000 7f833ea00640 1 --2- 172.21.15.62:0/3420392524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f39a40 0x55ea19f69830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.336+0000 7f833ea00640 1 -- 172.21.15.62:0/3420392524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f3d5a0 msgr2=0x55ea19f3f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.336+0000 7f833ea00640 1 --2- 172.21.15.62:0/3420392524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f3d5a0 0x55ea19f3f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.336+0000 7f833ea00640 1 -- 172.21.15.62:0/3420392524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea19e6e230 con 0x55ea19f87520 2024-03-23T11:28:20.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.337+0000 7f833ea00640 1 --2- 172.21.15.62:0/3420392524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f3eb50 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f8330005a20 tx=0x7f8330019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e55d73c91f47d58 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.338+0000 7f832fe00640 1 -- 172.21.15.62:0/3420392524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330007550 con 0x55ea19f87520 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.338+0000 7f832fe00640 1 -- 172.21.15.62:0/3420392524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83300076b0 con 0x55ea19f87520 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.338+0000 7f832fe00640 1 -- 172.21.15.62:0/3420392524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833000e990 con 0x55ea19f87520 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.338+0000 7f8348b86740 1 -- 172.21.15.62:0/3420392524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 msgr2=0x55ea19f3eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.338+0000 7f8348b86740 1 --2- 172.21.15.62:0/3420392524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f3eb50 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f8330005a20 tx=0x7f8330019000 comp rx=0 tx=0).stop 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.338+0000 7f8348b86740 1 -- 172.21.15.62:0/3420392524 shutdown_connections 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.338+0000 7f8348b86740 1 --2- 172.21.15.62:0/3420392524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f39a40 0x55ea19f69830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.339+0000 7f8348b86740 1 --2- 172.21.15.62:0/3420392524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f3d5a0 0x55ea19f3f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.339+0000 7f8348b86740 1 --2- 172.21.15.62:0/3420392524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f3eb50 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.339+0000 7f8348b86740 1 -- 172.21.15.62:0/3420392524 >> 172.21.15.62:0/3420392524 conn(0x55ea19f3bcb0 msgr2=0x55ea19f84830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.339+0000 7f8348b86740 1 -- 172.21.15.62:0/3420392524 shutdown_connections 2024-03-23T11:28:20.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.339+0000 7f8348b86740 1 -- 172.21.15.62:0/3420392524 wait complete. 2024-03-23T11:28:20.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.340+0000 7f8348b86740 1 -- start start 2024-03-23T11:28:20.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.340+0000 7f8348b86740 1 Processor -- start 2024-03-23T11:28:20.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.341+0000 7f8348b86740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f3d5a0 0x55ea19f89a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.341+0000 7f8348b86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f87520 0x55ea19f64b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.341+0000 7f8348b86740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f65070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.341+0000 7f8348b86740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea19e448c0 con 0x55ea19f8a0a0 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.341+0000 7f8348b86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea19e44560 con 0x55ea19f87520 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.341+0000 7f8348b86740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea19c0af20 con 0x55ea19f3d5a0 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.341+0000 7f8348b86740 10 client.0 _ll_register_callbacks cb 0x55ea19e9c800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f833f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f65070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f833f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f65070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58878/0 (socket says 172.21.15.62:58878) 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f833f400640 1 -- 172.21.15.62:0/1233621336 learned_addr learned my addr 172.21.15.62:0/1233621336 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f833ea00640 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f3d5a0 0x55ea19f89a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f8337e00640 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f87520 0x55ea19f64b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f833f400640 1 -- 172.21.15.62:0/1233621336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f3d5a0 msgr2=0x55ea19f89a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f833f400640 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f3d5a0 0x55ea19f89a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f833f400640 1 -- 172.21.15.62:0/1233621336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f87520 msgr2=0x55ea19f64b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f833f400640 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f87520 0x55ea19f64b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f833f400640 1 -- 172.21.15.62:0/1233621336 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833000f040 con 0x55ea19f8a0a0 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f8337e00640 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f87520 0x55ea19f64b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f833f400640 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f65070 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f833802bef0 tx=0x7f833807b590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ed614f316b4bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f832cc00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.342+0000 7f832cc00640 1 -- 172.21.15.62:0/1233621336 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83380831c0 con 0x55ea19f8a0a0 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f832cc00640 1 -- 172.21.15.62:0/1233621336 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f833807b030 con 0x55ea19f8a0a0 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f8348b86740 1 -- 172.21.15.62:0/1233621336 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea19ecd280 con 0x55ea19f8a0a0 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f832cc00640 1 -- 172.21.15.62:0/1233621336 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f833807b1f0 con 0x55ea19f8a0a0 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f8348b86740 10 client.8001 fetch_fsmap learned FSMap version 70 2024-03-23T11:28:20.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f8348b86740 1 -- 172.21.15.62:0/1233621336 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55ea19ea1810 con 0x55ea19f8a0a0 2024-03-23T11:28:20.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f832cc00640 1 -- 172.21.15.62:0/1233621336 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8338087020 con 0x55ea19f8a0a0 2024-03-23T11:28:20.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f832cc00640 1 -- 172.21.15.62:0/1233621336 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8338050020 con 0x55ea19f8a0a0 2024-03-23T11:28:20.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f8348b86740 10 client.8001 fetch_fsmap finished waiting for FSMap version 70 2024-03-23T11:28:20.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f8348b86740 10 client.8001 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:28:20.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f8348b86740 20 client.8001 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:20.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f8348b86740 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x55ea19f4cc10 0x55ea19f4f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f8348b86740 4 client.8001 mds_command: new command op to 7926 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:28:20.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.343+0000 7f8348b86740 1 -- 172.21.15.62:0/1233621336 --> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea19e959d0 con 0x55ea19f4cc10 2024-03-23T11:28:20.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.344+0000 7f833ea00640 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x55ea19f4cc10 0x55ea19f4f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.344+0000 7f833ea00640 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x55ea19f4cc10 0x55ea19f4f0a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dad06231627f27bf server_cookie=29420a96b882c09f in_seq=0 out_seq=0 2024-03-23T11:28:20.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.344+0000 7f832cc00640 10 client.8001 ms_handle_connect on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.345+0000 7f832cc00640 1 -- 172.21.15.62:0/1233621336 <== mds.0 v2:172.21.15.62:6832/1594260761 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55ea19e959d0 con 0x55ea19f4cc10 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.345+0000 7f832cc00640 10 client.8001 handle_command_reply: tid=0 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.345+0000 7f8348b86740 1 client.8001 shutdown 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.345+0000 7f8348b86740 1 -- 172.21.15.62:0/1233621336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 msgr2=0x55ea19f65070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.345+0000 7f8348b86740 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f65070 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f833802bef0 tx=0x7f833807b590 comp rx=0 tx=0).stop 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.346+0000 7f8348b86740 1 -- 172.21.15.62:0/1233621336 shutdown_connections 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.346+0000 7f8348b86740 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x55ea19f4cc10 0x55ea19f4f0a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.346+0000 7f8348b86740 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f65070 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.346+0000 7f8348b86740 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f87520 0x55ea19f64b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.346+0000 7f8348b86740 1 --2- 172.21.15.62:0/1233621336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f3d5a0 0x55ea19f89a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.346+0000 7f832cc00640 0 client.8001 ms_handle_reset on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.346+0000 7f8348b86740 1 -- 172.21.15.62:0/1233621336 >> 172.21.15.62:0/1233621336 conn(0x55ea19f39a40 msgr2=0x55ea19f3fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:20.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.346+0000 7f8348b86740 1 -- 172.21.15.62:0/1233621336 shutdown_connections 2024-03-23T11:28:20.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.346+0000 7f8348b86740 1 -- 172.21.15.62:0/1233621336 wait complete. 2024-03-23T11:28:20.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.346+0000 7f8348b86740 20 client.8001 trim_cache size 0 max 16384 2024-03-23T11:28:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.363+0000 7f8348b86740 1 Processor -- start 2024-03-23T11:28:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.363+0000 7f8348b86740 1 -- start start 2024-03-23T11:28:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8348b86740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f3d5a0 0x55ea19f892c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8348b86740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f89800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8348b86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f8a0a0 0x55ea19f6d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8348b86740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea19f61710 con 0x55ea19f87520 2024-03-23T11:28:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8348b86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea19f398c0 con 0x55ea19f8a0a0 2024-03-23T11:28:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8348b86740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea19f40200 con 0x55ea19f3d5a0 2024-03-23T11:28:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8337e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f89800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8337e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f89800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58884/0 (socket says 172.21.15.62:58884) 2024-03-23T11:28:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8337e00640 1 -- 172.21.15.62:0/3425510581 learned_addr learned my addr 172.21.15.62:0/3425510581 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8337e00640 1 -- 172.21.15.62:0/3425510581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f3d5a0 msgr2=0x55ea19f892c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8337e00640 1 --2- 172.21.15.62:0/3425510581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f3d5a0 0x55ea19f892c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8337e00640 1 -- 172.21.15.62:0/3425510581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f8a0a0 msgr2=0x55ea19f6d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8337e00640 1 --2- 172.21.15.62:0/3425510581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f8a0a0 0x55ea19f6d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.364+0000 7f8337e00640 1 -- 172.21.15.62:0/3425510581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea19f61a80 con 0x55ea19f87520 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.365+0000 7f8337e00640 1 --2- 172.21.15.62:0/3425510581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f89800 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f832800b810 tx=0x7f832800bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e9936013f10a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.365+0000 7f8334c00640 1 -- 172.21.15.62:0/3425510581 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83280129e0 con 0x55ea19f87520 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.365+0000 7f8334c00640 1 -- 172.21.15.62:0/3425510581 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8328012b40 con 0x55ea19f87520 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.365+0000 7f8334c00640 1 -- 172.21.15.62:0/3425510581 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83280153f0 con 0x55ea19f87520 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.366+0000 7f8348b86740 1 -- 172.21.15.62:0/3425510581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 msgr2=0x55ea19f89800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.366+0000 7f8348b86740 1 --2- 172.21.15.62:0/3425510581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f89800 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f832800b810 tx=0x7f832800bcd0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.366+0000 7f8348b86740 1 -- 172.21.15.62:0/3425510581 shutdown_connections 2024-03-23T11:28:20.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.366+0000 7f8348b86740 1 --2- 172.21.15.62:0/3425510581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f87520 0x55ea19f89800 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.366+0000 7f8348b86740 1 --2- 172.21.15.62:0/3425510581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f8a0a0 0x55ea19f6d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.366+0000 7f8348b86740 1 --2- 172.21.15.62:0/3425510581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f3d5a0 0x55ea19f892c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.366+0000 7f8348b86740 1 -- 172.21.15.62:0/3425510581 >> 172.21.15.62:0/3425510581 conn(0x55ea19f39a40 msgr2=0x55ea19f65640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:20.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.366+0000 7f8348b86740 1 -- 172.21.15.62:0/3425510581 shutdown_connections 2024-03-23T11:28:20.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.366+0000 7f8348b86740 1 -- 172.21.15.62:0/3425510581 wait complete. 2024-03-23T11:28:20.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.367+0000 7f8348b86740 1 -- start start 2024-03-23T11:28:20.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.367+0000 7f8348b86740 1 Processor -- start 2024-03-23T11:28:20.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f8348b86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f3d5a0 0x55ea19fea1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f8348b86740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f87520 0x55ea19f3e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f8348b86740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f3edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f8348b86740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea19f848d0 con 0x55ea19f8a0a0 2024-03-23T11:28:20.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f8348b86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea19f3fb70 con 0x55ea19f3d5a0 2024-03-23T11:28:20.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f8348b86740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea19f84400 con 0x55ea19f87520 2024-03-23T11:28:20.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f8348b86740 10 client.0 _ll_register_callbacks cb 0x7f833807a730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:20.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f833f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f3edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f833f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f3edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58898/0 (socket says 172.21.15.62:58898) 2024-03-23T11:28:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f833f400640 1 -- 172.21.15.62:0/2039600977 learned_addr learned my addr 172.21.15.62:0/2039600977 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f833ea00640 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f3d5a0 0x55ea19fea1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.368+0000 7f833f400640 1 -- 172.21.15.62:0/2039600977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f87520 msgr2=0x55ea19f3e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.369+0000 7f8337e00640 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f87520 0x55ea19f3e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.369+0000 7f833f400640 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f87520 0x55ea19f3e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.369+0000 7f833f400640 1 -- 172.21.15.62:0/2039600977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f3d5a0 msgr2=0x55ea19fea1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.369+0000 7f833f400640 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f3d5a0 0x55ea19fea1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.369+0000 7f833f400640 1 -- 172.21.15.62:0/2039600977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832800e040 con 0x55ea19f8a0a0 2024-03-23T11:28:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.369+0000 7f833ea00640 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f3d5a0 0x55ea19fea1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.369+0000 7f833f400640 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f3edf0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f8338079120 tx=0x7f83380a9fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dcc5c5c5057c11c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.369+0000 7f832cc00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f832cc00640 1 -- 172.21.15.62:0/2039600977 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83380831c0 con 0x55ea19f8a0a0 2024-03-23T11:28:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f832cc00640 1 -- 172.21.15.62:0/2039600977 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f8338088080 con 0x55ea19f8a0a0 2024-03-23T11:28:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f8348b86740 1 -- 172.21.15.62:0/2039600977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f83300076b0 con 0x55ea19f8a0a0 2024-03-23T11:28:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f832cc00640 1 -- 172.21.15.62:0/2039600977 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f833807b240 con 0x55ea19f8a0a0 2024-03-23T11:28:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f8348b86740 10 client.8007 fetch_fsmap learned FSMap version 70 2024-03-23T11:28:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f8348b86740 1 -- 172.21.15.62:0/2039600977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55ea19e43ad0 con 0x55ea19f8a0a0 2024-03-23T11:28:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f832cc00640 1 -- 172.21.15.62:0/2039600977 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83300076b0 con 0x55ea19f8a0a0 2024-03-23T11:28:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f832cc00640 1 -- 172.21.15.62:0/2039600977 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8338050040 con 0x55ea19f8a0a0 2024-03-23T11:28:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f8348b86740 10 client.8007 fetch_fsmap finished waiting for FSMap version 70 2024-03-23T11:28:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f8348b86740 10 client.8007 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:28:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f8348b86740 20 client.8007 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f8348b86740 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x55ea19f4b4f0 0x55ea19f4b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f8348b86740 4 client.8007 mds_command: new command op to 7926 tid=0[{"prefix": "status"}] 2024-03-23T11:28:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.371+0000 7f8348b86740 1 -- 172.21.15.62:0/2039600977 --> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ea19b973c0 con 0x55ea19f4b4f0 2024-03-23T11:28:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.372+0000 7f833ea00640 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x55ea19f4b4f0 0x55ea19f4b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.372+0000 7f833ea00640 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x55ea19f4b4f0 0x55ea19f4b8d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46eba54195af98c7 server_cookie=9992007a1a753bfb in_seq=0 out_seq=0 2024-03-23T11:28:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.372+0000 7f832cc00640 10 client.8007 ms_handle_connect on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:20.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f832cc00640 1 -- 172.21.15.62:0/2039600977 <== mds.0 v2:172.21.15.62:6832/1594260761 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55ea19b973c0 con 0x55ea19f4b4f0 2024-03-23T11:28:20.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f832cc00640 10 client.8007 handle_command_reply: tid=0 2024-03-23T11:28:20.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 1 client.8007 shutdown 2024-03-23T11:28:20.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 1 -- 172.21.15.62:0/2039600977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 msgr2=0x55ea19f3edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f3edf0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f8338079120 tx=0x7f83380a9fd0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 1 -- 172.21.15.62:0/2039600977 shutdown_connections 2024-03-23T11:28:20.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x55ea19f4b4f0 0x55ea19f4b8d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea19f8a0a0 0x55ea19f3edf0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55ea19f87520 0x55ea19f3e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 1 --2- 172.21.15.62:0/2039600977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ea19f3d5a0 0x55ea19fea1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f832cc00640 0 client.8007 ms_handle_reset on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:20.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 1 -- 172.21.15.62:0/2039600977 >> 172.21.15.62:0/2039600977 conn(0x55ea19f39a40 msgr2=0x55ea19f65640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:20.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 1 -- 172.21.15.62:0/2039600977 shutdown_connections 2024-03-23T11:28:20.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 1 -- 172.21.15.62:0/2039600977 wait complete. 2024-03-23T11:28:20.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.373+0000 7f8348b86740 20 client.8007 trim_cache size 0 max 16384 2024-03-23T11:28:20.367 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:28:20.367 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:28:20.368 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:28:20.368 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 7926, 2024-03-23T11:28:20.368 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:28:20.368 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:28:20.368 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:28:20.368 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 5.0331436409999997, 2024-03-23T11:28:20.368 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 70, 2024-03-23T11:28:20.368 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 93, 2024-03-23T11:28:20.368 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 93, 2024-03-23T11:28:20.368 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 6.8213810019999999 2024-03-23T11:28:20.368 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:28:20.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 -- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f831c06a7a0 msgr2=0x7f831c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f831c06a7a0 0x7f831c06cc60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8330005880 tx=0x7f8330001a40 comp rx=0 tx=0).stop 2024-03-23T11:28:20.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 -- 172.21.15.62:0/1964733384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834015c730 msgr2=0x7f83401f1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834015c730 0x7f83401f1d80 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f8338009d90 tx=0x7f8338004a40 comp rx=0 tx=0).stop 2024-03-23T11:28:20.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 -- 172.21.15.62:0/1964733384 shutdown_connections 2024-03-23T11:28:20.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f831c06a7a0 0x7f831c06cc60 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834015c730 0x7f83401f1d80 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83400ad370 0x7f834015b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 --2- 172.21.15.62:0/1964733384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8340069c60 0x7f83400c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 -- 172.21.15.62:0/1964733384 >> 172.21.15.62:0/1964733384 conn(0x7f83400b0090 msgr2=0x7f834015f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 -- 172.21.15.62:0/1964733384 shutdown_connections 2024-03-23T11:28:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.374+0000 7f833fe00640 1 -- 172.21.15.62:0/1964733384 wait complete. 2024-03-23T11:28:20.383 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:28:20.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.604+0000 7fde5ca00640 1 Processor -- start 2024-03-23T11:28:20.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.605+0000 7fde5ca00640 1 -- start start 2024-03-23T11:28:20.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.606+0000 7fde5ca00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fde5815ca00 0x7fde580ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.606+0000 7fde5ca00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fde580d0d40 0x7fde580d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.606+0000 7fde5ca00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde580cfe40 con 0x7fde580b24e0 2024-03-23T11:28:20.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.606+0000 7fde5ca00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde580d1700 con 0x7fde580d0d40 2024-03-23T11:28:20.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.606+0000 7fde5ca00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde580cd680 con 0x7fde5815ca00 2024-03-23T11:28:20.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.606+0000 7fde4de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fde580d0d40 0x7fde580d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.606+0000 7fde57200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fde580b24e0 0x7fde580b2970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54554/0 (socket says 172.21.15.62:54554) 2024-03-23T11:28:20.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.606+0000 7fde57200640 1 -- 172.21.15.62:0/1020026557 learned_addr learned my addr 172.21.15.62:0/1020026557 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:20.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.606+0000 7fde56800640 1 --2- 172.21.15.62:0/1020026557 >> v2:172.21.15.120:3300/0 conn(0x7fde5815ca00 0x7fde580ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.607+0000 7fde4de00640 1 -- 172.21.15.62:0/1020026557 >> v2:172.21.15.120:3300/0 conn(0x7fde5815ca00 msgr2=0x7fde580ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:20.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.607+0000 7fde4de00640 1 --2- 172.21.15.62:0/1020026557 >> v2:172.21.15.120:3300/0 conn(0x7fde5815ca00 0x7fde580ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.607+0000 7fde4de00640 1 -- 172.21.15.62:0/1020026557 >> v1:172.21.15.62:6789/0 conn(0x7fde580b24e0 legacy=0x7fde580b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.607+0000 7fde4de00640 1 -- 172.21.15.62:0/1020026557 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde580cd960 con 0x7fde580d0d40 2024-03-23T11:28:20.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.607+0000 7fde55e00640 1 -- 172.21.15.62:0/1020026557 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323505748 0 0) 0x7fde580cfe40 con 0x7fde580b24e0 2024-03-23T11:28:20.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.608+0000 7fde4de00640 1 --2- 172.21.15.62:0/1020026557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde580d0d40 0x7fde580d11c0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fde580a05e0 tx=0x7fde4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f48a7e73902f24 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:20.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.608+0000 7fde55e00640 1 -- 172.21.15.62:0/1020026557 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40031830 con 0x7fde580d0d40 2024-03-23T11:28:20.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.608+0000 7fde55e00640 1 -- 172.21.15.62:0/1020026557 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde40031990 con 0x7fde580d0d40 2024-03-23T11:28:20.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.608+0000 7fde55e00640 1 -- 172.21.15.62:0/1020026557 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40031d20 con 0x7fde580d0d40 2024-03-23T11:28:20.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.609+0000 7fde5ca00640 1 -- 172.21.15.62:0/1020026557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde580d0d40 msgr2=0x7fde580d11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.609+0000 7fde5ca00640 1 --2- 172.21.15.62:0/1020026557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde580d0d40 0x7fde580d11c0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fde580a05e0 tx=0x7fde4002b8d0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.609+0000 7fde5ca00640 1 -- 172.21.15.62:0/1020026557 shutdown_connections 2024-03-23T11:28:20.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.609+0000 7fde5ca00640 1 --2- 172.21.15.62:0/1020026557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde580d0d40 0x7fde580d11c0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.609+0000 7fde5ca00640 1 --2- 172.21.15.62:0/1020026557 >> v2:172.21.15.120:3300/0 conn(0x7fde5815ca00 0x7fde580ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.609+0000 7fde5ca00640 1 -- 172.21.15.62:0/1020026557 >> 172.21.15.62:0/1020026557 conn(0x7fde5809ba10 msgr2=0x7fde58089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:20.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.609+0000 7fde5ca00640 1 -- 172.21.15.62:0/1020026557 shutdown_connections 2024-03-23T11:28:20.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.609+0000 7fde5ca00640 1 -- 172.21.15.62:0/1020026557 wait complete. 2024-03-23T11:28:20.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.611+0000 7fde5ca00640 1 Processor -- start 2024-03-23T11:28:20.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.611+0000 7fde5ca00640 1 -- start start 2024-03-23T11:28:20.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.612+0000 7fde5ca00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde580b24e0 0x7fde5815ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.612+0000 7fde5ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde580d0d40 0x7fde580b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.612+0000 7fde5ca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde5815ca00 0x7fde580cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.612+0000 7fde5ca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde581f3210 con 0x7fde580d0d40 2024-03-23T11:28:20.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.612+0000 7fde5ca00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5815b150 con 0x7fde580b24e0 2024-03-23T11:28:20.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.612+0000 7fde5ca00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde580b6500 con 0x7fde5815ca00 2024-03-23T11:28:20.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.612+0000 7fde4de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde580d0d40 0x7fde580b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.612+0000 7fde4de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde580d0d40 0x7fde580b5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58910/0 (socket says 172.21.15.62:58910) 2024-03-23T11:28:20.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.612+0000 7fde4de00640 1 -- 172.21.15.62:0/3063613567 learned_addr learned my addr 172.21.15.62:0/3063613567 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:20.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.613+0000 7fde56800640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde580b24e0 0x7fde5815ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.613+0000 7fde57200640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde5815ca00 0x7fde580cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.613+0000 7fde4de00640 1 -- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde5815ca00 msgr2=0x7fde580cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.613+0000 7fde4de00640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde5815ca00 0x7fde580cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.613+0000 7fde4de00640 1 -- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde580b24e0 msgr2=0x7fde5815ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.613+0000 7fde4de00640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde580b24e0 0x7fde5815ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.613+0000 7fde4de00640 1 -- 172.21.15.62:0/3063613567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde40009400 con 0x7fde580d0d40 2024-03-23T11:28:20.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.613+0000 7fde57200640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde5815ca00 0x7fde580cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:20.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.613+0000 7fde4de00640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde580d0d40 0x7fde580b5bb0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fde40009890 tx=0x7fde400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:20.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.614+0000 7fde4f400640 1 -- 172.21.15.62:0/3063613567 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40004230 con 0x7fde580d0d40 2024-03-23T11:28:20.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.614+0000 7fde4f400640 1 -- 172.21.15.62:0/3063613567 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde40004390 con 0x7fde580d0d40 2024-03-23T11:28:20.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.614+0000 7fde4f400640 1 -- 172.21.15.62:0/3063613567 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4003a880 con 0x7fde580d0d40 2024-03-23T11:28:20.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.614+0000 7fde5ca00640 1 -- 172.21.15.62:0/3063613567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde581f2450 con 0x7fde580d0d40 2024-03-23T11:28:20.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.614+0000 7fde5ca00640 1 -- 172.21.15.62:0/3063613567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde581f2a10 con 0x7fde580d0d40 2024-03-23T11:28:20.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.615+0000 7fde5ca00640 1 -- 172.21.15.62:0/3063613567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde580672c0 con 0x7fde580d0d40 2024-03-23T11:28:20.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.616+0000 7fde4f400640 1 -- 172.21.15.62:0/3063613567 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fde4003a9e0 con 0x7fde580d0d40 2024-03-23T11:28:20.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.617+0000 7fde4f400640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fde3406a7a0 0x7fde3406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:20.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.617+0000 7fde4f400640 1 -- 172.21.15.62:0/3063613567 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fde400ab560 con 0x7fde580d0d40 2024-03-23T11:28:20.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.617+0000 7fde56800640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fde3406a7a0 0x7fde3406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:20.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.619+0000 7fde56800640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fde3406a7a0 0x7fde3406cc60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fde4800b9f0 tx=0x7fde48008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:20.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.622+0000 7fde4f400640 1 -- 172.21.15.62:0/3063613567 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fde400769c0 con 0x7fde580d0d40 2024-03-23T11:28:20.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.860+0000 7fde5ca00640 1 -- 172.21.15.62:0/3063613567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fde580b8680 con 0x7fde580d0d40 2024-03-23T11:28:20.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.861+0000 7fde4f400640 1 -- 172.21.15.62:0/3063613567 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) v1 ==== 53+0+73 (secure 0 0 0) 0x7fde40079fe0 con 0x7fde580d0d40 2024-03-23T11:28:20.853 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:28:20.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.863+0000 7fde5ca00640 1 -- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fde3406a7a0 msgr2=0x7fde3406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.863+0000 7fde5ca00640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fde3406a7a0 0x7fde3406cc60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fde4800b9f0 tx=0x7fde48008040 comp rx=0 tx=0).stop 2024-03-23T11:28:20.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.863+0000 7fde5ca00640 1 -- 172.21.15.62:0/3063613567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde580d0d40 msgr2=0x7fde580b5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:20.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.863+0000 7fde5ca00640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde580d0d40 0x7fde580b5bb0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fde40009890 tx=0x7fde400046c0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.863+0000 7fde5ca00640 1 -- 172.21.15.62:0/3063613567 shutdown_connections 2024-03-23T11:28:20.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.863+0000 7fde5ca00640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fde3406a7a0 0x7fde3406cc60 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.863+0000 7fde5ca00640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde580d0d40 0x7fde580b5bb0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.863+0000 7fde5ca00640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde5815ca00 0x7fde580cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.863+0000 7fde5ca00640 1 --2- 172.21.15.62:0/3063613567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde580b24e0 0x7fde5815ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:20.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.863+0000 7fde5ca00640 1 -- 172.21.15.62:0/3063613567 >> 172.21.15.62:0/3063613567 conn(0x7fde5809ba10 msgr2=0x7fde580b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:20.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.864+0000 7fde5ca00640 1 -- 172.21.15.62:0/3063613567 shutdown_connections 2024-03-23T11:28:20.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:20.864+0000 7fde5ca00640 1 -- 172.21.15.62:0/3063613567 wait complete. 2024-03-23T11:28:20.876 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:28:20.876 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:28:20.876 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:28:20.876 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:28:20.876 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:28:20.876 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:28:20.876 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:28:20.876 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:28:20.876 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:28:20.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:20.877 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:28:20.934 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:28:20.934 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:28:20.934 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:28:20.934 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:28:20.934 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:28:20.934 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:28:20.935 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:28:20.935 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:28:20.935 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:28:20.935 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:28:20.935 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:28:20.935 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:28:20.935 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:28:20.935 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:28:20.935 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:28:20.935 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:28:20.935 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42056sec preferred_lft 42056sec 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:28:20.936 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:28:20.937 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:28:20.937 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:28:20.937 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:28:20.937 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:28:20.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:20.937 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:28:21.030 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:28:21.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:21.035 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:28:21.090 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:28:21.090 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:28:21.090 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:28:21.090 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:28:21.090 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:28:21.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:21.091 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:28:21.092 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:28:21.092 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:28:21.092 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:28:21.092 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:28:21.092 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:28:21.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:21.288 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:28:21.342 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:28:21.342 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:28:21.343 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:28:21.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:21.343 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:28:21.439 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:28:21.440 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:28:21.440 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:28:21.496 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:28:21.496 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:28:21.496 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:28:21.551 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:28:21.647 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:28:21.647 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:28:21.745 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:28:21.750 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:28:21.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:21.750 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:28:21.807 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:28:21.807 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:28:21.813 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:28:21.899 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:28:21.899 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:28:21.968 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:28:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:28:21.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:21.973 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:28:22.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:28:22.012+0000 7f023823a480 -1 init, newargv = 0x55621ea9b0a0 newargc=15 2024-03-23T11:28:22.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[69642]: starting ceph client 2024-03-23T11:28:22.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[69642]: starting fuse 2024-03-23T11:28:22.015 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:28:22.016 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:28:22.016 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:28:22.017 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:28:22.211 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.69642.asok 2024-03-23T11:28:22.391 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.69642.asok 2024-03-23T11:28:22.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:22.391 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.69642.asok status 2024-03-23T11:28:22.638 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:28:22.638 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:28:22.638 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:28:22.638 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:28:22.638 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:28:22.638 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:28:22.638 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "69642", 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 8022, 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 8022 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:28:22.639 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2280780157 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2280780157 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.8022 192.168.0.1:0/2280780157", 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/2280780157", 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:28:22.640 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 70, 2024-03-23T11:28:22.641 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 93, 2024-03-23T11:28:22.641 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:28:22.641 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:28:22.641 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:28:22.641 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:28:22.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:22.653 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:28:22.653 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:28:22.710 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:28:22.710 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:28:22.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:22.710 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:28:22.811 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-03-23T11:28:23.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.049+0000 7f4cea000640 1 Processor -- start 2024-03-23T11:28:23.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.049+0000 7f4cea000640 1 -- start start 2024-03-23T11:28:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.050+0000 7f4cea000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ce40cf000 0x7f4ce40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.050+0000 7f4cea000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ce40d0c50 0x7f4ce40d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.050+0000 7f4cea000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce40d0200 con 0x7f4ce415c820 2024-03-23T11:28:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.050+0000 7f4cea000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce40d07c0 con 0x7f4ce40d0c50 2024-03-23T11:28:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.050+0000 7f4cea000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce40cf980 con 0x7f4ce40cf000 2024-03-23T11:28:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.051+0000 7f4ce9600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ce40d0c50 0x7f4ce40d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.051+0000 7f4ce9600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ce40d0c50 0x7f4ce40d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58924/0 (socket says 172.21.15.62:58924) 2024-03-23T11:28:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.051+0000 7f4ce9600640 1 -- 172.21.15.62:0/737971126 learned_addr learned my addr 172.21.15.62:0/737971126 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.051+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/737971126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ce40cf000 0x7f4ce40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.051+0000 7f4ce9600640 1 -- 172.21.15.62:0/737971126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ce40cf000 msgr2=0x7f4ce40cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.051+0000 7f4ce9600640 1 --2- 172.21.15.62:0/737971126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ce40cf000 0x7f4ce40cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.051+0000 7f4ce9600640 1 -- 172.21.15.62:0/737971126 >> v1:172.21.15.120:6789/0 conn(0x7f4ce415c820 legacy=0x7f4ce40b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.051+0000 7f4ce9600640 1 -- 172.21.15.62:0/737971126 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce40d15f0 con 0x7f4ce40d0c50 2024-03-23T11:28:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.052+0000 7f4ce9600640 1 --2- 172.21.15.62:0/737971126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ce40d0c50 0x7f4ce40d10b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009870 tx=0x7f4cdc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d161c90910271edd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.053+0000 7f4ce3400640 1 -- 172.21.15.62:0/737971126 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc0317c0 con 0x7f4ce40d0c50 2024-03-23T11:28:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.053+0000 7f4ce3400640 1 -- 172.21.15.62:0/737971126 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cdc031920 con 0x7f4ce40d0c50 2024-03-23T11:28:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.053+0000 7f4ce3400640 1 -- 172.21.15.62:0/737971126 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc031cb0 con 0x7f4ce40d0c50 2024-03-23T11:28:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.053+0000 7f4cea000640 1 -- 172.21.15.62:0/737971126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ce40d0c50 msgr2=0x7f4ce40d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.053+0000 7f4cea000640 1 --2- 172.21.15.62:0/737971126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ce40d0c50 0x7f4ce40d10b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009870 tx=0x7f4cdc02b860 comp rx=0 tx=0).stop 2024-03-23T11:28:23.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.053+0000 7f4cea000640 1 -- 172.21.15.62:0/737971126 shutdown_connections 2024-03-23T11:28:23.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.053+0000 7f4cea000640 1 --2- 172.21.15.62:0/737971126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ce40d0c50 0x7f4ce40d10b0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.053+0000 7f4cea000640 1 --2- 172.21.15.62:0/737971126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ce40cf000 0x7f4ce40cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.053+0000 7f4cea000640 1 -- 172.21.15.62:0/737971126 >> 172.21.15.62:0/737971126 conn(0x7f4ce409b9d0 msgr2=0x7f4ce4089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.054+0000 7f4cea000640 1 -- 172.21.15.62:0/737971126 shutdown_connections 2024-03-23T11:28:23.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.054+0000 7f4cea000640 1 -- 172.21.15.62:0/737971126 wait complete. 2024-03-23T11:28:23.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.055+0000 7f4cea000640 1 Processor -- start 2024-03-23T11:28:23.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.055+0000 7f4cea000640 1 -- start start 2024-03-23T11:28:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.056+0000 7f4cea000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ce40cf000 0x7f4ce41f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.056+0000 7f4cea000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce40d0c50 0x7f4ce4157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.056+0000 7f4cea000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ce415c820 0x7f4ce40b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.056+0000 7f4cea000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce41f2f30 con 0x7f4ce415c820 2024-03-23T11:28:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.056+0000 7f4cea000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce41581f0 con 0x7f4ce40d0c50 2024-03-23T11:28:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.056+0000 7f4cea000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce40b6810 con 0x7f4ce40cf000 2024-03-23T11:28:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.057+0000 7f4ce9600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ce415c820 0x7f4ce40b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.057+0000 7f4ce9600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ce415c820 0x7f4ce40b7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58926/0 (socket says 172.21.15.62:58926) 2024-03-23T11:28:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.057+0000 7f4ce9600640 1 -- 172.21.15.62:0/2453766802 learned_addr learned my addr 172.21.15.62:0/2453766802 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.057+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce40d0c50 0x7f4ce4157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.057+0000 7f4ce8c00640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ce40cf000 0x7f4ce41f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.057+0000 7f4ce9600640 1 -- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ce40cf000 msgr2=0x7f4ce41f29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.057+0000 7f4ce9600640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ce40cf000 0x7f4ce41f29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.057+0000 7f4ce9600640 1 -- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce40d0c50 msgr2=0x7f4ce4157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.057+0000 7f4ce9600640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce40d0c50 0x7f4ce4157cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.057+0000 7f4ce9600640 1 -- 172.21.15.62:0/2453766802 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cdc0094d0 con 0x7f4ce415c820 2024-03-23T11:28:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.058+0000 7f4ce9600640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ce415c820 0x7f4ce40b7180 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009870 tx=0x7f4cdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.058+0000 7f4ce0c00640 1 -- 172.21.15.62:0/2453766802 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc032860 con 0x7f4ce415c820 2024-03-23T11:28:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.058+0000 7f4ce0c00640 1 -- 172.21.15.62:0/2453766802 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cdc0329e0 con 0x7f4ce415c820 2024-03-23T11:28:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.058+0000 7f4ce0c00640 1 -- 172.21.15.62:0/2453766802 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc032d90 con 0x7f4ce415c820 2024-03-23T11:28:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.058+0000 7f4cea000640 1 -- 172.21.15.62:0/2453766802 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce40b5770 con 0x7f4ce415c820 2024-03-23T11:28:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.058+0000 7f4cea000640 1 -- 172.21.15.62:0/2453766802 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce40b5d30 con 0x7f4ce415c820 2024-03-23T11:28:23.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.060+0000 7f4ce0c00640 1 -- 172.21.15.62:0/2453766802 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4cdc04e450 con 0x7f4ce415c820 2024-03-23T11:28:23.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.061+0000 7f4ce0c00640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4cc006a7a0 0x7f4cc006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.061+0000 7f4ce0c00640 1 -- 172.21.15.62:0/2453766802 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f4cdc0ab0c0 con 0x7f4ce415c820 2024-03-23T11:28:23.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.062+0000 7f4ce8c00640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4cc006a7a0 0x7f4cc006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.062+0000 7f4cee23a740 1 Processor -- start 2024-03-23T11:28:23.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.062+0000 7f4cee23a740 1 -- start start 2024-03-23T11:28:23.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.063+0000 7f4ce8c00640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4cc006a7a0 0x7f4cc006cc60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4007bf0 tx=0x7f4cd4007240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.063+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9fa4f0 0x56355d9b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.063+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9b0570 0x56355d9b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9acca0 0x56355d9dc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4cee23a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355d91cf40 con 0x56355d9acca0 2024-03-23T11:28:23.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4cee23a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355d91a6d0 con 0x56355d9b0570 2024-03-23T11:28:23.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4cee23a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56355d915fc0 con 0x56355d9fa4f0 2024-03-23T11:28:23.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4ce9600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9acca0 0x56355d9dc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4ce9600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9acca0 0x56355d9dc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58942/0 (socket says 172.21.15.62:58942) 2024-03-23T11:28:23.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4ce9600640 1 -- 172.21.15.62:0/968678217 learned_addr learned my addr 172.21.15.62:0/968678217 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:23.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/968678217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9b0570 0x56355d9b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4ce9600640 1 -- 172.21.15.62:0/968678217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9fa4f0 msgr2=0x56355d9b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4ce8c00640 1 --2- 172.21.15.62:0/968678217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9fa4f0 0x56355d9b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4ce9600640 1 --2- 172.21.15.62:0/968678217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9fa4f0 0x56355d9b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4ce9600640 1 -- 172.21.15.62:0/968678217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9b0570 msgr2=0x56355d9b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4ce9600640 1 --2- 172.21.15.62:0/968678217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9b0570 0x56355d9b2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4ce9600640 1 -- 172.21.15.62:0/968678217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56355d8dd230 con 0x56355d9acca0 2024-03-23T11:28:23.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.064+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/968678217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9b0570 0x56355d9b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.065+0000 7f4ce9600640 1 --2- 172.21.15.62:0/968678217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9acca0 0x56355d9dc800 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc081f80 tx=0x7f4cdc076480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b06b0b2bc5b0a1e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.065+0000 7f4cd9600640 1 -- 172.21.15.62:0/968678217 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc07d070 con 0x56355d9acca0 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.065+0000 7f4cd9600640 1 -- 172.21.15.62:0/968678217 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cdc07def0 con 0x56355d9acca0 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.065+0000 7f4cd9600640 1 -- 172.21.15.62:0/968678217 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc0824b0 con 0x56355d9acca0 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.066+0000 7f4cee23a740 1 -- 172.21.15.62:0/968678217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9acca0 msgr2=0x56355d9dc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.066+0000 7f4cee23a740 1 --2- 172.21.15.62:0/968678217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9acca0 0x56355d9dc800 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc081f80 tx=0x7f4cdc076480 comp rx=0 tx=0).stop 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.066+0000 7f4cee23a740 1 -- 172.21.15.62:0/968678217 shutdown_connections 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.066+0000 7f4cee23a740 1 --2- 172.21.15.62:0/968678217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9acca0 0x56355d9dc800 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.066+0000 7f4cee23a740 1 --2- 172.21.15.62:0/968678217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9b0570 0x56355d9b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.066+0000 7f4cee23a740 1 --2- 172.21.15.62:0/968678217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9fa4f0 0x56355d9b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.066+0000 7f4cee23a740 1 -- 172.21.15.62:0/968678217 >> 172.21.15.62:0/968678217 conn(0x56355d9aef10 msgr2=0x56355d9f7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.066+0000 7f4cee23a740 1 -- 172.21.15.62:0/968678217 shutdown_connections 2024-03-23T11:28:23.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.066+0000 7f4cee23a740 1 -- 172.21.15.62:0/968678217 wait complete. 2024-03-23T11:28:23.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.067+0000 7f4cee23a740 1 -- start start 2024-03-23T11:28:23.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.067+0000 7f4cee23a740 1 Processor -- start 2024-03-23T11:28:23.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9d4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 0x56355d9c1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9c1dd0 0x56355d9f8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4cee23a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355d8b7320 con 0x56355d9b0570 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4cee23a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355d8b6fc0 con 0x56355d9fa4f0 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4cee23a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56355d67e0c0 con 0x56355d9c1dd0 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4cee23a740 10 client.0 _ll_register_callbacks cb 0x56355d8dca10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4ce8c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9d4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4ce8c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9d4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58958/0 (socket says 172.21.15.62:58958) 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4ce8c00640 1 -- 172.21.15.62:0/3610179437 learned_addr learned my addr 172.21.15.62:0/3610179437 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 0x56355d9c1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4ce8c00640 1 -- 172.21.15.62:0/3610179437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9c1dd0 msgr2=0x56355d9f8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4ce8c00640 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9c1dd0 0x56355d9f8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.068+0000 7f4ce8c00640 1 -- 172.21.15.62:0/3610179437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 msgr2=0x56355d9c1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.069+0000 7f4ce8c00640 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 0x56355d9c1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.069+0000 7f4ce8c00640 1 -- 172.21.15.62:0/3610179437 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce40b5d30 con 0x56355d9b0570 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.069+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 0x56355d9c1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.069+0000 7f4ce8c00640 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9d4670 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4013440 tx=0x7f4cd4004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb31b5ba8c65aa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.069+0000 7f4cd2000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.069+0000 7f4cd2000640 1 -- 172.21.15.62:0/3610179437 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd401e410 con 0x56355d9b0570 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.069+0000 7f4cd2000640 1 -- 172.21.15.62:0/3610179437 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f4cd401e570 con 0x56355d9b0570 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.069+0000 7f4cee23a740 1 -- 172.21.15.62:0/3610179437 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56355d93fbb0 con 0x56355d9b0570 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.069+0000 7f4cd2000640 1 -- 172.21.15.62:0/3610179437 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4cd401e730 con 0x56355d9b0570 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.069+0000 7f4cee23a740 10 client.8034 fetch_fsmap learned FSMap version 70 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.069+0000 7f4cee23a740 1 -- 172.21.15.62:0/3610179437 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x56355d914640 con 0x56355d9b0570 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.070+0000 7f4cd2000640 1 -- 172.21.15.62:0/3610179437 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56355d93fbb0 con 0x56355d9b0570 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.070+0000 7f4cd2000640 1 -- 172.21.15.62:0/3610179437 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4cd401e890 con 0x56355d9b0570 2024-03-23T11:28:23.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.070+0000 7f4cee23a740 10 client.8034 fetch_fsmap finished waiting for FSMap version 70 2024-03-23T11:28:23.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.070+0000 7f4cee23a740 10 client.8034 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:28:23.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.070+0000 7f4cee23a740 20 client.8034 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:23.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.070+0000 7f4cee23a740 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x56355d9fc3a0 0x56355d9fc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.070+0000 7f4cee23a740 4 client.8034 mds_command: new command op to 7926 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:28:23.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.070+0000 7f4cee23a740 1 -- 172.21.15.62:0/3610179437 --> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56355d60a560 con 0x56355d9fc3a0 2024-03-23T11:28:23.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.070+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x56355d9fc3a0 0x56355d9fc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.071+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x56355d9fc3a0 0x56355d9fc780 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4330ad9554e6b796 server_cookie=dd2af002673e1b93 in_seq=0 out_seq=0 2024-03-23T11:28:23.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.071+0000 7f4cd2000640 10 client.8034 ms_handle_connect on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:23.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.071+0000 7f4cd2000640 1 -- 172.21.15.62:0/3610179437 <== mds.0 v2:172.21.15.62:6832/1594260761 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56355d60a560 con 0x56355d9fc3a0 2024-03-23T11:28:23.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.071+0000 7f4cd2000640 10 client.8034 handle_command_reply: tid=0 2024-03-23T11:28:23.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 1 client.8034 shutdown 2024-03-23T11:28:23.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 1 -- 172.21.15.62:0/3610179437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 msgr2=0x56355d9d4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9d4670 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4013440 tx=0x7f4cd4004a50 comp rx=0 tx=0).stop 2024-03-23T11:28:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 1 -- 172.21.15.62:0/3610179437 shutdown_connections 2024-03-23T11:28:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x56355d9fc3a0 0x56355d9fc780 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9c1dd0 0x56355d9f8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 0x56355d9c1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 1 --2- 172.21.15.62:0/3610179437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9d4670 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cd2000640 0 client.8034 ms_handle_reset on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 1 -- 172.21.15.62:0/3610179437 >> 172.21.15.62:0/3610179437 conn(0x56355d9acca0 msgr2=0x56355d9b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 1 -- 172.21.15.62:0/3610179437 shutdown_connections 2024-03-23T11:28:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 1 -- 172.21.15.62:0/3610179437 wait complete. 2024-03-23T11:28:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.072+0000 7f4cee23a740 20 client.8034 trim_cache size 0 max 16384 2024-03-23T11:28:23.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.088+0000 7f4cee23a740 1 Processor -- start 2024-03-23T11:28:23.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.088+0000 7f4cee23a740 1 -- start start 2024-03-23T11:28:23.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.089+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9b0570 0x56355d9b1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.089+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9c1dd0 0x56355d9b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.089+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9fa4f0 0x56355d9abaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.089+0000 7f4cee23a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355d915160 con 0x56355d9c1dd0 2024-03-23T11:28:23.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.089+0000 7f4cee23a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355d9d3e50 con 0x56355d9b0570 2024-03-23T11:28:23.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.089+0000 7f4cee23a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56355d9f7bd0 con 0x56355d9fa4f0 2024-03-23T11:28:23.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.089+0000 7f4ce3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9c1dd0 0x56355d9b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.090+0000 7f4ce3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9c1dd0 0x56355d9b1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58964/0 (socket says 172.21.15.62:58964) 2024-03-23T11:28:23.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.090+0000 7f4ce3e00640 1 -- 172.21.15.62:0/2453793688 learned_addr learned my addr 172.21.15.62:0/2453793688 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:23.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.090+0000 7f4ce8c00640 1 --2- 172.21.15.62:0/2453793688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9b0570 0x56355d9b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.090+0000 7f4ce3e00640 1 -- 172.21.15.62:0/2453793688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9fa4f0 msgr2=0x56355d9abaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.090+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/2453793688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9fa4f0 0x56355d9abaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.090+0000 7f4ce3e00640 1 -- 172.21.15.62:0/2453793688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9b0570 msgr2=0x56355d9b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.090+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/2453793688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9b0570 0x56355d9b1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.090+0000 7f4ce3e00640 1 -- 172.21.15.62:0/2453793688 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56355d948f00 con 0x56355d9c1dd0 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.090+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/2453793688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9c1dd0 0x56355d9b1bd0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc00ca10 tx=0x7f4ccc016800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961a730c6c870ca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.091+0000 7f4cda000640 1 -- 172.21.15.62:0/2453793688 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc016ea0 con 0x56355d9c1dd0 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.091+0000 7f4cda000640 1 -- 172.21.15.62:0/2453793688 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ccc00cd10 con 0x56355d9c1dd0 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.091+0000 7f4cda000640 1 -- 172.21.15.62:0/2453793688 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc004250 con 0x56355d9c1dd0 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.091+0000 7f4cee23a740 1 -- 172.21.15.62:0/2453793688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9c1dd0 msgr2=0x56355d9b1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.091+0000 7f4cee23a740 1 --2- 172.21.15.62:0/2453793688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9c1dd0 0x56355d9b1bd0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc00ca10 tx=0x7f4ccc016800 comp rx=0 tx=0).stop 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.091+0000 7f4cee23a740 1 -- 172.21.15.62:0/2453793688 shutdown_connections 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.091+0000 7f4cee23a740 1 --2- 172.21.15.62:0/2453793688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9c1dd0 0x56355d9b1bd0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.091+0000 7f4cee23a740 1 --2- 172.21.15.62:0/2453793688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9fa4f0 0x56355d9abaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.091+0000 7f4cee23a740 1 --2- 172.21.15.62:0/2453793688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9b0570 0x56355d9b1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.091+0000 7f4cee23a740 1 -- 172.21.15.62:0/2453793688 >> 172.21.15.62:0/2453793688 conn(0x56355d9acca0 msgr2=0x56355d9f9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.091+0000 7f4cee23a740 1 -- 172.21.15.62:0/2453793688 shutdown_connections 2024-03-23T11:28:23.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.092+0000 7f4cee23a740 1 -- 172.21.15.62:0/2453793688 wait complete. 2024-03-23T11:28:23.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.092+0000 7f4cee23a740 1 -- start start 2024-03-23T11:28:23.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.093+0000 7f4cee23a740 1 Processor -- start 2024-03-23T11:28:23.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9c01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9c1dd0 0x56355d9b1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4cee23a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 0x56355d9b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4cee23a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355d9ad6a0 con 0x56355d9b0570 2024-03-23T11:28:23.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4cee23a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355d9d2ed0 con 0x56355d9fa4f0 2024-03-23T11:28:23.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4cee23a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56355d9d3b80 con 0x56355d9c1dd0 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4cee23a740 10 client.0 _ll_register_callbacks cb 0x7f4cdc032ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4ce8c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9c01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4ce8c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9c01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58972/0 (socket says 172.21.15.62:58972) 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4ce8c00640 1 -- 172.21.15.62:0/579055692 learned_addr learned my addr 172.21.15.62:0/579055692 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9c1dd0 0x56355d9b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4ce9600640 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 0x56355d9b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4ce8c00640 1 -- 172.21.15.62:0/579055692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9c1dd0 msgr2=0x56355d9b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4ce8c00640 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9c1dd0 0x56355d9b1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.094+0000 7f4ce8c00640 1 -- 172.21.15.62:0/579055692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 msgr2=0x56355d9b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.095+0000 7f4ce8c00640 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 0x56355d9b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.095+0000 7f4ce8c00640 1 -- 172.21.15.62:0/579055692 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ccc00c660 con 0x56355d9b0570 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.095+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9c1dd0 0x56355d9b1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:23.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.095+0000 7f4ce9600640 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 0x56355d9b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.095+0000 7f4ce8c00640 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9c01d0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4003cd0 tx=0x7f4cd4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e737cd3a642a01b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.095+0000 7f4cd2000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.095+0000 7f4cd2000640 1 -- 172.21.15.62:0/579055692 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd401e820 con 0x56355d9b0570 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.095+0000 7f4cd2000640 1 -- 172.21.15.62:0/579055692 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f4cd401edf0 con 0x56355d9b0570 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.095+0000 7f4cee23a740 1 -- 172.21.15.62:0/579055692 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4cdc07def0 con 0x56355d9b0570 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4cd2000640 1 -- 172.21.15.62:0/579055692 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4cd400cbc0 con 0x56355d9b0570 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4cee23a740 10 client.8040 fetch_fsmap learned FSMap version 70 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4cee23a740 1 -- 172.21.15.62:0/579055692 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x56355d8b64c0 con 0x56355d9b0570 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4cd2000640 1 -- 172.21.15.62:0/579055692 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cdc07def0 con 0x56355d9b0570 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4cd2000640 1 -- 172.21.15.62:0/579055692 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4cd4011740 con 0x56355d9b0570 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4cee23a740 10 client.8040 fetch_fsmap finished waiting for FSMap version 70 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4cee23a740 10 client.8040 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:28:23.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4cee23a740 20 client.8040 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:23.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4cee23a740 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x56355da4ca50 0x56355da4d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4cee23a740 4 client.8040 mds_command: new command op to 7926 tid=0[{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-03-23T11:28:23.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4cee23a740 1 -- 172.21.15.62:0/579055692 --> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) v1 -- 0x56355d61b860 con 0x56355da4ca50 2024-03-23T11:28:23.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.096+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x56355da4ca50 0x56355da4d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.097+0000 7f4ce3e00640 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x56355da4ca50 0x56355da4d070 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7395835beb71cb66 server_cookie=df2492673780bdcc in_seq=0 out_seq=0 2024-03-23T11:28:23.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.097+0000 7f4cd2000640 10 client.8040 ms_handle_connect on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:23.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.097+0000 7f4cd2000640 1 -- 172.21.15.62:0/579055692 <== mds.0 v2:172.21.15.62:6832/1594260761 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56355d61b860 con 0x56355da4ca50 2024-03-23T11:28:23.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.097+0000 7f4cd2000640 10 client.8040 handle_command_reply: tid=0 2024-03-23T11:28:23.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.097+0000 7f4cee23a740 1 client.8040 shutdown 2024-03-23T11:28:23.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cee23a740 1 -- 172.21.15.62:0/579055692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 msgr2=0x56355d9c01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cee23a740 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9c01d0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4003cd0 tx=0x7f4cd4013000 comp rx=0 tx=0).stop 2024-03-23T11:28:23.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cee23a740 1 -- 172.21.15.62:0/579055692 shutdown_connections 2024-03-23T11:28:23.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cee23a740 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x56355da4ca50 0x56355da4d070 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cee23a740 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56355d9fa4f0 0x56355d9b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cee23a740 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56355d9c1dd0 0x56355d9b1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cee23a740 1 --2- 172.21.15.62:0/579055692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56355d9b0570 0x56355d9c01d0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cee23a740 1 -- 172.21.15.62:0/579055692 >> 172.21.15.62:0/579055692 conn(0x56355d9acca0 msgr2=0x56355d9d9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cd2000640 0 client.8040 ms_handle_reset on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:23.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cee23a740 1 -- 172.21.15.62:0/579055692 shutdown_connections 2024-03-23T11:28:23.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cee23a740 1 -- 172.21.15.62:0/579055692 wait complete. 2024-03-23T11:28:23.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.098+0000 7f4cee23a740 20 client.8040 trim_cache size 0 max 16384 2024-03-23T11:28:23.093 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:28:23.094 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-03-23T11:28:23.094 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "f7aa278f-0fce-4a29-b72b-19beffeff14e", 2024-03-23T11:28:23.094 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-03-23T11:28:23.094 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:28:23.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 -- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4cc006a7a0 msgr2=0x7f4cc006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4cc006a7a0 0x7f4cc006cc60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4007bf0 tx=0x7f4cd4007240 comp rx=0 tx=0).stop 2024-03-23T11:28:23.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 -- 172.21.15.62:0/2453766802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ce415c820 msgr2=0x7f4ce40b7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ce415c820 0x7f4ce40b7180 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009870 tx=0x7f4cdc0046c0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 -- 172.21.15.62:0/2453766802 shutdown_connections 2024-03-23T11:28:23.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4cc006a7a0 0x7f4cc006cc60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ce415c820 0x7f4ce40b7180 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce40d0c50 0x7f4ce4157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 --2- 172.21.15.62:0/2453766802 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4ce40cf000 0x7f4ce41f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 -- 172.21.15.62:0/2453766802 >> 172.21.15.62:0/2453766802 conn(0x7f4ce409b9d0 msgr2=0x7f4ce406c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 -- 172.21.15.62:0/2453766802 shutdown_connections 2024-03-23T11:28:23.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.099+0000 7f4cea000640 1 -- 172.21.15.62:0/2453766802 wait complete. 2024-03-23T11:28:23.096 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:28:23.100+0000 7fa2cbc00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() (iversion lock) | dirtyparent=1 authpin=1 scrubqueue=0 0x55d3eef9a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(13)0x10000000001:[<0x10000000000/file v6>,<0x1/dir v13>]//[]","memoryvalue":"(13)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-23T11:28:23.110 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-23T11:28:23.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.328+0000 7f598b400640 1 Processor -- start 2024-03-23T11:28:23.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.328+0000 7f598b400640 1 -- start start 2024-03-23T11:28:23.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.329+0000 7f598b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598415c9e0 0x7f59841607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.329+0000 7f598b400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598406bfc0 con 0x7f598415cdc0 2024-03-23T11:28:23.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.329+0000 7f598b400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59840b5260 con 0x7f59840cf650 2024-03-23T11:28:23.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.329+0000 7f598b400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59840d2630 con 0x7f598415c9e0 2024-03-23T11:28:23.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.329+0000 7f5983e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f598415cdc0 0x7f59840d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54566/0 (socket says 172.21.15.62:54566) 2024-03-23T11:28:23.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.329+0000 7f5983e00640 1 -- 172.21.15.62:0/1111053000 learned_addr learned my addr 172.21.15.62:0/1111053000 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:23.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5988a00640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598415c9e0 0x7f59841607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894632166 0 0) 0x7f598406bfc0 con 0x7f598415cdc0 2024-03-23T11:28:23.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968003520 con 0x7f598415cdc0 2024-03-23T11:28:23.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2169794531 0 0) 0x7f5968003520 con 0x7f598415cdc0 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598415c9e0 msgr2=0x7f59841607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598415c9e0 0x7f59841607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 >> v1:172.21.15.120:6789/0 conn(0x7f59840cf650 legacy=0x7f598415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59840ce140 con 0x7f598415cdc0 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f5970001670 con 0x7f598415cdc0 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 >> v1:172.21.15.62:6789/0 conn(0x7f598415cdc0 legacy=0x7f59840d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59680039a0 0x7f5968003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598415c9e0 0x7f59680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5968004d00 0x7f5968005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840d2630 con 0x7f5968004d00 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596800ba60 con 0x7f59680039a0 2024-03-23T11:28:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.330+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596800bc20 con 0x7f598415c9e0 2024-03-23T11:28:23.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.331+0000 7f5989400640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5968004d00 0x7f5968005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.331+0000 7f5988a00640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59680039a0 0x7f5968003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.331+0000 7f5983e00640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598415c9e0 0x7f59680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.331+0000 7f5989400640 1 -- 172.21.15.62:0/1111053000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598415c9e0 msgr2=0x7f59680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.331+0000 7f5989400640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f598415c9e0 0x7f59680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.331+0000 7f5989400640 1 -- 172.21.15.62:0/1111053000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59680039a0 msgr2=0x7f5968003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.331+0000 7f5989400640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59680039a0 0x7f5968003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.331+0000 7f5989400640 1 -- 172.21.15.62:0/1111053000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968003520 con 0x7f5968004d00 2024-03-23T11:28:23.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.332+0000 7f5989400640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5968004d00 0x7f5968005180 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f597c001e20 tx=0x7f59680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8982fae1c668c6c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.332+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c003b70 con 0x7f5968004d00 2024-03-23T11:28:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.332+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c003cd0 con 0x7f5968004d00 2024-03-23T11:28:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.332+0000 7f5983400640 1 -- 172.21.15.62:0/1111053000 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c005310 con 0x7f5968004d00 2024-03-23T11:28:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.333+0000 7f598b400640 1 -- 172.21.15.62:0/1111053000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5968004d00 msgr2=0x7f5968005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.333+0000 7f598b400640 1 --2- 172.21.15.62:0/1111053000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5968004d00 0x7f5968005180 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f597c001e20 tx=0x7f59680057f0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.333+0000 7f5989400640 1 -- 172.21.15.62:0/1111053000 reap_dead start 2024-03-23T11:28:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.333+0000 7f598b400640 1 -- 172.21.15.62:0/1111053000 shutdown_connections 2024-03-23T11:28:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.333+0000 7f598b400640 1 -- 172.21.15.62:0/1111053000 >> 172.21.15.62:0/1111053000 conn(0x7f5984071570 msgr2=0x7f5984071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.333+0000 7f598b400640 1 -- 172.21.15.62:0/1111053000 shutdown_connections 2024-03-23T11:28:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.333+0000 7f598b400640 1 -- 172.21.15.62:0/1111053000 wait complete. 2024-03-23T11:28:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.335+0000 7f598b400640 1 Processor -- start 2024-03-23T11:28:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.335+0000 7f598b400640 1 -- start start 2024-03-23T11:28:23.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.336+0000 7f598b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59841609a0 0x7f59840c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.336+0000 7f598b400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5984160d80 0x7f59840c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.336+0000 7f598b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59840c6960 0x7f598414e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.336+0000 7f598b400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840c6ee0 con 0x7f59840c6960 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.336+0000 7f598b400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840c71c0 con 0x7f5984160d80 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.336+0000 7f598b400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59840c74a0 con 0x7f59841609a0 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.336+0000 7f5989400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59840c6960 0x7f598414e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.336+0000 7f5989400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59840c6960 0x7f598414e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58982/0 (socket says 172.21.15.62:58982) 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.336+0000 7f5989400640 1 -- 172.21.15.62:0/2759844908 learned_addr learned my addr 172.21.15.62:0/2759844908 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5988a00640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59841609a0 0x7f59840c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5983e00640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5984160d80 0x7f59840c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5989400640 1 -- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59841609a0 msgr2=0x7f59840c7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5989400640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59841609a0 0x7f59840c7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5989400640 1 -- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5984160d80 msgr2=0x7f59840c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5989400640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5984160d80 0x7f59840c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5989400640 1 -- 172.21.15.62:0/2759844908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59840c7780 con 0x7f59840c6960 2024-03-23T11:28:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5983e00640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5984160d80 0x7f59840c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:23.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5989400640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59840c6960 0x7f598414e920 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f597c001f40 tx=0x7f597c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5980c00640 1 -- 172.21.15.62:0/2759844908 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c03e070 con 0x7f59840c6960 2024-03-23T11:28:23.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5980c00640 1 -- 172.21.15.62:0/2759844908 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c003e40 con 0x7f59840c6960 2024-03-23T11:28:23.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f5980c00640 1 -- 172.21.15.62:0/2759844908 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c006d90 con 0x7f59840c6960 2024-03-23T11:28:23.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.337+0000 7f598b400640 1 -- 172.21.15.62:0/2759844908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59840c7a60 con 0x7f59840c6960 2024-03-23T11:28:23.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.338+0000 7f598b400640 1 -- 172.21.15.62:0/2759844908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59840c84c0 con 0x7f59840c6960 2024-03-23T11:28:23.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.340+0000 7f5980c00640 1 -- 172.21.15.62:0/2759844908 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f597c007030 con 0x7f59840c6960 2024-03-23T11:28:23.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.341+0000 7f5980c00640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f596406a7d0 0x7f596406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.341+0000 7f5980c00640 1 -- 172.21.15.62:0/2759844908 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f597c0aa100 con 0x7f59840c6960 2024-03-23T11:28:23.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.341+0000 7f598dfa1740 1 Processor -- start 2024-03-23T11:28:23.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.341+0000 7f598dfa1740 1 -- start start 2024-03-23T11:28:23.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.342+0000 7f5988a00640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f596406a7d0 0x7f596406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.342+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3d8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.342+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d9280 0x555d8c3b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.342+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0cf0 0x555d8c3b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.342+0000 7f598dfa1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8c36ccb0 con 0x555d8c3d0910 2024-03-23T11:28:23.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.342+0000 7f598dfa1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8c31b500 con 0x555d8c3d9280 2024-03-23T11:28:23.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.342+0000 7f598dfa1740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d8c318c90 con 0x555d8c3d0cf0 2024-03-23T11:28:23.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.343+0000 7f5988a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3d8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.343+0000 7f5988a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3d8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58998/0 (socket says 172.21.15.62:58998) 2024-03-23T11:28:23.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.343+0000 7f5988a00640 1 -- 172.21.15.62:0/2142502764 learned_addr learned my addr 172.21.15.62:0/2142502764 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:23.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.343+0000 7f5989400640 1 --2- 172.21.15.62:0/2142502764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0cf0 0x555d8c3b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.343+0000 7f5983e00640 1 --2- 172.21.15.62:0/2142502764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d9280 0x555d8c3b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.343+0000 7f5988a00640 1 -- 172.21.15.62:0/2142502764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0cf0 msgr2=0x555d8c3b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.343+0000 7f5988a00640 1 --2- 172.21.15.62:0/2142502764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0cf0 0x555d8c3b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.343+0000 7f5988a00640 1 -- 172.21.15.62:0/2142502764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d9280 msgr2=0x555d8c3b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.343+0000 7f5988a00640 1 --2- 172.21.15.62:0/2142502764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d9280 0x555d8c3b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.343+0000 7f5988a00640 1 -- 172.21.15.62:0/2142502764 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d8c2c6360 con 0x555d8c3d0910 2024-03-23T11:28:23.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.343+0000 7f5988a00640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f596406a7d0 0x7f596406cc90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5974009a60 tx=0x7f5974009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.344+0000 7f5988a00640 1 --2- 172.21.15.62:0/2142502764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3d8b80 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f5974013c90 tx=0x7f597401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b59d019cd73895dd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.344+0000 7f5979600640 1 -- 172.21.15.62:0/2142502764 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974027070 con 0x555d8c3d0910 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.344+0000 7f5979600640 1 -- 172.21.15.62:0/2142502764 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5974013e40 con 0x555d8c3d0910 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.344+0000 7f5979600640 1 -- 172.21.15.62:0/2142502764 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974004240 con 0x555d8c3d0910 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.344+0000 7f598dfa1740 1 -- 172.21.15.62:0/2142502764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 msgr2=0x555d8c3d8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.344+0000 7f598dfa1740 1 --2- 172.21.15.62:0/2142502764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3d8b80 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f5974013c90 tx=0x7f597401e000 comp rx=0 tx=0).stop 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.345+0000 7f598dfa1740 1 -- 172.21.15.62:0/2142502764 shutdown_connections 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.345+0000 7f598dfa1740 1 --2- 172.21.15.62:0/2142502764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0cf0 0x555d8c3b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.345+0000 7f598dfa1740 1 --2- 172.21.15.62:0/2142502764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d9280 0x555d8c3b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.345+0000 7f598dfa1740 1 --2- 172.21.15.62:0/2142502764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3d8b80 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.345+0000 7f598dfa1740 1 -- 172.21.15.62:0/2142502764 >> 172.21.15.62:0/2142502764 conn(0x555d8c378b40 msgr2=0x555d8c3f6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.345+0000 7f598dfa1740 1 -- 172.21.15.62:0/2142502764 shutdown_connections 2024-03-23T11:28:23.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.345+0000 7f598dfa1740 1 -- 172.21.15.62:0/2142502764 wait complete. 2024-03-23T11:28:23.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.345+0000 7f598dfa1740 1 -- start start 2024-03-23T11:28:23.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.346+0000 7f598dfa1740 1 Processor -- start 2024-03-23T11:28:23.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0910 0x555d8c3c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 0x555d8c3dd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d9280 0x555d8c3f6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f598dfa1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8c314580 con 0x555d8c3d9280 2024-03-23T11:28:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f598dfa1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8c2b5a50 con 0x555d8c3d0cf0 2024-03-23T11:28:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f598dfa1740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d8c2b56f0 con 0x555d8c3d0910 2024-03-23T11:28:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f598dfa1740 10 client.0 _ll_register_callbacks cb 0x555d8c304ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f5989400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d9280 0x555d8c3f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f5989400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d9280 0x555d8c3f6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59004/0 (socket says 172.21.15.62:59004) 2024-03-23T11:28:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f5989400640 1 -- 172.21.15.62:0/1540808201 learned_addr learned my addr 172.21.15.62:0/1540808201 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f5988a00640 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0910 0x555d8c3c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.347+0000 7f5983e00640 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 0x555d8c3dd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f5989400640 1 -- 172.21.15.62:0/1540808201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0910 msgr2=0x555d8c3c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f5989400640 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0910 0x555d8c3c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f5989400640 1 -- 172.21.15.62:0/1540808201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 msgr2=0x555d8c3dd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f5989400640 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 0x555d8c3dd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f5989400640 1 -- 172.21.15.62:0/1540808201 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59740139c0 con 0x555d8c3d9280 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f5983e00640 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 0x555d8c3dd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f5989400640 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d9280 0x555d8c3f6270 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f597c02c040 tx=0x7f597c0789d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1512b8cc2bfa67e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f596e000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f596e000640 1 -- 172.21.15.62:0/1540808201 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c07cea0 con 0x555d8c3d9280 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f596e000640 1 -- 172.21.15.62:0/1540808201 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f597c081460 con 0x555d8c3d9280 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f598dfa1740 1 -- 172.21.15.62:0/1540808201 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d8c33efe0 con 0x555d8c3d9280 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f596e000640 1 -- 172.21.15.62:0/1540808201 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f597c07f0b0 con 0x555d8c3d9280 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f598dfa1740 10 client.8052 fetch_fsmap learned FSMap version 70 2024-03-23T11:28:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.348+0000 7f598dfa1740 1 -- 172.21.15.62:0/1540808201 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x555d8c312c00 con 0x555d8c3d9280 2024-03-23T11:28:23.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.349+0000 7f596e000640 1 -- 172.21.15.62:0/1540808201 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555d8c33efe0 con 0x555d8c3d9280 2024-03-23T11:28:23.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.349+0000 7f596e000640 1 -- 172.21.15.62:0/1540808201 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f597c02d040 con 0x555d8c3d9280 2024-03-23T11:28:23.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.349+0000 7f598dfa1740 10 client.8052 fetch_fsmap finished waiting for FSMap version 70 2024-03-23T11:28:23.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.349+0000 7f598dfa1740 10 client.8052 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:28:23.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.349+0000 7f598dfa1740 20 client.8052 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:23.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.349+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x555d8c4062b0 0x555d8c406690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.349+0000 7f598dfa1740 4 client.8052 mds_command: new command op to 7926 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:28:23.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.349+0000 7f598dfa1740 1 -- 172.21.15.62:0/1540808201 --> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d8c307240 con 0x555d8c4062b0 2024-03-23T11:28:23.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.349+0000 7f5988a00640 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x555d8c4062b0 0x555d8c406690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.350+0000 7f5988a00640 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x555d8c4062b0 0x555d8c406690 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=427fea67adee033a server_cookie=fc8c256037a70495 in_seq=0 out_seq=0 2024-03-23T11:28:23.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.350+0000 7f596e000640 10 client.8052 ms_handle_connect on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:23.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f596e000640 1 -- 172.21.15.62:0/1540808201 <== mds.0 v2:172.21.15.62:6832/1594260761 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x555d8c307240 con 0x555d8c4062b0 2024-03-23T11:28:23.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f596e000640 10 client.8052 handle_command_reply: tid=0 2024-03-23T11:28:23.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 1 client.8052 shutdown 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 1 -- 172.21.15.62:0/1540808201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d9280 msgr2=0x555d8c3f6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d9280 0x555d8c3f6270 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f597c02c040 tx=0x7f597c0789d0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 1 -- 172.21.15.62:0/1540808201 shutdown_connections 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x555d8c4062b0 0x555d8c406690 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d9280 0x555d8c3f6270 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 0x555d8c3dd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1540808201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0910 0x555d8c3c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 1 -- 172.21.15.62:0/1540808201 >> 172.21.15.62:0/1540808201 conn(0x555d8c378b40 msgr2=0x555d8c3f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f596e000640 0 client.8052 ms_handle_reset on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 1 -- 172.21.15.62:0/1540808201 shutdown_connections 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 1 -- 172.21.15.62:0/1540808201 wait complete. 2024-03-23T11:28:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.351+0000 7f598dfa1740 20 client.8052 trim_cache size 0 max 16384 2024-03-23T11:28:23.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.368+0000 7f598dfa1740 1 Processor -- start 2024-03-23T11:28:23.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.368+0000 7f598dfa1740 1 -- start start 2024-03-23T11:28:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.368+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3a7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.368+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 0x555d8c3f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.368+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d9280 0x555d8c3f3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f598dfa1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8c011f90 con 0x555d8c3d0910 2024-03-23T11:28:23.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f598dfa1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8c3a79c0 con 0x555d8c3d0cf0 2024-03-23T11:28:23.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f598dfa1740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d8c3b2090 con 0x555d8c3d9280 2024-03-23T11:28:23.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f5988a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3a7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f5988a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3a7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59006/0 (socket says 172.21.15.62:59006) 2024-03-23T11:28:23.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f5988a00640 1 -- 172.21.15.62:0/3963601924 learned_addr learned my addr 172.21.15.62:0/3963601924 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f5988a00640 1 -- 172.21.15.62:0/3963601924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d9280 msgr2=0x555d8c3f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f5988a00640 1 --2- 172.21.15.62:0/3963601924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d9280 0x555d8c3f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f5988a00640 1 -- 172.21.15.62:0/3963601924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 msgr2=0x555d8c3f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f5983e00640 1 --2- 172.21.15.62:0/3963601924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 0x555d8c3f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f5988a00640 1 --2- 172.21.15.62:0/3963601924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 0x555d8c3f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f5988a00640 1 -- 172.21.15.62:0/3963601924 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d8c2b4c60 con 0x555d8c3d0910 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.369+0000 7f5983e00640 1 --2- 172.21.15.62:0/3963601924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 0x555d8c3f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.370+0000 7f5988a00640 1 --2- 172.21.15.62:0/3963601924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3a7480 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f597401d000 tx=0x7f597401a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88acb9b1e4f61b31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.370+0000 7f597a000640 1 -- 172.21.15.62:0/3963601924 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974027040 con 0x555d8c3d0910 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.370+0000 7f597a000640 1 -- 172.21.15.62:0/3963601924 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59740323e0 con 0x555d8c3d0910 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.370+0000 7f597a000640 1 -- 172.21.15.62:0/3963601924 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974032820 con 0x555d8c3d0910 2024-03-23T11:28:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.371+0000 7f598dfa1740 1 -- 172.21.15.62:0/3963601924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 msgr2=0x555d8c3a7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.371+0000 7f598dfa1740 1 --2- 172.21.15.62:0/3963601924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3a7480 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f597401d000 tx=0x7f597401a040 comp rx=0 tx=0).stop 2024-03-23T11:28:23.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.371+0000 7f598dfa1740 1 -- 172.21.15.62:0/3963601924 shutdown_connections 2024-03-23T11:28:23.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.371+0000 7f598dfa1740 1 --2- 172.21.15.62:0/3963601924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d9280 0x555d8c3f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.371+0000 7f598dfa1740 1 --2- 172.21.15.62:0/3963601924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d0cf0 0x555d8c3f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.371+0000 7f598dfa1740 1 --2- 172.21.15.62:0/3963601924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0910 0x555d8c3a7480 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.371+0000 7f598dfa1740 1 -- 172.21.15.62:0/3963601924 >> 172.21.15.62:0/3963601924 conn(0x555d8c378b40 msgr2=0x555d8c3f90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.371+0000 7f598dfa1740 1 -- 172.21.15.62:0/3963601924 shutdown_connections 2024-03-23T11:28:23.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.371+0000 7f598dfa1740 1 -- 172.21.15.62:0/3963601924 wait complete. 2024-03-23T11:28:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.372+0000 7f598dfa1740 1 -- start start 2024-03-23T11:28:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.372+0000 7f598dfa1740 1 Processor -- start 2024-03-23T11:28:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0910 0x555d8c455850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0cf0 0x555d8c3c3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f598dfa1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d9280 0x555d8c3f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f598dfa1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8c455d90 con 0x555d8c3d0cf0 2024-03-23T11:28:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f598dfa1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8c3c43d0 con 0x555d8c3d9280 2024-03-23T11:28:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f598dfa1740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d8c3af9d0 con 0x555d8c3d0910 2024-03-23T11:28:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f598dfa1740 10 client.0 _ll_register_callbacks cb 0x7f597401d030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f5983e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0cf0 0x555d8c3c3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f5983e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0cf0 0x555d8c3c3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59018/0 (socket says 172.21.15.62:59018) 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f5983e00640 1 -- 172.21.15.62:0/1224189462 learned_addr learned my addr 172.21.15.62:0/1224189462 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f5988a00640 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0910 0x555d8c455850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.373+0000 7f5989400640 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d9280 0x555d8c3f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f5983e00640 1 -- 172.21.15.62:0/1224189462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0910 msgr2=0x555d8c455850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f5983e00640 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0910 0x555d8c455850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f5983e00640 1 -- 172.21.15.62:0/1224189462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d9280 msgr2=0x555d8c3f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f5983e00640 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d9280 0x555d8c3f33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f5983e00640 1 -- 172.21.15.62:0/1224189462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d8c2c6360 con 0x555d8c3d0cf0 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f5983e00640 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0cf0 0x555d8c3c3190 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f5970001050 tx=0x7f59700139a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fec10e1671671e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f596e000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f596e000640 1 -- 172.21.15.62:0/1224189462 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970015620 con 0x555d8c3d0cf0 2024-03-23T11:28:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f596e000640 1 -- 172.21.15.62:0/1224189462 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f5970015780 con 0x555d8c3d0cf0 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f598dfa1740 1 -- 172.21.15.62:0/1224189462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5974013e40 con 0x555d8c3d0cf0 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f596e000640 1 -- 172.21.15.62:0/1224189462 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5970015940 con 0x555d8c3d0cf0 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.374+0000 7f598dfa1740 10 client.8058 fetch_fsmap learned FSMap version 70 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.375+0000 7f598dfa1740 1 -- 172.21.15.62:0/1224189462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x555d8c2da170 con 0x555d8c3d0cf0 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.375+0000 7f596e000640 1 -- 172.21.15.62:0/1224189462 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5974013e40 con 0x555d8c3d0cf0 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.375+0000 7f596e000640 1 -- 172.21.15.62:0/1224189462 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5970015aa0 con 0x555d8c3d0cf0 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.375+0000 7f598dfa1740 10 client.8058 fetch_fsmap finished waiting for FSMap version 70 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.375+0000 7f598dfa1740 10 client.8058 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.375+0000 7f598dfa1740 20 client.8058 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.375+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x555d8c3f3930 0x555d8c45bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.375+0000 7f598dfa1740 4 client.8058 mds_command: new command op to 7926 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.375+0000 7f598dfa1740 1 -- 172.21.15.62:0/1224189462 --> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555d8c0084e0 con 0x555d8c3f3930 2024-03-23T11:28:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.375+0000 7f5988a00640 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x555d8c3f3930 0x555d8c45bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f5988a00640 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x555d8c3f3930 0x555d8c45bbd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb05655db271e8bc server_cookie=36a9bce828bc5014 in_seq=0 out_seq=0 2024-03-23T11:28:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f596e000640 10 client.8058 ms_handle_connect on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f596e000640 1 -- 172.21.15.62:0/1224189462 <== mds.0 v2:172.21.15.62:6832/1594260761 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+263 (crc 0 0 0) 0x555d8c0084e0 con 0x555d8c3f3930 2024-03-23T11:28:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f596e000640 10 client.8058 handle_command_reply: tid=0 2024-03-23T11:28:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f598dfa1740 1 client.8058 shutdown 2024-03-23T11:28:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f598dfa1740 1 -- 172.21.15.62:0/1224189462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0cf0 msgr2=0x555d8c3c3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0cf0 0x555d8c3c3190 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f5970001050 tx=0x7f59700139a0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f598dfa1740 1 -- 172.21.15.62:0/1224189462 shutdown_connections 2024-03-23T11:28:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x555d8c3f3930 0x555d8c45bbd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d8c3d0cf0 0x555d8c3c3190 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d8c3d9280 0x555d8c3f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f598dfa1740 1 --2- 172.21.15.62:0/1224189462 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555d8c3d0910 0x555d8c455850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f598dfa1740 1 -- 172.21.15.62:0/1224189462 >> 172.21.15.62:0/1224189462 conn(0x555d8c378b40 msgr2=0x555d8c3f90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f596e000640 0 client.8058 ms_handle_reset on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f598dfa1740 1 -- 172.21.15.62:0/1224189462 shutdown_connections 2024-03-23T11:28:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.376+0000 7f598dfa1740 1 -- 172.21.15.62:0/1224189462 wait complete. 2024-03-23T11:28:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.377+0000 7f598dfa1740 20 client.8058 trim_cache size 0 max 16384 2024-03-23T11:28:23.372 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:28:23.372 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T11:28:23.372 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-03-23T11:28:23.372 INFO:teuthology.orchestra.run.smithi062.stdout: "f7aa278f-0fce-4a29-b72b-19beffeff14e": { 2024-03-23T11:28:23.373 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/dir", 2024-03-23T11:28:23.373 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "f7aa278f-0fce-4a29-b72b-19beffeff14e", 2024-03-23T11:28:23.373 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,repair" 2024-03-23T11:28:23.373 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:28:23.373 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:28:23.373 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:28:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.377+0000 7f598b400640 1 -- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f596406a7d0 msgr2=0x7f596406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.377+0000 7f598b400640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f596406a7d0 0x7f596406cc90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5974009a60 tx=0x7f5974009190 comp rx=0 tx=0).stop 2024-03-23T11:28:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.377+0000 7f598b400640 1 -- 172.21.15.62:0/2759844908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59840c6960 msgr2=0x7f598414e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.377+0000 7f598b400640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59840c6960 0x7f598414e920 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f597c001f40 tx=0x7f597c0014a0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.377+0000 7f598b400640 1 -- 172.21.15.62:0/2759844908 shutdown_connections 2024-03-23T11:28:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.377+0000 7f598b400640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f596406a7d0 0x7f596406cc90 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.377+0000 7f598b400640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59840c6960 0x7f598414e920 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.377+0000 7f598b400640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5984160d80 0x7f59840c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.377+0000 7f598b400640 1 --2- 172.21.15.62:0/2759844908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59841609a0 0x7f59840c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.377+0000 7f598b400640 1 -- 172.21.15.62:0/2759844908 >> 172.21.15.62:0/2759844908 conn(0x7f5984071570 msgr2=0x7f59840b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.378+0000 7f598b400640 1 -- 172.21.15.62:0/2759844908 shutdown_connections 2024-03-23T11:28:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:23.378+0000 7f598b400640 1 -- 172.21.15.62:0/2759844908 wait complete. 2024-03-23T11:28:23.386 INFO:tasks.cephfs.filesystem:scrub status for tag:f7aa278f-0fce-4a29-b72b-19beffeff14e - {'path': '/dir', 'tag': 'f7aa278f-0fce-4a29-b72b-19beffeff14e', 'options': 'recursive,repair'} 2024-03-23T11:28:47.171 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:28:47.179 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:28:53.387 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-03-23T11:28:53.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.626+0000 7f669f200640 1 Processor -- start 2024-03-23T11:28:53.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.627+0000 7f669f200640 1 -- start start 2024-03-23T11:28:53.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.628+0000 7f669f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f66a015cba0 0x7f66a00d3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.628+0000 7f669f200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f66a00b4880 0x7f66a00cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.628+0000 7f669f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a00d3600 con 0x7f66a00ce8e0 2024-03-23T11:28:53.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.628+0000 7f669f200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a015dc10 con 0x7f66a00b4880 2024-03-23T11:28:53.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.628+0000 7f669f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a015d270 con 0x7f66a015cba0 2024-03-23T11:28:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.629+0000 7f669e800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f66a00b4880 0x7f66a00cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.629+0000 7f669e800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f66a00b4880 0x7f66a00cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44462/0 (socket says 172.21.15.62:44462) 2024-03-23T11:28:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.629+0000 7f669e800640 1 -- 172.21.15.62:0/830751295 learned_addr learned my addr 172.21.15.62:0/830751295 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.629+0000 7f669de00640 1 --2- 172.21.15.62:0/830751295 >> v2:172.21.15.120:3300/0 conn(0x7f66a015cba0 0x7f66a00d3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.629+0000 7f669e800640 1 -- 172.21.15.62:0/830751295 >> v2:172.21.15.120:3300/0 conn(0x7f66a015cba0 msgr2=0x7f66a00d3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.629+0000 7f669e800640 1 --2- 172.21.15.62:0/830751295 >> v2:172.21.15.120:3300/0 conn(0x7f66a015cba0 0x7f66a00d3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.629+0000 7f669e800640 1 -- 172.21.15.62:0/830751295 >> v1:172.21.15.120:6789/0 conn(0x7f66a00ce8e0 legacy=0x7f66a00b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.629+0000 7f669e800640 1 -- 172.21.15.62:0/830751295 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a015e030 con 0x7f66a00b4880 2024-03-23T11:28:53.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.631+0000 7f669e800640 1 --2- 172.21.15.62:0/830751295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a00b4880 0x7f66a00cd100 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f66a008aad0 tx=0x7f669802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf61ef22efb0da9e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:53.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.631+0000 7f669d400640 1 -- 172.21.15.62:0/830751295 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669802c9f0 con 0x7f66a00b4880 2024-03-23T11:28:53.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.631+0000 7f669d400640 1 -- 172.21.15.62:0/830751295 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669802cb50 con 0x7f66a00b4880 2024-03-23T11:28:53.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.631+0000 7f669d400640 1 -- 172.21.15.62:0/830751295 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66980343f0 con 0x7f66a00b4880 2024-03-23T11:28:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.632+0000 7f669f200640 1 -- 172.21.15.62:0/830751295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a00b4880 msgr2=0x7f66a00cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.632+0000 7f669f200640 1 --2- 172.21.15.62:0/830751295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a00b4880 0x7f66a00cd100 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f66a008aad0 tx=0x7f669802be50 comp rx=0 tx=0).stop 2024-03-23T11:28:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.632+0000 7f669f200640 1 -- 172.21.15.62:0/830751295 shutdown_connections 2024-03-23T11:28:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.632+0000 7f669f200640 1 --2- 172.21.15.62:0/830751295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a00b4880 0x7f66a00cd100 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.632+0000 7f669f200640 1 --2- 172.21.15.62:0/830751295 >> v2:172.21.15.120:3300/0 conn(0x7f66a015cba0 0x7f66a00d3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.632+0000 7f669f200640 1 -- 172.21.15.62:0/830751295 >> 172.21.15.62:0/830751295 conn(0x7f66a0089f60 msgr2=0x7f66a00d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.632+0000 7f669f200640 1 -- 172.21.15.62:0/830751295 shutdown_connections 2024-03-23T11:28:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.632+0000 7f669f200640 1 -- 172.21.15.62:0/830751295 wait complete. 2024-03-23T11:28:53.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.634+0000 7f669f200640 1 Processor -- start 2024-03-23T11:28:53.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.634+0000 7f669f200640 1 -- start start 2024-03-23T11:28:53.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.635+0000 7f669f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a00b4880 0x7f66a01f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.635+0000 7f669f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a00ce8e0 0x7f66a015af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.635+0000 7f669f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a015cba0 0x7f66a0156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.635+0000 7f669f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a015b490 con 0x7f66a00ce8e0 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.635+0000 7f669f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a015b6c0 con 0x7f66a00b4880 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.635+0000 7f669f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a00cabd0 con 0x7f66a015cba0 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.635+0000 7f6697e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a00ce8e0 0x7f66a015af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.636+0000 7f6697e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a00ce8e0 0x7f66a015af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44470/0 (socket says 172.21.15.62:44470) 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.636+0000 7f6697e00640 1 -- 172.21.15.62:0/4198310576 learned_addr learned my addr 172.21.15.62:0/4198310576 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.636+0000 7f669de00640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a00b4880 0x7f66a01f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.636+0000 7f669e800640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a015cba0 0x7f66a0156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.636+0000 7f6697e00640 1 -- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a015cba0 msgr2=0x7f66a0156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.636+0000 7f6697e00640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a015cba0 0x7f66a0156f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.636+0000 7f6697e00640 1 -- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a00b4880 msgr2=0x7f66a01f22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.636+0000 7f6697e00640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a00b4880 0x7f66a01f22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.636+0000 7f6697e00640 1 -- 172.21.15.62:0/4198310576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6698009b30 con 0x7f66a00ce8e0 2024-03-23T11:28:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.636+0000 7f6697e00640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a00ce8e0 0x7f66a015af50 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f668800b3c0 tx=0x7f668800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.637+0000 7f6696000640 1 -- 172.21.15.62:0/4198310576 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6688011830 con 0x7f66a00ce8e0 2024-03-23T11:28:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.637+0000 7f6696000640 1 -- 172.21.15.62:0/4198310576 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6688011990 con 0x7f66a00ce8e0 2024-03-23T11:28:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.637+0000 7f6696000640 1 -- 172.21.15.62:0/4198310576 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6688011d80 con 0x7f66a00ce8e0 2024-03-23T11:28:53.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.637+0000 7f669f200640 1 -- 172.21.15.62:0/4198310576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a00caeb0 con 0x7f66a00ce8e0 2024-03-23T11:28:53.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.637+0000 7f669f200640 1 -- 172.21.15.62:0/4198310576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a00cb470 con 0x7f66a00ce8e0 2024-03-23T11:28:53.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.639+0000 7f6696000640 1 -- 172.21.15.62:0/4198310576 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f668801b460 con 0x7f66a00ce8e0 2024-03-23T11:28:53.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.640+0000 7f6696000640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f667c06a7a0 0x7f667c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.640+0000 7f6696000640 1 -- 172.21.15.62:0/4198310576 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f6688088cf0 con 0x7f66a00ce8e0 2024-03-23T11:28:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.641+0000 7f669de00640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f667c06a7a0 0x7f667c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.641+0000 7f66a7860740 1 Processor -- start 2024-03-23T11:28:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.641+0000 7f66a7860740 1 -- start start 2024-03-23T11:28:53.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f669de00640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f667c06a7a0 0x7f667c06cc60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6690006fd0 tx=0x7f6690008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:53.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f66a7860740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706af0 0x560d0070ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f66a7860740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d006e6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f66a7860740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706ed0 0x560d006e7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f66a7860740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d006a2cb0 con 0x560d0070f460 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f66a7860740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d00651500 con 0x560d00706ed0 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f66a7860740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560d0064ec90 con 0x560d00706af0 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f6697e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d006e6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f6697e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d006e6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44486/0 (socket says 172.21.15.62:44486) 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f6697e00640 1 -- 172.21.15.62:0/2200375411 learned_addr learned my addr 172.21.15.62:0/2200375411 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f669e800640 1 --2- 172.21.15.62:0/2200375411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706ed0 0x560d006e7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f6697e00640 1 -- 172.21.15.62:0/2200375411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706af0 msgr2=0x560d0070ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f669de00640 1 --2- 172.21.15.62:0/2200375411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706af0 0x560d0070ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f6697e00640 1 --2- 172.21.15.62:0/2200375411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706af0 0x560d0070ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f6697e00640 1 -- 172.21.15.62:0/2200375411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706ed0 msgr2=0x560d006e7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.642+0000 7f6697e00640 1 --2- 172.21.15.62:0/2200375411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706ed0 0x560d006e7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.643+0000 7f6697e00640 1 -- 172.21.15.62:0/2200375411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d005fc360 con 0x560d0070f460 2024-03-23T11:28:53.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.643+0000 7f669e800640 1 --2- 172.21.15.62:0/2200375411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706ed0 0x560d006e7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:53.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.643+0000 7f6697e00640 1 --2- 172.21.15.62:0/2200375411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d006e6c20 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f668805fb20 tx=0x7f6688058360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8054498c3a0235a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:53.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.643+0000 7f668ea00640 1 -- 172.21.15.62:0/2200375411 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668806f070 con 0x560d0070f460 2024-03-23T11:28:53.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.643+0000 7f668ea00640 1 -- 172.21.15.62:0/2200375411 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6688060050 con 0x560d0070f460 2024-03-23T11:28:53.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.643+0000 7f668ea00640 1 -- 172.21.15.62:0/2200375411 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6688054370 con 0x560d0070f460 2024-03-23T11:28:53.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.644+0000 7f66a7860740 1 -- 172.21.15.62:0/2200375411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 msgr2=0x560d006e6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.644+0000 7f66a7860740 1 --2- 172.21.15.62:0/2200375411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d006e6c20 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f668805fb20 tx=0x7f6688058360 comp rx=0 tx=0).stop 2024-03-23T11:28:53.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.644+0000 7f66a7860740 1 -- 172.21.15.62:0/2200375411 shutdown_connections 2024-03-23T11:28:53.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.644+0000 7f66a7860740 1 --2- 172.21.15.62:0/2200375411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d006e6c20 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.644+0000 7f66a7860740 1 --2- 172.21.15.62:0/2200375411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706ed0 0x560d006e7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.644+0000 7f66a7860740 1 --2- 172.21.15.62:0/2200375411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706af0 0x560d0070ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.644+0000 7f66a7860740 1 -- 172.21.15.62:0/2200375411 >> 172.21.15.62:0/2200375411 conn(0x560d006aeb40 msgr2=0x560d0072c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:53.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.644+0000 7f66a7860740 1 -- 172.21.15.62:0/2200375411 shutdown_connections 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.644+0000 7f66a7860740 1 -- 172.21.15.62:0/2200375411 wait complete. 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.645+0000 7f66a7860740 1 -- start start 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.645+0000 7f66a7860740 1 Processor -- start 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f66a7860740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706af0 0x560d0073baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f66a7860740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706ed0 0x560d00711050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f66a7860740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d0070f460 0x560d0072ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f66a7860740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d0064a580 con 0x560d00706af0 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f66a7860740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d005eba50 con 0x560d0070f460 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f66a7860740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560d005eb6f0 con 0x560d00706ed0 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f66a7860740 10 client.0 _ll_register_callbacks cb 0x560d0063aed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f669de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706af0 0x560d0073baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f669de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706af0 0x560d0073baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44488/0 (socket says 172.21.15.62:44488) 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f669de00640 1 -- 172.21.15.62:0/3962218417 learned_addr learned my addr 172.21.15.62:0/3962218417 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f669e800640 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d0070f460 0x560d0072ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f669de00640 1 -- 172.21.15.62:0/3962218417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706ed0 msgr2=0x560d00711050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.646+0000 7f669de00640 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706ed0 0x560d00711050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.647+0000 7f669de00640 1 -- 172.21.15.62:0/3962218417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d0070f460 msgr2=0x560d0072ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.647+0000 7f669de00640 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d0070f460 0x560d0072ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.647+0000 7f669de00640 1 -- 172.21.15.62:0/3962218417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a00cb470 con 0x560d00706af0 2024-03-23T11:28:53.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.647+0000 7f669e800640 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d0070f460 0x560d0072ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:53.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.647+0000 7f669de00640 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706af0 0x560d0073baf0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f669000b590 tx=0x7f6690004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1df8a18ae4b27d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:53.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.647+0000 7f6683400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:53.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.647+0000 7f6683400640 1 -- 172.21.15.62:0/3962218417 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6690015820 con 0x560d00706af0 2024-03-23T11:28:53.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.647+0000 7f6683400640 1 -- 172.21.15.62:0/3962218417 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f6690015980 con 0x560d00706af0 2024-03-23T11:28:53.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.647+0000 7f66a7860740 1 -- 172.21.15.62:0/3962218417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560d00674fe0 con 0x560d00706af0 2024-03-23T11:28:53.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f6683400640 1 -- 172.21.15.62:0/3962218417 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6690015b40 con 0x560d00706af0 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f66a7860740 10 client.8070 fetch_fsmap learned FSMap version 70 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f66a7860740 1 -- 172.21.15.62:0/3962218417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x560d00648c00 con 0x560d00706af0 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f6683400640 1 -- 172.21.15.62:0/3962218417 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560d00674fe0 con 0x560d00706af0 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f6683400640 1 -- 172.21.15.62:0/3962218417 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6690015ca0 con 0x560d00706af0 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f66a7860740 10 client.8070 fetch_fsmap finished waiting for FSMap version 70 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f66a7860740 10 client.8070 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f66a7860740 20 client.8070 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f66a7860740 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x560d00713e60 0x560d00714260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f66a7860740 4 client.8070 mds_command: new command op to 7926 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f66a7860740 1 -- 172.21.15.62:0/3962218417 --> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560d0063d240 con 0x560d00713e60 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.648+0000 7f6697e00640 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x560d00713e60 0x560d00714260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.649+0000 7f6697e00640 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x560d00713e60 0x560d00714260 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef6aa19445857a25 server_cookie=3222be7f45870615 in_seq=0 out_seq=0 2024-03-23T11:28:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.649+0000 7f6683400640 10 client.8070 ms_handle_connect on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f6683400640 1 -- 172.21.15.62:0/3962218417 <== mds.0 v2:172.21.15.62:6832/1594260761 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x560d0063d240 con 0x560d00713e60 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f6683400640 10 client.8070 handle_command_reply: tid=0 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 1 client.8070 shutdown 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 1 -- 172.21.15.62:0/3962218417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706af0 msgr2=0x560d0073baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706af0 0x560d0073baf0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f669000b590 tx=0x7f6690004730 comp rx=0 tx=0).stop 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 1 -- 172.21.15.62:0/3962218417 shutdown_connections 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x560d00713e60 0x560d00714260 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d0070f460 0x560d0072ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706ed0 0x560d00711050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 1 --2- 172.21.15.62:0/3962218417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706af0 0x560d0073baf0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 1 -- 172.21.15.62:0/3962218417 >> 172.21.15.62:0/3962218417 conn(0x560d006aeb40 msgr2=0x560d00706040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f6683400640 0 client.8070 ms_handle_reset on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 1 -- 172.21.15.62:0/3962218417 shutdown_connections 2024-03-23T11:28:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 1 -- 172.21.15.62:0/3962218417 wait complete. 2024-03-23T11:28:53.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.650+0000 7f66a7860740 20 client.8070 trim_cache size 0 max 16384 2024-03-23T11:28:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.666+0000 7f66a7860740 1 Processor -- start 2024-03-23T11:28:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.666+0000 7f66a7860740 1 -- start start 2024-03-23T11:28:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.667+0000 7f66a7860740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706af0 0x560d006e6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.667+0000 7f66a7860740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706ed0 0x560d0070ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.667+0000 7f66a7860740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d0070f460 0x560d0073aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.667+0000 7f66a7860740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d00347f90 con 0x560d00706ed0 2024-03-23T11:28:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.667+0000 7f66a7860740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d0072fa10 con 0x560d00706af0 2024-03-23T11:28:53.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.667+0000 7f66a7860740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560d00714880 con 0x560d0070f460 2024-03-23T11:28:53.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.667+0000 7f6697e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706ed0 0x560d0070ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.667+0000 7f6697e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706ed0 0x560d0070ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44492/0 (socket says 172.21.15.62:44492) 2024-03-23T11:28:53.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.667+0000 7f6697e00640 1 -- 172.21.15.62:0/2781296837 learned_addr learned my addr 172.21.15.62:0/2781296837 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:53.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.668+0000 7f669de00640 1 --2- 172.21.15.62:0/2781296837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706af0 0x560d006e6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.668+0000 7f6697e00640 1 -- 172.21.15.62:0/2781296837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d0070f460 msgr2=0x560d0073aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.668+0000 7f6697e00640 1 --2- 172.21.15.62:0/2781296837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d0070f460 0x560d0073aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.668+0000 7f6697e00640 1 -- 172.21.15.62:0/2781296837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706af0 msgr2=0x560d006e6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.668+0000 7f6697e00640 1 --2- 172.21.15.62:0/2781296837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706af0 0x560d006e6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.668+0000 7f6697e00640 1 -- 172.21.15.62:0/2781296837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d005eac60 con 0x560d00706ed0 2024-03-23T11:28:53.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.668+0000 7f6697e00640 1 --2- 172.21.15.62:0/2781296837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706ed0 0x560d0070ec30 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f6688057960 tx=0x7f668801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef389dc31f42e3d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.668+0000 7f668f400640 1 -- 172.21.15.62:0/2781296837 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668806f040 con 0x560d00706ed0 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.668+0000 7f668f400640 1 -- 172.21.15.62:0/2781296837 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668805ba90 con 0x560d00706ed0 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.668+0000 7f668f400640 1 -- 172.21.15.62:0/2781296837 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6688054370 con 0x560d00706ed0 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.669+0000 7f66a7860740 1 -- 172.21.15.62:0/2781296837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706ed0 msgr2=0x560d0070ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.669+0000 7f66a7860740 1 --2- 172.21.15.62:0/2781296837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706ed0 0x560d0070ec30 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f6688057960 tx=0x7f668801c000 comp rx=0 tx=0).stop 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.669+0000 7f66a7860740 1 -- 172.21.15.62:0/2781296837 shutdown_connections 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.669+0000 7f66a7860740 1 --2- 172.21.15.62:0/2781296837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d00706ed0 0x560d0070ec30 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.669+0000 7f66a7860740 1 --2- 172.21.15.62:0/2781296837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d0070f460 0x560d0073aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.669+0000 7f66a7860740 1 --2- 172.21.15.62:0/2781296837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706af0 0x560d006e6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.669+0000 7f66a7860740 1 -- 172.21.15.62:0/2781296837 >> 172.21.15.62:0/2781296837 conn(0x560d006aeb40 msgr2=0x560d006e7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.669+0000 7f66a7860740 1 -- 172.21.15.62:0/2781296837 shutdown_connections 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.669+0000 7f66a7860740 1 -- 172.21.15.62:0/2781296837 wait complete. 2024-03-23T11:28:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.670+0000 7f66a7860740 1 -- start start 2024-03-23T11:28:53.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.670+0000 7f66a7860740 1 Processor -- start 2024-03-23T11:28:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.671+0000 7f66a7860740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706af0 0x560d00791a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.671+0000 7f66a7860740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706ed0 0x560d0073aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.671+0000 7f66a7860740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d0073b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.671+0000 7f66a7860740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d006ffe20 con 0x560d0070f460 2024-03-23T11:28:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.671+0000 7f66a7860740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d00713650 con 0x560d00706af0 2024-03-23T11:28:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.671+0000 7f66a7860740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560d006e1cd0 con 0x560d00706ed0 2024-03-23T11:28:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.671+0000 7f66a7860740 10 client.0 _ll_register_callbacks cb 0x7f6688054020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f669e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d0073b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f669e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d0073b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44502/0 (socket says 172.21.15.62:44502) 2024-03-23T11:28:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f669e800640 1 -- 172.21.15.62:0/161256469 learned_addr learned my addr 172.21.15.62:0/161256469 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f669de00640 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706af0 0x560d00791a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f6697e00640 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706ed0 0x560d0073aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f669e800640 1 -- 172.21.15.62:0/161256469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706ed0 msgr2=0x560d0073aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f669e800640 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706ed0 0x560d0073aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f669e800640 1 -- 172.21.15.62:0/161256469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706af0 msgr2=0x560d00791a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f669e800640 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706af0 0x560d00791a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f669e800640 1 -- 172.21.15.62:0/161256469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560d005fc360 con 0x560d0070f460 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f669e800640 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d0073b0e0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f669802be90 tx=0x7f6698004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de62e880e594ace4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.672+0000 7f6683400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f6683400640 1 -- 172.21.15.62:0/161256469 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6698034c00 con 0x560d0070f460 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f6683400640 1 -- 172.21.15.62:0/161256469 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f6698034d60 con 0x560d0070f460 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f66a7860740 1 -- 172.21.15.62:0/161256469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6688060050 con 0x560d0070f460 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f6683400640 1 -- 172.21.15.62:0/161256469 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6698048410 con 0x560d0070f460 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f66a7860740 10 client.8076 fetch_fsmap learned FSMap version 70 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f66a7860740 1 -- 172.21.15.62:0/161256469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x560d00610170 con 0x560d0070f460 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f6683400640 1 -- 172.21.15.62:0/161256469 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6688060050 con 0x560d0070f460 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f6683400640 1 -- 172.21.15.62:0/161256469 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6698048570 con 0x560d0070f460 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f66a7860740 10 client.8076 fetch_fsmap finished waiting for FSMap version 70 2024-03-23T11:28:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f66a7860740 10 client.8076 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-03-23T11:28:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f66a7860740 20 client.8076 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f66a7860740 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x560d0073e9a0 0x560d00711c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f66a7860740 4 client.8076 mds_command: new command op to 7926 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:28:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.673+0000 7f66a7860740 1 -- 172.21.15.62:0/161256469 --> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560d0033e4e0 con 0x560d0073e9a0 2024-03-23T11:28:53.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.674+0000 7f669de00640 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x560d0073e9a0 0x560d00711c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.674+0000 7f669de00640 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x560d0073e9a0 0x560d00711c50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bf6a7e8ad3addd1 server_cookie=9b2f5128923799ee in_seq=0 out_seq=0 2024-03-23T11:28:53.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.674+0000 7f6683400640 10 client.8076 ms_handle_connect on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:53.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f6683400640 1 -- 172.21.15.62:0/161256469 <== mds.0 v2:172.21.15.62:6832/1594260761 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560d0033e4e0 con 0x560d0073e9a0 2024-03-23T11:28:53.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f6683400640 10 client.8076 handle_command_reply: tid=0 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 1 client.8076 shutdown 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 1 -- 172.21.15.62:0/161256469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 msgr2=0x560d0073b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d0073b0e0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f669802be90 tx=0x7f6698004a40 comp rx=0 tx=0).stop 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 1 -- 172.21.15.62:0/161256469 shutdown_connections 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x560d0073e9a0 0x560d00711c50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d0070f460 0x560d0073b0e0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560d00706ed0 0x560d0073aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 1 --2- 172.21.15.62:0/161256469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d00706af0 0x560d00791a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f6683400640 0 client.8076 ms_handle_reset on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 1 -- 172.21.15.62:0/161256469 >> 172.21.15.62:0/161256469 conn(0x560d006aeb40 msgr2=0x560d006e6890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 1 -- 172.21.15.62:0/161256469 shutdown_connections 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 1 -- 172.21.15.62:0/161256469 wait complete. 2024-03-23T11:28:53.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.675+0000 7f66a7860740 20 client.8076 trim_cache size 0 max 16384 2024-03-23T11:28:53.671 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:28:53.671 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-03-23T11:28:53.671 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-03-23T11:28:53.671 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:28:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 -- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f667c06a7a0 msgr2=0x7f667c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f667c06a7a0 0x7f667c06cc60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6690006fd0 tx=0x7f6690008040 comp rx=0 tx=0).stop 2024-03-23T11:28:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 -- 172.21.15.62:0/4198310576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a00ce8e0 msgr2=0x7f66a015af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a00ce8e0 0x7f66a015af50 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f668800b3c0 tx=0x7f668800b880 comp rx=0 tx=0).stop 2024-03-23T11:28:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 -- 172.21.15.62:0/4198310576 shutdown_connections 2024-03-23T11:28:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f667c06a7a0 0x7f667c06cc60 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a00ce8e0 0x7f66a015af50 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66a015cba0 0x7f66a0156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 --2- 172.21.15.62:0/4198310576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a00b4880 0x7f66a01f22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 -- 172.21.15.62:0/4198310576 >> 172.21.15.62:0/4198310576 conn(0x7f66a0089f60 msgr2=0x7f66a00b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 -- 172.21.15.62:0/4198310576 shutdown_connections 2024-03-23T11:28:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.676+0000 7f669f200640 1 -- 172.21.15.62:0/4198310576 wait complete. 2024-03-23T11:28:53.685 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T11:28:53.685 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:28:53.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.905+0000 7fd68fe00640 1 Processor -- start 2024-03-23T11:28:53.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.905+0000 7fd68fe00640 1 -- start start 2024-03-23T11:28:53.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.907+0000 7fd68fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd69015c830 0x7fd6900ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.907+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd690160310 0x7fd6900ab980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.907+0000 7fd68fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6900abec0 0x7fd6900ce250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.907+0000 7fd68fe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6900addb0 con 0x7fd69015c830 2024-03-23T11:28:53.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.907+0000 7fd68fe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69015cc10 con 0x7fd6900abec0 2024-03-23T11:28:53.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.907+0000 7fd68fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900ad850 con 0x7fd690160310 2024-03-23T11:28:53.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.907+0000 7fd68ea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd69015c830 0x7fd6900ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.907+0000 7fd68ea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd69015c830 0x7fd6900ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44510/0 (socket says 172.21.15.62:44510) 2024-03-23T11:28:53.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.907+0000 7fd68ea00640 1 -- 172.21.15.62:0/3842377490 learned_addr learned my addr 172.21.15.62:0/3842377490 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:53.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.908+0000 7fd686000640 1 --2- 172.21.15.62:0/3842377490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd690160310 0x7fd6900ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.908+0000 7fd68f400640 1 --2- 172.21.15.62:0/3842377490 >> v2:172.21.15.120:3300/0 conn(0x7fd6900abec0 0x7fd6900ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.908+0000 7fd68ea00640 1 -- 172.21.15.62:0/3842377490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd690160310 msgr2=0x7fd6900ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.908+0000 7fd68ea00640 1 --2- 172.21.15.62:0/3842377490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd690160310 0x7fd6900ab980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.908+0000 7fd68ea00640 1 -- 172.21.15.62:0/3842377490 >> v2:172.21.15.120:3300/0 conn(0x7fd6900abec0 msgr2=0x7fd6900ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:53.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.908+0000 7fd68ea00640 1 --2- 172.21.15.62:0/3842377490 >> v2:172.21.15.120:3300/0 conn(0x7fd6900abec0 0x7fd6900ce250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.908+0000 7fd68ea00640 1 -- 172.21.15.62:0/3842377490 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6901608d0 con 0x7fd69015c830 2024-03-23T11:28:53.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.909+0000 7fd68ea00640 1 --2- 172.21.15.62:0/3842377490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd69015c830 0x7fd6900ad310 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fd69006ac90 tx=0x7fd68002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144c7431fd8f82ed server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.909+0000 7fd68e000640 1 -- 172.21.15.62:0/3842377490 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680031640 con 0x7fd69015c830 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.909+0000 7fd68e000640 1 -- 172.21.15.62:0/3842377490 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6800317a0 con 0x7fd69015c830 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.909+0000 7fd68e000640 1 -- 172.21.15.62:0/3842377490 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680031b30 con 0x7fd69015c830 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.910+0000 7fd68fe00640 1 -- 172.21.15.62:0/3842377490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd69015c830 msgr2=0x7fd6900ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.910+0000 7fd68fe00640 1 --2- 172.21.15.62:0/3842377490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd69015c830 0x7fd6900ad310 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fd69006ac90 tx=0x7fd68002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.910+0000 7fd68fe00640 1 -- 172.21.15.62:0/3842377490 shutdown_connections 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.910+0000 7fd68fe00640 1 --2- 172.21.15.62:0/3842377490 >> v2:172.21.15.120:3300/0 conn(0x7fd6900abec0 0x7fd6900ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.910+0000 7fd68fe00640 1 --2- 172.21.15.62:0/3842377490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd690160310 0x7fd6900ab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.910+0000 7fd68fe00640 1 --2- 172.21.15.62:0/3842377490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd69015c830 0x7fd6900ad310 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.910+0000 7fd68fe00640 1 -- 172.21.15.62:0/3842377490 >> 172.21.15.62:0/3842377490 conn(0x7fd690066050 msgr2=0x7fd690069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.910+0000 7fd68fe00640 1 -- 172.21.15.62:0/3842377490 shutdown_connections 2024-03-23T11:28:53.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.910+0000 7fd68fe00640 1 -- 172.21.15.62:0/3842377490 wait complete. 2024-03-23T11:28:53.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.912+0000 7fd68fe00640 1 Processor -- start 2024-03-23T11:28:53.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.912+0000 7fd68fe00640 1 -- start start 2024-03-23T11:28:53.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.913+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6900abec0 0x7fd6900c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.913+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd69015c830 0x7fd6900cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.913+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd690160310 0x7fd690159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.913+0000 7fd68fe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900c5570 con 0x7fd690160310 2024-03-23T11:28:53.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.913+0000 7fd68fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900b19d0 con 0x7fd69015c830 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.913+0000 7fd68fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900b1cb0 con 0x7fd6900abec0 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd68f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd690160310 0x7fd690159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd68f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd690160310 0x7fd690159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44524/0 (socket says 172.21.15.62:44524) 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd68f400640 1 -- 172.21.15.62:0/2703959368 learned_addr learned my addr 172.21.15.62:0/2703959368 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd68ea00640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6900abec0 0x7fd6900c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd686000640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd69015c830 0x7fd6900cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd68f400640 1 -- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6900abec0 msgr2=0x7fd6900c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd68f400640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6900abec0 0x7fd6900c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd68f400640 1 -- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd69015c830 msgr2=0x7fd6900cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd68f400640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd69015c830 0x7fd6900cd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd68f400640 1 -- 172.21.15.62:0/2703959368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6800094d0 con 0x7fd690160310 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd68ea00640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6900abec0 0x7fd6900c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.914+0000 7fd68f400640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd690160310 0x7fd690159220 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fd688009fd0 tx=0x7fd68800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:53.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.915+0000 7fd687400640 1 -- 172.21.15.62:0/2703959368 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688014420 con 0x7fd690160310 2024-03-23T11:28:53.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.915+0000 7fd687400640 1 -- 172.21.15.62:0/2703959368 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd688014580 con 0x7fd690160310 2024-03-23T11:28:53.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.915+0000 7fd687400640 1 -- 172.21.15.62:0/2703959368 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688014930 con 0x7fd690160310 2024-03-23T11:28:53.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.915+0000 7fd68fe00640 1 -- 172.21.15.62:0/2703959368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6900b1f90 con 0x7fd690160310 2024-03-23T11:28:53.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.915+0000 7fd68fe00640 1 -- 172.21.15.62:0/2703959368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6900c5260 con 0x7fd690160310 2024-03-23T11:28:53.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.916+0000 7fd68fe00640 1 -- 172.21.15.62:0/2703959368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6900b24b0 con 0x7fd690160310 2024-03-23T11:28:53.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.917+0000 7fd687400640 1 -- 172.21.15.62:0/2703959368 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd688014a90 con 0x7fd690160310 2024-03-23T11:28:53.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.918+0000 7fd687400640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd66c06a7a0 0x7fd66c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:53.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.918+0000 7fd687400640 1 -- 172.21.15.62:0/2703959368 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fd688089830 con 0x7fd690160310 2024-03-23T11:28:53.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.919+0000 7fd68ea00640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd66c06a7a0 0x7fd66c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:53.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.920+0000 7fd68ea00640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd66c06a7a0 0x7fd66c06cc60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd680036000 tx=0x7fd68003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:53.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:53.923+0000 7fd687400640 1 -- 172.21.15.62:0/2703959368 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd6900b24b0 con 0x7fd690160310 2024-03-23T11:28:54.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.167+0000 7fd68fe00640 1 -- 172.21.15.62:0/2703959368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6900b24b0 con 0x7fd690160310 2024-03-23T11:28:54.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.168+0000 7fd687400640 1 -- 172.21.15.62:0/2703959368 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fd6900b24b0 con 0x7fd690160310 2024-03-23T11:28:54.160 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:54.160 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":70,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:28:18.362214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6833/1594260761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1594260761},{"type":"v1","addr":"172.21.15.62:6833","nonce":1594260761}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:28:54.161 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 70 2024-03-23T11:28:54.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.171+0000 7fd68fe00640 1 -- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd66c06a7a0 msgr2=0x7fd66c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.171+0000 7fd68fe00640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd66c06a7a0 0x7fd66c06cc60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd680036000 tx=0x7fd68003c040 comp rx=0 tx=0).stop 2024-03-23T11:28:54.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.171+0000 7fd68fe00640 1 -- 172.21.15.62:0/2703959368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd690160310 msgr2=0x7fd690159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.171+0000 7fd68fe00640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd690160310 0x7fd690159220 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fd688009fd0 tx=0x7fd68800e4b0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.171+0000 7fd68fe00640 1 -- 172.21.15.62:0/2703959368 shutdown_connections 2024-03-23T11:28:54.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.171+0000 7fd68fe00640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd66c06a7a0 0x7fd66c06cc60 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.171+0000 7fd68fe00640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd690160310 0x7fd690159220 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.172+0000 7fd68fe00640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd69015c830 0x7fd6900cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.172+0000 7fd68fe00640 1 --2- 172.21.15.62:0/2703959368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6900abec0 0x7fd6900c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.172+0000 7fd68fe00640 1 -- 172.21.15.62:0/2703959368 >> 172.21.15.62:0/2703959368 conn(0x7fd690066050 msgr2=0x7fd6900b0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:54.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.172+0000 7fd68fe00640 1 -- 172.21.15.62:0/2703959368 shutdown_connections 2024-03-23T11:28:54.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.172+0000 7fd68fe00640 1 -- 172.21.15.62:0/2703959368 wait complete. 2024-03-23T11:28:54.181 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-03-23T11:28:54.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.388+0000 7f1741a00640 1 Processor -- start 2024-03-23T11:28:54.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.389+0000 7f1741a00640 1 -- start start 2024-03-23T11:28:54.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.390+0000 7f1741a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f173c142f10 0x7f173c0caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.390+0000 7f1741a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f173c0cb530 0x7f173c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.390+0000 7f1741a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173c159d00 con 0x7f173c0b25a0 2024-03-23T11:28:54.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.391+0000 7f1741a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f173c0cbd40 con 0x7f173c0cb530 2024-03-23T11:28:54.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.391+0000 7f1741a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f173c159fe0 con 0x7f173c142f10 2024-03-23T11:28:54.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.391+0000 7f1741000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f173c0cb530 0x7f173c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.391+0000 7f1741000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f173c0cb530 0x7f173c06c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44540/0 (socket says 172.21.15.62:44540) 2024-03-23T11:28:54.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.391+0000 7f1741000640 1 -- 172.21.15.62:0/2198820164 learned_addr learned my addr 172.21.15.62:0/2198820164 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:54.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.391+0000 7f1733e00640 1 --2- 172.21.15.62:0/2198820164 >> v2:172.21.15.120:3300/0 conn(0x7f173c142f10 0x7f173c0caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.391+0000 7f1741000640 1 -- 172.21.15.62:0/2198820164 >> v2:172.21.15.120:3300/0 conn(0x7f173c142f10 msgr2=0x7f173c0caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:54.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.391+0000 7f1741000640 1 --2- 172.21.15.62:0/2198820164 >> v2:172.21.15.120:3300/0 conn(0x7f173c142f10 0x7f173c0caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.391+0000 7f1741000640 1 -- 172.21.15.62:0/2198820164 >> v1:172.21.15.62:6789/0 conn(0x7f173c0b25a0 legacy=0x7f173c0b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.391+0000 7f1741000640 1 -- 172.21.15.62:0/2198820164 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c15a2c0 con 0x7f173c0cb530 2024-03-23T11:28:54.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.393+0000 7f1741000640 1 --2- 172.21.15.62:0/2198820164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173c0cb530 0x7f173c06c7e0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f173c0b0df0 tx=0x7f173402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c61a9598e07f59 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:54.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.393+0000 7f173b400640 1 -- 172.21.15.62:0/2198820164 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173402cb20 con 0x7f173c0cb530 2024-03-23T11:28:54.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.393+0000 7f173b400640 1 -- 172.21.15.62:0/2198820164 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173402cc80 con 0x7f173c0cb530 2024-03-23T11:28:54.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.393+0000 7f173b400640 1 -- 172.21.15.62:0/2198820164 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17340343f0 con 0x7f173c0cb530 2024-03-23T11:28:54.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.394+0000 7f1741a00640 1 -- 172.21.15.62:0/2198820164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173c0cb530 msgr2=0x7f173c06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.394+0000 7f1741a00640 1 --2- 172.21.15.62:0/2198820164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173c0cb530 0x7f173c06c7e0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f173c0b0df0 tx=0x7f173402bf80 comp rx=0 tx=0).stop 2024-03-23T11:28:54.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.394+0000 7f1741a00640 1 -- 172.21.15.62:0/2198820164 shutdown_connections 2024-03-23T11:28:54.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.394+0000 7f1741a00640 1 --2- 172.21.15.62:0/2198820164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173c0cb530 0x7f173c06c7e0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.394+0000 7f1741a00640 1 --2- 172.21.15.62:0/2198820164 >> v2:172.21.15.120:3300/0 conn(0x7f173c142f10 0x7f173c0caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.394+0000 7f1741a00640 1 -- 172.21.15.62:0/2198820164 >> 172.21.15.62:0/2198820164 conn(0x7f173c0b0090 msgr2=0x7f173c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:54.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.394+0000 7f1741a00640 1 -- 172.21.15.62:0/2198820164 shutdown_connections 2024-03-23T11:28:54.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.394+0000 7f1741a00640 1 -- 172.21.15.62:0/2198820164 wait complete. 2024-03-23T11:28:54.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.396+0000 7f1741a00640 1 Processor -- start 2024-03-23T11:28:54.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.396+0000 7f1741a00640 1 -- start start 2024-03-23T11:28:54.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.397+0000 7f1741a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173c0b25a0 0x7f173c13fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.397+0000 7f1741a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f173c0cb530 0x7f173c0adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.397+0000 7f1741a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f173c1f55d0 0x7f173c0acf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.397+0000 7f1741a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c0af400 con 0x7f173c0b25a0 2024-03-23T11:28:54.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.397+0000 7f1741a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c0ae520 con 0x7f173c0cb530 2024-03-23T11:28:54.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.397+0000 7f1741a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173c0ae780 con 0x7f173c1f55d0 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.397+0000 7f173be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173c0b25a0 0x7f173c13fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.398+0000 7f173be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173c0b25a0 0x7f173c13fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44544/0 (socket says 172.21.15.62:44544) 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.398+0000 7f173be00640 1 -- 172.21.15.62:0/2531708307 learned_addr learned my addr 172.21.15.62:0/2531708307 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.398+0000 7f1733e00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f173c0cb530 0x7f173c0adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.398+0000 7f1741000640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f173c1f55d0 0x7f173c0acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.398+0000 7f173be00640 1 -- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f173c1f55d0 msgr2=0x7f173c0acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.398+0000 7f173be00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f173c1f55d0 0x7f173c0acf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.398+0000 7f173be00640 1 -- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f173c0cb530 msgr2=0x7f173c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.398+0000 7f173be00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f173c0cb530 0x7f173c0adfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.398+0000 7f173be00640 1 -- 172.21.15.62:0/2531708307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1734009b30 con 0x7f173c0b25a0 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.398+0000 7f173be00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173c0b25a0 0x7f173c13fba0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f172c00c450 tx=0x7f172c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:54.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.398+0000 7f1738c00640 1 -- 172.21.15.62:0/2531708307 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c0052d0 con 0x7f173c0b25a0 2024-03-23T11:28:54.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.399+0000 7f1738c00640 1 -- 172.21.15.62:0/2531708307 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172c005430 con 0x7f173c0b25a0 2024-03-23T11:28:54.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.399+0000 7f1741a00640 1 -- 172.21.15.62:0/2531708307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173c0aea60 con 0x7f173c0b25a0 2024-03-23T11:28:54.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.399+0000 7f1738c00640 1 -- 172.21.15.62:0/2531708307 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c012870 con 0x7f173c0b25a0 2024-03-23T11:28:54.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.399+0000 7f1741a00640 1 -- 172.21.15.62:0/2531708307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173c0af020 con 0x7f173c0b25a0 2024-03-23T11:28:54.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.401+0000 7f1738c00640 1 -- 172.21.15.62:0/2531708307 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f172c0129d0 con 0x7f173c0b25a0 2024-03-23T11:28:54.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.402+0000 7f1738c00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f171806a7a0 0x7f171806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.402+0000 7f1738c00640 1 -- 172.21.15.62:0/2531708307 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f172c0888f0 con 0x7f173c0b25a0 2024-03-23T11:28:54.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.402+0000 7f1733e00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f171806a7a0 0x7f171806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.402+0000 7f1745bc6740 1 Processor -- start 2024-03-23T11:28:54.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.403+0000 7f1745bc6740 1 -- start start 2024-03-23T11:28:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.403+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5236b60 0x561ab5236f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.403+0000 7f1733e00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f171806a7a0 0x7f171806cc60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1724006fd0 tx=0x7f1724008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.403+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab523abb0 0x561ab523c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.403+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52668f0 0x561ab5266d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.403+0000 7f1745bc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab51a6d80 con 0x561ab523abb0 2024-03-23T11:28:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.403+0000 7f1745bc6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab51a4350 con 0x561ab5236b60 2024-03-23T11:28:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.403+0000 7f1745bc6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ab519fc40 con 0x561ab52668f0 2024-03-23T11:28:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f1733e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab523abb0 0x561ab523c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f1733e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab523abb0 0x561ab523c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44550/0 (socket says 172.21.15.62:44550) 2024-03-23T11:28:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f1733e00640 1 -- 172.21.15.62:0/938648436 learned_addr learned my addr 172.21.15.62:0/938648436 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f1741000640 1 --2- 172.21.15.62:0/938648436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52668f0 0x561ab5266d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f173be00640 1 --2- 172.21.15.62:0/938648436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5236b60 0x561ab5236f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f1733e00640 1 -- 172.21.15.62:0/938648436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52668f0 msgr2=0x561ab5266d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f1733e00640 1 --2- 172.21.15.62:0/938648436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52668f0 0x561ab5266d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f1733e00640 1 -- 172.21.15.62:0/938648436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5236b60 msgr2=0x561ab5236f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f1733e00640 1 --2- 172.21.15.62:0/938648436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5236b60 0x561ab5236f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f1733e00640 1 -- 172.21.15.62:0/938648436 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ab516a3e0 con 0x561ab523abb0 2024-03-23T11:28:54.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f173be00640 1 --2- 172.21.15.62:0/938648436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5236b60 0x561ab5236f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:54.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.404+0000 7f1733e00640 1 --2- 172.21.15.62:0/938648436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab523abb0 0x561ab523c650 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f172400f990 tx=0x7f1724012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed70e23d668e078 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:54.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.405+0000 7f1730c00640 1 -- 172.21.15.62:0/938648436 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724014be0 con 0x561ab523abb0 2024-03-23T11:28:54.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.405+0000 7f1730c00640 1 -- 172.21.15.62:0/938648436 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1724019070 con 0x561ab523abb0 2024-03-23T11:28:54.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.405+0000 7f1730c00640 1 -- 172.21.15.62:0/938648436 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724020440 con 0x561ab523abb0 2024-03-23T11:28:54.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.405+0000 7f1745bc6740 1 -- 172.21.15.62:0/938648436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab523abb0 msgr2=0x561ab523c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.405+0000 7f1745bc6740 1 --2- 172.21.15.62:0/938648436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab523abb0 0x561ab523c650 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f172400f990 tx=0x7f1724012000 comp rx=0 tx=0).stop 2024-03-23T11:28:54.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.406+0000 7f1745bc6740 1 -- 172.21.15.62:0/938648436 shutdown_connections 2024-03-23T11:28:54.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.406+0000 7f1745bc6740 1 --2- 172.21.15.62:0/938648436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab523abb0 0x561ab523c650 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.406+0000 7f1745bc6740 1 --2- 172.21.15.62:0/938648436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52668f0 0x561ab5266d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.406+0000 7f1745bc6740 1 --2- 172.21.15.62:0/938648436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5236b60 0x561ab5236f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.406+0000 7f1745bc6740 1 -- 172.21.15.62:0/938648436 >> 172.21.15.62:0/938648436 conn(0x561ab5238be0 msgr2=0x561ab523a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:54.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.406+0000 7f1745bc6740 1 -- 172.21.15.62:0/938648436 shutdown_connections 2024-03-23T11:28:54.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.406+0000 7f1745bc6740 1 -- 172.21.15.62:0/938648436 wait complete. 2024-03-23T11:28:54.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.406+0000 7f1745bc6740 1 -- start start 2024-03-23T11:28:54.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.407+0000 7f1745bc6740 1 Processor -- start 2024-03-23T11:28:54.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.407+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52668f0 0x561ab5257aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.407+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52815f0 0x561ab5255e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5280090 0x561ab52804f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1745bc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab5140ce0 con 0x561ab52815f0 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1745bc6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab5140980 con 0x561ab5280090 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1745bc6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ab4f08000 con 0x561ab52668f0 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1745bc6740 10 client.0 _ll_register_callbacks cb 0x561ab5199410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1733e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52815f0 0x561ab5255e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1733e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52815f0 0x561ab5255e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44560/0 (socket says 172.21.15.62:44560) 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1733e00640 1 -- 172.21.15.62:0/811774789 learned_addr learned my addr 172.21.15.62:0/811774789 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f173be00640 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52668f0 0x561ab5257aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1741000640 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5280090 0x561ab52804f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1733e00640 1 -- 172.21.15.62:0/811774789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52668f0 msgr2=0x561ab5257aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1733e00640 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52668f0 0x561ab5257aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1733e00640 1 -- 172.21.15.62:0/811774789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5280090 msgr2=0x561ab52804f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:54.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1733e00640 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5280090 0x561ab52804f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1733e00640 1 -- 172.21.15.62:0/811774789 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f172400e040 con 0x561ab52815f0 2024-03-23T11:28:54.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.408+0000 7f1741000640 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5280090 0x561ab52804f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:54.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.409+0000 7f1733e00640 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52815f0 0x561ab5255e00 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x561ab52645c0 tx=0x7f1724004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44632e9285b8de server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:54.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.409+0000 7f1729600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:54.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.409+0000 7f1729600640 1 -- 172.21.15.62:0/811774789 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724020c80 con 0x561ab52815f0 2024-03-23T11:28:54.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.409+0000 7f1729600640 1 -- 172.21.15.62:0/811774789 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f1724020de0 con 0x561ab52815f0 2024-03-23T11:28:54.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.409+0000 7f1745bc6740 1 -- 172.21.15.62:0/811774789 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ab4e518d0 con 0x561ab52815f0 2024-03-23T11:28:54.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.409+0000 7f1729600640 1 -- 172.21.15.62:0/811774789 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1724014e90 con 0x561ab52815f0 2024-03-23T11:28:54.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.409+0000 7f1745bc6740 10 client.8097 fetch_fsmap learned FSMap version 70 2024-03-23T11:28:54.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.409+0000 7f1745bc6740 1 -- 172.21.15.62:0/811774789 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x561ab519e2c0 con 0x561ab52815f0 2024-03-23T11:28:54.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.410+0000 7f1729600640 1 -- 172.21.15.62:0/811774789 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561ab4e518d0 con 0x561ab52815f0 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.410+0000 7f1729600640 1 -- 172.21.15.62:0/811774789 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1724034440 con 0x561ab52815f0 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.410+0000 7f1745bc6740 10 client.8097 fetch_fsmap finished waiting for FSMap version 70 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.410+0000 7f1745bc6740 10 client.8097 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.410+0000 7f1745bc6740 20 client.8097 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.410+0000 7f1745bc6740 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x561ab52645f0 0x561ab52649d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.410+0000 7f1745bc6740 4 client.8097 mds_command: new command op to 7926 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.410+0000 7f1745bc6740 1 -- 172.21.15.62:0/811774789 --> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ab4e944a0 con 0x561ab52645f0 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.410+0000 7f173be00640 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x561ab52645f0 0x561ab52649d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.411+0000 7f173be00640 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x561ab52645f0 0x561ab52649d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=700b1fde148749c server_cookie=6d27160d0d96bef7 in_seq=0 out_seq=0 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.411+0000 7f1729600640 10 client.8097 ms_handle_connect on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.411+0000 7f1729600640 1 -- 172.21.15.62:0/811774789 <== mds.0 v2:172.21.15.62:6832/1594260761 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x561ab4e944a0 con 0x561ab52645f0 2024-03-23T11:28:54.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.411+0000 7f1729600640 10 client.8097 handle_command_reply: tid=0 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.411+0000 7f1745bc6740 1 client.8097 shutdown 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1745bc6740 1 -- 172.21.15.62:0/811774789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52815f0 msgr2=0x561ab5255e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1745bc6740 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52815f0 0x561ab5255e00 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x561ab52645c0 tx=0x7f1724004730 comp rx=0 tx=0).stop 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1745bc6740 1 -- 172.21.15.62:0/811774789 shutdown_connections 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1745bc6740 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x561ab52645f0 0x561ab52649d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1745bc6740 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52815f0 0x561ab5255e00 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1745bc6740 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5280090 0x561ab52804f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1745bc6740 1 --2- 172.21.15.62:0/811774789 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52668f0 0x561ab5257aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1729600640 0 client.8097 ms_handle_reset on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1745bc6740 1 -- 172.21.15.62:0/811774789 >> 172.21.15.62:0/811774789 conn(0x561ab5236b60 msgr2=0x561ab523b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1745bc6740 1 -- 172.21.15.62:0/811774789 shutdown_connections 2024-03-23T11:28:54.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1745bc6740 1 -- 172.21.15.62:0/811774789 wait complete. 2024-03-23T11:28:54.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.412+0000 7f1745bc6740 20 client.8097 trim_cache size 0 max 16384 2024-03-23T11:28:54.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.424+0000 7f1745bc6740 1 Processor -- start 2024-03-23T11:28:54.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.424+0000 7f1745bc6740 1 -- start start 2024-03-23T11:28:54.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.425+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab52668f0 0x561ab527f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.425+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab5280090 0x561ab5255c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.425+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52815f0 0x561ab5235d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.425+0000 7f1745bc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab5262d30 con 0x561ab5280090 2024-03-23T11:28:54.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.425+0000 7f1745bc6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab5237c10 con 0x561ab52668f0 2024-03-23T11:28:54.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.425+0000 7f1745bc6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ab52378c0 con 0x561ab52815f0 2024-03-23T11:28:54.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.425+0000 7f1733e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab5280090 0x561ab5255c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.426+0000 7f1733e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab5280090 0x561ab5255c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44576/0 (socket says 172.21.15.62:44576) 2024-03-23T11:28:54.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.426+0000 7f1733e00640 1 -- 172.21.15.62:0/3180293305 learned_addr learned my addr 172.21.15.62:0/3180293305 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:54.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.426+0000 7f1733e00640 1 -- 172.21.15.62:0/3180293305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52815f0 msgr2=0x561ab5235d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:54.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.426+0000 7f1733e00640 1 --2- 172.21.15.62:0/3180293305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52815f0 0x561ab5235d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.426+0000 7f1733e00640 1 -- 172.21.15.62:0/3180293305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab52668f0 msgr2=0x561ab527f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:54.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.426+0000 7f1733e00640 1 --2- 172.21.15.62:0/3180293305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab52668f0 0x561ab527f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.426+0000 7f1733e00640 1 -- 172.21.15.62:0/3180293305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ab5140170 con 0x561ab5280090 2024-03-23T11:28:54.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.426+0000 7f1733e00640 1 --2- 172.21.15.62:0/3180293305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab5280090 0x561ab5255c10 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f172400f680 tx=0x7f172402d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee907c9ce58c233a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:54.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.426+0000 7f1731600640 1 -- 172.21.15.62:0/3180293305 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172400f830 con 0x561ab5280090 2024-03-23T11:28:54.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.426+0000 7f1731600640 1 -- 172.21.15.62:0/3180293305 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1724014830 con 0x561ab5280090 2024-03-23T11:28:54.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.426+0000 7f1731600640 1 -- 172.21.15.62:0/3180293305 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724034950 con 0x561ab5280090 2024-03-23T11:28:54.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.427+0000 7f1745bc6740 1 -- 172.21.15.62:0/3180293305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab5280090 msgr2=0x561ab5255c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.427+0000 7f1745bc6740 1 --2- 172.21.15.62:0/3180293305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab5280090 0x561ab5255c10 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f172400f680 tx=0x7f172402d040 comp rx=0 tx=0).stop 2024-03-23T11:28:54.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.427+0000 7f1745bc6740 1 -- 172.21.15.62:0/3180293305 shutdown_connections 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.427+0000 7f1745bc6740 1 --2- 172.21.15.62:0/3180293305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab5280090 0x561ab5255c10 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.427+0000 7f1745bc6740 1 --2- 172.21.15.62:0/3180293305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52815f0 0x561ab5235d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.427+0000 7f1745bc6740 1 --2- 172.21.15.62:0/3180293305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab52668f0 0x561ab527f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.427+0000 7f1745bc6740 1 -- 172.21.15.62:0/3180293305 >> 172.21.15.62:0/3180293305 conn(0x561ab5236b60 msgr2=0x561ab52804a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.427+0000 7f1745bc6740 1 -- 172.21.15.62:0/3180293305 shutdown_connections 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.427+0000 7f1745bc6740 1 -- 172.21.15.62:0/3180293305 wait complete. 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.428+0000 7f1745bc6740 1 -- start start 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.428+0000 7f1745bc6740 1 Processor -- start 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.429+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52668f0 0x561ab529b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.429+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5280090 0x561ab52e8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.429+0000 7f1745bc6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52815f0 0x561ab52e9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.429+0000 7f1745bc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab523d400 con 0x561ab52668f0 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.429+0000 7f1745bc6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ab52e9aa0 con 0x561ab5280090 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.429+0000 7f1745bc6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ab525a380 con 0x561ab52815f0 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.429+0000 7f1745bc6740 10 client.0 _ll_register_callbacks cb 0x561ab5236770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.429+0000 7f173be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52668f0 0x561ab529b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.429+0000 7f173be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52668f0 0x561ab529b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44588/0 (socket says 172.21.15.62:44588) 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.429+0000 7f173be00640 1 -- 172.21.15.62:0/1810712816 learned_addr learned my addr 172.21.15.62:0/1810712816 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f173be00640 1 -- 172.21.15.62:0/1810712816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52815f0 msgr2=0x561ab52e9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f173be00640 1 --2- 172.21.15.62:0/1810712816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52815f0 0x561ab52e9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f173be00640 1 -- 172.21.15.62:0/1810712816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5280090 msgr2=0x561ab52e8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f173be00640 1 --2- 172.21.15.62:0/1810712816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5280090 0x561ab52e8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f173be00640 1 -- 172.21.15.62:0/1810712816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ab519e2c0 con 0x561ab52668f0 2024-03-23T11:28:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f173be00640 1 --2- 172.21.15.62:0/1810712816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52668f0 0x561ab529b080 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f172c053e90 tx=0x7f172c053df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8599e15bdccf4b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f1729600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:28:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f1729600640 1 -- 172.21.15.62:0/1810712816 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c06f220 con 0x561ab52668f0 2024-03-23T11:28:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f1729600640 1 -- 172.21.15.62:0/1810712816 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f172c06f380 con 0x561ab52668f0 2024-03-23T11:28:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f1745bc6740 1 -- 172.21.15.62:0/1810712816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1724019070 con 0x561ab52668f0 2024-03-23T11:28:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f1729600640 1 -- 172.21.15.62:0/1810712816 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f172c0540a0 con 0x561ab52668f0 2024-03-23T11:28:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f1745bc6740 10 client.8103 fetch_fsmap learned FSMap version 70 2024-03-23T11:28:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f1745bc6740 1 -- 172.21.15.62:0/1810712816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x561ab5165a50 con 0x561ab52668f0 2024-03-23T11:28:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.430+0000 7f1729600640 1 -- 172.21.15.62:0/1810712816 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1724019070 con 0x561ab52668f0 2024-03-23T11:28:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.431+0000 7f1729600640 1 -- 172.21.15.62:0/1810712816 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f172c019020 con 0x561ab52668f0 2024-03-23T11:28:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.431+0000 7f1745bc6740 10 client.8103 fetch_fsmap finished waiting for FSMap version 70 2024-03-23T11:28:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.431+0000 7f1745bc6740 10 client.8103 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.431+0000 7f1745bc6740 20 client.8103 populate_metadata read hostname 'smithi062' 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.431+0000 7f1745bc6740 1 --2- 172.21.15.62:0/1810712816 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x561ab523b7b0 0x561ab52649d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.431+0000 7f1745bc6740 4 client.8103 mds_command: new command op to 7926 tid=0[{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.431+0000 7f1745bc6740 1 -- 172.21.15.62:0/1810712816 --> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x561ab4ea57a0 con 0x561ab523b7b0 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.431+0000 7f1733e00640 1 --2- 172.21.15.62:0/1810712816 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x561ab523b7b0 0x561ab52649d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1729600640 10 client.8103 ms_handle_connect on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1733e00640 1 --2- 172.21.15.62:0/1810712816 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x561ab523b7b0 0x561ab52649d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a305f1eb2ad877 server_cookie=70828ef354e5a88a in_seq=0 out_seq=0 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1729600640 1 -- 172.21.15.62:0/1810712816 <== mds.0 v2:172.21.15.62:6832/1594260761 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561ab4ea57a0 con 0x561ab523b7b0 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1729600640 10 client.8103 handle_command_reply: tid=0 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1745bc6740 1 client.8103 shutdown 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1745bc6740 1 -- 172.21.15.62:0/1810712816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52668f0 msgr2=0x561ab529b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1745bc6740 1 --2- 172.21.15.62:0/1810712816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52668f0 0x561ab529b080 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f172c053e90 tx=0x7f172c053df0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1745bc6740 1 -- 172.21.15.62:0/1810712816 shutdown_connections 2024-03-23T11:28:54.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1745bc6740 1 --2- 172.21.15.62:0/1810712816 >> [v2:172.21.15.62:6832/1594260761,v1:172.21.15.62:6833/1594260761] conn(0x561ab523b7b0 0x561ab52649d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1745bc6740 1 --2- 172.21.15.62:0/1810712816 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561ab52815f0 0x561ab52e9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1745bc6740 1 --2- 172.21.15.62:0/1810712816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ab5280090 0x561ab52e8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1745bc6740 1 --2- 172.21.15.62:0/1810712816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ab52668f0 0x561ab529b080 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1729600640 0 client.8103 ms_handle_reset on v2:172.21.15.62:6832/1594260761 2024-03-23T11:28:54.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1745bc6740 1 -- 172.21.15.62:0/1810712816 >> 172.21.15.62:0/1810712816 conn(0x561ab5236b60 msgr2=0x561ab5256480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:54.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1745bc6740 1 -- 172.21.15.62:0/1810712816 shutdown_connections 2024-03-23T11:28:54.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.432+0000 7f1745bc6740 1 -- 172.21.15.62:0/1810712816 wait complete. 2024-03-23T11:28:54.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1745bc6740 20 client.8103 trim_cache size 0 max 16384 2024-03-23T11:28:54.430 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:54.430 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-03-23T11:28:54.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 -- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f171806a7a0 msgr2=0x7f171806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f171806a7a0 0x7f171806cc60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1724006fd0 tx=0x7f1724008040 comp rx=0 tx=0).stop 2024-03-23T11:28:54.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 -- 172.21.15.62:0/2531708307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173c0b25a0 msgr2=0x7f173c13fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173c0b25a0 0x7f173c13fba0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f172c00c450 tx=0x7f172c00c910 comp rx=0 tx=0).stop 2024-03-23T11:28:54.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 -- 172.21.15.62:0/2531708307 shutdown_connections 2024-03-23T11:28:54.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f171806a7a0 0x7f171806cc60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f173c1f55d0 0x7f173c0acf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f173c0cb530 0x7f173c0adfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 --2- 172.21.15.62:0/2531708307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f173c0b25a0 0x7f173c13fba0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 -- 172.21.15.62:0/2531708307 >> 172.21.15.62:0/2531708307 conn(0x7f173c0b0090 msgr2=0x7f173c0b2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 -- 172.21.15.62:0/2531708307 shutdown_connections 2024-03-23T11:28:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.433+0000 7f1741a00640 1 -- 172.21.15.62:0/2531708307 wait complete. 2024-03-23T11:28:54.441 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-23T11:28:54.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.665+0000 7f156e600640 1 Processor -- start 2024-03-23T11:28:54.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.665+0000 7f156e600640 1 -- start start 2024-03-23T11:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.667+0000 7f156e600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f156806c300 0x7f15680ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.667+0000 7f156e600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f156806c6e0 0x7f156806cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.667+0000 7f156e600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15680cbc60 con 0x7f15680cafa0 2024-03-23T11:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.667+0000 7f156e600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15680cbf40 con 0x7f156806c6e0 2024-03-23T11:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.667+0000 7f156e600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15680cc220 con 0x7f156806c300 2024-03-23T11:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.667+0000 7f1567e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f156806c6e0 0x7f156806cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.668+0000 7f1567e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f156806c6e0 0x7f156806cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44600/0 (socket says 172.21.15.62:44600) 2024-03-23T11:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.668+0000 7f1567e00640 1 -- 172.21.15.62:0/1359647096 learned_addr learned my addr 172.21.15.62:0/1359647096 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.668+0000 7f1567400640 1 --2- 172.21.15.62:0/1359647096 >> v2:172.21.15.120:3300/0 conn(0x7f156806c300 0x7f15680ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.668+0000 7f1567e00640 1 -- 172.21.15.62:0/1359647096 >> v2:172.21.15.120:3300/0 conn(0x7f156806c300 msgr2=0x7f15680ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.668+0000 7f1567e00640 1 --2- 172.21.15.62:0/1359647096 >> v2:172.21.15.120:3300/0 conn(0x7f156806c300 0x7f15680ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.668+0000 7f1567e00640 1 -- 172.21.15.62:0/1359647096 >> v1:172.21.15.120:6789/0 conn(0x7f15680cafa0 legacy=0x7f15680cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.668+0000 7f1567e00640 1 -- 172.21.15.62:0/1359647096 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15680cc500 con 0x7f156806c6e0 2024-03-23T11:28:54.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.669+0000 7f1567e00640 1 --2- 172.21.15.62:0/1359647096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f156806c6e0 0x7f156806cb30 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f156806c1b0 tx=0x7f156002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a02425b2cc4a02 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:54.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.669+0000 7f1566a00640 1 -- 172.21.15.62:0/1359647096 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156002ca80 con 0x7f156806c6e0 2024-03-23T11:28:54.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.670+0000 7f1566a00640 1 -- 172.21.15.62:0/1359647096 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156002cbe0 con 0x7f156806c6e0 2024-03-23T11:28:54.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.670+0000 7f1566a00640 1 -- 172.21.15.62:0/1359647096 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15600343f0 con 0x7f156806c6e0 2024-03-23T11:28:54.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.670+0000 7f156e600640 1 -- 172.21.15.62:0/1359647096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f156806c6e0 msgr2=0x7f156806cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.670+0000 7f156e600640 1 --2- 172.21.15.62:0/1359647096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f156806c6e0 0x7f156806cb30 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f156806c1b0 tx=0x7f156002bee0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.670+0000 7f156e600640 1 -- 172.21.15.62:0/1359647096 shutdown_connections 2024-03-23T11:28:54.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.670+0000 7f156e600640 1 --2- 172.21.15.62:0/1359647096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f156806c6e0 0x7f156806cb30 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.670+0000 7f156e600640 1 --2- 172.21.15.62:0/1359647096 >> v2:172.21.15.120:3300/0 conn(0x7f156806c300 0x7f15680ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.670+0000 7f156e600640 1 -- 172.21.15.62:0/1359647096 >> 172.21.15.62:0/1359647096 conn(0x7f15680b02e0 msgr2=0x7f15680b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:54.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.670+0000 7f156e600640 1 -- 172.21.15.62:0/1359647096 shutdown_connections 2024-03-23T11:28:54.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.670+0000 7f156e600640 1 -- 172.21.15.62:0/1359647096 wait complete. 2024-03-23T11:28:54.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.672+0000 7f156e600640 1 Processor -- start 2024-03-23T11:28:54.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.672+0000 7f156e600640 1 -- start start 2024-03-23T11:28:54.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.673+0000 7f156e600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f156806c300 0x7f15681f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.673+0000 7f156e600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f156806c6e0 0x7f156815c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.673+0000 7f156e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15680cafa0 0x7f156813cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.673+0000 7f156e600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156815d100 con 0x7f156806c6e0 2024-03-23T11:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.673+0000 7f156e600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156815d3e0 con 0x7f156806c300 2024-03-23T11:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.673+0000 7f156e600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156815d6c0 con 0x7f15680cafa0 2024-03-23T11:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.673+0000 7f155fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f156806c6e0 0x7f156815c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.674+0000 7f155fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f156806c6e0 0x7f156815c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44608/0 (socket says 172.21.15.62:44608) 2024-03-23T11:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.674+0000 7f155fe00640 1 -- 172.21.15.62:0/3196675562 learned_addr learned my addr 172.21.15.62:0/3196675562 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.674+0000 7f1567400640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f156806c300 0x7f15681f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.674+0000 7f1567e00640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15680cafa0 0x7f156813cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.674+0000 7f155fe00640 1 -- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15680cafa0 msgr2=0x7f156813cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.674+0000 7f155fe00640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15680cafa0 0x7f156813cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.674+0000 7f155fe00640 1 -- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f156806c300 msgr2=0x7f15681f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.674+0000 7f155fe00640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f156806c300 0x7f15681f28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.674+0000 7f155fe00640 1 -- 172.21.15.62:0/3196675562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1560009b30 con 0x7f156806c6e0 2024-03-23T11:28:54.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.674+0000 7f155fe00640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f156806c6e0 0x7f156815c720 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f155000b3f0 tx=0x7f155000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:54.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.675+0000 7f155f400640 1 -- 172.21.15.62:0/3196675562 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550011830 con 0x7f156806c6e0 2024-03-23T11:28:54.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.675+0000 7f155f400640 1 -- 172.21.15.62:0/3196675562 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1550011990 con 0x7f156806c6e0 2024-03-23T11:28:54.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.675+0000 7f155f400640 1 -- 172.21.15.62:0/3196675562 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550011d40 con 0x7f156806c6e0 2024-03-23T11:28:54.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.675+0000 7f156e600640 1 -- 172.21.15.62:0/3196675562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156815d9a0 con 0x7f156806c6e0 2024-03-23T11:28:54.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.675+0000 7f156e600640 1 -- 172.21.15.62:0/3196675562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15681f2df0 con 0x7f156806c6e0 2024-03-23T11:28:54.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.676+0000 7f156e600640 1 -- 172.21.15.62:0/3196675562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f156815dec0 con 0x7f156806c6e0 2024-03-23T11:28:54.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.677+0000 7f155f400640 1 -- 172.21.15.62:0/3196675562 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f155001a400 con 0x7f156806c6e0 2024-03-23T11:28:54.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.678+0000 7f155f400640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f154406a7d0 0x7f154406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:54.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.678+0000 7f155f400640 1 -- 172.21.15.62:0/3196675562 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f1550088cc0 con 0x7f156806c6e0 2024-03-23T11:28:54.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.679+0000 7f1567400640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f154406a7d0 0x7f154406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:54.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.680+0000 7f1567400640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f154406a7d0 0x7f154406cc90 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1558006fd0 tx=0x7f1558008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:54.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.683+0000 7f155f400640 1 -- 172.21.15.62:0/3196675562 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f156815dec0 con 0x7f156806c6e0 2024-03-23T11:28:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.922+0000 7f156e600640 1 -- 172.21.15.62:0/3196675562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f156815dec0 con 0x7f156806c6e0 2024-03-23T11:28:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.923+0000 7f155f400640 1 -- 172.21.15.62:0/3196675562 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1550015040 con 0x7f156806c6e0 2024-03-23T11:28:54.914 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:54.914 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-23T11:28:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.925+0000 7f156e600640 1 -- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f154406a7d0 msgr2=0x7f154406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.925+0000 7f156e600640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f154406a7d0 0x7f154406cc90 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1558006fd0 tx=0x7f1558008040 comp rx=0 tx=0).stop 2024-03-23T11:28:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.925+0000 7f156e600640 1 -- 172.21.15.62:0/3196675562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f156806c6e0 msgr2=0x7f156815c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.925+0000 7f156e600640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f156806c6e0 0x7f156815c720 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f155000b3f0 tx=0x7f155000b8b0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.926+0000 7f156e600640 1 -- 172.21.15.62:0/3196675562 shutdown_connections 2024-03-23T11:28:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.926+0000 7f156e600640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f154406a7d0 0x7f154406cc90 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.926+0000 7f156e600640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f156806c6e0 0x7f156815c720 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.926+0000 7f156e600640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15680cafa0 0x7f156813cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.926+0000 7f156e600640 1 --2- 172.21.15.62:0/3196675562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f156806c300 0x7f15681f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.926+0000 7f156e600640 1 -- 172.21.15.62:0/3196675562 >> 172.21.15.62:0/3196675562 conn(0x7f15680b02e0 msgr2=0x7f156813eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.926+0000 7f156e600640 1 -- 172.21.15.62:0/3196675562 shutdown_connections 2024-03-23T11:28:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:54.926+0000 7f156e600640 1 -- 172.21.15.62:0/3196675562 wait complete. 2024-03-23T11:28:54.937 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-23T11:28:54.937 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:28:54.937 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:28:55.041 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:28:55.149 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:28:55.150 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:28:55.150 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:28:55.211 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:28:55.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.448+0000 7fe5ab200640 1 Processor -- start 2024-03-23T11:28:55.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.449+0000 7fe5ab200640 1 -- start start 2024-03-23T11:28:55.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.450+0000 7fe5ab200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a406cc60 0x7fe5a40cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.450+0000 7fe5ab200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe5a40ce420 0x7fe5a40b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.450+0000 7fe5ab200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a40ce8b0 con 0x7fe5a4159490 2024-03-23T11:28:55.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.451+0000 7fe5ab200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a40ced90 con 0x7fe5a40ce420 2024-03-23T11:28:55.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.451+0000 7fe5ab200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a406e3e0 con 0x7fe5a406cc60 2024-03-23T11:28:55.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.451+0000 7fe5a9e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe5a4159490 0x7fe5a415da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:45822/0 (socket says 172.21.15.62:45822) 2024-03-23T11:28:55.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.451+0000 7fe5a9e00640 1 -- 172.21.15.62:0/312848922 learned_addr learned my addr 172.21.15.62:0/312848922 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:55.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.451+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371561827 0 0) 0x7fe5a40ce8b0 con 0x7fe5a4159490 2024-03-23T11:28:55.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.451+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe58c003520 con 0x7fe5a4159490 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.451+0000 7fe5aa800640 1 --2- 172.21.15.62:0/312848922 >> v2:172.21.15.120:3300/0 conn(0x7fe5a40ce420 0x7fe5a40b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.451+0000 7fe5a9400640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a406cc60 0x7fe5a40cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1188290209 0 0) 0x7fe58c003520 con 0x7fe5a4159490 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a406cc60 msgr2=0x7fe5a40cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a406cc60 0x7fe5a40cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 >> v2:172.21.15.120:3300/0 conn(0x7fe5a40ce420 msgr2=0x7fe5a40b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 --2- 172.21.15.62:0/312848922 >> v2:172.21.15.120:3300/0 conn(0x7fe5a40ce420 0x7fe5a40b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a406c1c0 con 0x7fe5a4159490 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fe598001670 con 0x7fe5a4159490 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 >> v1:172.21.15.62:6789/0 conn(0x7fe5a4159490 legacy=0x7fe5a415da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a406cc60 0x7fe58c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 0x7fe58c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe58c004d00 0x7fe58c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40ced90 con 0x7fe58c003fa0 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a406e3e0 con 0x7fe58c004d00 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c00bb40 con 0x7fe5a406cc60 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a9400640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 0x7fe58c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.452+0000 7fe5a9e00640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a406cc60 0x7fe58c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.453+0000 7fe5aa800640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe58c004d00 0x7fe58c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.453+0000 7fe5a9400640 1 -- 172.21.15.62:0/312848922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a406cc60 msgr2=0x7fe58c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.453+0000 7fe5a9400640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a406cc60 0x7fe58c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.453+0000 7fe5a9400640 1 -- 172.21.15.62:0/312848922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe58c004d00 msgr2=0x7fe58c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.453+0000 7fe5a9400640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe58c004d00 0x7fe58c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.453+0000 7fe5a9400640 1 -- 172.21.15.62:0/312848922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c003520 con 0x7fe58c003fa0 2024-03-23T11:28:55.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.454+0000 7fe5a9400640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 0x7fe58c0047c0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fe59400ba30 tx=0x7fe58c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c596f7785e494bda server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:55.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.454+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594032630 con 0x7fe58c003fa0 2024-03-23T11:28:55.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.454+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe594032790 con 0x7fe58c003fa0 2024-03-23T11:28:55.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.454+0000 7fe5a8a00640 1 -- 172.21.15.62:0/312848922 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594032b20 con 0x7fe58c003fa0 2024-03-23T11:28:55.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.455+0000 7fe5ab200640 1 -- 172.21.15.62:0/312848922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 msgr2=0x7fe58c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:55.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.455+0000 7fe5ab200640 1 --2- 172.21.15.62:0/312848922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 0x7fe58c0047c0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fe59400ba30 tx=0x7fe58c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.455+0000 7fe5aa800640 1 -- 172.21.15.62:0/312848922 reap_dead start 2024-03-23T11:28:55.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.455+0000 7fe5ab200640 1 -- 172.21.15.62:0/312848922 shutdown_connections 2024-03-23T11:28:55.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.455+0000 7fe5ab200640 1 -- 172.21.15.62:0/312848922 >> 172.21.15.62:0/312848922 conn(0x7fe5a4066050 msgr2=0x7fe5a4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:55.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.455+0000 7fe5ab200640 1 -- 172.21.15.62:0/312848922 shutdown_connections 2024-03-23T11:28:55.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.455+0000 7fe5ab200640 1 -- 172.21.15.62:0/312848922 wait complete. 2024-03-23T11:28:55.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.457+0000 7fe5ab200640 1 Processor -- start 2024-03-23T11:28:55.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.457+0000 7fe5ab200640 1 -- start start 2024-03-23T11:28:55.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.458+0000 7fe5ab200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 0x7fe5a41f05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.458+0000 7fe5ab200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a40a7f50 0x7fe5a40a83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.458+0000 7fe5ab200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5a4152780 0x7fe5a4152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.458+0000 7fe5ab200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a4153490 con 0x7fe58c003fa0 2024-03-23T11:28:55.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.458+0000 7fe5ab200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40a88f0 con 0x7fe5a4152780 2024-03-23T11:28:55.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.458+0000 7fe5ab200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41f0af0 con 0x7fe5a40a7f50 2024-03-23T11:28:55.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.458+0000 7fe5a9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 0x7fe5a41f05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.458+0000 7fe5a9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 0x7fe5a41f05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44618/0 (socket says 172.21.15.62:44618) 2024-03-23T11:28:55.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.458+0000 7fe5a9e00640 1 -- 172.21.15.62:0/2040379191 learned_addr learned my addr 172.21.15.62:0/2040379191 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:55.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe5a9400640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a40a7f50 0x7fe5a40a83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe5aa800640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5a4152780 0x7fe5a4152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe5a9e00640 1 -- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a40a7f50 msgr2=0x7fe5a40a83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:55.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe5a9e00640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a40a7f50 0x7fe5a40a83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe5a9e00640 1 -- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5a4152780 msgr2=0x7fe5a4152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:55.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe5a9e00640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5a4152780 0x7fe5a4152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe5a9e00640 1 -- 172.21.15.62:0/2040379191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a41ef7f0 con 0x7fe58c003fa0 2024-03-23T11:28:55.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe5a9e00640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 0x7fe5a41f05b0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe598001820 tx=0x7fe59800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:55.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe59e000640 1 -- 172.21.15.62:0/2040379191 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59800ee60 con 0x7fe58c003fa0 2024-03-23T11:28:55.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe59e000640 1 -- 172.21.15.62:0/2040379191 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59800c8d0 con 0x7fe58c003fa0 2024-03-23T11:28:55.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe59e000640 1 -- 172.21.15.62:0/2040379191 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59800cc80 con 0x7fe58c003fa0 2024-03-23T11:28:55.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.459+0000 7fe5ab200640 1 -- 172.21.15.62:0/2040379191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a41efad0 con 0x7fe58c003fa0 2024-03-23T11:28:55.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.460+0000 7fe5ab200640 1 -- 172.21.15.62:0/2040379191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a41f0090 con 0x7fe58c003fa0 2024-03-23T11:28:55.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.461+0000 7fe5ab200640 1 -- 172.21.15.62:0/2040379191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a406c420 con 0x7fe58c003fa0 2024-03-23T11:28:55.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.462+0000 7fe59e000640 1 -- 172.21.15.62:0/2040379191 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe598013420 con 0x7fe58c003fa0 2024-03-23T11:28:55.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.463+0000 7fe59e000640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe58806a7d0 0x7fe58806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.463+0000 7fe59e000640 1 -- 172.21.15.62:0/2040379191 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fe598088b00 con 0x7fe58c003fa0 2024-03-23T11:28:55.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.463+0000 7fe5a9400640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe58806a7d0 0x7fe58806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.464+0000 7fe5a9400640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe58806a7d0 0x7fe58806cc90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe59400ba00 tx=0x7fe59403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:55.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.468+0000 7fe59e000640 1 -- 172.21.15.62:0/2040379191 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe598057850 con 0x7fe58c003fa0 2024-03-23T11:28:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.675+0000 7fe5ab200640 1 -- 172.21.15.62:0/2040379191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5a406d0c0 con 0x7fe58c003fa0 2024-03-23T11:28:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.678+0000 7fe59e000640 1 -- 172.21.15.62:0/2040379191 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16401 (secure 0 0 0) 0x7fe5980573e0 con 0x7fe58c003fa0 2024-03-23T11:28:55.670 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:55.671 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":93,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:28:13.436047+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:26:22.794149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-23T11:26:24.150997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6832/2573443114":"2024-03-24T11:28:13.436002+0000","172.21.15.62:6833/2573443114":"2024-03-24T11:28:13.436002+0000","172.21.15.62:6833/3056962984":"2024-03-24T11:26:47.481437+0000","172.21.15.62:6832/3056962984":"2024-03-24T11:26:47.481437+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:28:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.681+0000 7fe5ab200640 1 -- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe58806a7d0 msgr2=0x7fe58806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.681+0000 7fe5ab200640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe58806a7d0 0x7fe58806cc90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe59400ba00 tx=0x7fe59403d040 comp rx=0 tx=0).stop 2024-03-23T11:28:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.681+0000 7fe5ab200640 1 -- 172.21.15.62:0/2040379191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 msgr2=0x7fe5a41f05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.681+0000 7fe5ab200640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 0x7fe5a41f05b0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe598001820 tx=0x7fe59800ec10 comp rx=0 tx=0).stop 2024-03-23T11:28:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.681+0000 7fe5ab200640 1 -- 172.21.15.62:0/2040379191 shutdown_connections 2024-03-23T11:28:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.681+0000 7fe5ab200640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe58806a7d0 0x7fe58806cc90 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.681+0000 7fe5ab200640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe5a4152780 0x7fe5a4152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.681+0000 7fe5ab200640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5a40a7f50 0x7fe5a40a83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.681+0000 7fe5ab200640 1 --2- 172.21.15.62:0/2040379191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c003fa0 0x7fe5a41f05b0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.681+0000 7fe5ab200640 1 -- 172.21.15.62:0/2040379191 >> 172.21.15.62:0/2040379191 conn(0x7fe5a4066050 msgr2=0x7fe5a41586d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:55.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.682+0000 7fe5ab200640 1 -- 172.21.15.62:0/2040379191 shutdown_connections 2024-03-23T11:28:55.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.682+0000 7fe5ab200640 1 -- 172.21.15.62:0/2040379191 wait complete. 2024-03-23T11:28:55.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:26:22.794149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:26:24.150997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:28:55.694 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:28:55.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.931+0000 7fabb3200640 1 Processor -- start 2024-03-23T11:28:55.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.931+0000 7fabb3200640 1 -- start start 2024-03-23T11:28:55.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.933+0000 7fabb3200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabb40d0d40 0x7fabb40d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.933+0000 7fabb3200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fabb40b24e0 0x7fabb40b2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.933+0000 7fabb3200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb40ce870 con 0x7fabb415ca00 2024-03-23T11:28:55.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.933+0000 7fabb3200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb40d1700 con 0x7fabb40b24e0 2024-03-23T11:28:55.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.933+0000 7fabb3200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb40b2eb0 con 0x7fabb40d0d40 2024-03-23T11:28:55.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.933+0000 7fabb2800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fabb40b24e0 0x7fabb40b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.933+0000 7fabb2800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fabb40b24e0 0x7fabb40b2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44622/0 (socket says 172.21.15.62:44622) 2024-03-23T11:28:55.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.933+0000 7fabb2800640 1 -- 172.21.15.62:0/3390563767 learned_addr learned my addr 172.21.15.62:0/3390563767 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:55.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.933+0000 7faba9400640 1 --2- 172.21.15.62:0/3390563767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabb40d0d40 0x7fabb40d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.934+0000 7fabb2800640 1 -- 172.21.15.62:0/3390563767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabb40d0d40 msgr2=0x7fabb40d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:55.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.934+0000 7fabb2800640 1 --2- 172.21.15.62:0/3390563767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabb40d0d40 0x7fabb40d11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.934+0000 7fabb2800640 1 -- 172.21.15.62:0/3390563767 >> v1:172.21.15.120:6789/0 conn(0x7fabb415ca00 legacy=0x7fabb40ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:55.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.934+0000 7fabb2800640 1 -- 172.21.15.62:0/3390563767 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb40cfe40 con 0x7fabb40b24e0 2024-03-23T11:28:55.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.935+0000 7fabb2800640 1 --2- 172.21.15.62:0/3390563767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabb40b24e0 0x7fabb40b2970 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fabac009870 tx=0x7fabac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3804c2fea3f592 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:55.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.935+0000 7fabb1400640 1 -- 172.21.15.62:0/3390563767 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac030840 con 0x7fabb40b24e0 2024-03-23T11:28:55.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.935+0000 7fabb1400640 1 -- 172.21.15.62:0/3390563767 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabac0309a0 con 0x7fabb40b24e0 2024-03-23T11:28:55.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.935+0000 7fabb1400640 1 -- 172.21.15.62:0/3390563767 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac030d30 con 0x7fabb40b24e0 2024-03-23T11:28:55.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.936+0000 7fabb3200640 1 -- 172.21.15.62:0/3390563767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabb40b24e0 msgr2=0x7fabb40b2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:55.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.936+0000 7fabb3200640 1 --2- 172.21.15.62:0/3390563767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabb40b24e0 0x7fabb40b2970 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fabac009870 tx=0x7fabac02b980 comp rx=0 tx=0).stop 2024-03-23T11:28:55.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.936+0000 7fabb3200640 1 -- 172.21.15.62:0/3390563767 shutdown_connections 2024-03-23T11:28:55.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.936+0000 7fabb3200640 1 --2- 172.21.15.62:0/3390563767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabb40b24e0 0x7fabb40b2970 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.936+0000 7fabb3200640 1 --2- 172.21.15.62:0/3390563767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabb40d0d40 0x7fabb40d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.936+0000 7fabb3200640 1 -- 172.21.15.62:0/3390563767 >> 172.21.15.62:0/3390563767 conn(0x7fabb409ba10 msgr2=0x7fabb4089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:55.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.936+0000 7fabb3200640 1 -- 172.21.15.62:0/3390563767 shutdown_connections 2024-03-23T11:28:55.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.936+0000 7fabb3200640 1 -- 172.21.15.62:0/3390563767 wait complete. 2024-03-23T11:28:55.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.938+0000 7fabb3200640 1 Processor -- start 2024-03-23T11:28:55.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.938+0000 7fabb3200640 1 -- start start 2024-03-23T11:28:55.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.939+0000 7fabb3200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabb40b24e0 0x7fabb40b6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.939+0000 7fabb3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabb40d0d40 0x7fabb415f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.939+0000 7fabb3200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabb415ca00 0x7fabb40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.939+0000 7fabb3200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb415f930 con 0x7fabb40d0d40 2024-03-23T11:28:55.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.939+0000 7fabb3200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb40ca8f0 con 0x7fabb415ca00 2024-03-23T11:28:55.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.939+0000 7fabb3200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb415fbd0 con 0x7fabb40b24e0 2024-03-23T11:28:55.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.939+0000 7faba9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabb40d0d40 0x7fabb415f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.939+0000 7faba9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabb40d0d40 0x7fabb415f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44630/0 (socket says 172.21.15.62:44630) 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.939+0000 7faba9400640 1 -- 172.21.15.62:0/2834731246 learned_addr learned my addr 172.21.15.62:0/2834731246 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7fabb2800640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabb415ca00 0x7fabb40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7faba9400640 1 -- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabb40b24e0 msgr2=0x7fabb40b6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7fabb1e00640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabb40b24e0 0x7fabb40b6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7faba9400640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabb40b24e0 0x7fabb40b6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7faba9400640 1 -- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabb415ca00 msgr2=0x7fabb40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7faba9400640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabb415ca00 0x7fabb40ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7faba9400640 1 -- 172.21.15.62:0/2834731246 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac0094d0 con 0x7fabb40d0d40 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7fabb2800640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabb415ca00 0x7fabb40ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7fabb1e00640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabb40b24e0 0x7fabb40b6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7faba9400640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabb40d0d40 0x7fabb415f3f0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00d5f0 tx=0x7fab9c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7fabaaa00640 1 -- 172.21.15.62:0/2834731246 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c0125f0 con 0x7fabb40d0d40 2024-03-23T11:28:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7fabaaa00640 1 -- 172.21.15.62:0/2834731246 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9c012750 con 0x7fabb40d0d40 2024-03-23T11:28:55.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.940+0000 7fabb3200640 1 -- 172.21.15.62:0/2834731246 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb415ce80 con 0x7fabb40d0d40 2024-03-23T11:28:55.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.941+0000 7fabaaa00640 1 -- 172.21.15.62:0/2834731246 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c012b00 con 0x7fabb40d0d40 2024-03-23T11:28:55.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.941+0000 7fabb3200640 1 -- 172.21.15.62:0/2834731246 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb415d440 con 0x7fabb40d0d40 2024-03-23T11:28:55.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.942+0000 7fabb3200640 1 -- 172.21.15.62:0/2834731246 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb40672c0 con 0x7fabb40d0d40 2024-03-23T11:28:55.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.943+0000 7fabaaa00640 1 -- 172.21.15.62:0/2834731246 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fab9c012c60 con 0x7fabb40d0d40 2024-03-23T11:28:55.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.943+0000 7fabaaa00640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab9006a7d0 0x7fab9006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:55.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.944+0000 7fabaaa00640 1 -- 172.21.15.62:0/2834731246 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fab9c0893c0 con 0x7fabb40d0d40 2024-03-23T11:28:55.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.944+0000 7fabb1e00640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab9006a7d0 0x7fab9006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:55.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.945+0000 7fabb1e00640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab9006a7d0 0x7fab9006cc90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7faba4007fd0 tx=0x7faba4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:55.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:55.948+0000 7fabaaa00640 1 -- 172.21.15.62:0/2834731246 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fab9c054670 con 0x7fabb40d0d40 2024-03-23T11:28:56.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.145+0000 7fabb3200640 1 -- 172.21.15.62:0/2834731246 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fabb40b22c0 con 0x7fabb40d0d40 2024-03-23T11:28:56.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.146+0000 7fabaaa00640 1 -- 172.21.15.62:0/2834731246 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab9c057c90 con 0x7fabb40d0d40 2024-03-23T11:28:56.137 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:28:56.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 -- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab9006a7d0 msgr2=0x7fab9006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab9006a7d0 0x7fab9006cc90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7faba4007fd0 tx=0x7faba4009040 comp rx=0 tx=0).stop 2024-03-23T11:28:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 -- 172.21.15.62:0/2834731246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabb40d0d40 msgr2=0x7fabb415f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabb40d0d40 0x7fabb415f3f0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00d5f0 tx=0x7fab9c00dab0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 -- 172.21.15.62:0/2834731246 shutdown_connections 2024-03-23T11:28:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fab9006a7d0 0x7fab9006cc90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabb40d0d40 0x7fabb415f3f0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabb415ca00 0x7fabb40ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 --2- 172.21.15.62:0/2834731246 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fabb40b24e0 0x7fabb40b6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 -- 172.21.15.62:0/2834731246 >> 172.21.15.62:0/2834731246 conn(0x7fabb409ba10 msgr2=0x7fabb4072c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 -- 172.21.15.62:0/2834731246 shutdown_connections 2024-03-23T11:28:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.149+0000 7fabb3200640 1 -- 172.21.15.62:0/2834731246 wait complete. 2024-03-23T11:28:56.160 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:28:56.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.387+0000 7f57e3e00640 1 Processor -- start 2024-03-23T11:28:56.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.388+0000 7f57e3e00640 1 -- start start 2024-03-23T11:28:56.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.389+0000 7f57e3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f57e415fcf0 0x7f57e40d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.389+0000 7f57e3e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f57e40b6dc0 0x7f57e41608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.389+0000 7f57e3e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e40d3540 con 0x7f57e4160420 2024-03-23T11:28:56.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.389+0000 7f57e3e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e4160df0 con 0x7f57e415fcf0 2024-03-23T11:28:56.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.389+0000 7f57e3e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e40ce510 con 0x7f57e40b6dc0 2024-03-23T11:28:56.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.390+0000 7f57e2a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f57e415fcf0 0x7f57e40d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.390+0000 7f57e2000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f57e4160420 0x7f57e40d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:45826/0 (socket says 172.21.15.62:45826) 2024-03-23T11:28:56.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.390+0000 7f57e2a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f57e415fcf0 0x7f57e40d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44634/0 (socket says 172.21.15.62:44634) 2024-03-23T11:28:56.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.390+0000 7f57e2000640 1 -- 172.21.15.62:0/3702347914 learned_addr learned my addr 172.21.15.62:0/3702347914 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:56.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.390+0000 7f57e3400640 1 --2- 172.21.15.62:0/3702347914 >> v2:172.21.15.120:3300/0 conn(0x7f57e40b6dc0 0x7f57e41608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.390+0000 7f57e2a00640 1 -- 172.21.15.62:0/3702347914 >> v2:172.21.15.120:3300/0 conn(0x7f57e40b6dc0 msgr2=0x7f57e41608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:56.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.390+0000 7f57e2a00640 1 --2- 172.21.15.62:0/3702347914 >> v2:172.21.15.120:3300/0 conn(0x7f57e40b6dc0 0x7f57e41608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.390+0000 7f57e2a00640 1 -- 172.21.15.62:0/3702347914 >> v1:172.21.15.62:6789/0 conn(0x7f57e4160420 legacy=0x7f57e40d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:56.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.390+0000 7f57e2a00640 1 -- 172.21.15.62:0/3702347914 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40cdc70 con 0x7f57e415fcf0 2024-03-23T11:28:56.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.390+0000 7f57e1600640 1 -- 172.21.15.62:0/3702347914 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170547345 0 0) 0x7f57e40d3540 con 0x7f57e4160420 2024-03-23T11:28:56.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.391+0000 7f57e2a00640 1 --2- 172.21.15.62:0/3702347914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57e415fcf0 0x7f57e40d3000 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f57e408bb20 tx=0x7f57d402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a823aa045a51ead0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:56.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.392+0000 7f57e1600640 1 -- 172.21.15.62:0/3702347914 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d402cbf0 con 0x7f57e415fcf0 2024-03-23T11:28:56.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.392+0000 7f57e1600640 1 -- 172.21.15.62:0/3702347914 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d402cd50 con 0x7f57e415fcf0 2024-03-23T11:28:56.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.392+0000 7f57e1600640 1 -- 172.21.15.62:0/3702347914 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d4034660 con 0x7f57e415fcf0 2024-03-23T11:28:56.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.392+0000 7f57e3e00640 1 -- 172.21.15.62:0/3702347914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57e415fcf0 msgr2=0x7f57e40d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:56.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.392+0000 7f57e3e00640 1 --2- 172.21.15.62:0/3702347914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57e415fcf0 0x7f57e40d3000 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f57e408bb20 tx=0x7f57d402be40 comp rx=0 tx=0).stop 2024-03-23T11:28:56.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.392+0000 7f57e3e00640 1 -- 172.21.15.62:0/3702347914 shutdown_connections 2024-03-23T11:28:56.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.392+0000 7f57e3e00640 1 --2- 172.21.15.62:0/3702347914 >> v2:172.21.15.120:3300/0 conn(0x7f57e40b6dc0 0x7f57e41608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.392+0000 7f57e3e00640 1 --2- 172.21.15.62:0/3702347914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57e415fcf0 0x7f57e40d3000 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.392+0000 7f57e3e00640 1 -- 172.21.15.62:0/3702347914 >> 172.21.15.62:0/3702347914 conn(0x7f57e4089f60 msgr2=0x7f57e40d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:56.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.392+0000 7f57e3e00640 1 -- 172.21.15.62:0/3702347914 shutdown_connections 2024-03-23T11:28:56.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.392+0000 7f57e3e00640 1 -- 172.21.15.62:0/3702347914 wait complete. 2024-03-23T11:28:56.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.394+0000 7f57e3e00640 1 Processor -- start 2024-03-23T11:28:56.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.394+0000 7f57e3e00640 1 -- start start 2024-03-23T11:28:56.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.395+0000 7f57e3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57e40b6dc0 0x7f57e414c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.395+0000 7f57e3e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e415fcf0 0x7f57e414ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.395+0000 7f57e3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57e4160420 0x7f57e414a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.395+0000 7f57e3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e414bd70 con 0x7f57e4160420 2024-03-23T11:28:56.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.395+0000 7f57e3e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e414b160 con 0x7f57e415fcf0 2024-03-23T11:28:56.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.395+0000 7f57e3e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e414b440 con 0x7f57e40b6dc0 2024-03-23T11:28:56.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.395+0000 7f57e3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57e4160420 0x7f57e414a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57e3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57e4160420 0x7f57e414a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44642/0 (socket says 172.21.15.62:44642) 2024-03-23T11:28:56.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57e3400640 1 -- 172.21.15.62:0/1035486040 learned_addr learned my addr 172.21.15.62:0/1035486040 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:56.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57e2a00640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57e40b6dc0 0x7f57e414c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57e2000640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e415fcf0 0x7f57e414ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57e3400640 1 -- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57e40b6dc0 msgr2=0x7f57e414c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:56.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57e3400640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57e40b6dc0 0x7f57e414c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57e3400640 1 -- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e415fcf0 msgr2=0x7f57e414ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:56.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57e3400640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e415fcf0 0x7f57e414ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57e3400640 1 -- 172.21.15.62:0/1035486040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d4009a70 con 0x7f57e4160420 2024-03-23T11:28:56.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57e2000640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e415fcf0 0x7f57e414ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:28:56.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57e3400640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57e4160420 0x7f57e414a780 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00d800 tx=0x7f57dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:56.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.396+0000 7f57daa00640 1 -- 172.21.15.62:0/1035486040 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00b6c0 con 0x7f57e4160420 2024-03-23T11:28:56.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.397+0000 7f57daa00640 1 -- 172.21.15.62:0/1035486040 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc00b840 con 0x7f57e4160420 2024-03-23T11:28:56.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.397+0000 7f57daa00640 1 -- 172.21.15.62:0/1035486040 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00bbf0 con 0x7f57e4160420 2024-03-23T11:28:56.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.397+0000 7f57e3e00640 1 -- 172.21.15.62:0/1035486040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e414b720 con 0x7f57e4160420 2024-03-23T11:28:56.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.397+0000 7f57e3e00640 1 -- 172.21.15.62:0/1035486040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e414d490 con 0x7f57e4160420 2024-03-23T11:28:56.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.398+0000 7f57e3e00640 1 -- 172.21.15.62:0/1035486040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e414bf20 con 0x7f57e4160420 2024-03-23T11:28:56.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.399+0000 7f57daa00640 1 -- 172.21.15.62:0/1035486040 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f57dc00bd50 con 0x7f57e4160420 2024-03-23T11:28:56.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.400+0000 7f57daa00640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f57c006a7a0 0x7f57c006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.400+0000 7f57daa00640 1 -- 172.21.15.62:0/1035486040 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f57dc089450 con 0x7f57e4160420 2024-03-23T11:28:56.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.400+0000 7f57e2a00640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f57c006a7a0 0x7f57c006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.401+0000 7f57e2a00640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f57c006a7a0 0x7f57c006cc60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f57d4036000 tx=0x7f57d4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:56.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.404+0000 7f57daa00640 1 -- 172.21.15.62:0/1035486040 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f57e414bf20 con 0x7f57e4160420 2024-03-23T11:28:56.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.603+0000 7f57e3e00640 1 -- 172.21.15.62:0/1035486040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f57e414d620 con 0x7f57e4160420 2024-03-23T11:28:56.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.603+0000 7f57daa00640 1 -- 172.21.15.62:0/1035486040 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f57dc054700 con 0x7f57e4160420 2024-03-23T11:28:56.595 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:28:56.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.605+0000 7f57e3e00640 1 -- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f57c006a7a0 msgr2=0x7f57c006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:56.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.605+0000 7f57e3e00640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f57c006a7a0 0x7f57c006cc60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f57d4036000 tx=0x7f57d4030000 comp rx=0 tx=0).stop 2024-03-23T11:28:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.605+0000 7f57e3e00640 1 -- 172.21.15.62:0/1035486040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57e4160420 msgr2=0x7f57e414a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.606+0000 7f57e3e00640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57e4160420 0x7f57e414a780 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00d800 tx=0x7f57dc00dcc0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.606+0000 7f57e3e00640 1 -- 172.21.15.62:0/1035486040 shutdown_connections 2024-03-23T11:28:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.606+0000 7f57e3e00640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f57c006a7a0 0x7f57c006cc60 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.606+0000 7f57e3e00640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57e4160420 0x7f57e414a780 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.606+0000 7f57e3e00640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e415fcf0 0x7f57e414ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.606+0000 7f57e3e00640 1 --2- 172.21.15.62:0/1035486040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57e40b6dc0 0x7f57e414c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.606+0000 7f57e3e00640 1 -- 172.21.15.62:0/1035486040 >> 172.21.15.62:0/1035486040 conn(0x7f57e4089f60 msgr2=0x7f57e406ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.606+0000 7f57e3e00640 1 -- 172.21.15.62:0/1035486040 shutdown_connections 2024-03-23T11:28:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.606+0000 7f57e3e00640 1 -- 172.21.15.62:0/1035486040 wait complete. 2024-03-23T11:28:56.615 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:28:56.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.842+0000 7f2539e00640 1 Processor -- start 2024-03-23T11:28:56.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.842+0000 7f2539e00640 1 -- start start 2024-03-23T11:28:56.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.843+0000 7f2539e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f253415ca00 0x7f25340ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.843+0000 7f2539e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f25340d0d40 0x7f25340d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.843+0000 7f2539e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25340cfe40 con 0x7f25340b24e0 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.843+0000 7f2539e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25340d1700 con 0x7f253415ca00 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.843+0000 7f2539e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25340cd680 con 0x7f25340d0d40 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.844+0000 7f2538a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f253415ca00 0x7f25340ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.844+0000 7f2538a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f253415ca00 0x7f25340ce160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44650/0 (socket says 172.21.15.62:44650) 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.844+0000 7f2538a00640 1 -- 172.21.15.62:0/2470910811 learned_addr learned my addr 172.21.15.62:0/2470910811 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.844+0000 7f2533e00640 1 --2- 172.21.15.62:0/2470910811 >> v2:172.21.15.120:3300/0 conn(0x7f25340d0d40 0x7f25340d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.844+0000 7f2538a00640 1 -- 172.21.15.62:0/2470910811 >> v2:172.21.15.120:3300/0 conn(0x7f25340d0d40 msgr2=0x7f25340d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.844+0000 7f2538a00640 1 --2- 172.21.15.62:0/2470910811 >> v2:172.21.15.120:3300/0 conn(0x7f25340d0d40 0x7f25340d11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.844+0000 7f2538a00640 1 -- 172.21.15.62:0/2470910811 >> v1:172.21.15.120:6789/0 conn(0x7f25340b24e0 legacy=0x7f25340b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:56.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.844+0000 7f2538a00640 1 -- 172.21.15.62:0/2470910811 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25340cd960 con 0x7f253415ca00 2024-03-23T11:28:56.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.845+0000 7f2538a00640 1 --2- 172.21.15.62:0/2470910811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f253415ca00 0x7f25340ce160 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f253408bcd0 tx=0x7f252802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b57a90ae2e5c55 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:56.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.846+0000 7f2533400640 1 -- 172.21.15.62:0/2470910811 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528004230 con 0x7f253415ca00 2024-03-23T11:28:56.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.846+0000 7f2533400640 1 -- 172.21.15.62:0/2470910811 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2528004390 con 0x7f253415ca00 2024-03-23T11:28:56.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.846+0000 7f2533400640 1 -- 172.21.15.62:0/2470910811 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528003810 con 0x7f253415ca00 2024-03-23T11:28:56.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.846+0000 7f2539e00640 1 -- 172.21.15.62:0/2470910811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f253415ca00 msgr2=0x7f25340ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:56.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.846+0000 7f2539e00640 1 --2- 172.21.15.62:0/2470910811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f253415ca00 0x7f25340ce160 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f253408bcd0 tx=0x7f252802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.847+0000 7f2539e00640 1 -- 172.21.15.62:0/2470910811 shutdown_connections 2024-03-23T11:28:56.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.847+0000 7f2539e00640 1 --2- 172.21.15.62:0/2470910811 >> v2:172.21.15.120:3300/0 conn(0x7f25340d0d40 0x7f25340d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.847+0000 7f2539e00640 1 --2- 172.21.15.62:0/2470910811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f253415ca00 0x7f25340ce160 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.847+0000 7f2539e00640 1 -- 172.21.15.62:0/2470910811 >> 172.21.15.62:0/2470910811 conn(0x7f253409ba10 msgr2=0x7f2534089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:56.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.847+0000 7f2539e00640 1 -- 172.21.15.62:0/2470910811 shutdown_connections 2024-03-23T11:28:56.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.847+0000 7f2539e00640 1 -- 172.21.15.62:0/2470910811 wait complete. 2024-03-23T11:28:56.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.848+0000 7f2539e00640 1 Processor -- start 2024-03-23T11:28:56.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.848+0000 7f2539e00640 1 -- start start 2024-03-23T11:28:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.849+0000 7f2539e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25340b24e0 0x7f253415f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.849+0000 7f2539e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25340d0d40 0x7f253415fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.849+0000 7f2539e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f253415ca00 0x7f25341f56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2539e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253406c480 con 0x7f253415ca00 2024-03-23T11:28:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2539e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25340b7900 con 0x7f25340d0d40 2024-03-23T11:28:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2539e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25340b7be0 con 0x7f25340b24e0 2024-03-23T11:28:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2539400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f253415ca00 0x7f25341f56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2539400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f253415ca00 0x7f25341f56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44656/0 (socket says 172.21.15.62:44656) 2024-03-23T11:28:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2539400640 1 -- 172.21.15.62:0/941509589 learned_addr learned my addr 172.21.15.62:0/941509589 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2538a00640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25340b24e0 0x7f253415f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2533e00640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25340d0d40 0x7f253415fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2539400640 1 -- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25340b24e0 msgr2=0x7f253415f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2539400640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25340b24e0 0x7f253415f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2539400640 1 -- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25340d0d40 msgr2=0x7f253415fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2539400640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25340d0d40 0x7f253415fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.850+0000 7f2539400640 1 -- 172.21.15.62:0/941509589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2528009460 con 0x7f253415ca00 2024-03-23T11:28:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.851+0000 7f2539400640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f253415ca00 0x7f25341f56e0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f252c00c780 tx=0x7f252c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.851+0000 7f2530c00640 1 -- 172.21.15.62:0/941509589 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c003d80 con 0x7f253415ca00 2024-03-23T11:28:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.851+0000 7f2530c00640 1 -- 172.21.15.62:0/941509589 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252c00ce30 con 0x7f253415ca00 2024-03-23T11:28:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.851+0000 7f2530c00640 1 -- 172.21.15.62:0/941509589 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c005290 con 0x7f253415ca00 2024-03-23T11:28:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.851+0000 7f2539e00640 1 -- 172.21.15.62:0/941509589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25340b7ec0 con 0x7f253415ca00 2024-03-23T11:28:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.851+0000 7f2539e00640 1 -- 172.21.15.62:0/941509589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253406c030 con 0x7f253415ca00 2024-03-23T11:28:56.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.852+0000 7f2539e00640 1 -- 172.21.15.62:0/941509589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25340672c0 con 0x7f253415ca00 2024-03-23T11:28:56.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.853+0000 7f2530c00640 1 -- 172.21.15.62:0/941509589 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f252c0053f0 con 0x7f253415ca00 2024-03-23T11:28:56.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.854+0000 7f2530c00640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f251806a7d0 0x7f251806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.854+0000 7f2530c00640 1 -- 172.21.15.62:0/941509589 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f252c088320 con 0x7f253415ca00 2024-03-23T11:28:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.855+0000 7f2538a00640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f251806a7d0 0x7f251806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.856+0000 7f2538a00640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f251806a7d0 0x7f251806cc90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2528035000 tx=0x7f2528008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:56.858+0000 7f2530c00640 1 -- 172.21.15.62:0/941509589 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f252c053660 con 0x7f253415ca00 2024-03-23T11:28:57.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.055+0000 7f2539e00640 1 -- 172.21.15.62:0/941509589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f253415cde0 con 0x7f253415ca00 2024-03-23T11:28:57.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.056+0000 7f2530c00640 1 -- 172.21.15.62:0/941509589 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f252c056c80 con 0x7f253415ca00 2024-03-23T11:28:57.047 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:28:57.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 -- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f251806a7d0 msgr2=0x7f251806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:57.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f251806a7d0 0x7f251806cc90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2528035000 tx=0x7f2528008030 comp rx=0 tx=0).stop 2024-03-23T11:28:57.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 -- 172.21.15.62:0/941509589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f253415ca00 msgr2=0x7f25341f56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:57.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f253415ca00 0x7f25341f56e0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f252c00c780 tx=0x7f252c00cc40 comp rx=0 tx=0).stop 2024-03-23T11:28:57.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 -- 172.21.15.62:0/941509589 shutdown_connections 2024-03-23T11:28:57.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f251806a7d0 0x7f251806cc90 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f253415ca00 0x7f25341f56e0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25340d0d40 0x7f253415fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 --2- 172.21.15.62:0/941509589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25340b24e0 0x7f253415f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 -- 172.21.15.62:0/941509589 >> 172.21.15.62:0/941509589 conn(0x7f253409ba10 msgr2=0x7f253406ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:57.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 -- 172.21.15.62:0/941509589 shutdown_connections 2024-03-23T11:28:57.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.059+0000 7f2539e00640 1 -- 172.21.15.62:0/941509589 wait complete. 2024-03-23T11:28:57.070 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:28:57.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.300+0000 7f9edec00640 1 Processor -- start 2024-03-23T11:28:57.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.301+0000 7f9edec00640 1 -- start start 2024-03-23T11:28:57.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.302+0000 7f9edec00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9ed80d13b0 0x7f9ed80cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:57.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.302+0000 7f9edec00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9ed80cd260 0x7f9ed80b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:57.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.302+0000 7f9edec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed80d0810 con 0x7f9ed815c960 2024-03-23T11:28:57.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.302+0000 7f9edec00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed80cd9d0 con 0x7f9ed80d13b0 2024-03-23T11:28:57.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.302+0000 7f9edec00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed80cdcb0 con 0x7f9ed80cd260 2024-03-23T11:28:57.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.303+0000 7f9edce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9ed80d13b0 0x7f9ed80cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:57.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.303+0000 7f9ede200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9ed80cd260 0x7f9ed80b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:57.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.303+0000 7f9edce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9ed80d13b0 0x7f9ed80cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41436/0 (socket says 172.21.15.62:41436) 2024-03-23T11:28:57.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.303+0000 7f9edce00640 1 -- 172.21.15.62:0/2611967986 learned_addr learned my addr 172.21.15.62:0/2611967986 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:57.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.303+0000 7f9edce00640 1 -- 172.21.15.62:0/2611967986 >> v2:172.21.15.120:3300/0 conn(0x7f9ed80cd260 msgr2=0x7f9ed80b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:57.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.303+0000 7f9edce00640 1 --2- 172.21.15.62:0/2611967986 >> v2:172.21.15.120:3300/0 conn(0x7f9ed80cd260 0x7f9ed80b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.303+0000 7f9edce00640 1 -- 172.21.15.62:0/2611967986 >> v1:172.21.15.120:6789/0 conn(0x7f9ed815c960 legacy=0x7f9ed80d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:57.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.304+0000 7f9edce00640 1 -- 172.21.15.62:0/2611967986 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed80cdf90 con 0x7f9ed80d13b0 2024-03-23T11:28:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.305+0000 7f9edce00640 1 --2- 172.21.15.62:0/2611967986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ed80d13b0 0x7f9ed80cf790 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80b1a80 tx=0x7f9ec402c060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea832279e49027f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.305+0000 7f9ed7e00640 1 -- 172.21.15.62:0/2611967986 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec402cc00 con 0x7f9ed80d13b0 2024-03-23T11:28:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.305+0000 7f9ed7e00640 1 -- 172.21.15.62:0/2611967986 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec402cd60 con 0x7f9ed80d13b0 2024-03-23T11:28:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.305+0000 7f9ed7e00640 1 -- 172.21.15.62:0/2611967986 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4034610 con 0x7f9ed80d13b0 2024-03-23T11:28:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.306+0000 7f9edec00640 1 -- 172.21.15.62:0/2611967986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ed80d13b0 msgr2=0x7f9ed80cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.306+0000 7f9edec00640 1 --2- 172.21.15.62:0/2611967986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ed80d13b0 0x7f9ed80cf790 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80b1a80 tx=0x7f9ec402c060 comp rx=0 tx=0).stop 2024-03-23T11:28:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.306+0000 7f9edec00640 1 -- 172.21.15.62:0/2611967986 shutdown_connections 2024-03-23T11:28:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.306+0000 7f9edec00640 1 --2- 172.21.15.62:0/2611967986 >> v2:172.21.15.120:3300/0 conn(0x7f9ed80cd260 0x7f9ed80b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.306+0000 7f9edec00640 1 --2- 172.21.15.62:0/2611967986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ed80d13b0 0x7f9ed80cf790 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.306+0000 7f9edec00640 1 -- 172.21.15.62:0/2611967986 >> 172.21.15.62:0/2611967986 conn(0x7f9ed8089f60 msgr2=0x7f9ed80d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.306+0000 7f9edec00640 1 -- 172.21.15.62:0/2611967986 shutdown_connections 2024-03-23T11:28:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.306+0000 7f9edec00640 1 -- 172.21.15.62:0/2611967986 wait complete. 2024-03-23T11:28:57.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.308+0000 7f9edec00640 1 Processor -- start 2024-03-23T11:28:57.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.308+0000 7f9edec00640 1 -- start start 2024-03-23T11:28:57.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.309+0000 7f9edec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ed80cd260 0x7f9ed80cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:57.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.309+0000 7f9edec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ed80d13b0 0x7f9ed80c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:57.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.309+0000 7f9edec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ed815c960 0x7f9ed8153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:57.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.309+0000 7f9edec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed815e2b0 con 0x7f9ed80cd260 2024-03-23T11:28:57.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.309+0000 7f9edec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed80ca140 con 0x7f9ed80d13b0 2024-03-23T11:28:57.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.309+0000 7f9edec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed81f5e80 con 0x7f9ed815c960 2024-03-23T11:28:57.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.310+0000 7f9edd800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ed80cd260 0x7f9ed80cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.310+0000 7f9edd800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ed80cd260 0x7f9ed80cc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41442/0 (socket says 172.21.15.62:41442) 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.310+0000 7f9edd800640 1 -- 172.21.15.62:0/2246627659 learned_addr learned my addr 172.21.15.62:0/2246627659 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.310+0000 7f9ede200640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ed815c960 0x7f9ed8153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.310+0000 7f9edce00640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ed80d13b0 0x7f9ed80c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.310+0000 7f9edd800640 1 -- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ed815c960 msgr2=0x7f9ed8153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.310+0000 7f9edd800640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ed815c960 0x7f9ed8153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.310+0000 7f9edd800640 1 -- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ed80d13b0 msgr2=0x7f9ed80c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.310+0000 7f9edd800640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ed80d13b0 0x7f9ed80c74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.310+0000 7f9edd800640 1 -- 172.21.15.62:0/2246627659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec4009b30 con 0x7f9ed80cd260 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.310+0000 7f9edd800640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ed80cd260 0x7f9ed80cc350 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f9ec800c400 tx=0x7f9ec800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.311+0000 7f9ed5600640 1 -- 172.21.15.62:0/2246627659 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec80052d0 con 0x7f9ed80cd260 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.311+0000 7f9ed5600640 1 -- 172.21.15.62:0/2246627659 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec8005430 con 0x7f9ed80cd260 2024-03-23T11:28:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.311+0000 7f9ed5600640 1 -- 172.21.15.62:0/2246627659 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8003830 con 0x7f9ed80cd260 2024-03-23T11:28:57.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.311+0000 7f9edec00640 1 -- 172.21.15.62:0/2246627659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed81f6160 con 0x7f9ed80cd260 2024-03-23T11:28:57.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.311+0000 7f9edec00640 1 -- 172.21.15.62:0/2246627659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed81f6720 con 0x7f9ed80cd260 2024-03-23T11:28:57.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.312+0000 7f9edec00640 1 -- 172.21.15.62:0/2246627659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed8066bb0 con 0x7f9ed80cd260 2024-03-23T11:28:57.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.313+0000 7f9ed5600640 1 -- 172.21.15.62:0/2246627659 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9ec8003990 con 0x7f9ed80cd260 2024-03-23T11:28:57.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.314+0000 7f9ed5600640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9eb806a7d0 0x7f9eb806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:57.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.314+0000 7f9ed5600640 1 -- 172.21.15.62:0/2246627659 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f9ec80882d0 con 0x7f9ed80cd260 2024-03-23T11:28:57.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.315+0000 7f9edce00640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9eb806a7d0 0x7f9eb806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:57.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.316+0000 7f9edce00640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9eb806a7d0 0x7f9eb806cc90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4035000 tx=0x7f9ec4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:57.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.318+0000 7f9ed5600640 1 -- 172.21.15.62:0/2246627659 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ec8053580 con 0x7f9ed80cd260 2024-03-23T11:28:57.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.534+0000 7f9edec00640 1 -- 172.21.15.62:0/2246627659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9ed80d0810 con 0x7f9ed80cd260 2024-03-23T11:28:57.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.534+0000 7f9ed5600640 1 -- 172.21.15.62:0/2246627659 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v93) v1 ==== 80+0+244 (secure 0 0 0) 0x7f9ed80d0810 con 0x7f9ed80cd260 2024-03-23T11:28:57.526 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6832/2573443114 2024-03-24T11:28:13.436002+0000 2024-03-23T11:28:57.526 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6833/2573443114 2024-03-24T11:28:13.436002+0000 2024-03-23T11:28:57.526 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6833/3056962984 2024-03-24T11:26:47.481437+0000 2024-03-23T11:28:57.526 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6832/3056962984 2024-03-24T11:26:47.481437+0000 2024-03-23T11:28:57.527 INFO:teuthology.orchestra.run.smithi062.stderr:listed 4 entries 2024-03-23T11:28:57.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.537+0000 7f9edec00640 1 -- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9eb806a7d0 msgr2=0x7f9eb806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:57.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.537+0000 7f9edec00640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9eb806a7d0 0x7f9eb806cc90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4035000 tx=0x7f9ec4030000 comp rx=0 tx=0).stop 2024-03-23T11:28:57.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.537+0000 7f9edec00640 1 -- 172.21.15.62:0/2246627659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ed80cd260 msgr2=0x7f9ed80cc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:57.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.537+0000 7f9edec00640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ed80cd260 0x7f9ed80cc350 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f9ec800c400 tx=0x7f9ec800c8c0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.538+0000 7f9edec00640 1 -- 172.21.15.62:0/2246627659 shutdown_connections 2024-03-23T11:28:57.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.538+0000 7f9edec00640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9eb806a7d0 0x7f9eb806cc90 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.538+0000 7f9edec00640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ed815c960 0x7f9ed8153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.538+0000 7f9edec00640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ed80d13b0 0x7f9ed80c74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.538+0000 7f9edec00640 1 --2- 172.21.15.62:0/2246627659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ed80cd260 0x7f9ed80cc350 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:57.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.538+0000 7f9edec00640 1 -- 172.21.15.62:0/2246627659 >> 172.21.15.62:0/2246627659 conn(0x7f9ed8089f60 msgr2=0x7f9ed815a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:57.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.538+0000 7f9edec00640 1 -- 172.21.15.62:0/2246627659 shutdown_connections 2024-03-23T11:28:57.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:57.538+0000 7f9edec00640 1 -- 172.21.15.62:0/2246627659 wait complete. 2024-03-23T11:28:57.550 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:28:57.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:57.550 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:28:57.642 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[69642]: fuse finished with error 0 and tester_r 0 2024-03-23T11:28:57.649 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:28:57.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:57.650 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:28:57.828 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:28:57.829 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:28:57.887 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:28:57.887 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:28:57.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:57.887 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T11:28:57.941 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-23T11:28:57.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:28:57.943 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:28:57.943 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:28:58.003 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:28:58.004 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:28:58.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:58.004 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T11:28:58.060 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-23T11:28:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:28:58.061 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:28:58.061 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:28:58.117 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:28:58.117 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:28:58.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:28:58.118 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T11:28:58.167 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-23T11:28:58.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:28:58.168 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:28:58.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.412+0000 7f1c43000640 1 Processor -- start 2024-03-23T11:28:58.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.412+0000 7f1c43000640 1 -- start start 2024-03-23T11:28:58.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.414+0000 7f1c43000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1c3c06a370 0x7f1c3c161100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:58.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.414+0000 7f1c43000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c0cf2a0 con 0x7f1c3c0cb8c0 2024-03-23T11:28:58.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.414+0000 7f1c43000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c0cefc0 con 0x7f1c3c0b0980 2024-03-23T11:28:58.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.414+0000 7f1c43000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c3c160620 con 0x7f1c3c06a370 2024-03-23T11:28:58.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.414+0000 7f1c41200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1c3c06a370 0x7f1c3c161100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:58.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.414+0000 7f1c42600640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1c3c0cb8c0 0x7f1c3c15c210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42920/0 (socket says 172.21.15.62:42920) 2024-03-23T11:28:58.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.414+0000 7f1c42600640 1 -- 172.21.15.62:0/3771498339 learned_addr learned my addr 172.21.15.62:0/3771498339 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:58.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.415+0000 7f1c41200640 1 -- 172.21.15.62:0/3771498339 >> v1:172.21.15.120:6789/0 conn(0x7f1c3c0b0980 legacy=0x7f1c3c069c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:58.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.415+0000 7f1c41200640 1 -- 172.21.15.62:0/3771498339 >> v1:172.21.15.62:6789/0 conn(0x7f1c3c0cb8c0 legacy=0x7f1c3c15c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:58.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.415+0000 7f1c41200640 1 -- 172.21.15.62:0/3771498339 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c160900 con 0x7f1c3c06a370 2024-03-23T11:28:58.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.416+0000 7f1c41200640 1 --2- 172.21.15.62:0/3771498339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c3c06a370 0x7f1c3c161100 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f1c280097a0 tx=0x7f1c2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe31cf66f79ebb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:58.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.416+0000 7f1c3be00640 1 -- 172.21.15.62:0/3771498339 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28031830 con 0x7f1c3c06a370 2024-03-23T11:28:58.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.416+0000 7f1c3be00640 1 -- 172.21.15.62:0/3771498339 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c28031990 con 0x7f1c3c06a370 2024-03-23T11:28:58.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.417+0000 7f1c3be00640 1 -- 172.21.15.62:0/3771498339 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28031d20 con 0x7f1c3c06a370 2024-03-23T11:28:58.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.417+0000 7f1c43000640 1 -- 172.21.15.62:0/3771498339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c3c06a370 msgr2=0x7f1c3c161100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:58.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.417+0000 7f1c43000640 1 --2- 172.21.15.62:0/3771498339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c3c06a370 0x7f1c3c161100 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f1c280097a0 tx=0x7f1c2802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.417+0000 7f1c43000640 1 -- 172.21.15.62:0/3771498339 shutdown_connections 2024-03-23T11:28:58.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.417+0000 7f1c43000640 1 --2- 172.21.15.62:0/3771498339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c3c06a370 0x7f1c3c161100 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.417+0000 7f1c43000640 1 -- 172.21.15.62:0/3771498339 >> 172.21.15.62:0/3771498339 conn(0x7f1c3c0989d0 msgr2=0x7f1c3c0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:58.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.417+0000 7f1c43000640 1 -- 172.21.15.62:0/3771498339 shutdown_connections 2024-03-23T11:28:58.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.417+0000 7f1c43000640 1 -- 172.21.15.62:0/3771498339 wait complete. 2024-03-23T11:28:58.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.419+0000 7f1c43000640 1 Processor -- start 2024-03-23T11:28:58.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.419+0000 7f1c43000640 1 -- start start 2024-03-23T11:28:58.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.420+0000 7f1c43000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c3c06a370 0x7f1c3c1561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:58.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.420+0000 7f1c43000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c3c0b0980 0x7f1c3c0c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:58.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.420+0000 7f1c43000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c3c0cb8c0 0x7f1c3c15e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:58.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.420+0000 7f1c43000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c0c31d0 con 0x7f1c3c0b0980 2024-03-23T11:28:58.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.420+0000 7f1c43000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c0c3470 con 0x7f1c3c0cb8c0 2024-03-23T11:28:58.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.420+0000 7f1c43000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c0c3750 con 0x7f1c3c06a370 2024-03-23T11:28:58.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.420+0000 7f1c41200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c3c0b0980 0x7f1c3c0c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:58.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.421+0000 7f1c41200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c3c0b0980 0x7f1c3c0c4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41462/0 (socket says 172.21.15.62:41462) 2024-03-23T11:28:58.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.421+0000 7f1c41200640 1 -- 172.21.15.62:0/2109057252 learned_addr learned my addr 172.21.15.62:0/2109057252 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:58.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.421+0000 7f1c41c00640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c3c06a370 0x7f1c3c1561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:58.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.421+0000 7f1c42600640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c3c0cb8c0 0x7f1c3c15e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.421+0000 7f1c41200640 1 -- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c3c06a370 msgr2=0x7f1c3c1561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.421+0000 7f1c41200640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c3c06a370 0x7f1c3c1561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.421+0000 7f1c41200640 1 -- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c3c0cb8c0 msgr2=0x7f1c3c15e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.421+0000 7f1c41200640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c3c0cb8c0 0x7f1c3c15e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.421+0000 7f1c41200640 1 -- 172.21.15.62:0/2109057252 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c28009400 con 0x7f1c3c0b0980 2024-03-23T11:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.422+0000 7f1c41200640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c3c0b0980 0x7f1c3c0c4cb0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f1c280098c0 tx=0x7f1c280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.422+0000 7f1c39600640 1 -- 172.21.15.62:0/2109057252 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2803ba70 con 0x7f1c3c0b0980 2024-03-23T11:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.422+0000 7f1c39600640 1 -- 172.21.15.62:0/2109057252 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2803bbf0 con 0x7f1c3c0b0980 2024-03-23T11:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.422+0000 7f1c39600640 1 -- 172.21.15.62:0/2109057252 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c280453f0 con 0x7f1c3c0b0980 2024-03-23T11:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.422+0000 7f1c43000640 1 -- 172.21.15.62:0/2109057252 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3c0c3a30 con 0x7f1c3c0b0980 2024-03-23T11:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.422+0000 7f1c43000640 1 -- 172.21.15.62:0/2109057252 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3c1591b0 con 0x7f1c3c0b0980 2024-03-23T11:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.423+0000 7f1c43000640 1 -- 172.21.15.62:0/2109057252 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3c0c3f50 con 0x7f1c3c0b0980 2024-03-23T11:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.424+0000 7f1c39600640 1 -- 172.21.15.62:0/2109057252 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f1c2804e450 con 0x7f1c3c0b0980 2024-03-23T11:28:58.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.425+0000 7f1c39600640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1c1c06a7a0 0x7f1c1c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:58.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.425+0000 7f1c39600640 1 -- 172.21.15.62:0/2109057252 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f1c280ab160 con 0x7f1c3c0b0980 2024-03-23T11:28:58.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.426+0000 7f1c41c00640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1c1c06a7a0 0x7f1c1c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:58.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.427+0000 7f1c41c00640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1c1c06a7a0 0x7f1c1c06cc60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1c30004ea0 tx=0x7f1c30009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:58.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.430+0000 7f1c39600640 1 -- 172.21.15.62:0/2109057252 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1c3c0c3f50 con 0x7f1c3c0b0980 2024-03-23T11:28:58.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.665+0000 7f1c43000640 1 -- 172.21.15.62:0/2109057252 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c3c0c3f50 con 0x7f1c3c0b0980 2024-03-23T11:28:58.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.666+0000 7f1c39600640 1 -- 172.21.15.62:0/2109057252 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f1c3c0c3f50 con 0x7f1c3c0b0980 2024-03-23T11:28:58.658 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:58.658 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":70,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:28:18.362214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6833/1594260761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1594260761},{"type":"v1","addr":"172.21.15.62:6833","nonce":1594260761}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:28:58.658 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 70 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.669+0000 7f1c43000640 1 -- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1c1c06a7a0 msgr2=0x7f1c1c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.669+0000 7f1c43000640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1c1c06a7a0 0x7f1c1c06cc60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1c30004ea0 tx=0x7f1c30009040 comp rx=0 tx=0).stop 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.669+0000 7f1c43000640 1 -- 172.21.15.62:0/2109057252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c3c0b0980 msgr2=0x7f1c3c0c4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.669+0000 7f1c43000640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c3c0b0980 0x7f1c3c0c4cb0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f1c280098c0 tx=0x7f1c280046c0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.669+0000 7f1c43000640 1 -- 172.21.15.62:0/2109057252 shutdown_connections 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.669+0000 7f1c43000640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1c1c06a7a0 0x7f1c1c06cc60 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.669+0000 7f1c43000640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c3c0b0980 0x7f1c3c0c4cb0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.669+0000 7f1c43000640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c3c0cb8c0 0x7f1c3c15e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.669+0000 7f1c43000640 1 --2- 172.21.15.62:0/2109057252 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c3c06a370 0x7f1c3c1561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.669+0000 7f1c43000640 1 -- 172.21.15.62:0/2109057252 >> 172.21.15.62:0/2109057252 conn(0x7f1c3c0989d0 msgr2=0x7f1c3c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.669+0000 7f1c43000640 1 -- 172.21.15.62:0/2109057252 shutdown_connections 2024-03-23T11:28:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.670+0000 7f1c43000640 1 -- 172.21.15.62:0/2109057252 wait complete. 2024-03-23T11:28:58.679 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:28:58.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.902+0000 7f52e9a00640 1 Processor -- start 2024-03-23T11:28:58.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.902+0000 7f52e9a00640 1 -- start start 2024-03-23T11:28:58.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.904+0000 7f52e9a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e41571e0 0x7f52e40ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:58.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.904+0000 7f52e9a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f52e41613a0 0x7f52e415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:58.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.904+0000 7f52e9a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e406cf40 con 0x7f52e40ade70 2024-03-23T11:28:58.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.904+0000 7f52e9a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e41575c0 con 0x7f52e41613a0 2024-03-23T11:28:58.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.904+0000 7f52e9a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e406c3a0 con 0x7f52e41571e0 2024-03-23T11:28:58.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.904+0000 7f52e3400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f52e41613a0 0x7f52e415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:58.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.905+0000 7f52e3400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f52e41613a0 0x7f52e415e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41472/0 (socket says 172.21.15.62:41472) 2024-03-23T11:28:58.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.905+0000 7f52e3400640 1 -- 172.21.15.62:0/2184721636 learned_addr learned my addr 172.21.15.62:0/2184721636 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:58.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.905+0000 7f52e2a00640 1 --2- 172.21.15.62:0/2184721636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e41571e0 0x7f52e40ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:58.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.905+0000 7f52e3400640 1 -- 172.21.15.62:0/2184721636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e41571e0 msgr2=0x7f52e40ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:58.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.905+0000 7f52e3400640 1 --2- 172.21.15.62:0/2184721636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e41571e0 0x7f52e40ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.905+0000 7f52e3400640 1 -- 172.21.15.62:0/2184721636 >> v1:172.21.15.120:6789/0 conn(0x7f52e40ade70 legacy=0x7f52e406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:58.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.905+0000 7f52e3400640 1 -- 172.21.15.62:0/2184721636 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e406c610 con 0x7f52e41613a0 2024-03-23T11:28:58.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.906+0000 7f52e3400640 1 --2- 172.21.15.62:0/2184721636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e41613a0 0x7f52e415e280 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009e50 tx=0x7f52dc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb4be9569c95ca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:58.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.907+0000 7f52e1600640 1 -- 172.21.15.62:0/2184721636 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc02ca80 con 0x7f52e41613a0 2024-03-23T11:28:58.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.907+0000 7f52e1600640 1 -- 172.21.15.62:0/2184721636 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52dc02cbe0 con 0x7f52e41613a0 2024-03-23T11:28:58.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.907+0000 7f52e1600640 1 -- 172.21.15.62:0/2184721636 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc0343f0 con 0x7f52e41613a0 2024-03-23T11:28:58.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.907+0000 7f52e9a00640 1 -- 172.21.15.62:0/2184721636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e41613a0 msgr2=0x7f52e415e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:58.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.907+0000 7f52e9a00640 1 --2- 172.21.15.62:0/2184721636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e41613a0 0x7f52e415e280 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009e50 tx=0x7f52dc02bee0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.908+0000 7f52e9a00640 1 -- 172.21.15.62:0/2184721636 shutdown_connections 2024-03-23T11:28:58.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.908+0000 7f52e9a00640 1 --2- 172.21.15.62:0/2184721636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e41613a0 0x7f52e415e280 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.908+0000 7f52e9a00640 1 --2- 172.21.15.62:0/2184721636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e41571e0 0x7f52e40ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.908+0000 7f52e9a00640 1 -- 172.21.15.62:0/2184721636 >> 172.21.15.62:0/2184721636 conn(0x7f52e40b02e0 msgr2=0x7f52e40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:58.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.908+0000 7f52e9a00640 1 -- 172.21.15.62:0/2184721636 shutdown_connections 2024-03-23T11:28:58.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.908+0000 7f52e9a00640 1 -- 172.21.15.62:0/2184721636 wait complete. 2024-03-23T11:28:58.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.910+0000 7f52e9a00640 1 Processor -- start 2024-03-23T11:28:58.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.910+0000 7f52e9a00640 1 -- start start 2024-03-23T11:28:58.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.911+0000 7f52e9a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e40ade70 0x7f52e40cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:58.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.911+0000 7f52e9a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e41571e0 0x7f52e41388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:58.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.911+0000 7f52e9a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e41613a0 0x7f52e4138e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:58.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.911+0000 7f52e9a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e4158190 con 0x7f52e41613a0 2024-03-23T11:28:58.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.911+0000 7f52e9a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e4139360 con 0x7f52e41571e0 2024-03-23T11:28:58.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.911+0000 7f52e9a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e41f6a10 con 0x7f52e40ade70 2024-03-23T11:28:58.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.911+0000 7f52e3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e41613a0 0x7f52e4138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:58.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.911+0000 7f52e3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e41613a0 0x7f52e4138e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41476/0 (socket says 172.21.15.62:41476) 2024-03-23T11:28:58.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.911+0000 7f52e3400640 1 -- 172.21.15.62:0/989518107 learned_addr learned my addr 172.21.15.62:0/989518107 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:58.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.911+0000 7f52e2a00640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e40ade70 0x7f52e40cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:58.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.911+0000 7f52e2000640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e41571e0 0x7f52e41388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:58.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.912+0000 7f52e3400640 1 -- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e40ade70 msgr2=0x7f52e40cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:58.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.912+0000 7f52e3400640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e40ade70 0x7f52e40cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.912+0000 7f52e3400640 1 -- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e41571e0 msgr2=0x7f52e41388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:58.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.912+0000 7f52e3400640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e41571e0 0x7f52e41388e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:58.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.912+0000 7f52e3400640 1 -- 172.21.15.62:0/989518107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc009b30 con 0x7f52e41613a0 2024-03-23T11:28:58.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.912+0000 7f52e3400640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e41613a0 0x7f52e4138e20 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f52dc036040 tx=0x7f52dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:58.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.912+0000 7f52daa00640 1 -- 172.21.15.62:0/989518107 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc009510 con 0x7f52e41613a0 2024-03-23T11:28:58.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.912+0000 7f52daa00640 1 -- 172.21.15.62:0/989518107 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52dc03e450 con 0x7f52e41613a0 2024-03-23T11:28:58.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.912+0000 7f52e9a00640 1 -- 172.21.15.62:0/989518107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e41577f0 con 0x7f52e41613a0 2024-03-23T11:28:58.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.912+0000 7f52daa00640 1 -- 172.21.15.62:0/989518107 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc03e800 con 0x7f52e41613a0 2024-03-23T11:28:58.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.912+0000 7f52e9a00640 1 -- 172.21.15.62:0/989518107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e4157db0 con 0x7f52e41613a0 2024-03-23T11:28:58.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.914+0000 7f52e9a00640 1 -- 172.21.15.62:0/989518107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e41582d0 con 0x7f52e41613a0 2024-03-23T11:28:58.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.915+0000 7f52daa00640 1 -- 172.21.15.62:0/989518107 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f52dc02cd50 con 0x7f52e41613a0 2024-03-23T11:28:58.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.915+0000 7f52daa00640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52c406a7f0 0x7f52c406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:58.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.916+0000 7f52daa00640 1 -- 172.21.15.62:0/989518107 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f52dc0abbb0 con 0x7f52e41613a0 2024-03-23T11:28:58.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.916+0000 7f52e2a00640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52c406a7f0 0x7f52c406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:58.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.917+0000 7f52e2a00640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52c406a7f0 0x7f52c406ccb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f52d4006fd0 tx=0x7f52d4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:58.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:58.920+0000 7f52daa00640 1 -- 172.21.15.62:0/989518107 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f52e41582d0 con 0x7f52e41613a0 2024-03-23T11:28:59.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.119+0000 7f52e9a00640 1 -- 172.21.15.62:0/989518107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52e41582d0 con 0x7f52e41613a0 2024-03-23T11:28:59.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.121+0000 7f52daa00640 1 -- 172.21.15.62:0/989518107 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16401 (secure 0 0 0) 0x7f52e41582d0 con 0x7f52e41613a0 2024-03-23T11:28:59.113 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:28:59.113 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":93,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:28:13.436047+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:26:22.794149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-23T11:26:24.150997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6832/2573443114":"2024-03-24T11:28:13.436002+0000","172.21.15.62:6833/2573443114":"2024-03-24T11:28:13.436002+0000","172.21.15.62:6833/3056962984":"2024-03-24T11:26:47.481437+0000","172.21.15.62:6832/3056962984":"2024-03-24T11:26:47.481437+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:28:59.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 -- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52c406a7f0 msgr2=0x7f52c406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:59.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52c406a7f0 0x7f52c406ccb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f52d4006fd0 tx=0x7f52d4008040 comp rx=0 tx=0).stop 2024-03-23T11:28:59.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 -- 172.21.15.62:0/989518107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e41613a0 msgr2=0x7f52e4138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:59.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e41613a0 0x7f52e4138e20 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f52dc036040 tx=0x7f52dc004a40 comp rx=0 tx=0).stop 2024-03-23T11:28:59.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 -- 172.21.15.62:0/989518107 shutdown_connections 2024-03-23T11:28:59.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52c406a7f0 0x7f52c406ccb0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e41613a0 0x7f52e4138e20 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e41571e0 0x7f52e41388e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 --2- 172.21.15.62:0/989518107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e40ade70 0x7f52e40cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 -- 172.21.15.62:0/989518107 >> 172.21.15.62:0/989518107 conn(0x7f52e40b02e0 msgr2=0x7f52e40ad400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:59.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 -- 172.21.15.62:0/989518107 shutdown_connections 2024-03-23T11:28:59.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.124+0000 7f52e9a00640 1 -- 172.21.15.62:0/989518107 wait complete. 2024-03-23T11:28:59.135 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:26:22.794149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:26:24.150997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:28:59.135 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:28:59.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.372+0000 7fcd99000640 1 Processor -- start 2024-03-23T11:28:59.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.372+0000 7fcd99000640 1 -- start start 2024-03-23T11:28:59.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.373+0000 7fcd99000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcd9415fcf0 0x7fcd940d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.373+0000 7fcd99000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcd94160420 0x7fcd941608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.373+0000 7fcd99000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd940d3540 con 0x7fcd940d06d0 2024-03-23T11:28:59.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.373+0000 7fcd99000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd94160df0 con 0x7fcd94160420 2024-03-23T11:28:59.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.373+0000 7fcd99000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd940ce510 con 0x7fcd9415fcf0 2024-03-23T11:28:59.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.373+0000 7fcd8aa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcd94160420 0x7fcd941608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:59.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.373+0000 7fcd93e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fcd940d06d0 0x7fcd940d0ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42930/0 (socket says 172.21.15.62:42930) 2024-03-23T11:28:59.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.373+0000 7fcd93e00640 1 -- 172.21.15.62:0/326901306 learned_addr learned my addr 172.21.15.62:0/326901306 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:28:59.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.374+0000 7fcd93400640 1 --2- 172.21.15.62:0/326901306 >> v2:172.21.15.120:3300/0 conn(0x7fcd9415fcf0 0x7fcd940d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:59.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.374+0000 7fcd8aa00640 1 -- 172.21.15.62:0/326901306 >> v2:172.21.15.120:3300/0 conn(0x7fcd9415fcf0 msgr2=0x7fcd940d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:59.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.374+0000 7fcd8aa00640 1 --2- 172.21.15.62:0/326901306 >> v2:172.21.15.120:3300/0 conn(0x7fcd9415fcf0 0x7fcd940d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.374+0000 7fcd8aa00640 1 -- 172.21.15.62:0/326901306 >> v1:172.21.15.62:6789/0 conn(0x7fcd940d06d0 legacy=0x7fcd940d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:59.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.374+0000 7fcd8aa00640 1 -- 172.21.15.62:0/326901306 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940cdc70 con 0x7fcd94160420 2024-03-23T11:28:59.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.374+0000 7fcd92a00640 1 -- 172.21.15.62:0/326901306 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300827634 0 0) 0x7fcd940d3540 con 0x7fcd940d06d0 2024-03-23T11:28:59.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.375+0000 7fcd8aa00640 1 --2- 172.21.15.62:0/326901306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd94160420 0x7fcd941608b0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fcd94066480 tx=0x7fcd7c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7edf954b163ff8d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:59.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.375+0000 7fcd92a00640 1 -- 172.21.15.62:0/326901306 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c02e070 con 0x7fcd94160420 2024-03-23T11:28:59.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.375+0000 7fcd92a00640 1 -- 172.21.15.62:0/326901306 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c0315a0 con 0x7fcd94160420 2024-03-23T11:28:59.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.375+0000 7fcd92a00640 1 -- 172.21.15.62:0/326901306 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c031930 con 0x7fcd94160420 2024-03-23T11:28:59.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.376+0000 7fcd99000640 1 -- 172.21.15.62:0/326901306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd94160420 msgr2=0x7fcd941608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:59.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.376+0000 7fcd99000640 1 --2- 172.21.15.62:0/326901306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd94160420 0x7fcd941608b0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fcd94066480 tx=0x7fcd7c02b840 comp rx=0 tx=0).stop 2024-03-23T11:28:59.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.376+0000 7fcd99000640 1 -- 172.21.15.62:0/326901306 shutdown_connections 2024-03-23T11:28:59.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.376+0000 7fcd99000640 1 --2- 172.21.15.62:0/326901306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd94160420 0x7fcd941608b0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.376+0000 7fcd99000640 1 --2- 172.21.15.62:0/326901306 >> v2:172.21.15.120:3300/0 conn(0x7fcd9415fcf0 0x7fcd940d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.376+0000 7fcd99000640 1 -- 172.21.15.62:0/326901306 >> 172.21.15.62:0/326901306 conn(0x7fcd94089f60 msgr2=0x7fcd940d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:59.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.376+0000 7fcd99000640 1 -- 172.21.15.62:0/326901306 shutdown_connections 2024-03-23T11:28:59.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.376+0000 7fcd99000640 1 -- 172.21.15.62:0/326901306 wait complete. 2024-03-23T11:28:59.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.378+0000 7fcd99000640 1 Processor -- start 2024-03-23T11:28:59.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.378+0000 7fcd99000640 1 -- start start 2024-03-23T11:28:59.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.379+0000 7fcd99000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd940d06d0 0x7fcd940cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.379+0000 7fcd99000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9415fcf0 0x7fcd940c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.379+0000 7fcd99000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd94160420 0x7fcd941f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.379+0000 7fcd99000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd94155470 con 0x7fcd9415fcf0 2024-03-23T11:28:59.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.379+0000 7fcd99000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd94155a30 con 0x7fcd940d06d0 2024-03-23T11:28:59.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.379+0000 7fcd99000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940c9f70 con 0x7fcd94160420 2024-03-23T11:28:59.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.379+0000 7fcd8aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9415fcf0 0x7fcd940c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:59.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.379+0000 7fcd8aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9415fcf0 0x7fcd940c9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41488/0 (socket says 172.21.15.62:41488) 2024-03-23T11:28:59.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.379+0000 7fcd8aa00640 1 -- 172.21.15.62:0/241538699 learned_addr learned my addr 172.21.15.62:0/241538699 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:59.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.380+0000 7fcd93400640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd940d06d0 0x7fcd940cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:59.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.380+0000 7fcd8aa00640 1 -- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd94160420 msgr2=0x7fcd941f1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:59.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.380+0000 7fcd8aa00640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd94160420 0x7fcd941f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.380+0000 7fcd8aa00640 1 -- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd940d06d0 msgr2=0x7fcd940cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:59.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.380+0000 7fcd8aa00640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd940d06d0 0x7fcd940cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.380+0000 7fcd8aa00640 1 -- 172.21.15.62:0/241538699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c009400 con 0x7fcd9415fcf0 2024-03-23T11:28:59.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.380+0000 7fcd8aa00640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9415fcf0 0x7fcd940c9720 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c02be50 tx=0x7fcd7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:59.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.380+0000 7fcd8be00640 1 -- 172.21.15.62:0/241538699 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c032840 con 0x7fcd9415fcf0 2024-03-23T11:28:59.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.380+0000 7fcd8be00640 1 -- 172.21.15.62:0/241538699 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c0329a0 con 0x7fcd9415fcf0 2024-03-23T11:28:59.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.380+0000 7fcd8be00640 1 -- 172.21.15.62:0/241538699 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c032d50 con 0x7fcd9415fcf0 2024-03-23T11:28:59.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.380+0000 7fcd99000640 1 -- 172.21.15.62:0/241538699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd940ca250 con 0x7fcd9415fcf0 2024-03-23T11:28:59.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.381+0000 7fcd99000640 1 -- 172.21.15.62:0/241538699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd940c9c60 con 0x7fcd9415fcf0 2024-03-23T11:28:59.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.382+0000 7fcd99000640 1 -- 172.21.15.62:0/241538699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd94066bb0 con 0x7fcd9415fcf0 2024-03-23T11:28:59.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.383+0000 7fcd8be00640 1 -- 172.21.15.62:0/241538699 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fcd7c0453f0 con 0x7fcd9415fcf0 2024-03-23T11:28:59.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.384+0000 7fcd8be00640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd7006a7a0 0x7fcd7006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.384+0000 7fcd8be00640 1 -- 172.21.15.62:0/241538699 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fcd7c0aaed0 con 0x7fcd9415fcf0 2024-03-23T11:28:59.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.384+0000 7fcd93400640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd7006a7a0 0x7fcd7006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:59.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.385+0000 7fcd93400640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd7006a7a0 0x7fcd7006cc60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcd84006fd0 tx=0x7fcd84008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:59.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.388+0000 7fcd8be00640 1 -- 172.21.15.62:0/241538699 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd7c0762c0 con 0x7fcd9415fcf0 2024-03-23T11:28:59.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.589+0000 7fcd99000640 1 -- 172.21.15.62:0/241538699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcd940ca440 con 0x7fcd9415fcf0 2024-03-23T11:28:59.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.590+0000 7fcd8be00640 1 -- 172.21.15.62:0/241538699 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcd7c0798e0 con 0x7fcd9415fcf0 2024-03-23T11:28:59.581 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:28:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 -- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd7006a7a0 msgr2=0x7fcd7006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd7006a7a0 0x7fcd7006cc60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcd84006fd0 tx=0x7fcd84008040 comp rx=0 tx=0).stop 2024-03-23T11:28:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 -- 172.21.15.62:0/241538699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9415fcf0 msgr2=0x7fcd940c9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9415fcf0 0x7fcd940c9720 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c02be50 tx=0x7fcd7c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 -- 172.21.15.62:0/241538699 shutdown_connections 2024-03-23T11:28:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd7006a7a0 0x7fcd7006cc60 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd9415fcf0 0x7fcd940c9720 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd94160420 0x7fcd941f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 --2- 172.21.15.62:0/241538699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd940d06d0 0x7fcd940cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 -- 172.21.15.62:0/241538699 >> 172.21.15.62:0/241538699 conn(0x7fcd94089f60 msgr2=0x7fcd9415f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 -- 172.21.15.62:0/241538699 shutdown_connections 2024-03-23T11:28:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.593+0000 7fcd99000640 1 -- 172.21.15.62:0/241538699 wait complete. 2024-03-23T11:28:59.605 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:28:59.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.836+0000 7fd086000640 1 Processor -- start 2024-03-23T11:28:59.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.836+0000 7fd086000640 1 -- start start 2024-03-23T11:28:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.837+0000 7fd086000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd08015fda0 0x7fd0800d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.837+0000 7fd086000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd080160420 0x7fd0801608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.837+0000 7fd086000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0800d3540 con 0x7fd0800d06d0 2024-03-23T11:28:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.837+0000 7fd086000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd080160df0 con 0x7fd08015fda0 2024-03-23T11:28:59.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.837+0000 7fd086000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0800ce510 con 0x7fd080160420 2024-03-23T11:28:59.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.837+0000 7fd084c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd08015fda0 0x7fd0800d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:59.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.838+0000 7fd084c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd08015fda0 0x7fd0800d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41504/0 (socket says 172.21.15.62:41504) 2024-03-23T11:28:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.838+0000 7fd084c00640 1 -- 172.21.15.62:0/1661785277 learned_addr learned my addr 172.21.15.62:0/1661785277 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.838+0000 7fd07fe00640 1 --2- 172.21.15.62:0/1661785277 >> v2:172.21.15.120:3300/0 conn(0x7fd080160420 0x7fd0801608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.838+0000 7fd084c00640 1 -- 172.21.15.62:0/1661785277 >> v2:172.21.15.120:3300/0 conn(0x7fd080160420 msgr2=0x7fd0801608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.838+0000 7fd084c00640 1 --2- 172.21.15.62:0/1661785277 >> v2:172.21.15.120:3300/0 conn(0x7fd080160420 0x7fd0801608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.838+0000 7fd084c00640 1 -- 172.21.15.62:0/1661785277 >> v1:172.21.15.120:6789/0 conn(0x7fd0800d06d0 legacy=0x7fd0800d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:28:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.838+0000 7fd084c00640 1 -- 172.21.15.62:0/1661785277 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd080160180 con 0x7fd08015fda0 2024-03-23T11:28:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.839+0000 7fd084c00640 1 --2- 172.21.15.62:0/1661785277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd08015fda0 0x7fd0800d3000 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fd080161540 tx=0x7fd07002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab618d8c75b4b50 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:59.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.839+0000 7fd07f400640 1 -- 172.21.15.62:0/1661785277 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0700317e0 con 0x7fd08015fda0 2024-03-23T11:28:59.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.839+0000 7fd07f400640 1 -- 172.21.15.62:0/1661785277 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070031940 con 0x7fd08015fda0 2024-03-23T11:28:59.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.840+0000 7fd07f400640 1 -- 172.21.15.62:0/1661785277 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070031cd0 con 0x7fd08015fda0 2024-03-23T11:28:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.840+0000 7fd086000640 1 -- 172.21.15.62:0/1661785277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd08015fda0 msgr2=0x7fd0800d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.840+0000 7fd086000640 1 --2- 172.21.15.62:0/1661785277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd08015fda0 0x7fd0800d3000 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fd080161540 tx=0x7fd07002b880 comp rx=0 tx=0).stop 2024-03-23T11:28:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.840+0000 7fd086000640 1 -- 172.21.15.62:0/1661785277 shutdown_connections 2024-03-23T11:28:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.840+0000 7fd086000640 1 --2- 172.21.15.62:0/1661785277 >> v2:172.21.15.120:3300/0 conn(0x7fd080160420 0x7fd0801608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.840+0000 7fd086000640 1 --2- 172.21.15.62:0/1661785277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd08015fda0 0x7fd0800d3000 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.840+0000 7fd086000640 1 -- 172.21.15.62:0/1661785277 >> 172.21.15.62:0/1661785277 conn(0x7fd080089f60 msgr2=0x7fd0800d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:28:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.840+0000 7fd086000640 1 -- 172.21.15.62:0/1661785277 shutdown_connections 2024-03-23T11:28:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.840+0000 7fd086000640 1 -- 172.21.15.62:0/1661785277 wait complete. 2024-03-23T11:28:59.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.842+0000 7fd086000640 1 Processor -- start 2024-03-23T11:28:59.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.842+0000 7fd086000640 1 -- start start 2024-03-23T11:28:59.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.843+0000 7fd086000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0800d06d0 0x7fd0801f1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.843+0000 7fd086000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd08015fda0 0x7fd080158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.843+0000 7fd086000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd080160420 0x7fd08006b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.843+0000 7fd086000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0801f20a0 con 0x7fd08015fda0 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.843+0000 7fd086000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08015b7d0 con 0x7fd0800d06d0 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.843+0000 7fd086000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800c6f70 con 0x7fd080160420 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.843+0000 7fd07fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd08015fda0 0x7fd080158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.844+0000 7fd07fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd08015fda0 0x7fd080158410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41510/0 (socket says 172.21.15.62:41510) 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.844+0000 7fd07fe00640 1 -- 172.21.15.62:0/2341818724 learned_addr learned my addr 172.21.15.62:0/2341818724 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.844+0000 7fd084c00640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0800d06d0 0x7fd0801f1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.844+0000 7fd085600640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd080160420 0x7fd08006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.844+0000 7fd07fe00640 1 -- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd080160420 msgr2=0x7fd08006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.844+0000 7fd07fe00640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd080160420 0x7fd08006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.844+0000 7fd07fe00640 1 -- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0800d06d0 msgr2=0x7fd0801f1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.844+0000 7fd07fe00640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0800d06d0 0x7fd0801f1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:28:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.844+0000 7fd07fe00640 1 -- 172.21.15.62:0/2341818724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0700094d0 con 0x7fd08015fda0 2024-03-23T11:28:59.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.844+0000 7fd07fe00640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd08015fda0 0x7fd080158410 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fd06800d800 tx=0x7fd06800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:59.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.845+0000 7fd07cc00640 1 -- 172.21.15.62:0/2341818724 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06800b6c0 con 0x7fd08015fda0 2024-03-23T11:28:59.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.845+0000 7fd07cc00640 1 -- 172.21.15.62:0/2341818724 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06800b820 con 0x7fd08015fda0 2024-03-23T11:28:59.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.845+0000 7fd07cc00640 1 -- 172.21.15.62:0/2341818724 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06800bbb0 con 0x7fd08015fda0 2024-03-23T11:28:59.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.845+0000 7fd086000640 1 -- 172.21.15.62:0/2341818724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd08015af30 con 0x7fd08015fda0 2024-03-23T11:28:59.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.845+0000 7fd086000640 1 -- 172.21.15.62:0/2341818724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08015b4f0 con 0x7fd08015fda0 2024-03-23T11:28:59.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.846+0000 7fd086000640 1 -- 172.21.15.62:0/2341818724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08004e030 con 0x7fd08015fda0 2024-03-23T11:28:59.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.847+0000 7fd07cc00640 1 -- 172.21.15.62:0/2341818724 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd06800bd10 con 0x7fd08015fda0 2024-03-23T11:28:59.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.848+0000 7fd07cc00640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd05c06a7d0 0x7fd05c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:28:59.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.848+0000 7fd07cc00640 1 -- 172.21.15.62:0/2341818724 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fd068089470 con 0x7fd08015fda0 2024-03-23T11:28:59.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.848+0000 7fd084c00640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd05c06a7d0 0x7fd05c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:28:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.849+0000 7fd084c00640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd05c06a7d0 0x7fd05c06cc90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd070036000 tx=0x7fd07003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:28:59.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:28:59.853+0000 7fd07cc00640 1 -- 172.21.15.62:0/2341818724 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd068054720 con 0x7fd08015fda0 2024-03-23T11:29:00.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.050+0000 7fd086000640 1 -- 172.21.15.62:0/2341818724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd0800c70b0 con 0x7fd08015fda0 2024-03-23T11:29:00.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.051+0000 7fd07cc00640 1 -- 172.21.15.62:0/2341818724 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd068057d40 con 0x7fd08015fda0 2024-03-23T11:29:00.042 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:29:00.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.053+0000 7fd086000640 1 -- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd05c06a7d0 msgr2=0x7fd05c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.053+0000 7fd086000640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd05c06a7d0 0x7fd05c06cc90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd070036000 tx=0x7fd07003d040 comp rx=0 tx=0).stop 2024-03-23T11:29:00.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.053+0000 7fd086000640 1 -- 172.21.15.62:0/2341818724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd08015fda0 msgr2=0x7fd080158410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.053+0000 7fd086000640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd08015fda0 0x7fd080158410 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fd06800d800 tx=0x7fd06800dcc0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.053+0000 7fd086000640 1 -- 172.21.15.62:0/2341818724 shutdown_connections 2024-03-23T11:29:00.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.053+0000 7fd086000640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd05c06a7d0 0x7fd05c06cc90 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.053+0000 7fd086000640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd08015fda0 0x7fd080158410 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.053+0000 7fd086000640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd080160420 0x7fd08006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.053+0000 7fd086000640 1 --2- 172.21.15.62:0/2341818724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0800d06d0 0x7fd0801f1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.053+0000 7fd086000640 1 -- 172.21.15.62:0/2341818724 >> 172.21.15.62:0/2341818724 conn(0x7fd080089f60 msgr2=0x7fd0800d0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:00.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.054+0000 7fd086000640 1 -- 172.21.15.62:0/2341818724 shutdown_connections 2024-03-23T11:29:00.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.054+0000 7fd086000640 1 -- 172.21.15.62:0/2341818724 wait complete. 2024-03-23T11:29:00.063 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:29:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.284+0000 7f5c13e00640 1 Processor -- start 2024-03-23T11:29:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.285+0000 7f5c13e00640 1 -- start start 2024-03-23T11:29:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.286+0000 7f5c13e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5c14160420 0x7f5c141608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.286+0000 7f5c13e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5c140b5db0 0x7f5c140b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.286+0000 7f5c13e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c140ce510 con 0x7f5c1415fcf0 2024-03-23T11:29:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.286+0000 7f5c13e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c14160df0 con 0x7f5c140b5db0 2024-03-23T11:29:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.286+0000 7f5c13e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c140cdc70 con 0x7f5c14160420 2024-03-23T11:29:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.287+0000 7f5c12a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5c1415fcf0 0x7f5c140d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42932/0 (socket says 172.21.15.62:42932) 2024-03-23T11:29:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.287+0000 7f5c12a00640 1 -- 172.21.15.62:0/4084720745 learned_addr learned my addr 172.21.15.62:0/4084720745 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.287+0000 7f5c13400640 1 --2- 172.21.15.62:0/4084720745 >> v2:172.21.15.62:3300/0 conn(0x7f5c140b5db0 0x7f5c140b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:00.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.287+0000 7f5c12000640 1 --2- 172.21.15.62:0/4084720745 >> v2:172.21.15.120:3300/0 conn(0x7f5c14160420 0x7f5c141608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:00.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.287+0000 7f5c11600640 1 -- 172.21.15.62:0/4084720745 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991934361 0 0) 0x7f5c140ce510 con 0x7f5c1415fcf0 2024-03-23T11:29:00.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.287+0000 7f5c11600640 1 -- 172.21.15.62:0/4084720745 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bf8003520 con 0x7f5c1415fcf0 2024-03-23T11:29:00.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.287+0000 7f5c13400640 1 -- 172.21.15.62:0/4084720745 >> v2:172.21.15.120:3300/0 conn(0x7f5c14160420 msgr2=0x7f5c141608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:00.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.287+0000 7f5c13400640 1 --2- 172.21.15.62:0/4084720745 >> v2:172.21.15.120:3300/0 conn(0x7f5c14160420 0x7f5c141608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.287+0000 7f5c13400640 1 -- 172.21.15.62:0/4084720745 >> v1:172.21.15.62:6789/0 conn(0x7f5c1415fcf0 legacy=0x7f5c140d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.287+0000 7f5c13400640 1 -- 172.21.15.62:0/4084720745 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c140cdf50 con 0x7f5c140b5db0 2024-03-23T11:29:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.288+0000 7f5c13400640 1 --2- 172.21.15.62:0/4084720745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c140b5db0 0x7f5c140b61e0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f5c14161540 tx=0x7f5c0c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55344aa1634f952e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.289+0000 7f5c11600640 1 -- 172.21.15.62:0/4084720745 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c02e070 con 0x7f5c140b5db0 2024-03-23T11:29:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.289+0000 7f5c11600640 1 -- 172.21.15.62:0/4084720745 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c0315a0 con 0x7f5c140b5db0 2024-03-23T11:29:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.289+0000 7f5c11600640 1 -- 172.21.15.62:0/4084720745 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c031930 con 0x7f5c140b5db0 2024-03-23T11:29:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.289+0000 7f5c13e00640 1 -- 172.21.15.62:0/4084720745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c140b5db0 msgr2=0x7f5c140b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.289+0000 7f5c13e00640 1 --2- 172.21.15.62:0/4084720745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c140b5db0 0x7f5c140b61e0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f5c14161540 tx=0x7f5c0c02b840 comp rx=0 tx=0).stop 2024-03-23T11:29:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.289+0000 7f5c13e00640 1 -- 172.21.15.62:0/4084720745 shutdown_connections 2024-03-23T11:29:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.289+0000 7f5c13e00640 1 --2- 172.21.15.62:0/4084720745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c140b5db0 0x7f5c140b61e0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.289+0000 7f5c13e00640 1 --2- 172.21.15.62:0/4084720745 >> v2:172.21.15.120:3300/0 conn(0x7f5c14160420 0x7f5c141608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.289+0000 7f5c13e00640 1 -- 172.21.15.62:0/4084720745 >> 172.21.15.62:0/4084720745 conn(0x7f5c14089f60 msgr2=0x7f5c140d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.290+0000 7f5c13e00640 1 -- 172.21.15.62:0/4084720745 shutdown_connections 2024-03-23T11:29:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.290+0000 7f5c13e00640 1 -- 172.21.15.62:0/4084720745 wait complete. 2024-03-23T11:29:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.291+0000 7f5c13e00640 1 Processor -- start 2024-03-23T11:29:00.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.291+0000 7f5c13e00640 1 -- start start 2024-03-23T11:29:00.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.292+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c140b5db0 0x7f5c14157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.292+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c1415fcf0 0x7f5c1414ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.292+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c14160420 0x7f5c1414cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.292+0000 7f5c13e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1414d4c0 con 0x7f5c1415fcf0 2024-03-23T11:29:00.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.292+0000 7f5c13e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c141588c0 con 0x7f5c14160420 2024-03-23T11:29:00.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.292+0000 7f5c13e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c14155c80 con 0x7f5c140b5db0 2024-03-23T11:29:00.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c12000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c1415fcf0 0x7f5c1414ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:00.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c12000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c1415fcf0 0x7f5c1414ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41532/0 (socket says 172.21.15.62:41532) 2024-03-23T11:29:00.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c12000640 1 -- 172.21.15.62:0/720792565 learned_addr learned my addr 172.21.15.62:0/720792565 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:00.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c12a00640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c140b5db0 0x7f5c14157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:00.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c13400640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c14160420 0x7f5c1414cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:00.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c12000640 1 -- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c140b5db0 msgr2=0x7f5c14157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c12000640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c140b5db0 0x7f5c14157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c12000640 1 -- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c14160420 msgr2=0x7f5c1414cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c12000640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c14160420 0x7f5c1414cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c12000640 1 -- 172.21.15.62:0/720792565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c009400 con 0x7f5c1415fcf0 2024-03-23T11:29:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c13400640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c14160420 0x7f5c1414cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.293+0000 7f5c12000640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c1415fcf0 0x7f5c1414ca40 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f5c04007ef0 tx=0x7f5c0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.294+0000 7f5c0aa00640 1 -- 172.21.15.62:0/720792565 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04014620 con 0x7f5c1415fcf0 2024-03-23T11:29:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.294+0000 7f5c0aa00640 1 -- 172.21.15.62:0/720792565 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c04014780 con 0x7f5c1415fcf0 2024-03-23T11:29:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.294+0000 7f5c0aa00640 1 -- 172.21.15.62:0/720792565 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04014b30 con 0x7f5c1415fcf0 2024-03-23T11:29:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.294+0000 7f5c13e00640 1 -- 172.21.15.62:0/720792565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c14155f60 con 0x7f5c1415fcf0 2024-03-23T11:29:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.294+0000 7f5c13e00640 1 -- 172.21.15.62:0/720792565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c14156520 con 0x7f5c1415fcf0 2024-03-23T11:29:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.295+0000 7f5c13e00640 1 -- 172.21.15.62:0/720792565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c141566b0 con 0x7f5c1415fcf0 2024-03-23T11:29:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.296+0000 7f5c0aa00640 1 -- 172.21.15.62:0/720792565 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5c04014c90 con 0x7f5c1415fcf0 2024-03-23T11:29:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.297+0000 7f5c0aa00640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 0x7f5bf006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.297+0000 7f5c0aa00640 1 -- 172.21.15.62:0/720792565 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f5c04089840 con 0x7f5c1415fcf0 2024-03-23T11:29:00.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.298+0000 7f5c12a00640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 0x7f5bf006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:00.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.299+0000 7f5c12a00640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 0x7f5bf006cc60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc0011e0 tx=0x7f5bfc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.302+0000 7f5c0aa00640 1 -- 172.21.15.62:0/720792565 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c04054b90 con 0x7f5c1415fcf0 2024-03-23T11:29:00.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.500+0000 7f5c13e00640 1 -- 172.21.15.62:0/720792565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5c14161110 con 0x7f5c1415fcf0 2024-03-23T11:29:00.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.501+0000 7f5c0aa00640 1 -- 172.21.15.62:0/720792565 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5c040581b0 con 0x7f5c1415fcf0 2024-03-23T11:29:00.493 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:29:00.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 -- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 msgr2=0x7f5bf006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 0x7f5bf006cc60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc0011e0 tx=0x7f5bfc009040 comp rx=0 tx=0).stop 2024-03-23T11:29:00.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 -- 172.21.15.62:0/720792565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c1415fcf0 msgr2=0x7f5c1414ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c1415fcf0 0x7f5c1414ca40 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f5c04007ef0 tx=0x7f5c0400e4b0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 -- 172.21.15.62:0/720792565 shutdown_connections 2024-03-23T11:29:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5bf006a7a0 0x7f5bf006cc60 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c1415fcf0 0x7f5c1414ca40 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c14160420 0x7f5c1414cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 --2- 172.21.15.62:0/720792565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c140b5db0 0x7f5c14157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 -- 172.21.15.62:0/720792565 >> 172.21.15.62:0/720792565 conn(0x7f5c14089f60 msgr2=0x7f5c140b7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 -- 172.21.15.62:0/720792565 shutdown_connections 2024-03-23T11:29:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.504+0000 7f5c13e00640 1 -- 172.21.15.62:0/720792565 wait complete. 2024-03-23T11:29:00.514 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:29:00.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.725+0000 7f758ce00640 1 Processor -- start 2024-03-23T11:29:00.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.726+0000 7f758ce00640 1 -- start start 2024-03-23T11:29:00.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.727+0000 7f758ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f758815c9e0 0x7f75881607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.727+0000 7f758ce00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f758815cdc0 0x7f75880d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.727+0000 7f758ce00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758806bfc0 con 0x7f75880cf650 2024-03-23T11:29:00.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.727+0000 7f758ce00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75880ce140 con 0x7f758815c9e0 2024-03-23T11:29:00.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.727+0000 7f758ce00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75880b4370 con 0x7f758815cdc0 2024-03-23T11:29:00.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.728+0000 7f7585e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f758815c9e0 0x7f75881607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:00.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.728+0000 7f7586800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f75880cf650 0x7f758815d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42938/0 (socket says 172.21.15.62:42938) 2024-03-23T11:29:00.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.728+0000 7f7586800640 1 -- 172.21.15.62:0/3790938898 learned_addr learned my addr 172.21.15.62:0/3790938898 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:00.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.728+0000 7f7585400640 1 --2- 172.21.15.62:0/3790938898 >> v2:172.21.15.120:3300/0 conn(0x7f758815cdc0 0x7f75880d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:00.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.728+0000 7f7585e00640 1 -- 172.21.15.62:0/3790938898 >> v2:172.21.15.120:3300/0 conn(0x7f758815cdc0 msgr2=0x7f75880d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:00.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.728+0000 7f7585e00640 1 --2- 172.21.15.62:0/3790938898 >> v2:172.21.15.120:3300/0 conn(0x7f758815cdc0 0x7f75880d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.728+0000 7f7585e00640 1 -- 172.21.15.62:0/3790938898 >> v1:172.21.15.62:6789/0 conn(0x7f75880cf650 legacy=0x7f758815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.728+0000 7f7585e00640 1 -- 172.21.15.62:0/3790938898 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75880b5260 con 0x7f758815c9e0 2024-03-23T11:29:00.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.728+0000 7f7584a00640 1 -- 172.21.15.62:0/3790938898 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801378597 0 0) 0x7f758806bfc0 con 0x7f75880cf650 2024-03-23T11:29:00.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.730+0000 7f7585e00640 1 --2- 172.21.15.62:0/3790938898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f758815c9e0 0x7f75881607c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f75880b89a0 tx=0x7f757802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7e7e052d0fa922 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:00.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.730+0000 7f7584a00640 1 -- 172.21.15.62:0/3790938898 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578031830 con 0x7f758815c9e0 2024-03-23T11:29:00.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.730+0000 7f7584a00640 1 -- 172.21.15.62:0/3790938898 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7578031990 con 0x7f758815c9e0 2024-03-23T11:29:00.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.730+0000 7f7584a00640 1 -- 172.21.15.62:0/3790938898 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578031d20 con 0x7f758815c9e0 2024-03-23T11:29:00.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.731+0000 7f758ce00640 1 -- 172.21.15.62:0/3790938898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f758815c9e0 msgr2=0x7f75881607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.731+0000 7f758ce00640 1 --2- 172.21.15.62:0/3790938898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f758815c9e0 0x7f75881607c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f75880b89a0 tx=0x7f757802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.731+0000 7f758ce00640 1 -- 172.21.15.62:0/3790938898 shutdown_connections 2024-03-23T11:29:00.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.731+0000 7f758ce00640 1 --2- 172.21.15.62:0/3790938898 >> v2:172.21.15.120:3300/0 conn(0x7f758815cdc0 0x7f75880d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.731+0000 7f758ce00640 1 --2- 172.21.15.62:0/3790938898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f758815c9e0 0x7f75881607c0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.731+0000 7f758ce00640 1 -- 172.21.15.62:0/3790938898 >> 172.21.15.62:0/3790938898 conn(0x7f7588071570 msgr2=0x7f7588071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:00.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.731+0000 7f758ce00640 1 -- 172.21.15.62:0/3790938898 shutdown_connections 2024-03-23T11:29:00.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.731+0000 7f758ce00640 1 -- 172.21.15.62:0/3790938898 wait complete. 2024-03-23T11:29:00.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.733+0000 7f758ce00640 1 Processor -- start 2024-03-23T11:29:00.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.733+0000 7f758ce00640 1 -- start start 2024-03-23T11:29:00.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.734+0000 7f758ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75880cf650 0x7f75881505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.734+0000 7f758ce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f758815c9e0 0x7f75880ac070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.734+0000 7f758ce00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f758815cdc0 0x7f758815f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.734+0000 7f758ce00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758814dad0 con 0x7f75880cf650 2024-03-23T11:29:00.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.734+0000 7f758ce00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75880aa3a0 con 0x7f758815cdc0 2024-03-23T11:29:00.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.734+0000 7f758ce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758814d5e0 con 0x7f758815c9e0 2024-03-23T11:29:00.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.734+0000 7f7585e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75880cf650 0x7f75881505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:00.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.734+0000 7f7585e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75880cf650 0x7f75881505c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41550/0 (socket says 172.21.15.62:41550) 2024-03-23T11:29:00.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.734+0000 7f7585e00640 1 -- 172.21.15.62:0/286293199 learned_addr learned my addr 172.21.15.62:0/286293199 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.734+0000 7f7585e00640 1 -- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f758815c9e0 msgr2=0x7f75880ac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.734+0000 7f7585e00640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f758815c9e0 0x7f75880ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.735+0000 7f7585e00640 1 -- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f758815cdc0 msgr2=0x7f758815f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.735+0000 7f7585e00640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f758815cdc0 0x7f758815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.735+0000 7f7585e00640 1 -- 172.21.15.62:0/286293199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7578009400 con 0x7f75880cf650 2024-03-23T11:29:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.735+0000 7f7585e00640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75880cf650 0x7f75881505c0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f7578009890 tx=0x7f75780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.735+0000 7f757e000640 1 -- 172.21.15.62:0/286293199 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757803ba70 con 0x7f75880cf650 2024-03-23T11:29:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.735+0000 7f757e000640 1 -- 172.21.15.62:0/286293199 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757803bbf0 con 0x7f75880cf650 2024-03-23T11:29:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.735+0000 7f757e000640 1 -- 172.21.15.62:0/286293199 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578045400 con 0x7f75880cf650 2024-03-23T11:29:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.735+0000 7f758ce00640 1 -- 172.21.15.62:0/286293199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75880a9790 con 0x7f75880cf650 2024-03-23T11:29:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.735+0000 7f758ce00640 1 -- 172.21.15.62:0/286293199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75880a9d50 con 0x7f75880cf650 2024-03-23T11:29:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.736+0000 7f758ce00640 1 -- 172.21.15.62:0/286293199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7588066550 con 0x7f75880cf650 2024-03-23T11:29:00.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.737+0000 7f757e000640 1 -- 172.21.15.62:0/286293199 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f757804e440 con 0x7f75880cf650 2024-03-23T11:29:00.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.738+0000 7f757e000640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f756806a7d0 0x7f756806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:00.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.738+0000 7f757e000640 1 -- 172.21.15.62:0/286293199 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f75780ab110 con 0x7f75880cf650 2024-03-23T11:29:00.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.739+0000 7f7585400640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f756806a7d0 0x7f756806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:00.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.740+0000 7f7585400640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f756806a7d0 0x7f756806cc90 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f75740070b0 tx=0x7f7574007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:00.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.743+0000 7f757e000640 1 -- 172.21.15.62:0/286293199 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7578076450 con 0x7f75880cf650 2024-03-23T11:29:00.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.976+0000 7f758ce00640 1 -- 172.21.15.62:0/286293199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7588066c90 con 0x7f75880cf650 2024-03-23T11:29:00.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.977+0000 7f757e000640 1 -- 172.21.15.62:0/286293199 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f757807ee60 con 0x7f75880cf650 2024-03-23T11:29:00.969 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:00.969 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":70,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:28:18.362214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6833/1594260761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1594260761},{"type":"v1","addr":"172.21.15.62:6833","nonce":1594260761}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:29:00.969 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 70 2024-03-23T11:29:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 -- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f756806a7d0 msgr2=0x7f756806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f756806a7d0 0x7f756806cc90 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f75740070b0 tx=0x7f7574007040 comp rx=0 tx=0).stop 2024-03-23T11:29:00.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 -- 172.21.15.62:0/286293199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75880cf650 msgr2=0x7f75881505c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75880cf650 0x7f75881505c0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f7578009890 tx=0x7f75780046c0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 -- 172.21.15.62:0/286293199 shutdown_connections 2024-03-23T11:29:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f756806a7d0 0x7f756806cc90 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f758815cdc0 0x7f758815f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f758815c9e0 0x7f75880ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 --2- 172.21.15.62:0/286293199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75880cf650 0x7f75881505c0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 -- 172.21.15.62:0/286293199 >> 172.21.15.62:0/286293199 conn(0x7f7588071570 msgr2=0x7f75880b3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 -- 172.21.15.62:0/286293199 shutdown_connections 2024-03-23T11:29:00.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:00.980+0000 7f758ce00640 1 -- 172.21.15.62:0/286293199 wait complete. 2024-03-23T11:29:00.990 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:29:01.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.205+0000 7fe52a600640 1 Processor -- start 2024-03-23T11:29:01.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.205+0000 7fe52a600640 1 -- start start 2024-03-23T11:29:01.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.207+0000 7fe52a600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe52415c830 0x7fe5240ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.207+0000 7fe52a600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5241608d0 con 0x7fe5240ce250 2024-03-23T11:29:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.207+0000 7fe52a600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe524160bb0 con 0x7fe524160310 2024-03-23T11:29:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.207+0000 7fe52a600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5240addb0 con 0x7fe52415c830 2024-03-23T11:29:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.207+0000 7fe529c00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe5240ce250 0x7fe52415d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42942/0 (socket says 172.21.15.62:42942) 2024-03-23T11:29:01.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.207+0000 7fe529c00640 1 -- 172.21.15.62:0/3606249412 learned_addr learned my addr 172.21.15.62:0/3606249412 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.208+0000 7fe529200640 1 --2- 172.21.15.62:0/3606249412 >> v2:172.21.15.120:3300/0 conn(0x7fe52415c830 0x7fe5240ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.208+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046894238 0 0) 0x7fe5241608d0 con 0x7fe5240ce250 2024-03-23T11:29:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.208+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe50c003520 con 0x7fe5240ce250 2024-03-23T11:29:01.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.208+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2760486354 0 0) 0x7fe50c003520 con 0x7fe5240ce250 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.208+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 >> v2:172.21.15.120:3300/0 conn(0x7fe52415c830 msgr2=0x7fe5240ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.208+0000 7fe523400640 1 --2- 172.21.15.62:0/3606249412 >> v2:172.21.15.120:3300/0 conn(0x7fe52415c830 0x7fe5240ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.208+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 >> v1:172.21.15.120:6789/0 conn(0x7fe524160310 legacy=0x7fe5240ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.208+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524160e90 con 0x7fe5240ce250 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.208+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fe51c0019f0 con 0x7fe5240ce250 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.209+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 >> v1:172.21.15.62:6789/0 conn(0x7fe5240ce250 legacy=0x7fe52415d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.209+0000 7fe523400640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe50c003970 0x7fe50c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.209+0000 7fe523400640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe50c0047c0 0x7fe50c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.209+0000 7fe523400640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe50c005180 0x7fe50c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.209+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c00bff0 con 0x7fe50c005180 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.209+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c00c160 con 0x7fe50c0047c0 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.209+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c00c320 con 0x7fe50c003970 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.209+0000 7fe529c00640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe50c005180 0x7fe50c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.209+0000 7fe523e00640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe50c0047c0 0x7fe50c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.209+0000 7fe529200640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe50c003970 0x7fe50c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.210+0000 7fe529c00640 1 -- 172.21.15.62:0/3606249412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe50c003970 msgr2=0x7fe50c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:01.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.210+0000 7fe529c00640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe50c003970 0x7fe50c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.210+0000 7fe529c00640 1 -- 172.21.15.62:0/3606249412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe50c0047c0 msgr2=0x7fe50c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:01.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.210+0000 7fe529c00640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe50c0047c0 0x7fe50c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.210+0000 7fe529c00640 1 -- 172.21.15.62:0/3606249412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c003520 con 0x7fe50c005180 2024-03-23T11:29:01.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.210+0000 7fe523e00640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe50c0047c0 0x7fe50c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.211+0000 7fe529c00640 1 -- 172.21.15.62:0/3606249412 reap_dead start 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.211+0000 7fe529c00640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe50c005180 0x7fe50c00b980 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00a830 tx=0x7fe50c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f757645b37c7158 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.211+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0072d0 con 0x7fe50c005180 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.211+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c02f070 con 0x7fe50c005180 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.211+0000 7fe523400640 1 -- 172.21.15.62:0/3606249412 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0076b0 con 0x7fe50c005180 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.212+0000 7fe52a600640 1 -- 172.21.15.62:0/3606249412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe50c005180 msgr2=0x7fe50c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.212+0000 7fe52a600640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe50c005180 0x7fe50c00b980 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00a830 tx=0x7fe50c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.212+0000 7fe52a600640 1 -- 172.21.15.62:0/3606249412 shutdown_connections 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.212+0000 7fe52a600640 1 --2- 172.21.15.62:0/3606249412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe50c005180 0x7fe50c00b980 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.212+0000 7fe52a600640 1 -- 172.21.15.62:0/3606249412 >> 172.21.15.62:0/3606249412 conn(0x7fe524066050 msgr2=0x7fe524069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.212+0000 7fe52a600640 1 -- 172.21.15.62:0/3606249412 shutdown_connections 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.212+0000 7fe52a600640 1 -- 172.21.15.62:0/3606249412 wait complete. 2024-03-23T11:29:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.214+0000 7fe52a600640 1 Processor -- start 2024-03-23T11:29:01.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.214+0000 7fe52a600640 1 -- start start 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.215+0000 7fe52a600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe50c005180 0x7fe524148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.215+0000 7fe52a600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe524159270 0x7fe5240aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.215+0000 7fe52a600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5240af390 0x7fe524070090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.215+0000 7fe52a600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240af810 con 0x7fe524159270 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.215+0000 7fe52a600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240c4710 con 0x7fe50c005180 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.215+0000 7fe52a600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240c4280 con 0x7fe5240af390 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.215+0000 7fe523e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe524159270 0x7fe5240aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.215+0000 7fe523e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe524159270 0x7fe5240aee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41562/0 (socket says 172.21.15.62:41562) 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.215+0000 7fe523e00640 1 -- 172.21.15.62:0/112094239 learned_addr learned my addr 172.21.15.62:0/112094239 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.215+0000 7fe529c00640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5240af390 0x7fe524070090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.215+0000 7fe529200640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe50c005180 0x7fe524148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.216+0000 7fe523e00640 1 -- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5240af390 msgr2=0x7fe524070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.216+0000 7fe523e00640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5240af390 0x7fe524070090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.216+0000 7fe523e00640 1 -- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe50c005180 msgr2=0x7fe524148f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.216+0000 7fe523e00640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe50c005180 0x7fe524148f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.216+0000 7fe523e00640 1 -- 172.21.15.62:0/112094239 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5240c3670 con 0x7fe524159270 2024-03-23T11:29:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.216+0000 7fe523e00640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe524159270 0x7fe5240aee50 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fe51000e490 tx=0x7fe51000e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.216+0000 7fe520c00640 1 -- 172.21.15.62:0/112094239 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5100052e0 con 0x7fe524159270 2024-03-23T11:29:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.216+0000 7fe520c00640 1 -- 172.21.15.62:0/112094239 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe510005440 con 0x7fe524159270 2024-03-23T11:29:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.216+0000 7fe520c00640 1 -- 172.21.15.62:0/112094239 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510003830 con 0x7fe524159270 2024-03-23T11:29:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.216+0000 7fe52a600640 1 -- 172.21.15.62:0/112094239 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5240c3950 con 0x7fe524159270 2024-03-23T11:29:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.216+0000 7fe52a600640 1 -- 172.21.15.62:0/112094239 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5240c3f10 con 0x7fe524159270 2024-03-23T11:29:01.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.217+0000 7fe52a600640 1 -- 172.21.15.62:0/112094239 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe524066bb0 con 0x7fe524159270 2024-03-23T11:29:01.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.218+0000 7fe520c00640 1 -- 172.21.15.62:0/112094239 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe510003990 con 0x7fe524159270 2024-03-23T11:29:01.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.219+0000 7fe520c00640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe50406a7d0 0x7fe50406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.219+0000 7fe520c00640 1 -- 172.21.15.62:0/112094239 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fe510088c00 con 0x7fe524159270 2024-03-23T11:29:01.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.220+0000 7fe529200640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe50406a7d0 0x7fe50406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.221+0000 7fe529200640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe50406a7d0 0x7fe50406cc90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe5180098d0 tx=0x7fe518008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:01.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.224+0000 7fe520c00640 1 -- 172.21.15.62:0/112094239 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe510053eb0 con 0x7fe524159270 2024-03-23T11:29:01.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.415+0000 7fe52a600640 1 -- 172.21.15.62:0/112094239 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5240c40a0 con 0x7fe524159270 2024-03-23T11:29:01.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.418+0000 7fe520c00640 1 -- 172.21.15.62:0/112094239 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16401 (secure 0 0 0) 0x7fe5100574d0 con 0x7fe524159270 2024-03-23T11:29:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:01.410 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":93,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:28:13.436047+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:26:22.794149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-23T11:26:24.150997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6832/2573443114":"2024-03-24T11:28:13.436002+0000","172.21.15.62:6833/2573443114":"2024-03-24T11:28:13.436002+0000","172.21.15.62:6833/3056962984":"2024-03-24T11:26:47.481437+0000","172.21.15.62:6832/3056962984":"2024-03-24T11:26:47.481437+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:29:01.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 -- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe50406a7d0 msgr2=0x7fe50406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe50406a7d0 0x7fe50406cc90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe5180098d0 tx=0x7fe518008040 comp rx=0 tx=0).stop 2024-03-23T11:29:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 -- 172.21.15.62:0/112094239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe524159270 msgr2=0x7fe5240aee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe524159270 0x7fe5240aee50 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fe51000e490 tx=0x7fe51000e950 comp rx=0 tx=0).stop 2024-03-23T11:29:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 -- 172.21.15.62:0/112094239 shutdown_connections 2024-03-23T11:29:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe50406a7d0 0x7fe50406cc90 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe524159270 0x7fe5240aee50 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe5240af390 0x7fe524070090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 --2- 172.21.15.62:0/112094239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe50c005180 0x7fe524148f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 -- 172.21.15.62:0/112094239 >> 172.21.15.62:0/112094239 conn(0x7fe524066050 msgr2=0x7fe52415bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 -- 172.21.15.62:0/112094239 shutdown_connections 2024-03-23T11:29:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.421+0000 7fe52a600640 1 -- 172.21.15.62:0/112094239 wait complete. 2024-03-23T11:29:01.431 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T11:29:01.432 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:29:01.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.649+0000 7fa049200640 1 Processor -- start 2024-03-23T11:29:01.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.649+0000 7fa049200640 1 -- start start 2024-03-23T11:29:01.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.650+0000 7fa049200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa0440b2f30 0x7fa044070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.650+0000 7fa049200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0440cd140 con 0x7fa044143dd0 2024-03-23T11:29:01.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.650+0000 7fa049200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0440cd3e0 con 0x7fa044070090 2024-03-23T11:29:01.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.650+0000 7fa049200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa044070a60 con 0x7fa0440b2f30 2024-03-23T11:29:01.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.651+0000 7fa043400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa044143dd0 0x7fa04415abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42950/0 (socket says 172.21.15.62:42950) 2024-03-23T11:29:01.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.651+0000 7fa043400640 1 -- 172.21.15.62:0/842633245 learned_addr learned my addr 172.21.15.62:0/842633245 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:01.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.651+0000 7fa043e00640 1 --2- 172.21.15.62:0/842633245 >> v2:172.21.15.120:3300/0 conn(0x7fa0440b2f30 0x7fa044070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.651+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342339488 0 0) 0x7fa0440cd140 con 0x7fa044143dd0 2024-03-23T11:29:01.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.651+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa028003520 con 0x7fa044143dd0 2024-03-23T11:29:01.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.651+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3876207061 0 0) 0x7fa028003520 con 0x7fa044143dd0 2024-03-23T11:29:01.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.651+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 >> v2:172.21.15.120:3300/0 conn(0x7fa0440b2f30 msgr2=0x7fa044070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:01.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.651+0000 7fa042000640 1 --2- 172.21.15.62:0/842633245 >> v2:172.21.15.120:3300/0 conn(0x7fa0440b2f30 0x7fa044070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 >> v1:172.21.15.120:6789/0 conn(0x7fa044070090 legacy=0x7fa0440b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0440cd6c0 con 0x7fa044143dd0 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fa034001670 con 0x7fa044143dd0 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 >> v1:172.21.15.62:6789/0 conn(0x7fa044143dd0 legacy=0x7fa04415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042000640 1 --2- 172.21.15.62:0/842633245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0280039c0 0x7fa028003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042000640 1 --2- 172.21.15.62:0/842633245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0280047c0 0x7fa028004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042000640 1 --2- 172.21.15.62:0/842633245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa028005180 0x7fa02800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa044070a60 con 0x7fa0280047c0 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02800c080 con 0x7fa028005180 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02800c240 con 0x7fa0280039c0 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042a00640 1 --2- 172.21.15.62:0/842633245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0280047c0 0x7fa028004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa043400640 1 --2- 172.21.15.62:0/842633245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0280039c0 0x7fa028003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042a00640 1 -- 172.21.15.62:0/842633245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0280039c0 msgr2=0x7fa028003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042a00640 1 --2- 172.21.15.62:0/842633245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0280039c0 0x7fa028003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042a00640 1 -- 172.21.15.62:0/842633245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa028005180 msgr2=0x7fa02800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:01.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042a00640 1 --2- 172.21.15.62:0/842633245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa028005180 0x7fa02800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa042a00640 1 -- 172.21.15.62:0/842633245 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa028003520 con 0x7fa0280047c0 2024-03-23T11:29:01.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.652+0000 7fa043e00640 1 -- 172.21.15.62:0/842633245 reap_dead start 2024-03-23T11:29:01.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.653+0000 7fa042a00640 1 --2- 172.21.15.62:0/842633245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0280047c0 0x7fa028004c40 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fa030001750 tx=0x7fa0280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee7c43d486bfcb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:01.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.654+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0300052d0 con 0x7fa0280047c0 2024-03-23T11:29:01.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.654+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03002e050 con 0x7fa0280047c0 2024-03-23T11:29:01.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.654+0000 7fa042000640 1 -- 172.21.15.62:0/842633245 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030003830 con 0x7fa0280047c0 2024-03-23T11:29:01.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.654+0000 7fa049200640 1 -- 172.21.15.62:0/842633245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0280047c0 msgr2=0x7fa028004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.654+0000 7fa049200640 1 --2- 172.21.15.62:0/842633245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0280047c0 0x7fa028004c40 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fa030001750 tx=0x7fa0280057f0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.654+0000 7fa049200640 1 -- 172.21.15.62:0/842633245 shutdown_connections 2024-03-23T11:29:01.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.654+0000 7fa049200640 1 --2- 172.21.15.62:0/842633245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0280047c0 0x7fa028004c40 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.654+0000 7fa049200640 1 -- 172.21.15.62:0/842633245 >> 172.21.15.62:0/842633245 conn(0x7fa044066050 msgr2=0x7fa044069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:01.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.654+0000 7fa049200640 1 -- 172.21.15.62:0/842633245 shutdown_connections 2024-03-23T11:29:01.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.654+0000 7fa049200640 1 -- 172.21.15.62:0/842633245 wait complete. 2024-03-23T11:29:01.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.656+0000 7fa049200640 1 Processor -- start 2024-03-23T11:29:01.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.656+0000 7fa049200640 1 -- start start 2024-03-23T11:29:01.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.657+0000 7fa049200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0280047c0 0x7fa04413ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.657+0000 7fa049200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0440caf70 0x7fa0441f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.657+0000 7fa049200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0440cb3d0 0x7fa0441f3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.657+0000 7fa049200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa044139dd0 con 0x7fa0440cb3d0 2024-03-23T11:29:01.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.657+0000 7fa049200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0441f3cc0 con 0x7fa0280047c0 2024-03-23T11:29:01.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.657+0000 7fa049200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0441f3f70 con 0x7fa0440caf70 2024-03-23T11:29:01.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.657+0000 7fa043e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0440cb3d0 0x7fa0441f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.657+0000 7fa043e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0440cb3d0 0x7fa0441f3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41578/0 (socket says 172.21.15.62:41578) 2024-03-23T11:29:01.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.657+0000 7fa043e00640 1 -- 172.21.15.62:0/2151771887 learned_addr learned my addr 172.21.15.62:0/2151771887 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa043400640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0280047c0 0x7fa04413ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa042a00640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0440caf70 0x7fa0441f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa043e00640 1 -- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0440caf70 msgr2=0x7fa0441f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa043e00640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0440caf70 0x7fa0441f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa043e00640 1 -- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0280047c0 msgr2=0x7fa04413ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa043e00640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0280047c0 0x7fa04413ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa043e00640 1 -- 172.21.15.62:0/2151771887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04413a070 con 0x7fa0440cb3d0 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa043e00640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0440cb3d0 0x7fa0441f3780 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fa04406e330 tx=0x7fa03c00ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa03b400640 1 -- 172.21.15.62:0/2151771887 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c010040 con 0x7fa0440cb3d0 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa03b400640 1 -- 172.21.15.62:0/2151771887 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03c00bb30 con 0x7fa0440cb3d0 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa03b400640 1 -- 172.21.15.62:0/2151771887 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c014400 con 0x7fa0440cb3d0 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.658+0000 7fa049200640 1 -- 172.21.15.62:0/2151771887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04413a350 con 0x7fa0440cb3d0 2024-03-23T11:29:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.659+0000 7fa049200640 1 -- 172.21.15.62:0/2151771887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa044159a10 con 0x7fa0440cb3d0 2024-03-23T11:29:01.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.660+0000 7fa049200640 1 -- 172.21.15.62:0/2151771887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa044066bb0 con 0x7fa0440cb3d0 2024-03-23T11:29:01.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.661+0000 7fa03b400640 1 -- 172.21.15.62:0/2151771887 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa03c014560 con 0x7fa0440cb3d0 2024-03-23T11:29:01.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.662+0000 7fa03b400640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa02406a7d0 0x7fa02406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:01.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.662+0000 7fa03b400640 1 -- 172.21.15.62:0/2151771887 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fa03c0892e0 con 0x7fa0440cb3d0 2024-03-23T11:29:01.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.662+0000 7fa043400640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa02406a7d0 0x7fa02406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:01.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.663+0000 7fa043400640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa02406a7d0 0x7fa02406cc90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa034001eb0 tx=0x7fa034009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:01.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.666+0000 7fa03b400640 1 -- 172.21.15.62:0/2151771887 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa03c054630 con 0x7fa0440cb3d0 2024-03-23T11:29:01.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.905+0000 7fa049200640 1 -- 172.21.15.62:0/2151771887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0440b29a0 con 0x7fa0440cb3d0 2024-03-23T11:29:01.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.907+0000 7fa03b400640 1 -- 172.21.15.62:0/2151771887 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fa03c057c50 con 0x7fa0440cb3d0 2024-03-23T11:29:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:01.898 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":70,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:28:18.362214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6833/1594260761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1594260761},{"type":"v1","addr":"172.21.15.62:6833","nonce":1594260761}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:29:01.899 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 70 2024-03-23T11:29:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.909+0000 7fa049200640 1 -- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa02406a7d0 msgr2=0x7fa02406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.909+0000 7fa049200640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa02406a7d0 0x7fa02406cc90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa034001eb0 tx=0x7fa034009040 comp rx=0 tx=0).stop 2024-03-23T11:29:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.909+0000 7fa049200640 1 -- 172.21.15.62:0/2151771887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0440cb3d0 msgr2=0x7fa0441f3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.909+0000 7fa049200640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0440cb3d0 0x7fa0441f3780 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fa04406e330 tx=0x7fa03c00ef50 comp rx=0 tx=0).stop 2024-03-23T11:29:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.909+0000 7fa049200640 1 -- 172.21.15.62:0/2151771887 shutdown_connections 2024-03-23T11:29:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.909+0000 7fa049200640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa02406a7d0 0x7fa02406cc90 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.909+0000 7fa049200640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0440cb3d0 0x7fa0441f3780 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.909+0000 7fa049200640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0440caf70 0x7fa0441f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.909+0000 7fa049200640 1 --2- 172.21.15.62:0/2151771887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0280047c0 0x7fa04413ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.909+0000 7fa049200640 1 -- 172.21.15.62:0/2151771887 >> 172.21.15.62:0/2151771887 conn(0x7fa044066050 msgr2=0x7fa044069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.910+0000 7fa049200640 1 -- 172.21.15.62:0/2151771887 shutdown_connections 2024-03-23T11:29:01.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:01.910+0000 7fa049200640 1 -- 172.21.15.62:0/2151771887 wait complete. 2024-03-23T11:29:01.921 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:29:02.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.152+0000 7f682f200640 1 Processor -- start 2024-03-23T11:29:02.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.152+0000 7f682f200640 1 -- start start 2024-03-23T11:29:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.154+0000 7f682f200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f68300ceb00 0x7f68300ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.154+0000 7f682f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f68300ce0f0 0x7f683015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.154+0000 7f682f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68300cf420 con 0x7f68300b12b0 2024-03-23T11:29:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.154+0000 7f682f200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68300cbc40 con 0x7f68300ceb00 2024-03-23T11:29:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.154+0000 7f682f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f683015c3a0 con 0x7f68300ce0f0 2024-03-23T11:29:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.154+0000 7f682de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f68300ceb00 0x7f68300ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.154+0000 7f682de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f68300ceb00 0x7f68300ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41582/0 (socket says 172.21.15.62:41582) 2024-03-23T11:29:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.154+0000 7f682de00640 1 -- 172.21.15.62:0/1712448930 learned_addr learned my addr 172.21.15.62:0/1712448930 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.155+0000 7f6827e00640 1 --2- 172.21.15.62:0/1712448930 >> v2:172.21.15.120:3300/0 conn(0x7f68300ce0f0 0x7f683015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.155+0000 7f682de00640 1 -- 172.21.15.62:0/1712448930 >> v2:172.21.15.120:3300/0 conn(0x7f68300ce0f0 msgr2=0x7f683015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.155+0000 7f682de00640 1 --2- 172.21.15.62:0/1712448930 >> v2:172.21.15.120:3300/0 conn(0x7f68300ce0f0 0x7f683015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.155+0000 7f682de00640 1 -- 172.21.15.62:0/1712448930 >> v1:172.21.15.120:6789/0 conn(0x7f68300b12b0 legacy=0x7f683015e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.155+0000 7f682de00640 1 -- 172.21.15.62:0/1712448930 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68300ce660 con 0x7f68300ceb00 2024-03-23T11:29:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.156+0000 7f682de00640 1 --2- 172.21.15.62:0/1712448930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68300ceb00 0x7f68300ceee0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f683015f4f0 tx=0x7f682002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28587d84f2086ed server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:02.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.156+0000 7f682d400640 1 -- 172.21.15.62:0/1712448930 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820031870 con 0x7f68300ceb00 2024-03-23T11:29:02.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.157+0000 7f682d400640 1 -- 172.21.15.62:0/1712448930 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68200319d0 con 0x7f68300ceb00 2024-03-23T11:29:02.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.157+0000 7f682d400640 1 -- 172.21.15.62:0/1712448930 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820031d60 con 0x7f68300ceb00 2024-03-23T11:29:02.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.157+0000 7f682f200640 1 -- 172.21.15.62:0/1712448930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68300ceb00 msgr2=0x7f68300ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:02.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.157+0000 7f682f200640 1 --2- 172.21.15.62:0/1712448930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68300ceb00 0x7f68300ceee0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f683015f4f0 tx=0x7f682002b910 comp rx=0 tx=0).stop 2024-03-23T11:29:02.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.157+0000 7f682f200640 1 -- 172.21.15.62:0/1712448930 shutdown_connections 2024-03-23T11:29:02.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.157+0000 7f682f200640 1 --2- 172.21.15.62:0/1712448930 >> v2:172.21.15.120:3300/0 conn(0x7f68300ce0f0 0x7f683015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.157+0000 7f682f200640 1 --2- 172.21.15.62:0/1712448930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68300ceb00 0x7f68300ceee0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.157+0000 7f682f200640 1 -- 172.21.15.62:0/1712448930 >> 172.21.15.62:0/1712448930 conn(0x7f68300b2010 msgr2=0x7f68300b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:02.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.157+0000 7f682f200640 1 -- 172.21.15.62:0/1712448930 shutdown_connections 2024-03-23T11:29:02.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.158+0000 7f682f200640 1 -- 172.21.15.62:0/1712448930 wait complete. 2024-03-23T11:29:02.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.159+0000 7f682f200640 1 Processor -- start 2024-03-23T11:29:02.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.160+0000 7f682f200640 1 -- start start 2024-03-23T11:29:02.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.160+0000 7f682f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68300b12b0 0x7f68300c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.160+0000 7f682f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68300ce0f0 0x7f683015e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.160+0000 7f682f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68300ceb00 0x7f683015e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.160+0000 7f682f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68300c8fc0 con 0x7f68300ce0f0 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f682f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68300c7b30 con 0x7f68300ceb00 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f682f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68300b4f80 con 0x7f68300b12b0 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f6827e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68300ce0f0 0x7f683015e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f6827e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68300ce0f0 0x7f683015e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41590/0 (socket says 172.21.15.62:41590) 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f6827e00640 1 -- 172.21.15.62:0/4260563021 learned_addr learned my addr 172.21.15.62:0/4260563021 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f682de00640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68300b12b0 0x7f68300c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f682e800640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68300ceb00 0x7f683015e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f6827e00640 1 -- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68300b12b0 msgr2=0x7f68300c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f6827e00640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68300b12b0 0x7f68300c75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f6827e00640 1 -- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68300ceb00 msgr2=0x7f683015e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f6827e00640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68300ceb00 0x7f683015e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f6827e00640 1 -- 172.21.15.62:0/4260563021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68200094d0 con 0x7f68300ce0f0 2024-03-23T11:29:02.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.161+0000 7f682e800640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68300ceb00 0x7f683015e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:02.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.162+0000 7f6827e00640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68300ce0f0 0x7f683015e370 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f681800d5f0 tx=0x7f681800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:02.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.162+0000 7f6826000640 1 -- 172.21.15.62:0/4260563021 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68180125f0 con 0x7f68300ce0f0 2024-03-23T11:29:02.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.162+0000 7f6826000640 1 -- 172.21.15.62:0/4260563021 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6818012750 con 0x7f68300ce0f0 2024-03-23T11:29:02.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.162+0000 7f6826000640 1 -- 172.21.15.62:0/4260563021 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6818012b00 con 0x7f68300ce0f0 2024-03-23T11:29:02.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.162+0000 7f682f200640 1 -- 172.21.15.62:0/4260563021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68300c3ee0 con 0x7f68300ce0f0 2024-03-23T11:29:02.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.162+0000 7f682f200640 1 -- 172.21.15.62:0/4260563021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6830153d40 con 0x7f68300ce0f0 2024-03-23T11:29:02.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.163+0000 7f682f200640 1 -- 172.21.15.62:0/4260563021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68300c3d40 con 0x7f68300ce0f0 2024-03-23T11:29:02.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.164+0000 7f6826000640 1 -- 172.21.15.62:0/4260563021 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6818012c60 con 0x7f68300ce0f0 2024-03-23T11:29:02.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.165+0000 7f6826000640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f680c06a7a0 0x7f680c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.165+0000 7f6826000640 1 -- 172.21.15.62:0/4260563021 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f6818089360 con 0x7f68300ce0f0 2024-03-23T11:29:02.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.166+0000 7f682de00640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f680c06a7a0 0x7f680c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.167+0000 7f682de00640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f680c06a7a0 0x7f680c06cc60 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6820036000 tx=0x7f682003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:02.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.170+0000 7f6826000640 1 -- 172.21.15.62:0/4260563021 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f68300c3d40 con 0x7f68300ce0f0 2024-03-23T11:29:02.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.408+0000 7f682f200640 1 -- 172.21.15.62:0/4260563021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68300c3d40 con 0x7f68300ce0f0 2024-03-23T11:29:02.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.409+0000 7f6826000640 1 -- 172.21.15.62:0/4260563021 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f68300c3d40 con 0x7f68300ce0f0 2024-03-23T11:29:02.400 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:02.401 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":70,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:26:25.131291+0000","modified":"2024-03-23T11:28:18.362214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6833/1594260761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1594260761},{"type":"v1","addr":"172.21.15.62:6833","nonce":1594260761}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T11:29:02.401 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 70 2024-03-23T11:29:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 -- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f680c06a7a0 msgr2=0x7f680c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f680c06a7a0 0x7f680c06cc60 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6820036000 tx=0x7f682003c040 comp rx=0 tx=0).stop 2024-03-23T11:29:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 -- 172.21.15.62:0/4260563021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68300ce0f0 msgr2=0x7f683015e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68300ce0f0 0x7f683015e370 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f681800d5f0 tx=0x7f681800dab0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 -- 172.21.15.62:0/4260563021 shutdown_connections 2024-03-23T11:29:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f680c06a7a0 0x7f680c06cc60 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68300ce0f0 0x7f683015e370 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68300ceb00 0x7f683015e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 --2- 172.21.15.62:0/4260563021 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68300b12b0 0x7f68300c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 -- 172.21.15.62:0/4260563021 >> 172.21.15.62:0/4260563021 conn(0x7f68300b2010 msgr2=0x7f68301567f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 -- 172.21.15.62:0/4260563021 shutdown_connections 2024-03-23T11:29:02.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.412+0000 7f682f200640 1 -- 172.21.15.62:0/4260563021 wait complete. 2024-03-23T11:29:02.421 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:29:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.642+0000 7f845da00640 1 Processor -- start 2024-03-23T11:29:02.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.642+0000 7f845da00640 1 -- start start 2024-03-23T11:29:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.644+0000 7f845da00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f845806dca0 0x7f84580b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.644+0000 7f845da00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f845806c300 0x7f84580cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.644+0000 7f845da00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84580cad20 con 0x7f84580cd7c0 2024-03-23T11:29:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.644+0000 7f845da00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845806c770 con 0x7f845806c300 2024-03-23T11:29:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.644+0000 7f845da00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845806e130 con 0x7f845806dca0 2024-03-23T11:29:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.644+0000 7f8457400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f845806c300 0x7f84580cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.644+0000 7f8457400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f845806c300 0x7f84580cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41604/0 (socket says 172.21.15.62:41604) 2024-03-23T11:29:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.644+0000 7f8457400640 1 -- 172.21.15.62:0/365599700 learned_addr learned my addr 172.21.15.62:0/365599700 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.645+0000 7f844fe00640 1 --2- 172.21.15.62:0/365599700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f845806dca0 0x7f84580b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.645+0000 7f8457400640 1 -- 172.21.15.62:0/365599700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f845806dca0 msgr2=0x7f84580b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.645+0000 7f8457400640 1 --2- 172.21.15.62:0/365599700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f845806dca0 0x7f84580b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.645+0000 7f8457400640 1 -- 172.21.15.62:0/365599700 >> v1:172.21.15.120:6789/0 conn(0x7f84580cd7c0 legacy=0x7f84580cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.645+0000 7f8457400640 1 -- 172.21.15.62:0/365599700 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845815a8a0 con 0x7f845806c300 2024-03-23T11:29:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.646+0000 7f8457400640 1 --2- 172.21.15.62:0/365599700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845806c300 0x7f84580cb490 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f8450009e50 tx=0x7f845002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3078cc53d3dec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.646+0000 7f8456000640 1 -- 172.21.15.62:0/365599700 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845002c9e0 con 0x7f845806c300 2024-03-23T11:29:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.646+0000 7f8456000640 1 -- 172.21.15.62:0/365599700 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845002cb40 con 0x7f845806c300 2024-03-23T11:29:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.646+0000 7f8456000640 1 -- 172.21.15.62:0/365599700 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84500343f0 con 0x7f845806c300 2024-03-23T11:29:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.647+0000 7f845da00640 1 -- 172.21.15.62:0/365599700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845806c300 msgr2=0x7f84580cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.647+0000 7f845da00640 1 --2- 172.21.15.62:0/365599700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845806c300 0x7f84580cb490 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f8450009e50 tx=0x7f845002be40 comp rx=0 tx=0).stop 2024-03-23T11:29:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.647+0000 7f845da00640 1 -- 172.21.15.62:0/365599700 shutdown_connections 2024-03-23T11:29:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.647+0000 7f845da00640 1 --2- 172.21.15.62:0/365599700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845806c300 0x7f84580cb490 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.647+0000 7f845da00640 1 --2- 172.21.15.62:0/365599700 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f845806dca0 0x7f84580b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.647+0000 7f845da00640 1 -- 172.21.15.62:0/365599700 >> 172.21.15.62:0/365599700 conn(0x7f84580b02e0 msgr2=0x7f84580b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.647+0000 7f845da00640 1 -- 172.21.15.62:0/365599700 shutdown_connections 2024-03-23T11:29:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.647+0000 7f845da00640 1 -- 172.21.15.62:0/365599700 wait complete. 2024-03-23T11:29:02.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.649+0000 7f845da00640 1 Processor -- start 2024-03-23T11:29:02.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.649+0000 7f845da00640 1 -- start start 2024-03-23T11:29:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.650+0000 7f845da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845806c300 0x7f845814daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.650+0000 7f845da00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84580cd7c0 0x7f84581564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.650+0000 7f845da00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84580afd00 0x7f84581e6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.650+0000 7f845da00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84580aed40 con 0x7f845806c300 2024-03-23T11:29:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.650+0000 7f845da00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84580ae130 con 0x7f84580afd00 2024-03-23T11:29:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.650+0000 7f845da00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84580ae410 con 0x7f84580cd7c0 2024-03-23T11:29:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.650+0000 7f8456a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845806c300 0x7f845814daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.650+0000 7f8456a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845806c300 0x7f845814daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41606/0 (socket says 172.21.15.62:41606) 2024-03-23T11:29:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.650+0000 7f8456a00640 1 -- 172.21.15.62:0/2914784212 learned_addr learned my addr 172.21.15.62:0/2914784212 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f844fe00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84580cd7c0 0x7f84581564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f8456a00640 1 -- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84580cd7c0 msgr2=0x7f84581564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f8456a00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84580cd7c0 0x7f84581564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f8457400640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84580afd00 0x7f84581e6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f8456a00640 1 -- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84580afd00 msgr2=0x7f84581e6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f8456a00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84580afd00 0x7f84581e6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f8456a00640 1 -- 172.21.15.62:0/2914784212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8450009b30 con 0x7f845806c300 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f844fe00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84580cd7c0 0x7f84581564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f8456a00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845806c300 0x7f845814daa0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f844800b3f0 tx=0x7f844800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f844ea00640 1 -- 172.21.15.62:0/2914784212 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448011830 con 0x7f845806c300 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f844ea00640 1 -- 172.21.15.62:0/2914784212 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8448011990 con 0x7f845806c300 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f845da00640 1 -- 172.21.15.62:0/2914784212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84580ae6f0 con 0x7f845806c300 2024-03-23T11:29:02.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.651+0000 7f844ea00640 1 -- 172.21.15.62:0/2914784212 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448011d40 con 0x7f845806c300 2024-03-23T11:29:02.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.652+0000 7f845da00640 1 -- 172.21.15.62:0/2914784212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8458156a20 con 0x7f845806c300 2024-03-23T11:29:02.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.653+0000 7f845da00640 1 -- 172.21.15.62:0/2914784212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8458066c90 con 0x7f845806c300 2024-03-23T11:29:02.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.654+0000 7f844ea00640 1 -- 172.21.15.62:0/2914784212 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f844801a400 con 0x7f845806c300 2024-03-23T11:29:02.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.655+0000 7f844ea00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843406a7a0 0x7f843406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:02.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.655+0000 7f844ea00640 1 -- 172.21.15.62:0/2914784212 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f8448088d10 con 0x7f845806c300 2024-03-23T11:29:02.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.655+0000 7f844fe00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843406a7a0 0x7f843406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:02.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.656+0000 7f844fe00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843406a7a0 0x7f843406cc60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f84581f59d0 tx=0x7f8440008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:02.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.659+0000 7f844ea00640 1 -- 172.21.15.62:0/2914784212 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f844808e020 con 0x7f845806c300 2024-03-23T11:29:02.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.862+0000 7f845da00640 1 -- 172.21.15.62:0/2914784212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f845806c970 con 0x7f845806c300 2024-03-23T11:29:02.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.865+0000 7f844ea00640 1 -- 172.21.15.62:0/2914784212 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16401 (secure 0 0 0) 0x7f8448015090 con 0x7f845806c300 2024-03-23T11:29:02.857 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:02.857 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":93,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:28:13.436047+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:26:22.794149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-23T11:26:24.150997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6832/2573443114":"2024-03-24T11:28:13.436002+0000","172.21.15.62:6833/2573443114":"2024-03-24T11:28:13.436002+0000","172.21.15.62:6833/3056962984":"2024-03-24T11:26:47.481437+0000","172.21.15.62:6832/3056962984":"2024-03-24T11:26:47.481437+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:29:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.868+0000 7f845da00640 1 -- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843406a7a0 msgr2=0x7f843406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.868+0000 7f845da00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843406a7a0 0x7f843406cc60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f84581f59d0 tx=0x7f8440008040 comp rx=0 tx=0).stop 2024-03-23T11:29:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.868+0000 7f845da00640 1 -- 172.21.15.62:0/2914784212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845806c300 msgr2=0x7f845814daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.868+0000 7f845da00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845806c300 0x7f845814daa0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f844800b3f0 tx=0x7f844800b8b0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.868+0000 7f845da00640 1 -- 172.21.15.62:0/2914784212 shutdown_connections 2024-03-23T11:29:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.868+0000 7f845da00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843406a7a0 0x7f843406cc60 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.868+0000 7f845da00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f84580afd00 0x7f84581e6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.868+0000 7f845da00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84580cd7c0 0x7f84581564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.868+0000 7f845da00640 1 --2- 172.21.15.62:0/2914784212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845806c300 0x7f845814daa0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.868+0000 7f845da00640 1 -- 172.21.15.62:0/2914784212 >> 172.21.15.62:0/2914784212 conn(0x7f84580b02e0 msgr2=0x7f84580702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.869+0000 7f845da00640 1 -- 172.21.15.62:0/2914784212 shutdown_connections 2024-03-23T11:29:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:02.869+0000 7f845da00640 1 -- 172.21.15.62:0/2914784212 wait complete. 2024-03-23T11:29:02.880 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:29:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.114+0000 7fcd6ce00640 1 Processor -- start 2024-03-23T11:29:03.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.114+0000 7fcd6ce00640 1 -- start start 2024-03-23T11:29:03.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.116+0000 7fcd6ce00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcd6815c9e0 0x7fcd681607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.116+0000 7fcd6ce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd6815cdc0 0x7fcd680d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.116+0000 7fcd6ce00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6806bfc0 con 0x7fcd680cf650 2024-03-23T11:29:03.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.116+0000 7fcd6ce00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd680ce140 con 0x7fcd6815c9e0 2024-03-23T11:29:03.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.116+0000 7fcd6ce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680b4370 con 0x7fcd6815cdc0 2024-03-23T11:29:03.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.116+0000 7fcd65e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcd6815c9e0 0x7fcd681607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.117+0000 7fcd65e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcd6815c9e0 0x7fcd681607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41620/0 (socket says 172.21.15.62:41620) 2024-03-23T11:29:03.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.117+0000 7fcd65e00640 1 -- 172.21.15.62:0/3488090415 learned_addr learned my addr 172.21.15.62:0/3488090415 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:03.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.117+0000 7fcd5fe00640 1 --2- 172.21.15.62:0/3488090415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd6815cdc0 0x7fcd680d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.117+0000 7fcd65e00640 1 -- 172.21.15.62:0/3488090415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd6815cdc0 msgr2=0x7fcd680d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:03.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.117+0000 7fcd65e00640 1 --2- 172.21.15.62:0/3488090415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd6815cdc0 0x7fcd680d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.117+0000 7fcd65e00640 1 -- 172.21.15.62:0/3488090415 >> v1:172.21.15.120:6789/0 conn(0x7fcd680cf650 legacy=0x7fcd6815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:03.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.117+0000 7fcd65e00640 1 -- 172.21.15.62:0/3488090415 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd680b5260 con 0x7fcd6815c9e0 2024-03-23T11:29:03.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.119+0000 7fcd65e00640 1 --2- 172.21.15.62:0/3488090415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd6815c9e0 0x7fcd681607c0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fcd58009870 tx=0x7fcd5802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210b6d52663f34e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.120+0000 7fcd65400640 1 -- 172.21.15.62:0/3488090415 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd58030840 con 0x7fcd6815c9e0 2024-03-23T11:29:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.120+0000 7fcd65400640 1 -- 172.21.15.62:0/3488090415 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd580309a0 con 0x7fcd6815c9e0 2024-03-23T11:29:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.120+0000 7fcd65400640 1 -- 172.21.15.62:0/3488090415 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd58030d30 con 0x7fcd6815c9e0 2024-03-23T11:29:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.121+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3488090415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd6815c9e0 msgr2=0x7fcd681607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.121+0000 7fcd6ce00640 1 --2- 172.21.15.62:0/3488090415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd6815c9e0 0x7fcd681607c0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fcd58009870 tx=0x7fcd5802b980 comp rx=0 tx=0).stop 2024-03-23T11:29:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.121+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3488090415 shutdown_connections 2024-03-23T11:29:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.121+0000 7fcd6ce00640 1 --2- 172.21.15.62:0/3488090415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd6815cdc0 0x7fcd680d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.121+0000 7fcd6ce00640 1 --2- 172.21.15.62:0/3488090415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd6815c9e0 0x7fcd681607c0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.121+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3488090415 >> 172.21.15.62:0/3488090415 conn(0x7fcd68071570 msgr2=0x7fcd68071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:03.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.121+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3488090415 shutdown_connections 2024-03-23T11:29:03.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.121+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3488090415 wait complete. 2024-03-23T11:29:03.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.123+0000 7fcd6ce00640 1 Processor -- start 2024-03-23T11:29:03.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.123+0000 7fcd6ce00640 1 -- start start 2024-03-23T11:29:03.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.124+0000 7fcd6ce00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd680cf650 0x7fcd680d2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.124+0000 7fcd6ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd6815c9e0 0x7fcd681f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.124+0000 7fcd6ce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd6815cdc0 0x7fcd6815ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.124+0000 7fcd6ce00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680ab660 con 0x7fcd6815c9e0 2024-03-23T11:29:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.124+0000 7fcd6ce00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6814ef70 con 0x7fcd680cf650 2024-03-23T11:29:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.124+0000 7fcd6ce00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6814f4a0 con 0x7fcd6815cdc0 2024-03-23T11:29:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.124+0000 7fcd5fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd6815c9e0 0x7fcd681f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.124+0000 7fcd5fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd6815c9e0 0x7fcd681f5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41624/0 (socket says 172.21.15.62:41624) 2024-03-23T11:29:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.124+0000 7fcd5fe00640 1 -- 172.21.15.62:0/3922678635 learned_addr learned my addr 172.21.15.62:0/3922678635 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.124+0000 7fcd65e00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd680cf650 0x7fcd680d2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.124+0000 7fcd66800640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd6815cdc0 0x7fcd6815ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd5fe00640 1 -- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd6815cdc0 msgr2=0x7fcd6815ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd5fe00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd6815cdc0 0x7fcd6815ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd5fe00640 1 -- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd680cf650 msgr2=0x7fcd680d2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd5fe00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd680cf650 0x7fcd680d2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd5fe00640 1 -- 172.21.15.62:0/3922678635 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580094d0 con 0x7fcd6815c9e0 2024-03-23T11:29:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd65e00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd680cf650 0x7fcd680d2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd5fe00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd6815c9e0 0x7fcd681f5fa0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fcd5000e8a0 tx=0x7fcd5000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd5e000640 1 -- 172.21.15.62:0/3922678635 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd500094f0 con 0x7fcd6815c9e0 2024-03-23T11:29:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd5e000640 1 -- 172.21.15.62:0/3922678635 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5000c940 con 0x7fcd6815c9e0 2024-03-23T11:29:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd5e000640 1 -- 172.21.15.62:0/3922678635 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5000ccf0 con 0x7fcd6815c9e0 2024-03-23T11:29:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3922678635 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd681f5b10 con 0x7fcd6815c9e0 2024-03-23T11:29:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.125+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3922678635 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd6815b3a0 con 0x7fcd6815c9e0 2024-03-23T11:29:03.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.127+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3922678635 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd680b4910 con 0x7fcd6815c9e0 2024-03-23T11:29:03.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.127+0000 7fcd5e000640 1 -- 172.21.15.62:0/3922678635 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fcd50013470 con 0x7fcd6815c9e0 2024-03-23T11:29:03.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.128+0000 7fcd5e000640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd4406a7d0 0x7fcd4406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.128+0000 7fcd5e000640 1 -- 172.21.15.62:0/3922678635 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fcd50088fe0 con 0x7fcd6815c9e0 2024-03-23T11:29:03.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.129+0000 7fcd65e00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd4406a7d0 0x7fcd4406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.130+0000 7fcd65e00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd4406a7d0 0x7fcd4406cc90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fcd58036000 tx=0x7fcd5803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:03.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.133+0000 7fcd5e000640 1 -- 172.21.15.62:0/3922678635 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd680b4910 con 0x7fcd6815c9e0 2024-03-23T11:29:03.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.375+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3922678635 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcd680b4910 con 0x7fcd6815c9e0 2024-03-23T11:29:03.368 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:29:03.377+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:29:03.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.390+0000 7fcd5e000640 1 -- 172.21.15.62:0/3922678635 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v71) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcd680b4910 con 0x7fcd6815c9e0 2024-03-23T11:29:03.382 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:29:03.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.393+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd4406a7d0 msgr2=0x7fcd4406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:03.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.393+0000 7fcd6ce00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd4406a7d0 0x7fcd4406cc90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fcd58036000 tx=0x7fcd5803b040 comp rx=0 tx=0).stop 2024-03-23T11:29:03.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.393+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3922678635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd6815c9e0 msgr2=0x7fcd681f5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:03.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.393+0000 7fcd6ce00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd6815c9e0 0x7fcd681f5fa0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fcd5000e8a0 tx=0x7fcd5000ed60 comp rx=0 tx=0).stop 2024-03-23T11:29:03.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.393+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3922678635 shutdown_connections 2024-03-23T11:29:03.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.393+0000 7fcd6ce00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcd4406a7d0 0x7fcd4406cc90 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.393+0000 7fcd6ce00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd6815c9e0 0x7fcd681f5fa0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.393+0000 7fcd6ce00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcd6815cdc0 0x7fcd6815ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.393+0000 7fcd6ce00640 1 --2- 172.21.15.62:0/3922678635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd680cf650 0x7fcd680d2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.394+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3922678635 >> 172.21.15.62:0/3922678635 conn(0x7fcd68071570 msgr2=0x7fcd680cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:03.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.394+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3922678635 shutdown_connections 2024-03-23T11:29:03.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.394+0000 7fcd6ce00640 1 -- 172.21.15.62:0/3922678635 wait complete. 2024-03-23T11:29:03.395 INFO:tasks.ceph.mds.a.smithi062.stderr: -2288> 2024-03-23T11:28:19.468+0000 7fa2d0200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:29:03.397 INFO:tasks.ceph.mds.a.smithi062.stderr: -1842> 2024-03-23T11:28:23.100+0000 7fa2cbc00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() (iversion lock) | dirtyparent=1 authpin=1 scrubqueue=0 0x55d3eef9a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(13)0x10000000001:[<0x10000000000/file v6>,<0x1/dir v13>]//[]","memoryvalue":"(13)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-23T11:29:03.407 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T11:29:03.474 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:29:03.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.624+0000 7fe647800640 1 Processor -- start 2024-03-23T11:29:03.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.625+0000 7fe647800640 1 -- start start 2024-03-23T11:29:03.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.626+0000 7fe647800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe6400cd7c0 0x7fe6400b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.626+0000 7fe647800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64006c300 0x7fe6400cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.626+0000 7fe647800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6400cad20 con 0x7fe64006dca0 2024-03-23T11:29:03.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.626+0000 7fe647800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe64006c750 con 0x7fe6400cd7c0 2024-03-23T11:29:03.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.626+0000 7fe647800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64006e130 con 0x7fe64006c300 2024-03-23T11:29:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.627+0000 7fe644e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe6400cd7c0 0x7fe6400b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.627+0000 7fe645800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64006c300 0x7fe6400cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.627+0000 7fe644e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe6400cd7c0 0x7fe6400b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:52318/0 (socket says 172.21.15.62:52318) 2024-03-23T11:29:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.627+0000 7fe644e00640 1 -- 172.21.15.62:0/674166134 learned_addr learned my addr 172.21.15.62:0/674166134 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:03.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.627+0000 7fe645800640 1 --2- 172.21.15.62:0/674166134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64006c300 0x7fe6400cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:53702/0 (socket says 172.21.15.62:53702) 2024-03-23T11:29:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.628+0000 7fe63fe00640 1 -- 172.21.15.62:0/674166134 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283539192 0 0) 0x7fe6400cad20 con 0x7fe64006dca0 2024-03-23T11:29:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.628+0000 7fe63fe00640 1 -- 172.21.15.62:0/674166134 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe620005730 con 0x7fe64006dca0 2024-03-23T11:29:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.628+0000 7fe644e00640 1 -- 172.21.15.62:0/674166134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64006c300 msgr2=0x7fe6400cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.628+0000 7fe644e00640 1 --2- 172.21.15.62:0/674166134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64006c300 0x7fe6400cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.628+0000 7fe644e00640 1 -- 172.21.15.62:0/674166134 >> v1:172.21.15.120:6789/0 conn(0x7fe64006dca0 legacy=0x7fe6400cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.628+0000 7fe644e00640 1 -- 172.21.15.62:0/674166134 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64015a8a0 con 0x7fe6400cd7c0 2024-03-23T11:29:03.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.630+0000 7fe644e00640 1 --2- 172.21.15.62:0/674166134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6400cd7c0 0x7fe6400b15e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe628009870 tx=0x7fe62802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80a6eb481aa82a52 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:03.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.630+0000 7fe63fe00640 1 -- 172.21.15.62:0/674166134 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628031640 con 0x7fe6400cd7c0 2024-03-23T11:29:03.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.630+0000 7fe63fe00640 1 -- 172.21.15.62:0/674166134 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6280317a0 con 0x7fe6400cd7c0 2024-03-23T11:29:03.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.630+0000 7fe63fe00640 1 -- 172.21.15.62:0/674166134 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628031b30 con 0x7fe6400cd7c0 2024-03-23T11:29:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.631+0000 7fe647800640 1 -- 172.21.15.62:0/674166134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6400cd7c0 msgr2=0x7fe6400b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.631+0000 7fe647800640 1 --2- 172.21.15.62:0/674166134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6400cd7c0 0x7fe6400b15e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe628009870 tx=0x7fe62802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.631+0000 7fe647800640 1 -- 172.21.15.62:0/674166134 shutdown_connections 2024-03-23T11:29:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.631+0000 7fe647800640 1 --2- 172.21.15.62:0/674166134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64006c300 0x7fe6400cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.631+0000 7fe647800640 1 --2- 172.21.15.62:0/674166134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6400cd7c0 0x7fe6400b15e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.631+0000 7fe647800640 1 -- 172.21.15.62:0/674166134 >> 172.21.15.62:0/674166134 conn(0x7fe6400b02e0 msgr2=0x7fe6400b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:03.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.631+0000 7fe647800640 1 -- 172.21.15.62:0/674166134 shutdown_connections 2024-03-23T11:29:03.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.631+0000 7fe647800640 1 -- 172.21.15.62:0/674166134 wait complete. 2024-03-23T11:29:03.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.633+0000 7fe647800640 1 Processor -- start 2024-03-23T11:29:03.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.633+0000 7fe647800640 1 -- start start 2024-03-23T11:29:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.634+0000 7fe647800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64006c300 0x7fe640149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.634+0000 7fe647800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6400cd7c0 0x7fe640146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.634+0000 7fe647800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe640146970 0x7fe640147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.634+0000 7fe647800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe640149950 con 0x7fe640146970 2024-03-23T11:29:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.634+0000 7fe647800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe640146ea0 con 0x7fe64006c300 2024-03-23T11:29:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.634+0000 7fe647800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe640147180 con 0x7fe6400cd7c0 2024-03-23T11:29:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.634+0000 7fe645800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe640146970 0x7fe640147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.634+0000 7fe645800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe640146970 0x7fe640147f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41654/0 (socket says 172.21.15.62:41654) 2024-03-23T11:29:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.634+0000 7fe645800640 1 -- 172.21.15.62:0/677693086 learned_addr learned my addr 172.21.15.62:0/677693086 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.634+0000 7fe644e00640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64006c300 0x7fe640149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.634+0000 7fe637e00640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6400cd7c0 0x7fe640146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.635+0000 7fe645800640 1 -- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6400cd7c0 msgr2=0x7fe640146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.635+0000 7fe645800640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6400cd7c0 0x7fe640146020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.635+0000 7fe645800640 1 -- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64006c300 msgr2=0x7fe640149410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.635+0000 7fe645800640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64006c300 0x7fe640149410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.635+0000 7fe645800640 1 -- 172.21.15.62:0/677693086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6280094d0 con 0x7fe640146970 2024-03-23T11:29:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.635+0000 7fe645800640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe640146970 0x7fe640147f40 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fe63800efc0 tx=0x7fe63800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.635+0000 7fe63d600640 1 -- 172.21.15.62:0/677693086 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638014480 con 0x7fe640146970 2024-03-23T11:29:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.635+0000 7fe63d600640 1 -- 172.21.15.62:0/677693086 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63800f040 con 0x7fe640146970 2024-03-23T11:29:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.635+0000 7fe63d600640 1 -- 172.21.15.62:0/677693086 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638014820 con 0x7fe640146970 2024-03-23T11:29:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.635+0000 7fe647800640 1 -- 172.21.15.62:0/677693086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe640147460 con 0x7fe640146970 2024-03-23T11:29:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.636+0000 7fe647800640 1 -- 172.21.15.62:0/677693086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe640147a20 con 0x7fe640146970 2024-03-23T11:29:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.637+0000 7fe647800640 1 -- 172.21.15.62:0/677693086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe640147c40 con 0x7fe640146970 2024-03-23T11:29:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.638+0000 7fe63d600640 1 -- 172.21.15.62:0/677693086 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe638014980 con 0x7fe640146970 2024-03-23T11:29:03.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.639+0000 7fe63d600640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61c06a7d0 0x7fe61c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:03.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.639+0000 7fe63d600640 1 -- 172.21.15.62:0/677693086 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5996+0+0 (secure 0 0 0) 0x7fe638089b30 con 0x7fe640146970 2024-03-23T11:29:03.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.639+0000 7fe644e00640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61c06a7d0 0x7fe61c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:03.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.640+0000 7fe644e00640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61c06a7d0 0x7fe61c06cc90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe628036000 tx=0x7fe62803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:03.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.643+0000 7fe63d600640 1 -- 172.21.15.62:0/677693086 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe638054d90 con 0x7fe640146970 2024-03-23T11:29:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:03.888+0000 7fe647800640 1 -- 172.21.15.62:0/677693086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe640066c90 con 0x7fe640146970 2024-03-23T11:29:04.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.397+0000 7fe63d600640 1 -- 172.21.15.62:0/677693086 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v72) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe6380583b0 con 0x7fe640146970 2024-03-23T11:29:04.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 -- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61c06a7d0 msgr2=0x7fe61c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:04.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61c06a7d0 0x7fe61c06cc90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe628036000 tx=0x7fe62803c040 comp rx=0 tx=0).stop 2024-03-23T11:29:04.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 -- 172.21.15.62:0/677693086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe640146970 msgr2=0x7fe640147f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:04.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe640146970 0x7fe640147f40 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fe63800efc0 tx=0x7fe63800c490 comp rx=0 tx=0).stop 2024-03-23T11:29:04.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 -- 172.21.15.62:0/677693086 shutdown_connections 2024-03-23T11:29:04.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe61c06a7d0 0x7fe61c06cc90 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:04.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe640146970 0x7fe640147f40 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:04.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6400cd7c0 0x7fe640146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:04.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 --2- 172.21.15.62:0/677693086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64006c300 0x7fe640149410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:04.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 -- 172.21.15.62:0/677693086 >> 172.21.15.62:0/677693086 conn(0x7fe6400b02e0 msgr2=0x7fe64015c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:04.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 -- 172.21.15.62:0/677693086 shutdown_connections 2024-03-23T11:29:04.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.401+0000 7fe647800640 1 -- 172.21.15.62:0/677693086 wait complete. 2024-03-23T11:29:04.414 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T11:29:04.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.645+0000 7f245f200640 1 Processor -- start 2024-03-23T11:29:04.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.645+0000 7f245f200640 1 -- start start 2024-03-23T11:29:04.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.646+0000 7f245f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f245815c820 0x7f24580b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:04.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.646+0000 7f245f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24580cf710 0x7f24580d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:04.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.646+0000 7f245f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24580d0200 con 0x7f24580b28e0 2024-03-23T11:29:04.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.646+0000 7f245f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24580d07c0 con 0x7f245815c820 2024-03-23T11:29:04.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.646+0000 7f245f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24580ce070 con 0x7f24580cf710 2024-03-23T11:29:04.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.647+0000 7f245de00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f245815c820 0x7f24580b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:04.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.647+0000 7f245e800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24580cf710 0x7f24580d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:04.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.647+0000 7f245de00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f245815c820 0x7f24580b23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:52346/0 (socket says 172.21.15.62:52346) 2024-03-23T11:29:04.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.647+0000 7f245de00640 1 -- 172.21.15.62:0/1191167510 learned_addr learned my addr 172.21.15.62:0/1191167510 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:04.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.648+0000 7f245e800640 1 -- 172.21.15.62:0/1191167510 >> v2:172.21.15.120:3300/0 conn(0x7f245815c820 msgr2=0x7f24580b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:04.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.648+0000 7f245e800640 1 --2- 172.21.15.62:0/1191167510 >> v2:172.21.15.120:3300/0 conn(0x7f245815c820 0x7f24580b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:04.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.648+0000 7f245e800640 1 -- 172.21.15.62:0/1191167510 >> v1:172.21.15.120:6789/0 conn(0x7f24580b28e0 legacy=0x7f24580cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:04.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.648+0000 7f245e800640 1 -- 172.21.15.62:0/1191167510 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24580ce350 con 0x7f24580cf710 2024-03-23T11:29:04.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.649+0000 7f245e800640 1 --2- 172.21.15.62:0/1191167510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24580cf710 0x7f24580d0c50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2454009d90 tx=0x7f245402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2371ae0eeaee6358 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:04.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.650+0000 7f245ca00640 1 -- 172.21.15.62:0/1191167510 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245402cbc0 con 0x7f24580cf710 2024-03-23T11:29:04.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.650+0000 7f245ca00640 1 -- 172.21.15.62:0/1191167510 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245402cd20 con 0x7f24580cf710 2024-03-23T11:29:04.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.650+0000 7f245ca00640 1 -- 172.21.15.62:0/1191167510 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454034660 con 0x7f24580cf710 2024-03-23T11:29:04.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.651+0000 7f245f200640 1 -- 172.21.15.62:0/1191167510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24580cf710 msgr2=0x7f24580d0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:04.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.651+0000 7f245f200640 1 --2- 172.21.15.62:0/1191167510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24580cf710 0x7f24580d0c50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2454009d90 tx=0x7f245402be10 comp rx=0 tx=0).stop 2024-03-23T11:29:04.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.651+0000 7f245f200640 1 -- 172.21.15.62:0/1191167510 shutdown_connections 2024-03-23T11:29:04.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.651+0000 7f245f200640 1 --2- 172.21.15.62:0/1191167510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24580cf710 0x7f24580d0c50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:04.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.651+0000 7f245f200640 1 --2- 172.21.15.62:0/1191167510 >> v2:172.21.15.120:3300/0 conn(0x7f245815c820 0x7f24580b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:04.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.651+0000 7f245f200640 1 -- 172.21.15.62:0/1191167510 >> 172.21.15.62:0/1191167510 conn(0x7f245809b9d0 msgr2=0x7f2458089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:04.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.651+0000 7f245f200640 1 -- 172.21.15.62:0/1191167510 shutdown_connections 2024-03-23T11:29:04.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.651+0000 7f245f200640 1 -- 172.21.15.62:0/1191167510 wait complete. 2024-03-23T11:29:04.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.653+0000 7f245f200640 1 Processor -- start 2024-03-23T11:29:04.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.653+0000 7f245f200640 1 -- start start 2024-03-23T11:29:04.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24580b28e0 0x7f24580ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:04.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24580cf710 0x7f245815efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:04.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f245815c820 0x7f24581577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:04.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245815f520 con 0x7f24580cf710 2024-03-23T11:29:04.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2458157d30 con 0x7f245815c820 2024-03-23T11:29:04.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24580cae50 con 0x7f24580b28e0 2024-03-23T11:29:04.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24580cf710 0x7f245815efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:04.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24580cf710 0x7f245815efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41662/0 (socket says 172.21.15.62:41662) 2024-03-23T11:29:04.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245d400640 1 -- 172.21.15.62:0/3587575071 learned_addr learned my addr 172.21.15.62:0/3587575071 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:04.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245de00640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24580b28e0 0x7f24580ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:04.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245e800640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f245815c820 0x7f24581577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:04.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.654+0000 7f245d400640 1 -- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24580b28e0 msgr2=0x7f24580ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:04.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.655+0000 7f245d400640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24580b28e0 0x7f24580ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:04.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.655+0000 7f245d400640 1 -- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f245815c820 msgr2=0x7f24581577f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:04.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.655+0000 7f245d400640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f245815c820 0x7f24581577f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:04.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.655+0000 7f245d400640 1 -- 172.21.15.62:0/3587575071 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2454009a70 con 0x7f24580cf710 2024-03-23T11:29:04.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.655+0000 7f245e800640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f245815c820 0x7f24581577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:04.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.655+0000 7f245d400640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24580cf710 0x7f245815efe0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f244800c420 tx=0x7f244800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:04.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.655+0000 7f2452000640 1 -- 172.21.15.62:0/3587575071 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24480052d0 con 0x7f24580cf710 2024-03-23T11:29:04.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.655+0000 7f2452000640 1 -- 172.21.15.62:0/3587575071 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2448005430 con 0x7f24580cf710 2024-03-23T11:29:04.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.655+0000 7f2452000640 1 -- 172.21.15.62:0/3587575071 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448003880 con 0x7f24580cf710 2024-03-23T11:29:04.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.655+0000 7f245f200640 1 -- 172.21.15.62:0/3587575071 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24580cb130 con 0x7f24580cf710 2024-03-23T11:29:04.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.655+0000 7f245f200640 1 -- 172.21.15.62:0/3587575071 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24580cb6f0 con 0x7f24580cf710 2024-03-23T11:29:04.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.656+0000 7f245f200640 1 -- 172.21.15.62:0/3587575071 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2458066bb0 con 0x7f24580cf710 2024-03-23T11:29:04.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.658+0000 7f2452000640 1 -- 172.21.15.62:0/3587575071 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f24480039e0 con 0x7f24580cf710 2024-03-23T11:29:04.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.659+0000 7f2452000640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f243c06a7a0 0x7f243c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:04.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.659+0000 7f2452000640 1 -- 172.21.15.62:0/3587575071 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5996+0+0 (secure 0 0 0) 0x7f2448088420 con 0x7f24580cf710 2024-03-23T11:29:04.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.660+0000 7f245de00640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f243c06a7a0 0x7f243c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:04.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.661+0000 7f245de00640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f243c06a7a0 0x7f243c06cc60 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f244c009a20 tx=0x7f244c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:04.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.664+0000 7f2452000640 1 -- 172.21.15.62:0/3587575071 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2448053720 con 0x7f24580cf710 2024-03-23T11:29:04.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:04.862+0000 7f245f200640 1 -- 172.21.15.62:0/3587575071 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2458069d80 con 0x7f24580cf710 2024-03-23T11:29:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.590+0000 7f2452000640 1 -- 172.21.15.62:0/3587575071 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2448056d40 con 0x7f24580cf710 2024-03-23T11:29:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' removed 2024-03-23T11:29:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.595+0000 7f245f200640 1 -- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f243c06a7a0 msgr2=0x7f243c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.595+0000 7f245f200640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f243c06a7a0 0x7f243c06cc60 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f244c009a20 tx=0x7f244c008040 comp rx=0 tx=0).stop 2024-03-23T11:29:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.595+0000 7f245f200640 1 -- 172.21.15.62:0/3587575071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24580cf710 msgr2=0x7f245815efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.595+0000 7f245f200640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24580cf710 0x7f245815efe0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f244800c420 tx=0x7f244800c8e0 comp rx=0 tx=0).stop 2024-03-23T11:29:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.595+0000 7f245f200640 1 -- 172.21.15.62:0/3587575071 shutdown_connections 2024-03-23T11:29:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.595+0000 7f245f200640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f243c06a7a0 0x7f243c06cc60 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.595+0000 7f245f200640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24580cf710 0x7f245815efe0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.595+0000 7f245f200640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f245815c820 0x7f24581577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.595+0000 7f245f200640 1 --2- 172.21.15.62:0/3587575071 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24580b28e0 0x7f24580ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.596+0000 7f245f200640 1 -- 172.21.15.62:0/3587575071 >> 172.21.15.62:0/3587575071 conn(0x7f245809b9d0 msgr2=0x7f24580b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.596+0000 7f245f200640 1 -- 172.21.15.62:0/3587575071 shutdown_connections 2024-03-23T11:29:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.596+0000 7f245f200640 1 -- 172.21.15.62:0/3587575071 wait complete. 2024-03-23T11:29:05.612 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T11:29:05.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.844+0000 7fb73cc00640 1 Processor -- start 2024-03-23T11:29:05.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.845+0000 7fb73cc00640 1 -- start start 2024-03-23T11:29:05.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.846+0000 7fb73cc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb73815c820 0x7fb7380b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:05.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.846+0000 7fb73cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380b28e0 0x7fb7380cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:05.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.846+0000 7fb73cc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7380d0200 con 0x7fb7380cf540 2024-03-23T11:29:05.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.846+0000 7fb73cc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7380cf9c0 con 0x7fb73815c820 2024-03-23T11:29:05.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.846+0000 7fb73cc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380d07c0 con 0x7fb7380b28e0 2024-03-23T11:29:05.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.847+0000 7fb72fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380b28e0 0x7fb7380cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:05.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.847+0000 7fb737400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb73815c820 0x7fb7380b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:05.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.847+0000 7fb72fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380b28e0 0x7fb7380cf000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:53756/0 (socket says 172.21.15.62:53756) 2024-03-23T11:29:05.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.847+0000 7fb72fe00640 1 -- 172.21.15.62:0/2211082238 learned_addr learned my addr 172.21.15.62:0/2211082238 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:05.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.848+0000 7fb72fe00640 1 -- 172.21.15.62:0/2211082238 >> v2:172.21.15.120:3300/0 conn(0x7fb73815c820 msgr2=0x7fb7380b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:05.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.848+0000 7fb72fe00640 1 --2- 172.21.15.62:0/2211082238 >> v2:172.21.15.120:3300/0 conn(0x7fb73815c820 0x7fb7380b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:05.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.848+0000 7fb72fe00640 1 -- 172.21.15.62:0/2211082238 >> v1:172.21.15.120:6789/0 conn(0x7fb7380cf540 legacy=0x7fb7380d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:05.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.848+0000 7fb72fe00640 1 -- 172.21.15.62:0/2211082238 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7380ce070 con 0x7fb7380b28e0 2024-03-23T11:29:05.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.849+0000 7fb72fe00640 1 --2- 172.21.15.62:0/2211082238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380b28e0 0x7fb7380cf000 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb720009850 tx=0x7fb72002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e7dae79409e138e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:05.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.850+0000 7fb736a00640 1 -- 172.21.15.62:0/2211082238 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb720004230 con 0x7fb7380b28e0 2024-03-23T11:29:05.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.850+0000 7fb736a00640 1 -- 172.21.15.62:0/2211082238 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb720004390 con 0x7fb7380b28e0 2024-03-23T11:29:05.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.850+0000 7fb736a00640 1 -- 172.21.15.62:0/2211082238 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb720031830 con 0x7fb7380b28e0 2024-03-23T11:29:05.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.850+0000 7fb73cc00640 1 -- 172.21.15.62:0/2211082238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380b28e0 msgr2=0x7fb7380cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:05.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.850+0000 7fb73cc00640 1 --2- 172.21.15.62:0/2211082238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380b28e0 0x7fb7380cf000 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb720009850 tx=0x7fb72002b8d0 comp rx=0 tx=0).stop 2024-03-23T11:29:05.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.851+0000 7fb73cc00640 1 -- 172.21.15.62:0/2211082238 shutdown_connections 2024-03-23T11:29:05.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.851+0000 7fb73cc00640 1 --2- 172.21.15.62:0/2211082238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380b28e0 0x7fb7380cf000 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:05.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.851+0000 7fb73cc00640 1 --2- 172.21.15.62:0/2211082238 >> v2:172.21.15.120:3300/0 conn(0x7fb73815c820 0x7fb7380b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:05.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.851+0000 7fb73cc00640 1 -- 172.21.15.62:0/2211082238 >> 172.21.15.62:0/2211082238 conn(0x7fb73809b9d0 msgr2=0x7fb738089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:05.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.851+0000 7fb73cc00640 1 -- 172.21.15.62:0/2211082238 shutdown_connections 2024-03-23T11:29:05.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.851+0000 7fb73cc00640 1 -- 172.21.15.62:0/2211082238 wait complete. 2024-03-23T11:29:05.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.852+0000 7fb73cc00640 1 Processor -- start 2024-03-23T11:29:05.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.853+0000 7fb73cc00640 1 -- start start 2024-03-23T11:29:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb73cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7380b28e0 0x7fb73814ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb73cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380cf540 0x7fb7380cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb73cc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb73815c820 0x7fb7380c6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb73cc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73814f150 con 0x7fb7380b28e0 2024-03-23T11:29:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb73cc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380c8890 con 0x7fb73815c820 2024-03-23T11:29:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb73cc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380ca860 con 0x7fb7380cf540 2024-03-23T11:29:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb737400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7380b28e0 0x7fb73814ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:05.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb737400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7380b28e0 0x7fb73814ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41668/0 (socket says 172.21.15.62:41668) 2024-03-23T11:29:05.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb737400640 1 -- 172.21.15.62:0/569578441 learned_addr learned my addr 172.21.15.62:0/569578441 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb737400640 1 -- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380cf540 msgr2=0x7fb7380cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb72fe00640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380cf540 0x7fb7380cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb737e00640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb73815c820 0x7fb7380c6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb737400640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380cf540 0x7fb7380cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb737400640 1 -- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb73815c820 msgr2=0x7fb7380c6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.854+0000 7fb737400640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb73815c820 0x7fb7380c6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.855+0000 7fb737400640 1 -- 172.21.15.62:0/569578441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720009460 con 0x7fb7380b28e0 2024-03-23T11:29:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.855+0000 7fb737400640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7380b28e0 0x7fb73814ec10 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fb72800e870 tx=0x7fb72800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.855+0000 7fb72f400640 1 -- 172.21.15.62:0/569578441 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280095c0 con 0x7fb7380b28e0 2024-03-23T11:29:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.855+0000 7fb72f400640 1 -- 172.21.15.62:0/569578441 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72800c940 con 0x7fb7380b28e0 2024-03-23T11:29:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.855+0000 7fb72f400640 1 -- 172.21.15.62:0/569578441 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72800ccb0 con 0x7fb7380b28e0 2024-03-23T11:29:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.855+0000 7fb73cc00640 1 -- 172.21.15.62:0/569578441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7381f2150 con 0x7fb7380b28e0 2024-03-23T11:29:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.855+0000 7fb73cc00640 1 -- 172.21.15.62:0/569578441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7380c8da0 con 0x7fb7380b28e0 2024-03-23T11:29:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.856+0000 7fb73cc00640 1 -- 172.21.15.62:0/569578441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb738066bb0 con 0x7fb7380b28e0 2024-03-23T11:29:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.857+0000 7fb72f400640 1 -- 172.21.15.62:0/569578441 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb728013470 con 0x7fb7380b28e0 2024-03-23T11:29:05.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.858+0000 7fb72f400640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb71406a7a0 0x7fb71406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:05.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.858+0000 7fb72f400640 1 -- 172.21.15.62:0/569578441 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5558+0+0 (secure 0 0 0) 0x7fb728089290 con 0x7fb7380b28e0 2024-03-23T11:29:05.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.859+0000 7fb72fe00640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb71406a7a0 0x7fb71406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:05.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.860+0000 7fb72fe00640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb71406a7a0 0x7fb71406cc60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb720035000 tx=0x7fb720008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:05.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:05.863+0000 7fb72f400640 1 -- 172.21.15.62:0/569578441 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7280547f0 con 0x7fb7380b28e0 2024-03-23T11:29:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.060+0000 7fb73cc00640 1 -- 172.21.15.62:0/569578441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb7380cf2c0 con 0x7fb7380b28e0 2024-03-23T11:29:06.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.594+0000 7fb72f400640 1 -- 172.21.15.62:0/569578441 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb728057e10 con 0x7fb7380b28e0 2024-03-23T11:29:06.589 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' removed 2024-03-23T11:29:06.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.600+0000 7fb73cc00640 1 -- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb71406a7a0 msgr2=0x7fb71406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:06.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.600+0000 7fb73cc00640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb71406a7a0 0x7fb71406cc60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb720035000 tx=0x7fb720008030 comp rx=0 tx=0).stop 2024-03-23T11:29:06.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.600+0000 7fb73cc00640 1 -- 172.21.15.62:0/569578441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7380b28e0 msgr2=0x7fb73814ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:06.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.600+0000 7fb73cc00640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7380b28e0 0x7fb73814ec10 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fb72800e870 tx=0x7fb72800ed30 comp rx=0 tx=0).stop 2024-03-23T11:29:06.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.600+0000 7fb73cc00640 1 -- 172.21.15.62:0/569578441 shutdown_connections 2024-03-23T11:29:06.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.600+0000 7fb73cc00640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb71406a7a0 0x7fb71406cc60 secure :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb720035000 tx=0x7fb720008030 comp rx=0 tx=0).stop 2024-03-23T11:29:06.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.600+0000 7fb73cc00640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb73815c820 0x7fb7380c6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:06.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.600+0000 7fb73cc00640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7380cf540 0x7fb7380cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:06.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.600+0000 7fb73cc00640 1 --2- 172.21.15.62:0/569578441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7380b28e0 0x7fb73814ec10 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:06.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.600+0000 7fb73cc00640 1 -- 172.21.15.62:0/569578441 >> 172.21.15.62:0/569578441 conn(0x7fb73809b9d0 msgr2=0x7fb7380b3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:06.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.601+0000 7fb73cc00640 1 -- 172.21.15.62:0/569578441 shutdown_connections 2024-03-23T11:29:06.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.601+0000 7fb73cc00640 1 -- 172.21.15.62:0/569578441 wait complete. 2024-03-23T11:29:06.620 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T11:29:06.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.827+0000 7f6811400640 1 Processor -- start 2024-03-23T11:29:06.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.827+0000 7f6811400640 1 -- start start 2024-03-23T11:29:06.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.829+0000 7f6811400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f680c15c730 0x7f680c0b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:06.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.829+0000 7f6811400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f680c069c60 0x7f680c06a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:06.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.829+0000 7f6811400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680c0abe50 con 0x7f680c0ad370 2024-03-23T11:29:06.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.829+0000 7f6811400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680c06a650 con 0x7f680c069c60 2024-03-23T11:29:06.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.829+0000 7f6811400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680c06cba0 con 0x7f680c15c730 2024-03-23T11:29:06.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.830+0000 7f6810a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f680c069c60 0x7f680c06a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:06.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.830+0000 7f680be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f680c15c730 0x7f680c0b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:06.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.830+0000 7f680b400640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f680c0ad370 0x7f680c160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:43296/0 (socket says 172.21.15.62:43296) 2024-03-23T11:29:06.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.830+0000 7f680b400640 1 -- 172.21.15.62:0/3034050486 learned_addr learned my addr 172.21.15.62:0/3034050486 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:06.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.831+0000 7f6810a00640 1 -- 172.21.15.62:0/3034050486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f680c15c730 msgr2=0x7f680c0b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:06.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.831+0000 7f6810a00640 1 --2- 172.21.15.62:0/3034050486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f680c15c730 0x7f680c0b1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:06.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.831+0000 7f6810a00640 1 -- 172.21.15.62:0/3034050486 >> v1:172.21.15.120:6789/0 conn(0x7f680c0ad370 legacy=0x7f680c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:06.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.831+0000 7f6810a00640 1 -- 172.21.15.62:0/3034050486 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c06ce80 con 0x7f680c069c60 2024-03-23T11:29:06.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.831+0000 7f680be00640 1 --2- 172.21.15.62:0/3034050486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f680c15c730 0x7f680c0b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:29:06.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.832+0000 7f6810a00640 1 --2- 172.21.15.62:0/3034050486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f680c069c60 0x7f680c06a110 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6804009d90 tx=0x7f680402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ca433984ddf5935 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:06.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.833+0000 7f680aa00640 1 -- 172.21.15.62:0/3034050486 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680402cc60 con 0x7f680c069c60 2024-03-23T11:29:06.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.833+0000 7f680aa00640 1 -- 172.21.15.62:0/3034050486 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680402cdc0 con 0x7f680c069c60 2024-03-23T11:29:06.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.833+0000 7f680aa00640 1 -- 172.21.15.62:0/3034050486 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804034660 con 0x7f680c069c60 2024-03-23T11:29:06.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.833+0000 7f6811400640 1 -- 172.21.15.62:0/3034050486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f680c069c60 msgr2=0x7f680c06a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:06.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.833+0000 7f6811400640 1 --2- 172.21.15.62:0/3034050486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f680c069c60 0x7f680c06a110 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6804009d90 tx=0x7f680402beb0 comp rx=0 tx=0).stop 2024-03-23T11:29:06.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.834+0000 7f6811400640 1 -- 172.21.15.62:0/3034050486 shutdown_connections 2024-03-23T11:29:06.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.834+0000 7f6811400640 1 --2- 172.21.15.62:0/3034050486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f680c069c60 0x7f680c06a110 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:06.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.834+0000 7f6811400640 1 --2- 172.21.15.62:0/3034050486 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f680c15c730 0x7f680c0b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:06.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.834+0000 7f6811400640 1 -- 172.21.15.62:0/3034050486 >> 172.21.15.62:0/3034050486 conn(0x7f680c0b0090 msgr2=0x7f680c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:06.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.834+0000 7f6811400640 1 -- 172.21.15.62:0/3034050486 shutdown_connections 2024-03-23T11:29:06.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.834+0000 7f6811400640 1 -- 172.21.15.62:0/3034050486 wait complete. 2024-03-23T11:29:06.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.835+0000 7f6811400640 1 Processor -- start 2024-03-23T11:29:06.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.836+0000 7f6811400640 1 -- start start 2024-03-23T11:29:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.836+0000 7f6811400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f680c069c60 0x7f680c146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.836+0000 7f6811400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f680c0ad370 0x7f680c149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.836+0000 7f6811400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f680c15c730 0x7f680c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.836+0000 7f6811400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c0c62a0 con 0x7f680c069c60 2024-03-23T11:29:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.836+0000 7f6811400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c14a330 con 0x7f680c0ad370 2024-03-23T11:29:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.836+0000 7f6811400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680c15ad30 con 0x7f680c15c730 2024-03-23T11:29:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.837+0000 7f680be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f680c069c60 0x7f680c146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.837+0000 7f680be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f680c069c60 0x7f680c146d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41670/0 (socket says 172.21.15.62:41670) 2024-03-23T11:29:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.837+0000 7f680be00640 1 -- 172.21.15.62:0/2163817115 learned_addr learned my addr 172.21.15.62:0/2163817115 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.837+0000 7f6810a00640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f680c15c730 0x7f680c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.837+0000 7f680b400640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f680c0ad370 0x7f680c149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.837+0000 7f680be00640 1 -- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f680c15c730 msgr2=0x7f680c0b2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.837+0000 7f680be00640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f680c15c730 0x7f680c0b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.837+0000 7f680be00640 1 -- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f680c0ad370 msgr2=0x7f680c149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.837+0000 7f680be00640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f680c0ad370 0x7f680c149df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.837+0000 7f680be00640 1 -- 172.21.15.62:0/2163817115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc009400 con 0x7f680c069c60 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.838+0000 7f680be00640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f680c069c60 0x7f680c146d00 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00e960 tx=0x7f67fc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.838+0000 7f6803e00640 1 -- 172.21.15.62:0/2163817115 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc00ca00 con 0x7f680c069c60 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.838+0000 7f6803e00640 1 -- 172.21.15.62:0/2163817115 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc00cb60 con 0x7f680c069c60 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.838+0000 7f6803e00640 1 -- 172.21.15.62:0/2163817115 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc013470 con 0x7f680c069c60 2024-03-23T11:29:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.838+0000 7f6811400640 1 -- 172.21.15.62:0/2163817115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6804009a70 con 0x7f680c069c60 2024-03-23T11:29:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.838+0000 7f6811400640 1 -- 172.21.15.62:0/2163817115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c15b2f0 con 0x7f680c069c60 2024-03-23T11:29:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.839+0000 7f6811400640 1 -- 172.21.15.62:0/2163817115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680c147350 con 0x7f680c069c60 2024-03-23T11:29:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.840+0000 7f6803e00640 1 -- 172.21.15.62:0/2163817115 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f67fc0135d0 con 0x7f680c069c60 2024-03-23T11:29:06.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.841+0000 7f6803e00640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f67ec06a7a0 0x7f67ec06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:06.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.841+0000 7f6803e00640 1 -- 172.21.15.62:0/2163817115 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f67fc088ee0 con 0x7f680c069c60 2024-03-23T11:29:06.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.842+0000 7f680b400640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f67ec06a7a0 0x7f67ec06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:06.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.843+0000 7f680b400640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f67ec06a7a0 0x7f67ec06cc60 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f680c15c140 tx=0x7f67f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:06.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:06.846+0000 7f6803e00640 1 -- 172.21.15.62:0/2163817115 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67fc0544b0 con 0x7f680c069c60 2024-03-23T11:29:07.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.046+0000 7f6811400640 1 -- 172.21.15.62:0/2163817115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f680c066bb0 con 0x7f680c069c60 2024-03-23T11:29:07.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.055+0000 7f6803e00640 1 -- 172.21.15.62:0/2163817115 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f67fc057ad0 con 0x7f680c069c60 2024-03-23T11:29:07.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.057+0000 7f6811400640 1 -- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f67ec06a7a0 msgr2=0x7f67ec06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.057+0000 7f6811400640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f67ec06a7a0 0x7f67ec06cc60 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f680c15c140 tx=0x7f67f8009040 comp rx=0 tx=0).stop 2024-03-23T11:29:07.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.058+0000 7f6811400640 1 -- 172.21.15.62:0/2163817115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f680c069c60 msgr2=0x7f680c146d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.058+0000 7f6811400640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f680c069c60 0x7f680c146d00 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00e960 tx=0x7f67fc00ee20 comp rx=0 tx=0).stop 2024-03-23T11:29:07.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.058+0000 7f6811400640 1 -- 172.21.15.62:0/2163817115 shutdown_connections 2024-03-23T11:29:07.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.058+0000 7f6811400640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f67ec06a7a0 0x7f67ec06cc60 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.058+0000 7f6811400640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f680c15c730 0x7f680c0b2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.058+0000 7f6811400640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f680c0ad370 0x7f680c149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.058+0000 7f6811400640 1 --2- 172.21.15.62:0/2163817115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f680c069c60 0x7f680c146d00 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.058+0000 7f6811400640 1 -- 172.21.15.62:0/2163817115 >> 172.21.15.62:0/2163817115 conn(0x7f680c0b0090 msgr2=0x7f680c0b30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:07.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.058+0000 7f6811400640 1 -- 172.21.15.62:0/2163817115 shutdown_connections 2024-03-23T11:29:07.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.058+0000 7f6811400640 1 -- 172.21.15.62:0/2163817115 wait complete. 2024-03-23T11:29:07.069 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-03-23T11:29:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.295+0000 7f8b43200640 1 Processor -- start 2024-03-23T11:29:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.296+0000 7f8b43200640 1 -- start start 2024-03-23T11:29:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.297+0000 7f8b43200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b3c15c9e0 0x7f8b3c1607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.297+0000 7f8b43200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8b3c15cdc0 0x7f8b3c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.297+0000 7f8b43200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8b3c0cf650 0x7f8b3c0cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.297+0000 7f8b43200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c0d0020 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.297+0000 7f8b43200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c06bfc0 con 0x7f8b3c0cf650 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.297+0000 7f8b43200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0b5260 con 0x7f8b3c15c9e0 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.297+0000 7f8b3b400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8b3c15cdc0 0x7f8b3c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.298+0000 7f8b3b400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8b3c15cdc0 0x7f8b3c0d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33408/0 (socket says 172.21.15.62:33408) 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.298+0000 7f8b3b400640 1 -- 172.21.15.62:0/607238934 learned_addr learned my addr 172.21.15.62:0/607238934 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.298+0000 7f8b41200640 1 --2- 172.21.15.62:0/607238934 >> v2:172.21.15.120:3300/0 conn(0x7f8b3c0cf650 0x7f8b3c0cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.298+0000 7f8b3b400640 1 -- 172.21.15.62:0/607238934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b3c15c9e0 msgr2=0x7f8b3c1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.298+0000 7f8b3be00640 1 --2- 172.21.15.62:0/607238934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b3c15c9e0 0x7f8b3c1607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.298+0000 7f8b3b400640 1 --2- 172.21.15.62:0/607238934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b3c15c9e0 0x7f8b3c1607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.298+0000 7f8b3b400640 1 -- 172.21.15.62:0/607238934 >> v2:172.21.15.120:3300/0 conn(0x7f8b3c0cf650 msgr2=0x7f8b3c0cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.298+0000 7f8b3b400640 1 --2- 172.21.15.62:0/607238934 >> v2:172.21.15.120:3300/0 conn(0x7f8b3c0cf650 0x7f8b3c0cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.298+0000 7f8b3b400640 1 -- 172.21.15.62:0/607238934 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0ce140 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.298+0000 7f8b3be00640 1 --2- 172.21.15.62:0/607238934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b3c15c9e0 0x7f8b3c1607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:07.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.299+0000 7f8b3b400640 1 --2- 172.21.15.62:0/607238934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15cdc0 0x7f8b3c0d1f20 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c0b51d0 tx=0x7f8b2402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca16bd660d84e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:07.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.300+0000 7f8b3aa00640 1 -- 172.21.15.62:0/607238934 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b24031640 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.300+0000 7f8b3aa00640 1 -- 172.21.15.62:0/607238934 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b240317a0 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.300+0000 7f8b3aa00640 1 -- 172.21.15.62:0/607238934 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b24031b30 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.300+0000 7f8b43200640 1 -- 172.21.15.62:0/607238934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15cdc0 msgr2=0x7f8b3c0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.300+0000 7f8b43200640 1 --2- 172.21.15.62:0/607238934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15cdc0 0x7f8b3c0d1f20 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c0b51d0 tx=0x7f8b2402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.300+0000 7f8b43200640 1 -- 172.21.15.62:0/607238934 shutdown_connections 2024-03-23T11:29:07.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.300+0000 7f8b43200640 1 --2- 172.21.15.62:0/607238934 >> v2:172.21.15.120:3300/0 conn(0x7f8b3c0cf650 0x7f8b3c0cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.300+0000 7f8b43200640 1 --2- 172.21.15.62:0/607238934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15cdc0 0x7f8b3c0d1f20 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.300+0000 7f8b43200640 1 --2- 172.21.15.62:0/607238934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b3c15c9e0 0x7f8b3c1607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.300+0000 7f8b43200640 1 -- 172.21.15.62:0/607238934 >> 172.21.15.62:0/607238934 conn(0x7f8b3c071570 msgr2=0x7f8b3c071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:07.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.301+0000 7f8b43200640 1 -- 172.21.15.62:0/607238934 shutdown_connections 2024-03-23T11:29:07.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.301+0000 7f8b43200640 1 -- 172.21.15.62:0/607238934 wait complete. 2024-03-23T11:29:07.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.302+0000 7f8b43200640 1 Processor -- start 2024-03-23T11:29:07.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.302+0000 7f8b43200640 1 -- start start 2024-03-23T11:29:07.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.303+0000 7f8b43200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b3c0cf650 0x7f8b3c0aac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.303+0000 7f8b43200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3c15c9e0 0x7f8b3c0cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.303+0000 7f8b43200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15cdc0 0x7f8b3c0cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.303+0000 7f8b43200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c14c0e0 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.303+0000 7f8b43200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c1f33f0 con 0x7f8b3c15c9e0 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.303+0000 7f8b43200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c14c500 con 0x7f8b3c0cf650 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.304+0000 7f8b41200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15cdc0 0x7f8b3c0cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.304+0000 7f8b41200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15cdc0 0x7f8b3c0cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33418/0 (socket says 172.21.15.62:33418) 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.304+0000 7f8b41200640 1 -- 172.21.15.62:0/3856803342 learned_addr learned my addr 172.21.15.62:0/3856803342 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.304+0000 7f8b3be00640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b3c0cf650 0x7f8b3c0aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.304+0000 7f8b3b400640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3c15c9e0 0x7f8b3c0cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.304+0000 7f8b41200640 1 -- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b3c0cf650 msgr2=0x7f8b3c0aac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.304+0000 7f8b41200640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b3c0cf650 0x7f8b3c0aac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.304+0000 7f8b41200640 1 -- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3c15c9e0 msgr2=0x7f8b3c0cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.304+0000 7f8b41200640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3c15c9e0 0x7f8b3c0cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.304+0000 7f8b41200640 1 -- 172.21.15.62:0/3856803342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b240094d0 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.304+0000 7f8b41200640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15cdc0 0x7f8b3c0cbf10 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f8b34007e90 tx=0x7f8b3400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.305+0000 7f8b33e00640 1 -- 172.21.15.62:0/3856803342 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34014420 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.305+0000 7f8b33e00640 1 -- 172.21.15.62:0/3856803342 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b340145a0 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.305+0000 7f8b33e00640 1 -- 172.21.15.62:0/3856803342 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34014950 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.305+0000 7f8b43200640 1 -- 172.21.15.62:0/3856803342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c0ab150 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.305+0000 7f8b43200640 1 -- 172.21.15.62:0/3856803342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c14b740 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.306+0000 7f8b43200640 1 -- 172.21.15.62:0/3856803342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c066c90 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.307+0000 7f8b33e00640 1 -- 172.21.15.62:0/3856803342 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8b34014ab0 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.308+0000 7f8b33e00640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b1806a7a0 0x7f8b1806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.308+0000 7f8b33e00640 1 -- 172.21.15.62:0/3856803342 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f8b340894b0 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.309+0000 7f8b3be00640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b1806a7a0 0x7f8b1806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.310+0000 7f8b3be00640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b1806a7a0 0x7f8b1806cc60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00a990 tx=0x7f8b2c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:07.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.313+0000 7f8b33e00640 1 -- 172.21.15.62:0/3856803342 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8b34054b10 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.582+0000 7f8b43200640 1 -- 172.21.15.62:0/3856803342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) v1 -- 0x7f8b3c0cda60 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.613+0000 7f8b33e00640 1 -- 172.21.15.62:0/3856803342 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v473) v1 ==== 167+0+0 (secure 0 0 0) 0x7f8b34058130 con 0x7f8b3c15cdc0 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.615+0000 7f8b43200640 1 -- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b1806a7a0 msgr2=0x7f8b1806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.615+0000 7f8b43200640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b1806a7a0 0x7f8b1806cc60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00a990 tx=0x7f8b2c008040 comp rx=0 tx=0).stop 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.615+0000 7f8b43200640 1 -- 172.21.15.62:0/3856803342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15cdc0 msgr2=0x7f8b3c0cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.616+0000 7f8b43200640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15cdc0 0x7f8b3c0cbf10 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f8b34007e90 tx=0x7f8b3400e4b0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.616+0000 7f8b43200640 1 -- 172.21.15.62:0/3856803342 shutdown_connections 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.616+0000 7f8b43200640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b1806a7a0 0x7f8b1806cc60 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.616+0000 7f8b43200640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b3c15cdc0 0x7f8b3c0cbf10 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.616+0000 7f8b43200640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3c15c9e0 0x7f8b3c0cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.616+0000 7f8b43200640 1 --2- 172.21.15.62:0/3856803342 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b3c0cf650 0x7f8b3c0aac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.616+0000 7f8b43200640 1 -- 172.21.15.62:0/3856803342 >> 172.21.15.62:0/3856803342 conn(0x7f8b3c071570 msgr2=0x7f8b3c15bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.616+0000 7f8b43200640 1 -- 172.21.15.62:0/3856803342 shutdown_connections 2024-03-23T11:29:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.616+0000 7f8b43200640 1 -- 172.21.15.62:0/3856803342 wait complete. 2024-03-23T11:29:07.629 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-03-23T11:29:07.629 INFO:tasks.cephfs_test_runner:Starting test: test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-23T11:29:07.629 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-03-23T11:29:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.859+0000 7fbc9a200640 1 Processor -- start 2024-03-23T11:29:07.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.859+0000 7fbc9a200640 1 -- start start 2024-03-23T11:29:07.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.861+0000 7fbc9a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc94160310 0x7fbc940ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.861+0000 7fbc9a200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc941608d0 con 0x7fbc9415c830 2024-03-23T11:29:07.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.861+0000 7fbc9a200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc94160bb0 con 0x7fbc940ce250 2024-03-23T11:29:07.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.861+0000 7fbc9a200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc940addb0 con 0x7fbc94160310 2024-03-23T11:29:07.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.861+0000 7fbc98e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbc9415c830 0x7fbc940ab980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57232/0 (socket says 172.21.15.62:57232) 2024-03-23T11:29:07.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.861+0000 7fbc98e00640 1 -- 172.21.15.62:0/803681953 learned_addr learned my addr 172.21.15.62:0/803681953 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:07.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.861+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836567889 0 0) 0x7fbc941608d0 con 0x7fbc9415c830 2024-03-23T11:29:07.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc7c003520 con 0x7fbc9415c830 2024-03-23T11:29:07.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93e00640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc94160310 0x7fbc940ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4053923129 0 0) 0x7fbc7c003520 con 0x7fbc9415c830 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc94160310 msgr2=0x7fbc940ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc94160310 0x7fbc940ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 >> v1:172.21.15.120:6789/0 conn(0x7fbc940ce250 legacy=0x7fbc9415d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc94160e90 con 0x7fbc9415c830 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fbc88001670 con 0x7fbc9415c830 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 >> v1:172.21.15.62:6789/0 conn(0x7fbc9415c830 legacy=0x7fbc940ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc7c0039a0 0x7fbc7c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc94160310 0x7fbc7c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc7c004d00 0x7fbc7c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c00ba30 con 0x7fbc7c0039a0 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c00bba0 con 0x7fbc7c004d00 2024-03-23T11:29:07.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c00bd60 con 0x7fbc94160310 2024-03-23T11:29:07.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.862+0000 7fbc98e00640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc7c0039a0 0x7fbc7c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.863+0000 7fbc93e00640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc94160310 0x7fbc7c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.863+0000 7fbc98e00640 1 -- 172.21.15.62:0/803681953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc94160310 msgr2=0x7fbc7c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:07.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.863+0000 7fbc98e00640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc94160310 0x7fbc7c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.863+0000 7fbc98e00640 1 -- 172.21.15.62:0/803681953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc7c004d00 msgr2=0x7fbc7c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:07.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.863+0000 7fbc98e00640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc7c004d00 0x7fbc7c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.863+0000 7fbc98e00640 1 -- 172.21.15.62:0/803681953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c003520 con 0x7fbc7c0039a0 2024-03-23T11:29:07.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.863+0000 7fbc93e00640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc94160310 0x7fbc7c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:07.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.864+0000 7fbc98e00640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc7c0039a0 0x7fbc7c003dc0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fbc88001eb0 tx=0x7fbc7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684b7bf0c63f983e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:07.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.864+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88005ad0 con 0x7fbc7c0039a0 2024-03-23T11:29:07.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.864+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc88005c50 con 0x7fbc7c0039a0 2024-03-23T11:29:07.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.864+0000 7fbc93400640 1 -- 172.21.15.62:0/803681953 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8802c7c0 con 0x7fbc7c0039a0 2024-03-23T11:29:07.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.865+0000 7fbc9a200640 1 -- 172.21.15.62:0/803681953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc7c0039a0 msgr2=0x7fbc7c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.865+0000 7fbc9a200640 1 --2- 172.21.15.62:0/803681953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc7c0039a0 0x7fbc7c003dc0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fbc88001eb0 tx=0x7fbc7c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.865+0000 7fbc99800640 1 -- 172.21.15.62:0/803681953 reap_dead start 2024-03-23T11:29:07.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.865+0000 7fbc9a200640 1 -- 172.21.15.62:0/803681953 shutdown_connections 2024-03-23T11:29:07.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.865+0000 7fbc9a200640 1 -- 172.21.15.62:0/803681953 >> 172.21.15.62:0/803681953 conn(0x7fbc94066050 msgr2=0x7fbc94069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:07.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.865+0000 7fbc9a200640 1 -- 172.21.15.62:0/803681953 shutdown_connections 2024-03-23T11:29:07.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.865+0000 7fbc9a200640 1 -- 172.21.15.62:0/803681953 wait complete. 2024-03-23T11:29:07.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.867+0000 7fbc9a200640 1 Processor -- start 2024-03-23T11:29:07.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.867+0000 7fbc9a200640 1 -- start start 2024-03-23T11:29:07.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.868+0000 7fbc9a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c0039a0 0x7fbc940affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.868+0000 7fbc9a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc9414d9c0 0x7fbc9414d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.868+0000 7fbc9a200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc940c3b80 0x7fbc940c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.868+0000 7fbc9a200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc940c4540 con 0x7fbc9414d9c0 2024-03-23T11:29:07.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.868+0000 7fbc9a200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc940cc0d0 con 0x7fbc940c3b80 2024-03-23T11:29:07.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.868+0000 7fbc9a200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc940cb7e0 con 0x7fbc7c0039a0 2024-03-23T11:29:07.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.868+0000 7fbc93e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc9414d9c0 0x7fbc9414d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.868+0000 7fbc93e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc9414d9c0 0x7fbc9414d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33444/0 (socket says 172.21.15.62:33444) 2024-03-23T11:29:07.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.868+0000 7fbc93e00640 1 -- 172.21.15.62:0/135123310 learned_addr learned my addr 172.21.15.62:0/135123310 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:07.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc98e00640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c0039a0 0x7fbc940affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc99800640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc940c3b80 0x7fbc940c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc93e00640 1 -- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c0039a0 msgr2=0x7fbc940affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc93e00640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c0039a0 0x7fbc940affa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc93e00640 1 -- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc940c3b80 msgr2=0x7fbc940c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:07.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc93e00640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc940c3b80 0x7fbc940c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:07.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc93e00640 1 -- 172.21.15.62:0/135123310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9414e210 con 0x7fbc9414d9c0 2024-03-23T11:29:07.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc93e00640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc9414d9c0 0x7fbc9414d180 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fbc840108e0 tx=0x7fbc84010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:07.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc90c00640 1 -- 172.21.15.62:0/135123310 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8400ebc0 con 0x7fbc9414d9c0 2024-03-23T11:29:07.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc90c00640 1 -- 172.21.15.62:0/135123310 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8400ed20 con 0x7fbc9414d9c0 2024-03-23T11:29:07.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc90c00640 1 -- 172.21.15.62:0/135123310 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8401c600 con 0x7fbc9414d9c0 2024-03-23T11:29:07.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.869+0000 7fbc9a200640 1 -- 172.21.15.62:0/135123310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc940cba80 con 0x7fbc9414d9c0 2024-03-23T11:29:07.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.870+0000 7fbc9a200640 1 -- 172.21.15.62:0/135123310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc940b0630 con 0x7fbc9414d9c0 2024-03-23T11:29:07.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.871+0000 7fbc9a200640 1 -- 172.21.15.62:0/135123310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc940cc260 con 0x7fbc9414d9c0 2024-03-23T11:29:07.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.872+0000 7fbc90c00640 1 -- 172.21.15.62:0/135123310 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbc8401c760 con 0x7fbc9414d9c0 2024-03-23T11:29:07.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.873+0000 7fbc90c00640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc7806a7d0 0x7fbc7806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:07.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.873+0000 7fbc90c00640 1 -- 172.21.15.62:0/135123310 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fbc84089070 con 0x7fbc9414d9c0 2024-03-23T11:29:07.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.873+0000 7fbc98e00640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc7806a7d0 0x7fbc7806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:07.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.875+0000 7fbc98e00640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc7806a7d0 0x7fbc7806cc90 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbc88034000 tx=0x7fbc88009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:07.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:07.877+0000 7fbc90c00640 1 -- 172.21.15.62:0/135123310 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc840bc580 con 0x7fbc9414d9c0 2024-03-23T11:29:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.149+0000 7fbc9a200640 1 -- 172.21.15.62:0/135123310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) v1 -- 0x7fbc94066bb0 con 0x7fbc9414d9c0 2024-03-23T11:29:08.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.617+0000 7fbc90c00640 1 -- 172.21.15.62:0/135123310 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v474) v1 ==== 149+0+0 (secure 0 0 0) 0x7fbc84054780 con 0x7fbc9414d9c0 2024-03-23T11:29:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.620+0000 7fbc9a200640 1 -- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc7806a7d0 msgr2=0x7fbc7806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.620+0000 7fbc9a200640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc7806a7d0 0x7fbc7806cc90 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbc88034000 tx=0x7fbc88009030 comp rx=0 tx=0).stop 2024-03-23T11:29:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.620+0000 7fbc9a200640 1 -- 172.21.15.62:0/135123310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc9414d9c0 msgr2=0x7fbc9414d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.620+0000 7fbc9a200640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc9414d9c0 0x7fbc9414d180 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fbc840108e0 tx=0x7fbc84010da0 comp rx=0 tx=0).stop 2024-03-23T11:29:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.621+0000 7fbc9a200640 1 -- 172.21.15.62:0/135123310 shutdown_connections 2024-03-23T11:29:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.621+0000 7fbc9a200640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbc7806a7d0 0x7fbc7806cc90 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.621+0000 7fbc9a200640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc9414d9c0 0x7fbc9414d180 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.621+0000 7fbc9a200640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc940c3b80 0x7fbc940c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:08.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.621+0000 7fbc9a200640 1 --2- 172.21.15.62:0/135123310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc7c0039a0 0x7fbc940affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:08.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.621+0000 7fbc9a200640 1 -- 172.21.15.62:0/135123310 >> 172.21.15.62:0/135123310 conn(0x7fbc94066050 msgr2=0x7fbc9415e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:08.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.621+0000 7fbc9a200640 1 -- 172.21.15.62:0/135123310 shutdown_connections 2024-03-23T11:29:08.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.621+0000 7fbc9a200640 1 -- 172.21.15.62:0/135123310 wait complete. 2024-03-23T11:29:08.632 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T11:29:08.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.863+0000 7f83dc600640 1 Processor -- start 2024-03-23T11:29:08.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.864+0000 7f83dc600640 1 -- start start 2024-03-23T11:29:08.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.865+0000 7f83dc600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d415cdc0 0x7f83d40d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:08.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.865+0000 7f83dc600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f83d40cf650 0x7f83d40cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:08.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.865+0000 7f83dc600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d406bfc0 con 0x7f83d415c9e0 2024-03-23T11:29:08.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.865+0000 7f83dc600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d40d0020 con 0x7f83d40cf650 2024-03-23T11:29:08.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.865+0000 7f83dc600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d40ce140 con 0x7f83d415cdc0 2024-03-23T11:29:08.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.865+0000 7f83da600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f83d40cf650 0x7f83d40cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:08.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.865+0000 7f83d9c00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f83d415c9e0 0x7f83d41607c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57248/0 (socket says 172.21.15.62:57248) 2024-03-23T11:29:08.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.866+0000 7f83d9c00640 1 -- 172.21.15.62:0/1240316853 learned_addr learned my addr 172.21.15.62:0/1240316853 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.866+0000 7f83d9200640 1 --2- 172.21.15.62:0/1240316853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d415cdc0 0x7f83d40d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.866+0000 7f83da600640 1 -- 172.21.15.62:0/1240316853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d415cdc0 msgr2=0x7f83d40d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.866+0000 7f83da600640 1 --2- 172.21.15.62:0/1240316853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d415cdc0 0x7f83d40d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.866+0000 7f83da600640 1 -- 172.21.15.62:0/1240316853 >> v1:172.21.15.62:6789/0 conn(0x7f83d415c9e0 legacy=0x7f83d41607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.866+0000 7f83da600640 1 -- 172.21.15.62:0/1240316853 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d40b5260 con 0x7f83d40cf650 2024-03-23T11:29:08.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.867+0000 7f83da600640 1 --2- 172.21.15.62:0/1240316853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d40cf650 0x7f83d40cfae0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f83cc009d90 tx=0x7f83cc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db765d5d025a44 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:08.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.868+0000 7f83d3e00640 1 -- 172.21.15.62:0/1240316853 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc02cc50 con 0x7f83d40cf650 2024-03-23T11:29:08.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.868+0000 7f83d3e00640 1 -- 172.21.15.62:0/1240316853 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83cc02cdb0 con 0x7f83d40cf650 2024-03-23T11:29:08.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.868+0000 7f83d3e00640 1 -- 172.21.15.62:0/1240316853 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc034660 con 0x7f83d40cf650 2024-03-23T11:29:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.868+0000 7f83dc600640 1 -- 172.21.15.62:0/1240316853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d40cf650 msgr2=0x7f83d40cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.868+0000 7f83dc600640 1 --2- 172.21.15.62:0/1240316853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d40cf650 0x7f83d40cfae0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f83cc009d90 tx=0x7f83cc02bea0 comp rx=0 tx=0).stop 2024-03-23T11:29:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.868+0000 7f83dc600640 1 -- 172.21.15.62:0/1240316853 shutdown_connections 2024-03-23T11:29:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.868+0000 7f83dc600640 1 --2- 172.21.15.62:0/1240316853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d40cf650 0x7f83d40cfae0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.868+0000 7f83dc600640 1 --2- 172.21.15.62:0/1240316853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d415cdc0 0x7f83d40d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.868+0000 7f83dc600640 1 -- 172.21.15.62:0/1240316853 >> 172.21.15.62:0/1240316853 conn(0x7f83d4071570 msgr2=0x7f83d4071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.868+0000 7f83dc600640 1 -- 172.21.15.62:0/1240316853 shutdown_connections 2024-03-23T11:29:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.868+0000 7f83dc600640 1 -- 172.21.15.62:0/1240316853 wait complete. 2024-03-23T11:29:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.870+0000 7f83dc600640 1 Processor -- start 2024-03-23T11:29:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.870+0000 7f83dc600640 1 -- start start 2024-03-23T11:29:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.871+0000 7f83dc600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83d40cf650 0x7f83d41f1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.871+0000 7f83dc600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d415c9e0 0x7f83d414b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.871+0000 7f83dc600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d415cdc0 0x7f83d4160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.871+0000 7f83dc600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d414bc00 con 0x7f83d415cdc0 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.871+0000 7f83dc600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d414bee0 con 0x7f83d40cf650 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.871+0000 7f83dc600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d414c1c0 con 0x7f83d415c9e0 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.872+0000 7f83da600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d415cdc0 0x7f83d4160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.872+0000 7f83da600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d415cdc0 0x7f83d4160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33464/0 (socket says 172.21.15.62:33464) 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.872+0000 7f83da600640 1 -- 172.21.15.62:0/1769062721 learned_addr learned my addr 172.21.15.62:0/1769062721 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.872+0000 7f83d9c00640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83d40cf650 0x7f83d41f1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.872+0000 7f83d9200640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d415c9e0 0x7f83d414b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.872+0000 7f83da600640 1 -- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d415c9e0 msgr2=0x7f83d414b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.872+0000 7f83da600640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d415c9e0 0x7f83d414b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.872+0000 7f83da600640 1 -- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83d40cf650 msgr2=0x7f83d41f1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.872+0000 7f83da600640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83d40cf650 0x7f83d41f1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.872+0000 7f83da600640 1 -- 172.21.15.62:0/1769062721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83cc009a70 con 0x7f83d415cdc0 2024-03-23T11:29:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.872+0000 7f83d9200640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d415c9e0 0x7f83d414b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.873+0000 7f83da600640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d415cdc0 0x7f83d4160730 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f83cc009d90 tx=0x7f83cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.873+0000 7f83d1600640 1 -- 172.21.15.62:0/1769062721 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc040650 con 0x7f83d415cdc0 2024-03-23T11:29:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.873+0000 7f83d1600640 1 -- 172.21.15.62:0/1769062721 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83cc0407d0 con 0x7f83d415cdc0 2024-03-23T11:29:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.873+0000 7f83d1600640 1 -- 172.21.15.62:0/1769062721 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc040b80 con 0x7f83d415cdc0 2024-03-23T11:29:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.873+0000 7f83dc600640 1 -- 172.21.15.62:0/1769062721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d414c4a0 con 0x7f83d415cdc0 2024-03-23T11:29:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.873+0000 7f83dc600640 1 -- 172.21.15.62:0/1769062721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d40a9530 con 0x7f83d415cdc0 2024-03-23T11:29:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.874+0000 7f83dc600640 1 -- 172.21.15.62:0/1769062721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d414df40 con 0x7f83d415cdc0 2024-03-23T11:29:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.875+0000 7f83d1600640 1 -- 172.21.15.62:0/1769062721 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f83cc040ce0 con 0x7f83d415cdc0 2024-03-23T11:29:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.876+0000 7f83d1600640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f83b406a7a0 0x7f83b406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.876+0000 7f83d1600640 1 -- 172.21.15.62:0/1769062721 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f83cc0ab990 con 0x7f83d415cdc0 2024-03-23T11:29:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.876+0000 7f83d9c00640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f83b406a7a0 0x7f83b406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.877+0000 7f83d9c00640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f83b406a7a0 0x7f83b406cc60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f83c8005880 tx=0x7f83c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:08.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:08.881+0000 7f83d1600640 1 -- 172.21.15.62:0/1769062721 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f83d414df40 con 0x7f83d415cdc0 2024-03-23T11:29:09.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.084+0000 7f83dc600640 1 -- 172.21.15.62:0/1769062721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f83d414df40 con 0x7f83d415cdc0 2024-03-23T11:29:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.094+0000 7f83d1600640 1 -- 172.21.15.62:0/1769062721 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f83d414df40 con 0x7f83d415cdc0 2024-03-23T11:29:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 -- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f83b406a7a0 msgr2=0x7f83b406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f83b406a7a0 0x7f83b406cc60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f83c8005880 tx=0x7f83c8009040 comp rx=0 tx=0).stop 2024-03-23T11:29:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 -- 172.21.15.62:0/1769062721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d415cdc0 msgr2=0x7f83d4160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d415cdc0 0x7f83d4160730 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f83cc009d90 tx=0x7f83cc004a40 comp rx=0 tx=0).stop 2024-03-23T11:29:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 -- 172.21.15.62:0/1769062721 shutdown_connections 2024-03-23T11:29:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f83b406a7a0 0x7f83b406cc60 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d415cdc0 0x7f83d4160730 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83d415c9e0 0x7f83d414b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 --2- 172.21.15.62:0/1769062721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83d40cf650 0x7f83d41f1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 -- 172.21.15.62:0/1769062721 >> 172.21.15.62:0/1769062721 conn(0x7f83d4071570 msgr2=0x7f83d415bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 -- 172.21.15.62:0/1769062721 shutdown_connections 2024-03-23T11:29:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.097+0000 7f83dc600640 1 -- 172.21.15.62:0/1769062721 wait complete. 2024-03-23T11:29:09.108 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:29:09.108 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:29:09.230 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:29:09.347 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:29:09.347 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:29:09.347 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:09.408 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:29:09.408 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:29:09.408 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:09.464 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:29:09.464 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:29:09.464 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:09.523 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:29:09.523 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:29:09.523 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:09.577 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:29:09.577 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T11:29:09.577 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:29:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.814+0000 7f6d5f200640 1 Processor -- start 2024-03-23T11:29:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.814+0000 7f6d5f200640 1 -- start start 2024-03-23T11:29:09.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.816+0000 7f6d5f200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d60160420 0x7f6d601608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:09.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.816+0000 7f6d5f200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d600ce510 con 0x7f6d600b5db0 2024-03-23T11:29:09.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.816+0000 7f6d5f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d600cdc70 con 0x7f6d6015fcf0 2024-03-23T11:29:09.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.816+0000 7f6d5f200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d60160df0 con 0x7f6d60160420 2024-03-23T11:29:09.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.816+0000 7f6d55400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d60160420 0x7f6d601608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:09.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.816+0000 7f6d5e800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6d600b5db0 0x7f6d600b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57256/0 (socket says 172.21.15.62:57256) 2024-03-23T11:29:09.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.816+0000 7f6d5e800640 1 -- 172.21.15.62:0/647068255 learned_addr learned my addr 172.21.15.62:0/647068255 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:09.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.817+0000 7f6d55400640 1 -- 172.21.15.62:0/647068255 >> v1:172.21.15.120:6789/0 conn(0x7f6d6015fcf0 legacy=0x7f6d600d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:09.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.817+0000 7f6d55400640 1 -- 172.21.15.62:0/647068255 >> v1:172.21.15.62:6789/0 conn(0x7f6d600b5db0 legacy=0x7f6d600b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:09.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.817+0000 7f6d55400640 1 -- 172.21.15.62:0/647068255 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d600cdf50 con 0x7f6d60160420 2024-03-23T11:29:09.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.818+0000 7f6d55400640 1 --2- 172.21.15.62:0/647068255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60160420 0x7f6d601608b0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f6d480097a0 tx=0x7f6d4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4af0208c8a4feb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:09.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.818+0000 7f6d5d400640 1 -- 172.21.15.62:0/647068255 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48031830 con 0x7f6d60160420 2024-03-23T11:29:09.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.818+0000 7f6d5d400640 1 -- 172.21.15.62:0/647068255 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d48031990 con 0x7f6d60160420 2024-03-23T11:29:09.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.818+0000 7f6d5d400640 1 -- 172.21.15.62:0/647068255 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48031d20 con 0x7f6d60160420 2024-03-23T11:29:09.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.819+0000 7f6d5f200640 1 -- 172.21.15.62:0/647068255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60160420 msgr2=0x7f6d601608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:09.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.819+0000 7f6d5f200640 1 --2- 172.21.15.62:0/647068255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60160420 0x7f6d601608b0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f6d480097a0 tx=0x7f6d4802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:29:09.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.819+0000 7f6d5f200640 1 -- 172.21.15.62:0/647068255 shutdown_connections 2024-03-23T11:29:09.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.819+0000 7f6d5f200640 1 --2- 172.21.15.62:0/647068255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60160420 0x7f6d601608b0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:09.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.819+0000 7f6d5f200640 1 -- 172.21.15.62:0/647068255 >> 172.21.15.62:0/647068255 conn(0x7f6d60089f60 msgr2=0x7f6d600d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:09.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.819+0000 7f6d5f200640 1 -- 172.21.15.62:0/647068255 shutdown_connections 2024-03-23T11:29:09.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.819+0000 7f6d5f200640 1 -- 172.21.15.62:0/647068255 wait complete. 2024-03-23T11:29:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.821+0000 7f6d5f200640 1 Processor -- start 2024-03-23T11:29:09.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.821+0000 7f6d5f200640 1 -- start start 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.822+0000 7f6d5f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d600b5db0 0x7f6d6015ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.822+0000 7f6d5f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d6015fcf0 0x7f6d600b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.822+0000 7f6d5f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d60160420 0x7f6d600c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.822+0000 7f6d5f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600a0090 con 0x7f6d600b5db0 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.822+0000 7f6d5f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6015e300 con 0x7f6d6015fcf0 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.822+0000 7f6d5f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6009f760 con 0x7f6d60160420 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.822+0000 7f6d5de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d600b5db0 0x7f6d6015ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.822+0000 7f6d5de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d600b5db0 0x7f6d6015ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33484/0 (socket says 172.21.15.62:33484) 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.822+0000 7f6d5de00640 1 -- 172.21.15.62:0/26232950 learned_addr learned my addr 172.21.15.62:0/26232950 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.822+0000 7f6d55400640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d6015fcf0 0x7f6d600b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d5e800640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d60160420 0x7f6d600c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d5de00640 1 -- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d60160420 msgr2=0x7f6d600c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:09.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d5de00640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d60160420 0x7f6d600c80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d5de00640 1 -- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d6015fcf0 msgr2=0x7f6d600b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d5de00640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d6015fcf0 0x7f6d600b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d5de00640 1 -- 172.21.15.62:0/26232950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d48009400 con 0x7f6d600b5db0 2024-03-23T11:29:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d5de00640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d600b5db0 0x7f6d6015ddc0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000c450 tx=0x7f6d5000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d56a00640 1 -- 172.21.15.62:0/26232950 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d500052d0 con 0x7f6d600b5db0 2024-03-23T11:29:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d56a00640 1 -- 172.21.15.62:0/26232950 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d50005430 con 0x7f6d600b5db0 2024-03-23T11:29:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d5f200640 1 -- 172.21.15.62:0/26232950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6009fa40 con 0x7f6d600b5db0 2024-03-23T11:29:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d56a00640 1 -- 172.21.15.62:0/26232950 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d50012870 con 0x7f6d600b5db0 2024-03-23T11:29:09.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.823+0000 7f6d5f200640 1 -- 172.21.15.62:0/26232950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d600b6fd0 con 0x7f6d600b5db0 2024-03-23T11:29:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.824+0000 7f6d5f200640 1 -- 172.21.15.62:0/26232950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d600a0200 con 0x7f6d600b5db0 2024-03-23T11:29:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.825+0000 7f6d56a00640 1 -- 172.21.15.62:0/26232950 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6d500129d0 con 0x7f6d600b5db0 2024-03-23T11:29:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.826+0000 7f6d56a00640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d3c06a7a0 0x7f6d3c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:09.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.826+0000 7f6d56a00640 1 -- 172.21.15.62:0/26232950 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f6d500886c0 con 0x7f6d600b5db0 2024-03-23T11:29:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.827+0000 7f6d55400640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d3c06a7a0 0x7f6d3c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:09.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.828+0000 7f6d55400640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d3c06a7a0 0x7f6d3c06cc60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6d48036000 tx=0x7f6d4803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:09.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:09.831+0000 7f6d56a00640 1 -- 172.21.15.62:0/26232950 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6d50053d20 con 0x7f6d600b5db0 2024-03-23T11:29:10.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.066+0000 7f6d5f200640 1 -- 172.21.15.62:0/26232950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d60066bb0 con 0x7f6d600b5db0 2024-03-23T11:29:10.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.067+0000 7f6d56a00640 1 -- 172.21.15.62:0/26232950 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f6d50057340 con 0x7f6d600b5db0 2024-03-23T11:29:10.059 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:10.059 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":72,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5552,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4274280849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4274280849},{"type":"v1","addr":"172.21.15.62:6835","nonce":4274280849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/2782258283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2782258283},{"type":"v1","addr":"172.21.15.120:6835","nonce":2782258283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6151,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/713128344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":713128344},{"type":"v1","addr":"172.21.15.120:6837","nonce":713128344}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":8223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/554675928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":554675928},{"type":"v1","addr":"172.21.15.62:6833","nonce":554675928}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[]} 2024-03-23T11:29:10.059 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 72 2024-03-23T11:29:10.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.069+0000 7f6d5f200640 1 -- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d3c06a7a0 msgr2=0x7f6d3c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:10.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.069+0000 7f6d5f200640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d3c06a7a0 0x7f6d3c06cc60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6d48036000 tx=0x7f6d4803c040 comp rx=0 tx=0).stop 2024-03-23T11:29:10.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.070+0000 7f6d5f200640 1 -- 172.21.15.62:0/26232950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d600b5db0 msgr2=0x7f6d6015ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:10.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.070+0000 7f6d5f200640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d600b5db0 0x7f6d6015ddc0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000c450 tx=0x7f6d5000c910 comp rx=0 tx=0).stop 2024-03-23T11:29:10.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.070+0000 7f6d5f200640 1 -- 172.21.15.62:0/26232950 shutdown_connections 2024-03-23T11:29:10.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.070+0000 7f6d5f200640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d3c06a7a0 0x7f6d3c06cc60 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:10.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.070+0000 7f6d5f200640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6d60160420 0x7f6d600c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:10.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.070+0000 7f6d5f200640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d6015fcf0 0x7f6d600b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:10.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.070+0000 7f6d5f200640 1 --2- 172.21.15.62:0/26232950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d600b5db0 0x7f6d6015ddc0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:10.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.070+0000 7f6d5f200640 1 -- 172.21.15.62:0/26232950 >> 172.21.15.62:0/26232950 conn(0x7f6d60089f60 msgr2=0x7f6d600d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:10.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.070+0000 7f6d5f200640 1 -- 172.21.15.62:0/26232950 shutdown_connections 2024-03-23T11:29:10.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.070+0000 7f6d5f200640 1 -- 172.21.15.62:0/26232950 wait complete. 2024-03-23T11:29:10.082 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T11:29:10.082 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T11:29:10.082 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T11:29:10.082 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:29:10.082 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T11:29:10.082 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T11:29:10.083 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T11:29:10.083 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:29:10.083 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T11:29:10.083 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T11:29:10.083 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T11:29:10.083 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:29:10.083 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T11:29:10.083 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T11:29:10.084 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T11:29:10.084 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:29:10.100 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T11:29:10.100 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T11:29:10.104 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T11:29:10.104 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T11:29:10.108 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T11:29:10.108 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T11:29:10.116 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T11:29:10.117 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T11:29:10.149 INFO:tasks.ceph.mds.c:Started 2024-03-23T11:29:10.150 DEBUG:teuthology.parallel:result is None 2024-03-23T11:29:10.156 INFO:tasks.ceph.mds.a:Started 2024-03-23T11:29:10.156 DEBUG:teuthology.parallel:result is None 2024-03-23T11:29:10.163 INFO:tasks.ceph.mds.b:Started 2024-03-23T11:29:10.164 DEBUG:teuthology.parallel:result is None 2024-03-23T11:29:10.169 INFO:tasks.ceph.mds.d:Started 2024-03-23T11:29:10.169 DEBUG:teuthology.parallel:result is None 2024-03-23T11:29:10.170 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T11:29:10.312 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:29:10.313 INFO:tasks.ceph.mds.c.smithi062.stdout:starting mds.c at 2024-03-23T11:29:10.338 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-23T11:29:10.338 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-23T11:29:10.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.376+0000 7fa28d400640 1 Processor -- start 2024-03-23T11:29:10.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.376+0000 7fa28d400640 1 -- start start 2024-03-23T11:29:10.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.377+0000 7fa28d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa28815cb90 0x7fa2880b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:10.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.377+0000 7fa28d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2880cde10 0x7fa2880d1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:10.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.377+0000 7fa28d400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2880b5ad0 con 0x7fa28806b040 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.377+0000 7fa28d400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa28806c0c0 con 0x7fa28815cb90 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.377+0000 7fa28d400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2880d1010 con 0x7fa2880cde10 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.378+0000 7fa286800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa28815cb90 0x7fa2880b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.378+0000 7fa286800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa28815cb90 0x7fa2880b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33530/0 (socket says 172.21.15.62:33530) 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.378+0000 7fa286800640 1 -- 172.21.15.62:0/2549369369 learned_addr learned my addr 172.21.15.62:0/2549369369 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.378+0000 7fa285e00640 1 --1- 172.21.15.62:0/2549369369 >> v1:172.21.15.62:6789/0 conn(0x7fa28806b040 0x7fa2880cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57258/0 (socket says 172.21.15.62:57258) 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.378+0000 7fa287200640 1 --2- 172.21.15.62:0/2549369369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2880cde10 0x7fa2880d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.378+0000 7fa286800640 1 -- 172.21.15.62:0/2549369369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2880cde10 msgr2=0x7fa2880d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.378+0000 7fa286800640 1 --2- 172.21.15.62:0/2549369369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2880cde10 0x7fa2880d1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.378+0000 7fa286800640 1 -- 172.21.15.62:0/2549369369 >> v1:172.21.15.62:6789/0 conn(0x7fa28806b040 legacy=0x7fa2880cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.378+0000 7fa286800640 1 -- 172.21.15.62:0/2549369369 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28806b820 con 0x7fa28815cb90 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.378+0000 7fa287200640 1 --2- 172.21.15.62:0/2549369369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2880cde10 0x7fa2880d1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:29:10.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.378+0000 7fa285400640 1 -- 172.21.15.62:0/2549369369 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32286864 0 0) 0x7fa2880b5ad0 con 0x7fa28806b040 2024-03-23T11:29:10.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.379+0000 7fa286800640 1 --2- 172.21.15.62:0/2549369369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa28815cb90 0x7fa2880b84f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fa278009870 tx=0x7fa27802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a8aad1482dffaf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.380+0000 7fa285400640 1 -- 172.21.15.62:0/2549369369 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278030840 con 0x7fa28815cb90 2024-03-23T11:29:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.380+0000 7fa285400640 1 -- 172.21.15.62:0/2549369369 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2780309a0 con 0x7fa28815cb90 2024-03-23T11:29:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.380+0000 7fa285400640 1 -- 172.21.15.62:0/2549369369 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278030d30 con 0x7fa28815cb90 2024-03-23T11:29:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.380+0000 7fa28d400640 1 -- 172.21.15.62:0/2549369369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa28815cb90 msgr2=0x7fa2880b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.380+0000 7fa28d400640 1 --2- 172.21.15.62:0/2549369369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa28815cb90 0x7fa2880b84f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fa278009870 tx=0x7fa27802b980 comp rx=0 tx=0).stop 2024-03-23T11:29:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.380+0000 7fa28d400640 1 -- 172.21.15.62:0/2549369369 shutdown_connections 2024-03-23T11:29:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.380+0000 7fa28d400640 1 --2- 172.21.15.62:0/2549369369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2880cde10 0x7fa2880d1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.380+0000 7fa28d400640 1 --2- 172.21.15.62:0/2549369369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa28815cb90 0x7fa2880b84f0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.380+0000 7fa28d400640 1 -- 172.21.15.62:0/2549369369 >> 172.21.15.62:0/2549369369 conn(0x7fa2880cf7d0 msgr2=0x7fa2880cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.380+0000 7fa28d400640 1 -- 172.21.15.62:0/2549369369 shutdown_connections 2024-03-23T11:29:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.380+0000 7fa28d400640 1 -- 172.21.15.62:0/2549369369 wait complete. 2024-03-23T11:29:10.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.382+0000 7fa28d400640 1 Processor -- start 2024-03-23T11:29:10.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.382+0000 7fa28d400640 1 -- start start 2024-03-23T11:29:10.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.383+0000 7fa28d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa28806b040 0x7fa28815eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:10.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.383+0000 7fa28d400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2880cde10 0x7fa288153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:10.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.383+0000 7fa28d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa28815cb90 0x7fa288153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.383+0000 7fa28d400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa288153fc0 con 0x7fa28806b040 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.383+0000 7fa28d400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2881542a0 con 0x7fa2880cde10 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.383+0000 7fa28d400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa288154580 con 0x7fa28815cb90 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.383+0000 7fa286800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa28806b040 0x7fa28815eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.384+0000 7fa286800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa28806b040 0x7fa28815eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33538/0 (socket says 172.21.15.62:33538) 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.384+0000 7fa286800640 1 -- 172.21.15.62:0/312963564 learned_addr learned my addr 172.21.15.62:0/312963564 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.384+0000 7fa285e00640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2880cde10 0x7fa288153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.384+0000 7fa287200640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa28815cb90 0x7fa288153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.384+0000 7fa286800640 1 -- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa28815cb90 msgr2=0x7fa288153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.384+0000 7fa286800640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa28815cb90 0x7fa288153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.384+0000 7fa286800640 1 -- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2880cde10 msgr2=0x7fa288153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.384+0000 7fa286800640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2880cde10 0x7fa288153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.384+0000 7fa286800640 1 -- 172.21.15.62:0/312963564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2780094d0 con 0x7fa28806b040 2024-03-23T11:29:10.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.384+0000 7fa286800640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa28806b040 0x7fa28815eaf0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fa278009870 tx=0x7fa278004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:10.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.385+0000 7fa27ea00640 1 -- 172.21.15.62:0/312963564 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278039c90 con 0x7fa28806b040 2024-03-23T11:29:10.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.385+0000 7fa27ea00640 1 -- 172.21.15.62:0/312963564 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa278039df0 con 0x7fa28806b040 2024-03-23T11:29:10.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.385+0000 7fa28d400640 1 -- 172.21.15.62:0/312963564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa288154860 con 0x7fa28806b040 2024-03-23T11:29:10.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.385+0000 7fa27ea00640 1 -- 172.21.15.62:0/312963564 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278043620 con 0x7fa28806b040 2024-03-23T11:29:10.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.385+0000 7fa28d400640 1 -- 172.21.15.62:0/312963564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28815f030 con 0x7fa28806b040 2024-03-23T11:29:10.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.386+0000 7fa28d400640 1 -- 172.21.15.62:0/312963564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa288066c90 con 0x7fa28806b040 2024-03-23T11:29:10.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.387+0000 7fa27ea00640 1 -- 172.21.15.62:0/312963564 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa278043780 con 0x7fa28806b040 2024-03-23T11:29:10.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.388+0000 7fa27ea00640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa26806a7a0 0x7fa26806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:10.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.388+0000 7fa27ea00640 1 -- 172.21.15.62:0/312963564 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fa27803e070 con 0x7fa28806b040 2024-03-23T11:29:10.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.389+0000 7fa285e00640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa26806a7a0 0x7fa26806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:10.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.389+0000 7fa285e00640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa26806a7a0 0x7fa26806cc60 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa2880c62e0 tx=0x7fa274007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:10.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.392+0000 7fa27ea00640 1 -- 172.21.15.62:0/312963564 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa278077260 con 0x7fa28806b040 2024-03-23T11:29:10.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:10.601+0000 7fa28d400640 1 -- 172.21.15.62:0/312963564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa2880cfea0 con 0x7fa28806b040 2024-03-23T11:29:11.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.376+0000 7fa27ea00640 1 -- 172.21.15.62:0/312963564 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v101) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa27807a880 con 0x7fa28806b040 2024-03-23T11:29:11.368 INFO:teuthology.orchestra.run.smithi062.stderr: removed all blocklist entries 2024-03-23T11:29:11.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.380+0000 7fa28d400640 1 -- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa26806a7a0 msgr2=0x7fa26806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:11.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.380+0000 7fa28d400640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa26806a7a0 0x7fa26806cc60 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa2880c62e0 tx=0x7fa274007de0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.380+0000 7fa28d400640 1 -- 172.21.15.62:0/312963564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa28806b040 msgr2=0x7fa28815eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:11.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.380+0000 7fa28d400640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa28806b040 0x7fa28815eaf0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fa278009870 tx=0x7fa278004730 comp rx=0 tx=0).stop 2024-03-23T11:29:11.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.381+0000 7fa28d400640 1 -- 172.21.15.62:0/312963564 shutdown_connections 2024-03-23T11:29:11.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.381+0000 7fa28d400640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa26806a7a0 0x7fa26806cc60 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.381+0000 7fa28d400640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa28815cb90 0x7fa288153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.381+0000 7fa28d400640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2880cde10 0x7fa288153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.381+0000 7fa28d400640 1 --2- 172.21.15.62:0/312963564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa28806b040 0x7fa28815eaf0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.381+0000 7fa28d400640 1 -- 172.21.15.62:0/312963564 >> 172.21.15.62:0/312963564 conn(0x7fa2880cf7d0 msgr2=0x7fa2880b7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:11.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.381+0000 7fa28d400640 1 -- 172.21.15.62:0/312963564 shutdown_connections 2024-03-23T11:29:11.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.381+0000 7fa28d400640 1 -- 172.21.15.62:0/312963564 wait complete. 2024-03-23T11:29:11.397 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T11:29:11.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.629+0000 7efed6c00640 1 Processor -- start 2024-03-23T11:29:11.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.629+0000 7efed6c00640 1 -- start start 2024-03-23T11:29:11.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.631+0000 7efed6c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed0143dd0 0x7efed015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:11.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.631+0000 7efed6c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efed0070090 0x7efed00704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:11.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.631+0000 7efed6c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0070a30 con 0x7efed00b2820 2024-03-23T11:29:11.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.631+0000 7efed6c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed00cd140 con 0x7efed0070090 2024-03-23T11:29:11.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.631+0000 7efed6c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed00cd3e0 con 0x7efed0143dd0 2024-03-23T11:29:11.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.631+0000 7efed6200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7efed00b2820 0x7efed00b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57274/0 (socket says 172.21.15.62:57274) 2024-03-23T11:29:11.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.631+0000 7efed6200640 1 -- 172.21.15.62:0/3289170830 learned_addr learned my addr 172.21.15.62:0/3289170830 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:11.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.632+0000 7efed4e00640 1 --2- 172.21.15.62:0/3289170830 >> v2:172.21.15.120:3300/0 conn(0x7efed0070090 0x7efed00704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:11.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.632+0000 7efed5800640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed0143dd0 0x7efed015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:11.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.632+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129466022 0 0) 0x7efed0070a30 con 0x7efed00b2820 2024-03-23T11:29:11.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.632+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb8003520 con 0x7efed00b2820 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 688362627 0 0) 0x7efeb8003520 con 0x7efed00b2820 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed0143dd0 msgr2=0x7efed015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed0143dd0 0x7efed015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 >> v2:172.21.15.120:3300/0 conn(0x7efed0070090 msgr2=0x7efed00704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 --2- 172.21.15.62:0/3289170830 >> v2:172.21.15.120:3300/0 conn(0x7efed0070090 0x7efed00704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed00cd6c0 con 0x7efed00b2820 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7efec80019f0 con 0x7efed00b2820 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 >> v1:172.21.15.62:6789/0 conn(0x7efed00b2820 legacy=0x7efed00b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb8003970 0x7efeb8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed0143dd0 0x7efeb80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb8004d00 0x7efeb8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed00cd140 con 0x7efeb8003970 2024-03-23T11:29:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed00cd3e0 con 0x7efeb8004d00 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb800b980 con 0x7efed0143dd0 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.633+0000 7efed5800640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb8003970 0x7efeb8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.634+0000 7efed6200640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb8004d00 0x7efeb8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.634+0000 7efed5800640 1 -- 172.21.15.62:0/3289170830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed0143dd0 msgr2=0x7efeb80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.634+0000 7efed5800640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efed0143dd0 0x7efeb80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.634+0000 7efed5800640 1 -- 172.21.15.62:0/3289170830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb8004d00 msgr2=0x7efeb8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.634+0000 7efed5800640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb8004d00 0x7efeb8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.634+0000 7efed5800640 1 -- 172.21.15.62:0/3289170830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed00cd6c0 con 0x7efeb8003970 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.634+0000 7efed6200640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb8004d00 0x7efeb8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.635+0000 7efed5800640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb8003970 0x7efeb8003d90 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7efec4009ac0 tx=0x7efeb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6070e0efa3bd3ee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.635+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec402c710 con 0x7efeb8003970 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.635+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec402c870 con 0x7efeb8003970 2024-03-23T11:29:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.635+0000 7efecfe00640 1 -- 172.21.15.62:0/3289170830 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec402cc00 con 0x7efeb8003970 2024-03-23T11:29:11.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.636+0000 7efed6c00640 1 -- 172.21.15.62:0/3289170830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb8003970 msgr2=0x7efeb8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:11.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.636+0000 7efed6c00640 1 --2- 172.21.15.62:0/3289170830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efeb8003970 0x7efeb8003d90 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7efec4009ac0 tx=0x7efeb80057f0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.636+0000 7efed6200640 1 -- 172.21.15.62:0/3289170830 reap_dead start 2024-03-23T11:29:11.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.636+0000 7efed6c00640 1 -- 172.21.15.62:0/3289170830 shutdown_connections 2024-03-23T11:29:11.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.636+0000 7efed6c00640 1 -- 172.21.15.62:0/3289170830 >> 172.21.15.62:0/3289170830 conn(0x7efed0066050 msgr2=0x7efed0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:11.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.636+0000 7efed6c00640 1 -- 172.21.15.62:0/3289170830 shutdown_connections 2024-03-23T11:29:11.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.636+0000 7efed6c00640 1 -- 172.21.15.62:0/3289170830 wait complete. 2024-03-23T11:29:11.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.638+0000 7efed6c00640 1 Processor -- start 2024-03-23T11:29:11.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.638+0000 7efed6c00640 1 -- start start 2024-03-23T11:29:11.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.639+0000 7efed6c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeb8003970 0x7efed013df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:11.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.639+0000 7efed6c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed006c8a0 0x7efed00b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:11.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.639+0000 7efed6c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed006cd20 0x7efed00b0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:11.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.639+0000 7efed6c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed01f1d20 con 0x7efed006cd20 2024-03-23T11:29:11.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.639+0000 7efed6c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed01f1a40 con 0x7efed006c8a0 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.639+0000 7efed6c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed013d460 con 0x7efeb8003970 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.639+0000 7efed6200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed006cd20 0x7efed00b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.639+0000 7efed6200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed006cd20 0x7efed00b0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33558/0 (socket says 172.21.15.62:33558) 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.639+0000 7efed6200640 1 -- 172.21.15.62:0/478938093 learned_addr learned my addr 172.21.15.62:0/478938093 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efed4e00640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed006c8a0 0x7efed00b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efed5800640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeb8003970 0x7efed013df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efed6200640 1 -- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeb8003970 msgr2=0x7efed013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efed6200640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeb8003970 0x7efed013df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efed6200640 1 -- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed006c8a0 msgr2=0x7efed00b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efed6200640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed006c8a0 0x7efed00b03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efed6200640 1 -- 172.21.15.62:0/478938093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed013d740 con 0x7efed006cd20 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efed4e00640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed006c8a0 0x7efed00b03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efed6200640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed006cd20 0x7efed00b0910 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7efed006e380 tx=0x7efec800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:11.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efecd600640 1 -- 172.21.15.62:0/478938093 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec800bb80 con 0x7efed006cd20 2024-03-23T11:29:11.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efecd600640 1 -- 172.21.15.62:0/478938093 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec800bd00 con 0x7efed006cd20 2024-03-23T11:29:11.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.640+0000 7efecd600640 1 -- 172.21.15.62:0/478938093 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8007350 con 0x7efed006cd20 2024-03-23T11:29:11.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.641+0000 7efed6c00640 1 -- 172.21.15.62:0/478938093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed013da20 con 0x7efed006cd20 2024-03-23T11:29:11.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.641+0000 7efed6c00640 1 -- 172.21.15.62:0/478938093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed013e660 con 0x7efed006cd20 2024-03-23T11:29:11.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.642+0000 7efed6c00640 1 -- 172.21.15.62:0/478938093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed013d2c0 con 0x7efed006cd20 2024-03-23T11:29:11.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.643+0000 7efecd600640 1 -- 172.21.15.62:0/478938093 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7efec80074b0 con 0x7efed006cd20 2024-03-23T11:29:11.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.644+0000 7efecd600640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efeb006a7a0 0x7efeb006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:11.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.644+0000 7efecd600640 1 -- 172.21.15.62:0/478938093 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efec8088dd0 con 0x7efed006cd20 2024-03-23T11:29:11.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.644+0000 7efed5800640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efeb006a7a0 0x7efeb006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:11.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.645+0000 7efed5800640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efeb006a7a0 0x7efeb006cc60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7efec4036000 tx=0x7efec4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:11.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.648+0000 7efecd600640 1 -- 172.21.15.62:0/478938093 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efed013d2c0 con 0x7efed006cd20 2024-03-23T11:29:11.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.907+0000 7efed6c00640 1 -- 172.21.15.62:0/478938093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7efed013e660 con 0x7efed006cd20 2024-03-23T11:29:11.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.908+0000 7efecd600640 1 -- 172.21.15.62:0/478938093 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7efed013e660 con 0x7efed006cd20 2024-03-23T11:29:11.900 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:11.900 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:29:11.900 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_dump": [ 2024-03-23T11:29:11.900 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.900 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.a", 2024-03-23T11:29:11.900 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lYNxYERAAPVHWwRHM5V7OejL294b2zg==", 2024-03-23T11:29:11.900 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.900 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:29:11.900 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:29:11.900 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.b", 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lmOUANBAAifuZ0DUPD9az/mturTndSA==", 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.901 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.c", 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5ltqrjIhAA9+f82fvlrFlIEXqswPE0ig==", 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.902 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.d", 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5l7ly4CBAA8+oim0z+bbljfP0VH7XSaw==", 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.0", 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBiuv5lRfaQBRAAvrtCZXwt00PUQfCqqKBzMQ==", 2024-03-23T11:29:11.903 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.1", 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBluv5lzwS/IBAAmYU8yMLtCUJNr/AyE0MJDA==", 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.904 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.2", 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBouv5lyAy1OhAAJoUsg6DFpEPxAcH5oI2exQ==", 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.3", 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBsuv5lb0YcGBAA220D0uRhAZpG4w97wqsB5A==", 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:29:11.905 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.4", 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBzuv5lC8RIGhAAH+5HaeCgQhrQMrYm9iWaYA==", 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.906 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.5", 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB2uv5lYi7ZIhAAMNIm/jPckCAbX9oEx5Li/Q==", 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.6", 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB5uv5lgoRtLBAAFT7uJtyqHtHAegOYyDH/WQ==", 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.907 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.7", 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB9uv5l/dzrCBAA3jsua0LvBPI7I551/rRHrg==", 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.908 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.0", 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5ltavQFhAAvV2RWuHOPK2T9Q7gWkYA4w==", 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow r", 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.1", 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5luQ/zLRAAwjDtV64TCKnQliPq4iUAYg==", 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.909 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:29:11.910 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:29:11.910 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:29:11.910 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:29:11.910 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.910 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.910 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.910 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.2", 2024-03-23T11:29:11.910 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lDRG1IhAAjVaXRR30G1985oMou7Ambg==", 2024-03-23T11:29:11.910 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.3", 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lomZyOBAA8BMIBD+Zhtp6Fa2RcEXCJg==", 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:29:11.911 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.admin", 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lNtZHBhAAhiAC1rdmANClOSwzrX9iSQ==", 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.912 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.x", 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lT6+KBRAAl5TZsJGNr4gAB2cEDZo/hQ==", 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.y", 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lqnG8NRAAvyswMz5kpW0rGjzWYRTAAg==", 2024-03-23T11:29:11.913 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:29:11.914 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:29:11.914 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:29:11.914 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:29:11.914 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.914 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:11.914 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-03-23T11:29:11.914 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:29:11.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.911+0000 7efed6c00640 1 -- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efeb006a7a0 msgr2=0x7efeb006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:11.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.911+0000 7efed6c00640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efeb006a7a0 0x7efeb006cc60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7efec4036000 tx=0x7efec4030000 comp rx=0 tx=0).stop 2024-03-23T11:29:11.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.911+0000 7efed6c00640 1 -- 172.21.15.62:0/478938093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed006cd20 msgr2=0x7efed00b0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:11.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.911+0000 7efed6c00640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed006cd20 0x7efed00b0910 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7efed006e380 tx=0x7efec800b4a0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.911+0000 7efed6c00640 1 -- 172.21.15.62:0/478938093 shutdown_connections 2024-03-23T11:29:11.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.911+0000 7efed6c00640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7efeb006a7a0 0x7efeb006cc60 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.911+0000 7efed6c00640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efed006cd20 0x7efed00b0910 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.911+0000 7efed6c00640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efed006c8a0 0x7efed00b03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.911+0000 7efed6c00640 1 --2- 172.21.15.62:0/478938093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efeb8003970 0x7efed013df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:11.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.911+0000 7efed6c00640 1 -- 172.21.15.62:0/478938093 >> 172.21.15.62:0/478938093 conn(0x7efed0066050 msgr2=0x7efed00704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:11.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.911+0000 7efed6c00640 1 -- 172.21.15.62:0/478938093 shutdown_connections 2024-03-23T11:29:11.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:11.912+0000 7efed6c00640 1 -- 172.21.15.62:0/478938093 wait complete. 2024-03-23T11:29:11.923 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:29:12.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.151+0000 7ff977200640 1 Processor -- start 2024-03-23T11:29:12.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.151+0000 7ff977200640 1 -- start start 2024-03-23T11:29:12.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.153+0000 7ff977200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff9780b2f30 0x7ff978070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.153+0000 7ff977200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9780cd140 con 0x7ff978070090 2024-03-23T11:29:12.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.153+0000 7ff977200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9780cd3e0 con 0x7ff978143dd0 2024-03-23T11:29:12.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.153+0000 7ff977200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff978070a60 con 0x7ff9780b2f30 2024-03-23T11:29:12.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.153+0000 7ff976800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff9780b2f30 0x7ff978070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.153+0000 7ff976800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff9780b2f30 0x7ff978070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33560/0 (socket says 172.21.15.62:33560) 2024-03-23T11:29:12.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.153+0000 7ff976800640 1 -- 172.21.15.62:0/3010346057 learned_addr learned my addr 172.21.15.62:0/3010346057 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:12.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.153+0000 7ff976800640 1 -- 172.21.15.62:0/3010346057 >> v1:172.21.15.120:6789/0 conn(0x7ff978143dd0 legacy=0x7ff97815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:12.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.153+0000 7ff976800640 1 -- 172.21.15.62:0/3010346057 >> v1:172.21.15.62:6789/0 conn(0x7ff978070090 legacy=0x7ff9780b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.153+0000 7ff976800640 1 -- 172.21.15.62:0/3010346057 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9780cd6c0 con 0x7ff9780b2f30 2024-03-23T11:29:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.155+0000 7ff976800640 1 --2- 172.21.15.62:0/3010346057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9780b2f30 0x7ff978070520 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7ff970009e50 tx=0x7ff97002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa24186a46f16101 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.155+0000 7ff975400640 1 -- 172.21.15.62:0/3010346057 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97002cb10 con 0x7ff9780b2f30 2024-03-23T11:29:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.155+0000 7ff975400640 1 -- 172.21.15.62:0/3010346057 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97002cc70 con 0x7ff9780b2f30 2024-03-23T11:29:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.155+0000 7ff975400640 1 -- 172.21.15.62:0/3010346057 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9700343f0 con 0x7ff9780b2f30 2024-03-23T11:29:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.155+0000 7ff977200640 1 -- 172.21.15.62:0/3010346057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9780b2f30 msgr2=0x7ff978070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.155+0000 7ff977200640 1 --2- 172.21.15.62:0/3010346057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9780b2f30 0x7ff978070520 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7ff970009e50 tx=0x7ff97002bf70 comp rx=0 tx=0).stop 2024-03-23T11:29:12.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.156+0000 7ff977200640 1 -- 172.21.15.62:0/3010346057 shutdown_connections 2024-03-23T11:29:12.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.156+0000 7ff977200640 1 --2- 172.21.15.62:0/3010346057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9780b2f30 0x7ff978070520 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.156+0000 7ff977200640 1 -- 172.21.15.62:0/3010346057 >> 172.21.15.62:0/3010346057 conn(0x7ff978066050 msgr2=0x7ff978069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:12.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.156+0000 7ff977200640 1 -- 172.21.15.62:0/3010346057 shutdown_connections 2024-03-23T11:29:12.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.156+0000 7ff977200640 1 -- 172.21.15.62:0/3010346057 wait complete. 2024-03-23T11:29:12.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.157+0000 7ff977200640 1 Processor -- start 2024-03-23T11:29:12.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.158+0000 7ff977200640 1 -- start start 2024-03-23T11:29:12.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.158+0000 7ff977200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff978070090 0x7ff978159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.158+0000 7ff977200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9780b2f30 0x7ff9780abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.158+0000 7ff977200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff978143dd0 0x7ff97813dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.158+0000 7ff977200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff978159ca0 con 0x7ff978070090 2024-03-23T11:29:12.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.158+0000 7ff977200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97813e730 con 0x7ff978143dd0 2024-03-23T11:29:12.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.158+0000 7ff977200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9780ab300 con 0x7ff9780b2f30 2024-03-23T11:29:12.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff975e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff978070090 0x7ff978159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff975e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff978070090 0x7ff978159760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33572/0 (socket says 172.21.15.62:33572) 2024-03-23T11:29:12.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff975e00640 1 -- 172.21.15.62:0/1249979132 learned_addr learned my addr 172.21.15.62:0/1249979132 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:12.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff975e00640 1 -- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9780b2f30 msgr2=0x7ff9780abde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff976800640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff978143dd0 0x7ff97813dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff96fe00640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9780b2f30 0x7ff9780abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff975e00640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9780b2f30 0x7ff9780abde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff975e00640 1 -- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff978143dd0 msgr2=0x7ff97813dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff975e00640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff978143dd0 0x7ff97813dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff975e00640 1 -- 172.21.15.62:0/1249979132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff970009b30 con 0x7ff978070090 2024-03-23T11:29:12.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff96fe00640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9780b2f30 0x7ff9780abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:12.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.159+0000 7ff976800640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff978143dd0 0x7ff97813dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:12.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.160+0000 7ff975e00640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff978070090 0x7ff978159760 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7ff96800b3f0 tx=0x7ff96800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:12.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.160+0000 7ff96e000640 1 -- 172.21.15.62:0/1249979132 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff968011830 con 0x7ff978070090 2024-03-23T11:29:12.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.160+0000 7ff96e000640 1 -- 172.21.15.62:0/1249979132 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff968011990 con 0x7ff978070090 2024-03-23T11:29:12.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.160+0000 7ff96e000640 1 -- 172.21.15.62:0/1249979132 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff968011d40 con 0x7ff978070090 2024-03-23T11:29:12.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.160+0000 7ff977200640 1 -- 172.21.15.62:0/1249979132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9780ab5e0 con 0x7ff978070090 2024-03-23T11:29:12.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.160+0000 7ff977200640 1 -- 172.21.15.62:0/1249979132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9780ac320 con 0x7ff978070090 2024-03-23T11:29:12.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.162+0000 7ff977200640 1 -- 172.21.15.62:0/1249979132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9780b2460 con 0x7ff978070090 2024-03-23T11:29:12.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.163+0000 7ff96e000640 1 -- 172.21.15.62:0/1249979132 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff96801a400 con 0x7ff978070090 2024-03-23T11:29:12.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.163+0000 7ff96e000640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff95406a7d0 0x7ff95406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.164+0000 7ff96e000640 1 -- 172.21.15.62:0/1249979132 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff968088810 con 0x7ff978070090 2024-03-23T11:29:12.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.164+0000 7ff96fe00640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff95406a7d0 0x7ff95406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.165+0000 7ff96fe00640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff95406a7d0 0x7ff95406cc90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff97813d800 tx=0x7ff960009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:12.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.168+0000 7ff96e000640 1 -- 172.21.15.62:0/1249979132 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff968053f80 con 0x7ff978070090 2024-03-23T11:29:12.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.368+0000 7ff977200640 1 -- 172.21.15.62:0/1249979132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff978066bb0 con 0x7ff978070090 2024-03-23T11:29:12.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.370+0000 7ff96e000640 1 -- 172.21.15.62:0/1249979132 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+12391 (secure 0 0 0) 0x7ff968015090 con 0x7ff978070090 2024-03-23T11:29:12.361 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:12.361 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":101,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:29:11.367500+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:29:12.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 -- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff95406a7d0 msgr2=0x7ff95406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff95406a7d0 0x7ff95406cc90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff97813d800 tx=0x7ff960009040 comp rx=0 tx=0).stop 2024-03-23T11:29:12.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 -- 172.21.15.62:0/1249979132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff978070090 msgr2=0x7ff978159760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff978070090 0x7ff978159760 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7ff96800b3f0 tx=0x7ff96800b8b0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 -- 172.21.15.62:0/1249979132 shutdown_connections 2024-03-23T11:29:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff95406a7d0 0x7ff95406cc90 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff978143dd0 0x7ff97813dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9780b2f30 0x7ff9780abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 --2- 172.21.15.62:0/1249979132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff978070090 0x7ff978159760 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 -- 172.21.15.62:0/1249979132 >> 172.21.15.62:0/1249979132 conn(0x7ff978066050 msgr2=0x7ff9780aefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 -- 172.21.15.62:0/1249979132 shutdown_connections 2024-03-23T11:29:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.373+0000 7ff977200640 1 -- 172.21.15.62:0/1249979132 wait complete. 2024-03-23T11:29:12.385 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:29:12.385 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:29:12.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.618+0000 7fb675200640 1 Processor -- start 2024-03-23T11:29:12.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.619+0000 7fb675200640 1 -- start start 2024-03-23T11:29:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.620+0000 7fb675200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb6700b2f30 0x7fb670070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.620+0000 7fb675200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6700cd140 con 0x7fb670070090 2024-03-23T11:29:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.620+0000 7fb675200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6700cd3e0 con 0x7fb670143dd0 2024-03-23T11:29:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.620+0000 7fb675200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb670070a60 con 0x7fb6700b2f30 2024-03-23T11:29:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.621+0000 7fb666a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb670070090 0x7fb6700b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57284/0 (socket says 172.21.15.62:57284) 2024-03-23T11:29:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.621+0000 7fb666a00640 1 -- 172.21.15.62:0/1995153112 learned_addr learned my addr 172.21.15.62:0/1995153112 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.621+0000 7fb66fe00640 1 --2- 172.21.15.62:0/1995153112 >> v2:172.21.15.120:3300/0 conn(0x7fb6700b2f30 0x7fb670070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.621+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984751893 0 0) 0x7fb6700cd140 con 0x7fb670070090 2024-03-23T11:29:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.621+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb654003520 con 0x7fb670070090 2024-03-23T11:29:12.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1759435941 0 0) 0x7fb654003520 con 0x7fb670070090 2024-03-23T11:29:12.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 >> v2:172.21.15.120:3300/0 conn(0x7fb6700b2f30 msgr2=0x7fb670070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:12.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 --2- 172.21.15.62:0/1995153112 >> v2:172.21.15.120:3300/0 conn(0x7fb6700b2f30 0x7fb670070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 >> v1:172.21.15.120:6789/0 conn(0x7fb670143dd0 legacy=0x7fb67015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6700cd6c0 con 0x7fb670070090 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fb658001670 con 0x7fb670070090 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 >> v1:172.21.15.62:6789/0 conn(0x7fb670070090 legacy=0x7fb6700b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 --2- 172.21.15.62:0/1995153112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb654003970 0x7fb654003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 --2- 172.21.15.62:0/1995153112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6540047c0 0x7fb654004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 --2- 172.21.15.62:0/1995153112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb654005180 0x7fb65400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb670070a60 con 0x7fb654003970 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65400c020 con 0x7fb6540047c0 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65400c1e0 con 0x7fb654005180 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66f400640 1 --2- 172.21.15.62:0/1995153112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb654003970 0x7fb654003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb666a00640 1 --2- 172.21.15.62:0/1995153112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6540047c0 0x7fb654004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66f400640 1 -- 172.21.15.62:0/1995153112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb654005180 msgr2=0x7fb65400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.622+0000 7fb66f400640 1 --2- 172.21.15.62:0/1995153112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb654005180 0x7fb65400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.623+0000 7fb66f400640 1 -- 172.21.15.62:0/1995153112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6540047c0 msgr2=0x7fb654004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.623+0000 7fb66f400640 1 --2- 172.21.15.62:0/1995153112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6540047c0 0x7fb654004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.623+0000 7fb66f400640 1 -- 172.21.15.62:0/1995153112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb654003520 con 0x7fb654003970 2024-03-23T11:29:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.623+0000 7fb66fe00640 1 -- 172.21.15.62:0/1995153112 reap_dead start 2024-03-23T11:29:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.623+0000 7fb666a00640 1 --2- 172.21.15.62:0/1995153112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6540047c0 0x7fb654004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.624+0000 7fb66f400640 1 --2- 172.21.15.62:0/1995153112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb654003970 0x7fb654003d90 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fb660001750 tx=0x7fb6540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7992fa02da6d92d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.624+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6600052d0 con 0x7fb654003970 2024-03-23T11:29:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.624+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66002e050 con 0x7fb654003970 2024-03-23T11:29:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.624+0000 7fb66ea00640 1 -- 172.21.15.62:0/1995153112 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb660003830 con 0x7fb654003970 2024-03-23T11:29:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.625+0000 7fb675200640 1 -- 172.21.15.62:0/1995153112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb654003970 msgr2=0x7fb654003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.625+0000 7fb675200640 1 --2- 172.21.15.62:0/1995153112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb654003970 0x7fb654003d90 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fb660001750 tx=0x7fb6540057f0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.625+0000 7fb675200640 1 -- 172.21.15.62:0/1995153112 shutdown_connections 2024-03-23T11:29:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.625+0000 7fb675200640 1 --2- 172.21.15.62:0/1995153112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb654003970 0x7fb654003d90 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.625+0000 7fb675200640 1 -- 172.21.15.62:0/1995153112 >> 172.21.15.62:0/1995153112 conn(0x7fb670066050 msgr2=0x7fb670069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.625+0000 7fb675200640 1 -- 172.21.15.62:0/1995153112 shutdown_connections 2024-03-23T11:29:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.625+0000 7fb675200640 1 -- 172.21.15.62:0/1995153112 wait complete. 2024-03-23T11:29:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.627+0000 7fb675200640 1 Processor -- start 2024-03-23T11:29:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.627+0000 7fb675200640 1 -- start start 2024-03-23T11:29:12.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb675200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb654003970 0x7fb6700c4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb675200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6701376e0 0x7fb67015c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb675200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6700c8d60 0x7fb6700c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb675200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb670144c10 con 0x7fb6700c8d60 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb675200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb670134cc0 con 0x7fb6701376e0 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb675200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700c9720 con 0x7fb654003970 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb66fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6700c8d60 0x7fb6700c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb66fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6700c8d60 0x7fb6700c91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33590/0 (socket says 172.21.15.62:33590) 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb66fe00640 1 -- 172.21.15.62:0/3146342701 learned_addr learned my addr 172.21.15.62:0/3146342701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb66f400640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb654003970 0x7fb6700c4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb66fe00640 1 -- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb654003970 msgr2=0x7fb6700c4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb66fe00640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb654003970 0x7fb6700c4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb66fe00640 1 -- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6701376e0 msgr2=0x7fb67015c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb666a00640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6701376e0 0x7fb67015c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb66fe00640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6701376e0 0x7fb67015c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb66fe00640 1 -- 172.21.15.62:0/3146342701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb670137b60 con 0x7fb6700c8d60 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.628+0000 7fb66f400640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb654003970 0x7fb6700c4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.629+0000 7fb66fe00640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6700c8d60 0x7fb6700c91e0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fb66800eaa0 tx=0x7fb66800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:12.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.629+0000 7fb667e00640 1 -- 172.21.15.62:0/3146342701 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668010040 con 0x7fb6700c8d60 2024-03-23T11:29:12.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.629+0000 7fb667e00640 1 -- 172.21.15.62:0/3146342701 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66800bbd0 con 0x7fb6700c8d60 2024-03-23T11:29:12.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.629+0000 7fb667e00640 1 -- 172.21.15.62:0/3146342701 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668014400 con 0x7fb6700c8d60 2024-03-23T11:29:12.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.629+0000 7fb675200640 1 -- 172.21.15.62:0/3146342701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6700c5420 con 0x7fb6700c8d60 2024-03-23T11:29:12.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.629+0000 7fb675200640 1 -- 172.21.15.62:0/3146342701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6700c59e0 con 0x7fb6700c8d60 2024-03-23T11:29:12.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.630+0000 7fb675200640 1 -- 172.21.15.62:0/3146342701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb670134e50 con 0x7fb6700c8d60 2024-03-23T11:29:12.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.631+0000 7fb667e00640 1 -- 172.21.15.62:0/3146342701 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb668014560 con 0x7fb6700c8d60 2024-03-23T11:29:12.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.632+0000 7fb667e00640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb64c06a7d0 0x7fb64c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:12.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.632+0000 7fb667e00640 1 -- 172.21.15.62:0/3146342701 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb668088d20 con 0x7fb6700c8d60 2024-03-23T11:29:12.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.633+0000 7fb66f400640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb64c06a7d0 0x7fb64c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:12.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.634+0000 7fb66f400640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb64c06a7d0 0x7fb64c06cc90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb660033000 tx=0x7fb660009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:12.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.636+0000 7fb667e00640 1 -- 172.21.15.62:0/3146342701 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb668054360 con 0x7fb6700c8d60 2024-03-23T11:29:12.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.836+0000 7fb675200640 1 -- 172.21.15.62:0/3146342701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb670066bb0 con 0x7fb6700c8d60 2024-03-23T11:29:12.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.837+0000 7fb667e00640 1 -- 172.21.15.62:0/3146342701 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb668057980 con 0x7fb6700c8d60 2024-03-23T11:29:12.829 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:29:12.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 -- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb64c06a7d0 msgr2=0x7fb64c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb64c06a7d0 0x7fb64c06cc90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb660033000 tx=0x7fb660009030 comp rx=0 tx=0).stop 2024-03-23T11:29:12.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 -- 172.21.15.62:0/3146342701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6700c8d60 msgr2=0x7fb6700c91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:12.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6700c8d60 0x7fb6700c91e0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fb66800eaa0 tx=0x7fb66800ef60 comp rx=0 tx=0).stop 2024-03-23T11:29:12.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 -- 172.21.15.62:0/3146342701 shutdown_connections 2024-03-23T11:29:12.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb64c06a7d0 0x7fb64c06cc90 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6700c8d60 0x7fb6700c91e0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6701376e0 0x7fb67015c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 --2- 172.21.15.62:0/3146342701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb654003970 0x7fb6700c4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:12.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 -- 172.21.15.62:0/3146342701 >> 172.21.15.62:0/3146342701 conn(0x7fb670066050 msgr2=0x7fb6700ab840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:12.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 -- 172.21.15.62:0/3146342701 shutdown_connections 2024-03-23T11:29:12.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:12.840+0000 7fb675200640 1 -- 172.21.15.62:0/3146342701 wait complete. 2024-03-23T11:29:12.852 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T11:29:13.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.080+0000 7f0446400640 1 Processor -- start 2024-03-23T11:29:13.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.080+0000 7f0446400640 1 -- start start 2024-03-23T11:29:13.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.082+0000 7f0446400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0440143510 0x7f04400cc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.082+0000 7f0446400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f04400b41f0 0x7f04400caea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.082+0000 7f0446400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04400a7640 con 0x7f0440070fa0 2024-03-23T11:29:13.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.082+0000 7f0446400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0440070270 con 0x7f04400b41f0 2024-03-23T11:29:13.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.082+0000 7f0446400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04400cd2e0 con 0x7f0440143510 2024-03-23T11:29:13.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.082+0000 7f043ea00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0440070fa0 0x7f04400b3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57290/0 (socket says 172.21.15.62:57290) 2024-03-23T11:29:13.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.082+0000 7f043ea00640 1 -- 172.21.15.62:0/4180893060 learned_addr learned my addr 172.21.15.62:0/4180893060 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:13.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.082+0000 7f043fe00640 1 --2- 172.21.15.62:0/4180893060 >> v2:172.21.15.62:3300/0 conn(0x7f04400b41f0 0x7f04400caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.083+0000 7f043f400640 1 --2- 172.21.15.62:0/4180893060 >> v2:172.21.15.120:3300/0 conn(0x7f0440143510 0x7f04400cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.083+0000 7f043fe00640 1 -- 172.21.15.62:0/4180893060 >> v2:172.21.15.120:3300/0 conn(0x7f0440143510 msgr2=0x7f04400cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:13.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.083+0000 7f043fe00640 1 --2- 172.21.15.62:0/4180893060 >> v2:172.21.15.120:3300/0 conn(0x7f0440143510 0x7f04400cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.083+0000 7f043fe00640 1 -- 172.21.15.62:0/4180893060 >> v1:172.21.15.62:6789/0 conn(0x7f0440070fa0 legacy=0x7f04400b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:13.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.083+0000 7f043fe00640 1 -- 172.21.15.62:0/4180893060 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04400704c0 con 0x7f04400b41f0 2024-03-23T11:29:13.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.084+0000 7f043fe00640 1 --2- 172.21.15.62:0/4180893060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04400b41f0 0x7f04400caea0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f0440071410 tx=0x7f043802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a686889e7acf5152 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:13.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.085+0000 7f043e000640 1 -- 172.21.15.62:0/4180893060 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438031830 con 0x7f04400b41f0 2024-03-23T11:29:13.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.085+0000 7f043e000640 1 -- 172.21.15.62:0/4180893060 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0438031990 con 0x7f04400b41f0 2024-03-23T11:29:13.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.085+0000 7f043e000640 1 -- 172.21.15.62:0/4180893060 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438031d20 con 0x7f04400b41f0 2024-03-23T11:29:13.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.085+0000 7f0446400640 1 -- 172.21.15.62:0/4180893060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04400b41f0 msgr2=0x7f04400caea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:13.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.085+0000 7f0446400640 1 --2- 172.21.15.62:0/4180893060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04400b41f0 0x7f04400caea0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f0440071410 tx=0x7f043802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.085+0000 7f0446400640 1 -- 172.21.15.62:0/4180893060 shutdown_connections 2024-03-23T11:29:13.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.085+0000 7f0446400640 1 --2- 172.21.15.62:0/4180893060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04400b41f0 0x7f04400caea0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.085+0000 7f0446400640 1 --2- 172.21.15.62:0/4180893060 >> v2:172.21.15.120:3300/0 conn(0x7f0440143510 0x7f04400cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.085+0000 7f0446400640 1 -- 172.21.15.62:0/4180893060 >> 172.21.15.62:0/4180893060 conn(0x7f0440069c60 msgr2=0x7f044006a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:13.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.086+0000 7f0446400640 1 -- 172.21.15.62:0/4180893060 shutdown_connections 2024-03-23T11:29:13.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.086+0000 7f0446400640 1 -- 172.21.15.62:0/4180893060 wait complete. 2024-03-23T11:29:13.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.087+0000 7f0446400640 1 Processor -- start 2024-03-23T11:29:13.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.087+0000 7f0446400640 1 -- start start 2024-03-23T11:29:13.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.088+0000 7f0446400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0440070fa0 0x7f044013c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.088+0000 7f0446400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04400b41f0 0x7f0440139760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.088+0000 7f0446400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0440143510 0x7f04400ac580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.088+0000 7f0446400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0440139ca0 con 0x7f04400b41f0 2024-03-23T11:29:13.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.088+0000 7f0446400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04400abc80 con 0x7f0440143510 2024-03-23T11:29:13.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.088+0000 7f0446400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04400abf60 con 0x7f0440070fa0 2024-03-23T11:29:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.089+0000 7f043ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04400b41f0 0x7f0440139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.089+0000 7f043ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04400b41f0 0x7f0440139760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33606/0 (socket says 172.21.15.62:33606) 2024-03-23T11:29:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.089+0000 7f043ea00640 1 -- 172.21.15.62:0/2904729001 learned_addr learned my addr 172.21.15.62:0/2904729001 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.089+0000 7f043f400640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0440070fa0 0x7f044013c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.089+0000 7f043fe00640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0440143510 0x7f04400ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.089+0000 7f043ea00640 1 -- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0440070fa0 msgr2=0x7f044013c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.089+0000 7f043ea00640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0440070fa0 0x7f044013c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.089+0000 7f043ea00640 1 -- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0440143510 msgr2=0x7f04400ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.089+0000 7f043ea00640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0440143510 0x7f04400ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.089+0000 7f043ea00640 1 -- 172.21.15.62:0/2904729001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438009400 con 0x7f04400b41f0 2024-03-23T11:29:13.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.089+0000 7f043ea00640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04400b41f0 0x7f0440139760 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f042c00c480 tx=0x7f042c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:13.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.090+0000 7f0437400640 1 -- 172.21.15.62:0/2904729001 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c005ce0 con 0x7f04400b41f0 2024-03-23T11:29:13.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.090+0000 7f0437400640 1 -- 172.21.15.62:0/2904729001 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c005e40 con 0x7f04400b41f0 2024-03-23T11:29:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.090+0000 7f0437400640 1 -- 172.21.15.62:0/2904729001 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c0072f0 con 0x7f04400b41f0 2024-03-23T11:29:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.090+0000 7f0446400640 1 -- 172.21.15.62:0/2904729001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04400ac240 con 0x7f04400b41f0 2024-03-23T11:29:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.090+0000 7f0446400640 1 -- 172.21.15.62:0/2904729001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04400b2660 con 0x7f04400b41f0 2024-03-23T11:29:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.091+0000 7f0446400640 1 -- 172.21.15.62:0/2904729001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04400b2850 con 0x7f04400b41f0 2024-03-23T11:29:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.092+0000 7f0437400640 1 -- 172.21.15.62:0/2904729001 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f042c007510 con 0x7f04400b41f0 2024-03-23T11:29:13.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.093+0000 7f0437400640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042006a7a0 0x7f042006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.093+0000 7f0437400640 1 -- 172.21.15.62:0/2904729001 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f042c087f10 con 0x7f04400b41f0 2024-03-23T11:29:13.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.094+0000 7f043f400640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042006a7a0 0x7f042006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.095+0000 7f043f400640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042006a7a0 0x7f042006cc60 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0440067700 tx=0x7f04300039b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:13.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.097+0000 7f0437400640 1 -- 172.21.15.62:0/2904729001 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f042c0537c0 con 0x7f04400b41f0 2024-03-23T11:29:13.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.300+0000 7f0446400640 1 -- 172.21.15.62:0/2904729001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0440066c20 con 0x7f04400b41f0 2024-03-23T11:29:13.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.300+0000 7f0437400640 1 -- 172.21.15.62:0/2904729001 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v101) v1 ==== 84+0+65 (secure 0 0 0) 0x7f042c017020 con 0x7f04400b41f0 2024-03-23T11:29:13.292 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:13.292 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:29:13.292 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:29:13.292 INFO:teuthology.orchestra.run.smithi062.stdout: "poolnum": 1, 2024-03-23T11:29:13.292 INFO:teuthology.orchestra.run.smithi062.stdout: "poolname": ".mgr" 2024-03-23T11:29:13.292 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:13.292 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:29:13.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 -- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042006a7a0 msgr2=0x7f042006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:13.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042006a7a0 0x7f042006cc60 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0440067700 tx=0x7f04300039b0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 -- 172.21.15.62:0/2904729001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04400b41f0 msgr2=0x7f0440139760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:13.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04400b41f0 0x7f0440139760 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f042c00c480 tx=0x7f042c00c940 comp rx=0 tx=0).stop 2024-03-23T11:29:13.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 -- 172.21.15.62:0/2904729001 shutdown_connections 2024-03-23T11:29:13.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f042006a7a0 0x7f042006cc60 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04400b41f0 0x7f0440139760 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0440143510 0x7f04400ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 --2- 172.21.15.62:0/2904729001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0440070fa0 0x7f044013c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 -- 172.21.15.62:0/2904729001 >> 172.21.15.62:0/2904729001 conn(0x7f0440069c60 msgr2=0x7f04400ca9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:13.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 -- 172.21.15.62:0/2904729001 shutdown_connections 2024-03-23T11:29:13.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.303+0000 7f0446400640 1 -- 172.21.15.62:0/2904729001 wait complete. 2024-03-23T11:29:13.315 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T11:29:13.315 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T11:29:13.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.546+0000 7feb2be00640 1 Processor -- start 2024-03-23T11:29:13.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.547+0000 7feb2be00640 1 -- start start 2024-03-23T11:29:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.548+0000 7feb2be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feb2c15c820 0x7feb2c0b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.548+0000 7feb2be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7feb2c0b28e0 0x7feb2c0cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.548+0000 7feb2be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb2c0cf540 0x7feb2c0d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.548+0000 7feb2be00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c0cf9f0 con 0x7feb2c0b28e0 2024-03-23T11:29:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.548+0000 7feb2be00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c0d0200 con 0x7feb2c15c820 2024-03-23T11:29:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.548+0000 7feb2be00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0d07c0 con 0x7feb2c0cf540 2024-03-23T11:29:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2a000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7feb2c0b28e0 0x7feb2c0cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2a000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7feb2c0b28e0 0x7feb2c0cf000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33616/0 (socket says 172.21.15.62:33616) 2024-03-23T11:29:13.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2a000640 1 -- 172.21.15.62:0/2234741264 learned_addr learned my addr 172.21.15.62:0/2234741264 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:13.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2a000640 1 -- 172.21.15.62:0/2234741264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb2c0cf540 msgr2=0x7feb2c0d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:13.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2aa00640 1 --2- 172.21.15.62:0/2234741264 >> v2:172.21.15.120:3300/0 conn(0x7feb2c15c820 0x7feb2c0b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2b400640 1 --2- 172.21.15.62:0/2234741264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb2c0cf540 0x7feb2c0d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2a000640 1 --2- 172.21.15.62:0/2234741264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb2c0cf540 0x7feb2c0d0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2a000640 1 -- 172.21.15.62:0/2234741264 >> v2:172.21.15.120:3300/0 conn(0x7feb2c15c820 msgr2=0x7feb2c0b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2a000640 1 --2- 172.21.15.62:0/2234741264 >> v2:172.21.15.120:3300/0 conn(0x7feb2c15c820 0x7feb2c0b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2a000640 1 -- 172.21.15.62:0/2234741264 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0ce070 con 0x7feb2c0b28e0 2024-03-23T11:29:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2aa00640 1 --2- 172.21.15.62:0/2234741264 >> v2:172.21.15.120:3300/0 conn(0x7feb2c15c820 0x7feb2c0b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.549+0000 7feb2b400640 1 --2- 172.21.15.62:0/2234741264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb2c0cf540 0x7feb2c0d0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.550+0000 7feb2a000640 1 --2- 172.21.15.62:0/2234741264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb2c0b28e0 0x7feb2c0cf000 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7feb2c09c100 tx=0x7feb1402b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e125dd63753f60b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.550+0000 7feb29600640 1 -- 172.21.15.62:0/2234741264 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14004230 con 0x7feb2c0b28e0 2024-03-23T11:29:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.551+0000 7feb29600640 1 -- 172.21.15.62:0/2234741264 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1402e070 con 0x7feb2c0b28e0 2024-03-23T11:29:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.551+0000 7feb29600640 1 -- 172.21.15.62:0/2234741264 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14003610 con 0x7feb2c0b28e0 2024-03-23T11:29:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.551+0000 7feb2be00640 1 -- 172.21.15.62:0/2234741264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb2c0b28e0 msgr2=0x7feb2c0cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.551+0000 7feb2be00640 1 --2- 172.21.15.62:0/2234741264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb2c0b28e0 0x7feb2c0cf000 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7feb2c09c100 tx=0x7feb1402b850 comp rx=0 tx=0).stop 2024-03-23T11:29:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.551+0000 7feb2be00640 1 -- 172.21.15.62:0/2234741264 shutdown_connections 2024-03-23T11:29:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.551+0000 7feb2be00640 1 --2- 172.21.15.62:0/2234741264 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb2c0cf540 0x7feb2c0d0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.551+0000 7feb2be00640 1 --2- 172.21.15.62:0/2234741264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb2c0b28e0 0x7feb2c0cf000 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.551+0000 7feb2be00640 1 --2- 172.21.15.62:0/2234741264 >> v2:172.21.15.120:3300/0 conn(0x7feb2c15c820 0x7feb2c0b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.551+0000 7feb2be00640 1 -- 172.21.15.62:0/2234741264 >> 172.21.15.62:0/2234741264 conn(0x7feb2c09b9d0 msgr2=0x7feb2c089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:13.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.551+0000 7feb2be00640 1 -- 172.21.15.62:0/2234741264 shutdown_connections 2024-03-23T11:29:13.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.551+0000 7feb2be00640 1 -- 172.21.15.62:0/2234741264 wait complete. 2024-03-23T11:29:13.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.553+0000 7feb2be00640 1 Processor -- start 2024-03-23T11:29:13.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.553+0000 7feb2be00640 1 -- start start 2024-03-23T11:29:13.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.554+0000 7feb2be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb2c0b28e0 0x7feb2c1f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.554+0000 7feb2be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb2c0cf540 0x7feb2c0c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.554+0000 7feb2be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb2c15c820 0x7feb2c0cb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.554+0000 7feb2be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c1f1fa0 con 0x7feb2c0b28e0 2024-03-23T11:29:13.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.554+0000 7feb2be00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c15e900 con 0x7feb2c0cf540 2024-03-23T11:29:13.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.554+0000 7feb2be00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c15ebe0 con 0x7feb2c15c820 2024-03-23T11:29:13.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.554+0000 7feb2aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb2c0b28e0 0x7feb2c1f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.555+0000 7feb2aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb2c0b28e0 0x7feb2c1f1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33628/0 (socket says 172.21.15.62:33628) 2024-03-23T11:29:13.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.555+0000 7feb2aa00640 1 -- 172.21.15.62:0/1064287026 learned_addr learned my addr 172.21.15.62:0/1064287026 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:13.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.555+0000 7feb2a000640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb2c0cf540 0x7feb2c0c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.555+0000 7feb2b400640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb2c15c820 0x7feb2c0cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.555+0000 7feb2aa00640 1 -- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb2c15c820 msgr2=0x7feb2c0cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.555+0000 7feb2aa00640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb2c15c820 0x7feb2c0cb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.555+0000 7feb2aa00640 1 -- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb2c0cf540 msgr2=0x7feb2c0c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.555+0000 7feb2aa00640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb2c0cf540 0x7feb2c0c7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.555+0000 7feb2aa00640 1 -- 172.21.15.62:0/1064287026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb14009460 con 0x7feb2c0b28e0 2024-03-23T11:29:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.555+0000 7feb2aa00640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb2c0b28e0 0x7feb2c1f1a60 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00c7f0 tx=0x7feb1c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.556+0000 7feb22a00640 1 -- 172.21.15.62:0/1064287026 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c004220 con 0x7feb2c0b28e0 2024-03-23T11:29:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.556+0000 7feb22a00640 1 -- 172.21.15.62:0/1064287026 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c004380 con 0x7feb2c0b28e0 2024-03-23T11:29:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.556+0000 7feb22a00640 1 -- 172.21.15.62:0/1064287026 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c005bb0 con 0x7feb2c0b28e0 2024-03-23T11:29:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.556+0000 7feb2be00640 1 -- 172.21.15.62:0/1064287026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c15eec0 con 0x7feb2c0b28e0 2024-03-23T11:29:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.556+0000 7feb2be00640 1 -- 172.21.15.62:0/1064287026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c15f480 con 0x7feb2c0b28e0 2024-03-23T11:29:13.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.557+0000 7feb2be00640 1 -- 172.21.15.62:0/1064287026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c0cffa0 con 0x7feb2c0b28e0 2024-03-23T11:29:13.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.558+0000 7feb22a00640 1 -- 172.21.15.62:0/1064287026 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7feb1c006030 con 0x7feb2c0b28e0 2024-03-23T11:29:13.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.559+0000 7feb22a00640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb0806a7a0 0x7feb0806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:13.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.559+0000 7feb22a00640 1 -- 172.21.15.62:0/1064287026 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feb1c088fb0 con 0x7feb2c0b28e0 2024-03-23T11:29:13.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.559+0000 7feb2a000640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb0806a7a0 0x7feb0806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:13.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.561+0000 7feb2a000640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb0806a7a0 0x7feb0806cc60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7feb14034000 tx=0x7feb14008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:13.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.563+0000 7feb22a00640 1 -- 172.21.15.62:0/1064287026 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feb1c054680 con 0x7feb2c0b28e0 2024-03-23T11:29:13.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:13.763+0000 7feb2be00640 1 -- 172.21.15.62:0/1064287026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7feb2c15f670 con 0x7feb2c0b28e0 2024-03-23T11:29:14.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.124+0000 7feb22a00640 1 -- 172.21.15.62:0/1064287026 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v102) v1 ==== 138+0+0 (secure 0 0 0) 0x7feb1c057ca0 con 0x7feb2c0b28e0 2024-03-23T11:29:14.116 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' created 2024-03-23T11:29:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.129+0000 7feb2be00640 1 -- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb0806a7a0 msgr2=0x7feb0806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.129+0000 7feb2be00640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb0806a7a0 0x7feb0806cc60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7feb14034000 tx=0x7feb14008030 comp rx=0 tx=0).stop 2024-03-23T11:29:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.129+0000 7feb2be00640 1 -- 172.21.15.62:0/1064287026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb2c0b28e0 msgr2=0x7feb2c1f1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.129+0000 7feb2be00640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb2c0b28e0 0x7feb2c1f1a60 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00c7f0 tx=0x7feb1c00ccb0 comp rx=0 tx=0).stop 2024-03-23T11:29:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.129+0000 7feb2be00640 1 -- 172.21.15.62:0/1064287026 shutdown_connections 2024-03-23T11:29:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.129+0000 7feb2be00640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb0806a7a0 0x7feb0806cc60 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.129+0000 7feb2be00640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb2c15c820 0x7feb2c0cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.129+0000 7feb2be00640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb2c0cf540 0x7feb2c0c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.129+0000 7feb2be00640 1 --2- 172.21.15.62:0/1064287026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb2c0b28e0 0x7feb2c1f1a60 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.129+0000 7feb2be00640 1 -- 172.21.15.62:0/1064287026 >> 172.21.15.62:0/1064287026 conn(0x7feb2c09b9d0 msgr2=0x7feb2c0af610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:14.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.130+0000 7feb2be00640 1 -- 172.21.15.62:0/1064287026 shutdown_connections 2024-03-23T11:29:14.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.131+0000 7feb2be00640 1 -- 172.21.15.62:0/1064287026 wait complete. 2024-03-23T11:29:14.146 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T11:29:14.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.369+0000 7f6a7fe00640 1 Processor -- start 2024-03-23T11:29:14.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.369+0000 7f6a7fe00640 1 -- start start 2024-03-23T11:29:14.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.371+0000 7f6a7fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a7815cb40 0x7f6a780d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:14.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.371+0000 7f6a7fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6a780b83e0 0x7f6a780cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:14.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.371+0000 7f6a7fe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a780b6a00 con 0x7f6a780cdbd0 2024-03-23T11:29:14.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.371+0000 7f6a7fe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a780ce060 con 0x7f6a780b83e0 2024-03-23T11:29:14.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.371+0000 7f6a7fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a780d1350 con 0x7f6a7815cb40 2024-03-23T11:29:14.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.371+0000 7f6a7ca00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6a780b83e0 0x7f6a780cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:14.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.372+0000 7f6a7d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a7815cb40 0x7f6a780d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:14.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.372+0000 7f6a7ca00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6a780b83e0 0x7f6a780cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:33166/0 (socket says 172.21.15.62:33166) 2024-03-23T11:29:14.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.372+0000 7f6a7ca00640 1 -- 172.21.15.62:0/584575792 learned_addr learned my addr 172.21.15.62:0/584575792 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:14.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.372+0000 7f6a7ca00640 1 -- 172.21.15.62:0/584575792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a7815cb40 msgr2=0x7f6a780d0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:14.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.372+0000 7f6a7ca00640 1 --2- 172.21.15.62:0/584575792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a7815cb40 0x7f6a780d0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:14.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.372+0000 7f6a7ca00640 1 -- 172.21.15.62:0/584575792 >> v1:172.21.15.120:6789/0 conn(0x7f6a780cdbd0 legacy=0x7f6a780d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:14.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.372+0000 7f6a7ca00640 1 -- 172.21.15.62:0/584575792 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a780d1630 con 0x7f6a780b83e0 2024-03-23T11:29:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.374+0000 7f6a7ca00640 1 --2- 172.21.15.62:0/584575792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a780b83e0 0x7f6a780cd690 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009850 tx=0x7f6a6402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c484a6762f89da2d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.374+0000 7f6a77e00640 1 -- 172.21.15.62:0/584575792 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64004230 con 0x7f6a780b83e0 2024-03-23T11:29:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.374+0000 7f6a77e00640 1 -- 172.21.15.62:0/584575792 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a64004390 con 0x7f6a780b83e0 2024-03-23T11:29:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.374+0000 7f6a77e00640 1 -- 172.21.15.62:0/584575792 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64003810 con 0x7f6a780b83e0 2024-03-23T11:29:14.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.375+0000 7f6a7fe00640 1 -- 172.21.15.62:0/584575792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a780b83e0 msgr2=0x7f6a780cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:14.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.375+0000 7f6a7fe00640 1 --2- 172.21.15.62:0/584575792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a780b83e0 0x7f6a780cd690 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009850 tx=0x7f6a6402b9f0 comp rx=0 tx=0).stop 2024-03-23T11:29:14.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.375+0000 7f6a7fe00640 1 -- 172.21.15.62:0/584575792 shutdown_connections 2024-03-23T11:29:14.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.375+0000 7f6a7fe00640 1 --2- 172.21.15.62:0/584575792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a780b83e0 0x7f6a780cd690 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:14.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.375+0000 7f6a7fe00640 1 --2- 172.21.15.62:0/584575792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a7815cb40 0x7f6a780d0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:14.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.375+0000 7f6a7fe00640 1 -- 172.21.15.62:0/584575792 >> 172.21.15.62:0/584575792 conn(0x7f6a780717b0 msgr2=0x7f6a78071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:14.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.375+0000 7f6a7fe00640 1 -- 172.21.15.62:0/584575792 shutdown_connections 2024-03-23T11:29:14.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.375+0000 7f6a7fe00640 1 -- 172.21.15.62:0/584575792 wait complete. 2024-03-23T11:29:14.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.377+0000 7f6a7fe00640 1 Processor -- start 2024-03-23T11:29:14.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.377+0000 7f6a7fe00640 1 -- start start 2024-03-23T11:29:14.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.378+0000 7f6a7fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a780b83e0 0x7f6a7815fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:14.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.378+0000 7f6a7fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a780cdbd0 0x7f6a780c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:14.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.378+0000 7f6a7fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a7815cb40 0x7f6a78153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.378+0000 7f6a7fe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a780c6dd0 con 0x7f6a780cdbd0 2024-03-23T11:29:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.378+0000 7f6a7fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7806bcd0 con 0x7f6a780b83e0 2024-03-23T11:29:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.378+0000 7f6a7fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a780c70a0 con 0x7f6a7815cb40 2024-03-23T11:29:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.378+0000 7f6a7ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a780cdbd0 0x7f6a780c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.378+0000 7f6a7ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a780cdbd0 0x7f6a780c8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33632/0 (socket says 172.21.15.62:33632) 2024-03-23T11:29:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.378+0000 7f6a7ca00640 1 -- 172.21.15.62:0/1643672353 learned_addr learned my addr 172.21.15.62:0/1643672353 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.379+0000 7f6a7d400640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a780b83e0 0x7f6a7815fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.379+0000 7f6a7de00640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a7815cb40 0x7f6a78153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.379+0000 7f6a7ca00640 1 -- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a7815cb40 msgr2=0x7f6a78153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.379+0000 7f6a7ca00640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a7815cb40 0x7f6a78153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.379+0000 7f6a7ca00640 1 -- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a780b83e0 msgr2=0x7f6a7815fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:14.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.379+0000 7f6a7ca00640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a780b83e0 0x7f6a7815fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:14.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.379+0000 7f6a7ca00640 1 -- 172.21.15.62:0/1643672353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a64009460 con 0x7f6a780cdbd0 2024-03-23T11:29:14.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.379+0000 7f6a7ca00640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a780cdbd0 0x7f6a780c8640 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f6a6402ba30 tx=0x7f6a640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:14.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.380+0000 7f6a75600640 1 -- 172.21.15.62:0/1643672353 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6403d070 con 0x7f6a780cdbd0 2024-03-23T11:29:14.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.380+0000 7f6a75600640 1 -- 172.21.15.62:0/1643672353 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a64003610 con 0x7f6a780cdbd0 2024-03-23T11:29:14.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.380+0000 7f6a75600640 1 -- 172.21.15.62:0/1643672353 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640039a0 con 0x7f6a780cdbd0 2024-03-23T11:29:14.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.380+0000 7f6a7fe00640 1 -- 172.21.15.62:0/1643672353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a780c7380 con 0x7f6a780cdbd0 2024-03-23T11:29:14.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.380+0000 7f6a7fe00640 1 -- 172.21.15.62:0/1643672353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7806b930 con 0x7f6a780cdbd0 2024-03-23T11:29:14.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.381+0000 7f6a7fe00640 1 -- 172.21.15.62:0/1643672353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a780672c0 con 0x7f6a780cdbd0 2024-03-23T11:29:14.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.382+0000 7f6a75600640 1 -- 172.21.15.62:0/1643672353 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6a64003b00 con 0x7f6a780cdbd0 2024-03-23T11:29:14.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.383+0000 7f6a75600640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6a5806a7d0 0x7f6a5806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:14.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.384+0000 7f6a75600640 1 -- 172.21.15.62:0/1643672353 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f6a640aa370 con 0x7f6a780cdbd0 2024-03-23T11:29:14.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.384+0000 7f6a7d400640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6a5806a7d0 0x7f6a5806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:14.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.386+0000 7f6a7d400640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6a5806a7d0 0x7f6a5806cc90 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009a30 tx=0x7f6a6c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:14.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.390+0000 7f6a75600640 1 -- 172.21.15.62:0/1643672353 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a64075a00 con 0x7f6a780cdbd0 2024-03-23T11:29:14.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:14.595+0000 7f6a7fe00640 1 -- 172.21.15.62:0/1643672353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6a780c6f10 con 0x7f6a780cdbd0 2024-03-23T11:29:15.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.136+0000 7f6a75600640 1 -- 172.21.15.62:0/1643672353 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v103) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6a64079020 con 0x7f6a780cdbd0 2024-03-23T11:29:15.129 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' created 2024-03-23T11:29:15.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.141+0000 7f6a7fe00640 1 -- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6a5806a7d0 msgr2=0x7f6a5806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:15.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.141+0000 7f6a7fe00640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6a5806a7d0 0x7f6a5806cc90 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009a30 tx=0x7f6a6c008040 comp rx=0 tx=0).stop 2024-03-23T11:29:15.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.141+0000 7f6a7fe00640 1 -- 172.21.15.62:0/1643672353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a780cdbd0 msgr2=0x7f6a780c8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:15.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.141+0000 7f6a7fe00640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a780cdbd0 0x7f6a780c8640 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f6a6402ba30 tx=0x7f6a640046c0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.147+0000 7f6a7fe00640 1 -- 172.21.15.62:0/1643672353 shutdown_connections 2024-03-23T11:29:15.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.147+0000 7f6a7fe00640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6a5806a7d0 0x7f6a5806cc90 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.147+0000 7f6a7fe00640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a780cdbd0 0x7f6a780c8640 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.147+0000 7f6a7fe00640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a7815cb40 0x7f6a78153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.147+0000 7f6a7fe00640 1 --2- 172.21.15.62:0/1643672353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a780b83e0 0x7f6a7815fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.147+0000 7f6a7fe00640 1 -- 172.21.15.62:0/1643672353 >> 172.21.15.62:0/1643672353 conn(0x7f6a780717b0 msgr2=0x7f6a780cf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:15.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.151+0000 7f6a7fe00640 1 -- 172.21.15.62:0/1643672353 shutdown_connections 2024-03-23T11:29:15.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.152+0000 7f6a7fe00640 1 -- 172.21.15.62:0/1643672353 wait complete. 2024-03-23T11:29:15.172 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T11:29:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.366+0000 7fc65d800640 1 Processor -- start 2024-03-23T11:29:15.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.366+0000 7fc65d800640 1 -- start start 2024-03-23T11:29:15.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.368+0000 7fc65d800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc65806c300 0x7fc6580ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.368+0000 7fc65d800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc65806c6e0 0x7fc65806cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.368+0000 7fc65d800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6580cbc60 con 0x7fc6580cafa0 2024-03-23T11:29:15.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.368+0000 7fc65d800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6580cbf40 con 0x7fc65806c300 2024-03-23T11:29:15.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.368+0000 7fc65d800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6580cc220 con 0x7fc65806c6e0 2024-03-23T11:29:15.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.368+0000 7fc656a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc65806c300 0x7fc6580ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:15.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.368+0000 7fc656a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc65806c300 0x7fc6580ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33642/0 (socket says 172.21.15.62:33642) 2024-03-23T11:29:15.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.368+0000 7fc656a00640 1 -- 172.21.15.62:0/2923165021 learned_addr learned my addr 172.21.15.62:0/2923165021 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:15.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.368+0000 7fc657400640 1 --2- 172.21.15.62:0/2923165021 >> v2:172.21.15.120:3300/0 conn(0x7fc65806c6e0 0x7fc65806cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:15.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.369+0000 7fc656a00640 1 -- 172.21.15.62:0/2923165021 >> v2:172.21.15.120:3300/0 conn(0x7fc65806c6e0 msgr2=0x7fc65806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:15.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.369+0000 7fc656a00640 1 --2- 172.21.15.62:0/2923165021 >> v2:172.21.15.120:3300/0 conn(0x7fc65806c6e0 0x7fc65806cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.369+0000 7fc656a00640 1 -- 172.21.15.62:0/2923165021 >> v1:172.21.15.120:6789/0 conn(0x7fc6580cafa0 legacy=0x7fc6580cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:15.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.369+0000 7fc656a00640 1 -- 172.21.15.62:0/2923165021 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6580cc500 con 0x7fc65806c300 2024-03-23T11:29:15.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.370+0000 7fc656a00640 1 --2- 172.21.15.62:0/2923165021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65806c300 0x7fc6580ccd00 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fc658070270 tx=0x7fc64802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47227944b4066add server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:15.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.370+0000 7fc656000640 1 -- 172.21.15.62:0/2923165021 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648031870 con 0x7fc65806c300 2024-03-23T11:29:15.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.370+0000 7fc656000640 1 -- 172.21.15.62:0/2923165021 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6480319d0 con 0x7fc65806c300 2024-03-23T11:29:15.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.370+0000 7fc656000640 1 -- 172.21.15.62:0/2923165021 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648031d60 con 0x7fc65806c300 2024-03-23T11:29:15.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.371+0000 7fc65d800640 1 -- 172.21.15.62:0/2923165021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65806c300 msgr2=0x7fc6580ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:15.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.371+0000 7fc65d800640 1 --2- 172.21.15.62:0/2923165021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65806c300 0x7fc6580ccd00 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fc658070270 tx=0x7fc64802b910 comp rx=0 tx=0).stop 2024-03-23T11:29:15.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.371+0000 7fc65d800640 1 -- 172.21.15.62:0/2923165021 shutdown_connections 2024-03-23T11:29:15.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.371+0000 7fc65d800640 1 --2- 172.21.15.62:0/2923165021 >> v2:172.21.15.120:3300/0 conn(0x7fc65806c6e0 0x7fc65806cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.371+0000 7fc65d800640 1 --2- 172.21.15.62:0/2923165021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65806c300 0x7fc6580ccd00 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.371+0000 7fc65d800640 1 -- 172.21.15.62:0/2923165021 >> 172.21.15.62:0/2923165021 conn(0x7fc6580b02e0 msgr2=0x7fc6580b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:15.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.371+0000 7fc65d800640 1 -- 172.21.15.62:0/2923165021 shutdown_connections 2024-03-23T11:29:15.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.371+0000 7fc65d800640 1 -- 172.21.15.62:0/2923165021 wait complete. 2024-03-23T11:29:15.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.373+0000 7fc65d800640 1 Processor -- start 2024-03-23T11:29:15.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.373+0000 7fc65d800640 1 -- start start 2024-03-23T11:29:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.374+0000 7fc65d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65806c300 0x7fc658150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.374+0000 7fc65d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc65806c6e0 0x7fc6581f1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.374+0000 7fc65d800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6580cafa0 0x7fc6580b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.374+0000 7fc65d800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65813ea40 con 0x7fc65806c300 2024-03-23T11:29:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.374+0000 7fc65d800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65813ddc0 con 0x7fc6580cafa0 2024-03-23T11:29:15.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.374+0000 7fc65d800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65813e0a0 con 0x7fc65806c6e0 2024-03-23T11:29:15.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.374+0000 7fc656a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65806c300 0x7fc658150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:15.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.374+0000 7fc656a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65806c300 0x7fc658150d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33648/0 (socket says 172.21.15.62:33648) 2024-03-23T11:29:15.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.374+0000 7fc656a00640 1 -- 172.21.15.62:0/499759854 learned_addr learned my addr 172.21.15.62:0/499759854 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:15.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc657400640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6580cafa0 0x7fc6580b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:15.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc656a00640 1 -- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc65806c6e0 msgr2=0x7fc6581f1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:15.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc64fe00640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc65806c6e0 0x7fc6581f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:15.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc656a00640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc65806c6e0 0x7fc6581f1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc656a00640 1 -- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6580cafa0 msgr2=0x7fc6580b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:15.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc656a00640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6580cafa0 0x7fc6580b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc656a00640 1 -- 172.21.15.62:0/499759854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6480094d0 con 0x7fc65806c300 2024-03-23T11:29:15.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc64fe00640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc65806c6e0 0x7fc6581f1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:15.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc657400640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6580cafa0 0x7fc6580b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:15.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc656a00640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65806c300 0x7fc658150d30 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fc648000c00 tx=0x7fc6480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:15.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc64ea00640 1 -- 172.21.15.62:0/499759854 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64803abe0 con 0x7fc65806c300 2024-03-23T11:29:15.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc64ea00640 1 -- 172.21.15.62:0/499759854 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64803ad40 con 0x7fc65806c300 2024-03-23T11:29:15.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc65d800640 1 -- 172.21.15.62:0/499759854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65813e380 con 0x7fc65806c300 2024-03-23T11:29:15.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.375+0000 7fc64ea00640 1 -- 172.21.15.62:0/499759854 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648044610 con 0x7fc65806c300 2024-03-23T11:29:15.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.376+0000 7fc65d800640 1 -- 172.21.15.62:0/499759854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6580c37c0 con 0x7fc65806c300 2024-03-23T11:29:15.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.377+0000 7fc65d800640 1 -- 172.21.15.62:0/499759854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc658066c90 con 0x7fc65806c300 2024-03-23T11:29:15.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.378+0000 7fc64ea00640 1 -- 172.21.15.62:0/499759854 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc648044770 con 0x7fc65806c300 2024-03-23T11:29:15.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.379+0000 7fc64ea00640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc63406a7a0 0x7fc63406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.379+0000 7fc64ea00640 1 -- 172.21.15.62:0/499759854 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fc64802d040 con 0x7fc65806c300 2024-03-23T11:29:15.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.380+0000 7fc64fe00640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc63406a7a0 0x7fc63406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:15.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.381+0000 7fc64fe00640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc63406a7a0 0x7fc63406cc60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc6581518f0 tx=0x7fc640008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:15.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.385+0000 7fc64ea00640 1 -- 172.21.15.62:0/499759854 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc648076830 con 0x7fc65806c300 2024-03-23T11:29:15.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.623+0000 7fc65d800640 1 -- 172.21.15.62:0/499759854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc6580673f0 con 0x7fc65806c300 2024-03-23T11:29:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.634+0000 7fc64ea00640 1 -- 172.21.15.62:0/499759854 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v74) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc648079e50 con 0x7fc65806c300 2024-03-23T11:29:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:new fs with metadata pool 14 and data pool 15 2024-03-23T11:29:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.640+0000 7fc65d800640 1 -- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc63406a7a0 msgr2=0x7fc63406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.640+0000 7fc65d800640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc63406a7a0 0x7fc63406cc60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc6581518f0 tx=0x7fc640008040 comp rx=0 tx=0).stop 2024-03-23T11:29:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.640+0000 7fc65d800640 1 -- 172.21.15.62:0/499759854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65806c300 msgr2=0x7fc658150d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.640+0000 7fc65d800640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65806c300 0x7fc658150d30 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fc648000c00 tx=0x7fc6480046c0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.642+0000 7fc65d800640 1 -- 172.21.15.62:0/499759854 shutdown_connections 2024-03-23T11:29:15.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.642+0000 7fc65d800640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc63406a7a0 0x7fc63406cc60 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.642+0000 7fc65d800640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6580cafa0 0x7fc6580b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.642+0000 7fc65d800640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc65806c6e0 0x7fc6581f1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.642+0000 7fc65d800640 1 --2- 172.21.15.62:0/499759854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc65806c300 0x7fc658150d30 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.642+0000 7fc65d800640 1 -- 172.21.15.62:0/499759854 >> 172.21.15.62:0/499759854 conn(0x7fc6580b02e0 msgr2=0x7fc658159c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:15.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.643+0000 7fc65d800640 1 -- 172.21.15.62:0/499759854 shutdown_connections 2024-03-23T11:29:15.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.643+0000 7fc65d800640 1 -- 172.21.15.62:0/499759854 wait complete. 2024-03-23T11:29:15.668 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:29:15.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.852+0000 7f4143400640 1 Processor -- start 2024-03-23T11:29:15.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.852+0000 7f4143400640 1 -- start start 2024-03-23T11:29:15.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.854+0000 7f4143400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f413c06c300 0x7f413c0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.854+0000 7f4143400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cafa0 0x7f413c0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.854+0000 7f4143400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413c0cbc60 con 0x7f413c06c6e0 2024-03-23T11:29:15.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.854+0000 7f4143400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413c0cbf40 con 0x7f413c06c300 2024-03-23T11:29:15.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.854+0000 7f4143400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413c0cc220 con 0x7f413c0cafa0 2024-03-23T11:29:15.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.854+0000 7f4140a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f413c06c300 0x7f413c0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:15.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.854+0000 7f4141400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f413c06c6e0 0x7f413c06cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57302/0 (socket says 172.21.15.62:57302) 2024-03-23T11:29:15.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.854+0000 7f4140a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f413c06c300 0x7f413c0ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33650/0 (socket says 172.21.15.62:33650) 2024-03-23T11:29:15.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.854+0000 7f4141400640 1 -- 172.21.15.62:0/1400485001 learned_addr learned my addr 172.21.15.62:0/1400485001 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.855+0000 7f413be00640 1 --2- 172.21.15.62:0/1400485001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cafa0 0x7f413c0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.855+0000 7f4140a00640 1 -- 172.21.15.62:0/1400485001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cafa0 msgr2=0x7f413c0cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.855+0000 7f4140a00640 1 --2- 172.21.15.62:0/1400485001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cafa0 0x7f413c0cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.855+0000 7f4140a00640 1 -- 172.21.15.62:0/1400485001 >> v1:172.21.15.62:6789/0 conn(0x7f413c06c6e0 legacy=0x7f413c06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.855+0000 7f4140a00640 1 -- 172.21.15.62:0/1400485001 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c0cc500 con 0x7f413c06c300 2024-03-23T11:29:15.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.855+0000 7f413be00640 1 --2- 172.21.15.62:0/1400485001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cafa0 0x7f413c0cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:15.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.856+0000 7f4140a00640 1 --2- 172.21.15.62:0/1400485001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f413c06c300 0x7f413c0ccd00 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f412c0097a0 tx=0x7f412c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d165d00ca0e7f064 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:15.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.856+0000 7f413b400640 1 -- 172.21.15.62:0/1400485001 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c031780 con 0x7f413c06c300 2024-03-23T11:29:15.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.856+0000 7f413b400640 1 -- 172.21.15.62:0/1400485001 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c0318e0 con 0x7f413c06c300 2024-03-23T11:29:15.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.856+0000 7f413b400640 1 -- 172.21.15.62:0/1400485001 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c031c70 con 0x7f413c06c300 2024-03-23T11:29:15.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.857+0000 7f4143400640 1 -- 172.21.15.62:0/1400485001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f413c06c300 msgr2=0x7f413c0ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.857+0000 7f4143400640 1 --2- 172.21.15.62:0/1400485001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f413c06c300 0x7f413c0ccd00 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f412c0097a0 tx=0x7f412c02b820 comp rx=0 tx=0).stop 2024-03-23T11:29:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.857+0000 7f4143400640 1 -- 172.21.15.62:0/1400485001 shutdown_connections 2024-03-23T11:29:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.857+0000 7f4143400640 1 --2- 172.21.15.62:0/1400485001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c0cafa0 0x7f413c0cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.857+0000 7f4143400640 1 --2- 172.21.15.62:0/1400485001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f413c06c300 0x7f413c0ccd00 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.857+0000 7f4143400640 1 -- 172.21.15.62:0/1400485001 >> 172.21.15.62:0/1400485001 conn(0x7f413c0b02e0 msgr2=0x7f413c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.857+0000 7f4143400640 1 -- 172.21.15.62:0/1400485001 shutdown_connections 2024-03-23T11:29:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.857+0000 7f4143400640 1 -- 172.21.15.62:0/1400485001 wait complete. 2024-03-23T11:29:15.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.859+0000 7f4143400640 1 Processor -- start 2024-03-23T11:29:15.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.859+0000 7f4143400640 1 -- start start 2024-03-23T11:29:15.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f4143400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c06c300 0x7f413c156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f4143400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f413c06c6e0 0x7f413c140570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f4143400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c0cafa0 0x7f413c1f1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f4143400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c0b16e0 con 0x7f413c06c6e0 2024-03-23T11:29:15.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f4143400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c0b1980 con 0x7f413c0cafa0 2024-03-23T11:29:15.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f4143400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413c0b2f80 con 0x7f413c06c300 2024-03-23T11:29:15.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f413be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f413c06c6e0 0x7f413c140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:15.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f413be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f413c06c6e0 0x7f413c140570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33658/0 (socket says 172.21.15.62:33658) 2024-03-23T11:29:15.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f413be00640 1 -- 172.21.15.62:0/3932991582 learned_addr learned my addr 172.21.15.62:0/3932991582 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f413be00640 1 -- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c06c300 msgr2=0x7f413c156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f413be00640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c06c300 0x7f413c156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f4141400640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c0cafa0 0x7f413c1f1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f413be00640 1 -- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c0cafa0 msgr2=0x7f413c1f1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f413be00640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c0cafa0 0x7f413c1f1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.860+0000 7f413be00640 1 -- 172.21.15.62:0/3932991582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c009400 con 0x7f413c06c6e0 2024-03-23T11:29:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.861+0000 7f413be00640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f413c06c6e0 0x7f413c140570 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f413000c7f0 tx=0x7f413000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.861+0000 7f4138c00640 1 -- 172.21.15.62:0/3932991582 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130007820 con 0x7f413c06c6e0 2024-03-23T11:29:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.861+0000 7f4138c00640 1 -- 172.21.15.62:0/3932991582 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4130007980 con 0x7f413c06c6e0 2024-03-23T11:29:15.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.861+0000 7f4143400640 1 -- 172.21.15.62:0/3932991582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413c0b3260 con 0x7f413c06c6e0 2024-03-23T11:29:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.861+0000 7f4138c00640 1 -- 172.21.15.62:0/3932991582 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130007cf0 con 0x7f413c06c6e0 2024-03-23T11:29:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.861+0000 7f4143400640 1 -- 172.21.15.62:0/3932991582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413c0b2bc0 con 0x7f413c06c6e0 2024-03-23T11:29:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.862+0000 7f4143400640 1 -- 172.21.15.62:0/3932991582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413c066c90 con 0x7f413c06c6e0 2024-03-23T11:29:15.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.863+0000 7f4138c00640 1 -- 172.21.15.62:0/3932991582 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f41300113f0 con 0x7f413c06c6e0 2024-03-23T11:29:15.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.864+0000 7f4138c00640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f412006a7a0 0x7f412006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:15.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.865+0000 7f4138c00640 1 -- 172.21.15.62:0/3932991582 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4130089110 con 0x7f413c06c6e0 2024-03-23T11:29:15.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.865+0000 7f4140a00640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f412006a7a0 0x7f412006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:15.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.866+0000 7f4140a00640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f412006a7a0 0x7f412006cc60 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f412c036000 tx=0x7f412c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:15.869+0000 7f4138c00640 1 -- 172.21.15.62:0/3932991582 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4130054520 con 0x7f413c06c6e0 2024-03-23T11:29:16.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.066+0000 7f4143400640 1 -- 172.21.15.62:0/3932991582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f413c0b1ac0 con 0x7f413c06c6e0 2024-03-23T11:29:16.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.069+0000 7f4138c00640 1 -- 172.21.15.62:0/3932991582 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+16064 (secure 0 0 0) 0x7f4130057b40 con 0x7f413c06c6e0 2024-03-23T11:29:16.061 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:16.061 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":104,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:29:15.625446+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:29:13.817605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-23T11:29:14.652617+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:29:16.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 -- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f412006a7a0 msgr2=0x7f412006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:16.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f412006a7a0 0x7f412006cc60 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f412c036000 tx=0x7f412c030000 comp rx=0 tx=0).stop 2024-03-23T11:29:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 -- 172.21.15.62:0/3932991582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f413c06c6e0 msgr2=0x7f413c140570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f413c06c6e0 0x7f413c140570 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f413000c7f0 tx=0x7f413000ccb0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 -- 172.21.15.62:0/3932991582 shutdown_connections 2024-03-23T11:29:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f412006a7a0 0x7f412006cc60 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f413c06c6e0 0x7f413c140570 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413c0cafa0 0x7f413c1f1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 --2- 172.21.15.62:0/3932991582 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413c06c300 0x7f413c156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 -- 172.21.15.62:0/3932991582 >> 172.21.15.62:0/3932991582 conn(0x7f413c0b02e0 msgr2=0x7f413c15a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 -- 172.21.15.62:0/3932991582 shutdown_connections 2024-03-23T11:29:16.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.072+0000 7f4143400640 1 -- 172.21.15.62:0/3932991582 wait complete. 2024-03-23T11:29:16.081 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:29:16.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.294+0000 7f8fdf200640 1 Processor -- start 2024-03-23T11:29:16.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.295+0000 7f8fdf200640 1 -- start start 2024-03-23T11:29:16.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.296+0000 7f8fdf200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fe015fcf0 0x7f8fe00d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:16.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.296+0000 7f8fdf200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00d14f0 con 0x7f8fe0160420 2024-03-23T11:29:16.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.296+0000 7f8fdf200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe00d3540 con 0x7f8fe00b6dc0 2024-03-23T11:29:16.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.296+0000 7f8fdf200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe00ce510 con 0x7f8fe015fcf0 2024-03-23T11:29:16.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.296+0000 7f8fdde00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fe015fcf0 0x7f8fe00d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:16.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.296+0000 7f8fdde00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fe015fcf0 0x7f8fe00d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33668/0 (socket says 172.21.15.62:33668) 2024-03-23T11:29:16.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.296+0000 7f8fdde00640 1 -- 172.21.15.62:0/1944878491 learned_addr learned my addr 172.21.15.62:0/1944878491 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:16.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.297+0000 7f8fdde00640 1 -- 172.21.15.62:0/1944878491 >> v1:172.21.15.120:6789/0 conn(0x7f8fe00b6dc0 legacy=0x7f8fe00d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:16.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.297+0000 7f8fdde00640 1 -- 172.21.15.62:0/1944878491 >> v1:172.21.15.62:6789/0 conn(0x7f8fe0160420 legacy=0x7f8fe00d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:16.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.297+0000 7f8fdde00640 1 -- 172.21.15.62:0/1944878491 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe00cdc70 con 0x7f8fe015fcf0 2024-03-23T11:29:16.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.297+0000 7f8fdd400640 1 -- 172.21.15.62:0/1944878491 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187457019 0 0) 0x7f8fe00d14f0 con 0x7f8fe0160420 2024-03-23T11:29:16.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.298+0000 7f8fdde00640 1 --2- 172.21.15.62:0/1944878491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe015fcf0 0x7f8fe00d3000 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009870 tx=0x7f8fd002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62227c0603a6b48 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:16.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.299+0000 7f8fdd400640 1 -- 172.21.15.62:0/1944878491 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0031860 con 0x7f8fe015fcf0 2024-03-23T11:29:16.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.299+0000 7f8fdd400640 1 -- 172.21.15.62:0/1944878491 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd00319c0 con 0x7f8fe015fcf0 2024-03-23T11:29:16.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.299+0000 7f8fdd400640 1 -- 172.21.15.62:0/1944878491 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0031d50 con 0x7f8fe015fcf0 2024-03-23T11:29:16.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.299+0000 7f8fdf200640 1 -- 172.21.15.62:0/1944878491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe015fcf0 msgr2=0x7f8fe00d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:16.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.299+0000 7f8fdf200640 1 --2- 172.21.15.62:0/1944878491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe015fcf0 0x7f8fe00d3000 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009870 tx=0x7f8fd002b900 comp rx=0 tx=0).stop 2024-03-23T11:29:16.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.299+0000 7f8fdf200640 1 -- 172.21.15.62:0/1944878491 shutdown_connections 2024-03-23T11:29:16.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.299+0000 7f8fdf200640 1 --2- 172.21.15.62:0/1944878491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe015fcf0 0x7f8fe00d3000 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.299+0000 7f8fdf200640 1 -- 172.21.15.62:0/1944878491 >> 172.21.15.62:0/1944878491 conn(0x7f8fe0089f60 msgr2=0x7f8fe00d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:16.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.300+0000 7f8fdf200640 1 -- 172.21.15.62:0/1944878491 shutdown_connections 2024-03-23T11:29:16.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.300+0000 7f8fdf200640 1 -- 172.21.15.62:0/1944878491 wait complete. 2024-03-23T11:29:16.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.301+0000 7f8fdf200640 1 Processor -- start 2024-03-23T11:29:16.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.301+0000 7f8fdf200640 1 -- start start 2024-03-23T11:29:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.302+0000 7f8fdf200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe00b6dc0 0x7f8fe015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.302+0000 7f8fdf200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe015fcf0 0x7f8fe00c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.302+0000 7f8fdf200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0160420 0x7f8fe00cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.302+0000 7f8fdf200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe00c74f0 con 0x7f8fe015fcf0 2024-03-23T11:29:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.302+0000 7f8fdf200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe015b140 con 0x7f8fe0160420 2024-03-23T11:29:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.302+0000 7f8fdf200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe00c68e0 con 0x7f8fe00b6dc0 2024-03-23T11:29:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.303+0000 7f8fd5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe015fcf0 0x7f8fe00c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.303+0000 7f8fd5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe015fcf0 0x7f8fe00c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33672/0 (socket says 172.21.15.62:33672) 2024-03-23T11:29:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.303+0000 7f8fd5400640 1 -- 172.21.15.62:0/619791921 learned_addr learned my addr 172.21.15.62:0/619791921 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.303+0000 7f8fdde00640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe00b6dc0 0x7f8fe015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.303+0000 7f8fde800640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0160420 0x7f8fe00cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.303+0000 7f8fd5400640 1 -- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe00b6dc0 msgr2=0x7f8fe015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.303+0000 7f8fd5400640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe00b6dc0 0x7f8fe015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.303+0000 7f8fd5400640 1 -- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0160420 msgr2=0x7f8fe00cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.303+0000 7f8fd5400640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0160420 0x7f8fe00cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.303+0000 7f8fd5400640 1 -- 172.21.15.62:0/619791921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd00094d0 con 0x7f8fe015fcf0 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.303+0000 7f8fd5400640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe015fcf0 0x7f8fe00c5ed0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800c410 tx=0x7f8fc800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.304+0000 7f8fd6a00640 1 -- 172.21.15.62:0/619791921 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80035f0 con 0x7f8fe015fcf0 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.304+0000 7f8fd6a00640 1 -- 172.21.15.62:0/619791921 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc8003770 con 0x7f8fe015fcf0 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.304+0000 7f8fd6a00640 1 -- 172.21.15.62:0/619791921 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8003b20 con 0x7f8fe015fcf0 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.304+0000 7f8fdf200640 1 -- 172.21.15.62:0/619791921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe00c6bc0 con 0x7f8fe015fcf0 2024-03-23T11:29:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.304+0000 7f8fdf200640 1 -- 172.21.15.62:0/619791921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe00c7180 con 0x7f8fe015fcf0 2024-03-23T11:29:16.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.305+0000 7f8fdf200640 1 -- 172.21.15.62:0/619791921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe00c76a0 con 0x7f8fe015fcf0 2024-03-23T11:29:16.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.306+0000 7f8fd6a00640 1 -- 172.21.15.62:0/619791921 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8fc8003c80 con 0x7f8fe015fcf0 2024-03-23T11:29:16.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.307+0000 7f8fd6a00640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fbc06a7a0 0x7f8fbc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:16.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.307+0000 7f8fd6a00640 1 -- 172.21.15.62:0/619791921 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fc80884c0 con 0x7f8fe015fcf0 2024-03-23T11:29:16.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.308+0000 7f8fdde00640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fbc06a7a0 0x7f8fbc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:16.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.309+0000 7f8fdde00640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fbc06a7a0 0x7f8fbc06cc60 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0036000 tx=0x7f8fd0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.311+0000 7f8fd6a00640 1 -- 172.21.15.62:0/619791921 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8fe00c76a0 con 0x7f8fe015fcf0 2024-03-23T11:29:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.503+0000 7f8fdf200640 1 -- 172.21.15.62:0/619791921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fe0066bb0 con 0x7f8fe015fcf0 2024-03-23T11:29:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.505+0000 7f8fd6a00640 1 -- 172.21.15.62:0/619791921 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+16064 (secure 0 0 0) 0x7f8fc8053840 con 0x7f8fe015fcf0 2024-03-23T11:29:16.496 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:16.496 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":104,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:29:15.625446+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:29:13.817605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-23T11:29:14.652617+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:29:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.507+0000 7f8fdf200640 1 -- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fbc06a7a0 msgr2=0x7f8fbc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.507+0000 7f8fdf200640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fbc06a7a0 0x7f8fbc06cc60 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0036000 tx=0x7f8fd0008030 comp rx=0 tx=0).stop 2024-03-23T11:29:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.507+0000 7f8fdf200640 1 -- 172.21.15.62:0/619791921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe015fcf0 msgr2=0x7f8fe00c5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.507+0000 7f8fdf200640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe015fcf0 0x7f8fe00c5ed0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800c410 tx=0x7f8fc800c8d0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.507+0000 7f8fdf200640 1 -- 172.21.15.62:0/619791921 shutdown_connections 2024-03-23T11:29:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.507+0000 7f8fdf200640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fbc06a7a0 0x7f8fbc06cc60 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.507+0000 7f8fdf200640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe015fcf0 0x7f8fe00c5ed0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.507+0000 7f8fdf200640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe0160420 0x7f8fe00cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.507+0000 7f8fdf200640 1 --2- 172.21.15.62:0/619791921 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe00b6dc0 0x7f8fe015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.507+0000 7f8fdf200640 1 -- 172.21.15.62:0/619791921 >> 172.21.15.62:0/619791921 conn(0x7f8fe0089f60 msgr2=0x7f8fe006c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:16.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.508+0000 7f8fdf200640 1 -- 172.21.15.62:0/619791921 shutdown_connections 2024-03-23T11:29:16.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.508+0000 7f8fdf200640 1 -- 172.21.15.62:0/619791921 wait complete. 2024-03-23T11:29:16.518 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T11:29:16.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.742+0000 7f2a12800640 1 Processor -- start 2024-03-23T11:29:16.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.742+0000 7f2a12800640 1 -- start start 2024-03-23T11:29:16.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.744+0000 7f2a12800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c143dd0 0x7f2a0c15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:16.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.744+0000 7f2a12800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2a0c070090 0x7f2a0c0704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:16.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.744+0000 7f2a12800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c070a30 con 0x7f2a0c0b2820 2024-03-23T11:29:16.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.744+0000 7f2a12800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0c0cd140 con 0x7f2a0c070090 2024-03-23T11:29:16.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.744+0000 7f2a12800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c0cd3e0 con 0x7f2a0c143dd0 2024-03-23T11:29:16.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.744+0000 7f2a10a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2a0c070090 0x7f2a0c0704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:16.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.744+0000 7f2a10a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2a0c070090 0x7f2a0c0704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33676/0 (socket says 172.21.15.62:33676) 2024-03-23T11:29:16.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.744+0000 7f2a10a00640 1 -- 172.21.15.62:0/1901416951 learned_addr learned my addr 172.21.15.62:0/1901416951 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:16.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.744+0000 7f2a11400640 1 --2- 172.21.15.62:0/1901416951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c143dd0 0x7f2a0c15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:16.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.744+0000 7f2a10a00640 1 -- 172.21.15.62:0/1901416951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c143dd0 msgr2=0x7f2a0c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:16.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.745+0000 7f2a10a00640 1 --2- 172.21.15.62:0/1901416951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c143dd0 0x7f2a0c15abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.745+0000 7f2a10a00640 1 -- 172.21.15.62:0/1901416951 >> v1:172.21.15.120:6789/0 conn(0x7f2a0c0b2820 legacy=0x7f2a0c0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:16.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.745+0000 7f2a10a00640 1 -- 172.21.15.62:0/1901416951 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c0cd6c0 con 0x7f2a0c070090 2024-03-23T11:29:16.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.745+0000 7f2a11400640 1 --2- 172.21.15.62:0/1901416951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c143dd0 0x7f2a0c15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:16.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.746+0000 7f2a10a00640 1 --2- 172.21.15.62:0/1901416951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c070090 0x7f2a0c0704f0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009870 tx=0x7f29f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410572dd155b942d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:16.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.746+0000 7f2a0be00640 1 -- 172.21.15.62:0/1901416951 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8031640 con 0x7f2a0c070090 2024-03-23T11:29:16.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.746+0000 7f2a0be00640 1 -- 172.21.15.62:0/1901416951 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f80317a0 con 0x7f2a0c070090 2024-03-23T11:29:16.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.746+0000 7f2a0be00640 1 -- 172.21.15.62:0/1901416951 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8031b30 con 0x7f2a0c070090 2024-03-23T11:29:16.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.747+0000 7f2a12800640 1 -- 172.21.15.62:0/1901416951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c070090 msgr2=0x7f2a0c0704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:16.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.747+0000 7f2a12800640 1 --2- 172.21.15.62:0/1901416951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c070090 0x7f2a0c0704f0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009870 tx=0x7f29f802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.747+0000 7f2a12800640 1 -- 172.21.15.62:0/1901416951 shutdown_connections 2024-03-23T11:29:16.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.747+0000 7f2a12800640 1 --2- 172.21.15.62:0/1901416951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c070090 0x7f2a0c0704f0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.747+0000 7f2a12800640 1 --2- 172.21.15.62:0/1901416951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c143dd0 0x7f2a0c15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.747+0000 7f2a12800640 1 -- 172.21.15.62:0/1901416951 >> 172.21.15.62:0/1901416951 conn(0x7f2a0c066050 msgr2=0x7f2a0c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:16.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.747+0000 7f2a12800640 1 -- 172.21.15.62:0/1901416951 shutdown_connections 2024-03-23T11:29:16.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.747+0000 7f2a12800640 1 -- 172.21.15.62:0/1901416951 wait complete. 2024-03-23T11:29:16.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.748+0000 7f2a12800640 1 Processor -- start 2024-03-23T11:29:16.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.749+0000 7f2a12800640 1 -- start start 2024-03-23T11:29:16.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.749+0000 7f2a12800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c070090 0x7f2a0c1f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:16.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.749+0000 7f2a12800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a0c0b2820 0x7f2a0c15c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:16.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.749+0000 7f2a12800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c143dd0 0x7f2a0c159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:16.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a12800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c136b70 con 0x7f2a0c070090 2024-03-23T11:29:16.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a12800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c136e50 con 0x7f2a0c0b2820 2024-03-23T11:29:16.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a12800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c137130 con 0x7f2a0c143dd0 2024-03-23T11:29:16.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a11400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c070090 0x7f2a0c1f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:16.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a11400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c070090 0x7f2a0c1f2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33688/0 (socket says 172.21.15.62:33688) 2024-03-23T11:29:16.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a11400640 1 -- 172.21.15.62:0/1188359202 learned_addr learned my addr 172.21.15.62:0/1188359202 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:16.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a11e00640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c143dd0 0x7f2a0c159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:16.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a10a00640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a0c0b2820 0x7f2a0c15c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:16.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a11400640 1 -- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c143dd0 msgr2=0x7f2a0c159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:16.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a11400640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c143dd0 0x7f2a0c159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a11400640 1 -- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a0c0b2820 msgr2=0x7f2a0c15c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:16.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a11400640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a0c0b2820 0x7f2a0c15c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:16.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a11400640 1 -- 172.21.15.62:0/1188359202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f80094d0 con 0x7f2a0c070090 2024-03-23T11:29:16.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.750+0000 7f2a10a00640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a0c0b2820 0x7f2a0c15c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:16.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.751+0000 7f2a11400640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c070090 0x7f2a0c1f2740 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000c7f0 tx=0x7f2a0000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:16.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.751+0000 7f2a09600640 1 -- 172.21.15.62:0/1188359202 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00007820 con 0x7f2a0c070090 2024-03-23T11:29:16.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.751+0000 7f2a09600640 1 -- 172.21.15.62:0/1188359202 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a00007980 con 0x7f2a0c070090 2024-03-23T11:29:16.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.751+0000 7f2a09600640 1 -- 172.21.15.62:0/1188359202 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00007cf0 con 0x7f2a0c070090 2024-03-23T11:29:16.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.751+0000 7f2a12800640 1 -- 172.21.15.62:0/1188359202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c137410 con 0x7f2a0c070090 2024-03-23T11:29:16.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.751+0000 7f2a12800640 1 -- 172.21.15.62:0/1188359202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c1f2c80 con 0x7f2a0c070090 2024-03-23T11:29:16.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.752+0000 7f2a12800640 1 -- 172.21.15.62:0/1188359202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c066bb0 con 0x7f2a0c070090 2024-03-23T11:29:16.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.754+0000 7f2a09600640 1 -- 172.21.15.62:0/1188359202 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2a000113f0 con 0x7f2a0c070090 2024-03-23T11:29:16.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.755+0000 7f2a09600640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29ec06a7a0 0x7f29ec06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:16.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.755+0000 7f2a09600640 1 -- 172.21.15.62:0/1188359202 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a000890d0 con 0x7f2a0c070090 2024-03-23T11:29:16.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.756+0000 7f2a10a00640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29ec06a7a0 0x7f29ec06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:16.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.757+0000 7f2a10a00640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29ec06a7a0 0x7f29ec06cc60 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f29f8036000 tx=0x7f29f803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:16.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.760+0000 7f2a09600640 1 -- 172.21.15.62:0/1188359202 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2a000544d0 con 0x7f2a0c070090 2024-03-23T11:29:16.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:16.987+0000 7f2a12800640 1 -- 172.21.15.62:0/1188359202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2a0c0acdd0 con 0x7f2a0c070090 2024-03-23T11:29:17.321 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:29:17.329 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:29:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.648+0000 7f2a09600640 1 -- 172.21.15.62:0/1188359202 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v77) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2a00057af0 con 0x7f2a0c070090 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.652+0000 7f2a12800640 1 -- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29ec06a7a0 msgr2=0x7f29ec06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.652+0000 7f2a12800640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29ec06a7a0 0x7f29ec06cc60 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f29f8036000 tx=0x7f29f803c040 comp rx=0 tx=0).stop 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.652+0000 7f2a12800640 1 -- 172.21.15.62:0/1188359202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c070090 msgr2=0x7f2a0c1f2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.652+0000 7f2a12800640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c070090 0x7f2a0c1f2740 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000c7f0 tx=0x7f2a0000ccb0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.652+0000 7f2a12800640 1 -- 172.21.15.62:0/1188359202 shutdown_connections 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.652+0000 7f2a12800640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29ec06a7a0 0x7f29ec06cc60 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.652+0000 7f2a12800640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2a0c143dd0 0x7f2a0c159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.652+0000 7f2a12800640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a0c0b2820 0x7f2a0c15c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.652+0000 7f2a12800640 1 --2- 172.21.15.62:0/1188359202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c070090 0x7f2a0c1f2740 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.652+0000 7f2a12800640 1 -- 172.21.15.62:0/1188359202 >> 172.21.15.62:0/1188359202 conn(0x7f2a0c066050 msgr2=0x7f2a0c0b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.653+0000 7f2a12800640 1 -- 172.21.15.62:0/1188359202 shutdown_connections 2024-03-23T11:29:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.653+0000 7f2a12800640 1 -- 172.21.15.62:0/1188359202 wait complete. 2024-03-23T11:29:17.666 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T11:29:17.666 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T11:29:17.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.903+0000 7fa708a00640 1 Processor -- start 2024-03-23T11:29:17.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.903+0000 7fa708a00640 1 -- start start 2024-03-23T11:29:17.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.905+0000 7fa708a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa704098b20 0x7fa704099770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:17.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.905+0000 7fa708a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70406bc30 con 0x7fa70406b780 2024-03-23T11:29:17.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.905+0000 7fa708a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa704099f50 con 0x7fa70415e7a0 2024-03-23T11:29:17.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.905+0000 7fa708a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7040cf5b0 con 0x7fa704098b20 2024-03-23T11:29:17.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.905+0000 7fa701400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa70406b780 0x7fa704098f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47090/0 (socket says 172.21.15.62:47090) 2024-03-23T11:29:17.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.905+0000 7fa701400640 1 -- 172.21.15.62:0/3546900727 learned_addr learned my addr 172.21.15.62:0/3546900727 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:17.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.905+0000 7fa701e00640 1 --2- 172.21.15.62:0/3546900727 >> v2:172.21.15.120:3300/0 conn(0x7fa704098b20 0x7fa704099770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:17.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303015283 0 0) 0x7fa70406bc30 con 0x7fa70406b780 2024-03-23T11:29:17.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6e8003520 con 0x7fa70406b780 2024-03-23T11:29:17.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 528796041 0 0) 0x7fa6e8003520 con 0x7fa70406b780 2024-03-23T11:29:17.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 >> v2:172.21.15.120:3300/0 conn(0x7fa704098b20 msgr2=0x7fa704099770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 --2- 172.21.15.62:0/3546900727 >> v2:172.21.15.120:3300/0 conn(0x7fa704098b20 0x7fa704099770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 >> v1:172.21.15.120:6789/0 conn(0x7fa70415e7a0 legacy=0x7fa70415ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7040cf890 con 0x7fa70406b780 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fa6f0001670 con 0x7fa70406b780 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 >> v1:172.21.15.62:6789/0 conn(0x7fa70406b780 legacy=0x7fa704098f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 --2- 172.21.15.62:0/3546900727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80039c0 0x7fa6e8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 --2- 172.21.15.62:0/3546900727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6e80047c0 0x7fa6e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 --2- 172.21.15.62:0/3546900727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e80051d0 0x7fa6e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040cf5b0 con 0x7fa6e80051d0 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e800c120 con 0x7fa6e80047c0 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.906+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e800c2e0 con 0x7fa6e80039c0 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.907+0000 7fa702800640 1 --2- 172.21.15.62:0/3546900727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e80051d0 0x7fa6e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.907+0000 7fa701400640 1 --2- 172.21.15.62:0/3546900727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6e80047c0 0x7fa6e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:17.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.907+0000 7fa701e00640 1 --2- 172.21.15.62:0/3546900727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80039c0 0x7fa6e8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:17.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.907+0000 7fa702800640 1 -- 172.21.15.62:0/3546900727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80039c0 msgr2=0x7fa6e8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:17.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.907+0000 7fa702800640 1 --2- 172.21.15.62:0/3546900727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80039c0 0x7fa6e8003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.907+0000 7fa702800640 1 -- 172.21.15.62:0/3546900727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6e80047c0 msgr2=0x7fa6e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:17.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.907+0000 7fa702800640 1 --2- 172.21.15.62:0/3546900727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6e80047c0 0x7fa6e8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.907+0000 7fa702800640 1 -- 172.21.15.62:0/3546900727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e8003520 con 0x7fa6e80051d0 2024-03-23T11:29:17.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.908+0000 7fa702800640 1 -- 172.21.15.62:0/3546900727 reap_dead start 2024-03-23T11:29:17.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.908+0000 7fa702800640 1 --2- 172.21.15.62:0/3546900727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e80051d0 0x7fa6e800b980 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc001e20 tx=0x7fa6e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621b6df0e53185e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:17.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.908+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc003b70 con 0x7fa6e80051d0 2024-03-23T11:29:17.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.908+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6fc003cd0 con 0x7fa6e80051d0 2024-03-23T11:29:17.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.908+0000 7fa700a00640 1 -- 172.21.15.62:0/3546900727 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc0052f0 con 0x7fa6e80051d0 2024-03-23T11:29:17.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.909+0000 7fa708a00640 1 -- 172.21.15.62:0/3546900727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e80051d0 msgr2=0x7fa6e800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:17.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.909+0000 7fa708a00640 1 --2- 172.21.15.62:0/3546900727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e80051d0 0x7fa6e800b980 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc001e20 tx=0x7fa6e80057f0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.909+0000 7fa708a00640 1 -- 172.21.15.62:0/3546900727 shutdown_connections 2024-03-23T11:29:17.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.909+0000 7fa708a00640 1 --2- 172.21.15.62:0/3546900727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e80051d0 0x7fa6e800b980 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.909+0000 7fa708a00640 1 -- 172.21.15.62:0/3546900727 >> 172.21.15.62:0/3546900727 conn(0x7fa7040b23f0 msgr2=0x7fa7040b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:17.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.909+0000 7fa708a00640 1 -- 172.21.15.62:0/3546900727 shutdown_connections 2024-03-23T11:29:17.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.909+0000 7fa708a00640 1 -- 172.21.15.62:0/3546900727 wait complete. 2024-03-23T11:29:17.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.911+0000 7fa708a00640 1 Processor -- start 2024-03-23T11:29:17.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.911+0000 7fa708a00640 1 -- start start 2024-03-23T11:29:17.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.912+0000 7fa708a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80051d0 0x7fa7041f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:17.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.912+0000 7fa708a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa704154760 0x7fa7040b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:17.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.912+0000 7fa708a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7040b1760 0x7fa7040b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:17.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.912+0000 7fa708a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040b4d60 con 0x7fa704154760 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.912+0000 7fa708a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040b2120 con 0x7fa7040b1760 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.912+0000 7fa708a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa704158450 con 0x7fa6e80051d0 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.913+0000 7fa701400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa704154760 0x7fa7040b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.913+0000 7fa701400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa704154760 0x7fa7040b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54366/0 (socket says 172.21.15.62:54366) 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.913+0000 7fa701400640 1 -- 172.21.15.62:0/1280354029 learned_addr learned my addr 172.21.15.62:0/1280354029 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.913+0000 7fa701e00640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80051d0 0x7fa7041f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.913+0000 7fa702800640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7040b1760 0x7fa7040b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.913+0000 7fa701400640 1 -- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80051d0 msgr2=0x7fa7041f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.913+0000 7fa701400640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80051d0 0x7fa7041f32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.913+0000 7fa701400640 1 -- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7040b1760 msgr2=0x7fa7040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.913+0000 7fa701400640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7040b1760 0x7fa7040b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.913+0000 7fa701400640 1 -- 172.21.15.62:0/1280354029 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7041f0270 con 0x7fa704154760 2024-03-23T11:29:17.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.913+0000 7fa701400640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa704154760 0x7fa7040b5180 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000e730 tx=0x7fa6f000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:17.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.914+0000 7fa6fa000640 1 -- 172.21.15.62:0/1280354029 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f000c6c0 con 0x7fa704154760 2024-03-23T11:29:17.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.914+0000 7fa6fa000640 1 -- 172.21.15.62:0/1280354029 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f000c820 con 0x7fa704154760 2024-03-23T11:29:17.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.914+0000 7fa6fa000640 1 -- 172.21.15.62:0/1280354029 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f000cbd0 con 0x7fa704154760 2024-03-23T11:29:17.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.914+0000 7fa708a00640 1 -- 172.21.15.62:0/1280354029 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7041f37e0 con 0x7fa704154760 2024-03-23T11:29:17.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.914+0000 7fa708a00640 1 -- 172.21.15.62:0/1280354029 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa704158170 con 0x7fa704154760 2024-03-23T11:29:17.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.915+0000 7fa708a00640 1 -- 172.21.15.62:0/1280354029 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7040cd180 con 0x7fa704154760 2024-03-23T11:29:17.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.916+0000 7fa6fa000640 1 -- 172.21.15.62:0/1280354029 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa6f000cd30 con 0x7fa704154760 2024-03-23T11:29:17.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.917+0000 7fa6fa000640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa6e406a7d0 0x7fa6e406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:17.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.917+0000 7fa6fa000640 1 -- 172.21.15.62:0/1280354029 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6f0088c50 con 0x7fa704154760 2024-03-23T11:29:17.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.917+0000 7fa701e00640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa6e406a7d0 0x7fa6e406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:17.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.919+0000 7fa701e00640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa6e406a7d0 0x7fa6e406cc90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4009a10 tx=0x7fa6f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:17.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:17.921+0000 7fa6fa000640 1 -- 172.21.15.62:0/1280354029 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa6f0054060 con 0x7fa704154760 2024-03-23T11:29:18.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:18.157+0000 7fa708a00640 1 -- 172.21.15.62:0/1280354029 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa7040b4ea0 con 0x7fa704154760 2024-03-23T11:29:19.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.163+0000 7fa6fa000640 1 -- 172.21.15.62:0/1280354029 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v78) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa7040b4ea0 con 0x7fa704154760 2024-03-23T11:29:19.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 -- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa6e406a7d0 msgr2=0x7fa6e406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:19.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa6e406a7d0 0x7fa6e406cc90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4009a10 tx=0x7fa6f4008040 comp rx=0 tx=0).stop 2024-03-23T11:29:19.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 -- 172.21.15.62:0/1280354029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa704154760 msgr2=0x7fa7040b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:19.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa704154760 0x7fa7040b5180 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000e730 tx=0x7fa6f000ebf0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 -- 172.21.15.62:0/1280354029 shutdown_connections 2024-03-23T11:29:19.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa6e406a7d0 0x7fa6e406cc90 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa704154760 0x7fa7040b5180 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7040b1760 0x7fa7040b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 --2- 172.21.15.62:0/1280354029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6e80051d0 0x7fa7041f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 -- 172.21.15.62:0/1280354029 >> 172.21.15.62:0/1280354029 conn(0x7fa7040b23f0 msgr2=0x7fa7040af580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:19.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 -- 172.21.15.62:0/1280354029 shutdown_connections 2024-03-23T11:29:19.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.167+0000 7fa708a00640 1 -- 172.21.15.62:0/1280354029 wait complete. 2024-03-23T11:29:19.183 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T11:29:19.183 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:29:19.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.423+0000 7fb180c00640 1 Processor -- start 2024-03-23T11:29:19.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.423+0000 7fb180c00640 1 -- start start 2024-03-23T11:29:19.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.424+0000 7fb180c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb17c06dca0 0x7fb17c0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.424+0000 7fb180c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb17c06c300 0x7fb17c0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.424+0000 7fb180c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c0cad20 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.424+0000 7fb180c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb17c06e130 con 0x7fb17c06dca0 2024-03-23T11:29:19.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.425+0000 7fb180c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb17c15a8a0 con 0x7fb17c06c300 2024-03-23T11:29:19.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.425+0000 7fb179400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb17c06dca0 0x7fb17c0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.425+0000 7fb179400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb17c06dca0 0x7fb17c0b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54380/0 (socket says 172.21.15.62:54380) 2024-03-23T11:29:19.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.425+0000 7fb179400640 1 -- 172.21.15.62:0/3829718489 learned_addr learned my addr 172.21.15.62:0/3829718489 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:19.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.425+0000 7fb17a800640 1 --2- 172.21.15.62:0/3829718489 >> v2:172.21.15.120:3300/0 conn(0x7fb17c06c300 0x7fb17c0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.426+0000 7fb179400640 1 -- 172.21.15.62:0/3829718489 >> v2:172.21.15.120:3300/0 conn(0x7fb17c06c300 msgr2=0x7fb17c0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:19.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.426+0000 7fb179400640 1 --2- 172.21.15.62:0/3829718489 >> v2:172.21.15.120:3300/0 conn(0x7fb17c06c300 0x7fb17c0cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.426+0000 7fb179400640 1 -- 172.21.15.62:0/3829718489 >> v1:172.21.15.120:6789/0 conn(0x7fb17c0cd7c0 legacy=0x7fb17c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:19.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.426+0000 7fb179400640 1 -- 172.21.15.62:0/3829718489 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17c15ab80 con 0x7fb17c06dca0 2024-03-23T11:29:19.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.427+0000 7fb179400640 1 --2- 172.21.15.62:0/3829718489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb17c06dca0 0x7fb17c0b15e0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0b3a20 tx=0x7fb16802da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7253620437446051 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:19.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.427+0000 7fb178a00640 1 -- 172.21.15.62:0/3829718489 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16802f850 con 0x7fb17c06dca0 2024-03-23T11:29:19.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.427+0000 7fb178a00640 1 -- 172.21.15.62:0/3829718489 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16802f9b0 con 0x7fb17c06dca0 2024-03-23T11:29:19.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.427+0000 7fb178a00640 1 -- 172.21.15.62:0/3829718489 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16802fd40 con 0x7fb17c06dca0 2024-03-23T11:29:19.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.428+0000 7fb180c00640 1 -- 172.21.15.62:0/3829718489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb17c06dca0 msgr2=0x7fb17c0b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:19.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.428+0000 7fb180c00640 1 --2- 172.21.15.62:0/3829718489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb17c06dca0 0x7fb17c0b15e0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0b3a20 tx=0x7fb16802da90 comp rx=0 tx=0).stop 2024-03-23T11:29:19.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.428+0000 7fb180c00640 1 -- 172.21.15.62:0/3829718489 shutdown_connections 2024-03-23T11:29:19.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.428+0000 7fb180c00640 1 --2- 172.21.15.62:0/3829718489 >> v2:172.21.15.120:3300/0 conn(0x7fb17c06c300 0x7fb17c0cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.428+0000 7fb180c00640 1 --2- 172.21.15.62:0/3829718489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb17c06dca0 0x7fb17c0b15e0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.428+0000 7fb180c00640 1 -- 172.21.15.62:0/3829718489 >> 172.21.15.62:0/3829718489 conn(0x7fb17c0b02e0 msgr2=0x7fb17c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:19.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.428+0000 7fb180c00640 1 -- 172.21.15.62:0/3829718489 shutdown_connections 2024-03-23T11:29:19.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.428+0000 7fb180c00640 1 -- 172.21.15.62:0/3829718489 wait complete. 2024-03-23T11:29:19.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.430+0000 7fb180c00640 1 Processor -- start 2024-03-23T11:29:19.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.430+0000 7fb180c00640 1 -- start start 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.431+0000 7fb180c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb17c06c300 0x7fb17c144df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.431+0000 7fb180c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb17c0cd7c0 0x7fb17c157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.431+0000 7fb180c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c1582e0 0x7fb17c153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.431+0000 7fb180c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c145330 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.431+0000 7fb180c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c158860 con 0x7fb17c06c300 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.431+0000 7fb180c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c158b40 con 0x7fb17c1582e0 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.431+0000 7fb179400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb17c0cd7c0 0x7fb17c157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.432+0000 7fb179400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb17c0cd7c0 0x7fb17c157da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54384/0 (socket says 172.21.15.62:54384) 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.432+0000 7fb179400640 1 -- 172.21.15.62:0/2617023019 learned_addr learned my addr 172.21.15.62:0/2617023019 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.432+0000 7fb179e00640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb17c06c300 0x7fb17c144df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.432+0000 7fb179400640 1 -- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c1582e0 msgr2=0x7fb17c153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:19.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.432+0000 7fb17a800640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c1582e0 0x7fb17c153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.432+0000 7fb179400640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c1582e0 0x7fb17c153c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.432+0000 7fb179400640 1 -- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb17c06c300 msgr2=0x7fb17c144df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:19.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.432+0000 7fb179400640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb17c06c300 0x7fb17c144df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.432+0000 7fb179400640 1 -- 172.21.15.62:0/2617023019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1680094d0 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.432+0000 7fb179e00640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb17c06c300 0x7fb17c144df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:19.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.432+0000 7fb179400640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb17c0cd7c0 0x7fb17c157da0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fb168009960 tx=0x7fb168004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:19.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.433+0000 7fb172000640 1 -- 172.21.15.62:0/2617023019 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16803e040 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.433+0000 7fb172000640 1 -- 172.21.15.62:0/2617023019 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb168039950 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.433+0000 7fb172000640 1 -- 172.21.15.62:0/2617023019 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb168039d00 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.433+0000 7fb180c00640 1 -- 172.21.15.62:0/2617023019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb17c158e20 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.433+0000 7fb180c00640 1 -- 172.21.15.62:0/2617023019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17c1593e0 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.434+0000 7fb180c00640 1 -- 172.21.15.62:0/2617023019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17c1595d0 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.435+0000 7fb172000640 1 -- 172.21.15.62:0/2617023019 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb168043390 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.436+0000 7fb172000640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb15c06a7a0 0x7fb15c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.436+0000 7fb172000640 1 -- 172.21.15.62:0/2617023019 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb168034070 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.436+0000 7fb179e00640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb15c06a7a0 0x7fb15c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.437+0000 7fb179e00640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb15c06a7a0 0x7fb15c06cc60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00a830 tx=0x7fb16c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:19.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.440+0000 7fb172000640 1 -- 172.21.15.62:0/2617023019 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb168077e80 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.676+0000 7fb180c00640 1 -- 172.21.15.62:0/2617023019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb17c066c90 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.677+0000 7fb172000640 1 -- 172.21.15.62:0/2617023019 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fb168039e60 con 0x7fb17c0cd7c0 2024-03-23T11:29:19.668 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:19.668 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":78,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:29:19.157087+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8271":{"gid":8271,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6834/692790652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":692790652},{"type":"v1","addr":"172.21.15.62:6834","nonce":692790652}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:29:19.669 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 78 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.680+0000 7fb180c00640 1 -- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb15c06a7a0 msgr2=0x7fb15c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.680+0000 7fb180c00640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb15c06a7a0 0x7fb15c06cc60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00a830 tx=0x7fb16c008040 comp rx=0 tx=0).stop 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.680+0000 7fb180c00640 1 -- 172.21.15.62:0/2617023019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb17c0cd7c0 msgr2=0x7fb17c157da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.680+0000 7fb180c00640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb17c0cd7c0 0x7fb17c157da0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fb168009960 tx=0x7fb168004730 comp rx=0 tx=0).stop 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.680+0000 7fb180c00640 1 -- 172.21.15.62:0/2617023019 shutdown_connections 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.680+0000 7fb180c00640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb15c06a7a0 0x7fb15c06cc60 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.680+0000 7fb180c00640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb17c0cd7c0 0x7fb17c157da0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.680+0000 7fb180c00640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb17c1582e0 0x7fb17c153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.680+0000 7fb180c00640 1 --2- 172.21.15.62:0/2617023019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb17c06c300 0x7fb17c144df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.680+0000 7fb180c00640 1 -- 172.21.15.62:0/2617023019 >> 172.21.15.62:0/2617023019 conn(0x7fb17c0b02e0 msgr2=0x7fb17c0b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.681+0000 7fb180c00640 1 -- 172.21.15.62:0/2617023019 shutdown_connections 2024-03-23T11:29:19.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.681+0000 7fb180c00640 1 -- 172.21.15.62:0/2617023019 wait complete. 2024-03-23T11:29:19.691 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:29:19.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.920+0000 7f5aa5c00640 1 Processor -- start 2024-03-23T11:29:19.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.920+0000 7f5aa5c00640 1 -- start start 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.922+0000 7f5aa5c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5aa0160310 0x7f5aa00ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.922+0000 7f5aa5c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5aa00ce250 0x7f5aa00ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.922+0000 7f5aa5c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa01608d0 con 0x7f5aa015c830 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.922+0000 7f5aa5c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aa00addb0 con 0x7f5aa00ce250 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.922+0000 7f5aa5c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aa015cc10 con 0x7f5aa0160310 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.922+0000 7f5aa5200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5aa00ce250 0x7f5aa00ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.922+0000 7f5aa5200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5aa00ce250 0x7f5aa00ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54400/0 (socket says 172.21.15.62:54400) 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.922+0000 7f5aa5200640 1 -- 172.21.15.62:0/395850772 learned_addr learned my addr 172.21.15.62:0/395850772 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.922+0000 7f5a9f400640 1 --2- 172.21.15.62:0/395850772 >> v2:172.21.15.120:3300/0 conn(0x7f5aa0160310 0x7f5aa00ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.923+0000 7f5aa5200640 1 -- 172.21.15.62:0/395850772 >> v2:172.21.15.120:3300/0 conn(0x7f5aa0160310 msgr2=0x7f5aa00ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.923+0000 7f5aa5200640 1 --2- 172.21.15.62:0/395850772 >> v2:172.21.15.120:3300/0 conn(0x7f5aa0160310 0x7f5aa00ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.923+0000 7f5aa5200640 1 -- 172.21.15.62:0/395850772 >> v1:172.21.15.120:6789/0 conn(0x7f5aa015c830 legacy=0x7f5aa00ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.923+0000 7f5aa5200640 1 -- 172.21.15.62:0/395850772 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa0160bb0 con 0x7f5aa00ce250 2024-03-23T11:29:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.924+0000 7f5aa5200640 1 --2- 172.21.15.62:0/395850772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa00ce250 0x7f5aa00ce6e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00b3a80 tx=0x7f5a9802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e7f7d75ce1a770 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.924+0000 7f5a9ea00640 1 -- 172.21.15.62:0/395850772 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9802ca80 con 0x7f5aa00ce250 2024-03-23T11:29:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.924+0000 7f5a9ea00640 1 -- 172.21.15.62:0/395850772 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9802cbe0 con 0x7f5aa00ce250 2024-03-23T11:29:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.924+0000 7f5a9ea00640 1 -- 172.21.15.62:0/395850772 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a980343f0 con 0x7f5aa00ce250 2024-03-23T11:29:19.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.925+0000 7f5aa5c00640 1 -- 172.21.15.62:0/395850772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa00ce250 msgr2=0x7f5aa00ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:19.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.925+0000 7f5aa5c00640 1 --2- 172.21.15.62:0/395850772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa00ce250 0x7f5aa00ce6e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00b3a80 tx=0x7f5a9802bee0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.925+0000 7f5aa5c00640 1 -- 172.21.15.62:0/395850772 shutdown_connections 2024-03-23T11:29:19.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.925+0000 7f5aa5c00640 1 --2- 172.21.15.62:0/395850772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa00ce250 0x7f5aa00ce6e0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.925+0000 7f5aa5c00640 1 --2- 172.21.15.62:0/395850772 >> v2:172.21.15.120:3300/0 conn(0x7f5aa0160310 0x7f5aa00ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.925+0000 7f5aa5c00640 1 -- 172.21.15.62:0/395850772 >> 172.21.15.62:0/395850772 conn(0x7f5aa0066050 msgr2=0x7f5aa0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:19.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.925+0000 7f5aa5c00640 1 -- 172.21.15.62:0/395850772 shutdown_connections 2024-03-23T11:29:19.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.925+0000 7f5aa5c00640 1 -- 172.21.15.62:0/395850772 wait complete. 2024-03-23T11:29:19.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.927+0000 7f5aa5c00640 1 Processor -- start 2024-03-23T11:29:19.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.927+0000 7f5aa5c00640 1 -- start start 2024-03-23T11:29:19.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5aa5c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa00ce250 0x7f5aa00b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5aa5c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa015c830 0x7f5aa00c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5aa5c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5aa0160310 0x7f5aa00ae430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5aa5c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa00c67f0 con 0x7f5aa015c830 2024-03-23T11:29:19.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5aa5c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa00c5a10 con 0x7f5aa00ce250 2024-03-23T11:29:19.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5aa5c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa00c5c70 con 0x7f5aa0160310 2024-03-23T11:29:19.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5a9f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa015c830 0x7f5aa00c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5a9f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa015c830 0x7f5aa00c54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54416/0 (socket says 172.21.15.62:54416) 2024-03-23T11:29:19.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5a9f400640 1 -- 172.21.15.62:0/2210428649 learned_addr learned my addr 172.21.15.62:0/2210428649 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:19.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5a9fe00640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa00ce250 0x7f5aa00b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5a9f400640 1 -- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5aa0160310 msgr2=0x7f5aa00ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:19.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5aa5200640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5aa0160310 0x7f5aa00ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5a9f400640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5aa0160310 0x7f5aa00ae430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5a9f400640 1 -- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa00ce250 msgr2=0x7f5aa00b0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:19.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5a9f400640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa00ce250 0x7f5aa00b0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:19.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5a9f400640 1 -- 172.21.15.62:0/2210428649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a98009b30 con 0x7f5aa015c830 2024-03-23T11:29:19.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.928+0000 7f5a9fe00640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa00ce250 0x7f5aa00b0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:19.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.929+0000 7f5a9f400640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa015c830 0x7f5aa00c54d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c00d800 tx=0x7f5a8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:19.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.929+0000 7f5a97e00640 1 -- 172.21.15.62:0/2210428649 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c00b6c0 con 0x7f5aa015c830 2024-03-23T11:29:19.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.929+0000 7f5a97e00640 1 -- 172.21.15.62:0/2210428649 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8c00b820 con 0x7f5aa015c830 2024-03-23T11:29:19.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.929+0000 7f5aa5c00640 1 -- 172.21.15.62:0/2210428649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa00c5f50 con 0x7f5aa015c830 2024-03-23T11:29:19.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.929+0000 7f5a97e00640 1 -- 172.21.15.62:0/2210428649 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c00bbb0 con 0x7f5aa015c830 2024-03-23T11:29:19.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.929+0000 7f5aa5c00640 1 -- 172.21.15.62:0/2210428649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa00c6510 con 0x7f5aa015c830 2024-03-23T11:29:19.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.930+0000 7f5aa5c00640 1 -- 172.21.15.62:0/2210428649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa0066bb0 con 0x7f5aa015c830 2024-03-23T11:29:19.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.931+0000 7f5a97e00640 1 -- 172.21.15.62:0/2210428649 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5a8c00bd10 con 0x7f5aa015c830 2024-03-23T11:29:19.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.932+0000 7f5a97e00640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a8006a7f0 0x7f5a8006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:19.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.932+0000 7f5a97e00640 1 -- 172.21.15.62:0/2210428649 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a8c089350 con 0x7f5aa015c830 2024-03-23T11:29:19.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.933+0000 7f5a9fe00640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a8006a7f0 0x7f5a8006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:19.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.934+0000 7f5a9fe00640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a8006a7f0 0x7f5a8006ccb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5a90007fd0 tx=0x7f5a90009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:19.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:19.938+0000 7f5a97e00640 1 -- 172.21.15.62:0/2210428649 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a8c054750 con 0x7f5aa015c830 2024-03-23T11:29:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.143+0000 7f5aa5c00640 1 -- 172.21.15.62:0/2210428649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5aa00c69b0 con 0x7f5aa015c830 2024-03-23T11:29:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.144+0000 7f5a97e00640 1 -- 172.21.15.62:0/2210428649 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16051 (secure 0 0 0) 0x7f5a8c057d70 con 0x7f5aa015c830 2024-03-23T11:29:20.136 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:20.136 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":105,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:29:16.631279+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:29:13.817605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-23T11:29:14.652617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:29:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 -- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a8006a7f0 msgr2=0x7f5a8006ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a8006a7f0 0x7f5a8006ccb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5a90007fd0 tx=0x7f5a90009040 comp rx=0 tx=0).stop 2024-03-23T11:29:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 -- 172.21.15.62:0/2210428649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa015c830 msgr2=0x7f5aa00c54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa015c830 0x7f5aa00c54d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c00d800 tx=0x7f5a8c00dcc0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 -- 172.21.15.62:0/2210428649 shutdown_connections 2024-03-23T11:29:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5a8006a7f0 0x7f5a8006ccb0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa015c830 0x7f5aa00c54d0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5aa0160310 0x7f5aa00ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 --2- 172.21.15.62:0/2210428649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5aa00ce250 0x7f5aa00b0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 -- 172.21.15.62:0/2210428649 >> 172.21.15.62:0/2210428649 conn(0x7f5aa0066050 msgr2=0x7f5aa015a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 -- 172.21.15.62:0/2210428649 shutdown_connections 2024-03-23T11:29:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.147+0000 7f5aa5c00640 1 -- 172.21.15.62:0/2210428649 wait complete. 2024-03-23T11:29:20.158 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T11:29:20.158 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:29:20.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.395+0000 7f271dc00640 1 Processor -- start 2024-03-23T11:29:20.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.395+0000 7f271dc00640 1 -- start start 2024-03-23T11:29:20.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.397+0000 7f271dc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f271806dca0 0x7f27180b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.397+0000 7f271dc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f271806c300 0x7f27180cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.397+0000 7f271dc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27180cad20 con 0x7f27180cd7c0 2024-03-23T11:29:20.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.397+0000 7f271dc00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f271806e130 con 0x7f271806c300 2024-03-23T11:29:20.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.397+0000 7f271dc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f271815a8a0 con 0x7f271806dca0 2024-03-23T11:29:20.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.397+0000 7f2717400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f271806c300 0x7f27180cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:20.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.397+0000 7f2717400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f271806c300 0x7f27180cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54430/0 (socket says 172.21.15.62:54430) 2024-03-23T11:29:20.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.397+0000 7f2717400640 1 -- 172.21.15.62:0/1769872288 learned_addr learned my addr 172.21.15.62:0/1769872288 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:20.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.397+0000 7f2716000640 1 --2- 172.21.15.62:0/1769872288 >> v2:172.21.15.120:3300/0 conn(0x7f271806dca0 0x7f27180b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:20.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.398+0000 7f2717400640 1 -- 172.21.15.62:0/1769872288 >> v2:172.21.15.120:3300/0 conn(0x7f271806dca0 msgr2=0x7f27180b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:20.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.398+0000 7f2717400640 1 --2- 172.21.15.62:0/1769872288 >> v2:172.21.15.120:3300/0 conn(0x7f271806dca0 0x7f27180b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.398+0000 7f2717400640 1 -- 172.21.15.62:0/1769872288 >> v1:172.21.15.120:6789/0 conn(0x7f27180cd7c0 legacy=0x7f27180cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.398+0000 7f2717400640 1 -- 172.21.15.62:0/1769872288 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271815ab80 con 0x7f271806c300 2024-03-23T11:29:20.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.399+0000 7f2717400640 1 --2- 172.21.15.62:0/1769872288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f271806c300 0x7f27180cb490 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f2718074040 tx=0x7f271002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d3122dd15229ee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:20.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.399+0000 7f2715600640 1 -- 172.21.15.62:0/1769872288 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271002ec90 con 0x7f271806c300 2024-03-23T11:29:20.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.399+0000 7f2715600640 1 -- 172.21.15.62:0/1769872288 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271002edf0 con 0x7f271806c300 2024-03-23T11:29:20.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.399+0000 7f2715600640 1 -- 172.21.15.62:0/1769872288 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710036610 con 0x7f271806c300 2024-03-23T11:29:20.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.400+0000 7f271dc00640 1 -- 172.21.15.62:0/1769872288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f271806c300 msgr2=0x7f27180cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.400+0000 7f271dc00640 1 --2- 172.21.15.62:0/1769872288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f271806c300 0x7f27180cb490 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f2718074040 tx=0x7f271002e0f0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.400+0000 7f271dc00640 1 -- 172.21.15.62:0/1769872288 shutdown_connections 2024-03-23T11:29:20.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.400+0000 7f271dc00640 1 --2- 172.21.15.62:0/1769872288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f271806c300 0x7f27180cb490 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.400+0000 7f271dc00640 1 --2- 172.21.15.62:0/1769872288 >> v2:172.21.15.120:3300/0 conn(0x7f271806dca0 0x7f27180b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.400+0000 7f271dc00640 1 -- 172.21.15.62:0/1769872288 >> 172.21.15.62:0/1769872288 conn(0x7f27180b02e0 msgr2=0x7f27180b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:20.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.400+0000 7f271dc00640 1 -- 172.21.15.62:0/1769872288 shutdown_connections 2024-03-23T11:29:20.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.400+0000 7f271dc00640 1 -- 172.21.15.62:0/1769872288 wait complete. 2024-03-23T11:29:20.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.402+0000 7f271dc00640 1 Processor -- start 2024-03-23T11:29:20.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.402+0000 7f271dc00640 1 -- start start 2024-03-23T11:29:20.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f271dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271806c300 0x7f27180ae720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f271dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27180cd7c0 0x7f27180ac650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f271dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27180ab840 0x7f27180af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f271dc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27181518c0 con 0x7f27180cd7c0 2024-03-23T11:29:20.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f271dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27180aec60 con 0x7f27180ab840 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f271dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2718151020 con 0x7f271806c300 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2716000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27180cd7c0 0x7f27180ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2716000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27180cd7c0 0x7f27180ac650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54432/0 (socket says 172.21.15.62:54432) 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2716000640 1 -- 172.21.15.62:0/1326828662 learned_addr learned my addr 172.21.15.62:0/1326828662 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2717400640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27180ab840 0x7f27180af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2716000640 1 -- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271806c300 msgr2=0x7f27180ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2716a00640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271806c300 0x7f27180ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2716000640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271806c300 0x7f27180ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2716000640 1 -- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27180ab840 msgr2=0x7f27180af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2716000640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27180ab840 0x7f27180af0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2716000640 1 -- 172.21.15.62:0/1326828662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2710009b30 con 0x7f27180cd7c0 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2716a00640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271806c300 0x7f27180ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.403+0000 7f2717400640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27180ab840 0x7f27180af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.404+0000 7f2716000640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27180cd7c0 0x7f27180ac650 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f270400d800 tx=0x7f270400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.404+0000 7f270ea00640 1 -- 172.21.15.62:0/1326828662 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270400b6c0 con 0x7f27180cd7c0 2024-03-23T11:29:20.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.404+0000 7f270ea00640 1 -- 172.21.15.62:0/1326828662 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270400b820 con 0x7f27180cd7c0 2024-03-23T11:29:20.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.404+0000 7f270ea00640 1 -- 172.21.15.62:0/1326828662 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270400bbb0 con 0x7f27180cd7c0 2024-03-23T11:29:20.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.404+0000 7f271dc00640 1 -- 172.21.15.62:0/1326828662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2718151300 con 0x7f27180cd7c0 2024-03-23T11:29:20.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.404+0000 7f271dc00640 1 -- 172.21.15.62:0/1326828662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27180acb90 con 0x7f27180cd7c0 2024-03-23T11:29:20.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.406+0000 7f271dc00640 1 -- 172.21.15.62:0/1326828662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27180acd50 con 0x7f27180cd7c0 2024-03-23T11:29:20.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.406+0000 7f270ea00640 1 -- 172.21.15.62:0/1326828662 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f270400bd10 con 0x7f27180cd7c0 2024-03-23T11:29:20.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.407+0000 7f270ea00640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26f806a7a0 0x7f26f806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.407+0000 7f270ea00640 1 -- 172.21.15.62:0/1326828662 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27040892b0 con 0x7f27180cd7c0 2024-03-23T11:29:20.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.408+0000 7f2716a00640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26f806a7a0 0x7f26f806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:20.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.409+0000 7f2716a00640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26f806a7a0 0x7f26f806cc60 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2708000f60 tx=0x7f2708009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:20.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.412+0000 7f270ea00640 1 -- 172.21.15.62:0/1326828662 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2704054760 con 0x7f27180cd7c0 2024-03-23T11:29:20.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.608+0000 7f271dc00640 1 -- 172.21.15.62:0/1326828662 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27180b1060 con 0x7f26f806a7a0 2024-03-23T11:29:20.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.634+0000 7f270ea00640 1 -- 172.21.15.62:0/1326828662 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287295 (secure 0 0 0) 0x7f2708001c90 con 0x7f26f806a7a0 2024-03-23T11:29:20.630 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:20.631 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:29:20.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 -- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26f806a7a0 msgr2=0x7f26f806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26f806a7a0 0x7f26f806cc60 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2708000f60 tx=0x7f2708009040 comp rx=0 tx=0).stop 2024-03-23T11:29:20.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 -- 172.21.15.62:0/1326828662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27180cd7c0 msgr2=0x7f27180ac650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27180cd7c0 0x7f27180ac650 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f270400d800 tx=0x7f270400dcc0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 -- 172.21.15.62:0/1326828662 shutdown_connections 2024-03-23T11:29:20.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f26f806a7a0 0x7f26f806cc60 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27180cd7c0 0x7f27180ac650 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27180ab840 0x7f27180af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 --2- 172.21.15.62:0/1326828662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f271806c300 0x7f27180ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 -- 172.21.15.62:0/1326828662 >> 172.21.15.62:0/1326828662 conn(0x7f27180b02e0 msgr2=0x7f271815aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:20.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 -- 172.21.15.62:0/1326828662 shutdown_connections 2024-03-23T11:29:20.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.640+0000 7f271dc00640 1 -- 172.21.15.62:0/1326828662 wait complete. 2024-03-23T11:29:20.651 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-03-23T11:29:19.463436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":159,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260292,"kb_used_data":11324,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754714428,"statfs":{"total":773094113280,"available":772827574272,"internally_reserved":0,"allocated":11595776,"data_stored":5657125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.342932"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649246+0000","last_change":"2024-03-23T11:29:15.669730+0000","last_active":"2024-03-23T11:29:16.649246+0000","last_peered":"2024-03-23T11:29:16.649246+0000","last_clean":"2024-03-23T11:29:16.649246+0000","last_became_active":"2024-03-23T11:29:15.669463+0000","last_became_peered":"2024-03-23T11:29:15.669463+0000","last_unstale":"2024-03-23T11:29:16.649246+0000","last_undegraded":"2024-03-23T11:29:16.649246+0000","last_fullsized":"2024-03-23T11:29:16.649246+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:21:46.428250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077624+0000","last_change":"2024-03-23T11:29:15.673453+0000","last_active":"2024-03-23T11:29:17.077624+0000","last_peered":"2024-03-23T11:29:17.077624+0000","last_clean":"2024-03-23T11:29:17.077624+0000","last_became_active":"2024-03-23T11:29:15.672342+0000","last_became_peered":"2024-03-23T11:29:15.672342+0000","last_unstale":"2024-03-23T11:29:17.077624+0000","last_undegraded":"2024-03-23T11:29:17.077624+0000","last_fullsized":"2024-03-23T11:29:17.077624+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:15:37.630453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649271+0000","last_change":"2024-03-23T11:29:15.658308+0000","last_active":"2024-03-23T11:29:16.649271+0000","last_peered":"2024-03-23T11:29:16.649271+0000","last_clean":"2024-03-23T11:29:16.649271+0000","last_became_active":"2024-03-23T11:29:15.658030+0000","last_became_peered":"2024-03-23T11:29:15.658030+0000","last_unstale":"2024-03-23T11:29:16.649271+0000","last_undegraded":"2024-03-23T11:29:16.649271+0000","last_fullsized":"2024-03-23T11:29:16.649271+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:07:07.568405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079069+0000","last_change":"2024-03-23T11:29:15.651333+0000","last_active":"2024-03-23T11:29:17.079069+0000","last_peered":"2024-03-23T11:29:17.079069+0000","last_clean":"2024-03-23T11:29:17.079069+0000","last_became_active":"2024-03-23T11:29:15.651076+0000","last_became_peered":"2024-03-23T11:29:15.651076+0000","last_unstale":"2024-03-23T11:29:17.079069+0000","last_undegraded":"2024-03-23T11:29:17.079069+0000","last_fullsized":"2024-03-23T11:29:17.079069+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.902089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929345+0000","last_change":"2024-03-23T11:29:15.665566+0000","last_active":"2024-03-23T11:29:16.929345+0000","last_peered":"2024-03-23T11:29:16.929345+0000","last_clean":"2024-03-23T11:29:16.929345+0000","last_became_active":"2024-03-23T11:29:15.664851+0000","last_became_peered":"2024-03-23T11:29:15.664851+0000","last_unstale":"2024-03-23T11:29:16.929345+0000","last_undegraded":"2024-03-23T11:29:16.929345+0000","last_fullsized":"2024-03-23T11:29:16.929345+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:27.274279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649914+0000","last_change":"2024-03-23T11:29:15.663227+0000","last_active":"2024-03-23T11:29:16.649914+0000","last_peered":"2024-03-23T11:29:16.649914+0000","last_clean":"2024-03-23T11:29:16.649914+0000","last_became_active":"2024-03-23T11:29:15.661436+0000","last_became_peered":"2024-03-23T11:29:15.661436+0000","last_unstale":"2024-03-23T11:29:16.649914+0000","last_undegraded":"2024-03-23T11:29:16.649914+0000","last_fullsized":"2024-03-23T11:29:16.649914+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:20:09.497186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649956+0000","last_change":"2024-03-23T11:29:15.662335+0000","last_active":"2024-03-23T11:29:16.649956+0000","last_peered":"2024-03-23T11:29:16.649956+0000","last_clean":"2024-03-23T11:29:16.649956+0000","last_became_active":"2024-03-23T11:29:15.662022+0000","last_became_peered":"2024-03-23T11:29:15.662022+0000","last_unstale":"2024-03-23T11:29:16.649956+0000","last_undegraded":"2024-03-23T11:29:16.649956+0000","last_fullsized":"2024-03-23T11:29:16.649956+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:01:37.986646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077905+0000","last_change":"2024-03-23T11:29:15.669108+0000","last_active":"2024-03-23T11:29:17.077905+0000","last_peered":"2024-03-23T11:29:17.077905+0000","last_clean":"2024-03-23T11:29:17.077905+0000","last_became_active":"2024-03-23T11:29:15.668763+0000","last_became_peered":"2024-03-23T11:29:15.668763+0000","last_unstale":"2024-03-23T11:29:17.077905+0000","last_undegraded":"2024-03-23T11:29:17.077905+0000","last_fullsized":"2024-03-23T11:29:17.077905+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:25.131893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052620+0000","last_change":"2024-03-23T11:29:15.668474+0000","last_active":"2024-03-23T11:29:17.052620+0000","last_peered":"2024-03-23T11:29:17.052620+0000","last_clean":"2024-03-23T11:29:17.052620+0000","last_became_active":"2024-03-23T11:29:15.668114+0000","last_became_peered":"2024-03-23T11:29:15.668114+0000","last_unstale":"2024-03-23T11:29:17.052620+0000","last_undegraded":"2024-03-23T11:29:17.052620+0000","last_fullsized":"2024-03-23T11:29:17.052620+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:42:00.302069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929241+0000","last_change":"2024-03-23T11:29:15.666201+0000","last_active":"2024-03-23T11:29:16.929241+0000","last_peered":"2024-03-23T11:29:16.929241+0000","last_clean":"2024-03-23T11:29:16.929241+0000","last_became_active":"2024-03-23T11:29:15.665939+0000","last_became_peered":"2024-03-23T11:29:15.665939+0000","last_unstale":"2024-03-23T11:29:16.929241+0000","last_undegraded":"2024-03-23T11:29:16.929241+0000","last_fullsized":"2024-03-23T11:29:16.929241+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:03.126355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649783+0000","last_change":"2024-03-23T11:29:15.666875+0000","last_active":"2024-03-23T11:29:16.649783+0000","last_peered":"2024-03-23T11:29:16.649783+0000","last_clean":"2024-03-23T11:29:16.649783+0000","last_became_active":"2024-03-23T11:29:15.661755+0000","last_became_peered":"2024-03-23T11:29:15.661755+0000","last_unstale":"2024-03-23T11:29:16.649783+0000","last_undegraded":"2024-03-23T11:29:16.649783+0000","last_fullsized":"2024-03-23T11:29:16.649783+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:34:53.607989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052654+0000","last_change":"2024-03-23T11:29:15.660658+0000","last_active":"2024-03-23T11:29:17.052654+0000","last_peered":"2024-03-23T11:29:17.052654+0000","last_clean":"2024-03-23T11:29:17.052654+0000","last_became_active":"2024-03-23T11:29:15.658934+0000","last_became_peered":"2024-03-23T11:29:15.658934+0000","last_unstale":"2024-03-23T11:29:17.052654+0000","last_undegraded":"2024-03-23T11:29:17.052654+0000","last_fullsized":"2024-03-23T11:29:17.052654+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:21:17.856878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078618+0000","last_change":"2024-03-23T11:29:15.662724+0000","last_active":"2024-03-23T11:29:17.078618+0000","last_peered":"2024-03-23T11:29:17.078618+0000","last_clean":"2024-03-23T11:29:17.078618+0000","last_became_active":"2024-03-23T11:29:15.662420+0000","last_became_peered":"2024-03-23T11:29:15.662420+0000","last_unstale":"2024-03-23T11:29:17.078618+0000","last_undegraded":"2024-03-23T11:29:17.078618+0000","last_fullsized":"2024-03-23T11:29:17.078618+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:27.274279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076113+0000","last_change":"2024-03-23T11:29:15.664201+0000","last_active":"2024-03-23T11:29:17.076113+0000","last_peered":"2024-03-23T11:29:17.076113+0000","last_clean":"2024-03-23T11:29:17.076113+0000","last_became_active":"2024-03-23T11:29:15.663923+0000","last_became_peered":"2024-03-23T11:29:15.663923+0000","last_unstale":"2024-03-23T11:29:17.076113+0000","last_undegraded":"2024-03-23T11:29:17.076113+0000","last_fullsized":"2024-03-23T11:29:17.076113+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:46:09.117552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077065+0000","last_change":"2024-03-23T11:29:15.653031+0000","last_active":"2024-03-23T11:29:17.077065+0000","last_peered":"2024-03-23T11:29:17.077065+0000","last_clean":"2024-03-23T11:29:17.077065+0000","last_became_active":"2024-03-23T11:29:15.652677+0000","last_became_peered":"2024-03-23T11:29:15.652677+0000","last_unstale":"2024-03-23T11:29:17.077065+0000","last_undegraded":"2024-03-23T11:29:17.077065+0000","last_fullsized":"2024-03-23T11:29:17.077065+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:20:38.886782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649060+0000","last_change":"2024-03-23T11:29:15.658243+0000","last_active":"2024-03-23T11:29:16.649060+0000","last_peered":"2024-03-23T11:29:16.649060+0000","last_clean":"2024-03-23T11:29:16.649060+0000","last_became_active":"2024-03-23T11:29:15.657964+0000","last_became_peered":"2024-03-23T11:29:15.657964+0000","last_unstale":"2024-03-23T11:29:16.649060+0000","last_undegraded":"2024-03-23T11:29:16.649060+0000","last_fullsized":"2024-03-23T11:29:16.649060+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:40.286477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.082158+0000","last_change":"2024-03-23T11:29:15.666060+0000","last_active":"2024-03-23T11:29:17.082158+0000","last_peered":"2024-03-23T11:29:17.082158+0000","last_clean":"2024-03-23T11:29:17.082158+0000","last_became_active":"2024-03-23T11:29:15.665037+0000","last_became_peered":"2024-03-23T11:29:15.665037+0000","last_unstale":"2024-03-23T11:29:17.082158+0000","last_undegraded":"2024-03-23T11:29:17.082158+0000","last_fullsized":"2024-03-23T11:29:17.082158+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:23:08.931665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078923+0000","last_change":"2024-03-23T11:29:15.651649+0000","last_active":"2024-03-23T11:29:17.078923+0000","last_peered":"2024-03-23T11:29:17.078923+0000","last_clean":"2024-03-23T11:29:17.078923+0000","last_became_active":"2024-03-23T11:29:15.651296+0000","last_became_peered":"2024-03-23T11:29:15.651296+0000","last_unstale":"2024-03-23T11:29:17.078923+0000","last_undegraded":"2024-03-23T11:29:17.078923+0000","last_fullsized":"2024-03-23T11:29:17.078923+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:17.074421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079382+0000","last_change":"2024-03-23T11:29:15.652434+0000","last_active":"2024-03-23T11:29:17.079382+0000","last_peered":"2024-03-23T11:29:17.079382+0000","last_clean":"2024-03-23T11:29:17.079382+0000","last_became_active":"2024-03-23T11:29:15.652172+0000","last_became_peered":"2024-03-23T11:29:15.652172+0000","last_unstale":"2024-03-23T11:29:17.079382+0000","last_undegraded":"2024-03-23T11:29:17.079382+0000","last_fullsized":"2024-03-23T11:29:17.079382+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:19.474314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052560+0000","last_change":"2024-03-23T11:29:15.671106+0000","last_active":"2024-03-23T11:29:17.052560+0000","last_peered":"2024-03-23T11:29:17.052560+0000","last_clean":"2024-03-23T11:29:17.052560+0000","last_became_active":"2024-03-23T11:29:15.670839+0000","last_became_peered":"2024-03-23T11:29:15.670839+0000","last_unstale":"2024-03-23T11:29:17.052560+0000","last_undegraded":"2024-03-23T11:29:17.052560+0000","last_fullsized":"2024-03-23T11:29:17.052560+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:56:56.282431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076764+0000","last_change":"2024-03-23T11:29:15.659203+0000","last_active":"2024-03-23T11:29:17.076764+0000","last_peered":"2024-03-23T11:29:17.076764+0000","last_clean":"2024-03-23T11:29:17.076764+0000","last_became_active":"2024-03-23T11:29:15.658693+0000","last_became_peered":"2024-03-23T11:29:15.658693+0000","last_unstale":"2024-03-23T11:29:17.076764+0000","last_undegraded":"2024-03-23T11:29:17.076764+0000","last_fullsized":"2024-03-23T11:29:17.076764+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:32:17.491368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077429+0000","last_change":"2024-03-23T11:29:15.661680+0000","last_active":"2024-03-23T11:29:17.077429+0000","last_peered":"2024-03-23T11:29:17.077429+0000","last_clean":"2024-03-23T11:29:17.077429+0000","last_became_active":"2024-03-23T11:29:15.661421+0000","last_became_peered":"2024-03-23T11:29:15.661421+0000","last_unstale":"2024-03-23T11:29:17.077429+0000","last_undegraded":"2024-03-23T11:29:17.077429+0000","last_fullsized":"2024-03-23T11:29:17.077429+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:13:25.413831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079748+0000","last_change":"2024-03-23T11:29:15.673377+0000","last_active":"2024-03-23T11:29:17.079748+0000","last_peered":"2024-03-23T11:29:17.079748+0000","last_clean":"2024-03-23T11:29:17.079748+0000","last_became_active":"2024-03-23T11:29:15.673077+0000","last_became_peered":"2024-03-23T11:29:15.673077+0000","last_unstale":"2024-03-23T11:29:17.079748+0000","last_undegraded":"2024-03-23T11:29:17.079748+0000","last_fullsized":"2024-03-23T11:29:17.079748+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:55:04.251123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649131+0000","last_change":"2024-03-23T11:29:15.652916+0000","last_active":"2024-03-23T11:29:16.649131+0000","last_peered":"2024-03-23T11:29:16.649131+0000","last_clean":"2024-03-23T11:29:16.649131+0000","last_became_active":"2024-03-23T11:29:15.652566+0000","last_became_peered":"2024-03-23T11:29:15.652566+0000","last_unstale":"2024-03-23T11:29:16.649131+0000","last_undegraded":"2024-03-23T11:29:16.649131+0000","last_fullsized":"2024-03-23T11:29:16.649131+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:22:05.889747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929778+0000","last_change":"2024-03-23T11:29:15.673239+0000","last_active":"2024-03-23T11:29:16.929778+0000","last_peered":"2024-03-23T11:29:16.929778+0000","last_clean":"2024-03-23T11:29:16.929778+0000","last_became_active":"2024-03-23T11:29:15.671796+0000","last_became_peered":"2024-03-23T11:29:15.671796+0000","last_unstale":"2024-03-23T11:29:16.929778+0000","last_undegraded":"2024-03-23T11:29:16.929778+0000","last_fullsized":"2024-03-23T11:29:16.929778+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:17.074421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077900+0000","last_change":"2024-03-23T11:29:15.664158+0000","last_active":"2024-03-23T11:29:17.077900+0000","last_peered":"2024-03-23T11:29:17.077900+0000","last_clean":"2024-03-23T11:29:17.077900+0000","last_became_active":"2024-03-23T11:29:15.662983+0000","last_became_peered":"2024-03-23T11:29:15.662983+0000","last_unstale":"2024-03-23T11:29:17.077900+0000","last_undegraded":"2024-03-23T11:29:17.077900+0000","last_fullsized":"2024-03-23T11:29:17.077900+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:03:38.240579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649710+0000","last_change":"2024-03-23T11:29:15.658623+0000","last_active":"2024-03-23T11:29:16.649710+0000","last_peered":"2024-03-23T11:29:16.649710+0000","last_clean":"2024-03-23T11:29:16.649710+0000","last_became_active":"2024-03-23T11:29:15.656411+0000","last_became_peered":"2024-03-23T11:29:15.656411+0000","last_unstale":"2024-03-23T11:29:16.649710+0000","last_undegraded":"2024-03-23T11:29:16.649710+0000","last_fullsized":"2024-03-23T11:29:16.649710+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:50:49.301705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929338+0000","last_change":"2024-03-23T11:29:15.670061+0000","last_active":"2024-03-23T11:29:16.929338+0000","last_peered":"2024-03-23T11:29:16.929338+0000","last_clean":"2024-03-23T11:29:16.929338+0000","last_became_active":"2024-03-23T11:29:15.669663+0000","last_became_peered":"2024-03-23T11:29:15.669663+0000","last_unstale":"2024-03-23T11:29:16.929338+0000","last_undegraded":"2024-03-23T11:29:16.929338+0000","last_fullsized":"2024-03-23T11:29:16.929338+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:50:21.272634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079978+0000","last_change":"2024-03-23T11:29:15.666505+0000","last_active":"2024-03-23T11:29:17.079978+0000","last_peered":"2024-03-23T11:29:17.079978+0000","last_clean":"2024-03-23T11:29:17.079978+0000","last_became_active":"2024-03-23T11:29:15.666264+0000","last_became_peered":"2024-03-23T11:29:15.666264+0000","last_unstale":"2024-03-23T11:29:17.079978+0000","last_undegraded":"2024-03-23T11:29:17.079978+0000","last_fullsized":"2024-03-23T11:29:17.079978+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:13:07.960167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076589+0000","last_change":"2024-03-23T11:29:15.651998+0000","last_active":"2024-03-23T11:29:17.076589+0000","last_peered":"2024-03-23T11:29:17.076589+0000","last_clean":"2024-03-23T11:29:17.076589+0000","last_became_active":"2024-03-23T11:29:15.650650+0000","last_became_peered":"2024-03-23T11:29:15.650650+0000","last_unstale":"2024-03-23T11:29:17.076589+0000","last_undegraded":"2024-03-23T11:29:17.076589+0000","last_fullsized":"2024-03-23T11:29:17.076589+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:59:16.956887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077760+0000","last_change":"2024-03-23T11:29:15.668910+0000","last_active":"2024-03-23T11:29:17.077760+0000","last_peered":"2024-03-23T11:29:17.077760+0000","last_clean":"2024-03-23T11:29:17.077760+0000","last_became_active":"2024-03-23T11:29:15.668357+0000","last_became_peered":"2024-03-23T11:29:15.668357+0000","last_unstale":"2024-03-23T11:29:17.077760+0000","last_undegraded":"2024-03-23T11:29:17.077760+0000","last_fullsized":"2024-03-23T11:29:17.077760+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:22:40.156625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649018+0000","last_change":"2024-03-23T11:29:15.657660+0000","last_active":"2024-03-23T11:29:16.649018+0000","last_peered":"2024-03-23T11:29:16.649018+0000","last_clean":"2024-03-23T11:29:16.649018+0000","last_became_active":"2024-03-23T11:29:15.656966+0000","last_became_peered":"2024-03-23T11:29:15.656966+0000","last_unstale":"2024-03-23T11:29:16.649018+0000","last_undegraded":"2024-03-23T11:29:16.649018+0000","last_fullsized":"2024-03-23T11:29:16.649018+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:56.613315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079556+0000","last_change":"2024-03-23T11:29:15.654194+0000","last_active":"2024-03-23T11:29:17.079556+0000","last_peered":"2024-03-23T11:29:17.079556+0000","last_clean":"2024-03-23T11:29:17.079556+0000","last_became_active":"2024-03-23T11:29:15.653914+0000","last_became_peered":"2024-03-23T11:29:15.653914+0000","last_unstale":"2024-03-23T11:29:17.079556+0000","last_undegraded":"2024-03-23T11:29:17.079556+0000","last_fullsized":"2024-03-23T11:29:17.079556+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:17:39.719198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"104'4","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.082180+0000","last_change":"2024-03-23T11:29:15.167697+0000","last_active":"2024-03-23T11:29:17.082180+0000","last_peered":"2024-03-23T11:29:17.082180+0000","last_clean":"2024-03-23T11:29:17.082180+0000","last_became_active":"2024-03-23T11:29:15.167198+0000","last_became_peered":"2024-03-23T11:29:15.167198+0000","last_unstale":"2024-03-23T11:29:17.082180+0000","last_undegraded":"2024-03-23T11:29:17.082180+0000","last_fullsized":"2024-03-23T11:29:17.082180+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:24:59.167502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.080175+0000","last_change":"2024-03-23T11:29:15.668009+0000","last_active":"2024-03-23T11:29:17.080175+0000","last_peered":"2024-03-23T11:29:17.080175+0000","last_clean":"2024-03-23T11:29:17.080175+0000","last_became_active":"2024-03-23T11:29:15.667751+0000","last_became_peered":"2024-03-23T11:29:15.667751+0000","last_unstale":"2024-03-23T11:29:17.080175+0000","last_undegraded":"2024-03-23T11:29:17.080175+0000","last_fullsized":"2024-03-23T11:29:17.080175+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:50:21.272634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"104'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.650334+0000","last_change":"2024-03-23T11:29:15.163279+0000","last_active":"2024-03-23T11:29:16.650334+0000","last_peered":"2024-03-23T11:29:16.650334+0000","last_clean":"2024-03-23T11:29:16.650334+0000","last_became_active":"2024-03-23T11:29:15.162911+0000","last_became_peered":"2024-03-23T11:29:15.162911+0000","last_unstale":"2024-03-23T11:29:16.650334+0000","last_undegraded":"2024-03-23T11:29:16.650334+0000","last_fullsized":"2024-03-23T11:29:16.650334+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:51:23.085524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.648986+0000","last_change":"2024-03-23T11:29:15.655253+0000","last_active":"2024-03-23T11:29:16.648986+0000","last_peered":"2024-03-23T11:29:16.648986+0000","last_clean":"2024-03-23T11:29:16.648986+0000","last_became_active":"2024-03-23T11:29:15.654942+0000","last_became_peered":"2024-03-23T11:29:15.654942+0000","last_unstale":"2024-03-23T11:29:16.648986+0000","last_undegraded":"2024-03-23T11:29:16.648986+0000","last_fullsized":"2024-03-23T11:29:16.648986+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:13:19.425483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.081531+0000","last_change":"2024-03-23T11:29:15.168589+0000","last_active":"2024-03-23T11:29:17.081531+0000","last_peered":"2024-03-23T11:29:17.081531+0000","last_clean":"2024-03-23T11:29:17.081531+0000","last_became_active":"2024-03-23T11:29:15.166723+0000","last_became_peered":"2024-03-23T11:29:15.166723+0000","last_unstale":"2024-03-23T11:29:17.081531+0000","last_undegraded":"2024-03-23T11:29:17.081531+0000","last_fullsized":"2024-03-23T11:29:17.081531+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:46.791548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052889+0000","last_change":"2024-03-23T11:29:15.659552+0000","last_active":"2024-03-23T11:29:17.052889+0000","last_peered":"2024-03-23T11:29:17.052889+0000","last_clean":"2024-03-23T11:29:17.052889+0000","last_became_active":"2024-03-23T11:29:15.659196+0000","last_became_peered":"2024-03-23T11:29:15.659196+0000","last_unstale":"2024-03-23T11:29:17.052889+0000","last_undegraded":"2024-03-23T11:29:17.052889+0000","last_fullsized":"2024-03-23T11:29:17.052889+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:03.126355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.080571+0000","last_change":"2024-03-23T11:29:15.167939+0000","last_active":"2024-03-23T11:29:17.080571+0000","last_peered":"2024-03-23T11:29:17.080571+0000","last_clean":"2024-03-23T11:29:17.080571+0000","last_became_active":"2024-03-23T11:29:15.165542+0000","last_became_peered":"2024-03-23T11:29:15.165542+0000","last_unstale":"2024-03-23T11:29:17.080571+0000","last_undegraded":"2024-03-23T11:29:17.080571+0000","last_fullsized":"2024-03-23T11:29:17.080571+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:27:48.629258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.648411+0000","last_change":"2024-03-23T11:29:15.667879+0000","last_active":"2024-03-23T11:29:16.648411+0000","last_peered":"2024-03-23T11:29:16.648411+0000","last_clean":"2024-03-23T11:29:16.648411+0000","last_became_active":"2024-03-23T11:29:15.666726+0000","last_became_peered":"2024-03-23T11:29:15.666726+0000","last_unstale":"2024-03-23T11:29:16.648411+0000","last_undegraded":"2024-03-23T11:29:16.648411+0000","last_fullsized":"2024-03-23T11:29:16.648411+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:03.126355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"104'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649014+0000","last_change":"2024-03-23T11:29:15.170048+0000","last_active":"2024-03-23T11:29:16.649014+0000","last_peered":"2024-03-23T11:29:16.649014+0000","last_clean":"2024-03-23T11:29:16.649014+0000","last_became_active":"2024-03-23T11:29:15.169324+0000","last_became_peered":"2024-03-23T11:29:15.169324+0000","last_unstale":"2024-03-23T11:29:16.649014+0000","last_undegraded":"2024-03-23T11:29:16.649014+0000","last_fullsized":"2024-03-23T11:29:16.649014+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:19:12.246415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.650011+0000","last_change":"2024-03-23T11:29:15.664162+0000","last_active":"2024-03-23T11:29:16.650011+0000","last_peered":"2024-03-23T11:29:16.650011+0000","last_clean":"2024-03-23T11:29:16.650011+0000","last_became_active":"2024-03-23T11:29:15.663884+0000","last_became_peered":"2024-03-23T11:29:15.663884+0000","last_unstale":"2024-03-23T11:29:16.650011+0000","last_undegraded":"2024-03-23T11:29:16.650011+0000","last_fullsized":"2024-03-23T11:29:16.650011+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:55:14.250603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"104'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077570+0000","last_change":"2024-03-23T11:29:15.167254+0000","last_active":"2024-03-23T11:29:17.077570+0000","last_peered":"2024-03-23T11:29:17.077570+0000","last_clean":"2024-03-23T11:29:17.077570+0000","last_became_active":"2024-03-23T11:29:15.165937+0000","last_became_peered":"2024-03-23T11:29:15.165937+0000","last_unstale":"2024-03-23T11:29:17.077570+0000","last_undegraded":"2024-03-23T11:29:17.077570+0000","last_fullsized":"2024-03-23T11:29:17.077570+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:02.110388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077030+0000","last_change":"2024-03-23T11:29:15.652851+0000","last_active":"2024-03-23T11:29:17.077030+0000","last_peered":"2024-03-23T11:29:17.077030+0000","last_clean":"2024-03-23T11:29:17.077030+0000","last_became_active":"2024-03-23T11:29:15.652536+0000","last_became_peered":"2024-03-23T11:29:15.652536+0000","last_unstale":"2024-03-23T11:29:17.077030+0000","last_undegraded":"2024-03-23T11:29:17.077030+0000","last_fullsized":"2024-03-23T11:29:17.077030+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:44.511761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929189+0000","last_change":"2024-03-23T11:29:15.171133+0000","last_active":"2024-03-23T11:29:16.929189+0000","last_peered":"2024-03-23T11:29:16.929189+0000","last_clean":"2024-03-23T11:29:16.929189+0000","last_became_active":"2024-03-23T11:29:15.170830+0000","last_became_peered":"2024-03-23T11:29:15.170830+0000","last_unstale":"2024-03-23T11:29:16.929189+0000","last_undegraded":"2024-03-23T11:29:16.929189+0000","last_fullsized":"2024-03-23T11:29:16.929189+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:36:01.499746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.051906+0000","last_change":"2024-03-23T11:29:15.660248+0000","last_active":"2024-03-23T11:29:17.051906+0000","last_peered":"2024-03-23T11:29:17.051906+0000","last_clean":"2024-03-23T11:29:17.051906+0000","last_became_active":"2024-03-23T11:29:15.659942+0000","last_became_peered":"2024-03-23T11:29:15.659942+0000","last_unstale":"2024-03-23T11:29:17.051906+0000","last_undegraded":"2024-03-23T11:29:17.051906+0000","last_fullsized":"2024-03-23T11:29:17.051906+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:22.855663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649166+0000","last_change":"2024-03-23T11:29:15.159736+0000","last_active":"2024-03-23T11:29:16.649166+0000","last_peered":"2024-03-23T11:29:16.649166+0000","last_clean":"2024-03-23T11:29:16.649166+0000","last_became_active":"2024-03-23T11:29:15.159171+0000","last_became_peered":"2024-03-23T11:29:15.159171+0000","last_unstale":"2024-03-23T11:29:16.649166+0000","last_undegraded":"2024-03-23T11:29:16.649166+0000","last_fullsized":"2024-03-23T11:29:16.649166+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:44:57.248852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078147+0000","last_change":"2024-03-23T11:29:15.662195+0000","last_active":"2024-03-23T11:29:17.078147+0000","last_peered":"2024-03-23T11:29:17.078147+0000","last_clean":"2024-03-23T11:29:17.078147+0000","last_became_active":"2024-03-23T11:29:15.661948+0000","last_became_peered":"2024-03-23T11:29:15.661948+0000","last_unstale":"2024-03-23T11:29:17.078147+0000","last_undegraded":"2024-03-23T11:29:17.078147+0000","last_fullsized":"2024-03-23T11:29:17.078147+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:22:25.026968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649307+0000","last_change":"2024-03-23T11:29:15.161141+0000","last_active":"2024-03-23T11:29:16.649307+0000","last_peered":"2024-03-23T11:29:16.649307+0000","last_clean":"2024-03-23T11:29:16.649307+0000","last_became_active":"2024-03-23T11:29:15.160884+0000","last_became_peered":"2024-03-23T11:29:15.160884+0000","last_unstale":"2024-03-23T11:29:16.649307+0000","last_undegraded":"2024-03-23T11:29:16.649307+0000","last_fullsized":"2024-03-23T11:29:16.649307+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:11.396958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052772+0000","last_change":"2024-03-23T11:29:15.673901+0000","last_active":"2024-03-23T11:29:17.052772+0000","last_peered":"2024-03-23T11:29:17.052772+0000","last_clean":"2024-03-23T11:29:17.052772+0000","last_became_active":"2024-03-23T11:29:15.673013+0000","last_became_peered":"2024-03-23T11:29:15.673013+0000","last_unstale":"2024-03-23T11:29:17.052772+0000","last_undegraded":"2024-03-23T11:29:17.052772+0000","last_fullsized":"2024-03-23T11:29:17.052772+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:00:56.194551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079165+0000","last_change":"2024-03-23T11:29:15.164764+0000","last_active":"2024-03-23T11:29:17.079165+0000","last_peered":"2024-03-23T11:29:17.079165+0000","last_clean":"2024-03-23T11:29:17.079165+0000","last_became_active":"2024-03-23T11:29:15.163778+0000","last_became_peered":"2024-03-23T11:29:15.163778+0000","last_unstale":"2024-03-23T11:29:17.079165+0000","last_undegraded":"2024-03-23T11:29:17.079165+0000","last_fullsized":"2024-03-23T11:29:17.079165+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:31:55.748276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929396+0000","last_change":"2024-03-23T11:29:15.671469+0000","last_active":"2024-03-23T11:29:16.929396+0000","last_peered":"2024-03-23T11:29:16.929396+0000","last_clean":"2024-03-23T11:29:16.929396+0000","last_became_active":"2024-03-23T11:29:15.669339+0000","last_became_peered":"2024-03-23T11:29:15.669339+0000","last_unstale":"2024-03-23T11:29:16.929396+0000","last_undegraded":"2024-03-23T11:29:16.929396+0000","last_fullsized":"2024-03-23T11:29:16.929396+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:22:40.156625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"104'3","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.650069+0000","last_change":"2024-03-23T11:29:15.161417+0000","last_active":"2024-03-23T11:29:16.650069+0000","last_peered":"2024-03-23T11:29:16.650069+0000","last_clean":"2024-03-23T11:29:16.650069+0000","last_became_active":"2024-03-23T11:29:15.161009+0000","last_became_peered":"2024-03-23T11:29:15.161009+0000","last_unstale":"2024-03-23T11:29:16.650069+0000","last_undegraded":"2024-03-23T11:29:16.650069+0000","last_fullsized":"2024-03-23T11:29:16.650069+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:15:46.288971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649866+0000","last_change":"2024-03-23T11:29:15.656957+0000","last_active":"2024-03-23T11:29:16.649866+0000","last_peered":"2024-03-23T11:29:16.649866+0000","last_clean":"2024-03-23T11:29:16.649866+0000","last_became_active":"2024-03-23T11:29:15.655820+0000","last_became_peered":"2024-03-23T11:29:15.655820+0000","last_unstale":"2024-03-23T11:29:16.649866+0000","last_undegraded":"2024-03-23T11:29:16.649866+0000","last_fullsized":"2024-03-23T11:29:16.649866+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:33:06.699830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"104'6","reported_seq":17,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052814+0000","last_change":"2024-03-23T11:29:15.162213+0000","last_active":"2024-03-23T11:29:17.052814+0000","last_peered":"2024-03-23T11:29:17.052814+0000","last_clean":"2024-03-23T11:29:17.052814+0000","last_became_active":"2024-03-23T11:29:15.161871+0000","last_became_peered":"2024-03-23T11:29:15.161871+0000","last_unstale":"2024-03-23T11:29:17.052814+0000","last_undegraded":"2024-03-23T11:29:17.052814+0000","last_fullsized":"2024-03-23T11:29:17.052814+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:40:27.480724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078197+0000","last_change":"2024-03-23T11:29:15.664516+0000","last_active":"2024-03-23T11:29:17.078197+0000","last_peered":"2024-03-23T11:29:17.078197+0000","last_clean":"2024-03-23T11:29:17.078197+0000","last_became_active":"2024-03-23T11:29:15.661188+0000","last_became_peered":"2024-03-23T11:29:15.661188+0000","last_unstale":"2024-03-23T11:29:17.078197+0000","last_undegraded":"2024-03-23T11:29:17.078197+0000","last_fullsized":"2024-03-23T11:29:17.078197+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:27:49.645225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078851+0000","last_change":"2024-03-23T11:29:15.162980+0000","last_active":"2024-03-23T11:29:17.078851+0000","last_peered":"2024-03-23T11:29:17.078851+0000","last_clean":"2024-03-23T11:29:17.078851+0000","last_became_active":"2024-03-23T11:29:15.161438+0000","last_became_peered":"2024-03-23T11:29:15.161438+0000","last_unstale":"2024-03-23T11:29:17.078851+0000","last_undegraded":"2024-03-23T11:29:17.078851+0000","last_fullsized":"2024-03-23T11:29:17.078851+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:57:53.683724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649432+0000","last_change":"2024-03-23T11:29:15.646405+0000","last_active":"2024-03-23T11:29:16.649432+0000","last_peered":"2024-03-23T11:29:16.649432+0000","last_clean":"2024-03-23T11:29:16.649432+0000","last_became_active":"2024-03-23T11:29:15.645120+0000","last_became_peered":"2024-03-23T11:29:15.645120+0000","last_unstale":"2024-03-23T11:29:16.649432+0000","last_undegraded":"2024-03-23T11:29:16.649432+0000","last_fullsized":"2024-03-23T11:29:16.649432+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:22:51.383135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077114+0000","last_change":"2024-03-23T11:29:15.165402+0000","last_active":"2024-03-23T11:29:17.077114+0000","last_peered":"2024-03-23T11:29:17.077114+0000","last_clean":"2024-03-23T11:29:17.077114+0000","last_became_active":"2024-03-23T11:29:15.165117+0000","last_became_peered":"2024-03-23T11:29:15.165117+0000","last_unstale":"2024-03-23T11:29:17.077114+0000","last_undegraded":"2024-03-23T11:29:17.077114+0000","last_fullsized":"2024-03-23T11:29:17.077114+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:08:22.178636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.051916+0000","last_change":"2024-03-23T11:29:15.659027+0000","last_active":"2024-03-23T11:29:17.051916+0000","last_peered":"2024-03-23T11:29:17.051916+0000","last_clean":"2024-03-23T11:29:17.051916+0000","last_became_active":"2024-03-23T11:29:15.658730+0000","last_became_peered":"2024-03-23T11:29:15.658730+0000","last_unstale":"2024-03-23T11:29:17.051916+0000","last_undegraded":"2024-03-23T11:29:17.051916+0000","last_fullsized":"2024-03-23T11:29:17.051916+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:25:00.183469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052089+0000","last_change":"2024-03-23T11:29:15.166013+0000","last_active":"2024-03-23T11:29:17.052089+0000","last_peered":"2024-03-23T11:29:17.052089+0000","last_clean":"2024-03-23T11:29:17.052089+0000","last_became_active":"2024-03-23T11:29:15.165701+0000","last_became_peered":"2024-03-23T11:29:15.165701+0000","last_unstale":"2024-03-23T11:29:17.052089+0000","last_undegraded":"2024-03-23T11:29:17.052089+0000","last_fullsized":"2024-03-23T11:29:17.052089+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:26.319833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076103+0000","last_change":"2024-03-23T11:29:15.663722+0000","last_active":"2024-03-23T11:29:17.076103+0000","last_peered":"2024-03-23T11:29:17.076103+0000","last_clean":"2024-03-23T11:29:17.076103+0000","last_became_active":"2024-03-23T11:29:15.663438+0000","last_became_peered":"2024-03-23T11:29:15.663438+0000","last_unstale":"2024-03-23T11:29:17.076103+0000","last_undegraded":"2024-03-23T11:29:17.076103+0000","last_fullsized":"2024-03-23T11:29:17.076103+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:53:36.693645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"104'8","reported_seq":19,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649292+0000","last_change":"2024-03-23T11:29:15.146975+0000","last_active":"2024-03-23T11:29:16.649292+0000","last_peered":"2024-03-23T11:29:16.649292+0000","last_clean":"2024-03-23T11:29:16.649292+0000","last_became_active":"2024-03-23T11:29:15.146669+0000","last_became_peered":"2024-03-23T11:29:15.146669+0000","last_unstale":"2024-03-23T11:29:16.649292+0000","last_undegraded":"2024-03-23T11:29:16.649292+0000","last_fullsized":"2024-03-23T11:29:16.649292+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:18:47.004845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929851+0000","last_change":"2024-03-23T11:29:15.668120+0000","last_active":"2024-03-23T11:29:16.929851+0000","last_peered":"2024-03-23T11:29:16.929851+0000","last_clean":"2024-03-23T11:29:16.929851+0000","last_became_active":"2024-03-23T11:29:15.667698+0000","last_became_peered":"2024-03-23T11:29:15.667698+0000","last_unstale":"2024-03-23T11:29:16.929851+0000","last_undegraded":"2024-03-23T11:29:16.929851+0000","last_fullsized":"2024-03-23T11:29:16.929851+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.902089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052562+0000","last_change":"2024-03-23T11:29:15.160035+0000","last_active":"2024-03-23T11:29:17.052562+0000","last_peered":"2024-03-23T11:29:17.052562+0000","last_clean":"2024-03-23T11:29:17.052562+0000","last_became_active":"2024-03-23T11:29:15.159640+0000","last_became_peered":"2024-03-23T11:29:15.159640+0000","last_unstale":"2024-03-23T11:29:17.052562+0000","last_undegraded":"2024-03-23T11:29:17.052562+0000","last_fullsized":"2024-03-23T11:29:17.052562+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:04.617758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649027+0000","last_change":"2024-03-23T11:29:15.672165+0000","last_active":"2024-03-23T11:29:16.649027+0000","last_peered":"2024-03-23T11:29:16.649027+0000","last_clean":"2024-03-23T11:29:16.649027+0000","last_became_active":"2024-03-23T11:29:15.670772+0000","last_became_peered":"2024-03-23T11:29:15.670772+0000","last_unstale":"2024-03-23T11:29:16.649027+0000","last_undegraded":"2024-03-23T11:29:16.649027+0000","last_fullsized":"2024-03-23T11:29:16.649027+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:56:56.282431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"104'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929678+0000","last_change":"2024-03-23T11:29:15.164914+0000","last_active":"2024-03-23T11:29:16.929678+0000","last_peered":"2024-03-23T11:29:16.929678+0000","last_clean":"2024-03-23T11:29:16.929678+0000","last_became_active":"2024-03-23T11:29:15.164608+0000","last_became_peered":"2024-03-23T11:29:15.164608+0000","last_unstale":"2024-03-23T11:29:16.929678+0000","last_undegraded":"2024-03-23T11:29:16.929678+0000","last_fullsized":"2024-03-23T11:29:16.929678+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:19:57.036431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.650070+0000","last_change":"2024-03-23T11:29:15.668131+0000","last_active":"2024-03-23T11:29:16.650070+0000","last_peered":"2024-03-23T11:29:16.650070+0000","last_clean":"2024-03-23T11:29:16.650070+0000","last_became_active":"2024-03-23T11:29:15.666919+0000","last_became_peered":"2024-03-23T11:29:15.666919+0000","last_unstale":"2024-03-23T11:29:16.650070+0000","last_undegraded":"2024-03-23T11:29:16.650070+0000","last_fullsized":"2024-03-23T11:29:16.650070+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:18:50.991323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.080780+0000","last_change":"2024-03-23T11:29:15.167426+0000","last_active":"2024-03-23T11:29:17.080780+0000","last_peered":"2024-03-23T11:29:17.080780+0000","last_clean":"2024-03-23T11:29:17.080780+0000","last_became_active":"2024-03-23T11:29:15.165774+0000","last_became_peered":"2024-03-23T11:29:15.165774+0000","last_unstale":"2024-03-23T11:29:17.080780+0000","last_undegraded":"2024-03-23T11:29:17.080780+0000","last_fullsized":"2024-03-23T11:29:17.080780+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:41:59.286102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076874+0000","last_change":"2024-03-23T11:29:15.654221+0000","last_active":"2024-03-23T11:29:17.076874+0000","last_peered":"2024-03-23T11:29:17.076874+0000","last_clean":"2024-03-23T11:29:17.076874+0000","last_became_active":"2024-03-23T11:29:15.653905+0000","last_became_peered":"2024-03-23T11:29:15.653905+0000","last_unstale":"2024-03-23T11:29:17.076874+0000","last_undegraded":"2024-03-23T11:29:17.076874+0000","last_fullsized":"2024-03-23T11:29:17.076874+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:26:06.732162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.080087+0000","last_change":"2024-03-23T11:29:15.167420+0000","last_active":"2024-03-23T11:29:17.080087+0000","last_peered":"2024-03-23T11:29:17.080087+0000","last_clean":"2024-03-23T11:29:17.080087+0000","last_became_active":"2024-03-23T11:29:15.167065+0000","last_became_peered":"2024-03-23T11:29:15.167065+0000","last_unstale":"2024-03-23T11:29:17.080087+0000","last_undegraded":"2024-03-23T11:29:17.080087+0000","last_fullsized":"2024-03-23T11:29:17.080087+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:26:01.038104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077577+0000","last_change":"2024-03-23T11:29:15.667475+0000","last_active":"2024-03-23T11:29:17.077577+0000","last_peered":"2024-03-23T11:29:17.077577+0000","last_clean":"2024-03-23T11:29:17.077577+0000","last_became_active":"2024-03-23T11:29:15.658307+0000","last_became_peered":"2024-03-23T11:29:15.658307+0000","last_unstale":"2024-03-23T11:29:17.077577+0000","last_undegraded":"2024-03-23T11:29:17.077577+0000","last_fullsized":"2024-03-23T11:29:17.077577+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:10:36.931539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.648477+0000","last_change":"2024-03-23T11:29:15.161859+0000","last_active":"2024-03-23T11:29:16.648477+0000","last_peered":"2024-03-23T11:29:16.648477+0000","last_clean":"2024-03-23T11:29:16.648477+0000","last_became_active":"2024-03-23T11:29:15.161564+0000","last_became_peered":"2024-03-23T11:29:15.161564+0000","last_unstale":"2024-03-23T11:29:16.648477+0000","last_undegraded":"2024-03-23T11:29:16.648477+0000","last_fullsized":"2024-03-23T11:29:16.648477+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:26:01.038104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076109+0000","last_change":"2024-03-23T11:29:15.648783+0000","last_active":"2024-03-23T11:29:17.076109+0000","last_peered":"2024-03-23T11:29:17.076109+0000","last_clean":"2024-03-23T11:29:17.076109+0000","last_became_active":"2024-03-23T11:29:15.648519+0000","last_became_peered":"2024-03-23T11:29:15.648519+0000","last_unstale":"2024-03-23T11:29:17.076109+0000","last_undegraded":"2024-03-23T11:29:17.076109+0000","last_fullsized":"2024-03-23T11:29:17.076109+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:28:49.056332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.648180+0000","last_change":"2024-03-23T11:29:15.157608+0000","last_active":"2024-03-23T11:29:16.648180+0000","last_peered":"2024-03-23T11:29:16.648180+0000","last_clean":"2024-03-23T11:29:16.648180+0000","last_became_active":"2024-03-23T11:29:15.157259+0000","last_became_peered":"2024-03-23T11:29:15.157259+0000","last_unstale":"2024-03-23T11:29:16.648180+0000","last_undegraded":"2024-03-23T11:29:16.648180+0000","last_fullsized":"2024-03-23T11:29:16.648180+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:19:57.036431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.928916+0000","last_change":"2024-03-23T11:29:15.664280+0000","last_active":"2024-03-23T11:29:16.928916+0000","last_peered":"2024-03-23T11:29:16.928916+0000","last_clean":"2024-03-23T11:29:16.928916+0000","last_became_active":"2024-03-23T11:29:15.663918+0000","last_became_peered":"2024-03-23T11:29:15.663918+0000","last_unstale":"2024-03-23T11:29:16.928916+0000","last_undegraded":"2024-03-23T11:29:16.928916+0000","last_fullsized":"2024-03-23T11:29:16.928916+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:44.511761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078124+0000","last_change":"2024-03-23T11:29:15.171195+0000","last_active":"2024-03-23T11:29:17.078124+0000","last_peered":"2024-03-23T11:29:17.078124+0000","last_clean":"2024-03-23T11:29:17.078124+0000","last_became_active":"2024-03-23T11:29:15.168383+0000","last_became_peered":"2024-03-23T11:29:15.168383+0000","last_unstale":"2024-03-23T11:29:17.078124+0000","last_undegraded":"2024-03-23T11:29:17.078124+0000","last_fullsized":"2024-03-23T11:29:17.078124+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:00:55.178584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.082050+0000","last_change":"2024-03-23T11:29:15.665043+0000","last_active":"2024-03-23T11:29:17.082050+0000","last_peered":"2024-03-23T11:29:17.082050+0000","last_clean":"2024-03-23T11:29:17.082050+0000","last_became_active":"2024-03-23T11:29:15.663990+0000","last_became_peered":"2024-03-23T11:29:15.663990+0000","last_unstale":"2024-03-23T11:29:17.082050+0000","last_undegraded":"2024-03-23T11:29:17.082050+0000","last_fullsized":"2024-03-23T11:29:17.082050+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:13.398610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649462+0000","last_change":"2024-03-23T11:29:15.160347+0000","last_active":"2024-03-23T11:29:16.649462+0000","last_peered":"2024-03-23T11:29:16.649462+0000","last_clean":"2024-03-23T11:29:16.649462+0000","last_became_active":"2024-03-23T11:29:15.160035+0000","last_became_peered":"2024-03-23T11:29:15.160035+0000","last_unstale":"2024-03-23T11:29:16.649462+0000","last_undegraded":"2024-03-23T11:29:16.649462+0000","last_fullsized":"2024-03-23T11:29:16.649462+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:31:55.748276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649218+0000","last_change":"2024-03-23T11:29:15.644839+0000","last_active":"2024-03-23T11:29:16.649218+0000","last_peered":"2024-03-23T11:29:16.649218+0000","last_clean":"2024-03-23T11:29:16.649218+0000","last_became_active":"2024-03-23T11:29:15.644500+0000","last_became_peered":"2024-03-23T11:29:15.644500+0000","last_unstale":"2024-03-23T11:29:16.649218+0000","last_undegraded":"2024-03-23T11:29:16.649218+0000","last_fullsized":"2024-03-23T11:29:16.649218+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:18.726617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.082131+0000","last_change":"2024-03-23T11:29:15.162600+0000","last_active":"2024-03-23T11:29:17.082131+0000","last_peered":"2024-03-23T11:29:17.082131+0000","last_clean":"2024-03-23T11:29:17.082131+0000","last_became_active":"2024-03-23T11:29:15.162335+0000","last_became_peered":"2024-03-23T11:29:15.162335+0000","last_unstale":"2024-03-23T11:29:17.082131+0000","last_undegraded":"2024-03-23T11:29:17.082131+0000","last_fullsized":"2024-03-23T11:29:17.082131+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:21.839696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649072+0000","last_change":"2024-03-23T11:29:15.670658+0000","last_active":"2024-03-23T11:29:16.649072+0000","last_peered":"2024-03-23T11:29:16.649072+0000","last_clean":"2024-03-23T11:29:16.649072+0000","last_became_active":"2024-03-23T11:29:15.670379+0000","last_became_peered":"2024-03-23T11:29:15.670379+0000","last_unstale":"2024-03-23T11:29:16.649072+0000","last_undegraded":"2024-03-23T11:29:16.649072+0000","last_fullsized":"2024-03-23T11:29:16.649072+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:33:29.272656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.081308+0000","last_change":"2024-03-23T11:29:15.166452+0000","last_active":"2024-03-23T11:29:17.081308+0000","last_peered":"2024-03-23T11:29:17.081308+0000","last_clean":"2024-03-23T11:29:17.081308+0000","last_became_active":"2024-03-23T11:29:15.166192+0000","last_became_peered":"2024-03-23T11:29:15.166192+0000","last_unstale":"2024-03-23T11:29:17.081308+0000","last_undegraded":"2024-03-23T11:29:17.081308+0000","last_fullsized":"2024-03-23T11:29:17.081308+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:21:16.840911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929649+0000","last_change":"2024-03-23T11:29:15.670394+0000","last_active":"2024-03-23T11:29:16.929649+0000","last_peered":"2024-03-23T11:29:16.929649+0000","last_clean":"2024-03-23T11:29:16.929649+0000","last_became_active":"2024-03-23T11:29:15.669874+0000","last_became_peered":"2024-03-23T11:29:15.669874+0000","last_unstale":"2024-03-23T11:29:16.929649+0000","last_undegraded":"2024-03-23T11:29:16.929649+0000","last_fullsized":"2024-03-23T11:29:16.929649+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:42:00.302069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929594+0000","last_change":"2024-03-23T11:29:15.165222+0000","last_active":"2024-03-23T11:29:16.929594+0000","last_peered":"2024-03-23T11:29:16.929594+0000","last_clean":"2024-03-23T11:29:16.929594+0000","last_became_active":"2024-03-23T11:29:15.164799+0000","last_became_peered":"2024-03-23T11:29:15.164799+0000","last_unstale":"2024-03-23T11:29:16.929594+0000","last_undegraded":"2024-03-23T11:29:16.929594+0000","last_fullsized":"2024-03-23T11:29:16.929594+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:49:27.211897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077150+0000","last_change":"2024-03-23T11:29:15.660807+0000","last_active":"2024-03-23T11:29:17.077150+0000","last_peered":"2024-03-23T11:29:17.077150+0000","last_clean":"2024-03-23T11:29:17.077150+0000","last_became_active":"2024-03-23T11:29:15.660489+0000","last_became_peered":"2024-03-23T11:29:15.660489+0000","last_unstale":"2024-03-23T11:29:17.077150+0000","last_undegraded":"2024-03-23T11:29:17.077150+0000","last_fullsized":"2024-03-23T11:29:17.077150+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:24:05.630578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.080180+0000","last_change":"2024-03-23T11:29:15.164486+0000","last_active":"2024-03-23T11:29:17.080180+0000","last_peered":"2024-03-23T11:29:17.080180+0000","last_clean":"2024-03-23T11:29:17.080180+0000","last_became_active":"2024-03-23T11:29:15.163263+0000","last_became_peered":"2024-03-23T11:29:15.163263+0000","last_unstale":"2024-03-23T11:29:17.080180+0000","last_undegraded":"2024-03-23T11:29:17.080180+0000","last_fullsized":"2024-03-23T11:29:17.080180+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:49:27.211897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052222+0000","last_change":"2024-03-23T11:29:15.673835+0000","last_active":"2024-03-23T11:29:17.052222+0000","last_peered":"2024-03-23T11:29:17.052222+0000","last_clean":"2024-03-23T11:29:17.052222+0000","last_became_active":"2024-03-23T11:29:15.672375+0000","last_became_peered":"2024-03-23T11:29:15.672375+0000","last_unstale":"2024-03-23T11:29:17.052222+0000","last_undegraded":"2024-03-23T11:29:17.052222+0000","last_fullsized":"2024-03-23T11:29:17.052222+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:47.807515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.928913+0000","last_change":"2024-03-23T11:29:15.164188+0000","last_active":"2024-03-23T11:29:16.928913+0000","last_peered":"2024-03-23T11:29:16.928913+0000","last_clean":"2024-03-23T11:29:16.928913+0000","last_became_active":"2024-03-23T11:29:15.163837+0000","last_became_peered":"2024-03-23T11:29:15.163837+0000","last_unstale":"2024-03-23T11:29:16.928913+0000","last_undegraded":"2024-03-23T11:29:16.928913+0000","last_fullsized":"2024-03-23T11:29:16.928913+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:57:18.783732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076206+0000","last_change":"2024-03-23T11:29:15.663646+0000","last_active":"2024-03-23T11:29:17.076206+0000","last_peered":"2024-03-23T11:29:17.076206+0000","last_clean":"2024-03-23T11:29:17.076206+0000","last_became_active":"2024-03-23T11:29:15.663344+0000","last_became_peered":"2024-03-23T11:29:15.663344+0000","last_unstale":"2024-03-23T11:29:17.076206+0000","last_undegraded":"2024-03-23T11:29:17.076206+0000","last_fullsized":"2024-03-23T11:29:17.076206+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:40:12.294892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077858+0000","last_change":"2024-03-23T11:29:15.169786+0000","last_active":"2024-03-23T11:29:17.077858+0000","last_peered":"2024-03-23T11:29:17.077858+0000","last_clean":"2024-03-23T11:29:17.077858+0000","last_became_active":"2024-03-23T11:29:15.167122+0000","last_became_peered":"2024-03-23T11:29:15.167122+0000","last_unstale":"2024-03-23T11:29:17.077858+0000","last_undegraded":"2024-03-23T11:29:17.077858+0000","last_fullsized":"2024-03-23T11:29:17.077858+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:26:01.038104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.928937+0000","last_change":"2024-03-23T11:29:15.666879+0000","last_active":"2024-03-23T11:29:16.928937+0000","last_peered":"2024-03-23T11:29:16.928937+0000","last_clean":"2024-03-23T11:29:16.928937+0000","last_became_active":"2024-03-23T11:29:15.666394+0000","last_became_peered":"2024-03-23T11:29:15.666394+0000","last_unstale":"2024-03-23T11:29:16.928937+0000","last_undegraded":"2024-03-23T11:29:16.928937+0000","last_fullsized":"2024-03-23T11:29:16.928937+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:20:38.886782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076344+0000","last_change":"2024-03-23T11:29:15.164528+0000","last_active":"2024-03-23T11:29:17.076344+0000","last_peered":"2024-03-23T11:29:17.076344+0000","last_clean":"2024-03-23T11:29:17.076344+0000","last_became_active":"2024-03-23T11:29:15.164222+0000","last_became_peered":"2024-03-23T11:29:15.164222+0000","last_unstale":"2024-03-23T11:29:17.076344+0000","last_undegraded":"2024-03-23T11:29:17.076344+0000","last_fullsized":"2024-03-23T11:29:17.076344+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:31:06.346410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079509+0000","last_change":"2024-03-23T11:29:15.650442+0000","last_active":"2024-03-23T11:29:17.079509+0000","last_peered":"2024-03-23T11:29:17.079509+0000","last_clean":"2024-03-23T11:29:17.079509+0000","last_became_active":"2024-03-23T11:29:15.650158+0000","last_became_peered":"2024-03-23T11:29:15.650158+0000","last_unstale":"2024-03-23T11:29:17.079509+0000","last_undegraded":"2024-03-23T11:29:17.079509+0000","last_fullsized":"2024-03-23T11:29:17.079509+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:47:14.336720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"104'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.650329+0000","last_change":"2024-03-23T11:29:15.159260+0000","last_active":"2024-03-23T11:29:16.650329+0000","last_peered":"2024-03-23T11:29:16.650329+0000","last_clean":"2024-03-23T11:29:16.650329+0000","last_became_active":"2024-03-23T11:29:15.157891+0000","last_became_peered":"2024-03-23T11:29:15.157891+0000","last_unstale":"2024-03-23T11:29:16.650329+0000","last_undegraded":"2024-03-23T11:29:16.650329+0000","last_fullsized":"2024-03-23T11:29:16.650329+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:05:26.465230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":252,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T11:29:15.641662+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:29:15.641662+0000","last_peered":"2024-03-23T11:29:15.641662+0000","last_clean":"2024-03-23T11:29:15.641662+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:29:15.641662+0000","last_undegraded":"2024-03-23T11:29:15.641662+0000","last_fullsized":"2024-03-23T11:29:15.641662+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542273,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1896448,"data_stored":1149689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.083,"15min":0.98999999999999999},"min":{"1min":0.623,"5min":0.5,"15min":0.5},"max":{"1min":1.599,"5min":2.8370000000000002,"15min":2.8370000000000002},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.0309999999999999},"min":{"1min":0.749,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.466,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.90400000000000003}]},{"osd":1,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.26,"15min":1.198},"min":{"1min":0.73499999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.708,"5min":3.569,"15min":3.569},"last":1.083},{"interface":"front","average":{"1min":1.345,"5min":1.3100000000000001,"15min":1.2649999999999999},"min":{"1min":0.877,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.653,"5min":1.8009999999999999,"15min":2.4350000000000001},"last":1.24}]},{"osd":2,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.1399999999999999,"15min":1.077},"min":{"1min":0.69599999999999995,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.379,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":1.625},{"interface":"front","average":{"1min":1.212,"5min":1.238,"15min":1.179},"min":{"1min":0.84199999999999997,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.6060000000000001,"5min":3.524,"15min":3.524},"last":1.3939999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.224,"15min":1.175},"min":{"1min":0.72199999999999998,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.5189999999999999,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.3180000000000001,"15min":1.27},"min":{"1min":0.93500000000000005,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.653,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":1.2729999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.22,"15min":1.222},"min":{"1min":0.80500000000000005,"5min":0.65300000000000002,"15min":0.54300000000000004},"max":{"1min":2.1560000000000001,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":1.026},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.407,"15min":1.3839999999999999},"min":{"1min":0.94699999999999995,"5min":0.63800000000000001,"15min":0.60199999999999998},"max":{"1min":1.623,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":1.5029999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.365,"15min":1.341},"min":{"1min":0.84599999999999997,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.7969999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4830000000000001,"15min":1.407},"min":{"1min":0.91700000000000004,"5min":0.77100000000000002,"15min":0.58199999999999996},"max":{"1min":1.978,"5min":2.0219999999999998,"15min":2.0859999999999999},"last":1.357}]},{"osd":6,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.446,"15min":1.4259999999999999},"min":{"1min":0.98699999999999999,"5min":0.73599999999999999,"15min":0.61399999999999999},"max":{"1min":2.0680000000000001,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.4039999999999999,"15min":1.363},"min":{"1min":1.034,"5min":0.61699999999999999,"15min":0.59899999999999998},"max":{"1min":1.6359999999999999,"5min":3.218,"15min":3.218},"last":1.4179999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542273,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1298432,"data_stored":559192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.107,"15min":1.1020000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.522,"5min":2.403,"15min":3.9140000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1839999999999999,"15min":1.141},"min":{"1min":0.751,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.669,"5min":1.774,"15min":6.6539999999999999},"last":1.0549999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1659999999999999,"15min":1.2030000000000001},"min":{"1min":0.77300000000000002,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.6100000000000001,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":1.611},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1850000000000001,"15min":1.2270000000000001},"min":{"1min":0.82299999999999995,"5min":0.76200000000000001,"15min":0.70399999999999996},"max":{"1min":1.514,"5min":1.9510000000000001,"15min":2.0859999999999999},"last":1.1890000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.1870000000000001,"15min":1.228},"min":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.5049999999999999,"5min":1.6419999999999999,"15min":3.73},"last":0.87},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.284,"15min":1.2869999999999999},"min":{"1min":0.73499999999999999,"5min":0.72899999999999998,"15min":0.68100000000000005},"max":{"1min":1.6140000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.3540000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.2270000000000001,"15min":1.2649999999999999},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.749},"max":{"1min":1.4790000000000001,"5min":1.9890000000000001,"15min":12.914},"last":1.006},{"interface":"front","average":{"1min":1.369,"5min":1.335,"15min":1.349},"min":{"1min":0.82099999999999995,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6559999999999999,"5min":1.9670000000000001,"15min":3.8100000000000001},"last":1.8919999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3100000000000001,"15min":1.349},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.51100000000000001},"max":{"1min":1.772,"5min":2.4420000000000002,"15min":2.702},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3400000000000001,"15min":1.3819999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.82299999999999995},"max":{"1min":1.845,"5min":2.1080000000000001,"15min":7.2140000000000004},"last":1.1040000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4139999999999999,"15min":1.4890000000000001},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.73899999999999999},"max":{"1min":1.7549999999999999,"5min":2.1099999999999999,"15min":3.097},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.4610000000000001,"15min":1.478},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.72499999999999998},"max":{"1min":1.8640000000000001,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":1.702}]},{"osd":7,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5389999999999999,"15min":1.621},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.80400000000000005},"max":{"1min":2.0760000000000001,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.488,"15min":1.4890000000000001},"min":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.57699999999999996},"max":{"1min":1.911,"5min":4.085,"15min":4.085},"last":2.0649999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542273,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1298432,"data_stored":559279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.093,"15min":1.139},"min":{"1min":0.755,"5min":0.65500000000000003,"15min":0.55400000000000005},"max":{"1min":1.484,"5min":1.518,"15min":2.2999999999999998},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1220000000000001,"15min":1.147},"min":{"1min":0.75,"5min":0.61499999999999999,"15min":0.47799999999999998},"max":{"1min":1.407,"5min":2.1190000000000002,"15min":2.355},"last":1.6259999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.1899999999999999,"15min":1.1779999999999999},"min":{"1min":0.89800000000000002,"5min":0.70599999999999996,"15min":0.65300000000000002},"max":{"1min":1.5109999999999999,"5min":1.6699999999999999,"15min":2.2559999999999998},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.159,"15min":1.1930000000000001},"min":{"1min":0.86599999999999999,"5min":0.66900000000000004,"15min":0.495},"max":{"1min":1.391,"5min":1.627,"15min":1.8580000000000001},"last":1.278}]},{"osd":2,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1910000000000001,"15min":1.234},"min":{"1min":0.89900000000000002,"5min":0.71699999999999997,"15min":0.60199999999999998},"max":{"1min":1.3959999999999999,"5min":1.6659999999999999,"15min":2.012},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.2569999999999999,"15min":1.24},"min":{"1min":0.86499999999999999,"5min":0.77600000000000002,"15min":0.64400000000000002},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.946},"last":1.579}]},{"osd":3,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.262,"15min":1.2949999999999999},"min":{"1min":0.93500000000000005,"5min":0.69299999999999995,"15min":0.66800000000000004},"max":{"1min":1.6499999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.224},{"interface":"front","average":{"1min":1.367,"5min":1.3480000000000001,"15min":1.3839999999999999},"min":{"1min":0.97999999999999998,"5min":0.78600000000000003,"15min":0.73199999999999998},"max":{"1min":1.6950000000000001,"5min":1.7549999999999999,"15min":2.3999999999999999},"last":1.72}]},{"osd":4,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.2889999999999999,"15min":1.357},"min":{"1min":0.98599999999999999,"5min":0.754,"15min":0.56699999999999995},"max":{"1min":1.6679999999999999,"5min":1.8779999999999999,"15min":2.2090000000000001},"last":1.385},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.4450000000000001,"15min":1.502},"min":{"1min":1.071,"5min":1.0169999999999999,"15min":0.65000000000000002},"max":{"1min":1.774,"5min":2.2839999999999998,"15min":2.5009999999999999},"last":1.6859999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.363,"15min":1.4239999999999999},"min":{"1min":1.0609999999999999,"5min":0.85699999999999998,"15min":0.48899999999999999},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":2.4860000000000002},"last":1.329},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.371,"15min":1.4570000000000001},"min":{"1min":1.2370000000000001,"5min":0.86599999999999999,"15min":0.624},"max":{"1min":1.7909999999999999,"5min":2.0569999999999999,"15min":2.3679999999999999},"last":1.5329999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.4279999999999999,"15min":1.4970000000000001},"min":{"1min":1.1000000000000001,"5min":0.90900000000000003,"15min":0.54000000000000004},"max":{"1min":1.841,"5min":2.5720000000000001,"15min":2.7229999999999999},"last":1.462},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.504,"15min":1.5700000000000001},"min":{"1min":1.2689999999999999,"5min":1.0469999999999999,"15min":0.60099999999999998},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.7509999999999999},"last":1.409}]}]},{"osd":4,"up_from":14,"seq":60129542273,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1314816,"data_stored":560897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.238,"15min":1.1539999999999999},"min":{"1min":0.79800000000000004,"5min":0.66800000000000004,"15min":0.58999999999999997},"max":{"1min":1.5860000000000001,"5min":1.627,"15min":1.6759999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.337,"15min":1.26},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.63700000000000001},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.8009999999999999},"last":1.512}]},{"osd":1,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1839999999999999,"15min":1.1479999999999999},"min":{"1min":0.76700000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.4970000000000001,"5min":1.6990000000000001,"15min":4.327},"last":1.409},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.169,"15min":1.137},"min":{"1min":0.86199999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.5389999999999999,"5min":1.7470000000000001,"15min":2.4369999999999998},"last":1.3260000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.194,"15min":1.2110000000000001},"min":{"1min":0.91100000000000003,"5min":0.879,"15min":0.624},"max":{"1min":1.518,"5min":1.627,"15min":2.6080000000000001},"last":1.119},{"interface":"front","average":{"1min":1.175,"5min":1.244,"15min":1.248},"min":{"1min":0.876,"5min":0.80600000000000005,"15min":0.71799999999999997},"max":{"1min":1.73,"5min":1.927,"15min":4.1310000000000002},"last":1.1639999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.234,"15min":1.2250000000000001},"min":{"1min":0.83499999999999996,"5min":0.747,"15min":0.622},"max":{"1min":1.4550000000000001,"5min":1.6499999999999999,"15min":2.5579999999999998},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.345,"5min":1.3420000000000001,"15min":1.3},"min":{"1min":0.97399999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.5880000000000001,"5min":1.665,"15min":4.2400000000000002},"last":1.5900000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.238,"15min":1.2609999999999999},"min":{"1min":0.94499999999999995,"5min":0.83899999999999997,"15min":0.74299999999999999},"max":{"1min":1.7,"5min":1.966,"15min":2.508},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.3400000000000001,"15min":1.359},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.60499999999999998},"max":{"1min":1.6679999999999999,"5min":1.877,"15min":5.859},"last":1.075}]},{"osd":6,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.409,"15min":1.3959999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.58899999999999997},"max":{"1min":1.784,"5min":2.222,"15min":5.3339999999999996},"last":1.885},{"interface":"front","average":{"1min":1.421,"5min":1.47,"15min":1.4830000000000001},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.72599999999999998},"max":{"1min":1.71,"5min":1.875,"15min":5.3940000000000001},"last":1.732}]},{"osd":7,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.5169999999999999,"15min":1.4870000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.754},"max":{"1min":1.879,"5min":2.1549999999999998,"15min":2.6859999999999999},"last":1.944},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.365,"15min":1.3899999999999999},"min":{"1min":0.79000000000000004,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.6930000000000001,"5min":2.028,"15min":2.641},"last":1.569}]}]},{"osd":3,"up_from":14,"seq":60129542273,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1310720,"data_stored":560795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.86899999999999999,"15min":0.93100000000000005},"min":{"1min":0.47499999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.27,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.099},{"interface":"front","average":{"1min":1.103,"5min":1.0760000000000001,"15min":1.097},"min":{"1min":0.65400000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.54,"5min":1.54,"15min":2.0790000000000002},"last":1.4910000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":0.98599999999999999},"min":{"1min":0.70499999999999996,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.3320000000000001,"5min":1.8049999999999999,"15min":1.976},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.218,"15min":1.246},"min":{"1min":0.74399999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.774,"5min":1.8100000000000001,"15min":2.2309999999999999},"last":1.5700000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.107,"15min":1.127},"min":{"1min":0.63200000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.746,"5min":1.839,"15min":3.3109999999999999},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.177,"15min":1.216},"min":{"1min":0.65100000000000002,"5min":0.51900000000000002,"15min":0.48399999999999999},"max":{"1min":1.595,"5min":2.2440000000000002,"15min":2.7109999999999999},"last":1.274}]},{"osd":4,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2609999999999999,"15min":1.276},"min":{"1min":0.93200000000000005,"5min":0.79000000000000004,"15min":0.73199999999999998},"max":{"1min":1.6899999999999999,"5min":1.897,"15min":2.3490000000000002},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.429,"15min":1.4870000000000001},"min":{"1min":1.0800000000000001,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.081,"5min":2.081,"15min":4.8970000000000002},"last":1.627}]},{"osd":5,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.458,"15min":1.494},"min":{"1min":1.167,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":1.6819999999999999,"5min":2.1520000000000001,"15min":3.4910000000000001},"last":1.704},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3660000000000001,"15min":1.423},"min":{"1min":0.92600000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":3.125},"last":1.339}]},{"osd":6,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3640000000000001,"15min":1.4530000000000001},"min":{"1min":1.117,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.8640000000000001,"5min":1.974,"15min":4.5499999999999998},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4419999999999999,"15min":1.5129999999999999},"min":{"1min":0.91100000000000003,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6439999999999999,"5min":2.0569999999999999,"15min":2.8450000000000002},"last":1.4410000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3939999999999999,"15min":1.47},"min":{"1min":1.032,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.7789999999999999,"5min":1.9219999999999999,"15min":2.952},"last":1.222},{"interface":"front","average":{"1min":1.468,"5min":1.4690000000000001,"15min":1.5629999999999999},"min":{"1min":0.88600000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":4.798},"last":1.5229999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542273,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1298432,"data_stored":559243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.90100000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.2290000000000001,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.052,"5min":1.097,"15min":1.0389999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.35099999999999998},"max":{"1min":1.667,"5min":1.7310000000000001,"15min":1.8220000000000001},"last":1.2210000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.85599999999999998,"15min":0.91500000000000004},"min":{"1min":0.504,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.006,"5min":1.365,"15min":1.9430000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.0580000000000001,"15min":1.0189999999999999},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.35799999999999998},"max":{"1min":1.49,"5min":1.593,"15min":1.7230000000000001},"last":1.5349999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1060000000000001,"15min":1.1140000000000001},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.5900000000000001,"5min":1.905,"15min":1.98},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.6879999999999999,"5min":1.827,"15min":2.1560000000000001},"last":0.93300000000000005}]},{"osd":4,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.341,"15min":1.3500000000000001},"min":{"1min":0.91800000000000004,"5min":0.73299999999999998,"15min":0.69599999999999995},"max":{"1min":1.758,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.421,"15min":1.3899999999999999},"min":{"1min":0.93799999999999994,"5min":0.85799999999999998,"15min":0.83499999999999996},"max":{"1min":1.879,"5min":2.0230000000000001,"15min":2.085},"last":1.7130000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3100000000000001,"15min":1.3759999999999999},"min":{"1min":0.752,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7130000000000001,"5min":1.9530000000000001,"15min":2.5830000000000002},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3600000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.66000000000000003},"max":{"1min":1.6839999999999999,"5min":1.96,"15min":2.6469999999999998},"last":1.28}]},{"osd":6,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3799999999999999,"15min":1.3759999999999999},"min":{"1min":0.84499999999999997,"5min":0.76300000000000001,"15min":0.68600000000000005},"max":{"1min":1.905,"5min":2.0600000000000001,"15min":2.758},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.3799999999999999,"15min":1.464},"min":{"1min":0.79100000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.879,"5min":1.889,"15min":2.8580000000000001},"last":1.3440000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.4350000000000001,"15min":1.4219999999999999},"min":{"1min":0.84199999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.9039999999999999,"5min":2.0150000000000001,"15min":2.246},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.4670000000000001,"15min":1.538},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.919,"5min":2.012,"15min":2.7120000000000002},"last":1.4930000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542274,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1290240,"data_stored":558540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.90800000000000003,"15min":0.89200000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":3.2959999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.03,"15min":1.0089999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.99},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.95299999999999996,"15min":0.96499999999999997},"min":{"1min":0.55400000000000005,"5min":0.505,"15min":0.40699999999999997},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.738},"last":0.499},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0880000000000001,"15min":1.1120000000000001},"min":{"1min":0.57399999999999995,"5min":0.55100000000000005,"15min":0.49399999999999999},"max":{"1min":1.5569999999999999,"5min":1.762,"15min":2.2160000000000002},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2130000000000001,"15min":1.226},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.6519999999999999,"5min":1.653,"15min":2.4430000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.113,"5min":1.1120000000000001,"15min":1.1379999999999999},"min":{"1min":0.65700000000000003,"5min":0.52000000000000002,"15min":0.434},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.4169999999999998},"last":0.45900000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3580000000000001,"15min":1.3009999999999999},"min":{"1min":1.0549999999999999,"5min":0.83899999999999997,"15min":0.79400000000000004},"max":{"1min":1.7969999999999999,"5min":1.9019999999999999,"15min":2.153},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.292,"5min":1.2749999999999999,"15min":1.3129999999999999},"min":{"1min":0.90300000000000002,"5min":0.81299999999999994,"15min":0.74199999999999999},"max":{"1min":1.8049999999999999,"5min":1.867,"15min":2.758},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3400000000000001,"15min":1.3879999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.81699999999999995},"max":{"1min":1.992,"5min":1.992,"15min":2.6589999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4159999999999999,"15min":1.4399999999999999},"min":{"1min":0.96899999999999997,"5min":0.89900000000000002,"15min":0.72199999999999998},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":3.0329999999999999},"last":0.66000000000000003}]},{"osd":6,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.421,"15min":1.4299999999999999},"min":{"1min":0.88600000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.8839999999999999,"5min":2.1600000000000001,"15min":2.2730000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.421,"15min":1.504},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":2.9630000000000001},"last":0.747}]},{"osd":7,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.5009999999999999,"15min":1.548},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":3.2749999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.482,"5min":1.522,"15min":1.5529999999999999},"min":{"1min":0.93500000000000005,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.962,"5min":2.242,"15min":2.7829999999999999},"last":0.71999999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542272,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32964,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338876,"statfs":{"total":96636764160,"available":96603009024,"internally_reserved":0,"allocated":1888256,"data_stored":1149490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.47599999999999998,"5min":0.379,"15min":0.376},"max":{"1min":1.8660000000000001,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97499999999999998,"15min":0.95799999999999996},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.30399999999999999},"max":{"1min":1.536,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.71699999999999997}]},{"osd":2,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.014,"15min":1.036},"min":{"1min":0.80200000000000005,"5min":0.44500000000000001,"15min":0.33700000000000002},"max":{"1min":1.4910000000000001,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.155,"5min":1.093,"15min":1.0469999999999999},"min":{"1min":0.63200000000000001,"5min":0.54000000000000004,"15min":0.432},"max":{"1min":1.6299999999999999,"5min":2.5760000000000001,"15min":2.5760000000000001},"last":0.90400000000000003}]},{"osd":3,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1499999999999999,"15min":1.1140000000000001},"min":{"1min":0.66800000000000004,"5min":0.40899999999999997,"15min":0.35799999999999998},"max":{"1min":1.718,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.335,"15min":1.3180000000000001},"min":{"1min":0.75600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.968,"5min":2.9390000000000001,"15min":2.9390000000000001},"last":1.2869999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.298,"15min":1.343},"min":{"1min":0.94399999999999995,"5min":0.77900000000000003,"15min":0.56100000000000005},"max":{"1min":1.5860000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.3069999999999999,"15min":1.284},"min":{"1min":0.82099999999999995,"5min":0.73099999999999998,"15min":0.39900000000000002},"max":{"1min":1.8460000000000001,"5min":3.125,"15min":3.125},"last":1.968}]},{"osd":5,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.3520000000000001,"15min":1.391},"min":{"1min":0.91200000000000003,"5min":0.90200000000000002,"15min":0.51200000000000001},"max":{"1min":1.925,"5min":3.0830000000000002,"15min":3.0830000000000002},"last":1.8740000000000001},{"interface":"front","average":{"1min":1.345,"5min":1.349,"15min":1.454},"min":{"1min":0.877,"5min":0.76300000000000001,"15min":0.52700000000000002},"max":{"1min":2.0449999999999999,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":1.837}]},{"osd":6,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.399,"15min":1.4810000000000001},"min":{"1min":0.90200000000000002,"5min":0.876,"15min":0.47999999999999998},"max":{"1min":2.1030000000000002,"5min":3.391,"15min":3.391},"last":1.704},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.391,"15min":1.4119999999999999},"min":{"1min":0.94899999999999995,"5min":0.746,"15min":0.58399999999999996},"max":{"1min":2.2170000000000001,"5min":3.153,"15min":3.153},"last":1.3939999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4890000000000001,"15min":1.522},"min":{"1min":0.98699999999999999,"5min":0.77300000000000002,"15min":0.60599999999999998},"max":{"1min":2.1400000000000001,"5min":3.5350000000000001,"15min":3.5350000000000001},"last":1.579},{"interface":"front","average":{"1min":1.536,"5min":1.4830000000000001,"15min":1.4970000000000001},"min":{"1min":1.125,"5min":0.83799999999999997,"15min":0.45300000000000001},"max":{"1min":2.2610000000000001,"5min":3.3650000000000002,"15min":3.3650000000000002},"last":2.0350000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:29:20.656 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:29:20.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.920+0000 7fa098600640 1 Processor -- start 2024-03-23T11:29:20.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.921+0000 7fa098600640 1 -- start start 2024-03-23T11:29:20.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.922+0000 7fa098600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0900b8350 0x7fa0900b4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.922+0000 7fa098600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0900b4980 0x7fa0900d1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.922+0000 7fa098600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0900d0ff0 con 0x7fa09015cab0 2024-03-23T11:29:20.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.922+0000 7fa098600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0900b4e10 con 0x7fa0900b8350 2024-03-23T11:29:20.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.922+0000 7fa098600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0900b5b60 con 0x7fa0900b4980 2024-03-23T11:29:20.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.923+0000 7fa095200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0900b8350 0x7fa0900b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:20.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.923+0000 7fa095200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0900b8350 0x7fa0900b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54446/0 (socket says 172.21.15.62:54446) 2024-03-23T11:29:20.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.923+0000 7fa095200640 1 -- 172.21.15.62:0/1191678420 learned_addr learned my addr 172.21.15.62:0/1191678420 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:20.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.923+0000 7fa096600640 1 --2- 172.21.15.62:0/1191678420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0900b4980 0x7fa0900d1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:20.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.923+0000 7fa095200640 1 -- 172.21.15.62:0/1191678420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0900b4980 msgr2=0x7fa0900d1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:20.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.923+0000 7fa095200640 1 --2- 172.21.15.62:0/1191678420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0900b4980 0x7fa0900d1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.923+0000 7fa095200640 1 -- 172.21.15.62:0/1191678420 >> v1:172.21.15.120:6789/0 conn(0x7fa09015cab0 legacy=0x7fa09006b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.923+0000 7fa095200640 1 -- 172.21.15.62:0/1191678420 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09006b9e0 con 0x7fa0900b8350 2024-03-23T11:29:20.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.925+0000 7fa095200640 1 --2- 172.21.15.62:0/1191678420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0900b8350 0x7fa0900b4440 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009870 tx=0x7fa07c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66d0cc90915210 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:20.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.925+0000 7fa08fe00640 1 -- 172.21.15.62:0/1191678420 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07c030840 con 0x7fa0900b8350 2024-03-23T11:29:20.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.925+0000 7fa08fe00640 1 -- 172.21.15.62:0/1191678420 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07c0309a0 con 0x7fa0900b8350 2024-03-23T11:29:20.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.925+0000 7fa08fe00640 1 -- 172.21.15.62:0/1191678420 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07c030d30 con 0x7fa0900b8350 2024-03-23T11:29:20.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.926+0000 7fa098600640 1 -- 172.21.15.62:0/1191678420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0900b8350 msgr2=0x7fa0900b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.926+0000 7fa098600640 1 --2- 172.21.15.62:0/1191678420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0900b8350 0x7fa0900b4440 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009870 tx=0x7fa07c02b980 comp rx=0 tx=0).stop 2024-03-23T11:29:20.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.926+0000 7fa098600640 1 -- 172.21.15.62:0/1191678420 shutdown_connections 2024-03-23T11:29:20.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.926+0000 7fa098600640 1 --2- 172.21.15.62:0/1191678420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0900b4980 0x7fa0900d1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.926+0000 7fa098600640 1 --2- 172.21.15.62:0/1191678420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0900b8350 0x7fa0900b4440 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.926+0000 7fa098600640 1 -- 172.21.15.62:0/1191678420 >> 172.21.15.62:0/1191678420 conn(0x7fa0900cf620 msgr2=0x7fa0900cfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:20.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.926+0000 7fa098600640 1 -- 172.21.15.62:0/1191678420 shutdown_connections 2024-03-23T11:29:20.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.926+0000 7fa098600640 1 -- 172.21.15.62:0/1191678420 wait complete. 2024-03-23T11:29:20.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.928+0000 7fa098600640 1 Processor -- start 2024-03-23T11:29:20.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.928+0000 7fa098600640 1 -- start start 2024-03-23T11:29:20.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.929+0000 7fa098600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0900b4980 0x7fa090157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.929+0000 7fa098600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0900b8350 0x7fa090152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.929+0000 7fa098600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa09015cab0 0x7fa090144d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.929+0000 7fa098600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0900cb3d0 con 0x7fa0900b4980 2024-03-23T11:29:20.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.929+0000 7fa098600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0901521b0 con 0x7fa09015cab0 2024-03-23T11:29:20.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.929+0000 7fa098600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0901f2270 con 0x7fa0900b8350 2024-03-23T11:29:20.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.929+0000 7fa095c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0900b4980 0x7fa090157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:20.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.929+0000 7fa095c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0900b4980 0x7fa090157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54454/0 (socket says 172.21.15.62:54454) 2024-03-23T11:29:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.929+0000 7fa095c00640 1 -- 172.21.15.62:0/2116898083 learned_addr learned my addr 172.21.15.62:0/2116898083 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.929+0000 7fa096600640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa09015cab0 0x7fa090144d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.930+0000 7fa095c00640 1 -- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0900b8350 msgr2=0x7fa090152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.930+0000 7fa095c00640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0900b8350 0x7fa090152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.930+0000 7fa095c00640 1 -- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa09015cab0 msgr2=0x7fa090144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.930+0000 7fa095c00640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa09015cab0 0x7fa090144d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.930+0000 7fa095c00640 1 -- 172.21.15.62:0/2116898083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c0094d0 con 0x7fa0900b4980 2024-03-23T11:29:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.930+0000 7fa095c00640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0900b4980 0x7fa090157d60 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fa08400c400 tx=0x7fa08400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.930+0000 7fa08d600640 1 -- 172.21.15.62:0/2116898083 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0840052d0 con 0x7fa0900b4980 2024-03-23T11:29:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.930+0000 7fa08d600640 1 -- 172.21.15.62:0/2116898083 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa084005430 con 0x7fa0900b4980 2024-03-23T11:29:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.930+0000 7fa08d600640 1 -- 172.21.15.62:0/2116898083 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa084003830 con 0x7fa0900b4980 2024-03-23T11:29:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.930+0000 7fa098600640 1 -- 172.21.15.62:0/2116898083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa090153640 con 0x7fa0900b4980 2024-03-23T11:29:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.931+0000 7fa098600640 1 -- 172.21.15.62:0/2116898083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0900cb060 con 0x7fa0900b4980 2024-03-23T11:29:20.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.932+0000 7fa098600640 1 -- 172.21.15.62:0/2116898083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0900738e0 con 0x7fa0900b4980 2024-03-23T11:29:20.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.933+0000 7fa08d600640 1 -- 172.21.15.62:0/2116898083 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa084003990 con 0x7fa0900b4980 2024-03-23T11:29:20.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.933+0000 7fa08d600640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa07006a7a0 0x7fa07006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:20.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.933+0000 7fa08d600640 1 -- 172.21.15.62:0/2116898083 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa084088430 con 0x7fa0900b4980 2024-03-23T11:29:20.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.934+0000 7fa095200640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa07006a7a0 0x7fa07006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.935+0000 7fa095200640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa07006a7a0 0x7fa07006cc60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa07c036000 tx=0x7fa07c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:20.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:20.938+0000 7fa08d600640 1 -- 172.21.15.62:0/2116898083 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa084053720 con 0x7fa0900b4980 2024-03-23T11:29:21.102 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:29:21.111+0000 7fd542c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:29:21.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.134+0000 7fa098600640 1 -- 172.21.15.62:0/2116898083 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa09015ce90 con 0x7fa07006a7a0 2024-03-23T11:29:21.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.157+0000 7fa08d600640 1 -- 172.21.15.62:0/2116898083 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287295 (secure 0 0 0) 0x7fa07c030b10 con 0x7fa07006a7a0 2024-03-23T11:29:21.152 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:21.153 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:29:21.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.162+0000 7fa098600640 1 -- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa07006a7a0 msgr2=0x7fa07006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:21.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.162+0000 7fa098600640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa07006a7a0 0x7fa07006cc60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa07c036000 tx=0x7fa07c03b040 comp rx=0 tx=0).stop 2024-03-23T11:29:21.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.162+0000 7fa098600640 1 -- 172.21.15.62:0/2116898083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0900b4980 msgr2=0x7fa090157d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:21.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.162+0000 7fa098600640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0900b4980 0x7fa090157d60 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fa08400c400 tx=0x7fa08400c8c0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.162+0000 7fa098600640 1 -- 172.21.15.62:0/2116898083 shutdown_connections 2024-03-23T11:29:21.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.162+0000 7fa098600640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa07006a7a0 0x7fa07006cc60 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.162+0000 7fa098600640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa09015cab0 0x7fa090144d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.162+0000 7fa098600640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0900b8350 0x7fa090152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.162+0000 7fa098600640 1 --2- 172.21.15.62:0/2116898083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0900b4980 0x7fa090157d60 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.162+0000 7fa098600640 1 -- 172.21.15.62:0/2116898083 >> 172.21.15.62:0/2116898083 conn(0x7fa0900cf620 msgr2=0x7fa09015bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:21.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.162+0000 7fa098600640 1 -- 172.21.15.62:0/2116898083 shutdown_connections 2024-03-23T11:29:21.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.163+0000 7fa098600640 1 -- 172.21.15.62:0/2116898083 wait complete. 2024-03-23T11:29:21.172 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-03-23T11:29:19.463436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":159,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260292,"kb_used_data":11324,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754714428,"statfs":{"total":773094113280,"available":772827574272,"internally_reserved":0,"allocated":11595776,"data_stored":5657125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.342932"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649246+0000","last_change":"2024-03-23T11:29:15.669730+0000","last_active":"2024-03-23T11:29:16.649246+0000","last_peered":"2024-03-23T11:29:16.649246+0000","last_clean":"2024-03-23T11:29:16.649246+0000","last_became_active":"2024-03-23T11:29:15.669463+0000","last_became_peered":"2024-03-23T11:29:15.669463+0000","last_unstale":"2024-03-23T11:29:16.649246+0000","last_undegraded":"2024-03-23T11:29:16.649246+0000","last_fullsized":"2024-03-23T11:29:16.649246+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:21:46.428250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077624+0000","last_change":"2024-03-23T11:29:15.673453+0000","last_active":"2024-03-23T11:29:17.077624+0000","last_peered":"2024-03-23T11:29:17.077624+0000","last_clean":"2024-03-23T11:29:17.077624+0000","last_became_active":"2024-03-23T11:29:15.672342+0000","last_became_peered":"2024-03-23T11:29:15.672342+0000","last_unstale":"2024-03-23T11:29:17.077624+0000","last_undegraded":"2024-03-23T11:29:17.077624+0000","last_fullsized":"2024-03-23T11:29:17.077624+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:15:37.630453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649271+0000","last_change":"2024-03-23T11:29:15.658308+0000","last_active":"2024-03-23T11:29:16.649271+0000","last_peered":"2024-03-23T11:29:16.649271+0000","last_clean":"2024-03-23T11:29:16.649271+0000","last_became_active":"2024-03-23T11:29:15.658030+0000","last_became_peered":"2024-03-23T11:29:15.658030+0000","last_unstale":"2024-03-23T11:29:16.649271+0000","last_undegraded":"2024-03-23T11:29:16.649271+0000","last_fullsized":"2024-03-23T11:29:16.649271+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:07:07.568405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079069+0000","last_change":"2024-03-23T11:29:15.651333+0000","last_active":"2024-03-23T11:29:17.079069+0000","last_peered":"2024-03-23T11:29:17.079069+0000","last_clean":"2024-03-23T11:29:17.079069+0000","last_became_active":"2024-03-23T11:29:15.651076+0000","last_became_peered":"2024-03-23T11:29:15.651076+0000","last_unstale":"2024-03-23T11:29:17.079069+0000","last_undegraded":"2024-03-23T11:29:17.079069+0000","last_fullsized":"2024-03-23T11:29:17.079069+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.902089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929345+0000","last_change":"2024-03-23T11:29:15.665566+0000","last_active":"2024-03-23T11:29:16.929345+0000","last_peered":"2024-03-23T11:29:16.929345+0000","last_clean":"2024-03-23T11:29:16.929345+0000","last_became_active":"2024-03-23T11:29:15.664851+0000","last_became_peered":"2024-03-23T11:29:15.664851+0000","last_unstale":"2024-03-23T11:29:16.929345+0000","last_undegraded":"2024-03-23T11:29:16.929345+0000","last_fullsized":"2024-03-23T11:29:16.929345+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:27.274279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649914+0000","last_change":"2024-03-23T11:29:15.663227+0000","last_active":"2024-03-23T11:29:16.649914+0000","last_peered":"2024-03-23T11:29:16.649914+0000","last_clean":"2024-03-23T11:29:16.649914+0000","last_became_active":"2024-03-23T11:29:15.661436+0000","last_became_peered":"2024-03-23T11:29:15.661436+0000","last_unstale":"2024-03-23T11:29:16.649914+0000","last_undegraded":"2024-03-23T11:29:16.649914+0000","last_fullsized":"2024-03-23T11:29:16.649914+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:20:09.497186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649956+0000","last_change":"2024-03-23T11:29:15.662335+0000","last_active":"2024-03-23T11:29:16.649956+0000","last_peered":"2024-03-23T11:29:16.649956+0000","last_clean":"2024-03-23T11:29:16.649956+0000","last_became_active":"2024-03-23T11:29:15.662022+0000","last_became_peered":"2024-03-23T11:29:15.662022+0000","last_unstale":"2024-03-23T11:29:16.649956+0000","last_undegraded":"2024-03-23T11:29:16.649956+0000","last_fullsized":"2024-03-23T11:29:16.649956+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:01:37.986646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077905+0000","last_change":"2024-03-23T11:29:15.669108+0000","last_active":"2024-03-23T11:29:17.077905+0000","last_peered":"2024-03-23T11:29:17.077905+0000","last_clean":"2024-03-23T11:29:17.077905+0000","last_became_active":"2024-03-23T11:29:15.668763+0000","last_became_peered":"2024-03-23T11:29:15.668763+0000","last_unstale":"2024-03-23T11:29:17.077905+0000","last_undegraded":"2024-03-23T11:29:17.077905+0000","last_fullsized":"2024-03-23T11:29:17.077905+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:25.131893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052620+0000","last_change":"2024-03-23T11:29:15.668474+0000","last_active":"2024-03-23T11:29:17.052620+0000","last_peered":"2024-03-23T11:29:17.052620+0000","last_clean":"2024-03-23T11:29:17.052620+0000","last_became_active":"2024-03-23T11:29:15.668114+0000","last_became_peered":"2024-03-23T11:29:15.668114+0000","last_unstale":"2024-03-23T11:29:17.052620+0000","last_undegraded":"2024-03-23T11:29:17.052620+0000","last_fullsized":"2024-03-23T11:29:17.052620+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:42:00.302069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929241+0000","last_change":"2024-03-23T11:29:15.666201+0000","last_active":"2024-03-23T11:29:16.929241+0000","last_peered":"2024-03-23T11:29:16.929241+0000","last_clean":"2024-03-23T11:29:16.929241+0000","last_became_active":"2024-03-23T11:29:15.665939+0000","last_became_peered":"2024-03-23T11:29:15.665939+0000","last_unstale":"2024-03-23T11:29:16.929241+0000","last_undegraded":"2024-03-23T11:29:16.929241+0000","last_fullsized":"2024-03-23T11:29:16.929241+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:03.126355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649783+0000","last_change":"2024-03-23T11:29:15.666875+0000","last_active":"2024-03-23T11:29:16.649783+0000","last_peered":"2024-03-23T11:29:16.649783+0000","last_clean":"2024-03-23T11:29:16.649783+0000","last_became_active":"2024-03-23T11:29:15.661755+0000","last_became_peered":"2024-03-23T11:29:15.661755+0000","last_unstale":"2024-03-23T11:29:16.649783+0000","last_undegraded":"2024-03-23T11:29:16.649783+0000","last_fullsized":"2024-03-23T11:29:16.649783+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:34:53.607989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052654+0000","last_change":"2024-03-23T11:29:15.660658+0000","last_active":"2024-03-23T11:29:17.052654+0000","last_peered":"2024-03-23T11:29:17.052654+0000","last_clean":"2024-03-23T11:29:17.052654+0000","last_became_active":"2024-03-23T11:29:15.658934+0000","last_became_peered":"2024-03-23T11:29:15.658934+0000","last_unstale":"2024-03-23T11:29:17.052654+0000","last_undegraded":"2024-03-23T11:29:17.052654+0000","last_fullsized":"2024-03-23T11:29:17.052654+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:21:17.856878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078618+0000","last_change":"2024-03-23T11:29:15.662724+0000","last_active":"2024-03-23T11:29:17.078618+0000","last_peered":"2024-03-23T11:29:17.078618+0000","last_clean":"2024-03-23T11:29:17.078618+0000","last_became_active":"2024-03-23T11:29:15.662420+0000","last_became_peered":"2024-03-23T11:29:15.662420+0000","last_unstale":"2024-03-23T11:29:17.078618+0000","last_undegraded":"2024-03-23T11:29:17.078618+0000","last_fullsized":"2024-03-23T11:29:17.078618+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:27.274279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076113+0000","last_change":"2024-03-23T11:29:15.664201+0000","last_active":"2024-03-23T11:29:17.076113+0000","last_peered":"2024-03-23T11:29:17.076113+0000","last_clean":"2024-03-23T11:29:17.076113+0000","last_became_active":"2024-03-23T11:29:15.663923+0000","last_became_peered":"2024-03-23T11:29:15.663923+0000","last_unstale":"2024-03-23T11:29:17.076113+0000","last_undegraded":"2024-03-23T11:29:17.076113+0000","last_fullsized":"2024-03-23T11:29:17.076113+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:46:09.117552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077065+0000","last_change":"2024-03-23T11:29:15.653031+0000","last_active":"2024-03-23T11:29:17.077065+0000","last_peered":"2024-03-23T11:29:17.077065+0000","last_clean":"2024-03-23T11:29:17.077065+0000","last_became_active":"2024-03-23T11:29:15.652677+0000","last_became_peered":"2024-03-23T11:29:15.652677+0000","last_unstale":"2024-03-23T11:29:17.077065+0000","last_undegraded":"2024-03-23T11:29:17.077065+0000","last_fullsized":"2024-03-23T11:29:17.077065+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:20:38.886782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649060+0000","last_change":"2024-03-23T11:29:15.658243+0000","last_active":"2024-03-23T11:29:16.649060+0000","last_peered":"2024-03-23T11:29:16.649060+0000","last_clean":"2024-03-23T11:29:16.649060+0000","last_became_active":"2024-03-23T11:29:15.657964+0000","last_became_peered":"2024-03-23T11:29:15.657964+0000","last_unstale":"2024-03-23T11:29:16.649060+0000","last_undegraded":"2024-03-23T11:29:16.649060+0000","last_fullsized":"2024-03-23T11:29:16.649060+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:56:40.286477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.082158+0000","last_change":"2024-03-23T11:29:15.666060+0000","last_active":"2024-03-23T11:29:17.082158+0000","last_peered":"2024-03-23T11:29:17.082158+0000","last_clean":"2024-03-23T11:29:17.082158+0000","last_became_active":"2024-03-23T11:29:15.665037+0000","last_became_peered":"2024-03-23T11:29:15.665037+0000","last_unstale":"2024-03-23T11:29:17.082158+0000","last_undegraded":"2024-03-23T11:29:17.082158+0000","last_fullsized":"2024-03-23T11:29:17.082158+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:23:08.931665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078923+0000","last_change":"2024-03-23T11:29:15.651649+0000","last_active":"2024-03-23T11:29:17.078923+0000","last_peered":"2024-03-23T11:29:17.078923+0000","last_clean":"2024-03-23T11:29:17.078923+0000","last_became_active":"2024-03-23T11:29:15.651296+0000","last_became_peered":"2024-03-23T11:29:15.651296+0000","last_unstale":"2024-03-23T11:29:17.078923+0000","last_undegraded":"2024-03-23T11:29:17.078923+0000","last_fullsized":"2024-03-23T11:29:17.078923+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:17.074421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079382+0000","last_change":"2024-03-23T11:29:15.652434+0000","last_active":"2024-03-23T11:29:17.079382+0000","last_peered":"2024-03-23T11:29:17.079382+0000","last_clean":"2024-03-23T11:29:17.079382+0000","last_became_active":"2024-03-23T11:29:15.652172+0000","last_became_peered":"2024-03-23T11:29:15.652172+0000","last_unstale":"2024-03-23T11:29:17.079382+0000","last_undegraded":"2024-03-23T11:29:17.079382+0000","last_fullsized":"2024-03-23T11:29:17.079382+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:19.474314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052560+0000","last_change":"2024-03-23T11:29:15.671106+0000","last_active":"2024-03-23T11:29:17.052560+0000","last_peered":"2024-03-23T11:29:17.052560+0000","last_clean":"2024-03-23T11:29:17.052560+0000","last_became_active":"2024-03-23T11:29:15.670839+0000","last_became_peered":"2024-03-23T11:29:15.670839+0000","last_unstale":"2024-03-23T11:29:17.052560+0000","last_undegraded":"2024-03-23T11:29:17.052560+0000","last_fullsized":"2024-03-23T11:29:17.052560+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:56:56.282431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076764+0000","last_change":"2024-03-23T11:29:15.659203+0000","last_active":"2024-03-23T11:29:17.076764+0000","last_peered":"2024-03-23T11:29:17.076764+0000","last_clean":"2024-03-23T11:29:17.076764+0000","last_became_active":"2024-03-23T11:29:15.658693+0000","last_became_peered":"2024-03-23T11:29:15.658693+0000","last_unstale":"2024-03-23T11:29:17.076764+0000","last_undegraded":"2024-03-23T11:29:17.076764+0000","last_fullsized":"2024-03-23T11:29:17.076764+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:32:17.491368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077429+0000","last_change":"2024-03-23T11:29:15.661680+0000","last_active":"2024-03-23T11:29:17.077429+0000","last_peered":"2024-03-23T11:29:17.077429+0000","last_clean":"2024-03-23T11:29:17.077429+0000","last_became_active":"2024-03-23T11:29:15.661421+0000","last_became_peered":"2024-03-23T11:29:15.661421+0000","last_unstale":"2024-03-23T11:29:17.077429+0000","last_undegraded":"2024-03-23T11:29:17.077429+0000","last_fullsized":"2024-03-23T11:29:17.077429+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:13:25.413831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079748+0000","last_change":"2024-03-23T11:29:15.673377+0000","last_active":"2024-03-23T11:29:17.079748+0000","last_peered":"2024-03-23T11:29:17.079748+0000","last_clean":"2024-03-23T11:29:17.079748+0000","last_became_active":"2024-03-23T11:29:15.673077+0000","last_became_peered":"2024-03-23T11:29:15.673077+0000","last_unstale":"2024-03-23T11:29:17.079748+0000","last_undegraded":"2024-03-23T11:29:17.079748+0000","last_fullsized":"2024-03-23T11:29:17.079748+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:55:04.251123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649131+0000","last_change":"2024-03-23T11:29:15.652916+0000","last_active":"2024-03-23T11:29:16.649131+0000","last_peered":"2024-03-23T11:29:16.649131+0000","last_clean":"2024-03-23T11:29:16.649131+0000","last_became_active":"2024-03-23T11:29:15.652566+0000","last_became_peered":"2024-03-23T11:29:15.652566+0000","last_unstale":"2024-03-23T11:29:16.649131+0000","last_undegraded":"2024-03-23T11:29:16.649131+0000","last_fullsized":"2024-03-23T11:29:16.649131+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:22:05.889747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929778+0000","last_change":"2024-03-23T11:29:15.673239+0000","last_active":"2024-03-23T11:29:16.929778+0000","last_peered":"2024-03-23T11:29:16.929778+0000","last_clean":"2024-03-23T11:29:16.929778+0000","last_became_active":"2024-03-23T11:29:15.671796+0000","last_became_peered":"2024-03-23T11:29:15.671796+0000","last_unstale":"2024-03-23T11:29:16.929778+0000","last_undegraded":"2024-03-23T11:29:16.929778+0000","last_fullsized":"2024-03-23T11:29:16.929778+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:17.074421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077900+0000","last_change":"2024-03-23T11:29:15.664158+0000","last_active":"2024-03-23T11:29:17.077900+0000","last_peered":"2024-03-23T11:29:17.077900+0000","last_clean":"2024-03-23T11:29:17.077900+0000","last_became_active":"2024-03-23T11:29:15.662983+0000","last_became_peered":"2024-03-23T11:29:15.662983+0000","last_unstale":"2024-03-23T11:29:17.077900+0000","last_undegraded":"2024-03-23T11:29:17.077900+0000","last_fullsized":"2024-03-23T11:29:17.077900+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:03:38.240579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649710+0000","last_change":"2024-03-23T11:29:15.658623+0000","last_active":"2024-03-23T11:29:16.649710+0000","last_peered":"2024-03-23T11:29:16.649710+0000","last_clean":"2024-03-23T11:29:16.649710+0000","last_became_active":"2024-03-23T11:29:15.656411+0000","last_became_peered":"2024-03-23T11:29:15.656411+0000","last_unstale":"2024-03-23T11:29:16.649710+0000","last_undegraded":"2024-03-23T11:29:16.649710+0000","last_fullsized":"2024-03-23T11:29:16.649710+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:50:49.301705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929338+0000","last_change":"2024-03-23T11:29:15.670061+0000","last_active":"2024-03-23T11:29:16.929338+0000","last_peered":"2024-03-23T11:29:16.929338+0000","last_clean":"2024-03-23T11:29:16.929338+0000","last_became_active":"2024-03-23T11:29:15.669663+0000","last_became_peered":"2024-03-23T11:29:15.669663+0000","last_unstale":"2024-03-23T11:29:16.929338+0000","last_undegraded":"2024-03-23T11:29:16.929338+0000","last_fullsized":"2024-03-23T11:29:16.929338+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:50:21.272634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079978+0000","last_change":"2024-03-23T11:29:15.666505+0000","last_active":"2024-03-23T11:29:17.079978+0000","last_peered":"2024-03-23T11:29:17.079978+0000","last_clean":"2024-03-23T11:29:17.079978+0000","last_became_active":"2024-03-23T11:29:15.666264+0000","last_became_peered":"2024-03-23T11:29:15.666264+0000","last_unstale":"2024-03-23T11:29:17.079978+0000","last_undegraded":"2024-03-23T11:29:17.079978+0000","last_fullsized":"2024-03-23T11:29:17.079978+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:13:07.960167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076589+0000","last_change":"2024-03-23T11:29:15.651998+0000","last_active":"2024-03-23T11:29:17.076589+0000","last_peered":"2024-03-23T11:29:17.076589+0000","last_clean":"2024-03-23T11:29:17.076589+0000","last_became_active":"2024-03-23T11:29:15.650650+0000","last_became_peered":"2024-03-23T11:29:15.650650+0000","last_unstale":"2024-03-23T11:29:17.076589+0000","last_undegraded":"2024-03-23T11:29:17.076589+0000","last_fullsized":"2024-03-23T11:29:17.076589+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:59:16.956887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077760+0000","last_change":"2024-03-23T11:29:15.668910+0000","last_active":"2024-03-23T11:29:17.077760+0000","last_peered":"2024-03-23T11:29:17.077760+0000","last_clean":"2024-03-23T11:29:17.077760+0000","last_became_active":"2024-03-23T11:29:15.668357+0000","last_became_peered":"2024-03-23T11:29:15.668357+0000","last_unstale":"2024-03-23T11:29:17.077760+0000","last_undegraded":"2024-03-23T11:29:17.077760+0000","last_fullsized":"2024-03-23T11:29:17.077760+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:22:40.156625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649018+0000","last_change":"2024-03-23T11:29:15.657660+0000","last_active":"2024-03-23T11:29:16.649018+0000","last_peered":"2024-03-23T11:29:16.649018+0000","last_clean":"2024-03-23T11:29:16.649018+0000","last_became_active":"2024-03-23T11:29:15.656966+0000","last_became_peered":"2024-03-23T11:29:15.656966+0000","last_unstale":"2024-03-23T11:29:16.649018+0000","last_undegraded":"2024-03-23T11:29:16.649018+0000","last_fullsized":"2024-03-23T11:29:16.649018+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:56.613315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079556+0000","last_change":"2024-03-23T11:29:15.654194+0000","last_active":"2024-03-23T11:29:17.079556+0000","last_peered":"2024-03-23T11:29:17.079556+0000","last_clean":"2024-03-23T11:29:17.079556+0000","last_became_active":"2024-03-23T11:29:15.653914+0000","last_became_peered":"2024-03-23T11:29:15.653914+0000","last_unstale":"2024-03-23T11:29:17.079556+0000","last_undegraded":"2024-03-23T11:29:17.079556+0000","last_fullsized":"2024-03-23T11:29:17.079556+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:17:39.719198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"104'4","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.082180+0000","last_change":"2024-03-23T11:29:15.167697+0000","last_active":"2024-03-23T11:29:17.082180+0000","last_peered":"2024-03-23T11:29:17.082180+0000","last_clean":"2024-03-23T11:29:17.082180+0000","last_became_active":"2024-03-23T11:29:15.167198+0000","last_became_peered":"2024-03-23T11:29:15.167198+0000","last_unstale":"2024-03-23T11:29:17.082180+0000","last_undegraded":"2024-03-23T11:29:17.082180+0000","last_fullsized":"2024-03-23T11:29:17.082180+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:24:59.167502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.080175+0000","last_change":"2024-03-23T11:29:15.668009+0000","last_active":"2024-03-23T11:29:17.080175+0000","last_peered":"2024-03-23T11:29:17.080175+0000","last_clean":"2024-03-23T11:29:17.080175+0000","last_became_active":"2024-03-23T11:29:15.667751+0000","last_became_peered":"2024-03-23T11:29:15.667751+0000","last_unstale":"2024-03-23T11:29:17.080175+0000","last_undegraded":"2024-03-23T11:29:17.080175+0000","last_fullsized":"2024-03-23T11:29:17.080175+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:50:21.272634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"104'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.650334+0000","last_change":"2024-03-23T11:29:15.163279+0000","last_active":"2024-03-23T11:29:16.650334+0000","last_peered":"2024-03-23T11:29:16.650334+0000","last_clean":"2024-03-23T11:29:16.650334+0000","last_became_active":"2024-03-23T11:29:15.162911+0000","last_became_peered":"2024-03-23T11:29:15.162911+0000","last_unstale":"2024-03-23T11:29:16.650334+0000","last_undegraded":"2024-03-23T11:29:16.650334+0000","last_fullsized":"2024-03-23T11:29:16.650334+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:51:23.085524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.648986+0000","last_change":"2024-03-23T11:29:15.655253+0000","last_active":"2024-03-23T11:29:16.648986+0000","last_peered":"2024-03-23T11:29:16.648986+0000","last_clean":"2024-03-23T11:29:16.648986+0000","last_became_active":"2024-03-23T11:29:15.654942+0000","last_became_peered":"2024-03-23T11:29:15.654942+0000","last_unstale":"2024-03-23T11:29:16.648986+0000","last_undegraded":"2024-03-23T11:29:16.648986+0000","last_fullsized":"2024-03-23T11:29:16.648986+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:13:19.425483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.081531+0000","last_change":"2024-03-23T11:29:15.168589+0000","last_active":"2024-03-23T11:29:17.081531+0000","last_peered":"2024-03-23T11:29:17.081531+0000","last_clean":"2024-03-23T11:29:17.081531+0000","last_became_active":"2024-03-23T11:29:15.166723+0000","last_became_peered":"2024-03-23T11:29:15.166723+0000","last_unstale":"2024-03-23T11:29:17.081531+0000","last_undegraded":"2024-03-23T11:29:17.081531+0000","last_fullsized":"2024-03-23T11:29:17.081531+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:46.791548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052889+0000","last_change":"2024-03-23T11:29:15.659552+0000","last_active":"2024-03-23T11:29:17.052889+0000","last_peered":"2024-03-23T11:29:17.052889+0000","last_clean":"2024-03-23T11:29:17.052889+0000","last_became_active":"2024-03-23T11:29:15.659196+0000","last_became_peered":"2024-03-23T11:29:15.659196+0000","last_unstale":"2024-03-23T11:29:17.052889+0000","last_undegraded":"2024-03-23T11:29:17.052889+0000","last_fullsized":"2024-03-23T11:29:17.052889+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:03.126355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.080571+0000","last_change":"2024-03-23T11:29:15.167939+0000","last_active":"2024-03-23T11:29:17.080571+0000","last_peered":"2024-03-23T11:29:17.080571+0000","last_clean":"2024-03-23T11:29:17.080571+0000","last_became_active":"2024-03-23T11:29:15.165542+0000","last_became_peered":"2024-03-23T11:29:15.165542+0000","last_unstale":"2024-03-23T11:29:17.080571+0000","last_undegraded":"2024-03-23T11:29:17.080571+0000","last_fullsized":"2024-03-23T11:29:17.080571+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:27:48.629258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.648411+0000","last_change":"2024-03-23T11:29:15.667879+0000","last_active":"2024-03-23T11:29:16.648411+0000","last_peered":"2024-03-23T11:29:16.648411+0000","last_clean":"2024-03-23T11:29:16.648411+0000","last_became_active":"2024-03-23T11:29:15.666726+0000","last_became_peered":"2024-03-23T11:29:15.666726+0000","last_unstale":"2024-03-23T11:29:16.648411+0000","last_undegraded":"2024-03-23T11:29:16.648411+0000","last_fullsized":"2024-03-23T11:29:16.648411+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:03.126355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"104'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649014+0000","last_change":"2024-03-23T11:29:15.170048+0000","last_active":"2024-03-23T11:29:16.649014+0000","last_peered":"2024-03-23T11:29:16.649014+0000","last_clean":"2024-03-23T11:29:16.649014+0000","last_became_active":"2024-03-23T11:29:15.169324+0000","last_became_peered":"2024-03-23T11:29:15.169324+0000","last_unstale":"2024-03-23T11:29:16.649014+0000","last_undegraded":"2024-03-23T11:29:16.649014+0000","last_fullsized":"2024-03-23T11:29:16.649014+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:19:12.246415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.650011+0000","last_change":"2024-03-23T11:29:15.664162+0000","last_active":"2024-03-23T11:29:16.650011+0000","last_peered":"2024-03-23T11:29:16.650011+0000","last_clean":"2024-03-23T11:29:16.650011+0000","last_became_active":"2024-03-23T11:29:15.663884+0000","last_became_peered":"2024-03-23T11:29:15.663884+0000","last_unstale":"2024-03-23T11:29:16.650011+0000","last_undegraded":"2024-03-23T11:29:16.650011+0000","last_fullsized":"2024-03-23T11:29:16.650011+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:55:14.250603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"104'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077570+0000","last_change":"2024-03-23T11:29:15.167254+0000","last_active":"2024-03-23T11:29:17.077570+0000","last_peered":"2024-03-23T11:29:17.077570+0000","last_clean":"2024-03-23T11:29:17.077570+0000","last_became_active":"2024-03-23T11:29:15.165937+0000","last_became_peered":"2024-03-23T11:29:15.165937+0000","last_unstale":"2024-03-23T11:29:17.077570+0000","last_undegraded":"2024-03-23T11:29:17.077570+0000","last_fullsized":"2024-03-23T11:29:17.077570+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:02.110388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077030+0000","last_change":"2024-03-23T11:29:15.652851+0000","last_active":"2024-03-23T11:29:17.077030+0000","last_peered":"2024-03-23T11:29:17.077030+0000","last_clean":"2024-03-23T11:29:17.077030+0000","last_became_active":"2024-03-23T11:29:15.652536+0000","last_became_peered":"2024-03-23T11:29:15.652536+0000","last_unstale":"2024-03-23T11:29:17.077030+0000","last_undegraded":"2024-03-23T11:29:17.077030+0000","last_fullsized":"2024-03-23T11:29:17.077030+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:44.511761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929189+0000","last_change":"2024-03-23T11:29:15.171133+0000","last_active":"2024-03-23T11:29:16.929189+0000","last_peered":"2024-03-23T11:29:16.929189+0000","last_clean":"2024-03-23T11:29:16.929189+0000","last_became_active":"2024-03-23T11:29:15.170830+0000","last_became_peered":"2024-03-23T11:29:15.170830+0000","last_unstale":"2024-03-23T11:29:16.929189+0000","last_undegraded":"2024-03-23T11:29:16.929189+0000","last_fullsized":"2024-03-23T11:29:16.929189+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:36:01.499746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.051906+0000","last_change":"2024-03-23T11:29:15.660248+0000","last_active":"2024-03-23T11:29:17.051906+0000","last_peered":"2024-03-23T11:29:17.051906+0000","last_clean":"2024-03-23T11:29:17.051906+0000","last_became_active":"2024-03-23T11:29:15.659942+0000","last_became_peered":"2024-03-23T11:29:15.659942+0000","last_unstale":"2024-03-23T11:29:17.051906+0000","last_undegraded":"2024-03-23T11:29:17.051906+0000","last_fullsized":"2024-03-23T11:29:17.051906+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:22.855663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649166+0000","last_change":"2024-03-23T11:29:15.159736+0000","last_active":"2024-03-23T11:29:16.649166+0000","last_peered":"2024-03-23T11:29:16.649166+0000","last_clean":"2024-03-23T11:29:16.649166+0000","last_became_active":"2024-03-23T11:29:15.159171+0000","last_became_peered":"2024-03-23T11:29:15.159171+0000","last_unstale":"2024-03-23T11:29:16.649166+0000","last_undegraded":"2024-03-23T11:29:16.649166+0000","last_fullsized":"2024-03-23T11:29:16.649166+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:44:57.248852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078147+0000","last_change":"2024-03-23T11:29:15.662195+0000","last_active":"2024-03-23T11:29:17.078147+0000","last_peered":"2024-03-23T11:29:17.078147+0000","last_clean":"2024-03-23T11:29:17.078147+0000","last_became_active":"2024-03-23T11:29:15.661948+0000","last_became_peered":"2024-03-23T11:29:15.661948+0000","last_unstale":"2024-03-23T11:29:17.078147+0000","last_undegraded":"2024-03-23T11:29:17.078147+0000","last_fullsized":"2024-03-23T11:29:17.078147+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:22:25.026968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649307+0000","last_change":"2024-03-23T11:29:15.161141+0000","last_active":"2024-03-23T11:29:16.649307+0000","last_peered":"2024-03-23T11:29:16.649307+0000","last_clean":"2024-03-23T11:29:16.649307+0000","last_became_active":"2024-03-23T11:29:15.160884+0000","last_became_peered":"2024-03-23T11:29:15.160884+0000","last_unstale":"2024-03-23T11:29:16.649307+0000","last_undegraded":"2024-03-23T11:29:16.649307+0000","last_fullsized":"2024-03-23T11:29:16.649307+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:11.396958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052772+0000","last_change":"2024-03-23T11:29:15.673901+0000","last_active":"2024-03-23T11:29:17.052772+0000","last_peered":"2024-03-23T11:29:17.052772+0000","last_clean":"2024-03-23T11:29:17.052772+0000","last_became_active":"2024-03-23T11:29:15.673013+0000","last_became_peered":"2024-03-23T11:29:15.673013+0000","last_unstale":"2024-03-23T11:29:17.052772+0000","last_undegraded":"2024-03-23T11:29:17.052772+0000","last_fullsized":"2024-03-23T11:29:17.052772+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:00:56.194551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079165+0000","last_change":"2024-03-23T11:29:15.164764+0000","last_active":"2024-03-23T11:29:17.079165+0000","last_peered":"2024-03-23T11:29:17.079165+0000","last_clean":"2024-03-23T11:29:17.079165+0000","last_became_active":"2024-03-23T11:29:15.163778+0000","last_became_peered":"2024-03-23T11:29:15.163778+0000","last_unstale":"2024-03-23T11:29:17.079165+0000","last_undegraded":"2024-03-23T11:29:17.079165+0000","last_fullsized":"2024-03-23T11:29:17.079165+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:31:55.748276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929396+0000","last_change":"2024-03-23T11:29:15.671469+0000","last_active":"2024-03-23T11:29:16.929396+0000","last_peered":"2024-03-23T11:29:16.929396+0000","last_clean":"2024-03-23T11:29:16.929396+0000","last_became_active":"2024-03-23T11:29:15.669339+0000","last_became_peered":"2024-03-23T11:29:15.669339+0000","last_unstale":"2024-03-23T11:29:16.929396+0000","last_undegraded":"2024-03-23T11:29:16.929396+0000","last_fullsized":"2024-03-23T11:29:16.929396+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:22:40.156625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"104'3","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.650069+0000","last_change":"2024-03-23T11:29:15.161417+0000","last_active":"2024-03-23T11:29:16.650069+0000","last_peered":"2024-03-23T11:29:16.650069+0000","last_clean":"2024-03-23T11:29:16.650069+0000","last_became_active":"2024-03-23T11:29:15.161009+0000","last_became_peered":"2024-03-23T11:29:15.161009+0000","last_unstale":"2024-03-23T11:29:16.650069+0000","last_undegraded":"2024-03-23T11:29:16.650069+0000","last_fullsized":"2024-03-23T11:29:16.650069+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:15:46.288971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649866+0000","last_change":"2024-03-23T11:29:15.656957+0000","last_active":"2024-03-23T11:29:16.649866+0000","last_peered":"2024-03-23T11:29:16.649866+0000","last_clean":"2024-03-23T11:29:16.649866+0000","last_became_active":"2024-03-23T11:29:15.655820+0000","last_became_peered":"2024-03-23T11:29:15.655820+0000","last_unstale":"2024-03-23T11:29:16.649866+0000","last_undegraded":"2024-03-23T11:29:16.649866+0000","last_fullsized":"2024-03-23T11:29:16.649866+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:33:06.699830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"104'6","reported_seq":17,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052814+0000","last_change":"2024-03-23T11:29:15.162213+0000","last_active":"2024-03-23T11:29:17.052814+0000","last_peered":"2024-03-23T11:29:17.052814+0000","last_clean":"2024-03-23T11:29:17.052814+0000","last_became_active":"2024-03-23T11:29:15.161871+0000","last_became_peered":"2024-03-23T11:29:15.161871+0000","last_unstale":"2024-03-23T11:29:17.052814+0000","last_undegraded":"2024-03-23T11:29:17.052814+0000","last_fullsized":"2024-03-23T11:29:17.052814+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:40:27.480724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078197+0000","last_change":"2024-03-23T11:29:15.664516+0000","last_active":"2024-03-23T11:29:17.078197+0000","last_peered":"2024-03-23T11:29:17.078197+0000","last_clean":"2024-03-23T11:29:17.078197+0000","last_became_active":"2024-03-23T11:29:15.661188+0000","last_became_peered":"2024-03-23T11:29:15.661188+0000","last_unstale":"2024-03-23T11:29:17.078197+0000","last_undegraded":"2024-03-23T11:29:17.078197+0000","last_fullsized":"2024-03-23T11:29:17.078197+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:27:49.645225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078851+0000","last_change":"2024-03-23T11:29:15.162980+0000","last_active":"2024-03-23T11:29:17.078851+0000","last_peered":"2024-03-23T11:29:17.078851+0000","last_clean":"2024-03-23T11:29:17.078851+0000","last_became_active":"2024-03-23T11:29:15.161438+0000","last_became_peered":"2024-03-23T11:29:15.161438+0000","last_unstale":"2024-03-23T11:29:17.078851+0000","last_undegraded":"2024-03-23T11:29:17.078851+0000","last_fullsized":"2024-03-23T11:29:17.078851+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:57:53.683724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649432+0000","last_change":"2024-03-23T11:29:15.646405+0000","last_active":"2024-03-23T11:29:16.649432+0000","last_peered":"2024-03-23T11:29:16.649432+0000","last_clean":"2024-03-23T11:29:16.649432+0000","last_became_active":"2024-03-23T11:29:15.645120+0000","last_became_peered":"2024-03-23T11:29:15.645120+0000","last_unstale":"2024-03-23T11:29:16.649432+0000","last_undegraded":"2024-03-23T11:29:16.649432+0000","last_fullsized":"2024-03-23T11:29:16.649432+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:22:51.383135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077114+0000","last_change":"2024-03-23T11:29:15.165402+0000","last_active":"2024-03-23T11:29:17.077114+0000","last_peered":"2024-03-23T11:29:17.077114+0000","last_clean":"2024-03-23T11:29:17.077114+0000","last_became_active":"2024-03-23T11:29:15.165117+0000","last_became_peered":"2024-03-23T11:29:15.165117+0000","last_unstale":"2024-03-23T11:29:17.077114+0000","last_undegraded":"2024-03-23T11:29:17.077114+0000","last_fullsized":"2024-03-23T11:29:17.077114+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:08:22.178636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.051916+0000","last_change":"2024-03-23T11:29:15.659027+0000","last_active":"2024-03-23T11:29:17.051916+0000","last_peered":"2024-03-23T11:29:17.051916+0000","last_clean":"2024-03-23T11:29:17.051916+0000","last_became_active":"2024-03-23T11:29:15.658730+0000","last_became_peered":"2024-03-23T11:29:15.658730+0000","last_unstale":"2024-03-23T11:29:17.051916+0000","last_undegraded":"2024-03-23T11:29:17.051916+0000","last_fullsized":"2024-03-23T11:29:17.051916+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:25:00.183469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052089+0000","last_change":"2024-03-23T11:29:15.166013+0000","last_active":"2024-03-23T11:29:17.052089+0000","last_peered":"2024-03-23T11:29:17.052089+0000","last_clean":"2024-03-23T11:29:17.052089+0000","last_became_active":"2024-03-23T11:29:15.165701+0000","last_became_peered":"2024-03-23T11:29:15.165701+0000","last_unstale":"2024-03-23T11:29:17.052089+0000","last_undegraded":"2024-03-23T11:29:17.052089+0000","last_fullsized":"2024-03-23T11:29:17.052089+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:26.319833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076103+0000","last_change":"2024-03-23T11:29:15.663722+0000","last_active":"2024-03-23T11:29:17.076103+0000","last_peered":"2024-03-23T11:29:17.076103+0000","last_clean":"2024-03-23T11:29:17.076103+0000","last_became_active":"2024-03-23T11:29:15.663438+0000","last_became_peered":"2024-03-23T11:29:15.663438+0000","last_unstale":"2024-03-23T11:29:17.076103+0000","last_undegraded":"2024-03-23T11:29:17.076103+0000","last_fullsized":"2024-03-23T11:29:17.076103+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:53:36.693645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"104'8","reported_seq":19,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649292+0000","last_change":"2024-03-23T11:29:15.146975+0000","last_active":"2024-03-23T11:29:16.649292+0000","last_peered":"2024-03-23T11:29:16.649292+0000","last_clean":"2024-03-23T11:29:16.649292+0000","last_became_active":"2024-03-23T11:29:15.146669+0000","last_became_peered":"2024-03-23T11:29:15.146669+0000","last_unstale":"2024-03-23T11:29:16.649292+0000","last_undegraded":"2024-03-23T11:29:16.649292+0000","last_fullsized":"2024-03-23T11:29:16.649292+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:18:47.004845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929851+0000","last_change":"2024-03-23T11:29:15.668120+0000","last_active":"2024-03-23T11:29:16.929851+0000","last_peered":"2024-03-23T11:29:16.929851+0000","last_clean":"2024-03-23T11:29:16.929851+0000","last_became_active":"2024-03-23T11:29:15.667698+0000","last_became_peered":"2024-03-23T11:29:15.667698+0000","last_unstale":"2024-03-23T11:29:16.929851+0000","last_undegraded":"2024-03-23T11:29:16.929851+0000","last_fullsized":"2024-03-23T11:29:16.929851+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:30:24.902089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052562+0000","last_change":"2024-03-23T11:29:15.160035+0000","last_active":"2024-03-23T11:29:17.052562+0000","last_peered":"2024-03-23T11:29:17.052562+0000","last_clean":"2024-03-23T11:29:17.052562+0000","last_became_active":"2024-03-23T11:29:15.159640+0000","last_became_peered":"2024-03-23T11:29:15.159640+0000","last_unstale":"2024-03-23T11:29:17.052562+0000","last_undegraded":"2024-03-23T11:29:17.052562+0000","last_fullsized":"2024-03-23T11:29:17.052562+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:31:04.617758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649027+0000","last_change":"2024-03-23T11:29:15.672165+0000","last_active":"2024-03-23T11:29:16.649027+0000","last_peered":"2024-03-23T11:29:16.649027+0000","last_clean":"2024-03-23T11:29:16.649027+0000","last_became_active":"2024-03-23T11:29:15.670772+0000","last_became_peered":"2024-03-23T11:29:15.670772+0000","last_unstale":"2024-03-23T11:29:16.649027+0000","last_undegraded":"2024-03-23T11:29:16.649027+0000","last_fullsized":"2024-03-23T11:29:16.649027+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:56:56.282431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"104'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929678+0000","last_change":"2024-03-23T11:29:15.164914+0000","last_active":"2024-03-23T11:29:16.929678+0000","last_peered":"2024-03-23T11:29:16.929678+0000","last_clean":"2024-03-23T11:29:16.929678+0000","last_became_active":"2024-03-23T11:29:15.164608+0000","last_became_peered":"2024-03-23T11:29:15.164608+0000","last_unstale":"2024-03-23T11:29:16.929678+0000","last_undegraded":"2024-03-23T11:29:16.929678+0000","last_fullsized":"2024-03-23T11:29:16.929678+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:19:57.036431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.650070+0000","last_change":"2024-03-23T11:29:15.668131+0000","last_active":"2024-03-23T11:29:16.650070+0000","last_peered":"2024-03-23T11:29:16.650070+0000","last_clean":"2024-03-23T11:29:16.650070+0000","last_became_active":"2024-03-23T11:29:15.666919+0000","last_became_peered":"2024-03-23T11:29:15.666919+0000","last_unstale":"2024-03-23T11:29:16.650070+0000","last_undegraded":"2024-03-23T11:29:16.650070+0000","last_fullsized":"2024-03-23T11:29:16.650070+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:18:50.991323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.080780+0000","last_change":"2024-03-23T11:29:15.167426+0000","last_active":"2024-03-23T11:29:17.080780+0000","last_peered":"2024-03-23T11:29:17.080780+0000","last_clean":"2024-03-23T11:29:17.080780+0000","last_became_active":"2024-03-23T11:29:15.165774+0000","last_became_peered":"2024-03-23T11:29:15.165774+0000","last_unstale":"2024-03-23T11:29:17.080780+0000","last_undegraded":"2024-03-23T11:29:17.080780+0000","last_fullsized":"2024-03-23T11:29:17.080780+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:41:59.286102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076874+0000","last_change":"2024-03-23T11:29:15.654221+0000","last_active":"2024-03-23T11:29:17.076874+0000","last_peered":"2024-03-23T11:29:17.076874+0000","last_clean":"2024-03-23T11:29:17.076874+0000","last_became_active":"2024-03-23T11:29:15.653905+0000","last_became_peered":"2024-03-23T11:29:15.653905+0000","last_unstale":"2024-03-23T11:29:17.076874+0000","last_undegraded":"2024-03-23T11:29:17.076874+0000","last_fullsized":"2024-03-23T11:29:17.076874+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:26:06.732162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.080087+0000","last_change":"2024-03-23T11:29:15.167420+0000","last_active":"2024-03-23T11:29:17.080087+0000","last_peered":"2024-03-23T11:29:17.080087+0000","last_clean":"2024-03-23T11:29:17.080087+0000","last_became_active":"2024-03-23T11:29:15.167065+0000","last_became_peered":"2024-03-23T11:29:15.167065+0000","last_unstale":"2024-03-23T11:29:17.080087+0000","last_undegraded":"2024-03-23T11:29:17.080087+0000","last_fullsized":"2024-03-23T11:29:17.080087+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:26:01.038104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077577+0000","last_change":"2024-03-23T11:29:15.667475+0000","last_active":"2024-03-23T11:29:17.077577+0000","last_peered":"2024-03-23T11:29:17.077577+0000","last_clean":"2024-03-23T11:29:17.077577+0000","last_became_active":"2024-03-23T11:29:15.658307+0000","last_became_peered":"2024-03-23T11:29:15.658307+0000","last_unstale":"2024-03-23T11:29:17.077577+0000","last_undegraded":"2024-03-23T11:29:17.077577+0000","last_fullsized":"2024-03-23T11:29:17.077577+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:10:36.931539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.648477+0000","last_change":"2024-03-23T11:29:15.161859+0000","last_active":"2024-03-23T11:29:16.648477+0000","last_peered":"2024-03-23T11:29:16.648477+0000","last_clean":"2024-03-23T11:29:16.648477+0000","last_became_active":"2024-03-23T11:29:15.161564+0000","last_became_peered":"2024-03-23T11:29:15.161564+0000","last_unstale":"2024-03-23T11:29:16.648477+0000","last_undegraded":"2024-03-23T11:29:16.648477+0000","last_fullsized":"2024-03-23T11:29:16.648477+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:26:01.038104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076109+0000","last_change":"2024-03-23T11:29:15.648783+0000","last_active":"2024-03-23T11:29:17.076109+0000","last_peered":"2024-03-23T11:29:17.076109+0000","last_clean":"2024-03-23T11:29:17.076109+0000","last_became_active":"2024-03-23T11:29:15.648519+0000","last_became_peered":"2024-03-23T11:29:15.648519+0000","last_unstale":"2024-03-23T11:29:17.076109+0000","last_undegraded":"2024-03-23T11:29:17.076109+0000","last_fullsized":"2024-03-23T11:29:17.076109+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:28:49.056332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.648180+0000","last_change":"2024-03-23T11:29:15.157608+0000","last_active":"2024-03-23T11:29:16.648180+0000","last_peered":"2024-03-23T11:29:16.648180+0000","last_clean":"2024-03-23T11:29:16.648180+0000","last_became_active":"2024-03-23T11:29:15.157259+0000","last_became_peered":"2024-03-23T11:29:15.157259+0000","last_unstale":"2024-03-23T11:29:16.648180+0000","last_undegraded":"2024-03-23T11:29:16.648180+0000","last_fullsized":"2024-03-23T11:29:16.648180+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:19:57.036431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.928916+0000","last_change":"2024-03-23T11:29:15.664280+0000","last_active":"2024-03-23T11:29:16.928916+0000","last_peered":"2024-03-23T11:29:16.928916+0000","last_clean":"2024-03-23T11:29:16.928916+0000","last_became_active":"2024-03-23T11:29:15.663918+0000","last_became_peered":"2024-03-23T11:29:15.663918+0000","last_unstale":"2024-03-23T11:29:16.928916+0000","last_undegraded":"2024-03-23T11:29:16.928916+0000","last_fullsized":"2024-03-23T11:29:16.928916+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:44.511761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.078124+0000","last_change":"2024-03-23T11:29:15.171195+0000","last_active":"2024-03-23T11:29:17.078124+0000","last_peered":"2024-03-23T11:29:17.078124+0000","last_clean":"2024-03-23T11:29:17.078124+0000","last_became_active":"2024-03-23T11:29:15.168383+0000","last_became_peered":"2024-03-23T11:29:15.168383+0000","last_unstale":"2024-03-23T11:29:17.078124+0000","last_undegraded":"2024-03-23T11:29:17.078124+0000","last_fullsized":"2024-03-23T11:29:17.078124+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:00:55.178584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.082050+0000","last_change":"2024-03-23T11:29:15.665043+0000","last_active":"2024-03-23T11:29:17.082050+0000","last_peered":"2024-03-23T11:29:17.082050+0000","last_clean":"2024-03-23T11:29:17.082050+0000","last_became_active":"2024-03-23T11:29:15.663990+0000","last_became_peered":"2024-03-23T11:29:15.663990+0000","last_unstale":"2024-03-23T11:29:17.082050+0000","last_undegraded":"2024-03-23T11:29:17.082050+0000","last_fullsized":"2024-03-23T11:29:17.082050+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:13.398610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"104'2","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649462+0000","last_change":"2024-03-23T11:29:15.160347+0000","last_active":"2024-03-23T11:29:16.649462+0000","last_peered":"2024-03-23T11:29:16.649462+0000","last_clean":"2024-03-23T11:29:16.649462+0000","last_became_active":"2024-03-23T11:29:15.160035+0000","last_became_peered":"2024-03-23T11:29:15.160035+0000","last_unstale":"2024-03-23T11:29:16.649462+0000","last_undegraded":"2024-03-23T11:29:16.649462+0000","last_fullsized":"2024-03-23T11:29:16.649462+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:31:55.748276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649218+0000","last_change":"2024-03-23T11:29:15.644839+0000","last_active":"2024-03-23T11:29:16.649218+0000","last_peered":"2024-03-23T11:29:16.649218+0000","last_clean":"2024-03-23T11:29:16.649218+0000","last_became_active":"2024-03-23T11:29:15.644500+0000","last_became_peered":"2024-03-23T11:29:15.644500+0000","last_unstale":"2024-03-23T11:29:16.649218+0000","last_undegraded":"2024-03-23T11:29:16.649218+0000","last_fullsized":"2024-03-23T11:29:16.649218+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:18.726617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.082131+0000","last_change":"2024-03-23T11:29:15.162600+0000","last_active":"2024-03-23T11:29:17.082131+0000","last_peered":"2024-03-23T11:29:17.082131+0000","last_clean":"2024-03-23T11:29:17.082131+0000","last_became_active":"2024-03-23T11:29:15.162335+0000","last_became_peered":"2024-03-23T11:29:15.162335+0000","last_unstale":"2024-03-23T11:29:17.082131+0000","last_undegraded":"2024-03-23T11:29:17.082131+0000","last_fullsized":"2024-03-23T11:29:17.082131+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:21.839696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.649072+0000","last_change":"2024-03-23T11:29:15.670658+0000","last_active":"2024-03-23T11:29:16.649072+0000","last_peered":"2024-03-23T11:29:16.649072+0000","last_clean":"2024-03-23T11:29:16.649072+0000","last_became_active":"2024-03-23T11:29:15.670379+0000","last_became_peered":"2024-03-23T11:29:15.670379+0000","last_unstale":"2024-03-23T11:29:16.649072+0000","last_undegraded":"2024-03-23T11:29:16.649072+0000","last_fullsized":"2024-03-23T11:29:16.649072+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:33:29.272656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.081308+0000","last_change":"2024-03-23T11:29:15.166452+0000","last_active":"2024-03-23T11:29:17.081308+0000","last_peered":"2024-03-23T11:29:17.081308+0000","last_clean":"2024-03-23T11:29:17.081308+0000","last_became_active":"2024-03-23T11:29:15.166192+0000","last_became_peered":"2024-03-23T11:29:15.166192+0000","last_unstale":"2024-03-23T11:29:17.081308+0000","last_undegraded":"2024-03-23T11:29:17.081308+0000","last_fullsized":"2024-03-23T11:29:17.081308+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:21:16.840911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929649+0000","last_change":"2024-03-23T11:29:15.670394+0000","last_active":"2024-03-23T11:29:16.929649+0000","last_peered":"2024-03-23T11:29:16.929649+0000","last_clean":"2024-03-23T11:29:16.929649+0000","last_became_active":"2024-03-23T11:29:15.669874+0000","last_became_peered":"2024-03-23T11:29:15.669874+0000","last_unstale":"2024-03-23T11:29:16.929649+0000","last_undegraded":"2024-03-23T11:29:16.929649+0000","last_fullsized":"2024-03-23T11:29:16.929649+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:42:00.302069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.929594+0000","last_change":"2024-03-23T11:29:15.165222+0000","last_active":"2024-03-23T11:29:16.929594+0000","last_peered":"2024-03-23T11:29:16.929594+0000","last_clean":"2024-03-23T11:29:16.929594+0000","last_became_active":"2024-03-23T11:29:15.164799+0000","last_became_peered":"2024-03-23T11:29:15.164799+0000","last_unstale":"2024-03-23T11:29:16.929594+0000","last_undegraded":"2024-03-23T11:29:16.929594+0000","last_fullsized":"2024-03-23T11:29:16.929594+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:49:27.211897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077150+0000","last_change":"2024-03-23T11:29:15.660807+0000","last_active":"2024-03-23T11:29:17.077150+0000","last_peered":"2024-03-23T11:29:17.077150+0000","last_clean":"2024-03-23T11:29:17.077150+0000","last_became_active":"2024-03-23T11:29:15.660489+0000","last_became_peered":"2024-03-23T11:29:15.660489+0000","last_unstale":"2024-03-23T11:29:17.077150+0000","last_undegraded":"2024-03-23T11:29:17.077150+0000","last_fullsized":"2024-03-23T11:29:17.077150+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:24:05.630578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.080180+0000","last_change":"2024-03-23T11:29:15.164486+0000","last_active":"2024-03-23T11:29:17.080180+0000","last_peered":"2024-03-23T11:29:17.080180+0000","last_clean":"2024-03-23T11:29:17.080180+0000","last_became_active":"2024-03-23T11:29:15.163263+0000","last_became_peered":"2024-03-23T11:29:15.163263+0000","last_unstale":"2024-03-23T11:29:17.080180+0000","last_undegraded":"2024-03-23T11:29:17.080180+0000","last_fullsized":"2024-03-23T11:29:17.080180+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:49:27.211897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.052222+0000","last_change":"2024-03-23T11:29:15.673835+0000","last_active":"2024-03-23T11:29:17.052222+0000","last_peered":"2024-03-23T11:29:17.052222+0000","last_clean":"2024-03-23T11:29:17.052222+0000","last_became_active":"2024-03-23T11:29:15.672375+0000","last_became_peered":"2024-03-23T11:29:15.672375+0000","last_unstale":"2024-03-23T11:29:17.052222+0000","last_undegraded":"2024-03-23T11:29:17.052222+0000","last_fullsized":"2024-03-23T11:29:17.052222+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:57:47.807515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.928913+0000","last_change":"2024-03-23T11:29:15.164188+0000","last_active":"2024-03-23T11:29:16.928913+0000","last_peered":"2024-03-23T11:29:16.928913+0000","last_clean":"2024-03-23T11:29:16.928913+0000","last_became_active":"2024-03-23T11:29:15.163837+0000","last_became_peered":"2024-03-23T11:29:15.163837+0000","last_unstale":"2024-03-23T11:29:16.928913+0000","last_undegraded":"2024-03-23T11:29:16.928913+0000","last_fullsized":"2024-03-23T11:29:16.928913+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:57:18.783732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076206+0000","last_change":"2024-03-23T11:29:15.663646+0000","last_active":"2024-03-23T11:29:17.076206+0000","last_peered":"2024-03-23T11:29:17.076206+0000","last_clean":"2024-03-23T11:29:17.076206+0000","last_became_active":"2024-03-23T11:29:15.663344+0000","last_became_peered":"2024-03-23T11:29:15.663344+0000","last_unstale":"2024-03-23T11:29:17.076206+0000","last_undegraded":"2024-03-23T11:29:17.076206+0000","last_fullsized":"2024-03-23T11:29:17.076206+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:40:12.294892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.077858+0000","last_change":"2024-03-23T11:29:15.169786+0000","last_active":"2024-03-23T11:29:17.077858+0000","last_peered":"2024-03-23T11:29:17.077858+0000","last_clean":"2024-03-23T11:29:17.077858+0000","last_became_active":"2024-03-23T11:29:15.167122+0000","last_became_peered":"2024-03-23T11:29:15.167122+0000","last_unstale":"2024-03-23T11:29:17.077858+0000","last_undegraded":"2024-03-23T11:29:17.077858+0000","last_fullsized":"2024-03-23T11:29:17.077858+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:26:01.038104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.928937+0000","last_change":"2024-03-23T11:29:15.666879+0000","last_active":"2024-03-23T11:29:16.928937+0000","last_peered":"2024-03-23T11:29:16.928937+0000","last_clean":"2024-03-23T11:29:16.928937+0000","last_became_active":"2024-03-23T11:29:15.666394+0000","last_became_peered":"2024-03-23T11:29:15.666394+0000","last_unstale":"2024-03-23T11:29:16.928937+0000","last_undegraded":"2024-03-23T11:29:16.928937+0000","last_fullsized":"2024-03-23T11:29:16.928937+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:20:38.886782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":11,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.076344+0000","last_change":"2024-03-23T11:29:15.164528+0000","last_active":"2024-03-23T11:29:17.076344+0000","last_peered":"2024-03-23T11:29:17.076344+0000","last_clean":"2024-03-23T11:29:17.076344+0000","last_became_active":"2024-03-23T11:29:15.164222+0000","last_became_peered":"2024-03-23T11:29:15.164222+0000","last_unstale":"2024-03-23T11:29:17.076344+0000","last_undegraded":"2024-03-23T11:29:17.076344+0000","last_fullsized":"2024-03-23T11:29:17.076344+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:31:06.346410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:17.079509+0000","last_change":"2024-03-23T11:29:15.650442+0000","last_active":"2024-03-23T11:29:17.079509+0000","last_peered":"2024-03-23T11:29:17.079509+0000","last_clean":"2024-03-23T11:29:17.079509+0000","last_became_active":"2024-03-23T11:29:15.650158+0000","last_became_peered":"2024-03-23T11:29:15.650158+0000","last_unstale":"2024-03-23T11:29:17.079509+0000","last_undegraded":"2024-03-23T11:29:17.079509+0000","last_fullsized":"2024-03-23T11:29:17.079509+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:15.119178+0000","last_clean_scrub_stamp":"2024-03-23T11:29:15.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:47:14.336720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"104'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T11:29:16.650329+0000","last_change":"2024-03-23T11:29:15.159260+0000","last_active":"2024-03-23T11:29:16.650329+0000","last_peered":"2024-03-23T11:29:16.650329+0000","last_clean":"2024-03-23T11:29:16.650329+0000","last_became_active":"2024-03-23T11:29:15.157891+0000","last_became_peered":"2024-03-23T11:29:15.157891+0000","last_unstale":"2024-03-23T11:29:16.650329+0000","last_undegraded":"2024-03-23T11:29:16.650329+0000","last_fullsized":"2024-03-23T11:29:16.650329+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:29:14.103211+0000","last_clean_scrub_stamp":"2024-03-23T11:29:14.103211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:05:26.465230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":252,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T11:29:15.641662+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:29:15.641662+0000","last_peered":"2024-03-23T11:29:15.641662+0000","last_clean":"2024-03-23T11:29:15.641662+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:29:15.641662+0000","last_undegraded":"2024-03-23T11:29:15.641662+0000","last_fullsized":"2024-03-23T11:29:15.641662+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542273,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1896448,"data_stored":1149689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.083,"15min":0.98999999999999999},"min":{"1min":0.623,"5min":0.5,"15min":0.5},"max":{"1min":1.599,"5min":2.8370000000000002,"15min":2.8370000000000002},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.0309999999999999},"min":{"1min":0.749,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.466,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.90400000000000003}]},{"osd":1,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.26,"15min":1.198},"min":{"1min":0.73499999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.708,"5min":3.569,"15min":3.569},"last":1.083},{"interface":"front","average":{"1min":1.345,"5min":1.3100000000000001,"15min":1.2649999999999999},"min":{"1min":0.877,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.653,"5min":1.8009999999999999,"15min":2.4350000000000001},"last":1.24}]},{"osd":2,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.1399999999999999,"15min":1.077},"min":{"1min":0.69599999999999995,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.379,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":1.625},{"interface":"front","average":{"1min":1.212,"5min":1.238,"15min":1.179},"min":{"1min":0.84199999999999997,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.6060000000000001,"5min":3.524,"15min":3.524},"last":1.3939999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.224,"15min":1.175},"min":{"1min":0.72199999999999998,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.5189999999999999,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.3180000000000001,"15min":1.27},"min":{"1min":0.93500000000000005,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.653,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":1.2729999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.22,"15min":1.222},"min":{"1min":0.80500000000000005,"5min":0.65300000000000002,"15min":0.54300000000000004},"max":{"1min":2.1560000000000001,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":1.026},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.407,"15min":1.3839999999999999},"min":{"1min":0.94699999999999995,"5min":0.63800000000000001,"15min":0.60199999999999998},"max":{"1min":1.623,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":1.5029999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.365,"15min":1.341},"min":{"1min":0.84599999999999997,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.7969999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4830000000000001,"15min":1.407},"min":{"1min":0.91700000000000004,"5min":0.77100000000000002,"15min":0.58199999999999996},"max":{"1min":1.978,"5min":2.0219999999999998,"15min":2.0859999999999999},"last":1.357}]},{"osd":6,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.446,"15min":1.4259999999999999},"min":{"1min":0.98699999999999999,"5min":0.73599999999999999,"15min":0.61399999999999999},"max":{"1min":2.0680000000000001,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.4039999999999999,"15min":1.363},"min":{"1min":1.034,"5min":0.61699999999999999,"15min":0.59899999999999998},"max":{"1min":1.6359999999999999,"5min":3.218,"15min":3.218},"last":1.4179999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542273,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1298432,"data_stored":559192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.107,"15min":1.1020000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.522,"5min":2.403,"15min":3.9140000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1839999999999999,"15min":1.141},"min":{"1min":0.751,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.669,"5min":1.774,"15min":6.6539999999999999},"last":1.0549999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1659999999999999,"15min":1.2030000000000001},"min":{"1min":0.77300000000000002,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.6100000000000001,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":1.611},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1850000000000001,"15min":1.2270000000000001},"min":{"1min":0.82299999999999995,"5min":0.76200000000000001,"15min":0.70399999999999996},"max":{"1min":1.514,"5min":1.9510000000000001,"15min":2.0859999999999999},"last":1.1890000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.1870000000000001,"15min":1.228},"min":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.5049999999999999,"5min":1.6419999999999999,"15min":3.73},"last":0.87},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.284,"15min":1.2869999999999999},"min":{"1min":0.73499999999999999,"5min":0.72899999999999998,"15min":0.68100000000000005},"max":{"1min":1.6140000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.3540000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.2270000000000001,"15min":1.2649999999999999},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.749},"max":{"1min":1.4790000000000001,"5min":1.9890000000000001,"15min":12.914},"last":1.006},{"interface":"front","average":{"1min":1.369,"5min":1.335,"15min":1.349},"min":{"1min":0.82099999999999995,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6559999999999999,"5min":1.9670000000000001,"15min":3.8100000000000001},"last":1.8919999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3100000000000001,"15min":1.349},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.51100000000000001},"max":{"1min":1.772,"5min":2.4420000000000002,"15min":2.702},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3400000000000001,"15min":1.3819999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.82299999999999995},"max":{"1min":1.845,"5min":2.1080000000000001,"15min":7.2140000000000004},"last":1.1040000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4139999999999999,"15min":1.4890000000000001},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.73899999999999999},"max":{"1min":1.7549999999999999,"5min":2.1099999999999999,"15min":3.097},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.4610000000000001,"15min":1.478},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.72499999999999998},"max":{"1min":1.8640000000000001,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":1.702}]},{"osd":7,"last update":"Sat Mar 23 11:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5389999999999999,"15min":1.621},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.80400000000000005},"max":{"1min":2.0760000000000001,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.488,"15min":1.4890000000000001},"min":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.57699999999999996},"max":{"1min":1.911,"5min":4.085,"15min":4.085},"last":2.0649999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542273,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1298432,"data_stored":559279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.093,"15min":1.139},"min":{"1min":0.755,"5min":0.65500000000000003,"15min":0.55400000000000005},"max":{"1min":1.484,"5min":1.518,"15min":2.2999999999999998},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1220000000000001,"15min":1.147},"min":{"1min":0.75,"5min":0.61499999999999999,"15min":0.47799999999999998},"max":{"1min":1.407,"5min":2.1190000000000002,"15min":2.355},"last":1.6259999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.1899999999999999,"15min":1.1779999999999999},"min":{"1min":0.89800000000000002,"5min":0.70599999999999996,"15min":0.65300000000000002},"max":{"1min":1.5109999999999999,"5min":1.6699999999999999,"15min":2.2559999999999998},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.159,"15min":1.1930000000000001},"min":{"1min":0.86599999999999999,"5min":0.66900000000000004,"15min":0.495},"max":{"1min":1.391,"5min":1.627,"15min":1.8580000000000001},"last":1.278}]},{"osd":2,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1910000000000001,"15min":1.234},"min":{"1min":0.89900000000000002,"5min":0.71699999999999997,"15min":0.60199999999999998},"max":{"1min":1.3959999999999999,"5min":1.6659999999999999,"15min":2.012},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.2569999999999999,"15min":1.24},"min":{"1min":0.86499999999999999,"5min":0.77600000000000002,"15min":0.64400000000000002},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.946},"last":1.579}]},{"osd":3,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.262,"15min":1.2949999999999999},"min":{"1min":0.93500000000000005,"5min":0.69299999999999995,"15min":0.66800000000000004},"max":{"1min":1.6499999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.224},{"interface":"front","average":{"1min":1.367,"5min":1.3480000000000001,"15min":1.3839999999999999},"min":{"1min":0.97999999999999998,"5min":0.78600000000000003,"15min":0.73199999999999998},"max":{"1min":1.6950000000000001,"5min":1.7549999999999999,"15min":2.3999999999999999},"last":1.72}]},{"osd":4,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.2889999999999999,"15min":1.357},"min":{"1min":0.98599999999999999,"5min":0.754,"15min":0.56699999999999995},"max":{"1min":1.6679999999999999,"5min":1.8779999999999999,"15min":2.2090000000000001},"last":1.385},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.4450000000000001,"15min":1.502},"min":{"1min":1.071,"5min":1.0169999999999999,"15min":0.65000000000000002},"max":{"1min":1.774,"5min":2.2839999999999998,"15min":2.5009999999999999},"last":1.6859999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.363,"15min":1.4239999999999999},"min":{"1min":1.0609999999999999,"5min":0.85699999999999998,"15min":0.48899999999999999},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":2.4860000000000002},"last":1.329},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.371,"15min":1.4570000000000001},"min":{"1min":1.2370000000000001,"5min":0.86599999999999999,"15min":0.624},"max":{"1min":1.7909999999999999,"5min":2.0569999999999999,"15min":2.3679999999999999},"last":1.5329999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.4279999999999999,"15min":1.4970000000000001},"min":{"1min":1.1000000000000001,"5min":0.90900000000000003,"15min":0.54000000000000004},"max":{"1min":1.841,"5min":2.5720000000000001,"15min":2.7229999999999999},"last":1.462},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.504,"15min":1.5700000000000001},"min":{"1min":1.2689999999999999,"5min":1.0469999999999999,"15min":0.60099999999999998},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.7509999999999999},"last":1.409}]}]},{"osd":4,"up_from":14,"seq":60129542273,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1314816,"data_stored":560897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.238,"15min":1.1539999999999999},"min":{"1min":0.79800000000000004,"5min":0.66800000000000004,"15min":0.58999999999999997},"max":{"1min":1.5860000000000001,"5min":1.627,"15min":1.6759999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.337,"15min":1.26},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.63700000000000001},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.8009999999999999},"last":1.512}]},{"osd":1,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1839999999999999,"15min":1.1479999999999999},"min":{"1min":0.76700000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.4970000000000001,"5min":1.6990000000000001,"15min":4.327},"last":1.409},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.169,"15min":1.137},"min":{"1min":0.86199999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.5389999999999999,"5min":1.7470000000000001,"15min":2.4369999999999998},"last":1.3260000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.194,"15min":1.2110000000000001},"min":{"1min":0.91100000000000003,"5min":0.879,"15min":0.624},"max":{"1min":1.518,"5min":1.627,"15min":2.6080000000000001},"last":1.119},{"interface":"front","average":{"1min":1.175,"5min":1.244,"15min":1.248},"min":{"1min":0.876,"5min":0.80600000000000005,"15min":0.71799999999999997},"max":{"1min":1.73,"5min":1.927,"15min":4.1310000000000002},"last":1.1639999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.234,"15min":1.2250000000000001},"min":{"1min":0.83499999999999996,"5min":0.747,"15min":0.622},"max":{"1min":1.4550000000000001,"5min":1.6499999999999999,"15min":2.5579999999999998},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.345,"5min":1.3420000000000001,"15min":1.3},"min":{"1min":0.97399999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.5880000000000001,"5min":1.665,"15min":4.2400000000000002},"last":1.5900000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.238,"15min":1.2609999999999999},"min":{"1min":0.94499999999999995,"5min":0.83899999999999997,"15min":0.74299999999999999},"max":{"1min":1.7,"5min":1.966,"15min":2.508},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.3400000000000001,"15min":1.359},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.60499999999999998},"max":{"1min":1.6679999999999999,"5min":1.877,"15min":5.859},"last":1.075}]},{"osd":6,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.409,"15min":1.3959999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.58899999999999997},"max":{"1min":1.784,"5min":2.222,"15min":5.3339999999999996},"last":1.885},{"interface":"front","average":{"1min":1.421,"5min":1.47,"15min":1.4830000000000001},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.72599999999999998},"max":{"1min":1.71,"5min":1.875,"15min":5.3940000000000001},"last":1.732}]},{"osd":7,"last update":"Sat Mar 23 11:29:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.5169999999999999,"15min":1.4870000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.754},"max":{"1min":1.879,"5min":2.1549999999999998,"15min":2.6859999999999999},"last":1.944},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.365,"15min":1.3899999999999999},"min":{"1min":0.79000000000000004,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.6930000000000001,"5min":2.028,"15min":2.641},"last":1.569}]}]},{"osd":3,"up_from":14,"seq":60129542273,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1310720,"data_stored":560795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.86899999999999999,"15min":0.93100000000000005},"min":{"1min":0.47499999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.27,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.099},{"interface":"front","average":{"1min":1.103,"5min":1.0760000000000001,"15min":1.097},"min":{"1min":0.65400000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.54,"5min":1.54,"15min":2.0790000000000002},"last":1.4910000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":0.98599999999999999},"min":{"1min":0.70499999999999996,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.3320000000000001,"5min":1.8049999999999999,"15min":1.976},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.218,"15min":1.246},"min":{"1min":0.74399999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.774,"5min":1.8100000000000001,"15min":2.2309999999999999},"last":1.5700000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.107,"15min":1.127},"min":{"1min":0.63200000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.746,"5min":1.839,"15min":3.3109999999999999},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.177,"15min":1.216},"min":{"1min":0.65100000000000002,"5min":0.51900000000000002,"15min":0.48399999999999999},"max":{"1min":1.595,"5min":2.2440000000000002,"15min":2.7109999999999999},"last":1.274}]},{"osd":4,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2609999999999999,"15min":1.276},"min":{"1min":0.93200000000000005,"5min":0.79000000000000004,"15min":0.73199999999999998},"max":{"1min":1.6899999999999999,"5min":1.897,"15min":2.3490000000000002},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.429,"15min":1.4870000000000001},"min":{"1min":1.0800000000000001,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.081,"5min":2.081,"15min":4.8970000000000002},"last":1.627}]},{"osd":5,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.458,"15min":1.494},"min":{"1min":1.167,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":1.6819999999999999,"5min":2.1520000000000001,"15min":3.4910000000000001},"last":1.704},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3660000000000001,"15min":1.423},"min":{"1min":0.92600000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":3.125},"last":1.339}]},{"osd":6,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3640000000000001,"15min":1.4530000000000001},"min":{"1min":1.117,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.8640000000000001,"5min":1.974,"15min":4.5499999999999998},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4419999999999999,"15min":1.5129999999999999},"min":{"1min":0.91100000000000003,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6439999999999999,"5min":2.0569999999999999,"15min":2.8450000000000002},"last":1.4410000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3939999999999999,"15min":1.47},"min":{"1min":1.032,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.7789999999999999,"5min":1.9219999999999999,"15min":2.952},"last":1.222},{"interface":"front","average":{"1min":1.468,"5min":1.4690000000000001,"15min":1.5629999999999999},"min":{"1min":0.88600000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":4.798},"last":1.5229999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542273,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1298432,"data_stored":559243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91200000000000003,"15min":0.90100000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.2290000000000001,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.052,"5min":1.097,"15min":1.0389999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.35099999999999998},"max":{"1min":1.667,"5min":1.7310000000000001,"15min":1.8220000000000001},"last":1.2210000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.85599999999999998,"15min":0.91500000000000004},"min":{"1min":0.504,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.006,"5min":1.365,"15min":1.9430000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.0580000000000001,"15min":1.0189999999999999},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.35799999999999998},"max":{"1min":1.49,"5min":1.593,"15min":1.7230000000000001},"last":1.5349999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1060000000000001,"15min":1.1140000000000001},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.5900000000000001,"5min":1.905,"15min":1.98},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.6879999999999999,"5min":1.827,"15min":2.1560000000000001},"last":0.93300000000000005}]},{"osd":4,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.341,"15min":1.3500000000000001},"min":{"1min":0.91800000000000004,"5min":0.73299999999999998,"15min":0.69599999999999995},"max":{"1min":1.758,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.421,"15min":1.3899999999999999},"min":{"1min":0.93799999999999994,"5min":0.85799999999999998,"15min":0.83499999999999996},"max":{"1min":1.879,"5min":2.0230000000000001,"15min":2.085},"last":1.7130000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3100000000000001,"15min":1.3759999999999999},"min":{"1min":0.752,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7130000000000001,"5min":1.9530000000000001,"15min":2.5830000000000002},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3600000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.66000000000000003},"max":{"1min":1.6839999999999999,"5min":1.96,"15min":2.6469999999999998},"last":1.28}]},{"osd":6,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3799999999999999,"15min":1.3759999999999999},"min":{"1min":0.84499999999999997,"5min":0.76300000000000001,"15min":0.68600000000000005},"max":{"1min":1.905,"5min":2.0600000000000001,"15min":2.758},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.3799999999999999,"15min":1.464},"min":{"1min":0.79100000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.879,"5min":1.889,"15min":2.8580000000000001},"last":1.3440000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.4350000000000001,"15min":1.4219999999999999},"min":{"1min":0.84199999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.9039999999999999,"5min":2.0150000000000001,"15min":2.246},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.4670000000000001,"15min":1.538},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.919,"5min":2.012,"15min":2.7120000000000002},"last":1.4930000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542274,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1290240,"data_stored":558540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.90800000000000003,"15min":0.89200000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":3.2959999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.03,"15min":1.0089999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.99},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.95299999999999996,"15min":0.96499999999999997},"min":{"1min":0.55400000000000005,"5min":0.505,"15min":0.40699999999999997},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.738},"last":0.499},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0880000000000001,"15min":1.1120000000000001},"min":{"1min":0.57399999999999995,"5min":0.55100000000000005,"15min":0.49399999999999999},"max":{"1min":1.5569999999999999,"5min":1.762,"15min":2.2160000000000002},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2130000000000001,"15min":1.226},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.6519999999999999,"5min":1.653,"15min":2.4430000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.113,"5min":1.1120000000000001,"15min":1.1379999999999999},"min":{"1min":0.65700000000000003,"5min":0.52000000000000002,"15min":0.434},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.4169999999999998},"last":0.45900000000000002}]},{"osd":4,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3580000000000001,"15min":1.3009999999999999},"min":{"1min":1.0549999999999999,"5min":0.83899999999999997,"15min":0.79400000000000004},"max":{"1min":1.7969999999999999,"5min":1.9019999999999999,"15min":2.153},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.292,"5min":1.2749999999999999,"15min":1.3129999999999999},"min":{"1min":0.90300000000000002,"5min":0.81299999999999994,"15min":0.74199999999999999},"max":{"1min":1.8049999999999999,"5min":1.867,"15min":2.758},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3400000000000001,"15min":1.3879999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.81699999999999995},"max":{"1min":1.992,"5min":1.992,"15min":2.6589999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4159999999999999,"15min":1.4399999999999999},"min":{"1min":0.96899999999999997,"5min":0.89900000000000002,"15min":0.72199999999999998},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":3.0329999999999999},"last":0.66000000000000003}]},{"osd":6,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.421,"15min":1.4299999999999999},"min":{"1min":0.88600000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.8839999999999999,"5min":2.1600000000000001,"15min":2.2730000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.421,"15min":1.504},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":2.9630000000000001},"last":0.747}]},{"osd":7,"last update":"Sat Mar 23 11:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.5009999999999999,"15min":1.548},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":3.2749999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.482,"5min":1.522,"15min":1.5529999999999999},"min":{"1min":0.93500000000000005,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.962,"5min":2.242,"15min":2.7829999999999999},"last":0.71999999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542272,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32964,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338876,"statfs":{"total":96636764160,"available":96603009024,"internally_reserved":0,"allocated":1888256,"data_stored":1149490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.47599999999999998,"5min":0.379,"15min":0.376},"max":{"1min":1.8660000000000001,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97499999999999998,"15min":0.95799999999999996},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.30399999999999999},"max":{"1min":1.536,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.71699999999999997}]},{"osd":2,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.014,"15min":1.036},"min":{"1min":0.80200000000000005,"5min":0.44500000000000001,"15min":0.33700000000000002},"max":{"1min":1.4910000000000001,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.155,"5min":1.093,"15min":1.0469999999999999},"min":{"1min":0.63200000000000001,"5min":0.54000000000000004,"15min":0.432},"max":{"1min":1.6299999999999999,"5min":2.5760000000000001,"15min":2.5760000000000001},"last":0.90400000000000003}]},{"osd":3,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1499999999999999,"15min":1.1140000000000001},"min":{"1min":0.66800000000000004,"5min":0.40899999999999997,"15min":0.35799999999999998},"max":{"1min":1.718,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.335,"15min":1.3180000000000001},"min":{"1min":0.75600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.968,"5min":2.9390000000000001,"15min":2.9390000000000001},"last":1.2869999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.298,"15min":1.343},"min":{"1min":0.94399999999999995,"5min":0.77900000000000003,"15min":0.56100000000000005},"max":{"1min":1.5860000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.3069999999999999,"15min":1.284},"min":{"1min":0.82099999999999995,"5min":0.73099999999999998,"15min":0.39900000000000002},"max":{"1min":1.8460000000000001,"5min":3.125,"15min":3.125},"last":1.968}]},{"osd":5,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.3520000000000001,"15min":1.391},"min":{"1min":0.91200000000000003,"5min":0.90200000000000002,"15min":0.51200000000000001},"max":{"1min":1.925,"5min":3.0830000000000002,"15min":3.0830000000000002},"last":1.8740000000000001},{"interface":"front","average":{"1min":1.345,"5min":1.349,"15min":1.454},"min":{"1min":0.877,"5min":0.76300000000000001,"15min":0.52700000000000002},"max":{"1min":2.0449999999999999,"5min":3.1899999999999999,"15min":3.1899999999999999},"last":1.837}]},{"osd":6,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.399,"15min":1.4810000000000001},"min":{"1min":0.90200000000000002,"5min":0.876,"15min":0.47999999999999998},"max":{"1min":2.1030000000000002,"5min":3.391,"15min":3.391},"last":1.704},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.391,"15min":1.4119999999999999},"min":{"1min":0.94899999999999995,"5min":0.746,"15min":0.58399999999999996},"max":{"1min":2.2170000000000001,"5min":3.153,"15min":3.153},"last":1.3939999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4890000000000001,"15min":1.522},"min":{"1min":0.98699999999999999,"5min":0.77300000000000002,"15min":0.60599999999999998},"max":{"1min":2.1400000000000001,"5min":3.5350000000000001,"15min":3.5350000000000001},"last":1.579},{"interface":"front","average":{"1min":1.536,"5min":1.4830000000000001,"15min":1.4970000000000001},"min":{"1min":1.125,"5min":0.83799999999999997,"15min":0.45300000000000001},"max":{"1min":2.2610000000000001,"5min":3.3650000000000002,"15min":3.3650000000000002},"last":2.0350000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:29:21.181 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T11:29:21.181 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T11:29:21.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.456+0000 7f4292e00640 1 Processor -- start 2024-03-23T11:29:21.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.456+0000 7f4292e00640 1 -- start start 2024-03-23T11:29:21.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.458+0000 7f4292e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f428c143fe0 0x7f428c0ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:21.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.458+0000 7f4292e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f428c0ce780 0x7f428c15e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:21.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.458+0000 7f4292e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428c0cebd0 con 0x7f428c143c00 2024-03-23T11:29:21.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.458+0000 7f4292e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f428c15eec0 con 0x7f428c143fe0 2024-03-23T11:29:21.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.458+0000 7f4292e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f428c0b1a50 con 0x7f428c0ce780 2024-03-23T11:29:21.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.458+0000 7f4291000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f428c143fe0 0x7f428c0ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:21.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.458+0000 7f4291000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f428c143fe0 0x7f428c0ce240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54466/0 (socket says 172.21.15.62:54466) 2024-03-23T11:29:21.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.458+0000 7f4291000640 1 -- 172.21.15.62:0/3205031577 learned_addr learned my addr 172.21.15.62:0/3205031577 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:21.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.459+0000 7f4292400640 1 --2- 172.21.15.62:0/3205031577 >> v2:172.21.15.120:3300/0 conn(0x7f428c0ce780 0x7f428c15e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:21.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.459+0000 7f4291000640 1 -- 172.21.15.62:0/3205031577 >> v2:172.21.15.120:3300/0 conn(0x7f428c0ce780 msgr2=0x7f428c15e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:21.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.459+0000 7f4291000640 1 --2- 172.21.15.62:0/3205031577 >> v2:172.21.15.120:3300/0 conn(0x7f428c0ce780 0x7f428c15e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.459+0000 7f4291000640 1 -- 172.21.15.62:0/3205031577 >> v1:172.21.15.120:6789/0 conn(0x7f428c143c00 legacy=0x7f428c15abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:21.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.459+0000 7f4291000640 1 -- 172.21.15.62:0/3205031577 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c0b1d30 con 0x7f428c143fe0 2024-03-23T11:29:21.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.460+0000 7f4291000640 1 --2- 172.21.15.62:0/3205031577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f428c143fe0 0x7f428c0ce240 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f428c0b3960 tx=0x7f427802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968ff9acb26418fe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:21.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.460+0000 7f428be00640 1 -- 172.21.15.62:0/3205031577 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278031870 con 0x7f428c143fe0 2024-03-23T11:29:21.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.460+0000 7f428be00640 1 -- 172.21.15.62:0/3205031577 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42780319d0 con 0x7f428c143fe0 2024-03-23T11:29:21.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.460+0000 7f428be00640 1 -- 172.21.15.62:0/3205031577 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278031d60 con 0x7f428c143fe0 2024-03-23T11:29:21.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.461+0000 7f4292e00640 1 -- 172.21.15.62:0/3205031577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f428c143fe0 msgr2=0x7f428c0ce240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:21.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.461+0000 7f4292e00640 1 --2- 172.21.15.62:0/3205031577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f428c143fe0 0x7f428c0ce240 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f428c0b3960 tx=0x7f427802b910 comp rx=0 tx=0).stop 2024-03-23T11:29:21.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.461+0000 7f4292e00640 1 -- 172.21.15.62:0/3205031577 shutdown_connections 2024-03-23T11:29:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.461+0000 7f4292e00640 1 --2- 172.21.15.62:0/3205031577 >> v2:172.21.15.120:3300/0 conn(0x7f428c0ce780 0x7f428c15e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.461+0000 7f4292e00640 1 --2- 172.21.15.62:0/3205031577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f428c143fe0 0x7f428c0ce240 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.461+0000 7f4292e00640 1 -- 172.21.15.62:0/3205031577 >> 172.21.15.62:0/3205031577 conn(0x7f428c066050 msgr2=0x7f428c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.461+0000 7f4292e00640 1 -- 172.21.15.62:0/3205031577 shutdown_connections 2024-03-23T11:29:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.461+0000 7f4292e00640 1 -- 172.21.15.62:0/3205031577 wait complete. 2024-03-23T11:29:21.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.463+0000 7f4292e00640 1 Processor -- start 2024-03-23T11:29:21.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.463+0000 7f4292e00640 1 -- start start 2024-03-23T11:29:21.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.464+0000 7f4292e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f428c0ce780 0x7f428c1f52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:21.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.464+0000 7f4292e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f428c143c00 0x7f428c1f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.464+0000 7f4292e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f428c143fe0 0x7f428c0ac200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.464+0000 7f4292e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c06cf20 con 0x7f428c0ce780 2024-03-23T11:29:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.464+0000 7f4292e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c13b5e0 con 0x7f428c143c00 2024-03-23T11:29:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.464+0000 7f4292e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428c06c050 con 0x7f428c143fe0 2024-03-23T11:29:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.464+0000 7f4291a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f428c0ce780 0x7f428c1f52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.464+0000 7f4291a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f428c0ce780 0x7f428c1f52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54470/0 (socket says 172.21.15.62:54470) 2024-03-23T11:29:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.464+0000 7f4291a00640 1 -- 172.21.15.62:0/3377524318 learned_addr learned my addr 172.21.15.62:0/3377524318 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:21.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4291000640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f428c143c00 0x7f428c1f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4291a00640 1 -- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f428c143fe0 msgr2=0x7f428c0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4292400640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f428c143fe0 0x7f428c0ac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4291a00640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f428c143fe0 0x7f428c0ac200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4291a00640 1 -- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f428c143c00 msgr2=0x7f428c1f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4291a00640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f428c143c00 0x7f428c1f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4291a00640 1 -- 172.21.15.62:0/3377524318 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42780094d0 con 0x7f428c0ce780 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4291000640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f428c143c00 0x7f428c1f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4292400640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f428c143fe0 0x7f428c0ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4291a00640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f428c0ce780 0x7f428c1f52f0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f428000c440 tx=0x7f428000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4289600640 1 -- 172.21.15.62:0/3377524318 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42800035f0 con 0x7f428c0ce780 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4289600640 1 -- 172.21.15.62:0/3377524318 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4280003770 con 0x7f428c0ce780 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.465+0000 7f4292e00640 1 -- 172.21.15.62:0/3377524318 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c06c2a0 con 0x7f428c0ce780 2024-03-23T11:29:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.466+0000 7f4289600640 1 -- 172.21.15.62:0/3377524318 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4280003b20 con 0x7f428c0ce780 2024-03-23T11:29:21.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.466+0000 7f4292e00640 1 -- 172.21.15.62:0/3377524318 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c06c860 con 0x7f428c0ce780 2024-03-23T11:29:21.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.467+0000 7f4292e00640 1 -- 172.21.15.62:0/3377524318 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428c06d060 con 0x7f428c0ce780 2024-03-23T11:29:21.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.468+0000 7f4289600640 1 -- 172.21.15.62:0/3377524318 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4280003c80 con 0x7f428c0ce780 2024-03-23T11:29:21.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.469+0000 7f4289600640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f426c06a7a0 0x7f426c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:21.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.469+0000 7f4289600640 1 -- 172.21.15.62:0/3377524318 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4280088370 con 0x7f428c0ce780 2024-03-23T11:29:21.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.469+0000 7f4291000640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f426c06a7a0 0x7f426c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:21.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.470+0000 7f4291000640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f426c06a7a0 0x7f426c06cc60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4278036000 tx=0x7f427803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:21.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.473+0000 7f4289600640 1 -- 172.21.15.62:0/3377524318 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f428c06d060 con 0x7f428c0ce780 2024-03-23T11:29:21.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.752+0000 7f4292e00640 1 -- 172.21.15.62:0/3377524318 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f428c06d060 con 0x7f428c0ce780 2024-03-23T11:29:21.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.758+0000 7f4289600640 1 -- 172.21.15.62:0/3377524318 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+0 (secure 0 0 0) 0x7f428c06d060 con 0x7f428c0ce780 2024-03-23T11:29:21.749 INFO:teuthology.orchestra.run.smithi062.stderr:updated caps for client.0 2024-03-23T11:29:21.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.761+0000 7f4292e00640 1 -- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f426c06a7a0 msgr2=0x7f426c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:21.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.761+0000 7f4292e00640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f426c06a7a0 0x7f426c06cc60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4278036000 tx=0x7f427803c040 comp rx=0 tx=0).stop 2024-03-23T11:29:21.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.761+0000 7f4292e00640 1 -- 172.21.15.62:0/3377524318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f428c0ce780 msgr2=0x7f428c1f52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:21.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.761+0000 7f4292e00640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f428c0ce780 0x7f428c1f52f0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f428000c440 tx=0x7f428000c900 comp rx=0 tx=0).stop 2024-03-23T11:29:21.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.761+0000 7f4292e00640 1 -- 172.21.15.62:0/3377524318 shutdown_connections 2024-03-23T11:29:21.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.761+0000 7f4292e00640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f426c06a7a0 0x7f426c06cc60 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.761+0000 7f4292e00640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f428c143fe0 0x7f428c0ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.761+0000 7f4292e00640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f428c143c00 0x7f428c1f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.761+0000 7f4292e00640 1 --2- 172.21.15.62:0/3377524318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f428c0ce780 0x7f428c1f52f0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.761+0000 7f4292e00640 1 -- 172.21.15.62:0/3377524318 >> 172.21.15.62:0/3377524318 conn(0x7f428c066050 msgr2=0x7f428c141a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:21.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.761+0000 7f4292e00640 1 -- 172.21.15.62:0/3377524318 shutdown_connections 2024-03-23T11:29:21.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.762+0000 7f4292e00640 1 -- 172.21.15.62:0/3377524318 wait complete. 2024-03-23T11:29:21.772 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:29:21.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.998+0000 7f041c400640 1 Processor -- start 2024-03-23T11:29:21.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:21.998+0000 7f041c400640 1 -- start start 2024-03-23T11:29:21.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.000+0000 7f041c400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f041406c300 0x7f04140b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:21.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.000+0000 7f041c400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f04140b3110 0x7f0414159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:21.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.000+0000 7f041c400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04140b38b0 con 0x7f0414152f60 2024-03-23T11:29:21.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.000+0000 7f041c400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04140cc9e0 con 0x7f04140b3110 2024-03-23T11:29:21.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.000+0000 7f041c400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04140b1bc0 con 0x7f041406c300 2024-03-23T11:29:21.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.001+0000 7f0419a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0414152f60 0x7f04140b2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47102/0 (socket says 172.21.15.62:47102) 2024-03-23T11:29:21.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.001+0000 7f0419a00640 1 -- 172.21.15.62:0/4026263546 learned_addr learned my addr 172.21.15.62:0/4026263546 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.001+0000 7f0419000640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f041406c300 0x7f04140b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.001+0000 7f041a400640 1 --2- 172.21.15.62:0/4026263546 >> v2:172.21.15.120:3300/0 conn(0x7f04140b3110 0x7f0414159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.001+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039540612 0 0) 0x7f04140b38b0 con 0x7f0414152f60 2024-03-23T11:29:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.001+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f8003520 con 0x7f0414152f60 2024-03-23T11:29:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2246615384 0 0) 0x7f03f8003520 con 0x7f0414152f60 2024-03-23T11:29:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f041406c300 msgr2=0x7f04140b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f041406c300 0x7f04140b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 >> v2:172.21.15.120:3300/0 conn(0x7f04140b3110 msgr2=0x7f0414159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 --2- 172.21.15.62:0/4026263546 >> v2:172.21.15.120:3300/0 conn(0x7f04140b3110 0x7f0414159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04140b1ea0 con 0x7f0414152f60 2024-03-23T11:29:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f0404001670 con 0x7f0414152f60 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 >> v1:172.21.15.62:6789/0 conn(0x7f0414152f60 legacy=0x7f04140b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03f8003970 0x7f03f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03f80047c0 0x7f03f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f041406c300 0x7f03f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04140cc9e0 con 0x7f03f8003970 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04140b1bc0 con 0x7f03f80047c0 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f800fd90 con 0x7f041406c300 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0419a00640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03f8003970 0x7f03f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f0419000640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03f80047c0 0x7f03f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.002+0000 7f041a400640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f041406c300 0x7f03f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.003+0000 7f0419a00640 1 -- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f041406c300 msgr2=0x7f03f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.003+0000 7f0419a00640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f041406c300 0x7f03f8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.003+0000 7f0419a00640 1 -- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03f80047c0 msgr2=0x7f03f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.003+0000 7f0419a00640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03f80047c0 0x7f03f8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.003+0000 7f0419a00640 1 -- 172.21.15.62:0/4026263546 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f8003520 con 0x7f03f8003970 2024-03-23T11:29:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.003+0000 7f041a400640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f041406c300 0x7f03f8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.004+0000 7f0419a00640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03f8003970 0x7f03f8003d90 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f040400c0e0 tx=0x7f03f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ba317bed2830a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.004+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0404005ad0 con 0x7f03f8003970 2024-03-23T11:29:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.004+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0404005c30 con 0x7f03f8003970 2024-03-23T11:29:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.004+0000 7f0413e00640 1 -- 172.21.15.62:0/4026263546 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040402c7f0 con 0x7f03f8003970 2024-03-23T11:29:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.005+0000 7f041c400640 1 -- 172.21.15.62:0/4026263546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03f8003970 msgr2=0x7f03f8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.005+0000 7f041c400640 1 --2- 172.21.15.62:0/4026263546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03f8003970 0x7f03f8003d90 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f040400c0e0 tx=0x7f03f80057f0 comp rx=0 tx=0).stop 2024-03-23T11:29:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.005+0000 7f041a400640 1 -- 172.21.15.62:0/4026263546 reap_dead start 2024-03-23T11:29:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.005+0000 7f041c400640 1 -- 172.21.15.62:0/4026263546 shutdown_connections 2024-03-23T11:29:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.005+0000 7f041c400640 1 -- 172.21.15.62:0/4026263546 >> 172.21.15.62:0/4026263546 conn(0x7f04140b02e0 msgr2=0x7f04140b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.005+0000 7f041c400640 1 -- 172.21.15.62:0/4026263546 shutdown_connections 2024-03-23T11:29:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.005+0000 7f041c400640 1 -- 172.21.15.62:0/4026263546 wait complete. 2024-03-23T11:29:21.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.007+0000 7f041c400640 1 Processor -- start 2024-03-23T11:29:21.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.007+0000 7f041c400640 1 -- start start 2024-03-23T11:29:22.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.008+0000 7f041c400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03f8003970 0x7f0414153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.008+0000 7f041c400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0414141dc0 0x7f04140add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.008+0000 7f041c400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04140ae240 0x7f04141508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.008+0000 7f041c400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04140ae6f0 con 0x7f04140ae240 2024-03-23T11:29:22.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.008+0000 7f041c400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414153b90 con 0x7f0414141dc0 2024-03-23T11:29:22.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.008+0000 7f041c400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0414156b10 con 0x7f03f8003970 2024-03-23T11:29:22.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.008+0000 7f041a400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04140ae240 0x7f04141508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.008+0000 7f041a400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04140ae240 0x7f04141508d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54488/0 (socket says 172.21.15.62:54488) 2024-03-23T11:29:22.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.008+0000 7f041a400640 1 -- 172.21.15.62:0/2930618023 learned_addr learned my addr 172.21.15.62:0/2930618023 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:22.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.009+0000 7f0419a00640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03f8003970 0x7f0414153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.009+0000 7f0419000640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0414141dc0 0x7f04140add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.009+0000 7f041a400640 1 -- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03f8003970 msgr2=0x7f0414153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.009+0000 7f041a400640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03f8003970 0x7f0414153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.009+0000 7f041a400640 1 -- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0414141dc0 msgr2=0x7f04140add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.009+0000 7f041a400640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0414141dc0 0x7f04140add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.009+0000 7f041a400640 1 -- 172.21.15.62:0/2930618023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04141534c0 con 0x7f04140ae240 2024-03-23T11:29:22.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.009+0000 7f041a400640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04140ae240 0x7f04141508d0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f041406c090 tx=0x7f040c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:22.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.009+0000 7f0411600640 1 -- 172.21.15.62:0/2930618023 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c016620 con 0x7f04140ae240 2024-03-23T11:29:22.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.010+0000 7f0411600640 1 -- 172.21.15.62:0/2930618023 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c0167a0 con 0x7f04140ae240 2024-03-23T11:29:22.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.010+0000 7f0411600640 1 -- 172.21.15.62:0/2930618023 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c016b50 con 0x7f04140ae240 2024-03-23T11:29:22.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.010+0000 7f041c400640 1 -- 172.21.15.62:0/2930618023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0414156df0 con 0x7f04140ae240 2024-03-23T11:29:22.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.010+0000 7f041c400640 1 -- 172.21.15.62:0/2930618023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04141573b0 con 0x7f04140ae240 2024-03-23T11:29:22.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.011+0000 7f041c400640 1 -- 172.21.15.62:0/2930618023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0414066c90 con 0x7f04140ae240 2024-03-23T11:29:22.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.012+0000 7f0411600640 1 -- 172.21.15.62:0/2930618023 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f040c016cb0 con 0x7f04140ae240 2024-03-23T11:29:22.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.013+0000 7f0411600640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03f406a7a0 0x7f03f406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.013+0000 7f0411600640 1 -- 172.21.15.62:0/2930618023 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f040c089b50 con 0x7f04140ae240 2024-03-23T11:29:22.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.013+0000 7f0419a00640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03f406a7a0 0x7f03f406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.014+0000 7f0419a00640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03f406a7a0 0x7f03f406cc60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0404034000 tx=0x7f0404009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:22.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.017+0000 7f0411600640 1 -- 172.21.15.62:0/2930618023 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f040c054ed0 con 0x7f04140ae240 2024-03-23T11:29:22.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.256+0000 7f041c400640 1 -- 172.21.15.62:0/2930618023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04140cb9f0 con 0x7f04140ae240 2024-03-23T11:29:22.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.257+0000 7f0411600640 1 -- 172.21.15.62:0/2930618023 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f040c0584f0 con 0x7f04140ae240 2024-03-23T11:29:22.248 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:22.248 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":78,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:29:19.157087+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8271":{"gid":8271,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6834/692790652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":692790652},{"type":"v1","addr":"172.21.15.62:6834","nonce":692790652}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:29:22.249 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 78 2024-03-23T11:29:22.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 -- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03f406a7a0 msgr2=0x7f03f406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03f406a7a0 0x7f03f406cc60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0404034000 tx=0x7f0404009030 comp rx=0 tx=0).stop 2024-03-23T11:29:22.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 -- 172.21.15.62:0/2930618023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04140ae240 msgr2=0x7f04141508d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04140ae240 0x7f04141508d0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f041406c090 tx=0x7f040c00f520 comp rx=0 tx=0).stop 2024-03-23T11:29:22.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 -- 172.21.15.62:0/2930618023 shutdown_connections 2024-03-23T11:29:22.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f03f406a7a0 0x7f03f406cc60 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04140ae240 0x7f04141508d0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0414141dc0 0x7f04140add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 --2- 172.21.15.62:0/2930618023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03f8003970 0x7f0414153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 -- 172.21.15.62:0/2930618023 >> 172.21.15.62:0/2930618023 conn(0x7f04140b02e0 msgr2=0x7f041414aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:22.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 -- 172.21.15.62:0/2930618023 shutdown_connections 2024-03-23T11:29:22.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.260+0000 7f041c400640 1 -- 172.21.15.62:0/2930618023 wait complete. 2024-03-23T11:29:22.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:29:15.626274+0000', 'modified': '2024-03-23T11:29:19.157087+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8271}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8271': {'gid': 8271, 'name': 'a', 'rank': 0, 'incarnation': 74, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.62:6834/692790652', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 692790652}, {'type': 'v1', 'addr': '172.21.15.62:6834', 'nonce': 692790652}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:29:22.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:29:22.674 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T11:29:22.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.913+0000 7f60bc200640 1 Processor -- start 2024-03-23T11:29:22.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.913+0000 7f60bc200640 1 -- start start 2024-03-23T11:29:22.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.915+0000 7f60bc200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60b406c6e0 0x7f60b406cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.915+0000 7f60bc200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b40cbc60 con 0x7f60b406c300 2024-03-23T11:29:22.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.915+0000 7f60bc200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b40cbf40 con 0x7f60b40cafa0 2024-03-23T11:29:22.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.915+0000 7f60bc200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b40cd410 con 0x7f60b406c6e0 2024-03-23T11:29:22.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.915+0000 7f60b9800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f60b406c300 0x7f60b40ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47114/0 (socket says 172.21.15.62:47114) 2024-03-23T11:29:22.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.915+0000 7f60b9800640 1 -- 172.21.15.62:0/3066248204 learned_addr learned my addr 172.21.15.62:0/3066248204 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:22.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.915+0000 7f60ba200640 1 --2- 172.21.15.62:0/3066248204 >> v2:172.21.15.62:3300/0 conn(0x7f60b406c6e0 0x7f60b406cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.915+0000 7f60b3e00640 1 -- 172.21.15.62:0/3066248204 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851414531 0 0) 0x7f60b40cbc60 con 0x7f60b406c300 2024-03-23T11:29:22.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.915+0000 7f60b3e00640 1 -- 172.21.15.62:0/3066248204 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f609c003520 con 0x7f60b406c300 2024-03-23T11:29:22.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.916+0000 7f60ba200640 1 -- 172.21.15.62:0/3066248204 >> v1:172.21.15.120:6789/0 conn(0x7f60b40cafa0 legacy=0x7f60b40cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:22.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.916+0000 7f60ba200640 1 -- 172.21.15.62:0/3066248204 >> v1:172.21.15.62:6789/0 conn(0x7f60b406c300 legacy=0x7f60b40ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.916+0000 7f60ba200640 1 -- 172.21.15.62:0/3066248204 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b40cc220 con 0x7f60b406c6e0 2024-03-23T11:29:22.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.917+0000 7f60ba200640 1 --2- 172.21.15.62:0/3066248204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60b406c6e0 0x7f60b406cb30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009d90 tx=0x7f60ac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df5cf64f22f0294 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:22.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.917+0000 7f60b3e00640 1 -- 172.21.15.62:0/3066248204 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac02cc60 con 0x7f60b406c6e0 2024-03-23T11:29:22.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.917+0000 7f60b3e00640 1 -- 172.21.15.62:0/3066248204 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac02cdc0 con 0x7f60b406c6e0 2024-03-23T11:29:22.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.917+0000 7f60b3e00640 1 -- 172.21.15.62:0/3066248204 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac034660 con 0x7f60b406c6e0 2024-03-23T11:29:22.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.917+0000 7f60bc200640 1 -- 172.21.15.62:0/3066248204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60b406c6e0 msgr2=0x7f60b406cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.918+0000 7f60bc200640 1 --2- 172.21.15.62:0/3066248204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60b406c6e0 0x7f60b406cb30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009d90 tx=0x7f60ac02beb0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.918+0000 7f60bc200640 1 -- 172.21.15.62:0/3066248204 shutdown_connections 2024-03-23T11:29:22.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.918+0000 7f60bc200640 1 --2- 172.21.15.62:0/3066248204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60b406c6e0 0x7f60b406cb30 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.918+0000 7f60bc200640 1 -- 172.21.15.62:0/3066248204 >> 172.21.15.62:0/3066248204 conn(0x7f60b40b02e0 msgr2=0x7f60b40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:22.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.918+0000 7f60bc200640 1 -- 172.21.15.62:0/3066248204 shutdown_connections 2024-03-23T11:29:22.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.918+0000 7f60bc200640 1 -- 172.21.15.62:0/3066248204 wait complete. 2024-03-23T11:29:22.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.919+0000 7f60bc200640 1 Processor -- start 2024-03-23T11:29:22.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.920+0000 7f60bc200640 1 -- start start 2024-03-23T11:29:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.920+0000 7f60bc200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60b406c300 0x7f60b41f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60bc200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b406c6e0 0x7f60b40acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60bc200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b40cafa0 0x7f60b40ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60bc200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4130660 con 0x7f60b406c300 2024-03-23T11:29:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60bc200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41311e0 con 0x7f60b40cafa0 2024-03-23T11:29:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60bc200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41308b0 con 0x7f60b406c6e0 2024-03-23T11:29:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60b9800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60b406c300 0x7f60b41f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60b9800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60b406c300 0x7f60b41f48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54508/0 (socket says 172.21.15.62:54508) 2024-03-23T11:29:22.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60b9800640 1 -- 172.21.15.62:0/3869128653 learned_addr learned my addr 172.21.15.62:0/3869128653 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:22.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60ba200640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b40cafa0 0x7f60b40ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60b9800640 1 -- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b406c6e0 msgr2=0x7f60b40acad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60b8e00640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b406c6e0 0x7f60b40acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60b9800640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b406c6e0 0x7f60b40acad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60b9800640 1 -- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b40cafa0 msgr2=0x7f60b40ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60b9800640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b40cafa0 0x7f60b40ad010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.921+0000 7f60b9800640 1 -- 172.21.15.62:0/3869128653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac009a70 con 0x7f60b406c300 2024-03-23T11:29:22.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.922+0000 7f60b9800640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60b406c300 0x7f60b41f48b0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f60a800c500 tx=0x7f60a800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:22.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.922+0000 7f60b1600640 1 -- 172.21.15.62:0/3869128653 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8005ce0 con 0x7f60b406c300 2024-03-23T11:29:22.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.922+0000 7f60b1600640 1 -- 172.21.15.62:0/3869128653 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a8005e40 con 0x7f60b406c300 2024-03-23T11:29:22.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.922+0000 7f60bc200640 1 -- 172.21.15.62:0/3869128653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b4130b90 con 0x7f60b406c300 2024-03-23T11:29:22.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.922+0000 7f60b1600640 1 -- 172.21.15.62:0/3869128653 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8007330 con 0x7f60b406c300 2024-03-23T11:29:22.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.922+0000 7f60bc200640 1 -- 172.21.15.62:0/3869128653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b40ad550 con 0x7f60b406c300 2024-03-23T11:29:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.924+0000 7f60b1600640 1 -- 172.21.15.62:0/3869128653 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f60a8007550 con 0x7f60b406c300 2024-03-23T11:29:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.925+0000 7f60b1600640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f609406a7a0 0x7f609406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.925+0000 7f60b1600640 1 -- 172.21.15.62:0/3869128653 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60a8088400 con 0x7f60b406c300 2024-03-23T11:29:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.926+0000 7f60bedc6740 1 Processor -- start 2024-03-23T11:29:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.926+0000 7f60b8e00640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f609406a7a0 0x7f609406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.926+0000 7f60bedc6740 1 -- start start 2024-03-23T11:29:22.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.927+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74579a390 0x55e7457519c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.927+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 0x55e745751f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.927+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74574c8b0 0x55e74577c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.927+0000 7f60bedc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7456bd110 con 0x55e74574c8b0 2024-03-23T11:29:22.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.927+0000 7f60bedc6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7456ba8a0 con 0x55e74579a390 2024-03-23T11:29:22.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.927+0000 7f60bedc6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7456b6190 con 0x55e745750410 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.927+0000 7f60ba200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74574c8b0 0x55e74577c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.927+0000 7f60ba200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74574c8b0 0x55e74577c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54522/0 (socket says 172.21.15.62:54522) 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.927+0000 7f60ba200640 1 -- 172.21.15.62:0/3698019394 learned_addr learned my addr 172.21.15.62:0/3698019394 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.927+0000 7f60b8e00640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f609406a7a0 0x7f609406cc60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f60b4157800 tx=0x7f60a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.928+0000 7f60b9800640 1 --2- 172.21.15.62:0/3698019394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74579a390 0x55e7457519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.928+0000 7f60b8e00640 1 --2- 172.21.15.62:0/3698019394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 0x55e745751f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.928+0000 7f60ba200640 1 -- 172.21.15.62:0/3698019394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 msgr2=0x55e745751f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.928+0000 7f60ba200640 1 --2- 172.21.15.62:0/3698019394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 0x55e745751f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.928+0000 7f60ba200640 1 -- 172.21.15.62:0/3698019394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74579a390 msgr2=0x55e7457519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.928+0000 7f60ba200640 1 --2- 172.21.15.62:0/3698019394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74579a390 0x55e7457519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.928+0000 7f60ba200640 1 -- 172.21.15.62:0/3698019394 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e745681230 con 0x55e74574c8b0 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.928+0000 7f60b8e00640 1 --2- 172.21.15.62:0/3698019394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 0x55e745751f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:22.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.928+0000 7f60ba200640 1 --2- 172.21.15.62:0/3698019394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74574c8b0 0x55e74577c6a0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f60ac037040 tx=0x7f60ac02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f796d65dc14fe146 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.929+0000 7f60a6000640 1 -- 172.21.15.62:0/3698019394 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac034440 con 0x55e74574c8b0 2024-03-23T11:29:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.929+0000 7f60a6000640 1 -- 172.21.15.62:0/3698019394 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac0345c0 con 0x55e74574c8b0 2024-03-23T11:29:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.929+0000 7f60a6000640 1 -- 172.21.15.62:0/3698019394 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac034970 con 0x55e74574c8b0 2024-03-23T11:29:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.929+0000 7f60bedc6740 1 -- 172.21.15.62:0/3698019394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74574c8b0 msgr2=0x55e74577c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.929+0000 7f60bedc6740 1 --2- 172.21.15.62:0/3698019394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74574c8b0 0x55e74577c6a0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f60ac037040 tx=0x7f60ac02e000 comp rx=0 tx=0).stop 2024-03-23T11:29:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.929+0000 7f60bedc6740 1 -- 172.21.15.62:0/3698019394 shutdown_connections 2024-03-23T11:29:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.929+0000 7f60bedc6740 1 --2- 172.21.15.62:0/3698019394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74574c8b0 0x55e74577c6a0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.929+0000 7f60bedc6740 1 --2- 172.21.15.62:0/3698019394 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 0x55e745751f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.929+0000 7f60bedc6740 1 --2- 172.21.15.62:0/3698019394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74579a390 0x55e7457519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.929+0000 7f60bedc6740 1 -- 172.21.15.62:0/3698019394 >> 172.21.15.62:0/3698019394 conn(0x55e74574eb20 msgr2=0x55e7457976a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:22.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.930+0000 7f60bedc6740 1 -- 172.21.15.62:0/3698019394 shutdown_connections 2024-03-23T11:29:22.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.930+0000 7f60bedc6740 1 -- 172.21.15.62:0/3698019394 wait complete. 2024-03-23T11:29:22.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.930+0000 7f60bedc6740 1 -- start start 2024-03-23T11:29:22.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.930+0000 7f60bedc6740 1 Processor -- start 2024-03-23T11:29:22.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.931+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 0x55e74579f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.931+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74579a390 0x55e745798d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.931+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e7457740d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.931+0000 7f60bedc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7456578c0 con 0x55e74577a820 2024-03-23T11:29:22.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.931+0000 7f60bedc6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e745657560 con 0x55e74579a390 2024-03-23T11:29:22.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.931+0000 7f60bedc6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e74541df20 con 0x55e745750410 2024-03-23T11:29:22.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60bedc6740 10 client.0 _ll_register_callbacks cb 0x55e7456af800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:29:22.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60ba200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e7457740d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60ba200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e7457740d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54532/0 (socket says 172.21.15.62:54532) 2024-03-23T11:29:22.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60ba200640 1 -- 172.21.15.62:0/902019501 learned_addr learned my addr 172.21.15.62:0/902019501 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:22.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60b9800640 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 0x55e74579f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60ba200640 1 -- 172.21.15.62:0/902019501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 msgr2=0x55e74579f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60ba200640 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 0x55e74579f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60ba200640 1 -- 172.21.15.62:0/902019501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74579a390 msgr2=0x55e745798d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60ba200640 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74579a390 0x55e745798d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60ba200640 1 -- 172.21.15.62:0/902019501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b40cc220 con 0x55e74577a820 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60b9800640 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 0x55e74579f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f60ba200640 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e7457740d0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009eb0 tx=0x7f60ac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef379e249c0e6d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.932+0000 7f609aa00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.933+0000 7f609aa00640 1 -- 172.21.15.62:0/902019501 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac003b90 con 0x55e74577a820 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.933+0000 7f609aa00640 1 -- 172.21.15.62:0/902019501 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60ac003cf0 con 0x55e74577a820 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.933+0000 7f60bedc6740 1 -- 172.21.15.62:0/902019501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e7456e0280 con 0x55e74577a820 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.933+0000 7f609aa00640 1 -- 172.21.15.62:0/902019501 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60ac007080 con 0x55e74577a820 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.933+0000 7f60bedc6740 10 client.8421 fetch_fsmap learned FSMap version 78 2024-03-23T11:29:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.933+0000 7f60bedc6740 1 -- 172.21.15.62:0/902019501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55e7456b4810 con 0x55e74577a820 2024-03-23T11:29:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.933+0000 7f609aa00640 1 -- 172.21.15.62:0/902019501 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e7456e0280 con 0x55e74577a820 2024-03-23T11:29:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.933+0000 7f609aa00640 1 -- 172.21.15.62:0/902019501 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f60ac004030 con 0x55e74577a820 2024-03-23T11:29:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.933+0000 7f60bedc6740 10 client.8421 fetch_fsmap finished waiting for FSMap version 78 2024-03-23T11:29:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.933+0000 7f60bedc6740 10 client.8421 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:29:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.934+0000 7f60bedc6740 20 client.8421 populate_metadata read hostname 'smithi062' 2024-03-23T11:29:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.934+0000 7f60bedc6740 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55e7457a7c40 0x55e74579bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.934+0000 7f60bedc6740 4 client.8421 mds_command: new command op to 8271 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:29:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.934+0000 7f60bedc6740 1 -- 172.21.15.62:0/902019501 --> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7456a89d0 con 0x55e7457a7c40 2024-03-23T11:29:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.934+0000 7f60b9800640 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55e7457a7c40 0x55e74579bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.934+0000 7f609aa00640 10 client.8421 ms_handle_connect on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.934+0000 7f60b9800640 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55e7457a7c40 0x55e74579bd70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93fbe280abb99dfb server_cookie=c6cfe3ced9ade19c in_seq=0 out_seq=0 2024-03-23T11:29:22.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.935+0000 7f609aa00640 1 -- 172.21.15.62:0/902019501 <== mds.0 v2:172.21.15.62:6832/692790652 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55e7456a89d0 con 0x55e7457a7c40 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.935+0000 7f609aa00640 10 client.8421 handle_command_reply: tid=0 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.935+0000 7f60bedc6740 1 client.8421 shutdown 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.935+0000 7f60bedc6740 1 -- 172.21.15.62:0/902019501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 msgr2=0x55e7457740d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.935+0000 7f60bedc6740 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e7457740d0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009eb0 tx=0x7f60ac004ab0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.935+0000 7f60bedc6740 1 -- 172.21.15.62:0/902019501 shutdown_connections 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.935+0000 7f60bedc6740 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55e7457a7c40 0x55e74579bd70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.935+0000 7f60bedc6740 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e7457740d0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.935+0000 7f60bedc6740 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74579a390 0x55e745798d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.935+0000 7f60bedc6740 1 --2- 172.21.15.62:0/902019501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e745750410 0x55e74579f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.936+0000 7f60bedc6740 1 -- 172.21.15.62:0/902019501 >> 172.21.15.62:0/902019501 conn(0x55e74574c8b0 msgr2=0x55e745750c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.936+0000 7f609aa00640 0 client.8421 ms_handle_reset on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:22.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.936+0000 7f60bedc6740 1 -- 172.21.15.62:0/902019501 shutdown_connections 2024-03-23T11:29:22.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.936+0000 7f60bedc6740 1 -- 172.21.15.62:0/902019501 wait complete. 2024-03-23T11:29:22.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.936+0000 7f60bedc6740 20 client.8421 trim_cache size 0 max 16384 2024-03-23T11:29:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.952+0000 7f60bedc6740 1 Processor -- start 2024-03-23T11:29:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.952+0000 7f60bedc6740 1 -- start start 2024-03-23T11:29:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.953+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e745750410 0x55e7457524d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.953+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74577a820 0x55e745752a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.953+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e74579a390 0x55e7457731f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.953+0000 7f60bedc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e74567c950 con 0x55e745750410 2024-03-23T11:29:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.953+0000 7f60bedc6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7456b5290 con 0x55e74577a820 2024-03-23T11:29:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.953+0000 7f60bedc6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e74574c0d0 con 0x55e74579a390 2024-03-23T11:29:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.953+0000 7f60b9800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e745750410 0x55e7457524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.954+0000 7f60b9800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e745750410 0x55e7457524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54536/0 (socket says 172.21.15.62:54536) 2024-03-23T11:29:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.954+0000 7f60b9800640 1 -- 172.21.15.62:0/4034821554 learned_addr learned my addr 172.21.15.62:0/4034821554 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.954+0000 7f60b8e00640 1 --2- 172.21.15.62:0/4034821554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74577a820 0x55e745752a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.954+0000 7f60b9800640 1 -- 172.21.15.62:0/4034821554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e74579a390 msgr2=0x55e7457731f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.954+0000 7f60b9800640 1 --2- 172.21.15.62:0/4034821554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e74579a390 0x55e7457731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.954+0000 7f60b9800640 1 -- 172.21.15.62:0/4034821554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74577a820 msgr2=0x55e745752a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.954+0000 7f60b9800640 1 --2- 172.21.15.62:0/4034821554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74577a820 0x55e745752a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.954+0000 7f60b9800640 1 -- 172.21.15.62:0/4034821554 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e745656ad0 con 0x55e745750410 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.954+0000 7f60b9800640 1 --2- 172.21.15.62:0/4034821554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e745750410 0x55e7457524d0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f60a8086920 tx=0x7f60a8088040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb2c656452bb9054 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60a6a00640 1 -- 172.21.15.62:0/4034821554 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8071070 con 0x55e745750410 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60a6a00640 1 -- 172.21.15.62:0/4034821554 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a8055790 con 0x55e745750410 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60a6a00640 1 -- 172.21.15.62:0/4034821554 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a805b0b0 con 0x55e745750410 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60bedc6740 1 -- 172.21.15.62:0/4034821554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e745750410 msgr2=0x55e7457524d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60bedc6740 1 --2- 172.21.15.62:0/4034821554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e745750410 0x55e7457524d0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f60a8086920 tx=0x7f60a8088040 comp rx=0 tx=0).stop 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60bedc6740 1 -- 172.21.15.62:0/4034821554 shutdown_connections 2024-03-23T11:29:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60bedc6740 1 --2- 172.21.15.62:0/4034821554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e74579a390 0x55e7457731f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60bedc6740 1 --2- 172.21.15.62:0/4034821554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e74577a820 0x55e745752a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60bedc6740 1 --2- 172.21.15.62:0/4034821554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e745750410 0x55e7457524d0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60bedc6740 1 -- 172.21.15.62:0/4034821554 >> 172.21.15.62:0/4034821554 conn(0x55e74574c8b0 msgr2=0x55e74574cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60bedc6740 1 -- 172.21.15.62:0/4034821554 shutdown_connections 2024-03-23T11:29:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.955+0000 7f60bedc6740 1 -- 172.21.15.62:0/4034821554 wait complete. 2024-03-23T11:29:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.956+0000 7f60bedc6740 1 -- start start 2024-03-23T11:29:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.956+0000 7f60bedc6740 1 Processor -- start 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.957+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e745750410 0x55e7457524d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.957+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e745752a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.957+0000 7f60bedc6740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e74579a390 0x55e74578d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.957+0000 7f60bedc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7457fdb00 con 0x55e74577a820 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.957+0000 7f60bedc6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e74579caf0 con 0x55e745750410 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.957+0000 7f60bedc6740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7457b0ea0 con 0x55e74579a390 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.957+0000 7f60bedc6740 10 client.0 _ll_register_callbacks cb 0x7f609c0036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.957+0000 7f60b8e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e745752a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.957+0000 7f60b8e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e745752a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54546/0 (socket says 172.21.15.62:54546) 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.957+0000 7f60b8e00640 1 -- 172.21.15.62:0/2191286747 learned_addr learned my addr 172.21.15.62:0/2191286747 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f60b9800640 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e745750410 0x55e7457524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f60b8e00640 1 -- 172.21.15.62:0/2191286747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e74579a390 msgr2=0x55e74578d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f60ba200640 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e74579a390 0x55e74578d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f60b8e00640 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e74579a390 0x55e74578d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f60b8e00640 1 -- 172.21.15.62:0/2191286747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e745750410 msgr2=0x55e7457524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f60b8e00640 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e745750410 0x55e7457524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f60b8e00640 1 -- 172.21.15.62:0/2191286747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b40ad550 con 0x55e74577a820 2024-03-23T11:29:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f60b8e00640 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e745752a10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f60b40cd320 tx=0x7f60a00046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256cd4998240aa8f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f609aa00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:29:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f609aa00640 1 -- 172.21.15.62:0/2191286747 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0019040 con 0x55e74577a820 2024-03-23T11:29:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f609aa00640 1 -- 172.21.15.62:0/2191286747 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60a001f430 con 0x55e74577a820 2024-03-23T11:29:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.958+0000 7f60bedc6740 1 -- 172.21.15.62:0/2191286747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e7453aa3c0 con 0x55e74577a820 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.959+0000 7f609aa00640 1 -- 172.21.15.62:0/2191286747 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60a001f5f0 con 0x55e74577a820 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.959+0000 7f60bedc6740 10 client.8427 fetch_fsmap learned FSMap version 78 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.959+0000 7f60bedc6740 1 -- 172.21.15.62:0/2191286747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55e74567bfe0 con 0x55e74577a820 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.959+0000 7f609aa00640 1 -- 172.21.15.62:0/2191286747 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e7453aa3c0 con 0x55e74577a820 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.959+0000 7f609aa00640 1 -- 172.21.15.62:0/2191286747 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f60a001f750 con 0x55e74577a820 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.959+0000 7f60bedc6740 10 client.8427 fetch_fsmap finished waiting for FSMap version 78 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.959+0000 7f60bedc6740 10 client.8427 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.959+0000 7f60bedc6740 20 client.8427 populate_metadata read hostname 'smithi062' 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.959+0000 7f60bedc6740 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55e74578d950 0x55e7457fc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.959+0000 7f60bedc6740 4 client.8427 mds_command: new command op to 8271 tid=0[{"prefix": "status"}] 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.959+0000 7f60bedc6740 1 -- 172.21.15.62:0/2191286747 --> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e7453bb6c0 con 0x55e74578d950 2024-03-23T11:29:22.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.960+0000 7f60b9800640 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55e74578d950 0x55e7457fc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.960+0000 7f60b9800640 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55e74578d950 0x55e7457fc5c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3954e790050a5dbb server_cookie=35b3ecd61834b23e in_seq=0 out_seq=0 2024-03-23T11:29:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.960+0000 7f609aa00640 10 client.8427 ms_handle_connect on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.960+0000 7f609aa00640 1 -- 172.21.15.62:0/2191286747 <== mds.0 v2:172.21.15.62:6832/692790652 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55e7453bb6c0 con 0x55e74578d950 2024-03-23T11:29:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.960+0000 7f609aa00640 10 client.8427 handle_command_reply: tid=0 2024-03-23T11:29:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.960+0000 7f60bedc6740 1 client.8427 shutdown 2024-03-23T11:29:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f60bedc6740 1 -- 172.21.15.62:0/2191286747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 msgr2=0x55e745752a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f60bedc6740 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e745752a10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f60b40cd320 tx=0x7f60a00046f0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f60bedc6740 1 -- 172.21.15.62:0/2191286747 shutdown_connections 2024-03-23T11:29:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f60bedc6740 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55e74578d950 0x55e7457fc5c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f60bedc6740 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e74577a820 0x55e745752a10 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f60bedc6740 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e74579a390 0x55e74578d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f609aa00640 0 client.8427 ms_handle_reset on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f60bedc6740 1 --2- 172.21.15.62:0/2191286747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e745750410 0x55e7457524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f60bedc6740 1 -- 172.21.15.62:0/2191286747 >> 172.21.15.62:0/2191286747 conn(0x55e74574c8b0 msgr2=0x55e74574cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f60bedc6740 1 -- 172.21.15.62:0/2191286747 shutdown_connections 2024-03-23T11:29:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f60bedc6740 1 -- 172.21.15.62:0/2191286747 wait complete. 2024-03-23T11:29:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.961+0000 7f60bedc6740 20 client.8427 trim_cache size 0 max 16384 2024-03-23T11:29:22.957 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:29:22.957 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:29:22.957 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:29:22.957 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 8271, 2024-03-23T11:29:22.958 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:29:22.958 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:29:22.958 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:29:22.958 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 7.329290715, 2024-03-23T11:29:22.958 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 78, 2024-03-23T11:29:22.958 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 105, 2024-03-23T11:29:22.958 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 105, 2024-03-23T11:29:22.958 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 12.635713301999999 2024-03-23T11:29:22.958 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:29:22.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 -- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f609406a7a0 msgr2=0x7f609406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f609406a7a0 0x7f609406cc60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f60b4157800 tx=0x7f60a0008040 comp rx=0 tx=0).stop 2024-03-23T11:29:22.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 -- 172.21.15.62:0/3869128653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60b406c300 msgr2=0x7f60b41f48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:22.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60b406c300 0x7f60b41f48b0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f60a800c500 tx=0x7f60a800c9c0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 -- 172.21.15.62:0/3869128653 shutdown_connections 2024-03-23T11:29:22.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f609406a7a0 0x7f609406cc60 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60b40cafa0 0x7f60b40ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60b406c6e0 0x7f60b40acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 --2- 172.21.15.62:0/3869128653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60b406c300 0x7f60b41f48b0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:22.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 -- 172.21.15.62:0/3869128653 >> 172.21.15.62:0/3869128653 conn(0x7f60b40b02e0 msgr2=0x7f60b413eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:22.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 -- 172.21.15.62:0/3869128653 shutdown_connections 2024-03-23T11:29:22.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:22.962+0000 7f60bc200640 1 -- 172.21.15.62:0/3869128653 wait complete. 2024-03-23T11:29:22.970 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:29:23.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.197+0000 7fb801200640 1 Processor -- start 2024-03-23T11:29:23.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.197+0000 7fb801200640 1 -- start start 2024-03-23T11:29:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.199+0000 7fb801200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb7fc1571e0 0x7fb7fc0ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.199+0000 7fb801200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7fc1613a0 0x7fb7fc15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.199+0000 7fb801200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc06cf40 con 0x7fb7fc0ade70 2024-03-23T11:29:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.199+0000 7fb801200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7fc1575c0 con 0x7fb7fc1571e0 2024-03-23T11:29:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.199+0000 7fb801200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc06c3a0 con 0x7fb7fc1613a0 2024-03-23T11:29:23.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.200+0000 7fb7f9e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb7fc1571e0 0x7fb7fc0ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:23.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.200+0000 7fb7fa800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7fc1613a0 0x7fb7fc15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:23.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.200+0000 7fb7f9e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb7fc1571e0 0x7fb7fc0ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:56188/0 (socket says 172.21.15.62:56188) 2024-03-23T11:29:23.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.200+0000 7fb7f9e00640 1 -- 172.21.15.62:0/3626417245 learned_addr learned my addr 172.21.15.62:0/3626417245 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:23.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.201+0000 7fb7f9e00640 1 -- 172.21.15.62:0/3626417245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7fc1613a0 msgr2=0x7fb7fc15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:23.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.201+0000 7fb7f9e00640 1 --2- 172.21.15.62:0/3626417245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7fc1613a0 0x7fb7fc15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:23.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.201+0000 7fb7f9e00640 1 -- 172.21.15.62:0/3626417245 >> v1:172.21.15.120:6789/0 conn(0x7fb7fc0ade70 legacy=0x7fb7fc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:23.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.201+0000 7fb7f9400640 1 -- 172.21.15.62:0/3626417245 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539223285 0 0) 0x7fb7fc06cf40 con 0x7fb7fc0ade70 2024-03-23T11:29:23.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.201+0000 7fb7f9e00640 1 -- 172.21.15.62:0/3626417245 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc06c610 con 0x7fb7fc1571e0 2024-03-23T11:29:23.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.203+0000 7fb7f9e00640 1 --2- 172.21.15.62:0/3626417245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7fc1571e0 0x7fb7fc0ad930 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec009870 tx=0x7fb7ec02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30c7bfd7d0e9b1c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:23.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.203+0000 7fb7f9400640 1 -- 172.21.15.62:0/3626417245 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec031640 con 0x7fb7fc1571e0 2024-03-23T11:29:23.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.203+0000 7fb7f9400640 1 -- 172.21.15.62:0/3626417245 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ec0317a0 con 0x7fb7fc1571e0 2024-03-23T11:29:23.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.203+0000 7fb7f9400640 1 -- 172.21.15.62:0/3626417245 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec031b30 con 0x7fb7fc1571e0 2024-03-23T11:29:23.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.204+0000 7fb801200640 1 -- 172.21.15.62:0/3626417245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7fc1571e0 msgr2=0x7fb7fc0ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:23.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.204+0000 7fb801200640 1 --2- 172.21.15.62:0/3626417245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7fc1571e0 0x7fb7fc0ad930 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec009870 tx=0x7fb7ec02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:29:23.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.204+0000 7fb801200640 1 -- 172.21.15.62:0/3626417245 shutdown_connections 2024-03-23T11:29:23.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.204+0000 7fb801200640 1 --2- 172.21.15.62:0/3626417245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7fc1613a0 0x7fb7fc15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:23.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.204+0000 7fb801200640 1 --2- 172.21.15.62:0/3626417245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7fc1571e0 0x7fb7fc0ad930 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:23.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.204+0000 7fb801200640 1 -- 172.21.15.62:0/3626417245 >> 172.21.15.62:0/3626417245 conn(0x7fb7fc0b02e0 msgr2=0x7fb7fc0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:23.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.204+0000 7fb801200640 1 -- 172.21.15.62:0/3626417245 shutdown_connections 2024-03-23T11:29:23.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.204+0000 7fb801200640 1 -- 172.21.15.62:0/3626417245 wait complete. 2024-03-23T11:29:23.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.206+0000 7fb801200640 1 Processor -- start 2024-03-23T11:29:23.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.206+0000 7fb801200640 1 -- start start 2024-03-23T11:29:23.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.207+0000 7fb801200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7fc0ade70 0x7fb7fc1f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:23.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.207+0000 7fb801200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7fc1571e0 0x7fb7fc152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:23.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.207+0000 7fb801200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7fc1613a0 0x7fb7fc0cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:23.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.207+0000 7fb801200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc150d10 con 0x7fb7fc1613a0 2024-03-23T11:29:23.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.207+0000 7fb801200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc0ae660 con 0x7fb7fc0ade70 2024-03-23T11:29:23.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.207+0000 7fb801200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc1f6650 con 0x7fb7fc1571e0 2024-03-23T11:29:23.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.208+0000 7fb7fa800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7fc1613a0 0x7fb7fc0cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:23.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.208+0000 7fb7fa800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7fc1613a0 0x7fb7fc0cd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54556/0 (socket says 172.21.15.62:54556) 2024-03-23T11:29:23.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.208+0000 7fb7fa800640 1 -- 172.21.15.62:0/3864243203 learned_addr learned my addr 172.21.15.62:0/3864243203 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:23.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.208+0000 7fb7f9e00640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7fc0ade70 0x7fb7fc1f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:23.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.208+0000 7fb7f3e00640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7fc1571e0 0x7fb7fc152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:23.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.208+0000 7fb7fa800640 1 -- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7fc1571e0 msgr2=0x7fb7fc152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:23.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.208+0000 7fb7fa800640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7fc1571e0 0x7fb7fc152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:23.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.208+0000 7fb7fa800640 1 -- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7fc0ade70 msgr2=0x7fb7fc1f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:23.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.208+0000 7fb7fa800640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7fc0ade70 0x7fb7fc1f57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:23.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.208+0000 7fb7fa800640 1 -- 172.21.15.62:0/3864243203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec0094d0 con 0x7fb7fc1613a0 2024-03-23T11:29:23.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.209+0000 7fb7fa800640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7fc1613a0 0x7fb7fc0cd2e0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fb7f400efc0 tx=0x7fb7f400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:23.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.209+0000 7fb7f2000640 1 -- 172.21.15.62:0/3864243203 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4014480 con 0x7fb7fc1613a0 2024-03-23T11:29:23.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.209+0000 7fb7f2000640 1 -- 172.21.15.62:0/3864243203 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f400f040 con 0x7fb7fc1613a0 2024-03-23T11:29:23.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.209+0000 7fb7f2000640 1 -- 172.21.15.62:0/3864243203 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4014820 con 0x7fb7fc1613a0 2024-03-23T11:29:23.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.209+0000 7fb801200640 1 -- 172.21.15.62:0/3864243203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc1f6090 con 0x7fb7fc1613a0 2024-03-23T11:29:23.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.209+0000 7fb801200640 1 -- 172.21.15.62:0/3864243203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc0ae2f0 con 0x7fb7fc1613a0 2024-03-23T11:29:23.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.210+0000 7fb801200640 1 -- 172.21.15.62:0/3864243203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc0ae810 con 0x7fb7fc1613a0 2024-03-23T11:29:23.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.211+0000 7fb7f2000640 1 -- 172.21.15.62:0/3864243203 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb7f4014980 con 0x7fb7fc1613a0 2024-03-23T11:29:23.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.212+0000 7fb7f2000640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb7d806a7a0 0x7fb7d806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:23.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.212+0000 7fb7f2000640 1 -- 172.21.15.62:0/3864243203 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7f4089b70 con 0x7fb7fc1613a0 2024-03-23T11:29:23.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.213+0000 7fb7f9e00640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb7d806a7a0 0x7fb7d806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:23.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.214+0000 7fb7f9e00640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb7d806a7a0 0x7fb7d806cc60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec036000 tx=0x7fb7ec03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:23.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.217+0000 7fb7f2000640 1 -- 172.21.15.62:0/3864243203 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7fc0ae810 con 0x7fb7fc1613a0 2024-03-23T11:29:23.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.453+0000 7fb801200640 1 -- 172.21.15.62:0/3864243203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb7fc0ae7d0 con 0x7fb7fc1613a0 2024-03-23T11:29:23.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.453+0000 7fb7f2000640 1 -- 172.21.15.62:0/3864243203 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v78) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb7f4018070 con 0x7fb7fc1613a0 2024-03-23T11:29:23.444 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:29:23.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 -- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb7d806a7a0 msgr2=0x7fb7d806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:23.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb7d806a7a0 0x7fb7d806cc60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec036000 tx=0x7fb7ec03c040 comp rx=0 tx=0).stop 2024-03-23T11:29:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 -- 172.21.15.62:0/3864243203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7fc1613a0 msgr2=0x7fb7fc0cd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7fc1613a0 0x7fb7fc0cd2e0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fb7f400efc0 tx=0x7fb7f400c490 comp rx=0 tx=0).stop 2024-03-23T11:29:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 -- 172.21.15.62:0/3864243203 shutdown_connections 2024-03-23T11:29:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb7d806a7a0 0x7fb7d806cc60 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7fc1613a0 0x7fb7fc0cd2e0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb7fc1571e0 0x7fb7fc152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 --2- 172.21.15.62:0/3864243203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7fc0ade70 0x7fb7fc1f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 -- 172.21.15.62:0/3864243203 >> 172.21.15.62:0/3864243203 conn(0x7fb7fc0b02e0 msgr2=0x7fb7fc15f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 -- 172.21.15.62:0/3864243203 shutdown_connections 2024-03-23T11:29:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:23.456+0000 7fb801200640 1 -- 172.21.15.62:0/3864243203 wait complete. 2024-03-23T11:29:23.468 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:29:23.468 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:29:23.468 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:29:23.468 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:29:23.468 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:23.468 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:29:23.468 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:29:23.468 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:29:23.468 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:29:23.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:23.468 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:29:23.525 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:29:23.525 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:29:23.525 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:29:23.525 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:23.525 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:29:23.526 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:23.526 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:29:23.526 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:29:23.526 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:29:23.526 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:29:23.526 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:29:23.527 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:29:23.527 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:29:23.527 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:29:23.527 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:29:23.527 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:29:23.527 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:29:23.527 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 41993sec preferred_lft 41993sec 2024-03-23T11:29:23.527 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:29:23.527 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:23.528 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:29:23.528 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:29:23.528 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:29:23.528 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:23.528 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:29:23.528 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:23.528 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:29:23.528 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:29:23.529 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:29:23.529 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:23.529 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:29:23.529 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:29:23.529 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:29:23.529 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:23.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:23.530 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:29:23.623 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:29:23.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:23.628 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:29:23.685 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:29:23.686 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:29:23.686 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:29:23.686 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:29:23.686 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:29:23.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:23.687 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:29:23.687 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:29:23.687 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:29:23.687 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:29:23.687 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:29:23.687 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:29:23.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:23.876 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:29:23.928 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:29:23.929 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:29:23.930 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:29:23.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:23.930 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:29:24.022 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:29:24.022 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:29:24.023 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:24.078 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:29:24.078 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:29:24.079 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:24.133 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:29:24.231 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:29:24.231 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:29:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:29:24.329 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:29:24.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:24.330 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:29:24.387 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:29:24.387 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:29:24.392 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:29:24.468 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:29:24.468 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:29:24.543 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:29:24.547 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:29:24.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:24.547 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:29:24.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:29:24.583+0000 7fdec6d87480 -1 init, newargv = 0x561ac6921a30 newargc=15 2024-03-23T11:29:24.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[72544]: starting ceph client 2024-03-23T11:29:24.584 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[72544]: starting fuse 2024-03-23T11:29:24.594 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:29:24.594 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:29:24.594 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:29:24.594 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:29:24.594 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:29:24.594 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:29:24.594 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:29:24.595 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:29:24.596 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:29:24.596 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:29:24.596 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:24.596 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:29:24.596 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:29:24.800 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.72544.asok 2024-03-23T11:29:24.979 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.72544.asok 2024-03-23T11:29:24.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:24.979 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.72544.asok status 2024-03-23T11:29:25.228 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:29:25.228 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:29:25.228 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:29:25.228 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "72544", 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 8439, 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:29:25.229 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 8439 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1593162478 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1593162478 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:25.230 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.8439 192.168.0.1:0/1593162478", 2024-03-23T11:29:25.231 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/1593162478", 2024-03-23T11:29:25.231 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:29:25.231 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 78, 2024-03-23T11:29:25.231 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 105, 2024-03-23T11:29:25.231 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:29:25.231 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:29:25.231 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:29:25.231 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:29:25.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:25.245 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:29:25.245 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:25.298 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:29:25.298 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:25.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:25.298 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:25.391 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:29:25.391 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:25.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:25.445 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file1_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-03-23T11:29:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:6+0 records in 2024-03-23T11:29:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:6+0 records out 2024-03-23T11:29:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.180528 s, 34.9 MB/s 2024-03-23T11:29:25.693 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:29:25.693 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:25.753 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:29:25.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.986+0000 7fe159a00640 1 Processor -- start 2024-03-23T11:29:25.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.986+0000 7fe159a00640 1 -- start start 2024-03-23T11:29:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.988+0000 7fe159a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe154160310 0x7fe1540ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.988+0000 7fe159a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1541608d0 con 0x7fe1540ce250 2024-03-23T11:29:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.988+0000 7fe159a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe154160bb0 con 0x7fe15415c830 2024-03-23T11:29:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.988+0000 7fe159a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1540addb0 con 0x7fe154160310 2024-03-23T11:29:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.988+0000 7fe153400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe154160310 0x7fe1540ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:25.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.988+0000 7fe159000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe1540ce250 0x7fe15415d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47128/0 (socket says 172.21.15.62:47128) 2024-03-23T11:29:25.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.988+0000 7fe159000640 1 -- 172.21.15.62:0/2834513233 learned_addr learned my addr 172.21.15.62:0/2834513233 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:25.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.989+0000 7fe153400640 1 -- 172.21.15.62:0/2834513233 >> v1:172.21.15.120:6789/0 conn(0x7fe15415c830 legacy=0x7fe1540ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:25.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.989+0000 7fe152a00640 1 -- 172.21.15.62:0/2834513233 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857010378 0 0) 0x7fe1541608d0 con 0x7fe1540ce250 2024-03-23T11:29:25.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.989+0000 7fe153400640 1 -- 172.21.15.62:0/2834513233 >> v1:172.21.15.62:6789/0 conn(0x7fe1540ce250 legacy=0x7fe15415d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:25.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.989+0000 7fe153400640 1 -- 172.21.15.62:0/2834513233 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe154160e90 con 0x7fe154160310 2024-03-23T11:29:25.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.990+0000 7fe153400640 1 --2- 172.21.15.62:0/2834513233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe154160310 0x7fe1540ad310 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fe1400097a0 tx=0x7fe14002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5781d48e3b4dbd0d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:25.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.990+0000 7fe152a00640 1 -- 172.21.15.62:0/2834513233 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140030610 con 0x7fe154160310 2024-03-23T11:29:25.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.990+0000 7fe152a00640 1 -- 172.21.15.62:0/2834513233 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe140030770 con 0x7fe154160310 2024-03-23T11:29:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.990+0000 7fe152a00640 1 -- 172.21.15.62:0/2834513233 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140030b00 con 0x7fe154160310 2024-03-23T11:29:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.991+0000 7fe159a00640 1 -- 172.21.15.62:0/2834513233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe154160310 msgr2=0x7fe1540ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.991+0000 7fe159a00640 1 --2- 172.21.15.62:0/2834513233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe154160310 0x7fe1540ad310 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fe1400097a0 tx=0x7fe14002b960 comp rx=0 tx=0).stop 2024-03-23T11:29:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.991+0000 7fe159a00640 1 -- 172.21.15.62:0/2834513233 shutdown_connections 2024-03-23T11:29:25.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.991+0000 7fe159a00640 1 --2- 172.21.15.62:0/2834513233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe154160310 0x7fe1540ad310 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:25.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.991+0000 7fe159a00640 1 -- 172.21.15.62:0/2834513233 >> 172.21.15.62:0/2834513233 conn(0x7fe154066050 msgr2=0x7fe154069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:25.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.991+0000 7fe159a00640 1 -- 172.21.15.62:0/2834513233 shutdown_connections 2024-03-23T11:29:25.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.991+0000 7fe159a00640 1 -- 172.21.15.62:0/2834513233 wait complete. 2024-03-23T11:29:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.993+0000 7fe159a00640 1 Processor -- start 2024-03-23T11:29:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.993+0000 7fe159a00640 1 -- start start 2024-03-23T11:29:25.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.994+0000 7fe159a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1540ce250 0x7fe154157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:25.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.994+0000 7fe159a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe15415c830 0x7fe15415f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.994+0000 7fe159a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe154160310 0x7fe1541f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.994+0000 7fe159a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe154157560 con 0x7fe1540ce250 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.994+0000 7fe159a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15415ed00 con 0x7fe15415c830 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.994+0000 7fe159a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15415f6d0 con 0x7fe154160310 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.994+0000 7fe153e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1540ce250 0x7fe154157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.994+0000 7fe153e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1540ce250 0x7fe154157020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54584/0 (socket says 172.21.15.62:54584) 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.994+0000 7fe153e00640 1 -- 172.21.15.62:0/2428797721 learned_addr learned my addr 172.21.15.62:0/2428797721 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.995+0000 7fe153e00640 1 -- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe154160310 msgr2=0x7fe1541f21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.995+0000 7fe153e00640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe154160310 0x7fe1541f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.995+0000 7fe153400640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe15415c830 0x7fe15415f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.995+0000 7fe153e00640 1 -- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe15415c830 msgr2=0x7fe15415f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.995+0000 7fe153e00640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe15415c830 0x7fe15415f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.995+0000 7fe153e00640 1 -- 172.21.15.62:0/2428797721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe140009400 con 0x7fe1540ce250 2024-03-23T11:29:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.995+0000 7fe153400640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe15415c830 0x7fe15415f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:25.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.995+0000 7fe153e00640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1540ce250 0x7fe154157020 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fe14400c450 tx=0x7fe14400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:25.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.995+0000 7fe14be00640 1 -- 172.21.15.62:0/2428797721 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1440052d0 con 0x7fe1540ce250 2024-03-23T11:29:25.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.995+0000 7fe14be00640 1 -- 172.21.15.62:0/2428797721 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe144005430 con 0x7fe1540ce250 2024-03-23T11:29:25.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.995+0000 7fe14be00640 1 -- 172.21.15.62:0/2428797721 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe144012870 con 0x7fe1540ce250 2024-03-23T11:29:25.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.996+0000 7fe159a00640 1 -- 172.21.15.62:0/2428797721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15415e0f0 con 0x7fe1540ce250 2024-03-23T11:29:25.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.996+0000 7fe159a00640 1 -- 172.21.15.62:0/2428797721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15415e6b0 con 0x7fe1540ce250 2024-03-23T11:29:25.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.997+0000 7fe159a00640 1 -- 172.21.15.62:0/2428797721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe154066bb0 con 0x7fe1540ce250 2024-03-23T11:29:25.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.998+0000 7fe14be00640 1 -- 172.21.15.62:0/2428797721 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe1440129d0 con 0x7fe1540ce250 2024-03-23T11:29:25.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.999+0000 7fe14be00640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe13406a7a0 0x7fe13406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:25.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.999+0000 7fe14be00640 1 -- 172.21.15.62:0/2428797721 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1440889a0 con 0x7fe1540ce250 2024-03-23T11:29:25.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:25.999+0000 7fe153400640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe13406a7a0 0x7fe13406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:25.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.001+0000 7fe153400640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe13406a7a0 0x7fe13406cc60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe140036000 tx=0x7fe14003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:25.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.004+0000 7fe14be00640 1 -- 172.21.15.62:0/2428797721 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe144053d20 con 0x7fe1540ce250 2024-03-23T11:29:26.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.204+0000 7fe159a00640 1 -- 172.21.15.62:0/2428797721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1540b0a90 con 0x7fe1540ce250 2024-03-23T11:29:26.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.207+0000 7fe14be00640 1 -- 172.21.15.62:0/2428797721 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16051 (secure 0 0 0) 0x7fe144057340 con 0x7fe1540ce250 2024-03-23T11:29:26.198 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:26.198 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":105,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:29:16.631279+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:29:13.817605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-23T11:29:14.652617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:29:26.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.210+0000 7fe159a00640 1 -- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe13406a7a0 msgr2=0x7fe13406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.210+0000 7fe159a00640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe13406a7a0 0x7fe13406cc60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe140036000 tx=0x7fe14003b040 comp rx=0 tx=0).stop 2024-03-23T11:29:26.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.210+0000 7fe159a00640 1 -- 172.21.15.62:0/2428797721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1540ce250 msgr2=0x7fe154157020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.210+0000 7fe159a00640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1540ce250 0x7fe154157020 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fe14400c450 tx=0x7fe14400c910 comp rx=0 tx=0).stop 2024-03-23T11:29:26.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.210+0000 7fe159a00640 1 -- 172.21.15.62:0/2428797721 shutdown_connections 2024-03-23T11:29:26.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.210+0000 7fe159a00640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe13406a7a0 0x7fe13406cc60 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.210+0000 7fe159a00640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe154160310 0x7fe1541f21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.210+0000 7fe159a00640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe15415c830 0x7fe15415f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.210+0000 7fe159a00640 1 --2- 172.21.15.62:0/2428797721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1540ce250 0x7fe154157020 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.210+0000 7fe159a00640 1 -- 172.21.15.62:0/2428797721 >> 172.21.15.62:0/2428797721 conn(0x7fe154066050 msgr2=0x7fe1540adf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:26.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.211+0000 7fe159a00640 1 -- 172.21.15.62:0/2428797721 shutdown_connections 2024-03-23T11:29:26.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.211+0000 7fe159a00640 1 -- 172.21.15.62:0/2428797721 wait complete. 2024-03-23T11:29:26.222 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:29:13.817605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:29:14.652617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:29:26.223 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:29:26.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.459+0000 7fb0dfe00640 1 Processor -- start 2024-03-23T11:29:26.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.460+0000 7fb0dfe00640 1 -- start start 2024-03-23T11:29:26.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.461+0000 7fb0dfe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb0e015fcf0 0x7fb0e00d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.461+0000 7fb0dfe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e00d14f0 con 0x7fb0e0160420 2024-03-23T11:29:26.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.461+0000 7fb0dfe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e00d3540 con 0x7fb0e00b6dc0 2024-03-23T11:29:26.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.461+0000 7fb0dfe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e00ce510 con 0x7fb0e015fcf0 2024-03-23T11:29:26.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.462+0000 7fb0d7e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb0e0160420 0x7fb0e00d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47132/0 (socket says 172.21.15.62:47132) 2024-03-23T11:29:26.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.462+0000 7fb0d7e00640 1 -- 172.21.15.62:0/3742695321 learned_addr learned my addr 172.21.15.62:0/3742695321 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:26.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.462+0000 7fb0dea00640 1 --2- 172.21.15.62:0/3742695321 >> v2:172.21.15.120:3300/0 conn(0x7fb0e015fcf0 0x7fb0e00d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.462+0000 7fb0de000640 1 -- 172.21.15.62:0/3742695321 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020045666 0 0) 0x7fb0e00d14f0 con 0x7fb0e0160420 2024-03-23T11:29:26.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.462+0000 7fb0de000640 1 -- 172.21.15.62:0/3742695321 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c4003520 con 0x7fb0e0160420 2024-03-23T11:29:26.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.463+0000 7fb0dea00640 1 -- 172.21.15.62:0/3742695321 >> v1:172.21.15.120:6789/0 conn(0x7fb0e00b6dc0 legacy=0x7fb0e00d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.463+0000 7fb0dea00640 1 -- 172.21.15.62:0/3742695321 >> v1:172.21.15.62:6789/0 conn(0x7fb0e0160420 legacy=0x7fb0e00d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.463+0000 7fb0de000640 1 -- 172.21.15.62:0/3742695321 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 206582889 0 0) 0x7fb0c4003520 con 0x7fb0e0160420 2024-03-23T11:29:26.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.463+0000 7fb0dea00640 1 -- 172.21.15.62:0/3742695321 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e00cdc70 con 0x7fb0e015fcf0 2024-03-23T11:29:26.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.464+0000 7fb0dea00640 1 --2- 172.21.15.62:0/3742695321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e015fcf0 0x7fb0e00d3000 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00097a0 tx=0x7fb0d002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c0184a5df53b58d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:26.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.465+0000 7fb0de000640 1 -- 172.21.15.62:0/3742695321 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8001670 con 0x7fb0e015fcf0 2024-03-23T11:29:26.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.465+0000 7fb0de000640 1 -- 172.21.15.62:0/3742695321 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d002e050 con 0x7fb0e015fcf0 2024-03-23T11:29:26.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.465+0000 7fb0de000640 1 -- 172.21.15.62:0/3742695321 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0030730 con 0x7fb0e015fcf0 2024-03-23T11:29:26.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.466+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3742695321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e015fcf0 msgr2=0x7fb0e00d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.466+0000 7fb0dfe00640 1 --2- 172.21.15.62:0/3742695321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e015fcf0 0x7fb0e00d3000 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00097a0 tx=0x7fb0d002b790 comp rx=0 tx=0).stop 2024-03-23T11:29:26.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.466+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3742695321 shutdown_connections 2024-03-23T11:29:26.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.466+0000 7fb0dfe00640 1 --2- 172.21.15.62:0/3742695321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e015fcf0 0x7fb0e00d3000 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.466+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3742695321 >> 172.21.15.62:0/3742695321 conn(0x7fb0e0089f60 msgr2=0x7fb0e00d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:26.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.466+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3742695321 shutdown_connections 2024-03-23T11:29:26.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.466+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3742695321 wait complete. 2024-03-23T11:29:26.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.468+0000 7fb0dfe00640 1 Processor -- start 2024-03-23T11:29:26.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.468+0000 7fb0dfe00640 1 -- start start 2024-03-23T11:29:26.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.469+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0e00b6dc0 0x7fb0e00c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.469+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e015fcf0 0x7fb0e015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.469+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0e0160420 0x7fb0e00c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.469+0000 7fb0dfe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e00c8be0 con 0x7fb0e0160420 2024-03-23T11:29:26.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.469+0000 7fb0dfe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e00a0460 con 0x7fb0e015fcf0 2024-03-23T11:29:26.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.469+0000 7fb0dfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e015b1d0 con 0x7fb0e00b6dc0 2024-03-23T11:29:26.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.470+0000 7fb0df400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0e0160420 0x7fb0e00c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.470+0000 7fb0df400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0e0160420 0x7fb0e00c7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54596/0 (socket says 172.21.15.62:54596) 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.470+0000 7fb0df400640 1 -- 172.21.15.62:0/3940169602 learned_addr learned my addr 172.21.15.62:0/3940169602 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.470+0000 7fb0d7e00640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e015fcf0 0x7fb0e015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.470+0000 7fb0df400640 1 -- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0e00b6dc0 msgr2=0x7fb0e00c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.470+0000 7fb0dea00640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0e00b6dc0 0x7fb0e00c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.470+0000 7fb0df400640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0e00b6dc0 0x7fb0e00c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.470+0000 7fb0df400640 1 -- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e015fcf0 msgr2=0x7fb0e015ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.470+0000 7fb0df400640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e015fcf0 0x7fb0e015ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.470+0000 7fb0df400640 1 -- 172.21.15.62:0/3940169602 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d0009400 con 0x7fb0e0160420 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.470+0000 7fb0d7e00640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e015fcf0 0x7fb0e015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.471+0000 7fb0df400640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0e0160420 0x7fb0e00c7670 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800c7a0 tx=0x7fb0d800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.471+0000 7fb0d7400640 1 -- 172.21.15.62:0/3940169602 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8003b70 con 0x7fb0e0160420 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.471+0000 7fb0d7400640 1 -- 172.21.15.62:0/3940169602 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d8003cf0 con 0x7fb0e0160420 2024-03-23T11:29:26.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.471+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3940169602 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e00c65d0 con 0x7fb0e0160420 2024-03-23T11:29:26.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.471+0000 7fb0d7400640 1 -- 172.21.15.62:0/3940169602 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d800aa90 con 0x7fb0e0160420 2024-03-23T11:29:26.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.471+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3940169602 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e00c6b90 con 0x7fb0e0160420 2024-03-23T11:29:26.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.472+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3940169602 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e0066bb0 con 0x7fb0e0160420 2024-03-23T11:29:26.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.473+0000 7fb0d7400640 1 -- 172.21.15.62:0/3940169602 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb0d800abf0 con 0x7fb0e0160420 2024-03-23T11:29:26.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.474+0000 7fb0d7400640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0bc06a7a0 0x7fb0bc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.474+0000 7fb0d7400640 1 -- 172.21.15.62:0/3940169602 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0d8088ca0 con 0x7fb0e0160420 2024-03-23T11:29:26.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.475+0000 7fb0dea00640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0bc06a7a0 0x7fb0bc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.476+0000 7fb0dea00640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0bc06a7a0 0x7fb0bc06cc60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0034000 tx=0x7fb0d002f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:26.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.479+0000 7fb0d7400640 1 -- 172.21.15.62:0/3940169602 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb0d8054020 con 0x7fb0e0160420 2024-03-23T11:29:26.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.675+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3940169602 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb0e00cea40 con 0x7fb0e0160420 2024-03-23T11:29:26.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.676+0000 7fb0d7400640 1 -- 172.21.15.62:0/3940169602 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb0d8003e60 con 0x7fb0e0160420 2024-03-23T11:29:26.667 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:29:26.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0bc06a7a0 msgr2=0x7fb0bc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0bc06a7a0 0x7fb0bc06cc60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0034000 tx=0x7fb0d002f000 comp rx=0 tx=0).stop 2024-03-23T11:29:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3940169602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0e0160420 msgr2=0x7fb0e00c7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0e0160420 0x7fb0e00c7670 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800c7a0 tx=0x7fb0d800cc60 comp rx=0 tx=0).stop 2024-03-23T11:29:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3940169602 shutdown_connections 2024-03-23T11:29:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0bc06a7a0 0x7fb0bc06cc60 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0e0160420 0x7fb0e00c7670 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e015fcf0 0x7fb0e015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 --2- 172.21.15.62:0/3940169602 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb0e00b6dc0 0x7fb0e00c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3940169602 >> 172.21.15.62:0/3940169602 conn(0x7fb0e0089f60 msgr2=0x7fb0e00cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3940169602 shutdown_connections 2024-03-23T11:29:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.679+0000 7fb0dfe00640 1 -- 172.21.15.62:0/3940169602 wait complete. 2024-03-23T11:29:26.690 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:29:26.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.924+0000 7f4602200640 1 Processor -- start 2024-03-23T11:29:26.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.925+0000 7f4602200640 1 -- start start 2024-03-23T11:29:26.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.926+0000 7f4602200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc06c300 0x7f45fc0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.926+0000 7f4602200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f45fc0cafa0 0x7f45fc0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.926+0000 7f4602200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc0cbc60 con 0x7f45fc06c6e0 2024-03-23T11:29:26.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.926+0000 7f4602200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45fc0cbf40 con 0x7f45fc0cafa0 2024-03-23T11:29:26.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.926+0000 7f4602200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0cc220 con 0x7f45fc06c300 2024-03-23T11:29:26.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.927+0000 7f45fbe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f45fc06c6e0 0x7f45fc06cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47144/0 (socket says 172.21.15.62:47144) 2024-03-23T11:29:26.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.927+0000 7f45fbe00640 1 -- 172.21.15.62:0/2694790779 learned_addr learned my addr 172.21.15.62:0/2694790779 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:26.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.927+0000 7f45f2a00640 1 --2- 172.21.15.62:0/2694790779 >> v2:172.21.15.120:3300/0 conn(0x7f45fc0cafa0 0x7f45fc0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.927+0000 7f45fb400640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc06c300 0x7f45fc0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.927+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609183255 0 0) 0x7f45fc0cbc60 con 0x7f45fc06c6e0 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.927+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e0003520 con 0x7f45fc06c6e0 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3441309198 0 0) 0x7f45e0003520 con 0x7f45fc06c6e0 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc06c300 msgr2=0x7f45fc0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc06c300 0x7f45fc0ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 >> v2:172.21.15.120:3300/0 conn(0x7f45fc0cafa0 msgr2=0x7f45fc0cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 --2- 172.21.15.62:0/2694790779 >> v2:172.21.15.120:3300/0 conn(0x7f45fc0cafa0 0x7f45fc0cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc0cc500 con 0x7f45fc06c6e0 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f45f4001670 con 0x7f45fc06c6e0 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 >> v1:172.21.15.62:6789/0 conn(0x7f45fc06c6e0 legacy=0x7f45fc06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 0x7f45e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45e00047c0 0x7f45e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc06c300 0x7f45e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0cbf40 con 0x7f45e0003970 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0cc220 con 0x7f45e00047c0 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e000b980 con 0x7f45fc06c300 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.928+0000 7f45fb400640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 0x7f45e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.929+0000 7f45f2a00640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45e00047c0 0x7f45e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.929+0000 7f45fbe00640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc06c300 0x7f45e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.929+0000 7f45fb400640 1 -- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc06c300 msgr2=0x7f45e0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.929+0000 7f45fb400640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc06c300 0x7f45e0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.929+0000 7f45fb400640 1 -- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45e00047c0 msgr2=0x7f45e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.929+0000 7f45fb400640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45e00047c0 0x7f45e0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.929+0000 7f45fb400640 1 -- 172.21.15.62:0/2694790779 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e0003520 con 0x7f45e0003970 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.929+0000 7f45f2a00640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45e00047c0 0x7f45e0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.929+0000 7f45fbe00640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc06c300 0x7f45e0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.930+0000 7f45fb400640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 0x7f45e0003d90 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f45ec00e170 tx=0x7f45e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3287fa9a308bf5a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.930+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec02cbf0 con 0x7f45e0003970 2024-03-23T11:29:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.930+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec00f040 con 0x7f45e0003970 2024-03-23T11:29:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.930+0000 7f45faa00640 1 -- 172.21.15.62:0/2694790779 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec034440 con 0x7f45e0003970 2024-03-23T11:29:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.931+0000 7f4602200640 1 -- 172.21.15.62:0/2694790779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 msgr2=0x7f45e0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.931+0000 7f4602200640 1 --2- 172.21.15.62:0/2694790779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 0x7f45e0003d90 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f45ec00e170 tx=0x7f45e00057f0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.931+0000 7f45fbe00640 1 -- 172.21.15.62:0/2694790779 reap_dead start 2024-03-23T11:29:26.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.931+0000 7f4602200640 1 -- 172.21.15.62:0/2694790779 shutdown_connections 2024-03-23T11:29:26.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.931+0000 7f4602200640 1 -- 172.21.15.62:0/2694790779 >> 172.21.15.62:0/2694790779 conn(0x7f45fc0b02e0 msgr2=0x7f45fc0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:26.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.931+0000 7f4602200640 1 -- 172.21.15.62:0/2694790779 shutdown_connections 2024-03-23T11:29:26.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.931+0000 7f4602200640 1 -- 172.21.15.62:0/2694790779 wait complete. 2024-03-23T11:29:26.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.933+0000 7f4602200640 1 Processor -- start 2024-03-23T11:29:26.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.933+0000 7f4602200640 1 -- start start 2024-03-23T11:29:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.934+0000 7f4602200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 0x7f45fc157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.934+0000 7f4602200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45fc152fe0 0x7f45fc153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.934+0000 7f4602200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc0af2a0 0x7f45fc0af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.934+0000 7f4602200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0afc60 con 0x7f45e0003970 2024-03-23T11:29:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.934+0000 7f4602200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0b26b0 con 0x7f45fc152fe0 2024-03-23T11:29:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.934+0000 7f4602200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0b23d0 con 0x7f45fc0af2a0 2024-03-23T11:29:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.934+0000 7f45fb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 0x7f45fc157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.934+0000 7f45fb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 0x7f45fc157360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54616/0 (socket says 172.21.15.62:54616) 2024-03-23T11:29:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.934+0000 7f45fb400640 1 -- 172.21.15.62:0/3275475916 learned_addr learned my addr 172.21.15.62:0/3275475916 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.935+0000 7f45fbe00640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc0af2a0 0x7f45fc0af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.935+0000 7f45f2a00640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45fc152fe0 0x7f45fc153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.935+0000 7f45fb400640 1 -- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc0af2a0 msgr2=0x7f45fc0af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.935+0000 7f45fb400640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc0af2a0 0x7f45fc0af720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.935+0000 7f45fb400640 1 -- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45fc152fe0 msgr2=0x7f45fc153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.935+0000 7f45fb400640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45fc152fe0 0x7f45fc153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.935+0000 7f45fb400640 1 -- 172.21.15.62:0/3275475916 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc152500 con 0x7f45e0003970 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.935+0000 7f45fb400640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 0x7f45fc157360 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f45ec037040 tx=0x7f45ec009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.936+0000 7f45f3e00640 1 -- 172.21.15.62:0/3275475916 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec040450 con 0x7f45e0003970 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.936+0000 7f45f3e00640 1 -- 172.21.15.62:0/3275475916 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec0405b0 con 0x7f45e0003970 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.936+0000 7f45f3e00640 1 -- 172.21.15.62:0/3275475916 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec040960 con 0x7f45e0003970 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.936+0000 7f4602200640 1 -- 172.21.15.62:0/3275475916 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc1527e0 con 0x7f45e0003970 2024-03-23T11:29:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.936+0000 7f4602200640 1 -- 172.21.15.62:0/3275475916 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc154cc0 con 0x7f45e0003970 2024-03-23T11:29:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.937+0000 7f4602200640 1 -- 172.21.15.62:0/3275475916 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc066c90 con 0x7f45e0003970 2024-03-23T11:29:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.938+0000 7f45f3e00640 1 -- 172.21.15.62:0/3275475916 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f45ec040ac0 con 0x7f45e0003970 2024-03-23T11:29:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.939+0000 7f45f3e00640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45d806a7a0 0x7f45d806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.940+0000 7f45f3e00640 1 -- 172.21.15.62:0/3275475916 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45ec0ab630 con 0x7f45e0003970 2024-03-23T11:29:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.940+0000 7f45f2a00640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45d806a7a0 0x7f45d806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:26.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.941+0000 7f45f2a00640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45d806a7a0 0x7f45d806cc60 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0b2c80 tx=0x7f45e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:26.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:26.944+0000 7f45f3e00640 1 -- 172.21.15.62:0/3275475916 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f45ec076a40 con 0x7f45e0003970 2024-03-23T11:29:27.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.138+0000 7f4602200640 1 -- 172.21.15.62:0/3275475916 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f45fc13d300 con 0x7f45e0003970 2024-03-23T11:29:27.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.139+0000 7f45f3e00640 1 -- 172.21.15.62:0/3275475916 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) v1 ==== 104+0+11 (secure 0 0 0) 0x7f45ec07a060 con 0x7f45e0003970 2024-03-23T11:29:27.131 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:29:27.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 -- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45d806a7a0 msgr2=0x7f45d806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:27.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45d806a7a0 0x7f45d806cc60 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0b2c80 tx=0x7f45e4008040 comp rx=0 tx=0).stop 2024-03-23T11:29:27.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 -- 172.21.15.62:0/3275475916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 msgr2=0x7f45fc157360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:27.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 0x7f45fc157360 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f45ec037040 tx=0x7f45ec009750 comp rx=0 tx=0).stop 2024-03-23T11:29:27.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 -- 172.21.15.62:0/3275475916 shutdown_connections 2024-03-23T11:29:27.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45d806a7a0 0x7f45d806cc60 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45fc0af2a0 0x7f45fc0af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45fc152fe0 0x7f45fc153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 --2- 172.21.15.62:0/3275475916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e0003970 0x7f45fc157360 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 -- 172.21.15.62:0/3275475916 >> 172.21.15.62:0/3275475916 conn(0x7f45fc0b02e0 msgr2=0x7f45fc13e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:27.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 -- 172.21.15.62:0/3275475916 shutdown_connections 2024-03-23T11:29:27.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.142+0000 7f4602200640 1 -- 172.21.15.62:0/3275475916 wait complete. 2024-03-23T11:29:27.150 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:29:27.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.375+0000 7fd84c000640 1 Processor -- start 2024-03-23T11:29:27.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.376+0000 7fd84c000640 1 -- start start 2024-03-23T11:29:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.377+0000 7fd84c000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd84406c300 0x7fd8440ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.377+0000 7fd84c000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84406c6e0 0x7fd84406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.377+0000 7fd84c000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8440cbc60 con 0x7fd8440cafa0 2024-03-23T11:29:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.378+0000 7fd84c000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8440cbf40 con 0x7fd84406c300 2024-03-23T11:29:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.378+0000 7fd84c000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8440cc220 con 0x7fd84406c6e0 2024-03-23T11:29:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.378+0000 7fd849600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd84406c300 0x7fd8440ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.378+0000 7fd849600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd84406c300 0x7fd8440ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41980/0 (socket says 172.21.15.62:41980) 2024-03-23T11:29:27.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.378+0000 7fd849600640 1 -- 172.21.15.62:0/1371245854 learned_addr learned my addr 172.21.15.62:0/1371245854 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:27.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.378+0000 7fd849600640 1 -- 172.21.15.62:0/1371245854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84406c6e0 msgr2=0x7fd84406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:27.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.378+0000 7fd849600640 1 --2- 172.21.15.62:0/1371245854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84406c6e0 0x7fd84406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.378+0000 7fd849600640 1 -- 172.21.15.62:0/1371245854 >> v1:172.21.15.120:6789/0 conn(0x7fd8440cafa0 legacy=0x7fd8440cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:27.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.378+0000 7fd849600640 1 -- 172.21.15.62:0/1371245854 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8440cc500 con 0x7fd84406c300 2024-03-23T11:29:27.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.379+0000 7fd849600640 1 --2- 172.21.15.62:0/1371245854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84406c300 0x7fd8440ccd00 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fd838009870 tx=0x7fd83802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9e882bd1855175 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:27.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.384+0000 7fd843e00640 1 -- 172.21.15.62:0/1371245854 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838031640 con 0x7fd84406c300 2024-03-23T11:29:27.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.384+0000 7fd843e00640 1 -- 172.21.15.62:0/1371245854 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8380317a0 con 0x7fd84406c300 2024-03-23T11:29:27.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.384+0000 7fd843e00640 1 -- 172.21.15.62:0/1371245854 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838031b30 con 0x7fd84406c300 2024-03-23T11:29:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.385+0000 7fd84c000640 1 -- 172.21.15.62:0/1371245854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84406c300 msgr2=0x7fd8440ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.385+0000 7fd84c000640 1 --2- 172.21.15.62:0/1371245854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84406c300 0x7fd8440ccd00 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fd838009870 tx=0x7fd83802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.385+0000 7fd84c000640 1 -- 172.21.15.62:0/1371245854 shutdown_connections 2024-03-23T11:29:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.385+0000 7fd84c000640 1 --2- 172.21.15.62:0/1371245854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84406c6e0 0x7fd84406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.385+0000 7fd84c000640 1 --2- 172.21.15.62:0/1371245854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84406c300 0x7fd8440ccd00 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.385+0000 7fd84c000640 1 -- 172.21.15.62:0/1371245854 >> 172.21.15.62:0/1371245854 conn(0x7fd8440b02e0 msgr2=0x7fd8440b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.385+0000 7fd84c000640 1 -- 172.21.15.62:0/1371245854 shutdown_connections 2024-03-23T11:29:27.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.385+0000 7fd84c000640 1 -- 172.21.15.62:0/1371245854 wait complete. 2024-03-23T11:29:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.387+0000 7fd84c000640 1 Processor -- start 2024-03-23T11:29:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.387+0000 7fd84c000640 1 -- start start 2024-03-23T11:29:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84c000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd84406c300 0x7fd8441570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84c000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84406c6e0 0x7fd8441404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84c000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8440cafa0 0x7fd84413e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84c000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84413fb10 con 0x7fd8440cafa0 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84c000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd844157630 con 0x7fd84406c300 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84c000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8440ad1a0 con 0x7fd84406c6e0 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8440cafa0 0x7fd84413e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8440cafa0 0x7fd84413e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41994/0 (socket says 172.21.15.62:41994) 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84a000640 1 -- 172.21.15.62:0/3691739294 learned_addr learned my addr 172.21.15.62:0/3691739294 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd849600640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd84406c300 0x7fd8441570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd848c00640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84406c6e0 0x7fd8441404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84a000640 1 -- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84406c6e0 msgr2=0x7fd8441404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84a000640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84406c6e0 0x7fd8441404f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84a000640 1 -- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd84406c300 msgr2=0x7fd8441570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84a000640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd84406c300 0x7fd8441570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.388+0000 7fd84a000640 1 -- 172.21.15.62:0/3691739294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8380094d0 con 0x7fd8440cafa0 2024-03-23T11:29:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.389+0000 7fd848c00640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84406c6e0 0x7fd8441404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.389+0000 7fd84a000640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8440cafa0 0x7fd84413e0b0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00b9a0 tx=0x7fd83c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.389+0000 7fd841600640 1 -- 172.21.15.62:0/3691739294 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c00cba0 con 0x7fd8440cafa0 2024-03-23T11:29:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.389+0000 7fd841600640 1 -- 172.21.15.62:0/3691739294 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c00cd20 con 0x7fd8440cafa0 2024-03-23T11:29:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.389+0000 7fd841600640 1 -- 172.21.15.62:0/3691739294 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c014680 con 0x7fd8440cafa0 2024-03-23T11:29:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.389+0000 7fd84c000640 1 -- 172.21.15.62:0/3691739294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8440ad480 con 0x7fd8440cafa0 2024-03-23T11:29:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.389+0000 7fd84c000640 1 -- 172.21.15.62:0/3691739294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8440ada40 con 0x7fd8440cafa0 2024-03-23T11:29:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.390+0000 7fd84c000640 1 -- 172.21.15.62:0/3691739294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd844066c90 con 0x7fd8440cafa0 2024-03-23T11:29:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.391+0000 7fd841600640 1 -- 172.21.15.62:0/3691739294 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd83c01d3b0 con 0x7fd8440cafa0 2024-03-23T11:29:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.392+0000 7fd841600640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd82406a7a0 0x7fd82406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.392+0000 7fd841600640 1 -- 172.21.15.62:0/3691739294 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd83c088dc0 con 0x7fd8440cafa0 2024-03-23T11:29:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.393+0000 7fd849600640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd82406a7a0 0x7fd82406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.394+0000 7fd849600640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd82406a7a0 0x7fd82406cc60 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd838036000 tx=0x7fd83803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:27.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.396+0000 7fd841600640 1 -- 172.21.15.62:0/3691739294 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd83c054140 con 0x7fd8440cafa0 2024-03-23T11:29:27.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.597+0000 7fd84c000640 1 -- 172.21.15.62:0/3691739294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd8440ade50 con 0x7fd8440cafa0 2024-03-23T11:29:27.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.598+0000 7fd841600640 1 -- 172.21.15.62:0/3691739294 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd83c0190a0 con 0x7fd8440cafa0 2024-03-23T11:29:27.590 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:29:27.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 -- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd82406a7a0 msgr2=0x7fd82406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:27.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd82406a7a0 0x7fd82406cc60 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd838036000 tx=0x7fd83803c040 comp rx=0 tx=0).stop 2024-03-23T11:29:27.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 -- 172.21.15.62:0/3691739294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8440cafa0 msgr2=0x7fd84413e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:27.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8440cafa0 0x7fd84413e0b0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00b9a0 tx=0x7fd83c00be60 comp rx=0 tx=0).stop 2024-03-23T11:29:27.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 -- 172.21.15.62:0/3691739294 shutdown_connections 2024-03-23T11:29:27.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd82406a7a0 0x7fd82406cc60 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8440cafa0 0x7fd84413e0b0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd84406c6e0 0x7fd8441404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 --2- 172.21.15.62:0/3691739294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd84406c300 0x7fd8441570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 -- 172.21.15.62:0/3691739294 >> 172.21.15.62:0/3691739294 conn(0x7fd8440b02e0 msgr2=0x7fd8440cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:27.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 -- 172.21.15.62:0/3691739294 shutdown_connections 2024-03-23T11:29:27.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.601+0000 7fd84c000640 1 -- 172.21.15.62:0/3691739294 wait complete. 2024-03-23T11:29:27.612 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:29:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.839+0000 7f4033200640 1 Processor -- start 2024-03-23T11:29:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.840+0000 7f4033200640 1 -- start start 2024-03-23T11:29:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.841+0000 7f4033200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f40340d13b0 0x7f40340cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.841+0000 7f4033200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40340cd260 0x7f40340b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.841+0000 7f4033200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340d0810 con 0x7f403415c960 2024-03-23T11:29:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.841+0000 7f4033200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40340cd9d0 con 0x7f40340d13b0 2024-03-23T11:29:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.841+0000 7f4033200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40340cdcb0 con 0x7f40340cd260 2024-03-23T11:29:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.842+0000 7f4031e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f403415c960 0x7f40340d0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:51994/0 (socket says 172.21.15.62:51994) 2024-03-23T11:29:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.842+0000 7f4031e00640 1 -- 172.21.15.62:0/824542615 learned_addr learned my addr 172.21.15.62:0/824542615 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.842+0000 7f4031400640 1 --2- 172.21.15.62:0/824542615 >> v2:172.21.15.62:3300/0 conn(0x7f40340d13b0 0x7f40340cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.842+0000 7f4032800640 1 --2- 172.21.15.62:0/824542615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40340cd260 0x7f40340b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.842+0000 7f4030a00640 1 -- 172.21.15.62:0/824542615 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983616899 0 0) 0x7f4024001110 con 0x7f403415c960 2024-03-23T11:29:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.842+0000 7f4030a00640 1 -- 172.21.15.62:0/824542615 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4018003520 con 0x7f403415c960 2024-03-23T11:29:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.842+0000 7f4031400640 1 -- 172.21.15.62:0/824542615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40340cd260 msgr2=0x7f40340b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.842+0000 7f4031400640 1 --2- 172.21.15.62:0/824542615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40340cd260 0x7f40340b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.842+0000 7f4031400640 1 -- 172.21.15.62:0/824542615 >> v1:172.21.15.62:6789/0 conn(0x7f403415c960 legacy=0x7f40340d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.842+0000 7f4031400640 1 -- 172.21.15.62:0/824542615 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40340cdf90 con 0x7f40340d13b0 2024-03-23T11:29:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.844+0000 7f4031400640 1 --2- 172.21.15.62:0/824542615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40340d13b0 0x7f40340cf790 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f401c009790 tx=0x7f401c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e816657a8c6ac51c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.844+0000 7f4030a00640 1 -- 172.21.15.62:0/824542615 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c030800 con 0x7f40340d13b0 2024-03-23T11:29:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.844+0000 7f4030a00640 1 -- 172.21.15.62:0/824542615 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401c030960 con 0x7f40340d13b0 2024-03-23T11:29:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.844+0000 7f4030a00640 1 -- 172.21.15.62:0/824542615 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c030cf0 con 0x7f40340d13b0 2024-03-23T11:29:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.844+0000 7f4033200640 1 -- 172.21.15.62:0/824542615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40340d13b0 msgr2=0x7f40340cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.844+0000 7f4033200640 1 --2- 172.21.15.62:0/824542615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40340d13b0 0x7f40340cf790 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f401c009790 tx=0x7f401c02b940 comp rx=0 tx=0).stop 2024-03-23T11:29:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.845+0000 7f4033200640 1 -- 172.21.15.62:0/824542615 shutdown_connections 2024-03-23T11:29:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.845+0000 7f4033200640 1 --2- 172.21.15.62:0/824542615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40340cd260 0x7f40340b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.845+0000 7f4033200640 1 --2- 172.21.15.62:0/824542615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40340d13b0 0x7f40340cf790 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.845+0000 7f4033200640 1 -- 172.21.15.62:0/824542615 >> 172.21.15.62:0/824542615 conn(0x7f4034089f60 msgr2=0x7f40340d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.845+0000 7f4033200640 1 -- 172.21.15.62:0/824542615 shutdown_connections 2024-03-23T11:29:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.845+0000 7f4033200640 1 -- 172.21.15.62:0/824542615 wait complete. 2024-03-23T11:29:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.846+0000 7f4033200640 1 Processor -- start 2024-03-23T11:29:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.847+0000 7f4033200640 1 -- start start 2024-03-23T11:29:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.847+0000 7f4033200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40340cd260 0x7f403415f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4033200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40340d13b0 0x7f403415e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4033200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403415c960 0x7f403414c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4033200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403415f1b0 con 0x7f40340cd260 2024-03-23T11:29:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4033200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40340cc710 con 0x7f40340d13b0 2024-03-23T11:29:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4033200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40340cc9f0 con 0x7f403415c960 2024-03-23T11:29:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4031e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40340cd260 0x7f403415f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4031e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40340cd260 0x7f403415f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42002/0 (socket says 172.21.15.62:42002) 2024-03-23T11:29:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4031e00640 1 -- 172.21.15.62:0/3109000242 learned_addr learned my addr 172.21.15.62:0/3109000242 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4031e00640 1 -- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403415c960 msgr2=0x7f403414c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4031e00640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403415c960 0x7f403414c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4031e00640 1 -- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40340d13b0 msgr2=0x7f403415e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4031e00640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40340d13b0 0x7f403415e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4031e00640 1 -- 172.21.15.62:0/3109000242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c0093a0 con 0x7f40340cd260 2024-03-23T11:29:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.848+0000 7f4031e00640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40340cd260 0x7f403415f5d0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f402400c700 tx=0x7f402400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.849+0000 7f402a000640 1 -- 172.21.15.62:0/3109000242 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024005ce0 con 0x7f40340cd260 2024-03-23T11:29:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.849+0000 7f402a000640 1 -- 172.21.15.62:0/3109000242 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4024005e40 con 0x7f40340cd260 2024-03-23T11:29:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.849+0000 7f402a000640 1 -- 172.21.15.62:0/3109000242 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024007270 con 0x7f40340cd260 2024-03-23T11:29:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.849+0000 7f4033200640 1 -- 172.21.15.62:0/3109000242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403415edd0 con 0x7f40340cd260 2024-03-23T11:29:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.849+0000 7f4033200640 1 -- 172.21.15.62:0/3109000242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403414efd0 con 0x7f40340cd260 2024-03-23T11:29:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.850+0000 7f4033200640 1 -- 172.21.15.62:0/3109000242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403415f2f0 con 0x7f40340cd260 2024-03-23T11:29:27.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.851+0000 7f402a000640 1 -- 172.21.15.62:0/3109000242 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4024007490 con 0x7f40340cd260 2024-03-23T11:29:27.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.852+0000 7f402a000640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f401006a7a0 0x7f401006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:27.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.852+0000 7f402a000640 1 -- 172.21.15.62:0/3109000242 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40240880f0 con 0x7f40340cd260 2024-03-23T11:29:27.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.852+0000 7f4031400640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f401006a7a0 0x7f401006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:27.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.854+0000 7f4031400640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f401006a7a0 0x7f401006cc60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f401c036000 tx=0x7f401c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:27.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:27.857+0000 7f402a000640 1 -- 172.21.15.62:0/3109000242 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40240535b0 con 0x7f40340cd260 2024-03-23T11:29:28.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.072+0000 7f4033200640 1 -- 172.21.15.62:0/3109000242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f40340d0810 con 0x7f40340cd260 2024-03-23T11:29:28.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.072+0000 7f402a000640 1 -- 172.21.15.62:0/3109000242 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v105) v1 ==== 80+0+0 (secure 0 0 0) 0x7f40340d0810 con 0x7f40340cd260 2024-03-23T11:29:28.064 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:29:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.075+0000 7f4033200640 1 -- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f401006a7a0 msgr2=0x7f401006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.075+0000 7f4033200640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f401006a7a0 0x7f401006cc60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f401c036000 tx=0x7f401c03b040 comp rx=0 tx=0).stop 2024-03-23T11:29:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.075+0000 7f4033200640 1 -- 172.21.15.62:0/3109000242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40340cd260 msgr2=0x7f403415f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.075+0000 7f4033200640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40340cd260 0x7f403415f5d0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f402400c700 tx=0x7f402400cb00 comp rx=0 tx=0).stop 2024-03-23T11:29:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.075+0000 7f4033200640 1 -- 172.21.15.62:0/3109000242 shutdown_connections 2024-03-23T11:29:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.075+0000 7f4033200640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f401006a7a0 0x7f401006cc60 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.075+0000 7f4033200640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f403415c960 0x7f403414c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.075+0000 7f4033200640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40340d13b0 0x7f403415e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.075+0000 7f4033200640 1 --2- 172.21.15.62:0/3109000242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40340cd260 0x7f403415f5d0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.075+0000 7f4033200640 1 -- 172.21.15.62:0/3109000242 >> 172.21.15.62:0/3109000242 conn(0x7f4034089f60 msgr2=0x7f403415a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.076+0000 7f4033200640 1 -- 172.21.15.62:0/3109000242 shutdown_connections 2024-03-23T11:29:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.076+0000 7f4033200640 1 -- 172.21.15.62:0/3109000242 wait complete. 2024-03-23T11:29:28.086 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:29:28.086 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:28.142 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:29:28.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.379+0000 7efffd600640 1 Processor -- start 2024-03-23T11:29:28.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.380+0000 7efffd600640 1 -- start start 2024-03-23T11:29:28.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.381+0000 7efffd600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efff815fcf0 0x7efff80d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:28.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.381+0000 7efffd600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff80d14f0 con 0x7efff80b6dc0 2024-03-23T11:29:28.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.381+0000 7efffd600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff80d3540 con 0x7efff8160420 2024-03-23T11:29:28.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.381+0000 7efffd600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff80ce510 con 0x7efff815fcf0 2024-03-23T11:29:28.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.381+0000 7efff7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efff815fcf0 0x7efff80d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:28.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.381+0000 7efffcc00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7efff80b6dc0 0x7efff80d0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:52010/0 (socket says 172.21.15.62:52010) 2024-03-23T11:29:28.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.381+0000 7efff7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efff815fcf0 0x7efff80d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42010/0 (socket says 172.21.15.62:42010) 2024-03-23T11:29:28.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.381+0000 7efffcc00640 1 -- 172.21.15.62:0/4211035073 learned_addr learned my addr 172.21.15.62:0/4211035073 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:28.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.382+0000 7efff7e00640 1 -- 172.21.15.62:0/4211035073 >> v1:172.21.15.120:6789/0 conn(0x7efff8160420 legacy=0x7efff80d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:28.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.382+0000 7efff7e00640 1 -- 172.21.15.62:0/4211035073 >> v1:172.21.15.62:6789/0 conn(0x7efff80b6dc0 legacy=0x7efff80d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.382+0000 7efff7e00640 1 -- 172.21.15.62:0/4211035073 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff80cdc70 con 0x7efff815fcf0 2024-03-23T11:29:28.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.383+0000 7efff7e00640 1 --2- 172.21.15.62:0/4211035073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff815fcf0 0x7efff80d3000 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7effe80097a0 tx=0x7effe802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615280695990856c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:28.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.386+0000 7efff6a00640 1 -- 172.21.15.62:0/4211035073 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe802e070 con 0x7efff815fcf0 2024-03-23T11:29:28.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.386+0000 7efff6a00640 1 -- 172.21.15.62:0/4211035073 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe80315a0 con 0x7efff815fcf0 2024-03-23T11:29:28.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.386+0000 7efff6a00640 1 -- 172.21.15.62:0/4211035073 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe8031930 con 0x7efff815fcf0 2024-03-23T11:29:28.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.387+0000 7efffd600640 1 -- 172.21.15.62:0/4211035073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff815fcf0 msgr2=0x7efff80d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.387+0000 7efffd600640 1 --2- 172.21.15.62:0/4211035073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff815fcf0 0x7efff80d3000 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7effe80097a0 tx=0x7effe802b840 comp rx=0 tx=0).stop 2024-03-23T11:29:28.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.387+0000 7efffd600640 1 -- 172.21.15.62:0/4211035073 shutdown_connections 2024-03-23T11:29:28.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.387+0000 7efffd600640 1 --2- 172.21.15.62:0/4211035073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff815fcf0 0x7efff80d3000 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.387+0000 7efffd600640 1 -- 172.21.15.62:0/4211035073 >> 172.21.15.62:0/4211035073 conn(0x7efff8089f60 msgr2=0x7efff80d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:28.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.387+0000 7efffd600640 1 -- 172.21.15.62:0/4211035073 shutdown_connections 2024-03-23T11:29:28.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.387+0000 7efffd600640 1 -- 172.21.15.62:0/4211035073 wait complete. 2024-03-23T11:29:28.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.389+0000 7efffd600640 1 Processor -- start 2024-03-23T11:29:28.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.389+0000 7efffd600640 1 -- start start 2024-03-23T11:29:28.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.390+0000 7efffd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80b6dc0 0x7efff81e8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:28.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.390+0000 7efffd600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efff815fcf0 0x7efff80cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:28.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.390+0000 7efffd600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff8160420 0x7efff814b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:28.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.390+0000 7efffd600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff81e76c0 con 0x7efff80b6dc0 2024-03-23T11:29:28.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.390+0000 7efffd600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff80c6320 con 0x7efff815fcf0 2024-03-23T11:29:28.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.390+0000 7efffd600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff80c6600 con 0x7efff8160420 2024-03-23T11:29:28.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.391+0000 7efff7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80b6dc0 0x7efff81e8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:28.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.391+0000 7efff7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80b6dc0 0x7efff81e8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42018/0 (socket says 172.21.15.62:42018) 2024-03-23T11:29:28.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.391+0000 7efff7e00640 1 -- 172.21.15.62:0/3811075737 learned_addr learned my addr 172.21.15.62:0/3811075737 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:28.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.391+0000 7efff7e00640 1 -- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff8160420 msgr2=0x7efff814b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.391+0000 7efff7e00640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff8160420 0x7efff814b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.391+0000 7efff7e00640 1 -- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efff815fcf0 msgr2=0x7efff80cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.391+0000 7efff7e00640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efff815fcf0 0x7efff80cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.391+0000 7efff7e00640 1 -- 172.21.15.62:0/3811075737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe8009400 con 0x7efff80b6dc0 2024-03-23T11:29:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.392+0000 7efff7e00640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80b6dc0 0x7efff81e8230 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7effe8034040 tx=0x7effe80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.392+0000 7effefe00640 1 -- 172.21.15.62:0/3811075737 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe8032900 con 0x7efff80b6dc0 2024-03-23T11:29:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.392+0000 7effefe00640 1 -- 172.21.15.62:0/3811075737 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe8032a80 con 0x7efff80b6dc0 2024-03-23T11:29:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.392+0000 7efffd600640 1 -- 172.21.15.62:0/3811075737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff80c68e0 con 0x7efff80b6dc0 2024-03-23T11:29:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.392+0000 7effefe00640 1 -- 172.21.15.62:0/3811075737 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe8032e30 con 0x7efff80b6dc0 2024-03-23T11:29:28.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.392+0000 7efffd600640 1 -- 172.21.15.62:0/3811075737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff80c6ea0 con 0x7efff80b6dc0 2024-03-23T11:29:28.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.393+0000 7efffd600640 1 -- 172.21.15.62:0/3811075737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff80c7090 con 0x7efff80b6dc0 2024-03-23T11:29:28.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.394+0000 7effefe00640 1 -- 172.21.15.62:0/3811075737 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7effe803b3f0 con 0x7efff80b6dc0 2024-03-23T11:29:28.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.395+0000 7effefe00640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7effd406a7a0 0x7effd406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:28.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.395+0000 7effefe00640 1 -- 172.21.15.62:0/3811075737 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7effe80aaf60 con 0x7efff80b6dc0 2024-03-23T11:29:28.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.396+0000 7efff7400640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7effd406a7a0 0x7effd406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:28.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.397+0000 7efff7400640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7effd406a7a0 0x7effd406cc60 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7effe0006fd0 tx=0x7effe0006bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:28.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.400+0000 7effefe00640 1 -- 172.21.15.62:0/3811075737 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7effe8076260 con 0x7efff80b6dc0 2024-03-23T11:29:28.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.598+0000 7efffd600640 1 -- 172.21.15.62:0/3811075737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efff8066bb0 con 0x7efff80b6dc0 2024-03-23T11:29:28.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.599+0000 7effefe00640 1 -- 172.21.15.62:0/3811075737 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16051 (secure 0 0 0) 0x7effe8079880 con 0x7efff80b6dc0 2024-03-23T11:29:28.591 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:28.591 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":105,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:29:16.631279+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:29:13.817605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-23T11:29:14.652617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:29:28.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 -- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7effd406a7a0 msgr2=0x7effd406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7effd406a7a0 0x7effd406cc60 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7effe0006fd0 tx=0x7effe0006bc0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 -- 172.21.15.62:0/3811075737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80b6dc0 msgr2=0x7efff81e8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80b6dc0 0x7efff81e8230 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7effe8034040 tx=0x7effe80046c0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 -- 172.21.15.62:0/3811075737 shutdown_connections 2024-03-23T11:29:28.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7effd406a7a0 0x7effd406cc60 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efff8160420 0x7efff814b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efff815fcf0 0x7efff80cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 --2- 172.21.15.62:0/3811075737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80b6dc0 0x7efff81e8230 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 -- 172.21.15.62:0/3811075737 >> 172.21.15.62:0/3811075737 conn(0x7efff8089f60 msgr2=0x7efff80d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:28.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 -- 172.21.15.62:0/3811075737 shutdown_connections 2024-03-23T11:29:28.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.602+0000 7efffd600640 1 -- 172.21.15.62:0/3811075737 wait complete. 2024-03-23T11:29:28.613 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:29:13.817605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:29:14.652617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:29:28.614 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:29:28.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.850+0000 7fc375200640 1 Processor -- start 2024-03-23T11:29:28.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.850+0000 7fc375200640 1 -- start start 2024-03-23T11:29:28.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.851+0000 7fc375200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc37015c9e0 0x7fc3701607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:28.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.851+0000 7fc375200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc37015cdc0 0x7fc3700d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:28.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.852+0000 7fc375200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37006bfc0 con 0x7fc3700cf650 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.852+0000 7fc375200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3700ce140 con 0x7fc37015cdc0 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.852+0000 7fc375200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3700b4370 con 0x7fc37015c9e0 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.852+0000 7fc36d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc37015cdc0 0x7fc3700d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.852+0000 7fc36d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc37015cdc0 0x7fc3700d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42030/0 (socket says 172.21.15.62:42030) 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.852+0000 7fc36e800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc3700cf650 0x7fc37015d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:52020/0 (socket says 172.21.15.62:52020) 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.852+0000 7fc36d400640 1 -- 172.21.15.62:0/1682228920 learned_addr learned my addr 172.21.15.62:0/1682228920 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.852+0000 7fc36de00640 1 --2- 172.21.15.62:0/1682228920 >> v2:172.21.15.120:3300/0 conn(0x7fc37015c9e0 0x7fc3701607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.853+0000 7fc36d400640 1 -- 172.21.15.62:0/1682228920 >> v2:172.21.15.120:3300/0 conn(0x7fc37015c9e0 msgr2=0x7fc3701607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.853+0000 7fc36d400640 1 --2- 172.21.15.62:0/1682228920 >> v2:172.21.15.120:3300/0 conn(0x7fc37015c9e0 0x7fc3701607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.853+0000 7fc36d400640 1 -- 172.21.15.62:0/1682228920 >> v1:172.21.15.62:6789/0 conn(0x7fc3700cf650 legacy=0x7fc37015d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.853+0000 7fc36d400640 1 -- 172.21.15.62:0/1682228920 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3700b5260 con 0x7fc37015cdc0 2024-03-23T11:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.853+0000 7fc36ca00640 1 -- 172.21.15.62:0/1682228920 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926286586 0 0) 0x7fc37006bfc0 con 0x7fc3700cf650 2024-03-23T11:29:28.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.854+0000 7fc36d400640 1 --2- 172.21.15.62:0/1682228920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc37015cdc0 0x7fc3700d1f20 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fc3700d0380 tx=0x7fc36002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9420cd351a7fd3ce server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:28.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.854+0000 7fc36ca00640 1 -- 172.21.15.62:0/1682228920 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc360030860 con 0x7fc37015cdc0 2024-03-23T11:29:28.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.854+0000 7fc36ca00640 1 -- 172.21.15.62:0/1682228920 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3600309c0 con 0x7fc37015cdc0 2024-03-23T11:29:28.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.854+0000 7fc36ca00640 1 -- 172.21.15.62:0/1682228920 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc360030d50 con 0x7fc37015cdc0 2024-03-23T11:29:28.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.855+0000 7fc375200640 1 -- 172.21.15.62:0/1682228920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc37015cdc0 msgr2=0x7fc3700d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.855+0000 7fc375200640 1 --2- 172.21.15.62:0/1682228920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc37015cdc0 0x7fc3700d1f20 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fc3700d0380 tx=0x7fc36002b9a0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.855+0000 7fc375200640 1 -- 172.21.15.62:0/1682228920 shutdown_connections 2024-03-23T11:29:28.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.855+0000 7fc375200640 1 --2- 172.21.15.62:0/1682228920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc37015cdc0 0x7fc3700d1f20 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.855+0000 7fc375200640 1 --2- 172.21.15.62:0/1682228920 >> v2:172.21.15.120:3300/0 conn(0x7fc37015c9e0 0x7fc3701607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.855+0000 7fc375200640 1 -- 172.21.15.62:0/1682228920 >> 172.21.15.62:0/1682228920 conn(0x7fc370071570 msgr2=0x7fc370071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:28.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.855+0000 7fc375200640 1 -- 172.21.15.62:0/1682228920 shutdown_connections 2024-03-23T11:29:28.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.855+0000 7fc375200640 1 -- 172.21.15.62:0/1682228920 wait complete. 2024-03-23T11:29:28.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.857+0000 7fc375200640 1 Processor -- start 2024-03-23T11:29:28.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.857+0000 7fc375200640 1 -- start start 2024-03-23T11:29:28.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.858+0000 7fc375200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3700cf650 0x7fc3700d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:28.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.858+0000 7fc375200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc37015c9e0 0x7fc37015e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:28.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.858+0000 7fc375200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc37015cdc0 0x7fc37015e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:28.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.858+0000 7fc375200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3701f35e0 con 0x7fc37015cdc0 2024-03-23T11:29:28.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.858+0000 7fc375200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700d11a0 con 0x7fc37015c9e0 2024-03-23T11:29:28.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.858+0000 7fc375200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700d1480 con 0x7fc3700cf650 2024-03-23T11:29:28.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.858+0000 7fc36e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc37015cdc0 0x7fc37015e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:28.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.858+0000 7fc36e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc37015cdc0 0x7fc37015e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42044/0 (socket says 172.21.15.62:42044) 2024-03-23T11:29:28.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.858+0000 7fc36e800640 1 -- 172.21.15.62:0/3893743306 learned_addr learned my addr 172.21.15.62:0/3893743306 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:28.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.859+0000 7fc36de00640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3700cf650 0x7fc3700d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:28.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.859+0000 7fc36d400640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc37015c9e0 0x7fc37015e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:28.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.859+0000 7fc36e800640 1 -- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3700cf650 msgr2=0x7fc3700d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.859+0000 7fc36e800640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3700cf650 0x7fc3700d21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.859+0000 7fc36e800640 1 -- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc37015c9e0 msgr2=0x7fc37015e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:28.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.859+0000 7fc36e800640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc37015c9e0 0x7fc37015e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:28.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.859+0000 7fc36e800640 1 -- 172.21.15.62:0/3893743306 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600094d0 con 0x7fc37015cdc0 2024-03-23T11:29:28.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.859+0000 7fc36de00640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3700cf650 0x7fc3700d21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:28.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.859+0000 7fc36d400640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc37015c9e0 0x7fc37015e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:28.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.859+0000 7fc36e800640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc37015cdc0 0x7fc37015e8b0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fc36800c7e0 tx=0x7fc36800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:28.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.859+0000 7fc366000640 1 -- 172.21.15.62:0/3893743306 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36800a640 con 0x7fc37015cdc0 2024-03-23T11:29:28.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.860+0000 7fc366000640 1 -- 172.21.15.62:0/3893743306 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36800a7c0 con 0x7fc37015cdc0 2024-03-23T11:29:28.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.860+0000 7fc366000640 1 -- 172.21.15.62:0/3893743306 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36800ab70 con 0x7fc37015cdc0 2024-03-23T11:29:28.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.860+0000 7fc375200640 1 -- 172.21.15.62:0/3893743306 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3700d1760 con 0x7fc37015cdc0 2024-03-23T11:29:28.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.860+0000 7fc375200640 1 -- 172.21.15.62:0/3893743306 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3700d1d20 con 0x7fc37015cdc0 2024-03-23T11:29:28.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.861+0000 7fc375200640 1 -- 172.21.15.62:0/3893743306 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3700c91c0 con 0x7fc37015cdc0 2024-03-23T11:29:28.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.862+0000 7fc366000640 1 -- 172.21.15.62:0/3893743306 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc36800acd0 con 0x7fc37015cdc0 2024-03-23T11:29:28.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.863+0000 7fc366000640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc34c06a7a0 0x7fc34c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:28.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.863+0000 7fc366000640 1 -- 172.21.15.62:0/3893743306 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc368088390 con 0x7fc37015cdc0 2024-03-23T11:29:28.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.863+0000 7fc36de00640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc34c06a7a0 0x7fc34c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:28.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.865+0000 7fc36de00640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc34c06a7a0 0x7fc34c06cc60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc358006fd0 tx=0x7fc358008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:28.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:28.867+0000 7fc366000640 1 -- 172.21.15.62:0/3893743306 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc3680537a0 con 0x7fc37015cdc0 2024-03-23T11:29:29.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.063+0000 7fc375200640 1 -- 172.21.15.62:0/3893743306 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc370066c90 con 0x7fc37015cdc0 2024-03-23T11:29:29.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.064+0000 7fc366000640 1 -- 172.21.15.62:0/3893743306 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc3680180a0 con 0x7fc37015cdc0 2024-03-23T11:29:29.055 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.066+0000 7fc375200640 1 -- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc34c06a7a0 msgr2=0x7fc34c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.066+0000 7fc375200640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc34c06a7a0 0x7fc34c06cc60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc358006fd0 tx=0x7fc358008040 comp rx=0 tx=0).stop 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.066+0000 7fc375200640 1 -- 172.21.15.62:0/3893743306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc37015cdc0 msgr2=0x7fc37015e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.066+0000 7fc375200640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc37015cdc0 0x7fc37015e8b0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fc36800c7e0 tx=0x7fc36800cca0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.066+0000 7fc375200640 1 -- 172.21.15.62:0/3893743306 shutdown_connections 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.066+0000 7fc375200640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc34c06a7a0 0x7fc34c06cc60 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.066+0000 7fc375200640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc37015cdc0 0x7fc37015e8b0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.066+0000 7fc375200640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc37015c9e0 0x7fc37015e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.066+0000 7fc375200640 1 --2- 172.21.15.62:0/3893743306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3700cf650 0x7fc3700d21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.066+0000 7fc375200640 1 -- 172.21.15.62:0/3893743306 >> 172.21.15.62:0/3893743306 conn(0x7fc370071570 msgr2=0x7fc3700cfac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.067+0000 7fc375200640 1 -- 172.21.15.62:0/3893743306 shutdown_connections 2024-03-23T11:29:29.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.067+0000 7fc375200640 1 -- 172.21.15.62:0/3893743306 wait complete. 2024-03-23T11:29:29.078 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:29:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.307+0000 7f89c3e00640 1 Processor -- start 2024-03-23T11:29:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.307+0000 7f89c3e00640 1 -- start start 2024-03-23T11:29:29.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.309+0000 7f89c3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c4070090 0x7f89c40704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.309+0000 7f89c3e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c4070a30 con 0x7f89c4143dd0 2024-03-23T11:29:29.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.309+0000 7f89c3e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c40cd140 con 0x7f89c40b2820 2024-03-23T11:29:29.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.309+0000 7f89c3e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c40cd3e0 con 0x7f89c4070090 2024-03-23T11:29:29.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.309+0000 7f89c2a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f89c4143dd0 0x7f89c415abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:52034/0 (socket says 172.21.15.62:52034) 2024-03-23T11:29:29.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.309+0000 7f89c2a00640 1 -- 172.21.15.62:0/3278114042 learned_addr learned my addr 172.21.15.62:0/3278114042 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.309+0000 7f89c2000640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c4070090 0x7f89c40704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158265311 0 0) 0x7f89c4070a30 con 0x7f89c4143dd0 2024-03-23T11:29:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a8003520 con 0x7f89c4143dd0 2024-03-23T11:29:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1594171501 0 0) 0x7f89a8003520 con 0x7f89c4143dd0 2024-03-23T11:29:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c4070090 msgr2=0x7f89c40704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c4070090 0x7f89c40704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 >> v1:172.21.15.120:6789/0 conn(0x7f89c40b2820 legacy=0x7f89c40b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c40cd6c0 con 0x7f89c4143dd0 2024-03-23T11:29:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f89b4001670 con 0x7f89c4143dd0 2024-03-23T11:29:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 >> v1:172.21.15.62:6789/0 conn(0x7f89c4143dd0 legacy=0x7f89c415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89a80039a0 0x7f89a8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89a80047c0 0x7f89a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89a8005180 0x7f89a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c40cd3e0 con 0x7f89a80047c0 2024-03-23T11:29:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a800c050 con 0x7f89a8005180 2024-03-23T11:29:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a800c210 con 0x7f89a80039a0 2024-03-23T11:29:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.310+0000 7f89c2000640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c4070090 0x7f89c40704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:29:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.311+0000 7f89c2000640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89a80047c0 0x7f89a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.311+0000 7f89c2a00640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89a80039a0 0x7f89a8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.311+0000 7f89c3400640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89a8005180 0x7f89a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.311+0000 7f89c2000640 1 -- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89a80039a0 msgr2=0x7f89a8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.311+0000 7f89c2000640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89a80039a0 0x7f89a8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.311+0000 7f89c2000640 1 -- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89a8005180 msgr2=0x7f89a800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.311+0000 7f89c2000640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89a8005180 0x7f89a800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.311+0000 7f89c2000640 1 -- 172.21.15.62:0/3278114042 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a8003520 con 0x7f89a80047c0 2024-03-23T11:29:29.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.312+0000 7f89c2000640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89a80047c0 0x7f89a8004c40 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f89b0006fd0 tx=0x7f89a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4e2b27e2afaaf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:29.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.312+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0033600 con 0x7f89a80047c0 2024-03-23T11:29:29.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.312+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b002f050 con 0x7f89a80047c0 2024-03-23T11:29:29.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.312+0000 7f89c1600640 1 -- 172.21.15.62:0/3278114042 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0033980 con 0x7f89a80047c0 2024-03-23T11:29:29.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.313+0000 7f89c3e00640 1 -- 172.21.15.62:0/3278114042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89a80047c0 msgr2=0x7f89a8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.313+0000 7f89c3e00640 1 --2- 172.21.15.62:0/3278114042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89a80047c0 0x7f89a8004c40 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f89b0006fd0 tx=0x7f89a80057f0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.313+0000 7f89c3400640 1 -- 172.21.15.62:0/3278114042 reap_dead start 2024-03-23T11:29:29.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.313+0000 7f89c3e00640 1 -- 172.21.15.62:0/3278114042 shutdown_connections 2024-03-23T11:29:29.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.313+0000 7f89c3e00640 1 -- 172.21.15.62:0/3278114042 >> 172.21.15.62:0/3278114042 conn(0x7f89c4066050 msgr2=0x7f89c4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:29.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.313+0000 7f89c3e00640 1 -- 172.21.15.62:0/3278114042 shutdown_connections 2024-03-23T11:29:29.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.313+0000 7f89c3e00640 1 -- 172.21.15.62:0/3278114042 wait complete. 2024-03-23T11:29:29.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.315+0000 7f89c3e00640 1 Processor -- start 2024-03-23T11:29:29.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.315+0000 7f89c3e00640 1 -- start start 2024-03-23T11:29:29.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.316+0000 7f89c3e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89a80047c0 0x7f89c4142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.316+0000 7f89c3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c41f6080 0x7f89c413b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c41f28c0 0x7f89c41f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c4141f70 con 0x7f89c41f28c0 2024-03-23T11:29:29.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c413ed80 con 0x7f89a80047c0 2024-03-23T11:29:29.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c41412f0 con 0x7f89c41f6080 2024-03-23T11:29:29.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c41f28c0 0x7f89c41f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c41f28c0 0x7f89c41f5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42062/0 (socket says 172.21.15.62:42062) 2024-03-23T11:29:29.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3400640 1 -- 172.21.15.62:0/781342074 learned_addr learned my addr 172.21.15.62:0/781342074 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:29.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c2000640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c41f6080 0x7f89c413b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c2a00640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89a80047c0 0x7f89c4142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3400640 1 -- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c41f6080 msgr2=0x7f89c413b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3400640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c41f6080 0x7f89c413b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3400640 1 -- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89a80047c0 msgr2=0x7f89c4142390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3400640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89a80047c0 0x7f89c4142390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c3400640 1 -- 172.21.15.62:0/781342074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c41415d0 con 0x7f89c41f28c0 2024-03-23T11:29:29.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.317+0000 7f89c2a00640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89a80047c0 0x7f89c4142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:29.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.318+0000 7f89c3400640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c41f28c0 0x7f89c41f5430 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f89c406e450 tx=0x7f89bc00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:29.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.318+0000 7f89baa00640 1 -- 172.21.15.62:0/781342074 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc003970 con 0x7f89c41f28c0 2024-03-23T11:29:29.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.318+0000 7f89baa00640 1 -- 172.21.15.62:0/781342074 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89bc003ad0 con 0x7f89c41f28c0 2024-03-23T11:29:29.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.318+0000 7f89baa00640 1 -- 172.21.15.62:0/781342074 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc003e80 con 0x7f89c41f28c0 2024-03-23T11:29:29.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.318+0000 7f89c3e00640 1 -- 172.21.15.62:0/781342074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c41418b0 con 0x7f89c41f28c0 2024-03-23T11:29:29.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.318+0000 7f89c3e00640 1 -- 172.21.15.62:0/781342074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c41f2d40 con 0x7f89c41f28c0 2024-03-23T11:29:29.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.319+0000 7f89c3e00640 1 -- 172.21.15.62:0/781342074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c41420b0 con 0x7f89c41f28c0 2024-03-23T11:29:29.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.320+0000 7f89baa00640 1 -- 172.21.15.62:0/781342074 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f89bc0050d0 con 0x7f89c41f28c0 2024-03-23T11:29:29.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.322+0000 7f89baa00640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f89a406a7a0 0x7f89a406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.322+0000 7f89baa00640 1 -- 172.21.15.62:0/781342074 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89bc088af0 con 0x7f89c41f28c0 2024-03-23T11:29:29.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.322+0000 7f89c2a00640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f89a406a7a0 0x7f89a406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.324+0000 7f89c2a00640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f89a406a7a0 0x7f89a406cc60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f89b4001050 tx=0x7f89b4001d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:29.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.326+0000 7f89baa00640 1 -- 172.21.15.62:0/781342074 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f89c41420b0 con 0x7f89c41f28c0 2024-03-23T11:29:29.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.531+0000 7f89c3e00640 1 -- 172.21.15.62:0/781342074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89c41420b0 con 0x7f89c41f28c0 2024-03-23T11:29:29.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.532+0000 7f89baa00640 1 -- 172.21.15.62:0/781342074 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89c41420b0 con 0x7f89c41f28c0 2024-03-23T11:29:29.524 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:29:29.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.535+0000 7f89c3e00640 1 -- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f89a406a7a0 msgr2=0x7f89a406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.535+0000 7f89c3e00640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f89a406a7a0 0x7f89a406cc60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f89b4001050 tx=0x7f89b4001d30 comp rx=0 tx=0).stop 2024-03-23T11:29:29.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.535+0000 7f89c3e00640 1 -- 172.21.15.62:0/781342074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c41f28c0 msgr2=0x7f89c41f5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.535+0000 7f89c3e00640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c41f28c0 0x7f89c41f5430 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f89c406e450 tx=0x7f89bc00ec10 comp rx=0 tx=0).stop 2024-03-23T11:29:29.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.535+0000 7f89c3e00640 1 -- 172.21.15.62:0/781342074 shutdown_connections 2024-03-23T11:29:29.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.535+0000 7f89c3e00640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f89a406a7a0 0x7f89a406cc60 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.535+0000 7f89c3e00640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c41f28c0 0x7f89c41f5430 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.535+0000 7f89c3e00640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f89c41f6080 0x7f89c413b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.535+0000 7f89c3e00640 1 --2- 172.21.15.62:0/781342074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89a80047c0 0x7f89c4142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.535+0000 7f89c3e00640 1 -- 172.21.15.62:0/781342074 >> 172.21.15.62:0/781342074 conn(0x7f89c4066050 msgr2=0x7f89c406d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:29.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.535+0000 7f89c3e00640 1 -- 172.21.15.62:0/781342074 shutdown_connections 2024-03-23T11:29:29.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.536+0000 7f89c3e00640 1 -- 172.21.15.62:0/781342074 wait complete. 2024-03-23T11:29:29.546 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:29:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.776+0000 7f081ac00640 1 Processor -- start 2024-03-23T11:29:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.776+0000 7f081ac00640 1 -- start start 2024-03-23T11:29:29.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.778+0000 7f081ac00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f08141613a0 0x7f081406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.778+0000 7f081ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f081406d910 0x7f081415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.778+0000 7f081ac00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081406cf40 con 0x7f08141571e0 2024-03-23T11:29:29.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.778+0000 7f081ac00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08141575c0 con 0x7f08141613a0 2024-03-23T11:29:29.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.778+0000 7f081ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081406c3a0 con 0x7f081406d910 2024-03-23T11:29:29.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.778+0000 7f0813400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f08141613a0 0x7f081406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.778+0000 7f0813400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f08141613a0 0x7f081406d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42066/0 (socket says 172.21.15.62:42066) 2024-03-23T11:29:29.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.778+0000 7f0813400640 1 -- 172.21.15.62:0/2418195529 learned_addr learned my addr 172.21.15.62:0/2418195529 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:29.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.779+0000 7f0818c00640 1 --2- 172.21.15.62:0/2418195529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f081406d910 0x7f081415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.779+0000 7f0813400640 1 -- 172.21.15.62:0/2418195529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f081406d910 msgr2=0x7f081415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.779+0000 7f0813400640 1 --2- 172.21.15.62:0/2418195529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f081406d910 0x7f081415e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.779+0000 7f0813400640 1 -- 172.21.15.62:0/2418195529 >> v1:172.21.15.120:6789/0 conn(0x7f08141571e0 legacy=0x7f08140ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.779+0000 7f0813400640 1 -- 172.21.15.62:0/2418195529 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08140ae040 con 0x7f08141613a0 2024-03-23T11:29:29.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.780+0000 7f0813400640 1 --2- 172.21.15.62:0/2418195529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08141613a0 0x7f081406d3d0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f0800009870 tx=0x7f080002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae15ba097083cc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:29.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.780+0000 7f0812a00640 1 -- 172.21.15.62:0/2418195529 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800031640 con 0x7f08141613a0 2024-03-23T11:29:29.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.780+0000 7f0812a00640 1 -- 172.21.15.62:0/2418195529 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08000317a0 con 0x7f08141613a0 2024-03-23T11:29:29.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.780+0000 7f0812a00640 1 -- 172.21.15.62:0/2418195529 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800031b30 con 0x7f08141613a0 2024-03-23T11:29:29.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.781+0000 7f081ac00640 1 -- 172.21.15.62:0/2418195529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08141613a0 msgr2=0x7f081406d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.781+0000 7f081ac00640 1 --2- 172.21.15.62:0/2418195529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08141613a0 0x7f081406d3d0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f0800009870 tx=0x7f080002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.781+0000 7f081ac00640 1 -- 172.21.15.62:0/2418195529 shutdown_connections 2024-03-23T11:29:29.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.781+0000 7f081ac00640 1 --2- 172.21.15.62:0/2418195529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f081406d910 0x7f081415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.781+0000 7f081ac00640 1 --2- 172.21.15.62:0/2418195529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08141613a0 0x7f081406d3d0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.781+0000 7f081ac00640 1 -- 172.21.15.62:0/2418195529 >> 172.21.15.62:0/2418195529 conn(0x7f08140b02e0 msgr2=0x7f08140b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:29.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.782+0000 7f081ac00640 1 -- 172.21.15.62:0/2418195529 shutdown_connections 2024-03-23T11:29:29.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.782+0000 7f081ac00640 1 -- 172.21.15.62:0/2418195529 wait complete. 2024-03-23T11:29:29.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.784+0000 7f081ac00640 1 Processor -- start 2024-03-23T11:29:29.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.784+0000 7f081ac00640 1 -- start start 2024-03-23T11:29:29.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.785+0000 7f081ac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f081406d910 0x7f0814141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.785+0000 7f081ac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08141571e0 0x7f08140cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.785+0000 7f081ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08141613a0 0x7f0814155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.785+0000 7f081ac00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814153340 con 0x7f08141571e0 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.785+0000 7f081ac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08141423e0 con 0x7f081406d910 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.785+0000 7f081ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08140cc250 con 0x7f08141613a0 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.786+0000 7f0813400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08141571e0 0x7f08140cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.786+0000 7f0813400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08141571e0 0x7f08140cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42078/0 (socket says 172.21.15.62:42078) 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.786+0000 7f0813400640 1 -- 172.21.15.62:0/386797821 learned_addr learned my addr 172.21.15.62:0/386797821 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.786+0000 7f0813e00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f081406d910 0x7f0814141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.786+0000 7f0818c00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08141613a0 0x7f0814155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.786+0000 7f0813400640 1 -- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08141613a0 msgr2=0x7f0814155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.786+0000 7f0813400640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08141613a0 0x7f0814155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.786+0000 7f0813400640 1 -- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f081406d910 msgr2=0x7f0814141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.786+0000 7f0813400640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f081406d910 0x7f0814141ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.786+0000 7f0813400640 1 -- 172.21.15.62:0/386797821 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08000094d0 con 0x7f08141571e0 2024-03-23T11:29:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.786+0000 7f0818c00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08141613a0 0x7f0814155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.787+0000 7f0813400640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08141571e0 0x7f08140cae00 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f0800000c00 tx=0x7f08000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.787+0000 7f080be00640 1 -- 172.21.15.62:0/386797821 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080003ac60 con 0x7f08141571e0 2024-03-23T11:29:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.787+0000 7f080be00640 1 -- 172.21.15.62:0/386797821 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080003adc0 con 0x7f08141571e0 2024-03-23T11:29:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.787+0000 7f080be00640 1 -- 172.21.15.62:0/386797821 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800042630 con 0x7f08141571e0 2024-03-23T11:29:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.787+0000 7f081ac00640 1 -- 172.21.15.62:0/386797821 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08140cc530 con 0x7f08141571e0 2024-03-23T11:29:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.787+0000 7f081ac00640 1 -- 172.21.15.62:0/386797821 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08140cba10 con 0x7f08141571e0 2024-03-23T11:29:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.788+0000 7f081ac00640 1 -- 172.21.15.62:0/386797821 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0814066c90 con 0x7f08141571e0 2024-03-23T11:29:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.789+0000 7f080be00640 1 -- 172.21.15.62:0/386797821 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f080002f050 con 0x7f08141571e0 2024-03-23T11:29:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.791+0000 7f080be00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f07f406a7a0 0x7f07f406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:29.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.791+0000 7f080be00640 1 -- 172.21.15.62:0/386797821 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08000aa690 con 0x7f08141571e0 2024-03-23T11:29:29.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.791+0000 7f0813e00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f07f406a7a0 0x7f07f406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:29.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.792+0000 7f0813e00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f07f406a7a0 0x7f07f406cc60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f08040015a0 tx=0x7f0804009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.795+0000 7f080be00640 1 -- 172.21.15.62:0/386797821 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0800075a10 con 0x7f08141571e0 2024-03-23T11:29:29.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.994+0000 7f081ac00640 1 -- 172.21.15.62:0/386797821 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f08140b38c0 con 0x7f08141571e0 2024-03-23T11:29:29.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.995+0000 7f080be00640 1 -- 172.21.15.62:0/386797821 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0800031e60 con 0x7f08141571e0 2024-03-23T11:29:29.986 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:29:29.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 -- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f07f406a7a0 msgr2=0x7f07f406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f07f406a7a0 0x7f07f406cc60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f08040015a0 tx=0x7f0804009040 comp rx=0 tx=0).stop 2024-03-23T11:29:29.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 -- 172.21.15.62:0/386797821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08141571e0 msgr2=0x7f08140cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:29.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08141571e0 0x7f08140cae00 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f0800000c00 tx=0x7f08000046c0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 -- 172.21.15.62:0/386797821 shutdown_connections 2024-03-23T11:29:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f07f406a7a0 0x7f07f406cc60 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08141571e0 0x7f08140cae00 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f08141613a0 0x7f0814155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 --2- 172.21.15.62:0/386797821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f081406d910 0x7f0814141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 -- 172.21.15.62:0/386797821 >> 172.21.15.62:0/386797821 conn(0x7f08140b02e0 msgr2=0x7f08140ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 -- 172.21.15.62:0/386797821 shutdown_connections 2024-03-23T11:29:29.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:29.998+0000 7f081ac00640 1 -- 172.21.15.62:0/386797821 wait complete. 2024-03-23T11:29:30.010 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:29:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.232+0000 7f627d200640 1 Processor -- start 2024-03-23T11:29:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.233+0000 7f627d200640 1 -- start start 2024-03-23T11:29:30.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.234+0000 7f627d200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6278069c60 0x7f627806a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:30.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.234+0000 7f627d200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62780caff0 0x7f62780cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:30.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.234+0000 7f627d200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627815bb10 con 0x7f6278143450 2024-03-23T11:29:30.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.234+0000 7f627d200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62780ce1a0 con 0x7f6278069c60 2024-03-23T11:29:30.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.234+0000 7f627d200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62780cbcb0 con 0x7f62780caff0 2024-03-23T11:29:30.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.235+0000 7f6277400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6278069c60 0x7f627806a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:30.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.235+0000 7f6277400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6278069c60 0x7f627806a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42088/0 (socket says 172.21.15.62:42088) 2024-03-23T11:29:30.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.235+0000 7f6277400640 1 -- 172.21.15.62:0/1975994136 learned_addr learned my addr 172.21.15.62:0/1975994136 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.235+0000 7f6276a00640 1 --2- 172.21.15.62:0/1975994136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62780caff0 0x7f62780cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.235+0000 7f6277400640 1 -- 172.21.15.62:0/1975994136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62780caff0 msgr2=0x7f62780cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.236+0000 7f6277400640 1 --2- 172.21.15.62:0/1975994136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62780caff0 0x7f62780cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.236+0000 7f6277400640 1 -- 172.21.15.62:0/1975994136 >> v1:172.21.15.120:6789/0 conn(0x7f6278143450 legacy=0x7f62781438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.236+0000 7f6277400640 1 -- 172.21.15.62:0/1975994136 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627806a580 con 0x7f6278069c60 2024-03-23T11:29:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.237+0000 7f6277400640 1 --2- 172.21.15.62:0/1975994136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278069c60 0x7f627806a040 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f62680098a0 tx=0x7f626802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c003ecbf8354bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.238+0000 7f6276000640 1 -- 172.21.15.62:0/1975994136 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268031640 con 0x7f6278069c60 2024-03-23T11:29:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.238+0000 7f6276000640 1 -- 172.21.15.62:0/1975994136 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62680317a0 con 0x7f6278069c60 2024-03-23T11:29:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.238+0000 7f6276000640 1 -- 172.21.15.62:0/1975994136 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268031b30 con 0x7f6278069c60 2024-03-23T11:29:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.238+0000 7f627d200640 1 -- 172.21.15.62:0/1975994136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278069c60 msgr2=0x7f627806a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.238+0000 7f627d200640 1 --2- 172.21.15.62:0/1975994136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278069c60 0x7f627806a040 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f62680098a0 tx=0x7f626802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:29:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.239+0000 7f627d200640 1 -- 172.21.15.62:0/1975994136 shutdown_connections 2024-03-23T11:29:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.239+0000 7f627d200640 1 --2- 172.21.15.62:0/1975994136 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62780caff0 0x7f62780cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.239+0000 7f627d200640 1 --2- 172.21.15.62:0/1975994136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278069c60 0x7f627806a040 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.239+0000 7f627d200640 1 -- 172.21.15.62:0/1975994136 >> 172.21.15.62:0/1975994136 conn(0x7f62780b0090 msgr2=0x7f62780b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.239+0000 7f627d200640 1 -- 172.21.15.62:0/1975994136 shutdown_connections 2024-03-23T11:29:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.239+0000 7f627d200640 1 -- 172.21.15.62:0/1975994136 wait complete. 2024-03-23T11:29:30.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.241+0000 7f627d200640 1 Processor -- start 2024-03-23T11:29:30.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.241+0000 7f627d200640 1 -- start start 2024-03-23T11:29:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.242+0000 7f627d200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278069c60 0x7f62781f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f627d200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62780caff0 0x7f62781f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f627d200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6278143450 0x7f62781f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f627d200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62781f0e80 con 0x7f6278069c60 2024-03-23T11:29:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f627d200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62781f1ce0 con 0x7f6278143450 2024-03-23T11:29:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f627d200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62781f10d0 con 0x7f62780caff0 2024-03-23T11:29:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6277400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278069c60 0x7f62781f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6277400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278069c60 0x7f62781f32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42094/0 (socket says 172.21.15.62:42094) 2024-03-23T11:29:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6277400640 1 -- 172.21.15.62:0/1118033402 learned_addr learned my addr 172.21.15.62:0/1118033402 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6276a00640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62780caff0 0x7f62781f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6277e00640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6278143450 0x7f62781f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6277400640 1 -- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62780caff0 msgr2=0x7f62781f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6277400640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62780caff0 0x7f62781f37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6277400640 1 -- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6278143450 msgr2=0x7f62781f2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6277400640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6278143450 0x7f62781f2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6277400640 1 -- 172.21.15.62:0/1118033402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62680094d0 con 0x7f6278069c60 2024-03-23T11:29:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6276a00640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62780caff0 0x7f62781f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.243+0000 7f6277e00640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6278143450 0x7f62781f2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.244+0000 7f6277400640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278069c60 0x7f62781f32b0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f62680098a0 tx=0x7f62680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.244+0000 7f626f400640 1 -- 172.21.15.62:0/1118033402 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626803ac60 con 0x7f6278069c60 2024-03-23T11:29:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.244+0000 7f626f400640 1 -- 172.21.15.62:0/1118033402 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626803adc0 con 0x7f6278069c60 2024-03-23T11:29:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.244+0000 7f627d200640 1 -- 172.21.15.62:0/1118033402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62781f13b0 con 0x7f6278069c60 2024-03-23T11:29:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.244+0000 7f626f400640 1 -- 172.21.15.62:0/1118033402 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268044630 con 0x7f6278069c60 2024-03-23T11:29:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.244+0000 7f627d200640 1 -- 172.21.15.62:0/1118033402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62781f1970 con 0x7f6278069c60 2024-03-23T11:29:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.246+0000 7f627d200640 1 -- 172.21.15.62:0/1118033402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6278066bb0 con 0x7f6278069c60 2024-03-23T11:29:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.246+0000 7f626f400640 1 -- 172.21.15.62:0/1118033402 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f626802f050 con 0x7f6278069c60 2024-03-23T11:29:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.248+0000 7f626f400640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f625806a7d0 0x7f625806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.248+0000 7f626f400640 1 -- 172.21.15.62:0/1118033402 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62680aac10 con 0x7f6278069c60 2024-03-23T11:29:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.248+0000 7f6276a00640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f625806a7d0 0x7f625806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.249+0000 7f6276a00640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f625806a7d0 0x7f625806cc90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f62781429e0 tx=0x7f626400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:30.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.252+0000 7f626f400640 1 -- 172.21.15.62:0/1118033402 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6268076030 con 0x7f6278069c60 2024-03-23T11:29:30.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.465+0000 7f627d200640 1 -- 172.21.15.62:0/1118033402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f62780ae180 con 0x7f6278069c60 2024-03-23T11:29:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.466+0000 7f626f400640 1 -- 172.21.15.62:0/1118033402 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v105) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6268031e60 con 0x7f6278069c60 2024-03-23T11:29:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:29:30.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 -- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f625806a7d0 msgr2=0x7f625806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:30.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f625806a7d0 0x7f625806cc90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f62781429e0 tx=0x7f626400b040 comp rx=0 tx=0).stop 2024-03-23T11:29:30.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 -- 172.21.15.62:0/1118033402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278069c60 msgr2=0x7f62781f32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:30.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278069c60 0x7f62781f32b0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f62680098a0 tx=0x7f62680046c0 comp rx=0 tx=0).stop 2024-03-23T11:29:30.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 -- 172.21.15.62:0/1118033402 shutdown_connections 2024-03-23T11:29:30.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f625806a7d0 0x7f625806cc90 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6278143450 0x7f62781f2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62780caff0 0x7f62781f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 --2- 172.21.15.62:0/1118033402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278069c60 0x7f62781f32b0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 -- 172.21.15.62:0/1118033402 >> 172.21.15.62:0/1118033402 conn(0x7f62780b0090 msgr2=0x7f627806de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 -- 172.21.15.62:0/1118033402 shutdown_connections 2024-03-23T11:29:30.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:30.469+0000 7f627d200640 1 -- 172.21.15.62:0/1118033402 wait complete. 2024-03-23T11:29:30.479 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:29:30.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:30.480 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:30.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[72544]: fuse finished with error 0 and tester_r 0 2024-03-23T11:29:30.587 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:29:36.689 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:29:36.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:36.689 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:29:36.750 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:29:36.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.990+0000 7ffa83600640 1 Processor -- start 2024-03-23T11:29:36.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.991+0000 7ffa83600640 1 -- start start 2024-03-23T11:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.992+0000 7ffa83600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffa7c0b2f90 0x7ffa7c161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.992+0000 7ffa83600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7c0ae0c0 con 0x7ffa7c06c300 2024-03-23T11:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.992+0000 7ffa83600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7c0b2740 con 0x7ffa7c0b15e0 2024-03-23T11:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.992+0000 7ffa83600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa7c1623b0 con 0x7ffa7c0b2f90 2024-03-23T11:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.993+0000 7ffa81600640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ffa7c06c300 0x7ffa7c06c7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:52044/0 (socket says 172.21.15.62:52044) 2024-03-23T11:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.993+0000 7ffa81600640 1 -- 172.21.15.62:0/2128937786 learned_addr learned my addr 172.21.15.62:0/2128937786 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.993+0000 7ffa80c00640 1 --2- 172.21.15.62:0/2128937786 >> v2:172.21.15.120:3300/0 conn(0x7ffa7c0b2f90 0x7ffa7c161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.993+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278634380 0 0) 0x7ffa7c0ae0c0 con 0x7ffa7c06c300 2024-03-23T11:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.993+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa60003520 con 0x7ffa7c06c300 2024-03-23T11:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1855670192 0 0) 0x7ffa60003520 con 0x7ffa7c06c300 2024-03-23T11:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 >> v2:172.21.15.120:3300/0 conn(0x7ffa7c0b2f90 msgr2=0x7ffa7c161e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 --2- 172.21.15.62:0/2128937786 >> v2:172.21.15.120:3300/0 conn(0x7ffa7c0b2f90 0x7ffa7c161e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 >> v1:172.21.15.120:6789/0 conn(0x7ffa7c0b15e0 legacy=0x7ffa7c0b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7c0b1100 con 0x7ffa7c06c300 2024-03-23T11:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7ffa740019f0 con 0x7ffa7c06c300 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 >> v1:172.21.15.62:6789/0 conn(0x7ffa7c06c300 legacy=0x7ffa7c06c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa60003970 0x7ffa60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa600047c0 0x7ffa60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa60005180 0x7ffa6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c1623b0 con 0x7ffa60005180 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6000c020 con 0x7ffa60003970 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6000c1e0 con 0x7ffa600047c0 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa81600640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa60005180 0x7ffa6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.994+0000 7ffa80c00640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa60003970 0x7ffa60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.995+0000 7ffa7be00640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa600047c0 0x7ffa60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.995+0000 7ffa81600640 1 -- 172.21.15.62:0/2128937786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa600047c0 msgr2=0x7ffa60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.995+0000 7ffa81600640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa600047c0 0x7ffa60004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.995+0000 7ffa81600640 1 -- 172.21.15.62:0/2128937786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa60003970 msgr2=0x7ffa60003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.995+0000 7ffa81600640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa60003970 0x7ffa60003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.995+0000 7ffa81600640 1 -- 172.21.15.62:0/2128937786 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa60003520 con 0x7ffa60005180 2024-03-23T11:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.995+0000 7ffa80c00640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa60003970 0x7ffa60003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.996+0000 7ffa81600640 1 -- 172.21.15.62:0/2128937786 reap_dead start 2024-03-23T11:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.996+0000 7ffa81600640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa60005180 0x7ffa6000b980 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7ffa7400a830 tx=0x7ffa600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab202b906d9a45fe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.996+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa740072d0 con 0x7ffa60005180 2024-03-23T11:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.996+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7402f070 con 0x7ffa60005180 2024-03-23T11:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.996+0000 7ffa7b400640 1 -- 172.21.15.62:0/2128937786 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa740076b0 con 0x7ffa60005180 2024-03-23T11:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.997+0000 7ffa83600640 1 -- 172.21.15.62:0/2128937786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa60005180 msgr2=0x7ffa6000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.997+0000 7ffa83600640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa60005180 0x7ffa6000b980 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7ffa7400a830 tx=0x7ffa600057f0 comp rx=0 tx=0).stop 2024-03-23T11:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.997+0000 7ffa83600640 1 -- 172.21.15.62:0/2128937786 shutdown_connections 2024-03-23T11:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.997+0000 7ffa83600640 1 --2- 172.21.15.62:0/2128937786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa60005180 0x7ffa6000b980 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.997+0000 7ffa83600640 1 -- 172.21.15.62:0/2128937786 >> 172.21.15.62:0/2128937786 conn(0x7ffa7c0b02e0 msgr2=0x7ffa7c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.997+0000 7ffa83600640 1 -- 172.21.15.62:0/2128937786 shutdown_connections 2024-03-23T11:29:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.997+0000 7ffa83600640 1 -- 172.21.15.62:0/2128937786 wait complete. 2024-03-23T11:29:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.999+0000 7ffa83600640 1 Processor -- start 2024-03-23T11:29:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:36.999+0000 7ffa83600640 1 -- start start 2024-03-23T11:29:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.000+0000 7ffa83600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa60005180 0x7ffa7c1f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.000+0000 7ffa83600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c1f9c50 0x7ffa7c0c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.000+0000 7ffa83600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c1fa310 0x7ffa7c1fa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.000+0000 7ffa83600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c0ab460 con 0x7ffa7c1fa310 2024-03-23T11:29:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.000+0000 7ffa83600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c1facd0 con 0x7ffa7c1f9c50 2024-03-23T11:29:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.000+0000 7ffa83600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c1fa0d0 con 0x7ffa60005180 2024-03-23T11:29:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.000+0000 7ffa81600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c1fa310 0x7ffa7c1fa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.000+0000 7ffa81600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c1fa310 0x7ffa7c1fa790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42114/0 (socket says 172.21.15.62:42114) 2024-03-23T11:29:36.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.000+0000 7ffa81600640 1 -- 172.21.15.62:0/2507525339 learned_addr learned my addr 172.21.15.62:0/2507525339 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.000+0000 7ffa80c00640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa60005180 0x7ffa7c1f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.000+0000 7ffa7be00640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c1f9c50 0x7ffa7c0c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.001+0000 7ffa81600640 1 -- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa60005180 msgr2=0x7ffa7c1f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.001+0000 7ffa81600640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa60005180 0x7ffa7c1f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.001+0000 7ffa81600640 1 -- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c1f9c50 msgr2=0x7ffa7c0c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.001+0000 7ffa81600640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c1f9c50 0x7ffa7c0c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.001+0000 7ffa81600640 1 -- 172.21.15.62:0/2507525339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7c0ab180 con 0x7ffa7c1fa310 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.001+0000 7ffa81600640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c1fa310 0x7ffa7c1fa790 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7ffa74000f80 tx=0x7ffa74001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.001+0000 7ffa78c00640 1 -- 172.21.15.62:0/2507525339 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74004e20 con 0x7ffa7c1fa310 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.001+0000 7ffa78c00640 1 -- 172.21.15.62:0/2507525339 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa740017b0 con 0x7ffa7c1fa310 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.001+0000 7ffa83600640 1 -- 172.21.15.62:0/2507525339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7c0ab960 con 0x7ffa7c1fa310 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.001+0000 7ffa78c00640 1 -- 172.21.15.62:0/2507525339 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74001b20 con 0x7ffa7c1fa310 2024-03-23T11:29:36.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.001+0000 7ffa83600640 1 -- 172.21.15.62:0/2507525339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7c0abf20 con 0x7ffa7c1fa310 2024-03-23T11:29:36.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.003+0000 7ffa83600640 1 -- 172.21.15.62:0/2507525339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa7c0ac140 con 0x7ffa7c1fa310 2024-03-23T11:29:36.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.004+0000 7ffa78c00640 1 -- 172.21.15.62:0/2507525339 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ffa74031030 con 0x7ffa7c1fa310 2024-03-23T11:29:36.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.004+0000 7ffa78c00640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa5806a7d0 0x7ffa5806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:36.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.005+0000 7ffa78c00640 1 -- 172.21.15.62:0/2507525339 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa740aa4c0 con 0x7ffa7c1fa310 2024-03-23T11:29:36.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.005+0000 7ffa80c00640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa5806a7d0 0x7ffa5806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:36.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.006+0000 7ffa80c00640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa5806a7d0 0x7ffa5806cc90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009990 tx=0x7ffa6c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:37.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.010+0000 7ffa78c00640 1 -- 172.21.15.62:0/2507525339 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa740758f0 con 0x7ffa7c1fa310 2024-03-23T11:29:37.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.252+0000 7ffa83600640 1 -- 172.21.15.62:0/2507525339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa7c06e510 con 0x7ffa7c1fa310 2024-03-23T11:29:37.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.253+0000 7ffa78c00640 1 -- 172.21.15.62:0/2507525339 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4739 (secure 0 0 0) 0x7ffa74078f10 con 0x7ffa7c1fa310 2024-03-23T11:29:37.245 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:37.245 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":78,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:29:19.157087+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8271":{"gid":8271,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6834/692790652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":692790652},{"type":"v1","addr":"172.21.15.62:6834","nonce":692790652}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:29:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 78 2024-03-23T11:29:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.255+0000 7ffa83600640 1 -- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa5806a7d0 msgr2=0x7ffa5806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.255+0000 7ffa83600640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa5806a7d0 0x7ffa5806cc90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009990 tx=0x7ffa6c008040 comp rx=0 tx=0).stop 2024-03-23T11:29:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.255+0000 7ffa83600640 1 -- 172.21.15.62:0/2507525339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c1fa310 msgr2=0x7ffa7c1fa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.255+0000 7ffa83600640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c1fa310 0x7ffa7c1fa790 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7ffa74000f80 tx=0x7ffa74001d80 comp rx=0 tx=0).stop 2024-03-23T11:29:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.256+0000 7ffa83600640 1 -- 172.21.15.62:0/2507525339 shutdown_connections 2024-03-23T11:29:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.256+0000 7ffa83600640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa5806a7d0 0x7ffa5806cc90 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:37.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.256+0000 7ffa83600640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c1fa310 0x7ffa7c1fa790 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.256+0000 7ffa83600640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c1f9c50 0x7ffa7c0c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.256+0000 7ffa83600640 1 --2- 172.21.15.62:0/2507525339 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa60005180 0x7ffa7c1f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.256+0000 7ffa83600640 1 -- 172.21.15.62:0/2507525339 >> 172.21.15.62:0/2507525339 conn(0x7ffa7c0b02e0 msgr2=0x7ffa7c0af160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.256+0000 7ffa83600640 1 -- 172.21.15.62:0/2507525339 shutdown_connections 2024-03-23T11:29:37.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.256+0000 7ffa83600640 1 -- 172.21.15.62:0/2507525339 wait complete. 2024-03-23T11:29:37.269 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-23T11:29:37.534 INFO:teuthology.orchestra.run.smithi062.stdout:{"message":"","return_code":0} 2024-03-23T11:29:37.535 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-23T11:29:37.535 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:29:37.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.768+0000 7f1376200640 1 Processor -- start 2024-03-23T11:29:37.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.769+0000 7f1376200640 1 -- start start 2024-03-23T11:29:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.770+0000 7f1376200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f137015fcf0 0x7f13700d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.770+0000 7f1376200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f13700b6dc0 0x7f13701608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.770+0000 7f1376200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13700d3540 con 0x7f1370160420 2024-03-23T11:29:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.770+0000 7f1376200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1370160df0 con 0x7f137015fcf0 2024-03-23T11:29:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.770+0000 7f1376200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13700ce510 con 0x7f13700b6dc0 2024-03-23T11:29:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.770+0000 7f1374e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f137015fcf0 0x7f13700d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.770+0000 7f1374e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f137015fcf0 0x7f13700d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35302/0 (socket says 172.21.15.62:35302) 2024-03-23T11:29:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.770+0000 7f1374e00640 1 -- 172.21.15.62:0/1750964204 learned_addr learned my addr 172.21.15.62:0/1750964204 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.771+0000 7f1374e00640 1 -- 172.21.15.62:0/1750964204 >> v2:172.21.15.120:3300/0 conn(0x7f13700b6dc0 msgr2=0x7f13701608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.771+0000 7f1375800640 1 --2- 172.21.15.62:0/1750964204 >> v2:172.21.15.120:3300/0 conn(0x7f13700b6dc0 0x7f13701608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.771+0000 7f1374e00640 1 --2- 172.21.15.62:0/1750964204 >> v2:172.21.15.120:3300/0 conn(0x7f13700b6dc0 0x7f13701608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.771+0000 7f1374e00640 1 -- 172.21.15.62:0/1750964204 >> v1:172.21.15.120:6789/0 conn(0x7f1370160420 legacy=0x7f13700d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.771+0000 7f1374e00640 1 -- 172.21.15.62:0/1750964204 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13700cdc70 con 0x7f137015fcf0 2024-03-23T11:29:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.771+0000 7f1375800640 1 --2- 172.21.15.62:0/1750964204 >> v2:172.21.15.120:3300/0 conn(0x7f13700b6dc0 0x7f13701608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.772+0000 7f1374e00640 1 --2- 172.21.15.62:0/1750964204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f137015fcf0 0x7f13700d3000 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f137008bb20 tx=0x7f136002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78446ec2c87790fa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.772+0000 7f136f400640 1 -- 172.21.15.62:0/1750964204 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13600317e0 con 0x7f137015fcf0 2024-03-23T11:29:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.773+0000 7f136f400640 1 -- 172.21.15.62:0/1750964204 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1360031940 con 0x7f137015fcf0 2024-03-23T11:29:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.773+0000 7f136f400640 1 -- 172.21.15.62:0/1750964204 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360031cd0 con 0x7f137015fcf0 2024-03-23T11:29:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.773+0000 7f1376200640 1 -- 172.21.15.62:0/1750964204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f137015fcf0 msgr2=0x7f13700d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.773+0000 7f1376200640 1 --2- 172.21.15.62:0/1750964204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f137015fcf0 0x7f13700d3000 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f137008bb20 tx=0x7f136002b880 comp rx=0 tx=0).stop 2024-03-23T11:29:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.773+0000 7f1376200640 1 -- 172.21.15.62:0/1750964204 shutdown_connections 2024-03-23T11:29:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.773+0000 7f1376200640 1 --2- 172.21.15.62:0/1750964204 >> v2:172.21.15.120:3300/0 conn(0x7f13700b6dc0 0x7f13701608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.773+0000 7f1376200640 1 --2- 172.21.15.62:0/1750964204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f137015fcf0 0x7f13700d3000 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.773+0000 7f1376200640 1 -- 172.21.15.62:0/1750964204 >> 172.21.15.62:0/1750964204 conn(0x7f1370089f60 msgr2=0x7f13700d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.773+0000 7f1376200640 1 -- 172.21.15.62:0/1750964204 shutdown_connections 2024-03-23T11:29:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.773+0000 7f1376200640 1 -- 172.21.15.62:0/1750964204 wait complete. 2024-03-23T11:29:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.775+0000 7f1376200640 1 Processor -- start 2024-03-23T11:29:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.775+0000 7f1376200640 1 -- start start 2024-03-23T11:29:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.776+0000 7f1376200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13700b6dc0 0x7f13701f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.776+0000 7f1376200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f137015fcf0 0x7f137009cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.776+0000 7f1376200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1370160420 0x7f137009d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.776+0000 7f1376200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137015c4c0 con 0x7f1370160420 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.776+0000 7f1376200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13701f2340 con 0x7f13700b6dc0 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.776+0000 7f1376200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13701f2620 con 0x7f137015fcf0 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.777+0000 7f1375800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1370160420 0x7f137009d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.777+0000 7f1375800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1370160420 0x7f137009d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35310/0 (socket says 172.21.15.62:35310) 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.777+0000 7f1375800640 1 -- 172.21.15.62:0/2681292080 learned_addr learned my addr 172.21.15.62:0/2681292080 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.777+0000 7f136fe00640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f137015fcf0 0x7f137009cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.777+0000 7f1374e00640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13700b6dc0 0x7f13701f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.777+0000 7f1375800640 1 -- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f137015fcf0 msgr2=0x7f137009cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.777+0000 7f1375800640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f137015fcf0 0x7f137009cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.777+0000 7f1375800640 1 -- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13700b6dc0 msgr2=0x7f13701f1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.777+0000 7f1375800640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13700b6dc0 0x7f13701f1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:37.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.777+0000 7f1375800640 1 -- 172.21.15.62:0/2681292080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13600094d0 con 0x7f1370160420 2024-03-23T11:29:37.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.778+0000 7f1375800640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1370160420 0x7f137009d190 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f136800cef0 tx=0x7f136800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:37.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.778+0000 7f136cc00640 1 -- 172.21.15.62:0/2681292080 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136800d840 con 0x7f1370160420 2024-03-23T11:29:37.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.778+0000 7f136cc00640 1 -- 172.21.15.62:0/2681292080 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136800d9c0 con 0x7f1370160420 2024-03-23T11:29:37.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.778+0000 7f1376200640 1 -- 172.21.15.62:0/2681292080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13701f2900 con 0x7f1370160420 2024-03-23T11:29:37.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.778+0000 7f136cc00640 1 -- 172.21.15.62:0/2681292080 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136800dd70 con 0x7f1370160420 2024-03-23T11:29:37.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.778+0000 7f1376200640 1 -- 172.21.15.62:0/2681292080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13701f2ec0 con 0x7f1370160420 2024-03-23T11:29:37.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.779+0000 7f1376200640 1 -- 172.21.15.62:0/2681292080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13701f30e0 con 0x7f1370160420 2024-03-23T11:29:37.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.780+0000 7f136cc00640 1 -- 172.21.15.62:0/2681292080 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f136801e3b0 con 0x7f1370160420 2024-03-23T11:29:37.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.781+0000 7f136cc00640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f135006a7a0 0x7f135006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:37.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.781+0000 7f136cc00640 1 -- 172.21.15.62:0/2681292080 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1368089910 con 0x7f1370160420 2024-03-23T11:29:37.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.782+0000 7f1374e00640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f135006a7a0 0x7f135006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:37.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.783+0000 7f1374e00640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f135006a7a0 0x7f135006cc60 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1360036000 tx=0x7f136003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:37.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:37.786+0000 7f136cc00640 1 -- 172.21.15.62:0/2681292080 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1368054d50 con 0x7f1370160420 2024-03-23T11:29:38.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.027+0000 7f1376200640 1 -- 172.21.15.62:0/2681292080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1370066bb0 con 0x7f1370160420 2024-03-23T11:29:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.028+0000 7f136cc00640 1 -- 172.21.15.62:0/2681292080 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f136801a020 con 0x7f1370160420 2024-03-23T11:29:38.021 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:38.021 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":78,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:29:19.157087+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8271":{"gid":8271,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6834/692790652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":692790652},{"type":"v1","addr":"172.21.15.62:6834","nonce":692790652}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:29:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 78 2024-03-23T11:29:38.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.031+0000 7f1376200640 1 -- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f135006a7a0 msgr2=0x7f135006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.031+0000 7f1376200640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f135006a7a0 0x7f135006cc60 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1360036000 tx=0x7f136003d040 comp rx=0 tx=0).stop 2024-03-23T11:29:38.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.031+0000 7f1376200640 1 -- 172.21.15.62:0/2681292080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1370160420 msgr2=0x7f137009d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.031+0000 7f1376200640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1370160420 0x7f137009d190 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f136800cef0 tx=0x7f136800a710 comp rx=0 tx=0).stop 2024-03-23T11:29:38.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.031+0000 7f1376200640 1 -- 172.21.15.62:0/2681292080 shutdown_connections 2024-03-23T11:29:38.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.031+0000 7f1376200640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f135006a7a0 0x7f135006cc60 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.031+0000 7f1376200640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1370160420 0x7f137009d190 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.031+0000 7f1376200640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f137015fcf0 0x7f137009cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.031+0000 7f1376200640 1 --2- 172.21.15.62:0/2681292080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13700b6dc0 0x7f13701f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.031+0000 7f1376200640 1 -- 172.21.15.62:0/2681292080 >> 172.21.15.62:0/2681292080 conn(0x7f1370089f60 msgr2=0x7f137006ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:38.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.031+0000 7f1376200640 1 -- 172.21.15.62:0/2681292080 shutdown_connections 2024-03-23T11:29:38.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.032+0000 7f1376200640 1 -- 172.21.15.62:0/2681292080 wait complete. 2024-03-23T11:29:38.044 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata get mds0_inotable - 2024-03-23T11:29:38.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.161+0000 7f24b4a44f40 1 Processor -- start 2024-03-23T11:29:38.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.162+0000 7f24b4a44f40 1 -- start start 2024-03-23T11:29:38.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.162+0000 7f24b4a44f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 0x557ea079ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.162+0000 7f24b4a44f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ea07bbc20 0x557ea07ba7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.162+0000 7f24b4a44f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ea07bc710 0x557ea07bad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b4a44f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ea062c800 con 0x557ea0717910 2024-03-23T11:29:38.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b4a44f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ea062c4a0 con 0x557ea07bbc20 2024-03-23T11:29:38.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b4a44f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ea0623110 con 0x557ea07bc710 2024-03-23T11:29:38.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 0x557ea079ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 0x557ea079ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35324/0 (socket says 172.21.15.62:35324) 2024-03-23T11:29:38.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b1400640 1 -- 172.21.15.62:0/3581058535 learned_addr learned my addr 172.21.15.62:0/3581058535 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:38.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b0a00640 1 --2- 172.21.15.62:0/3581058535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ea07bbc20 0x557ea07ba7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b1e00640 1 --2- 172.21.15.62:0/3581058535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ea07bc710 0x557ea07bad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b1400640 1 -- 172.21.15.62:0/3581058535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ea07bc710 msgr2=0x557ea07bad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:38.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b1400640 1 --2- 172.21.15.62:0/3581058535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ea07bc710 0x557ea07bad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b1400640 1 -- 172.21.15.62:0/3581058535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ea07bbc20 msgr2=0x557ea07ba7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:38.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b1400640 1 --2- 172.21.15.62:0/3581058535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ea07bbc20 0x557ea07ba7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b1400640 1 -- 172.21.15.62:0/3581058535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a4005de0 con 0x557ea0717910 2024-03-23T11:29:38.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.163+0000 7f24b0a00640 1 --2- 172.21.15.62:0/3581058535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ea07bbc20 0x557ea07ba7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:38.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.164+0000 7f24b1400640 1 --2- 172.21.15.62:0/3581058535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 0x557ea079ea80 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f24a401bae0 tx=0x7f24a404c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c671899f7a31e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:38.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.164+0000 7f24ab400640 1 -- 172.21.15.62:0/3581058535 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a40527b0 con 0x557ea0717910 2024-03-23T11:29:38.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.164+0000 7f24ab400640 1 -- 172.21.15.62:0/3581058535 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a4052930 con 0x557ea0717910 2024-03-23T11:29:38.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.164+0000 7f24ab400640 1 -- 172.21.15.62:0/3581058535 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4052ce0 con 0x557ea0717910 2024-03-23T11:29:38.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.165+0000 7f24b4a44f40 1 -- 172.21.15.62:0/3581058535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 msgr2=0x557ea079ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.165+0000 7f24b4a44f40 1 --2- 172.21.15.62:0/3581058535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 0x557ea079ea80 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f24a401bae0 tx=0x7f24a404c000 comp rx=0 tx=0).stop 2024-03-23T11:29:38.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.165+0000 7f24b4a44f40 1 -- 172.21.15.62:0/3581058535 shutdown_connections 2024-03-23T11:29:38.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.165+0000 7f24b4a44f40 1 --2- 172.21.15.62:0/3581058535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ea07bc710 0x557ea07bad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.165+0000 7f24b4a44f40 1 --2- 172.21.15.62:0/3581058535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ea07bbc20 0x557ea07ba7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.165+0000 7f24b4a44f40 1 --2- 172.21.15.62:0/3581058535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 0x557ea079ea80 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.165+0000 7f24b4a44f40 1 -- 172.21.15.62:0/3581058535 >> 172.21.15.62:0/3581058535 conn(0x557ea0731ff0 msgr2=0x557ea07ba0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:38.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.165+0000 7f24b4a44f40 1 -- 172.21.15.62:0/3581058535 shutdown_connections 2024-03-23T11:29:38.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.165+0000 7f24b4a44f40 1 -- 172.21.15.62:0/3581058535 wait complete. 2024-03-23T11:29:38.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.166+0000 7f24b4a44f40 1 Processor -- start 2024-03-23T11:29:38.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.166+0000 7f24b4a44f40 1 -- start start 2024-03-23T11:29:38.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.167+0000 7f24b4a44f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 0x557ea07995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.167+0000 7f24b4a44f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ea07bbc20 0x557ea079d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.167+0000 7f24b4a44f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ea07bc710 0x557ea079d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.167+0000 7f24b4a44f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ea07c1da0 con 0x557ea0717910 2024-03-23T11:29:38.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.167+0000 7f24b4a44f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ea07b80d0 con 0x557ea07bbc20 2024-03-23T11:29:38.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.167+0000 7f24b4a44f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ea07b83b0 con 0x557ea07bc710 2024-03-23T11:29:38.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.167+0000 7f24b1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 0x557ea07995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.167+0000 7f24b1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 0x557ea07995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35326/0 (socket says 172.21.15.62:35326) 2024-03-23T11:29:38.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.167+0000 7f24b1400640 1 -- 172.21.15.62:0/1774444549 learned_addr learned my addr 172.21.15.62:0/1774444549 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:38.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.167+0000 7f24b1e00640 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ea07bc710 0x557ea079d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24b0a00640 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ea07bbc20 0x557ea079d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24b1400640 1 -- 172.21.15.62:0/1774444549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ea07bc710 msgr2=0x557ea079d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24b1400640 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ea07bc710 0x557ea079d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24b1400640 1 -- 172.21.15.62:0/1774444549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ea07bbc20 msgr2=0x557ea079d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24b1400640 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ea07bbc20 0x557ea079d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24b1400640 1 -- 172.21.15.62:0/1774444549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ea06554e0 con 0x557ea0717910 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24b0a00640 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ea07bbc20 0x557ea079d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24b1400640 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 0x557ea07995e0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f24a40445e0 tx=0x7f24a4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24aa000640 1 -- 172.21.15.62:0/1774444549 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a405c850 con 0x557ea0717910 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24aa000640 1 -- 172.21.15.62:0/1774444549 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a405c9d0 con 0x557ea0717910 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24aa000640 1 -- 172.21.15.62:0/1774444549 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a405cd80 con 0x557ea0717910 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.168+0000 7f24b4a44f40 1 -- 172.21.15.62:0/1774444549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557ea063b340 con 0x557ea0717910 2024-03-23T11:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.169+0000 7f24b4a44f40 1 -- 172.21.15.62:0/1774444549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ea062e780 con 0x557ea0717910 2024-03-23T11:29:38.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.171+0000 7f24aa000640 1 -- 172.21.15.62:0/1774444549 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f24a40653f0 con 0x557ea0717910 2024-03-23T11:29:38.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.171+0000 7f24aa000640 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2494069180 0x7f249406b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.172+0000 7f24aa000640 1 -- 172.21.15.62:0/1774444549 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24a4069070 con 0x557ea0717910 2024-03-23T11:29:38.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.172+0000 7f24b4a44f40 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x557ea07ba0a0 0x557ea07ba4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.172+0000 7f24b4a44f40 1 -- 172.21.15.62:0/1774444549 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:1 14.13 14:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e105) v8 -- 0x557ea07ba9f0 con 0x557ea07ba0a0 2024-03-23T11:29:38.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.172+0000 7f24b1e00640 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x557ea07ba0a0 0x557ea07ba4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.172+0000 7f24b0a00640 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2494069180 0x7f249406b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.173+0000 7f24b1e00640 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x557ea07ba0a0 0x557ea07ba4b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:38.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.173+0000 7f24b1e00640 1 -- 172.21.15.62:0/1774444549 <== osd.2 v2:172.21.15.62:6800/36996 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv3 ondisk = 0) v8 ==== 157+0+34 (crc 0 0 0) 0x7f24ac00f4a0 con 0x557ea07ba0a0 2024-03-23T11:29:38.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.173+0000 7f24b4a44f40 1 -- 172.21.15.62:0/1774444549 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x557ea07ba0a0 msgr2=0x557ea07ba4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.173+0000 7f24b4a44f40 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x557ea07ba0a0 0x557ea07ba4b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.173+0000 7f24b4a44f40 1 -- 172.21.15.62:0/1774444549 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2494069180 msgr2=0x7f249406b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.173+0000 7f24b4a44f40 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2494069180 0x7f249406b640 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x557ea0857050 tx=0x7f24a0002580 comp rx=0 tx=0).stop 2024-03-23T11:29:38.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.173+0000 7f24b4a44f40 1 -- 172.21.15.62:0/1774444549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 msgr2=0x557ea07995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.173+0000 7f24b4a44f40 1 --2- 172.21.15.62:0/1774444549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557ea0717910 0x557ea07995e0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f24a40445e0 tx=0x7f24a4002fb0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.174+0000 7f24b1e00640 1 -- 172.21.15.62:0/1774444549 reap_dead start 2024-03-23T11:29:38.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.174+0000 7f24b4a44f40 1 -- 172.21.15.62:0/1774444549 shutdown_connections 2024-03-23T11:29:38.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.174+0000 7f24b4a44f40 1 -- 172.21.15.62:0/1774444549 >> 172.21.15.62:0/1774444549 conn(0x557ea0731ff0 msgr2=0x557ea0712b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:38.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.174+0000 7f24b4a44f40 1 -- 172.21.15.62:0/1774444549 shutdown_connections 2024-03-23T11:29:38.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.174+0000 7f24b4a44f40 1 -- 172.21.15.62:0/1774444549 wait complete. 2024-03-23T11:29:38.170 INFO:teuthology.orchestra.run.smithi062.stdout:����� 2024-03-23T11:29:38.171 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:29:38.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.395+0000 7ff540c00640 1 Processor -- start 2024-03-23T11:29:38.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.396+0000 7ff540c00640 1 -- start start 2024-03-23T11:29:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.397+0000 7ff540c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c15c9e0 0x7ff53c1607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.397+0000 7ff540c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff53c15cdc0 0x7ff53c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.397+0000 7ff540c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff53c0cf650 0x7ff53c0cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.397+0000 7ff540c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53c0d0020 con 0x7ff53c0cf650 2024-03-23T11:29:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.397+0000 7ff540c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53c06bfc0 con 0x7ff53c15cdc0 2024-03-23T11:29:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.397+0000 7ff540c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0b5260 con 0x7ff53c15c9e0 2024-03-23T11:29:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.398+0000 7ff53a800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff53c0cf650 0x7ff53c0cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.398+0000 7ff53a800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff53c0cf650 0x7ff53c0cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35340/0 (socket says 172.21.15.62:35340) 2024-03-23T11:29:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.398+0000 7ff53a800640 1 -- 172.21.15.62:0/3702441425 learned_addr learned my addr 172.21.15.62:0/3702441425 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.398+0000 7ff539e00640 1 --2- 172.21.15.62:0/3702441425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c15c9e0 0x7ff53c1607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.398+0000 7ff539400640 1 --2- 172.21.15.62:0/3702441425 >> v2:172.21.15.120:3300/0 conn(0x7ff53c15cdc0 0x7ff53c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.398+0000 7ff53a800640 1 -- 172.21.15.62:0/3702441425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c15c9e0 msgr2=0x7ff53c1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.398+0000 7ff53a800640 1 --2- 172.21.15.62:0/3702441425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c15c9e0 0x7ff53c1607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.398+0000 7ff53a800640 1 -- 172.21.15.62:0/3702441425 >> v2:172.21.15.120:3300/0 conn(0x7ff53c15cdc0 msgr2=0x7ff53c0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.398+0000 7ff53a800640 1 --2- 172.21.15.62:0/3702441425 >> v2:172.21.15.120:3300/0 conn(0x7ff53c15cdc0 0x7ff53c0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.398+0000 7ff53a800640 1 -- 172.21.15.62:0/3702441425 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c0ce140 con 0x7ff53c0cf650 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.399+0000 7ff53a800640 1 --2- 172.21.15.62:0/3702441425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff53c0cf650 0x7ff53c0cfae0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0b53f0 tx=0x7ff53402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba22bcc614a6af5a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.400+0000 7ff538a00640 1 -- 172.21.15.62:0/3702441425 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53402ca80 con 0x7ff53c0cf650 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.400+0000 7ff538a00640 1 -- 172.21.15.62:0/3702441425 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53402cbe0 con 0x7ff53c0cf650 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.400+0000 7ff538a00640 1 -- 172.21.15.62:0/3702441425 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5340343f0 con 0x7ff53c0cf650 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.400+0000 7ff540c00640 1 -- 172.21.15.62:0/3702441425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff53c0cf650 msgr2=0x7ff53c0cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.400+0000 7ff540c00640 1 --2- 172.21.15.62:0/3702441425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff53c0cf650 0x7ff53c0cfae0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0b53f0 tx=0x7ff53402bee0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.400+0000 7ff540c00640 1 -- 172.21.15.62:0/3702441425 shutdown_connections 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.400+0000 7ff540c00640 1 --2- 172.21.15.62:0/3702441425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff53c0cf650 0x7ff53c0cfae0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.400+0000 7ff540c00640 1 --2- 172.21.15.62:0/3702441425 >> v2:172.21.15.120:3300/0 conn(0x7ff53c15cdc0 0x7ff53c0d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.400+0000 7ff540c00640 1 --2- 172.21.15.62:0/3702441425 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c15c9e0 0x7ff53c1607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.400+0000 7ff540c00640 1 -- 172.21.15.62:0/3702441425 >> 172.21.15.62:0/3702441425 conn(0x7ff53c071570 msgr2=0x7ff53c071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.400+0000 7ff540c00640 1 -- 172.21.15.62:0/3702441425 shutdown_connections 2024-03-23T11:29:38.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.401+0000 7ff540c00640 1 -- 172.21.15.62:0/3702441425 wait complete. 2024-03-23T11:29:38.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.402+0000 7ff540c00640 1 Processor -- start 2024-03-23T11:29:38.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.402+0000 7ff540c00640 1 -- start start 2024-03-23T11:29:38.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.403+0000 7ff540c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff53c0cf650 0x7ff53c15f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.403+0000 7ff540c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c15c9e0 0x7ff53c0c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.403+0000 7ff540c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c15cdc0 0x7ff53c151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.403+0000 7ff540c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0c6700 con 0x7ff53c0cf650 2024-03-23T11:29:38.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.403+0000 7ff540c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c157320 con 0x7ff53c15c9e0 2024-03-23T11:29:38.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.403+0000 7ff540c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0c6b70 con 0x7ff53c15cdc0 2024-03-23T11:29:38.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.404+0000 7ff539e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff53c0cf650 0x7ff53c15f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.404+0000 7ff539e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff53c0cf650 0x7ff53c15f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35348/0 (socket says 172.21.15.62:35348) 2024-03-23T11:29:38.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.404+0000 7ff539e00640 1 -- 172.21.15.62:0/3914737421 learned_addr learned my addr 172.21.15.62:0/3914737421 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:38.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.404+0000 7ff539e00640 1 -- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c15cdc0 msgr2=0x7ff53c151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.404+0000 7ff539e00640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c15cdc0 0x7ff53c151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.404+0000 7ff539400640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c15c9e0 0x7ff53c0c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.404+0000 7ff539e00640 1 -- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c15c9e0 msgr2=0x7ff53c0c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.404+0000 7ff539e00640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c15c9e0 0x7ff53c0c61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.404+0000 7ff539e00640 1 -- 172.21.15.62:0/3914737421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff534009b30 con 0x7ff53c0cf650 2024-03-23T11:29:38.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.404+0000 7ff539e00640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff53c0cf650 0x7ff53c15f5c0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00d800 tx=0x7ff52c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:38.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.405+0000 7ff532000640 1 -- 172.21.15.62:0/3914737421 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c0096a0 con 0x7ff53c0cf650 2024-03-23T11:29:38.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.405+0000 7ff532000640 1 -- 172.21.15.62:0/3914737421 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52c009800 con 0x7ff53c0cf650 2024-03-23T11:29:38.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.405+0000 7ff532000640 1 -- 172.21.15.62:0/3914737421 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c009b90 con 0x7ff53c0cf650 2024-03-23T11:29:38.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.405+0000 7ff540c00640 1 -- 172.21.15.62:0/3914737421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c0c6e50 con 0x7ff53c0cf650 2024-03-23T11:29:38.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.405+0000 7ff540c00640 1 -- 172.21.15.62:0/3914737421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c0c7130 con 0x7ff53c0cf650 2024-03-23T11:29:38.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.406+0000 7ff540c00640 1 -- 172.21.15.62:0/3914737421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53c066c90 con 0x7ff53c0cf650 2024-03-23T11:29:38.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.407+0000 7ff532000640 1 -- 172.21.15.62:0/3914737421 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff52c009cf0 con 0x7ff53c0cf650 2024-03-23T11:29:38.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.409+0000 7ff532000640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff51c06a7a0 0x7ff51c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:38.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.410+0000 7ff532000640 1 -- 172.21.15.62:0/3914737421 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff52c0895d0 con 0x7ff53c0cf650 2024-03-23T11:29:38.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.410+0000 7ff539400640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff51c06a7a0 0x7ff51c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:38.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.411+0000 7ff539400640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff51c06a7a0 0x7ff51c06cc60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff528006fd0 tx=0x7ff528008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:38.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.417+0000 7ff532000640 1 -- 172.21.15.62:0/3914737421 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff52c054950 con 0x7ff53c0cf650 2024-03-23T11:29:38.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.653+0000 7ff540c00640 1 -- 172.21.15.62:0/3914737421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff53c0c5d70 con 0x7ff53c0cf650 2024-03-23T11:29:38.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.654+0000 7ff532000640 1 -- 172.21.15.62:0/3914737421 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v78) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff52c00b440 con 0x7ff53c0cf650 2024-03-23T11:29:38.646 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:29:38.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 -- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff51c06a7a0 msgr2=0x7ff51c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff51c06a7a0 0x7ff51c06cc60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff528006fd0 tx=0x7ff528008040 comp rx=0 tx=0).stop 2024-03-23T11:29:38.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 -- 172.21.15.62:0/3914737421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff53c0cf650 msgr2=0x7ff53c15f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:38.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff53c0cf650 0x7ff53c15f5c0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00d800 tx=0x7ff52c00dcc0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 -- 172.21.15.62:0/3914737421 shutdown_connections 2024-03-23T11:29:38.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff51c06a7a0 0x7ff51c06cc60 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c15cdc0 0x7ff53c151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c15c9e0 0x7ff53c0c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 --2- 172.21.15.62:0/3914737421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff53c0cf650 0x7ff53c15f5c0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:38.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 -- 172.21.15.62:0/3914737421 >> 172.21.15.62:0/3914737421 conn(0x7ff53c071570 msgr2=0x7ff53c0b7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:38.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 -- 172.21.15.62:0/3914737421 shutdown_connections 2024-03-23T11:29:38.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:38.657+0000 7ff540c00640 1 -- 172.21.15.62:0/3914737421 wait complete. 2024-03-23T11:29:38.670 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:29:38.670 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:29:38.670 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:29:38.670 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:29:38.670 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:38.670 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:29:38.670 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:29:38.670 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:29:38.670 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:29:38.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:38.671 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:29:38.727 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 41978sec preferred_lft 41978sec 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:38.728 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:29:38.729 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:29:38.729 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:29:38.729 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:38.729 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:29:38.729 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:29:38.729 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:29:38.729 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:29:38.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:38.730 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:29:38.825 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:29:38.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:38.830 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:29:38.884 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:29:38.884 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:29:38.884 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:29:38.884 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:29:38.884 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:29:38.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:38.885 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:29:38.885 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:29:38.885 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:29:38.885 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:29:38.885 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:29:38.885 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:29:39.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:39.067 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:29:39.118 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:29:39.118 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:29:39.118 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:29:39.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:39.119 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:29:39.209 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:29:39.209 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:29:39.209 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:39.263 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:29:39.263 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:29:39.264 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:39.319 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:29:39.417 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:29:39.418 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:29:39.514 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:29:39.518 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:29:39.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:39.519 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:29:39.575 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:29:39.576 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:29:39.582 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:29:39.666 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:29:39.666 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:29:39.734 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:29:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:29:39.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:39.737 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:29:39.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:29:39.780+0000 7f58f6c4c480 -1 init, newargv = 0x557a0c4f9e20 newargc=15 2024-03-23T11:29:39.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[73624]: starting ceph client 2024-03-23T11:29:39.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[73624]: starting fuse 2024-03-23T11:29:40.878 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:29:40.987 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:29:40.987 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:29:41.082 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:29:41.087 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:29:41.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:41.087 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:29:41.144 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:29:41.144 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:29:41.144 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:29:41.144 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:29:41.144 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:29:41.145 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:29:41.146 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:29:41.146 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:29:41.146 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:29:41.146 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:29:41.146 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:41.146 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:29:41.146 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:29:41.331 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.73624.asok 2024-03-23T11:29:41.511 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.73624.asok 2024-03-23T11:29:41.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:41.511 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.73624.asok status 2024-03-23T11:29:41.758 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:29:41.758 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:29:41.758 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:29:41.758 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:29:41.758 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:29:41.758 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "73624", 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 8541, 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 8541 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:29:41.759 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2298284726 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2298284726 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.8541 192.168.0.1:0/2298284726", 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/2298284726", 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 78, 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 105, 2024-03-23T11:29:41.760 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:29:41.761 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:29:41.761 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:29:41.761 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:29:41.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:41.774 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:29:41.774 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:41.828 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:29:41.828 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:41.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:41.828 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:41.927 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:29:41.927 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:41.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:41.983 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file2_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-03-23T11:29:42.205 INFO:teuthology.orchestra.run.smithi062.stderr:6+0 records in 2024-03-23T11:29:42.205 INFO:teuthology.orchestra.run.smithi062.stderr:6+0 records out 2024-03-23T11:29:42.205 INFO:teuthology.orchestra.run.smithi062.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.157843 s, 39.9 MB/s 2024-03-23T11:29:42.207 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:29:42.207 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:42.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:42.265 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file3_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-03-23T11:29:42.493 INFO:teuthology.orchestra.run.smithi062.stderr:6+0 records in 2024-03-23T11:29:42.494 INFO:teuthology.orchestra.run.smithi062.stderr:6+0 records out 2024-03-23T11:29:42.494 INFO:teuthology.orchestra.run.smithi062.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.170657 s, 36.9 MB/s 2024-03-23T11:29:42.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:42.496 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-23T11:29:42.561 INFO:teuthology.orchestra.run.smithi062.stdout:./ 2024-03-23T11:29:42.561 INFO:teuthology.orchestra.run.smithi062.stdout:./file2_sixmegs 2024-03-23T11:29:42.561 INFO:teuthology.orchestra.run.smithi062.stdout:./file3_sixmegs 2024-03-23T11:29:42.561 INFO:teuthology.orchestra.run.smithi062.stdout:./file1_sixmegs 2024-03-23T11:29:42.563 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:29:42.563 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:29:42.563 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:29:42.563 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:42.563 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-23T11:29:42.563 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:29:42.712 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:29:42.892 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:29:42.893 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:29:42.893 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:29:42.893 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:42.893 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file2_sixmegs").st_ino) 2024-03-23T11:29:42.893 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:29:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627777 2024-03-23T11:29:43.264 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:29:43.265 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:29:43.265 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:29:43.265 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:43.265 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file3_sixmegs").st_ino) 2024-03-23T11:29:43.265 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:29:43.462 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627778 2024-03-23T11:29:43.641 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:29:43.641 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:29:43.641 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:29:43.641 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:29:43.641 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file1_sixmegs").st_ino) 2024-03-23T11:29:43.641 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:29:43.835 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627776 2024-03-23T11:29:44.015 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:29:44.015 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:44.115 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:29:44.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.341+0000 7f6da5e00640 1 Processor -- start 2024-03-23T11:29:44.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.341+0000 7f6da5e00640 1 -- start start 2024-03-23T11:29:44.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.343+0000 7f6da5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6da015fcf0 0x7f6da00d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.343+0000 7f6da5e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6da0160420 0x7f6da0160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.343+0000 7f6da5e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da0160dc0 con 0x7f6da00d06d0 2024-03-23T11:29:44.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.343+0000 7f6da5e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da00d3540 con 0x7f6da015fcf0 2024-03-23T11:29:44.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.343+0000 7f6da5e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da00ce510 con 0x7f6da0160420 2024-03-23T11:29:44.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.343+0000 7f6da4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6da015fcf0 0x7f6da00d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.343+0000 7f6da4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6da015fcf0 0x7f6da00d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35352/0 (socket says 172.21.15.62:35352) 2024-03-23T11:29:44.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.343+0000 7f6da4a00640 1 -- 172.21.15.62:0/206946329 learned_addr learned my addr 172.21.15.62:0/206946329 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:44.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.344+0000 7f6d9fe00640 1 --2- 172.21.15.62:0/206946329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6da0160420 0x7f6da0160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.344+0000 7f6da4a00640 1 -- 172.21.15.62:0/206946329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6da0160420 msgr2=0x7f6da0160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:44.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.344+0000 7f6da4a00640 1 --2- 172.21.15.62:0/206946329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6da0160420 0x7f6da0160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.344+0000 7f6da4a00640 1 -- 172.21.15.62:0/206946329 >> v1:172.21.15.120:6789/0 conn(0x7f6da00d06d0 legacy=0x7f6da00d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:44.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.344+0000 7f6da4a00640 1 -- 172.21.15.62:0/206946329 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da00cdc70 con 0x7f6da015fcf0 2024-03-23T11:29:44.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.345+0000 7f6da4a00640 1 --2- 172.21.15.62:0/206946329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6da015fcf0 0x7f6da00d3000 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009870 tx=0x7f6d9402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3342c49afbe8ddaf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:44.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.345+0000 7f6d9f400640 1 -- 172.21.15.62:0/206946329 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d940317c0 con 0x7f6da015fcf0 2024-03-23T11:29:44.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.346+0000 7f6d9f400640 1 -- 172.21.15.62:0/206946329 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d94031920 con 0x7f6da015fcf0 2024-03-23T11:29:44.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.346+0000 7f6d9f400640 1 -- 172.21.15.62:0/206946329 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d94031cb0 con 0x7f6da015fcf0 2024-03-23T11:29:44.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.346+0000 7f6da5e00640 1 -- 172.21.15.62:0/206946329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6da015fcf0 msgr2=0x7f6da00d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:44.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.346+0000 7f6da5e00640 1 --2- 172.21.15.62:0/206946329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6da015fcf0 0x7f6da00d3000 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009870 tx=0x7f6d9402b860 comp rx=0 tx=0).stop 2024-03-23T11:29:44.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.346+0000 7f6da5e00640 1 -- 172.21.15.62:0/206946329 shutdown_connections 2024-03-23T11:29:44.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.346+0000 7f6da5e00640 1 --2- 172.21.15.62:0/206946329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6da0160420 0x7f6da0160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.346+0000 7f6da5e00640 1 --2- 172.21.15.62:0/206946329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6da015fcf0 0x7f6da00d3000 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.346+0000 7f6da5e00640 1 -- 172.21.15.62:0/206946329 >> 172.21.15.62:0/206946329 conn(0x7f6da0089f60 msgr2=0x7f6da00d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:44.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.346+0000 7f6da5e00640 1 -- 172.21.15.62:0/206946329 shutdown_connections 2024-03-23T11:29:44.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.346+0000 7f6da5e00640 1 -- 172.21.15.62:0/206946329 wait complete. 2024-03-23T11:29:44.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.348+0000 7f6da5e00640 1 Processor -- start 2024-03-23T11:29:44.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.348+0000 7f6da5e00640 1 -- start start 2024-03-23T11:29:44.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.349+0000 7f6da5e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6da00d06d0 0x7f6da009c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.349+0000 7f6da5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6da015fcf0 0x7f6da0152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.349+0000 7f6da5e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6da0160420 0x7f6da015c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.349+0000 7f6da5e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da009ece0 con 0x7f6da015fcf0 2024-03-23T11:29:44.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.349+0000 7f6da5e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da009e3b0 con 0x7f6da00d06d0 2024-03-23T11:29:44.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.349+0000 7f6da5e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da009e690 con 0x7f6da0160420 2024-03-23T11:29:44.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.350+0000 7f6d9fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6da015fcf0 0x7f6da0152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.350+0000 7f6d9fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6da015fcf0 0x7f6da0152860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35366/0 (socket says 172.21.15.62:35366) 2024-03-23T11:29:44.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.350+0000 7f6d9fe00640 1 -- 172.21.15.62:0/3915072594 learned_addr learned my addr 172.21.15.62:0/3915072594 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:44.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.350+0000 7f6da4a00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6da00d06d0 0x7f6da009c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.350+0000 7f6da5400640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6da0160420 0x7f6da015c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.350+0000 7f6d9fe00640 1 -- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6da0160420 msgr2=0x7f6da015c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:44.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.350+0000 7f6d9fe00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6da0160420 0x7f6da015c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.350+0000 7f6d9fe00640 1 -- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6da00d06d0 msgr2=0x7f6da009c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:44.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.350+0000 7f6d9fe00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6da00d06d0 0x7f6da009c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.350+0000 7f6d9fe00640 1 -- 172.21.15.62:0/3915072594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d940094d0 con 0x7f6da015fcf0 2024-03-23T11:29:44.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.351+0000 7f6d9fe00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6da015fcf0 0x7f6da0152860 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00d5f0 tx=0x7f6d8c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:44.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.351+0000 7f6d9cc00640 1 -- 172.21.15.62:0/3915072594 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c0125f0 con 0x7f6da015fcf0 2024-03-23T11:29:44.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.351+0000 7f6d9cc00640 1 -- 172.21.15.62:0/3915072594 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8c012750 con 0x7f6da015fcf0 2024-03-23T11:29:44.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.351+0000 7f6d9cc00640 1 -- 172.21.15.62:0/3915072594 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c012b00 con 0x7f6da015fcf0 2024-03-23T11:29:44.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.351+0000 7f6da5e00640 1 -- 172.21.15.62:0/3915072594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da009e970 con 0x7f6da015fcf0 2024-03-23T11:29:44.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.351+0000 7f6da5e00640 1 -- 172.21.15.62:0/3915072594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da009c8a0 con 0x7f6da015fcf0 2024-03-23T11:29:44.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.352+0000 7f6da5e00640 1 -- 172.21.15.62:0/3915072594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da009ee90 con 0x7f6da015fcf0 2024-03-23T11:29:44.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.353+0000 7f6d9cc00640 1 -- 172.21.15.62:0/3915072594 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6d8c012c60 con 0x7f6da015fcf0 2024-03-23T11:29:44.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.354+0000 7f6d9cc00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d8006a7a0 0x7f6d8006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.355+0000 7f6d9cc00640 1 -- 172.21.15.62:0/3915072594 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d8c0892c0 con 0x7f6da015fcf0 2024-03-23T11:29:44.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.355+0000 7f6da4a00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d8006a7a0 0x7f6d8006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.356+0000 7f6da4a00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d8006a7a0 0x7f6d8006cc60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6d94036000 tx=0x7f6d9403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:44.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.360+0000 7f6d9cc00640 1 -- 172.21.15.62:0/3915072594 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6da009ee90 con 0x7f6da015fcf0 2024-03-23T11:29:44.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.558+0000 7f6da5e00640 1 -- 172.21.15.62:0/3915072594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6da00675b0 con 0x7f6da015fcf0 2024-03-23T11:29:44.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.559+0000 7f6d9cc00640 1 -- 172.21.15.62:0/3915072594 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16051 (secure 0 0 0) 0x7f6d8c054640 con 0x7f6da015fcf0 2024-03-23T11:29:44.551 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:44.552 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":105,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:29:16.631279+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:29:13.817605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-23T11:29:14.652617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:29:44.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 -- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d8006a7a0 msgr2=0x7f6d8006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:44.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d8006a7a0 0x7f6d8006cc60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6d94036000 tx=0x7f6d9403d040 comp rx=0 tx=0).stop 2024-03-23T11:29:44.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 -- 172.21.15.62:0/3915072594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6da015fcf0 msgr2=0x7f6da0152860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:44.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6da015fcf0 0x7f6da0152860 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c00d5f0 tx=0x7f6d8c00dab0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 -- 172.21.15.62:0/3915072594 shutdown_connections 2024-03-23T11:29:44.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6d8006a7a0 0x7f6d8006cc60 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6da015fcf0 0x7f6da0152860 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6da0160420 0x7f6da015c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 --2- 172.21.15.62:0/3915072594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6da00d06d0 0x7f6da009c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 -- 172.21.15.62:0/3915072594 >> 172.21.15.62:0/3915072594 conn(0x7f6da0089f60 msgr2=0x7f6da00b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:44.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 -- 172.21.15.62:0/3915072594 shutdown_connections 2024-03-23T11:29:44.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.562+0000 7f6da5e00640 1 -- 172.21.15.62:0/3915072594 wait complete. 2024-03-23T11:29:44.572 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:29:13.817605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:29:14.652617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:29:44.573 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:29:44.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.803+0000 7f29f8800640 1 Processor -- start 2024-03-23T11:29:44.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.803+0000 7f29f8800640 1 -- start start 2024-03-23T11:29:44.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.804+0000 7f29f8800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f29f00cd7c0 0x7f29f00b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f8800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f29f006c300 0x7f29f00cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f8800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f00cad20 con 0x7f29f006dca0 2024-03-23T11:29:44.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f8800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f006e130 con 0x7f29f00cd7c0 2024-03-23T11:29:44.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f8800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f015a8a0 con 0x7f29f006c300 2024-03-23T11:29:44.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f29f00cd7c0 0x7f29f00b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f29f00cd7c0 0x7f29f00b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35374/0 (socket says 172.21.15.62:35374) 2024-03-23T11:29:44.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f5400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f29f006dca0 0x7f29f00cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53700/0 (socket says 172.21.15.62:53700) 2024-03-23T11:29:44.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f5e00640 1 -- 172.21.15.62:0/2380942499 learned_addr learned my addr 172.21.15.62:0/2380942499 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:44.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f5e00640 1 -- 172.21.15.62:0/2380942499 >> v2:172.21.15.120:3300/0 conn(0x7f29f006c300 msgr2=0x7f29f00cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:44.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f6800640 1 --2- 172.21.15.62:0/2380942499 >> v2:172.21.15.120:3300/0 conn(0x7f29f006c300 0x7f29f00cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f5e00640 1 --2- 172.21.15.62:0/2380942499 >> v2:172.21.15.120:3300/0 conn(0x7f29f006c300 0x7f29f00cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.805+0000 7f29f4a00640 1 -- 172.21.15.62:0/2380942499 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250908039 0 0) 0x7f29f00cad20 con 0x7f29f006dca0 2024-03-23T11:29:44.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.806+0000 7f29f5e00640 1 -- 172.21.15.62:0/2380942499 >> v1:172.21.15.62:6789/0 conn(0x7f29f006dca0 legacy=0x7f29f00cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:44.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.806+0000 7f29f5e00640 1 -- 172.21.15.62:0/2380942499 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f015ab80 con 0x7f29f00cd7c0 2024-03-23T11:29:44.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.806+0000 7f29f6800640 1 --2- 172.21.15.62:0/2380942499 >> v2:172.21.15.120:3300/0 conn(0x7f29f006c300 0x7f29f00cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:44.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.807+0000 7f29f5e00640 1 --2- 172.21.15.62:0/2380942499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f00cd7c0 0x7f29f00b15e0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f29f00b3a20 tx=0x7f29e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6508234c5e93876 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:44.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.807+0000 7f29f4a00640 1 -- 172.21.15.62:0/2380942499 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4031870 con 0x7f29f00cd7c0 2024-03-23T11:29:44.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.807+0000 7f29f4a00640 1 -- 172.21.15.62:0/2380942499 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e40319d0 con 0x7f29f00cd7c0 2024-03-23T11:29:44.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.807+0000 7f29f4a00640 1 -- 172.21.15.62:0/2380942499 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4031d60 con 0x7f29f00cd7c0 2024-03-23T11:29:44.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.808+0000 7f29f8800640 1 -- 172.21.15.62:0/2380942499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f00cd7c0 msgr2=0x7f29f00b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:44.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.808+0000 7f29f8800640 1 --2- 172.21.15.62:0/2380942499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f00cd7c0 0x7f29f00b15e0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f29f00b3a20 tx=0x7f29e402b910 comp rx=0 tx=0).stop 2024-03-23T11:29:44.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.808+0000 7f29f8800640 1 -- 172.21.15.62:0/2380942499 shutdown_connections 2024-03-23T11:29:44.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.808+0000 7f29f8800640 1 --2- 172.21.15.62:0/2380942499 >> v2:172.21.15.120:3300/0 conn(0x7f29f006c300 0x7f29f00cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.808+0000 7f29f8800640 1 --2- 172.21.15.62:0/2380942499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f00cd7c0 0x7f29f00b15e0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.808+0000 7f29f8800640 1 -- 172.21.15.62:0/2380942499 >> 172.21.15.62:0/2380942499 conn(0x7f29f00b02e0 msgr2=0x7f29f00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:44.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.808+0000 7f29f8800640 1 -- 172.21.15.62:0/2380942499 shutdown_connections 2024-03-23T11:29:44.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.808+0000 7f29f8800640 1 -- 172.21.15.62:0/2380942499 wait complete. 2024-03-23T11:29:44.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.810+0000 7f29f8800640 1 Processor -- start 2024-03-23T11:29:44.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.810+0000 7f29f8800640 1 -- start start 2024-03-23T11:29:44.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.811+0000 7f29f8800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f006c300 0x7f29f0150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.811+0000 7f29f8800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29f00cd7c0 0x7f29f0156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.811+0000 7f29f8800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0158600 0x7f29f01519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.811+0000 7f29f8800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f0150650 con 0x7f29f0158600 2024-03-23T11:29:44.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.811+0000 7f29f8800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f0158bc0 con 0x7f29f006c300 2024-03-23T11:29:44.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.811+0000 7f29f8800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f0158ea0 con 0x7f29f00cd7c0 2024-03-23T11:29:44.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.811+0000 7f29f6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0158600 0x7f29f01519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.811+0000 7f29f6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0158600 0x7f29f01519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35388/0 (socket says 172.21.15.62:35388) 2024-03-23T11:29:44.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.811+0000 7f29f6800640 1 -- 172.21.15.62:0/1231650458 learned_addr learned my addr 172.21.15.62:0/1231650458 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:44.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.812+0000 7f29f5e00640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f006c300 0x7f29f0150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.812+0000 7f29f5400640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29f00cd7c0 0x7f29f0156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.812+0000 7f29f6800640 1 -- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29f00cd7c0 msgr2=0x7f29f0156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:44.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.812+0000 7f29f6800640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29f00cd7c0 0x7f29f0156840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.812+0000 7f29f6800640 1 -- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f006c300 msgr2=0x7f29f0150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:44.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.812+0000 7f29f6800640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f006c300 0x7f29f0150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:44.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.812+0000 7f29f6800640 1 -- 172.21.15.62:0/1231650458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e40094d0 con 0x7f29f0158600 2024-03-23T11:29:44.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.812+0000 7f29f5400640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29f00cd7c0 0x7f29f0156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:44.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.812+0000 7f29f6800640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0158600 0x7f29f01519f0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f29ec00cef0 tx=0x7f29ec00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:44.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.813+0000 7f29ea000640 1 -- 172.21.15.62:0/1231650458 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec00d840 con 0x7f29f0158600 2024-03-23T11:29:44.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.813+0000 7f29ea000640 1 -- 172.21.15.62:0/1231650458 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec00d9c0 con 0x7f29f0158600 2024-03-23T11:29:44.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.813+0000 7f29ea000640 1 -- 172.21.15.62:0/1231650458 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec00dd70 con 0x7f29f0158600 2024-03-23T11:29:44.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.813+0000 7f29f8800640 1 -- 172.21.15.62:0/1231650458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f0159180 con 0x7f29f0158600 2024-03-23T11:29:44.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.813+0000 7f29f8800640 1 -- 172.21.15.62:0/1231650458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f0159740 con 0x7f29f0158600 2024-03-23T11:29:44.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.814+0000 7f29f8800640 1 -- 172.21.15.62:0/1231650458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f0066c90 con 0x7f29f0158600 2024-03-23T11:29:44.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.815+0000 7f29ea000640 1 -- 172.21.15.62:0/1231650458 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f29ec01f440 con 0x7f29f0158600 2024-03-23T11:29:44.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.816+0000 7f29ea000640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29d406a7a0 0x7f29d406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:44.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.816+0000 7f29ea000640 1 -- 172.21.15.62:0/1231650458 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29ec0898b0 con 0x7f29f0158600 2024-03-23T11:29:44.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.817+0000 7f29f5e00640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29d406a7a0 0x7f29d406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:44.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.818+0000 7f29f5e00640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29d406a7a0 0x7f29d406cc60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f29e4036000 tx=0x7f29e403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:44.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:44.821+0000 7f29ea000640 1 -- 172.21.15.62:0/1231650458 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f29ec054cf0 con 0x7f29f0158600 2024-03-23T11:29:45.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.023+0000 7f29f8800640 1 -- 172.21.15.62:0/1231650458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29f006c970 con 0x7f29f0158600 2024-03-23T11:29:45.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.024+0000 7f29ea000640 1 -- 172.21.15.62:0/1231650458 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29ec01a020 con 0x7f29f0158600 2024-03-23T11:29:45.016 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:29:45.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 -- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29d406a7a0 msgr2=0x7f29d406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29d406a7a0 0x7f29d406cc60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f29e4036000 tx=0x7f29e403c040 comp rx=0 tx=0).stop 2024-03-23T11:29:45.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 -- 172.21.15.62:0/1231650458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0158600 msgr2=0x7f29f01519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0158600 0x7f29f01519f0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f29ec00cef0 tx=0x7f29ec00a710 comp rx=0 tx=0).stop 2024-03-23T11:29:45.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 -- 172.21.15.62:0/1231650458 shutdown_connections 2024-03-23T11:29:45.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f29d406a7a0 0x7f29d406cc60 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29f0158600 0x7f29f01519f0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29f00cd7c0 0x7f29f0156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 --2- 172.21.15.62:0/1231650458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29f006c300 0x7f29f0150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 -- 172.21.15.62:0/1231650458 >> 172.21.15.62:0/1231650458 conn(0x7f29f00b02e0 msgr2=0x7f29f015b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:45.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 -- 172.21.15.62:0/1231650458 shutdown_connections 2024-03-23T11:29:45.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.027+0000 7f29f8800640 1 -- 172.21.15.62:0/1231650458 wait complete. 2024-03-23T11:29:45.037 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:29:45.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.272+0000 7f7a1ba00640 1 Processor -- start 2024-03-23T11:29:45.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.272+0000 7f7a1ba00640 1 -- start start 2024-03-23T11:29:45.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.274+0000 7f7a1ba00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a140cafa0 0x7f7a140cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:45.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.274+0000 7f7a1ba00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7a1406c6e0 0x7f7a1406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:45.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.274+0000 7f7a1ba00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a140cbc60 con 0x7f7a1406c300 2024-03-23T11:29:45.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.274+0000 7f7a1ba00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a140cd410 con 0x7f7a1406c6e0 2024-03-23T11:29:45.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.274+0000 7f7a1ba00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140cbf40 con 0x7f7a140cafa0 2024-03-23T11:29:45.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.274+0000 7f7a19a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7a1406c6e0 0x7f7a1406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:45.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.275+0000 7f7a19a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7a1406c6e0 0x7f7a1406cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35402/0 (socket says 172.21.15.62:35402) 2024-03-23T11:29:45.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.275+0000 7f7a19a00640 1 -- 172.21.15.62:0/3972631325 learned_addr learned my addr 172.21.15.62:0/3972631325 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:45.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.275+0000 7f7a13e00640 1 --2- 172.21.15.62:0/3972631325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a140cafa0 0x7f7a140cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:45.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.275+0000 7f7a19a00640 1 -- 172.21.15.62:0/3972631325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a140cafa0 msgr2=0x7f7a140cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:45.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.275+0000 7f7a19a00640 1 --2- 172.21.15.62:0/3972631325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a140cafa0 0x7f7a140cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.275+0000 7f7a19a00640 1 -- 172.21.15.62:0/3972631325 >> v1:172.21.15.120:6789/0 conn(0x7f7a1406c300 legacy=0x7f7a140ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.275+0000 7f7a19a00640 1 -- 172.21.15.62:0/3972631325 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a140cc220 con 0x7f7a1406c6e0 2024-03-23T11:29:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.277+0000 7f7a19a00640 1 --2- 172.21.15.62:0/3972631325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a1406c6e0 0x7f7a1406cb70 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009e50 tx=0x7f7a0c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46abd2585a4f2486 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.277+0000 7f7a13400640 1 -- 172.21.15.62:0/3972631325 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c02cb00 con 0x7f7a1406c6e0 2024-03-23T11:29:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.277+0000 7f7a13400640 1 -- 172.21.15.62:0/3972631325 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c02cc60 con 0x7f7a1406c6e0 2024-03-23T11:29:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.277+0000 7f7a13400640 1 -- 172.21.15.62:0/3972631325 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0343f0 con 0x7f7a1406c6e0 2024-03-23T11:29:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.278+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3972631325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a1406c6e0 msgr2=0x7f7a1406cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.278+0000 7f7a1ba00640 1 --2- 172.21.15.62:0/3972631325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a1406c6e0 0x7f7a1406cb70 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009e50 tx=0x7f7a0c02bf60 comp rx=0 tx=0).stop 2024-03-23T11:29:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.278+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3972631325 shutdown_connections 2024-03-23T11:29:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.278+0000 7f7a1ba00640 1 --2- 172.21.15.62:0/3972631325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a1406c6e0 0x7f7a1406cb70 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.278+0000 7f7a1ba00640 1 --2- 172.21.15.62:0/3972631325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a140cafa0 0x7f7a140cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.278+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3972631325 >> 172.21.15.62:0/3972631325 conn(0x7f7a140b02e0 msgr2=0x7f7a140b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.278+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3972631325 shutdown_connections 2024-03-23T11:29:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.278+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3972631325 wait complete. 2024-03-23T11:29:45.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.280+0000 7f7a1ba00640 1 Processor -- start 2024-03-23T11:29:45.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.280+0000 7f7a1ba00640 1 -- start start 2024-03-23T11:29:45.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.281+0000 7f7a1ba00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1406c300 0x7f7a141efae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:45.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.281+0000 7f7a1ba00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a1406c6e0 0x7f7a14140580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:45.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.281+0000 7f7a1ba00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140cafa0 0x7f7a14158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:45.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.281+0000 7f7a1ba00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1415c2c0 con 0x7f7a140cafa0 2024-03-23T11:29:45.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.281+0000 7f7a1ba00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1415da30 con 0x7f7a1406c300 2024-03-23T11:29:45.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.281+0000 7f7a1ba00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1415dd10 con 0x7f7a1406c6e0 2024-03-23T11:29:45.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.281+0000 7f7a19a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140cafa0 0x7f7a14158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:45.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.281+0000 7f7a19a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140cafa0 0x7f7a14158a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35412/0 (socket says 172.21.15.62:35412) 2024-03-23T11:29:45.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.282+0000 7f7a19a00640 1 -- 172.21.15.62:0/3672695787 learned_addr learned my addr 172.21.15.62:0/3672695787 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:45.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.282+0000 7f7a19000640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1406c300 0x7f7a141efae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:45.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.282+0000 7f7a13e00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a1406c6e0 0x7f7a14140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:45.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.282+0000 7f7a19a00640 1 -- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a1406c6e0 msgr2=0x7f7a14140580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.282+0000 7f7a19a00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a1406c6e0 0x7f7a14140580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.282+0000 7f7a19a00640 1 -- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1406c300 msgr2=0x7f7a141efae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.282+0000 7f7a19a00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1406c300 0x7f7a141efae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.282+0000 7f7a19a00640 1 -- 172.21.15.62:0/3672695787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c009b30 con 0x7f7a140cafa0 2024-03-23T11:29:45.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.282+0000 7f7a19000640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1406c300 0x7f7a141efae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:29:45.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.282+0000 7f7a13e00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a1406c6e0 0x7f7a14140580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:29:45.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.283+0000 7f7a19a00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140cafa0 0x7f7a14158a60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009e50 tx=0x7f7a0c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:45.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.283+0000 7f7a10c00640 1 -- 172.21.15.62:0/3672695787 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c009510 con 0x7f7a140cafa0 2024-03-23T11:29:45.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.283+0000 7f7a10c00640 1 -- 172.21.15.62:0/3672695787 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c03e450 con 0x7f7a140cafa0 2024-03-23T11:29:45.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.283+0000 7f7a10c00640 1 -- 172.21.15.62:0/3672695787 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c03e800 con 0x7f7a140cafa0 2024-03-23T11:29:45.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.283+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3672695787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1415dff0 con 0x7f7a140cafa0 2024-03-23T11:29:45.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.283+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3672695787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a14140ca0 con 0x7f7a140cafa0 2024-03-23T11:29:45.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.284+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3672695787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1415c400 con 0x7f7a140cafa0 2024-03-23T11:29:45.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.285+0000 7f7a10c00640 1 -- 172.21.15.62:0/3672695787 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7a0c03e960 con 0x7f7a140cafa0 2024-03-23T11:29:45.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.286+0000 7f7a10c00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f79f406a7a0 0x7f79f406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:45.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.286+0000 7f7a10c00640 1 -- 172.21.15.62:0/3672695787 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a0c0ab910 con 0x7f7a140cafa0 2024-03-23T11:29:45.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.287+0000 7f7a19000640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f79f406a7a0 0x7f79f406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:45.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.288+0000 7f7a19000640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f79f406a7a0 0x7f79f406cc60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7a08007fd0 tx=0x7f7a08009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.290+0000 7f7a10c00640 1 -- 172.21.15.62:0/3672695787 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7a1415c400 con 0x7f7a140cafa0 2024-03-23T11:29:45.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.493+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3672695787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7a1415c400 con 0x7f7a140cafa0 2024-03-23T11:29:45.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.494+0000 7f7a10c00640 1 -- 172.21.15.62:0/3672695787 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7a1415c400 con 0x7f7a140cafa0 2024-03-23T11:29:45.487 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:29:45.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.497+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f79f406a7a0 msgr2=0x7f79f406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.497+0000 7f7a1ba00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f79f406a7a0 0x7f79f406cc60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7a08007fd0 tx=0x7f7a08009040 comp rx=0 tx=0).stop 2024-03-23T11:29:45.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.497+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3672695787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140cafa0 msgr2=0x7f7a14158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.497+0000 7f7a1ba00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140cafa0 0x7f7a14158a60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009e50 tx=0x7f7a0c004a40 comp rx=0 tx=0).stop 2024-03-23T11:29:45.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.497+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3672695787 shutdown_connections 2024-03-23T11:29:45.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.497+0000 7f7a1ba00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f79f406a7a0 0x7f79f406cc60 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.497+0000 7f7a1ba00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140cafa0 0x7f7a14158a60 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.497+0000 7f7a1ba00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a1406c6e0 0x7f7a14140580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.497+0000 7f7a1ba00640 1 --2- 172.21.15.62:0/3672695787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a1406c300 0x7f7a141efae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.497+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3672695787 >> 172.21.15.62:0/3672695787 conn(0x7f7a140b02e0 msgr2=0x7f7a14072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:45.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.497+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3672695787 shutdown_connections 2024-03-23T11:29:45.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.498+0000 7f7a1ba00640 1 -- 172.21.15.62:0/3672695787 wait complete. 2024-03-23T11:29:45.508 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:29:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.737+0000 7f5d12600640 1 Processor -- start 2024-03-23T11:29:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.738+0000 7f5d12600640 1 -- start start 2024-03-23T11:29:45.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.739+0000 7f5d12600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d0c0b2f30 0x7f5d0c070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:45.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.739+0000 7f5d12600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c0cd140 con 0x7f5d0c143dd0 2024-03-23T11:29:45.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.739+0000 7f5d12600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c0cd3e0 con 0x7f5d0c070090 2024-03-23T11:29:45.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.739+0000 7f5d12600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0c070a60 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.739+0000 7f5d11c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d0c0b2f30 0x7f5d0c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.739+0000 7f5d11200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5d0c143dd0 0x7f5d0c15abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53706/0 (socket says 172.21.15.62:53706) 2024-03-23T11:29:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.740+0000 7f5d11200640 1 -- 172.21.15.62:0/3316813687 learned_addr learned my addr 172.21.15.62:0/3316813687 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.740+0000 7f5d11c00640 1 -- 172.21.15.62:0/3316813687 >> v1:172.21.15.120:6789/0 conn(0x7f5d0c070090 legacy=0x7f5d0c0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.740+0000 7f5d11c00640 1 -- 172.21.15.62:0/3316813687 >> v1:172.21.15.62:6789/0 conn(0x7f5d0c143dd0 legacy=0x7f5d0c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.740+0000 7f5d11c00640 1 -- 172.21.15.62:0/3316813687 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c0cd6c0 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.741+0000 7f5d11c00640 1 --2- 172.21.15.62:0/3316813687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d0c0b2f30 0x7f5d0c070520 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f5d04009d90 tx=0x7f5d0402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99566060864ba639 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.742+0000 7f5d0b400640 1 -- 172.21.15.62:0/3316813687 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0402cc60 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.742+0000 7f5d0b400640 1 -- 172.21.15.62:0/3316813687 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0402cdc0 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.742+0000 7f5d0b400640 1 -- 172.21.15.62:0/3316813687 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d04034660 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.742+0000 7f5d12600640 1 -- 172.21.15.62:0/3316813687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d0c0b2f30 msgr2=0x7f5d0c070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.742+0000 7f5d12600640 1 --2- 172.21.15.62:0/3316813687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d0c0b2f30 0x7f5d0c070520 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f5d04009d90 tx=0x7f5d0402beb0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.742+0000 7f5d12600640 1 -- 172.21.15.62:0/3316813687 shutdown_connections 2024-03-23T11:29:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.742+0000 7f5d12600640 1 --2- 172.21.15.62:0/3316813687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d0c0b2f30 0x7f5d0c070520 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.742+0000 7f5d12600640 1 -- 172.21.15.62:0/3316813687 >> 172.21.15.62:0/3316813687 conn(0x7f5d0c066050 msgr2=0x7f5d0c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.742+0000 7f5d12600640 1 -- 172.21.15.62:0/3316813687 shutdown_connections 2024-03-23T11:29:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.742+0000 7f5d12600640 1 -- 172.21.15.62:0/3316813687 wait complete. 2024-03-23T11:29:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.744+0000 7f5d12600640 1 Processor -- start 2024-03-23T11:29:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.744+0000 7f5d12600640 1 -- start start 2024-03-23T11:29:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.745+0000 7f5d12600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d0c070090 0x7f5d0c134960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.745+0000 7f5d12600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d0c0b2f30 0x7f5d0c0c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.745+0000 7f5d12600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d0c143dd0 0x7f5d0c159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.745+0000 7f5d12600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c134ea0 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.745+0000 7f5d12600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c135140 con 0x7f5d0c143dd0 2024-03-23T11:29:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.745+0000 7f5d12600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c135420 con 0x7f5d0c070090 2024-03-23T11:29:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.746+0000 7f5d0be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d0c0b2f30 0x7f5d0c0c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.746+0000 7f5d0be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d0c0b2f30 0x7f5d0c0c7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35424/0 (socket says 172.21.15.62:35424) 2024-03-23T11:29:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.746+0000 7f5d0be00640 1 -- 172.21.15.62:0/2919966011 learned_addr learned my addr 172.21.15.62:0/2919966011 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.746+0000 7f5d11c00640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d0c143dd0 0x7f5d0c159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.746+0000 7f5d11200640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d0c070090 0x7f5d0c134960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.746+0000 7f5d0be00640 1 -- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d0c070090 msgr2=0x7f5d0c134960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.746+0000 7f5d0be00640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d0c070090 0x7f5d0c134960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.746+0000 7f5d0be00640 1 -- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d0c143dd0 msgr2=0x7f5d0c159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.746+0000 7f5d0be00640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d0c143dd0 0x7f5d0c159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.746+0000 7f5d0be00640 1 -- 172.21.15.62:0/2919966011 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d04009a70 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.746+0000 7f5d11c00640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d0c143dd0 0x7f5d0c159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:29:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.747+0000 7f5d0be00640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d0c0b2f30 0x7f5d0c0c7690 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c430 tx=0x7f5cfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.747+0000 7f5d08c00640 1 -- 172.21.15.62:0/2919966011 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc011790 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.747+0000 7f5d08c00640 1 -- 172.21.15.62:0/2919966011 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc0118f0 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.747+0000 7f5d08c00640 1 -- 172.21.15.62:0/2919966011 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc011ce0 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.747+0000 7f5d12600640 1 -- 172.21.15.62:0/2919966011 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0c135700 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.747+0000 7f5d12600640 1 -- 172.21.15.62:0/2919966011 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0c135cc0 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.748+0000 7f5d12600640 1 -- 172.21.15.62:0/2919966011 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d0c066bb0 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.749+0000 7f5d08c00640 1 -- 172.21.15.62:0/2919966011 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5cfc01c460 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.750+0000 7f5d08c00640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5cf006a7a0 0x7f5cf006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.750+0000 7f5d08c00640 1 -- 172.21.15.62:0/2919966011 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cfc088d40 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.751+0000 7f5d11200640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5cf006a7a0 0x7f5cf006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:45.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.752+0000 7f5d11200640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5cf006a7a0 0x7f5cf006cc60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f5d00000f60 tx=0x7f5d00009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.755+0000 7f5d08c00640 1 -- 172.21.15.62:0/2919966011 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5cfc054150 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.953+0000 7f5d12600640 1 -- 172.21.15.62:0/2919966011 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5d0c135e50 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.953+0000 7f5d08c00640 1 -- 172.21.15.62:0/2919966011 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5cfc057770 con 0x7f5d0c0b2f30 2024-03-23T11:29:45.946 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:29:45.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.956+0000 7f5d12600640 1 -- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5cf006a7a0 msgr2=0x7f5cf006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.956+0000 7f5d12600640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5cf006a7a0 0x7f5cf006cc60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f5d00000f60 tx=0x7f5d00009040 comp rx=0 tx=0).stop 2024-03-23T11:29:45.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.956+0000 7f5d12600640 1 -- 172.21.15.62:0/2919966011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d0c0b2f30 msgr2=0x7f5d0c0c7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:45.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.956+0000 7f5d12600640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d0c0b2f30 0x7f5d0c0c7690 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c430 tx=0x7f5cfc00c8f0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.956+0000 7f5d12600640 1 -- 172.21.15.62:0/2919966011 shutdown_connections 2024-03-23T11:29:45.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.956+0000 7f5d12600640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5cf006a7a0 0x7f5cf006cc60 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.956+0000 7f5d12600640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d0c0b2f30 0x7f5d0c0c7690 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.956+0000 7f5d12600640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d0c143dd0 0x7f5d0c159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.956+0000 7f5d12600640 1 --2- 172.21.15.62:0/2919966011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5d0c070090 0x7f5d0c134960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:45.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.956+0000 7f5d12600640 1 -- 172.21.15.62:0/2919966011 >> 172.21.15.62:0/2919966011 conn(0x7f5d0c066050 msgr2=0x7f5d0c06d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:45.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.956+0000 7f5d12600640 1 -- 172.21.15.62:0/2919966011 shutdown_connections 2024-03-23T11:29:45.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:45.957+0000 7f5d12600640 1 -- 172.21.15.62:0/2919966011 wait complete. 2024-03-23T11:29:45.969 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:29:46.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.201+0000 7f8196800640 1 Processor -- start 2024-03-23T11:29:46.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.201+0000 7f8196800640 1 -- start start 2024-03-23T11:29:46.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.203+0000 7f8196800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f81900ad370 0x7f81900b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:46.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.203+0000 7f8196800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81900abe50 con 0x7f819015c730 2024-03-23T11:29:46.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.203+0000 7f8196800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819006cba0 con 0x7f8190069c60 2024-03-23T11:29:46.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.203+0000 7f8196800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f819006ce80 con 0x7f81900ad370 2024-03-23T11:29:46.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.203+0000 7f8195400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f819015c730 0x7f8190160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53714/0 (socket says 172.21.15.62:53714) 2024-03-23T11:29:46.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.203+0000 7f8195400640 1 -- 172.21.15.62:0/2904488651 learned_addr learned my addr 172.21.15.62:0/2904488651 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:46.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.203+0000 7f8194a00640 1 --2- 172.21.15.62:0/2904488651 >> v2:172.21.15.62:3300/0 conn(0x7f81900ad370 0x7f81900b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:46.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.204+0000 7f818fe00640 1 -- 172.21.15.62:0/2904488651 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061226643 0 0) 0x7f81900abe50 con 0x7f819015c730 2024-03-23T11:29:46.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.204+0000 7f818fe00640 1 -- 172.21.15.62:0/2904488651 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8178003520 con 0x7f819015c730 2024-03-23T11:29:46.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.204+0000 7f8194a00640 1 -- 172.21.15.62:0/2904488651 >> v1:172.21.15.120:6789/0 conn(0x7f8190069c60 legacy=0x7f81900ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:46.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.204+0000 7f8194a00640 1 -- 172.21.15.62:0/2904488651 >> v1:172.21.15.62:6789/0 conn(0x7f819015c730 legacy=0x7f8190160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.204+0000 7f8194a00640 1 -- 172.21.15.62:0/2904488651 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819006d160 con 0x7f81900ad370 2024-03-23T11:29:46.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.205+0000 7f8194a00640 1 --2- 172.21.15.62:0/2904488651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81900ad370 0x7f81900b1ab0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f817c0097a0 tx=0x7f817c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c4f0dfaa33c93c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:46.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.205+0000 7f818fe00640 1 -- 172.21.15.62:0/2904488651 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c030610 con 0x7f81900ad370 2024-03-23T11:29:46.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.205+0000 7f818fe00640 1 -- 172.21.15.62:0/2904488651 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c030770 con 0x7f81900ad370 2024-03-23T11:29:46.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.205+0000 7f818fe00640 1 -- 172.21.15.62:0/2904488651 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c030b00 con 0x7f81900ad370 2024-03-23T11:29:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.206+0000 7f8196800640 1 -- 172.21.15.62:0/2904488651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81900ad370 msgr2=0x7f81900b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.206+0000 7f8196800640 1 --2- 172.21.15.62:0/2904488651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81900ad370 0x7f81900b1ab0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f817c0097a0 tx=0x7f817c02b960 comp rx=0 tx=0).stop 2024-03-23T11:29:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.206+0000 7f8196800640 1 -- 172.21.15.62:0/2904488651 shutdown_connections 2024-03-23T11:29:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.206+0000 7f8196800640 1 --2- 172.21.15.62:0/2904488651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81900ad370 0x7f81900b1ab0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.206+0000 7f8196800640 1 -- 172.21.15.62:0/2904488651 >> 172.21.15.62:0/2904488651 conn(0x7f81900b0090 msgr2=0x7f81900b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.206+0000 7f8196800640 1 -- 172.21.15.62:0/2904488651 shutdown_connections 2024-03-23T11:29:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.206+0000 7f8196800640 1 -- 172.21.15.62:0/2904488651 wait complete. 2024-03-23T11:29:46.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.208+0000 7f8196800640 1 Processor -- start 2024-03-23T11:29:46.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.208+0000 7f8196800640 1 -- start start 2024-03-23T11:29:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.209+0000 7f8196800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8190069c60 0x7f81901551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.209+0000 7f8196800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81900ad370 0x7f8190159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.209+0000 7f8196800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f819015c730 0x7f819015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.209+0000 7f8196800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8190159ad0 con 0x7f8190069c60 2024-03-23T11:29:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.209+0000 7f8196800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819015a440 con 0x7f81900ad370 2024-03-23T11:29:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.209+0000 7f8196800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819015a720 con 0x7f819015c730 2024-03-23T11:29:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.209+0000 7f8195400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8190069c60 0x7f81901551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:46.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.209+0000 7f8195400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8190069c60 0x7f81901551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35450/0 (socket says 172.21.15.62:35450) 2024-03-23T11:29:46.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.210+0000 7f8195400640 1 -- 172.21.15.62:0/799288926 learned_addr learned my addr 172.21.15.62:0/799288926 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:46.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.210+0000 7f8194a00640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81900ad370 0x7f8190159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:46.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.210+0000 7f8195e00640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f819015c730 0x7f819015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:46.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.210+0000 7f8195400640 1 -- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f819015c730 msgr2=0x7f819015dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.210+0000 7f8195400640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f819015c730 0x7f819015dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.210+0000 7f8195400640 1 -- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81900ad370 msgr2=0x7f8190159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.210+0000 7f8195400640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81900ad370 0x7f8190159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.210+0000 7f8195400640 1 -- 172.21.15.62:0/799288926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c009400 con 0x7f8190069c60 2024-03-23T11:29:46.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.210+0000 7f8194a00640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81900ad370 0x7f8190159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:46.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.210+0000 7f8195e00640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f819015c730 0x7f819015dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:46.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.210+0000 7f8195400640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8190069c60 0x7f81901551e0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f818400c410 tx=0x7f818400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:46.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.211+0000 7f818d600640 1 -- 172.21.15.62:0/799288926 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184005ce0 con 0x7f8190069c60 2024-03-23T11:29:46.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.211+0000 7f818d600640 1 -- 172.21.15.62:0/799288926 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8184005e40 con 0x7f8190069c60 2024-03-23T11:29:46.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.211+0000 7f818d600640 1 -- 172.21.15.62:0/799288926 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81840072f0 con 0x7f8190069c60 2024-03-23T11:29:46.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.211+0000 7f8196800640 1 -- 172.21.15.62:0/799288926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819015aa00 con 0x7f8190069c60 2024-03-23T11:29:46.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.211+0000 7f8196800640 1 -- 172.21.15.62:0/799288926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819015afc0 con 0x7f8190069c60 2024-03-23T11:29:46.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.212+0000 7f8196800640 1 -- 172.21.15.62:0/799288926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8190066bb0 con 0x7f8190069c60 2024-03-23T11:29:46.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.213+0000 7f818d600640 1 -- 172.21.15.62:0/799288926 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8184007510 con 0x7f8190069c60 2024-03-23T11:29:46.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.214+0000 7f818d600640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817006a7a0 0x7f817006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:46.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.214+0000 7f818d600640 1 -- 172.21.15.62:0/799288926 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8184088410 con 0x7f8190069c60 2024-03-23T11:29:46.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.214+0000 7f8194a00640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817006a7a0 0x7f817006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:46.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.215+0000 7f8194a00640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817006a7a0 0x7f817006cc60 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f817c036000 tx=0x7f817c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:46.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.218+0000 7f818d600640 1 -- 172.21.15.62:0/799288926 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f81840538e0 con 0x7f8190069c60 2024-03-23T11:29:46.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.434+0000 7f8196800640 1 -- 172.21.15.62:0/799288926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f819015bf50 con 0x7f8190069c60 2024-03-23T11:29:46.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.435+0000 7f818d600640 1 -- 172.21.15.62:0/799288926 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v105) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8184056f00 con 0x7f8190069c60 2024-03-23T11:29:46.427 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:29:46.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.438+0000 7f8196800640 1 -- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817006a7a0 msgr2=0x7f817006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.438+0000 7f8196800640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817006a7a0 0x7f817006cc60 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f817c036000 tx=0x7f817c03b040 comp rx=0 tx=0).stop 2024-03-23T11:29:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.438+0000 7f8196800640 1 -- 172.21.15.62:0/799288926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8190069c60 msgr2=0x7f81901551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.438+0000 7f8196800640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8190069c60 0x7f81901551e0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f818400c410 tx=0x7f818400c8d0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.438+0000 7f8196800640 1 -- 172.21.15.62:0/799288926 shutdown_connections 2024-03-23T11:29:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.438+0000 7f8196800640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f817006a7a0 0x7f817006cc60 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.438+0000 7f8196800640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f819015c730 0x7f819015dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.438+0000 7f8196800640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81900ad370 0x7f8190159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.438+0000 7f8196800640 1 --2- 172.21.15.62:0/799288926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8190069c60 0x7f81901551e0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.438+0000 7f8196800640 1 -- 172.21.15.62:0/799288926 >> 172.21.15.62:0/799288926 conn(0x7f81900b0090 msgr2=0x7f81900709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.439+0000 7f8196800640 1 -- 172.21.15.62:0/799288926 shutdown_connections 2024-03-23T11:29:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.439+0000 7f8196800640 1 -- 172.21.15.62:0/799288926 wait complete. 2024-03-23T11:29:46.449 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:29:46.449 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:46.500 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:29:46.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.722+0000 7f7ddb200640 1 Processor -- start 2024-03-23T11:29:46.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.722+0000 7f7ddb200640 1 -- start start 2024-03-23T11:29:46.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.723+0000 7f7ddb200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7ddc143dd0 0x7f7ddc15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:46.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.723+0000 7f7ddb200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7ddc070090 0x7f7ddc070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:46.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.723+0000 7f7ddb200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ddc0cd140 con 0x7f7ddc0b2820 2024-03-23T11:29:46.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.723+0000 7f7ddb200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ddc070a60 con 0x7f7ddc070090 2024-03-23T11:29:46.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.723+0000 7f7ddb200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ddc0cd3e0 con 0x7f7ddc143dd0 2024-03-23T11:29:46.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.724+0000 7f7dd9400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7ddc070090 0x7f7ddc070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:46.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.724+0000 7f7dd9400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7ddc070090 0x7f7ddc070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35464/0 (socket says 172.21.15.62:35464) 2024-03-23T11:29:46.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.724+0000 7f7dd9400640 1 -- 172.21.15.62:0/2655272932 learned_addr learned my addr 172.21.15.62:0/2655272932 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:46.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.724+0000 7f7dda800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7ddc0b2820 0x7f7ddc0b2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53726/0 (socket says 172.21.15.62:53726) 2024-03-23T11:29:46.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.724+0000 7f7dd9e00640 1 --2- 172.21.15.62:0/2655272932 >> v2:172.21.15.120:3300/0 conn(0x7f7ddc143dd0 0x7f7ddc15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:46.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.724+0000 7f7dd9400640 1 -- 172.21.15.62:0/2655272932 >> v2:172.21.15.120:3300/0 conn(0x7f7ddc143dd0 msgr2=0x7f7ddc15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:46.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.724+0000 7f7dd9400640 1 --2- 172.21.15.62:0/2655272932 >> v2:172.21.15.120:3300/0 conn(0x7f7ddc143dd0 0x7f7ddc15abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.724+0000 7f7dd9400640 1 -- 172.21.15.62:0/2655272932 >> v1:172.21.15.62:6789/0 conn(0x7f7ddc0b2820 legacy=0x7f7ddc0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.724+0000 7f7dd9400640 1 -- 172.21.15.62:0/2655272932 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ddc0cd6c0 con 0x7f7ddc070090 2024-03-23T11:29:46.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.724+0000 7f7dd8a00640 1 -- 172.21.15.62:0/2655272932 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518338019 0 0) 0x7f7ddc0cd140 con 0x7f7ddc0b2820 2024-03-23T11:29:46.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.725+0000 7f7dd9400640 1 --2- 172.21.15.62:0/2655272932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ddc070090 0x7f7ddc070520 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc06e380 tx=0x7f7dc802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1046bda9e781237d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:46.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.726+0000 7f7dd8a00640 1 -- 172.21.15.62:0/2655272932 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8030860 con 0x7f7ddc070090 2024-03-23T11:29:46.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.726+0000 7f7dd8a00640 1 -- 172.21.15.62:0/2655272932 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc80309c0 con 0x7f7ddc070090 2024-03-23T11:29:46.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.726+0000 7f7dd8a00640 1 -- 172.21.15.62:0/2655272932 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8030d50 con 0x7f7ddc070090 2024-03-23T11:29:46.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.726+0000 7f7ddb200640 1 -- 172.21.15.62:0/2655272932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ddc070090 msgr2=0x7f7ddc070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.726+0000 7f7ddb200640 1 --2- 172.21.15.62:0/2655272932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ddc070090 0x7f7ddc070520 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc06e380 tx=0x7f7dc802b9a0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.726+0000 7f7ddb200640 1 -- 172.21.15.62:0/2655272932 shutdown_connections 2024-03-23T11:29:46.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.726+0000 7f7ddb200640 1 --2- 172.21.15.62:0/2655272932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ddc070090 0x7f7ddc070520 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.726+0000 7f7ddb200640 1 --2- 172.21.15.62:0/2655272932 >> v2:172.21.15.120:3300/0 conn(0x7f7ddc143dd0 0x7f7ddc15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.726+0000 7f7ddb200640 1 -- 172.21.15.62:0/2655272932 >> 172.21.15.62:0/2655272932 conn(0x7f7ddc066050 msgr2=0x7f7ddc069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:46.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.727+0000 7f7ddb200640 1 -- 172.21.15.62:0/2655272932 shutdown_connections 2024-03-23T11:29:46.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.727+0000 7f7ddb200640 1 -- 172.21.15.62:0/2655272932 wait complete. 2024-03-23T11:29:46.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.728+0000 7f7ddb200640 1 Processor -- start 2024-03-23T11:29:46.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.728+0000 7f7ddb200640 1 -- start start 2024-03-23T11:29:46.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.729+0000 7f7ddb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ddc070090 0x7f7ddc159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:46.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.729+0000 7f7ddb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ddc0b2820 0x7f7ddc06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:46.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.729+0000 7f7ddb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ddc143dd0 0x7f7ddc0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:46.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.729+0000 7f7ddb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc0ab840 con 0x7f7ddc143dd0 2024-03-23T11:29:46.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.729+0000 7f7ddb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc0abb20 con 0x7f7ddc070090 2024-03-23T11:29:46.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.729+0000 7f7ddb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc0abe00 con 0x7f7ddc0b2820 2024-03-23T11:29:46.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.730+0000 7f7dda800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ddc143dd0 0x7f7ddc0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:46.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.730+0000 7f7dda800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ddc143dd0 0x7f7ddc0ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35472/0 (socket says 172.21.15.62:35472) 2024-03-23T11:29:46.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.730+0000 7f7dda800640 1 -- 172.21.15.62:0/1548627873 learned_addr learned my addr 172.21.15.62:0/1548627873 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:46.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.730+0000 7f7dd9e00640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ddc070090 0x7f7ddc159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:46.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.730+0000 7f7dd9400640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ddc0b2820 0x7f7ddc06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:46.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.730+0000 7f7dda800640 1 -- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ddc0b2820 msgr2=0x7f7ddc06d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.730+0000 7f7dda800640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ddc0b2820 0x7f7ddc06d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.730+0000 7f7dda800640 1 -- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ddc070090 msgr2=0x7f7ddc159b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.730+0000 7f7dda800640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ddc070090 0x7f7ddc159b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.730+0000 7f7dda800640 1 -- 172.21.15.62:0/1548627873 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc80094d0 con 0x7f7ddc143dd0 2024-03-23T11:29:46.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.731+0000 7f7dda800640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ddc143dd0 0x7f7ddc0ac8e0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400c7e0 tx=0x7f7dd400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:46.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.731+0000 7f7dd2000640 1 -- 172.21.15.62:0/1548627873 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd400a640 con 0x7f7ddc143dd0 2024-03-23T11:29:46.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.731+0000 7f7dd2000640 1 -- 172.21.15.62:0/1548627873 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd400a7c0 con 0x7f7ddc143dd0 2024-03-23T11:29:46.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.731+0000 7f7dd2000640 1 -- 172.21.15.62:0/1548627873 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd400ab70 con 0x7f7ddc143dd0 2024-03-23T11:29:46.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.731+0000 7f7ddb200640 1 -- 172.21.15.62:0/1548627873 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ddc0ac0e0 con 0x7f7ddc143dd0 2024-03-23T11:29:46.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.731+0000 7f7ddb200640 1 -- 172.21.15.62:0/1548627873 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ddc06cae0 con 0x7f7ddc143dd0 2024-03-23T11:29:46.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.732+0000 7f7ddb200640 1 -- 172.21.15.62:0/1548627873 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ddc0ac600 con 0x7f7ddc143dd0 2024-03-23T11:29:46.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.733+0000 7f7dd2000640 1 -- 172.21.15.62:0/1548627873 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7dd400acd0 con 0x7f7ddc143dd0 2024-03-23T11:29:46.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.734+0000 7f7dd2000640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7dbc06a7a0 0x7f7dbc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:46.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.734+0000 7f7dd2000640 1 -- 172.21.15.62:0/1548627873 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7dd4088390 con 0x7f7ddc143dd0 2024-03-23T11:29:46.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.735+0000 7f7dd9e00640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7dbc06a7a0 0x7f7dbc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:46.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.736+0000 7f7dd9e00640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7dbc06a7a0 0x7f7dbc06cc60 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc006fd0 tx=0x7f7dcc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:46.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.739+0000 7f7dd2000640 1 -- 172.21.15.62:0/1548627873 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7ddc0ac600 con 0x7f7ddc143dd0 2024-03-23T11:29:46.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.939+0000 7f7ddb200640 1 -- 172.21.15.62:0/1548627873 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ddc0ac600 con 0x7f7ddc143dd0 2024-03-23T11:29:46.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.942+0000 7f7dd2000640 1 -- 172.21.15.62:0/1548627873 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16051 (secure 0 0 0) 0x7f7ddc0ac600 con 0x7f7ddc143dd0 2024-03-23T11:29:46.934 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:46.935 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":105,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:29:16.631279+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:29:13.817605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-23T11:29:14.652617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:29:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 -- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7dbc06a7a0 msgr2=0x7f7dbc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7dbc06a7a0 0x7f7dbc06cc60 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc006fd0 tx=0x7f7dcc008040 comp rx=0 tx=0).stop 2024-03-23T11:29:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 -- 172.21.15.62:0/1548627873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ddc143dd0 msgr2=0x7f7ddc0ac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ddc143dd0 0x7f7ddc0ac8e0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400c7e0 tx=0x7f7dd400cca0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 -- 172.21.15.62:0/1548627873 shutdown_connections 2024-03-23T11:29:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7dbc06a7a0 0x7f7dbc06cc60 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ddc143dd0 0x7f7ddc0ac8e0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ddc0b2820 0x7f7ddc06d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 --2- 172.21.15.62:0/1548627873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ddc070090 0x7f7ddc159b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 -- 172.21.15.62:0/1548627873 >> 172.21.15.62:0/1548627873 conn(0x7f7ddc066050 msgr2=0x7f7ddc15d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 -- 172.21.15.62:0/1548627873 shutdown_connections 2024-03-23T11:29:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:46.946+0000 7f7ddb200640 1 -- 172.21.15.62:0/1548627873 wait complete. 2024-03-23T11:29:46.956 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:29:13.817605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:29:14.652617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:29:46.957 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:29:47.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.194+0000 7f8f4d400640 1 Processor -- start 2024-03-23T11:29:47.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.194+0000 7f8f4d400640 1 -- start start 2024-03-23T11:29:47.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.194+0000 7f8f4d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f4806c300 0x7f8f480ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.194+0000 7f8f4d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8f480cafa0 0x7f8f480cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.194+0000 7f8f4d400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f480cbc60 con 0x7f8f4806c6e0 2024-03-23T11:29:47.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.194+0000 7f8f4d400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f480cbf40 con 0x7f8f480cafa0 2024-03-23T11:29:47.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.194+0000 7f8f4d400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f480cc220 con 0x7f8f4806c300 2024-03-23T11:29:47.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.195+0000 7f8f45e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8f480cafa0 0x7f8f480cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:47.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.195+0000 7f8f45e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8f480cafa0 0x7f8f480cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35474/0 (socket says 172.21.15.62:35474) 2024-03-23T11:29:47.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.195+0000 7f8f45e00640 1 -- 172.21.15.62:0/3968968432 learned_addr learned my addr 172.21.15.62:0/3968968432 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:47.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.195+0000 7f8f47200640 1 --1- 172.21.15.62:0/3968968432 >> v1:172.21.15.62:6789/0 conn(0x7f8f4806c6e0 0x7f8f4806cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53740/0 (socket says 172.21.15.62:53740) 2024-03-23T11:29:47.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.195+0000 7f8f46800640 1 --2- 172.21.15.62:0/3968968432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f4806c300 0x7f8f480ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:47.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.195+0000 7f8f45e00640 1 -- 172.21.15.62:0/3968968432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f4806c300 msgr2=0x7f8f480ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:47.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.195+0000 7f8f45e00640 1 --2- 172.21.15.62:0/3968968432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f4806c300 0x7f8f480ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.195+0000 7f8f45e00640 1 -- 172.21.15.62:0/3968968432 >> v1:172.21.15.62:6789/0 conn(0x7f8f4806c6e0 legacy=0x7f8f4806cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:47.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.195+0000 7f8f45e00640 1 -- 172.21.15.62:0/3968968432 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f480cc500 con 0x7f8f480cafa0 2024-03-23T11:29:47.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.195+0000 7f8f46800640 1 --2- 172.21.15.62:0/3968968432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f4806c300 0x7f8f480ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:47.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.196+0000 7f8f45e00640 1 --2- 172.21.15.62:0/3968968432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f480cafa0 0x7f8f480cb400 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009870 tx=0x7f8f3402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a503f7cc3af5bb6c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:47.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.197+0000 7f8f45400640 1 -- 172.21.15.62:0/3968968432 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f34031640 con 0x7f8f480cafa0 2024-03-23T11:29:47.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.197+0000 7f8f45400640 1 -- 172.21.15.62:0/3968968432 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f340317a0 con 0x7f8f480cafa0 2024-03-23T11:29:47.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.197+0000 7f8f45400640 1 -- 172.21.15.62:0/3968968432 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f34031b30 con 0x7f8f480cafa0 2024-03-23T11:29:47.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.197+0000 7f8f4d400640 1 -- 172.21.15.62:0/3968968432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f480cafa0 msgr2=0x7f8f480cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:47.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.197+0000 7f8f4d400640 1 --2- 172.21.15.62:0/3968968432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f480cafa0 0x7f8f480cb400 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009870 tx=0x7f8f3402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.197+0000 7f8f4d400640 1 -- 172.21.15.62:0/3968968432 shutdown_connections 2024-03-23T11:29:47.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.197+0000 7f8f4d400640 1 --2- 172.21.15.62:0/3968968432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f480cafa0 0x7f8f480cb400 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.197+0000 7f8f4d400640 1 --2- 172.21.15.62:0/3968968432 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f4806c300 0x7f8f480ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.197+0000 7f8f4d400640 1 -- 172.21.15.62:0/3968968432 >> 172.21.15.62:0/3968968432 conn(0x7f8f480b02e0 msgr2=0x7f8f480b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:47.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.198+0000 7f8f4d400640 1 -- 172.21.15.62:0/3968968432 shutdown_connections 2024-03-23T11:29:47.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.198+0000 7f8f4d400640 1 -- 172.21.15.62:0/3968968432 wait complete. 2024-03-23T11:29:47.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.199+0000 7f8f4d400640 1 Processor -- start 2024-03-23T11:29:47.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.199+0000 7f8f4d400640 1 -- start start 2024-03-23T11:29:47.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.200+0000 7f8f4d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f4806c300 0x7f8f4813a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.200+0000 7f8f4d400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f4806c6e0 0x7f8f481f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.200+0000 7f8f4d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f480cafa0 0x7f8f481f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.200+0000 7f8f4d400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4814f650 con 0x7f8f4806c300 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.200+0000 7f8f4d400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4815d820 con 0x7f8f4806c6e0 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.200+0000 7f8f4d400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4815c970 con 0x7f8f480cafa0 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.201+0000 7f8f46800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f4806c300 0x7f8f4813a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.201+0000 7f8f46800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f4806c300 0x7f8f4813a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35490/0 (socket says 172.21.15.62:35490) 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.201+0000 7f8f46800640 1 -- 172.21.15.62:0/1237938135 learned_addr learned my addr 172.21.15.62:0/1237938135 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.201+0000 7f8f45e00640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f4806c6e0 0x7f8f481f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.201+0000 7f8f46800640 1 -- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f480cafa0 msgr2=0x7f8f481f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.201+0000 7f8f46800640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f480cafa0 0x7f8f481f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.201+0000 7f8f46800640 1 -- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f4806c6e0 msgr2=0x7f8f481f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.201+0000 7f8f46800640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f4806c6e0 0x7f8f481f3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.201+0000 7f8f46800640 1 -- 172.21.15.62:0/1237938135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f340094d0 con 0x7f8f4806c300 2024-03-23T11:29:47.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.201+0000 7f8f46800640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f4806c300 0x7f8f4813a720 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800c7f0 tx=0x7f8f3800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:47.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.202+0000 7f8f3ea00640 1 -- 172.21.15.62:0/1237938135 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38007a30 con 0x7f8f4806c300 2024-03-23T11:29:47.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.202+0000 7f8f3ea00640 1 -- 172.21.15.62:0/1237938135 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f38007b90 con 0x7f8f4806c300 2024-03-23T11:29:47.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.202+0000 7f8f3ea00640 1 -- 172.21.15.62:0/1237938135 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3800ce40 con 0x7f8f4806c300 2024-03-23T11:29:47.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.202+0000 7f8f4d400640 1 -- 172.21.15.62:0/1237938135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4815cc10 con 0x7f8f4806c300 2024-03-23T11:29:47.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.202+0000 7f8f4d400640 1 -- 172.21.15.62:0/1237938135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4815d1d0 con 0x7f8f4806c300 2024-03-23T11:29:47.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.203+0000 7f8f4d400640 1 -- 172.21.15.62:0/1237938135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f48066c90 con 0x7f8f4806c300 2024-03-23T11:29:47.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.204+0000 7f8f3ea00640 1 -- 172.21.15.62:0/1237938135 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8f380113f0 con 0x7f8f4806c300 2024-03-23T11:29:47.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.205+0000 7f8f3ea00640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8f2806a7a0 0x7f8f2806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.205+0000 7f8f3ea00640 1 -- 172.21.15.62:0/1237938135 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f38088fe0 con 0x7f8f4806c300 2024-03-23T11:29:47.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.206+0000 7f8f45e00640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8f2806a7a0 0x7f8f2806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:47.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.207+0000 7f8f45e00640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8f2806a7a0 0x7f8f2806cc60 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8f34036000 tx=0x7f8f3403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:47.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.210+0000 7f8f3ea00640 1 -- 172.21.15.62:0/1237938135 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8f380543a0 con 0x7f8f4806c300 2024-03-23T11:29:47.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.408+0000 7f8f4d400640 1 -- 172.21.15.62:0/1237938135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8f480ca650 con 0x7f8f4806c300 2024-03-23T11:29:47.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.408+0000 7f8f3ea00640 1 -- 172.21.15.62:0/1237938135 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8f380579c0 con 0x7f8f4806c300 2024-03-23T11:29:47.400 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:29:47.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 -- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8f2806a7a0 msgr2=0x7f8f2806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:47.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8f2806a7a0 0x7f8f2806cc60 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8f34036000 tx=0x7f8f3403c040 comp rx=0 tx=0).stop 2024-03-23T11:29:47.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 -- 172.21.15.62:0/1237938135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f4806c300 msgr2=0x7f8f4813a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:47.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f4806c300 0x7f8f4813a720 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800c7f0 tx=0x7f8f3800cbf0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 -- 172.21.15.62:0/1237938135 shutdown_connections 2024-03-23T11:29:47.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8f2806a7a0 0x7f8f2806cc60 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f480cafa0 0x7f8f481f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f4806c6e0 0x7f8f481f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 --2- 172.21.15.62:0/1237938135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f4806c300 0x7f8f4813a720 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 -- 172.21.15.62:0/1237938135 >> 172.21.15.62:0/1237938135 conn(0x7f8f480b02e0 msgr2=0x7f8f4813eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:47.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 -- 172.21.15.62:0/1237938135 shutdown_connections 2024-03-23T11:29:47.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.412+0000 7f8f4d400640 1 -- 172.21.15.62:0/1237938135 wait complete. 2024-03-23T11:29:47.423 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:29:47.493 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:29:47.498 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:29:47.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.657+0000 7f9a3cc00640 1 Processor -- start 2024-03-23T11:29:47.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.657+0000 7f9a3cc00640 1 -- start start 2024-03-23T11:29:47.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.659+0000 7f9a3cc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9a3815c9e0 0x7f9a381607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.659+0000 7f9a3cc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9a3815cdc0 0x7f9a380d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.659+0000 7f9a3cc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3806bfc0 con 0x7f9a380cf650 2024-03-23T11:29:47.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.659+0000 7f9a3cc00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a380ce140 con 0x7f9a3815cdc0 2024-03-23T11:29:47.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.659+0000 7f9a3cc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a380b4370 con 0x7f9a3815c9e0 2024-03-23T11:29:47.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.659+0000 7f9a2d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9a3815cdc0 0x7f9a380d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:47.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.659+0000 7f9a2d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9a3815cdc0 0x7f9a380d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45838/0 (socket says 172.21.15.62:45838) 2024-03-23T11:29:47.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.659+0000 7f9a2d400640 1 -- 172.21.15.62:0/2664872549 learned_addr learned my addr 172.21.15.62:0/2664872549 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:47.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.659+0000 7f9a35e00640 1 --2- 172.21.15.62:0/2664872549 >> v2:172.21.15.120:3300/0 conn(0x7f9a3815c9e0 0x7f9a381607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:47.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.660+0000 7f9a2d400640 1 -- 172.21.15.62:0/2664872549 >> v2:172.21.15.120:3300/0 conn(0x7f9a3815c9e0 msgr2=0x7f9a381607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:47.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.660+0000 7f9a2d400640 1 --2- 172.21.15.62:0/2664872549 >> v2:172.21.15.120:3300/0 conn(0x7f9a3815c9e0 0x7f9a381607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.660+0000 7f9a2d400640 1 -- 172.21.15.62:0/2664872549 >> v1:172.21.15.120:6789/0 conn(0x7f9a380cf650 legacy=0x7f9a3815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:47.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.660+0000 7f9a2d400640 1 -- 172.21.15.62:0/2664872549 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a380b5260 con 0x7f9a3815cdc0 2024-03-23T11:29:47.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.662+0000 7f9a2d400640 1 --2- 172.21.15.62:0/2664872549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a3815cdc0 0x7f9a380d1f20 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f9a380d0380 tx=0x7f9a2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28153cff745b0cc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:47.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.662+0000 7f9a35400640 1 -- 172.21.15.62:0/2664872549 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a20031870 con 0x7f9a3815cdc0 2024-03-23T11:29:47.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.662+0000 7f9a35400640 1 -- 172.21.15.62:0/2664872549 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a200319d0 con 0x7f9a3815cdc0 2024-03-23T11:29:47.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.662+0000 7f9a35400640 1 -- 172.21.15.62:0/2664872549 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a20031d60 con 0x7f9a3815cdc0 2024-03-23T11:29:47.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.663+0000 7f9a3cc00640 1 -- 172.21.15.62:0/2664872549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a3815cdc0 msgr2=0x7f9a380d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:47.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.663+0000 7f9a3cc00640 1 --2- 172.21.15.62:0/2664872549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a3815cdc0 0x7f9a380d1f20 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f9a380d0380 tx=0x7f9a2002b910 comp rx=0 tx=0).stop 2024-03-23T11:29:47.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.663+0000 7f9a3cc00640 1 -- 172.21.15.62:0/2664872549 shutdown_connections 2024-03-23T11:29:47.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.663+0000 7f9a3cc00640 1 --2- 172.21.15.62:0/2664872549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a3815cdc0 0x7f9a380d1f20 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.663+0000 7f9a3cc00640 1 --2- 172.21.15.62:0/2664872549 >> v2:172.21.15.120:3300/0 conn(0x7f9a3815c9e0 0x7f9a381607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.663+0000 7f9a3cc00640 1 -- 172.21.15.62:0/2664872549 >> 172.21.15.62:0/2664872549 conn(0x7f9a38071570 msgr2=0x7f9a38071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:47.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.663+0000 7f9a3cc00640 1 -- 172.21.15.62:0/2664872549 shutdown_connections 2024-03-23T11:29:47.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.663+0000 7f9a3cc00640 1 -- 172.21.15.62:0/2664872549 wait complete. 2024-03-23T11:29:47.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.665+0000 7f9a3cc00640 1 Processor -- start 2024-03-23T11:29:47.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.665+0000 7f9a3cc00640 1 -- start start 2024-03-23T11:29:47.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.666+0000 7f9a3cc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a380cf650 0x7f9a380cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.666+0000 7f9a3cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a3815c9e0 0x7f9a381f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.666+0000 7f9a3cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a3815cdc0 0x7f9a380b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.666+0000 7f9a3cc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a380aaf80 con 0x7f9a3815c9e0 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.666+0000 7f9a3cc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a381f34d0 con 0x7f9a380cf650 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.666+0000 7f9a3cc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a381f29a0 con 0x7f9a3815cdc0 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.666+0000 7f9a2d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a3815c9e0 0x7f9a381f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.666+0000 7f9a2d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a3815c9e0 0x7f9a381f2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45848/0 (socket says 172.21.15.62:45848) 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.666+0000 7f9a2d400640 1 -- 172.21.15.62:0/1695765039 learned_addr learned my addr 172.21.15.62:0/1695765039 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.667+0000 7f9a35e00640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a380cf650 0x7f9a380cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.667+0000 7f9a2d400640 1 -- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a3815cdc0 msgr2=0x7f9a380b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.667+0000 7f9a2d400640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a3815cdc0 0x7f9a380b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.667+0000 7f9a2d400640 1 -- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a380cf650 msgr2=0x7f9a380cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.667+0000 7f9a2d400640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a380cf650 0x7f9a380cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.667+0000 7f9a2d400640 1 -- 172.21.15.62:0/1695765039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a200094d0 con 0x7f9a3815c9e0 2024-03-23T11:29:47.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.667+0000 7f9a2d400640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a3815c9e0 0x7f9a381f2460 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f9a2002b950 tx=0x7f9a200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:47.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.667+0000 7f9a2ea00640 1 -- 172.21.15.62:0/1695765039 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2003ac60 con 0x7f9a3815c9e0 2024-03-23T11:29:47.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.667+0000 7f9a2ea00640 1 -- 172.21.15.62:0/1695765039 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a2003adc0 con 0x7f9a3815c9e0 2024-03-23T11:29:47.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.667+0000 7f9a2ea00640 1 -- 172.21.15.62:0/1695765039 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a20044630 con 0x7f9a3815c9e0 2024-03-23T11:29:47.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.667+0000 7f9a3cc00640 1 -- 172.21.15.62:0/1695765039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a381f2c30 con 0x7f9a3815c9e0 2024-03-23T11:29:47.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.668+0000 7f9a3cc00640 1 -- 172.21.15.62:0/1695765039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a381f31f0 con 0x7f9a3815c9e0 2024-03-23T11:29:47.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.669+0000 7f9a3cc00640 1 -- 172.21.15.62:0/1695765039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a380ab0c0 con 0x7f9a3815c9e0 2024-03-23T11:29:47.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.670+0000 7f9a2ea00640 1 -- 172.21.15.62:0/1695765039 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9a2004d450 con 0x7f9a3815c9e0 2024-03-23T11:29:47.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.671+0000 7f9a2ea00640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a1406a7a0 0x7f9a1406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:47.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.671+0000 7f9a2ea00640 1 -- 172.21.15.62:0/1695765039 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a2002d040 con 0x7f9a3815c9e0 2024-03-23T11:29:47.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.672+0000 7f9a35e00640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a1406a7a0 0x7f9a1406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:47.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.673+0000 7f9a35e00640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a1406a7a0 0x7f9a1406cc60 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9a28009f30 tx=0x7f9a2800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:47.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.675+0000 7f9a2ea00640 1 -- 172.21.15.62:0/1695765039 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9a380ab0c0 con 0x7f9a3815c9e0 2024-03-23T11:29:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.877+0000 7f9a3cc00640 1 -- 172.21.15.62:0/1695765039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9a380ab0c0 con 0x7f9a3815c9e0 2024-03-23T11:29:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.878+0000 7f9a2ea00640 1 -- 172.21.15.62:0/1695765039 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9a380ab0c0 con 0x7f9a3815c9e0 2024-03-23T11:29:47.870 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:29:47.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 -- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a1406a7a0 msgr2=0x7f9a1406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:47.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a1406a7a0 0x7f9a1406cc60 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9a28009f30 tx=0x7f9a2800b040 comp rx=0 tx=0).stop 2024-03-23T11:29:47.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 -- 172.21.15.62:0/1695765039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a3815c9e0 msgr2=0x7f9a381f2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:47.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a3815c9e0 0x7f9a381f2460 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f9a2002b950 tx=0x7f9a200046c0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 -- 172.21.15.62:0/1695765039 shutdown_connections 2024-03-23T11:29:47.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a1406a7a0 0x7f9a1406cc60 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a3815c9e0 0x7f9a381f2460 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a3815cdc0 0x7f9a380b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 --2- 172.21.15.62:0/1695765039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a380cf650 0x7f9a380cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:47.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 -- 172.21.15.62:0/1695765039 >> 172.21.15.62:0/1695765039 conn(0x7f9a38071570 msgr2=0x7f9a380cfac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:47.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 -- 172.21.15.62:0/1695765039 shutdown_connections 2024-03-23T11:29:47.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:47.881+0000 7f9a3cc00640 1 -- 172.21.15.62:0/1695765039 wait complete. 2024-03-23T11:29:47.892 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:29:48.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.178+0000 7ff10ca00640 1 Processor -- start 2024-03-23T11:29:48.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.179+0000 7ff10ca00640 1 -- start start 2024-03-23T11:29:48.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.180+0000 7ff10ca00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff108143dd0 0x7ff10815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:48.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.180+0000 7ff10ca00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1080cd140 con 0x7ff108070090 2024-03-23T11:29:48.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.180+0000 7ff10ca00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1080cd3e0 con 0x7ff1080b2f30 2024-03-23T11:29:48.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.180+0000 7ff10ca00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1080cd6c0 con 0x7ff108143dd0 2024-03-23T11:29:48.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.180+0000 7ff0ffe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff108070090 0x7ff1080b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42650/0 (socket says 172.21.15.62:42650) 2024-03-23T11:29:48.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.180+0000 7ff107400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff108143dd0 0x7ff10815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:48.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.180+0000 7ff0ffe00640 1 -- 172.21.15.62:0/1555922711 learned_addr learned my addr 172.21.15.62:0/1555922711 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:48.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.181+0000 7ff107400640 1 -- 172.21.15.62:0/1555922711 >> v1:172.21.15.120:6789/0 conn(0x7ff1080b2f30 legacy=0x7ff1080ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:48.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.181+0000 7ff107400640 1 -- 172.21.15.62:0/1555922711 >> v1:172.21.15.62:6789/0 conn(0x7ff108070090 legacy=0x7ff1080b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.181+0000 7ff107400640 1 -- 172.21.15.62:0/1555922711 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1080cd9a0 con 0x7ff108143dd0 2024-03-23T11:29:48.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.181+0000 7ff106a00640 1 -- 172.21.15.62:0/1555922711 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417570356 0 0) 0x7ff1080cd140 con 0x7ff108070090 2024-03-23T11:29:48.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.182+0000 7ff107400640 1 --2- 172.21.15.62:0/1555922711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff108143dd0 0x7ff10815abe0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097a0 tx=0x7ff0f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff3b3f5f989249b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:48.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.182+0000 7ff106a00640 1 -- 172.21.15.62:0/1555922711 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8031830 con 0x7ff108143dd0 2024-03-23T11:29:48.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.182+0000 7ff106a00640 1 -- 172.21.15.62:0/1555922711 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f8031990 con 0x7ff108143dd0 2024-03-23T11:29:48.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.182+0000 7ff106a00640 1 -- 172.21.15.62:0/1555922711 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8031d20 con 0x7ff108143dd0 2024-03-23T11:29:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.183+0000 7ff10ca00640 1 -- 172.21.15.62:0/1555922711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff108143dd0 msgr2=0x7ff10815abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.183+0000 7ff10ca00640 1 --2- 172.21.15.62:0/1555922711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff108143dd0 0x7ff10815abe0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097a0 tx=0x7ff0f802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.183+0000 7ff10ca00640 1 -- 172.21.15.62:0/1555922711 shutdown_connections 2024-03-23T11:29:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.183+0000 7ff10ca00640 1 --2- 172.21.15.62:0/1555922711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff108143dd0 0x7ff10815abe0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.183+0000 7ff10ca00640 1 -- 172.21.15.62:0/1555922711 >> 172.21.15.62:0/1555922711 conn(0x7ff108066050 msgr2=0x7ff108069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.183+0000 7ff10ca00640 1 -- 172.21.15.62:0/1555922711 shutdown_connections 2024-03-23T11:29:48.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.183+0000 7ff10ca00640 1 -- 172.21.15.62:0/1555922711 wait complete. 2024-03-23T11:29:48.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.185+0000 7ff10ca00640 1 Processor -- start 2024-03-23T11:29:48.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.185+0000 7ff10ca00640 1 -- start start 2024-03-23T11:29:48.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.186+0000 7ff10ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff108070090 0x7ff1080ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:48.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.186+0000 7ff10ca00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1080b2f30 0x7ff10815a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:48.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.186+0000 7ff10ca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff108143dd0 0x7ff1081f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:48.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.186+0000 7ff10ca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10813d500 con 0x7ff108070090 2024-03-23T11:29:48.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.186+0000 7ff10ca00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff108139e70 con 0x7ff1080b2f30 2024-03-23T11:29:48.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.186+0000 7ff10ca00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff108140310 con 0x7ff108143dd0 2024-03-23T11:29:48.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.186+0000 7ff107400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff108070090 0x7ff1080ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:48.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.186+0000 7ff107400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff108070090 0x7ff1080ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45868/0 (socket says 172.21.15.62:45868) 2024-03-23T11:29:48.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.186+0000 7ff107400640 1 -- 172.21.15.62:0/1717281152 learned_addr learned my addr 172.21.15.62:0/1717281152 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.187+0000 7ff0ffe00640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1080b2f30 0x7ff10815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.187+0000 7ff107400640 1 -- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff108143dd0 msgr2=0x7ff1081f6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.187+0000 7ff107e00640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff108143dd0 0x7ff1081f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.187+0000 7ff107400640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff108143dd0 0x7ff1081f6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.187+0000 7ff107400640 1 -- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1080b2f30 msgr2=0x7ff10815a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.187+0000 7ff107400640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1080b2f30 0x7ff10815a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.187+0000 7ff107400640 1 -- 172.21.15.62:0/1717281152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f8009400 con 0x7ff108070090 2024-03-23T11:29:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.187+0000 7ff0ffe00640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1080b2f30 0x7ff10815a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.187+0000 7ff107400640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff108070090 0x7ff1080ab840 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097a0 tx=0x7ff0f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:48.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.188+0000 7ff0ff400640 1 -- 172.21.15.62:0/1717281152 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f803aa60 con 0x7ff108070090 2024-03-23T11:29:48.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.188+0000 7ff0ff400640 1 -- 172.21.15.62:0/1717281152 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f803abc0 con 0x7ff108070090 2024-03-23T11:29:48.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.188+0000 7ff0ff400640 1 -- 172.21.15.62:0/1717281152 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80443f0 con 0x7ff108070090 2024-03-23T11:29:48.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.188+0000 7ff10ca00640 1 -- 172.21.15.62:0/1717281152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff10815a570 con 0x7ff108070090 2024-03-23T11:29:48.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.188+0000 7ff10ca00640 1 -- 172.21.15.62:0/1717281152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1081f2930 con 0x7ff108070090 2024-03-23T11:29:48.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.189+0000 7ff10ca00640 1 -- 172.21.15.62:0/1717281152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1080ccd20 con 0x7ff108070090 2024-03-23T11:29:48.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.190+0000 7ff0ff400640 1 -- 172.21.15.62:0/1717281152 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff0f804d450 con 0x7ff108070090 2024-03-23T11:29:48.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.191+0000 7ff0ff400640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0e406a7d0 0x7ff0e406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:48.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.192+0000 7ff0ff400640 1 -- 172.21.15.62:0/1717281152 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0f80aafd0 con 0x7ff108070090 2024-03-23T11:29:48.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.192+0000 7ff0ffe00640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0e406a7d0 0x7ff0e406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:48.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.193+0000 7ff0ffe00640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0e406a7d0 0x7ff0e406cc90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff1081437e0 tx=0x7ff0f0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:48.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.196+0000 7ff0ff400640 1 -- 172.21.15.62:0/1717281152 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff0f8076480 con 0x7ff108070090 2024-03-23T11:29:48.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.395+0000 7ff10ca00640 1 -- 172.21.15.62:0/1717281152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff108066bb0 con 0x7ff108070090 2024-03-23T11:29:48.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.396+0000 7ff0ff400640 1 -- 172.21.15.62:0/1717281152 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff0f8079aa0 con 0x7ff108070090 2024-03-23T11:29:48.387 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:29:48.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 -- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0e406a7d0 msgr2=0x7ff0e406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0e406a7d0 0x7ff0e406cc90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff1081437e0 tx=0x7ff0f0009040 comp rx=0 tx=0).stop 2024-03-23T11:29:48.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 -- 172.21.15.62:0/1717281152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff108070090 msgr2=0x7ff1080ab840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff108070090 0x7ff1080ab840 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097a0 tx=0x7ff0f80046c0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 -- 172.21.15.62:0/1717281152 shutdown_connections 2024-03-23T11:29:48.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff0e406a7d0 0x7ff0e406cc90 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff108143dd0 0x7ff1081f6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1080b2f30 0x7ff10815a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 --2- 172.21.15.62:0/1717281152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff108070090 0x7ff1080ab840 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 -- 172.21.15.62:0/1717281152 >> 172.21.15.62:0/1717281152 conn(0x7ff108066050 msgr2=0x7ff108070520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:48.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 -- 172.21.15.62:0/1717281152 shutdown_connections 2024-03-23T11:29:48.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.399+0000 7ff10ca00640 1 -- 172.21.15.62:0/1717281152 wait complete. 2024-03-23T11:29:48.407 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:29:48.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.631+0000 7f0bf3c00640 1 Processor -- start 2024-03-23T11:29:48.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.631+0000 7f0bf3c00640 1 -- start start 2024-03-23T11:29:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.633+0000 7f0bf3c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0bec0cafa0 0x7f0bec0cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.633+0000 7f0bf3c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bec06c6e0 0x7f0bec06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.633+0000 7f0bf3c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bec0cbc60 con 0x7f0bec06c300 2024-03-23T11:29:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.633+0000 7f0bf3c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bec0cd410 con 0x7f0bec0cafa0 2024-03-23T11:29:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.633+0000 7f0bf3c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec0cbf40 con 0x7f0bec06c6e0 2024-03-23T11:29:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.633+0000 7f0bebe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0bec0cafa0 0x7f0bec0cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.633+0000 7f0bf1200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0bec06c300 0x7f0bec0ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42654/0 (socket says 172.21.15.62:42654) 2024-03-23T11:29:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.633+0000 7f0bf1200640 1 -- 172.21.15.62:0/3857411884 learned_addr learned my addr 172.21.15.62:0/3857411884 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.633+0000 7f0bf1c00640 1 --2- 172.21.15.62:0/3857411884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bec06c6e0 0x7f0bec06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.634+0000 7f0bebe00640 1 -- 172.21.15.62:0/3857411884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bec06c6e0 msgr2=0x7f0bec06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.634+0000 7f0bebe00640 1 --2- 172.21.15.62:0/3857411884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bec06c6e0 0x7f0bec06cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.634+0000 7f0bebe00640 1 -- 172.21.15.62:0/3857411884 >> v1:172.21.15.62:6789/0 conn(0x7f0bec06c300 legacy=0x7f0bec0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.634+0000 7f0bebe00640 1 -- 172.21.15.62:0/3857411884 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec0cc220 con 0x7f0bec0cafa0 2024-03-23T11:29:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.634+0000 7f0bf1c00640 1 --2- 172.21.15.62:0/3857411884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bec06c6e0 0x7f0bec06cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.635+0000 7f0bebe00640 1 --2- 172.21.15.62:0/3857411884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bec0cafa0 0x7f0bec0cb430 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80097a0 tx=0x7f0bd802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608f8c2b5e3a30b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.635+0000 7f0beb400640 1 -- 172.21.15.62:0/3857411884 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd802d040 con 0x7f0bec0cafa0 2024-03-23T11:29:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.635+0000 7f0beb400640 1 -- 172.21.15.62:0/3857411884 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd8031810 con 0x7f0bec0cafa0 2024-03-23T11:29:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.635+0000 7f0beb400640 1 -- 172.21.15.62:0/3857411884 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8031ba0 con 0x7f0bec0cafa0 2024-03-23T11:29:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.636+0000 7f0bf3c00640 1 -- 172.21.15.62:0/3857411884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bec0cafa0 msgr2=0x7f0bec0cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.636+0000 7f0bf3c00640 1 --2- 172.21.15.62:0/3857411884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bec0cafa0 0x7f0bec0cb430 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80097a0 tx=0x7f0bd802b8b0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.636+0000 7f0bf3c00640 1 -- 172.21.15.62:0/3857411884 shutdown_connections 2024-03-23T11:29:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.636+0000 7f0bf3c00640 1 --2- 172.21.15.62:0/3857411884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bec06c6e0 0x7f0bec06cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.636+0000 7f0bf3c00640 1 --2- 172.21.15.62:0/3857411884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bec0cafa0 0x7f0bec0cb430 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.636+0000 7f0bf3c00640 1 -- 172.21.15.62:0/3857411884 >> 172.21.15.62:0/3857411884 conn(0x7f0bec0b02e0 msgr2=0x7f0bec0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.636+0000 7f0bf3c00640 1 -- 172.21.15.62:0/3857411884 shutdown_connections 2024-03-23T11:29:48.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.636+0000 7f0bf3c00640 1 -- 172.21.15.62:0/3857411884 wait complete. 2024-03-23T11:29:48.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.638+0000 7f0bf3c00640 1 Processor -- start 2024-03-23T11:29:48.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.638+0000 7f0bf3c00640 1 -- start start 2024-03-23T11:29:48.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.639+0000 7f0bf3c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bec06c300 0x7f0bec0a92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:48.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.639+0000 7f0bf3c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bec06c6e0 0x7f0bec156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:48.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.639+0000 7f0bf3c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bec0cafa0 0x7f0bec15bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:48.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.639+0000 7f0bf3c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec15d4c0 con 0x7f0bec0cafa0 2024-03-23T11:29:48.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.639+0000 7f0bf3c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec1574a0 con 0x7f0bec06c6e0 2024-03-23T11:29:48.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.639+0000 7f0bf3c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec15c840 con 0x7f0bec06c300 2024-03-23T11:29:48.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.639+0000 7f0bf1c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bec0cafa0 0x7f0bec15bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:48.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.640+0000 7f0bf1c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bec0cafa0 0x7f0bec15bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45878/0 (socket says 172.21.15.62:45878) 2024-03-23T11:29:48.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.640+0000 7f0bf1c00640 1 -- 172.21.15.62:0/2441928560 learned_addr learned my addr 172.21.15.62:0/2441928560 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:48.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.640+0000 7f0bf1200640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bec06c300 0x7f0bec0a92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:48.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.640+0000 7f0bebe00640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bec06c6e0 0x7f0bec156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:48.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.640+0000 7f0bf1c00640 1 -- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bec06c300 msgr2=0x7f0bec0a92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.640+0000 7f0bf1c00640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bec06c300 0x7f0bec0a92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.640+0000 7f0bf1c00640 1 -- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bec06c6e0 msgr2=0x7f0bec156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.640+0000 7f0bf1c00640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bec06c6e0 0x7f0bec156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.640+0000 7f0bf1c00640 1 -- 172.21.15.62:0/2441928560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd8009400 con 0x7f0bec0cafa0 2024-03-23T11:29:48.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.640+0000 7f0bf1c00640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bec0cafa0 0x7f0bec15bf60 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f0be400cef0 tx=0x7f0be400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:48.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.641+0000 7f0be8c00640 1 -- 172.21.15.62:0/2441928560 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be400d840 con 0x7f0bec0cafa0 2024-03-23T11:29:48.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.641+0000 7f0be8c00640 1 -- 172.21.15.62:0/2441928560 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be400d9c0 con 0x7f0bec0cafa0 2024-03-23T11:29:48.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.641+0000 7f0be8c00640 1 -- 172.21.15.62:0/2441928560 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be400dd70 con 0x7f0bec0cafa0 2024-03-23T11:29:48.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.641+0000 7f0bf3c00640 1 -- 172.21.15.62:0/2441928560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bec15cb20 con 0x7f0bec0cafa0 2024-03-23T11:29:48.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.641+0000 7f0bf3c00640 1 -- 172.21.15.62:0/2441928560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bec15d0e0 con 0x7f0bec0cafa0 2024-03-23T11:29:48.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.642+0000 7f0bf3c00640 1 -- 172.21.15.62:0/2441928560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bec15d600 con 0x7f0bec0cafa0 2024-03-23T11:29:48.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.643+0000 7f0be8c00640 1 -- 172.21.15.62:0/2441928560 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0be401e3b0 con 0x7f0bec0cafa0 2024-03-23T11:29:48.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.644+0000 7f0be8c00640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0bcc06a7a0 0x7f0bcc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:48.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.644+0000 7f0be8c00640 1 -- 172.21.15.62:0/2441928560 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0be4089910 con 0x7f0bec0cafa0 2024-03-23T11:29:48.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.644+0000 7f0bf1200640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0bcc06a7a0 0x7f0bcc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:48.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.645+0000 7f0bf1200640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0bcc06a7a0 0x7f0bcc06cc60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0be0005880 tx=0x7f0be0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.649+0000 7f0be8c00640 1 -- 172.21.15.62:0/2441928560 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0bec15d600 con 0x7f0bec0cafa0 2024-03-23T11:29:48.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.857+0000 7f0bf3c00640 1 -- 172.21.15.62:0/2441928560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0bec15d600 con 0x7f0bec0cafa0 2024-03-23T11:29:48.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.858+0000 7f0be8c00640 1 -- 172.21.15.62:0/2441928560 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v105) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0bec15d600 con 0x7f0bec0cafa0 2024-03-23T11:29:48.849 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:29:48.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.860+0000 7f0bf3c00640 1 -- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0bcc06a7a0 msgr2=0x7f0bcc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.860+0000 7f0bf3c00640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0bcc06a7a0 0x7f0bcc06cc60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0be0005880 tx=0x7f0be0009040 comp rx=0 tx=0).stop 2024-03-23T11:29:48.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.860+0000 7f0bf3c00640 1 -- 172.21.15.62:0/2441928560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bec0cafa0 msgr2=0x7f0bec15bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:48.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.860+0000 7f0bf3c00640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bec0cafa0 0x7f0bec15bf60 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f0be400cef0 tx=0x7f0be400a710 comp rx=0 tx=0).stop 2024-03-23T11:29:48.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.861+0000 7f0bf3c00640 1 -- 172.21.15.62:0/2441928560 shutdown_connections 2024-03-23T11:29:48.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.861+0000 7f0bf3c00640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0bcc06a7a0 0x7f0bcc06cc60 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.861+0000 7f0bf3c00640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bec0cafa0 0x7f0bec15bf60 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.861+0000 7f0bf3c00640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bec06c6e0 0x7f0bec156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.861+0000 7f0bf3c00640 1 --2- 172.21.15.62:0/2441928560 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0bec06c300 0x7f0bec0a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:48.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.861+0000 7f0bf3c00640 1 -- 172.21.15.62:0/2441928560 >> 172.21.15.62:0/2441928560 conn(0x7f0bec0b02e0 msgr2=0x7f0bec13dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:48.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.861+0000 7f0bf3c00640 1 -- 172.21.15.62:0/2441928560 shutdown_connections 2024-03-23T11:29:48.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:48.861+0000 7f0bf3c00640 1 -- 172.21.15.62:0/2441928560 wait complete. 2024-03-23T11:29:48.869 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:29:48.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:48.869 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:48.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[73624]: fuse finished with error 0 and tester_r 0 2024-03-23T11:29:48.965 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:29:55.066 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:29:55.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:55.067 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:29:55.129 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:29:55.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.356+0000 7f941b000640 1 Processor -- start 2024-03-23T11:29:55.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.356+0000 7f941b000640 1 -- start start 2024-03-23T11:29:55.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.357+0000 7f941b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f941415c900 0x7f94140d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:55.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.357+0000 7f941b000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f94140cd690 0x7f94140cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:55.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.357+0000 7f941b000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94140ce020 con 0x7f941415cce0 2024-03-23T11:29:55.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.357+0000 7f941b000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94140b89f0 con 0x7f94140cd690 2024-03-23T11:29:55.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.357+0000 7f941b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94140d1100 con 0x7f941415c900 2024-03-23T11:29:55.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.358+0000 7f9419000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f94140cd690 0x7f94140cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:55.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.358+0000 7f9419000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f94140cd690 0x7f94140cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45880/0 (socket says 172.21.15.62:45880) 2024-03-23T11:29:55.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.358+0000 7f9419000640 1 -- 172.21.15.62:0/3655135131 learned_addr learned my addr 172.21.15.62:0/3655135131 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:55.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.358+0000 7f9413e00640 1 --2- 172.21.15.62:0/3655135131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f941415c900 0x7f94140d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:55.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.358+0000 7f9419000640 1 -- 172.21.15.62:0/3655135131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f941415c900 msgr2=0x7f94140d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:55.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.358+0000 7f9419000640 1 --2- 172.21.15.62:0/3655135131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f941415c900 0x7f94140d21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:55.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.359+0000 7f9419000640 1 -- 172.21.15.62:0/3655135131 >> v1:172.21.15.120:6789/0 conn(0x7f941415cce0 legacy=0x7f94140b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:55.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.359+0000 7f9419000640 1 -- 172.21.15.62:0/3655135131 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94140d13e0 con 0x7f94140cd690 2024-03-23T11:29:55.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.360+0000 7f9419000640 1 --2- 172.21.15.62:0/3655135131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94140cd690 0x7f94140cdae0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f940c009e50 tx=0x7f940c02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb20c921be70b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:55.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.360+0000 7f9412a00640 1 -- 172.21.15.62:0/3655135131 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c02cb90 con 0x7f94140cd690 2024-03-23T11:29:55.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.360+0000 7f9412a00640 1 -- 172.21.15.62:0/3655135131 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940c02ccf0 con 0x7f94140cd690 2024-03-23T11:29:55.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.360+0000 7f9412a00640 1 -- 172.21.15.62:0/3655135131 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c034610 con 0x7f94140cd690 2024-03-23T11:29:55.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.361+0000 7f941b000640 1 -- 172.21.15.62:0/3655135131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94140cd690 msgr2=0x7f94140cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:55.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.361+0000 7f941b000640 1 --2- 172.21.15.62:0/3655135131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94140cd690 0x7f94140cdae0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f940c009e50 tx=0x7f940c02bff0 comp rx=0 tx=0).stop 2024-03-23T11:29:55.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.361+0000 7f941b000640 1 -- 172.21.15.62:0/3655135131 shutdown_connections 2024-03-23T11:29:55.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.361+0000 7f941b000640 1 --2- 172.21.15.62:0/3655135131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94140cd690 0x7f94140cdae0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:55.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.361+0000 7f941b000640 1 --2- 172.21.15.62:0/3655135131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f941415c900 0x7f94140d21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:55.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.361+0000 7f941b000640 1 -- 172.21.15.62:0/3655135131 >> 172.21.15.62:0/3655135131 conn(0x7f9414071390 msgr2=0x7f9414071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:55.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.361+0000 7f941b000640 1 -- 172.21.15.62:0/3655135131 shutdown_connections 2024-03-23T11:29:55.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.361+0000 7f941b000640 1 -- 172.21.15.62:0/3655135131 wait complete. 2024-03-23T11:29:55.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.363+0000 7f941b000640 1 Processor -- start 2024-03-23T11:29:55.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.363+0000 7f941b000640 1 -- start start 2024-03-23T11:29:55.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f941b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94140cd690 0x7f94141f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:55.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f941b000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f941415c900 0x7f94140c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f941b000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f941415cce0 0x7f94140cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f941b000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94140cc0e0 con 0x7f941415cce0 2024-03-23T11:29:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f941b000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94140d34f0 con 0x7f941415c900 2024-03-23T11:29:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f941b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94140ca4a0 con 0x7f94140cd690 2024-03-23T11:29:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f9419000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f941415cce0 0x7f94140cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f9419000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f941415cce0 0x7f94140cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45894/0 (socket says 172.21.15.62:45894) 2024-03-23T11:29:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f9419000640 1 -- 172.21.15.62:0/126323618 learned_addr learned my addr 172.21.15.62:0/126323618 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f9413400640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f941415c900 0x7f94140c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f9413e00640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94140cd690 0x7f94141f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f9419000640 1 -- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94140cd690 msgr2=0x7f94141f50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f9419000640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94140cd690 0x7f94141f50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:55.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.364+0000 7f9419000640 1 -- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f941415c900 msgr2=0x7f94140c5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:55.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.365+0000 7f9419000640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f941415c900 0x7f94140c5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:55.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.365+0000 7f9419000640 1 -- 172.21.15.62:0/126323618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c009b30 con 0x7f941415cce0 2024-03-23T11:29:55.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.365+0000 7f9413400640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f941415c900 0x7f94140c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:55.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.365+0000 7f9419000640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f941415cce0 0x7f94140cb260 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f940c009e50 tx=0x7f940c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:55.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.365+0000 7f940be00640 1 -- 172.21.15.62:0/126323618 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c009510 con 0x7f941415cce0 2024-03-23T11:29:55.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.365+0000 7f940be00640 1 -- 172.21.15.62:0/126323618 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940c03e450 con 0x7f941415cce0 2024-03-23T11:29:55.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.365+0000 7f940be00640 1 -- 172.21.15.62:0/126323618 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c03e800 con 0x7f941415cce0 2024-03-23T11:29:55.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.365+0000 7f941b000640 1 -- 172.21.15.62:0/126323618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94140ca780 con 0x7f941415cce0 2024-03-23T11:29:55.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.366+0000 7f941b000640 1 -- 172.21.15.62:0/126323618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94140cad40 con 0x7f941415cce0 2024-03-23T11:29:55.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.367+0000 7f941b000640 1 -- 172.21.15.62:0/126323618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94140cde10 con 0x7f941415cce0 2024-03-23T11:29:55.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.368+0000 7f940be00640 1 -- 172.21.15.62:0/126323618 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f940c03e960 con 0x7f941415cce0 2024-03-23T11:29:55.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.368+0000 7f940be00640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f006a7a0 0x7f93f006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:55.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.368+0000 7f940be00640 1 -- 172.21.15.62:0/126323618 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f940c0aba10 con 0x7f941415cce0 2024-03-23T11:29:55.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.369+0000 7f9413e00640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f006a7a0 0x7f93f006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:55.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.370+0000 7f9413e00640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f006a7a0 0x7f93f006cc60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f9404006fd0 tx=0x7f9404008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:55.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.373+0000 7f940be00640 1 -- 172.21.15.62:0/126323618 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f940c076e30 con 0x7f941415cce0 2024-03-23T11:29:55.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.612+0000 7f941b000640 1 -- 172.21.15.62:0/126323618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94140672c0 con 0x7f941415cce0 2024-03-23T11:29:55.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.613+0000 7f940be00640 1 -- 172.21.15.62:0/126323618 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f940c02ce60 con 0x7f941415cce0 2024-03-23T11:29:55.604 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:55.604 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":78,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:29:19.157087+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8271":{"gid":8271,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6834/692790652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":692790652},{"type":"v1","addr":"172.21.15.62:6834","nonce":692790652}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:29:55.604 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 78 2024-03-23T11:29:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 -- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f006a7a0 msgr2=0x7f93f006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f006a7a0 0x7f93f006cc60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f9404006fd0 tx=0x7f9404008040 comp rx=0 tx=0).stop 2024-03-23T11:29:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 -- 172.21.15.62:0/126323618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f941415cce0 msgr2=0x7f94140cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f941415cce0 0x7f94140cb260 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f940c009e50 tx=0x7f940c004a40 comp rx=0 tx=0).stop 2024-03-23T11:29:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 -- 172.21.15.62:0/126323618 shutdown_connections 2024-03-23T11:29:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f93f006a7a0 0x7f93f006cc60 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f941415cce0 0x7f94140cb260 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f941415c900 0x7f94140c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 --2- 172.21.15.62:0/126323618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f94140cd690 0x7f94141f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 -- 172.21.15.62:0/126323618 >> 172.21.15.62:0/126323618 conn(0x7f9414071390 msgr2=0x7f9414157200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:55.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 -- 172.21.15.62:0/126323618 shutdown_connections 2024-03-23T11:29:55.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:55.616+0000 7f941b000640 1 -- 172.21.15.62:0/126323618 wait complete. 2024-03-23T11:29:55.627 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-23T11:29:55.870 INFO:teuthology.orchestra.run.smithi062.stdout:{"message":"","return_code":0} 2024-03-23T11:29:55.870 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-23T11:29:55.871 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:29:55.871 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:29:55.923 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:29:55.923 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi062.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-03-23T11:29:55.923 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:29:55.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:29:55.924 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:29:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-23T11:29:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:29:55.980 DEBUG:teuthology.orchestra.run.smithi062:> sudo daemon-helper kill ceph --cluster ceph -w 2024-03-23T11:29:55.985 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-03-23T11:29:56.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.198+0000 7fc2b4a00640 1 Processor -- start 2024-03-23T11:29:56.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.199+0000 7fc2b4a00640 1 -- start start 2024-03-23T11:29:56.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.200+0000 7fc2b4a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015c860 0x7fc2b00d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.200+0000 7fc2b4a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc2b00d0c30 0x7fc2b01604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.200+0000 7fc2b4a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2b00af3a0 con 0x7fc2b0160a00 2024-03-23T11:29:56.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.200+0000 7fc2b4a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b0160e90 con 0x7fc2b00d0c30 2024-03-23T11:29:56.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.200+0000 7fc2b4a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00d3540 con 0x7fc2b015c860 2024-03-23T11:29:56.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.201+0000 7fc2afe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc2b0160a00 0x7fc2b00cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42660/0 (socket says 172.21.15.62:42660) 2024-03-23T11:29:56.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.201+0000 7fc2afe00640 1 -- 172.21.15.62:0/3626864436 learned_addr learned my addr 172.21.15.62:0/3626864436 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:56.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.201+0000 7fc2af400640 1 --2- 172.21.15.62:0/3626864436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015c860 0x7fc2b00d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.201+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/3626864436 >> v2:172.21.15.120:3300/0 conn(0x7fc2b00d0c30 0x7fc2b01604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.201+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270621521 0 0) 0x7fc2b00af3a0 con 0x7fc2b0160a00 2024-03-23T11:29:56.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.201+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc294003520 con 0x7fc2b0160a00 2024-03-23T11:29:56.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2488100940 0 0) 0x7fc294003520 con 0x7fc2b0160a00 2024-03-23T11:29:56.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015c860 msgr2=0x7fc2b00d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 --2- 172.21.15.62:0/3626864436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015c860 0x7fc2b00d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 >> v2:172.21.15.120:3300/0 conn(0x7fc2b00d0c30 msgr2=0x7fc2b01604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 --2- 172.21.15.62:0/3626864436 >> v2:172.21.15.120:3300/0 conn(0x7fc2b00d0c30 0x7fc2b01604c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b00d1110 con 0x7fc2b0160a00 2024-03-23T11:29:56.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fc2a8001670 con 0x7fc2b0160a00 2024-03-23T11:29:56.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 >> v1:172.21.15.62:6789/0 conn(0x7fc2b0160a00 legacy=0x7fc2b00cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 --2- 172.21.15.62:0/3626864436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 0x7fc294003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 --2- 172.21.15.62:0/3626864436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015c860 0x7fc2940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 --2- 172.21.15.62:0/3626864436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc294004d00 0x7fc294005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b0160e90 con 0x7fc294003970 2024-03-23T11:29:56.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00d3540 con 0x7fc294004d00 2024-03-23T11:29:56.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29400b980 con 0x7fc2b015c860 2024-03-23T11:29:56.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.202+0000 7fc2af400640 1 --2- 172.21.15.62:0/3626864436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 0x7fc294003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.203+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/3626864436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015c860 0x7fc2940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.203+0000 7fc2af400640 1 -- 172.21.15.62:0/3626864436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015c860 msgr2=0x7fc2940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.203+0000 7fc2af400640 1 --2- 172.21.15.62:0/3626864436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015c860 0x7fc2940047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.203+0000 7fc2af400640 1 -- 172.21.15.62:0/3626864436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc294004d00 msgr2=0x7fc294005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.203+0000 7fc2af400640 1 --2- 172.21.15.62:0/3626864436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc294004d00 0x7fc294005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.203+0000 7fc2af400640 1 -- 172.21.15.62:0/3626864436 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc294003520 con 0x7fc294003970 2024-03-23T11:29:56.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.204+0000 7fc2af400640 1 --2- 172.21.15.62:0/3626864436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 0x7fc294003d90 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fc2a000e170 tx=0x7fc2940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c776a53f5fb3ba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.204+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a002cc60 con 0x7fc294003970 2024-03-23T11:29:56.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.204+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a000f040 con 0x7fc294003970 2024-03-23T11:29:56.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.204+0000 7fc2aea00640 1 -- 172.21.15.62:0/3626864436 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0034440 con 0x7fc294003970 2024-03-23T11:29:56.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.205+0000 7fc2b4a00640 1 -- 172.21.15.62:0/3626864436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 msgr2=0x7fc294003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.205+0000 7fc2b4a00640 1 --2- 172.21.15.62:0/3626864436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 0x7fc294003d90 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fc2a000e170 tx=0x7fc2940057f0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.205+0000 7fc2afe00640 1 -- 172.21.15.62:0/3626864436 reap_dead start 2024-03-23T11:29:56.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.205+0000 7fc2b4a00640 1 -- 172.21.15.62:0/3626864436 shutdown_connections 2024-03-23T11:29:56.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.205+0000 7fc2b4a00640 1 -- 172.21.15.62:0/3626864436 >> 172.21.15.62:0/3626864436 conn(0x7fc2b0089f60 msgr2=0x7fc2b00d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.205+0000 7fc2b4a00640 1 -- 172.21.15.62:0/3626864436 shutdown_connections 2024-03-23T11:29:56.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.205+0000 7fc2b4a00640 1 -- 172.21.15.62:0/3626864436 wait complete. 2024-03-23T11:29:56.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.207+0000 7fc2b4a00640 1 Processor -- start 2024-03-23T11:29:56.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.207+0000 7fc2b4a00640 1 -- start start 2024-03-23T11:29:56.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.208+0000 7fc2b4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 0x7fc2b0159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.208+0000 7fc2b4a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015ef50 0x7fc2b015f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.208+0000 7fc2b4a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b00cb690 0x7fc2b00cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.208+0000 7fc2b4a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00ccc60 con 0x7fc294003970 2024-03-23T11:29:56.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.208+0000 7fc2b4a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b015f910 con 0x7fc2b00cb690 2024-03-23T11:29:56.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.208+0000 7fc2b4a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00cc3c0 con 0x7fc2b015ef50 2024-03-23T11:29:56.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.208+0000 7fc2af400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 0x7fc2b0159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.208+0000 7fc2af400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 0x7fc2b0159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45922/0 (socket says 172.21.15.62:45922) 2024-03-23T11:29:56.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.208+0000 7fc2af400640 1 -- 172.21.15.62:0/1501233194 learned_addr learned my addr 172.21.15.62:0/1501233194 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.209+0000 7fc2afe00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b00cb690 0x7fc2b00cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.209+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015ef50 0x7fc2b015f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.209+0000 7fc2af400640 1 -- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015ef50 msgr2=0x7fc2b015f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.209+0000 7fc2af400640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015ef50 0x7fc2b015f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.209+0000 7fc2af400640 1 -- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b00cb690 msgr2=0x7fc2b00cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.209+0000 7fc2af400640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b00cb690 0x7fc2b00cbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.209+0000 7fc2af400640 1 -- 172.21.15.62:0/1501233194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b00cc6a0 con 0x7fc294003970 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.209+0000 7fc2afe00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b00cb690 0x7fc2b00cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.209+0000 7fc2af400640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 0x7fc2b0159bf0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fc2a002c600 tx=0x7fc2a0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.210+0000 7fc2a7e00640 1 -- 172.21.15.62:0/1501233194 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0040450 con 0x7fc294003970 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.210+0000 7fc2a7e00640 1 -- 172.21.15.62:0/1501233194 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a00405d0 con 0x7fc294003970 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.210+0000 7fc2a7e00640 1 -- 172.21.15.62:0/1501233194 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0040980 con 0x7fc294003970 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.210+0000 7fc2b4a00640 1 -- 172.21.15.62:0/1501233194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b00cc980 con 0x7fc294003970 2024-03-23T11:29:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.210+0000 7fc2b4a00640 1 -- 172.21.15.62:0/1501233194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b015a130 con 0x7fc294003970 2024-03-23T11:29:56.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.212+0000 7fc2a7e00640 1 -- 172.21.15.62:0/1501233194 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc2a0040ae0 con 0x7fc294003970 2024-03-23T11:29:56.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.213+0000 7fc2a7e00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28c06a7a0 0x7fc28c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.213+0000 7fc2a7e00640 1 -- 172.21.15.62:0/1501233194 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2a00ab950 con 0x7fc294003970 2024-03-23T11:29:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.214+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28c06a7a0 0x7fc28c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.214+0000 7fc2b8dc0740 1 Processor -- start 2024-03-23T11:29:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.214+0000 7fc2b8dc0740 1 -- start start 2024-03-23T11:29:56.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28c06a7a0 0x7fc28c06cc60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00afb80 tx=0x7fc298008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af715b04f0 0x55af71567b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71568060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af71592800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af714d2f40 con 0x55af71566570 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af714d06d0 con 0x55af71562ca0 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af714cbfc0 con 0x55af715b04f0 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2a6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71568060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2a6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71568060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45928/0 (socket says 172.21.15.62:45928) 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2a6a00640 1 -- 172.21.15.62:0/2206686569 learned_addr learned my addr 172.21.15.62:0/2206686569 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2afe00640 1 --2- 172.21.15.62:0/2206686569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af71592800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.215+0000 7fc2af400640 1 --2- 172.21.15.62:0/2206686569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af715b04f0 0x55af71567b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.216+0000 7fc2a6a00640 1 -- 172.21.15.62:0/2206686569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af715b04f0 msgr2=0x55af71567b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.216+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/2206686569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af715b04f0 0x55af71567b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.216+0000 7fc2a6a00640 1 -- 172.21.15.62:0/2206686569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 msgr2=0x55af71592800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.216+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/2206686569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af71592800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.216+0000 7fc2a6a00640 1 -- 172.21.15.62:0/2206686569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af71493230 con 0x55af71566570 2024-03-23T11:29:56.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.216+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/2206686569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71568060 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fc298006de0 tx=0x7fc298014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c7ca2b0835d004 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.216+0000 7fc29fe00640 1 -- 172.21.15.62:0/2206686569 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298015a80 con 0x55af71566570 2024-03-23T11:29:56.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.216+0000 7fc29fe00640 1 -- 172.21.15.62:0/2206686569 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc298015be0 con 0x55af71566570 2024-03-23T11:29:56.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.216+0000 7fc29fe00640 1 -- 172.21.15.62:0/2206686569 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298021440 con 0x55af71566570 2024-03-23T11:29:56.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.217+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/2206686569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 msgr2=0x55af71568060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.217+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/2206686569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71568060 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fc298006de0 tx=0x7fc298014000 comp rx=0 tx=0).stop 2024-03-23T11:29:56.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.217+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/2206686569 shutdown_connections 2024-03-23T11:29:56.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.217+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/2206686569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71568060 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.217+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/2206686569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af71592800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.217+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/2206686569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af715b04f0 0x55af71567b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.217+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/2206686569 >> 172.21.15.62:0/2206686569 conn(0x55af71564f10 msgr2=0x55af715ad800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.217+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/2206686569 shutdown_connections 2024-03-23T11:29:56.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.217+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/2206686569 wait complete. 2024-03-23T11:29:56.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.218+0000 7fc2b8dc0740 1 -- start start 2024-03-23T11:29:56.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.218+0000 7fc2b8dc0740 1 Processor -- start 2024-03-23T11:29:56.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.219+0000 7f533f800640 1 Processor -- start 2024-03-23T11:29:56.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.219+0000 7f533f800640 1 -- start start 2024-03-23T11:29:56.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.219+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af71566570 0x55af71590c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.219+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af715b04f0 0x55af71577e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.219+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af7158db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.219+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af7146d320 con 0x55af7158ab70 2024-03-23T11:29:56.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.219+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af7146cfc0 con 0x55af715b04f0 2024-03-23T11:29:56.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.219+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af712340c0 con 0x55af71566570 2024-03-23T11:29:56.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2b8dc0740 10 client.0 _ll_register_callbacks cb 0x55af71492a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:29:56.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af7158db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af7158db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45938/0 (socket says 172.21.15.62:45938) 2024-03-23T11:29:56.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2afe00640 1 -- 172.21.15.62:0/803637590 learned_addr learned my addr 172.21.15.62:0/803637590 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2af400640 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af71566570 0x55af71590c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af715b04f0 0x55af71577e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2afe00640 1 -- 172.21.15.62:0/803637590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af71566570 msgr2=0x55af71590c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2afe00640 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af71566570 0x55af71590c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2afe00640 1 -- 172.21.15.62:0/803637590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af715b04f0 msgr2=0x55af71577e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2afe00640 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af715b04f0 0x55af71577e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2afe00640 1 -- 172.21.15.62:0/803637590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2980118e0 con 0x55af7158ab70 2024-03-23T11:29:56.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7fc2af400640 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af71566570 0x55af71590c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:29:56.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.220+0000 7f533f800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f53380d1bb0 0x7f53380b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7f533f800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53380b7d10 con 0x7f533815c990 2024-03-23T11:29:56.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7f533f800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53380b7100 con 0x7f53380b4860 2024-03-23T11:29:56.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7f533f800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53380b4cd0 con 0x7f53380d1bb0 2024-03-23T11:29:56.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7fc2afe00640 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af7158db00 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fc2b008a520 tx=0x7fc2a800e880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c58920ca458547e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7fc29cc00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:29:56.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7f533ce00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f533815c990 0x7f53380b82e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42672/0 (socket says 172.21.15.62:42672) 2024-03-23T11:29:56.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7f533ce00640 1 -- 172.21.15.62:0/3100730099 learned_addr learned my addr 172.21.15.62:0/3100730099 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:29:56.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7fc29cc00640 1 -- 172.21.15.62:0/803637590 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8011750 con 0x55af7158ab70 2024-03-23T11:29:56.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7f5337e00640 1 --2- 172.21.15.62:0/3100730099 >> v2:172.21.15.120:3300/0 conn(0x7f53380d1bb0 0x7f53380b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7fc29cc00640 1 -- 172.21.15.62:0/803637590 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2a80118b0 con 0x55af7158ab70 2024-03-23T11:29:56.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/803637590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55af714f5bb0 con 0x55af7158ab70 2024-03-23T11:29:56.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.221+0000 7f5337400640 1 -- 172.21.15.62:0/3100730099 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679498056 0 0) 0x7f53380b7d10 con 0x7f533815c990 2024-03-23T11:29:56.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7fc29cc00640 1 -- 172.21.15.62:0/803637590 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2a8011a70 con 0x55af7158ab70 2024-03-23T11:29:56.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7f5337400640 1 -- 172.21.15.62:0/3100730099 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5320003520 con 0x7f533815c990 2024-03-23T11:29:56.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7fc2b8dc0740 10 client.8640 fetch_fsmap learned FSMap version 78 2024-03-23T11:29:56.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/803637590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55af714ca640 con 0x55af7158ab70 2024-03-23T11:29:56.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7f5337e00640 1 -- 172.21.15.62:0/3100730099 >> v1:172.21.15.120:6789/0 conn(0x7f53380b4860 legacy=0x7f53380b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7f5337e00640 1 -- 172.21.15.62:0/3100730099 >> v1:172.21.15.62:6789/0 conn(0x7f533815c990 legacy=0x7f53380b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7f5337e00640 1 -- 172.21.15.62:0/3100730099 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53380b73e0 con 0x7f53380d1bb0 2024-03-23T11:29:56.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7fc29cc00640 1 -- 172.21.15.62:0/803637590 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55af714f5bb0 con 0x55af7158ab70 2024-03-23T11:29:56.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7fc29cc00640 1 -- 172.21.15.62:0/803637590 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc2a8011bd0 con 0x55af7158ab70 2024-03-23T11:29:56.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7fc2b8dc0740 10 client.8640 fetch_fsmap finished waiting for FSMap version 78 2024-03-23T11:29:56.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7fc2b8dc0740 10 client.8640 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:29:56.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7fc2b8dc0740 20 client.8640 populate_metadata read hostname 'smithi062' 2024-03-23T11:29:56.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55af71575a50 0x55af71575e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7fc2b8dc0740 4 client.8640 mds_command: new command op to 8271 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:29:56.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.222+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/803637590 --> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af711c0560 con 0x55af71575a50 2024-03-23T11:29:56.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.223+0000 7fc2af400640 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55af71575a50 0x55af71575e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.223+0000 7fc2af400640 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55af71575a50 0x55af71575e60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfe3d526adf1b65c server_cookie=45225fcb12f5df31 in_seq=0 out_seq=0 2024-03-23T11:29:56.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.223+0000 7fc29cc00640 10 client.8640 ms_handle_connect on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:56.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.223+0000 7f5337e00640 1 --2- 172.21.15.62:0/3100730099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380d1bb0 0x7f53380b4320 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f53240097a0 tx=0x7f532402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=525c28f9f79d2158 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc29cc00640 1 -- 172.21.15.62:0/803637590 <== mds.0 v2:172.21.15.62:6832/692790652 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55af711c0560 con 0x55af71575a50 2024-03-23T11:29:56.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc29cc00640 10 client.8640 handle_command_reply: tid=0 2024-03-23T11:29:56.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7f5337400640 1 -- 172.21.15.62:0/3100730099 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324030610 con 0x7f53380d1bb0 2024-03-23T11:29:56.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc2b8dc0740 1 client.8640 shutdown 2024-03-23T11:29:56.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7f5337400640 1 -- 172.21.15.62:0/3100730099 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5324030770 con 0x7f53380d1bb0 2024-03-23T11:29:56.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7f5337400640 1 -- 172.21.15.62:0/3100730099 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324030ae0 con 0x7f53380d1bb0 2024-03-23T11:29:56.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/803637590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 msgr2=0x55af7158db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af7158db00 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fc2b008a520 tx=0x7fc2a800e880 comp rx=0 tx=0).stop 2024-03-23T11:29:56.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/803637590 shutdown_connections 2024-03-23T11:29:56.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55af71575a50 0x55af71575e60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af7158db00 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af715b04f0 0x55af71577e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/803637590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af71566570 0x55af71590c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/803637590 >> 172.21.15.62:0/803637590 conn(0x55af71562ca0 msgr2=0x55af71566ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc29cc00640 0 client.8640 ms_handle_reset on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:56.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/803637590 shutdown_connections 2024-03-23T11:29:56.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.224+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/803637590 wait complete. 2024-03-23T11:29:56.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.225+0000 7f533f800640 1 -- 172.21.15.62:0/3100730099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380d1bb0 msgr2=0x7f53380b4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.225+0000 7f533f800640 1 --2- 172.21.15.62:0/3100730099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380d1bb0 0x7f53380b4320 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f53240097a0 tx=0x7f532402b960 comp rx=0 tx=0).stop 2024-03-23T11:29:56.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.225+0000 7f533f800640 1 -- 172.21.15.62:0/3100730099 shutdown_connections 2024-03-23T11:29:56.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.225+0000 7f533f800640 1 --2- 172.21.15.62:0/3100730099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380d1bb0 0x7f53380b4320 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.225+0000 7fc2b8dc0740 20 client.8640 trim_cache size 0 max 16384 2024-03-23T11:29:56.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.225+0000 7f533f800640 1 -- 172.21.15.62:0/3100730099 >> 172.21.15.62:0/3100730099 conn(0x7f53380717b0 msgr2=0x7f5338071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.225+0000 7f533f800640 1 -- 172.21.15.62:0/3100730099 shutdown_connections 2024-03-23T11:29:56.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.225+0000 7f533f800640 1 -- 172.21.15.62:0/3100730099 wait complete. 2024-03-23T11:29:56.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.228+0000 7f533f800640 1 Processor -- start 2024-03-23T11:29:56.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.228+0000 7f533f800640 1 -- start start 2024-03-23T11:29:56.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533f800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53380b4860 0x7f53380c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533f800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380d1bb0 0x7f53381501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533f800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f533815c990 0x7f53381f6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533f800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53380c8960 con 0x7f53380b4860 2024-03-23T11:29:56.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533f800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5338150710 con 0x7f53380d1bb0 2024-03-23T11:29:56.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533f800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5338152bb0 con 0x7f533815c990 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53380b4860 0x7f53380c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53380b4860 0x7f53380c8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45948/0 (socket says 172.21.15.62:45948) 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533ce00640 1 -- 172.21.15.62:0/4175894408 learned_addr learned my addr 172.21.15.62:0/4175894408 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f5337e00640 1 --2- 172.21.15.62:0/4175894408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380d1bb0 0x7f53381501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533d800640 1 --2- 172.21.15.62:0/4175894408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f533815c990 0x7f53381f6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533ce00640 1 -- 172.21.15.62:0/4175894408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f533815c990 msgr2=0x7f53381f6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533ce00640 1 --2- 172.21.15.62:0/4175894408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f533815c990 0x7f53381f6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533ce00640 1 -- 172.21.15.62:0/4175894408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380d1bb0 msgr2=0x7f53381501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533ce00640 1 --2- 172.21.15.62:0/4175894408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380d1bb0 0x7f53381501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f533ce00640 1 -- 172.21.15.62:0/4175894408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5324009400 con 0x7f53380b4860 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.230+0000 7f5337e00640 1 --2- 172.21.15.62:0/4175894408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53380d1bb0 0x7f53381501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:56.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.231+0000 7f533d800640 1 --2- 172.21.15.62:0/4175894408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f533815c990 0x7f53381f6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:56.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.231+0000 7f533ce00640 1 --2- 172.21.15.62:0/4175894408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53380b4860 0x7f53380c8d80 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f532c00c410 tx=0x7f532c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.231+0000 7f5334c00640 1 -- 172.21.15.62:0/4175894408 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c005ce0 con 0x7f53380b4860 2024-03-23T11:29:56.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.231+0000 7f5334c00640 1 -- 172.21.15.62:0/4175894408 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532c005e40 con 0x7f53380b4860 2024-03-23T11:29:56.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.231+0000 7f5334c00640 1 -- 172.21.15.62:0/4175894408 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c0072f0 con 0x7f53380b4860 2024-03-23T11:29:56.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.231+0000 7f533f800640 1 -- 172.21.15.62:0/4175894408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53380c7d20 con 0x7f53380b4860 2024-03-23T11:29:56.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.231+0000 7f533f800640 1 -- 172.21.15.62:0/4175894408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53380c82a0 con 0x7f53380b4860 2024-03-23T11:29:56.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.232+0000 7f533f800640 1 -- 172.21.15.62:0/4175894408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f5338066bb0 con 0x7f53380b4860 2024-03-23T11:29:56.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.234+0000 7f5334c00640 1 -- 172.21.15.62:0/4175894408 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f532c007510 con 0x7f53380b4860 2024-03-23T11:29:56.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.235+0000 7f5334c00640 1 --2- 172.21.15.62:0/4175894408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f531806a7a0 0x7f531806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.235+0000 7f5334c00640 1 -- 172.21.15.62:0/4175894408 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f532c088a90 con 0x7f53380b4860 2024-03-23T11:29:56.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.235+0000 7f5334c00640 1 -- 172.21.15.62:0/4175894408 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+504 (secure 0 0 0) 0x7f532c0892d0 con 0x7f53380b4860 2024-03-23T11:29:56.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.236+0000 7f5337e00640 1 --2- 172.21.15.62:0/4175894408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f531806a7a0 0x7f531806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.236+0000 7f533f800640 1 -- 172.21.15.62:0/4175894408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7f53380b23a0 con 0x7f53380b4860 2024-03-23T11:29:56.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.237+0000 7f5337e00640 1 --2- 172.21.15.62:0/4175894408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f531806a7a0 0x7f531806cc60 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5324036000 tx=0x7f5324008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.245+0000 7fc2b8dc0740 1 Processor -- start 2024-03-23T11:29:56.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.245+0000 7fc2b8dc0740 1 -- start start 2024-03-23T11:29:56.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.245+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af71567ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71588c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af7158ab70 0x55af71589150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af7158ded0 con 0x55af71566570 2024-03-23T11:29:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af715ac6c0 con 0x55af71562ca0 2024-03-23T11:29:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af71569ee0 con 0x55af7158ab70 2024-03-23T11:29:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2a6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71588c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2a6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71588c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45962/0 (socket says 172.21.15.62:45962) 2024-03-23T11:29:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2a6a00640 1 -- 172.21.15.62:0/3358528685 learned_addr learned my addr 172.21.15.62:0/3358528685 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2af400640 1 --2- 172.21.15.62:0/3358528685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af71567ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2afe00640 1 --2- 172.21.15.62:0/3358528685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af7158ab70 0x55af71589150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2a6a00640 1 -- 172.21.15.62:0/3358528685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af7158ab70 msgr2=0x55af71589150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/3358528685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af7158ab70 0x55af71589150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2a6a00640 1 -- 172.21.15.62:0/3358528685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 msgr2=0x55af71567ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/3358528685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af71567ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.246+0000 7fc2a6a00640 1 -- 172.21.15.62:0/3358528685 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af715b1270 con 0x55af71566570 2024-03-23T11:29:56.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.247+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/3358528685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71588c10 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fc298011b40 tx=0x7fc298018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac422c5727d41408 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.247+0000 7fc2a4c00640 1 -- 172.21.15.62:0/3358528685 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29802f070 con 0x55af71566570 2024-03-23T11:29:56.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.247+0000 7fc2a4c00640 1 -- 172.21.15.62:0/3358528685 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc298011700 con 0x55af71566570 2024-03-23T11:29:56.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.247+0000 7fc2a4c00640 1 -- 172.21.15.62:0/3358528685 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298015d50 con 0x55af71566570 2024-03-23T11:29:56.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.248+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3358528685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 msgr2=0x55af71588c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.248+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/3358528685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71588c10 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fc298011b40 tx=0x7fc298018000 comp rx=0 tx=0).stop 2024-03-23T11:29:56.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.248+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3358528685 shutdown_connections 2024-03-23T11:29:56.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.248+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/3358528685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af71566570 0x55af71588c10 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.248+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/3358528685 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af7158ab70 0x55af71589150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.248+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/3358528685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af71567ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.248+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3358528685 >> 172.21.15.62:0/3358528685 conn(0x55af71567a00 msgr2=0x55af715b53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.248+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3358528685 shutdown_connections 2024-03-23T11:29:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.248+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3358528685 wait complete. 2024-03-23T11:29:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.249+0000 7fc2b8dc0740 1 -- start start 2024-03-23T11:29:56.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.249+0000 7fc2b8dc0740 1 Processor -- start 2024-03-23T11:29:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.250+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af715aba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.250+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af71566570 0x55af71574310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.250+0000 7fc2b8dc0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af715c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.250+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af715ab5e0 con 0x55af7158ab70 2024-03-23T11:29:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.250+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af715af9b0 con 0x55af71562ca0 2024-03-23T11:29:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.250+0000 7fc2b8dc0740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af71568870 con 0x55af71566570 2024-03-23T11:29:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.250+0000 7fc2b8dc0740 10 client.0 _ll_register_callbacks cb 0x7fc2980095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:29:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.250+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af715c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.250+0000 7fc2afe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af715c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45974/0 (socket says 172.21.15.62:45974) 2024-03-23T11:29:56.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.250+0000 7fc2afe00640 1 -- 172.21.15.62:0/3477797413 learned_addr learned my addr 172.21.15.62:0/3477797413 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.250+0000 7fc2af400640 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af715aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.251+0000 7fc2a6a00640 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af71566570 0x55af71574310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.251+0000 7fc2afe00640 1 -- 172.21.15.62:0/3477797413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af71566570 msgr2=0x55af71574310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.251+0000 7fc2afe00640 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af71566570 0x55af71574310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.251+0000 7fc2afe00640 1 -- 172.21.15.62:0/3477797413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 msgr2=0x55af715aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.251+0000 7fc2afe00640 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af715aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.251+0000 7fc2afe00640 1 -- 172.21.15.62:0/3477797413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af71493230 con 0x55af7158ab70 2024-03-23T11:29:56.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.251+0000 7fc2afe00640 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af715c61a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8013b40 tx=0x7fc2a8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fbd2ee4bbb5eb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.251+0000 7fc29cc00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:29:56.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.251+0000 7fc29cc00640 1 -- 172.21.15.62:0/3477797413 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8027070 con 0x55af7158ab70 2024-03-23T11:29:56.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.251+0000 7fc29cc00640 1 -- 172.21.15.62:0/3477797413 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2a8001680 con 0x55af7158ab70 2024-03-23T11:29:56.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.251+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3477797413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55af711d1860 con 0x55af7158ab70 2024-03-23T11:29:56.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.252+0000 7fc29cc00640 1 -- 172.21.15.62:0/3477797413 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2a8011760 con 0x55af7158ab70 2024-03-23T11:29:56.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.252+0000 7fc2b8dc0740 10 client.8652 fetch_fsmap learned FSMap version 78 2024-03-23T11:29:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.252+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3477797413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55af714fef00 con 0x55af7158ab70 2024-03-23T11:29:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.252+0000 7fc29cc00640 1 -- 172.21.15.62:0/3477797413 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55af711d1860 con 0x55af7158ab70 2024-03-23T11:29:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.252+0000 7fc29cc00640 1 -- 172.21.15.62:0/3477797413 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc2a80118c0 con 0x55af7158ab70 2024-03-23T11:29:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.252+0000 7fc2b8dc0740 10 client.8652 fetch_fsmap finished waiting for FSMap version 78 2024-03-23T11:29:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.252+0000 7fc2b8dc0740 10 client.8652 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:29:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.252+0000 7fc2b8dc0740 20 client.8652 populate_metadata read hostname 'smithi062' 2024-03-23T11:29:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.252+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55af715b04f0 0x55af715b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.252+0000 7fc2b8dc0740 4 client.8652 mds_command: new command op to 8271 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-03-23T11:29:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.252+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3477797413 --> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) v1 -- 0x55af71224eb0 con 0x55af715b04f0 2024-03-23T11:29:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.253+0000 7fc2af400640 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55af715b04f0 0x55af715b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.253+0000 7fc29cc00640 10 client.8652 ms_handle_connect on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:56.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.253+0000 7fc2af400640 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55af715b04f0 0x55af715b25c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a869c340cd9267c server_cookie=4248ed1cfa6babf9 in_seq=0 out_seq=0 2024-03-23T11:29:56.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.254+0000 7fc29cc00640 1 -- 172.21.15.62:0/3477797413 <== mds.0 v2:172.21.15.62:6832/692790652 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55af71224eb0 con 0x55af715b04f0 2024-03-23T11:29:56.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.254+0000 7fc29cc00640 10 client.8652 handle_command_reply: tid=0 2024-03-23T11:29:56.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.254+0000 7fc2b8dc0740 1 client.8652 shutdown 2024-03-23T11:29:56.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3477797413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 msgr2=0x55af715c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af715c61a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8013b40 tx=0x7fc2a8000bc0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3477797413 shutdown_connections 2024-03-23T11:29:56.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x55af715b04f0 0x55af715b25c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af7158ab70 0x55af715c61a0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55af71566570 0x55af71574310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc2b8dc0740 1 --2- 172.21.15.62:0/3477797413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55af71562ca0 0x55af715aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3477797413 >> 172.21.15.62:0/3477797413 conn(0x55af71567a00 msgr2=0x55af71588c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc29cc00640 0 client.8652 ms_handle_reset on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:56.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3477797413 shutdown_connections 2024-03-23T11:29:56.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc2b8dc0740 1 -- 172.21.15.62:0/3477797413 wait complete. 2024-03-23T11:29:56.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.255+0000 7fc2b8dc0740 20 client.8652 trim_cache size 0 max 16384 2024-03-23T11:29:56.251 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:29:56.251 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-03-23T11:29:56.251 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "58dbe2e2-33e9-412a-b3eb-7d05a2a40ee2", 2024-03-23T11:29:56.251 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-03-23T11:29:56.251 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:29:56.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 -- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28c06a7a0 msgr2=0x7fc28c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28c06a7a0 0x7fc28c06cc60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00afb80 tx=0x7fc298008040 comp rx=0 tx=0).stop 2024-03-23T11:29:56.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 -- 172.21.15.62:0/1501233194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 msgr2=0x7fc2b0159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 0x7fc2b0159bf0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fc2a002c600 tx=0x7fc2a0009750 comp rx=0 tx=0).stop 2024-03-23T11:29:56.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 -- 172.21.15.62:0/1501233194 shutdown_connections 2024-03-23T11:29:56.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28c06a7a0 0x7fc28c06cc60 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b00cb690 0x7fc2b00cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2b015ef50 0x7fc2b015f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 --2- 172.21.15.62:0/1501233194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc294003970 0x7fc2b0159bf0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 -- 172.21.15.62:0/1501233194 >> 172.21.15.62:0/1501233194 conn(0x7fc2b0089f60 msgr2=0x7fc2b00b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 -- 172.21.15.62:0/1501233194 shutdown_connections 2024-03-23T11:29:56.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.256+0000 7fc2b4a00640 1 -- 172.21.15.62:0/1501233194 wait complete. 2024-03-23T11:29:56.264 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-23T11:29:56.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.390+0000 7f5334c00640 1 -- 172.21.15.62:0/4175894408 <== mon.0 v2:172.21.15.62:3300/0 7 ==== log(4 entries from seq 1 at 2024-03-23T11:29:56.254696+0000) v1 ==== 695+0+0 (secure 0 0 0) 0x7f532c088e40 con 0x7f53380b4860 2024-03-23T11:29:56.381 INFO:teuthology.orchestra.run.smithi062.stdout: cluster: 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: id: e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: health: HEALTH_OK 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: services: 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: mon: 3 daemons, quorum a,b,c (age 11m) 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: mgr: x(active, since 11m), standbys: y 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: mds: 1/1 daemons up, 3 standby 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: osd: 8 osds: 8 up (since 11m), 8 in (since 11m) 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: data: 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: volumes: 1/1 healthy 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: pools: 3 pools, 97 pgs 2024-03-23T11:29:56.382 INFO:teuthology.orchestra.run.smithi062.stdout: objects: 30 objects, 19 MiB 2024-03-23T11:29:56.383 INFO:teuthology.orchestra.run.smithi062.stdout: usage: 290 MiB used, 720 GiB / 720 GiB avail 2024-03-23T11:29:56.383 INFO:teuthology.orchestra.run.smithi062.stdout: pgs: 97 active+clean 2024-03-23T11:29:56.383 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:56.383 INFO:teuthology.orchestra.run.smithi062.stdout: io: 2024-03-23T11:29:56.383 INFO:teuthology.orchestra.run.smithi062.stdout: client: 1.0 MiB/s wr, 0 op/s rd, 1 op/s wr 2024-03-23T11:29:56.383 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:56.383 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:29:56.383 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:29:56.254696+0000 mds.a [INF] scrub queued for path: / 2024-03-23T11:29:56.383 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:29:56.254700+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-23T11:29:56.383 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:29:56.254705+0000 mds.a [INF] scrub summary: active paths [/] 2024-03-23T11:29:56.383 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:29:56.256792+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-23T11:29:56.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.463+0000 7fe9b9600640 1 Processor -- start 2024-03-23T11:29:56.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.463+0000 7fe9b9600640 1 -- start start 2024-03-23T11:29:56.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.465+0000 7fe9b9600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b41571e0 0x7fe9b40ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.465+0000 7fe9b9600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe9b41613a0 0x7fe9b415e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.465+0000 7fe9b9600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b406cf40 con 0x7fe9b40ade70 2024-03-23T11:29:56.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.465+0000 7fe9b9600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b41575c0 con 0x7fe9b41613a0 2024-03-23T11:29:56.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.465+0000 7fe9b9600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b406c3a0 con 0x7fe9b41571e0 2024-03-23T11:29:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.466+0000 7fe9b3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe9b41613a0 0x7fe9b415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.466+0000 7fe9b3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe9b41613a0 0x7fe9b415e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:52960/0 (socket says 172.21.15.62:52960) 2024-03-23T11:29:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.466+0000 7fe9b3400640 1 -- 172.21.15.62:0/3181005856 learned_addr learned my addr 172.21.15.62:0/3181005856 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.466+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/3181005856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b41571e0 0x7fe9b40ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.467+0000 7fe9b3400640 1 -- 172.21.15.62:0/3181005856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b41571e0 msgr2=0x7fe9b40ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.467+0000 7fe9b3400640 1 --2- 172.21.15.62:0/3181005856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b41571e0 0x7fe9b40ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.467+0000 7fe9b3400640 1 -- 172.21.15.62:0/3181005856 >> v1:172.21.15.120:6789/0 conn(0x7fe9b40ade70 legacy=0x7fe9b406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.467+0000 7fe9b3400640 1 -- 172.21.15.62:0/3181005856 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b406c610 con 0x7fe9b41613a0 2024-03-23T11:29:56.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.468+0000 7fe9b3400640 1 --2- 172.21.15.62:0/3181005856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b41613a0 0x7fe9b415e280 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009e50 tx=0x7fe9ac02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68aeebb0c52d2624 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.468+0000 7fe9b2000640 1 -- 172.21.15.62:0/3181005856 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac02ca80 con 0x7fe9b41613a0 2024-03-23T11:29:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.468+0000 7fe9b2000640 1 -- 172.21.15.62:0/3181005856 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac02cbe0 con 0x7fe9b41613a0 2024-03-23T11:29:56.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.468+0000 7fe9b2000640 1 -- 172.21.15.62:0/3181005856 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac0343f0 con 0x7fe9b41613a0 2024-03-23T11:29:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.469+0000 7fe9b9600640 1 -- 172.21.15.62:0/3181005856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b41613a0 msgr2=0x7fe9b415e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.469+0000 7fe9b9600640 1 --2- 172.21.15.62:0/3181005856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b41613a0 0x7fe9b415e280 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009e50 tx=0x7fe9ac02bee0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.469+0000 7fe9b9600640 1 -- 172.21.15.62:0/3181005856 shutdown_connections 2024-03-23T11:29:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.469+0000 7fe9b9600640 1 --2- 172.21.15.62:0/3181005856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b41613a0 0x7fe9b415e280 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.469+0000 7fe9b9600640 1 --2- 172.21.15.62:0/3181005856 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b41571e0 0x7fe9b40ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.469+0000 7fe9b9600640 1 -- 172.21.15.62:0/3181005856 >> 172.21.15.62:0/3181005856 conn(0x7fe9b40b02e0 msgr2=0x7fe9b40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.469+0000 7fe9b9600640 1 -- 172.21.15.62:0/3181005856 shutdown_connections 2024-03-23T11:29:56.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.469+0000 7fe9b9600640 1 -- 172.21.15.62:0/3181005856 wait complete. 2024-03-23T11:29:56.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.471+0000 7fe9b9600640 1 Processor -- start 2024-03-23T11:29:56.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.472+0000 7fe9b9600640 1 -- start start 2024-03-23T11:29:56.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.473+0000 7fe9b9600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b40ade70 0x7fe9b41f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.473+0000 7fe9b9600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b41571e0 0x7fe9b41f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.473+0000 7fe9b9600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9b41613a0 0x7fe9b40b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.473+0000 7fe9b9600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b4134c80 con 0x7fe9b41613a0 2024-03-23T11:29:56.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.473+0000 7fe9b9600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b4134ed0 con 0x7fe9b41571e0 2024-03-23T11:29:56.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.473+0000 7fe9b9600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b41351b0 con 0x7fe9b40ade70 2024-03-23T11:29:56.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.473+0000 7fe9b3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9b41613a0 0x7fe9b40b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.473+0000 7fe9b3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9b41613a0 0x7fe9b40b1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45980/0 (socket says 172.21.15.62:45980) 2024-03-23T11:29:56.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.473+0000 7fe9b3400640 1 -- 172.21.15.62:0/3209953052 learned_addr learned my addr 172.21.15.62:0/3209953052 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.473+0000 7fe9abe00640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b41571e0 0x7fe9b41f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.473+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b40ade70 0x7fe9b41f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.474+0000 7fe9b3400640 1 -- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b40ade70 msgr2=0x7fe9b41f17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.474+0000 7fe9b3400640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b40ade70 0x7fe9b41f17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.474+0000 7fe9b3400640 1 -- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b41571e0 msgr2=0x7fe9b41f1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.474+0000 7fe9b3400640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b41571e0 0x7fe9b41f1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.474+0000 7fe9b3400640 1 -- 172.21.15.62:0/3209953052 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac009b30 con 0x7fe9b41613a0 2024-03-23T11:29:56.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.474+0000 7fe9b3400640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9b41613a0 0x7fe9b40b1fb0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009e50 tx=0x7fe9ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.474+0000 7fe9aaa00640 1 -- 172.21.15.62:0/3209953052 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac009510 con 0x7fe9b41613a0 2024-03-23T11:29:56.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.474+0000 7fe9aaa00640 1 -- 172.21.15.62:0/3209953052 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac03e450 con 0x7fe9b41613a0 2024-03-23T11:29:56.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.474+0000 7fe9b9600640 1 -- 172.21.15.62:0/3209953052 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b4135490 con 0x7fe9b41613a0 2024-03-23T11:29:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.474+0000 7fe9aaa00640 1 -- 172.21.15.62:0/3209953052 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac03e800 con 0x7fe9b41613a0 2024-03-23T11:29:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.475+0000 7fe9b9600640 1 -- 172.21.15.62:0/3209953052 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b4158810 con 0x7fe9b41613a0 2024-03-23T11:29:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.477+0000 7fe9aaa00640 1 -- 172.21.15.62:0/3209953052 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe9ac02cd50 con 0x7fe9b41613a0 2024-03-23T11:29:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.478+0000 7fe9aaa00640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe99006a7d0 0x7fe99006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.478+0000 7fe9aaa00640 1 -- 172.21.15.62:0/3209953052 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9ac0ab850 con 0x7fe9b41613a0 2024-03-23T11:29:56.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.478+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe99006a7d0 0x7fe99006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.478+0000 7fe9bc228740 1 Processor -- start 2024-03-23T11:29:56.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.479+0000 7fe9bc228740 1 -- start start 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.479+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x557993435b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.479+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 0x55799340da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.479+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe99006a7d0 0x7fe99006cc90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4009a30 tx=0x7fe9a4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.479+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x55799340df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9bc228740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579933c9cb0 con 0x55799342dcf0 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9bc228740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557993378500 con 0x55799342d910 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9bc228740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557993375c90 con 0x557993436280 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9b3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x55799340df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9b3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x55799340df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45986/0 (socket says 172.21.15.62:45986) 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9b3400640 1 -- 172.21.15.62:0/1990085657 learned_addr learned my addr 172.21.15.62:0/1990085657 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/1990085657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x557993435b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9abe00640 1 --2- 172.21.15.62:0/1990085657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 0x55799340da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9b3400640 1 -- 172.21.15.62:0/1990085657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 msgr2=0x55799340da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9b3400640 1 --2- 172.21.15.62:0/1990085657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 0x55799340da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9b3400640 1 -- 172.21.15.62:0/1990085657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 msgr2=0x557993435b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9b3400640 1 --2- 172.21.15.62:0/1990085657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x557993435b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.480+0000 7fe9b3400640 1 -- 172.21.15.62:0/1990085657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557993323360 con 0x55799342dcf0 2024-03-23T11:29:56.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.481+0000 7fe9b3400640 1 --2- 172.21.15.62:0/1990085657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x55799340df80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac02c4f0 tx=0x7fe9ac07a3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7055ab5bcf755811 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.481+0000 7fe9a3400640 1 -- 172.21.15.62:0/1990085657 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac092070 con 0x55799342dcf0 2024-03-23T11:29:56.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.481+0000 7fe9a3400640 1 -- 172.21.15.62:0/1990085657 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac0ab040 con 0x55799342dcf0 2024-03-23T11:29:56.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.481+0000 7fe9a3400640 1 -- 172.21.15.62:0/1990085657 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac07a890 con 0x55799342dcf0 2024-03-23T11:29:56.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.482+0000 7fe9bc228740 1 -- 172.21.15.62:0/1990085657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 msgr2=0x55799340df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.482+0000 7fe9bc228740 1 --2- 172.21.15.62:0/1990085657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x55799340df80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac02c4f0 tx=0x7fe9ac07a3d0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.482+0000 7fe9bc228740 1 -- 172.21.15.62:0/1990085657 shutdown_connections 2024-03-23T11:29:56.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.482+0000 7fe9bc228740 1 --2- 172.21.15.62:0/1990085657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x55799340df80 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.482+0000 7fe9bc228740 1 --2- 172.21.15.62:0/1990085657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 0x55799340da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.482+0000 7fe9bc228740 1 --2- 172.21.15.62:0/1990085657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x557993435b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.482+0000 7fe9bc228740 1 -- 172.21.15.62:0/1990085657 >> 172.21.15.62:0/1990085657 conn(0x5579933d5b40 msgr2=0x557993453720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.482+0000 7fe9bc228740 1 -- 172.21.15.62:0/1990085657 shutdown_connections 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.482+0000 7fe9bc228740 1 -- 172.21.15.62:0/1990085657 wait complete. 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.483+0000 7fe9bc228740 1 -- start start 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.483+0000 7fe9bc228740 1 Processor -- start 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.484+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342d910 0x557993455c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.484+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342dcf0 0x55799342bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.484+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 0x557993451700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.484+0000 7fe9bc228740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557993371580 con 0x55799342d910 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.484+0000 7fe9bc228740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557993312a50 con 0x55799342dcf0 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.484+0000 7fe9bc228740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5579933126f0 con 0x557993436280 2024-03-23T11:29:56.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.484+0000 7fe9bc228740 10 client.0 _ll_register_callbacks cb 0x557993361ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.484+0000 7fe9b2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342d910 0x557993455c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.484+0000 7fe9b2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342d910 0x557993455c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45994/0 (socket says 172.21.15.62:45994) 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.484+0000 7fe9b2a00640 1 -- 172.21.15.62:0/3166947549 learned_addr learned my addr 172.21.15.62:0/3166947549 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.484+0000 7fe9abe00640 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342dcf0 0x55799342bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe9b2a00640 1 -- 172.21.15.62:0/3166947549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 msgr2=0x557993451700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe9b3400640 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 0x557993451700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 0x557993451700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe9b2a00640 1 -- 172.21.15.62:0/3166947549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342dcf0 msgr2=0x55799342bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342dcf0 0x55799342bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe9b2a00640 1 -- 172.21.15.62:0/3166947549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b4158810 con 0x55799342d910 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe9abe00640 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342dcf0 0x55799342bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342d910 0x557993455c20 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4015720 tx=0x7fe9a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41ca3bcd81423a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe997e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:29:56.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe997e00640 1 -- 172.21.15.62:0/3166947549 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40147b0 con 0x55799342d910 2024-03-23T11:29:56.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe997e00640 1 -- 172.21.15.62:0/3166947549 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9a4014910 con 0x55799342d910 2024-03-23T11:29:56.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.485+0000 7fe9bc228740 1 -- 172.21.15.62:0/3166947549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55799339bfe0 con 0x55799342d910 2024-03-23T11:29:56.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe997e00640 1 -- 172.21.15.62:0/3166947549 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9a4014ad0 con 0x55799342d910 2024-03-23T11:29:56.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe9bc228740 10 client.8661 fetch_fsmap learned FSMap version 78 2024-03-23T11:29:56.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe9bc228740 1 -- 172.21.15.62:0/3166947549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55799336fc00 con 0x55799342d910 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe997e00640 1 -- 172.21.15.62:0/3166947549 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55799339bfe0 con 0x55799342d910 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe997e00640 1 -- 172.21.15.62:0/3166947549 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe9a4014c30 con 0x55799342d910 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe9bc228740 10 client.8661 fetch_fsmap finished waiting for FSMap version 78 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe9bc228740 10 client.8661 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe9bc228740 20 client.8661 populate_metadata read hostname 'smithi062' 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x557993439d90 0x55799343a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe9bc228740 4 client.8661 mds_command: new command op to 8271 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe9bc228740 1 -- 172.21.15.62:0/3166947549 --> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557993364240 con 0x557993439d90 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.486+0000 7fe9abe00640 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x557993439d90 0x55799343a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.487+0000 7fe9abe00640 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x557993439d90 0x55799343a1a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7219984114510312 server_cookie=8260a2dfa349c462 in_seq=0 out_seq=0 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.487+0000 7fe997e00640 10 client.8661 ms_handle_connect on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe997e00640 1 -- 172.21.15.62:0/3166947549 <== mds.0 v2:172.21.15.62:6832/692790652 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x557993364240 con 0x557993439d90 2024-03-23T11:29:56.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe997e00640 10 client.8661 handle_command_reply: tid=0 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 1 client.8661 shutdown 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 1 -- 172.21.15.62:0/3166947549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342d910 msgr2=0x557993455c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342d910 0x557993455c20 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4015720 tx=0x7fe9a40046c0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 1 -- 172.21.15.62:0/3166947549 shutdown_connections 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x557993439d90 0x55799343a1a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 0x557993451700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342dcf0 0x55799342bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3166947549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342d910 0x557993455c20 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe997e00640 0 client.8661 ms_handle_reset on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 1 -- 172.21.15.62:0/3166947549 >> 172.21.15.62:0/3166947549 conn(0x5579933d5b40 msgr2=0x557993456ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 1 -- 172.21.15.62:0/3166947549 shutdown_connections 2024-03-23T11:29:56.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 1 -- 172.21.15.62:0/3166947549 wait complete. 2024-03-23T11:29:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.488+0000 7fe9bc228740 20 client.8661 trim_cache size 0 max 16384 2024-03-23T11:29:56.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.504+0000 7fe9bc228740 1 Processor -- start 2024-03-23T11:29:56.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.504+0000 7fe9bc228740 1 -- start start 2024-03-23T11:29:56.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.505+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x557993439060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.505+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55799342dcf0 0x557993434210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.505+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557993436280 0x5579934622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.505+0000 7fe9bc228740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55799306ef90 con 0x557993436280 2024-03-23T11:29:56.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.505+0000 7fe9bc228740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55799340e8d0 con 0x55799342d910 2024-03-23T11:29:56.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.505+0000 7fe9bc228740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55799343afc0 con 0x55799342dcf0 2024-03-23T11:29:56.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.505+0000 7fe9b3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557993436280 0x5579934622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.505+0000 7fe9b3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557993436280 0x5579934622a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46000/0 (socket says 172.21.15.62:46000) 2024-03-23T11:29:56.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.505+0000 7fe9b3400640 1 -- 172.21.15.62:0/245875047 learned_addr learned my addr 172.21.15.62:0/245875047 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.505+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/245875047 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x557993439060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9abe00640 1 --2- 172.21.15.62:0/245875047 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55799342dcf0 0x557993434210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9b3400640 1 -- 172.21.15.62:0/245875047 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55799342dcf0 msgr2=0x557993434210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9b3400640 1 --2- 172.21.15.62:0/245875047 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55799342dcf0 0x557993434210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9b3400640 1 -- 172.21.15.62:0/245875047 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 msgr2=0x557993439060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9b3400640 1 --2- 172.21.15.62:0/245875047 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x557993439060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9b3400640 1 -- 172.21.15.62:0/245875047 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557993311c60 con 0x557993436280 2024-03-23T11:29:56.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/245875047 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x557993439060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:56.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9abe00640 1 --2- 172.21.15.62:0/245875047 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55799342dcf0 0x557993434210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:29:56.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9b3400640 1 --2- 172.21.15.62:0/245875047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557993436280 0x5579934622a0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac082750 tx=0x7fe9ac07d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ac1c4f1e405da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9a3e00640 1 -- 172.21.15.62:0/245875047 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac096070 con 0x557993436280 2024-03-23T11:29:56.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9a3e00640 1 -- 172.21.15.62:0/245875047 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac0a7070 con 0x557993436280 2024-03-23T11:29:56.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.506+0000 7fe9a3e00640 1 -- 172.21.15.62:0/245875047 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac092040 con 0x557993436280 2024-03-23T11:29:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.507+0000 7fe9bc228740 1 -- 172.21.15.62:0/245875047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557993436280 msgr2=0x5579934622a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.507+0000 7fe9bc228740 1 --2- 172.21.15.62:0/245875047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557993436280 0x5579934622a0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac082750 tx=0x7fe9ac07d000 comp rx=0 tx=0).stop 2024-03-23T11:29:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.507+0000 7fe9bc228740 1 -- 172.21.15.62:0/245875047 shutdown_connections 2024-03-23T11:29:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.507+0000 7fe9bc228740 1 --2- 172.21.15.62:0/245875047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557993436280 0x5579934622a0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.507+0000 7fe9bc228740 1 --2- 172.21.15.62:0/245875047 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55799342dcf0 0x557993434210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.507+0000 7fe9bc228740 1 --2- 172.21.15.62:0/245875047 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x557993439060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.507+0000 7fe9bc228740 1 -- 172.21.15.62:0/245875047 >> 172.21.15.62:0/245875047 conn(0x5579933d5b40 msgr2=0x557993434e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.507+0000 7fe9bc228740 1 -- 172.21.15.62:0/245875047 shutdown_connections 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.507+0000 7fe9bc228740 1 -- 172.21.15.62:0/245875047 wait complete. 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.508+0000 7fe9bc228740 1 -- start start 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.508+0000 7fe9bc228740 1 Processor -- start 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.509+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x5579934546c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.509+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x5579934b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.509+0000 7fe9bc228740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 0x5579934507b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.509+0000 7fe9bc228740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557993455090 con 0x55799342dcf0 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.509+0000 7fe9bc228740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557993456920 con 0x55799342d910 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.509+0000 7fe9bc228740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5579934b8710 con 0x557993436280 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.509+0000 7fe9bc228740 10 client.0 _ll_register_callbacks cb 0x7fe9ac0092e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.509+0000 7fe9abe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x5579934b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.509+0000 7fe9abe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x5579934b8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46002/0 (socket says 172.21.15.62:46002) 2024-03-23T11:29:56.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.509+0000 7fe9abe00640 1 -- 172.21.15.62:0/3429409615 learned_addr learned my addr 172.21.15.62:0/3429409615 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:29:56.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe9abe00640 1 -- 172.21.15.62:0/3429409615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 msgr2=0x5579934507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe9abe00640 1 --2- 172.21.15.62:0/3429409615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 0x5579934507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe9abe00640 1 -- 172.21.15.62:0/3429409615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 msgr2=0x5579934546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:29:56.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe9abe00640 1 --2- 172.21.15.62:0/3429409615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x5579934546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe9abe00640 1 -- 172.21.15.62:0/3429409615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557993323360 con 0x55799342dcf0 2024-03-23T11:29:56.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe9abe00640 1 --2- 172.21.15.62:0/3429409615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x5579934b8ba0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe99c000fd0 tx=0x7fe99c018d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c39254226968dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe997e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe997e00640 1 -- 172.21.15.62:0/3429409615 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c004e90 con 0x55799342dcf0 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe997e00640 1 -- 172.21.15.62:0/3429409615 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe99c022ac0 con 0x55799342dcf0 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe9bc228740 1 -- 172.21.15.62:0/3429409615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe9ac0ab040 con 0x55799342dcf0 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe997e00640 1 -- 172.21.15.62:0/3429409615 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe99c022c80 con 0x55799342dcf0 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe9bc228740 10 client.8667 fetch_fsmap learned FSMap version 78 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.510+0000 7fe9bc228740 1 -- 172.21.15.62:0/3429409615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x557993337170 con 0x55799342dcf0 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.511+0000 7fe997e00640 1 -- 172.21.15.62:0/3429409615 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac0ab040 con 0x55799342dcf0 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.511+0000 7fe997e00640 1 -- 172.21.15.62:0/3429409615 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe99c007030 con 0x55799342dcf0 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.511+0000 7fe9bc228740 10 client.8667 fetch_fsmap finished waiting for FSMap version 78 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.511+0000 7fe9bc228740 10 client.8667 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:29:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.511+0000 7fe9bc228740 20 client.8667 populate_metadata read hostname 'smithi062' 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.511+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3429409615 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5579934b90e0 0x557993450cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.511+0000 7fe9bc228740 4 client.8667 mds_command: new command op to 8271 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.511+0000 7fe9bc228740 1 -- 172.21.15.62:0/3429409615 --> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5579930654e0 con 0x5579934b90e0 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.511+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/3429409615 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5579934b90e0 0x557993450cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe997e00640 10 client.8667 ms_handle_connect on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe9b2a00640 1 --2- 172.21.15.62:0/3429409615 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5579934b90e0 0x557993450cf0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edce3595c30ac3c7 server_cookie=242326ccef0484fa in_seq=0 out_seq=0 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe997e00640 1 -- 172.21.15.62:0/3429409615 <== mds.0 v2:172.21.15.62:6832/692790652 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x5579930654e0 con 0x5579934b90e0 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe997e00640 10 client.8667 handle_command_reply: tid=0 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe9bc228740 1 client.8667 shutdown 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe9bc228740 1 -- 172.21.15.62:0/3429409615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 msgr2=0x5579934b8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3429409615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x5579934b8ba0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe99c000fd0 tx=0x7fe99c018d40 comp rx=0 tx=0).stop 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe9bc228740 1 -- 172.21.15.62:0/3429409615 shutdown_connections 2024-03-23T11:29:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3429409615 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5579934b90e0 0x557993450cf0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3429409615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799342dcf0 0x5579934b8ba0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3429409615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557993436280 0x5579934507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe9bc228740 1 --2- 172.21.15.62:0/3429409615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55799342d910 0x5579934546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe997e00640 0 client.8667 ms_handle_reset on v2:172.21.15.62:6832/692790652 2024-03-23T11:29:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.512+0000 7fe9bc228740 1 -- 172.21.15.62:0/3429409615 >> 172.21.15.62:0/3429409615 conn(0x5579933d5b40 msgr2=0x5579934622a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9bc228740 1 -- 172.21.15.62:0/3429409615 shutdown_connections 2024-03-23T11:29:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9bc228740 1 -- 172.21.15.62:0/3429409615 wait complete. 2024-03-23T11:29:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9bc228740 20 client.8667 trim_cache size 0 max 16384 2024-03-23T11:29:56.507 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:29:56.507 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T11:29:56.507 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-03-23T11:29:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: "58dbe2e2-33e9-412a-b3eb-7d05a2a40ee2": { 2024-03-23T11:29:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-03-23T11:29:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "58dbe2e2-33e9-412a-b3eb-7d05a2a40ee2", 2024-03-23T11:29:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,repair" 2024-03-23T11:29:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:29:56.508 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:29:56.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9b9600640 1 -- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe99006a7d0 msgr2=0x7fe99006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9b9600640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe99006a7d0 0x7fe99006cc90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4009a30 tx=0x7fe9a4008040 comp rx=0 tx=0).stop 2024-03-23T11:29:56.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9b9600640 1 -- 172.21.15.62:0/3209953052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9b41613a0 msgr2=0x7fe9b40b1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:29:56.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9b9600640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9b41613a0 0x7fe9b40b1fb0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009e50 tx=0x7fe9ac004a40 comp rx=0 tx=0).stop 2024-03-23T11:29:56.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9b9600640 1 -- 172.21.15.62:0/3209953052 shutdown_connections 2024-03-23T11:29:56.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9b9600640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe99006a7d0 0x7fe99006cc90 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9b9600640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9b41613a0 0x7fe9b40b1fb0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9b9600640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9b41571e0 0x7fe9b41f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9b9600640 1 --2- 172.21.15.62:0/3209953052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9b40ade70 0x7fe9b41f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:29:56.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.513+0000 7fe9b9600640 1 -- 172.21.15.62:0/3209953052 >> 172.21.15.62:0/3209953052 conn(0x7fe9b40b02e0 msgr2=0x7fe9b40aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:29:56.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.514+0000 7fe9b9600640 1 -- 172.21.15.62:0/3209953052 shutdown_connections 2024-03-23T11:29:56.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:29:56.514+0000 7fe9b9600640 1 -- 172.21.15.62:0/3209953052 wait complete. 2024-03-23T11:29:56.522 INFO:tasks.cephfs.filesystem:scrub status for tag:58dbe2e2-33e9-412a-b3eb-7d05a2a40ee2 - {'path': '/', 'tag': '58dbe2e2-33e9-412a-b3eb-7d05a2a40ee2', 'options': 'recursive,repair'} 2024-03-23T11:30:00.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:00.543+0000 7f5334c00640 1 -- 172.21.15.62:0/4175894408 <== mon.0 v2:172.21.15.62:3300/0 8 ==== log(2 entries from seq 1148 at 2024-03-23T11:30:00.000160+0000) v1 ==== 338+0+0 (secure 0 0 0) 0x7f532c057cc0 con 0x7f53380b4860 2024-03-23T11:30:00.535 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:30:00.000160+0000 mon.a [INF] overall HEALTH_OK 2024-03-23T11:30:00.535 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:30:00.330672+0000 mds.a [INF] scrub summary: idle 2024-03-23T11:30:17.630 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:30:17.638 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:30:26.523 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-23T11:30:26.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.770+0000 7f71ed000640 1 Processor -- start 2024-03-23T11:30:26.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.770+0000 7f71ed000640 1 -- start start 2024-03-23T11:30:26.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.772+0000 7f71ed000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71e80cd7c0 0x7f71e80b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.772+0000 7f71ed000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e80cad20 con 0x7f71e806c300 2024-03-23T11:30:26.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.772+0000 7f71ed000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e806e130 con 0x7f71e806dca0 2024-03-23T11:30:26.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.772+0000 7f71ed000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e815a8a0 con 0x7f71e80cd7c0 2024-03-23T11:30:26.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.772+0000 7f71e6800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f71e806c300 0x7f71e80cb490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:40312/0 (socket says 172.21.15.62:40312) 2024-03-23T11:30:26.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.772+0000 7f71e6800640 1 -- 172.21.15.62:0/3221466891 learned_addr learned my addr 172.21.15.62:0/3221466891 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:30:26.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.773+0000 7f71e5e00640 1 --2- 172.21.15.62:0/3221466891 >> v2:172.21.15.120:3300/0 conn(0x7f71e80cd7c0 0x7f71e80b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.773+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946467906 0 0) 0x7f71e80cad20 con 0x7f71e806c300 2024-03-23T11:30:26.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.773+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71cc005730 con 0x7f71e806c300 2024-03-23T11:30:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.773+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1682816550 0 0) 0x7f71cc005730 con 0x7f71e806c300 2024-03-23T11:30:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.773+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 >> v2:172.21.15.120:3300/0 conn(0x7f71e80cd7c0 msgr2=0x7f71e80b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.773+0000 7f71e4a00640 1 --2- 172.21.15.62:0/3221466891 >> v2:172.21.15.120:3300/0 conn(0x7f71e80cd7c0 0x7f71e80b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.773+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 >> v1:172.21.15.120:6789/0 conn(0x7f71e806dca0 legacy=0x7f71e80cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.773+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e815ab80 con 0x7f71e806c300 2024-03-23T11:30:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.773+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f71e00019f0 con 0x7f71e806c300 2024-03-23T11:30:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.773+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 >> v1:172.21.15.62:6789/0 conn(0x7f71e806c300 legacy=0x7f71e80cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.773+0000 7f71e4a00640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71cc005bd0 0x7f71cc005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e4a00640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71cc0069d0 0x7f71cc006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e4a00640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71cc0073e0 0x7f71cc00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e815a8a0 con 0x7f71cc0073e0 2024-03-23T11:30:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc00e330 con 0x7f71cc005bd0 2024-03-23T11:30:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc00e4f0 con 0x7f71cc0069d0 2024-03-23T11:30:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e6800640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71cc0073e0 0x7f71cc00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e5e00640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71cc005bd0 0x7f71cc005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e5400640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71cc0069d0 0x7f71cc006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e6800640 1 -- 172.21.15.62:0/3221466891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71cc0069d0 msgr2=0x7f71cc006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e6800640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71cc0069d0 0x7f71cc006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e6800640 1 -- 172.21.15.62:0/3221466891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71cc005bd0 msgr2=0x7f71cc005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.774+0000 7f71e6800640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71cc005bd0 0x7f71cc005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.775+0000 7f71e6800640 1 -- 172.21.15.62:0/3221466891 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc005730 con 0x7f71cc0073e0 2024-03-23T11:30:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.775+0000 7f71e5400640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71cc0069d0 0x7f71cc006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:26.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.776+0000 7f71e6800640 1 -- 172.21.15.62:0/3221466891 reap_dead start 2024-03-23T11:30:26.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.776+0000 7f71e6800640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71cc0073e0 0x7f71cc00db90 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f71e0002230 tx=0x7f71cc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b196d2c283753713 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:26.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.776+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e0034630 con 0x7f71cc0073e0 2024-03-23T11:30:26.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.776+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e002f070 con 0x7f71cc0073e0 2024-03-23T11:30:26.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.776+0000 7f71e4a00640 1 -- 172.21.15.62:0/3221466891 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e00349d0 con 0x7f71cc0073e0 2024-03-23T11:30:26.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.777+0000 7f71ed000640 1 -- 172.21.15.62:0/3221466891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71cc0073e0 msgr2=0x7f71cc00db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:26.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.777+0000 7f71ed000640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71cc0073e0 0x7f71cc00db90 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f71e0002230 tx=0x7f71cc007a00 comp rx=0 tx=0).stop 2024-03-23T11:30:26.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.777+0000 7f71ed000640 1 -- 172.21.15.62:0/3221466891 shutdown_connections 2024-03-23T11:30:26.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.777+0000 7f71ed000640 1 --2- 172.21.15.62:0/3221466891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71cc0073e0 0x7f71cc00db90 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.777+0000 7f71ed000640 1 -- 172.21.15.62:0/3221466891 >> 172.21.15.62:0/3221466891 conn(0x7f71e80b02e0 msgr2=0x7f71e80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:26.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.777+0000 7f71ed000640 1 -- 172.21.15.62:0/3221466891 shutdown_connections 2024-03-23T11:30:26.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.777+0000 7f71ed000640 1 -- 172.21.15.62:0/3221466891 wait complete. 2024-03-23T11:30:26.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.779+0000 7f71ed000640 1 Processor -- start 2024-03-23T11:30:26.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.779+0000 7f71ed000640 1 -- start start 2024-03-23T11:30:26.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.780+0000 7f71ed000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71cc0073e0 0x7f71e8154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.780+0000 7f71ed000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e80af3f0 0x7f71e81524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.780+0000 7f71ed000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71e80af870 0x7f71e8152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.780+0000 7f71ed000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e81554c0 con 0x7f71e80af870 2024-03-23T11:30:26.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.780+0000 7f71ed000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e81557a0 con 0x7f71e80af3f0 2024-03-23T11:30:26.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.780+0000 7f71ed000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e8155a80 con 0x7f71cc0073e0 2024-03-23T11:30:26.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.780+0000 7f71e6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71e80af870 0x7f71e8152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.780+0000 7f71e6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71e80af870 0x7f71e8152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37444/0 (socket says 172.21.15.62:37444) 2024-03-23T11:30:26.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.780+0000 7f71e6800640 1 -- 172.21.15.62:0/3443439237 learned_addr learned my addr 172.21.15.62:0/3443439237 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:26.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.781+0000 7f71e5e00640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71cc0073e0 0x7f71e8154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.781+0000 7f71e5400640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e80af3f0 0x7f71e81524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.781+0000 7f71e6800640 1 -- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71cc0073e0 msgr2=0x7f71e8154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:26.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.781+0000 7f71e6800640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71cc0073e0 0x7f71e8154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.781+0000 7f71e6800640 1 -- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e80af3f0 msgr2=0x7f71e81524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:26.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.781+0000 7f71e6800640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e80af3f0 0x7f71e81524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.781+0000 7f71e6800640 1 -- 172.21.15.62:0/3443439237 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e8155d60 con 0x7f71e80af870 2024-03-23T11:30:26.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.781+0000 7f71e6800640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71e80af870 0x7f71e8152a00 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f71e0000f80 tx=0x7f71e0001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:26.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.782+0000 7f71de000640 1 -- 172.21.15.62:0/3443439237 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e0004d60 con 0x7f71e80af870 2024-03-23T11:30:26.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.782+0000 7f71de000640 1 -- 172.21.15.62:0/3443439237 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e003f470 con 0x7f71e80af870 2024-03-23T11:30:26.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.782+0000 7f71de000640 1 -- 172.21.15.62:0/3443439237 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e003f820 con 0x7f71e80af870 2024-03-23T11:30:26.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.782+0000 7f71ed000640 1 -- 172.21.15.62:0/3443439237 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e8156040 con 0x7f71e80af870 2024-03-23T11:30:26.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.782+0000 7f71ed000640 1 -- 172.21.15.62:0/3443439237 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e81f23b0 con 0x7f71e80af870 2024-03-23T11:30:26.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.784+0000 7f71de000640 1 -- 172.21.15.62:0/3443439237 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f71e0031030 con 0x7f71e80af870 2024-03-23T11:30:26.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.785+0000 7f71de000640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71c806a7a0 0x7f71c806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.785+0000 7f71de000640 1 -- 172.21.15.62:0/3443439237 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71e00aa420 con 0x7f71e80af870 2024-03-23T11:30:26.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.786+0000 7f71e5e00640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71c806a7a0 0x7f71c806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.786+0000 7f71efdf2740 1 Processor -- start 2024-03-23T11:30:26.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.786+0000 7f71efdf2740 1 -- start start 2024-03-23T11:30:26.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.787+0000 7f71e5e00640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71c806a7a0 0x7f71c806cc60 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f71d8009970 tx=0x7f71d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:26.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.787+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633a2910 0x5588633aab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.787+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633ab280 0x558863382a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.787+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x558863382f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.787+0000 7f71efdf2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55886333ecb0 con 0x5588633a2cf0 2024-03-23T11:30:26.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.787+0000 7f71efdf2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588632ed500 con 0x5588633ab280 2024-03-23T11:30:26.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.787+0000 7f71efdf2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588632eac90 con 0x5588633a2910 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71e6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x558863382f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71e5400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633ab280 0x558863382a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71e6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x558863382f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37456/0 (socket says 172.21.15.62:37456) 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71e6800640 1 -- 172.21.15.62:0/1031953465 learned_addr learned my addr 172.21.15.62:0/1031953465 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71e5e00640 1 --2- 172.21.15.62:0/1031953465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633a2910 0x5588633aab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71e6800640 1 -- 172.21.15.62:0/1031953465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633a2910 msgr2=0x5588633aab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71e6800640 1 --2- 172.21.15.62:0/1031953465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633a2910 0x5588633aab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71e6800640 1 -- 172.21.15.62:0/1031953465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633ab280 msgr2=0x558863382a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71e6800640 1 --2- 172.21.15.62:0/1031953465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633ab280 0x558863382a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71e6800640 1 -- 172.21.15.62:0/1031953465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558863298360 con 0x5588633a2cf0 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71e6800640 1 --2- 172.21.15.62:0/1031953465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x558863382f80 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f71e0081320 tx=0x7f71e0079060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cffbc16572c03dab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:26.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71d2a00640 1 -- 172.21.15.62:0/1031953465 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e0079b60 con 0x5588633a2cf0 2024-03-23T11:30:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.788+0000 7f71d2a00640 1 -- 172.21.15.62:0/1031953465 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e0079460 con 0x5588633a2cf0 2024-03-23T11:30:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.789+0000 7f71d2a00640 1 -- 172.21.15.62:0/1031953465 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e007d290 con 0x5588633a2cf0 2024-03-23T11:30:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.789+0000 7f71efdf2740 1 -- 172.21.15.62:0/1031953465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 msgr2=0x558863382f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.789+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1031953465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x558863382f80 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f71e0081320 tx=0x7f71e0079060 comp rx=0 tx=0).stop 2024-03-23T11:30:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.789+0000 7f71efdf2740 1 -- 172.21.15.62:0/1031953465 shutdown_connections 2024-03-23T11:30:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.789+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1031953465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x558863382f80 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.789+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1031953465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633ab280 0x558863382a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.789+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1031953465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633a2910 0x5588633aab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.789+0000 7f71efdf2740 1 -- 172.21.15.62:0/1031953465 >> 172.21.15.62:0/1031953465 conn(0x55886334ab40 msgr2=0x5588633c8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.789+0000 7f71efdf2740 1 -- 172.21.15.62:0/1031953465 shutdown_connections 2024-03-23T11:30:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.789+0000 7f71efdf2740 1 -- 172.21.15.62:0/1031953465 wait complete. 2024-03-23T11:30:26.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.790+0000 7f71efdf2740 1 -- start start 2024-03-23T11:30:26.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.790+0000 7f71efdf2740 1 Processor -- start 2024-03-23T11:30:26.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.791+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.791+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633a2cf0 0x5588633c9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.791+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633ab280 0x5588633c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.791+0000 7f71efdf2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588632e6580 con 0x5588633a2910 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.791+0000 7f71efdf2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558863287a50 con 0x5588633a2cf0 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.791+0000 7f71efdf2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588632876f0 con 0x5588633ab280 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.791+0000 7f71efdf2740 10 client.0 _ll_register_callbacks cb 0x5588632d6ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.791+0000 7f71e5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.792+0000 7f71e5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37466/0 (socket says 172.21.15.62:37466) 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.792+0000 7f71e5e00640 1 -- 172.21.15.62:0/1837285487 learned_addr learned my addr 172.21.15.62:0/1837285487 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.792+0000 7f71e5e00640 1 -- 172.21.15.62:0/1837285487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633ab280 msgr2=0x5588633c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.792+0000 7f71e5e00640 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633ab280 0x5588633c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.792+0000 7f71e5400640 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633a2cf0 0x5588633c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.792+0000 7f71e5e00640 1 -- 172.21.15.62:0/1837285487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633a2cf0 msgr2=0x5588633c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.792+0000 7f71e5e00640 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633a2cf0 0x5588633c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.792+0000 7f71e5e00640 1 -- 172.21.15.62:0/1837285487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e81f23b0 con 0x5588633a2910 2024-03-23T11:30:26.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.792+0000 7f71e5e00640 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633a0a70 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f71d8013f40 tx=0x7f71d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fcdafa3b2577cbf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:26.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.792+0000 7f71c7400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:30:26.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.793+0000 7f71c7400640 1 -- 172.21.15.62:0/1837285487 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d8024830 con 0x5588633a2910 2024-03-23T11:30:26.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.793+0000 7f71c7400640 1 -- 172.21.15.62:0/1837285487 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71d8024990 con 0x5588633a2910 2024-03-23T11:30:26.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.793+0000 7f71efdf2740 1 -- 172.21.15.62:0/1837285487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558863310fe0 con 0x5588633a2910 2024-03-23T11:30:26.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.793+0000 7f71c7400640 1 -- 172.21.15.62:0/1837285487 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71d8024b50 con 0x5588633a2910 2024-03-23T11:30:26.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.793+0000 7f71efdf2740 10 client.8679 fetch_fsmap learned FSMap version 78 2024-03-23T11:30:26.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.793+0000 7f71efdf2740 1 -- 172.21.15.62:0/1837285487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x5588632e4c00 con 0x5588633a2910 2024-03-23T11:30:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.793+0000 7f71c7400640 1 -- 172.21.15.62:0/1837285487 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558863310fe0 con 0x5588633a2910 2024-03-23T11:30:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.793+0000 7f71c7400640 1 -- 172.21.15.62:0/1837285487 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f71d8024cb0 con 0x5588633a2910 2024-03-23T11:30:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.793+0000 7f71efdf2740 10 client.8679 fetch_fsmap finished waiting for FSMap version 78 2024-03-23T11:30:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.793+0000 7f71efdf2740 10 client.8679 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:30:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.793+0000 7f71efdf2740 20 client.8679 populate_metadata read hostname 'smithi062' 2024-03-23T11:30:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.794+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5588633c7250 0x5588633c7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.794+0000 7f71efdf2740 4 client.8679 mds_command: new command op to 8271 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:30:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.794+0000 7f71efdf2740 1 -- 172.21.15.62:0/1837285487 --> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588632d9240 con 0x5588633c7250 2024-03-23T11:30:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.794+0000 7f71e5400640 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5588633c7250 0x5588633c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.794+0000 7f71e5400640 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5588633c7250 0x5588633c7660 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25391627f8d7890b server_cookie=2812ae58a49e5b7c in_seq=0 out_seq=0 2024-03-23T11:30:26.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.794+0000 7f71c7400640 10 client.8679 ms_handle_connect on v2:172.21.15.62:6832/692790652 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71c7400640 1 -- 172.21.15.62:0/1837285487 <== mds.0 v2:172.21.15.62:6832/692790652 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5588632d9240 con 0x5588633c7250 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71c7400640 10 client.8679 handle_command_reply: tid=0 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71efdf2740 1 client.8679 shutdown 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71efdf2740 1 -- 172.21.15.62:0/1837285487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 msgr2=0x5588633a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633a0a70 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f71d8013f40 tx=0x7f71d8004730 comp rx=0 tx=0).stop 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71efdf2740 1 -- 172.21.15.62:0/1837285487 shutdown_connections 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5588633c7250 0x5588633c7660 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633ab280 0x5588633c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633a2cf0 0x5588633c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71c7400640 0 client.8679 ms_handle_reset on v2:172.21.15.62:6832/692790652 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1837285487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633a0a70 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.795+0000 7f71efdf2740 1 -- 172.21.15.62:0/1837285487 >> 172.21.15.62:0/1837285487 conn(0x55886334ab40 msgr2=0x5588633cb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.796+0000 7f71efdf2740 1 -- 172.21.15.62:0/1837285487 shutdown_connections 2024-03-23T11:30:26.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.796+0000 7f71efdf2740 1 -- 172.21.15.62:0/1837285487 wait complete. 2024-03-23T11:30:26.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.796+0000 7f71efdf2740 20 client.8679 trim_cache size 0 max 16384 2024-03-23T11:30:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.812+0000 7f71efdf2740 1 Processor -- start 2024-03-23T11:30:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.812+0000 7f71efdf2740 1 -- start start 2024-03-23T11:30:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633c7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633a2cf0 0x5588633d6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633ab280 0x5588633d7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71efdf2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x558862fe3f90 con 0x5588633a2910 2024-03-23T11:30:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71efdf2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588632e5730 con 0x5588633ab280 2024-03-23T11:30:26.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71efdf2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588633ab100 con 0x5588633a2cf0 2024-03-23T11:30:26.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71e5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71e5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633c7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37474/0 (socket says 172.21.15.62:37474) 2024-03-23T11:30:26.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71e5e00640 1 -- 172.21.15.62:0/1793710634 learned_addr learned my addr 172.21.15.62:0/1793710634 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:26.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71e5e00640 1 -- 172.21.15.62:0/1793710634 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633a2cf0 msgr2=0x5588633d6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:26.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71e5e00640 1 --2- 172.21.15.62:0/1793710634 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633a2cf0 0x5588633d6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71e5e00640 1 -- 172.21.15.62:0/1793710634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633ab280 msgr2=0x5588633d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:26.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71e5e00640 1 --2- 172.21.15.62:0/1793710634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633ab280 0x5588633d7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.813+0000 7f71e5e00640 1 -- 172.21.15.62:0/1793710634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558863286c60 con 0x5588633a2910 2024-03-23T11:30:26.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.814+0000 7f71e5e00640 1 --2- 172.21.15.62:0/1793710634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633c7820 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f71d80137b0 tx=0x7f71d8024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef483acb21008d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:26.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.814+0000 7f71d3400640 1 -- 172.21.15.62:0/1793710634 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d8015d80 con 0x5588633a2910 2024-03-23T11:30:26.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.814+0000 7f71d3400640 1 -- 172.21.15.62:0/1793710634 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d803b070 con 0x5588633a2910 2024-03-23T11:30:26.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.814+0000 7f71d3400640 1 -- 172.21.15.62:0/1793710634 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d80157f0 con 0x5588633a2910 2024-03-23T11:30:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.814+0000 7f71efdf2740 1 -- 172.21.15.62:0/1793710634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 msgr2=0x5588633c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.814+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1793710634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633c7820 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f71d80137b0 tx=0x7f71d8024000 comp rx=0 tx=0).stop 2024-03-23T11:30:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.815+0000 7f71efdf2740 1 -- 172.21.15.62:0/1793710634 shutdown_connections 2024-03-23T11:30:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.815+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1793710634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633ab280 0x5588633d7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.815+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1793710634 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633a2cf0 0x5588633d6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.815+0000 7f71efdf2740 1 --2- 172.21.15.62:0/1793710634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2910 0x5588633c7820 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.815+0000 7f71efdf2740 1 -- 172.21.15.62:0/1793710634 >> 172.21.15.62:0/1793710634 conn(0x55886334ab40 msgr2=0x55886337eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.815+0000 7f71efdf2740 1 -- 172.21.15.62:0/1793710634 shutdown_connections 2024-03-23T11:30:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.815+0000 7f71efdf2740 1 -- 172.21.15.62:0/1793710634 wait complete. 2024-03-23T11:30:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.815+0000 7f71efdf2740 1 -- start start 2024-03-23T11:30:26.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.816+0000 7f71efdf2740 1 Processor -- start 2024-03-23T11:30:26.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.816+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633a2910 0x558863427880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x5588633ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71efdf2740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633ab280 0x5588633cac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71efdf2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588633cb330 con 0x5588633a2cf0 2024-03-23T11:30:26.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71efdf2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588633aad20 con 0x5588633a2910 2024-03-23T11:30:26.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71efdf2740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558863383210 con 0x5588633ab280 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71efdf2740 10 client.0 _ll_register_callbacks cb 0x7f71e002ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71e5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x5588633ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71e5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x5588633ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37486/0 (socket says 172.21.15.62:37486) 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71e5400640 1 -- 172.21.15.62:0/4235211636 learned_addr learned my addr 172.21.15.62:0/4235211636 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71e5400640 1 -- 172.21.15.62:0/4235211636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633ab280 msgr2=0x5588633cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71e5400640 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633ab280 0x5588633cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71e5400640 1 -- 172.21.15.62:0/4235211636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633a2910 msgr2=0x558863427880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71e5e00640 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633a2910 0x558863427880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71e5400640 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633a2910 0x558863427880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.817+0000 7f71e5400640 1 -- 172.21.15.62:0/4235211636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588632e4c00 con 0x5588633a2cf0 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.818+0000 7f71e5400640 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x5588633ca6e0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f71d4005510 tx=0x7f71d4019e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa94b52607b2a227 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.818+0000 7f71c7400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.818+0000 7f71c7400640 1 -- 172.21.15.62:0/4235211636 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d4004d80 con 0x5588633a2cf0 2024-03-23T11:30:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.818+0000 7f71c7400640 1 -- 172.21.15.62:0/4235211636 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71d401f900 con 0x5588633a2cf0 2024-03-23T11:30:26.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.818+0000 7f71efdf2740 1 -- 172.21.15.62:0/4235211636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f71e0079460 con 0x5588633a2cf0 2024-03-23T11:30:26.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.818+0000 7f71c7400640 1 -- 172.21.15.62:0/4235211636 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71d4004ee0 con 0x5588633a2cf0 2024-03-23T11:30:26.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.818+0000 7f71efdf2740 10 client.8685 fetch_fsmap learned FSMap version 78 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.818+0000 7f71efdf2740 1 -- 172.21.15.62:0/4235211636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x5588632ac170 con 0x5588633a2cf0 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.819+0000 7f71c7400640 1 -- 172.21.15.62:0/4235211636 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e0079460 con 0x5588633a2cf0 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.819+0000 7f71c7400640 1 -- 172.21.15.62:0/4235211636 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f71d4021030 con 0x5588633a2cf0 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.819+0000 7f71efdf2740 10 client.8685 fetch_fsmap finished waiting for FSMap version 78 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.819+0000 7f71efdf2740 10 client.8685 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.819+0000 7f71efdf2740 20 client.8685 populate_metadata read hostname 'smithi062' 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.819+0000 7f71efdf2740 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5588633c92b0 0x5588633c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.819+0000 7f71efdf2740 4 client.8685 mds_command: new command op to 8271 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.819+0000 7f71efdf2740 1 -- 172.21.15.62:0/4235211636 --> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558862fda4e0 con 0x5588633c92b0 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.819+0000 7f71e5e00640 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5588633c92b0 0x5588633c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71e5e00640 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5588633c92b0 0x5588633c5f50 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a3d73cf28071576 server_cookie=553bef704edb260c in_seq=0 out_seq=0 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71c7400640 10 client.8685 ms_handle_connect on v2:172.21.15.62:6832/692790652 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71c7400640 1 -- 172.21.15.62:0/4235211636 <== mds.0 v2:172.21.15.62:6832/692790652 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558862fda4e0 con 0x5588633c92b0 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71c7400640 10 client.8685 handle_command_reply: tid=0 2024-03-23T11:30:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71efdf2740 1 client.8685 shutdown 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71efdf2740 1 -- 172.21.15.62:0/4235211636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 msgr2=0x5588633ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71efdf2740 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x5588633ca6e0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f71d4005510 tx=0x7f71d4019e40 comp rx=0 tx=0).stop 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71efdf2740 1 -- 172.21.15.62:0/4235211636 shutdown_connections 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71efdf2740 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.62:6832/692790652,v1:172.21.15.62:6834/692790652] conn(0x5588633c92b0 0x5588633c5f50 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71efdf2740 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588633a2cf0 0x5588633ca6e0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71efdf2740 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5588633ab280 0x5588633cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71efdf2740 1 --2- 172.21.15.62:0/4235211636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588633a2910 0x558863427880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71c7400640 0 client.8685 ms_handle_reset on v2:172.21.15.62:6832/692790652 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71efdf2740 1 -- 172.21.15.62:0/4235211636 >> 172.21.15.62:0/4235211636 conn(0x55886334ab40 msgr2=0x5588633d73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71efdf2740 1 -- 172.21.15.62:0/4235211636 shutdown_connections 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.820+0000 7f71efdf2740 1 -- 172.21.15.62:0/4235211636 wait complete. 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71efdf2740 20 client.8685 trim_cache size 0 max 16384 2024-03-23T11:30:26.813 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:30:26.814 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-03-23T11:30:26.814 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-03-23T11:30:26.814 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:30:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 -- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71c806a7a0 msgr2=0x7f71c806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71c806a7a0 0x7f71c806cc60 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f71d8009970 tx=0x7f71d8008040 comp rx=0 tx=0).stop 2024-03-23T11:30:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 -- 172.21.15.62:0/3443439237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71e80af870 msgr2=0x7f71e8152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71e80af870 0x7f71e8152a00 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f71e0000f80 tx=0x7f71e0001cb0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 -- 172.21.15.62:0/3443439237 shutdown_connections 2024-03-23T11:30:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f71c806a7a0 0x7f71c806cc60 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71e80af870 0x7f71e8152a00 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e80af3f0 0x7f71e81524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 --2- 172.21.15.62:0/3443439237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71cc0073e0 0x7f71e8154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 -- 172.21.15.62:0/3443439237 >> 172.21.15.62:0/3443439237 conn(0x7f71e80b02e0 msgr2=0x7f71e80cbb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 -- 172.21.15.62:0/3443439237 shutdown_connections 2024-03-23T11:30:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:26.821+0000 7f71ed000640 1 -- 172.21.15.62:0/3443439237 wait complete. 2024-03-23T11:30:26.828 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T11:30:26.929 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:30:27.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.156+0000 7f6031c00640 1 Processor -- start 2024-03-23T11:30:27.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.157+0000 7f6031c00640 1 -- start start 2024-03-23T11:30:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.158+0000 7f6031c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f602c06c300 0x7f602c0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.158+0000 7f6031c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c0cafa0 0x7f602c0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.158+0000 7f6031c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c0cbc60 con 0x7f602c06c6e0 2024-03-23T11:30:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.158+0000 7f6031c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602c0cbf40 con 0x7f602c06c300 2024-03-23T11:30:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.158+0000 7f6031c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602c0cc220 con 0x7f602c0cafa0 2024-03-23T11:30:27.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.159+0000 7f6023e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c0cafa0 0x7f602c0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.159+0000 7f602aa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f602c06c300 0x7f602c0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.159+0000 7f6023e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c0cafa0 0x7f602c0cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:36780/0 (socket says 172.21.15.62:36780) 2024-03-23T11:30:27.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.159+0000 7f6023e00640 1 -- 172.21.15.62:0/2568177474 learned_addr learned my addr 172.21.15.62:0/2568177474 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:27.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.160+0000 7f6023e00640 1 -- 172.21.15.62:0/2568177474 >> v2:172.21.15.120:3300/0 conn(0x7f602c06c300 msgr2=0x7f602c0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:27.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.160+0000 7f6023e00640 1 --2- 172.21.15.62:0/2568177474 >> v2:172.21.15.120:3300/0 conn(0x7f602c06c300 0x7f602c0ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.160+0000 7f6023e00640 1 -- 172.21.15.62:0/2568177474 >> v1:172.21.15.120:6789/0 conn(0x7f602c06c6e0 legacy=0x7f602c06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.160+0000 7f6023e00640 1 -- 172.21.15.62:0/2568177474 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c0cc500 con 0x7f602c0cafa0 2024-03-23T11:30:27.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.162+0000 7f6023e00640 1 --2- 172.21.15.62:0/2568177474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c0cafa0 0x7f602c0cb400 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6014009870 tx=0x7f601402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3540a9c97b93eaab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.162+0000 7f602a000640 1 -- 172.21.15.62:0/2568177474 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6014031640 con 0x7f602c0cafa0 2024-03-23T11:30:27.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.162+0000 7f602a000640 1 -- 172.21.15.62:0/2568177474 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60140317a0 con 0x7f602c0cafa0 2024-03-23T11:30:27.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.162+0000 7f602a000640 1 -- 172.21.15.62:0/2568177474 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6014031b30 con 0x7f602c0cafa0 2024-03-23T11:30:27.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.163+0000 7f6031c00640 1 -- 172.21.15.62:0/2568177474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c0cafa0 msgr2=0x7f602c0cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.163+0000 7f6031c00640 1 --2- 172.21.15.62:0/2568177474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c0cafa0 0x7f602c0cb400 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6014009870 tx=0x7f601402b8f0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.163+0000 7f6031c00640 1 -- 172.21.15.62:0/2568177474 shutdown_connections 2024-03-23T11:30:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.163+0000 7f6031c00640 1 --2- 172.21.15.62:0/2568177474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c0cafa0 0x7f602c0cb400 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.163+0000 7f6031c00640 1 --2- 172.21.15.62:0/2568177474 >> v2:172.21.15.120:3300/0 conn(0x7f602c06c300 0x7f602c0ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.163+0000 7f6031c00640 1 -- 172.21.15.62:0/2568177474 >> 172.21.15.62:0/2568177474 conn(0x7f602c0b02e0 msgr2=0x7f602c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.163+0000 7f6031c00640 1 -- 172.21.15.62:0/2568177474 shutdown_connections 2024-03-23T11:30:27.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.163+0000 7f6031c00640 1 -- 172.21.15.62:0/2568177474 wait complete. 2024-03-23T11:30:27.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.165+0000 7f6031c00640 1 Processor -- start 2024-03-23T11:30:27.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.165+0000 7f6031c00640 1 -- start start 2024-03-23T11:30:27.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.166+0000 7f6031c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f602c06c300 0x7f602c156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.166+0000 7f6031c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c06c6e0 0x7f602c0ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.166+0000 7f6031c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f602c0cafa0 0x7f602c0b2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.166+0000 7f6031c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c15c3d0 con 0x7f602c06c300 2024-03-23T11:30:27.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.166+0000 7f6031c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c15c640 con 0x7f602c0cafa0 2024-03-23T11:30:27.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.166+0000 7f6031c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602c15c920 con 0x7f602c06c6e0 2024-03-23T11:30:27.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.167+0000 7f602aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f602c06c300 0x7f602c156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.167+0000 7f602aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f602c06c300 0x7f602c156370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37490/0 (socket says 172.21.15.62:37490) 2024-03-23T11:30:27.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.167+0000 7f602aa00640 1 -- 172.21.15.62:0/921228829 learned_addr learned my addr 172.21.15.62:0/921228829 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:27.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.167+0000 7f602b400640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f602c0cafa0 0x7f602c0b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.167+0000 7f6023e00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c06c6e0 0x7f602c0ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.167+0000 7f602aa00640 1 -- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c06c6e0 msgr2=0x7f602c0ad7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.167+0000 7f602aa00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c06c6e0 0x7f602c0ad7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.167+0000 7f602aa00640 1 -- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f602c0cafa0 msgr2=0x7f602c0b2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.167+0000 7f602aa00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f602c0cafa0 0x7f602c0b2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.167+0000 7f602aa00640 1 -- 172.21.15.62:0/921228829 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60140094d0 con 0x7f602c06c300 2024-03-23T11:30:27.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.168+0000 7f602aa00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f602c06c300 0x7f602c156370 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f601c00e9b0 tx=0x7f601c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.168+0000 7f6022a00640 1 -- 172.21.15.62:0/921228829 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c009670 con 0x7f602c06c300 2024-03-23T11:30:27.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.168+0000 7f6022a00640 1 -- 172.21.15.62:0/921228829 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c00c9a0 con 0x7f602c06c300 2024-03-23T11:30:27.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.168+0000 7f6031c00640 1 -- 172.21.15.62:0/921228829 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602c15cc00 con 0x7f602c06c300 2024-03-23T11:30:27.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.168+0000 7f6022a00640 1 -- 172.21.15.62:0/921228829 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c00cd50 con 0x7f602c06c300 2024-03-23T11:30:27.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.168+0000 7f6031c00640 1 -- 172.21.15.62:0/921228829 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602c0add10 con 0x7f602c06c300 2024-03-23T11:30:27.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.170+0000 7f6031c00640 1 -- 172.21.15.62:0/921228829 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602c066c90 con 0x7f602c06c300 2024-03-23T11:30:27.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.171+0000 7f6022a00640 1 -- 172.21.15.62:0/921228829 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f601c013470 con 0x7f602c06c300 2024-03-23T11:30:27.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.172+0000 7f6022a00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f600806a7a0 0x7f600806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.172+0000 7f6022a00640 1 -- 172.21.15.62:0/921228829 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f601c088fe0 con 0x7f602c06c300 2024-03-23T11:30:27.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.172+0000 7f6023e00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f600806a7a0 0x7f600806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.174+0000 7f6023e00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f600806a7a0 0x7f600806cc60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6014036000 tx=0x7f601403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.177+0000 7f6022a00640 1 -- 172.21.15.62:0/921228829 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f601c054450 con 0x7f602c06c300 2024-03-23T11:30:27.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.419+0000 7f6031c00640 1 -- 172.21.15.62:0/921228829 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f602c0674d0 con 0x7f602c06c300 2024-03-23T11:30:27.411 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:30:27.421+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:30:27.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.437+0000 7f6022a00640 1 -- 172.21.15.62:0/921228829 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v79) v1 ==== 161+0+0 (secure 0 0 0) 0x7f601c05ce60 con 0x7f602c06c300 2024-03-23T11:30:27.427 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:30:27.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 -- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f600806a7a0 msgr2=0x7f600806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f600806a7a0 0x7f600806cc60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6014036000 tx=0x7f601403c040 comp rx=0 tx=0).stop 2024-03-23T11:30:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 -- 172.21.15.62:0/921228829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f602c06c300 msgr2=0x7f602c156370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f602c06c300 0x7f602c156370 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f601c00e9b0 tx=0x7f601c00ee70 comp rx=0 tx=0).stop 2024-03-23T11:30:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 -- 172.21.15.62:0/921228829 shutdown_connections 2024-03-23T11:30:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f600806a7a0 0x7f600806cc60 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f602c0cafa0 0x7f602c0b2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f602c06c6e0 0x7f602c0ad7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 --2- 172.21.15.62:0/921228829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f602c06c300 0x7f602c156370 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 -- 172.21.15.62:0/921228829 >> 172.21.15.62:0/921228829 conn(0x7f602c0b02e0 msgr2=0x7f602c06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 -- 172.21.15.62:0/921228829 shutdown_connections 2024-03-23T11:30:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.440+0000 7f6031c00640 1 -- 172.21.15.62:0/921228829 wait complete. 2024-03-23T11:30:27.451 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627777 summary 2024-03-23T11:30:27.464 INFO:tasks.ceph.mds.a.smithi062.stderr: -6335> 2024-03-23T11:29:21.111+0000 7fd542c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:30:27.533 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:30:27.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.558+0000 7fb086c52280 1 -- start start 2024-03-23T11:30:27.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.558+0000 7fb086c52280 1 Processor -- start 2024-03-23T11:30:27.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.559+0000 7fb086c52280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d6b04c00 0x5635d6b8e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.559+0000 7fb086c52280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d6b05800 0x5635d6b8eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.559+0000 7fb086c52280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6b05c00 0x5635d6b8f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.559+0000 7fb086c52280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635d6bd6a00 con 0x5635d6b05c00 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.559+0000 7fb086c52280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635d6bd6780 con 0x5635d6b05800 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.559+0000 7fb086c52280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635d6bd6640 con 0x5635d6b04c00 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.560+0000 7fb085000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6b05c00 0x5635d6b8f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.560+0000 7fb085000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6b05c00 0x5635d6b8f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49894/0 (socket says 172.21.15.62:49894) 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.560+0000 7fb085000640 1 -- 172.21.15.62:0/2238862884 learned_addr learned my addr 172.21.15.62:0/2238862884 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.560+0000 7fb084600640 1 --2- 172.21.15.62:0/2238862884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d6b04c00 0x5635d6b8e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.560+0000 7fb083c00640 1 --2- 172.21.15.62:0/2238862884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d6b05800 0x5635d6b8eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.560+0000 7fb085000640 1 -- 172.21.15.62:0/2238862884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d6b04c00 msgr2=0x5635d6b8e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:27.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.560+0000 7fb085000640 1 --2- 172.21.15.62:0/2238862884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d6b04c00 0x5635d6b8e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.560+0000 7fb085000640 1 -- 172.21.15.62:0/2238862884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d6b05800 msgr2=0x5635d6b8eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:27.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.560+0000 7fb085000640 1 --2- 172.21.15.62:0/2238862884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d6b05800 0x5635d6b8eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.560+0000 7fb085000640 1 -- 172.21.15.62:0/2238862884 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635d6b2a9c0 con 0x5635d6b05c00 2024-03-23T11:30:27.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.560+0000 7fb085000640 1 --2- 172.21.15.62:0/2238862884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6b05c00 0x5635d6b8f080 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x5635d792ecf0 tx=0x5635d7899080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ebc71fbb0f0598 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.561+0000 7fb081400640 1 -- 172.21.15.62:0/2238862884 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5635d6b4d080 con 0x5635d6b05c00 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.561+0000 7fb081400640 1 -- 172.21.15.62:0/2238862884 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5635d6b62480 con 0x5635d6b05c00 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.561+0000 7fb081400640 1 -- 172.21.15.62:0/2238862884 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5635d6b4d600 con 0x5635d6b05c00 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.561+0000 7fb086c52280 1 -- 172.21.15.62:0/2238862884 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635d6b2ad00 con 0x5635d6b05c00 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.561+0000 7fb086c52280 1 -- 172.21.15.62:0/2238862884 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5635d6b2a9c0 con 0x5635d6b05c00 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.561+0000 7fb086c52280 4 waiting for MDS map... 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.561+0000 7fb081400640 1 -- 172.21.15.62:0/2238862884 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x5635d6b30380 con 0x5635d6b05c00 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.561+0000 7fb081400640 1 -- 172.21.15.62:0/2238862884 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 79) v1 ==== 2573+0+0 (secure 0 0 0) 0x5635d6b56f00 con 0x5635d6b05c00 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.561+0000 7fb086c52280 4 Got MDS map 79 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.561+0000 7fb086c52280 10 main: JournalTool::main 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.561+0000 7fb086c52280 4 main: JournalTool: connecting to RADOS... 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.562+0000 7fb086c52280 1 Processor -- start 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.562+0000 7fb086c52280 1 -- start start 2024-03-23T11:30:27.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.563+0000 7fb086c52280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d78e5c00 0x5635d6b8f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.563+0000 7fb086c52280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d7a24000 0x5635d6b8fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.563+0000 7fb086c52280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d7a24400 0x5635d6b90100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.563+0000 7fb086c52280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635d6b4fe00 con 0x5635d7a24400 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.563+0000 7fb086c52280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635d6bd6780 con 0x5635d78e5c00 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.563+0000 7fb086c52280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635d6bd6a00 con 0x5635d7a24000 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.563+0000 7fb083c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d7a24400 0x5635d6b90100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.563+0000 7fb083c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d7a24400 0x5635d6b90100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49910/0 (socket says 172.21.15.62:49910) 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.563+0000 7fb083c00640 1 -- 172.21.15.62:0/3180941838 learned_addr learned my addr 172.21.15.62:0/3180941838 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.564+0000 7fb084600640 1 --2- 172.21.15.62:0/3180941838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d7a24000 0x5635d6b8fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.564+0000 7fb083c00640 1 --2- 172.21.15.62:0/3180941838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d78e5c00 0x5635d6b8f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.564+0000 7fb083c00640 1 -- 172.21.15.62:0/3180941838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d7a24000 msgr2=0x5635d6b8fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.564+0000 7fb083c00640 1 --2- 172.21.15.62:0/3180941838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d7a24000 0x5635d6b8fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.564+0000 7fb083c00640 1 -- 172.21.15.62:0/3180941838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d78e5c00 msgr2=0x5635d6b8f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:27.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.564+0000 7fb083c00640 1 --2- 172.21.15.62:0/3180941838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d78e5c00 0x5635d6b8f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.564+0000 7fb083c00640 1 -- 172.21.15.62:0/3180941838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635d6b2ab60 con 0x5635d7a24400 2024-03-23T11:30:27.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.564+0000 7fb083c00640 1 --2- 172.21.15.62:0/3180941838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d7a24400 0x5635d6b90100 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x5635d7a320f0 tx=0x5635d79e4c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e951842e296c84 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.564+0000 7fb07ec00640 1 -- 172.21.15.62:0/3180941838 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5635d6b4d080 con 0x5635d7a24400 2024-03-23T11:30:27.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.564+0000 7fb07ec00640 1 -- 172.21.15.62:0/3180941838 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5635d78de480 con 0x5635d7a24400 2024-03-23T11:30:27.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.564+0000 7fb07ec00640 1 -- 172.21.15.62:0/3180941838 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5635d6b4d600 con 0x5635d7a24400 2024-03-23T11:30:27.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.565+0000 7fb086c52280 1 -- 172.21.15.62:0/3180941838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d7a24400 msgr2=0x5635d6b90100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.565+0000 7fb086c52280 1 --2- 172.21.15.62:0/3180941838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d7a24400 0x5635d6b90100 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x5635d7a320f0 tx=0x5635d79e4c60 comp rx=0 tx=0).stop 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.565+0000 7fb086c52280 1 -- 172.21.15.62:0/3180941838 shutdown_connections 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.565+0000 7fb086c52280 1 --2- 172.21.15.62:0/3180941838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d7a24400 0x5635d6b90100 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.565+0000 7fb086c52280 1 --2- 172.21.15.62:0/3180941838 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d7a24000 0x5635d6b8fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.565+0000 7fb086c52280 1 --2- 172.21.15.62:0/3180941838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d78e5c00 0x5635d6b8f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.565+0000 7fb086c52280 1 -- 172.21.15.62:0/3180941838 >> 172.21.15.62:0/3180941838 conn(0x5635d78e5800 msgr2=0x5635d6bf7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.565+0000 7fb086c52280 1 -- 172.21.15.62:0/3180941838 shutdown_connections 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.565+0000 7fb086c52280 1 -- 172.21.15.62:0/3180941838 wait complete. 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.566+0000 7fb086c52280 1 Processor -- start 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.566+0000 7fb086c52280 1 -- start start 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.567+0000 7fb086c52280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d78e5800 0x5635d6b91180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.567+0000 7fb086c52280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d7a25000 0x5635d6b90c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.567+0000 7fb086c52280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d7a24c00 0x5635d6b90680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.567+0000 7fb086c52280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635d6bd6500 con 0x5635d78e5800 2024-03-23T11:30:27.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.567+0000 7fb086c52280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635d6bd63c0 con 0x5635d7a25000 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.567+0000 7fb086c52280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5635d6bd6280 con 0x5635d7a24c00 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.567+0000 7fb083c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d78e5800 0x5635d6b91180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.567+0000 7fb083c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d78e5800 0x5635d6b91180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49922/0 (socket says 172.21.15.62:49922) 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.567+0000 7fb083c00640 1 -- 172.21.15.62:0/2354530993 learned_addr learned my addr 172.21.15.62:0/2354530993 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.567+0000 7fb084600640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d7a25000 0x5635d6b90c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb083c00640 1 -- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d7a24c00 msgr2=0x5635d6b90680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb083c00640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d7a24c00 0x5635d6b90680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb083c00640 1 -- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d7a25000 msgr2=0x5635d6b90c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb083c00640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d7a25000 0x5635d6b90c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb083c00640 1 -- 172.21.15.62:0/2354530993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635d6b2a820 con 0x5635d78e5800 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb083c00640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d78e5800 0x5635d6b91180 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x5635d7a333b0 tx=0x5635d79c8d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb07ce00640 1 -- 172.21.15.62:0/2354530993 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5635d6b4cdc0 con 0x5635d78e5800 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb07ce00640 1 -- 172.21.15.62:0/2354530993 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5635d78de780 con 0x5635d78e5800 2024-03-23T11:30:27.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb07ce00640 1 -- 172.21.15.62:0/2354530993 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5635d6b4d080 con 0x5635d78e5800 2024-03-23T11:30:27.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5635d6b2a4e0 con 0x5635d78e5800 2024-03-23T11:30:27.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635d6b2a1a0 con 0x5635d78e5800 2024-03-23T11:30:27.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.568+0000 7fb086c52280 4 main: JournalTool: resolving pool 14 2024-03-23T11:30:27.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.570+0000 7fb07ce00640 1 -- 172.21.15.62:0/2354530993 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x5635d6bd3080 con 0x5635d78e5800 2024-03-23T11:30:27.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.571+0000 7fb07ce00640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5635d6e62000 0x5635d6b91700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.571+0000 7fb07ce00640 1 -- 172.21.15.62:0/2354530993 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x5635d6b30540 con 0x5635d78e5800 2024-03-23T11:30:27.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.571+0000 7fb086c52280 4 main: JournalTool: creating IoCtx.. 2024-03-23T11:30:27.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.571+0000 7fb086c52280 4 main: Executing for rank 0 2024-03-23T11:30:27.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.571+0000 7fb086c52280 4 Filtering by inode '1099511627777' 2024-03-23T11:30:27.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.571+0000 7fb086c52280 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5635d7a24800 0x5635d6b90100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.571+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:1 14.f 14:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x5635d7a24400 con 0x5635d7a24800 2024-03-23T11:30:27.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.571+0000 7fb084600640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5635d6e62000 0x5635d6b91700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.572+0000 7fb083c00640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5635d7a24800 0x5635d6b90100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.572+0000 7fb084600640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5635d6e62000 0x5635d6b91700 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x5635d792ebd0 tx=0x5635d79e5da0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.573+0000 7fb083c00640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5635d7a24800 0x5635d6b90100 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.574+0000 7fb083c00640 1 -- 172.21.15.62:0/2354530993 <== osd.7 v2:172.21.15.120:6816/37655 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x5635d79c4b40 con 0x5635d7a24800 2024-03-23T11:30:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.574+0000 7fb086c52280 4 Pointer 400.00000000 is readable 2024-03-23T11:30:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.574+0000 7fb086c52280 4 JournalScanner::scan: reading header object '200.00000000' 2024-03-23T11:30:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.574+0000 7fb086c52280 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x5635d7a24000 0x5635d6b8fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.574+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:2 14.14 14:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x5635d6e63000 con 0x5635d7a24000 2024-03-23T11:30:27.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.575+0000 7fb085000640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x5635d7a24000 0x5635d6b8fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.575+0000 7fb085000640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x5635d7a24000 0x5635d6b8fb80 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.576+0000 7fb085000640 1 -- 172.21.15.62:0/2354530993 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv31 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x5635d79c4d80 con 0x5635d7a24000 2024-03-23T11:30:27.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.577+0000 7fb086c52280 10 Header 0x400000 0x405ee0 0x40625c 2024-03-23T11:30:27.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.577+0000 7fb086c52280 10 Starting journal scan from offset 0x405ee0 2024-03-23T11:30:27.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.577+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:3 14.14 14:2e2fa760:::200.00000001:head [read 24288~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x5635d6e63400 con 0x5635d7a24000 2024-03-23T11:30:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.578+0000 7fb085000640 1 -- 172.21.15.62:0/2354530993 <== osd.4 v2:172.21.15.120:6803/37640 2 ==== osd_op_reply(3 200.00000001 [read 24288~892 out=892b] v0'0 uv27 ondisk = 0) v8 ==== 156+0+892 (crc 0 0 0) 0x5635d79c4d80 con 0x5635d7a24000 2024-03-23T11:30:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.578+0000 7fb086c52280 4 Read 0x37c bytes from 200.00000001 gap=0 2024-03-23T11:30:27.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.578+0000 7fb086c52280 10 Parsing data, 0x37c bytes available 2024-03-23T11:30:27.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.578+0000 7fb086c52280 10 Attempting decode at 0x405ee0 2024-03-23T11:30:27.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.578+0000 7fb086c52280 10 Consumed 0x37c bytes 2024-03-23T11:30:27.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.578+0000 7fb086c52280 10 Valid entry at 0x405ee0 2024-03-23T11:30:27.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.578+0000 7fb086c52280 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5635d6e63800 0x5635d6b8f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.578+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:4 14.1b 14:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x5635d6e63c00 con 0x5635d6e63800 2024-03-23T11:30:27.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.578+0000 7fb084600640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5635d6e63800 0x5635d6b8f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb084600640 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5635d6e63800 0x5635d6b8f600 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb084600640 1 -- 172.21.15.62:0/2354530993 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5635d79c4fc0 con 0x5635d6e63800 2024-03-23T11:30:27.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 4 Reached end of journal objects 2024-03-23T11:30:27.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 4 Scanned objects, 0 missing, 1 valid 2024-03-23T11:30:27.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 4 Events scanned, 0 gaps 2024-03-23T11:30:27.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 4 Found 1 valid events 2024-03-23T11:30:27.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 4 Selected 0 events events for processing 2024-03-23T11:30:27.570 INFO:teuthology.orchestra.run.smithi062.stdout:Events by type: 2024-03-23T11:30:27.570 INFO:teuthology.orchestra.run.smithi062.stdout:Errors: 0 2024-03-23T11:30:27.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 4 0 events 2024-03-23T11:30:27.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5635d6e63800 msgr2=0x5635d6b8f600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5635d6e63800 0x5635d6b8f600 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x5635d7a24000 msgr2=0x5635d6b8fb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x5635d7a24000 0x5635d6b8fb80 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5635d7a24800 msgr2=0x5635d6b90100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5635d7a24800 0x5635d6b90100 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb084600640 1 -- 172.21.15.62:0/2354530993 reap_dead start 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5635d6e62000 msgr2=0x5635d6b91700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5635d6e62000 0x5635d6b91700 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x5635d792ebd0 tx=0x5635d79e5da0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d78e5800 msgr2=0x5635d6b91180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d78e5800 0x5635d6b91180 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x5635d7a333b0 tx=0x5635d79c8d80 comp rx=0 tx=0).stop 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 shutdown_connections 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5635d6e62000 0x5635d6b91700 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 --2- 172.21.15.62:0/2354530993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d78e5800 0x5635d6b91180 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 >> 172.21.15.62:0/2354530993 conn(0x5635d78e5c00 msgr2=0x5635d6bf7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 shutdown_connections 2024-03-23T11:30:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.579+0000 7fb086c52280 1 -- 172.21.15.62:0/2354530993 wait complete. 2024-03-23T11:30:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.580+0000 7fb086c52280 1 -- 172.21.15.62:0/2238862884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6b05c00 msgr2=0x5635d6b8f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.580+0000 7fb086c52280 1 --2- 172.21.15.62:0/2238862884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6b05c00 0x5635d6b8f080 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x5635d792ecf0 tx=0x5635d7899080 comp rx=0 tx=0).stop 2024-03-23T11:30:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.580+0000 7fb086c52280 1 -- 172.21.15.62:0/2238862884 shutdown_connections 2024-03-23T11:30:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.580+0000 7fb086c52280 1 --2- 172.21.15.62:0/2238862884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6b05c00 0x5635d6b8f080 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.580+0000 7fb086c52280 1 --2- 172.21.15.62:0/2238862884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635d6b05800 0x5635d6b8eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.580+0000 7fb086c52280 1 --2- 172.21.15.62:0/2238862884 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5635d6b04c00 0x5635d6b8e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.580+0000 7fb086c52280 1 -- 172.21.15.62:0/2238862884 >> 172.21.15.62:0/2238862884 conn(0x5635d6b05000 msgr2=0x5635d6bf6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.580+0000 7fb086c52280 1 -- 172.21.15.62:0/2238862884 shutdown_connections 2024-03-23T11:30:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.580+0000 7fb086c52280 1 -- 172.21.15.62:0/2238862884 wait complete. 2024-03-23T11:30:27.576 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627777', 'summary'] in time 0:00:00.125177, result: Events by type: Errors: 0 2024-03-23T11:30:27.576 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627778 summary 2024-03-23T11:30:27.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.727+0000 7f062400d280 1 -- start start 2024-03-23T11:30:27.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.727+0000 7f062400d280 1 Processor -- start 2024-03-23T11:30:27.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.728+0000 7f062400d280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b61672c00 0x561b616fc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.728+0000 7f062400d280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b61673800 0x561b616fcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.728+0000 7f062400d280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b61673c00 0x561b616fd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.728+0000 7f062400d280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b61744a00 con 0x561b61672c00 2024-03-23T11:30:27.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.728+0000 7f062400d280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b61744780 con 0x561b61673c00 2024-03-23T11:30:27.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.728+0000 7f062400d280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b61744640 con 0x561b61673800 2024-03-23T11:30:27.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.729+0000 7f0621600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b61672c00 0x561b616fc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.729+0000 7f0621600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b61672c00 0x561b616fc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49946/0 (socket says 172.21.15.62:49946) 2024-03-23T11:30:27.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.729+0000 7f0621600640 1 -- 172.21.15.62:0/540155704 learned_addr learned my addr 172.21.15.62:0/540155704 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:27.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.729+0000 7f0622000640 1 --2- 172.21.15.62:0/540155704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b61673c00 0x561b616fd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.729+0000 7f0620c00640 1 --2- 172.21.15.62:0/540155704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b61673800 0x561b616fcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.729+0000 7f0621600640 1 -- 172.21.15.62:0/540155704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b61673800 msgr2=0x561b616fcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.729+0000 7f0621600640 1 --2- 172.21.15.62:0/540155704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b61673800 0x561b616fcb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.729+0000 7f0621600640 1 -- 172.21.15.62:0/540155704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b61673c00 msgr2=0x561b616fd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.729+0000 7f0621600640 1 --2- 172.21.15.62:0/540155704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b61673c00 0x561b616fd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.729+0000 7f0621600640 1 -- 172.21.15.62:0/540155704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b616989c0 con 0x561b61672c00 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.730+0000 7f0621600640 1 --2- 172.21.15.62:0/540155704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b61672c00 0x561b616fc580 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x561b6249ed50 tx=0x561b62498e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9be26cdfedea8a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.730+0000 7f061e400640 1 -- 172.21.15.62:0/540155704 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561b616bb600 con 0x561b61672c00 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.730+0000 7f061e400640 1 -- 172.21.15.62:0/540155704 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561b616d0480 con 0x561b61672c00 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.730+0000 7f061e400640 1 -- 172.21.15.62:0/540155704 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561b616bb1e0 con 0x561b61672c00 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.730+0000 7f062400d280 1 -- 172.21.15.62:0/540155704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b61698b60 con 0x561b61672c00 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.730+0000 7f062400d280 1 -- 172.21.15.62:0/540155704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x561b61698820 con 0x561b61672c00 2024-03-23T11:30:27.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.730+0000 7f062400d280 4 waiting for MDS map... 2024-03-23T11:30:27.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.730+0000 7f061e400640 1 -- 172.21.15.62:0/540155704 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x561b6169e380 con 0x561b61672c00 2024-03-23T11:30:27.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.730+0000 7f061e400640 1 -- 172.21.15.62:0/540155704 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 79) v1 ==== 2573+0+0 (secure 0 0 0) 0x561b616c5200 con 0x561b61672c00 2024-03-23T11:30:27.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.730+0000 7f062400d280 4 Got MDS map 79 2024-03-23T11:30:27.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.730+0000 7f062400d280 10 main: JournalTool::main 2024-03-23T11:30:27.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.731+0000 7f062400d280 4 main: JournalTool: connecting to RADOS... 2024-03-23T11:30:27.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.731+0000 7f062400d280 1 Processor -- start 2024-03-23T11:30:27.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.731+0000 7f062400d280 1 -- start start 2024-03-23T11:30:27.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.732+0000 7f062400d280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547000 0x561b616fd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.732+0000 7f062400d280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b62547400 0x561b616fdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.732+0000 7f062400d280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b62547800 0x561b616fe100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.732+0000 7f062400d280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b616bde00 con 0x561b62547000 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.732+0000 7f062400d280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b61744780 con 0x561b62547800 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.732+0000 7f062400d280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b61744500 con 0x561b62547400 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.732+0000 7f0622000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547000 0x561b616fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.732+0000 7f0622000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547000 0x561b616fd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49948/0 (socket says 172.21.15.62:49948) 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.732+0000 7f0622000640 1 -- 172.21.15.62:0/2149200948 learned_addr learned my addr 172.21.15.62:0/2149200948 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.733+0000 7f0620c00640 1 --2- 172.21.15.62:0/2149200948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b62547800 0x561b616fe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.733+0000 7f0621600640 1 --2- 172.21.15.62:0/2149200948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b62547400 0x561b616fdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.733+0000 7f0622000640 1 -- 172.21.15.62:0/2149200948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b62547400 msgr2=0x561b616fdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.733+0000 7f0622000640 1 --2- 172.21.15.62:0/2149200948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b62547400 0x561b616fdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.733+0000 7f0622000640 1 -- 172.21.15.62:0/2149200948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b62547800 msgr2=0x561b616fe100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.733+0000 7f0622000640 1 --2- 172.21.15.62:0/2149200948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b62547800 0x561b616fe100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.733+0000 7f0622000640 1 -- 172.21.15.62:0/2149200948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b616984e0 con 0x561b62547000 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.733+0000 7f0621600640 1 --2- 172.21.15.62:0/2149200948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b62547400 0x561b616fdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.733+0000 7f0622000640 1 --2- 172.21.15.62:0/2149200948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547000 0x561b616fd600 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x561b625aa060 tx=0x561b62558c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358227f79213ee3a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f061bc00640 1 -- 172.21.15.62:0/2149200948 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561b616bb600 con 0x561b62547000 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f061bc00640 1 -- 172.21.15.62:0/2149200948 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561b616d0900 con 0x561b62547000 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f061bc00640 1 -- 172.21.15.62:0/2149200948 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561b616bb1e0 con 0x561b62547000 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f062400d280 1 -- 172.21.15.62:0/2149200948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547000 msgr2=0x561b616fd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f062400d280 1 --2- 172.21.15.62:0/2149200948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547000 0x561b616fd600 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x561b625aa060 tx=0x561b62558c00 comp rx=0 tx=0).stop 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f062400d280 1 -- 172.21.15.62:0/2149200948 shutdown_connections 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f062400d280 1 --2- 172.21.15.62:0/2149200948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b62547800 0x561b616fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f062400d280 1 --2- 172.21.15.62:0/2149200948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b62547400 0x561b616fdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f062400d280 1 --2- 172.21.15.62:0/2149200948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547000 0x561b616fd600 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f062400d280 1 -- 172.21.15.62:0/2149200948 >> 172.21.15.62:0/2149200948 conn(0x561b62546c00 msgr2=0x561b623ef000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:27.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f062400d280 1 -- 172.21.15.62:0/2149200948 shutdown_connections 2024-03-23T11:30:27.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.734+0000 7f062400d280 1 -- 172.21.15.62:0/2149200948 wait complete. 2024-03-23T11:30:27.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.735+0000 7f062400d280 1 Processor -- start 2024-03-23T11:30:27.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.735+0000 7f062400d280 1 -- start start 2024-03-23T11:30:27.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.736+0000 7f062400d280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b62547000 0x561b616ff180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.736+0000 7f062400d280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547400 0x561b616fec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.736+0000 7f062400d280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b62547800 0x561b616fe680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.736+0000 7f062400d280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b617443c0 con 0x561b62547400 2024-03-23T11:30:27.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.736+0000 7f062400d280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b61744280 con 0x561b62547800 2024-03-23T11:30:27.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.736+0000 7f062400d280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b61744140 con 0x561b62547000 2024-03-23T11:30:27.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.736+0000 7f0621600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547400 0x561b616fec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.736+0000 7f0621600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547400 0x561b616fec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49956/0 (socket says 172.21.15.62:49956) 2024-03-23T11:30:27.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.736+0000 7f0621600640 1 -- 172.21.15.62:0/471066928 learned_addr learned my addr 172.21.15.62:0/471066928 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:27.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.736+0000 7f0622000640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b62547000 0x561b616ff180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f0621600640 1 -- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b62547000 msgr2=0x561b616ff180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.736+0000 7f0620c00640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b62547800 0x561b616fe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f0621600640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b62547000 0x561b616ff180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f0621600640 1 -- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b62547800 msgr2=0x561b616fe680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f0621600640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b62547800 0x561b616fe680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f0621600640 1 -- 172.21.15.62:0/471066928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b616989c0 con 0x561b62547400 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f0622000640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b62547000 0x561b616ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f0621600640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547400 0x561b616fec00 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x561b625aadb0 tx=0x561b62406300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f0619e00640 1 -- 172.21.15.62:0/471066928 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561b616baf20 con 0x561b62547400 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f0619e00640 1 -- 172.21.15.62:0/471066928 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561b62450c00 con 0x561b62547400 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f0619e00640 1 -- 172.21.15.62:0/471066928 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561b616bb600 con 0x561b62547400 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561b61698b60 con 0x561b62547400 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b616981a0 con 0x561b62547400 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.737+0000 7f062400d280 4 main: JournalTool: resolving pool 14 2024-03-23T11:30:27.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.739+0000 7f0619e00640 1 -- 172.21.15.62:0/471066928 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x561b61741080 con 0x561b62547400 2024-03-23T11:30:27.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.739+0000 7f0619e00640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561b62465800 0x561b616ff700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.740+0000 7f0619e00640 1 -- 172.21.15.62:0/471066928 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x561b6169e540 con 0x561b62547400 2024-03-23T11:30:27.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.740+0000 7f062400d280 4 main: JournalTool: creating IoCtx.. 2024-03-23T11:30:27.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.740+0000 7f062400d280 4 main: Executing for rank 0 2024-03-23T11:30:27.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.740+0000 7f062400d280 4 Filtering by inode '1099511627778' 2024-03-23T11:30:27.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.740+0000 7f062400d280 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x561b62465c00 0x561b616fe100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.740+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:1 14.f 14:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x561b619d4000 con 0x561b62465c00 2024-03-23T11:30:27.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.740+0000 7f0622000640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561b62465800 0x561b616ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.741+0000 7f0620c00640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x561b62465c00 0x561b616fe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.742+0000 7f0622000640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561b62465800 0x561b616ff700 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x561b625aa060 tx=0x561b62498d80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.742+0000 7f0620c00640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x561b62465c00 0x561b616fe100 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.742+0000 7f0620c00640 1 -- 172.21.15.62:0/471066928 <== osd.7 v2:172.21.15.120:6816/37655 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x561b6252cb40 con 0x561b62465c00 2024-03-23T11:30:27.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.742+0000 7f062400d280 4 Pointer 400.00000000 is readable 2024-03-23T11:30:27.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.742+0000 7f062400d280 4 JournalScanner::scan: reading header object '200.00000000' 2024-03-23T11:30:27.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.742+0000 7f062400d280 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x561b619d4400 0x561b616fdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.742+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:2 14.14 14:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x561b619d4800 con 0x561b619d4400 2024-03-23T11:30:27.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.743+0000 7f0622000640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x561b619d4400 0x561b616fdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.744+0000 7f0622000640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x561b619d4400 0x561b616fdb80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.745+0000 7f0622000640 1 -- 172.21.15.62:0/471066928 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv31 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x561b6252cd80 con 0x561b619d4400 2024-03-23T11:30:27.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.745+0000 7f062400d280 10 Header 0x400000 0x405ee0 0x40625c 2024-03-23T11:30:27.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.745+0000 7f062400d280 10 Starting journal scan from offset 0x405ee0 2024-03-23T11:30:27.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.745+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:3 14.14 14:2e2fa760:::200.00000001:head [read 24288~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x561b619d4c00 con 0x561b619d4400 2024-03-23T11:30:27.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.745+0000 7f0622000640 1 -- 172.21.15.62:0/471066928 <== osd.4 v2:172.21.15.120:6803/37640 2 ==== osd_op_reply(3 200.00000001 [read 24288~892 out=892b] v0'0 uv27 ondisk = 0) v8 ==== 156+0+892 (crc 0 0 0) 0x561b6252cd80 con 0x561b619d4400 2024-03-23T11:30:27.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.746+0000 7f062400d280 4 Read 0x37c bytes from 200.00000001 gap=0 2024-03-23T11:30:27.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.746+0000 7f062400d280 10 Parsing data, 0x37c bytes available 2024-03-23T11:30:27.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.746+0000 7f062400d280 10 Attempting decode at 0x405ee0 2024-03-23T11:30:27.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.746+0000 7f062400d280 10 Consumed 0x37c bytes 2024-03-23T11:30:27.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.746+0000 7f062400d280 10 Valid entry at 0x405ee0 2024-03-23T11:30:27.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.746+0000 7f062400d280 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x561b619d5000 0x561b616fd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.746+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:4 14.1b 14:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x561b619d5400 con 0x561b619d5000 2024-03-23T11:30:27.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.746+0000 7f0621600640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x561b619d5000 0x561b616fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.746+0000 7f0621600640 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x561b619d5000 0x561b616fd600 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f0621600640 1 -- 172.21.15.62:0/471066928 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561b6252cfc0 con 0x561b619d5000 2024-03-23T11:30:27.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 4 Reached end of journal objects 2024-03-23T11:30:27.738 INFO:teuthology.orchestra.run.smithi062.stdout:Events by type: 2024-03-23T11:30:27.738 INFO:teuthology.orchestra.run.smithi062.stdout:Errors: 0 2024-03-23T11:30:27.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 4 Scanned objects, 0 missing, 1 valid 2024-03-23T11:30:27.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 4 Events scanned, 0 gaps 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 4 Found 1 valid events 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 4 Selected 0 events events for processing 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 4 0 events 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x561b619d5000 msgr2=0x561b616fd600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x561b619d5000 0x561b616fd600 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x561b619d4400 msgr2=0x561b616fdb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x561b619d4400 0x561b616fdb80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x561b62465c00 msgr2=0x561b616fe100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x561b62465c00 0x561b616fe100 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f0620c00640 1 -- 172.21.15.62:0/471066928 reap_dead start 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561b62465800 msgr2=0x561b616ff700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561b62465800 0x561b616ff700 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x561b625aa060 tx=0x561b62498d80 comp rx=0 tx=0).stop 2024-03-23T11:30:27.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547400 msgr2=0x561b616fec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547400 0x561b616fec00 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x561b625aadb0 tx=0x561b62406300 comp rx=0 tx=0).stop 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 shutdown_connections 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561b62465800 0x561b616ff700 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 --2- 172.21.15.62:0/471066928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b62547400 0x561b616fec00 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 >> 172.21.15.62:0/471066928 conn(0x561b62546c00 msgr2=0x561b623ef000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 shutdown_connections 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.747+0000 7f062400d280 1 -- 172.21.15.62:0/471066928 wait complete. 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.748+0000 7f062400d280 1 -- 172.21.15.62:0/540155704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b61672c00 msgr2=0x561b616fc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.748+0000 7f062400d280 1 --2- 172.21.15.62:0/540155704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b61672c00 0x561b616fc580 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x561b6249ed50 tx=0x561b62498e40 comp rx=0 tx=0).stop 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.748+0000 7f062400d280 1 -- 172.21.15.62:0/540155704 shutdown_connections 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.748+0000 7f062400d280 1 --2- 172.21.15.62:0/540155704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b61673c00 0x561b616fd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.748+0000 7f062400d280 1 --2- 172.21.15.62:0/540155704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561b61673800 0x561b616fcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.748+0000 7f062400d280 1 --2- 172.21.15.62:0/540155704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b61672c00 0x561b616fc580 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.748+0000 7f062400d280 1 -- 172.21.15.62:0/540155704 >> 172.21.15.62:0/540155704 conn(0x561b61673000 msgr2=0x561b623ee800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:27.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.748+0000 7f062400d280 1 -- 172.21.15.62:0/540155704 shutdown_connections 2024-03-23T11:30:27.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.748+0000 7f062400d280 1 -- 172.21.15.62:0/540155704 wait complete. 2024-03-23T11:30:27.744 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627778', 'summary'] in time 0:00:00.167714, result: Events by type: Errors: 0 2024-03-23T11:30:27.744 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds0_inotable - 2024-03-23T11:30:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.844+0000 7f6fc040cf40 1 Processor -- start 2024-03-23T11:30:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.844+0000 7f6fc040cf40 1 -- start start 2024-03-23T11:30:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.844+0000 7f6fc040cf40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fa0002810 0x559308445270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.844+0000 7f6fc040cf40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55930839d9d0 0x559308442c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fc040cf40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593084d4770 0x5593084d4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fc040cf40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593082b5800 con 0x7f6fa0002810 2024-03-23T11:30:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fc040cf40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593082b54a0 con 0x5593084d4770 2024-03-23T11:30:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fc040cf40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5593082ac110 con 0x55930839d9d0 2024-03-23T11:30:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fbcc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fa0002810 0x559308445270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fbcc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fa0002810 0x559308445270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49978/0 (socket says 172.21.15.62:49978) 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fbcc00640 1 -- 172.21.15.62:0/1762692411 learned_addr learned my addr 172.21.15.62:0/1762692411 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fbd600640 1 --2- 172.21.15.62:0/1762692411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593084d4770 0x5593084d4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fbcc00640 1 -- 172.21.15.62:0/1762692411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55930839d9d0 msgr2=0x559308442c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fbcc00640 1 --2- 172.21.15.62:0/1762692411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55930839d9d0 0x559308442c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fbcc00640 1 -- 172.21.15.62:0/1762692411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593084d4770 msgr2=0x5593084d4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fbcc00640 1 --2- 172.21.15.62:0/1762692411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593084d4770 0x5593084d4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fbcc00640 1 -- 172.21.15.62:0/1762692411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593082c4340 con 0x7f6fa0002810 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.845+0000 7f6fbcc00640 1 --2- 172.21.15.62:0/1762692411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fa0002810 0x559308445270 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f6fac006970 tx=0x7f6fac006e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889f85fd1faf89d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fb7400640 1 -- 172.21.15.62:0/1762692411 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac00c070 con 0x7f6fa0002810 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fb7400640 1 -- 172.21.15.62:0/1762692411 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac00ea10 con 0x7f6fa0002810 2024-03-23T11:30:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fb7400640 1 -- 172.21.15.62:0/1762692411 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac00edc0 con 0x7f6fa0002810 2024-03-23T11:30:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fc040cf40 1 -- 172.21.15.62:0/1762692411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fa0002810 msgr2=0x559308445270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fc040cf40 1 --2- 172.21.15.62:0/1762692411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fa0002810 0x559308445270 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f6fac006970 tx=0x7f6fac006e30 comp rx=0 tx=0).stop 2024-03-23T11:30:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fc040cf40 1 -- 172.21.15.62:0/1762692411 shutdown_connections 2024-03-23T11:30:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fc040cf40 1 --2- 172.21.15.62:0/1762692411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593084d4770 0x5593084d4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fc040cf40 1 --2- 172.21.15.62:0/1762692411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55930839d9d0 0x559308442c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fc040cf40 1 --2- 172.21.15.62:0/1762692411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fa0002810 0x559308445270 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fc040cf40 1 -- 172.21.15.62:0/1762692411 >> 172.21.15.62:0/1762692411 conn(0x55930836b6f0 msgr2=0x5593083b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fc040cf40 1 -- 172.21.15.62:0/1762692411 shutdown_connections 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.846+0000 7f6fc040cf40 1 -- 172.21.15.62:0/1762692411 wait complete. 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.847+0000 7f6fc040cf40 1 Processor -- start 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.847+0000 7f6fc040cf40 1 -- start start 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.848+0000 7f6fc040cf40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55930839d9d0 0x5593084cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.848+0000 7f6fc040cf40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5593084d4770 0x55930842c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.848+0000 7f6fc040cf40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fa0002810 0x55930836df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.848+0000 7f6fc040cf40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593084435d0 con 0x55930839d9d0 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.848+0000 7f6fc040cf40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593084438b0 con 0x7f6fa0002810 2024-03-23T11:30:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.848+0000 7f6fc040cf40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559308443b90 con 0x5593084d4770 2024-03-23T11:30:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.848+0000 7f6fbcc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55930839d9d0 0x5593084cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.849+0000 7f6fbcc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55930839d9d0 0x5593084cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49984/0 (socket says 172.21.15.62:49984) 2024-03-23T11:30:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.849+0000 7f6fbcc00640 1 -- 172.21.15.62:0/41365995 learned_addr learned my addr 172.21.15.62:0/41365995 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.849+0000 7f6fbd600640 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fa0002810 0x55930836df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.849+0000 7f6fbcc00640 1 -- 172.21.15.62:0/41365995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5593084d4770 msgr2=0x55930842c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.849+0000 7f6fbcc00640 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5593084d4770 0x55930842c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.849+0000 7f6fbcc00640 1 -- 172.21.15.62:0/41365995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fa0002810 msgr2=0x55930836df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.849+0000 7f6fbcc00640 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fa0002810 0x55930836df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.849+0000 7f6fbcc00640 1 -- 172.21.15.62:0/41365995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593082de4e0 con 0x55930839d9d0 2024-03-23T11:30:27.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.849+0000 7f6fbcc00640 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55930839d9d0 0x5593084cc160 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f6fac004510 tx=0x7f6fac001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.849+0000 7f6fb6000640 1 -- 172.21.15.62:0/41365995 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac001b20 con 0x55930839d9d0 2024-03-23T11:30:27.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.850+0000 7f6fb6000640 1 -- 172.21.15.62:0/41365995 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac013d30 con 0x55930839d9d0 2024-03-23T11:30:27.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.850+0000 7f6fb6000640 1 -- 172.21.15.62:0/41365995 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac013ea0 con 0x55930839d9d0 2024-03-23T11:30:27.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.850+0000 7f6fc040cf40 1 -- 172.21.15.62:0/41365995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5593082b7780 con 0x55930839d9d0 2024-03-23T11:30:27.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.850+0000 7f6fc040cf40 1 -- 172.21.15.62:0/41365995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5593082b71e0 con 0x55930839d9d0 2024-03-23T11:30:27.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.852+0000 7f6fb6000640 1 -- 172.21.15.62:0/41365995 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6fac01f670 con 0x55930839d9d0 2024-03-23T11:30:27.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.852+0000 7f6fb6000640 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6f98069180 0x7f6f9806b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.853+0000 7f6fb6000640 1 -- 172.21.15.62:0/41365995 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6fac08b3c0 con 0x55930839d9d0 2024-03-23T11:30:27.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.853+0000 7f6fc040cf40 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55930842cc80 0x559308437b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:27.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.853+0000 7f6fc040cf40 1 -- 172.21.15.62:0/41365995 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:1 14.13 14:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) v8 -- 0x559308430600 con 0x55930842cc80 2024-03-23T11:30:27.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.853+0000 7f6fb4200640 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6f98069180 0x7f6f9806b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.853+0000 7f6fbd600640 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55930842cc80 0x559308437b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:27.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.854+0000 7f6fbd600640 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55930842cc80 0x559308437b20 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.854+0000 7f6fb4200640 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6f98069180 0x7f6f9806b640 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400a7a0 tx=0x7f6fa4002fb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:27.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.857+0000 7f6fbd600640 1 -- 172.21.15.62:0/41365995 <== osd.2 v2:172.21.15.62:6800/36996 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~34] v106'5 uv5 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6fb8009470 con 0x55930842cc80 2024-03-23T11:30:27.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.858+0000 7f6fc040cf40 1 -- 172.21.15.62:0/41365995 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55930842cc80 msgr2=0x559308437b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.858+0000 7f6fc040cf40 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55930842cc80 0x559308437b20 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.858+0000 7f6fc040cf40 1 -- 172.21.15.62:0/41365995 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6f98069180 msgr2=0x7f6f9806b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.858+0000 7f6fc040cf40 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6f98069180 0x7f6f9806b640 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400a7a0 tx=0x7f6fa4002fb0 comp rx=0 tx=0).stop 2024-03-23T11:30:27.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.858+0000 7f6fc040cf40 1 -- 172.21.15.62:0/41365995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55930839d9d0 msgr2=0x5593084cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:27.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.858+0000 7f6fc040cf40 1 --2- 172.21.15.62:0/41365995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55930839d9d0 0x5593084cc160 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f6fac004510 tx=0x7f6fac001d80 comp rx=0 tx=0).stop 2024-03-23T11:30:27.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.858+0000 7f6fbd600640 1 -- 172.21.15.62:0/41365995 reap_dead start 2024-03-23T11:30:27.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.858+0000 7f6fc040cf40 1 -- 172.21.15.62:0/41365995 shutdown_connections 2024-03-23T11:30:27.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.858+0000 7f6fc040cf40 1 -- 172.21.15.62:0/41365995 >> 172.21.15.62:0/41365995 conn(0x55930836b6f0 msgr2=0x55930836bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:27.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.858+0000 7f6fc040cf40 1 -- 172.21.15.62:0/41365995 shutdown_connections 2024-03-23T11:30:27.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:27.858+0000 7f6fc040cf40 1 -- 172.21.15.62:0/41365995 wait complete. 2024-03-23T11:30:27.853 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-23T11:30:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.077+0000 7f61a2800640 1 Processor -- start 2024-03-23T11:30:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.078+0000 7f61a2800640 1 -- start start 2024-03-23T11:30:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.079+0000 7f61a2800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c15fcf0 0x7f619c0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.079+0000 7f61a2800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f619c0b6dc0 0x7f619c160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.079+0000 7f61a2800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c160dc0 con 0x7f619c160420 2024-03-23T11:30:28.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.079+0000 7f61a2800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619c0d3540 con 0x7f619c0b6dc0 2024-03-23T11:30:28.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.079+0000 7f61a2800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c0ce510 con 0x7f619c15fcf0 2024-03-23T11:30:28.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.080+0000 7f61a1e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f619c0b6dc0 0x7f619c160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.080+0000 7f61a1e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f619c0b6dc0 0x7f619c160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50000/0 (socket says 172.21.15.62:50000) 2024-03-23T11:30:28.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.080+0000 7f61a1e00640 1 -- 172.21.15.62:0/2231350116 learned_addr learned my addr 172.21.15.62:0/2231350116 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:28.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.080+0000 7f61a1400640 1 --2- 172.21.15.62:0/2231350116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c15fcf0 0x7f619c0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.080+0000 7f61a1e00640 1 -- 172.21.15.62:0/2231350116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c15fcf0 msgr2=0x7f619c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:28.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.080+0000 7f61a1e00640 1 --2- 172.21.15.62:0/2231350116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c15fcf0 0x7f619c0d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.080+0000 7f61a1e00640 1 -- 172.21.15.62:0/2231350116 >> v1:172.21.15.120:6789/0 conn(0x7f619c160420 legacy=0x7f619c0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:28.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.080+0000 7f61a1e00640 1 -- 172.21.15.62:0/2231350116 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c0cdc70 con 0x7f619c0b6dc0 2024-03-23T11:30:28.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.082+0000 7f61a1e00640 1 --2- 172.21.15.62:0/2231350116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f619c0b6dc0 0x7f619c160880 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f6194009e50 tx=0x7f619402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba3598bd40c4d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:28.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.082+0000 7f619be00640 1 -- 172.21.15.62:0/2231350116 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619402c9e0 con 0x7f619c0b6dc0 2024-03-23T11:30:28.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.082+0000 7f619be00640 1 -- 172.21.15.62:0/2231350116 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619402cb40 con 0x7f619c0b6dc0 2024-03-23T11:30:28.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.082+0000 7f619be00640 1 -- 172.21.15.62:0/2231350116 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61940343f0 con 0x7f619c0b6dc0 2024-03-23T11:30:28.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.083+0000 7f61a2800640 1 -- 172.21.15.62:0/2231350116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f619c0b6dc0 msgr2=0x7f619c160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:28.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.083+0000 7f61a2800640 1 --2- 172.21.15.62:0/2231350116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f619c0b6dc0 0x7f619c160880 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f6194009e50 tx=0x7f619402be40 comp rx=0 tx=0).stop 2024-03-23T11:30:28.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.083+0000 7f61a2800640 1 -- 172.21.15.62:0/2231350116 shutdown_connections 2024-03-23T11:30:28.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.083+0000 7f61a2800640 1 --2- 172.21.15.62:0/2231350116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f619c0b6dc0 0x7f619c160880 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.083+0000 7f61a2800640 1 --2- 172.21.15.62:0/2231350116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c15fcf0 0x7f619c0d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.083+0000 7f61a2800640 1 -- 172.21.15.62:0/2231350116 >> 172.21.15.62:0/2231350116 conn(0x7f619c089f60 msgr2=0x7f619c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:28.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.083+0000 7f61a2800640 1 -- 172.21.15.62:0/2231350116 shutdown_connections 2024-03-23T11:30:28.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.083+0000 7f61a2800640 1 -- 172.21.15.62:0/2231350116 wait complete. 2024-03-23T11:30:28.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.085+0000 7f61a2800640 1 Processor -- start 2024-03-23T11:30:28.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.085+0000 7f61a2800640 1 -- start start 2024-03-23T11:30:28.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.086+0000 7f61a2800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0b6dc0 0x7f619c1f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.086+0000 7f61a2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f619c15fcf0 0x7f619c158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.086+0000 7f61a2800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c160420 0x7f619c159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.086+0000 7f61a2800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c159a80 con 0x7f619c15fcf0 2024-03-23T11:30:28.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.086+0000 7f61a2800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c1f6940 con 0x7f619c160420 2024-03-23T11:30:28.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.086+0000 7f61a2800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c1596e0 con 0x7f619c0b6dc0 2024-03-23T11:30:28.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.086+0000 7f61a0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f619c15fcf0 0x7f619c158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.086+0000 7f61a0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f619c15fcf0 0x7f619c158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50014/0 (socket says 172.21.15.62:50014) 2024-03-23T11:30:28.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.086+0000 7f61a0a00640 1 -- 172.21.15.62:0/3809746686 learned_addr learned my addr 172.21.15.62:0/3809746686 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:28.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.087+0000 7f61a1400640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0b6dc0 0x7f619c1f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.087+0000 7f61a1e00640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c160420 0x7f619c159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.087+0000 7f61a0a00640 1 -- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0b6dc0 msgr2=0x7f619c1f6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:28.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.087+0000 7f61a0a00640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0b6dc0 0x7f619c1f6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.087+0000 7f61a0a00640 1 -- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c160420 msgr2=0x7f619c159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:28.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.087+0000 7f61a0a00640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c160420 0x7f619c159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.087+0000 7f61a0a00640 1 -- 172.21.15.62:0/3809746686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6194009b30 con 0x7f619c15fcf0 2024-03-23T11:30:28.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.087+0000 7f61a0a00640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f619c15fcf0 0x7f619c158e00 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f618800c480 tx=0x7f618800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:28.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.088+0000 7f6199600640 1 -- 172.21.15.62:0/3809746686 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61880052d0 con 0x7f619c15fcf0 2024-03-23T11:30:28.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.088+0000 7f6199600640 1 -- 172.21.15.62:0/3809746686 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6188005430 con 0x7f619c15fcf0 2024-03-23T11:30:28.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.088+0000 7f6199600640 1 -- 172.21.15.62:0/3809746686 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188012870 con 0x7f619c15fcf0 2024-03-23T11:30:28.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.088+0000 7f61a2800640 1 -- 172.21.15.62:0/3809746686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619c09fa20 con 0x7f619c15fcf0 2024-03-23T11:30:28.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.088+0000 7f61a2800640 1 -- 172.21.15.62:0/3809746686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c09ffe0 con 0x7f619c15fcf0 2024-03-23T11:30:28.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.089+0000 7f61a2800640 1 -- 172.21.15.62:0/3809746686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619c159bc0 con 0x7f619c15fcf0 2024-03-23T11:30:28.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.090+0000 7f6199600640 1 -- 172.21.15.62:0/3809746686 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f61880129d0 con 0x7f619c15fcf0 2024-03-23T11:30:28.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.091+0000 7f6199600640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f617c06a7a0 0x7f617c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.091+0000 7f6199600640 1 -- 172.21.15.62:0/3809746686 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f61880889a0 con 0x7f619c15fcf0 2024-03-23T11:30:28.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.091+0000 7f61a1400640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f617c06a7a0 0x7f617c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.092+0000 7f61a1400640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f617c06a7a0 0x7f617c06cc60 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6190006fd0 tx=0x7f6190008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:28.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.095+0000 7f6199600640 1 -- 172.21.15.62:0/3809746686 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f619c159bc0 con 0x7f619c15fcf0 2024-03-23T11:30:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.337+0000 7f61a2800640 1 -- 172.21.15.62:0/3809746686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f619c159bc0 con 0x7f619c15fcf0 2024-03-23T11:30:28.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.493+0000 7f6199600640 1 -- 172.21.15.62:0/3809746686 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v80) v1 ==== 162+0+0 (secure 0 0 0) 0x7f619c159bc0 con 0x7f619c15fcf0 2024-03-23T11:30:28.484 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-23T11:30:28.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.496+0000 7f61a2800640 1 -- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f617c06a7a0 msgr2=0x7f617c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:28.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.496+0000 7f61a2800640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f617c06a7a0 0x7f617c06cc60 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6190006fd0 tx=0x7f6190008040 comp rx=0 tx=0).stop 2024-03-23T11:30:28.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.496+0000 7f61a2800640 1 -- 172.21.15.62:0/3809746686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f619c15fcf0 msgr2=0x7f619c158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:28.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.496+0000 7f61a2800640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f619c15fcf0 0x7f619c158e00 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f618800c480 tx=0x7f618800c940 comp rx=0 tx=0).stop 2024-03-23T11:30:28.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.496+0000 7f61a2800640 1 -- 172.21.15.62:0/3809746686 shutdown_connections 2024-03-23T11:30:28.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.497+0000 7f61a2800640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f617c06a7a0 0x7f617c06cc60 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.497+0000 7f61a2800640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f619c15fcf0 0x7f619c158e00 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.497+0000 7f61a2800640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f619c160420 0x7f619c159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.497+0000 7f61a2800640 1 --2- 172.21.15.62:0/3809746686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f619c0b6dc0 0x7f619c1f6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.497+0000 7f61a2800640 1 -- 172.21.15.62:0/3809746686 >> 172.21.15.62:0/3809746686 conn(0x7f619c089f60 msgr2=0x7f619c06c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.497+0000 7f61a2800640 1 -- 172.21.15.62:0/3809746686 shutdown_connections 2024-03-23T11:30:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.497+0000 7f61a2800640 1 -- 172.21.15.62:0/3809746686 wait complete. 2024-03-23T11:30:28.511 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:30:28.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.718+0000 7f964ea00640 1 Processor -- start 2024-03-23T11:30:28.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.718+0000 7f964ea00640 1 -- start start 2024-03-23T11:30:28.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.720+0000 7f964ea00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f96480b2f90 0x7f9648161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.720+0000 7f964ea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f96480b15e0 0x7f96480b3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.720+0000 7f964ea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9648164a30 0x7f9648164ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.720+0000 7f964ea00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96480ae0c0 con 0x7f96480b15e0 2024-03-23T11:30:28.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.720+0000 7f964ea00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96480b2740 con 0x7f96480b2f90 2024-03-23T11:30:28.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.720+0000 7f964ea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96481623b0 con 0x7f9648164a30 2024-03-23T11:30:28.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.720+0000 7f963f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f96480b15e0 0x7f96480b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.720+0000 7f963f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f96480b15e0 0x7f96480b3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50016/0 (socket says 172.21.15.62:50016) 2024-03-23T11:30:28.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.720+0000 7f963f400640 1 -- 172.21.15.62:0/1789014806 learned_addr learned my addr 172.21.15.62:0/1789014806 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:28.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.721+0000 7f9647e00640 1 --2- 172.21.15.62:0/1789014806 >> v2:172.21.15.120:3300/0 conn(0x7f96480b2f90 0x7f9648161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.721+0000 7f964ca00640 1 --2- 172.21.15.62:0/1789014806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9648164a30 0x7f9648164ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.721+0000 7f963f400640 1 -- 172.21.15.62:0/1789014806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9648164a30 msgr2=0x7f9648164ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.721+0000 7f963f400640 1 --2- 172.21.15.62:0/1789014806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9648164a30 0x7f9648164ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.721+0000 7f963f400640 1 -- 172.21.15.62:0/1789014806 >> v2:172.21.15.120:3300/0 conn(0x7f96480b2f90 msgr2=0x7f9648161e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.721+0000 7f963f400640 1 --2- 172.21.15.62:0/1789014806 >> v2:172.21.15.120:3300/0 conn(0x7f96480b2f90 0x7f9648161e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.721+0000 7f963f400640 1 -- 172.21.15.62:0/1789014806 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96480b1100 con 0x7f96480b15e0 2024-03-23T11:30:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.721+0000 7f9647e00640 1 --2- 172.21.15.62:0/1789014806 >> v2:172.21.15.120:3300/0 conn(0x7f96480b2f90 0x7f9648161e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.721+0000 7f964ca00640 1 --2- 172.21.15.62:0/1789014806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9648164a30 0x7f9648164ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.722+0000 7f963f400640 1 --2- 172.21.15.62:0/1789014806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96480b15e0 0x7f96480b3370 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f96480719e0 tx=0x7f963002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fa52f5bcd5990 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.722+0000 7f9647400640 1 -- 172.21.15.62:0/1789014806 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630031870 con 0x7f96480b15e0 2024-03-23T11:30:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.722+0000 7f9647400640 1 -- 172.21.15.62:0/1789014806 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96300319d0 con 0x7f96480b15e0 2024-03-23T11:30:28.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.722+0000 7f9647400640 1 -- 172.21.15.62:0/1789014806 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630031d60 con 0x7f96480b15e0 2024-03-23T11:30:28.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.723+0000 7f964ea00640 1 -- 172.21.15.62:0/1789014806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96480b15e0 msgr2=0x7f96480b3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:28.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.723+0000 7f964ea00640 1 --2- 172.21.15.62:0/1789014806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96480b15e0 0x7f96480b3370 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f96480719e0 tx=0x7f963002b910 comp rx=0 tx=0).stop 2024-03-23T11:30:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.723+0000 7f964ea00640 1 -- 172.21.15.62:0/1789014806 shutdown_connections 2024-03-23T11:30:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.723+0000 7f964ea00640 1 --2- 172.21.15.62:0/1789014806 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9648164a30 0x7f9648164ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.723+0000 7f964ea00640 1 --2- 172.21.15.62:0/1789014806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96480b15e0 0x7f96480b3370 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.723+0000 7f964ea00640 1 --2- 172.21.15.62:0/1789014806 >> v2:172.21.15.120:3300/0 conn(0x7f96480b2f90 0x7f9648161e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.723+0000 7f964ea00640 1 -- 172.21.15.62:0/1789014806 >> 172.21.15.62:0/1789014806 conn(0x7f96480b02e0 msgr2=0x7f96480b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.723+0000 7f964ea00640 1 -- 172.21.15.62:0/1789014806 shutdown_connections 2024-03-23T11:30:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.723+0000 7f964ea00640 1 -- 172.21.15.62:0/1789014806 wait complete. 2024-03-23T11:30:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.725+0000 7f964ea00640 1 Processor -- start 2024-03-23T11:30:28.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.725+0000 7f964ea00640 1 -- start start 2024-03-23T11:30:28.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.726+0000 7f964ea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96480b15e0 0x7f9648153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.726+0000 7f964ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96480b2f90 0x7f964814e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.726+0000 7f964ea00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9648164a30 0x7f964814e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.726+0000 7f964ea00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96480cd930 con 0x7f96480b2f90 2024-03-23T11:30:28.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.726+0000 7f964ea00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9648152b50 con 0x7f9648164a30 2024-03-23T11:30:28.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.726+0000 7f964ea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9648152e30 con 0x7f96480b15e0 2024-03-23T11:30:28.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.726+0000 7f963f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96480b2f90 0x7f964814e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.726+0000 7f963f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96480b2f90 0x7f964814e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50022/0 (socket says 172.21.15.62:50022) 2024-03-23T11:30:28.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.726+0000 7f963f400640 1 -- 172.21.15.62:0/3505764694 learned_addr learned my addr 172.21.15.62:0/3505764694 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:28.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f964ca00640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9648164a30 0x7f964814e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f963f400640 1 -- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96480b15e0 msgr2=0x7f9648153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:28.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f963f400640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96480b15e0 0x7f9648153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f963f400640 1 -- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9648164a30 msgr2=0x7f964814e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:28.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f963f400640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9648164a30 0x7f964814e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f963f400640 1 -- 172.21.15.62:0/3505764694 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96300094d0 con 0x7f96480b2f90 2024-03-23T11:30:28.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f964ca00640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9648164a30 0x7f964814e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:28.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f963f400640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96480b2f90 0x7f964814e0e0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f963002b950 tx=0x7f96300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:28.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f9644c00640 1 -- 172.21.15.62:0/3505764694 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630031860 con 0x7f96480b2f90 2024-03-23T11:30:28.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f9644c00640 1 -- 172.21.15.62:0/3505764694 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96300443f0 con 0x7f96480b2f90 2024-03-23T11:30:28.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f964ea00640 1 -- 172.21.15.62:0/3505764694 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9648153110 con 0x7f96480b2f90 2024-03-23T11:30:28.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f9644c00640 1 -- 172.21.15.62:0/3505764694 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96300447a0 con 0x7f96480b2f90 2024-03-23T11:30:28.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.727+0000 7f964ea00640 1 -- 172.21.15.62:0/3505764694 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96481536d0 con 0x7f96480b2f90 2024-03-23T11:30:28.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.729+0000 7f964ea00640 1 -- 172.21.15.62:0/3505764694 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9648073860 con 0x7f96480b2f90 2024-03-23T11:30:28.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.730+0000 7f9644c00640 1 -- 172.21.15.62:0/3505764694 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f963004d450 con 0x7f96480b2f90 2024-03-23T11:30:28.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.731+0000 7f9644c00640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f962406a7a0 0x7f962406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:28.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.731+0000 7f9644c00640 1 -- 172.21.15.62:0/3505764694 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f963002d040 con 0x7f96480b2f90 2024-03-23T11:30:28.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.731+0000 7f9647e00640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f962406a7a0 0x7f962406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:28.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.733+0000 7f9647e00640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f962406a7a0 0x7f962406cc60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f963800a8b0 tx=0x7f9638008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.735+0000 7f9644c00640 1 -- 172.21.15.62:0/3505764694 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9630076a00 con 0x7f96480b2f90 2024-03-23T11:30:28.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.971+0000 7f964ea00640 1 -- 172.21.15.62:0/3505764694 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9648072270 con 0x7f96480b2f90 2024-03-23T11:30:28.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.972+0000 7f9644c00640 1 -- 172.21.15.62:0/3505764694 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4744 (secure 0 0 0) 0x7f963007a020 con 0x7f96480b2f90 2024-03-23T11:30:28.963 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:30:28.963 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":81,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:30:28.494245+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8691},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8691":{"gid":8691,"name":"a","rank":0,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.62:6834/1070252332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1070252332},{"type":"v1","addr":"172.21.15.62:6834","nonce":1070252332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:30:28.963 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 81 2024-03-23T11:30:28.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 -- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f962406a7a0 msgr2=0x7f962406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:28.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f962406a7a0 0x7f962406cc60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f963800a8b0 tx=0x7f9638008040 comp rx=0 tx=0).stop 2024-03-23T11:30:28.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 -- 172.21.15.62:0/3505764694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96480b2f90 msgr2=0x7f964814e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:28.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96480b2f90 0x7f964814e0e0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f963002b950 tx=0x7f96300046c0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 -- 172.21.15.62:0/3505764694 shutdown_connections 2024-03-23T11:30:28.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f962406a7a0 0x7f962406cc60 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96480b2f90 0x7f964814e0e0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9648164a30 0x7f964814e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 --2- 172.21.15.62:0/3505764694 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f96480b15e0 0x7f9648153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:28.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 -- 172.21.15.62:0/3505764694 >> 172.21.15.62:0/3505764694 conn(0x7f96480b02e0 msgr2=0x7f9648164220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:28.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 -- 172.21.15.62:0/3505764694 shutdown_connections 2024-03-23T11:30:28.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:28.975+0000 7f964ea00640 1 -- 172.21.15.62:0/3505764694 wait complete. 2024-03-23T11:30:29.386 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:29:15.626274+0000', 'modified': '2024-03-23T11:30:28.494245+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 106, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8691}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8691': {'gid': 8691, 'name': 'a', 'rank': 0, 'incarnation': 81, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.62:6834/1070252332', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 1070252332}, {'type': 'v1', 'addr': '172.21.15.62:6834', 'nonce': 1070252332}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:30:29.386 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8691:up:replay 2024-03-23T11:30:30.387 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:30:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.627+0000 7f1a07e00640 1 Processor -- start 2024-03-23T11:30:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.627+0000 7f1a07e00640 1 -- start start 2024-03-23T11:30:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.629+0000 7f1a07e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a0806cc60 0x7f1a080cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.629+0000 7f1a07e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1a080ce420 0x7f1a080b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.629+0000 7f1a07e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a080ce8b0 con 0x7f1a08159490 2024-03-23T11:30:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.629+0000 7f1a07e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a080ced90 con 0x7f1a080ce420 2024-03-23T11:30:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.629+0000 7f1a07e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0806e3e0 con 0x7f1a0806cc60 2024-03-23T11:30:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.630+0000 7f1a07400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1a080ce420 0x7f1a080b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.630+0000 7f1a07400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1a080ce420 0x7f1a080b26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:36470/0 (socket says 172.21.15.62:36470) 2024-03-23T11:30:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.630+0000 7f1a07400640 1 -- 172.21.15.62:0/1540086280 learned_addr learned my addr 172.21.15.62:0/1540086280 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.630+0000 7f1a06000640 1 --2- 172.21.15.62:0/1540086280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a0806cc60 0x7f1a080cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.630+0000 7f1a07400640 1 -- 172.21.15.62:0/1540086280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a0806cc60 msgr2=0x7f1a080cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.630+0000 7f1a07400640 1 --2- 172.21.15.62:0/1540086280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a0806cc60 0x7f1a080cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.630+0000 7f1a07400640 1 -- 172.21.15.62:0/1540086280 >> v1:172.21.15.120:6789/0 conn(0x7f1a08159490 legacy=0x7f1a0815da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.630+0000 7f1a07400640 1 -- 172.21.15.62:0/1540086280 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0806c1c0 con 0x7f1a080ce420 2024-03-23T11:30:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.632+0000 7f1a07400640 1 --2- 172.21.15.62:0/1540086280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a080ce420 0x7f1a080b26b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009e50 tx=0x7f1a0002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6f2cac5dff55c35 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.632+0000 7f1a05600640 1 -- 172.21.15.62:0/1540086280 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0002ca70 con 0x7f1a080ce420 2024-03-23T11:30:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.632+0000 7f1a05600640 1 -- 172.21.15.62:0/1540086280 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0002cbd0 con 0x7f1a080ce420 2024-03-23T11:30:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.632+0000 7f1a05600640 1 -- 172.21.15.62:0/1540086280 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a000343f0 con 0x7f1a080ce420 2024-03-23T11:30:30.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.633+0000 7f1a07e00640 1 -- 172.21.15.62:0/1540086280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a080ce420 msgr2=0x7f1a080b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:30.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.633+0000 7f1a07e00640 1 --2- 172.21.15.62:0/1540086280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a080ce420 0x7f1a080b26b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009e50 tx=0x7f1a0002bed0 comp rx=0 tx=0).stop 2024-03-23T11:30:30.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.633+0000 7f1a07e00640 1 -- 172.21.15.62:0/1540086280 shutdown_connections 2024-03-23T11:30:30.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.633+0000 7f1a07e00640 1 --2- 172.21.15.62:0/1540086280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a080ce420 0x7f1a080b26b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:30.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.633+0000 7f1a07e00640 1 --2- 172.21.15.62:0/1540086280 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a0806cc60 0x7f1a080cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:30.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.633+0000 7f1a07e00640 1 -- 172.21.15.62:0/1540086280 >> 172.21.15.62:0/1540086280 conn(0x7f1a08066050 msgr2=0x7f1a08069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:30.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.633+0000 7f1a07e00640 1 -- 172.21.15.62:0/1540086280 shutdown_connections 2024-03-23T11:30:30.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.633+0000 7f1a07e00640 1 -- 172.21.15.62:0/1540086280 wait complete. 2024-03-23T11:30:30.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.635+0000 7f1a07e00640 1 Processor -- start 2024-03-23T11:30:30.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.635+0000 7f1a07e00640 1 -- start start 2024-03-23T11:30:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.636+0000 7f1a07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a0806cc60 0x7f1a080b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.636+0000 7f1a07e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a080ce420 0x7f1a080ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.636+0000 7f1a07e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a08159490 0x7f1a080cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.636+0000 7f1a07e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080ac660 con 0x7f1a0806cc60 2024-03-23T11:30:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.636+0000 7f1a07e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080b1370 con 0x7f1a08159490 2024-03-23T11:30:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.636+0000 7f1a07e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080ab9e0 con 0x7f1a080ce420 2024-03-23T11:30:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.636+0000 7f1a06a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a0806cc60 0x7f1a080b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.636+0000 7f1a06a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a0806cc60 0x7f1a080b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50026/0 (socket says 172.21.15.62:50026) 2024-03-23T11:30:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.636+0000 7f1a06a00640 1 -- 172.21.15.62:0/3212402792 learned_addr learned my addr 172.21.15.62:0/3212402792 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f1a06000640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a080ce420 0x7f1a080ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f1a07400640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a08159490 0x7f1a080cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f1a06a00640 1 -- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a080ce420 msgr2=0x7f1a080ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f1a06a00640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a080ce420 0x7f1a080ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f1a06a00640 1 -- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a08159490 msgr2=0x7f1a080cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f1a06a00640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a08159490 0x7f1a080cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f1a06a00640 1 -- 172.21.15.62:0/3212402792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a00009b30 con 0x7f1a0806cc60 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f1a07400640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a08159490 0x7f1a080cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f1a06a00640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a0806cc60 0x7f1a080b0e30 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f19f800c400 tx=0x7f19f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f19fea00640 1 -- 172.21.15.62:0/3212402792 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f80052d0 con 0x7f1a0806cc60 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f19fea00640 1 -- 172.21.15.62:0/3212402792 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f8005450 con 0x7f1a0806cc60 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f1a07e00640 1 -- 172.21.15.62:0/3212402792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a080abcc0 con 0x7f1a0806cc60 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f19fea00640 1 -- 172.21.15.62:0/3212402792 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8007260 con 0x7f1a0806cc60 2024-03-23T11:30:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.637+0000 7f1a07e00640 1 -- 172.21.15.62:0/3212402792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a080ac280 con 0x7f1a0806cc60 2024-03-23T11:30:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.638+0000 7f1a07e00640 1 -- 172.21.15.62:0/3212402792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a08066bb0 con 0x7f1a0806cc60 2024-03-23T11:30:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.639+0000 7f19fea00640 1 -- 172.21.15.62:0/3212402792 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f19f80073c0 con 0x7f1a0806cc60 2024-03-23T11:30:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.640+0000 7f19fea00640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19e806a7a0 0x7f19e806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.640+0000 7f19fea00640 1 -- 172.21.15.62:0/3212402792 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f19f8088260 con 0x7f1a0806cc60 2024-03-23T11:30:30.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.641+0000 7f1a06000640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19e806a7a0 0x7f19e806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:30.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.642+0000 7f1a06000640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19e806a7a0 0x7f19e806cc60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009a30 tx=0x7f19f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:30.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.645+0000 7f19fea00640 1 -- 172.21.15.62:0/3212402792 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f19f8053600 con 0x7f1a0806cc60 2024-03-23T11:30:30.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.886+0000 7f1a07e00640 1 -- 172.21.15.62:0/3212402792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a080caaf0 con 0x7f1a0806cc60 2024-03-23T11:30:30.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.887+0000 7f19fea00640 1 -- 172.21.15.62:0/3212402792 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4744 (secure 0 0 0) 0x7f19f8056c20 con 0x7f1a0806cc60 2024-03-23T11:30:30.878 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:30:30.878 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:30:30.503239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8691},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8691":{"gid":8691,"name":"a","rank":0,"incarnation":81,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.62:6834/1070252332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1070252332},{"type":"v1","addr":"172.21.15.62:6834","nonce":1070252332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:30:30.878 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 83 2024-03-23T11:30:30.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 -- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19e806a7a0 msgr2=0x7f19e806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:30.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19e806a7a0 0x7f19e806cc60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009a30 tx=0x7f19f4008040 comp rx=0 tx=0).stop 2024-03-23T11:30:30.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 -- 172.21.15.62:0/3212402792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a0806cc60 msgr2=0x7f1a080b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:30.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a0806cc60 0x7f1a080b0e30 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f19f800c400 tx=0x7f19f800c8c0 comp rx=0 tx=0).stop 2024-03-23T11:30:30.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 -- 172.21.15.62:0/3212402792 shutdown_connections 2024-03-23T11:30:30.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f19e806a7a0 0x7f19e806cc60 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:30.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a08159490 0x7f1a080cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:30.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a080ce420 0x7f1a080ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:30.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 --2- 172.21.15.62:0/3212402792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a0806cc60 0x7f1a080b0e30 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:30.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 -- 172.21.15.62:0/3212402792 >> 172.21.15.62:0/3212402792 conn(0x7f1a08066050 msgr2=0x7f1a080b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:30.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 -- 172.21.15.62:0/3212402792 shutdown_connections 2024-03-23T11:30:30.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:30.890+0000 7f1a07e00640 1 -- 172.21.15.62:0/3212402792 wait complete. 2024-03-23T11:30:31.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:29:15.626274+0000', 'modified': '2024-03-23T11:30:30.503239+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 106, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8691}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8691': {'gid': 8691, 'name': 'a', 'rank': 0, 'incarnation': 81, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.62:6834/1070252332', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 1070252332}, {'type': 'v1', 'addr': '172.21.15.62:6834', 'nonce': 1070252332}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:30:31.304 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8691:up:rejoin 2024-03-23T11:30:32.304 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:30:32.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.545+0000 7fd889c00640 1 Processor -- start 2024-03-23T11:30:32.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.545+0000 7fd889c00640 1 -- start start 2024-03-23T11:30:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.547+0000 7fd889c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd88406c300 0x7fd8840ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.547+0000 7fd889c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd8840cafa0 0x7fd8840cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.547+0000 7fd889c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8840cbc60 con 0x7fd88406c6e0 2024-03-23T11:30:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.547+0000 7fd889c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8840cbf40 con 0x7fd88406c300 2024-03-23T11:30:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.547+0000 7fd889c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8840cc220 con 0x7fd8840cafa0 2024-03-23T11:30:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.547+0000 7fd882a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd88406c300 0x7fd8840ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.547+0000 7fd882a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd88406c300 0x7fd8840ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50028/0 (socket says 172.21.15.62:50028) 2024-03-23T11:30:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.547+0000 7fd882a00640 1 -- 172.21.15.62:0/1402174798 learned_addr learned my addr 172.21.15.62:0/1402174798 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.548+0000 7fd882a00640 1 -- 172.21.15.62:0/1402174798 >> v2:172.21.15.120:3300/0 conn(0x7fd8840cafa0 msgr2=0x7fd8840cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.548+0000 7fd87a000640 1 --2- 172.21.15.62:0/1402174798 >> v2:172.21.15.120:3300/0 conn(0x7fd8840cafa0 0x7fd8840cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.548+0000 7fd882a00640 1 --2- 172.21.15.62:0/1402174798 >> v2:172.21.15.120:3300/0 conn(0x7fd8840cafa0 0x7fd8840cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.548+0000 7fd882a00640 1 -- 172.21.15.62:0/1402174798 >> v1:172.21.15.120:6789/0 conn(0x7fd88406c6e0 legacy=0x7fd88406cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.548+0000 7fd882a00640 1 -- 172.21.15.62:0/1402174798 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8840cc500 con 0x7fd88406c300 2024-03-23T11:30:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.549+0000 7fd882a00640 1 --2- 172.21.15.62:0/1402174798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd88406c300 0x7fd8840ccd00 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fd88406e370 tx=0x7fd87402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb7fd2529bbc89d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:32.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.549+0000 7fd882000640 1 -- 172.21.15.62:0/1402174798 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874031870 con 0x7fd88406c300 2024-03-23T11:30:32.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.549+0000 7fd882000640 1 -- 172.21.15.62:0/1402174798 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8740319d0 con 0x7fd88406c300 2024-03-23T11:30:32.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.549+0000 7fd882000640 1 -- 172.21.15.62:0/1402174798 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874031d60 con 0x7fd88406c300 2024-03-23T11:30:32.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.550+0000 7fd889c00640 1 -- 172.21.15.62:0/1402174798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd88406c300 msgr2=0x7fd8840ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:32.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.550+0000 7fd889c00640 1 --2- 172.21.15.62:0/1402174798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd88406c300 0x7fd8840ccd00 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fd88406e370 tx=0x7fd87402b910 comp rx=0 tx=0).stop 2024-03-23T11:30:32.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.550+0000 7fd889c00640 1 -- 172.21.15.62:0/1402174798 shutdown_connections 2024-03-23T11:30:32.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.550+0000 7fd889c00640 1 --2- 172.21.15.62:0/1402174798 >> v2:172.21.15.120:3300/0 conn(0x7fd8840cafa0 0x7fd8840cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:32.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.550+0000 7fd889c00640 1 --2- 172.21.15.62:0/1402174798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd88406c300 0x7fd8840ccd00 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:32.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.550+0000 7fd889c00640 1 -- 172.21.15.62:0/1402174798 >> 172.21.15.62:0/1402174798 conn(0x7fd8840b02e0 msgr2=0x7fd8840b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:32.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.550+0000 7fd889c00640 1 -- 172.21.15.62:0/1402174798 shutdown_connections 2024-03-23T11:30:32.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.550+0000 7fd889c00640 1 -- 172.21.15.62:0/1402174798 wait complete. 2024-03-23T11:30:32.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.552+0000 7fd889c00640 1 Processor -- start 2024-03-23T11:30:32.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.552+0000 7fd889c00640 1 -- start start 2024-03-23T11:30:32.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.553+0000 7fd889c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd88406c300 0x7fd8841f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:32.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.553+0000 7fd889c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd88406c6e0 0x7fd8840a7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:32.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.553+0000 7fd889c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8840cafa0 0x7fd884154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:32.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.553+0000 7fd889c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd884153d40 con 0x7fd88406c6e0 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.553+0000 7fd889c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8841f39b0 con 0x7fd88406c300 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.553+0000 7fd889c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8840a7cc0 con 0x7fd8840cafa0 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.553+0000 7fd87a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd88406c6e0 0x7fd8840a7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.553+0000 7fd87a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd88406c6e0 0x7fd8840a7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50034/0 (socket says 172.21.15.62:50034) 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.553+0000 7fd87a000640 1 -- 172.21.15.62:0/4223368445 learned_addr learned my addr 172.21.15.62:0/4223368445 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.554+0000 7fd882a00640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd88406c300 0x7fd8841f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.554+0000 7fd883400640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8840cafa0 0x7fd884154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.554+0000 7fd87a000640 1 -- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8840cafa0 msgr2=0x7fd884154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.554+0000 7fd87a000640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8840cafa0 0x7fd884154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.554+0000 7fd87a000640 1 -- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd88406c300 msgr2=0x7fd8841f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.554+0000 7fd87a000640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd88406c300 0x7fd8841f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.554+0000 7fd87a000640 1 -- 172.21.15.62:0/4223368445 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8740094d0 con 0x7fd88406c6e0 2024-03-23T11:30:32.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.554+0000 7fd883400640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8840cafa0 0x7fd884154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:32.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.554+0000 7fd87a000640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd88406c6e0 0x7fd8840a7780 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00c820 tx=0x7fd86c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:32.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.555+0000 7fd87b400640 1 -- 172.21.15.62:0/4223368445 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c004020 con 0x7fd88406c6e0 2024-03-23T11:30:32.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.555+0000 7fd87b400640 1 -- 172.21.15.62:0/4223368445 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c004180 con 0x7fd88406c6e0 2024-03-23T11:30:32.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.555+0000 7fd87b400640 1 -- 172.21.15.62:0/4223368445 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c007830 con 0x7fd88406c6e0 2024-03-23T11:30:32.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.555+0000 7fd889c00640 1 -- 172.21.15.62:0/4223368445 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8841312d0 con 0x7fd88406c6e0 2024-03-23T11:30:32.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.555+0000 7fd889c00640 1 -- 172.21.15.62:0/4223368445 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd884131890 con 0x7fd88406c6e0 2024-03-23T11:30:32.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.556+0000 7fd889c00640 1 -- 172.21.15.62:0/4223368445 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd884066c90 con 0x7fd88406c6e0 2024-03-23T11:30:32.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.557+0000 7fd87b400640 1 -- 172.21.15.62:0/4223368445 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd86c007990 con 0x7fd88406c6e0 2024-03-23T11:30:32.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.558+0000 7fd87b400640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd86006a7d0 0x7fd86006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:32.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.558+0000 7fd87b400640 1 -- 172.21.15.62:0/4223368445 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd86c088b50 con 0x7fd88406c6e0 2024-03-23T11:30:32.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.559+0000 7fd882a00640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd86006a7d0 0x7fd86006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:32.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.560+0000 7fd882a00640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd86006a7d0 0x7fd86006cc90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd874036000 tx=0x7fd87403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:32.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.563+0000 7fd87b400640 1 -- 172.21.15.62:0/4223368445 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd86c053e80 con 0x7fd88406c6e0 2024-03-23T11:30:32.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.799+0000 7fd889c00640 1 -- 172.21.15.62:0/4223368445 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8840cabc0 con 0x7fd88406c6e0 2024-03-23T11:30:32.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.800+0000 7fd87b400640 1 -- 172.21.15.62:0/4223368445 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4744 (secure 0 0 0) 0x7fd86c01e070 con 0x7fd88406c6e0 2024-03-23T11:30:32.791 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:30:32.791 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":84,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:30:31.513188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8691},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8691":{"gid":8691,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6834/1070252332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1070252332},{"type":"v1","addr":"172.21.15.62:6834","nonce":1070252332}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:30:32.791 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 84 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.803+0000 7fd889c00640 1 -- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd86006a7d0 msgr2=0x7fd86006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.803+0000 7fd889c00640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd86006a7d0 0x7fd86006cc90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd874036000 tx=0x7fd87403c040 comp rx=0 tx=0).stop 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.803+0000 7fd889c00640 1 -- 172.21.15.62:0/4223368445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd88406c6e0 msgr2=0x7fd8840a7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.803+0000 7fd889c00640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd88406c6e0 0x7fd8840a7780 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00c820 tx=0x7fd86c00cce0 comp rx=0 tx=0).stop 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.803+0000 7fd889c00640 1 -- 172.21.15.62:0/4223368445 shutdown_connections 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.803+0000 7fd889c00640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd86006a7d0 0x7fd86006cc90 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.803+0000 7fd889c00640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd88406c6e0 0x7fd8840a7780 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.803+0000 7fd889c00640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8840cafa0 0x7fd884154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.803+0000 7fd889c00640 1 --2- 172.21.15.62:0/4223368445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd88406c300 0x7fd8841f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.803+0000 7fd889c00640 1 -- 172.21.15.62:0/4223368445 >> 172.21.15.62:0/4223368445 conn(0x7fd8840b02e0 msgr2=0x7fd88406cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.803+0000 7fd889c00640 1 -- 172.21.15.62:0/4223368445 shutdown_connections 2024-03-23T11:30:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:32.804+0000 7fd889c00640 1 -- 172.21.15.62:0/4223368445 wait complete. 2024-03-23T11:30:33.216 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:29:15.626274+0000', 'modified': '2024-03-23T11:30:31.513188+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 106, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8691}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8691': {'gid': 8691, 'name': 'a', 'rank': 0, 'incarnation': 81, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.62:6834/1070252332', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 1070252332}, {'type': 'v1', 'addr': '172.21.15.62:6834', 'nonce': 1070252332}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:30:33.216 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:30:33.217 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T11:30:33.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.454+0000 7f053ae00640 1 Processor -- start 2024-03-23T11:30:33.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.455+0000 7f053ae00640 1 -- start start 2024-03-23T11:30:33.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.456+0000 7f053ae00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 0x7f05340cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.456+0000 7f053ae00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f05340d13b0 0x7f05340b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.456+0000 7f053ae00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05340d0810 con 0x7f05340cd260 2024-03-23T11:30:33.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.456+0000 7f053ae00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05340cd9d0 con 0x7f05340d13b0 2024-03-23T11:30:33.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.456+0000 7f053ae00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05340cdcb0 con 0x7f053415c960 2024-03-23T11:30:33.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.457+0000 7f053a400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f05340d13b0 0x7f05340b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.457+0000 7f0539a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 0x7f05340cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.457+0000 7f053a400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f05340d13b0 0x7f05340b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:36508/0 (socket says 172.21.15.62:36508) 2024-03-23T11:30:33.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.457+0000 7f053a400640 1 -- 172.21.15.62:0/4286912643 learned_addr learned my addr 172.21.15.62:0/4286912643 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.457+0000 7f0539a00640 1 -- 172.21.15.62:0/4286912643 >> v2:172.21.15.120:3300/0 conn(0x7f05340d13b0 msgr2=0x7f05340b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.457+0000 7f0539a00640 1 --2- 172.21.15.62:0/4286912643 >> v2:172.21.15.120:3300/0 conn(0x7f05340d13b0 0x7f05340b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.457+0000 7f0539a00640 1 -- 172.21.15.62:0/4286912643 >> v1:172.21.15.120:6789/0 conn(0x7f05340cd260 legacy=0x7f05340d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.458+0000 7f0539a00640 1 -- 172.21.15.62:0/4286912643 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05340cdf90 con 0x7f053415c960 2024-03-23T11:30:33.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.459+0000 7f0539a00640 1 --2- 172.21.15.62:0/4286912643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 0x7f05340cf790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f05280097a0 tx=0x7f052802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6308c5601d8adb94 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.460+0000 7f0533e00640 1 -- 172.21.15.62:0/4286912643 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528030800 con 0x7f053415c960 2024-03-23T11:30:33.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.460+0000 7f0533e00640 1 -- 172.21.15.62:0/4286912643 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0528030960 con 0x7f053415c960 2024-03-23T11:30:33.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.460+0000 7f0533e00640 1 -- 172.21.15.62:0/4286912643 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528030cf0 con 0x7f053415c960 2024-03-23T11:30:33.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.460+0000 7f053ae00640 1 -- 172.21.15.62:0/4286912643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 msgr2=0x7f05340cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.460+0000 7f053ae00640 1 --2- 172.21.15.62:0/4286912643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 0x7f05340cf790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f05280097a0 tx=0x7f052802b940 comp rx=0 tx=0).stop 2024-03-23T11:30:33.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.460+0000 7f053ae00640 1 -- 172.21.15.62:0/4286912643 shutdown_connections 2024-03-23T11:30:33.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.460+0000 7f053ae00640 1 --2- 172.21.15.62:0/4286912643 >> v2:172.21.15.120:3300/0 conn(0x7f05340d13b0 0x7f05340b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.460+0000 7f053ae00640 1 --2- 172.21.15.62:0/4286912643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 0x7f05340cf790 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.460+0000 7f053ae00640 1 -- 172.21.15.62:0/4286912643 >> 172.21.15.62:0/4286912643 conn(0x7f0534089f60 msgr2=0x7f05340d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.461+0000 7f053ae00640 1 -- 172.21.15.62:0/4286912643 shutdown_connections 2024-03-23T11:30:33.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.461+0000 7f053ae00640 1 -- 172.21.15.62:0/4286912643 wait complete. 2024-03-23T11:30:33.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.462+0000 7f053ae00640 1 Processor -- start 2024-03-23T11:30:33.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.463+0000 7f053ae00640 1 -- start start 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.463+0000 7f053ae00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05340cd260 0x7f053414a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f053ae00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05340d13b0 0x7f053415f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f053ae00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 0x7f05340afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f053ae00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0534160000 con 0x7f05340d13b0 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f053ae00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053414aef0 con 0x7f05340cd260 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f053ae00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05340c9930 con 0x7f053415c960 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f0539000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05340d13b0 0x7f053415f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f0539000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05340d13b0 0x7f053415f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50050/0 (socket says 172.21.15.62:50050) 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f0539000640 1 -- 172.21.15.62:0/3894580062 learned_addr learned my addr 172.21.15.62:0/3894580062 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f053a400640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 0x7f05340afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f0539a00640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05340cd260 0x7f053414a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f0539000640 1 -- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 msgr2=0x7f05340afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f0539000640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 0x7f05340afd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f0539000640 1 -- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05340cd260 msgr2=0x7f053414a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f0539000640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05340cd260 0x7f053414a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f0539000640 1 -- 172.21.15.62:0/3894580062 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0528009400 con 0x7f05340d13b0 2024-03-23T11:30:33.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f0539a00640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05340cd260 0x7f053414a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:33.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.464+0000 7f053a400640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 0x7f05340afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:33.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.465+0000 7f0539000640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05340d13b0 0x7f053415f5f0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f052000c450 tx=0x7f052000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.465+0000 7f0531600640 1 -- 172.21.15.62:0/3894580062 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05200052d0 con 0x7f05340d13b0 2024-03-23T11:30:33.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.465+0000 7f0531600640 1 -- 172.21.15.62:0/3894580062 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0520005450 con 0x7f05340d13b0 2024-03-23T11:30:33.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.465+0000 7f053ae00640 1 -- 172.21.15.62:0/3894580062 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05340c9c10 con 0x7f05340d13b0 2024-03-23T11:30:33.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.465+0000 7f0531600640 1 -- 172.21.15.62:0/3894580062 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520003880 con 0x7f05340d13b0 2024-03-23T11:30:33.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.465+0000 7f053ae00640 1 -- 172.21.15.62:0/3894580062 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05340ca1d0 con 0x7f05340d13b0 2024-03-23T11:30:33.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.468+0000 7f0531600640 1 -- 172.21.15.62:0/3894580062 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f05200039e0 con 0x7f05340d13b0 2024-03-23T11:30:33.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.468+0000 7f0531600640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f051406a7a0 0x7f051406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.469+0000 7f0531600640 1 -- 172.21.15.62:0/3894580062 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0520088270 con 0x7f05340d13b0 2024-03-23T11:30:33.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.469+0000 7f0539a00640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f051406a7a0 0x7f051406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.469+0000 7f053f1c4740 1 Processor -- start 2024-03-23T11:30:33.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.469+0000 7f053f1c4740 1 -- start start 2024-03-23T11:30:33.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.470+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1ceedb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.470+0000 7f0539a00640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f051406a7a0 0x7f051406cc60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0528036000 tx=0x7f052803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.470+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1ceec5a0 0x559d1ceee090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.470+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cee8a40 0x559d1cf18830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.470+0000 7f053f1c4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d1ce59110 con 0x559d1cf36520 2024-03-23T11:30:33.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.470+0000 7f053f1c4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d1ce568a0 con 0x559d1cee8a40 2024-03-23T11:30:33.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.470+0000 7f053f1c4740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d1ce52190 con 0x559d1ceec5a0 2024-03-23T11:30:33.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.471+0000 7f0539a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1ceedb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.471+0000 7f0539a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1ceedb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50066/0 (socket says 172.21.15.62:50066) 2024-03-23T11:30:33.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.471+0000 7f0539a00640 1 -- 172.21.15.62:0/663413590 learned_addr learned my addr 172.21.15.62:0/663413590 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.471+0000 7f0539000640 1 --2- 172.21.15.62:0/663413590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1ceec5a0 0x559d1ceee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.471+0000 7f053a400640 1 --2- 172.21.15.62:0/663413590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cee8a40 0x559d1cf18830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.471+0000 7f0539a00640 1 -- 172.21.15.62:0/663413590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1ceec5a0 msgr2=0x559d1ceee090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.471+0000 7f0539a00640 1 --2- 172.21.15.62:0/663413590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1ceec5a0 0x559d1ceee090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.471+0000 7f0539a00640 1 -- 172.21.15.62:0/663413590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cee8a40 msgr2=0x559d1cf18830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.471+0000 7f0539a00640 1 --2- 172.21.15.62:0/663413590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cee8a40 0x559d1cf18830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.471+0000 7f0539a00640 1 -- 172.21.15.62:0/663413590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d1ce1d230 con 0x559d1cf36520 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.472+0000 7f0539a00640 1 --2- 172.21.15.62:0/663413590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1ceedb50 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f0528009890 tx=0x7f0528030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfbe84088acf32f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.472+0000 7f0526000640 1 -- 172.21.15.62:0/663413590 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052804a070 con 0x559d1cf36520 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.472+0000 7f0526000640 1 -- 172.21.15.62:0/663413590 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0528030cb0 con 0x559d1cf36520 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.472+0000 7f0526000640 1 -- 172.21.15.62:0/663413590 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052803e040 con 0x559d1cf36520 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.472+0000 7f053f1c4740 1 -- 172.21.15.62:0/663413590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 msgr2=0x559d1ceedb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.472+0000 7f053f1c4740 1 --2- 172.21.15.62:0/663413590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1ceedb50 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f0528009890 tx=0x7f0528030000 comp rx=0 tx=0).stop 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.473+0000 7f053f1c4740 1 -- 172.21.15.62:0/663413590 shutdown_connections 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.473+0000 7f053f1c4740 1 --2- 172.21.15.62:0/663413590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cee8a40 0x559d1cf18830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.473+0000 7f053f1c4740 1 --2- 172.21.15.62:0/663413590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1ceec5a0 0x559d1ceee090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.473+0000 7f053f1c4740 1 --2- 172.21.15.62:0/663413590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1ceedb50 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.473+0000 7f053f1c4740 1 -- 172.21.15.62:0/663413590 >> 172.21.15.62:0/663413590 conn(0x559d1ceeacb0 msgr2=0x559d1cf33830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.473+0000 7f053f1c4740 1 -- 172.21.15.62:0/663413590 shutdown_connections 2024-03-23T11:30:33.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.473+0000 7f053f1c4740 1 -- 172.21.15.62:0/663413590 wait complete. 2024-03-23T11:30:33.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.473+0000 7f053f1c4740 1 -- start start 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.474+0000 7f053f1c4740 1 Processor -- start 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.475+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1ceec5a0 0x559d1cf10810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.475+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1cf36520 0x559d1cf34a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.475+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf34fa0 0x559d1cefb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.475+0000 7f053f1c4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d1cdf38c0 con 0x559d1cf34fa0 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.475+0000 7f053f1c4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d1cdf3560 con 0x559d1ceec5a0 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.475+0000 7f053f1c4740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d1cbb9f20 con 0x559d1cf36520 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.475+0000 7f053f1c4740 10 client.0 _ll_register_callbacks cb 0x559d1ce4b800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.475+0000 7f053a400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf34fa0 0x559d1cefb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.475+0000 7f053a400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf34fa0 0x559d1cefb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50080/0 (socket says 172.21.15.62:50080) 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.475+0000 7f053a400640 1 -- 172.21.15.62:0/3479064771 learned_addr learned my addr 172.21.15.62:0/3479064771 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.475+0000 7f0539a00640 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1ceec5a0 0x559d1cf10810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f053a400640 1 -- 172.21.15.62:0/3479064771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1cf36520 msgr2=0x559d1cf34a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f053a400640 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1cf36520 0x559d1cf34a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f053a400640 1 -- 172.21.15.62:0/3479064771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1ceec5a0 msgr2=0x559d1cf10810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f053a400640 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1ceec5a0 0x559d1cf10810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f053a400640 1 -- 172.21.15.62:0/3479064771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05340cdf90 con 0x559d1cf34fa0 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f0539a00640 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1ceec5a0 0x559d1cf10810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f053a400640 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf34fa0 0x559d1cefb8a0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f052c007ab0 tx=0x7f052c009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a3e4598b3a242b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f051aa00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f051aa00640 1 -- 172.21.15.62:0/3479064771 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c0119a0 con 0x559d1cf34fa0 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f051aa00640 1 -- 172.21.15.62:0/3479064771 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f052c011b00 con 0x559d1cf34fa0 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f053f1c4740 1 -- 172.21.15.62:0/3479064771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d1ce7c280 con 0x559d1cf34fa0 2024-03-23T11:30:33.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.476+0000 7f051aa00640 1 -- 172.21.15.62:0/3479064771 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f052c011cc0 con 0x559d1cf34fa0 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.477+0000 7f053f1c4740 10 client.8757 fetch_fsmap learned FSMap version 84 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.477+0000 7f053f1c4740 1 -- 172.21.15.62:0/3479064771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x559d1ce50810 con 0x559d1cf34fa0 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.477+0000 7f051aa00640 1 -- 172.21.15.62:0/3479064771 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559d1ce7c280 con 0x559d1cf34fa0 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.477+0000 7f051aa00640 1 -- 172.21.15.62:0/3479064771 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 84) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f052c01d060 con 0x559d1cf34fa0 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.477+0000 7f053f1c4740 10 client.8757 fetch_fsmap finished waiting for FSMap version 84 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.477+0000 7f053f1c4740 10 client.8757 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.477+0000 7f053f1c4740 20 client.8757 populate_metadata read hostname 'smithi062' 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.477+0000 7f053f1c4740 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x559d1cefd8f0 0x559d1cefdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.477+0000 7f053f1c4740 4 client.8757 mds_command: new command op to 8691 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.477+0000 7f053f1c4740 1 -- 172.21.15.62:0/3479064771 --> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d1ce449d0 con 0x559d1cefd8f0 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.477+0000 7f0539a00640 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x559d1cefd8f0 0x559d1cefdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.478+0000 7f0539a00640 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x559d1cefd8f0 0x559d1cefdcd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3959859c634072dd server_cookie=d2488c42e8c23c8e in_seq=0 out_seq=0 2024-03-23T11:30:33.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.478+0000 7f051aa00640 10 client.8757 ms_handle_connect on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:33.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f051aa00640 1 -- 172.21.15.62:0/3479064771 <== mds.0 v2:172.21.15.62:6832/1070252332 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x559d1ce449d0 con 0x559d1cefd8f0 2024-03-23T11:30:33.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f051aa00640 10 client.8757 handle_command_reply: tid=0 2024-03-23T11:30:33.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f053f1c4740 1 client.8757 shutdown 2024-03-23T11:30:33.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f053f1c4740 1 -- 172.21.15.62:0/3479064771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf34fa0 msgr2=0x559d1cefb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f053f1c4740 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf34fa0 0x559d1cefb8a0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f052c007ab0 tx=0x7f052c009f70 comp rx=0 tx=0).stop 2024-03-23T11:30:33.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f053f1c4740 1 -- 172.21.15.62:0/3479064771 shutdown_connections 2024-03-23T11:30:33.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f053f1c4740 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x559d1cefd8f0 0x559d1cefdcd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f053f1c4740 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf34fa0 0x559d1cefb8a0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f053f1c4740 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1cf36520 0x559d1cf34a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f053f1c4740 1 --2- 172.21.15.62:0/3479064771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1ceec5a0 0x559d1cf10810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f053f1c4740 1 -- 172.21.15.62:0/3479064771 >> 172.21.15.62:0/3479064771 conn(0x559d1cee8a40 msgr2=0x559d1ceed910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f051aa00640 0 client.8757 ms_handle_reset on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:33.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f053f1c4740 1 -- 172.21.15.62:0/3479064771 shutdown_connections 2024-03-23T11:30:33.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.479+0000 7f053f1c4740 1 -- 172.21.15.62:0/3479064771 wait complete. 2024-03-23T11:30:33.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.480+0000 7f053f1c4740 20 client.8757 trim_cache size 0 max 16384 2024-03-23T11:30:33.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.496+0000 7f053f1c4740 1 Processor -- start 2024-03-23T11:30:33.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.496+0000 7f053f1c4740 1 -- start start 2024-03-23T11:30:33.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.497+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1ceec5a0 0x559d1cf0f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.497+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cf34fa0 0x559d1cf15f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.497+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1cf349b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.497+0000 7f053f1c4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d1cf37d20 con 0x559d1cf36520 2024-03-23T11:30:33.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.497+0000 7f053f1c4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d1cf380c0 con 0x559d1cf34fa0 2024-03-23T11:30:33.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.497+0000 7f053f1c4740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d1cefaac0 con 0x559d1ceec5a0 2024-03-23T11:30:33.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.497+0000 7f053a400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1cf349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.497+0000 7f053a400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1cf349b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50096/0 (socket says 172.21.15.62:50096) 2024-03-23T11:30:33.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.497+0000 7f053a400640 1 -- 172.21.15.62:0/4068976127 learned_addr learned my addr 172.21.15.62:0/4068976127 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.498+0000 7f053a400640 1 -- 172.21.15.62:0/4068976127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1ceec5a0 msgr2=0x559d1cf0f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.498+0000 7f053a400640 1 --2- 172.21.15.62:0/4068976127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1ceec5a0 0x559d1cf0f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.498+0000 7f053a400640 1 -- 172.21.15.62:0/4068976127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cf34fa0 msgr2=0x559d1cf15f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.498+0000 7f053a400640 1 --2- 172.21.15.62:0/4068976127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cf34fa0 0x559d1cf15f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.498+0000 7f053a400640 1 -- 172.21.15.62:0/4068976127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d1cdf2ad0 con 0x559d1cf36520 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.498+0000 7f053a400640 1 --2- 172.21.15.62:0/4068976127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1cf349b0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f052c007a80 tx=0x7f052c00d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee7f1831fa27bde server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.498+0000 7f0526a00640 1 -- 172.21.15.62:0/4068976127 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c011ba0 con 0x559d1cf36520 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.498+0000 7f0526a00640 1 -- 172.21.15.62:0/4068976127 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c00f050 con 0x559d1cf36520 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.498+0000 7f0526a00640 1 -- 172.21.15.62:0/4068976127 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c0168f0 con 0x559d1cf36520 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.499+0000 7f053f1c4740 1 -- 172.21.15.62:0/4068976127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 msgr2=0x559d1cf349b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.499+0000 7f053f1c4740 1 --2- 172.21.15.62:0/4068976127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1cf349b0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f052c007a80 tx=0x7f052c00d000 comp rx=0 tx=0).stop 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.499+0000 7f053f1c4740 1 -- 172.21.15.62:0/4068976127 shutdown_connections 2024-03-23T11:30:33.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.499+0000 7f053f1c4740 1 --2- 172.21.15.62:0/4068976127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1cf36520 0x559d1cf349b0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.499+0000 7f053f1c4740 1 --2- 172.21.15.62:0/4068976127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cf34fa0 0x559d1cf15f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.499+0000 7f053f1c4740 1 --2- 172.21.15.62:0/4068976127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1ceec5a0 0x559d1cf0f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.499+0000 7f053f1c4740 1 -- 172.21.15.62:0/4068976127 >> 172.21.15.62:0/4068976127 conn(0x559d1cee8a40 msgr2=0x559d1cf10db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.499+0000 7f053f1c4740 1 -- 172.21.15.62:0/4068976127 shutdown_connections 2024-03-23T11:30:33.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.499+0000 7f053f1c4740 1 -- 172.21.15.62:0/4068976127 wait complete. 2024-03-23T11:30:33.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.500+0000 7f053f1c4740 1 -- start start 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.500+0000 7f053f1c4740 1 Processor -- start 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1ceec5a0 0x559d1cf36900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cf34fa0 0x559d1cf36e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f053f1c4740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1cf36520 0x559d1cf99f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f053f1c4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d1cefa460 con 0x559d1ceec5a0 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f053f1c4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d1cf29820 con 0x559d1cf34fa0 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f053f1c4740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d1cf110f0 con 0x559d1cf36520 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f053f1c4740 10 client.0 _ll_register_callbacks cb 0x7f052c009130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f0539a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1ceec5a0 0x559d1cf36900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f0539a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1ceec5a0 0x559d1cf36900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50110/0 (socket says 172.21.15.62:50110) 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f0539a00640 1 -- 172.21.15.62:0/2483556654 learned_addr learned my addr 172.21.15.62:0/2483556654 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f0539a00640 1 -- 172.21.15.62:0/2483556654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1cf36520 msgr2=0x559d1cf99f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f0539a00640 1 --2- 172.21.15.62:0/2483556654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1cf36520 0x559d1cf99f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f0539a00640 1 -- 172.21.15.62:0/2483556654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cf34fa0 msgr2=0x559d1cf36e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.501+0000 7f0539a00640 1 --2- 172.21.15.62:0/2483556654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cf34fa0 0x559d1cf36e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.502+0000 7f0539a00640 1 -- 172.21.15.62:0/2483556654 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d1ce50810 con 0x559d1ceec5a0 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.502+0000 7f0539a00640 1 --2- 172.21.15.62:0/2483556654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1ceec5a0 0x559d1cf36900 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f0528006f80 tx=0x7f05280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f829796ef7ed5d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.502+0000 7f051aa00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.502+0000 7f051aa00640 1 -- 172.21.15.62:0/2483556654 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052804a040 con 0x559d1ceec5a0 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.502+0000 7f053f1c4740 1 -- 172.21.15.62:0/2483556654 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0528030cb0 con 0x559d1ceec5a0 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.502+0000 7f051aa00640 1 -- 172.21.15.62:0/2483556654 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f052805a0d0 con 0x559d1ceec5a0 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.502+0000 7f051aa00640 1 -- 172.21.15.62:0/2483556654 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f052805a290 con 0x559d1ceec5a0 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.502+0000 7f053f1c4740 10 client.8763 fetch_fsmap learned FSMap version 84 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.502+0000 7f053f1c4740 1 -- 172.21.15.62:0/2483556654 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x559d1ce17fe0 con 0x559d1ceec5a0 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.503+0000 7f051aa00640 1 -- 172.21.15.62:0/2483556654 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0528030cb0 con 0x559d1ceec5a0 2024-03-23T11:30:33.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.503+0000 7f051aa00640 1 -- 172.21.15.62:0/2483556654 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 84) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f052805a860 con 0x559d1ceec5a0 2024-03-23T11:30:33.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.503+0000 7f053f1c4740 10 client.8763 fetch_fsmap finished waiting for FSMap version 84 2024-03-23T11:30:33.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.503+0000 7f053f1c4740 10 client.8763 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:30:33.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.503+0000 7f053f1c4740 20 client.8763 populate_metadata read hostname 'smithi062' 2024-03-23T11:30:33.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.503+0000 7f053f1c4740 1 --2- 172.21.15.62:0/2483556654 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x559d1cee4130 0x559d1cf9a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.503+0000 7f053f1c4740 4 client.8763 mds_command: new command op to 8691 tid=0[{"prefix": "status"}] 2024-03-23T11:30:33.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.503+0000 7f053f1c4740 1 -- 172.21.15.62:0/2483556654 --> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559d1cb463c0 con 0x559d1cee4130 2024-03-23T11:30:33.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.503+0000 7f0539000640 1 --2- 172.21.15.62:0/2483556654 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x559d1cee4130 0x559d1cf9a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f0539000640 1 --2- 172.21.15.62:0/2483556654 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x559d1cee4130 0x559d1cf9a4c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84fa79d99cc3d951 server_cookie=241b71ec7b3b9c33 in_seq=0 out_seq=0 2024-03-23T11:30:33.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f051aa00640 10 client.8763 ms_handle_connect on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:33.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f051aa00640 1 -- 172.21.15.62:0/2483556654 <== mds.0 v2:172.21.15.62:6832/1070252332 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x559d1cb463c0 con 0x559d1cee4130 2024-03-23T11:30:33.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f051aa00640 10 client.8763 handle_command_reply: tid=0 2024-03-23T11:30:33.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f053f1c4740 1 client.8763 shutdown 2024-03-23T11:30:33.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f053f1c4740 1 -- 172.21.15.62:0/2483556654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1ceec5a0 msgr2=0x559d1cf36900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f053f1c4740 1 --2- 172.21.15.62:0/2483556654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1ceec5a0 0x559d1cf36900 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f0528006f80 tx=0x7f05280046c0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f053f1c4740 1 -- 172.21.15.62:0/2483556654 shutdown_connections 2024-03-23T11:30:33.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f053f1c4740 1 --2- 172.21.15.62:0/2483556654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559d1cf36520 0x559d1cf99f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f053f1c4740 1 --2- 172.21.15.62:0/2483556654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d1cf34fa0 0x559d1cf36e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f053f1c4740 1 --2- 172.21.15.62:0/2483556654 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x559d1cee4130 0x559d1cf9a4c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f053f1c4740 1 --2- 172.21.15.62:0/2483556654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559d1ceec5a0 0x559d1cf36900 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f053f1c4740 1 -- 172.21.15.62:0/2483556654 >> 172.21.15.62:0/2483556654 conn(0x559d1cee8a40 msgr2=0x559d1cf0ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f051aa00640 0 client.8763 ms_handle_reset on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:33.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f053f1c4740 1 -- 172.21.15.62:0/2483556654 shutdown_connections 2024-03-23T11:30:33.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.504+0000 7f053f1c4740 1 -- 172.21.15.62:0/2483556654 wait complete. 2024-03-23T11:30:33.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053f1c4740 20 client.8763 trim_cache size 0 max 16384 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 8691, 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 5.0007312070000003, 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 84, 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 106, 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 106, 2024-03-23T11:30:33.499 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 5.955688812 2024-03-23T11:30:33.500 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:30:33.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 -- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f051406a7a0 msgr2=0x7f051406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f051406a7a0 0x7f051406cc60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0528036000 tx=0x7f052803b040 comp rx=0 tx=0).stop 2024-03-23T11:30:33.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 -- 172.21.15.62:0/3894580062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05340d13b0 msgr2=0x7f053415f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05340d13b0 0x7f053415f5f0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f052000c450 tx=0x7f052000c910 comp rx=0 tx=0).stop 2024-03-23T11:30:33.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 -- 172.21.15.62:0/3894580062 shutdown_connections 2024-03-23T11:30:33.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f051406a7a0 0x7f051406cc60 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05340d13b0 0x7f053415f5f0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f053415c960 0x7f05340afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 --2- 172.21.15.62:0/3894580062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05340cd260 0x7f053414a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 -- 172.21.15.62:0/3894580062 >> 172.21.15.62:0/3894580062 conn(0x7f0534089f60 msgr2=0x7f05340af270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 -- 172.21.15.62:0/3894580062 shutdown_connections 2024-03-23T11:30:33.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.505+0000 7f053ae00640 1 -- 172.21.15.62:0/3894580062 wait complete. 2024-03-23T11:30:33.501 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:30:33.506+0000 7fb13fe00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:30:33.513 DEBUG:teuthology.orchestra.run.smithi062:> sudo daemon-helper kill ceph --cluster ceph -w 2024-03-23T11:30:33.517 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-03-23T11:30:33.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.722+0000 7fdc62200640 1 Processor -- start 2024-03-23T11:30:33.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.723+0000 7fdc62200640 1 -- start start 2024-03-23T11:30:33.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.724+0000 7fdc62200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdc5c15ca70 0x7fdc5c1606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.724+0000 7fdc62200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdc5c160be0 0x7fdc5c0cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.724+0000 7fdc62200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5c0b58b0 con 0x7fdc5c0cf800 2024-03-23T11:30:33.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.724+0000 7fdc62200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5c0d0910 con 0x7fdc5c15ca70 2024-03-23T11:30:33.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.724+0000 7fdc62200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5c0cfc10 con 0x7fdc5c160be0 2024-03-23T11:30:33.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.724+0000 7fdc5b400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdc5c15ca70 0x7fdc5c1606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.724+0000 7fdc5b400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdc5c15ca70 0x7fdc5c1606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50116/0 (socket says 172.21.15.62:50116) 2024-03-23T11:30:33.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.724+0000 7fdc5be00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fdc5c0cf800 0x7fdc5c0b4370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43122/0 (socket says 172.21.15.62:43122) 2024-03-23T11:30:33.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.724+0000 7fdc5b400640 1 -- 172.21.15.62:0/332540089 learned_addr learned my addr 172.21.15.62:0/332540089 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.725+0000 7fdc5aa00640 1 --2- 172.21.15.62:0/332540089 >> v2:172.21.15.120:3300/0 conn(0x7fdc5c160be0 0x7fdc5c0cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.725+0000 7fdc5b400640 1 -- 172.21.15.62:0/332540089 >> v2:172.21.15.120:3300/0 conn(0x7fdc5c160be0 msgr2=0x7fdc5c0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.725+0000 7fdc5b400640 1 --2- 172.21.15.62:0/332540089 >> v2:172.21.15.120:3300/0 conn(0x7fdc5c160be0 0x7fdc5c0cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.725+0000 7fdc5b400640 1 -- 172.21.15.62:0/332540089 >> v1:172.21.15.62:6789/0 conn(0x7fdc5c0cf800 legacy=0x7fdc5c0b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.725+0000 7fdc5b400640 1 -- 172.21.15.62:0/332540089 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c161160 con 0x7fdc5c15ca70 2024-03-23T11:30:33.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.726+0000 7fdc5b400640 1 --2- 172.21.15.62:0/332540089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc5c15ca70 0x7fdc5c1606a0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c06c0a0 tx=0x7fdc4c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88e31497c569c6d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.726+0000 7fdc5a000640 1 -- 172.21.15.62:0/332540089 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c02f850 con 0x7fdc5c15ca70 2024-03-23T11:30:33.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.726+0000 7fdc5a000640 1 -- 172.21.15.62:0/332540089 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c02f9b0 con 0x7fdc5c15ca70 2024-03-23T11:30:33.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.726+0000 7fdc5a000640 1 -- 172.21.15.62:0/332540089 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c02fd40 con 0x7fdc5c15ca70 2024-03-23T11:30:33.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.727+0000 7fdc62200640 1 -- 172.21.15.62:0/332540089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc5c15ca70 msgr2=0x7fdc5c1606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.727+0000 7fdc62200640 1 --2- 172.21.15.62:0/332540089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc5c15ca70 0x7fdc5c1606a0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c06c0a0 tx=0x7fdc4c02ba30 comp rx=0 tx=0).stop 2024-03-23T11:30:33.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.727+0000 7fdc62200640 1 -- 172.21.15.62:0/332540089 shutdown_connections 2024-03-23T11:30:33.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.727+0000 7fdc62200640 1 --2- 172.21.15.62:0/332540089 >> v2:172.21.15.120:3300/0 conn(0x7fdc5c160be0 0x7fdc5c0cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.727+0000 7fdc62200640 1 --2- 172.21.15.62:0/332540089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc5c15ca70 0x7fdc5c1606a0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.727+0000 7fdc62200640 1 -- 172.21.15.62:0/332540089 >> 172.21.15.62:0/332540089 conn(0x7fdc5c071390 msgr2=0x7fdc5c071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.727+0000 7fdc62200640 1 -- 172.21.15.62:0/332540089 shutdown_connections 2024-03-23T11:30:33.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.727+0000 7fdc62200640 1 -- 172.21.15.62:0/332540089 wait complete. 2024-03-23T11:30:33.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.729+0000 7fdc62200640 1 Processor -- start 2024-03-23T11:30:33.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.729+0000 7fdc62200640 1 -- start start 2024-03-23T11:30:33.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.730+0000 7fdc62200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc5c0cf800 0x7fdc5c157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.730+0000 7fdc62200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5c15ca70 0x7fdc5c0b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.730+0000 7fdc62200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc5c160be0 0x7fdc5c0b2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.730+0000 7fdc62200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c0cc600 con 0x7fdc5c160be0 2024-03-23T11:30:33.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.730+0000 7fdc62200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c0c9af0 con 0x7fdc5c0cf800 2024-03-23T11:30:33.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.730+0000 7fdc62200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c0c9dd0 con 0x7fdc5c15ca70 2024-03-23T11:30:33.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.730+0000 7fdc5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc5c160be0 0x7fdc5c0b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.730+0000 7fdc5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc5c160be0 0x7fdc5c0b2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50126/0 (socket says 172.21.15.62:50126) 2024-03-23T11:30:33.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.730+0000 7fdc5be00640 1 -- 172.21.15.62:0/467409511 learned_addr learned my addr 172.21.15.62:0/467409511 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.731+0000 7fdc5aa00640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5c15ca70 0x7fdc5c0b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.731+0000 7fdc5b400640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc5c0cf800 0x7fdc5c157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.731+0000 7fdc5be00640 1 -- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5c15ca70 msgr2=0x7fdc5c0b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.731+0000 7fdc5be00640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5c15ca70 0x7fdc5c0b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.731+0000 7fdc5be00640 1 -- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc5c0cf800 msgr2=0x7fdc5c157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.731+0000 7fdc5be00640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc5c0cf800 0x7fdc5c157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.731+0000 7fdc5be00640 1 -- 172.21.15.62:0/467409511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c0094d0 con 0x7fdc5c160be0 2024-03-23T11:30:33.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.731+0000 7fdc5be00640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc5c160be0 0x7fdc5c0b2790 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fdc54001eb0 tx=0x7fdc5400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.732+0000 7fdc53400640 1 -- 172.21.15.62:0/467409511 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc540074e0 con 0x7fdc5c160be0 2024-03-23T11:30:33.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.732+0000 7fdc53400640 1 -- 172.21.15.62:0/467409511 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc54007640 con 0x7fdc5c160be0 2024-03-23T11:30:33.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.732+0000 7fdc53400640 1 -- 172.21.15.62:0/467409511 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc540049e0 con 0x7fdc5c160be0 2024-03-23T11:30:33.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.732+0000 7fdc62200640 1 -- 172.21.15.62:0/467409511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc5c0ca0b0 con 0x7fdc5c160be0 2024-03-23T11:30:33.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.732+0000 7fdc62200640 1 -- 172.21.15.62:0/467409511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5c0cca50 con 0x7fdc5c160be0 2024-03-23T11:30:33.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.734+0000 7fdc53400640 1 -- 172.21.15.62:0/467409511 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fdc54004b40 con 0x7fdc5c160be0 2024-03-23T11:30:33.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.735+0000 7fdc53400640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc3806a7a0 0x7fdc3806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.735+0000 7fdc53400640 1 -- 172.21.15.62:0/467409511 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdc54088770 con 0x7fdc5c160be0 2024-03-23T11:30:33.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.735+0000 7fdc5b400640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc3806a7a0 0x7fdc3806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.736+0000 7fdc64e3a740 1 Processor -- start 2024-03-23T11:30:33.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.736+0000 7fdc64e3a740 1 -- start start 2024-03-23T11:30:33.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.737+0000 7fdc5b400640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc3806a7a0 0x7fdc3806cc60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c036000 tx=0x7fdc4c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.737+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665279340 0x556665230970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.737+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666522f3c0 0x556665230eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.737+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522baf0 0x55666525b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.737+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55666519bf40 con 0x55666522baf0 2024-03-23T11:30:33.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.737+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566651996d0 con 0x55666522f3c0 2024-03-23T11:30:33.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.737+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x556665194fc0 con 0x556665279340 2024-03-23T11:30:33.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.737+0000 7fdc5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522baf0 0x55666525b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.737+0000 7fdc5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522baf0 0x55666525b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50138/0 (socket says 172.21.15.62:50138) 2024-03-23T11:30:33.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.737+0000 7fdc5be00640 1 -- 172.21.15.62:0/111446704 learned_addr learned my addr 172.21.15.62:0/111446704 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.737+0000 7fdc5aa00640 1 --2- 172.21.15.62:0/111446704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666522f3c0 0x556665230eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7fdc5b400640 1 --2- 172.21.15.62:0/111446704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665279340 0x556665230970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7fdc5be00640 1 -- 172.21.15.62:0/111446704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665279340 msgr2=0x556665230970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7fdc5be00640 1 --2- 172.21.15.62:0/111446704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665279340 0x556665230970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7fdc5be00640 1 -- 172.21.15.62:0/111446704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666522f3c0 msgr2=0x556665230eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7fdc5be00640 1 --2- 172.21.15.62:0/111446704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666522f3c0 0x556665230eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7fdc5be00640 1 -- 172.21.15.62:0/111446704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55666515c230 con 0x55666522baf0 2024-03-23T11:30:33.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7fdc5aa00640 1 --2- 172.21.15.62:0/111446704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666522f3c0 0x556665230eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:30:33.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7fdc5be00640 1 --2- 172.21.15.62:0/111446704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522baf0 0x55666525b650 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fdc54053ce0 tx=0x7fdc54057fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=627b0c936f0288c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7f6629800640 1 Processor -- start 2024-03-23T11:30:33.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7fdc4be00640 1 -- 172.21.15.62:0/111446704 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc54055920 con 0x55666522baf0 2024-03-23T11:30:33.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7fdc4be00640 1 -- 172.21.15.62:0/111446704 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc54055b80 con 0x55666522baf0 2024-03-23T11:30:33.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.738+0000 7fdc4be00640 1 -- 172.21.15.62:0/111446704 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc54057a40 con 0x55666522baf0 2024-03-23T11:30:33.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.739+0000 7f6629800640 1 -- start start 2024-03-23T11:30:33.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.739+0000 7fdc64e3a740 1 -- 172.21.15.62:0/111446704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522baf0 msgr2=0x55666525b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.739+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/111446704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522baf0 0x55666525b650 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fdc54053ce0 tx=0x7fdc54057fa0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.739+0000 7fdc64e3a740 1 -- 172.21.15.62:0/111446704 shutdown_connections 2024-03-23T11:30:33.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.739+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/111446704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522baf0 0x55666525b650 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.739+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/111446704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666522f3c0 0x556665230eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.739+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/111446704 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665279340 0x556665230970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.739+0000 7fdc64e3a740 1 -- 172.21.15.62:0/111446704 >> 172.21.15.62:0/111446704 conn(0x55666522dd60 msgr2=0x556665276650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.739+0000 7fdc64e3a740 1 -- 172.21.15.62:0/111446704 shutdown_connections 2024-03-23T11:30:33.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.739+0000 7fdc64e3a740 1 -- 172.21.15.62:0/111446704 wait complete. 2024-03-23T11:30:33.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.740+0000 7f6629800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66241440d0 0x7f66240cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.740+0000 7f6629800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f662415efa0 0x7f66240cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.740+0000 7fdc64e3a740 1 -- start start 2024-03-23T11:30:33.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.740+0000 7f6629800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662415eb10 con 0x7f662415ad00 2024-03-23T11:30:33.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.740+0000 7f6629800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f662415b880 con 0x7f662415efa0 2024-03-23T11:30:33.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.740+0000 7f6629800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66241444b0 con 0x7f66241440d0 2024-03-23T11:30:33.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.740+0000 7fdc64e3a740 1 Processor -- start 2024-03-23T11:30:33.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.740+0000 7f6622000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f662415efa0 0x7f66240cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.741+0000 7f6622000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f662415efa0 0x7f66240cd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50144/0 (socket says 172.21.15.62:50144) 2024-03-23T11:30:33.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.741+0000 7f6622000640 1 -- 172.21.15.62:0/344468114 learned_addr learned my addr 172.21.15.62:0/344468114 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.741+0000 7f6622a00640 1 --2- 172.21.15.62:0/344468114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66241440d0 0x7f66240cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.741+0000 7f6622000640 1 -- 172.21.15.62:0/344468114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66241440d0 msgr2=0x7f66240cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.741+0000 7f6622000640 1 --2- 172.21.15.62:0/344468114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66241440d0 0x7f66240cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.741+0000 7f6622000640 1 -- 172.21.15.62:0/344468114 >> v1:172.21.15.120:6789/0 conn(0x7f662415ad00 legacy=0x7f662415b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.741+0000 7f6622000640 1 -- 172.21.15.62:0/344468114 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662415baf0 con 0x7f662415efa0 2024-03-23T11:30:33.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55666522f3c0 0x556665277c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 0x55666527e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556665253280 0x556665253700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556665136320 con 0x556665253280 2024-03-23T11:30:33.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x556665135fc0 con 0x556665279340 2024-03-23T11:30:33.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x556664efd0c0 con 0x55666522f3c0 2024-03-23T11:30:33.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc64e3a740 10 client.0 _ll_register_callbacks cb 0x55666515ba10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:30:33.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556665253280 0x556665253700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556665253280 0x556665253700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50148/0 (socket says 172.21.15.62:50148) 2024-03-23T11:30:33.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc5be00640 1 -- 172.21.15.62:0/1848178781 learned_addr learned my addr 172.21.15.62:0/1848178781 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7f6622000640 1 --2- 172.21.15.62:0/344468114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f662415efa0 0x7f66240cd8e0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f6610009870 tx=0x7f661002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d3c7a238609803 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc5b400640 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55666522f3c0 0x556665277c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc5be00640 1 -- 172.21.15.62:0/1848178781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55666522f3c0 msgr2=0x556665277c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc5be00640 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55666522f3c0 0x556665277c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc5aa00640 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 0x55666527e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc5be00640 1 -- 172.21.15.62:0/1848178781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 msgr2=0x55666527e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc5be00640 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 0x55666527e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc5be00640 1 -- 172.21.15.62:0/1848178781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5c0cca50 con 0x556665253280 2024-03-23T11:30:33.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.742+0000 7fdc5aa00640 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 0x55666527e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:33.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.743+0000 7f6621600640 1 -- 172.21.15.62:0/344468114 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610031640 con 0x7f662415efa0 2024-03-23T11:30:33.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.743+0000 7f6621600640 1 -- 172.21.15.62:0/344468114 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66100317a0 con 0x7f662415efa0 2024-03-23T11:30:33.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.743+0000 7fdc5be00640 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556665253280 0x556665253700 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fdc54057770 tx=0x7fdc5400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ea54a36b0cda50 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.743+0000 7f6621600640 1 -- 172.21.15.62:0/344468114 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610031b30 con 0x7f662415efa0 2024-03-23T11:30:33.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.743+0000 7fdc48c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:30:33.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.743+0000 7fdc48c00640 1 -- 172.21.15.62:0/1848178781 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc54056ed0 con 0x556665253280 2024-03-23T11:30:33.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.743+0000 7fdc48c00640 1 -- 172.21.15.62:0/1848178781 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdc5406f1d0 con 0x556665253280 2024-03-23T11:30:33.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.743+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1848178781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566651bebb0 con 0x556665253280 2024-03-23T11:30:33.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7fdc48c00640 1 -- 172.21.15.62:0/1848178781 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc54057a40 con 0x556665253280 2024-03-23T11:30:33.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7fdc64e3a740 10 client.8781 fetch_fsmap learned FSMap version 84 2024-03-23T11:30:33.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1848178781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x556665193640 con 0x556665253280 2024-03-23T11:30:33.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7f6629800640 1 -- 172.21.15.62:0/344468114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f662415efa0 msgr2=0x7f66240cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7f6629800640 1 --2- 172.21.15.62:0/344468114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f662415efa0 0x7f66240cd8e0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f6610009870 tx=0x7f661002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7fdc48c00640 1 -- 172.21.15.62:0/1848178781 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5566651bebb0 con 0x556665253280 2024-03-23T11:30:33.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7fdc48c00640 1 -- 172.21.15.62:0/1848178781 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 84) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fdc5405f740 con 0x556665253280 2024-03-23T11:30:33.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7fdc64e3a740 10 client.8781 fetch_fsmap finished waiting for FSMap version 84 2024-03-23T11:30:33.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7fdc64e3a740 10 client.8781 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:30:33.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7fdc64e3a740 20 client.8781 populate_metadata read hostname 'smithi062' 2024-03-23T11:30:33.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x556665242210 0x5566652425f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7fdc64e3a740 4 client.8781 mds_command: new command op to 8691 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:30:33.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1848178781 --> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556664e89560 con 0x556665242210 2024-03-23T11:30:33.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7f6629800640 1 -- 172.21.15.62:0/344468114 shutdown_connections 2024-03-23T11:30:33.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7f6629800640 1 --2- 172.21.15.62:0/344468114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f662415efa0 0x7f66240cd8e0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7f6629800640 1 --2- 172.21.15.62:0/344468114 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f66241440d0 0x7f66240cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.744+0000 7f6629800640 1 -- 172.21.15.62:0/344468114 >> 172.21.15.62:0/344468114 conn(0x7f6624066050 msgr2=0x7f6624069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.745+0000 7f6629800640 1 -- 172.21.15.62:0/344468114 shutdown_connections 2024-03-23T11:30:33.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.745+0000 7fdc5b400640 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x556665242210 0x5566652425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.745+0000 7f6629800640 1 -- 172.21.15.62:0/344468114 wait complete. 2024-03-23T11:30:33.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.745+0000 7fdc5b400640 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x556665242210 0x5566652425f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8da3a401ae767e5 server_cookie=67de09bfa52d82a4 in_seq=0 out_seq=0 2024-03-23T11:30:33.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.745+0000 7fdc48c00640 10 client.8781 ms_handle_connect on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:33.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.746+0000 7fdc48c00640 1 -- 172.21.15.62:0/1848178781 <== mds.0 v2:172.21.15.62:6832/1070252332 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x556664e89560 con 0x556665242210 2024-03-23T11:30:33.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.746+0000 7fdc48c00640 10 client.8781 handle_command_reply: tid=0 2024-03-23T11:30:33.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.746+0000 7fdc64e3a740 1 client.8781 shutdown 2024-03-23T11:30:33.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.746+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1848178781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556665253280 msgr2=0x556665253700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.746+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556665253280 0x556665253700 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fdc54057770 tx=0x7fdc5400ced0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1848178781 shutdown_connections 2024-03-23T11:30:33.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x556665242210 0x5566652425f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556665253280 0x556665253700 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 0x55666527e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1848178781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55666522f3c0 0x556665277c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1848178781 >> 172.21.15.62:0/1848178781 conn(0x55666522baf0 msgr2=0x55666522fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7fdc48c00640 0 client.8781 ms_handle_reset on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:33.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1848178781 shutdown_connections 2024-03-23T11:30:33.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1848178781 wait complete. 2024-03-23T11:30:33.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7f6629800640 1 Processor -- start 2024-03-23T11:30:33.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7fdc64e3a740 20 client.8781 trim_cache size 0 max 16384 2024-03-23T11:30:33.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.747+0000 7f6629800640 1 -- start start 2024-03-23T11:30:33.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.748+0000 7f6629800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66241440d0 0x7f662413db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.748+0000 7f6629800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f662415ad00 0x7f66241f8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.748+0000 7f6629800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f662415efa0 0x7f662415c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.748+0000 7f6629800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66241f8d30 con 0x7f66241440d0 2024-03-23T11:30:33.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.748+0000 7f6629800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66241f9010 con 0x7f662415efa0 2024-03-23T11:30:33.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.748+0000 7f6629800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66241f92f0 con 0x7f662415ad00 2024-03-23T11:30:33.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.748+0000 7f6622a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66241440d0 0x7f662413db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.749+0000 7f6622a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66241440d0 0x7f662413db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50164/0 (socket says 172.21.15.62:50164) 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.749+0000 7f6622a00640 1 -- 172.21.15.62:0/3621539899 learned_addr learned my addr 172.21.15.62:0/3621539899 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.749+0000 7f6622000640 1 --2- 172.21.15.62:0/3621539899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f662415ad00 0x7f66241f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.749+0000 7f6622a00640 1 -- 172.21.15.62:0/3621539899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f662415ad00 msgr2=0x7f66241f8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.749+0000 7f6622a00640 1 --2- 172.21.15.62:0/3621539899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f662415ad00 0x7f66241f8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.749+0000 7f6622a00640 1 -- 172.21.15.62:0/3621539899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f662415efa0 msgr2=0x7f662415c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.749+0000 7f6622a00640 1 --2- 172.21.15.62:0/3621539899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f662415efa0 0x7f662415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.749+0000 7f6622a00640 1 -- 172.21.15.62:0/3621539899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66100094d0 con 0x7f66241440d0 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.749+0000 7f6622000640 1 --2- 172.21.15.62:0/3621539899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f662415ad00 0x7f66241f8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.749+0000 7f6622a00640 1 --2- 172.21.15.62:0/3621539899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66241440d0 0x7f662413db20 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f661400d800 tx=0x7f661400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.750+0000 7f661aa00640 1 -- 172.21.15.62:0/3621539899 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661400b6c0 con 0x7f66241440d0 2024-03-23T11:30:33.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.750+0000 7f661aa00640 1 -- 172.21.15.62:0/3621539899 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661400b820 con 0x7f66241440d0 2024-03-23T11:30:33.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.750+0000 7f661aa00640 1 -- 172.21.15.62:0/3621539899 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661400bbb0 con 0x7f66241440d0 2024-03-23T11:30:33.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.750+0000 7f6629800640 1 -- 172.21.15.62:0/3621539899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66241f95d0 con 0x7f66241440d0 2024-03-23T11:30:33.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.750+0000 7f6629800640 1 -- 172.21.15.62:0/3621539899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662413e060 con 0x7f66241440d0 2024-03-23T11:30:33.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.751+0000 7f6629800640 1 -- 172.21.15.62:0/3621539899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f6624066bb0 con 0x7f66241440d0 2024-03-23T11:30:33.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.752+0000 7f661aa00640 1 -- 172.21.15.62:0/3621539899 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f661400bd10 con 0x7f66241440d0 2024-03-23T11:30:33.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.753+0000 7f661aa00640 1 --2- 172.21.15.62:0/3621539899 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f660406a7a0 0x7f660406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.753+0000 7f661aa00640 1 -- 172.21.15.62:0/3621539899 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6614089db0 con 0x7f66241440d0 2024-03-23T11:30:33.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.753+0000 7f661aa00640 1 -- 172.21.15.62:0/3621539899 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+516 (secure 0 0 0) 0x7f661408a220 con 0x7f66241440d0 2024-03-23T11:30:33.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.754+0000 7f6629800640 1 -- 172.21.15.62:0/3621539899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7f66240ce150 con 0x7f66241440d0 2024-03-23T11:30:33.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.754+0000 7f6622000640 1 --2- 172.21.15.62:0/3621539899 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f660406a7a0 0x7f660406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.755+0000 7f6622000640 1 --2- 172.21.15.62:0/3621539899 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f660406a7a0 0x7f660406cc60 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6610036000 tx=0x7f661003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.764+0000 7fdc64e3a740 1 Processor -- start 2024-03-23T11:30:33.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.764+0000 7fdc64e3a740 1 -- start start 2024-03-23T11:30:33.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.764+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665254350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.764+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665253280 0x55666522fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 0x556665275490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556665276c50 con 0x55666522f3c0 2024-03-23T11:30:33.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55666523ea10 con 0x556665279340 2024-03-23T11:30:33.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55666523e750 con 0x556665253280 2024-03-23T11:30:33.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc5b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665254350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc5b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665254350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50166/0 (socket says 172.21.15.62:50166) 2024-03-23T11:30:33.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc5b400640 1 -- 172.21.15.62:0/3823204042 learned_addr learned my addr 172.21.15.62:0/3823204042 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc5b400640 1 -- 172.21.15.62:0/3823204042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665253280 msgr2=0x55666522fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc5b400640 1 --2- 172.21.15.62:0/3823204042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665253280 0x55666522fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc5b400640 1 -- 172.21.15.62:0/3823204042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 msgr2=0x556665275490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc5b400640 1 --2- 172.21.15.62:0/3823204042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 0x556665275490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.765+0000 7fdc5b400640 1 -- 172.21.15.62:0/3823204042 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566651c7f00 con 0x55666522f3c0 2024-03-23T11:30:33.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.766+0000 7fdc5b400640 1 --2- 172.21.15.62:0/3823204042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665254350 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c009960 tx=0x7fdc4c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3ac59d701d6abf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.766+0000 7fdc50c00640 1 -- 172.21.15.62:0/3823204042 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c047030 con 0x55666522f3c0 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.766+0000 7fdc50c00640 1 -- 172.21.15.62:0/3823204042 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c048040 con 0x55666522f3c0 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.766+0000 7fdc50c00640 1 -- 172.21.15.62:0/3823204042 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c0502f0 con 0x55666522f3c0 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.766+0000 7fdc64e3a740 1 -- 172.21.15.62:0/3823204042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 msgr2=0x556665254350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.766+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/3823204042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665254350 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c009960 tx=0x7fdc4c009240 comp rx=0 tx=0).stop 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.767+0000 7fdc64e3a740 1 -- 172.21.15.62:0/3823204042 shutdown_connections 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.767+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/3823204042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 0x556665275490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.767+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/3823204042 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665253280 0x55666522fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.767+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/3823204042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665254350 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.767+0000 7fdc64e3a740 1 -- 172.21.15.62:0/3823204042 >> 172.21.15.62:0/3823204042 conn(0x55666522baf0 msgr2=0x5566652426f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.767+0000 7fdc64e3a740 1 -- 172.21.15.62:0/3823204042 shutdown_connections 2024-03-23T11:30:33.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.767+0000 7fdc64e3a740 1 -- 172.21.15.62:0/3823204042 wait complete. 2024-03-23T11:30:33.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.767+0000 7fdc64e3a740 1 -- start start 2024-03-23T11:30:33.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.768+0000 7fdc64e3a740 1 Processor -- start 2024-03-23T11:30:33.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665275490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665253280 0x5566652759d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc64e3a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 0x55666526bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556665231e50 con 0x55666522f3c0 2024-03-23T11:30:33.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566652de4d0 con 0x556665279340 2024-03-23T11:30:33.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc64e3a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55666522fec0 con 0x556665253280 2024-03-23T11:30:33.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc64e3a740 10 client.0 _ll_register_callbacks cb 0x7fdc540575b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:30:33.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc5b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665275490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc5b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665275490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50174/0 (socket says 172.21.15.62:50174) 2024-03-23T11:30:33.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc5b400640 1 -- 172.21.15.62:0/1371562406 learned_addr learned my addr 172.21.15.62:0/1371562406 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc5b400640 1 -- 172.21.15.62:0/1371562406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665253280 msgr2=0x5566652759d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc5b400640 1 --2- 172.21.15.62:0/1371562406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665253280 0x5566652759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc5b400640 1 -- 172.21.15.62:0/1371562406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 msgr2=0x55666526bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc5b400640 1 --2- 172.21.15.62:0/1371562406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 0x55666526bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc5b400640 1 -- 172.21.15.62:0/1371562406 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5c161160 con 0x55666522f3c0 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc5b400640 1 --2- 172.21.15.62:0/1371562406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665275490 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c009960 tx=0x7fdc4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46d5b445d53d9b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.769+0000 7fdc48c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc48c00640 1 -- 172.21.15.62:0/1371562406 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c047040 con 0x55666522f3c0 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc48c00640 1 -- 172.21.15.62:0/1371562406 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdc4c03c040 con 0x55666522f3c0 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1371562406 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc54055b80 con 0x55666522f3c0 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc48c00640 1 -- 172.21.15.62:0/1371562406 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc4c042640 con 0x55666522f3c0 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc64e3a740 10 client.8790 fetch_fsmap learned FSMap version 84 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1371562406 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x5566651354c0 con 0x55666522f3c0 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc48c00640 1 -- 172.21.15.62:0/1371562406 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc54055b80 con 0x55666522f3c0 2024-03-23T11:30:33.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc48c00640 1 -- 172.21.15.62:0/1371562406 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 84) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fdc4c057070 con 0x55666522f3c0 2024-03-23T11:30:33.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc64e3a740 10 client.8790 fetch_fsmap finished waiting for FSMap version 84 2024-03-23T11:30:33.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc64e3a740 10 client.8790 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:30:33.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc64e3a740 20 client.8790 populate_metadata read hostname 'smithi062' 2024-03-23T11:30:33.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1371562406 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x5566652dcc70 0x55666526c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc64e3a740 4 client.8790 mds_command: new command op to 8691 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-03-23T11:30:33.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.770+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1371562406 --> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) v1 -- 0x556664e9a860 con 0x5566652dcc70 2024-03-23T11:30:33.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.771+0000 7fdc5aa00640 1 --2- 172.21.15.62:0/1371562406 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x5566652dcc70 0x55666526c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.771+0000 7fdc5aa00640 1 --2- 172.21.15.62:0/1371562406 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x5566652dcc70 0x55666526c520 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc4ddf0647a6a051 server_cookie=c1ee307271a7ae5d in_seq=0 out_seq=0 2024-03-23T11:30:33.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.771+0000 7fdc48c00640 10 client.8790 ms_handle_connect on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:33.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.772+0000 7fdc48c00640 1 -- 172.21.15.62:0/1371562406 <== mds.0 v2:172.21.15.62:6832/1070252332 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556664e9a860 con 0x5566652dcc70 2024-03-23T11:30:33.765 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:30:33.772+0000 7fb13b800640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-03-23T11:30:33.765 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:30:33.772+0000 7fb13b800640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-03-23T11:30:33.765 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:30:33.772+0000 7fb13b800640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-03-23T11:30:33.765 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:30:33.772+0000 7fb13b800640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-03-23T11:30:33.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.772+0000 7fdc48c00640 10 client.8790 handle_command_reply: tid=0 2024-03-23T11:30:33.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.772+0000 7fdc64e3a740 1 client.8790 shutdown 2024-03-23T11:30:33.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1371562406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 msgr2=0x556665275490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1371562406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665275490 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c009960 tx=0x7fdc4c004730 comp rx=0 tx=0).stop 2024-03-23T11:30:33.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1371562406 shutdown_connections 2024-03-23T11:30:33.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1371562406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665279340 0x55666526bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1371562406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556665253280 0x5566652759d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1371562406 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x5566652dcc70 0x55666526c520 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc64e3a740 1 --2- 172.21.15.62:0/1371562406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55666522f3c0 0x556665275490 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1371562406 >> 172.21.15.62:0/1371562406 conn(0x55666522baf0 msgr2=0x556665278340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc48c00640 0 client.8790 ms_handle_reset on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:33.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1371562406 shutdown_connections 2024-03-23T11:30:33.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc64e3a740 1 -- 172.21.15.62:0/1371562406 wait complete. 2024-03-23T11:30:33.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.773+0000 7fdc64e3a740 20 client.8790 trim_cache size 0 max 16384 2024-03-23T11:30:33.768 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:30:33.768 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-03-23T11:30:33.768 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "6d3ac3a4-a1f7-4be5-84c6-372aff6ccca3", 2024-03-23T11:30:33.768 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-03-23T11:30:33.768 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.774+0000 7fdc62200640 1 -- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc3806a7a0 msgr2=0x7fdc3806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.774+0000 7fdc62200640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc3806a7a0 0x7fdc3806cc60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c036000 tx=0x7fdc4c008030 comp rx=0 tx=0).stop 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.774+0000 7fdc62200640 1 -- 172.21.15.62:0/467409511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc5c160be0 msgr2=0x7fdc5c0b2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.774+0000 7fdc62200640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc5c160be0 0x7fdc5c0b2790 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fdc54001eb0 tx=0x7fdc5400c820 comp rx=0 tx=0).stop 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.774+0000 7fdc62200640 1 -- 172.21.15.62:0/467409511 shutdown_connections 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.774+0000 7fdc62200640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fdc3806a7a0 0x7fdc3806cc60 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.774+0000 7fdc62200640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc5c160be0 0x7fdc5c0b2790 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.774+0000 7fdc62200640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdc5c15ca70 0x7fdc5c0b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.774+0000 7fdc62200640 1 --2- 172.21.15.62:0/467409511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc5c0cf800 0x7fdc5c157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.774+0000 7fdc62200640 1 -- 172.21.15.62:0/467409511 >> 172.21.15.62:0/467409511 conn(0x7fdc5c071390 msgr2=0x7fdc5c0b7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.775+0000 7fdc62200640 1 -- 172.21.15.62:0/467409511 shutdown_connections 2024-03-23T11:30:33.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.775+0000 7fdc62200640 1 -- 172.21.15.62:0/467409511 wait complete. 2024-03-23T11:30:33.782 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-23T11:30:33.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.988+0000 7f7636c00640 1 Processor -- start 2024-03-23T11:30:33.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.988+0000 7f7636c00640 1 -- start start 2024-03-23T11:30:33.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.989+0000 7f7636c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f763015cba0 0x7f76300d3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.989+0000 7f7636c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76300ce8e0 0x7f76300b4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.989+0000 7f7636c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76300d3600 con 0x7f76300b46b0 2024-03-23T11:30:33.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.989+0000 7f7636c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763015dc10 con 0x7f763015cba0 2024-03-23T11:30:33.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.989+0000 7f7636c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76300b4b60 con 0x7f76300ce8e0 2024-03-23T11:30:33.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.989+0000 7f7635800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f763015cba0 0x7f76300d3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.989+0000 7f7635800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f763015cba0 0x7f76300d3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50190/0 (socket says 172.21.15.62:50190) 2024-03-23T11:30:33.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.989+0000 7f7635800640 1 -- 172.21.15.62:0/2738710300 learned_addr learned my addr 172.21.15.62:0/2738710300 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.990+0000 7f7634e00640 1 --2- 172.21.15.62:0/2738710300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76300ce8e0 0x7f76300b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.990+0000 7f7635800640 1 -- 172.21.15.62:0/2738710300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76300ce8e0 msgr2=0x7f76300b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.990+0000 7f7635800640 1 --2- 172.21.15.62:0/2738710300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76300ce8e0 0x7f76300b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.990+0000 7f7635800640 1 -- 172.21.15.62:0/2738710300 >> v1:172.21.15.120:6789/0 conn(0x7f76300b46b0 legacy=0x7f76301605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.990+0000 7f7635800640 1 -- 172.21.15.62:0/2738710300 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763015e030 con 0x7f763015cba0 2024-03-23T11:30:33.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.991+0000 7f7635800640 1 --2- 172.21.15.62:0/2738710300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f763015cba0 0x7f76300d3020 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7620009870 tx=0x7f762002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17bcba0b6fc203d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.991+0000 7f762fe00640 1 -- 172.21.15.62:0/2738710300 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76200317c0 con 0x7f763015cba0 2024-03-23T11:30:33.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.991+0000 7f762fe00640 1 -- 172.21.15.62:0/2738710300 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7620031920 con 0x7f763015cba0 2024-03-23T11:30:33.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.992+0000 7f762fe00640 1 -- 172.21.15.62:0/2738710300 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7620031cb0 con 0x7f763015cba0 2024-03-23T11:30:33.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.992+0000 7f7636c00640 1 -- 172.21.15.62:0/2738710300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f763015cba0 msgr2=0x7f76300d3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.992+0000 7f7636c00640 1 --2- 172.21.15.62:0/2738710300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f763015cba0 0x7f76300d3020 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7620009870 tx=0x7f762002b860 comp rx=0 tx=0).stop 2024-03-23T11:30:33.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.992+0000 7f7636c00640 1 -- 172.21.15.62:0/2738710300 shutdown_connections 2024-03-23T11:30:33.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.992+0000 7f7636c00640 1 --2- 172.21.15.62:0/2738710300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f76300ce8e0 0x7f76300b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.992+0000 7f7636c00640 1 --2- 172.21.15.62:0/2738710300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f763015cba0 0x7f76300d3020 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.992+0000 7f7636c00640 1 -- 172.21.15.62:0/2738710300 >> 172.21.15.62:0/2738710300 conn(0x7f7630089f60 msgr2=0x7f76300d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.992+0000 7f7636c00640 1 -- 172.21.15.62:0/2738710300 shutdown_connections 2024-03-23T11:30:33.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.992+0000 7f7636c00640 1 -- 172.21.15.62:0/2738710300 wait complete. 2024-03-23T11:30:33.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.994+0000 7f7636c00640 1 Processor -- start 2024-03-23T11:30:33.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.994+0000 7f7636c00640 1 -- start start 2024-03-23T11:30:33.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.995+0000 7f7636c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300b46b0 0x7f763015b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.995+0000 7f7636c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300ce8e0 0x7f7630154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.995+0000 7f7636c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f763015cba0 0x7f7630157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.995+0000 7f7636c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76301583e0 con 0x7f76300b46b0 2024-03-23T11:30:33.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.995+0000 7f7636c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7630154a70 con 0x7f76300ce8e0 2024-03-23T11:30:33.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.995+0000 7f7636c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7630158100 con 0x7f763015cba0 2024-03-23T11:30:33.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.995+0000 7f7635800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300b46b0 0x7f763015b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.995+0000 7f7635800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300b46b0 0x7f763015b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50198/0 (socket says 172.21.15.62:50198) 2024-03-23T11:30:33.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.995+0000 7f7635800640 1 -- 172.21.15.62:0/207713586 learned_addr learned my addr 172.21.15.62:0/207713586 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.996+0000 7f7634e00640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300ce8e0 0x7f7630154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.996+0000 7f7635800640 1 -- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f763015cba0 msgr2=0x7f7630157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.996+0000 7f7635800640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f763015cba0 0x7f7630157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.996+0000 7f7635800640 1 -- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300ce8e0 msgr2=0x7f7630154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.996+0000 7f7635800640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300ce8e0 0x7f7630154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.996+0000 7f7635800640 1 -- 172.21.15.62:0/207713586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76200094d0 con 0x7f76300b46b0 2024-03-23T11:30:33.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.996+0000 7f7635800640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300b46b0 0x7f763015b460 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f7620000c00 tx=0x7f76200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.997+0000 7f762d600640 1 -- 172.21.15.62:0/207713586 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76200329a0 con 0x7f76300b46b0 2024-03-23T11:30:33.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.997+0000 7f762d600640 1 -- 172.21.15.62:0/207713586 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7620032b40 con 0x7f76300b46b0 2024-03-23T11:30:33.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.997+0000 7f7636c00640 1 -- 172.21.15.62:0/207713586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76300c98d0 con 0x7f76300b46b0 2024-03-23T11:30:33.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.997+0000 7f762d600640 1 -- 172.21.15.62:0/207713586 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76200453f0 con 0x7f76300b46b0 2024-03-23T11:30:33.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.997+0000 7f7636c00640 1 -- 172.21.15.62:0/207713586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76300c9e90 con 0x7f76300b46b0 2024-03-23T11:30:33.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:33.999+0000 7f762d600640 1 -- 172.21.15.62:0/207713586 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f762004e450 con 0x7f76300b46b0 2024-03-23T11:30:33.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.000+0000 7f762d600640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f761006a7a0 0x7f761006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.000+0000 7f762d600640 1 -- 172.21.15.62:0/207713586 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f76200ab1d0 con 0x7f76300b46b0 2024-03-23T11:30:33.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.001+0000 7f7634e00640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f761006a7a0 0x7f761006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.001+0000 7f763af8f740 1 Processor -- start 2024-03-23T11:30:33.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.001+0000 7f763af8f740 1 -- start start 2024-03-23T11:30:33.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.002+0000 7f7634e00640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f761006a7a0 0x7f761006cc60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f761c006fd0 tx=0x7f761c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.003+0000 7f763af8f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722af0 0x55712b72ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.003+0000 7f763af8f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b702c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.003+0000 7f763af8f740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 0x55712b703160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.003+0000 7f763af8f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55712b6becb0 con 0x55712b72b460 2024-03-23T11:30:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.003+0000 7f763af8f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55712b66d500 con 0x55712b722af0 2024-03-23T11:30:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.003+0000 7f763af8f740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55712b66ac90 con 0x55712b722ed0 2024-03-23T11:30:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.003+0000 7f7634e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b702c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.003+0000 7f7634e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b702c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50212/0 (socket says 172.21.15.62:50212) 2024-03-23T11:30:33.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.003+0000 7f7634e00640 1 -- 172.21.15.62:0/4071554513 learned_addr learned my addr 172.21.15.62:0/4071554513 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:33.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.003+0000 7f7635800640 1 --2- 172.21.15.62:0/4071554513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722af0 0x55712b72ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.004+0000 7f7636200640 1 --2- 172.21.15.62:0/4071554513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 0x55712b703160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:33.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.004+0000 7f7634e00640 1 -- 172.21.15.62:0/4071554513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 msgr2=0x55712b703160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.004+0000 7f7634e00640 1 --2- 172.21.15.62:0/4071554513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 0x55712b703160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.004+0000 7f7634e00640 1 -- 172.21.15.62:0/4071554513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722af0 msgr2=0x55712b72ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:33.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.004+0000 7f7634e00640 1 --2- 172.21.15.62:0/4071554513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722af0 0x55712b72ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.004+0000 7f7634e00640 1 -- 172.21.15.62:0/4071554513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55712b618360 con 0x55712b72b460 2024-03-23T11:30:33.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.004+0000 7f7634e00640 1 --2- 172.21.15.62:0/4071554513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b702c20 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f761c00a850 tx=0x7f761c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23990f949d38263d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:33.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.005+0000 7f7626000640 1 -- 172.21.15.62:0/4071554513 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c013ca0 con 0x55712b72b460 2024-03-23T11:30:33.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.005+0000 7f7626000640 1 -- 172.21.15.62:0/4071554513 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c018070 con 0x55712b72b460 2024-03-23T11:30:33.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.005+0000 7f7626000640 1 -- 172.21.15.62:0/4071554513 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c01f440 con 0x55712b72b460 2024-03-23T11:30:33.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.006+0000 7f763af8f740 1 -- 172.21.15.62:0/4071554513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 msgr2=0x55712b702c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:33.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.006+0000 7f763af8f740 1 --2- 172.21.15.62:0/4071554513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b702c20 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f761c00a850 tx=0x7f761c011000 comp rx=0 tx=0).stop 2024-03-23T11:30:33.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.006+0000 7f763af8f740 1 -- 172.21.15.62:0/4071554513 shutdown_connections 2024-03-23T11:30:33.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.006+0000 7f763af8f740 1 --2- 172.21.15.62:0/4071554513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b702c20 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.006+0000 7f763af8f740 1 --2- 172.21.15.62:0/4071554513 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 0x55712b703160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.006+0000 7f763af8f740 1 --2- 172.21.15.62:0/4071554513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722af0 0x55712b72ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:33.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.006+0000 7f763af8f740 1 -- 172.21.15.62:0/4071554513 >> 172.21.15.62:0/4071554513 conn(0x55712b6cab40 msgr2=0x55712b748900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:33.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.006+0000 7f763af8f740 1 -- 172.21.15.62:0/4071554513 shutdown_connections 2024-03-23T11:30:33.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.006+0000 7f763af8f740 1 -- 172.21.15.62:0/4071554513 wait complete. 2024-03-23T11:30:33.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.007+0000 7f763af8f740 1 -- start start 2024-03-23T11:30:33.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.007+0000 7f763af8f740 1 Processor -- start 2024-03-23T11:30:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.008+0000 7f763af8f740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722af0 0x55712b712ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.008+0000 7f763af8f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722ed0 0x55712b7573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.008+0000 7f763af8f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b7578e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.008+0000 7f763af8f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55712b666580 con 0x55712b72b460 2024-03-23T11:30:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.008+0000 7f763af8f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55712b607a50 con 0x55712b722ed0 2024-03-23T11:30:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.008+0000 7f763af8f740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55712b6076f0 con 0x55712b722af0 2024-03-23T11:30:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.008+0000 7f763af8f740 10 client.0 _ll_register_callbacks cb 0x55712b656ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:30:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.008+0000 7f7636200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b7578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.008+0000 7f7636200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b7578e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50224/0 (socket says 172.21.15.62:50224) 2024-03-23T11:30:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.008+0000 7f7636200640 1 -- 172.21.15.62:0/2927429609 learned_addr learned my addr 172.21.15.62:0/2927429609 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:34.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.009+0000 7f7635800640 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722af0 0x55712b712ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.009+0000 7f7634e00640 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722ed0 0x55712b7573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.009+0000 7f7636200640 1 -- 172.21.15.62:0/2927429609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722af0 msgr2=0x55712b712ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.009+0000 7f7636200640 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722af0 0x55712b712ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.009+0000 7f7636200640 1 -- 172.21.15.62:0/2927429609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722ed0 msgr2=0x55712b7573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.009+0000 7f7636200640 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722ed0 0x55712b7573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.009+0000 7f7636200640 1 -- 172.21.15.62:0/2927429609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c00e820 con 0x55712b72b460 2024-03-23T11:30:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.009+0000 7f7636200640 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b7578e0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f762800c970 tx=0x7f762800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c48d14b9027d9a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.009+0000 7f761aa00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:30:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.009+0000 7f761aa00640 1 -- 172.21.15.62:0/2927429609 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7628011070 con 0x55712b72b460 2024-03-23T11:30:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f761aa00640 1 -- 172.21.15.62:0/2927429609 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7628007b40 con 0x55712b72b460 2024-03-23T11:30:34.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f763af8f740 1 -- 172.21.15.62:0/2927429609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55712b690fe0 con 0x55712b72b460 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f761aa00640 1 -- 172.21.15.62:0/2927429609 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7628007d00 con 0x55712b72b460 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f763af8f740 10 client.8802 fetch_fsmap learned FSMap version 84 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f763af8f740 1 -- 172.21.15.62:0/2927429609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x55712b664c00 con 0x55712b72b460 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f761aa00640 1 -- 172.21.15.62:0/2927429609 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55712b690fe0 con 0x55712b72b460 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f761aa00640 1 -- 172.21.15.62:0/2927429609 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 84) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7628015410 con 0x55712b72b460 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f763af8f740 10 client.8802 fetch_fsmap finished waiting for FSMap version 84 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f763af8f740 10 client.8802 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f763af8f740 20 client.8802 populate_metadata read hostname 'smithi062' 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f763af8f740 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x55712b763610 0x55712b765a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f763af8f740 4 client.8802 mds_command: new command op to 8691 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.010+0000 7f763af8f740 1 -- 172.21.15.62:0/2927429609 --> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55712b659240 con 0x55712b763610 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.011+0000 7f7635800640 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x55712b763610 0x55712b765a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:34.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.011+0000 7f7635800640 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x55712b763610 0x55712b765a00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8a8070d5c31c399 server_cookie=4310f5a11099aeee in_seq=0 out_seq=0 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.011+0000 7f761aa00640 10 client.8802 ms_handle_connect on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f761aa00640 1 -- 172.21.15.62:0/2927429609 <== mds.0 v2:172.21.15.62:6832/1070252332 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55712b659240 con 0x55712b763610 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f761aa00640 10 client.8802 handle_command_reply: tid=0 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f763af8f740 1 client.8802 shutdown 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f763af8f740 1 -- 172.21.15.62:0/2927429609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 msgr2=0x55712b7578e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f763af8f740 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b7578e0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f762800c970 tx=0x7f762800ce30 comp rx=0 tx=0).stop 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f763af8f740 1 -- 172.21.15.62:0/2927429609 shutdown_connections 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f763af8f740 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x55712b763610 0x55712b765a00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f763af8f740 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b7578e0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f763af8f740 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722ed0 0x55712b7573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f763af8f740 1 --2- 172.21.15.62:0/2927429609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722af0 0x55712b712ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f763af8f740 1 -- 172.21.15.62:0/2927429609 >> 172.21.15.62:0/2927429609 conn(0x55712b6cab40 msgr2=0x55712b7029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:34.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f761aa00640 0 client.8802 ms_handle_reset on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:34.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.012+0000 7f763af8f740 1 -- 172.21.15.62:0/2927429609 shutdown_connections 2024-03-23T11:30:34.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.013+0000 7f763af8f740 1 -- 172.21.15.62:0/2927429609 wait complete. 2024-03-23T11:30:34.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.013+0000 7f763af8f740 20 client.8802 trim_cache size 0 max 16384 2024-03-23T11:30:34.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.030+0000 7f763af8f740 1 Processor -- start 2024-03-23T11:30:34.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.030+0000 7f763af8f740 1 -- start start 2024-03-23T11:30:34.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.031+0000 7f763af8f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722af0 0x55712b74b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:34.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.031+0000 7f763af8f740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 0x55712b72a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:34.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.031+0000 7f763af8f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b72ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:34.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.031+0000 7f763af8f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55712b363f90 con 0x55712b72b460 2024-03-23T11:30:34.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.031+0000 7f763af8f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55712b7482e0 con 0x55712b722af0 2024-03-23T11:30:34.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.031+0000 7f763af8f740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55712b7042d0 con 0x55712b722ed0 2024-03-23T11:30:34.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.031+0000 7f7636200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b72ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:34.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.032+0000 7f7636200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b72ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50226/0 (socket says 172.21.15.62:50226) 2024-03-23T11:30:34.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.032+0000 7f7636200640 1 -- 172.21.15.62:0/3532776177 learned_addr learned my addr 172.21.15.62:0/3532776177 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:34.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.032+0000 7f7634e00640 1 --2- 172.21.15.62:0/3532776177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 0x55712b72a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:34.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.032+0000 7f7635800640 1 --2- 172.21.15.62:0/3532776177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722af0 0x55712b74b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:34.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.032+0000 7f7636200640 1 -- 172.21.15.62:0/3532776177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 msgr2=0x55712b72a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:34.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.032+0000 7f7636200640 1 --2- 172.21.15.62:0/3532776177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 0x55712b72a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.032+0000 7f7636200640 1 -- 172.21.15.62:0/3532776177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722af0 msgr2=0x55712b74b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:34.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.032+0000 7f7636200640 1 --2- 172.21.15.62:0/3532776177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722af0 0x55712b74b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.032+0000 7f7636200640 1 -- 172.21.15.62:0/3532776177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55712b606c60 con 0x55712b72b460 2024-03-23T11:30:34.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.032+0000 7f7636200640 1 --2- 172.21.15.62:0/3532776177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b72ac10 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f7628000f80 tx=0x7f7628011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a9ab3757a949c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:34.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f7626a00640 1 -- 172.21.15.62:0/3532776177 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7628009370 con 0x55712b72b460 2024-03-23T11:30:34.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f7626a00640 1 -- 172.21.15.62:0/3532776177 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7628007660 con 0x55712b72b460 2024-03-23T11:30:34.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f7626a00640 1 -- 172.21.15.62:0/3532776177 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7628015920 con 0x55712b72b460 2024-03-23T11:30:34.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f763af8f740 1 -- 172.21.15.62:0/3532776177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 msgr2=0x55712b72ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:34.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f763af8f740 1 --2- 172.21.15.62:0/3532776177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b72ac10 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f7628000f80 tx=0x7f7628011000 comp rx=0 tx=0).stop 2024-03-23T11:30:34.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f763af8f740 1 -- 172.21.15.62:0/3532776177 shutdown_connections 2024-03-23T11:30:34.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f763af8f740 1 --2- 172.21.15.62:0/3532776177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b72b460 0x55712b72ac10 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f763af8f740 1 --2- 172.21.15.62:0/3532776177 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 0x55712b72a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f763af8f740 1 --2- 172.21.15.62:0/3532776177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b722af0 0x55712b74b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f763af8f740 1 -- 172.21.15.62:0/3532776177 >> 172.21.15.62:0/3532776177 conn(0x55712b6cab40 msgr2=0x55712b769de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:34.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f763af8f740 1 -- 172.21.15.62:0/3532776177 shutdown_connections 2024-03-23T11:30:34.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.033+0000 7f763af8f740 1 -- 172.21.15.62:0/3532776177 wait complete. 2024-03-23T11:30:34.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.034+0000 7f763af8f740 1 -- start start 2024-03-23T11:30:34.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.034+0000 7f763af8f740 1 Processor -- start 2024-03-23T11:30:34.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.035+0000 7f763af8f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b722af0 0x55712b7aea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:34.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.035+0000 7f763af8f740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 0x55712b721e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:34.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.035+0000 7f763af8f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b72b460 0x55712b7b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:34.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.035+0000 7f763af8f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55712b74bd10 con 0x55712b722af0 2024-03-23T11:30:34.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.035+0000 7f763af8f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55712b74b540 con 0x55712b72b460 2024-03-23T11:30:34.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.035+0000 7f763af8f740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55712b7a4fa0 con 0x55712b722ed0 2024-03-23T11:30:34.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.035+0000 7f763af8f740 10 client.0 _ll_register_callbacks cb 0x7f761c00a1d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:30:34.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f7635800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b722af0 0x55712b7aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:34.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f7635800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b722af0 0x55712b7aea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50228/0 (socket says 172.21.15.62:50228) 2024-03-23T11:30:34.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f7635800640 1 -- 172.21.15.62:0/3486263673 learned_addr learned my addr 172.21.15.62:0/3486263673 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:30:34.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f7636200640 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b72b460 0x55712b7b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:34.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f7635800640 1 -- 172.21.15.62:0/3486263673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 msgr2=0x55712b721e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:34.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f7635800640 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 0x55712b721e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f7635800640 1 -- 172.21.15.62:0/3486263673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b72b460 msgr2=0x55712b7b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:30:34.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f7635800640 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b72b460 0x55712b7b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f7635800640 1 -- 172.21.15.62:0/3486263673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55712b664c00 con 0x55712b722af0 2024-03-23T11:30:34.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f7636200640 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b72b460 0x55712b7b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:30:34.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f7635800640 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b722af0 0x55712b7aea80 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f76200766b0 tx=0x7f7620006ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2fc7450cc93fbe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:30:34.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f761aa00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:30:34.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.036+0000 7f761aa00640 1 -- 172.21.15.62:0/3486263673 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76200825a0 con 0x55712b722af0 2024-03-23T11:30:34.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f761aa00640 1 -- 172.21.15.62:0/3486263673 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f76200a9210 con 0x55712b722af0 2024-03-23T11:30:34.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f763af8f740 1 -- 172.21.15.62:0/3486263673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f761c018070 con 0x55712b722af0 2024-03-23T11:30:34.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f761aa00640 1 -- 172.21.15.62:0/3486263673 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76200768c0 con 0x55712b722af0 2024-03-23T11:30:34.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f763af8f740 10 client.8808 fetch_fsmap learned FSMap version 84 2024-03-23T11:30:34.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f763af8f740 1 -- 172.21.15.62:0/3486263673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x55712b62c170 con 0x55712b722af0 2024-03-23T11:30:34.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f761aa00640 1 -- 172.21.15.62:0/3486263673 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c018070 con 0x55712b722af0 2024-03-23T11:30:34.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f761aa00640 1 -- 172.21.15.62:0/3486263673 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 84) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f76200796a0 con 0x55712b722af0 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f763af8f740 10 client.8808 fetch_fsmap finished waiting for FSMap version 84 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f763af8f740 10 client.8808 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f763af8f740 20 client.8808 populate_metadata read hostname 'smithi062' 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f763af8f740 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x55712b7020f0 0x55712b72a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f763af8f740 4 client.8808 mds_command: new command op to 8691 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f763af8f740 1 -- 172.21.15.62:0/3486263673 --> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55712b35a4e0 con 0x55712b7020f0 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.037+0000 7f7634e00640 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x55712b7020f0 0x55712b72a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.038+0000 7f7634e00640 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x55712b7020f0 0x55712b72a840 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51fb084da7d16997 server_cookie=289bb033e82c37fc in_seq=0 out_seq=0 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.038+0000 7f761aa00640 10 client.8808 ms_handle_connect on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.038+0000 7f761aa00640 1 -- 172.21.15.62:0/3486263673 <== mds.0 v2:172.21.15.62:6832/1070252332 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x55712b35a4e0 con 0x55712b7020f0 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.038+0000 7f761aa00640 10 client.8808 handle_command_reply: tid=0 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.038+0000 7f763af8f740 1 client.8808 shutdown 2024-03-23T11:30:34.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f763af8f740 1 -- 172.21.15.62:0/3486263673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b722af0 msgr2=0x55712b7aea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f763af8f740 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b722af0 0x55712b7aea80 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f76200766b0 tx=0x7f7620006ed0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f763af8f740 1 -- 172.21.15.62:0/3486263673 shutdown_connections 2024-03-23T11:30:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f763af8f740 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55712b72b460 0x55712b7b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f763af8f740 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55712b722ed0 0x55712b721e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f763af8f740 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x55712b7020f0 0x55712b72a840 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f763af8f740 1 --2- 172.21.15.62:0/3486263673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55712b722af0 0x55712b7aea80 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f763af8f740 1 -- 172.21.15.62:0/3486263673 >> 172.21.15.62:0/3486263673 conn(0x55712b6cab40 msgr2=0x55712b6f96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f761aa00640 0 client.8808 ms_handle_reset on v2:172.21.15.62:6832/1070252332 2024-03-23T11:30:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f763af8f740 1 -- 172.21.15.62:0/3486263673 shutdown_connections 2024-03-23T11:30:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f763af8f740 1 -- 172.21.15.62:0/3486263673 wait complete. 2024-03-23T11:30:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.039+0000 7f763af8f740 20 client.8808 trim_cache size 0 max 16384 2024-03-23T11:30:34.038 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:30:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-23T11:30:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-03-23T11:30:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "6d3ac3a4-a1f7-4be5-84c6-372aff6ccca3": { 2024-03-23T11:30:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-03-23T11:30:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "6d3ac3a4-a1f7-4be5-84c6-372aff6ccca3", 2024-03-23T11:30:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,repair" 2024-03-23T11:30:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:30:34.038 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:30:34.038 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 -- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f761006a7a0 msgr2=0x7f761006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f761006a7a0 0x7f761006cc60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f761c006fd0 tx=0x7f761c008040 comp rx=0 tx=0).stop 2024-03-23T11:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 -- 172.21.15.62:0/207713586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300b46b0 msgr2=0x7f763015b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300b46b0 0x7f763015b460 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f7620000c00 tx=0x7f76200046c0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 -- 172.21.15.62:0/207713586 shutdown_connections 2024-03-23T11:30:34.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f761006a7a0 0x7f761006cc60 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f763015cba0 0x7f7630157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76300ce8e0 0x7f7630154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 --2- 172.21.15.62:0/207713586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300b46b0 0x7f763015b460 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:30:34.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 -- 172.21.15.62:0/207713586 >> 172.21.15.62:0/207713586 conn(0x7f7630089f60 msgr2=0x7f76300b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:30:34.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 -- 172.21.15.62:0/207713586 shutdown_connections 2024-03-23T11:30:34.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:34.040+0000 7f7636c00640 1 -- 172.21.15.62:0/207713586 wait complete. 2024-03-23T11:30:34.047 INFO:tasks.cephfs.filesystem:scrub status for tag:6d3ac3a4-a1f7-4be5-84c6-372aff6ccca3 - {'path': '/', 'tag': '6d3ac3a4-a1f7-4be5-84c6-372aff6ccca3', 'options': 'recursive,repair'} 2024-03-23T11:30:35.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:35.534+0000 7f661aa00640 1 -- 172.21.15.62:0/3621539899 <== mon.0 v2:172.21.15.62:3300/0 7 ==== log(8 entries from seq 1 at 2024-03-23T11:30:33.772838+0000) v1 ==== 1407+0+0 (secure 0 0 0) 0x7f6614059530 con 0x7f66241440d0 2024-03-23T11:30:35.525 INFO:teuthology.orchestra.run.smithi062.stdout: cluster: 2024-03-23T11:30:35.525 INFO:teuthology.orchestra.run.smithi062.stdout: id: e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:30:35.525 INFO:teuthology.orchestra.run.smithi062.stdout: health: HEALTH_OK 2024-03-23T11:30:35.525 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:30:35.525 INFO:teuthology.orchestra.run.smithi062.stdout: services: 2024-03-23T11:30:35.525 INFO:teuthology.orchestra.run.smithi062.stdout: mon: 3 daemons, quorum a,b,c (age 11m) 2024-03-23T11:30:35.525 INFO:teuthology.orchestra.run.smithi062.stdout: mgr: x(active, since 11m), standbys: y 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: mds: 1/1 daemons up, 3 standby 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: osd: 8 osds: 8 up (since 11m), 8 in (since 11m) 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: data: 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: volumes: 1/1 healthy 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: pools: 3 pools, 97 pgs 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: objects: 30 objects, 19 MiB 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: usage: 290 MiB used, 720 GiB / 720 GiB avail 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: pgs: 97 active+clean 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: io: 2024-03-23T11:30:35.526 INFO:teuthology.orchestra.run.smithi062.stdout: client: 8.9 KiB/s rd, 102 B/s wr, 8 op/s rd, 0 op/s wr 2024-03-23T11:30:35.527 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:30:35.527 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:30:35.527 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:30:33.772838+0000 mds.a [INF] scrub queued for path: / 2024-03-23T11:30:35.527 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:30:33.772843+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-23T11:30:35.527 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:30:33.772848+0000 mds.a [INF] scrub summary: active paths [/] 2024-03-23T11:30:35.527 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:30:33.773686+0000 mds.a [ERR] scrub: inode wrongly marked free: 0x10000000001 2024-03-23T11:30:35.527 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:30:33.773695+0000 mds.a [ERR] inode table repaired for inode: 0x10000000001 2024-03-23T11:30:35.527 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:30:33.773776+0000 mds.a [ERR] scrub: inode wrongly marked free: 0x10000000002 2024-03-23T11:30:35.527 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:30:33.773782+0000 mds.a [ERR] inode table repaired for inode: 0x10000000002 2024-03-23T11:30:35.527 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:30:33.775093+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-03-23T11:30:39.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:30:39.556+0000 7f661aa00640 1 -- 172.21.15.62:0/3621539899 <== mon.0 v2:172.21.15.62:3300/0 8 ==== log(1 entries from seq 9 at 2024-03-23T11:30:37.552342+0000) v1 ==== 189+0+0 (secure 0 0 0) 0x7f661405cc60 con 0x7f66241440d0 2024-03-23T11:30:39.548 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:30:37.552342+0000 mds.a [INF] scrub summary: idle 2024-03-23T11:30:47.788 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:30:47.796 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:31:04.048 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub status 2024-03-23T11:31:04.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.286+0000 7f675a400640 1 Processor -- start 2024-03-23T11:31:04.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.286+0000 7f675a400640 1 -- start start 2024-03-23T11:31:04.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.288+0000 7f675a400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f675415cba0 0x7f67540d3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.288+0000 7f675a400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f67540ce8e0 0x7f67540b4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.288+0000 7f675a400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67540d3600 con 0x7f67540b46b0 2024-03-23T11:31:04.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.288+0000 7f675a400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f675415dc10 con 0x7f67540ce8e0 2024-03-23T11:31:04.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.288+0000 7f675a400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67540b4ac0 con 0x7f675415cba0 2024-03-23T11:31:04.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.288+0000 7f6759a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f67540b46b0 0x7f67541605d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47582/0 (socket says 172.21.15.62:47582) 2024-03-23T11:31:04.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.288+0000 7f6759a00640 1 -- 172.21.15.62:0/499135775 learned_addr learned my addr 172.21.15.62:0/499135775 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:04.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.288+0000 7f6753e00640 1 --2- 172.21.15.62:0/499135775 >> v2:172.21.15.62:3300/0 conn(0x7f67540ce8e0 0x7f67540b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.289+0000 7f6759000640 1 --2- 172.21.15.62:0/499135775 >> v2:172.21.15.120:3300/0 conn(0x7f675415cba0 0x7f67540d3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.289+0000 7f6753400640 1 -- 172.21.15.62:0/499135775 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153712536 0 0) 0x7f67540d3600 con 0x7f67540b46b0 2024-03-23T11:31:04.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.289+0000 7f6753400640 1 -- 172.21.15.62:0/499135775 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f673c003520 con 0x7f67540b46b0 2024-03-23T11:31:04.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.289+0000 7f6753e00640 1 -- 172.21.15.62:0/499135775 >> v2:172.21.15.120:3300/0 conn(0x7f675415cba0 msgr2=0x7f67540d3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:04.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.289+0000 7f6753e00640 1 --2- 172.21.15.62:0/499135775 >> v2:172.21.15.120:3300/0 conn(0x7f675415cba0 0x7f67540d3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.289+0000 7f6753e00640 1 -- 172.21.15.62:0/499135775 >> v1:172.21.15.62:6789/0 conn(0x7f67540b46b0 legacy=0x7f67541605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.289+0000 7f6753e00640 1 -- 172.21.15.62:0/499135775 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675415e030 con 0x7f67540ce8e0 2024-03-23T11:31:04.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.290+0000 7f6753e00640 1 --2- 172.21.15.62:0/499135775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67540ce8e0 0x7f67540b4170 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f675408aad0 tx=0x7f674002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77ad894c6595168 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:04.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.290+0000 7f6753400640 1 -- 172.21.15.62:0/499135775 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674002e070 con 0x7f67540ce8e0 2024-03-23T11:31:04.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.290+0000 7f6753400640 1 -- 172.21.15.62:0/499135775 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67400315a0 con 0x7f67540ce8e0 2024-03-23T11:31:04.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.290+0000 7f6753400640 1 -- 172.21.15.62:0/499135775 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6740031930 con 0x7f67540ce8e0 2024-03-23T11:31:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.291+0000 7f675a400640 1 -- 172.21.15.62:0/499135775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67540ce8e0 msgr2=0x7f67540b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.291+0000 7f675a400640 1 --2- 172.21.15.62:0/499135775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67540ce8e0 0x7f67540b4170 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f675408aad0 tx=0x7f674002b840 comp rx=0 tx=0).stop 2024-03-23T11:31:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.291+0000 7f675a400640 1 -- 172.21.15.62:0/499135775 shutdown_connections 2024-03-23T11:31:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.291+0000 7f675a400640 1 --2- 172.21.15.62:0/499135775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67540ce8e0 0x7f67540b4170 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.291+0000 7f675a400640 1 --2- 172.21.15.62:0/499135775 >> v2:172.21.15.120:3300/0 conn(0x7f675415cba0 0x7f67540d3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.291+0000 7f675a400640 1 -- 172.21.15.62:0/499135775 >> 172.21.15.62:0/499135775 conn(0x7f6754089f60 msgr2=0x7f67540d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.291+0000 7f675a400640 1 -- 172.21.15.62:0/499135775 shutdown_connections 2024-03-23T11:31:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.291+0000 7f675a400640 1 -- 172.21.15.62:0/499135775 wait complete. 2024-03-23T11:31:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.293+0000 7f675a400640 1 Processor -- start 2024-03-23T11:31:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.293+0000 7f675a400640 1 -- start start 2024-03-23T11:31:04.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.294+0000 7f675a400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67540b46b0 0x7f67540ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.294+0000 7f675a400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67540ce8e0 0x7f675409d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.294+0000 7f675a400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f675415cba0 0x7f6754150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.294+0000 7f675a400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67540ca680 con 0x7f67540b46b0 2024-03-23T11:31:04.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.294+0000 7f675a400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675409d7e0 con 0x7f675415cba0 2024-03-23T11:31:04.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.294+0000 7f675a400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675409f6c0 con 0x7f67540ce8e0 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.294+0000 7f6759000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67540b46b0 0x7f67540ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.294+0000 7f6759000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67540b46b0 0x7f67540ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41366/0 (socket says 172.21.15.62:41366) 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.295+0000 7f6759000640 1 -- 172.21.15.62:0/1094074428 learned_addr learned my addr 172.21.15.62:0/1094074428 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.295+0000 7f6753e00640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67540ce8e0 0x7f675409d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.295+0000 7f6759a00640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f675415cba0 0x7f6754150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.295+0000 7f6759000640 1 -- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67540ce8e0 msgr2=0x7f675409d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.295+0000 7f6759000640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67540ce8e0 0x7f675409d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.295+0000 7f6759000640 1 -- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f675415cba0 msgr2=0x7f6754150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.295+0000 7f6759000640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f675415cba0 0x7f6754150b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.295+0000 7f6759000640 1 -- 172.21.15.62:0/1094074428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6740009400 con 0x7f67540b46b0 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.295+0000 7f6753e00640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67540ce8e0 0x7f675409d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.295+0000 7f6759000640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67540b46b0 0x7f67540ca140 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f674800e8a0 tx=0x7f674800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:04.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.296+0000 7f6750c00640 1 -- 172.21.15.62:0/1094074428 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67480094f0 con 0x7f67540b46b0 2024-03-23T11:31:04.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.296+0000 7f6750c00640 1 -- 172.21.15.62:0/1094074428 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674800c940 con 0x7f67540b46b0 2024-03-23T11:31:04.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.296+0000 7f675a400640 1 -- 172.21.15.62:0/1094074428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67540ca890 con 0x7f67540b46b0 2024-03-23T11:31:04.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.296+0000 7f6750c00640 1 -- 172.21.15.62:0/1094074428 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674800ccf0 con 0x7f67540b46b0 2024-03-23T11:31:04.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.296+0000 7f675a400640 1 -- 172.21.15.62:0/1094074428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675409f070 con 0x7f67540b46b0 2024-03-23T11:31:04.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.298+0000 7f6750c00640 1 -- 172.21.15.62:0/1094074428 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6748013470 con 0x7f67540b46b0 2024-03-23T11:31:04.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.299+0000 7f6750c00640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f673406a7a0 0x7f673406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.299+0000 7f6750c00640 1 -- 172.21.15.62:0/1094074428 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6748088f60 con 0x7f67540b46b0 2024-03-23T11:31:04.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.299+0000 7f6753e00640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f673406a7a0 0x7f673406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.300+0000 7f675e57a740 1 Processor -- start 2024-03-23T11:31:04.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.300+0000 7f675e57a740 1 -- start start 2024-03-23T11:31:04.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.301+0000 7f6753e00640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f673406a7a0 0x7f673406cc60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6740036000 tx=0x7f674003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.301+0000 7f675e57a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e21bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.301+0000 7f675e57a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e1f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.301+0000 7f675e57a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213ed0 0x560c4e1f4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.301+0000 7f675e57a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e1afcb0 con 0x560c4e21c460 2024-03-23T11:31:04.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.301+0000 7f675e57a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e15e500 con 0x560c4e213af0 2024-03-23T11:31:04.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.301+0000 7f675e57a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c4e15bc90 con 0x560c4e213ed0 2024-03-23T11:31:04.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.302+0000 7f6753e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e1f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.302+0000 7f6753e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e1f3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41374/0 (socket says 172.21.15.62:41374) 2024-03-23T11:31:04.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.302+0000 7f6753e00640 1 -- 172.21.15.62:0/1282717831 learned_addr learned my addr 172.21.15.62:0/1282717831 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:04.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.302+0000 7f6759000640 1 --2- 172.21.15.62:0/1282717831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e21bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.302+0000 7f6759a00640 1 --2- 172.21.15.62:0/1282717831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213ed0 0x560c4e1f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.302+0000 7f6753e00640 1 -- 172.21.15.62:0/1282717831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213ed0 msgr2=0x560c4e1f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:04.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.302+0000 7f6753e00640 1 --2- 172.21.15.62:0/1282717831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213ed0 0x560c4e1f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.302+0000 7f6753e00640 1 -- 172.21.15.62:0/1282717831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 msgr2=0x560c4e21bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:04.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.302+0000 7f6753e00640 1 --2- 172.21.15.62:0/1282717831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e21bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.302+0000 7f6753e00640 1 -- 172.21.15.62:0/1282717831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c4e109360 con 0x560c4e21c460 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.302+0000 7f6759000640 1 --2- 172.21.15.62:0/1282717831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e21bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.303+0000 7f6753e00640 1 --2- 172.21.15.62:0/1282717831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e1f3c20 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f674002b880 tx=0x7f6740030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa9eb83a8a610f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.303+0000 7f6745600640 1 -- 172.21.15.62:0/1282717831 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6740032a40 con 0x560c4e21c460 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.303+0000 7f6745600640 1 -- 172.21.15.62:0/1282717831 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6740032ba0 con 0x560c4e21c460 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.303+0000 7f6745600640 1 -- 172.21.15.62:0/1282717831 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6740045420 con 0x560c4e21c460 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.303+0000 7f675e57a740 1 -- 172.21.15.62:0/1282717831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 msgr2=0x560c4e1f3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.303+0000 7f675e57a740 1 --2- 172.21.15.62:0/1282717831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e1f3c20 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f674002b880 tx=0x7f6740030000 comp rx=0 tx=0).stop 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.304+0000 7f675e57a740 1 -- 172.21.15.62:0/1282717831 shutdown_connections 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.304+0000 7f675e57a740 1 --2- 172.21.15.62:0/1282717831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e1f3c20 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.304+0000 7f675e57a740 1 --2- 172.21.15.62:0/1282717831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213ed0 0x560c4e1f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.304+0000 7f675e57a740 1 --2- 172.21.15.62:0/1282717831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e21bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.304+0000 7f675e57a740 1 -- 172.21.15.62:0/1282717831 >> 172.21.15.62:0/1282717831 conn(0x560c4e1bbb40 msgr2=0x560c4e239900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.304+0000 7f675e57a740 1 -- 172.21.15.62:0/1282717831 shutdown_connections 2024-03-23T11:31:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.304+0000 7f675e57a740 1 -- 172.21.15.62:0/1282717831 wait complete. 2024-03-23T11:31:04.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.304+0000 7f675e57a740 1 -- start start 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.305+0000 7f675e57a740 1 Processor -- start 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f675e57a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e213240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f675e57a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213ed0 0x560c4e215000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f675e57a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e2494d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f675e57a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e157580 con 0x560c4e21c460 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f675e57a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e0f8a50 con 0x560c4e213af0 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f675e57a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c4e0f86f0 con 0x560c4e213ed0 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f675e57a740 10 client.0 _ll_register_callbacks cb 0x560c4e147ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f6759a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e2494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f6759a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e2494d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41378/0 (socket says 172.21.15.62:41378) 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f6759a00640 1 -- 172.21.15.62:0/3053756018 learned_addr learned my addr 172.21.15.62:0/3053756018 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f6759000640 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e213240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f6759a00640 1 -- 172.21.15.62:0/3053756018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213ed0 msgr2=0x560c4e215000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f6759a00640 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213ed0 0x560c4e215000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f6759a00640 1 -- 172.21.15.62:0/3053756018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 msgr2=0x560c4e213240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f6759a00640 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e213240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.306+0000 7f6759a00640 1 -- 172.21.15.62:0/3053756018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675415e030 con 0x560c4e21c460 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.307+0000 7f6759a00640 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e2494d0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f674c00f820 tx=0x7f674c00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b10fb6de3934b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.307+0000 7f673a000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.307+0000 7f673a000640 1 -- 172.21.15.62:0/3053756018 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c005de0 con 0x560c4e21c460 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.307+0000 7f673a000640 1 -- 172.21.15.62:0/3053756018 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f674c00d730 con 0x560c4e21c460 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.307+0000 7f675e57a740 1 -- 172.21.15.62:0/3053756018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560c4e181fe0 con 0x560c4e21c460 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.307+0000 7f673a000640 1 -- 172.21.15.62:0/3053756018 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f674c001be0 con 0x560c4e21c460 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.307+0000 7f675e57a740 10 client.8823 fetch_fsmap learned FSMap version 84 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.307+0000 7f675e57a740 1 -- 172.21.15.62:0/3053756018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x560c4e155c00 con 0x560c4e21c460 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.307+0000 7f673a000640 1 -- 172.21.15.62:0/3053756018 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560c4e181fe0 con 0x560c4e21c460 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.308+0000 7f673a000640 1 -- 172.21.15.62:0/3053756018 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 84) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f674c012070 con 0x560c4e21c460 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.308+0000 7f675e57a740 10 client.8823 fetch_fsmap finished waiting for FSMap version 84 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.308+0000 7f675e57a740 10 client.8823 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:31:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.308+0000 7f675e57a740 20 client.8823 populate_metadata read hostname 'smithi062' 2024-03-23T11:31:04.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.308+0000 7f675e57a740 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x560c4e24a120 0x560c4e21ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.308+0000 7f675e57a740 4 client.8823 mds_command: new command op to 8691 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:31:04.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.308+0000 7f675e57a740 1 -- 172.21.15.62:0/3053756018 --> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c4e14a240 con 0x560c4e24a120 2024-03-23T11:31:04.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.308+0000 7f6759000640 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x560c4e24a120 0x560c4e21ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.309+0000 7f6759000640 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x560c4e24a120 0x560c4e21ed40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=416667ef4ecaa1c4 server_cookie=baee027b5414b722 in_seq=0 out_seq=0 2024-03-23T11:31:04.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.309+0000 7f673a000640 10 client.8823 ms_handle_connect on v2:172.21.15.62:6832/1070252332 2024-03-23T11:31:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.309+0000 7f673a000640 1 -- 172.21.15.62:0/3053756018 <== mds.0 v2:172.21.15.62:6832/1070252332 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x560c4e14a240 con 0x560c4e24a120 2024-03-23T11:31:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.309+0000 7f673a000640 10 client.8823 handle_command_reply: tid=0 2024-03-23T11:31:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.309+0000 7f675e57a740 1 client.8823 shutdown 2024-03-23T11:31:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.309+0000 7f675e57a740 1 -- 172.21.15.62:0/3053756018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 msgr2=0x560c4e2494d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.309+0000 7f675e57a740 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e2494d0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f674c00f820 tx=0x7f674c00fce0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.310+0000 7f675e57a740 1 -- 172.21.15.62:0/3053756018 shutdown_connections 2024-03-23T11:31:04.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.310+0000 7f675e57a740 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x560c4e24a120 0x560c4e21ed40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.310+0000 7f675e57a740 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e21c460 0x560c4e2494d0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.310+0000 7f675e57a740 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213ed0 0x560c4e215000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.310+0000 7f675e57a740 1 --2- 172.21.15.62:0/3053756018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e213240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.310+0000 7f675e57a740 1 -- 172.21.15.62:0/3053756018 >> 172.21.15.62:0/3053756018 conn(0x560c4e1bbb40 msgr2=0x560c4e239150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:04.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.310+0000 7f673a000640 0 client.8823 ms_handle_reset on v2:172.21.15.62:6832/1070252332 2024-03-23T11:31:04.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.310+0000 7f675e57a740 1 -- 172.21.15.62:0/3053756018 shutdown_connections 2024-03-23T11:31:04.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.310+0000 7f675e57a740 1 -- 172.21.15.62:0/3053756018 wait complete. 2024-03-23T11:31:04.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.310+0000 7f675e57a740 20 client.8823 trim_cache size 0 max 16384 2024-03-23T11:31:04.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.326+0000 7f675e57a740 1 Processor -- start 2024-03-23T11:31:04.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.326+0000 7f675e57a740 1 -- start start 2024-03-23T11:31:04.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.326+0000 7f675e57a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213af0 0x560c4e2391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.326+0000 7f675e57a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e1f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.326+0000 7f675e57a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e21c460 0x560c4e1ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.326+0000 7f675e57a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4de54f90 con 0x560c4e213ed0 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.327+0000 7f675e57a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e2305d0 con 0x560c4e21c460 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.327+0000 7f675e57a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c4e1f4470 con 0x560c4e213af0 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.327+0000 7f6753e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e1f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.327+0000 7f6753e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e1f39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41386/0 (socket says 172.21.15.62:41386) 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.327+0000 7f6753e00640 1 -- 172.21.15.62:0/2952148011 learned_addr learned my addr 172.21.15.62:0/2952148011 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.327+0000 7f6753e00640 1 -- 172.21.15.62:0/2952148011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213af0 msgr2=0x560c4e2391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.327+0000 7f6753e00640 1 --2- 172.21.15.62:0/2952148011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213af0 0x560c4e2391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.327+0000 7f6753e00640 1 -- 172.21.15.62:0/2952148011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e21c460 msgr2=0x560c4e1ee0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.327+0000 7f6753e00640 1 --2- 172.21.15.62:0/2952148011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e21c460 0x560c4e1ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.327+0000 7f6753e00640 1 -- 172.21.15.62:0/2952148011 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c4e0f7c60 con 0x560c4e213ed0 2024-03-23T11:31:04.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f6753e00640 1 --2- 172.21.15.62:0/2952148011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e1f39e0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f674002be50 tx=0x7f674002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8698e3cd2502594 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:04.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f6746000640 1 -- 172.21.15.62:0/2952148011 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6740004230 con 0x560c4e213ed0 2024-03-23T11:31:04.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f6746000640 1 -- 172.21.15.62:0/2952148011 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6740004390 con 0x560c4e213ed0 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f6746000640 1 -- 172.21.15.62:0/2952148011 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6740003830 con 0x560c4e213ed0 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f675e57a740 1 -- 172.21.15.62:0/2952148011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 msgr2=0x560c4e1f39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f675e57a740 1 --2- 172.21.15.62:0/2952148011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e1f39e0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f674002be50 tx=0x7f674002e000 comp rx=0 tx=0).stop 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f675e57a740 1 -- 172.21.15.62:0/2952148011 shutdown_connections 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f675e57a740 1 --2- 172.21.15.62:0/2952148011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e1f39e0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f675e57a740 1 --2- 172.21.15.62:0/2952148011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e21c460 0x560c4e1ee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f675e57a740 1 --2- 172.21.15.62:0/2952148011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e213af0 0x560c4e2391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f675e57a740 1 -- 172.21.15.62:0/2952148011 >> 172.21.15.62:0/2952148011 conn(0x560c4e1bbb40 msgr2=0x560c4e213050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.328+0000 7f675e57a740 1 -- 172.21.15.62:0/2952148011 shutdown_connections 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.329+0000 7f675e57a740 1 -- 172.21.15.62:0/2952148011 wait complete. 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.329+0000 7f675e57a740 1 -- start start 2024-03-23T11:31:04.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.329+0000 7f675e57a740 1 Processor -- start 2024-03-23T11:31:04.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.330+0000 7f675e57a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e1f2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.330+0000 7f675e57a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e212010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.330+0000 7f675e57a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e21c460 0x560c4e212550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.330+0000 7f675e57a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e23ddc0 con 0x560c4e213ed0 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.330+0000 7f675e57a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c4e24af60 con 0x560c4e213af0 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.330+0000 7f675e57a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c4e23d890 con 0x560c4e21c460 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.330+0000 7f675e57a740 10 client.0 _ll_register_callbacks cb 0x7f673c0036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f6753e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e212010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f6753e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e212010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41402/0 (socket says 172.21.15.62:41402) 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f6753e00640 1 -- 172.21.15.62:0/1090644350 learned_addr learned my addr 172.21.15.62:0/1090644350 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f6759000640 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e1f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f6753e00640 1 -- 172.21.15.62:0/1090644350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e21c460 msgr2=0x560c4e212550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f6753e00640 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e21c460 0x560c4e212550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f6753e00640 1 -- 172.21.15.62:0/1090644350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 msgr2=0x560c4e1f2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:04.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f6753e00640 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e1f2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f6753e00640 1 -- 172.21.15.62:0/1090644350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c4e109360 con 0x560c4e213ed0 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f6759000640 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e1f2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f6753e00640 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e212010 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f674002b880 tx=0x7f6740004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fdb684c9bcc0000 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f673a000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.331+0000 7f673a000640 1 -- 172.21.15.62:0/1090644350 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6740003dd0 con 0x560c4e213ed0 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f673a000640 1 -- 172.21.15.62:0/1090644350 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6740058450 con 0x560c4e213ed0 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f675e57a740 1 -- 172.21.15.62:0/1090644350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6740032ba0 con 0x560c4e213ed0 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f673a000640 1 -- 172.21.15.62:0/1090644350 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6740004500 con 0x560c4e213ed0 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f675e57a740 10 client.8829 fetch_fsmap learned FSMap version 84 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f675e57a740 1 -- 172.21.15.62:0/1090644350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x560c4e11d170 con 0x560c4e213ed0 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f673a000640 1 -- 172.21.15.62:0/1090644350 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6740032ba0 con 0x560c4e213ed0 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f673a000640 1 -- 172.21.15.62:0/1090644350 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 84) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6740056070 con 0x560c4e213ed0 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f675e57a740 10 client.8829 fetch_fsmap finished waiting for FSMap version 84 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f675e57a740 10 client.8829 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f675e57a740 20 client.8829 populate_metadata read hostname 'smithi062' 2024-03-23T11:31:04.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f675e57a740 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x560c4e24a860 0x560c4e2143c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f675e57a740 4 client.8829 mds_command: new command op to 8691 tid=0[{"prefix": "scrub status"}] 2024-03-23T11:31:04.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f675e57a740 1 -- 172.21.15.62:0/1090644350 --> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560c4de4b4e0 con 0x560c4e24a860 2024-03-23T11:31:04.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.332+0000 7f6759000640 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x560c4e24a860 0x560c4e2143c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.333+0000 7f6759000640 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x560c4e24a860 0x560c4e2143c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c55adf688c9cb78 server_cookie=43d106a610d42a25 in_seq=0 out_seq=0 2024-03-23T11:31:04.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.333+0000 7f673a000640 10 client.8829 ms_handle_connect on v2:172.21.15.62:6832/1070252332 2024-03-23T11:31:04.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.333+0000 7f673a000640 1 -- 172.21.15.62:0/1090644350 <== mds.0 v2:172.21.15.62:6832/1070252332 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560c4de4b4e0 con 0x560c4e24a860 2024-03-23T11:31:04.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.333+0000 7f673a000640 10 client.8829 handle_command_reply: tid=0 2024-03-23T11:31:04.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.333+0000 7f675e57a740 1 client.8829 shutdown 2024-03-23T11:31:04.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.333+0000 7f675e57a740 1 -- 172.21.15.62:0/1090644350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 msgr2=0x560c4e212010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.333+0000 7f675e57a740 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e212010 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f674002b880 tx=0x7f6740004730 comp rx=0 tx=0).stop 2024-03-23T11:31:04.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.333+0000 7f675e57a740 1 -- 172.21.15.62:0/1090644350 shutdown_connections 2024-03-23T11:31:04.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.333+0000 7f675e57a740 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.62:6832/1070252332,v1:172.21.15.62:6834/1070252332] conn(0x560c4e24a860 0x560c4e2143c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f675e57a740 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c4e213ed0 0x560c4e212010 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f675e57a740 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x560c4e21c460 0x560c4e212550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f675e57a740 1 --2- 172.21.15.62:0/1090644350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c4e213af0 0x560c4e1f2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f673a000640 0 client.8829 ms_handle_reset on v2:172.21.15.62:6832/1070252332 2024-03-23T11:31:04.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f675e57a740 1 -- 172.21.15.62:0/1090644350 >> 172.21.15.62:0/1090644350 conn(0x560c4e1bbb40 msgr2=0x560c4e2390d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:04.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f675e57a740 1 -- 172.21.15.62:0/1090644350 shutdown_connections 2024-03-23T11:31:04.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f675e57a740 1 -- 172.21.15.62:0/1090644350 wait complete. 2024-03-23T11:31:04.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f675e57a740 20 client.8829 trim_cache size 0 max 16384 2024-03-23T11:31:04.327 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:31:04.327 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-03-23T11:31:04.327 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-03-23T11:31:04.327 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:31:04.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f675a400640 1 -- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f673406a7a0 msgr2=0x7f673406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f675a400640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f673406a7a0 0x7f673406cc60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6740036000 tx=0x7f674003d040 comp rx=0 tx=0).stop 2024-03-23T11:31:04.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f675a400640 1 -- 172.21.15.62:0/1094074428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67540b46b0 msgr2=0x7f67540ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.334+0000 7f675a400640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67540b46b0 0x7f67540ca140 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f674800e8a0 tx=0x7f674800ed60 comp rx=0 tx=0).stop 2024-03-23T11:31:04.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.335+0000 7f675a400640 1 -- 172.21.15.62:0/1094074428 shutdown_connections 2024-03-23T11:31:04.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.335+0000 7f675a400640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f673406a7a0 0x7f673406cc60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.335+0000 7f675a400640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f675415cba0 0x7f6754150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.335+0000 7f675a400640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67540ce8e0 0x7f675409d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.335+0000 7f675a400640 1 --2- 172.21.15.62:0/1094074428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67540b46b0 0x7f67540ca140 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.335+0000 7f675a400640 1 -- 172.21.15.62:0/1094074428 >> 172.21.15.62:0/1094074428 conn(0x7f6754089f60 msgr2=0x7f67540b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:04.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.335+0000 7f675a400640 1 -- 172.21.15.62:0/1094074428 shutdown_connections 2024-03-23T11:31:04.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.335+0000 7f675a400640 1 -- 172.21.15.62:0/1094074428 wait complete. 2024-03-23T11:31:04.343 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-23T11:31:04.444 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:31:04.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.680+0000 7f73d7200640 1 Processor -- start 2024-03-23T11:31:04.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.681+0000 7f73d7200640 1 -- start start 2024-03-23T11:31:04.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.682+0000 7f73d7200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f73d80d13b0 0x7f73d80cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.682+0000 7f73d7200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d80d0810 con 0x7f73d80cd260 2024-03-23T11:31:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.682+0000 7f73d7200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d80cd9d0 con 0x7f73d815c960 2024-03-23T11:31:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.682+0000 7f73d7200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d80cdcb0 con 0x7f73d80d13b0 2024-03-23T11:31:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.683+0000 7f73d5400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f73d80d13b0 0x7f73d80cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.683+0000 7f73d5400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f73d80d13b0 0x7f73d80cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41410/0 (socket says 172.21.15.62:41410) 2024-03-23T11:31:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.683+0000 7f73d6800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f73d80cd260 0x7f73d80b3310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47588/0 (socket says 172.21.15.62:47588) 2024-03-23T11:31:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.683+0000 7f73d5400640 1 -- 172.21.15.62:0/1895950225 learned_addr learned my addr 172.21.15.62:0/1895950225 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.683+0000 7f73d5400640 1 -- 172.21.15.62:0/1895950225 >> v1:172.21.15.120:6789/0 conn(0x7f73d815c960 legacy=0x7f73d80d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.683+0000 7f73d5400640 1 -- 172.21.15.62:0/1895950225 >> v1:172.21.15.62:6789/0 conn(0x7f73d80cd260 legacy=0x7f73d80b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.683+0000 7f73d5400640 1 -- 172.21.15.62:0/1895950225 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d80cdf90 con 0x7f73d80d13b0 2024-03-23T11:31:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.684+0000 7f73d5400640 1 --2- 172.21.15.62:0/1895950225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d80d13b0 0x7f73d80cf790 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009860 tx=0x7f73c402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25114fa312c21770 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.684+0000 7f73d4a00640 1 -- 172.21.15.62:0/1895950225 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c402f840 con 0x7f73d80d13b0 2024-03-23T11:31:04.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.685+0000 7f73d4a00640 1 -- 172.21.15.62:0/1895950225 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c402f9a0 con 0x7f73d80d13b0 2024-03-23T11:31:04.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.685+0000 7f73d4a00640 1 -- 172.21.15.62:0/1895950225 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c402fd30 con 0x7f73d80d13b0 2024-03-23T11:31:04.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.685+0000 7f73d7200640 1 -- 172.21.15.62:0/1895950225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d80d13b0 msgr2=0x7f73d80cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.685+0000 7f73d7200640 1 --2- 172.21.15.62:0/1895950225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d80d13b0 0x7f73d80cf790 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009860 tx=0x7f73c402ba20 comp rx=0 tx=0).stop 2024-03-23T11:31:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.685+0000 7f73d7200640 1 -- 172.21.15.62:0/1895950225 shutdown_connections 2024-03-23T11:31:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.685+0000 7f73d7200640 1 --2- 172.21.15.62:0/1895950225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d80d13b0 0x7f73d80cf790 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.685+0000 7f73d7200640 1 -- 172.21.15.62:0/1895950225 >> 172.21.15.62:0/1895950225 conn(0x7f73d8089f60 msgr2=0x7f73d80d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.685+0000 7f73d7200640 1 -- 172.21.15.62:0/1895950225 shutdown_connections 2024-03-23T11:31:04.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.685+0000 7f73d7200640 1 -- 172.21.15.62:0/1895950225 wait complete. 2024-03-23T11:31:04.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.687+0000 7f73d7200640 1 Processor -- start 2024-03-23T11:31:04.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.687+0000 7f73d7200640 1 -- start start 2024-03-23T11:31:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.688+0000 7f73d7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73d80cd260 0x7f73d8158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.688+0000 7f73d7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73d80d13b0 0x7f73d80cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.688+0000 7f73d7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d815c960 0x7f73d80cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.688+0000 7f73d7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d809f980 con 0x7f73d815c960 2024-03-23T11:31:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.688+0000 7f73d7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d809ed00 con 0x7f73d80d13b0 2024-03-23T11:31:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.688+0000 7f73d7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d809efe0 con 0x7f73d80cd260 2024-03-23T11:31:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.688+0000 7f73d6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d815c960 0x7f73d80cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.688+0000 7f73d6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d815c960 0x7f73d80cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41412/0 (socket says 172.21.15.62:41412) 2024-03-23T11:31:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.688+0000 7f73d6800640 1 -- 172.21.15.62:0/676913048 learned_addr learned my addr 172.21.15.62:0/676913048 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.689+0000 7f73d5400640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73d80d13b0 0x7f73d80cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.689+0000 7f73d5e00640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73d80cd260 0x7f73d8158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.689+0000 7f73d6800640 1 -- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73d80cd260 msgr2=0x7f73d8158f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.689+0000 7f73d6800640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73d80cd260 0x7f73d8158f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.689+0000 7f73d6800640 1 -- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73d80d13b0 msgr2=0x7f73d80cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.689+0000 7f73d6800640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73d80d13b0 0x7f73d80cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.689+0000 7f73d6800640 1 -- 172.21.15.62:0/676913048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c4009470 con 0x7f73d815c960 2024-03-23T11:31:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.689+0000 7f73d5400640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73d80d13b0 0x7f73d80cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.689+0000 7f73d5e00640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73d80cd260 0x7f73d8158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.689+0000 7f73d6800640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d815c960 0x7f73d80cb2c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f73d000c730 tx=0x7f73d000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:04.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.690+0000 7f73ce000640 1 -- 172.21.15.62:0/676913048 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d000ce40 con 0x7f73d815c960 2024-03-23T11:31:04.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.690+0000 7f73ce000640 1 -- 172.21.15.62:0/676913048 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d000a640 con 0x7f73d815c960 2024-03-23T11:31:04.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.690+0000 7f73ce000640 1 -- 172.21.15.62:0/676913048 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d000a9f0 con 0x7f73d815c960 2024-03-23T11:31:04.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.690+0000 7f73d7200640 1 -- 172.21.15.62:0/676913048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d809f2c0 con 0x7f73d815c960 2024-03-23T11:31:04.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.690+0000 7f73d7200640 1 -- 172.21.15.62:0/676913048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d80cc4a0 con 0x7f73d815c960 2024-03-23T11:31:04.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.691+0000 7f73d7200640 1 -- 172.21.15.62:0/676913048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d8066bb0 con 0x7f73d815c960 2024-03-23T11:31:04.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.692+0000 7f73ce000640 1 -- 172.21.15.62:0/676913048 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f73d000ab50 con 0x7f73d815c960 2024-03-23T11:31:04.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.693+0000 7f73ce000640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f73b806a7a0 0x7f73b806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:04.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.693+0000 7f73ce000640 1 -- 172.21.15.62:0/676913048 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f73d00882e0 con 0x7f73d815c960 2024-03-23T11:31:04.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.693+0000 7f73d5e00640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f73b806a7a0 0x7f73b806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:04.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.695+0000 7f73d5e00640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f73b806a7a0 0x7f73b806cc60 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f73c8003eb0 tx=0x7f73c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:04.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.698+0000 7f73ce000640 1 -- 172.21.15.62:0/676913048 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f73d0053740 con 0x7f73d815c960 2024-03-23T11:31:04.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.933+0000 7f73d7200640 1 -- 172.21.15.62:0/676913048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f73d80d0810 con 0x7f73d815c960 2024-03-23T11:31:04.924 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:31:04.934+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:31:04.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.950+0000 7f73ce000640 1 -- 172.21.15.62:0/676913048 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v85) v1 ==== 161+0+0 (secure 0 0 0) 0x7f73d80d0810 con 0x7f73d815c960 2024-03-23T11:31:04.941 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:31:04.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.953+0000 7f73d7200640 1 -- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f73b806a7a0 msgr2=0x7f73b806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.953+0000 7f73d7200640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f73b806a7a0 0x7f73b806cc60 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f73c8003eb0 tx=0x7f73c8009040 comp rx=0 tx=0).stop 2024-03-23T11:31:04.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.953+0000 7f73d7200640 1 -- 172.21.15.62:0/676913048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d815c960 msgr2=0x7f73d80cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:04.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.953+0000 7f73d7200640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d815c960 0x7f73d80cb2c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f73d000c730 tx=0x7f73d000cbf0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.953+0000 7f73d7200640 1 -- 172.21.15.62:0/676913048 shutdown_connections 2024-03-23T11:31:04.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.953+0000 7f73d7200640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f73b806a7a0 0x7f73b806cc60 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.953+0000 7f73d7200640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d815c960 0x7f73d80cb2c0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.953+0000 7f73d7200640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73d80d13b0 0x7f73d80cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.953+0000 7f73d7200640 1 --2- 172.21.15.62:0/676913048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73d80cd260 0x7f73d8158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:04.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.953+0000 7f73d7200640 1 -- 172.21.15.62:0/676913048 >> 172.21.15.62:0/676913048 conn(0x7f73d8089f60 msgr2=0x7f73d80b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:04.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.954+0000 7f73d7200640 1 -- 172.21.15.62:0/676913048 shutdown_connections 2024-03-23T11:31:04.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:04.954+0000 7f73d7200640 1 -- 172.21.15.62:0/676913048 wait complete. 2024-03-23T11:31:04.950 INFO:tasks.ceph.mds.a.smithi062.stderr: -1155> 2024-03-23T11:30:33.506+0000 7fb13fe00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:31:04.951 INFO:tasks.ceph.mds.a.smithi062.stderr: -1005> 2024-03-23T11:30:33.772+0000 7fb13b800640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-03-23T11:31:04.951 INFO:tasks.ceph.mds.a.smithi062.stderr: -1002> 2024-03-23T11:30:33.772+0000 7fb13b800640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-03-23T11:31:04.951 INFO:tasks.ceph.mds.a.smithi062.stderr: -986> 2024-03-23T11:30:33.772+0000 7fb13b800640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-03-23T11:31:04.951 INFO:tasks.ceph.mds.a.smithi062.stderr: -983> 2024-03-23T11:30:33.772+0000 7fb13b800640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-03-23T11:31:04.963 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 0 show inode 2024-03-23T11:31:05.004 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:31:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.057+0000 7f36a503a280 1 -- start start 2024-03-23T11:31:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.058+0000 7f36a503a280 1 Processor -- start 2024-03-23T11:31:05.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.058+0000 7f36a503a280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649df618c00 0x5649df6a3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:05.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.058+0000 7f36a503a280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649df619800 0x5649df6a3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.058+0000 7f36a503a280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649df619c00 0x5649df6a3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.058+0000 7f36a503a280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649df6ea640 con 0x5649df618c00 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.058+0000 7f36a503a280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649df6ea500 con 0x5649df619c00 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.058+0000 7f36a503a280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649df6ea3c0 con 0x5649df619800 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.059+0000 7f36a2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649df618c00 0x5649df6a3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.059+0000 7f36a2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649df618c00 0x5649df6a3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41436/0 (socket says 172.21.15.62:41436) 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.059+0000 7f36a2a00640 1 -- 172.21.15.62:0/2257116008 learned_addr learned my addr 172.21.15.62:0/2257116008 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.059+0000 7f36a3400640 1 --2- 172.21.15.62:0/2257116008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649df619c00 0x5649df6a3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.059+0000 7f36a2a00640 1 -- 172.21.15.62:0/2257116008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649df619800 msgr2=0x5649df6a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.059+0000 7f36a2a00640 1 --2- 172.21.15.62:0/2257116008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649df619800 0x5649df6a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.059+0000 7f36a2a00640 1 -- 172.21.15.62:0/2257116008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649df619c00 msgr2=0x5649df6a3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.059+0000 7f36a2a00640 1 --2- 172.21.15.62:0/2257116008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649df619c00 0x5649df6a3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.059+0000 7f36a2a00640 1 -- 172.21.15.62:0/2257116008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649df63e9c0 con 0x5649df618c00 2024-03-23T11:31:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.059+0000 7f36a3400640 1 --2- 172.21.15.62:0/2257116008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649df619c00 0x5649df6a3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.060+0000 7f36a2a00640 1 --2- 172.21.15.62:0/2257116008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649df618c00 0x5649df6a3b80 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x5649e0380c00 tx=0x5649e0436de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da50e6df818a8b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.060+0000 7f369f800640 1 -- 172.21.15.62:0/2257116008 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5649df661600 con 0x5649df618c00 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.060+0000 7f369f800640 1 -- 172.21.15.62:0/2257116008 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5649e04e6000 con 0x5649df618c00 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.060+0000 7f369f800640 1 -- 172.21.15.62:0/2257116008 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5649df6611e0 con 0x5649df618c00 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.060+0000 7f36a503a280 1 -- 172.21.15.62:0/2257116008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649df63eb60 con 0x5649df618c00 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.060+0000 7f36a503a280 1 -- 172.21.15.62:0/2257116008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5649df63e820 con 0x5649df618c00 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.060+0000 7f36a503a280 4 waiting for MDS map... 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.060+0000 7f369f800640 1 -- 172.21.15.62:0/2257116008 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x5649df644380 con 0x5649df618c00 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.060+0000 7f369f800640 1 -- 172.21.15.62:0/2257116008 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 85) v1 ==== 2573+0+0 (secure 0 0 0) 0x5649df66af00 con 0x5649df618c00 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.060+0000 7f36a503a280 4 Got MDS map 85 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.060+0000 7f36a503a280 10 main: main 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.061+0000 7f36a503a280 4 main: connecting to RADOS... 2024-03-23T11:31:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.061+0000 7f36a503a280 1 Processor -- start 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.062+0000 7f36a503a280 1 -- start start 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.062+0000 7f36a503a280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649e04e1000 0x5649df6a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.062+0000 7f36a503a280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e04e1400 0x5649df6a2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.062+0000 7f36a503a280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 0x5649df6a4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.062+0000 7f36a503a280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649df6ea280 con 0x5649e04e1800 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.062+0000 7f36a503a280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649df6ea500 con 0x5649e04e1400 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.062+0000 7f36a503a280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649df6ea140 con 0x5649e04e1000 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.063+0000 7f36a2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 0x5649df6a4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.063+0000 7f36a2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 0x5649df6a4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41438/0 (socket says 172.21.15.62:41438) 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.063+0000 7f36a2000640 1 -- 172.21.15.62:0/4129128436 learned_addr learned my addr 172.21.15.62:0/4129128436 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.063+0000 7f36a3400640 1 --2- 172.21.15.62:0/4129128436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649e04e1000 0x5649df6a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.063+0000 7f36a2a00640 1 --2- 172.21.15.62:0/4129128436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e04e1400 0x5649df6a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.063+0000 7f36a2000640 1 -- 172.21.15.62:0/4129128436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649e04e1000 msgr2=0x5649df6a2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.063+0000 7f36a2000640 1 --2- 172.21.15.62:0/4129128436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649e04e1000 0x5649df6a2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.063+0000 7f36a2000640 1 -- 172.21.15.62:0/4129128436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e04e1400 msgr2=0x5649df6a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.063+0000 7f36a2000640 1 --2- 172.21.15.62:0/4129128436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e04e1400 0x5649df6a2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.063+0000 7f36a2000640 1 -- 172.21.15.62:0/4129128436 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649df63e4e0 con 0x5649e04e1800 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.063+0000 7f36a2000640 1 --2- 172.21.15.62:0/4129128436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 0x5649df6a4100 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x5649e03802d0 tx=0x5649e04360c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581163a35b04d8b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.064+0000 7f369d000640 1 -- 172.21.15.62:0/4129128436 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5649df661080 con 0x5649e04e1800 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.064+0000 7f369d000640 1 -- 172.21.15.62:0/4129128436 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5649e04e6480 con 0x5649e04e1800 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.064+0000 7f369d000640 1 -- 172.21.15.62:0/4129128436 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5649df661600 con 0x5649e04e1800 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.064+0000 7f36a503a280 1 -- 172.21.15.62:0/4129128436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 msgr2=0x5649df6a4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.064+0000 7f36a503a280 1 --2- 172.21.15.62:0/4129128436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 0x5649df6a4100 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x5649e03802d0 tx=0x5649e04360c0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.064+0000 7f36a503a280 1 -- 172.21.15.62:0/4129128436 shutdown_connections 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.064+0000 7f36a503a280 1 --2- 172.21.15.62:0/4129128436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 0x5649df6a4100 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.064+0000 7f36a503a280 1 --2- 172.21.15.62:0/4129128436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e04e1400 0x5649df6a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.064+0000 7f36a503a280 1 --2- 172.21.15.62:0/4129128436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649e04e1000 0x5649df6a2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.064+0000 7f36a503a280 1 -- 172.21.15.62:0/4129128436 >> 172.21.15.62:0/4129128436 conn(0x5649e04e0c00 msgr2=0x5649df6ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.064+0000 7f36a503a280 1 -- 172.21.15.62:0/4129128436 shutdown_connections 2024-03-23T11:31:05.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.065+0000 7f36a503a280 1 -- 172.21.15.62:0/4129128436 wait complete. 2024-03-23T11:31:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.065+0000 7f36a503a280 1 Processor -- start 2024-03-23T11:31:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.065+0000 7f36a503a280 1 -- start start 2024-03-23T11:31:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.066+0000 7f36a503a280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e04e1000 0x5649df6a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.066+0000 7f36a503a280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649e04e1400 0x5649df6a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.066+0000 7f36a503a280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 0x5649df6a4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.066+0000 7f36a503a280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649df6ea000 con 0x5649e04e1800 2024-03-23T11:31:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.066+0000 7f36a503a280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649df6eb040 con 0x5649e04e1000 2024-03-23T11:31:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.066+0000 7f36a503a280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649df6eb180 con 0x5649e04e1400 2024-03-23T11:31:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.066+0000 7f36a2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 0x5649df6a4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f36a2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 0x5649df6a4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41450/0 (socket says 172.21.15.62:41450) 2024-03-23T11:31:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f36a2000640 1 -- 172.21.15.62:0/528350444 learned_addr learned my addr 172.21.15.62:0/528350444 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f36a3400640 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e04e1000 0x5649df6a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f36a2a00640 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649e04e1400 0x5649df6a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f36a2000640 1 -- 172.21.15.62:0/528350444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649e04e1400 msgr2=0x5649df6a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f36a2000640 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649e04e1400 0x5649df6a2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f36a2000640 1 -- 172.21.15.62:0/528350444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e04e1000 msgr2=0x5649df6a2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f36a2000640 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649e04e1000 0x5649df6a2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f36a2000640 1 -- 172.21.15.62:0/528350444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649df63ed00 con 0x5649e04e1800 2024-03-23T11:31:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f36a2000640 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 0x5649df6a4100 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x5649e0381b00 tx=0x5649e0436000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f369b200640 1 -- 172.21.15.62:0/528350444 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5649df6611e0 con 0x5649e04e1800 2024-03-23T11:31:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.067+0000 7f369b200640 1 -- 172.21.15.62:0/528350444 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5649e04e6180 con 0x5649e04e1800 2024-03-23T11:31:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.068+0000 7f369b200640 1 -- 172.21.15.62:0/528350444 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5649df660dc0 con 0x5649e04e1800 2024-03-23T11:31:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.068+0000 7f36a503a280 1 -- 172.21.15.62:0/528350444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5649df63e4e0 con 0x5649e04e1800 2024-03-23T11:31:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.068+0000 7f36a503a280 1 -- 172.21.15.62:0/528350444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649df63e9c0 con 0x5649e04e1800 2024-03-23T11:31:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.068+0000 7f36a503a280 4 main: resolving pool 14 2024-03-23T11:31:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.069+0000 7f369b200640 1 -- 172.21.15.62:0/528350444 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x5649df6e7080 con 0x5649e04e1800 2024-03-23T11:31:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.070+0000 7f369b200640 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5649e04e1c00 0x5649df978000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.070+0000 7f369b200640 1 -- 172.21.15.62:0/528350444 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x5649df644540 con 0x5649e04e1800 2024-03-23T11:31:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.070+0000 7f36a503a280 4 main: creating IoCtx.. 2024-03-23T11:31:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.070+0000 7f36a503a280 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x5649df97c000 0x5649df6a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.070+0000 7f36a503a280 1 -- 172.21.15.62:0/528350444 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:1 14.13 14:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107) v8 -- 0x5649df97c400 con 0x5649df97c000 2024-03-23T11:31:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.070+0000 7f36a2a00640 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x5649df97c000 0x5649df6a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.071+0000 7f36a3400640 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5649e04e1c00 0x5649df978000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:05.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.071+0000 7f36a2a00640 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x5649df97c000 0x5649df6a4680 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:05.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.072+0000 7f36a3400640 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5649e04e1c00 0x5649df978000 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x5649df64be60 tx=0x5649e0505a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:05.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.074+0000 7f36a2a00640 1 -- 172.21.15.62:0/528350444 <== osd.2 v2:172.21.15.62:6800/36996 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv7 ondisk = 0) v8 ==== 157+0+34 (crc 0 0 0) 0x5649e04d4b40 con 0x5649df97c000 2024-03-23T11:31:05.065 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:31:05.065 INFO:teuthology.orchestra.run.smithi062.stdout: "0": { 2024-03-23T11:31:05.065 INFO:teuthology.orchestra.run.smithi062.stdout: "data": { 2024-03-23T11:31:05.065 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 5, 2024-03-23T11:31:05.065 INFO:teuthology.orchestra.run.smithi062.stdout: "inotable": { 2024-03-23T11:31:05.065 INFO:teuthology.orchestra.run.smithi062.stdout: "projected_free": [ 2024-03-23T11:31:05.065 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:05.065 INFO:teuthology.orchestra.run.smithi062.stdout: "start": 1099511627779, 2024-03-23T11:31:05.065 INFO:teuthology.orchestra.run.smithi062.stdout: "len": 1099511627773 2024-03-23T11:31:05.065 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:05.065 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-03-23T11:31:05.066 INFO:teuthology.orchestra.run.smithi062.stdout: "free": [ 2024-03-23T11:31:05.066 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:05.066 INFO:teuthology.orchestra.run.smithi062.stdout: "start": 1099511627779, 2024-03-23T11:31:05.066 INFO:teuthology.orchestra.run.smithi062.stdout: "len": 1099511627773 2024-03-23T11:31:05.066 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:05.066 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-03-23T11:31:05.066 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:05.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:05.067 INFO:teuthology.orchestra.run.smithi062.stdout: "result": 0 2024-03-23T11:31:05.067 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:05.067 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:31:05.067 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.074+0000 7f36a503a280 1 -- 172.21.15.62:0/528350444 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x5649df97c000 msgr2=0x5649df6a4680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.074+0000 7f36a503a280 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x5649df97c000 0x5649df6a4680 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.074+0000 7f36a503a280 1 -- 172.21.15.62:0/528350444 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5649e04e1c00 msgr2=0x5649df978000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.074+0000 7f36a503a280 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x5649e04e1c00 0x5649df978000 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x5649df64be60 tx=0x5649e0505a40 comp rx=0 tx=0).stop 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.074+0000 7f36a503a280 1 -- 172.21.15.62:0/528350444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 msgr2=0x5649df6a4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.074+0000 7f36a503a280 1 --2- 172.21.15.62:0/528350444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649e04e1800 0x5649df6a4100 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x5649e0381b00 tx=0x5649e0436000 comp rx=0 tx=0).stop 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.074+0000 7f36a2000640 1 -- 172.21.15.62:0/528350444 reap_dead start 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.074+0000 7f36a503a280 1 -- 172.21.15.62:0/528350444 shutdown_connections 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 -- 172.21.15.62:0/528350444 >> 172.21.15.62:0/528350444 conn(0x5649e04e0c00 msgr2=0x5649df6ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 -- 172.21.15.62:0/528350444 shutdown_connections 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 -- 172.21.15.62:0/528350444 wait complete. 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 -- 172.21.15.62:0/2257116008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649df618c00 msgr2=0x5649df6a3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 --2- 172.21.15.62:0/2257116008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649df618c00 0x5649df6a3b80 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x5649e0380c00 tx=0x5649e0436de0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 -- 172.21.15.62:0/2257116008 shutdown_connections 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 --2- 172.21.15.62:0/2257116008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5649df619c00 0x5649df6a3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 --2- 172.21.15.62:0/2257116008 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5649df619800 0x5649df6a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 --2- 172.21.15.62:0/2257116008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649df618c00 0x5649df6a3b80 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 -- 172.21.15.62:0/2257116008 >> 172.21.15.62:0/2257116008 conn(0x5649df619000 msgr2=0x5649df6fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:05.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 -- 172.21.15.62:0/2257116008 shutdown_connections 2024-03-23T11:31:05.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:05.075+0000 7f36a503a280 1 -- 172.21.15.62:0/2257116008 wait complete. 2024-03-23T11:31:05.071 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '0', 'show', 'inode'] in time 0:00:00.107248, result: { "0": { "data": { "version": 5, "inotable": { "projected_free": [ { "start": 1099511627779, "len": 1099511627773 } ], "free": [ { "start": 1099511627779, "len": 1099511627773 } ] } }, "result": 0 } } 2024-03-23T11:31:05.071 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:31:05.071 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:31:05.231 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:31:05.357 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:31:05.357 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:31:05.357 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:05.418 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:31:05.418 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:31:05.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:05.418 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:31:05.474 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-23T11:31:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:31:05.475 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:31:05.475 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:05.531 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:31:05.532 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:31:05.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:05.532 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T11:31:05.585 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-23T11:31:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:31:05.586 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:31:05.586 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:05.644 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:31:05.644 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:31:05.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:05.644 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T11:31:05.696 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-23T11:31:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:31:05.698 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:31:05.698 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:05.753 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:31:05.753 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:31:05.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:05.753 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T11:31:05.803 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-23T11:31:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:31:05.804 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:31:06.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.046+0000 7ff632a00640 1 Processor -- start 2024-03-23T11:31:06.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.046+0000 7ff632a00640 1 -- start start 2024-03-23T11:31:06.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.048+0000 7ff632a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff62c159760 0x7ff62c0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:06.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.048+0000 7ff632a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff62c0b3ea0 0x7ff62c0b4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:06.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.048+0000 7ff632a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62c0cd9a0 con 0x7ff62c071440 2024-03-23T11:31:06.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.048+0000 7ff632a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62c0b4850 con 0x7ff62c0b3ea0 2024-03-23T11:31:06.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.048+0000 7ff632a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62c06f210 con 0x7ff62c159760 2024-03-23T11:31:06.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.049+0000 7ff630a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff62c0b3ea0 0x7ff62c0b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:06.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.049+0000 7ff630a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff62c0b3ea0 0x7ff62c0b4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41460/0 (socket says 172.21.15.62:41460) 2024-03-23T11:31:06.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.049+0000 7ff630a00640 1 -- 172.21.15.62:0/2312390571 learned_addr learned my addr 172.21.15.62:0/2312390571 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:06.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.049+0000 7ff62be00640 1 --2- 172.21.15.62:0/2312390571 >> v2:172.21.15.120:3300/0 conn(0x7ff62c159760 0x7ff62c0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:06.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.049+0000 7ff630a00640 1 -- 172.21.15.62:0/2312390571 >> v2:172.21.15.120:3300/0 conn(0x7ff62c159760 msgr2=0x7ff62c0b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:06.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.049+0000 7ff630a00640 1 --2- 172.21.15.62:0/2312390571 >> v2:172.21.15.120:3300/0 conn(0x7ff62c159760 0x7ff62c0b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.049+0000 7ff630a00640 1 -- 172.21.15.62:0/2312390571 >> v1:172.21.15.120:6789/0 conn(0x7ff62c071440 legacy=0x7ff62c06eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:06.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.049+0000 7ff630a00640 1 -- 172.21.15.62:0/2312390571 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c06f470 con 0x7ff62c0b3ea0 2024-03-23T11:31:06.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.050+0000 7ff630a00640 1 --2- 172.21.15.62:0/2312390571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c0b3ea0 0x7ff62c0b4310 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7ff62c06ccd0 tx=0x7ff62402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e5fd59512112f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:06.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.051+0000 7ff62aa00640 1 -- 172.21.15.62:0/2312390571 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff624031870 con 0x7ff62c0b3ea0 2024-03-23T11:31:06.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.051+0000 7ff62aa00640 1 -- 172.21.15.62:0/2312390571 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6240319d0 con 0x7ff62c0b3ea0 2024-03-23T11:31:06.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.051+0000 7ff62aa00640 1 -- 172.21.15.62:0/2312390571 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff624031d60 con 0x7ff62c0b3ea0 2024-03-23T11:31:06.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.052+0000 7ff632a00640 1 -- 172.21.15.62:0/2312390571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c0b3ea0 msgr2=0x7ff62c0b4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:06.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.052+0000 7ff632a00640 1 --2- 172.21.15.62:0/2312390571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c0b3ea0 0x7ff62c0b4310 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7ff62c06ccd0 tx=0x7ff62402b910 comp rx=0 tx=0).stop 2024-03-23T11:31:06.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.052+0000 7ff632a00640 1 -- 172.21.15.62:0/2312390571 shutdown_connections 2024-03-23T11:31:06.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.052+0000 7ff632a00640 1 --2- 172.21.15.62:0/2312390571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c0b3ea0 0x7ff62c0b4310 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.052+0000 7ff632a00640 1 --2- 172.21.15.62:0/2312390571 >> v2:172.21.15.120:3300/0 conn(0x7ff62c159760 0x7ff62c0b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.052+0000 7ff632a00640 1 -- 172.21.15.62:0/2312390571 >> 172.21.15.62:0/2312390571 conn(0x7ff62c0b1370 msgr2=0x7ff62c0b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:06.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.052+0000 7ff632a00640 1 -- 172.21.15.62:0/2312390571 shutdown_connections 2024-03-23T11:31:06.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.052+0000 7ff632a00640 1 -- 172.21.15.62:0/2312390571 wait complete. 2024-03-23T11:31:06.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.054+0000 7ff632a00640 1 Processor -- start 2024-03-23T11:31:06.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.054+0000 7ff632a00640 1 -- start start 2024-03-23T11:31:06.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.055+0000 7ff632a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff62c071440 0x7ff62c14edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:06.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.055+0000 7ff632a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff62c0b3ea0 0x7ff62c14ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:06.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff632a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c159760 0x7ff62c14b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:06.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff632a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c15a9f0 con 0x7ff62c159760 2024-03-23T11:31:06.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff632a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c15acd0 con 0x7ff62c071440 2024-03-23T11:31:06.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff632a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c15afb0 con 0x7ff62c0b3ea0 2024-03-23T11:31:06.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff630a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c159760 0x7ff62c14b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:06.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff630a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c159760 0x7ff62c14b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41466/0 (socket says 172.21.15.62:41466) 2024-03-23T11:31:06.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff630a00640 1 -- 172.21.15.62:0/374512326 learned_addr learned my addr 172.21.15.62:0/374512326 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:06.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff62be00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff62c071440 0x7ff62c14edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:06.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff62b400640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff62c0b3ea0 0x7ff62c14ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:06.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff630a00640 1 -- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff62c0b3ea0 msgr2=0x7ff62c14ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:06.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff630a00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff62c0b3ea0 0x7ff62c14ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff630a00640 1 -- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff62c071440 msgr2=0x7ff62c14edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:06.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff630a00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff62c071440 0x7ff62c14edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.056+0000 7ff630a00640 1 -- 172.21.15.62:0/374512326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6240094d0 con 0x7ff62c159760 2024-03-23T11:31:06.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.057+0000 7ff630a00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c159760 0x7ff62c14b2c0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7ff6240090b0 tx=0x7ff6240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:06.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.057+0000 7ff623e00640 1 -- 172.21.15.62:0/374512326 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62403ac60 con 0x7ff62c159760 2024-03-23T11:31:06.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.057+0000 7ff623e00640 1 -- 172.21.15.62:0/374512326 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62403adc0 con 0x7ff62c159760 2024-03-23T11:31:06.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.057+0000 7ff623e00640 1 -- 172.21.15.62:0/374512326 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff624044630 con 0x7ff62c159760 2024-03-23T11:31:06.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.057+0000 7ff632a00640 1 -- 172.21.15.62:0/374512326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62c15b290 con 0x7ff62c159760 2024-03-23T11:31:06.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.057+0000 7ff632a00640 1 -- 172.21.15.62:0/374512326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62c14f2f0 con 0x7ff62c159760 2024-03-23T11:31:06.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.059+0000 7ff632a00640 1 -- 172.21.15.62:0/374512326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62c14f4e0 con 0x7ff62c159760 2024-03-23T11:31:06.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.059+0000 7ff623e00640 1 -- 172.21.15.62:0/374512326 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff62404d450 con 0x7ff62c159760 2024-03-23T11:31:06.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.060+0000 7ff623e00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff60806a7a0 0x7ff60806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:06.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.060+0000 7ff623e00640 1 -- 172.21.15.62:0/374512326 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ff62402d040 con 0x7ff62c159760 2024-03-23T11:31:06.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.061+0000 7ff62be00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff60806a7a0 0x7ff60806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.062+0000 7ff62be00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff60806a7a0 0x7ff60806cc60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff61c006fd0 tx=0x7ff61c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:06.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.067+0000 7ff623e00640 1 -- 172.21.15.62:0/374512326 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff624076820 con 0x7ff62c159760 2024-03-23T11:31:06.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.303+0000 7ff632a00640 1 -- 172.21.15.62:0/374512326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff62c15b7b0 con 0x7ff62c159760 2024-03-23T11:31:06.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.304+0000 7ff623e00640 1 -- 172.21.15.62:0/374512326 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4736 (secure 0 0 0) 0x7ff62c15b7b0 con 0x7ff62c159760 2024-03-23T11:31:06.295 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:06.295 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":86,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/4217933052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":4217933052},{"type":"v1","addr":"172.21.15.62:6834","nonce":4217933052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":85,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:31:04.935197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:31:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 86 2024-03-23T11:31:06.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 -- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff60806a7a0 msgr2=0x7ff60806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:06.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff60806a7a0 0x7ff60806cc60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff61c006fd0 tx=0x7ff61c008040 comp rx=0 tx=0).stop 2024-03-23T11:31:06.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 -- 172.21.15.62:0/374512326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c159760 msgr2=0x7ff62c14b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:06.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c159760 0x7ff62c14b2c0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7ff6240090b0 tx=0x7ff6240046c0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 -- 172.21.15.62:0/374512326 shutdown_connections 2024-03-23T11:31:06.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff60806a7a0 0x7ff60806cc60 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff62c159760 0x7ff62c14b2c0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff62c0b3ea0 0x7ff62c14ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 --2- 172.21.15.62:0/374512326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff62c071440 0x7ff62c14edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 -- 172.21.15.62:0/374512326 >> 172.21.15.62:0/374512326 conn(0x7ff62c0b1370 msgr2=0x7ff62c158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:06.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 -- 172.21.15.62:0/374512326 shutdown_connections 2024-03-23T11:31:06.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.307+0000 7ff632a00640 1 -- 172.21.15.62:0/374512326 wait complete. 2024-03-23T11:31:06.319 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:31:06.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.549+0000 7f9e30800640 1 Processor -- start 2024-03-23T11:31:06.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.549+0000 7f9e30800640 1 -- start start 2024-03-23T11:31:06.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.551+0000 7f9e30800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9e2806c300 0x7f9e280ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:06.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.551+0000 7f9e30800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280cbc60 con 0x7f9e2806c6e0 2024-03-23T11:31:06.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.551+0000 7f9e30800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280cbf40 con 0x7f9e280cafa0 2024-03-23T11:31:06.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.551+0000 7f9e30800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e280cc220 con 0x7f9e2806c300 2024-03-23T11:31:06.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.551+0000 7f9e2de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9e2806c300 0x7f9e280ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:06.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.551+0000 7f9e2e800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9e2806c6e0 0x7f9e2806cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47604/0 (socket says 172.21.15.62:47604) 2024-03-23T11:31:06.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.552+0000 7f9e2e800640 1 -- 172.21.15.62:0/2395937480 learned_addr learned my addr 172.21.15.62:0/2395937480 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:06.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.552+0000 7f9e2de00640 1 -- 172.21.15.62:0/2395937480 >> v1:172.21.15.120:6789/0 conn(0x7f9e280cafa0 legacy=0x7f9e280cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:06.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.552+0000 7f9e2ca00640 1 -- 172.21.15.62:0/2395937480 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487706231 0 0) 0x7f9e280cbc60 con 0x7f9e2806c6e0 2024-03-23T11:31:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.552+0000 7f9e2de00640 1 -- 172.21.15.62:0/2395937480 >> v1:172.21.15.62:6789/0 conn(0x7f9e2806c6e0 legacy=0x7f9e2806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.552+0000 7f9e2de00640 1 -- 172.21.15.62:0/2395937480 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e280cc500 con 0x7f9e2806c300 2024-03-23T11:31:06.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.553+0000 7f9e2de00640 1 --2- 172.21.15.62:0/2395937480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e2806c300 0x7f9e280ccd00 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0097a0 tx=0x7f9e1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b85d3fc1029eac9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.554+0000 7f9e2ca00640 1 -- 172.21.15.62:0/2395937480 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c031830 con 0x7f9e2806c300 2024-03-23T11:31:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.554+0000 7f9e2ca00640 1 -- 172.21.15.62:0/2395937480 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c031990 con 0x7f9e2806c300 2024-03-23T11:31:06.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.554+0000 7f9e2ca00640 1 -- 172.21.15.62:0/2395937480 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c031d20 con 0x7f9e2806c300 2024-03-23T11:31:06.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.554+0000 7f9e30800640 1 -- 172.21.15.62:0/2395937480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e2806c300 msgr2=0x7f9e280ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:06.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.554+0000 7f9e30800640 1 --2- 172.21.15.62:0/2395937480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e2806c300 0x7f9e280ccd00 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0097a0 tx=0x7f9e1c02b8d0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.554+0000 7f9e30800640 1 -- 172.21.15.62:0/2395937480 shutdown_connections 2024-03-23T11:31:06.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.554+0000 7f9e30800640 1 --2- 172.21.15.62:0/2395937480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e2806c300 0x7f9e280ccd00 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.554+0000 7f9e30800640 1 -- 172.21.15.62:0/2395937480 >> 172.21.15.62:0/2395937480 conn(0x7f9e280b02e0 msgr2=0x7f9e280b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:06.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.554+0000 7f9e30800640 1 -- 172.21.15.62:0/2395937480 shutdown_connections 2024-03-23T11:31:06.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.554+0000 7f9e30800640 1 -- 172.21.15.62:0/2395937480 wait complete. 2024-03-23T11:31:06.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.556+0000 7f9e30800640 1 Processor -- start 2024-03-23T11:31:06.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.556+0000 7f9e30800640 1 -- start start 2024-03-23T11:31:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.557+0000 7f9e30800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e2806c300 0x7f9e2812dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.557+0000 7f9e30800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e2806c6e0 0x7f9e28130890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:06.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.557+0000 7f9e30800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e280cafa0 0x7f9e281f0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:06.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.557+0000 7f9e30800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2815cc50 con 0x7f9e280cafa0 2024-03-23T11:31:06.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.557+0000 7f9e30800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e281f09b0 con 0x7f9e2806c6e0 2024-03-23T11:31:06.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.557+0000 7f9e30800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e281f0c90 con 0x7f9e2806c300 2024-03-23T11:31:06.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.558+0000 7f9e2e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e280cafa0 0x7f9e281f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:06.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.558+0000 7f9e2e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e280cafa0 0x7f9e281f0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41490/0 (socket says 172.21.15.62:41490) 2024-03-23T11:31:06.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.558+0000 7f9e2e800640 1 -- 172.21.15.62:0/4248802319 learned_addr learned my addr 172.21.15.62:0/4248802319 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:06.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.558+0000 7f9e2d400640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e2806c6e0 0x7f9e28130890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:06.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.558+0000 7f9e2de00640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e2806c300 0x7f9e2812dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.558+0000 7f9e2e800640 1 -- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e2806c300 msgr2=0x7f9e2812dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.558+0000 7f9e2e800640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e2806c300 0x7f9e2812dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.558+0000 7f9e2e800640 1 -- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e2806c6e0 msgr2=0x7f9e28130890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.558+0000 7f9e2e800640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e2806c6e0 0x7f9e28130890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.558+0000 7f9e2e800640 1 -- 172.21.15.62:0/4248802319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c009400 con 0x7f9e280cafa0 2024-03-23T11:31:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.558+0000 7f9e2e800640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e280cafa0 0x7f9e281f0160 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400c7e0 tx=0x7f9e2400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.559+0000 7f9e22000640 1 -- 172.21.15.62:0/4248802319 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2400a640 con 0x7f9e280cafa0 2024-03-23T11:31:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.559+0000 7f9e22000640 1 -- 172.21.15.62:0/4248802319 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2400a7c0 con 0x7f9e280cafa0 2024-03-23T11:31:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.559+0000 7f9e22000640 1 -- 172.21.15.62:0/4248802319 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2400ab70 con 0x7f9e280cafa0 2024-03-23T11:31:06.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.559+0000 7f9e30800640 1 -- 172.21.15.62:0/4248802319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e281f0f70 con 0x7f9e280cafa0 2024-03-23T11:31:06.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.559+0000 7f9e30800640 1 -- 172.21.15.62:0/4248802319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e281f06a0 con 0x7f9e280cafa0 2024-03-23T11:31:06.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.560+0000 7f9e30800640 1 -- 172.21.15.62:0/4248802319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2815d110 con 0x7f9e280cafa0 2024-03-23T11:31:06.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.561+0000 7f9e22000640 1 -- 172.21.15.62:0/4248802319 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9e2400acd0 con 0x7f9e280cafa0 2024-03-23T11:31:06.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.562+0000 7f9e22000640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0c06a7a0 0x7f9e0c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:06.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.563+0000 7f9e22000640 1 -- 172.21.15.62:0/4248802319 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9e240883a0 con 0x7f9e280cafa0 2024-03-23T11:31:06.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.563+0000 7f9e2de00640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0c06a7a0 0x7f9e0c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:06.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.564+0000 7f9e2de00640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0c06a7a0 0x7f9e0c06cc60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c036000 tx=0x7f9e1c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:06.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.568+0000 7f9e22000640 1 -- 172.21.15.62:0/4248802319 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e2815d110 con 0x7f9e280cafa0 2024-03-23T11:31:06.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.768+0000 7f9e30800640 1 -- 172.21.15.62:0/4248802319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e2815d110 con 0x7f9e280cafa0 2024-03-23T11:31:06.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.771+0000 7f9e22000640 1 -- 172.21.15.62:0/4248802319 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16308 (secure 0 0 0) 0x7f9e2815d110 con 0x7f9e280cafa0 2024-03-23T11:31:06.761 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:06.761 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":107,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:31:04.934594+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:29:13.817605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-23T11:29:14.652617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6834/1070252332":"2024-03-24T11:31:04.934570+0000","172.21.15.62:6832/1070252332":"2024-03-24T11:31:04.934570+0000","172.21.15.62:6834/692790652":"2024-03-24T11:30:27.420514+0000","172.21.15.62:6832/692790652":"2024-03-24T11:30:27.420514+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:31:06.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.773+0000 7f9e30800640 1 -- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0c06a7a0 msgr2=0x7f9e0c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.773+0000 7f9e30800640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0c06a7a0 0x7f9e0c06cc60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c036000 tx=0x7f9e1c03c040 comp rx=0 tx=0).stop 2024-03-23T11:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.773+0000 7f9e30800640 1 -- 172.21.15.62:0/4248802319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e280cafa0 msgr2=0x7f9e281f0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.774+0000 7f9e30800640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e280cafa0 0x7f9e281f0160 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400c7e0 tx=0x7f9e2400cca0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.774+0000 7f9e30800640 1 -- 172.21.15.62:0/4248802319 shutdown_connections 2024-03-23T11:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.774+0000 7f9e30800640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e0c06a7a0 0x7f9e0c06cc60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.774+0000 7f9e30800640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e280cafa0 0x7f9e281f0160 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.774+0000 7f9e30800640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e2806c6e0 0x7f9e28130890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.774+0000 7f9e30800640 1 --2- 172.21.15.62:0/4248802319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e2806c300 0x7f9e2812dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.774+0000 7f9e30800640 1 -- 172.21.15.62:0/4248802319 >> 172.21.15.62:0/4248802319 conn(0x7f9e280b02e0 msgr2=0x7f9e2813ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.774+0000 7f9e30800640 1 -- 172.21.15.62:0/4248802319 shutdown_connections 2024-03-23T11:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:06.774+0000 7f9e30800640 1 -- 172.21.15.62:0/4248802319 wait complete. 2024-03-23T11:31:06.783 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:29:13.817605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:29:14.652617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:31:06.783 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:31:07.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.019+0000 7f8ff0400640 1 Processor -- start 2024-03-23T11:31:07.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.019+0000 7f8ff0400640 1 -- start start 2024-03-23T11:31:07.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.020+0000 7f8ff0400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8fe8160240 0x7f8fe806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.020+0000 7f8ff0400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fe80995b0 0x7f8fe80ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.020+0000 7f8ff0400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe806b2b0 con 0x7f8fe80989d0 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.020+0000 7f8ff0400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe80cf690 con 0x7f8fe80995b0 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.020+0000 7f8ff0400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe80cb730 con 0x7f8fe8160240 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.021+0000 7f8fed000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fe80995b0 0x7f8fe80ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.021+0000 7f8fed000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fe80995b0 0x7f8fe80ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41494/0 (socket says 172.21.15.62:41494) 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.021+0000 7f8fed000640 1 -- 172.21.15.62:0/2826574496 learned_addr learned my addr 172.21.15.62:0/2826574496 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.021+0000 7f8fee400640 1 --1- 172.21.15.62:0/2826574496 >> v1:172.21.15.62:6789/0 conn(0x7f8fe80989d0 0x7f8fe80cc210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:47620/0 (socket says 172.21.15.62:47620) 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.021+0000 7f8fed000640 1 -- 172.21.15.62:0/2826574496 >> v2:172.21.15.120:3300/0 conn(0x7f8fe8160240 msgr2=0x7f8fe806ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.021+0000 7f8feda00640 1 --2- 172.21.15.62:0/2826574496 >> v2:172.21.15.120:3300/0 conn(0x7f8fe8160240 0x7f8fe806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.021+0000 7f8fed000640 1 --2- 172.21.15.62:0/2826574496 >> v2:172.21.15.120:3300/0 conn(0x7f8fe8160240 0x7f8fe806ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.021+0000 7f8fed000640 1 -- 172.21.15.62:0/2826574496 >> v1:172.21.15.62:6789/0 conn(0x7f8fe80989d0 legacy=0x7f8fe80cc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.021+0000 7f8fed000640 1 -- 172.21.15.62:0/2826574496 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe80cba10 con 0x7f8fe80995b0 2024-03-23T11:31:07.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.021+0000 7f8feda00640 1 --2- 172.21.15.62:0/2826574496 >> v2:172.21.15.120:3300/0 conn(0x7f8fe8160240 0x7f8fe806ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:07.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.021+0000 7f8fe7e00640 1 -- 172.21.15.62:0/2826574496 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797683098 0 0) 0x7f8fe806b2b0 con 0x7f8fe80989d0 2024-03-23T11:31:07.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.023+0000 7f8fed000640 1 --2- 172.21.15.62:0/2826574496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe80995b0 0x7f8fe80ced80 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8071410 tx=0x7f8fd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa66c729d9e057c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:07.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.023+0000 7f8fe7e00640 1 -- 172.21.15.62:0/2826574496 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8031870 con 0x7f8fe80995b0 2024-03-23T11:31:07.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.023+0000 7f8fe7e00640 1 -- 172.21.15.62:0/2826574496 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd80319d0 con 0x7f8fe80995b0 2024-03-23T11:31:07.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.023+0000 7f8fe7e00640 1 -- 172.21.15.62:0/2826574496 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8031d60 con 0x7f8fe80995b0 2024-03-23T11:31:07.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.023+0000 7f8ff0400640 1 -- 172.21.15.62:0/2826574496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe80995b0 msgr2=0x7f8fe80ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.023+0000 7f8ff0400640 1 --2- 172.21.15.62:0/2826574496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe80995b0 0x7f8fe80ced80 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8071410 tx=0x7f8fd802b910 comp rx=0 tx=0).stop 2024-03-23T11:31:07.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.024+0000 7f8ff0400640 1 -- 172.21.15.62:0/2826574496 shutdown_connections 2024-03-23T11:31:07.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.024+0000 7f8ff0400640 1 --2- 172.21.15.62:0/2826574496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe80995b0 0x7f8fe80ced80 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.024+0000 7f8ff0400640 1 --2- 172.21.15.62:0/2826574496 >> v2:172.21.15.120:3300/0 conn(0x7f8fe8160240 0x7f8fe806ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.024+0000 7f8ff0400640 1 -- 172.21.15.62:0/2826574496 >> 172.21.15.62:0/2826574496 conn(0x7f8fe80b22c0 msgr2=0x7f8fe80b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:07.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.024+0000 7f8ff0400640 1 -- 172.21.15.62:0/2826574496 shutdown_connections 2024-03-23T11:31:07.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.024+0000 7f8ff0400640 1 -- 172.21.15.62:0/2826574496 wait complete. 2024-03-23T11:31:07.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.025+0000 7f8ff0400640 1 Processor -- start 2024-03-23T11:31:07.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.026+0000 7f8ff0400640 1 -- start start 2024-03-23T11:31:07.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.026+0000 7f8ff0400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe80989d0 0x7f8fe81f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.026+0000 7f8ff0400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe80995b0 0x7f8fe814e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8ff0400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe8160240 0x7f8fe806bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8ff0400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80ca470 con 0x7f8fe8160240 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8ff0400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe81f6560 con 0x7f8fe80989d0 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8ff0400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80c93d0 con 0x7f8fe80995b0 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8fee400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe8160240 0x7f8fe806bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8fee400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe8160240 0x7f8fe806bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41500/0 (socket says 172.21.15.62:41500) 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8fee400640 1 -- 172.21.15.62:0/3058583288 learned_addr learned my addr 172.21.15.62:0/3058583288 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8feda00640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe80989d0 0x7f8fe81f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8fed000640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe80995b0 0x7f8fe814e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8fee400640 1 -- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe80995b0 msgr2=0x7f8fe814e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8fee400640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe80995b0 0x7f8fe814e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8fee400640 1 -- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe80989d0 msgr2=0x7f8fe81f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8fee400640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe80989d0 0x7f8fe81f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.027+0000 7f8fee400640 1 -- 172.21.15.62:0/3058583288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd80094d0 con 0x7f8fe8160240 2024-03-23T11:31:07.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.028+0000 7f8fee400640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe8160240 0x7f8fe806bed0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000c7e0 tx=0x7f8fe000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:07.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.028+0000 7f8fe5600640 1 -- 172.21.15.62:0/3058583288 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe000a640 con 0x7f8fe8160240 2024-03-23T11:31:07.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.028+0000 7f8fe5600640 1 -- 172.21.15.62:0/3058583288 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe000a7c0 con 0x7f8fe8160240 2024-03-23T11:31:07.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.028+0000 7f8fe5600640 1 -- 172.21.15.62:0/3058583288 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe000ab70 con 0x7f8fe8160240 2024-03-23T11:31:07.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.028+0000 7f8ff0400640 1 -- 172.21.15.62:0/3058583288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe80c96b0 con 0x7f8fe8160240 2024-03-23T11:31:07.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.028+0000 7f8ff0400640 1 -- 172.21.15.62:0/3058583288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe80c9c70 con 0x7f8fe8160240 2024-03-23T11:31:07.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.030+0000 7f8ff0400640 1 -- 172.21.15.62:0/3058583288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe80738e0 con 0x7f8fe8160240 2024-03-23T11:31:07.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.031+0000 7f8fe5600640 1 -- 172.21.15.62:0/3058583288 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8fe000acd0 con 0x7f8fe8160240 2024-03-23T11:31:07.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.031+0000 7f8fe5600640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fcc06a7a0 0x7f8fcc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.031+0000 7f8fe5600640 1 -- 172.21.15.62:0/3058583288 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f8fe00884d0 con 0x7f8fe8160240 2024-03-23T11:31:07.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.032+0000 7f8feda00640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fcc06a7a0 0x7f8fcc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.033+0000 7f8feda00640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fcc06a7a0 0x7f8fcc06cc60 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00a950 tx=0x7f8fdc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:07.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.036+0000 7f8fe5600640 1 -- 172.21.15.62:0/3058583288 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8fe00537a0 con 0x7f8fe8160240 2024-03-23T11:31:07.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.237+0000 7f8ff0400640 1 -- 172.21.15.62:0/3058583288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8fe8066c90 con 0x7f8fe8160240 2024-03-23T11:31:07.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.238+0000 7f8fe5600640 1 -- 172.21.15.62:0/3058583288 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8fe00180a0 con 0x7f8fe8160240 2024-03-23T11:31:07.228 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:31:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 -- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fcc06a7a0 msgr2=0x7f8fcc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fcc06a7a0 0x7f8fcc06cc60 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00a950 tx=0x7f8fdc008040 comp rx=0 tx=0).stop 2024-03-23T11:31:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 -- 172.21.15.62:0/3058583288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe8160240 msgr2=0x7f8fe806bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe8160240 0x7f8fe806bed0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000c7e0 tx=0x7f8fe000cca0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 -- 172.21.15.62:0/3058583288 shutdown_connections 2024-03-23T11:31:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8fcc06a7a0 0x7f8fcc06cc60 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe8160240 0x7f8fe806bed0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fe80995b0 0x7f8fe814e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 --2- 172.21.15.62:0/3058583288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fe80989d0 0x7f8fe81f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 -- 172.21.15.62:0/3058583288 >> 172.21.15.62:0/3058583288 conn(0x7f8fe80b22c0 msgr2=0x7f8fe815ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:07.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 -- 172.21.15.62:0/3058583288 shutdown_connections 2024-03-23T11:31:07.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.240+0000 7f8ff0400640 1 -- 172.21.15.62:0/3058583288 wait complete. 2024-03-23T11:31:07.248 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:31:07.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.477+0000 7fc52f200640 1 Processor -- start 2024-03-23T11:31:07.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.478+0000 7fc52f200640 1 -- start start 2024-03-23T11:31:07.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.479+0000 7fc52f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc528160420 0x7fc5281608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.479+0000 7fc52f200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc5280b5db0 0x7fc5280b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.479+0000 7fc52f200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5280ce510 con 0x7fc52815fcf0 2024-03-23T11:31:07.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.479+0000 7fc52f200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc528160df0 con 0x7fc5280b5db0 2024-03-23T11:31:07.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.479+0000 7fc52f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5280cdc70 con 0x7fc528160420 2024-03-23T11:31:07.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.479+0000 7fc52de00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc52815fcf0 0x7fc5280d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37530/0 (socket says 172.21.15.62:37530) 2024-03-23T11:31:07.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.479+0000 7fc52de00640 1 -- 172.21.15.62:0/4255781109 learned_addr learned my addr 172.21.15.62:0/4255781109 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:07.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.479+0000 7fc52e800640 1 --2- 172.21.15.62:0/4255781109 >> v2:172.21.15.62:3300/0 conn(0x7fc5280b5db0 0x7fc5280b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.480+0000 7fc52d400640 1 --2- 172.21.15.62:0/4255781109 >> v2:172.21.15.120:3300/0 conn(0x7fc528160420 0x7fc5281608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.480+0000 7fc52ca00640 1 -- 172.21.15.62:0/4255781109 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246789754 0 0) 0x7fc5280ce510 con 0x7fc52815fcf0 2024-03-23T11:31:07.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.480+0000 7fc52ca00640 1 -- 172.21.15.62:0/4255781109 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc514003520 con 0x7fc52815fcf0 2024-03-23T11:31:07.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.480+0000 7fc52e800640 1 -- 172.21.15.62:0/4255781109 >> v2:172.21.15.120:3300/0 conn(0x7fc528160420 msgr2=0x7fc5281608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:07.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.480+0000 7fc52e800640 1 --2- 172.21.15.62:0/4255781109 >> v2:172.21.15.120:3300/0 conn(0x7fc528160420 0x7fc5281608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.480+0000 7fc52e800640 1 -- 172.21.15.62:0/4255781109 >> v1:172.21.15.62:6789/0 conn(0x7fc52815fcf0 legacy=0x7fc5280d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.480+0000 7fc52ca00640 1 -- 172.21.15.62:0/4255781109 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2046539682 0 0) 0x7fc514003520 con 0x7fc52815fcf0 2024-03-23T11:31:07.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.480+0000 7fc52e800640 1 -- 172.21.15.62:0/4255781109 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5280cdf50 con 0x7fc5280b5db0 2024-03-23T11:31:07.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.480+0000 7fc52d400640 1 --2- 172.21.15.62:0/4255781109 >> v2:172.21.15.120:3300/0 conn(0x7fc528160420 0x7fc5281608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:31:07.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.482+0000 7fc52e800640 1 --2- 172.21.15.62:0/4255781109 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5280b5db0 0x7fc5280b61e0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fc528161540 tx=0x7fc52402b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61efea4e14680a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:07.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.482+0000 7fc52ca00640 1 -- 172.21.15.62:0/4255781109 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5200019f0 con 0x7fc5280b5db0 2024-03-23T11:31:07.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.482+0000 7fc52ca00640 1 -- 172.21.15.62:0/4255781109 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52402e050 con 0x7fc5280b5db0 2024-03-23T11:31:07.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.482+0000 7fc52ca00640 1 -- 172.21.15.62:0/4255781109 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc524030730 con 0x7fc5280b5db0 2024-03-23T11:31:07.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.482+0000 7fc52f200640 1 -- 172.21.15.62:0/4255781109 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5280b5db0 msgr2=0x7fc5280b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.482+0000 7fc52f200640 1 --2- 172.21.15.62:0/4255781109 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5280b5db0 0x7fc5280b61e0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fc528161540 tx=0x7fc52402b710 comp rx=0 tx=0).stop 2024-03-23T11:31:07.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.483+0000 7fc52f200640 1 -- 172.21.15.62:0/4255781109 shutdown_connections 2024-03-23T11:31:07.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.483+0000 7fc52f200640 1 --2- 172.21.15.62:0/4255781109 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5280b5db0 0x7fc5280b61e0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.483+0000 7fc52f200640 1 --2- 172.21.15.62:0/4255781109 >> v2:172.21.15.120:3300/0 conn(0x7fc528160420 0x7fc5281608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.483+0000 7fc52f200640 1 -- 172.21.15.62:0/4255781109 >> 172.21.15.62:0/4255781109 conn(0x7fc528089f60 msgr2=0x7fc5280d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:07.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.483+0000 7fc52f200640 1 -- 172.21.15.62:0/4255781109 shutdown_connections 2024-03-23T11:31:07.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.483+0000 7fc52f200640 1 -- 172.21.15.62:0/4255781109 wait complete. 2024-03-23T11:31:07.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.485+0000 7fc52f200640 1 Processor -- start 2024-03-23T11:31:07.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.485+0000 7fc52f200640 1 -- start start 2024-03-23T11:31:07.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.486+0000 7fc52f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5280b5db0 0x7fc5281f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.486+0000 7fc52f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52815fcf0 0x7fc528158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.486+0000 7fc52f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc528160420 0x7fc5280c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.486+0000 7fc52f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc528154b20 con 0x7fc5280b5db0 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.486+0000 7fc52f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5281550e0 con 0x7fc528160420 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.486+0000 7fc52f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280c7a90 con 0x7fc52815fcf0 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.486+0000 7fc52de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5280b5db0 0x7fc5281f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.486+0000 7fc52de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5280b5db0 0x7fc5281f1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54426/0 (socket says 172.21.15.62:54426) 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.486+0000 7fc52de00640 1 -- 172.21.15.62:0/1649790507 learned_addr learned my addr 172.21.15.62:0/1649790507 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.486+0000 7fc52e800640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc528160420 0x7fc5280c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.486+0000 7fc52d400640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52815fcf0 0x7fc528158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.487+0000 7fc52de00640 1 -- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52815fcf0 msgr2=0x7fc528158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.487+0000 7fc52de00640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52815fcf0 0x7fc528158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.487+0000 7fc52de00640 1 -- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc528160420 msgr2=0x7fc5280c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.487+0000 7fc52de00640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc528160420 0x7fc5280c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.487+0000 7fc52de00640 1 -- 172.21.15.62:0/1649790507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc524009400 con 0x7fc5280b5db0 2024-03-23T11:31:07.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.487+0000 7fc52de00640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5280b5db0 0x7fc5281f1b80 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fc520001930 tx=0x7fc52000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:07.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.487+0000 7fc51e000640 1 -- 172.21.15.62:0/1649790507 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc520012400 con 0x7fc5280b5db0 2024-03-23T11:31:07.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.487+0000 7fc51e000640 1 -- 172.21.15.62:0/1649790507 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc520012560 con 0x7fc5280b5db0 2024-03-23T11:31:07.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.487+0000 7fc51e000640 1 -- 172.21.15.62:0/1649790507 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc520012950 con 0x7fc5280b5db0 2024-03-23T11:31:07.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.487+0000 7fc52f200640 1 -- 172.21.15.62:0/1649790507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5280c7d70 con 0x7fc5280b5db0 2024-03-23T11:31:07.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.488+0000 7fc52f200640 1 -- 172.21.15.62:0/1649790507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5281f20c0 con 0x7fc5280b5db0 2024-03-23T11:31:07.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.489+0000 7fc52f200640 1 -- 172.21.15.62:0/1649790507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc528066bb0 con 0x7fc5280b5db0 2024-03-23T11:31:07.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.490+0000 7fc51e000640 1 -- 172.21.15.62:0/1649790507 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc520012ab0 con 0x7fc5280b5db0 2024-03-23T11:31:07.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.490+0000 7fc51e000640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc50c06a7a0 0x7fc50c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.490+0000 7fc51e000640 1 -- 172.21.15.62:0/1649790507 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fc5200898a0 con 0x7fc5280b5db0 2024-03-23T11:31:07.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.491+0000 7fc52d400640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc50c06a7a0 0x7fc50c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.492+0000 7fc52d400640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc50c06a7a0 0x7fc50c06cc60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc528065f30 tx=0x7fc518008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:07.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.495+0000 7fc51e000640 1 -- 172.21.15.62:0/1649790507 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc520054b70 con 0x7fc5280b5db0 2024-03-23T11:31:07.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.697+0000 7fc52f200640 1 -- 172.21.15.62:0/1649790507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc5280cea40 con 0x7fc5280b5db0 2024-03-23T11:31:07.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.698+0000 7fc51e000640 1 -- 172.21.15.62:0/1649790507 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc520058190 con 0x7fc5280b5db0 2024-03-23T11:31:07.688 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:31:07.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 -- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc50c06a7a0 msgr2=0x7fc50c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc50c06a7a0 0x7fc50c06cc60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc528065f30 tx=0x7fc518008040 comp rx=0 tx=0).stop 2024-03-23T11:31:07.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 -- 172.21.15.62:0/1649790507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5280b5db0 msgr2=0x7fc5281f1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5280b5db0 0x7fc5281f1b80 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fc520001930 tx=0x7fc52000a430 comp rx=0 tx=0).stop 2024-03-23T11:31:07.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 -- 172.21.15.62:0/1649790507 shutdown_connections 2024-03-23T11:31:07.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc50c06a7a0 0x7fc50c06cc60 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc528160420 0x7fc5280c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52815fcf0 0x7fc528158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 --2- 172.21.15.62:0/1649790507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5280b5db0 0x7fc5281f1b80 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 -- 172.21.15.62:0/1649790507 >> 172.21.15.62:0/1649790507 conn(0x7fc528089f60 msgr2=0x7fc5280b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:07.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 -- 172.21.15.62:0/1649790507 shutdown_connections 2024-03-23T11:31:07.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.701+0000 7fc52f200640 1 -- 172.21.15.62:0/1649790507 wait complete. 2024-03-23T11:31:07.712 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:31:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.934+0000 7fc5cbe00640 1 Processor -- start 2024-03-23T11:31:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.935+0000 7fc5cbe00640 1 -- start start 2024-03-23T11:31:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.936+0000 7fc5cbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5cc0cda20 0x7fc5cc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.936+0000 7fc5cbe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc160fe0 con 0x7fc5cc160c00 2024-03-23T11:31:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.936+0000 7fc5cbe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0b1510 con 0x7fc5cc0cde90 2024-03-23T11:31:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.936+0000 7fc5cbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc0b1ad0 con 0x7fc5cc0cda20 2024-03-23T11:31:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.937+0000 7fc5caa00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc5cc160c00 0x7fc5cc06de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37536/0 (socket says 172.21.15.62:37536) 2024-03-23T11:31:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.937+0000 7fc5caa00640 1 -- 172.21.15.62:0/4266234872 learned_addr learned my addr 172.21.15.62:0/4266234872 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:07.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.937+0000 7fc5c3e00640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5cc0cda20 0x7fc5cc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.937+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368132596 0 0) 0x7fc5cc160fe0 con 0x7fc5cc160c00 2024-03-23T11:31:07.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.937+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b0003520 con 0x7fc5cc160c00 2024-03-23T11:31:07.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2752689140 0 0) 0x7fc5b0003520 con 0x7fc5cc160c00 2024-03-23T11:31:07.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5cc0cda20 msgr2=0x7fc5cc06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:07.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5cc0cda20 0x7fc5cc06d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 >> v1:172.21.15.120:6789/0 conn(0x7fc5cc0cde90 legacy=0x7fc5cc0caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc0b17f0 con 0x7fc5cc160c00 2024-03-23T11:31:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fc5bc001670 con 0x7fc5cc160c00 2024-03-23T11:31:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 >> v1:172.21.15.62:6789/0 conn(0x7fc5cc160c00 legacy=0x7fc5cc06de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b00039a0 0x7fc5b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5b00047c0 0x7fc5b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5cc0cda20 0x7fc5b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc0b1ad0 con 0x7fc5b00047c0 2024-03-23T11:31:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b000ba90 con 0x7fc5b00039a0 2024-03-23T11:31:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b000bc50 con 0x7fc5cc0cda20 2024-03-23T11:31:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.938+0000 7fc5c3e00640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5b00047c0 0x7fc5b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.939+0000 7fc5cb400640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5cc0cda20 0x7fc5b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.939+0000 7fc5caa00640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b00039a0 0x7fc5b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.939+0000 7fc5c3e00640 1 -- 172.21.15.62:0/4266234872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5cc0cda20 msgr2=0x7fc5b0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:07.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.939+0000 7fc5c3e00640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5cc0cda20 0x7fc5b0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.939+0000 7fc5c3e00640 1 -- 172.21.15.62:0/4266234872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b00039a0 msgr2=0x7fc5b0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:07.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.939+0000 7fc5c3e00640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b00039a0 0x7fc5b0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.939+0000 7fc5c3e00640 1 -- 172.21.15.62:0/4266234872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0003520 con 0x7fc5b00047c0 2024-03-23T11:31:07.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.940+0000 7fc5c3e00640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5b00047c0 0x7fc5b0004c40 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009a90 tx=0x7fc5b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e82d5e90549da5d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:07.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.940+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b402c680 con 0x7fc5b00047c0 2024-03-23T11:31:07.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.941+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b402c7e0 con 0x7fc5b00047c0 2024-03-23T11:31:07.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.941+0000 7fc5ca000640 1 -- 172.21.15.62:0/4266234872 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b402cb70 con 0x7fc5b00047c0 2024-03-23T11:31:07.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.941+0000 7fc5cbe00640 1 -- 172.21.15.62:0/4266234872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5b00047c0 msgr2=0x7fc5b0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.941+0000 7fc5cbe00640 1 --2- 172.21.15.62:0/4266234872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5b00047c0 0x7fc5b0004c40 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009a90 tx=0x7fc5b00057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.941+0000 7fc5cb400640 1 -- 172.21.15.62:0/4266234872 reap_dead start 2024-03-23T11:31:07.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.941+0000 7fc5cbe00640 1 -- 172.21.15.62:0/4266234872 shutdown_connections 2024-03-23T11:31:07.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.941+0000 7fc5cbe00640 1 -- 172.21.15.62:0/4266234872 >> 172.21.15.62:0/4266234872 conn(0x7fc5cc066050 msgr2=0x7fc5cc069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:07.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.942+0000 7fc5cbe00640 1 -- 172.21.15.62:0/4266234872 shutdown_connections 2024-03-23T11:31:07.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.942+0000 7fc5cbe00640 1 -- 172.21.15.62:0/4266234872 wait complete. 2024-03-23T11:31:07.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.944+0000 7fc5cbe00640 1 Processor -- start 2024-03-23T11:31:07.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.944+0000 7fc5cbe00640 1 -- start start 2024-03-23T11:31:07.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.945+0000 7fc5cbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b00047c0 0x7fc5cc1eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.945+0000 7fc5cbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5cc15cce0 0x7fc5cc0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.945+0000 7fc5cbe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5cc15c160 0x7fc5cc15c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.945+0000 7fc5cbe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc15cb50 con 0x7fc5cc15cce0 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.945+0000 7fc5cbe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc15d530 con 0x7fc5cc15c160 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.945+0000 7fc5cbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc1eb0c0 con 0x7fc5b00047c0 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.946+0000 7fc5c3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5cc15cce0 0x7fc5cc0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.946+0000 7fc5c3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5cc15cce0 0x7fc5cc0b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54444/0 (socket says 172.21.15.62:54444) 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.946+0000 7fc5c3e00640 1 -- 172.21.15.62:0/1467088903 learned_addr learned my addr 172.21.15.62:0/1467088903 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.946+0000 7fc5caa00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b00047c0 0x7fc5cc1eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.946+0000 7fc5cb400640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5cc15c160 0x7fc5cc15c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.946+0000 7fc5c3e00640 1 -- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b00047c0 msgr2=0x7fc5cc1eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.946+0000 7fc5c3e00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b00047c0 0x7fc5cc1eaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.946+0000 7fc5c3e00640 1 -- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5cc15c160 msgr2=0x7fc5cc15c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:07.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.946+0000 7fc5c3e00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5cc15c160 0x7fc5cc15c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:07.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.946+0000 7fc5c3e00640 1 -- 172.21.15.62:0/1467088903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc1eb3a0 con 0x7fc5cc15cce0 2024-03-23T11:31:07.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.947+0000 7fc5c3e00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5cc15cce0 0x7fc5cc0b0440 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fc5b402c220 tx=0x7fc5b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:07.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.947+0000 7fc5c2a00640 1 -- 172.21.15.62:0/1467088903 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4003810 con 0x7fc5cc15cce0 2024-03-23T11:31:07.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.947+0000 7fc5c2a00640 1 -- 172.21.15.62:0/1467088903 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b4003970 con 0x7fc5cc15cce0 2024-03-23T11:31:07.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.947+0000 7fc5c2a00640 1 -- 172.21.15.62:0/1467088903 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4003d00 con 0x7fc5cc15cce0 2024-03-23T11:31:07.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.947+0000 7fc5cbe00640 1 -- 172.21.15.62:0/1467088903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc1eb680 con 0x7fc5cc15cce0 2024-03-23T11:31:07.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.947+0000 7fc5cbe00640 1 -- 172.21.15.62:0/1467088903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc1ebc40 con 0x7fc5cc15cce0 2024-03-23T11:31:07.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.948+0000 7fc5cbe00640 1 -- 172.21.15.62:0/1467088903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc1ebdd0 con 0x7fc5cc15cce0 2024-03-23T11:31:07.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.949+0000 7fc5c2a00640 1 -- 172.21.15.62:0/1467088903 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc5b4034440 con 0x7fc5cc15cce0 2024-03-23T11:31:07.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.950+0000 7fc5c2a00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc5a806a7a0 0x7fc5a806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:07.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.950+0000 7fc5c2a00640 1 -- 172.21.15.62:0/1467088903 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fc5b40aaf20 con 0x7fc5cc15cce0 2024-03-23T11:31:07.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.951+0000 7fc5caa00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc5a806a7a0 0x7fc5a806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:07.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.952+0000 7fc5caa00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc5a806a7a0 0x7fc5a806cc60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc001050 tx=0x7fc5bc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:07.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:07.955+0000 7fc5c2a00640 1 -- 172.21.15.62:0/1467088903 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc5b40763c0 con 0x7fc5cc15cce0 2024-03-23T11:31:08.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.155+0000 7fc5cbe00640 1 -- 172.21.15.62:0/1467088903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc5cc066bb0 con 0x7fc5cc15cce0 2024-03-23T11:31:08.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.156+0000 7fc5c2a00640 1 -- 172.21.15.62:0/1467088903 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc5b4003e60 con 0x7fc5cc15cce0 2024-03-23T11:31:08.146 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:31:08.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 -- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc5a806a7a0 msgr2=0x7fc5a806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc5a806a7a0 0x7fc5a806cc60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc001050 tx=0x7fc5bc009040 comp rx=0 tx=0).stop 2024-03-23T11:31:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 -- 172.21.15.62:0/1467088903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5cc15cce0 msgr2=0x7fc5cc0b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5cc15cce0 0x7fc5cc0b0440 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fc5b402c220 tx=0x7fc5b4004730 comp rx=0 tx=0).stop 2024-03-23T11:31:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 -- 172.21.15.62:0/1467088903 shutdown_connections 2024-03-23T11:31:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc5a806a7a0 0x7fc5a806cc60 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5cc15cce0 0x7fc5cc0b0440 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5cc15c160 0x7fc5cc15c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 --2- 172.21.15.62:0/1467088903 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5b00047c0 0x7fc5cc1eaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 -- 172.21.15.62:0/1467088903 >> 172.21.15.62:0/1467088903 conn(0x7fc5cc066050 msgr2=0x7fc5cc070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 -- 172.21.15.62:0/1467088903 shutdown_connections 2024-03-23T11:31:08.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.158+0000 7fc5cbe00640 1 -- 172.21.15.62:0/1467088903 wait complete. 2024-03-23T11:31:08.165 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:31:08.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.382+0000 7fb95a200640 1 Processor -- start 2024-03-23T11:31:08.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.382+0000 7fb95a200640 1 -- start start 2024-03-23T11:31:08.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb95a200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb954143dd0 0x7fb95415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb95a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb954070090 0x7fb9540704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb95a200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb954070a30 con 0x7fb9540b2820 2024-03-23T11:31:08.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb95a200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9540cd140 con 0x7fb954143dd0 2024-03-23T11:31:08.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb95a200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540cd3e0 con 0x7fb954070090 2024-03-23T11:31:08.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb958e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb954143dd0 0x7fb95415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb958e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb954143dd0 0x7fb95415abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54454/0 (socket says 172.21.15.62:54454) 2024-03-23T11:31:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb958e00640 1 -- 172.21.15.62:0/2134837384 learned_addr learned my addr 172.21.15.62:0/2134837384 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb959800640 1 --1- 172.21.15.62:0/2134837384 >> v1:172.21.15.62:6789/0 conn(0x7fb9540b2820 0x7fb9540b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37540/0 (socket says 172.21.15.62:37540) 2024-03-23T11:31:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb953e00640 1 --2- 172.21.15.62:0/2134837384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb954070090 0x7fb9540704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb958e00640 1 -- 172.21.15.62:0/2134837384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb954070090 msgr2=0x7fb9540704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb958e00640 1 --2- 172.21.15.62:0/2134837384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb954070090 0x7fb9540704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.384+0000 7fb958e00640 1 -- 172.21.15.62:0/2134837384 >> v1:172.21.15.62:6789/0 conn(0x7fb9540b2820 legacy=0x7fb9540b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.385+0000 7fb953400640 1 -- 172.21.15.62:0/2134837384 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854142139 0 0) 0x7fb954070a30 con 0x7fb9540b2820 2024-03-23T11:31:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.385+0000 7fb958e00640 1 -- 172.21.15.62:0/2134837384 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9540cd6c0 con 0x7fb954143dd0 2024-03-23T11:31:08.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.385+0000 7fb953e00640 1 --2- 172.21.15.62:0/2134837384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb954070090 0x7fb9540704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:08.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.386+0000 7fb958e00640 1 --2- 172.21.15.62:0/2134837384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb954143dd0 0x7fb95415abe0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fb944009870 tx=0x7fb94402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56bf422af2648ff server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:08.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.386+0000 7fb953400640 1 -- 172.21.15.62:0/2134837384 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944030840 con 0x7fb954143dd0 2024-03-23T11:31:08.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.386+0000 7fb953400640 1 -- 172.21.15.62:0/2134837384 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9440309a0 con 0x7fb954143dd0 2024-03-23T11:31:08.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.386+0000 7fb953400640 1 -- 172.21.15.62:0/2134837384 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944030d30 con 0x7fb954143dd0 2024-03-23T11:31:08.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.387+0000 7fb95a200640 1 -- 172.21.15.62:0/2134837384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb954143dd0 msgr2=0x7fb95415abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.387+0000 7fb95a200640 1 --2- 172.21.15.62:0/2134837384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb954143dd0 0x7fb95415abe0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fb944009870 tx=0x7fb94402b980 comp rx=0 tx=0).stop 2024-03-23T11:31:08.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.387+0000 7fb95a200640 1 -- 172.21.15.62:0/2134837384 shutdown_connections 2024-03-23T11:31:08.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.387+0000 7fb95a200640 1 --2- 172.21.15.62:0/2134837384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb954070090 0x7fb9540704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.387+0000 7fb95a200640 1 --2- 172.21.15.62:0/2134837384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb954143dd0 0x7fb95415abe0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.387+0000 7fb95a200640 1 -- 172.21.15.62:0/2134837384 >> 172.21.15.62:0/2134837384 conn(0x7fb954066050 msgr2=0x7fb954069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:08.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.387+0000 7fb95a200640 1 -- 172.21.15.62:0/2134837384 shutdown_connections 2024-03-23T11:31:08.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.387+0000 7fb95a200640 1 -- 172.21.15.62:0/2134837384 wait complete. 2024-03-23T11:31:08.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.389+0000 7fb95a200640 1 Processor -- start 2024-03-23T11:31:08.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.389+0000 7fb95a200640 1 -- start start 2024-03-23T11:31:08.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb95a200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb954070090 0x7fb9541eda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb95a200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9540b2820 0x7fb9541427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb95a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb954143dd0 0x7fb954159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb95a200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9541eea10 con 0x7fb954143dd0 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb95a200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9541edf90 con 0x7fb9540b2820 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb95a200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb954142d30 con 0x7fb954070090 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb959800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb954143dd0 0x7fb954159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb959800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb954143dd0 0x7fb954159b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54464/0 (socket says 172.21.15.62:54464) 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb959800640 1 -- 172.21.15.62:0/2323759026 learned_addr learned my addr 172.21.15.62:0/2323759026 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb959800640 1 -- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb954070090 msgr2=0x7fb9541eda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb959800640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb954070090 0x7fb9541eda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb953e00640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9540b2820 0x7fb9541427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb959800640 1 -- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9540b2820 msgr2=0x7fb9541427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb959800640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9540b2820 0x7fb9541427f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.390+0000 7fb959800640 1 -- 172.21.15.62:0/2323759026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9440094d0 con 0x7fb954143dd0 2024-03-23T11:31:08.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.391+0000 7fb959800640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb954143dd0 0x7fb954159b60 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb94c001560 tx=0x7fb94c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:08.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.391+0000 7fb950c00640 1 -- 172.21.15.62:0/2323759026 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c00a640 con 0x7fb954143dd0 2024-03-23T11:31:08.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.391+0000 7fb950c00640 1 -- 172.21.15.62:0/2323759026 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c00a7a0 con 0x7fb954143dd0 2024-03-23T11:31:08.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.391+0000 7fb95a200640 1 -- 172.21.15.62:0/2323759026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95415a0a0 con 0x7fb954143dd0 2024-03-23T11:31:08.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.391+0000 7fb950c00640 1 -- 172.21.15.62:0/2323759026 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c00ab30 con 0x7fb954143dd0 2024-03-23T11:31:08.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.391+0000 7fb95a200640 1 -- 172.21.15.62:0/2323759026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9541ee450 con 0x7fb954143dd0 2024-03-23T11:31:08.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.392+0000 7fb95a200640 1 -- 172.21.15.62:0/2323759026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb954066bb0 con 0x7fb954143dd0 2024-03-23T11:31:08.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.393+0000 7fb950c00640 1 -- 172.21.15.62:0/2323759026 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb94c00ac90 con 0x7fb954143dd0 2024-03-23T11:31:08.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.395+0000 7fb950c00640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb93406a7a0 0x7fb93406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.395+0000 7fb950c00640 1 -- 172.21.15.62:0/2323759026 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fb94c088640 con 0x7fb954143dd0 2024-03-23T11:31:08.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.395+0000 7fb958e00640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb93406a7a0 0x7fb93406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:08.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.396+0000 7fb958e00640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb93406a7a0 0x7fb93406cc60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb944036000 tx=0x7fb94403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:08.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.399+0000 7fb950c00640 1 -- 172.21.15.62:0/2323759026 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb94c0539b0 con 0x7fb954143dd0 2024-03-23T11:31:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.635+0000 7fb95a200640 1 -- 172.21.15.62:0/2323759026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb95415b440 con 0x7fb954143dd0 2024-03-23T11:31:08.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.636+0000 7fb950c00640 1 -- 172.21.15.62:0/2323759026 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4736 (secure 0 0 0) 0x7fb94c056fd0 con 0x7fb954143dd0 2024-03-23T11:31:08.626 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:08.627 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":86,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/4217933052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":4217933052},{"type":"v1","addr":"172.21.15.62:6834","nonce":4217933052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":85,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:31:04.935197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:31:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 86 2024-03-23T11:31:08.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 -- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb93406a7a0 msgr2=0x7fb93406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb93406a7a0 0x7fb93406cc60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb944036000 tx=0x7fb94403b040 comp rx=0 tx=0).stop 2024-03-23T11:31:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 -- 172.21.15.62:0/2323759026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb954143dd0 msgr2=0x7fb954159b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb954143dd0 0x7fb954159b60 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb94c001560 tx=0x7fb94c00cc70 comp rx=0 tx=0).stop 2024-03-23T11:31:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 -- 172.21.15.62:0/2323759026 shutdown_connections 2024-03-23T11:31:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb93406a7a0 0x7fb93406cc60 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb954143dd0 0x7fb954159b60 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9540b2820 0x7fb9541427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 --2- 172.21.15.62:0/2323759026 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb954070090 0x7fb9541eda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 -- 172.21.15.62:0/2323759026 >> 172.21.15.62:0/2323759026 conn(0x7fb954066050 msgr2=0x7fb9540ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 -- 172.21.15.62:0/2323759026 shutdown_connections 2024-03-23T11:31:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.639+0000 7fb95a200640 1 -- 172.21.15.62:0/2323759026 wait complete. 2024-03-23T11:31:08.647 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:31:08.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.873+0000 7f51da600640 1 Processor -- start 2024-03-23T11:31:08.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.873+0000 7f51da600640 1 -- start start 2024-03-23T11:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.874+0000 7f51da600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f51d415fcf0 0x7f51d40d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.874+0000 7f51da600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40d14f0 con 0x7f51d4160420 2024-03-23T11:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.874+0000 7f51da600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40d3540 con 0x7f51d40b6dc0 2024-03-23T11:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.874+0000 7f51da600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d40ce510 con 0x7f51d415fcf0 2024-03-23T11:31:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d3e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f51d4160420 0x7f51d40d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37542/0 (socket says 172.21.15.62:37542) 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d3e00640 1 -- 172.21.15.62:0/10277027 learned_addr learned my addr 172.21.15.62:0/10277027 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d9200640 1 --2- 172.21.15.62:0/10277027 >> v2:172.21.15.120:3300/0 conn(0x7f51d415fcf0 0x7f51d40d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525858335 0 0) 0x7f51d40d14f0 con 0x7f51d4160420 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b8003520 con 0x7f51d4160420 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 827253854 0 0) 0x7f51b8003520 con 0x7f51d4160420 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 >> v2:172.21.15.120:3300/0 conn(0x7f51d415fcf0 msgr2=0x7f51d40d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d3400640 1 --2- 172.21.15.62:0/10277027 >> v2:172.21.15.120:3300/0 conn(0x7f51d415fcf0 0x7f51d40d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 >> v1:172.21.15.120:6789/0 conn(0x7f51d40b6dc0 legacy=0x7f51d40d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40cdc70 con 0x7f51d4160420 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f51c40019f0 con 0x7f51d4160420 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.875+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 >> v1:172.21.15.62:6789/0 conn(0x7f51d4160420 legacy=0x7f51d40d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d3400640 1 --2- 172.21.15.62:0/10277027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51b8003970 0x7f51b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d3400640 1 --2- 172.21.15.62:0/10277027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51b80047c0 0x7f51b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d3400640 1 --2- 172.21.15.62:0/10277027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b8005180 0x7f51b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40ce510 con 0x7f51b8003970 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b800c120 con 0x7f51b80047c0 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b800c2e0 con 0x7f51b8005180 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d9200640 1 --2- 172.21.15.62:0/10277027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51b8003970 0x7f51b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d9200640 1 -- 172.21.15.62:0/10277027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b8005180 msgr2=0x7f51b800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d9200640 1 --2- 172.21.15.62:0/10277027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b8005180 0x7f51b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d9200640 1 -- 172.21.15.62:0/10277027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51b80047c0 msgr2=0x7f51b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d9200640 1 --2- 172.21.15.62:0/10277027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51b80047c0 0x7f51b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d9200640 1 -- 172.21.15.62:0/10277027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b8003520 con 0x7f51b8003970 2024-03-23T11:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.876+0000 7f51d9c00640 1 -- 172.21.15.62:0/10277027 reap_dead start 2024-03-23T11:31:08.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.877+0000 7f51d9200640 1 --2- 172.21.15.62:0/10277027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51b8003970 0x7f51b8003d90 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f51bc009ac0 tx=0x7f51b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63bd505ab501bcb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:08.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.877+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc02c6b0 con 0x7f51b8003970 2024-03-23T11:31:08.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.877+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51bc02c810 con 0x7f51b8003970 2024-03-23T11:31:08.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.878+0000 7f51d3400640 1 -- 172.21.15.62:0/10277027 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc02cba0 con 0x7f51b8003970 2024-03-23T11:31:08.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.878+0000 7f51da600640 1 -- 172.21.15.62:0/10277027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51b8003970 msgr2=0x7f51b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.878+0000 7f51da600640 1 --2- 172.21.15.62:0/10277027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51b8003970 0x7f51b8003d90 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f51bc009ac0 tx=0x7f51b80057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.878+0000 7f51da600640 1 -- 172.21.15.62:0/10277027 shutdown_connections 2024-03-23T11:31:08.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.878+0000 7f51da600640 1 --2- 172.21.15.62:0/10277027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51b8003970 0x7f51b8003d90 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.878+0000 7f51da600640 1 -- 172.21.15.62:0/10277027 >> 172.21.15.62:0/10277027 conn(0x7f51d4089f60 msgr2=0x7f51d40d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:08.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.878+0000 7f51da600640 1 -- 172.21.15.62:0/10277027 shutdown_connections 2024-03-23T11:31:08.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.878+0000 7f51da600640 1 -- 172.21.15.62:0/10277027 wait complete. 2024-03-23T11:31:08.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.880+0000 7f51da600640 1 Processor -- start 2024-03-23T11:31:08.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.880+0000 7f51da600640 1 -- start start 2024-03-23T11:31:08.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.881+0000 7f51da600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b8003970 0x7f51d41f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.881+0000 7f51da600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d406c310 0x7f51d40c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.881+0000 7f51da600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d406c770 0x7f51d40c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.881+0000 7f51da600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d409c400 con 0x7f51d406c770 2024-03-23T11:31:08.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.881+0000 7f51da600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d409c670 con 0x7f51d406c310 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.881+0000 7f51da600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d409c950 con 0x7f51b8003970 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.881+0000 7f51d9c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d406c770 0x7f51d40c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.881+0000 7f51d9c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d406c770 0x7f51d40c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54488/0 (socket says 172.21.15.62:54488) 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.881+0000 7f51d9c00640 1 -- 172.21.15.62:0/1330933250 learned_addr learned my addr 172.21.15.62:0/1330933250 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51d3e00640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d406c310 0x7f51d40c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51d9200640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b8003970 0x7f51d41f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51d9c00640 1 -- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b8003970 msgr2=0x7f51d41f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51d9c00640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b8003970 0x7f51d41f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51d9c00640 1 -- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d406c310 msgr2=0x7f51d40c7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51d9c00640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d406c310 0x7f51d40c7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51d9c00640 1 -- 172.21.15.62:0/1330933250 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d409cc30 con 0x7f51d406c770 2024-03-23T11:31:08.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51d9c00640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d406c770 0x7f51d40c8150 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f51d408a610 tx=0x7f51cc00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:08.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51d0c00640 1 -- 172.21.15.62:0/1330933250 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc004220 con 0x7f51d406c770 2024-03-23T11:31:08.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51d0c00640 1 -- 172.21.15.62:0/1330933250 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc0043a0 con 0x7f51d406c770 2024-03-23T11:31:08.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51d0c00640 1 -- 172.21.15.62:0/1330933250 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc007310 con 0x7f51d406c770 2024-03-23T11:31:08.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.882+0000 7f51da600640 1 -- 172.21.15.62:0/1330933250 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d409cf10 con 0x7f51d406c770 2024-03-23T11:31:08.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.883+0000 7f51da600640 1 -- 172.21.15.62:0/1330933250 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d40c8690 con 0x7f51d406c770 2024-03-23T11:31:08.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.884+0000 7f51da600640 1 -- 172.21.15.62:0/1330933250 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d4066bb0 con 0x7f51d406c770 2024-03-23T11:31:08.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.885+0000 7f51d0c00640 1 -- 172.21.15.62:0/1330933250 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f51cc007470 con 0x7f51d406c770 2024-03-23T11:31:08.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.886+0000 7f51d0c00640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b006a7d0 0x7f51b006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:08.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.886+0000 7f51d0c00640 1 -- 172.21.15.62:0/1330933250 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f51cc087f20 con 0x7f51d406c770 2024-03-23T11:31:08.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.887+0000 7f51d9200640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b006a7d0 0x7f51b006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:08.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.888+0000 7f51d9200640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b006a7d0 0x7f51b006cc90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f51bc036000 tx=0x7f51bc030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:08.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:08.890+0000 7f51d0c00640 1 -- 172.21.15.62:0/1330933250 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51cc0531f0 con 0x7f51d406c770 2024-03-23T11:31:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.084+0000 7f51da600640 1 -- 172.21.15.62:0/1330933250 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51d40b42c0 con 0x7f51d406c770 2024-03-23T11:31:09.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.087+0000 7f51d0c00640 1 -- 172.21.15.62:0/1330933250 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16308 (secure 0 0 0) 0x7f51cc056810 con 0x7f51d406c770 2024-03-23T11:31:09.077 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:09.077 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":107,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:31:04.934594+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:29:13.817605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-23T11:29:14.652617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6834/1070252332":"2024-03-24T11:31:04.934570+0000","172.21.15.62:6832/1070252332":"2024-03-24T11:31:04.934570+0000","172.21.15.62:6834/692790652":"2024-03-24T11:30:27.420514+0000","172.21.15.62:6832/692790652":"2024-03-24T11:30:27.420514+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.089+0000 7f51da600640 1 -- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b006a7d0 msgr2=0x7f51b006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.089+0000 7f51da600640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b006a7d0 0x7f51b006cc90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f51bc036000 tx=0x7f51bc030000 comp rx=0 tx=0).stop 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.089+0000 7f51da600640 1 -- 172.21.15.62:0/1330933250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d406c770 msgr2=0x7f51d40c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.089+0000 7f51da600640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d406c770 0x7f51d40c8150 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f51d408a610 tx=0x7f51cc00c820 comp rx=0 tx=0).stop 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.089+0000 7f51da600640 1 -- 172.21.15.62:0/1330933250 shutdown_connections 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.089+0000 7f51da600640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b006a7d0 0x7f51b006cc90 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.089+0000 7f51da600640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d406c770 0x7f51d40c8150 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.089+0000 7f51da600640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d406c310 0x7f51d40c7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.089+0000 7f51da600640 1 --2- 172.21.15.62:0/1330933250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51b8003970 0x7f51d41f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.090+0000 7f51da600640 1 -- 172.21.15.62:0/1330933250 >> 172.21.15.62:0/1330933250 conn(0x7f51d4089f60 msgr2=0x7f51d40cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.090+0000 7f51da600640 1 -- 172.21.15.62:0/1330933250 shutdown_connections 2024-03-23T11:31:09.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.090+0000 7f51da600640 1 -- 172.21.15.62:0/1330933250 wait complete. 2024-03-23T11:31:09.099 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T11:31:09.099 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:31:09.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.323+0000 7f55fbe00640 1 Processor -- start 2024-03-23T11:31:09.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.323+0000 7f55fbe00640 1 -- start start 2024-03-23T11:31:09.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.324+0000 7f55fbe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f55fc143dd0 0x7f55fc15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:09.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.324+0000 7f55fbe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc0cd140 con 0x7f55fc070090 2024-03-23T11:31:09.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.324+0000 7f55fbe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc0cd3e0 con 0x7f55fc0b2f30 2024-03-23T11:31:09.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.324+0000 7f55fbe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55fc0cd6c0 con 0x7f55fc143dd0 2024-03-23T11:31:09.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.325+0000 7f55faa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f55fc143dd0 0x7f55fc15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:09.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.325+0000 7f55fa000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f55fc070090 0x7f55fc0b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37544/0 (socket says 172.21.15.62:37544) 2024-03-23T11:31:09.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.325+0000 7f55fa000640 1 -- 172.21.15.62:0/2018069441 learned_addr learned my addr 172.21.15.62:0/2018069441 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:09.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.325+0000 7f55faa00640 1 -- 172.21.15.62:0/2018069441 >> v1:172.21.15.120:6789/0 conn(0x7f55fc0b2f30 legacy=0x7f55fc0ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:09.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.325+0000 7f55faa00640 1 -- 172.21.15.62:0/2018069441 >> v1:172.21.15.62:6789/0 conn(0x7f55fc070090 legacy=0x7f55fc0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:09.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.325+0000 7f55faa00640 1 -- 172.21.15.62:0/2018069441 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc0cd9a0 con 0x7f55fc143dd0 2024-03-23T11:31:09.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.325+0000 7f55f9600640 1 -- 172.21.15.62:0/2018069441 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299854588 0 0) 0x7f55fc0cd140 con 0x7f55fc070090 2024-03-23T11:31:09.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.326+0000 7f55faa00640 1 --2- 172.21.15.62:0/2018069441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55fc143dd0 0x7f55fc15abe0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0097a0 tx=0x7f55ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee08ea82de98992b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:09.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.327+0000 7f55f9600640 1 -- 172.21.15.62:0/2018069441 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec031830 con 0x7f55fc143dd0 2024-03-23T11:31:09.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.327+0000 7f55f9600640 1 -- 172.21.15.62:0/2018069441 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ec031990 con 0x7f55fc143dd0 2024-03-23T11:31:09.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.327+0000 7f55f9600640 1 -- 172.21.15.62:0/2018069441 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec031d20 con 0x7f55fc143dd0 2024-03-23T11:31:09.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.327+0000 7f55fbe00640 1 -- 172.21.15.62:0/2018069441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55fc143dd0 msgr2=0x7f55fc15abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:09.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.327+0000 7f55fbe00640 1 --2- 172.21.15.62:0/2018069441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55fc143dd0 0x7f55fc15abe0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0097a0 tx=0x7f55ec02b8d0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.327+0000 7f55fbe00640 1 -- 172.21.15.62:0/2018069441 shutdown_connections 2024-03-23T11:31:09.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.327+0000 7f55fbe00640 1 --2- 172.21.15.62:0/2018069441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55fc143dd0 0x7f55fc15abe0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.327+0000 7f55fbe00640 1 -- 172.21.15.62:0/2018069441 >> 172.21.15.62:0/2018069441 conn(0x7f55fc066050 msgr2=0x7f55fc069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:09.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.327+0000 7f55fbe00640 1 -- 172.21.15.62:0/2018069441 shutdown_connections 2024-03-23T11:31:09.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.327+0000 7f55fbe00640 1 -- 172.21.15.62:0/2018069441 wait complete. 2024-03-23T11:31:09.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.329+0000 7f55fbe00640 1 Processor -- start 2024-03-23T11:31:09.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.329+0000 7f55fbe00640 1 -- start start 2024-03-23T11:31:09.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.330+0000 7f55fbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc070090 0x7f55fc0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:09.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.330+0000 7f55fbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55fc0b2f30 0x7f55fc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:09.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.330+0000 7f55fbe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc143dd0 0x7f55fc1f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:09.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.330+0000 7f55fbe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc0b02f0 con 0x7f55fc0b2f30 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.330+0000 7f55fbe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc06c470 con 0x7f55fc143dd0 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.330+0000 7f55fbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc06c750 con 0x7f55fc070090 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.330+0000 7f55fa000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55fc0b2f30 0x7f55fc06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.331+0000 7f55fa000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55fc0b2f30 0x7f55fc06d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54502/0 (socket says 172.21.15.62:54502) 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.331+0000 7f55fa000640 1 -- 172.21.15.62:0/1910226100 learned_addr learned my addr 172.21.15.62:0/1910226100 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.331+0000 7f55fb400640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc143dd0 0x7f55fc1f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.331+0000 7f55faa00640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc070090 0x7f55fc0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.331+0000 7f55fa000640 1 -- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc070090 msgr2=0x7f55fc0afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.331+0000 7f55fa000640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc070090 0x7f55fc0afdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.331+0000 7f55fa000640 1 -- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc143dd0 msgr2=0x7f55fc1f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.331+0000 7f55fa000640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc143dd0 0x7f55fc1f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.331+0000 7f55fa000640 1 -- 172.21.15.62:0/1910226100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec009400 con 0x7f55fc0b2f30 2024-03-23T11:31:09.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.331+0000 7f55fb400640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc143dd0 0x7f55fc1f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:09.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.331+0000 7f55fa000640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55fc0b2f30 0x7f55fc06d510 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c4d0 tx=0x7f55e800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:09.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.332+0000 7f55f2a00640 1 -- 172.21.15.62:0/1910226100 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e80074e0 con 0x7f55fc0b2f30 2024-03-23T11:31:09.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.332+0000 7f55f2a00640 1 -- 172.21.15.62:0/1910226100 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e8007660 con 0x7f55fc0b2f30 2024-03-23T11:31:09.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.332+0000 7f55fbe00640 1 -- 172.21.15.62:0/1910226100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc06ca30 con 0x7f55fc0b2f30 2024-03-23T11:31:09.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.332+0000 7f55f2a00640 1 -- 172.21.15.62:0/1910226100 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8004a20 con 0x7f55fc0b2f30 2024-03-23T11:31:09.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.332+0000 7f55fbe00640 1 -- 172.21.15.62:0/1910226100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc06cff0 con 0x7f55fc0b2f30 2024-03-23T11:31:09.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.333+0000 7f55fbe00640 1 -- 172.21.15.62:0/1910226100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc06d210 con 0x7f55fc0b2f30 2024-03-23T11:31:09.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.334+0000 7f55f2a00640 1 -- 172.21.15.62:0/1910226100 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f55e8004b80 con 0x7f55fc0b2f30 2024-03-23T11:31:09.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.336+0000 7f55f2a00640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55dc06a7a0 0x7f55dc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:09.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.336+0000 7f55f2a00640 1 -- 172.21.15.62:0/1910226100 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f55e80883f0 con 0x7f55fc0b2f30 2024-03-23T11:31:09.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.336+0000 7f55faa00640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55dc06a7a0 0x7f55dc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:09.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.337+0000 7f55faa00640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55dc06a7a0 0x7f55dc06cc60 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f55ec036000 tx=0x7f55ec03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:09.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.340+0000 7f55f2a00640 1 -- 172.21.15.62:0/1910226100 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f55e8053760 con 0x7f55fc0b2f30 2024-03-23T11:31:09.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.584+0000 7f55fbe00640 1 -- 172.21.15.62:0/1910226100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55fc066bb0 con 0x7f55fc0b2f30 2024-03-23T11:31:09.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.585+0000 7f55f2a00640 1 -- 172.21.15.62:0/1910226100 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f55e805d020 con 0x7f55fc0b2f30 2024-03-23T11:31:09.576 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:09.576 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":86,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/4217933052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":4217933052},{"type":"v1","addr":"172.21.15.62:6834","nonce":4217933052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":85,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:31:04.935197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:31:09.576 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 86 2024-03-23T11:31:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.588+0000 7f55fbe00640 1 -- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55dc06a7a0 msgr2=0x7f55dc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.588+0000 7f55fbe00640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55dc06a7a0 0x7f55dc06cc60 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f55ec036000 tx=0x7f55ec03c040 comp rx=0 tx=0).stop 2024-03-23T11:31:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.588+0000 7f55fbe00640 1 -- 172.21.15.62:0/1910226100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55fc0b2f30 msgr2=0x7f55fc06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.588+0000 7f55fbe00640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55fc0b2f30 0x7f55fc06d510 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c4d0 tx=0x7f55e800c990 comp rx=0 tx=0).stop 2024-03-23T11:31:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.588+0000 7f55fbe00640 1 -- 172.21.15.62:0/1910226100 shutdown_connections 2024-03-23T11:31:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.588+0000 7f55fbe00640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f55dc06a7a0 0x7f55dc06cc60 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.588+0000 7f55fbe00640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55fc0b2f30 0x7f55fc06d510 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.588+0000 7f55fbe00640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc143dd0 0x7f55fc1f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.588+0000 7f55fbe00640 1 --2- 172.21.15.62:0/1910226100 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc070090 0x7f55fc0afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.588+0000 7f55fbe00640 1 -- 172.21.15.62:0/1910226100 >> 172.21.15.62:0/1910226100 conn(0x7f55fc066050 msgr2=0x7f55fc069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.589+0000 7f55fbe00640 1 -- 172.21.15.62:0/1910226100 shutdown_connections 2024-03-23T11:31:09.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.589+0000 7f55fbe00640 1 -- 172.21.15.62:0/1910226100 wait complete. 2024-03-23T11:31:09.597 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:31:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.828+0000 7fa0b9200640 1 Processor -- start 2024-03-23T11:31:09.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.828+0000 7fa0b9200640 1 -- start start 2024-03-23T11:31:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.830+0000 7fa0b9200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa0b4070090 0x7fa0b4070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.830+0000 7fa0b9200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0b40b2820 0x7fa0b40b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.830+0000 7fa0b9200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b40b31b0 con 0x7fa0b4143dd0 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.830+0000 7fa0b9200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b4070a60 con 0x7fa0b40b2820 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.830+0000 7fa0b9200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b40cd140 con 0x7fa0b4070090 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.830+0000 7fa0b3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0b40b2820 0x7fa0b40b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.830+0000 7fa0b3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0b40b2820 0x7fa0b40b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54506/0 (socket says 172.21.15.62:54506) 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.830+0000 7fa0b3e00640 1 -- 172.21.15.62:0/60575024 learned_addr learned my addr 172.21.15.62:0/60575024 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.830+0000 7fa0b2a00640 1 --2- 172.21.15.62:0/60575024 >> v2:172.21.15.120:3300/0 conn(0x7fa0b4070090 0x7fa0b4070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.831+0000 7fa0b3e00640 1 -- 172.21.15.62:0/60575024 >> v2:172.21.15.120:3300/0 conn(0x7fa0b4070090 msgr2=0x7fa0b4070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.831+0000 7fa0b3e00640 1 --2- 172.21.15.62:0/60575024 >> v2:172.21.15.120:3300/0 conn(0x7fa0b4070090 0x7fa0b4070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.831+0000 7fa0b3e00640 1 -- 172.21.15.62:0/60575024 >> v1:172.21.15.120:6789/0 conn(0x7fa0b4143dd0 legacy=0x7fa0b415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.831+0000 7fa0b3e00640 1 -- 172.21.15.62:0/60575024 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b40cd3e0 con 0x7fa0b40b2820 2024-03-23T11:31:09.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.832+0000 7fa0b3e00640 1 --2- 172.21.15.62:0/60575024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b40b2820 0x7fa0b40b2c70 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4071530 tx=0x7fa0ac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc00c349fc57626 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:09.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.832+0000 7fa0b2000640 1 -- 172.21.15.62:0/60575024 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac02ca80 con 0x7fa0b40b2820 2024-03-23T11:31:09.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.832+0000 7fa0b2000640 1 -- 172.21.15.62:0/60575024 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ac02cbe0 con 0x7fa0b40b2820 2024-03-23T11:31:09.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.832+0000 7fa0b2000640 1 -- 172.21.15.62:0/60575024 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac0343f0 con 0x7fa0b40b2820 2024-03-23T11:31:09.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.833+0000 7fa0b9200640 1 -- 172.21.15.62:0/60575024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b40b2820 msgr2=0x7fa0b40b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:09.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.833+0000 7fa0b9200640 1 --2- 172.21.15.62:0/60575024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b40b2820 0x7fa0b40b2c70 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4071530 tx=0x7fa0ac02bee0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.833+0000 7fa0b9200640 1 -- 172.21.15.62:0/60575024 shutdown_connections 2024-03-23T11:31:09.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.833+0000 7fa0b9200640 1 --2- 172.21.15.62:0/60575024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b40b2820 0x7fa0b40b2c70 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.833+0000 7fa0b9200640 1 --2- 172.21.15.62:0/60575024 >> v2:172.21.15.120:3300/0 conn(0x7fa0b4070090 0x7fa0b4070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.833+0000 7fa0b9200640 1 -- 172.21.15.62:0/60575024 >> 172.21.15.62:0/60575024 conn(0x7fa0b4066050 msgr2=0x7fa0b4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:09.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.833+0000 7fa0b9200640 1 -- 172.21.15.62:0/60575024 shutdown_connections 2024-03-23T11:31:09.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.833+0000 7fa0b9200640 1 -- 172.21.15.62:0/60575024 wait complete. 2024-03-23T11:31:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.835+0000 7fa0b9200640 1 Processor -- start 2024-03-23T11:31:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.835+0000 7fa0b9200640 1 -- start start 2024-03-23T11:31:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.836+0000 7fa0b9200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0b4070090 0x7fa0b415a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.836+0000 7fa0b9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b40b2820 0x7fa0b40c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:09.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.836+0000 7fa0b9200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b4143dd0 0x7fa0b40c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.836+0000 7fa0b9200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b40cb440 con 0x7fa0b40b2820 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.836+0000 7fa0b9200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b40c34a0 con 0x7fa0b4070090 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.836+0000 7fa0b9200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b40cae80 con 0x7fa0b4143dd0 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.836+0000 7fa0b2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b40b2820 0x7fa0b40c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.836+0000 7fa0b2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b40b2820 0x7fa0b40c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54518/0 (socket says 172.21.15.62:54518) 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.836+0000 7fa0b2a00640 1 -- 172.21.15.62:0/32816192 learned_addr learned my addr 172.21.15.62:0/32816192 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.837+0000 7fa0b2a00640 1 -- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b4143dd0 msgr2=0x7fa0b40c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.837+0000 7fa0b2a00640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b4143dd0 0x7fa0b40c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.837+0000 7fa0b2a00640 1 -- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0b4070090 msgr2=0x7fa0b415a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.837+0000 7fa0b2a00640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0b4070090 0x7fa0b415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.837+0000 7fa0b2a00640 1 -- 172.21.15.62:0/32816192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac009b30 con 0x7fa0b40b2820 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.837+0000 7fa0b2a00640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b40b2820 0x7fa0b40c7cf0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000d5f0 tx=0x7fa0a000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.837+0000 7fa0ab400640 1 -- 172.21.15.62:0/32816192 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a00125f0 con 0x7fa0b40b2820 2024-03-23T11:31:09.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.837+0000 7fa0ab400640 1 -- 172.21.15.62:0/32816192 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a0012750 con 0x7fa0b40b2820 2024-03-23T11:31:09.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.837+0000 7fa0ab400640 1 -- 172.21.15.62:0/32816192 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0012b00 con 0x7fa0b40b2820 2024-03-23T11:31:09.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.837+0000 7fa0b9200640 1 -- 172.21.15.62:0/32816192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b40cb160 con 0x7fa0b40b2820 2024-03-23T11:31:09.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.837+0000 7fa0b9200640 1 -- 172.21.15.62:0/32816192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b4134ab0 con 0x7fa0b40b2820 2024-03-23T11:31:09.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.839+0000 7fa0b9200640 1 -- 172.21.15.62:0/32816192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b40c3300 con 0x7fa0b40b2820 2024-03-23T11:31:09.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.839+0000 7fa0ab400640 1 -- 172.21.15.62:0/32816192 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa0a0012c60 con 0x7fa0b40b2820 2024-03-23T11:31:09.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.840+0000 7fa0ab400640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa09406a7a0 0x7fa09406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:09.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.840+0000 7fa0ab400640 1 -- 172.21.15.62:0/32816192 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fa0a0089290 con 0x7fa0b40b2820 2024-03-23T11:31:09.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.841+0000 7fa0b3400640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa09406a7a0 0x7fa09406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:09.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.842+0000 7fa0b3400640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa09406a7a0 0x7fa09406cc60 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4003cc0 tx=0x7fa0a40079b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:09.845+0000 7fa0ab400640 1 -- 172.21.15.62:0/32816192 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa0b40c3300 con 0x7fa0b40b2820 2024-03-23T11:31:10.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.082+0000 7fa0b9200640 1 -- 172.21.15.62:0/32816192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0b40c3300 con 0x7fa0b40b2820 2024-03-23T11:31:10.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.084+0000 7fa0ab400640 1 -- 172.21.15.62:0/32816192 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4736 (secure 0 0 0) 0x7fa0b40c3300 con 0x7fa0b40b2820 2024-03-23T11:31:10.074 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:10.074 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":86,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/4217933052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":4217933052},{"type":"v1","addr":"172.21.15.62:6834","nonce":4217933052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":85,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:29:15.626274+0000","modified":"2024-03-23T11:31:04.935197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T11:31:10.074 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 86 2024-03-23T11:31:10.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 -- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa09406a7a0 msgr2=0x7fa09406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa09406a7a0 0x7fa09406cc60 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4003cc0 tx=0x7fa0a40079b0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 -- 172.21.15.62:0/32816192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b40b2820 msgr2=0x7fa0b40c7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b40b2820 0x7fa0b40c7cf0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000d5f0 tx=0x7fa0a000dab0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 -- 172.21.15.62:0/32816192 shutdown_connections 2024-03-23T11:31:10.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa09406a7a0 0x7fa09406cc60 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b40b2820 0x7fa0b40c7cf0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa0b4143dd0 0x7fa0b40c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 --2- 172.21.15.62:0/32816192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0b4070090 0x7fa0b415a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 -- 172.21.15.62:0/32816192 >> 172.21.15.62:0/32816192 conn(0x7fa0b4066050 msgr2=0x7fa0b4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:10.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 -- 172.21.15.62:0/32816192 shutdown_connections 2024-03-23T11:31:10.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.087+0000 7fa0b9200640 1 -- 172.21.15.62:0/32816192 wait complete. 2024-03-23T11:31:10.097 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:31:10.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.326+0000 7f09a6200640 1 Processor -- start 2024-03-23T11:31:10.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.326+0000 7f09a6200640 1 -- start start 2024-03-23T11:31:10.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.328+0000 7f09a6200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f09a00cafa0 0x7f09a00cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.328+0000 7f09a6200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f09a006c6e0 0x7f09a006cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.328+0000 7f09a6200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a00cbc60 con 0x7f09a006c300 2024-03-23T11:31:10.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.328+0000 7f09a6200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a00cd410 con 0x7f09a006c6e0 2024-03-23T11:31:10.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.328+0000 7f09a6200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a00cbf40 con 0x7f09a00cafa0 2024-03-23T11:31:10.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.328+0000 7f099fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f09a006c6e0 0x7f09a006cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:10.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.328+0000 7f099fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f09a006c6e0 0x7f09a006cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54520/0 (socket says 172.21.15.62:54520) 2024-03-23T11:31:10.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.328+0000 7f099fe00640 1 -- 172.21.15.62:0/2252018326 learned_addr learned my addr 172.21.15.62:0/2252018326 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:10.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.328+0000 7f099ea00640 1 --2- 172.21.15.62:0/2252018326 >> v2:172.21.15.120:3300/0 conn(0x7f09a00cafa0 0x7f09a00cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:10.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.329+0000 7f099fe00640 1 -- 172.21.15.62:0/2252018326 >> v2:172.21.15.120:3300/0 conn(0x7f09a00cafa0 msgr2=0x7f09a00cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:10.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.329+0000 7f099fe00640 1 --2- 172.21.15.62:0/2252018326 >> v2:172.21.15.120:3300/0 conn(0x7f09a00cafa0 0x7f09a00cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.329+0000 7f099fe00640 1 -- 172.21.15.62:0/2252018326 >> v1:172.21.15.120:6789/0 conn(0x7f09a006c300 legacy=0x7f09a00ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.329+0000 7f099fe00640 1 -- 172.21.15.62:0/2252018326 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a00cc220 con 0x7f09a006c6e0 2024-03-23T11:31:10.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.330+0000 7f099fe00640 1 --2- 172.21.15.62:0/2252018326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09a006c6e0 0x7f09a006cb30 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f09a006e370 tx=0x7f099802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30af647ad31eb1cb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:10.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.330+0000 7f099e000640 1 -- 172.21.15.62:0/2252018326 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099802cb10 con 0x7f09a006c6e0 2024-03-23T11:31:10.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.330+0000 7f099e000640 1 -- 172.21.15.62:0/2252018326 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099802cc70 con 0x7f09a006c6e0 2024-03-23T11:31:10.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.330+0000 7f099e000640 1 -- 172.21.15.62:0/2252018326 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09980343f0 con 0x7f09a006c6e0 2024-03-23T11:31:10.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.331+0000 7f09a6200640 1 -- 172.21.15.62:0/2252018326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09a006c6e0 msgr2=0x7f09a006cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.331+0000 7f09a6200640 1 --2- 172.21.15.62:0/2252018326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09a006c6e0 0x7f09a006cb30 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f09a006e370 tx=0x7f099802bf70 comp rx=0 tx=0).stop 2024-03-23T11:31:10.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.331+0000 7f09a6200640 1 -- 172.21.15.62:0/2252018326 shutdown_connections 2024-03-23T11:31:10.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.331+0000 7f09a6200640 1 --2- 172.21.15.62:0/2252018326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09a006c6e0 0x7f09a006cb30 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.331+0000 7f09a6200640 1 --2- 172.21.15.62:0/2252018326 >> v2:172.21.15.120:3300/0 conn(0x7f09a00cafa0 0x7f09a00cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.331+0000 7f09a6200640 1 -- 172.21.15.62:0/2252018326 >> 172.21.15.62:0/2252018326 conn(0x7f09a00b02e0 msgr2=0x7f09a00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:10.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.331+0000 7f09a6200640 1 -- 172.21.15.62:0/2252018326 shutdown_connections 2024-03-23T11:31:10.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.331+0000 7f09a6200640 1 -- 172.21.15.62:0/2252018326 wait complete. 2024-03-23T11:31:10.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.333+0000 7f09a6200640 1 Processor -- start 2024-03-23T11:31:10.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.333+0000 7f09a6200640 1 -- start start 2024-03-23T11:31:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.334+0000 7f09a6200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09a006c300 0x7f09a00a7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.334+0000 7f09a6200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a006c6e0 0x7f09a015d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.334+0000 7f09a6200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09a00cafa0 0x7f09a013b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.334+0000 7f09a6200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a01f15f0 con 0x7f09a006c300 2024-03-23T11:31:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.334+0000 7f09a6200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00a7cc0 con 0x7f09a00cafa0 2024-03-23T11:31:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.334+0000 7f09a6200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a01f17d0 con 0x7f09a006c6e0 2024-03-23T11:31:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.334+0000 7f099f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09a006c300 0x7f09a00a7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:10.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.334+0000 7f099f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09a006c300 0x7f09a00a7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54536/0 (socket says 172.21.15.62:54536) 2024-03-23T11:31:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.334+0000 7f099f400640 1 -- 172.21.15.62:0/2317271066 learned_addr learned my addr 172.21.15.62:0/2317271066 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.335+0000 7f099fe00640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09a00cafa0 0x7f09a013b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.335+0000 7f099f400640 1 -- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a006c6e0 msgr2=0x7f09a015d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.335+0000 7f099f400640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a006c6e0 0x7f09a015d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.335+0000 7f099f400640 1 -- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09a00cafa0 msgr2=0x7f09a013b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.335+0000 7f099f400640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09a00cafa0 0x7f09a013b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.335+0000 7f099f400640 1 -- 172.21.15.62:0/2317271066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0998009b30 con 0x7f09a006c300 2024-03-23T11:31:10.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.335+0000 7f099f400640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09a006c300 0x7f09a00a7780 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f099000c450 tx=0x7f099000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:10.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.335+0000 7f0997400640 1 -- 172.21.15.62:0/2317271066 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09900052d0 con 0x7f09a006c300 2024-03-23T11:31:10.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.335+0000 7f0997400640 1 -- 172.21.15.62:0/2317271066 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0990005430 con 0x7f09a006c300 2024-03-23T11:31:10.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.335+0000 7f0997400640 1 -- 172.21.15.62:0/2317271066 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0990012870 con 0x7f09a006c300 2024-03-23T11:31:10.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.335+0000 7f09a6200640 1 -- 172.21.15.62:0/2317271066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a01f1ab0 con 0x7f09a006c300 2024-03-23T11:31:10.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.336+0000 7f09a6200640 1 -- 172.21.15.62:0/2317271066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a01f2070 con 0x7f09a006c300 2024-03-23T11:31:10.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.337+0000 7f09a6200640 1 -- 172.21.15.62:0/2317271066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a01f2200 con 0x7f09a006c300 2024-03-23T11:31:10.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.338+0000 7f0997400640 1 -- 172.21.15.62:0/2317271066 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f09900129d0 con 0x7f09a006c300 2024-03-23T11:31:10.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.338+0000 7f0997400640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f098006ebb0 0x7f0980071070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.339+0000 7f0997400640 1 -- 172.21.15.62:0/2317271066 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f09900889c0 con 0x7f09a006c300 2024-03-23T11:31:10.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.339+0000 7f099ea00640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f098006ebb0 0x7f0980071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:10.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.341+0000 7f099ea00640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f098006ebb0 0x7f0980071070 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f098c006a90 tx=0x7f098c006a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:10.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.343+0000 7f0997400640 1 -- 172.21.15.62:0/2317271066 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0990053d20 con 0x7f09a006c300 2024-03-23T11:31:10.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.545+0000 7f09a6200640 1 -- 172.21.15.62:0/2317271066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09a0066c90 con 0x7f09a006c300 2024-03-23T11:31:10.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.548+0000 7f0997400640 1 -- 172.21.15.62:0/2317271066 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16308 (secure 0 0 0) 0x7f0990057340 con 0x7f09a006c300 2024-03-23T11:31:10.539 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:10.539 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":107,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:31:04.934594+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:29:13.817605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-23T11:29:14.652617+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6834/1070252332":"2024-03-24T11:31:04.934570+0000","172.21.15.62:6832/1070252332":"2024-03-24T11:31:04.934570+0000","172.21.15.62:6834/692790652":"2024-03-24T11:30:27.420514+0000","172.21.15.62:6832/692790652":"2024-03-24T11:30:27.420514+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:31:10.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 -- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f098006ebb0 msgr2=0x7f0980071070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f098006ebb0 0x7f0980071070 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f098c006a90 tx=0x7f098c006a20 comp rx=0 tx=0).stop 2024-03-23T11:31:10.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 -- 172.21.15.62:0/2317271066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09a006c300 msgr2=0x7f09a00a7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09a006c300 0x7f09a00a7780 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f099000c450 tx=0x7f099000c910 comp rx=0 tx=0).stop 2024-03-23T11:31:10.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 -- 172.21.15.62:0/2317271066 shutdown_connections 2024-03-23T11:31:10.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f098006ebb0 0x7f0980071070 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09a00cafa0 0x7f09a013b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f09a006c6e0 0x7f09a015d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 --2- 172.21.15.62:0/2317271066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09a006c300 0x7f09a00a7780 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 -- 172.21.15.62:0/2317271066 >> 172.21.15.62:0/2317271066 conn(0x7f09a00b02e0 msgr2=0x7f09a013eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:10.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 -- 172.21.15.62:0/2317271066 shutdown_connections 2024-03-23T11:31:10.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.551+0000 7f09a6200640 1 -- 172.21.15.62:0/2317271066 wait complete. 2024-03-23T11:31:10.562 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:31:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.795+0000 7ff2ad000640 1 Processor -- start 2024-03-23T11:31:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.796+0000 7ff2ad000640 1 -- start start 2024-03-23T11:31:10.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.797+0000 7ff2ad000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a80d13b0 0x7ff2a80cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.797+0000 7ff2ad000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff2a80cd260 0x7ff2a80b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.797+0000 7ff2ad000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a80d0810 con 0x7ff2a815c960 2024-03-23T11:31:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.797+0000 7ff2ad000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a80cd9d0 con 0x7ff2a80cd260 2024-03-23T11:31:10.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.797+0000 7ff2ad000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a80cdcb0 con 0x7ff2a80d13b0 2024-03-23T11:31:10.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.798+0000 7ff2a7e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff2a80cd260 0x7ff2a80b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:10.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.798+0000 7ff2a7e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff2a80cd260 0x7ff2a80b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:49424/0 (socket says 172.21.15.62:49424) 2024-03-23T11:31:10.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.798+0000 7ff2a7e00640 1 -- 172.21.15.62:0/2165475076 learned_addr learned my addr 172.21.15.62:0/2165475076 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:10.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.798+0000 7ff2a6a00640 1 --2- 172.21.15.62:0/2165475076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a80d13b0 0x7ff2a80cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:10.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.799+0000 7ff2a7e00640 1 -- 172.21.15.62:0/2165475076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a80d13b0 msgr2=0x7ff2a80cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:10.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.799+0000 7ff2a7e00640 1 --2- 172.21.15.62:0/2165475076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a80d13b0 0x7ff2a80cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.799+0000 7ff2a7e00640 1 -- 172.21.15.62:0/2165475076 >> v1:172.21.15.120:6789/0 conn(0x7ff2a815c960 legacy=0x7ff2a80d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.799+0000 7ff2a7e00640 1 -- 172.21.15.62:0/2165475076 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a80cdf90 con 0x7ff2a80cd260 2024-03-23T11:31:10.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.800+0000 7ff2a7e00640 1 --2- 172.21.15.62:0/2165475076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a80cd260 0x7ff2a80b3310 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009870 tx=0x7ff2a002ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bd49e90d89c744d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.801+0000 7ff2a6000640 1 -- 172.21.15.62:0/2165475076 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a002f830 con 0x7ff2a80cd260 2024-03-23T11:31:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.801+0000 7ff2a6000640 1 -- 172.21.15.62:0/2165475076 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a002f990 con 0x7ff2a80cd260 2024-03-23T11:31:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.801+0000 7ff2a6000640 1 -- 172.21.15.62:0/2165475076 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a002fd20 con 0x7ff2a80cd260 2024-03-23T11:31:10.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.802+0000 7ff2ad000640 1 -- 172.21.15.62:0/2165475076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a80cd260 msgr2=0x7ff2a80b3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.802+0000 7ff2ad000640 1 --2- 172.21.15.62:0/2165475076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a80cd260 0x7ff2a80b3310 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009870 tx=0x7ff2a002ba10 comp rx=0 tx=0).stop 2024-03-23T11:31:10.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.802+0000 7ff2ad000640 1 -- 172.21.15.62:0/2165475076 shutdown_connections 2024-03-23T11:31:10.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.802+0000 7ff2ad000640 1 --2- 172.21.15.62:0/2165475076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a80cd260 0x7ff2a80b3310 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.802+0000 7ff2ad000640 1 --2- 172.21.15.62:0/2165475076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a80d13b0 0x7ff2a80cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.802+0000 7ff2ad000640 1 -- 172.21.15.62:0/2165475076 >> 172.21.15.62:0/2165475076 conn(0x7ff2a8089f60 msgr2=0x7ff2a80d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:10.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.802+0000 7ff2ad000640 1 -- 172.21.15.62:0/2165475076 shutdown_connections 2024-03-23T11:31:10.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.802+0000 7ff2ad000640 1 -- 172.21.15.62:0/2165475076 wait complete. 2024-03-23T11:31:10.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.804+0000 7ff2ad000640 1 Processor -- start 2024-03-23T11:31:10.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.804+0000 7ff2ad000640 1 -- start start 2024-03-23T11:31:10.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.806+0000 7ff2ad000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a80cd260 0x7ff2a81f2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.806+0000 7ff2ad000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a80d13b0 0x7ff2a809c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.806+0000 7ff2ad000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a815c960 0x7ff2a81f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.806+0000 7ff2ad000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a809cb30 con 0x7ff2a80cd260 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.806+0000 7ff2ad000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a81f29b0 con 0x7ff2a80d13b0 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.806+0000 7ff2ad000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a81f2c90 con 0x7ff2a815c960 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.806+0000 7ff2a7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a80cd260 0x7ff2a81f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.806+0000 7ff2a7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a80cd260 0x7ff2a81f2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54550/0 (socket says 172.21.15.62:54550) 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.806+0000 7ff2a7400640 1 -- 172.21.15.62:0/4188110325 learned_addr learned my addr 172.21.15.62:0/4188110325 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.806+0000 7ff2a6a00640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a80d13b0 0x7ff2a809c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff2a7400640 1 -- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a815c960 msgr2=0x7ff2a81f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff2a7e00640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a815c960 0x7ff2a81f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff2a7400640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a815c960 0x7ff2a81f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff2a7400640 1 -- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a80d13b0 msgr2=0x7ff2a809c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff2a7400640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a80d13b0 0x7ff2a809c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff2a7400640 1 -- 172.21.15.62:0/4188110325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00094d0 con 0x7ff2a80cd260 2024-03-23T11:31:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff2a7400640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a80cd260 0x7ff2a81f2360 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c710 tx=0x7ff29800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff29f400640 1 -- 172.21.15.62:0/4188110325 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff298003b90 con 0x7ff2a80cd260 2024-03-23T11:31:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff29f400640 1 -- 172.21.15.62:0/4188110325 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff298003cf0 con 0x7ff2a80cd260 2024-03-23T11:31:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff29f400640 1 -- 172.21.15.62:0/4188110325 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29800aa90 con 0x7ff2a80cd260 2024-03-23T11:31:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff2ad000640 1 -- 172.21.15.62:0/4188110325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a81f2f70 con 0x7ff2a80cd260 2024-03-23T11:31:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.807+0000 7ff2ad000640 1 -- 172.21.15.62:0/4188110325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a81f3790 con 0x7ff2a80cd260 2024-03-23T11:31:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.809+0000 7ff2ad000640 1 -- 172.21.15.62:0/4188110325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a809e470 con 0x7ff2a80cd260 2024-03-23T11:31:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.809+0000 7ff29f400640 1 -- 172.21.15.62:0/4188110325 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff29800abf0 con 0x7ff2a80cd260 2024-03-23T11:31:10.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.811+0000 7ff29f400640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff28406a7a0 0x7ff28406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:10.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.811+0000 7ff29f400640 1 -- 172.21.15.62:0/4188110325 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ff298088b20 con 0x7ff2a80cd260 2024-03-23T11:31:10.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.811+0000 7ff2a6a00640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff28406a7a0 0x7ff28406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:10.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.813+0000 7ff2a6a00640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff28406a7a0 0x7ff28406cc60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff2a806c540 tx=0x7ff290008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:10.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:10.816+0000 7ff29f400640 1 -- 172.21.15.62:0/4188110325 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff2a809e470 con 0x7ff2a80cd260 2024-03-23T11:31:11.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:11.057+0000 7ff2ad000640 1 -- 172.21.15.62:0/4188110325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff2a80d0810 con 0x7ff2a80cd260 2024-03-23T11:31:11.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:11.997+0000 7ff29f400640 1 -- 172.21.15.62:0/4188110325 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v87) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff2a80d0810 con 0x7ff2a80cd260 2024-03-23T11:31:11.988 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:31:11.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 -- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff28406a7a0 msgr2=0x7ff28406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:11.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff28406a7a0 0x7ff28406cc60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff2a806c540 tx=0x7ff290008040 comp rx=0 tx=0).stop 2024-03-23T11:31:11.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 -- 172.21.15.62:0/4188110325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a80cd260 msgr2=0x7ff2a81f2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:11.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a80cd260 0x7ff2a81f2360 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c710 tx=0x7ff29800cbd0 comp rx=0 tx=0).stop 2024-03-23T11:31:11.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 -- 172.21.15.62:0/4188110325 shutdown_connections 2024-03-23T11:31:11.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff28406a7a0 0x7ff28406cc60 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:11.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a815c960 0x7ff2a81f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:11.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a80d13b0 0x7ff2a809c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:11.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 --2- 172.21.15.62:0/4188110325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a80cd260 0x7ff2a81f2360 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:11.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 -- 172.21.15.62:0/4188110325 >> 172.21.15.62:0/4188110325 conn(0x7ff2a8089f60 msgr2=0x7ff2a80cf4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:11.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 -- 172.21.15.62:0/4188110325 shutdown_connections 2024-03-23T11:31:11.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.001+0000 7ff2ad000640 1 -- 172.21.15.62:0/4188110325 wait complete. 2024-03-23T11:31:12.015 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T11:31:12.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.252+0000 7f873ec00640 1 Processor -- start 2024-03-23T11:31:12.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.253+0000 7f873ec00640 1 -- start start 2024-03-23T11:31:12.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.254+0000 7f873ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f873815fe40 0x7f87380cd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:12.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.254+0000 7f873ec00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f87380d0310 0x7f87380cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:12.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.254+0000 7f873ec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87380b1fd0 con 0x7f8738160220 2024-03-23T11:31:12.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.254+0000 7f873ec00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87380cdeb0 con 0x7f87380d0310 2024-03-23T11:31:12.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.254+0000 7f873ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87380ce190 con 0x7f873815fe40 2024-03-23T11:31:12.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.255+0000 7f873ce00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f87380d0310 0x7f87380cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:12.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.255+0000 7f873d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f873815fe40 0x7f87380cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:12.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.255+0000 7f873ce00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f87380d0310 0x7f87380cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:49442/0 (socket says 172.21.15.62:49442) 2024-03-23T11:31:12.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.255+0000 7f873ce00640 1 -- 172.21.15.62:0/904696221 learned_addr learned my addr 172.21.15.62:0/904696221 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:12.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.256+0000 7f873ce00640 1 -- 172.21.15.62:0/904696221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f873815fe40 msgr2=0x7f87380cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:12.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.256+0000 7f873ce00640 1 --2- 172.21.15.62:0/904696221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f873815fe40 0x7f87380cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:12.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.256+0000 7f873ce00640 1 -- 172.21.15.62:0/904696221 >> v1:172.21.15.120:6789/0 conn(0x7f8738160220 legacy=0x7f87380cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:12.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.256+0000 7f873ce00640 1 -- 172.21.15.62:0/904696221 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380ce470 con 0x7f87380d0310 2024-03-23T11:31:12.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.257+0000 7f873ce00640 1 --2- 172.21.15.62:0/904696221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87380d0310 0x7f87380cd890 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8724009850 tx=0x7f872402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f5901445b612f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:12.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.258+0000 7f8737e00640 1 -- 172.21.15.62:0/904696221 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724004230 con 0x7f87380d0310 2024-03-23T11:31:12.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.258+0000 7f8737e00640 1 -- 172.21.15.62:0/904696221 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8724004390 con 0x7f87380d0310 2024-03-23T11:31:12.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.258+0000 7f8737e00640 1 -- 172.21.15.62:0/904696221 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724003810 con 0x7f87380d0310 2024-03-23T11:31:12.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.258+0000 7f873ec00640 1 -- 172.21.15.62:0/904696221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87380d0310 msgr2=0x7f87380cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:12.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.258+0000 7f873ec00640 1 --2- 172.21.15.62:0/904696221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87380d0310 0x7f87380cd890 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8724009850 tx=0x7f872402b960 comp rx=0 tx=0).stop 2024-03-23T11:31:12.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.258+0000 7f873ec00640 1 -- 172.21.15.62:0/904696221 shutdown_connections 2024-03-23T11:31:12.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.258+0000 7f873ec00640 1 --2- 172.21.15.62:0/904696221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87380d0310 0x7f87380cd890 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:12.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.258+0000 7f873ec00640 1 --2- 172.21.15.62:0/904696221 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f873815fe40 0x7f87380cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:12.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.258+0000 7f873ec00640 1 -- 172.21.15.62:0/904696221 >> 172.21.15.62:0/904696221 conn(0x7f8738089f60 msgr2=0x7f87380d2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:12.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.259+0000 7f873ec00640 1 -- 172.21.15.62:0/904696221 shutdown_connections 2024-03-23T11:31:12.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.259+0000 7f873ec00640 1 -- 172.21.15.62:0/904696221 wait complete. 2024-03-23T11:31:12.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.260+0000 7f873ec00640 1 Processor -- start 2024-03-23T11:31:12.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.261+0000 7f873ec00640 1 -- start start 2024-03-23T11:31:12.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.262+0000 7f873ec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87380d0310 0x7f87381496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.262+0000 7f873ec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873815fe40 0x7f873815b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.262+0000 7f873ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8738160220 0x7f87381f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.262+0000 7f873ec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380b7430 con 0x7f873815fe40 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.262+0000 7f873ec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8738149bf0 con 0x7f87380d0310 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.262+0000 7f873ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87380b7120 con 0x7f8738160220 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.262+0000 7f873ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873815fe40 0x7f873815b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.262+0000 7f873ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873815fe40 0x7f873815b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54558/0 (socket says 172.21.15.62:54558) 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.262+0000 7f873ce00640 1 -- 172.21.15.62:0/3869182853 learned_addr learned my addr 172.21.15.62:0/3869182853 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.262+0000 7f873d800640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87380d0310 0x7f87381496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.262+0000 7f873e200640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8738160220 0x7f87381f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.263+0000 7f873ce00640 1 -- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8738160220 msgr2=0x7f87381f5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.263+0000 7f873ce00640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8738160220 0x7f87381f5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:12.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.263+0000 7f873ce00640 1 -- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87380d0310 msgr2=0x7f87381496b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:12.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.263+0000 7f873ce00640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87380d0310 0x7f87381496b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:12.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.263+0000 7f873ce00640 1 -- 172.21.15.62:0/3869182853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8724009460 con 0x7f873815fe40 2024-03-23T11:31:12.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.263+0000 7f873d800640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87380d0310 0x7f87381496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:31:12.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.263+0000 7f873ce00640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873815fe40 0x7f873815b7a0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f8724009970 tx=0x7f87240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:12.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.263+0000 7f8735600640 1 -- 172.21.15.62:0/3869182853 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872403d070 con 0x7f873815fe40 2024-03-23T11:31:12.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.263+0000 7f8735600640 1 -- 172.21.15.62:0/3869182853 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8724004500 con 0x7f873815fe40 2024-03-23T11:31:12.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.263+0000 7f873ec00640 1 -- 172.21.15.62:0/3869182853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873815c1d0 con 0x7f873815fe40 2024-03-23T11:31:12.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.263+0000 7f8735600640 1 -- 172.21.15.62:0/3869182853 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724003df0 con 0x7f873815fe40 2024-03-23T11:31:12.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.264+0000 7f873ec00640 1 -- 172.21.15.62:0/3869182853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873815c790 con 0x7f873815fe40 2024-03-23T11:31:12.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.265+0000 7f873ec00640 1 -- 172.21.15.62:0/3869182853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87380b75a0 con 0x7f873815fe40 2024-03-23T11:31:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.266+0000 7f8735600640 1 -- 172.21.15.62:0/3869182853 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8724006040 con 0x7f873815fe40 2024-03-23T11:31:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.266+0000 7f8735600640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f871806a7a0 0x7f871806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:12.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.267+0000 7f8735600640 1 -- 172.21.15.62:0/3869182853 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f87240aac20 con 0x7f873815fe40 2024-03-23T11:31:12.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.267+0000 7f873d800640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f871806a7a0 0x7f871806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:12.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.268+0000 7f873d800640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f871806a7a0 0x7f871806cc60 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f872c007460 tx=0x7f872c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:12.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.273+0000 7f8735600640 1 -- 172.21.15.62:0/3869182853 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8724075ef0 con 0x7f873815fe40 2024-03-23T11:31:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:12.510+0000 7f873ec00640 1 -- 172.21.15.62:0/3869182853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f873815f980 con 0x7f873815fe40 2024-03-23T11:31:12.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.009+0000 7f8735600640 1 -- 172.21.15.62:0/3869182853 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v88) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8724079510 con 0x7f873815fe40 2024-03-23T11:31:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.012+0000 7f873ec00640 1 -- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f871806a7a0 msgr2=0x7f871806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.012+0000 7f873ec00640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f871806a7a0 0x7f871806cc60 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f872c007460 tx=0x7f872c00b040 comp rx=0 tx=0).stop 2024-03-23T11:31:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.013+0000 7f873ec00640 1 -- 172.21.15.62:0/3869182853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873815fe40 msgr2=0x7f873815b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.013+0000 7f873ec00640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873815fe40 0x7f873815b7a0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f8724009970 tx=0x7f87240046c0 comp rx=0 tx=0).stop 2024-03-23T11:31:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.013+0000 7f873ec00640 1 -- 172.21.15.62:0/3869182853 shutdown_connections 2024-03-23T11:31:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.013+0000 7f873ec00640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f871806a7a0 0x7f871806cc60 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.013+0000 7f873ec00640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873815fe40 0x7f873815b7a0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:13.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.013+0000 7f873ec00640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8738160220 0x7f87381f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:13.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.013+0000 7f873ec00640 1 --2- 172.21.15.62:0/3869182853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87380d0310 0x7f87381496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:13.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.013+0000 7f873ec00640 1 -- 172.21.15.62:0/3869182853 >> 172.21.15.62:0/3869182853 conn(0x7f8738089f60 msgr2=0x7f87380cefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:13.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.013+0000 7f873ec00640 1 -- 172.21.15.62:0/3869182853 shutdown_connections 2024-03-23T11:31:13.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.013+0000 7f873ec00640 1 -- 172.21.15.62:0/3869182853 wait complete. 2024-03-23T11:31:13.025 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T11:31:13.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.265+0000 7f2513200640 1 Processor -- start 2024-03-23T11:31:13.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.265+0000 7f2513200640 1 -- start start 2024-03-23T11:31:13.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.266+0000 7f2513200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f250c15ca70 0x7f250c1606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:13.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.267+0000 7f2513200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f250c0cf9d0 0x7f250c0d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:13.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.267+0000 7f2513200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c0b58b0 con 0x7f250c160be0 2024-03-23T11:31:13.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.267+0000 7f2513200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250c0d0910 con 0x7f250c0cf9d0 2024-03-23T11:31:13.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.267+0000 7f2513200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250c161160 con 0x7f250c15ca70 2024-03-23T11:31:13.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.267+0000 7f250b400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f250c160be0 0x7f250c0cf2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37548/0 (socket says 172.21.15.62:37548) 2024-03-23T11:31:13.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.267+0000 7f250b400640 1 -- 172.21.15.62:0/3219919873 learned_addr learned my addr 172.21.15.62:0/3219919873 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:13.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.267+0000 7f2511200640 1 --2- 172.21.15.62:0/3219919873 >> v2:172.21.15.62:3300/0 conn(0x7f250c0cf9d0 0x7f250c0d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:13.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.267+0000 7f250be00640 1 --2- 172.21.15.62:0/3219919873 >> v2:172.21.15.120:3300/0 conn(0x7f250c15ca70 0x7f250c1606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:13.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.268+0000 7f250aa00640 1 -- 172.21.15.62:0/3219919873 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864670435 0 0) 0x7f250c0b58b0 con 0x7f250c160be0 2024-03-23T11:31:13.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.268+0000 7f250aa00640 1 -- 172.21.15.62:0/3219919873 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f0003520 con 0x7f250c160be0 2024-03-23T11:31:13.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.268+0000 7f2511200640 1 -- 172.21.15.62:0/3219919873 >> v2:172.21.15.120:3300/0 conn(0x7f250c15ca70 msgr2=0x7f250c1606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:13.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.268+0000 7f2511200640 1 --2- 172.21.15.62:0/3219919873 >> v2:172.21.15.120:3300/0 conn(0x7f250c15ca70 0x7f250c1606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:13.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.268+0000 7f2511200640 1 -- 172.21.15.62:0/3219919873 >> v1:172.21.15.62:6789/0 conn(0x7f250c160be0 legacy=0x7f250c0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:13.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.268+0000 7f2511200640 1 -- 172.21.15.62:0/3219919873 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c161440 con 0x7f250c0cf9d0 2024-03-23T11:31:13.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.269+0000 7f2511200640 1 --2- 172.21.15.62:0/3219919873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f250c0cf9d0 0x7f250c0d1320 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f250c08af80 tx=0x7f250402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54775169f92823e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:13.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.269+0000 7f250aa00640 1 -- 172.21.15.62:0/3219919873 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504030610 con 0x7f250c0cf9d0 2024-03-23T11:31:13.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.269+0000 7f250aa00640 1 -- 172.21.15.62:0/3219919873 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2504030770 con 0x7f250c0cf9d0 2024-03-23T11:31:13.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.269+0000 7f250aa00640 1 -- 172.21.15.62:0/3219919873 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504030b00 con 0x7f250c0cf9d0 2024-03-23T11:31:13.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.270+0000 7f2513200640 1 -- 172.21.15.62:0/3219919873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f250c0cf9d0 msgr2=0x7f250c0d1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:13.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.270+0000 7f2513200640 1 --2- 172.21.15.62:0/3219919873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f250c0cf9d0 0x7f250c0d1320 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f250c08af80 tx=0x7f250402b960 comp rx=0 tx=0).stop 2024-03-23T11:31:13.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.270+0000 7f2513200640 1 -- 172.21.15.62:0/3219919873 shutdown_connections 2024-03-23T11:31:13.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.270+0000 7f2513200640 1 --2- 172.21.15.62:0/3219919873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f250c0cf9d0 0x7f250c0d1320 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:13.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.270+0000 7f2513200640 1 --2- 172.21.15.62:0/3219919873 >> v2:172.21.15.120:3300/0 conn(0x7f250c15ca70 0x7f250c1606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:13.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.270+0000 7f2513200640 1 -- 172.21.15.62:0/3219919873 >> 172.21.15.62:0/3219919873 conn(0x7f250c071390 msgr2=0x7f250c071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:13.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.270+0000 7f2513200640 1 -- 172.21.15.62:0/3219919873 shutdown_connections 2024-03-23T11:31:13.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.270+0000 7f2513200640 1 -- 172.21.15.62:0/3219919873 wait complete. 2024-03-23T11:31:13.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.272+0000 7f2513200640 1 Processor -- start 2024-03-23T11:31:13.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.272+0000 7f2513200640 1 -- start start 2024-03-23T11:31:13.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.273+0000 7f2513200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f250c0cf9d0 0x7f250c0b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.273+0000 7f2513200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f250c15ca70 0x7f250c0b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.273+0000 7f2513200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250c160be0 0x7f250c1f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.273+0000 7f2513200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c0b4840 con 0x7f250c15ca70 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.273+0000 7f2513200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c1f20f0 con 0x7f250c0cf9d0 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.273+0000 7f2513200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250c1f6750 con 0x7f250c160be0 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.273+0000 7f250b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f250c15ca70 0x7f250c0b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.273+0000 7f250b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f250c15ca70 0x7f250c0b4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54570/0 (socket says 172.21.15.62:54570) 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.273+0000 7f250b400640 1 -- 172.21.15.62:0/1125930507 learned_addr learned my addr 172.21.15.62:0/1125930507 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f2511200640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250c160be0 0x7f250c1f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f250be00640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f250c0cf9d0 0x7f250c0b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f250b400640 1 -- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250c160be0 msgr2=0x7f250c1f6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:13.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f250b400640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250c160be0 0x7f250c1f6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:13.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f250b400640 1 -- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f250c0cf9d0 msgr2=0x7f250c0b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:13.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f250b400640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f250c0cf9d0 0x7f250c0b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:13.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f250b400640 1 -- 172.21.15.62:0/1125930507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2504009400 con 0x7f250c15ca70 2024-03-23T11:31:13.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f2511200640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250c160be0 0x7f250c1f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:13.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f250be00640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f250c0cf9d0 0x7f250c0b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:13.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f250b400640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f250c15ca70 0x7f250c0b4cd0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f24fc00cbc0 tx=0x7f24fc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:13.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f2503e00640 1 -- 172.21.15.62:0/1125930507 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc00ad40 con 0x7f250c15ca70 2024-03-23T11:31:13.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f2503e00640 1 -- 172.21.15.62:0/1125930507 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc012400 con 0x7f250c15ca70 2024-03-23T11:31:13.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.274+0000 7f2503e00640 1 -- 172.21.15.62:0/1125930507 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc00aea0 con 0x7f250c15ca70 2024-03-23T11:31:13.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.275+0000 7f2513200640 1 -- 172.21.15.62:0/1125930507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250c0b3f50 con 0x7f250c15ca70 2024-03-23T11:31:13.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.275+0000 7f2513200640 1 -- 172.21.15.62:0/1125930507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250c0b44d0 con 0x7f250c15ca70 2024-03-23T11:31:13.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.276+0000 7f2513200640 1 -- 172.21.15.62:0/1125930507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250c0b49f0 con 0x7f250c15ca70 2024-03-23T11:31:13.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.277+0000 7f2503e00640 1 -- 172.21.15.62:0/1125930507 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f24fc007070 con 0x7f250c15ca70 2024-03-23T11:31:13.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.278+0000 7f2503e00640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f24e806a7d0 0x7f24e806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:13.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.278+0000 7f2503e00640 1 -- 172.21.15.62:0/1125930507 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f24fc088ea0 con 0x7f250c15ca70 2024-03-23T11:31:13.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.279+0000 7f250be00640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f24e806a7d0 0x7f24e806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:13.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.280+0000 7f250be00640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f24e806a7d0 0x7f24e806cc90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f24f4009ac0 tx=0x7f24f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:13.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.283+0000 7f2503e00640 1 -- 172.21.15.62:0/1125930507 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f250c0b49f0 con 0x7f250c15ca70 2024-03-23T11:31:13.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:13.487+0000 7f2513200640 1 -- 172.21.15.62:0/1125930507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f250c0ced00 con 0x7f250c15ca70 2024-03-23T11:31:14.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.022+0000 7f2503e00640 1 -- 172.21.15.62:0/1125930507 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v108) v1 ==== 181+0+0 (secure 0 0 0) 0x7f24fc0170d0 con 0x7f250c15ca70 2024-03-23T11:31:14.012 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' removed 2024-03-23T11:31:14.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.027+0000 7f2513200640 1 -- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f24e806a7d0 msgr2=0x7f24e806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:14.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.027+0000 7f2513200640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f24e806a7d0 0x7f24e806cc90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f24f4009ac0 tx=0x7f24f4008040 comp rx=0 tx=0).stop 2024-03-23T11:31:14.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.027+0000 7f2513200640 1 -- 172.21.15.62:0/1125930507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f250c15ca70 msgr2=0x7f250c0b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:14.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.027+0000 7f2513200640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f250c15ca70 0x7f250c0b4cd0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f24fc00cbc0 tx=0x7f24fc00a430 comp rx=0 tx=0).stop 2024-03-23T11:31:14.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.028+0000 7f2513200640 1 -- 172.21.15.62:0/1125930507 shutdown_connections 2024-03-23T11:31:14.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.028+0000 7f2513200640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f24e806a7d0 0x7f24e806cc90 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:14.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.028+0000 7f2513200640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f250c15ca70 0x7f250c0b4cd0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:14.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.028+0000 7f2513200640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f250c160be0 0x7f250c1f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:14.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.028+0000 7f2513200640 1 --2- 172.21.15.62:0/1125930507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f250c0cf9d0 0x7f250c0b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:14.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.028+0000 7f2513200640 1 -- 172.21.15.62:0/1125930507 >> 172.21.15.62:0/1125930507 conn(0x7f250c071390 msgr2=0x7f250c0d29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:14.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.028+0000 7f2513200640 1 -- 172.21.15.62:0/1125930507 shutdown_connections 2024-03-23T11:31:14.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.028+0000 7f2513200640 1 -- 172.21.15.62:0/1125930507 wait complete. 2024-03-23T11:31:14.040 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T11:31:14.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.271+0000 7fd70fe00640 1 Processor -- start 2024-03-23T11:31:14.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.271+0000 7fd70fe00640 1 -- start start 2024-03-23T11:31:14.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.273+0000 7fd70fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd7100b1bb0 0x7fd7100ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:14.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.273+0000 7fd70fe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7100b1160 con 0x7fd7100cebd0 2024-03-23T11:31:14.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.273+0000 7fd70fe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7100cf000 con 0x7fd71006c890 2024-03-23T11:31:14.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.273+0000 7fd70fe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7100b2040 con 0x7fd7100b1bb0 2024-03-23T11:31:14.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.273+0000 7fd70f400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd7100cebd0 0x7fd71006ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37554/0 (socket says 172.21.15.62:37554) 2024-03-23T11:31:14.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.273+0000 7fd70f400640 1 -- 172.21.15.62:0/607498216 learned_addr learned my addr 172.21.15.62:0/607498216 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:14.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.274+0000 7fd706000640 1 --2- 172.21.15.62:0/607498216 >> v2:172.21.15.120:3300/0 conn(0x7fd7100b1bb0 0x7fd7100ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:14.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.274+0000 7fd70e000640 1 -- 172.21.15.62:0/607498216 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482905312 0 0) 0x7fd7100b1160 con 0x7fd7100cebd0 2024-03-23T11:31:14.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.274+0000 7fd70e000640 1 -- 172.21.15.62:0/607498216 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6f4003520 con 0x7fd7100cebd0 2024-03-23T11:31:14.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.274+0000 7fd70e000640 1 -- 172.21.15.62:0/607498216 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1026741196 0 0) 0x7fd6f4003520 con 0x7fd7100cebd0 2024-03-23T11:31:14.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.274+0000 7fd706000640 1 -- 172.21.15.62:0/607498216 >> v1:172.21.15.120:6789/0 conn(0x7fd71006c890 legacy=0x7fd71015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:14.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.274+0000 7fd706000640 1 -- 172.21.15.62:0/607498216 >> v1:172.21.15.62:6789/0 conn(0x7fd7100cebd0 legacy=0x7fd71006ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:14.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.274+0000 7fd706000640 1 -- 172.21.15.62:0/607498216 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71015d840 con 0x7fd7100b1bb0 2024-03-23T11:31:14.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.276+0000 7fd706000640 1 --2- 172.21.15.62:0/607498216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7100b1bb0 0x7fd7100ce690 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd7100cf190 tx=0x7fd6f802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66f258cd45804419 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:14.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.277+0000 7fd70e000640 1 -- 172.21.15.62:0/607498216 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708001670 con 0x7fd7100b1bb0 2024-03-23T11:31:14.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.277+0000 7fd70e000640 1 -- 172.21.15.62:0/607498216 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f80314f0 con 0x7fd7100b1bb0 2024-03-23T11:31:14.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.277+0000 7fd70e000640 1 -- 172.21.15.62:0/607498216 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f8031880 con 0x7fd7100b1bb0 2024-03-23T11:31:14.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.277+0000 7fd70fe00640 1 -- 172.21.15.62:0/607498216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7100b1bb0 msgr2=0x7fd7100ce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:14.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.277+0000 7fd70fe00640 1 --2- 172.21.15.62:0/607498216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7100b1bb0 0x7fd7100ce690 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd7100cf190 tx=0x7fd6f802b810 comp rx=0 tx=0).stop 2024-03-23T11:31:14.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.278+0000 7fd70fe00640 1 -- 172.21.15.62:0/607498216 shutdown_connections 2024-03-23T11:31:14.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.278+0000 7fd70fe00640 1 --2- 172.21.15.62:0/607498216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7100b1bb0 0x7fd7100ce690 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:14.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.278+0000 7fd70fe00640 1 -- 172.21.15.62:0/607498216 >> 172.21.15.62:0/607498216 conn(0x7fd7100b0280 msgr2=0x7fd7100b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:14.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.278+0000 7fd70fe00640 1 -- 172.21.15.62:0/607498216 shutdown_connections 2024-03-23T11:31:14.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.278+0000 7fd70fe00640 1 -- 172.21.15.62:0/607498216 wait complete. 2024-03-23T11:31:14.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.279+0000 7fd70fe00640 1 Processor -- start 2024-03-23T11:31:14.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.280+0000 7fd70fe00640 1 -- start start 2024-03-23T11:31:14.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd71006c890 0x7fd7101f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:14.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7100b1bb0 0x7fd71015c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7100cebd0 0x7fd7101f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70fe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7101f23e0 con 0x7fd7100cebd0 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7101f5e10 con 0x7fd7100b1bb0 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7101f60f0 con 0x7fd71006c890 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7100cebd0 0x7fd7101f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7100cebd0 0x7fd7101f2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54572/0 (socket says 172.21.15.62:54572) 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70f400640 1 -- 172.21.15.62:0/3554328930 learned_addr learned my addr 172.21.15.62:0/3554328930 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70ea00640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd71006c890 0x7fd7101f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd706000640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7100b1bb0 0x7fd71015c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70f400640 1 -- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd71006c890 msgr2=0x7fd7101f55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70f400640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd71006c890 0x7fd7101f55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:14.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70f400640 1 -- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7100b1bb0 msgr2=0x7fd71015c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:14.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70f400640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7100b1bb0 0x7fd71015c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:14.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70f400640 1 -- 172.21.15.62:0/3554328930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f8009400 con 0x7fd7100cebd0 2024-03-23T11:31:14.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd70ea00640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd71006c890 0x7fd7101f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:14.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.281+0000 7fd706000640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7100b1bb0 0x7fd71015c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:14.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.282+0000 7fd70f400640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7100cebd0 0x7fd7101f2800 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fd70800c750 tx=0x7fd70800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:14.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.282+0000 7fd707400640 1 -- 172.21.15.62:0/3554328930 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708005ad0 con 0x7fd7100cebd0 2024-03-23T11:31:14.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.282+0000 7fd707400640 1 -- 172.21.15.62:0/3554328930 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd708005c30 con 0x7fd7100cebd0 2024-03-23T11:31:14.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.282+0000 7fd70fe00640 1 -- 172.21.15.62:0/3554328930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7101f63d0 con 0x7fd7100cebd0 2024-03-23T11:31:14.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.282+0000 7fd707400640 1 -- 172.21.15.62:0/3554328930 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708007070 con 0x7fd7100cebd0 2024-03-23T11:31:14.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.282+0000 7fd70fe00640 1 -- 172.21.15.62:0/3554328930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7101f1f50 con 0x7fd7100cebd0 2024-03-23T11:31:14.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.283+0000 7fd70fe00640 1 -- 172.21.15.62:0/3554328930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7101f2520 con 0x7fd7100cebd0 2024-03-23T11:31:14.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.284+0000 7fd707400640 1 -- 172.21.15.62:0/3554328930 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd7080071d0 con 0x7fd7100cebd0 2024-03-23T11:31:14.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.285+0000 7fd707400640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6ec06a7d0 0x7fd6ec06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:14.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.285+0000 7fd707400640 1 -- 172.21.15.62:0/3554328930 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5443+0+0 (secure 0 0 0) 0x7fd7080883c0 con 0x7fd7100cebd0 2024-03-23T11:31:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.286+0000 7fd70ea00640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6ec06a7d0 0x7fd6ec06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:14.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.287+0000 7fd70ea00640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6ec06a7d0 0x7fd6ec06cc90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd7000012e0 tx=0x7fd700009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:14.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.289+0000 7fd707400640 1 -- 172.21.15.62:0/3554328930 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd7101f2520 con 0x7fd7100cebd0 2024-03-23T11:31:14.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:14.487+0000 7fd70fe00640 1 -- 172.21.15.62:0/3554328930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd7101f2520 con 0x7fd7100cebd0 2024-03-23T11:31:15.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.027+0000 7fd707400640 1 -- 172.21.15.62:0/3554328930 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v109) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd7101f2520 con 0x7fd7100cebd0 2024-03-23T11:31:15.018 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' removed 2024-03-23T11:31:15.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.032+0000 7fd70fe00640 1 -- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6ec06a7d0 msgr2=0x7fd6ec06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.032+0000 7fd70fe00640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6ec06a7d0 0x7fd6ec06cc90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd7000012e0 tx=0x7fd700009040 comp rx=0 tx=0).stop 2024-03-23T11:31:15.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.032+0000 7fd70fe00640 1 -- 172.21.15.62:0/3554328930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7100cebd0 msgr2=0x7fd7101f2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.032+0000 7fd70fe00640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7100cebd0 0x7fd7101f2800 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fd70800c750 tx=0x7fd70800cc10 comp rx=0 tx=0).stop 2024-03-23T11:31:15.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.034+0000 7fd70fe00640 1 -- 172.21.15.62:0/3554328930 shutdown_connections 2024-03-23T11:31:15.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.034+0000 7fd70fe00640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6ec06a7d0 0x7fd6ec06cc90 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.034+0000 7fd70fe00640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7100cebd0 0x7fd7101f2800 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.034+0000 7fd70fe00640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7100b1bb0 0x7fd71015c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.034+0000 7fd70fe00640 1 --2- 172.21.15.62:0/3554328930 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd71006c890 0x7fd7101f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.034+0000 7fd70fe00640 1 -- 172.21.15.62:0/3554328930 >> 172.21.15.62:0/3554328930 conn(0x7fd7100b0280 msgr2=0x7fd71006ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:15.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.035+0000 7fd70fe00640 1 -- 172.21.15.62:0/3554328930 shutdown_connections 2024-03-23T11:31:15.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.035+0000 7fd70fe00640 1 -- 172.21.15.62:0/3554328930 wait complete. 2024-03-23T11:31:15.051 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T11:31:15.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.274+0000 7ffaaf200640 1 Processor -- start 2024-03-23T11:31:15.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.274+0000 7ffaaf200640 1 -- start start 2024-03-23T11:31:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.275+0000 7ffaaf200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00caff0 0x7ffab00cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.275+0000 7ffaaf200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffab0143450 0x7ffab01438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.276+0000 7ffaaf200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab015bb10 con 0x7ffab0069c60 2024-03-23T11:31:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.276+0000 7ffaaf200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab00ce1a0 con 0x7ffab0143450 2024-03-23T11:31:15.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.276+0000 7ffaaf200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00cbcb0 con 0x7ffab00caff0 2024-03-23T11:31:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.276+0000 7ffaade00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ffab0069c60 0x7ffab006a040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37560/0 (socket says 172.21.15.62:37560) 2024-03-23T11:31:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.276+0000 7ffaade00640 1 -- 172.21.15.62:0/3511984226 learned_addr learned my addr 172.21.15.62:0/3511984226 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.276+0000 7ffaae800640 1 --2- 172.21.15.62:0/3511984226 >> v2:172.21.15.62:3300/0 conn(0x7ffab0143450 0x7ffab01438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.277+0000 7ffaa7e00640 1 --2- 172.21.15.62:0/3511984226 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00caff0 0x7ffab00cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.277+0000 7ffaad400640 1 -- 172.21.15.62:0/3511984226 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260849410 0 0) 0x7ffab015bb10 con 0x7ffab0069c60 2024-03-23T11:31:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.277+0000 7ffaad400640 1 -- 172.21.15.62:0/3511984226 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa94003520 con 0x7ffab0069c60 2024-03-23T11:31:15.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.277+0000 7ffaae800640 1 -- 172.21.15.62:0/3511984226 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00caff0 msgr2=0x7ffab00cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:15.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.277+0000 7ffaae800640 1 --2- 172.21.15.62:0/3511984226 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00caff0 0x7ffab00cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.277+0000 7ffaae800640 1 -- 172.21.15.62:0/3511984226 >> v1:172.21.15.62:6789/0 conn(0x7ffab0069c60 legacy=0x7ffab006a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.277+0000 7ffaae800640 1 -- 172.21.15.62:0/3511984226 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab015ad50 con 0x7ffab0143450 2024-03-23T11:31:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.279+0000 7ffaae800640 1 --2- 172.21.15.62:0/3511984226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab0143450 0x7ffab01438e0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8009dc0 tx=0x7ffaa802be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15662afe778900ab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.281+0000 7ffaad400640 1 -- 172.21.15.62:0/3511984226 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa802cc30 con 0x7ffab0143450 2024-03-23T11:31:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.281+0000 7ffaad400640 1 -- 172.21.15.62:0/3511984226 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa802cd90 con 0x7ffab0143450 2024-03-23T11:31:15.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.281+0000 7ffaad400640 1 -- 172.21.15.62:0/3511984226 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8034660 con 0x7ffab0143450 2024-03-23T11:31:15.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.282+0000 7ffaaf200640 1 -- 172.21.15.62:0/3511984226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab0143450 msgr2=0x7ffab01438e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.282+0000 7ffaaf200640 1 --2- 172.21.15.62:0/3511984226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab0143450 0x7ffab01438e0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8009dc0 tx=0x7ffaa802be80 comp rx=0 tx=0).stop 2024-03-23T11:31:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.283+0000 7ffaaf200640 1 -- 172.21.15.62:0/3511984226 shutdown_connections 2024-03-23T11:31:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.283+0000 7ffaaf200640 1 --2- 172.21.15.62:0/3511984226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab0143450 0x7ffab01438e0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.283+0000 7ffaaf200640 1 --2- 172.21.15.62:0/3511984226 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00caff0 0x7ffab00cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.283+0000 7ffaaf200640 1 -- 172.21.15.62:0/3511984226 >> 172.21.15.62:0/3511984226 conn(0x7ffab00b0090 msgr2=0x7ffab00b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:15.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.284+0000 7ffaaf200640 1 -- 172.21.15.62:0/3511984226 shutdown_connections 2024-03-23T11:31:15.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.284+0000 7ffaaf200640 1 -- 172.21.15.62:0/3511984226 wait complete. 2024-03-23T11:31:15.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.287+0000 7ffaaf200640 1 Processor -- start 2024-03-23T11:31:15.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.287+0000 7ffaaf200640 1 -- start start 2024-03-23T11:31:15.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.289+0000 7ffaaf200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab0069c60 0x7ffab0142ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.289+0000 7ffaaf200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00caff0 0x7ffab00ac3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.289+0000 7ffaaf200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffab0143450 0x7ffab00ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.289+0000 7ffaaf200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00cad20 con 0x7ffab0069c60 2024-03-23T11:31:15.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.289+0000 7ffaaf200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00adc70 con 0x7ffab0143450 2024-03-23T11:31:15.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.289+0000 7ffaaf200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00ac910 con 0x7ffab00caff0 2024-03-23T11:31:15.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.290+0000 7ffaade00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab0069c60 0x7ffab0142ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.290+0000 7ffaade00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab0069c60 0x7ffab0142ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54594/0 (socket says 172.21.15.62:54594) 2024-03-23T11:31:15.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.290+0000 7ffaade00640 1 -- 172.21.15.62:0/256396126 learned_addr learned my addr 172.21.15.62:0/256396126 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:15.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.290+0000 7ffaa7e00640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00caff0 0x7ffab00ac3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.291+0000 7ffaae800640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffab0143450 0x7ffab00ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.291+0000 7ffaade00640 1 -- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00caff0 msgr2=0x7ffab00ac3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.291+0000 7ffaade00640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00caff0 0x7ffab00ac3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.291+0000 7ffaade00640 1 -- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffab0143450 msgr2=0x7ffab00ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.291+0000 7ffaade00640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffab0143450 0x7ffab00ae9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.291+0000 7ffaade00640 1 -- 172.21.15.62:0/256396126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa8009a70 con 0x7ffab0069c60 2024-03-23T11:31:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.292+0000 7ffaade00640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab0069c60 0x7ffab0142ca0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000c500 tx=0x7ffaa000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.293+0000 7ffaa6000640 1 -- 172.21.15.62:0/256396126 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0005ce0 con 0x7ffab0069c60 2024-03-23T11:31:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.293+0000 7ffaa6000640 1 -- 172.21.15.62:0/256396126 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa0005e40 con 0x7ffab0069c60 2024-03-23T11:31:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.293+0000 7ffaa6000640 1 -- 172.21.15.62:0/256396126 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0007330 con 0x7ffab0069c60 2024-03-23T11:31:15.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.293+0000 7ffaaf200640 1 -- 172.21.15.62:0/256396126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab00acbd0 con 0x7ffab0069c60 2024-03-23T11:31:15.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.294+0000 7ffaaf200640 1 -- 172.21.15.62:0/256396126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab00ad190 con 0x7ffab0069c60 2024-03-23T11:31:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.296+0000 7ffaaf200640 1 -- 172.21.15.62:0/256396126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab00b3670 con 0x7ffab0069c60 2024-03-23T11:31:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.300+0000 7ffaa6000640 1 -- 172.21.15.62:0/256396126 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ffaa0007550 con 0x7ffab0069c60 2024-03-23T11:31:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.301+0000 7ffaa6000640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa8c06a7f0 0x7ffa8c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.301+0000 7ffaa6000640 1 -- 172.21.15.62:0/256396126 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5029+0+0 (secure 0 0 0) 0x7ffaa0088190 con 0x7ffab0069c60 2024-03-23T11:31:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.302+0000 7ffaa7e00640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa8c06a7f0 0x7ffa8c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.305+0000 7ffaa7e00640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa8c06a7f0 0x7ffa8c06ccb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ffab00afff0 tx=0x7ffa98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:15.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.308+0000 7ffaa6000640 1 -- 172.21.15.62:0/256396126 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffaa0053980 con 0x7ffab0069c60 2024-03-23T11:31:15.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.508+0000 7ffaaf200640 1 -- 172.21.15.62:0/256396126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ffab0066bb0 con 0x7ffab0069c60 2024-03-23T11:31:15.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.514+0000 7ffaa6000640 1 -- 172.21.15.62:0/256396126 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7ffaa0056fa0 con 0x7ffab0069c60 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.516+0000 7ffaaf200640 1 -- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa8c06a7f0 msgr2=0x7ffa8c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.516+0000 7ffaaf200640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa8c06a7f0 0x7ffa8c06ccb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ffab00afff0 tx=0x7ffa98008040 comp rx=0 tx=0).stop 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.516+0000 7ffaaf200640 1 -- 172.21.15.62:0/256396126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab0069c60 msgr2=0x7ffab0142ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.516+0000 7ffaaf200640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab0069c60 0x7ffab0142ca0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000c500 tx=0x7ffaa000c9c0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.516+0000 7ffaaf200640 1 -- 172.21.15.62:0/256396126 shutdown_connections 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.516+0000 7ffaaf200640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa8c06a7f0 0x7ffa8c06ccb0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.516+0000 7ffaaf200640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffab0143450 0x7ffab00ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.516+0000 7ffaaf200640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab00caff0 0x7ffab00ac3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.516+0000 7ffaaf200640 1 --2- 172.21.15.62:0/256396126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab0069c60 0x7ffab0142ca0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.516+0000 7ffaaf200640 1 -- 172.21.15.62:0/256396126 >> 172.21.15.62:0/256396126 conn(0x7ffab00b0090 msgr2=0x7ffab00cd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.517+0000 7ffaaf200640 1 -- 172.21.15.62:0/256396126 shutdown_connections 2024-03-23T11:31:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.517+0000 7ffaaf200640 1 -- 172.21.15.62:0/256396126 wait complete. 2024-03-23T11:31:15.527 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-03-23T11:31:15.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.742+0000 7fd377200640 1 Processor -- start 2024-03-23T11:31:15.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.743+0000 7fd377200640 1 -- start start 2024-03-23T11:31:15.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.744+0000 7fd377200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370143dd0 0x7fd37015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.744+0000 7fd377200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd3700b2f30 0x7fd3700704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.744+0000 7fd377200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370070a30 con 0x7fd370070090 2024-03-23T11:31:15.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.744+0000 7fd377200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3700cd140 con 0x7fd3700b2f30 2024-03-23T11:31:15.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.744+0000 7fd377200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3700cd3e0 con 0x7fd370143dd0 2024-03-23T11:31:15.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.745+0000 7fd375400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd370070090 0x7fd3700b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37572/0 (socket says 172.21.15.62:37572) 2024-03-23T11:31:15.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.745+0000 7fd375400640 1 -- 172.21.15.62:0/2818576481 learned_addr learned my addr 172.21.15.62:0/2818576481 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:15.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.745+0000 7fd376800640 1 --2- 172.21.15.62:0/2818576481 >> v2:172.21.15.120:3300/0 conn(0x7fd3700b2f30 0x7fd3700704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.745+0000 7fd375e00640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370143dd0 0x7fd37015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.745+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521110677 0 0) 0x7fd370070a30 con 0x7fd370070090 2024-03-23T11:31:15.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.745+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd358003520 con 0x7fd370070090 2024-03-23T11:31:15.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.745+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 255729217 0 0) 0x7fd358003520 con 0x7fd370070090 2024-03-23T11:31:15.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370143dd0 msgr2=0x7fd37015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:15.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370143dd0 0x7fd37015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 >> v2:172.21.15.120:3300/0 conn(0x7fd3700b2f30 msgr2=0x7fd3700704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:15.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 --2- 172.21.15.62:0/2818576481 >> v2:172.21.15.120:3300/0 conn(0x7fd3700b2f30 0x7fd3700704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3700cd6c0 con 0x7fd370070090 2024-03-23T11:31:15.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fd360001670 con 0x7fd370070090 2024-03-23T11:31:15.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 >> v1:172.21.15.62:6789/0 conn(0x7fd370070090 legacy=0x7fd3700b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 0x7fd358003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3580047c0 0x7fd358004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370143dd0 0x7fd358005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3700cd140 con 0x7fd358003970 2024-03-23T11:31:15.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3700cd3e0 con 0x7fd3580047c0 2024-03-23T11:31:15.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35800b980 con 0x7fd370143dd0 2024-03-23T11:31:15.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd375e00640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 0x7fd358003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd375400640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3580047c0 0x7fd358004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd376800640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370143dd0 0x7fd358005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd375e00640 1 -- 172.21.15.62:0/2818576481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370143dd0 msgr2=0x7fd358005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:15.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd375e00640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370143dd0 0x7fd358005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd375e00640 1 -- 172.21.15.62:0/2818576481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3580047c0 msgr2=0x7fd358004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:15.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd375e00640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3580047c0 0x7fd358004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.746+0000 7fd375e00640 1 -- 172.21.15.62:0/2818576481 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd358003520 con 0x7fd358003970 2024-03-23T11:31:15.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.748+0000 7fd375e00640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 0x7fd358003d90 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ac0 tx=0x7fd3580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde97418fbe7bdee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:15.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.748+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36402c6b0 con 0x7fd358003970 2024-03-23T11:31:15.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.748+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36402c810 con 0x7fd358003970 2024-03-23T11:31:15.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.748+0000 7fd374a00640 1 -- 172.21.15.62:0/2818576481 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36402cba0 con 0x7fd358003970 2024-03-23T11:31:15.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.748+0000 7fd377200640 1 -- 172.21.15.62:0/2818576481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 msgr2=0x7fd358003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.748+0000 7fd377200640 1 --2- 172.21.15.62:0/2818576481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 0x7fd358003d90 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ac0 tx=0x7fd3580057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.748+0000 7fd376800640 1 -- 172.21.15.62:0/2818576481 reap_dead start 2024-03-23T11:31:15.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.749+0000 7fd377200640 1 -- 172.21.15.62:0/2818576481 shutdown_connections 2024-03-23T11:31:15.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.749+0000 7fd377200640 1 -- 172.21.15.62:0/2818576481 >> 172.21.15.62:0/2818576481 conn(0x7fd370066050 msgr2=0x7fd370069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:15.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.749+0000 7fd377200640 1 -- 172.21.15.62:0/2818576481 shutdown_connections 2024-03-23T11:31:15.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.749+0000 7fd377200640 1 -- 172.21.15.62:0/2818576481 wait complete. 2024-03-23T11:31:15.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.750+0000 7fd377200640 1 Processor -- start 2024-03-23T11:31:15.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.750+0000 7fd377200640 1 -- start start 2024-03-23T11:31:15.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.751+0000 7fd377200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 0x7fd37015c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.751+0000 7fd377200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd370154af0 0x7fd370154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.751+0000 7fd377200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370159680 0x7fd370159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.751+0000 7fd377200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3700ae170 con 0x7fd358003970 2024-03-23T11:31:15.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.751+0000 7fd377200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3701554b0 con 0x7fd370154af0 2024-03-23T11:31:15.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.751+0000 7fd377200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37015a3d0 con 0x7fd370159680 2024-03-23T11:31:15.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd375e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 0x7fd37015c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd375e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 0x7fd37015c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54600/0 (socket says 172.21.15.62:54600) 2024-03-23T11:31:15.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd375e00640 1 -- 172.21.15.62:0/3748839454 learned_addr learned my addr 172.21.15.62:0/3748839454 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:15.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd375400640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd370154af0 0x7fd370154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd375e00640 1 -- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370159680 msgr2=0x7fd370159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd376800640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370159680 0x7fd370159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd375e00640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370159680 0x7fd370159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd375e00640 1 -- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd370154af0 msgr2=0x7fd370154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:15.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd375e00640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd370154af0 0x7fd370154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:15.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd375e00640 1 -- 172.21.15.62:0/3748839454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37015a6b0 con 0x7fd358003970 2024-03-23T11:31:15.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd375400640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd370154af0 0x7fd370154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:15.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.752+0000 7fd375e00640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 0x7fd37015c4c0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ac0 tx=0x7fd364004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:15.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.753+0000 7fd36a000640 1 -- 172.21.15.62:0/3748839454 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd364003810 con 0x7fd358003970 2024-03-23T11:31:15.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.753+0000 7fd36a000640 1 -- 172.21.15.62:0/3748839454 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd364003970 con 0x7fd358003970 2024-03-23T11:31:15.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.753+0000 7fd36a000640 1 -- 172.21.15.62:0/3748839454 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd364003d00 con 0x7fd358003970 2024-03-23T11:31:15.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.753+0000 7fd377200640 1 -- 172.21.15.62:0/3748839454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37015a990 con 0x7fd358003970 2024-03-23T11:31:15.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.753+0000 7fd377200640 1 -- 172.21.15.62:0/3748839454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd370137a90 con 0x7fd358003970 2024-03-23T11:31:15.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.754+0000 7fd377200640 1 -- 172.21.15.62:0/3748839454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd370066bb0 con 0x7fd358003970 2024-03-23T11:31:15.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.755+0000 7fd36a000640 1 -- 172.21.15.62:0/3748839454 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd364040450 con 0x7fd358003970 2024-03-23T11:31:15.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.756+0000 7fd36a000640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35406a7d0 0x7fd35406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:15.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.756+0000 7fd36a000640 1 -- 172.21.15.62:0/3748839454 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd3640aae60 con 0x7fd358003970 2024-03-23T11:31:15.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.756+0000 7fd375400640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35406a7d0 0x7fd35406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:15.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.758+0000 7fd375400640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35406a7d0 0x7fd35406cc90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd360001eb0 tx=0x7fd360000f70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:15.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:15.761+0000 7fd36a000640 1 -- 172.21.15.62:0/3748839454 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd364076480 con 0x7fd358003970 2024-03-23T11:31:16.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.023+0000 7fd377200640 1 -- 172.21.15.62:0/3748839454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) v1 -- 0x7fd37013a320 con 0x7fd358003970 2024-03-23T11:31:16.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.044+0000 7fd36a000640 1 -- 172.21.15.62:0/3748839454 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v564) v1 ==== 146+0+0 (secure 0 0 0) 0x7fd3640044e0 con 0x7fd358003970 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.046+0000 7fd377200640 1 -- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35406a7d0 msgr2=0x7fd35406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.046+0000 7fd377200640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35406a7d0 0x7fd35406cc90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd360001eb0 tx=0x7fd360000f70 comp rx=0 tx=0).stop 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.046+0000 7fd377200640 1 -- 172.21.15.62:0/3748839454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 msgr2=0x7fd37015c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.046+0000 7fd377200640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 0x7fd37015c4c0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ac0 tx=0x7fd364004730 comp rx=0 tx=0).stop 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.046+0000 7fd377200640 1 -- 172.21.15.62:0/3748839454 shutdown_connections 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.046+0000 7fd377200640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd35406a7d0 0x7fd35406cc90 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.046+0000 7fd377200640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd370159680 0x7fd370159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.046+0000 7fd377200640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd370154af0 0x7fd370154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.046+0000 7fd377200640 1 --2- 172.21.15.62:0/3748839454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd358003970 0x7fd37015c4c0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.046+0000 7fd377200640 1 -- 172.21.15.62:0/3748839454 >> 172.21.15.62:0/3748839454 conn(0x7fd370066050 msgr2=0x7fd37015b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.047+0000 7fd377200640 1 -- 172.21.15.62:0/3748839454 shutdown_connections 2024-03-23T11:31:16.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.047+0000 7fd377200640 1 -- 172.21.15.62:0/3748839454 wait complete. 2024-03-23T11:31:16.055 INFO:tasks.cephfs_test_runner:test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-03-23T11:31:16.055 INFO:tasks.cephfs_test_runner:Starting test: test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-03-23T11:31:16.055 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-03-23T11:31:16.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.282+0000 7f8e37000640 1 Processor -- start 2024-03-23T11:31:16.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.283+0000 7f8e37000640 1 -- start start 2024-03-23T11:31:16.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.284+0000 7f8e37000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8e3015fcf0 0x7f8e300d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:16.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.284+0000 7f8e37000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e30160420 0x7f8e30160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:16.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.284+0000 7f8e37000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8e300d06d0 0x7f8e300d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:16.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.284+0000 7f8e37000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e30160dc0 con 0x7f8e300d06d0 2024-03-23T11:31:16.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.284+0000 7f8e37000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e300d3540 con 0x7f8e3015fcf0 2024-03-23T11:31:16.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.284+0000 7f8e37000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e300ce510 con 0x7f8e30160420 2024-03-23T11:31:16.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.285+0000 7f8e36600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8e300d06d0 0x7f8e300d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:16.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.285+0000 7f8e36600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8e300d06d0 0x7f8e300d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54606/0 (socket says 172.21.15.62:54606) 2024-03-23T11:31:16.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.285+0000 7f8e36600640 1 -- 172.21.15.62:0/1744809870 learned_addr learned my addr 172.21.15.62:0/1744809870 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:16.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.285+0000 7f8e35c00640 1 --2- 172.21.15.62:0/1744809870 >> v2:172.21.15.120:3300/0 conn(0x7f8e3015fcf0 0x7f8e300d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:16.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.285+0000 7f8e36600640 1 -- 172.21.15.62:0/1744809870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e30160420 msgr2=0x7f8e30160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:16.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.285+0000 7f8e36600640 1 --2- 172.21.15.62:0/1744809870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e30160420 0x7f8e30160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:16.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.285+0000 7f8e36600640 1 -- 172.21.15.62:0/1744809870 >> v2:172.21.15.120:3300/0 conn(0x7f8e3015fcf0 msgr2=0x7f8e300d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:16.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.285+0000 7f8e36600640 1 --2- 172.21.15.62:0/1744809870 >> v2:172.21.15.120:3300/0 conn(0x7f8e3015fcf0 0x7f8e300d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:16.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.285+0000 7f8e36600640 1 -- 172.21.15.62:0/1744809870 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e300cdc70 con 0x7f8e300d06d0 2024-03-23T11:31:16.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.286+0000 7f8e36600640 1 --2- 172.21.15.62:0/1744809870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e300d06d0 0x7f8e300d0b20 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f8e30066480 tx=0x7f8e2802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514d2cbdfe3582ea server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:16.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.287+0000 7f8e2fe00640 1 -- 172.21.15.62:0/1744809870 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2802c9e0 con 0x7f8e300d06d0 2024-03-23T11:31:16.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.287+0000 7f8e2fe00640 1 -- 172.21.15.62:0/1744809870 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2802cb40 con 0x7f8e300d06d0 2024-03-23T11:31:16.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.287+0000 7f8e2fe00640 1 -- 172.21.15.62:0/1744809870 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e280343f0 con 0x7f8e300d06d0 2024-03-23T11:31:16.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.287+0000 7f8e37000640 1 -- 172.21.15.62:0/1744809870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e300d06d0 msgr2=0x7f8e300d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:16.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.287+0000 7f8e37000640 1 --2- 172.21.15.62:0/1744809870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e300d06d0 0x7f8e300d0b20 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f8e30066480 tx=0x7f8e2802be40 comp rx=0 tx=0).stop 2024-03-23T11:31:16.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.287+0000 7f8e37000640 1 -- 172.21.15.62:0/1744809870 shutdown_connections 2024-03-23T11:31:16.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.287+0000 7f8e37000640 1 --2- 172.21.15.62:0/1744809870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e300d06d0 0x7f8e300d0b20 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:16.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.287+0000 7f8e37000640 1 --2- 172.21.15.62:0/1744809870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e30160420 0x7f8e30160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:16.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.288+0000 7f8e37000640 1 --2- 172.21.15.62:0/1744809870 >> v2:172.21.15.120:3300/0 conn(0x7f8e3015fcf0 0x7f8e300d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:16.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.288+0000 7f8e37000640 1 -- 172.21.15.62:0/1744809870 >> 172.21.15.62:0/1744809870 conn(0x7f8e30089f60 msgr2=0x7f8e300d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:16.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.288+0000 7f8e37000640 1 -- 172.21.15.62:0/1744809870 shutdown_connections 2024-03-23T11:31:16.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.288+0000 7f8e37000640 1 -- 172.21.15.62:0/1744809870 wait complete. 2024-03-23T11:31:16.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.289+0000 7f8e37000640 1 Processor -- start 2024-03-23T11:31:16.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.290+0000 7f8e37000640 1 -- start start 2024-03-23T11:31:16.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.290+0000 7f8e37000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e300d06d0 0x7f8e300c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:16.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.290+0000 7f8e37000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e3015fcf0 0x7f8e300cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:16.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.290+0000 7f8e37000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e30160420 0x7f8e3009c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:16.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.290+0000 7f8e37000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e300c8ee0 con 0x7f8e300d06d0 2024-03-23T11:31:16.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.290+0000 7f8e37000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e300cbc40 con 0x7f8e3015fcf0 2024-03-23T11:31:16.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.290+0000 7f8e37000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e300cbf20 con 0x7f8e30160420 2024-03-23T11:31:16.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.291+0000 7f8e35c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e300d06d0 0x7f8e300c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:16.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.291+0000 7f8e35c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e300d06d0 0x7f8e300c89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54618/0 (socket says 172.21.15.62:54618) 2024-03-23T11:31:16.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.291+0000 7f8e35c00640 1 -- 172.21.15.62:0/980953581 learned_addr learned my addr 172.21.15.62:0/980953581 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:16.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.291+0000 7f8e35c00640 1 -- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e30160420 msgr2=0x7f8e3009c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:16.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.291+0000 7f8e35c00640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e30160420 0x7f8e3009c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:16.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.291+0000 7f8e35c00640 1 -- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e3015fcf0 msgr2=0x7f8e300cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:16.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.291+0000 7f8e35c00640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e3015fcf0 0x7f8e300cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:16.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.291+0000 7f8e35c00640 1 -- 172.21.15.62:0/980953581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e28009b30 con 0x7f8e300d06d0 2024-03-23T11:31:16.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.291+0000 7f8e35c00640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e300d06d0 0x7f8e300c89a0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400d5f0 tx=0x7f8e2400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:16.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.292+0000 7f8e2d600640 1 -- 172.21.15.62:0/980953581 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e240125f0 con 0x7f8e300d06d0 2024-03-23T11:31:16.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.292+0000 7f8e2d600640 1 -- 172.21.15.62:0/980953581 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e24012750 con 0x7f8e300d06d0 2024-03-23T11:31:16.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.292+0000 7f8e2d600640 1 -- 172.21.15.62:0/980953581 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e24012ae0 con 0x7f8e300d06d0 2024-03-23T11:31:16.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.292+0000 7f8e37000640 1 -- 172.21.15.62:0/980953581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e300cc200 con 0x7f8e300d06d0 2024-03-23T11:31:16.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.292+0000 7f8e37000640 1 -- 172.21.15.62:0/980953581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e300c6880 con 0x7f8e300d06d0 2024-03-23T11:31:16.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.293+0000 7f8e37000640 1 -- 172.21.15.62:0/980953581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e300c6a40 con 0x7f8e300d06d0 2024-03-23T11:31:16.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.294+0000 7f8e2d600640 1 -- 172.21.15.62:0/980953581 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8e24012c40 con 0x7f8e300d06d0 2024-03-23T11:31:16.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.295+0000 7f8e2d600640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e1006a7d0 0x7f8e1006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:16.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.295+0000 7f8e2d600640 1 -- 172.21.15.62:0/980953581 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8e24088f60 con 0x7f8e300d06d0 2024-03-23T11:31:16.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.296+0000 7f8e35200640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e1006a7d0 0x7f8e1006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:16.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.297+0000 7f8e35200640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e1006a7d0 0x7f8e1006cc90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8e3008c710 tx=0x7f8e1c006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:16.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.300+0000 7f8e2d600640 1 -- 172.21.15.62:0/980953581 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8e24054620 con 0x7f8e300d06d0 2024-03-23T11:31:16.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:16.568+0000 7f8e37000640 1 -- 172.21.15.62:0/980953581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) v1 -- 0x7f8e30066bb0 con 0x7f8e300d06d0 2024-03-23T11:31:17.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.050+0000 7f8e2d600640 1 -- 172.21.15.62:0/980953581 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v565) v1 ==== 147+0+0 (secure 0 0 0) 0x7f8e24004640 con 0x7f8e300d06d0 2024-03-23T11:31:17.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.053+0000 7f8e37000640 1 -- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e1006a7d0 msgr2=0x7f8e1006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:17.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.053+0000 7f8e37000640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e1006a7d0 0x7f8e1006cc90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8e3008c710 tx=0x7f8e1c006c90 comp rx=0 tx=0).stop 2024-03-23T11:31:17.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.053+0000 7f8e37000640 1 -- 172.21.15.62:0/980953581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e300d06d0 msgr2=0x7f8e300c89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:17.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.053+0000 7f8e37000640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e300d06d0 0x7f8e300c89a0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400d5f0 tx=0x7f8e2400dab0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.053+0000 7f8e37000640 1 -- 172.21.15.62:0/980953581 shutdown_connections 2024-03-23T11:31:17.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.053+0000 7f8e37000640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8e1006a7d0 0x7f8e1006cc90 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.053+0000 7f8e37000640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e30160420 0x7f8e3009c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.053+0000 7f8e37000640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e3015fcf0 0x7f8e300cb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.053+0000 7f8e37000640 1 --2- 172.21.15.62:0/980953581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e300d06d0 0x7f8e300c89a0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.053+0000 7f8e37000640 1 -- 172.21.15.62:0/980953581 >> 172.21.15.62:0/980953581 conn(0x7f8e30089f60 msgr2=0x7f8e3006ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:17.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.054+0000 7f8e37000640 1 -- 172.21.15.62:0/980953581 shutdown_connections 2024-03-23T11:31:17.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.054+0000 7f8e37000640 1 -- 172.21.15.62:0/980953581 wait complete. 2024-03-23T11:31:17.065 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T11:31:17.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.272+0000 7f4518a00640 1 Processor -- start 2024-03-23T11:31:17.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.272+0000 7f4518a00640 1 -- start start 2024-03-23T11:31:17.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.273+0000 7f4518a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4514143dd0 0x7f451415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:17.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.273+0000 7f4518a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45140b2f30 0x7f4514070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:17.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.273+0000 7f4518a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45140cd140 con 0x7f4514070090 2024-03-23T11:31:17.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.273+0000 7f4518a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4514070a60 con 0x7f4514143dd0 2024-03-23T11:31:17.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.273+0000 7f4518a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45140cd3e0 con 0x7f45140b2f30 2024-03-23T11:31:17.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.274+0000 7f4512800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4514143dd0 0x7f451415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:17.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.274+0000 7f4513200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45140b2f30 0x7f4514070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:17.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.274+0000 7f4512800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4514143dd0 0x7f451415abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:49532/0 (socket says 172.21.15.62:49532) 2024-03-23T11:31:17.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.274+0000 7f4512800640 1 -- 172.21.15.62:0/2171468263 learned_addr learned my addr 172.21.15.62:0/2171468263 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:17.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.275+0000 7f4512800640 1 -- 172.21.15.62:0/2171468263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45140b2f30 msgr2=0x7f4514070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:17.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.275+0000 7f4512800640 1 --2- 172.21.15.62:0/2171468263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45140b2f30 0x7f4514070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.275+0000 7f4512800640 1 -- 172.21.15.62:0/2171468263 >> v1:172.21.15.120:6789/0 conn(0x7f4514070090 legacy=0x7f45140b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:17.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.276+0000 7f4512800640 1 -- 172.21.15.62:0/2171468263 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140cd6c0 con 0x7f4514143dd0 2024-03-23T11:31:17.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.276+0000 7f4511400640 1 -- 172.21.15.62:0/2171468263 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723266066 0 0) 0x7f45140cd140 con 0x7f4514070090 2024-03-23T11:31:17.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.277+0000 7f4512800640 1 --2- 172.21.15.62:0/2171468263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4514143dd0 0x7f451415abe0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4504009870 tx=0x7f450402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=accaa30aa4043ef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:17.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.277+0000 7f4511400640 1 -- 172.21.15.62:0/2171468263 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504030840 con 0x7f4514143dd0 2024-03-23T11:31:17.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.277+0000 7f4511400640 1 -- 172.21.15.62:0/2171468263 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45040309a0 con 0x7f4514143dd0 2024-03-23T11:31:17.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.277+0000 7f4511400640 1 -- 172.21.15.62:0/2171468263 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504030d30 con 0x7f4514143dd0 2024-03-23T11:31:17.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.278+0000 7f4518a00640 1 -- 172.21.15.62:0/2171468263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4514143dd0 msgr2=0x7f451415abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:17.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.278+0000 7f4518a00640 1 --2- 172.21.15.62:0/2171468263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4514143dd0 0x7f451415abe0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4504009870 tx=0x7f450402b980 comp rx=0 tx=0).stop 2024-03-23T11:31:17.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.278+0000 7f4518a00640 1 -- 172.21.15.62:0/2171468263 shutdown_connections 2024-03-23T11:31:17.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.278+0000 7f4518a00640 1 --2- 172.21.15.62:0/2171468263 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45140b2f30 0x7f4514070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.278+0000 7f4518a00640 1 --2- 172.21.15.62:0/2171468263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4514143dd0 0x7f451415abe0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.278+0000 7f4518a00640 1 -- 172.21.15.62:0/2171468263 >> 172.21.15.62:0/2171468263 conn(0x7f4514066050 msgr2=0x7f4514069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:17.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.278+0000 7f4518a00640 1 -- 172.21.15.62:0/2171468263 shutdown_connections 2024-03-23T11:31:17.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.278+0000 7f4518a00640 1 -- 172.21.15.62:0/2171468263 wait complete. 2024-03-23T11:31:17.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.280+0000 7f4518a00640 1 Processor -- start 2024-03-23T11:31:17.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.280+0000 7f4518a00640 1 -- start start 2024-03-23T11:31:17.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.281+0000 7f4518a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4514070090 0x7f45141f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:17.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.281+0000 7f4518a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45140b2f30 0x7f451413a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:17.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.281+0000 7f4518a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4514143dd0 0x7f451406c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:17.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.281+0000 7f4518a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45141f6280 con 0x7f45140b2f30 2024-03-23T11:31:17.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.281+0000 7f4518a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4514136e50 con 0x7f4514143dd0 2024-03-23T11:31:17.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.281+0000 7f4518a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4514137130 con 0x7f4514070090 2024-03-23T11:31:17.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.281+0000 7f4511e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45140b2f30 0x7f451413a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:17.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.281+0000 7f4511e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45140b2f30 0x7f451413a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47876/0 (socket says 172.21.15.62:47876) 2024-03-23T11:31:17.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.281+0000 7f4511e00640 1 -- 172.21.15.62:0/2552811758 learned_addr learned my addr 172.21.15.62:0/2552811758 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:17.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.282+0000 7f4512800640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4514070090 0x7f45141f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:17.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.282+0000 7f4513200640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4514143dd0 0x7f451406c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:17.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.282+0000 7f4511e00640 1 -- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4514070090 msgr2=0x7f45141f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:17.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.282+0000 7f4511e00640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4514070090 0x7f45141f5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.282+0000 7f4511e00640 1 -- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4514143dd0 msgr2=0x7f451406c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:17.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.282+0000 7f4511e00640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4514143dd0 0x7f451406c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.282+0000 7f4511e00640 1 -- 172.21.15.62:0/2552811758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45040094d0 con 0x7f45140b2f30 2024-03-23T11:31:17.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.282+0000 7f4511e00640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45140b2f30 0x7f451413a6a0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f450000c4b0 tx=0x7f450000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.282+0000 7f450aa00640 1 -- 172.21.15.62:0/2552811758 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500005480 con 0x7f45140b2f30 2024-03-23T11:31:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.282+0000 7f450aa00640 1 -- 172.21.15.62:0/2552811758 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45000035f0 con 0x7f45140b2f30 2024-03-23T11:31:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.282+0000 7f450aa00640 1 -- 172.21.15.62:0/2552811758 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45000039e0 con 0x7f45140b2f30 2024-03-23T11:31:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.283+0000 7f4518a00640 1 -- 172.21.15.62:0/2552811758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4514137410 con 0x7f45140b2f30 2024-03-23T11:31:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.283+0000 7f4518a00640 1 -- 172.21.15.62:0/2552811758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45141379d0 con 0x7f45140b2f30 2024-03-23T11:31:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.284+0000 7f4518a00640 1 -- 172.21.15.62:0/2552811758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4514066bb0 con 0x7f45140b2f30 2024-03-23T11:31:17.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.285+0000 7f450aa00640 1 -- 172.21.15.62:0/2552811758 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4500003b40 con 0x7f45140b2f30 2024-03-23T11:31:17.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.285+0000 7f450aa00640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f44f406a7a0 0x7f44f406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:17.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.286+0000 7f450aa00640 1 -- 172.21.15.62:0/2552811758 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4500088180 con 0x7f45140b2f30 2024-03-23T11:31:17.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.286+0000 7f4512800640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f44f406a7a0 0x7f44f406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:17.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.287+0000 7f4512800640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f44f406a7a0 0x7f44f406cc60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4504036000 tx=0x7f450403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:17.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.290+0000 7f450aa00640 1 -- 172.21.15.62:0/2552811758 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f45000537a0 con 0x7f45140b2f30 2024-03-23T11:31:17.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.494+0000 7f4518a00640 1 -- 172.21.15.62:0/2552811758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f45140ab8e0 con 0x7f45140b2f30 2024-03-23T11:31:17.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.501+0000 7f450aa00640 1 -- 172.21.15.62:0/2552811758 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f45000170c0 con 0x7f45140b2f30 2024-03-23T11:31:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.504+0000 7f4518a00640 1 -- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f44f406a7a0 msgr2=0x7f44f406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.504+0000 7f4518a00640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f44f406a7a0 0x7f44f406cc60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4504036000 tx=0x7f450403b040 comp rx=0 tx=0).stop 2024-03-23T11:31:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.505+0000 7f4518a00640 1 -- 172.21.15.62:0/2552811758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45140b2f30 msgr2=0x7f451413a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.505+0000 7f4518a00640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45140b2f30 0x7f451413a6a0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f450000c4b0 tx=0x7f450000c970 comp rx=0 tx=0).stop 2024-03-23T11:31:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.505+0000 7f4518a00640 1 -- 172.21.15.62:0/2552811758 shutdown_connections 2024-03-23T11:31:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.505+0000 7f4518a00640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f44f406a7a0 0x7f44f406cc60 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.505+0000 7f4518a00640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45140b2f30 0x7f451413a6a0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.505+0000 7f4518a00640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4514143dd0 0x7f451406c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.505+0000 7f4518a00640 1 --2- 172.21.15.62:0/2552811758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4514070090 0x7f45141f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:17.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.505+0000 7f4518a00640 1 -- 172.21.15.62:0/2552811758 >> 172.21.15.62:0/2552811758 conn(0x7f4514066050 msgr2=0x7f45140acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:17.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.505+0000 7f4518a00640 1 -- 172.21.15.62:0/2552811758 shutdown_connections 2024-03-23T11:31:17.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:17.505+0000 7f4518a00640 1 -- 172.21.15.62:0/2552811758 wait complete. 2024-03-23T11:31:17.513 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:31:17.513 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:31:17.637 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:31:17.753 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:31:17.753 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:31:17.753 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:17.811 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:31:17.812 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:31:17.812 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:17.867 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:31:17.868 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:31:17.868 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:17.927 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:31:17.928 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:31:17.928 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:17.932 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:31:17.938 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:31:17.984 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:31:17.985 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T11:31:17.985 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:31:18.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.196+0000 7f6562800640 1 Processor -- start 2024-03-23T11:31:18.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.196+0000 7f6562800640 1 -- start start 2024-03-23T11:31:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.197+0000 7f6562800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f655c06c300 0x7f655c0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.197+0000 7f6562800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f655c0cafa0 0x7f655c0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.198+0000 7f6562800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f655c06c6e0 0x7f655c06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.198+0000 7f6562800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f655c0cbc60 con 0x7f655c06c6e0 2024-03-23T11:31:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.198+0000 7f6562800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f655c0cbf40 con 0x7f655c0cafa0 2024-03-23T11:31:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.198+0000 7f6562800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655c0cc220 con 0x7f655c06c300 2024-03-23T11:31:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.198+0000 7f655be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f655c06c6e0 0x7f655c06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.198+0000 7f655be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f655c06c6e0 0x7f655c06cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47892/0 (socket says 172.21.15.62:47892) 2024-03-23T11:31:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.198+0000 7f655be00640 1 -- 172.21.15.62:0/3352866104 learned_addr learned my addr 172.21.15.62:0/3352866104 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.198+0000 7f6553e00640 1 --2- 172.21.15.62:0/3352866104 >> v2:172.21.15.120:3300/0 conn(0x7f655c0cafa0 0x7f655c0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.198+0000 7f655b400640 1 --2- 172.21.15.62:0/3352866104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f655c06c300 0x7f655c0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.198+0000 7f655be00640 1 -- 172.21.15.62:0/3352866104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f655c06c300 msgr2=0x7f655c0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.198+0000 7f655be00640 1 --2- 172.21.15.62:0/3352866104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f655c06c300 0x7f655c0ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.199+0000 7f655be00640 1 -- 172.21.15.62:0/3352866104 >> v2:172.21.15.120:3300/0 conn(0x7f655c0cafa0 msgr2=0x7f655c0cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.199+0000 7f655be00640 1 --2- 172.21.15.62:0/3352866104 >> v2:172.21.15.120:3300/0 conn(0x7f655c0cafa0 0x7f655c0cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.199+0000 7f655be00640 1 -- 172.21.15.62:0/3352866104 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655c0cc500 con 0x7f655c06c6e0 2024-03-23T11:31:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.200+0000 7f655be00640 1 --2- 172.21.15.62:0/3352866104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f655c06c6e0 0x7f655c06cb70 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f655c0738b0 tx=0x7f655402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d77c0b18c3a9266 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.200+0000 7f655aa00640 1 -- 172.21.15.62:0/3352866104 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655402ca70 con 0x7f655c06c6e0 2024-03-23T11:31:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.200+0000 7f655aa00640 1 -- 172.21.15.62:0/3352866104 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655402cbd0 con 0x7f655c06c6e0 2024-03-23T11:31:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.200+0000 7f655aa00640 1 -- 172.21.15.62:0/3352866104 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65540343f0 con 0x7f655c06c6e0 2024-03-23T11:31:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.201+0000 7f6562800640 1 -- 172.21.15.62:0/3352866104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f655c06c6e0 msgr2=0x7f655c06cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.201+0000 7f6562800640 1 --2- 172.21.15.62:0/3352866104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f655c06c6e0 0x7f655c06cb70 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f655c0738b0 tx=0x7f655402bed0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.201+0000 7f6562800640 1 -- 172.21.15.62:0/3352866104 shutdown_connections 2024-03-23T11:31:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.201+0000 7f6562800640 1 --2- 172.21.15.62:0/3352866104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f655c06c6e0 0x7f655c06cb70 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.201+0000 7f6562800640 1 --2- 172.21.15.62:0/3352866104 >> v2:172.21.15.120:3300/0 conn(0x7f655c0cafa0 0x7f655c0cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.201+0000 7f6562800640 1 --2- 172.21.15.62:0/3352866104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f655c06c300 0x7f655c0ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.201+0000 7f6562800640 1 -- 172.21.15.62:0/3352866104 >> 172.21.15.62:0/3352866104 conn(0x7f655c0b02e0 msgr2=0x7f655c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.201+0000 7f6562800640 1 -- 172.21.15.62:0/3352866104 shutdown_connections 2024-03-23T11:31:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.201+0000 7f6562800640 1 -- 172.21.15.62:0/3352866104 wait complete. 2024-03-23T11:31:18.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.203+0000 7f6562800640 1 Processor -- start 2024-03-23T11:31:18.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.204+0000 7f6562800640 1 -- start start 2024-03-23T11:31:18.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.205+0000 7f6562800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f655c06c300 0x7f655c155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.205+0000 7f6562800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f655c06c6e0 0x7f655c1536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.205+0000 7f6562800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655c0cafa0 0x7f655c153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.205+0000 7f6562800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c1f3c10 con 0x7f655c06c300 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.205+0000 7f6562800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c1f3000 con 0x7f655c0cafa0 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.205+0000 7f6562800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655c1f32e0 con 0x7f655c06c6e0 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.205+0000 7f655b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f655c06c300 0x7f655c155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.205+0000 7f655b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f655c06c300 0x7f655c155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47896/0 (socket says 172.21.15.62:47896) 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.205+0000 7f655b400640 1 -- 172.21.15.62:0/4105155943 learned_addr learned my addr 172.21.15.62:0/4105155943 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f6553e00640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f655c06c6e0 0x7f655c1536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f655b400640 1 -- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f655c06c6e0 msgr2=0x7f655c1536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f655b400640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f655c06c6e0 0x7f655c1536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f655b400640 1 -- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655c0cafa0 msgr2=0x7f655c153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f655b400640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655c0cafa0 0x7f655c153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f655b400640 1 -- 172.21.15.62:0/4105155943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6554009b30 con 0x7f655c06c300 2024-03-23T11:31:18.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f6553e00640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f655c06c6e0 0x7f655c1536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:18.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f655b400640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f655c06c300 0x7f655c155ce0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f654c00d800 tx=0x7f654c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:18.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f6553400640 1 -- 172.21.15.62:0/4105155943 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c0096a0 con 0x7f655c06c300 2024-03-23T11:31:18.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f6553400640 1 -- 172.21.15.62:0/4105155943 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654c009800 con 0x7f655c06c300 2024-03-23T11:31:18.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f6553400640 1 -- 172.21.15.62:0/4105155943 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c009b90 con 0x7f655c06c300 2024-03-23T11:31:18.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.206+0000 7f6562800640 1 -- 172.21.15.62:0/4105155943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655c1f35c0 con 0x7f655c06c300 2024-03-23T11:31:18.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.207+0000 7f6562800640 1 -- 172.21.15.62:0/4105155943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655c13d260 con 0x7f655c06c300 2024-03-23T11:31:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.208+0000 7f6562800640 1 -- 172.21.15.62:0/4105155943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f655c0ce100 con 0x7f655c06c300 2024-03-23T11:31:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.209+0000 7f6553400640 1 -- 172.21.15.62:0/4105155943 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f654c009cf0 con 0x7f655c06c300 2024-03-23T11:31:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.210+0000 7f6553400640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f653806a7a0 0x7f653806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.210+0000 7f6553400640 1 -- 172.21.15.62:0/4105155943 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f654c089200 con 0x7f655c06c300 2024-03-23T11:31:18.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.210+0000 7f6553e00640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f653806a7a0 0x7f653806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.212+0000 7f6553e00640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f653806a7a0 0x7f653806cc60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6544006fd0 tx=0x7f6544008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:18.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.215+0000 7f6553400640 1 -- 172.21.15.62:0/4105155943 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f654c0548c0 con 0x7f655c06c300 2024-03-23T11:31:18.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.454+0000 7f6562800640 1 -- 172.21.15.62:0/4105155943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f655c066c90 con 0x7f655c06c300 2024-03-23T11:31:18.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.456+0000 7f6553400640 1 -- 172.21.15.62:0/4105155943 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f654c00b440 con 0x7f655c06c300 2024-03-23T11:31:18.446 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:18.446 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6598,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/2630459054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6833","nonce":2630459054},{"type":"v1","addr":"172.21.15.62:6835","nonce":2630459054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2384680918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2384680918},{"type":"v1","addr":"172.21.15.120:6836","nonce":2384680918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6610,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1015778684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1015778684},{"type":"v1","addr":"172.21.15.120:6837","nonce":1015778684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6834/4217933052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":4217933052},{"type":"v1","addr":"172.21.15.62:6834","nonce":4217933052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":86}],"filesystems":[]} 2024-03-23T11:31:18.446 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 88 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.458+0000 7f6562800640 1 -- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f653806a7a0 msgr2=0x7f653806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.458+0000 7f6562800640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f653806a7a0 0x7f653806cc60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6544006fd0 tx=0x7f6544008040 comp rx=0 tx=0).stop 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.458+0000 7f6562800640 1 -- 172.21.15.62:0/4105155943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f655c06c300 msgr2=0x7f655c155ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.458+0000 7f6562800640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f655c06c300 0x7f655c155ce0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f654c00d800 tx=0x7f654c00dcc0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.458+0000 7f6562800640 1 -- 172.21.15.62:0/4105155943 shutdown_connections 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.458+0000 7f6562800640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f653806a7a0 0x7f653806cc60 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.458+0000 7f6562800640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655c0cafa0 0x7f655c153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.458+0000 7f6562800640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f655c06c6e0 0x7f655c1536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.458+0000 7f6562800640 1 --2- 172.21.15.62:0/4105155943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f655c06c300 0x7f655c155ce0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.459+0000 7f6562800640 1 -- 172.21.15.62:0/4105155943 >> 172.21.15.62:0/4105155943 conn(0x7f655c0b02e0 msgr2=0x7f655c0af590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.459+0000 7f6562800640 1 -- 172.21.15.62:0/4105155943 shutdown_connections 2024-03-23T11:31:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.459+0000 7f6562800640 1 -- 172.21.15.62:0/4105155943 wait complete. 2024-03-23T11:31:18.468 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T11:31:18.468 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T11:31:18.469 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T11:31:18.469 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:31:18.469 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T11:31:18.469 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T11:31:18.469 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T11:31:18.469 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:31:18.469 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T11:31:18.469 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T11:31:18.470 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T11:31:18.470 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:31:18.470 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T11:31:18.470 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T11:31:18.470 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T11:31:18.470 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:31:18.488 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T11:31:18.488 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T11:31:18.489 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T11:31:18.489 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T11:31:18.491 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T11:31:18.491 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T11:31:18.492 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T11:31:18.492 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T11:31:18.534 INFO:tasks.ceph.mds.a:Started 2024-03-23T11:31:18.534 DEBUG:teuthology.parallel:result is None 2024-03-23T11:31:18.538 INFO:tasks.ceph.mds.c:Started 2024-03-23T11:31:18.538 DEBUG:teuthology.parallel:result is None 2024-03-23T11:31:18.539 INFO:tasks.ceph.mds.b:Started 2024-03-23T11:31:18.539 DEBUG:teuthology.parallel:result is None 2024-03-23T11:31:18.544 INFO:tasks.ceph.mds.d:Started 2024-03-23T11:31:18.544 DEBUG:teuthology.parallel:result is None 2024-03-23T11:31:18.545 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T11:31:18.686 INFO:tasks.ceph.mds.c.smithi062.stdout:starting mds.c at 2024-03-23T11:31:18.688 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:31:18.735 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-23T11:31:18.735 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-23T11:31:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.738+0000 7fafee200640 1 Processor -- start 2024-03-23T11:31:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.738+0000 7fafee200640 1 -- start start 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafee200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe815c960 0x7fafe80cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafee200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fafe80cd260 0x7fafe80d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafee200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fafe80d11e0 0x7fafe80b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafee200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe80d1650 con 0x7fafe80d11e0 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafee200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe80d0810 con 0x7fafe80cd260 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafee200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80cd9d0 con 0x7fafe815c960 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafed800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fafe80d11e0 0x7fafe80b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafece00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe815c960 0x7fafe80cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafe7e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fafe80cd260 0x7fafe80d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafece00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe815c960 0x7fafe80cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:45690/0 (socket says 172.21.15.62:45690) 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafece00640 1 -- 172.21.15.62:0/3134581636 learned_addr learned my addr 172.21.15.62:0/3134581636 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.739+0000 7fafed800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fafe80d11e0 0x7fafe80b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47952/0 (socket says 172.21.15.62:47952) 2024-03-23T11:31:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.740+0000 7fafed800640 1 -- 172.21.15.62:0/3134581636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe815c960 msgr2=0x7fafe80cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.740+0000 7fafed800640 1 --2- 172.21.15.62:0/3134581636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe815c960 0x7fafe80cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.740+0000 7fafed800640 1 -- 172.21.15.62:0/3134581636 >> v2:172.21.15.120:3300/0 conn(0x7fafe80cd260 msgr2=0x7fafe80d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.740+0000 7fafed800640 1 --2- 172.21.15.62:0/3134581636 >> v2:172.21.15.120:3300/0 conn(0x7fafe80cd260 0x7fafe80d0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.740+0000 7fafed800640 1 -- 172.21.15.62:0/3134581636 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe80cdcb0 con 0x7fafe80d11e0 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.741+0000 7fafed800640 1 --2- 172.21.15.62:0/3134581636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafe80d11e0 0x7fafe80b3310 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fafe8066520 tx=0x7fafe002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18bd02abdf3a546 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.744+0000 7fafe7400640 1 -- 172.21.15.62:0/3134581636 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe002cc60 con 0x7fafe80d11e0 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.744+0000 7fafe7400640 1 -- 172.21.15.62:0/3134581636 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe002cdc0 con 0x7fafe80d11e0 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.744+0000 7fafe7400640 1 -- 172.21.15.62:0/3134581636 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0034660 con 0x7fafe80d11e0 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.744+0000 7fafee200640 1 -- 172.21.15.62:0/3134581636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafe80d11e0 msgr2=0x7fafe80b3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.744+0000 7fafee200640 1 --2- 172.21.15.62:0/3134581636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafe80d11e0 0x7fafe80b3310 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fafe8066520 tx=0x7fafe002beb0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.745+0000 7fafee200640 1 -- 172.21.15.62:0/3134581636 shutdown_connections 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.745+0000 7fafee200640 1 --2- 172.21.15.62:0/3134581636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafe80d11e0 0x7fafe80b3310 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.745+0000 7fafee200640 1 --2- 172.21.15.62:0/3134581636 >> v2:172.21.15.120:3300/0 conn(0x7fafe80cd260 0x7fafe80d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.745+0000 7fafee200640 1 --2- 172.21.15.62:0/3134581636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe815c960 0x7fafe80cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.745+0000 7fafee200640 1 -- 172.21.15.62:0/3134581636 >> 172.21.15.62:0/3134581636 conn(0x7fafe8089f60 msgr2=0x7fafe80d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.745+0000 7fafee200640 1 -- 172.21.15.62:0/3134581636 shutdown_connections 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.745+0000 7fafee200640 1 -- 172.21.15.62:0/3134581636 wait complete. 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.746+0000 7fafee200640 1 Processor -- start 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.747+0000 7fafee200640 1 -- start start 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.747+0000 7fafee200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe80cd260 0x7fafe815f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.747+0000 7fafee200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafe80d11e0 0x7fafe8159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.747+0000 7fafee200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafe815c960 0x7fafe80cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.747+0000 7fafee200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe81f4220 con 0x7fafe815c960 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.747+0000 7fafee200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe815f760 con 0x7fafe80d11e0 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.747+0000 7fafee200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe81f3610 con 0x7fafe80cd260 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.748+0000 7fafed800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafe815c960 0x7fafe80cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.748+0000 7fafece00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe80cd260 0x7fafe815f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.748+0000 7fafe7e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafe80d11e0 0x7fafe8159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.748+0000 7fafece00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe80cd260 0x7fafe815f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:45698/0 (socket says 172.21.15.62:45698) 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.748+0000 7fafece00640 1 -- 172.21.15.62:0/1609750304 learned_addr learned my addr 172.21.15.62:0/1609750304 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.748+0000 7fafece00640 1 -- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafe80d11e0 msgr2=0x7fafe8159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.748+0000 7fafece00640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafe80d11e0 0x7fafe8159bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.748+0000 7fafece00640 1 -- 172.21.15.62:0/1609750304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafe815c960 msgr2=0x7fafe80cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.748+0000 7fafece00640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafe815c960 0x7fafe80cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.748+0000 7fafece00640 1 -- 172.21.15.62:0/1609750304 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe0009a70 con 0x7fafe80cd260 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.748+0000 7fafed800640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafe815c960 0x7fafe80cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.749+0000 7fafece00640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe80cd260 0x7fafe815f220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009fd0 tx=0x7fafdc00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.749+0000 7fafe4c00640 1 -- 172.21.15.62:0/1609750304 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc00c940 con 0x7fafe80cd260 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.749+0000 7fafe4c00640 1 -- 172.21.15.62:0/1609750304 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafdc00caa0 con 0x7fafe80cd260 2024-03-23T11:31:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.749+0000 7fafe4c00640 1 -- 172.21.15.62:0/1609750304 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc00ce30 con 0x7fafe80cd260 2024-03-23T11:31:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.749+0000 7fafee200640 1 -- 172.21.15.62:0/1609750304 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe81f38f0 con 0x7fafe80cd260 2024-03-23T11:31:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.749+0000 7fafee200640 1 -- 172.21.15.62:0/1609750304 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe81f3eb0 con 0x7fafe80cd260 2024-03-23T11:31:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.750+0000 7fafee200640 1 -- 172.21.15.62:0/1609750304 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe8066bb0 con 0x7fafe80cd260 2024-03-23T11:31:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.751+0000 7fafe4c00640 1 -- 172.21.15.62:0/1609750304 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fafdc013470 con 0x7fafe80cd260 2024-03-23T11:31:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.752+0000 7fafe4c00640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fafc806a7a0 0x7fafc806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.752+0000 7fafe4c00640 1 -- 172.21.15.62:0/1609750304 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fafdc088fd0 con 0x7fafe80cd260 2024-03-23T11:31:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.752+0000 7fafe7e00640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fafc806a7a0 0x7fafc806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.753+0000 7fafe7e00640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fafc806a7a0 0x7fafc806cc60 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fafe815fcb0 tx=0x7fafd4009110 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:18.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.757+0000 7fafe4c00640 1 -- 172.21.15.62:0/1609750304 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fafdc0546a0 con 0x7fafe80cd260 2024-03-23T11:31:18.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:18.968+0000 7fafee200640 1 -- 172.21.15.62:0/1609750304 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fafe80d0a80 con 0x7fafe80cd260 2024-03-23T11:31:19.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.753+0000 7fafe4c00640 1 -- 172.21.15.62:0/1609750304 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v114) v1 ==== 97+0+0 (secure 0 0 0) 0x7fafdc057cc0 con 0x7fafe80cd260 2024-03-23T11:31:19.743 INFO:teuthology.orchestra.run.smithi062.stderr: removed all blocklist entries 2024-03-23T11:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.756+0000 7fafee200640 1 -- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fafc806a7a0 msgr2=0x7fafc806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.756+0000 7fafee200640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fafc806a7a0 0x7fafc806cc60 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fafe815fcb0 tx=0x7fafd4009110 comp rx=0 tx=0).stop 2024-03-23T11:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.756+0000 7fafee200640 1 -- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe80cd260 msgr2=0x7fafe815f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.756+0000 7fafee200640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe80cd260 0x7fafe815f220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009fd0 tx=0x7fafdc00ed40 comp rx=0 tx=0).stop 2024-03-23T11:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.756+0000 7fafee200640 1 -- 172.21.15.62:0/1609750304 shutdown_connections 2024-03-23T11:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.756+0000 7fafee200640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fafc806a7a0 0x7fafc806cc60 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.756+0000 7fafee200640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fafe815c960 0x7fafe80cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.756+0000 7fafee200640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafe80d11e0 0x7fafe8159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.756+0000 7fafee200640 1 --2- 172.21.15.62:0/1609750304 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fafe80cd260 0x7fafe815f220 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.756+0000 7fafee200640 1 -- 172.21.15.62:0/1609750304 >> 172.21.15.62:0/1609750304 conn(0x7fafe8089f60 msgr2=0x7fafe815a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.756+0000 7fafee200640 1 -- 172.21.15.62:0/1609750304 shutdown_connections 2024-03-23T11:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.757+0000 7fafee200640 1 -- 172.21.15.62:0/1609750304 wait complete. 2024-03-23T11:31:19.766 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T11:31:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.995+0000 7fc6b7600640 1 Processor -- start 2024-03-23T11:31:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.995+0000 7fc6b7600640 1 -- start start 2024-03-23T11:31:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.997+0000 7fc6b7600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6b006c300 0x7fc6b00ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.997+0000 7fc6b7600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc6b006c6e0 0x7fc6b006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.997+0000 7fc6b7600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b00cbc60 con 0x7fc6b00cafa0 2024-03-23T11:31:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.997+0000 7fc6b7600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b00cbf40 con 0x7fc6b006c6e0 2024-03-23T11:31:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.997+0000 7fc6b7600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b00cc220 con 0x7fc6b006c300 2024-03-23T11:31:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.997+0000 7fc6b5600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc6b006c6e0 0x7fc6b006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.997+0000 7fc6afe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc6b00cafa0 0x7fc6b00cb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:51408/0 (socket says 172.21.15.62:51408) 2024-03-23T11:31:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.997+0000 7fc6afe00640 1 -- 172.21.15.62:0/3707371702 learned_addr learned my addr 172.21.15.62:0/3707371702 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.998+0000 7fc6b4c00640 1 --2- 172.21.15.62:0/3707371702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6b006c300 0x7fc6b00ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.998+0000 7fc6b5600640 1 -- 172.21.15.62:0/3707371702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6b006c300 msgr2=0x7fc6b00ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.998+0000 7fc6b5600640 1 --2- 172.21.15.62:0/3707371702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6b006c300 0x7fc6b00ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.998+0000 7fc6b5600640 1 -- 172.21.15.62:0/3707371702 >> v1:172.21.15.62:6789/0 conn(0x7fc6b00cafa0 legacy=0x7fc6b00cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.998+0000 7fc6b5600640 1 -- 172.21.15.62:0/3707371702 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b00cc500 con 0x7fc6b006c6e0 2024-03-23T11:31:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.998+0000 7fc6af400640 1 -- 172.21.15.62:0/3707371702 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337657983 0 0) 0x7fc6b00cbc60 con 0x7fc6b00cafa0 2024-03-23T11:31:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.998+0000 7fc6b4c00640 1 --2- 172.21.15.62:0/3707371702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6b006c300 0x7fc6b00ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:31:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.999+0000 7fc6b5600640 1 --2- 172.21.15.62:0/3707371702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6b006c6e0 0x7fc6b006cb70 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009d90 tx=0x7fc6a802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993736530ac495b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.999+0000 7fc6af400640 1 -- 172.21.15.62:0/3707371702 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a802cc50 con 0x7fc6b006c6e0 2024-03-23T11:31:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.999+0000 7fc6af400640 1 -- 172.21.15.62:0/3707371702 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a802cdb0 con 0x7fc6b006c6e0 2024-03-23T11:31:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:19.999+0000 7fc6af400640 1 -- 172.21.15.62:0/3707371702 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8034660 con 0x7fc6b006c6e0 2024-03-23T11:31:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.000+0000 7fc6b7600640 1 -- 172.21.15.62:0/3707371702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6b006c6e0 msgr2=0x7fc6b006cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.000+0000 7fc6b7600640 1 --2- 172.21.15.62:0/3707371702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6b006c6e0 0x7fc6b006cb70 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009d90 tx=0x7fc6a802bea0 comp rx=0 tx=0).stop 2024-03-23T11:31:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.000+0000 7fc6b7600640 1 -- 172.21.15.62:0/3707371702 shutdown_connections 2024-03-23T11:31:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.000+0000 7fc6b7600640 1 --2- 172.21.15.62:0/3707371702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6b006c6e0 0x7fc6b006cb70 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.000+0000 7fc6b7600640 1 --2- 172.21.15.62:0/3707371702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6b006c300 0x7fc6b00ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.000+0000 7fc6b7600640 1 -- 172.21.15.62:0/3707371702 >> 172.21.15.62:0/3707371702 conn(0x7fc6b00b02e0 msgr2=0x7fc6b00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.000+0000 7fc6b7600640 1 -- 172.21.15.62:0/3707371702 shutdown_connections 2024-03-23T11:31:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.000+0000 7fc6b7600640 1 -- 172.21.15.62:0/3707371702 wait complete. 2024-03-23T11:31:19.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.002+0000 7fc6b7600640 1 Processor -- start 2024-03-23T11:31:19.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.002+0000 7fc6b7600640 1 -- start start 2024-03-23T11:31:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.003+0000 7fc6b7600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6b006c300 0x7fc6b013ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.003+0000 7fc6b7600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6b006c6e0 0x7fc6b00a8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.003+0000 7fc6b7600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6b00cafa0 0x7fc6b00a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.003+0000 7fc6b7600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b013b320 con 0x7fc6b006c6e0 2024-03-23T11:31:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.003+0000 7fc6b7600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b01f1500 con 0x7fc6b00cafa0 2024-03-23T11:31:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.003+0000 7fc6b7600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b01f17e0 con 0x7fc6b006c300 2024-03-23T11:31:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.003+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6b006c6e0 0x7fc6b00a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.003+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6b006c6e0 0x7fc6b00a8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47972/0 (socket says 172.21.15.62:47972) 2024-03-23T11:31:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.003+0000 7fc6afe00640 1 -- 172.21.15.62:0/1296055636 learned_addr learned my addr 172.21.15.62:0/1296055636 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6b4c00640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6b006c300 0x7fc6b013ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6afe00640 1 -- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6b006c300 msgr2=0x7fc6b013ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6b5600640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6b00cafa0 0x7fc6b00a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6afe00640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6b006c300 0x7fc6b013ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6afe00640 1 -- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6b00cafa0 msgr2=0x7fc6b00a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6afe00640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6b00cafa0 0x7fc6b00a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6afe00640 1 -- 172.21.15.62:0/1296055636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a8009a70 con 0x7fc6b006c6e0 2024-03-23T11:31:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6afe00640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6b006c6e0 0x7fc6b00a8710 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00c430 tx=0x7fc69c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6acc00640 1 -- 172.21.15.62:0/1296055636 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c0074e0 con 0x7fc6b006c6e0 2024-03-23T11:31:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6acc00640 1 -- 172.21.15.62:0/1296055636 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69c007660 con 0x7fc6b006c6e0 2024-03-23T11:31:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6b7600640 1 -- 172.21.15.62:0/1296055636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b01f1ac0 con 0x7fc6b006c6e0 2024-03-23T11:31:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6acc00640 1 -- 172.21.15.62:0/1296055636 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c004a20 con 0x7fc6b006c6e0 2024-03-23T11:31:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.004+0000 7fc6b7600640 1 -- 172.21.15.62:0/1296055636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b014aa70 con 0x7fc6b006c6e0 2024-03-23T11:31:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.006+0000 7fc6b7600640 1 -- 172.21.15.62:0/1296055636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b01f1fe0 con 0x7fc6b006c6e0 2024-03-23T11:31:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.006+0000 7fc6acc00640 1 -- 172.21.15.62:0/1296055636 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc69c004b80 con 0x7fc6b006c6e0 2024-03-23T11:31:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.008+0000 7fc6acc00640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc69006a7d0 0x7fc69006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.008+0000 7fc6acc00640 1 -- 172.21.15.62:0/1296055636 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc69c088910 con 0x7fc6b006c6e0 2024-03-23T11:31:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.009+0000 7fc6b4c00640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc69006a7d0 0x7fc69006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.010+0000 7fc6b4c00640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc69006a7d0 0x7fc69006cc90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400d8b0 tx=0x7fc6a4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:20.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.013+0000 7fc6acc00640 1 -- 172.21.15.62:0/1296055636 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc6b01f1fe0 con 0x7fc6b006c6e0 2024-03-23T11:31:20.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.282+0000 7fc6b7600640 1 -- 172.21.15.62:0/1296055636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc6b01f1fe0 con 0x7fc6b006c6e0 2024-03-23T11:31:20.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.283+0000 7fc6acc00640 1 -- 172.21.15.62:0/1296055636 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc6b01f1fe0 con 0x7fc6b006c6e0 2024-03-23T11:31:20.274 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:20.274 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:31:20.274 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_dump": [ 2024-03-23T11:31:20.274 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.274 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.a", 2024-03-23T11:31:20.274 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lYNxYERAAPVHWwRHM5V7OejL294b2zg==", 2024-03-23T11:31:20.274 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.274 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:31:20.274 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:31:20.274 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:31:20.274 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.b", 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lmOUANBAAifuZ0DUPD9az/mturTndSA==", 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.275 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.c", 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5ltqrjIhAA9+f82fvlrFlIEXqswPE0ig==", 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.d", 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5l7ly4CBAA8+oim0z+bbljfP0VH7XSaw==", 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.276 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.0", 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBiuv5lRfaQBRAAvrtCZXwt00PUQfCqqKBzMQ==", 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.277 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.1", 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBluv5lzwS/IBAAmYU8yMLtCUJNr/AyE0MJDA==", 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.2", 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBouv5lyAy1OhAAJoUsg6DFpEPxAcH5oI2exQ==", 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.278 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.3", 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBsuv5lb0YcGBAA220D0uRhAZpG4w97wqsB5A==", 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.279 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.4", 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBzuv5lC8RIGhAAH+5HaeCgQhrQMrYm9iWaYA==", 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.5", 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB2uv5lYi7ZIhAAMNIm/jPckCAbX9oEx5Li/Q==", 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:31:20.280 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.6", 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB5uv5lgoRtLBAAFT7uJtyqHtHAegOYyDH/WQ==", 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.281 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.7", 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQB9uv5l/dzrCBAA3jsua0LvBPI7I551/rRHrg==", 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.0", 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5ltavQFhAAvV2RWuHOPK2T9Q7gWkYA4w==", 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.282 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow r", 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.1", 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5luQ/zLRAAwjDtV64TCKnQliPq4iUAYg==", 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.283 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.2", 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lDRG1IhAAjVaXRR30G1985oMou7Ambg==", 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.3", 2024-03-23T11:31:20.284 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBduv5lomZyOBAA8BMIBD+Zhtp6Fa2RcEXCJg==", 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.admin", 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lNtZHBhAAhiAC1rdmANClOSwzrX9iSQ==", 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-03-23T11:31:20.285 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.x", 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBcuv5lT6+KBRAAl5TZsJGNr4gAB2cEDZo/hQ==", 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:20.286 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.y", 2024-03-23T11:31:20.287 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBbuv5lqnG8NRAAvyswMz5kpW0rGjzWYRTAAg==", 2024-03-23T11:31:20.287 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-03-23T11:31:20.287 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-03-23T11:31:20.287 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-03-23T11:31:20.287 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-03-23T11:31:20.287 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.287 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:20.287 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-03-23T11:31:20.287 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:31:20.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 -- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc69006a7d0 msgr2=0x7fc69006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc69006a7d0 0x7fc69006cc90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400d8b0 tx=0x7fc6a4008040 comp rx=0 tx=0).stop 2024-03-23T11:31:20.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 -- 172.21.15.62:0/1296055636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6b006c6e0 msgr2=0x7fc6b00a8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6b006c6e0 0x7fc6b00a8710 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00c430 tx=0x7fc69c00c8f0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 -- 172.21.15.62:0/1296055636 shutdown_connections 2024-03-23T11:31:20.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc69006a7d0 0x7fc69006cc90 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6b006c6e0 0x7fc6b00a8710 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6b00cafa0 0x7fc6b00a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 --2- 172.21.15.62:0/1296055636 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6b006c300 0x7fc6b013ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 -- 172.21.15.62:0/1296055636 >> 172.21.15.62:0/1296055636 conn(0x7fc6b00b02e0 msgr2=0x7fc6b015a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:20.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 -- 172.21.15.62:0/1296055636 shutdown_connections 2024-03-23T11:31:20.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.286+0000 7fc6b7600640 1 -- 172.21.15.62:0/1296055636 wait complete. 2024-03-23T11:31:20.293 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:31:20.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.518+0000 7f69e3c00640 1 Processor -- start 2024-03-23T11:31:20.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.519+0000 7f69e3c00640 1 -- start start 2024-03-23T11:31:20.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.520+0000 7f69e3c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f69dc0cd7c0 0x7f69dc0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.520+0000 7f69e3c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f69dc06dca0 0x7f69dc06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.520+0000 7f69e3c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0cad20 con 0x7f69dc0cdba0 2024-03-23T11:31:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.520+0000 7f69e3c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69dc06e130 con 0x7f69dc06dca0 2024-03-23T11:31:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.520+0000 7f69e3c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69dc15a8a0 con 0x7f69dc0cd7c0 2024-03-23T11:31:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.521+0000 7f69e1c00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f69dc0cdba0 0x7f69dc0cb490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:51410/0 (socket says 172.21.15.62:51410) 2024-03-23T11:31:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.521+0000 7f69e1c00640 1 -- 172.21.15.62:0/3899495740 learned_addr learned my addr 172.21.15.62:0/3899495740 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.521+0000 7f69dbe00640 1 --2- 172.21.15.62:0/3899495740 >> v2:172.21.15.62:3300/0 conn(0x7f69dc06dca0 0x7f69dc06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.521+0000 7f69e1200640 1 --2- 172.21.15.62:0/3899495740 >> v2:172.21.15.120:3300/0 conn(0x7f69dc0cd7c0 0x7f69dc0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.521+0000 7f69dbe00640 1 -- 172.21.15.62:0/3899495740 >> v2:172.21.15.120:3300/0 conn(0x7f69dc0cd7c0 msgr2=0x7f69dc0b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:20.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.521+0000 7f69dbe00640 1 --2- 172.21.15.62:0/3899495740 >> v2:172.21.15.120:3300/0 conn(0x7f69dc0cd7c0 0x7f69dc0b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.521+0000 7f69dbe00640 1 -- 172.21.15.62:0/3899495740 >> v1:172.21.15.62:6789/0 conn(0x7f69dc0cdba0 legacy=0x7f69dc0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.521+0000 7f69dbe00640 1 -- 172.21.15.62:0/3899495740 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc15ab80 con 0x7f69dc06dca0 2024-03-23T11:31:20.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.522+0000 7f69dbe00640 1 --2- 172.21.15.62:0/3899495740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69dc06dca0 0x7f69dc06c300 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f69dc06ac20 tx=0x7f69c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e128fd890f9a1db2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:20.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.522+0000 7f69db400640 1 -- 172.21.15.62:0/3899495740 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c802e070 con 0x7f69dc06dca0 2024-03-23T11:31:20.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.523+0000 7f69db400640 1 -- 172.21.15.62:0/3899495740 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c80315a0 con 0x7f69dc06dca0 2024-03-23T11:31:20.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.523+0000 7f69db400640 1 -- 172.21.15.62:0/3899495740 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8031930 con 0x7f69dc06dca0 2024-03-23T11:31:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.523+0000 7f69e3c00640 1 -- 172.21.15.62:0/3899495740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69dc06dca0 msgr2=0x7f69dc06c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.523+0000 7f69e3c00640 1 --2- 172.21.15.62:0/3899495740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69dc06dca0 0x7f69dc06c300 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f69dc06ac20 tx=0x7f69c802b840 comp rx=0 tx=0).stop 2024-03-23T11:31:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.523+0000 7f69e3c00640 1 -- 172.21.15.62:0/3899495740 shutdown_connections 2024-03-23T11:31:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.523+0000 7f69e3c00640 1 --2- 172.21.15.62:0/3899495740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69dc06dca0 0x7f69dc06c300 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.523+0000 7f69e3c00640 1 --2- 172.21.15.62:0/3899495740 >> v2:172.21.15.120:3300/0 conn(0x7f69dc0cd7c0 0x7f69dc0b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.523+0000 7f69e3c00640 1 -- 172.21.15.62:0/3899495740 >> 172.21.15.62:0/3899495740 conn(0x7f69dc0b02e0 msgr2=0x7f69dc0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.523+0000 7f69e3c00640 1 -- 172.21.15.62:0/3899495740 shutdown_connections 2024-03-23T11:31:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.523+0000 7f69e3c00640 1 -- 172.21.15.62:0/3899495740 wait complete. 2024-03-23T11:31:20.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.525+0000 7f69e3c00640 1 Processor -- start 2024-03-23T11:31:20.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.525+0000 7f69e3c00640 1 -- start start 2024-03-23T11:31:20.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.526+0000 7f69e3c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69dc0cd7c0 0x7f69dc13d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.526+0000 7f69e3c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69dc0cdba0 0x7f69dc147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.526+0000 7f69e3c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69dc0acb40 0x7f69dc14be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.526+0000 7f69e3c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc13df00 con 0x7f69dc0cdba0 2024-03-23T11:31:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.526+0000 7f69e3c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc154c80 con 0x7f69dc0cd7c0 2024-03-23T11:31:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.526+0000 7f69e3c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc154f60 con 0x7f69dc0acb40 2024-03-23T11:31:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.526+0000 7f69dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69dc0cdba0 0x7f69dc147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.526+0000 7f69dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69dc0cdba0 0x7f69dc147730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47992/0 (socket says 172.21.15.62:47992) 2024-03-23T11:31:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.526+0000 7f69dbe00640 1 -- 172.21.15.62:0/1474848154 learned_addr learned my addr 172.21.15.62:0/1474848154 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.527+0000 7f69e1200640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69dc0cd7c0 0x7f69dc13d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.527+0000 7f69e1c00640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69dc0acb40 0x7f69dc14be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.527+0000 7f69dbe00640 1 -- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69dc0acb40 msgr2=0x7f69dc14be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.527+0000 7f69dbe00640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69dc0acb40 0x7f69dc14be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.527+0000 7f69dbe00640 1 -- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69dc0cd7c0 msgr2=0x7f69dc13d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.527+0000 7f69dbe00640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69dc0cd7c0 0x7f69dc13d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.527+0000 7f69dbe00640 1 -- 172.21.15.62:0/1474848154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c8009400 con 0x7f69dc0cdba0 2024-03-23T11:31:20.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.527+0000 7f69dbe00640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69dc0cdba0 0x7f69dc147730 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009890 tx=0x7f69c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:20.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.528+0000 7f69d8c00640 1 -- 172.21.15.62:0/1474848154 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8032840 con 0x7f69dc0cdba0 2024-03-23T11:31:20.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.528+0000 7f69d8c00640 1 -- 172.21.15.62:0/1474848154 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c80329a0 con 0x7f69dc0cdba0 2024-03-23T11:31:20.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.528+0000 7f69d8c00640 1 -- 172.21.15.62:0/1474848154 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8032d50 con 0x7f69dc0cdba0 2024-03-23T11:31:20.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.528+0000 7f69e3c00640 1 -- 172.21.15.62:0/1474848154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc155240 con 0x7f69dc0cdba0 2024-03-23T11:31:20.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.528+0000 7f69e3c00640 1 -- 172.21.15.62:0/1474848154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc165b10 con 0x7f69dc0cdba0 2024-03-23T11:31:20.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.529+0000 7f69e3c00640 1 -- 172.21.15.62:0/1474848154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69dc165d00 con 0x7f69dc0cdba0 2024-03-23T11:31:20.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.530+0000 7f69d8c00640 1 -- 172.21.15.62:0/1474848154 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f69c803b3f0 con 0x7f69dc0cdba0 2024-03-23T11:31:20.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.531+0000 7f69d8c00640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69bc06a7a0 0x7f69bc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.531+0000 7f69d8c00640 1 -- 172.21.15.62:0/1474848154 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f69c80aaa10 con 0x7f69dc0cdba0 2024-03-23T11:31:20.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.532+0000 7f69e1200640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69bc06a7a0 0x7f69bc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.533+0000 7f69e1200640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69bc06a7a0 0x7f69bc06cc60 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f69d0006fd0 tx=0x7f69d0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:20.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.536+0000 7f69d8c00640 1 -- 172.21.15.62:0/1474848154 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69c8076170 con 0x7f69dc0cdba0 2024-03-23T11:31:20.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.733+0000 7f69e3c00640 1 -- 172.21.15.62:0/1474848154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69dc066c90 con 0x7f69dc0cdba0 2024-03-23T11:31:20.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.735+0000 7f69d8c00640 1 -- 172.21.15.62:0/1474848154 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+12399 (secure 0 0 0) 0x7f69c8079790 con 0x7f69dc0cdba0 2024-03-23T11:31:20.725 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:20.725 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":114,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:31:19.740697+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:31:20.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.737+0000 7f69e3c00640 1 -- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69bc06a7a0 msgr2=0x7f69bc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.737+0000 7f69e3c00640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69bc06a7a0 0x7f69bc06cc60 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f69d0006fd0 tx=0x7f69d0008040 comp rx=0 tx=0).stop 2024-03-23T11:31:20.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.737+0000 7f69e3c00640 1 -- 172.21.15.62:0/1474848154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69dc0cdba0 msgr2=0x7f69dc147730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.737+0000 7f69e3c00640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69dc0cdba0 0x7f69dc147730 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009890 tx=0x7f69c80046c0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.737+0000 7f69e3c00640 1 -- 172.21.15.62:0/1474848154 shutdown_connections 2024-03-23T11:31:20.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.737+0000 7f69e3c00640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69bc06a7a0 0x7f69bc06cc60 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.737+0000 7f69e3c00640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69dc0cdba0 0x7f69dc147730 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.737+0000 7f69e3c00640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69dc0acb40 0x7f69dc14be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.737+0000 7f69e3c00640 1 --2- 172.21.15.62:0/1474848154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69dc0cd7c0 0x7f69dc13d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.737+0000 7f69e3c00640 1 -- 172.21.15.62:0/1474848154 >> 172.21.15.62:0/1474848154 conn(0x7f69dc0b02e0 msgr2=0x7f69dc0b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:20.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.737+0000 7f69e3c00640 1 -- 172.21.15.62:0/1474848154 shutdown_connections 2024-03-23T11:31:20.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.738+0000 7f69e3c00640 1 -- 172.21.15.62:0/1474848154 wait complete. 2024-03-23T11:31:20.747 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:31:20.747 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:31:20.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.970+0000 7f7fb0e00640 1 Processor -- start 2024-03-23T11:31:20.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.970+0000 7f7fb0e00640 1 -- start start 2024-03-23T11:31:20.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.972+0000 7f7fb0e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7fac15ac50 0x7f7fac0989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.972+0000 7f7fb0e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac099360 con 0x7f7fac098f10 2024-03-23T11:31:20.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.972+0000 7f7fb0e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac0cf320 con 0x7f7fac15a870 2024-03-23T11:31:20.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.972+0000 7f7fb0e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fac0cf7b0 con 0x7f7fac15ac50 2024-03-23T11:31:20.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.972+0000 7f7faa800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7fac098f10 0x7f7fac0b1480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:51418/0 (socket says 172.21.15.62:51418) 2024-03-23T11:31:20.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.972+0000 7f7faa800640 1 -- 172.21.15.62:0/3421376968 learned_addr learned my addr 172.21.15.62:0/3421376968 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:20.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.972+0000 7f7fa9400640 1 --2- 172.21.15.62:0/3421376968 >> v2:172.21.15.120:3300/0 conn(0x7f7fac15ac50 0x7f7fac0989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254944663 0 0) 0x7f7fa4001110 con 0x7f7fac098f10 2024-03-23T11:31:20.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f90003520 con 0x7f7fac098f10 2024-03-23T11:31:20.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 820192418 0 0) 0x7f7f90003520 con 0x7f7fac098f10 2024-03-23T11:31:20.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 >> v2:172.21.15.120:3300/0 conn(0x7f7fac15ac50 msgr2=0x7f7fac0989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 --2- 172.21.15.62:0/3421376968 >> v2:172.21.15.120:3300/0 conn(0x7f7fac15ac50 0x7f7fac0989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 >> v1:172.21.15.120:6789/0 conn(0x7f7fac15a870 legacy=0x7f7fac0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac0ce710 con 0x7f7fac098f10 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7fa4001800 con 0x7f7fac098f10 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 >> v1:172.21.15.62:6789/0 conn(0x7f7fac098f10 legacy=0x7f7fac0b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 --2- 172.21.15.62:0/3421376968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f90003970 0x7f7f90003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 --2- 172.21.15.62:0/3421376968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f900047c0 0x7f7f90004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 --2- 172.21.15.62:0/3421376968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 0x7f7f9000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac0cf7b0 con 0x7f7f90005180 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9000c020 con 0x7f7f90003970 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.973+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9000c1e0 con 0x7f7f900047c0 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.974+0000 7f7faa800640 1 --2- 172.21.15.62:0/3421376968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 0x7f7f9000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.974+0000 7f7fa9e00640 1 --2- 172.21.15.62:0/3421376968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f90003970 0x7f7f90003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.974+0000 7f7faa800640 1 -- 172.21.15.62:0/3421376968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f900047c0 msgr2=0x7f7f90004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:20.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.974+0000 7f7faa800640 1 --2- 172.21.15.62:0/3421376968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f900047c0 0x7f7f90004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.974+0000 7f7faa800640 1 -- 172.21.15.62:0/3421376968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f90003970 msgr2=0x7f7f90003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:20.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.974+0000 7f7faa800640 1 --2- 172.21.15.62:0/3421376968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f90003970 0x7f7f90003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.974+0000 7f7faa800640 1 -- 172.21.15.62:0/3421376968 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f90003520 con 0x7f7f90005180 2024-03-23T11:31:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.975+0000 7f7faa800640 1 -- 172.21.15.62:0/3421376968 reap_dead start 2024-03-23T11:31:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.975+0000 7f7faa800640 1 --2- 172.21.15.62:0/3421376968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 0x7f7f9000b980 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400c240 tx=0x7f7f900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77076fddb666b283 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.975+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4005ad0 con 0x7f7f90005180 2024-03-23T11:31:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.975+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa4005c50 con 0x7f7f90005180 2024-03-23T11:31:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.975+0000 7f7fa8a00640 1 -- 172.21.15.62:0/3421376968 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa402c8b0 con 0x7f7f90005180 2024-03-23T11:31:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.976+0000 7f7fb0e00640 1 -- 172.21.15.62:0/3421376968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 msgr2=0x7f7f9000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.976+0000 7f7fb0e00640 1 --2- 172.21.15.62:0/3421376968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 0x7f7f9000b980 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400c240 tx=0x7f7f900057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.976+0000 7f7fb0e00640 1 -- 172.21.15.62:0/3421376968 shutdown_connections 2024-03-23T11:31:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.976+0000 7f7fb0e00640 1 --2- 172.21.15.62:0/3421376968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 0x7f7f9000b980 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.976+0000 7f7fb0e00640 1 -- 172.21.15.62:0/3421376968 >> 172.21.15.62:0/3421376968 conn(0x7f7fac0b2270 msgr2=0x7f7fac0b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.976+0000 7f7fb0e00640 1 -- 172.21.15.62:0/3421376968 shutdown_connections 2024-03-23T11:31:20.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.976+0000 7f7fb0e00640 1 -- 172.21.15.62:0/3421376968 wait complete. 2024-03-23T11:31:20.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.978+0000 7f7fb0e00640 1 Processor -- start 2024-03-23T11:31:20.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.978+0000 7f7fb0e00640 1 -- start start 2024-03-23T11:31:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.979+0000 7f7fb0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 0x7f7fac14fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.979+0000 7f7fb0e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fac0c4670 0x7f7fac0c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.979+0000 7f7fb0e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fac0c88a0 0x7f7fac0c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.979+0000 7f7fb0e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac0c9260 con 0x7f7f90005180 2024-03-23T11:31:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.979+0000 7f7fb0e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1502e0 con 0x7f7fac0c4670 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.979+0000 7f7fb0e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac0c56d0 con 0x7f7fac0c88a0 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.980+0000 7f7fa9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 0x7f7fac14fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.980+0000 7f7fa9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 0x7f7fac14fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48002/0 (socket says 172.21.15.62:48002) 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.980+0000 7f7fa9e00640 1 -- 172.21.15.62:0/2019052150 learned_addr learned my addr 172.21.15.62:0/2019052150 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.980+0000 7f7fa9400640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fac0c4670 0x7f7fac0c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.980+0000 7f7fa9e00640 1 -- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fac0c88a0 msgr2=0x7f7fac0c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.980+0000 7f7fa9e00640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fac0c88a0 0x7f7fac0c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.980+0000 7f7fa9e00640 1 -- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fac0c4670 msgr2=0x7f7fac0c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.980+0000 7f7fa9e00640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fac0c4670 0x7f7fac0c4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.980+0000 7f7fa9e00640 1 -- 172.21.15.62:0/2019052150 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac0c5110 con 0x7f7f90005180 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.980+0000 7f7fa9400640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fac0c4670 0x7f7fac0c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.980+0000 7f7fa9e00640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 0x7f7fac14fda0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00e850 tx=0x7f7f9c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.981+0000 7f7fa2000640 1 -- 172.21.15.62:0/2019052150 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c003b80 con 0x7f7f90005180 2024-03-23T11:31:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.981+0000 7f7fa2000640 1 -- 172.21.15.62:0/2019052150 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9c003ce0 con 0x7f7f90005180 2024-03-23T11:31:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.981+0000 7f7fa2000640 1 -- 172.21.15.62:0/2019052150 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0052f0 con 0x7f7f90005180 2024-03-23T11:31:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.981+0000 7f7fb0e00640 1 -- 172.21.15.62:0/2019052150 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac0c53f0 con 0x7f7f90005180 2024-03-23T11:31:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.981+0000 7f7fb0e00640 1 -- 172.21.15.62:0/2019052150 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac0b3380 con 0x7f7f90005180 2024-03-23T11:31:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.982+0000 7f7fb0e00640 1 -- 172.21.15.62:0/2019052150 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fac0b3540 con 0x7f7f90005180 2024-03-23T11:31:20.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.983+0000 7f7fa2000640 1 -- 172.21.15.62:0/2019052150 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7f9c005450 con 0x7f7f90005180 2024-03-23T11:31:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.984+0000 7f7fa2000640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f8c06a7d0 0x7f7f8c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.984+0000 7f7fa2000640 1 -- 172.21.15.62:0/2019052150 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7f9c088b30 con 0x7f7f90005180 2024-03-23T11:31:20.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.985+0000 7f7fa9400640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f8c06a7d0 0x7f7f8c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:20.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.986+0000 7f7fa9400640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f8c06a7d0 0x7f7f8c06cc90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0c9af0 tx=0x7f7f98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:20.989+0000 7f7fa2000640 1 -- 172.21.15.62:0/2019052150 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f9c054210 con 0x7f7f90005180 2024-03-23T11:31:21.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.179+0000 7f7fb0e00640 1 -- 172.21.15.62:0/2019052150 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7fac0b2f00 con 0x7f7f90005180 2024-03-23T11:31:21.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.180+0000 7f7fa2000640 1 -- 172.21.15.62:0/2019052150 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7fac0b2f00 con 0x7f7f90005180 2024-03-23T11:31:21.170 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:31:21.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.182+0000 7f7fb0e00640 1 -- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f8c06a7d0 msgr2=0x7f7f8c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.182+0000 7f7fb0e00640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f8c06a7d0 0x7f7f8c06cc90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0c9af0 tx=0x7f7f98008040 comp rx=0 tx=0).stop 2024-03-23T11:31:21.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.182+0000 7f7fb0e00640 1 -- 172.21.15.62:0/2019052150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 msgr2=0x7f7fac14fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.182+0000 7f7fb0e00640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 0x7f7fac14fda0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00e850 tx=0x7f7f9c00ed10 comp rx=0 tx=0).stop 2024-03-23T11:31:21.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.183+0000 7f7fb0e00640 1 -- 172.21.15.62:0/2019052150 shutdown_connections 2024-03-23T11:31:21.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.183+0000 7f7fb0e00640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f8c06a7d0 0x7f7f8c06cc90 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.183+0000 7f7fb0e00640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7fac0c88a0 0x7f7fac0c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.183+0000 7f7fb0e00640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fac0c4670 0x7f7fac0c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.183+0000 7f7fb0e00640 1 --2- 172.21.15.62:0/2019052150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f90005180 0x7f7fac14fda0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.183+0000 7f7fb0e00640 1 -- 172.21.15.62:0/2019052150 >> 172.21.15.62:0/2019052150 conn(0x7f7fac0b2270 msgr2=0x7f7fac099500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:21.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.183+0000 7f7fb0e00640 1 -- 172.21.15.62:0/2019052150 shutdown_connections 2024-03-23T11:31:21.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.183+0000 7f7fb0e00640 1 -- 172.21.15.62:0/2019052150 wait complete. 2024-03-23T11:31:21.190 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T11:31:21.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.412+0000 7fac39000640 1 Processor -- start 2024-03-23T11:31:21.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.412+0000 7fac39000640 1 -- start start 2024-03-23T11:31:21.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.414+0000 7fac39000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fac3415b340 0x7fac3406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:21.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.414+0000 7fac39000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac340a7540 con 0x7fac34140f80 2024-03-23T11:31:21.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.414+0000 7fac39000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3415be80 con 0x7fac3415d370 2024-03-23T11:31:21.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.414+0000 7fac39000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac3406d5c0 con 0x7fac3415b340 2024-03-23T11:31:21.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.414+0000 7fac33400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fac34140f80 0x7fac3415ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:51434/0 (socket says 172.21.15.62:51434) 2024-03-23T11:31:21.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.414+0000 7fac33400640 1 -- 172.21.15.62:0/448121781 learned_addr learned my addr 172.21.15.62:0/448121781 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:21.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.414+0000 7fac33e00640 1 --2- 172.21.15.62:0/448121781 >> v2:172.21.15.62:3300/0 conn(0x7fac3415b340 0x7fac3406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:21.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.414+0000 7fac32a00640 1 -- 172.21.15.62:0/448121781 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291216118 0 0) 0x7fac340a7540 con 0x7fac34140f80 2024-03-23T11:31:21.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.414+0000 7fac32a00640 1 -- 172.21.15.62:0/448121781 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac18003520 con 0x7fac34140f80 2024-03-23T11:31:21.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.415+0000 7fac33e00640 1 -- 172.21.15.62:0/448121781 >> v1:172.21.15.120:6789/0 conn(0x7fac3415d370 legacy=0x7fac3406c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:21.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.415+0000 7fac33e00640 1 -- 172.21.15.62:0/448121781 >> v1:172.21.15.62:6789/0 conn(0x7fac34140f80 legacy=0x7fac3415ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.415+0000 7fac33e00640 1 -- 172.21.15.62:0/448121781 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3406d8a0 con 0x7fac3415b340 2024-03-23T11:31:21.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.416+0000 7fac33e00640 1 --2- 172.21.15.62:0/448121781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3415b340 0x7fac3406cfe0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0097a0 tx=0x7fac2c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce698534a9adc2e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:21.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.416+0000 7fac32a00640 1 -- 172.21.15.62:0/448121781 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c02f810 con 0x7fac3415b340 2024-03-23T11:31:21.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.416+0000 7fac32a00640 1 -- 172.21.15.62:0/448121781 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c02f970 con 0x7fac3415b340 2024-03-23T11:31:21.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.416+0000 7fac32a00640 1 -- 172.21.15.62:0/448121781 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c02fd00 con 0x7fac3415b340 2024-03-23T11:31:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.417+0000 7fac39000640 1 -- 172.21.15.62:0/448121781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3415b340 msgr2=0x7fac3406cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.417+0000 7fac39000640 1 --2- 172.21.15.62:0/448121781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3415b340 0x7fac3406cfe0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0097a0 tx=0x7fac2c02b9f0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.417+0000 7fac39000640 1 -- 172.21.15.62:0/448121781 shutdown_connections 2024-03-23T11:31:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.417+0000 7fac39000640 1 --2- 172.21.15.62:0/448121781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3415b340 0x7fac3406cfe0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.417+0000 7fac39000640 1 -- 172.21.15.62:0/448121781 >> 172.21.15.62:0/448121781 conn(0x7fac340b0100 msgr2=0x7fac340b0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.417+0000 7fac39000640 1 -- 172.21.15.62:0/448121781 shutdown_connections 2024-03-23T11:31:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.417+0000 7fac39000640 1 -- 172.21.15.62:0/448121781 wait complete. 2024-03-23T11:31:21.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.418+0000 7fac39000640 1 Processor -- start 2024-03-23T11:31:21.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.419+0000 7fac39000640 1 -- start start 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac39000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac34140f80 0x7fac34159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac39000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3415b340 0x7fac34162360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac39000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac3415d370 0x7fac341600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac39000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac340b34a0 con 0x7fac3415b340 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac39000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac340b1c50 con 0x7fac3415d370 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac39000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac340b2ee0 con 0x7fac34140f80 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac2be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3415b340 0x7fac34162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac2be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3415b340 0x7fac34162360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48010/0 (socket says 172.21.15.62:48010) 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac2be00640 1 -- 172.21.15.62:0/4111720845 learned_addr learned my addr 172.21.15.62:0/4111720845 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac33e00640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac3415d370 0x7fac341600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac2be00640 1 -- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac34140f80 msgr2=0x7fac34159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac33400640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac34140f80 0x7fac34159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:21.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac2be00640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac34140f80 0x7fac34159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac2be00640 1 -- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac3415d370 msgr2=0x7fac341600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac2be00640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac3415d370 0x7fac341600b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac2be00640 1 -- 172.21.15.62:0/4111720845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c009400 con 0x7fac3415b340 2024-03-23T11:31:21.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.420+0000 7fac33e00640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac3415d370 0x7fac341600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:21.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.421+0000 7fac2be00640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3415b340 0x7fac34162360 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00c430 tx=0x7fac1c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:21.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.421+0000 7fac2b400640 1 -- 172.21.15.62:0/4111720845 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1c0052d0 con 0x7fac3415b340 2024-03-23T11:31:21.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.421+0000 7fac2b400640 1 -- 172.21.15.62:0/4111720845 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1c005430 con 0x7fac3415b340 2024-03-23T11:31:21.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.421+0000 7fac2b400640 1 -- 172.21.15.62:0/4111720845 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1c003830 con 0x7fac3415b340 2024-03-23T11:31:21.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.421+0000 7fac39000640 1 -- 172.21.15.62:0/4111720845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac340b31c0 con 0x7fac3415b340 2024-03-23T11:31:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.421+0000 7fac39000640 1 -- 172.21.15.62:0/4111720845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac341595b0 con 0x7fac3415b340 2024-03-23T11:31:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.423+0000 7fac39000640 1 -- 172.21.15.62:0/4111720845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac34066bb0 con 0x7fac3415b340 2024-03-23T11:31:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.423+0000 7fac2b400640 1 -- 172.21.15.62:0/4111720845 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fac1c003990 con 0x7fac3415b340 2024-03-23T11:31:21.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.425+0000 7fac2b400640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac1006a7d0 0x7fac1006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.425+0000 7fac2b400640 1 -- 172.21.15.62:0/4111720845 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fac1c087ff0 con 0x7fac3415b340 2024-03-23T11:31:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.425+0000 7fac33400640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac1006a7d0 0x7fac1006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.426+0000 7fac33400640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac1006a7d0 0x7fac1006cc90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fac24001560 tx=0x7fac24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:21.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.429+0000 7fac2b400640 1 -- 172.21.15.62:0/4111720845 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac1c053760 con 0x7fac3415b340 2024-03-23T11:31:21.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.626+0000 7fac39000640 1 -- 172.21.15.62:0/4111720845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fac340b1d90 con 0x7fac3415b340 2024-03-23T11:31:21.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.627+0000 7fac2b400640 1 -- 172.21.15.62:0/4111720845 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v114) v1 ==== 84+0+65 (secure 0 0 0) 0x7fac1c017060 con 0x7fac3415b340 2024-03-23T11:31:21.617 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:21.618 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-03-23T11:31:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-03-23T11:31:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "poolnum": 1, 2024-03-23T11:31:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: "poolname": ".mgr" 2024-03-23T11:31:21.618 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:21.618 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-03-23T11:31:21.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 -- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac1006a7d0 msgr2=0x7fac1006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac1006a7d0 0x7fac1006cc90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fac24001560 tx=0x7fac24009040 comp rx=0 tx=0).stop 2024-03-23T11:31:21.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 -- 172.21.15.62:0/4111720845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3415b340 msgr2=0x7fac34162360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3415b340 0x7fac34162360 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00c430 tx=0x7fac1c00c8f0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 -- 172.21.15.62:0/4111720845 shutdown_connections 2024-03-23T11:31:21.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac1006a7d0 0x7fac1006cc90 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3415b340 0x7fac34162360 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac3415d370 0x7fac341600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 --2- 172.21.15.62:0/4111720845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac34140f80 0x7fac34159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 -- 172.21.15.62:0/4111720845 >> 172.21.15.62:0/4111720845 conn(0x7fac340b0100 msgr2=0x7fac340b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:21.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 -- 172.21.15.62:0/4111720845 shutdown_connections 2024-03-23T11:31:21.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.630+0000 7fac39000640 1 -- 172.21.15.62:0/4111720845 wait complete. 2024-03-23T11:31:21.637 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T11:31:21.637 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T11:31:21.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.853+0000 7f2857400640 1 Processor -- start 2024-03-23T11:31:21.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.854+0000 7f2857400640 1 -- start start 2024-03-23T11:31:21.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.856+0000 7f2857400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f28500cf2c0 0x7f28500cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:21.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.856+0000 7f2857400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28500d1320 0x7f28500d1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:21.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.856+0000 7f2857400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28500b58b0 con 0x7f285015ca70 2024-03-23T11:31:21.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.856+0000 7f2857400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28500cfc90 con 0x7f28500cf2c0 2024-03-23T11:31:21.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.856+0000 7f2857400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28500d0910 con 0x7f28500d1320 2024-03-23T11:31:21.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.856+0000 7f2854a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f285015ca70 0x7f28501606a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:51438/0 (socket says 172.21.15.62:51438) 2024-03-23T11:31:21.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.856+0000 7f2854a00640 1 -- 172.21.15.62:0/3464294674 learned_addr learned my addr 172.21.15.62:0/3464294674 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:21.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.856+0000 7f284fe00640 1 --2- 172.21.15.62:0/3464294674 >> v2:172.21.15.62:3300/0 conn(0x7f28500cf2c0 0x7f28500cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:21.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.856+0000 7f2855400640 1 --2- 172.21.15.62:0/3464294674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28500d1320 0x7f28500d1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:21.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.856+0000 7f284f400640 1 -- 172.21.15.62:0/3464294674 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493775896 0 0) 0x7f28500b58b0 con 0x7f285015ca70 2024-03-23T11:31:21.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.857+0000 7f284f400640 1 -- 172.21.15.62:0/3464294674 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2830003520 con 0x7f285015ca70 2024-03-23T11:31:21.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.857+0000 7f284fe00640 1 -- 172.21.15.62:0/3464294674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28500d1320 msgr2=0x7f28500d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:21.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.857+0000 7f284fe00640 1 --2- 172.21.15.62:0/3464294674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28500d1320 0x7f28500d1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.857+0000 7f284fe00640 1 -- 172.21.15.62:0/3464294674 >> v1:172.21.15.62:6789/0 conn(0x7f285015ca70 legacy=0x7f28501606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.857+0000 7f284fe00640 1 -- 172.21.15.62:0/3464294674 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850161160 con 0x7f28500cf2c0 2024-03-23T11:31:21.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.858+0000 7f284fe00640 1 --2- 172.21.15.62:0/3464294674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28500cf2c0 0x7f28500cf750 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f2838009d90 tx=0x7f283802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc229ad8174160d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:21.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.858+0000 7f284f400640 1 -- 172.21.15.62:0/3464294674 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283802cce0 con 0x7f28500cf2c0 2024-03-23T11:31:21.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.858+0000 7f284f400640 1 -- 172.21.15.62:0/3464294674 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283802ce40 con 0x7f28500cf2c0 2024-03-23T11:31:21.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.858+0000 7f284f400640 1 -- 172.21.15.62:0/3464294674 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2838034660 con 0x7f28500cf2c0 2024-03-23T11:31:21.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.859+0000 7f2857400640 1 -- 172.21.15.62:0/3464294674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28500cf2c0 msgr2=0x7f28500cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.859+0000 7f2857400640 1 --2- 172.21.15.62:0/3464294674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28500cf2c0 0x7f28500cf750 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f2838009d90 tx=0x7f283802bf30 comp rx=0 tx=0).stop 2024-03-23T11:31:21.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.859+0000 7f2857400640 1 -- 172.21.15.62:0/3464294674 shutdown_connections 2024-03-23T11:31:21.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.859+0000 7f2857400640 1 --2- 172.21.15.62:0/3464294674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28500d1320 0x7f28500d1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.859+0000 7f2857400640 1 --2- 172.21.15.62:0/3464294674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28500cf2c0 0x7f28500cf750 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.859+0000 7f2857400640 1 -- 172.21.15.62:0/3464294674 >> 172.21.15.62:0/3464294674 conn(0x7f2850071390 msgr2=0x7f2850071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:21.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.859+0000 7f2857400640 1 -- 172.21.15.62:0/3464294674 shutdown_connections 2024-03-23T11:31:21.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.859+0000 7f2857400640 1 -- 172.21.15.62:0/3464294674 wait complete. 2024-03-23T11:31:21.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.861+0000 7f2857400640 1 Processor -- start 2024-03-23T11:31:21.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.861+0000 7f2857400640 1 -- start start 2024-03-23T11:31:21.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.862+0000 7f2857400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28500cf2c0 0x7f285015f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:21.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.862+0000 7f2857400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28500d1320 0x7f28501f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:21.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.862+0000 7f2857400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f285015ca70 0x7f28500cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:21.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.862+0000 7f2857400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28500b2dc0 con 0x7f28500cf2c0 2024-03-23T11:31:21.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.862+0000 7f2857400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285015cfd0 con 0x7f285015ca70 2024-03-23T11:31:21.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.862+0000 7f2857400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285015d2b0 con 0x7f28500d1320 2024-03-23T11:31:21.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.862+0000 7f2854a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28500cf2c0 0x7f285015f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:21.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.862+0000 7f2854a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28500cf2c0 0x7f285015f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48024/0 (socket says 172.21.15.62:48024) 2024-03-23T11:31:21.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.862+0000 7f2854a00640 1 -- 172.21.15.62:0/1702258030 learned_addr learned my addr 172.21.15.62:0/1702258030 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:21.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f284fe00640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28500d1320 0x7f28501f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:21.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f2854a00640 1 -- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28500d1320 msgr2=0x7f28501f1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f2854a00640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28500d1320 0x7f28501f1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f2854a00640 1 -- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f285015ca70 msgr2=0x7f28500cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:21.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f2855400640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f285015ca70 0x7f28500cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:21.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f2854a00640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f285015ca70 0x7f28500cb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:21.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f2854a00640 1 -- 172.21.15.62:0/1702258030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2838009a70 con 0x7f28500cf2c0 2024-03-23T11:31:21.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f2854a00640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28500cf2c0 0x7f285015f510 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f284000c500 tx=0x7f284000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:21.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f284cc00640 1 -- 172.21.15.62:0/1702258030 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840005ce0 con 0x7f28500cf2c0 2024-03-23T11:31:21.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f284cc00640 1 -- 172.21.15.62:0/1702258030 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2840005e40 con 0x7f28500cf2c0 2024-03-23T11:31:21.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f284cc00640 1 -- 172.21.15.62:0/1702258030 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840007330 con 0x7f28500cf2c0 2024-03-23T11:31:21.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.863+0000 7f2857400640 1 -- 172.21.15.62:0/1702258030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285015d590 con 0x7f28500cf2c0 2024-03-23T11:31:21.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.864+0000 7f2857400640 1 -- 172.21.15.62:0/1702258030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28500ca820 con 0x7f28500cf2c0 2024-03-23T11:31:21.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.865+0000 7f2857400640 1 -- 172.21.15.62:0/1702258030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28500caa10 con 0x7f28500cf2c0 2024-03-23T11:31:21.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.865+0000 7f284cc00640 1 -- 172.21.15.62:0/1702258030 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2840007490 con 0x7f28500cf2c0 2024-03-23T11:31:21.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.866+0000 7f284cc00640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f282c06a7d0 0x7f282c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:21.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.866+0000 7f284cc00640 1 -- 172.21.15.62:0/1702258030 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f28400880e0 con 0x7f28500cf2c0 2024-03-23T11:31:21.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.867+0000 7f284fe00640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f282c06a7d0 0x7f282c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:21.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.868+0000 7f284fe00640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f282c06a7d0 0x7f282c06cc90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2838036000 tx=0x7f2838030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:21.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:21.871+0000 7f284cc00640 1 -- 172.21.15.62:0/1702258030 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2840053900 con 0x7f28500cf2c0 2024-03-23T11:31:22.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.070+0000 7f2857400640 1 -- 172.21.15.62:0/1702258030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2850066c90 con 0x7f28500cf2c0 2024-03-23T11:31:22.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.542+0000 7f284cc00640 1 -- 172.21.15.62:0/1702258030 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v115) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2840056f20 con 0x7f28500cf2c0 2024-03-23T11:31:22.533 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' created 2024-03-23T11:31:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.549+0000 7f2857400640 1 -- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f282c06a7d0 msgr2=0x7f282c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.549+0000 7f2857400640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f282c06a7d0 0x7f282c06cc90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2838036000 tx=0x7f2838030000 comp rx=0 tx=0).stop 2024-03-23T11:31:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.549+0000 7f2857400640 1 -- 172.21.15.62:0/1702258030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28500cf2c0 msgr2=0x7f285015f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.549+0000 7f2857400640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28500cf2c0 0x7f285015f510 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f284000c500 tx=0x7f284000c9c0 comp rx=0 tx=0).stop 2024-03-23T11:31:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.549+0000 7f2857400640 1 -- 172.21.15.62:0/1702258030 shutdown_connections 2024-03-23T11:31:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.549+0000 7f2857400640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f282c06a7d0 0x7f282c06cc90 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.549+0000 7f2857400640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f285015ca70 0x7f28500cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.549+0000 7f2857400640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28500d1320 0x7f28501f1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.549+0000 7f2857400640 1 --2- 172.21.15.62:0/1702258030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28500cf2c0 0x7f285015f510 secure :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f284000c500 tx=0x7f284000c9c0 comp rx=0 tx=0).stop 2024-03-23T11:31:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.549+0000 7f2857400640 1 -- 172.21.15.62:0/1702258030 >> 172.21.15.62:0/1702258030 conn(0x7f2850071390 msgr2=0x7f285015bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.551+0000 7f2857400640 1 -- 172.21.15.62:0/1702258030 shutdown_connections 2024-03-23T11:31:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.552+0000 7f2857400640 1 -- 172.21.15.62:0/1702258030 wait complete. 2024-03-23T11:31:22.566 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T11:31:22.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.790+0000 7f17fee00640 1 Processor -- start 2024-03-23T11:31:22.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.790+0000 7f17fee00640 1 -- start start 2024-03-23T11:31:22.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.791+0000 7f17fee00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f17f80c5390 0x7f17f80b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:22.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.791+0000 7f17fee00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f8070230 con 0x7f17f815de20 2024-03-23T11:31:22.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.791+0000 7f17fee00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f806ce30 con 0x7f17f80cd330 2024-03-23T11:31:22.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.791+0000 7f17fee00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f806d110 con 0x7f17f80c5390 2024-03-23T11:31:22.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.792+0000 7f17f7e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f17f815de20 0x7f17f815e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:51444/0 (socket says 172.21.15.62:51444) 2024-03-23T11:31:22.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.792+0000 7f17efe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f17f80c5390 0x7f17f80b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:22.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.792+0000 7f17f7e00640 1 -- 172.21.15.62:0/1450860600 learned_addr learned my addr 172.21.15.62:0/1450860600 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:22.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.792+0000 7f17efe00640 1 -- 172.21.15.62:0/1450860600 >> v1:172.21.15.120:6789/0 conn(0x7f17f80cd330 legacy=0x7f17f80706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:22.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.792+0000 7f17efe00640 1 -- 172.21.15.62:0/1450860600 >> v1:172.21.15.62:6789/0 conn(0x7f17f815de20 legacy=0x7f17f815e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:22.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.792+0000 7f17efe00640 1 -- 172.21.15.62:0/1450860600 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f806d3f0 con 0x7f17f80c5390 2024-03-23T11:31:22.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.792+0000 7f17f7400640 1 -- 172.21.15.62:0/1450860600 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402659514 0 0) 0x7f17f8070230 con 0x7f17f815de20 2024-03-23T11:31:22.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.794+0000 7f17efe00640 1 --2- 172.21.15.62:0/1450860600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17f80c5390 0x7f17f80b06d0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f17e00097a0 tx=0x7f17e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86da2d6ea07e184 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.794+0000 7f17f7400640 1 -- 172.21.15.62:0/1450860600 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e002e070 con 0x7f17f80c5390 2024-03-23T11:31:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.794+0000 7f17f7400640 1 -- 172.21.15.62:0/1450860600 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e00315a0 con 0x7f17f80c5390 2024-03-23T11:31:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.794+0000 7f17f7400640 1 -- 172.21.15.62:0/1450860600 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0031930 con 0x7f17f80c5390 2024-03-23T11:31:22.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.795+0000 7f17fee00640 1 -- 172.21.15.62:0/1450860600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17f80c5390 msgr2=0x7f17f80b06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:22.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.795+0000 7f17fee00640 1 --2- 172.21.15.62:0/1450860600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17f80c5390 0x7f17f80b06d0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f17e00097a0 tx=0x7f17e002b840 comp rx=0 tx=0).stop 2024-03-23T11:31:22.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.795+0000 7f17fee00640 1 -- 172.21.15.62:0/1450860600 shutdown_connections 2024-03-23T11:31:22.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.795+0000 7f17fee00640 1 --2- 172.21.15.62:0/1450860600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17f80c5390 0x7f17f80b06d0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:22.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.795+0000 7f17fee00640 1 -- 172.21.15.62:0/1450860600 >> 172.21.15.62:0/1450860600 conn(0x7f17f8065a20 msgr2=0x7f17f8065e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:22.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.795+0000 7f17fee00640 1 -- 172.21.15.62:0/1450860600 shutdown_connections 2024-03-23T11:31:22.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.795+0000 7f17fee00640 1 -- 172.21.15.62:0/1450860600 wait complete. 2024-03-23T11:31:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.797+0000 7f17fee00640 1 Processor -- start 2024-03-23T11:31:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.797+0000 7f17fee00640 1 -- start start 2024-03-23T11:31:22.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17fee00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17f80c5390 0x7f17f80cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:22.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17fee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17f80cd330 0x7f17f8149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:22.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17fee00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17f815de20 0x7f17f80cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:22.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17fee00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f80cc4f0 con 0x7f17f80c5390 2024-03-23T11:31:22.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17fee00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f80cb660 con 0x7f17f815de20 2024-03-23T11:31:22.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17fee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f80cb870 con 0x7f17f80cd330 2024-03-23T11:31:22.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17f7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17f80c5390 0x7f17f80cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:22.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17f7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17f80c5390 0x7f17f80cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48040/0 (socket says 172.21.15.62:48040) 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17f7e00640 1 -- 172.21.15.62:0/1217679080 learned_addr learned my addr 172.21.15.62:0/1217679080 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17f7e00640 1 -- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17f80cd330 msgr2=0x7f17f8149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17f7e00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17f80cd330 0x7f17f8149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17f7e00640 1 -- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17f815de20 msgr2=0x7f17f80cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17fce00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17f815de20 0x7f17f80cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17f7e00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17f815de20 0x7f17f80cc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.798+0000 7f17f7e00640 1 -- 172.21.15.62:0/1217679080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e0009400 con 0x7f17f80c5390 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.799+0000 7f17f7e00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17f80c5390 0x7f17f80cb120 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c4d0 tx=0x7f17e800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.799+0000 7f17f4c00640 1 -- 172.21.15.62:0/1217679080 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e80074e0 con 0x7f17f80c5390 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.799+0000 7f17f4c00640 1 -- 172.21.15.62:0/1217679080 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e8007660 con 0x7f17f80c5390 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.799+0000 7f17fee00640 1 -- 172.21.15.62:0/1217679080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f80cbb50 con 0x7f17f80c5390 2024-03-23T11:31:22.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.799+0000 7f17f4c00640 1 -- 172.21.15.62:0/1217679080 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8004a20 con 0x7f17f80c5390 2024-03-23T11:31:22.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.799+0000 7f17fee00640 1 -- 172.21.15.62:0/1217679080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f80cc110 con 0x7f17f80c5390 2024-03-23T11:31:22.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.800+0000 7f17fee00640 1 -- 172.21.15.62:0/1217679080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f80cc630 con 0x7f17f80c5390 2024-03-23T11:31:22.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.802+0000 7f17f4c00640 1 -- 172.21.15.62:0/1217679080 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f17e8004b80 con 0x7f17f80c5390 2024-03-23T11:31:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.802+0000 7f17f4c00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f17d406a7a0 0x7f17d406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.802+0000 7f17f4c00640 1 -- 172.21.15.62:0/1217679080 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f17e8087d90 con 0x7f17f80c5390 2024-03-23T11:31:22.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.803+0000 7f17efe00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f17d406a7a0 0x7f17d406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:22.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.805+0000 7f17efe00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f17d406a7a0 0x7f17d406cc60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f17e0036000 tx=0x7f17e0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:22.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:22.808+0000 7f17f4c00640 1 -- 172.21.15.62:0/1217679080 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17f80cc630 con 0x7f17f80c5390 2024-03-23T11:31:22.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.005+0000 7f17fee00640 1 -- 172.21.15.62:0/1217679080 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f17f80cc630 con 0x7f17f80c5390 2024-03-23T11:31:23.545 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' created 2024-03-23T11:31:23.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.555+0000 7f17f4c00640 1 -- 172.21.15.62:0/1217679080 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v116) v1 ==== 170+0+0 (secure 0 0 0) 0x7f17f80cc630 con 0x7f17f80c5390 2024-03-23T11:31:23.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.562+0000 7f17fee00640 1 -- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f17d406a7a0 msgr2=0x7f17d406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:23.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.562+0000 7f17fee00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f17d406a7a0 0x7f17d406cc60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f17e0036000 tx=0x7f17e0030000 comp rx=0 tx=0).stop 2024-03-23T11:31:23.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.562+0000 7f17fee00640 1 -- 172.21.15.62:0/1217679080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17f80c5390 msgr2=0x7f17f80cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:23.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.562+0000 7f17fee00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17f80c5390 0x7f17f80cb120 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c4d0 tx=0x7f17e800c990 comp rx=0 tx=0).stop 2024-03-23T11:31:23.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.563+0000 7f17fee00640 1 -- 172.21.15.62:0/1217679080 shutdown_connections 2024-03-23T11:31:23.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.563+0000 7f17fee00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f17d406a7a0 0x7f17d406cc60 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:23.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.563+0000 7f17fee00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17f815de20 0x7f17f80cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:23.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.563+0000 7f17fee00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17f80cd330 0x7f17f8149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:23.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.563+0000 7f17fee00640 1 --2- 172.21.15.62:0/1217679080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17f80c5390 0x7f17f80cb120 secure :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c4d0 tx=0x7f17e800c990 comp rx=0 tx=0).stop 2024-03-23T11:31:23.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.563+0000 7f17fee00640 1 -- 172.21.15.62:0/1217679080 >> 172.21.15.62:0/1217679080 conn(0x7f17f8065a20 msgr2=0x7f17f80b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:23.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.566+0000 7f17fee00640 1 -- 172.21.15.62:0/1217679080 shutdown_connections 2024-03-23T11:31:23.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.566+0000 7f17fee00640 1 -- 172.21.15.62:0/1217679080 wait complete. 2024-03-23T11:31:23.585 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T11:31:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.780+0000 7f6cfd400640 1 Processor -- start 2024-03-23T11:31:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.781+0000 7f6cfd400640 1 -- start start 2024-03-23T11:31:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.782+0000 7f6cfd400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6cf806c300 0x7f6cf80ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.782+0000 7f6cfd400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6cf806c6e0 0x7f6cf806cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.782+0000 7f6cfd400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf80cbc60 con 0x7f6cf80cafa0 2024-03-23T11:31:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.782+0000 7f6cfd400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf80cbf40 con 0x7f6cf806c300 2024-03-23T11:31:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.782+0000 7f6cfd400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf80cc220 con 0x7f6cf806c6e0 2024-03-23T11:31:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.782+0000 7f6cf6800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6cf806c300 0x7f6cf80ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.783+0000 7f6cf6800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6cf806c300 0x7f6cf80ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48054/0 (socket says 172.21.15.62:48054) 2024-03-23T11:31:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.783+0000 7f6cf6800640 1 -- 172.21.15.62:0/3681018692 learned_addr learned my addr 172.21.15.62:0/3681018692 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.783+0000 7f6cf6800640 1 -- 172.21.15.62:0/3681018692 >> v2:172.21.15.120:3300/0 conn(0x7f6cf806c6e0 msgr2=0x7f6cf806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.783+0000 7f6cf6800640 1 --2- 172.21.15.62:0/3681018692 >> v2:172.21.15.120:3300/0 conn(0x7f6cf806c6e0 0x7f6cf806cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.783+0000 7f6cf6800640 1 -- 172.21.15.62:0/3681018692 >> v1:172.21.15.120:6789/0 conn(0x7f6cf80cafa0 legacy=0x7f6cf80cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.783+0000 7f6cf6800640 1 -- 172.21.15.62:0/3681018692 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf80cc500 con 0x7f6cf806c300 2024-03-23T11:31:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.784+0000 7f6cf6800640 1 --2- 172.21.15.62:0/3681018692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf806c300 0x7f6cf80ccd00 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8070270 tx=0x7f6ce802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3dfa06c0defb5e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.784+0000 7f6cf5e00640 1 -- 172.21.15.62:0/3681018692 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8031870 con 0x7f6cf806c300 2024-03-23T11:31:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.784+0000 7f6cf5e00640 1 -- 172.21.15.62:0/3681018692 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce80319d0 con 0x7f6cf806c300 2024-03-23T11:31:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.784+0000 7f6cf5e00640 1 -- 172.21.15.62:0/3681018692 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8031d60 con 0x7f6cf806c300 2024-03-23T11:31:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.785+0000 7f6cfd400640 1 -- 172.21.15.62:0/3681018692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf806c300 msgr2=0x7f6cf80ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.785+0000 7f6cfd400640 1 --2- 172.21.15.62:0/3681018692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf806c300 0x7f6cf80ccd00 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8070270 tx=0x7f6ce802b910 comp rx=0 tx=0).stop 2024-03-23T11:31:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.785+0000 7f6cfd400640 1 -- 172.21.15.62:0/3681018692 shutdown_connections 2024-03-23T11:31:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.785+0000 7f6cfd400640 1 --2- 172.21.15.62:0/3681018692 >> v2:172.21.15.120:3300/0 conn(0x7f6cf806c6e0 0x7f6cf806cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.785+0000 7f6cfd400640 1 --2- 172.21.15.62:0/3681018692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf806c300 0x7f6cf80ccd00 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.785+0000 7f6cfd400640 1 -- 172.21.15.62:0/3681018692 >> 172.21.15.62:0/3681018692 conn(0x7f6cf80b02e0 msgr2=0x7f6cf80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.785+0000 7f6cfd400640 1 -- 172.21.15.62:0/3681018692 shutdown_connections 2024-03-23T11:31:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.785+0000 7f6cfd400640 1 -- 172.21.15.62:0/3681018692 wait complete. 2024-03-23T11:31:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.787+0000 7f6cfd400640 1 Processor -- start 2024-03-23T11:31:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.787+0000 7f6cfd400640 1 -- start start 2024-03-23T11:31:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.788+0000 7f6cfd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf806c300 0x7f6cf813d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.788+0000 7f6cfd400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cf806c6e0 0x7f6cf813de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.788+0000 7f6cfd400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cf80cafa0 0x7f6cf8149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.788+0000 7f6cfd400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf813b730 con 0x7f6cf806c300 2024-03-23T11:31:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.788+0000 7f6cfd400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf813e660 con 0x7f6cf80cafa0 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.788+0000 7f6cfd400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf813e940 con 0x7f6cf806c6e0 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.788+0000 7f6cf6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf806c300 0x7f6cf813d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.788+0000 7f6cf6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf806c300 0x7f6cf813d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48060/0 (socket says 172.21.15.62:48060) 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.788+0000 7f6cf6800640 1 -- 172.21.15.62:0/2713107517 learned_addr learned my addr 172.21.15.62:0/2713107517 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.788+0000 7f6cede00640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cf806c6e0 0x7f6cf813de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cf6800640 1 -- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cf806c6e0 msgr2=0x7f6cf813de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cf6800640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cf806c6e0 0x7f6cf813de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cf7200640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cf80cafa0 0x7f6cf8149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cf6800640 1 -- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cf80cafa0 msgr2=0x7f6cf8149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cf6800640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cf80cafa0 0x7f6cf8149610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cf6800640 1 -- 172.21.15.62:0/2713107517 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce80094d0 con 0x7f6cf806c300 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cede00640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cf806c6e0 0x7f6cf813de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cf6800640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf806c300 0x7f6cf813d8f0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8000c00 tx=0x7f6ce80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cef400640 1 -- 172.21.15.62:0/2713107517 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce803ad10 con 0x7f6cf806c300 2024-03-23T11:31:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cef400640 1 -- 172.21.15.62:0/2713107517 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce803ae70 con 0x7f6cf806c300 2024-03-23T11:31:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cef400640 1 -- 172.21.15.62:0/2713107517 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8044610 con 0x7f6cf806c300 2024-03-23T11:31:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.789+0000 7f6cfd400640 1 -- 172.21.15.62:0/2713107517 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf813ec20 con 0x7f6cf806c300 2024-03-23T11:31:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.790+0000 7f6cfd400640 1 -- 172.21.15.62:0/2713107517 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf813e370 con 0x7f6cf806c300 2024-03-23T11:31:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.791+0000 7f6cfd400640 1 -- 172.21.15.62:0/2713107517 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf813f270 con 0x7f6cf806c300 2024-03-23T11:31:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.792+0000 7f6cef400640 1 -- 172.21.15.62:0/2713107517 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6ce804d450 con 0x7f6cf806c300 2024-03-23T11:31:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.793+0000 7f6cef400640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6cd406a7d0 0x7f6cd406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.793+0000 7f6cef400640 1 -- 172.21.15.62:0/2713107517 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f6ce802d040 con 0x7f6cf806c300 2024-03-23T11:31:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.793+0000 7f6cede00640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6cd406a7d0 0x7f6cd406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.794+0000 7f6cede00640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6cd406a7d0 0x7f6cd406cc90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80aab30 tx=0x7f6ce0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:23.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:23.798+0000 7f6cef400640 1 -- 172.21.15.62:0/2713107517 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ce8076900 con 0x7f6cf806c300 2024-03-23T11:31:24.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.042+0000 7f6cfd400640 1 -- 172.21.15.62:0/2713107517 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6cf8072270 con 0x7f6cf806c300 2024-03-23T11:31:24.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.053+0000 7f6cef400640 1 -- 172.21.15.62:0/2713107517 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v90) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6ce8079f20 con 0x7f6cf806c300 2024-03-23T11:31:24.043 INFO:teuthology.orchestra.run.smithi062.stderr:new fs with metadata pool 16 and data pool 17 2024-03-23T11:31:24.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.057+0000 7f6cfd400640 1 -- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6cd406a7d0 msgr2=0x7f6cd406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.058+0000 7f6cfd400640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6cd406a7d0 0x7f6cd406cc90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80aab30 tx=0x7f6ce0008040 comp rx=0 tx=0).stop 2024-03-23T11:31:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.058+0000 7f6cfd400640 1 -- 172.21.15.62:0/2713107517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf806c300 msgr2=0x7f6cf813d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.058+0000 7f6cfd400640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf806c300 0x7f6cf813d8f0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8000c00 tx=0x7f6ce80046c0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.058+0000 7f6cfd400640 1 -- 172.21.15.62:0/2713107517 shutdown_connections 2024-03-23T11:31:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.058+0000 7f6cfd400640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6cd406a7d0 0x7f6cd406cc90 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.058+0000 7f6cfd400640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cf80cafa0 0x7f6cf8149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.058+0000 7f6cfd400640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cf806c6e0 0x7f6cf813de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.058+0000 7f6cfd400640 1 --2- 172.21.15.62:0/2713107517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf806c300 0x7f6cf813d8f0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.058+0000 7f6cfd400640 1 -- 172.21.15.62:0/2713107517 >> 172.21.15.62:0/2713107517 conn(0x7f6cf80b02e0 msgr2=0x7f6cf80cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:24.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.062+0000 7f6cfd400640 1 -- 172.21.15.62:0/2713107517 shutdown_connections 2024-03-23T11:31:24.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.063+0000 7f6cfd400640 1 -- 172.21.15.62:0/2713107517 wait complete. 2024-03-23T11:31:24.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:31:24.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.276+0000 7f4617200640 1 Processor -- start 2024-03-23T11:31:24.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.276+0000 7f4617200640 1 -- start start 2024-03-23T11:31:24.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.277+0000 7f4617200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f461006c300 0x7f46100ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.277+0000 7f4617200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f46100cafa0 0x7f46100cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.277+0000 7f4617200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461006c6e0 0x7f461006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.277+0000 7f4617200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46100cbc60 con 0x7f461006c300 2024-03-23T11:31:24.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.277+0000 7f4617200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46100cbf40 con 0x7f46100cafa0 2024-03-23T11:31:24.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.277+0000 7f4617200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46100cc220 con 0x7f461006c6e0 2024-03-23T11:31:24.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.278+0000 7f460fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f461006c300 0x7f46100ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.278+0000 7f460fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f461006c300 0x7f46100ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48074/0 (socket says 172.21.15.62:48074) 2024-03-23T11:31:24.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.278+0000 7f460fe00640 1 -- 172.21.15.62:0/2802681572 learned_addr learned my addr 172.21.15.62:0/2802681572 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:24.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.278+0000 7f4615200640 1 --2- 172.21.15.62:0/2802681572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461006c6e0 0x7f461006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.278+0000 7f460fe00640 1 -- 172.21.15.62:0/2802681572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461006c6e0 msgr2=0x7f461006cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:24.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.278+0000 7f460fe00640 1 --2- 172.21.15.62:0/2802681572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461006c6e0 0x7f461006cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.278+0000 7f460fe00640 1 -- 172.21.15.62:0/2802681572 >> v2:172.21.15.120:3300/0 conn(0x7f46100cafa0 msgr2=0x7f46100cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:24.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.278+0000 7f4607400640 1 --2- 172.21.15.62:0/2802681572 >> v2:172.21.15.120:3300/0 conn(0x7f46100cafa0 0x7f46100cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.278+0000 7f460fe00640 1 --2- 172.21.15.62:0/2802681572 >> v2:172.21.15.120:3300/0 conn(0x7f46100cafa0 0x7f46100cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.278+0000 7f460fe00640 1 -- 172.21.15.62:0/2802681572 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46100cc500 con 0x7f461006c300 2024-03-23T11:31:24.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.279+0000 7f460fe00640 1 --2- 172.21.15.62:0/2802681572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f461006c300 0x7f46100ccd00 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f461006e370 tx=0x7f460002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad2b46d133c4bac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:24.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.280+0000 7f460f400640 1 -- 172.21.15.62:0/2802681572 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460002ca70 con 0x7f461006c300 2024-03-23T11:31:24.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.280+0000 7f460f400640 1 -- 172.21.15.62:0/2802681572 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460002cbd0 con 0x7f461006c300 2024-03-23T11:31:24.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.280+0000 7f460f400640 1 -- 172.21.15.62:0/2802681572 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46000343f0 con 0x7f461006c300 2024-03-23T11:31:24.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.280+0000 7f4617200640 1 -- 172.21.15.62:0/2802681572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f461006c300 msgr2=0x7f46100ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.280+0000 7f4617200640 1 --2- 172.21.15.62:0/2802681572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f461006c300 0x7f46100ccd00 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f461006e370 tx=0x7f460002bed0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.281+0000 7f4617200640 1 -- 172.21.15.62:0/2802681572 shutdown_connections 2024-03-23T11:31:24.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.281+0000 7f4617200640 1 --2- 172.21.15.62:0/2802681572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461006c6e0 0x7f461006cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.281+0000 7f4617200640 1 --2- 172.21.15.62:0/2802681572 >> v2:172.21.15.120:3300/0 conn(0x7f46100cafa0 0x7f46100cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.281+0000 7f4617200640 1 --2- 172.21.15.62:0/2802681572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f461006c300 0x7f46100ccd00 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.281+0000 7f4617200640 1 -- 172.21.15.62:0/2802681572 >> 172.21.15.62:0/2802681572 conn(0x7f46100b02e0 msgr2=0x7f46100b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:24.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.281+0000 7f4617200640 1 -- 172.21.15.62:0/2802681572 shutdown_connections 2024-03-23T11:31:24.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.281+0000 7f4617200640 1 -- 172.21.15.62:0/2802681572 wait complete. 2024-03-23T11:31:24.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.282+0000 7f4617200640 1 Processor -- start 2024-03-23T11:31:24.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.283+0000 7f4617200640 1 -- start start 2024-03-23T11:31:24.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.283+0000 7f4617200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f461006c300 0x7f461015caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.283+0000 7f4617200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461006c6e0 0x7f46101f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.283+0000 7f4617200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46100cafa0 0x7f46101f23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.283+0000 7f4617200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461015ad80 con 0x7f461006c300 2024-03-23T11:31:24.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.283+0000 7f4617200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46101f2ed0 con 0x7f46100cafa0 2024-03-23T11:31:24.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.284+0000 7f4617200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46101f31b0 con 0x7f461006c6e0 2024-03-23T11:31:24.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.284+0000 7f460fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f461006c300 0x7f461015caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.284+0000 7f460fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f461006c300 0x7f461015caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48090/0 (socket says 172.21.15.62:48090) 2024-03-23T11:31:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.284+0000 7f460fe00640 1 -- 172.21.15.62:0/1007013243 learned_addr learned my addr 172.21.15.62:0/1007013243 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.284+0000 7f4607400640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461006c6e0 0x7f46101f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.284+0000 7f460fe00640 1 -- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461006c6e0 msgr2=0x7f46101f3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.284+0000 7f460fe00640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461006c6e0 0x7f46101f3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.284+0000 7f4615200640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46100cafa0 0x7f46101f23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.284+0000 7f460fe00640 1 -- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46100cafa0 msgr2=0x7f46101f23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.284+0000 7f460fe00640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46100cafa0 0x7f46101f23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.284+0000 7f460fe00640 1 -- 172.21.15.62:0/1007013243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4600009b30 con 0x7f461006c300 2024-03-23T11:31:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.285+0000 7f460fe00640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f461006c300 0x7f461015caa0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f4600000f80 tx=0x7f4600004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.285+0000 7f460cc00640 1 -- 172.21.15.62:0/1007013243 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460003e450 con 0x7f461006c300 2024-03-23T11:31:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.285+0000 7f460cc00640 1 -- 172.21.15.62:0/1007013243 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460003e5f0 con 0x7f461006c300 2024-03-23T11:31:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.285+0000 7f460cc00640 1 -- 172.21.15.62:0/1007013243 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460003e9a0 con 0x7f461006c300 2024-03-23T11:31:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.285+0000 7f4617200640 1 -- 172.21.15.62:0/1007013243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f461015af90 con 0x7f461006c300 2024-03-23T11:31:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.285+0000 7f4617200640 1 -- 172.21.15.62:0/1007013243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461013f970 con 0x7f461006c300 2024-03-23T11:31:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.286+0000 7f4617200640 1 -- 172.21.15.62:0/1007013243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461015b4b0 con 0x7f461006c300 2024-03-23T11:31:24.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.287+0000 7f460cc00640 1 -- 172.21.15.62:0/1007013243 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f460003eb00 con 0x7f461006c300 2024-03-23T11:31:24.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.288+0000 7f460cc00640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45ec06a7a0 0x7f45ec06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.288+0000 7f460cc00640 1 -- 172.21.15.62:0/1007013243 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46000abc80 con 0x7f461006c300 2024-03-23T11:31:24.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.289+0000 7f4607400640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45ec06a7a0 0x7f45ec06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.290+0000 7f4607400640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45ec06a7a0 0x7f45ec06cc60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f46100ccf50 tx=0x7f45f8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:24.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.293+0000 7f460cc00640 1 -- 172.21.15.62:0/1007013243 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f461015b4b0 con 0x7f461006c300 2024-03-23T11:31:24.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.492+0000 7f4617200640 1 -- 172.21.15.62:0/1007013243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f461015b4b0 con 0x7f461006c300 2024-03-23T11:31:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.494+0000 7f460cc00640 1 -- 172.21.15.62:0/1007013243 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+16072 (secure 0 0 0) 0x7f461015b4b0 con 0x7f461006c300 2024-03-23T11:31:24.485 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:24.485 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":117,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:31:24.043970+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:31:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 -- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45ec06a7a0 msgr2=0x7f45ec06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45ec06a7a0 0x7f45ec06cc60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f46100ccf50 tx=0x7f45f8008040 comp rx=0 tx=0).stop 2024-03-23T11:31:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 -- 172.21.15.62:0/1007013243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f461006c300 msgr2=0x7f461015caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f461006c300 0x7f461015caa0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f4600000f80 tx=0x7f4600004a40 comp rx=0 tx=0).stop 2024-03-23T11:31:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 -- 172.21.15.62:0/1007013243 shutdown_connections 2024-03-23T11:31:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f45ec06a7a0 0x7f45ec06cc60 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46100cafa0 0x7f46101f23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f461006c6e0 0x7f46101f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 --2- 172.21.15.62:0/1007013243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f461006c300 0x7f461015caa0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 -- 172.21.15.62:0/1007013243 >> 172.21.15.62:0/1007013243 conn(0x7f46100b02e0 msgr2=0x7f46100cd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:24.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 -- 172.21.15.62:0/1007013243 shutdown_connections 2024-03-23T11:31:24.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.498+0000 7f4617200640 1 -- 172.21.15.62:0/1007013243 wait complete. 2024-03-23T11:31:24.506 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:31:24.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.730+0000 7f8ee8e00640 1 Processor -- start 2024-03-23T11:31:24.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.730+0000 7f8ee8e00640 1 -- start start 2024-03-23T11:31:24.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.731+0000 7f8ee8e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ee4143dd0 0x7f8ee415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.731+0000 7f8ee8e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8ee4070090 0x7f8ee40704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.731+0000 7f8ee8e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee4070a30 con 0x7f8ee40b2820 2024-03-23T11:31:24.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.731+0000 7f8ee8e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee40cd140 con 0x7f8ee4070090 2024-03-23T11:31:24.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.731+0000 7f8ee8e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee40cd3e0 con 0x7f8ee4143dd0 2024-03-23T11:31:24.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.732+0000 7f8ee3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ee4143dd0 0x7f8ee415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.732+0000 7f8ee3e00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f8ee40b2820 0x7f8ee40b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:56626/0 (socket says 172.21.15.62:56626) 2024-03-23T11:31:24.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.733+0000 7f8ee3e00640 1 -- 172.21.15.62:0/3701823821 learned_addr learned my addr 172.21.15.62:0/3701823821 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:24.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.733+0000 7f8ee2a00640 1 --2- 172.21.15.62:0/3701823821 >> v2:172.21.15.120:3300/0 conn(0x7f8ee4070090 0x7f8ee40704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.734+0000 7f8ee2a00640 1 -- 172.21.15.62:0/3701823821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ee4143dd0 msgr2=0x7f8ee415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:24.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.734+0000 7f8ee2a00640 1 --2- 172.21.15.62:0/3701823821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ee4143dd0 0x7f8ee415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.734+0000 7f8ee2a00640 1 -- 172.21.15.62:0/3701823821 >> v1:172.21.15.120:6789/0 conn(0x7f8ee40b2820 legacy=0x7f8ee40b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.734+0000 7f8ee2a00640 1 -- 172.21.15.62:0/3701823821 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee40cd6c0 con 0x7f8ee4070090 2024-03-23T11:31:24.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.735+0000 7f8ee2a00640 1 --2- 172.21.15.62:0/3701823821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ee4070090 0x7f8ee40704f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00097a0 tx=0x7f8ed002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c2cefca83b344c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:24.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.735+0000 7f8ee2000640 1 -- 172.21.15.62:0/3701823821 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed002d040 con 0x7f8ee4070090 2024-03-23T11:31:24.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.735+0000 7f8ee2000640 1 -- 172.21.15.62:0/3701823821 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed0031810 con 0x7f8ee4070090 2024-03-23T11:31:24.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.735+0000 7f8ee2000640 1 -- 172.21.15.62:0/3701823821 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0031ba0 con 0x7f8ee4070090 2024-03-23T11:31:24.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.736+0000 7f8ee8e00640 1 -- 172.21.15.62:0/3701823821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ee4070090 msgr2=0x7f8ee40704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.736+0000 7f8ee8e00640 1 --2- 172.21.15.62:0/3701823821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ee4070090 0x7f8ee40704f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00097a0 tx=0x7f8ed002b8b0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.736+0000 7f8ee8e00640 1 -- 172.21.15.62:0/3701823821 shutdown_connections 2024-03-23T11:31:24.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.736+0000 7f8ee8e00640 1 --2- 172.21.15.62:0/3701823821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ee4070090 0x7f8ee40704f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.736+0000 7f8ee8e00640 1 --2- 172.21.15.62:0/3701823821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ee4143dd0 0x7f8ee415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.736+0000 7f8ee8e00640 1 -- 172.21.15.62:0/3701823821 >> 172.21.15.62:0/3701823821 conn(0x7f8ee4066050 msgr2=0x7f8ee4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:24.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.736+0000 7f8ee8e00640 1 -- 172.21.15.62:0/3701823821 shutdown_connections 2024-03-23T11:31:24.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.736+0000 7f8ee8e00640 1 -- 172.21.15.62:0/3701823821 wait complete. 2024-03-23T11:31:24.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.738+0000 7f8ee8e00640 1 Processor -- start 2024-03-23T11:31:24.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.738+0000 7f8ee8e00640 1 -- start start 2024-03-23T11:31:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.739+0000 7f8ee8e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ee4070090 0x7f8ee4159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.739+0000 7f8ee8e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ee40b2820 0x7f8ee406d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.739+0000 7f8ee8e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ee4143dd0 0x7f8ee406d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.739+0000 7f8ee8e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee413fda0 con 0x7f8ee4070090 2024-03-23T11:31:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.739+0000 7f8ee8e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee413d6d0 con 0x7f8ee40b2820 2024-03-23T11:31:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.739+0000 7f8ee8e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee413d9b0 con 0x7f8ee4143dd0 2024-03-23T11:31:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.739+0000 7f8ee3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ee4070090 0x7f8ee4159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.739+0000 7f8ee3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ee4070090 0x7f8ee4159b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48102/0 (socket says 172.21.15.62:48102) 2024-03-23T11:31:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.739+0000 7f8ee3400640 1 -- 172.21.15.62:0/1951397231 learned_addr learned my addr 172.21.15.62:0/1951397231 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.740+0000 7f8ee2a00640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ee40b2820 0x7f8ee406d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.740+0000 7f8ee3e00640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ee4143dd0 0x7f8ee406d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.740+0000 7f8ee3400640 1 -- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ee4143dd0 msgr2=0x7f8ee406d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.740+0000 7f8ee3400640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ee4143dd0 0x7f8ee406d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.740+0000 7f8ee3400640 1 -- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ee40b2820 msgr2=0x7f8ee406d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.740+0000 7f8ee3400640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ee40b2820 0x7f8ee406d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.740+0000 7f8ee3400640 1 -- 172.21.15.62:0/1951397231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed0009400 con 0x7f8ee4070090 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.740+0000 7f8ee2a00640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ee40b2820 0x7f8ee406d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.740+0000 7f8ee3400640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ee4070090 0x7f8ee4159b80 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400e870 tx=0x7f8ed400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.741+0000 7f8edb400640 1 -- 172.21.15.62:0/1951397231 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed40094f0 con 0x7f8ee4070090 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.741+0000 7f8edb400640 1 -- 172.21.15.62:0/1951397231 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed400c940 con 0x7f8ee4070090 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.741+0000 7f8edb400640 1 -- 172.21.15.62:0/1951397231 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed400ccf0 con 0x7f8ee4070090 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.741+0000 7f8ee8e00640 1 -- 172.21.15.62:0/1951397231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee413dc90 con 0x7f8ee4070090 2024-03-23T11:31:24.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.741+0000 7f8ee8e00640 1 -- 172.21.15.62:0/1951397231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee41f2e00 con 0x7f8ee4070090 2024-03-23T11:31:24.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.742+0000 7f8ee8e00640 1 -- 172.21.15.62:0/1951397231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee4066bb0 con 0x7f8ee4070090 2024-03-23T11:31:24.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.743+0000 7f8edb400640 1 -- 172.21.15.62:0/1951397231 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8ed4013470 con 0x7f8ee4070090 2024-03-23T11:31:24.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.744+0000 7f8edb400640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ec406a7a0 0x7f8ec406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:24.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.744+0000 7f8edb400640 1 -- 172.21.15.62:0/1951397231 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ed4088ee0 con 0x7f8ee4070090 2024-03-23T11:31:24.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.745+0000 7f8ee2a00640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ec406a7a0 0x7f8ec406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:24.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.746+0000 7f8ee2a00640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ec406a7a0 0x7f8ec406cc60 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0036000 tx=0x7f8ed003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:24.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.749+0000 7f8edb400640 1 -- 172.21.15.62:0/1951397231 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ed40542f0 con 0x7f8ee4070090 2024-03-23T11:31:24.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.946+0000 7f8ee8e00640 1 -- 172.21.15.62:0/1951397231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ee40706a0 con 0x7f8ee4070090 2024-03-23T11:31:24.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.949+0000 7f8edb400640 1 -- 172.21.15.62:0/1951397231 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+16072 (secure 0 0 0) 0x7f8ed4057910 con 0x7f8ee4070090 2024-03-23T11:31:24.940 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:24.940 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":117,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:31:24.043970+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:31:24.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.952+0000 7f8ee8e00640 1 -- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ec406a7a0 msgr2=0x7f8ec406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.952+0000 7f8ee8e00640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ec406a7a0 0x7f8ec406cc60 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0036000 tx=0x7f8ed003c040 comp rx=0 tx=0).stop 2024-03-23T11:31:24.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.952+0000 7f8ee8e00640 1 -- 172.21.15.62:0/1951397231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ee4070090 msgr2=0x7f8ee4159b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:24.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.952+0000 7f8ee8e00640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ee4070090 0x7f8ee4159b80 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400e870 tx=0x7f8ed400ed30 comp rx=0 tx=0).stop 2024-03-23T11:31:24.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.953+0000 7f8ee8e00640 1 -- 172.21.15.62:0/1951397231 shutdown_connections 2024-03-23T11:31:24.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.953+0000 7f8ee8e00640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ec406a7a0 0x7f8ec406cc60 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.953+0000 7f8ee8e00640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8ee4143dd0 0x7f8ee406d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.953+0000 7f8ee8e00640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ee40b2820 0x7f8ee406d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.953+0000 7f8ee8e00640 1 --2- 172.21.15.62:0/1951397231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ee4070090 0x7f8ee4159b80 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:24.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.953+0000 7f8ee8e00640 1 -- 172.21.15.62:0/1951397231 >> 172.21.15.62:0/1951397231 conn(0x7f8ee4066050 msgr2=0x7f8ee40adde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:24.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.953+0000 7f8ee8e00640 1 -- 172.21.15.62:0/1951397231 shutdown_connections 2024-03-23T11:31:24.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:24.953+0000 7f8ee8e00640 1 -- 172.21.15.62:0/1951397231 wait complete. 2024-03-23T11:31:24.964 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T11:31:25.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.202+0000 7f02dfe00640 1 Processor -- start 2024-03-23T11:31:25.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.203+0000 7f02dfe00640 1 -- start start 2024-03-23T11:31:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.204+0000 7f02dfe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02d806c6e0 0x7f02d806cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.204+0000 7f02dfe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d80cbc60 con 0x7f02d80cafa0 2024-03-23T11:31:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.204+0000 7f02dfe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d80cbf40 con 0x7f02d806c300 2024-03-23T11:31:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.204+0000 7f02dfe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d80cd410 con 0x7f02d806c6e0 2024-03-23T11:31:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.205+0000 7f02dca00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f02d80cafa0 0x7f02d80cb430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:51456/0 (socket says 172.21.15.62:51456) 2024-03-23T11:31:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.205+0000 7f02dca00640 1 -- 172.21.15.62:0/1613764344 learned_addr learned my addr 172.21.15.62:0/1613764344 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.205+0000 7f02dde00640 1 --2- 172.21.15.62:0/1613764344 >> v2:172.21.15.120:3300/0 conn(0x7f02d806c6e0 0x7f02d806cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.206+0000 7f02d7e00640 1 -- 172.21.15.62:0/1613764344 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389999664 0 0) 0x7f02d80cbc60 con 0x7f02d80cafa0 2024-03-23T11:31:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.206+0000 7f02d7e00640 1 -- 172.21.15.62:0/1613764344 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c0003520 con 0x7f02d80cafa0 2024-03-23T11:31:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.206+0000 7f02dde00640 1 -- 172.21.15.62:0/1613764344 >> v1:172.21.15.120:6789/0 conn(0x7f02d806c300 legacy=0x7f02d80ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.206+0000 7f02dde00640 1 -- 172.21.15.62:0/1613764344 >> v1:172.21.15.62:6789/0 conn(0x7f02d80cafa0 legacy=0x7f02d80cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.206+0000 7f02dde00640 1 -- 172.21.15.62:0/1613764344 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d80cc220 con 0x7f02d806c6e0 2024-03-23T11:31:25.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.208+0000 7f02dde00640 1 --2- 172.21.15.62:0/1613764344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d806c6e0 0x7f02d806cb30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f02d0009d90 tx=0x7f02d002bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4d0eb4f2cd180bd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:25.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.209+0000 7f02d7e00640 1 -- 172.21.15.62:0/1613764344 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d002ccf0 con 0x7f02d806c6e0 2024-03-23T11:31:25.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.209+0000 7f02d7e00640 1 -- 172.21.15.62:0/1613764344 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d002ce50 con 0x7f02d806c6e0 2024-03-23T11:31:25.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.209+0000 7f02d7e00640 1 -- 172.21.15.62:0/1613764344 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d0034640 con 0x7f02d806c6e0 2024-03-23T11:31:25.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.210+0000 7f02dfe00640 1 -- 172.21.15.62:0/1613764344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d806c6e0 msgr2=0x7f02d806cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:25.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.210+0000 7f02dfe00640 1 --2- 172.21.15.62:0/1613764344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d806c6e0 0x7f02d806cb30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f02d0009d90 tx=0x7f02d002bf40 comp rx=0 tx=0).stop 2024-03-23T11:31:25.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.210+0000 7f02dfe00640 1 -- 172.21.15.62:0/1613764344 shutdown_connections 2024-03-23T11:31:25.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.210+0000 7f02dfe00640 1 --2- 172.21.15.62:0/1613764344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d806c6e0 0x7f02d806cb30 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:25.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.210+0000 7f02dfe00640 1 -- 172.21.15.62:0/1613764344 >> 172.21.15.62:0/1613764344 conn(0x7f02d80b02e0 msgr2=0x7f02d80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:25.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.210+0000 7f02dfe00640 1 -- 172.21.15.62:0/1613764344 shutdown_connections 2024-03-23T11:31:25.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.210+0000 7f02dfe00640 1 -- 172.21.15.62:0/1613764344 wait complete. 2024-03-23T11:31:25.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.214+0000 7f02dfe00640 1 Processor -- start 2024-03-23T11:31:25.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.214+0000 7f02dfe00640 1 -- start start 2024-03-23T11:31:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.216+0000 7f02dfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02d806c300 0x7f02d813d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.217+0000 7f02dfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d806c6e0 0x7f02d813ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.217+0000 7f02dfe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d80cafa0 0x7f02d813b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.217+0000 7f02dfe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d813d5c0 con 0x7f02d806c300 2024-03-23T11:31:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.217+0000 7f02dfe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d813b290 con 0x7f02d80cafa0 2024-03-23T11:31:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.217+0000 7f02dfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d813bfe0 con 0x7f02d806c6e0 2024-03-23T11:31:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.217+0000 7f02dd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02d806c300 0x7f02d813d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.217+0000 7f02dd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02d806c300 0x7f02d813d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48106/0 (socket says 172.21.15.62:48106) 2024-03-23T11:31:25.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.217+0000 7f02dd400640 1 -- 172.21.15.62:0/1253627493 learned_addr learned my addr 172.21.15.62:0/1253627493 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:25.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.217+0000 7f02dde00640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d80cafa0 0x7f02d813b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:25.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.217+0000 7f02dca00640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d806c6e0 0x7f02d813ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:25.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.218+0000 7f02dd400640 1 -- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d806c6e0 msgr2=0x7f02d813ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:25.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.218+0000 7f02dd400640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d806c6e0 0x7f02d813ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:25.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.218+0000 7f02dd400640 1 -- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d80cafa0 msgr2=0x7f02d813b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:25.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.218+0000 7f02dd400640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d80cafa0 0x7f02d813b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:25.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.218+0000 7f02dd400640 1 -- 172.21.15.62:0/1253627493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d0009a70 con 0x7f02d806c300 2024-03-23T11:31:25.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.218+0000 7f02dca00640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d806c6e0 0x7f02d813ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:31:25.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.218+0000 7f02dd400640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02d806c300 0x7f02d813d080 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00c450 tx=0x7f02cc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:25.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.218+0000 7f02d5600640 1 -- 172.21.15.62:0/1253627493 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc0052d0 con 0x7f02d806c300 2024-03-23T11:31:25.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.219+0000 7f02d5600640 1 -- 172.21.15.62:0/1253627493 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc005430 con 0x7f02d806c300 2024-03-23T11:31:25.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.219+0000 7f02d5600640 1 -- 172.21.15.62:0/1253627493 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc012870 con 0x7f02d806c300 2024-03-23T11:31:25.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.219+0000 7f02dfe00640 1 -- 172.21.15.62:0/1253627493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d813c2c0 con 0x7f02d806c300 2024-03-23T11:31:25.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.219+0000 7f02dfe00640 1 -- 172.21.15.62:0/1253627493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d813c880 con 0x7f02d806c300 2024-03-23T11:31:25.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.220+0000 7f02dfe00640 1 -- 172.21.15.62:0/1253627493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d8066700 con 0x7f02d806c300 2024-03-23T11:31:25.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.221+0000 7f02d5600640 1 -- 172.21.15.62:0/1253627493 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f02cc0129d0 con 0x7f02d806c300 2024-03-23T11:31:25.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.222+0000 7f02d5600640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7a0 0x7f02b806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:25.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.222+0000 7f02d5600640 1 -- 172.21.15.62:0/1253627493 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02cc088820 con 0x7f02d806c300 2024-03-23T11:31:25.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.223+0000 7f02dca00640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7a0 0x7f02b806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:25.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.224+0000 7f02dca00640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7a0 0x7f02b806cc60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f02d815d910 tx=0x7f02c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:25.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.226+0000 7f02d5600640 1 -- 172.21.15.62:0/1253627493 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f02cc053cd0 con 0x7f02d806c300 2024-03-23T11:31:25.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:25.471+0000 7f02dfe00640 1 -- 172.21.15.62:0/1253627493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f02d8066c90 con 0x7f02d806c300 2024-03-23T11:31:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.425+0000 7f02d5600640 1 -- 172.21.15.62:0/1253627493 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v93) v1 ==== 118+0+0 (secure 0 0 0) 0x7f02cc088070 con 0x7f02d806c300 2024-03-23T11:31:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 -- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7a0 msgr2=0x7f02b806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7a0 0x7f02b806cc60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f02d815d910 tx=0x7f02c4009040 comp rx=0 tx=0).stop 2024-03-23T11:31:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 -- 172.21.15.62:0/1253627493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02d806c300 msgr2=0x7f02d813d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02d806c300 0x7f02d813d080 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00c450 tx=0x7f02cc00c910 comp rx=0 tx=0).stop 2024-03-23T11:31:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 -- 172.21.15.62:0/1253627493 shutdown_connections 2024-03-23T11:31:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f02b806a7a0 0x7f02b806cc60 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02d80cafa0 0x7f02d813b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02d806c6e0 0x7f02d813ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 --2- 172.21.15.62:0/1253627493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02d806c300 0x7f02d813d080 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 -- 172.21.15.62:0/1253627493 >> 172.21.15.62:0/1253627493 conn(0x7f02d80b02e0 msgr2=0x7f02d80afad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 -- 172.21.15.62:0/1253627493 shutdown_connections 2024-03-23T11:31:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.429+0000 7f02dfe00640 1 -- 172.21.15.62:0/1253627493 wait complete. 2024-03-23T11:31:26.441 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T11:31:26.441 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T11:31:26.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.677+0000 7fbf51e00640 1 Processor -- start 2024-03-23T11:31:26.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.677+0000 7fbf51e00640 1 -- start start 2024-03-23T11:31:26.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.679+0000 7fbf51e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbf4c156920 0x7fbf4c1594c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:26.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.679+0000 7fbf51e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c06de90 0x7fbf4c0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:26.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.679+0000 7fbf51e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf4c0cd3d0 con 0x7fbf4c156d00 2024-03-23T11:31:26.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.679+0000 7fbf51e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4c0cd6b0 con 0x7fbf4c156920 2024-03-23T11:31:26.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.679+0000 7fbf51e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c0cd990 con 0x7fbf4c06de90 2024-03-23T11:31:26.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.680+0000 7fbf4aa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbf4c156920 0x7fbf4c1594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:26.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.680+0000 7fbf4b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c06de90 0x7fbf4c0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:26.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.680+0000 7fbf4aa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbf4c156920 0x7fbf4c1594c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:42722/0 (socket says 172.21.15.62:42722) 2024-03-23T11:31:26.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.680+0000 7fbf4aa00640 1 -- 172.21.15.62:0/3892156538 learned_addr learned my addr 172.21.15.62:0/3892156538 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:26.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.680+0000 7fbf4aa00640 1 -- 172.21.15.62:0/3892156538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c06de90 msgr2=0x7fbf4c0b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:26.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.680+0000 7fbf4aa00640 1 --2- 172.21.15.62:0/3892156538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c06de90 0x7fbf4c0b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:26.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.680+0000 7fbf4aa00640 1 -- 172.21.15.62:0/3892156538 >> v1:172.21.15.120:6789/0 conn(0x7fbf4c156d00 legacy=0x7fbf4c06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:26.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.680+0000 7fbf4aa00640 1 -- 172.21.15.62:0/3892156538 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c0cdc70 con 0x7fbf4c156920 2024-03-23T11:31:26.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.682+0000 7fbf4aa00640 1 --2- 172.21.15.62:0/3892156538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c156920 0x7fbf4c1594c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0098a0 tx=0x7fbf3c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c38f7ed9f567921a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:26.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.682+0000 7fbf49600640 1 -- 172.21.15.62:0/3892156538 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c031640 con 0x7fbf4c156920 2024-03-23T11:31:26.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.682+0000 7fbf49600640 1 -- 172.21.15.62:0/3892156538 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c0317a0 con 0x7fbf4c156920 2024-03-23T11:31:26.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.682+0000 7fbf49600640 1 -- 172.21.15.62:0/3892156538 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c031b30 con 0x7fbf4c156920 2024-03-23T11:31:26.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.683+0000 7fbf51e00640 1 -- 172.21.15.62:0/3892156538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c156920 msgr2=0x7fbf4c1594c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:26.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.683+0000 7fbf51e00640 1 --2- 172.21.15.62:0/3892156538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c156920 0x7fbf4c1594c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0098a0 tx=0x7fbf3c02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:31:26.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.683+0000 7fbf51e00640 1 -- 172.21.15.62:0/3892156538 shutdown_connections 2024-03-23T11:31:26.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.683+0000 7fbf51e00640 1 --2- 172.21.15.62:0/3892156538 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c06de90 0x7fbf4c0b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:26.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.683+0000 7fbf51e00640 1 --2- 172.21.15.62:0/3892156538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c156920 0x7fbf4c1594c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:26.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.683+0000 7fbf51e00640 1 -- 172.21.15.62:0/3892156538 >> 172.21.15.62:0/3892156538 conn(0x7fbf4c0b02e0 msgr2=0x7fbf4c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:26.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.683+0000 7fbf51e00640 1 -- 172.21.15.62:0/3892156538 shutdown_connections 2024-03-23T11:31:26.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.683+0000 7fbf51e00640 1 -- 172.21.15.62:0/3892156538 wait complete. 2024-03-23T11:31:26.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.685+0000 7fbf51e00640 1 Processor -- start 2024-03-23T11:31:26.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.685+0000 7fbf51e00640 1 -- start start 2024-03-23T11:31:26.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.686+0000 7fbf51e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c06de90 0x7fbf4c13fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:26.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.686+0000 7fbf51e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c156920 0x7fbf4c158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:26.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.686+0000 7fbf51e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c156d00 0x7fbf4c1f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:26.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.686+0000 7fbf51e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c14f990 con 0x7fbf4c156d00 2024-03-23T11:31:26.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.686+0000 7fbf51e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c14fc70 con 0x7fbf4c156920 2024-03-23T11:31:26.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.686+0000 7fbf51e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c14ff50 con 0x7fbf4c06de90 2024-03-23T11:31:26.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.686+0000 7fbf4b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c156d00 0x7fbf4c1f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:26.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.686+0000 7fbf4b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c156d00 0x7fbf4c1f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48108/0 (socket says 172.21.15.62:48108) 2024-03-23T11:31:26.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.686+0000 7fbf4b400640 1 -- 172.21.15.62:0/3998037357 learned_addr learned my addr 172.21.15.62:0/3998037357 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:26.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.687+0000 7fbf4aa00640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c06de90 0x7fbf4c13fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:26.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.687+0000 7fbf4a000640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c156920 0x7fbf4c158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:26.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.687+0000 7fbf4b400640 1 -- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c06de90 msgr2=0x7fbf4c13fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:26.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.687+0000 7fbf4b400640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c06de90 0x7fbf4c13fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:26.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.687+0000 7fbf4b400640 1 -- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c156920 msgr2=0x7fbf4c158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:26.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.687+0000 7fbf4b400640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c156920 0x7fbf4c158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:26.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.687+0000 7fbf4b400640 1 -- 172.21.15.62:0/3998037357 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c0094d0 con 0x7fbf4c156d00 2024-03-23T11:31:26.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.687+0000 7fbf4a000640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c156920 0x7fbf4c158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:31:26.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.687+0000 7fbf4b400640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c156d00 0x7fbf4c1f22b0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400efc0 tx=0x7fbf4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:26.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.687+0000 7fbf43400640 1 -- 172.21.15.62:0/3998037357 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44014480 con 0x7fbf4c156d00 2024-03-23T11:31:26.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.688+0000 7fbf43400640 1 -- 172.21.15.62:0/3998037357 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4400f040 con 0x7fbf4c156d00 2024-03-23T11:31:26.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.688+0000 7fbf43400640 1 -- 172.21.15.62:0/3998037357 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44014820 con 0x7fbf4c156d00 2024-03-23T11:31:26.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.688+0000 7fbf51e00640 1 -- 172.21.15.62:0/3998037357 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4c150230 con 0x7fbf4c156d00 2024-03-23T11:31:26.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.688+0000 7fbf51e00640 1 -- 172.21.15.62:0/3998037357 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4c140240 con 0x7fbf4c156d00 2024-03-23T11:31:26.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.689+0000 7fbf51e00640 1 -- 172.21.15.62:0/3998037357 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4c150750 con 0x7fbf4c156d00 2024-03-23T11:31:26.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.690+0000 7fbf43400640 1 -- 172.21.15.62:0/3998037357 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbf44014980 con 0x7fbf4c156d00 2024-03-23T11:31:26.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.690+0000 7fbf43400640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 0x7fbf2c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:26.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.691+0000 7fbf43400640 1 -- 172.21.15.62:0/3998037357 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf44089ce0 con 0x7fbf4c156d00 2024-03-23T11:31:26.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.691+0000 7fbf4aa00640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 0x7fbf2c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:26.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.692+0000 7fbf4aa00640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 0x7fbf2c06cc60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c036000 tx=0x7fbf3c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:26.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.695+0000 7fbf43400640 1 -- 172.21.15.62:0/3998037357 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf4c150750 con 0x7fbf4c156d00 2024-03-23T11:31:26.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:26.932+0000 7fbf51e00640 1 -- 172.21.15.62:0/3998037357 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbf4c150750 con 0x7fbf4c156d00 2024-03-23T11:31:27.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.589+0000 7fbf43400640 1 -- 172.21.15.62:0/3998037357 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbf4c150750 con 0x7fbf4c156d00 2024-03-23T11:31:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.592+0000 7fbf51e00640 1 -- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 msgr2=0x7fbf2c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.592+0000 7fbf51e00640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 0x7fbf2c06cc60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c036000 tx=0x7fbf3c03c040 comp rx=0 tx=0).stop 2024-03-23T11:31:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.592+0000 7fbf51e00640 1 -- 172.21.15.62:0/3998037357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c156d00 msgr2=0x7fbf4c1f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.592+0000 7fbf51e00640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c156d00 0x7fbf4c1f22b0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400efc0 tx=0x7fbf4400c490 comp rx=0 tx=0).stop 2024-03-23T11:31:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.593+0000 7fbf51e00640 1 -- 172.21.15.62:0/3998037357 shutdown_connections 2024-03-23T11:31:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.593+0000 7fbf51e00640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbf2c06a7a0 0x7fbf2c06cc60 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.593+0000 7fbf51e00640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c156d00 0x7fbf4c1f22b0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.593+0000 7fbf51e00640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf4c156920 0x7fbf4c158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.593+0000 7fbf51e00640 1 --2- 172.21.15.62:0/3998037357 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf4c06de90 0x7fbf4c13fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.593+0000 7fbf51e00640 1 -- 172.21.15.62:0/3998037357 >> 172.21.15.62:0/3998037357 conn(0x7fbf4c0b02e0 msgr2=0x7fbf4c0702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:27.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.593+0000 7fbf51e00640 1 -- 172.21.15.62:0/3998037357 shutdown_connections 2024-03-23T11:31:27.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.593+0000 7fbf51e00640 1 -- 172.21.15.62:0/3998037357 wait complete. 2024-03-23T11:31:27.605 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T11:31:27.605 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:31:27.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.836+0000 7fa7fd800640 1 Processor -- start 2024-03-23T11:31:27.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.836+0000 7fa7fd800640 1 -- start start 2024-03-23T11:31:27.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.837+0000 7fa7fd800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa7f806fd20 0x7fa7f8070100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:27.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.837+0000 7fa7fd800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa7f80b43d0 0x7fa7f815cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:27.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.837+0000 7fa7fd800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7f815d1b0 0x7fa7f8160550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:27.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.837+0000 7fa7fd800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f815df50 con 0x7fa7f80b43d0 2024-03-23T11:31:27.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.837+0000 7fa7fd800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f806af80 con 0x7fa7f806fd20 2024-03-23T11:31:27.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.837+0000 7fa7fd800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f8070640 con 0x7fa7f815d1b0 2024-03-23T11:31:27.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.838+0000 7fa7f6000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa7f80b43d0 0x7fa7f815cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:27.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.838+0000 7fa7f6000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa7f80b43d0 0x7fa7f815cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37644/0 (socket says 172.21.15.62:37644) 2024-03-23T11:31:27.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.838+0000 7fa7f6000640 1 -- 172.21.15.62:0/2595776455 learned_addr learned my addr 172.21.15.62:0/2595776455 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.838+0000 7fa7f6a00640 1 --2- 172.21.15.62:0/2595776455 >> v2:172.21.15.120:3300/0 conn(0x7fa7f806fd20 0x7fa7f8070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.838+0000 7fa7f7400640 1 --2- 172.21.15.62:0/2595776455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7f815d1b0 0x7fa7f8160550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.838+0000 7fa7f6000640 1 -- 172.21.15.62:0/2595776455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7f815d1b0 msgr2=0x7fa7f8160550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.838+0000 7fa7f6000640 1 --2- 172.21.15.62:0/2595776455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7f815d1b0 0x7fa7f8160550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.838+0000 7fa7f6000640 1 -- 172.21.15.62:0/2595776455 >> v2:172.21.15.120:3300/0 conn(0x7fa7f806fd20 msgr2=0x7fa7f8070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.838+0000 7fa7f6000640 1 --2- 172.21.15.62:0/2595776455 >> v2:172.21.15.120:3300/0 conn(0x7fa7f806fd20 0x7fa7f8070100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.838+0000 7fa7f6000640 1 -- 172.21.15.62:0/2595776455 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f815d6b0 con 0x7fa7f80b43d0 2024-03-23T11:31:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.839+0000 7fa7f6000640 1 --2- 172.21.15.62:0/2595776455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7f80b43d0 0x7fa7f815cc70 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80d17d0 tx=0x7fa7e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc99f387cc3aabe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.840+0000 7fa7f5600640 1 -- 172.21.15.62:0/2595776455 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0031870 con 0x7fa7f80b43d0 2024-03-23T11:31:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.840+0000 7fa7f5600640 1 -- 172.21.15.62:0/2595776455 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e00319d0 con 0x7fa7f80b43d0 2024-03-23T11:31:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.840+0000 7fa7f5600640 1 -- 172.21.15.62:0/2595776455 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0031d60 con 0x7fa7f80b43d0 2024-03-23T11:31:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.840+0000 7fa7fd800640 1 -- 172.21.15.62:0/2595776455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7f80b43d0 msgr2=0x7fa7f815cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.840+0000 7fa7fd800640 1 --2- 172.21.15.62:0/2595776455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7f80b43d0 0x7fa7f815cc70 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80d17d0 tx=0x7fa7e002b910 comp rx=0 tx=0).stop 2024-03-23T11:31:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.841+0000 7fa7fd800640 1 -- 172.21.15.62:0/2595776455 shutdown_connections 2024-03-23T11:31:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.841+0000 7fa7fd800640 1 --2- 172.21.15.62:0/2595776455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7f815d1b0 0x7fa7f8160550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.841+0000 7fa7fd800640 1 --2- 172.21.15.62:0/2595776455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7f80b43d0 0x7fa7f815cc70 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.841+0000 7fa7fd800640 1 --2- 172.21.15.62:0/2595776455 >> v2:172.21.15.120:3300/0 conn(0x7fa7f806fd20 0x7fa7f8070100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.841+0000 7fa7fd800640 1 -- 172.21.15.62:0/2595776455 >> 172.21.15.62:0/2595776455 conn(0x7fa7f806e580 msgr2=0x7fa7f806e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.841+0000 7fa7fd800640 1 -- 172.21.15.62:0/2595776455 shutdown_connections 2024-03-23T11:31:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.841+0000 7fa7fd800640 1 -- 172.21.15.62:0/2595776455 wait complete. 2024-03-23T11:31:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.842+0000 7fa7fd800640 1 Processor -- start 2024-03-23T11:31:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.843+0000 7fa7fd800640 1 -- start start 2024-03-23T11:31:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.843+0000 7fa7fd800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7f806fd20 0x7fa7f815bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7fd800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7f80b43d0 0x7fa7f81f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7fd800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7f815d1b0 0x7fa7f80cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7fd800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f81f2de0 con 0x7fa7f806fd20 2024-03-23T11:31:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7fd800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f80cabe0 con 0x7fa7f80b43d0 2024-03-23T11:31:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7fd800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f80caec0 con 0x7fa7f815d1b0 2024-03-23T11:31:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7f6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7f806fd20 0x7fa7f815bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7f6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7f806fd20 0x7fa7f815bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37656/0 (socket says 172.21.15.62:37656) 2024-03-23T11:31:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7f6a00640 1 -- 172.21.15.62:0/2775571799 learned_addr learned my addr 172.21.15.62:0/2775571799 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7f6a00640 1 -- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7f815d1b0 msgr2=0x7fa7f80cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7f6a00640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7f815d1b0 0x7fa7f80cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7f6a00640 1 -- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7f80b43d0 msgr2=0x7fa7f81f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7f6a00640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7f80b43d0 0x7fa7f81f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.844+0000 7fa7f6a00640 1 -- 172.21.15.62:0/2775571799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e00094d0 con 0x7fa7f806fd20 2024-03-23T11:31:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.845+0000 7fa7f6a00640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7f806fd20 0x7fa7f815bf20 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fa7e800d5f0 tx=0x7fa7e800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.845+0000 7fa7eea00640 1 -- 172.21.15.62:0/2775571799 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e80125f0 con 0x7fa7f806fd20 2024-03-23T11:31:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.845+0000 7fa7eea00640 1 -- 172.21.15.62:0/2775571799 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e8012750 con 0x7fa7f806fd20 2024-03-23T11:31:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.845+0000 7fa7fd800640 1 -- 172.21.15.62:0/2775571799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f80cb1a0 con 0x7fa7f806fd20 2024-03-23T11:31:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.845+0000 7fa7eea00640 1 -- 172.21.15.62:0/2775571799 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8012b00 con 0x7fa7f806fd20 2024-03-23T11:31:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.845+0000 7fa7fd800640 1 -- 172.21.15.62:0/2775571799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f81f3480 con 0x7fa7f806fd20 2024-03-23T11:31:27.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.846+0000 7fa7fd800640 1 -- 172.21.15.62:0/2775571799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f81f3610 con 0x7fa7f806fd20 2024-03-23T11:31:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.847+0000 7fa7eea00640 1 -- 172.21.15.62:0/2775571799 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa7e8012c60 con 0x7fa7f806fd20 2024-03-23T11:31:27.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.848+0000 7fa7eea00640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a7a0 0x7fa7d406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:27.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.848+0000 7fa7eea00640 1 -- 172.21.15.62:0/2775571799 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7e8089310 con 0x7fa7f806fd20 2024-03-23T11:31:27.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.849+0000 7fa7f6000640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a7a0 0x7fa7d406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:27.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.850+0000 7fa7f6000640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a7a0 0x7fa7d406cc60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0036000 tx=0x7fa7e0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:27.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:27.852+0000 7fa7eea00640 1 -- 172.21.15.62:0/2775571799 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7e8054690 con 0x7fa7f806fd20 2024-03-23T11:31:28.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.092+0000 7fa7fd800640 1 -- 172.21.15.62:0/2775571799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7f80672c0 con 0x7fa7f806fd20 2024-03-23T11:31:28.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.093+0000 7fa7eea00640 1 -- 172.21.15.62:0/2775571799 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fa7e8057cb0 con 0x7fa7f806fd20 2024-03-23T11:31:28.083 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:28.083 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":94,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4129977589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4129977589},{"type":"v1","addr":"172.21.15.62:6835","nonce":4129977589}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:31:27.575868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9006},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9006":{"gid":9006,"name":"b","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/2585024035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":2585024035},{"type":"v1","addr":"172.21.15.120:6837","nonce":2585024035}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:31:28.083 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 94 2024-03-23T11:31:28.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 -- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a7a0 msgr2=0x7fa7d406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:28.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a7a0 0x7fa7d406cc60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0036000 tx=0x7fa7e0030000 comp rx=0 tx=0).stop 2024-03-23T11:31:28.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 -- 172.21.15.62:0/2775571799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7f806fd20 msgr2=0x7fa7f815bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:28.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7f806fd20 0x7fa7f815bf20 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fa7e800d5f0 tx=0x7fa7e800dab0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 -- 172.21.15.62:0/2775571799 shutdown_connections 2024-03-23T11:31:28.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7d406a7a0 0x7fa7d406cc60 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7f815d1b0 0x7fa7f80cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7f80b43d0 0x7fa7f81f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 --2- 172.21.15.62:0/2775571799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7f806fd20 0x7fa7f815bf20 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 -- 172.21.15.62:0/2775571799 >> 172.21.15.62:0/2775571799 conn(0x7fa7f806e580 msgr2=0x7fa7f80b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:28.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 -- 172.21.15.62:0/2775571799 shutdown_connections 2024-03-23T11:31:28.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.096+0000 7fa7fd800640 1 -- 172.21.15.62:0/2775571799 wait complete. 2024-03-23T11:31:28.105 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:31:28.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.340+0000 7f4915a00640 1 Processor -- start 2024-03-23T11:31:28.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.340+0000 7f4915a00640 1 -- start start 2024-03-23T11:31:28.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.341+0000 7f4915a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f49100ce8d0 0x7f49100af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.341+0000 7f4915a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f49100cecb0 0x7f49100ad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.341+0000 7f4915a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49100addb0 0x7f49100ae240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.341+0000 7f4915a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49100cac10 con 0x7f49100cecb0 2024-03-23T11:31:28.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.341+0000 7f4915a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49100cc860 con 0x7f49100ce8d0 2024-03-23T11:31:28.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.341+0000 7f4915a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49100cbfc0 con 0x7f49100addb0 2024-03-23T11:31:28.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.342+0000 7f490f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f49100cecb0 0x7f49100ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:28.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.342+0000 7f490f400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f49100cecb0 0x7f49100ad870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37666/0 (socket says 172.21.15.62:37666) 2024-03-23T11:31:28.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.342+0000 7f490f400640 1 -- 172.21.15.62:0/327583308 learned_addr learned my addr 172.21.15.62:0/327583308 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:28.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.342+0000 7f490fe00640 1 --2- 172.21.15.62:0/327583308 >> v2:172.21.15.120:3300/0 conn(0x7f49100ce8d0 0x7f49100af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:28.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.342+0000 7f4915000640 1 --2- 172.21.15.62:0/327583308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49100addb0 0x7f49100ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:28.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.342+0000 7f490f400640 1 -- 172.21.15.62:0/327583308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49100addb0 msgr2=0x7f49100ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:28.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.342+0000 7f490f400640 1 --2- 172.21.15.62:0/327583308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49100addb0 0x7f49100ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.342+0000 7f490f400640 1 -- 172.21.15.62:0/327583308 >> v2:172.21.15.120:3300/0 conn(0x7f49100ce8d0 msgr2=0x7f49100af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:28.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.342+0000 7f490f400640 1 --2- 172.21.15.62:0/327583308 >> v2:172.21.15.120:3300/0 conn(0x7f49100ce8d0 0x7f49100af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.342+0000 7f490f400640 1 -- 172.21.15.62:0/327583308 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100cc2a0 con 0x7f49100cecb0 2024-03-23T11:31:28.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.343+0000 7f490f400640 1 --2- 172.21.15.62:0/327583308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49100cecb0 0x7f49100ad870 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f49100716d0 tx=0x7f48fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd70c2c7767abf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:28.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.344+0000 7f490ea00640 1 -- 172.21.15.62:0/327583308 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc031870 con 0x7f49100cecb0 2024-03-23T11:31:28.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.344+0000 7f490ea00640 1 -- 172.21.15.62:0/327583308 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc0319d0 con 0x7f49100cecb0 2024-03-23T11:31:28.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.344+0000 7f490ea00640 1 -- 172.21.15.62:0/327583308 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc031d60 con 0x7f49100cecb0 2024-03-23T11:31:28.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.344+0000 7f4915a00640 1 -- 172.21.15.62:0/327583308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49100cecb0 msgr2=0x7f49100ad870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:28.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.344+0000 7f4915a00640 1 --2- 172.21.15.62:0/327583308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49100cecb0 0x7f49100ad870 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f49100716d0 tx=0x7f48fc02b910 comp rx=0 tx=0).stop 2024-03-23T11:31:28.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.344+0000 7f4915a00640 1 -- 172.21.15.62:0/327583308 shutdown_connections 2024-03-23T11:31:28.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.344+0000 7f4915a00640 1 --2- 172.21.15.62:0/327583308 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49100addb0 0x7f49100ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.344+0000 7f4915a00640 1 --2- 172.21.15.62:0/327583308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49100cecb0 0x7f49100ad870 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.344+0000 7f4915a00640 1 --2- 172.21.15.62:0/327583308 >> v2:172.21.15.120:3300/0 conn(0x7f49100ce8d0 0x7f49100af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.344+0000 7f4915a00640 1 -- 172.21.15.62:0/327583308 >> 172.21.15.62:0/327583308 conn(0x7f49100b1fb0 msgr2=0x7f49100b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:28.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.345+0000 7f4915a00640 1 -- 172.21.15.62:0/327583308 shutdown_connections 2024-03-23T11:31:28.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.345+0000 7f4915a00640 1 -- 172.21.15.62:0/327583308 wait complete. 2024-03-23T11:31:28.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.347+0000 7f4915a00640 1 Processor -- start 2024-03-23T11:31:28.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.347+0000 7f4915a00640 1 -- start start 2024-03-23T11:31:28.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.348+0000 7f4915a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49100addb0 0x7f49100b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.348+0000 7f4915a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49100ce8d0 0x7f491015cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.348+0000 7f4915a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49100cecb0 0x7f49100b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.348+0000 7f4915a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49100af770 con 0x7f49100addb0 2024-03-23T11:31:28.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.348+0000 7f4915a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49100afa50 con 0x7f49100cecb0 2024-03-23T11:31:28.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.348+0000 7f4915a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49100afd30 con 0x7f49100ce8d0 2024-03-23T11:31:28.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.348+0000 7f490fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49100addb0 0x7f49100b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:28.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.348+0000 7f490fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49100addb0 0x7f49100b5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37676/0 (socket says 172.21.15.62:37676) 2024-03-23T11:31:28.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.348+0000 7f490fe00640 1 -- 172.21.15.62:0/3549736727 learned_addr learned my addr 172.21.15.62:0/3549736727 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.349+0000 7f490fe00640 1 -- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49100ce8d0 msgr2=0x7f491015cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.349+0000 7f490fe00640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49100ce8d0 0x7f491015cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.349+0000 7f490fe00640 1 -- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49100cecb0 msgr2=0x7f49100b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.349+0000 7f490fe00640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49100cecb0 0x7f49100b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.349+0000 7f490fe00640 1 -- 172.21.15.62:0/3549736727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc0094d0 con 0x7f49100addb0 2024-03-23T11:31:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.349+0000 7f490fe00640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49100addb0 0x7f49100b5d30 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f490000d800 tx=0x7f490000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.349+0000 7f4907e00640 1 -- 172.21.15.62:0/3549736727 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490000b6c0 con 0x7f49100addb0 2024-03-23T11:31:28.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.349+0000 7f4907e00640 1 -- 172.21.15.62:0/3549736727 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490000b820 con 0x7f49100addb0 2024-03-23T11:31:28.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.349+0000 7f4907e00640 1 -- 172.21.15.62:0/3549736727 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490000bbb0 con 0x7f49100addb0 2024-03-23T11:31:28.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.349+0000 7f4915a00640 1 -- 172.21.15.62:0/3549736727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49100b0010 con 0x7f49100addb0 2024-03-23T11:31:28.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.350+0000 7f4915a00640 1 -- 172.21.15.62:0/3549736727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491015d1a0 con 0x7f49100addb0 2024-03-23T11:31:28.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.351+0000 7f4915a00640 1 -- 172.21.15.62:0/3549736727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4910073540 con 0x7f49100addb0 2024-03-23T11:31:28.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.352+0000 7f4907e00640 1 -- 172.21.15.62:0/3549736727 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f490000bd10 con 0x7f49100addb0 2024-03-23T11:31:28.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.352+0000 7f4907e00640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48f006a7a0 0x7f48f006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.352+0000 7f4907e00640 1 -- 172.21.15.62:0/3549736727 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49000893d0 con 0x7f49100addb0 2024-03-23T11:31:28.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.353+0000 7f490f400640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48f006a7a0 0x7f48f006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:28.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.354+0000 7f490f400640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48f006a7a0 0x7f48f006cc60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f48fc036000 tx=0x7f48fc030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:28.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.357+0000 7f4907e00640 1 -- 172.21.15.62:0/3549736727 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4900054750 con 0x7f49100addb0 2024-03-23T11:31:28.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.562+0000 7f4915a00640 1 -- 172.21.15.62:0/3549736727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49100672c0 con 0x7f49100addb0 2024-03-23T11:31:28.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.565+0000 7f4907e00640 1 -- 172.21.15.62:0/3549736727 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16059 (secure 0 0 0) 0x7f4900057d70 con 0x7f49100addb0 2024-03-23T11:31:28.555 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:28.556 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":118,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:31:25.048209+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:31:28.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.567+0000 7f4915a00640 1 -- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48f006a7a0 msgr2=0x7f48f006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:28.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.567+0000 7f4915a00640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48f006a7a0 0x7f48f006cc60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f48fc036000 tx=0x7f48fc030000 comp rx=0 tx=0).stop 2024-03-23T11:31:28.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.568+0000 7f4915a00640 1 -- 172.21.15.62:0/3549736727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49100addb0 msgr2=0x7f49100b5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:28.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.568+0000 7f4915a00640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49100addb0 0x7f49100b5d30 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f490000d800 tx=0x7f490000dcc0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.568+0000 7f4915a00640 1 -- 172.21.15.62:0/3549736727 shutdown_connections 2024-03-23T11:31:28.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.568+0000 7f4915a00640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48f006a7a0 0x7f48f006cc60 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.568+0000 7f4915a00640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49100cecb0 0x7f49100b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.568+0000 7f4915a00640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49100ce8d0 0x7f491015cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.568+0000 7f4915a00640 1 --2- 172.21.15.62:0/3549736727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49100addb0 0x7f49100b5d30 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.568+0000 7f4915a00640 1 -- 172.21.15.62:0/3549736727 >> 172.21.15.62:0/3549736727 conn(0x7f49100b1fb0 msgr2=0x7f49100cb520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:28.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.568+0000 7f4915a00640 1 -- 172.21.15.62:0/3549736727 shutdown_connections 2024-03-23T11:31:28.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.568+0000 7f4915a00640 1 -- 172.21.15.62:0/3549736727 wait complete. 2024-03-23T11:31:28.583 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T11:31:28.584 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:31:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.807+0000 7f048ee00640 1 Processor -- start 2024-03-23T11:31:28.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.807+0000 7f048ee00640 1 -- start start 2024-03-23T11:31:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.809+0000 7f048ee00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0488160420 0x7f04881608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.809+0000 7f048ee00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04880ce510 con 0x7f04880b5db0 2024-03-23T11:31:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.809+0000 7f048ee00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04880cdc70 con 0x7f048815fcf0 2024-03-23T11:31:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.809+0000 7f048ee00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0488160df0 con 0x7f0488160420 2024-03-23T11:31:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.809+0000 7f048e400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f04880b5db0 0x7f04880b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42892/0 (socket says 172.21.15.62:42892) 2024-03-23T11:31:28.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.809+0000 7f048d000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0488160420 0x7f04881608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:28.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.809+0000 7f048e400640 1 -- 172.21.15.62:0/2777833165 learned_addr learned my addr 172.21.15.62:0/2777833165 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:28.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.810+0000 7f0487e00640 1 -- 172.21.15.62:0/2777833165 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096737067 0 0) 0x7f04880ce510 con 0x7f04880b5db0 2024-03-23T11:31:28.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.810+0000 7f048d000640 1 -- 172.21.15.62:0/2777833165 >> v1:172.21.15.120:6789/0 conn(0x7f048815fcf0 legacy=0x7f04880d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:28.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.810+0000 7f048d000640 1 -- 172.21.15.62:0/2777833165 >> v1:172.21.15.62:6789/0 conn(0x7f04880b5db0 legacy=0x7f04880b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:28.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.810+0000 7f048d000640 1 -- 172.21.15.62:0/2777833165 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04880cdf50 con 0x7f0488160420 2024-03-23T11:31:28.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.811+0000 7f048d000640 1 --2- 172.21.15.62:0/2777833165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0488160420 0x7f04881608b0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f04780097a0 tx=0x7f047802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573ba2cfbb8ee034 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:28.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.811+0000 7f0487e00640 1 -- 172.21.15.62:0/2777833165 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478031830 con 0x7f0488160420 2024-03-23T11:31:28.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.812+0000 7f0487e00640 1 -- 172.21.15.62:0/2777833165 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0478031990 con 0x7f0488160420 2024-03-23T11:31:28.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.812+0000 7f0487e00640 1 -- 172.21.15.62:0/2777833165 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478031d20 con 0x7f0488160420 2024-03-23T11:31:28.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.812+0000 7f048ee00640 1 -- 172.21.15.62:0/2777833165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0488160420 msgr2=0x7f04881608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:28.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.812+0000 7f048ee00640 1 --2- 172.21.15.62:0/2777833165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0488160420 0x7f04881608b0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f04780097a0 tx=0x7f047802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.812+0000 7f048ee00640 1 -- 172.21.15.62:0/2777833165 shutdown_connections 2024-03-23T11:31:28.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.812+0000 7f048ee00640 1 --2- 172.21.15.62:0/2777833165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0488160420 0x7f04881608b0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.812+0000 7f048ee00640 1 -- 172.21.15.62:0/2777833165 >> 172.21.15.62:0/2777833165 conn(0x7f0488089f60 msgr2=0x7f04880d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:28.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.812+0000 7f048ee00640 1 -- 172.21.15.62:0/2777833165 shutdown_connections 2024-03-23T11:31:28.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.812+0000 7f048ee00640 1 -- 172.21.15.62:0/2777833165 wait complete. 2024-03-23T11:31:28.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.814+0000 7f048ee00640 1 Processor -- start 2024-03-23T11:31:28.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.814+0000 7f048ee00640 1 -- start start 2024-03-23T11:31:28.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.815+0000 7f048ee00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04880b5db0 0x7f04880cf690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.815+0000 7f048ee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f048815fcf0 0x7f048814b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.815+0000 7f048ee00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0488160420 0x7f048814bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.815+0000 7f048ee00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04881e6c50 con 0x7f04880b5db0 2024-03-23T11:31:28.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.815+0000 7f048ee00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04881e62f0 con 0x7f0488160420 2024-03-23T11:31:28.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.815+0000 7f048ee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04881e6590 con 0x7f048815fcf0 2024-03-23T11:31:28.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.815+0000 7f048da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04880b5db0 0x7f04880cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:28.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.815+0000 7f048da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04880b5db0 0x7f04880cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37696/0 (socket says 172.21.15.62:37696) 2024-03-23T11:31:28.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.815+0000 7f048da00640 1 -- 172.21.15.62:0/665277691 learned_addr learned my addr 172.21.15.62:0/665277691 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:28.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f048d000640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f048815fcf0 0x7f048814b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:28.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f048e400640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0488160420 0x7f048814bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f048da00640 1 -- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f048815fcf0 msgr2=0x7f048814b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f048da00640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f048815fcf0 0x7f048814b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f048da00640 1 -- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0488160420 msgr2=0x7f048814bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f048da00640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0488160420 0x7f048814bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f048da00640 1 -- 172.21.15.62:0/665277691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0478009400 con 0x7f04880b5db0 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f048d000640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f048815fcf0 0x7f048814b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f048e400640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0488160420 0x7f048814bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f048da00640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04880b5db0 0x7f04880cf690 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f047c00c400 tx=0x7f047c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f0485600640 1 -- 172.21.15.62:0/665277691 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c0052d0 con 0x7f04880b5db0 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f0485600640 1 -- 172.21.15.62:0/665277691 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c005450 con 0x7f04880b5db0 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f0485600640 1 -- 172.21.15.62:0/665277691 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c007260 con 0x7f04880b5db0 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.816+0000 7f048ee00640 1 -- 172.21.15.62:0/665277691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04881e6870 con 0x7f04880b5db0 2024-03-23T11:31:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.817+0000 7f048ee00640 1 -- 172.21.15.62:0/665277691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04880cfbd0 con 0x7f04880b5db0 2024-03-23T11:31:28.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.818+0000 7f048ee00640 1 -- 172.21.15.62:0/665277691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04881e6d90 con 0x7f04880b5db0 2024-03-23T11:31:28.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.819+0000 7f0485600640 1 -- 172.21.15.62:0/665277691 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f047c0073c0 con 0x7f04880b5db0 2024-03-23T11:31:28.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.820+0000 7f0485600640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f046c06a7a0 0x7f046c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:28.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.820+0000 7f0485600640 1 -- 172.21.15.62:0/665277691 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f047c088150 con 0x7f04880b5db0 2024-03-23T11:31:28.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.820+0000 7f048d000640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f046c06a7a0 0x7f046c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:28.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.822+0000 7f048d000640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f046c06a7a0 0x7f046c06cc60 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f0478036000 tx=0x7f0478008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:28.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:28.824+0000 7f0485600640 1 -- 172.21.15.62:0/665277691 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f04881e6d90 con 0x7f04880b5db0 2024-03-23T11:31:29.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.022+0000 7f048ee00640 1 -- 172.21.15.62:0/665277691 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f04880b58b0 con 0x7f046c06a7a0 2024-03-23T11:31:29.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.046+0000 7f0485600640 1 -- 172.21.15.62:0/665277691 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+275865 (secure 0 0 0) 0x7f0478031c00 con 0x7f046c06a7a0 2024-03-23T11:31:29.040 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:29.042 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:31:29.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 -- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f046c06a7a0 msgr2=0x7f046c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f046c06a7a0 0x7f046c06cc60 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f0478036000 tx=0x7f0478008030 comp rx=0 tx=0).stop 2024-03-23T11:31:29.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 -- 172.21.15.62:0/665277691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04880b5db0 msgr2=0x7f04880cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04880b5db0 0x7f04880cf690 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f047c00c400 tx=0x7f047c00c8c0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 -- 172.21.15.62:0/665277691 shutdown_connections 2024-03-23T11:31:29.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f046c06a7a0 0x7f046c06cc60 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0488160420 0x7f048814bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f048815fcf0 0x7f048814b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 --2- 172.21.15.62:0/665277691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04880b5db0 0x7f04880cf690 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 -- 172.21.15.62:0/665277691 >> 172.21.15.62:0/665277691 conn(0x7f0488089f60 msgr2=0x7f04880b67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:29.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 -- 172.21.15.62:0/665277691 shutdown_connections 2024-03-23T11:31:29.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.052+0000 7f048ee00640 1 -- 172.21.15.62:0/665277691 wait complete. 2024-03-23T11:31:29.064 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-03-23T11:31:27.512010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261216,"kb_used_data":12284,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754713504,"statfs":{"total":773094113280,"available":772826628096,"internally_reserved":0,"allocated":12578816,"data_stored":6099745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.977252"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.546238+0000","last_change":"2024-03-23T11:31:22.546238+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.546238+0000","last_undegraded":"2024-03-23T11:31:22.546238+0000","last_fullsized":"2024-03-23T11:31:22.546238+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:21:25.524815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.540464+0000","last_change":"2024-03-23T11:31:22.540464+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.540464+0000","last_undegraded":"2024-03-23T11:31:22.540464+0000","last_fullsized":"2024-03-23T11:31:22.540464+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:51:44.775155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.547144+0000","last_change":"2024-03-23T11:31:22.547144+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.547144+0000","last_undegraded":"2024-03-23T11:31:22.547144+0000","last_fullsized":"2024-03-23T11:31:22.547144+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:53:45.695329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.546894+0000","last_change":"2024-03-23T11:31:22.546894+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.546894+0000","last_undegraded":"2024-03-23T11:31:22.546894+0000","last_fullsized":"2024-03-23T11:31:22.546894+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:12:13.014635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.547876+0000","last_change":"2024-03-23T11:31:22.547876+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.547876+0000","last_undegraded":"2024-03-23T11:31:22.547876+0000","last_fullsized":"2024-03-23T11:31:22.547876+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:46.152155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.548141+0000","last_change":"2024-03-23T11:31:22.548141+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.548141+0000","last_undegraded":"2024-03-23T11:31:22.548141+0000","last_fullsized":"2024-03-23T11:31:22.548141+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:24:08.473235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.541027+0000","last_change":"2024-03-23T11:31:22.541027+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.541027+0000","last_undegraded":"2024-03-23T11:31:22.541027+0000","last_fullsized":"2024-03-23T11:31:22.541027+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:14:30.980614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.554204+0000","last_change":"2024-03-23T11:31:22.554204+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.554204+0000","last_undegraded":"2024-03-23T11:31:22.554204+0000","last_fullsized":"2024-03-23T11:31:22.554204+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:36:56.686495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.546189+0000","last_change":"2024-03-23T11:31:22.546189+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.546189+0000","last_undegraded":"2024-03-23T11:31:22.546189+0000","last_fullsized":"2024-03-23T11:31:22.546189+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:25:12.126975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.548008+0000","last_change":"2024-03-23T11:31:22.548008+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.548008+0000","last_undegraded":"2024-03-23T11:31:22.548008+0000","last_fullsized":"2024-03-23T11:31:22.548008+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:08.838349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.552729+0000","last_change":"2024-03-23T11:31:22.552729+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.552729+0000","last_undegraded":"2024-03-23T11:31:22.552729+0000","last_fullsized":"2024-03-23T11:31:22.552729+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:46:53.689805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.552467+0000","last_change":"2024-03-23T11:31:22.552467+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.552467+0000","last_undegraded":"2024-03-23T11:31:22.552467+0000","last_fullsized":"2024-03-23T11:31:22.552467+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:48:53.978419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.546847+0000","last_change":"2024-03-23T11:31:22.546847+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.546847+0000","last_undegraded":"2024-03-23T11:31:22.546847+0000","last_fullsized":"2024-03-23T11:31:22.546847+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:02:56.394144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.555284+0000","last_change":"2024-03-23T11:31:22.555284+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.555284+0000","last_undegraded":"2024-03-23T11:31:22.555284+0000","last_fullsized":"2024-03-23T11:31:22.555284+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:08.838349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.546118+0000","last_change":"2024-03-23T11:31:22.546118+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.546118+0000","last_undegraded":"2024-03-23T11:31:22.546118+0000","last_fullsized":"2024-03-23T11:31:22.546118+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:21:01.865009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.553224+0000","last_change":"2024-03-23T11:31:22.553224+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.553224+0000","last_undegraded":"2024-03-23T11:31:22.553224+0000","last_fullsized":"2024-03-23T11:31:22.553224+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:31:43.172254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.547346+0000","last_change":"2024-03-23T11:31:22.547346+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.547346+0000","last_undegraded":"2024-03-23T11:31:22.547346+0000","last_fullsized":"2024-03-23T11:31:22.547346+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:23.091705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.554978+0000","last_change":"2024-03-23T11:31:22.554978+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.554978+0000","last_undegraded":"2024-03-23T11:31:22.554978+0000","last_fullsized":"2024-03-23T11:31:22.554978+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:29:42.109160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.546147+0000","last_change":"2024-03-23T11:31:22.546147+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.546147+0000","last_undegraded":"2024-03-23T11:31:22.546147+0000","last_fullsized":"2024-03-23T11:31:22.546147+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:14:30.980614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.546129+0000","last_change":"2024-03-23T11:31:22.546129+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.546129+0000","last_undegraded":"2024-03-23T11:31:22.546129+0000","last_fullsized":"2024-03-23T11:31:22.546129+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:41.262297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.545588+0000","last_change":"2024-03-23T11:31:22.545588+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.545588+0000","last_undegraded":"2024-03-23T11:31:22.545588+0000","last_fullsized":"2024-03-23T11:31:22.545588+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:59:59.275689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.546269+0000","last_change":"2024-03-23T11:31:22.546269+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.546269+0000","last_undegraded":"2024-03-23T11:31:22.546269+0000","last_fullsized":"2024-03-23T11:31:22.546269+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:00.392534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.553634+0000","last_change":"2024-03-23T11:31:22.553634+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.553634+0000","last_undegraded":"2024-03-23T11:31:22.553634+0000","last_fullsized":"2024-03-23T11:31:22.553634+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:29:34.803613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.548094+0000","last_change":"2024-03-23T11:31:22.548094+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.548094+0000","last_undegraded":"2024-03-23T11:31:22.548094+0000","last_fullsized":"2024-03-23T11:31:22.548094+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:29:42.109160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.554991+0000","last_change":"2024-03-23T11:31:22.554991+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.554991+0000","last_undegraded":"2024-03-23T11:31:22.554991+0000","last_fullsized":"2024-03-23T11:31:22.554991+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:24:08.473235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.552081+0000","last_change":"2024-03-23T11:31:22.552081+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.552081+0000","last_undegraded":"2024-03-23T11:31:22.552081+0000","last_fullsized":"2024-03-23T11:31:22.552081+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:26.294819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.547039+0000","last_change":"2024-03-23T11:31:22.547039+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.547039+0000","last_undegraded":"2024-03-23T11:31:22.547039+0000","last_fullsized":"2024-03-23T11:31:22.547039+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:55:31.725691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.546188+0000","last_change":"2024-03-23T11:31:22.546188+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.546188+0000","last_undegraded":"2024-03-23T11:31:22.546188+0000","last_fullsized":"2024-03-23T11:31:22.546188+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:29:14.159484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.541078+0000","last_change":"2024-03-23T11:31:22.541078+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.541078+0000","last_undegraded":"2024-03-23T11:31:22.541078+0000","last_fullsized":"2024-03-23T11:31:22.541078+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:45:52.136449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.547125+0000","last_change":"2024-03-23T11:31:22.547125+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.547125+0000","last_undegraded":"2024-03-23T11:31:22.547125+0000","last_fullsized":"2024-03-23T11:31:22.547125+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:37:48.520573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.551993+0000","last_change":"2024-03-23T11:31:22.551993+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.551993+0000","last_undegraded":"2024-03-23T11:31:22.551993+0000","last_fullsized":"2024-03-23T11:31:22.551993+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:45:53.606801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T11:31:23.533089+0000","last_change":"2024-03-23T11:31:23.533089+0000","last_active":"2024-03-23T11:31:23.533089+0000","last_peered":"2024-03-23T11:31:23.533089+0000","last_clean":"2024-03-23T11:31:23.533089+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:23.533089+0000","last_undegraded":"2024-03-23T11:31:23.533089+0000","last_fullsized":"2024-03-23T11:31:23.533089+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":115,"state":"creating+peering","last_fresh":"2024-03-23T11:31:22.546182+0000","last_change":"2024-03-23T11:31:22.546182+0000","last_active":"2024-03-23T11:31:22.528125+0000","last_peered":"2024-03-23T11:31:22.528125+0000","last_clean":"2024-03-23T11:31:22.528125+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T11:31:22.546182+0000","last_undegraded":"2024-03-23T11:31:22.546182+0000","last_fullsized":"2024-03-23T11:31:22.546182+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:45:52.136449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":263,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-23T11:31:19.753120+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:31:19.753120+0000","last_peered":"2024-03-23T11:31:19.753120+0000","last_clean":"2024-03-23T11:31:19.753120+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:31:19.753120+0000","last_undegraded":"2024-03-23T11:31:19.753120+0000","last_fullsized":"2024-03-23T11:31:19.753120+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33088,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338752,"statfs":{"total":96636764160,"available":96602882048,"internally_reserved":0,"allocated":2019328,"data_stored":1205495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.0960000000000001,"15min":1.0169999999999999},"min":{"1min":0.77500000000000002,"5min":0.5,"15min":0.5},"max":{"1min":1.556,"5min":2.8370000000000002,"15min":2.8370000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.1120000000000001,"15min":1.052},"min":{"1min":0.86899999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3300000000000001,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.79500000000000004}]},{"osd":1,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.2929999999999999,"15min":1.212},"min":{"1min":0.80000000000000004,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.6559999999999999,"5min":3.569,"15min":3.569},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.306,"15min":1.27},"min":{"1min":1.0649999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.5649999999999999,"5min":1.8009999999999999,"15min":2.4350000000000001},"last":1.2370000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1319999999999999,"15min":1.0900000000000001},"min":{"1min":0.65500000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.4379999999999999,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.262,"5min":1.2410000000000001,"15min":1.1950000000000001},"min":{"1min":0.81499999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.7130000000000001,"5min":3.524,"15min":3.524},"last":1.1319999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.22,"15min":1.1870000000000001},"min":{"1min":0.996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.55,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.091},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3240000000000001,"15min":1.2769999999999999},"min":{"1min":1.026,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.605,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":1.3340000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.1890000000000001,"15min":1.212},"min":{"1min":0.73199999999999998,"5min":0.65300000000000002,"15min":0.54300000000000004},"max":{"1min":1.496,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":1.425},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.411,"15min":1.395},"min":{"1min":1.2050000000000001,"5min":0.63800000000000001,"15min":0.60199999999999998},"max":{"1min":1.744,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":1.1899999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.3799999999999999,"15min":1.3500000000000001},"min":{"1min":0.877,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.7789999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":1.472},{"interface":"front","average":{"1min":1.466,"5min":1.4730000000000001,"15min":1.425},"min":{"1min":0.96399999999999997,"5min":0.77100000000000002,"15min":0.58199999999999996},"max":{"1min":1.869,"5min":2.0219999999999998,"15min":2.0859999999999999},"last":1.542}]},{"osd":6,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.4279999999999999,"15min":1.4259999999999999},"min":{"1min":1.0489999999999999,"5min":0.73599999999999999,"15min":0.61399999999999999},"max":{"1min":1.923,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.383,"5min":1.401,"15min":1.373},"min":{"1min":1.03,"5min":0.61699999999999999,"15min":0.59899999999999998},"max":{"1min":1.794,"5min":3.218,"15min":3.218},"last":1.637}]}]},{"osd":6,"up_from":14,"seq":60129542298,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1421312,"data_stored":614458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1020000000000001,"15min":1.1120000000000001},"min":{"1min":0.72999999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.7450000000000001,"5min":2.403,"15min":3.9140000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.2010000000000001,"15min":1.1579999999999999},"min":{"1min":0.92400000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.597,"5min":1.8600000000000001,"15min":6.6539999999999999},"last":1.53}]},{"osd":1,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.1859999999999999,"15min":1.206},"min":{"1min":0.84899999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.627,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.161,"15min":1.2070000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.448,"5min":1.9510000000000001,"15min":2.0859999999999999},"last":1.2549999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.161,"15min":1.214},"min":{"1min":0.81499999999999995,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.5980000000000001,"5min":1.649,"15min":3.73},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.3220000000000001,"15min":1.296},"min":{"1min":1.077,"5min":0.73499999999999999,"15min":0.68100000000000005},"max":{"1min":1.6859999999999999,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.393}]},{"osd":3,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.2470000000000001,"15min":1.278},"min":{"1min":1.0840000000000001,"5min":0.76700000000000002,"15min":0.749},"max":{"1min":1.738,"5min":1.9890000000000001,"15min":12.914},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.351,"5min":1.3220000000000001,"15min":1.337},"min":{"1min":1.0169999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6220000000000001,"5min":1.9670000000000001,"15min":3.8100000000000001},"last":0.85099999999999998}]},{"osd":4,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.3260000000000001,"15min":1.335},"min":{"1min":0.89400000000000002,"5min":0.68000000000000005,"15min":0.51100000000000001},"max":{"1min":1.8100000000000001,"5min":2.4420000000000002,"15min":2.702},"last":1.556},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.3089999999999999,"15min":1.361},"min":{"1min":0.92200000000000004,"5min":0.752,"15min":0.752},"max":{"1min":1.552,"5min":2.1080000000000001,"15min":7.2140000000000004},"last":1.4470000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4119999999999999,"15min":1.4610000000000001},"min":{"1min":1.1100000000000001,"5min":0.92900000000000005,"15min":0.73899999999999999},"max":{"1min":1.6799999999999999,"5min":2.0270000000000001,"15min":3.097},"last":1.143},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.458,"15min":1.468},"min":{"1min":1.1799999999999999,"5min":0.86899999999999999,"15min":0.72499999999999998},"max":{"1min":1.8680000000000001,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":1.6839999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.5609999999999999,"15min":1.597},"min":{"1min":1.377,"5min":0.97799999999999998,"15min":0.80400000000000005},"max":{"1min":1.738,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":1.593},{"interface":"front","average":{"1min":1.383,"5min":1.482,"15min":1.4710000000000001},"min":{"1min":1.0049999999999999,"5min":0.70399999999999996,"15min":0.57699999999999996},"max":{"1min":1.72,"5min":4.085,"15min":4.085},"last":1.47}]}]},{"osd":5,"up_from":14,"seq":60129542298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1425408,"data_stored":615127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.1519999999999999,"15min":1.1359999999999999},"min":{"1min":0.81899999999999995,"5min":0.65500000000000003,"15min":0.55400000000000005},"max":{"1min":1.538,"5min":2.0179999999999998,"15min":2.2999999999999998},"last":1.204},{"interface":"front","average":{"1min":1.214,"5min":1.155,"15min":1.1479999999999999},"min":{"1min":0.624,"5min":0.624,"15min":0.47799999999999998},"max":{"1min":1.6619999999999999,"5min":2.1190000000000002,"15min":2.355},"last":1.0329999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.198,"15min":1.173},"min":{"1min":0.94399999999999995,"5min":0.76700000000000002,"15min":0.65300000000000002},"max":{"1min":1.605,"5min":1.6699999999999999,"15min":2.2559999999999998},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.202,"15min":1.1910000000000001},"min":{"1min":0.81599999999999995,"5min":0.69299999999999995,"15min":0.495},"max":{"1min":1.5589999999999999,"5min":1.627,"15min":1.8580000000000001},"last":1.375}]},{"osd":2,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2010000000000001,"15min":1.2250000000000001},"min":{"1min":0.93000000000000005,"5min":0.70999999999999996,"15min":0.60199999999999998},"max":{"1min":1.5269999999999999,"5min":1.6659999999999999,"15min":2.012},"last":1.147},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.2969999999999999,"15min":1.254},"min":{"1min":1.117,"5min":0.77600000000000002,"15min":0.64400000000000002},"max":{"1min":1.76,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.2989999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2769999999999999,"15min":1.29},"min":{"1min":0.89300000000000002,"5min":0.69299999999999995,"15min":0.66800000000000004},"max":{"1min":1.611,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3879999999999999,"15min":1.381},"min":{"1min":1.02,"5min":0.78600000000000003,"15min":0.73199999999999998},"max":{"1min":1.7509999999999999,"5min":2.0750000000000002,"15min":2.3999999999999999},"last":1.5569999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3109999999999999,"15min":1.3400000000000001},"min":{"1min":1.052,"5min":0.754,"15min":0.56699999999999995},"max":{"1min":1.653,"5min":2.1499999999999999,"15min":2.2090000000000001},"last":1.587},{"interface":"front","average":{"1min":1.518,"5min":1.4830000000000001,"15min":1.492},"min":{"1min":1.2250000000000001,"5min":1.0169999999999999,"15min":0.65000000000000002},"max":{"1min":1.7410000000000001,"5min":2.2839999999999998,"15min":2.5009999999999999},"last":1.4319999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.393,"15min":1.4059999999999999},"min":{"1min":1.0209999999999999,"5min":0.91200000000000003,"15min":0.48899999999999999},"max":{"1min":1.681,"5min":2.3170000000000002,"15min":2.4860000000000002},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.4179999999999999,"15min":1.4350000000000001},"min":{"1min":1.03,"5min":0.86599999999999999,"15min":0.624},"max":{"1min":1.7390000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.5}]},{"osd":7,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.4690000000000001,"15min":1.484},"min":{"1min":1.1479999999999999,"5min":0.90900000000000003,"15min":0.54000000000000004},"max":{"1min":2.1549999999999998,"5min":2.5720000000000001,"15min":2.7229999999999999},"last":1.534},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":1.345,"5min":1.099,"15min":0.60099999999999998},"max":{"1min":1.851,"5min":2.2269999999999999,"15min":2.7509999999999999},"last":1.6399999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542298,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1425408,"data_stored":615127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.278,"15min":1.1819999999999999},"min":{"1min":1.044,"5min":0.71199999999999997,"15min":0.58999999999999997},"max":{"1min":1.73,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.069},{"interface":"front","average":{"1min":1.319,"5min":1.335,"15min":1.276},"min":{"1min":0.94199999999999995,"5min":0.74399999999999999,"15min":0.63700000000000001},"max":{"1min":1.5269999999999999,"5min":1.7769999999999999,"15min":1.8009999999999999},"last":1.2689999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.171,"15min":1.1619999999999999},"min":{"1min":0.84699999999999998,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.4079999999999999,"5min":1.9350000000000001,"15min":4.327},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.276,"5min":1.2130000000000001,"15min":1.1599999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.66200000000000003},"max":{"1min":1.6970000000000001,"5min":1.7470000000000001,"15min":2.4369999999999998},"last":1.554}]},{"osd":2,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2,"15min":1.2110000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.624},"max":{"1min":1.5780000000000001,"5min":2.0960000000000001,"15min":2.6080000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.2689999999999999,"15min":1.2649999999999999},"min":{"1min":1.0960000000000001,"5min":0.80600000000000005,"15min":0.71799999999999997},"max":{"1min":1.6299999999999999,"5min":1.8919999999999999,"15min":4.1310000000000002},"last":1.1339999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2509999999999999,"15min":1.2330000000000001},"min":{"1min":0.89600000000000002,"5min":0.747,"15min":0.622},"max":{"1min":1.659,"5min":1.659,"15min":2.5579999999999998},"last":1.341},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.365,"15min":1.325},"min":{"1min":1.165,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.655,"5min":2.024,"15min":4.2400000000000002},"last":1.462}]},{"osd":5,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.246,"15min":1.2509999999999999},"min":{"1min":0.85799999999999998,"5min":0.77100000000000002,"15min":0.74299999999999999},"max":{"1min":1.5940000000000001,"5min":1.9490000000000001,"15min":2.508},"last":1.044},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.321,"15min":1.355},"min":{"1min":1.004,"5min":0.67800000000000005,"15min":0.60499999999999998},"max":{"1min":1.7849999999999999,"5min":1.8460000000000001,"15min":5.859},"last":1.5760000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.393,"15min":1.3979999999999999},"min":{"1min":1.095,"5min":0.753,"15min":0.58899999999999997},"max":{"1min":1.609,"5min":2.194,"15min":5.3339999999999996},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.488,"5min":1.468,"15min":1.486},"min":{"1min":1.0349999999999999,"5min":0.81000000000000005,"15min":0.72599999999999998},"max":{"1min":1.762,"5min":1.875,"15min":5.3940000000000001},"last":1.639}]},{"osd":7,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.512,"15min":1.494},"min":{"1min":1.24,"5min":0.84699999999999998,"15min":0.754},"max":{"1min":1.681,"5min":2.0790000000000002,"15min":2.6859999999999999},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.3799999999999999,"15min":1.3839999999999999},"min":{"1min":1.032,"5min":0.79000000000000004,"15min":0.75600000000000001},"max":{"1min":1.794,"5min":1.8560000000000001,"15min":2.641},"last":1.766}]}]},{"osd":3,"up_from":14,"seq":60129542298,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1425408,"data_stored":615127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.873,"15min":0.92400000000000004},"min":{"1min":0.39300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.29,"5min":1.323,"15min":1.6819999999999999},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.0980000000000001,"15min":1.1040000000000001},"min":{"1min":0.625,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.5329999999999999,"5min":1.8540000000000001,"15min":2.0790000000000002},"last":1.5589999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.96299999999999997,"15min":0.98899999999999999},"min":{"1min":0.68899999999999995,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.976},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.1890000000000001,"15min":1.2390000000000001},"min":{"1min":0.55300000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6020000000000001,"5min":1.774,"15min":2.2309999999999999},"last":1.1619999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.105,"15min":1.125},"min":{"1min":0.753,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.532,"5min":1.7729999999999999,"15min":3.3109999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.27,"5min":1.167,"15min":1.2030000000000001},"min":{"1min":0.67600000000000005,"5min":0.51900000000000002,"15min":0.48399999999999999},"max":{"1min":1.5820000000000001,"5min":1.635,"15min":2.7109999999999999},"last":1.45}]},{"osd":4,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2450000000000001,"15min":1.268},"min":{"1min":0.95599999999999996,"5min":0.81499999999999995,"15min":0.73199999999999998},"max":{"1min":1.5669999999999999,"5min":1.752,"15min":2.3490000000000002},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.458,"5min":1.431,"15min":1.4710000000000001},"min":{"1min":1.119,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.649,"5min":2.081,"15min":4.8970000000000002},"last":1.619}]},{"osd":5,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4199999999999999,"15min":1.4770000000000001},"min":{"1min":1.0629999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":1.75,"5min":2.121,"15min":3.4910000000000001},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.339,"15min":1.407},"min":{"1min":1.0129999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.855,"5min":1.8700000000000001,"15min":3.125},"last":1.7010000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.371,"15min":1.4299999999999999},"min":{"1min":0.873,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":4.5499999999999998},"last":1.413},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4219999999999999,"15min":1.4970000000000001},"min":{"1min":0.995,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.8450000000000002},"last":1.893}]},{"osd":7,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.3979999999999999,"15min":1.458},"min":{"1min":1.0880000000000001,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.998,"5min":1.998,"15min":2.952},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.482,"15min":1.544},"min":{"1min":0.98699999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.964,"5min":1.964,"15min":4.798},"last":1.744}]}]},{"osd":2,"up_from":14,"seq":60129542298,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1425408,"data_stored":615127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.86199999999999999,"15min":0.89200000000000002},"min":{"1min":0.63900000000000001,"5min":0.246,"15min":0.246},"max":{"1min":1.202,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.0660000000000001,"15min":1.056},"min":{"1min":0.78100000000000003,"5min":0.40300000000000002,"15min":0.35099999999999998},"max":{"1min":1.641,"5min":1.6759999999999999,"15min":1.8220000000000001},"last":0.61699999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.85899999999999999,"15min":0.90100000000000002},"min":{"1min":0.66300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.798,"5min":1.798,"15min":1.9430000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0309999999999999},"min":{"1min":0.71499999999999997,"5min":0.39600000000000002,"15min":0.35799999999999998},"max":{"1min":1.458,"5min":1.593,"15min":1.7230000000000001},"last":0.68200000000000005}]},{"osd":3,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.099,"15min":1.113},"min":{"1min":0.82399999999999995,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.589,"5min":1.758,"15min":1.98},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.117,"5min":1.0669999999999999,"15min":1.0740000000000001},"min":{"1min":0.90000000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4239999999999999,"5min":1.6879999999999999,"15min":2.1560000000000001},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.343,"15min":1.3540000000000001},"min":{"1min":1.224,"5min":0.73299999999999998,"15min":0.69599999999999995},"max":{"1min":1.597,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.048},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4219999999999999,"15min":1.407},"min":{"1min":1.1839999999999999,"5min":0.93700000000000006,"15min":0.83499999999999996},"max":{"1min":1.7470000000000001,"5min":1.9470000000000001,"15min":2.085},"last":1.0029999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.319,"15min":1.3620000000000001},"min":{"1min":1.0429999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7689999999999999,"5min":1.8720000000000001,"15min":2.5830000000000002},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.3080000000000001,"15min":1.3580000000000001},"min":{"1min":1.2370000000000001,"5min":0.73999999999999999,"15min":0.66000000000000003},"max":{"1min":1.5429999999999999,"5min":1.96,"15min":2.6469999999999998},"last":0.95099999999999996}]},{"osd":6,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.3979999999999999,"15min":1.385},"min":{"1min":1.256,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.859,"5min":2.0600000000000001,"15min":2.758},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.389,"15min":1.444},"min":{"1min":1.135,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.738,"5min":1.883,"15min":2.8580000000000001},"last":1.135}]},{"osd":7,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.454,"15min":1.4359999999999999},"min":{"1min":1.3300000000000001,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8049999999999999,"5min":2.0150000000000001,"15min":2.246},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.45,"15min":1.512},"min":{"1min":1.204,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.714,"5min":2.012,"15min":2.7120000000000002},"last":1.226}]}]},{"osd":1,"up_from":14,"seq":60129542299,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1421312,"data_stored":614458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.88800000000000001,"15min":0.89100000000000001},"min":{"1min":0.70599999999999996,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.6379999999999999,"5min":1.6930000000000001,"15min":3.2959999999999998},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.0600000000000001,"15min":1.014},"min":{"1min":0.56699999999999995,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.5840000000000001,"5min":1.8480000000000001,"15min":1.99},"last":1.0629999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.93700000000000006,"15min":0.95099999999999996},"min":{"1min":0.64400000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.212,"5min":1.5549999999999999,"15min":1.738},"last":1.25},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.109,"15min":1.1060000000000001},"min":{"1min":0.70399999999999996,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.53,"5min":1.762,"15min":2.2160000000000002},"last":1.296}]},{"osd":3,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.23,"15min":1.2250000000000001},"min":{"1min":0.69799999999999995,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.837,"5min":1.8819999999999999,"15min":2.4430000000000001},"last":1.486},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1319999999999999,"15min":1.1299999999999999},"min":{"1min":0.496,"5min":0.45900000000000002,"15min":0.434},"max":{"1min":1.6120000000000001,"5min":2.0539999999999998,"15min":2.4169999999999998},"last":0.93899999999999995}]},{"osd":4,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.339,"15min":1.3100000000000001},"min":{"1min":1.139,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.556,"5min":1.9019999999999999,"15min":2.153},"last":1.355},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.296,"15min":1.3029999999999999},"min":{"1min":0.95999999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.6140000000000001,"5min":1.867,"15min":2.758},"last":1.4219999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3200000000000001,"15min":1.3620000000000001},"min":{"1min":0.86799999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6579999999999999,"5min":1.992,"15min":2.6589999999999998},"last":1.589},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.423,"15min":1.431},"min":{"1min":1.214,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":3.0329999999999999},"last":1.5629999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.413,"15min":1.4199999999999999},"min":{"1min":1.022,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.7310000000000001,"5min":2.1600000000000001,"15min":2.2730000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4319999999999999,"15min":1.474},"min":{"1min":1.196,"5min":0.747,"15min":0.747},"max":{"1min":1.732,"5min":1.8959999999999999,"15min":2.9630000000000001},"last":1.6830000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.5269999999999999,"15min":1.5329999999999999},"min":{"1min":1.2310000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.0030000000000001,"5min":2.2450000000000001,"15min":3.2749999999999999},"last":1.629},{"interface":"front","average":{"1min":1.5,"5min":1.5149999999999999,"15min":1.534},"min":{"1min":1.163,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.714,"5min":2.242,"15min":2.7829999999999999},"last":1.6539999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33084,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338756,"statfs":{"total":96636764160,"available":96602886144,"internally_reserved":0,"allocated":2015232,"data_stored":1204826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.83299999999999996,"15min":0.85599999999999998},"min":{"1min":0.72599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.2789999999999999,"5min":1.8660000000000001,"15min":2.8479999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.052,"5min":0.99099999999999999,"15min":0.97099999999999997},"min":{"1min":0.77300000000000002,"5min":0.46600000000000003,"15min":0.30399999999999999},"max":{"1min":1.466,"5min":1.544,"15min":2.4319999999999999},"last":0.96799999999999997}]},{"osd":2,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.01,"15min":1.036},"min":{"1min":0.86799999999999999,"5min":0.41799999999999998,"15min":0.33700000000000002},"max":{"1min":1.446,"5min":1.5509999999999999,"15min":2.4980000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.198,"5min":1.1240000000000001,"15min":1.0669999999999999},"min":{"1min":0.65700000000000003,"5min":0.50600000000000001,"15min":0.432},"max":{"1min":1.579,"5min":1.7589999999999999,"15min":2.5760000000000001},"last":0.84099999999999997}]},{"osd":3,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.145,"15min":1.125},"min":{"1min":0.85299999999999998,"5min":0.51500000000000001,"15min":0.35799999999999998},"max":{"1min":1.6890000000000001,"5min":1.7589999999999999,"15min":2.4580000000000002},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.3220000000000001,"15min":1.3240000000000001},"min":{"1min":1.1240000000000001,"5min":0.61299999999999999,"15min":0.48599999999999999},"max":{"1min":1.7769999999999999,"5min":1.968,"15min":2.9390000000000001},"last":1.345}]},{"osd":4,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.321,"15min":1.3420000000000001},"min":{"1min":1.05,"5min":0.73699999999999999,"15min":0.56100000000000005},"max":{"1min":1.7470000000000001,"5min":1.855,"15min":2.9750000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.3160000000000001,"15min":1.2969999999999999},"min":{"1min":1.024,"5min":0.81399999999999995,"15min":0.39900000000000002},"max":{"1min":1.9079999999999999,"5min":1.968,"15min":3.125},"last":1.4470000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.359,"15min":1.381},"min":{"1min":1.0489999999999999,"5min":0.90200000000000002,"15min":0.51200000000000001},"max":{"1min":1.792,"5min":1.925,"15min":3.0830000000000002},"last":1.595},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.3340000000000001,"15min":1.427},"min":{"1min":1.0980000000000001,"5min":0.72799999999999998,"15min":0.52700000000000002},"max":{"1min":1.9370000000000001,"5min":2.0449999999999999,"15min":3.1899999999999999},"last":1.639}]},{"osd":6,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.3899999999999999,"15min":1.462},"min":{"1min":1.2010000000000001,"5min":0.876,"15min":0.47999999999999998},"max":{"1min":2.0089999999999999,"5min":2.1030000000000002,"15min":3.391},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4199999999999999,"15min":1.4139999999999999},"min":{"1min":1.018,"5min":0.71399999999999997,"15min":0.58399999999999996},"max":{"1min":1.823,"5min":2.2170000000000001,"15min":3.153},"last":1.151}]},{"osd":7,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.5169999999999999,"15min":1.5249999999999999},"min":{"1min":1.1619999999999999,"5min":0.85799999999999998,"15min":0.60599999999999998},"max":{"1min":1.8879999999999999,"5min":2.1400000000000001,"15min":3.5350000000000001},"last":1.55},{"interface":"front","average":{"1min":1.635,"5min":1.5,"15min":1.4990000000000001},"min":{"1min":1.2689999999999999,"5min":0.78300000000000003,"15min":0.45300000000000001},"max":{"1min":2.2440000000000002,"5min":2.2610000000000001,"15min":3.3650000000000002},"last":1.6870000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:31:29.069 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:31:29.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.339+0000 7fa7df200640 1 Processor -- start 2024-03-23T11:31:29.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.339+0000 7fa7df200640 1 -- start start 2024-03-23T11:31:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.341+0000 7fa7df200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e00ce5f0 0x7fa7e00b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.341+0000 7fa7df200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e00ced90 con 0x7fa7e006cc60 2024-03-23T11:31:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.341+0000 7fa7df200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e006e3e0 con 0x7fa7e0159490 2024-03-23T11:31:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.341+0000 7fa7df200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e006c1c0 con 0x7fa7e00ce5f0 2024-03-23T11:31:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.341+0000 7fa7dd400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa7e006cc60 0x7fa7e00cdee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42908/0 (socket says 172.21.15.62:42908) 2024-03-23T11:31:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.341+0000 7fa7dd400640 1 -- 172.21.15.62:0/429083058 learned_addr learned my addr 172.21.15.62:0/429083058 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.341+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986016411 0 0) 0x7fa7e00ced90 con 0x7fa7e006cc60 2024-03-23T11:31:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.341+0000 7fa7de800640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e00ce5f0 0x7fa7e00b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.341+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c4003520 con 0x7fa7e006cc60 2024-03-23T11:31:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1590124349 0 0) 0x7fa7c4003520 con 0x7fa7e006cc60 2024-03-23T11:31:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e00ce5f0 msgr2=0x7fa7e00b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e00ce5f0 0x7fa7e00b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 >> v1:172.21.15.120:6789/0 conn(0x7fa7e0159490 legacy=0x7fa7e015da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e006d0c0 con 0x7fa7e006cc60 2024-03-23T11:31:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fa7d0001670 con 0x7fa7e006cc60 2024-03-23T11:31:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 >> v1:172.21.15.62:6789/0 conn(0x7fa7e006cc60 legacy=0x7fa7e00cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c40039a0 0x7fa7c4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e00ce5f0 0x7fa7c40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c4004d00 0x7fa7c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e006c1c0 con 0x7fa7c4004d00 2024-03-23T11:31:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c400bb60 con 0x7fa7c40039a0 2024-03-23T11:31:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c400df30 con 0x7fa7e00ce5f0 2024-03-23T11:31:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7de800640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c4004d00 0x7fa7c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dd400640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e00ce5f0 0x7fa7c40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.342+0000 7fa7dde00640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c40039a0 0x7fa7c4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.343+0000 7fa7de800640 1 -- 172.21.15.62:0/429083058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e00ce5f0 msgr2=0x7fa7c40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.343+0000 7fa7de800640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e00ce5f0 0x7fa7c40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.343+0000 7fa7de800640 1 -- 172.21.15.62:0/429083058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c40039a0 msgr2=0x7fa7c4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.343+0000 7fa7de800640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c40039a0 0x7fa7c4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.343+0000 7fa7de800640 1 -- 172.21.15.62:0/429083058 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c4003520 con 0x7fa7c4004d00 2024-03-23T11:31:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.344+0000 7fa7de800640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c4004d00 0x7fa7c4005180 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8007330 tx=0x7fa7c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0913896ba1ce43 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.344+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d802cca0 con 0x7fa7c4004d00 2024-03-23T11:31:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.344+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d802ce00 con 0x7fa7c4004d00 2024-03-23T11:31:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.344+0000 7fa7dca00640 1 -- 172.21.15.62:0/429083058 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8034680 con 0x7fa7c4004d00 2024-03-23T11:31:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.345+0000 7fa7df200640 1 -- 172.21.15.62:0/429083058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c4004d00 msgr2=0x7fa7c4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.345+0000 7fa7df200640 1 --2- 172.21.15.62:0/429083058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7c4004d00 0x7fa7c4005180 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8007330 tx=0x7fa7c40057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.345+0000 7fa7de800640 1 -- 172.21.15.62:0/429083058 reap_dead start 2024-03-23T11:31:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.345+0000 7fa7df200640 1 -- 172.21.15.62:0/429083058 shutdown_connections 2024-03-23T11:31:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.345+0000 7fa7df200640 1 -- 172.21.15.62:0/429083058 >> 172.21.15.62:0/429083058 conn(0x7fa7e0066050 msgr2=0x7fa7e0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.345+0000 7fa7df200640 1 -- 172.21.15.62:0/429083058 shutdown_connections 2024-03-23T11:31:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.345+0000 7fa7df200640 1 -- 172.21.15.62:0/429083058 wait complete. 2024-03-23T11:31:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.347+0000 7fa7df200640 1 Processor -- start 2024-03-23T11:31:29.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.347+0000 7fa7df200640 1 -- start start 2024-03-23T11:31:29.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.348+0000 7fa7df200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e0144c40 0x7fa7e0151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.348+0000 7fa7df200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7e0145020 0x7fa7e014ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7df200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7e014efb0 0x7fa7e014f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7df200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e014f430 con 0x7fa7e014efb0 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7df200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e0150580 con 0x7fa7e0145020 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7df200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e0150e70 con 0x7fa7e0144c40 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7de800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7e014efb0 0x7fa7e014f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7de800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7e014efb0 0x7fa7e014f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37716/0 (socket says 172.21.15.62:37716) 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7de800640 1 -- 172.21.15.62:0/3720444360 learned_addr learned my addr 172.21.15.62:0/3720444360 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7dde00640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e0144c40 0x7fa7e0151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7dd400640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7e0145020 0x7fa7e014ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7de800640 1 -- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e0144c40 msgr2=0x7fa7e0151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7de800640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e0144c40 0x7fa7e0151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7de800640 1 -- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7e0145020 msgr2=0x7fa7e014ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7de800640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7e0145020 0x7fa7e014ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.349+0000 7fa7de800640 1 -- 172.21.15.62:0/3720444360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e0151150 con 0x7fa7e014efb0 2024-03-23T11:31:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.350+0000 7fa7de800640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7e014efb0 0x7fa7e014f7e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8007450 tx=0x7fa7d8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.350+0000 7fa7d6000640 1 -- 172.21.15.62:0/3720444360 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d803e600 con 0x7fa7e014efb0 2024-03-23T11:31:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.350+0000 7fa7d6000640 1 -- 172.21.15.62:0/3720444360 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d803e760 con 0x7fa7e014efb0 2024-03-23T11:31:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.350+0000 7fa7df200640 1 -- 172.21.15.62:0/3720444360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e0151430 con 0x7fa7e014efb0 2024-03-23T11:31:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.350+0000 7fa7d6000640 1 -- 172.21.15.62:0/3720444360 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d803ead0 con 0x7fa7e014efb0 2024-03-23T11:31:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.350+0000 7fa7df200640 1 -- 172.21.15.62:0/3720444360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e0152170 con 0x7fa7e014efb0 2024-03-23T11:31:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.352+0000 7fa7df200640 1 -- 172.21.15.62:0/3720444360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e0066bb0 con 0x7fa7e014efb0 2024-03-23T11:31:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.352+0000 7fa7d6000640 1 -- 172.21.15.62:0/3720444360 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa7d803ec30 con 0x7fa7e014efb0 2024-03-23T11:31:29.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.353+0000 7fa7d6000640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7bc06a7d0 0x7fa7bc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.353+0000 7fa7d6000640 1 -- 172.21.15.62:0/3720444360 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7d80aaca0 con 0x7fa7e014efb0 2024-03-23T11:31:29.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.354+0000 7fa7dde00640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7bc06a7d0 0x7fa7bc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.355+0000 7fa7dde00640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7bc06a7d0 0x7fa7bc06cc90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8007fd0 tx=0x7fa7c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:29.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.358+0000 7fa7d6000640 1 -- 172.21.15.62:0/3720444360 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7d8076150 con 0x7fa7e014efb0 2024-03-23T11:31:29.511 INFO:tasks.ceph.mds.b.smithi120.stderr:2024-03-23T11:31:29.521+0000 7f577a800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:31:29.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.549+0000 7fa7df200640 1 -- 172.21.15.62:0/3720444360 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7e006c740 con 0x7fa7bc06a7d0 2024-03-23T11:31:29.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.571+0000 7fa7d6000640 1 -- 172.21.15.62:0/3720444360 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286872 (secure 0 0 0) 0x7fa7c80013b0 con 0x7fa7bc06a7d0 2024-03-23T11:31:29.565 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:29.567 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:31:29.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.577+0000 7fa7df200640 1 -- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7bc06a7d0 msgr2=0x7fa7bc06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.577+0000 7fa7df200640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7bc06a7d0 0x7fa7bc06cc90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8007fd0 tx=0x7fa7c8009040 comp rx=0 tx=0).stop 2024-03-23T11:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.577+0000 7fa7df200640 1 -- 172.21.15.62:0/3720444360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7e014efb0 msgr2=0x7fa7e014f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.577+0000 7fa7df200640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7e014efb0 0x7fa7e014f7e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8007450 tx=0x7fa7d8004ab0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.577+0000 7fa7df200640 1 -- 172.21.15.62:0/3720444360 shutdown_connections 2024-03-23T11:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.577+0000 7fa7df200640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa7bc06a7d0 0x7fa7bc06cc90 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.577+0000 7fa7df200640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7e014efb0 0x7fa7e014f7e0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.577+0000 7fa7df200640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7e0145020 0x7fa7e014ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.577+0000 7fa7df200640 1 --2- 172.21.15.62:0/3720444360 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa7e0144c40 0x7fa7e0151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.577+0000 7fa7df200640 1 -- 172.21.15.62:0/3720444360 >> 172.21.15.62:0/3720444360 conn(0x7fa7e0066050 msgr2=0x7fa7e00ccfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.578+0000 7fa7df200640 1 -- 172.21.15.62:0/3720444360 shutdown_connections 2024-03-23T11:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.578+0000 7fa7df200640 1 -- 172.21.15.62:0/3720444360 wait complete. 2024-03-23T11:31:29.587 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-03-23T11:31:29.512679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261572,"kb_used_data":12636,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754713148,"statfs":{"total":773094113280,"available":772826263552,"internally_reserved":0,"allocated":12939264,"data_stored":6259269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002058"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.364017+0000","last_change":"2024-03-23T11:31:24.087898+0000","last_active":"2024-03-23T11:31:26.364017+0000","last_peered":"2024-03-23T11:31:26.364017+0000","last_clean":"2024-03-23T11:31:26.364017+0000","last_became_active":"2024-03-23T11:31:24.087620+0000","last_became_peered":"2024-03-23T11:31:24.087620+0000","last_unstale":"2024-03-23T11:31:26.364017+0000","last_undegraded":"2024-03-23T11:31:26.364017+0000","last_fullsized":"2024-03-23T11:31:26.364017+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:38.653958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363487+0000","last_change":"2024-03-23T11:31:24.071891+0000","last_active":"2024-03-23T11:31:26.363487+0000","last_peered":"2024-03-23T11:31:26.363487+0000","last_clean":"2024-03-23T11:31:26.363487+0000","last_became_active":"2024-03-23T11:31:24.071398+0000","last_became_peered":"2024-03-23T11:31:24.071398+0000","last_unstale":"2024-03-23T11:31:26.363487+0000","last_undegraded":"2024-03-23T11:31:26.363487+0000","last_fullsized":"2024-03-23T11:31:26.363487+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:15:26.611633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.072839+0000","last_change":"2024-03-23T11:31:24.070173+0000","last_active":"2024-03-23T11:31:25.072839+0000","last_peered":"2024-03-23T11:31:25.072839+0000","last_clean":"2024-03-23T11:31:25.072839+0000","last_became_active":"2024-03-23T11:31:24.069877+0000","last_became_peered":"2024-03-23T11:31:24.069877+0000","last_unstale":"2024-03-23T11:31:25.072839+0000","last_undegraded":"2024-03-23T11:31:25.072839+0000","last_fullsized":"2024-03-23T11:31:25.072839+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:48:53.858111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.364093+0000","last_change":"2024-03-23T11:31:24.090534+0000","last_active":"2024-03-23T11:31:26.364093+0000","last_peered":"2024-03-23T11:31:26.364093+0000","last_clean":"2024-03-23T11:31:26.364093+0000","last_became_active":"2024-03-23T11:31:24.090081+0000","last_became_peered":"2024-03-23T11:31:24.090081+0000","last_unstale":"2024-03-23T11:31:26.364093+0000","last_undegraded":"2024-03-23T11:31:26.364093+0000","last_fullsized":"2024-03-23T11:31:26.364093+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:14:25.736995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.360579+0000","last_change":"2024-03-23T11:31:24.073178+0000","last_active":"2024-03-23T11:31:25.360579+0000","last_peered":"2024-03-23T11:31:25.360579+0000","last_clean":"2024-03-23T11:31:25.360579+0000","last_became_active":"2024-03-23T11:31:24.072741+0000","last_became_peered":"2024-03-23T11:31:24.072741+0000","last_unstale":"2024-03-23T11:31:25.360579+0000","last_undegraded":"2024-03-23T11:31:25.360579+0000","last_fullsized":"2024-03-23T11:31:25.360579+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:03:46.114026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.066678+0000","last_change":"2024-03-23T11:31:24.071990+0000","last_active":"2024-03-23T11:31:25.066678+0000","last_peered":"2024-03-23T11:31:25.066678+0000","last_clean":"2024-03-23T11:31:25.066678+0000","last_became_active":"2024-03-23T11:31:24.071720+0000","last_became_peered":"2024-03-23T11:31:24.071720+0000","last_unstale":"2024-03-23T11:31:25.066678+0000","last_undegraded":"2024-03-23T11:31:25.066678+0000","last_fullsized":"2024-03-23T11:31:25.066678+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:48:27.561521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.070671+0000","last_change":"2024-03-23T11:31:24.079802+0000","last_active":"2024-03-23T11:31:25.070671+0000","last_peered":"2024-03-23T11:31:25.070671+0000","last_clean":"2024-03-23T11:31:25.070671+0000","last_became_active":"2024-03-23T11:31:24.079447+0000","last_became_peered":"2024-03-23T11:31:24.079447+0000","last_unstale":"2024-03-23T11:31:25.070671+0000","last_undegraded":"2024-03-23T11:31:25.070671+0000","last_fullsized":"2024-03-23T11:31:25.070671+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:10:33.600614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.072544+0000","last_change":"2024-03-23T11:31:24.075662+0000","last_active":"2024-03-23T11:31:25.072544+0000","last_peered":"2024-03-23T11:31:25.072544+0000","last_clean":"2024-03-23T11:31:25.072544+0000","last_became_active":"2024-03-23T11:31:24.075340+0000","last_became_peered":"2024-03-23T11:31:24.075340+0000","last_unstale":"2024-03-23T11:31:25.072544+0000","last_undegraded":"2024-03-23T11:31:25.072544+0000","last_fullsized":"2024-03-23T11:31:25.072544+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:59:01.687590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.364154+0000","last_change":"2024-03-23T11:31:24.087579+0000","last_active":"2024-03-23T11:31:26.364154+0000","last_peered":"2024-03-23T11:31:26.364154+0000","last_clean":"2024-03-23T11:31:26.364154+0000","last_became_active":"2024-03-23T11:31:24.087262+0000","last_became_peered":"2024-03-23T11:31:24.087262+0000","last_unstale":"2024-03-23T11:31:26.364154+0000","last_undegraded":"2024-03-23T11:31:26.364154+0000","last_fullsized":"2024-03-23T11:31:26.364154+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:14:12.812328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363678+0000","last_change":"2024-03-23T11:31:24.095511+0000","last_active":"2024-03-23T11:31:26.363678+0000","last_peered":"2024-03-23T11:31:26.363678+0000","last_clean":"2024-03-23T11:31:26.363678+0000","last_became_active":"2024-03-23T11:31:24.095309+0000","last_became_peered":"2024-03-23T11:31:24.095309+0000","last_unstale":"2024-03-23T11:31:26.363678+0000","last_undegraded":"2024-03-23T11:31:26.363678+0000","last_fullsized":"2024-03-23T11:31:26.363678+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:56:54.286721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363044+0000","last_change":"2024-03-23T11:31:24.081019+0000","last_active":"2024-03-23T11:31:26.363044+0000","last_peered":"2024-03-23T11:31:26.363044+0000","last_clean":"2024-03-23T11:31:26.363044+0000","last_became_active":"2024-03-23T11:31:24.080735+0000","last_became_peered":"2024-03-23T11:31:24.080735+0000","last_unstale":"2024-03-23T11:31:26.363044+0000","last_undegraded":"2024-03-23T11:31:26.363044+0000","last_fullsized":"2024-03-23T11:31:26.363044+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:21:35.835742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363974+0000","last_change":"2024-03-23T11:31:24.095578+0000","last_active":"2024-03-23T11:31:26.363974+0000","last_peered":"2024-03-23T11:31:26.363974+0000","last_clean":"2024-03-23T11:31:26.363974+0000","last_became_active":"2024-03-23T11:31:24.095148+0000","last_became_peered":"2024-03-23T11:31:24.095148+0000","last_unstale":"2024-03-23T11:31:26.363974+0000","last_undegraded":"2024-03-23T11:31:26.363974+0000","last_fullsized":"2024-03-23T11:31:26.363974+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:09:04.413734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.072666+0000","last_change":"2024-03-23T11:31:24.082007+0000","last_active":"2024-03-23T11:31:25.072666+0000","last_peered":"2024-03-23T11:31:25.072666+0000","last_clean":"2024-03-23T11:31:25.072666+0000","last_became_active":"2024-03-23T11:31:24.081678+0000","last_became_peered":"2024-03-23T11:31:24.081678+0000","last_unstale":"2024-03-23T11:31:25.072666+0000","last_undegraded":"2024-03-23T11:31:25.072666+0000","last_fullsized":"2024-03-23T11:31:25.072666+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:20:40.119818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.072216+0000","last_change":"2024-03-23T11:31:24.073216+0000","last_active":"2024-03-23T11:31:25.072216+0000","last_peered":"2024-03-23T11:31:25.072216+0000","last_clean":"2024-03-23T11:31:25.072216+0000","last_became_active":"2024-03-23T11:31:24.072918+0000","last_became_peered":"2024-03-23T11:31:24.072918+0000","last_unstale":"2024-03-23T11:31:25.072216+0000","last_undegraded":"2024-03-23T11:31:25.072216+0000","last_fullsized":"2024-03-23T11:31:25.072216+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:38:46.314357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.066956+0000","last_change":"2024-03-23T11:31:24.076040+0000","last_active":"2024-03-23T11:31:25.066956+0000","last_peered":"2024-03-23T11:31:25.066956+0000","last_clean":"2024-03-23T11:31:25.066956+0000","last_became_active":"2024-03-23T11:31:24.075557+0000","last_became_peered":"2024-03-23T11:31:24.075557+0000","last_unstale":"2024-03-23T11:31:25.066956+0000","last_undegraded":"2024-03-23T11:31:25.066956+0000","last_fullsized":"2024-03-23T11:31:25.066956+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:46:09.528465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363074+0000","last_change":"2024-03-23T11:31:24.080597+0000","last_active":"2024-03-23T11:31:26.363074+0000","last_peered":"2024-03-23T11:31:26.363074+0000","last_clean":"2024-03-23T11:31:26.363074+0000","last_became_active":"2024-03-23T11:31:24.080329+0000","last_became_peered":"2024-03-23T11:31:24.080329+0000","last_unstale":"2024-03-23T11:31:26.363074+0000","last_undegraded":"2024-03-23T11:31:26.363074+0000","last_fullsized":"2024-03-23T11:31:26.363074+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:58.793014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363556+0000","last_change":"2024-03-23T11:31:24.080666+0000","last_active":"2024-03-23T11:31:26.363556+0000","last_peered":"2024-03-23T11:31:26.363556+0000","last_clean":"2024-03-23T11:31:26.363556+0000","last_became_active":"2024-03-23T11:31:24.080404+0000","last_became_peered":"2024-03-23T11:31:24.080404+0000","last_unstale":"2024-03-23T11:31:26.363556+0000","last_undegraded":"2024-03-23T11:31:26.363556+0000","last_fullsized":"2024-03-23T11:31:26.363556+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:33.287144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.562202+0000","last_change":"2024-03-23T11:31:24.076780+0000","last_active":"2024-03-23T11:31:25.562202+0000","last_peered":"2024-03-23T11:31:25.562202+0000","last_clean":"2024-03-23T11:31:25.562202+0000","last_became_active":"2024-03-23T11:31:24.076379+0000","last_became_peered":"2024-03-23T11:31:24.076379+0000","last_unstale":"2024-03-23T11:31:25.562202+0000","last_undegraded":"2024-03-23T11:31:25.562202+0000","last_fullsized":"2024-03-23T11:31:25.562202+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:33.287144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.073045+0000","last_change":"2024-03-23T11:31:24.082271+0000","last_active":"2024-03-23T11:31:25.073045+0000","last_peered":"2024-03-23T11:31:25.073045+0000","last_clean":"2024-03-23T11:31:25.073045+0000","last_became_active":"2024-03-23T11:31:24.081832+0000","last_became_peered":"2024-03-23T11:31:24.081832+0000","last_unstale":"2024-03-23T11:31:25.073045+0000","last_undegraded":"2024-03-23T11:31:25.073045+0000","last_fullsized":"2024-03-23T11:31:25.073045+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:32:56.321991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.362842+0000","last_change":"2024-03-23T11:31:24.073190+0000","last_active":"2024-03-23T11:31:26.362842+0000","last_peered":"2024-03-23T11:31:26.362842+0000","last_clean":"2024-03-23T11:31:26.362842+0000","last_became_active":"2024-03-23T11:31:24.072206+0000","last_became_peered":"2024-03-23T11:31:24.072206+0000","last_unstale":"2024-03-23T11:31:26.362842+0000","last_undegraded":"2024-03-23T11:31:26.362842+0000","last_fullsized":"2024-03-23T11:31:26.362842+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:58:36.736303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363053+0000","last_change":"2024-03-23T11:31:24.075119+0000","last_active":"2024-03-23T11:31:26.363053+0000","last_peered":"2024-03-23T11:31:26.363053+0000","last_clean":"2024-03-23T11:31:26.363053+0000","last_became_active":"2024-03-23T11:31:24.074348+0000","last_became_peered":"2024-03-23T11:31:24.074348+0000","last_unstale":"2024-03-23T11:31:26.363053+0000","last_undegraded":"2024-03-23T11:31:26.363053+0000","last_fullsized":"2024-03-23T11:31:26.363053+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:52:56.302746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.474889+0000","last_change":"2024-03-23T11:31:24.081538+0000","last_active":"2024-03-23T11:31:25.474889+0000","last_peered":"2024-03-23T11:31:25.474889+0000","last_clean":"2024-03-23T11:31:25.474889+0000","last_became_active":"2024-03-23T11:31:24.081239+0000","last_became_peered":"2024-03-23T11:31:24.081239+0000","last_unstale":"2024-03-23T11:31:25.474889+0000","last_undegraded":"2024-03-23T11:31:25.474889+0000","last_fullsized":"2024-03-23T11:31:25.474889+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:29:35.808578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.362614+0000","last_change":"2024-03-23T11:31:24.068827+0000","last_active":"2024-03-23T11:31:26.362614+0000","last_peered":"2024-03-23T11:31:26.362614+0000","last_clean":"2024-03-23T11:31:26.362614+0000","last_became_active":"2024-03-23T11:31:24.068311+0000","last_became_peered":"2024-03-23T11:31:24.068311+0000","last_unstale":"2024-03-23T11:31:26.362614+0000","last_undegraded":"2024-03-23T11:31:26.362614+0000","last_fullsized":"2024-03-23T11:31:26.362614+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:22:29.505465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.474013+0000","last_change":"2024-03-23T11:31:24.066437+0000","last_active":"2024-03-23T11:31:25.474013+0000","last_peered":"2024-03-23T11:31:25.474013+0000","last_clean":"2024-03-23T11:31:25.474013+0000","last_became_active":"2024-03-23T11:31:24.065132+0000","last_became_peered":"2024-03-23T11:31:24.065132+0000","last_unstale":"2024-03-23T11:31:25.474013+0000","last_undegraded":"2024-03-23T11:31:25.474013+0000","last_fullsized":"2024-03-23T11:31:25.474013+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:58:36.736303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363654+0000","last_change":"2024-03-23T11:31:24.081141+0000","last_active":"2024-03-23T11:31:26.363654+0000","last_peered":"2024-03-23T11:31:26.363654+0000","last_clean":"2024-03-23T11:31:26.363654+0000","last_became_active":"2024-03-23T11:31:24.078948+0000","last_became_peered":"2024-03-23T11:31:24.078948+0000","last_unstale":"2024-03-23T11:31:26.363654+0000","last_undegraded":"2024-03-23T11:31:26.363654+0000","last_fullsized":"2024-03-23T11:31:26.363654+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:31:36.918271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363952+0000","last_change":"2024-03-23T11:31:24.085746+0000","last_active":"2024-03-23T11:31:26.363952+0000","last_peered":"2024-03-23T11:31:26.363952+0000","last_clean":"2024-03-23T11:31:26.363952+0000","last_became_active":"2024-03-23T11:31:24.085464+0000","last_became_peered":"2024-03-23T11:31:24.085464+0000","last_unstale":"2024-03-23T11:31:26.363952+0000","last_undegraded":"2024-03-23T11:31:26.363952+0000","last_fullsized":"2024-03-23T11:31:26.363952+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:12:58.656613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.067435+0000","last_change":"2024-03-23T11:31:24.087209+0000","last_active":"2024-03-23T11:31:25.067435+0000","last_peered":"2024-03-23T11:31:25.067435+0000","last_clean":"2024-03-23T11:31:25.067435+0000","last_became_active":"2024-03-23T11:31:24.086892+0000","last_became_peered":"2024-03-23T11:31:24.086892+0000","last_unstale":"2024-03-23T11:31:25.067435+0000","last_undegraded":"2024-03-23T11:31:25.067435+0000","last_fullsized":"2024-03-23T11:31:25.067435+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:27.519507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.474420+0000","last_change":"2024-03-23T11:31:24.094576+0000","last_active":"2024-03-23T11:31:25.474420+0000","last_peered":"2024-03-23T11:31:25.474420+0000","last_clean":"2024-03-23T11:31:25.474420+0000","last_became_active":"2024-03-23T11:31:24.094183+0000","last_became_peered":"2024-03-23T11:31:24.094183+0000","last_unstale":"2024-03-23T11:31:25.474420+0000","last_undegraded":"2024-03-23T11:31:25.474420+0000","last_fullsized":"2024-03-23T11:31:25.474420+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:31:44.177219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.362568+0000","last_change":"2024-03-23T11:31:24.065302+0000","last_active":"2024-03-23T11:31:26.362568+0000","last_peered":"2024-03-23T11:31:26.362568+0000","last_clean":"2024-03-23T11:31:26.362568+0000","last_became_active":"2024-03-23T11:31:24.064944+0000","last_became_peered":"2024-03-23T11:31:24.064944+0000","last_unstale":"2024-03-23T11:31:26.362568+0000","last_undegraded":"2024-03-23T11:31:26.362568+0000","last_fullsized":"2024-03-23T11:31:26.362568+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:48:54.983383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.070258+0000","last_change":"2024-03-23T11:31:24.085174+0000","last_active":"2024-03-23T11:31:25.070258+0000","last_peered":"2024-03-23T11:31:25.070258+0000","last_clean":"2024-03-23T11:31:25.070258+0000","last_became_active":"2024-03-23T11:31:24.084736+0000","last_became_peered":"2024-03-23T11:31:24.084736+0000","last_unstale":"2024-03-23T11:31:25.070258+0000","last_undegraded":"2024-03-23T11:31:25.070258+0000","last_fullsized":"2024-03-23T11:31:25.070258+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:24:09.478200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363983+0000","last_change":"2024-03-23T11:31:24.085490+0000","last_active":"2024-03-23T11:31:26.363983+0000","last_peered":"2024-03-23T11:31:26.363983+0000","last_clean":"2024-03-23T11:31:26.363983+0000","last_became_active":"2024-03-23T11:31:24.085220+0000","last_became_peered":"2024-03-23T11:31:24.085220+0000","last_unstale":"2024-03-23T11:31:26.363983+0000","last_undegraded":"2024-03-23T11:31:26.363983+0000","last_fullsized":"2024-03-23T11:31:26.363983+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:53:07.492278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.066651+0000","last_change":"2024-03-23T11:31:24.076047+0000","last_active":"2024-03-23T11:31:25.066651+0000","last_peered":"2024-03-23T11:31:25.066651+0000","last_clean":"2024-03-23T11:31:25.066651+0000","last_became_active":"2024-03-23T11:31:24.075624+0000","last_became_peered":"2024-03-23T11:31:24.075624+0000","last_unstale":"2024-03-23T11:31:25.066651+0000","last_undegraded":"2024-03-23T11:31:25.066651+0000","last_fullsized":"2024-03-23T11:31:25.066651+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:56:56.808997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.067334+0000","last_change":"2024-03-23T11:31:24.093266+0000","last_active":"2024-03-23T11:31:25.067334+0000","last_peered":"2024-03-23T11:31:25.067334+0000","last_clean":"2024-03-23T11:31:25.067334+0000","last_became_active":"2024-03-23T11:31:24.092985+0000","last_became_peered":"2024-03-23T11:31:24.092985+0000","last_unstale":"2024-03-23T11:31:25.067334+0000","last_undegraded":"2024-03-23T11:31:25.067334+0000","last_fullsized":"2024-03-23T11:31:25.067334+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:27:17.107454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"117'4","reported_seq":15,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.082948+0000","last_change":"2024-03-23T11:31:23.574298+0000","last_active":"2024-03-23T11:31:24.082948+0000","last_peered":"2024-03-23T11:31:24.082948+0000","last_clean":"2024-03-23T11:31:24.082948+0000","last_became_active":"2024-03-23T11:31:23.573930+0000","last_became_peered":"2024-03-23T11:31:23.573930+0000","last_unstale":"2024-03-23T11:31:24.082948+0000","last_undegraded":"2024-03-23T11:31:24.082948+0000","last_fullsized":"2024-03-23T11:31:24.082948+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:21:25.524815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.562684+0000","last_change":"2024-03-23T11:31:24.076666+0000","last_active":"2024-03-23T11:31:25.562684+0000","last_peered":"2024-03-23T11:31:25.562684+0000","last_clean":"2024-03-23T11:31:25.562684+0000","last_became_active":"2024-03-23T11:31:24.076306+0000","last_became_peered":"2024-03-23T11:31:24.076306+0000","last_unstale":"2024-03-23T11:31:25.562684+0000","last_undegraded":"2024-03-23T11:31:25.562684+0000","last_fullsized":"2024-03-23T11:31:25.562684+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:31:36.918271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"117'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.080272+0000","last_change":"2024-03-23T11:31:23.573596+0000","last_active":"2024-03-23T11:31:24.080272+0000","last_peered":"2024-03-23T11:31:24.080272+0000","last_clean":"2024-03-23T11:31:24.080272+0000","last_became_active":"2024-03-23T11:31:23.573155+0000","last_became_peered":"2024-03-23T11:31:23.573155+0000","last_unstale":"2024-03-23T11:31:24.080272+0000","last_undegraded":"2024-03-23T11:31:24.080272+0000","last_fullsized":"2024-03-23T11:31:24.080272+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:51:44.775155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.067223+0000","last_change":"2024-03-23T11:31:24.079516+0000","last_active":"2024-03-23T11:31:25.067223+0000","last_peered":"2024-03-23T11:31:25.067223+0000","last_clean":"2024-03-23T11:31:25.067223+0000","last_became_active":"2024-03-23T11:31:24.078774+0000","last_became_peered":"2024-03-23T11:31:24.078774+0000","last_unstale":"2024-03-23T11:31:25.067223+0000","last_undegraded":"2024-03-23T11:31:25.067223+0000","last_fullsized":"2024-03-23T11:31:25.067223+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:06.389983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"117'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.092901+0000","last_change":"2024-03-23T11:31:23.583124+0000","last_active":"2024-03-23T11:31:24.092901+0000","last_peered":"2024-03-23T11:31:24.092901+0000","last_clean":"2024-03-23T11:31:24.092901+0000","last_became_active":"2024-03-23T11:31:23.582769+0000","last_became_peered":"2024-03-23T11:31:23.582769+0000","last_unstale":"2024-03-23T11:31:24.092901+0000","last_undegraded":"2024-03-23T11:31:24.092901+0000","last_fullsized":"2024-03-23T11:31:24.092901+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:53:45.695329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.072929+0000","last_change":"2024-03-23T11:31:24.083141+0000","last_active":"2024-03-23T11:31:25.072929+0000","last_peered":"2024-03-23T11:31:25.072929+0000","last_clean":"2024-03-23T11:31:25.072929+0000","last_became_active":"2024-03-23T11:31:24.082550+0000","last_became_peered":"2024-03-23T11:31:24.082550+0000","last_unstale":"2024-03-23T11:31:25.072929+0000","last_undegraded":"2024-03-23T11:31:25.072929+0000","last_fullsized":"2024-03-23T11:31:25.072929+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:03:33.602220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"117'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.094490+0000","last_change":"2024-03-23T11:31:23.579437+0000","last_active":"2024-03-23T11:31:24.094490+0000","last_peered":"2024-03-23T11:31:24.094490+0000","last_clean":"2024-03-23T11:31:24.094490+0000","last_became_active":"2024-03-23T11:31:23.578988+0000","last_became_peered":"2024-03-23T11:31:23.578988+0000","last_unstale":"2024-03-23T11:31:24.094490+0000","last_undegraded":"2024-03-23T11:31:24.094490+0000","last_fullsized":"2024-03-23T11:31:24.094490+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:12:13.014635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363114+0000","last_change":"2024-03-23T11:31:24.084659+0000","last_active":"2024-03-23T11:31:26.363114+0000","last_peered":"2024-03-23T11:31:26.363114+0000","last_clean":"2024-03-23T11:31:26.363114+0000","last_became_active":"2024-03-23T11:31:24.084262+0000","last_became_peered":"2024-03-23T11:31:24.084262+0000","last_unstale":"2024-03-23T11:31:26.363114+0000","last_undegraded":"2024-03-23T11:31:26.363114+0000","last_fullsized":"2024-03-23T11:31:26.363114+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:17.674501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"117'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.081669+0000","last_change":"2024-03-23T11:31:23.582049+0000","last_active":"2024-03-23T11:31:24.081669+0000","last_peered":"2024-03-23T11:31:24.081669+0000","last_clean":"2024-03-23T11:31:24.081669+0000","last_became_active":"2024-03-23T11:31:23.581728+0000","last_became_peered":"2024-03-23T11:31:23.581728+0000","last_unstale":"2024-03-23T11:31:24.081669+0000","last_undegraded":"2024-03-23T11:31:24.081669+0000","last_fullsized":"2024-03-23T11:31:24.081669+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:46.152155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.362619+0000","last_change":"2024-03-23T11:31:24.070293+0000","last_active":"2024-03-23T11:31:26.362619+0000","last_peered":"2024-03-23T11:31:26.362619+0000","last_clean":"2024-03-23T11:31:26.362619+0000","last_became_active":"2024-03-23T11:31:24.069518+0000","last_became_peered":"2024-03-23T11:31:24.069518+0000","last_unstale":"2024-03-23T11:31:26.362619+0000","last_undegraded":"2024-03-23T11:31:26.362619+0000","last_fullsized":"2024-03-23T11:31:26.362619+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:02:29.547845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"117'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.080972+0000","last_change":"2024-03-23T11:31:23.578933+0000","last_active":"2024-03-23T11:31:24.080972+0000","last_peered":"2024-03-23T11:31:24.080972+0000","last_clean":"2024-03-23T11:31:24.080972+0000","last_became_active":"2024-03-23T11:31:23.578647+0000","last_became_peered":"2024-03-23T11:31:23.578647+0000","last_unstale":"2024-03-23T11:31:24.080972+0000","last_undegraded":"2024-03-23T11:31:24.080972+0000","last_fullsized":"2024-03-23T11:31:24.080972+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:24:08.473235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.363108+0000","last_change":"2024-03-23T11:31:24.082842+0000","last_active":"2024-03-23T11:31:26.363108+0000","last_peered":"2024-03-23T11:31:26.363108+0000","last_clean":"2024-03-23T11:31:26.363108+0000","last_became_active":"2024-03-23T11:31:24.082551+0000","last_became_peered":"2024-03-23T11:31:24.082551+0000","last_unstale":"2024-03-23T11:31:26.363108+0000","last_undegraded":"2024-03-23T11:31:26.363108+0000","last_fullsized":"2024-03-23T11:31:26.363108+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:53:25.691050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.056574+0000","last_change":"2024-03-23T11:31:23.571302+0000","last_active":"2024-03-23T11:31:24.056574+0000","last_peered":"2024-03-23T11:31:24.056574+0000","last_clean":"2024-03-23T11:31:24.056574+0000","last_became_active":"2024-03-23T11:31:23.571023+0000","last_became_peered":"2024-03-23T11:31:23.571023+0000","last_unstale":"2024-03-23T11:31:24.056574+0000","last_undegraded":"2024-03-23T11:31:24.056574+0000","last_fullsized":"2024-03-23T11:31:24.056574+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:14:30.980614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.070264+0000","last_change":"2024-03-23T11:31:24.080118+0000","last_active":"2024-03-23T11:31:25.070264+0000","last_peered":"2024-03-23T11:31:25.070264+0000","last_clean":"2024-03-23T11:31:25.070264+0000","last_became_active":"2024-03-23T11:31:24.079863+0000","last_became_peered":"2024-03-23T11:31:24.079863+0000","last_unstale":"2024-03-23T11:31:25.070264+0000","last_undegraded":"2024-03-23T11:31:25.070264+0000","last_fullsized":"2024-03-23T11:31:25.070264+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:21:35.835742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.067643+0000","last_change":"2024-03-23T11:31:23.574360+0000","last_active":"2024-03-23T11:31:24.067643+0000","last_peered":"2024-03-23T11:31:24.067643+0000","last_clean":"2024-03-23T11:31:24.067643+0000","last_became_active":"2024-03-23T11:31:23.571456+0000","last_became_peered":"2024-03-23T11:31:23.571456+0000","last_unstale":"2024-03-23T11:31:24.067643+0000","last_undegraded":"2024-03-23T11:31:24.067643+0000","last_fullsized":"2024-03-23T11:31:24.067643+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:36:56.686495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.070731+0000","last_change":"2024-03-23T11:31:24.091421+0000","last_active":"2024-03-23T11:31:25.070731+0000","last_peered":"2024-03-23T11:31:25.070731+0000","last_clean":"2024-03-23T11:31:25.070731+0000","last_became_active":"2024-03-23T11:31:24.091083+0000","last_became_peered":"2024-03-23T11:31:24.091083+0000","last_unstale":"2024-03-23T11:31:25.070731+0000","last_undegraded":"2024-03-23T11:31:25.070731+0000","last_fullsized":"2024-03-23T11:31:25.070731+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:59.077050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.061189+0000","last_change":"2024-03-23T11:31:23.583429+0000","last_active":"2024-03-23T11:31:24.061189+0000","last_peered":"2024-03-23T11:31:24.061189+0000","last_clean":"2024-03-23T11:31:24.061189+0000","last_became_active":"2024-03-23T11:31:23.583040+0000","last_became_peered":"2024-03-23T11:31:23.583040+0000","last_unstale":"2024-03-23T11:31:24.061189+0000","last_undegraded":"2024-03-23T11:31:24.061189+0000","last_fullsized":"2024-03-23T11:31:24.061189+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:25:12.126975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.562830+0000","last_change":"2024-03-23T11:31:24.086217+0000","last_active":"2024-03-23T11:31:25.562830+0000","last_peered":"2024-03-23T11:31:25.562830+0000","last_clean":"2024-03-23T11:31:25.562830+0000","last_became_active":"2024-03-23T11:31:24.085139+0000","last_became_peered":"2024-03-23T11:31:24.085139+0000","last_unstale":"2024-03-23T11:31:25.562830+0000","last_undegraded":"2024-03-23T11:31:25.562830+0000","last_fullsized":"2024-03-23T11:31:25.562830+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:34:52.128761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.055696+0000","last_change":"2024-03-23T11:31:23.578131+0000","last_active":"2024-03-23T11:31:24.055696+0000","last_peered":"2024-03-23T11:31:24.055696+0000","last_clean":"2024-03-23T11:31:24.055696+0000","last_became_active":"2024-03-23T11:31:23.575712+0000","last_became_peered":"2024-03-23T11:31:23.575712+0000","last_unstale":"2024-03-23T11:31:24.055696+0000","last_undegraded":"2024-03-23T11:31:24.055696+0000","last_fullsized":"2024-03-23T11:31:24.055696+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:08.838349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.474512+0000","last_change":"2024-03-23T11:31:24.068963+0000","last_active":"2024-03-23T11:31:25.474512+0000","last_peered":"2024-03-23T11:31:25.474512+0000","last_clean":"2024-03-23T11:31:25.474512+0000","last_became_active":"2024-03-23T11:31:24.068704+0000","last_became_peered":"2024-03-23T11:31:24.068704+0000","last_unstale":"2024-03-23T11:31:25.474512+0000","last_undegraded":"2024-03-23T11:31:25.474512+0000","last_fullsized":"2024-03-23T11:31:25.474512+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:15:26.611633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"117'3","reported_seq":14,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.097411+0000","last_change":"2024-03-23T11:31:23.580681+0000","last_active":"2024-03-23T11:31:24.097411+0000","last_peered":"2024-03-23T11:31:24.097411+0000","last_clean":"2024-03-23T11:31:24.097411+0000","last_became_active":"2024-03-23T11:31:23.580245+0000","last_became_peered":"2024-03-23T11:31:23.580245+0000","last_unstale":"2024-03-23T11:31:24.097411+0000","last_undegraded":"2024-03-23T11:31:24.097411+0000","last_fullsized":"2024-03-23T11:31:24.097411+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:46:53.689805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.070902+0000","last_change":"2024-03-23T11:31:24.085798+0000","last_active":"2024-03-23T11:31:25.070902+0000","last_peered":"2024-03-23T11:31:25.070902+0000","last_clean":"2024-03-23T11:31:25.070902+0000","last_became_active":"2024-03-23T11:31:24.085511+0000","last_became_peered":"2024-03-23T11:31:24.085511+0000","last_unstale":"2024-03-23T11:31:25.070902+0000","last_undegraded":"2024-03-23T11:31:25.070902+0000","last_fullsized":"2024-03-23T11:31:25.070902+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:09.843314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"117'6","reported_seq":17,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.102269+0000","last_change":"2024-03-23T11:31:23.571676+0000","last_active":"2024-03-23T11:31:24.102269+0000","last_peered":"2024-03-23T11:31:24.102269+0000","last_clean":"2024-03-23T11:31:24.102269+0000","last_became_active":"2024-03-23T11:31:23.571381+0000","last_became_peered":"2024-03-23T11:31:23.571381+0000","last_unstale":"2024-03-23T11:31:24.102269+0000","last_undegraded":"2024-03-23T11:31:24.102269+0000","last_fullsized":"2024-03-23T11:31:24.102269+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:48:53.978419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.474737+0000","last_change":"2024-03-23T11:31:24.077960+0000","last_active":"2024-03-23T11:31:25.474737+0000","last_peered":"2024-03-23T11:31:25.474737+0000","last_clean":"2024-03-23T11:31:25.474737+0000","last_became_active":"2024-03-23T11:31:24.077680+0000","last_became_peered":"2024-03-23T11:31:24.077680+0000","last_unstale":"2024-03-23T11:31:25.474737+0000","last_undegraded":"2024-03-23T11:31:25.474737+0000","last_fullsized":"2024-03-23T11:31:25.474737+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:27:30.987735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"117'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.083575+0000","last_change":"2024-03-23T11:31:23.577869+0000","last_active":"2024-03-23T11:31:24.083575+0000","last_peered":"2024-03-23T11:31:24.083575+0000","last_clean":"2024-03-23T11:31:24.083575+0000","last_became_active":"2024-03-23T11:31:23.577586+0000","last_became_peered":"2024-03-23T11:31:23.577586+0000","last_unstale":"2024-03-23T11:31:24.083575+0000","last_undegraded":"2024-03-23T11:31:24.083575+0000","last_fullsized":"2024-03-23T11:31:24.083575+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:02:56.394144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.474818+0000","last_change":"2024-03-23T11:31:24.071521+0000","last_active":"2024-03-23T11:31:25.474818+0000","last_peered":"2024-03-23T11:31:25.474818+0000","last_clean":"2024-03-23T11:31:25.474818+0000","last_became_active":"2024-03-23T11:31:24.071220+0000","last_became_peered":"2024-03-23T11:31:24.071220+0000","last_unstale":"2024-03-23T11:31:25.474818+0000","last_undegraded":"2024-03-23T11:31:25.474818+0000","last_fullsized":"2024-03-23T11:31:25.474818+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:36:57.691459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.071870+0000","last_change":"2024-03-23T11:31:23.571213+0000","last_active":"2024-03-23T11:31:24.071870+0000","last_peered":"2024-03-23T11:31:24.071870+0000","last_clean":"2024-03-23T11:31:24.071870+0000","last_became_active":"2024-03-23T11:31:23.570947+0000","last_became_peered":"2024-03-23T11:31:23.570947+0000","last_unstale":"2024-03-23T11:31:24.071870+0000","last_undegraded":"2024-03-23T11:31:24.071870+0000","last_fullsized":"2024-03-23T11:31:24.071870+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:08.838349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.474861+0000","last_change":"2024-03-23T11:31:24.071642+0000","last_active":"2024-03-23T11:31:25.474861+0000","last_peered":"2024-03-23T11:31:25.474861+0000","last_clean":"2024-03-23T11:31:25.474861+0000","last_became_active":"2024-03-23T11:31:24.071365+0000","last_became_peered":"2024-03-23T11:31:24.071365+0000","last_unstale":"2024-03-23T11:31:25.474861+0000","last_undegraded":"2024-03-23T11:31:25.474861+0000","last_fullsized":"2024-03-23T11:31:25.474861+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:34:52.128761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"117'8","reported_seq":19,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.097423+0000","last_change":"2024-03-23T11:31:23.570283+0000","last_active":"2024-03-23T11:31:24.097423+0000","last_peered":"2024-03-23T11:31:24.097423+0000","last_clean":"2024-03-23T11:31:24.097423+0000","last_became_active":"2024-03-23T11:31:23.569976+0000","last_became_peered":"2024-03-23T11:31:23.569976+0000","last_unstale":"2024-03-23T11:31:24.097423+0000","last_undegraded":"2024-03-23T11:31:24.097423+0000","last_fullsized":"2024-03-23T11:31:24.097423+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:21:01.865009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.360973+0000","last_change":"2024-03-23T11:31:24.085015+0000","last_active":"2024-03-23T11:31:25.360973+0000","last_peered":"2024-03-23T11:31:25.360973+0000","last_clean":"2024-03-23T11:31:25.360973+0000","last_became_active":"2024-03-23T11:31:24.080717+0000","last_became_peered":"2024-03-23T11:31:24.080717+0000","last_unstale":"2024-03-23T11:31:25.360973+0000","last_undegraded":"2024-03-23T11:31:25.360973+0000","last_fullsized":"2024-03-23T11:31:25.360973+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:23:00.517508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"117'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.086188+0000","last_change":"2024-03-23T11:31:23.577189+0000","last_active":"2024-03-23T11:31:24.086188+0000","last_peered":"2024-03-23T11:31:24.086188+0000","last_clean":"2024-03-23T11:31:24.086188+0000","last_became_active":"2024-03-23T11:31:23.576357+0000","last_became_peered":"2024-03-23T11:31:23.576357+0000","last_unstale":"2024-03-23T11:31:24.086188+0000","last_undegraded":"2024-03-23T11:31:24.086188+0000","last_fullsized":"2024-03-23T11:31:24.086188+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:31:43.172254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.067608+0000","last_change":"2024-03-23T11:31:24.089173+0000","last_active":"2024-03-23T11:31:25.067608+0000","last_peered":"2024-03-23T11:31:25.067608+0000","last_clean":"2024-03-23T11:31:25.067608+0000","last_became_active":"2024-03-23T11:31:24.087728+0000","last_became_peered":"2024-03-23T11:31:24.087728+0000","last_unstale":"2024-03-23T11:31:25.067608+0000","last_undegraded":"2024-03-23T11:31:25.067608+0000","last_fullsized":"2024-03-23T11:31:25.067608+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:25:20.896556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"117'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.094682+0000","last_change":"2024-03-23T11:31:23.577092+0000","last_active":"2024-03-23T11:31:24.094682+0000","last_peered":"2024-03-23T11:31:24.094682+0000","last_clean":"2024-03-23T11:31:24.094682+0000","last_became_active":"2024-03-23T11:31:23.574763+0000","last_became_peered":"2024-03-23T11:31:23.574763+0000","last_unstale":"2024-03-23T11:31:24.094682+0000","last_undegraded":"2024-03-23T11:31:24.094682+0000","last_fullsized":"2024-03-23T11:31:24.094682+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:23.091705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.072528+0000","last_change":"2024-03-23T11:31:24.091837+0000","last_active":"2024-03-23T11:31:25.072528+0000","last_peered":"2024-03-23T11:31:25.072528+0000","last_clean":"2024-03-23T11:31:25.072528+0000","last_became_active":"2024-03-23T11:31:24.091540+0000","last_became_peered":"2024-03-23T11:31:24.091540+0000","last_unstale":"2024-03-23T11:31:25.072528+0000","last_undegraded":"2024-03-23T11:31:25.072528+0000","last_fullsized":"2024-03-23T11:31:25.072528+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:28:11.838070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"117'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.083949+0000","last_change":"2024-03-23T11:31:23.570669+0000","last_active":"2024-03-23T11:31:24.083949+0000","last_peered":"2024-03-23T11:31:24.083949+0000","last_clean":"2024-03-23T11:31:24.083949+0000","last_became_active":"2024-03-23T11:31:23.570374+0000","last_became_peered":"2024-03-23T11:31:23.570374+0000","last_unstale":"2024-03-23T11:31:24.083949+0000","last_undegraded":"2024-03-23T11:31:24.083949+0000","last_fullsized":"2024-03-23T11:31:24.083949+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:29:42.109160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.073013+0000","last_change":"2024-03-23T11:31:24.082109+0000","last_active":"2024-03-23T11:31:25.073013+0000","last_peered":"2024-03-23T11:31:25.073013+0000","last_clean":"2024-03-23T11:31:25.073013+0000","last_became_active":"2024-03-23T11:31:24.080842+0000","last_became_peered":"2024-03-23T11:31:24.080842+0000","last_unstale":"2024-03-23T11:31:25.073013+0000","last_undegraded":"2024-03-23T11:31:25.073013+0000","last_fullsized":"2024-03-23T11:31:25.073013+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:22:29.190096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.066837+0000","last_change":"2024-03-23T11:31:23.574872+0000","last_active":"2024-03-23T11:31:24.066837+0000","last_peered":"2024-03-23T11:31:24.066837+0000","last_clean":"2024-03-23T11:31:24.066837+0000","last_became_active":"2024-03-23T11:31:23.572315+0000","last_became_peered":"2024-03-23T11:31:23.572315+0000","last_unstale":"2024-03-23T11:31:24.066837+0000","last_undegraded":"2024-03-23T11:31:24.066837+0000","last_fullsized":"2024-03-23T11:31:24.066837+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:14:30.980614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.072257+0000","last_change":"2024-03-23T11:31:24.070748+0000","last_active":"2024-03-23T11:31:25.072257+0000","last_peered":"2024-03-23T11:31:25.072257+0000","last_clean":"2024-03-23T11:31:25.072257+0000","last_became_active":"2024-03-23T11:31:24.070468+0000","last_became_peered":"2024-03-23T11:31:24.070468+0000","last_unstale":"2024-03-23T11:31:25.072257+0000","last_undegraded":"2024-03-23T11:31:25.072257+0000","last_fullsized":"2024-03-23T11:31:25.072257+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:55:09.712352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"117'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.093227+0000","last_change":"2024-03-23T11:31:23.583134+0000","last_active":"2024-03-23T11:31:24.093227+0000","last_peered":"2024-03-23T11:31:24.093227+0000","last_clean":"2024-03-23T11:31:24.093227+0000","last_became_active":"2024-03-23T11:31:23.582448+0000","last_became_peered":"2024-03-23T11:31:23.582448+0000","last_unstale":"2024-03-23T11:31:24.093227+0000","last_undegraded":"2024-03-23T11:31:24.093227+0000","last_fullsized":"2024-03-23T11:31:24.093227+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:41.262297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.362971+0000","last_change":"2024-03-23T11:31:24.083929+0000","last_active":"2024-03-23T11:31:26.362971+0000","last_peered":"2024-03-23T11:31:26.362971+0000","last_clean":"2024-03-23T11:31:26.362971+0000","last_became_active":"2024-03-23T11:31:24.083190+0000","last_became_peered":"2024-03-23T11:31:24.083190+0000","last_unstale":"2024-03-23T11:31:26.362971+0000","last_undegraded":"2024-03-23T11:31:26.362971+0000","last_fullsized":"2024-03-23T11:31:26.362971+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:14:53.768296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.060206+0000","last_change":"2024-03-23T11:31:23.573054+0000","last_active":"2024-03-23T11:31:24.060206+0000","last_peered":"2024-03-23T11:31:24.060206+0000","last_clean":"2024-03-23T11:31:24.060206+0000","last_became_active":"2024-03-23T11:31:23.572721+0000","last_became_peered":"2024-03-23T11:31:23.572721+0000","last_unstale":"2024-03-23T11:31:24.060206+0000","last_undegraded":"2024-03-23T11:31:24.060206+0000","last_fullsized":"2024-03-23T11:31:24.060206+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:59:59.275689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.072931+0000","last_change":"2024-03-23T11:31:24.090542+0000","last_active":"2024-03-23T11:31:25.072931+0000","last_peered":"2024-03-23T11:31:25.072931+0000","last_clean":"2024-03-23T11:31:25.072931+0000","last_became_active":"2024-03-23T11:31:24.090233+0000","last_became_peered":"2024-03-23T11:31:24.090233+0000","last_unstale":"2024-03-23T11:31:25.072931+0000","last_undegraded":"2024-03-23T11:31:25.072931+0000","last_fullsized":"2024-03-23T11:31:25.072931+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:59:30.438327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.063470+0000","last_change":"2024-03-23T11:31:23.581618+0000","last_active":"2024-03-23T11:31:24.063470+0000","last_peered":"2024-03-23T11:31:24.063470+0000","last_clean":"2024-03-23T11:31:24.063470+0000","last_became_active":"2024-03-23T11:31:23.578691+0000","last_became_peered":"2024-03-23T11:31:23.578691+0000","last_unstale":"2024-03-23T11:31:24.063470+0000","last_undegraded":"2024-03-23T11:31:24.063470+0000","last_fullsized":"2024-03-23T11:31:24.063470+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:00.392534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.474541+0000","last_change":"2024-03-23T11:31:24.084759+0000","last_active":"2024-03-23T11:31:25.474541+0000","last_peered":"2024-03-23T11:31:25.474541+0000","last_clean":"2024-03-23T11:31:25.474541+0000","last_became_active":"2024-03-23T11:31:24.084403+0000","last_became_peered":"2024-03-23T11:31:24.084403+0000","last_unstale":"2024-03-23T11:31:25.474541+0000","last_undegraded":"2024-03-23T11:31:25.474541+0000","last_fullsized":"2024-03-23T11:31:25.474541+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:46:54.694770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.067747+0000","last_change":"2024-03-23T11:31:23.571707+0000","last_active":"2024-03-23T11:31:24.067747+0000","last_peered":"2024-03-23T11:31:24.067747+0000","last_clean":"2024-03-23T11:31:24.067747+0000","last_became_active":"2024-03-23T11:31:23.570962+0000","last_became_peered":"2024-03-23T11:31:23.570962+0000","last_unstale":"2024-03-23T11:31:24.067747+0000","last_undegraded":"2024-03-23T11:31:24.067747+0000","last_fullsized":"2024-03-23T11:31:24.067747+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:29:34.803613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.562424+0000","last_change":"2024-03-23T11:31:24.085632+0000","last_active":"2024-03-23T11:31:25.562424+0000","last_peered":"2024-03-23T11:31:25.562424+0000","last_clean":"2024-03-23T11:31:25.562424+0000","last_became_active":"2024-03-23T11:31:24.085266+0000","last_became_peered":"2024-03-23T11:31:24.085266+0000","last_unstale":"2024-03-23T11:31:25.562424+0000","last_undegraded":"2024-03-23T11:31:25.562424+0000","last_fullsized":"2024-03-23T11:31:25.562424+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:43:42.414576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"117'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.088322+0000","last_change":"2024-03-23T11:31:23.579560+0000","last_active":"2024-03-23T11:31:24.088322+0000","last_peered":"2024-03-23T11:31:24.088322+0000","last_clean":"2024-03-23T11:31:24.088322+0000","last_became_active":"2024-03-23T11:31:23.577407+0000","last_became_peered":"2024-03-23T11:31:23.577407+0000","last_unstale":"2024-03-23T11:31:24.088322+0000","last_undegraded":"2024-03-23T11:31:24.088322+0000","last_fullsized":"2024-03-23T11:31:24.088322+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:29:42.109160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.364287+0000","last_change":"2024-03-23T11:31:24.087316+0000","last_active":"2024-03-23T11:31:26.364287+0000","last_peered":"2024-03-23T11:31:26.364287+0000","last_clean":"2024-03-23T11:31:26.364287+0000","last_became_active":"2024-03-23T11:31:24.087000+0000","last_became_peered":"2024-03-23T11:31:24.087000+0000","last_unstale":"2024-03-23T11:31:26.364287+0000","last_undegraded":"2024-03-23T11:31:26.364287+0000","last_fullsized":"2024-03-23T11:31:26.364287+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:14:46.381125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.072983+0000","last_change":"2024-03-23T11:31:23.577071+0000","last_active":"2024-03-23T11:31:24.072983+0000","last_peered":"2024-03-23T11:31:24.072983+0000","last_clean":"2024-03-23T11:31:24.072983+0000","last_became_active":"2024-03-23T11:31:23.576626+0000","last_became_peered":"2024-03-23T11:31:23.576626+0000","last_unstale":"2024-03-23T11:31:24.072983+0000","last_undegraded":"2024-03-23T11:31:24.072983+0000","last_fullsized":"2024-03-23T11:31:24.072983+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:24:08.473235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.364279+0000","last_change":"2024-03-23T11:31:24.090433+0000","last_active":"2024-03-23T11:31:26.364279+0000","last_peered":"2024-03-23T11:31:26.364279+0000","last_clean":"2024-03-23T11:31:26.364279+0000","last_became_active":"2024-03-23T11:31:24.089737+0000","last_became_peered":"2024-03-23T11:31:24.089737+0000","last_unstale":"2024-03-23T11:31:26.364279+0000","last_undegraded":"2024-03-23T11:31:26.364279+0000","last_fullsized":"2024-03-23T11:31:26.364279+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:30:38.332361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.074007+0000","last_change":"2024-03-23T11:31:23.575691+0000","last_active":"2024-03-23T11:31:24.074007+0000","last_peered":"2024-03-23T11:31:24.074007+0000","last_clean":"2024-03-23T11:31:24.074007+0000","last_became_active":"2024-03-23T11:31:23.575362+0000","last_became_peered":"2024-03-23T11:31:23.575362+0000","last_unstale":"2024-03-23T11:31:24.074007+0000","last_undegraded":"2024-03-23T11:31:24.074007+0000","last_fullsized":"2024-03-23T11:31:24.074007+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:51:26.294819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.562289+0000","last_change":"2024-03-23T11:31:24.081588+0000","last_active":"2024-03-23T11:31:25.562289+0000","last_peered":"2024-03-23T11:31:25.562289+0000","last_clean":"2024-03-23T11:31:25.562289+0000","last_became_active":"2024-03-23T11:31:24.081169+0000","last_became_peered":"2024-03-23T11:31:24.081169+0000","last_unstale":"2024-03-23T11:31:25.562289+0000","last_undegraded":"2024-03-23T11:31:25.562289+0000","last_fullsized":"2024-03-23T11:31:25.562289+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:15:26.611633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.057207+0000","last_change":"2024-03-23T11:31:23.583746+0000","last_active":"2024-03-23T11:31:24.057207+0000","last_peered":"2024-03-23T11:31:24.057207+0000","last_clean":"2024-03-23T11:31:24.057207+0000","last_became_active":"2024-03-23T11:31:23.582877+0000","last_became_peered":"2024-03-23T11:31:23.582877+0000","last_unstale":"2024-03-23T11:31:24.057207+0000","last_undegraded":"2024-03-23T11:31:24.057207+0000","last_fullsized":"2024-03-23T11:31:24.057207+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:55:31.725691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.362614+0000","last_change":"2024-03-23T11:31:24.073937+0000","last_active":"2024-03-23T11:31:26.362614+0000","last_peered":"2024-03-23T11:31:26.362614+0000","last_clean":"2024-03-23T11:31:26.362614+0000","last_became_active":"2024-03-23T11:31:24.073619+0000","last_became_peered":"2024-03-23T11:31:24.073619+0000","last_unstale":"2024-03-23T11:31:26.362614+0000","last_undegraded":"2024-03-23T11:31:26.362614+0000","last_fullsized":"2024-03-23T11:31:26.362614+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:43:42.414576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.066763+0000","last_change":"2024-03-23T11:31:23.575974+0000","last_active":"2024-03-23T11:31:24.066763+0000","last_peered":"2024-03-23T11:31:24.066763+0000","last_clean":"2024-03-23T11:31:24.066763+0000","last_became_active":"2024-03-23T11:31:23.575521+0000","last_became_peered":"2024-03-23T11:31:23.575521+0000","last_unstale":"2024-03-23T11:31:24.066763+0000","last_undegraded":"2024-03-23T11:31:24.066763+0000","last_fullsized":"2024-03-23T11:31:24.066763+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:29:14.159484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.362905+0000","last_change":"2024-03-23T11:31:24.071290+0000","last_active":"2024-03-23T11:31:26.362905+0000","last_peered":"2024-03-23T11:31:26.362905+0000","last_clean":"2024-03-23T11:31:26.362905+0000","last_became_active":"2024-03-23T11:31:24.070624+0000","last_became_peered":"2024-03-23T11:31:24.070624+0000","last_unstale":"2024-03-23T11:31:26.362905+0000","last_undegraded":"2024-03-23T11:31:26.362905+0000","last_fullsized":"2024-03-23T11:31:26.362905+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:34:52.128761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.055659+0000","last_change":"2024-03-23T11:31:23.581381+0000","last_active":"2024-03-23T11:31:24.055659+0000","last_peered":"2024-03-23T11:31:24.055659+0000","last_clean":"2024-03-23T11:31:24.055659+0000","last_became_active":"2024-03-23T11:31:23.581060+0000","last_became_peered":"2024-03-23T11:31:23.581060+0000","last_unstale":"2024-03-23T11:31:24.055659+0000","last_undegraded":"2024-03-23T11:31:24.055659+0000","last_fullsized":"2024-03-23T11:31:24.055659+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:45:52.136449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.073190+0000","last_change":"2024-03-23T11:31:24.079015+0000","last_active":"2024-03-23T11:31:25.073190+0000","last_peered":"2024-03-23T11:31:25.073190+0000","last_clean":"2024-03-23T11:31:25.073190+0000","last_became_active":"2024-03-23T11:31:24.078551+0000","last_became_peered":"2024-03-23T11:31:24.078551+0000","last_unstale":"2024-03-23T11:31:25.073190+0000","last_undegraded":"2024-03-23T11:31:25.073190+0000","last_fullsized":"2024-03-23T11:31:25.073190+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:25:49.848713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.060482+0000","last_change":"2024-03-23T11:31:23.571091+0000","last_active":"2024-03-23T11:31:24.060482+0000","last_peered":"2024-03-23T11:31:24.060482+0000","last_clean":"2024-03-23T11:31:24.060482+0000","last_became_active":"2024-03-23T11:31:23.570813+0000","last_became_peered":"2024-03-23T11:31:23.570813+0000","last_unstale":"2024-03-23T11:31:24.060482+0000","last_undegraded":"2024-03-23T11:31:24.060482+0000","last_fullsized":"2024-03-23T11:31:24.060482+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:37:48.520573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:26.364182+0000","last_change":"2024-03-23T11:31:24.082917+0000","last_active":"2024-03-23T11:31:26.364182+0000","last_peered":"2024-03-23T11:31:26.364182+0000","last_clean":"2024-03-23T11:31:26.364182+0000","last_became_active":"2024-03-23T11:31:24.082582+0000","last_became_peered":"2024-03-23T11:31:24.082582+0000","last_unstale":"2024-03-23T11:31:26.364182+0000","last_undegraded":"2024-03-23T11:31:26.364182+0000","last_fullsized":"2024-03-23T11:31:26.364182+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:34:30.877715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.067921+0000","last_change":"2024-03-23T11:31:23.571209+0000","last_active":"2024-03-23T11:31:24.067921+0000","last_peered":"2024-03-23T11:31:24.067921+0000","last_clean":"2024-03-23T11:31:24.067921+0000","last_became_active":"2024-03-23T11:31:23.570618+0000","last_became_peered":"2024-03-23T11:31:23.570618+0000","last_unstale":"2024-03-23T11:31:24.067921+0000","last_undegraded":"2024-03-23T11:31:24.067921+0000","last_fullsized":"2024-03-23T11:31:24.067921+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:45:53.606801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-23T11:31:25.474114+0000","last_change":"2024-03-23T11:31:24.066657+0000","last_active":"2024-03-23T11:31:25.474114+0000","last_peered":"2024-03-23T11:31:25.474114+0000","last_clean":"2024-03-23T11:31:25.474114+0000","last_became_active":"2024-03-23T11:31:24.065593+0000","last_became_peered":"2024-03-23T11:31:24.065593+0000","last_unstale":"2024-03-23T11:31:25.474114+0000","last_undegraded":"2024-03-23T11:31:25.474114+0000","last_fullsized":"2024-03-23T11:31:25.474114+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:23.533089+0000","last_clean_scrub_stamp":"2024-03-23T11:31:23.533089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:48:54.983383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.072179+0000","last_change":"2024-03-23T11:31:23.573935+0000","last_active":"2024-03-23T11:31:24.072179+0000","last_peered":"2024-03-23T11:31:24.072179+0000","last_clean":"2024-03-23T11:31:24.072179+0000","last_became_active":"2024-03-23T11:31:23.573650+0000","last_became_peered":"2024-03-23T11:31:23.573650+0000","last_unstale":"2024-03-23T11:31:24.072179+0000","last_undegraded":"2024-03-23T11:31:24.072179+0000","last_fullsized":"2024-03-23T11:31:24.072179+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:31:22.528125+0000","last_clean_scrub_stamp":"2024-03-23T11:31:22.528125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:45:52.136449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":266,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-23T11:31:24.058027+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:31:24.058027+0000","last_peered":"2024-03-23T11:31:24.058027+0000","last_clean":"2024-03-23T11:31:24.058027+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:31:24.058027+0000","last_undegraded":"2024-03-23T11:31:24.058027+0000","last_fullsized":"2024-03-23T11:31:24.058027+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542299,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33132,"kb_used_data":2016,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338708,"statfs":{"total":96636764160,"available":96602836992,"internally_reserved":0,"allocated":2064384,"data_stored":1225461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.0960000000000001,"15min":1.0169999999999999},"min":{"1min":0.77500000000000002,"5min":0.5,"15min":0.5},"max":{"1min":1.556,"5min":2.8370000000000002,"15min":2.8370000000000002},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.1120000000000001,"15min":1.052},"min":{"1min":0.86899999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3300000000000001,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.5449999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.2929999999999999,"15min":1.212},"min":{"1min":0.80000000000000004,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.6559999999999999,"5min":3.569,"15min":3.569},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.306,"15min":1.27},"min":{"1min":1.0649999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.5649999999999999,"5min":1.8009999999999999,"15min":2.4350000000000001},"last":1.147}]},{"osd":2,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1319999999999999,"15min":1.0900000000000001},"min":{"1min":0.65500000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.4379999999999999,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.262,"5min":1.2410000000000001,"15min":1.1950000000000001},"min":{"1min":0.81499999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.7130000000000001,"5min":3.524,"15min":3.524},"last":1.7609999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.22,"15min":1.1870000000000001},"min":{"1min":0.996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.55,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.2},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3240000000000001,"15min":1.2769999999999999},"min":{"1min":1.026,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.605,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":1.5129999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.1890000000000001,"15min":1.212},"min":{"1min":0.73199999999999998,"5min":0.65300000000000002,"15min":0.54300000000000004},"max":{"1min":1.496,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.411,"15min":1.395},"min":{"1min":1.2050000000000001,"5min":0.63800000000000001,"15min":0.60199999999999998},"max":{"1min":1.744,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":1.597}]},{"osd":5,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.3799999999999999,"15min":1.3500000000000001},"min":{"1min":0.877,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.7789999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.4730000000000001,"15min":1.425},"min":{"1min":0.96399999999999997,"5min":0.77100000000000002,"15min":0.58199999999999996},"max":{"1min":1.869,"5min":2.0219999999999998,"15min":2.0859999999999999},"last":1.3580000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:31:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.4279999999999999,"15min":1.4259999999999999},"min":{"1min":1.0489999999999999,"5min":0.73599999999999999,"15min":0.61399999999999999},"max":{"1min":1.923,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.401,"15min":1.373},"min":{"1min":1.03,"5min":0.61699999999999999,"15min":0.59899999999999998},"max":{"1min":1.794,"5min":3.218,"15min":3.218},"last":1.48}]}]},{"osd":6,"up_from":14,"seq":60129542299,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1462272,"data_stored":633878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1020000000000001,"15min":1.1120000000000001},"min":{"1min":0.72999999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.7450000000000001,"5min":2.403,"15min":3.9140000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.264,"5min":1.2010000000000001,"15min":1.1579999999999999},"min":{"1min":0.92400000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.597,"5min":1.8600000000000001,"15min":6.6539999999999999},"last":1.165}]},{"osd":1,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.1859999999999999,"15min":1.206},"min":{"1min":0.84899999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.627,"5min":2.4820000000000002,"15min":2.4820000000000002},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.161,"15min":1.2070000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.448,"5min":1.9510000000000001,"15min":2.0859999999999999},"last":1.857}]},{"osd":2,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.161,"15min":1.214},"min":{"1min":0.81499999999999995,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.5980000000000001,"5min":1.649,"15min":3.73},"last":1.526},{"interface":"front","average":{"1min":1.399,"5min":1.3220000000000001,"15min":1.296},"min":{"1min":1.077,"5min":0.73499999999999999,"15min":0.68100000000000005},"max":{"1min":1.6859999999999999,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.224}]},{"osd":3,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.2470000000000001,"15min":1.278},"min":{"1min":1.0840000000000001,"5min":0.76700000000000002,"15min":0.749},"max":{"1min":1.738,"5min":1.9890000000000001,"15min":12.914},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.3220000000000001,"15min":1.337},"min":{"1min":1.0169999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6220000000000001,"5min":1.9670000000000001,"15min":3.8100000000000001},"last":1.3160000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.3260000000000001,"15min":1.335},"min":{"1min":0.89400000000000002,"5min":0.68000000000000005,"15min":0.51100000000000001},"max":{"1min":1.8100000000000001,"5min":2.4420000000000002,"15min":2.702},"last":1.363},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.3089999999999999,"15min":1.361},"min":{"1min":0.92200000000000004,"5min":0.752,"15min":0.752},"max":{"1min":1.552,"5min":2.1080000000000001,"15min":7.2140000000000004},"last":1.746}]},{"osd":5,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4119999999999999,"15min":1.4610000000000001},"min":{"1min":1.1100000000000001,"5min":0.92900000000000005,"15min":0.73899999999999999},"max":{"1min":1.6799999999999999,"5min":2.0270000000000001,"15min":3.097},"last":1.9370000000000001},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.458,"15min":1.468},"min":{"1min":1.1799999999999999,"5min":0.86899999999999999,"15min":0.72499999999999998},"max":{"1min":1.8680000000000001,"5min":3.3109999999999999,"15min":3.3109999999999999},"last":1.6879999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.5609999999999999,"15min":1.597},"min":{"1min":1.377,"5min":0.97799999999999998,"15min":0.80400000000000005},"max":{"1min":1.738,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":2.0150000000000001},{"interface":"front","average":{"1min":1.383,"5min":1.482,"15min":1.4710000000000001},"min":{"1min":1.0049999999999999,"5min":0.70399999999999996,"15min":0.57699999999999996},"max":{"1min":1.72,"5min":4.085,"15min":4.085},"last":2.0640000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542299,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32552,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339288,"statfs":{"total":96636764160,"available":96603430912,"internally_reserved":0,"allocated":1470464,"data_stored":635081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.1519999999999999,"15min":1.1359999999999999},"min":{"1min":0.81899999999999995,"5min":0.65500000000000003,"15min":0.55400000000000005},"max":{"1min":1.538,"5min":2.0179999999999998,"15min":2.2999999999999998},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.214,"5min":1.155,"15min":1.1479999999999999},"min":{"1min":0.624,"5min":0.624,"15min":0.47799999999999998},"max":{"1min":1.6619999999999999,"5min":2.1190000000000002,"15min":2.355},"last":1.518}]},{"osd":1,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.198,"15min":1.173},"min":{"1min":0.94399999999999995,"5min":0.76700000000000002,"15min":0.65300000000000002},"max":{"1min":1.605,"5min":1.6699999999999999,"15min":2.2559999999999998},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.202,"15min":1.1910000000000001},"min":{"1min":0.81599999999999995,"5min":0.69299999999999995,"15min":0.495},"max":{"1min":1.5589999999999999,"5min":1.627,"15min":1.8580000000000001},"last":1.3939999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2010000000000001,"15min":1.2250000000000001},"min":{"1min":0.93000000000000005,"5min":0.70999999999999996,"15min":0.60199999999999998},"max":{"1min":1.5269999999999999,"5min":1.6659999999999999,"15min":2.012},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.2969999999999999,"15min":1.254},"min":{"1min":1.117,"5min":0.77600000000000002,"15min":0.64400000000000002},"max":{"1min":1.76,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.4850000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2769999999999999,"15min":1.29},"min":{"1min":0.89300000000000002,"5min":0.69299999999999995,"15min":0.66800000000000004},"max":{"1min":1.611,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3879999999999999,"15min":1.381},"min":{"1min":1.02,"5min":0.78600000000000003,"15min":0.73199999999999998},"max":{"1min":1.7509999999999999,"5min":2.0750000000000002,"15min":2.3999999999999999},"last":1.593}]},{"osd":4,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3109999999999999,"15min":1.3400000000000001},"min":{"1min":1.052,"5min":0.754,"15min":0.56699999999999995},"max":{"1min":1.653,"5min":2.1499999999999999,"15min":2.2090000000000001},"last":1.236},{"interface":"front","average":{"1min":1.518,"5min":1.4830000000000001,"15min":1.492},"min":{"1min":1.2250000000000001,"5min":1.0169999999999999,"15min":0.65000000000000002},"max":{"1min":1.7410000000000001,"5min":2.2839999999999998,"15min":2.5009999999999999},"last":1.1240000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.393,"15min":1.4059999999999999},"min":{"1min":1.0209999999999999,"5min":0.91200000000000003,"15min":0.48899999999999999},"max":{"1min":1.681,"5min":2.3170000000000002,"15min":2.4860000000000002},"last":1.333},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.4179999999999999,"15min":1.4350000000000001},"min":{"1min":1.03,"5min":0.86599999999999999,"15min":0.624},"max":{"1min":1.7390000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.98699999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.4690000000000001,"15min":1.484},"min":{"1min":1.1479999999999999,"5min":0.90900000000000003,"15min":0.54000000000000004},"max":{"1min":2.1549999999999998,"5min":2.5720000000000001,"15min":2.7229999999999999},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":1.345,"5min":1.099,"15min":0.60099999999999998},"max":{"1min":1.851,"5min":2.2269999999999999,"15min":2.7509999999999999},"last":1.542}]}]},{"osd":4,"up_from":14,"seq":60129542299,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1462272,"data_stored":634477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.278,"15min":1.1819999999999999},"min":{"1min":1.044,"5min":0.71199999999999997,"15min":0.58999999999999997},"max":{"1min":1.73,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.361},{"interface":"front","average":{"1min":1.319,"5min":1.335,"15min":1.276},"min":{"1min":0.94199999999999995,"5min":0.74399999999999999,"15min":0.63700000000000001},"max":{"1min":1.5269999999999999,"5min":1.7769999999999999,"15min":1.8009999999999999},"last":1.2849999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.171,"15min":1.1619999999999999},"min":{"1min":0.84699999999999998,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.4079999999999999,"5min":1.9350000000000001,"15min":4.327},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.2130000000000001,"15min":1.1599999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.66200000000000003},"max":{"1min":1.6970000000000001,"5min":1.7470000000000001,"15min":2.4369999999999998},"last":1.155}]},{"osd":2,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2,"15min":1.2110000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.624},"max":{"1min":1.5780000000000001,"5min":2.0960000000000001,"15min":2.6080000000000001},"last":0.998},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.2689999999999999,"15min":1.2649999999999999},"min":{"1min":1.0960000000000001,"5min":0.80600000000000005,"15min":0.71799999999999997},"max":{"1min":1.6299999999999999,"5min":1.8919999999999999,"15min":4.1310000000000002},"last":1.3400000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2509999999999999,"15min":1.2330000000000001},"min":{"1min":0.89600000000000002,"5min":0.747,"15min":0.622},"max":{"1min":1.659,"5min":1.659,"15min":2.5579999999999998},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.365,"15min":1.325},"min":{"1min":1.165,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.655,"5min":2.024,"15min":4.2400000000000002},"last":1.522}]},{"osd":5,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.246,"15min":1.2509999999999999},"min":{"1min":0.85799999999999998,"5min":0.77100000000000002,"15min":0.74299999999999999},"max":{"1min":1.5940000000000001,"5min":1.9490000000000001,"15min":2.508},"last":1.474},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.321,"15min":1.355},"min":{"1min":1.004,"5min":0.67800000000000005,"15min":0.60499999999999998},"max":{"1min":1.7849999999999999,"5min":1.8460000000000001,"15min":5.859},"last":1.577}]},{"osd":6,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.393,"15min":1.3979999999999999},"min":{"1min":1.095,"5min":0.753,"15min":0.58899999999999997},"max":{"1min":1.609,"5min":2.194,"15min":5.3339999999999996},"last":1.109},{"interface":"front","average":{"1min":1.488,"5min":1.468,"15min":1.486},"min":{"1min":1.0349999999999999,"5min":0.81000000000000005,"15min":0.72599999999999998},"max":{"1min":1.762,"5min":1.875,"15min":5.3940000000000001},"last":1.4470000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:31:08 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.512,"15min":1.494},"min":{"1min":1.24,"5min":0.84699999999999998,"15min":0.754},"max":{"1min":1.681,"5min":2.0790000000000002,"15min":2.6859999999999999},"last":1.409},{"interface":"front","average":{"1min":1.444,"5min":1.3799999999999999,"15min":1.3839999999999999},"min":{"1min":1.032,"5min":0.79000000000000004,"15min":0.75600000000000001},"max":{"1min":1.794,"5min":1.8560000000000001,"15min":2.641},"last":1.0589999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542299,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32560,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339280,"statfs":{"total":96636764160,"available":96603422720,"internally_reserved":0,"allocated":1478656,"data_stored":636111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.873,"15min":0.92400000000000004},"min":{"1min":0.39300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.29,"5min":1.323,"15min":1.6819999999999999},"last":1.177},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.0980000000000001,"15min":1.1040000000000001},"min":{"1min":0.625,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.5329999999999999,"5min":1.8540000000000001,"15min":2.0790000000000002},"last":0.85699999999999998}]},{"osd":1,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.96299999999999997,"15min":0.98899999999999999},"min":{"1min":0.68899999999999995,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.976},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.1890000000000001,"15min":1.2390000000000001},"min":{"1min":0.55300000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6020000000000001,"5min":1.774,"15min":2.2309999999999999},"last":1.4950000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.105,"15min":1.125},"min":{"1min":0.753,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.532,"5min":1.7729999999999999,"15min":3.3109999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.27,"5min":1.167,"15min":1.2030000000000001},"min":{"1min":0.67600000000000005,"5min":0.51900000000000002,"15min":0.48399999999999999},"max":{"1min":1.5820000000000001,"5min":1.635,"15min":2.7109999999999999},"last":1.0589999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2450000000000001,"15min":1.268},"min":{"1min":0.95599999999999996,"5min":0.81499999999999995,"15min":0.73199999999999998},"max":{"1min":1.5669999999999999,"5min":1.752,"15min":2.3490000000000002},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.458,"5min":1.431,"15min":1.4710000000000001},"min":{"1min":1.119,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.649,"5min":2.081,"15min":4.8970000000000002},"last":1.825}]},{"osd":5,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.4199999999999999,"15min":1.4770000000000001},"min":{"1min":1.0629999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":1.75,"5min":2.121,"15min":3.4910000000000001},"last":1.7729999999999999},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.339,"15min":1.407},"min":{"1min":1.0129999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.855,"5min":1.8700000000000001,"15min":3.125},"last":1.7010000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.371,"15min":1.4299999999999999},"min":{"1min":0.873,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":4.5499999999999998},"last":1.9550000000000001},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4219999999999999,"15min":1.4970000000000001},"min":{"1min":0.995,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":2.2080000000000002,"5min":2.2080000000000002,"15min":2.8450000000000002},"last":1.9339999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.3979999999999999,"15min":1.458},"min":{"1min":1.0880000000000001,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.998,"5min":1.998,"15min":2.952},"last":1.879},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.482,"15min":1.544},"min":{"1min":0.98699999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.964,"5min":1.964,"15min":4.798},"last":1.629}]}]},{"osd":2,"up_from":14,"seq":60129542299,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1462272,"data_stored":634477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.86199999999999999,"15min":0.89200000000000002},"min":{"1min":0.63900000000000001,"5min":0.246,"15min":0.246},"max":{"1min":1.202,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.0660000000000001,"15min":1.056},"min":{"1min":0.78100000000000003,"5min":0.40300000000000002,"15min":0.35099999999999998},"max":{"1min":1.641,"5min":1.6759999999999999,"15min":1.8220000000000001},"last":1.103}]},{"osd":1,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.85899999999999999,"15min":0.90100000000000002},"min":{"1min":0.66300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.798,"5min":1.798,"15min":1.9430000000000001},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0309999999999999},"min":{"1min":0.71499999999999997,"5min":0.39600000000000002,"15min":0.35799999999999998},"max":{"1min":1.458,"5min":1.593,"15min":1.7230000000000001},"last":0.67100000000000004}]},{"osd":3,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.099,"15min":1.113},"min":{"1min":0.82399999999999995,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.589,"5min":1.758,"15min":1.98},"last":1.179},{"interface":"front","average":{"1min":1.117,"5min":1.0669999999999999,"15min":1.0740000000000001},"min":{"1min":0.90000000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4239999999999999,"5min":1.6879999999999999,"15min":2.1560000000000001},"last":1.0169999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.343,"15min":1.3540000000000001},"min":{"1min":1.224,"5min":0.73299999999999998,"15min":0.69599999999999995},"max":{"1min":1.597,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.157},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4219999999999999,"15min":1.407},"min":{"1min":1.1839999999999999,"5min":0.93700000000000006,"15min":0.83499999999999996},"max":{"1min":1.7470000000000001,"5min":1.9470000000000001,"15min":2.085},"last":1.2909999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.319,"15min":1.3620000000000001},"min":{"1min":1.0429999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.7689999999999999,"5min":1.8720000000000001,"15min":2.5830000000000002},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.401,"5min":1.3080000000000001,"15min":1.3580000000000001},"min":{"1min":1.2370000000000001,"5min":0.73999999999999999,"15min":0.66000000000000003},"max":{"1min":1.5429999999999999,"5min":1.96,"15min":2.6469999999999998},"last":1.246}]},{"osd":6,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.3979999999999999,"15min":1.385},"min":{"1min":1.256,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.859,"5min":2.0600000000000001,"15min":2.758},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.389,"15min":1.444},"min":{"1min":1.135,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.738,"5min":1.883,"15min":2.8580000000000001},"last":1.593}]},{"osd":7,"last update":"Sat Mar 23 11:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.454,"15min":1.4359999999999999},"min":{"1min":1.3300000000000001,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8049999999999999,"5min":2.0150000000000001,"15min":2.246},"last":1.413},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.45,"15min":1.512},"min":{"1min":1.204,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.714,"5min":2.012,"15min":2.7120000000000002},"last":1.7310000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542300,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32556,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339284,"statfs":{"total":96636764160,"available":96603426816,"internally_reserved":0,"allocated":1466368,"data_stored":633932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.88800000000000001,"15min":0.89100000000000001},"min":{"1min":0.70599999999999996,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.6379999999999999,"5min":1.6930000000000001,"15min":3.2959999999999998},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.081,"5min":1.0600000000000001,"15min":1.014},"min":{"1min":0.56699999999999995,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.5840000000000001,"5min":1.8480000000000001,"15min":1.99},"last":1.1950000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.93700000000000006,"15min":0.95099999999999996},"min":{"1min":0.64400000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.212,"5min":1.5549999999999999,"15min":1.738},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.109,"15min":1.1060000000000001},"min":{"1min":0.70399999999999996,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.53,"5min":1.762,"15min":2.2160000000000002},"last":0.998}]},{"osd":3,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.23,"15min":1.2250000000000001},"min":{"1min":0.69799999999999995,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.837,"5min":1.8819999999999999,"15min":2.4430000000000001},"last":1.117},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1319999999999999,"15min":1.1299999999999999},"min":{"1min":0.496,"5min":0.45900000000000002,"15min":0.434},"max":{"1min":1.6120000000000001,"5min":2.0539999999999998,"15min":2.4169999999999998},"last":1.3260000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.339,"15min":1.3100000000000001},"min":{"1min":1.139,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.556,"5min":1.9019999999999999,"15min":2.153},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.296,"15min":1.3029999999999999},"min":{"1min":0.95999999999999996,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.6140000000000001,"5min":1.867,"15min":2.758},"last":1.1599999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3200000000000001,"15min":1.3620000000000001},"min":{"1min":0.86799999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6579999999999999,"5min":1.992,"15min":2.6589999999999998},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.423,"15min":1.431},"min":{"1min":1.214,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":3.0329999999999999},"last":1.609}]},{"osd":6,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.413,"15min":1.4199999999999999},"min":{"1min":1.022,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.7310000000000001,"5min":2.1600000000000001,"15min":2.2730000000000001},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4319999999999999,"15min":1.474},"min":{"1min":1.196,"5min":0.747,"15min":0.747},"max":{"1min":1.732,"5min":1.8959999999999999,"15min":2.9630000000000001},"last":1.2969999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.5269999999999999,"15min":1.5329999999999999},"min":{"1min":1.2310000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.0030000000000001,"5min":2.2450000000000001,"15min":3.2749999999999999},"last":1.7370000000000001},{"interface":"front","average":{"1min":1.5,"5min":1.5149999999999999,"15min":1.534},"min":{"1min":1.163,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.714,"5min":2.242,"15min":2.7829999999999999},"last":1.508}]}]},{"osd":0,"up_from":14,"seq":60129542298,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33140,"kb_used_data":2024,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338700,"statfs":{"total":96636764160,"available":96602828800,"internally_reserved":0,"allocated":2072576,"data_stored":1225852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.83299999999999996,"15min":0.85599999999999998},"min":{"1min":0.72599999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.2789999999999999,"5min":1.8660000000000001,"15min":2.8479999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.052,"5min":0.99099999999999999,"15min":0.97099999999999997},"min":{"1min":0.77300000000000002,"5min":0.46600000000000003,"15min":0.30399999999999999},"max":{"1min":1.466,"5min":1.544,"15min":2.4319999999999999},"last":1.0620000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.01,"15min":1.036},"min":{"1min":0.86799999999999999,"5min":0.41799999999999998,"15min":0.33700000000000002},"max":{"1min":1.446,"5min":1.5509999999999999,"15min":2.4980000000000002},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.198,"5min":1.1240000000000001,"15min":1.0669999999999999},"min":{"1min":0.65700000000000003,"5min":0.50600000000000001,"15min":0.432},"max":{"1min":1.579,"5min":1.7589999999999999,"15min":2.5760000000000001},"last":1.204}]},{"osd":3,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.145,"15min":1.125},"min":{"1min":0.85299999999999998,"5min":0.51500000000000001,"15min":0.35799999999999998},"max":{"1min":1.6890000000000001,"5min":1.7589999999999999,"15min":2.4580000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.3220000000000001,"15min":1.3240000000000001},"min":{"1min":1.1240000000000001,"5min":0.61299999999999999,"15min":0.48599999999999999},"max":{"1min":1.7769999999999999,"5min":1.968,"15min":2.9390000000000001},"last":1.014}]},{"osd":4,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.321,"15min":1.3420000000000001},"min":{"1min":1.05,"5min":0.73699999999999999,"15min":0.56100000000000005},"max":{"1min":1.7470000000000001,"5min":1.855,"15min":2.9750000000000001},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.3160000000000001,"15min":1.2969999999999999},"min":{"1min":1.024,"5min":0.81399999999999995,"15min":0.39900000000000002},"max":{"1min":1.9079999999999999,"5min":1.968,"15min":3.125},"last":0.91800000000000004}]},{"osd":5,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.359,"15min":1.381},"min":{"1min":1.0489999999999999,"5min":0.90200000000000002,"15min":0.51200000000000001},"max":{"1min":1.792,"5min":1.925,"15min":3.0830000000000002},"last":1.244},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.3340000000000001,"15min":1.427},"min":{"1min":1.0980000000000001,"5min":0.72799999999999998,"15min":0.52700000000000002},"max":{"1min":1.9370000000000001,"5min":2.0449999999999999,"15min":3.1899999999999999},"last":0.96399999999999997}]},{"osd":6,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.3899999999999999,"15min":1.462},"min":{"1min":1.2010000000000001,"5min":0.876,"15min":0.47999999999999998},"max":{"1min":2.0089999999999999,"5min":2.1030000000000002,"15min":3.391},"last":1.115},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4199999999999999,"15min":1.4139999999999999},"min":{"1min":1.018,"5min":0.71399999999999997,"15min":0.58399999999999996},"max":{"1min":1.823,"5min":2.2170000000000001,"15min":3.153},"last":1.359}]},{"osd":7,"last update":"Sat Mar 23 11:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.5169999999999999,"15min":1.5249999999999999},"min":{"1min":1.1619999999999999,"5min":0.85799999999999998,"15min":0.60599999999999998},"max":{"1min":1.8879999999999999,"5min":2.1400000000000001,"15min":3.5350000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.635,"5min":1.5,"15min":1.4990000000000001},"min":{"1min":1.2689999999999999,"5min":0.78300000000000003,"15min":0.45300000000000001},"max":{"1min":2.2440000000000002,"5min":2.2610000000000001,"15min":3.3650000000000002},"last":1.1519999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:31:29.593 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T11:31:29.593 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T11:31:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.869+0000 7fcaa7000640 1 Processor -- start 2024-03-23T11:31:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.869+0000 7fcaa7000640 1 -- start start 2024-03-23T11:31:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.871+0000 7fcaa7000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa00c5390 0x7fcaa00b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.871+0000 7fcaa7000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa0070230 con 0x7fcaa015de20 2024-03-23T11:31:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.871+0000 7fcaa7000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa006ce30 con 0x7fcaa00cd330 2024-03-23T11:31:29.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.871+0000 7fcaa7000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa006d110 con 0x7fcaa00c5390 2024-03-23T11:31:29.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.872+0000 7fca9fe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fcaa015de20 0x7fcaa015e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42916/0 (socket says 172.21.15.62:42916) 2024-03-23T11:31:29.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.872+0000 7fca9fe00640 1 -- 172.21.15.62:0/1088257319 learned_addr learned my addr 172.21.15.62:0/1088257319 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:29.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.872+0000 7fca9f400640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa00c5390 0x7fcaa00b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.872+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143984307 0 0) 0x7fcaa0070230 con 0x7fcaa015de20 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.872+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca84003520 con 0x7fcaa015de20 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1588865977 0 0) 0x7fca84003520 con 0x7fcaa015de20 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa00c5390 msgr2=0x7fcaa00b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa00c5390 0x7fcaa00b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 >> v1:172.21.15.120:6789/0 conn(0x7fcaa00cd330 legacy=0x7fcaa00706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa006d3f0 con 0x7fcaa015de20 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fca90001670 con 0x7fcaa015de20 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 >> v1:172.21.15.62:6789/0 conn(0x7fcaa015de20 legacy=0x7fcaa015e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa00c5390 0x7fca84003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca84003f90 0x7fca840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca84004d00 0x7fca84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa006d110 con 0x7fca84004d00 2024-03-23T11:31:29.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8400bb70 con 0x7fca84003f90 2024-03-23T11:31:29.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8400bd30 con 0x7fcaa00c5390 2024-03-23T11:31:29.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fcaa5000640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca84004d00 0x7fca84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9fe00640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa00c5390 0x7fca84003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.873+0000 7fca9f400640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca84003f90 0x7fca840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.874+0000 7fcaa5000640 1 -- 172.21.15.62:0/1088257319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa00c5390 msgr2=0x7fca84003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:29.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.874+0000 7fcaa5000640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa00c5390 0x7fca84003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.874+0000 7fcaa5000640 1 -- 172.21.15.62:0/1088257319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca84003f90 msgr2=0x7fca840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:29.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.874+0000 7fcaa5000640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca84003f90 0x7fca840047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.874+0000 7fcaa5000640 1 -- 172.21.15.62:0/1088257319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca84003520 con 0x7fca84004d00 2024-03-23T11:31:29.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.875+0000 7fcaa5000640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca84004d00 0x7fca84005180 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fca98001e50 tx=0x7fca840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4dc9ddb6c905ec8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:29.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.875+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca98003b70 con 0x7fca84004d00 2024-03-23T11:31:29.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.875+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca98003cd0 con 0x7fca84004d00 2024-03-23T11:31:29.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.875+0000 7fca9ea00640 1 -- 172.21.15.62:0/1088257319 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca98005310 con 0x7fca84004d00 2024-03-23T11:31:29.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.875+0000 7fcaa7000640 1 -- 172.21.15.62:0/1088257319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca84004d00 msgr2=0x7fca84005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.875+0000 7fcaa7000640 1 --2- 172.21.15.62:0/1088257319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca84004d00 0x7fca84005180 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fca98001e50 tx=0x7fca840057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.876+0000 7fcaa5000640 1 -- 172.21.15.62:0/1088257319 reap_dead start 2024-03-23T11:31:29.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.876+0000 7fcaa7000640 1 -- 172.21.15.62:0/1088257319 shutdown_connections 2024-03-23T11:31:29.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.876+0000 7fcaa7000640 1 -- 172.21.15.62:0/1088257319 >> 172.21.15.62:0/1088257319 conn(0x7fcaa0065a20 msgr2=0x7fcaa0065e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:29.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.876+0000 7fcaa7000640 1 -- 172.21.15.62:0/1088257319 shutdown_connections 2024-03-23T11:31:29.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.876+0000 7fcaa7000640 1 -- 172.21.15.62:0/1088257319 wait complete. 2024-03-23T11:31:29.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.877+0000 7fcaa7000640 1 Processor -- start 2024-03-23T11:31:29.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.878+0000 7fcaa7000640 1 -- start start 2024-03-23T11:31:29.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.878+0000 7fcaa7000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa014a250 0x7fcaa014cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.878+0000 7fcaa7000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa014a630 0x7fcaa014d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.878+0000 7fcaa7000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcaa00cba40 0x7fcaa00cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.878+0000 7fcaa7000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa014abe0 con 0x7fcaa014a250 2024-03-23T11:31:29.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.878+0000 7fcaa7000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa014aec0 con 0x7fcaa00cba40 2024-03-23T11:31:29.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.878+0000 7fcaa7000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa014b1a0 con 0x7fcaa014a630 2024-03-23T11:31:29.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.879+0000 7fca9fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa014a250 0x7fcaa014cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.879+0000 7fca9fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa014a250 0x7fcaa014cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37738/0 (socket says 172.21.15.62:37738) 2024-03-23T11:31:29.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.879+0000 7fca9fe00640 1 -- 172.21.15.62:0/1380590208 learned_addr learned my addr 172.21.15.62:0/1380590208 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:29.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.879+0000 7fca9fe00640 1 -- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa014a630 msgr2=0x7fcaa014d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.879+0000 7fca9f400640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa014a630 0x7fcaa014d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.879+0000 7fca9fe00640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa014a630 0x7fcaa014d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.879+0000 7fcaa5000640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcaa00cba40 0x7fcaa00cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.879+0000 7fca9fe00640 1 -- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcaa00cba40 msgr2=0x7fcaa00cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:29.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.879+0000 7fca9fe00640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcaa00cba40 0x7fcaa00cbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:29.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.879+0000 7fca9fe00640 1 -- 172.21.15.62:0/1380590208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa014b480 con 0x7fcaa014a250 2024-03-23T11:31:29.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.880+0000 7fca9fe00640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa014a250 0x7fcaa014cbc0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fca9000eaa0 tx=0x7fca9000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.880+0000 7fca97e00640 1 -- 172.21.15.62:0/1380590208 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9000c820 con 0x7fcaa014a250 2024-03-23T11:31:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.880+0000 7fca97e00640 1 -- 172.21.15.62:0/1380590208 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9000c9a0 con 0x7fcaa014a250 2024-03-23T11:31:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.880+0000 7fca97e00640 1 -- 172.21.15.62:0/1380590208 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9000cd50 con 0x7fcaa014a250 2024-03-23T11:31:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.880+0000 7fcaa7000640 1 -- 172.21.15.62:0/1380590208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa014b760 con 0x7fcaa014a250 2024-03-23T11:31:29.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.880+0000 7fcaa7000640 1 -- 172.21.15.62:0/1380590208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa00c6ac0 con 0x7fcaa014a250 2024-03-23T11:31:29.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.881+0000 7fcaa7000640 1 -- 172.21.15.62:0/1380590208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa014b980 con 0x7fcaa014a250 2024-03-23T11:31:29.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.882+0000 7fca97e00640 1 -- 172.21.15.62:0/1380590208 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fca9001d460 con 0x7fcaa014a250 2024-03-23T11:31:29.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.883+0000 7fca97e00640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fca8006a7d0 0x7fca8006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:29.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.883+0000 7fca97e00640 1 -- 172.21.15.62:0/1380590208 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca90088e10 con 0x7fcaa014a250 2024-03-23T11:31:29.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.884+0000 7fca9f400640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fca8006a7d0 0x7fca8006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:29.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.885+0000 7fca9f400640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fca8006a7d0 0x7fca8006cc90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcaa00664a0 tx=0x7fca8c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:29.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:29.887+0000 7fca97e00640 1 -- 172.21.15.62:0/1380590208 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fca9008c030 con 0x7fcaa014a250 2024-03-23T11:31:30.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.152+0000 7fcaa7000640 1 -- 172.21.15.62:0/1380590208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fcaa0066bb0 con 0x7fcaa014a250 2024-03-23T11:31:30.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.162+0000 7fca97e00640 1 -- 172.21.15.62:0/1380590208 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7fca90054220 con 0x7fcaa014a250 2024-03-23T11:31:30.152 INFO:teuthology.orchestra.run.smithi062.stderr:updated caps for client.0 2024-03-23T11:31:30.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.165+0000 7fcaa7000640 1 -- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fca8006a7d0 msgr2=0x7fca8006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:30.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.165+0000 7fcaa7000640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fca8006a7d0 0x7fca8006cc90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcaa00664a0 tx=0x7fca8c008040 comp rx=0 tx=0).stop 2024-03-23T11:31:30.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.165+0000 7fcaa7000640 1 -- 172.21.15.62:0/1380590208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa014a250 msgr2=0x7fcaa014cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:30.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.165+0000 7fcaa7000640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa014a250 0x7fcaa014cbc0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fca9000eaa0 tx=0x7fca9000ef60 comp rx=0 tx=0).stop 2024-03-23T11:31:30.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.165+0000 7fcaa7000640 1 -- 172.21.15.62:0/1380590208 shutdown_connections 2024-03-23T11:31:30.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.165+0000 7fcaa7000640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fca8006a7d0 0x7fca8006cc90 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.165+0000 7fcaa7000640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcaa00cba40 0x7fcaa00cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.165+0000 7fcaa7000640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcaa014a630 0x7fcaa014d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.165+0000 7fcaa7000640 1 --2- 172.21.15.62:0/1380590208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa014a250 0x7fcaa014cbc0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.165+0000 7fcaa7000640 1 -- 172.21.15.62:0/1380590208 >> 172.21.15.62:0/1380590208 conn(0x7fcaa0065a20 msgr2=0x7fcaa00c4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:30.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.166+0000 7fcaa7000640 1 -- 172.21.15.62:0/1380590208 shutdown_connections 2024-03-23T11:31:30.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.166+0000 7fcaa7000640 1 -- 172.21.15.62:0/1380590208 wait complete. 2024-03-23T11:31:30.175 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:31:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.404+0000 7f728ac00640 1 Processor -- start 2024-03-23T11:31:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.405+0000 7f728ac00640 1 -- start start 2024-03-23T11:31:30.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.406+0000 7f728ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7284143dd0 0x7f728415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.406+0000 7f728ac00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f72840b2f30 0x7f72840704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.406+0000 7f728ac00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7284070a30 con 0x7f7284070090 2024-03-23T11:31:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.406+0000 7f728ac00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72840cd140 con 0x7f72840b2f30 2024-03-23T11:31:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.406+0000 7f728ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72840cd3e0 con 0x7f7284143dd0 2024-03-23T11:31:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.407+0000 7f7288e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7284070090 0x7f72840b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42928/0 (socket says 172.21.15.62:42928) 2024-03-23T11:31:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.407+0000 7f7288e00640 1 -- 172.21.15.62:0/580307083 learned_addr learned my addr 172.21.15.62:0/580307083 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:30.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.407+0000 7f7289800640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7284143dd0 0x7f728415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:30.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.407+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062963114 0 0) 0x7f7284070a30 con 0x7f7284070090 2024-03-23T11:31:30.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.407+0000 7f728a200640 1 --2- 172.21.15.62:0/580307083 >> v2:172.21.15.120:3300/0 conn(0x7f72840b2f30 0x7f72840704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:30.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.407+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f726c003520 con 0x7f7284070090 2024-03-23T11:31:30.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1887344278 0 0) 0x7f726c003520 con 0x7f7284070090 2024-03-23T11:31:30.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7284143dd0 msgr2=0x7f728415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7284143dd0 0x7f728415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 >> v2:172.21.15.120:3300/0 conn(0x7f72840b2f30 msgr2=0x7f72840704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 --2- 172.21.15.62:0/580307083 >> v2:172.21.15.120:3300/0 conn(0x7f72840b2f30 0x7f72840704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72840cd6c0 con 0x7f7284070090 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7270001670 con 0x7f7284070090 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 >> v1:172.21.15.62:6789/0 conn(0x7f7284070090 legacy=0x7f72840b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7284143dd0 0x7f726c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f726c003ef0 0x7f726c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f726c004d00 0x7f726c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72840cd140 con 0x7f726c003ef0 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72840cd3e0 con 0x7f726c004d00 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726c00fd90 con 0x7f7284143dd0 2024-03-23T11:31:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.408+0000 7f7288e00640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f726c003ef0 0x7f726c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.409+0000 7f7289800640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7284143dd0 0x7f726c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.409+0000 7f728a200640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f726c004d00 0x7f726c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.409+0000 7f7288e00640 1 -- 172.21.15.62:0/580307083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7284143dd0 msgr2=0x7f726c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.409+0000 7f7288e00640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7284143dd0 0x7f726c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.409+0000 7f7288e00640 1 -- 172.21.15.62:0/580307083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f726c004d00 msgr2=0x7f726c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.409+0000 7f7288e00640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f726c004d00 0x7f726c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.409+0000 7f7288e00640 1 -- 172.21.15.62:0/580307083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c003520 con 0x7f726c003ef0 2024-03-23T11:31:30.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.410+0000 7f7288e00640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f726c003ef0 0x7f726c0047c0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f727000c240 tx=0x7f726c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986842f26867d979 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:30.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.410+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727002cb00 con 0x7f726c003ef0 2024-03-23T11:31:30.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.410+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727002cc60 con 0x7f726c003ef0 2024-03-23T11:31:30.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.410+0000 7f7283e00640 1 -- 172.21.15.62:0/580307083 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270005af0 con 0x7f726c003ef0 2024-03-23T11:31:30.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.411+0000 7f728ac00640 1 -- 172.21.15.62:0/580307083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f726c003ef0 msgr2=0x7f726c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:30.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.411+0000 7f728ac00640 1 --2- 172.21.15.62:0/580307083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f726c003ef0 0x7f726c0047c0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f727000c240 tx=0x7f726c0057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.411+0000 7f728a200640 1 -- 172.21.15.62:0/580307083 reap_dead start 2024-03-23T11:31:30.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.411+0000 7f728ac00640 1 -- 172.21.15.62:0/580307083 shutdown_connections 2024-03-23T11:31:30.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.411+0000 7f728ac00640 1 -- 172.21.15.62:0/580307083 >> 172.21.15.62:0/580307083 conn(0x7f7284066050 msgr2=0x7f7284069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:30.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.411+0000 7f728ac00640 1 -- 172.21.15.62:0/580307083 shutdown_connections 2024-03-23T11:31:30.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.411+0000 7f728ac00640 1 -- 172.21.15.62:0/580307083 wait complete. 2024-03-23T11:31:30.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.413+0000 7f728ac00640 1 Processor -- start 2024-03-23T11:31:30.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.413+0000 7f728ac00640 1 -- start start 2024-03-23T11:31:30.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f728ac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f726c003ef0 0x7f728413e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:30.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f728ac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72841ee520 0x7f72840a8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:30.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f728ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f728415b320 0x7f728415b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:30.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f728ac00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728415ae90 con 0x7f72841ee520 2024-03-23T11:31:30.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f728ac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72840a8a50 con 0x7f726c003ef0 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f728ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72840a8d30 con 0x7f728415b320 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f7288e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72841ee520 0x7f72840a8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f7288e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72841ee520 0x7f72840a8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37752/0 (socket says 172.21.15.62:37752) 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f7288e00640 1 -- 172.21.15.62:0/1908346218 learned_addr learned my addr 172.21.15.62:0/1908346218 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f728a200640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f728415b320 0x7f728415b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f7289800640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f726c003ef0 0x7f728413e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f7288e00640 1 -- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f728415b320 msgr2=0x7f728415b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f7288e00640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f728415b320 0x7f728415b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.414+0000 7f7288e00640 1 -- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f726c003ef0 msgr2=0x7f728413e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.415+0000 7f7288e00640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f726c003ef0 0x7f728413e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.415+0000 7f7288e00640 1 -- 172.21.15.62:0/1908346218 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728415a840 con 0x7f72841ee520 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.415+0000 7f7289800640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f726c003ef0 0x7f728413e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.415+0000 7f728a200640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f728415b320 0x7f728415b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.415+0000 7f7288e00640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72841ee520 0x7f72840a8150 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f727002c500 tx=0x7f7270001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.415+0000 7f7281600640 1 -- 172.21.15.62:0/1908346218 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270039070 con 0x7f72841ee520 2024-03-23T11:31:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.415+0000 7f7281600640 1 -- 172.21.15.62:0/1908346218 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7270005c80 con 0x7f72841ee520 2024-03-23T11:31:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.415+0000 7f7281600640 1 -- 172.21.15.62:0/1908346218 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727003c040 con 0x7f72841ee520 2024-03-23T11:31:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.415+0000 7f728ac00640 1 -- 172.21.15.62:0/1908346218 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728415ab20 con 0x7f72841ee520 2024-03-23T11:31:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.416+0000 7f728ac00640 1 -- 172.21.15.62:0/1908346218 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72840a8690 con 0x7f72841ee520 2024-03-23T11:31:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.417+0000 7f728ac00640 1 -- 172.21.15.62:0/1908346218 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728415b000 con 0x7f72841ee520 2024-03-23T11:31:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.420+0000 7f7281600640 1 -- 172.21.15.62:0/1908346218 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7270032070 con 0x7f72841ee520 2024-03-23T11:31:30.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.421+0000 7f7281600640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f726406a7d0 0x7f726406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:30.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.421+0000 7f7281600640 1 -- 172.21.15.62:0/1908346218 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72700aa1b0 con 0x7f72841ee520 2024-03-23T11:31:30.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.422+0000 7f7289800640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f726406a7d0 0x7f726406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:30.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.423+0000 7f7289800640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f726406a7d0 0x7f726406cc90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7278006ce0 tx=0x7f7278008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.424+0000 7f7281600640 1 -- 172.21.15.62:0/1908346218 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72700757d0 con 0x7f72841ee520 2024-03-23T11:31:30.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.663+0000 7f728ac00640 1 -- 172.21.15.62:0/1908346218 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7284066bb0 con 0x7f72841ee520 2024-03-23T11:31:30.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.664+0000 7f7281600640 1 -- 172.21.15.62:0/1908346218 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f7270046090 con 0x7f72841ee520 2024-03-23T11:31:30.654 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:30.654 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":94,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/4129977589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4129977589},{"type":"v1","addr":"172.21.15.62:6835","nonce":4129977589}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:31:27.575868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9006},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9006":{"gid":9006,"name":"b","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6837/2585024035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":2585024035},{"type":"v1","addr":"172.21.15.120:6837","nonce":2585024035}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:31:30.654 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 94 2024-03-23T11:31:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.666+0000 7f728ac00640 1 -- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f726406a7d0 msgr2=0x7f726406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.666+0000 7f728ac00640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f726406a7d0 0x7f726406cc90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7278006ce0 tx=0x7f7278008040 comp rx=0 tx=0).stop 2024-03-23T11:31:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.666+0000 7f728ac00640 1 -- 172.21.15.62:0/1908346218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72841ee520 msgr2=0x7f72840a8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.666+0000 7f728ac00640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72841ee520 0x7f72840a8150 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f727002c500 tx=0x7f7270001a00 comp rx=0 tx=0).stop 2024-03-23T11:31:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.666+0000 7f728ac00640 1 -- 172.21.15.62:0/1908346218 shutdown_connections 2024-03-23T11:31:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.666+0000 7f728ac00640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f726406a7d0 0x7f726406cc90 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.666+0000 7f728ac00640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72841ee520 0x7f72840a8150 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.666+0000 7f728ac00640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f728415b320 0x7f728415b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.666+0000 7f728ac00640 1 --2- 172.21.15.62:0/1908346218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f726c003ef0 0x7f728413e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.667+0000 7f728ac00640 1 -- 172.21.15.62:0/1908346218 >> 172.21.15.62:0/1908346218 conn(0x7f7284066050 msgr2=0x7f72840ae6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.667+0000 7f728ac00640 1 -- 172.21.15.62:0/1908346218 shutdown_connections 2024-03-23T11:31:30.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:30.667+0000 7f728ac00640 1 -- 172.21.15.62:0/1908346218 wait complete. 2024-03-23T11:31:31.076 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:31:24.044342+0000', 'modified': '2024-03-23T11:31:27.575868+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9006}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9006': {'gid': 9006, 'name': 'b', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6837/2585024035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6835', 'nonce': 2585024035}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 2585024035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:31:31.077 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:31:31.077 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-23T11:31:31.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.323+0000 7f3b8f200640 1 Processor -- start 2024-03-23T11:31:31.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.323+0000 7f3b8f200640 1 -- start start 2024-03-23T11:31:31.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.324+0000 7f3b8f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b90069c60 0x7f3b9006a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.325+0000 7f3b8f200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9015bb10 con 0x7f3b900caff0 2024-03-23T11:31:31.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.325+0000 7f3b8f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9006a580 con 0x7f3b90143450 2024-03-23T11:31:31.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.325+0000 7f3b8f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b900ce1a0 con 0x7f3b90069c60 2024-03-23T11:31:31.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.325+0000 7f3b87e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3b900caff0 0x7f3b900cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42944/0 (socket says 172.21.15.62:42944) 2024-03-23T11:31:31.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.325+0000 7f3b87e00640 1 -- 172.21.15.62:0/2168969275 learned_addr learned my addr 172.21.15.62:0/2168969275 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:31.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.325+0000 7f3b8de00640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b90069c60 0x7f3b9006a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.325+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912619738 0 0) 0x7f3b9015bb10 con 0x7f3b900caff0 2024-03-23T11:31:31.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b74003520 con 0x7f3b900caff0 2024-03-23T11:31:31.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 863589580 0 0) 0x7f3b74003520 con 0x7f3b900caff0 2024-03-23T11:31:31.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b90069c60 msgr2=0x7f3b9006a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b90069c60 0x7f3b9006a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 >> v1:172.21.15.120:6789/0 conn(0x7f3b90143450 legacy=0x7f3b901438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9015ad50 con 0x7f3b900caff0 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f3b78001670 con 0x7f3b900caff0 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 >> v1:172.21.15.62:6789/0 conn(0x7f3b900caff0 legacy=0x7f3b900cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b90069c60 0x7f3b740039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b74003f20 0x7f3b740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b74004d00 0x7f3b74005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b900ce1a0 con 0x7f3b74004d00 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7400bb10 con 0x7f3b74003f20 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7400bcd0 con 0x7f3b90069c60 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.326+0000 7f3b8e800640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b74004d00 0x7f3b74005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.327+0000 7f3b87e00640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b74003f20 0x7f3b740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.327+0000 7f3b8de00640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b90069c60 0x7f3b740039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.327+0000 7f3b8e800640 1 -- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b90069c60 msgr2=0x7f3b740039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.327+0000 7f3b8e800640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b90069c60 0x7f3b740039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.327+0000 7f3b8e800640 1 -- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b74003f20 msgr2=0x7f3b740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.327+0000 7f3b8e800640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b74003f20 0x7f3b740047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.327+0000 7f3b8e800640 1 -- 172.21.15.62:0/2168969275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b74003520 con 0x7f3b74004d00 2024-03-23T11:31:31.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.327+0000 7f3b8de00640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b90069c60 0x7f3b740039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.328+0000 7f3b8e800640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b74004d00 0x7f3b74005180 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f3b88001e50 tx=0x7f3b740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f424fa937f08e47 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.328+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88003b70 con 0x7f3b74004d00 2024-03-23T11:31:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.328+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b88003cd0 con 0x7f3b74004d00 2024-03-23T11:31:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.328+0000 7f3b8d400640 1 -- 172.21.15.62:0/2168969275 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88005310 con 0x7f3b74004d00 2024-03-23T11:31:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.329+0000 7f3b8f200640 1 -- 172.21.15.62:0/2168969275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b74004d00 msgr2=0x7f3b74005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.329+0000 7f3b8f200640 1 --2- 172.21.15.62:0/2168969275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b74004d00 0x7f3b74005180 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f3b88001e50 tx=0x7f3b740057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.329+0000 7f3b8e800640 1 -- 172.21.15.62:0/2168969275 reap_dead start 2024-03-23T11:31:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.329+0000 7f3b8f200640 1 -- 172.21.15.62:0/2168969275 shutdown_connections 2024-03-23T11:31:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.329+0000 7f3b8f200640 1 -- 172.21.15.62:0/2168969275 >> 172.21.15.62:0/2168969275 conn(0x7f3b900b0090 msgr2=0x7f3b900b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:31.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.329+0000 7f3b8f200640 1 -- 172.21.15.62:0/2168969275 shutdown_connections 2024-03-23T11:31:31.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.329+0000 7f3b8f200640 1 -- 172.21.15.62:0/2168969275 wait complete. 2024-03-23T11:31:31.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.331+0000 7f3b8f200640 1 Processor -- start 2024-03-23T11:31:31.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.331+0000 7f3b8f200640 1 -- start start 2024-03-23T11:31:31.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.332+0000 7f3b8f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b74004d00 0x7f3b900ac250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.332+0000 7f3b8f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b900b22c0 0x7f3b900ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.332+0000 7f3b8f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b901f55f0 0x7f3b901f5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.332+0000 7f3b8f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b900cbcb0 con 0x7f3b900b22c0 2024-03-23T11:31:31.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.332+0000 7f3b8f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b901f5fb0 con 0x7f3b901f55f0 2024-03-23T11:31:31.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.332+0000 7f3b8f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b900c33a0 con 0x7f3b74004d00 2024-03-23T11:31:31.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.332+0000 7f3b87e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b900b22c0 0x7f3b900ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.332+0000 7f3b87e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b900b22c0 0x7f3b900ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37774/0 (socket says 172.21.15.62:37774) 2024-03-23T11:31:31.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.332+0000 7f3b87e00640 1 -- 172.21.15.62:0/3171880868 learned_addr learned my addr 172.21.15.62:0/3171880868 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:31.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.332+0000 7f3b87e00640 1 -- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b74004d00 msgr2=0x7f3b900ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.332+0000 7f3b87e00640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b74004d00 0x7f3b900ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.333+0000 7f3b87e00640 1 -- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b901f55f0 msgr2=0x7f3b901f5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.333+0000 7f3b8e800640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b901f55f0 0x7f3b901f5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.333+0000 7f3b87e00640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b901f55f0 0x7f3b901f5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.333+0000 7f3b87e00640 1 -- 172.21.15.62:0/3171880868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b900ca5a0 con 0x7f3b900b22c0 2024-03-23T11:31:31.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.333+0000 7f3b8e800640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b901f55f0 0x7f3b901f5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:31.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.333+0000 7f3b87e00640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b900b22c0 0x7f3b900ca060 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f3b7800e820 tx=0x7f3b7800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:31.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.333+0000 7f3b86000640 1 -- 172.21.15.62:0/3171880868 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7800c870 con 0x7f3b900b22c0 2024-03-23T11:31:31.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.333+0000 7f3b86000640 1 -- 172.21.15.62:0/3171880868 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7800c9d0 con 0x7f3b900b22c0 2024-03-23T11:31:31.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.333+0000 7f3b86000640 1 -- 172.21.15.62:0/3171880868 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7800cd60 con 0x7f3b900b22c0 2024-03-23T11:31:31.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.333+0000 7f3b8f200640 1 -- 172.21.15.62:0/3171880868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b900ac830 con 0x7f3b900b22c0 2024-03-23T11:31:31.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.333+0000 7f3b8f200640 1 -- 172.21.15.62:0/3171880868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b900acdf0 con 0x7f3b900b22c0 2024-03-23T11:31:31.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.335+0000 7f3b86000640 1 -- 172.21.15.62:0/3171880868 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3b78013420 con 0x7f3b900b22c0 2024-03-23T11:31:31.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.336+0000 7f3b86000640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b6c06a7d0 0x7f3b6c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.336+0000 7f3b86000640 1 -- 172.21.15.62:0/3171880868 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b78088b30 con 0x7f3b900b22c0 2024-03-23T11:31:31.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.337+0000 7f3b8de00640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b6c06a7d0 0x7f3b6c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.337+0000 7f3b97874740 1 Processor -- start 2024-03-23T11:31:31.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.338+0000 7f3b97874740 1 -- start start 2024-03-23T11:31:31.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.338+0000 7f3b8de00640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b6c06a7d0 0x7f3b6c06cc90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3b800099e0 tx=0x7f3b80008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:31.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.338+0000 7f3b97874740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36db520 0x559bd3692b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b97874740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 0x559bd3693090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b97874740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd368da40 0x559bd36bd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b97874740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bd35fe110 con 0x559bd368da40 2024-03-23T11:31:31.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b97874740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bd35fb8a0 con 0x559bd36915a0 2024-03-23T11:31:31.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b97874740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559bd35f7190 con 0x559bd36db520 2024-03-23T11:31:31.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b8e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd368da40 0x559bd36bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b8e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd368da40 0x559bd36bd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37784/0 (socket says 172.21.15.62:37784) 2024-03-23T11:31:31.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b8e800640 1 -- 172.21.15.62:0/2242069745 learned_addr learned my addr 172.21.15.62:0/2242069745 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:31.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b87e00640 1 --2- 172.21.15.62:0/2242069745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 0x559bd3693090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b8de00640 1 --2- 172.21.15.62:0/2242069745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36db520 0x559bd3692b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b8e800640 1 -- 172.21.15.62:0/2242069745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36db520 msgr2=0x559bd3692b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b8e800640 1 --2- 172.21.15.62:0/2242069745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36db520 0x559bd3692b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b8e800640 1 -- 172.21.15.62:0/2242069745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 msgr2=0x559bd3693090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b8e800640 1 --2- 172.21.15.62:0/2242069745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 0x559bd3693090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.339+0000 7f3b8e800640 1 -- 172.21.15.62:0/2242069745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559bd35c2230 con 0x559bd368da40 2024-03-23T11:31:31.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.340+0000 7f3b8e800640 1 --2- 172.21.15.62:0/2242069745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd368da40 0x559bd36bd830 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f3b88001f70 tx=0x7f3b88030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd85cc0840211d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:31.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.340+0000 7f3b7ea00640 1 -- 172.21.15.62:0/2242069745 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88003b70 con 0x559bd368da40 2024-03-23T11:31:31.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.340+0000 7f3b7ea00640 1 -- 172.21.15.62:0/2242069745 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b88003e40 con 0x559bd368da40 2024-03-23T11:31:31.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.340+0000 7f3b7ea00640 1 -- 172.21.15.62:0/2242069745 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88006d50 con 0x559bd368da40 2024-03-23T11:31:31.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.341+0000 7f3b97874740 1 -- 172.21.15.62:0/2242069745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd368da40 msgr2=0x559bd36bd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.341+0000 7f3b97874740 1 --2- 172.21.15.62:0/2242069745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd368da40 0x559bd36bd830 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f3b88001f70 tx=0x7f3b88030000 comp rx=0 tx=0).stop 2024-03-23T11:31:31.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.341+0000 7f3b97874740 1 -- 172.21.15.62:0/2242069745 shutdown_connections 2024-03-23T11:31:31.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.341+0000 7f3b97874740 1 --2- 172.21.15.62:0/2242069745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd368da40 0x559bd36bd830 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.341+0000 7f3b97874740 1 --2- 172.21.15.62:0/2242069745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 0x559bd3693090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.341+0000 7f3b97874740 1 --2- 172.21.15.62:0/2242069745 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36db520 0x559bd3692b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.341+0000 7f3b97874740 1 -- 172.21.15.62:0/2242069745 >> 172.21.15.62:0/2242069745 conn(0x559bd368fcb0 msgr2=0x559bd36d8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:31.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.341+0000 7f3b97874740 1 -- 172.21.15.62:0/2242069745 shutdown_connections 2024-03-23T11:31:31.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.341+0000 7f3b97874740 1 -- 172.21.15.62:0/2242069745 wait complete. 2024-03-23T11:31:31.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.342+0000 7f3b97874740 1 -- start start 2024-03-23T11:31:31.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.342+0000 7f3b97874740 1 Processor -- start 2024-03-23T11:31:31.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.343+0000 7f3b97874740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 0x559bd36e08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.343+0000 7f3b97874740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36b4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.343+0000 7f3b97874740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36bb710 0x559bd36b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.343+0000 7f3b97874740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bd35988c0 con 0x559bd36db520 2024-03-23T11:31:31.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.343+0000 7f3b97874740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bd3598560 con 0x559bd36915a0 2024-03-23T11:31:31.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.343+0000 7f3b97874740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559bd335ef20 con 0x559bd36bb710 2024-03-23T11:31:31.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.343+0000 7f3b97874740 10 client.0 _ll_register_callbacks cb 0x559bd35f0800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:31:31.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.343+0000 7f3b87e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.343+0000 7f3b87e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37786/0 (socket says 172.21.15.62:37786) 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.343+0000 7f3b87e00640 1 -- 172.21.15.62:0/1643305467 learned_addr learned my addr 172.21.15.62:0/1643305467 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.343+0000 7f3b8de00640 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 0x559bd36e08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b8e800640 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36bb710 0x559bd36b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b87e00640 1 -- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36bb710 msgr2=0x559bd36b8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b87e00640 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36bb710 0x559bd36b8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b87e00640 1 -- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 msgr2=0x559bd36e08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b87e00640 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 0x559bd36e08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b87e00640 1 -- 172.21.15.62:0/1643305467 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b74003520 con 0x559bd36db520 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b87e00640 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36b4270 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f3b78086fc0 tx=0x7f3b78057b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09df270a0bb0c7f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b73400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b73400640 1 -- 172.21.15.62:0/1643305467 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7805ff60 con 0x559bd36db520 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b73400640 1 -- 172.21.15.62:0/1643305467 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b78057060 con 0x559bd36db520 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b97874740 1 -- 172.21.15.62:0/1643305467 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559bd3621280 con 0x559bd36db520 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b73400640 1 -- 172.21.15.62:0/1643305467 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b78057220 con 0x559bd36db520 2024-03-23T11:31:31.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b97874740 10 client.9138 fetch_fsmap learned FSMap version 94 2024-03-23T11:31:31.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.344+0000 7f3b97874740 1 -- 172.21.15.62:0/1643305467 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x559bd35f5810 con 0x559bd36db520 2024-03-23T11:31:31.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.345+0000 7f3b73400640 1 -- 172.21.15.62:0/1643305467 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559bd3621280 con 0x559bd36db520 2024-03-23T11:31:31.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.345+0000 7f3b73400640 1 -- 172.21.15.62:0/1643305467 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 94) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3b7805fad0 con 0x559bd36db520 2024-03-23T11:31:31.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.345+0000 7f3b97874740 10 client.9138 fetch_fsmap finished waiting for FSMap version 94 2024-03-23T11:31:31.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.345+0000 7f3b97874740 10 client.9138 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:31:31.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.345+0000 7f3b97874740 20 client.9138 populate_metadata read hostname 'smithi062' 2024-03-23T11:31:31.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.345+0000 7f3b97874740 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x559bd36ddb10 0x559bd36de4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.345+0000 7f3b97874740 4 client.9138 mds_command: new command op to 9006 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:31:31.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.345+0000 7f3b97874740 1 -- 172.21.15.62:0/1643305467 --> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559bd35e99d0 con 0x559bd36ddb10 2024-03-23T11:31:31.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.346+0000 7f3b8de00640 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x559bd36ddb10 0x559bd36de4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.347+0000 7f3b8de00640 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x559bd36ddb10 0x559bd36de4c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9e6bd420f8eeb14 server_cookie=eec089e28a3a8a02 in_seq=0 out_seq=0 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.347+0000 7f3b73400640 10 client.9138 ms_handle_connect on v2:172.21.15.120:6835/2585024035 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.348+0000 7f3b73400640 1 -- 172.21.15.62:0/1643305467 <== mds.0 v2:172.21.15.120:6835/2585024035 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x559bd35e99d0 con 0x559bd36ddb10 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.348+0000 7f3b73400640 10 client.9138 handle_command_reply: tid=0 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 1 client.9138 shutdown 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 1 -- 172.21.15.62:0/1643305467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 msgr2=0x559bd36b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36b4270 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f3b78086fc0 tx=0x7f3b78057b70 comp rx=0 tx=0).stop 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 1 -- 172.21.15.62:0/1643305467 shutdown_connections 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x559bd36ddb10 0x559bd36de4c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36b4270 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36bb710 0x559bd36b8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 1 --2- 172.21.15.62:0/1643305467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 0x559bd36e08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b73400640 0 client.9138 ms_handle_reset on v2:172.21.15.120:6835/2585024035 2024-03-23T11:31:31.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 1 -- 172.21.15.62:0/1643305467 >> 172.21.15.62:0/1643305467 conn(0x559bd368da40 msgr2=0x559bd36d9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:31.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 1 -- 172.21.15.62:0/1643305467 shutdown_connections 2024-03-23T11:31:31.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 1 -- 172.21.15.62:0/1643305467 wait complete. 2024-03-23T11:31:31.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.349+0000 7f3b97874740 20 client.9138 trim_cache size 0 max 16384 2024-03-23T11:31:31.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.367+0000 7f3b97874740 1 Processor -- start 2024-03-23T11:31:31.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.367+0000 7f3b97874740 1 -- start start 2024-03-23T11:31:31.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.368+0000 7f3b97874740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 0x559bd36b4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.368+0000 7f3b97874740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36bb710 0x559bd368e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.368+0000 7f3b97874740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36db520 0x559bd368cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.368+0000 7f3b97874740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bd36b94a0 con 0x559bd36bb710 2024-03-23T11:31:31.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.368+0000 7f3b97874740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bd36b4ea0 con 0x559bd36915a0 2024-03-23T11:31:31.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.368+0000 7f3b97874740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559bd36ddef0 con 0x559bd36db520 2024-03-23T11:31:31.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.368+0000 7f3b87e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36bb710 0x559bd368e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.368+0000 7f3b87e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36bb710 0x559bd368e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37792/0 (socket says 172.21.15.62:37792) 2024-03-23T11:31:31.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.368+0000 7f3b87e00640 1 -- 172.21.15.62:0/1123809783 learned_addr learned my addr 172.21.15.62:0/1123809783 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:31.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.369+0000 7f3b87e00640 1 -- 172.21.15.62:0/1123809783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36db520 msgr2=0x559bd368cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.369+0000 7f3b87e00640 1 --2- 172.21.15.62:0/1123809783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36db520 0x559bd368cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.369+0000 7f3b87e00640 1 -- 172.21.15.62:0/1123809783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 msgr2=0x559bd36b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.369+0000 7f3b87e00640 1 --2- 172.21.15.62:0/1123809783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 0x559bd36b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.369+0000 7f3b87e00640 1 -- 172.21.15.62:0/1123809783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559bd3597ad0 con 0x559bd36bb710 2024-03-23T11:31:31.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.369+0000 7f3b87e00640 1 --2- 172.21.15.62:0/1123809783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36bb710 0x559bd368e110 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f3b78001a60 tx=0x7f3b7805b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24c33cb088a9dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:31.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.369+0000 7f3b7f400640 1 -- 172.21.15.62:0/1123809783 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7805ff60 con 0x559bd36bb710 2024-03-23T11:31:31.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.369+0000 7f3b7f400640 1 -- 172.21.15.62:0/1123809783 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7805fad0 con 0x559bd36bb710 2024-03-23T11:31:31.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.369+0000 7f3b7f400640 1 -- 172.21.15.62:0/1123809783 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b780572a0 con 0x559bd36bb710 2024-03-23T11:31:31.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.370+0000 7f3b97874740 1 -- 172.21.15.62:0/1123809783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36bb710 msgr2=0x559bd368e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.370+0000 7f3b97874740 1 --2- 172.21.15.62:0/1123809783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36bb710 0x559bd368e110 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f3b78001a60 tx=0x7f3b7805b000 comp rx=0 tx=0).stop 2024-03-23T11:31:31.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.370+0000 7f3b97874740 1 -- 172.21.15.62:0/1123809783 shutdown_connections 2024-03-23T11:31:31.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.370+0000 7f3b97874740 1 --2- 172.21.15.62:0/1123809783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36bb710 0x559bd368e110 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.370+0000 7f3b97874740 1 --2- 172.21.15.62:0/1123809783 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36db520 0x559bd368cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.370+0000 7f3b97874740 1 --2- 172.21.15.62:0/1123809783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36915a0 0x559bd36b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.370+0000 7f3b97874740 1 -- 172.21.15.62:0/1123809783 >> 172.21.15.62:0/1123809783 conn(0x559bd368da40 msgr2=0x559bd36b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:31.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.370+0000 7f3b97874740 1 -- 172.21.15.62:0/1123809783 shutdown_connections 2024-03-23T11:31:31.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.370+0000 7f3b97874740 1 -- 172.21.15.62:0/1123809783 wait complete. 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.371+0000 7f3b97874740 1 -- start start 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.371+0000 7f3b97874740 1 Processor -- start 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.372+0000 7f3b97874740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36915a0 0x559bd36b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.372+0000 7f3b97874740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36bb710 0x559bd36b6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.372+0000 7f3b97874740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36de700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.372+0000 7f3b97874740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bd36bf6c0 con 0x559bd36db520 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.372+0000 7f3b97874740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bd3689550 con 0x559bd36bb710 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.372+0000 7f3b97874740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559bd36dc7e0 con 0x559bd36915a0 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.372+0000 7f3b97874740 10 client.0 _ll_register_callbacks cb 0x7f3b7800cec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.372+0000 7f3b8e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36de700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.372+0000 7f3b8e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36de700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37794/0 (socket says 172.21.15.62:37794) 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.372+0000 7f3b8e800640 1 -- 172.21.15.62:0/1413977819 learned_addr learned my addr 172.21.15.62:0/1413977819 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b8de00640 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36915a0 0x559bd36b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b87e00640 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36bb710 0x559bd36b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b8e800640 1 -- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36915a0 msgr2=0x559bd36b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b8e800640 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36915a0 0x559bd36b5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b8e800640 1 -- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36bb710 msgr2=0x559bd36b6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b8e800640 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36bb710 0x559bd36b6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b8e800640 1 -- 172.21.15.62:0/1413977819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b74003520 con 0x559bd36db520 2024-03-23T11:31:31.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b8e800640 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36de700 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f3b88005040 tx=0x7f3b880014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a8c6e1ac909027 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:31.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b73400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:31:31.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b73400640 1 -- 172.21.15.62:0/1413977819 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88005790 con 0x559bd36db520 2024-03-23T11:31:31.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b73400640 1 -- 172.21.15.62:0/1413977819 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b8802f460 con 0x559bd36db520 2024-03-23T11:31:31.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.373+0000 7f3b97874740 1 -- 172.21.15.62:0/1413977819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b88003e40 con 0x559bd36db520 2024-03-23T11:31:31.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.374+0000 7f3b73400640 1 -- 172.21.15.62:0/1413977819 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b88007db0 con 0x559bd36db520 2024-03-23T11:31:31.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.374+0000 7f3b97874740 10 client.9144 fetch_fsmap learned FSMap version 94 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.374+0000 7f3b97874740 1 -- 172.21.15.62:0/1413977819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x559bd35bcfe0 con 0x559bd36db520 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.374+0000 7f3b73400640 1 -- 172.21.15.62:0/1413977819 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b88003e40 con 0x559bd36db520 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.374+0000 7f3b73400640 1 -- 172.21.15.62:0/1413977819 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 94) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3b8804b020 con 0x559bd36db520 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.374+0000 7f3b97874740 10 client.9144 fetch_fsmap finished waiting for FSMap version 94 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.374+0000 7f3b97874740 10 client.9144 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.374+0000 7f3b97874740 20 client.9144 populate_metadata read hostname 'smithi062' 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.374+0000 7f3b97874740 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x559bd373e660 0x559bd36dec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.374+0000 7f3b97874740 4 client.9144 mds_command: new command op to 9006 tid=0[{"prefix": "status"}] 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.374+0000 7f3b97874740 1 -- 172.21.15.62:0/1413977819 --> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559bd32eb3c0 con 0x559bd373e660 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.375+0000 7f3b8de00640 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x559bd373e660 0x559bd36dec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b8de00640 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x559bd373e660 0x559bd36dec40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce28e1410318c3bf server_cookie=71bc6c3c4bdc7f4 in_seq=0 out_seq=0 2024-03-23T11:31:31.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b73400640 10 client.9144 ms_handle_connect on v2:172.21.15.120:6835/2585024035 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b73400640 1 -- 172.21.15.62:0/1413977819 <== mds.0 v2:172.21.15.120:6835/2585024035 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x559bd32eb3c0 con 0x559bd373e660 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b73400640 10 client.9144 handle_command_reply: tid=0 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b97874740 1 client.9144 shutdown 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b97874740 1 -- 172.21.15.62:0/1413977819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 msgr2=0x559bd36de700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b97874740 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36de700 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f3b88005040 tx=0x7f3b880014a0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b97874740 1 -- 172.21.15.62:0/1413977819 shutdown_connections 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b97874740 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x559bd373e660 0x559bd36dec40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b97874740 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559bd36db520 0x559bd36de700 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b97874740 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559bd36bb710 0x559bd36b6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b97874740 1 --2- 172.21.15.62:0/1413977819 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x559bd36915a0 0x559bd36b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b73400640 0 client.9144 ms_handle_reset on v2:172.21.15.120:6835/2585024035 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.376+0000 7f3b97874740 1 -- 172.21.15.62:0/1413977819 >> 172.21.15.62:0/1413977819 conn(0x559bd368da40 msgr2=0x559bd36b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:31.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b97874740 1 -- 172.21.15.62:0/1413977819 shutdown_connections 2024-03-23T11:31:31.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b97874740 1 -- 172.21.15.62:0/1413977819 wait complete. 2024-03-23T11:31:31.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b97874740 20 client.9144 trim_cache size 0 max 16384 2024-03-23T11:31:31.369 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:31:31.369 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:31:31.369 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:31:31.369 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 9006, 2024-03-23T11:31:31.369 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:31:31.369 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:31:31.369 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:31:31.369 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 7.327799422, 2024-03-23T11:31:31.370 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 94, 2024-03-23T11:31:31.370 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 118, 2024-03-23T11:31:31.370 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 118, 2024-03-23T11:31:31.370 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 12.663522017 2024-03-23T11:31:31.370 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:31:31.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b8f200640 1 -- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b6c06a7d0 msgr2=0x7f3b6c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b8f200640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b6c06a7d0 0x7f3b6c06cc90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3b800099e0 tx=0x7f3b80008040 comp rx=0 tx=0).stop 2024-03-23T11:31:31.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b8f200640 1 -- 172.21.15.62:0/3171880868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b900b22c0 msgr2=0x7f3b900ca060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b8f200640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b900b22c0 0x7f3b900ca060 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f3b7800e820 tx=0x7f3b7800ece0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b8f200640 1 -- 172.21.15.62:0/3171880868 shutdown_connections 2024-03-23T11:31:31.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b8f200640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3b6c06a7d0 0x7f3b6c06cc90 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b8f200640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b900b22c0 0x7f3b900ca060 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b8f200640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b901f55f0 0x7f3b901f5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b8f200640 1 --2- 172.21.15.62:0/3171880868 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b74004d00 0x7f3b900ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.377+0000 7f3b8f200640 1 -- 172.21.15.62:0/3171880868 >> 172.21.15.62:0/3171880868 conn(0x7f3b900b0090 msgr2=0x7f3b90157400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:31.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.378+0000 7f3b8f200640 1 -- 172.21.15.62:0/3171880868 shutdown_connections 2024-03-23T11:31:31.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.378+0000 7f3b8f200640 1 -- 172.21.15.62:0/3171880868 wait complete. 2024-03-23T11:31:31.383 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:31:31.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.600+0000 7f8b0b600640 1 Processor -- start 2024-03-23T11:31:31.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.600+0000 7f8b0b600640 1 -- start start 2024-03-23T11:31:31.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.602+0000 7f8b0b600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8b040cafa0 0x7f8b040cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.602+0000 7f8b0b600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 0x7f8b0406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.602+0000 7f8b0b600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b040cbc60 con 0x7f8b0406c300 2024-03-23T11:31:31.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.602+0000 7f8b0b600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b040cd410 con 0x7f8b040cafa0 2024-03-23T11:31:31.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.602+0000 7f8b0b600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b040cbf40 con 0x7f8b0406c6e0 2024-03-23T11:31:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.603+0000 7f8b09600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 0x7f8b0406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.603+0000 7f8b09600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 0x7f8b0406cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:59070/0 (socket says 172.21.15.62:59070) 2024-03-23T11:31:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.603+0000 7f8b09600640 1 -- 172.21.15.62:0/3283129829 learned_addr learned my addr 172.21.15.62:0/3283129829 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.603+0000 7f8b08c00640 1 --1- 172.21.15.62:0/3283129829 >> v1:172.21.15.120:6789/0 conn(0x7f8b0406c300 0x7f8b040ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:60754/0 (socket says 172.21.15.62:60754) 2024-03-23T11:31:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.603+0000 7f8b03e00640 1 --2- 172.21.15.62:0/3283129829 >> v2:172.21.15.120:3300/0 conn(0x7f8b040cafa0 0x7f8b040cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.604+0000 7f8b09600640 1 -- 172.21.15.62:0/3283129829 >> v2:172.21.15.120:3300/0 conn(0x7f8b040cafa0 msgr2=0x7f8b040cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:31.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.604+0000 7f8b09600640 1 --2- 172.21.15.62:0/3283129829 >> v2:172.21.15.120:3300/0 conn(0x7f8b040cafa0 0x7f8b040cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.604+0000 7f8b09600640 1 -- 172.21.15.62:0/3283129829 >> v1:172.21.15.120:6789/0 conn(0x7f8b0406c300 legacy=0x7f8b040ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.604+0000 7f8b09600640 1 -- 172.21.15.62:0/3283129829 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b040cc220 con 0x7f8b0406c6e0 2024-03-23T11:31:31.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.605+0000 7f8b09600640 1 --2- 172.21.15.62:0/3283129829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 0x7f8b0406cb70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009e50 tx=0x7f8afc02bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18b17601f2b24fc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:31.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.612+0000 7f8b03400640 1 -- 172.21.15.62:0/3283129829 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc02caf0 con 0x7f8b0406c6e0 2024-03-23T11:31:31.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.613+0000 7f8b03400640 1 -- 172.21.15.62:0/3283129829 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc02cc50 con 0x7f8b0406c6e0 2024-03-23T11:31:31.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.613+0000 7f8b03400640 1 -- 172.21.15.62:0/3283129829 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc0343f0 con 0x7f8b0406c6e0 2024-03-23T11:31:31.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.613+0000 7f8b0b600640 1 -- 172.21.15.62:0/3283129829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 msgr2=0x7f8b0406cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.613+0000 7f8b0b600640 1 --2- 172.21.15.62:0/3283129829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 0x7f8b0406cb70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009e50 tx=0x7f8afc02bf50 comp rx=0 tx=0).stop 2024-03-23T11:31:31.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.613+0000 7f8b0b600640 1 -- 172.21.15.62:0/3283129829 shutdown_connections 2024-03-23T11:31:31.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.613+0000 7f8b0b600640 1 --2- 172.21.15.62:0/3283129829 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 0x7f8b0406cb70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.613+0000 7f8b0b600640 1 --2- 172.21.15.62:0/3283129829 >> v2:172.21.15.120:3300/0 conn(0x7f8b040cafa0 0x7f8b040cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.613+0000 7f8b0b600640 1 -- 172.21.15.62:0/3283129829 >> 172.21.15.62:0/3283129829 conn(0x7f8b040b02e0 msgr2=0x7f8b040b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:31.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.613+0000 7f8b0b600640 1 -- 172.21.15.62:0/3283129829 shutdown_connections 2024-03-23T11:31:31.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.614+0000 7f8b0b600640 1 -- 172.21.15.62:0/3283129829 wait complete. 2024-03-23T11:31:31.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.616+0000 7f8b0b600640 1 Processor -- start 2024-03-23T11:31:31.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.617+0000 7f8b0b600640 1 -- start start 2024-03-23T11:31:31.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.618+0000 7f8b0b600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b0406c300 0x7f8b0413ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.618+0000 7f8b0b600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 0x7f8b04152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.618+0000 7f8b0b600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b040cafa0 0x7f8b0413ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.618+0000 7f8b0b600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0413ea50 con 0x7f8b040cafa0 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.618+0000 7f8b0b600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0413ddd0 con 0x7f8b0406c300 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.618+0000 7f8b0b600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0413e0b0 con 0x7f8b0406c6e0 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.618+0000 7f8b09600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b040cafa0 0x7f8b0413ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.618+0000 7f8b09600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b040cafa0 0x7f8b0413ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37802/0 (socket says 172.21.15.62:37802) 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.618+0000 7f8b09600640 1 -- 172.21.15.62:0/4161846507 learned_addr learned my addr 172.21.15.62:0/4161846507 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.618+0000 7f8b08c00640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b0406c300 0x7f8b0413ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.619+0000 7f8b03e00640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 0x7f8b04152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.619+0000 7f8b09600640 1 -- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 msgr2=0x7f8b04152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.619+0000 7f8b09600640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 0x7f8b04152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.619+0000 7f8b09600640 1 -- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b0406c300 msgr2=0x7f8b0413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.619+0000 7f8b09600640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b0406c300 0x7f8b0413ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.619+0000 7f8b09600640 1 -- 172.21.15.62:0/4161846507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc009b30 con 0x7f8b040cafa0 2024-03-23T11:31:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.619+0000 7f8b09600640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b040cafa0 0x7f8b0413ad50 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009e50 tx=0x7f8afc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.619+0000 7f8b00c00640 1 -- 172.21.15.62:0/4161846507 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc009510 con 0x7f8b040cafa0 2024-03-23T11:31:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.619+0000 7f8b00c00640 1 -- 172.21.15.62:0/4161846507 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc03e450 con 0x7f8b040cafa0 2024-03-23T11:31:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.619+0000 7f8b00c00640 1 -- 172.21.15.62:0/4161846507 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc03e800 con 0x7f8b040cafa0 2024-03-23T11:31:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.619+0000 7f8b0b600640 1 -- 172.21.15.62:0/4161846507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0413e390 con 0x7f8b040cafa0 2024-03-23T11:31:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.620+0000 7f8b0b600640 1 -- 172.21.15.62:0/4161846507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b04153130 con 0x7f8b040cafa0 2024-03-23T11:31:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.621+0000 7f8b0b600640 1 -- 172.21.15.62:0/4161846507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0413eb90 con 0x7f8b040cafa0 2024-03-23T11:31:31.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.621+0000 7f8b00c00640 1 -- 172.21.15.62:0/4161846507 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8afc03e960 con 0x7f8b040cafa0 2024-03-23T11:31:31.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.622+0000 7f8b00c00640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ae406a7a0 0x7f8ae406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:31.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.622+0000 7f8b00c00640 1 -- 172.21.15.62:0/4161846507 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8afc0abc10 con 0x7f8b040cafa0 2024-03-23T11:31:31.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.623+0000 7f8b08c00640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ae406a7a0 0x7f8ae406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:31.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.624+0000 7f8b08c00640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ae406a7a0 0x7f8ae406cc60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8af8000f60 tx=0x7f8af8007c80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.627+0000 7f8b00c00640 1 -- 172.21.15.62:0/4161846507 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8b0413eb90 con 0x7f8b040cafa0 2024-03-23T11:31:31.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.870+0000 7f8b0b600640 1 -- 172.21.15.62:0/4161846507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8b0413eb90 con 0x7f8b040cafa0 2024-03-23T11:31:31.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.871+0000 7f8b00c00640 1 -- 172.21.15.62:0/4161846507 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v94) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8afc043060 con 0x7f8b040cafa0 2024-03-23T11:31:31.861 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:31:31.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 -- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ae406a7a0 msgr2=0x7f8ae406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ae406a7a0 0x7f8ae406cc60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8af8000f60 tx=0x7f8af8007c80 comp rx=0 tx=0).stop 2024-03-23T11:31:31.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 -- 172.21.15.62:0/4161846507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b040cafa0 msgr2=0x7f8b0413ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:31.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b040cafa0 0x7f8b0413ad50 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009e50 tx=0x7f8afc004a40 comp rx=0 tx=0).stop 2024-03-23T11:31:31.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 -- 172.21.15.62:0/4161846507 shutdown_connections 2024-03-23T11:31:31.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8ae406a7a0 0x7f8ae406cc60 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b040cafa0 0x7f8b0413ad50 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b0406c6e0 0x7f8b04152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 --2- 172.21.15.62:0/4161846507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b0406c300 0x7f8b0413ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:31.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 -- 172.21.15.62:0/4161846507 >> 172.21.15.62:0/4161846507 conn(0x7f8b040b02e0 msgr2=0x7f8b040cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:31.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 -- 172.21.15.62:0/4161846507 shutdown_connections 2024-03-23T11:31:31.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:31.874+0000 7f8b0b600640 1 -- 172.21.15.62:0/4161846507 wait complete. 2024-03-23T11:31:31.883 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:31:31.884 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:31:31.884 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:31:31.884 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:31:31.884 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:31.884 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:31:31.884 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:31:31.884 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:31:31.884 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:31:31.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:31.884 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:31:31.941 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:31:31.941 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:31:31.942 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 41865sec preferred_lft 41865sec 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:31:31.943 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:31:31.944 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:31.944 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:31:31.944 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:31:31.944 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:31:31.944 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:31.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:31.944 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:31:32.040 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:31:32.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:32.045 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:31:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:31:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:31:32.104 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:31:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:31:32.105 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:31:32.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:32.105 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:31:32.105 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:31:32.105 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:31:32.105 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:31:32.105 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:31:32.105 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:31:32.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:32.303 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:31:32.359 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:31:32.359 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:31:32.360 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:31:32.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:32.360 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:31:32.459 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:31:32.459 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:31:32.459 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:32.514 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:31:32.514 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:31:32.514 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:32.568 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:31:32.667 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:31:32.667 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:31:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:31:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:31:32.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:32.767 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:31:32.824 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:31:32.824 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:31:32.831 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:31:32.912 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:31:32.913 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:31:32.987 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:31:32.991 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:31:32.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:32.991 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:31:33.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:31:33.033+0000 7f78cbc66480 -1 init, newargv = 0x55a84d8d6ea0 newargc=15 2024-03-23T11:31:33.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[77759]: starting ceph client 2024-03-23T11:31:33.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[77759]: starting fuse 2024-03-23T11:31:33.036 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:31:33.036 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:31:33.036 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:31:33.036 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:31:33.036 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:31:33.037 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:33.038 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:31:33.038 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:31:33.038 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:31:33.038 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:31:33.038 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:31:33.038 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:31:33.038 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:31:33.038 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:31:33.038 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:33.038 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:31:33.038 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:31:33.245 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.77759.asok 2024-03-23T11:31:33.427 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.77759.asok 2024-03-23T11:31:33.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:33.427 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.77759.asok status 2024-03-23T11:31:33.675 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:31:33.675 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:31:33.675 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:31:33.675 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:31:33.675 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "77759", 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 9153, 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:31:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 9153 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 822623953 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 822623953 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.9153 192.168.0.1:0/822623953", 2024-03-23T11:31:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/822623953", 2024-03-23T11:31:33.678 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:31:33.678 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 94, 2024-03-23T11:31:33.678 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 118, 2024-03-23T11:31:33.678 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:31:33.678 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:31:33.678 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:31:33.678 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:31:33.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:33.691 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:31:33.691 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:33.748 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:31:33.749 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:33.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:33.749 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:33.854 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:31:33.854 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:31:33.855 DEBUG:teuthology.orchestra.run.smithi062:> mount_point = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:31:33.855 DEBUG:teuthology.orchestra.run.smithi062:> parent = os.path.join(mount_point, "parent") 2024-03-23T11:31:33.855 DEBUG:teuthology.orchestra.run.smithi062:> os.mkdir(parent) 2024-03-23T11:31:33.855 DEBUG:teuthology.orchestra.run.smithi062:> flushed = os.path.join(parent, "flushed") 2024-03-23T11:31:33.855 DEBUG:teuthology.orchestra.run.smithi062:> os.mkdir(flushed) 2024-03-23T11:31:33.855 DEBUG:teuthology.orchestra.run.smithi062:> for f in ["alpha", "bravo", "charlie"]: 2024-03-23T11:31:33.855 DEBUG:teuthology.orchestra.run.smithi062:> open(os.path.join(flushed, f), '"'"'w'"'"').write(f) 2024-03-23T11:31:33.855 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:31:34.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:34.186 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-23T11:31:34.320 INFO:teuthology.orchestra.run.smithi062.stdout:./ 2024-03-23T11:31:34.320 INFO:teuthology.orchestra.run.smithi062.stdout:./parent 2024-03-23T11:31:34.320 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/flushed 2024-03-23T11:31:34.320 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/flushed/charlie 2024-03-23T11:31:34.321 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/flushed/alpha 2024-03-23T11:31:34.321 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/flushed/bravo 2024-03-23T11:31:34.322 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:31:34.322 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:31:34.322 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:31:34.322 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:34.322 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-23T11:31:34.323 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:31:34.481 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:31:34.662 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:31:34.662 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:31:34.662 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:31:34.662 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:34.662 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-03-23T11:31:34.662 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:31:34.858 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627776 2024-03-23T11:31:35.037 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:31:35.037 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:31:35.037 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:31:35.037 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:35.037 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-03-23T11:31:35.037 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:31:35.231 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627777 2024-03-23T11:31:35.412 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:31:35.412 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:31:35.412 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:31:35.412 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:35.412 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-03-23T11:31:35.412 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:31:35.591 INFO:teuthology.orchestra.run.smithi062.stdout:1099511628280 2024-03-23T11:31:35.770 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:31:35.770 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:31:35.770 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:31:35.770 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:35.770 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-03-23T11:31:35.770 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:31:35.970 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627778 2024-03-23T11:31:36.147 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:31:36.148 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:31:36.148 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:31:36.148 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:36.148 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-03-23T11:31:36.148 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:31:36.342 INFO:teuthology.orchestra.run.smithi062.stdout:1099511628279 2024-03-23T11:31:36.522 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:31:36.522 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:36.623 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:31:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.856+0000 7f4f53200640 1 Processor -- start 2024-03-23T11:31:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.856+0000 7f4f53200640 1 -- start start 2024-03-23T11:31:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.858+0000 7f4f53200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4f4c070090 0x7f4f4c070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.858+0000 7f4f53200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c0cd140 con 0x7f4f4c0b2820 2024-03-23T11:31:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.858+0000 7f4f53200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c0cd3e0 con 0x7f4f4c143dd0 2024-03-23T11:31:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.858+0000 7f4f53200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4c070a60 con 0x7f4f4c070090 2024-03-23T11:31:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.859+0000 7f4f52800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4f4c0b2820 0x7f4f4c0b2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42946/0 (socket says 172.21.15.62:42946) 2024-03-23T11:31:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.859+0000 7f4f52800640 1 -- 172.21.15.62:0/3032901871 learned_addr learned my addr 172.21.15.62:0/3032901871 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.859+0000 7f4f51400640 1 --2- 172.21.15.62:0/3032901871 >> v2:172.21.15.120:3300/0 conn(0x7f4f4c070090 0x7f4f4c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.859+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549694165 0 0) 0x7f4f4c0cd140 con 0x7f4f4c0b2820 2024-03-23T11:31:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.859+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f34003520 con 0x7f4f4c0b2820 2024-03-23T11:31:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.859+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2472181575 0 0) 0x7f4f34003520 con 0x7f4f4c0b2820 2024-03-23T11:31:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 >> v2:172.21.15.120:3300/0 conn(0x7f4f4c070090 msgr2=0x7f4f4c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 --2- 172.21.15.62:0/3032901871 >> v2:172.21.15.120:3300/0 conn(0x7f4f4c070090 0x7f4f4c070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 >> v1:172.21.15.120:6789/0 conn(0x7f4f4c143dd0 legacy=0x7f4f4c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c0cd6c0 con 0x7f4f4c0b2820 2024-03-23T11:31:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f4f480019f0 con 0x7f4f4c0b2820 2024-03-23T11:31:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 >> v1:172.21.15.62:6789/0 conn(0x7f4f4c0b2820 legacy=0x7f4f4c0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 --2- 172.21.15.62:0/3032901871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f34003970 0x7f4f34003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 --2- 172.21.15.62:0/3032901871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f340047c0 0x7f4f34004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 --2- 172.21.15.62:0/3032901871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 0x7f4f3400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c070a60 con 0x7f4f34005180 2024-03-23T11:31:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3400bfd0 con 0x7f4f34003970 2024-03-23T11:31:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3400c190 con 0x7f4f340047c0 2024-03-23T11:31:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f52800640 1 --2- 172.21.15.62:0/3032901871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 0x7f4f3400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f51e00640 1 --2- 172.21.15.62:0/3032901871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f34003970 0x7f4f34003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.860+0000 7f4f51400640 1 --2- 172.21.15.62:0/3032901871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f340047c0 0x7f4f34004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.861+0000 7f4f52800640 1 -- 172.21.15.62:0/3032901871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f340047c0 msgr2=0x7f4f34004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.861+0000 7f4f52800640 1 --2- 172.21.15.62:0/3032901871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f340047c0 0x7f4f34004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.861+0000 7f4f52800640 1 -- 172.21.15.62:0/3032901871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f34003970 msgr2=0x7f4f34003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.861+0000 7f4f52800640 1 --2- 172.21.15.62:0/3032901871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f34003970 0x7f4f34003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.861+0000 7f4f52800640 1 -- 172.21.15.62:0/3032901871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f34003520 con 0x7f4f34005180 2024-03-23T11:31:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.862+0000 7f4f52800640 1 -- 172.21.15.62:0/3032901871 reap_dead start 2024-03-23T11:31:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.862+0000 7f4f52800640 1 --2- 172.21.15.62:0/3032901871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 0x7f4f3400b980 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800a810 tx=0x7f4f340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93507f89c68ade8b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.862+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48034630 con 0x7f4f34005180 2024-03-23T11:31:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.862+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4802f070 con 0x7f4f34005180 2024-03-23T11:31:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.862+0000 7f4f50a00640 1 -- 172.21.15.62:0/3032901871 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48034a10 con 0x7f4f34005180 2024-03-23T11:31:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.862+0000 7f4f53200640 1 -- 172.21.15.62:0/3032901871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 msgr2=0x7f4f3400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.862+0000 7f4f53200640 1 --2- 172.21.15.62:0/3032901871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 0x7f4f3400b980 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800a810 tx=0x7f4f340057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.863+0000 7f4f53200640 1 -- 172.21.15.62:0/3032901871 shutdown_connections 2024-03-23T11:31:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.863+0000 7f4f53200640 1 --2- 172.21.15.62:0/3032901871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 0x7f4f3400b980 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.863+0000 7f4f53200640 1 -- 172.21.15.62:0/3032901871 >> 172.21.15.62:0/3032901871 conn(0x7f4f4c066050 msgr2=0x7f4f4c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.863+0000 7f4f53200640 1 -- 172.21.15.62:0/3032901871 shutdown_connections 2024-03-23T11:31:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.863+0000 7f4f53200640 1 -- 172.21.15.62:0/3032901871 wait complete. 2024-03-23T11:31:36.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.864+0000 7f4f53200640 1 Processor -- start 2024-03-23T11:31:36.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.865+0000 7f4f53200640 1 -- start start 2024-03-23T11:31:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.865+0000 7f4f53200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 0x7f4f4c159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.865+0000 7f4f53200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f4c0aed00 0x7f4f4c0af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f53200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f4c06c550 0x7f4f4c06ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f53200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c06cf40 con 0x7f4f34005180 2024-03-23T11:31:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f53200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c13fe50 con 0x7f4f4c06c550 2024-03-23T11:31:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f53200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c0b01e0 con 0x7f4f4c0aed00 2024-03-23T11:31:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f51e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 0x7f4f4c159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f51e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 0x7f4f4c159f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37814/0 (socket says 172.21.15.62:37814) 2024-03-23T11:31:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f51e00640 1 -- 172.21.15.62:0/2779766447 learned_addr learned my addr 172.21.15.62:0/2779766447 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f51e00640 1 -- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f4c0aed00 msgr2=0x7f4f4c0af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:36.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f51e00640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f4c0aed00 0x7f4f4c0af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:36.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f51e00640 1 -- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f4c06c550 msgr2=0x7f4f4c06ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:36.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f51e00640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f4c06c550 0x7f4f4c06ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:36.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.866+0000 7f4f51e00640 1 -- 172.21.15.62:0/2779766447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c0b0cd0 con 0x7f4f34005180 2024-03-23T11:31:36.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.867+0000 7f4f51e00640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 0x7f4f4c159f60 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000e560 tx=0x7f4f4000ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:36.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.867+0000 7f4f46000640 1 -- 172.21.15.62:0/2779766447 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f400052e0 con 0x7f4f34005180 2024-03-23T11:31:36.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.867+0000 7f4f46000640 1 -- 172.21.15.62:0/2779766447 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f40005440 con 0x7f4f34005180 2024-03-23T11:31:36.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.867+0000 7f4f46000640 1 -- 172.21.15.62:0/2779766447 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40003830 con 0x7f4f34005180 2024-03-23T11:31:36.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.867+0000 7f4f53200640 1 -- 172.21.15.62:0/2779766447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c0b0430 con 0x7f4f34005180 2024-03-23T11:31:36.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.867+0000 7f4f53200640 1 -- 172.21.15.62:0/2779766447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4c0b09f0 con 0x7f4f34005180 2024-03-23T11:31:36.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.868+0000 7f4f53200640 1 -- 172.21.15.62:0/2779766447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4c066bb0 con 0x7f4f34005180 2024-03-23T11:31:36.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.869+0000 7f4f46000640 1 -- 172.21.15.62:0/2779766447 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4f40003990 con 0x7f4f34005180 2024-03-23T11:31:36.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.870+0000 7f4f46000640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4f3006a7d0 0x7f4f3006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:36.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.870+0000 7f4f46000640 1 -- 172.21.15.62:0/2779766447 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f40088b80 con 0x7f4f34005180 2024-03-23T11:31:36.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.871+0000 7f4f51400640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4f3006a7d0 0x7f4f3006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:36.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.872+0000 7f4f51400640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4f3006a7d0 0x7f4f3006cc90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c06d170 tx=0x7f4f3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:36.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:36.875+0000 7f4f46000640 1 -- 172.21.15.62:0/2779766447 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4f40053f90 con 0x7f4f34005180 2024-03-23T11:31:37.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.076+0000 7f4f53200640 1 -- 172.21.15.62:0/2779766447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f4c0ca970 con 0x7f4f34005180 2024-03-23T11:31:37.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.078+0000 7f4f46000640 1 -- 172.21.15.62:0/2779766447 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16059 (secure 0 0 0) 0x7f4f40088e60 con 0x7f4f34005180 2024-03-23T11:31:37.069 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:37.070 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":118,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:31:25.048209+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:31:37.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.081+0000 7f4f53200640 1 -- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4f3006a7d0 msgr2=0x7f4f3006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.081+0000 7f4f53200640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4f3006a7d0 0x7f4f3006cc90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c06d170 tx=0x7f4f3c008040 comp rx=0 tx=0).stop 2024-03-23T11:31:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.081+0000 7f4f53200640 1 -- 172.21.15.62:0/2779766447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 msgr2=0x7f4f4c159f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.081+0000 7f4f53200640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 0x7f4f4c159f60 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000e560 tx=0x7f4f4000ea20 comp rx=0 tx=0).stop 2024-03-23T11:31:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.081+0000 7f4f53200640 1 -- 172.21.15.62:0/2779766447 shutdown_connections 2024-03-23T11:31:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.081+0000 7f4f53200640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4f3006a7d0 0x7f4f3006cc90 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.081+0000 7f4f53200640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f4c06c550 0x7f4f4c06ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.081+0000 7f4f53200640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4f4c0aed00 0x7f4f4c0af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.081+0000 7f4f53200640 1 --2- 172.21.15.62:0/2779766447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f34005180 0x7f4f4c159f60 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.081+0000 7f4f53200640 1 -- 172.21.15.62:0/2779766447 >> 172.21.15.62:0/2779766447 conn(0x7f4f4c066050 msgr2=0x7f4f4c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.082+0000 7f4f53200640 1 -- 172.21.15.62:0/2779766447 shutdown_connections 2024-03-23T11:31:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.082+0000 7f4f53200640 1 -- 172.21.15.62:0/2779766447 wait complete. 2024-03-23T11:31:37.093 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:31:22.127079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:31:23.062387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:31:37.093 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:31:37.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.329+0000 7f90b6c00640 1 Processor -- start 2024-03-23T11:31:37.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.330+0000 7f90b6c00640 1 -- start start 2024-03-23T11:31:37.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.331+0000 7f90b6c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b0160420 0x7f90b0160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.331+0000 7f90b6c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f90b00b5db0 0x7f90b00b6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.331+0000 7f90b6c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b0160dc0 con 0x7f90b015fcf0 2024-03-23T11:31:37.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.331+0000 7f90b6c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b00ce510 con 0x7f90b00b5db0 2024-03-23T11:31:37.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.331+0000 7f90b6c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b00cdc70 con 0x7f90b0160420 2024-03-23T11:31:37.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.332+0000 7f90b6200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f90b00b5db0 0x7f90b00b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.332+0000 7f90b5800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f90b015fcf0 0x7f90b00d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57124/0 (socket says 172.21.15.62:57124) 2024-03-23T11:31:37.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.332+0000 7f90b5800640 1 -- 172.21.15.62:0/2723658640 learned_addr learned my addr 172.21.15.62:0/2723658640 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:37.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.332+0000 7f90b4e00640 1 --2- 172.21.15.62:0/2723658640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b0160420 0x7f90b0160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.332+0000 7f90b6200640 1 -- 172.21.15.62:0/2723658640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b0160420 msgr2=0x7f90b0160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:37.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.332+0000 7f90b6200640 1 --2- 172.21.15.62:0/2723658640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b0160420 0x7f90b0160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.332+0000 7f90b6200640 1 -- 172.21.15.62:0/2723658640 >> v1:172.21.15.62:6789/0 conn(0x7f90b015fcf0 legacy=0x7f90b00d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.332+0000 7f90b6200640 1 -- 172.21.15.62:0/2723658640 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b00cdf50 con 0x7f90b00b5db0 2024-03-23T11:31:37.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.332+0000 7f90afe00640 1 -- 172.21.15.62:0/2723658640 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262338380 0 0) 0x7f90b0160dc0 con 0x7f90b015fcf0 2024-03-23T11:31:37.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.334+0000 7f90b6200640 1 --2- 172.21.15.62:0/2723658640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00b5db0 0x7f90b00b6220 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009d90 tx=0x7f90a802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71be753ad70cd0f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:37.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.334+0000 7f90afe00640 1 -- 172.21.15.62:0/2723658640 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a802cbc0 con 0x7f90b00b5db0 2024-03-23T11:31:37.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.334+0000 7f90afe00640 1 -- 172.21.15.62:0/2723658640 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a802cd20 con 0x7f90b00b5db0 2024-03-23T11:31:37.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.334+0000 7f90afe00640 1 -- 172.21.15.62:0/2723658640 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a8034660 con 0x7f90b00b5db0 2024-03-23T11:31:37.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.334+0000 7f90b6c00640 1 -- 172.21.15.62:0/2723658640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00b5db0 msgr2=0x7f90b00b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.334+0000 7f90b6c00640 1 --2- 172.21.15.62:0/2723658640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00b5db0 0x7f90b00b6220 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009d90 tx=0x7f90a802be10 comp rx=0 tx=0).stop 2024-03-23T11:31:37.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.334+0000 7f90b6c00640 1 -- 172.21.15.62:0/2723658640 shutdown_connections 2024-03-23T11:31:37.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.334+0000 7f90b6c00640 1 --2- 172.21.15.62:0/2723658640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00b5db0 0x7f90b00b6220 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.334+0000 7f90b6c00640 1 --2- 172.21.15.62:0/2723658640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b0160420 0x7f90b0160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.335+0000 7f90b6c00640 1 -- 172.21.15.62:0/2723658640 >> 172.21.15.62:0/2723658640 conn(0x7f90b0089f60 msgr2=0x7f90b00d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:37.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.335+0000 7f90b6c00640 1 -- 172.21.15.62:0/2723658640 shutdown_connections 2024-03-23T11:31:37.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.335+0000 7f90b6c00640 1 -- 172.21.15.62:0/2723658640 wait complete. 2024-03-23T11:31:37.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.336+0000 7f90b6c00640 1 Processor -- start 2024-03-23T11:31:37.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.336+0000 7f90b6c00640 1 -- start start 2024-03-23T11:31:37.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.337+0000 7f90b6c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00b5db0 0x7f90b01ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.337+0000 7f90b6c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b015fcf0 0x7f90b01f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.337+0000 7f90b6c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b0160420 0x7f90b01f5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.337+0000 7f90b6c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b015a7c0 con 0x7f90b00b5db0 2024-03-23T11:31:37.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.337+0000 7f90b6c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b01f69b0 con 0x7f90b015fcf0 2024-03-23T11:31:37.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.337+0000 7f90b6c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b00c62d0 con 0x7f90b0160420 2024-03-23T11:31:37.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.338+0000 7f90b5800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00b5db0 0x7f90b01ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.338+0000 7f90b5800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00b5db0 0x7f90b01ef8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42338/0 (socket says 172.21.15.62:42338) 2024-03-23T11:31:37.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.338+0000 7f90b5800640 1 -- 172.21.15.62:0/3925266435 learned_addr learned my addr 172.21.15.62:0/3925266435 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:37.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.338+0000 7f90b6200640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b0160420 0x7f90b01f5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.338+0000 7f90b5800640 1 -- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b0160420 msgr2=0x7f90b01f5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.338+0000 7f90b5800640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b0160420 0x7f90b01f5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.338+0000 7f90b5800640 1 -- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b015fcf0 msgr2=0x7f90b01f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.338+0000 7f90b5800640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b015fcf0 0x7f90b01f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.338+0000 7f90b5800640 1 -- 172.21.15.62:0/3925266435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a8009a70 con 0x7f90b00b5db0 2024-03-23T11:31:37.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.338+0000 7f90b5800640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00b5db0 0x7f90b01ef8d0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f90a000c4d0 tx=0x7f90a000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:37.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.339+0000 7f90ad600640 1 -- 172.21.15.62:0/3925266435 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a00074e0 con 0x7f90b00b5db0 2024-03-23T11:31:37.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.339+0000 7f90ad600640 1 -- 172.21.15.62:0/3925266435 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a0007660 con 0x7f90b00b5db0 2024-03-23T11:31:37.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.339+0000 7f90ad600640 1 -- 172.21.15.62:0/3925266435 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a0004a20 con 0x7f90b00b5db0 2024-03-23T11:31:37.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.339+0000 7f90b6c00640 1 -- 172.21.15.62:0/3925266435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b00c65b0 con 0x7f90b00b5db0 2024-03-23T11:31:37.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.339+0000 7f90b6c00640 1 -- 172.21.15.62:0/3925266435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b00c6b70 con 0x7f90b00b5db0 2024-03-23T11:31:37.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.340+0000 7f90b6c00640 1 -- 172.21.15.62:0/3925266435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b00b5580 con 0x7f90b00b5db0 2024-03-23T11:31:37.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.341+0000 7f90ad600640 1 -- 172.21.15.62:0/3925266435 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f90a0004b80 con 0x7f90b00b5db0 2024-03-23T11:31:37.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.342+0000 7f90ad600640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f909006a7f0 0x7f909006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.342+0000 7f90ad600640 1 -- 172.21.15.62:0/3925266435 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90a0088430 con 0x7f90b00b5db0 2024-03-23T11:31:37.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.343+0000 7f90b4e00640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f909006a7f0 0x7f909006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.344+0000 7f90b4e00640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f909006a7f0 0x7f909006ccb0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f909c006fd0 tx=0x7f909c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:37.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.349+0000 7f90ad600640 1 -- 172.21.15.62:0/3925266435 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f90a0053830 con 0x7f90b00b5db0 2024-03-23T11:31:37.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.547+0000 7f90b6c00640 1 -- 172.21.15.62:0/3925266435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f90b00c6d90 con 0x7f90b00b5db0 2024-03-23T11:31:37.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.547+0000 7f90ad600640 1 -- 172.21.15.62:0/3925266435 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7f90a00180a0 con 0x7f90b00b5db0 2024-03-23T11:31:37.539 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:31:37.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.550+0000 7f90b6c00640 1 -- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f909006a7f0 msgr2=0x7f909006ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.550+0000 7f90b6c00640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f909006a7f0 0x7f909006ccb0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f909c006fd0 tx=0x7f909c008040 comp rx=0 tx=0).stop 2024-03-23T11:31:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.550+0000 7f90b6c00640 1 -- 172.21.15.62:0/3925266435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00b5db0 msgr2=0x7f90b01ef8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.550+0000 7f90b6c00640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00b5db0 0x7f90b01ef8d0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f90a000c4d0 tx=0x7f90a000c990 comp rx=0 tx=0).stop 2024-03-23T11:31:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.550+0000 7f90b6c00640 1 -- 172.21.15.62:0/3925266435 shutdown_connections 2024-03-23T11:31:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.550+0000 7f90b6c00640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f909006a7f0 0x7f909006ccb0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.550+0000 7f90b6c00640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b0160420 0x7f90b01f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.550+0000 7f90b6c00640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b015fcf0 0x7f90b01f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.550+0000 7f90b6c00640 1 --2- 172.21.15.62:0/3925266435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00b5db0 0x7f90b01ef8d0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.550+0000 7f90b6c00640 1 -- 172.21.15.62:0/3925266435 >> 172.21.15.62:0/3925266435 conn(0x7f90b0089f60 msgr2=0x7f90b00d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.551+0000 7f90b6c00640 1 -- 172.21.15.62:0/3925266435 shutdown_connections 2024-03-23T11:31:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.551+0000 7f90b6c00640 1 -- 172.21.15.62:0/3925266435 wait complete. 2024-03-23T11:31:37.559 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:31:37.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.785+0000 7f61cde00640 1 Processor -- start 2024-03-23T11:31:37.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.785+0000 7f61cde00640 1 -- start start 2024-03-23T11:31:37.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.787+0000 7f61cde00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80ae880 0x7f61c80af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.787+0000 7f61cde00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c80cc860 con 0x7f61c80989d0 2024-03-23T11:31:37.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.787+0000 7f61cde00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c80cbfc0 con 0x7f61c80ce8d0 2024-03-23T11:31:37.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.787+0000 7f61cde00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c80cac10 con 0x7f61c80ae880 2024-03-23T11:31:37.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.787+0000 7f61cd400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f61c80989d0 0x7f61c8098e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57126/0 (socket says 172.21.15.62:57126) 2024-03-23T11:31:37.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.787+0000 7f61cd400640 1 -- 172.21.15.62:0/1810945890 learned_addr learned my addr 172.21.15.62:0/1810945890 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:37.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7e00640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80ae880 0x7f61c80af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080553887 0 0) 0x7f61c80cc860 con 0x7f61c80989d0 2024-03-23T11:31:37.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b0003520 con 0x7f61c80989d0 2024-03-23T11:31:37.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981678664 0 0) 0x7f61b0003520 con 0x7f61c80989d0 2024-03-23T11:31:37.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80ae880 msgr2=0x7f61c80af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:37.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80ae880 0x7f61c80af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 >> v1:172.21.15.120:6789/0 conn(0x7f61c80ce8d0 legacy=0x7f61c80cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c80cc2a0 con 0x7f61c80989d0 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f61c00019f0 con 0x7f61c80989d0 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 >> v1:172.21.15.62:6789/0 conn(0x7f61c80989d0 legacy=0x7f61c8098e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b00039a0 0x7f61b0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80ae880 0x7f61b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0004d00 0x7f61b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b000b980 con 0x7f61b0004d00 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b000bac0 con 0x7f61b00039a0 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.788+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b000bc80 con 0x7f61c80ae880 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.789+0000 7f61cd400640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0004d00 0x7f61b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.789+0000 7f61c7e00640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80ae880 0x7f61b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.789+0000 7f61cca00640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b00039a0 0x7f61b0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.789+0000 7f61cd400640 1 -- 172.21.15.62:0/1810945890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80ae880 msgr2=0x7f61b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:37.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.789+0000 7f61cd400640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80ae880 0x7f61b00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.789+0000 7f61cd400640 1 -- 172.21.15.62:0/1810945890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b00039a0 msgr2=0x7f61b0003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:37.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.789+0000 7f61cd400640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b00039a0 0x7f61b0003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.789+0000 7f61cd400640 1 -- 172.21.15.62:0/1810945890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b0003520 con 0x7f61b0004d00 2024-03-23T11:31:37.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.790+0000 7f61cd400640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0004d00 0x7f61b0005180 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f61c000a830 tx=0x7f61b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc77165e0ae58a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:37.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.790+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c002cea0 con 0x7f61b0004d00 2024-03-23T11:31:37.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.791+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c002f070 con 0x7f61b0004d00 2024-03-23T11:31:37.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.791+0000 7f61c7400640 1 -- 172.21.15.62:0/1810945890 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0034890 con 0x7f61b0004d00 2024-03-23T11:31:37.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.791+0000 7f61cde00640 1 -- 172.21.15.62:0/1810945890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0004d00 msgr2=0x7f61b0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.791+0000 7f61cde00640 1 --2- 172.21.15.62:0/1810945890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0004d00 0x7f61b0005180 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f61c000a830 tx=0x7f61b00057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.791+0000 7f61cd400640 1 -- 172.21.15.62:0/1810945890 reap_dead start 2024-03-23T11:31:37.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.791+0000 7f61cde00640 1 -- 172.21.15.62:0/1810945890 shutdown_connections 2024-03-23T11:31:37.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.791+0000 7f61cde00640 1 -- 172.21.15.62:0/1810945890 >> 172.21.15.62:0/1810945890 conn(0x7f61c80b1fb0 msgr2=0x7f61c80b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:37.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.791+0000 7f61cde00640 1 -- 172.21.15.62:0/1810945890 shutdown_connections 2024-03-23T11:31:37.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.791+0000 7f61cde00640 1 -- 172.21.15.62:0/1810945890 wait complete. 2024-03-23T11:31:37.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.793+0000 7f61cde00640 1 Processor -- start 2024-03-23T11:31:37.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.793+0000 7f61cde00640 1 -- start start 2024-03-23T11:31:37.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.794+0000 7f61cde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c814ee00 0x7f61c815b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.794+0000 7f61cde00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61c815bb40 0x7f61c80b07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.794+0000 7f61cde00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80b0ce0 0x7f61c80c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.794+0000 7f61cde00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c80b1190 con 0x7f61c814ee00 2024-03-23T11:31:37.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.794+0000 7f61cde00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c815bfc0 con 0x7f61c815bb40 2024-03-23T11:31:37.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.794+0000 7f61cde00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c80c9da0 con 0x7f61c80b0ce0 2024-03-23T11:31:37.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61cca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c814ee00 0x7f61c815b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61cca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c814ee00 0x7f61c815b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42368/0 (socket says 172.21.15.62:42368) 2024-03-23T11:31:37.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61cca00640 1 -- 172.21.15.62:0/290271039 learned_addr learned my addr 172.21.15.62:0/290271039 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:37.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61cd400640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80b0ce0 0x7f61c80c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61c7e00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61c815bb40 0x7f61c80b07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61cca00640 1 -- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80b0ce0 msgr2=0x7f61c80c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61cca00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80b0ce0 0x7f61c80c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61cca00640 1 -- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61c815bb40 msgr2=0x7f61c80b07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:37.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61cca00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61c815bb40 0x7f61c80b07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:37.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61cca00640 1 -- 172.21.15.62:0/290271039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c8156200 con 0x7f61c814ee00 2024-03-23T11:31:37.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61c7e00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61c815bb40 0x7f61c80b07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:37.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61cd400640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80b0ce0 0x7f61c80c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:37.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.795+0000 7f61cca00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c814ee00 0x7f61c815b600 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f61b800e5d0 tx=0x7f61b800ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:37.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.796+0000 7f61c4c00640 1 -- 172.21.15.62:0/290271039 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b80053a0 con 0x7f61c814ee00 2024-03-23T11:31:37.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.796+0000 7f61c4c00640 1 -- 172.21.15.62:0/290271039 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b80035f0 con 0x7f61c814ee00 2024-03-23T11:31:37.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.796+0000 7f61c4c00640 1 -- 172.21.15.62:0/290271039 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b80039a0 con 0x7f61c814ee00 2024-03-23T11:31:37.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.796+0000 7f61cde00640 1 -- 172.21.15.62:0/290271039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c814f1e0 con 0x7f61c814ee00 2024-03-23T11:31:37.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.796+0000 7f61cde00640 1 -- 172.21.15.62:0/290271039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c80b1660 con 0x7f61c814ee00 2024-03-23T11:31:37.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.797+0000 7f61cde00640 1 -- 172.21.15.62:0/290271039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c8073540 con 0x7f61c814ee00 2024-03-23T11:31:37.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.798+0000 7f61c4c00640 1 -- 172.21.15.62:0/290271039 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f61b8003b00 con 0x7f61c814ee00 2024-03-23T11:31:37.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.799+0000 7f61c4c00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f61ac06a7d0 0x7f61ac06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:37.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.799+0000 7f61c4c00640 1 -- 172.21.15.62:0/290271039 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61b8088c50 con 0x7f61c814ee00 2024-03-23T11:31:37.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.800+0000 7f61c7e00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f61ac06a7d0 0x7f61ac06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:37.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.801+0000 7f61c7e00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f61ac06a7d0 0x7f61ac06cc90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f61c80cb1f0 tx=0x7f61bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:37.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:37.803+0000 7f61c4c00640 1 -- 172.21.15.62:0/290271039 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f61b8053fd0 con 0x7f61c814ee00 2024-03-23T11:31:37.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.004+0000 7f61cde00640 1 -- 172.21.15.62:0/290271039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f61c80672c0 con 0x7f61c814ee00 2024-03-23T11:31:37.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.005+0000 7f61c4c00640 1 -- 172.21.15.62:0/290271039 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) v1 ==== 104+0+11 (secure 0 0 0) 0x7f61b8003db0 con 0x7f61c814ee00 2024-03-23T11:31:37.996 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.009+0000 7f61cde00640 1 -- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f61ac06a7d0 msgr2=0x7f61ac06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.009+0000 7f61cde00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f61ac06a7d0 0x7f61ac06cc90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f61c80cb1f0 tx=0x7f61bc008040 comp rx=0 tx=0).stop 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.009+0000 7f61cde00640 1 -- 172.21.15.62:0/290271039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c814ee00 msgr2=0x7f61c815b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.009+0000 7f61cde00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c814ee00 0x7f61c815b600 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f61b800e5d0 tx=0x7f61b800ea90 comp rx=0 tx=0).stop 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.009+0000 7f61cde00640 1 -- 172.21.15.62:0/290271039 shutdown_connections 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.010+0000 7f61cde00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f61ac06a7d0 0x7f61ac06cc90 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.010+0000 7f61cde00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61c80b0ce0 0x7f61c80c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.010+0000 7f61cde00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61c815bb40 0x7f61c80b07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.010+0000 7f61cde00640 1 --2- 172.21.15.62:0/290271039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c814ee00 0x7f61c815b600 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.010+0000 7f61cde00640 1 -- 172.21.15.62:0/290271039 >> 172.21.15.62:0/290271039 conn(0x7f61c80b1fb0 msgr2=0x7f61c80cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.010+0000 7f61cde00640 1 -- 172.21.15.62:0/290271039 shutdown_connections 2024-03-23T11:31:38.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.010+0000 7f61cde00640 1 -- 172.21.15.62:0/290271039 wait complete. 2024-03-23T11:31:38.019 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:31:38.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.239+0000 7f597fe00640 1 Processor -- start 2024-03-23T11:31:38.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.239+0000 7f597fe00640 1 -- start start 2024-03-23T11:31:38.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.241+0000 7f597fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5980160c00 0x7f598006d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.241+0000 7f597fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f59800cda20 0x7f598006de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.241+0000 7f597fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59800cde90 0x7f59800ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.241+0000 7f597fe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59800b1ad0 con 0x7f5980160c00 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.241+0000 7f597fe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59800ce840 con 0x7f59800cda20 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.241+0000 7f597fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5980160fe0 con 0x7f59800cde90 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.241+0000 7f597ea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5980160c00 0x7f598006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.241+0000 7f597ea00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5980160c00 0x7f598006d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42376/0 (socket says 172.21.15.62:42376) 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.241+0000 7f597ea00640 1 -- 172.21.15.62:0/1157062039 learned_addr learned my addr 172.21.15.62:0/1157062039 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.242+0000 7f597ea00640 1 -- 172.21.15.62:0/1157062039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59800cde90 msgr2=0x7f59800ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.242+0000 7f597ea00640 1 --2- 172.21.15.62:0/1157062039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59800cde90 0x7f59800ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.242+0000 7f597ea00640 1 -- 172.21.15.62:0/1157062039 >> v2:172.21.15.120:3300/0 conn(0x7f59800cda20 msgr2=0x7f598006de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.242+0000 7f597ea00640 1 --2- 172.21.15.62:0/1157062039 >> v2:172.21.15.120:3300/0 conn(0x7f59800cda20 0x7f598006de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.242+0000 7f597ea00640 1 -- 172.21.15.62:0/1157062039 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59800b1510 con 0x7f5980160c00 2024-03-23T11:31:38.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.243+0000 7f597ea00640 1 --2- 172.21.15.62:0/1157062039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5980160c00 0x7f598006d530 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f59800b3730 tx=0x7f597002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a95e0dcc1662eef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:38.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.243+0000 7f597e000640 1 -- 172.21.15.62:0/1157062039 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970031640 con 0x7f5980160c00 2024-03-23T11:31:38.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.243+0000 7f597e000640 1 -- 172.21.15.62:0/1157062039 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59700317a0 con 0x7f5980160c00 2024-03-23T11:31:38.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.243+0000 7f597e000640 1 -- 172.21.15.62:0/1157062039 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970031b30 con 0x7f5980160c00 2024-03-23T11:31:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.244+0000 7f597fe00640 1 -- 172.21.15.62:0/1157062039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5980160c00 msgr2=0x7f598006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.244+0000 7f597fe00640 1 --2- 172.21.15.62:0/1157062039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5980160c00 0x7f598006d530 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f59800b3730 tx=0x7f597002b8f0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.244+0000 7f597fe00640 1 -- 172.21.15.62:0/1157062039 shutdown_connections 2024-03-23T11:31:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.244+0000 7f597fe00640 1 --2- 172.21.15.62:0/1157062039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59800cde90 0x7f59800ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.244+0000 7f597fe00640 1 --2- 172.21.15.62:0/1157062039 >> v2:172.21.15.120:3300/0 conn(0x7f59800cda20 0x7f598006de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.244+0000 7f597fe00640 1 --2- 172.21.15.62:0/1157062039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5980160c00 0x7f598006d530 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.244+0000 7f597fe00640 1 -- 172.21.15.62:0/1157062039 >> 172.21.15.62:0/1157062039 conn(0x7f5980066050 msgr2=0x7f5980069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.244+0000 7f597fe00640 1 -- 172.21.15.62:0/1157062039 shutdown_connections 2024-03-23T11:31:38.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.244+0000 7f597fe00640 1 -- 172.21.15.62:0/1157062039 wait complete. 2024-03-23T11:31:38.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.246+0000 7f597fe00640 1 Processor -- start 2024-03-23T11:31:38.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.246+0000 7f597fe00640 1 -- start start 2024-03-23T11:31:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.247+0000 7f597fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59800cda20 0x7f598015cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.247+0000 7f597fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59800cde90 0x7f59801ebb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.247+0000 7f597fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5980160c00 0x7f59801ec060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.247+0000 7f597fe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59800a92b0 con 0x7f5980160c00 2024-03-23T11:31:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.247+0000 7f597fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59800a8210 con 0x7f59800cde90 2024-03-23T11:31:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.247+0000 7f597fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59800a84f0 con 0x7f59800cda20 2024-03-23T11:31:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.247+0000 7f597f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5980160c00 0x7f59801ec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.248+0000 7f597f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5980160c00 0x7f59801ec060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42388/0 (socket says 172.21.15.62:42388) 2024-03-23T11:31:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.248+0000 7f597f400640 1 -- 172.21.15.62:0/4247269979 learned_addr learned my addr 172.21.15.62:0/4247269979 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:38.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.248+0000 7f5976000640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59800cde90 0x7f59801ebb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:38.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.248+0000 7f597ea00640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59800cda20 0x7f598015cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:38.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.248+0000 7f597f400640 1 -- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59800cda20 msgr2=0x7f598015cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.248+0000 7f597f400640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59800cda20 0x7f598015cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.248+0000 7f597f400640 1 -- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59800cde90 msgr2=0x7f59801ebb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.248+0000 7f597f400640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59800cde90 0x7f59801ebb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.248+0000 7f597f400640 1 -- 172.21.15.62:0/4247269979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59700094d0 con 0x7f5980160c00 2024-03-23T11:31:38.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.248+0000 7f597f400640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5980160c00 0x7f59801ec060 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f597800b970 tx=0x7f597800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:38.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.249+0000 7f5977e00640 1 -- 172.21.15.62:0/4247269979 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597800cb40 con 0x7f5980160c00 2024-03-23T11:31:38.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.249+0000 7f5977e00640 1 -- 172.21.15.62:0/4247269979 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597800ccc0 con 0x7f5980160c00 2024-03-23T11:31:38.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.249+0000 7f597fe00640 1 -- 172.21.15.62:0/4247269979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59800a87d0 con 0x7f5980160c00 2024-03-23T11:31:38.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.249+0000 7f5977e00640 1 -- 172.21.15.62:0/4247269979 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5978014680 con 0x7f5980160c00 2024-03-23T11:31:38.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.249+0000 7f597fe00640 1 -- 172.21.15.62:0/4247269979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59800a8d90 con 0x7f5980160c00 2024-03-23T11:31:38.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.250+0000 7f597fe00640 1 -- 172.21.15.62:0/4247269979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59800a8fb0 con 0x7f5980160c00 2024-03-23T11:31:38.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.251+0000 7f5977e00640 1 -- 172.21.15.62:0/4247269979 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f597801d3b0 con 0x7f5980160c00 2024-03-23T11:31:38.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.252+0000 7f5977e00640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f595c06a7a0 0x7f595c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.252+0000 7f5977e00640 1 -- 172.21.15.62:0/4247269979 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5978088d20 con 0x7f5980160c00 2024-03-23T11:31:38.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.252+0000 7f597ea00640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f595c06a7a0 0x7f595c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:38.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.253+0000 7f597ea00640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f595c06a7a0 0x7f595c06cc60 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5970036000 tx=0x7f597003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:38.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.256+0000 7f5977e00640 1 -- 172.21.15.62:0/4247269979 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5978054130 con 0x7f5980160c00 2024-03-23T11:31:38.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.457+0000 7f597fe00640 1 -- 172.21.15.62:0/4247269979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5980066bb0 con 0x7f5980160c00 2024-03-23T11:31:38.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.458+0000 7f5977e00640 1 -- 172.21.15.62:0/4247269979 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) v1 ==== 100+0+11 (secure 0 0 0) 0x7f59780190a0 con 0x7f5980160c00 2024-03-23T11:31:38.448 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:31:38.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 -- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f595c06a7a0 msgr2=0x7f595c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f595c06a7a0 0x7f595c06cc60 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5970036000 tx=0x7f597003c040 comp rx=0 tx=0).stop 2024-03-23T11:31:38.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 -- 172.21.15.62:0/4247269979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5980160c00 msgr2=0x7f59801ec060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5980160c00 0x7f59801ec060 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f597800b970 tx=0x7f597800be30 comp rx=0 tx=0).stop 2024-03-23T11:31:38.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 -- 172.21.15.62:0/4247269979 shutdown_connections 2024-03-23T11:31:38.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f595c06a7a0 0x7f595c06cc60 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5980160c00 0x7f59801ec060 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59800cde90 0x7f59801ebb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 --2- 172.21.15.62:0/4247269979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59800cda20 0x7f598015cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 -- 172.21.15.62:0/4247269979 >> 172.21.15.62:0/4247269979 conn(0x7f5980066050 msgr2=0x7f598015e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:38.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 -- 172.21.15.62:0/4247269979 shutdown_connections 2024-03-23T11:31:38.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.461+0000 7f597fe00640 1 -- 172.21.15.62:0/4247269979 wait complete. 2024-03-23T11:31:38.472 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:31:38.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.693+0000 7f5922400640 1 Processor -- start 2024-03-23T11:31:38.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.693+0000 7f5922400640 1 -- start start 2024-03-23T11:31:38.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.695+0000 7f5922400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f591c0caff0 0x7f591c0cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.695+0000 7f5922400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f591c143450 0x7f591c1438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.695+0000 7f5922400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c15bb10 con 0x7f591c069c60 2024-03-23T11:31:38.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.695+0000 7f5922400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591c0ce1a0 con 0x7f591c0caff0 2024-03-23T11:31:38.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.695+0000 7f5922400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0cbcb0 con 0x7f591c143450 2024-03-23T11:31:38.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.695+0000 7f5921a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f591c143450 0x7f591c1438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:38.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.695+0000 7f5921a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f591c143450 0x7f591c1438e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:48242/0 (socket says 172.21.15.62:48242) 2024-03-23T11:31:38.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.696+0000 7f5921a00640 1 -- 172.21.15.62:0/4233326463 learned_addr learned my addr 172.21.15.62:0/4233326463 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:38.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.696+0000 7f5921000640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f591c069c60 0x7f591c06a040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:59168/0 (socket says 172.21.15.62:59168) 2024-03-23T11:31:38.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.696+0000 7f591be00640 1 --2- 172.21.15.62:0/4233326463 >> v2:172.21.15.120:3300/0 conn(0x7f591c0caff0 0x7f591c0cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:38.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.696+0000 7f591be00640 1 -- 172.21.15.62:0/4233326463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f591c143450 msgr2=0x7f591c1438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:38.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.696+0000 7f591be00640 1 --2- 172.21.15.62:0/4233326463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f591c143450 0x7f591c1438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.696+0000 7f591be00640 1 -- 172.21.15.62:0/4233326463 >> v1:172.21.15.120:6789/0 conn(0x7f591c069c60 legacy=0x7f591c06a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.696+0000 7f591be00640 1 -- 172.21.15.62:0/4233326463 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c0cb9e0 con 0x7f591c0caff0 2024-03-23T11:31:38.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.698+0000 7f591be00640 1 --2- 172.21.15.62:0/4233326463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f591c0caff0 0x7f591c0cb4a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f59080097d0 tx=0x7f590802b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbfda85c247dd4db server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:38.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.698+0000 7f591b400640 1 -- 172.21.15.62:0/4233326463 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908031800 con 0x7f591c0caff0 2024-03-23T11:31:38.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.698+0000 7f591b400640 1 -- 172.21.15.62:0/4233326463 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5908031960 con 0x7f591c0caff0 2024-03-23T11:31:38.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.698+0000 7f591b400640 1 -- 172.21.15.62:0/4233326463 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908031cf0 con 0x7f591c0caff0 2024-03-23T11:31:38.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.699+0000 7f5922400640 1 -- 172.21.15.62:0/4233326463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f591c0caff0 msgr2=0x7f591c0cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.699+0000 7f5922400640 1 --2- 172.21.15.62:0/4233326463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f591c0caff0 0x7f591c0cb4a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f59080097d0 tx=0x7f590802b8a0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.699+0000 7f5922400640 1 -- 172.21.15.62:0/4233326463 shutdown_connections 2024-03-23T11:31:38.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.699+0000 7f5922400640 1 --2- 172.21.15.62:0/4233326463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f591c143450 0x7f591c1438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.699+0000 7f5922400640 1 --2- 172.21.15.62:0/4233326463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f591c0caff0 0x7f591c0cb4a0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.699+0000 7f5922400640 1 -- 172.21.15.62:0/4233326463 >> 172.21.15.62:0/4233326463 conn(0x7f591c0b0090 msgr2=0x7f591c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:38.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.699+0000 7f5922400640 1 -- 172.21.15.62:0/4233326463 shutdown_connections 2024-03-23T11:31:38.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.699+0000 7f5922400640 1 -- 172.21.15.62:0/4233326463 wait complete. 2024-03-23T11:31:38.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.701+0000 7f5922400640 1 Processor -- start 2024-03-23T11:31:38.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.701+0000 7f5922400640 1 -- start start 2024-03-23T11:31:38.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.702+0000 7f5922400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f591c069c60 0x7f591c1e6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.702+0000 7f5922400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f591c0caff0 0x7f591c1e2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.702+0000 7f5922400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f591c143450 0x7f591c1f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.702+0000 7f5922400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c1e3d70 con 0x7f591c0caff0 2024-03-23T11:31:38.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.702+0000 7f5922400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c1e34d0 con 0x7f591c143450 2024-03-23T11:31:38.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.702+0000 7f5922400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c1e37b0 con 0x7f591c069c60 2024-03-23T11:31:38.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.703+0000 7f591be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f591c0caff0 0x7f591c1e2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:38.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.703+0000 7f591be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f591c0caff0 0x7f591c1e2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42398/0 (socket says 172.21.15.62:42398) 2024-03-23T11:31:38.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.703+0000 7f591be00640 1 -- 172.21.15.62:0/20302572 learned_addr learned my addr 172.21.15.62:0/20302572 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:38.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.703+0000 7f5921000640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f591c069c60 0x7f591c1e6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:38.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.703+0000 7f5921a00640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f591c143450 0x7f591c1f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:38.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.703+0000 7f591be00640 1 -- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f591c069c60 msgr2=0x7f591c1e6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.703+0000 7f591be00640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f591c069c60 0x7f591c1e6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.703+0000 7f591be00640 1 -- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f591c143450 msgr2=0x7f591c1f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.703+0000 7f591be00640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f591c143450 0x7f591c1f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.703+0000 7f591be00640 1 -- 172.21.15.62:0/20302572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5908009400 con 0x7f591c0caff0 2024-03-23T11:31:38.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.704+0000 7f591be00640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f591c0caff0 0x7f591c1e2dd0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f59080096f0 tx=0x7f59080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:38.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.704+0000 7f5918c00640 1 -- 172.21.15.62:0/20302572 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590803bc70 con 0x7f591c0caff0 2024-03-23T11:31:38.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.704+0000 7f5918c00640 1 -- 172.21.15.62:0/20302572 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590803bdf0 con 0x7f591c0caff0 2024-03-23T11:31:38.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.704+0000 7f5922400640 1 -- 172.21.15.62:0/20302572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c1e3a90 con 0x7f591c0caff0 2024-03-23T11:31:38.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.704+0000 7f5918c00640 1 -- 172.21.15.62:0/20302572 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908045630 con 0x7f591c0caff0 2024-03-23T11:31:38.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.704+0000 7f5922400640 1 -- 172.21.15.62:0/20302572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c1f1620 con 0x7f591c0caff0 2024-03-23T11:31:38.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.705+0000 7f5922400640 1 -- 172.21.15.62:0/20302572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591c066bb0 con 0x7f591c0caff0 2024-03-23T11:31:38.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.707+0000 7f5918c00640 1 -- 172.21.15.62:0/20302572 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f590804e450 con 0x7f591c0caff0 2024-03-23T11:31:38.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.707+0000 7f5918c00640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f58fc06a7a0 0x7f58fc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:38.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.708+0000 7f5918c00640 1 -- 172.21.15.62:0/20302572 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f590802e070 con 0x7f591c0caff0 2024-03-23T11:31:38.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.708+0000 7f5921000640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f58fc06a7a0 0x7f58fc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:38.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.709+0000 7f5921000640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f58fc06a7a0 0x7f58fc06cc60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f591c15ad00 tx=0x7f5910007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:38.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.712+0000 7f5918c00640 1 -- 172.21.15.62:0/20302572 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5908076540 con 0x7f591c0caff0 2024-03-23T11:31:38.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.923+0000 7f5922400640 1 -- 172.21.15.62:0/20302572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f591c1e44f0 con 0x7f591c0caff0 2024-03-23T11:31:38.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.924+0000 7f5918c00640 1 -- 172.21.15.62:0/20302572 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v118) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5908006e10 con 0x7f591c0caff0 2024-03-23T11:31:38.915 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:31:38.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.927+0000 7f5922400640 1 -- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f58fc06a7a0 msgr2=0x7f58fc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.927+0000 7f5922400640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f58fc06a7a0 0x7f58fc06cc60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f591c15ad00 tx=0x7f5910007e80 comp rx=0 tx=0).stop 2024-03-23T11:31:38.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.927+0000 7f5922400640 1 -- 172.21.15.62:0/20302572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f591c0caff0 msgr2=0x7f591c1e2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:38.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.927+0000 7f5922400640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f591c0caff0 0x7f591c1e2dd0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f59080096f0 tx=0x7f59080046c0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.928+0000 7f5922400640 1 -- 172.21.15.62:0/20302572 shutdown_connections 2024-03-23T11:31:38.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.928+0000 7f5922400640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f58fc06a7a0 0x7f58fc06cc60 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.928+0000 7f5922400640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f591c0caff0 0x7f591c1e2dd0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.928+0000 7f5922400640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f591c143450 0x7f591c1f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.928+0000 7f5922400640 1 --2- 172.21.15.62:0/20302572 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f591c069c60 0x7f591c1e6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:38.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.928+0000 7f5922400640 1 -- 172.21.15.62:0/20302572 >> 172.21.15.62:0/20302572 conn(0x7f591c0b0090 msgr2=0x7f591c0aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:38.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.928+0000 7f5922400640 1 -- 172.21.15.62:0/20302572 shutdown_connections 2024-03-23T11:31:38.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:38.928+0000 7f5922400640 1 -- 172.21.15.62:0/20302572 wait complete. 2024-03-23T11:31:38.938 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:31:38.939 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:38.991 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:31:39.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.212+0000 7fedaa600640 1 Processor -- start 2024-03-23T11:31:39.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.213+0000 7fedaa600640 1 -- start start 2024-03-23T11:31:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.214+0000 7fedaa600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda40cb380 0x7feda40cb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.214+0000 7fedaa600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feda40b09a0 0x7feda40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.214+0000 7fedaa600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda40cc630 con 0x7feda40cec90 2024-03-23T11:31:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.214+0000 7fedaa600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda40b01e0 con 0x7feda40b09a0 2024-03-23T11:31:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.214+0000 7fedaa600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda40cbca0 con 0x7feda40cb380 2024-03-23T11:31:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.215+0000 7feda2a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7feda40cec90 0x7feda40cbf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57130/0 (socket says 172.21.15.62:57130) 2024-03-23T11:31:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.215+0000 7feda2a00640 1 -- 172.21.15.62:0/4169572290 learned_addr learned my addr 172.21.15.62:0/4169572290 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.215+0000 7feda3e00640 1 --2- 172.21.15.62:0/4169572290 >> v2:172.21.15.120:3300/0 conn(0x7feda40b09a0 0x7feda40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.215+0000 7feda3400640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda40cb380 0x7feda40cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.215+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484087474 0 0) 0x7feda40cc630 con 0x7feda40cec90 2024-03-23T11:31:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.215+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed88003520 con 0x7feda40cec90 2024-03-23T11:31:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3840950424 0 0) 0x7fed88003520 con 0x7feda40cec90 2024-03-23T11:31:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda40cb380 msgr2=0x7feda40cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda40cb380 0x7feda40cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 >> v2:172.21.15.120:3300/0 conn(0x7feda40b09a0 msgr2=0x7feda40b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 --2- 172.21.15.62:0/4169572290 >> v2:172.21.15.120:3300/0 conn(0x7feda40b09a0 0x7feda40b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda40cf710 con 0x7feda40cec90 2024-03-23T11:31:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fed90001670 con 0x7feda40cec90 2024-03-23T11:31:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 >> v1:172.21.15.62:6789/0 conn(0x7feda40cec90 legacy=0x7feda40cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed88003970 0x7fed88003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda40cb380 0x7fed880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed88004d00 0x7fed88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda40b01e0 con 0x7fed88003970 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda40cbca0 con 0x7fed88004d00 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8800b980 con 0x7feda40cb380 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda3e00640 1 --2- 172.21.15.62:0/4169572290 >> v2:172.21.15.120:3300/0 conn(0x7feda40b09a0 0x7feda40b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.216+0000 7feda3400640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed88003970 0x7fed88003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.217+0000 7feda2a00640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda40cb380 0x7fed880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.217+0000 7feda3400640 1 -- 172.21.15.62:0/4169572290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda40cb380 msgr2=0x7fed880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.217+0000 7feda3400640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda40cb380 0x7fed880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.217+0000 7feda3400640 1 -- 172.21.15.62:0/4169572290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed88004d00 msgr2=0x7fed88005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.217+0000 7feda3400640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed88004d00 0x7fed88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.217+0000 7feda3400640 1 -- 172.21.15.62:0/4169572290 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed88003520 con 0x7fed88003970 2024-03-23T11:31:39.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.217+0000 7feda2a00640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda40cb380 0x7fed880047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:39.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.218+0000 7feda3400640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed88003970 0x7fed88003d50 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fed94009ac0 tx=0x7fed880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12dce9ca7f7e8909 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.218+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9402c6b0 con 0x7fed88003970 2024-03-23T11:31:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.218+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9402c810 con 0x7fed88003970 2024-03-23T11:31:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.218+0000 7feda2000640 1 -- 172.21.15.62:0/4169572290 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9402cba0 con 0x7fed88003970 2024-03-23T11:31:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.219+0000 7fedaa600640 1 -- 172.21.15.62:0/4169572290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed88003970 msgr2=0x7fed88003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.219+0000 7fedaa600640 1 --2- 172.21.15.62:0/4169572290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed88003970 0x7fed88003d50 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fed94009ac0 tx=0x7fed880057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.219+0000 7feda3e00640 1 -- 172.21.15.62:0/4169572290 reap_dead start 2024-03-23T11:31:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.219+0000 7fedaa600640 1 -- 172.21.15.62:0/4169572290 shutdown_connections 2024-03-23T11:31:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.219+0000 7fedaa600640 1 -- 172.21.15.62:0/4169572290 >> 172.21.15.62:0/4169572290 conn(0x7feda40b21d0 msgr2=0x7feda40b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.219+0000 7fedaa600640 1 -- 172.21.15.62:0/4169572290 shutdown_connections 2024-03-23T11:31:39.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.219+0000 7fedaa600640 1 -- 172.21.15.62:0/4169572290 wait complete. 2024-03-23T11:31:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.221+0000 7fedaa600640 1 Processor -- start 2024-03-23T11:31:39.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.221+0000 7fedaa600640 1 -- start start 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.222+0000 7fedaa600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed88003970 0x7feda406c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.222+0000 7fedaa600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feda40c3670 0x7feda41f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.222+0000 7fedaa600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda41f3940 0x7feda41e2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.222+0000 7fedaa600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda406cd90 con 0x7feda40c3670 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.222+0000 7fedaa600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda40c97a0 con 0x7fed88003970 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.222+0000 7fedaa600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda40ca5a0 con 0x7feda41f3940 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.222+0000 7feda2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feda40c3670 0x7feda41f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.222+0000 7feda2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feda40c3670 0x7feda41f1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42418/0 (socket says 172.21.15.62:42418) 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.222+0000 7feda2a00640 1 -- 172.21.15.62:0/192676269 learned_addr learned my addr 172.21.15.62:0/192676269 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.223+0000 7feda3400640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed88003970 0x7feda406c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.223+0000 7feda2a00640 1 -- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda41f3940 msgr2=0x7feda41e2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.223+0000 7feda2a00640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda41f3940 0x7feda41e2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.223+0000 7feda2a00640 1 -- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed88003970 msgr2=0x7feda406c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.223+0000 7feda2a00640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed88003970 0x7feda406c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.223+0000 7feda2a00640 1 -- 172.21.15.62:0/192676269 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda40c9a20 con 0x7feda40c3670 2024-03-23T11:31:39.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.223+0000 7feda2a00640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feda40c3670 0x7feda41f1ce0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fed9000e7c0 tx=0x7fed9000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:39.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.223+0000 7fed9b400640 1 -- 172.21.15.62:0/192676269 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed90004dd0 con 0x7feda40c3670 2024-03-23T11:31:39.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.223+0000 7fed9b400640 1 -- 172.21.15.62:0/192676269 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9000eed0 con 0x7feda40c3670 2024-03-23T11:31:39.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.223+0000 7fed9b400640 1 -- 172.21.15.62:0/192676269 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed900083f0 con 0x7feda40c3670 2024-03-23T11:31:39.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.224+0000 7fedaa600640 1 -- 172.21.15.62:0/192676269 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda40c9d00 con 0x7feda40c3670 2024-03-23T11:31:39.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.224+0000 7fedaa600640 1 -- 172.21.15.62:0/192676269 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda40ca2c0 con 0x7feda40c3670 2024-03-23T11:31:39.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.225+0000 7fedaa600640 1 -- 172.21.15.62:0/192676269 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda4066c90 con 0x7feda40c3670 2024-03-23T11:31:39.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.226+0000 7fed9b400640 1 -- 172.21.15.62:0/192676269 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fed90008550 con 0x7feda40c3670 2024-03-23T11:31:39.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.227+0000 7fed9b400640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fed8406a7d0 0x7fed8406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.227+0000 7fed9b400640 1 -- 172.21.15.62:0/192676269 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed90088b40 con 0x7feda40c3670 2024-03-23T11:31:39.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.228+0000 7feda3400640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fed8406a7d0 0x7fed8406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.229+0000 7feda3400640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fed8406a7d0 0x7fed8406cc90 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fed94036000 tx=0x7fed94030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.232+0000 7fed9b400640 1 -- 172.21.15.62:0/192676269 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fed90053ec0 con 0x7feda40c3670 2024-03-23T11:31:39.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.438+0000 7fedaa600640 1 -- 172.21.15.62:0/192676269 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feda4070300 con 0x7feda40c3670 2024-03-23T11:31:39.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.441+0000 7fed9b400640 1 -- 172.21.15.62:0/192676269 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16059 (secure 0 0 0) 0x7fed900574e0 con 0x7feda40c3670 2024-03-23T11:31:39.432 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:39.432 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":118,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:31:25.048209+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:31:39.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 -- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fed8406a7d0 msgr2=0x7fed8406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fed8406a7d0 0x7fed8406cc90 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fed94036000 tx=0x7fed94030000 comp rx=0 tx=0).stop 2024-03-23T11:31:39.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 -- 172.21.15.62:0/192676269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feda40c3670 msgr2=0x7feda41f1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feda40c3670 0x7feda41f1ce0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fed9000e7c0 tx=0x7fed9000ec80 comp rx=0 tx=0).stop 2024-03-23T11:31:39.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 -- 172.21.15.62:0/192676269 shutdown_connections 2024-03-23T11:31:39.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fed8406a7d0 0x7fed8406cc90 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feda40c3670 0x7feda41f1ce0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feda41f3940 0x7feda41e2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 --2- 172.21.15.62:0/192676269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed88003970 0x7feda406c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 -- 172.21.15.62:0/192676269 >> 172.21.15.62:0/192676269 conn(0x7feda40b21d0 msgr2=0x7feda40cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:39.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 -- 172.21.15.62:0/192676269 shutdown_connections 2024-03-23T11:31:39.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.444+0000 7fedaa600640 1 -- 172.21.15.62:0/192676269 wait complete. 2024-03-23T11:31:39.455 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:31:22.127079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:31:23.062387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:31:39.456 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:31:39.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.693+0000 7ff6d8a00640 1 Processor -- start 2024-03-23T11:31:39.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.693+0000 7ff6d8a00640 1 -- start start 2024-03-23T11:31:39.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.695+0000 7ff6d8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6d4160310 0x7ff6d40ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.695+0000 7ff6d8a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6d40ce250 0x7ff6d40ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.695+0000 7ff6d8a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d41608d0 con 0x7ff6d415c830 2024-03-23T11:31:39.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.695+0000 7ff6d8a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d40addb0 con 0x7ff6d40ce250 2024-03-23T11:31:39.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.695+0000 7ff6d8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d415cc10 con 0x7ff6d4160310 2024-03-23T11:31:39.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.695+0000 7ff6d3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6d40ce250 0x7ff6d40ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.695+0000 7ff6d3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6d40ce250 0x7ff6d40ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42420/0 (socket says 172.21.15.62:42420) 2024-03-23T11:31:39.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.695+0000 7ff6d3e00640 1 -- 172.21.15.62:0/240089379 learned_addr learned my addr 172.21.15.62:0/240089379 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:39.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.696+0000 7ff6d2a00640 1 --2- 172.21.15.62:0/240089379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6d4160310 0x7ff6d40ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.696+0000 7ff6d3e00640 1 -- 172.21.15.62:0/240089379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6d4160310 msgr2=0x7ff6d40ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:39.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.696+0000 7ff6d3e00640 1 --2- 172.21.15.62:0/240089379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6d4160310 0x7ff6d40ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.696+0000 7ff6d3e00640 1 -- 172.21.15.62:0/240089379 >> v1:172.21.15.120:6789/0 conn(0x7ff6d415c830 legacy=0x7ff6d40ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.696+0000 7ff6d3e00640 1 -- 172.21.15.62:0/240089379 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d4160bb0 con 0x7ff6d40ce250 2024-03-23T11:31:39.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.697+0000 7ff6d3e00640 1 --2- 172.21.15.62:0/240089379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6d40ce250 0x7ff6d40ce6e0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009e80 tx=0x7ff6cc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e26c863101ab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:39.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.697+0000 7ff6d2000640 1 -- 172.21.15.62:0/240089379 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc02caf0 con 0x7ff6d40ce250 2024-03-23T11:31:39.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.697+0000 7ff6d2000640 1 -- 172.21.15.62:0/240089379 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc02cc50 con 0x7ff6d40ce250 2024-03-23T11:31:39.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.697+0000 7ff6d2000640 1 -- 172.21.15.62:0/240089379 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc0343f0 con 0x7ff6d40ce250 2024-03-23T11:31:39.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.698+0000 7ff6d8a00640 1 -- 172.21.15.62:0/240089379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6d40ce250 msgr2=0x7ff6d40ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.698+0000 7ff6d8a00640 1 --2- 172.21.15.62:0/240089379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6d40ce250 0x7ff6d40ce6e0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009e80 tx=0x7ff6cc02bf50 comp rx=0 tx=0).stop 2024-03-23T11:31:39.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.698+0000 7ff6d8a00640 1 -- 172.21.15.62:0/240089379 shutdown_connections 2024-03-23T11:31:39.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.698+0000 7ff6d8a00640 1 --2- 172.21.15.62:0/240089379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6d40ce250 0x7ff6d40ce6e0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.698+0000 7ff6d8a00640 1 --2- 172.21.15.62:0/240089379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6d4160310 0x7ff6d40ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.698+0000 7ff6d8a00640 1 -- 172.21.15.62:0/240089379 >> 172.21.15.62:0/240089379 conn(0x7ff6d4066050 msgr2=0x7ff6d4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:39.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.698+0000 7ff6d8a00640 1 -- 172.21.15.62:0/240089379 shutdown_connections 2024-03-23T11:31:39.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.698+0000 7ff6d8a00640 1 -- 172.21.15.62:0/240089379 wait complete. 2024-03-23T11:31:39.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.700+0000 7ff6d8a00640 1 Processor -- start 2024-03-23T11:31:39.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.700+0000 7ff6d8a00640 1 -- start start 2024-03-23T11:31:39.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.701+0000 7ff6d8a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6d40ce250 0x7ff6d4142ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.701+0000 7ff6d8a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d415c830 0x7ff6d40ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.701+0000 7ff6d8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6d4160310 0x7ff6d40cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.701+0000 7ff6d8a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d40cb9f0 con 0x7ff6d40ce250 2024-03-23T11:31:39.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.701+0000 7ff6d8a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d40cbcd0 con 0x7ff6d415c830 2024-03-23T11:31:39.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.701+0000 7ff6d8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d40cbfb0 con 0x7ff6d4160310 2024-03-23T11:31:39.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.701+0000 7ff6d3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6d40ce250 0x7ff6d4142ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.701+0000 7ff6d3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6d40ce250 0x7ff6d4142ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42424/0 (socket says 172.21.15.62:42424) 2024-03-23T11:31:39.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.701+0000 7ff6d3400640 1 -- 172.21.15.62:0/2187957755 learned_addr learned my addr 172.21.15.62:0/2187957755 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:39.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6d2a00640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d415c830 0x7ff6d40ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6d3e00640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6d4160310 0x7ff6d40cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6d3400640 1 -- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6d4160310 msgr2=0x7ff6d40cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6d3400640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6d4160310 0x7ff6d40cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6d3400640 1 -- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d415c830 msgr2=0x7ff6d40ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6d3400640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d415c830 0x7ff6d40ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6d3400640 1 -- 172.21.15.62:0/2187957755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc009b30 con 0x7ff6d40ce250 2024-03-23T11:31:39.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6d3e00640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6d4160310 0x7ff6d40cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:39.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6d3400640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6d40ce250 0x7ff6d4142ee0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400c400 tx=0x7ff6c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:39.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6cb400640 1 -- 172.21.15.62:0/2187957755 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c40052d0 con 0x7ff6d40ce250 2024-03-23T11:31:39.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6cb400640 1 -- 172.21.15.62:0/2187957755 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c4005430 con 0x7ff6d40ce250 2024-03-23T11:31:39.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6cb400640 1 -- 172.21.15.62:0/2187957755 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4003830 con 0x7ff6d40ce250 2024-03-23T11:31:39.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.702+0000 7ff6d8a00640 1 -- 172.21.15.62:0/2187957755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d40cc290 con 0x7ff6d40ce250 2024-03-23T11:31:39.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.703+0000 7ff6d8a00640 1 -- 172.21.15.62:0/2187957755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d4143420 con 0x7ff6d40ce250 2024-03-23T11:31:39.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.704+0000 7ff6d8a00640 1 -- 172.21.15.62:0/2187957755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d40cc7b0 con 0x7ff6d40ce250 2024-03-23T11:31:39.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.705+0000 7ff6cb400640 1 -- 172.21.15.62:0/2187957755 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff6c4003990 con 0x7ff6d40ce250 2024-03-23T11:31:39.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.706+0000 7ff6cb400640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6b406a7a0 0x7ff6b406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:39.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.706+0000 7ff6cb400640 1 -- 172.21.15.62:0/2187957755 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6c4088310 con 0x7ff6d40ce250 2024-03-23T11:31:39.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.706+0000 7ff6d2a00640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6b406a7a0 0x7ff6b406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:39.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.708+0000 7ff6d2a00640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6b406a7a0 0x7ff6b406cc60 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff6d406ddb0 tx=0x7ff6c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:39.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.710+0000 7ff6cb400640 1 -- 172.21.15.62:0/2187957755 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6d40cc7b0 con 0x7ff6d40ce250 2024-03-23T11:31:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.913+0000 7ff6d8a00640 1 -- 172.21.15.62:0/2187957755 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff6d40cc7b0 con 0x7ff6d40ce250 2024-03-23T11:31:39.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.914+0000 7ff6cb400640 1 -- 172.21.15.62:0/2187957755 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff6d40cc7b0 con 0x7ff6d40ce250 2024-03-23T11:31:39.905 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:31:39.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.917+0000 7ff6d8a00640 1 -- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6b406a7a0 msgr2=0x7ff6b406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.917+0000 7ff6d8a00640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6b406a7a0 0x7ff6b406cc60 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff6d406ddb0 tx=0x7ff6c0008040 comp rx=0 tx=0).stop 2024-03-23T11:31:39.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.917+0000 7ff6d8a00640 1 -- 172.21.15.62:0/2187957755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6d40ce250 msgr2=0x7ff6d4142ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:39.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.917+0000 7ff6d8a00640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6d40ce250 0x7ff6d4142ee0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400c400 tx=0x7ff6c400c8c0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.917+0000 7ff6d8a00640 1 -- 172.21.15.62:0/2187957755 shutdown_connections 2024-03-23T11:31:39.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.917+0000 7ff6d8a00640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff6b406a7a0 0x7ff6b406cc60 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.917+0000 7ff6d8a00640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6d4160310 0x7ff6d40cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.917+0000 7ff6d8a00640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6d415c830 0x7ff6d40ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.917+0000 7ff6d8a00640 1 --2- 172.21.15.62:0/2187957755 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6d40ce250 0x7ff6d4142ee0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:39.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.917+0000 7ff6d8a00640 1 -- 172.21.15.62:0/2187957755 >> 172.21.15.62:0/2187957755 conn(0x7ff6d4066050 msgr2=0x7ff6d40adf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:39.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.917+0000 7ff6d8a00640 1 -- 172.21.15.62:0/2187957755 shutdown_connections 2024-03-23T11:31:39.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:39.918+0000 7ff6d8a00640 1 -- 172.21.15.62:0/2187957755 wait complete. 2024-03-23T11:31:39.928 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:31:40.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.164+0000 7f09a1c00640 1 Processor -- start 2024-03-23T11:31:40.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.165+0000 7f09a1c00640 1 -- start start 2024-03-23T11:31:40.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.166+0000 7f09a1c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f099c15c9e0 0x7f099c1607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.166+0000 7f09a1c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f099c15cdc0 0x7f099c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.166+0000 7f09a1c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099c06bfc0 con 0x7f099c0cf650 2024-03-23T11:31:40.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.166+0000 7f09a1c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f099c0ce140 con 0x7f099c15cdc0 2024-03-23T11:31:40.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.166+0000 7f09a1c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f099c0b4370 con 0x7f099c15c9e0 2024-03-23T11:31:40.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.166+0000 7f099b400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f099c0cf650 0x7f099c15d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57134/0 (socket says 172.21.15.62:57134) 2024-03-23T11:31:40.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.166+0000 7f099b400640 1 -- 172.21.15.62:0/626326550 learned_addr learned my addr 172.21.15.62:0/626326550 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:40.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.167+0000 7f099a000640 1 --2- 172.21.15.62:0/626326550 >> v2:172.21.15.62:3300/0 conn(0x7f099c15cdc0 0x7f099c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.167+0000 7f099aa00640 1 --2- 172.21.15.62:0/626326550 >> v2:172.21.15.120:3300/0 conn(0x7f099c15c9e0 0x7f099c1607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.167+0000 7f0999600640 1 -- 172.21.15.62:0/626326550 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256043329 0 0) 0x7f099c06bfc0 con 0x7f099c0cf650 2024-03-23T11:31:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.167+0000 7f0999600640 1 -- 172.21.15.62:0/626326550 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0980003520 con 0x7f099c0cf650 2024-03-23T11:31:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.167+0000 7f099a000640 1 -- 172.21.15.62:0/626326550 >> v2:172.21.15.120:3300/0 conn(0x7f099c15c9e0 msgr2=0x7f099c1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.167+0000 7f099a000640 1 --2- 172.21.15.62:0/626326550 >> v2:172.21.15.120:3300/0 conn(0x7f099c15c9e0 0x7f099c1607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.167+0000 7f099a000640 1 -- 172.21.15.62:0/626326550 >> v1:172.21.15.62:6789/0 conn(0x7f099c0cf650 legacy=0x7f099c15d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.167+0000 7f099a000640 1 -- 172.21.15.62:0/626326550 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c0b5260 con 0x7f099c15cdc0 2024-03-23T11:31:40.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.168+0000 7f099a000640 1 --2- 172.21.15.62:0/626326550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f099c15cdc0 0x7f099c0d1f20 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f099c0d0380 tx=0x7f098802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d25aa1b12f05289 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:40.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.169+0000 7f0999600640 1 -- 172.21.15.62:0/626326550 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988031830 con 0x7f099c15cdc0 2024-03-23T11:31:40.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.169+0000 7f0999600640 1 -- 172.21.15.62:0/626326550 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0988031990 con 0x7f099c15cdc0 2024-03-23T11:31:40.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.169+0000 7f0999600640 1 -- 172.21.15.62:0/626326550 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988031d20 con 0x7f099c15cdc0 2024-03-23T11:31:40.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.169+0000 7f09a1c00640 1 -- 172.21.15.62:0/626326550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f099c15cdc0 msgr2=0x7f099c0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.169+0000 7f09a1c00640 1 --2- 172.21.15.62:0/626326550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f099c15cdc0 0x7f099c0d1f20 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f099c0d0380 tx=0x7f098802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.169+0000 7f09a1c00640 1 -- 172.21.15.62:0/626326550 shutdown_connections 2024-03-23T11:31:40.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.169+0000 7f09a1c00640 1 --2- 172.21.15.62:0/626326550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f099c15cdc0 0x7f099c0d1f20 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.169+0000 7f09a1c00640 1 --2- 172.21.15.62:0/626326550 >> v2:172.21.15.120:3300/0 conn(0x7f099c15c9e0 0x7f099c1607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.169+0000 7f09a1c00640 1 -- 172.21.15.62:0/626326550 >> 172.21.15.62:0/626326550 conn(0x7f099c071570 msgr2=0x7f099c071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:40.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.170+0000 7f09a1c00640 1 -- 172.21.15.62:0/626326550 shutdown_connections 2024-03-23T11:31:40.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.170+0000 7f09a1c00640 1 -- 172.21.15.62:0/626326550 wait complete. 2024-03-23T11:31:40.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.171+0000 7f09a1c00640 1 Processor -- start 2024-03-23T11:31:40.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.171+0000 7f09a1c00640 1 -- start start 2024-03-23T11:31:40.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.172+0000 7f09a1c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c0cf650 0x7f099c15fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.172+0000 7f09a1c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f099c15c9e0 0x7f099c15a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.172+0000 7f09a1c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f099c15cdc0 0x7f099c1f4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.172+0000 7f09a1c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099c0a96c0 con 0x7f099c15cdc0 2024-03-23T11:31:40.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.172+0000 7f09a1c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099c0a99a0 con 0x7f099c0cf650 2024-03-23T11:31:40.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.172+0000 7f09a1c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099c0a9c80 con 0x7f099c15c9e0 2024-03-23T11:31:40.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.173+0000 7f099b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f099c15cdc0 0x7f099c1f4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.173+0000 7f099b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f099c15cdc0 0x7f099c1f4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42444/0 (socket says 172.21.15.62:42444) 2024-03-23T11:31:40.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.173+0000 7f099b400640 1 -- 172.21.15.62:0/2267743524 learned_addr learned my addr 172.21.15.62:0/2267743524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:40.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.173+0000 7f099aa00640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c0cf650 0x7f099c15fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.173+0000 7f099a000640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f099c15c9e0 0x7f099c15a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.173+0000 7f099b400640 1 -- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f099c15c9e0 msgr2=0x7f099c15a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.173+0000 7f099b400640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f099c15c9e0 0x7f099c15a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.173+0000 7f099b400640 1 -- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c0cf650 msgr2=0x7f099c15fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.173+0000 7f099b400640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c0cf650 0x7f099c15fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.173+0000 7f099b400640 1 -- 172.21.15.62:0/2267743524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0988009400 con 0x7f099c15cdc0 2024-03-23T11:31:40.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.174+0000 7f099b400640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f099c15cdc0 0x7f099c1f4980 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f099400cb40 tx=0x7f099400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:40.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.174+0000 7f0992a00640 1 -- 172.21.15.62:0/2267743524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099400aca0 con 0x7f099c15cdc0 2024-03-23T11:31:40.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.174+0000 7f0992a00640 1 -- 172.21.15.62:0/2267743524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099400ae00 con 0x7f099c15cdc0 2024-03-23T11:31:40.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.174+0000 7f0992a00640 1 -- 172.21.15.62:0/2267743524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0994012620 con 0x7f099c15cdc0 2024-03-23T11:31:40.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.174+0000 7f09a1c00640 1 -- 172.21.15.62:0/2267743524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099c0a9f60 con 0x7f099c15cdc0 2024-03-23T11:31:40.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.174+0000 7f09a1c00640 1 -- 172.21.15.62:0/2267743524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099c0aa760 con 0x7f099c15cdc0 2024-03-23T11:31:40.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.175+0000 7f09a1c00640 1 -- 172.21.15.62:0/2267743524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f099c066c90 con 0x7f099c15cdc0 2024-03-23T11:31:40.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.176+0000 7f0992a00640 1 -- 172.21.15.62:0/2267743524 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0994007070 con 0x7f099c15cdc0 2024-03-23T11:31:40.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.177+0000 7f0992a00640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f097c06a7a0 0x7f097c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.177+0000 7f0992a00640 1 -- 172.21.15.62:0/2267743524 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0994088eb0 con 0x7f099c15cdc0 2024-03-23T11:31:40.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.178+0000 7f099aa00640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f097c06a7a0 0x7f097c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.179+0000 7f099aa00640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f097c06a7a0 0x7f097c06cc60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f098c006fd0 tx=0x7f098c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:40.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.183+0000 7f0992a00640 1 -- 172.21.15.62:0/2267743524 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0994054230 con 0x7f099c15cdc0 2024-03-23T11:31:40.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.383+0000 7f09a1c00640 1 -- 172.21.15.62:0/2267743524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f099c06b720 con 0x7f099c15cdc0 2024-03-23T11:31:40.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.384+0000 7f0992a00640 1 -- 172.21.15.62:0/2267743524 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0994057850 con 0x7f099c15cdc0 2024-03-23T11:31:40.375 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:31:40.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.386+0000 7f09a1c00640 1 -- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f097c06a7a0 msgr2=0x7f097c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.386+0000 7f09a1c00640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f097c06a7a0 0x7f097c06cc60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f098c006fd0 tx=0x7f098c008040 comp rx=0 tx=0).stop 2024-03-23T11:31:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.386+0000 7f09a1c00640 1 -- 172.21.15.62:0/2267743524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f099c15cdc0 msgr2=0x7f099c1f4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.386+0000 7f09a1c00640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f099c15cdc0 0x7f099c1f4980 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f099400cb40 tx=0x7f099400a060 comp rx=0 tx=0).stop 2024-03-23T11:31:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.386+0000 7f09a1c00640 1 -- 172.21.15.62:0/2267743524 shutdown_connections 2024-03-23T11:31:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.386+0000 7f09a1c00640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f097c06a7a0 0x7f097c06cc60 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.386+0000 7f09a1c00640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f099c15cdc0 0x7f099c1f4980 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.386+0000 7f09a1c00640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f099c15c9e0 0x7f099c15a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.386+0000 7f09a1c00640 1 --2- 172.21.15.62:0/2267743524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c0cf650 0x7f099c15fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.386+0000 7f09a1c00640 1 -- 172.21.15.62:0/2267743524 >> 172.21.15.62:0/2267743524 conn(0x7f099c071570 msgr2=0x7f099c15e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.387+0000 7f09a1c00640 1 -- 172.21.15.62:0/2267743524 shutdown_connections 2024-03-23T11:31:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.387+0000 7f09a1c00640 1 -- 172.21.15.62:0/2267743524 wait complete. 2024-03-23T11:31:40.398 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:31:40.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.632+0000 7f6dce800640 1 Processor -- start 2024-03-23T11:31:40.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.633+0000 7f6dce800640 1 -- start start 2024-03-23T11:31:40.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.634+0000 7f6dce800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6dc815c9e0 0x7f6dc81607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.634+0000 7f6dce800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc806bfc0 con 0x7f6dc815cdc0 2024-03-23T11:31:40.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.634+0000 7f6dce800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc80b5260 con 0x7f6dc80cf650 2024-03-23T11:31:40.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.634+0000 7f6dce800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dc80d2630 con 0x7f6dc815c9e0 2024-03-23T11:31:40.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.634+0000 7f6dc6a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6dc815cdc0 0x7f6dc80d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57150/0 (socket says 172.21.15.62:57150) 2024-03-23T11:31:40.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.634+0000 7f6dc6a00640 1 -- 172.21.15.62:0/560011510 learned_addr learned my addr 172.21.15.62:0/560011510 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:40.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc7400640 1 --2- 172.21.15.62:0/560011510 >> v2:172.21.15.120:3300/0 conn(0x7f6dc815c9e0 0x7f6dc81607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49382527 0 0) 0x7f6dc806bfc0 con 0x7f6dc815cdc0 2024-03-23T11:31:40.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dac003520 con 0x7f6dc815cdc0 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1197603061 0 0) 0x7f6dac003520 con 0x7f6dc815cdc0 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 >> v2:172.21.15.120:3300/0 conn(0x7f6dc815c9e0 msgr2=0x7f6dc81607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 --2- 172.21.15.62:0/560011510 >> v2:172.21.15.120:3300/0 conn(0x7f6dc815c9e0 0x7f6dc81607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 >> v1:172.21.15.120:6789/0 conn(0x7f6dc80cf650 legacy=0x7f6dc815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc80ce140 con 0x7f6dc815cdc0 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f6db0001670 con 0x7f6dc815cdc0 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 >> v1:172.21.15.62:6789/0 conn(0x7f6dc815cdc0 legacy=0x7f6dc80d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 --2- 172.21.15.62:0/560011510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dac0039c0 0x7f6dac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 --2- 172.21.15.62:0/560011510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dac0047c0 0x7f6dac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 --2- 172.21.15.62:0/560011510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dac005180 0x7f6dac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc80d2630 con 0x7f6dac0047c0 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac00c080 con 0x7f6dac005180 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.635+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac00c240 con 0x7f6dac0039c0 2024-03-23T11:31:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.636+0000 7f6dc6a00640 1 --2- 172.21.15.62:0/560011510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dac0047c0 0x7f6dac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.636+0000 7f6dc6a00640 1 -- 172.21.15.62:0/560011510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dac0039c0 msgr2=0x7f6dac003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.636+0000 7f6dc6a00640 1 --2- 172.21.15.62:0/560011510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dac0039c0 0x7f6dac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.636+0000 7f6dc6a00640 1 -- 172.21.15.62:0/560011510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dac005180 msgr2=0x7f6dac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.636+0000 7f6dc6a00640 1 --2- 172.21.15.62:0/560011510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dac005180 0x7f6dac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.636+0000 7f6dc6a00640 1 -- 172.21.15.62:0/560011510 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac003520 con 0x7f6dac0047c0 2024-03-23T11:31:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.636+0000 7f6dc7e00640 1 -- 172.21.15.62:0/560011510 reap_dead start 2024-03-23T11:31:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.637+0000 7f6dc6a00640 1 --2- 172.21.15.62:0/560011510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dac0047c0 0x7f6dac004c40 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f6db000c0e0 tx=0x7f6dac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9d2ee7aa710783 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.637+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db0005ad0 con 0x7f6dac0047c0 2024-03-23T11:31:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.637+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db0005c30 con 0x7f6dac0047c0 2024-03-23T11:31:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.637+0000 7f6dc6000640 1 -- 172.21.15.62:0/560011510 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db002ca50 con 0x7f6dac0047c0 2024-03-23T11:31:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.638+0000 7f6dce800640 1 -- 172.21.15.62:0/560011510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dac0047c0 msgr2=0x7f6dac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.638+0000 7f6dce800640 1 --2- 172.21.15.62:0/560011510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dac0047c0 0x7f6dac004c40 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f6db000c0e0 tx=0x7f6dac0057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.638+0000 7f6dce800640 1 -- 172.21.15.62:0/560011510 shutdown_connections 2024-03-23T11:31:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.638+0000 7f6dce800640 1 --2- 172.21.15.62:0/560011510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dac0047c0 0x7f6dac004c40 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.638+0000 7f6dce800640 1 -- 172.21.15.62:0/560011510 >> 172.21.15.62:0/560011510 conn(0x7f6dc8071570 msgr2=0x7f6dc8071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.638+0000 7f6dce800640 1 -- 172.21.15.62:0/560011510 shutdown_connections 2024-03-23T11:31:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.638+0000 7f6dce800640 1 -- 172.21.15.62:0/560011510 wait complete. 2024-03-23T11:31:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.640+0000 7f6dce800640 1 Processor -- start 2024-03-23T11:31:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.640+0000 7f6dce800640 1 -- start start 2024-03-23T11:31:40.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.641+0000 7f6dce800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dac0047c0 0x7f6dc80ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.641+0000 7f6dce800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc81f2500 0x7f6dc80cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.641+0000 7f6dce800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dc80cba10 0x7f6dc815d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.641+0000 7f6dce800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc80c64c0 con 0x7f6dc81f2500 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.641+0000 7f6dce800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc80c6760 con 0x7f6dac0047c0 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.641+0000 7f6dce800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc80c6a40 con 0x7f6dc80cba10 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.641+0000 7f6dc6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc81f2500 0x7f6dc80cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.641+0000 7f6dc6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc81f2500 0x7f6dc80cb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42450/0 (socket says 172.21.15.62:42450) 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.641+0000 7f6dc6a00640 1 -- 172.21.15.62:0/4107803659 learned_addr learned my addr 172.21.15.62:0/4107803659 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dc6a00640 1 -- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dc80cba10 msgr2=0x7f6dc815d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dc7400640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dac0047c0 0x7f6dc80ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dc7e00640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dc80cba10 0x7f6dc815d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dc6a00640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dc80cba10 0x7f6dc815d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dc6a00640 1 -- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dac0047c0 msgr2=0x7f6dc80ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dc6a00640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dac0047c0 0x7f6dc80ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dc6a00640 1 -- 172.21.15.62:0/4107803659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc80c6d20 con 0x7f6dc81f2500 2024-03-23T11:31:40.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dc7400640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dac0047c0 0x7f6dc80ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:40.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dc6a00640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc81f2500 0x7f6dc80cb4d0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6db0000c00 tx=0x7f6db002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:40.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dbf400640 1 -- 172.21.15.62:0/4107803659 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db003c040 con 0x7f6dc81f2500 2024-03-23T11:31:40.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dbf400640 1 -- 172.21.15.62:0/4107803659 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db002cd00 con 0x7f6dc81f2500 2024-03-23T11:31:40.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.642+0000 7f6dbf400640 1 -- 172.21.15.62:0/4107803659 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db00072b0 con 0x7f6dc81f2500 2024-03-23T11:31:40.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.643+0000 7f6dce800640 1 -- 172.21.15.62:0/4107803659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc80c7000 con 0x7f6dc81f2500 2024-03-23T11:31:40.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.643+0000 7f6dce800640 1 -- 172.21.15.62:0/4107803659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc815d980 con 0x7f6dc81f2500 2024-03-23T11:31:40.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.644+0000 7f6dce800640 1 -- 172.21.15.62:0/4107803659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc80c6b80 con 0x7f6dc81f2500 2024-03-23T11:31:40.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.645+0000 7f6dbf400640 1 -- 172.21.15.62:0/4107803659 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6db0032070 con 0x7f6dc81f2500 2024-03-23T11:31:40.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.645+0000 7f6dbf400640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6da406a7d0 0x7f6da406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:40.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.646+0000 7f6dbf400640 1 -- 172.21.15.62:0/4107803659 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6db00aa310 con 0x7f6dc81f2500 2024-03-23T11:31:40.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.646+0000 7f6dc7400640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6da406a7d0 0x7f6da406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:40.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.647+0000 7f6dc7400640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6da406a7d0 0x7f6da406cc90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6db8007520 tx=0x7f6db800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:40.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.650+0000 7f6dbf400640 1 -- 172.21.15.62:0/4107803659 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6dc80c6b80 con 0x7f6dc81f2500 2024-03-23T11:31:40.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.846+0000 7f6dce800640 1 -- 172.21.15.62:0/4107803659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6dc80c6b80 con 0x7f6dc81f2500 2024-03-23T11:31:40.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.847+0000 7f6dbf400640 1 -- 172.21.15.62:0/4107803659 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6dc80c6b80 con 0x7f6dc81f2500 2024-03-23T11:31:40.838 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:31:40.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 -- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6da406a7d0 msgr2=0x7f6da406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6da406a7d0 0x7f6da406cc90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6db8007520 tx=0x7f6db800b040 comp rx=0 tx=0).stop 2024-03-23T11:31:40.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 -- 172.21.15.62:0/4107803659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc81f2500 msgr2=0x7f6dc80cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:40.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc81f2500 0x7f6dc80cb4d0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6db0000c00 tx=0x7f6db002e070 comp rx=0 tx=0).stop 2024-03-23T11:31:40.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 -- 172.21.15.62:0/4107803659 shutdown_connections 2024-03-23T11:31:40.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6da406a7d0 0x7f6da406cc90 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc81f2500 0x7f6dc80cb4d0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6dc80cba10 0x7f6dc815d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 --2- 172.21.15.62:0/4107803659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6dac0047c0 0x7f6dc80ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:40.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 -- 172.21.15.62:0/4107803659 >> 172.21.15.62:0/4107803659 conn(0x7f6dc8071570 msgr2=0x7f6dc80b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:40.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 -- 172.21.15.62:0/4107803659 shutdown_connections 2024-03-23T11:31:40.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:40.850+0000 7f6dce800640 1 -- 172.21.15.62:0/4107803659 wait complete. 2024-03-23T11:31:40.861 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:31:41.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.097+0000 7f0765a00640 1 Processor -- start 2024-03-23T11:31:41.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.097+0000 7f0765a00640 1 -- start start 2024-03-23T11:31:41.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.099+0000 7f0765a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0760069c60 0x7f076006a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.099+0000 7f0765a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076015bb10 con 0x7f07600caff0 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.099+0000 7f0765a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076006a580 con 0x7f0760143450 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.099+0000 7f0765a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07600ce1a0 con 0x7f0760069c60 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.099+0000 7f0757e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f07600caff0 0x7f07600cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:57166/0 (socket says 172.21.15.62:57166) 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.099+0000 7f0757e00640 1 -- 172.21.15.62:0/2644100151 learned_addr learned my addr 172.21.15.62:0/2644100151 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.099+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918902636 0 0) 0x7f076015bb10 con 0x7f07600caff0 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.099+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0744003520 con 0x7f07600caff0 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.099+0000 7f075fe00640 1 --2- 172.21.15.62:0/2644100151 >> v2:172.21.15.120:3300/0 conn(0x7f0760069c60 0x7f076006a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.100+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3788876891 0 0) 0x7f0744003520 con 0x7f07600caff0 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.100+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 >> v2:172.21.15.120:3300/0 conn(0x7f0760069c60 msgr2=0x7f076006a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.100+0000 7f075f400640 1 --2- 172.21.15.62:0/2644100151 >> v2:172.21.15.120:3300/0 conn(0x7f0760069c60 0x7f076006a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.100+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 >> v1:172.21.15.120:6789/0 conn(0x7f0760143450 legacy=0x7f07601438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:41.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.100+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076015ad50 con 0x7f07600caff0 2024-03-23T11:31:41.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f0748001670 con 0x7f07600caff0 2024-03-23T11:31:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 >> v1:172.21.15.62:6789/0 conn(0x7f07600caff0 legacy=0x7f07600cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f075f400640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0744003970 0x7f0744003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f075f400640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07440047c0 0x7f0744004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f075f400640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0744005180 0x7f074400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07600ce1a0 con 0x7f07440047c0 2024-03-23T11:31:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074400c0d0 con 0x7f0744005180 2024-03-23T11:31:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074400c290 con 0x7f0744003970 2024-03-23T11:31:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f0757e00640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07440047c0 0x7f0744004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f075fe00640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0744003970 0x7f0744003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f0765000640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0744005180 0x7f074400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.101+0000 7f0757e00640 1 -- 172.21.15.62:0/2644100151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0744003970 msgr2=0x7f0744003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.102+0000 7f0757e00640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0744003970 0x7f0744003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.102+0000 7f0757e00640 1 -- 172.21.15.62:0/2644100151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0744005180 msgr2=0x7f074400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.102+0000 7f0757e00640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0744005180 0x7f074400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.102+0000 7f0757e00640 1 -- 172.21.15.62:0/2644100151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0744003520 con 0x7f07440047c0 2024-03-23T11:31:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.102+0000 7f0765000640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0744005180 0x7f074400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.102+0000 7f0765000640 1 -- 172.21.15.62:0/2644100151 reap_dead start 2024-03-23T11:31:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.102+0000 7f075fe00640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0744003970 0x7f0744003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.103+0000 7f0757e00640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07440047c0 0x7f0744004c40 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f074800c1b0 tx=0x7f07440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2efcb0d4e3c13ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.103+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0748005ad0 con 0x7f07440047c0 2024-03-23T11:31:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.103+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0748005c30 con 0x7f07440047c0 2024-03-23T11:31:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.103+0000 7f075f400640 1 -- 172.21.15.62:0/2644100151 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074802c870 con 0x7f07440047c0 2024-03-23T11:31:41.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.104+0000 7f0765a00640 1 -- 172.21.15.62:0/2644100151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07440047c0 msgr2=0x7f0744004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.104+0000 7f0765a00640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07440047c0 0x7f0744004c40 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f074800c1b0 tx=0x7f07440057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.104+0000 7f0765a00640 1 -- 172.21.15.62:0/2644100151 shutdown_connections 2024-03-23T11:31:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.104+0000 7f0765a00640 1 --2- 172.21.15.62:0/2644100151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07440047c0 0x7f0744004c40 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.104+0000 7f0765a00640 1 -- 172.21.15.62:0/2644100151 >> 172.21.15.62:0/2644100151 conn(0x7f07600b0090 msgr2=0x7f07600b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.104+0000 7f0765a00640 1 -- 172.21.15.62:0/2644100151 shutdown_connections 2024-03-23T11:31:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.104+0000 7f0765a00640 1 -- 172.21.15.62:0/2644100151 wait complete. 2024-03-23T11:31:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.106+0000 7f0765a00640 1 Processor -- start 2024-03-23T11:31:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.106+0000 7f0765a00640 1 -- start start 2024-03-23T11:31:41.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07440047c0 0x7f07601428e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:41.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07601f2ca0 0x7f076015ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:41.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015d380 0x7f07601f5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07600cbcb0 con 0x7f076015d380 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076015d800 con 0x7f07601f2ca0 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07601f6540 con 0x7f07440047c0 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015d380 0x7f07601f5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015d380 0x7f07601f5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42468/0 (socket says 172.21.15.62:42468) 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765000640 1 -- 172.21.15.62:0/2530189236 learned_addr learned my addr 172.21.15.62:0/2530189236 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765000640 1 -- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07440047c0 msgr2=0x7f07601428e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765000640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07440047c0 0x7f07601428e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0757e00640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07601f2ca0 0x7f076015ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.107+0000 7f0765000640 1 -- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07601f2ca0 msgr2=0x7f076015ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.108+0000 7f0765000640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07601f2ca0 0x7f076015ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.108+0000 7f0765000640 1 -- 172.21.15.62:0/2530189236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07601f6260 con 0x7f076015d380 2024-03-23T11:31:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.108+0000 7f0757e00640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07601f2ca0 0x7f076015ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:41.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.108+0000 7f0765000640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015d380 0x7f07601f5610 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f076006a9a0 tx=0x7f075800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:41.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.108+0000 7f075cc00640 1 -- 172.21.15.62:0/2530189236 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0758017070 con 0x7f076015d380 2024-03-23T11:31:41.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.108+0000 7f075cc00640 1 -- 172.21.15.62:0/2530189236 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0758003b80 con 0x7f076015d380 2024-03-23T11:31:41.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.108+0000 7f075cc00640 1 -- 172.21.15.62:0/2530189236 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0758006b50 con 0x7f076015d380 2024-03-23T11:31:41.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.108+0000 7f0765a00640 1 -- 172.21.15.62:0/2530189236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07601f21c0 con 0x7f076015d380 2024-03-23T11:31:41.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.108+0000 7f0765a00640 1 -- 172.21.15.62:0/2530189236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07601f2780 con 0x7f076015d380 2024-03-23T11:31:41.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.109+0000 7f0765a00640 1 -- 172.21.15.62:0/2530189236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07601f2020 con 0x7f076015d380 2024-03-23T11:31:41.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.110+0000 7f075cc00640 1 -- 172.21.15.62:0/2530189236 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0758006cb0 con 0x7f076015d380 2024-03-23T11:31:41.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.111+0000 7f075cc00640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f073c06a7a0 0x7f073c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:41.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.111+0000 7f075cc00640 1 -- 172.21.15.62:0/2530189236 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0758088530 con 0x7f076015d380 2024-03-23T11:31:41.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.112+0000 7f075fe00640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f073c06a7a0 0x7f073c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:41.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.113+0000 7f075fe00640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f073c06a7a0 0x7f073c06cc60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0760067260 tx=0x7f0750009340 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:41.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.116+0000 7f075cc00640 1 -- 172.21.15.62:0/2530189236 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f07601f2020 con 0x7f076015d380 2024-03-23T11:31:41.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.326+0000 7f0765a00640 1 -- 172.21.15.62:0/2530189236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f07601f2020 con 0x7f076015d380 2024-03-23T11:31:41.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.326+0000 7f075cc00640 1 -- 172.21.15.62:0/2530189236 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v118) v1 ==== 80+0+0 (secure 0 0 0) 0x7f07601f2020 con 0x7f076015d380 2024-03-23T11:31:41.317 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:31:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.329+0000 7f0765a00640 1 -- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f073c06a7a0 msgr2=0x7f073c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.329+0000 7f0765a00640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f073c06a7a0 0x7f073c06cc60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0760067260 tx=0x7f0750009340 comp rx=0 tx=0).stop 2024-03-23T11:31:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.329+0000 7f0765a00640 1 -- 172.21.15.62:0/2530189236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015d380 msgr2=0x7f07601f5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.329+0000 7f0765a00640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015d380 0x7f07601f5610 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f076006a9a0 tx=0x7f075800d9d0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.329+0000 7f0765a00640 1 -- 172.21.15.62:0/2530189236 shutdown_connections 2024-03-23T11:31:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.329+0000 7f0765a00640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f073c06a7a0 0x7f073c06cc60 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.329+0000 7f0765a00640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f076015d380 0x7f07601f5610 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.329+0000 7f0765a00640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07601f2ca0 0x7f076015ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.329+0000 7f0765a00640 1 --2- 172.21.15.62:0/2530189236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f07440047c0 0x7f07601428e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.329+0000 7f0765a00640 1 -- 172.21.15.62:0/2530189236 >> 172.21.15.62:0/2530189236 conn(0x7f07600b0090 msgr2=0x7f0760070790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.329+0000 7f0765a00640 1 -- 172.21.15.62:0/2530189236 shutdown_connections 2024-03-23T11:31:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:41.330+0000 7f0765a00640 1 -- 172.21.15.62:0/2530189236 wait complete. 2024-03-23T11:31:41.338 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:31:41.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:41.338 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:41.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[77759]: fuse finished with error 0 and tester_r 0 2024-03-23T11:31:41.428 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:31:47.529 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:31:47.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:47.530 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:31:47.592 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-03-23T11:31:47.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.822+0000 7fd6ace00640 1 Processor -- start 2024-03-23T11:31:47.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.822+0000 7fd6ace00640 1 -- start start 2024-03-23T11:31:47.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.823+0000 7fd6ace00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6a8142c80 0x7fd6a81603b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.823+0000 7fd6ace00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd6a81608f0 0x7fd6a80ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6ace00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6a80adb10 0x7fd6a806c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6ace00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a8160e50 con 0x7fd6a80adb10 2024-03-23T11:31:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6ace00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a8161130 con 0x7fd6a81608f0 2024-03-23T11:31:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6ace00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a80acdd0 con 0x7fd6a8142c80 2024-03-23T11:31:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6a7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6a80adb10 0x7fd6a806c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6a7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6a80adb10 0x7fd6a806c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48154/0 (socket says 172.21.15.62:48154) 2024-03-23T11:31:47.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6a7e00640 1 -- 172.21.15.62:0/463998646 learned_addr learned my addr 172.21.15.62:0/463998646 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/463998646 >> v2:172.21.15.120:3300/0 conn(0x7fd6a81608f0 0x7fd6a80ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6a7400640 1 --2- 172.21.15.62:0/463998646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6a8142c80 0x7fd6a81603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6a7e00640 1 -- 172.21.15.62:0/463998646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6a8142c80 msgr2=0x7fd6a81603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6a7e00640 1 --2- 172.21.15.62:0/463998646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6a8142c80 0x7fd6a81603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6a7e00640 1 -- 172.21.15.62:0/463998646 >> v2:172.21.15.120:3300/0 conn(0x7fd6a81608f0 msgr2=0x7fd6a80ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.824+0000 7fd6a7e00640 1 --2- 172.21.15.62:0/463998646 >> v2:172.21.15.120:3300/0 conn(0x7fd6a81608f0 0x7fd6a80ad5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.825+0000 7fd6a7e00640 1 -- 172.21.15.62:0/463998646 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a80ad0b0 con 0x7fd6a80adb10 2024-03-23T11:31:47.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.825+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/463998646 >> v2:172.21.15.120:3300/0 conn(0x7fd6a81608f0 0x7fd6a80ad5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:47.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.825+0000 7fd6a7400640 1 --2- 172.21.15.62:0/463998646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6a8142c80 0x7fd6a81603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:47.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.826+0000 7fd6a7e00640 1 --2- 172.21.15.62:0/463998646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6a80adb10 0x7fd6a806c0d0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fd6a806fd50 tx=0x7fd6a002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84e26c8a4728949 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:47.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.826+0000 7fd6a6000640 1 -- 172.21.15.62:0/463998646 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a002ca80 con 0x7fd6a80adb10 2024-03-23T11:31:47.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.826+0000 7fd6a6000640 1 -- 172.21.15.62:0/463998646 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a002cbe0 con 0x7fd6a80adb10 2024-03-23T11:31:47.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.826+0000 7fd6a6000640 1 -- 172.21.15.62:0/463998646 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a00343f0 con 0x7fd6a80adb10 2024-03-23T11:31:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.827+0000 7fd6ace00640 1 -- 172.21.15.62:0/463998646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6a80adb10 msgr2=0x7fd6a806c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.827+0000 7fd6ace00640 1 --2- 172.21.15.62:0/463998646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6a80adb10 0x7fd6a806c0d0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fd6a806fd50 tx=0x7fd6a002bee0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.827+0000 7fd6ace00640 1 -- 172.21.15.62:0/463998646 shutdown_connections 2024-03-23T11:31:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.827+0000 7fd6ace00640 1 --2- 172.21.15.62:0/463998646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6a80adb10 0x7fd6a806c0d0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.827+0000 7fd6ace00640 1 --2- 172.21.15.62:0/463998646 >> v2:172.21.15.120:3300/0 conn(0x7fd6a81608f0 0x7fd6a80ad5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.827+0000 7fd6ace00640 1 --2- 172.21.15.62:0/463998646 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6a8142c80 0x7fd6a81603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.827+0000 7fd6ace00640 1 -- 172.21.15.62:0/463998646 >> 172.21.15.62:0/463998646 conn(0x7fd6a80b0090 msgr2=0x7fd6a80b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:47.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.827+0000 7fd6ace00640 1 -- 172.21.15.62:0/463998646 shutdown_connections 2024-03-23T11:31:47.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.827+0000 7fd6ace00640 1 -- 172.21.15.62:0/463998646 wait complete. 2024-03-23T11:31:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.829+0000 7fd6ace00640 1 Processor -- start 2024-03-23T11:31:47.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.829+0000 7fd6ace00640 1 -- start start 2024-03-23T11:31:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.831+0000 7fd6ace00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6a80adb10 0x7fd6a815e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.831+0000 7fd6ace00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6a8142c80 0x7fd6a815ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.831+0000 7fd6ace00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6a81608f0 0x7fd6a815f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.831+0000 7fd6ace00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a815eae0 con 0x7fd6a8142c80 2024-03-23T11:31:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.831+0000 7fd6ace00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a815e800 con 0x7fd6a81608f0 2024-03-23T11:31:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.831+0000 7fd6ace00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a813e530 con 0x7fd6a80adb10 2024-03-23T11:31:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.831+0000 7fd6a6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6a8142c80 0x7fd6a815ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.831+0000 7fd6a6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6a8142c80 0x7fd6a815ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48170/0 (socket says 172.21.15.62:48170) 2024-03-23T11:31:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.831+0000 7fd6a6a00640 1 -- 172.21.15.62:0/256950533 learned_addr learned my addr 172.21.15.62:0/256950533 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:47.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.831+0000 7fd6a7400640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6a80adb10 0x7fd6a815e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.831+0000 7fd6a7e00640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6a81608f0 0x7fd6a815f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.832+0000 7fd6a6a00640 1 -- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6a80adb10 msgr2=0x7fd6a815e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.832+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6a80adb10 0x7fd6a815e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.832+0000 7fd6a6a00640 1 -- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6a81608f0 msgr2=0x7fd6a815f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.832+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6a81608f0 0x7fd6a815f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.832+0000 7fd6a6a00640 1 -- 172.21.15.62:0/256950533 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a0009b30 con 0x7fd6a8142c80 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.832+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6a8142c80 0x7fd6a815ee50 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fd69400c7f0 tx=0x7fd69400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.832+0000 7fd69f400640 1 -- 172.21.15.62:0/256950533 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694007820 con 0x7fd6a8142c80 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.832+0000 7fd69f400640 1 -- 172.21.15.62:0/256950533 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd694007980 con 0x7fd6a8142c80 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.832+0000 7fd6ace00640 1 -- 172.21.15.62:0/256950533 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a813e810 con 0x7fd6a8142c80 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.832+0000 7fd69f400640 1 -- 172.21.15.62:0/256950533 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694007cf0 con 0x7fd6a8142c80 2024-03-23T11:31:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.832+0000 7fd6ace00640 1 -- 172.21.15.62:0/256950533 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a81404e0 con 0x7fd6a8142c80 2024-03-23T11:31:47.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.835+0000 7fd69f400640 1 -- 172.21.15.62:0/256950533 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd6940113f0 con 0x7fd6a8142c80 2024-03-23T11:31:47.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.836+0000 7fd69f400640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd68806a7a0 0x7fd68806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.836+0000 7fd69f400640 1 -- 172.21.15.62:0/256950533 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd694089100 con 0x7fd6a8142c80 2024-03-23T11:31:47.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.836+0000 7fd6b11b7740 1 Processor -- start 2024-03-23T11:31:47.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.836+0000 7fd6b11b7740 1 -- start start 2024-03-23T11:31:47.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.837+0000 7fd6a7400640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd68806a7a0 0x7fd68806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.837+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0caf0 0x556a54b14d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.837+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54aecc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.837+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 0x556a54aed160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.837+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a54aa8cb0 con 0x556a54b15460 2024-03-23T11:31:47.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.837+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a54a57500 con 0x556a54b0caf0 2024-03-23T11:31:47.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.837+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a54a54c90 con 0x556a54b0ced0 2024-03-23T11:31:47.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.838+0000 7fd6a7400640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd68806a7a0 0x7fd68806cc60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd698006b10 tx=0x7fd698008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:47.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.838+0000 7fd6a6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54aecc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.838+0000 7fd6a6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54aecc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48174/0 (socket says 172.21.15.62:48174) 2024-03-23T11:31:47.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.838+0000 7fd6a6a00640 1 -- 172.21.15.62:0/3443381443 learned_addr learned my addr 172.21.15.62:0/3443381443 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:47.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.838+0000 7fd6a7400640 1 --2- 172.21.15.62:0/3443381443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0caf0 0x556a54b14d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.838+0000 7fd6a7e00640 1 --2- 172.21.15.62:0/3443381443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 0x556a54aed160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.838+0000 7fd6a6a00640 1 -- 172.21.15.62:0/3443381443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 msgr2=0x556a54aed160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:47.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.838+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/3443381443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 0x556a54aed160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.838+0000 7fd6a6a00640 1 -- 172.21.15.62:0/3443381443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0caf0 msgr2=0x556a54b14d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:47.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.838+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/3443381443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0caf0 0x556a54b14d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.838+0000 7fd6a6a00640 1 -- 172.21.15.62:0/3443381443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556a54a02360 con 0x556a54b15460 2024-03-23T11:31:47.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.839+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/3443381443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54aecc20 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fd694004790 tx=0x7fd694057e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea05235046ba2a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:47.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.839+0000 7fd693e00640 1 -- 172.21.15.62:0/3443381443 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694060580 con 0x556a54b15460 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.839+0000 7fd693e00640 1 -- 172.21.15.62:0/3443381443 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd694056530 con 0x556a54b15460 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.839+0000 7fd693e00640 1 -- 172.21.15.62:0/3443381443 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69405e070 con 0x556a54b15460 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.839+0000 7fd6b11b7740 1 -- 172.21.15.62:0/3443381443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 msgr2=0x556a54aecc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.839+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/3443381443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54aecc20 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fd694004790 tx=0x7fd694057e00 comp rx=0 tx=0).stop 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.840+0000 7fd6b11b7740 1 -- 172.21.15.62:0/3443381443 shutdown_connections 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.840+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/3443381443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54aecc20 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.840+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/3443381443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 0x556a54aed160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.840+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/3443381443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0caf0 0x556a54b14d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.840+0000 7fd6b11b7740 1 -- 172.21.15.62:0/3443381443 >> 172.21.15.62:0/3443381443 conn(0x556a54ab4b40 msgr2=0x556a54b32900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.840+0000 7fd6b11b7740 1 -- 172.21.15.62:0/3443381443 shutdown_connections 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.840+0000 7fd6b11b7740 1 -- 172.21.15.62:0/3443381443 wait complete. 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.840+0000 7fd6b11b7740 1 -- start start 2024-03-23T11:31:47.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.841+0000 7fd6b11b7740 1 Processor -- start 2024-03-23T11:31:47.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.842+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b0bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.842+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0ced0 0x556a54b29960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.842+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b15460 0x556a54b18170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.842+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a54a50580 con 0x556a54b0caf0 2024-03-23T11:31:47.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.842+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a549f1a50 con 0x556a54b0ced0 2024-03-23T11:31:47.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.842+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a549f16f0 con 0x556a54b15460 2024-03-23T11:31:47.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.842+0000 7fd6b11b7740 10 client.0 _ll_register_callbacks cb 0x556a54a40ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:31:47.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.842+0000 7fd6a7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b0bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.842+0000 7fd6a7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b0bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48178/0 (socket says 172.21.15.62:48178) 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.842+0000 7fd6a7400640 1 -- 172.21.15.62:0/1312433724 learned_addr learned my addr 172.21.15.62:0/1312433724 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.842+0000 7fd6a7e00640 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b15460 0x556a54b18170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0ced0 0x556a54b29960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd6a7400640 1 -- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b15460 msgr2=0x556a54b18170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd6a7400640 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b15460 0x556a54b18170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd6a7400640 1 -- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0ced0 msgr2=0x556a54b29960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd6a7400640 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0ced0 0x556a54b29960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd6a7400640 1 -- 172.21.15.62:0/1312433724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a81404e0 con 0x556a54b0caf0 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd6a7400640 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b0bb20 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fd698013700 tx=0x7fd698004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74d01c9842ffdab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd690c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd690c00640 1 -- 172.21.15.62:0/1312433724 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6980147d0 con 0x556a54b0caf0 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd690c00640 1 -- 172.21.15.62:0/1312433724 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd698014930 con 0x556a54b0caf0 2024-03-23T11:31:47.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd6b11b7740 1 -- 172.21.15.62:0/1312433724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556a54a7afe0 con 0x556a54b0caf0 2024-03-23T11:31:47.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd690c00640 1 -- 172.21.15.62:0/1312433724 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd698014af0 con 0x556a54b0caf0 2024-03-23T11:31:47.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd6b11b7740 10 client.9228 fetch_fsmap learned FSMap version 94 2024-03-23T11:31:47.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.843+0000 7fd6b11b7740 1 -- 172.21.15.62:0/1312433724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x556a54a4ec00 con 0x556a54b0caf0 2024-03-23T11:31:47.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.844+0000 7fd690c00640 1 -- 172.21.15.62:0/1312433724 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556a54a7afe0 con 0x556a54b0caf0 2024-03-23T11:31:47.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.844+0000 7fd690c00640 1 -- 172.21.15.62:0/1312433724 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 94) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd698011070 con 0x556a54b0caf0 2024-03-23T11:31:47.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.844+0000 7fd6b11b7740 10 client.9228 fetch_fsmap finished waiting for FSMap version 94 2024-03-23T11:31:47.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.844+0000 7fd6b11b7740 10 client.9228 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.b 2024-03-23T11:31:47.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.844+0000 7fd6b11b7740 20 client.9228 populate_metadata read hostname 'smithi062' 2024-03-23T11:31:47.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.844+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x556a54b34a30 0x556a54b34e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.844+0000 7fd6b11b7740 4 client.9228 mds_command: new command op to 9006 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:31:47.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.844+0000 7fd6b11b7740 1 -- 172.21.15.62:0/1312433724 --> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556a54a43240 con 0x556a54b34a30 2024-03-23T11:31:47.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.845+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x556a54b34a30 0x556a54b34e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.846+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x556a54b34a30 0x556a54b34e10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7d5d0fc6dc257d6 server_cookie=4f1ca60c07417b37 in_seq=0 out_seq=0 2024-03-23T11:31:47.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.846+0000 7fd690c00640 10 client.9228 ms_handle_connect on v2:172.21.15.120:6835/2585024035 2024-03-23T11:31:47.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.847+0000 7fd690c00640 1 -- 172.21.15.62:0/1312433724 <== mds.0 v2:172.21.15.120:6835/2585024035 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x556a54a43240 con 0x556a54b34a30 2024-03-23T11:31:47.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.847+0000 7fd690c00640 10 client.9228 handle_command_reply: tid=0 2024-03-23T11:31:47.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.847+0000 7fd6b11b7740 1 client.9228 shutdown 2024-03-23T11:31:47.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd6b11b7740 1 -- 172.21.15.62:0/1312433724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 msgr2=0x556a54b0bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:47.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b0bb20 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fd698013700 tx=0x7fd698004730 comp rx=0 tx=0).stop 2024-03-23T11:31:47.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd6b11b7740 1 -- 172.21.15.62:0/1312433724 shutdown_connections 2024-03-23T11:31:47.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x556a54b34a30 0x556a54b34e10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b15460 0x556a54b18170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0ced0 0x556a54b29960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/1312433724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b0bb20 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd690c00640 0 client.9228 ms_handle_reset on v2:172.21.15.120:6835/2585024035 2024-03-23T11:31:47.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd6b11b7740 1 -- 172.21.15.62:0/1312433724 >> 172.21.15.62:0/1312433724 conn(0x556a54ab4b40 msgr2=0x556a54ae6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:47.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd6b11b7740 1 -- 172.21.15.62:0/1312433724 shutdown_connections 2024-03-23T11:31:47.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd6b11b7740 1 -- 172.21.15.62:0/1312433724 wait complete. 2024-03-23T11:31:47.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.848+0000 7fd6b11b7740 20 client.9228 trim_cache size 0 max 16384 2024-03-23T11:31:47.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.862+0000 7fd6b11b7740 1 Processor -- start 2024-03-23T11:31:47.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.862+0000 7fd6b11b7740 1 -- start start 2024-03-23T11:31:47.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.863+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0caf0 0x556a54ae33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.863+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 0x556a54ae6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.863+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54b28290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.863+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a5474df90 con 0x556a54b15460 2024-03-23T11:31:47.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.863+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a54b45040 con 0x556a54b0caf0 2024-03-23T11:31:47.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.863+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a54b18f80 con 0x556a54b0ced0 2024-03-23T11:31:47.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.864+0000 7fd6a7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54b28290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.864+0000 7fd6a7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54b28290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48192/0 (socket says 172.21.15.62:48192) 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.864+0000 7fd6a7e00640 1 -- 172.21.15.62:0/2896988475 learned_addr learned my addr 172.21.15.62:0/2896988475 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.864+0000 7fd6a7400640 1 --2- 172.21.15.62:0/2896988475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0caf0 0x556a54ae33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.864+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/2896988475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 0x556a54ae6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.864+0000 7fd6a7e00640 1 -- 172.21.15.62:0/2896988475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 msgr2=0x556a54ae6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.864+0000 7fd6a7e00640 1 --2- 172.21.15.62:0/2896988475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 0x556a54ae6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.864+0000 7fd6a7e00640 1 -- 172.21.15.62:0/2896988475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0caf0 msgr2=0x556a54ae33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.864+0000 7fd6a7e00640 1 --2- 172.21.15.62:0/2896988475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0caf0 0x556a54ae33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.864+0000 7fd6a7e00640 1 -- 172.21.15.62:0/2896988475 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556a549f0c60 con 0x556a54b15460 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.865+0000 7fd6a7e00640 1 --2- 172.21.15.62:0/2896988475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54b28290 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0036040 tx=0x7fd6a002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d717da6761ab858a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.865+0000 7fd69cc00640 1 -- 172.21.15.62:0/2896988475 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0049480 con 0x556a54b15460 2024-03-23T11:31:47.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.865+0000 7fd69cc00640 1 -- 172.21.15.62:0/2896988475 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a0042070 con 0x556a54b15460 2024-03-23T11:31:47.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.865+0000 7fd69cc00640 1 -- 172.21.15.62:0/2896988475 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0049800 con 0x556a54b15460 2024-03-23T11:31:47.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.865+0000 7fd6b11b7740 1 -- 172.21.15.62:0/2896988475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 msgr2=0x556a54b28290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:47.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.865+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/2896988475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54b28290 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0036040 tx=0x7fd6a002e000 comp rx=0 tx=0).stop 2024-03-23T11:31:47.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.866+0000 7fd6b11b7740 1 -- 172.21.15.62:0/2896988475 shutdown_connections 2024-03-23T11:31:47.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.866+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/2896988475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b15460 0x556a54b28290 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.866+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/2896988475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 0x556a54ae6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.866+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/2896988475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b0caf0 0x556a54ae33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.866+0000 7fd6b11b7740 1 -- 172.21.15.62:0/2896988475 >> 172.21.15.62:0/2896988475 conn(0x556a54ab4b40 msgr2=0x556a54b35e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:47.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.866+0000 7fd6b11b7740 1 -- 172.21.15.62:0/2896988475 shutdown_connections 2024-03-23T11:31:47.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.866+0000 7fd6b11b7740 1 -- 172.21.15.62:0/2896988475 wait complete. 2024-03-23T11:31:47.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.866+0000 7fd6b11b7740 1 -- start start 2024-03-23T11:31:47.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.867+0000 7fd6b11b7740 1 Processor -- start 2024-03-23T11:31:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b86940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 0x556a54b14870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6b11b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b15460 0x556a54b14db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a54b19570 con 0x556a54b0caf0 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a54b2c8c0 con 0x556a54b15460 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6b11b7740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a54b86e80 con 0x556a54b0ced0 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6b11b7740 10 client.0 _ll_register_callbacks cb 0x7fd694054550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6a7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b86940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6a7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b86940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48204/0 (socket says 172.21.15.62:48204) 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6a7400640 1 -- 172.21.15.62:0/6078212 learned_addr learned my addr 172.21.15.62:0/6078212 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6a7400640 1 -- 172.21.15.62:0/6078212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 msgr2=0x556a54b14870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6a7400640 1 --2- 172.21.15.62:0/6078212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 0x556a54b14870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6a7400640 1 -- 172.21.15.62:0/6078212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b15460 msgr2=0x556a54b14db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6a7400640 1 --2- 172.21.15.62:0/6078212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b15460 0x556a54b14db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.868+0000 7fd6a7400640 1 -- 172.21.15.62:0/6078212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a80ad0b0 con 0x556a54b0caf0 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.869+0000 7fd6a7400640 1 --2- 172.21.15.62:0/6078212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b86940 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fd69800a3a0 tx=0x7fd698013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942dc6f7420e8d95 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.869+0000 7fd690c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.869+0000 7fd690c00640 1 -- 172.21.15.62:0/6078212 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6980060d0 con 0x556a54b0caf0 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.869+0000 7fd690c00640 1 -- 172.21.15.62:0/6078212 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd698014d90 con 0x556a54b0caf0 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.869+0000 7fd6b11b7740 1 -- 172.21.15.62:0/6078212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd694056530 con 0x556a54b0caf0 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.869+0000 7fd690c00640 1 -- 172.21.15.62:0/6078212 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd698006230 con 0x556a54b0caf0 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.869+0000 7fd6b11b7740 10 client.9234 fetch_fsmap learned FSMap version 94 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.869+0000 7fd6b11b7740 1 -- 172.21.15.62:0/6078212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x556a54a16170 con 0x556a54b0caf0 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.869+0000 7fd690c00640 1 -- 172.21.15.62:0/6078212 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd694056530 con 0x556a54b0caf0 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.870+0000 7fd690c00640 1 -- 172.21.15.62:0/6078212 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 94) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd698011050 con 0x556a54b0caf0 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.870+0000 7fd6b11b7740 10 client.9234 fetch_fsmap finished waiting for FSMap version 94 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.870+0000 7fd6b11b7740 10 client.9234 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.b 2024-03-23T11:31:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.870+0000 7fd6b11b7740 20 client.9234 populate_metadata read hostname 'smithi062' 2024-03-23T11:31:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.870+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/6078212 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x556a54aecc10 0x556a54b19990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.870+0000 7fd6b11b7740 4 client.9234 mds_command: new command op to 9006 tid=0[{"prefix": "flush journal"}] 2024-03-23T11:31:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.870+0000 7fd6b11b7740 1 -- 172.21.15.62:0/6078212 --> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556a547444e0 con 0x556a54aecc10 2024-03-23T11:31:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.870+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/6078212 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x556a54aecc10 0x556a54b19990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.871+0000 7fd6a6a00640 1 --2- 172.21.15.62:0/6078212 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x556a54aecc10 0x556a54b19990 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d73282710d6424 server_cookie=fea32c2b1ac8de68 in_seq=0 out_seq=0 2024-03-23T11:31:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.871+0000 7fd690c00640 10 client.9234 ms_handle_connect on v2:172.21.15.120:6835/2585024035 2024-03-23T11:31:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd690c00640 1 -- 172.21.15.62:0/6078212 <== mds.0 v2:172.21.15.120:6835/2585024035 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556a547444e0 con 0x556a54aecc10 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd690c00640 10 client.9234 handle_command_reply: tid=0 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd6b11b7740 1 client.9234 shutdown 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd6b11b7740 1 -- 172.21.15.62:0/6078212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 msgr2=0x556a54b86940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/6078212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b86940 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fd69800a3a0 tx=0x7fd698013000 comp rx=0 tx=0).stop 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd6b11b7740 1 -- 172.21.15.62:0/6078212 shutdown_connections 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/6078212 >> [v2:172.21.15.120:6835/2585024035,v1:172.21.15.120:6837/2585024035] conn(0x556a54aecc10 0x556a54b19990 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/6078212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a54b15460 0x556a54b14db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/6078212 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556a54b0ced0 0x556a54b14870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd6b11b7740 1 --2- 172.21.15.62:0/6078212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556a54b0caf0 0x556a54b86940 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd690c00640 0 client.9234 ms_handle_reset on v2:172.21.15.120:6835/2585024035 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.887+0000 7fd6b11b7740 1 -- 172.21.15.62:0/6078212 >> 172.21.15.62:0/6078212 conn(0x556a54ab4b40 msgr2=0x556a54b17e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.888+0000 7fd6b11b7740 1 -- 172.21.15.62:0/6078212 shutdown_connections 2024-03-23T11:31:47.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.888+0000 7fd6b11b7740 1 -- 172.21.15.62:0/6078212 wait complete. 2024-03-23T11:31:47.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.888+0000 7fd6b11b7740 20 client.9234 trim_cache size 0 max 16384 2024-03-23T11:31:47.880 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:31:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "", 2024-03-23T11:31:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0 2024-03-23T11:31:47.880 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:31:47.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.888+0000 7fd6ace00640 1 -- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd68806a7a0 msgr2=0x7fd68806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:47.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.888+0000 7fd6ace00640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd68806a7a0 0x7fd68806cc60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd698006b10 tx=0x7fd698008040 comp rx=0 tx=0).stop 2024-03-23T11:31:47.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.889+0000 7fd6ace00640 1 -- 172.21.15.62:0/256950533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6a8142c80 msgr2=0x7fd6a815ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:47.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.889+0000 7fd6ace00640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6a8142c80 0x7fd6a815ee50 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fd69400c7f0 tx=0x7fd69400ccb0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.889+0000 7fd6ace00640 1 -- 172.21.15.62:0/256950533 shutdown_connections 2024-03-23T11:31:47.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.889+0000 7fd6ace00640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd68806a7a0 0x7fd68806cc60 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.889+0000 7fd6ace00640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6a8142c80 0x7fd6a815ee50 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.889+0000 7fd6ace00640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6a81608f0 0x7fd6a815f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.889+0000 7fd6ace00640 1 --2- 172.21.15.62:0/256950533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6a80adb10 0x7fd6a815e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:47.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.889+0000 7fd6ace00640 1 -- 172.21.15.62:0/256950533 >> 172.21.15.62:0/256950533 conn(0x7fd6a80b0090 msgr2=0x7fd6a80b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:47.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.889+0000 7fd6ace00640 1 -- 172.21.15.62:0/256950533 shutdown_connections 2024-03-23T11:31:47.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:47.889+0000 7fd6ace00640 1 -- 172.21.15.62:0/256950533 wait complete. 2024-03-23T11:31:47.897 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:31:48.095 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:31:48.102 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:31:48.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.117+0000 7fe33ae00640 1 Processor -- start 2024-03-23T11:31:48.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.117+0000 7fe33ae00640 1 -- start start 2024-03-23T11:31:48.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.118+0000 7fe33ae00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe33406a370 0x7fe33415ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:48.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe33ae00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe3340b4df0 0x7fe3340b5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:48.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe33ae00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3340b57c0 con 0x7fe3341443d0 2024-03-23T11:31:48.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe33ae00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33415b3c0 con 0x7fe33406a370 2024-03-23T11:31:48.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe33ae00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33406eb70 con 0x7fe3340b4df0 2024-03-23T11:31:48.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe333400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe33406a370 0x7fe33415ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:48.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe333400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe33406a370 0x7fe33415ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48206/0 (socket says 172.21.15.62:48206) 2024-03-23T11:31:48.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe333400640 1 -- 172.21.15.62:0/3859068871 learned_addr learned my addr 172.21.15.62:0/3859068871 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:48.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe338e00640 1 --2- 172.21.15.62:0/3859068871 >> v2:172.21.15.120:3300/0 conn(0x7fe3340b4df0 0x7fe3340b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:48.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe333400640 1 -- 172.21.15.62:0/3859068871 >> v2:172.21.15.120:3300/0 conn(0x7fe3340b4df0 msgr2=0x7fe3340b5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe333400640 1 --2- 172.21.15.62:0/3859068871 >> v2:172.21.15.120:3300/0 conn(0x7fe3340b4df0 0x7fe3340b5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe333400640 1 -- 172.21.15.62:0/3859068871 >> v1:172.21.15.120:6789/0 conn(0x7fe3341443d0 legacy=0x7fe334069c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.119+0000 7fe333400640 1 -- 172.21.15.62:0/3859068871 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33406ee50 con 0x7fe33406a370 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.120+0000 7fe338e00640 1 --2- 172.21.15.62:0/3859068871 >> v2:172.21.15.120:3300/0 conn(0x7fe3340b4df0 0x7fe3340b5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.121+0000 7fe333400640 1 --2- 172.21.15.62:0/3859068871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33406a370 0x7fe33415ae80 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fe320009870 tx=0x7fe32002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7460835e0720d2ca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.121+0000 7fe332a00640 1 -- 172.21.15.62:0/3859068871 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe320031870 con 0x7fe33406a370 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.121+0000 7fe332a00640 1 -- 172.21.15.62:0/3859068871 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3200319d0 con 0x7fe33406a370 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.121+0000 7fe332a00640 1 -- 172.21.15.62:0/3859068871 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe320031d60 con 0x7fe33406a370 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.122+0000 7fe33ae00640 1 -- 172.21.15.62:0/3859068871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33406a370 msgr2=0x7fe33415ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.122+0000 7fe33ae00640 1 --2- 172.21.15.62:0/3859068871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33406a370 0x7fe33415ae80 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fe320009870 tx=0x7fe32002b910 comp rx=0 tx=0).stop 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.122+0000 7fe33ae00640 1 -- 172.21.15.62:0/3859068871 shutdown_connections 2024-03-23T11:31:48.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.122+0000 7fe33ae00640 1 --2- 172.21.15.62:0/3859068871 >> v2:172.21.15.120:3300/0 conn(0x7fe3340b4df0 0x7fe3340b5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:48.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.122+0000 7fe33ae00640 1 --2- 172.21.15.62:0/3859068871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33406a370 0x7fe33415ae80 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:48.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.122+0000 7fe33ae00640 1 -- 172.21.15.62:0/3859068871 >> 172.21.15.62:0/3859068871 conn(0x7fe3340b1370 msgr2=0x7fe3340b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:48.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.122+0000 7fe33ae00640 1 -- 172.21.15.62:0/3859068871 shutdown_connections 2024-03-23T11:31:48.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.122+0000 7fe33ae00640 1 -- 172.21.15.62:0/3859068871 wait complete. 2024-03-23T11:31:48.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.124+0000 7fe33ae00640 1 Processor -- start 2024-03-23T11:31:48.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.124+0000 7fe33ae00640 1 -- start start 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe33ae00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33406a370 0x7fe3340c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe33ae00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3340b4df0 0x7fe334137e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe33ae00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3341443d0 0x7fe33415d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe33ae00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe334138350 con 0x7fe33406a370 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe33ae00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3341387e0 con 0x7fe3340b4df0 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe33ae00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340cfa90 con 0x7fe3341443d0 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe333e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33406a370 0x7fe3340c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe333e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33406a370 0x7fe3340c45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48218/0 (socket says 172.21.15.62:48218) 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe333e00640 1 -- 172.21.15.62:0/2814703149 learned_addr learned my addr 172.21.15.62:0/2814703149 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe338e00640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3341443d0 0x7fe33415d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe333400640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3340b4df0 0x7fe334137e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:48.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe333e00640 1 -- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3341443d0 msgr2=0x7fe33415d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:48.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe333e00640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3341443d0 0x7fe33415d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:48.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe333e00640 1 -- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3340b4df0 msgr2=0x7fe334137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:48.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.125+0000 7fe333e00640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3340b4df0 0x7fe334137e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:48.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.126+0000 7fe333e00640 1 -- 172.21.15.62:0/2814703149 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3200094d0 con 0x7fe33406a370 2024-03-23T11:31:48.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.126+0000 7fe333400640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3340b4df0 0x7fe334137e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:48.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.126+0000 7fe333e00640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33406a370 0x7fe3340c45e0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fe32400b3f0 tx=0x7fe32400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:48.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.126+0000 7fe32be00640 1 -- 172.21.15.62:0/2814703149 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324011830 con 0x7fe33406a370 2024-03-23T11:31:48.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.126+0000 7fe32be00640 1 -- 172.21.15.62:0/2814703149 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe324011990 con 0x7fe33406a370 2024-03-23T11:31:48.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.126+0000 7fe32be00640 1 -- 172.21.15.62:0/2814703149 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324011d40 con 0x7fe33406a370 2024-03-23T11:31:48.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.126+0000 7fe33ae00640 1 -- 172.21.15.62:0/2814703149 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe334141a70 con 0x7fe33406a370 2024-03-23T11:31:48.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.126+0000 7fe33ae00640 1 -- 172.21.15.62:0/2814703149 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3340cfd50 con 0x7fe33406a370 2024-03-23T11:31:48.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.128+0000 7fe33ae00640 1 -- 172.21.15.62:0/2814703149 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3340672c0 con 0x7fe33406a370 2024-03-23T11:31:48.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.128+0000 7fe32be00640 1 -- 172.21.15.62:0/2814703149 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe32401a400 con 0x7fe33406a370 2024-03-23T11:31:48.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.130+0000 7fe32be00640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe31406a7a0 0x7fe31406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:48.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.130+0000 7fe32be00640 1 -- 172.21.15.62:0/2814703149 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe324088c30 con 0x7fe33406a370 2024-03-23T11:31:48.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.131+0000 7fe333400640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe31406a7a0 0x7fe31406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:48.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.132+0000 7fe333400640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe31406a7a0 0x7fe31406cc60 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe320036000 tx=0x7fe32003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:48.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.134+0000 7fe32be00640 1 -- 172.21.15.62:0/2814703149 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe324054000 con 0x7fe33406a370 2024-03-23T11:31:48.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.376+0000 7fe33ae00640 1 -- 172.21.15.62:0/2814703149 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe33415cf60 con 0x7fe33406a370 2024-03-23T11:31:48.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.377+0000 7fe32be00640 1 -- 172.21.15.62:0/2814703149 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v94) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe324015070 con 0x7fe33406a370 2024-03-23T11:31:48.368 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:31:48.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 -- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe31406a7a0 msgr2=0x7fe31406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:48.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe31406a7a0 0x7fe31406cc60 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe320036000 tx=0x7fe32003c040 comp rx=0 tx=0).stop 2024-03-23T11:31:48.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 -- 172.21.15.62:0/2814703149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33406a370 msgr2=0x7fe3340c45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:48.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33406a370 0x7fe3340c45e0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fe32400b3f0 tx=0x7fe32400b8b0 comp rx=0 tx=0).stop 2024-03-23T11:31:48.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 -- 172.21.15.62:0/2814703149 shutdown_connections 2024-03-23T11:31:48.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe31406a7a0 0x7fe31406cc60 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:48.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3341443d0 0x7fe33415d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:48.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3340b4df0 0x7fe334137e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:48.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 --2- 172.21.15.62:0/2814703149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33406a370 0x7fe3340c45e0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:48.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 -- 172.21.15.62:0/2814703149 >> 172.21.15.62:0/2814703149 conn(0x7fe3340b1370 msgr2=0x7fe3340af340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:48.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 -- 172.21.15.62:0/2814703149 shutdown_connections 2024-03-23T11:31:48.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:48.380+0000 7fe33ae00640 1 -- 172.21.15.62:0/2814703149 wait complete. 2024-03-23T11:31:48.392 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:31:48.392 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:31:48.392 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:31:48.392 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:31:48.392 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:48.392 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:31:48.392 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:31:48.392 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:31:48.392 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:31:48.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:48.393 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:31:48.444 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:31:48.445 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:31:48.445 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:31:48.445 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:48.445 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:31:48.445 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:48.445 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:31:48.445 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 41848sec preferred_lft 41848sec 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:31:48.446 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:31:48.447 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:31:48.448 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:31:48.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:48.449 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:31:48.541 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:31:48.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:48.546 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:31:48.599 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:31:48.599 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:31:48.599 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:31:48.600 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:31:48.600 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:31:48.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:48.600 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:31:48.600 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:31:48.600 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:31:48.601 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:31:48.601 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:31:48.601 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:31:48.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:48.785 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:31:48.834 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:31:48.834 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:31:48.835 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:31:48.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:48.835 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:31:48.933 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:31:48.933 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:31:48.933 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:48.991 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:31:48.991 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:31:48.991 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:49.044 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:31:49.139 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:31:49.139 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:31:49.234 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:31:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:31:49.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:49.239 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:31:49.295 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:31:49.295 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:31:49.300 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:31:49.377 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:31:49.377 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:31:49.446 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:31:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:31:49.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:49.450 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:31:49.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:31:49.497+0000 7f2cfca37480 -1 init, newargv = 0x55a0c84fab10 newargc=15 2024-03-23T11:31:49.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[78892]: starting ceph client 2024-03-23T11:31:49.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[78892]: starting fuse 2024-03-23T11:31:50.595 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:31:50.703 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:31:50.703 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:31:50.799 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:31:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:31:50.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:50.804 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:31:50.862 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:31:50.862 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:31:50.863 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:31:50.863 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:31:50.863 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:31:50.863 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:31:50.863 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:31:50.863 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:50.863 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:31:50.863 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:31:50.863 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:31:50.863 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:31:50.863 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:31:50.864 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:31:50.865 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:50.865 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:31:50.865 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:31:51.049 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.78892.asok 2024-03-23T11:31:51.227 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.78892.asok 2024-03-23T11:31:51.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:51.228 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.78892.asok status 2024-03-23T11:31:51.474 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:31:51.474 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:31:51.474 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:31:51.474 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:31:51.474 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:31:51.474 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:31:51.474 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "78892", 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 9246, 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 9246 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:31:51.475 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2944917105 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2944917105 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.9246 192.168.0.1:0/2944917105", 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/2944917105", 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 94, 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 120, 2024-03-23T11:31:51.476 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:31:51.477 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:31:51.477 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:31:51.477 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:31:51.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:51.487 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:31:51.487 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:51.540 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:31:51.540 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:51.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:51.540 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:51.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:51.641 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent/unflushed) 2024-03-23T11:31:51.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:51.712 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=./parent/unflushed/jfile bs=1M count=8) 2024-03-23T11:31:51.918 INFO:teuthology.orchestra.run.smithi062.stderr:8+0 records in 2024-03-23T11:31:51.918 INFO:teuthology.orchestra.run.smithi062.stderr:8+0 records out 2024-03-23T11:31:51.918 INFO:teuthology.orchestra.run.smithi062.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.14362 s, 58.4 MB/s 2024-03-23T11:31:51.921 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:31:51.922 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:31:51.922 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:31:51.922 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:51.922 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-03-23T11:31:51.922 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:31:52.074 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627779 2024-03-23T11:31:52.254 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:31:52.254 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:31:52.254 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:31:52.254 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:31:52.254 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-03-23T11:31:52.254 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:31:52.443 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627780 2024-03-23T11:31:52.622 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:31:52.622 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:52.722 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:31:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.956+0000 7fb1ff200640 1 Processor -- start 2024-03-23T11:31:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.957+0000 7fb1ff200640 1 -- start start 2024-03-23T11:31:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.958+0000 7fb1ff200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb200070090 0x7fb200070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.958+0000 7fb1ff200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2000cd140 con 0x7fb2000b2820 2024-03-23T11:31:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.958+0000 7fb1ff200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2000cd3e0 con 0x7fb200143dd0 2024-03-23T11:31:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.958+0000 7fb1ff200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb200070a60 con 0x7fb200070090 2024-03-23T11:31:52.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.958+0000 7fb1fe800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb2000b2820 0x7fb2000b2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:34304/0 (socket says 172.21.15.62:34304) 2024-03-23T11:31:52.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.958+0000 7fb1fe800640 1 -- 172.21.15.62:0/2171668164 learned_addr learned my addr 172.21.15.62:0/2171668164 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:52.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.959+0000 7fb1fd400640 1 --2- 172.21.15.62:0/2171668164 >> v2:172.21.15.120:3300/0 conn(0x7fb200070090 0x7fb200070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:52.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.959+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926771766 0 0) 0x7fb2000cd140 con 0x7fb2000b2820 2024-03-23T11:31:52.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.959+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e0003520 con 0x7fb2000b2820 2024-03-23T11:31:52.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.959+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 379579228 0 0) 0x7fb1e0003520 con 0x7fb2000b2820 2024-03-23T11:31:52.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.959+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 >> v2:172.21.15.120:3300/0 conn(0x7fb200070090 msgr2=0x7fb200070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.959+0000 7fb1fca00640 1 --2- 172.21.15.62:0/2171668164 >> v2:172.21.15.120:3300/0 conn(0x7fb200070090 0x7fb200070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.959+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 >> v1:172.21.15.120:6789/0 conn(0x7fb200143dd0 legacy=0x7fb20015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.959+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2000cd6c0 con 0x7fb2000b2820 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.959+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fb1f8001670 con 0x7fb2000b2820 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.959+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 >> v1:172.21.15.62:6789/0 conn(0x7fb2000b2820 legacy=0x7fb2000b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.959+0000 7fb1fca00640 1 --2- 172.21.15.62:0/2171668164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 0x7fb1e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fca00640 1 --2- 172.21.15.62:0/2171668164 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1e00047c0 0x7fb1e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fca00640 1 --2- 172.21.15.62:0/2171668164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1e0005180 0x7fb1e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200070a60 con 0x7fb1e0003970 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e000c020 con 0x7fb1e0005180 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e000c1e0 con 0x7fb1e00047c0 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fde00640 1 --2- 172.21.15.62:0/2171668164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 0x7fb1e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:52.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fe800640 1 --2- 172.21.15.62:0/2171668164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1e0005180 0x7fb1e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fde00640 1 -- 172.21.15.62:0/2171668164 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1e00047c0 msgr2=0x7fb1e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fde00640 1 --2- 172.21.15.62:0/2171668164 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1e00047c0 0x7fb1e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fde00640 1 -- 172.21.15.62:0/2171668164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1e0005180 msgr2=0x7fb1e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fde00640 1 --2- 172.21.15.62:0/2171668164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1e0005180 0x7fb1e000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fe800640 1 -- 172.21.15.62:0/2171668164 reap_dead start 2024-03-23T11:31:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.960+0000 7fb1fde00640 1 -- 172.21.15.62:0/2171668164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e0003520 con 0x7fb1e0003970 2024-03-23T11:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.962+0000 7fb1fde00640 1 --2- 172.21.15.62:0/2171668164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 0x7fb1e0003d90 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0001750 tx=0x7fb1e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7b0315982396aa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.962+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f00052d0 con 0x7fb1e0003970 2024-03-23T11:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.962+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f002e050 con 0x7fb1e0003970 2024-03-23T11:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.962+0000 7fb1fca00640 1 -- 172.21.15.62:0/2171668164 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0003830 con 0x7fb1e0003970 2024-03-23T11:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.963+0000 7fb1ff200640 1 -- 172.21.15.62:0/2171668164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 msgr2=0x7fb1e0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.963+0000 7fb1ff200640 1 --2- 172.21.15.62:0/2171668164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 0x7fb1e0003d90 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0001750 tx=0x7fb1e00057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.963+0000 7fb1ff200640 1 -- 172.21.15.62:0/2171668164 shutdown_connections 2024-03-23T11:31:52.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.963+0000 7fb1ff200640 1 --2- 172.21.15.62:0/2171668164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 0x7fb1e0003d90 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:52.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.963+0000 7fb1ff200640 1 -- 172.21.15.62:0/2171668164 >> 172.21.15.62:0/2171668164 conn(0x7fb200066050 msgr2=0x7fb200069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:52.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.963+0000 7fb1ff200640 1 -- 172.21.15.62:0/2171668164 shutdown_connections 2024-03-23T11:31:52.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.963+0000 7fb1ff200640 1 -- 172.21.15.62:0/2171668164 wait complete. 2024-03-23T11:31:52.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.965+0000 7fb1ff200640 1 Processor -- start 2024-03-23T11:31:52.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.965+0000 7fb1ff200640 1 -- start start 2024-03-23T11:31:52.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.966+0000 7fb1ff200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 0x7fb200159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:52.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.966+0000 7fb1ff200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20013aaf0 0x7fb20013c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.966+0000 7fb1ff200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb200140ca0 0x7fb200141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.966+0000 7fb1ff200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20013a6d0 con 0x7fb1e0003970 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.966+0000 7fb1ff200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20015a530 con 0x7fb200140ca0 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.966+0000 7fb1ff200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20013e330 con 0x7fb20013aaf0 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.966+0000 7fb1fde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 0x7fb200159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.966+0000 7fb1fde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 0x7fb200159ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48238/0 (socket says 172.21.15.62:48238) 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.966+0000 7fb1fde00640 1 -- 172.21.15.62:0/534091244 learned_addr learned my addr 172.21.15.62:0/534091244 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.967+0000 7fb1fde00640 1 -- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20013aaf0 msgr2=0x7fb20013c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.967+0000 7fb1fde00640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20013aaf0 0x7fb20013c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.967+0000 7fb1fde00640 1 -- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb200140ca0 msgr2=0x7fb200141120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.967+0000 7fb1fde00640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb200140ca0 0x7fb200141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:52.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.967+0000 7fb1fde00640 1 -- 172.21.15.62:0/534091244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb200139a50 con 0x7fb1e0003970 2024-03-23T11:31:52.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.967+0000 7fb1fde00640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 0x7fb200159ff0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0005040 tx=0x7fb1f0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:52.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.967+0000 7fb1f6000640 1 -- 172.21.15.62:0/534091244 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0007a30 con 0x7fb1e0003970 2024-03-23T11:31:52.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.967+0000 7fb1f6000640 1 -- 172.21.15.62:0/534091244 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f00069e0 con 0x7fb1e0003970 2024-03-23T11:31:52.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.967+0000 7fb1f6000640 1 -- 172.21.15.62:0/534091244 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0006d70 con 0x7fb1e0003970 2024-03-23T11:31:52.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.967+0000 7fb1ff200640 1 -- 172.21.15.62:0/534091244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb200139d30 con 0x7fb1e0003970 2024-03-23T11:31:52.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.967+0000 7fb1ff200640 1 -- 172.21.15.62:0/534091244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20013a2f0 con 0x7fb1e0003970 2024-03-23T11:31:52.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.969+0000 7fb1ff200640 1 -- 172.21.15.62:0/534091244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb200066bb0 con 0x7fb1e0003970 2024-03-23T11:31:52.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.969+0000 7fb1f6000640 1 -- 172.21.15.62:0/534091244 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb1f00035f0 con 0x7fb1e0003970 2024-03-23T11:31:52.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.970+0000 7fb1f6000640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb1dc06a7d0 0x7fb1dc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:52.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.970+0000 7fb1f6000640 1 -- 172.21.15.62:0/534091244 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb1f00aa930 con 0x7fb1e0003970 2024-03-23T11:31:52.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.971+0000 7fb1fd400640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb1dc06a7d0 0x7fb1dc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:52.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.972+0000 7fb1fd400640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb1dc06a7d0 0x7fb1dc06cc90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb200138ff0 tx=0x7fb1e800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:52.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:52.975+0000 7fb1f6000640 1 -- 172.21.15.62:0/534091244 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb1f0075d00 con 0x7fb1e0003970 2024-03-23T11:31:53.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.172+0000 7fb1ff200640 1 -- 172.21.15.62:0/534091244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2000ca610 con 0x7fb1e0003970 2024-03-23T11:31:53.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.174+0000 7fb1f6000640 1 -- 172.21.15.62:0/534091244 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16152 (secure 0 0 0) 0x7fb1f0041090 con 0x7fb1e0003970 2024-03-23T11:31:53.164 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:53.165 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":120,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:31:44.554756+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:31:53.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 -- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb1dc06a7d0 msgr2=0x7fb1dc06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb1dc06a7d0 0x7fb1dc06cc90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb200138ff0 tx=0x7fb1e800b040 comp rx=0 tx=0).stop 2024-03-23T11:31:53.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 -- 172.21.15.62:0/534091244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 msgr2=0x7fb200159ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 0x7fb200159ff0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0005040 tx=0x7fb1f0001120 comp rx=0 tx=0).stop 2024-03-23T11:31:53.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 -- 172.21.15.62:0/534091244 shutdown_connections 2024-03-23T11:31:53.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb1dc06a7d0 0x7fb1dc06cc90 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb200140ca0 0x7fb200141120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb20013aaf0 0x7fb20013c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 --2- 172.21.15.62:0/534091244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1e0003970 0x7fb200159ff0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 -- 172.21.15.62:0/534091244 >> 172.21.15.62:0/534091244 conn(0x7fb200066050 msgr2=0x7fb2000ae780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:53.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 -- 172.21.15.62:0/534091244 shutdown_connections 2024-03-23T11:31:53.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.177+0000 7fb1ff200640 1 -- 172.21.15.62:0/534091244 wait complete. 2024-03-23T11:31:53.187 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:31:22.127079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:31:23.062387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:31:53.188 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:31:53.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.430+0000 7fbd14e00640 1 Processor -- start 2024-03-23T11:31:53.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.430+0000 7fbd14e00640 1 -- start start 2024-03-23T11:31:53.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.432+0000 7fbd14e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd101440b0 0x7fbd10071b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.432+0000 7fbd14e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbd100720a0 0x7fbd10069c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.432+0000 7fbd14e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd100b2e50 con 0x7fbd10143580 2024-03-23T11:31:53.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.432+0000 7fbd14e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd10144510 con 0x7fbd100720a0 2024-03-23T11:31:53.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.432+0000 7fbd14e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd100b3130 con 0x7fbd101440b0 2024-03-23T11:31:53.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.432+0000 7fbd0e800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbd100720a0 0x7fbd10069c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.432+0000 7fbd0de00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbd10143580 0x7fbd100b2670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:34306/0 (socket says 172.21.15.62:34306) 2024-03-23T11:31:53.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.432+0000 7fbd0de00640 1 -- 172.21.15.62:0/2360305557 learned_addr learned my addr 172.21.15.62:0/2360305557 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:53.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.433+0000 7fbd0d400640 1 --2- 172.21.15.62:0/2360305557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd101440b0 0x7fbd10071b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.433+0000 7fbd0e800640 1 -- 172.21.15.62:0/2360305557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd101440b0 msgr2=0x7fbd10071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:53.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.433+0000 7fbd0e800640 1 --2- 172.21.15.62:0/2360305557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd101440b0 0x7fbd10071b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.433+0000 7fbd0e800640 1 -- 172.21.15.62:0/2360305557 >> v1:172.21.15.62:6789/0 conn(0x7fbd10143580 legacy=0x7fbd100b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.433+0000 7fbd0e800640 1 -- 172.21.15.62:0/2360305557 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd100b3410 con 0x7fbd100720a0 2024-03-23T11:31:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.434+0000 7fbd0e800640 1 --2- 172.21.15.62:0/2360305557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd100720a0 0x7fbd10069c60 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009d90 tx=0x7fbd0802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2851f7239967980 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.434+0000 7fbd0ca00640 1 -- 172.21.15.62:0/2360305557 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0802cc50 con 0x7fbd100720a0 2024-03-23T11:31:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.434+0000 7fbd0ca00640 1 -- 172.21.15.62:0/2360305557 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0802cdb0 con 0x7fbd100720a0 2024-03-23T11:31:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.434+0000 7fbd0ca00640 1 -- 172.21.15.62:0/2360305557 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd08034660 con 0x7fbd100720a0 2024-03-23T11:31:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.435+0000 7fbd14e00640 1 -- 172.21.15.62:0/2360305557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd100720a0 msgr2=0x7fbd10069c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.435+0000 7fbd14e00640 1 --2- 172.21.15.62:0/2360305557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd100720a0 0x7fbd10069c60 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009d90 tx=0x7fbd0802bea0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.435+0000 7fbd14e00640 1 -- 172.21.15.62:0/2360305557 shutdown_connections 2024-03-23T11:31:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.435+0000 7fbd14e00640 1 --2- 172.21.15.62:0/2360305557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd100720a0 0x7fbd10069c60 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.435+0000 7fbd14e00640 1 --2- 172.21.15.62:0/2360305557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd101440b0 0x7fbd10071b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.435+0000 7fbd14e00640 1 -- 172.21.15.62:0/2360305557 >> 172.21.15.62:0/2360305557 conn(0x7fbd100b1370 msgr2=0x7fbd100b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.435+0000 7fbd14e00640 1 -- 172.21.15.62:0/2360305557 shutdown_connections 2024-03-23T11:31:53.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.435+0000 7fbd14e00640 1 -- 172.21.15.62:0/2360305557 wait complete. 2024-03-23T11:31:53.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.437+0000 7fbd14e00640 1 Processor -- start 2024-03-23T11:31:53.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.437+0000 7fbd14e00640 1 -- start start 2024-03-23T11:31:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.438+0000 7fbd14e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd100720a0 0x7fbd100c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.438+0000 7fbd14e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd10143580 0x7fbd1015a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.438+0000 7fbd14e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd101440b0 0x7fbd100ce070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.438+0000 7fbd14e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd100ce8c0 con 0x7fbd101440b0 2024-03-23T11:31:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.438+0000 7fbd14e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd100ceba0 con 0x7fbd10143580 2024-03-23T11:31:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.438+0000 7fbd14e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd100cee80 con 0x7fbd100720a0 2024-03-23T11:31:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.439+0000 7fbd0e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd101440b0 0x7fbd100ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.439+0000 7fbd0e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd101440b0 0x7fbd100ce070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48262/0 (socket says 172.21.15.62:48262) 2024-03-23T11:31:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.439+0000 7fbd0e800640 1 -- 172.21.15.62:0/410258934 learned_addr learned my addr 172.21.15.62:0/410258934 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.439+0000 7fbd0de00640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd100720a0 0x7fbd100c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.439+0000 7fbd0d400640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd10143580 0x7fbd1015a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.439+0000 7fbd0e800640 1 -- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd100720a0 msgr2=0x7fbd100c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.439+0000 7fbd0e800640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd100720a0 0x7fbd100c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.439+0000 7fbd0e800640 1 -- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd10143580 msgr2=0x7fbd1015a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.439+0000 7fbd0e800640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd10143580 0x7fbd1015a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.439+0000 7fbd0e800640 1 -- 172.21.15.62:0/410258934 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd08009a70 con 0x7fbd101440b0 2024-03-23T11:31:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.439+0000 7fbd0e800640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd101440b0 0x7fbd100ce070 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009d90 tx=0x7fbd08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.440+0000 7fbd06000640 1 -- 172.21.15.62:0/410258934 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd08040650 con 0x7fbd101440b0 2024-03-23T11:31:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.440+0000 7fbd06000640 1 -- 172.21.15.62:0/410258934 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd080407d0 con 0x7fbd101440b0 2024-03-23T11:31:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.440+0000 7fbd06000640 1 -- 172.21.15.62:0/410258934 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd08040b80 con 0x7fbd101440b0 2024-03-23T11:31:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.440+0000 7fbd14e00640 1 -- 172.21.15.62:0/410258934 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd100cf160 con 0x7fbd101440b0 2024-03-23T11:31:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.440+0000 7fbd14e00640 1 -- 172.21.15.62:0/410258934 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1013df00 con 0x7fbd101440b0 2024-03-23T11:31:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.441+0000 7fbd14e00640 1 -- 172.21.15.62:0/410258934 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd100672c0 con 0x7fbd101440b0 2024-03-23T11:31:53.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.442+0000 7fbd06000640 1 -- 172.21.15.62:0/410258934 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fbd08040ce0 con 0x7fbd101440b0 2024-03-23T11:31:53.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.443+0000 7fbd06000640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcf006a7d0 0x7fbcf006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.443+0000 7fbd06000640 1 -- 172.21.15.62:0/410258934 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbd080abc70 con 0x7fbd101440b0 2024-03-23T11:31:53.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.444+0000 7fbd0de00640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcf006a7d0 0x7fbcf006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.445+0000 7fbd0de00640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcf006a7d0 0x7fbcf006cc90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fbd00005880 tx=0x7fbd00009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.448+0000 7fbd06000640 1 -- 172.21.15.62:0/410258934 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd080770d0 con 0x7fbd101440b0 2024-03-23T11:31:53.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.648+0000 7fbd14e00640 1 -- 172.21.15.62:0/410258934 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbd101f30e0 con 0x7fbd101440b0 2024-03-23T11:31:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.649+0000 7fbd06000640 1 -- 172.21.15.62:0/410258934 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbd08039020 con 0x7fbd101440b0 2024-03-23T11:31:53.639 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:31:53.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.652+0000 7fbd14e00640 1 -- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcf006a7d0 msgr2=0x7fbcf006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.652+0000 7fbd14e00640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcf006a7d0 0x7fbcf006cc90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fbd00005880 tx=0x7fbd00009040 comp rx=0 tx=0).stop 2024-03-23T11:31:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.652+0000 7fbd14e00640 1 -- 172.21.15.62:0/410258934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd101440b0 msgr2=0x7fbd100ce070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.652+0000 7fbd14e00640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd101440b0 0x7fbd100ce070 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009d90 tx=0x7fbd08004a40 comp rx=0 tx=0).stop 2024-03-23T11:31:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.652+0000 7fbd14e00640 1 -- 172.21.15.62:0/410258934 shutdown_connections 2024-03-23T11:31:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.652+0000 7fbd14e00640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fbcf006a7d0 0x7fbcf006cc90 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.652+0000 7fbd14e00640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd101440b0 0x7fbd100ce070 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.652+0000 7fbd14e00640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd10143580 0x7fbd1015a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.652+0000 7fbd14e00640 1 --2- 172.21.15.62:0/410258934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd100720a0 0x7fbd100c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.652+0000 7fbd14e00640 1 -- 172.21.15.62:0/410258934 >> 172.21.15.62:0/410258934 conn(0x7fbd100b1370 msgr2=0x7fbd10072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.653+0000 7fbd14e00640 1 -- 172.21.15.62:0/410258934 shutdown_connections 2024-03-23T11:31:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.653+0000 7fbd14e00640 1 -- 172.21.15.62:0/410258934 wait complete. 2024-03-23T11:31:53.663 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:31:53.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.888+0000 7f34b6400640 1 Processor -- start 2024-03-23T11:31:53.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.889+0000 7f34b6400640 1 -- start start 2024-03-23T11:31:53.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.890+0000 7f34b6400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f34b00cafa0 0x7f34b00cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.890+0000 7f34b6400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34b006c6e0 0x7f34b006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.890+0000 7f34b6400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b00cbc60 con 0x7f34b006c300 2024-03-23T11:31:53.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.890+0000 7f34b6400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b00cd410 con 0x7f34b00cafa0 2024-03-23T11:31:53.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.890+0000 7f34b6400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b00cbf40 con 0x7f34b006c6e0 2024-03-23T11:31:53.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.891+0000 7f34af400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f34b006c300 0x7f34b00ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:34308/0 (socket says 172.21.15.62:34308) 2024-03-23T11:31:53.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.891+0000 7f34af400640 1 -- 172.21.15.62:0/3721047004 learned_addr learned my addr 172.21.15.62:0/3721047004 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:53.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.891+0000 7f34a7e00640 1 --2- 172.21.15.62:0/3721047004 >> v2:172.21.15.120:3300/0 conn(0x7f34b00cafa0 0x7f34b00cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.891+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499955146 0 0) 0x7f34b00cbc60 con 0x7f34b006c300 2024-03-23T11:31:53.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.891+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3494003520 con 0x7f34b006c300 2024-03-23T11:31:53.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.891+0000 7f34afe00640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34b006c6e0 0x7f34b006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2263111577 0 0) 0x7f3494003520 con 0x7f34b006c300 2024-03-23T11:31:53.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34b006c6e0 msgr2=0x7f34b006cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:53.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34b006c6e0 0x7f34b006cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 >> v2:172.21.15.120:3300/0 conn(0x7f34b00cafa0 msgr2=0x7f34b00cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:53.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 --2- 172.21.15.62:0/3721047004 >> v2:172.21.15.120:3300/0 conn(0x7f34b00cafa0 0x7f34b00cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b00cc220 con 0x7f34b006c300 2024-03-23T11:31:53.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f34a0001670 con 0x7f34b006c300 2024-03-23T11:31:53.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 >> v1:172.21.15.62:6789/0 conn(0x7f34b006c300 legacy=0x7f34b00ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494003970 0x7f3494003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940047c0 0x7f3494004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34b006c6e0 0x7f3494005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b00cd410 con 0x7f3494003970 2024-03-23T11:31:53.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b00cbf40 con 0x7f34940047c0 2024-03-23T11:31:53.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349400ba30 con 0x7f34b006c6e0 2024-03-23T11:31:53.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.892+0000 7f34af400640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494003970 0x7f3494003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.893+0000 7f34a7e00640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940047c0 0x7f3494004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.893+0000 7f34af400640 1 -- 172.21.15.62:0/3721047004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34b006c6e0 msgr2=0x7f3494005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:53.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.893+0000 7f34af400640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34b006c6e0 0x7f3494005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.893+0000 7f34af400640 1 -- 172.21.15.62:0/3721047004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940047c0 msgr2=0x7f3494004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:53.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.893+0000 7f34af400640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940047c0 0x7f3494004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.893+0000 7f34af400640 1 -- 172.21.15.62:0/3721047004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3494003520 con 0x7f3494003970 2024-03-23T11:31:53.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.893+0000 7f34a7e00640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34940047c0 0x7f3494004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:53.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.894+0000 7f34af400640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494003970 0x7f3494003d90 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f34a000c210 tx=0x7f34940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a46ff7a6d6cb8d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:53.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.894+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0005ad0 con 0x7f3494003970 2024-03-23T11:31:53.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.894+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a0005c30 con 0x7f3494003970 2024-03-23T11:31:53.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.894+0000 7f34aea00640 1 -- 172.21.15.62:0/3721047004 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a002c8a0 con 0x7f3494003970 2024-03-23T11:31:53.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.895+0000 7f34b6400640 1 -- 172.21.15.62:0/3721047004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494003970 msgr2=0x7f3494003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.895+0000 7f34b6400640 1 --2- 172.21.15.62:0/3721047004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3494003970 0x7f3494003d90 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f34a000c210 tx=0x7f34940057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.895+0000 7f34afe00640 1 -- 172.21.15.62:0/3721047004 reap_dead start 2024-03-23T11:31:53.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.895+0000 7f34b6400640 1 -- 172.21.15.62:0/3721047004 shutdown_connections 2024-03-23T11:31:53.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.895+0000 7f34b6400640 1 -- 172.21.15.62:0/3721047004 >> 172.21.15.62:0/3721047004 conn(0x7f34b00b02e0 msgr2=0x7f34b00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:53.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.895+0000 7f34b6400640 1 -- 172.21.15.62:0/3721047004 shutdown_connections 2024-03-23T11:31:53.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.895+0000 7f34b6400640 1 -- 172.21.15.62:0/3721047004 wait complete. 2024-03-23T11:31:53.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.897+0000 7f34b6400640 1 Processor -- start 2024-03-23T11:31:53.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.897+0000 7f34b6400640 1 -- start start 2024-03-23T11:31:53.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.898+0000 7f34b6400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3494003970 0x7f34b013d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.898+0000 7f34b6400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34b015daa0 0x7f34b00ad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.898+0000 7f34b6400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34b013ec60 0x7f34b01570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.898+0000 7f34b6400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b00ad600 con 0x7f34b013ec60 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.898+0000 7f34b6400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b013e7d0 con 0x7f34b015daa0 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.898+0000 7f34b6400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b015df20 con 0x7f3494003970 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.898+0000 7f34afe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34b013ec60 0x7f34b01570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.898+0000 7f34afe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34b013ec60 0x7f34b01570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48286/0 (socket says 172.21.15.62:48286) 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.898+0000 7f34afe00640 1 -- 172.21.15.62:0/2300053434 learned_addr learned my addr 172.21.15.62:0/2300053434 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.899+0000 7f34afe00640 1 -- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3494003970 msgr2=0x7f34b013d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.899+0000 7f34af400640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3494003970 0x7f34b013d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.899+0000 7f34a7e00640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34b015daa0 0x7f34b00ad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.899+0000 7f34afe00640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3494003970 0x7f34b013d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.899+0000 7f34afe00640 1 -- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34b015daa0 msgr2=0x7f34b00ad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.899+0000 7f34afe00640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34b015daa0 0x7f34b00ad0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.899+0000 7f34afe00640 1 -- 172.21.15.62:0/2300053434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b013dbc0 con 0x7f34b013ec60 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.899+0000 7f34a7e00640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34b015daa0 0x7f34b00ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:53.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.899+0000 7f34afe00640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34b013ec60 0x7f34b01570e0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f34b00cb380 tx=0x7f34a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:53.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.900+0000 7f34a7400640 1 -- 172.21.15.62:0/2300053434 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a8018070 con 0x7f34b013ec60 2024-03-23T11:31:53.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.900+0000 7f34a7400640 1 -- 172.21.15.62:0/2300053434 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a800eb90 con 0x7f34b013ec60 2024-03-23T11:31:53.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.900+0000 7f34b6400640 1 -- 172.21.15.62:0/2300053434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b013dea0 con 0x7f34b013ec60 2024-03-23T11:31:53.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.900+0000 7f34a7400640 1 -- 172.21.15.62:0/2300053434 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a8003990 con 0x7f34b013ec60 2024-03-23T11:31:53.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.900+0000 7f34b6400640 1 -- 172.21.15.62:0/2300053434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b013e460 con 0x7f34b013ec60 2024-03-23T11:31:53.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.901+0000 7f34b6400640 1 -- 172.21.15.62:0/2300053434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b013e5f0 con 0x7f34b013ec60 2024-03-23T11:31:53.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.902+0000 7f34a7400640 1 -- 172.21.15.62:0/2300053434 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f34a8014420 con 0x7f34b013ec60 2024-03-23T11:31:53.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.903+0000 7f34a7400640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f348c06a7a0 0x7f348c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:53.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.903+0000 7f34a7400640 1 -- 172.21.15.62:0/2300053434 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f34a8088d00 con 0x7f34b013ec60 2024-03-23T11:31:53.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.903+0000 7f34af400640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f348c06a7a0 0x7f348c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:53.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.904+0000 7f34af400640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f348c06a7a0 0x7f348c06cc60 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f34a0034000 tx=0x7f34a0007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:53.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:53.907+0000 7f34a7400640 1 -- 172.21.15.62:0/2300053434 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f34a8054040 con 0x7f34b013ec60 2024-03-23T11:31:54.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.105+0000 7f34b6400640 1 -- 172.21.15.62:0/2300053434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34b0066c90 con 0x7f34b013ec60 2024-03-23T11:31:54.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.106+0000 7f34a7400640 1 -- 172.21.15.62:0/2300053434 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7f34a80097a0 con 0x7f34b013ec60 2024-03-23T11:31:54.096 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:31:54.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.108+0000 7f34b6400640 1 -- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f348c06a7a0 msgr2=0x7f348c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:54.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.108+0000 7f34b6400640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f348c06a7a0 0x7f348c06cc60 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f34a0034000 tx=0x7f34a0007000 comp rx=0 tx=0).stop 2024-03-23T11:31:54.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.108+0000 7f34b6400640 1 -- 172.21.15.62:0/2300053434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34b013ec60 msgr2=0x7f34b01570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:54.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.108+0000 7f34b6400640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34b013ec60 0x7f34b01570e0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f34b00cb380 tx=0x7f34a800e4b0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.109+0000 7f34b6400640 1 -- 172.21.15.62:0/2300053434 shutdown_connections 2024-03-23T11:31:54.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.109+0000 7f34b6400640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f348c06a7a0 0x7f348c06cc60 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.109+0000 7f34b6400640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34b013ec60 0x7f34b01570e0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.109+0000 7f34b6400640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34b015daa0 0x7f34b00ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.109+0000 7f34b6400640 1 --2- 172.21.15.62:0/2300053434 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3494003970 0x7f34b013d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.109+0000 7f34b6400640 1 -- 172.21.15.62:0/2300053434 >> 172.21.15.62:0/2300053434 conn(0x7f34b00b02e0 msgr2=0x7f34b013c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:54.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.109+0000 7f34b6400640 1 -- 172.21.15.62:0/2300053434 shutdown_connections 2024-03-23T11:31:54.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.109+0000 7f34b6400640 1 -- 172.21.15.62:0/2300053434 wait complete. 2024-03-23T11:31:54.118 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:31:54.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.355+0000 7fd231e00640 1 Processor -- start 2024-03-23T11:31:54.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.356+0000 7fd231e00640 1 -- start start 2024-03-23T11:31:54.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.357+0000 7fd231e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd22c15fcf0 0x7fd22c0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.357+0000 7fd231e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd22c160420 0x7fd22c160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.357+0000 7fd231e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd22c0d06d0 0x7fd22c0d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.357+0000 7fd231e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22c160dc0 con 0x7fd22c160420 2024-03-23T11:31:54.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.357+0000 7fd231e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22c0d3540 con 0x7fd22c0d06d0 2024-03-23T11:31:54.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.357+0000 7fd231e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0ce510 con 0x7fd22c15fcf0 2024-03-23T11:31:54.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.358+0000 7fd22be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd22c160420 0x7fd22c160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.358+0000 7fd22be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd22c160420 0x7fd22c160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48290/0 (socket says 172.21.15.62:48290) 2024-03-23T11:31:54.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.358+0000 7fd22be00640 1 -- 172.21.15.62:0/1472510734 learned_addr learned my addr 172.21.15.62:0/1472510734 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:54.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.358+0000 7fd230a00640 1 --2- 172.21.15.62:0/1472510734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd22c15fcf0 0x7fd22c0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.358+0000 7fd231400640 1 --2- 172.21.15.62:0/1472510734 >> v2:172.21.15.120:3300/0 conn(0x7fd22c0d06d0 0x7fd22c0d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.358+0000 7fd22be00640 1 -- 172.21.15.62:0/1472510734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd22c15fcf0 msgr2=0x7fd22c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:54.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.358+0000 7fd22be00640 1 --2- 172.21.15.62:0/1472510734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd22c15fcf0 0x7fd22c0d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.358+0000 7fd22be00640 1 -- 172.21.15.62:0/1472510734 >> v2:172.21.15.120:3300/0 conn(0x7fd22c0d06d0 msgr2=0x7fd22c0d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:54.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.358+0000 7fd22be00640 1 --2- 172.21.15.62:0/1472510734 >> v2:172.21.15.120:3300/0 conn(0x7fd22c0d06d0 0x7fd22c0d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.358+0000 7fd22be00640 1 -- 172.21.15.62:0/1472510734 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c0cdc70 con 0x7fd22c160420 2024-03-23T11:31:54.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.360+0000 7fd22be00640 1 --2- 172.21.15.62:0/1472510734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c160420 0x7fd22c160880 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fd22c08c3e0 tx=0x7fd21802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448f816dc0b7bf13 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.360+0000 7fd22b400640 1 -- 172.21.15.62:0/1472510734 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2180317d0 con 0x7fd22c160420 2024-03-23T11:31:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.360+0000 7fd22b400640 1 -- 172.21.15.62:0/1472510734 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd218031930 con 0x7fd22c160420 2024-03-23T11:31:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.360+0000 7fd22b400640 1 -- 172.21.15.62:0/1472510734 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218031cc0 con 0x7fd22c160420 2024-03-23T11:31:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.361+0000 7fd231e00640 1 -- 172.21.15.62:0/1472510734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c160420 msgr2=0x7fd22c160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.361+0000 7fd231e00640 1 --2- 172.21.15.62:0/1472510734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c160420 0x7fd22c160880 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fd22c08c3e0 tx=0x7fd21802b870 comp rx=0 tx=0).stop 2024-03-23T11:31:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.361+0000 7fd231e00640 1 -- 172.21.15.62:0/1472510734 shutdown_connections 2024-03-23T11:31:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.361+0000 7fd231e00640 1 --2- 172.21.15.62:0/1472510734 >> v2:172.21.15.120:3300/0 conn(0x7fd22c0d06d0 0x7fd22c0d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.361+0000 7fd231e00640 1 --2- 172.21.15.62:0/1472510734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c160420 0x7fd22c160880 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.361+0000 7fd231e00640 1 --2- 172.21.15.62:0/1472510734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd22c15fcf0 0x7fd22c0d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.361+0000 7fd231e00640 1 -- 172.21.15.62:0/1472510734 >> 172.21.15.62:0/1472510734 conn(0x7fd22c089f60 msgr2=0x7fd22c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.361+0000 7fd231e00640 1 -- 172.21.15.62:0/1472510734 shutdown_connections 2024-03-23T11:31:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.361+0000 7fd231e00640 1 -- 172.21.15.62:0/1472510734 wait complete. 2024-03-23T11:31:54.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.363+0000 7fd231e00640 1 Processor -- start 2024-03-23T11:31:54.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.363+0000 7fd231e00640 1 -- start start 2024-03-23T11:31:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.364+0000 7fd231e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd22c0d06d0 0x7fd22c15ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.364+0000 7fd231e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c15fcf0 0x7fd22c0c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.364+0000 7fd231e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c160420 0x7fd22c15c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.364+0000 7fd231e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c7230 con 0x7fd22c15fcf0 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.364+0000 7fd231e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c7510 con 0x7fd22c160420 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.364+0000 7fd231e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c77f0 con 0x7fd22c0d06d0 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.364+0000 7fd22be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c15fcf0 0x7fd22c0c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.364+0000 7fd22be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c15fcf0 0x7fd22c0c68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48292/0 (socket says 172.21.15.62:48292) 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.364+0000 7fd22be00640 1 -- 172.21.15.62:0/2298824496 learned_addr learned my addr 172.21.15.62:0/2298824496 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.365+0000 7fd22be00640 1 -- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd22c0d06d0 msgr2=0x7fd22c15ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.365+0000 7fd231400640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c160420 0x7fd22c15c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.365+0000 7fd22be00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd22c0d06d0 0x7fd22c15ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.365+0000 7fd22be00640 1 -- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c160420 msgr2=0x7fd22c15c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:54.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.365+0000 7fd22be00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c160420 0x7fd22c15c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.365+0000 7fd22be00640 1 -- 172.21.15.62:0/2298824496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180094d0 con 0x7fd22c15fcf0 2024-03-23T11:31:54.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.365+0000 7fd231400640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c160420 0x7fd22c15c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:54.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.365+0000 7fd22be00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c15fcf0 0x7fd22c0c68e0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fd218000c00 tx=0x7fd2180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:54.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.365+0000 7fd228c00640 1 -- 172.21.15.62:0/2298824496 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218032870 con 0x7fd22c15fcf0 2024-03-23T11:31:54.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.365+0000 7fd228c00640 1 -- 172.21.15.62:0/2298824496 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd218032a10 con 0x7fd22c15fcf0 2024-03-23T11:31:54.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.365+0000 7fd231e00640 1 -- 172.21.15.62:0/2298824496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c0c7ad0 con 0x7fd22c15fcf0 2024-03-23T11:31:54.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.366+0000 7fd228c00640 1 -- 172.21.15.62:0/2298824496 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218032dc0 con 0x7fd22c15fcf0 2024-03-23T11:31:54.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.366+0000 7fd231e00640 1 -- 172.21.15.62:0/2298824496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c0c6e20 con 0x7fd22c15fcf0 2024-03-23T11:31:54.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.367+0000 7fd231e00640 1 -- 172.21.15.62:0/2298824496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22c09dce0 con 0x7fd22c15fcf0 2024-03-23T11:31:54.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.368+0000 7fd228c00640 1 -- 172.21.15.62:0/2298824496 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd218030070 con 0x7fd22c15fcf0 2024-03-23T11:31:54.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.369+0000 7fd228c00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd20c06a7a0 0x7fd20c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.369+0000 7fd228c00640 1 -- 172.21.15.62:0/2298824496 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd2180aacc0 con 0x7fd22c15fcf0 2024-03-23T11:31:54.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.369+0000 7fd230a00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd20c06a7a0 0x7fd20c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.371+0000 7fd230a00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd20c06a7a0 0x7fd20c06cc60 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd220006fd0 tx=0x7fd220008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:54.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.373+0000 7fd228c00640 1 -- 172.21.15.62:0/2298824496 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd22c09dce0 con 0x7fd22c15fcf0 2024-03-23T11:31:54.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.571+0000 7fd231e00640 1 -- 172.21.15.62:0/2298824496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd22c09dce0 con 0x7fd22c15fcf0 2024-03-23T11:31:54.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.572+0000 7fd228c00640 1 -- 172.21.15.62:0/2298824496 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd22c09dce0 con 0x7fd22c15fcf0 2024-03-23T11:31:54.562 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:31:54.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 -- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd20c06a7a0 msgr2=0x7fd20c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:54.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd20c06a7a0 0x7fd20c06cc60 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd220006fd0 tx=0x7fd220008040 comp rx=0 tx=0).stop 2024-03-23T11:31:54.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 -- 172.21.15.62:0/2298824496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c15fcf0 msgr2=0x7fd22c0c68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:54.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c15fcf0 0x7fd22c0c68e0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fd218000c00 tx=0x7fd2180046c0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 -- 172.21.15.62:0/2298824496 shutdown_connections 2024-03-23T11:31:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd20c06a7a0 0x7fd20c06cc60 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd22c15fcf0 0x7fd22c0c68e0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c160420 0x7fd22c15c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 --2- 172.21.15.62:0/2298824496 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd22c0d06d0 0x7fd22c15ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 -- 172.21.15.62:0/2298824496 >> 172.21.15.62:0/2298824496 conn(0x7fd22c089f60 msgr2=0x7fd22c0b6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:54.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 -- 172.21.15.62:0/2298824496 shutdown_connections 2024-03-23T11:31:54.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.575+0000 7fd231e00640 1 -- 172.21.15.62:0/2298824496 wait complete. 2024-03-23T11:31:54.585 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:31:54.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.822+0000 7f90b4a00640 1 Processor -- start 2024-03-23T11:31:54.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.822+0000 7f90b4a00640 1 -- start start 2024-03-23T11:31:54.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.824+0000 7f90b4a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f90b015c960 0x7f90b00cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.824+0000 7f90b4a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f90b00d13b0 0x7f90b00b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.824+0000 7f90b4a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b00d0810 con 0x7f90b00cd260 2024-03-23T11:31:54.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.824+0000 7f90b4a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b00cd9d0 con 0x7f90b015c960 2024-03-23T11:31:54.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.824+0000 7f90b4a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b00cdcb0 con 0x7f90b00d13b0 2024-03-23T11:31:54.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.824+0000 7f90af400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f90b015c960 0x7f90b00cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.824+0000 7f90af400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f90b015c960 0x7f90b00cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48298/0 (socket says 172.21.15.62:48298) 2024-03-23T11:31:54.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.824+0000 7f90af400640 1 -- 172.21.15.62:0/3582565757 learned_addr learned my addr 172.21.15.62:0/3582565757 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:54.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.824+0000 7f90afe00640 1 --2- 172.21.15.62:0/3582565757 >> v2:172.21.15.120:3300/0 conn(0x7f90b00d13b0 0x7f90b00b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.825+0000 7f90af400640 1 -- 172.21.15.62:0/3582565757 >> v2:172.21.15.120:3300/0 conn(0x7f90b00d13b0 msgr2=0x7f90b00b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:54.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.825+0000 7f90af400640 1 --2- 172.21.15.62:0/3582565757 >> v2:172.21.15.120:3300/0 conn(0x7f90b00d13b0 0x7f90b00b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.825+0000 7f90af400640 1 -- 172.21.15.62:0/3582565757 >> v1:172.21.15.62:6789/0 conn(0x7f90b00cd260 legacy=0x7f90b00d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:54.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.825+0000 7f90af400640 1 -- 172.21.15.62:0/3582565757 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b00cdf90 con 0x7f90b015c960 2024-03-23T11:31:54.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.825+0000 7f90ae000640 1 -- 172.21.15.62:0/3582565757 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369417043 0 0) 0x7f90a0001500 con 0x7f90b00cd260 2024-03-23T11:31:54.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.826+0000 7f90af400640 1 --2- 172.21.15.62:0/3582565757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b015c960 0x7f90b00cf790 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f90b00b8740 tx=0x7f909802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8bba522abf9780 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:54.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.827+0000 7f90ae000640 1 -- 172.21.15.62:0/3582565757 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909802f850 con 0x7f90b015c960 2024-03-23T11:31:54.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.827+0000 7f90ae000640 1 -- 172.21.15.62:0/3582565757 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909802f9b0 con 0x7f90b015c960 2024-03-23T11:31:54.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.827+0000 7f90ae000640 1 -- 172.21.15.62:0/3582565757 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909802fd40 con 0x7f90b015c960 2024-03-23T11:31:54.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.827+0000 7f90b4a00640 1 -- 172.21.15.62:0/3582565757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b015c960 msgr2=0x7f90b00cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:54.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.827+0000 7f90b4a00640 1 --2- 172.21.15.62:0/3582565757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b015c960 0x7f90b00cf790 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f90b00b8740 tx=0x7f909802ba30 comp rx=0 tx=0).stop 2024-03-23T11:31:54.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.827+0000 7f90b4a00640 1 -- 172.21.15.62:0/3582565757 shutdown_connections 2024-03-23T11:31:54.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.827+0000 7f90b4a00640 1 --2- 172.21.15.62:0/3582565757 >> v2:172.21.15.120:3300/0 conn(0x7f90b00d13b0 0x7f90b00b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.827+0000 7f90b4a00640 1 --2- 172.21.15.62:0/3582565757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b015c960 0x7f90b00cf790 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.827+0000 7f90b4a00640 1 -- 172.21.15.62:0/3582565757 >> 172.21.15.62:0/3582565757 conn(0x7f90b0089f60 msgr2=0x7f90b00d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:54.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.827+0000 7f90b4a00640 1 -- 172.21.15.62:0/3582565757 shutdown_connections 2024-03-23T11:31:54.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.827+0000 7f90b4a00640 1 -- 172.21.15.62:0/3582565757 wait complete. 2024-03-23T11:31:54.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.829+0000 7f90b4a00640 1 Processor -- start 2024-03-23T11:31:54.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.829+0000 7f90b4a00640 1 -- start start 2024-03-23T11:31:54.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.830+0000 7f90b4a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b00cd260 0x7f90b015e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.830+0000 7f90b4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00d13b0 0x7f90b0152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.830+0000 7f90b4a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b015c960 0x7f90b009db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.830+0000 7f90b4a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b015e920 con 0x7f90b00d13b0 2024-03-23T11:31:54.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.830+0000 7f90b4a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b015e6c0 con 0x7f90b00cd260 2024-03-23T11:31:54.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.830+0000 7f90b4a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b00ca980 con 0x7f90b015c960 2024-03-23T11:31:54.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90aea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00d13b0 0x7f90b0152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90aea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00d13b0 0x7f90b0152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48304/0 (socket says 172.21.15.62:48304) 2024-03-23T11:31:54.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90aea00640 1 -- 172.21.15.62:0/2320263368 learned_addr learned my addr 172.21.15.62:0/2320263368 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:54.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90af400640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b00cd260 0x7f90b015e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90afe00640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b015c960 0x7f90b009db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90aea00640 1 -- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b015c960 msgr2=0x7f90b009db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:54.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90aea00640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b015c960 0x7f90b009db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90aea00640 1 -- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b00cd260 msgr2=0x7f90b015e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:54.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90aea00640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b00cd260 0x7f90b015e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:54.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90aea00640 1 -- 172.21.15.62:0/2320263368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90980094d0 con 0x7f90b00d13b0 2024-03-23T11:31:54.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90afe00640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b015c960 0x7f90b009db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:54.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.831+0000 7f90aea00640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00d13b0 0x7f90b0152e80 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f90a000c890 tx=0x7f90a000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:54.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.832+0000 7f90a7400640 1 -- 172.21.15.62:0/2320263368 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a0003da0 con 0x7f90b00d13b0 2024-03-23T11:31:54.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.832+0000 7f90a7400640 1 -- 172.21.15.62:0/2320263368 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a0005070 con 0x7f90b00d13b0 2024-03-23T11:31:54.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.832+0000 7f90a7400640 1 -- 172.21.15.62:0/2320263368 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a0005420 con 0x7f90b00d13b0 2024-03-23T11:31:54.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.832+0000 7f90b4a00640 1 -- 172.21.15.62:0/2320263368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b00cac60 con 0x7f90b00d13b0 2024-03-23T11:31:54.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.832+0000 7f90b4a00640 1 -- 172.21.15.62:0/2320263368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b00cb220 con 0x7f90b00d13b0 2024-03-23T11:31:54.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.833+0000 7f90b4a00640 1 -- 172.21.15.62:0/2320263368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b0066bb0 con 0x7f90b00d13b0 2024-03-23T11:31:54.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.834+0000 7f90a7400640 1 -- 172.21.15.62:0/2320263368 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f90a0005580 con 0x7f90b00d13b0 2024-03-23T11:31:54.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.835+0000 7f90a7400640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f908c06a7a0 0x7f908c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:54.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.835+0000 7f90a7400640 1 -- 172.21.15.62:0/2320263368 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f90a0088690 con 0x7f90b00d13b0 2024-03-23T11:31:54.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.836+0000 7f90af400640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f908c06a7a0 0x7f908c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.837+0000 7f90af400640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f908c06a7a0 0x7f908c06cc60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9098036000 tx=0x7f9098008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:54.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:54.840+0000 7f90a7400640 1 -- 172.21.15.62:0/2320263368 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f90a00539d0 con 0x7f90b00d13b0 2024-03-23T11:31:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.052+0000 7f90b4a00640 1 -- 172.21.15.62:0/2320263368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f90b00d0810 con 0x7f90b00d13b0 2024-03-23T11:31:55.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.053+0000 7f90a7400640 1 -- 172.21.15.62:0/2320263368 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v120) v1 ==== 80+0+0 (secure 0 0 0) 0x7f90b00d0810 con 0x7f90b00d13b0 2024-03-23T11:31:55.043 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:31:55.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.056+0000 7f90b4a00640 1 -- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f908c06a7a0 msgr2=0x7f908c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.056+0000 7f90b4a00640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f908c06a7a0 0x7f908c06cc60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9098036000 tx=0x7f9098008030 comp rx=0 tx=0).stop 2024-03-23T11:31:55.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.056+0000 7f90b4a00640 1 -- 172.21.15.62:0/2320263368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00d13b0 msgr2=0x7f90b0152e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.056+0000 7f90b4a00640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00d13b0 0x7f90b0152e80 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f90a000c890 tx=0x7f90a000cd50 comp rx=0 tx=0).stop 2024-03-23T11:31:55.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.056+0000 7f90b4a00640 1 -- 172.21.15.62:0/2320263368 shutdown_connections 2024-03-23T11:31:55.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.056+0000 7f90b4a00640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f908c06a7a0 0x7f908c06cc60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.056+0000 7f90b4a00640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b00d13b0 0x7f90b0152e80 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.056+0000 7f90b4a00640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90b015c960 0x7f90b009db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.056+0000 7f90b4a00640 1 --2- 172.21.15.62:0/2320263368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b00cd260 0x7f90b015e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.056+0000 7f90b4a00640 1 -- 172.21.15.62:0/2320263368 >> 172.21.15.62:0/2320263368 conn(0x7f90b0089f60 msgr2=0x7f90b00af270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:55.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.057+0000 7f90b4a00640 1 -- 172.21.15.62:0/2320263368 shutdown_connections 2024-03-23T11:31:55.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.057+0000 7f90b4a00640 1 -- 172.21.15.62:0/2320263368 wait complete. 2024-03-23T11:31:55.066 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:31:55.066 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:31:55.116 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:31:55.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.395+0000 7f4fff200640 1 Processor -- start 2024-03-23T11:31:55.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.395+0000 7f4fff200640 1 -- start start 2024-03-23T11:31:55.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.397+0000 7f4fff200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f500015fcf0 0x7f50000d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.397+0000 7f4fff200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50000d14f0 con 0x7f5000160420 2024-03-23T11:31:55.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.397+0000 7f4fff200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50000d3540 con 0x7f50000b6dc0 2024-03-23T11:31:55.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.397+0000 7f4fff200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50000ce510 con 0x7f500015fcf0 2024-03-23T11:31:55.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.397+0000 7f4ff5400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5000160420 0x7f50000d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:34326/0 (socket says 172.21.15.62:34326) 2024-03-23T11:31:55.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.397+0000 7f4ff5400640 1 -- 172.21.15.62:0/3582810867 learned_addr learned my addr 172.21.15.62:0/3582810867 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:55.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffde00640 1 --2- 172.21.15.62:0/3582810867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f500015fcf0 0x7f50000d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948032750 0 0) 0x7f50000d14f0 con 0x7f5000160420 2024-03-23T11:31:55.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe4003520 con 0x7f5000160420 2024-03-23T11:31:55.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4049425474 0 0) 0x7f4fe4003520 con 0x7f5000160420 2024-03-23T11:31:55.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f500015fcf0 msgr2=0x7f50000d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:55.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffd400640 1 --2- 172.21.15.62:0/3582810867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f500015fcf0 0x7f50000d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 >> v1:172.21.15.120:6789/0 conn(0x7f50000b6dc0 legacy=0x7f50000d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50000cdc70 con 0x7f5000160420 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f4fe8001670 con 0x7f5000160420 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 >> v1:172.21.15.62:6789/0 conn(0x7f5000160420 legacy=0x7f50000d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffd400640 1 --2- 172.21.15.62:0/3582810867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe40039a0 0x7f4fe4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.398+0000 7f4ffd400640 1 --2- 172.21.15.62:0/3582810867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fe40047c0 0x7f4fe4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.399+0000 7f4ffd400640 1 --2- 172.21.15.62:0/3582810867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f500015fcf0 0x7f4fe400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.399+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50000ce510 con 0x7f4fe40047c0 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.399+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe40054e0 con 0x7f4fe40039a0 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.399+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe400bec0 con 0x7f500015fcf0 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.399+0000 7f4ff5400640 1 --2- 172.21.15.62:0/3582810867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fe40047c0 0x7f4fe4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.399+0000 7f4ffde00640 1 --2- 172.21.15.62:0/3582810867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe40039a0 0x7f4fe4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.399+0000 7f4ff5400640 1 -- 172.21.15.62:0/3582810867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f500015fcf0 msgr2=0x7f4fe400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:55.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.399+0000 7f4ff5400640 1 --2- 172.21.15.62:0/3582810867 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f500015fcf0 0x7f4fe400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.399+0000 7f4ff5400640 1 -- 172.21.15.62:0/3582810867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe40039a0 msgr2=0x7f4fe4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:55.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.399+0000 7f4ff5400640 1 --2- 172.21.15.62:0/3582810867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe40039a0 0x7f4fe4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.399+0000 7f4ff5400640 1 -- 172.21.15.62:0/3582810867 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe4003520 con 0x7f4fe40047c0 2024-03-23T11:31:55.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.400+0000 7f4ff5400640 1 --2- 172.21.15.62:0/3582810867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fe40047c0 0x7f4fe4004c40 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800c0e0 tx=0x7f4fe40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584cd50d573cdf93 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:55.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.401+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8005ad0 con 0x7f4fe40047c0 2024-03-23T11:31:55.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.401+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe8005c30 con 0x7f4fe40047c0 2024-03-23T11:31:55.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.401+0000 7f4ffd400640 1 -- 172.21.15.62:0/3582810867 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe802c810 con 0x7f4fe40047c0 2024-03-23T11:31:55.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.401+0000 7f4fff200640 1 -- 172.21.15.62:0/3582810867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fe40047c0 msgr2=0x7f4fe4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.401+0000 7f4fff200640 1 --2- 172.21.15.62:0/3582810867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fe40047c0 0x7f4fe4004c40 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800c0e0 tx=0x7f4fe40057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.401+0000 7f4ffe800640 1 -- 172.21.15.62:0/3582810867 reap_dead start 2024-03-23T11:31:55.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.401+0000 7f4fff200640 1 -- 172.21.15.62:0/3582810867 shutdown_connections 2024-03-23T11:31:55.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.401+0000 7f4fff200640 1 -- 172.21.15.62:0/3582810867 >> 172.21.15.62:0/3582810867 conn(0x7f5000089f60 msgr2=0x7f50000d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:55.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.401+0000 7f4fff200640 1 -- 172.21.15.62:0/3582810867 shutdown_connections 2024-03-23T11:31:55.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.401+0000 7f4fff200640 1 -- 172.21.15.62:0/3582810867 wait complete. 2024-03-23T11:31:55.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.403+0000 7f4fff200640 1 Processor -- start 2024-03-23T11:31:55.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.403+0000 7f4fff200640 1 -- start start 2024-03-23T11:31:55.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.404+0000 7f4fff200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe40047c0 0x7f5000148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.404+0000 7f4fff200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f500015b0b0 0x7f50000c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.404+0000 7f4fff200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5000160800 0x7f5000160c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.404+0000 7f4fff200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50001611c0 con 0x7f500015b0b0 2024-03-23T11:31:55.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.404+0000 7f4fff200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50001469e0 con 0x7f4fe40047c0 2024-03-23T11:31:55.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.404+0000 7f4fff200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50001492c0 con 0x7f5000160800 2024-03-23T11:31:55.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.405+0000 7f4ff5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f500015b0b0 0x7f50000c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.405+0000 7f4ff5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f500015b0b0 0x7f50000c8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48322/0 (socket says 172.21.15.62:48322) 2024-03-23T11:31:55.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.405+0000 7f4ff5400640 1 -- 172.21.15.62:0/2698496555 learned_addr learned my addr 172.21.15.62:0/2698496555 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:55.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.405+0000 7f4ffde00640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe40047c0 0x7f5000148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.405+0000 7f4ffe800640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5000160800 0x7f5000160c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.405+0000 7f4ff5400640 1 -- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5000160800 msgr2=0x7f5000160c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.405+0000 7f4ff5400640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5000160800 0x7f5000160c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.405+0000 7f4ff5400640 1 -- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe40047c0 msgr2=0x7f5000148a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.405+0000 7f4ff5400640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe40047c0 0x7f5000148a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.405+0000 7f4ff5400640 1 -- 172.21.15.62:0/2698496555 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50001495a0 con 0x7f500015b0b0 2024-03-23T11:31:55.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.406+0000 7f4ff5400640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f500015b0b0 0x7f50000c8750 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f4fe802c3b0 tx=0x7f4fe8001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:55.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.406+0000 7f4ff6a00640 1 -- 172.21.15.62:0/2698496555 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8039070 con 0x7f500015b0b0 2024-03-23T11:31:55.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.406+0000 7f4ff6a00640 1 -- 172.21.15.62:0/2698496555 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe802cc20 con 0x7f500015b0b0 2024-03-23T11:31:55.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.406+0000 7f4ff6a00640 1 -- 172.21.15.62:0/2698496555 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe803c040 con 0x7f500015b0b0 2024-03-23T11:31:55.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.406+0000 7f4fff200640 1 -- 172.21.15.62:0/2698496555 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5000149880 con 0x7f500015b0b0 2024-03-23T11:31:55.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.406+0000 7f4fff200640 1 -- 172.21.15.62:0/2698496555 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5000148f60 con 0x7f500015b0b0 2024-03-23T11:31:55.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.407+0000 7f4fff200640 1 -- 172.21.15.62:0/2698496555 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5000066bb0 con 0x7f500015b0b0 2024-03-23T11:31:55.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.409+0000 7f4ff6a00640 1 -- 172.21.15.62:0/2698496555 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4fe8032070 con 0x7f500015b0b0 2024-03-23T11:31:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.409+0000 7f4ff6a00640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4fdc06a7a0 0x7f4fdc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.410+0000 7f4ff6a00640 1 -- 172.21.15.62:0/2698496555 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4fe80aa1c0 con 0x7f500015b0b0 2024-03-23T11:31:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.410+0000 7f4ffde00640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4fdc06a7a0 0x7f4fdc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.411+0000 7f4ffde00640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4fdc06a7a0 0x7f4fdc06cc60 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009a90 tx=0x7f4ff0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:55.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.414+0000 7f4ff6a00640 1 -- 172.21.15.62:0/2698496555 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4fe8075500 con 0x7f500015b0b0 2024-03-23T11:31:55.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.619+0000 7f4fff200640 1 -- 172.21.15.62:0/2698496555 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50000cca80 con 0x7f500015b0b0 2024-03-23T11:31:55.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.621+0000 7f4ff6a00640 1 -- 172.21.15.62:0/2698496555 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16152 (secure 0 0 0) 0x7f4fe8078b20 con 0x7f500015b0b0 2024-03-23T11:31:55.611 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:31:55.611 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":120,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:31:44.554756+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:31:55.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 -- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4fdc06a7a0 msgr2=0x7f4fdc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4fdc06a7a0 0x7f4fdc06cc60 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009a90 tx=0x7f4ff0008040 comp rx=0 tx=0).stop 2024-03-23T11:31:55.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 -- 172.21.15.62:0/2698496555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f500015b0b0 msgr2=0x7f50000c8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f500015b0b0 0x7f50000c8750 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f4fe802c3b0 tx=0x7f4fe8001930 comp rx=0 tx=0).stop 2024-03-23T11:31:55.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 -- 172.21.15.62:0/2698496555 shutdown_connections 2024-03-23T11:31:55.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4fdc06a7a0 0x7f4fdc06cc60 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f500015b0b0 0x7f50000c8750 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5000160800 0x7f5000160c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 --2- 172.21.15.62:0/2698496555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4fe40047c0 0x7f5000148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 -- 172.21.15.62:0/2698496555 >> 172.21.15.62:0/2698496555 conn(0x7f5000089f60 msgr2=0x7f500006c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:55.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 -- 172.21.15.62:0/2698496555 shutdown_connections 2024-03-23T11:31:55.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.624+0000 7f4fff200640 1 -- 172.21.15.62:0/2698496555 wait complete. 2024-03-23T11:31:55.636 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:31:22.127079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:31:23.062387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:31:55.637 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:31:55.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.875+0000 7f7cb5a00640 1 Processor -- start 2024-03-23T11:31:55.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.875+0000 7f7cb5a00640 1 -- start start 2024-03-23T11:31:55.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.877+0000 7f7cb5a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7cb0160310 0x7f7cb00ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.877+0000 7f7cb5a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb01608d0 con 0x7f7cb015c830 2024-03-23T11:31:55.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.877+0000 7f7cb5a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb0160bb0 con 0x7f7cb00ce250 2024-03-23T11:31:55.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.877+0000 7f7cb5a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb00addb0 con 0x7f7cb0160310 2024-03-23T11:31:55.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.877+0000 7f7cafe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7cb015c830 0x7f7cb00ab980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:34332/0 (socket says 172.21.15.62:34332) 2024-03-23T11:31:55.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.877+0000 7f7cafe00640 1 -- 172.21.15.62:0/1037144105 learned_addr learned my addr 172.21.15.62:0/1037144105 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:55.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358787353 0 0) 0x7f7cb01608d0 con 0x7f7cb015c830 2024-03-23T11:31:55.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caf400640 1 --2- 172.21.15.62:0/1037144105 >> v2:172.21.15.120:3300/0 conn(0x7f7cb0160310 0x7f7cb00ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c94003520 con 0x7f7cb015c830 2024-03-23T11:31:55.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 753773411 0 0) 0x7f7c94003520 con 0x7f7cb015c830 2024-03-23T11:31:55.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 >> v2:172.21.15.120:3300/0 conn(0x7f7cb0160310 msgr2=0x7f7cb00ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:55.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 --2- 172.21.15.62:0/1037144105 >> v2:172.21.15.120:3300/0 conn(0x7f7cb0160310 0x7f7cb00ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 >> v1:172.21.15.120:6789/0 conn(0x7f7cb00ce250 legacy=0x7f7cb015d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb0160e90 con 0x7f7cb015c830 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7ca0001670 con 0x7f7cb015c830 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 >> v1:172.21.15.62:6789/0 conn(0x7f7cb015c830 legacy=0x7f7cb00ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 --2- 172.21.15.62:0/1037144105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 0x7f7c94003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 --2- 172.21.15.62:0/1037144105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c940047c0 0x7f7c94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 --2- 172.21.15.62:0/1037144105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c940051d0 0x7f7c9400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9400c0f0 con 0x7f7c940039c0 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9400c260 con 0x7f7c940051d0 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.878+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9400c420 con 0x7f7c940047c0 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.879+0000 7f7cafe00640 1 --2- 172.21.15.62:0/1037144105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 0x7f7c94003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.879+0000 7f7cb5000640 1 --2- 172.21.15.62:0/1037144105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c940051d0 0x7f7c9400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.879+0000 7f7cafe00640 1 -- 172.21.15.62:0/1037144105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c940047c0 msgr2=0x7f7c94004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.879+0000 7f7cafe00640 1 --2- 172.21.15.62:0/1037144105 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c940047c0 0x7f7c94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.879+0000 7f7cafe00640 1 -- 172.21.15.62:0/1037144105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c940051d0 msgr2=0x7f7c9400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:55.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.879+0000 7f7cafe00640 1 --2- 172.21.15.62:0/1037144105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c940051d0 0x7f7c9400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.879+0000 7f7cafe00640 1 -- 172.21.15.62:0/1037144105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c94003520 con 0x7f7c940039c0 2024-03-23T11:31:55.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.879+0000 7f7cb5000640 1 -- 172.21.15.62:0/1037144105 reap_dead start 2024-03-23T11:31:55.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.880+0000 7f7cafe00640 1 --2- 172.21.15.62:0/1037144105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 0x7f7c94003de0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000c0e0 tx=0x7f7c940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee316dc62579d3b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:55.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.881+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0005ad0 con 0x7f7c940039c0 2024-03-23T11:31:55.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.881+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca0005c30 con 0x7f7c940039c0 2024-03-23T11:31:55.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.881+0000 7f7caea00640 1 -- 172.21.15.62:0/1037144105 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca002c7f0 con 0x7f7c940039c0 2024-03-23T11:31:55.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.881+0000 7f7cb5a00640 1 -- 172.21.15.62:0/1037144105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 msgr2=0x7f7c94003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.881+0000 7f7cb5a00640 1 --2- 172.21.15.62:0/1037144105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 0x7f7c94003de0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000c0e0 tx=0x7f7c940057f0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.881+0000 7f7cb5a00640 1 -- 172.21.15.62:0/1037144105 shutdown_connections 2024-03-23T11:31:55.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.881+0000 7f7cb5a00640 1 --2- 172.21.15.62:0/1037144105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 0x7f7c94003de0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.881+0000 7f7cb5a00640 1 -- 172.21.15.62:0/1037144105 >> 172.21.15.62:0/1037144105 conn(0x7f7cb0066050 msgr2=0x7f7cb0069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:55.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.881+0000 7f7cb5a00640 1 -- 172.21.15.62:0/1037144105 shutdown_connections 2024-03-23T11:31:55.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.881+0000 7f7cb5a00640 1 -- 172.21.15.62:0/1037144105 wait complete. 2024-03-23T11:31:55.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.883+0000 7f7cb5a00640 1 Processor -- start 2024-03-23T11:31:55.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.883+0000 7f7cb5a00640 1 -- start start 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.884+0000 7f7cb5a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 0x7f7cb006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.884+0000 7f7cb5a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cb0147a00 0x7f7cb0147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.884+0000 7f7cb5a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cb00c7ca0 0x7f7cb00c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.884+0000 7f7cb5a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb01483a0 con 0x7f7c940039c0 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.884+0000 7f7cb5a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb01f5260 con 0x7f7cb00c7ca0 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.884+0000 7f7cb5a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb01f5500 con 0x7f7cb0147a00 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.885+0000 7f7cafe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 0x7f7cb006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.885+0000 7f7cafe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 0x7f7cb006d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48330/0 (socket says 172.21.15.62:48330) 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.885+0000 7f7cafe00640 1 -- 172.21.15.62:0/3636103597 learned_addr learned my addr 172.21.15.62:0/3636103597 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.885+0000 7f7cafe00640 1 -- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cb0147a00 msgr2=0x7f7cb0147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.885+0000 7f7cafe00640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cb0147a00 0x7f7cb0147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.885+0000 7f7cafe00640 1 -- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cb00c7ca0 msgr2=0x7f7cb00c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:55.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.885+0000 7f7cafe00640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cb00c7ca0 0x7f7cb00c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:55.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.885+0000 7f7cafe00640 1 -- 172.21.15.62:0/3636103597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb01f57e0 con 0x7f7c940039c0 2024-03-23T11:31:55.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.885+0000 7f7cafe00640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 0x7f7cb006d790 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f7ca002c390 tx=0x7f7ca0001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:55.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.886+0000 7f7ca7e00640 1 -- 172.21.15.62:0/3636103597 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca003c070 con 0x7f7c940039c0 2024-03-23T11:31:55.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.886+0000 7f7ca7e00640 1 -- 172.21.15.62:0/3636103597 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca002ca00 con 0x7f7c940039c0 2024-03-23T11:31:55.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.886+0000 7f7ca7e00640 1 -- 172.21.15.62:0/3636103597 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca002cd70 con 0x7f7c940039c0 2024-03-23T11:31:55.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.886+0000 7f7cb5a00640 1 -- 172.21.15.62:0/3636103597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb01f5ac0 con 0x7f7c940039c0 2024-03-23T11:31:55.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.886+0000 7f7cb5a00640 1 -- 172.21.15.62:0/3636103597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb00c8670 con 0x7f7c940039c0 2024-03-23T11:31:55.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.887+0000 7f7cb5a00640 1 -- 172.21.15.62:0/3636103597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb0066bb0 con 0x7f7c940039c0 2024-03-23T11:31:55.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.888+0000 7f7ca7e00640 1 -- 172.21.15.62:0/3636103597 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7ca0032070 con 0x7f7c940039c0 2024-03-23T11:31:55.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.889+0000 7f7ca7e00640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c9006a7d0 0x7f7c9006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:55.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.889+0000 7f7ca7e00640 1 -- 172.21.15.62:0/3636103597 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7ca00a97f0 con 0x7f7c940039c0 2024-03-23T11:31:55.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.890+0000 7f7caf400640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c9006a7d0 0x7f7c9006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:55.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.891+0000 7f7caf400640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c9006a7d0 0x7f7c9006cc90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7cb014ed60 tx=0x7f7c9c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:55.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:55.894+0000 7f7ca7e00640 1 -- 172.21.15.62:0/3636103597 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7ca0074d90 con 0x7f7c940039c0 2024-03-23T11:31:56.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.095+0000 7f7cb5a00640 1 -- 172.21.15.62:0/3636103597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7cb00b2930 con 0x7f7c940039c0 2024-03-23T11:31:56.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.096+0000 7f7ca7e00640 1 -- 172.21.15.62:0/3636103597 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ca0045020 con 0x7f7c940039c0 2024-03-23T11:31:56.086 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:31:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 -- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c9006a7d0 msgr2=0x7f7c9006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c9006a7d0 0x7f7c9006cc90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7cb014ed60 tx=0x7f7c9c008040 comp rx=0 tx=0).stop 2024-03-23T11:31:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 -- 172.21.15.62:0/3636103597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 msgr2=0x7f7cb006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 0x7f7cb006d790 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f7ca002c390 tx=0x7f7ca0001930 comp rx=0 tx=0).stop 2024-03-23T11:31:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 -- 172.21.15.62:0/3636103597 shutdown_connections 2024-03-23T11:31:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7c9006a7d0 0x7f7c9006cc90 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cb00c7ca0 0x7f7cb00c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cb0147a00 0x7f7cb0147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 --2- 172.21.15.62:0/3636103597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c940039c0 0x7f7cb006d790 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 -- 172.21.15.62:0/3636103597 >> 172.21.15.62:0/3636103597 conn(0x7f7cb0066050 msgr2=0x7f7cb00ae1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 -- 172.21.15.62:0/3636103597 shutdown_connections 2024-03-23T11:31:56.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.099+0000 7f7cb5a00640 1 -- 172.21.15.62:0/3636103597 wait complete. 2024-03-23T11:31:56.109 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:31:56.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.339+0000 7f0f5b200640 1 Processor -- start 2024-03-23T11:31:56.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.340+0000 7f0f5b200640 1 -- start start 2024-03-23T11:31:56.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.341+0000 7f0f5b200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0f5c160420 0x7f0f5c1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.341+0000 7f0f5b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0f5c0b5db0 0x7f0f5c0b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.341+0000 7f0f5b200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5c0ce510 con 0x7f0f5c15fcf0 2024-03-23T11:31:56.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.341+0000 7f0f5b200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5c160df0 con 0x7f0f5c160420 2024-03-23T11:31:56.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.341+0000 7f0f5b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5c0cdc70 con 0x7f0f5c0b5db0 2024-03-23T11:31:56.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.342+0000 7f0f53e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0f5c160420 0x7f0f5c1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.342+0000 7f0f59e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0f5c15fcf0 0x7f0f5c0d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:34348/0 (socket says 172.21.15.62:34348) 2024-03-23T11:31:56.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.342+0000 7f0f59e00640 1 -- 172.21.15.62:0/1416668962 learned_addr learned my addr 172.21.15.62:0/1416668962 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:31:56.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.342+0000 7f0f5a800640 1 --2- 172.21.15.62:0/1416668962 >> v2:172.21.15.120:3300/0 conn(0x7f0f5c0b5db0 0x7f0f5c0b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.342+0000 7f0f53e00640 1 -- 172.21.15.62:0/1416668962 >> v2:172.21.15.120:3300/0 conn(0x7f0f5c0b5db0 msgr2=0x7f0f5c0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:56.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.342+0000 7f0f53e00640 1 --2- 172.21.15.62:0/1416668962 >> v2:172.21.15.120:3300/0 conn(0x7f0f5c0b5db0 0x7f0f5c0b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.342+0000 7f0f53e00640 1 -- 172.21.15.62:0/1416668962 >> v1:172.21.15.62:6789/0 conn(0x7f0f5c15fcf0 legacy=0x7f0f5c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:56.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.342+0000 7f0f53e00640 1 -- 172.21.15.62:0/1416668962 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c0cdf50 con 0x7f0f5c160420 2024-03-23T11:31:56.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.342+0000 7f0f59400640 1 -- 172.21.15.62:0/1416668962 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206182369 0 0) 0x7f0f5c0ce510 con 0x7f0f5c15fcf0 2024-03-23T11:31:56.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.344+0000 7f0f53e00640 1 --2- 172.21.15.62:0/1416668962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f5c160420 0x7f0f5c1608b0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c0b52b0 tx=0x7f0f4402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afc9b6c108f9bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:56.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.344+0000 7f0f59400640 1 -- 172.21.15.62:0/1416668962 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4402e070 con 0x7f0f5c160420 2024-03-23T11:31:56.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.344+0000 7f0f59400640 1 -- 172.21.15.62:0/1416668962 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f440315a0 con 0x7f0f5c160420 2024-03-23T11:31:56.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.344+0000 7f0f59400640 1 -- 172.21.15.62:0/1416668962 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f44031930 con 0x7f0f5c160420 2024-03-23T11:31:56.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.345+0000 7f0f5b200640 1 -- 172.21.15.62:0/1416668962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f5c160420 msgr2=0x7f0f5c1608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:56.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.345+0000 7f0f5b200640 1 --2- 172.21.15.62:0/1416668962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f5c160420 0x7f0f5c1608b0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c0b52b0 tx=0x7f0f4402b840 comp rx=0 tx=0).stop 2024-03-23T11:31:56.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.345+0000 7f0f5b200640 1 -- 172.21.15.62:0/1416668962 shutdown_connections 2024-03-23T11:31:56.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.345+0000 7f0f5b200640 1 --2- 172.21.15.62:0/1416668962 >> v2:172.21.15.120:3300/0 conn(0x7f0f5c0b5db0 0x7f0f5c0b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.345+0000 7f0f5b200640 1 --2- 172.21.15.62:0/1416668962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f5c160420 0x7f0f5c1608b0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.345+0000 7f0f5b200640 1 -- 172.21.15.62:0/1416668962 >> 172.21.15.62:0/1416668962 conn(0x7f0f5c089f60 msgr2=0x7f0f5c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:56.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.345+0000 7f0f5b200640 1 -- 172.21.15.62:0/1416668962 shutdown_connections 2024-03-23T11:31:56.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.345+0000 7f0f5b200640 1 -- 172.21.15.62:0/1416668962 wait complete. 2024-03-23T11:31:56.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.347+0000 7f0f5b200640 1 Processor -- start 2024-03-23T11:31:56.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.347+0000 7f0f5b200640 1 -- start start 2024-03-23T11:31:56.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.348+0000 7f0f5b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f5c0b5db0 0x7f0f5c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.348+0000 7f0f5b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f5c15fcf0 0x7f0f5c153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.348+0000 7f0f5b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f5c160420 0x7f0f5c153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.348+0000 7f0f5b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c154530 con 0x7f0f5c160420 2024-03-23T11:31:56.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.348+0000 7f0f5b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1f58e0 con 0x7f0f5c15fcf0 2024-03-23T11:31:56.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.348+0000 7f0f5b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1f30b0 con 0x7f0f5c0b5db0 2024-03-23T11:31:56.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.348+0000 7f0f5a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f5c160420 0x7f0f5c153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.348+0000 7f0f5a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f5c160420 0x7f0f5c153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48354/0 (socket says 172.21.15.62:48354) 2024-03-23T11:31:56.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.348+0000 7f0f5a800640 1 -- 172.21.15.62:0/33719608 learned_addr learned my addr 172.21.15.62:0/33719608 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.348+0000 7f0f53e00640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f5c15fcf0 0x7f0f5c153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.348+0000 7f0f59e00640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f5c0b5db0 0x7f0f5c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f5a800640 1 -- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f5c0b5db0 msgr2=0x7f0f5c06c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f5a800640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f5c0b5db0 0x7f0f5c06c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f5a800640 1 -- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f5c15fcf0 msgr2=0x7f0f5c153ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f5a800640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f5c15fcf0 0x7f0f5c153ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f5a800640 1 -- 172.21.15.62:0/33719608 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f44009400 con 0x7f0f5c160420 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f53e00640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f5c15fcf0 0x7f0f5c153ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f5a800640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f5c160420 0x7f0f5c153ff0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f0f54007e90 tx=0x7f0f5400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f52000640 1 -- 172.21.15.62:0/33719608 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54014420 con 0x7f0f5c160420 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f52000640 1 -- 172.21.15.62:0/33719608 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f54014580 con 0x7f0f5c160420 2024-03-23T11:31:56.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f52000640 1 -- 172.21.15.62:0/33719608 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54014930 con 0x7f0f5c160420 2024-03-23T11:31:56.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f5b200640 1 -- 172.21.15.62:0/33719608 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c1f5b30 con 0x7f0f5c160420 2024-03-23T11:31:56.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.349+0000 7f0f5b200640 1 -- 172.21.15.62:0/33719608 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c1f60f0 con 0x7f0f5c160420 2024-03-23T11:31:56.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.350+0000 7f0f5b200640 1 -- 172.21.15.62:0/33719608 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5c0672c0 con 0x7f0f5c160420 2024-03-23T11:31:56.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.352+0000 7f0f52000640 1 -- 172.21.15.62:0/33719608 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0f54014a90 con 0x7f0f5c160420 2024-03-23T11:31:56.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.353+0000 7f0f52000640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f3806a7a0 0x7f0f3806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.353+0000 7f0f52000640 1 -- 172.21.15.62:0/33719608 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0f540896b0 con 0x7f0f5c160420 2024-03-23T11:31:56.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.353+0000 7f0f59e00640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f3806a7a0 0x7f0f3806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.354+0000 7f0f59e00640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f3806a7a0 0x7f0f3806cc60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c000f60 tx=0x7f0f4c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:56.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.357+0000 7f0f52000640 1 -- 172.21.15.62:0/33719608 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f54054b20 con 0x7f0f5c160420 2024-03-23T11:31:56.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.560+0000 7f0f5b200640 1 -- 172.21.15.62:0/33719608 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0f5c066bb0 con 0x7f0f5c160420 2024-03-23T11:31:56.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.561+0000 7f0f52000640 1 -- 172.21.15.62:0/33719608 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0f54058140 con 0x7f0f5c160420 2024-03-23T11:31:56.551 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:31:56.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 -- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f3806a7a0 msgr2=0x7f0f3806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:56.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f3806a7a0 0x7f0f3806cc60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c000f60 tx=0x7f0f4c009040 comp rx=0 tx=0).stop 2024-03-23T11:31:56.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 -- 172.21.15.62:0/33719608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f5c160420 msgr2=0x7f0f5c153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:56.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f5c160420 0x7f0f5c153ff0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f0f54007e90 tx=0x7f0f5400e4b0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 -- 172.21.15.62:0/33719608 shutdown_connections 2024-03-23T11:31:56.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0f3806a7a0 0x7f0f3806cc60 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f5c160420 0x7f0f5c153ff0 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f5c15fcf0 0x7f0f5c153ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 --2- 172.21.15.62:0/33719608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f5c0b5db0 0x7f0f5c06c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 -- 172.21.15.62:0/33719608 >> 172.21.15.62:0/33719608 conn(0x7f0f5c089f60 msgr2=0x7f0f5c0b6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:56.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 -- 172.21.15.62:0/33719608 shutdown_connections 2024-03-23T11:31:56.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.564+0000 7f0f5b200640 1 -- 172.21.15.62:0/33719608 wait complete. 2024-03-23T11:31:56.571 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:31:56.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.790+0000 7f0b8da00640 1 Processor -- start 2024-03-23T11:31:56.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.791+0000 7f0b8da00640 1 -- start start 2024-03-23T11:31:56.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.792+0000 7f0b8da00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0b8815c830 0x7f0b880ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.792+0000 7f0b8da00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0b88160310 0x7f0b880ab980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.792+0000 7f0b8da00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b881608d0 con 0x7f0b880abec0 2024-03-23T11:31:56.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.792+0000 7f0b8da00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b880addb0 con 0x7f0b88160310 2024-03-23T11:31:56.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.792+0000 7f0b8da00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b8815cc10 con 0x7f0b8815c830 2024-03-23T11:31:56.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.793+0000 7f0b7fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0b88160310 0x7f0b880ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.793+0000 7f0b7fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0b88160310 0x7f0b880ab980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48360/0 (socket says 172.21.15.62:48360) 2024-03-23T11:31:56.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.793+0000 7f0b7fe00640 1 -- 172.21.15.62:0/3090680920 learned_addr learned my addr 172.21.15.62:0/3090680920 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:56.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.793+0000 7f0b87e00640 1 --2- 172.21.15.62:0/3090680920 >> v2:172.21.15.120:3300/0 conn(0x7f0b8815c830 0x7f0b880ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.793+0000 7f0b7fe00640 1 -- 172.21.15.62:0/3090680920 >> v2:172.21.15.120:3300/0 conn(0x7f0b8815c830 msgr2=0x7f0b880ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:56.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.793+0000 7f0b7fe00640 1 --2- 172.21.15.62:0/3090680920 >> v2:172.21.15.120:3300/0 conn(0x7f0b8815c830 0x7f0b880ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.793+0000 7f0b7fe00640 1 -- 172.21.15.62:0/3090680920 >> v1:172.21.15.120:6789/0 conn(0x7f0b880abec0 legacy=0x7f0b880ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:56.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.793+0000 7f0b7fe00640 1 -- 172.21.15.62:0/3090680920 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b88160bb0 con 0x7f0b88160310 2024-03-23T11:31:56.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.795+0000 7f0b7fe00640 1 --2- 172.21.15.62:0/3090680920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b88160310 0x7f0b880ab980 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f0b880ce220 tx=0x7f0b7002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ce3231654c3e11 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:56.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.795+0000 7f0b87400640 1 -- 172.21.15.62:0/3090680920 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70030860 con 0x7f0b88160310 2024-03-23T11:31:56.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.795+0000 7f0b87400640 1 -- 172.21.15.62:0/3090680920 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b700309c0 con 0x7f0b88160310 2024-03-23T11:31:56.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.795+0000 7f0b87400640 1 -- 172.21.15.62:0/3090680920 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70030d50 con 0x7f0b88160310 2024-03-23T11:31:56.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.796+0000 7f0b8da00640 1 -- 172.21.15.62:0/3090680920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b88160310 msgr2=0x7f0b880ab980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:56.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.796+0000 7f0b8da00640 1 --2- 172.21.15.62:0/3090680920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b88160310 0x7f0b880ab980 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f0b880ce220 tx=0x7f0b7002b9a0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.796+0000 7f0b8da00640 1 -- 172.21.15.62:0/3090680920 shutdown_connections 2024-03-23T11:31:56.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.796+0000 7f0b8da00640 1 --2- 172.21.15.62:0/3090680920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b88160310 0x7f0b880ab980 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.796+0000 7f0b8da00640 1 --2- 172.21.15.62:0/3090680920 >> v2:172.21.15.120:3300/0 conn(0x7f0b8815c830 0x7f0b880ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.796+0000 7f0b8da00640 1 -- 172.21.15.62:0/3090680920 >> 172.21.15.62:0/3090680920 conn(0x7f0b88066050 msgr2=0x7f0b88069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:56.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.796+0000 7f0b8da00640 1 -- 172.21.15.62:0/3090680920 shutdown_connections 2024-03-23T11:31:56.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.796+0000 7f0b8da00640 1 -- 172.21.15.62:0/3090680920 wait complete. 2024-03-23T11:31:56.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.799+0000 7f0b8da00640 1 Processor -- start 2024-03-23T11:31:56.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.799+0000 7f0b8da00640 1 -- start start 2024-03-23T11:31:56.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.800+0000 7f0b8da00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880abec0 0x7f0b8815af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.800+0000 7f0b8da00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8815c830 0x7f0b8815b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.800+0000 7f0b8da00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b88160310 0x7f0b88142a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.800+0000 7f0b8da00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8815f2c0 con 0x7f0b88160310 2024-03-23T11:31:56.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.800+0000 7f0b8da00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b88154610 con 0x7f0b8815c830 2024-03-23T11:31:56.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.800+0000 7f0b8da00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8815e640 con 0x7f0b880abec0 2024-03-23T11:31:56.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.800+0000 7f0b8d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b88160310 0x7f0b88142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.800+0000 7f0b8d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b88160310 0x7f0b88142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48362/0 (socket says 172.21.15.62:48362) 2024-03-23T11:31:56.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.800+0000 7f0b8d000640 1 -- 172.21.15.62:0/2173597618 learned_addr learned my addr 172.21.15.62:0/2173597618 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:56.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.801+0000 7f0b7fe00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8815c830 0x7f0b8815b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.801+0000 7f0b87e00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880abec0 0x7f0b8815af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.801+0000 7f0b8d000640 1 -- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880abec0 msgr2=0x7f0b8815af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:56.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.801+0000 7f0b8d000640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880abec0 0x7f0b8815af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.801+0000 7f0b8d000640 1 -- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8815c830 msgr2=0x7f0b8815b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:56.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.801+0000 7f0b8d000640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8815c830 0x7f0b8815b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:56.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.801+0000 7f0b8d000640 1 -- 172.21.15.62:0/2173597618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b700094d0 con 0x7f0b88160310 2024-03-23T11:31:56.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.801+0000 7f0b7fe00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8815c830 0x7f0b8815b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:31:56.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.801+0000 7f0b8d000640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b88160310 0x7f0b88142a20 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000c680 tx=0x7f0b8000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:56.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.801+0000 7f0b84c00640 1 -- 172.21.15.62:0/2173597618 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80003b70 con 0x7f0b88160310 2024-03-23T11:31:56.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.801+0000 7f0b84c00640 1 -- 172.21.15.62:0/2173597618 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b80003cd0 con 0x7f0b88160310 2024-03-23T11:31:56.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.802+0000 7f0b8da00640 1 -- 172.21.15.62:0/2173597618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8815e920 con 0x7f0b88160310 2024-03-23T11:31:56.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.802+0000 7f0b84c00640 1 -- 172.21.15.62:0/2173597618 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80005310 con 0x7f0b88160310 2024-03-23T11:31:56.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.802+0000 7f0b8da00640 1 -- 172.21.15.62:0/2173597618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8815eee0 con 0x7f0b88160310 2024-03-23T11:31:56.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.803+0000 7f0b8da00640 1 -- 172.21.15.62:0/2173597618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b881563b0 con 0x7f0b88160310 2024-03-23T11:31:56.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.804+0000 7f0b84c00640 1 -- 172.21.15.62:0/2173597618 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0b80005470 con 0x7f0b88160310 2024-03-23T11:31:56.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.805+0000 7f0b84c00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0b6406a7d0 0x7f0b6406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:56.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.805+0000 7f0b84c00640 1 -- 172.21.15.62:0/2173597618 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0b80088b80 con 0x7f0b88160310 2024-03-23T11:31:56.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.805+0000 7f0b87e00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0b6406a7d0 0x7f0b6406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:56.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.807+0000 7f0b87e00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0b6406a7d0 0x7f0b6406cc90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0b78006fd0 tx=0x7f0b78008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:56.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:56.810+0000 7f0b84c00640 1 -- 172.21.15.62:0/2173597618 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b80053f50 con 0x7f0b88160310 2024-03-23T11:31:57.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.012+0000 7f0b8da00640 1 -- 172.21.15.62:0/2173597618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0b88066bb0 con 0x7f0b88160310 2024-03-23T11:31:57.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.013+0000 7f0b84c00640 1 -- 172.21.15.62:0/2173597618 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0b80088e60 con 0x7f0b88160310 2024-03-23T11:31:57.002 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:31:57.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 -- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0b6406a7d0 msgr2=0x7f0b6406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:57.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0b6406a7d0 0x7f0b6406cc90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0b78006fd0 tx=0x7f0b78008040 comp rx=0 tx=0).stop 2024-03-23T11:31:57.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 -- 172.21.15.62:0/2173597618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b88160310 msgr2=0x7f0b88142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:57.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b88160310 0x7f0b88142a20 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000c680 tx=0x7f0b8000cb40 comp rx=0 tx=0).stop 2024-03-23T11:31:57.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 -- 172.21.15.62:0/2173597618 shutdown_connections 2024-03-23T11:31:57.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0b6406a7d0 0x7f0b6406cc90 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b88160310 0x7f0b88142a20 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b8815c830 0x7f0b8815b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 --2- 172.21.15.62:0/2173597618 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b880abec0 0x7f0b8815af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 -- 172.21.15.62:0/2173597618 >> 172.21.15.62:0/2173597618 conn(0x7f0b88066050 msgr2=0x7f0b88069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:57.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 -- 172.21.15.62:0/2173597618 shutdown_connections 2024-03-23T11:31:57.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.015+0000 7f0b8da00640 1 -- 172.21.15.62:0/2173597618 wait complete. 2024-03-23T11:31:57.022 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:31:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.246+0000 7f3d12a00640 1 Processor -- start 2024-03-23T11:31:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.246+0000 7f3d12a00640 1 -- start start 2024-03-23T11:31:57.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.248+0000 7f3d12a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c15c730 0x7f3d0c0b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:57.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.248+0000 7f3d12a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3d0c0ad370 0x7f3d0c160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:57.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.248+0000 7f3d12a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0c0abe50 con 0x7f3d0c069c60 2024-03-23T11:31:57.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.248+0000 7f3d12a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0c1612c0 con 0x7f3d0c0ad370 2024-03-23T11:31:57.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.248+0000 7f3d12a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c06cba0 con 0x7f3d0c15c730 2024-03-23T11:31:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.249+0000 7f3d11600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c15c730 0x7f3d0c0b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.249+0000 7f3d11600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c15c730 0x7f3d0c0b1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:57180/0 (socket says 172.21.15.62:57180) 2024-03-23T11:31:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.249+0000 7f3d11600640 1 -- 172.21.15.62:0/657178701 learned_addr learned my addr 172.21.15.62:0/657178701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.249+0000 7f3d10c00640 1 --2- 172.21.15.62:0/657178701 >> v2:172.21.15.120:3300/0 conn(0x7f3d0c0ad370 0x7f3d0c160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.249+0000 7f3d11600640 1 -- 172.21.15.62:0/657178701 >> v2:172.21.15.120:3300/0 conn(0x7f3d0c0ad370 msgr2=0x7f3d0c160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:31:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.249+0000 7f3d11600640 1 --2- 172.21.15.62:0/657178701 >> v2:172.21.15.120:3300/0 conn(0x7f3d0c0ad370 0x7f3d0c160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.249+0000 7f3d11600640 1 -- 172.21.15.62:0/657178701 >> v1:172.21.15.120:6789/0 conn(0x7f3d0c069c60 legacy=0x7f3d0c0ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.249+0000 7f3d11600640 1 -- 172.21.15.62:0/657178701 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c06ce80 con 0x7f3d0c15c730 2024-03-23T11:31:57.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.251+0000 7f3d11600640 1 --2- 172.21.15.62:0/657178701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c15c730 0x7f3d0c0b1ab0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009870 tx=0x7f3d0002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4e81b0bf0f2b268 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.251+0000 7f3d0be00640 1 -- 172.21.15.62:0/657178701 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00030840 con 0x7f3d0c15c730 2024-03-23T11:31:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.251+0000 7f3d0be00640 1 -- 172.21.15.62:0/657178701 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d000309a0 con 0x7f3d0c15c730 2024-03-23T11:31:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.251+0000 7f3d0be00640 1 -- 172.21.15.62:0/657178701 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00030d30 con 0x7f3d0c15c730 2024-03-23T11:31:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.252+0000 7f3d12a00640 1 -- 172.21.15.62:0/657178701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c15c730 msgr2=0x7f3d0c0b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.252+0000 7f3d12a00640 1 --2- 172.21.15.62:0/657178701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c15c730 0x7f3d0c0b1ab0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009870 tx=0x7f3d0002b980 comp rx=0 tx=0).stop 2024-03-23T11:31:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.252+0000 7f3d12a00640 1 -- 172.21.15.62:0/657178701 shutdown_connections 2024-03-23T11:31:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.252+0000 7f3d12a00640 1 --2- 172.21.15.62:0/657178701 >> v2:172.21.15.120:3300/0 conn(0x7f3d0c0ad370 0x7f3d0c160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.252+0000 7f3d12a00640 1 --2- 172.21.15.62:0/657178701 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c15c730 0x7f3d0c0b1ab0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.252+0000 7f3d12a00640 1 -- 172.21.15.62:0/657178701 >> 172.21.15.62:0/657178701 conn(0x7f3d0c0b0090 msgr2=0x7f3d0c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.252+0000 7f3d12a00640 1 -- 172.21.15.62:0/657178701 shutdown_connections 2024-03-23T11:31:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.252+0000 7f3d12a00640 1 -- 172.21.15.62:0/657178701 wait complete. 2024-03-23T11:31:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.254+0000 7f3d12a00640 1 Processor -- start 2024-03-23T11:31:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.255+0000 7f3d12a00640 1 -- start start 2024-03-23T11:31:57.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.256+0000 7f3d12a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d0c069c60 0x7f3d0c1f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:57.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.256+0000 7f3d12a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c0ad370 0x7f3d0c157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:57.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.256+0000 7f3d12a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d0c15c730 0x7f3d0c157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:57.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.256+0000 7f3d12a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c156fe0 con 0x7f3d0c15c730 2024-03-23T11:31:57.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.256+0000 7f3d12a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c1563a0 con 0x7f3d0c069c60 2024-03-23T11:31:57.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.256+0000 7f3d12a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c14d320 con 0x7f3d0c0ad370 2024-03-23T11:31:57.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.256+0000 7f3d12000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d0c15c730 0x7f3d0c157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:57.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.256+0000 7f3d12000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d0c15c730 0x7f3d0c157940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48374/0 (socket says 172.21.15.62:48374) 2024-03-23T11:31:57.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.256+0000 7f3d12000640 1 -- 172.21.15.62:0/1235203556 learned_addr learned my addr 172.21.15.62:0/1235203556 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:31:57.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.257+0000 7f3d11600640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d0c069c60 0x7f3d0c1f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:57.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.257+0000 7f3d10c00640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c0ad370 0x7f3d0c157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.257+0000 7f3d12000640 1 -- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c0ad370 msgr2=0x7f3d0c157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.257+0000 7f3d12000640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c0ad370 0x7f3d0c157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.257+0000 7f3d12000640 1 -- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d0c069c60 msgr2=0x7f3d0c1f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.257+0000 7f3d12000640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d0c069c60 0x7f3d0c1f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.257+0000 7f3d12000640 1 -- 172.21.15.62:0/1235203556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d000094d0 con 0x7f3d0c15c730 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.257+0000 7f3d11600640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d0c069c60 0x7f3d0c1f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.257+0000 7f3d12000640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d0c15c730 0x7f3d0c157940 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400c7a0 tx=0x7f3d0400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.257+0000 7f3d09600640 1 -- 172.21.15.62:0/1235203556 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d04003b70 con 0x7f3d0c15c730 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.258+0000 7f3d09600640 1 -- 172.21.15.62:0/1235203556 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d04003cf0 con 0x7f3d0c15c730 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.258+0000 7f3d09600640 1 -- 172.21.15.62:0/1235203556 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d04003ea0 con 0x7f3d0c15c730 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.258+0000 7f3d12a00640 1 -- 172.21.15.62:0/1235203556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0c156640 con 0x7f3d0c15c730 2024-03-23T11:31:57.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.258+0000 7f3d12a00640 1 -- 172.21.15.62:0/1235203556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0c156c00 con 0x7f3d0c15c730 2024-03-23T11:31:57.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.259+0000 7f3d12a00640 1 -- 172.21.15.62:0/1235203556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0c066bb0 con 0x7f3d0c15c730 2024-03-23T11:31:57.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.260+0000 7f3d09600640 1 -- 172.21.15.62:0/1235203556 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3d0400aa90 con 0x7f3d0c15c730 2024-03-23T11:31:57.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.261+0000 7f3d09600640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cec06a7a0 0x7f3cec06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:31:57.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.261+0000 7f3d09600640 1 -- 172.21.15.62:0/1235203556 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3d04088c40 con 0x7f3d0c15c730 2024-03-23T11:31:57.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.261+0000 7f3d11600640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cec06a7a0 0x7f3cec06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:31:57.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.262+0000 7f3d11600640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cec06a7a0 0x7f3cec06cc60 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f3d00036000 tx=0x7f3d0003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:31:57.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.265+0000 7f3d09600640 1 -- 172.21.15.62:0/1235203556 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3d04054020 con 0x7f3d0c15c730 2024-03-23T11:31:57.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.477+0000 7f3d12a00640 1 -- 172.21.15.62:0/1235203556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3d0c0b15d0 con 0x7f3d0c15c730 2024-03-23T11:31:57.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.478+0000 7f3d09600640 1 -- 172.21.15.62:0/1235203556 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v120) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3d04057640 con 0x7f3d0c15c730 2024-03-23T11:31:57.468 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 -- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cec06a7a0 msgr2=0x7f3cec06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cec06a7a0 0x7f3cec06cc60 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f3d00036000 tx=0x7f3d0003b040 comp rx=0 tx=0).stop 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 -- 172.21.15.62:0/1235203556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d0c15c730 msgr2=0x7f3d0c157940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d0c15c730 0x7f3d0c157940 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400c7a0 tx=0x7f3d0400cc60 comp rx=0 tx=0).stop 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 -- 172.21.15.62:0/1235203556 shutdown_connections 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cec06a7a0 0x7f3cec06cc60 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d0c15c730 0x7f3d0c157940 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d0c0ad370 0x7f3d0c157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 --2- 172.21.15.62:0/1235203556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d0c069c60 0x7f3d0c1f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 -- 172.21.15.62:0/1235203556 >> 172.21.15.62:0/1235203556 conn(0x7f3d0c0b0090 msgr2=0x7f3d0c0aead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 -- 172.21.15.62:0/1235203556 shutdown_connections 2024-03-23T11:31:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:31:57.481+0000 7f3d12a00640 1 -- 172.21.15.62:0/1235203556 wait complete. 2024-03-23T11:31:57.491 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:31:57.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:31:57.491 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:31:57.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[78892]: fuse finished with error 0 and tester_r 0 2024-03-23T11:31:57.592 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:32:03.694 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:32:03.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:03.695 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:32:03.755 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:32:03.755 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:32:03.814 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:32:03.814 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi062.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:03.814 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:32:03.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:03.814 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:32:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-23T11:32:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:32:03.872 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:32:04.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.108+0000 7fcc08c00640 1 Processor -- start 2024-03-23T11:32:04.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.109+0000 7fcc08c00640 1 -- start start 2024-03-23T11:32:04.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.110+0000 7fcc08c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc0415a770 0x7fcc04099710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.110+0000 7fcc08c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcc040cb070 0x7fcc040989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.110+0000 7fcc08c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc04099340 con 0x7fcc04098f10 2024-03-23T11:32:04.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.110+0000 7fcc08c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc040cb5c0 con 0x7fcc040cb070 2024-03-23T11:32:04.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.110+0000 7fcc08c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040cb8a0 con 0x7fcc0415a770 2024-03-23T11:32:04.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.111+0000 7fcc03400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc0415a770 0x7fcc04099710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.111+0000 7fcc03400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc0415a770 0x7fcc04099710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:46986/0 (socket says 172.21.15.62:46986) 2024-03-23T11:32:04.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.111+0000 7fcc03400640 1 -- 172.21.15.62:0/3795552706 learned_addr learned my addr 172.21.15.62:0/3795552706 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:04.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.111+0000 7fcc02a00640 1 --2- 172.21.15.62:0/3795552706 >> v2:172.21.15.120:3300/0 conn(0x7fcc040cb070 0x7fcc040989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.111+0000 7fcc03400640 1 -- 172.21.15.62:0/3795552706 >> v2:172.21.15.120:3300/0 conn(0x7fcc040cb070 msgr2=0x7fcc040989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:04.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.111+0000 7fcc03400640 1 --2- 172.21.15.62:0/3795552706 >> v2:172.21.15.120:3300/0 conn(0x7fcc040cb070 0x7fcc040989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.111+0000 7fcc03400640 1 -- 172.21.15.62:0/3795552706 >> v1:172.21.15.120:6789/0 conn(0x7fcc04098f10 legacy=0x7fcc040b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.111+0000 7fcc03400640 1 -- 172.21.15.62:0/3795552706 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc040cbb80 con 0x7fcc0415a770 2024-03-23T11:32:04.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.113+0000 7fcc03400640 1 --2- 172.21.15.62:0/3795552706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc0415a770 0x7fcc04099710 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009870 tx=0x7fcbf402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=471b8615a3931240 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:04.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.113+0000 7fcc02000640 1 -- 172.21.15.62:0/3795552706 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40317c0 con 0x7fcc0415a770 2024-03-23T11:32:04.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.113+0000 7fcc02000640 1 -- 172.21.15.62:0/3795552706 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf4031920 con 0x7fcc0415a770 2024-03-23T11:32:04.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.113+0000 7fcc02000640 1 -- 172.21.15.62:0/3795552706 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4031cb0 con 0x7fcc0415a770 2024-03-23T11:32:04.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.114+0000 7fcc08c00640 1 -- 172.21.15.62:0/3795552706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc0415a770 msgr2=0x7fcc04099710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.114+0000 7fcc08c00640 1 --2- 172.21.15.62:0/3795552706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc0415a770 0x7fcc04099710 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009870 tx=0x7fcbf402b860 comp rx=0 tx=0).stop 2024-03-23T11:32:04.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.114+0000 7fcc08c00640 1 -- 172.21.15.62:0/3795552706 shutdown_connections 2024-03-23T11:32:04.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.114+0000 7fcc08c00640 1 --2- 172.21.15.62:0/3795552706 >> v2:172.21.15.120:3300/0 conn(0x7fcc040cb070 0x7fcc040989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.114+0000 7fcc08c00640 1 --2- 172.21.15.62:0/3795552706 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc0415a770 0x7fcc04099710 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.114+0000 7fcc08c00640 1 -- 172.21.15.62:0/3795552706 >> 172.21.15.62:0/3795552706 conn(0x7fcc040b1f50 msgr2=0x7fcc040b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.114+0000 7fcc08c00640 1 -- 172.21.15.62:0/3795552706 shutdown_connections 2024-03-23T11:32:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.114+0000 7fcc08c00640 1 -- 172.21.15.62:0/3795552706 wait complete. 2024-03-23T11:32:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.116+0000 7fcc08c00640 1 Processor -- start 2024-03-23T11:32:04.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.116+0000 7fcc08c00640 1 -- start start 2024-03-23T11:32:04.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.117+0000 7fcc08c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc04098f10 0x7fcc04153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.117+0000 7fcc08c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc040cb070 0x7fcc040b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.117+0000 7fcc08c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc0415a770 0x7fcc0415bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.117+0000 7fcc08c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040b33f0 con 0x7fcc04098f10 2024-03-23T11:32:04.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.117+0000 7fcc08c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040b36d0 con 0x7fcc0415a770 2024-03-23T11:32:04.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.117+0000 7fcc08c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc040b39b0 con 0x7fcc040cb070 2024-03-23T11:32:04.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.117+0000 7fcc03400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc04098f10 0x7fcc04153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.118+0000 7fcc03400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc04098f10 0x7fcc04153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51704/0 (socket says 172.21.15.62:51704) 2024-03-23T11:32:04.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.118+0000 7fcc03400640 1 -- 172.21.15.62:0/1366647345 learned_addr learned my addr 172.21.15.62:0/1366647345 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:04.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.118+0000 7fcc02a00640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc040cb070 0x7fcc040b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.118+0000 7fcc03e00640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc0415a770 0x7fcc0415bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.118+0000 7fcc03400640 1 -- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc040cb070 msgr2=0x7fcc040b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.118+0000 7fcc03400640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc040cb070 0x7fcc040b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.118+0000 7fcc03400640 1 -- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc0415a770 msgr2=0x7fcc0415bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.118+0000 7fcc03400640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc0415a770 0x7fcc0415bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.118+0000 7fcc03400640 1 -- 172.21.15.62:0/1366647345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf40094d0 con 0x7fcc04098f10 2024-03-23T11:32:04.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.118+0000 7fcc03400640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc04098f10 0x7fcc04153830 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fcbf402b8a0 tx=0x7fcbf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:04.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.119+0000 7fcbfb400640 1 -- 172.21.15.62:0/1366647345 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4032860 con 0x7fcc04098f10 2024-03-23T11:32:04.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.119+0000 7fcbfb400640 1 -- 172.21.15.62:0/1366647345 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf4032a00 con 0x7fcc04098f10 2024-03-23T11:32:04.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.119+0000 7fcbfb400640 1 -- 172.21.15.62:0/1366647345 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4032db0 con 0x7fcc04098f10 2024-03-23T11:32:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.119+0000 7fcc08c00640 1 -- 172.21.15.62:0/1366647345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc040b3c90 con 0x7fcc04098f10 2024-03-23T11:32:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.119+0000 7fcc08c00640 1 -- 172.21.15.62:0/1366647345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc04153d70 con 0x7fcc04098f10 2024-03-23T11:32:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.120+0000 7fcc08c00640 1 -- 172.21.15.62:0/1366647345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc040ccc50 con 0x7fcc04098f10 2024-03-23T11:32:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.121+0000 7fcbfb400640 1 -- 172.21.15.62:0/1366647345 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fcbf404e450 con 0x7fcc04098f10 2024-03-23T11:32:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.122+0000 7fcbfb400640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbe006a7a0 0x7fcbe006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.122+0000 7fcbfb400640 1 -- 172.21.15.62:0/1366647345 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcbf40aaec0 con 0x7fcc04098f10 2024-03-23T11:32:04.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.123+0000 7fcc02a00640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbe006a7a0 0x7fcbe006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.124+0000 7fcc02a00640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbe006a7a0 0x7fcbe006cc60 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fcc040c7230 tx=0x7fcbec009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:04.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.127+0000 7fcbfb400640 1 -- 172.21.15.62:0/1366647345 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcbf4076200 con 0x7fcc04098f10 2024-03-23T11:32:04.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.364+0000 7fcc08c00640 1 -- 172.21.15.62:0/1366647345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcc040672c0 con 0x7fcc04098f10 2024-03-23T11:32:04.355 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:32:04.365+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:32:04.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.381+0000 7fcbfb400640 1 -- 172.21.15.62:0/1366647345 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v95) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcbf4079820 con 0x7fcc04098f10 2024-03-23T11:32:04.371 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:32:04.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.384+0000 7fcc08c00640 1 -- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbe006a7a0 msgr2=0x7fcbe006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.384+0000 7fcc08c00640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbe006a7a0 0x7fcbe006cc60 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fcc040c7230 tx=0x7fcbec009190 comp rx=0 tx=0).stop 2024-03-23T11:32:04.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.384+0000 7fcc08c00640 1 -- 172.21.15.62:0/1366647345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc04098f10 msgr2=0x7fcc04153830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.384+0000 7fcc08c00640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc04098f10 0x7fcc04153830 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fcbf402b8a0 tx=0x7fcbf40046c0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.384+0000 7fcc08c00640 1 -- 172.21.15.62:0/1366647345 shutdown_connections 2024-03-23T11:32:04.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.384+0000 7fcc08c00640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fcbe006a7a0 0x7fcbe006cc60 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.384+0000 7fcc08c00640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc0415a770 0x7fcc0415bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.384+0000 7fcc08c00640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc040cb070 0x7fcc040b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.384+0000 7fcc08c00640 1 --2- 172.21.15.62:0/1366647345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc04098f10 0x7fcc04153830 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.384+0000 7fcc08c00640 1 -- 172.21.15.62:0/1366647345 >> 172.21.15.62:0/1366647345 conn(0x7fcc040b1f50 msgr2=0x7fcc040cebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:04.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.384+0000 7fcc08c00640 1 -- 172.21.15.62:0/1366647345 shutdown_connections 2024-03-23T11:32:04.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.385+0000 7fcc08c00640 1 -- 172.21.15.62:0/1366647345 wait complete. 2024-03-23T11:32:04.394 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T11:32:04.401 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T11:32:04.414 INFO:tasks.ceph.mds.b.smithi120.stderr: -8132> 2024-03-23T11:31:29.521+0000 7f577a800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:32:04.482 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-23T11:32:04.556 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2024-03-23T11:32:04.556 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2024-03-23T11:32:04.556 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T11:32:04.557 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2024-03-23T11:32:04.557 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.557 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2024-03-23T11:32:04.557 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.557 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2024-03-23T11:32:04.557 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.557 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2024-03-23T11:32:04.557 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2024-03-23T11:32:04.557 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.557 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2024-03-23T11:32:04.557 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2024-03-23T11:32:04.558 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.558 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T11:32:04.558 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2024-03-23T11:32:04.558 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.558 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T11:32:04.558 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T11:32:04.558 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.558 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2024-03-23T11:32:04.558 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.558 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2024-03-23T11:32:04.558 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2024-03-23T11:32:04.559 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2024-03-23T11:32:04.559 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2024-03-23T11:32:04.559 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2024-03-23T11:32:04.559 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2024-03-23T11:32:04.559 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2024-03-23T11:32:04.559 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T11:32:04.559 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T11:32:04.559 INFO:teuthology.orchestra.run.smithi120.stdout: mon_allow_pool_size_one = true 2024-03-23T11:32:04.559 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.559 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T11:32:04.559 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.560 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2024-03-23T11:32:04.560 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.560 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2024-03-23T11:32:04.560 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.560 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2024-03-23T11:32:04.560 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2024-03-23T11:32:04.560 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2024-03-23T11:32:04.560 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "172.21.15.62,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-23T11:32:04.560 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:32:04.560 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.560 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2024-03-23T11:32:04.561 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2024-03-23T11:32:04.561 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.561 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2024-03-23T11:32:04.561 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2024-03-23T11:32:04.561 INFO:teuthology.orchestra.run.smithi120.stdout: osd mclock profile = high_recovery_ops 2024-03-23T11:32:04.561 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.561 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2024-03-23T11:32:04.561 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2024-03-23T11:32:04.561 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.561 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2024-03-23T11:32:04.561 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2024-03-23T11:32:04.562 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2024-03-23T11:32:04.562 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.562 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2024-03-23T11:32:04.562 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2024-03-23T11:32:04.562 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.562 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2024-03-23T11:32:04.562 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.562 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2024-03-23T11:32:04.562 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.562 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2024-03-23T11:32:04.562 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.563 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2024-03-23T11:32:04.563 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2024-03-23T11:32:04.563 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.563 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2024-03-23T11:32:04.563 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2024-03-23T11:32:04.563 INFO:teuthology.orchestra.run.smithi120.stdout: bdev async discard = True 2024-03-23T11:32:04.563 INFO:teuthology.orchestra.run.smithi120.stdout: bdev enable discard = True 2024-03-23T11:32:04.563 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore allocator = bitmap 2024-03-23T11:32:04.563 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2024-03-23T11:32:04.563 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2024-03-23T11:32:04.563 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 1/20 2024-03-23T11:32:04.564 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 1/20 2024-03-23T11:32:04.564 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:04.564 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2024-03-23T11:32:04.564 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 4/10 2024-03-23T11:32:04.564 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T11:32:04.564 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2024-03-23T11:32:04.564 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2024-03-23T11:32:04.564 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2024-03-23T11:32:04.564 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2024-03-23T11:32:04.564 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-23T11:32:04.565 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.565 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2024-03-23T11:32:04.565 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:04.565 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2024-03-23T11:32:04.565 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-23T11:32:04.565 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-23T11:32:04.565 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:04.565 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:04.565 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2024-03-23T11:32:04.565 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.565 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2024-03-23T11:32:04.566 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:04.566 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-23T11:32:04.566 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2024-03-23T11:32:04.566 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-23T11:32:04.566 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2024-03-23T11:32:04.566 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2024-03-23T11:32:04.566 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:04.566 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2024-03-23T11:32:04.566 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2024-03-23T11:32:04.566 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: # 1m isn't quite enough 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: mon_down_mkfs_grace = 2m 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.567 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_filestore_osds = false 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout: mon op complaint time = 120 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout: client mount timeout = 600 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout: debug client = 20 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:04.568 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = "" 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.d] 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout:[mds] 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds = 20 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds balancer = 20 2024-03-23T11:32:04.569 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:04.570 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug frag = True 2024-03-23T11:32:04.570 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug scatterstat = True 2024-03-23T11:32:04.570 INFO:teuthology.orchestra.run.smithi120.stdout: mds op complaint time = 180 2024-03-23T11:32:04.570 INFO:teuthology.orchestra.run.smithi120.stdout: mds verify scatter = False 2024-03-23T11:32:04.570 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-23T11:32:04.570 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-23T11:32:04.570 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-23T11:32:04.573 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-03-23T11:32:04.573 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2024-03-23T11:32:04.573 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T11:32:04.573 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2024-03-23T11:32:04.573 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.573 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2024-03-23T11:32:04.574 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.574 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2024-03-23T11:32:04.574 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.574 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2024-03-23T11:32:04.574 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2024-03-23T11:32:04.574 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.574 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2024-03-23T11:32:04.574 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2024-03-23T11:32:04.574 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.574 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2024-03-23T11:32:04.575 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.576 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = "172.21.15.62,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock profile = high_recovery_ops 2024-03-23T11:32:04.577 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.578 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2024-03-23T11:32:04.578 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2024-03-23T11:32:04.578 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.578 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2024-03-23T11:32:04.578 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2024-03-23T11:32:04.578 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2024-03-23T11:32:04.578 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.578 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2024-03-23T11:32:04.578 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2024-03-23T11:32:04.578 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.578 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: bdev async discard = True 2024-03-23T11:32:04.579 INFO:teuthology.orchestra.run.smithi062.stdout: bdev enable discard = True 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore allocator = bitmap 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 1/20 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 1/20 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 4/10 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2024-03-23T11:32:04.580 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.581 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.582 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: mon op complaint time = 120 2024-03-23T11:32:04.583 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout: client mount timeout = 600 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout: debug client = 20 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = "" 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-03-23T11:32:04.584 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.b] 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.c] 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a] 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.c] 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b] 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.d] 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout:[mds] 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds = 20 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds balancer = 20 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:04.585 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug frag = True 2024-03-23T11:32:04.586 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug scatterstat = True 2024-03-23T11:32:04.586 INFO:teuthology.orchestra.run.smithi062.stdout: mds op complaint time = 180 2024-03-23T11:32:04.586 INFO:teuthology.orchestra.run.smithi062.stdout: mds verify scatter = False 2024-03-23T11:32:04.586 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-03-23T11:32:04.586 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-03-23T11:32:04.586 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-03-23T11:32:04.624 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T11:32:04.631 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T11:32:04.800 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2024-03-23T11:32:04.804 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2024-03-23T11:32:04.805 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: mon_allow_pool_size_one = true 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2024-03-23T11:32:04.806 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "172.21.15.62,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: osd mclock profile = high_recovery_ops 2024-03-23T11:32:04.807 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2024-03-23T11:32:04.808 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: bdev async discard = True 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: bdev enable discard = True 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore allocator = bitmap 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2024-03-23T11:32:04.809 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 1/20 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 1/20 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 4/10 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:04.810 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-23T11:32:04.811 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T11:32:04.812 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: # 1m isn't quite enough 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: mon_down_mkfs_grace = 2m 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_filestore_osds = false 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: mon op complaint time = 120 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T11:32:04.813 INFO:teuthology.orchestra.run.smithi120.stdout: client mount timeout = 600 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout: debug client = 20 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = "" 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.d] 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout:[mds] 2024-03-23T11:32:04.814 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds = 20 2024-03-23T11:32:04.815 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds balancer = 20 2024-03-23T11:32:04.815 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:04.815 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug frag = True 2024-03-23T11:32:04.815 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug scatterstat = False 2024-03-23T11:32:04.815 INFO:teuthology.orchestra.run.smithi120.stdout: mds op complaint time = 180 2024-03-23T11:32:04.815 INFO:teuthology.orchestra.run.smithi120.stdout: mds verify scatter = False 2024-03-23T11:32:04.815 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-23T11:32:04.815 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-23T11:32:04.815 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2024-03-23T11:32:04.816 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2024-03-23T11:32:04.817 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2024-03-23T11:32:04.818 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = "172.21.15.62,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2024-03-23T11:32:04.819 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock profile = high_recovery_ops 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2024-03-23T11:32:04.820 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: bdev async discard = True 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: bdev enable discard = True 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore allocator = bitmap 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2024-03-23T11:32:04.821 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 1/20 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 1/20 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 4/10 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2024-03-23T11:32:04.822 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2024-03-23T11:32:04.823 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.824 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: mon op complaint time = 120 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: client mount timeout = 600 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: debug client = 20 2024-03-23T11:32:04.825 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = "" 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.b] 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.c] 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a] 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.c] 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b] 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.d] 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout:[mds] 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds = 20 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds balancer = 20 2024-03-23T11:32:04.826 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:04.827 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug frag = True 2024-03-23T11:32:04.827 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug scatterstat = False 2024-03-23T11:32:04.827 INFO:teuthology.orchestra.run.smithi062.stdout: mds op complaint time = 180 2024-03-23T11:32:04.827 INFO:teuthology.orchestra.run.smithi062.stdout: mds verify scatter = False 2024-03-23T11:32:04.827 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-03-23T11:32:04.827 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-03-23T11:32:04.827 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-03-23T11:32:04.851 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmomapkey 10000000001.00000000 bravo_head 2024-03-23T11:32:04.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.967+0000 7f9975c3df40 1 Processor -- start 2024-03-23T11:32:04.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.967+0000 7f9975c3df40 1 -- start start 2024-03-23T11:32:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.968+0000 7f9975c3df40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99580030b0 0x5576f3cbdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.968+0000 7f9975c3df40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576f3ccdd80 0x5576f3c4fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.968+0000 7f9975c3df40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5576f3ccd830 0x5576f3c500d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.968+0000 7f9975c3df40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576f3b4b800 con 0x5576f3ccd830 2024-03-23T11:32:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.968+0000 7f9975c3df40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576f3b4b4a0 con 0x7f99580030b0 2024-03-23T11:32:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.968+0000 7f9975c3df40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576f3b42110 con 0x5576f3ccdd80 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.969+0000 7f9973000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5576f3ccd830 0x5576f3c500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.969+0000 7f9973000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5576f3ccd830 0x5576f3c500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51714/0 (socket says 172.21.15.62:51714) 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.969+0000 7f9973000640 1 -- 172.21.15.62:0/1826902575 learned_addr learned my addr 172.21.15.62:0/1826902575 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.969+0000 7f9972600640 1 --2- 172.21.15.62:0/1826902575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99580030b0 0x5576f3cbdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.969+0000 7f9973000640 1 -- 172.21.15.62:0/1826902575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576f3ccdd80 msgr2=0x5576f3c4fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.969+0000 7f9973000640 1 --2- 172.21.15.62:0/1826902575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576f3ccdd80 0x5576f3c4fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.969+0000 7f9973000640 1 -- 172.21.15.62:0/1826902575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99580030b0 msgr2=0x5576f3cbdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.969+0000 7f9973000640 1 --2- 172.21.15.62:0/1826902575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99580030b0 0x5576f3cbdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.969+0000 7f9973000640 1 -- 172.21.15.62:0/1826902575 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576f3b5a340 con 0x5576f3ccd830 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.969+0000 7f9972600640 1 --2- 172.21.15.62:0/1826902575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99580030b0 0x5576f3cbdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.969+0000 7f9973000640 1 --2- 172.21.15.62:0/1826902575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5576f3ccd830 0x5576f3c500d0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f996c00e4b0 tx=0x7f996c00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4faa25039820d5d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.970+0000 7f996be00640 1 -- 172.21.15.62:0/1826902575 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c00f710 con 0x5576f3ccd830 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.970+0000 7f996be00640 1 -- 172.21.15.62:0/1826902575 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c013070 con 0x5576f3ccd830 2024-03-23T11:32:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.970+0000 7f996be00640 1 -- 172.21.15.62:0/1826902575 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c00fab0 con 0x5576f3ccd830 2024-03-23T11:32:04.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.970+0000 7f9975c3df40 1 -- 172.21.15.62:0/1826902575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5576f3ccd830 msgr2=0x5576f3c500d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.970+0000 7f9975c3df40 1 --2- 172.21.15.62:0/1826902575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5576f3ccd830 0x5576f3c500d0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f996c00e4b0 tx=0x7f996c00e970 comp rx=0 tx=0).stop 2024-03-23T11:32:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.970+0000 7f9975c3df40 1 -- 172.21.15.62:0/1826902575 shutdown_connections 2024-03-23T11:32:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.970+0000 7f9975c3df40 1 --2- 172.21.15.62:0/1826902575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5576f3ccd830 0x5576f3c500d0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.970+0000 7f9975c3df40 1 --2- 172.21.15.62:0/1826902575 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576f3ccdd80 0x5576f3c4fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.970+0000 7f9975c3df40 1 --2- 172.21.15.62:0/1826902575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99580030b0 0x5576f3cbdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.970+0000 7f9975c3df40 1 -- 172.21.15.62:0/1826902575 >> 172.21.15.62:0/1826902575 conn(0x5576f3c50ff0 msgr2=0x5576f3c513d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.970+0000 7f9975c3df40 1 -- 172.21.15.62:0/1826902575 shutdown_connections 2024-03-23T11:32:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.971+0000 7f9975c3df40 1 -- 172.21.15.62:0/1826902575 wait complete. 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.971+0000 7f9975c3df40 1 Processor -- start 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.971+0000 7f9975c3df40 1 -- start start 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.972+0000 7f9975c3df40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576f3ccd830 0x5576f3cd09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.972+0000 7f9975c3df40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f3ccdd80 0x5576f3cd60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.972+0000 7f9975c3df40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99580030b0 0x5576f3c4f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.972+0000 7f9975c3df40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576f3cd65e0 con 0x7f99580030b0 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.972+0000 7f9975c3df40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576f3cb8300 con 0x5576f3ccdd80 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.972+0000 7f9975c3df40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576f3cda230 con 0x5576f3ccd830 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.973+0000 7f9973000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99580030b0 0x5576f3c4f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.973+0000 7f9973000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99580030b0 0x5576f3c4f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51716/0 (socket says 172.21.15.62:51716) 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.973+0000 7f9973000640 1 -- 172.21.15.62:0/2619637475 learned_addr learned my addr 172.21.15.62:0/2619637475 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.973+0000 7f9971c00640 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f3ccdd80 0x5576f3cd60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.973+0000 7f9972600640 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576f3ccd830 0x5576f3cd09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.973+0000 7f9973000640 1 -- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576f3ccd830 msgr2=0x5576f3cd09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.973+0000 7f9973000640 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576f3ccd830 0x5576f3cd09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.973+0000 7f9973000640 1 -- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f3ccdd80 msgr2=0x5576f3cd60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.973+0000 7f9973000640 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f3ccdd80 0x5576f3cd60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.973+0000 7f9973000640 1 -- 172.21.15.62:0/2619637475 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576f3b744e0 con 0x7f99580030b0 2024-03-23T11:32:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.973+0000 7f9972600640 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576f3ccd830 0x5576f3cd09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:04.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.974+0000 7f9973000640 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99580030b0 0x5576f3c4f2c0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f996c000ff0 tx=0x7f996c001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.974+0000 7f996aa00640 1 -- 172.21.15.62:0/2619637475 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c018420 con 0x7f99580030b0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.974+0000 7f996aa00640 1 -- 172.21.15.62:0/2619637475 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c018580 con 0x7f99580030b0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.974+0000 7f996aa00640 1 -- 172.21.15.62:0/2619637475 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c018910 con 0x7f99580030b0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.974+0000 7f9975c3df40 1 -- 172.21.15.62:0/2619637475 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5576f3b4d780 con 0x7f99580030b0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.974+0000 7f9975c3df40 1 -- 172.21.15.62:0/2619637475 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576f3b4d1e0 con 0x7f99580030b0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.976+0000 7f996aa00640 1 -- 172.21.15.62:0/2619637475 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f996c018a70 con 0x7f99580030b0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.977+0000 7f996aa00640 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f99540691b0 0x7f995406b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.977+0000 7f996aa00640 1 -- 172.21.15.62:0/2619637475 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f996c08b220 con 0x7f99580030b0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.977+0000 7f9975c3df40 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5576f3c4ffa0 0x5576f3cd85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.977+0000 7f9975c3df40 1 -- 172.21.15.62:0/2619637475 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:1 16.7 16:e2e07930:::10000000001.00000000:head [omap-rm-keys in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) v8 -- 0x5576f3cd8b20 con 0x5576f3c4ffa0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.978+0000 7f9972600640 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f99540691b0 0x7f995406b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.978+0000 7f9971c00640 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5576f3c4ffa0 0x5576f3cd85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.978+0000 7f9971c00640 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5576f3c4ffa0 0x5576f3cd85e0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.979+0000 7f9972600640 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f99540691b0 0x7f995406b670 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9964005af0 tx=0x7f9964007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:04.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9971c00640 1 -- 172.21.15.62:0/2619637475 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== osd_op_reply(1 10000000001.00000000 [omap-rm-keys] v121'5 uv5 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f9960053820 con 0x5576f3c4ffa0 2024-03-23T11:32:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9975c3df40 1 -- 172.21.15.62:0/2619637475 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5576f3c4ffa0 msgr2=0x5576f3cd85e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9975c3df40 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5576f3c4ffa0 0x5576f3cd85e0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9975c3df40 1 -- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f99540691b0 msgr2=0x7f995406b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9975c3df40 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f99540691b0 0x7f995406b670 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9964005af0 tx=0x7f9964007040 comp rx=0 tx=0).stop 2024-03-23T11:32:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9975c3df40 1 -- 172.21.15.62:0/2619637475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99580030b0 msgr2=0x5576f3c4f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9975c3df40 1 --2- 172.21.15.62:0/2619637475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99580030b0 0x5576f3c4f2c0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f996c000ff0 tx=0x7f996c001930 comp rx=0 tx=0).stop 2024-03-23T11:32:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9973000640 1 -- 172.21.15.62:0/2619637475 reap_dead start 2024-03-23T11:32:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9975c3df40 1 -- 172.21.15.62:0/2619637475 shutdown_connections 2024-03-23T11:32:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9975c3df40 1 -- 172.21.15.62:0/2619637475 >> 172.21.15.62:0/2619637475 conn(0x5576f3c50ff0 msgr2=0x5576f3c513d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9975c3df40 1 -- 172.21.15.62:0/2619637475 shutdown_connections 2024-03-23T11:32:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:04.980+0000 7f9975c3df40 1 -- 172.21.15.62:0/2619637475 wait complete. 2024-03-23T11:32:04.975 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-23T11:32:05.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.177+0000 7f0cb0a00640 1 Processor -- start 2024-03-23T11:32:05.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.177+0000 7f0cb0a00640 1 -- start start 2024-03-23T11:32:05.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.178+0000 7f0cb0a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0cac15a8b0 0x7f0cac15b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:05.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.178+0000 7f0cb0a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0cac0cbfa0 0x7f0cac0cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:05.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.178+0000 7f0cb0a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cac15b3c0 con 0x7f0cac0b0260 2024-03-23T11:32:05.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.178+0000 7f0cb0a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cac0cca40 con 0x7f0cac0cbfa0 2024-03-23T11:32:05.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.178+0000 7f0cb0a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cac0ccd20 con 0x7f0cac15a8b0 2024-03-23T11:32:05.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.179+0000 7f0caaa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0cac0cbfa0 0x7f0cac0cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:05.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.179+0000 7f0caaa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0cac0cbfa0 0x7f0cac0cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51718/0 (socket says 172.21.15.62:51718) 2024-03-23T11:32:05.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.179+0000 7f0caaa00640 1 -- 172.21.15.62:0/1709489868 learned_addr learned my addr 172.21.15.62:0/1709489868 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:05.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.179+0000 7f0cabe00640 1 --1- 172.21.15.62:0/1709489868 >> v1:172.21.15.62:6789/0 conn(0x7f0cac0b0260 0x7f0cac0aeef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:51588/0 (socket says 172.21.15.62:51588) 2024-03-23T11:32:05.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.179+0000 7f0cab400640 1 --2- 172.21.15.62:0/1709489868 >> v2:172.21.15.120:3300/0 conn(0x7f0cac15a8b0 0x7f0cac15b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:05.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.180+0000 7f0caaa00640 1 -- 172.21.15.62:0/1709489868 >> v2:172.21.15.120:3300/0 conn(0x7f0cac15a8b0 msgr2=0x7f0cac15b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:05.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.180+0000 7f0caaa00640 1 --2- 172.21.15.62:0/1709489868 >> v2:172.21.15.120:3300/0 conn(0x7f0cac15a8b0 0x7f0cac15b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:05.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.180+0000 7f0caaa00640 1 -- 172.21.15.62:0/1709489868 >> v1:172.21.15.62:6789/0 conn(0x7f0cac0b0260 legacy=0x7f0cac0aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:05.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.180+0000 7f0caaa00640 1 -- 172.21.15.62:0/1709489868 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cac15b050 con 0x7f0cac0cbfa0 2024-03-23T11:32:05.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.180+0000 7f0cab400640 1 --2- 172.21.15.62:0/1709489868 >> v2:172.21.15.120:3300/0 conn(0x7f0cac15a8b0 0x7f0cac15b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:05.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.181+0000 7f0caaa00640 1 --2- 172.21.15.62:0/1709489868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cac0cbfa0 0x7f0cac0cc3e0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f0cac070e70 tx=0x7f0c9c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a32d32754d9aa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:05.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.181+0000 7f0caa000640 1 -- 172.21.15.62:0/1709489868 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c02cb10 con 0x7f0cac0cbfa0 2024-03-23T11:32:05.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.181+0000 7f0caa000640 1 -- 172.21.15.62:0/1709489868 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9c02cc70 con 0x7f0cac0cbfa0 2024-03-23T11:32:05.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.181+0000 7f0caa000640 1 -- 172.21.15.62:0/1709489868 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c0343f0 con 0x7f0cac0cbfa0 2024-03-23T11:32:05.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.182+0000 7f0cb0a00640 1 -- 172.21.15.62:0/1709489868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cac0cbfa0 msgr2=0x7f0cac0cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:05.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.182+0000 7f0cb0a00640 1 --2- 172.21.15.62:0/1709489868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cac0cbfa0 0x7f0cac0cc3e0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f0cac070e70 tx=0x7f0c9c02bf70 comp rx=0 tx=0).stop 2024-03-23T11:32:05.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.182+0000 7f0cb0a00640 1 -- 172.21.15.62:0/1709489868 shutdown_connections 2024-03-23T11:32:05.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.182+0000 7f0cb0a00640 1 --2- 172.21.15.62:0/1709489868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cac0cbfa0 0x7f0cac0cc3e0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:05.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.182+0000 7f0cb0a00640 1 --2- 172.21.15.62:0/1709489868 >> v2:172.21.15.120:3300/0 conn(0x7f0cac15a8b0 0x7f0cac15b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:05.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.182+0000 7f0cb0a00640 1 -- 172.21.15.62:0/1709489868 >> 172.21.15.62:0/1709489868 conn(0x7f0cac0989d0 msgr2=0x7f0cac0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:05.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.182+0000 7f0cb0a00640 1 -- 172.21.15.62:0/1709489868 shutdown_connections 2024-03-23T11:32:05.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.182+0000 7f0cb0a00640 1 -- 172.21.15.62:0/1709489868 wait complete. 2024-03-23T11:32:05.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.184+0000 7f0cb0a00640 1 Processor -- start 2024-03-23T11:32:05.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.184+0000 7f0cb0a00640 1 -- start start 2024-03-23T11:32:05.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.185+0000 7f0cb0a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cac0b0260 0x7f0cac0c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.185+0000 7f0cb0a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cac0cbfa0 0x7f0cac0c3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.185+0000 7f0cb0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cac15a8b0 0x7f0cac0c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.185+0000 7f0cb0a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac0c74b0 con 0x7f0cac15a8b0 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.185+0000 7f0cb0a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac0c7790 con 0x7f0cac0cbfa0 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.185+0000 7f0cb0a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac0c7a70 con 0x7f0cac0b0260 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.185+0000 7f0cabe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cac15a8b0 0x7f0cac0c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.185+0000 7f0cabe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cac15a8b0 0x7f0cac0c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51734/0 (socket says 172.21.15.62:51734) 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.185+0000 7f0cabe00640 1 -- 172.21.15.62:0/354775013 learned_addr learned my addr 172.21.15.62:0/354775013 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.186+0000 7f0cab400640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cac0b0260 0x7f0cac0c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.186+0000 7f0caaa00640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cac0cbfa0 0x7f0cac0c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.186+0000 7f0cabe00640 1 -- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cac0b0260 msgr2=0x7f0cac0c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.186+0000 7f0cabe00640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cac0b0260 0x7f0cac0c30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:05.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.186+0000 7f0cabe00640 1 -- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cac0cbfa0 msgr2=0x7f0cac0c3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:05.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.186+0000 7f0cabe00640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cac0cbfa0 0x7f0cac0c3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:05.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.186+0000 7f0cabe00640 1 -- 172.21.15.62:0/354775013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9c009b30 con 0x7f0cac15a8b0 2024-03-23T11:32:05.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.186+0000 7f0caaa00640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cac0cbfa0 0x7f0cac0c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:05.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.186+0000 7f0cabe00640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cac15a8b0 0x7f0cac0c6740 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400c400 tx=0x7f0ca400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:05.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.186+0000 7f0ca3400640 1 -- 172.21.15.62:0/354775013 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca40074e0 con 0x7f0cac15a8b0 2024-03-23T11:32:05.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.187+0000 7f0ca3400640 1 -- 172.21.15.62:0/354775013 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca4007640 con 0x7f0cac15a8b0 2024-03-23T11:32:05.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.187+0000 7f0ca3400640 1 -- 172.21.15.62:0/354775013 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca40049e0 con 0x7f0cac15a8b0 2024-03-23T11:32:05.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.187+0000 7f0cb0a00640 1 -- 172.21.15.62:0/354775013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cac0c7d50 con 0x7f0cac15a8b0 2024-03-23T11:32:05.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.187+0000 7f0cb0a00640 1 -- 172.21.15.62:0/354775013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cac14ed20 con 0x7f0cac15a8b0 2024-03-23T11:32:05.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.188+0000 7f0cb0a00640 1 -- 172.21.15.62:0/354775013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cac155e50 con 0x7f0cac15a8b0 2024-03-23T11:32:05.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.189+0000 7f0ca3400640 1 -- 172.21.15.62:0/354775013 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f0ca4004b40 con 0x7f0cac15a8b0 2024-03-23T11:32:05.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.190+0000 7f0ca3400640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c8806a7d0 0x7f0c8806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:05.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.190+0000 7f0ca3400640 1 -- 172.21.15.62:0/354775013 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0ca4088950 con 0x7f0cac15a8b0 2024-03-23T11:32:05.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.190+0000 7f0cab400640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c8806a7d0 0x7f0c8806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:05.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.191+0000 7f0cab400640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c8806a7d0 0x7f0c8806cc90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0c94006fd0 tx=0x7f0c94008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:05.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.194+0000 7f0ca3400640 1 -- 172.21.15.62:0/354775013 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0cac155e50 con 0x7f0cac15a8b0 2024-03-23T11:32:05.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:05.433+0000 7f0cb0a00640 1 -- 172.21.15.62:0/354775013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f0cac155e50 con 0x7f0cac15a8b0 2024-03-23T11:32:06.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.443+0000 7f0ca3400640 1 -- 172.21.15.62:0/354775013 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v97) v1 ==== 162+0+0 (secure 0 0 0) 0x7f0cac155e50 con 0x7f0cac15a8b0 2024-03-23T11:32:06.433 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-23T11:32:06.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.447+0000 7f0cb0a00640 1 -- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c8806a7d0 msgr2=0x7f0c8806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.447+0000 7f0cb0a00640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c8806a7d0 0x7f0c8806cc90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0c94006fd0 tx=0x7f0c94008040 comp rx=0 tx=0).stop 2024-03-23T11:32:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.447+0000 7f0cb0a00640 1 -- 172.21.15.62:0/354775013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cac15a8b0 msgr2=0x7f0cac0c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.447+0000 7f0cb0a00640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cac15a8b0 0x7f0cac0c6740 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f0ca400c400 tx=0x7f0ca400c8c0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.448+0000 7f0cb0a00640 1 -- 172.21.15.62:0/354775013 shutdown_connections 2024-03-23T11:32:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.448+0000 7f0cb0a00640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f0c8806a7d0 0x7f0c8806cc90 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.448+0000 7f0cb0a00640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cac15a8b0 0x7f0cac0c6740 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.448+0000 7f0cb0a00640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cac0cbfa0 0x7f0cac0c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.448+0000 7f0cb0a00640 1 --2- 172.21.15.62:0/354775013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cac0b0260 0x7f0cac0c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.448+0000 7f0cb0a00640 1 -- 172.21.15.62:0/354775013 >> 172.21.15.62:0/354775013 conn(0x7f0cac0989d0 msgr2=0x7f0cac0ae710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.448+0000 7f0cb0a00640 1 -- 172.21.15.62:0/354775013 shutdown_connections 2024-03-23T11:32:06.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.448+0000 7f0cb0a00640 1 -- 172.21.15.62:0/354775013 wait complete. 2024-03-23T11:32:06.463 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:32:06.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.677+0000 7f8b63e00640 1 Processor -- start 2024-03-23T11:32:06.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.677+0000 7f8b63e00640 1 -- start start 2024-03-23T11:32:06.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.678+0000 7f8b63e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8b640b0980 0x7f8b64161100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:06.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.678+0000 7f8b63e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b6406a370 0x7f8b640ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:06.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.679+0000 7f8b63e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b640ced60 con 0x7f8b640cb8c0 2024-03-23T11:32:06.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.679+0000 7f8b63e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b640cf2a0 con 0x7f8b640b0980 2024-03-23T11:32:06.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.679+0000 7f8b63e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b640cefc0 con 0x7f8b6406a370 2024-03-23T11:32:06.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.679+0000 7f8b62a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8b640b0980 0x7f8b64161100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.679+0000 7f8b62a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8b640b0980 0x7f8b64161100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:51248/0 (socket says 172.21.15.62:51248) 2024-03-23T11:32:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.679+0000 7f8b62a00640 1 -- 172.21.15.62:0/2969745644 learned_addr learned my addr 172.21.15.62:0/2969745644 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.679+0000 7f8b63400640 1 --2- 172.21.15.62:0/2969745644 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b6406a370 0x7f8b640ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.679+0000 7f8b62000640 1 --1- 172.21.15.62:0/2969745644 >> v1:172.21.15.120:6789/0 conn(0x7f8b640cb8c0 0x7f8b64069c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:39266/0 (socket says 172.21.15.62:39266) 2024-03-23T11:32:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.680+0000 7f8b62a00640 1 -- 172.21.15.62:0/2969745644 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b6406a370 msgr2=0x7f8b640ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.680+0000 7f8b62a00640 1 --2- 172.21.15.62:0/2969745644 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b6406a370 0x7f8b640ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.680+0000 7f8b62a00640 1 -- 172.21.15.62:0/2969745644 >> v1:172.21.15.120:6789/0 conn(0x7f8b640cb8c0 legacy=0x7f8b64069c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.680+0000 7f8b62a00640 1 -- 172.21.15.62:0/2969745644 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b64160620 con 0x7f8b640b0980 2024-03-23T11:32:06.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.682+0000 7f8b62a00640 1 --2- 172.21.15.62:0/2969745644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b640b0980 0x7f8b64161100 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009870 tx=0x7f8b5402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e14b7aee3fbd0c07 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:06.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.682+0000 7f8b61600640 1 -- 172.21.15.62:0/2969745644 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54030840 con 0x7f8b640b0980 2024-03-23T11:32:06.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.682+0000 7f8b61600640 1 -- 172.21.15.62:0/2969745644 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b540309a0 con 0x7f8b640b0980 2024-03-23T11:32:06.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.682+0000 7f8b61600640 1 -- 172.21.15.62:0/2969745644 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54030d30 con 0x7f8b640b0980 2024-03-23T11:32:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.683+0000 7f8b63e00640 1 -- 172.21.15.62:0/2969745644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b640b0980 msgr2=0x7f8b64161100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.683+0000 7f8b63e00640 1 --2- 172.21.15.62:0/2969745644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b640b0980 0x7f8b64161100 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009870 tx=0x7f8b5402b980 comp rx=0 tx=0).stop 2024-03-23T11:32:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.683+0000 7f8b63e00640 1 -- 172.21.15.62:0/2969745644 shutdown_connections 2024-03-23T11:32:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.683+0000 7f8b63e00640 1 --2- 172.21.15.62:0/2969745644 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b6406a370 0x7f8b640ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.683+0000 7f8b63e00640 1 --2- 172.21.15.62:0/2969745644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b640b0980 0x7f8b64161100 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.683+0000 7f8b63e00640 1 -- 172.21.15.62:0/2969745644 >> 172.21.15.62:0/2969745644 conn(0x7f8b640989d0 msgr2=0x7f8b640cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.683+0000 7f8b63e00640 1 -- 172.21.15.62:0/2969745644 shutdown_connections 2024-03-23T11:32:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.683+0000 7f8b63e00640 1 -- 172.21.15.62:0/2969745644 wait complete. 2024-03-23T11:32:06.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.686+0000 7f8b63e00640 1 Processor -- start 2024-03-23T11:32:06.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.686+0000 7f8b63e00640 1 -- start start 2024-03-23T11:32:06.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.687+0000 7f8b63e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b6406a370 0x7f8b641f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:06.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.687+0000 7f8b63e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b640b0980 0x7f8b6406e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:06.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.687+0000 7f8b63e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b640cb8c0 0x7f8b64158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:06.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.687+0000 7f8b63e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b640c30f0 con 0x7f8b640b0980 2024-03-23T11:32:06.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.687+0000 7f8b63e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b640c8ee0 con 0x7f8b6406a370 2024-03-23T11:32:06.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.687+0000 7f8b63e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64159d20 con 0x7f8b640cb8c0 2024-03-23T11:32:06.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.688+0000 7f8b62000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b640b0980 0x7f8b6406e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:06.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.688+0000 7f8b62000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b640b0980 0x7f8b6406e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51740/0 (socket says 172.21.15.62:51740) 2024-03-23T11:32:06.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.688+0000 7f8b62000640 1 -- 172.21.15.62:0/1322556206 learned_addr learned my addr 172.21.15.62:0/1322556206 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:06.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.688+0000 7f8b62a00640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b6406a370 0x7f8b641f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:06.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.688+0000 7f8b63400640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b640cb8c0 0x7f8b64158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:06.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.688+0000 7f8b62000640 1 -- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b640cb8c0 msgr2=0x7f8b64158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:06.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.688+0000 7f8b62000640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b640cb8c0 0x7f8b64158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.688+0000 7f8b62000640 1 -- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b6406a370 msgr2=0x7f8b641f6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:06.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.688+0000 7f8b62000640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b6406a370 0x7f8b641f6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.688+0000 7f8b62000640 1 -- 172.21.15.62:0/1322556206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b540094d0 con 0x7f8b640b0980 2024-03-23T11:32:06.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.689+0000 7f8b62000640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b640b0980 0x7f8b6406e760 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c00c7d0 tx=0x7f8b4c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:06.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.689+0000 7f8b5aa00640 1 -- 172.21.15.62:0/1322556206 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c00a640 con 0x7f8b640b0980 2024-03-23T11:32:06.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.689+0000 7f8b5aa00640 1 -- 172.21.15.62:0/1322556206 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4c00a7a0 con 0x7f8b640b0980 2024-03-23T11:32:06.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.689+0000 7f8b63e00640 1 -- 172.21.15.62:0/1322556206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b640c6140 con 0x7f8b640b0980 2024-03-23T11:32:06.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.689+0000 7f8b5aa00640 1 -- 172.21.15.62:0/1322556206 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c00ab10 con 0x7f8b640b0980 2024-03-23T11:32:06.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.689+0000 7f8b63e00640 1 -- 172.21.15.62:0/1322556206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b641596d0 con 0x7f8b640b0980 2024-03-23T11:32:06.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.690+0000 7f8b63e00640 1 -- 172.21.15.62:0/1322556206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b64159eb0 con 0x7f8b640b0980 2024-03-23T11:32:06.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.692+0000 7f8b5aa00640 1 -- 172.21.15.62:0/1322556206 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8b4c00ac70 con 0x7f8b640b0980 2024-03-23T11:32:06.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.693+0000 7f8b5aa00640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b4006a7d0 0x7f8b4006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:06.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.693+0000 7f8b5aa00640 1 -- 172.21.15.62:0/1322556206 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8b4c088590 con 0x7f8b640b0980 2024-03-23T11:32:06.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.693+0000 7f8b62a00640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b4006a7d0 0x7f8b4006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:06.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.695+0000 7f8b62a00640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b4006a7d0 0x7f8b4006cc90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8b54036000 tx=0x7f8b54008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:06.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.697+0000 7f8b5aa00640 1 -- 172.21.15.62:0/1322556206 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8b4c053930 con 0x7f8b640b0980 2024-03-23T11:32:06.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.942+0000 7f8b63e00640 1 -- 172.21.15.62:0/1322556206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b640672c0 con 0x7f8b640b0980 2024-03-23T11:32:06.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.944+0000 7f8b5aa00640 1 -- 172.21.15.62:0/1322556206 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f8b4c001820 con 0x7f8b640b0980 2024-03-23T11:32:06.934 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:06.934 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:32:06.444204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9000":{"gid":9000,"name":"a","rank":0,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.62:6835/4129977589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4129977589},{"type":"v1","addr":"172.21.15.62:6835","nonce":4129977589}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:32:06.934 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 98 2024-03-23T11:32:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 -- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b4006a7d0 msgr2=0x7f8b4006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b4006a7d0 0x7f8b4006cc90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8b54036000 tx=0x7f8b54008030 comp rx=0 tx=0).stop 2024-03-23T11:32:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 -- 172.21.15.62:0/1322556206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b640b0980 msgr2=0x7f8b6406e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b640b0980 0x7f8b6406e760 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c00c7d0 tx=0x7f8b4c00cc90 comp rx=0 tx=0).stop 2024-03-23T11:32:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 -- 172.21.15.62:0/1322556206 shutdown_connections 2024-03-23T11:32:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8b4006a7d0 0x7f8b4006cc90 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b640b0980 0x7f8b6406e760 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8b640cb8c0 0x7f8b64158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 --2- 172.21.15.62:0/1322556206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b6406a370 0x7f8b641f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 -- 172.21.15.62:0/1322556206 >> 172.21.15.62:0/1322556206 conn(0x7f8b640989d0 msgr2=0x7f8b6415cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:06.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 -- 172.21.15.62:0/1322556206 shutdown_connections 2024-03-23T11:32:06.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:06.947+0000 7f8b63e00640 1 -- 172.21.15.62:0/1322556206 wait complete. 2024-03-23T11:32:07.359 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:31:24.044342+0000', 'modified': '2024-03-23T11:32:06.444204+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 121, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9000}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9000': {'gid': 9000, 'name': 'a', 'rank': 0, 'incarnation': 98, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.62:6835/4129977589', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6834', 'nonce': 4129977589}, {'type': 'v1', 'addr': '172.21.15.62:6835', 'nonce': 4129977589}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:32:07.360 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9000:up:replay 2024-03-23T11:32:08.360 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:32:08.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.600+0000 7fc980800640 1 Processor -- start 2024-03-23T11:32:08.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.600+0000 7fc980800640 1 -- start start 2024-03-23T11:32:08.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.602+0000 7fc980800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc9780cafa0 0x7fc9780cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:08.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.602+0000 7fc980800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9780cbc60 con 0x7fc97806c300 2024-03-23T11:32:08.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.602+0000 7fc980800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9780cbf40 con 0x7fc97806c6e0 2024-03-23T11:32:08.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.602+0000 7fc980800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9780cd410 con 0x7fc9780cafa0 2024-03-23T11:32:08.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.602+0000 7fc97de00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc97806c300 0x7fc9780ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37380/0 (socket says 172.21.15.62:37380) 2024-03-23T11:32:08.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.602+0000 7fc97de00640 1 -- 172.21.15.62:0/551115663 learned_addr learned my addr 172.21.15.62:0/551115663 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:08.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.602+0000 7fc97d400640 1 --2- 172.21.15.62:0/551115663 >> v2:172.21.15.62:3300/0 conn(0x7fc9780cafa0 0x7fc9780cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.603+0000 7fc97ca00640 1 -- 172.21.15.62:0/551115663 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143376208 0 0) 0x7fc9780cbc60 con 0x7fc97806c300 2024-03-23T11:32:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.603+0000 7fc97d400640 1 -- 172.21.15.62:0/551115663 >> v1:172.21.15.120:6789/0 conn(0x7fc97806c6e0 legacy=0x7fc97806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.603+0000 7fc97d400640 1 -- 172.21.15.62:0/551115663 >> v1:172.21.15.62:6789/0 conn(0x7fc97806c300 legacy=0x7fc9780ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.603+0000 7fc97d400640 1 -- 172.21.15.62:0/551115663 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9780cc220 con 0x7fc9780cafa0 2024-03-23T11:32:08.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.604+0000 7fc97d400640 1 --2- 172.21.15.62:0/551115663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9780cafa0 0x7fc9780cb430 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fc9680097a0 tx=0x7fc96802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cffdc05b78cbd2c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:08.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.605+0000 7fc97ca00640 1 -- 172.21.15.62:0/551115663 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc968031830 con 0x7fc9780cafa0 2024-03-23T11:32:08.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.605+0000 7fc97ca00640 1 -- 172.21.15.62:0/551115663 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc968031990 con 0x7fc9780cafa0 2024-03-23T11:32:08.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.605+0000 7fc97ca00640 1 -- 172.21.15.62:0/551115663 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc968031d20 con 0x7fc9780cafa0 2024-03-23T11:32:08.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.605+0000 7fc980800640 1 -- 172.21.15.62:0/551115663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9780cafa0 msgr2=0x7fc9780cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:08.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.605+0000 7fc980800640 1 --2- 172.21.15.62:0/551115663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9780cafa0 0x7fc9780cb430 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fc9680097a0 tx=0x7fc96802b8d0 comp rx=0 tx=0).stop 2024-03-23T11:32:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.605+0000 7fc980800640 1 -- 172.21.15.62:0/551115663 shutdown_connections 2024-03-23T11:32:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.605+0000 7fc980800640 1 --2- 172.21.15.62:0/551115663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9780cafa0 0x7fc9780cb430 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.606+0000 7fc980800640 1 -- 172.21.15.62:0/551115663 >> 172.21.15.62:0/551115663 conn(0x7fc9780b02e0 msgr2=0x7fc9780b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.606+0000 7fc980800640 1 -- 172.21.15.62:0/551115663 shutdown_connections 2024-03-23T11:32:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.606+0000 7fc980800640 1 -- 172.21.15.62:0/551115663 wait complete. 2024-03-23T11:32:08.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.607+0000 7fc980800640 1 Processor -- start 2024-03-23T11:32:08.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.607+0000 7fc980800640 1 -- start start 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.608+0000 7fc980800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc97806c300 0x7fc9780acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.608+0000 7fc980800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc97806c6e0 0x7fc978139d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.608+0000 7fc980800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9780cafa0 0x7fc97813a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.608+0000 7fc980800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97813c240 con 0x7fc9780cafa0 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.608+0000 7fc980800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97813a800 con 0x7fc97806c300 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.608+0000 7fc980800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97813b630 con 0x7fc97806c6e0 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.609+0000 7fc97e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9780cafa0 0x7fc97813a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.609+0000 7fc97e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9780cafa0 0x7fc97813a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42348/0 (socket says 172.21.15.62:42348) 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.609+0000 7fc97e800640 1 -- 172.21.15.62:0/3857279662 learned_addr learned my addr 172.21.15.62:0/3857279662 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.609+0000 7fc97de00640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc97806c300 0x7fc9780acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.609+0000 7fc97d400640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc97806c6e0 0x7fc978139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.609+0000 7fc97e800640 1 -- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc97806c6e0 msgr2=0x7fc978139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:08.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.609+0000 7fc97e800640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc97806c6e0 0x7fc978139d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:08.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.609+0000 7fc97e800640 1 -- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc97806c300 msgr2=0x7fc9780acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:08.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.609+0000 7fc97e800640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc97806c300 0x7fc9780acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:08.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.609+0000 7fc97e800640 1 -- 172.21.15.62:0/3857279662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc968009400 con 0x7fc9780cafa0 2024-03-23T11:32:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.610+0000 7fc97e800640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9780cafa0 0x7fc97813a2c0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fc97400c780 tx=0x7fc97400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.610+0000 7fc972000640 1 -- 172.21.15.62:0/3857279662 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974004b60 con 0x7fc9780cafa0 2024-03-23T11:32:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.610+0000 7fc972000640 1 -- 172.21.15.62:0/3857279662 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97400ce90 con 0x7fc9780cafa0 2024-03-23T11:32:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.610+0000 7fc972000640 1 -- 172.21.15.62:0/3857279662 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97400a850 con 0x7fc9780cafa0 2024-03-23T11:32:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.610+0000 7fc980800640 1 -- 172.21.15.62:0/3857279662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97813b910 con 0x7fc9780cafa0 2024-03-23T11:32:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.610+0000 7fc980800640 1 -- 172.21.15.62:0/3857279662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97813bed0 con 0x7fc9780cafa0 2024-03-23T11:32:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.611+0000 7fc980800640 1 -- 172.21.15.62:0/3857279662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9780cde60 con 0x7fc9780cafa0 2024-03-23T11:32:08.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.612+0000 7fc972000640 1 -- 172.21.15.62:0/3857279662 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc97400a9b0 con 0x7fc9780cafa0 2024-03-23T11:32:08.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.613+0000 7fc972000640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc95c06a7d0 0x7fc95c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:08.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.613+0000 7fc972000640 1 -- 172.21.15.62:0/3857279662 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc974088ff0 con 0x7fc9780cafa0 2024-03-23T11:32:08.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.614+0000 7fc97de00640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc95c06a7d0 0x7fc95c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:08.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.615+0000 7fc97de00640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc95c06a7d0 0x7fc95c06cc90 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc96c000f60 tx=0x7fc96c007b50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:08.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.617+0000 7fc972000640 1 -- 172.21.15.62:0/3857279662 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc9740bc580 con 0x7fc9780cafa0 2024-03-23T11:32:08.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.852+0000 7fc980800640 1 -- 172.21.15.62:0/3857279662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc978066c90 con 0x7fc9780cafa0 2024-03-23T11:32:08.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.854+0000 7fc972000640 1 -- 172.21.15.62:0/3857279662 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fc974054300 con 0x7fc9780cafa0 2024-03-23T11:32:08.843 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:08.844 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":100,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:32:08.454105+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9000":{"gid":9000,"name":"a","rank":0,"incarnation":98,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.62:6835/4129977589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4129977589},{"type":"v1","addr":"172.21.15.62:6835","nonce":4129977589}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:32:08.844 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 100 2024-03-23T11:32:08.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.856+0000 7fc980800640 1 -- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc95c06a7d0 msgr2=0x7fc95c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:08.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.856+0000 7fc980800640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc95c06a7d0 0x7fc95c06cc90 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc96c000f60 tx=0x7fc96c007b50 comp rx=0 tx=0).stop 2024-03-23T11:32:08.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.856+0000 7fc980800640 1 -- 172.21.15.62:0/3857279662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9780cafa0 msgr2=0x7fc97813a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:08.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.856+0000 7fc980800640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9780cafa0 0x7fc97813a2c0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fc97400c780 tx=0x7fc97400cc40 comp rx=0 tx=0).stop 2024-03-23T11:32:08.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.856+0000 7fc980800640 1 -- 172.21.15.62:0/3857279662 shutdown_connections 2024-03-23T11:32:08.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.856+0000 7fc980800640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc95c06a7d0 0x7fc95c06cc90 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.856+0000 7fc980800640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9780cafa0 0x7fc97813a2c0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.856+0000 7fc980800640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc97806c6e0 0x7fc978139d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.856+0000 7fc980800640 1 --2- 172.21.15.62:0/3857279662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc97806c300 0x7fc9780acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.856+0000 7fc980800640 1 -- 172.21.15.62:0/3857279662 >> 172.21.15.62:0/3857279662 conn(0x7fc9780b02e0 msgr2=0x7fc97815a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.857+0000 7fc980800640 1 -- 172.21.15.62:0/3857279662 shutdown_connections 2024-03-23T11:32:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:08.857+0000 7fc980800640 1 -- 172.21.15.62:0/3857279662 wait complete. 2024-03-23T11:32:09.269 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:31:24.044342+0000', 'modified': '2024-03-23T11:32:08.454105+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 121, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9000}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9000': {'gid': 9000, 'name': 'a', 'rank': 0, 'incarnation': 98, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.62:6835/4129977589', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6834', 'nonce': 4129977589}, {'type': 'v1', 'addr': '172.21.15.62:6835', 'nonce': 4129977589}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:32:09.270 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9000:up:rejoin 2024-03-23T11:32:09.513 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-03-23T11:32:09.523+0000 7f045d600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:32:10.270 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:32:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.504+0000 7fe24de00640 1 Processor -- start 2024-03-23T11:32:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.505+0000 7fe24de00640 1 -- start start 2024-03-23T11:32:10.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.506+0000 7fe24de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe24815c9e0 0x7fe2481607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:10.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.506+0000 7fe24de00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe24815cdc0 0x7fe2480d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:10.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.506+0000 7fe24de00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24806bfc0 con 0x7fe2480cf650 2024-03-23T11:32:10.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.506+0000 7fe24de00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2480ce140 con 0x7fe24815cdc0 2024-03-23T11:32:10.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.506+0000 7fe24de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480b4370 con 0x7fe24815c9e0 2024-03-23T11:32:10.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.507+0000 7fe246000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe24815cdc0 0x7fe2480d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:10.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.507+0000 7fe246000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe24815cdc0 0x7fe2480d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:33010/0 (socket says 172.21.15.62:33010) 2024-03-23T11:32:10.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.507+0000 7fe246000640 1 -- 172.21.15.62:0/85898904 learned_addr learned my addr 172.21.15.62:0/85898904 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:10.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.507+0000 7fe246a00640 1 --2- 172.21.15.62:0/85898904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe24815c9e0 0x7fe2481607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:10.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.508+0000 7fe246a00640 1 -- 172.21.15.62:0/85898904 >> v2:172.21.15.120:3300/0 conn(0x7fe24815cdc0 msgr2=0x7fe2480d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:10.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.508+0000 7fe246a00640 1 --2- 172.21.15.62:0/85898904 >> v2:172.21.15.120:3300/0 conn(0x7fe24815cdc0 0x7fe2480d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:10.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.508+0000 7fe246a00640 1 -- 172.21.15.62:0/85898904 >> v1:172.21.15.120:6789/0 conn(0x7fe2480cf650 legacy=0x7fe24815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:10.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.508+0000 7fe246a00640 1 -- 172.21.15.62:0/85898904 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2480b5260 con 0x7fe24815c9e0 2024-03-23T11:32:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.509+0000 7fe246a00640 1 --2- 172.21.15.62:0/85898904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe24815c9e0 0x7fe2481607c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe2380097a0 tx=0x7fe23802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bb65e579502a52 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.510+0000 7fe245600640 1 -- 172.21.15.62:0/85898904 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23802d040 con 0x7fe24815c9e0 2024-03-23T11:32:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.510+0000 7fe245600640 1 -- 172.21.15.62:0/85898904 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe238031810 con 0x7fe24815c9e0 2024-03-23T11:32:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.510+0000 7fe245600640 1 -- 172.21.15.62:0/85898904 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe238031ba0 con 0x7fe24815c9e0 2024-03-23T11:32:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.510+0000 7fe24de00640 1 -- 172.21.15.62:0/85898904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe24815c9e0 msgr2=0x7fe2481607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.510+0000 7fe24de00640 1 --2- 172.21.15.62:0/85898904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe24815c9e0 0x7fe2481607c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe2380097a0 tx=0x7fe23802b8b0 comp rx=0 tx=0).stop 2024-03-23T11:32:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.510+0000 7fe24de00640 1 -- 172.21.15.62:0/85898904 shutdown_connections 2024-03-23T11:32:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.510+0000 7fe24de00640 1 --2- 172.21.15.62:0/85898904 >> v2:172.21.15.120:3300/0 conn(0x7fe24815cdc0 0x7fe2480d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.510+0000 7fe24de00640 1 --2- 172.21.15.62:0/85898904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe24815c9e0 0x7fe2481607c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.510+0000 7fe24de00640 1 -- 172.21.15.62:0/85898904 >> 172.21.15.62:0/85898904 conn(0x7fe248071570 msgr2=0x7fe248071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.511+0000 7fe24de00640 1 -- 172.21.15.62:0/85898904 shutdown_connections 2024-03-23T11:32:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.511+0000 7fe24de00640 1 -- 172.21.15.62:0/85898904 wait complete. 2024-03-23T11:32:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.512+0000 7fe24de00640 1 Processor -- start 2024-03-23T11:32:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.512+0000 7fe24de00640 1 -- start start 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.513+0000 7fe24de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2480cf650 0x7fe2481f2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.513+0000 7fe24de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe24815c9e0 0x7fe2481f5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.513+0000 7fe24de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe24815cdc0 0x7fe2480cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.513+0000 7fe24de00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480c8d20 con 0x7fe24815cdc0 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.513+0000 7fe24de00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2481f59e0 con 0x7fe2480cf650 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.513+0000 7fe24de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2481f5cc0 con 0x7fe24815c9e0 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.514+0000 7fe247400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe24815cdc0 0x7fe2480cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.514+0000 7fe247400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe24815cdc0 0x7fe2480cb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42360/0 (socket says 172.21.15.62:42360) 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.514+0000 7fe247400640 1 -- 172.21.15.62:0/1835752875 learned_addr learned my addr 172.21.15.62:0/1835752875 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.514+0000 7fe246a00640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2480cf650 0x7fe2481f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.514+0000 7fe247400640 1 -- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe24815c9e0 msgr2=0x7fe2481f5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.514+0000 7fe247400640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe24815c9e0 0x7fe2481f5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.514+0000 7fe247400640 1 -- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2480cf650 msgr2=0x7fe2481f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.514+0000 7fe247400640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2480cf650 0x7fe2481f2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.514+0000 7fe247400640 1 -- 172.21.15.62:0/1835752875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe238009400 con 0x7fe24815cdc0 2024-03-23T11:32:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.515+0000 7fe247400640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe24815cdc0 0x7fe2480cb150 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fe24000c7a0 tx=0x7fe24000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.515+0000 7fe23f400640 1 -- 172.21.15.62:0/1835752875 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240003b70 con 0x7fe24815cdc0 2024-03-23T11:32:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.515+0000 7fe23f400640 1 -- 172.21.15.62:0/1835752875 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe240003cf0 con 0x7fe24815cdc0 2024-03-23T11:32:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.515+0000 7fe23f400640 1 -- 172.21.15.62:0/1835752875 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240003ea0 con 0x7fe24815cdc0 2024-03-23T11:32:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.515+0000 7fe24de00640 1 -- 172.21.15.62:0/1835752875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2481f5fa0 con 0x7fe24815cdc0 2024-03-23T11:32:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.515+0000 7fe24de00640 1 -- 172.21.15.62:0/1835752875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2481f6560 con 0x7fe24815cdc0 2024-03-23T11:32:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.516+0000 7fe24de00640 1 -- 172.21.15.62:0/1835752875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2481f66f0 con 0x7fe24815cdc0 2024-03-23T11:32:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.517+0000 7fe23f400640 1 -- 172.21.15.62:0/1835752875 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe24000aa90 con 0x7fe24815cdc0 2024-03-23T11:32:10.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.518+0000 7fe23f400640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe22806a7a0 0x7fe22806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:10.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.518+0000 7fe23f400640 1 -- 172.21.15.62:0/1835752875 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe240088c00 con 0x7fe24815cdc0 2024-03-23T11:32:10.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.519+0000 7fe246a00640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe22806a7a0 0x7fe22806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:10.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.520+0000 7fe246a00640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe22806a7a0 0x7fe22806cc60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe238036000 tx=0x7fe23803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:10.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.523+0000 7fe23f400640 1 -- 172.21.15.62:0/1835752875 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe24008f080 con 0x7fe24815cdc0 2024-03-23T11:32:10.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.765+0000 7fe24de00640 1 -- 172.21.15.62:0/1835752875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe248066c90 con 0x7fe24815cdc0 2024-03-23T11:32:10.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.766+0000 7fe23f400640 1 -- 172.21.15.62:0/1835752875 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fe240054020 con 0x7fe24815cdc0 2024-03-23T11:32:10.756 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:10.756 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":101,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:32:09.464564+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9000":{"gid":9000,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":15,"addr":"172.21.15.62:6835/4129977589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4129977589},{"type":"v1","addr":"172.21.15.62:6835","nonce":4129977589}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:32:10.756 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 101 2024-03-23T11:32:10.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.769+0000 7fe24de00640 1 -- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe22806a7a0 msgr2=0x7fe22806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:10.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.769+0000 7fe24de00640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe22806a7a0 0x7fe22806cc60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe238036000 tx=0x7fe23803c040 comp rx=0 tx=0).stop 2024-03-23T11:32:10.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.769+0000 7fe24de00640 1 -- 172.21.15.62:0/1835752875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe24815cdc0 msgr2=0x7fe2480cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:10.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.769+0000 7fe24de00640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe24815cdc0 0x7fe2480cb150 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fe24000c7a0 tx=0x7fe24000cc60 comp rx=0 tx=0).stop 2024-03-23T11:32:10.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.769+0000 7fe24de00640 1 -- 172.21.15.62:0/1835752875 shutdown_connections 2024-03-23T11:32:10.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.769+0000 7fe24de00640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe22806a7a0 0x7fe22806cc60 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:10.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.769+0000 7fe24de00640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe24815cdc0 0x7fe2480cb150 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:10.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.769+0000 7fe24de00640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe24815c9e0 0x7fe2481f5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:10.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.769+0000 7fe24de00640 1 --2- 172.21.15.62:0/1835752875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2480cf650 0x7fe2481f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:10.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.769+0000 7fe24de00640 1 -- 172.21.15.62:0/1835752875 >> 172.21.15.62:0/1835752875 conn(0x7fe248071570 msgr2=0x7fe2480b7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:10.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.769+0000 7fe24de00640 1 -- 172.21.15.62:0/1835752875 shutdown_connections 2024-03-23T11:32:10.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:10.770+0000 7fe24de00640 1 -- 172.21.15.62:0/1835752875 wait complete. 2024-03-23T11:32:11.181 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:31:24.044342+0000', 'modified': '2024-03-23T11:32:09.464564+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 121, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9000}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9000': {'gid': 9000, 'name': 'a', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.62:6835/4129977589', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6834', 'nonce': 4129977589}, {'type': 'v1', 'addr': '172.21.15.62:6835', 'nonce': 4129977589}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:32:11.181 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:32:11.182 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T11:32:11.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.423+0000 7f7cdbe00640 1 Processor -- start 2024-03-23T11:32:11.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.423+0000 7f7cdbe00640 1 -- start start 2024-03-23T11:32:11.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.424+0000 7f7cdbe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7cdc069c60 0x7f7cdc06a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.424+0000 7f7cdbe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7cdc143450 0x7f7cdc1438a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.424+0000 7f7cdbe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cdc15bb10 con 0x7f7cdc0caff0 2024-03-23T11:32:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.424+0000 7f7cdbe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cdc0ce1a0 con 0x7f7cdc069c60 2024-03-23T11:32:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.424+0000 7f7cdbe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cdc0cbcb0 con 0x7f7cdc143450 2024-03-23T11:32:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.425+0000 7f7cdaa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7cdc069c60 0x7f7cdc06a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.425+0000 7f7cdaa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7cdc069c60 0x7f7cdc06a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42362/0 (socket says 172.21.15.62:42362) 2024-03-23T11:32:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.425+0000 7f7cdaa00640 1 -- 172.21.15.62:0/645836436 learned_addr learned my addr 172.21.15.62:0/645836436 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.425+0000 7f7cda000640 1 --1- 172.21.15.62:0/645836436 >> v1:172.21.15.62:6789/0 conn(0x7f7cdc0caff0 0x7f7cdc0cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37392/0 (socket says 172.21.15.62:37392) 2024-03-23T11:32:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.425+0000 7f7cdb400640 1 --2- 172.21.15.62:0/645836436 >> v2:172.21.15.120:3300/0 conn(0x7f7cdc143450 0x7f7cdc1438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.425+0000 7f7cdaa00640 1 -- 172.21.15.62:0/645836436 >> v2:172.21.15.120:3300/0 conn(0x7f7cdc143450 msgr2=0x7f7cdc1438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.425+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/645836436 >> v2:172.21.15.120:3300/0 conn(0x7f7cdc143450 0x7f7cdc1438a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.425+0000 7f7cd9600640 1 -- 172.21.15.62:0/645836436 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649761365 0 0) 0x7f7cdc15bb10 con 0x7f7cdc0caff0 2024-03-23T11:32:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.425+0000 7f7cdaa00640 1 -- 172.21.15.62:0/645836436 >> v1:172.21.15.62:6789/0 conn(0x7f7cdc0caff0 legacy=0x7f7cdc0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.425+0000 7f7cdaa00640 1 -- 172.21.15.62:0/645836436 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc06a580 con 0x7f7cdc069c60 2024-03-23T11:32:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.425+0000 7f7cdb400640 1 --2- 172.21.15.62:0/645836436 >> v2:172.21.15.120:3300/0 conn(0x7f7cdc143450 0x7f7cdc1438a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.427+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/645836436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cdc069c60 0x7f7cdc06a040 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc06fee0 tx=0x7f7ccc02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd660536942e175 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.427+0000 7f7cd9600640 1 -- 172.21.15.62:0/645836436 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc030850 con 0x7f7cdc069c60 2024-03-23T11:32:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.427+0000 7f7cd9600640 1 -- 172.21.15.62:0/645836436 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc0309b0 con 0x7f7cdc069c60 2024-03-23T11:32:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.427+0000 7f7cd9600640 1 -- 172.21.15.62:0/645836436 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc030d40 con 0x7f7cdc069c60 2024-03-23T11:32:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.427+0000 7f7cdbe00640 1 -- 172.21.15.62:0/645836436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cdc069c60 msgr2=0x7f7cdc06a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.427+0000 7f7cdbe00640 1 --2- 172.21.15.62:0/645836436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cdc069c60 0x7f7cdc06a040 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc06fee0 tx=0x7f7ccc02b990 comp rx=0 tx=0).stop 2024-03-23T11:32:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.428+0000 7f7cdbe00640 1 -- 172.21.15.62:0/645836436 shutdown_connections 2024-03-23T11:32:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.428+0000 7f7cdbe00640 1 --2- 172.21.15.62:0/645836436 >> v2:172.21.15.120:3300/0 conn(0x7f7cdc143450 0x7f7cdc1438a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.428+0000 7f7cdbe00640 1 --2- 172.21.15.62:0/645836436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cdc069c60 0x7f7cdc06a040 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.428+0000 7f7cdbe00640 1 -- 172.21.15.62:0/645836436 >> 172.21.15.62:0/645836436 conn(0x7f7cdc0b0090 msgr2=0x7f7cdc0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.428+0000 7f7cdbe00640 1 -- 172.21.15.62:0/645836436 shutdown_connections 2024-03-23T11:32:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.428+0000 7f7cdbe00640 1 -- 172.21.15.62:0/645836436 wait complete. 2024-03-23T11:32:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.429+0000 7f7cdbe00640 1 Processor -- start 2024-03-23T11:32:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.430+0000 7f7cdbe00640 1 -- start start 2024-03-23T11:32:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.430+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cdc069c60 0x7f7cdc0b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.430+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cdc0caff0 0x7f7cdc1f14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.430+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cdc143450 0x7f7cdc0c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.430+0000 7f7cdbe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc0b2090 con 0x7f7cdc069c60 2024-03-23T11:32:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.430+0000 7f7cdbe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc0b2370 con 0x7f7cdc143450 2024-03-23T11:32:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.430+0000 7f7cdbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc0b2650 con 0x7f7cdc0caff0 2024-03-23T11:32:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.431+0000 7f7cdaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cdc069c60 0x7f7cdc0b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.431+0000 7f7cdaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cdc069c60 0x7f7cdc0b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42366/0 (socket says 172.21.15.62:42366) 2024-03-23T11:32:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.431+0000 7f7cdaa00640 1 -- 172.21.15.62:0/3812728428 learned_addr learned my addr 172.21.15.62:0/3812728428 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.431+0000 7f7cdaa00640 1 -- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cdc0caff0 msgr2=0x7f7cdc1f14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.431+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cdc0caff0 0x7f7cdc1f14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.431+0000 7f7cdaa00640 1 -- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cdc143450 msgr2=0x7f7cdc0c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.431+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cdc143450 0x7f7cdc0c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.431+0000 7f7cdaa00640 1 -- 172.21.15.62:0/3812728428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc0094d0 con 0x7f7cdc069c60 2024-03-23T11:32:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.432+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cdc069c60 0x7f7cdc0b3130 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc000c00 tx=0x7f7ccc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.432+0000 7f7cd2a00640 1 -- 172.21.15.62:0/3812728428 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc030820 con 0x7f7cdc069c60 2024-03-23T11:32:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.432+0000 7f7cd2a00640 1 -- 172.21.15.62:0/3812728428 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc039d90 con 0x7f7cdc069c60 2024-03-23T11:32:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.432+0000 7f7cd2a00640 1 -- 172.21.15.62:0/3812728428 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc043670 con 0x7f7cdc069c60 2024-03-23T11:32:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.432+0000 7f7cdbe00640 1 -- 172.21.15.62:0/3812728428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cdc0b2930 con 0x7f7cdc069c60 2024-03-23T11:32:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.432+0000 7f7cdbe00640 1 -- 172.21.15.62:0/3812728428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cdc1f1a20 con 0x7f7cdc069c60 2024-03-23T11:32:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.434+0000 7f7cd2a00640 1 -- 172.21.15.62:0/3812728428 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7ccc02e070 con 0x7f7cdc069c60 2024-03-23T11:32:11.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.436+0000 7f7cd2a00640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cbc06a7a0 0x7f7cbc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.436+0000 7f7cd2a00640 1 -- 172.21.15.62:0/3812728428 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7ccc0aaff0 con 0x7f7cdc069c60 2024-03-23T11:32:11.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.437+0000 7f7cda000640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cbc06a7a0 0x7f7cbc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.437+0000 7f7ce4829740 1 Processor -- start 2024-03-23T11:32:11.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.437+0000 7f7ce4829740 1 -- start start 2024-03-23T11:32:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.438+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8b00520 0x55c3c8ab7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.438+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8ab65a0 0x55c3c8ab8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.438+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab2a40 0x55c3c8ae2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.438+0000 7f7ce4829740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c8a23110 con 0x55c3c8ab2a40 2024-03-23T11:32:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.438+0000 7f7ce4829740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c8a208a0 con 0x55c3c8ab65a0 2024-03-23T11:32:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.438+0000 7f7ce4829740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c3c8a1c190 con 0x55c3c8b00520 2024-03-23T11:32:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.438+0000 7f7cda000640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cbc06a7a0 0x7f7cbc06cc60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8007fd0 tx=0x7f7cc8007920 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.438+0000 7f7cdb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab2a40 0x55c3c8ae2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.438+0000 7f7cdb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab2a40 0x55c3c8ae2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42370/0 (socket says 172.21.15.62:42370) 2024-03-23T11:32:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.438+0000 7f7cdb400640 1 -- 172.21.15.62:0/522114372 learned_addr learned my addr 172.21.15.62:0/522114372 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:11.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.439+0000 7f7cda000640 1 --2- 172.21.15.62:0/522114372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8ab65a0 0x55c3c8ab8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.439+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/522114372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8b00520 0x55c3c8ab7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.439+0000 7f7cdb400640 1 -- 172.21.15.62:0/522114372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8b00520 msgr2=0x55c3c8ab7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:11.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.439+0000 7f7cdb400640 1 --2- 172.21.15.62:0/522114372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8b00520 0x55c3c8ab7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.439+0000 7f7cdb400640 1 -- 172.21.15.62:0/522114372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8ab65a0 msgr2=0x55c3c8ab8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:11.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.439+0000 7f7cdb400640 1 --2- 172.21.15.62:0/522114372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8ab65a0 0x55c3c8ab8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.439+0000 7f7cdb400640 1 -- 172.21.15.62:0/522114372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3c89e7230 con 0x55c3c8ab2a40 2024-03-23T11:32:11.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.439+0000 7f7cdb400640 1 --2- 172.21.15.62:0/522114372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab2a40 0x55c3c8ae2830 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4008f00 tx=0x7f7cd400d7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b242c105b0398f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.439+0000 7f7cc7400640 1 -- 172.21.15.62:0/522114372 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd400e850 con 0x55c3c8ab2a40 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.439+0000 7f7cc7400640 1 -- 172.21.15.62:0/522114372 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd400e9b0 con 0x55c3c8ab2a40 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.440+0000 7f7cc7400640 1 -- 172.21.15.62:0/522114372 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd400ed20 con 0x55c3c8ab2a40 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.440+0000 7f7ce4829740 1 -- 172.21.15.62:0/522114372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab2a40 msgr2=0x55c3c8ae2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.440+0000 7f7ce4829740 1 --2- 172.21.15.62:0/522114372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab2a40 0x55c3c8ae2830 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4008f00 tx=0x7f7cd400d7a0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.440+0000 7f7ce4829740 1 -- 172.21.15.62:0/522114372 shutdown_connections 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.440+0000 7f7ce4829740 1 --2- 172.21.15.62:0/522114372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab2a40 0x55c3c8ae2830 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.440+0000 7f7ce4829740 1 --2- 172.21.15.62:0/522114372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8ab65a0 0x55c3c8ab8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.440+0000 7f7ce4829740 1 --2- 172.21.15.62:0/522114372 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8b00520 0x55c3c8ab7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.440+0000 7f7ce4829740 1 -- 172.21.15.62:0/522114372 >> 172.21.15.62:0/522114372 conn(0x55c3c8ab4cb0 msgr2=0x55c3c8afd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.440+0000 7f7ce4829740 1 -- 172.21.15.62:0/522114372 shutdown_connections 2024-03-23T11:32:11.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.440+0000 7f7ce4829740 1 -- 172.21.15.62:0/522114372 wait complete. 2024-03-23T11:32:11.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.441+0000 7f7ce4829740 1 -- start start 2024-03-23T11:32:11.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.441+0000 7f7ce4829740 1 Processor -- start 2024-03-23T11:32:11.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.442+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8ab65a0 0x55c3c8afeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.442+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8b00520 0x55c3c8afefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.442+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8adb100 0x55c3c8b036a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.442+0000 7f7ce4829740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c89bd8c0 con 0x55c3c8b00520 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.442+0000 7f7ce4829740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c89bd560 con 0x55c3c8ab65a0 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.442+0000 7f7ce4829740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c3c8783f20 con 0x55c3c8adb100 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.442+0000 7f7ce4829740 10 client.0 _ll_register_callbacks cb 0x55c3c8a15800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.443+0000 7f7cda000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8b00520 0x55c3c8afefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.443+0000 7f7cda000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8b00520 0x55c3c8afefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42372/0 (socket says 172.21.15.62:42372) 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.443+0000 7f7cda000640 1 -- 172.21.15.62:0/1597911448 learned_addr learned my addr 172.21.15.62:0/1597911448 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.443+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8ab65a0 0x55c3c8afeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.443+0000 7f7cda000640 1 -- 172.21.15.62:0/1597911448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8adb100 msgr2=0x55c3c8b036a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.443+0000 7f7cdb400640 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8adb100 0x55c3c8b036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.443+0000 7f7cda000640 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8adb100 0x55c3c8b036a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.443+0000 7f7cda000640 1 -- 172.21.15.62:0/1597911448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8ab65a0 msgr2=0x55c3c8afeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.443+0000 7f7cda000640 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8ab65a0 0x55c3c8afeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.443+0000 7f7cda000640 1 -- 172.21.15.62:0/1597911448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd4008b30 con 0x55c3c8b00520 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.443+0000 7f7cda000640 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8b00520 0x55c3c8afefe0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0ac960 tx=0x7f7cc8001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73bca4fefd223db server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.444+0000 7f7cbbe00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.444+0000 7f7cbbe00640 1 -- 172.21.15.62:0/1597911448 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8016b50 con 0x55c3c8b00520 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.444+0000 7f7cbbe00640 1 -- 172.21.15.62:0/1597911448 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7cc8005e30 con 0x55c3c8b00520 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.444+0000 7f7ce4829740 1 -- 172.21.15.62:0/1597911448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c3c8a46280 con 0x55c3c8b00520 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.444+0000 7f7cbbe00640 1 -- 172.21.15.62:0/1597911448 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cc8016cb0 con 0x55c3c8b00520 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.444+0000 7f7ce4829740 10 client.9363 fetch_fsmap learned FSMap version 101 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.444+0000 7f7ce4829740 1 -- 172.21.15.62:0/1597911448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=2+}) v3 -- 0x55c3c8a1a810 con 0x55c3c8b00520 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.444+0000 7f7cbbe00640 1 -- 172.21.15.62:0/1597911448 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c3c8a46280 con 0x55c3c8b00520 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.445+0000 7f7cbbe00640 1 -- 172.21.15.62:0/1597911448 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 101) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7cc801f070 con 0x55c3c8b00520 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.445+0000 7f7ce4829740 10 client.9363 fetch_fsmap finished waiting for FSMap version 101 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.445+0000 7f7ce4829740 10 client.9363 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.445+0000 7f7ce4829740 20 client.9363 populate_metadata read hostname 'smithi062' 2024-03-23T11:32:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.445+0000 7f7ce4829740 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55c3c8b020b0 0x55c3c8b02490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.445+0000 7f7ce4829740 4 client.9363 mds_command: new command op to 9000 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:32:11.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.445+0000 7f7ce4829740 1 -- 172.21.15.62:0/1597911448 --> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c3c8a0e9d0 con 0x55c3c8b020b0 2024-03-23T11:32:11.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.445+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55c3c8b020b0 0x55c3c8b02490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.445+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55c3c8b020b0 0x55c3c8b02490 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4a5eaf87f37970 server_cookie=7abf2a51cdebf7db in_seq=0 out_seq=0 2024-03-23T11:32:11.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.445+0000 7f7cbbe00640 10 client.9363 ms_handle_connect on v2:172.21.15.62:6834/4129977589 2024-03-23T11:32:11.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.446+0000 7f7cbbe00640 1 -- 172.21.15.62:0/1597911448 <== mds.0 v2:172.21.15.62:6834/4129977589 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55c3c8a0e9d0 con 0x55c3c8b020b0 2024-03-23T11:32:11.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.446+0000 7f7cbbe00640 10 client.9363 handle_command_reply: tid=0 2024-03-23T11:32:11.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.446+0000 7f7ce4829740 1 client.9363 shutdown 2024-03-23T11:32:11.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7ce4829740 1 -- 172.21.15.62:0/1597911448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8b00520 msgr2=0x55c3c8afefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7ce4829740 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8b00520 0x55c3c8afefe0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0ac960 tx=0x7f7cc8001080 comp rx=0 tx=0).stop 2024-03-23T11:32:11.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7ce4829740 1 -- 172.21.15.62:0/1597911448 shutdown_connections 2024-03-23T11:32:11.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7ce4829740 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55c3c8b020b0 0x55c3c8b02490 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7ce4829740 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8b00520 0x55c3c8afefe0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7ce4829740 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8adb100 0x55c3c8b036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7ce4829740 1 --2- 172.21.15.62:0/1597911448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8ab65a0 0x55c3c8afeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7cbbe00640 0 client.9363 ms_handle_reset on v2:172.21.15.62:6834/4129977589 2024-03-23T11:32:11.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7ce4829740 1 -- 172.21.15.62:0/1597911448 >> 172.21.15.62:0/1597911448 conn(0x55c3c8ab2a40 msgr2=0x55c3c8ab6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:11.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7ce4829740 1 -- 172.21.15.62:0/1597911448 shutdown_connections 2024-03-23T11:32:11.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7ce4829740 1 -- 172.21.15.62:0/1597911448 wait complete. 2024-03-23T11:32:11.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.447+0000 7f7ce4829740 20 client.9363 trim_cache size 0 max 16384 2024-03-23T11:32:11.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.464+0000 7f7ce4829740 1 Processor -- start 2024-03-23T11:32:11.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.464+0000 7f7ce4829740 1 -- start start 2024-03-23T11:32:11.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8ab65a0 0x55c3c8aae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8adb100 0x55c3c8aae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8b00520 0x55c3c8b02490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7ce4829740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c8afccd0 con 0x55c3c8adb100 2024-03-23T11:32:11.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7ce4829740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c8ae1a40 con 0x55c3c8b00520 2024-03-23T11:32:11.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7ce4829740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c3c8b03ca0 con 0x55c3c8ab65a0 2024-03-23T11:32:11.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7cda000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8adb100 0x55c3c8aae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7cda000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8adb100 0x55c3c8aae620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42380/0 (socket says 172.21.15.62:42380) 2024-03-23T11:32:11.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7cda000640 1 -- 172.21.15.62:0/3757166018 learned_addr learned my addr 172.21.15.62:0/3757166018 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:11.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7cda000640 1 -- 172.21.15.62:0/3757166018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8ab65a0 msgr2=0x55c3c8aae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:11.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7cda000640 1 --2- 172.21.15.62:0/3757166018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8ab65a0 0x55c3c8aae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7cda000640 1 -- 172.21.15.62:0/3757166018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8b00520 msgr2=0x55c3c8b02490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:11.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7cda000640 1 --2- 172.21.15.62:0/3757166018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8b00520 0x55c3c8b02490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.465+0000 7f7cda000640 1 -- 172.21.15.62:0/3757166018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3c89bcad0 con 0x55c3c8adb100 2024-03-23T11:32:11.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.466+0000 7f7cda000640 1 --2- 172.21.15.62:0/3757166018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8adb100 0x55c3c8aae620 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80011e0 tx=0x7f7cc801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d302f6d1834a7b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:11.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.466+0000 7f7cc7e00640 1 -- 172.21.15.62:0/3757166018 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8005e30 con 0x55c3c8adb100 2024-03-23T11:32:11.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.466+0000 7f7cc7e00640 1 -- 172.21.15.62:0/3757166018 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc801c040 con 0x55c3c8adb100 2024-03-23T11:32:11.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.466+0000 7f7cc7e00640 1 -- 172.21.15.62:0/3757166018 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8014680 con 0x55c3c8adb100 2024-03-23T11:32:11.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.466+0000 7f7ce4829740 1 -- 172.21.15.62:0/3757166018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8adb100 msgr2=0x55c3c8aae620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.466+0000 7f7ce4829740 1 --2- 172.21.15.62:0/3757166018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8adb100 0x55c3c8aae620 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80011e0 tx=0x7f7cc801a000 comp rx=0 tx=0).stop 2024-03-23T11:32:11.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.467+0000 7f7ce4829740 1 -- 172.21.15.62:0/3757166018 shutdown_connections 2024-03-23T11:32:11.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.467+0000 7f7ce4829740 1 --2- 172.21.15.62:0/3757166018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8adb100 0x55c3c8aae620 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.467+0000 7f7ce4829740 1 --2- 172.21.15.62:0/3757166018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8b00520 0x55c3c8b02490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.467+0000 7f7ce4829740 1 --2- 172.21.15.62:0/3757166018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8ab65a0 0x55c3c8aae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.467+0000 7f7ce4829740 1 -- 172.21.15.62:0/3757166018 >> 172.21.15.62:0/3757166018 conn(0x55c3c8ab2a40 msgr2=0x55c3c8ab6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:11.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.467+0000 7f7ce4829740 1 -- 172.21.15.62:0/3757166018 shutdown_connections 2024-03-23T11:32:11.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.467+0000 7f7ce4829740 1 -- 172.21.15.62:0/3757166018 wait complete. 2024-03-23T11:32:11.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.467+0000 7f7ce4829740 1 -- start start 2024-03-23T11:32:11.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.468+0000 7f7ce4829740 1 Processor -- start 2024-03-23T11:32:11.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab65a0 0x55c3c8afc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8adb100 0x55c3c8b63a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7ce4829740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8b00520 0x55c3c8ae3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7ce4829740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c8afcb60 con 0x55c3c8ab65a0 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7ce4829740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c8b63fb0 con 0x55c3c8adb100 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7ce4829740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c3c8ae4780 con 0x55c3c8b00520 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7ce4829740 10 client.0 _ll_register_callbacks cb 0x55c3c8b05e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7cdaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab65a0 0x55c3c8afc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7cdaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab65a0 0x55c3c8afc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42390/0 (socket says 172.21.15.62:42390) 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7cdaa00640 1 -- 172.21.15.62:0/2704672907 learned_addr learned my addr 172.21.15.62:0/2704672907 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7cdaa00640 1 -- 172.21.15.62:0/2704672907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8b00520 msgr2=0x55c3c8ae3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7cdb400640 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8b00520 0x55c3c8ae3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7cda000640 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8adb100 0x55c3c8b63a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8b00520 0x55c3c8ae3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7cdaa00640 1 -- 172.21.15.62:0/2704672907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8adb100 msgr2=0x55c3c8b63a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8adb100 0x55c3c8b63a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7cdaa00640 1 -- 172.21.15.62:0/2704672907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3c8a1a810 con 0x55c3c8ab65a0 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.469+0000 7f7cdb400640 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8b00520 0x55c3c8ae3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.470+0000 7f7cdaa00640 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab65a0 0x55c3c8afc620 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0764a0 tx=0x7f7ccc0091c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191cab48edf03983 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.470+0000 7f7cbbe00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.470+0000 7f7cbbe00640 1 -- 172.21.15.62:0/2704672907 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc082390 con 0x55c3c8ab65a0 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.470+0000 7f7cbbe00640 1 -- 172.21.15.62:0/2704672907 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7ccc078340 con 0x55c3c8ab65a0 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.470+0000 7f7ce4829740 1 -- 172.21.15.62:0/2704672907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cd400e9b0 con 0x55c3c8ab65a0 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.470+0000 7f7cbbe00640 1 -- 172.21.15.62:0/2704672907 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ccc0766b0 con 0x55c3c8ab65a0 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.470+0000 7f7ce4829740 10 client.9369 fetch_fsmap learned FSMap version 101 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.470+0000 7f7ce4829740 1 -- 172.21.15.62:0/2704672907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=2+}) v3 -- 0x55c3c89e1fe0 con 0x55c3c8ab65a0 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.470+0000 7f7cbbe00640 1 -- 172.21.15.62:0/2704672907 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd400e9b0 con 0x55c3c8ab65a0 2024-03-23T11:32:11.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.471+0000 7f7cbbe00640 1 -- 172.21.15.62:0/2704672907 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 101) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7ccc079490 con 0x55c3c8ab65a0 2024-03-23T11:32:11.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.471+0000 7f7ce4829740 10 client.9369 fetch_fsmap finished waiting for FSMap version 101 2024-03-23T11:32:11.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.471+0000 7f7ce4829740 10 client.9369 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:32:11.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.471+0000 7f7ce4829740 20 client.9369 populate_metadata read hostname 'smithi062' 2024-03-23T11:32:11.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.471+0000 7f7ce4829740 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55c3c8b00cd0 0x55c3c8b010b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.471+0000 7f7ce4829740 4 client.9369 mds_command: new command op to 9000 tid=0[{"prefix": "status"}] 2024-03-23T11:32:11.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.471+0000 7f7ce4829740 1 -- 172.21.15.62:0/2704672907 --> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c3c87103c0 con 0x55c3c8b00cd0 2024-03-23T11:32:11.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.471+0000 7f7cda000640 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55c3c8b00cd0 0x55c3c8b010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.471+0000 7f7cbbe00640 10 client.9369 ms_handle_connect on v2:172.21.15.62:6834/4129977589 2024-03-23T11:32:11.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.471+0000 7f7cda000640 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55c3c8b00cd0 0x55c3c8b010b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46c8fcdbcf7eb927 server_cookie=4ba2aa6d9a3ca055 in_seq=0 out_seq=0 2024-03-23T11:32:11.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7cbbe00640 1 -- 172.21.15.62:0/2704672907 <== mds.0 v2:172.21.15.62:6834/4129977589 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55c3c87103c0 con 0x55c3c8b00cd0 2024-03-23T11:32:11.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7cbbe00640 10 client.9369 handle_command_reply: tid=0 2024-03-23T11:32:11.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7ce4829740 1 client.9369 shutdown 2024-03-23T11:32:11.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7ce4829740 1 -- 172.21.15.62:0/2704672907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab65a0 msgr2=0x55c3c8afc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7ce4829740 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab65a0 0x55c3c8afc620 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0764a0 tx=0x7f7ccc0091c0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7ce4829740 1 -- 172.21.15.62:0/2704672907 shutdown_connections 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7ce4829740 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55c3c8b00cd0 0x55c3c8b010b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7ce4829740 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c3c8b00520 0x55c3c8ae3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7ce4829740 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3c8adb100 0x55c3c8b63a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7ce4829740 1 --2- 172.21.15.62:0/2704672907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c3c8ab65a0 0x55c3c8afc620 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7cbbe00640 0 client.9369 ms_handle_reset on v2:172.21.15.62:6834/4129977589 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7ce4829740 1 -- 172.21.15.62:0/2704672907 >> 172.21.15.62:0/2704672907 conn(0x55c3c8ab2a40 msgr2=0x55c3c8b168d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7ce4829740 1 -- 172.21.15.62:0/2704672907 shutdown_connections 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.472+0000 7f7ce4829740 1 -- 172.21.15.62:0/2704672907 wait complete. 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7ce4829740 20 client.9369 trim_cache size 0 max 16384 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:11.465 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:32:11.466 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:32:11.466 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 9000, 2024-03-23T11:32:11.466 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:32:11.466 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:32:11.466 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:32:11.466 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 5.0221167590000002, 2024-03-23T11:32:11.466 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 101, 2024-03-23T11:32:11.466 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 121, 2024-03-23T11:32:11.466 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 121, 2024-03-23T11:32:11.466 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 52.768290610000001 2024-03-23T11:32:11.466 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 -- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cbc06a7a0 msgr2=0x7f7cbc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cbc06a7a0 0x7f7cbc06cc60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8007fd0 tx=0x7f7cc8007920 comp rx=0 tx=0).stop 2024-03-23T11:32:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 -- 172.21.15.62:0/3812728428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cdc069c60 msgr2=0x7f7cdc0b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cdc069c60 0x7f7cdc0b3130 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc000c00 tx=0x7f7ccc004730 comp rx=0 tx=0).stop 2024-03-23T11:32:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 -- 172.21.15.62:0/3812728428 shutdown_connections 2024-03-23T11:32:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7cbc06a7a0 0x7f7cbc06cc60 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cdc143450 0x7f7cdc0c3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7cdc0caff0 0x7f7cdc1f14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 --2- 172.21.15.62:0/3812728428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cdc069c60 0x7f7cdc0b3130 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 -- 172.21.15.62:0/3812728428 >> 172.21.15.62:0/3812728428 conn(0x7f7cdc0b0090 msgr2=0x7f7cdc157370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:11.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 -- 172.21.15.62:0/3812728428 shutdown_connections 2024-03-23T11:32:11.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.473+0000 7f7cdbe00640 1 -- 172.21.15.62:0/3812728428 wait complete. 2024-03-23T11:32:11.480 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:32:11.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.708+0000 7f4681200640 1 Processor -- start 2024-03-23T11:32:11.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.708+0000 7f4681200640 1 -- start start 2024-03-23T11:32:11.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f4681200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f467c143dd0 0x7f467c15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f4681200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f467c0b2f30 0x7f467c070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f4681200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c0cd140 con 0x7f467c070090 2024-03-23T11:32:11.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f4681200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f467c070a60 con 0x7f467c143dd0 2024-03-23T11:32:11.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f4681200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f467c0cd3e0 con 0x7f467c0b2f30 2024-03-23T11:32:11.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f467b400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f467c143dd0 0x7f467c15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f4673e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f467c070090 0x7f467c0b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37404/0 (socket says 172.21.15.62:37404) 2024-03-23T11:32:11.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f467b400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f467c143dd0 0x7f467c15abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42392/0 (socket says 172.21.15.62:42392) 2024-03-23T11:32:11.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f4673e00640 1 -- 172.21.15.62:0/885078524 learned_addr learned my addr 172.21.15.62:0/885078524 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:11.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f467b400640 1 -- 172.21.15.62:0/885078524 >> v2:172.21.15.120:3300/0 conn(0x7f467c0b2f30 msgr2=0x7f467c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:11.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f467b400640 1 --2- 172.21.15.62:0/885078524 >> v2:172.21.15.120:3300/0 conn(0x7f467c0b2f30 0x7f467c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f467b400640 1 -- 172.21.15.62:0/885078524 >> v1:172.21.15.62:6789/0 conn(0x7f467c070090 legacy=0x7f467c0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f467b400640 1 -- 172.21.15.62:0/885078524 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c0cd6c0 con 0x7f467c143dd0 2024-03-23T11:32:11.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.710+0000 7f467aa00640 1 -- 172.21.15.62:0/885078524 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993672499 0 0) 0x7f467c0cd140 con 0x7f467c070090 2024-03-23T11:32:11.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.712+0000 7f467b400640 1 --2- 172.21.15.62:0/885078524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f467c143dd0 0x7f467c15abe0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f467c0cbe50 tx=0x7f466c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb45b64e2e746003 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:11.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.712+0000 7f467aa00640 1 -- 172.21.15.62:0/885078524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c031830 con 0x7f467c143dd0 2024-03-23T11:32:11.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.712+0000 7f467aa00640 1 -- 172.21.15.62:0/885078524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c031990 con 0x7f467c143dd0 2024-03-23T11:32:11.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.712+0000 7f467aa00640 1 -- 172.21.15.62:0/885078524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c031d20 con 0x7f467c143dd0 2024-03-23T11:32:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.712+0000 7f4681200640 1 -- 172.21.15.62:0/885078524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f467c143dd0 msgr2=0x7f467c15abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.712+0000 7f4681200640 1 --2- 172.21.15.62:0/885078524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f467c143dd0 0x7f467c15abe0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f467c0cbe50 tx=0x7f466c02b8d0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.713+0000 7f4681200640 1 -- 172.21.15.62:0/885078524 shutdown_connections 2024-03-23T11:32:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.713+0000 7f4681200640 1 --2- 172.21.15.62:0/885078524 >> v2:172.21.15.120:3300/0 conn(0x7f467c0b2f30 0x7f467c070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.713+0000 7f4681200640 1 --2- 172.21.15.62:0/885078524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f467c143dd0 0x7f467c15abe0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.713+0000 7f4681200640 1 -- 172.21.15.62:0/885078524 >> 172.21.15.62:0/885078524 conn(0x7f467c066050 msgr2=0x7f467c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.713+0000 7f4681200640 1 -- 172.21.15.62:0/885078524 shutdown_connections 2024-03-23T11:32:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.713+0000 7f4681200640 1 -- 172.21.15.62:0/885078524 wait complete. 2024-03-23T11:32:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.715+0000 7f4681200640 1 Processor -- start 2024-03-23T11:32:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.715+0000 7f4681200640 1 -- start start 2024-03-23T11:32:11.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4681200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f467c070090 0x7f467c0c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4681200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f467c0b2f30 0x7f467c0b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4681200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c143dd0 0x7f467c0b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4681200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0c4120 con 0x7f467c0b2f30 2024-03-23T11:32:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4681200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0c34a0 con 0x7f467c070090 2024-03-23T11:32:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4681200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0c3780 con 0x7f467c143dd0 2024-03-23T11:32:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4673e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f467c0b2f30 0x7f467c0b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4673e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f467c0b2f30 0x7f467c0b17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42400/0 (socket says 172.21.15.62:42400) 2024-03-23T11:32:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4673e00640 1 -- 172.21.15.62:0/1634863243 learned_addr learned my addr 172.21.15.62:0/1634863243 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:11.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4673e00640 1 -- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c143dd0 msgr2=0x7f467c0b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4673e00640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c143dd0 0x7f467c0b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4673e00640 1 -- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f467c070090 msgr2=0x7f467c0c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4673e00640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f467c070090 0x7f467c0c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.716+0000 7f4673e00640 1 -- 172.21.15.62:0/1634863243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c009400 con 0x7f467c0b2f30 2024-03-23T11:32:11.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.717+0000 7f4673e00640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f467c0b2f30 0x7f467c0b17e0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f466400c4d0 tx=0x7f466400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:11.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.717+0000 7f4673400640 1 -- 172.21.15.62:0/1634863243 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46640074e0 con 0x7f467c0b2f30 2024-03-23T11:32:11.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.717+0000 7f4673400640 1 -- 172.21.15.62:0/1634863243 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4664007660 con 0x7f467c0b2f30 2024-03-23T11:32:11.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.717+0000 7f4673400640 1 -- 172.21.15.62:0/1634863243 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4664004a20 con 0x7f467c0b2f30 2024-03-23T11:32:11.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.717+0000 7f4681200640 1 -- 172.21.15.62:0/1634863243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467c0c3a60 con 0x7f467c0b2f30 2024-03-23T11:32:11.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.717+0000 7f4681200640 1 -- 172.21.15.62:0/1634863243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467c0c7fa0 con 0x7f467c0b2f30 2024-03-23T11:32:11.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.718+0000 7f4681200640 1 -- 172.21.15.62:0/1634863243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467c0c4260 con 0x7f467c0b2f30 2024-03-23T11:32:11.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.719+0000 7f4673400640 1 -- 172.21.15.62:0/1634863243 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4664004b80 con 0x7f467c0b2f30 2024-03-23T11:32:11.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.720+0000 7f4673400640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f465806a7a0 0x7f465806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:11.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.720+0000 7f4673400640 1 -- 172.21.15.62:0/1634863243 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4664087fe0 con 0x7f467c0b2f30 2024-03-23T11:32:11.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.721+0000 7f467b400640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f465806a7a0 0x7f465806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:11.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.722+0000 7f467b400640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f465806a7a0 0x7f465806cc60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f466c036000 tx=0x7f466c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:11.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.724+0000 7f4673400640 1 -- 172.21.15.62:0/1634863243 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f467c0c4260 con 0x7f467c0b2f30 2024-03-23T11:32:11.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.966+0000 7f4681200640 1 -- 172.21.15.62:0/1634863243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f467c0c4260 con 0x7f467c0b2f30 2024-03-23T11:32:11.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.967+0000 7f4673400640 1 -- 172.21.15.62:0/1634863243 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v101) v1 ==== 53+0+73 (secure 0 0 0) 0x7f467c0c4260 con 0x7f467c0b2f30 2024-03-23T11:32:11.957 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:32:11.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 -- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f465806a7a0 msgr2=0x7f465806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f465806a7a0 0x7f465806cc60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f466c036000 tx=0x7f466c030000 comp rx=0 tx=0).stop 2024-03-23T11:32:11.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 -- 172.21.15.62:0/1634863243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f467c0b2f30 msgr2=0x7f467c0b17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:11.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f467c0b2f30 0x7f467c0b17e0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f466400c4d0 tx=0x7f466400c990 comp rx=0 tx=0).stop 2024-03-23T11:32:11.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 -- 172.21.15.62:0/1634863243 shutdown_connections 2024-03-23T11:32:11.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f465806a7a0 0x7f465806cc60 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f467c0b2f30 0x7f467c0b17e0 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f467c143dd0 0x7f467c0b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 --2- 172.21.15.62:0/1634863243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f467c070090 0x7f467c0c7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:11.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 -- 172.21.15.62:0/1634863243 >> 172.21.15.62:0/1634863243 conn(0x7f467c066050 msgr2=0x7f467c0ca5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:11.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 -- 172.21.15.62:0/1634863243 shutdown_connections 2024-03-23T11:32:11.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:11.970+0000 7f4681200640 1 -- 172.21.15.62:0/1634863243 wait complete. 2024-03-23T11:32:11.979 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:32:11.979 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:32:11.979 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:32:11.979 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:32:11.979 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:11.979 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:32:11.979 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:32:11.979 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:32:11.979 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:32:11.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:11.980 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:32:12.032 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:32:12.033 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 41825sec preferred_lft 41825sec 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:12.034 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:32:12.035 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:12.035 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:32:12.035 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:32:12.035 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:32:12.035 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:12.035 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:32:12.035 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:32:12.035 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:32:12.035 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:12.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:12.036 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:32:12.126 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:32:12.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:12.131 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:32:12.184 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:32:12.184 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:32:12.185 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:32:12.185 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:32:12.185 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:32:12.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:12.186 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:32:12.186 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:32:12.186 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:32:12.186 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:32:12.186 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:32:12.186 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:32:12.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:12.375 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:32:12.426 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:32:12.427 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:32:12.427 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:32:12.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:12.428 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:32:12.527 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:32:12.527 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:32:12.527 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:12.585 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:32:12.586 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:32:12.586 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:12.638 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:32:12.742 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:32:12.742 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:32:12.839 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:32:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:32:12.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:12.844 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:32:12.899 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:32:12.900 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:32:12.904 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:32:12.981 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:32:12.981 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:32:13.065 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:32:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:32:13.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:13.069 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:32:13.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:32:13.106+0000 7f712695c480 -1 init, newargv = 0x561a03df87d0 newargc=15 2024-03-23T11:32:13.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[80332]: starting ceph client 2024-03-23T11:32:13.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[80332]: starting fuse 2024-03-23T11:32:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:32:13.113 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:32:13.114 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:32:13.115 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:13.315 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.80332.asok 2024-03-23T11:32:13.495 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.80332.asok 2024-03-23T11:32:13.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:13.496 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.80332.asok status 2024-03-23T11:32:13.745 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:13.745 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:32:13.745 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:32:13.745 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:32:13.745 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:32:13.745 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:32:13.745 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:32:13.745 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "80332", 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 9384, 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 9384 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:32:13.746 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 3981555822 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 3981555822 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.9384 192.168.0.1:0/3981555822", 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/3981555822", 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 101, 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 121, 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:32:13.747 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:32:13.748 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:32:13.748 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:13.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:13.760 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:32:13.760 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:13.817 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:32:13.817 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:13.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:13.818 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:13.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:13.914 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-23T11:32:13.992 INFO:teuthology.orchestra.run.smithi062.stdout:./ 2024-03-23T11:32:13.992 INFO:teuthology.orchestra.run.smithi062.stdout:./parent 2024-03-23T11:32:13.992 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/flushed 2024-03-23T11:32:13.992 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/flushed/charlie 2024-03-23T11:32:13.993 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/flushed/alpha 2024-03-23T11:32:13.993 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/unflushed 2024-03-23T11:32:13.993 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/unflushed/jfile 2024-03-23T11:32:13.994 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:13.994 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:13.994 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:13.994 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:13.994 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-23T11:32:13.994 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:14.130 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:32:14.310 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:14.311 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:14.311 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:14.311 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:14.311 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-03-23T11:32:14.311 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:14.512 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627776 2024-03-23T11:32:14.692 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:14.692 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:14.692 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:14.692 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:14.692 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-03-23T11:32:14.692 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:14.873 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627777 2024-03-23T11:32:15.053 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:15.053 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:15.053 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:15.053 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:15.053 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-03-23T11:32:15.053 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:15.246 INFO:teuthology.orchestra.run.smithi062.stdout:1099511628280 2024-03-23T11:32:15.426 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:15.426 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:15.426 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:15.426 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:15.426 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-03-23T11:32:15.426 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:15.619 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627778 2024-03-23T11:32:15.799 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:15.799 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:15.799 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:15.799 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:15.799 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-03-23T11:32:15.799 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:15.995 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627779 2024-03-23T11:32:16.175 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:16.175 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:16.175 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:16.176 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:16.176 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-03-23T11:32:16.176 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:16.373 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627780 2024-03-23T11:32:16.552 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:32:16.552 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:32:16.653 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:32:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.885+0000 7f9e59a00640 1 Processor -- start 2024-03-23T11:32:16.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.885+0000 7f9e59a00640 1 -- start start 2024-03-23T11:32:16.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.887+0000 7f9e59a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54143dd0 0x7f9e5415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.887+0000 7f9e59a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e540cd140 con 0x7f9e540b2f30 2024-03-23T11:32:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.887+0000 7f9e59a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e540cd3e0 con 0x7f9e54070090 2024-03-23T11:32:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.887+0000 7f9e59a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e540cd6c0 con 0x7f9e54143dd0 2024-03-23T11:32:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.887+0000 7f9e59000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9e540b2f30 0x7f9e540ce480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37412/0 (socket says 172.21.15.62:37412) 2024-03-23T11:32:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.887+0000 7f9e59000640 1 -- 172.21.15.62:0/3560989 learned_addr learned my addr 172.21.15.62:0/3560989 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:16.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.888+0000 7f9e53e00640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54143dd0 0x7f9e5415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:16.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.888+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127389410 0 0) 0x7f9e540cd140 con 0x7f9e540b2f30 2024-03-23T11:32:16.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.888+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e38003520 con 0x7f9e540b2f30 2024-03-23T11:32:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.888+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1264339392 0 0) 0x7f9e38003520 con 0x7f9e540b2f30 2024-03-23T11:32:16.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.888+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54143dd0 msgr2=0x7f9e5415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.888+0000 7f9e53400640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54143dd0 0x7f9e5415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.888+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 >> v1:172.21.15.120:6789/0 conn(0x7f9e54070090 legacy=0x7f9e540b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e540cd9a0 con 0x7f9e540b2f30 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9e4c0019f0 con 0x7f9e540b2f30 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 >> v1:172.21.15.62:6789/0 conn(0x7f9e540b2f30 legacy=0x7f9e540ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e53400640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e380039a0 0x7f9e38003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e53400640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 0x7f9e38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e53400640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54143dd0 0x7f9e38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e540cd6c0 con 0x7f9e380047c0 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3800ba30 con 0x7f9e380039a0 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3800bbf0 con 0x7f9e54143dd0 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e4be00640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 0x7f9e38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:16.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e53e00640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e380039a0 0x7f9e38003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:16.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e59000640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54143dd0 0x7f9e38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:16.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.889+0000 7f9e4be00640 1 -- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54143dd0 msgr2=0x7f9e38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:16.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.890+0000 7f9e4be00640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54143dd0 0x7f9e38005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:16.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.890+0000 7f9e4be00640 1 -- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e380039a0 msgr2=0x7f9e38003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:16.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.890+0000 7f9e4be00640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e380039a0 0x7f9e38003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:16.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.890+0000 7f9e4be00640 1 -- 172.21.15.62:0/3560989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e38003520 con 0x7f9e380047c0 2024-03-23T11:32:16.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.890+0000 7f9e59000640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54143dd0 0x7f9e38005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.891+0000 7f9e4be00640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 0x7f9e38004c40 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c001720 tx=0x7f9e380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d39721437fc8b97 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.891+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c0052d0 con 0x7f9e380047c0 2024-03-23T11:32:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.891+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3c02e050 con 0x7f9e380047c0 2024-03-23T11:32:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.891+0000 7f9e53400640 1 -- 172.21.15.62:0/3560989 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c0035f0 con 0x7f9e380047c0 2024-03-23T11:32:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.892+0000 7f9e59a00640 1 -- 172.21.15.62:0/3560989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 msgr2=0x7f9e38004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:16.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.892+0000 7f9e59a00640 1 --2- 172.21.15.62:0/3560989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 0x7f9e38004c40 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c001720 tx=0x7f9e380057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:16.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.892+0000 7f9e59000640 1 -- 172.21.15.62:0/3560989 reap_dead start 2024-03-23T11:32:16.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.892+0000 7f9e59a00640 1 -- 172.21.15.62:0/3560989 shutdown_connections 2024-03-23T11:32:16.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.892+0000 7f9e59a00640 1 -- 172.21.15.62:0/3560989 >> 172.21.15.62:0/3560989 conn(0x7f9e54066050 msgr2=0x7f9e54069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:16.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.892+0000 7f9e59a00640 1 -- 172.21.15.62:0/3560989 shutdown_connections 2024-03-23T11:32:16.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.892+0000 7f9e59a00640 1 -- 172.21.15.62:0/3560989 wait complete. 2024-03-23T11:32:16.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.894+0000 7f9e59a00640 1 Processor -- start 2024-03-23T11:32:16.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.894+0000 7f9e59a00640 1 -- start start 2024-03-23T11:32:16.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e59a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 0x7f9e5413c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:16.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e59a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54159120 0x7f9e54158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:16.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e59a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5406d0e0 0x7f9e5406d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e59a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5413ca60 con 0x7f9e380047c0 2024-03-23T11:32:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e59a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5406daa0 con 0x7f9e5406d0e0 2024-03-23T11:32:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e59a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5415a480 con 0x7f9e54159120 2024-03-23T11:32:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e53e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 0x7f9e5413c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e53e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 0x7f9e5413c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42414/0 (socket says 172.21.15.62:42414) 2024-03-23T11:32:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e53e00640 1 -- 172.21.15.62:0/1746427763 learned_addr learned my addr 172.21.15.62:0/1746427763 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e53e00640 1 -- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54159120 msgr2=0x7f9e54158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e53e00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54159120 0x7f9e54158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e53e00640 1 -- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5406d0e0 msgr2=0x7f9e5406d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e53e00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5406d0e0 0x7f9e5406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.895+0000 7f9e53e00640 1 -- 172.21.15.62:0/1746427763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54159870 con 0x7f9e380047c0 2024-03-23T11:32:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.896+0000 7f9e53e00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 0x7f9e5413c520 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f9e44011630 tx=0x7f9e44011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.896+0000 7f9e50c00640 1 -- 172.21.15.62:0/1746427763 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4400f920 con 0x7f9e380047c0 2024-03-23T11:32:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.896+0000 7f9e50c00640 1 -- 172.21.15.62:0/1746427763 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4400fa80 con 0x7f9e380047c0 2024-03-23T11:32:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.896+0000 7f9e50c00640 1 -- 172.21.15.62:0/1746427763 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4400fdf0 con 0x7f9e380047c0 2024-03-23T11:32:16.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.896+0000 7f9e59a00640 1 -- 172.21.15.62:0/1746427763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e54159b50 con 0x7f9e380047c0 2024-03-23T11:32:16.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.896+0000 7f9e59a00640 1 -- 172.21.15.62:0/1746427763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5415a110 con 0x7f9e380047c0 2024-03-23T11:32:16.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.897+0000 7f9e59a00640 1 -- 172.21.15.62:0/1746427763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e54066bb0 con 0x7f9e380047c0 2024-03-23T11:32:16.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.898+0000 7f9e50c00640 1 -- 172.21.15.62:0/1746427763 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9e44015460 con 0x7f9e380047c0 2024-03-23T11:32:16.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.899+0000 7f9e50c00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e3006a7a0 0x7f9e3006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:16.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.899+0000 7f9e50c00640 1 -- 172.21.15.62:0/1746427763 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9e44089490 con 0x7f9e380047c0 2024-03-23T11:32:16.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.900+0000 7f9e4be00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e3006a7a0 0x7f9e3006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:16.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.901+0000 7f9e4be00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e3006a7a0 0x7f9e3006cc60 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c033000 tx=0x7f9e3c02f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:16.904+0000 7f9e50c00640 1 -- 172.21.15.62:0/1746427763 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e440546f0 con 0x7f9e380047c0 2024-03-23T11:32:17.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.104+0000 7f9e59a00640 1 -- 172.21.15.62:0/1746427763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e540aef60 con 0x7f9e380047c0 2024-03-23T11:32:17.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.107+0000 7f9e50c00640 1 -- 172.21.15.62:0/1746427763 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+16283 (secure 0 0 0) 0x7f9e44057d10 con 0x7f9e380047c0 2024-03-23T11:32:17.097 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:17.097 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":121,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:32:04.365698+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6837/2585024035":"2024-03-24T11:32:04.365655+0000","172.21.15.120:6835/2585024035":"2024-03-24T11:32:04.365655+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:32:17.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 -- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e3006a7a0 msgr2=0x7f9e3006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e3006a7a0 0x7f9e3006cc60 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c033000 tx=0x7f9e3c02f000 comp rx=0 tx=0).stop 2024-03-23T11:32:17.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 -- 172.21.15.62:0/1746427763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 msgr2=0x7f9e5413c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 0x7f9e5413c520 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f9e44011630 tx=0x7f9e44011af0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 -- 172.21.15.62:0/1746427763 shutdown_connections 2024-03-23T11:32:17.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9e3006a7a0 0x7f9e3006cc60 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5406d0e0 0x7f9e5406d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e54159120 0x7f9e54158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 --2- 172.21.15.62:0/1746427763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e380047c0 0x7f9e5413c520 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 -- 172.21.15.62:0/1746427763 >> 172.21.15.62:0/1746427763 conn(0x7f9e54066050 msgr2=0x7f9e54142760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:17.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 -- 172.21.15.62:0/1746427763 shutdown_connections 2024-03-23T11:32:17.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.110+0000 7f9e59a00640 1 -- 172.21.15.62:0/1746427763 wait complete. 2024-03-23T11:32:17.119 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:31:22.127079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:31:23.062387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:32:17.119 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:32:17.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.352+0000 7fe35f000640 1 Processor -- start 2024-03-23T11:32:17.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.353+0000 7fe35f000640 1 -- start start 2024-03-23T11:32:17.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.354+0000 7fe35f000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe3580af340 0x7fe3580990e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.354+0000 7fe35f000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580cc860 con 0x7fe3580ce8d0 2024-03-23T11:32:17.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.354+0000 7fe35f000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580cbfc0 con 0x7fe3580ae880 2024-03-23T11:32:17.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.354+0000 7fe35f000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3580cac10 con 0x7fe3580af340 2024-03-23T11:32:17.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.355+0000 7fe35dc00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe3580ce8d0 0x7fe3580cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58006/0 (socket says 172.21.15.62:58006) 2024-03-23T11:32:17.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.355+0000 7fe35dc00640 1 -- 172.21.15.62:0/2354935708 learned_addr learned my addr 172.21.15.62:0/2354935708 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:17.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.355+0000 7fe35e600640 1 --2- 172.21.15.62:0/2354935708 >> v2:172.21.15.120:3300/0 conn(0x7fe3580af340 0x7fe3580990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.355+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788486166 0 0) 0x7fe3580cc860 con 0x7fe3580ce8d0 2024-03-23T11:32:17.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.355+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe340003520 con 0x7fe3580ce8d0 2024-03-23T11:32:17.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1741535214 0 0) 0x7fe340003520 con 0x7fe3580ce8d0 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 >> v2:172.21.15.120:3300/0 conn(0x7fe3580af340 msgr2=0x7fe3580990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 --2- 172.21.15.62:0/2354935708 >> v2:172.21.15.120:3300/0 conn(0x7fe3580af340 0x7fe3580990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 >> v1:172.21.15.120:6789/0 conn(0x7fe3580ae880 legacy=0x7fe3580989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3580cc2a0 con 0x7fe3580ce8d0 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fe34c001670 con 0x7fe3580ce8d0 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 >> v1:172.21.15.62:6789/0 conn(0x7fe3580ce8d0 legacy=0x7fe3580cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 --2- 172.21.15.62:0/2354935708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe340003970 0x7fe340003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 --2- 172.21.15.62:0/2354935708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3400047c0 0x7fe340004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 --2- 172.21.15.62:0/2354935708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 0x7fe34000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34000bff0 con 0x7fe340005180 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34000c160 con 0x7fe3400047c0 2024-03-23T11:32:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34000c320 con 0x7fe340003970 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe35e600640 1 --2- 172.21.15.62:0/2354935708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 0x7fe34000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe35dc00640 1 --2- 172.21.15.62:0/2354935708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe340003970 0x7fe340003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe35d200640 1 --2- 172.21.15.62:0/2354935708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3400047c0 0x7fe340004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe35e600640 1 -- 172.21.15.62:0/2354935708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe340003970 msgr2=0x7fe340003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.356+0000 7fe35e600640 1 --2- 172.21.15.62:0/2354935708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe340003970 0x7fe340003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.357+0000 7fe35e600640 1 -- 172.21.15.62:0/2354935708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3400047c0 msgr2=0x7fe340004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.357+0000 7fe35e600640 1 --2- 172.21.15.62:0/2354935708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3400047c0 0x7fe340004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.357+0000 7fe35e600640 1 -- 172.21.15.62:0/2354935708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe340003520 con 0x7fe340005180 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.357+0000 7fe35e600640 1 -- 172.21.15.62:0/2354935708 reap_dead start 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.358+0000 7fe35e600640 1 --2- 172.21.15.62:0/2354935708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 0x7fe34000b980 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fe350009fc0 tx=0x7fe3400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d29dfcdfba1cd2c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.358+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35002cc60 con 0x7fe340005180 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.358+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35000f040 con 0x7fe340005180 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.358+0000 7fe357e00640 1 -- 172.21.15.62:0/2354935708 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe350003990 con 0x7fe340005180 2024-03-23T11:32:17.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.358+0000 7fe35f000640 1 -- 172.21.15.62:0/2354935708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 msgr2=0x7fe34000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.358+0000 7fe35f000640 1 --2- 172.21.15.62:0/2354935708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 0x7fe34000b980 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fe350009fc0 tx=0x7fe3400057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.359+0000 7fe35f000640 1 -- 172.21.15.62:0/2354935708 shutdown_connections 2024-03-23T11:32:17.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.359+0000 7fe35f000640 1 --2- 172.21.15.62:0/2354935708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 0x7fe34000b980 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.359+0000 7fe35f000640 1 -- 172.21.15.62:0/2354935708 >> 172.21.15.62:0/2354935708 conn(0x7fe3580b1fb0 msgr2=0x7fe3580b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:17.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.359+0000 7fe35f000640 1 -- 172.21.15.62:0/2354935708 shutdown_connections 2024-03-23T11:32:17.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.359+0000 7fe35f000640 1 -- 172.21.15.62:0/2354935708 wait complete. 2024-03-23T11:32:17.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.360+0000 7fe35f000640 1 Processor -- start 2024-03-23T11:32:17.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.361+0000 7fe35f000640 1 -- start start 2024-03-23T11:32:17.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35f000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 0x7fe35815b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe358157980 0x7fe358157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35f000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3580c6470 0x7fe3580c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35f000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3581f43f0 con 0x7fe340005180 2024-03-23T11:32:17.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35f000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe358158340 con 0x7fe3580c6470 2024-03-23T11:32:17.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3581f3520 con 0x7fe358157980 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 0x7fe35815b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 0x7fe35815b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60318/0 (socket says 172.21.15.62:60318) 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35dc00640 1 -- 172.21.15.62:0/697212408 learned_addr learned my addr 172.21.15.62:0/697212408 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35d200640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe358157980 0x7fe358157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35e600640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3580c6470 0x7fe3580c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35dc00640 1 -- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe358157980 msgr2=0x7fe358157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35dc00640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe358157980 0x7fe358157e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35dc00640 1 -- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3580c6470 msgr2=0x7fe3580c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35dc00640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3580c6470 0x7fe3580c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.362+0000 7fe35dc00640 1 -- 172.21.15.62:0/697212408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3580c6e60 con 0x7fe340005180 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.363+0000 7fe35dc00640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 0x7fe35815b770 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fe34c00e780 tx=0x7fe34c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.363+0000 7fe355600640 1 -- 172.21.15.62:0/697212408 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c00ee90 con 0x7fe340005180 2024-03-23T11:32:17.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.363+0000 7fe355600640 1 -- 172.21.15.62:0/697212408 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34c00c8d0 con 0x7fe340005180 2024-03-23T11:32:17.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.363+0000 7fe355600640 1 -- 172.21.15.62:0/697212408 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c00cc80 con 0x7fe340005180 2024-03-23T11:32:17.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.363+0000 7fe35f000640 1 -- 172.21.15.62:0/697212408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3581f3770 con 0x7fe340005180 2024-03-23T11:32:17.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.363+0000 7fe35f000640 1 -- 172.21.15.62:0/697212408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3581f3d30 con 0x7fe340005180 2024-03-23T11:32:17.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.364+0000 7fe35f000640 1 -- 172.21.15.62:0/697212408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe358073540 con 0x7fe340005180 2024-03-23T11:32:17.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.366+0000 7fe355600640 1 -- 172.21.15.62:0/697212408 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe34c013420 con 0x7fe340005180 2024-03-23T11:32:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.367+0000 7fe355600640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe33c06a7d0 0x7fe33c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.367+0000 7fe355600640 1 -- 172.21.15.62:0/697212408 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe34c088b20 con 0x7fe340005180 2024-03-23T11:32:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.367+0000 7fe35d200640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe33c06a7d0 0x7fe33c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.368+0000 7fe35d200640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe33c06a7d0 0x7fe33c06cc90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe348000c00 tx=0x7fe348009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.372+0000 7fe355600640 1 -- 172.21.15.62:0/697212408 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe34c053e10 con 0x7fe340005180 2024-03-23T11:32:17.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.569+0000 7fe35f000640 1 -- 172.21.15.62:0/697212408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe3580cce90 con 0x7fe340005180 2024-03-23T11:32:17.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.570+0000 7fe355600640 1 -- 172.21.15.62:0/697212408 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe34c057430 con 0x7fe340005180 2024-03-23T11:32:17.560 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:32:17.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.572+0000 7fe35f000640 1 -- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe33c06a7d0 msgr2=0x7fe33c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.572+0000 7fe35f000640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe33c06a7d0 0x7fe33c06cc90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe348000c00 tx=0x7fe348009040 comp rx=0 tx=0).stop 2024-03-23T11:32:17.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.572+0000 7fe35f000640 1 -- 172.21.15.62:0/697212408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 msgr2=0x7fe35815b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.572+0000 7fe35f000640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 0x7fe35815b770 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fe34c00e780 tx=0x7fe34c00ec40 comp rx=0 tx=0).stop 2024-03-23T11:32:17.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.573+0000 7fe35f000640 1 -- 172.21.15.62:0/697212408 shutdown_connections 2024-03-23T11:32:17.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.573+0000 7fe35f000640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe33c06a7d0 0x7fe33c06cc90 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.573+0000 7fe35f000640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3580c6470 0x7fe3580c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.573+0000 7fe35f000640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe358157980 0x7fe358157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.573+0000 7fe35f000640 1 --2- 172.21.15.62:0/697212408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe340005180 0x7fe35815b770 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.573+0000 7fe35f000640 1 -- 172.21.15.62:0/697212408 >> 172.21.15.62:0/697212408 conn(0x7fe3580b1fb0 msgr2=0x7fe358159c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:17.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.573+0000 7fe35f000640 1 -- 172.21.15.62:0/697212408 shutdown_connections 2024-03-23T11:32:17.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.573+0000 7fe35f000640 1 -- 172.21.15.62:0/697212408 wait complete. 2024-03-23T11:32:17.583 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:32:17.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.816+0000 7f5bbbe00640 1 Processor -- start 2024-03-23T11:32:17.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.817+0000 7f5bbbe00640 1 -- start start 2024-03-23T11:32:17.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.818+0000 7f5bbbe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5bbc15fcf0 0x7f5bbc0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.818+0000 7f5bbbe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5bbc0b6dc0 0x7f5bbc1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.818+0000 7f5bbbe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bbc0d3540 con 0x7f5bbc160420 2024-03-23T11:32:17.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.818+0000 7f5bbbe00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bbc160df0 con 0x7f5bbc15fcf0 2024-03-23T11:32:17.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.818+0000 7f5bbbe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bbc0ce510 con 0x7f5bbc0b6dc0 2024-03-23T11:32:17.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.819+0000 7f5bbaa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5bbc15fcf0 0x7f5bbc0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.819+0000 7f5bba000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5bbc160420 0x7f5bbc0d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58016/0 (socket says 172.21.15.62:58016) 2024-03-23T11:32:17.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.819+0000 7f5bba000640 1 -- 172.21.15.62:0/1077108401 learned_addr learned my addr 172.21.15.62:0/1077108401 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:17.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.819+0000 7f5bbb400640 1 --2- 172.21.15.62:0/1077108401 >> v2:172.21.15.120:3300/0 conn(0x7f5bbc0b6dc0 0x7f5bbc1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.819+0000 7f5bbaa00640 1 -- 172.21.15.62:0/1077108401 >> v2:172.21.15.120:3300/0 conn(0x7f5bbc0b6dc0 msgr2=0x7f5bbc1608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:17.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.819+0000 7f5bbaa00640 1 --2- 172.21.15.62:0/1077108401 >> v2:172.21.15.120:3300/0 conn(0x7f5bbc0b6dc0 0x7f5bbc1608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.819+0000 7f5bbaa00640 1 -- 172.21.15.62:0/1077108401 >> v1:172.21.15.62:6789/0 conn(0x7f5bbc160420 legacy=0x7f5bbc0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.819+0000 7f5bbaa00640 1 -- 172.21.15.62:0/1077108401 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc0cdc70 con 0x7f5bbc15fcf0 2024-03-23T11:32:17.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.819+0000 7f5bb9600640 1 -- 172.21.15.62:0/1077108401 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720410256 0 0) 0x7f5bbc0d3540 con 0x7f5bbc160420 2024-03-23T11:32:17.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.819+0000 7f5bbb400640 1 --2- 172.21.15.62:0/1077108401 >> v2:172.21.15.120:3300/0 conn(0x7f5bbc0b6dc0 0x7f5bbc1608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:17.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.821+0000 7f5bbaa00640 1 --2- 172.21.15.62:0/1077108401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bbc15fcf0 0x7f5bbc0d3000 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc08bb20 tx=0x7f5bac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739c695c75228912 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:17.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.821+0000 7f5bb9600640 1 -- 172.21.15.62:0/1077108401 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac02e070 con 0x7f5bbc15fcf0 2024-03-23T11:32:17.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.821+0000 7f5bb9600640 1 -- 172.21.15.62:0/1077108401 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bac0315a0 con 0x7f5bbc15fcf0 2024-03-23T11:32:17.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.821+0000 7f5bb9600640 1 -- 172.21.15.62:0/1077108401 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac031930 con 0x7f5bbc15fcf0 2024-03-23T11:32:17.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.822+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1077108401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bbc15fcf0 msgr2=0x7f5bbc0d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.822+0000 7f5bbbe00640 1 --2- 172.21.15.62:0/1077108401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bbc15fcf0 0x7f5bbc0d3000 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc08bb20 tx=0x7f5bac02b840 comp rx=0 tx=0).stop 2024-03-23T11:32:17.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.822+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1077108401 shutdown_connections 2024-03-23T11:32:17.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.822+0000 7f5bbbe00640 1 --2- 172.21.15.62:0/1077108401 >> v2:172.21.15.120:3300/0 conn(0x7f5bbc0b6dc0 0x7f5bbc1608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.822+0000 7f5bbbe00640 1 --2- 172.21.15.62:0/1077108401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bbc15fcf0 0x7f5bbc0d3000 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.822+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1077108401 >> 172.21.15.62:0/1077108401 conn(0x7f5bbc089f60 msgr2=0x7f5bbc0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:17.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.822+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1077108401 shutdown_connections 2024-03-23T11:32:17.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.822+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1077108401 wait complete. 2024-03-23T11:32:17.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.824+0000 7f5bbbe00640 1 Processor -- start 2024-03-23T11:32:17.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.824+0000 7f5bbbe00640 1 -- start start 2024-03-23T11:32:17.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.825+0000 7f5bbbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bbc0b6dc0 0x7f5bbc15a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.825+0000 7f5bbbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bbc15fcf0 0x7f5bbc0c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.825+0000 7f5bbbe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bbc160420 0x7f5bbc0c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.825+0000 7f5bbbe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc0c7690 con 0x7f5bbc0b6dc0 2024-03-23T11:32:17.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.825+0000 7f5bbbe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc0c6a80 con 0x7f5bbc160420 2024-03-23T11:32:17.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.825+0000 7f5bbbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc0c6d60 con 0x7f5bbc15fcf0 2024-03-23T11:32:17.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.825+0000 7f5bbaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bbc0b6dc0 0x7f5bbc15a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bbaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bbc0b6dc0 0x7f5bbc15a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60330/0 (socket says 172.21.15.62:60330) 2024-03-23T11:32:17.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bbaa00640 1 -- 172.21.15.62:0/1858370951 learned_addr learned my addr 172.21.15.62:0/1858370951 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:17.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bbb400640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bbc160420 0x7f5bbc0c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bba000640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bbc15fcf0 0x7f5bbc0c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bbaa00640 1 -- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bbc15fcf0 msgr2=0x7f5bbc0c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bbaa00640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bbc15fcf0 0x7f5bbc0c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bbaa00640 1 -- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bbc160420 msgr2=0x7f5bbc0c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:17.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bbaa00640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bbc160420 0x7f5bbc0c8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:17.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bbaa00640 1 -- 172.21.15.62:0/1858370951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bac009400 con 0x7f5bbc0b6dc0 2024-03-23T11:32:17.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bba000640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bbc15fcf0 0x7f5bbc0c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:17.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bbb400640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bbc160420 0x7f5bbc0c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:17.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.826+0000 7f5bbaa00640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bbc0b6dc0 0x7f5bbc15a7b0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f5bac034040 tx=0x7f5bac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:17.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.827+0000 7f5bb2a00640 1 -- 172.21.15.62:0/1858370951 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac004230 con 0x7f5bbc0b6dc0 2024-03-23T11:32:17.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.827+0000 7f5bb2a00640 1 -- 172.21.15.62:0/1858370951 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bac004390 con 0x7f5bbc0b6dc0 2024-03-23T11:32:17.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.827+0000 7f5bb2a00640 1 -- 172.21.15.62:0/1858370951 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac032880 con 0x7f5bbc0b6dc0 2024-03-23T11:32:17.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.827+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1858370951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bbc0c7040 con 0x7f5bbc0b6dc0 2024-03-23T11:32:17.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.827+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1858370951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bbc15acf0 con 0x7f5bbc0b6dc0 2024-03-23T11:32:17.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.828+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1858370951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bbc066bb0 con 0x7f5bbc0b6dc0 2024-03-23T11:32:17.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.829+0000 7f5bb2a00640 1 -- 172.21.15.62:0/1858370951 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5bac0329e0 con 0x7f5bbc0b6dc0 2024-03-23T11:32:17.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.830+0000 7f5bb2a00640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5b9806a7a0 0x7f5b9806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:17.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.830+0000 7f5bb2a00640 1 -- 172.21.15.62:0/1858370951 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5bac0aafe0 con 0x7f5bbc0b6dc0 2024-03-23T11:32:17.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.830+0000 7f5bba000640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5b9806a7a0 0x7f5b9806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:17.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.832+0000 7f5bba000640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5b9806a7a0 0x7f5b9806cc60 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0c83d0 tx=0x7f5ba4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:17.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:17.835+0000 7f5bb2a00640 1 -- 172.21.15.62:0/1858370951 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5bac076310 con 0x7f5bbc0b6dc0 2024-03-23T11:32:18.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.034+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1858370951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5bbc0b4d90 con 0x7f5bbc0b6dc0 2024-03-23T11:32:18.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.035+0000 7f5bb2a00640 1 -- 172.21.15.62:0/1858370951 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v121) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5bac079930 con 0x7f5bbc0b6dc0 2024-03-23T11:32:18.024 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:32:18.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.037+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5b9806a7a0 msgr2=0x7f5b9806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.037+0000 7f5bbbe00640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5b9806a7a0 0x7f5b9806cc60 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc0c83d0 tx=0x7f5ba4009040 comp rx=0 tx=0).stop 2024-03-23T11:32:18.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.037+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1858370951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bbc0b6dc0 msgr2=0x7f5bbc15a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.037+0000 7f5bbbe00640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bbc0b6dc0 0x7f5bbc15a7b0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f5bac034040 tx=0x7f5bac0046c0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.038+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1858370951 shutdown_connections 2024-03-23T11:32:18.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.038+0000 7f5bbbe00640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5b9806a7a0 0x7f5b9806cc60 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.038+0000 7f5bbbe00640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5bbc160420 0x7f5bbc0c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.038+0000 7f5bbbe00640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5bbc15fcf0 0x7f5bbc0c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.038+0000 7f5bbbe00640 1 --2- 172.21.15.62:0/1858370951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bbc0b6dc0 0x7f5bbc15a7b0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.038+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1858370951 >> 172.21.15.62:0/1858370951 conn(0x7f5bbc089f60 msgr2=0x7f5bbc06bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:18.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.038+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1858370951 shutdown_connections 2024-03-23T11:32:18.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.038+0000 7f5bbbe00640 1 -- 172.21.15.62:0/1858370951 wait complete. 2024-03-23T11:32:18.047 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:32:18.242 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:32:18.249 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:32:18.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.280+0000 7f346ae00640 1 Processor -- start 2024-03-23T11:32:18.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.280+0000 7f346ae00640 1 -- start start 2024-03-23T11:32:18.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.282+0000 7f346ae00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f346415c9e0 0x7f34641607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.282+0000 7f346ae00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f346415cdc0 0x7f34640d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.282+0000 7f346ae00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346406bfc0 con 0x7f34640cf650 2024-03-23T11:32:18.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.282+0000 7f346ae00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34640ce140 con 0x7f346415cdc0 2024-03-23T11:32:18.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.282+0000 7f346ae00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34640b4370 con 0x7f346415c9e0 2024-03-23T11:32:18.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.282+0000 7f3463400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f346415cdc0 0x7f34640d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:18.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.282+0000 7f3463400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f346415cdc0 0x7f34640d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60342/0 (socket says 172.21.15.62:60342) 2024-03-23T11:32:18.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.282+0000 7f3463400640 1 -- 172.21.15.62:0/3289825942 learned_addr learned my addr 172.21.15.62:0/3289825942 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:18.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.282+0000 7f3463e00640 1 --2- 172.21.15.62:0/3289825942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f346415c9e0 0x7f34641607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:18.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.283+0000 7f3463400640 1 -- 172.21.15.62:0/3289825942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f346415c9e0 msgr2=0x7f34641607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:18.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.283+0000 7f3463400640 1 --2- 172.21.15.62:0/3289825942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f346415c9e0 0x7f34641607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.283+0000 7f3463400640 1 -- 172.21.15.62:0/3289825942 >> v1:172.21.15.120:6789/0 conn(0x7f34640cf650 legacy=0x7f346415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.283+0000 7f3463400640 1 -- 172.21.15.62:0/3289825942 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34640b5260 con 0x7f346415cdc0 2024-03-23T11:32:18.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.284+0000 7f3463400640 1 --2- 172.21.15.62:0/3289825942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f346415cdc0 0x7f34640d1f20 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f3450009870 tx=0x7f345002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae94ce2a9c9c678 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:18.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.284+0000 7f3462a00640 1 -- 172.21.15.62:0/3289825942 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450030840 con 0x7f346415cdc0 2024-03-23T11:32:18.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.284+0000 7f3462a00640 1 -- 172.21.15.62:0/3289825942 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34500309a0 con 0x7f346415cdc0 2024-03-23T11:32:18.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.284+0000 7f3462a00640 1 -- 172.21.15.62:0/3289825942 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450030d30 con 0x7f346415cdc0 2024-03-23T11:32:18.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.285+0000 7f346ae00640 1 -- 172.21.15.62:0/3289825942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f346415cdc0 msgr2=0x7f34640d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.285+0000 7f346ae00640 1 --2- 172.21.15.62:0/3289825942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f346415cdc0 0x7f34640d1f20 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f3450009870 tx=0x7f345002b980 comp rx=0 tx=0).stop 2024-03-23T11:32:18.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.285+0000 7f346ae00640 1 -- 172.21.15.62:0/3289825942 shutdown_connections 2024-03-23T11:32:18.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.285+0000 7f346ae00640 1 --2- 172.21.15.62:0/3289825942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f346415cdc0 0x7f34640d1f20 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.285+0000 7f346ae00640 1 --2- 172.21.15.62:0/3289825942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f346415c9e0 0x7f34641607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.285+0000 7f346ae00640 1 -- 172.21.15.62:0/3289825942 >> 172.21.15.62:0/3289825942 conn(0x7f3464071570 msgr2=0x7f3464071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:18.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.285+0000 7f346ae00640 1 -- 172.21.15.62:0/3289825942 shutdown_connections 2024-03-23T11:32:18.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.285+0000 7f346ae00640 1 -- 172.21.15.62:0/3289825942 wait complete. 2024-03-23T11:32:18.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.287+0000 7f346ae00640 1 Processor -- start 2024-03-23T11:32:18.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.287+0000 7f346ae00640 1 -- start start 2024-03-23T11:32:18.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.288+0000 7f346ae00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34640cf650 0x7f34640c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.288+0000 7f346ae00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f346415c9e0 0x7f346414b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.288+0000 7f346ae00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f346415cdc0 0x7f3464160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.288+0000 7f346ae00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346414c540 con 0x7f34640cf650 2024-03-23T11:32:18.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.288+0000 7f346ae00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346414bf80 con 0x7f346415cdc0 2024-03-23T11:32:18.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.288+0000 7f346ae00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346414c260 con 0x7f346415c9e0 2024-03-23T11:32:18.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.288+0000 7f3463e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34640cf650 0x7f34640c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:18.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.289+0000 7f3463e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34640cf650 0x7f34640c9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60352/0 (socket says 172.21.15.62:60352) 2024-03-23T11:32:18.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.289+0000 7f3463e00640 1 -- 172.21.15.62:0/3347562259 learned_addr learned my addr 172.21.15.62:0/3347562259 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:18.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.289+0000 7f3463400640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f346415c9e0 0x7f346414b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:18.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.289+0000 7f3468e00640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f346415cdc0 0x7f3464160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:18.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.289+0000 7f3463e00640 1 -- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f346415c9e0 msgr2=0x7f346414b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.289+0000 7f3463e00640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f346415c9e0 0x7f346414b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.289+0000 7f3463e00640 1 -- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f346415cdc0 msgr2=0x7f3464160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.289+0000 7f3463e00640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f346415cdc0 0x7f3464160730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.289+0000 7f3463e00640 1 -- 172.21.15.62:0/3347562259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34500094d0 con 0x7f34640cf650 2024-03-23T11:32:18.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.290+0000 7f3463e00640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34640cf650 0x7f34640c9350 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f345400d800 tx=0x7f345400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:18.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.290+0000 7f345be00640 1 -- 172.21.15.62:0/3347562259 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345400b6c0 con 0x7f34640cf650 2024-03-23T11:32:18.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.290+0000 7f345be00640 1 -- 172.21.15.62:0/3347562259 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345400b820 con 0x7f34640cf650 2024-03-23T11:32:18.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.290+0000 7f345be00640 1 -- 172.21.15.62:0/3347562259 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345400bbb0 con 0x7f34640cf650 2024-03-23T11:32:18.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.290+0000 7f346ae00640 1 -- 172.21.15.62:0/3347562259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34641496b0 con 0x7f34640cf650 2024-03-23T11:32:18.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.290+0000 7f346ae00640 1 -- 172.21.15.62:0/3347562259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346414f180 con 0x7f34640cf650 2024-03-23T11:32:18.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.291+0000 7f346ae00640 1 -- 172.21.15.62:0/3347562259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3464149510 con 0x7f34640cf650 2024-03-23T11:32:18.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.292+0000 7f345be00640 1 -- 172.21.15.62:0/3347562259 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f345400bd10 con 0x7f34640cf650 2024-03-23T11:32:18.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.293+0000 7f345be00640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f344406a7a0 0x7f344406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.293+0000 7f345be00640 1 -- 172.21.15.62:0/3347562259 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3454089380 con 0x7f34640cf650 2024-03-23T11:32:18.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.293+0000 7f3463400640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f344406a7a0 0x7f344406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:18.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.294+0000 7f3463400640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f344406a7a0 0x7f344406cc60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3450036000 tx=0x7f3450008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:18.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.298+0000 7f345be00640 1 -- 172.21.15.62:0/3347562259 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3464149510 con 0x7f34640cf650 2024-03-23T11:32:18.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.504+0000 7f346ae00640 1 -- 172.21.15.62:0/3347562259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3464149510 con 0x7f34640cf650 2024-03-23T11:32:18.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.504+0000 7f345be00640 1 -- 172.21.15.62:0/3347562259 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3464149510 con 0x7f34640cf650 2024-03-23T11:32:18.494 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:32:18.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.507+0000 7f346ae00640 1 -- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f344406a7a0 msgr2=0x7f344406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.507+0000 7f346ae00640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f344406a7a0 0x7f344406cc60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3450036000 tx=0x7f3450008030 comp rx=0 tx=0).stop 2024-03-23T11:32:18.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.507+0000 7f346ae00640 1 -- 172.21.15.62:0/3347562259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34640cf650 msgr2=0x7f34640c9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.507+0000 7f346ae00640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34640cf650 0x7f34640c9350 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f345400d800 tx=0x7f345400dcc0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.508+0000 7f346ae00640 1 -- 172.21.15.62:0/3347562259 shutdown_connections 2024-03-23T11:32:18.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.508+0000 7f346ae00640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f344406a7a0 0x7f344406cc60 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.508+0000 7f346ae00640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f346415cdc0 0x7f3464160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.508+0000 7f346ae00640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f346415c9e0 0x7f346414b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.508+0000 7f346ae00640 1 --2- 172.21.15.62:0/3347562259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34640cf650 0x7f34640c9350 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.508+0000 7f346ae00640 1 -- 172.21.15.62:0/3347562259 >> 172.21.15.62:0/3347562259 conn(0x7f3464071570 msgr2=0x7f34640d2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:18.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.508+0000 7f346ae00640 1 -- 172.21.15.62:0/3347562259 shutdown_connections 2024-03-23T11:32:18.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.508+0000 7f346ae00640 1 -- 172.21.15.62:0/3347562259 wait complete. 2024-03-23T11:32:18.518 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:32:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.750+0000 7fb656a00640 1 Processor -- start 2024-03-23T11:32:18.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.751+0000 7fb656a00640 1 -- start start 2024-03-23T11:32:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.752+0000 7fb656a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb6500cafa0 0x7fb6500cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.752+0000 7fb656a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb65006c6e0 0x7fb65006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.752+0000 7fb656a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6500cbc60 con 0x7fb65006c300 2024-03-23T11:32:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.752+0000 7fb656a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6500cd410 con 0x7fb6500cafa0 2024-03-23T11:32:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.752+0000 7fb656a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500cbf40 con 0x7fb65006c6e0 2024-03-23T11:32:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.753+0000 7fb64fe00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb65006c300 0x7fb6500ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58022/0 (socket says 172.21.15.62:58022) 2024-03-23T11:32:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.753+0000 7fb64fe00640 1 -- 172.21.15.62:0/3182966386 learned_addr learned my addr 172.21.15.62:0/3182966386 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.753+0000 7fb647e00640 1 --2- 172.21.15.62:0/3182966386 >> v2:172.21.15.120:3300/0 conn(0x7fb6500cafa0 0x7fb6500cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.753+0000 7fb654a00640 1 --2- 172.21.15.62:0/3182966386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb65006c6e0 0x7fb65006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.754+0000 7fb64f400640 1 -- 172.21.15.62:0/3182966386 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733692807 0 0) 0x7fb6500cbc60 con 0x7fb65006c300 2024-03-23T11:32:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.754+0000 7fb64f400640 1 -- 172.21.15.62:0/3182966386 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb634003520 con 0x7fb65006c300 2024-03-23T11:32:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.754+0000 7fb647e00640 1 -- 172.21.15.62:0/3182966386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb65006c6e0 msgr2=0x7fb65006cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.754+0000 7fb647e00640 1 --2- 172.21.15.62:0/3182966386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb65006c6e0 0x7fb65006cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.754+0000 7fb647e00640 1 -- 172.21.15.62:0/3182966386 >> v1:172.21.15.62:6789/0 conn(0x7fb65006c300 legacy=0x7fb6500ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.754+0000 7fb647e00640 1 -- 172.21.15.62:0/3182966386 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6500cc220 con 0x7fb6500cafa0 2024-03-23T11:32:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.755+0000 7fb647e00640 1 --2- 172.21.15.62:0/3182966386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6500cafa0 0x7fb6500cb430 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb6380097a0 tx=0x7fb63802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a55338315fd207ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.756+0000 7fb64f400640 1 -- 172.21.15.62:0/3182966386 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63802d040 con 0x7fb6500cafa0 2024-03-23T11:32:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.756+0000 7fb64f400640 1 -- 172.21.15.62:0/3182966386 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb638031810 con 0x7fb6500cafa0 2024-03-23T11:32:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.756+0000 7fb64f400640 1 -- 172.21.15.62:0/3182966386 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb638031b80 con 0x7fb6500cafa0 2024-03-23T11:32:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.757+0000 7fb656a00640 1 -- 172.21.15.62:0/3182966386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6500cafa0 msgr2=0x7fb6500cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.757+0000 7fb656a00640 1 --2- 172.21.15.62:0/3182966386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6500cafa0 0x7fb6500cb430 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb6380097a0 tx=0x7fb63802b8b0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.757+0000 7fb656a00640 1 -- 172.21.15.62:0/3182966386 shutdown_connections 2024-03-23T11:32:18.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.757+0000 7fb656a00640 1 --2- 172.21.15.62:0/3182966386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb65006c6e0 0x7fb65006cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.757+0000 7fb656a00640 1 --2- 172.21.15.62:0/3182966386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6500cafa0 0x7fb6500cb430 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.757+0000 7fb656a00640 1 -- 172.21.15.62:0/3182966386 >> 172.21.15.62:0/3182966386 conn(0x7fb6500b02e0 msgr2=0x7fb6500b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:18.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.757+0000 7fb656a00640 1 -- 172.21.15.62:0/3182966386 shutdown_connections 2024-03-23T11:32:18.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.757+0000 7fb656a00640 1 -- 172.21.15.62:0/3182966386 wait complete. 2024-03-23T11:32:18.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.758+0000 7fb656a00640 1 Processor -- start 2024-03-23T11:32:18.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.759+0000 7fb656a00640 1 -- start start 2024-03-23T11:32:18.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.759+0000 7fb656a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65006c300 0x7fb6500aa180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb656a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb65006c6e0 0x7fb65013bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb656a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6500cafa0 0x7fb65013c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb656a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65015c570 con 0x7fb65006c300 2024-03-23T11:32:18.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb656a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6501f4460 con 0x7fb6500cafa0 2024-03-23T11:32:18.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb656a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500aa6c0 con 0x7fb65006c6e0 2024-03-23T11:32:18.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb64fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65006c300 0x7fb6500aa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:18.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb64fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65006c300 0x7fb6500aa180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60354/0 (socket says 172.21.15.62:60354) 2024-03-23T11:32:18.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb64fe00640 1 -- 172.21.15.62:0/2781873227 learned_addr learned my addr 172.21.15.62:0/2781873227 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb647e00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb65006c6e0 0x7fb65013bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb64fe00640 1 -- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb65006c6e0 msgr2=0x7fb65013bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb64fe00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb65006c6e0 0x7fb65013bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb64fe00640 1 -- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6500cafa0 msgr2=0x7fb65013c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb64fe00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6500cafa0 0x7fb65013c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.760+0000 7fb64fe00640 1 -- 172.21.15.62:0/2781873227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb638009400 con 0x7fb65006c300 2024-03-23T11:32:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.761+0000 7fb64fe00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65006c300 0x7fb6500aa180 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fb64000c430 tx=0x7fb64000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.761+0000 7fb64cc00640 1 -- 172.21.15.62:0/2781873227 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640005ce0 con 0x7fb65006c300 2024-03-23T11:32:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.761+0000 7fb64cc00640 1 -- 172.21.15.62:0/2781873227 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb640005e40 con 0x7fb65006c300 2024-03-23T11:32:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.761+0000 7fb656a00640 1 -- 172.21.15.62:0/2781873227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6501f4180 con 0x7fb65006c300 2024-03-23T11:32:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.761+0000 7fb64cc00640 1 -- 172.21.15.62:0/2781873227 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6400072f0 con 0x7fb65006c300 2024-03-23T11:32:18.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.761+0000 7fb656a00640 1 -- 172.21.15.62:0/2781873227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65015cd00 con 0x7fb65006c300 2024-03-23T11:32:18.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.762+0000 7fb656a00640 1 -- 172.21.15.62:0/2781873227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb650066c90 con 0x7fb65006c300 2024-03-23T11:32:18.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.763+0000 7fb64cc00640 1 -- 172.21.15.62:0/2781873227 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb640007450 con 0x7fb65006c300 2024-03-23T11:32:18.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.764+0000 7fb64cc00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb62c06a7a0 0x7fb62c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:18.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.764+0000 7fb64cc00640 1 -- 172.21.15.62:0/2781873227 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb640088420 con 0x7fb65006c300 2024-03-23T11:32:18.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.765+0000 7fb647e00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb62c06a7a0 0x7fb62c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:18.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.766+0000 7fb647e00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb62c06a7a0 0x7fb62c06cc60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb638036000 tx=0x7fb638008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:18.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.768+0000 7fb64cc00640 1 -- 172.21.15.62:0/2781873227 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb6400537a0 con 0x7fb65006c300 2024-03-23T11:32:18.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.982+0000 7fb656a00640 1 -- 172.21.15.62:0/2781873227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb65015a1d0 con 0x7fb65006c300 2024-03-23T11:32:18.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.982+0000 7fb64cc00640 1 -- 172.21.15.62:0/2781873227 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v121) v1 ==== 80+0+124 (secure 0 0 0) 0x7fb640017090 con 0x7fb65006c300 2024-03-23T11:32:18.972 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6837/2585024035 2024-03-24T11:32:04.365655+0000 2024-03-23T11:32:18.972 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6835/2585024035 2024-03-24T11:32:04.365655+0000 2024-03-23T11:32:18.972 INFO:teuthology.orchestra.run.smithi062.stderr:listed 2 entries 2024-03-23T11:32:18.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.985+0000 7fb656a00640 1 -- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb62c06a7a0 msgr2=0x7fb62c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.985+0000 7fb656a00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb62c06a7a0 0x7fb62c06cc60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb638036000 tx=0x7fb638008030 comp rx=0 tx=0).stop 2024-03-23T11:32:18.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.985+0000 7fb656a00640 1 -- 172.21.15.62:0/2781873227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65006c300 msgr2=0x7fb6500aa180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:18.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.985+0000 7fb656a00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65006c300 0x7fb6500aa180 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fb64000c430 tx=0x7fb64000c8f0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.985+0000 7fb656a00640 1 -- 172.21.15.62:0/2781873227 shutdown_connections 2024-03-23T11:32:18.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.985+0000 7fb656a00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb62c06a7a0 0x7fb62c06cc60 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.985+0000 7fb656a00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6500cafa0 0x7fb65013c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.985+0000 7fb656a00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb65006c6e0 0x7fb65013bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.985+0000 7fb656a00640 1 --2- 172.21.15.62:0/2781873227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65006c300 0x7fb6500aa180 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:18.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.985+0000 7fb656a00640 1 -- 172.21.15.62:0/2781873227 >> 172.21.15.62:0/2781873227 conn(0x7fb6500b02e0 msgr2=0x7fb65013eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:18.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.985+0000 7fb656a00640 1 -- 172.21.15.62:0/2781873227 shutdown_connections 2024-03-23T11:32:18.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:18.986+0000 7fb656a00640 1 -- 172.21.15.62:0/2781873227 wait complete. 2024-03-23T11:32:18.995 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:32:18.995 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:32:19.049 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:32:19.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.281+0000 7ff8dd000640 1 Processor -- start 2024-03-23T11:32:19.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.281+0000 7ff8dd000640 1 -- start start 2024-03-23T11:32:19.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.283+0000 7ff8dd000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff8d80707a0 0x7ff8d815d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.283+0000 7ff8dd000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff8d8159060 0x7ff8d8159490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.283+0000 7ff8dd000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d815ced0 con 0x7ff8d8158c80 2024-03-23T11:32:19.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.283+0000 7ff8dd000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d806d870 con 0x7ff8d80707a0 2024-03-23T11:32:19.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.283+0000 7ff8dd000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d8159e70 con 0x7ff8d8159060 2024-03-23T11:32:19.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.283+0000 7ff8cfe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff8d80707a0 0x7ff8d815d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.283+0000 7ff8cfe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff8d80707a0 0x7ff8d815d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60362/0 (socket says 172.21.15.62:60362) 2024-03-23T11:32:19.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.283+0000 7ff8cfe00640 1 -- 172.21.15.62:0/893456523 learned_addr learned my addr 172.21.15.62:0/893456523 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:19.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.284+0000 7ff8d7e00640 1 --2- 172.21.15.62:0/893456523 >> v2:172.21.15.120:3300/0 conn(0x7ff8d8159060 0x7ff8d8159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.284+0000 7ff8cfe00640 1 -- 172.21.15.62:0/893456523 >> v2:172.21.15.120:3300/0 conn(0x7ff8d8159060 msgr2=0x7ff8d8159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:19.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.284+0000 7ff8cfe00640 1 --2- 172.21.15.62:0/893456523 >> v2:172.21.15.120:3300/0 conn(0x7ff8d8159060 0x7ff8d8159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.284+0000 7ff8cfe00640 1 -- 172.21.15.62:0/893456523 >> v1:172.21.15.120:6789/0 conn(0x7ff8d8158c80 legacy=0x7ff8d8070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:19.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.284+0000 7ff8cfe00640 1 -- 172.21.15.62:0/893456523 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d815a150 con 0x7ff8d80707a0 2024-03-23T11:32:19.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.285+0000 7ff8cfe00640 1 --2- 172.21.15.62:0/893456523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d80707a0 0x7ff8d815d360 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80ce210 tx=0x7ff8c002cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150d772cf9e0d141 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:19.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.285+0000 7ff8d6a00640 1 -- 172.21.15.62:0/893456523 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c002dba0 con 0x7ff8d80707a0 2024-03-23T11:32:19.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.285+0000 7ff8d6a00640 1 -- 172.21.15.62:0/893456523 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c002dd00 con 0x7ff8d80707a0 2024-03-23T11:32:19.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.285+0000 7ff8d6a00640 1 -- 172.21.15.62:0/893456523 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0035660 con 0x7ff8d80707a0 2024-03-23T11:32:19.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.286+0000 7ff8dd000640 1 -- 172.21.15.62:0/893456523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d80707a0 msgr2=0x7ff8d815d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:19.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.286+0000 7ff8dd000640 1 --2- 172.21.15.62:0/893456523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d80707a0 0x7ff8d815d360 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80ce210 tx=0x7ff8c002cdf0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.286+0000 7ff8dd000640 1 -- 172.21.15.62:0/893456523 shutdown_connections 2024-03-23T11:32:19.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.286+0000 7ff8dd000640 1 --2- 172.21.15.62:0/893456523 >> v2:172.21.15.120:3300/0 conn(0x7ff8d8159060 0x7ff8d8159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.286+0000 7ff8dd000640 1 --2- 172.21.15.62:0/893456523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d80707a0 0x7ff8d815d360 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.286+0000 7ff8dd000640 1 -- 172.21.15.62:0/893456523 >> 172.21.15.62:0/893456523 conn(0x7ff8d8066050 msgr2=0x7ff8d8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:19.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.286+0000 7ff8dd000640 1 -- 172.21.15.62:0/893456523 shutdown_connections 2024-03-23T11:32:19.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.286+0000 7ff8dd000640 1 -- 172.21.15.62:0/893456523 wait complete. 2024-03-23T11:32:19.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.288+0000 7ff8dd000640 1 Processor -- start 2024-03-23T11:32:19.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.288+0000 7ff8dd000640 1 -- start start 2024-03-23T11:32:19.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.289+0000 7ff8dd000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8d80707a0 0x7ff8d81552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.289+0000 7ff8dd000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8d8158c80 0x7ff8d81efff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.289+0000 7ff8dd000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d8159060 0x7ff8d81edc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.289+0000 7ff8dd000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d81f1590 con 0x7ff8d8159060 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.289+0000 7ff8dd000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d81f1870 con 0x7ff8d8158c80 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.289+0000 7ff8dd000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d81f1b50 con 0x7ff8d80707a0 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.289+0000 7ff8d7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d8159060 0x7ff8d81edc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.289+0000 7ff8d7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d8159060 0x7ff8d81edc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60366/0 (socket says 172.21.15.62:60366) 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.289+0000 7ff8d7e00640 1 -- 172.21.15.62:0/1373169468 learned_addr learned my addr 172.21.15.62:0/1373169468 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8cfe00640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8d8158c80 0x7ff8d81efff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8d7400640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8d80707a0 0x7ff8d81552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8d7e00640 1 -- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8d80707a0 msgr2=0x7ff8d81552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8d7e00640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8d80707a0 0x7ff8d81552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8d7e00640 1 -- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8d8158c80 msgr2=0x7ff8d81efff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8d7e00640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8d8158c80 0x7ff8d81efff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8d7e00640 1 -- 172.21.15.62:0/1373169468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c000a670 con 0x7ff8d8159060 2024-03-23T11:32:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8d7400640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8d80707a0 0x7ff8d81552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8d7e00640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d8159060 0x7ff8d81edc30 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0007e60 tx=0x7ff8d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8cf400640 1 -- 172.21.15.62:0/1373169468 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0014420 con 0x7ff8d8159060 2024-03-23T11:32:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8cf400640 1 -- 172.21.15.62:0/1373169468 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d00145a0 con 0x7ff8d8159060 2024-03-23T11:32:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8cf400640 1 -- 172.21.15.62:0/1373169468 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0014950 con 0x7ff8d8159060 2024-03-23T11:32:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.290+0000 7ff8dd000640 1 -- 172.21.15.62:0/1373169468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d81f1e30 con 0x7ff8d8159060 2024-03-23T11:32:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.291+0000 7ff8dd000640 1 -- 172.21.15.62:0/1373169468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d81f11b0 con 0x7ff8d8159060 2024-03-23T11:32:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.292+0000 7ff8dd000640 1 -- 172.21.15.62:0/1373169468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d81f2350 con 0x7ff8d8159060 2024-03-23T11:32:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.293+0000 7ff8cf400640 1 -- 172.21.15.62:0/1373169468 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff8d0014ab0 con 0x7ff8d8159060 2024-03-23T11:32:19.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.294+0000 7ff8cf400640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8b406a7a0 0x7ff8b406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.294+0000 7ff8cf400640 1 -- 172.21.15.62:0/1373169468 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff8d00897d0 con 0x7ff8d8159060 2024-03-23T11:32:19.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.294+0000 7ff8d7400640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8b406a7a0 0x7ff8b406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.295+0000 7ff8d7400640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8b406a7a0 0x7ff8b406cc60 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8003eb0 tx=0x7ff8c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:19.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.298+0000 7ff8cf400640 1 -- 172.21.15.62:0/1373169468 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff8d81f2350 con 0x7ff8d8159060 2024-03-23T11:32:19.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.499+0000 7ff8dd000640 1 -- 172.21.15.62:0/1373169468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8d81f2350 con 0x7ff8d8159060 2024-03-23T11:32:19.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.502+0000 7ff8cf400640 1 -- 172.21.15.62:0/1373169468 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+16283 (secure 0 0 0) 0x7ff8d81f2350 con 0x7ff8d8159060 2024-03-23T11:32:19.491 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:19.492 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":121,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:32:04.365698+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6837/2585024035":"2024-03-24T11:32:04.365655+0000","172.21.15.120:6835/2585024035":"2024-03-24T11:32:04.365655+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:32:19.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.504+0000 7ff8dd000640 1 -- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8b406a7a0 msgr2=0x7ff8b406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:19.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.504+0000 7ff8dd000640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8b406a7a0 0x7ff8b406cc60 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8003eb0 tx=0x7ff8c8009040 comp rx=0 tx=0).stop 2024-03-23T11:32:19.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.504+0000 7ff8dd000640 1 -- 172.21.15.62:0/1373169468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d8159060 msgr2=0x7ff8d81edc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:19.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.504+0000 7ff8dd000640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d8159060 0x7ff8d81edc30 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0007e60 tx=0x7ff8d000e4b0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.505+0000 7ff8dd000640 1 -- 172.21.15.62:0/1373169468 shutdown_connections 2024-03-23T11:32:19.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.505+0000 7ff8dd000640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff8b406a7a0 0x7ff8b406cc60 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.505+0000 7ff8dd000640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d8159060 0x7ff8d81edc30 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.505+0000 7ff8dd000640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8d8158c80 0x7ff8d81efff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.505+0000 7ff8dd000640 1 --2- 172.21.15.62:0/1373169468 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8d80707a0 0x7ff8d81552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.505+0000 7ff8dd000640 1 -- 172.21.15.62:0/1373169468 >> 172.21.15.62:0/1373169468 conn(0x7ff8d8066050 msgr2=0x7ff8d80b0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:19.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.505+0000 7ff8dd000640 1 -- 172.21.15.62:0/1373169468 shutdown_connections 2024-03-23T11:32:19.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.505+0000 7ff8dd000640 1 -- 172.21.15.62:0/1373169468 wait complete. 2024-03-23T11:32:19.513 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:31:22.127079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:31:23.062387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:32:19.514 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:32:19.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.755+0000 7f6272000640 1 Processor -- start 2024-03-23T11:32:19.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.755+0000 7f6272000640 1 -- start start 2024-03-23T11:32:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.756+0000 7f6272000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f626c0ae880 0x7f626c0af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.757+0000 7f6272000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f626c0989d0 0x7f626c098e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.757+0000 7f6272000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c0cc860 con 0x7f626c0ce8d0 2024-03-23T11:32:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.757+0000 7f6272000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626c0cac10 con 0x7f626c0989d0 2024-03-23T11:32:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.757+0000 7f6272000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0cbfc0 con 0x7f626c0ae880 2024-03-23T11:32:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.757+0000 7f6270c00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f626c0ce8d0 0x7f626c0cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.62:42710/0 (socket says 172.21.15.62:42710) 2024-03-23T11:32:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.757+0000 7f6270c00640 1 -- 172.21.15.62:0/461328379 learned_addr learned my addr 172.21.15.62:0/461328379 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.757+0000 7f6271600640 1 --2- 172.21.15.62:0/461328379 >> v2:172.21.15.120:3300/0 conn(0x7f626c0989d0 0x7f626c098e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.757+0000 7f626be00640 1 --2- 172.21.15.62:0/461328379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f626c0ae880 0x7f626c0af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.758+0000 7f626b400640 1 -- 172.21.15.62:0/461328379 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811659619 0 0) 0x7f626c0cc860 con 0x7f626c0ce8d0 2024-03-23T11:32:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.758+0000 7f626b400640 1 -- 172.21.15.62:0/461328379 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6250003520 con 0x7f626c0ce8d0 2024-03-23T11:32:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.758+0000 7f6271600640 1 -- 172.21.15.62:0/461328379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f626c0ae880 msgr2=0x7f626c0af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.758+0000 7f6271600640 1 --2- 172.21.15.62:0/461328379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f626c0ae880 0x7f626c0af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.758+0000 7f6271600640 1 -- 172.21.15.62:0/461328379 >> v1:172.21.15.120:6789/0 conn(0x7f626c0ce8d0 legacy=0x7f626c0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.758+0000 7f6271600640 1 -- 172.21.15.62:0/461328379 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c0cc2a0 con 0x7f626c0989d0 2024-03-23T11:32:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.760+0000 7f6271600640 1 --2- 172.21.15.62:0/461328379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f626c0989d0 0x7f626c098e60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6264009d90 tx=0x7f626402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ad8c723b0926031 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.760+0000 7f626b400640 1 -- 172.21.15.62:0/461328379 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626402cc50 con 0x7f626c0989d0 2024-03-23T11:32:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.760+0000 7f626b400640 1 -- 172.21.15.62:0/461328379 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626402cdb0 con 0x7f626c0989d0 2024-03-23T11:32:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.760+0000 7f626b400640 1 -- 172.21.15.62:0/461328379 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6264034660 con 0x7f626c0989d0 2024-03-23T11:32:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.761+0000 7f6272000640 1 -- 172.21.15.62:0/461328379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f626c0989d0 msgr2=0x7f626c098e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.761+0000 7f6272000640 1 --2- 172.21.15.62:0/461328379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f626c0989d0 0x7f626c098e60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6264009d90 tx=0x7f626402bea0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.761+0000 7f6272000640 1 -- 172.21.15.62:0/461328379 shutdown_connections 2024-03-23T11:32:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.761+0000 7f6272000640 1 --2- 172.21.15.62:0/461328379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f626c0989d0 0x7f626c098e60 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.761+0000 7f6272000640 1 --2- 172.21.15.62:0/461328379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f626c0ae880 0x7f626c0af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.761+0000 7f6272000640 1 -- 172.21.15.62:0/461328379 >> 172.21.15.62:0/461328379 conn(0x7f626c0b1fb0 msgr2=0x7f626c0b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:19.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.761+0000 7f6272000640 1 -- 172.21.15.62:0/461328379 shutdown_connections 2024-03-23T11:32:19.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.761+0000 7f6272000640 1 -- 172.21.15.62:0/461328379 wait complete. 2024-03-23T11:32:19.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.762+0000 7f6272000640 1 Processor -- start 2024-03-23T11:32:19.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.763+0000 7f6272000640 1 -- start start 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.763+0000 7f6272000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f626c0989d0 0x7f626c14c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.763+0000 7f6272000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f626c0ae880 0x7f626c14ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.763+0000 7f6272000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f626c0ce8d0 0x7f626c1f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.763+0000 7f6272000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0ca410 con 0x7f626c0ae880 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.763+0000 7f6272000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c1efc30 con 0x7f626c0989d0 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.763+0000 7f6272000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0ca130 con 0x7f626c0ce8d0 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f626be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f626c0ae880 0x7f626c14ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f626be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f626c0ae880 0x7f626c14ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60376/0 (socket says 172.21.15.62:60376) 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f626be00640 1 -- 172.21.15.62:0/3546697036 learned_addr learned my addr 172.21.15.62:0/3546697036 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f6270c00640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f626c0989d0 0x7f626c14c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f6271600640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f626c0ce8d0 0x7f626c1f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f626be00640 1 -- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f626c0ce8d0 msgr2=0x7f626c1f35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:19.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f626be00640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f626c0ce8d0 0x7f626c1f35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f626be00640 1 -- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f626c0989d0 msgr2=0x7f626c14c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:19.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f626be00640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f626c0989d0 0x7f626c14c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f626be00640 1 -- 172.21.15.62:0/3546697036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6264009a70 con 0x7f626c0ae880 2024-03-23T11:32:19.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f6270c00640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f626c0989d0 0x7f626c14c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:19.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.764+0000 7f6271600640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f626c0ce8d0 0x7f626c1f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:19.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.765+0000 7f626be00640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f626c0ae880 0x7f626c14ca30 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f625800e9b0 tx=0x7f625800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:19.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.765+0000 7f6268c00640 1 -- 172.21.15.62:0/3546697036 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6258009670 con 0x7f626c0ae880 2024-03-23T11:32:19.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.765+0000 7f6268c00640 1 -- 172.21.15.62:0/3546697036 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625800c9a0 con 0x7f626c0ae880 2024-03-23T11:32:19.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.765+0000 7f6268c00640 1 -- 172.21.15.62:0/3546697036 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625800cd50 con 0x7f626c0ae880 2024-03-23T11:32:19.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.765+0000 7f6272000640 1 -- 172.21.15.62:0/3546697036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626c0c62b0 con 0x7f626c0ae880 2024-03-23T11:32:19.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.765+0000 7f6272000640 1 -- 172.21.15.62:0/3546697036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c0c6870 con 0x7f626c0ae880 2024-03-23T11:32:19.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.767+0000 7f6272000640 1 -- 172.21.15.62:0/3546697036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626c073540 con 0x7f626c0ae880 2024-03-23T11:32:19.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.767+0000 7f6268c00640 1 -- 172.21.15.62:0/3546697036 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6258013470 con 0x7f626c0ae880 2024-03-23T11:32:19.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.768+0000 7f6268c00640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f624c06a7d0 0x7f624c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:19.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.768+0000 7f6268c00640 1 -- 172.21.15.62:0/3546697036 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f6258089040 con 0x7f626c0ae880 2024-03-23T11:32:19.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.769+0000 7f6270c00640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f624c06a7d0 0x7f624c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:19.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.770+0000 7f6270c00640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f624c06a7d0 0x7f624c06cc90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f625c0011e0 tx=0x7f625c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:19.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.773+0000 7f6268c00640 1 -- 172.21.15.62:0/3546697036 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f62580bc580 con 0x7f626c0ae880 2024-03-23T11:32:19.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.975+0000 7f6272000640 1 -- 172.21.15.62:0/3546697036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f626c0672c0 con 0x7f626c0ae880 2024-03-23T11:32:19.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.976+0000 7f6268c00640 1 -- 172.21.15.62:0/3546697036 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62580543c0 con 0x7f626c0ae880 2024-03-23T11:32:19.966 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:32:19.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.979+0000 7f6272000640 1 -- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f624c06a7d0 msgr2=0x7f624c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:19.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.979+0000 7f6272000640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f624c06a7d0 0x7f624c06cc90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f625c0011e0 tx=0x7f625c009040 comp rx=0 tx=0).stop 2024-03-23T11:32:19.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.979+0000 7f6272000640 1 -- 172.21.15.62:0/3546697036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f626c0ae880 msgr2=0x7f626c14ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:19.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.979+0000 7f6272000640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f626c0ae880 0x7f626c14ca30 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f625800e9b0 tx=0x7f625800ee70 comp rx=0 tx=0).stop 2024-03-23T11:32:19.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.979+0000 7f6272000640 1 -- 172.21.15.62:0/3546697036 shutdown_connections 2024-03-23T11:32:19.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.979+0000 7f6272000640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f624c06a7d0 0x7f624c06cc90 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.979+0000 7f6272000640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f626c0ae880 0x7f626c14ca30 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.979+0000 7f6272000640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f626c0ce8d0 0x7f626c1f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.979+0000 7f6272000640 1 --2- 172.21.15.62:0/3546697036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f626c0989d0 0x7f626c14c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:19.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.979+0000 7f6272000640 1 -- 172.21.15.62:0/3546697036 >> 172.21.15.62:0/3546697036 conn(0x7f626c0b1fb0 msgr2=0x7f626c0ccc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:19.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.979+0000 7f6272000640 1 -- 172.21.15.62:0/3546697036 shutdown_connections 2024-03-23T11:32:19.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:19.980+0000 7f6272000640 1 -- 172.21.15.62:0/3546697036 wait complete. 2024-03-23T11:32:19.989 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:32:20.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.228+0000 7fecd4200640 1 Processor -- start 2024-03-23T11:32:20.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.228+0000 7fecd4200640 1 -- start start 2024-03-23T11:32:20.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.230+0000 7fecd4200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feccc06dca0 0x7feccc0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.230+0000 7fecd4200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feccc06c300 0x7feccc0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.230+0000 7fecd4200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feccc0cad20 con 0x7feccc0cd7c0 2024-03-23T11:32:20.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.230+0000 7fecd4200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feccc06c770 con 0x7feccc06c300 2024-03-23T11:32:20.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.230+0000 7fecd4200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feccc06e130 con 0x7feccc06dca0 2024-03-23T11:32:20.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.231+0000 7fecd0e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feccc06dca0 0x7feccc0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.231+0000 7fecd2200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feccc06c300 0x7feccc0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.231+0000 7fecd2200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feccc06c300 0x7feccc0cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:45172/0 (socket says 172.21.15.62:45172) 2024-03-23T11:32:20.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.231+0000 7fecd0e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feccc06dca0 0x7feccc0b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:57266/0 (socket says 172.21.15.62:57266) 2024-03-23T11:32:20.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.231+0000 7fecd2200640 1 -- 172.21.15.62:0/1268268769 learned_addr learned my addr 172.21.15.62:0/1268268769 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:20.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.231+0000 7fecd2200640 1 -- 172.21.15.62:0/1268268769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feccc06dca0 msgr2=0x7feccc0b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:20.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.231+0000 7fecd2200640 1 --2- 172.21.15.62:0/1268268769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feccc06dca0 0x7feccc0b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.231+0000 7fecd2200640 1 -- 172.21.15.62:0/1268268769 >> v1:172.21.15.120:6789/0 conn(0x7feccc0cd7c0 legacy=0x7feccc0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.231+0000 7fecd2200640 1 -- 172.21.15.62:0/1268268769 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc15a8a0 con 0x7feccc06c300 2024-03-23T11:32:20.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.233+0000 7fecd2200640 1 --2- 172.21.15.62:0/1268268769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feccc06c300 0x7feccc0cb490 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fecc4009e50 tx=0x7fecc402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47eb9894d92f68cb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:20.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.233+0000 7feccbe00640 1 -- 172.21.15.62:0/1268268769 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc402ca70 con 0x7feccc06c300 2024-03-23T11:32:20.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.233+0000 7feccbe00640 1 -- 172.21.15.62:0/1268268769 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc402cbd0 con 0x7feccc06c300 2024-03-23T11:32:20.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.233+0000 7feccbe00640 1 -- 172.21.15.62:0/1268268769 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc40343f0 con 0x7feccc06c300 2024-03-23T11:32:20.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.234+0000 7fecd4200640 1 -- 172.21.15.62:0/1268268769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feccc06c300 msgr2=0x7feccc0cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.234+0000 7fecd4200640 1 --2- 172.21.15.62:0/1268268769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feccc06c300 0x7feccc0cb490 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fecc4009e50 tx=0x7fecc402bed0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.234+0000 7fecd4200640 1 -- 172.21.15.62:0/1268268769 shutdown_connections 2024-03-23T11:32:20.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.234+0000 7fecd4200640 1 --2- 172.21.15.62:0/1268268769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feccc06c300 0x7feccc0cb490 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.234+0000 7fecd4200640 1 --2- 172.21.15.62:0/1268268769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feccc06dca0 0x7feccc0b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.234+0000 7fecd4200640 1 -- 172.21.15.62:0/1268268769 >> 172.21.15.62:0/1268268769 conn(0x7feccc0b02e0 msgr2=0x7feccc0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:20.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.234+0000 7fecd4200640 1 -- 172.21.15.62:0/1268268769 shutdown_connections 2024-03-23T11:32:20.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.234+0000 7fecd4200640 1 -- 172.21.15.62:0/1268268769 wait complete. 2024-03-23T11:32:20.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.236+0000 7fecd4200640 1 Processor -- start 2024-03-23T11:32:20.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.236+0000 7fecd4200640 1 -- start start 2024-03-23T11:32:20.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.237+0000 7fecd4200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feccc06c300 0x7feccc145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.237+0000 7fecd4200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feccc0cd7c0 0x7feccc155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.237+0000 7fecd4200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feccc14c320 0x7feccc0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.237+0000 7fecd4200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc156e90 con 0x7feccc06c300 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.237+0000 7fecd4200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc1462a0 con 0x7feccc0cd7c0 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.237+0000 7fecd4200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feccc1565f0 con 0x7feccc14c320 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.237+0000 7fecd1800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feccc06c300 0x7feccc145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.237+0000 7fecd1800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feccc06c300 0x7feccc145d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60390/0 (socket says 172.21.15.62:60390) 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.237+0000 7fecd1800640 1 -- 172.21.15.62:0/1205505985 learned_addr learned my addr 172.21.15.62:0/1205505985 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.238+0000 7fecd0e00640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feccc0cd7c0 0x7feccc155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.238+0000 7fecd2200640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feccc14c320 0x7feccc0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.238+0000 7fecd1800640 1 -- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feccc14c320 msgr2=0x7feccc0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.238+0000 7fecd1800640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feccc14c320 0x7feccc0702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.238+0000 7fecd1800640 1 -- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feccc0cd7c0 msgr2=0x7feccc155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.238+0000 7fecd1800640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feccc0cd7c0 0x7feccc155b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.238+0000 7fecd1800640 1 -- 172.21.15.62:0/1205505985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc4009b30 con 0x7feccc06c300 2024-03-23T11:32:20.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.238+0000 7fecd1800640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feccc06c300 0x7feccc145d60 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fecc000c3c0 tx=0x7fecc000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:20.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.239+0000 7fecc9600640 1 -- 172.21.15.62:0/1205505985 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc00052d0 con 0x7feccc06c300 2024-03-23T11:32:20.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.239+0000 7fecc9600640 1 -- 172.21.15.62:0/1205505985 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc0005450 con 0x7feccc06c300 2024-03-23T11:32:20.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.239+0000 7fecc9600640 1 -- 172.21.15.62:0/1205505985 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc0007260 con 0x7feccc06c300 2024-03-23T11:32:20.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.239+0000 7fecd4200640 1 -- 172.21.15.62:0/1205505985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feccc1568d0 con 0x7feccc06c300 2024-03-23T11:32:20.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.239+0000 7fecd4200640 1 -- 172.21.15.62:0/1205505985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feccc14fec0 con 0x7feccc06c300 2024-03-23T11:32:20.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.240+0000 7fecd4200640 1 -- 172.21.15.62:0/1205505985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feccc14fd20 con 0x7feccc06c300 2024-03-23T11:32:20.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.241+0000 7fecc9600640 1 -- 172.21.15.62:0/1205505985 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fecc00073c0 con 0x7feccc06c300 2024-03-23T11:32:20.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.242+0000 7fecc9600640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fecac06a7a0 0x7fecac06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.242+0000 7fecc9600640 1 -- 172.21.15.62:0/1205505985 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fecc0088200 con 0x7feccc06c300 2024-03-23T11:32:20.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.242+0000 7fecd0e00640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fecac06a7a0 0x7fecac06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.244+0000 7fecd0e00640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fecac06a7a0 0x7fecac06cc60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7feccc0acda0 tx=0x7fecb8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:20.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.246+0000 7fecc9600640 1 -- 172.21.15.62:0/1205505985 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feccc14fd20 con 0x7feccc06c300 2024-03-23T11:32:20.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.442+0000 7fecd4200640 1 -- 172.21.15.62:0/1205505985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feccc14fd20 con 0x7feccc06c300 2024-03-23T11:32:20.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.443+0000 7fecc9600640 1 -- 172.21.15.62:0/1205505985 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v121) v1 ==== 104+0+11 (secure 0 0 0) 0x7feccc14fd20 con 0x7feccc06c300 2024-03-23T11:32:20.432 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:32:20.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.445+0000 7fecd4200640 1 -- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fecac06a7a0 msgr2=0x7fecac06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.445+0000 7fecd4200640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fecac06a7a0 0x7fecac06cc60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7feccc0acda0 tx=0x7fecb8008040 comp rx=0 tx=0).stop 2024-03-23T11:32:20.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.445+0000 7fecd4200640 1 -- 172.21.15.62:0/1205505985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feccc06c300 msgr2=0x7feccc145d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.445+0000 7fecd4200640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feccc06c300 0x7feccc145d60 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fecc000c3c0 tx=0x7fecc000c880 comp rx=0 tx=0).stop 2024-03-23T11:32:20.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.445+0000 7fecd4200640 1 -- 172.21.15.62:0/1205505985 shutdown_connections 2024-03-23T11:32:20.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.445+0000 7fecd4200640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fecac06a7a0 0x7fecac06cc60 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.445+0000 7fecd4200640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feccc14c320 0x7feccc0702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.445+0000 7fecd4200640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feccc0cd7c0 0x7feccc155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.445+0000 7fecd4200640 1 --2- 172.21.15.62:0/1205505985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feccc06c300 0x7feccc145d60 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.445+0000 7fecd4200640 1 -- 172.21.15.62:0/1205505985 >> 172.21.15.62:0/1205505985 conn(0x7feccc0b02e0 msgr2=0x7feccc153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:20.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.445+0000 7fecd4200640 1 -- 172.21.15.62:0/1205505985 shutdown_connections 2024-03-23T11:32:20.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.446+0000 7fecd4200640 1 -- 172.21.15.62:0/1205505985 wait complete. 2024-03-23T11:32:20.455 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:32:20.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.679+0000 7f2901600640 1 Processor -- start 2024-03-23T11:32:20.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.680+0000 7f2901600640 1 -- start start 2024-03-23T11:32:20.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.681+0000 7f2901600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f28fc15fcf0 0x7f28fc0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.681+0000 7f2901600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0d14f0 con 0x7f28fc0b6dc0 2024-03-23T11:32:20.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.681+0000 7f2901600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0d3540 con 0x7f28fc160420 2024-03-23T11:32:20.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.681+0000 7f2901600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc0ce510 con 0x7f28fc15fcf0 2024-03-23T11:32:20.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.681+0000 7f2900c00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f28fc0b6dc0 0x7f28fc0d0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58036/0 (socket says 172.21.15.62:58036) 2024-03-23T11:32:20.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.681+0000 7f2900c00640 1 -- 172.21.15.62:0/1370332134 learned_addr learned my addr 172.21.15.62:0/1370332134 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:20.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529861522 0 0) 0x7f28fc0d14f0 con 0x7f28fc0b6dc0 2024-03-23T11:32:20.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e0003520 con 0x7f28fc0b6dc0 2024-03-23T11:32:20.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28fbe00640 1 --2- 172.21.15.62:0/1370332134 >> v2:172.21.15.120:3300/0 conn(0x7f28fc15fcf0 0x7f28fc0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1424533452 0 0) 0x7f28e0003520 con 0x7f28fc0b6dc0 2024-03-23T11:32:20.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 >> v2:172.21.15.120:3300/0 conn(0x7f28fc15fcf0 msgr2=0x7f28fc0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:20.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 --2- 172.21.15.62:0/1370332134 >> v2:172.21.15.120:3300/0 conn(0x7f28fc15fcf0 0x7f28fc0d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 >> v1:172.21.15.120:6789/0 conn(0x7f28fc160420 legacy=0x7f28fc0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:20.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0cdc70 con 0x7f28fc0b6dc0 2024-03-23T11:32:20.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f28f4001670 con 0x7f28fc0b6dc0 2024-03-23T11:32:20.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 >> v1:172.21.15.62:6789/0 conn(0x7f28fc0b6dc0 legacy=0x7f28fc0d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 --2- 172.21.15.62:0/1370332134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28e0003970 0x7f28e0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 --2- 172.21.15.62:0/1370332134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28e00047c0 0x7f28e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 --2- 172.21.15.62:0/1370332134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28e00052a0 0x7f28e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0ce510 con 0x7f28e00052a0 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e000c150 con 0x7f28e0003970 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.682+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e000c310 con 0x7f28e00047c0 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.683+0000 7f2900c00640 1 --2- 172.21.15.62:0/1370332134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28e00052a0 0x7f28e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.683+0000 7f28fbe00640 1 --2- 172.21.15.62:0/1370332134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28e0003970 0x7f28e0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.683+0000 7f28fb400640 1 --2- 172.21.15.62:0/1370332134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28e00047c0 0x7f28e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.683+0000 7f2900c00640 1 -- 172.21.15.62:0/1370332134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28e00047c0 msgr2=0x7f28e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.683+0000 7f2900c00640 1 --2- 172.21.15.62:0/1370332134 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28e00047c0 0x7f28e0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.683+0000 7f2900c00640 1 -- 172.21.15.62:0/1370332134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28e0003970 msgr2=0x7f28e0003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.683+0000 7f2900c00640 1 --2- 172.21.15.62:0/1370332134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28e0003970 0x7f28e0003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.683+0000 7f2900c00640 1 -- 172.21.15.62:0/1370332134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e0003520 con 0x7f28e00052a0 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.684+0000 7f2900c00640 1 -- 172.21.15.62:0/1370332134 reap_dead start 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.684+0000 7f2900c00640 1 --2- 172.21.15.62:0/1370332134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28e00052a0 0x7f28e000b980 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f28f4001eb0 tx=0x7f28e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71955e7644b146c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.685+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4005ad0 con 0x7f28e00052a0 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.685+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f4005c50 con 0x7f28e00052a0 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.685+0000 7f28faa00640 1 -- 172.21.15.62:0/1370332134 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f402c890 con 0x7f28e00052a0 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.685+0000 7f2901600640 1 -- 172.21.15.62:0/1370332134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28e00052a0 msgr2=0x7f28e000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.685+0000 7f2901600640 1 --2- 172.21.15.62:0/1370332134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28e00052a0 0x7f28e000b980 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f28f4001eb0 tx=0x7f28e00057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.685+0000 7f2901600640 1 -- 172.21.15.62:0/1370332134 shutdown_connections 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.685+0000 7f2901600640 1 --2- 172.21.15.62:0/1370332134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28e00052a0 0x7f28e000b980 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.685+0000 7f2901600640 1 -- 172.21.15.62:0/1370332134 >> 172.21.15.62:0/1370332134 conn(0x7f28fc089f60 msgr2=0x7f28fc0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.685+0000 7f2901600640 1 -- 172.21.15.62:0/1370332134 shutdown_connections 2024-03-23T11:32:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.685+0000 7f2901600640 1 -- 172.21.15.62:0/1370332134 wait complete. 2024-03-23T11:32:20.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.687+0000 7f2901600640 1 Processor -- start 2024-03-23T11:32:20.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.687+0000 7f2901600640 1 -- start start 2024-03-23T11:32:20.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.688+0000 7f2901600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28e00052a0 0x7f28fc0ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.688+0000 7f2901600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28fc1f57c0 0x7f28fc0cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.688+0000 7f2901600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28fc1f28e0 0x7f28fc1f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.688+0000 7f2901600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc1f21d0 con 0x7f28fc1f28e0 2024-03-23T11:32:20.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.688+0000 7f2901600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc1f6790 con 0x7f28e00052a0 2024-03-23T11:32:20.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.688+0000 7f2901600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc1f5c40 con 0x7f28fc1f57c0 2024-03-23T11:32:20.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.689+0000 7f2900c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28fc1f28e0 0x7f28fc1f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.689+0000 7f2900c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28fc1f28e0 0x7f28fc1f2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60416/0 (socket says 172.21.15.62:60416) 2024-03-23T11:32:20.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.689+0000 7f2900c00640 1 -- 172.21.15.62:0/2917273828 learned_addr learned my addr 172.21.15.62:0/2917273828 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:20.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.689+0000 7f28fbe00640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28e00052a0 0x7f28fc0ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.689+0000 7f28fb400640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28fc1f57c0 0x7f28fc0cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.689+0000 7f2900c00640 1 -- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28fc1f57c0 msgr2=0x7f28fc0cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.689+0000 7f2900c00640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28fc1f57c0 0x7f28fc0cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.689+0000 7f2900c00640 1 -- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28e00052a0 msgr2=0x7f28fc0ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.689+0000 7f2900c00640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28e00052a0 0x7f28fc0ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.689+0000 7f2900c00640 1 -- 172.21.15.62:0/2917273828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc1f5ef0 con 0x7f28fc1f28e0 2024-03-23T11:32:20.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.690+0000 7f2900c00640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28fc1f28e0 0x7f28fc1f2d60 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f28f400c300 tx=0x7f28f4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:20.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.690+0000 7f28f3e00640 1 -- 172.21.15.62:0/2917273828 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f40049e0 con 0x7f28fc1f28e0 2024-03-23T11:32:20.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.690+0000 7f28f3e00640 1 -- 172.21.15.62:0/2917273828 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f4004b40 con 0x7f28fc1f28e0 2024-03-23T11:32:20.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.690+0000 7f2901600640 1 -- 172.21.15.62:0/2917273828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28fc1f61d0 con 0x7f28fc1f28e0 2024-03-23T11:32:20.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.690+0000 7f28f3e00640 1 -- 172.21.15.62:0/2917273828 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4007070 con 0x7f28fc1f28e0 2024-03-23T11:32:20.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.690+0000 7f2901600640 1 -- 172.21.15.62:0/2917273828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28fc1f23c0 con 0x7f28fc1f28e0 2024-03-23T11:32:20.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.691+0000 7f2901600640 1 -- 172.21.15.62:0/2917273828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28fc1f25e0 con 0x7f28fc1f28e0 2024-03-23T11:32:20.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.692+0000 7f28f3e00640 1 -- 172.21.15.62:0/2917273828 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f28f4032070 con 0x7f28fc1f28e0 2024-03-23T11:32:20.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.693+0000 7f28f3e00640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28dc06a7a0 0x7f28dc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:20.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.693+0000 7f28f3e00640 1 -- 172.21.15.62:0/2917273828 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f28f403c070 con 0x7f28fc1f28e0 2024-03-23T11:32:20.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.694+0000 7f28fbe00640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28dc06a7a0 0x7f28dc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:20.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.695+0000 7f28fbe00640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28dc06a7a0 0x7f28dc06cc60 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f28ec006fd0 tx=0x7f28ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:20.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.698+0000 7f28f3e00640 1 -- 172.21.15.62:0/2917273828 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f28f4075520 con 0x7f28fc1f28e0 2024-03-23T11:32:20.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.895+0000 7f2901600640 1 -- 172.21.15.62:0/2917273828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f28fc066bb0 con 0x7f28fc1f28e0 2024-03-23T11:32:20.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.896+0000 7f28f3e00640 1 -- 172.21.15.62:0/2917273828 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) v1 ==== 100+0+11 (secure 0 0 0) 0x7f28f40450a0 con 0x7f28fc1f28e0 2024-03-23T11:32:20.885 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:32:20.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.898+0000 7f2901600640 1 -- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28dc06a7a0 msgr2=0x7f28dc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.898+0000 7f2901600640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28dc06a7a0 0x7f28dc06cc60 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f28ec006fd0 tx=0x7f28ec008040 comp rx=0 tx=0).stop 2024-03-23T11:32:20.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.898+0000 7f2901600640 1 -- 172.21.15.62:0/2917273828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28fc1f28e0 msgr2=0x7f28fc1f2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:20.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.898+0000 7f2901600640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28fc1f28e0 0x7f28fc1f2d60 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f28f400c300 tx=0x7f28f4001a00 comp rx=0 tx=0).stop 2024-03-23T11:32:20.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.898+0000 7f2901600640 1 -- 172.21.15.62:0/2917273828 shutdown_connections 2024-03-23T11:32:20.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.898+0000 7f2901600640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f28dc06a7a0 0x7f28dc06cc60 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.898+0000 7f2901600640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28fc1f28e0 0x7f28fc1f2d60 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.898+0000 7f2901600640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28fc1f57c0 0x7f28fc0cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.898+0000 7f2901600640 1 --2- 172.21.15.62:0/2917273828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28e00052a0 0x7f28fc0ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:20.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.898+0000 7f2901600640 1 -- 172.21.15.62:0/2917273828 >> 172.21.15.62:0/2917273828 conn(0x7f28fc089f60 msgr2=0x7f28fc15f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:20.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.898+0000 7f2901600640 1 -- 172.21.15.62:0/2917273828 shutdown_connections 2024-03-23T11:32:20.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:20.899+0000 7f2901600640 1 -- 172.21.15.62:0/2917273828 wait complete. 2024-03-23T11:32:20.907 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:32:21.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.137+0000 7f2daf200640 1 Processor -- start 2024-03-23T11:32:21.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.137+0000 7f2daf200640 1 -- start start 2024-03-23T11:32:21.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.139+0000 7f2daf200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db0069c60 0x7f2db006a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.139+0000 7f2daf200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2db0143450 0x7f2db01438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.139+0000 7f2daf200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db015bb10 con 0x7f2db00caff0 2024-03-23T11:32:21.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.139+0000 7f2daf200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db00ce1a0 con 0x7f2db0143450 2024-03-23T11:32:21.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.139+0000 7f2daf200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00cbcb0 con 0x7f2db0069c60 2024-03-23T11:32:21.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.139+0000 7f2da7e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2db00caff0 0x7f2db00cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58044/0 (socket says 172.21.15.62:58044) 2024-03-23T11:32:21.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.139+0000 7f2da7e00640 1 -- 172.21.15.62:0/335208770 learned_addr learned my addr 172.21.15.62:0/335208770 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:21.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.139+0000 7f2dade00640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db0069c60 0x7f2db006a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.139+0000 7f2dae800640 1 --2- 172.21.15.62:0/335208770 >> v2:172.21.15.120:3300/0 conn(0x7f2db0143450 0x7f2db01438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824937109 0 0) 0x7f2db015bb10 con 0x7f2db00caff0 2024-03-23T11:32:21.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d94003520 con 0x7f2db00caff0 2024-03-23T11:32:21.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3613669557 0 0) 0x7f2d94003520 con 0x7f2db00caff0 2024-03-23T11:32:21.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db0069c60 msgr2=0x7f2db006a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:21.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db0069c60 0x7f2db006a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 >> v2:172.21.15.120:3300/0 conn(0x7f2db0143450 msgr2=0x7f2db01438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:21.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 --2- 172.21.15.62:0/335208770 >> v2:172.21.15.120:3300/0 conn(0x7f2db0143450 0x7f2db01438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db006a580 con 0x7f2db00caff0 2024-03-23T11:32:21.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f2d98001670 con 0x7f2db00caff0 2024-03-23T11:32:21.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 >> v1:172.21.15.62:6789/0 conn(0x7f2db00caff0 legacy=0x7f2db00cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:21.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d940039a0 0x7f2d94003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d940047c0 0x7f2d94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db0069c60 0x7f2d94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00ce1a0 con 0x7f2d940039a0 2024-03-23T11:32:21.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00cbcb0 con 0x7f2d940047c0 2024-03-23T11:32:21.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.140+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9400b9b0 con 0x7f2db0069c60 2024-03-23T11:32:21.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.141+0000 7f2dade00640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d940039a0 0x7f2d94003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.141+0000 7f2da7e00640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d940047c0 0x7f2d94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.141+0000 7f2dae800640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db0069c60 0x7f2d94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.141+0000 7f2dade00640 1 -- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db0069c60 msgr2=0x7f2d94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:21.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.141+0000 7f2dade00640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db0069c60 0x7f2d94005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.141+0000 7f2dade00640 1 -- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d940047c0 msgr2=0x7f2d94004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:21.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.141+0000 7f2dade00640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d940047c0 0x7f2d94004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.141+0000 7f2dade00640 1 -- 172.21.15.62:0/335208770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d94003520 con 0x7f2d940039a0 2024-03-23T11:32:21.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.141+0000 7f2dae800640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db0069c60 0x7f2d94005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.142+0000 7f2dade00640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d940039a0 0x7f2d94003dc0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009ac0 tx=0x7f2d940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d669cb801dbd4ff server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.142+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da002c6b0 con 0x7f2d940039a0 2024-03-23T11:32:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.142+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da002c810 con 0x7f2d940039a0 2024-03-23T11:32:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.142+0000 7f2dad400640 1 -- 172.21.15.62:0/335208770 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da002cba0 con 0x7f2d940039a0 2024-03-23T11:32:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.143+0000 7f2daf200640 1 -- 172.21.15.62:0/335208770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d940039a0 msgr2=0x7f2d94003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.143+0000 7f2daf200640 1 --2- 172.21.15.62:0/335208770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d940039a0 0x7f2d94003dc0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009ac0 tx=0x7f2d940057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.143+0000 7f2dae800640 1 -- 172.21.15.62:0/335208770 reap_dead start 2024-03-23T11:32:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.143+0000 7f2daf200640 1 -- 172.21.15.62:0/335208770 shutdown_connections 2024-03-23T11:32:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.143+0000 7f2daf200640 1 -- 172.21.15.62:0/335208770 >> 172.21.15.62:0/335208770 conn(0x7f2db00b0090 msgr2=0x7f2db00b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.143+0000 7f2daf200640 1 -- 172.21.15.62:0/335208770 shutdown_connections 2024-03-23T11:32:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.143+0000 7f2daf200640 1 -- 172.21.15.62:0/335208770 wait complete. 2024-03-23T11:32:21.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.145+0000 7f2daf200640 1 Processor -- start 2024-03-23T11:32:21.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.145+0000 7f2daf200640 1 -- start start 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.146+0000 7f2daf200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d940039a0 0x7f2db01414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.146+0000 7f2daf200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db00caae0 0x7f2db01f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.146+0000 7f2daf200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db00c9c50 0x7f2db00b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.146+0000 7f2daf200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db013fc40 con 0x7f2db00caae0 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.146+0000 7f2daf200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db01f1ad0 con 0x7f2d940039a0 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.146+0000 7f2daf200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db01f1db0 con 0x7f2db00c9c50 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.147+0000 7f2da7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db00caae0 0x7f2db01f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.147+0000 7f2da7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db00caae0 0x7f2db01f25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60438/0 (socket says 172.21.15.62:60438) 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.147+0000 7f2da7e00640 1 -- 172.21.15.62:0/1890696537 learned_addr learned my addr 172.21.15.62:0/1890696537 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.147+0000 7f2dade00640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d940039a0 0x7f2db01414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.147+0000 7f2dae800640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db00c9c50 0x7f2db00b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.147+0000 7f2da7e00640 1 -- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db00c9c50 msgr2=0x7f2db00b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:21.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.147+0000 7f2da7e00640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db00c9c50 0x7f2db00b2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.147+0000 7f2da7e00640 1 -- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d940039a0 msgr2=0x7f2db01414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:21.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.147+0000 7f2da7e00640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d940039a0 0x7f2db01414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.147+0000 7f2da7e00640 1 -- 172.21.15.62:0/1890696537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db01f2090 con 0x7f2db00caae0 2024-03-23T11:32:21.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.147+0000 7f2da7e00640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db00caae0 0x7f2db01f25b0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800e830 tx=0x7f2d9800ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:21.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.148+0000 7f2da6000640 1 -- 172.21.15.62:0/1890696537 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9800ee20 con 0x7f2db00caae0 2024-03-23T11:32:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.148+0000 7f2da6000640 1 -- 172.21.15.62:0/1890696537 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9800c8d0 con 0x7f2db00caae0 2024-03-23T11:32:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.148+0000 7f2da6000640 1 -- 172.21.15.62:0/1890696537 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9800cc80 con 0x7f2db00caae0 2024-03-23T11:32:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.148+0000 7f2daf200640 1 -- 172.21.15.62:0/1890696537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db00ca0e0 con 0x7f2db00caae0 2024-03-23T11:32:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.148+0000 7f2daf200640 1 -- 172.21.15.62:0/1890696537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db013ff20 con 0x7f2db00caae0 2024-03-23T11:32:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.149+0000 7f2daf200640 1 -- 172.21.15.62:0/1890696537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db01f22d0 con 0x7f2db00caae0 2024-03-23T11:32:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.150+0000 7f2da6000640 1 -- 172.21.15.62:0/1890696537 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2d9801d460 con 0x7f2db00caae0 2024-03-23T11:32:21.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.151+0000 7f2da6000640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d8c06a7a0 0x7f2d8c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.151+0000 7f2da6000640 1 -- 172.21.15.62:0/1890696537 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2d98088cb0 con 0x7f2db00caae0 2024-03-23T11:32:21.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.152+0000 7f2dade00640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d8c06a7a0 0x7f2d8c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.153+0000 7f2dade00640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d8c06a7a0 0x7f2d8c06cc60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f2da0036000 tx=0x7f2da0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:21.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.155+0000 7f2da6000640 1 -- 172.21.15.62:0/1890696537 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2db01f22d0 con 0x7f2db00caae0 2024-03-23T11:32:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.366+0000 7f2daf200640 1 -- 172.21.15.62:0/1890696537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2db01f22d0 con 0x7f2db00caae0 2024-03-23T11:32:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.367+0000 7f2da6000640 1 -- 172.21.15.62:0/1890696537 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v121) v1 ==== 80+0+124 (secure 0 0 0) 0x7f2db01f22d0 con 0x7f2db00caae0 2024-03-23T11:32:21.357 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6837/2585024035 2024-03-24T11:32:04.365655+0000 2024-03-23T11:32:21.357 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6835/2585024035 2024-03-24T11:32:04.365655+0000 2024-03-23T11:32:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:listed 2 entries 2024-03-23T11:32:21.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 -- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d8c06a7a0 msgr2=0x7f2d8c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:21.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d8c06a7a0 0x7f2d8c06cc60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f2da0036000 tx=0x7f2da0030000 comp rx=0 tx=0).stop 2024-03-23T11:32:21.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 -- 172.21.15.62:0/1890696537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db00caae0 msgr2=0x7f2db01f25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db00caae0 0x7f2db01f25b0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800e830 tx=0x7f2d9800ecf0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 -- 172.21.15.62:0/1890696537 shutdown_connections 2024-03-23T11:32:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2d8c06a7a0 0x7f2d8c06cc60 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db00caae0 0x7f2db01f25b0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2db00c9c50 0x7f2db00b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 --2- 172.21.15.62:0/1890696537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d940039a0 0x7f2db01414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 -- 172.21.15.62:0/1890696537 >> 172.21.15.62:0/1890696537 conn(0x7f2db00b0090 msgr2=0x7f2db00cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 -- 172.21.15.62:0/1890696537 shutdown_connections 2024-03-23T11:32:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.370+0000 7f2daf200640 1 -- 172.21.15.62:0/1890696537 wait complete. 2024-03-23T11:32:21.380 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:32:21.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:21.380 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:21.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[80332]: fuse finished with error 0 and tester_r 0 2024-03-23T11:32:21.466 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:32:21.488 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:32:21.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:21.489 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:32:21.580 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:32:21.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.815+0000 7f1d41400640 1 Processor -- start 2024-03-23T11:32:21.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.816+0000 7f1d41400640 1 -- start start 2024-03-23T11:32:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.817+0000 7f1d41400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1d3c15e280 0x7f1d3c15e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.817+0000 7f1d41400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c06cf40 con 0x7f1d3c1613a0 2024-03-23T11:32:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.817+0000 7f1d41400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c1575c0 con 0x7f1d3c1571e0 2024-03-23T11:32:21.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.817+0000 7f1d41400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d3c06c3a0 con 0x7f1d3c15e280 2024-03-23T11:32:21.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.818+0000 7f1d39e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1d3c1613a0 0x7f1d3c06d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58050/0 (socket says 172.21.15.62:58050) 2024-03-23T11:32:21.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.818+0000 7f1d39e00640 1 -- 172.21.15.62:0/479729937 learned_addr learned my addr 172.21.15.62:0/479729937 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:21.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.818+0000 7f1d3b200640 1 --2- 172.21.15.62:0/479729937 >> v2:172.21.15.120:3300/0 conn(0x7f1d3c15e280 0x7f1d3c15e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.818+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199135630 0 0) 0x7f1d3c06cf40 con 0x7f1d3c1613a0 2024-03-23T11:32:21.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.818+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d20003520 con 0x7f1d3c1613a0 2024-03-23T11:32:21.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3554409532 0 0) 0x7f1d20003520 con 0x7f1d3c1613a0 2024-03-23T11:32:21.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 >> v2:172.21.15.120:3300/0 conn(0x7f1d3c15e280 msgr2=0x7f1d3c15e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:21.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 --2- 172.21.15.62:0/479729937 >> v2:172.21.15.120:3300/0 conn(0x7f1d3c15e280 0x7f1d3c15e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 >> v1:172.21.15.120:6789/0 conn(0x7f1d3c1571e0 legacy=0x7f1d3c0ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c0ae040 con 0x7f1d3c1613a0 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f1d2c0019f0 con 0x7f1d3c1613a0 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 >> v1:172.21.15.62:6789/0 conn(0x7f1d3c1613a0 legacy=0x7f1d3c06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d20003970 0x7f1d20003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d200047c0 0x7f1d20004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d20005180 0x7f1d2000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c06c3a0 con 0x7f1d20005180 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2000c120 con 0x7f1d20003970 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2000c2e0 con 0x7f1d200047c0 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.819+0000 7f1d3b200640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d20005180 0x7f1d2000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.820+0000 7f1d3a800640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d20003970 0x7f1d20003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.820+0000 7f1d39e00640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d200047c0 0x7f1d20004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.820+0000 7f1d3b200640 1 -- 172.21.15.62:0/479729937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d200047c0 msgr2=0x7f1d20004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:21.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.820+0000 7f1d3b200640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d200047c0 0x7f1d20004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.820+0000 7f1d3b200640 1 -- 172.21.15.62:0/479729937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d20003970 msgr2=0x7f1d20003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:21.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.820+0000 7f1d3b200640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d20003970 0x7f1d20003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.820+0000 7f1d3b200640 1 -- 172.21.15.62:0/479729937 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d20003520 con 0x7f1d20005180 2024-03-23T11:32:21.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.820+0000 7f1d3a800640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d20003970 0x7f1d20003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:21.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.821+0000 7f1d3b200640 1 -- 172.21.15.62:0/479729937 reap_dead start 2024-03-23T11:32:21.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.821+0000 7f1d3b200640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d20005180 0x7f1d2000b980 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009ac0 tx=0x7f1d200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460d67039b97dff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:21.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.821+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d34004020 con 0x7f1d20005180 2024-03-23T11:32:21.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.821+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d34004180 con 0x7f1d20005180 2024-03-23T11:32:21.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.821+0000 7f1d39400640 1 -- 172.21.15.62:0/479729937 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3402c710 con 0x7f1d20005180 2024-03-23T11:32:21.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.822+0000 7f1d41400640 1 -- 172.21.15.62:0/479729937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d20005180 msgr2=0x7f1d2000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:21.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.822+0000 7f1d41400640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d20005180 0x7f1d2000b980 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009ac0 tx=0x7f1d200057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.822+0000 7f1d41400640 1 -- 172.21.15.62:0/479729937 shutdown_connections 2024-03-23T11:32:21.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.822+0000 7f1d41400640 1 --2- 172.21.15.62:0/479729937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d20005180 0x7f1d2000b980 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.822+0000 7f1d41400640 1 -- 172.21.15.62:0/479729937 >> 172.21.15.62:0/479729937 conn(0x7f1d3c0b02e0 msgr2=0x7f1d3c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:21.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.822+0000 7f1d41400640 1 -- 172.21.15.62:0/479729937 shutdown_connections 2024-03-23T11:32:21.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.822+0000 7f1d41400640 1 -- 172.21.15.62:0/479729937 wait complete. 2024-03-23T11:32:21.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.824+0000 7f1d41400640 1 Processor -- start 2024-03-23T11:32:21.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.824+0000 7f1d41400640 1 -- start start 2024-03-23T11:32:21.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.825+0000 7f1d41400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d20005180 0x7f1d3c157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.825+0000 7f1d41400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d3c1412d0 0x7f1d3c1409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.825+0000 7f1d41400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d3c142930 0x7f1d3c143360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.825+0000 7f1d41400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c158f30 con 0x7f1d3c142930 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.825+0000 7f1d41400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c141890 con 0x7f1d20005180 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.825+0000 7f1d41400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c141b70 con 0x7f1d3c1412d0 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.826+0000 7f1d3b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d3c142930 0x7f1d3c143360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.826+0000 7f1d3b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d3c142930 0x7f1d3c143360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60454/0 (socket says 172.21.15.62:60454) 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.826+0000 7f1d3b200640 1 -- 172.21.15.62:0/171489594 learned_addr learned my addr 172.21.15.62:0/171489594 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.826+0000 7f1d3a800640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d20005180 0x7f1d3c157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.826+0000 7f1d39e00640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d3c1412d0 0x7f1d3c1409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.826+0000 7f1d3b200640 1 -- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d3c1412d0 msgr2=0x7f1d3c1409f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.826+0000 7f1d3b200640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d3c1412d0 0x7f1d3c1409f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.826+0000 7f1d3b200640 1 -- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d20005180 msgr2=0x7f1d3c157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.826+0000 7f1d3b200640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d20005180 0x7f1d3c157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.826+0000 7f1d3b200640 1 -- 172.21.15.62:0/171489594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c141e50 con 0x7f1d3c142930 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.826+0000 7f1d3b200640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d3c142930 0x7f1d3c143360 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f1d3402c380 tx=0x7f1d34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.827+0000 7f1d32a00640 1 -- 172.21.15.62:0/171489594 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3403a070 con 0x7f1d3c142930 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.827+0000 7f1d32a00640 1 -- 172.21.15.62:0/171489594 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d34004500 con 0x7f1d3c142930 2024-03-23T11:32:21.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.827+0000 7f1d32a00640 1 -- 172.21.15.62:0/171489594 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d34003830 con 0x7f1d3c142930 2024-03-23T11:32:21.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.827+0000 7f1d41400640 1 -- 172.21.15.62:0/171489594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c142130 con 0x7f1d3c142930 2024-03-23T11:32:21.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.827+0000 7f1d41400640 1 -- 172.21.15.62:0/171489594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c157e60 con 0x7f1d3c142930 2024-03-23T11:32:21.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.828+0000 7f1d41400640 1 -- 172.21.15.62:0/171489594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3c142650 con 0x7f1d3c142930 2024-03-23T11:32:21.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.829+0000 7f1d32a00640 1 -- 172.21.15.62:0/171489594 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f1d34003990 con 0x7f1d3c142930 2024-03-23T11:32:21.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.830+0000 7f1d32a00640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1d1806a7a0 0x7f1d1806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:21.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.830+0000 7f1d32a00640 1 -- 172.21.15.62:0/171489594 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1d340a9790 con 0x7f1d3c142930 2024-03-23T11:32:21.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.831+0000 7f1d3a800640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1d1806a7a0 0x7f1d1806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:21.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.832+0000 7f1d3a800640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1d1806a7a0 0x7f1d1806cc60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1d24005450 tx=0x7f1d24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:21.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:21.835+0000 7f1d32a00640 1 -- 172.21.15.62:0/171489594 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1d3c142650 con 0x7f1d3c142930 2024-03-23T11:32:22.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.075+0000 7f1d41400640 1 -- 172.21.15.62:0/171489594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d3c142650 con 0x7f1d3c142930 2024-03-23T11:32:22.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.076+0000 7f1d32a00640 1 -- 172.21.15.62:0/171489594 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f1d3c142650 con 0x7f1d3c142930 2024-03-23T11:32:22.066 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:22.066 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":101,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:32:09.464564+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9000":{"gid":9000,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":15,"addr":"172.21.15.62:6835/4129977589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":4129977589},{"type":"v1","addr":"172.21.15.62:6835","nonce":4129977589}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:32:22.067 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 101 2024-03-23T11:32:22.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.079+0000 7f1d41400640 1 -- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1d1806a7a0 msgr2=0x7f1d1806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.079+0000 7f1d41400640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1d1806a7a0 0x7f1d1806cc60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1d24005450 tx=0x7f1d24009040 comp rx=0 tx=0).stop 2024-03-23T11:32:22.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.079+0000 7f1d41400640 1 -- 172.21.15.62:0/171489594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d3c142930 msgr2=0x7f1d3c143360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.079+0000 7f1d41400640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d3c142930 0x7f1d3c143360 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f1d3402c380 tx=0x7f1d34004730 comp rx=0 tx=0).stop 2024-03-23T11:32:22.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.079+0000 7f1d41400640 1 -- 172.21.15.62:0/171489594 shutdown_connections 2024-03-23T11:32:22.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.079+0000 7f1d41400640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1d1806a7a0 0x7f1d1806cc60 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.079+0000 7f1d41400640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d3c142930 0x7f1d3c143360 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.079+0000 7f1d41400640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d3c1412d0 0x7f1d3c1409f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.079+0000 7f1d41400640 1 --2- 172.21.15.62:0/171489594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d20005180 0x7f1d3c157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.079+0000 7f1d41400640 1 -- 172.21.15.62:0/171489594 >> 172.21.15.62:0/171489594 conn(0x7f1d3c0b02e0 msgr2=0x7f1d3c156590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:22.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.080+0000 7f1d41400640 1 -- 172.21.15.62:0/171489594 shutdown_connections 2024-03-23T11:32:22.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.080+0000 7f1d41400640 1 -- 172.21.15.62:0/171489594 wait complete. 2024-03-23T11:32:22.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json tag path /parent mytag123 2024-03-23T11:32:22.332 INFO:teuthology.orchestra.run.smithi062.stdout:{"return_code":0,"scrub_tag":"mytag123","mode":"asynchronous"} 2024-03-23T11:32:22.333 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag123" } 2024-03-23T11:32:22.333 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-03-23T11:32:22.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.437+0000 7f5b5ecf5f40 1 Processor -- start 2024-03-23T11:32:22.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.437+0000 7f5b5ecf5f40 1 -- start start 2024-03-23T11:32:22.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.438+0000 7f5b5ecf5f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55975724fb00 0x5597572daca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.438+0000 7f5b5ecf5f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597572de220 0x5597572ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.438+0000 7f5b5ecf5f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559757250440 0x559757250820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.438+0000 7f5b5ecf5f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55975714f800 con 0x559757250440 2024-03-23T11:32:22.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.438+0000 7f5b5ecf5f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55975714f4a0 con 0x5597572de220 2024-03-23T11:32:22.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.438+0000 7f5b5ecf5f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559757146110 con 0x55975724fb00 2024-03-23T11:32:22.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.439+0000 7f5b5c800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559757250440 0x559757250820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.439+0000 7f5b5c800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559757250440 0x559757250820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60468/0 (socket says 172.21.15.62:60468) 2024-03-23T11:32:22.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.439+0000 7f5b5c800640 1 -- 172.21.15.62:0/1254707792 learned_addr learned my addr 172.21.15.62:0/1254707792 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:22.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.439+0000 7f5b5be00640 1 --2- 172.21.15.62:0/1254707792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55975724fb00 0x5597572daca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.439+0000 7f5b5c800640 1 -- 172.21.15.62:0/1254707792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55975724fb00 msgr2=0x5597572daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:22.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.439+0000 7f5b5c800640 1 --2- 172.21.15.62:0/1254707792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55975724fb00 0x5597572daca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.439+0000 7f5b5c800640 1 -- 172.21.15.62:0/1254707792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597572de220 msgr2=0x5597572ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:22.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.439+0000 7f5b5b400640 1 --2- 172.21.15.62:0/1254707792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597572de220 0x5597572ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.439+0000 7f5b5c800640 1 --2- 172.21.15.62:0/1254707792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597572de220 0x5597572ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.439+0000 7f5b5c800640 1 -- 172.21.15.62:0/1254707792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55975715e340 con 0x559757250440 2024-03-23T11:32:22.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.439+0000 7f5b5c800640 1 --2- 172.21.15.62:0/1254707792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559757250440 0x559757250820 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f5b5401c230 tx=0x7f5b5404d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9192926c154df815 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5a000640 1 -- 172.21.15.62:0/1254707792 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5404b9b0 con 0x559757250440 2024-03-23T11:32:22.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5a000640 1 -- 172.21.15.62:0/1254707792 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54001430 con 0x559757250440 2024-03-23T11:32:22.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5a000640 1 -- 172.21.15.62:0/1254707792 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5405f480 con 0x559757250440 2024-03-23T11:32:22.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/1254707792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559757250440 msgr2=0x559757250820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5ecf5f40 1 --2- 172.21.15.62:0/1254707792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559757250440 0x559757250820 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f5b5401c230 tx=0x7f5b5404d000 comp rx=0 tx=0).stop 2024-03-23T11:32:22.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/1254707792 shutdown_connections 2024-03-23T11:32:22.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5ecf5f40 1 --2- 172.21.15.62:0/1254707792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559757250440 0x559757250820 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5ecf5f40 1 --2- 172.21.15.62:0/1254707792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597572de220 0x5597572ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5ecf5f40 1 --2- 172.21.15.62:0/1254707792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55975724fb00 0x5597572daca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/1254707792 >> 172.21.15.62:0/1254707792 conn(0x5597572056f0 msgr2=0x559757206390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:22.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/1254707792 shutdown_connections 2024-03-23T11:32:22.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.440+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/1254707792 wait complete. 2024-03-23T11:32:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.441+0000 7f5b5ecf5f40 1 Processor -- start 2024-03-23T11:32:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.441+0000 7f5b5ecf5f40 1 -- start start 2024-03-23T11:32:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.442+0000 7f5b5ecf5f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55975724fb00 0x5597572c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.442+0000 7f5b5ecf5f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559757250440 0x5597572c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.442+0000 7f5b5ecf5f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597572de220 0x559757247dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.442+0000 7f5b5ecf5f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597572e03d0 con 0x55975724fb00 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.442+0000 7f5b5ecf5f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597572e0990 con 0x559757250440 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.442+0000 7f5b5ecf5f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x559757248c90 con 0x5597572de220 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.443+0000 7f5b5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55975724fb00 0x5597572c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.443+0000 7f5b5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55975724fb00 0x5597572c8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60474/0 (socket says 172.21.15.62:60474) 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.443+0000 7f5b5be00640 1 -- 172.21.15.62:0/3461549392 learned_addr learned my addr 172.21.15.62:0/3461549392 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.443+0000 7f5b5be00640 1 -- 172.21.15.62:0/3461549392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597572de220 msgr2=0x559757247dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.443+0000 7f5b5be00640 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5597572de220 0x559757247dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.443+0000 7f5b5b400640 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559757250440 0x5597572c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.443+0000 7f5b5be00640 1 -- 172.21.15.62:0/3461549392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559757250440 msgr2=0x5597572c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.443+0000 7f5b5be00640 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559757250440 0x5597572c85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.443+0000 7f5b5be00640 1 -- 172.21.15.62:0/3461549392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b40001e10 con 0x55975724fb00 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.443+0000 7f5b5be00640 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55975724fb00 0x5597572c8e80 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c010560 tx=0x7f5b4c010a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.444+0000 7f5b58c00640 1 -- 172.21.15.62:0/3461549392 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c014850 con 0x55975724fb00 2024-03-23T11:32:22.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.444+0000 7f5b58c00640 1 -- 172.21.15.62:0/3461549392 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4c0149b0 con 0x55975724fb00 2024-03-23T11:32:22.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.444+0000 7f5b58c00640 1 -- 172.21.15.62:0/3461549392 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c014d60 con 0x55975724fb00 2024-03-23T11:32:22.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.444+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/3461549392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559757151780 con 0x55975724fb00 2024-03-23T11:32:22.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.444+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/3461549392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5597571511e0 con 0x55975724fb00 2024-03-23T11:32:22.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.446+0000 7f5b58c00640 1 -- 172.21.15.62:0/3461549392 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f5b4c003040 con 0x55975724fb00 2024-03-23T11:32:22.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.447+0000 7f5b58c00640 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5b3c069180 0x7f5b3c06b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.447+0000 7f5b58c00640 1 -- 172.21.15.62:0/3461549392 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5b4c087e50 con 0x55975724fb00 2024-03-23T11:32:22.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.447+0000 7f5b5ecf5f40 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x5597572487f0 0x5597572e1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.447+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/3461549392 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:1 17.36 17:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121) v8 -- 0x5597572c9440 con 0x5597572487f0 2024-03-23T11:32:22.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.447+0000 7f5b5c800640 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x5597572487f0 0x5597572e1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.447+0000 7f5b5b400640 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5b3c069180 0x7f5b3c06b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.448+0000 7f5b5c800640 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x5597572487f0 0x5597572e1330 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.448+0000 7f5b5c800640 1 -- 172.21.15.62:0/3461549392 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f5b54065020 con 0x5597572487f0 2024-03-23T11:32:22.438 INFO:teuthology.orchestra.run.smithi062.stderr:error getting xattr cephfs_data/100000001f7.00000000/scrub_tag: (61) No data available 2024-03-23T11:32:22.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.448+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/3461549392 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x5597572487f0 msgr2=0x5597572e1330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.448+0000 7f5b5ecf5f40 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x5597572487f0 0x5597572e1330 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.449+0000 7f5b5b400640 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5b3c069180 0x7f5b3c06b640 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5b48005fd0 tx=0x7f5b4800c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.449+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/3461549392 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5b3c069180 msgr2=0x7f5b3c06b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.449+0000 7f5b5ecf5f40 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f5b3c069180 0x7f5b3c06b640 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5b48005fd0 tx=0x7f5b4800c040 comp rx=0 tx=0).stop 2024-03-23T11:32:22.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.449+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/3461549392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55975724fb00 msgr2=0x5597572c8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.449+0000 7f5b5ecf5f40 1 --2- 172.21.15.62:0/3461549392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55975724fb00 0x5597572c8e80 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c010560 tx=0x7f5b4c010a20 comp rx=0 tx=0).stop 2024-03-23T11:32:22.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.449+0000 7f5b5c800640 1 -- 172.21.15.62:0/3461549392 reap_dead start 2024-03-23T11:32:22.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.449+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/3461549392 shutdown_connections 2024-03-23T11:32:22.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.449+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/3461549392 >> 172.21.15.62:0/3461549392 conn(0x5597572056f0 msgr2=0x559757250820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:22.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.449+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/3461549392 shutdown_connections 2024-03-23T11:32:22.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.449+0000 7f5b5ecf5f40 1 -- 172.21.15.62:0/3461549392 wait complete. 2024-03-23T11:32:22.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:32:22.443 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-03-23T11:32:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.556+0000 7ff57ce35f40 1 Processor -- start 2024-03-23T11:32:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.556+0000 7ff57ce35f40 1 -- start start 2024-03-23T11:32:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.557+0000 7ff57ce35f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600035f0 0x5577dcbad180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.557+0000 7ff57ce35f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577dcc56380 0x5577dcc4c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.557+0000 7ff57ce35f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 0x5577dcc4cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.557+0000 7ff57ce35f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577dcac2800 con 0x5577dcc4bc80 2024-03-23T11:32:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.557+0000 7ff57ce35f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577dcac24a0 con 0x5577dcc56380 2024-03-23T11:32:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.557+0000 7ff57ce35f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5577dcab9110 con 0x7ff5600035f0 2024-03-23T11:32:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.558+0000 7ff57a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 0x5577dcc4cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.558+0000 7ff57a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 0x5577dcc4cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60488/0 (socket says 172.21.15.62:60488) 2024-03-23T11:32:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.558+0000 7ff57a200640 1 -- 172.21.15.62:0/2983371427 learned_addr learned my addr 172.21.15.62:0/2983371427 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.558+0000 7ff570e00640 1 --2- 172.21.15.62:0/2983371427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577dcc56380 0x5577dcc4c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.558+0000 7ff57a200640 1 -- 172.21.15.62:0/2983371427 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600035f0 msgr2=0x5577dcbad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.558+0000 7ff57a200640 1 --2- 172.21.15.62:0/2983371427 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600035f0 0x5577dcbad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.558+0000 7ff57a200640 1 -- 172.21.15.62:0/2983371427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577dcc56380 msgr2=0x5577dcc4c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.558+0000 7ff57a200640 1 --2- 172.21.15.62:0/2983371427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577dcc56380 0x5577dcc4c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.558+0000 7ff57a200640 1 -- 172.21.15.62:0/2983371427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577dcad1340 con 0x5577dcc4bc80 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.558+0000 7ff57a200640 1 --2- 172.21.15.62:0/2983371427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 0x5577dcc4cee0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7ff57401c030 tx=0x7ff574002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd1e38d3a1ea8b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff573e00640 1 -- 172.21.15.62:0/2983371427 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574052640 con 0x5577dcc4bc80 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff573e00640 1 -- 172.21.15.62:0/2983371427 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff574052910 con 0x5577dcc4bc80 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff573e00640 1 -- 172.21.15.62:0/2983371427 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574052c80 con 0x5577dcc4bc80 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff57ce35f40 1 -- 172.21.15.62:0/2983371427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 msgr2=0x5577dcc4cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff57ce35f40 1 --2- 172.21.15.62:0/2983371427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 0x5577dcc4cee0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7ff57401c030 tx=0x7ff574002fb0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff57ce35f40 1 -- 172.21.15.62:0/2983371427 shutdown_connections 2024-03-23T11:32:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff57ce35f40 1 --2- 172.21.15.62:0/2983371427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 0x5577dcc4cee0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff57ce35f40 1 --2- 172.21.15.62:0/2983371427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577dcc56380 0x5577dcc4c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff57ce35f40 1 --2- 172.21.15.62:0/2983371427 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600035f0 0x5577dcbad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff57ce35f40 1 -- 172.21.15.62:0/2983371427 >> 172.21.15.62:0/2983371427 conn(0x5577dcbc7ff0 msgr2=0x5577dcc55650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff57ce35f40 1 -- 172.21.15.62:0/2983371427 shutdown_connections 2024-03-23T11:32:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.559+0000 7ff57ce35f40 1 -- 172.21.15.62:0/2983371427 wait complete. 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.560+0000 7ff57ce35f40 1 Processor -- start 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.560+0000 7ff57ce35f40 1 -- start start 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.561+0000 7ff57ce35f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 0x5577dcc30cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.561+0000 7ff57ce35f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600035f0 0x5577dcc50a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.561+0000 7ff57ce35f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577dcc50f40 0x5577dcc32d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.561+0000 7ff57ce35f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577dcc4ccf0 con 0x5577dcc4bc80 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.561+0000 7ff57ce35f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577dcc44f90 con 0x5577dcc50f40 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.561+0000 7ff57ce35f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5577dcc44d40 con 0x7ff5600035f0 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.562+0000 7ff579800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 0x5577dcc30cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.562+0000 7ff579800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 0x5577dcc30cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60494/0 (socket says 172.21.15.62:60494) 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.562+0000 7ff579800640 1 -- 172.21.15.62:0/725919589 learned_addr learned my addr 172.21.15.62:0/725919589 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.562+0000 7ff579800640 1 -- 172.21.15.62:0/725919589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600035f0 msgr2=0x5577dcc50a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.562+0000 7ff579800640 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5600035f0 0x5577dcc50a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.562+0000 7ff579800640 1 -- 172.21.15.62:0/725919589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577dcc50f40 msgr2=0x5577dcc32d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.562+0000 7ff57a200640 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577dcc50f40 0x5577dcc32d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.562+0000 7ff579800640 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577dcc50f40 0x5577dcc32d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.562+0000 7ff579800640 1 -- 172.21.15.62:0/725919589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff560001e10 con 0x5577dcc4bc80 2024-03-23T11:32:22.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.562+0000 7ff579800640 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 0x5577dcc30cf0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00c1b0 tx=0x7ff56c00c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.563+0000 7ff572a00640 1 -- 172.21.15.62:0/725919589 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c00cea0 con 0x5577dcc4bc80 2024-03-23T11:32:22.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.563+0000 7ff572a00640 1 -- 172.21.15.62:0/725919589 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c003670 con 0x5577dcc4bc80 2024-03-23T11:32:22.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.563+0000 7ff572a00640 1 -- 172.21.15.62:0/725919589 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c003a20 con 0x5577dcc4bc80 2024-03-23T11:32:22.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.563+0000 7ff57ce35f40 1 -- 172.21.15.62:0/725919589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5577dcac4780 con 0x5577dcc4bc80 2024-03-23T11:32:22.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.563+0000 7ff57ce35f40 1 -- 172.21.15.62:0/725919589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5577dcac41e0 con 0x5577dcc4bc80 2024-03-23T11:32:22.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.565+0000 7ff572a00640 1 -- 172.21.15.62:0/725919589 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff56c003b80 con 0x5577dcc4bc80 2024-03-23T11:32:22.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.566+0000 7ff572a00640 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff558069180 0x7ff55806b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.566+0000 7ff572a00640 1 -- 172.21.15.62:0/725919589 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff56c087550 con 0x5577dcc4bc80 2024-03-23T11:32:22.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.566+0000 7ff57ce35f40 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5577dcc4e5e0 0x5577dcc4ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.566+0000 7ff57ce35f40 1 -- 172.21.15.62:0/725919589 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:1 17.12 17:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121) v8 -- 0x5577dcc2f660 con 0x5577dcc4e5e0 2024-03-23T11:32:22.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.566+0000 7ff57a200640 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5577dcc4e5e0 0x5577dcc4ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.566+0000 7ff570e00640 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff558069180 0x7ff55806b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.567+0000 7ff57a200640 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5577dcc4e5e0 0x5577dcc4ea10 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.567+0000 7ff57a200640 1 -- 172.21.15.62:0/725919589 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr (12) out=12b] v0'0 uv22 ondisk = 0) v8 ==== 164+0+12 (crc 0 0 0) 0x7ff574052a80 con 0x5577dcc4e5e0 2024-03-23T11:32:22.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.567+0000 7ff570e00640 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff558069180 0x7ff55806b640 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff564005380 tx=0x7ff564005310 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.567+0000 7ff57ce35f40 1 -- 172.21.15.62:0/725919589 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5577dcc4e5e0 msgr2=0x5577dcc4ea10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.567+0000 7ff57ce35f40 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5577dcc4e5e0 0x5577dcc4ea10 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.568+0000 7ff57ce35f40 1 -- 172.21.15.62:0/725919589 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff558069180 msgr2=0x7ff55806b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.568+0000 7ff57ce35f40 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff558069180 0x7ff55806b640 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff564005380 tx=0x7ff564005310 comp rx=0 tx=0).stop 2024-03-23T11:32:22.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.568+0000 7ff57ce35f40 1 -- 172.21.15.62:0/725919589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 msgr2=0x5577dcc30cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.568+0000 7ff57ce35f40 1 --2- 172.21.15.62:0/725919589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5577dcc4bc80 0x5577dcc30cf0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00c1b0 tx=0x7ff56c00c670 comp rx=0 tx=0).stop 2024-03-23T11:32:22.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.568+0000 7ff57a200640 1 -- 172.21.15.62:0/725919589 reap_dead start 2024-03-23T11:32:22.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.568+0000 7ff57ce35f40 1 -- 172.21.15.62:0/725919589 shutdown_connections 2024-03-23T11:32:22.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.568+0000 7ff57ce35f40 1 -- 172.21.15.62:0/725919589 >> 172.21.15.62:0/725919589 conn(0x5577dcbc7ff0 msgr2=0x5577dcc55610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:22.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.568+0000 7ff57ce35f40 1 -- 172.21.15.62:0/725919589 shutdown_connections 2024-03-23T11:32:22.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.568+0000 7ff57ce35f40 1 -- 172.21.15.62:0/725919589 wait complete. 2024-03-23T11:32:22.562 INFO:teuthology.orchestra.run.smithi062.stdout:mytag123 2024-03-23T11:32:22.562 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-03-23T11:32:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.662+0000 7f3cf6c49f40 1 Processor -- start 2024-03-23T11:32:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.662+0000 7f3cf6c49f40 1 -- start start 2024-03-23T11:32:22.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.663+0000 7f3cf6c49f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd80030b0 0x56296fbe7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.663+0000 7f3cf6c49f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56296fbe8730 0x56296fb723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.663+0000 7f3cf6c49f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 0x56296fb728e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.663+0000 7f3cf6c49f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56296fad6c20 con 0x56296fb5b020 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.663+0000 7f3cf6c49f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56296fa72800 con 0x56296fbe8730 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.663+0000 7f3cf6c49f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56296fa724a0 con 0x7f3cd80030b0 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.663+0000 7f3cf4000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 0x56296fb728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.664+0000 7f3cf4000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 0x56296fb728e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60512/0 (socket says 172.21.15.62:60512) 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.664+0000 7f3cf4000640 1 -- 172.21.15.62:0/3554328973 learned_addr learned my addr 172.21.15.62:0/3554328973 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.664+0000 7f3cf2c00640 1 --2- 172.21.15.62:0/3554328973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56296fbe8730 0x56296fb723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.664+0000 7f3cf3600640 1 --2- 172.21.15.62:0/3554328973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd80030b0 0x56296fbe7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.664+0000 7f3cf4000640 1 -- 172.21.15.62:0/3554328973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd80030b0 msgr2=0x56296fbe7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.664+0000 7f3cf4000640 1 --2- 172.21.15.62:0/3554328973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd80030b0 0x56296fbe7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.664+0000 7f3cf4000640 1 -- 172.21.15.62:0/3554328973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56296fbe8730 msgr2=0x56296fb723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.664+0000 7f3cf4000640 1 --2- 172.21.15.62:0/3554328973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56296fbe8730 0x56296fb723a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.664+0000 7f3cf4000640 1 -- 172.21.15.62:0/3554328973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56296fa74780 con 0x56296fb5b020 2024-03-23T11:32:22.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.664+0000 7f3cf4000640 1 --2- 172.21.15.62:0/3554328973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 0x56296fb728e0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f3cec001b80 tx=0x7f3cec00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52ab407f87b8d75 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf1800640 1 -- 172.21.15.62:0/3554328973 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec014070 con 0x56296fb5b020 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf1800640 1 -- 172.21.15.62:0/3554328973 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec008030 con 0x56296fb5b020 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf1800640 1 -- 172.21.15.62:0/3554328973 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec006070 con 0x56296fb5b020 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/3554328973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 msgr2=0x56296fb728e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf6c49f40 1 --2- 172.21.15.62:0/3554328973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 0x56296fb728e0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f3cec001b80 tx=0x7f3cec00b5b0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/3554328973 shutdown_connections 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf6c49f40 1 --2- 172.21.15.62:0/3554328973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 0x56296fb728e0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf6c49f40 1 --2- 172.21.15.62:0/3554328973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56296fbe8730 0x56296fb723a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf6c49f40 1 --2- 172.21.15.62:0/3554328973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd80030b0 0x56296fbe7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/3554328973 >> 172.21.15.62:0/3554328973 conn(0x56296fb286f0 msgr2=0x56296fc02180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/3554328973 shutdown_connections 2024-03-23T11:32:22.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.665+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/3554328973 wait complete. 2024-03-23T11:32:22.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.666+0000 7f3cf6c49f40 1 Processor -- start 2024-03-23T11:32:22.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.666+0000 7f3cf6c49f40 1 -- start start 2024-03-23T11:32:22.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.667+0000 7f3cf6c49f40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 0x56296fbfe5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.667+0000 7f3cf6c49f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56296fbe8730 0x56296fc014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.667+0000 7f3cf6c49f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd80030b0 0x56296fc01a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.667+0000 7f3cf6c49f40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56296fa69110 con 0x56296fb5b020 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.667+0000 7f3cf6c49f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56296fb6f430 con 0x56296fbe8730 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.667+0000 7f3cf6c49f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56296fb6e9b0 con 0x7f3cd80030b0 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf3600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 0x56296fbfe5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf3600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 0x56296fbfe5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60520/0 (socket says 172.21.15.62:60520) 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf3600640 1 -- 172.21.15.62:0/2737511949 learned_addr learned my addr 172.21.15.62:0/2737511949 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf2c00640 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56296fbe8730 0x56296fc014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf4000640 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd80030b0 0x56296fc01a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf3600640 1 -- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd80030b0 msgr2=0x56296fc01a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf3600640 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd80030b0 0x56296fc01a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf3600640 1 -- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56296fbe8730 msgr2=0x56296fc014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf3600640 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56296fbe8730 0x56296fc014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf3600640 1 -- 172.21.15.62:0/2737511949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec00a040 con 0x56296fb5b020 2024-03-23T11:32:22.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf2c00640 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56296fbe8730 0x56296fc014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:22.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.668+0000 7f3cf3600640 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 0x56296fbfe5c0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4011760 tx=0x7f3ce4011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.669+0000 7f3cebe00640 1 -- 172.21.15.62:0/2737511949 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4011e70 con 0x56296fb5b020 2024-03-23T11:32:22.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.669+0000 7f3cebe00640 1 -- 172.21.15.62:0/2737511949 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce4002a70 con 0x56296fb5b020 2024-03-23T11:32:22.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.669+0000 7f3cebe00640 1 -- 172.21.15.62:0/2737511949 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4002be0 con 0x56296fb5b020 2024-03-23T11:32:22.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.669+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/2737511949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56296fa741e0 con 0x56296fb5b020 2024-03-23T11:32:22.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.669+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/2737511949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56296fb6f9d0 con 0x56296fb5b020 2024-03-23T11:32:22.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.671+0000 7f3cebe00640 1 -- 172.21.15.62:0/2737511949 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3ce4003040 con 0x56296fb5b020 2024-03-23T11:32:22.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.672+0000 7f3cebe00640 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cd4069180 0x7f3cd406b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.672+0000 7f3cebe00640 1 -- 172.21.15.62:0/2737511949 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3ce4088210 con 0x56296fb5b020 2024-03-23T11:32:22.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.672+0000 7f3cf6c49f40 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x56296fb723a0 0x56296fc96030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.672+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/2737511949 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:1 17.19 17:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121) v8 -- 0x56296fc96570 con 0x56296fb723a0 2024-03-23T11:32:22.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.673+0000 7f3cf2c00640 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cd4069180 0x7f3cd406b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.673+0000 7f3cf4000640 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x56296fb723a0 0x56296fc96030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.674+0000 7f3cf4000640 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x56296fb723a0 0x56296fc96030 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.674+0000 7f3cf2c00640 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cd4069180 0x7f3cd406b640 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0001eb0 tx=0x7f3ce0050000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.675+0000 7f3cf4000640 1 -- 172.21.15.62:0/2737511949 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (12) out=12b] v0'0 uv23 ondisk = 0) v8 ==== 164+0+12 (crc 0 0 0) 0x7f3cec016070 con 0x56296fb723a0 2024-03-23T11:32:22.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.675+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x56296fb723a0 msgr2=0x56296fc96030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.675+0000 7f3cf6c49f40 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x56296fb723a0 0x56296fc96030 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.675+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cd4069180 msgr2=0x7f3cd406b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.675+0000 7f3cf6c49f40 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cd4069180 0x7f3cd406b640 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0001eb0 tx=0x7f3ce0050000 comp rx=0 tx=0).stop 2024-03-23T11:32:22.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.675+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/2737511949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 msgr2=0x56296fbfe5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.675+0000 7f3cf6c49f40 1 --2- 172.21.15.62:0/2737511949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56296fb5b020 0x56296fbfe5c0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4011760 tx=0x7f3ce4011c20 comp rx=0 tx=0).stop 2024-03-23T11:32:22.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.675+0000 7f3cf4000640 1 -- 172.21.15.62:0/2737511949 reap_dead start 2024-03-23T11:32:22.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.675+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/2737511949 shutdown_connections 2024-03-23T11:32:22.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.675+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/2737511949 >> 172.21.15.62:0/2737511949 conn(0x56296fb286f0 msgr2=0x56296fc02180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:22.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.676+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/2737511949 shutdown_connections 2024-03-23T11:32:22.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.676+0000 7f3cf6c49f40 1 -- 172.21.15.62:0/2737511949 wait complete. 2024-03-23T11:32:22.669 INFO:teuthology.orchestra.run.smithi062.stdout:mytag123 2024-03-23T11:32:22.670 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000004.00000000 scrub_tag 2024-03-23T11:32:22.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.793+0000 7f401e64bf40 1 Processor -- start 2024-03-23T11:32:22.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.793+0000 7f401e64bf40 1 -- start start 2024-03-23T11:32:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.794+0000 7f401e64bf40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52b1b910 0x55bb52b9dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.794+0000 7f401e64bf40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bb52bbfc20 0x55bb52ba0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.794+0000 7f401e64bf40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb52bc0710 0x55bb52ba1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.794+0000 7f401e64bf40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb52a30800 con 0x55bb52b1b910 2024-03-23T11:32:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.794+0000 7f401e64bf40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb52a304a0 con 0x55bb52bc0710 2024-03-23T11:32:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.794+0000 7f401e64bf40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb52a27110 con 0x55bb52bbfc20 2024-03-23T11:32:22.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f401b000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52b1b910 0x55bb52b9dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f401b000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52b1b910 0x55bb52b9dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60536/0 (socket says 172.21.15.62:60536) 2024-03-23T11:32:22.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f401b000640 1 -- 172.21.15.62:0/3330748300 learned_addr learned my addr 172.21.15.62:0/3330748300 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f401ba00640 1 --2- 172.21.15.62:0/3330748300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb52bc0710 0x55bb52ba1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f4012600640 1 --2- 172.21.15.62:0/3330748300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bb52bbfc20 0x55bb52ba0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f401b000640 1 -- 172.21.15.62:0/3330748300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bb52bbfc20 msgr2=0x55bb52ba0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f401b000640 1 --2- 172.21.15.62:0/3330748300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bb52bbfc20 0x55bb52ba0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f401b000640 1 -- 172.21.15.62:0/3330748300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb52bc0710 msgr2=0x55bb52ba1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f401b000640 1 --2- 172.21.15.62:0/3330748300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb52bc0710 0x55bb52ba1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f401b000640 1 -- 172.21.15.62:0/3330748300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c00a1c0 con 0x55bb52b1b910 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f401ba00640 1 --2- 172.21.15.62:0/3330748300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb52bc0710 0x55bb52ba1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.795+0000 7f4012600640 1 --2- 172.21.15.62:0/3330748300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bb52bbfc20 0x55bb52ba0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.796+0000 7f401b000640 1 --2- 172.21.15.62:0/3330748300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52b1b910 0x55bb52b9dfe0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f400c007000 tx=0x7f400c04c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796240a7dd7e6d51 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.796+0000 7f4019c00640 1 -- 172.21.15.62:0/3330748300 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c0528e0 con 0x55bb52b1b910 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.796+0000 7f4019c00640 1 -- 172.21.15.62:0/3330748300 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c052a40 con 0x55bb52b1b910 2024-03-23T11:32:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.796+0000 7f4019c00640 1 -- 172.21.15.62:0/3330748300 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c052db0 con 0x55bb52b1b910 2024-03-23T11:32:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.797+0000 7f401e64bf40 1 -- 172.21.15.62:0/3330748300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52b1b910 msgr2=0x55bb52b9dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.797+0000 7f401e64bf40 1 --2- 172.21.15.62:0/3330748300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52b1b910 0x55bb52b9dfe0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f400c007000 tx=0x7f400c04c000 comp rx=0 tx=0).stop 2024-03-23T11:32:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.797+0000 7f401e64bf40 1 -- 172.21.15.62:0/3330748300 shutdown_connections 2024-03-23T11:32:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.797+0000 7f401e64bf40 1 --2- 172.21.15.62:0/3330748300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb52bc0710 0x55bb52ba1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.797+0000 7f401e64bf40 1 --2- 172.21.15.62:0/3330748300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bb52bbfc20 0x55bb52ba0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.797+0000 7f401e64bf40 1 --2- 172.21.15.62:0/3330748300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52b1b910 0x55bb52b9dfe0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.797+0000 7f401e64bf40 1 -- 172.21.15.62:0/3330748300 >> 172.21.15.62:0/3330748300 conn(0x55bb52b35ff0 msgr2=0x55bb52b16a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.797+0000 7f401e64bf40 1 -- 172.21.15.62:0/3330748300 shutdown_connections 2024-03-23T11:32:22.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.797+0000 7f401e64bf40 1 -- 172.21.15.62:0/3330748300 wait complete. 2024-03-23T11:32:22.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.798+0000 7f401e64bf40 1 Processor -- start 2024-03-23T11:32:22.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.798+0000 7f401e64bf40 1 -- start start 2024-03-23T11:32:22.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.799+0000 7f401e64bf40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb52b1b910 0x55bb52bc92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.799+0000 7f401e64bf40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bb52bbfc20 0x55bb52c577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.799+0000 7f401e64bf40 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52bc0710 0x55bb52bb5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.799+0000 7f401e64bf40 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb52b169d0 con 0x55bb52bc0710 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.799+0000 7f401e64bf40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb52b17420 con 0x55bb52b1b910 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.799+0000 7f401e64bf40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb52b9ed40 con 0x55bb52bbfc20 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.799+0000 7f401ba00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52bc0710 0x55bb52bb5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.799+0000 7f401ba00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52bc0710 0x55bb52bb5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60544/0 (socket says 172.21.15.62:60544) 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.799+0000 7f401ba00640 1 -- 172.21.15.62:0/2373491762 learned_addr learned my addr 172.21.15.62:0/2373491762 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.800+0000 7f4012600640 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bb52bbfc20 0x55bb52c577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.800+0000 7f401b000640 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb52b1b910 0x55bb52bc92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.800+0000 7f401ba00640 1 -- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bb52bbfc20 msgr2=0x55bb52c577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.800+0000 7f401ba00640 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bb52bbfc20 0x55bb52c577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.800+0000 7f401ba00640 1 -- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb52b1b910 msgr2=0x55bb52bc92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.800+0000 7f401ba00640 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb52b1b910 0x55bb52bc92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.800+0000 7f401ba00640 1 -- 172.21.15.62:0/2373491762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb52a594e0 con 0x55bb52bc0710 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.800+0000 7f401ba00640 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52bc0710 0x55bb52bb5b70 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f4014009f20 tx=0x7f40140084c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.801+0000 7f4013e00640 1 -- 172.21.15.62:0/2373491762 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40140143f0 con 0x55bb52bc0710 2024-03-23T11:32:22.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.801+0000 7f4013e00640 1 -- 172.21.15.62:0/2373491762 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4014010040 con 0x55bb52bc0710 2024-03-23T11:32:22.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.801+0000 7f4013e00640 1 -- 172.21.15.62:0/2373491762 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40140147b0 con 0x55bb52bc0710 2024-03-23T11:32:22.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.801+0000 7f401e64bf40 1 -- 172.21.15.62:0/2373491762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bb52a3f340 con 0x55bb52bc0710 2024-03-23T11:32:22.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.801+0000 7f401e64bf40 1 -- 172.21.15.62:0/2373491762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb52a32780 con 0x55bb52bc0710 2024-03-23T11:32:22.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.803+0000 7f4013e00640 1 -- 172.21.15.62:0/2373491762 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4014005020 con 0x55bb52bc0710 2024-03-23T11:32:22.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.804+0000 7f4013e00640 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3ff8069180 0x7f3ff806b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.804+0000 7f4013e00640 1 -- 172.21.15.62:0/2373491762 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4014087540 con 0x55bb52bc0710 2024-03-23T11:32:22.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.804+0000 7f401e64bf40 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55bb52bc8e00 0x55bb52b9f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:22.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.804+0000 7f401e64bf40 1 -- 172.21.15.62:0/2373491762 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:1 17.5 17:a367a3e8:::10000000004.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121) v8 -- 0x55bb52b9e8d0 con 0x55bb52bc8e00 2024-03-23T11:32:22.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.805+0000 7f401b000640 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3ff8069180 0x7f3ff806b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.805+0000 7f4012600640 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55bb52bc8e00 0x55bb52b9f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:22.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.806+0000 7f4012600640 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55bb52bc8e00 0x55bb52b9f3e0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.806+0000 7f401b000640 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3ff8069180 0x7f3ff806b640 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f400c00a190 tx=0x7f400c007030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:22.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.807+0000 7f4012600640 1 -- 172.21.15.62:0/2373491762 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== osd_op_reply(1 10000000004.00000000 [getxattr (12) out=12b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+12 (crc 0 0 0) 0x7f400400e5a0 con 0x55bb52bc8e00 2024-03-23T11:32:22.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.807+0000 7f401e64bf40 1 -- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55bb52bc8e00 msgr2=0x55bb52b9f3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.807+0000 7f401e64bf40 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55bb52bc8e00 0x55bb52b9f3e0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.807+0000 7f401e64bf40 1 -- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3ff8069180 msgr2=0x7f3ff806b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.807+0000 7f401e64bf40 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3ff8069180 0x7f3ff806b640 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f400c00a190 tx=0x7f400c007030 comp rx=0 tx=0).stop 2024-03-23T11:32:22.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.807+0000 7f401e64bf40 1 -- 172.21.15.62:0/2373491762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52bc0710 msgr2=0x55bb52bb5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:22.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.807+0000 7f401e64bf40 1 --2- 172.21.15.62:0/2373491762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb52bc0710 0x55bb52bb5b70 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f4014009f20 tx=0x7f40140084c0 comp rx=0 tx=0).stop 2024-03-23T11:32:22.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.807+0000 7f401ba00640 1 -- 172.21.15.62:0/2373491762 reap_dead start 2024-03-23T11:32:22.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.807+0000 7f401e64bf40 1 -- 172.21.15.62:0/2373491762 shutdown_connections 2024-03-23T11:32:22.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.807+0000 7f401e64bf40 1 -- 172.21.15.62:0/2373491762 >> 172.21.15.62:0/2373491762 conn(0x55bb52b35ff0 msgr2=0x55bb52ba20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:22.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.808+0000 7f401e64bf40 1 -- 172.21.15.62:0/2373491762 shutdown_connections 2024-03-23T11:32:22.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:22.808+0000 7f401e64bf40 1 -- 172.21.15.62:0/2373491762 wait complete. 2024-03-23T11:32:22.802 INFO:teuthology.orchestra.run.smithi062.stdout:mytag123 2024-03-23T11:32:22.802 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-03-23T11:32:23.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.012+0000 7f9bdd600640 1 Processor -- start 2024-03-23T11:32:23.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.012+0000 7f9bdd600640 1 -- start start 2024-03-23T11:32:23.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.014+0000 7f9bdd600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9bd806c300 0x7f9bd80ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.014+0000 7f9bdd600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bd80cafa0 0x7f9bd80cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.014+0000 7f9bdd600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9bd806c6e0 0x7f9bd806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.014+0000 7f9bdd600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd80cbc60 con 0x7f9bd806c6e0 2024-03-23T11:32:23.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.014+0000 7f9bdd600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd80cbf40 con 0x7f9bd806c300 2024-03-23T11:32:23.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.014+0000 7f9bdd600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80cc220 con 0x7f9bd80cafa0 2024-03-23T11:32:23.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.014+0000 7f9bd7400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9bd806c6e0 0x7f9bd806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.014+0000 7f9bd7400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9bd806c6e0 0x7f9bd806cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60560/0 (socket says 172.21.15.62:60560) 2024-03-23T11:32:23.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.014+0000 7f9bd7400640 1 -- 172.21.15.62:0/25889059 learned_addr learned my addr 172.21.15.62:0/25889059 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.015+0000 7f9bd6000640 1 --2- 172.21.15.62:0/25889059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bd80cafa0 0x7f9bd80cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.015+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/25889059 >> v2:172.21.15.120:3300/0 conn(0x7f9bd806c300 0x7f9bd80ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.015+0000 7f9bd7400640 1 -- 172.21.15.62:0/25889059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bd80cafa0 msgr2=0x7f9bd80cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.015+0000 7f9bd7400640 1 --2- 172.21.15.62:0/25889059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bd80cafa0 0x7f9bd80cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.015+0000 7f9bd7400640 1 -- 172.21.15.62:0/25889059 >> v2:172.21.15.120:3300/0 conn(0x7f9bd806c300 msgr2=0x7f9bd80ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.015+0000 7f9bd7400640 1 --2- 172.21.15.62:0/25889059 >> v2:172.21.15.120:3300/0 conn(0x7f9bd806c300 0x7f9bd80ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.015+0000 7f9bd7400640 1 -- 172.21.15.62:0/25889059 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd80cc500 con 0x7f9bd806c6e0 2024-03-23T11:32:23.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.015+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/25889059 >> v2:172.21.15.120:3300/0 conn(0x7f9bd806c300 0x7f9bd80ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:23.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.016+0000 7f9bd7400640 1 --2- 172.21.15.62:0/25889059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd806c6e0 0x7f9bd806cb70 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f9bd806c1b0 tx=0x7f9bd002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c198e57533a089e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.017+0000 7f9bd5600640 1 -- 172.21.15.62:0/25889059 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd002ca80 con 0x7f9bd806c6e0 2024-03-23T11:32:23.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.017+0000 7f9bd5600640 1 -- 172.21.15.62:0/25889059 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd002cbe0 con 0x7f9bd806c6e0 2024-03-23T11:32:23.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.017+0000 7f9bd5600640 1 -- 172.21.15.62:0/25889059 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00343f0 con 0x7f9bd806c6e0 2024-03-23T11:32:23.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.017+0000 7f9bdd600640 1 -- 172.21.15.62:0/25889059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd806c6e0 msgr2=0x7f9bd806cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.017+0000 7f9bdd600640 1 --2- 172.21.15.62:0/25889059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd806c6e0 0x7f9bd806cb70 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f9bd806c1b0 tx=0x7f9bd002bee0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.018+0000 7f9bdd600640 1 -- 172.21.15.62:0/25889059 shutdown_connections 2024-03-23T11:32:23.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.018+0000 7f9bdd600640 1 --2- 172.21.15.62:0/25889059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd806c6e0 0x7f9bd806cb70 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.018+0000 7f9bdd600640 1 --2- 172.21.15.62:0/25889059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bd80cafa0 0x7f9bd80cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.018+0000 7f9bdd600640 1 --2- 172.21.15.62:0/25889059 >> v2:172.21.15.120:3300/0 conn(0x7f9bd806c300 0x7f9bd80ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.018+0000 7f9bdd600640 1 -- 172.21.15.62:0/25889059 >> 172.21.15.62:0/25889059 conn(0x7f9bd80b02e0 msgr2=0x7f9bd80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.018+0000 7f9bdd600640 1 -- 172.21.15.62:0/25889059 shutdown_connections 2024-03-23T11:32:23.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.018+0000 7f9bdd600640 1 -- 172.21.15.62:0/25889059 wait complete. 2024-03-23T11:32:23.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.019+0000 7f9bdd600640 1 Processor -- start 2024-03-23T11:32:23.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.020+0000 7f9bdd600640 1 -- start start 2024-03-23T11:32:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.020+0000 7f9bdd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd806c300 0x7f9bd813cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bdd600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bd806c6e0 0x7f9bd815b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bdd600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bd80cafa0 0x7f9bd813f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bdd600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd813fa80 con 0x7f9bd806c300 2024-03-23T11:32:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bdd600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd815c050 con 0x7f9bd80cafa0 2024-03-23T11:32:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bdd600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd815c330 con 0x7f9bd806c6e0 2024-03-23T11:32:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bd6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd806c300 0x7f9bd813cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bd6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd806c300 0x7f9bd813cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60562/0 (socket says 172.21.15.62:60562) 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bd6a00640 1 -- 172.21.15.62:0/501199697 learned_addr learned my addr 172.21.15.62:0/501199697 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bd6000640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bd806c6e0 0x7f9bd815b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bd7400640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bd80cafa0 0x7f9bd813f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bd6a00640 1 -- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bd806c6e0 msgr2=0x7f9bd815b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bd806c6e0 0x7f9bd815b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bd6a00640 1 -- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bd80cafa0 msgr2=0x7f9bd813f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bd80cafa0 0x7f9bd813f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.021+0000 7f9bd6a00640 1 -- 172.21.15.62:0/501199697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd0009b30 con 0x7f9bd806c300 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.022+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd806c300 0x7f9bd813cb40 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800e810 tx=0x7f9bc800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.022+0000 7f9bcea00640 1 -- 172.21.15.62:0/501199697 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8004030 con 0x7f9bd806c300 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.022+0000 7f9bcea00640 1 -- 172.21.15.62:0/501199697 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc8004190 con 0x7f9bd806c300 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.022+0000 7f9bcea00640 1 -- 172.21.15.62:0/501199697 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8004520 con 0x7f9bd806c300 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.022+0000 7f9bdd600640 1 -- 172.21.15.62:0/501199697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd815c610 con 0x7f9bd806c300 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.022+0000 7f9bdd600640 1 -- 172.21.15.62:0/501199697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd815cbd0 con 0x7f9bd806c300 2024-03-23T11:32:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.025+0000 7f9bcea00640 1 -- 172.21.15.62:0/501199697 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9bc800c6d0 con 0x7f9bd806c300 2024-03-23T11:32:23.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.026+0000 7f9bcea00640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9bb806a7a0 0x7f9bb806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.026+0000 7f9bcea00640 1 -- 172.21.15.62:0/501199697 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9bc80889f0 con 0x7f9bd806c300 2024-03-23T11:32:23.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.027+0000 7f9bd6000640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9bb806a7a0 0x7f9bb806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.027+0000 7f9be022d740 1 Processor -- start 2024-03-23T11:32:23.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.027+0000 7f9be022d740 1 -- start start 2024-03-23T11:32:23.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9be022d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31910 0x55bfadc39b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9be022d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc3a280 0x55bfadc11a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9bd6000640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9bb806a7a0 0x7f9bb806cc60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9bd806e6e0 tx=0x7f9bc4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9be022d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc31cf0 0x55bfadc11f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9be022d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfadbcdcb0 con 0x55bfadc3a280 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9be022d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfadb7c500 con 0x55bfadc31cf0 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9be022d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bfadb79c90 con 0x55bfadc31910 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9bd6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc3a280 0x55bfadc11a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9bd6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc3a280 0x55bfadc11a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60566/0 (socket says 172.21.15.62:60566) 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9bd6000640 1 -- 172.21.15.62:0/4043305307 learned_addr learned my addr 172.21.15.62:0/4043305307 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9bd7400640 1 --2- 172.21.15.62:0/4043305307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc31cf0 0x55bfadc11f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9bd6000640 1 -- 172.21.15.62:0/4043305307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31910 msgr2=0x55bfadc39b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/4043305307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31910 0x55bfadc39b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9bd6000640 1 --2- 172.21.15.62:0/4043305307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31910 0x55bfadc39b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9bd6000640 1 -- 172.21.15.62:0/4043305307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc31cf0 msgr2=0x55bfadc11f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.028+0000 7f9bd6000640 1 --2- 172.21.15.62:0/4043305307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc31cf0 0x55bfadc11f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.029+0000 7f9bd6000640 1 -- 172.21.15.62:0/4043305307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfadb27360 con 0x55bfadc3a280 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.029+0000 7f9bd6000640 1 --2- 172.21.15.62:0/4043305307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc3a280 0x55bfadc11a40 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4011ca0 tx=0x7f9bc4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2d326b4fdf5e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.029+0000 7f9bc3400640 1 -- 172.21.15.62:0/4043305307 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4011cd0 con 0x55bfadc3a280 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.029+0000 7f9bc3400640 1 -- 172.21.15.62:0/4043305307 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc4011e30 con 0x55bfadc3a280 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.029+0000 7f9bc3400640 1 -- 172.21.15.62:0/4043305307 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4021640 con 0x55bfadc3a280 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.030+0000 7f9be022d740 1 -- 172.21.15.62:0/4043305307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc3a280 msgr2=0x55bfadc11a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.030+0000 7f9be022d740 1 --2- 172.21.15.62:0/4043305307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc3a280 0x55bfadc11a40 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4011ca0 tx=0x7f9bc4014000 comp rx=0 tx=0).stop 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.030+0000 7f9be022d740 1 -- 172.21.15.62:0/4043305307 shutdown_connections 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.030+0000 7f9be022d740 1 --2- 172.21.15.62:0/4043305307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc3a280 0x55bfadc11a40 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.030+0000 7f9be022d740 1 --2- 172.21.15.62:0/4043305307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc31cf0 0x55bfadc11f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.030+0000 7f9be022d740 1 --2- 172.21.15.62:0/4043305307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31910 0x55bfadc39b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.030+0000 7f9be022d740 1 -- 172.21.15.62:0/4043305307 >> 172.21.15.62:0/4043305307 conn(0x55bfadbd9b40 msgr2=0x55bfadc57720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.030+0000 7f9be022d740 1 -- 172.21.15.62:0/4043305307 shutdown_connections 2024-03-23T11:32:23.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.030+0000 7f9be022d740 1 -- 172.21.15.62:0/4043305307 wait complete. 2024-03-23T11:32:23.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.031+0000 7f9be022d740 1 -- start start 2024-03-23T11:32:23.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.031+0000 7f9be022d740 1 Processor -- start 2024-03-23T11:32:23.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.032+0000 7f9be022d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc31910 0x55bfadc54fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.032+0000 7f9be022d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc5aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.032+0000 7f9be022d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc3a280 0x55bfadc5b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.032+0000 7f9be022d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfadb75580 con 0x55bfadc31cf0 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.032+0000 7f9be022d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfadb16a50 con 0x55bfadc31910 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.032+0000 7f9be022d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bfadb166f0 con 0x55bfadc3a280 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.032+0000 7f9be022d740 10 client.0 _ll_register_callbacks cb 0x55bfadb65ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.032+0000 7f9bd6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc5aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.032+0000 7f9bd6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc5aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60574/0 (socket says 172.21.15.62:60574) 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.032+0000 7f9bd6000640 1 -- 172.21.15.62:0/1969165611 learned_addr learned my addr 172.21.15.62:0/1969165611 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc31910 0x55bfadc54fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bd7400640 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc3a280 0x55bfadc5b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bd6000640 1 -- 172.21.15.62:0/1969165611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc3a280 msgr2=0x55bfadc5b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bd6000640 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc3a280 0x55bfadc5b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bd6000640 1 -- 172.21.15.62:0/1969165611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc31910 msgr2=0x55bfadc54fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bd6000640 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc31910 0x55bfadc54fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bd6000640 1 -- 172.21.15.62:0/1969165611 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc4011950 con 0x55bfadc31cf0 2024-03-23T11:32:23.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc31910 0x55bfadc54fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bd7400640 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc3a280 0x55bfadc5b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bd6000640 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc5aca0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4023040 tx=0x7f9bc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3da90a2353991b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bb7e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bb7e00640 1 -- 172.21.15.62:0/1969165611 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4009200 con 0x55bfadc31cf0 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9bb7e00640 1 -- 172.21.15.62:0/1969165611 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9bc4031070 con 0x55bfadc31cf0 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.033+0000 7f9be022d740 1 -- 172.21.15.62:0/1969165611 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bfadb9ffe0 con 0x55bfadc31cf0 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.034+0000 7f9bb7e00640 1 -- 172.21.15.62:0/1969165611 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bc4021c80 con 0x55bfadc31cf0 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.034+0000 7f9be022d740 10 client.9492 fetch_fsmap learned FSMap version 101 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.034+0000 7f9be022d740 1 -- 172.21.15.62:0/1969165611 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=2+}) v3 -- 0x55bfadb73c00 con 0x55bfadc31cf0 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.034+0000 7f9bb7e00640 1 -- 172.21.15.62:0/1969165611 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bfadb9ffe0 con 0x55bfadc31cf0 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.034+0000 7f9bb7e00640 1 -- 172.21.15.62:0/1969165611 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 101) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9bc4036480 con 0x55bfadc31cf0 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.034+0000 7f9be022d740 10 client.9492 fetch_fsmap finished waiting for FSMap version 101 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.034+0000 7f9be022d740 10 client.9492 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.a 2024-03-23T11:32:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.034+0000 7f9be022d740 20 client.9492 populate_metadata read hostname 'smithi062' 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.034+0000 7f9be022d740 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55bfadc3b3a0 0x55bfadc3b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.034+0000 7f9be022d740 4 client.9492 mds_command: new command op to 9000 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.034+0000 7f9be022d740 1 -- 172.21.15.62:0/1969165611 --> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bfadb68240 con 0x55bfadc3b3a0 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.035+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55bfadc3b3a0 0x55bfadc3b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.035+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55bfadc3b3a0 0x55bfadc3b780 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=186b3202699416ec server_cookie=5d64bc488b792e80 in_seq=0 out_seq=0 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.035+0000 7f9bb7e00640 10 client.9492 ms_handle_connect on v2:172.21.15.62:6834/4129977589 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9bb7e00640 1 -- 172.21.15.62:0/1969165611 <== mds.0 v2:172.21.15.62:6834/4129977589 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55bfadb68240 con 0x55bfadc3b3a0 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9bb7e00640 10 client.9492 handle_command_reply: tid=0 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9be022d740 1 client.9492 shutdown 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9be022d740 1 -- 172.21.15.62:0/1969165611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 msgr2=0x55bfadc5aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9be022d740 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc5aca0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4023040 tx=0x7f9bc40046c0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9be022d740 1 -- 172.21.15.62:0/1969165611 shutdown_connections 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9be022d740 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55bfadc3b3a0 0x55bfadc3b780 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9be022d740 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc5aca0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9be022d740 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc3a280 0x55bfadc5b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9be022d740 1 --2- 172.21.15.62:0/1969165611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc31910 0x55bfadc54fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9bb7e00640 0 client.9492 ms_handle_reset on v2:172.21.15.62:6834/4129977589 2024-03-23T11:32:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9be022d740 1 -- 172.21.15.62:0/1969165611 >> 172.21.15.62:0/1969165611 conn(0x55bfadbd9b40 msgr2=0x55bfadc126f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.036+0000 7f9be022d740 1 -- 172.21.15.62:0/1969165611 shutdown_connections 2024-03-23T11:32:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.037+0000 7f9be022d740 1 -- 172.21.15.62:0/1969165611 wait complete. 2024-03-23T11:32:23.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.037+0000 7f9be022d740 20 client.9492 trim_cache size 0 max 16384 2024-03-23T11:32:23.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.051+0000 7f9be022d740 1 Processor -- start 2024-03-23T11:32:23.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.051+0000 7f9be022d740 1 -- start start 2024-03-23T11:32:23.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.052+0000 7f9be022d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31910 0x55bfadc0bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.052+0000 7f9be022d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31cf0 0x55bfadc54850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.052+0000 7f9be022d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 0x55bfadc2c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.052+0000 7f9be022d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfad872f90 con 0x55bfadc31910 2024-03-23T11:32:23.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.052+0000 7f9be022d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfadc54d90 con 0x55bfadc3a280 2024-03-23T11:32:23.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.052+0000 7f9be022d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bfadc0cfa0 con 0x55bfadc31cf0 2024-03-23T11:32:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.052+0000 7f9bd6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31910 0x55bfadc0bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.052+0000 7f9bd6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31910 0x55bfadc0bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60584/0 (socket says 172.21.15.62:60584) 2024-03-23T11:32:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.052+0000 7f9bd6a00640 1 -- 172.21.15.62:0/3625421627 learned_addr learned my addr 172.21.15.62:0/3625421627 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.052+0000 7f9bd7400640 1 --2- 172.21.15.62:0/3625421627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 0x55bfadc2c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.052+0000 7f9bd6000640 1 --2- 172.21.15.62:0/3625421627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31cf0 0x55bfadc54850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.053+0000 7f9bd6a00640 1 -- 172.21.15.62:0/3625421627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31cf0 msgr2=0x55bfadc54850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.053+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/3625421627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31cf0 0x55bfadc54850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.053+0000 7f9bd6a00640 1 -- 172.21.15.62:0/3625421627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 msgr2=0x55bfadc2c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.053+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/3625421627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 0x55bfadc2c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.053+0000 7f9bd6a00640 1 -- 172.21.15.62:0/3625421627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfadc31590 con 0x55bfadc31910 2024-03-23T11:32:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.053+0000 7f9bd7400640 1 --2- 172.21.15.62:0/3625421627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 0x55bfadc2c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.053+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/3625421627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31910 0x55bfadc0bb40 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80576a0 tx=0x7f9bc8053d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c06204b3c85c80 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.053+0000 7f9bc3e00640 1 -- 172.21.15.62:0/3625421627 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8055ba0 con 0x55bfadc31910 2024-03-23T11:32:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.054+0000 7f9bc3e00640 1 -- 172.21.15.62:0/3625421627 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc8055d40 con 0x55bfadc31910 2024-03-23T11:32:23.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.054+0000 7f9bc3e00640 1 -- 172.21.15.62:0/3625421627 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc80540b0 con 0x55bfadc31910 2024-03-23T11:32:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.054+0000 7f9be022d740 1 -- 172.21.15.62:0/3625421627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31910 msgr2=0x55bfadc0bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.054+0000 7f9be022d740 1 --2- 172.21.15.62:0/3625421627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31910 0x55bfadc0bb40 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80576a0 tx=0x7f9bc8053d80 comp rx=0 tx=0).stop 2024-03-23T11:32:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.054+0000 7f9be022d740 1 -- 172.21.15.62:0/3625421627 shutdown_connections 2024-03-23T11:32:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.054+0000 7f9be022d740 1 --2- 172.21.15.62:0/3625421627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 0x55bfadc2c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.054+0000 7f9be022d740 1 --2- 172.21.15.62:0/3625421627 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31cf0 0x55bfadc54850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.054+0000 7f9be022d740 1 --2- 172.21.15.62:0/3625421627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31910 0x55bfadc0bb40 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.054+0000 7f9be022d740 1 -- 172.21.15.62:0/3625421627 >> 172.21.15.62:0/3625421627 conn(0x55bfadbd9b40 msgr2=0x55bfadc3a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.054+0000 7f9be022d740 1 -- 172.21.15.62:0/3625421627 shutdown_connections 2024-03-23T11:32:23.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.054+0000 7f9be022d740 1 -- 172.21.15.62:0/3625421627 wait complete. 2024-03-23T11:32:23.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.055+0000 7f9be022d740 1 -- start start 2024-03-23T11:32:23.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.055+0000 7f9be022d740 1 Processor -- start 2024-03-23T11:32:23.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.056+0000 7f9be022d740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31910 0x55bfadc59390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.056+0000 7f9be022d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc69210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.056+0000 7f9be022d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 0x55bfadc69750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.056+0000 7f9be022d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfadcbdb20 con 0x55bfadc31cf0 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.056+0000 7f9be022d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfadcbcf70 con 0x55bfadc3a280 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.056+0000 7f9be022d740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bfadc2c730 con 0x55bfadc31910 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.056+0000 7f9be022d740 10 client.0 _ll_register_callbacks cb 0x7f9bc40096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc69210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc69210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60600/0 (socket says 172.21.15.62:60600) 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd6000640 1 -- 172.21.15.62:0/2733616380 learned_addr learned my addr 172.21.15.62:0/2733616380 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd7400640 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 0x55bfadc69750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd6000640 1 -- 172.21.15.62:0/2733616380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31910 msgr2=0x55bfadc59390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31910 0x55bfadc59390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd6000640 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31910 0x55bfadc59390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd6000640 1 -- 172.21.15.62:0/2733616380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 msgr2=0x55bfadc69750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd6000640 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 0x55bfadc69750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd6000640 1 -- 172.21.15.62:0/2733616380 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd815cbd0 con 0x55bfadc31cf0 2024-03-23T11:32:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd7400640 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 0x55bfadc69750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bd6000640 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc69210 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40116a0 tx=0x7f9bc4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfd231aea256361 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.057+0000 7f9bb7e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:32:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9bb7e00640 1 -- 172.21.15.62:0/2733616380 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4036990 con 0x55bfadc31cf0 2024-03-23T11:32:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9be022d740 1 -- 172.21.15.62:0/2733616380 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9bc4011e30 con 0x55bfadc31cf0 2024-03-23T11:32:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9bb7e00640 1 -- 172.21.15.62:0/2733616380 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9bc4031040 con 0x55bfadc31cf0 2024-03-23T11:32:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9bb7e00640 1 -- 172.21.15.62:0/2733616380 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bc4006d30 con 0x55bfadc31cf0 2024-03-23T11:32:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9be022d740 10 client.9498 fetch_fsmap learned FSMap version 101 2024-03-23T11:32:23.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9be022d740 1 -- 172.21.15.62:0/2733616380 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=2+}) v3 -- 0x55bfadb15c60 con 0x55bfadc31cf0 2024-03-23T11:32:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9bb7e00640 1 -- 172.21.15.62:0/2733616380 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc4011e30 con 0x55bfadc31cf0 2024-03-23T11:32:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9bb7e00640 1 -- 172.21.15.62:0/2733616380 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 101) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9bc4036bd0 con 0x55bfadc31cf0 2024-03-23T11:32:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9be022d740 10 client.9498 fetch_fsmap finished waiting for FSMap version 101 2024-03-23T11:32:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9be022d740 10 client.9498 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.a 2024-03-23T11:32:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9be022d740 20 client.9498 populate_metadata read hostname 'smithi062' 2024-03-23T11:32:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9be022d740 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55bfadca2ee0 0x55bfadc4d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9be022d740 4 client.9498 mds_command: new command op to 9000 tid=0[{"prefix": "flush journal"}] 2024-03-23T11:32:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.058+0000 7f9be022d740 1 -- 172.21.15.62:0/2733616380 --> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bfad8694e0 con 0x55bfadca2ee0 2024-03-23T11:32:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.059+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55bfadca2ee0 0x55bfadc4d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.059+0000 7f9bd6a00640 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55bfadca2ee0 0x55bfadc4d0b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dac0de51ed9db86 server_cookie=991490e9c61c0a35 in_seq=0 out_seq=0 2024-03-23T11:32:23.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.059+0000 7f9bb7e00640 10 client.9498 ms_handle_connect on v2:172.21.15.62:6834/4129977589 2024-03-23T11:32:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.077+0000 7f9bb7e00640 1 -- 172.21.15.62:0/2733616380 <== mds.0 v2:172.21.15.62:6834/4129977589 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bfad8694e0 con 0x55bfadca2ee0 2024-03-23T11:32:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.077+0000 7f9bb7e00640 10 client.9498 handle_command_reply: tid=0 2024-03-23T11:32:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.077+0000 7f9be022d740 1 client.9498 shutdown 2024-03-23T11:32:23.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9be022d740 1 -- 172.21.15.62:0/2733616380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 msgr2=0x55bfadc69210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9be022d740 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc69210 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40116a0 tx=0x7f9bc4017000 comp rx=0 tx=0).stop 2024-03-23T11:32:23.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9be022d740 1 -- 172.21.15.62:0/2733616380 shutdown_connections 2024-03-23T11:32:23.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9be022d740 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.62:6834/4129977589,v1:172.21.15.62:6835/4129977589] conn(0x55bfadca2ee0 0x55bfadc4d0b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9be022d740 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bfadc31cf0 0x55bfadc69210 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9be022d740 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfadc3a280 0x55bfadc69750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9be022d740 1 --2- 172.21.15.62:0/2733616380 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55bfadc31910 0x55bfadc59390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9bb7e00640 0 client.9498 ms_handle_reset on v2:172.21.15.62:6834/4129977589 2024-03-23T11:32:23.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9be022d740 1 -- 172.21.15.62:0/2733616380 >> 172.21.15.62:0/2733616380 conn(0x55bfadbd9b40 msgr2=0x55bfadc0e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9be022d740 1 -- 172.21.15.62:0/2733616380 shutdown_connections 2024-03-23T11:32:23.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9be022d740 1 -- 172.21.15.62:0/2733616380 wait complete. 2024-03-23T11:32:23.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.078+0000 7f9be022d740 20 client.9498 trim_cache size 0 max 16384 2024-03-23T11:32:23.069 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:23.069 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "", 2024-03-23T11:32:23.069 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0 2024-03-23T11:32:23.069 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:23.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 -- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9bb806a7a0 msgr2=0x7f9bb806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9bb806a7a0 0x7f9bb806cc60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9bd806e6e0 tx=0x7f9bc4008040 comp rx=0 tx=0).stop 2024-03-23T11:32:23.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 -- 172.21.15.62:0/501199697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd806c300 msgr2=0x7f9bd813cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd806c300 0x7f9bd813cb40 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800e810 tx=0x7f9bc800ecd0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 -- 172.21.15.62:0/501199697 shutdown_connections 2024-03-23T11:32:23.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9bb806a7a0 0x7f9bb806cc60 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9bd80cafa0 0x7f9bd813f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9bd806c6e0 0x7f9bd815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 --2- 172.21.15.62:0/501199697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd806c300 0x7f9bd813cb40 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 -- 172.21.15.62:0/501199697 >> 172.21.15.62:0/501199697 conn(0x7f9bd80b02e0 msgr2=0x7f9bd806d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 -- 172.21.15.62:0/501199697 shutdown_connections 2024-03-23T11:32:23.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.079+0000 7f9bdd600640 1 -- 172.21.15.62:0/501199697 wait complete. 2024-03-23T11:32:23.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:32:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.294+0000 7f9dfe000640 1 Processor -- start 2024-03-23T11:32:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.295+0000 7f9dfe000640 1 -- start start 2024-03-23T11:32:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.296+0000 7f9dfe000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9df80d13b0 0x7f9df80cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.296+0000 7f9dfe000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df80cd260 0x7f9df80b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.296+0000 7f9dfe000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df80d0810 con 0x7f9df815c960 2024-03-23T11:32:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.296+0000 7f9dfe000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df80cd9d0 con 0x7f9df80d13b0 2024-03-23T11:32:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.296+0000 7f9dfe000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80cdcb0 con 0x7f9df80cd260 2024-03-23T11:32:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.296+0000 7f9dfcc00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9df815c960 0x7f9df80d0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58072/0 (socket says 172.21.15.62:58072) 2024-03-23T11:32:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.296+0000 7f9dfcc00640 1 -- 172.21.15.62:0/1550337494 learned_addr learned my addr 172.21.15.62:0/1550337494 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.297+0000 7f9df7e00640 1 --2- 172.21.15.62:0/1550337494 >> v2:172.21.15.120:3300/0 conn(0x7f9df80d13b0 0x7f9df80cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.297+0000 7f9dfd600640 1 --2- 172.21.15.62:0/1550337494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df80cd260 0x7f9df80b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.297+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983448209 0 0) 0x7f9de8001110 con 0x7f9df815c960 2024-03-23T11:32:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.297+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ddc003520 con 0x7f9df815c960 2024-03-23T11:32:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.297+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 839945423 0 0) 0x7f9ddc003520 con 0x7f9df815c960 2024-03-23T11:32:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.297+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df80cd260 msgr2=0x7f9df80b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.297+0000 7f9df7400640 1 --2- 172.21.15.62:0/1550337494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df80cd260 0x7f9df80b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.297+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 >> v2:172.21.15.120:3300/0 conn(0x7f9df80d13b0 msgr2=0x7f9df80cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.297+0000 7f9df7400640 1 --2- 172.21.15.62:0/1550337494 >> v2:172.21.15.120:3300/0 conn(0x7f9df80d13b0 0x7f9df80cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.297+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df80cdf90 con 0x7f9df815c960 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.297+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9de8001800 con 0x7f9df815c960 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.298+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 >> v1:172.21.15.62:6789/0 conn(0x7f9df815c960 legacy=0x7f9df80d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.298+0000 7f9df7400640 1 --2- 172.21.15.62:0/1550337494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ddc003970 0x7f9ddc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.298+0000 7f9df7400640 1 --2- 172.21.15.62:0/1550337494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df80cd260 0x7f9ddc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.298+0000 7f9df7400640 1 --2- 172.21.15.62:0/1550337494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ddc004d00 0x7f9ddc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.298+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80cd9d0 con 0x7f9ddc004d00 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.298+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80cdcb0 con 0x7f9ddc003970 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.298+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc00ba00 con 0x7f9df80cd260 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.298+0000 7f9df7e00640 1 --2- 172.21.15.62:0/1550337494 >> v2:172.21.15.120:3300/0 conn(0x7f9df80d13b0 0x7f9df80cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.298+0000 7f9dfd600640 1 --2- 172.21.15.62:0/1550337494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ddc004d00 0x7f9ddc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.298+0000 7f9df7e00640 1 --2- 172.21.15.62:0/1550337494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df80cd260 0x7f9ddc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.298+0000 7f9dfd600640 1 -- 172.21.15.62:0/1550337494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df80cd260 msgr2=0x7f9ddc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.299+0000 7f9dfd600640 1 --2- 172.21.15.62:0/1550337494 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df80cd260 0x7f9ddc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.299+0000 7f9dfd600640 1 -- 172.21.15.62:0/1550337494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ddc003970 msgr2=0x7f9ddc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.299+0000 7f9dfd600640 1 --2- 172.21.15.62:0/1550337494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ddc003970 0x7f9ddc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.299+0000 7f9dfd600640 1 -- 172.21.15.62:0/1550337494 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc003520 con 0x7f9ddc004d00 2024-03-23T11:32:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.300+0000 7f9dfd600640 1 --2- 172.21.15.62:0/1550337494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ddc004d00 0x7f9ddc005180 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009fc0 tx=0x7f9ddc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81cb108e6e2dc3b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.300+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df002cc20 con 0x7f9ddc004d00 2024-03-23T11:32:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.300+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df000f040 con 0x7f9ddc004d00 2024-03-23T11:32:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.300+0000 7f9df7400640 1 -- 172.21.15.62:0/1550337494 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0034440 con 0x7f9ddc004d00 2024-03-23T11:32:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.301+0000 7f9dfe000640 1 -- 172.21.15.62:0/1550337494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ddc004d00 msgr2=0x7f9ddc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.301+0000 7f9dfe000640 1 --2- 172.21.15.62:0/1550337494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ddc004d00 0x7f9ddc005180 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009fc0 tx=0x7f9ddc0057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.301+0000 7f9dfd600640 1 -- 172.21.15.62:0/1550337494 reap_dead start 2024-03-23T11:32:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.301+0000 7f9dfe000640 1 -- 172.21.15.62:0/1550337494 shutdown_connections 2024-03-23T11:32:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.301+0000 7f9dfe000640 1 -- 172.21.15.62:0/1550337494 >> 172.21.15.62:0/1550337494 conn(0x7f9df8089f60 msgr2=0x7f9df80d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.301+0000 7f9dfe000640 1 -- 172.21.15.62:0/1550337494 shutdown_connections 2024-03-23T11:32:23.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.301+0000 7f9dfe000640 1 -- 172.21.15.62:0/1550337494 wait complete. 2024-03-23T11:32:23.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.303+0000 7f9dfe000640 1 Processor -- start 2024-03-23T11:32:23.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.303+0000 7f9dfe000640 1 -- start start 2024-03-23T11:32:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.304+0000 7f9dfe000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9df815f900 0x7f9df80b5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.304+0000 7f9dfe000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df815fce0 0x7f9df80b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.304+0000 7f9dfe000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df814bf90 0x7f9df814c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.304+0000 7f9dfe000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80b7080 con 0x7f9df814bf90 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.304+0000 7f9dfe000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81594b0 con 0x7f9df815f900 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.304+0000 7f9dfe000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80b6470 con 0x7f9df815fce0 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.304+0000 7f9dfd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df814bf90 0x7f9df814c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.304+0000 7f9dfd600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df814bf90 0x7f9df814c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60626/0 (socket says 172.21.15.62:60626) 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.304+0000 7f9dfd600640 1 -- 172.21.15.62:0/4025341848 learned_addr learned my addr 172.21.15.62:0/4025341848 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.305+0000 7f9dfcc00640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9df815f900 0x7f9df80b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.305+0000 7f9dfd600640 1 -- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df815fce0 msgr2=0x7f9df80b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.305+0000 7f9df7e00640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df815fce0 0x7f9df80b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.305+0000 7f9dfd600640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df815fce0 0x7f9df80b5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.305+0000 7f9dfd600640 1 -- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9df815f900 msgr2=0x7f9df80b5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.305+0000 7f9dfd600640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9df815f900 0x7f9df80b5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.305+0000 7f9dfd600640 1 -- 172.21.15.62:0/4025341848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df80b6750 con 0x7f9df814bf90 2024-03-23T11:32:23.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.305+0000 7f9df7e00640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df815fce0 0x7f9df80b5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.305+0000 7f9dfd600640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df814bf90 0x7f9df814c410 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f9df0002230 tx=0x7f9df0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.306+0000 7f9df4c00640 1 -- 172.21.15.62:0/4025341848 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0043070 con 0x7f9df814bf90 2024-03-23T11:32:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.306+0000 7f9df4c00640 1 -- 172.21.15.62:0/4025341848 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df002de00 con 0x7f9df814bf90 2024-03-23T11:32:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.306+0000 7f9df4c00640 1 -- 172.21.15.62:0/4025341848 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0034d10 con 0x7f9df814bf90 2024-03-23T11:32:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.306+0000 7f9dfe000640 1 -- 172.21.15.62:0/4025341848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df80b6a30 con 0x7f9df814bf90 2024-03-23T11:32:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.306+0000 7f9dfe000640 1 -- 172.21.15.62:0/4025341848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df814c950 con 0x7f9df814bf90 2024-03-23T11:32:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.307+0000 7f9dfe000640 1 -- 172.21.15.62:0/4025341848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df8066bb0 con 0x7f9df814bf90 2024-03-23T11:32:23.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.308+0000 7f9df4c00640 1 -- 172.21.15.62:0/4025341848 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9df003e440 con 0x7f9df814bf90 2024-03-23T11:32:23.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.309+0000 7f9df4c00640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9dd406a7a0 0x7f9dd406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.309+0000 7f9df4c00640 1 -- 172.21.15.62:0/4025341848 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9df00aa9a0 con 0x7f9df814bf90 2024-03-23T11:32:23.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.310+0000 7f9dfcc00640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9dd406a7a0 0x7f9dd406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.311+0000 7f9dfcc00640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9dd406a7a0 0x7f9dd406cc60 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9de8005ad0 tx=0x7f9de8001600 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.313+0000 7f9df4c00640 1 -- 172.21.15.62:0/4025341848 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9df0075d20 con 0x7f9df814bf90 2024-03-23T11:32:23.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.550+0000 7f9dfe000640 1 -- 172.21.15.62:0/4025341848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9df8066bb0 con 0x7f9df814bf90 2024-03-23T11:32:23.541 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:32:23.552+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:32:23.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.568+0000 7f9df4c00640 1 -- 172.21.15.62:0/4025341848 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v102) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9df0039090 con 0x7f9df814bf90 2024-03-23T11:32:23.558 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:32:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.572+0000 7f9dfe000640 1 -- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9dd406a7a0 msgr2=0x7f9dd406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.572+0000 7f9dfe000640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9dd406a7a0 0x7f9dd406cc60 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9de8005ad0 tx=0x7f9de8001600 comp rx=0 tx=0).stop 2024-03-23T11:32:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.572+0000 7f9dfe000640 1 -- 172.21.15.62:0/4025341848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df814bf90 msgr2=0x7f9df814c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.572+0000 7f9dfe000640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df814bf90 0x7f9df814c410 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f9df0002230 tx=0x7f9df0009750 comp rx=0 tx=0).stop 2024-03-23T11:32:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.573+0000 7f9dfe000640 1 -- 172.21.15.62:0/4025341848 shutdown_connections 2024-03-23T11:32:23.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.573+0000 7f9dfe000640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9dd406a7a0 0x7f9dd406cc60 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.573+0000 7f9dfe000640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df814bf90 0x7f9df814c410 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.573+0000 7f9dfe000640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9df815fce0 0x7f9df80b5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.573+0000 7f9dfe000640 1 --2- 172.21.15.62:0/4025341848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9df815f900 0x7f9df80b5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.573+0000 7f9dfe000640 1 -- 172.21.15.62:0/4025341848 >> 172.21.15.62:0/4025341848 conn(0x7f9df8089f60 msgr2=0x7f9df806bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.573+0000 7f9dfe000640 1 -- 172.21.15.62:0/4025341848 shutdown_connections 2024-03-23T11:32:23.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.573+0000 7f9dfe000640 1 -- 172.21.15.62:0/4025341848 wait complete. 2024-03-23T11:32:23.567 INFO:tasks.ceph.mds.a.smithi062.stderr: -2301> 2024-03-23T11:32:09.523+0000 7f045d600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T11:32:23.586 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset --force 2024-03-23T11:32:23.630 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:32:23.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.684+0000 7ff862640280 1 -- start start 2024-03-23T11:32:23.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.684+0000 7ff862640280 1 Processor -- start 2024-03-23T11:32:23.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.685+0000 7ff862640280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561914cc6c00 0x561914d50580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.685+0000 7ff862640280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561914cc7800 0x561914d50b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.685+0000 7ff862640280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561914cc7c00 0x561914d51080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.685+0000 7ff862640280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561914d98a00 con 0x561914cc7c00 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.685+0000 7ff862640280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561914d98780 con 0x561914cc7800 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.685+0000 7ff862640280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561914d98640 con 0x561914cc6c00 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.685+0000 7ff861000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561914cc7c00 0x561914d51080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.685+0000 7ff861000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561914cc7c00 0x561914d51080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60642/0 (socket says 172.21.15.62:60642) 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.685+0000 7ff861000640 1 -- 172.21.15.62:0/1398772438 learned_addr learned my addr 172.21.15.62:0/1398772438 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff85fc00640 1 --2- 172.21.15.62:0/1398772438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561914cc7800 0x561914d50b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff860600640 1 --2- 172.21.15.62:0/1398772438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561914cc6c00 0x561914d50580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff861000640 1 -- 172.21.15.62:0/1398772438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561914cc6c00 msgr2=0x561914d50580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff861000640 1 --2- 172.21.15.62:0/1398772438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561914cc6c00 0x561914d50580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff861000640 1 -- 172.21.15.62:0/1398772438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561914cc7800 msgr2=0x561914d50b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff861000640 1 --2- 172.21.15.62:0/1398772438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561914cc7800 0x561914d50b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff861000640 1 -- 172.21.15.62:0/1398772438 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561914cec9c0 con 0x561914cc7c00 2024-03-23T11:32:23.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff861000640 1 --2- 172.21.15.62:0/1398772438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561914cc7c00 0x561914d51080 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x561915aeacc0 tx=0x561915b8e120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0fe80d4c5b7cfb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff85d400640 1 -- 172.21.15.62:0/1398772438 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561914d0f600 con 0x561914cc7c00 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff85d400640 1 -- 172.21.15.62:0/1398772438 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561914d24180 con 0x561914cc7c00 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff862640280 1 -- 172.21.15.62:0/1398772438 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561914cecb60 con 0x561914cc7c00 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff862640280 1 -- 172.21.15.62:0/1398772438 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x561914cec820 con 0x561914cc7c00 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff862640280 4 waiting for MDS map... 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.686+0000 7ff85d400640 1 -- 172.21.15.62:0/1398772438 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561914d0f1e0 con 0x561914cc7c00 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.687+0000 7ff85d400640 1 -- 172.21.15.62:0/1398772438 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x561914cf2380 con 0x561914cc7c00 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.687+0000 7ff85d400640 1 -- 172.21.15.62:0/1398772438 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 102) v1 ==== 2573+0+0 (secure 0 0 0) 0x561914d18f00 con 0x561914cc7c00 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.687+0000 7ff862640280 4 Got MDS map 102 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.687+0000 7ff862640280 10 main: JournalTool::main 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.687+0000 7ff862640280 4 main: JournalTool: connecting to RADOS... 2024-03-23T11:32:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.688+0000 7ff862640280 1 Processor -- start 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.688+0000 7ff862640280 1 -- start start 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff862640280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561915b9d000 0x561914d51600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff862640280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561915b9d400 0x561914d51b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff862640280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d800 0x561914d52100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff862640280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561914d11e00 con 0x561915b9d800 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff862640280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561914d98780 con 0x561915b9d400 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff862640280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561914d98500 con 0x561915b9d000 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff85fc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d800 0x561914d52100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff85fc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d800 0x561914d52100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60644/0 (socket says 172.21.15.62:60644) 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff85fc00640 1 -- 172.21.15.62:0/2440548417 learned_addr learned my addr 172.21.15.62:0/2440548417 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff85fc00640 1 -- 172.21.15.62:0/2440548417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561915b9d000 msgr2=0x561914d51600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff85fc00640 1 --2- 172.21.15.62:0/2440548417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561915b9d000 0x561914d51600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff85fc00640 1 -- 172.21.15.62:0/2440548417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561915b9d400 msgr2=0x561914d51b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff85fc00640 1 --2- 172.21.15.62:0/2440548417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561915b9d400 0x561914d51b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.689+0000 7ff85fc00640 1 -- 172.21.15.62:0/2440548417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561914cec4e0 con 0x561915b9d800 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff85fc00640 1 --2- 172.21.15.62:0/2440548417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d800 0x561914d52100 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x561915bf6120 tx=0x561915ba8c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141fe99d25c534ad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff85ac00640 1 -- 172.21.15.62:0/2440548417 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561914d0f080 con 0x561915b9d800 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff85ac00640 1 -- 172.21.15.62:0/2440548417 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561914d24900 con 0x561915b9d800 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff85ac00640 1 -- 172.21.15.62:0/2440548417 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561914d0f600 con 0x561915b9d800 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff862640280 1 -- 172.21.15.62:0/2440548417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d800 msgr2=0x561914d52100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff862640280 1 --2- 172.21.15.62:0/2440548417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d800 0x561914d52100 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x561915bf6120 tx=0x561915ba8c60 comp rx=0 tx=0).stop 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff862640280 1 -- 172.21.15.62:0/2440548417 shutdown_connections 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff862640280 1 --2- 172.21.15.62:0/2440548417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d800 0x561914d52100 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff862640280 1 --2- 172.21.15.62:0/2440548417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561915b9d400 0x561914d51b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff862640280 1 --2- 172.21.15.62:0/2440548417 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561915b9d000 0x561914d51600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff862640280 1 -- 172.21.15.62:0/2440548417 >> 172.21.15.62:0/2440548417 conn(0x561915b9cc00 msgr2=0x561914db9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff862640280 1 -- 172.21.15.62:0/2440548417 shutdown_connections 2024-03-23T11:32:23.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.690+0000 7ff862640280 1 -- 172.21.15.62:0/2440548417 wait complete. 2024-03-23T11:32:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.691+0000 7ff862640280 1 Processor -- start 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.691+0000 7ff862640280 1 -- start start 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.692+0000 7ff862640280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d000 0x561914d53180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.692+0000 7ff862640280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561915b9d400 0x561914d52c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.692+0000 7ff862640280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561915b9d800 0x561914d52680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.692+0000 7ff862640280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561914d983c0 con 0x561915b9d000 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.692+0000 7ff862640280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561914d98280 con 0x561915b9d800 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.692+0000 7ff862640280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561914d98140 con 0x561915b9d400 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.692+0000 7ff85fc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d000 0x561914d53180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.692+0000 7ff85fc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d000 0x561914d53180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60652/0 (socket says 172.21.15.62:60652) 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.692+0000 7ff85fc00640 1 -- 172.21.15.62:0/3635791487 learned_addr learned my addr 172.21.15.62:0/3635791487 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.692+0000 7ff860600640 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561915b9d400 0x561914d52c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff85fc00640 1 -- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561915b9d400 msgr2=0x561914d52c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff85fc00640 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561915b9d400 0x561914d52c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff85fc00640 1 -- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561915b9d800 msgr2=0x561914d52680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff85fc00640 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561915b9d800 0x561914d52680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff85fc00640 1 -- 172.21.15.62:0/3635791487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561914cecd00 con 0x561915b9d000 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff860600640 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561915b9d400 0x561914d52c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff85fc00640 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d000 0x561914d53180 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x561915bf6de0 tx=0x561915ba8e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff858e00640 1 -- 172.21.15.62:0/3635791487 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561914d0f1e0 con 0x561915b9d000 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff858e00640 1 -- 172.21.15.62:0/3635791487 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561914d24480 con 0x561915b9d000 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff858e00640 1 -- 172.21.15.62:0/3635791487 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561914d0edc0 con 0x561915b9d000 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff862640280 1 -- 172.21.15.62:0/3635791487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561914cec4e0 con 0x561915b9d000 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff862640280 1 -- 172.21.15.62:0/3635791487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561914cec9c0 con 0x561915b9d000 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.693+0000 7ff862640280 4 main: JournalTool: resolving pool 16 2024-03-23T11:32:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.695+0000 7ff858e00640 1 -- 172.21.15.62:0/3635791487 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x561914d95080 con 0x561915b9d000 2024-03-23T11:32:23.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.696+0000 7ff858e00640 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561915b9dc00 0x561914d53700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.696+0000 7ff858e00640 1 -- 172.21.15.62:0/3635791487 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x561914cf2540 con 0x561915b9d000 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.696+0000 7ff862640280 4 main: JournalTool: creating IoCtx.. 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.696+0000 7ff862640280 4 main: Executing for rank 0 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.696+0000 7ff860600640 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561915b9dc00 0x561914d53700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.697+0000 7ff862640280 1 -- start start 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.697+0000 7ff862640280 1 Processor -- start 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.698+0000 7ff860600640 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561915b9dc00 0x561914d53700 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x561914e380f0 tx=0x561915028c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.698+0000 7ff862640280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56191506e400 0x561914d52100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.698+0000 7ff862640280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56191506e800 0x561914d51b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.698+0000 7ff862640280 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56191506ec00 0x561914d51600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.698+0000 7ff862640280 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561914d98500 con 0x56191506ec00 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.698+0000 7ff862640280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561914d98780 con 0x56191506e400 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.698+0000 7ff862640280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561914d98000 con 0x56191506e800 2024-03-23T11:32:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.698+0000 7ff85fc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56191506ec00 0x561914d51600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.698+0000 7ff85fc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56191506ec00 0x561914d51600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60668/0 (socket says 172.21.15.62:60668) 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.698+0000 7ff85fc00640 1 -- 172.21.15.62:0/1983740992 learned_addr learned my addr 172.21.15.62:0/1983740992 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff85fc00640 1 -- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56191506e800 msgr2=0x561914d51b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff85fc00640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56191506e800 0x561914d51b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff85fc00640 1 -- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56191506e400 msgr2=0x561914d52100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff85fc00640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56191506e400 0x561914d52100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff85fc00640 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561915c38680 con 0x56191506ec00 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff85fc00640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56191506ec00 0x561914d51600 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x561915c5ad50 tx=0x561915028d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=176a3bb1a1416f7a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff855200640 1 -- 172.21.15.62:0/1983740992 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561914d0f600 con 0x56191506ec00 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff855200640 1 -- 172.21.15.62:0/1983740992 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561914d24180 con 0x56191506ec00 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff855200640 1 -- 172.21.15.62:0/1983740992 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561914d0e420 con 0x56191506ec00 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561915c384e0 con 0x56191506ec00 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x561915c37d40 con 0x56191506ec00 2024-03-23T11:32:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff862640280 4 waiting for MDS map... 2024-03-23T11:32:23.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.699+0000 7ff855200640 1 -- 172.21.15.62:0/1983740992 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x561914cf2a80 con 0x56191506ec00 2024-03-23T11:32:23.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.700+0000 7ff855200640 1 -- 172.21.15.62:0/1983740992 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 102) v1 ==== 2573+0+0 (secure 0 0 0) 0x56191508a900 con 0x56191506ec00 2024-03-23T11:32:23.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.700+0000 7ff862640280 4 Got MDS map 102 2024-03-23T11:32:23.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.700+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x56191506f000 0x561915098000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.700+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:1 16.f 16:f1f69726:::400.00000000:head [writefull 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x561915087400 con 0x56191506f000 2024-03-23T11:32:23.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.700+0000 7ff861000640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x56191506f000 0x561915098000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.701+0000 7ff861000640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x56191506f000 0x561915098000 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=4d36efd46e9c66ba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.704+0000 7ff861000640 1 -- 172.21.15.62:0/1983740992 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== osd_op_reply(1 400.00000000 [writefull 0~22] v122'2 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x561915b7eb40 con 0x56191506f000 2024-03-23T11:32:23.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.704+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:2 16.14 16:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x561915087800 con 0x56191506f000 2024-03-23T11:32:23.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff861000640 1 -- 172.21.15.62:0/1983740992 <== osd.0 v2:172.21.15.62:6811/36998 2 ==== osd_op_reply(2 200.00000000 [writefull 0~90] v122'41 uv41 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x561915b7eb40 con 0x56191506f000 2024-03-23T11:32:23.695 INFO:teuthology.orchestra.run.smithi062.stdout:writing EResetJournal entry 2024-03-23T11:32:23.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:3 16.14 16:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x561915087c00 con 0x56191506f000 2024-03-23T11:32:23.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x5619150a6000 0x561915098580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:4 16.1b 16:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x5619150a6400 con 0x5619150a6000 2024-03-23T11:32:23.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5619150a6800 0x561915098b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:5 16.b 16:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x5619150a6c00 con 0x5619150a6800 2024-03-23T11:32:23.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:6 16.1f 16:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x5619150a7000 con 0x5619150a6000 2024-03-23T11:32:23.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5619150a7400 0x561915099080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:7 16.1a 16:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x5619150a7800 con 0x5619150a7400 2024-03-23T11:32:23.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5619150a7c00 0x561915099600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:8 16.1e 16:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x5619150b0000 con 0x5619150a7c00 2024-03-23T11:32:23.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.706+0000 7ff861000640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5619150a6800 0x561915098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.707+0000 7ff860600640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5619150a7400 0x561915099080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.707+0000 7ff860600640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x5619150a6000 0x561915098580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.707+0000 7ff85fc00640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5619150a7c00 0x561915099600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.707+0000 7ff861000640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5619150a6800 0x561915098b00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=7962eb7fb51c57c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.708+0000 7ff860600640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5619150a7400 0x561915099080 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=b59b863814a63ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.708+0000 7ff860600640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x5619150a6000 0x561915098580 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=5835c6e5045491a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.708+0000 7ff85fc00640 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5619150a7c00 0x561915099600 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=f717781fef44e8d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.709+0000 7ff861000640 1 -- 172.21.15.62:0/1983740992 <== osd.0 v2:172.21.15.62:6811/36998 3 ==== osd_op_reply(3 200.00000001 [delete] v122'42 uv42 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x561915b7eb40 con 0x56191506f000 2024-03-23T11:32:23.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.709+0000 7ff861000640 1 -- 172.21.15.62:0/1983740992 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(5 200.00000003 [delete] v122'6 uv5 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561915b7eb40 con 0x5619150a6800 2024-03-23T11:32:23.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.711+0000 7ff860600640 1 -- 172.21.15.62:0/1983740992 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== osd_op_reply(7 200.00000005 [delete] v122'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561915b7f200 con 0x5619150a7400 2024-03-23T11:32:23.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.715+0000 7ff860600640 1 -- 172.21.15.62:0/1983740992 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== osd_op_reply(4 200.00000002 [delete] v122'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561915b7f200 con 0x5619150a6000 2024-03-23T11:32:23.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.715+0000 7ff853400640 1 -- 172.21.15.62:0/1983740992 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:9 16.14 16:2e2fa760:::200.00000001:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x56191506fc00 con 0x56191506f000 2024-03-23T11:32:23.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.715+0000 7ff860600640 1 -- 172.21.15.62:0/1983740992 <== osd.5 v2:172.21.15.120:6824/37642 2 ==== osd_op_reply(6 200.00000004 [delete] v122'9 uv8 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561915b7f200 con 0x5619150a6000 2024-03-23T11:32:23.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.715+0000 7ff85fc00640 1 -- 172.21.15.62:0/1983740992 <== osd.7 v2:172.21.15.120:6816/37655 1 ==== osd_op_reply(8 200.00000006 [delete] v122'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561915b7f8c0 con 0x5619150a7c00 2024-03-23T11:32:23.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff861000640 1 -- 172.21.15.62:0/1983740992 <== osd.0 v2:172.21.15.62:6811/36998 4 ==== osd_op_reply(9 200.00000001 [write 0~48] v122'43 uv43 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x561915b7eb40 con 0x56191506f000 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 4 Successfully wrote new journal pointer and header for rank 8:0 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x56191506f000 msgr2=0x561915098000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x56191506f000 0x561915098000 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5619150a6800 msgr2=0x561915098b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x5619150a6800 0x561915098b00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5619150a7400 msgr2=0x561915099080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x5619150a7400 0x561915099080 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x5619150a6000 msgr2=0x561915098580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x5619150a6000 0x561915098580 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5619150a7c00 msgr2=0x561915099600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x5619150a7c00 0x561915099600 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff85fc00640 1 -- 172.21.15.62:0/1983740992 reap_dead start 2024-03-23T11:32:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff85fc00640 1 -- 172.21.15.62:0/1983740992 reap_dead start 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56191506ec00 msgr2=0x561914d51600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.716+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56191506ec00 0x561914d51600 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x561915c5ad50 tx=0x561915028d20 comp rx=0 tx=0).stop 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 shutdown_connections 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 --2- 172.21.15.62:0/1983740992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56191506ec00 0x561914d51600 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 >> 172.21.15.62:0/1983740992 conn(0x56191506e000 msgr2=0x561914db9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 shutdown_connections 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 -- 172.21.15.62:0/1983740992 wait complete. 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 -- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561915b9dc00 msgr2=0x561914d53700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561915b9dc00 0x561914d53700 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x561914e380f0 tx=0x561915028c00 comp rx=0 tx=0).stop 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 -- 172.21.15.62:0/3635791487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d000 msgr2=0x561914d53180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d000 0x561914d53180 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x561915bf6de0 tx=0x561915ba8e40 comp rx=0 tx=0).stop 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 -- 172.21.15.62:0/3635791487 shutdown_connections 2024-03-23T11:32:23.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x561915b9dc00 0x561914d53700 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561915b9d800 0x561914d52680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561915b9d400 0x561914d52c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 --2- 172.21.15.62:0/3635791487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561915b9d000 0x561914d53180 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 -- 172.21.15.62:0/3635791487 >> 172.21.15.62:0/3635791487 conn(0x561915b9cc00 msgr2=0x561914db9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 -- 172.21.15.62:0/3635791487 shutdown_connections 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.717+0000 7ff862640280 1 -- 172.21.15.62:0/3635791487 wait complete. 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.718+0000 7ff862640280 1 -- 172.21.15.62:0/1398772438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561914cc7c00 msgr2=0x561914d51080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.718+0000 7ff862640280 1 --2- 172.21.15.62:0/1398772438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561914cc7c00 0x561914d51080 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x561915aeacc0 tx=0x561915b8e120 comp rx=0 tx=0).stop 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.718+0000 7ff862640280 1 -- 172.21.15.62:0/1398772438 shutdown_connections 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.718+0000 7ff862640280 1 --2- 172.21.15.62:0/1398772438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561914cc7c00 0x561914d51080 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.718+0000 7ff862640280 1 --2- 172.21.15.62:0/1398772438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561914cc7800 0x561914d50b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.718+0000 7ff862640280 1 --2- 172.21.15.62:0/1398772438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561914cc6c00 0x561914d50580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.718+0000 7ff862640280 1 -- 172.21.15.62:0/1398772438 >> 172.21.15.62:0/1398772438 conn(0x561914cc7000 msgr2=0x561914db8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.718+0000 7ff862640280 1 -- 172.21.15.62:0/1398772438 shutdown_connections 2024-03-23T11:32:23.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.718+0000 7ff862640280 1 -- 172.21.15.62:0/1398772438 wait complete. 2024-03-23T11:32:23.713 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset', '--force'] in time 0:00:00.127336, result: writing EResetJournal entry 2024-03-23T11:32:23.713 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-03-23T11:32:23.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.865+0000 7fc2b4438740 1 -- start start 2024-03-23T11:32:23.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.865+0000 7fc2b4438740 1 Processor -- start 2024-03-23T11:32:23.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.865+0000 7fc2b4438740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d03d7800 0x55c5d045eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.865+0000 7fc2b4438740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d7c00 0x55c5d045e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.866+0000 7fc2b4438740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d1169800 0x55c5d045f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.866+0000 7fc2b4438740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5d04a8a00 con 0x55c5d03d7c00 2024-03-23T11:32:23.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.866+0000 7fc2b4438740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5d04a8780 con 0x55c5d1169800 2024-03-23T11:32:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.866+0000 7fc2b4438740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5d04a8640 con 0x55c5d03d7800 2024-03-23T11:32:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.866+0000 7fc2b1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d7c00 0x55c5d045e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.866+0000 7fc2b1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d7c00 0x55c5d045e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60682/0 (socket says 172.21.15.62:60682) 2024-03-23T11:32:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.866+0000 7fc2b1400640 1 -- 172.21.15.62:0/1527844830 learned_addr learned my addr 172.21.15.62:0/1527844830 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.866+0000 7fc2b1e00640 1 --2- 172.21.15.62:0/1527844830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d03d7800 0x55c5d045eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.866+0000 7fc2b2800640 1 --2- 172.21.15.62:0/1527844830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d1169800 0x55c5d045f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2b1400640 1 -- 172.21.15.62:0/1527844830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d03d7800 msgr2=0x55c5d045eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2b1400640 1 --2- 172.21.15.62:0/1527844830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d03d7800 0x55c5d045eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2b1400640 1 -- 172.21.15.62:0/1527844830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d1169800 msgr2=0x55c5d045f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2b1400640 1 --2- 172.21.15.62:0/1527844830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d1169800 0x55c5d045f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2b1400640 1 -- 172.21.15.62:0/1527844830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5d03fc9c0 con 0x55c5d03d7c00 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2b1400640 1 --2- 172.21.15.62:0/1527844830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d7c00 0x55c5d045e580 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x55c5d11f1350 tx=0x55c5d11fcde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d2b3304a9aab1e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2aec00640 1 -- 172.21.15.62:0/1527844830 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c5d041d600 con 0x55c5d03d7c00 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2aec00640 1 -- 172.21.15.62:0/1527844830 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c5d0432180 con 0x55c5d03d7c00 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2b4438740 1 -- 172.21.15.62:0/1527844830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5d03fcb60 con 0x55c5d03d7c00 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2b4438740 1 -- 172.21.15.62:0/1527844830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55c5d03fc820 con 0x55c5d03d7c00 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2b4438740 4 waiting for MDS map... 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2aec00640 1 -- 172.21.15.62:0/1527844830 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c5d041d1e0 con 0x55c5d03d7c00 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.867+0000 7fc2aec00640 1 -- 172.21.15.62:0/1527844830 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x55c5d042cc40 con 0x55c5d03d7c00 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.868+0000 7fc2aec00640 1 -- 172.21.15.62:0/1527844830 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 102) v1 ==== 2573+0+0 (secure 0 0 0) 0x55c5d0427200 con 0x55c5d03d7c00 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.868+0000 7fc2b4438740 4 Got MDS map 102 2024-03-23T11:32:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.868+0000 7fc2b4438740 4 datascan.main: Using metadata pool output 2024-03-23T11:32:23.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.868+0000 7fc2b4438740 4 datascan.main: connecting to RADOS... 2024-03-23T11:32:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.869+0000 7fc2b4438740 1 Processor -- start 2024-03-23T11:32:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.869+0000 7fc2b4438740 1 -- start start 2024-03-23T11:32:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b4438740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d03d6c00 0x55c5d045f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b4438740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d12b8c00 0x55c5d045fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b4438740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d12b9000 0x55c5d0460100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b4438740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5d041fe00 con 0x55c5d12b8c00 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b4438740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5d04a8780 con 0x55c5d12b9000 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b4438740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5d04a8500 con 0x55c5d03d6c00 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d12b8c00 0x55c5d045fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d12b8c00 0x55c5d045fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60692/0 (socket says 172.21.15.62:60692) 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b1e00640 1 -- 172.21.15.62:0/1652166132 learned_addr learned my addr 172.21.15.62:0/1652166132 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b2800640 1 --2- 172.21.15.62:0/1652166132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d03d6c00 0x55c5d045f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b1400640 1 --2- 172.21.15.62:0/1652166132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d12b9000 0x55c5d0460100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b1e00640 1 -- 172.21.15.62:0/1652166132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d03d6c00 msgr2=0x55c5d045f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b1e00640 1 --2- 172.21.15.62:0/1652166132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d03d6c00 0x55c5d045f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b1e00640 1 -- 172.21.15.62:0/1652166132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d12b9000 msgr2=0x55c5d0460100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b1e00640 1 --2- 172.21.15.62:0/1652166132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d12b9000 0x55c5d0460100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.870+0000 7fc2b1e00640 1 -- 172.21.15.62:0/1652166132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5d03fc4e0 con 0x55c5d12b8c00 2024-03-23T11:32:23.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.871+0000 7fc2b1e00640 1 --2- 172.21.15.62:0/1652166132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d12b8c00 0x55c5d045fb80 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x55c5d12ca690 tx=0x55c5d12ccc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3de79ae16a05ec server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.871+0000 7fc2ac400640 1 -- 172.21.15.62:0/1652166132 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c5d041d600 con 0x55c5d12b8c00 2024-03-23T11:32:23.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.871+0000 7fc2ac400640 1 -- 172.21.15.62:0/1652166132 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c5d0432480 con 0x55c5d12b8c00 2024-03-23T11:32:23.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.871+0000 7fc2ac400640 1 -- 172.21.15.62:0/1652166132 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c5d041d1e0 con 0x55c5d12b8c00 2024-03-23T11:32:23.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.871+0000 7fc2b4438740 1 -- 172.21.15.62:0/1652166132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d12b8c00 msgr2=0x55c5d045fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.871+0000 7fc2b4438740 1 --2- 172.21.15.62:0/1652166132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d12b8c00 0x55c5d045fb80 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x55c5d12ca690 tx=0x55c5d12ccc00 comp rx=0 tx=0).stop 2024-03-23T11:32:23.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.872+0000 7fc2b4438740 1 -- 172.21.15.62:0/1652166132 shutdown_connections 2024-03-23T11:32:23.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.872+0000 7fc2b4438740 1 --2- 172.21.15.62:0/1652166132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d12b8c00 0x55c5d045fb80 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.872+0000 7fc2b4438740 1 --2- 172.21.15.62:0/1652166132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d12b9000 0x55c5d0460100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.872+0000 7fc2b4438740 1 --2- 172.21.15.62:0/1652166132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d03d6c00 0x55c5d045f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.872+0000 7fc2b4438740 1 -- 172.21.15.62:0/1652166132 >> 172.21.15.62:0/1652166132 conn(0x55c5d1169c00 msgr2=0x55c5d04c7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.872+0000 7fc2b4438740 1 -- 172.21.15.62:0/1652166132 shutdown_connections 2024-03-23T11:32:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.872+0000 7fc2b4438740 1 -- 172.21.15.62:0/1652166132 wait complete. 2024-03-23T11:32:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.872+0000 7fc2b4438740 1 Processor -- start 2024-03-23T11:32:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.873+0000 7fc2b4438740 1 -- start start 2024-03-23T11:32:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.873+0000 7fc2b4438740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d6c00 0x55c5d045f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.873+0000 7fc2b4438740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d12b9c00 0x55c5d045fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.873+0000 7fc2b4438740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d12b9800 0x55c5d0460100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.873+0000 7fc2b4438740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5d04a83c0 con 0x55c5d03d6c00 2024-03-23T11:32:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.873+0000 7fc2b4438740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5d04a8280 con 0x55c5d12b9c00 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.873+0000 7fc2b4438740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5d04a8140 con 0x55c5d12b9800 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.873+0000 7fc2b2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d6c00 0x55c5d045f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.874+0000 7fc2b2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d6c00 0x55c5d045f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60706/0 (socket says 172.21.15.62:60706) 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.874+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 learned_addr learned my addr 172.21.15.62:0/333937937 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.874+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d12b9800 msgr2=0x55c5d0460100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.874+0000 7fc2b2800640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d12b9800 0x55c5d0460100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.874+0000 7fc2b1e00640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d12b9c00 0x55c5d045fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.874+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d12b9c00 msgr2=0x55c5d045fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.874+0000 7fc2b2800640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d12b9c00 0x55c5d045fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.874+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5d03fc9c0 con 0x55c5d03d6c00 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.874+0000 7fc2b2800640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d6c00 0x55c5d045f600 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x55c5d12cb290 tx=0x55c5d11fc0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.874+0000 7fc2aa600640 1 -- 172.21.15.62:0/333937937 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c5d041d080 con 0x55c5d03d6c00 2024-03-23T11:32:23.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.875+0000 7fc2aa600640 1 -- 172.21.15.62:0/333937937 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c5d1206180 con 0x55c5d03d6c00 2024-03-23T11:32:23.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.875+0000 7fc2aa600640 1 -- 172.21.15.62:0/333937937 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c5d041d600 con 0x55c5d03d6c00 2024-03-23T11:32:23.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.875+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c5d03fcb60 con 0x55c5d03d6c00 2024-03-23T11:32:23.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.875+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5d03fc1a0 con 0x55c5d03d6c00 2024-03-23T11:32:23.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.875+0000 7fc2b4438740 4 datascan.init: resolving metadata pool 16 2024-03-23T11:32:23.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.876+0000 7fc2aa600640 1 -- 172.21.15.62:0/333937937 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x55c5d0409340 con 0x55c5d03d6c00 2024-03-23T11:32:23.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.877+0000 7fc2aa600640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x55c5d131ec00 0x55c5d0461180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.877+0000 7fc2aa600640 1 -- 172.21.15.62:0/333937937 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x55c5d042ca80 con 0x55c5d03d6c00 2024-03-23T11:32:23.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.877+0000 7fc2b4438740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-23T11:32:23.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.877+0000 7fc2b4438740 4 datascan.main: data pool 'cephfs_data' has ID 17 2024-03-23T11:32:23.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.877+0000 7fc2b4438740 4 datascan.main: opening data pool 'cephfs_data' 2024-03-23T11:32:23.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.877+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x55c5d12b9400 0x55c5d0460680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.877+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d12b9000 con 0x55c5d12b9400 2024-03-23T11:32:23.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.878+0000 7fc2b1e00640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x55c5d131ec00 0x55c5d0461180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.878+0000 7fc2b1400640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x55c5d12b9400 0x55c5d0460680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.879+0000 7fc2b1400640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x55c5d12b9400 0x55c5d0460680 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.879+0000 7fc2b1e00640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x55c5d131ec00 0x55c5d0461180 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x55c5d12ca690 tx=0x55c5d11fcd20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.880+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.7 v2:172.21.15.120:6816/37655 1 ==== osd_op_reply(1 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d12b9400 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.880+0000 7fc2aba00640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55c5d131f000 0x55c5d0461700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.880+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d131f400 con 0x55c5d131f000 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.880+0000 7fc2b2800640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55c5d131f000 0x55c5d0461700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.881+0000 7fc2b2800640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55c5d131f000 0x55c5d0461700 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.881+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(2 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d131f000 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.881+0000 7fc2ac400640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x55c5d131fc00 0x55c5d075a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.881+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075e000 con 0x55c5d131fc00 2024-03-23T11:32:23.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.882+0000 7fc2b1e00640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x55c5d131fc00 0x55c5d075a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.882+0000 7fc2b1e00640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x55c5d131fc00 0x55c5d075a000 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.882+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== osd_op_reply(3 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d131fc00 2024-03-23T11:32:23.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.882+0000 7fc2aba00640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55c5d131f800 0x55c5d075a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.882+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075e800 con 0x55c5d131f800 2024-03-23T11:32:23.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.883+0000 7fc2b1400640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55c5d131f800 0x55c5d075a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.883+0000 7fc2b1400640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55c5d131f800 0x55c5d075a580 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.884+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== osd_op_reply(4 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.884+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075e400 con 0x55c5d12b9400 2024-03-23T11:32:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.885+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.7 v2:172.21.15.120:6816/37655 2 ==== osd_op_reply(5 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d12b9400 2024-03-23T11:32:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.885+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075ec00 con 0x55c5d131fc00 2024-03-23T11:32:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.885+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.3 v2:172.21.15.62:6808/37000 2 ==== osd_op_reply(6 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d131fc00 2024-03-23T11:32:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.885+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075f400 con 0x55c5d131f800 2024-03-23T11:32:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.885+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 2 ==== osd_op_reply(7 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.885+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075f000 con 0x55c5d12b9400 2024-03-23T11:32:23.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.886+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.7 v2:172.21.15.120:6816/37655 3 ==== osd_op_reply(8 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d12b9400 2024-03-23T11:32:23.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.886+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075f800 con 0x55c5d131f800 2024-03-23T11:32:23.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.887+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 3 ==== osd_op_reply(9 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.887+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075e800 con 0x55c5d131fc00 2024-03-23T11:32:23.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.887+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.3 v2:172.21.15.62:6808/37000 3 ==== osd_op_reply(10 [pgnls start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d131fc00 2024-03-23T11:32:23.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.887+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075fc00 con 0x55c5d131fc00 2024-03-23T11:32:23.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.887+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.3 v2:172.21.15.62:6808/37000 4 ==== osd_op_reply(11 [pgnls start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d131fc00 2024-03-23T11:32:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.887+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075e400 con 0x55c5d131f800 2024-03-23T11:32:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.888+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 4 ==== osd_op_reply(12 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.888+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075e000 con 0x55c5d12b9400 2024-03-23T11:32:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.889+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.7 v2:172.21.15.120:6816/37655 4 ==== osd_op_reply(13 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d12b9400 2024-03-23T11:32:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.889+0000 7fc2aba00640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55c5d075f400 0x55c5d075ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.889+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d12b9000 con 0x55c5d075f400 2024-03-23T11:32:23.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.889+0000 7fc2b2800640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55c5d075f400 0x55c5d075ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.890+0000 7fc2b2800640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55c5d075f400 0x55c5d075ab00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.891+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== osd_op_reply(14 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075f400 2024-03-23T11:32:23.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.891+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075ec00 con 0x55c5d131f000 2024-03-23T11:32:23.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.891+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.1 v2:172.21.15.62:6824/37001 2 ==== osd_op_reply(15 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d131f000 2024-03-23T11:32:23.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.891+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0768000 con 0x55c5d131f800 2024-03-23T11:32:23.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.892+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 5 ==== osd_op_reply(16 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.892+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075e800 con 0x55c5d075f400 2024-03-23T11:32:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.893+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.4 v2:172.21.15.120:6803/37640 2 ==== osd_op_reply(17 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075f400 2024-03-23T11:32:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.893+0000 7fc2aba00640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x55c5d0768400 0x55c5d075b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.893+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0768800 con 0x55c5d0768400 2024-03-23T11:32:23.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.893+0000 7fc2b1e00640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x55c5d0768400 0x55c5d075b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.894+0000 7fc2b1e00640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x55c5d0768400 0x55c5d075b080 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.895+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== osd_op_reply(18 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d0768400 2024-03-23T11:32:23.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.895+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d131e800 con 0x55c5d131fc00 2024-03-23T11:32:23.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.896+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.3 v2:172.21.15.62:6808/37000 5 ==== osd_op_reply(19 [pgnls start_epoch 122 out=81b] v121'22 uv22 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d131fc00 2024-03-23T11:32:23.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.896+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:20 17.12 17:4bec3886:::10000000002.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d12b8c00 con 0x55c5d131fc00 2024-03-23T11:32:23.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.896+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.3 v2:172.21.15.62:6808/37000 6 ==== osd_op_reply(20 10000000002.00000000 [stat out=16b] v0'0 uv22 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d131fc00 2024-03-23T11:32:23.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.896+0000 7fc2b4438740 10 datascan.operator(): handling object 1099511627778.0 2024-03-23T11:32:23.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.896+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:21 17.12 17:4bec3886:::10000000002.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075ec00 con 0x55c5d131fc00 2024-03-23T11:32:23.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.899+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.3 v2:172.21.15.62:6808/37000 7 ==== osd_op_reply(21 10000000002.00000000 [call] v122'23 uv23 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d131fc00 2024-03-23T11:32:23.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.899+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:22 17.32 17:4c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075e800 con 0x55c5d12b9400 2024-03-23T11:32:23.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.900+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.7 v2:172.21.15.120:6816/37655 5 ==== osd_op_reply(22 [pgnls start_epoch 122 out=49b] v120'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d12b9400 2024-03-23T11:32:23.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.900+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:23 17.a 17:50000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075fc00 con 0x55c5d12b9400 2024-03-23T11:32:23.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.901+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.7 v2:172.21.15.120:6816/37655 6 ==== osd_op_reply(23 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d12b9400 2024-03-23T11:32:23.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.901+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:24 17.2a 17:54000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0768c00 con 0x55c5d131fc00 2024-03-23T11:32:23.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.901+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.3 v2:172.21.15.62:6808/37000 8 ==== osd_op_reply(24 [pgnls start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d131fc00 2024-03-23T11:32:23.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.901+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:25 17.1a 17:58000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075ec00 con 0x55c5d0768400 2024-03-23T11:32:23.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.902+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.5 v2:172.21.15.120:6824/37642 2 ==== osd_op_reply(25 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d0768400 2024-03-23T11:32:23.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.902+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:26 17.3a 17:5c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0769000 con 0x55c5d131f000 2024-03-23T11:32:23.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.902+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.1 v2:172.21.15.62:6824/37001 3 ==== osd_op_reply(26 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d131f000 2024-03-23T11:32:23.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.903+0000 7fc2ac400640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55c5d075e000 0x55c5d075c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.903+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:27 17.6 17:60000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d12b8c00 con 0x55c5d075e000 2024-03-23T11:32:23.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.903+0000 7fc2b1400640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55c5d075e000 0x55c5d075c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.904+0000 7fc2b1400640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55c5d075e000 0x55c5d075c100 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.905+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== osd_op_reply(27 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d075e000 2024-03-23T11:32:23.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.905+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:28 17.26 17:64000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0768800 con 0x55c5d131f800 2024-03-23T11:32:23.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.905+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 6 ==== osd_op_reply(28 [pgnls start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.906+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:29 17.16 17:68000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d131e800 con 0x55c5d0768400 2024-03-23T11:32:23.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.906+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.5 v2:172.21.15.120:6824/37642 3 ==== osd_op_reply(29 [pgnls start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d0768400 2024-03-23T11:32:23.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.906+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:30 17.36 17:6c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0768000 con 0x55c5d131f800 2024-03-23T11:32:23.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.907+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 7 ==== osd_op_reply(30 [pgnls start_epoch 122 out=81b] v120'33 uv17 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.907+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:31 17.36 17:6fe38dc2:::100000001f7.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075f000 con 0x55c5d131f800 2024-03-23T11:32:23.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.908+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 8 ==== osd_op_reply(31 100000001f7.00000000 [stat out=16b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.908+0000 7fc2b4438740 10 datascan.operator(): handling object 1099511628279.0 2024-03-23T11:32:23.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.908+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:32 17.36 17:6fe38dc2:::100000001f7.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075f800 con 0x55c5d131f800 2024-03-23T11:32:23.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.909+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 9 ==== osd_op_reply(32 100000001f7.00000000 [call] v122'34 uv34 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.909+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55c5d075e400 0x55c5d0460c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:23.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.909+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:33 17.e 17:70000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d12b9000 con 0x55c5d075e400 2024-03-23T11:32:23.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.910+0000 7fc2b2800640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55c5d075e400 0x55c5d0460c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:23.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.910+0000 7fc2b2800640 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55c5d075e400 0x55c5d0460c00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:23.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.911+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.2 v2:172.21.15.62:6800/36996 1 ==== osd_op_reply(33 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075e400 2024-03-23T11:32:23.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.911+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:34 17.2e 17:74000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0769400 con 0x55c5d075e000 2024-03-23T11:32:23.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.912+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.6 v2:172.21.15.120:6800/37641 2 ==== osd_op_reply(34 [pgnls start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d075e000 2024-03-23T11:32:23.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.912+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:35 17.1e 17:78000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075e800 con 0x55c5d131f000 2024-03-23T11:32:23.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.912+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.1 v2:172.21.15.62:6824/37001 4 ==== osd_op_reply(35 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d131f000 2024-03-23T11:32:23.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.912+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:36 17.3e 17:7c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0769800 con 0x55c5d075f400 2024-03-23T11:32:23.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.913+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.4 v2:172.21.15.120:6803/37640 3 ==== osd_op_reply(36 [pgnls start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075f400 2024-03-23T11:32:23.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.913+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:37 17.1 17:80000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075fc00 con 0x55c5d075e400 2024-03-23T11:32:23.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.914+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.2 v2:172.21.15.62:6800/36996 2 ==== osd_op_reply(37 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075e400 2024-03-23T11:32:23.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.914+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:38 17.21 17:84000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0769c00 con 0x55c5d131f800 2024-03-23T11:32:23.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.914+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 10 ==== osd_op_reply(38 [pgnls start_epoch 122 out=49b] v120'24 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.914+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:39 17.11 17:88000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075f000 con 0x55c5d131fc00 2024-03-23T11:32:23.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.915+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.3 v2:172.21.15.62:6808/37000 9 ==== osd_op_reply(39 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d131fc00 2024-03-23T11:32:23.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.915+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:40 17.31 17:8c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0786000 con 0x55c5d131f000 2024-03-23T11:32:23.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.915+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.1 v2:172.21.15.62:6824/37001 5 ==== osd_op_reply(40 [pgnls start_epoch 122 out=81b] v120'15 uv8 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d131f000 2024-03-23T11:32:23.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.915+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:41 17.31 17:8de29a03:::10000000004.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d131f400 con 0x55c5d131f000 2024-03-23T11:32:23.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.916+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.1 v2:172.21.15.62:6824/37001 6 ==== osd_op_reply(41 10000000004.00000001 [stat out=16b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d131f000 2024-03-23T11:32:23.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.916+0000 7fc2b4438740 10 datascan.operator(): handling object 1099511627780.1 2024-03-23T11:32:23.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.916+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:42 17.5 17:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d1169400 con 0x55c5d075e000 2024-03-23T11:32:23.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.919+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.6 v2:172.21.15.120:6800/37641 3 ==== osd_op_reply(42 10000000004.00000000 [call] v122'14 uv14 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d075e000 2024-03-23T11:32:23.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.919+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:43 17.9 17:90000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0786000 con 0x55c5d131fc00 2024-03-23T11:32:23.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.920+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.3 v2:172.21.15.62:6808/37000 10 ==== osd_op_reply(43 [pgnls start_epoch 122 out=49b] v120'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d131fc00 2024-03-23T11:32:23.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.920+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:44 17.29 17:94000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0786400 con 0x55c5d075f400 2024-03-23T11:32:23.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.921+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.4 v2:172.21.15.120:6803/37640 4 ==== osd_op_reply(44 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075f400 2024-03-23T11:32:23.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.921+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:45 17.19 17:98000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0768800 con 0x55c5d0768400 2024-03-23T11:32:23.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.922+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.5 v2:172.21.15.120:6824/37642 4 ==== osd_op_reply(45 [pgnls start_epoch 122 out=81b] v121'23 uv23 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d0768400 2024-03-23T11:32:23.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.922+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:46 17.19 17:997b7129:::100000001f8.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0787c00 con 0x55c5d0768400 2024-03-23T11:32:23.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.923+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.5 v2:172.21.15.120:6824/37642 5 ==== osd_op_reply(46 100000001f8.00000000 [stat out=16b] v0'0 uv23 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d0768400 2024-03-23T11:32:23.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.923+0000 7fc2b4438740 10 datascan.operator(): handling object 1099511628280.0 2024-03-23T11:32:23.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.923+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:47 17.19 17:997b7129:::100000001f8.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075e800 con 0x55c5d0768400 2024-03-23T11:32:23.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.927+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.5 v2:172.21.15.120:6824/37642 6 ==== osd_op_reply(47 100000001f8.00000000 [call] v122'24 uv24 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d0768400 2024-03-23T11:32:23.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.927+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:48 17.39 17:9c000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075fc00 con 0x55c5d0768400 2024-03-23T11:32:23.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.928+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.5 v2:172.21.15.120:6824/37642 7 ==== osd_op_reply(48 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d0768400 2024-03-23T11:32:23.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.928+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:49 17.5 17:a0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0768000 con 0x55c5d075e000 2024-03-23T11:32:23.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.929+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.6 v2:172.21.15.120:6800/37641 4 ==== osd_op_reply(49 [pgnls start_epoch 122 out=81b] v122'14 uv14 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d075e000 2024-03-23T11:32:23.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.929+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:50 17.5 17:a367a3e8:::10000000004.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0769000 con 0x55c5d075e000 2024-03-23T11:32:23.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.930+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.6 v2:172.21.15.120:6800/37641 5 ==== osd_op_reply(50 10000000004.00000000 [stat out=16b] v0'0 uv14 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d075e000 2024-03-23T11:32:23.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.930+0000 7fc2b4438740 10 datascan.operator(): handling object 1099511627780.0 2024-03-23T11:32:23.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.930+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:51 17.5 17:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0769800 con 0x55c5d075e000 2024-03-23T11:32:23.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.933+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.6 v2:172.21.15.120:6800/37641 6 ==== osd_op_reply(51 10000000004.00000000 [call] v122'15 uv15 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d075e000 2024-03-23T11:32:23.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.934+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:52 17.25 17:a4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d12b9000 con 0x55c5d131f000 2024-03-23T11:32:23.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.934+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.1 v2:172.21.15.62:6824/37001 7 ==== osd_op_reply(52 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d131f000 2024-03-23T11:32:23.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.934+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:53 17.15 17:a8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d12b8c00 con 0x55c5d0768400 2024-03-23T11:32:23.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.935+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.5 v2:172.21.15.120:6824/37642 8 ==== osd_op_reply(53 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d0768400 2024-03-23T11:32:23.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.935+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:54 17.35 17:ac000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0786800 con 0x55c5d075f400 2024-03-23T11:32:23.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.937+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.4 v2:172.21.15.120:6803/37640 5 ==== osd_op_reply(54 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075f400 2024-03-23T11:32:23.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.937+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:55 17.d 17:b0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0769400 con 0x55c5d12b9400 2024-03-23T11:32:23.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.938+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.7 v2:172.21.15.120:6816/37655 7 ==== osd_op_reply(55 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d12b9400 2024-03-23T11:32:23.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.938+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:56 17.2d 17:b4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0786c00 con 0x55c5d12b9400 2024-03-23T11:32:23.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.939+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.7 v2:172.21.15.120:6816/37655 8 ==== osd_op_reply(56 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d12b9400 2024-03-23T11:32:23.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.939+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:57 17.1d 17:b8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0786000 con 0x55c5d12b9400 2024-03-23T11:32:23.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.940+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.7 v2:172.21.15.120:6816/37655 9 ==== osd_op_reply(57 [pgnls start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d12b9400 2024-03-23T11:32:23.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.941+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:58 17.3d 17:bc000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0787000 con 0x55c5d12b9400 2024-03-23T11:32:23.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.942+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.7 v2:172.21.15.120:6816/37655 10 ==== osd_op_reply(58 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d12b9400 2024-03-23T11:32:23.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.942+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:59 17.3 17:c0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075f800 con 0x55c5d075f400 2024-03-23T11:32:23.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.943+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.4 v2:172.21.15.120:6803/37640 6 ==== osd_op_reply(59 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075f400 2024-03-23T11:32:23.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.943+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:60 17.23 17:c4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0787400 con 0x55c5d075f400 2024-03-23T11:32:23.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.944+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.4 v2:172.21.15.120:6803/37640 7 ==== osd_op_reply(60 [pgnls start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075f400 2024-03-23T11:32:23.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.944+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:61 17.13 17:c8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075ec00 con 0x55c5d131fc00 2024-03-23T11:32:23.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.945+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 <== osd.3 v2:172.21.15.62:6808/37000 11 ==== osd_op_reply(61 [pgnls start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2fc0 con 0x55c5d131fc00 2024-03-23T11:32:23.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.945+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:62 17.33 17:cc000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0787800 con 0x55c5d12b9400 2024-03-23T11:32:23.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.946+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.7 v2:172.21.15.120:6816/37655 11 ==== osd_op_reply(62 [pgnls start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d12b9400 2024-03-23T11:32:23.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.946+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:63 17.b 17:d0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d075f000 con 0x55c5d131f800 2024-03-23T11:32:23.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.947+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 11 ==== osd_op_reply(63 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.947+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:64 17.2b 17:d4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0769c00 con 0x55c5d075f400 2024-03-23T11:32:23.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.948+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.4 v2:172.21.15.120:6803/37640 8 ==== osd_op_reply(64 [pgnls start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075f400 2024-03-23T11:32:23.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.948+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:65 17.1b 17:d8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d131e800 con 0x55c5d075f400 2024-03-23T11:32:23.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.949+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.4 v2:172.21.15.120:6803/37640 9 ==== osd_op_reply(65 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075f400 2024-03-23T11:32:23.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.949+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:66 17.3b 17:dc000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0768000 con 0x55c5d075e400 2024-03-23T11:32:23.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.950+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.2 v2:172.21.15.62:6800/36996 3 ==== osd_op_reply(66 [pgnls start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075e400 2024-03-23T11:32:23.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.950+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:67 17.7 17:e0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d1169400 con 0x55c5d131f800 2024-03-23T11:32:23.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.951+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 12 ==== osd_op_reply(67 [pgnls start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.951+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:68 17.27 17:e4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0769000 con 0x55c5d075f400 2024-03-23T11:32:23.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.952+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.4 v2:172.21.15.120:6803/37640 10 ==== osd_op_reply(68 [pgnls start_epoch 122 out=49b] v120'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075f400 2024-03-23T11:32:23.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.952+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:69 17.17 17:e8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0769c00 con 0x55c5d075e000 2024-03-23T11:32:23.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.953+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.6 v2:172.21.15.120:6800/37641 7 ==== osd_op_reply(69 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d075e000 2024-03-23T11:32:23.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.953+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:70 17.37 17:ec000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0769800 con 0x55c5d131f800 2024-03-23T11:32:23.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.954+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 13 ==== osd_op_reply(70 [pgnls start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.954+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:71 17.f 17:f0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0786400 con 0x55c5d131f000 2024-03-23T11:32:23.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.955+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.1 v2:172.21.15.62:6824/37001 8 ==== osd_op_reply(71 [pgnls start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d131f000 2024-03-23T11:32:23.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.955+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:72 17.2f 17:f4000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0768c00 con 0x55c5d075f400 2024-03-23T11:32:23.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.956+0000 7fc2b2800640 1 -- 172.21.15.62:0/333937937 <== osd.4 v2:172.21.15.120:6803/37640 11 ==== osd_op_reply(72 [pgnls start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2d80 con 0x55c5d075f400 2024-03-23T11:32:23.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.956+0000 7fc2ac400640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:73 17.1f 17:f8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0786800 con 0x55c5d075e000 2024-03-23T11:32:23.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.957+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.6 v2:172.21.15.120:6800/37641 8 ==== osd_op_reply(73 [pgnls start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d075e000 2024-03-23T11:32:23.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.957+0000 7fc2aba00640 1 -- 172.21.15.62:0/333937937 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:74 17.3f 17:fc000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55c5d0768800 con 0x55c5d131f800 2024-03-23T11:32:23.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b1400640 1 -- 172.21.15.62:0/333937937 <== osd.0 v2:172.21.15.62:6811/36998 14 ==== osd_op_reply(74 [pgnls start_epoch 122 out=49b] v120'17 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c5d12a2b40 con 0x55c5d131f800 2024-03-23T11:32:23.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55c5d131f800 msgr2=0x55c5d075a580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55c5d131f800 0x55c5d075a580 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55c5d131f000 msgr2=0x55c5d0461700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55c5d131f000 0x55c5d0461700 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55c5d075e400 msgr2=0x55c5d0460c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55c5d075e400 0x55c5d0460c00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x55c5d131fc00 msgr2=0x55c5d075a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x55c5d131fc00 0x55c5d075a000 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55c5d075f400 msgr2=0x55c5d075ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55c5d075f400 0x55c5d075ab00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 reap_dead start 2024-03-23T11:32:23.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.958+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 reap_dead start 2024-03-23T11:32:23.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x55c5d0768400 msgr2=0x55c5d075b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x55c5d0768400 0x55c5d075b080 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55c5d075e000 msgr2=0x55c5d075c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55c5d075e000 0x55c5d075c100 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x55c5d12b9400 msgr2=0x55c5d0460680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x55c5d12b9400 0x55c5d0460680 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x55c5d131ec00 msgr2=0x55c5d0461180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x55c5d131ec00 0x55c5d0461180 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x55c5d12ca690 tx=0x55c5d11fcd20 comp rx=0 tx=0).stop 2024-03-23T11:32:23.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d6c00 msgr2=0x55c5d045f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 --2- 172.21.15.62:0/333937937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d6c00 0x55c5d045f600 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x55c5d12cb290 tx=0x55c5d11fc0c0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b1e00640 1 -- 172.21.15.62:0/333937937 reap_dead start 2024-03-23T11:32:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 shutdown_connections 2024-03-23T11:32:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.959+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 >> 172.21.15.62:0/333937937 conn(0x55c5d1169c00 msgr2=0x55c5d04c7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.960+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 shutdown_connections 2024-03-23T11:32:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.960+0000 7fc2b4438740 1 -- 172.21.15.62:0/333937937 wait complete. 2024-03-23T11:32:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.961+0000 7fc2b4438740 1 -- 172.21.15.62:0/1527844830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d7c00 msgr2=0x55c5d045e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.961+0000 7fc2b4438740 1 --2- 172.21.15.62:0/1527844830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d7c00 0x55c5d045e580 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x55c5d11f1350 tx=0x55c5d11fcde0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.961+0000 7fc2b4438740 1 -- 172.21.15.62:0/1527844830 shutdown_connections 2024-03-23T11:32:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.961+0000 7fc2b4438740 1 --2- 172.21.15.62:0/1527844830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c5d03d7c00 0x55c5d045e580 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.961+0000 7fc2b4438740 1 --2- 172.21.15.62:0/1527844830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d1169800 0x55c5d045f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.961+0000 7fc2b4438740 1 --2- 172.21.15.62:0/1527844830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c5d03d7800 0x55c5d045eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:23.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.961+0000 7fc2b4438740 1 -- 172.21.15.62:0/1527844830 >> 172.21.15.62:0/1527844830 conn(0x55c5d03d7000 msgr2=0x55c5d04c6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:23.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.961+0000 7fc2b4438740 1 -- 172.21.15.62:0/1527844830 shutdown_connections 2024-03-23T11:32:23.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:23.962+0000 7fc2b4438740 1 -- 172.21.15.62:0/1527844830 wait complete. 2024-03-23T11:32:23.962 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.248885, result: 2024-03-23T11:32:23.962 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes --filter-tag mytag123 cephfs_data 2024-03-23T11:32:24.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.085+0000 7fb07de4f740 1 -- start start 2024-03-23T11:32:24.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.085+0000 7fb07de4f740 1 Processor -- start 2024-03-23T11:32:24.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.086+0000 7fb07de4f740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f085d0c00 0x558f08658580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.087+0000 7fb07de4f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f085d1800 0x558f08658b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.087+0000 7fb07de4f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f085d1c00 0x558f08659080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.087+0000 7fb07de4f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f086a2a00 con 0x558f085d1800 2024-03-23T11:32:24.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.087+0000 7fb07de4f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f086a2780 con 0x558f085d1c00 2024-03-23T11:32:24.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.087+0000 7fb07de4f740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558f086a2640 con 0x558f085d0c00 2024-03-23T11:32:24.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.087+0000 7fb07ac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f085d1800 0x558f08658b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.087+0000 7fb07ac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f085d1800 0x558f08658b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60736/0 (socket says 172.21.15.62:60736) 2024-03-23T11:32:24.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.087+0000 7fb07ac00640 1 -- 172.21.15.62:0/3512760686 learned_addr learned my addr 172.21.15.62:0/3512760686 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:24.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.087+0000 7fb07b600640 1 --2- 172.21.15.62:0/3512760686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f085d0c00 0x558f08658580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.087+0000 7fb07c000640 1 --2- 172.21.15.62:0/3512760686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f085d1c00 0x558f08659080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb07ac00640 1 -- 172.21.15.62:0/3512760686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f085d0c00 msgr2=0x558f08658580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:24.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb07ac00640 1 --2- 172.21.15.62:0/3512760686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f085d0c00 0x558f08658580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb07ac00640 1 -- 172.21.15.62:0/3512760686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f085d1c00 msgr2=0x558f08659080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb07ac00640 1 --2- 172.21.15.62:0/3512760686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f085d1c00 0x558f08659080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb07ac00640 1 -- 172.21.15.62:0/3512760686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f085f69c0 con 0x558f085d1800 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb07b600640 1 --2- 172.21.15.62:0/3512760686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f085d0c00 0x558f08658580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb07ac00640 1 --2- 172.21.15.62:0/3512760686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f085d1800 0x558f08658b00 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x558f093e6a80 tx=0x558f093f20c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3977bfc6e3c102c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb078400640 1 -- 172.21.15.62:0/3512760686 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f08617080 con 0x558f085d1800 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb078400640 1 -- 172.21.15.62:0/3512760686 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558f0862c480 con 0x558f085d1800 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb078400640 1 -- 172.21.15.62:0/3512760686 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f08617600 con 0x558f085d1800 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb07de4f740 1 -- 172.21.15.62:0/3512760686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f085f6d00 con 0x558f085d1800 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb07de4f740 1 -- 172.21.15.62:0/3512760686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x558f085f69c0 con 0x558f085d1800 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.088+0000 7fb07de4f740 4 waiting for MDS map... 2024-03-23T11:32:24.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.089+0000 7fb078400640 1 -- 172.21.15.62:0/3512760686 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x558f08626c40 con 0x558f085d1800 2024-03-23T11:32:24.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.089+0000 7fb078400640 1 -- 172.21.15.62:0/3512760686 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 102) v1 ==== 2573+0+0 (secure 0 0 0) 0x558f08620f00 con 0x558f085d1800 2024-03-23T11:32:24.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.089+0000 7fb07de4f740 4 Got MDS map 102 2024-03-23T11:32:24.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.089+0000 7fb07de4f740 10 datascan.parse_kwarg: Applying tag filter: 'mytag123' 2024-03-23T11:32:24.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.089+0000 7fb07de4f740 4 datascan.main: Using metadata pool output 2024-03-23T11:32:24.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.089+0000 7fb07de4f740 4 datascan.main: connecting to RADOS... 2024-03-23T11:32:24.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.090+0000 7fb07de4f740 1 Processor -- start 2024-03-23T11:32:24.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.090+0000 7fb07de4f740 1 -- start start 2024-03-23T11:32:24.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.090+0000 7fb07de4f740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f093adc00 0x558f08659600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07de4f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f094e8000 0x558f08659b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07de4f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f094e8400 0x558f0865a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07de4f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f08619e00 con 0x558f094e8400 2024-03-23T11:32:24.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07de4f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f086a2780 con 0x558f094e8000 2024-03-23T11:32:24.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07de4f740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558f086a2a00 con 0x558f093adc00 2024-03-23T11:32:24.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07ac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f094e8400 0x558f0865a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07ac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f094e8400 0x558f0865a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60738/0 (socket says 172.21.15.62:60738) 2024-03-23T11:32:24.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07ac00640 1 -- 172.21.15.62:0/3066317387 learned_addr learned my addr 172.21.15.62:0/3066317387 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:24.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07c000640 1 --2- 172.21.15.62:0/3066317387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f093adc00 0x558f08659600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07b600640 1 --2- 172.21.15.62:0/3066317387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f094e8000 0x558f08659b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07ac00640 1 -- 172.21.15.62:0/3066317387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f093adc00 msgr2=0x558f08659600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:24.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07ac00640 1 --2- 172.21.15.62:0/3066317387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f093adc00 0x558f08659600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07ac00640 1 -- 172.21.15.62:0/3066317387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f094e8000 msgr2=0x558f08659b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:24.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07ac00640 1 --2- 172.21.15.62:0/3066317387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f094e8000 0x558f08659b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07ac00640 1 -- 172.21.15.62:0/3066317387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f085f6b60 con 0x558f094e8400 2024-03-23T11:32:24.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07b600640 1 --2- 172.21.15.62:0/3066317387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f094e8000 0x558f08659b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:24.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.091+0000 7fb07c000640 1 --2- 172.21.15.62:0/3066317387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f093adc00 0x558f08659600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:24.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.092+0000 7fb07ac00640 1 --2- 172.21.15.62:0/3066317387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f094e8400 0x558f0865a100 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x558f094ee150 tx=0x558f093641e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7999ff52f6b17b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.092+0000 7fb075c00640 1 -- 172.21.15.62:0/3066317387 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f086171e0 con 0x558f094e8400 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.092+0000 7fb075c00640 1 -- 172.21.15.62:0/3066317387 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558f086b4480 con 0x558f094e8400 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.092+0000 7fb075c00640 1 -- 172.21.15.62:0/3066317387 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f08616f20 con 0x558f094e8400 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.092+0000 7fb07de4f740 1 -- 172.21.15.62:0/3066317387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f094e8400 msgr2=0x558f0865a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.092+0000 7fb07de4f740 1 --2- 172.21.15.62:0/3066317387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f094e8400 0x558f0865a100 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x558f094ee150 tx=0x558f093641e0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.093+0000 7fb07de4f740 1 -- 172.21.15.62:0/3066317387 shutdown_connections 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.093+0000 7fb07de4f740 1 --2- 172.21.15.62:0/3066317387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f094e8400 0x558f0865a100 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.093+0000 7fb07de4f740 1 --2- 172.21.15.62:0/3066317387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f094e8000 0x558f08659b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.093+0000 7fb07de4f740 1 --2- 172.21.15.62:0/3066317387 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f093adc00 0x558f08659600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.093+0000 7fb07de4f740 1 -- 172.21.15.62:0/3066317387 >> 172.21.15.62:0/3066317387 conn(0x558f093ad800 msgr2=0x558f0934f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.093+0000 7fb07de4f740 1 -- 172.21.15.62:0/3066317387 shutdown_connections 2024-03-23T11:32:24.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.093+0000 7fb07de4f740 1 -- 172.21.15.62:0/3066317387 wait complete. 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.093+0000 7fb07de4f740 1 Processor -- start 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.094+0000 7fb07de4f740 1 -- start start 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.094+0000 7fb07de4f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f093adc00 0x558f08659600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.094+0000 7fb07de4f740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f094e8000 0x558f08659b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.094+0000 7fb07de4f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f094e8800 0x558f0865a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.094+0000 7fb07de4f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f086a2500 con 0x558f093adc00 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.094+0000 7fb07de4f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f086a23c0 con 0x558f094e8800 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.094+0000 7fb07de4f740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558f086a2280 con 0x558f094e8000 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07c000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f093adc00 0x558f08659600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07c000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f093adc00 0x558f08659600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60750/0 (socket says 172.21.15.62:60750) 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 learned_addr learned my addr 172.21.15.62:0/2435130711 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:24.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07b600640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f094e8000 0x558f08659b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07ac00640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f094e8800 0x558f0865a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f094e8000 msgr2=0x558f08659b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07c000640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f094e8000 0x558f08659b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f094e8800 msgr2=0x558f0865a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07c000640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f094e8800 0x558f0865a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f085f6d00 con 0x558f093adc00 2024-03-23T11:32:24.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07ac00640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f094e8800 0x558f0865a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:24.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.095+0000 7fb07c000640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f093adc00 0x558f08659600 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x558f094ef4a0 tx=0x558f093c4c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.096+0000 7fb073e00640 1 -- 172.21.15.62:0/2435130711 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f086171e0 con 0x558f093adc00 2024-03-23T11:32:24.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.096+0000 7fb073e00640 1 -- 172.21.15.62:0/2435130711 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558f086b5800 con 0x558f093adc00 2024-03-23T11:32:24.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.096+0000 7fb073e00640 1 -- 172.21.15.62:0/2435130711 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f08616f20 con 0x558f093adc00 2024-03-23T11:32:24.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.096+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558f085f69c0 con 0x558f093adc00 2024-03-23T11:32:24.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.096+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f085f6b60 con 0x558f093adc00 2024-03-23T11:32:24.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.096+0000 7fb07de4f740 4 datascan.init: resolving metadata pool 16 2024-03-23T11:32:24.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.098+0000 7fb073e00640 1 -- 172.21.15.62:0/2435130711 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x558f08602840 con 0x558f093adc00 2024-03-23T11:32:24.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.098+0000 7fb073e00640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558f09508c00 0x558f0865b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.098+0000 7fb073e00640 1 -- 172.21.15.62:0/2435130711 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x558f08626a80 con 0x558f093adc00 2024-03-23T11:32:24.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.099+0000 7fb07de4f740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-23T11:32:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.099+0000 7fb07de4f740 4 datascan.main: data pool 'cephfs_data' has ID 17 2024-03-23T11:32:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.099+0000 7fb07de4f740 4 datascan.main: opening data pool 'cephfs_data' 2024-03-23T11:32:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.099+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x558f094e8c00 0x558f0865ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.099+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:1 16.14 16:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f094e9000 con 0x558f094e8c00 2024-03-23T11:32:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.099+0000 7fb07ac00640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x558f094e8c00 0x558f0865ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.099+0000 7fb07b600640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558f09508c00 0x558f0865b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.100+0000 7fb07ac00640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x558f094e8c00 0x558f0865ac00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.100+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv36 ondisk = 0) v8 ==== 160+0+16 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.100+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x558f09509000 0x558f0865b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.100+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:2 16.1f 16:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f09509400 con 0x558f09509000 2024-03-23T11:32:24.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.100+0000 7fb07b600640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558f09508c00 0x558f0865b700 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x558f086ae690 tx=0x558f093c5da0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.101+0000 7fb07c000640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x558f09509000 0x558f0865b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.102+0000 7fb07c000640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x558f09509000 0x558f0865b180 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.102+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+16 (crc 0 0 0) 0x558f0948ad80 con 0x558f09509000 2024-03-23T11:32:24.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.102+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x558f09509800 0x558f08946000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.102+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:3 17.0 17:00000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f094e9400 con 0x558f09509800 2024-03-23T11:32:24.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.103+0000 7fb07b600640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x558f09509800 0x558f08946000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.104+0000 7fb07b600640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x558f09509800 0x558f08946000 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.105+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.105+0000 7fb075200640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x558f09509c00 0x558f08947b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.105+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:4 17.20 17:04000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958000 con 0x558f09509c00 2024-03-23T11:32:24.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.105+0000 7fb07ac00640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x558f09509c00 0x558f08947b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.106+0000 7fb07ac00640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x558f09509c00 0x558f08947b80 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.106+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(4 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f09509c00 2024-03-23T11:32:24.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.106+0000 7fb075c00640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x558f08958800 0x558f08948100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.106+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:5 17.10 17:08000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958c00 con 0x558f08958800 2024-03-23T11:32:24.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.106+0000 7fb07c000640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x558f08958800 0x558f08948100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.107+0000 7fb07c000640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x558f08958800 0x558f08948100 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.107+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== osd_op_reply(5 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.107+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:6 17.30 17:0c000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958400 con 0x558f094e8c00 2024-03-23T11:32:24.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.108+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 2 ==== osd_op_reply(6 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.108+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:7 17.8 17:10000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959000 con 0x558f09509800 2024-03-23T11:32:24.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.109+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 2 ==== osd_op_reply(7 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.109+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:8 17.28 17:14000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959400 con 0x558f08958800 2024-03-23T11:32:24.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.109+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 2 ==== osd_op_reply(8 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.109+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:9 17.18 17:18000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895e000 con 0x558f094e8c00 2024-03-23T11:32:24.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.109+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 3 ==== osd_op_reply(9 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.109+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:10 17.38 17:1c000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959800 con 0x558f09509800 2024-03-23T11:32:24.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.110+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 3 ==== osd_op_reply(10 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.110+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:11 17.4 17:20000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895e400 con 0x558f094e8c00 2024-03-23T11:32:24.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.111+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 4 ==== osd_op_reply(11 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.111+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:12 17.24 17:24000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959c00 con 0x558f08958800 2024-03-23T11:32:24.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.111+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 3 ==== osd_op_reply(12 [pgnls-filter start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.111+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:13 17.14 17:28000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895e800 con 0x558f08958800 2024-03-23T11:32:24.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.112+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 4 ==== osd_op_reply(13 [pgnls-filter start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.112+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:14 17.34 17:2c000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958c00 con 0x558f094e8c00 2024-03-23T11:32:24.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.112+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 5 ==== osd_op_reply(14 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.112+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:15 17.c 17:30000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959400 con 0x558f09509800 2024-03-23T11:32:24.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.113+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 4 ==== osd_op_reply(15 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.113+0000 7fb075200640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x558f094e9800 0x558f08948680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.113+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:16 17.2c 17:34000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f094e9400 con 0x558f094e9800 2024-03-23T11:32:24.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.114+0000 7fb07b600640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x558f094e9800 0x558f08948680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.115+0000 7fb07b600640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x558f094e9800 0x558f08948680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.115+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== osd_op_reply(16 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.115+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:17 17.1c 17:38000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959c00 con 0x558f09509c00 2024-03-23T11:32:24.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.116+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.1 v2:172.21.15.62:6824/37001 2 ==== osd_op_reply(17 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f09509c00 2024-03-23T11:32:24.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.116+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:18 17.3c 17:3c000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f09509400 con 0x558f094e8c00 2024-03-23T11:32:24.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.116+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 6 ==== osd_op_reply(18 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.116+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:19 17.2 17:40000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959000 con 0x558f094e9800 2024-03-23T11:32:24.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.117+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 2 ==== osd_op_reply(19 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.117+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:20 17.22 17:44000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f09509400 con 0x558f09509000 2024-03-23T11:32:24.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.118+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.5 v2:172.21.15.120:6824/37642 2 ==== osd_op_reply(20 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f09509000 2024-03-23T11:32:24.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.118+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:21 17.12 17:48000000::::head [pgnls-filter start_epoch 122 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959800 con 0x558f08958800 2024-03-23T11:32:24.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.118+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 5 ==== osd_op_reply(21 [pgnls-filter start_epoch 122 out=81b] v122'23 uv23 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.118+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:22 17.0 17:00000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f094e8400 con 0x558f09509800 2024-03-23T11:32:24.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.119+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 5 ==== osd_op_reply(22 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.120+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:23 17.20 17:04000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895e000 con 0x558f09509c00 2024-03-23T11:32:24.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.120+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.1 v2:172.21.15.62:6824/37001 3 ==== osd_op_reply(23 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f09509c00 2024-03-23T11:32:24.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.120+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:24 17.10 17:08000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958000 con 0x558f08958800 2024-03-23T11:32:24.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.120+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 6 ==== osd_op_reply(24 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.120+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:25 17.30 17:0c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895e400 con 0x558f094e8c00 2024-03-23T11:32:24.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.121+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 7 ==== osd_op_reply(25 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.121+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:26 17.8 17:10000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958400 con 0x558f09509800 2024-03-23T11:32:24.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.122+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 6 ==== osd_op_reply(26 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.122+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:27 17.28 17:14000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895ec00 con 0x558f08958800 2024-03-23T11:32:24.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.122+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 7 ==== osd_op_reply(27 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.122+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:28 17.18 17:18000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958c00 con 0x558f094e8c00 2024-03-23T11:32:24.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.123+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 8 ==== osd_op_reply(28 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.123+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:29 17.38 17:1c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895f000 con 0x558f09509800 2024-03-23T11:32:24.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.124+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 7 ==== osd_op_reply(29 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.124+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:30 17.4 17:20000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959c00 con 0x558f094e8c00 2024-03-23T11:32:24.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.124+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 9 ==== osd_op_reply(30 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.124+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:31 17.24 17:24000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895f400 con 0x558f08958800 2024-03-23T11:32:24.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.125+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 8 ==== osd_op_reply(31 [pgnls-filter start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.125+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:32 17.14 17:28000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959800 con 0x558f08958800 2024-03-23T11:32:24.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.125+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 9 ==== osd_op_reply(32 [pgnls-filter start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.126+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:33 17.34 17:2c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895e800 con 0x558f094e8c00 2024-03-23T11:32:24.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.126+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 10 ==== osd_op_reply(33 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.126+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:34 17.c 17:30000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958000 con 0x558f09509800 2024-03-23T11:32:24.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.127+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 8 ==== osd_op_reply(34 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.127+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:35 17.2c 17:34000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895ec00 con 0x558f094e9800 2024-03-23T11:32:24.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.128+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 3 ==== osd_op_reply(35 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.128+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:36 17.1c 17:38000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959400 con 0x558f09509c00 2024-03-23T11:32:24.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.129+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.1 v2:172.21.15.62:6824/37001 4 ==== osd_op_reply(36 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f09509c00 2024-03-23T11:32:24.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.129+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:37 17.3c 17:3c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895f800 con 0x558f094e8c00 2024-03-23T11:32:24.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.129+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 11 ==== osd_op_reply(37 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.129+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:38 17.2 17:40000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f094e9400 con 0x558f094e9800 2024-03-23T11:32:24.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.130+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 4 ==== osd_op_reply(38 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.130+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:39 17.22 17:44000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895fc00 con 0x558f09509000 2024-03-23T11:32:24.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.131+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.5 v2:172.21.15.120:6824/37642 3 ==== osd_op_reply(39 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f09509000 2024-03-23T11:32:24.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.131+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:40 17.12 17:48000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f094e8400 con 0x558f08958800 2024-03-23T11:32:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.132+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 10 ==== osd_op_reply(40 [pgnls-filter start_epoch 122 out=49b] v122'23 uv23 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.132+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:41 17.32 17:4c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959000 con 0x558f09509800 2024-03-23T11:32:24.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.133+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 9 ==== osd_op_reply(41 [pgnls-filter start_epoch 122 out=49b] v120'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.133+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:42 17.a 17:50000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f09509400 con 0x558f09509800 2024-03-23T11:32:24.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.134+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 10 ==== osd_op_reply(42 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.134+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:43 17.2a 17:54000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958400 con 0x558f08958800 2024-03-23T11:32:24.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.135+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 11 ==== osd_op_reply(43 [pgnls-filter start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.135+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:44 17.1a 17:58000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f09509400 con 0x558f09509000 2024-03-23T11:32:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.136+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.5 v2:172.21.15.120:6824/37642 4 ==== osd_op_reply(44 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f09509000 2024-03-23T11:32:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.136+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:45 17.3a 17:5c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f09509400 con 0x558f09509c00 2024-03-23T11:32:24.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.136+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.1 v2:172.21.15.62:6824/37001 5 ==== osd_op_reply(45 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f09509c00 2024-03-23T11:32:24.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.136+0000 7fb075200640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x558f08958000 0x558f08948c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.136+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:46 17.6 17:60000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959000 con 0x558f08958000 2024-03-23T11:32:24.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.137+0000 7fb07ac00640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x558f08958000 0x558f08948c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.138+0000 7fb07ac00640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x558f08958000 0x558f08948c00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.139+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== osd_op_reply(46 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f08958000 2024-03-23T11:32:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.139+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:47 17.26 17:64000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959800 con 0x558f094e8c00 2024-03-23T11:32:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.140+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 12 ==== osd_op_reply(47 [pgnls-filter start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.140+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:48 17.16 17:68000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f094e9400 con 0x558f09509000 2024-03-23T11:32:24.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.141+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.5 v2:172.21.15.120:6824/37642 5 ==== osd_op_reply(48 [pgnls-filter start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f09509000 2024-03-23T11:32:24.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.141+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:49 17.36 17:6c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958400 con 0x558f094e8c00 2024-03-23T11:32:24.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.142+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 13 ==== osd_op_reply(49 [pgnls-filter start_epoch 122 out=81b] v122'34 uv34 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.142+0000 7fb07de4f740 20 datascan.forall_objects: OSD matched oid 100000001f7.00000000 2024-03-23T11:32:24.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.142+0000 7fb07de4f740 10 datascan.operator(): handling object 100000001f7.0 2024-03-23T11:32:24.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.142+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:50 17.36 17:6fe38dc2:::100000001f7.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f094e9000 con 0x558f094e8c00 2024-03-23T11:32:24.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.143+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 14 ==== osd_op_reply(50 100000001f7.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (126) out=126b,getxattr (30) out=30b,getxattr] v0'0 uv34 ondisk = 0) v8 ==== 416+0+194 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.143+0000 7fb07de4f740 10 datascan.inject_with_backtrace: inode: 0x0x100000001f7 2024-03-23T11:32:24.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.143+0000 7fb07de4f740 10 datascan.inject_with_backtrace: backptr: 0x0x10000000001/bravo 2024-03-23T11:32:24.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.143+0000 7fb07de4f740 20 datascan.get_frag_of: dirino=0x10000000001 target_dname=bravo 2024-03-23T11:32:24.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.143+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:51 16.7 16:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895e000 con 0x558f08958800 2024-03-23T11:32:24.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.144+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 12 ==== osd_op_reply(51 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.144+0000 7fb07de4f740 20 datascan.get_frag_of: got backtrace parent 0x10000000000/flushed 2024-03-23T11:32:24.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.144+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x558f0895e400 0x558f08946b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.144+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:52 16.10 16:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895e800 con 0x558f0895e400 2024-03-23T11:32:24.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.144+0000 7fb07c000640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x558f0895e400 0x558f08946b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.145+0000 7fb07c000640 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x558f0895e400 0x558f08946b00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.146+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.2 v2:172.21.15.62:6800/36996 1 ==== osd_op_reply(52 10000000000.00000000 [omap-get-vals-by-keys out=524b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+524 (crc 0 0 0) 0x558f0948ad80 con 0x558f0895e400 2024-03-23T11:32:24.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.146+0000 7fb07de4f740 20 datascan.read_dentry: oid=10000000000.00000000 dname=flushed frag=*, r=0 2024-03-23T11:32:24.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.146+0000 7fb07de4f740 20 datascan.get_frag_of: fast path, fragtree is fragtree_t() 2024-03-23T11:32:24.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.146+0000 7fb07de4f740 20 datascan.get_frag_of: frag is * 2024-03-23T11:32:24.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.146+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:53 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895f800 con 0x558f08958800 2024-03-23T11:32:24.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.147+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 13 ==== osd_op_reply(53 10000000001.00000000 [omap-get-header out=274b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.147+0000 7fb07de4f740 10 datascan.find_or_create_dirfrag: read_version = 8 2024-03-23T11:32:24.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.147+0000 7fb07de4f740 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000001 * 2024-03-23T11:32:24.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.147+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:54 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958c00 con 0x558f08958800 2024-03-23T11:32:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.147+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 14 ==== osd_op_reply(54 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.147+0000 7fb07de4f740 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-03-23T11:32:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.147+0000 7fb07de4f740 20 datascan.read_dentry: bravo_head not found in result 2024-03-23T11:32:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.147+0000 7fb07de4f740 10 datascan.inject_with_backtrace: Linking inode 0x0x100000001f7 at 0x0x10000000001/bravo with size=5 bytes 2024-03-23T11:32:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.147+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:55 16.7 16:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959c00 con 0x558f08958800 2024-03-23T11:32:24.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.150+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 15 ==== osd_op_reply(55 10000000001.00000000 [omap-set-vals] v122'9 uv9 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.150+0000 7fb07de4f740 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-03-23T11:32:24.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.150+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:56 17.e 17:70000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959400 con 0x558f0895e400 2024-03-23T11:32:24.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.151+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.2 v2:172.21.15.62:6800/36996 2 ==== osd_op_reply(56 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f0895e400 2024-03-23T11:32:24.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.151+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:57 17.2e 17:74000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f094e8400 con 0x558f08958000 2024-03-23T11:32:24.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.152+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.6 v2:172.21.15.120:6800/37641 2 ==== osd_op_reply(57 [pgnls-filter start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f08958000 2024-03-23T11:32:24.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.152+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:58 17.1e 17:78000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895f000 con 0x558f09509c00 2024-03-23T11:32:24.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.153+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.1 v2:172.21.15.62:6824/37001 6 ==== osd_op_reply(58 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f09509c00 2024-03-23T11:32:24.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.153+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:59 17.3e 17:7c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895f400 con 0x558f094e9800 2024-03-23T11:32:24.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.154+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 5 ==== osd_op_reply(59 [pgnls-filter start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.154+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:60 17.1 17:80000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895ec00 con 0x558f0895e400 2024-03-23T11:32:24.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.155+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.2 v2:172.21.15.62:6800/36996 3 ==== osd_op_reply(60 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f0895e400 2024-03-23T11:32:24.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.155+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:61 17.21 17:84000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895fc00 con 0x558f094e8c00 2024-03-23T11:32:24.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.156+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 15 ==== osd_op_reply(61 [pgnls-filter start_epoch 122 out=49b] v120'24 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.156+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:62 17.11 17:88000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f09509400 con 0x558f08958800 2024-03-23T11:32:24.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.156+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 16 ==== osd_op_reply(62 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.156+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:63 17.31 17:8c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08978000 con 0x558f09509c00 2024-03-23T11:32:24.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.157+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.1 v2:172.21.15.62:6824/37001 7 ==== osd_op_reply(63 [pgnls-filter start_epoch 122 out=49b] v120'15 uv8 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f09509c00 2024-03-23T11:32:24.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.157+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:64 17.9 17:90000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958c00 con 0x558f08958800 2024-03-23T11:32:24.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.158+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 17 ==== osd_op_reply(64 [pgnls-filter start_epoch 122 out=49b] v120'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.158+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:65 17.29 17:94000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08978400 con 0x558f094e9800 2024-03-23T11:32:24.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.159+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 6 ==== osd_op_reply(65 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.159+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:66 17.19 17:98000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959c00 con 0x558f09509000 2024-03-23T11:32:24.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.161+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.5 v2:172.21.15.120:6824/37642 6 ==== osd_op_reply(66 [pgnls-filter start_epoch 122 out=49b] v122'24 uv24 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f09509000 2024-03-23T11:32:24.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.161+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:67 17.39 17:9c000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08978800 con 0x558f09509000 2024-03-23T11:32:24.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.162+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.5 v2:172.21.15.120:6824/37642 7 ==== osd_op_reply(67 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f09509000 2024-03-23T11:32:24.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.162+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:68 17.5 17:a0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959400 con 0x558f08958000 2024-03-23T11:32:24.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.164+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.6 v2:172.21.15.120:6800/37641 3 ==== osd_op_reply(68 [pgnls-filter start_epoch 122 out=49b] v122'15 uv15 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f08958000 2024-03-23T11:32:24.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.164+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:69 17.25 17:a4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08978c00 con 0x558f09509c00 2024-03-23T11:32:24.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.164+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.1 v2:172.21.15.62:6824/37001 8 ==== osd_op_reply(69 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f09509c00 2024-03-23T11:32:24.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.164+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:70 17.15 17:a8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895e000 con 0x558f09509000 2024-03-23T11:32:24.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.165+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.5 v2:172.21.15.120:6824/37642 8 ==== osd_op_reply(70 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f09509000 2024-03-23T11:32:24.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.165+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:71 17.35 17:ac000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08979000 con 0x558f094e9800 2024-03-23T11:32:24.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.167+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 7 ==== osd_op_reply(71 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.167+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:72 17.d 17:b0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895e800 con 0x558f09509800 2024-03-23T11:32:24.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.168+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 11 ==== osd_op_reply(72 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.168+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:73 17.2d 17:b4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08979400 con 0x558f09509800 2024-03-23T11:32:24.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.169+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 12 ==== osd_op_reply(73 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.169+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:74 17.1d 17:b8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895f800 con 0x558f09509800 2024-03-23T11:32:24.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.170+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 13 ==== osd_op_reply(74 [pgnls-filter start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.170+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:75 17.3d 17:bc000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08979800 con 0x558f09509800 2024-03-23T11:32:24.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.171+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 14 ==== osd_op_reply(75 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.171+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:76 17.3 17:c0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f094e9400 con 0x558f094e9800 2024-03-23T11:32:24.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.173+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 8 ==== osd_op_reply(76 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.173+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:77 17.23 17:c4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08979c00 con 0x558f094e9800 2024-03-23T11:32:24.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.174+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 9 ==== osd_op_reply(77 [pgnls-filter start_epoch 122 out=49b] v120'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.174+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:78 17.13 17:c8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08978400 con 0x558f08958800 2024-03-23T11:32:24.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.175+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.3 v2:172.21.15.62:6808/37000 18 ==== osd_op_reply(78 [pgnls-filter start_epoch 122 out=49b] v120'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f08958800 2024-03-23T11:32:24.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.175+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:79 17.33 17:cc000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959000 con 0x558f09509800 2024-03-23T11:32:24.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.176+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.7 v2:172.21.15.120:6816/37655 15 ==== osd_op_reply(79 [pgnls-filter start_epoch 122 out=49b] v120'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f09509800 2024-03-23T11:32:24.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.176+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:80 17.b 17:d0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08979000 con 0x558f094e8c00 2024-03-23T11:32:24.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.177+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 16 ==== osd_op_reply(80 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.177+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:81 17.2b 17:d4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959800 con 0x558f094e9800 2024-03-23T11:32:24.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.178+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 10 ==== osd_op_reply(81 [pgnls-filter start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.178+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:82 17.1b 17:d8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895f400 con 0x558f094e9800 2024-03-23T11:32:24.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.180+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 11 ==== osd_op_reply(82 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.180+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:83 17.3b 17:dc000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08958400 con 0x558f0895e400 2024-03-23T11:32:24.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.181+0000 7fb07c000640 1 -- 172.21.15.62:0/2435130711 <== osd.2 v2:172.21.15.62:6800/36996 4 ==== osd_op_reply(83 [pgnls-filter start_epoch 122 out=49b] v120'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ad80 con 0x558f0895e400 2024-03-23T11:32:24.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.181+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:84 17.7 17:e0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895e800 con 0x558f094e8c00 2024-03-23T11:32:24.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.182+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 17 ==== osd_op_reply(84 [pgnls-filter start_epoch 122 out=49b] v120'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.182+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:85 17.27 17:e4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08959400 con 0x558f094e9800 2024-03-23T11:32:24.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.183+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 12 ==== osd_op_reply(85 [pgnls-filter start_epoch 122 out=49b] v120'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.183+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:86 17.17 17:e8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895f800 con 0x558f08958000 2024-03-23T11:32:24.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.184+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.6 v2:172.21.15.120:6800/37641 4 ==== osd_op_reply(86 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f08958000 2024-03-23T11:32:24.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.184+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:87 17.37 17:ec000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08979400 con 0x558f094e8c00 2024-03-23T11:32:24.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.185+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 18 ==== osd_op_reply(87 [pgnls-filter start_epoch 122 out=49b] v120'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.185+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:88 17.f 17:f0000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895f000 con 0x558f09509c00 2024-03-23T11:32:24.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.186+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.1 v2:172.21.15.62:6824/37001 9 ==== osd_op_reply(88 [pgnls-filter start_epoch 122 out=49b] v120'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f09509c00 2024-03-23T11:32:24.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.186+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:89 17.2f 17:f4000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08979800 con 0x558f094e9800 2024-03-23T11:32:24.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.187+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 <== osd.4 v2:172.21.15.120:6803/37640 13 ==== osd_op_reply(89 [pgnls-filter start_epoch 122 out=49b] v120'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948afc0 con 0x558f094e9800 2024-03-23T11:32:24.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.187+0000 7fb075200640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:90 17.1f 17:f8000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f0895fc00 con 0x558f08958000 2024-03-23T11:32:24.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.188+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.6 v2:172.21.15.120:6800/37641 5 ==== osd_op_reply(90 [pgnls-filter start_epoch 122 out=49b] v120'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f08958000 2024-03-23T11:32:24.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.189+0000 7fb075c00640 1 -- 172.21.15.62:0/2435130711 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:91 17.3f 17:fc000000::::head [pgnls-filter start_epoch 122 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x558f08978000 con 0x558f094e8c00 2024-03-23T11:32:24.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.189+0000 7fb07ac00640 1 -- 172.21.15.62:0/2435130711 <== osd.0 v2:172.21.15.62:6811/36998 19 ==== osd_op_reply(91 [pgnls-filter start_epoch 122 out=49b] v120'17 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x558f0948ab40 con 0x558f094e8c00 2024-03-23T11:32:24.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x558f094e8c00 msgr2=0x558f0865ac00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x558f094e8c00 0x558f0865ac00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x558f09509c00 msgr2=0x558f08947b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x558f09509c00 0x558f08947b80 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x558f0895e400 msgr2=0x558f08946b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x558f0895e400 0x558f08946b00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x558f08958800 msgr2=0x558f08948100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x558f08958800 0x558f08948100 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 reap_dead start 2024-03-23T11:32:24.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x558f094e9800 msgr2=0x558f08948680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x558f094e9800 0x558f08948680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x558f09509000 msgr2=0x558f0865b180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x558f09509000 0x558f0865b180 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x558f08958000 msgr2=0x558f08948c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x558f08958000 0x558f08948c00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07b600640 1 -- 172.21.15.62:0/2435130711 reap_dead start 2024-03-23T11:32:24.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x558f09509800 msgr2=0x558f08946000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.190+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x558f09509800 0x558f08946000 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.191+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558f09508c00 msgr2=0x558f0865b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.191+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558f09508c00 0x558f0865b700 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x558f086ae690 tx=0x558f093c5da0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.191+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f093adc00 msgr2=0x558f08659600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.191+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f093adc00 0x558f08659600 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x558f094ef4a0 tx=0x558f093c4c00 comp rx=0 tx=0).stop 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.191+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 shutdown_connections 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.191+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x558f09509800 0x558f08946000 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.191+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x558f09508c00 0x558f0865b700 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.191+0000 7fb07de4f740 1 --2- 172.21.15.62:0/2435130711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f093adc00 0x558f08659600 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.191+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 >> 172.21.15.62:0/2435130711 conn(0x558f093ad800 msgr2=0x558f0934f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.191+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 shutdown_connections 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.191+0000 7fb07de4f740 1 -- 172.21.15.62:0/2435130711 wait complete. 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.192+0000 7fb07de4f740 1 -- 172.21.15.62:0/3512760686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f085d1800 msgr2=0x558f08658b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.192+0000 7fb07de4f740 1 --2- 172.21.15.62:0/3512760686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f085d1800 0x558f08658b00 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x558f093e6a80 tx=0x558f093f20c0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.193+0000 7fb07de4f740 1 -- 172.21.15.62:0/3512760686 shutdown_connections 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.193+0000 7fb07de4f740 1 --2- 172.21.15.62:0/3512760686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f085d1800 0x558f08658b00 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.193+0000 7fb07de4f740 1 --2- 172.21.15.62:0/3512760686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f085d1c00 0x558f08659080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.193+0000 7fb07de4f740 1 --2- 172.21.15.62:0/3512760686 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f085d0c00 0x558f08658580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.193+0000 7fb07de4f740 1 -- 172.21.15.62:0/3512760686 >> 172.21.15.62:0/3512760686 conn(0x558f085d1000 msgr2=0x558f0934e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.193+0000 7fb07de4f740 1 -- 172.21.15.62:0/3512760686 shutdown_connections 2024-03-23T11:32:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.193+0000 7fb07de4f740 1 -- 172.21.15.62:0/3512760686 wait complete. 2024-03-23T11:32:24.195 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', '--filter-tag', 'mytag123', 'cephfs_data'] in time 0:00:00.232599, result: 2024-03-23T11:32:24.195 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-03-23T11:32:24.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.312+0000 7f5149bf7740 1 -- start start 2024-03-23T11:32:24.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.312+0000 7f5149bf7740 1 Processor -- start 2024-03-23T11:32:24.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.313+0000 7f5149bf7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebdc74c00 0x55cebdcfc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.313+0000 7f5149bf7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebdc75800 0x55cebdcfcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.313+0000 7f5149bf7740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebdc75c00 0x55cebdcfd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.313+0000 7f5149bf7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cebdd46a00 con 0x55cebdc75800 2024-03-23T11:32:24.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.313+0000 7f5149bf7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cebdd46780 con 0x55cebdc74c00 2024-03-23T11:32:24.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.313+0000 7f5149bf7740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cebdd46640 con 0x55cebdc75c00 2024-03-23T11:32:24.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.313+0000 7f5147200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebdc75800 0x55cebdcfcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.313+0000 7f5147200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebdc75800 0x55cebdcfcb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60760/0 (socket says 172.21.15.62:60760) 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.313+0000 7f5147200640 1 -- 172.21.15.62:0/3731430887 learned_addr learned my addr 172.21.15.62:0/3731430887 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5148600640 1 --2- 172.21.15.62:0/3731430887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebdc75c00 0x55cebdcfd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5147c00640 1 --2- 172.21.15.62:0/3731430887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebdc74c00 0x55cebdcfc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5147200640 1 -- 172.21.15.62:0/3731430887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebdc75c00 msgr2=0x55cebdcfd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5147200640 1 --2- 172.21.15.62:0/3731430887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebdc75c00 0x55cebdcfd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5147200640 1 -- 172.21.15.62:0/3731430887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebdc74c00 msgr2=0x55cebdcfc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5147200640 1 --2- 172.21.15.62:0/3731430887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebdc74c00 0x55cebdcfc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5147200640 1 -- 172.21.15.62:0/3731430887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cebdc9a9c0 con 0x55cebdc75800 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5148600640 1 --2- 172.21.15.62:0/3731430887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebdc75c00 0x55cebdcfd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5147200640 1 --2- 172.21.15.62:0/3731430887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebdc75800 0x55cebdcfcb00 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x55cebea89320 tx=0x55cebea94d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b99d5dad2f008c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5144a00640 1 -- 172.21.15.62:0/3731430887 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55cebdcbb600 con 0x55cebdc75800 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5144a00640 1 -- 172.21.15.62:0/3731430887 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cebdcd0480 con 0x55cebdc75800 2024-03-23T11:32:24.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5144a00640 1 -- 172.21.15.62:0/3731430887 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55cebdcbb1e0 con 0x55cebdc75800 2024-03-23T11:32:24.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5149bf7740 1 -- 172.21.15.62:0/3731430887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cebdc9ab60 con 0x55cebdc75800 2024-03-23T11:32:24.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5149bf7740 1 -- 172.21.15.62:0/3731430887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55cebdc9a820 con 0x55cebdc75800 2024-03-23T11:32:24.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.314+0000 7f5149bf7740 4 waiting for MDS map... 2024-03-23T11:32:24.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.315+0000 7f5144a00640 1 -- 172.21.15.62:0/3731430887 <== mon.0 v2:172.21.15.62:3300/0 4 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x55cebdccac40 con 0x55cebdc75800 2024-03-23T11:32:24.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.315+0000 7f5144a00640 1 -- 172.21.15.62:0/3731430887 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 102) v1 ==== 2573+0+0 (secure 0 0 0) 0x55cebdcc5200 con 0x55cebdc75800 2024-03-23T11:32:24.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.315+0000 7f5149bf7740 4 Got MDS map 102 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.315+0000 7f5149bf7740 4 datascan.main: Using metadata pool output 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.315+0000 7f5149bf7740 4 datascan.main: connecting to RADOS... 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.316+0000 7f5149bf7740 1 Processor -- start 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.316+0000 7f5149bf7740 1 -- start start 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.317+0000 7f5149bf7740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebeb3d000 0x55cebdcfd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.317+0000 7f5149bf7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d400 0x55cebdcfdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.317+0000 7f5149bf7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebeb3d800 0x55cebdcfe100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.317+0000 7f5149bf7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cebdcbde00 con 0x55cebeb3d400 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.317+0000 7f5149bf7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cebdd46780 con 0x55cebeb3d800 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.317+0000 7f5149bf7740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cebdd46500 con 0x55cebeb3d000 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.317+0000 7f5147c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d400 0x55cebdcfdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.317+0000 7f5147c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d400 0x55cebdcfdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60770/0 (socket says 172.21.15.62:60770) 2024-03-23T11:32:24.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.317+0000 7f5147c00640 1 -- 172.21.15.62:0/3008604340 learned_addr learned my addr 172.21.15.62:0/3008604340 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.318+0000 7f5147c00640 1 -- 172.21.15.62:0/3008604340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebeb3d000 msgr2=0x55cebdcfd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.318+0000 7f5147c00640 1 --2- 172.21.15.62:0/3008604340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebeb3d000 0x55cebdcfd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.318+0000 7f5147c00640 1 -- 172.21.15.62:0/3008604340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebeb3d800 msgr2=0x55cebdcfe100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.318+0000 7f5147c00640 1 --2- 172.21.15.62:0/3008604340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebeb3d800 0x55cebdcfe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.318+0000 7f5147c00640 1 -- 172.21.15.62:0/3008604340 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cebdc9a4e0 con 0x55cebeb3d400 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.318+0000 7f5147c00640 1 --2- 172.21.15.62:0/3008604340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d400 0x55cebdcfdb80 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x55cebeb94600 tx=0x55cebea940c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f325f693b62ace54 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.318+0000 7f5142200640 1 -- 172.21.15.62:0/3008604340 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55cebdcbb080 con 0x55cebeb3d400 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.318+0000 7f5142200640 1 -- 172.21.15.62:0/3008604340 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cebea44900 con 0x55cebeb3d400 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.318+0000 7f5142200640 1 -- 172.21.15.62:0/3008604340 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55cebdcbb600 con 0x55cebeb3d400 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.319+0000 7f5149bf7740 1 -- 172.21.15.62:0/3008604340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d400 msgr2=0x55cebdcfdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.319+0000 7f5149bf7740 1 --2- 172.21.15.62:0/3008604340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d400 0x55cebdcfdb80 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x55cebeb94600 tx=0x55cebea940c0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.319+0000 7f5149bf7740 1 -- 172.21.15.62:0/3008604340 shutdown_connections 2024-03-23T11:32:24.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.319+0000 7f5149bf7740 1 --2- 172.21.15.62:0/3008604340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d400 0x55cebdcfdb80 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.319+0000 7f5149bf7740 1 --2- 172.21.15.62:0/3008604340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebeb3d800 0x55cebdcfe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.319+0000 7f5149bf7740 1 --2- 172.21.15.62:0/3008604340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebeb3d000 0x55cebdcfd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.319+0000 7f5149bf7740 1 -- 172.21.15.62:0/3008604340 >> 172.21.15.62:0/3008604340 conn(0x55cebeb3cc00 msgr2=0x55cebe9f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:24.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.319+0000 7f5149bf7740 1 -- 172.21.15.62:0/3008604340 shutdown_connections 2024-03-23T11:32:24.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.319+0000 7f5149bf7740 1 -- 172.21.15.62:0/3008604340 wait complete. 2024-03-23T11:32:24.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.319+0000 7f5149bf7740 1 Processor -- start 2024-03-23T11:32:24.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.320+0000 7f5149bf7740 1 -- start start 2024-03-23T11:32:24.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.320+0000 7f5149bf7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d000 0x55cebdcff180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.320+0000 7f5149bf7740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebeb3d400 0x55cebdcfec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.320+0000 7f5149bf7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebeb3d800 0x55cebdcfe680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.320+0000 7f5149bf7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cebdd463c0 con 0x55cebeb3d000 2024-03-23T11:32:24.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.320+0000 7f5149bf7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cebdd46280 con 0x55cebeb3d800 2024-03-23T11:32:24.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.321+0000 7f5149bf7740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cebdd46140 con 0x55cebeb3d400 2024-03-23T11:32:24.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.321+0000 7f5148600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d000 0x55cebdcff180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.321+0000 7f5148600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d000 0x55cebdcff180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60784/0 (socket says 172.21.15.62:60784) 2024-03-23T11:32:24.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.321+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 learned_addr learned my addr 172.21.15.62:0/2468410830 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:24.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.321+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebeb3d400 msgr2=0x55cebdcfec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.321+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebeb3d400 0x55cebdcfec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.321+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebeb3d800 msgr2=0x55cebdcfe680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.321+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebeb3d800 0x55cebdcfe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.321+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cebdc9ad00 con 0x55cebeb3d000 2024-03-23T11:32:24.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.321+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d000 0x55cebdcff180 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x55cebeb95470 tx=0x55cebeb52cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.322+0000 7f5140400640 1 -- 172.21.15.62:0/2468410830 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55cebdcbb080 con 0x55cebeb3d000 2024-03-23T11:32:24.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.322+0000 7f5140400640 1 -- 172.21.15.62:0/2468410830 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cebdcd0180 con 0x55cebeb3d000 2024-03-23T11:32:24.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.322+0000 7f5140400640 1 -- 172.21.15.62:0/2468410830 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55cebdcbb600 con 0x55cebeb3d000 2024-03-23T11:32:24.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.322+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55cebdc9a4e0 con 0x55cebeb3d000 2024-03-23T11:32:24.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.322+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cebdc9a9c0 con 0x55cebeb3d000 2024-03-23T11:32:24.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.322+0000 7f5149bf7740 4 datascan.init: resolving metadata pool 16 2024-03-23T11:32:24.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.324+0000 7f5140400640 1 -- 172.21.15.62:0/2468410830 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x55cebdca6840 con 0x55cebeb3d000 2024-03-23T11:32:24.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.324+0000 7f5140400640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x55cebebaa800 0x55cebdcff700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.324+0000 7f5140400640 1 -- 172.21.15.62:0/2468410830 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x55cebdccaa80 con 0x55cebeb3d000 2024-03-23T11:32:24.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.324+0000 7f5149bf7740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.324+0000 7f5149bf7740 4 datascan.main: resolving metadata pool 16 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.325+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55cebebaac00 0x55cebdcfe100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.325+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:1 16.0 16:00000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab000 con 0x55cebebaac00 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.325+0000 7f5147200640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55cebebaac00 0x55cebdcfe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.325+0000 7f5147c00640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x55cebebaa800 0x55cebdcff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.325+0000 7f5147200640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55cebebaac00 0x55cebdcfe100 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.326+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== osd_op_reply(1 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.326+0000 7f5147c00640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x55cebebaa800 0x55cebdcff700 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x55cebeb94600 tx=0x55cebea94000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.326+0000 7f5147200640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55cebebab000 0x55cebdfd6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.326+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:2 16.10 16:08000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebeb3c400 con 0x55cebebab000 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.326+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55cebebab000 0x55cebdfd6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.327+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55cebebab000 0x55cebdfd6000 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.327+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 1 ==== osd_op_reply(2 [pgnls start_epoch 122 out=177b] v121'16 uv16 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.327+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:3 16.8 16:10000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebeb3c400 con 0x55cebebaac00 2024-03-23T11:32:24.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.328+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 2 ==== osd_op_reply(3 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.328+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:4 16.18 16:18000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebeb3c400 con 0x55cebebaac00 2024-03-23T11:32:24.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.328+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 3 ==== osd_op_reply(4 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.328+0000 7f5147200640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x55cebeb3c400 0x55cebdfd6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.328+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:5 16.4 16:20000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebeb3c800 con 0x55cebeb3c400 2024-03-23T11:32:24.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.328+0000 7f5147c00640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x55cebeb3c400 0x55cebdfd6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.329+0000 7f5147c00640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x55cebeb3c400 0x55cebdfd6580 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.329+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== osd_op_reply(5 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.329+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:6 16.14 16:28000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebeb3c800 con 0x55cebebaac00 2024-03-23T11:32:24.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.330+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 4 ==== osd_op_reply(6 [pgnls start_epoch 122 out=149b] v122'43 uv43 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.330+0000 7f5147200640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x55cebeb3c800 0x55cebdfd6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.330+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:7 16.c 16:30000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6000 con 0x55cebeb3c800 2024-03-23T11:32:24.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.330+0000 7f5147200640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x55cebeb3c800 0x55cebdfd6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.331+0000 7f5147200640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x55cebeb3c800 0x55cebdfd6b00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.332+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== osd_op_reply(7 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.332+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:8 16.1c 16:38000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6000 con 0x55cebebab000 2024-03-23T11:32:24.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.332+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 2 ==== osd_op_reply(8 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.332+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55cebdfe6000 0x55cebdfd7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.332+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:9 16.2 16:40000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7000 con 0x55cebdfe6000 2024-03-23T11:32:24.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.333+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55cebdfe6000 0x55cebdfd7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.334+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55cebdfe6000 0x55cebdfd7080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.335+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== osd_op_reply(9 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe6000 2024-03-23T11:32:24.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.335+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55cebdfe7000 0x55cebdfd7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.335+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:10 16.12 16:48000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7400 con 0x55cebdfe7000 2024-03-23T11:32:24.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.335+0000 7f5147c00640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55cebdfe7000 0x55cebdfd7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.336+0000 7f5147c00640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55cebdfe7000 0x55cebdfd7600 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.337+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== osd_op_reply(10 [pgnls start_epoch 122 out=81b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebdfe7000 2024-03-23T11:32:24.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.337+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:11 16.a 16:50000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7400 con 0x55cebebab000 2024-03-23T11:32:24.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.337+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 3 ==== osd_op_reply(11 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.337+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:12 16.1a 16:58000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7400 con 0x55cebeb3c400 2024-03-23T11:32:24.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.338+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 2 ==== osd_op_reply(12 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.338+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:13 16.6 16:60000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7400 con 0x55cebdfe7000 2024-03-23T11:32:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.338+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.6 v2:172.21.15.120:6800/37641 2 ==== osd_op_reply(13 [pgnls start_epoch 122 out=77b] v121'10 uv10 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebdfe7000 2024-03-23T11:32:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.338+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:14 16.16 16:68000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7400 con 0x55cebeb3c400 2024-03-23T11:32:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.339+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 3 ==== osd_op_reply(14 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.339+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:15 16.e 16:70000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7400 con 0x55cebebab000 2024-03-23T11:32:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.339+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 4 ==== osd_op_reply(15 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.339+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x55cebdfe7400 0x55cebdfd7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.339+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:16 16.1e 16:78000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7800 con 0x55cebdfe7400 2024-03-23T11:32:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.340+0000 7f5147200640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x55cebdfe7400 0x55cebdfd7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.341+0000 7f5147200640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x55cebdfe7400 0x55cebdfd7b80 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.342+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.7 v2:172.21.15.120:6816/37655 1 ==== osd_op_reply(16 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebdfe7400 2024-03-23T11:32:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.342+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:17 16.1 16:80000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7800 con 0x55cebebab000 2024-03-23T11:32:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.342+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 5 ==== osd_op_reply(17 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.342+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:18 16.11 16:88000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7800 con 0x55cebeb3c800 2024-03-23T11:32:24.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.343+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 2 ==== osd_op_reply(18 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.343+0000 7f5147200640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55cebdfe7800 0x55cebdfd8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:24.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.343+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:19 16.9 16:90000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebdfe7800 2024-03-23T11:32:24.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.344+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55cebdfe7800 0x55cebdfd8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:24.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.344+0000 7f5148600640 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55cebdfe7800 0x55cebdfd8100 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:24.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.345+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== osd_op_reply(19 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe7800 2024-03-23T11:32:24.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.345+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:20 16.19 16:98000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebdfe6000 2024-03-23T11:32:24.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.346+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.4 v2:172.21.15.120:6803/37640 2 ==== osd_op_reply(20 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe6000 2024-03-23T11:32:24.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.346+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:21 16.5 16:a0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebebaac00 2024-03-23T11:32:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.346+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 5 ==== osd_op_reply(21 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.347+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:22 16.15 16:a8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebebaac00 2024-03-23T11:32:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.347+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 6 ==== osd_op_reply(22 [pgnls start_epoch 122 out=97b] v121'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.347+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:23 16.d 16:b0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebdfe7000 2024-03-23T11:32:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.348+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.6 v2:172.21.15.120:6800/37641 3 ==== osd_op_reply(23 [pgnls start_epoch 122 out=74b] v117'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebdfe7000 2024-03-23T11:32:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.348+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:24 16.1d 16:b8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebebab000 2024-03-23T11:32:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.349+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 6 ==== osd_op_reply(24 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.349+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:25 16.3 16:c0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebeb3c800 2024-03-23T11:32:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.350+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 3 ==== osd_op_reply(25 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.350+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:26 16.13 16:c8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebdfe7800 2024-03-23T11:32:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.350+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.1 v2:172.21.15.62:6824/37001 2 ==== osd_op_reply(26 [pgnls start_epoch 122 out=98b] v121'5 uv5 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe7800 2024-03-23T11:32:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.350+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:27 16.b 16:d0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebdfe7800 2024-03-23T11:32:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.351+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.1 v2:172.21.15.62:6824/37001 3 ==== osd_op_reply(27 [pgnls start_epoch 122 out=76b] v122'6 uv5 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe7800 2024-03-23T11:32:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.351+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:28 16.1b 16:d8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebeb3c800 2024-03-23T11:32:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.352+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 4 ==== osd_op_reply(28 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.352+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:29 16.7 16:e0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebeb3c400 2024-03-23T11:32:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.352+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 4 ==== osd_op_reply(29 [pgnls start_epoch 122 out=105b] v122'9 uv9 ondisk = 1) v8 ==== 144+0+105 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.352+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:30 16.17 16:e8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebebab000 2024-03-23T11:32:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.353+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 7 ==== osd_op_reply(30 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.353+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:31 16.f 16:f0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebebaac00 2024-03-23T11:32:24.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.354+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 7 ==== osd_op_reply(31 [pgnls start_epoch 122 out=73b] v122'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.354+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:32 16.1f 16:f8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebeb3c800 2024-03-23T11:32:24.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.355+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 5 ==== osd_op_reply(32 [pgnls start_epoch 122 out=101b] v122'9 uv8 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.355+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 601.00000000 2024-03-23T11:32:24.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.355+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:33 16.10 16:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab400 con 0x55cebebab000 2024-03-23T11:32:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.355+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 8 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.355+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 602.00000000 2024-03-23T11:32:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.355+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:34 16.10 16:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebab000 2024-03-23T11:32:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.356+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 9 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.356+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-03-23T11:32:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.356+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:35 16.10 16:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebabc00 con 0x55cebebab000 2024-03-23T11:32:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.356+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 10 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=1047b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+1047 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.356+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 600.00000000 2024-03-23T11:32:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.356+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:36 16.10 16:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7c00 con 0x55cebebab000 2024-03-23T11:32:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.356+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 11 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.356+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 603.00000000 2024-03-23T11:32:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.357+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:37 16.10 16:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe006000 con 0x55cebebab000 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.357+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 12 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.357+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.357+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.357+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 200.00000000 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.357+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.357+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 200.00000001 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.357+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.357+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 606.00000000 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.357+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:38 16.14 16:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe006400 con 0x55cebebaac00 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.357+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 8 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv38 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.358+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 607.00000000 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.358+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:39 16.1c 16:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe006800 con 0x55cebebab000 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.358+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 13 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.358+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 10000000003.00000000 2024-03-23T11:32:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.358+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:40 16.12 16:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe006c00 con 0x55cebdfe7000 2024-03-23T11:32:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.359+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.6 v2:172.21.15.120:6800/37641 4 ==== osd_op_reply(40 10000000003.00000000 [omap-get-vals out=523b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+523 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebdfe7000 2024-03-23T11:32:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.359+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-03-23T11:32:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.359+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-23T11:32:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.359+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 608.00000000 2024-03-23T11:32:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.359+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:41 16.e 16:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe007000 con 0x55cebebab000 2024-03-23T11:32:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.360+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 14 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.360+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 604.00000000 2024-03-23T11:32:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.360+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:42 16.15 16:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe007400 con 0x55cebebaac00 2024-03-23T11:32:24.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.361+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 9 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.361+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 500.00000000 2024-03-23T11:32:24.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.361+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-23T11:32:24.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.361+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-03-23T11:32:24.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.361+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-23T11:32:24.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.361+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 605.00000000 2024-03-23T11:32:24.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.361+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:43 16.1d 16:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe007800 con 0x55cebebab000 2024-03-23T11:32:24.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.362+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 15 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.362+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-03-23T11:32:24.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.362+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-23T11:32:24.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.362+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 100.00000000 2024-03-23T11:32:24.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.362+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:44 16.13 16:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe006000 con 0x55cebdfe7800 2024-03-23T11:32:24.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.362+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.1 v2:172.21.15.62:6824/37001 4 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe7800 2024-03-23T11:32:24.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.363+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-03-23T11:32:24.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.363+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-23T11:32:24.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.363+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 609.00000000 2024-03-23T11:32:24.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.363+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:45 16.7 16:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe007c00 con 0x55cebeb3c400 2024-03-23T11:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.363+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 5 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv7 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.363+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 10000000001.00000000 2024-03-23T11:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.364+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:46 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7c00 con 0x55cebeb3c400 2024-03-23T11:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.364+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 6 ==== osd_op_reply(46 10000000001.00000000 [omap-get-vals out=1545b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+1545 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.364+0000 7f5149bf7740 20 datascan.scan_links: injected ino detected 2024-03-23T11:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.364+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 400.00000000 2024-03-23T11:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.364+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-23T11:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.364+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-03-23T11:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.364+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-23T11:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.364+0000 7f5149bf7740 10 datascan.scan_links: step 1: handling object 1.00000000 2024-03-23T11:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.364+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:47 16.1f 16:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab400 con 0x55cebeb3c800 2024-03-23T11:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.366+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 6 ==== osd_op_reply(47 1.00000000 [omap-get-vals out=524b] v0'0 uv8 ondisk = 0) v8 ==== 154+0+524 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.366+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:48 16.0 16:00000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebaac00 2024-03-23T11:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.366+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 10 ==== osd_op_reply(48 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.366+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:49 16.10 16:08000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebab000 2024-03-23T11:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.367+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 16 ==== osd_op_reply(49 [pgnls start_epoch 122 out=177b] v121'16 uv16 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.367+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:50 16.8 16:10000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebaac00 2024-03-23T11:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.367+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 11 ==== osd_op_reply(50 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.368+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:51 16.18 16:18000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebaac00 2024-03-23T11:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.368+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 12 ==== osd_op_reply(51 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.368+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:52 16.4 16:20000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebeb3c400 2024-03-23T11:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.369+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 7 ==== osd_op_reply(52 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.369+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:53 16.14 16:28000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebaac00 2024-03-23T11:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.369+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 13 ==== osd_op_reply(53 [pgnls start_epoch 122 out=149b] v122'43 uv43 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.369+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:54 16.c 16:30000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebeb3c800 2024-03-23T11:32:24.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.370+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 7 ==== osd_op_reply(54 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.370+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:55 16.1c 16:38000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebab000 2024-03-23T11:32:24.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.371+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 17 ==== osd_op_reply(55 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.371+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:56 16.2 16:40000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebdfe6000 2024-03-23T11:32:24.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.372+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.4 v2:172.21.15.120:6803/37640 3 ==== osd_op_reply(56 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe6000 2024-03-23T11:32:24.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.372+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:57 16.12 16:48000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebdfe7000 2024-03-23T11:32:24.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.373+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.6 v2:172.21.15.120:6800/37641 5 ==== osd_op_reply(57 [pgnls start_epoch 122 out=81b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebdfe7000 2024-03-23T11:32:24.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.373+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:58 16.a 16:50000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebab000 2024-03-23T11:32:24.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.374+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 18 ==== osd_op_reply(58 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.374+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:59 16.1a 16:58000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebeb3c400 2024-03-23T11:32:24.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.375+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 8 ==== osd_op_reply(59 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.375+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:60 16.6 16:60000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebdfe7000 2024-03-23T11:32:24.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.376+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.6 v2:172.21.15.120:6800/37641 6 ==== osd_op_reply(60 [pgnls start_epoch 122 out=77b] v121'10 uv10 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebdfe7000 2024-03-23T11:32:24.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.376+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:61 16.16 16:68000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebeb3c400 2024-03-23T11:32:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.376+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 9 ==== osd_op_reply(61 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.376+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:62 16.e 16:70000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebab000 2024-03-23T11:32:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.377+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 19 ==== osd_op_reply(62 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.377+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- osd_op(unknown.0.0:63 16.1e 16:78000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebdfe7400 2024-03-23T11:32:24.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.378+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.7 v2:172.21.15.120:6816/37655 2 ==== osd_op_reply(63 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebdfe7400 2024-03-23T11:32:24.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.378+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:64 16.1 16:80000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebab000 2024-03-23T11:32:24.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.379+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 20 ==== osd_op_reply(64 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.379+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:65 16.11 16:88000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebeb3c800 2024-03-23T11:32:24.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.380+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 8 ==== osd_op_reply(65 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.380+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:66 16.9 16:90000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebdfe7800 2024-03-23T11:32:24.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.381+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.1 v2:172.21.15.62:6824/37001 5 ==== osd_op_reply(66 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe7800 2024-03-23T11:32:24.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.381+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- osd_op(unknown.0.0:67 16.19 16:98000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebdfe6000 2024-03-23T11:32:24.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.382+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.4 v2:172.21.15.120:6803/37640 4 ==== osd_op_reply(67 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe6000 2024-03-23T11:32:24.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.382+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:68 16.5 16:a0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebaac00 2024-03-23T11:32:24.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.382+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 14 ==== osd_op_reply(68 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.382+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:69 16.15 16:a8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebaac00 2024-03-23T11:32:24.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.383+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 15 ==== osd_op_reply(69 [pgnls start_epoch 122 out=97b] v121'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.383+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:70 16.d 16:b0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebdfe7000 2024-03-23T11:32:24.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.384+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.6 v2:172.21.15.120:6800/37641 7 ==== osd_op_reply(70 [pgnls start_epoch 122 out=74b] v117'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebdfe7000 2024-03-23T11:32:24.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.384+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:71 16.1d 16:b8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebab000 2024-03-23T11:32:24.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.385+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 21 ==== osd_op_reply(71 [pgnls start_epoch 122 out=73b] v121'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.385+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:72 16.3 16:c0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebeb3c800 2024-03-23T11:32:24.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.386+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 9 ==== osd_op_reply(72 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.386+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:73 16.13 16:c8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebdfe7800 2024-03-23T11:32:24.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.386+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.1 v2:172.21.15.62:6824/37001 6 ==== osd_op_reply(73 [pgnls start_epoch 122 out=98b] v121'5 uv5 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe7800 2024-03-23T11:32:24.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.387+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:74 16.b 16:d0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebdfe7800 2024-03-23T11:32:24.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.387+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.1 v2:172.21.15.62:6824/37001 7 ==== osd_op_reply(74 [pgnls start_epoch 122 out=76b] v122'6 uv5 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe7800 2024-03-23T11:32:24.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.387+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:75 16.1b 16:d8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebeb3c800 2024-03-23T11:32:24.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.388+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 10 ==== osd_op_reply(75 [pgnls start_epoch 122 out=49b] v122'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.388+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:76 16.7 16:e0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebeb3c400 2024-03-23T11:32:24.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.389+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 10 ==== osd_op_reply(76 [pgnls start_epoch 122 out=105b] v122'9 uv9 ondisk = 1) v8 ==== 144+0+105 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.389+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:77 16.17 16:e8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebab000 2024-03-23T11:32:24.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.390+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 22 ==== osd_op_reply(77 [pgnls start_epoch 122 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.390+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:78 16.f 16:f0000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebebaac00 2024-03-23T11:32:24.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.391+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 16 ==== osd_op_reply(78 [pgnls start_epoch 122 out=73b] v122'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.391+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:79 16.1f 16:f8000000::::head [pgnls start_epoch 122 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab800 con 0x55cebeb3c800 2024-03-23T11:32:24.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.392+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 11 ==== osd_op_reply(79 [pgnls start_epoch 122 out=101b] v122'9 uv8 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.392+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 601.00000000 2024-03-23T11:32:24.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.392+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:80 16.10 16:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebabc00 con 0x55cebebab000 2024-03-23T11:32:24.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.393+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 23 ==== osd_op_reply(80 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.393+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 602.00000000 2024-03-23T11:32:24.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.393+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:81 16.10 16:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6c00 con 0x55cebebab000 2024-03-23T11:32:24.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.393+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 24 ==== osd_op_reply(81 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.393+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-03-23T11:32:24.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.393+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:82 16.10 16:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6800 con 0x55cebebab000 2024-03-23T11:32:24.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.394+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 25 ==== osd_op_reply(82 10000000000.00000000 [omap-get-vals out=1047b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+1047 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.394+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 600.00000000 2024-03-23T11:32:24.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.394+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:83 16.10 16:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe006400 con 0x55cebebab000 2024-03-23T11:32:24.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.395+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 26 ==== osd_op_reply(83 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.395+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 603.00000000 2024-03-23T11:32:24.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.395+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:84 16.10 16:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe007400 con 0x55cebebab000 2024-03-23T11:32:24.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.396+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 27 ==== osd_op_reply(84 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.396+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-03-23T11:32:24.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.396+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-03-23T11:32:24.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.396+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 200.00000000 2024-03-23T11:32:24.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.396+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-03-23T11:32:24.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.396+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 200.00000001 2024-03-23T11:32:24.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.396+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-03-23T11:32:24.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.396+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 606.00000000 2024-03-23T11:32:24.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.396+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:85 16.14 16:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe006c00 con 0x55cebebaac00 2024-03-23T11:32:24.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.397+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 17 ==== osd_op_reply(85 606.00000000 [omap-get-vals out=5b] v0'0 uv38 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.397+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 607.00000000 2024-03-23T11:32:24.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.397+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:86 16.1c 16:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe007c00 con 0x55cebebab000 2024-03-23T11:32:24.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.398+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 28 ==== osd_op_reply(86 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.398+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 10000000003.00000000 2024-03-23T11:32:24.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.398+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:87 16.12 16:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebea47400 con 0x55cebdfe7000 2024-03-23T11:32:24.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.399+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.6 v2:172.21.15.120:6800/37641 8 ==== osd_op_reply(87 10000000003.00000000 [omap-get-vals out=523b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+523 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebdfe7000 2024-03-23T11:32:24.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.399+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-03-23T11:32:24.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.399+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-03-23T11:32:24.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.399+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 608.00000000 2024-03-23T11:32:24.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.399+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:88 16.e 16:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebea47000 con 0x55cebebab000 2024-03-23T11:32:24.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.400+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 29 ==== osd_op_reply(88 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.400+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 604.00000000 2024-03-23T11:32:24.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.400+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- osd_op(unknown.0.0:89 16.15 16:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebab400 con 0x55cebebaac00 2024-03-23T11:32:24.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.401+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.0 v2:172.21.15.62:6811/36998 18 ==== osd_op_reply(89 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebebaac00 2024-03-23T11:32:24.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.401+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 500.00000000 2024-03-23T11:32:24.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.401+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-03-23T11:32:24.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.401+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-03-23T11:32:24.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.401+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-03-23T11:32:24.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.401+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 605.00000000 2024-03-23T11:32:24.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.401+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- osd_op(unknown.0.0:90 16.1d 16:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6c00 con 0x55cebebab000 2024-03-23T11:32:24.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.402+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.2 v2:172.21.15.62:6800/36996 30 ==== osd_op_reply(90 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebebab000 2024-03-23T11:32:24.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.402+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-03-23T11:32:24.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.402+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-03-23T11:32:24.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.402+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 100.00000000 2024-03-23T11:32:24.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.402+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:91 16.13 16:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6400 con 0x55cebdfe7800 2024-03-23T11:32:24.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.403+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.1 v2:172.21.15.62:6824/37001 8 ==== osd_op_reply(91 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe7800 2024-03-23T11:32:24.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.403+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-03-23T11:32:24.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.404+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-03-23T11:32:24.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.404+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 609.00000000 2024-03-23T11:32:24.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.404+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:92 16.7 16:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe7c00 con 0x55cebeb3c400 2024-03-23T11:32:24.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.404+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 11 ==== osd_op_reply(92 609.00000000 [omap-get-vals out=5b] v0'0 uv7 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.404+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 10000000001.00000000 2024-03-23T11:32:24.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.404+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:93 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebebabc00 con 0x55cebeb3c400 2024-03-23T11:32:24.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.405+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 12 ==== osd_op_reply(93 10000000001.00000000 [omap-get-vals out=1545b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+1545 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.405+0000 7f5149bf7740 20 datascan.scan_links: injected ino detected 2024-03-23T11:32:24.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.405+0000 7f5149bf7740 20 datascan.scan_links: adding 0x100000001f7 for future processing to fix dnfirst 2024-03-23T11:32:24.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.405+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 400.00000000 2024-03-23T11:32:24.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.405+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-03-23T11:32:24.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.405+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-03-23T11:32:24.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.405+0000 7f5149bf7740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-03-23T11:32:24.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.405+0000 7f5149bf7740 10 datascan.scan_links: step 2: handling object 1.00000000 2024-03-23T11:32:24.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.405+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- osd_op(unknown.0.0:94 16.1f 16:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe006800 con 0x55cebeb3c800 2024-03-23T11:32:24.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.406+0000 7f5147200640 1 -- 172.21.15.62:0/2468410830 <== osd.5 v2:172.21.15.120:6824/37642 12 ==== osd_op_reply(94 1.00000000 [omap-get-vals out=524b] v0'0 uv8 ondisk = 0) v8 ==== 154+0+524 (crc 0 0 0) 0x55cebeb26b40 con 0x55cebeb3c800 2024-03-23T11:32:24.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.407+0000 7f5149bf7740 10 datascan.scan_links: processing 0 dup_primaries, 0 remote_links 2024-03-23T11:32:24.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.407+0000 7f5149bf7740 10 datascan.scan_links: removing dup dentries from 0 objects 2024-03-23T11:32:24.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.407+0000 7f5149bf7740 10 datascan.scan_links: processing 0 bad_nlink_inos 2024-03-23T11:32:24.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.407+0000 7f5149bf7740 10 datascan.scan_links: processing 1 injected_inos 2024-03-23T11:32:24.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.407+0000 7f5149bf7740 10 datascan.scan_links: handling injected_ino 0x100000001f7 2024-03-23T11:32:24.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.407+0000 7f5149bf7740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000001:bravo 2024-03-23T11:32:24.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.407+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:95 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe007000 con 0x55cebeb3c400 2024-03-23T11:32:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.407+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 13 ==== osd_op_reply(95 10000000001.00000000 [omap-get-vals-by-keys out=506b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+506 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.407+0000 7f5149bf7740 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-03-23T11:32:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.407+0000 7f5149bf7740 20 datascan.scan_links: first is now 2 2024-03-23T11:32:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.407+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- osd_op(unknown.0.0:96 16.7 16:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe007800 con 0x55cebeb3c400 2024-03-23T11:32:24.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.411+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.3 v2:172.21.15.62:6808/37000 14 ==== osd_op_reply(96 10000000001.00000000 [omap-set-vals] v122'10 uv10 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebeb3c400 2024-03-23T11:32:24.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.411+0000 7f5149bf7740 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-03-23T11:32:24.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.411+0000 7f5149bf7740 10 datascan.scan_links: updating inotable 2024-03-23T11:32:24.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.411+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:97 16.13 16:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebe006000 con 0x55cebdfe7800 2024-03-23T11:32:24.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.412+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.1 v2:172.21.15.62:6824/37001 9 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv4 ondisk = 0) v8 ==== 157+0+50 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe7800 2024-03-23T11:32:24.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.412+0000 7f5149bf7740 10 mds.0.inotable: skip_inos was [0x10000000005~0x1f2,0x100000001f9~0xfffffffe07] 2024-03-23T11:32:24.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.412+0000 7f5149bf7740 10 mds.0.inotable: skip_inos now [0x100000001f9~0xfffffffe07] 2024-03-23T11:32:24.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.412+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- osd_op(unknown.0.0:98 16.13 16:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6800 con 0x55cebdfe7800 2024-03-23T11:32:24.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.417+0000 7f5148600640 1 -- 172.21.15.62:0/2468410830 <== osd.1 v2:172.21.15.62:6824/37001 10 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v122'6 uv6 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x55cebeb26d80 con 0x55cebdfe7800 2024-03-23T11:32:24.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.417+0000 7f5149bf7740 10 datascan.scan_links: updating snaptable 2024-03-23T11:32:24.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.417+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- osd_op(unknown.0.0:99 16.d 16:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e122) v8 -- 0x55cebdfe6c00 con 0x55cebdfe7000 2024-03-23T11:32:24.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 <== osd.6 v2:172.21.15.120:6800/37641 9 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+70 (crc 0 0 0) 0x55cebeb26fc0 con 0x55cebdfe7000 2024-03-23T11:32:24.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55cebebaac00 msgr2=0x55cebdcfe100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x55cebebaac00 0x55cebdcfe100 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55cebdfe7800 msgr2=0x55cebdfd8100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x55cebdfe7800 0x55cebdfd8100 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55cebebab000 msgr2=0x55cebdfd6000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x55cebebab000 0x55cebdfd6000 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x55cebeb3c400 msgr2=0x55cebdfd6580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x55cebeb3c400 0x55cebdfd6580 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55cebdfe6000 msgr2=0x55cebdfd7080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x55cebdfe6000 0x55cebdfd7080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x55cebeb3c800 msgr2=0x55cebdfd6b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x55cebeb3c800 0x55cebdfd6b00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 reap_dead start 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55cebdfe7000 msgr2=0x55cebdfd7600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x55cebdfe7000 0x55cebdfd7600 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 reap_dead start 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x55cebdfe7400 msgr2=0x55cebdfd7b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.418+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x55cebdfe7400 0x55cebdfd7b80 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.419+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 reap_dead start 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.419+0000 7f5147c00640 1 -- 172.21.15.62:0/2468410830 reap_dead start 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.419+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x55cebebaa800 msgr2=0x55cebdcff700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.419+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x55cebebaa800 0x55cebdcff700 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x55cebeb94600 tx=0x55cebea94000 comp rx=0 tx=0).stop 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.419+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d000 msgr2=0x55cebdcff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.419+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d000 0x55cebdcff180 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x55cebeb95470 tx=0x55cebeb52cc0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.419+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 shutdown_connections 2024-03-23T11:32:24.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.419+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x55cebebaa800 0x55cebdcff700 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.419+0000 7f5149bf7740 1 --2- 172.21.15.62:0/2468410830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebeb3d000 0x55cebdcff180 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.419+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 >> 172.21.15.62:0/2468410830 conn(0x55cebeb3cc00 msgr2=0x55cebe9f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:24.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.420+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 shutdown_connections 2024-03-23T11:32:24.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.420+0000 7f5149bf7740 1 -- 172.21.15.62:0/2468410830 wait complete. 2024-03-23T11:32:24.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.421+0000 7f5149bf7740 1 -- 172.21.15.62:0/3731430887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebdc75800 msgr2=0x55cebdcfcb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:24.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.421+0000 7f5149bf7740 1 --2- 172.21.15.62:0/3731430887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebdc75800 0x55cebdcfcb00 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x55cebea89320 tx=0x55cebea94d80 comp rx=0 tx=0).stop 2024-03-23T11:32:24.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.421+0000 7f5149bf7740 1 -- 172.21.15.62:0/3731430887 shutdown_connections 2024-03-23T11:32:24.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.421+0000 7f5149bf7740 1 --2- 172.21.15.62:0/3731430887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cebdc75800 0x55cebdcfcb00 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.421+0000 7f5149bf7740 1 --2- 172.21.15.62:0/3731430887 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55cebdc75c00 0x55cebdcfd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.421+0000 7f5149bf7740 1 --2- 172.21.15.62:0/3731430887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cebdc74c00 0x55cebdcfc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:24.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.421+0000 7f5149bf7740 1 -- 172.21.15.62:0/3731430887 >> 172.21.15.62:0/3731430887 conn(0x55cebdc75000 msgr2=0x55cebe9f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:24.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.422+0000 7f5149bf7740 1 -- 172.21.15.62:0/3731430887 shutdown_connections 2024-03-23T11:32:24.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:24.422+0000 7f5149bf7740 1 -- 172.21.15.62:0/3731430887 wait complete. 2024-03-23T11:32:24.423 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.227781, result: 2024-03-23T11:32:24.424 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T11:32:24.430 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T11:32:24.605 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2024-03-23T11:32:24.605 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2024-03-23T11:32:24.605 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T11:32:24.605 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2024-03-23T11:32:24.605 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.605 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2024-03-23T11:32:24.605 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.605 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T11:32:24.606 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: mon_allow_pool_size_one = true 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.607 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "172.21.15.62,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2024-03-23T11:32:24.608 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: osd mclock profile = high_recovery_ops 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2024-03-23T11:32:24.609 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2024-03-23T11:32:24.610 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.610 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2024-03-23T11:32:24.610 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.610 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2024-03-23T11:32:24.610 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.610 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2024-03-23T11:32:24.610 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.610 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2024-03-23T11:32:24.611 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2024-03-23T11:32:24.611 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.611 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2024-03-23T11:32:24.611 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2024-03-23T11:32:24.611 INFO:teuthology.orchestra.run.smithi120.stdout: bdev async discard = True 2024-03-23T11:32:24.611 INFO:teuthology.orchestra.run.smithi120.stdout: bdev enable discard = True 2024-03-23T11:32:24.611 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore allocator = bitmap 2024-03-23T11:32:24.611 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2024-03-23T11:32:24.611 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2024-03-23T11:32:24.611 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 1/20 2024-03-23T11:32:24.611 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 1/20 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 4/10 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2024-03-23T11:32:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:24.613 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: # 1m isn't quite enough 2024-03-23T11:32:24.614 INFO:teuthology.orchestra.run.smithi120.stdout: mon_down_mkfs_grace = 2m 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_filestore_osds = false 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout: mon op complaint time = 120 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout: client mount timeout = 600 2024-03-23T11:32:24.615 INFO:teuthology.orchestra.run.smithi120.stdout: debug client = 20 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = "" 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.d] 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout:[mds] 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds = 20 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds balancer = 20 2024-03-23T11:32:24.616 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:24.617 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug frag = True 2024-03-23T11:32:24.617 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug scatterstat = False 2024-03-23T11:32:24.617 INFO:teuthology.orchestra.run.smithi120.stdout: mds op complaint time = 180 2024-03-23T11:32:24.617 INFO:teuthology.orchestra.run.smithi120.stdout: mds verify scatter = True 2024-03-23T11:32:24.617 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-23T11:32:24.617 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-23T11:32:24.617 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-23T11:32:24.617 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-03-23T11:32:24.618 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2024-03-23T11:32:24.618 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T11:32:24.618 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2024-03-23T11:32:24.618 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.618 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2024-03-23T11:32:24.618 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.618 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2024-03-23T11:32:24.618 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.618 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2024-03-23T11:32:24.619 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T11:32:24.622 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = "172.21.15.62,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.623 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock profile = high_recovery_ops 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: bdev async discard = True 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: bdev enable discard = True 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore allocator = bitmap 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2024-03-23T11:32:24.625 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 1/20 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 1/20 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 4/10 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2024-03-23T11:32:24.626 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2024-03-23T11:32:24.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.628 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: mon op complaint time = 120 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: client mount timeout = 600 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: debug client = 20 2024-03-23T11:32:24.629 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = "" 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.b] 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.c] 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a] 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.c] 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b] 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.d] 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout:[mds] 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds = 20 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds balancer = 20 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:24.630 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug frag = True 2024-03-23T11:32:24.631 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug scatterstat = False 2024-03-23T11:32:24.631 INFO:teuthology.orchestra.run.smithi062.stdout: mds op complaint time = 180 2024-03-23T11:32:24.631 INFO:teuthology.orchestra.run.smithi062.stdout: mds verify scatter = True 2024-03-23T11:32:24.631 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-03-23T11:32:24.631 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-03-23T11:32:24.631 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-03-23T11:32:24.661 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T11:32:24.666 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2024-03-23T11:32:24.833 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2024-03-23T11:32:24.834 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: mon_allow_pool_size_one = true 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2024-03-23T11:32:24.835 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "172.21.15.62,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2024-03-23T11:32:24.836 INFO:teuthology.orchestra.run.smithi120.stdout: osd mclock profile = high_recovery_ops 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.837 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: bdev async discard = True 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: bdev enable discard = True 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore allocator = bitmap 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2024-03-23T11:32:24.838 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 1/20 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 1/20 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 4/10 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:24.839 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2024-03-23T11:32:24.840 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-23T11:32:24.840 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-23T11:32:24.840 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:24.840 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:24.840 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2024-03-23T11:32:24.840 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.840 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2024-03-23T11:32:24.840 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T11:32:24.841 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: # 1m isn't quite enough 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: mon_down_mkfs_grace = 2m 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_filestore_osds = false 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: mon op complaint time = 120 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2024-03-23T11:32:24.842 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout: client mount timeout = 600 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout: debug client = 20 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = "" 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2024-03-23T11:32:24.843 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.d] 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout:[mds] 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds = 20 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds balancer = 20 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug frag = True 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug scatterstat = True 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout: mds op complaint time = 180 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout: mds verify scatter = True 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-23T11:32:24.844 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-23T11:32:24.845 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-03-23T11:32:24.845 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2024-03-23T11:32:24.845 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T11:32:24.845 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2024-03-23T11:32:24.845 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.845 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2024-03-23T11:32:24.845 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.845 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2024-03-23T11:32:24.845 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.845 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.846 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.847 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = "172.21.15.62,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = e8b2848c-f838-4b99-a310-de7a7069c38e 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock profile = high_recovery_ops 2024-03-23T11:32:24.848 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2024-03-23T11:32:24.849 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.850 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2024-03-23T11:32:24.850 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.850 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2024-03-23T11:32:24.850 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2024-03-23T11:32:24.850 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.850 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2024-03-23T11:32:24.850 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2024-03-23T11:32:24.850 INFO:teuthology.orchestra.run.smithi062.stdout: bdev async discard = True 2024-03-23T11:32:24.850 INFO:teuthology.orchestra.run.smithi062.stdout: bdev enable discard = True 2024-03-23T11:32:24.850 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore allocator = bitmap 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 1/20 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 1/20 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 4/10 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2024-03-23T11:32:24.851 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:24.852 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.853 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: mon op complaint time = 120 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2024-03-23T11:32:24.854 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout: client mount timeout = 600 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout: debug client = 20 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = "" 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.b] 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.c] 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a] 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.c] 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b] 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.d] 2024-03-23T11:32:24.855 INFO:teuthology.orchestra.run.smithi062.stdout:[mds] 2024-03-23T11:32:24.856 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds = 20 2024-03-23T11:32:24.856 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds balancer = 20 2024-03-23T11:32:24.856 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-23T11:32:24.856 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug frag = True 2024-03-23T11:32:24.856 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug scatterstat = True 2024-03-23T11:32:24.856 INFO:teuthology.orchestra.run.smithi062.stdout: mds op complaint time = 180 2024-03-23T11:32:24.856 INFO:teuthology.orchestra.run.smithi062.stdout: mds verify scatter = True 2024-03-23T11:32:24.856 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-03-23T11:32:24.856 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-03-23T11:32:24.856 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-03-23T11:32:24.887 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-23T11:32:25.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.126+0000 7fc0a1600640 1 Processor -- start 2024-03-23T11:32:25.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.126+0000 7fc0a1600640 1 -- start start 2024-03-23T11:32:25.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.127+0000 7fc0a1600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc09c160420 0x7fc09c1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.127+0000 7fc0a1600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0b5db0 0x7fc09c0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.127+0000 7fc0a1600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c0ce510 con 0x7fc09c15fcf0 2024-03-23T11:32:25.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.127+0000 7fc0a1600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09c160df0 con 0x7fc09c160420 2024-03-23T11:32:25.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.127+0000 7fc0a1600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0cdc70 con 0x7fc09c0b5db0 2024-03-23T11:32:25.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.128+0000 7fc09be00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc09c15fcf0 0x7fc09c0d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58094/0 (socket says 172.21.15.62:58094) 2024-03-23T11:32:25.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.128+0000 7fc09be00640 1 -- 172.21.15.62:0/863734027 learned_addr learned my addr 172.21.15.62:0/863734027 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:25.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.128+0000 7fc093400640 1 --2- 172.21.15.62:0/863734027 >> v2:172.21.15.62:3300/0 conn(0x7fc09c160420 0x7fc09c1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:25.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.128+0000 7fc0a0c00640 1 --2- 172.21.15.62:0/863734027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0b5db0 0x7fc09c0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:25.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.128+0000 7fc09b400640 1 -- 172.21.15.62:0/863734027 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320220469 0 0) 0x7fc09c0ce510 con 0x7fc09c15fcf0 2024-03-23T11:32:25.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.128+0000 7fc09b400640 1 -- 172.21.15.62:0/863734027 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc080003520 con 0x7fc09c15fcf0 2024-03-23T11:32:25.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.128+0000 7fc093400640 1 -- 172.21.15.62:0/863734027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0b5db0 msgr2=0x7fc09c0b6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:25.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.128+0000 7fc093400640 1 --2- 172.21.15.62:0/863734027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0b5db0 0x7fc09c0b6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.128+0000 7fc093400640 1 -- 172.21.15.62:0/863734027 >> v1:172.21.15.62:6789/0 conn(0x7fc09c15fcf0 legacy=0x7fc09c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:25.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.128+0000 7fc093400640 1 -- 172.21.15.62:0/863734027 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0cdf50 con 0x7fc09c160420 2024-03-23T11:32:25.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.128+0000 7fc0a0c00640 1 --2- 172.21.15.62:0/863734027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0b5db0 0x7fc09c0b6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:25.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.130+0000 7fc093400640 1 --2- 172.21.15.62:0/863734027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09c160420 0x7fc09c1608b0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fc0840097a0 tx=0x7fc08402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27dd6d735504286e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:25.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.130+0000 7fc09b400640 1 -- 172.21.15.62:0/863734027 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084031780 con 0x7fc09c160420 2024-03-23T11:32:25.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.130+0000 7fc09b400640 1 -- 172.21.15.62:0/863734027 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0840318e0 con 0x7fc09c160420 2024-03-23T11:32:25.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.130+0000 7fc09b400640 1 -- 172.21.15.62:0/863734027 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084031c70 con 0x7fc09c160420 2024-03-23T11:32:25.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.130+0000 7fc0a1600640 1 -- 172.21.15.62:0/863734027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09c160420 msgr2=0x7fc09c1608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:25.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.130+0000 7fc0a1600640 1 --2- 172.21.15.62:0/863734027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09c160420 0x7fc09c1608b0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fc0840097a0 tx=0x7fc08402b820 comp rx=0 tx=0).stop 2024-03-23T11:32:25.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.131+0000 7fc0a1600640 1 -- 172.21.15.62:0/863734027 shutdown_connections 2024-03-23T11:32:25.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.131+0000 7fc0a1600640 1 --2- 172.21.15.62:0/863734027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0b5db0 0x7fc09c0b6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.131+0000 7fc0a1600640 1 --2- 172.21.15.62:0/863734027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09c160420 0x7fc09c1608b0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.131+0000 7fc0a1600640 1 -- 172.21.15.62:0/863734027 >> 172.21.15.62:0/863734027 conn(0x7fc09c089f60 msgr2=0x7fc09c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:25.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.131+0000 7fc0a1600640 1 -- 172.21.15.62:0/863734027 shutdown_connections 2024-03-23T11:32:25.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.131+0000 7fc0a1600640 1 -- 172.21.15.62:0/863734027 wait complete. 2024-03-23T11:32:25.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.132+0000 7fc0a1600640 1 Processor -- start 2024-03-23T11:32:25.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.132+0000 7fc0a1600640 1 -- start start 2024-03-23T11:32:25.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.133+0000 7fc0a1600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0b5db0 0x7fc09c0c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.133+0000 7fc0a1600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c15fcf0 0x7fc09c150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.133+0000 7fc0a1600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09c160420 0x7fc09c1f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.133+0000 7fc0a1600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c09fdc0 con 0x7fc09c160420 2024-03-23T11:32:25.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.133+0000 7fc0a1600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0c8110 con 0x7fc09c15fcf0 2024-03-23T11:32:25.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.133+0000 7fc0a1600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0c7500 con 0x7fc09c0b5db0 2024-03-23T11:32:25.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.134+0000 7fc0a0c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09c160420 0x7fc09c1f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:25.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.134+0000 7fc0a0c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09c160420 0x7fc09c1f18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60808/0 (socket says 172.21.15.62:60808) 2024-03-23T11:32:25.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.134+0000 7fc0a0c00640 1 -- 172.21.15.62:0/2161535083 learned_addr learned my addr 172.21.15.62:0/2161535083 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.134+0000 7fc09be00640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0b5db0 0x7fc09c0c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.134+0000 7fc0a0c00640 1 -- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0b5db0 msgr2=0x7fc09c0c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.134+0000 7fc0a0c00640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0b5db0 0x7fc09c0c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.134+0000 7fc0a0c00640 1 -- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c15fcf0 msgr2=0x7fc09c150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.134+0000 7fc0a0c00640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c15fcf0 0x7fc09c150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.134+0000 7fc0a0c00640 1 -- 172.21.15.62:0/2161535083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc084009400 con 0x7fc09c160420 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.135+0000 7fc0a0c00640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09c160420 0x7fc09c1f18c0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fc094007df0 tx=0x7fc09400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.135+0000 7fc098c00640 1 -- 172.21.15.62:0/2161535083 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094003990 con 0x7fc09c160420 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.135+0000 7fc098c00640 1 -- 172.21.15.62:0/2161535083 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09400f040 con 0x7fc09c160420 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.135+0000 7fc098c00640 1 -- 172.21.15.62:0/2161535083 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094003d30 con 0x7fc09c160420 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.135+0000 7fc0a1600640 1 -- 172.21.15.62:0/2161535083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c0c77e0 con 0x7fc09c160420 2024-03-23T11:32:25.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.135+0000 7fc0a1600640 1 -- 172.21.15.62:0/2161535083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c0c7da0 con 0x7fc09c160420 2024-03-23T11:32:25.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.136+0000 7fc0a1600640 1 -- 172.21.15.62:0/2161535083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c0c8280 con 0x7fc09c160420 2024-03-23T11:32:25.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.137+0000 7fc098c00640 1 -- 172.21.15.62:0/2161535083 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc094004030 con 0x7fc09c160420 2024-03-23T11:32:25.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.138+0000 7fc098c00640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc07806a7a0 0x7fc07806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.138+0000 7fc098c00640 1 -- 172.21.15.62:0/2161535083 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fc0940891a0 con 0x7fc09c160420 2024-03-23T11:32:25.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.139+0000 7fc09be00640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc07806a7a0 0x7fc07806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:25.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.140+0000 7fc09be00640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc07806a7a0 0x7fc07806cc60 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0011e0 tx=0x7fc08c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:25.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.142+0000 7fc098c00640 1 -- 172.21.15.62:0/2161535083 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0940544c0 con 0x7fc09c160420 2024-03-23T11:32:25.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.381+0000 7fc0a1600640 1 -- 172.21.15.62:0/2161535083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7fc09c066bb0 con 0x7fc09c160420 2024-03-23T11:32:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.442+0000 7fc098c00640 1 -- 172.21.15.62:0/2161535083 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v104) v1 ==== 162+0+0 (secure 0 0 0) 0x7fc094003e90 con 0x7fc09c160420 2024-03-23T11:32:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-23T11:32:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 -- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc07806a7a0 msgr2=0x7fc07806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc07806a7a0 0x7fc07806cc60 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0011e0 tx=0x7fc08c009040 comp rx=0 tx=0).stop 2024-03-23T11:32:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 -- 172.21.15.62:0/2161535083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09c160420 msgr2=0x7fc09c1f18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09c160420 0x7fc09c1f18c0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fc094007df0 tx=0x7fc09400d4a0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 -- 172.21.15.62:0/2161535083 shutdown_connections 2024-03-23T11:32:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc07806a7a0 0x7fc07806cc60 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09c160420 0x7fc09c1f18c0 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc09c15fcf0 0x7fc09c150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 --2- 172.21.15.62:0/2161535083 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc09c0b5db0 0x7fc09c0c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 -- 172.21.15.62:0/2161535083 >> 172.21.15.62:0/2161535083 conn(0x7fc09c089f60 msgr2=0x7fc09c0cb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 -- 172.21.15.62:0/2161535083 shutdown_connections 2024-03-23T11:32:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.446+0000 7fc0a1600640 1 -- 172.21.15.62:0/2161535083 wait complete. 2024-03-23T11:32:25.459 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:32:25.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.664+0000 7f821e600640 1 Processor -- start 2024-03-23T11:32:25.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.665+0000 7f821e600640 1 -- start start 2024-03-23T11:32:25.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.666+0000 7f821e600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f82180b2f90 0x7f8218161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.666+0000 7f821e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82180b15e0 0x7f82180b3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.667+0000 7f821e600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82180ae0c0 con 0x7f8218164a30 2024-03-23T11:32:25.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.667+0000 7f821e600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82180b2740 con 0x7f82180b2f90 2024-03-23T11:32:25.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.667+0000 7f821e600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82181623b0 con 0x7f82180b15e0 2024-03-23T11:32:25.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.667+0000 7f8217400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f82180b2f90 0x7f8218161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:25.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.667+0000 7f8217400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f82180b2f90 0x7f8218161e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60816/0 (socket says 172.21.15.62:60816) 2024-03-23T11:32:25.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.667+0000 7f8217400640 1 -- 172.21.15.62:0/926885934 learned_addr learned my addr 172.21.15.62:0/926885934 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:25.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.667+0000 7f8217400640 1 -- 172.21.15.62:0/926885934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82180b15e0 msgr2=0x7f82180b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:25.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.667+0000 7f8217400640 1 --2- 172.21.15.62:0/926885934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82180b15e0 0x7f82180b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.667+0000 7f8217400640 1 -- 172.21.15.62:0/926885934 >> v1:172.21.15.120:6789/0 conn(0x7f8218164a30 legacy=0x7f8218164ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:25.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.667+0000 7f8217400640 1 -- 172.21.15.62:0/926885934 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82180b1100 con 0x7f82180b2f90 2024-03-23T11:32:25.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.668+0000 7f8217400640 1 --2- 172.21.15.62:0/926885934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82180b2f90 0x7f8218161e70 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f8208009870 tx=0x7f820802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df7af3d684580ab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:25.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.669+0000 7f8216000640 1 -- 172.21.15.62:0/926885934 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208031640 con 0x7f82180b2f90 2024-03-23T11:32:25.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.669+0000 7f8216000640 1 -- 172.21.15.62:0/926885934 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82080317a0 con 0x7f82180b2f90 2024-03-23T11:32:25.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.669+0000 7f8216000640 1 -- 172.21.15.62:0/926885934 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208031b30 con 0x7f82180b2f90 2024-03-23T11:32:25.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.669+0000 7f821e600640 1 -- 172.21.15.62:0/926885934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82180b2f90 msgr2=0x7f8218161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:25.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.669+0000 7f821e600640 1 --2- 172.21.15.62:0/926885934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82180b2f90 0x7f8218161e70 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f8208009870 tx=0x7f820802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.670+0000 7f821e600640 1 -- 172.21.15.62:0/926885934 shutdown_connections 2024-03-23T11:32:25.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.670+0000 7f821e600640 1 --2- 172.21.15.62:0/926885934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f82180b15e0 0x7f82180b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.670+0000 7f821e600640 1 --2- 172.21.15.62:0/926885934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82180b2f90 0x7f8218161e70 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.670+0000 7f821e600640 1 -- 172.21.15.62:0/926885934 >> 172.21.15.62:0/926885934 conn(0x7f82180b02e0 msgr2=0x7f82180b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:25.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.670+0000 7f821e600640 1 -- 172.21.15.62:0/926885934 shutdown_connections 2024-03-23T11:32:25.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.670+0000 7f821e600640 1 -- 172.21.15.62:0/926885934 wait complete. 2024-03-23T11:32:25.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.671+0000 7f821e600640 1 Processor -- start 2024-03-23T11:32:25.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.672+0000 7f821e600640 1 -- start start 2024-03-23T11:32:25.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.672+0000 7f821e600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82180b15e0 0x7f8218158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.672+0000 7f821e600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82180b2f90 0x7f8218163af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f821e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8218164a30 0x7f8218164030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f821e600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8218159fe0 con 0x7f82180b15e0 2024-03-23T11:32:25.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f821e600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8218159740 con 0x7f82180b2f90 2024-03-23T11:32:25.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f821e600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8218159a20 con 0x7f8218164a30 2024-03-23T11:32:25.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f8217400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82180b15e0 0x7f8218158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:25.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f8217400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82180b15e0 0x7f8218158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60818/0 (socket says 172.21.15.62:60818) 2024-03-23T11:32:25.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f8217400640 1 -- 172.21.15.62:0/3192139441 learned_addr learned my addr 172.21.15.62:0/3192139441 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:25.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f8217400640 1 -- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8218164a30 msgr2=0x7f8218164030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:25.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f8217400640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8218164a30 0x7f8218164030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f8217400640 1 -- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82180b2f90 msgr2=0x7f8218163af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:25.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f8217400640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82180b2f90 0x7f8218163af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.673+0000 7f8217400640 1 -- 172.21.15.62:0/3192139441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82080094d0 con 0x7f82180b15e0 2024-03-23T11:32:25.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.674+0000 7f8217400640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82180b15e0 0x7f8218158ef0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f8208009870 tx=0x7f82080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:25.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.674+0000 7f820f400640 1 -- 172.21.15.62:0/3192139441 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820803ac60 con 0x7f82180b15e0 2024-03-23T11:32:25.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.674+0000 7f820f400640 1 -- 172.21.15.62:0/3192139441 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820803adc0 con 0x7f82180b15e0 2024-03-23T11:32:25.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.674+0000 7f820f400640 1 -- 172.21.15.62:0/3192139441 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208044630 con 0x7f82180b15e0 2024-03-23T11:32:25.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.674+0000 7f821e600640 1 -- 172.21.15.62:0/3192139441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8218159d00 con 0x7f82180b15e0 2024-03-23T11:32:25.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.674+0000 7f821e600640 1 -- 172.21.15.62:0/3192139441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8218159430 con 0x7f82180b15e0 2024-03-23T11:32:25.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.675+0000 7f821e600640 1 -- 172.21.15.62:0/3192139441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82180b1df0 con 0x7f82180b15e0 2024-03-23T11:32:25.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.676+0000 7f820f400640 1 -- 172.21.15.62:0/3192139441 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f820802f050 con 0x7f82180b15e0 2024-03-23T11:32:25.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.677+0000 7f820f400640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81f406a7a0 0x7f81f406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:25.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.677+0000 7f820f400640 1 -- 172.21.15.62:0/3192139441 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f82080aaf00 con 0x7f82180b15e0 2024-03-23T11:32:25.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.678+0000 7f8216a00640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81f406a7a0 0x7f81f406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:25.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.679+0000 7f8216a00640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81f406a7a0 0x7f81f406cc60 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f82180afdc0 tx=0x7f8200006bf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:25.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.682+0000 7f820f400640 1 -- 172.21.15.62:0/3192139441 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82180b1df0 con 0x7f82180b15e0 2024-03-23T11:32:25.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.918+0000 7f821e600640 1 -- 172.21.15.62:0/3192139441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f821815b340 con 0x7f82180b15e0 2024-03-23T11:32:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.919+0000 7f820f400640 1 -- 172.21.15.62:0/3192139441 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f821815b340 con 0x7f82180b15e0 2024-03-23T11:32:25.908 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:25.909 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":104,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:32:25.434901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9507":{"gid":9507,"name":"a","rank":0,"incarnation":104,"state":"up:replay","state_seq":1,"addr":"172.21.15.62:6835/2500981044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2500981044},{"type":"v1","addr":"172.21.15.62:6835","nonce":2500981044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:32:25.909 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 104 2024-03-23T11:32:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.921+0000 7f821e600640 1 -- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81f406a7a0 msgr2=0x7f81f406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.921+0000 7f821e600640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81f406a7a0 0x7f81f406cc60 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f82180afdc0 tx=0x7f8200006bf0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.921+0000 7f821e600640 1 -- 172.21.15.62:0/3192139441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82180b15e0 msgr2=0x7f8218158ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.922+0000 7f821e600640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82180b15e0 0x7f8218158ef0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f8208009870 tx=0x7f82080046c0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.922+0000 7f821e600640 1 -- 172.21.15.62:0/3192139441 shutdown_connections 2024-03-23T11:32:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.922+0000 7f821e600640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f81f406a7a0 0x7f81f406cc60 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.922+0000 7f821e600640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8218164a30 0x7f8218164030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.922+0000 7f821e600640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82180b2f90 0x7f8218163af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.922+0000 7f821e600640 1 --2- 172.21.15.62:0/3192139441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82180b15e0 0x7f8218158ef0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.922+0000 7f821e600640 1 -- 172.21.15.62:0/3192139441 >> 172.21.15.62:0/3192139441 conn(0x7f82180b02e0 msgr2=0x7f8218161380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.922+0000 7f821e600640 1 -- 172.21.15.62:0/3192139441 shutdown_connections 2024-03-23T11:32:25.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:25.922+0000 7f821e600640 1 -- 172.21.15.62:0/3192139441 wait complete. 2024-03-23T11:32:26.332 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:31:24.044342+0000', 'modified': '2024-03-23T11:32:25.434901+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9507}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9507': {'gid': 9507, 'name': 'a', 'rank': 0, 'incarnation': 104, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.62:6835/2500981044', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6834', 'nonce': 2500981044}, {'type': 'v1', 'addr': '172.21.15.62:6835', 'nonce': 2500981044}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:32:26.333 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9507:up:replay 2024-03-23T11:32:27.334 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:32:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.577+0000 7f5164600640 1 Processor -- start 2024-03-23T11:32:27.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.577+0000 7f5164600640 1 -- start start 2024-03-23T11:32:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.578+0000 7f5164600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f515c06c300 0x7f515c0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.578+0000 7f5164600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f515c06c6e0 0x7f515c06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.578+0000 7f5164600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515c0cbc60 con 0x7f515c0cafa0 2024-03-23T11:32:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.578+0000 7f5164600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515c0cbf40 con 0x7f515c06c300 2024-03-23T11:32:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.578+0000 7f5164600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c0cc220 con 0x7f515c06c6e0 2024-03-23T11:32:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.578+0000 7f5161c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f515c06c300 0x7f515c0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.579+0000 7f5161c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f515c06c300 0x7f515c0ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55244/0 (socket says 172.21.15.62:55244) 2024-03-23T11:32:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.579+0000 7f5161200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f515c0cafa0 0x7f515c0cb430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59168/0 (socket says 172.21.15.62:59168) 2024-03-23T11:32:27.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.579+0000 7f5161c00640 1 -- 172.21.15.62:0/2175433112 learned_addr learned my addr 172.21.15.62:0/2175433112 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:27.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.579+0000 7f5162600640 1 --2- 172.21.15.62:0/2175433112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f515c06c6e0 0x7f515c06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:27.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.579+0000 7f5161c00640 1 -- 172.21.15.62:0/2175433112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f515c06c6e0 msgr2=0x7f515c06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:27.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.579+0000 7f5161c00640 1 --2- 172.21.15.62:0/2175433112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f515c06c6e0 0x7f515c06cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:27.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.579+0000 7f5161c00640 1 -- 172.21.15.62:0/2175433112 >> v1:172.21.15.62:6789/0 conn(0x7f515c0cafa0 legacy=0x7f515c0cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:27.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.579+0000 7f5161c00640 1 -- 172.21.15.62:0/2175433112 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c0cc500 con 0x7f515c06c300 2024-03-23T11:32:27.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.579+0000 7f515be00640 1 -- 172.21.15.62:0/2175433112 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790945880 0 0) 0x7f515c0cbc60 con 0x7f515c0cafa0 2024-03-23T11:32:27.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.581+0000 7f5161c00640 1 --2- 172.21.15.62:0/2175433112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f515c06c300 0x7f515c0ccd00 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f5150009870 tx=0x7f515002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5b62ab15eac46a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:27.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.582+0000 7f515be00640 1 -- 172.21.15.62:0/2175433112 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150030840 con 0x7f515c06c300 2024-03-23T11:32:27.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.582+0000 7f515be00640 1 -- 172.21.15.62:0/2175433112 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51500309a0 con 0x7f515c06c300 2024-03-23T11:32:27.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.582+0000 7f515be00640 1 -- 172.21.15.62:0/2175433112 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150030d30 con 0x7f515c06c300 2024-03-23T11:32:27.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.582+0000 7f5164600640 1 -- 172.21.15.62:0/2175433112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f515c06c300 msgr2=0x7f515c0ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:27.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.582+0000 7f5164600640 1 --2- 172.21.15.62:0/2175433112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f515c06c300 0x7f515c0ccd00 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f5150009870 tx=0x7f515002b980 comp rx=0 tx=0).stop 2024-03-23T11:32:27.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.582+0000 7f5164600640 1 -- 172.21.15.62:0/2175433112 shutdown_connections 2024-03-23T11:32:27.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.582+0000 7f5164600640 1 --2- 172.21.15.62:0/2175433112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f515c06c6e0 0x7f515c06cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:27.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.582+0000 7f5164600640 1 --2- 172.21.15.62:0/2175433112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f515c06c300 0x7f515c0ccd00 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.582+0000 7f5164600640 1 -- 172.21.15.62:0/2175433112 >> 172.21.15.62:0/2175433112 conn(0x7f515c0b02e0 msgr2=0x7f515c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.583+0000 7f5164600640 1 -- 172.21.15.62:0/2175433112 shutdown_connections 2024-03-23T11:32:27.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.583+0000 7f5164600640 1 -- 172.21.15.62:0/2175433112 wait complete. 2024-03-23T11:32:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.585+0000 7f5164600640 1 Processor -- start 2024-03-23T11:32:27.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.585+0000 7f5164600640 1 -- start start 2024-03-23T11:32:27.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.586+0000 7f5164600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f515c06c300 0x7f515c1f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:27.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.586+0000 7f5164600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f515c06c6e0 0x7f515c13e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:27.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.586+0000 7f5164600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f515c0cafa0 0x7f515c13c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:27.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.586+0000 7f5164600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c13fd90 con 0x7f515c06c300 2024-03-23T11:32:27.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.586+0000 7f5164600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c13ef00 con 0x7f515c0cafa0 2024-03-23T11:32:27.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.586+0000 7f5164600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c0abd10 con 0x7f515c06c6e0 2024-03-23T11:32:27.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.586+0000 7f5161c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f515c06c300 0x7f515c1f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:27.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.586+0000 7f5161c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f515c06c300 0x7f515c1f2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55250/0 (socket says 172.21.15.62:55250) 2024-03-23T11:32:27.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.586+0000 7f5161c00640 1 -- 172.21.15.62:0/2273314799 learned_addr learned my addr 172.21.15.62:0/2273314799 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:27.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.587+0000 7f5162600640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f515c0cafa0 0x7f515c13c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.587+0000 7f5161c00640 1 -- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f515c06c6e0 msgr2=0x7f515c13e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.587+0000 7f5161200640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f515c06c6e0 0x7f515c13e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.587+0000 7f5161c00640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f515c06c6e0 0x7f515c13e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.587+0000 7f5161c00640 1 -- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f515c0cafa0 msgr2=0x7f515c13c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.587+0000 7f5161c00640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f515c0cafa0 0x7f515c13c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.587+0000 7f5161c00640 1 -- 172.21.15.62:0/2273314799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500094d0 con 0x7f515c06c300 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.587+0000 7f5162600640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f515c0cafa0 0x7f515c13c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.587+0000 7f5161200640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f515c06c6e0 0x7f515c13e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.587+0000 7f5161c00640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f515c06c300 0x7f515c1f2e90 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f5150000c00 tx=0x7f5150004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.587+0000 7f5159600640 1 -- 172.21.15.62:0/2273314799 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150039c90 con 0x7f515c06c300 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.588+0000 7f5159600640 1 -- 172.21.15.62:0/2273314799 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5150039df0 con 0x7f515c06c300 2024-03-23T11:32:27.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.588+0000 7f5164600640 1 -- 172.21.15.62:0/2273314799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515c0abff0 con 0x7f515c06c300 2024-03-23T11:32:27.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.588+0000 7f5159600640 1 -- 172.21.15.62:0/2273314799 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150043640 con 0x7f515c06c300 2024-03-23T11:32:27.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.588+0000 7f5164600640 1 -- 172.21.15.62:0/2273314799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515c0ac5b0 con 0x7f515c06c300 2024-03-23T11:32:27.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.589+0000 7f5164600640 1 -- 172.21.15.62:0/2273314799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515c15c980 con 0x7f515c06c300 2024-03-23T11:32:27.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.590+0000 7f5159600640 1 -- 172.21.15.62:0/2273314799 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f51500437a0 con 0x7f515c06c300 2024-03-23T11:32:27.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.591+0000 7f5159600640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f513c06a7a0 0x7f513c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:27.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.591+0000 7f5159600640 1 -- 172.21.15.62:0/2273314799 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f515003e070 con 0x7f515c06c300 2024-03-23T11:32:27.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.591+0000 7f5161200640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f513c06a7a0 0x7f513c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:27.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.593+0000 7f5161200640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f513c06a7a0 0x7f513c06cc60 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f515c1efe40 tx=0x7f5148009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:27.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.595+0000 7f5159600640 1 -- 172.21.15.62:0/2273314799 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5150077230 con 0x7f515c06c300 2024-03-23T11:32:27.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.835+0000 7f5164600640 1 -- 172.21.15.62:0/2273314799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f515c0b14d0 con 0x7f515c06c300 2024-03-23T11:32:27.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.836+0000 7f5159600640 1 -- 172.21.15.62:0/2273314799 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f515007a850 con 0x7f515c06c300 2024-03-23T11:32:27.826 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:27.826 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":106,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:32:27.455767+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9507":{"gid":9507,"name":"a","rank":0,"incarnation":104,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.62:6835/2500981044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2500981044},{"type":"v1","addr":"172.21.15.62:6835","nonce":2500981044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:32:27.826 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 106 2024-03-23T11:32:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 -- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f513c06a7a0 msgr2=0x7f513c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f513c06a7a0 0x7f513c06cc60 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f515c1efe40 tx=0x7f5148009040 comp rx=0 tx=0).stop 2024-03-23T11:32:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 -- 172.21.15.62:0/2273314799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f515c06c300 msgr2=0x7f515c1f2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f515c06c300 0x7f515c1f2e90 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f5150000c00 tx=0x7f5150004730 comp rx=0 tx=0).stop 2024-03-23T11:32:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 -- 172.21.15.62:0/2273314799 shutdown_connections 2024-03-23T11:32:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f513c06a7a0 0x7f513c06cc60 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f515c0cafa0 0x7f515c13c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f515c06c6e0 0x7f515c13e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 --2- 172.21.15.62:0/2273314799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f515c06c300 0x7f515c1f2e90 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 -- 172.21.15.62:0/2273314799 >> 172.21.15.62:0/2273314799 conn(0x7f515c0b02e0 msgr2=0x7f515c0cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 -- 172.21.15.62:0/2273314799 shutdown_connections 2024-03-23T11:32:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:27.839+0000 7f5164600640 1 -- 172.21.15.62:0/2273314799 wait complete. 2024-03-23T11:32:28.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:31:24.044342+0000', 'modified': '2024-03-23T11:32:27.455767+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9507}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9507': {'gid': 9507, 'name': 'a', 'rank': 0, 'incarnation': 104, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.62:6835/2500981044', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6834', 'nonce': 2500981044}, {'type': 'v1', 'addr': '172.21.15.62:6835', 'nonce': 2500981044}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:32:28.251 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9507:up:rejoin 2024-03-23T11:32:29.252 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:32:29.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.490+0000 7f845ac00640 1 Processor -- start 2024-03-23T11:32:29.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.491+0000 7f845ac00640 1 -- start start 2024-03-23T11:32:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.492+0000 7f845ac00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f845406cc60 0x7f84540cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.492+0000 7f845ac00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f84540ce420 0x7f84540b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.492+0000 7f845ac00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84540ced90 con 0x7f8454159490 2024-03-23T11:32:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.492+0000 7f845ac00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845406e3e0 con 0x7f845406cc60 2024-03-23T11:32:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.492+0000 7f845ac00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84540ce850 con 0x7f84540ce420 2024-03-23T11:32:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.493+0000 7f8458e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f845406cc60 0x7f84540cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.493+0000 7f8458e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f845406cc60 0x7f84540cdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55256/0 (socket says 172.21.15.62:55256) 2024-03-23T11:32:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.493+0000 7f8458e00640 1 -- 172.21.15.62:0/54555580 learned_addr learned my addr 172.21.15.62:0/54555580 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.493+0000 7f845a200640 1 --2- 172.21.15.62:0/54555580 >> v2:172.21.15.120:3300/0 conn(0x7f84540ce420 0x7f84540b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.493+0000 7f8458e00640 1 -- 172.21.15.62:0/54555580 >> v2:172.21.15.120:3300/0 conn(0x7f84540ce420 msgr2=0x7f84540b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.493+0000 7f8458e00640 1 --2- 172.21.15.62:0/54555580 >> v2:172.21.15.120:3300/0 conn(0x7f84540ce420 0x7f84540b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.493+0000 7f8458e00640 1 -- 172.21.15.62:0/54555580 >> v1:172.21.15.120:6789/0 conn(0x7f8454159490 legacy=0x7f845415da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.493+0000 7f8458e00640 1 -- 172.21.15.62:0/54555580 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845406c1c0 con 0x7f845406cc60 2024-03-23T11:32:29.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.494+0000 7f8458e00640 1 --2- 172.21.15.62:0/54555580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845406cc60 0x7f84540cdee0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f84540ad990 tx=0x7f844802e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd67f64973508b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:29.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.495+0000 7f8453e00640 1 -- 172.21.15.62:0/54555580 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844802ea00 con 0x7f845406cc60 2024-03-23T11:32:29.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.495+0000 7f8453e00640 1 -- 172.21.15.62:0/54555580 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844802eb60 con 0x7f845406cc60 2024-03-23T11:32:29.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.495+0000 7f8453e00640 1 -- 172.21.15.62:0/54555580 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448009600 con 0x7f845406cc60 2024-03-23T11:32:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.495+0000 7f845ac00640 1 -- 172.21.15.62:0/54555580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845406cc60 msgr2=0x7f84540cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.495+0000 7f845ac00640 1 --2- 172.21.15.62:0/54555580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845406cc60 0x7f84540cdee0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f84540ad990 tx=0x7f844802e060 comp rx=0 tx=0).stop 2024-03-23T11:32:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.496+0000 7f845ac00640 1 -- 172.21.15.62:0/54555580 shutdown_connections 2024-03-23T11:32:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.496+0000 7f845ac00640 1 --2- 172.21.15.62:0/54555580 >> v2:172.21.15.120:3300/0 conn(0x7f84540ce420 0x7f84540b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.496+0000 7f845ac00640 1 --2- 172.21.15.62:0/54555580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f845406cc60 0x7f84540cdee0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.496+0000 7f845ac00640 1 -- 172.21.15.62:0/54555580 >> 172.21.15.62:0/54555580 conn(0x7f8454066050 msgr2=0x7f8454069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.496+0000 7f845ac00640 1 -- 172.21.15.62:0/54555580 shutdown_connections 2024-03-23T11:32:29.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.496+0000 7f845ac00640 1 -- 172.21.15.62:0/54555580 wait complete. 2024-03-23T11:32:29.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.498+0000 7f845ac00640 1 Processor -- start 2024-03-23T11:32:29.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.498+0000 7f845ac00640 1 -- start start 2024-03-23T11:32:29.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.499+0000 7f845ac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f845406cc60 0x7f8454146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.499+0000 7f845ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84540ce420 0x7f84541526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.499+0000 7f845ac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8454159490 0x7f8454152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.499+0000 7f845ac00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8454152420 con 0x7f8454159490 2024-03-23T11:32:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.499+0000 7f845ac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845415aad0 con 0x7f845406cc60 2024-03-23T11:32:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.499+0000 7f845ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845415adb0 con 0x7f84540ce420 2024-03-23T11:32:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.499+0000 7f845a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8454159490 0x7f8454152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.499+0000 7f845a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8454159490 0x7f8454152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55272/0 (socket says 172.21.15.62:55272) 2024-03-23T11:32:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.499+0000 7f845a200640 1 -- 172.21.15.62:0/1676256422 learned_addr learned my addr 172.21.15.62:0/1676256422 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:29.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f8459800640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f845406cc60 0x7f8454146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f845a200640 1 -- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84540ce420 msgr2=0x7f84541526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f845a200640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84540ce420 0x7f84541526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f845a200640 1 -- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f845406cc60 msgr2=0x7f8454146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f845a200640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f845406cc60 0x7f8454146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f845a200640 1 -- 172.21.15.62:0/1676256422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844800bd40 con 0x7f8454159490 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f8459800640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f845406cc60 0x7f8454146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f845a200640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8454159490 0x7f8454152c30 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f844c00d800 tx=0x7f844c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f8451600640 1 -- 172.21.15.62:0/1676256422 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c00b6c0 con 0x7f8454159490 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f8451600640 1 -- 172.21.15.62:0/1676256422 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844c00b840 con 0x7f8454159490 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f8451600640 1 -- 172.21.15.62:0/1676256422 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c00bbf0 con 0x7f8454159490 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.500+0000 7f845ac00640 1 -- 172.21.15.62:0/1676256422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845415b090 con 0x7f8454159490 2024-03-23T11:32:29.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.501+0000 7f845ac00640 1 -- 172.21.15.62:0/1676256422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845415b650 con 0x7f8454159490 2024-03-23T11:32:29.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.502+0000 7f845ac00640 1 -- 172.21.15.62:0/1676256422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845406d0c0 con 0x7f8454159490 2024-03-23T11:32:29.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.502+0000 7f8451600640 1 -- 172.21.15.62:0/1676256422 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f844c00bd50 con 0x7f8454159490 2024-03-23T11:32:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.503+0000 7f8451600640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843806a7a0 0x7f843806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.503+0000 7f8451600640 1 -- 172.21.15.62:0/1676256422 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f844c0893d0 con 0x7f8454159490 2024-03-23T11:32:29.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.504+0000 7f8459800640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843806a7a0 0x7f843806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:29.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.505+0000 7f8459800640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843806a7a0 0x7f843806cc60 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f844400a830 tx=0x7f8444008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:29.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.508+0000 7f8451600640 1 -- 172.21.15.62:0/1676256422 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f844c0546f0 con 0x7f8454159490 2024-03-23T11:32:29.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.747+0000 7f845ac00640 1 -- 172.21.15.62:0/1676256422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8454066bb0 con 0x7f8454159490 2024-03-23T11:32:29.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.748+0000 7f8451600640 1 -- 172.21.15.62:0/1676256422 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f844c057d10 con 0x7f8454159490 2024-03-23T11:32:29.738 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:29.738 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":107,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:32:28.458742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9507":{"gid":9507,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6835/2500981044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2500981044},{"type":"v1","addr":"172.21.15.62:6835","nonce":2500981044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:32:29.738 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 107 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 -- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843806a7a0 msgr2=0x7f843806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843806a7a0 0x7f843806cc60 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f844400a830 tx=0x7f8444008040 comp rx=0 tx=0).stop 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 -- 172.21.15.62:0/1676256422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8454159490 msgr2=0x7f8454152c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8454159490 0x7f8454152c30 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f844c00d800 tx=0x7f844c00dcc0 comp rx=0 tx=0).stop 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 -- 172.21.15.62:0/1676256422 shutdown_connections 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f843806a7a0 0x7f843806cc60 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8454159490 0x7f8454152c30 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f84540ce420 0x7f84541526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 --2- 172.21.15.62:0/1676256422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f845406cc60 0x7f8454146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 -- 172.21.15.62:0/1676256422 >> 172.21.15.62:0/1676256422 conn(0x7f8454066050 msgr2=0x7f84540aec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 -- 172.21.15.62:0/1676256422 shutdown_connections 2024-03-23T11:32:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:29.751+0000 7f845ac00640 1 -- 172.21.15.62:0/1676256422 wait complete. 2024-03-23T11:32:30.162 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T11:31:24.044342+0000', 'modified': '2024-03-23T11:32:28.458742+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9507}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9507': {'gid': 9507, 'name': 'a', 'rank': 0, 'incarnation': 104, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.62:6835/2500981044', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6834', 'nonce': 2500981044}, {'type': 'v1', 'addr': '172.21.15.62:6835', 'nonce': 2500981044}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T11:32:30.162 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T11:32:30.163 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T11:32:30.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.400+0000 7f9a6a600640 1 Processor -- start 2024-03-23T11:32:30.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.400+0000 7f9a6a600640 1 -- start start 2024-03-23T11:32:30.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.402+0000 7f9a6a600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b3240 0x7f9a6406a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.402+0000 7f9a6a600640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a640b14c0 con 0x7f9a6415d060 2024-03-23T11:32:30.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.402+0000 7f9a6a600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6415dee0 con 0x7f9a641593e0 2024-03-23T11:32:30.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.402+0000 7f9a6a600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6406a020 con 0x7f9a640b3240 2024-03-23T11:32:30.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.402+0000 7f9a5be00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9a6415d060 0x7f9a640b2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59176/0 (socket says 172.21.15.62:59176) 2024-03-23T11:32:30.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.402+0000 7f9a5be00640 1 -- 172.21.15.62:0/3022430590 learned_addr learned my addr 172.21.15.62:0/3022430590 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a69c00640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b3240 0x7f9a6406a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295730324 0 0) 0x7f9a640b14c0 con 0x7f9a6415d060 2024-03-23T11:32:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a48005730 con 0x7f9a6415d060 2024-03-23T11:32:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4286470345 0 0) 0x7f9a48005730 con 0x7f9a6415d060 2024-03-23T11:32:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b3240 msgr2=0x7f9a6406a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b3240 0x7f9a6406a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 >> v1:172.21.15.120:6789/0 conn(0x7f9a641593e0 legacy=0x7f9a640cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6415d640 con 0x7f9a6415d060 2024-03-23T11:32:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f9a4c001670 con 0x7f9a6415d060 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 >> v1:172.21.15.62:6789/0 conn(0x7f9a6415d060 legacy=0x7f9a640b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a48005bb0 0x7f9a48005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b3240 0x7f9a480069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a48006f10 0x7f9a48007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.403+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4800dc40 con 0x7f9a48005bb0 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.404+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4800ddb0 con 0x7f9a48006f10 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.404+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4800df70 con 0x7f9a640b3240 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.404+0000 7f9a69200640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a48005bb0 0x7f9a48005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.404+0000 7f9a5be00640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b3240 0x7f9a480069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.404+0000 7f9a69c00640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a48006f10 0x7f9a48007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.404+0000 7f9a69200640 1 -- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b3240 msgr2=0x7f9a480069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.404+0000 7f9a69200640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a640b3240 0x7f9a480069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.404+0000 7f9a69200640 1 -- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a48006f10 msgr2=0x7f9a48007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.404+0000 7f9a69200640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a48006f10 0x7f9a48007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.404+0000 7f9a69200640 1 -- 172.21.15.62:0/3022430590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a48005730 con 0x7f9a48005bb0 2024-03-23T11:32:30.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.404+0000 7f9a69c00640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a48006f10 0x7f9a48007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.405+0000 7f9a69200640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a48005bb0 0x7f9a48005fd0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f9a54001e20 tx=0x7f9a48007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbdaf771d9eef32 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.406+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54003b70 con 0x7f9a48005bb0 2024-03-23T11:32:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.406+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a54003cd0 con 0x7f9a48005bb0 2024-03-23T11:32:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.406+0000 7f9a63e00640 1 -- 172.21.15.62:0/3022430590 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54005310 con 0x7f9a48005bb0 2024-03-23T11:32:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.406+0000 7f9a6a600640 1 -- 172.21.15.62:0/3022430590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a48005bb0 msgr2=0x7f9a48005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.406+0000 7f9a6a600640 1 --2- 172.21.15.62:0/3022430590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a48005bb0 0x7f9a48005fd0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f9a54001e20 tx=0x7f9a48007a00 comp rx=0 tx=0).stop 2024-03-23T11:32:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.406+0000 7f9a69c00640 1 -- 172.21.15.62:0/3022430590 reap_dead start 2024-03-23T11:32:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.406+0000 7f9a6a600640 1 -- 172.21.15.62:0/3022430590 shutdown_connections 2024-03-23T11:32:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.406+0000 7f9a6a600640 1 -- 172.21.15.62:0/3022430590 >> 172.21.15.62:0/3022430590 conn(0x7f9a640b0090 msgr2=0x7f9a640b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.407+0000 7f9a6a600640 1 -- 172.21.15.62:0/3022430590 shutdown_connections 2024-03-23T11:32:30.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.407+0000 7f9a6a600640 1 -- 172.21.15.62:0/3022430590 wait complete. 2024-03-23T11:32:30.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.408+0000 7f9a6a600640 1 Processor -- start 2024-03-23T11:32:30.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.408+0000 7f9a6a600640 1 -- start start 2024-03-23T11:32:30.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.409+0000 7f9a6a600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a48005bb0 0x7f9a640cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.409+0000 7f9a6a600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a641518c0 0x7f9a64146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.409+0000 7f9a6a600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a6406d720 0x7f9a6406dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.409+0000 7f9a6a600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6406e0e0 con 0x7f9a641518c0 2024-03-23T11:32:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.409+0000 7f9a6a600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640abb70 con 0x7f9a6406d720 2024-03-23T11:32:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.409+0000 7f9a6a600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640ab840 con 0x7f9a48005bb0 2024-03-23T11:32:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.410+0000 7f9a5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a641518c0 0x7f9a64146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.410+0000 7f9a5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a641518c0 0x7f9a64146ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55284/0 (socket says 172.21.15.62:55284) 2024-03-23T11:32:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.410+0000 7f9a5be00640 1 -- 172.21.15.62:0/457459450 learned_addr learned my addr 172.21.15.62:0/457459450 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.410+0000 7f9a5be00640 1 -- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a48005bb0 msgr2=0x7f9a640cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.410+0000 7f9a5be00640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a48005bb0 0x7f9a640cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.410+0000 7f9a5be00640 1 -- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a6406d720 msgr2=0x7f9a6406dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.410+0000 7f9a5be00640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a6406d720 0x7f9a6406dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.410+0000 7f9a5be00640 1 -- 172.21.15.62:0/457459450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a641520b0 con 0x7f9a641518c0 2024-03-23T11:32:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.410+0000 7f9a5be00640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a641518c0 0x7f9a64146ce0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00d770 tx=0x7f9a4c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.411+0000 7f9a61600640 1 -- 172.21.15.62:0/457459450 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c0074e0 con 0x7f9a641518c0 2024-03-23T11:32:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.411+0000 7f9a61600640 1 -- 172.21.15.62:0/457459450 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4c007640 con 0x7f9a641518c0 2024-03-23T11:32:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.411+0000 7f9a61600640 1 -- 172.21.15.62:0/457459450 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c0049e0 con 0x7f9a641518c0 2024-03-23T11:32:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.411+0000 7f9a6a600640 1 -- 172.21.15.62:0/457459450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a64152390 con 0x7f9a641518c0 2024-03-23T11:32:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.411+0000 7f9a6a600640 1 -- 172.21.15.62:0/457459450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a64151d40 con 0x7f9a641518c0 2024-03-23T11:32:30.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.413+0000 7f9a61600640 1 -- 172.21.15.62:0/457459450 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f9a4c004b40 con 0x7f9a641518c0 2024-03-23T11:32:30.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.415+0000 7f9a61600640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a4006a7a0 0x7f9a4006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.415+0000 7f9a61600640 1 -- 172.21.15.62:0/457459450 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f9a4c088720 con 0x7f9a641518c0 2024-03-23T11:32:30.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.415+0000 7f9a69200640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a4006a7a0 0x7f9a4006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.416+0000 7f9a6e83b740 1 Processor -- start 2024-03-23T11:32:30.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.416+0000 7f9a6e83b740 1 -- start start 2024-03-23T11:32:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a69200640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a4006a7a0 0x7f9a4006cc60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9a54034000 tx=0x7f9a54030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6a520 0x565222f21b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f205a0 0x565222f22090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f1ca40 0x565222f4c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222e8d110 con 0x565222f6a520 2024-03-23T11:32:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222e8a8a0 con 0x565222f1ca40 2024-03-23T11:32:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x565222e86190 con 0x565222f205a0 2024-03-23T11:32:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a69200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6a520 0x565222f21b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a69200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6a520 0x565222f21b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55296/0 (socket says 172.21.15.62:55296) 2024-03-23T11:32:30.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a69200640 1 -- 172.21.15.62:0/1591842589 learned_addr learned my addr 172.21.15.62:0/1591842589 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:30.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a69200640 1 -- 172.21.15.62:0/1591842589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f205a0 msgr2=0x565222f22090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a69200640 1 --2- 172.21.15.62:0/1591842589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f205a0 0x565222f22090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a69200640 1 -- 172.21.15.62:0/1591842589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f1ca40 msgr2=0x565222f4c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.417+0000 7f9a69c00640 1 --2- 172.21.15.62:0/1591842589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f1ca40 0x565222f4c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.418+0000 7f9a69200640 1 --2- 172.21.15.62:0/1591842589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f1ca40 0x565222f4c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.418+0000 7f9a69200640 1 -- 172.21.15.62:0/1591842589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565222e51230 con 0x565222f6a520 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.418+0000 7f9a69200640 1 --2- 172.21.15.62:0/1591842589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6a520 0x565222f21b50 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f9a54001f10 tx=0x7f9a54031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f690299b8afa687f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.418+0000 7f9a59600640 1 -- 172.21.15.62:0/1591842589 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54005820 con 0x565222f6a520 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.418+0000 7f9a59600640 1 -- 172.21.15.62:0/1591842589 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a54006b50 con 0x565222f6a520 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.418+0000 7f9a59600640 1 -- 172.21.15.62:0/1591842589 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54007030 con 0x565222f6a520 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.419+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1591842589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6a520 msgr2=0x565222f21b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.419+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/1591842589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6a520 0x565222f21b50 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f9a54001f10 tx=0x7f9a54031000 comp rx=0 tx=0).stop 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.419+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1591842589 shutdown_connections 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.419+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/1591842589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f1ca40 0x565222f4c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.419+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/1591842589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f205a0 0x565222f22090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.419+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/1591842589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6a520 0x565222f21b50 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.419+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1591842589 >> 172.21.15.62:0/1591842589 conn(0x565222f1ecb0 msgr2=0x565222f67830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:30.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.419+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1591842589 shutdown_connections 2024-03-23T11:32:30.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.419+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1591842589 wait complete. 2024-03-23T11:32:30.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.420+0000 7f9a6e83b740 1 -- start start 2024-03-23T11:32:30.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.420+0000 7f9a6e83b740 1 Processor -- start 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.421+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222f447f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.421+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222f488c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.421+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f6df20 0x565222f47b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.421+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222e278c0 con 0x565222f205a0 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.421+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222e27560 con 0x565222f6a520 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.421+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x565222bedf20 con 0x565222f6df20 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.421+0000 7f9a6e83b740 10 client.0 _ll_register_callbacks cb 0x565222e7f800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.421+0000 7f9a69200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222f447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.421+0000 7f9a69200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222f447f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55306/0 (socket says 172.21.15.62:55306) 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.421+0000 7f9a69200640 1 -- 172.21.15.62:0/3793844143 learned_addr learned my addr 172.21.15.62:0/3793844143 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.422+0000 7f9a69200640 1 -- 172.21.15.62:0/3793844143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f6df20 msgr2=0x565222f47b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.422+0000 7f9a69200640 1 --2- 172.21.15.62:0/3793844143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f6df20 0x565222f47b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.422+0000 7f9a69200640 1 -- 172.21.15.62:0/3793844143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 msgr2=0x565222f488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.422+0000 7f9a69200640 1 --2- 172.21.15.62:0/3793844143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222f488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.422+0000 7f9a69200640 1 -- 172.21.15.62:0/3793844143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a48005730 con 0x565222f205a0 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.422+0000 7f9a69200640 1 --2- 172.21.15.62:0/3793844143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222f447f0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f9a54001cc0 tx=0x7f9a5402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd65d13059854dc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.422+0000 7f9a52000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.422+0000 7f9a52000640 1 -- 172.21.15.62:0/3793844143 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5402f640 con 0x565222f205a0 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.422+0000 7f9a52000640 1 -- 172.21.15.62:0/3793844143 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f9a5405abb0 con 0x565222f205a0 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.422+0000 7f9a6e83b740 1 -- 172.21.15.62:0/3793844143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565222eb0280 con 0x565222f205a0 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a52000640 1 -- 172.21.15.62:0/3793844143 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a5405ad70 con 0x565222f205a0 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a6e83b740 10 client.9597 fetch_fsmap learned FSMap version 107 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a6e83b740 1 -- 172.21.15.62:0/3793844143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x565222e84810 con 0x565222f205a0 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a52000640 1 -- 172.21.15.62:0/3793844143 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x565222eb0280 con 0x565222f205a0 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a52000640 1 -- 172.21.15.62:0/3793844143 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 107) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9a5402d040 con 0x565222f205a0 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a6e83b740 10 client.9597 fetch_fsmap finished waiting for FSMap version 107 2024-03-23T11:32:30.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a6e83b740 10 client.9597 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:32:30.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a6e83b740 20 client.9597 populate_metadata read hostname 'smithi062' 2024-03-23T11:32:30.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/3793844143 >> [v2:172.21.15.62:6834/2500981044,v1:172.21.15.62:6835/2500981044] conn(0x565222f77690 0x565222f6ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a6e83b740 4 client.9597 mds_command: new command op to 9507 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-23T11:32:30.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a6e83b740 1 -- 172.21.15.62:0/3793844143 --> [v2:172.21.15.62:6834/2500981044,v1:172.21.15.62:6835/2500981044] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565222e789d0 con 0x565222f77690 2024-03-23T11:32:30.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.423+0000 7f9a5be00640 1 --2- 172.21.15.62:0/3793844143 >> [v2:172.21.15.62:6834/2500981044,v1:172.21.15.62:6835/2500981044] conn(0x565222f77690 0x565222f6ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.424+0000 7f9a5be00640 1 --2- 172.21.15.62:0/3793844143 >> [v2:172.21.15.62:6834/2500981044,v1:172.21.15.62:6835/2500981044] conn(0x565222f77690 0x565222f6ee20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48f4b8b05140c44e server_cookie=b7529f73c57d3cc4 in_seq=0 out_seq=0 2024-03-23T11:32:30.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.424+0000 7f9a52000640 10 client.9597 ms_handle_connect on v2:172.21.15.62:6834/2500981044 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a52000640 1 -- 172.21.15.62:0/3793844143 <== mds.0 v2:172.21.15.62:6834/2500981044 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x565222e789d0 con 0x565222f77690 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a52000640 10 client.9597 handle_command_reply: tid=0 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a6e83b740 1 client.9597 shutdown 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a6e83b740 1 -- 172.21.15.62:0/3793844143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 msgr2=0x565222f447f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/3793844143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222f447f0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f9a54001cc0 tx=0x7f9a5402e000 comp rx=0 tx=0).stop 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a6e83b740 1 -- 172.21.15.62:0/3793844143 shutdown_connections 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/3793844143 >> [v2:172.21.15.62:6834/2500981044,v1:172.21.15.62:6835/2500981044] conn(0x565222f77690 0x565222f6ee20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/3793844143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f6df20 0x565222f47b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/3793844143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222f488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/3793844143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222f447f0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a6e83b740 1 -- 172.21.15.62:0/3793844143 >> 172.21.15.62:0/3793844143 conn(0x565222f1ca40 msgr2=0x565222f1bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a52000640 0 client.9597 ms_handle_reset on v2:172.21.15.62:6834/2500981044 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a6e83b740 1 -- 172.21.15.62:0/3793844143 shutdown_connections 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.425+0000 7f9a6e83b740 1 -- 172.21.15.62:0/3793844143 wait complete. 2024-03-23T11:32:30.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.426+0000 7f9a6e83b740 20 client.9597 trim_cache size 0 max 16384 2024-03-23T11:32:30.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.443+0000 7f9a6e83b740 1 Processor -- start 2024-03-23T11:32:30.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.443+0000 7f9a6e83b740 1 -- start start 2024-03-23T11:32:30.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.444+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222fc73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.444+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222f1ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.444+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f6df20 0x565222f1d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.444+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222f18210 con 0x565222f205a0 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.444+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222f18b60 con 0x565222f6a520 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.444+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x565222e85290 con 0x565222f6df20 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.444+0000 7f9a69200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222fc73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.444+0000 7f9a69200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222fc73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55312/0 (socket says 172.21.15.62:55312) 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.444+0000 7f9a69200640 1 -- 172.21.15.62:0/2096611438 learned_addr learned my addr 172.21.15.62:0/2096611438 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a5be00640 1 --2- 172.21.15.62:0/2096611438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222f1ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a69c00640 1 --2- 172.21.15.62:0/2096611438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f6df20 0x565222f1d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a69200640 1 -- 172.21.15.62:0/2096611438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f6df20 msgr2=0x565222f1d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a69200640 1 --2- 172.21.15.62:0/2096611438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f6df20 0x565222f1d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a69200640 1 -- 172.21.15.62:0/2096611438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 msgr2=0x565222f1ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a69200640 1 --2- 172.21.15.62:0/2096611438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222f1ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a69200640 1 -- 172.21.15.62:0/2096611438 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565222f4a070 con 0x565222f205a0 2024-03-23T11:32:30.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a5be00640 1 --2- 172.21.15.62:0/2096611438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222f1ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:30.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a69c00640 1 --2- 172.21.15.62:0/2096611438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f6df20 0x565222f1d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:30.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a69200640 1 --2- 172.21.15.62:0/2096611438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222fc73f0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9a5402c120 tx=0x7f9a54054000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ad33cbd192b2ce server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:30.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a5a000640 1 -- 172.21.15.62:0/2096611438 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5402fc80 con 0x565222f205a0 2024-03-23T11:32:30.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a5a000640 1 -- 172.21.15.62:0/2096611438 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5402fde0 con 0x565222f205a0 2024-03-23T11:32:30.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.445+0000 7f9a5a000640 1 -- 172.21.15.62:0/2096611438 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54050650 con 0x565222f205a0 2024-03-23T11:32:30.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.446+0000 7f9a6e83b740 1 -- 172.21.15.62:0/2096611438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 msgr2=0x565222fc73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.446+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/2096611438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222fc73f0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9a5402c120 tx=0x7f9a54054000 comp rx=0 tx=0).stop 2024-03-23T11:32:30.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.446+0000 7f9a6e83b740 1 -- 172.21.15.62:0/2096611438 shutdown_connections 2024-03-23T11:32:30.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.446+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/2096611438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f6df20 0x565222f1d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.446+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/2096611438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222f1ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.446+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/2096611438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f205a0 0x565222fc73f0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.446+0000 7f9a6e83b740 1 -- 172.21.15.62:0/2096611438 >> 172.21.15.62:0/2096611438 conn(0x565222f1ca40 msgr2=0x565222f20a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:30.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.446+0000 7f9a6e83b740 1 -- 172.21.15.62:0/2096611438 shutdown_connections 2024-03-23T11:32:30.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.446+0000 7f9a6e83b740 1 -- 172.21.15.62:0/2096611438 wait complete. 2024-03-23T11:32:30.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.447+0000 7f9a6e83b740 1 -- start start 2024-03-23T11:32:30.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.447+0000 7f9a6e83b740 1 Processor -- start 2024-03-23T11:32:30.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.448+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f205a0 0x565222f497c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.448+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222fc4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.448+0000 7f9a6e83b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6df20 0x565222fc6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.448+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222f89bc0 con 0x565222f6df20 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x565222f6e730 con 0x565222f6a520 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a6e83b740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x565222fc4dd0 con 0x565222f205a0 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a6e83b740 10 client.0 _ll_register_callbacks cb 0x7f9a4800db90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a69c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6df20 0x565222fc6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a69c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6df20 0x565222fc6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55314/0 (socket says 172.21.15.62:55314) 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a69c00640 1 -- 172.21.15.62:0/1179628709 learned_addr learned my addr 172.21.15.62:0/1179628709 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a69200640 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f205a0 0x565222f497c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a5be00640 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222fc4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a69c00640 1 -- 172.21.15.62:0/1179628709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f205a0 msgr2=0x565222f497c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a69c00640 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f205a0 0x565222f497c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a69c00640 1 -- 172.21.15.62:0/1179628709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 msgr2=0x565222fc4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a69c00640 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222fc4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.449+0000 7f9a69c00640 1 -- 172.21.15.62:0/1179628709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565222e84810 con 0x565222f6df20 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.450+0000 7f9a69c00640 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6df20 0x565222fc6450 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f9a640ce290 tx=0x7f9a5c012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c8e61345d03b26 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.450+0000 7f9a52000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.450+0000 7f9a52000640 1 -- 172.21.15.62:0/1179628709 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c015860 con 0x565222f6df20 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.450+0000 7f9a52000640 1 -- 172.21.15.62:0/1179628709 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f9a5c0159c0 con 0x565222f6df20 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.450+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1179628709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a54006b50 con 0x565222f6df20 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.450+0000 7f9a52000640 1 -- 172.21.15.62:0/1179628709 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a5c015b80 con 0x565222f6df20 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.450+0000 7f9a6e83b740 10 client.9603 fetch_fsmap learned FSMap version 107 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.450+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1179628709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x565222e26ad0 con 0x565222f6df20 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.450+0000 7f9a52000640 1 -- 172.21.15.62:0/1179628709 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a54006b50 con 0x565222f6df20 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.451+0000 7f9a52000640 1 -- 172.21.15.62:0/1179628709 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 107) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9a5c015ce0 con 0x565222f6df20 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.451+0000 7f9a6e83b740 10 client.9603 fetch_fsmap finished waiting for FSMap version 107 2024-03-23T11:32:30.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.451+0000 7f9a6e83b740 10 client.9603 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T11:32:30.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.451+0000 7f9a6e83b740 20 client.9603 populate_metadata read hostname 'smithi062' 2024-03-23T11:32:30.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.451+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.62:6834/2500981044,v1:172.21.15.62:6835/2500981044] conn(0x565222f1ce20 0x565222f1d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.451+0000 7f9a6e83b740 4 client.9603 mds_command: new command op to 9507 tid=0[{"prefix": "status"}] 2024-03-23T11:32:30.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.451+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1179628709 --> [v2:172.21.15.62:6834/2500981044,v1:172.21.15.62:6835/2500981044] -- command(tid 0: {"prefix": "status"}) v1 -- 0x565222b7a3c0 con 0x565222f1ce20 2024-03-23T11:32:30.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.451+0000 7f9a69200640 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.62:6834/2500981044,v1:172.21.15.62:6835/2500981044] conn(0x565222f1ce20 0x565222f1d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a69200640 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.62:6834/2500981044,v1:172.21.15.62:6835/2500981044] conn(0x565222f1ce20 0x565222f1d200 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92abf15f623a0619 server_cookie=7822f2859c362e90 in_seq=0 out_seq=0 2024-03-23T11:32:30.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a52000640 10 client.9603 ms_handle_connect on v2:172.21.15.62:6834/2500981044 2024-03-23T11:32:30.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a52000640 1 -- 172.21.15.62:0/1179628709 <== mds.0 v2:172.21.15.62:6834/2500981044 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x565222b7a3c0 con 0x565222f1ce20 2024-03-23T11:32:30.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a52000640 10 client.9603 handle_command_reply: tid=0 2024-03-23T11:32:30.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a6e83b740 1 client.9603 shutdown 2024-03-23T11:32:30.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1179628709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6df20 msgr2=0x565222fc6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6df20 0x565222fc6450 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f9a640ce290 tx=0x7f9a5c012760 comp rx=0 tx=0).stop 2024-03-23T11:32:30.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1179628709 shutdown_connections 2024-03-23T11:32:30.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.62:6834/2500981044,v1:172.21.15.62:6835/2500981044] conn(0x565222f1ce20 0x565222f1d200 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565222f6df20 0x565222fc6450 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565222f6a520 0x565222fc4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a6e83b740 1 --2- 172.21.15.62:0/1179628709 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565222f205a0 0x565222f497c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a52000640 0 client.9603 ms_handle_reset on v2:172.21.15.62:6834/2500981044 2024-03-23T11:32:30.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1179628709 >> 172.21.15.62:0/1179628709 conn(0x565222f1ca40 msgr2=0x565222f66440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:30.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1179628709 shutdown_connections 2024-03-23T11:32:30.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.452+0000 7f9a6e83b740 1 -- 172.21.15.62:0/1179628709 wait complete. 2024-03-23T11:32:30.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6e83b740 20 client.9603 trim_cache size 0 max 16384 2024-03-23T11:32:30.445 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "e8b2848c-f838-4b99-a310-de7a7069c38e", 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 9507, 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 5.0130239530000003, 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 107, 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 122, 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 122, 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 6.8064336399999998 2024-03-23T11:32:30.446 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:30.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 -- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a4006a7a0 msgr2=0x7f9a4006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a4006a7a0 0x7f9a4006cc60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9a54034000 tx=0x7f9a54030000 comp rx=0 tx=0).stop 2024-03-23T11:32:30.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 -- 172.21.15.62:0/457459450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a641518c0 msgr2=0x7f9a64146ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a641518c0 0x7f9a64146ce0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00d770 tx=0x7f9a4c00dc30 comp rx=0 tx=0).stop 2024-03-23T11:32:30.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 -- 172.21.15.62:0/457459450 shutdown_connections 2024-03-23T11:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f9a4006a7a0 0x7f9a4006cc60 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a641518c0 0x7f9a64146ce0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a6406d720 0x7f9a6406dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 --2- 172.21.15.62:0/457459450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a48005bb0 0x7f9a640cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 -- 172.21.15.62:0/457459450 >> 172.21.15.62:0/457459450 conn(0x7f9a640b0090 msgr2=0x7f9a64156090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 -- 172.21.15.62:0/457459450 shutdown_connections 2024-03-23T11:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.453+0000 7f9a6a600640 1 -- 172.21.15.62:0/457459450 wait complete. 2024-03-23T11:32:30.460 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T11:32:30.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.691+0000 7fb401000640 1 Processor -- start 2024-03-23T11:32:30.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.691+0000 7fb401000640 1 -- start start 2024-03-23T11:32:30.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.692+0000 7fb401000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3fc143dd0 0x7fb3fc15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.692+0000 7fb401000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb3fc070090 0x7fb3fc0704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.693+0000 7fb401000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc070a30 con 0x7fb3fc0b2820 2024-03-23T11:32:30.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.693+0000 7fb401000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc0cd140 con 0x7fb3fc070090 2024-03-23T11:32:30.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.693+0000 7fb401000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc0cd3e0 con 0x7fb3fc143dd0 2024-03-23T11:32:30.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.693+0000 7fb3faa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb3fc070090 0x7fb3fc0704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.693+0000 7fb3fb400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3fc143dd0 0x7fb3fc15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.693+0000 7fb3faa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb3fc070090 0x7fb3fc0704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:52952/0 (socket says 172.21.15.62:52952) 2024-03-23T11:32:30.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.693+0000 7fb3faa00640 1 -- 172.21.15.62:0/4205258567 learned_addr learned my addr 172.21.15.62:0/4205258567 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:30.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.694+0000 7fb3faa00640 1 -- 172.21.15.62:0/4205258567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3fc143dd0 msgr2=0x7fb3fc15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:30.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.694+0000 7fb3faa00640 1 --2- 172.21.15.62:0/4205258567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3fc143dd0 0x7fb3fc15abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.694+0000 7fb3faa00640 1 -- 172.21.15.62:0/4205258567 >> v1:172.21.15.120:6789/0 conn(0x7fb3fc0b2820 legacy=0x7fb3fc0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.694+0000 7fb3faa00640 1 -- 172.21.15.62:0/4205258567 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc0cd6c0 con 0x7fb3fc070090 2024-03-23T11:32:30.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.696+0000 7fb3faa00640 1 --2- 172.21.15.62:0/4205258567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3fc070090 0x7fb3fc0704f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009e50 tx=0x7fb3e402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=504d879ae2cedcc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:30.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.696+0000 7fb3fa000640 1 -- 172.21.15.62:0/4205258567 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e402cb00 con 0x7fb3fc070090 2024-03-23T11:32:30.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.696+0000 7fb3fa000640 1 -- 172.21.15.62:0/4205258567 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e402cc60 con 0x7fb3fc070090 2024-03-23T11:32:30.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.696+0000 7fb3fa000640 1 -- 172.21.15.62:0/4205258567 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40343f0 con 0x7fb3fc070090 2024-03-23T11:32:30.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.697+0000 7fb401000640 1 -- 172.21.15.62:0/4205258567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3fc070090 msgr2=0x7fb3fc0704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.697+0000 7fb401000640 1 --2- 172.21.15.62:0/4205258567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3fc070090 0x7fb3fc0704f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009e50 tx=0x7fb3e402bf60 comp rx=0 tx=0).stop 2024-03-23T11:32:30.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.697+0000 7fb401000640 1 -- 172.21.15.62:0/4205258567 shutdown_connections 2024-03-23T11:32:30.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.697+0000 7fb401000640 1 --2- 172.21.15.62:0/4205258567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3fc070090 0x7fb3fc0704f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.697+0000 7fb401000640 1 --2- 172.21.15.62:0/4205258567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3fc143dd0 0x7fb3fc15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.697+0000 7fb401000640 1 -- 172.21.15.62:0/4205258567 >> 172.21.15.62:0/4205258567 conn(0x7fb3fc066050 msgr2=0x7fb3fc069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:30.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.697+0000 7fb401000640 1 -- 172.21.15.62:0/4205258567 shutdown_connections 2024-03-23T11:32:30.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.697+0000 7fb401000640 1 -- 172.21.15.62:0/4205258567 wait complete. 2024-03-23T11:32:30.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.699+0000 7fb401000640 1 Processor -- start 2024-03-23T11:32:30.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.699+0000 7fb401000640 1 -- start start 2024-03-23T11:32:30.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.700+0000 7fb401000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3fc070090 0x7fb3fc15cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.700+0000 7fb401000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3fc0b2820 0x7fb3fc13c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.700+0000 7fb401000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3fc143dd0 0x7fb3fc13c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.700+0000 7fb401000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc139ea0 con 0x7fb3fc070090 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.700+0000 7fb401000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc13ef20 con 0x7fb3fc0b2820 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.700+0000 7fb401000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc13f200 con 0x7fb3fc143dd0 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.701+0000 7fb3fb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3fc070090 0x7fb3fc15cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.701+0000 7fb3fb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3fc070090 0x7fb3fc15cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55324/0 (socket says 172.21.15.62:55324) 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.701+0000 7fb3fb400640 1 -- 172.21.15.62:0/2881436799 learned_addr learned my addr 172.21.15.62:0/2881436799 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.701+0000 7fb3faa00640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3fc0b2820 0x7fb3fc13c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.701+0000 7fb3fbe00640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3fc143dd0 0x7fb3fc13c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.701+0000 7fb3fb400640 1 -- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3fc143dd0 msgr2=0x7fb3fc13c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.701+0000 7fb3fb400640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3fc143dd0 0x7fb3fc13c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.701+0000 7fb3fb400640 1 -- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3fc0b2820 msgr2=0x7fb3fc13c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.701+0000 7fb3fb400640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3fc0b2820 0x7fb3fc13c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.701+0000 7fb3fb400640 1 -- 172.21.15.62:0/2881436799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e4009b30 con 0x7fb3fc070090 2024-03-23T11:32:30.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.701+0000 7fb3fb400640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3fc070090 0x7fb3fc15cc80 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00e870 tx=0x7fb3ec00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:30.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.702+0000 7fb3f3400640 1 -- 172.21.15.62:0/2881436799 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec0094f0 con 0x7fb3fc070090 2024-03-23T11:32:30.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.702+0000 7fb3f3400640 1 -- 172.21.15.62:0/2881436799 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ec00c940 con 0x7fb3fc070090 2024-03-23T11:32:30.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.702+0000 7fb401000640 1 -- 172.21.15.62:0/2881436799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc13f4e0 con 0x7fb3fc070090 2024-03-23T11:32:30.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.702+0000 7fb3f3400640 1 -- 172.21.15.62:0/2881436799 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec00ccf0 con 0x7fb3fc070090 2024-03-23T11:32:30.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.702+0000 7fb401000640 1 -- 172.21.15.62:0/2881436799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc15d1c0 con 0x7fb3fc070090 2024-03-23T11:32:30.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.703+0000 7fb401000640 1 -- 172.21.15.62:0/2881436799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc15d3b0 con 0x7fb3fc070090 2024-03-23T11:32:30.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.704+0000 7fb3f3400640 1 -- 172.21.15.62:0/2881436799 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb3ec013470 con 0x7fb3fc070090 2024-03-23T11:32:30.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.705+0000 7fb3f3400640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3d806a7d0 0x7fb3d806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:30.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.705+0000 7fb3f3400640 1 -- 172.21.15.62:0/2881436799 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb3ec088fe0 con 0x7fb3fc070090 2024-03-23T11:32:30.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.706+0000 7fb3faa00640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3d806a7d0 0x7fb3d806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:30.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.707+0000 7fb3faa00640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3d806a7d0 0x7fb3d806cc90 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4035000 tx=0x7fb3e4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:30.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.710+0000 7fb3f3400640 1 -- 172.21.15.62:0/2881436799 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb3ec054290 con 0x7fb3fc070090 2024-03-23T11:32:30.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.950+0000 7fb401000640 1 -- 172.21.15.62:0/2881436799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb3fc066bb0 con 0x7fb3fc070090 2024-03-23T11:32:30.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.951+0000 7fb3f3400640 1 -- 172.21.15.62:0/2881436799 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v107) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb3ec0578b0 con 0x7fb3fc070090 2024-03-23T11:32:30.941 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T11:32:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 -- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3d806a7d0 msgr2=0x7fb3d806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3d806a7d0 0x7fb3d806cc90 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4035000 tx=0x7fb3e4030000 comp rx=0 tx=0).stop 2024-03-23T11:32:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 -- 172.21.15.62:0/2881436799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3fc070090 msgr2=0x7fb3fc15cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3fc070090 0x7fb3fc15cc80 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00e870 tx=0x7fb3ec00ed30 comp rx=0 tx=0).stop 2024-03-23T11:32:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 -- 172.21.15.62:0/2881436799 shutdown_connections 2024-03-23T11:32:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb3d806a7d0 0x7fb3d806cc90 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3fc143dd0 0x7fb3fc13c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3fc0b2820 0x7fb3fc13c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 --2- 172.21.15.62:0/2881436799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3fc070090 0x7fb3fc15cc80 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 -- 172.21.15.62:0/2881436799 >> 172.21.15.62:0/2881436799 conn(0x7fb3fc066050 msgr2=0x7fb3fc0aebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 -- 172.21.15.62:0/2881436799 shutdown_connections 2024-03-23T11:32:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:30.954+0000 7fb401000640 1 -- 172.21.15.62:0/2881436799 wait complete. 2024-03-23T11:32:30.961 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T11:32:30.961 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T11:32:30.962 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-03-23T11:32:30.962 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T11:32:30.962 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:30.962 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T11:32:30.962 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T11:32:30.962 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T11:32:30.962 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:32:30.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:30.962 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-03-23T11:32:31.012 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T11:32:31.012 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T11:32:31.012 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T11:32:31.012 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:31.012 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-03-23T11:32:31.012 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:31.012 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:32:31.012 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:32:31.012 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 41806sec preferred_lft 41806sec 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff 2024-03-23T11:32:31.013 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T11:32:31.014 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:31.014 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::707b:57ff:fe9a:eadf/64 scope link 2024-03-23T11:32:31.014 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:31.014 INFO:teuthology.orchestra.run.smithi062.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:32:31.014 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether b2:76:2c:eb:dd:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:32:31.014 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::b076:2cff:feeb:dde6/64 scope link 2024-03-23T11:32:31.014 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:31.014 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T11:32:31.014 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 26:6c:12:70:af:25 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:32:31.014 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::246c:12ff:fe70:af25/64 scope link 2024-03-23T11:32:31.014 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-03-23T11:32:31.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:31.015 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T11:32:31.109 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:32:31.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:31.114 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-03-23T11:32:31.170 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-03-23T11:32:31.171 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T11:32:31.171 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T11:32:31.171 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T11:32:31.171 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T11:32:31.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:31.172 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec bash -c ' 2024-03-23T11:32:31.172 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-03-23T11:32:31.172 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T11:32:31.172 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T11:32:31.172 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T11:32:31.172 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-03-23T11:32:31.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:31.365 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:32:31.418 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:32:31.418 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:32:31.419 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:32:31.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:31.420 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-03-23T11:32:31.520 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T11:32:31.520 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:32:31.520 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:31.571 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T11:32:31.571 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T11:32:31.571 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:31.626 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:32:31.730 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:32:31.730 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:32:31.830 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:32:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:32:31.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:31.835 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:32:31.890 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T11:32:31.890 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T11:32:31.897 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-03-23T11:32:31.975 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T11:32:31.975 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T11:32:32.043 INFO:teuthology.orchestra.run.smithi062.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T11:32:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T11:32:32.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:32.047 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-03-23T11:32:32.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-03-23T11:32:32.093+0000 7f0d76c0d480 -1 init, newargv = 0x555abd1836d0 newargc=15 2024-03-23T11:32:32.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[82214]: starting ceph client 2024-03-23T11:32:32.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[82214]: starting fuse 2024-03-23T11:32:32.091 INFO:teuthology.orchestra.run.smithi062.stdout:40 2024-03-23T11:32:32.092 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T11:32:32.092 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:32.092 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-03-23T11:32:32.092 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-03-23T11:32:32.092 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:32.092 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-03-23T11:32:32.092 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-03-23T11:32:32.093 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-03-23T11:32:32.094 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-03-23T11:32:32.094 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T11:32:32.094 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:32.094 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-03-23T11:32:32.094 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:32.283 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.82214.asok 2024-03-23T11:32:32.463 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.82214.asok 2024-03-23T11:32:32.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:32.463 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.82214.asok status 2024-03-23T11:32:32.715 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:32.715 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-03-23T11:32:32.715 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "b630d6cd702af63dfefcf906294a092ddce1c609", 2024-03-23T11:32:32.715 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.2-820-gb630d6cd (b630d6cd702af63dfefcf906294a092ddce1c609) reef (stable)", 2024-03-23T11:32:32.715 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-03-23T11:32:32.715 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-03-23T11:32:32.715 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T11:32:32.715 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "82214", 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 9612, 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 9612 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:32:32.716 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 598148745 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 598148745 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.9612 192.168.0.1:0/598148745", 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/598148745", 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 107, 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 122, 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-03-23T11:32:32.717 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-03-23T11:32:32.718 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:32.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:32.729 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-03-23T11:32:32.729 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:32.788 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-03-23T11:32:32.788 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:32.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:32.789 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:32.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:32.888 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-03-23T11:32:32.972 INFO:teuthology.orchestra.run.smithi062.stdout:./ 2024-03-23T11:32:32.973 INFO:teuthology.orchestra.run.smithi062.stdout:./parent 2024-03-23T11:32:32.973 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/flushed 2024-03-23T11:32:32.973 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/flushed/charlie 2024-03-23T11:32:32.973 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/flushed/alpha 2024-03-23T11:32:32.973 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/flushed/bravo 2024-03-23T11:32:32.973 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/unflushed 2024-03-23T11:32:32.973 INFO:teuthology.orchestra.run.smithi062.stdout:./parent/unflushed/jfile 2024-03-23T11:32:32.974 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:32.974 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:32.974 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:32.974 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:32.974 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-03-23T11:32:32.974 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:33.113 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-23T11:32:33.293 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:33.293 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:33.293 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:33.294 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:33.294 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-03-23T11:32:33.294 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:33.491 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627776 2024-03-23T11:32:33.672 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:33.672 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:33.672 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:33.672 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:33.672 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-03-23T11:32:33.672 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:33.864 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627777 2024-03-23T11:32:34.043 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:34.043 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:34.043 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:34.043 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:34.043 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-03-23T11:32:34.044 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:34.238 INFO:teuthology.orchestra.run.smithi062.stdout:1099511628280 2024-03-23T11:32:34.417 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:34.417 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:34.417 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:34.417 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:34.417 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-03-23T11:32:34.417 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:34.612 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627778 2024-03-23T11:32:34.790 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:34.791 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:34.791 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:34.791 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:34.791 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-03-23T11:32:34.791 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:34.984 INFO:teuthology.orchestra.run.smithi062.stdout:1099511628279 2024-03-23T11:32:35.162 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:35.163 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:35.163 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:35.163 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:35.163 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-03-23T11:32:35.163 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:35.357 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627779 2024-03-23T11:32:35.536 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T11:32:35.536 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-03-23T11:32:35.536 DEBUG:teuthology.orchestra.run.smithi062:> import stat 2024-03-23T11:32:35.536 DEBUG:teuthology.orchestra.run.smithi062:> 2024-03-23T11:32:35.536 DEBUG:teuthology.orchestra.run.smithi062:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-03-23T11:32:35.536 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-03-23T11:32:35.730 INFO:teuthology.orchestra.run.smithi062.stdout:1099511627780 2024-03-23T11:32:35.910 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T11:32:35.910 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:32:36.080 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T11:32:36.194 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T11:32:36.194 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:32:36.194 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:32:36.255 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:32:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.495+0000 7f3221a00640 1 Processor -- start 2024-03-23T11:32:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.495+0000 7f3221a00640 1 -- start start 2024-03-23T11:32:36.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.496+0000 7f3221a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f321c070090 0x7f321c0704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.496+0000 7f3221a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f321c0b2820 0x7f321c0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.496+0000 7f3221a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c070a30 con 0x7f321c143dd0 2024-03-23T11:32:36.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.496+0000 7f3221a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f321c0cd140 con 0x7f321c0b2820 2024-03-23T11:32:36.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.496+0000 7f3221a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c0b3180 con 0x7f321c070090 2024-03-23T11:32:36.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.497+0000 7f3221000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f321c0b2820 0x7f321c0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.497+0000 7f3221000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f321c0b2820 0x7f321c0b2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55328/0 (socket says 172.21.15.62:55328) 2024-03-23T11:32:36.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.497+0000 7f3221000640 1 -- 172.21.15.62:0/3858639450 learned_addr learned my addr 172.21.15.62:0/3858639450 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:36.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.497+0000 7f321b400640 1 --2- 172.21.15.62:0/3858639450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f321c070090 0x7f321c0704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.498+0000 7f3221000640 1 -- 172.21.15.62:0/3858639450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f321c070090 msgr2=0x7f321c0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:36.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.498+0000 7f3221000640 1 --2- 172.21.15.62:0/3858639450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f321c070090 0x7f321c0704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.498+0000 7f3221000640 1 -- 172.21.15.62:0/3858639450 >> v1:172.21.15.120:6789/0 conn(0x7f321c143dd0 legacy=0x7f321c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:36.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.498+0000 7f3221000640 1 -- 172.21.15.62:0/3858639450 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c0cd3e0 con 0x7f321c0b2820 2024-03-23T11:32:36.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.499+0000 7f3221000640 1 --2- 172.21.15.62:0/3858639450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f321c0b2820 0x7f321c0b2c40 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f3214009e50 tx=0x7f321402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9762ee176271401f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:36.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.499+0000 7f321aa00640 1 -- 172.21.15.62:0/3858639450 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321402cb00 con 0x7f321c0b2820 2024-03-23T11:32:36.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.499+0000 7f321aa00640 1 -- 172.21.15.62:0/3858639450 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321402cc60 con 0x7f321c0b2820 2024-03-23T11:32:36.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.499+0000 7f321aa00640 1 -- 172.21.15.62:0/3858639450 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32140343f0 con 0x7f321c0b2820 2024-03-23T11:32:36.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.500+0000 7f3221a00640 1 -- 172.21.15.62:0/3858639450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f321c0b2820 msgr2=0x7f321c0b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:36.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.500+0000 7f3221a00640 1 --2- 172.21.15.62:0/3858639450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f321c0b2820 0x7f321c0b2c40 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f3214009e50 tx=0x7f321402bf60 comp rx=0 tx=0).stop 2024-03-23T11:32:36.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.500+0000 7f3221a00640 1 -- 172.21.15.62:0/3858639450 shutdown_connections 2024-03-23T11:32:36.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.500+0000 7f3221a00640 1 --2- 172.21.15.62:0/3858639450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f321c0b2820 0x7f321c0b2c40 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.500+0000 7f3221a00640 1 --2- 172.21.15.62:0/3858639450 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f321c070090 0x7f321c0704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.500+0000 7f3221a00640 1 -- 172.21.15.62:0/3858639450 >> 172.21.15.62:0/3858639450 conn(0x7f321c066050 msgr2=0x7f321c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:36.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.500+0000 7f3221a00640 1 -- 172.21.15.62:0/3858639450 shutdown_connections 2024-03-23T11:32:36.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.500+0000 7f3221a00640 1 -- 172.21.15.62:0/3858639450 wait complete. 2024-03-23T11:32:36.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.502+0000 7f3221a00640 1 Processor -- start 2024-03-23T11:32:36.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.502+0000 7f3221a00640 1 -- start start 2024-03-23T11:32:36.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.503+0000 7f3221a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f321c070090 0x7f321c1354e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.503+0000 7f3221a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f321c0b2820 0x7f321c1332e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.503+0000 7f3221a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f321c143dd0 0x7f321c15a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.503+0000 7f3221a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c133820 con 0x7f321c0b2820 2024-03-23T11:32:36.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.503+0000 7f3221a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c133ac0 con 0x7f321c070090 2024-03-23T11:32:36.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.503+0000 7f3221a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c133da0 con 0x7f321c143dd0 2024-03-23T11:32:36.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.503+0000 7f321b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f321c0b2820 0x7f321c1332e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.503+0000 7f321b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f321c0b2820 0x7f321c1332e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55336/0 (socket says 172.21.15.62:55336) 2024-03-23T11:32:36.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.503+0000 7f321b400640 1 -- 172.21.15.62:0/1111614794 learned_addr learned my addr 172.21.15.62:0/1111614794 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:36.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.504+0000 7f321be00640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f321c070090 0x7f321c1354e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.504+0000 7f3221000640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f321c143dd0 0x7f321c15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.504+0000 7f321b400640 1 -- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f321c143dd0 msgr2=0x7f321c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:36.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.504+0000 7f321b400640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f321c143dd0 0x7f321c15a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.504+0000 7f321b400640 1 -- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f321c070090 msgr2=0x7f321c1354e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:36.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.504+0000 7f321b400640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f321c070090 0x7f321c1354e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.504+0000 7f321b400640 1 -- 172.21.15.62:0/1111614794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3214009b30 con 0x7f321c0b2820 2024-03-23T11:32:36.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.504+0000 7f321b400640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f321c0b2820 0x7f321c1332e0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f320800d800 tx=0x7f320800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:36.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.505+0000 7f3213e00640 1 -- 172.21.15.62:0/1111614794 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320800b6c0 con 0x7f321c0b2820 2024-03-23T11:32:36.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.505+0000 7f3213e00640 1 -- 172.21.15.62:0/1111614794 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320800b840 con 0x7f321c0b2820 2024-03-23T11:32:36.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.505+0000 7f3221a00640 1 -- 172.21.15.62:0/1111614794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321c134080 con 0x7f321c0b2820 2024-03-23T11:32:36.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.505+0000 7f3213e00640 1 -- 172.21.15.62:0/1111614794 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320800bbf0 con 0x7f321c0b2820 2024-03-23T11:32:36.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.505+0000 7f3221a00640 1 -- 172.21.15.62:0/1111614794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321c134640 con 0x7f321c0b2820 2024-03-23T11:32:36.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.506+0000 7f3221a00640 1 -- 172.21.15.62:0/1111614794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321c066bb0 con 0x7f321c0b2820 2024-03-23T11:32:36.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.507+0000 7f3213e00640 1 -- 172.21.15.62:0/1111614794 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f320800bd50 con 0x7f321c0b2820 2024-03-23T11:32:36.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.508+0000 7f3213e00640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f31fc06a7a0 0x7f31fc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.508+0000 7f3213e00640 1 -- 172.21.15.62:0/1111614794 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f3208089570 con 0x7f321c0b2820 2024-03-23T11:32:36.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.508+0000 7f321be00640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f31fc06a7a0 0x7f31fc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.509+0000 7f321be00640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f31fc06a7a0 0x7f31fc06cc60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f320c000f60 tx=0x7f320c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:36.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.512+0000 7f3213e00640 1 -- 172.21.15.62:0/1111614794 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3208054800 con 0x7f321c0b2820 2024-03-23T11:32:36.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.711+0000 7f3221a00640 1 -- 172.21.15.62:0/1111614794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f321c15d0c0 con 0x7f321c0b2820 2024-03-23T11:32:36.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.712+0000 7f3213e00640 1 -- 172.21.15.62:0/1111614794 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+16413 (secure 0 0 0) 0x7f3208057e20 con 0x7f321c0b2820 2024-03-23T11:32:36.702 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:36.702 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":122,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:32:23.551951+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6835/4129977589":"2024-03-24T11:32:23.551904+0000","172.21.15.62:6834/4129977589":"2024-03-24T11:32:23.551904+0000","172.21.15.120:6837/2585024035":"2024-03-24T11:32:04.365655+0000","172.21.15.120:6835/2585024035":"2024-03-24T11:32:04.365655+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:32:36.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.716+0000 7f3221a00640 1 -- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f31fc06a7a0 msgr2=0x7f31fc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:36.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.716+0000 7f3221a00640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f31fc06a7a0 0x7f31fc06cc60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f320c000f60 tx=0x7f320c009040 comp rx=0 tx=0).stop 2024-03-23T11:32:36.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.716+0000 7f3221a00640 1 -- 172.21.15.62:0/1111614794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f321c0b2820 msgr2=0x7f321c1332e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:36.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.716+0000 7f3221a00640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f321c0b2820 0x7f321c1332e0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f320800d800 tx=0x7f320800dcc0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.716+0000 7f3221a00640 1 -- 172.21.15.62:0/1111614794 shutdown_connections 2024-03-23T11:32:36.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.716+0000 7f3221a00640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f31fc06a7a0 0x7f31fc06cc60 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.716+0000 7f3221a00640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f321c0b2820 0x7f321c1332e0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.716+0000 7f3221a00640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f321c143dd0 0x7f321c15a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.716+0000 7f3221a00640 1 --2- 172.21.15.62:0/1111614794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f321c070090 0x7f321c1354e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.716+0000 7f3221a00640 1 -- 172.21.15.62:0/1111614794 >> 172.21.15.62:0/1111614794 conn(0x7f321c066050 msgr2=0x7f321c06c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:36.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.716+0000 7f3221a00640 1 -- 172.21.15.62:0/1111614794 shutdown_connections 2024-03-23T11:32:36.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.717+0000 7f3221a00640 1 -- 172.21.15.62:0/1111614794 wait complete. 2024-03-23T11:32:36.726 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:31:22.127079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:31:23.062387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:32:36.726 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:32:36.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.965+0000 7fb105e00640 1 Processor -- start 2024-03-23T11:32:36.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.965+0000 7fb105e00640 1 -- start start 2024-03-23T11:32:36.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.967+0000 7fb105e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb10015a530 0x7fb1000cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.967+0000 7fb105e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1000cc000 0x7fb10006f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.967+0000 7fb105e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb10006f820 0x7fb1000ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.967+0000 7fb105e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10006eb70 con 0x7fb10006f820 2024-03-23T11:32:36.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.967+0000 7fb105e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1000cc7e0 con 0x7fb10015a530 2024-03-23T11:32:36.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.967+0000 7fb105e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000ccac0 con 0x7fb1000cc000 2024-03-23T11:32:36.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.967+0000 7fb0ff400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb10006f820 0x7fb1000ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.968+0000 7fb0ff400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb10006f820 0x7fb1000ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55340/0 (socket says 172.21.15.62:55340) 2024-03-23T11:32:36.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.968+0000 7fb0ff400640 1 -- 172.21.15.62:0/1300783999 learned_addr learned my addr 172.21.15.62:0/1300783999 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:36.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.968+0000 7fb0fea00640 1 --2- 172.21.15.62:0/1300783999 >> v2:172.21.15.120:3300/0 conn(0x7fb10015a530 0x7fb1000cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.968+0000 7fb0f6000640 1 --2- 172.21.15.62:0/1300783999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1000cc000 0x7fb10006f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.968+0000 7fb0ff400640 1 -- 172.21.15.62:0/1300783999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1000cc000 msgr2=0x7fb10006f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:36.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.968+0000 7fb0ff400640 1 --2- 172.21.15.62:0/1300783999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1000cc000 0x7fb10006f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.968+0000 7fb0ff400640 1 -- 172.21.15.62:0/1300783999 >> v2:172.21.15.120:3300/0 conn(0x7fb10015a530 msgr2=0x7fb1000cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:36.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.968+0000 7fb0ff400640 1 --2- 172.21.15.62:0/1300783999 >> v2:172.21.15.120:3300/0 conn(0x7fb10015a530 0x7fb1000cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.968+0000 7fb0ff400640 1 -- 172.21.15.62:0/1300783999 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1000ccda0 con 0x7fb10006f820 2024-03-23T11:32:36.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.969+0000 7fb0ff400640 1 --2- 172.21.15.62:0/1300783999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb10006f820 0x7fb1000ae6b0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8009870 tx=0x7fb0f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e735631227249880 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:36.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb0fe000640 1 -- 172.21.15.62:0/1300783999 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8031640 con 0x7fb10006f820 2024-03-23T11:32:36.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb0fe000640 1 -- 172.21.15.62:0/1300783999 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f80317a0 con 0x7fb10006f820 2024-03-23T11:32:36.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb0fe000640 1 -- 172.21.15.62:0/1300783999 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8031b30 con 0x7fb10006f820 2024-03-23T11:32:36.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb105e00640 1 -- 172.21.15.62:0/1300783999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb10006f820 msgr2=0x7fb1000ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:36.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb105e00640 1 --2- 172.21.15.62:0/1300783999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb10006f820 0x7fb1000ae6b0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8009870 tx=0x7fb0f802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb105e00640 1 -- 172.21.15.62:0/1300783999 shutdown_connections 2024-03-23T11:32:36.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb105e00640 1 --2- 172.21.15.62:0/1300783999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb10006f820 0x7fb1000ae6b0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb105e00640 1 --2- 172.21.15.62:0/1300783999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1000cc000 0x7fb10006f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb105e00640 1 --2- 172.21.15.62:0/1300783999 >> v2:172.21.15.120:3300/0 conn(0x7fb10015a530 0x7fb1000cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb105e00640 1 -- 172.21.15.62:0/1300783999 >> 172.21.15.62:0/1300783999 conn(0x7fb1000b1370 msgr2=0x7fb1000b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:36.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb105e00640 1 -- 172.21.15.62:0/1300783999 shutdown_connections 2024-03-23T11:32:36.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.970+0000 7fb105e00640 1 -- 172.21.15.62:0/1300783999 wait complete. 2024-03-23T11:32:36.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.972+0000 7fb105e00640 1 Processor -- start 2024-03-23T11:32:36.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.972+0000 7fb105e00640 1 -- start start 2024-03-23T11:32:36.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.973+0000 7fb105e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1000cc000 0x7fb1001f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.973+0000 7fb105e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb10015a530 0x7fb1000cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.973+0000 7fb105e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb10015b670 0x7fb10015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.973+0000 7fb105e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000c9bc0 con 0x7fb10015a530 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.973+0000 7fb105e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000c9ea0 con 0x7fb10015b670 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.973+0000 7fb105e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000ca180 con 0x7fb1000cc000 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.973+0000 7fb0f6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb10015a530 0x7fb1000cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.973+0000 7fb0f6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb10015a530 0x7fb1000cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55346/0 (socket says 172.21.15.62:55346) 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.973+0000 7fb0f6000640 1 -- 172.21.15.62:0/1964804639 learned_addr learned my addr 172.21.15.62:0/1964804639 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.974+0000 7fb0ff400640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb10015b670 0x7fb10015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.974+0000 7fb0fea00640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1000cc000 0x7fb1001f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.974+0000 7fb0f6000640 1 -- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1000cc000 msgr2=0x7fb1001f1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.974+0000 7fb0f6000640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1000cc000 0x7fb1001f1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.974+0000 7fb0f6000640 1 -- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb10015b670 msgr2=0x7fb10015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.974+0000 7fb0f6000640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb10015b670 0x7fb10015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.974+0000 7fb0f6000640 1 -- 172.21.15.62:0/1964804639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f80094d0 con 0x7fb10015a530 2024-03-23T11:32:36.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.974+0000 7fb0ff400640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb10015b670 0x7fb10015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:36.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.974+0000 7fb0f6000640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb10015a530 0x7fb1000cb4c0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800d800 tx=0x7fb0e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:36.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.975+0000 7fb0f7e00640 1 -- 172.21.15.62:0/1964804639 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e800b6c0 con 0x7fb10015a530 2024-03-23T11:32:36.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.975+0000 7fb0f7e00640 1 -- 172.21.15.62:0/1964804639 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e800b820 con 0x7fb10015a530 2024-03-23T11:32:36.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.975+0000 7fb0f7e00640 1 -- 172.21.15.62:0/1964804639 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e800bbb0 con 0x7fb10015a530 2024-03-23T11:32:36.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.975+0000 7fb105e00640 1 -- 172.21.15.62:0/1964804639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1000ca460 con 0x7fb10015a530 2024-03-23T11:32:36.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.975+0000 7fb105e00640 1 -- 172.21.15.62:0/1964804639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10015c420 con 0x7fb10015a530 2024-03-23T11:32:36.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.976+0000 7fb105e00640 1 -- 172.21.15.62:0/1964804639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10006ab80 con 0x7fb10015a530 2024-03-23T11:32:36.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.977+0000 7fb0f7e00640 1 -- 172.21.15.62:0/1964804639 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fb0e800bd10 con 0x7fb10015a530 2024-03-23T11:32:36.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.978+0000 7fb0f7e00640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0dc06a7a0 0x7fb0dc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:36.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.978+0000 7fb0f7e00640 1 -- 172.21.15.62:0/1964804639 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb0e80894d0 con 0x7fb10015a530 2024-03-23T11:32:36.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.978+0000 7fb0fea00640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0dc06a7a0 0x7fb0dc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.980+0000 7fb0fea00640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0dc06a7a0 0x7fb0dc06cc60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb10015cb20 tx=0x7fb0f000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:36.982+0000 7fb0f7e00640 1 -- 172.21.15.62:0/1964804639 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb0e8054760 con 0x7fb10015a530 2024-03-23T11:32:37.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.187+0000 7fb105e00640 1 -- 172.21.15.62:0/1964804639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb1000672c0 con 0x7fb10015a530 2024-03-23T11:32:37.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.188+0000 7fb0f7e00640 1 -- 172.21.15.62:0/1964804639 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb0e805e030 con 0x7fb10015a530 2024-03-23T11:32:37.179 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:32:37.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 -- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0dc06a7a0 msgr2=0x7fb0dc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0dc06a7a0 0x7fb0dc06cc60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb10015cb20 tx=0x7fb0f000b040 comp rx=0 tx=0).stop 2024-03-23T11:32:37.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 -- 172.21.15.62:0/1964804639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb10015a530 msgr2=0x7fb1000cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb10015a530 0x7fb1000cb4c0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800d800 tx=0x7fb0e800dcc0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 -- 172.21.15.62:0/1964804639 shutdown_connections 2024-03-23T11:32:37.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fb0dc06a7a0 0x7fb0dc06cc60 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb10015a530 0x7fb1000cb4c0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb10015b670 0x7fb10015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 --2- 172.21.15.62:0/1964804639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb1000cc000 0x7fb1001f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 -- 172.21.15.62:0/1964804639 >> 172.21.15.62:0/1964804639 conn(0x7fb1000b1370 msgr2=0x7fb1000b4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:37.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 -- 172.21.15.62:0/1964804639 shutdown_connections 2024-03-23T11:32:37.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.191+0000 7fb105e00640 1 -- 172.21.15.62:0/1964804639 wait complete. 2024-03-23T11:32:37.199 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:32:37.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.421+0000 7fc9e7200640 1 Processor -- start 2024-03-23T11:32:37.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.422+0000 7fc9e7200640 1 -- start start 2024-03-23T11:32:37.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.423+0000 7fc9e7200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc9e815fcf0 0x7fc9e80d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.423+0000 7fc9e7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9e8160420 0x7fc9e8160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.423+0000 7fc9e7200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc9e80d06d0 0x7fc9e80d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.423+0000 7fc9e7200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e8160dc0 con 0x7fc9e815fcf0 2024-03-23T11:32:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.423+0000 7fc9e7200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e80d3540 con 0x7fc9e80d06d0 2024-03-23T11:32:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.423+0000 7fc9e7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e80ce510 con 0x7fc9e8160420 2024-03-23T11:32:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.423+0000 7fc9e5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc9e815fcf0 0x7fc9e80d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.423+0000 7fc9e5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc9e815fcf0 0x7fc9e80d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55582/0 (socket says 172.21.15.62:55582) 2024-03-23T11:32:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.423+0000 7fc9e5e00640 1 -- 172.21.15.62:0/1567204341 learned_addr learned my addr 172.21.15.62:0/1567204341 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:37.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.423+0000 7fc9e6800640 1 --2- 172.21.15.62:0/1567204341 >> v2:172.21.15.120:3300/0 conn(0x7fc9e80d06d0 0x7fc9e80d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:37.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.424+0000 7fc9e5400640 1 --2- 172.21.15.62:0/1567204341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9e8160420 0x7fc9e8160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:37.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.424+0000 7fc9e5e00640 1 -- 172.21.15.62:0/1567204341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9e8160420 msgr2=0x7fc9e8160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:37.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.424+0000 7fc9e5e00640 1 --2- 172.21.15.62:0/1567204341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9e8160420 0x7fc9e8160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.424+0000 7fc9e5e00640 1 -- 172.21.15.62:0/1567204341 >> v2:172.21.15.120:3300/0 conn(0x7fc9e80d06d0 msgr2=0x7fc9e80d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:37.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.424+0000 7fc9e5e00640 1 --2- 172.21.15.62:0/1567204341 >> v2:172.21.15.120:3300/0 conn(0x7fc9e80d06d0 0x7fc9e80d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.424+0000 7fc9e5e00640 1 -- 172.21.15.62:0/1567204341 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e80cdc70 con 0x7fc9e815fcf0 2024-03-23T11:32:37.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.425+0000 7fc9e5e00640 1 --2- 172.21.15.62:0/1567204341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9e815fcf0 0x7fc9e80d3000 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fc9e808c3e0 tx=0x7fc9d002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de5efa04e18d367 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:37.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.426+0000 7fc9e4a00640 1 -- 172.21.15.62:0/1567204341 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d00317d0 con 0x7fc9e815fcf0 2024-03-23T11:32:37.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.426+0000 7fc9e4a00640 1 -- 172.21.15.62:0/1567204341 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d0031930 con 0x7fc9e815fcf0 2024-03-23T11:32:37.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.426+0000 7fc9e4a00640 1 -- 172.21.15.62:0/1567204341 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0031cc0 con 0x7fc9e815fcf0 2024-03-23T11:32:37.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.426+0000 7fc9e7200640 1 -- 172.21.15.62:0/1567204341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9e815fcf0 msgr2=0x7fc9e80d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.426+0000 7fc9e7200640 1 --2- 172.21.15.62:0/1567204341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9e815fcf0 0x7fc9e80d3000 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fc9e808c3e0 tx=0x7fc9d002b870 comp rx=0 tx=0).stop 2024-03-23T11:32:37.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.426+0000 7fc9e7200640 1 -- 172.21.15.62:0/1567204341 shutdown_connections 2024-03-23T11:32:37.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.426+0000 7fc9e7200640 1 --2- 172.21.15.62:0/1567204341 >> v2:172.21.15.120:3300/0 conn(0x7fc9e80d06d0 0x7fc9e80d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.426+0000 7fc9e7200640 1 --2- 172.21.15.62:0/1567204341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9e8160420 0x7fc9e8160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.426+0000 7fc9e7200640 1 --2- 172.21.15.62:0/1567204341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9e815fcf0 0x7fc9e80d3000 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.426+0000 7fc9e7200640 1 -- 172.21.15.62:0/1567204341 >> 172.21.15.62:0/1567204341 conn(0x7fc9e8089f60 msgr2=0x7fc9e80d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:37.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.427+0000 7fc9e7200640 1 -- 172.21.15.62:0/1567204341 shutdown_connections 2024-03-23T11:32:37.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.427+0000 7fc9e7200640 1 -- 172.21.15.62:0/1567204341 wait complete. 2024-03-23T11:32:37.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.429+0000 7fc9e7200640 1 Processor -- start 2024-03-23T11:32:37.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.429+0000 7fc9e7200640 1 -- start start 2024-03-23T11:32:37.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.430+0000 7fc9e7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9e80d06d0 0x7fc9e8149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.430+0000 7fc9e7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9e815fcf0 0x7fc9e80c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.430+0000 7fc9e7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9e8160420 0x7fc9e80c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.430+0000 7fc9e7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e81592f0 con 0x7fc9e80d06d0 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.430+0000 7fc9e7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e814a460 con 0x7fc9e815fcf0 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.430+0000 7fc9e7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e80c8e90 con 0x7fc9e8160420 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9e80d06d0 0x7fc9e8149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9e80d06d0 0x7fc9e8149f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55584/0 (socket says 172.21.15.62:55584) 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e5e00640 1 -- 172.21.15.62:0/2633484588 learned_addr learned my addr 172.21.15.62:0/2633484588 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e5400640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9e815fcf0 0x7fc9e80c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e6800640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9e8160420 0x7fc9e80c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e5e00640 1 -- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9e8160420 msgr2=0x7fc9e80c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e5e00640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9e8160420 0x7fc9e80c5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e5e00640 1 -- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9e815fcf0 msgr2=0x7fc9e80c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e5e00640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9e815fcf0 0x7fc9e80c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e5e00640 1 -- 172.21.15.62:0/2633484588 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d00094d0 con 0x7fc9e80d06d0 2024-03-23T11:32:37.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e5400640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9e815fcf0 0x7fc9e80c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:37.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.431+0000 7fc9e6800640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9e8160420 0x7fc9e80c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.432+0000 7fc9e5e00640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9e80d06d0 0x7fc9e8149f20 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0000c00 tx=0x7fc9d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.432+0000 7fc9de000640 1 -- 172.21.15.62:0/2633484588 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0032870 con 0x7fc9e80d06d0 2024-03-23T11:32:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.432+0000 7fc9de000640 1 -- 172.21.15.62:0/2633484588 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d0032a10 con 0x7fc9e80d06d0 2024-03-23T11:32:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.432+0000 7fc9de000640 1 -- 172.21.15.62:0/2633484588 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0032dc0 con 0x7fc9e80d06d0 2024-03-23T11:32:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.432+0000 7fc9e7200640 1 -- 172.21.15.62:0/2633484588 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e80c88d0 con 0x7fc9e80d06d0 2024-03-23T11:32:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.432+0000 7fc9e7200640 1 -- 172.21.15.62:0/2633484588 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e8158f10 con 0x7fc9e80d06d0 2024-03-23T11:32:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.433+0000 7fc9e7200640 1 -- 172.21.15.62:0/2633484588 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e8066bb0 con 0x7fc9e80d06d0 2024-03-23T11:32:37.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.434+0000 7fc9de000640 1 -- 172.21.15.62:0/2633484588 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc9d0030070 con 0x7fc9e80d06d0 2024-03-23T11:32:37.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.435+0000 7fc9de000640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc9c406a7a0 0x7fc9c406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.435+0000 7fc9de000640 1 -- 172.21.15.62:0/2633484588 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fc9d00aad80 con 0x7fc9e80d06d0 2024-03-23T11:32:37.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.436+0000 7fc9e5400640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc9c406a7a0 0x7fc9c406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:37.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.437+0000 7fc9e5400640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc9c406a7a0 0x7fc9c406cc60 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc9e814ab10 tx=0x7fc9d800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:37.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.439+0000 7fc9de000640 1 -- 172.21.15.62:0/2633484588 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc9d0076090 con 0x7fc9e80d06d0 2024-03-23T11:32:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.639+0000 7fc9e7200640 1 -- 172.21.15.62:0/2633484588 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc9e80cea40 con 0x7fc9e80d06d0 2024-03-23T11:32:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.640+0000 7fc9de000640 1 -- 172.21.15.62:0/2633484588 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v122) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc9d0031ed0 con 0x7fc9e80d06d0 2024-03-23T11:32:37.631 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 -- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc9c406a7a0 msgr2=0x7fc9c406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc9c406a7a0 0x7fc9c406cc60 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc9e814ab10 tx=0x7fc9d800b040 comp rx=0 tx=0).stop 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 -- 172.21.15.62:0/2633484588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9e80d06d0 msgr2=0x7fc9e8149f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9e80d06d0 0x7fc9e8149f20 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0000c00 tx=0x7fc9d00046c0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 -- 172.21.15.62:0/2633484588 shutdown_connections 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc9c406a7a0 0x7fc9c406cc60 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9e8160420 0x7fc9e80c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9e815fcf0 0x7fc9e80c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 --2- 172.21.15.62:0/2633484588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9e80d06d0 0x7fc9e8149f20 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 -- 172.21.15.62:0/2633484588 >> 172.21.15.62:0/2633484588 conn(0x7fc9e8089f60 msgr2=0x7fc9e80d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 -- 172.21.15.62:0/2633484588 shutdown_connections 2024-03-23T11:32:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.643+0000 7fc9e7200640 1 -- 172.21.15.62:0/2633484588 wait complete. 2024-03-23T11:32:37.652 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:32:37.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.885+0000 7f7f47e00640 1 Processor -- start 2024-03-23T11:32:37.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.886+0000 7f7f47e00640 1 -- start start 2024-03-23T11:32:37.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.887+0000 7f7f47e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7f480ae880 0x7f7f480af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.887+0000 7f7f47e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f480cc860 con 0x7f7f480989d0 2024-03-23T11:32:37.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.887+0000 7f7f47e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f480cbfc0 con 0x7f7f480ce8d0 2024-03-23T11:32:37.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.887+0000 7f7f47e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f480cac10 con 0x7f7f480ae880 2024-03-23T11:32:37.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.888+0000 7f7f47400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7f480989d0 0x7f7f48098e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43970/0 (socket says 172.21.15.62:43970) 2024-03-23T11:32:37.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.888+0000 7f7f47400640 1 -- 172.21.15.62:0/608641348 learned_addr learned my addr 172.21.15.62:0/608641348 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:37.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.888+0000 7f7f46000640 1 --2- 172.21.15.62:0/608641348 >> v2:172.21.15.62:3300/0 conn(0x7f7f480ae880 0x7f7f480af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:37.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.888+0000 7f7f45600640 1 -- 172.21.15.62:0/608641348 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824329890 0 0) 0x7f7f480cc860 con 0x7f7f480989d0 2024-03-23T11:32:37.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.888+0000 7f7f46000640 1 -- 172.21.15.62:0/608641348 >> v1:172.21.15.120:6789/0 conn(0x7f7f480ce8d0 legacy=0x7f7f480cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.888+0000 7f7f46000640 1 -- 172.21.15.62:0/608641348 >> v1:172.21.15.62:6789/0 conn(0x7f7f480989d0 legacy=0x7f7f48098e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.889+0000 7f7f46000640 1 -- 172.21.15.62:0/608641348 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f480cc2a0 con 0x7f7f480ae880 2024-03-23T11:32:37.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.890+0000 7f7f46000640 1 --2- 172.21.15.62:0/608641348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f480ae880 0x7f7f480af340 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f7f340097a0 tx=0x7f7f3402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb9b1d0d3aecc50 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:37.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.890+0000 7f7f45600640 1 -- 172.21.15.62:0/608641348 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34030610 con 0x7f7f480ae880 2024-03-23T11:32:37.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.890+0000 7f7f45600640 1 -- 172.21.15.62:0/608641348 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f34030770 con 0x7f7f480ae880 2024-03-23T11:32:37.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.890+0000 7f7f45600640 1 -- 172.21.15.62:0/608641348 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34030b00 con 0x7f7f480ae880 2024-03-23T11:32:37.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.890+0000 7f7f47e00640 1 -- 172.21.15.62:0/608641348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f480ae880 msgr2=0x7f7f480af340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.890+0000 7f7f47e00640 1 --2- 172.21.15.62:0/608641348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f480ae880 0x7f7f480af340 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f7f340097a0 tx=0x7f7f3402b960 comp rx=0 tx=0).stop 2024-03-23T11:32:37.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.891+0000 7f7f47e00640 1 -- 172.21.15.62:0/608641348 shutdown_connections 2024-03-23T11:32:37.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.891+0000 7f7f47e00640 1 --2- 172.21.15.62:0/608641348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f480ae880 0x7f7f480af340 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.891+0000 7f7f47e00640 1 -- 172.21.15.62:0/608641348 >> 172.21.15.62:0/608641348 conn(0x7f7f480b1fb0 msgr2=0x7f7f480b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:37.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.891+0000 7f7f47e00640 1 -- 172.21.15.62:0/608641348 shutdown_connections 2024-03-23T11:32:37.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.891+0000 7f7f47e00640 1 -- 172.21.15.62:0/608641348 wait complete. 2024-03-23T11:32:37.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.892+0000 7f7f47e00640 1 Processor -- start 2024-03-23T11:32:37.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.893+0000 7f7f47e00640 1 -- start start 2024-03-23T11:32:37.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.893+0000 7f7f47e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f480989d0 0x7f7f481efa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.893+0000 7f7f47e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f480ae880 0x7f7f480c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.893+0000 7f7f47e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f480ce8d0 0x7f7f480c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.893+0000 7f7f47e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f480b40e0 con 0x7f7f480989d0 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.893+0000 7f7f47e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f48158540 con 0x7f7f480ae880 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.893+0000 7f7f47e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f480b3460 con 0x7f7f480ce8d0 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.894+0000 7f7f46a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f480989d0 0x7f7f481efa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.894+0000 7f7f46a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f480989d0 0x7f7f481efa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55596/0 (socket says 172.21.15.62:55596) 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.894+0000 7f7f46a00640 1 -- 172.21.15.62:0/2220262210 learned_addr learned my addr 172.21.15.62:0/2220262210 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.894+0000 7f7f46a00640 1 -- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f480ce8d0 msgr2=0x7f7f480c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.894+0000 7f7f46a00640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f480ce8d0 0x7f7f480c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.894+0000 7f7f46a00640 1 -- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f480ae880 msgr2=0x7f7f480c41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.894+0000 7f7f46000640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f480ae880 0x7f7f480c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.894+0000 7f7f46a00640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f480ae880 0x7f7f480c41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:37.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.894+0000 7f7f46a00640 1 -- 172.21.15.62:0/2220262210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f34009400 con 0x7f7f480989d0 2024-03-23T11:32:37.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.895+0000 7f7f46a00640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f480989d0 0x7f7f481efa60 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800c3c0 tx=0x7f7f3800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.895+0000 7f7f3ea00640 1 -- 172.21.15.62:0/2220262210 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f380052d0 con 0x7f7f480989d0 2024-03-23T11:32:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.895+0000 7f7f3ea00640 1 -- 172.21.15.62:0/2220262210 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f38005450 con 0x7f7f480989d0 2024-03-23T11:32:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.895+0000 7f7f47e00640 1 -- 172.21.15.62:0/2220262210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f480b3740 con 0x7f7f480989d0 2024-03-23T11:32:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.895+0000 7f7f3ea00640 1 -- 172.21.15.62:0/2220262210 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38007260 con 0x7f7f480989d0 2024-03-23T11:32:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.895+0000 7f7f47e00640 1 -- 172.21.15.62:0/2220262210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f480b3d00 con 0x7f7f480989d0 2024-03-23T11:32:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.896+0000 7f7f47e00640 1 -- 172.21.15.62:0/2220262210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f48073540 con 0x7f7f480989d0 2024-03-23T11:32:37.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.897+0000 7f7f3ea00640 1 -- 172.21.15.62:0/2220262210 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7f380073c0 con 0x7f7f480989d0 2024-03-23T11:32:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.898+0000 7f7f3ea00640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f2806a7a0 0x7f7f2806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:37.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.898+0000 7f7f3ea00640 1 -- 172.21.15.62:0/2220262210 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7f380882f0 con 0x7f7f480989d0 2024-03-23T11:32:37.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.899+0000 7f7f46000640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f2806a7a0 0x7f7f2806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:37.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.900+0000 7f7f46000640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f2806a7a0 0x7f7f2806cc60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7f34036000 tx=0x7f7f34008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:37.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:37.903+0000 7f7f3ea00640 1 -- 172.21.15.62:0/2220262210 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f38053580 con 0x7f7f480989d0 2024-03-23T11:32:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.101+0000 7f7f47e00640 1 -- 172.21.15.62:0/2220262210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7f48066d00 con 0x7f7f480989d0 2024-03-23T11:32:38.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.102+0000 7f7f3ea00640 1 -- 172.21.15.62:0/2220262210 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v122) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7f38056ba0 con 0x7f7f480989d0 2024-03-23T11:32:38.093 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:32:38.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.105+0000 7f7f47e00640 1 -- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f2806a7a0 msgr2=0x7f7f2806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:38.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.105+0000 7f7f47e00640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f2806a7a0 0x7f7f2806cc60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7f34036000 tx=0x7f7f34008030 comp rx=0 tx=0).stop 2024-03-23T11:32:38.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.105+0000 7f7f47e00640 1 -- 172.21.15.62:0/2220262210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f480989d0 msgr2=0x7f7f481efa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:38.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.105+0000 7f7f47e00640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f480989d0 0x7f7f481efa60 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800c3c0 tx=0x7f7f3800c880 comp rx=0 tx=0).stop 2024-03-23T11:32:38.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.105+0000 7f7f47e00640 1 -- 172.21.15.62:0/2220262210 shutdown_connections 2024-03-23T11:32:38.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.105+0000 7f7f47e00640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7f2806a7a0 0x7f7f2806cc60 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.105+0000 7f7f47e00640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7f480ce8d0 0x7f7f480c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.105+0000 7f7f47e00640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f480ae880 0x7f7f480c41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.105+0000 7f7f47e00640 1 --2- 172.21.15.62:0/2220262210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f480989d0 0x7f7f481efa60 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.105+0000 7f7f47e00640 1 -- 172.21.15.62:0/2220262210 >> 172.21.15.62:0/2220262210 conn(0x7f7f480b1fb0 msgr2=0x7f7f480cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:38.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.106+0000 7f7f47e00640 1 -- 172.21.15.62:0/2220262210 shutdown_connections 2024-03-23T11:32:38.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.106+0000 7f7f47e00640 1 -- 172.21.15.62:0/2220262210 wait complete. 2024-03-23T11:32:38.117 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T11:32:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.351+0000 7f6eefc00640 1 Processor -- start 2024-03-23T11:32:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.352+0000 7f6eefc00640 1 -- start start 2024-03-23T11:32:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.353+0000 7f6eefc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6ee80cafa0 0x7f6ee80cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.353+0000 7f6eefc00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee80cbc60 con 0x7f6ee806c300 2024-03-23T11:32:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.353+0000 7f6eefc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee80cbf40 con 0x7f6ee806c6e0 2024-03-23T11:32:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.353+0000 7f6eefc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee80cd410 con 0x7f6ee80cafa0 2024-03-23T11:32:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.353+0000 7f6eed200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6ee806c300 0x7f6ee80ccd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43980/0 (socket says 172.21.15.62:43980) 2024-03-23T11:32:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.353+0000 7f6eed200640 1 -- 172.21.15.62:0/2186303115 learned_addr learned my addr 172.21.15.62:0/2186303115 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:38.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.354+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881037550 0 0) 0x7f6ee80cbc60 con 0x7f6ee806c300 2024-03-23T11:32:38.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.354+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ecc003520 con 0x7f6ee806c300 2024-03-23T11:32:38.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.354+0000 7f6ee7e00640 1 --2- 172.21.15.62:0/2186303115 >> v2:172.21.15.120:3300/0 conn(0x7f6ee80cafa0 0x7f6ee80cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:38.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.354+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3624793184 0 0) 0x7f6ecc003520 con 0x7f6ee806c300 2024-03-23T11:32:38.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.354+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 >> v2:172.21.15.120:3300/0 conn(0x7f6ee80cafa0 msgr2=0x7f6ee80cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:38.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.354+0000 7f6ee7400640 1 --2- 172.21.15.62:0/2186303115 >> v2:172.21.15.120:3300/0 conn(0x7f6ee80cafa0 0x7f6ee80cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.354+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 >> v1:172.21.15.120:6789/0 conn(0x7f6ee806c6e0 legacy=0x7f6ee806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:38.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.354+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee80cc220 con 0x7f6ee806c300 2024-03-23T11:32:38.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f6ed8001670 con 0x7f6ee806c300 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 >> v1:172.21.15.62:6789/0 conn(0x7f6ee806c300 legacy=0x7f6ee80ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7400640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ecc0039c0 0x7f6ecc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7400640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 0x7f6ecc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7400640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ecc0051d0 0x7f6ecc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee80cd410 con 0x7f6ecc0047c0 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc00c120 con 0x7f6ecc0051d0 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc00c2e0 con 0x7f6ecc0039c0 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7e00640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 0x7f6ecc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6eed200640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ecc0039c0 0x7f6ecc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6eedc00640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ecc0051d0 0x7f6ecc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7e00640 1 -- 172.21.15.62:0/2186303115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ecc0039c0 msgr2=0x7f6ecc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7e00640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ecc0039c0 0x7f6ecc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7e00640 1 -- 172.21.15.62:0/2186303115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ecc0051d0 msgr2=0x7f6ecc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7e00640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ecc0051d0 0x7f6ecc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6ee7e00640 1 -- 172.21.15.62:0/2186303115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc003520 con 0x7f6ecc0047c0 2024-03-23T11:32:38.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6eed200640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ecc0039c0 0x7f6ecc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:38.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6eedc00640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ecc0051d0 0x7f6ecc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:38.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.355+0000 7f6eedc00640 1 -- 172.21.15.62:0/2186303115 reap_dead start 2024-03-23T11:32:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.356+0000 7f6ee7e00640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 0x7f6ecc004c40 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000ba30 tx=0x7f6ecc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3cee9d8db7c695b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.357+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0032640 con 0x7f6ecc0047c0 2024-03-23T11:32:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.357+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed00327a0 con 0x7f6ecc0047c0 2024-03-23T11:32:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.357+0000 7f6ee7400640 1 -- 172.21.15.62:0/2186303115 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0032b30 con 0x7f6ecc0047c0 2024-03-23T11:32:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.357+0000 7f6eefc00640 1 -- 172.21.15.62:0/2186303115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 msgr2=0x7f6ecc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.357+0000 7f6eefc00640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 0x7f6ecc004c40 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000ba30 tx=0x7f6ecc0057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.357+0000 7f6eefc00640 1 -- 172.21.15.62:0/2186303115 shutdown_connections 2024-03-23T11:32:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.357+0000 7f6eefc00640 1 --2- 172.21.15.62:0/2186303115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 0x7f6ecc004c40 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.357+0000 7f6eefc00640 1 -- 172.21.15.62:0/2186303115 >> 172.21.15.62:0/2186303115 conn(0x7f6ee80b02e0 msgr2=0x7f6ee80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.358+0000 7f6eefc00640 1 -- 172.21.15.62:0/2186303115 shutdown_connections 2024-03-23T11:32:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.358+0000 7f6eefc00640 1 -- 172.21.15.62:0/2186303115 wait complete. 2024-03-23T11:32:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.359+0000 7f6eefc00640 1 Processor -- start 2024-03-23T11:32:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.360+0000 7f6eefc00640 1 -- start start 2024-03-23T11:32:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.360+0000 7f6eefc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 0x7f6ee8157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eefc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee80b33d0 0x7f6ee815d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eefc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee815dee0 0x7f6ee80aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eefc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee81577a0 con 0x7f6ecc0047c0 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eefc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee80b2330 con 0x7f6ee815dee0 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eefc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee80b2610 con 0x7f6ee80b33d0 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eed200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 0x7f6ee8157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eed200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 0x7f6ee8157260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55622/0 (socket says 172.21.15.62:55622) 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eed200640 1 -- 172.21.15.62:0/149307161 learned_addr learned my addr 172.21.15.62:0/149307161 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eed200640 1 -- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee80b33d0 msgr2=0x7f6ee815d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eed200640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee80b33d0 0x7f6ee815d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eed200640 1 -- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee815dee0 msgr2=0x7f6ee80aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eed200640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee815dee0 0x7f6ee80aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.361+0000 7f6eed200640 1 -- 172.21.15.62:0/149307161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee80b28f0 con 0x7f6ecc0047c0 2024-03-23T11:32:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.362+0000 7f6eed200640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 0x7f6ee8157260 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f6ed800d770 tx=0x7f6ed800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.362+0000 7f6ee4c00640 1 -- 172.21.15.62:0/149307161 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed80074e0 con 0x7f6ecc0047c0 2024-03-23T11:32:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.362+0000 7f6ee4c00640 1 -- 172.21.15.62:0/149307161 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed8007640 con 0x7f6ecc0047c0 2024-03-23T11:32:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.362+0000 7f6ee4c00640 1 -- 172.21.15.62:0/149307161 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed8004a20 con 0x7f6ecc0047c0 2024-03-23T11:32:38.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.362+0000 7f6eefc00640 1 -- 172.21.15.62:0/149307161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee80b2bd0 con 0x7f6ecc0047c0 2024-03-23T11:32:38.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.362+0000 7f6eefc00640 1 -- 172.21.15.62:0/149307161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee80b2020 con 0x7f6ecc0047c0 2024-03-23T11:32:38.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.363+0000 7f6eefc00640 1 -- 172.21.15.62:0/149307161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee813e760 con 0x7f6ecc0047c0 2024-03-23T11:32:38.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.364+0000 7f6ee4c00640 1 -- 172.21.15.62:0/149307161 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f6ed8004b80 con 0x7f6ecc0047c0 2024-03-23T11:32:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.365+0000 7f6ee4c00640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6ec406a7a0 0x7f6ec406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:38.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.365+0000 7f6ee4c00640 1 -- 172.21.15.62:0/149307161 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f6ed8088720 con 0x7f6ecc0047c0 2024-03-23T11:32:38.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.366+0000 7f6ee7e00640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6ec406a7a0 0x7f6ec406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:38.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.367+0000 7f6ee7e00640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6ec406a7a0 0x7f6ec406cc60 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000ba00 tx=0x7f6ed0032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:38.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.370+0000 7f6ee4c00640 1 -- 172.21.15.62:0/149307161 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ed8053ae0 con 0x7f6ecc0047c0 2024-03-23T11:32:38.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.586+0000 7f6eefc00640 1 -- 172.21.15.62:0/149307161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6ee8066c90 con 0x7f6ecc0047c0 2024-03-23T11:32:38.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.587+0000 7f6ee4c00640 1 -- 172.21.15.62:0/149307161 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v122) v1 ==== 80+0+246 (secure 0 0 0) 0x7f6ed8019110 con 0x7f6ecc0047c0 2024-03-23T11:32:38.577 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6835/4129977589 2024-03-24T11:32:23.551904+0000 2024-03-23T11:32:38.577 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6834/4129977589 2024-03-24T11:32:23.551904+0000 2024-03-23T11:32:38.578 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6837/2585024035 2024-03-24T11:32:04.365655+0000 2024-03-23T11:32:38.578 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.120:6835/2585024035 2024-03-24T11:32:04.365655+0000 2024-03-23T11:32:38.578 INFO:teuthology.orchestra.run.smithi062.stderr:listed 4 entries 2024-03-23T11:32:38.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 -- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6ec406a7a0 msgr2=0x7f6ec406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:38.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6ec406a7a0 0x7f6ec406cc60 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000ba00 tx=0x7f6ed0032000 comp rx=0 tx=0).stop 2024-03-23T11:32:38.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 -- 172.21.15.62:0/149307161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 msgr2=0x7f6ee8157260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:38.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 0x7f6ee8157260 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f6ed800d770 tx=0x7f6ed800dc30 comp rx=0 tx=0).stop 2024-03-23T11:32:38.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 -- 172.21.15.62:0/149307161 shutdown_connections 2024-03-23T11:32:38.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f6ec406a7a0 0x7f6ec406cc60 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee815dee0 0x7f6ee80aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee80b33d0 0x7f6ee815d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 --2- 172.21.15.62:0/149307161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ecc0047c0 0x7f6ee8157260 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:38.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 -- 172.21.15.62:0/149307161 >> 172.21.15.62:0/149307161 conn(0x7f6ee80b02e0 msgr2=0x7f6ee806d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:38.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 -- 172.21.15.62:0/149307161 shutdown_connections 2024-03-23T11:32:38.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:38.590+0000 7f6eefc00640 1 -- 172.21.15.62:0/149307161 wait complete. 2024-03-23T11:32:38.599 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-03-23T11:32:38.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:38.599 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T11:32:38.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[82214]: fuse finished with error 0 and tester_r 0 2024-03-23T11:32:38.695 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:32:38.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:38.696 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T11:32:38.873 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:32:38.873 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:32:38.929 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:32:38.929 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:32:38.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:38.930 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T11:32:38.981 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-23T11:32:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:32:38.983 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:32:38.983 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:32:39.042 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:32:39.042 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:32:39.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:39.042 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T11:32:39.095 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-23T11:32:39.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:32:39.096 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:32:39.096 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:32:39.153 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:32:39.153 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:32:39.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:39.153 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T11:32:39.204 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-23T11:32:39.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:32:39.206 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:32:39.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.443+0000 7f2fea200640 1 Processor -- start 2024-03-23T11:32:39.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.443+0000 7f2fea200640 1 -- start start 2024-03-23T11:32:39.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.445+0000 7f2fea200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fe4143dd0 0x7f2fe415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fea200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2fe4070090 0x7f2fe40704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fea200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2fe40b2820 0x7f2fe40b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fea200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe4070a30 con 0x7f2fe40b2820 2024-03-23T11:32:39.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fea200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe40cd140 con 0x7f2fe4070090 2024-03-23T11:32:39.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fea200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40b3180 con 0x7f2fe4143dd0 2024-03-23T11:32:39.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fe9800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2fe40b2820 0x7f2fe40b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fe9800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2fe40b2820 0x7f2fe40b2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55636/0 (socket says 172.21.15.62:55636) 2024-03-23T11:32:39.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fe9800640 1 -- 172.21.15.62:0/1960048072 learned_addr learned my addr 172.21.15.62:0/1960048072 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:39.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fe8e00640 1 --2- 172.21.15.62:0/1960048072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fe4143dd0 0x7f2fe415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fdbe00640 1 --2- 172.21.15.62:0/1960048072 >> v2:172.21.15.120:3300/0 conn(0x7f2fe4070090 0x7f2fe40704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fe9800640 1 -- 172.21.15.62:0/1960048072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fe4143dd0 msgr2=0x7f2fe415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:39.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fe9800640 1 --2- 172.21.15.62:0/1960048072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fe4143dd0 0x7f2fe415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fe9800640 1 -- 172.21.15.62:0/1960048072 >> v2:172.21.15.120:3300/0 conn(0x7f2fe4070090 msgr2=0x7f2fe40704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:39.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fe9800640 1 --2- 172.21.15.62:0/1960048072 >> v2:172.21.15.120:3300/0 conn(0x7f2fe4070090 0x7f2fe40704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.446+0000 7f2fe9800640 1 -- 172.21.15.62:0/1960048072 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe40cd3e0 con 0x7f2fe40b2820 2024-03-23T11:32:39.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.447+0000 7f2fe8e00640 1 --2- 172.21.15.62:0/1960048072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fe4143dd0 0x7f2fe415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:39.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.447+0000 7f2fdbe00640 1 --2- 172.21.15.62:0/1960048072 >> v2:172.21.15.120:3300/0 conn(0x7f2fe4070090 0x7f2fe40704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.448+0000 7f2fe9800640 1 --2- 172.21.15.62:0/1960048072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe40b2820 0x7f2fe40b2c40 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f2fe406e3b0 tx=0x7f2fdc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd2d839599bb22e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.448+0000 7f2fe3e00640 1 -- 172.21.15.62:0/1960048072 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc02ca80 con 0x7f2fe40b2820 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.448+0000 7f2fe3e00640 1 -- 172.21.15.62:0/1960048072 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc02cbe0 con 0x7f2fe40b2820 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.448+0000 7f2fe3e00640 1 -- 172.21.15.62:0/1960048072 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc0343f0 con 0x7f2fe40b2820 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.449+0000 7f2fea200640 1 -- 172.21.15.62:0/1960048072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe40b2820 msgr2=0x7f2fe40b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.449+0000 7f2fea200640 1 --2- 172.21.15.62:0/1960048072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe40b2820 0x7f2fe40b2c40 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f2fe406e3b0 tx=0x7f2fdc02bee0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.449+0000 7f2fea200640 1 -- 172.21.15.62:0/1960048072 shutdown_connections 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.449+0000 7f2fea200640 1 --2- 172.21.15.62:0/1960048072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe40b2820 0x7f2fe40b2c40 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.449+0000 7f2fea200640 1 --2- 172.21.15.62:0/1960048072 >> v2:172.21.15.120:3300/0 conn(0x7f2fe4070090 0x7f2fe40704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.449+0000 7f2fea200640 1 --2- 172.21.15.62:0/1960048072 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fe4143dd0 0x7f2fe415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.449+0000 7f2fea200640 1 -- 172.21.15.62:0/1960048072 >> 172.21.15.62:0/1960048072 conn(0x7f2fe4066050 msgr2=0x7f2fe4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:39.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.449+0000 7f2fea200640 1 -- 172.21.15.62:0/1960048072 shutdown_connections 2024-03-23T11:32:39.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.449+0000 7f2fea200640 1 -- 172.21.15.62:0/1960048072 wait complete. 2024-03-23T11:32:39.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.451+0000 7f2fea200640 1 Processor -- start 2024-03-23T11:32:39.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.451+0000 7f2fea200640 1 -- start start 2024-03-23T11:32:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.452+0000 7f2fea200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4070090 0x7f2fe40aa550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.452+0000 7f2fea200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fe40b2820 0x7f2fe40acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.452+0000 7f2fea200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fe4143dd0 0x7f2fe4135dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.452+0000 7f2fea200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40ab840 con 0x7f2fe4070090 2024-03-23T11:32:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.452+0000 7f2fea200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40ad060 con 0x7f2fe4143dd0 2024-03-23T11:32:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.452+0000 7f2fea200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40aac30 con 0x7f2fe40b2820 2024-03-23T11:32:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.452+0000 7f2fe8e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4070090 0x7f2fe40aa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.452+0000 7f2fe8e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4070090 0x7f2fe40aa550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55648/0 (socket says 172.21.15.62:55648) 2024-03-23T11:32:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.452+0000 7f2fe8e00640 1 -- 172.21.15.62:0/297307293 learned_addr learned my addr 172.21.15.62:0/297307293 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.453+0000 7f2fe8e00640 1 -- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fe40b2820 msgr2=0x7f2fe40acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.453+0000 7f2fe8e00640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fe40b2820 0x7f2fe40acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.453+0000 7f2fe9800640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fe4143dd0 0x7f2fe4135dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.453+0000 7f2fe8e00640 1 -- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fe4143dd0 msgr2=0x7f2fe4135dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.453+0000 7f2fe8e00640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fe4143dd0 0x7f2fe4135dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.453+0000 7f2fe8e00640 1 -- 172.21.15.62:0/297307293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc009b30 con 0x7f2fe4070090 2024-03-23T11:32:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.453+0000 7f2fe8e00640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4070090 0x7f2fe40aa550 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400c7f0 tx=0x7f2fd400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.453+0000 7f2fe1600640 1 -- 172.21.15.62:0/297307293 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4007a30 con 0x7f2fe4070090 2024-03-23T11:32:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.453+0000 7f2fe1600640 1 -- 172.21.15.62:0/297307293 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd4007b90 con 0x7f2fe4070090 2024-03-23T11:32:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.453+0000 7f2fe1600640 1 -- 172.21.15.62:0/297307293 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd400ce40 con 0x7f2fe4070090 2024-03-23T11:32:39.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.453+0000 7f2fea200640 1 -- 172.21.15.62:0/297307293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe40aaf10 con 0x7f2fe4070090 2024-03-23T11:32:39.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.454+0000 7f2fea200640 1 -- 172.21.15.62:0/297307293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe40ab4d0 con 0x7f2fe4070090 2024-03-23T11:32:39.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.455+0000 7f2fea200640 1 -- 172.21.15.62:0/297307293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe40ccd20 con 0x7f2fe4070090 2024-03-23T11:32:39.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.456+0000 7f2fe1600640 1 -- 172.21.15.62:0/297307293 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2fd40113f0 con 0x7f2fe4070090 2024-03-23T11:32:39.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.457+0000 7f2fe1600640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fc006a7a0 0x7f2fc006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.457+0000 7f2fe1600640 1 -- 172.21.15.62:0/297307293 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f2fd4088fc0 con 0x7f2fe4070090 2024-03-23T11:32:39.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.457+0000 7f2fdbe00640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fc006a7a0 0x7f2fc006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.458+0000 7f2fdbe00640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fc006a7a0 0x7f2fc006cc60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc006fd0 tx=0x7f2fcc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:39.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.461+0000 7f2fe1600640 1 -- 172.21.15.62:0/297307293 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2fd40542e0 con 0x7f2fe4070090 2024-03-23T11:32:39.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.702+0000 7f2fea200640 1 -- 172.21.15.62:0/297307293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fe4066bb0 con 0x7f2fe4070090 2024-03-23T11:32:39.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.703+0000 7f2fe1600640 1 -- 172.21.15.62:0/297307293 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f2fd401f090 con 0x7f2fe4070090 2024-03-23T11:32:39.694 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:39.694 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":107,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:32:28.458742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9507":{"gid":9507,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6835/2500981044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2500981044},{"type":"v1","addr":"172.21.15.62:6835","nonce":2500981044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:32:39.695 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 107 2024-03-23T11:32:39.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.706+0000 7f2fea200640 1 -- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fc006a7a0 msgr2=0x7f2fc006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:39.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.706+0000 7f2fea200640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fc006a7a0 0x7f2fc006cc60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc006fd0 tx=0x7f2fcc008040 comp rx=0 tx=0).stop 2024-03-23T11:32:39.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.706+0000 7f2fea200640 1 -- 172.21.15.62:0/297307293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4070090 msgr2=0x7f2fe40aa550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:39.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.706+0000 7f2fea200640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4070090 0x7f2fe40aa550 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400c7f0 tx=0x7f2fd400cbf0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.706+0000 7f2fea200640 1 -- 172.21.15.62:0/297307293 shutdown_connections 2024-03-23T11:32:39.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.707+0000 7f2fea200640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2fc006a7a0 0x7f2fc006cc60 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.707+0000 7f2fea200640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fe4143dd0 0x7f2fe4135dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.707+0000 7f2fea200640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fe40b2820 0x7f2fe40acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.707+0000 7f2fea200640 1 --2- 172.21.15.62:0/297307293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4070090 0x7f2fe40aa550 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.707+0000 7f2fea200640 1 -- 172.21.15.62:0/297307293 >> 172.21.15.62:0/297307293 conn(0x7f2fe4066050 msgr2=0x7f2fe4143090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:39.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.707+0000 7f2fea200640 1 -- 172.21.15.62:0/297307293 shutdown_connections 2024-03-23T11:32:39.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.707+0000 7f2fea200640 1 -- 172.21.15.62:0/297307293 wait complete. 2024-03-23T11:32:39.718 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:32:39.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.941+0000 7f2d09600640 1 Processor -- start 2024-03-23T11:32:39.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.941+0000 7f2d09600640 1 -- start start 2024-03-23T11:32:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.942+0000 7f2d09600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2d0415fcf0 0x7f2d040d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.943+0000 7f2d09600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d040b6dc0 0x7f2d041608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.943+0000 7f2d09600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d040d3540 con 0x7f2d04160420 2024-03-23T11:32:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.943+0000 7f2d09600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d04160df0 con 0x7f2d0415fcf0 2024-03-23T11:32:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.943+0000 7f2d09600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040ce510 con 0x7f2d040b6dc0 2024-03-23T11:32:39.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.943+0000 7f2d03e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2d0415fcf0 0x7f2d040d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.943+0000 7f2d08c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d040b6dc0 0x7f2d041608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.943+0000 7f2d08c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d040b6dc0 0x7f2d041608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:60424/0 (socket says 172.21.15.62:60424) 2024-03-23T11:32:39.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.943+0000 7f2d03e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2d0415fcf0 0x7f2d040d3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:56970/0 (socket says 172.21.15.62:56970) 2024-03-23T11:32:39.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.944+0000 7f2d08c00640 1 -- 172.21.15.62:0/4272823066 learned_addr learned my addr 172.21.15.62:0/4272823066 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:39.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.944+0000 7f2d03e00640 1 -- 172.21.15.62:0/4272823066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d040b6dc0 msgr2=0x7f2d041608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:39.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.944+0000 7f2d03e00640 1 --2- 172.21.15.62:0/4272823066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d040b6dc0 0x7f2d041608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.944+0000 7f2d03e00640 1 -- 172.21.15.62:0/4272823066 >> v1:172.21.15.120:6789/0 conn(0x7f2d04160420 legacy=0x7f2d040d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:39.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.944+0000 7f2d03e00640 1 -- 172.21.15.62:0/4272823066 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d040cdc70 con 0x7f2d0415fcf0 2024-03-23T11:32:39.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.946+0000 7f2d03e00640 1 --2- 172.21.15.62:0/4272823066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d0415fcf0 0x7f2d040d3000 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0097a0 tx=0x7f2cec02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ad7bd78c855731 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:39.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.946+0000 7f2d02a00640 1 -- 172.21.15.62:0/4272823066 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec0316f0 con 0x7f2d0415fcf0 2024-03-23T11:32:39.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.946+0000 7f2d02a00640 1 -- 172.21.15.62:0/4272823066 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec031850 con 0x7f2d0415fcf0 2024-03-23T11:32:39.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.946+0000 7f2d02a00640 1 -- 172.21.15.62:0/4272823066 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec031be0 con 0x7f2d0415fcf0 2024-03-23T11:32:39.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.947+0000 7f2d09600640 1 -- 172.21.15.62:0/4272823066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d0415fcf0 msgr2=0x7f2d040d3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:39.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.947+0000 7f2d09600640 1 --2- 172.21.15.62:0/4272823066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d0415fcf0 0x7f2d040d3000 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0097a0 tx=0x7f2cec02b790 comp rx=0 tx=0).stop 2024-03-23T11:32:39.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.947+0000 7f2d09600640 1 -- 172.21.15.62:0/4272823066 shutdown_connections 2024-03-23T11:32:39.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.947+0000 7f2d09600640 1 --2- 172.21.15.62:0/4272823066 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d040b6dc0 0x7f2d041608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.947+0000 7f2d09600640 1 --2- 172.21.15.62:0/4272823066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d0415fcf0 0x7f2d040d3000 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.947+0000 7f2d09600640 1 -- 172.21.15.62:0/4272823066 >> 172.21.15.62:0/4272823066 conn(0x7f2d04089f60 msgr2=0x7f2d040d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:39.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.947+0000 7f2d09600640 1 -- 172.21.15.62:0/4272823066 shutdown_connections 2024-03-23T11:32:39.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.947+0000 7f2d09600640 1 -- 172.21.15.62:0/4272823066 wait complete. 2024-03-23T11:32:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.950+0000 7f2d09600640 1 Processor -- start 2024-03-23T11:32:39.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.950+0000 7f2d09600640 1 -- start start 2024-03-23T11:32:39.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.951+0000 7f2d09600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d040b6dc0 0x7f2d04157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.951+0000 7f2d09600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d0415fcf0 0x7f2d040cf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.951+0000 7f2d09600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d04160420 0x7f2d040cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.951+0000 7f2d09600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d041f28a0 con 0x7f2d0415fcf0 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.951+0000 7f2d09600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d041e7820 con 0x7f2d040b6dc0 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.951+0000 7f2d09600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d041e7b00 con 0x7f2d04160420 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.952+0000 7f2d03400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d0415fcf0 0x7f2d040cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.952+0000 7f2d03400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d0415fcf0 0x7f2d040cf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55664/0 (socket says 172.21.15.62:55664) 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.952+0000 7f2d03400640 1 -- 172.21.15.62:0/2529869893 learned_addr learned my addr 172.21.15.62:0/2529869893 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.952+0000 7f2d03e00640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d040b6dc0 0x7f2d04157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.952+0000 7f2d08c00640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d04160420 0x7f2d040cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.952+0000 7f2d03400640 1 -- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d04160420 msgr2=0x7f2d040cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.952+0000 7f2d03400640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d04160420 0x7f2d040cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.952+0000 7f2d03400640 1 -- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d040b6dc0 msgr2=0x7f2d04157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.952+0000 7f2d03400640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d040b6dc0 0x7f2d04157ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:39.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.952+0000 7f2d03400640 1 -- 172.21.15.62:0/2529869893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec009400 con 0x7f2d0415fcf0 2024-03-23T11:32:39.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.953+0000 7f2d03400640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d0415fcf0 0x7f2d040cf600 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400c450 tx=0x7f2cf400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:39.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.953+0000 7f2cfbe00640 1 -- 172.21.15.62:0/2529869893 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40052d0 con 0x7f2d0415fcf0 2024-03-23T11:32:39.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.953+0000 7f2cfbe00640 1 -- 172.21.15.62:0/2529869893 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf4005430 con 0x7f2d0415fcf0 2024-03-23T11:32:39.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.953+0000 7f2cfbe00640 1 -- 172.21.15.62:0/2529869893 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4012870 con 0x7f2d0415fcf0 2024-03-23T11:32:39.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.953+0000 7f2d09600640 1 -- 172.21.15.62:0/2529869893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d041e7de0 con 0x7f2d0415fcf0 2024-03-23T11:32:39.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.953+0000 7f2d09600640 1 -- 172.21.15.62:0/2529869893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d04158430 con 0x7f2d0415fcf0 2024-03-23T11:32:39.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.954+0000 7f2d09600640 1 -- 172.21.15.62:0/2529869893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d041f29e0 con 0x7f2d0415fcf0 2024-03-23T11:32:39.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.955+0000 7f2cfbe00640 1 -- 172.21.15.62:0/2529869893 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2cf40129d0 con 0x7f2d0415fcf0 2024-03-23T11:32:39.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.956+0000 7f2cfbe00640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ce006a7a0 0x7f2ce006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:39.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.956+0000 7f2cfbe00640 1 -- 172.21.15.62:0/2529869893 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f2cf40889b0 con 0x7f2d0415fcf0 2024-03-23T11:32:39.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.957+0000 7f2d03e00640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ce006a7a0 0x7f2ce006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:39.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.958+0000 7f2d03e00640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ce006a7a0 0x7f2ce006cc60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2cec036000 tx=0x7f2cec008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:39.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:39.961+0000 7f2cfbe00640 1 -- 172.21.15.62:0/2529869893 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d041f29e0 con 0x7f2d0415fcf0 2024-03-23T11:32:40.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.163+0000 7f2d09600640 1 -- 172.21.15.62:0/2529869893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d041f29e0 con 0x7f2d0415fcf0 2024-03-23T11:32:40.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.165+0000 7f2cfbe00640 1 -- 172.21.15.62:0/2529869893 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+16413 (secure 0 0 0) 0x7f2d041f29e0 con 0x7f2d0415fcf0 2024-03-23T11:32:40.158 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:40.158 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":122,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:32:23.551951+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6835/4129977589":"2024-03-24T11:32:23.551904+0000","172.21.15.62:6834/4129977589":"2024-03-24T11:32:23.551904+0000","172.21.15.120:6837/2585024035":"2024-03-24T11:32:04.365655+0000","172.21.15.120:6835/2585024035":"2024-03-24T11:32:04.365655+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:32:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 -- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ce006a7a0 msgr2=0x7f2ce006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ce006a7a0 0x7f2ce006cc60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2cec036000 tx=0x7f2cec008030 comp rx=0 tx=0).stop 2024-03-23T11:32:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 -- 172.21.15.62:0/2529869893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d0415fcf0 msgr2=0x7f2d040cf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d0415fcf0 0x7f2d040cf600 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400c450 tx=0x7f2cf400c910 comp rx=0 tx=0).stop 2024-03-23T11:32:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 -- 172.21.15.62:0/2529869893 shutdown_connections 2024-03-23T11:32:40.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2ce006a7a0 0x7f2ce006cc60 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d0415fcf0 0x7f2d040cf600 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d04160420 0x7f2d040cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 --2- 172.21.15.62:0/2529869893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d040b6dc0 0x7f2d04157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 -- 172.21.15.62:0/2529869893 >> 172.21.15.62:0/2529869893 conn(0x7f2d04089f60 msgr2=0x7f2d0406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:40.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 -- 172.21.15.62:0/2529869893 shutdown_connections 2024-03-23T11:32:40.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.168+0000 7f2d09600640 1 -- 172.21.15.62:0/2529869893 wait complete. 2024-03-23T11:32:40.177 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T11:18:49.304156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T11:31:22.127079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T11:31:23.062387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T11:32:40.177 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T11:32:40.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.413+0000 7eff8d600640 1 Processor -- start 2024-03-23T11:32:40.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.413+0000 7eff8d600640 1 -- start start 2024-03-23T11:32:40.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.414+0000 7eff8d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff88160420 0x7eff88160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.414+0000 7eff8d600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff880b5db0 0x7eff880b6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.414+0000 7eff8d600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff88160dc0 con 0x7eff8815fcf0 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.414+0000 7eff8d600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff880ce510 con 0x7eff880b5db0 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.414+0000 7eff8d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff880cdc70 con 0x7eff88160420 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.414+0000 7eff8cc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff880b5db0 0x7eff880b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.415+0000 7eff8cc00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff880b5db0 0x7eff880b6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55674/0 (socket says 172.21.15.62:55674) 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.415+0000 7eff8cc00640 1 -- 172.21.15.62:0/2216955196 learned_addr learned my addr 172.21.15.62:0/2216955196 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.415+0000 7eff7f400640 1 --2- 172.21.15.62:0/2216955196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff88160420 0x7eff88160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.415+0000 7eff8cc00640 1 -- 172.21.15.62:0/2216955196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff88160420 msgr2=0x7eff88160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.415+0000 7eff8cc00640 1 --2- 172.21.15.62:0/2216955196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff88160420 0x7eff88160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.415+0000 7eff8cc00640 1 -- 172.21.15.62:0/2216955196 >> v1:172.21.15.120:6789/0 conn(0x7eff8815fcf0 legacy=0x7eff880d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.415+0000 7eff8cc00640 1 -- 172.21.15.62:0/2216955196 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff880cdf50 con 0x7eff880b5db0 2024-03-23T11:32:40.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.415+0000 7eff7f400640 1 --2- 172.21.15.62:0/2216955196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff88160420 0x7eff88160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.416+0000 7eff8cc00640 1 --2- 172.21.15.62:0/2216955196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff880b5db0 0x7eff880b6220 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7eff80009e50 tx=0x7eff8002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff77b6341ade74b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.417+0000 7eff87400640 1 -- 172.21.15.62:0/2216955196 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8002c9f0 con 0x7eff880b5db0 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.417+0000 7eff87400640 1 -- 172.21.15.62:0/2216955196 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8002cb50 con 0x7eff880b5db0 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.417+0000 7eff87400640 1 -- 172.21.15.62:0/2216955196 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff800343f0 con 0x7eff880b5db0 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.417+0000 7eff8d600640 1 -- 172.21.15.62:0/2216955196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff880b5db0 msgr2=0x7eff880b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.417+0000 7eff8d600640 1 --2- 172.21.15.62:0/2216955196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff880b5db0 0x7eff880b6220 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7eff80009e50 tx=0x7eff8002be50 comp rx=0 tx=0).stop 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.417+0000 7eff8d600640 1 -- 172.21.15.62:0/2216955196 shutdown_connections 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.417+0000 7eff8d600640 1 --2- 172.21.15.62:0/2216955196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff880b5db0 0x7eff880b6220 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.417+0000 7eff8d600640 1 --2- 172.21.15.62:0/2216955196 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff88160420 0x7eff88160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.417+0000 7eff8d600640 1 -- 172.21.15.62:0/2216955196 >> 172.21.15.62:0/2216955196 conn(0x7eff88089f60 msgr2=0x7eff880d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.417+0000 7eff8d600640 1 -- 172.21.15.62:0/2216955196 shutdown_connections 2024-03-23T11:32:40.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.417+0000 7eff8d600640 1 -- 172.21.15.62:0/2216955196 wait complete. 2024-03-23T11:32:40.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.419+0000 7eff8d600640 1 Processor -- start 2024-03-23T11:32:40.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.419+0000 7eff8d600640 1 -- start start 2024-03-23T11:32:40.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.420+0000 7eff8d600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff880b5db0 0x7eff8809c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.420+0000 7eff8d600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff8815fcf0 0x7eff8814a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.420+0000 7eff8d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff88160420 0x7eff880cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.420+0000 7eff8d600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff880c9a60 con 0x7eff8815fcf0 2024-03-23T11:32:40.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.420+0000 7eff8d600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff880c9d40 con 0x7eff880b5db0 2024-03-23T11:32:40.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.420+0000 7eff8d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff880ca020 con 0x7eff88160420 2024-03-23T11:32:40.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.420+0000 7eff7f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff8815fcf0 0x7eff8814a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:40.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.421+0000 7eff7f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff8815fcf0 0x7eff8814a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55678/0 (socket says 172.21.15.62:55678) 2024-03-23T11:32:40.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.421+0000 7eff7f400640 1 -- 172.21.15.62:0/1308847942 learned_addr learned my addr 172.21.15.62:0/1308847942 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:40.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.421+0000 7eff7f400640 1 -- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff88160420 msgr2=0x7eff880cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:40.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.421+0000 7eff7f400640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff88160420 0x7eff880cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.421+0000 7eff87e00640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff880b5db0 0x7eff8809c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:40.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.421+0000 7eff7f400640 1 -- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff880b5db0 msgr2=0x7eff8809c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:40.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.421+0000 7eff7f400640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff880b5db0 0x7eff8809c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.421+0000 7eff7f400640 1 -- 172.21.15.62:0/1308847942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff80009b30 con 0x7eff8815fcf0 2024-03-23T11:32:40.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.421+0000 7eff7f400640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff8815fcf0 0x7eff8814a7f0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7eff7000c7f0 tx=0x7eff7000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:40.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.422+0000 7eff84c00640 1 -- 172.21.15.62:0/1308847942 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff70007820 con 0x7eff8815fcf0 2024-03-23T11:32:40.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.422+0000 7eff84c00640 1 -- 172.21.15.62:0/1308847942 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff70007980 con 0x7eff8815fcf0 2024-03-23T11:32:40.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.422+0000 7eff84c00640 1 -- 172.21.15.62:0/1308847942 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff70007cf0 con 0x7eff8815fcf0 2024-03-23T11:32:40.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.422+0000 7eff8d600640 1 -- 172.21.15.62:0/1308847942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff880ca300 con 0x7eff8815fcf0 2024-03-23T11:32:40.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.422+0000 7eff8d600640 1 -- 172.21.15.62:0/1308847942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8809c9a0 con 0x7eff8815fcf0 2024-03-23T11:32:40.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.423+0000 7eff8d600640 1 -- 172.21.15.62:0/1308847942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff88066bb0 con 0x7eff8815fcf0 2024-03-23T11:32:40.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.424+0000 7eff84c00640 1 -- 172.21.15.62:0/1308847942 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7eff700113f0 con 0x7eff8815fcf0 2024-03-23T11:32:40.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.425+0000 7eff84c00640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff6406a7a0 0x7eff6406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.425+0000 7eff84c00640 1 -- 172.21.15.62:0/1308847942 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7eff70089200 con 0x7eff8815fcf0 2024-03-23T11:32:40.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.425+0000 7eff87e00640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff6406a7a0 0x7eff6406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:40.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.426+0000 7eff87e00640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff6406a7a0 0x7eff6406cc60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7eff881f3470 tx=0x7eff78009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:40.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.429+0000 7eff84c00640 1 -- 172.21.15.62:0/1308847942 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff70054520 con 0x7eff8815fcf0 2024-03-23T11:32:40.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.632+0000 7eff8d600640 1 -- 172.21.15.62:0/1308847942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7eff880cea40 con 0x7eff8815fcf0 2024-03-23T11:32:40.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.633+0000 7eff84c00640 1 -- 172.21.15.62:0/1308847942 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7eff70057b40 con 0x7eff8815fcf0 2024-03-23T11:32:40.623 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-03-23T11:32:40.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.635+0000 7eff8d600640 1 -- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff6406a7a0 msgr2=0x7eff6406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:40.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.635+0000 7eff8d600640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff6406a7a0 0x7eff6406cc60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7eff881f3470 tx=0x7eff78009040 comp rx=0 tx=0).stop 2024-03-23T11:32:40.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.635+0000 7eff8d600640 1 -- 172.21.15.62:0/1308847942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff8815fcf0 msgr2=0x7eff8814a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:40.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.635+0000 7eff8d600640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff8815fcf0 0x7eff8814a7f0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7eff7000c7f0 tx=0x7eff7000ccb0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.636+0000 7eff8d600640 1 -- 172.21.15.62:0/1308847942 shutdown_connections 2024-03-23T11:32:40.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.636+0000 7eff8d600640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7eff6406a7a0 0x7eff6406cc60 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.636+0000 7eff8d600640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff8815fcf0 0x7eff8814a7f0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.636+0000 7eff8d600640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff88160420 0x7eff880cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.636+0000 7eff8d600640 1 --2- 172.21.15.62:0/1308847942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff880b5db0 0x7eff8809c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.636+0000 7eff8d600640 1 -- 172.21.15.62:0/1308847942 >> 172.21.15.62:0/1308847942 conn(0x7eff88089f60 msgr2=0x7eff8815f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:40.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.636+0000 7eff8d600640 1 -- 172.21.15.62:0/1308847942 shutdown_connections 2024-03-23T11:32:40.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.636+0000 7eff8d600640 1 -- 172.21.15.62:0/1308847942 wait complete. 2024-03-23T11:32:40.647 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T11:32:40.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.881+0000 7fa1fac00640 1 Processor -- start 2024-03-23T11:32:40.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.881+0000 7fa1fac00640 1 -- start start 2024-03-23T11:32:40.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.883+0000 7fa1fac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1f4143dd0 0x7fa1f415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.883+0000 7fa1fac00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa1f4070090 0x7fa1f40704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.883+0000 7fa1fac00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f4070a30 con 0x7fa1f40b2820 2024-03-23T11:32:40.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.883+0000 7fa1fac00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f40cd140 con 0x7fa1f4070090 2024-03-23T11:32:40.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.883+0000 7fa1fac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f40cd3e0 con 0x7fa1f4143dd0 2024-03-23T11:32:40.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.883+0000 7fa1f8e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa1f4070090 0x7fa1f40704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:40.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.883+0000 7fa1fa200640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa1f40b2820 0x7fa1f40b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43990/0 (socket says 172.21.15.62:43990) 2024-03-23T11:32:40.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.883+0000 7fa1f8e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa1f4070090 0x7fa1f40704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55680/0 (socket says 172.21.15.62:55680) 2024-03-23T11:32:40.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.883+0000 7fa1fa200640 1 -- 172.21.15.62:0/1726896001 learned_addr learned my addr 172.21.15.62:0/1726896001 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:40.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.884+0000 7fa1f9800640 1 --2- 172.21.15.62:0/1726896001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1f4143dd0 0x7fa1f415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:40.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.884+0000 7fa1f8e00640 1 -- 172.21.15.62:0/1726896001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1f4143dd0 msgr2=0x7fa1f415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:40.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.884+0000 7fa1f8e00640 1 --2- 172.21.15.62:0/1726896001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1f4143dd0 0x7fa1f415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.884+0000 7fa1f8e00640 1 -- 172.21.15.62:0/1726896001 >> v1:172.21.15.62:6789/0 conn(0x7fa1f40b2820 legacy=0x7fa1f40b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:40.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.884+0000 7fa1f8e00640 1 -- 172.21.15.62:0/1726896001 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f40cd6c0 con 0x7fa1f4070090 2024-03-23T11:32:40.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.885+0000 7fa1f8e00640 1 --2- 172.21.15.62:0/1726896001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4070090 0x7fa1f40704f0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00097a0 tx=0x7fa1e002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0875f702d2f2a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:40.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.885+0000 7fa1f3e00640 1 -- 172.21.15.62:0/1726896001 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e002d040 con 0x7fa1f4070090 2024-03-23T11:32:40.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.885+0000 7fa1f3e00640 1 -- 172.21.15.62:0/1726896001 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e0031810 con 0x7fa1f4070090 2024-03-23T11:32:40.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.885+0000 7fa1f3e00640 1 -- 172.21.15.62:0/1726896001 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0031ba0 con 0x7fa1f4070090 2024-03-23T11:32:40.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.886+0000 7fa1fac00640 1 -- 172.21.15.62:0/1726896001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4070090 msgr2=0x7fa1f40704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:40.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.886+0000 7fa1fac00640 1 --2- 172.21.15.62:0/1726896001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4070090 0x7fa1f40704f0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00097a0 tx=0x7fa1e002b8b0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.886+0000 7fa1fac00640 1 -- 172.21.15.62:0/1726896001 shutdown_connections 2024-03-23T11:32:40.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.886+0000 7fa1fac00640 1 --2- 172.21.15.62:0/1726896001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4070090 0x7fa1f40704f0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.886+0000 7fa1fac00640 1 --2- 172.21.15.62:0/1726896001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1f4143dd0 0x7fa1f415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.886+0000 7fa1fac00640 1 -- 172.21.15.62:0/1726896001 >> 172.21.15.62:0/1726896001 conn(0x7fa1f4066050 msgr2=0x7fa1f4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:40.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.886+0000 7fa1fac00640 1 -- 172.21.15.62:0/1726896001 shutdown_connections 2024-03-23T11:32:40.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.886+0000 7fa1fac00640 1 -- 172.21.15.62:0/1726896001 wait complete. 2024-03-23T11:32:40.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.888+0000 7fa1fac00640 1 Processor -- start 2024-03-23T11:32:40.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.888+0000 7fa1fac00640 1 -- start start 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.889+0000 7fa1fac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1f4070090 0x7fa1f41f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.889+0000 7fa1fac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1f40b2820 0x7fa1f413af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.889+0000 7fa1fac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4143dd0 0x7fa1f413e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.889+0000 7fa1fac00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f41f2da0 con 0x7fa1f4143dd0 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.889+0000 7fa1fac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f413b8c0 con 0x7fa1f4070090 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.889+0000 7fa1fac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f406c880 con 0x7fa1f40b2820 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.889+0000 7fa1fa200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4143dd0 0x7fa1f413e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.889+0000 7fa1fa200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4143dd0 0x7fa1f413e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55690/0 (socket says 172.21.15.62:55690) 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.889+0000 7fa1fa200640 1 -- 172.21.15.62:0/4077726974 learned_addr learned my addr 172.21.15.62:0/4077726974 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1f8e00640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1f40b2820 0x7fa1f413af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1f9800640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1f4070090 0x7fa1f41f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:40.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1fa200640 1 -- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1f40b2820 msgr2=0x7fa1f413af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:40.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1fa200640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1f40b2820 0x7fa1f413af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1fa200640 1 -- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1f4070090 msgr2=0x7fa1f41f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:40.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1fa200640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1f4070090 0x7fa1f41f27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:40.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1fa200640 1 -- 172.21.15.62:0/4077726974 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e0009400 con 0x7fa1f4143dd0 2024-03-23T11:32:40.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1f9800640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1f4070090 0x7fa1f41f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:40.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1fa200640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4143dd0 0x7fa1f413e310 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00caf0 tx=0x7fa1ec00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:40.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1f1600640 1 -- 172.21.15.62:0/4077726974 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec00aca0 con 0x7fa1f4143dd0 2024-03-23T11:32:40.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1f1600640 1 -- 172.21.15.62:0/4077726974 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ec00ae00 con 0x7fa1f4143dd0 2024-03-23T11:32:40.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1f1600640 1 -- 172.21.15.62:0/4077726974 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec012600 con 0x7fa1f4143dd0 2024-03-23T11:32:40.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.890+0000 7fa1fac00640 1 -- 172.21.15.62:0/4077726974 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f406d3b0 con 0x7fa1f4143dd0 2024-03-23T11:32:40.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.891+0000 7fa1fac00640 1 -- 172.21.15.62:0/4077726974 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f406cd60 con 0x7fa1f4143dd0 2024-03-23T11:32:40.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.892+0000 7fa1fac00640 1 -- 172.21.15.62:0/4077726974 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f413e8b0 con 0x7fa1f4143dd0 2024-03-23T11:32:40.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.893+0000 7fa1f1600640 1 -- 172.21.15.62:0/4077726974 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fa1ec007070 con 0x7fa1f4143dd0 2024-03-23T11:32:40.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.894+0000 7fa1f1600640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa1d406a820 0x7fa1d406cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:40.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.894+0000 7fa1f1600640 1 -- 172.21.15.62:0/4077726974 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa1ec088c80 con 0x7fa1f4143dd0 2024-03-23T11:32:40.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.894+0000 7fa1f9800640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa1d406a820 0x7fa1d406cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:40.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.895+0000 7fa1f9800640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa1d406a820 0x7fa1d406cce0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8006fd0 tx=0x7fa1e8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:40.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:40.898+0000 7fa1f1600640 1 -- 172.21.15.62:0/4077726974 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa1ec0540d0 con 0x7fa1f4143dd0 2024-03-23T11:32:41.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.097+0000 7fa1fac00640 1 -- 172.21.15.62:0/4077726974 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa1f406d540 con 0x7fa1f4143dd0 2024-03-23T11:32:41.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.098+0000 7fa1f1600640 1 -- 172.21.15.62:0/4077726974 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v122) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa1ec0170a0 con 0x7fa1f4143dd0 2024-03-23T11:32:41.089 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-03-23T11:32:41.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.101+0000 7fa1fac00640 1 -- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa1d406a820 msgr2=0x7fa1d406cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.101+0000 7fa1fac00640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa1d406a820 0x7fa1d406cce0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8006fd0 tx=0x7fa1e8008040 comp rx=0 tx=0).stop 2024-03-23T11:32:41.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.101+0000 7fa1fac00640 1 -- 172.21.15.62:0/4077726974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4143dd0 msgr2=0x7fa1f413e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.101+0000 7fa1fac00640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4143dd0 0x7fa1f413e310 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00caf0 tx=0x7fa1ec00a060 comp rx=0 tx=0).stop 2024-03-23T11:32:41.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.101+0000 7fa1fac00640 1 -- 172.21.15.62:0/4077726974 shutdown_connections 2024-03-23T11:32:41.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.101+0000 7fa1fac00640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fa1d406a820 0x7fa1d406cce0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.101+0000 7fa1fac00640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4143dd0 0x7fa1f413e310 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.101+0000 7fa1fac00640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa1f40b2820 0x7fa1f413af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.101+0000 7fa1fac00640 1 --2- 172.21.15.62:0/4077726974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1f4070090 0x7fa1f41f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.101+0000 7fa1fac00640 1 -- 172.21.15.62:0/4077726974 >> 172.21.15.62:0/4077726974 conn(0x7fa1f4066050 msgr2=0x7fa1f4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:41.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.102+0000 7fa1fac00640 1 -- 172.21.15.62:0/4077726974 shutdown_connections 2024-03-23T11:32:41.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.102+0000 7fa1fac00640 1 -- 172.21.15.62:0/4077726974 wait complete. 2024-03-23T11:32:41.112 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T11:32:41.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.346+0000 7f60f7200640 1 Processor -- start 2024-03-23T11:32:41.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.346+0000 7f60f7200640 1 -- start start 2024-03-23T11:32:41.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.348+0000 7f60f7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f81608b0 0x7f60f80b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.348+0000 7f60f7200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f80ce510 con 0x7f60f8160420 2024-03-23T11:32:41.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.348+0000 7f60f7200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f8160d20 con 0x7f60f815fcf0 2024-03-23T11:32:41.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.348+0000 7f60f7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f80cdc70 con 0x7f60f81608b0 2024-03-23T11:32:41.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.348+0000 7f60f5400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f60f8160420 0x7f60f80b5db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43998/0 (socket says 172.21.15.62:43998) 2024-03-23T11:32:41.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.348+0000 7f60f5400640 1 -- 172.21.15.62:0/2936880183 learned_addr learned my addr 172.21.15.62:0/2936880183 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:41.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.348+0000 7f60f6800640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f81608b0 0x7f60f80b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94252230 0 0) 0x7f60f80ce510 con 0x7f60f8160420 2024-03-23T11:32:41.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60dc003520 con 0x7f60f8160420 2024-03-23T11:32:41.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3279019644 0 0) 0x7f60dc003520 con 0x7f60f8160420 2024-03-23T11:32:41.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f81608b0 msgr2=0x7f60f80b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:41.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f81608b0 0x7f60f80b64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 >> v1:172.21.15.120:6789/0 conn(0x7f60f815fcf0 legacy=0x7f60f80d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f80cdf50 con 0x7f60f8160420 2024-03-23T11:32:41.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f60e4001670 con 0x7f60f8160420 2024-03-23T11:32:41.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 >> v1:172.21.15.62:6789/0 conn(0x7f60f8160420 legacy=0x7f60f80b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f81608b0 0x7f60dc0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60dc003f90 0x7f60dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60dc004d00 0x7f60dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f80cdc70 con 0x7f60dc004d00 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc00bb70 con 0x7f60dc003f90 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.349+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc00bd30 con 0x7f60f81608b0 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.350+0000 7f60f6800640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60dc004d00 0x7f60dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.350+0000 7f60f5400640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60dc003f90 0x7f60dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.350+0000 7f60f5e00640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f81608b0 0x7f60dc0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.350+0000 7f60f6800640 1 -- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f81608b0 msgr2=0x7f60dc0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.350+0000 7f60f6800640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f81608b0 0x7f60dc0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.350+0000 7f60f6800640 1 -- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60dc003f90 msgr2=0x7f60dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.350+0000 7f60f6800640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60dc003f90 0x7f60dc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.350+0000 7f60f6800640 1 -- 172.21.15.62:0/2936880183 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc003520 con 0x7f60dc004d00 2024-03-23T11:32:41.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.350+0000 7f60f5400640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60dc003f90 0x7f60dc0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:41.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.350+0000 7f60f5e00640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f81608b0 0x7f60dc0039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:41.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.351+0000 7f60f6800640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60dc004d00 0x7f60dc005180 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009fc0 tx=0x7f60dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141dab707f4e764e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:41.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.352+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f002cc20 con 0x7f60dc004d00 2024-03-23T11:32:41.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.352+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f000f040 con 0x7f60dc004d00 2024-03-23T11:32:41.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.352+0000 7f60f4a00640 1 -- 172.21.15.62:0/2936880183 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0034440 con 0x7f60dc004d00 2024-03-23T11:32:41.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.352+0000 7f60f7200640 1 -- 172.21.15.62:0/2936880183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60dc004d00 msgr2=0x7f60dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.352+0000 7f60f7200640 1 --2- 172.21.15.62:0/2936880183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60dc004d00 0x7f60dc005180 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009fc0 tx=0x7f60dc0057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.352+0000 7f60f6800640 1 -- 172.21.15.62:0/2936880183 reap_dead start 2024-03-23T11:32:41.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.352+0000 7f60f7200640 1 -- 172.21.15.62:0/2936880183 shutdown_connections 2024-03-23T11:32:41.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.352+0000 7f60f7200640 1 -- 172.21.15.62:0/2936880183 >> 172.21.15.62:0/2936880183 conn(0x7f60f8089f60 msgr2=0x7f60f80d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:41.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.352+0000 7f60f7200640 1 -- 172.21.15.62:0/2936880183 shutdown_connections 2024-03-23T11:32:41.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.352+0000 7f60f7200640 1 -- 172.21.15.62:0/2936880183 wait complete. 2024-03-23T11:32:41.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.354+0000 7f60f7200640 1 Processor -- start 2024-03-23T11:32:41.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.354+0000 7f60f7200640 1 -- start start 2024-03-23T11:32:41.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.355+0000 7f60f7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60dc004d00 0x7f60f8151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.355+0000 7f60f7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f809fa90 0x7f60f809fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.355+0000 7f60f7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f80cec60 0x7f60f80cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.355+0000 7f60f7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f80cf620 con 0x7f60f809fa90 2024-03-23T11:32:41.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.355+0000 7f60f7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f80c6dc0 con 0x7f60dc004d00 2024-03-23T11:32:41.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.355+0000 7f60f7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f8152090 con 0x7f60f80cec60 2024-03-23T11:32:41.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.355+0000 7f60f5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f809fa90 0x7f60f809fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.355+0000 7f60f5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f809fa90 0x7f60f809fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55706/0 (socket says 172.21.15.62:55706) 2024-03-23T11:32:41.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.355+0000 7f60f5400640 1 -- 172.21.15.62:0/956313324 learned_addr learned my addr 172.21.15.62:0/956313324 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60f5e00640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60dc004d00 0x7f60f8151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60f5400640 1 -- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f80cec60 msgr2=0x7f60f80cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60f6800640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f80cec60 0x7f60f80cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60f5400640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f80cec60 0x7f60f80cf0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60f5400640 1 -- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60dc004d00 msgr2=0x7f60f8151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60f5400640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60dc004d00 0x7f60f8151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60f5400640 1 -- 172.21.15.62:0/956313324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f80c5ed0 con 0x7f60f809fa90 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60f5e00640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60dc004d00 0x7f60f8151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60f5400640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f809fa90 0x7f60f809fef0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f60e4001200 tx=0x7f60e400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60ee000640 1 -- 172.21.15.62:0/956313324 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e400c8d0 con 0x7f60f809fa90 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60ee000640 1 -- 172.21.15.62:0/956313324 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e400ca30 con 0x7f60f809fa90 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.356+0000 7f60ee000640 1 -- 172.21.15.62:0/956313324 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e400cda0 con 0x7f60f809fa90 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.357+0000 7f60f7200640 1 -- 172.21.15.62:0/956313324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f80c61b0 con 0x7f60f809fa90 2024-03-23T11:32:41.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.357+0000 7f60f7200640 1 -- 172.21.15.62:0/956313324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f80c6770 con 0x7f60f809fa90 2024-03-23T11:32:41.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.358+0000 7f60f7200640 1 -- 172.21.15.62:0/956313324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f80c6f50 con 0x7f60f809fa90 2024-03-23T11:32:41.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.359+0000 7f60ee000640 1 -- 172.21.15.62:0/956313324 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f60e401d460 con 0x7f60f809fa90 2024-03-23T11:32:41.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.359+0000 7f60ee000640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60d806a7d0 0x7f60d806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.360+0000 7f60ee000640 1 -- 172.21.15.62:0/956313324 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f60e4088ee0 con 0x7f60f809fa90 2024-03-23T11:32:41.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.360+0000 7f60f5e00640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60d806a7d0 0x7f60d806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.361+0000 7f60f5e00640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60d806a7d0 0x7f60d806cc90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f60e8001690 tx=0x7f60e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:41.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.365+0000 7f60ee000640 1 -- 172.21.15.62:0/956313324 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f60e4054200 con 0x7f60f809fa90 2024-03-23T11:32:41.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.565+0000 7f60f7200640 1 -- 172.21.15.62:0/956313324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f60f8066bb0 con 0x7f60f809fa90 2024-03-23T11:32:41.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.566+0000 7f60ee000640 1 -- 172.21.15.62:0/956313324 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v122) v1 ==== 100+0+11 (secure 0 0 0) 0x7f60e4001610 con 0x7f60f809fa90 2024-03-23T11:32:41.557 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-03-23T11:32:41.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.569+0000 7f60f7200640 1 -- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60d806a7d0 msgr2=0x7f60d806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.569+0000 7f60f7200640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60d806a7d0 0x7f60d806cc90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f60e8001690 tx=0x7f60e8009040 comp rx=0 tx=0).stop 2024-03-23T11:32:41.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.569+0000 7f60f7200640 1 -- 172.21.15.62:0/956313324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f809fa90 msgr2=0x7f60f809fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.569+0000 7f60f7200640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f809fa90 0x7f60f809fef0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f60e4001200 tx=0x7f60e400ecd0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.569+0000 7f60f7200640 1 -- 172.21.15.62:0/956313324 shutdown_connections 2024-03-23T11:32:41.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.569+0000 7f60f7200640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f60d806a7d0 0x7f60d806cc90 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.569+0000 7f60f7200640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60f809fa90 0x7f60f809fef0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.569+0000 7f60f7200640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60f80cec60 0x7f60f80cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.569+0000 7f60f7200640 1 --2- 172.21.15.62:0/956313324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60dc004d00 0x7f60f8151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.569+0000 7f60f7200640 1 -- 172.21.15.62:0/956313324 >> 172.21.15.62:0/956313324 conn(0x7f60f8089f60 msgr2=0x7f60f80d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:41.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.570+0000 7f60f7200640 1 -- 172.21.15.62:0/956313324 shutdown_connections 2024-03-23T11:32:41.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.570+0000 7f60f7200640 1 -- 172.21.15.62:0/956313324 wait complete. 2024-03-23T11:32:41.580 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:32:41.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.813+0000 7fe180400640 1 Processor -- start 2024-03-23T11:32:41.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.813+0000 7fe180400640 1 -- start start 2024-03-23T11:32:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.815+0000 7fe180400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe17806c300 0x7fe1780ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.815+0000 7fe180400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17806c6e0 0x7fe17806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.815+0000 7fe180400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1780cbc60 con 0x7fe1780cafa0 2024-03-23T11:32:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.815+0000 7fe180400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1780cbf40 con 0x7fe17806c300 2024-03-23T11:32:41.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.815+0000 7fe180400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1780cc220 con 0x7fe17806c6e0 2024-03-23T11:32:41.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.815+0000 7fe17d000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe1780cafa0 0x7fe1780cb430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44010/0 (socket says 172.21.15.62:44010) 2024-03-23T11:32:41.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.815+0000 7fe17d000640 1 -- 172.21.15.62:0/1683250932 learned_addr learned my addr 172.21.15.62:0/1683250932 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe17da00640 1 --2- 172.21.15.62:0/1683250932 >> v2:172.21.15.120:3300/0 conn(0x7fe17806c300 0x7fe1780ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe17e400640 1 --2- 172.21.15.62:0/1683250932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17806c6e0 0x7fe17806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61143564 0 0) 0x7fe1780cbc60 con 0x7fe1780cafa0 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe160003520 con 0x7fe1780cafa0 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2240691161 0 0) 0x7fe160003520 con 0x7fe1780cafa0 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17806c6e0 msgr2=0x7fe17806cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 --2- 172.21.15.62:0/1683250932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17806c6e0 0x7fe17806cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 >> v2:172.21.15.120:3300/0 conn(0x7fe17806c300 msgr2=0x7fe1780ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 --2- 172.21.15.62:0/1683250932 >> v2:172.21.15.120:3300/0 conn(0x7fe17806c300 0x7fe1780ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1780cc500 con 0x7fe1780cafa0 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fe164001670 con 0x7fe1780cafa0 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 >> v1:172.21.15.62:6789/0 conn(0x7fe1780cafa0 legacy=0x7fe1780cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 --2- 172.21.15.62:0/1683250932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17806c6e0 0x7fe160003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 --2- 172.21.15.62:0/1683250932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe160003f40 0x7fe1600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 --2- 172.21.15.62:0/1683250932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160004d00 0x7fe160005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1780cbf40 con 0x7fe160003f40 2024-03-23T11:32:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.816+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1780cc220 con 0x7fe160004d00 2024-03-23T11:32:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.817+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16000b980 con 0x7fe17806c6e0 2024-03-23T11:32:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.817+0000 7fe17d000640 1 --2- 172.21.15.62:0/1683250932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe160003f40 0x7fe1600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.817+0000 7fe17d000640 1 -- 172.21.15.62:0/1683250932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17806c6e0 msgr2=0x7fe160003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.817+0000 7fe17d000640 1 --2- 172.21.15.62:0/1683250932 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17806c6e0 0x7fe160003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.817+0000 7fe17d000640 1 -- 172.21.15.62:0/1683250932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160004d00 msgr2=0x7fe160005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.817+0000 7fe17d000640 1 --2- 172.21.15.62:0/1683250932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160004d00 0x7fe160005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.817+0000 7fe17d000640 1 -- 172.21.15.62:0/1683250932 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe160003520 con 0x7fe160003f40 2024-03-23T11:32:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.818+0000 7fe17d000640 1 --2- 172.21.15.62:0/1683250932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe160003f40 0x7fe1600047c0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe16400c140 tx=0x7fe1600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e67793f9cf6e878 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.819+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe164005ad0 con 0x7fe160003f40 2024-03-23T11:32:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.819+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe164005c50 con 0x7fe160003f40 2024-03-23T11:32:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.819+0000 7fe177e00640 1 -- 172.21.15.62:0/1683250932 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16402c830 con 0x7fe160003f40 2024-03-23T11:32:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.819+0000 7fe180400640 1 -- 172.21.15.62:0/1683250932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe160003f40 msgr2=0x7fe1600047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.819+0000 7fe180400640 1 --2- 172.21.15.62:0/1683250932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe160003f40 0x7fe1600047c0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe16400c140 tx=0x7fe1600057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.819+0000 7fe17e400640 1 -- 172.21.15.62:0/1683250932 reap_dead start 2024-03-23T11:32:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.819+0000 7fe180400640 1 -- 172.21.15.62:0/1683250932 shutdown_connections 2024-03-23T11:32:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.819+0000 7fe180400640 1 -- 172.21.15.62:0/1683250932 >> 172.21.15.62:0/1683250932 conn(0x7fe1780b02e0 msgr2=0x7fe1780b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.819+0000 7fe180400640 1 -- 172.21.15.62:0/1683250932 shutdown_connections 2024-03-23T11:32:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.819+0000 7fe180400640 1 -- 172.21.15.62:0/1683250932 wait complete. 2024-03-23T11:32:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.821+0000 7fe180400640 1 Processor -- start 2024-03-23T11:32:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.821+0000 7fe180400640 1 -- start start 2024-03-23T11:32:41.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.822+0000 7fe180400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160003f40 0x7fe17813cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.822+0000 7fe180400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1780af2a0 0x7fe1780af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.822+0000 7fe180400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17815c6f0 0x7fe17815cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.822+0000 7fe180400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1780afc60 con 0x7fe1780af2a0 2024-03-23T11:32:41.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.822+0000 7fe180400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17815d420 con 0x7fe160003f40 2024-03-23T11:32:41.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.822+0000 7fe180400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe178151750 con 0x7fe17815c6f0 2024-03-23T11:32:41.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.822+0000 7fe17d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1780af2a0 0x7fe1780af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.822+0000 7fe17d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1780af2a0 0x7fe1780af720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55732/0 (socket says 172.21.15.62:55732) 2024-03-23T11:32:41.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.823+0000 7fe17d000640 1 -- 172.21.15.62:0/2213891052 learned_addr learned my addr 172.21.15.62:0/2213891052 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:41.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.823+0000 7fe17da00640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160003f40 0x7fe17813cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.823+0000 7fe17e400640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17815c6f0 0x7fe17815cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.823+0000 7fe17d000640 1 -- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17815c6f0 msgr2=0x7fe17815cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.823+0000 7fe17d000640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17815c6f0 0x7fe17815cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.823+0000 7fe17d000640 1 -- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160003f40 msgr2=0x7fe17813cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:41.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.823+0000 7fe17d000640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160003f40 0x7fe17813cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:41.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.823+0000 7fe17d000640 1 -- 172.21.15.62:0/2213891052 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe178150e20 con 0x7fe1780af2a0 2024-03-23T11:32:41.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.823+0000 7fe17d000640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1780af2a0 0x7fe1780af720 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe16402c3d0 tx=0x7fe164001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:41.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.824+0000 7fe175600640 1 -- 172.21.15.62:0/2213891052 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16403c040 con 0x7fe1780af2a0 2024-03-23T11:32:41.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.824+0000 7fe175600640 1 -- 172.21.15.62:0/2213891052 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16402dcd0 con 0x7fe1780af2a0 2024-03-23T11:32:41.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.824+0000 7fe175600640 1 -- 172.21.15.62:0/2213891052 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1640072b0 con 0x7fe1780af2a0 2024-03-23T11:32:41.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.824+0000 7fe180400640 1 -- 172.21.15.62:0/2213891052 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe178151100 con 0x7fe1780af2a0 2024-03-23T11:32:41.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.824+0000 7fe180400640 1 -- 172.21.15.62:0/2213891052 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17815d0b0 con 0x7fe1780af2a0 2024-03-23T11:32:41.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.825+0000 7fe180400640 1 -- 172.21.15.62:0/2213891052 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1780ccfa0 con 0x7fe1780af2a0 2024-03-23T11:32:41.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.826+0000 7fe175600640 1 -- 172.21.15.62:0/2213891052 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe164032070 con 0x7fe1780af2a0 2024-03-23T11:32:41.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.827+0000 7fe175600640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe15806a7a0 0x7fe15806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:41.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.827+0000 7fe175600640 1 -- 172.21.15.62:0/2213891052 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fe1640aa530 con 0x7fe1780af2a0 2024-03-23T11:32:41.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.828+0000 7fe17da00640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe15806a7a0 0x7fe15806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:41.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.829+0000 7fe17da00640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe15806a7a0 0x7fe15806cc60 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009ad0 tx=0x7fe16c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:41.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:41.832+0000 7fe175600640 1 -- 172.21.15.62:0/2213891052 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1640757c0 con 0x7fe1780af2a0 2024-03-23T11:32:42.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.073+0000 7fe180400640 1 -- 172.21.15.62:0/2213891052 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe178066c90 con 0x7fe1780af2a0 2024-03-23T11:32:42.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.074+0000 7fe175600640 1 -- 172.21.15.62:0/2213891052 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fe164078de0 con 0x7fe1780af2a0 2024-03-23T11:32:42.065 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:42.065 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":107,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:32:28.458742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9507":{"gid":9507,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6835/2500981044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2500981044},{"type":"v1","addr":"172.21.15.62:6835","nonce":2500981044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:32:42.065 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 107 2024-03-23T11:32:42.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.077+0000 7fe180400640 1 -- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe15806a7a0 msgr2=0x7fe15806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:42.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.077+0000 7fe180400640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe15806a7a0 0x7fe15806cc60 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009ad0 tx=0x7fe16c008040 comp rx=0 tx=0).stop 2024-03-23T11:32:42.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.077+0000 7fe180400640 1 -- 172.21.15.62:0/2213891052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1780af2a0 msgr2=0x7fe1780af720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:42.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.077+0000 7fe180400640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1780af2a0 0x7fe1780af720 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe16402c3d0 tx=0x7fe164001930 comp rx=0 tx=0).stop 2024-03-23T11:32:42.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.077+0000 7fe180400640 1 -- 172.21.15.62:0/2213891052 shutdown_connections 2024-03-23T11:32:42.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.077+0000 7fe180400640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe15806a7a0 0x7fe15806cc60 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.077+0000 7fe180400640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1780af2a0 0x7fe1780af720 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.077+0000 7fe180400640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe17815c6f0 0x7fe17815cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.077+0000 7fe180400640 1 --2- 172.21.15.62:0/2213891052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe160003f40 0x7fe17813cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.077+0000 7fe180400640 1 -- 172.21.15.62:0/2213891052 >> 172.21.15.62:0/2213891052 conn(0x7fe1780b02e0 msgr2=0x7fe17813ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.078+0000 7fe180400640 1 -- 172.21.15.62:0/2213891052 shutdown_connections 2024-03-23T11:32:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.078+0000 7fe180400640 1 -- 172.21.15.62:0/2213891052 wait complete. 2024-03-23T11:32:42.088 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:32:42.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.313+0000 7f52ff000640 1 Processor -- start 2024-03-23T11:32:42.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.313+0000 7f52ff000640 1 -- start start 2024-03-23T11:32:42.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.315+0000 7f52ff000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52f8070090 0x7f52f80704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.315+0000 7f52ff000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f52f80b2820 0x7f52f80b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.315+0000 7f52ff000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f8070a30 con 0x7f52f8143dd0 2024-03-23T11:32:42.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.315+0000 7f52ff000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f80cd140 con 0x7f52f80b2820 2024-03-23T11:32:42.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.315+0000 7f52ff000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f80b3180 con 0x7f52f8070090 2024-03-23T11:32:42.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.315+0000 7f52fdc00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f52f8143dd0 0x7f52f815abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44026/0 (socket says 172.21.15.62:44026) 2024-03-23T11:32:42.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.315+0000 7f52fdc00640 1 -- 172.21.15.62:0/4270590049 learned_addr learned my addr 172.21.15.62:0/4270590049 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:42.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52fe600640 1 --2- 172.21.15.62:0/4270590049 >> v2:172.21.15.120:3300/0 conn(0x7f52f80b2820 0x7f52f80b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52fd200640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52f8070090 0x7f52f80704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871965783 0 0) 0x7f52f8070a30 con 0x7f52f8143dd0 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e0003520 con 0x7f52f8143dd0 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1000981068 0 0) 0x7f52e0003520 con 0x7f52f8143dd0 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52f8070090 msgr2=0x7f52f80704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52f8070090 0x7f52f80704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 >> v2:172.21.15.120:3300/0 conn(0x7f52f80b2820 msgr2=0x7f52f80b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 --2- 172.21.15.62:0/4270590049 >> v2:172.21.15.120:3300/0 conn(0x7f52f80b2820 0x7f52f80b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f80cd3e0 con 0x7f52f8143dd0 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f52ec001670 con 0x7f52f8143dd0 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 >> v1:172.21.15.62:6789/0 conn(0x7f52f8143dd0 legacy=0x7f52f815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52f8070090 0x7f52e0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0003ef0 0x7f52e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e0004d00 0x7f52e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f80cd140 con 0x7f52e0003ef0 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f80b3180 con 0x7f52e0004d00 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.316+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e000b980 con 0x7f52f8070090 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.317+0000 7f52fd200640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0003ef0 0x7f52e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.317+0000 7f52fdc00640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52f8070090 0x7f52e0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.317+0000 7f52fe600640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e0004d00 0x7f52e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.317+0000 7f52fd200640 1 -- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52f8070090 msgr2=0x7f52e0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.317+0000 7f52fd200640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52f8070090 0x7f52e0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.317+0000 7f52fd200640 1 -- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e0004d00 msgr2=0x7f52e0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.317+0000 7f52fd200640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e0004d00 0x7f52e0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.317+0000 7f52fd200640 1 -- 172.21.15.62:0/4270590049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e0003520 con 0x7f52e0003ef0 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.317+0000 7f52fdc00640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52f8070090 0x7f52e0003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:42.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.317+0000 7f52fe600640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52e0004d00 0x7f52e0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:42.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.318+0000 7f52fd200640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0003ef0 0x7f52e00047c0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009ac0 tx=0x7f52e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96173f78c835a97a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:42.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.318+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e402c6b0 con 0x7f52e0003ef0 2024-03-23T11:32:42.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.318+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e402c810 con 0x7f52e0003ef0 2024-03-23T11:32:42.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.318+0000 7f52f7e00640 1 -- 172.21.15.62:0/4270590049 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e402cba0 con 0x7f52e0003ef0 2024-03-23T11:32:42.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.319+0000 7f52ff000640 1 -- 172.21.15.62:0/4270590049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0003ef0 msgr2=0x7f52e00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:42.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.319+0000 7f52ff000640 1 --2- 172.21.15.62:0/4270590049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0003ef0 0x7f52e00047c0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009ac0 tx=0x7f52e00057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.319+0000 7f52fe600640 1 -- 172.21.15.62:0/4270590049 reap_dead start 2024-03-23T11:32:42.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.319+0000 7f52ff000640 1 -- 172.21.15.62:0/4270590049 shutdown_connections 2024-03-23T11:32:42.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.319+0000 7f52ff000640 1 -- 172.21.15.62:0/4270590049 >> 172.21.15.62:0/4270590049 conn(0x7f52f8066050 msgr2=0x7f52f8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:42.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.319+0000 7f52ff000640 1 -- 172.21.15.62:0/4270590049 shutdown_connections 2024-03-23T11:32:42.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.319+0000 7f52ff000640 1 -- 172.21.15.62:0/4270590049 wait complete. 2024-03-23T11:32:42.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.321+0000 7f52ff000640 1 Processor -- start 2024-03-23T11:32:42.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.321+0000 7f52ff000640 1 -- start start 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.322+0000 7f52ff000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e0003ef0 0x7f52f815b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.322+0000 7f52ff000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52f80acb00 0x7f52f81f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.322+0000 7f52ff000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f80acf60 0x7f52f81f57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.322+0000 7f52ff000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f815b100 con 0x7f52f80acf60 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.322+0000 7f52ff000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f815c6a0 con 0x7f52f80acb00 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.322+0000 7f52ff000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f815a4a0 con 0x7f52e0003ef0 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fe600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f80acf60 0x7f52f81f57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fe600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f80acf60 0x7f52f81f57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55746/0 (socket says 172.21.15.62:55746) 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fe600640 1 -- 172.21.15.62:0/1392236160 learned_addr learned my addr 172.21.15.62:0/1392236160 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fdc00640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e0003ef0 0x7f52f815b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fe600640 1 -- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e0003ef0 msgr2=0x7f52f815b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fd200640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52f80acb00 0x7f52f81f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fe600640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e0003ef0 0x7f52f815b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fe600640 1 -- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52f80acb00 msgr2=0x7f52f81f5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:42.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fe600640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52f80acb00 0x7f52f81f5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fe600640 1 -- 172.21.15.62:0/1392236160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f81f5e70 con 0x7f52f80acf60 2024-03-23T11:32:42.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fdc00640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e0003ef0 0x7f52f815b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:42.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fd200640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52f80acb00 0x7f52f81f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:42.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.323+0000 7f52fe600640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f80acf60 0x7f52f81f57a0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f52f806e450 tx=0x7f52f000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:42.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.324+0000 7f52f5600640 1 -- 172.21.15.62:0/1392236160 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f0016620 con 0x7f52f80acf60 2024-03-23T11:32:42.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.324+0000 7f52f5600640 1 -- 172.21.15.62:0/1392236160 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f0016780 con 0x7f52f80acf60 2024-03-23T11:32:42.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.324+0000 7f52f5600640 1 -- 172.21.15.62:0/1392236160 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f0016b10 con 0x7f52f80acf60 2024-03-23T11:32:42.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.324+0000 7f52ff000640 1 -- 172.21.15.62:0/1392236160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f81f6150 con 0x7f52f80acf60 2024-03-23T11:32:42.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.324+0000 7f52ff000640 1 -- 172.21.15.62:0/1392236160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f815ad20 con 0x7f52f80acf60 2024-03-23T11:32:42.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.325+0000 7f52ff000640 1 -- 172.21.15.62:0/1392236160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f815b240 con 0x7f52f80acf60 2024-03-23T11:32:42.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.326+0000 7f52f5600640 1 -- 172.21.15.62:0/1392236160 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f52f0016c70 con 0x7f52f80acf60 2024-03-23T11:32:42.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.327+0000 7f52f5600640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52d806a7a0 0x7f52d806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.327+0000 7f52f5600640 1 -- 172.21.15.62:0/1392236160 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f52f0089b90 con 0x7f52f80acf60 2024-03-23T11:32:42.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.328+0000 7f52fdc00640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52d806a7a0 0x7f52d806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.329+0000 7f52fdc00640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52d806a7a0 0x7f52d806cc60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0018e0 tx=0x7f52ec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:42.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.331+0000 7f52f5600640 1 -- 172.21.15.62:0/1392236160 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f52f815b240 con 0x7f52f80acf60 2024-03-23T11:32:42.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.529+0000 7f52ff000640 1 -- 172.21.15.62:0/1392236160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52f8066bb0 con 0x7f52f80acf60 2024-03-23T11:32:42.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.532+0000 7f52f5600640 1 -- 172.21.15.62:0/1392236160 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+16413 (secure 0 0 0) 0x7f52f0054f50 con 0x7f52f80acf60 2024-03-23T11:32:42.523 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:42.523 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":122,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:32:23.551951+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6835/4129977589":"2024-03-24T11:32:23.551904+0000","172.21.15.62:6834/4129977589":"2024-03-24T11:32:23.551904+0000","172.21.15.120:6837/2585024035":"2024-03-24T11:32:04.365655+0000","172.21.15.120:6835/2585024035":"2024-03-24T11:32:04.365655+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:32:42.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 -- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52d806a7a0 msgr2=0x7f52d806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:42.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52d806a7a0 0x7f52d806cc60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0018e0 tx=0x7f52ec009040 comp rx=0 tx=0).stop 2024-03-23T11:32:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 -- 172.21.15.62:0/1392236160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f80acf60 msgr2=0x7f52f81f57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f80acf60 0x7f52f81f57a0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f52f806e450 tx=0x7f52f000f520 comp rx=0 tx=0).stop 2024-03-23T11:32:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 -- 172.21.15.62:0/1392236160 shutdown_connections 2024-03-23T11:32:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f52d806a7a0 0x7f52d806cc60 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52f80acf60 0x7f52f81f57a0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52f80acb00 0x7f52f81f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 --2- 172.21.15.62:0/1392236160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f52e0003ef0 0x7f52f815b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 -- 172.21.15.62:0/1392236160 >> 172.21.15.62:0/1392236160 conn(0x7f52f8066050 msgr2=0x7f52f80af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 -- 172.21.15.62:0/1392236160 shutdown_connections 2024-03-23T11:32:42.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.535+0000 7f52ff000640 1 -- 172.21.15.62:0/1392236160 wait complete. 2024-03-23T11:32:42.544 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T11:32:42.544 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:32:42.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.774+0000 7f3a17200640 1 Processor -- start 2024-03-23T11:32:42.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.774+0000 7f3a17200640 1 -- start start 2024-03-23T11:32:42.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.776+0000 7f3a17200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a18143dd0 0x7f3a1815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.776+0000 7f3a17200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3a18070090 0x7f3a180704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.776+0000 7f3a17200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3a180b2820 0x7f3a180b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.776+0000 7f3a17200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a18070a30 con 0x7f3a18070090 2024-03-23T11:32:42.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.776+0000 7f3a17200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a180cd140 con 0x7f3a180b2820 2024-03-23T11:32:42.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.776+0000 7f3a17200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a180b3180 con 0x7f3a18143dd0 2024-03-23T11:32:42.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.776+0000 7f3a15400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3a18070090 0x7f3a180704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.777+0000 7f3a15400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3a18070090 0x7f3a180704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55758/0 (socket says 172.21.15.62:55758) 2024-03-23T11:32:42.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.777+0000 7f3a15400640 1 -- 172.21.15.62:0/4160767879 learned_addr learned my addr 172.21.15.62:0/4160767879 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:42.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.777+0000 7f3a16800640 1 --2- 172.21.15.62:0/4160767879 >> v2:172.21.15.120:3300/0 conn(0x7f3a180b2820 0x7f3a180b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.777+0000 7f3a15e00640 1 --2- 172.21.15.62:0/4160767879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a18143dd0 0x7f3a1815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.777+0000 7f3a15400640 1 -- 172.21.15.62:0/4160767879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a18143dd0 msgr2=0x7f3a1815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:42.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.777+0000 7f3a15400640 1 --2- 172.21.15.62:0/4160767879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a18143dd0 0x7f3a1815abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.777+0000 7f3a15400640 1 -- 172.21.15.62:0/4160767879 >> v2:172.21.15.120:3300/0 conn(0x7f3a180b2820 msgr2=0x7f3a180b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:42.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.777+0000 7f3a15400640 1 --2- 172.21.15.62:0/4160767879 >> v2:172.21.15.120:3300/0 conn(0x7f3a180b2820 0x7f3a180b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.777+0000 7f3a15400640 1 -- 172.21.15.62:0/4160767879 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a180cd3e0 con 0x7f3a18070090 2024-03-23T11:32:42.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.779+0000 7f3a15400640 1 --2- 172.21.15.62:0/4160767879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a18070090 0x7f3a180704f0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f3a18071500 tx=0x7f3a0402b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2edeefcba12726b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:42.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.779+0000 7f3a14a00640 1 -- 172.21.15.62:0/4160767879 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04031880 con 0x7f3a18070090 2024-03-23T11:32:42.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.779+0000 7f3a14a00640 1 -- 172.21.15.62:0/4160767879 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a040319e0 con 0x7f3a18070090 2024-03-23T11:32:42.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.779+0000 7f3a14a00640 1 -- 172.21.15.62:0/4160767879 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04031d70 con 0x7f3a18070090 2024-03-23T11:32:42.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.780+0000 7f3a17200640 1 -- 172.21.15.62:0/4160767879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a18070090 msgr2=0x7f3a180704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:42.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.780+0000 7f3a17200640 1 --2- 172.21.15.62:0/4160767879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a18070090 0x7f3a180704f0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f3a18071500 tx=0x7f3a0402b920 comp rx=0 tx=0).stop 2024-03-23T11:32:42.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.780+0000 7f3a17200640 1 -- 172.21.15.62:0/4160767879 shutdown_connections 2024-03-23T11:32:42.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.780+0000 7f3a17200640 1 --2- 172.21.15.62:0/4160767879 >> v2:172.21.15.120:3300/0 conn(0x7f3a180b2820 0x7f3a180b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.780+0000 7f3a17200640 1 --2- 172.21.15.62:0/4160767879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a18070090 0x7f3a180704f0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.780+0000 7f3a17200640 1 --2- 172.21.15.62:0/4160767879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a18143dd0 0x7f3a1815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.780+0000 7f3a17200640 1 -- 172.21.15.62:0/4160767879 >> 172.21.15.62:0/4160767879 conn(0x7f3a18066050 msgr2=0x7f3a18069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:42.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.780+0000 7f3a17200640 1 -- 172.21.15.62:0/4160767879 shutdown_connections 2024-03-23T11:32:42.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.780+0000 7f3a17200640 1 -- 172.21.15.62:0/4160767879 wait complete. 2024-03-23T11:32:42.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.782+0000 7f3a17200640 1 Processor -- start 2024-03-23T11:32:42.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.782+0000 7f3a17200640 1 -- start start 2024-03-23T11:32:42.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a17200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a18070090 0x7f3a18136560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a17200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a180b2820 0x7f3a181f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a17200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a18143dd0 0x7f3a1815b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a17200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a181f1ae0 con 0x7f3a180b2820 2024-03-23T11:32:42.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a17200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a18136aa0 con 0x7f3a18143dd0 2024-03-23T11:32:42.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a17200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a181f0b20 con 0x7f3a18070090 2024-03-23T11:32:42.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a15400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a180b2820 0x7f3a181f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a15400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a180b2820 0x7f3a181f05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55764/0 (socket says 172.21.15.62:55764) 2024-03-23T11:32:42.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a15400640 1 -- 172.21.15.62:0/980240801 learned_addr learned my addr 172.21.15.62:0/980240801 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:42.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a16800640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a18143dd0 0x7f3a1815b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a15400640 1 -- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a18070090 msgr2=0x7f3a18136560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:42.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a15e00640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a18070090 0x7f3a18136560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.784+0000 7f3a15400640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a18070090 0x7f3a18136560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.785+0000 7f3a15400640 1 -- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a18143dd0 msgr2=0x7f3a1815b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:42.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.785+0000 7f3a15400640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a18143dd0 0x7f3a1815b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:42.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.785+0000 7f3a15400640 1 -- 172.21.15.62:0/980240801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a040094d0 con 0x7f3a180b2820 2024-03-23T11:32:42.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.785+0000 7f3a16800640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a18143dd0 0x7f3a1815b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:42.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.785+0000 7f3a15400640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a180b2820 0x7f3a181f05e0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f3a0402b960 tx=0x7f3a040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:42.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.785+0000 7f3a0e000640 1 -- 172.21.15.62:0/980240801 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0403ac60 con 0x7f3a180b2820 2024-03-23T11:32:42.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.785+0000 7f3a0e000640 1 -- 172.21.15.62:0/980240801 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0403ade0 con 0x7f3a180b2820 2024-03-23T11:32:42.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.785+0000 7f3a0e000640 1 -- 172.21.15.62:0/980240801 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04044630 con 0x7f3a180b2820 2024-03-23T11:32:42.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.785+0000 7f3a17200640 1 -- 172.21.15.62:0/980240801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a181f0ed0 con 0x7f3a180b2820 2024-03-23T11:32:42.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.786+0000 7f3a17200640 1 -- 172.21.15.62:0/980240801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a181f1490 con 0x7f3a180b2820 2024-03-23T11:32:42.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.787+0000 7f3a17200640 1 -- 172.21.15.62:0/980240801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a181f1c50 con 0x7f3a180b2820 2024-03-23T11:32:42.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.788+0000 7f3a0e000640 1 -- 172.21.15.62:0/980240801 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3a0404d450 con 0x7f3a180b2820 2024-03-23T11:32:42.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.789+0000 7f3a0e000640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39f806a7a0 0x7f39f806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:42.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.789+0000 7f3a0e000640 1 -- 172.21.15.62:0/980240801 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f3a040ab3e0 con 0x7f3a180b2820 2024-03-23T11:32:42.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.789+0000 7f3a15e00640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39f806a7a0 0x7f39f806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:42.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.790+0000 7f3a15e00640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39f806a7a0 0x7f39f806cc60 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f3a08006fd0 tx=0x7f3a08008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:42.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:42.793+0000 7f3a0e000640 1 -- 172.21.15.62:0/980240801 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3a04076710 con 0x7f3a180b2820 2024-03-23T11:32:43.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.025+0000 7f3a17200640 1 -- 172.21.15.62:0/980240801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a18066bb0 con 0x7f3a180b2820 2024-03-23T11:32:43.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.026+0000 7f3a0e000640 1 -- 172.21.15.62:0/980240801 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f3a04035090 con 0x7f3a180b2820 2024-03-23T11:32:43.016 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:43.016 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":107,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:32:28.458742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9507":{"gid":9507,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6835/2500981044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2500981044},{"type":"v1","addr":"172.21.15.62:6835","nonce":2500981044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:32:43.017 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 107 2024-03-23T11:32:43.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 -- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39f806a7a0 msgr2=0x7f39f806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39f806a7a0 0x7f39f806cc60 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f3a08006fd0 tx=0x7f3a08008040 comp rx=0 tx=0).stop 2024-03-23T11:32:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 -- 172.21.15.62:0/980240801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a180b2820 msgr2=0x7f3a181f05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a180b2820 0x7f3a181f05e0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f3a0402b960 tx=0x7f3a040046c0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 -- 172.21.15.62:0/980240801 shutdown_connections 2024-03-23T11:32:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f39f806a7a0 0x7f39f806cc60 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a180b2820 0x7f3a181f05e0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a18143dd0 0x7f3a1815b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 --2- 172.21.15.62:0/980240801 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a18070090 0x7f3a18136560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 -- 172.21.15.62:0/980240801 >> 172.21.15.62:0/980240801 conn(0x7f3a18066050 msgr2=0x7f3a180b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 -- 172.21.15.62:0/980240801 shutdown_connections 2024-03-23T11:32:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.029+0000 7f3a17200640 1 -- 172.21.15.62:0/980240801 wait complete. 2024-03-23T11:32:43.036 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:32:43.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.254+0000 7fef5fe00640 1 Processor -- start 2024-03-23T11:32:43.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.254+0000 7fef5fe00640 1 -- start start 2024-03-23T11:32:43.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.256+0000 7fef5fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6006cc60 0x7fef600cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.256+0000 7fef5fe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef600ce8b0 con 0x7fef600ce420 2024-03-23T11:32:43.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.256+0000 7fef5fe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef600ced90 con 0x7fef60159490 2024-03-23T11:32:43.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.256+0000 7fef5fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6006e3e0 con 0x7fef6006cc60 2024-03-23T11:32:43.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.256+0000 7fef5f400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fef600ce420 0x7fef600b26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44034/0 (socket says 172.21.15.62:44034) 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.256+0000 7fef5f400640 1 -- 172.21.15.62:0/3889116382 learned_addr learned my addr 172.21.15.62:0/3889116382 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.256+0000 7fef5e000640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6006cc60 0x7fef600cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654253197 0 0) 0x7fef600ce8b0 con 0x7fef600ce420 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef44003520 con 0x7fef600ce420 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1383708652 0 0) 0x7fef44003520 con 0x7fef600ce420 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6006cc60 msgr2=0x7fef600cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6006cc60 0x7fef600cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 >> v1:172.21.15.120:6789/0 conn(0x7fef60159490 legacy=0x7fef6015da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6006c1c0 con 0x7fef600ce420 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fef580019f0 con 0x7fef600ce420 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 >> v1:172.21.15.62:6789/0 conn(0x7fef600ce420 legacy=0x7fef600b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef440039a0 0x7fef44003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef440047c0 0x7fef44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef44005180 0x7fef4400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6006e3e0 con 0x7fef44005180 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4400c080 con 0x7fef440047c0 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4400c240 con 0x7fef440039a0 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.257+0000 7fef5e000640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef6006cc60 0x7fef600cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.258+0000 7fef5f400640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef44005180 0x7fef4400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.258+0000 7fef5ea00640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef440039a0 0x7fef44003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.258+0000 7fef5e000640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef440047c0 0x7fef44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.258+0000 7fef5f400640 1 -- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef440039a0 msgr2=0x7fef44003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.258+0000 7fef5f400640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef440039a0 0x7fef44003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.258+0000 7fef5f400640 1 -- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef440047c0 msgr2=0x7fef44004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.258+0000 7fef5f400640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef440047c0 0x7fef44004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.258+0000 7fef5f400640 1 -- 172.21.15.62:0/3889116382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef44003520 con 0x7fef44005180 2024-03-23T11:32:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.259+0000 7fef5f400640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef44005180 0x7fef4400b980 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fef5800a730 tx=0x7fef440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae7bf97be0b48e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.259+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58034630 con 0x7fef44005180 2024-03-23T11:32:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.259+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5802f070 con 0x7fef44005180 2024-03-23T11:32:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.259+0000 7fef5d600640 1 -- 172.21.15.62:0/3889116382 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58034a50 con 0x7fef44005180 2024-03-23T11:32:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.260+0000 7fef5fe00640 1 -- 172.21.15.62:0/3889116382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef44005180 msgr2=0x7fef4400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.260+0000 7fef5fe00640 1 --2- 172.21.15.62:0/3889116382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef44005180 0x7fef4400b980 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fef5800a730 tx=0x7fef440057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.260+0000 7fef5f400640 1 -- 172.21.15.62:0/3889116382 reap_dead start 2024-03-23T11:32:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.260+0000 7fef5fe00640 1 -- 172.21.15.62:0/3889116382 shutdown_connections 2024-03-23T11:32:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.260+0000 7fef5fe00640 1 -- 172.21.15.62:0/3889116382 >> 172.21.15.62:0/3889116382 conn(0x7fef60066050 msgr2=0x7fef60069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.260+0000 7fef5fe00640 1 -- 172.21.15.62:0/3889116382 shutdown_connections 2024-03-23T11:32:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.260+0000 7fef5fe00640 1 -- 172.21.15.62:0/3889116382 wait complete. 2024-03-23T11:32:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.262+0000 7fef5fe00640 1 Processor -- start 2024-03-23T11:32:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.262+0000 7fef5fe00640 1 -- start start 2024-03-23T11:32:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.263+0000 7fef5fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef44005180 0x7fef60154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.263+0000 7fef5fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef601e7090 0x7fef601e5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.263+0000 7fef5fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef60153430 0x7fef601538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.263+0000 7fef5fe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef601e55e0 con 0x7fef60153430 2024-03-23T11:32:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.263+0000 7fef5fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6015c3b0 con 0x7fef44005180 2024-03-23T11:32:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.263+0000 7fef5fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6015c690 con 0x7fef601e7090 2024-03-23T11:32:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.263+0000 7fef5f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef60153430 0x7fef601538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.263+0000 7fef5f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef60153430 0x7fef601538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55792/0 (socket says 172.21.15.62:55792) 2024-03-23T11:32:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.263+0000 7fef5f400640 1 -- 172.21.15.62:0/1782946898 learned_addr learned my addr 172.21.15.62:0/1782946898 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef5ea00640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef44005180 0x7fef60154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef5f400640 1 -- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef601e7090 msgr2=0x7fef601e5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef5f400640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef601e7090 0x7fef601e5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef5f400640 1 -- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef44005180 msgr2=0x7fef60154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef5f400640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef44005180 0x7fef60154aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef5f400640 1 -- 172.21.15.62:0/1782946898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef601e4f90 con 0x7fef60153430 2024-03-23T11:32:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef5f400640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef60153430 0x7fef601538b0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fef5800a850 tx=0x7fef6006c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef56a00640 1 -- 172.21.15.62:0/1782946898 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58034640 con 0x7fef60153430 2024-03-23T11:32:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef56a00640 1 -- 172.21.15.62:0/1782946898 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef580347a0 con 0x7fef60153430 2024-03-23T11:32:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef56a00640 1 -- 172.21.15.62:0/1782946898 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58034b30 con 0x7fef60153430 2024-03-23T11:32:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef5fe00640 1 -- 172.21.15.62:0/1782946898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef601e5270 con 0x7fef60153430 2024-03-23T11:32:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.264+0000 7fef5fe00640 1 -- 172.21.15.62:0/1782946898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef60153fd0 con 0x7fef60153430 2024-03-23T11:32:43.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.266+0000 7fef5fe00640 1 -- 172.21.15.62:0/1782946898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef601e5400 con 0x7fef60153430 2024-03-23T11:32:43.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.267+0000 7fef56a00640 1 -- 172.21.15.62:0/1782946898 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fef58031030 con 0x7fef60153430 2024-03-23T11:32:43.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.267+0000 7fef56a00640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3c06a7d0 0x7fef3c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.267+0000 7fef56a00640 1 -- 172.21.15.62:0/1782946898 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fef580ab1a0 con 0x7fef60153430 2024-03-23T11:32:43.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.268+0000 7fef5ea00640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3c06a7d0 0x7fef3c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.269+0000 7fef5ea00640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3c06a7d0 0x7fef3c06cc90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fef50007fd0 tx=0x7fef50009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.272+0000 7fef56a00640 1 -- 172.21.15.62:0/1782946898 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef580764d0 con 0x7fef60153430 2024-03-23T11:32:43.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.506+0000 7fef5fe00640 1 -- 172.21.15.62:0/1782946898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef601e5400 con 0x7fef60153430 2024-03-23T11:32:43.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.507+0000 7fef56a00640 1 -- 172.21.15.62:0/1782946898 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fef58079af0 con 0x7fef60153430 2024-03-23T11:32:43.497 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:43.497 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":107,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T11:31:24.044342+0000","modified":"2024-03-23T11:32:28.458742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9507},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9507":{"gid":9507,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":4,"addr":"172.21.15.62:6835/2500981044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":2500981044},{"type":"v1","addr":"172.21.15.62:6835","nonce":2500981044}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-23T11:32:43.498 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 107 2024-03-23T11:32:43.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.509+0000 7fef5fe00640 1 -- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3c06a7d0 msgr2=0x7fef3c06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.509+0000 7fef5fe00640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3c06a7d0 0x7fef3c06cc90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fef50007fd0 tx=0x7fef50009040 comp rx=0 tx=0).stop 2024-03-23T11:32:43.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.509+0000 7fef5fe00640 1 -- 172.21.15.62:0/1782946898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef60153430 msgr2=0x7fef601538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.509+0000 7fef5fe00640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef60153430 0x7fef601538b0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fef5800a850 tx=0x7fef6006c350 comp rx=0 tx=0).stop 2024-03-23T11:32:43.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.509+0000 7fef5fe00640 1 -- 172.21.15.62:0/1782946898 shutdown_connections 2024-03-23T11:32:43.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.509+0000 7fef5fe00640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef3c06a7d0 0x7fef3c06cc90 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.509+0000 7fef5fe00640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef60153430 0x7fef601538b0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.509+0000 7fef5fe00640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef601e7090 0x7fef601e5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.509+0000 7fef5fe00640 1 --2- 172.21.15.62:0/1782946898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef44005180 0x7fef60154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.509+0000 7fef5fe00640 1 -- 172.21.15.62:0/1782946898 >> 172.21.15.62:0/1782946898 conn(0x7fef60066050 msgr2=0x7fef600cd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:43.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.510+0000 7fef5fe00640 1 -- 172.21.15.62:0/1782946898 shutdown_connections 2024-03-23T11:32:43.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.510+0000 7fef5fe00640 1 -- 172.21.15.62:0/1782946898 wait complete. 2024-03-23T11:32:43.519 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:32:43.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.751+0000 7ffa28a00640 1 Processor -- start 2024-03-23T11:32:43.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.751+0000 7ffa28a00640 1 -- start start 2024-03-23T11:32:43.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.753+0000 7ffa28a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffa241608b0 0x7ffa240b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.753+0000 7ffa28a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa240ce510 con 0x7ffa24160420 2024-03-23T11:32:43.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.753+0000 7ffa28a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa240cdc70 con 0x7ffa2415fcf0 2024-03-23T11:32:43.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.753+0000 7ffa28a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa240cdf50 con 0x7ffa241608b0 2024-03-23T11:32:43.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.753+0000 7ffa19e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ffa24160420 0x7ffa240b5db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44050/0 (socket says 172.21.15.62:44050) 2024-03-23T11:32:43.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.753+0000 7ffa19e00640 1 -- 172.21.15.62:0/3878849827 learned_addr learned my addr 172.21.15.62:0/3878849827 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:43.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa23200640 1 --2- 172.21.15.62:0/3878849827 >> v2:172.21.15.120:3300/0 conn(0x7ffa241608b0 0x7ffa240b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191049506 0 0) 0x7ffa240ce510 con 0x7ffa24160420 2024-03-23T11:32:43.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa08003520 con 0x7ffa24160420 2024-03-23T11:32:43.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47875929 0 0) 0x7ffa08003520 con 0x7ffa24160420 2024-03-23T11:32:43.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 >> v2:172.21.15.120:3300/0 conn(0x7ffa241608b0 msgr2=0x7ffa240b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:43.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 --2- 172.21.15.62:0/3878849827 >> v2:172.21.15.120:3300/0 conn(0x7ffa241608b0 0x7ffa240b64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 >> v1:172.21.15.120:6789/0 conn(0x7ffa2415fcf0 legacy=0x7ffa240d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa240ce230 con 0x7ffa24160420 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7ffa0c001670 con 0x7ffa24160420 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 >> v1:172.21.15.62:6789/0 conn(0x7ffa24160420 legacy=0x7ffa240b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 --2- 172.21.15.62:0/3878849827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa080039c0 0x7ffa08003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 --2- 172.21.15.62:0/3878849827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 0x7ffa08004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.754+0000 7ffa21e00640 1 --2- 172.21.15.62:0/3878849827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa080051d0 0x7ffa0800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa240cdf50 con 0x7ffa080047c0 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0800c120 con 0x7ffa080051d0 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0800c2e0 con 0x7ffa080039c0 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa19e00640 1 --2- 172.21.15.62:0/3878849827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 0x7ffa08004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa23200640 1 --2- 172.21.15.62:0/3878849827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa080051d0 0x7ffa0800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa22800640 1 --2- 172.21.15.62:0/3878849827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa080039c0 0x7ffa08003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa19e00640 1 -- 172.21.15.62:0/3878849827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa080039c0 msgr2=0x7ffa08003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa19e00640 1 --2- 172.21.15.62:0/3878849827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa080039c0 0x7ffa08003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa19e00640 1 -- 172.21.15.62:0/3878849827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa080051d0 msgr2=0x7ffa0800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa19e00640 1 --2- 172.21.15.62:0/3878849827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa080051d0 0x7ffa0800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa23200640 1 -- 172.21.15.62:0/3878849827 reap_dead start 2024-03-23T11:32:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.755+0000 7ffa19e00640 1 -- 172.21.15.62:0/3878849827 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa08003520 con 0x7ffa080047c0 2024-03-23T11:32:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.756+0000 7ffa19e00640 1 --2- 172.21.15.62:0/3878849827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 0x7ffa08004c40 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c001eb0 tx=0x7ffa080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630022b02fcbc4ab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.757+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c005ad0 con 0x7ffa080047c0 2024-03-23T11:32:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.757+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0c005c50 con 0x7ffa080047c0 2024-03-23T11:32:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.757+0000 7ffa21e00640 1 -- 172.21.15.62:0/3878849827 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c02c790 con 0x7ffa080047c0 2024-03-23T11:32:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.757+0000 7ffa28a00640 1 -- 172.21.15.62:0/3878849827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 msgr2=0x7ffa08004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.757+0000 7ffa28a00640 1 --2- 172.21.15.62:0/3878849827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 0x7ffa08004c40 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c001eb0 tx=0x7ffa080057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.757+0000 7ffa28a00640 1 -- 172.21.15.62:0/3878849827 shutdown_connections 2024-03-23T11:32:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.757+0000 7ffa28a00640 1 --2- 172.21.15.62:0/3878849827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 0x7ffa08004c40 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.757+0000 7ffa28a00640 1 -- 172.21.15.62:0/3878849827 >> 172.21.15.62:0/3878849827 conn(0x7ffa24089f60 msgr2=0x7ffa240d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.758+0000 7ffa28a00640 1 -- 172.21.15.62:0/3878849827 shutdown_connections 2024-03-23T11:32:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.758+0000 7ffa28a00640 1 -- 172.21.15.62:0/3878849827 wait complete. 2024-03-23T11:32:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.759+0000 7ffa28a00640 1 Processor -- start 2024-03-23T11:32:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.760+0000 7ffa28a00640 1 -- start start 2024-03-23T11:32:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa28a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 0x7ffa2409ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa28a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa2409c580 0x7ffa2409f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa28a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa2414d240 0x7ffa2414d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa28a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa241f1ec0 con 0x7ffa080047c0 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa28a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2414e1c0 con 0x7ffa2409c580 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa28a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2414ed40 con 0x7ffa2414d240 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa22800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 0x7ffa2409ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa22800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 0x7ffa2409ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55806/0 (socket says 172.21.15.62:55806) 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa22800640 1 -- 172.21.15.62:0/4171280963 learned_addr learned my addr 172.21.15.62:0/4171280963 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa19e00640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa2409c580 0x7ffa2409f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa22800640 1 -- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa2414d240 msgr2=0x7ffa2414d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa22800640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa2414d240 0x7ffa2414d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa22800640 1 -- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa2409c580 msgr2=0x7ffa2409f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa22800640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa2409c580 0x7ffa2409f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.761+0000 7ffa22800640 1 -- 172.21.15.62:0/4171280963 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2414e410 con 0x7ffa080047c0 2024-03-23T11:32:43.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.762+0000 7ffa22800640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 0x7ffa2409ef20 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400d430 tx=0x7ffa1400d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:43.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.762+0000 7ffa1b400640 1 -- 172.21.15.62:0/4171280963 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1400b070 con 0x7ffa080047c0 2024-03-23T11:32:43.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.762+0000 7ffa1b400640 1 -- 172.21.15.62:0/4171280963 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa140054e0 con 0x7ffa080047c0 2024-03-23T11:32:43.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.762+0000 7ffa28a00640 1 -- 172.21.15.62:0/4171280963 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2414e6f0 con 0x7ffa080047c0 2024-03-23T11:32:43.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.762+0000 7ffa1b400640 1 -- 172.21.15.62:0/4171280963 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14003810 con 0x7ffa080047c0 2024-03-23T11:32:43.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.762+0000 7ffa28a00640 1 -- 172.21.15.62:0/4171280963 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2414dc00 con 0x7ffa080047c0 2024-03-23T11:32:43.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.763+0000 7ffa28a00640 1 -- 172.21.15.62:0/4171280963 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa24066bb0 con 0x7ffa080047c0 2024-03-23T11:32:43.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.764+0000 7ffa1b400640 1 -- 172.21.15.62:0/4171280963 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ffa14003970 con 0x7ffa080047c0 2024-03-23T11:32:43.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.765+0000 7ffa1b400640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa0006a7a0 0x7ffa0006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:43.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.765+0000 7ffa1b400640 1 -- 172.21.15.62:0/4171280963 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ffa14088730 con 0x7ffa080047c0 2024-03-23T11:32:43.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.766+0000 7ffa19e00640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa0006a7a0 0x7ffa0006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:43.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.767+0000 7ffa19e00640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa0006a7a0 0x7ffa0006cc60 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c034000 tx=0x7ffa0c009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:43.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.769+0000 7ffa1b400640 1 -- 172.21.15.62:0/4171280963 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa14053b30 con 0x7ffa080047c0 2024-03-23T11:32:43.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.970+0000 7ffa28a00640 1 -- 172.21.15.62:0/4171280963 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa240cea40 con 0x7ffa080047c0 2024-03-23T11:32:43.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.971+0000 7ffa1b400640 1 -- 172.21.15.62:0/4171280963 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+16413 (secure 0 0 0) 0x7ffa14057150 con 0x7ffa080047c0 2024-03-23T11:32:43.962 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:43.963 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":122,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:32:23.551951+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-23T11:31:22.127079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-23T11:31:23.062387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":2}]},{"pgid":"17.1a","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6835/4129977589":"2024-03-24T11:32:23.551904+0000","172.21.15.62:6834/4129977589":"2024-03-24T11:32:23.551904+0000","172.21.15.120:6837/2585024035":"2024-03-24T11:32:04.365655+0000","172.21.15.120:6835/2585024035":"2024-03-24T11:32:04.365655+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:32:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.974+0000 7ffa28a00640 1 -- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa0006a7a0 msgr2=0x7ffa0006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.974+0000 7ffa28a00640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa0006a7a0 0x7ffa0006cc60 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c034000 tx=0x7ffa0c009030 comp rx=0 tx=0).stop 2024-03-23T11:32:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.974+0000 7ffa28a00640 1 -- 172.21.15.62:0/4171280963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 msgr2=0x7ffa2409ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.974+0000 7ffa28a00640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 0x7ffa2409ef20 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400d430 tx=0x7ffa1400d8f0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.974+0000 7ffa28a00640 1 -- 172.21.15.62:0/4171280963 shutdown_connections 2024-03-23T11:32:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.974+0000 7ffa28a00640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ffa0006a7a0 0x7ffa0006cc60 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.974+0000 7ffa28a00640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa2414d240 0x7ffa2414d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.974+0000 7ffa28a00640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa2409c580 0x7ffa2409f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.974+0000 7ffa28a00640 1 --2- 172.21.15.62:0/4171280963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa080047c0 0x7ffa2409ef20 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.974+0000 7ffa28a00640 1 -- 172.21.15.62:0/4171280963 >> 172.21.15.62:0/4171280963 conn(0x7ffa24089f60 msgr2=0x7ffa240d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.975+0000 7ffa28a00640 1 -- 172.21.15.62:0/4171280963 shutdown_connections 2024-03-23T11:32:43.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:43.975+0000 7ffa28a00640 1 -- 172.21.15.62:0/4171280963 wait complete. 2024-03-23T11:32:43.985 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T11:32:44.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.204+0000 7fc7b0a00640 1 Processor -- start 2024-03-23T11:32:44.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.204+0000 7fc7b0a00640 1 -- start start 2024-03-23T11:32:44.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.205+0000 7fc7b0a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac15c8c0 0x7fc7ac069c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.205+0000 7fc7b0a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc7ac0b0450 0x7fc7ac0b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.205+0000 7fc7b0a00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac0b0de0 con 0x7fc7ac0cb990 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.205+0000 7fc7b0a00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ac06aa00 con 0x7fc7ac0b0450 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.205+0000 7fc7b0a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac06a1a0 con 0x7fc7ac15c8c0 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.206+0000 7fc7aaa00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc7ac0cb990 0x7fc7ac160e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44052/0 (socket says 172.21.15.62:44052) 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.206+0000 7fc7aaa00640 1 -- 172.21.15.62:0/938664152 learned_addr learned my addr 172.21.15.62:0/938664152 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.206+0000 7fc7abe00640 1 --2- 172.21.15.62:0/938664152 >> v2:172.21.15.62:3300/0 conn(0x7fc7ac0b0450 0x7fc7ac0b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.206+0000 7fc7ab400640 1 --2- 172.21.15.62:0/938664152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac15c8c0 0x7fc7ac069c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.206+0000 7fc7aa000640 1 -- 172.21.15.62:0/938664152 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770065539 0 0) 0x7fc794001110 con 0x7fc7ac0cb990 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.206+0000 7fc7aa000640 1 -- 172.21.15.62:0/938664152 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc790003520 con 0x7fc7ac0cb990 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.206+0000 7fc7abe00640 1 -- 172.21.15.62:0/938664152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac15c8c0 msgr2=0x7fc7ac069c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.206+0000 7fc7abe00640 1 --2- 172.21.15.62:0/938664152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac15c8c0 0x7fc7ac069c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.206+0000 7fc7abe00640 1 -- 172.21.15.62:0/938664152 >> v1:172.21.15.62:6789/0 conn(0x7fc7ac0cb990 legacy=0x7fc7ac160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.206+0000 7fc7abe00640 1 -- 172.21.15.62:0/938664152 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac06a440 con 0x7fc7ac0b0450 2024-03-23T11:32:44.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.208+0000 7fc7abe00640 1 --2- 172.21.15.62:0/938664152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7ac0b0450 0x7fc7ac0b08a0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0097a0 tx=0x7fc79c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4e8a9993a14fab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:44.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.208+0000 7fc7aa000640 1 -- 172.21.15.62:0/938664152 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c02d040 con 0x7fc7ac0b0450 2024-03-23T11:32:44.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.208+0000 7fc7aa000640 1 -- 172.21.15.62:0/938664152 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c031810 con 0x7fc7ac0b0450 2024-03-23T11:32:44.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.208+0000 7fc7aa000640 1 -- 172.21.15.62:0/938664152 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c031ba0 con 0x7fc7ac0b0450 2024-03-23T11:32:44.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.209+0000 7fc7b0a00640 1 -- 172.21.15.62:0/938664152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7ac0b0450 msgr2=0x7fc7ac0b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:44.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.209+0000 7fc7b0a00640 1 --2- 172.21.15.62:0/938664152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7ac0b0450 0x7fc7ac0b08a0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0097a0 tx=0x7fc79c02b8b0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.209+0000 7fc7b0a00640 1 -- 172.21.15.62:0/938664152 shutdown_connections 2024-03-23T11:32:44.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.209+0000 7fc7b0a00640 1 --2- 172.21.15.62:0/938664152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7ac0b0450 0x7fc7ac0b08a0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.209+0000 7fc7b0a00640 1 --2- 172.21.15.62:0/938664152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac15c8c0 0x7fc7ac069c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.209+0000 7fc7b0a00640 1 -- 172.21.15.62:0/938664152 >> 172.21.15.62:0/938664152 conn(0x7fc7ac0989d0 msgr2=0x7fc7ac0cf580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.209+0000 7fc7b0a00640 1 -- 172.21.15.62:0/938664152 shutdown_connections 2024-03-23T11:32:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.209+0000 7fc7b0a00640 1 -- 172.21.15.62:0/938664152 wait complete. 2024-03-23T11:32:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.211+0000 7fc7b0a00640 1 Processor -- start 2024-03-23T11:32:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.211+0000 7fc7b0a00640 1 -- start start 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7b0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7ac0b0450 0x7fc7ac1e7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7b0a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac0cb990 0x7fc7ac0ac310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7b0a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ac15c8c0 0x7fc7ac155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7b0a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0ac850 con 0x7fc7ac0b0450 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7b0a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0aca60 con 0x7fc7ac15c8c0 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7b0a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0acd40 con 0x7fc7ac0cb990 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7ab400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7ac0b0450 0x7fc7ac1e7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7ab400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7ac0b0450 0x7fc7ac1e7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55810/0 (socket says 172.21.15.62:55810) 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7ab400640 1 -- 172.21.15.62:0/2843206853 learned_addr learned my addr 172.21.15.62:0/2843206853 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7aaa00640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac0cb990 0x7fc7ac0ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7abe00640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ac15c8c0 0x7fc7ac155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7ab400640 1 -- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac0cb990 msgr2=0x7fc7ac0ac310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7ab400640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac0cb990 0x7fc7ac0ac310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7ab400640 1 -- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ac15c8c0 msgr2=0x7fc7ac155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7ab400640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ac15c8c0 0x7fc7ac155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.212+0000 7fc7ab400640 1 -- 172.21.15.62:0/2843206853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c009400 con 0x7fc7ac0b0450 2024-03-23T11:32:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.213+0000 7fc7ab400640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7ac0b0450 0x7fc7ac1e7db0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400efc0 tx=0x7fc7a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.213+0000 7fc7a3400640 1 -- 172.21.15.62:0/2843206853 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4009b80 con 0x7fc7ac0b0450 2024-03-23T11:32:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.213+0000 7fc7a3400640 1 -- 172.21.15.62:0/2843206853 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a400f040 con 0x7fc7ac0b0450 2024-03-23T11:32:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.213+0000 7fc7b0a00640 1 -- 172.21.15.62:0/2843206853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac0ad020 con 0x7fc7ac0b0450 2024-03-23T11:32:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.213+0000 7fc7a3400640 1 -- 172.21.15.62:0/2843206853 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a40146a0 con 0x7fc7ac0b0450 2024-03-23T11:32:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.213+0000 7fc7b0a00640 1 -- 172.21.15.62:0/2843206853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac0ad5e0 con 0x7fc7ac0b0450 2024-03-23T11:32:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.214+0000 7fc7b0a00640 1 -- 172.21.15.62:0/2843206853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac0cd150 con 0x7fc7ac0b0450 2024-03-23T11:32:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.215+0000 7fc7a3400640 1 -- 172.21.15.62:0/2843206853 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc7a4014800 con 0x7fc7ac0b0450 2024-03-23T11:32:44.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.216+0000 7fc7a3400640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc78806a7a0 0x7fc78806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.216+0000 7fc7a3400640 1 -- 172.21.15.62:0/2843206853 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fc7a4089a00 con 0x7fc7ac0b0450 2024-03-23T11:32:44.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.216+0000 7fc7aaa00640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc78806a7a0 0x7fc78806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:44.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.217+0000 7fc7aaa00640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc78806a7a0 0x7fc78806cc60 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0ccf20 tx=0x7fc794009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:44.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.220+0000 7fc7a3400640 1 -- 172.21.15.62:0/2843206853 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc7a4054d20 con 0x7fc7ac0b0450 2024-03-23T11:32:44.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.455+0000 7fc7b0a00640 1 -- 172.21.15.62:0/2843206853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc7ac0672c0 con 0x7fc7ac0b0450 2024-03-23T11:32:44.446 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-03-23T11:32:44.456+0000 7f91fe000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T11:32:44.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.475+0000 7fc7a3400640 1 -- 172.21.15.62:0/2843206853 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v108) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc7a4058340 con 0x7fc7ac0b0450 2024-03-23T11:32:44.466 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T11:32:44.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 -- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc78806a7a0 msgr2=0x7fc78806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:44.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc78806a7a0 0x7fc78806cc60 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0ccf20 tx=0x7fc794009040 comp rx=0 tx=0).stop 2024-03-23T11:32:44.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 -- 172.21.15.62:0/2843206853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7ac0b0450 msgr2=0x7fc7ac1e7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:44.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7ac0b0450 0x7fc7ac1e7db0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400efc0 tx=0x7fc7a400c490 comp rx=0 tx=0).stop 2024-03-23T11:32:44.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 -- 172.21.15.62:0/2843206853 shutdown_connections 2024-03-23T11:32:44.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc78806a7a0 0x7fc78806cc60 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7ac15c8c0 0x7fc7ac155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc7ac0cb990 0x7fc7ac0ac310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 --2- 172.21.15.62:0/2843206853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7ac0b0450 0x7fc7ac1e7db0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 -- 172.21.15.62:0/2843206853 >> 172.21.15.62:0/2843206853 conn(0x7fc7ac0989d0 msgr2=0x7fc7ac06d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:44.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 -- 172.21.15.62:0/2843206853 shutdown_connections 2024-03-23T11:32:44.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.478+0000 7fc7b0a00640 1 -- 172.21.15.62:0/2843206853 wait complete. 2024-03-23T11:32:44.485 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T11:32:44.541 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-03-23T11:32:44.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.677+0000 7f8128400640 1 Processor -- start 2024-03-23T11:32:44.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.677+0000 7f8128400640 1 -- start start 2024-03-23T11:32:44.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.679+0000 7f8128400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f812015cc20 0x7f81200b4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.679+0000 7f8128400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81200b46c0 0x7f812006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.679+0000 7f8128400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81200b0000 con 0x7f8120070260 2024-03-23T11:32:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.679+0000 7f8128400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8120070710 con 0x7f812015cc20 2024-03-23T11:32:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.679+0000 7f8128400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812006afa0 con 0x7f81200b46c0 2024-03-23T11:32:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.680+0000 7f8125a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f812015cc20 0x7f81200b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.680+0000 7f8125000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81200b46c0 0x7f812006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.680+0000 7f8125a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f812015cc20 0x7f81200b4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.62:57160/0 (socket says 172.21.15.62:57160) 2024-03-23T11:32:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.680+0000 7f8125000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81200b46c0 0x7f812006fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.62:60656/0 (socket says 172.21.15.62:60656) 2024-03-23T11:32:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.680+0000 7f8125a00640 1 -- 172.21.15.62:0/177822862 learned_addr learned my addr 172.21.15.62:0/177822862 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.680+0000 7f8125000640 1 -- 172.21.15.62:0/177822862 >> v2:172.21.15.120:3300/0 conn(0x7f812015cc20 msgr2=0x7f81200b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.680+0000 7f8125000640 1 --2- 172.21.15.62:0/177822862 >> v2:172.21.15.120:3300/0 conn(0x7f812015cc20 0x7f81200b4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.680+0000 7f8125000640 1 -- 172.21.15.62:0/177822862 >> v1:172.21.15.120:6789/0 conn(0x7f8120070260 legacy=0x7f812006b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.680+0000 7f8125000640 1 -- 172.21.15.62:0/177822862 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81200af760 con 0x7f81200b46c0 2024-03-23T11:32:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.680+0000 7f8125a00640 1 --2- 172.21.15.62:0/177822862 >> v2:172.21.15.120:3300/0 conn(0x7f812015cc20 0x7f81200b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T11:32:44.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.682+0000 7f8125000640 1 --2- 172.21.15.62:0/177822862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81200b46c0 0x7f812006fd20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f810c0097a0 tx=0x7f810c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9ed4107a6dfb6d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:44.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.682+0000 7f811fe00640 1 -- 172.21.15.62:0/177822862 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c02d040 con 0x7f81200b46c0 2024-03-23T11:32:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.682+0000 7f811fe00640 1 -- 172.21.15.62:0/177822862 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810c031810 con 0x7f81200b46c0 2024-03-23T11:32:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.682+0000 7f811fe00640 1 -- 172.21.15.62:0/177822862 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c031ba0 con 0x7f81200b46c0 2024-03-23T11:32:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.683+0000 7f8128400640 1 -- 172.21.15.62:0/177822862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81200b46c0 msgr2=0x7f812006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.683+0000 7f8128400640 1 --2- 172.21.15.62:0/177822862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81200b46c0 0x7f812006fd20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f810c0097a0 tx=0x7f810c02b8b0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.683+0000 7f8128400640 1 -- 172.21.15.62:0/177822862 shutdown_connections 2024-03-23T11:32:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.683+0000 7f8128400640 1 --2- 172.21.15.62:0/177822862 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f81200b46c0 0x7f812006fd20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.683+0000 7f8128400640 1 --2- 172.21.15.62:0/177822862 >> v2:172.21.15.120:3300/0 conn(0x7f812015cc20 0x7f81200b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.683+0000 7f8128400640 1 -- 172.21.15.62:0/177822862 >> 172.21.15.62:0/177822862 conn(0x7f812006e580 msgr2=0x7f812006e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.683+0000 7f8128400640 1 -- 172.21.15.62:0/177822862 shutdown_connections 2024-03-23T11:32:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.683+0000 7f8128400640 1 -- 172.21.15.62:0/177822862 wait complete. 2024-03-23T11:32:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.685+0000 7f8128400640 1 Processor -- start 2024-03-23T11:32:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.685+0000 7f8128400640 1 -- start start 2024-03-23T11:32:44.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8128400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8120070260 0x7f81201452d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8128400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81200b46c0 0x7f812014fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8128400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f812015cc20 0x7f81200cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8128400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81200cba80 con 0x7f81200b46c0 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8128400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81200cae70 con 0x7f812015cc20 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8128400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81200cb150 con 0x7f8120070260 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8125000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81200b46c0 0x7f812014fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8125000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81200b46c0 0x7f812014fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55850/0 (socket says 172.21.15.62:55850) 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8125000640 1 -- 172.21.15.62:0/2533059214 learned_addr learned my addr 172.21.15.62:0/2533059214 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8126400640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f812015cc20 0x7f81200cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8125000640 1 -- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8120070260 msgr2=0x7f81201452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8125000640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8120070260 0x7f81201452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.686+0000 7f8125000640 1 -- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f812015cc20 msgr2=0x7f81200cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:44.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.687+0000 7f8125000640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f812015cc20 0x7f81200cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:44.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.687+0000 7f8125000640 1 -- 172.21.15.62:0/2533059214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8114009b30 con 0x7f81200b46c0 2024-03-23T11:32:44.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.687+0000 7f8126400640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f812015cc20 0x7f81200cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:44.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.687+0000 7f8125000640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81200b46c0 0x7f812014fd70 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f810c0098c0 tx=0x7f810c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:44.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.687+0000 7f811d600640 1 -- 172.21.15.62:0/2533059214 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c006e10 con 0x7f81200b46c0 2024-03-23T11:32:44.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.687+0000 7f811d600640 1 -- 172.21.15.62:0/2533059214 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810c03ad60 con 0x7f81200b46c0 2024-03-23T11:32:44.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.687+0000 7f811d600640 1 -- 172.21.15.62:0/2533059214 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c044620 con 0x7f81200b46c0 2024-03-23T11:32:44.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.687+0000 7f8128400640 1 -- 172.21.15.62:0/2533059214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810c009400 con 0x7f81200b46c0 2024-03-23T11:32:44.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.687+0000 7f8128400640 1 -- 172.21.15.62:0/2533059214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81200cb710 con 0x7f81200b46c0 2024-03-23T11:32:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.688+0000 7f8128400640 1 -- 172.21.15.62:0/2533059214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8120071cf0 con 0x7f81200b46c0 2024-03-23T11:32:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.689+0000 7f811d600640 1 -- 172.21.15.62:0/2533059214 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f810c04d440 con 0x7f81200b46c0 2024-03-23T11:32:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.690+0000 7f811d600640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f810006a7d0 0x7f810006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.690+0000 7f811d600640 1 -- 172.21.15.62:0/2533059214 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f810c0ab220 con 0x7f81200b46c0 2024-03-23T11:32:44.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.691+0000 7f8125a00640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f810006a7d0 0x7f810006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:44.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.692+0000 7f8125a00640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f810006a7d0 0x7f810006cc90 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f811400e330 tx=0x7f811400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:44.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.694+0000 7f811d600640 1 -- 172.21.15.62:0/2533059214 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f810c076460 con 0x7f81200b46c0 2024-03-23T11:32:44.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:44.929+0000 7f8128400640 1 -- 172.21.15.62:0/2533059214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f81200672c0 con 0x7f81200b46c0 2024-03-23T11:32:45.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.510+0000 7f811d600640 1 -- 172.21.15.62:0/2533059214 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v109) v1 ==== 104+0+0 (secure 0 0 0) 0x7f810c079a80 con 0x7f81200b46c0 2024-03-23T11:32:45.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.514+0000 7f8128400640 1 -- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f810006a7d0 msgr2=0x7f810006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:45.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.514+0000 7f8128400640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f810006a7d0 0x7f810006cc90 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f811400e330 tx=0x7f811400b040 comp rx=0 tx=0).stop 2024-03-23T11:32:45.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.514+0000 7f8128400640 1 -- 172.21.15.62:0/2533059214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81200b46c0 msgr2=0x7f812014fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:45.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.514+0000 7f8128400640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81200b46c0 0x7f812014fd70 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f810c0098c0 tx=0x7f810c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:32:45.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.515+0000 7f8128400640 1 -- 172.21.15.62:0/2533059214 shutdown_connections 2024-03-23T11:32:45.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.515+0000 7f8128400640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f810006a7d0 0x7f810006cc90 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:45.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.515+0000 7f8128400640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81200b46c0 0x7f812014fd70 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:45.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.515+0000 7f8128400640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f812015cc20 0x7f81200cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:45.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.515+0000 7f8128400640 1 --2- 172.21.15.62:0/2533059214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8120070260 0x7f81201452d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:45.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.515+0000 7f8128400640 1 -- 172.21.15.62:0/2533059214 >> 172.21.15.62:0/2533059214 conn(0x7f812006e580 msgr2=0x7f81200cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:45.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.515+0000 7f8128400640 1 -- 172.21.15.62:0/2533059214 shutdown_connections 2024-03-23T11:32:45.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.515+0000 7f8128400640 1 -- 172.21.15.62:0/2533059214 wait complete. 2024-03-23T11:32:45.527 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T11:32:45.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.761+0000 7fac29e00640 1 Processor -- start 2024-03-23T11:32:45.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.761+0000 7fac29e00640 1 -- start start 2024-03-23T11:32:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.763+0000 7fac29e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fac24142e90 0x7fac24159d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.763+0000 7fac29e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fac24143270 0x7fac240c38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.763+0000 7fac29e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac240c4310 con 0x7fac240c3e00 2024-03-23T11:32:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.763+0000 7fac29e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac2406cf70 con 0x7fac24143270 2024-03-23T11:32:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.763+0000 7fac29e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac2406c8b0 con 0x7fac24142e90 2024-03-23T11:32:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.764+0000 7fac22000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fac24143270 0x7fac240c38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.764+0000 7fac22000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fac24143270 0x7fac240c38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55864/0 (socket says 172.21.15.62:55864) 2024-03-23T11:32:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.764+0000 7fac22000640 1 -- 172.21.15.62:0/2737764001 learned_addr learned my addr 172.21.15.62:0/2737764001 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.764+0000 7fac22a00640 1 --2- 172.21.15.62:0/2737764001 >> v2:172.21.15.120:3300/0 conn(0x7fac24142e90 0x7fac24159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.764+0000 7fac22000640 1 -- 172.21.15.62:0/2737764001 >> v2:172.21.15.120:3300/0 conn(0x7fac24142e90 msgr2=0x7fac24159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.764+0000 7fac22000640 1 --2- 172.21.15.62:0/2737764001 >> v2:172.21.15.120:3300/0 conn(0x7fac24142e90 0x7fac24159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.764+0000 7fac22000640 1 -- 172.21.15.62:0/2737764001 >> v1:172.21.15.120:6789/0 conn(0x7fac240c3e00 legacy=0x7fac240cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.764+0000 7fac22000640 1 -- 172.21.15.62:0/2737764001 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2406cb90 con 0x7fac24143270 2024-03-23T11:32:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.765+0000 7fac22000640 1 --2- 172.21.15.62:0/2737764001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac24143270 0x7fac240c38c0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fac240b2790 tx=0x7fac1002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e206ff96eadbd3c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.766+0000 7fac21600640 1 -- 172.21.15.62:0/2737764001 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac10031750 con 0x7fac24143270 2024-03-23T11:32:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.766+0000 7fac21600640 1 -- 172.21.15.62:0/2737764001 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac100318b0 con 0x7fac24143270 2024-03-23T11:32:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.766+0000 7fac21600640 1 -- 172.21.15.62:0/2737764001 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac10031c40 con 0x7fac24143270 2024-03-23T11:32:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.766+0000 7fac29e00640 1 -- 172.21.15.62:0/2737764001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac24143270 msgr2=0x7fac240c38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.766+0000 7fac29e00640 1 --2- 172.21.15.62:0/2737764001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac24143270 0x7fac240c38c0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fac240b2790 tx=0x7fac1002b7f0 comp rx=0 tx=0).stop 2024-03-23T11:32:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.766+0000 7fac29e00640 1 -- 172.21.15.62:0/2737764001 shutdown_connections 2024-03-23T11:32:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.766+0000 7fac29e00640 1 --2- 172.21.15.62:0/2737764001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac24143270 0x7fac240c38c0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.766+0000 7fac29e00640 1 --2- 172.21.15.62:0/2737764001 >> v2:172.21.15.120:3300/0 conn(0x7fac24142e90 0x7fac24159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.766+0000 7fac29e00640 1 -- 172.21.15.62:0/2737764001 >> 172.21.15.62:0/2737764001 conn(0x7fac240b17d0 msgr2=0x7fac240b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:45.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.767+0000 7fac29e00640 1 -- 172.21.15.62:0/2737764001 shutdown_connections 2024-03-23T11:32:45.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.767+0000 7fac29e00640 1 -- 172.21.15.62:0/2737764001 wait complete. 2024-03-23T11:32:45.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.768+0000 7fac29e00640 1 Processor -- start 2024-03-23T11:32:45.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.769+0000 7fac29e00640 1 -- start start 2024-03-23T11:32:45.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.769+0000 7fac29e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac240c3e00 0x7fac241f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:45.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac29e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac24142e90 0x7fac24159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:45.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac29e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac24143270 0x7fac241401b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:45.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac29e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2415af50 con 0x7fac240c3e00 2024-03-23T11:32:45.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac29e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac241f6540 con 0x7fac24142e90 2024-03-23T11:32:45.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac29e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2415a340 con 0x7fac24143270 2024-03-23T11:32:45.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac22a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac240c3e00 0x7fac241f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:45.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac22a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac240c3e00 0x7fac241f6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55874/0 (socket says 172.21.15.62:55874) 2024-03-23T11:32:45.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac22a00640 1 -- 172.21.15.62:0/3237865752 learned_addr learned my addr 172.21.15.62:0/3237865752 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:45.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac22a00640 1 -- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac24143270 msgr2=0x7fac241401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:45.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac22a00640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac24143270 0x7fac241401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:45.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac22a00640 1 -- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac24142e90 msgr2=0x7fac24159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:45.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac22a00640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac24142e90 0x7fac24159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:45.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.770+0000 7fac22a00640 1 -- 172.21.15.62:0/3237865752 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac100094d0 con 0x7fac240c3e00 2024-03-23T11:32:45.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.771+0000 7fac22a00640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac240c3e00 0x7fac241f6000 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fac1400d800 tx=0x7fac1400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:45.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.771+0000 7fac1b400640 1 -- 172.21.15.62:0/3237865752 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1400b6c0 con 0x7fac240c3e00 2024-03-23T11:32:45.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.771+0000 7fac1b400640 1 -- 172.21.15.62:0/3237865752 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1400b820 con 0x7fac240c3e00 2024-03-23T11:32:45.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.771+0000 7fac1b400640 1 -- 172.21.15.62:0/3237865752 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1400bb90 con 0x7fac240c3e00 2024-03-23T11:32:45.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.771+0000 7fac29e00640 1 -- 172.21.15.62:0/3237865752 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2415a620 con 0x7fac240c3e00 2024-03-23T11:32:45.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.771+0000 7fac29e00640 1 -- 172.21.15.62:0/3237865752 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2415abe0 con 0x7fac240c3e00 2024-03-23T11:32:45.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.772+0000 7fac29e00640 1 -- 172.21.15.62:0/3237865752 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac24066c90 con 0x7fac240c3e00 2024-03-23T11:32:45.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.773+0000 7fac1b400640 1 -- 172.21.15.62:0/3237865752 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fac1400bcf0 con 0x7fac240c3e00 2024-03-23T11:32:45.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.774+0000 7fac1b400640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac0406a7f0 0x7fac0406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:45.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.774+0000 7fac1b400640 1 -- 172.21.15.62:0/3237865752 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fac140894c0 con 0x7fac240c3e00 2024-03-23T11:32:45.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.775+0000 7fac22000640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac0406a7f0 0x7fac0406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:45.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.776+0000 7fac22000640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac0406a7f0 0x7fac0406ccb0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fac10035000 tx=0x7fac10030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:45.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.780+0000 7fac1b400640 1 -- 172.21.15.62:0/3237865752 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac14054780 con 0x7fac240c3e00 2024-03-23T11:32:45.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:45.987+0000 7fac29e00640 1 -- 172.21.15.62:0/3237865752 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fac240ce400 con 0x7fac240c3e00 2024-03-23T11:32:46.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.525+0000 7fac1b400640 1 -- 172.21.15.62:0/3237865752 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v124) v1 ==== 181+0+0 (secure 0 0 0) 0x7fac14057da0 con 0x7fac240c3e00 2024-03-23T11:32:46.516 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' removed 2024-03-23T11:32:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.529+0000 7fac29e00640 1 -- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac0406a7f0 msgr2=0x7fac0406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.529+0000 7fac29e00640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac0406a7f0 0x7fac0406ccb0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fac10035000 tx=0x7fac10030000 comp rx=0 tx=0).stop 2024-03-23T11:32:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.529+0000 7fac29e00640 1 -- 172.21.15.62:0/3237865752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac240c3e00 msgr2=0x7fac241f6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.529+0000 7fac29e00640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac240c3e00 0x7fac241f6000 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fac1400d800 tx=0x7fac1400dcc0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.529+0000 7fac29e00640 1 -- 172.21.15.62:0/3237865752 shutdown_connections 2024-03-23T11:32:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.529+0000 7fac29e00640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fac0406a7f0 0x7fac0406ccb0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.529+0000 7fac29e00640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fac24143270 0x7fac241401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.529+0000 7fac29e00640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fac24142e90 0x7fac24159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.529+0000 7fac29e00640 1 --2- 172.21.15.62:0/3237865752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac240c3e00 0x7fac241f6000 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.529+0000 7fac29e00640 1 -- 172.21.15.62:0/3237865752 >> 172.21.15.62:0/3237865752 conn(0x7fac240b17d0 msgr2=0x7fac240b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.530+0000 7fac29e00640 1 -- 172.21.15.62:0/3237865752 shutdown_connections 2024-03-23T11:32:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.530+0000 7fac29e00640 1 -- 172.21.15.62:0/3237865752 wait complete. 2024-03-23T11:32:46.548 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T11:32:46.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.768+0000 7f51de800640 1 Processor -- start 2024-03-23T11:32:46.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.769+0000 7f51de800640 1 -- start start 2024-03-23T11:32:46.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.770+0000 7f51de800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f51d815a8a0 0x7f51d8070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:46.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.770+0000 7f51de800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d80b07f0 0x7f51d80b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:46.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.770+0000 7f51de800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f51d80cd1a0 0x7f51d80cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:46.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.770+0000 7f51de800640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d80b1190 con 0x7f51d815a8a0 2024-03-23T11:32:46.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.770+0000 7f51de800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d80cdb70 con 0x7f51d80cd1a0 2024-03-23T11:32:46.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.770+0000 7f51de800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d806d610 con 0x7f51d80b07f0 2024-03-23T11:32:46.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.770+0000 7f51d7400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f51d815a8a0 0x7f51d8070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:46.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.770+0000 7f51d7400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f51d815a8a0 0x7f51d8070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55876/0 (socket says 172.21.15.62:55876) 2024-03-23T11:32:46.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.770+0000 7f51d7400640 1 -- 172.21.15.62:0/2208825295 learned_addr learned my addr 172.21.15.62:0/2208825295 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.771+0000 7f51d7e00640 1 --2- 172.21.15.62:0/2208825295 >> v2:172.21.15.120:3300/0 conn(0x7f51d80cd1a0 0x7f51d80cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.771+0000 7f51cfe00640 1 --2- 172.21.15.62:0/2208825295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d80b07f0 0x7f51d80b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.771+0000 7f51d7400640 1 -- 172.21.15.62:0/2208825295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d80b07f0 msgr2=0x7f51d80b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.771+0000 7f51d7400640 1 --2- 172.21.15.62:0/2208825295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d80b07f0 0x7f51d80b0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.771+0000 7f51d7400640 1 -- 172.21.15.62:0/2208825295 >> v2:172.21.15.120:3300/0 conn(0x7f51d80cd1a0 msgr2=0x7f51d80cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:46.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.771+0000 7f51d7400640 1 --2- 172.21.15.62:0/2208825295 >> v2:172.21.15.120:3300/0 conn(0x7f51d80cd1a0 0x7f51d80cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.771+0000 7f51d7400640 1 -- 172.21.15.62:0/2208825295 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d806d8f0 con 0x7f51d815a8a0 2024-03-23T11:32:46.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.772+0000 7f51d7400640 1 --2- 172.21.15.62:0/2208825295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d815a8a0 0x7f51d8070270 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f51d806e420 tx=0x7f51c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6565949c4a9238e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:46.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51d6a00640 1 -- 172.21.15.62:0/2208825295 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c80317e0 con 0x7f51d815a8a0 2024-03-23T11:32:46.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51d6a00640 1 -- 172.21.15.62:0/2208825295 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c8031940 con 0x7f51d815a8a0 2024-03-23T11:32:46.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51d6a00640 1 -- 172.21.15.62:0/2208825295 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8031cd0 con 0x7f51d815a8a0 2024-03-23T11:32:46.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51de800640 1 -- 172.21.15.62:0/2208825295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d815a8a0 msgr2=0x7f51d8070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:46.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51de800640 1 --2- 172.21.15.62:0/2208825295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d815a8a0 0x7f51d8070270 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f51d806e420 tx=0x7f51c802b880 comp rx=0 tx=0).stop 2024-03-23T11:32:46.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51de800640 1 -- 172.21.15.62:0/2208825295 shutdown_connections 2024-03-23T11:32:46.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51de800640 1 --2- 172.21.15.62:0/2208825295 >> v2:172.21.15.120:3300/0 conn(0x7f51d80cd1a0 0x7f51d80cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51de800640 1 --2- 172.21.15.62:0/2208825295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d80b07f0 0x7f51d80b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51de800640 1 --2- 172.21.15.62:0/2208825295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d815a8a0 0x7f51d8070270 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51de800640 1 -- 172.21.15.62:0/2208825295 >> 172.21.15.62:0/2208825295 conn(0x7f51d80b17d0 msgr2=0x7f51d80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:46.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51de800640 1 -- 172.21.15.62:0/2208825295 shutdown_connections 2024-03-23T11:32:46.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.773+0000 7f51de800640 1 -- 172.21.15.62:0/2208825295 wait complete. 2024-03-23T11:32:46.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.775+0000 7f51de800640 1 Processor -- start 2024-03-23T11:32:46.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.775+0000 7f51de800640 1 -- start start 2024-03-23T11:32:46.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.776+0000 7f51de800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d80b07f0 0x7f51d8149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:46.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.776+0000 7f51de800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d80cd1a0 0x7f51d814ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:46.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.776+0000 7f51de800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d815a8a0 0x7f51d8154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:46.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.776+0000 7f51de800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d81f4880 con 0x7f51d815a8a0 2024-03-23T11:32:46.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.776+0000 7f51de800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d81f45a0 con 0x7f51d80cd1a0 2024-03-23T11:32:46.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.776+0000 7f51de800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d814a280 con 0x7f51d80b07f0 2024-03-23T11:32:46.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.776+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d815a8a0 0x7f51d8154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:46.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.776+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d815a8a0 0x7f51d8154710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55882/0 (socket says 172.21.15.62:55882) 2024-03-23T11:32:46.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.777+0000 7f51d7e00640 1 -- 172.21.15.62:0/836135154 learned_addr learned my addr 172.21.15.62:0/836135154 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:46.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.777+0000 7f51cfe00640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d80cd1a0 0x7f51d814ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:46.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.777+0000 7f51d7400640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d80b07f0 0x7f51d8149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:46.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.777+0000 7f51d7e00640 1 -- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d80b07f0 msgr2=0x7f51d8149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:46.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.777+0000 7f51d7e00640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d80b07f0 0x7f51d8149890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.777+0000 7f51d7e00640 1 -- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d80cd1a0 msgr2=0x7f51d814ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:46.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.777+0000 7f51d7e00640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d80cd1a0 0x7f51d814ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:46.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.777+0000 7f51d7e00640 1 -- 172.21.15.62:0/836135154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c80094d0 con 0x7f51d815a8a0 2024-03-23T11:32:46.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.777+0000 7f51d7e00640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d815a8a0 0x7f51d8154710 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f51d0007e40 tx=0x7f51d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:46.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.778+0000 7f51cf400640 1 -- 172.21.15.62:0/836135154 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0014620 con 0x7f51d815a8a0 2024-03-23T11:32:46.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.778+0000 7f51cf400640 1 -- 172.21.15.62:0/836135154 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d000ee90 con 0x7f51d815a8a0 2024-03-23T11:32:46.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.778+0000 7f51cf400640 1 -- 172.21.15.62:0/836135154 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d00149e0 con 0x7f51d815a8a0 2024-03-23T11:32:46.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.778+0000 7f51de800640 1 -- 172.21.15.62:0/836135154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d814a560 con 0x7f51d815a8a0 2024-03-23T11:32:46.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.778+0000 7f51de800640 1 -- 172.21.15.62:0/836135154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d8149dd0 con 0x7f51d815a8a0 2024-03-23T11:32:46.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.779+0000 7f51de800640 1 -- 172.21.15.62:0/836135154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d814a0e0 con 0x7f51d815a8a0 2024-03-23T11:32:46.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.780+0000 7f51cf400640 1 -- 172.21.15.62:0/836135154 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f51d0014b40 con 0x7f51d815a8a0 2024-03-23T11:32:46.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.781+0000 7f51cf400640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b406a7a0 0x7f51b406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:46.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.781+0000 7f51cf400640 1 -- 172.21.15.62:0/836135154 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5587+0+0 (secure 0 0 0) 0x7f51d00897a0 con 0x7f51d815a8a0 2024-03-23T11:32:46.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.781+0000 7f51d7400640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b406a7a0 0x7f51b406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:46.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.783+0000 7f51d7400640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b406a7a0 0x7f51b406cc60 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f51c8036000 tx=0x7f51c803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:46.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.786+0000 7f51cf400640 1 -- 172.21.15.62:0/836135154 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51d814a0e0 con 0x7f51d815a8a0 2024-03-23T11:32:46.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:46.985+0000 7f51de800640 1 -- 172.21.15.62:0/836135154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f51d814a0e0 con 0x7f51d815a8a0 2024-03-23T11:32:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.550+0000 7f51cf400640 1 -- 172.21.15.62:0/836135154 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v125) v1 ==== 169+0+0 (secure 0 0 0) 0x7f51d814a0e0 con 0x7f51d815a8a0 2024-03-23T11:32:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' removed 2024-03-23T11:32:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.555+0000 7f51de800640 1 -- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b406a7a0 msgr2=0x7f51b406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.555+0000 7f51de800640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b406a7a0 0x7f51b406cc60 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f51c8036000 tx=0x7f51c803d040 comp rx=0 tx=0).stop 2024-03-23T11:32:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.556+0000 7f51de800640 1 -- 172.21.15.62:0/836135154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d815a8a0 msgr2=0x7f51d8154710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.556+0000 7f51de800640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d815a8a0 0x7f51d8154710 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f51d0007e40 tx=0x7f51d000e4b0 comp rx=0 tx=0).stop 2024-03-23T11:32:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.556+0000 7f51de800640 1 -- 172.21.15.62:0/836135154 shutdown_connections 2024-03-23T11:32:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.556+0000 7f51de800640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f51b406a7a0 0x7f51b406cc60 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.556+0000 7f51de800640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d815a8a0 0x7f51d8154710 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.556+0000 7f51de800640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d80cd1a0 0x7f51d814ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.556+0000 7f51de800640 1 --2- 172.21.15.62:0/836135154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51d80b07f0 0x7f51d8149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.556+0000 7f51de800640 1 -- 172.21.15.62:0/836135154 >> 172.21.15.62:0/836135154 conn(0x7f51d80b17d0 msgr2=0x7f51d80cb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.556+0000 7f51de800640 1 -- 172.21.15.62:0/836135154 shutdown_connections 2024-03-23T11:32:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.556+0000 7f51de800640 1 -- 172.21.15.62:0/836135154 wait complete. 2024-03-23T11:32:47.569 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T11:32:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.794+0000 7f7a87e00640 1 Processor -- start 2024-03-23T11:32:47.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.794+0000 7f7a87e00640 1 -- start start 2024-03-23T11:32:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.795+0000 7f7a87e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7a8815d060 0x7f7a8806a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.795+0000 7f7a87e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a880b14c0 con 0x7f7a881593e0 2024-03-23T11:32:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.795+0000 7f7a87e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8815dee0 con 0x7f7a880b2b30 2024-03-23T11:32:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.795+0000 7f7a87e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8806a020 con 0x7f7a8815d060 2024-03-23T11:32:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.796+0000 7f7a86a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7a881593e0 0x7f7a880cc100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35500/0 (socket says 172.21.15.62:35500) 2024-03-23T11:32:47.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.796+0000 7f7a86a00640 1 -- 172.21.15.62:0/3831163939 learned_addr learned my addr 172.21.15.62:0/3831163939 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:47.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.796+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94333106 0 0) 0x7f7a880b14c0 con 0x7f7a881593e0 2024-03-23T11:32:47.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.796+0000 7f7a86000640 1 --2- 172.21.15.62:0/3831163939 >> v2:172.21.15.120:3300/0 conn(0x7f7a8815d060 0x7f7a8806a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:47.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.796+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a6c005730 con 0x7f7a881593e0 2024-03-23T11:32:47.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 895656074 0 0) 0x7f7a6c005730 con 0x7f7a881593e0 2024-03-23T11:32:47.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 >> v2:172.21.15.120:3300/0 conn(0x7f7a8815d060 msgr2=0x7f7a8806a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:47.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 --2- 172.21.15.62:0/3831163939 >> v2:172.21.15.120:3300/0 conn(0x7f7a8815d060 0x7f7a8806a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:47.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 >> v1:172.21.15.120:6789/0 conn(0x7f7a880b2b30 legacy=0x7f7a880b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:47.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8815d640 con 0x7f7a881593e0 2024-03-23T11:32:47.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f7a78001670 con 0x7f7a881593e0 2024-03-23T11:32:47.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 >> v1:172.21.15.62:6789/0 conn(0x7f7a881593e0 legacy=0x7f7a880cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:47.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 --2- 172.21.15.62:0/3831163939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a6c005b80 0x7f7a6c005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 --2- 172.21.15.62:0/3831163939 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a6c0069d0 0x7f7a6c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 --2- 172.21.15.62:0/3831163939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c007390 0x7f7a6c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c00e290 con 0x7f7a6c007390 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c00e400 con 0x7f7a6c005b80 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c00e5c0 con 0x7f7a6c0069d0 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a87400640 1 --2- 172.21.15.62:0/3831163939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c007390 0x7f7a6c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.797+0000 7f7a86a00640 1 --2- 172.21.15.62:0/3831163939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a6c005b80 0x7f7a6c005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.798+0000 7f7a87400640 1 -- 172.21.15.62:0/3831163939 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a6c0069d0 msgr2=0x7f7a6c006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.798+0000 7f7a87400640 1 --2- 172.21.15.62:0/3831163939 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a6c0069d0 0x7f7a6c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.798+0000 7f7a87400640 1 -- 172.21.15.62:0/3831163939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a6c005b80 msgr2=0x7f7a6c005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.798+0000 7f7a87400640 1 --2- 172.21.15.62:0/3831163939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a6c005b80 0x7f7a6c005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:47.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.798+0000 7f7a87400640 1 -- 172.21.15.62:0/3831163939 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c005730 con 0x7f7a6c007390 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.799+0000 7f7a87400640 1 -- 172.21.15.62:0/3831163939 reap_dead start 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.799+0000 7f7a87400640 1 --2- 172.21.15.62:0/3831163939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c007390 0x7f7a6c00db90 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f7a80004480 tx=0x7f7a6c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3397d5ed9c8f18e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.799+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80006b50 con 0x7f7a6c007390 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.799+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a80006cd0 con 0x7f7a6c007390 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.799+0000 7f7a85600640 1 -- 172.21.15.62:0/3831163939 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80007260 con 0x7f7a6c007390 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.800+0000 7f7a87e00640 1 -- 172.21.15.62:0/3831163939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c007390 msgr2=0x7f7a6c00db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.800+0000 7f7a87e00640 1 --2- 172.21.15.62:0/3831163939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c007390 0x7f7a6c00db90 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f7a80004480 tx=0x7f7a6c007a00 comp rx=0 tx=0).stop 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.800+0000 7f7a87e00640 1 -- 172.21.15.62:0/3831163939 shutdown_connections 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.800+0000 7f7a87e00640 1 --2- 172.21.15.62:0/3831163939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c007390 0x7f7a6c00db90 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.800+0000 7f7a87e00640 1 -- 172.21.15.62:0/3831163939 >> 172.21.15.62:0/3831163939 conn(0x7f7a880b0090 msgr2=0x7f7a880b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.800+0000 7f7a87e00640 1 -- 172.21.15.62:0/3831163939 shutdown_connections 2024-03-23T11:32:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.800+0000 7f7a87e00640 1 -- 172.21.15.62:0/3831163939 wait complete. 2024-03-23T11:32:47.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.802+0000 7f7a87e00640 1 Processor -- start 2024-03-23T11:32:47.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.802+0000 7f7a87e00640 1 -- start start 2024-03-23T11:32:47.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a87e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a6c007390 0x7f7a881f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:47.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a87e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a881f55a0 0x7f7a880c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:47.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a87e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a88155cf0 0x7f7a88156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:47.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a87e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a88159b20 con 0x7f7a881f55a0 2024-03-23T11:32:47.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a87e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a881566c0 con 0x7f7a88155cf0 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a87e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a881f2f70 con 0x7f7a6c007390 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a86000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a881f55a0 0x7f7a880c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a86000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a881f55a0 0x7f7a880c9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39576/0 (socket says 172.21.15.62:39576) 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a86000640 1 -- 172.21.15.62:0/3418829407 learned_addr learned my addr 172.21.15.62:0/3418829407 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a86000640 1 -- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a6c007390 msgr2=0x7f7a881f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a86000640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a6c007390 0x7f7a881f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a86000640 1 -- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a88155cf0 msgr2=0x7f7a88156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a86000640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a88155cf0 0x7f7a88156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.803+0000 7f7a86000640 1 -- 172.21.15.62:0/3418829407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a881f5a30 con 0x7f7a881f55a0 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.804+0000 7f7a86000640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a881f55a0 0x7f7a880c9cc0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400d800 tx=0x7f7a7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.804+0000 7f7a7f400640 1 -- 172.21.15.62:0/3418829407 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7400b6c0 con 0x7f7a881f55a0 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.804+0000 7f7a7f400640 1 -- 172.21.15.62:0/3418829407 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7400b840 con 0x7f7a881f55a0 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.804+0000 7f7a7f400640 1 -- 172.21.15.62:0/3418829407 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7400bbf0 con 0x7f7a881f55a0 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.804+0000 7f7a87e00640 1 -- 172.21.15.62:0/3418829407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a881f5d10 con 0x7f7a881f55a0 2024-03-23T11:32:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.804+0000 7f7a87e00640 1 -- 172.21.15.62:0/3418829407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a881f62d0 con 0x7f7a881f55a0 2024-03-23T11:32:47.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.806+0000 7f7a87e00640 1 -- 172.21.15.62:0/3418829407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a88066bb0 con 0x7f7a881f55a0 2024-03-23T11:32:47.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.806+0000 7f7a7f400640 1 -- 172.21.15.62:0/3418829407 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f7a7400bd50 con 0x7f7a881f55a0 2024-03-23T11:32:47.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.807+0000 7f7a7f400640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a6806a7a0 0x7f7a6806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:47.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.808+0000 7f7a7f400640 1 -- 172.21.15.62:0/3418829407 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f7a740892d0 con 0x7f7a881f55a0 2024-03-23T11:32:47.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.808+0000 7f7a86a00640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a6806a7a0 0x7f7a6806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:47.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.809+0000 7f7a86a00640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a6806a7a0 0x7f7a6806cc60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f7a78005990 tx=0x7f7a78000fb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:47.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:47.812+0000 7f7a7f400640 1 -- 172.21.15.62:0/3418829407 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7a74054940 con 0x7f7a881f55a0 2024-03-23T11:32:48.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.015+0000 7f7a87e00640 1 -- 172.21.15.62:0/3418829407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7a88159c90 con 0x7f7a881f55a0 2024-03-23T11:32:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.023+0000 7f7a7f400640 1 -- 172.21.15.62:0/3418829407 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7a88159c90 con 0x7f7a881f55a0 2024-03-23T11:32:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.025+0000 7f7a87e00640 1 -- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a6806a7a0 msgr2=0x7f7a6806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.025+0000 7f7a87e00640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a6806a7a0 0x7f7a6806cc60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f7a78005990 tx=0x7f7a78000fb0 comp rx=0 tx=0).stop 2024-03-23T11:32:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.025+0000 7f7a87e00640 1 -- 172.21.15.62:0/3418829407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a881f55a0 msgr2=0x7f7a880c9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.025+0000 7f7a87e00640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a881f55a0 0x7f7a880c9cc0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400d800 tx=0x7f7a7400dcc0 comp rx=0 tx=0).stop 2024-03-23T11:32:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.025+0000 7f7a87e00640 1 -- 172.21.15.62:0/3418829407 shutdown_connections 2024-03-23T11:32:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.025+0000 7f7a87e00640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f7a6806a7a0 0x7f7a6806cc60 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.025+0000 7f7a87e00640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a881f55a0 0x7f7a880c9cc0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.026+0000 7f7a87e00640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a88155cf0 0x7f7a88156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.026+0000 7f7a87e00640 1 --2- 172.21.15.62:0/3418829407 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a6c007390 0x7f7a881f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.026+0000 7f7a87e00640 1 -- 172.21.15.62:0/3418829407 >> 172.21.15.62:0/3418829407 conn(0x7f7a880b0090 msgr2=0x7f7a880b1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.026+0000 7f7a87e00640 1 -- 172.21.15.62:0/3418829407 shutdown_connections 2024-03-23T11:32:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.026+0000 7f7a87e00640 1 -- 172.21.15.62:0/3418829407 wait complete. 2024-03-23T11:32:48.036 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-03-23T11:32:48.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.262+0000 7f926be00640 1 Processor -- start 2024-03-23T11:32:48.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.262+0000 7f926be00640 1 -- start start 2024-03-23T11:32:48.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.263+0000 7f926be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f926c15fcf0 0x7f926c0d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:48.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.263+0000 7f926be00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f926c0b6dc0 0x7f926c1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:48.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.263+0000 7f926be00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c0d3540 con 0x7f926c160420 2024-03-23T11:32:48.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.263+0000 7f926be00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c160df0 con 0x7f926c0b6dc0 2024-03-23T11:32:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.263+0000 7f926be00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c0ce510 con 0x7f926c15fcf0 2024-03-23T11:32:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.263+0000 7f926a000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f926c160420 0x7f926c0d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35508/0 (socket says 172.21.15.62:35508) 2024-03-23T11:32:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.263+0000 7f926a000640 1 -- 172.21.15.62:0/1566979603 learned_addr learned my addr 172.21.15.62:0/1566979603 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.263+0000 7f926b400640 1 --2- 172.21.15.62:0/1566979603 >> v2:172.21.15.62:3300/0 conn(0x7f926c0b6dc0 0x7f926c1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.264+0000 7f926aa00640 1 --2- 172.21.15.62:0/1566979603 >> v2:172.21.15.120:3300/0 conn(0x7f926c15fcf0 0x7f926c0d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.264+0000 7f9269600640 1 -- 172.21.15.62:0/1566979603 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204351093 0 0) 0x7f926c0d3540 con 0x7f926c160420 2024-03-23T11:32:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.264+0000 7f9269600640 1 -- 172.21.15.62:0/1566979603 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9250003520 con 0x7f926c160420 2024-03-23T11:32:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.264+0000 7f926b400640 1 -- 172.21.15.62:0/1566979603 >> v2:172.21.15.120:3300/0 conn(0x7f926c15fcf0 msgr2=0x7f926c0d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.264+0000 7f926b400640 1 --2- 172.21.15.62:0/1566979603 >> v2:172.21.15.120:3300/0 conn(0x7f926c15fcf0 0x7f926c0d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.264+0000 7f926b400640 1 -- 172.21.15.62:0/1566979603 >> v1:172.21.15.62:6789/0 conn(0x7f926c160420 legacy=0x7f926c0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:48.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.264+0000 7f926b400640 1 -- 172.21.15.62:0/1566979603 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0cdc70 con 0x7f926c0b6dc0 2024-03-23T11:32:48.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.265+0000 7f926b400640 1 --2- 172.21.15.62:0/1566979603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f926c0b6dc0 0x7f926c1608b0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f926c066480 tx=0x7f926402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea277b4be3a0376 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.266+0000 7f9269600640 1 -- 172.21.15.62:0/1566979603 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926402cbf0 con 0x7f926c0b6dc0 2024-03-23T11:32:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.266+0000 7f9269600640 1 -- 172.21.15.62:0/1566979603 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926402cd50 con 0x7f926c0b6dc0 2024-03-23T11:32:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.266+0000 7f9269600640 1 -- 172.21.15.62:0/1566979603 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264034660 con 0x7f926c0b6dc0 2024-03-23T11:32:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.266+0000 7f926be00640 1 -- 172.21.15.62:0/1566979603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f926c0b6dc0 msgr2=0x7f926c1608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.266+0000 7f926be00640 1 --2- 172.21.15.62:0/1566979603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f926c0b6dc0 0x7f926c1608b0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f926c066480 tx=0x7f926402be40 comp rx=0 tx=0).stop 2024-03-23T11:32:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.266+0000 7f926be00640 1 -- 172.21.15.62:0/1566979603 shutdown_connections 2024-03-23T11:32:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.266+0000 7f926be00640 1 --2- 172.21.15.62:0/1566979603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f926c0b6dc0 0x7f926c1608b0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.266+0000 7f926be00640 1 --2- 172.21.15.62:0/1566979603 >> v2:172.21.15.120:3300/0 conn(0x7f926c15fcf0 0x7f926c0d3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.266+0000 7f926be00640 1 -- 172.21.15.62:0/1566979603 >> 172.21.15.62:0/1566979603 conn(0x7f926c089f60 msgr2=0x7f926c0d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.266+0000 7f926be00640 1 -- 172.21.15.62:0/1566979603 shutdown_connections 2024-03-23T11:32:48.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.266+0000 7f926be00640 1 -- 172.21.15.62:0/1566979603 wait complete. 2024-03-23T11:32:48.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.268+0000 7f926be00640 1 Processor -- start 2024-03-23T11:32:48.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.268+0000 7f926be00640 1 -- start start 2024-03-23T11:32:48.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.269+0000 7f926be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f926c0b6dc0 0x7f926c0c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:48.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.269+0000 7f926be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f926c15fcf0 0x7f926c0c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:48.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.269+0000 7f926be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f926c160420 0x7f926c1f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:48.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.269+0000 7f926be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0a02f0 con 0x7f926c0b6dc0 2024-03-23T11:32:48.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.269+0000 7f926be00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0c6f90 con 0x7f926c160420 2024-03-23T11:32:48.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.269+0000 7f926be00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0c7270 con 0x7f926c15fcf0 2024-03-23T11:32:48.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.269+0000 7f926aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f926c0b6dc0 0x7f926c0c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:48.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.269+0000 7f926aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f926c0b6dc0 0x7f926c0c8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39596/0 (socket says 172.21.15.62:39596) 2024-03-23T11:32:48.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.269+0000 7f926aa00640 1 -- 172.21.15.62:0/4289410384 learned_addr learned my addr 172.21.15.62:0/4289410384 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:48.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.270+0000 7f926aa00640 1 -- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f926c15fcf0 msgr2=0x7f926c0c6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:48.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.270+0000 7f926aa00640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f926c15fcf0 0x7f926c0c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:48.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.270+0000 7f926aa00640 1 -- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f926c160420 msgr2=0x7f926c1f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:48.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.270+0000 7f926aa00640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f926c160420 0x7f926c1f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:48.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.270+0000 7f926aa00640 1 -- 172.21.15.62:0/4289410384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9264009a70 con 0x7f926c0b6dc0 2024-03-23T11:32:48.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.270+0000 7f926aa00640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f926c0b6dc0 0x7f926c0c8ef0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f925c00e9b0 tx=0x7f925c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:48.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.270+0000 7f9262a00640 1 -- 172.21.15.62:0/4289410384 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c009670 con 0x7f926c0b6dc0 2024-03-23T11:32:48.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.270+0000 7f9262a00640 1 -- 172.21.15.62:0/4289410384 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925c00c9a0 con 0x7f926c0b6dc0 2024-03-23T11:32:48.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.270+0000 7f9262a00640 1 -- 172.21.15.62:0/4289410384 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c00cd50 con 0x7f926c0b6dc0 2024-03-23T11:32:48.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.270+0000 7f926be00640 1 -- 172.21.15.62:0/4289410384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c0c7550 con 0x7f926c0b6dc0 2024-03-23T11:32:48.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.271+0000 7f926be00640 1 -- 172.21.15.62:0/4289410384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c0c6c80 con 0x7f926c0b6dc0 2024-03-23T11:32:48.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.272+0000 7f926be00640 1 -- 172.21.15.62:0/4289410384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c066bb0 con 0x7f926c0b6dc0 2024-03-23T11:32:48.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.273+0000 7f9262a00640 1 -- 172.21.15.62:0/4289410384 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f925c013470 con 0x7f926c0b6dc0 2024-03-23T11:32:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.274+0000 7f9262a00640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f924c06a7a0 0x7f924c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.274+0000 7f9262a00640 1 -- 172.21.15.62:0/4289410384 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f925c088e50 con 0x7f926c0b6dc0 2024-03-23T11:32:48.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.274+0000 7f926a000640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f924c06a7a0 0x7f924c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:48.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.275+0000 7f926a000640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f924c06a7a0 0x7f924c06cc60 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f926c0ccca0 tx=0x7f9258007930 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:48.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.279+0000 7f9262a00640 1 -- 172.21.15.62:0/4289410384 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f925c0544b0 con 0x7f926c0b6dc0 2024-03-23T11:32:48.373 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:32:48.382 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:32:48.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:48.550+0000 7f926be00640 1 -- 172.21.15.62:0/4289410384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) v1 -- 0x7f926c0b4d90 con 0x7f926c0b6dc0 2024-03-23T11:32:49.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.559+0000 7f9262a00640 1 -- 172.21.15.62:0/4289410384 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v642) v1 ==== 144+0+0 (secure 0 0 0) 0x7f925c057ad0 con 0x7f926c0b6dc0 2024-03-23T11:32:49.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 -- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f924c06a7a0 msgr2=0x7f924c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:49.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f924c06a7a0 0x7f924c06cc60 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f926c0ccca0 tx=0x7f9258007930 comp rx=0 tx=0).stop 2024-03-23T11:32:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 -- 172.21.15.62:0/4289410384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f926c0b6dc0 msgr2=0x7f926c0c8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f926c0b6dc0 0x7f926c0c8ef0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f925c00e9b0 tx=0x7f925c00ee70 comp rx=0 tx=0).stop 2024-03-23T11:32:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 -- 172.21.15.62:0/4289410384 shutdown_connections 2024-03-23T11:32:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f924c06a7a0 0x7f924c06cc60 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f926c160420 0x7f926c1f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f926c15fcf0 0x7f926c0c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 --2- 172.21.15.62:0/4289410384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f926c0b6dc0 0x7f926c0c8ef0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 -- 172.21.15.62:0/4289410384 >> 172.21.15.62:0/4289410384 conn(0x7f926c089f60 msgr2=0x7f926c15f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 -- 172.21.15.62:0/4289410384 shutdown_connections 2024-03-23T11:32:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:49.563+0000 7f926be00640 1 -- 172.21.15.62:0/4289410384 wait complete. 2024-03-23T11:32:49.574 INFO:tasks.cephfs_test_runner:test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-03-23T11:32:49.574 INFO:tasks.cephfs_test_runner: 2024-03-23T11:32:49.574 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-23T11:32:49.574 INFO:tasks.cephfs_test_runner:Ran 7 tests in 790.640s 2024-03-23T11:32:49.574 INFO:tasks.cephfs_test_runner: 2024-03-23T11:32:49.574 INFO:tasks.cephfs_test_runner:OK 2024-03-23T11:32:49.575 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-23T11:32:49.584 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-23T11:32:49.591 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-23T11:32:49.591 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:32:49.591 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:32:49.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:32:49.650 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-23T11:32:49.650 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:32:49.705 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498372k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T11:32:49.705 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:32:49.705 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:32:49.763 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:32:49.763 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-23T11:32:49.763 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T11:32:49.816 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T11:32:49.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:49.816 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-03-23T11:32:49.876 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T11:32:49.876 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T11:32:49.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:49.877 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T11:32:49.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:49.982 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T11:32:50.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:50.079 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link delete ceph-brx 2024-03-23T11:32:50.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:50.224 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2024-03-23T11:32:50.282 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T11:32:50.282 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-23T11:32:50.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:50.283 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T11:32:50.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:50.382 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T11:32:50.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T11:32:50.473 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link delete ceph-brx 2024-03-23T11:32:50.587 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-23T11:32:50.597 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-23T11:32:50.597 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:32:50.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.835+0000 7f3cdac00640 1 Processor -- start 2024-03-23T11:32:50.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.836+0000 7f3cdac00640 1 -- start start 2024-03-23T11:32:50.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.837+0000 7f3cdac00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3cd40b2f30 0x7f3cd4070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:50.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.837+0000 7f3cdac00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd40cd140 con 0x7f3cd4143dd0 2024-03-23T11:32:50.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.837+0000 7f3cdac00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd40cd3e0 con 0x7f3cd4070090 2024-03-23T11:32:50.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.837+0000 7f3cdac00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd4070a60 con 0x7f3cd40b2f30 2024-03-23T11:32:50.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.838+0000 7f3cda200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3cd40b2f30 0x7f3cd4070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:50.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.838+0000 7f3cda200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3cd40b2f30 0x7f3cd4070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39602/0 (socket says 172.21.15.62:39602) 2024-03-23T11:32:50.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.838+0000 7f3cda200640 1 -- 172.21.15.62:0/2094297079 learned_addr learned my addr 172.21.15.62:0/2094297079 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:50.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.838+0000 7f3cda200640 1 -- 172.21.15.62:0/2094297079 >> v1:172.21.15.120:6789/0 conn(0x7f3cd4070090 legacy=0x7f3cd40b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:50.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.838+0000 7f3cda200640 1 -- 172.21.15.62:0/2094297079 >> v1:172.21.15.62:6789/0 conn(0x7f3cd4143dd0 legacy=0x7f3cd415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:50.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.838+0000 7f3cda200640 1 -- 172.21.15.62:0/2094297079 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd40cd6c0 con 0x7f3cd40b2f30 2024-03-23T11:32:50.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.839+0000 7f3cda200640 1 --2- 172.21.15.62:0/2094297079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd40b2f30 0x7f3cd4070520 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009870 tx=0x7f3ccc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e680fc9aa28fd2bc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:50.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.840+0000 7f3cd3e00640 1 -- 172.21.15.62:0/2094297079 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc030860 con 0x7f3cd40b2f30 2024-03-23T11:32:50.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.840+0000 7f3cd3e00640 1 -- 172.21.15.62:0/2094297079 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc0309c0 con 0x7f3cd40b2f30 2024-03-23T11:32:50.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.840+0000 7f3cd3e00640 1 -- 172.21.15.62:0/2094297079 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc030d50 con 0x7f3cd40b2f30 2024-03-23T11:32:50.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.840+0000 7f3cdac00640 1 -- 172.21.15.62:0/2094297079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd40b2f30 msgr2=0x7f3cd4070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:50.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.840+0000 7f3cdac00640 1 --2- 172.21.15.62:0/2094297079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd40b2f30 0x7f3cd4070520 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009870 tx=0x7f3ccc02b9a0 comp rx=0 tx=0).stop 2024-03-23T11:32:50.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.840+0000 7f3cdac00640 1 -- 172.21.15.62:0/2094297079 shutdown_connections 2024-03-23T11:32:50.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.840+0000 7f3cdac00640 1 --2- 172.21.15.62:0/2094297079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd40b2f30 0x7f3cd4070520 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:50.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.841+0000 7f3cdac00640 1 -- 172.21.15.62:0/2094297079 >> 172.21.15.62:0/2094297079 conn(0x7f3cd4066050 msgr2=0x7f3cd4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:50.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.841+0000 7f3cdac00640 1 -- 172.21.15.62:0/2094297079 shutdown_connections 2024-03-23T11:32:50.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.841+0000 7f3cdac00640 1 -- 172.21.15.62:0/2094297079 wait complete. 2024-03-23T11:32:50.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.842+0000 7f3cdac00640 1 Processor -- start 2024-03-23T11:32:50.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.842+0000 7f3cdac00640 1 -- start start 2024-03-23T11:32:50.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.843+0000 7f3cdac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd4070090 0x7f3cd406dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:50.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.843+0000 7f3cdac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd40b2f30 0x7f3cd4136ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:50.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.843+0000 7f3cdac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4143dd0 0x7f3cd406c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:50.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.843+0000 7f3cdac00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd4137010 con 0x7f3cd4143dd0 2024-03-23T11:32:50.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.843+0000 7f3cdac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd406cb50 con 0x7f3cd4070090 2024-03-23T11:32:50.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.843+0000 7f3cdac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd406ce30 con 0x7f3cd40b2f30 2024-03-23T11:32:50.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.844+0000 7f3cda200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4143dd0 0x7f3cd406c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:50.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.844+0000 7f3cda200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4143dd0 0x7f3cd406c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39616/0 (socket says 172.21.15.62:39616) 2024-03-23T11:32:50.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.844+0000 7f3cda200640 1 -- 172.21.15.62:0/3947619152 learned_addr learned my addr 172.21.15.62:0/3947619152 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:50.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.844+0000 7f3cd9800640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd4070090 0x7f3cd406dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:50.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.844+0000 7f3cd8e00640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd40b2f30 0x7f3cd4136ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:50.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.844+0000 7f3cda200640 1 -- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd40b2f30 msgr2=0x7f3cd4136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:50.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.844+0000 7f3cda200640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd40b2f30 0x7f3cd4136ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:50.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.844+0000 7f3cda200640 1 -- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd4070090 msgr2=0x7f3cd406dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:50.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.844+0000 7f3cda200640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd4070090 0x7f3cd406dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:50.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.844+0000 7f3cda200640 1 -- 172.21.15.62:0/3947619152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc0094d0 con 0x7f3cd4143dd0 2024-03-23T11:32:50.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.844+0000 7f3cd9800640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd4070090 0x7f3cd406dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:50.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.845+0000 7f3cda200640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4143dd0 0x7f3cd406c140 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc02b9e0 tx=0x7f3ccc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:50.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.845+0000 7f3cd1600640 1 -- 172.21.15.62:0/3947619152 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc043400 con 0x7f3cd4143dd0 2024-03-23T11:32:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.845+0000 7f3cd1600640 1 -- 172.21.15.62:0/3947619152 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc043580 con 0x7f3cd4143dd0 2024-03-23T11:32:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.845+0000 7f3cd1600640 1 -- 172.21.15.62:0/3947619152 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc043930 con 0x7f3cd4143dd0 2024-03-23T11:32:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.845+0000 7f3cdac00640 1 -- 172.21.15.62:0/3947619152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd406d110 con 0x7f3cd4143dd0 2024-03-23T11:32:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.845+0000 7f3cdac00640 1 -- 172.21.15.62:0/3947619152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd406d6d0 con 0x7f3cd4143dd0 2024-03-23T11:32:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.846+0000 7f3cdac00640 1 -- 172.21.15.62:0/3947619152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd406d8c0 con 0x7f3cd4143dd0 2024-03-23T11:32:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.847+0000 7f3cd1600640 1 -- 172.21.15.62:0/3947619152 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f3ccc043a90 con 0x7f3cd4143dd0 2024-03-23T11:32:50.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.848+0000 7f3cd1600640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cb406a7a0 0x7f3cb406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:50.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.848+0000 7f3cd1600640 1 -- 172.21.15.62:0/3947619152 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3ccc03e070 con 0x7f3cd4143dd0 2024-03-23T11:32:50.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.849+0000 7f3cd9800640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cb406a7a0 0x7f3cb406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:50.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.850+0000 7f3cd9800640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cb406a7a0 0x7f3cb406cc60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40ccf00 tx=0x7f3cc8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:50.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:50.853+0000 7f3cd1600640 1 -- 172.21.15.62:0/3947619152 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ccc077490 con 0x7f3cd4143dd0 2024-03-23T11:32:51.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.045+0000 7f3cdac00640 1 -- 172.21.15.62:0/3947619152 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3cd406cfa0 con 0x7f3cb406a7a0 2024-03-23T11:32:51.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.052+0000 7f3cd1600640 1 -- 172.21.15.62:0/3947619152 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48102 (secure 0 0 0) 0x7f3cd406cfa0 con 0x7f3cb406a7a0 2024-03-23T11:32:51.042 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:51.042 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:32:51.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.054+0000 7f3cdac00640 1 -- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cb406a7a0 msgr2=0x7f3cb406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.054+0000 7f3cdac00640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cb406a7a0 0x7f3cb406cc60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40ccf00 tx=0x7f3cc8009040 comp rx=0 tx=0).stop 2024-03-23T11:32:51.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.054+0000 7f3cdac00640 1 -- 172.21.15.62:0/3947619152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4143dd0 msgr2=0x7f3cd406c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.054+0000 7f3cdac00640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4143dd0 0x7f3cd406c140 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc02b9e0 tx=0x7f3ccc004730 comp rx=0 tx=0).stop 2024-03-23T11:32:51.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.055+0000 7f3cdac00640 1 -- 172.21.15.62:0/3947619152 shutdown_connections 2024-03-23T11:32:51.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.055+0000 7f3cdac00640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f3cb406a7a0 0x7f3cb406cc60 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.055+0000 7f3cdac00640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4143dd0 0x7f3cd406c140 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.055+0000 7f3cdac00640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3cd40b2f30 0x7f3cd4136ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.055+0000 7f3cdac00640 1 --2- 172.21.15.62:0/3947619152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cd4070090 0x7f3cd406dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.055+0000 7f3cdac00640 1 -- 172.21.15.62:0/3947619152 >> 172.21.15.62:0/3947619152 conn(0x7f3cd4066050 msgr2=0x7f3cd41430b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:51.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.055+0000 7f3cdac00640 1 -- 172.21.15.62:0/3947619152 shutdown_connections 2024-03-23T11:32:51.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.055+0000 7f3cdac00640 1 -- 172.21.15.62:0/3947619152 wait complete. 2024-03-23T11:32:51.062 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":539,"stamp":"2024-03-23T11:32:49.544109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":274504,"kb_used_data":25560,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754700216,"statfs":{"total":773094113280,"available":772813021184,"internally_reserved":0,"allocated":26173440,"data_stored":19161657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8392377,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-264,"num_write":-179,"num_write_kb":-8297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002447"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":273,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-23T11:32:46.530200+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:32:46.530200+0000","last_peered":"2024-03-23T11:32:46.530200+0000","last_clean":"2024-03-23T11:32:46.530200+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:32:46.530200+0000","last_undegraded":"2024-03-23T11:32:46.530200+0000","last_fullsized":"2024-03-23T11:32:46.530200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542315,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33220,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338620,"statfs":{"total":96636764160,"available":96602746880,"internally_reserved":0,"allocated":2150400,"data_stored":1267532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.087,"15min":1.028},"min":{"1min":0.63600000000000001,"5min":0.623,"15min":0.5},"max":{"1min":1.456,"5min":1.631,"15min":2.8370000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.075,"5min":1.1220000000000001,"15min":1.0589999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.38300000000000001},"max":{"1min":1.5449999999999999,"5min":1.7230000000000001,"15min":1.8009999999999999},"last":1.0249999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.2809999999999999,"15min":1.214},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.46100000000000002},"max":{"1min":1.5720000000000001,"5min":1.708,"15min":3.569},"last":1.254},{"interface":"front","average":{"1min":1.208,"5min":1.2909999999999999,"15min":1.2669999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.69899999999999995},"max":{"1min":1.5309999999999999,"5min":1.8009999999999999,"15min":2.4350000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1299999999999999,"15min":1.0960000000000001},"min":{"1min":0.69899999999999995,"5min":0.65500000000000003,"15min":0.58099999999999996},"max":{"1min":1.4610000000000001,"5min":1.909,"15min":2.7440000000000002},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.23,"15min":1.1990000000000001},"min":{"1min":0.83799999999999997,"5min":0.81499999999999995,"15min":0.56999999999999995},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":3.524},"last":1.3839999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.204,"15min":1.1930000000000001},"min":{"1min":0.86799999999999999,"5min":0.72199999999999998,"15min":0.67600000000000005},"max":{"1min":1.5169999999999999,"5min":1.7889999999999999,"15min":2.6389999999999998},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.3169999999999999,"15min":1.2809999999999999},"min":{"1min":0.94399999999999995,"5min":0.93500000000000005,"15min":0.60199999999999998},"max":{"1min":1.641,"5min":1.7050000000000001,"15min":2.9470000000000001},"last":1.101}]},{"osd":4,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1679999999999999,"15min":1.208},"min":{"1min":0.84099999999999997,"5min":0.73199999999999998,"15min":0.54300000000000004},"max":{"1min":1.534,"5min":2.1560000000000001,"15min":2.2719999999999998},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.391,"15min":1.3979999999999999},"min":{"1min":1.071,"5min":0.81699999999999995,"15min":0.60199999999999998},"max":{"1min":1.7,"5min":2.0819999999999999,"15min":2.5569999999999999},"last":1.506}]},{"osd":5,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.369,"15min":1.351},"min":{"1min":0.745,"5min":0.745,"15min":0.53300000000000003},"max":{"1min":1.6719999999999999,"5min":1.8520000000000001,"15min":2.8999999999999999},"last":1.619},{"interface":"front","average":{"1min":1.379,"5min":1.448,"15min":1.427},"min":{"1min":1.073,"5min":0.91700000000000004,"15min":0.58199999999999996},"max":{"1min":1.6299999999999999,"5min":1.978,"15min":2.0859999999999999},"last":1.6910000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.391,"15min":1.417},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.61399999999999999},"max":{"1min":1.6379999999999999,"5min":2.0680000000000001,"15min":2.1299999999999999},"last":2.0510000000000002},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.357,"15min":1.3720000000000001},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.59899999999999998},"max":{"1min":1.637,"5min":1.9970000000000001,"15min":3.218},"last":1.3109999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542315,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36716,"kb_used_data":5600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335124,"statfs":{"total":96636764160,"available":96599166976,"internally_reserved":0,"allocated":5734400,"data_stored":4865035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.091,"15min":1.115},"min":{"1min":0.71299999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.536,"5min":1.7450000000000001,"15min":3.9140000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.133,"5min":1.1879999999999999,"15min":1.161},"min":{"1min":0.72299999999999998,"5min":0.71099999999999997,"15min":0.45200000000000001},"max":{"1min":1.53,"5min":1.8600000000000001,"15min":6.6539999999999999},"last":0.90000000000000002}]},{"osd":1,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1819999999999999,"15min":1.2},"min":{"1min":0.84999999999999998,"5min":0.77300000000000002,"15min":0.69399999999999995},"max":{"1min":1.5469999999999999,"5min":1.627,"15min":2.4820000000000002},"last":1.458},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1559999999999999,"15min":1.1970000000000001},"min":{"1min":0.79100000000000004,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.857,"5min":1.857,"15min":2.0859999999999999},"last":1.603}]},{"osd":2,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.173,"15min":1.2090000000000001},"min":{"1min":0.93000000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.6299999999999999,"5min":1.649,"15min":3.73},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.2989999999999999,"15min":1.29},"min":{"1min":0.91000000000000003,"5min":0.73499999999999999,"15min":0.68100000000000005},"max":{"1min":1.452,"5min":1.8,"15min":2.3460000000000001},"last":2.0529999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.246,"15min":1.278},"min":{"1min":0.83799999999999997,"5min":0.76700000000000002,"15min":0.749},"max":{"1min":1.6160000000000001,"5min":1.738,"15min":12.914},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.3080000000000001,"15min":1.3260000000000001},"min":{"1min":0.85099999999999998,"5min":0.76100000000000001,"15min":0.57799999999999996},"max":{"1min":1.5069999999999999,"5min":1.8919999999999999,"15min":3.8100000000000001},"last":1.2569999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.3149999999999999,"15min":1.3300000000000001},"min":{"1min":1.123,"5min":0.68000000000000005,"15min":0.51100000000000001},"max":{"1min":1.5920000000000001,"5min":1.827,"15min":2.702},"last":2.294},{"interface":"front","average":{"1min":1.325,"5min":1.2989999999999999,"15min":1.353},"min":{"1min":0.95499999999999996,"5min":0.752,"15min":0.752},"max":{"1min":1.875,"5min":1.875,"15min":7.2140000000000004},"last":1.8899999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.399,"15min":1.45},"min":{"1min":1.143,"5min":0.92900000000000005,"15min":0.73899999999999999},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":3.097},"last":2.2519999999999998},{"interface":"front","average":{"1min":1.502,"5min":1.4510000000000001,"15min":1.466},"min":{"1min":1.355,"5min":0.86899999999999999,"15min":0.72499999999999998},"max":{"1min":1.6879999999999999,"5min":1.9139999999999999,"15min":3.3109999999999999},"last":2.1579999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.534,"15min":1.5860000000000001},"min":{"1min":1.2949999999999999,"5min":0.97799999999999998,"15min":0.80400000000000005},"max":{"1min":2.0150000000000001,"5min":2.0760000000000001,"15min":4.9729999999999999},"last":2.0019999999999998},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4390000000000001,"15min":1.464},"min":{"1min":0.998,"5min":0.70399999999999996,"15min":0.57699999999999996},"max":{"1min":2.0640000000000001,"5min":2.0649999999999999,"15min":4.085},"last":1.5669999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542315,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.149,"15min":1.1240000000000001},"min":{"1min":0.63,"5min":0.63,"15min":0.55400000000000005},"max":{"1min":1.526,"5min":2.0179999999999998,"15min":2.2999999999999998},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.115,"5min":1.1439999999999999,"15min":1.145},"min":{"1min":0.73999999999999999,"5min":0.624,"15min":0.47799999999999998},"max":{"1min":1.518,"5min":1.7050000000000001,"15min":2.355},"last":0.68000000000000005}]},{"osd":1,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1990000000000001,"15min":1.171},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":2.2559999999999998},"last":1.706},{"interface":"front","average":{"1min":1.173,"5min":1.1970000000000001,"15min":1.1859999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.495},"max":{"1min":1.5,"5min":1.599,"15min":1.8580000000000001},"last":1.3560000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1799999999999999,"15min":1.212},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.60199999999999998},"max":{"1min":1.605,"5min":1.6659999999999999,"15min":2.012},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.294,"15min":1.2529999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.64400000000000002},"max":{"1min":1.661,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.036}]},{"osd":3,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.27,"15min":1.284},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5209999999999999,"5min":1.6499999999999999,"15min":2.3799999999999999},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.371,"15min":1.3700000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.6379999999999999,"5min":2.0750000000000002,"15min":2.3999999999999999},"last":1.153}]},{"osd":4,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.304,"15min":1.3240000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.587,"5min":2.1499999999999999,"15min":2.2090000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4730000000000001,"15min":1.48},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.6240000000000001,"5min":1.8919999999999999,"15min":2.5009999999999999},"last":1.6659999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3759999999999999,"15min":1.393},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7,"5min":2.3170000000000002,"15min":2.4860000000000002},"last":1.546},{"interface":"front","average":{"1min":1.292,"5min":1.403,"15min":1.415},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.609,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.756}]},{"osd":7,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.4530000000000001,"15min":1.468},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.661,"5min":2.1549999999999998,"15min":2.7229999999999999},"last":1.825},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.5389999999999999,"15min":1.55},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.8640000000000001,"5min":2.2269999999999999,"15min":2.7509999999999999},"last":1.411}]}]},{"osd":4,"up_from":14,"seq":60129542315,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.2330000000000001,"15min":1.1830000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.8360000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.3049999999999999,"15min":1.2789999999999999},"min":{"1min":0.78900000000000003,"5min":0.74399999999999999,"15min":0.63700000000000001},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":1.5389999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.1539999999999999,"15min":1.1619999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.776,"5min":1.9350000000000001,"15min":4.327},"last":1.196},{"interface":"front","average":{"1min":1.129,"5min":1.1950000000000001,"15min":1.1619999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.4369999999999998},"last":0.83499999999999996}]},{"osd":2,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.177,"15min":1.202},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.624},"max":{"1min":1.6299999999999999,"5min":2.0960000000000001,"15min":2.6080000000000001},"last":1.389},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.2809999999999999,"15min":1.2689999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.71799999999999997},"max":{"1min":1.7090000000000001,"5min":1.8919999999999999,"15min":4.1310000000000002},"last":1.3400000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.92400000000000004,"5min":0.747,"15min":0.622},"max":{"1min":1.6100000000000001,"5min":1.659,"15min":2.5579999999999998},"last":1.242},{"interface":"front","average":{"1min":1.347,"5min":1.3740000000000001,"15min":1.3320000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.70199999999999996},"max":{"1min":1.835,"5min":2.024,"15min":4.2400000000000002},"last":1.5649999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.23,"15min":1.252},"min":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.74299999999999999},"max":{"1min":1.764,"5min":1.8740000000000001,"15min":2.508},"last":1.073},{"interface":"front","average":{"1min":1.456,"5min":1.3360000000000001,"15min":1.3620000000000001},"min":{"1min":0.92800000000000005,"5min":0.67800000000000005,"15min":0.60499999999999998},"max":{"1min":1.7010000000000001,"5min":1.7849999999999999,"15min":5.859},"last":1.4570000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3759999999999999,"15min":1.401},"min":{"1min":0.94799999999999995,"5min":0.753,"15min":0.58899999999999997},"max":{"1min":2.1099999999999999,"5min":2.194,"15min":5.3339999999999996},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.4670000000000001,"15min":1.4870000000000001},"min":{"1min":1.26,"5min":0.81000000000000005,"15min":0.72599999999999998},"max":{"1min":1.885,"5min":1.885,"15min":5.3940000000000001},"last":1.645}]},{"osd":7,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.4910000000000001,"15min":1.498},"min":{"1min":1.0269999999999999,"5min":0.84699999999999998,"15min":0.754},"max":{"1min":2.0059999999999998,"5min":2.0790000000000002,"15min":2.6859999999999999},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.476,"5min":1.3839999999999999,"15min":1.3859999999999999},"min":{"1min":1.0589999999999999,"5min":0.79000000000000004,"15min":0.75600000000000001},"max":{"1min":1.978,"5min":1.978,"15min":2.641},"last":1.504}]}]},{"osd":3,"up_from":14,"seq":60129542315,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36720,"kb_used_data":5604,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335120,"statfs":{"total":96636764160,"available":96599162880,"internally_reserved":0,"allocated":5738496,"data_stored":4865702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.88100000000000001,"15min":0.91800000000000004},"min":{"1min":0.42399999999999999,"5min":0.39300000000000002,"15min":0.32100000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.6819999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.1160000000000001,"15min":1.101},"min":{"1min":0.48099999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.5589999999999999,"5min":1.8540000000000001,"15min":2.0790000000000002},"last":1.083}]},{"osd":1,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.96899999999999997,"15min":0.98299999999999998},"min":{"1min":0.51900000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.472,"5min":1.5249999999999999,"15min":1.976},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.202,"15min":1.2270000000000001},"min":{"1min":0.65200000000000002,"5min":0.47099999999999997,"15min":0.44600000000000001},"max":{"1min":1.5069999999999999,"5min":1.774,"15min":2.2309999999999999},"last":1.2270000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.107,"15min":1.1120000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5669999999999999,"5min":1.7729999999999999,"15min":3.3109999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.165,"15min":1.1879999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.48399999999999999},"max":{"1min":1.532,"5min":1.635,"15min":2.7109999999999999},"last":1.121}]},{"osd":4,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.26,"15min":1.2609999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.73199999999999998},"max":{"1min":1.6040000000000001,"5min":1.752,"15min":2.3490000000000002},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.4370000000000001,"15min":1.456},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.905,"5min":2.081,"15min":4.8970000000000002},"last":1.421}]},{"osd":5,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.431,"15min":1.4670000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.7729999999999999,"5min":2.121,"15min":3.4910000000000001},"last":1.296},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.361,"15min":1.3959999999999999},"min":{"1min":0.77200000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.7909999999999999,"5min":1.8700000000000001,"15min":3.125},"last":1.1679999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.3640000000000001,"15min":1.409},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.9550000000000001,"5min":2.0630000000000002,"15min":4.5499999999999998},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.4379999999999999,"15min":1.4830000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.9339999999999999,"5min":2.2080000000000002,"15min":2.8450000000000002},"last":1.401}]},{"osd":7,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.4179999999999999,"15min":1.4450000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.879,"5min":1.998,"15min":2.952},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.484,"15min":1.524},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8100000000000001,"5min":1.964,"15min":4.798},"last":1.3109999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542315,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.878,"15min":0.89200000000000002},"min":{"1min":0.61099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":1.375,"5min":1.411,"15min":2.2509999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.1000000000000001,"15min":1.0660000000000001},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.35099999999999998},"max":{"1min":1.371,"5min":1.6759999999999999,"15min":1.8220000000000001},"last":0.71999999999999997}]},{"osd":1,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88900000000000001,"15min":0.89600000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.32000000000000001},"max":{"1min":1.3200000000000001,"5min":1.798,"15min":1.9430000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.05,"5min":1.069,"15min":1.036},"min":{"1min":0.66700000000000004,"5min":0.45000000000000001,"15min":0.35799999999999998},"max":{"1min":1.4550000000000001,"5min":1.5349999999999999,"15min":1.7230000000000001},"last":1.1020000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1220000000000001,"15min":1.111},"min":{"1min":0.72399999999999998,"5min":0.47399999999999998,"15min":0.38300000000000001},"max":{"1min":1.3580000000000001,"5min":1.758,"15min":1.98},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.008,"5min":1.0629999999999999,"15min":1.069},"min":{"1min":0.53200000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.524,"5min":1.6879999999999999,"15min":2.1560000000000001},"last":1.006}]},{"osd":4,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3520000000000001,"15min":1.3540000000000001},"min":{"1min":0.90400000000000003,"5min":0.86699999999999999,"15min":0.69599999999999995},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":2.1280000000000001},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.4279999999999999,"15min":1.411},"min":{"1min":0.95299999999999996,"5min":0.93799999999999994,"15min":0.83499999999999996},"max":{"1min":1.774,"5min":1.9079999999999999,"15min":2.085},"last":1.2809999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.353,"15min":1.359},"min":{"1min":1.0209999999999999,"5min":0.752,"15min":0.64100000000000001},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":2.5830000000000002},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.395,"5min":1.341,"15min":1.359},"min":{"1min":0.93899999999999995,"5min":0.73999999999999999,"15min":0.66000000000000003},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.6469999999999998},"last":1.5209999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4159999999999999,"15min":1.3899999999999999},"min":{"1min":0.94899999999999995,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.758},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4139999999999999,"15min":1.4350000000000001},"min":{"1min":1.0429999999999999,"5min":0.79100000000000004,"15min":0.69599999999999995},"max":{"1min":1.7869999999999999,"5min":1.883,"15min":2.8580000000000001},"last":1.226}]},{"osd":7,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.468,"15min":1.4430000000000001},"min":{"1min":0.94299999999999995,"5min":0.83099999999999996,"15min":0.71899999999999997},"max":{"1min":2.23,"5min":2.23,"15min":2.246},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.476,"15min":1.5029999999999999},"min":{"1min":1.0489999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.8819999999999999,"5min":1.9550000000000001,"15min":2.7120000000000002},"last":1.3120000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542316,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32632,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1552384,"data_stored":676495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.876,"15min":0.89400000000000002},"min":{"1min":0.41599999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.6579999999999999,"5min":1.6930000000000001,"15min":3.2959999999999998},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.002,"5min":1.0620000000000001,"15min":1.0149999999999999},"min":{"1min":0.5,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.53,"5min":1.8480000000000001,"15min":1.99},"last":1.2669999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.94699999999999995},"min":{"1min":0.51200000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2949999999999999,"5min":1.5549999999999999,"15min":1.738},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.105,"15min":1.1040000000000001},"min":{"1min":0.60499999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5109999999999999,"5min":1.702,"15min":2.2160000000000002},"last":1.04}]},{"osd":3,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2250000000000001,"15min":1.222},"min":{"1min":0.60699999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.734,"5min":1.8819999999999999,"15min":2.4430000000000001},"last":1.661},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.133,"15min":1.1220000000000001},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.434},"max":{"1min":1.637,"5min":2.0539999999999998,"15min":2.4169999999999998},"last":1.411}]},{"osd":4,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.345,"15min":1.3140000000000001},"min":{"1min":0.92100000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7190000000000001,"5min":1.8080000000000001,"15min":2.153},"last":1.9890000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.286,"15min":1.3},"min":{"1min":0.747,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":2.758},"last":1.1120000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.357},"min":{"1min":0.81100000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":2.004,"5min":2.004,"15min":2.6589999999999998},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.4239999999999999,"15min":1.4259999999999999},"min":{"1min":0.97599999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.8109999999999999,"5min":1.9610000000000001,"15min":3.0329999999999999},"last":1.54}]},{"osd":6,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4119999999999999,"15min":1.4179999999999999},"min":{"1min":1.0369999999999999,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.984,"5min":1.984,"15min":2.2730000000000001},"last":2.0550000000000002},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4359999999999999,"15min":1.4650000000000001},"min":{"1min":0.91100000000000003,"5min":0.747,"15min":0.747},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.9630000000000001},"last":1.3919999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.524,"15min":1.528},"min":{"1min":0.93500000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.1269999999999998,"5min":2.2450000000000001,"15min":3.2749999999999999},"last":1.77},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4990000000000001,"15min":1.5249999999999999},"min":{"1min":1.0740000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.7829999999999999},"last":1.857}]}]},{"osd":0,"up_from":14,"seq":60129542314,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37296,"kb_used_data":6180,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94334544,"statfs":{"total":96636764160,"available":96598573056,"internally_reserved":0,"allocated":6328320,"data_stored":5455401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.85199999999999998,"15min":0.85799999999999998},"min":{"1min":0.55100000000000005,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.262,"5min":1.8660000000000001,"15min":2.8479999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":0.97599999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.30399999999999999},"max":{"1min":1.694,"5min":1.694,"15min":2.4319999999999999},"last":1.016}]},{"osd":2,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0269999999999999,"15min":1.0349999999999999},"min":{"1min":0.55800000000000005,"5min":0.41799999999999998,"15min":0.33700000000000002},"max":{"1min":1.849,"5min":1.849,"15min":2.4980000000000002},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.1180000000000001,"15min":1.069},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.432},"max":{"1min":1.5820000000000001,"5min":1.7569999999999999,"15min":2.5760000000000001},"last":1.448}]},{"osd":3,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1579999999999999,"15min":1.123},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.35799999999999998},"max":{"1min":1.552,"5min":1.7589999999999999,"15min":2.4580000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3460000000000001,"15min":1.3200000000000001},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.48599999999999999},"max":{"1min":1.6519999999999999,"5min":1.968,"15min":2.9390000000000001},"last":1.536}]},{"osd":4,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3320000000000001,"15min":1.3360000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.7410000000000001,"5min":1.855,"15min":2.9750000000000001},"last":1.78},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3360000000000001,"15min":1.298},"min":{"1min":0.88500000000000001,"5min":0.81399999999999995,"15min":0.39900000000000002},"max":{"1min":1.728,"5min":1.968,"15min":3.125},"last":1.3959999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.361,"15min":1.3660000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.51200000000000001},"max":{"1min":1.595,"5min":1.925,"15min":3.0830000000000002},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.3400000000000001,"15min":1.4079999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.52700000000000002},"max":{"1min":1.772,"5min":2.0449999999999999,"15min":3.1899999999999999},"last":1.2909999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.4139999999999999,"15min":1.448},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.47999999999999998},"max":{"1min":1.77,"5min":2.1030000000000002,"15min":3.391},"last":1.712},{"interface":"front","average":{"1min":1.367,"5min":1.4259999999999999,"15min":1.4079999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.58399999999999996},"max":{"1min":1.956,"5min":2.2170000000000001,"15min":3.153},"last":1.6679999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.5349999999999999,"15min":1.5189999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.60599999999999998},"max":{"1min":1.877,"5min":2.1400000000000001,"15min":3.5350000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.516,"15min":1.494},"min":{"1min":0.92600000000000005,"5min":0.78300000000000003,"15min":0.45300000000000001},"max":{"1min":1.7090000000000001,"5min":2.2610000000000001,"15min":3.3650000000000002},"last":1.583}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:32:51.063 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:32:51.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.296+0000 7f491fe00640 1 Processor -- start 2024-03-23T11:32:51.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.297+0000 7f491fe00640 1 -- start start 2024-03-23T11:32:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.298+0000 7f491fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f492015fcf0 0x7f49200d3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.298+0000 7f491fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4920160420 0x7f4920160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.298+0000 7f491fe00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4920160dc0 con 0x7f49200d06d0 2024-03-23T11:32:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.298+0000 7f491fe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49200d3540 con 0x7f492015fcf0 2024-03-23T11:32:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.298+0000 7f491fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49200ce510 con 0x7f4920160420 2024-03-23T11:32:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.299+0000 7f491f400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f49200d06d0 0x7f49200d0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35530/0 (socket says 172.21.15.62:35530) 2024-03-23T11:32:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.299+0000 7f491f400640 1 -- 172.21.15.62:0/10335498 learned_addr learned my addr 172.21.15.62:0/10335498 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.299+0000 7f491ea00640 1 --2- 172.21.15.62:0/10335498 >> v2:172.21.15.120:3300/0 conn(0x7f492015fcf0 0x7f49200d3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.299+0000 7f491e000640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4920160420 0x7f4920160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.299+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373144980 0 0) 0x7f4920160dc0 con 0x7f49200d06d0 2024-03-23T11:32:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.299+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904003520 con 0x7f49200d06d0 2024-03-23T11:32:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 333998749 0 0) 0x7f4904003520 con 0x7f49200d06d0 2024-03-23T11:32:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4920160420 msgr2=0x7f4920160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4920160420 0x7f4920160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 >> v2:172.21.15.120:3300/0 conn(0x7f492015fcf0 msgr2=0x7f49200d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 --2- 172.21.15.62:0/10335498 >> v2:172.21.15.120:3300/0 conn(0x7f492015fcf0 0x7f49200d3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49200cdc70 con 0x7f49200d06d0 2024-03-23T11:32:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f49180019f0 con 0x7f49200d06d0 2024-03-23T11:32:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 >> v1:172.21.15.62:6789/0 conn(0x7f49200d06d0 legacy=0x7f49200d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4904003970 0x7f4904003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49040047c0 0x7f4904004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4904005180 0x7f490400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49200d3540 con 0x7f4904005180 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49200ce510 con 0x7f4904003970 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490400bff0 con 0x7f49040047c0 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491e000640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4920160420 0x7f4920160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.300+0000 7f491f400640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4904005180 0x7f490400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.301+0000 7f491e000640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49040047c0 0x7f4904004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.301+0000 7f491ea00640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4904003970 0x7f4904003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.301+0000 7f491f400640 1 -- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49040047c0 msgr2=0x7f4904004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.301+0000 7f491f400640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49040047c0 0x7f4904004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.301+0000 7f491f400640 1 -- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4904003970 msgr2=0x7f4904003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.301+0000 7f491f400640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4904003970 0x7f4904003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.301+0000 7f491f400640 1 -- 172.21.15.62:0/10335498 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4904003520 con 0x7f4904005180 2024-03-23T11:32:51.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.301+0000 7f491ea00640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4904003970 0x7f4904003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:51.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.302+0000 7f491f400640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4904005180 0x7f490400b980 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f491800a790 tx=0x7f49040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1eab657e933ac1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:51.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.302+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4918034630 con 0x7f4904005180 2024-03-23T11:32:51.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.302+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491802f070 con 0x7f4904005180 2024-03-23T11:32:51.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.302+0000 7f491d600640 1 -- 172.21.15.62:0/10335498 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49180349f0 con 0x7f4904005180 2024-03-23T11:32:51.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.303+0000 7f491fe00640 1 -- 172.21.15.62:0/10335498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4904005180 msgr2=0x7f490400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.303+0000 7f491fe00640 1 --2- 172.21.15.62:0/10335498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4904005180 0x7f490400b980 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f491800a790 tx=0x7f49040057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.303+0000 7f491f400640 1 -- 172.21.15.62:0/10335498 reap_dead start 2024-03-23T11:32:51.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.303+0000 7f491fe00640 1 -- 172.21.15.62:0/10335498 shutdown_connections 2024-03-23T11:32:51.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.303+0000 7f491fe00640 1 -- 172.21.15.62:0/10335498 >> 172.21.15.62:0/10335498 conn(0x7f4920089f60 msgr2=0x7f49200d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:51.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.303+0000 7f491fe00640 1 -- 172.21.15.62:0/10335498 shutdown_connections 2024-03-23T11:32:51.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.303+0000 7f491fe00640 1 -- 172.21.15.62:0/10335498 wait complete. 2024-03-23T11:32:51.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.305+0000 7f491fe00640 1 Processor -- start 2024-03-23T11:32:51.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.305+0000 7f491fe00640 1 -- start start 2024-03-23T11:32:51.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4904005180 0x7f49201583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49201559f0 0x7f492015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4920155e70 0x7f492015cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491fe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4920158930 con 0x7f49201559f0 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49201576a0 con 0x7f4920155e70 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4920157910 con 0x7f4904005180 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49201559f0 0x7f492015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49201559f0 0x7f492015c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39644/0 (socket says 172.21.15.62:39644) 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491e000640 1 -- 172.21.15.62:0/4127106810 learned_addr learned my addr 172.21.15.62:0/4127106810 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491e000640 1 -- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4904005180 msgr2=0x7f49201583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491e000640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4904005180 0x7f49201583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491e000640 1 -- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4920155e70 msgr2=0x7f492015cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491f400640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4920155e70 0x7f492015cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491e000640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4920155e70 0x7f492015cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.306+0000 7f491e000640 1 -- 172.21.15.62:0/4127106810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4920157bf0 con 0x7f49201559f0 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.307+0000 7f491e000640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49201559f0 0x7f492015c7c0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f490800f4c0 tx=0x7f490800f980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.307+0000 7f4916a00640 1 -- 172.21.15.62:0/4127106810 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4908014890 con 0x7f49201559f0 2024-03-23T11:32:51.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.307+0000 7f4916a00640 1 -- 172.21.15.62:0/4127106810 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490800c030 con 0x7f49201559f0 2024-03-23T11:32:51.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.307+0000 7f4916a00640 1 -- 172.21.15.62:0/4127106810 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4908014c10 con 0x7f49201559f0 2024-03-23T11:32:51.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.307+0000 7f491fe00640 1 -- 172.21.15.62:0/4127106810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4920157ed0 con 0x7f49201559f0 2024-03-23T11:32:51.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.307+0000 7f491fe00640 1 -- 172.21.15.62:0/4127106810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4920146da0 con 0x7f49201559f0 2024-03-23T11:32:51.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.309+0000 7f491fe00640 1 -- 172.21.15.62:0/4127106810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4920158060 con 0x7f49201559f0 2024-03-23T11:32:51.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.309+0000 7f4916a00640 1 -- 172.21.15.62:0/4127106810 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f49080133e0 con 0x7f49201559f0 2024-03-23T11:32:51.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.311+0000 7f4916a00640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48fc06a7a0 0x7f48fc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.311+0000 7f4916a00640 1 -- 172.21.15.62:0/4127106810 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f4908088df0 con 0x7f49201559f0 2024-03-23T11:32:51.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.311+0000 7f491ea00640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48fc06a7a0 0x7f48fc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.312+0000 7f491ea00640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48fc06a7a0 0x7f48fc06cc60 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f4910009a60 tx=0x7f4910008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:51.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.315+0000 7f4916a00640 1 -- 172.21.15.62:0/4127106810 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4908054460 con 0x7f49201559f0 2024-03-23T11:32:51.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.511+0000 7f491fe00640 1 -- 172.21.15.62:0/4127106810 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49200b55d0 con 0x7f48fc06a7a0 2024-03-23T11:32:51.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.517+0000 7f4916a00640 1 -- 172.21.15.62:0/4127106810 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48102 (secure 0 0 0) 0x7f49100108b0 con 0x7f48fc06a7a0 2024-03-23T11:32:51.507 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:32:51.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 -- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48fc06a7a0 msgr2=0x7f48fc06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48fc06a7a0 0x7f48fc06cc60 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f4910009a60 tx=0x7f4910008040 comp rx=0 tx=0).stop 2024-03-23T11:32:51.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 -- 172.21.15.62:0/4127106810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49201559f0 msgr2=0x7f492015c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49201559f0 0x7f492015c7c0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f490800f4c0 tx=0x7f490800f980 comp rx=0 tx=0).stop 2024-03-23T11:32:51.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 -- 172.21.15.62:0/4127106810 shutdown_connections 2024-03-23T11:32:51.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f48fc06a7a0 0x7f48fc06cc60 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f49201559f0 0x7f492015c7c0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4920155e70 0x7f492015cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 --2- 172.21.15.62:0/4127106810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4904005180 0x7f49201583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 -- 172.21.15.62:0/4127106810 >> 172.21.15.62:0/4127106810 conn(0x7f4920089f60 msgr2=0x7f49200b4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:51.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 -- 172.21.15.62:0/4127106810 shutdown_connections 2024-03-23T11:32:51.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.520+0000 7f491fe00640 1 -- 172.21.15.62:0/4127106810 wait complete. 2024-03-23T11:32:51.529 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":539,"stamp":"2024-03-23T11:32:49.544109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":274504,"kb_used_data":25560,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754700216,"statfs":{"total":773094113280,"available":772813021184,"internally_reserved":0,"allocated":26173440,"data_stored":19161657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8392377,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-264,"num_write":-179,"num_write_kb":-8297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002447"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":273,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-23T11:32:46.530200+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:32:46.530200+0000","last_peered":"2024-03-23T11:32:46.530200+0000","last_clean":"2024-03-23T11:32:46.530200+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:32:46.530200+0000","last_undegraded":"2024-03-23T11:32:46.530200+0000","last_fullsized":"2024-03-23T11:32:46.530200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542315,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33220,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338620,"statfs":{"total":96636764160,"available":96602746880,"internally_reserved":0,"allocated":2150400,"data_stored":1267532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.087,"15min":1.028},"min":{"1min":0.63600000000000001,"5min":0.623,"15min":0.5},"max":{"1min":1.456,"5min":1.631,"15min":2.8370000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.075,"5min":1.1220000000000001,"15min":1.0589999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.38300000000000001},"max":{"1min":1.5449999999999999,"5min":1.7230000000000001,"15min":1.8009999999999999},"last":1.0249999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.2809999999999999,"15min":1.214},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.46100000000000002},"max":{"1min":1.5720000000000001,"5min":1.708,"15min":3.569},"last":1.254},{"interface":"front","average":{"1min":1.208,"5min":1.2909999999999999,"15min":1.2669999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.69899999999999995},"max":{"1min":1.5309999999999999,"5min":1.8009999999999999,"15min":2.4350000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1299999999999999,"15min":1.0960000000000001},"min":{"1min":0.69899999999999995,"5min":0.65500000000000003,"15min":0.58099999999999996},"max":{"1min":1.4610000000000001,"5min":1.909,"15min":2.7440000000000002},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.23,"15min":1.1990000000000001},"min":{"1min":0.83799999999999997,"5min":0.81499999999999995,"15min":0.56999999999999995},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":3.524},"last":1.3839999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.204,"15min":1.1930000000000001},"min":{"1min":0.86799999999999999,"5min":0.72199999999999998,"15min":0.67600000000000005},"max":{"1min":1.5169999999999999,"5min":1.7889999999999999,"15min":2.6389999999999998},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.3169999999999999,"15min":1.2809999999999999},"min":{"1min":0.94399999999999995,"5min":0.93500000000000005,"15min":0.60199999999999998},"max":{"1min":1.641,"5min":1.7050000000000001,"15min":2.9470000000000001},"last":1.101}]},{"osd":4,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1679999999999999,"15min":1.208},"min":{"1min":0.84099999999999997,"5min":0.73199999999999998,"15min":0.54300000000000004},"max":{"1min":1.534,"5min":2.1560000000000001,"15min":2.2719999999999998},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.391,"15min":1.3979999999999999},"min":{"1min":1.071,"5min":0.81699999999999995,"15min":0.60199999999999998},"max":{"1min":1.7,"5min":2.0819999999999999,"15min":2.5569999999999999},"last":1.506}]},{"osd":5,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.369,"15min":1.351},"min":{"1min":0.745,"5min":0.745,"15min":0.53300000000000003},"max":{"1min":1.6719999999999999,"5min":1.8520000000000001,"15min":2.8999999999999999},"last":1.619},{"interface":"front","average":{"1min":1.379,"5min":1.448,"15min":1.427},"min":{"1min":1.073,"5min":0.91700000000000004,"15min":0.58199999999999996},"max":{"1min":1.6299999999999999,"5min":1.978,"15min":2.0859999999999999},"last":1.6910000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.391,"15min":1.417},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.61399999999999999},"max":{"1min":1.6379999999999999,"5min":2.0680000000000001,"15min":2.1299999999999999},"last":2.0510000000000002},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.357,"15min":1.3720000000000001},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.59899999999999998},"max":{"1min":1.637,"5min":1.9970000000000001,"15min":3.218},"last":1.3109999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542315,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36716,"kb_used_data":5600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335124,"statfs":{"total":96636764160,"available":96599166976,"internally_reserved":0,"allocated":5734400,"data_stored":4865035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.091,"15min":1.115},"min":{"1min":0.71299999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.536,"5min":1.7450000000000001,"15min":3.9140000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.133,"5min":1.1879999999999999,"15min":1.161},"min":{"1min":0.72299999999999998,"5min":0.71099999999999997,"15min":0.45200000000000001},"max":{"1min":1.53,"5min":1.8600000000000001,"15min":6.6539999999999999},"last":0.90000000000000002}]},{"osd":1,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1819999999999999,"15min":1.2},"min":{"1min":0.84999999999999998,"5min":0.77300000000000002,"15min":0.69399999999999995},"max":{"1min":1.5469999999999999,"5min":1.627,"15min":2.4820000000000002},"last":1.458},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1559999999999999,"15min":1.1970000000000001},"min":{"1min":0.79100000000000004,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.857,"5min":1.857,"15min":2.0859999999999999},"last":1.603}]},{"osd":2,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.173,"15min":1.2090000000000001},"min":{"1min":0.93000000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.6299999999999999,"5min":1.649,"15min":3.73},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.2989999999999999,"15min":1.29},"min":{"1min":0.91000000000000003,"5min":0.73499999999999999,"15min":0.68100000000000005},"max":{"1min":1.452,"5min":1.8,"15min":2.3460000000000001},"last":2.0529999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.246,"15min":1.278},"min":{"1min":0.83799999999999997,"5min":0.76700000000000002,"15min":0.749},"max":{"1min":1.6160000000000001,"5min":1.738,"15min":12.914},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.3080000000000001,"15min":1.3260000000000001},"min":{"1min":0.85099999999999998,"5min":0.76100000000000001,"15min":0.57799999999999996},"max":{"1min":1.5069999999999999,"5min":1.8919999999999999,"15min":3.8100000000000001},"last":1.2569999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.3149999999999999,"15min":1.3300000000000001},"min":{"1min":1.123,"5min":0.68000000000000005,"15min":0.51100000000000001},"max":{"1min":1.5920000000000001,"5min":1.827,"15min":2.702},"last":2.294},{"interface":"front","average":{"1min":1.325,"5min":1.2989999999999999,"15min":1.353},"min":{"1min":0.95499999999999996,"5min":0.752,"15min":0.752},"max":{"1min":1.875,"5min":1.875,"15min":7.2140000000000004},"last":1.8899999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.399,"15min":1.45},"min":{"1min":1.143,"5min":0.92900000000000005,"15min":0.73899999999999999},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":3.097},"last":2.2519999999999998},{"interface":"front","average":{"1min":1.502,"5min":1.4510000000000001,"15min":1.466},"min":{"1min":1.355,"5min":0.86899999999999999,"15min":0.72499999999999998},"max":{"1min":1.6879999999999999,"5min":1.9139999999999999,"15min":3.3109999999999999},"last":2.1579999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.534,"15min":1.5860000000000001},"min":{"1min":1.2949999999999999,"5min":0.97799999999999998,"15min":0.80400000000000005},"max":{"1min":2.0150000000000001,"5min":2.0760000000000001,"15min":4.9729999999999999},"last":2.0019999999999998},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4390000000000001,"15min":1.464},"min":{"1min":0.998,"5min":0.70399999999999996,"15min":0.57699999999999996},"max":{"1min":2.0640000000000001,"5min":2.0649999999999999,"15min":4.085},"last":1.5669999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542315,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.149,"15min":1.1240000000000001},"min":{"1min":0.63,"5min":0.63,"15min":0.55400000000000005},"max":{"1min":1.526,"5min":2.0179999999999998,"15min":2.2999999999999998},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.115,"5min":1.1439999999999999,"15min":1.145},"min":{"1min":0.73999999999999999,"5min":0.624,"15min":0.47799999999999998},"max":{"1min":1.518,"5min":1.7050000000000001,"15min":2.355},"last":0.68000000000000005}]},{"osd":1,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1990000000000001,"15min":1.171},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":2.2559999999999998},"last":1.706},{"interface":"front","average":{"1min":1.173,"5min":1.1970000000000001,"15min":1.1859999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.495},"max":{"1min":1.5,"5min":1.599,"15min":1.8580000000000001},"last":1.3560000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1799999999999999,"15min":1.212},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.60199999999999998},"max":{"1min":1.605,"5min":1.6659999999999999,"15min":2.012},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.294,"15min":1.2529999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.64400000000000002},"max":{"1min":1.661,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.036}]},{"osd":3,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.27,"15min":1.284},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5209999999999999,"5min":1.6499999999999999,"15min":2.3799999999999999},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.371,"15min":1.3700000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.6379999999999999,"5min":2.0750000000000002,"15min":2.3999999999999999},"last":1.153}]},{"osd":4,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.304,"15min":1.3240000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.587,"5min":2.1499999999999999,"15min":2.2090000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4730000000000001,"15min":1.48},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.6240000000000001,"5min":1.8919999999999999,"15min":2.5009999999999999},"last":1.6659999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3759999999999999,"15min":1.393},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7,"5min":2.3170000000000002,"15min":2.4860000000000002},"last":1.546},{"interface":"front","average":{"1min":1.292,"5min":1.403,"15min":1.415},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.609,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.756}]},{"osd":7,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.4530000000000001,"15min":1.468},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.661,"5min":2.1549999999999998,"15min":2.7229999999999999},"last":1.825},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.5389999999999999,"15min":1.55},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.8640000000000001,"5min":2.2269999999999999,"15min":2.7509999999999999},"last":1.411}]}]},{"osd":4,"up_from":14,"seq":60129542315,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.2330000000000001,"15min":1.1830000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.8360000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.3049999999999999,"15min":1.2789999999999999},"min":{"1min":0.78900000000000003,"5min":0.74399999999999999,"15min":0.63700000000000001},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":1.5389999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.1539999999999999,"15min":1.1619999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.776,"5min":1.9350000000000001,"15min":4.327},"last":1.196},{"interface":"front","average":{"1min":1.129,"5min":1.1950000000000001,"15min":1.1619999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.4369999999999998},"last":0.83499999999999996}]},{"osd":2,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.177,"15min":1.202},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.624},"max":{"1min":1.6299999999999999,"5min":2.0960000000000001,"15min":2.6080000000000001},"last":1.389},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.2809999999999999,"15min":1.2689999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.71799999999999997},"max":{"1min":1.7090000000000001,"5min":1.8919999999999999,"15min":4.1310000000000002},"last":1.3400000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.92400000000000004,"5min":0.747,"15min":0.622},"max":{"1min":1.6100000000000001,"5min":1.659,"15min":2.5579999999999998},"last":1.242},{"interface":"front","average":{"1min":1.347,"5min":1.3740000000000001,"15min":1.3320000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.70199999999999996},"max":{"1min":1.835,"5min":2.024,"15min":4.2400000000000002},"last":1.5649999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.23,"15min":1.252},"min":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.74299999999999999},"max":{"1min":1.764,"5min":1.8740000000000001,"15min":2.508},"last":1.073},{"interface":"front","average":{"1min":1.456,"5min":1.3360000000000001,"15min":1.3620000000000001},"min":{"1min":0.92800000000000005,"5min":0.67800000000000005,"15min":0.60499999999999998},"max":{"1min":1.7010000000000001,"5min":1.7849999999999999,"15min":5.859},"last":1.4570000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3759999999999999,"15min":1.401},"min":{"1min":0.94799999999999995,"5min":0.753,"15min":0.58899999999999997},"max":{"1min":2.1099999999999999,"5min":2.194,"15min":5.3339999999999996},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.4670000000000001,"15min":1.4870000000000001},"min":{"1min":1.26,"5min":0.81000000000000005,"15min":0.72599999999999998},"max":{"1min":1.885,"5min":1.885,"15min":5.3940000000000001},"last":1.645}]},{"osd":7,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.4910000000000001,"15min":1.498},"min":{"1min":1.0269999999999999,"5min":0.84699999999999998,"15min":0.754},"max":{"1min":2.0059999999999998,"5min":2.0790000000000002,"15min":2.6859999999999999},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.476,"5min":1.3839999999999999,"15min":1.3859999999999999},"min":{"1min":1.0589999999999999,"5min":0.79000000000000004,"15min":0.75600000000000001},"max":{"1min":1.978,"5min":1.978,"15min":2.641},"last":1.504}]}]},{"osd":3,"up_from":14,"seq":60129542315,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36720,"kb_used_data":5604,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335120,"statfs":{"total":96636764160,"available":96599162880,"internally_reserved":0,"allocated":5738496,"data_stored":4865702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.88100000000000001,"15min":0.91800000000000004},"min":{"1min":0.42399999999999999,"5min":0.39300000000000002,"15min":0.32100000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.6819999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.1160000000000001,"15min":1.101},"min":{"1min":0.48099999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.5589999999999999,"5min":1.8540000000000001,"15min":2.0790000000000002},"last":1.083}]},{"osd":1,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.96899999999999997,"15min":0.98299999999999998},"min":{"1min":0.51900000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.472,"5min":1.5249999999999999,"15min":1.976},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.202,"15min":1.2270000000000001},"min":{"1min":0.65200000000000002,"5min":0.47099999999999997,"15min":0.44600000000000001},"max":{"1min":1.5069999999999999,"5min":1.774,"15min":2.2309999999999999},"last":1.2270000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.107,"15min":1.1120000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5669999999999999,"5min":1.7729999999999999,"15min":3.3109999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.165,"15min":1.1879999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.48399999999999999},"max":{"1min":1.532,"5min":1.635,"15min":2.7109999999999999},"last":1.121}]},{"osd":4,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.26,"15min":1.2609999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.73199999999999998},"max":{"1min":1.6040000000000001,"5min":1.752,"15min":2.3490000000000002},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.4370000000000001,"15min":1.456},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.905,"5min":2.081,"15min":4.8970000000000002},"last":1.421}]},{"osd":5,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.431,"15min":1.4670000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.7729999999999999,"5min":2.121,"15min":3.4910000000000001},"last":1.296},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.361,"15min":1.3959999999999999},"min":{"1min":0.77200000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.7909999999999999,"5min":1.8700000000000001,"15min":3.125},"last":1.1679999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.3640000000000001,"15min":1.409},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.9550000000000001,"5min":2.0630000000000002,"15min":4.5499999999999998},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.4379999999999999,"15min":1.4830000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.9339999999999999,"5min":2.2080000000000002,"15min":2.8450000000000002},"last":1.401}]},{"osd":7,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.4179999999999999,"15min":1.4450000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.879,"5min":1.998,"15min":2.952},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.484,"15min":1.524},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8100000000000001,"5min":1.964,"15min":4.798},"last":1.3109999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542315,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.878,"15min":0.89200000000000002},"min":{"1min":0.61099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":1.375,"5min":1.411,"15min":2.2509999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.1000000000000001,"15min":1.0660000000000001},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.35099999999999998},"max":{"1min":1.371,"5min":1.6759999999999999,"15min":1.8220000000000001},"last":0.71999999999999997}]},{"osd":1,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88900000000000001,"15min":0.89600000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.32000000000000001},"max":{"1min":1.3200000000000001,"5min":1.798,"15min":1.9430000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.05,"5min":1.069,"15min":1.036},"min":{"1min":0.66700000000000004,"5min":0.45000000000000001,"15min":0.35799999999999998},"max":{"1min":1.4550000000000001,"5min":1.5349999999999999,"15min":1.7230000000000001},"last":1.1020000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1220000000000001,"15min":1.111},"min":{"1min":0.72399999999999998,"5min":0.47399999999999998,"15min":0.38300000000000001},"max":{"1min":1.3580000000000001,"5min":1.758,"15min":1.98},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.008,"5min":1.0629999999999999,"15min":1.069},"min":{"1min":0.53200000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.524,"5min":1.6879999999999999,"15min":2.1560000000000001},"last":1.006}]},{"osd":4,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3520000000000001,"15min":1.3540000000000001},"min":{"1min":0.90400000000000003,"5min":0.86699999999999999,"15min":0.69599999999999995},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":2.1280000000000001},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.4279999999999999,"15min":1.411},"min":{"1min":0.95299999999999996,"5min":0.93799999999999994,"15min":0.83499999999999996},"max":{"1min":1.774,"5min":1.9079999999999999,"15min":2.085},"last":1.2809999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.353,"15min":1.359},"min":{"1min":1.0209999999999999,"5min":0.752,"15min":0.64100000000000001},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":2.5830000000000002},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.395,"5min":1.341,"15min":1.359},"min":{"1min":0.93899999999999995,"5min":0.73999999999999999,"15min":0.66000000000000003},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.6469999999999998},"last":1.5209999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4159999999999999,"15min":1.3899999999999999},"min":{"1min":0.94899999999999995,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.758},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4139999999999999,"15min":1.4350000000000001},"min":{"1min":1.0429999999999999,"5min":0.79100000000000004,"15min":0.69599999999999995},"max":{"1min":1.7869999999999999,"5min":1.883,"15min":2.8580000000000001},"last":1.226}]},{"osd":7,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.468,"15min":1.4430000000000001},"min":{"1min":0.94299999999999995,"5min":0.83099999999999996,"15min":0.71899999999999997},"max":{"1min":2.23,"5min":2.23,"15min":2.246},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.476,"15min":1.5029999999999999},"min":{"1min":1.0489999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.8819999999999999,"5min":1.9550000000000001,"15min":2.7120000000000002},"last":1.3120000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542316,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32632,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1552384,"data_stored":676495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.876,"15min":0.89400000000000002},"min":{"1min":0.41599999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.6579999999999999,"5min":1.6930000000000001,"15min":3.2959999999999998},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.002,"5min":1.0620000000000001,"15min":1.0149999999999999},"min":{"1min":0.5,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.53,"5min":1.8480000000000001,"15min":1.99},"last":1.2669999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.94699999999999995},"min":{"1min":0.51200000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2949999999999999,"5min":1.5549999999999999,"15min":1.738},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.105,"15min":1.1040000000000001},"min":{"1min":0.60499999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5109999999999999,"5min":1.702,"15min":2.2160000000000002},"last":1.04}]},{"osd":3,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2250000000000001,"15min":1.222},"min":{"1min":0.60699999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.734,"5min":1.8819999999999999,"15min":2.4430000000000001},"last":1.661},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.133,"15min":1.1220000000000001},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.434},"max":{"1min":1.637,"5min":2.0539999999999998,"15min":2.4169999999999998},"last":1.411}]},{"osd":4,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.345,"15min":1.3140000000000001},"min":{"1min":0.92100000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7190000000000001,"5min":1.8080000000000001,"15min":2.153},"last":1.9890000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.286,"15min":1.3},"min":{"1min":0.747,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":2.758},"last":1.1120000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.357},"min":{"1min":0.81100000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":2.004,"5min":2.004,"15min":2.6589999999999998},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.4239999999999999,"15min":1.4259999999999999},"min":{"1min":0.97599999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.8109999999999999,"5min":1.9610000000000001,"15min":3.0329999999999999},"last":1.54}]},{"osd":6,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4119999999999999,"15min":1.4179999999999999},"min":{"1min":1.0369999999999999,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.984,"5min":1.984,"15min":2.2730000000000001},"last":2.0550000000000002},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4359999999999999,"15min":1.4650000000000001},"min":{"1min":0.91100000000000003,"5min":0.747,"15min":0.747},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.9630000000000001},"last":1.3919999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.524,"15min":1.528},"min":{"1min":0.93500000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.1269999999999998,"5min":2.2450000000000001,"15min":3.2749999999999999},"last":1.77},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4990000000000001,"15min":1.5249999999999999},"min":{"1min":1.0740000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.7829999999999999},"last":1.857}]}]},{"osd":0,"up_from":14,"seq":60129542314,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37296,"kb_used_data":6180,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94334544,"statfs":{"total":96636764160,"available":96598573056,"internally_reserved":0,"allocated":6328320,"data_stored":5455401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.85199999999999998,"15min":0.85799999999999998},"min":{"1min":0.55100000000000005,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.262,"5min":1.8660000000000001,"15min":2.8479999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":0.97599999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.30399999999999999},"max":{"1min":1.694,"5min":1.694,"15min":2.4319999999999999},"last":1.016}]},{"osd":2,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0269999999999999,"15min":1.0349999999999999},"min":{"1min":0.55800000000000005,"5min":0.41799999999999998,"15min":0.33700000000000002},"max":{"1min":1.849,"5min":1.849,"15min":2.4980000000000002},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.1180000000000001,"15min":1.069},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.432},"max":{"1min":1.5820000000000001,"5min":1.7569999999999999,"15min":2.5760000000000001},"last":1.448}]},{"osd":3,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1579999999999999,"15min":1.123},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.35799999999999998},"max":{"1min":1.552,"5min":1.7589999999999999,"15min":2.4580000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3460000000000001,"15min":1.3200000000000001},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.48599999999999999},"max":{"1min":1.6519999999999999,"5min":1.968,"15min":2.9390000000000001},"last":1.536}]},{"osd":4,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3320000000000001,"15min":1.3360000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.7410000000000001,"5min":1.855,"15min":2.9750000000000001},"last":1.78},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3360000000000001,"15min":1.298},"min":{"1min":0.88500000000000001,"5min":0.81399999999999995,"15min":0.39900000000000002},"max":{"1min":1.728,"5min":1.968,"15min":3.125},"last":1.3959999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.361,"15min":1.3660000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.51200000000000001},"max":{"1min":1.595,"5min":1.925,"15min":3.0830000000000002},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.3400000000000001,"15min":1.4079999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.52700000000000002},"max":{"1min":1.772,"5min":2.0449999999999999,"15min":3.1899999999999999},"last":1.2909999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.4139999999999999,"15min":1.448},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.47999999999999998},"max":{"1min":1.77,"5min":2.1030000000000002,"15min":3.391},"last":1.712},{"interface":"front","average":{"1min":1.367,"5min":1.4259999999999999,"15min":1.4079999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.58399999999999996},"max":{"1min":1.956,"5min":2.2170000000000001,"15min":3.153},"last":1.6679999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.5349999999999999,"15min":1.5189999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.60599999999999998},"max":{"1min":1.877,"5min":2.1400000000000001,"15min":3.5350000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.516,"15min":1.494},"min":{"1min":0.92600000000000005,"5min":0.78300000000000003,"15min":0.45300000000000001},"max":{"1min":1.7090000000000001,"5min":2.2610000000000001,"15min":3.3650000000000002},"last":1.583}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:32:51.530 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-23T11:32:51.530 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:32:51.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.751+0000 7feb5be00640 1 Processor -- start 2024-03-23T11:32:51.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.751+0000 7feb5be00640 1 -- start start 2024-03-23T11:32:51.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.753+0000 7feb5be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feb5c159490 0x7feb5c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.753+0000 7feb5be00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c0ced90 con 0x7feb5c15da90 2024-03-23T11:32:51.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.753+0000 7feb5be00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c06e3e0 con 0x7feb5c0ce5f0 2024-03-23T11:32:51.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.753+0000 7feb5be00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5c06c1c0 con 0x7feb5c159490 2024-03-23T11:32:51.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.753+0000 7feb53e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7feb5c15da90 0x7feb5c0cdee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35532/0 (socket says 172.21.15.62:35532) 2024-03-23T11:32:51.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.753+0000 7feb53e00640 1 -- 172.21.15.62:0/1052998651 learned_addr learned my addr 172.21.15.62:0/1052998651 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:51.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5aa00640 1 --2- 172.21.15.62:0/1052998651 >> v2:172.21.15.120:3300/0 conn(0x7feb5c159490 0x7feb5c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818859303 0 0) 0x7feb5c0ced90 con 0x7feb5c15da90 2024-03-23T11:32:51.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb3c005730 con 0x7feb5c15da90 2024-03-23T11:32:51.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 327650454 0 0) 0x7feb3c005730 con 0x7feb5c15da90 2024-03-23T11:32:51.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 >> v2:172.21.15.120:3300/0 conn(0x7feb5c159490 msgr2=0x7feb5c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:51.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 --2- 172.21.15.62:0/1052998651 >> v2:172.21.15.120:3300/0 conn(0x7feb5c159490 0x7feb5c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 >> v1:172.21.15.120:6789/0 conn(0x7feb5c0ce5f0 legacy=0x7feb5c0b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c06e100 con 0x7feb5c15da90 2024-03-23T11:32:51.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7feb44001670 con 0x7feb5c15da90 2024-03-23T11:32:51.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 >> v1:172.21.15.62:6789/0 conn(0x7feb5c15da90 legacy=0x7feb5c0cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb3c005b80 0x7feb3c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3c0069d0 0x7feb3c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 0x7feb3c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c06c1c0 con 0x7feb3c007390 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c00e330 con 0x7feb3c005b80 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.754+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c00e4f0 con 0x7feb3c0069d0 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.755+0000 7feb5b400640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 0x7feb3c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.755+0000 7feb5aa00640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb3c005b80 0x7feb3c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.755+0000 7feb53e00640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3c0069d0 0x7feb3c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.755+0000 7feb5b400640 1 -- 172.21.15.62:0/1052998651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3c0069d0 msgr2=0x7feb3c006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.755+0000 7feb5b400640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3c0069d0 0x7feb3c006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.755+0000 7feb5b400640 1 -- 172.21.15.62:0/1052998651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb3c005b80 msgr2=0x7feb3c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.755+0000 7feb5b400640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb3c005b80 0x7feb3c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.755+0000 7feb5b400640 1 -- 172.21.15.62:0/1052998651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c005730 con 0x7feb3c007390 2024-03-23T11:32:51.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.755+0000 7feb53e00640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3c0069d0 0x7feb3c006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.756+0000 7feb5b400640 1 -- 172.21.15.62:0/1052998651 reap_dead start 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.756+0000 7feb5b400640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 0x7feb3c00db90 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7feb54001e50 tx=0x7feb3c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20571cf1ed38cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.757+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb54003b70 con 0x7feb3c007390 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.757+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb54003d10 con 0x7feb3c007390 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.757+0000 7feb5a000640 1 -- 172.21.15.62:0/1052998651 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb54005310 con 0x7feb3c007390 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.757+0000 7feb5be00640 1 -- 172.21.15.62:0/1052998651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 msgr2=0x7feb3c00db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.757+0000 7feb5be00640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 0x7feb3c00db90 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7feb54001e50 tx=0x7feb3c007a00 comp rx=0 tx=0).stop 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.757+0000 7feb5be00640 1 -- 172.21.15.62:0/1052998651 shutdown_connections 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.757+0000 7feb5be00640 1 --2- 172.21.15.62:0/1052998651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 0x7feb3c00db90 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.757+0000 7feb5be00640 1 -- 172.21.15.62:0/1052998651 >> 172.21.15.62:0/1052998651 conn(0x7feb5c066050 msgr2=0x7feb5c069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.757+0000 7feb5be00640 1 -- 172.21.15.62:0/1052998651 shutdown_connections 2024-03-23T11:32:51.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.757+0000 7feb5be00640 1 -- 172.21.15.62:0/1052998651 wait complete. 2024-03-23T11:32:51.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.759+0000 7feb5be00640 1 Processor -- start 2024-03-23T11:32:51.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.759+0000 7feb5be00640 1 -- start start 2024-03-23T11:32:51.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.760+0000 7feb5be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 0x7feb5c14aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.760+0000 7feb5be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb5c14a5b0 0x7feb5c149e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.760+0000 7feb5be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb5c152930 0x7feb5c152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.760+0000 7feb5be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c1532f0 con 0x7feb3c007390 2024-03-23T11:32:51.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.760+0000 7feb5be00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c15b0b0 con 0x7feb5c14a5b0 2024-03-23T11:32:51.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.760+0000 7feb5be00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c151d20 con 0x7feb5c152930 2024-03-23T11:32:51.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.760+0000 7feb5aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 0x7feb5c14aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.760+0000 7feb5aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 0x7feb5c14aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39656/0 (socket says 172.21.15.62:39656) 2024-03-23T11:32:51.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.760+0000 7feb5aa00640 1 -- 172.21.15.62:0/3539336107 learned_addr learned my addr 172.21.15.62:0/3539336107 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:51.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.761+0000 7feb53e00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb5c14a5b0 0x7feb5c149e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.761+0000 7feb5b400640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb5c152930 0x7feb5c152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.761+0000 7feb5aa00640 1 -- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb5c152930 msgr2=0x7feb5c152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.761+0000 7feb5aa00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb5c152930 0x7feb5c152db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.761+0000 7feb5aa00640 1 -- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb5c14a5b0 msgr2=0x7feb5c149e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.761+0000 7feb5aa00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb5c14a5b0 0x7feb5c149e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.761+0000 7feb5aa00640 1 -- 172.21.15.62:0/3539336107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c14a370 con 0x7feb3c007390 2024-03-23T11:32:51.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.761+0000 7feb5aa00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 0x7feb5c14aae0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7feb4c011540 tx=0x7feb4c011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:51.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.762+0000 7feb52a00640 1 -- 172.21.15.62:0/3539336107 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c015930 con 0x7feb3c007390 2024-03-23T11:32:51.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.762+0000 7feb52a00640 1 -- 172.21.15.62:0/3539336107 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4c015a90 con 0x7feb3c007390 2024-03-23T11:32:51.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.762+0000 7feb52a00640 1 -- 172.21.15.62:0/3539336107 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c015e60 con 0x7feb3c007390 2024-03-23T11:32:51.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.762+0000 7feb5be00640 1 -- 172.21.15.62:0/3539336107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5c1511a0 con 0x7feb3c007390 2024-03-23T11:32:51.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.762+0000 7feb5be00640 1 -- 172.21.15.62:0/3539336107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5c151760 con 0x7feb3c007390 2024-03-23T11:32:51.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.763+0000 7feb5be00640 1 -- 172.21.15.62:0/3539336107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5c066bb0 con 0x7feb3c007390 2024-03-23T11:32:51.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.764+0000 7feb52a00640 1 -- 172.21.15.62:0/3539336107 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7feb4c00f460 con 0x7feb3c007390 2024-03-23T11:32:51.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.765+0000 7feb52a00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb3806a7d0 0x7feb3806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:51.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.765+0000 7feb52a00640 1 -- 172.21.15.62:0/3539336107 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7feb4c0892c0 con 0x7feb3c007390 2024-03-23T11:32:51.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.765+0000 7feb53e00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb3806a7d0 0x7feb3806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:51.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.766+0000 7feb53e00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb3806a7d0 0x7feb3806cc90 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7feb5c158470 tx=0x7feb44009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:51.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.769+0000 7feb52a00640 1 -- 172.21.15.62:0/3539336107 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feb4c0549c0 con 0x7feb3c007390 2024-03-23T11:32:51.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.964+0000 7feb5be00640 1 -- 172.21.15.62:0/3539336107 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feb5c159870 con 0x7feb3806a7d0 2024-03-23T11:32:51.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.970+0000 7feb52a00640 1 -- 172.21.15.62:0/3539336107 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48102 (secure 0 0 0) 0x7feb4400d050 con 0x7feb3806a7d0 2024-03-23T11:32:51.960 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:51.960 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:32:51.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.973+0000 7feb5be00640 1 -- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb3806a7d0 msgr2=0x7feb3806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.973+0000 7feb5be00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb3806a7d0 0x7feb3806cc90 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7feb5c158470 tx=0x7feb44009040 comp rx=0 tx=0).stop 2024-03-23T11:32:51.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.973+0000 7feb5be00640 1 -- 172.21.15.62:0/3539336107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 msgr2=0x7feb5c14aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:51.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.973+0000 7feb5be00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 0x7feb5c14aae0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7feb4c011540 tx=0x7feb4c011a00 comp rx=0 tx=0).stop 2024-03-23T11:32:51.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.973+0000 7feb5be00640 1 -- 172.21.15.62:0/3539336107 shutdown_connections 2024-03-23T11:32:51.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.973+0000 7feb5be00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7feb3806a7d0 0x7feb3806cc90 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.973+0000 7feb5be00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb5c152930 0x7feb5c152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.973+0000 7feb5be00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb5c14a5b0 0x7feb5c149e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.973+0000 7feb5be00640 1 --2- 172.21.15.62:0/3539336107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb3c007390 0x7feb5c14aae0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:51.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.973+0000 7feb5be00640 1 -- 172.21.15.62:0/3539336107 >> 172.21.15.62:0/3539336107 conn(0x7feb5c066050 msgr2=0x7feb5c15d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:51.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.973+0000 7feb5be00640 1 -- 172.21.15.62:0/3539336107 shutdown_connections 2024-03-23T11:32:51.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:51.974+0000 7feb5be00640 1 -- 172.21.15.62:0/3539336107 wait complete. 2024-03-23T11:32:51.981 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":540,"stamp":"2024-03-23T11:32:51.544718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":274504,"kb_used_data":25560,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754700216,"statfs":{"total":773094113280,"available":772813021184,"internally_reserved":0,"allocated":26173440,"data_stored":19161657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8392377,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-264,"num_write":-179,"num_write_kb":-8297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.083094"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":273,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-23T11:32:46.530200+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:32:46.530200+0000","last_peered":"2024-03-23T11:32:46.530200+0000","last_clean":"2024-03-23T11:32:46.530200+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:32:46.530200+0000","last_undegraded":"2024-03-23T11:32:46.530200+0000","last_fullsized":"2024-03-23T11:32:46.530200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542315,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33220,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338620,"statfs":{"total":96636764160,"available":96602746880,"internally_reserved":0,"allocated":2150400,"data_stored":1267532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.087,"15min":1.028},"min":{"1min":0.63600000000000001,"5min":0.623,"15min":0.5},"max":{"1min":1.456,"5min":1.631,"15min":2.8370000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.075,"5min":1.1220000000000001,"15min":1.0589999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.38300000000000001},"max":{"1min":1.5449999999999999,"5min":1.7230000000000001,"15min":1.8009999999999999},"last":1.0249999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.2809999999999999,"15min":1.214},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.46100000000000002},"max":{"1min":1.5720000000000001,"5min":1.708,"15min":3.569},"last":1.254},{"interface":"front","average":{"1min":1.208,"5min":1.2909999999999999,"15min":1.2669999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.69899999999999995},"max":{"1min":1.5309999999999999,"5min":1.8009999999999999,"15min":2.4350000000000001},"last":1.444}]},{"osd":2,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1299999999999999,"15min":1.0960000000000001},"min":{"1min":0.69899999999999995,"5min":0.65500000000000003,"15min":0.58099999999999996},"max":{"1min":1.4610000000000001,"5min":1.909,"15min":2.7440000000000002},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.23,"15min":1.1990000000000001},"min":{"1min":0.83799999999999997,"5min":0.81499999999999995,"15min":0.56999999999999995},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":3.524},"last":1.3839999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.204,"15min":1.1930000000000001},"min":{"1min":0.86799999999999999,"5min":0.72199999999999998,"15min":0.67600000000000005},"max":{"1min":1.5169999999999999,"5min":1.7889999999999999,"15min":2.6389999999999998},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.3169999999999999,"15min":1.2809999999999999},"min":{"1min":0.94399999999999995,"5min":0.93500000000000005,"15min":0.60199999999999998},"max":{"1min":1.641,"5min":1.7050000000000001,"15min":2.9470000000000001},"last":1.101}]},{"osd":4,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1679999999999999,"15min":1.208},"min":{"1min":0.84099999999999997,"5min":0.73199999999999998,"15min":0.54300000000000004},"max":{"1min":1.534,"5min":2.1560000000000001,"15min":2.2719999999999998},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.391,"15min":1.3979999999999999},"min":{"1min":1.071,"5min":0.81699999999999995,"15min":0.60199999999999998},"max":{"1min":1.7,"5min":2.0819999999999999,"15min":2.5569999999999999},"last":1.506}]},{"osd":5,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.369,"15min":1.351},"min":{"1min":0.745,"5min":0.745,"15min":0.53300000000000003},"max":{"1min":1.6719999999999999,"5min":1.8520000000000001,"15min":2.8999999999999999},"last":1.619},{"interface":"front","average":{"1min":1.379,"5min":1.448,"15min":1.427},"min":{"1min":1.073,"5min":0.91700000000000004,"15min":0.58199999999999996},"max":{"1min":1.6299999999999999,"5min":1.978,"15min":2.0859999999999999},"last":1.6910000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.391,"15min":1.417},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.61399999999999999},"max":{"1min":1.6379999999999999,"5min":2.0680000000000001,"15min":2.1299999999999999},"last":2.0510000000000002},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.357,"15min":1.3720000000000001},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.59899999999999998},"max":{"1min":1.637,"5min":1.9970000000000001,"15min":3.218},"last":1.3109999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542315,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36716,"kb_used_data":5600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335124,"statfs":{"total":96636764160,"available":96599166976,"internally_reserved":0,"allocated":5734400,"data_stored":4865035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.091,"15min":1.115},"min":{"1min":0.71299999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.536,"5min":1.7450000000000001,"15min":3.9140000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.133,"5min":1.1879999999999999,"15min":1.161},"min":{"1min":0.72299999999999998,"5min":0.71099999999999997,"15min":0.45200000000000001},"max":{"1min":1.53,"5min":1.8600000000000001,"15min":6.6539999999999999},"last":0.90000000000000002}]},{"osd":1,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1819999999999999,"15min":1.2},"min":{"1min":0.84999999999999998,"5min":0.77300000000000002,"15min":0.69399999999999995},"max":{"1min":1.5469999999999999,"5min":1.627,"15min":2.4820000000000002},"last":1.458},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1559999999999999,"15min":1.1970000000000001},"min":{"1min":0.79100000000000004,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.857,"5min":1.857,"15min":2.0859999999999999},"last":1.603}]},{"osd":2,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.173,"15min":1.2090000000000001},"min":{"1min":0.93000000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.6299999999999999,"5min":1.649,"15min":3.73},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.2989999999999999,"15min":1.29},"min":{"1min":0.91000000000000003,"5min":0.73499999999999999,"15min":0.68100000000000005},"max":{"1min":1.452,"5min":1.8,"15min":2.3460000000000001},"last":2.0529999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.246,"15min":1.278},"min":{"1min":0.83799999999999997,"5min":0.76700000000000002,"15min":0.749},"max":{"1min":1.6160000000000001,"5min":1.738,"15min":12.914},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.3080000000000001,"15min":1.3260000000000001},"min":{"1min":0.85099999999999998,"5min":0.76100000000000001,"15min":0.57799999999999996},"max":{"1min":1.5069999999999999,"5min":1.8919999999999999,"15min":3.8100000000000001},"last":1.2569999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.3149999999999999,"15min":1.3300000000000001},"min":{"1min":1.123,"5min":0.68000000000000005,"15min":0.51100000000000001},"max":{"1min":1.5920000000000001,"5min":1.827,"15min":2.702},"last":2.294},{"interface":"front","average":{"1min":1.325,"5min":1.2989999999999999,"15min":1.353},"min":{"1min":0.95499999999999996,"5min":0.752,"15min":0.752},"max":{"1min":1.875,"5min":1.875,"15min":7.2140000000000004},"last":1.8899999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.399,"15min":1.45},"min":{"1min":1.143,"5min":0.92900000000000005,"15min":0.73899999999999999},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":3.097},"last":2.2519999999999998},{"interface":"front","average":{"1min":1.502,"5min":1.4510000000000001,"15min":1.466},"min":{"1min":1.355,"5min":0.86899999999999999,"15min":0.72499999999999998},"max":{"1min":1.6879999999999999,"5min":1.9139999999999999,"15min":3.3109999999999999},"last":2.1579999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.534,"15min":1.5860000000000001},"min":{"1min":1.2949999999999999,"5min":0.97799999999999998,"15min":0.80400000000000005},"max":{"1min":2.0150000000000001,"5min":2.0760000000000001,"15min":4.9729999999999999},"last":2.0019999999999998},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4390000000000001,"15min":1.464},"min":{"1min":0.998,"5min":0.70399999999999996,"15min":0.57699999999999996},"max":{"1min":2.0640000000000001,"5min":2.0649999999999999,"15min":4.085},"last":1.5669999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542315,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.149,"15min":1.1240000000000001},"min":{"1min":0.63,"5min":0.63,"15min":0.55400000000000005},"max":{"1min":1.526,"5min":2.0179999999999998,"15min":2.2999999999999998},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.115,"5min":1.1439999999999999,"15min":1.145},"min":{"1min":0.73999999999999999,"5min":0.624,"15min":0.47799999999999998},"max":{"1min":1.518,"5min":1.7050000000000001,"15min":2.355},"last":0.68000000000000005}]},{"osd":1,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1990000000000001,"15min":1.171},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":2.2559999999999998},"last":1.706},{"interface":"front","average":{"1min":1.173,"5min":1.1970000000000001,"15min":1.1859999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.495},"max":{"1min":1.5,"5min":1.599,"15min":1.8580000000000001},"last":1.3560000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1799999999999999,"15min":1.212},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.60199999999999998},"max":{"1min":1.605,"5min":1.6659999999999999,"15min":2.012},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.294,"15min":1.2529999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.64400000000000002},"max":{"1min":1.661,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.036}]},{"osd":3,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.27,"15min":1.284},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5209999999999999,"5min":1.6499999999999999,"15min":2.3799999999999999},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.371,"15min":1.3700000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.6379999999999999,"5min":2.0750000000000002,"15min":2.3999999999999999},"last":1.153}]},{"osd":4,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.304,"15min":1.3240000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.587,"5min":2.1499999999999999,"15min":2.2090000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4730000000000001,"15min":1.48},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.6240000000000001,"5min":1.8919999999999999,"15min":2.5009999999999999},"last":1.6659999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3759999999999999,"15min":1.393},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7,"5min":2.3170000000000002,"15min":2.4860000000000002},"last":1.546},{"interface":"front","average":{"1min":1.292,"5min":1.403,"15min":1.415},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.609,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.756}]},{"osd":7,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.4530000000000001,"15min":1.468},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.661,"5min":2.1549999999999998,"15min":2.7229999999999999},"last":1.825},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.5389999999999999,"15min":1.55},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.8640000000000001,"5min":2.2269999999999999,"15min":2.7509999999999999},"last":1.411}]}]},{"osd":4,"up_from":14,"seq":60129542315,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.2330000000000001,"15min":1.1830000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.8360000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.3049999999999999,"15min":1.2789999999999999},"min":{"1min":0.78900000000000003,"5min":0.74399999999999999,"15min":0.63700000000000001},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":1.5389999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.1539999999999999,"15min":1.1619999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.776,"5min":1.9350000000000001,"15min":4.327},"last":1.196},{"interface":"front","average":{"1min":1.129,"5min":1.1950000000000001,"15min":1.1619999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.4369999999999998},"last":0.83499999999999996}]},{"osd":2,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.177,"15min":1.202},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.624},"max":{"1min":1.6299999999999999,"5min":2.0960000000000001,"15min":2.6080000000000001},"last":1.389},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.2809999999999999,"15min":1.2689999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.71799999999999997},"max":{"1min":1.7090000000000001,"5min":1.8919999999999999,"15min":4.1310000000000002},"last":1.3400000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.92400000000000004,"5min":0.747,"15min":0.622},"max":{"1min":1.6100000000000001,"5min":1.659,"15min":2.5579999999999998},"last":1.242},{"interface":"front","average":{"1min":1.347,"5min":1.3740000000000001,"15min":1.3320000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.70199999999999996},"max":{"1min":1.835,"5min":2.024,"15min":4.2400000000000002},"last":1.5649999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.23,"15min":1.252},"min":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.74299999999999999},"max":{"1min":1.764,"5min":1.8740000000000001,"15min":2.508},"last":1.073},{"interface":"front","average":{"1min":1.456,"5min":1.3360000000000001,"15min":1.3620000000000001},"min":{"1min":0.92800000000000005,"5min":0.67800000000000005,"15min":0.60499999999999998},"max":{"1min":1.7010000000000001,"5min":1.7849999999999999,"15min":5.859},"last":1.4570000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3759999999999999,"15min":1.401},"min":{"1min":0.94799999999999995,"5min":0.753,"15min":0.58899999999999997},"max":{"1min":2.1099999999999999,"5min":2.194,"15min":5.3339999999999996},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.4670000000000001,"15min":1.4870000000000001},"min":{"1min":1.26,"5min":0.81000000000000005,"15min":0.72599999999999998},"max":{"1min":1.885,"5min":1.885,"15min":5.3940000000000001},"last":1.645}]},{"osd":7,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.4910000000000001,"15min":1.498},"min":{"1min":1.0269999999999999,"5min":0.84699999999999998,"15min":0.754},"max":{"1min":2.0059999999999998,"5min":2.0790000000000002,"15min":2.6859999999999999},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.476,"5min":1.3839999999999999,"15min":1.3859999999999999},"min":{"1min":1.0589999999999999,"5min":0.79000000000000004,"15min":0.75600000000000001},"max":{"1min":1.978,"5min":1.978,"15min":2.641},"last":1.504}]}]},{"osd":3,"up_from":14,"seq":60129542315,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36720,"kb_used_data":5604,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335120,"statfs":{"total":96636764160,"available":96599162880,"internally_reserved":0,"allocated":5738496,"data_stored":4865702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.88100000000000001,"15min":0.91800000000000004},"min":{"1min":0.42399999999999999,"5min":0.39300000000000002,"15min":0.32100000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.6819999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.1160000000000001,"15min":1.101},"min":{"1min":0.48099999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.5589999999999999,"5min":1.8540000000000001,"15min":2.0790000000000002},"last":1.083}]},{"osd":1,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.96899999999999997,"15min":0.98299999999999998},"min":{"1min":0.51900000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.472,"5min":1.5249999999999999,"15min":1.976},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.202,"15min":1.2270000000000001},"min":{"1min":0.65200000000000002,"5min":0.47099999999999997,"15min":0.44600000000000001},"max":{"1min":1.5069999999999999,"5min":1.774,"15min":2.2309999999999999},"last":1.2270000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.107,"15min":1.1120000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5669999999999999,"5min":1.7729999999999999,"15min":3.3109999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.165,"15min":1.1879999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.48399999999999999},"max":{"1min":1.532,"5min":1.635,"15min":2.7109999999999999},"last":1.121}]},{"osd":4,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.26,"15min":1.2609999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.73199999999999998},"max":{"1min":1.6040000000000001,"5min":1.752,"15min":2.3490000000000002},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.4370000000000001,"15min":1.456},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.905,"5min":2.081,"15min":4.8970000000000002},"last":1.421}]},{"osd":5,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.431,"15min":1.4670000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.7729999999999999,"5min":2.121,"15min":3.4910000000000001},"last":1.296},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.361,"15min":1.3959999999999999},"min":{"1min":0.77200000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.7909999999999999,"5min":1.8700000000000001,"15min":3.125},"last":1.1679999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.3640000000000001,"15min":1.409},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.9550000000000001,"5min":2.0630000000000002,"15min":4.5499999999999998},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.4379999999999999,"15min":1.4830000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.9339999999999999,"5min":2.2080000000000002,"15min":2.8450000000000002},"last":1.401}]},{"osd":7,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.4179999999999999,"15min":1.4450000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.879,"5min":1.998,"15min":2.952},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.484,"15min":1.524},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8100000000000001,"5min":1.964,"15min":4.798},"last":1.3109999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542315,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.878,"15min":0.89200000000000002},"min":{"1min":0.61099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":1.375,"5min":1.411,"15min":2.2509999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.1000000000000001,"15min":1.0660000000000001},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.35099999999999998},"max":{"1min":1.371,"5min":1.6759999999999999,"15min":1.8220000000000001},"last":0.71999999999999997}]},{"osd":1,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88900000000000001,"15min":0.89600000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.32000000000000001},"max":{"1min":1.3200000000000001,"5min":1.798,"15min":1.9430000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.05,"5min":1.069,"15min":1.036},"min":{"1min":0.66700000000000004,"5min":0.45000000000000001,"15min":0.35799999999999998},"max":{"1min":1.4550000000000001,"5min":1.5349999999999999,"15min":1.7230000000000001},"last":1.1020000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1220000000000001,"15min":1.111},"min":{"1min":0.72399999999999998,"5min":0.47399999999999998,"15min":0.38300000000000001},"max":{"1min":1.3580000000000001,"5min":1.758,"15min":1.98},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.008,"5min":1.0629999999999999,"15min":1.069},"min":{"1min":0.53200000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.524,"5min":1.6879999999999999,"15min":2.1560000000000001},"last":1.006}]},{"osd":4,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3520000000000001,"15min":1.3540000000000001},"min":{"1min":0.90400000000000003,"5min":0.86699999999999999,"15min":0.69599999999999995},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":2.1280000000000001},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.4279999999999999,"15min":1.411},"min":{"1min":0.95299999999999996,"5min":0.93799999999999994,"15min":0.83499999999999996},"max":{"1min":1.774,"5min":1.9079999999999999,"15min":2.085},"last":1.2809999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.353,"15min":1.359},"min":{"1min":1.0209999999999999,"5min":0.752,"15min":0.64100000000000001},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":2.5830000000000002},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.395,"5min":1.341,"15min":1.359},"min":{"1min":0.93899999999999995,"5min":0.73999999999999999,"15min":0.66000000000000003},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.6469999999999998},"last":1.5209999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4159999999999999,"15min":1.3899999999999999},"min":{"1min":0.94899999999999995,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.758},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4139999999999999,"15min":1.4350000000000001},"min":{"1min":1.0429999999999999,"5min":0.79100000000000004,"15min":0.69599999999999995},"max":{"1min":1.7869999999999999,"5min":1.883,"15min":2.8580000000000001},"last":1.226}]},{"osd":7,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.468,"15min":1.4430000000000001},"min":{"1min":0.94299999999999995,"5min":0.83099999999999996,"15min":0.71899999999999997},"max":{"1min":2.23,"5min":2.23,"15min":2.246},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.476,"15min":1.5029999999999999},"min":{"1min":1.0489999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.8819999999999999,"5min":1.9550000000000001,"15min":2.7120000000000002},"last":1.3120000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542316,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32632,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1552384,"data_stored":676495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.876,"15min":0.89400000000000002},"min":{"1min":0.41599999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.6579999999999999,"5min":1.6930000000000001,"15min":3.2959999999999998},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.002,"5min":1.0620000000000001,"15min":1.0149999999999999},"min":{"1min":0.5,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.53,"5min":1.8480000000000001,"15min":1.99},"last":1.2669999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.94699999999999995},"min":{"1min":0.51200000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2949999999999999,"5min":1.5549999999999999,"15min":1.738},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.105,"15min":1.1040000000000001},"min":{"1min":0.60499999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5109999999999999,"5min":1.702,"15min":2.2160000000000002},"last":1.04}]},{"osd":3,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2250000000000001,"15min":1.222},"min":{"1min":0.60699999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.734,"5min":1.8819999999999999,"15min":2.4430000000000001},"last":1.661},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.133,"15min":1.1220000000000001},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.434},"max":{"1min":1.637,"5min":2.0539999999999998,"15min":2.4169999999999998},"last":1.411}]},{"osd":4,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.345,"15min":1.3140000000000001},"min":{"1min":0.92100000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7190000000000001,"5min":1.8080000000000001,"15min":2.153},"last":1.9890000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.286,"15min":1.3},"min":{"1min":0.747,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":2.758},"last":1.1120000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.357},"min":{"1min":0.81100000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":2.004,"5min":2.004,"15min":2.6589999999999998},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.4239999999999999,"15min":1.4259999999999999},"min":{"1min":0.97599999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.8109999999999999,"5min":1.9610000000000001,"15min":3.0329999999999999},"last":1.54}]},{"osd":6,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4119999999999999,"15min":1.4179999999999999},"min":{"1min":1.0369999999999999,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.984,"5min":1.984,"15min":2.2730000000000001},"last":2.0550000000000002},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4359999999999999,"15min":1.4650000000000001},"min":{"1min":0.91100000000000003,"5min":0.747,"15min":0.747},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.9630000000000001},"last":1.3919999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.524,"15min":1.528},"min":{"1min":0.93500000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.1269999999999998,"5min":2.2450000000000001,"15min":3.2749999999999999},"last":1.77},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4990000000000001,"15min":1.5249999999999999},"min":{"1min":1.0740000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.7829999999999999},"last":1.857}]}]},{"osd":0,"up_from":14,"seq":60129542314,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37296,"kb_used_data":6180,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94334544,"statfs":{"total":96636764160,"available":96598573056,"internally_reserved":0,"allocated":6328320,"data_stored":5455401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.85199999999999998,"15min":0.85799999999999998},"min":{"1min":0.55100000000000005,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.262,"5min":1.8660000000000001,"15min":2.8479999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":0.97599999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.30399999999999999},"max":{"1min":1.694,"5min":1.694,"15min":2.4319999999999999},"last":1.016}]},{"osd":2,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0269999999999999,"15min":1.0349999999999999},"min":{"1min":0.55800000000000005,"5min":0.41799999999999998,"15min":0.33700000000000002},"max":{"1min":1.849,"5min":1.849,"15min":2.4980000000000002},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.1180000000000001,"15min":1.069},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.432},"max":{"1min":1.5820000000000001,"5min":1.7569999999999999,"15min":2.5760000000000001},"last":1.448}]},{"osd":3,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1579999999999999,"15min":1.123},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.35799999999999998},"max":{"1min":1.552,"5min":1.7589999999999999,"15min":2.4580000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3460000000000001,"15min":1.3200000000000001},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.48599999999999999},"max":{"1min":1.6519999999999999,"5min":1.968,"15min":2.9390000000000001},"last":1.536}]},{"osd":4,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3320000000000001,"15min":1.3360000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.7410000000000001,"5min":1.855,"15min":2.9750000000000001},"last":1.78},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3360000000000001,"15min":1.298},"min":{"1min":0.88500000000000001,"5min":0.81399999999999995,"15min":0.39900000000000002},"max":{"1min":1.728,"5min":1.968,"15min":3.125},"last":1.3959999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.361,"15min":1.3660000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.51200000000000001},"max":{"1min":1.595,"5min":1.925,"15min":3.0830000000000002},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.3400000000000001,"15min":1.4079999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.52700000000000002},"max":{"1min":1.772,"5min":2.0449999999999999,"15min":3.1899999999999999},"last":1.2909999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.4139999999999999,"15min":1.448},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.47999999999999998},"max":{"1min":1.77,"5min":2.1030000000000002,"15min":3.391},"last":1.712},{"interface":"front","average":{"1min":1.367,"5min":1.4259999999999999,"15min":1.4079999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.58399999999999996},"max":{"1min":1.956,"5min":2.2170000000000001,"15min":3.153},"last":1.6679999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.5349999999999999,"15min":1.5189999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.60599999999999998},"max":{"1min":1.877,"5min":2.1400000000000001,"15min":3.5350000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.516,"15min":1.494},"min":{"1min":0.92600000000000005,"5min":0.78300000000000003,"15min":0.45300000000000001},"max":{"1min":1.7090000000000001,"5min":2.2610000000000001,"15min":3.3650000000000002},"last":1.583}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:32:51.983 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T11:32:52.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.209+0000 7fee0de00640 1 Processor -- start 2024-03-23T11:32:52.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.209+0000 7fee0de00640 1 -- start start 2024-03-23T11:32:52.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.210+0000 7fee0de00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fee0815cbb0 0x7fee080996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:52.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.210+0000 7fee0de00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fee080990e0 0x7fee080b1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:52.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.211+0000 7fee0de00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee080cab10 con 0x7fee0806bfe0 2024-03-23T11:32:52.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.211+0000 7fee0de00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080cb3b0 con 0x7fee080990e0 2024-03-23T11:32:52.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.211+0000 7fee0de00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080afe20 con 0x7fee0815cbb0 2024-03-23T11:32:52.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.211+0000 7fee0d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fee080990e0 0x7fee080b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:52.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.211+0000 7fee0ca00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fee0815cbb0 0x7fee080996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:52.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.211+0000 7fee0d400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fee080990e0 0x7fee080b1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39664/0 (socket says 172.21.15.62:39664) 2024-03-23T11:32:52.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.211+0000 7fee0d400640 1 -- 172.21.15.62:0/439201308 learned_addr learned my addr 172.21.15.62:0/439201308 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:52.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.212+0000 7fee0d400640 1 -- 172.21.15.62:0/439201308 >> v2:172.21.15.120:3300/0 conn(0x7fee0815cbb0 msgr2=0x7fee080996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:52.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.212+0000 7fee0d400640 1 --2- 172.21.15.62:0/439201308 >> v2:172.21.15.120:3300/0 conn(0x7fee0815cbb0 0x7fee080996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:52.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.212+0000 7fee0d400640 1 -- 172.21.15.62:0/439201308 >> v1:172.21.15.120:6789/0 conn(0x7fee0806bfe0 legacy=0x7fee080989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.212+0000 7fee0d400640 1 -- 172.21.15.62:0/439201308 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0806c750 con 0x7fee080990e0 2024-03-23T11:32:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.213+0000 7fee0d400640 1 --2- 172.21.15.62:0/439201308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee080990e0 0x7fee080b1010 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fee0806d410 tx=0x7fee0002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39671ff45d62c4b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.214+0000 7fee07400640 1 -- 172.21.15.62:0/439201308 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0002cb20 con 0x7fee080990e0 2024-03-23T11:32:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.214+0000 7fee07400640 1 -- 172.21.15.62:0/439201308 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0002cc80 con 0x7fee080990e0 2024-03-23T11:32:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.214+0000 7fee07400640 1 -- 172.21.15.62:0/439201308 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee000343f0 con 0x7fee080990e0 2024-03-23T11:32:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.214+0000 7fee0de00640 1 -- 172.21.15.62:0/439201308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee080990e0 msgr2=0x7fee080b1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.214+0000 7fee0de00640 1 --2- 172.21.15.62:0/439201308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee080990e0 0x7fee080b1010 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fee0806d410 tx=0x7fee0002bf80 comp rx=0 tx=0).stop 2024-03-23T11:32:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.214+0000 7fee0de00640 1 -- 172.21.15.62:0/439201308 shutdown_connections 2024-03-23T11:32:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.214+0000 7fee0de00640 1 --2- 172.21.15.62:0/439201308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee080990e0 0x7fee080b1010 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.214+0000 7fee0de00640 1 --2- 172.21.15.62:0/439201308 >> v2:172.21.15.120:3300/0 conn(0x7fee0815cbb0 0x7fee080996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.214+0000 7fee0de00640 1 -- 172.21.15.62:0/439201308 >> 172.21.15.62:0/439201308 conn(0x7fee080b1fb0 msgr2=0x7fee080b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.215+0000 7fee0de00640 1 -- 172.21.15.62:0/439201308 shutdown_connections 2024-03-23T11:32:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.215+0000 7fee0de00640 1 -- 172.21.15.62:0/439201308 wait complete. 2024-03-23T11:32:52.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.216+0000 7fee0de00640 1 Processor -- start 2024-03-23T11:32:52.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.216+0000 7fee0de00640 1 -- start start 2024-03-23T11:32:52.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.217+0000 7fee0de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee0806bfe0 0x7fee0814ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.217+0000 7fee0de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee080990e0 0x7fee0814b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.217+0000 7fee0de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee0815cbb0 0x7fee081f5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.217+0000 7fee0de00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081f1f20 con 0x7fee0815cbb0 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.217+0000 7fee0de00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081f6450 con 0x7fee0806bfe0 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.217+0000 7fee0de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee080b6090 con 0x7fee080990e0 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.218+0000 7fee0d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee0815cbb0 0x7fee081f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.218+0000 7fee0d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee0815cbb0 0x7fee081f5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39672/0 (socket says 172.21.15.62:39672) 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.218+0000 7fee0d400640 1 -- 172.21.15.62:0/2280184006 learned_addr learned my addr 172.21.15.62:0/2280184006 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.218+0000 7fee0ca00640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee0806bfe0 0x7fee0814ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.218+0000 7fee07e00640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee080990e0 0x7fee0814b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.218+0000 7fee0d400640 1 -- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee080990e0 msgr2=0x7fee0814b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.218+0000 7fee0d400640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee080990e0 0x7fee0814b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:52.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.218+0000 7fee0d400640 1 -- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee0806bfe0 msgr2=0x7fee0814ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:52.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.218+0000 7fee0d400640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee0806bfe0 0x7fee0814ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:52.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.218+0000 7fee0d400640 1 -- 172.21.15.62:0/2280184006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee00009b30 con 0x7fee0815cbb0 2024-03-23T11:32:52.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.219+0000 7fee0d400640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee0815cbb0 0x7fee081f5570 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fee00036040 tx=0x7fee00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:52.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.219+0000 7fee04c00640 1 -- 172.21.15.62:0/2280184006 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee00009510 con 0x7fee0815cbb0 2024-03-23T11:32:52.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.219+0000 7fee04c00640 1 -- 172.21.15.62:0/2280184006 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0002c920 con 0x7fee0815cbb0 2024-03-23T11:32:52.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.219+0000 7fee04c00640 1 -- 172.21.15.62:0/2280184006 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0003e690 con 0x7fee0815cbb0 2024-03-23T11:32:52.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.219+0000 7fee0de00640 1 -- 172.21.15.62:0/2280184006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee080b5db0 con 0x7fee0815cbb0 2024-03-23T11:32:52.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.219+0000 7fee0de00640 1 -- 172.21.15.62:0/2280184006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee080c4450 con 0x7fee0815cbb0 2024-03-23T11:32:52.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.220+0000 7fee0de00640 1 -- 172.21.15.62:0/2280184006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee080b14d0 con 0x7fee0815cbb0 2024-03-23T11:32:52.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.221+0000 7fee04c00640 1 -- 172.21.15.62:0/2280184006 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fee0003e7f0 con 0x7fee0815cbb0 2024-03-23T11:32:52.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.222+0000 7fee04c00640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede806a7a0 0x7fede806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:52.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.222+0000 7fee04c00640 1 -- 172.21.15.62:0/2280184006 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fee000ab5d0 con 0x7fee0815cbb0 2024-03-23T11:32:52.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.222+0000 7fee0ca00640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede806a7a0 0x7fede806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:52.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.224+0000 7fee0ca00640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede806a7a0 0x7fede806cc60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009a30 tx=0x7fedf8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:52.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.227+0000 7fee04c00640 1 -- 172.21.15.62:0/2280184006 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fee00076cd0 con 0x7fee0815cbb0 2024-03-23T11:32:52.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.425+0000 7fee0de00640 1 -- 172.21.15.62:0/2280184006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee080672c0 con 0x7fee0815cbb0 2024-03-23T11:32:52.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.427+0000 7fee04c00640 1 -- 172.21.15.62:0/2280184006 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+12790 (secure 0 0 0) 0x7fee000ab070 con 0x7fee0815cbb0 2024-03-23T11:32:52.418 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:52.418 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":125,"fsid":"e8b2848c-f838-4b99-a310-de7a7069c38e","created":"2024-03-23T11:18:35.151206+0000","modified":"2024-03-23T11:32:47.535488+0000","last_up_change":"2024-03-23T11:18:48.216045+0000","last_in_change":"2024-03-23T11:18:39.629234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T11:18:49.304156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c85aebac-7bbe-48fe-8ff2-bab379b1927f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6811","nonce":36998},{"type":"v1","addr":"172.21.15.62:6812","nonce":36998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":36998},{"type":"v1","addr":"172.21.15.62:6816","nonce":36998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":36998},{"type":"v1","addr":"172.21.15.62:6823","nonce":36998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":36998},{"type":"v1","addr":"172.21.15.62:6819","nonce":36998}]},"public_addr":"172.21.15.62:6812/36998","cluster_addr":"172.21.15.62:6816/36998","heartbeat_back_addr":"172.21.15.62:6823/36998","heartbeat_front_addr":"172.21.15.62:6819/36998","state":["exists","up"]},{"osd":1,"uuid":"64153585-2982-4f26-98c5-b4cc9036dab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":37001},{"type":"v1","addr":"172.21.15.62:6825","nonce":37001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":37001},{"type":"v1","addr":"172.21.15.62:6827","nonce":37001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":37001},{"type":"v1","addr":"172.21.15.62:6831","nonce":37001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":37001},{"type":"v1","addr":"172.21.15.62:6829","nonce":37001}]},"public_addr":"172.21.15.62:6825/37001","cluster_addr":"172.21.15.62:6827/37001","heartbeat_back_addr":"172.21.15.62:6831/37001","heartbeat_front_addr":"172.21.15.62:6829/37001","state":["exists","up"]},{"osd":2,"uuid":"0ddbbec9-5271-4946-a229-c4b7924b2573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":36996},{"type":"v1","addr":"172.21.15.62:6801","nonce":36996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":36996},{"type":"v1","addr":"172.21.15.62:6803","nonce":36996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":36996},{"type":"v1","addr":"172.21.15.62:6807","nonce":36996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":36996},{"type":"v1","addr":"172.21.15.62:6805","nonce":36996}]},"public_addr":"172.21.15.62:6801/36996","cluster_addr":"172.21.15.62:6803/36996","heartbeat_back_addr":"172.21.15.62:6807/36996","heartbeat_front_addr":"172.21.15.62:6805/36996","state":["exists","up"]},{"osd":3,"uuid":"c7e6782f-d9c6-4e49-8191-bfc31bc3b274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":37000},{"type":"v1","addr":"172.21.15.62:6809","nonce":37000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":37000},{"type":"v1","addr":"172.21.15.62:6813","nonce":37000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":37000},{"type":"v1","addr":"172.21.15.62:6822","nonce":37000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6815","nonce":37000},{"type":"v1","addr":"172.21.15.62:6817","nonce":37000}]},"public_addr":"172.21.15.62:6809/37000","cluster_addr":"172.21.15.62:6813/37000","heartbeat_back_addr":"172.21.15.62:6822/37000","heartbeat_front_addr":"172.21.15.62:6817/37000","state":["exists","up"]},{"osd":4,"uuid":"5f5fd7ee-9bcc-4e93-ae08-9bb4f45ae1d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6803","nonce":37640},{"type":"v1","addr":"172.21.15.120:6805","nonce":37640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6807","nonce":37640},{"type":"v1","addr":"172.21.15.120:6809","nonce":37640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":37640},{"type":"v1","addr":"172.21.15.120:6815","nonce":37640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6811","nonce":37640},{"type":"v1","addr":"172.21.15.120:6813","nonce":37640}]},"public_addr":"172.21.15.120:6805/37640","cluster_addr":"172.21.15.120:6809/37640","heartbeat_back_addr":"172.21.15.120:6815/37640","heartbeat_front_addr":"172.21.15.120:6813/37640","state":["exists","up"]},{"osd":5,"uuid":"865988b2-5f37-4722-a2fc-8c611abd221a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":37642},{"type":"v1","addr":"172.21.15.120:6825","nonce":37642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":37642},{"type":"v1","addr":"172.21.15.120:6827","nonce":37642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":37642},{"type":"v1","addr":"172.21.15.120:6831","nonce":37642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":37642},{"type":"v1","addr":"172.21.15.120:6829","nonce":37642}]},"public_addr":"172.21.15.120:6825/37642","cluster_addr":"172.21.15.120:6827/37642","heartbeat_back_addr":"172.21.15.120:6831/37642","heartbeat_front_addr":"172.21.15.120:6829/37642","state":["exists","up"]},{"osd":6,"uuid":"b76f68e1-ec2c-438e-8275-90f2c3df293e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":37641},{"type":"v1","addr":"172.21.15.120:6801","nonce":37641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":37641},{"type":"v1","addr":"172.21.15.120:6804","nonce":37641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":37641},{"type":"v1","addr":"172.21.15.120:6812","nonce":37641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":37641},{"type":"v1","addr":"172.21.15.120:6808","nonce":37641}]},"public_addr":"172.21.15.120:6801/37641","cluster_addr":"172.21.15.120:6804/37641","heartbeat_back_addr":"172.21.15.120:6812/37641","heartbeat_front_addr":"172.21.15.120:6808/37641","state":["exists","up"]},{"osd":7,"uuid":"87fbbab4-1560-4530-8400-f57ca7f08fe6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":37655},{"type":"v1","addr":"172.21.15.120:6817","nonce":37655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":37655},{"type":"v1","addr":"172.21.15.120:6819","nonce":37655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":37655},{"type":"v1","addr":"172.21.15.120:6823","nonce":37655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":37655},{"type":"v1","addr":"172.21.15.120:6821","nonce":37655}]},"public_addr":"172.21.15.120:6817/37655","cluster_addr":"172.21.15.120:6819/37655","heartbeat_back_addr":"172.21.15.120:6823/37655","heartbeat_front_addr":"172.21.15.120:6821/37655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.241524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.212925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.214921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.982016+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.169940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.007109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:44.999943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T11:18:45.254775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6835/2500981044":"2024-03-24T11:32:44.457651+0000","172.21.15.62:6835/4129977589":"2024-03-24T11:32:23.551904+0000","172.21.15.62:6834/4129977589":"2024-03-24T11:32:23.551904+0000","172.21.15.120:6837/2585024035":"2024-03-24T11:32:04.365655+0000","172.21.15.62:6834/2500981044":"2024-03-24T11:32:44.457651+0000","172.21.15.120:6835/2585024035":"2024-03-24T11:32:04.365655+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T11:32:52.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 -- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede806a7a0 msgr2=0x7fede806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:52.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede806a7a0 0x7fede806cc60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009a30 tx=0x7fedf8008040 comp rx=0 tx=0).stop 2024-03-23T11:32:52.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 -- 172.21.15.62:0/2280184006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee0815cbb0 msgr2=0x7fee081f5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:52.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee0815cbb0 0x7fee081f5570 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fee00036040 tx=0x7fee00004a40 comp rx=0 tx=0).stop 2024-03-23T11:32:52.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 -- 172.21.15.62:0/2280184006 shutdown_connections 2024-03-23T11:32:52.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede806a7a0 0x7fede806cc60 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:52.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee0815cbb0 0x7fee081f5570 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:52.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee080990e0 0x7fee0814b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:52.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 --2- 172.21.15.62:0/2280184006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee0806bfe0 0x7fee0814ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:52.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 -- 172.21.15.62:0/2280184006 >> 172.21.15.62:0/2280184006 conn(0x7fee080b1fb0 msgr2=0x7fee080ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:52.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 -- 172.21.15.62:0/2280184006 shutdown_connections 2024-03-23T11:32:52.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:52.430+0000 7fee0de00640 1 -- 172.21.15.62:0/2280184006 wait complete. 2024-03-23T11:32:53.438 INFO:tasks.ceph:Scrubbing osd.0 2024-03-23T11:32:53.439 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-23T11:32:53.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.689+0000 7fd6ff400640 1 Processor -- start 2024-03-23T11:32:53.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.690+0000 7fd6ff400640 1 -- start start 2024-03-23T11:32:53.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.691+0000 7fd6ff400640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6f80cf2c0 0x7fd6f80cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.691+0000 7fd6ff400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f80b58b0 con 0x7fd6f815ca70 2024-03-23T11:32:53.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.691+0000 7fd6ff400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f8161160 con 0x7fd6f80d1320 2024-03-23T11:32:53.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.691+0000 7fd6ff400640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f80cfc90 con 0x7fd6f80cf2c0 2024-03-23T11:32:53.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.691+0000 7fd6f7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6f80cf2c0 0x7fd6f80cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.692+0000 7fd6f7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6f80cf2c0 0x7fd6f80cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39674/0 (socket says 172.21.15.62:39674) 2024-03-23T11:32:53.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.692+0000 7fd6f7e00640 1 -- 172.21.15.62:0/3122847797 learned_addr learned my addr 172.21.15.62:0/3122847797 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:53.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.692+0000 7fd6f7e00640 1 -- 172.21.15.62:0/3122847797 >> v1:172.21.15.120:6789/0 conn(0x7fd6f80d1320 legacy=0x7fd6f80d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:53.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.692+0000 7fd6f7e00640 1 -- 172.21.15.62:0/3122847797 >> v1:172.21.15.62:6789/0 conn(0x7fd6f815ca70 legacy=0x7fd6f81606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:53.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.692+0000 7fd6f7e00640 1 -- 172.21.15.62:0/3122847797 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f8161440 con 0x7fd6f80cf2c0 2024-03-23T11:32:53.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.693+0000 7fd6f7e00640 1 --2- 172.21.15.62:0/3122847797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f80cf2c0 0x7fd6f80cf750 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009870 tx=0x7fd6e402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f259f1c814bd068a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:53.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.694+0000 7fd6f7400640 1 -- 172.21.15.62:0/3122847797 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e402f850 con 0x7fd6f80cf2c0 2024-03-23T11:32:53.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.694+0000 7fd6f7400640 1 -- 172.21.15.62:0/3122847797 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e402f9b0 con 0x7fd6f80cf2c0 2024-03-23T11:32:53.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.694+0000 7fd6f7400640 1 -- 172.21.15.62:0/3122847797 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e402fd40 con 0x7fd6f80cf2c0 2024-03-23T11:32:53.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.694+0000 7fd6ff400640 1 -- 172.21.15.62:0/3122847797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f80cf2c0 msgr2=0x7fd6f80cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:53.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.694+0000 7fd6ff400640 1 --2- 172.21.15.62:0/3122847797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f80cf2c0 0x7fd6f80cf750 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009870 tx=0x7fd6e402ba30 comp rx=0 tx=0).stop 2024-03-23T11:32:53.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.694+0000 7fd6ff400640 1 -- 172.21.15.62:0/3122847797 shutdown_connections 2024-03-23T11:32:53.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.694+0000 7fd6ff400640 1 --2- 172.21.15.62:0/3122847797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f80cf2c0 0x7fd6f80cf750 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:53.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.694+0000 7fd6ff400640 1 -- 172.21.15.62:0/3122847797 >> 172.21.15.62:0/3122847797 conn(0x7fd6f8071390 msgr2=0x7fd6f8071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:53.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.694+0000 7fd6ff400640 1 -- 172.21.15.62:0/3122847797 shutdown_connections 2024-03-23T11:32:53.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.694+0000 7fd6ff400640 1 -- 172.21.15.62:0/3122847797 wait complete. 2024-03-23T11:32:53.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.696+0000 7fd6ff400640 1 Processor -- start 2024-03-23T11:32:53.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.696+0000 7fd6ff400640 1 -- start start 2024-03-23T11:32:53.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.697+0000 7fd6ff400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f80cf2c0 0x7fd6f80b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.697+0000 7fd6ff400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f80d1320 0x7fd6f80cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.697+0000 7fd6ff400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f815ca70 0x7fd6f80b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.697+0000 7fd6ff400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f80cbe00 con 0x7fd6f815ca70 2024-03-23T11:32:53.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.697+0000 7fd6ff400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f80cc0e0 con 0x7fd6f80d1320 2024-03-23T11:32:53.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.697+0000 7fd6ff400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f80cc3c0 con 0x7fd6f80cf2c0 2024-03-23T11:32:53.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6fd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f815ca70 0x7fd6f80b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6fd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f815ca70 0x7fd6f80b0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39688/0 (socket says 172.21.15.62:39688) 2024-03-23T11:32:53.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6fd400640 1 -- 172.21.15.62:0/2662364781 learned_addr learned my addr 172.21.15.62:0/2662364781 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:53.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6fca00640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f80cf2c0 0x7fd6f80b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6f7e00640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f80d1320 0x7fd6f80cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6fd400640 1 -- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f80cf2c0 msgr2=0x7fd6f80b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6fd400640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f80cf2c0 0x7fd6f80b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6fd400640 1 -- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f80d1320 msgr2=0x7fd6f80cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6fd400640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f80d1320 0x7fd6f80cb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6fd400640 1 -- 172.21.15.62:0/2662364781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e40094d0 con 0x7fd6f815ca70 2024-03-23T11:32:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6fca00640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6f80cf2c0 0x7fd6f80b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.698+0000 7fd6f7e00640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6f80d1320 0x7fd6f80cb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.699+0000 7fd6fd400640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f815ca70 0x7fd6f80b0350 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000c780 tx=0x7fd6f000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.699+0000 7fd6f4c00640 1 -- 172.21.15.62:0/2662364781 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0004b60 con 0x7fd6f815ca70 2024-03-23T11:32:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.699+0000 7fd6f4c00640 1 -- 172.21.15.62:0/2662364781 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f000ce90 con 0x7fd6f815ca70 2024-03-23T11:32:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.699+0000 7fd6ff400640 1 -- 172.21.15.62:0/2662364781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f80cc6a0 con 0x7fd6f815ca70 2024-03-23T11:32:53.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.699+0000 7fd6f4c00640 1 -- 172.21.15.62:0/2662364781 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f000a850 con 0x7fd6f815ca70 2024-03-23T11:32:53.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.699+0000 7fd6ff400640 1 -- 172.21.15.62:0/2662364781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f80b2a00 con 0x7fd6f815ca70 2024-03-23T11:32:53.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.700+0000 7fd6ff400640 1 -- 172.21.15.62:0/2662364781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd6f80b83c0 con 0x7fd6f815ca70 2024-03-23T11:32:53.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.701+0000 7fd6f4c00640 1 -- 172.21.15.62:0/2662364781 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fd6f000a9b0 con 0x7fd6f815ca70 2024-03-23T11:32:53.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.702+0000 7fd6f4c00640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6d806a7a0 0x7fd6d806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.702+0000 7fd6f4c00640 1 -- 172.21.15.62:0/2662364781 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fd6f0088e10 con 0x7fd6f815ca70 2024-03-23T11:32:53.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.702+0000 7fd6f4c00640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x7fd6d8070740 0x7fd6d8072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.702+0000 7fd6f4c00640 1 -- 172.21.15.62:0/2662364781 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6d8073210 con 0x7fd6d8070740 2024-03-23T11:32:53.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.702+0000 7fd6f4c00640 1 -- 172.21.15.62:0/2662364781 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6f008b3f0 con 0x7fd6f815ca70 2024-03-23T11:32:53.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.703+0000 7fd6f7e00640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x7fd6d8070740 0x7fd6d8072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.703+0000 7fd6fca00640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6d806a7a0 0x7fd6d806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.703+0000 7fd6f7e00640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x7fd6d8070740 0x7fd6d8072b60 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:53.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.704+0000 7fd6fca00640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6d806a7a0 0x7fd6d806cc60 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec007fd0 tx=0x7fd6ec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:53.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.704+0000 7fd6f4c00640 1 -- 172.21.15.62:0/2662364781 <== osd.0 v2:172.21.15.62:6811/36998 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd6d8073210 con 0x7fd6d8070740 2024-03-23T11:32:53.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.718+0000 7fd6ff400640 1 -- 172.21.15.62:0/2662364781 --> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd6f80d0e00 con 0x7fd6d8070740 2024-03-23T11:32:53.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.719+0000 7fd6f4c00640 1 -- 172.21.15.62:0/2662364781 <== osd.0 v2:172.21.15.62:6811/36998 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd6e402fd20 con 0x7fd6d8070740 2024-03-23T11:32:53.708 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:53.708 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T11:32:53.708 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:53.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.719+0000 7fd6ff400640 1 -- 172.21.15.62:0/2662364781 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x7fd6d8070740 msgr2=0x7fd6d8072b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:53.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.719+0000 7fd6ff400640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.62:6811/36998,v1:172.21.15.62:6812/36998] conn(0x7fd6d8070740 0x7fd6d8072b60 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:53.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.719+0000 7fd6ff400640 1 -- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6d806a7a0 msgr2=0x7fd6d806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:53.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.719+0000 7fd6ff400640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fd6d806a7a0 0x7fd6d806cc60 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec007fd0 tx=0x7fd6ec009040 comp rx=0 tx=0).stop 2024-03-23T11:32:53.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.719+0000 7fd6ff400640 1 -- 172.21.15.62:0/2662364781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f815ca70 msgr2=0x7fd6f80b0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:53.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.719+0000 7fd6ff400640 1 --2- 172.21.15.62:0/2662364781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f815ca70 0x7fd6f80b0350 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000c780 tx=0x7fd6f000cc40 comp rx=0 tx=0).stop 2024-03-23T11:32:53.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.719+0000 7fd6fd400640 1 -- 172.21.15.62:0/2662364781 reap_dead start 2024-03-23T11:32:53.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.720+0000 7fd6ff400640 1 -- 172.21.15.62:0/2662364781 shutdown_connections 2024-03-23T11:32:53.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.720+0000 7fd6ff400640 1 -- 172.21.15.62:0/2662364781 >> 172.21.15.62:0/2662364781 conn(0x7fd6f8071390 msgr2=0x7fd6f815b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:53.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.720+0000 7fd6ff400640 1 -- 172.21.15.62:0/2662364781 shutdown_connections 2024-03-23T11:32:53.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.720+0000 7fd6ff400640 1 -- 172.21.15.62:0/2662364781 wait complete. 2024-03-23T11:32:53.728 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-23T11:32:53.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.963+0000 7fc666000640 1 Processor -- start 2024-03-23T11:32:53.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.963+0000 7fc666000640 1 -- start start 2024-03-23T11:32:53.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.964+0000 7fc666000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6600ce8d0 0x7fc6600af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.964+0000 7fc666000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc6600cecb0 0x7fc6600ad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.964+0000 7fc666000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6600cc860 con 0x7fc6600addb0 2024-03-23T11:32:53.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.964+0000 7fc666000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6600cac10 con 0x7fc6600cecb0 2024-03-23T11:32:53.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.964+0000 7fc666000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600cbfc0 con 0x7fc6600ce8d0 2024-03-23T11:32:53.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.965+0000 7fc65fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc6600cecb0 0x7fc6600ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.965+0000 7fc65fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc6600cecb0 0x7fc6600ad870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39700/0 (socket says 172.21.15.62:39700) 2024-03-23T11:32:53.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.965+0000 7fc65fe00640 1 -- 172.21.15.62:0/2849362749 learned_addr learned my addr 172.21.15.62:0/2849362749 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:53.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.965+0000 7fc664c00640 1 --2- 172.21.15.62:0/2849362749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6600ce8d0 0x7fc6600af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.965+0000 7fc65fe00640 1 -- 172.21.15.62:0/2849362749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6600ce8d0 msgr2=0x7fc6600af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:53.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.965+0000 7fc65fe00640 1 --2- 172.21.15.62:0/2849362749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6600ce8d0 0x7fc6600af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:53.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.965+0000 7fc65fe00640 1 -- 172.21.15.62:0/2849362749 >> v1:172.21.15.62:6789/0 conn(0x7fc6600addb0 legacy=0x7fc6600ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:53.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.965+0000 7fc65fe00640 1 -- 172.21.15.62:0/2849362749 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6600cc2a0 con 0x7fc6600cecb0 2024-03-23T11:32:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.967+0000 7fc65fe00640 1 --2- 172.21.15.62:0/2849362749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6600cecb0 0x7fc6600ad870 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009870 tx=0x7fc64c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e30f7f3a91106f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.967+0000 7fc65f400640 1 -- 172.21.15.62:0/2849362749 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c030840 con 0x7fc6600cecb0 2024-03-23T11:32:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.967+0000 7fc65f400640 1 -- 172.21.15.62:0/2849362749 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c0309a0 con 0x7fc6600cecb0 2024-03-23T11:32:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.967+0000 7fc65f400640 1 -- 172.21.15.62:0/2849362749 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c030d30 con 0x7fc6600cecb0 2024-03-23T11:32:53.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.967+0000 7fc666000640 1 -- 172.21.15.62:0/2849362749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6600cecb0 msgr2=0x7fc6600ad870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:53.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.967+0000 7fc666000640 1 --2- 172.21.15.62:0/2849362749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6600cecb0 0x7fc6600ad870 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009870 tx=0x7fc64c02b980 comp rx=0 tx=0).stop 2024-03-23T11:32:53.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.968+0000 7fc666000640 1 -- 172.21.15.62:0/2849362749 shutdown_connections 2024-03-23T11:32:53.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.968+0000 7fc666000640 1 --2- 172.21.15.62:0/2849362749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6600cecb0 0x7fc6600ad870 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:53.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.968+0000 7fc666000640 1 --2- 172.21.15.62:0/2849362749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6600ce8d0 0x7fc6600af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:53.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.968+0000 7fc666000640 1 -- 172.21.15.62:0/2849362749 >> 172.21.15.62:0/2849362749 conn(0x7fc6600b1fb0 msgr2=0x7fc6600b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:53.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.968+0000 7fc666000640 1 -- 172.21.15.62:0/2849362749 shutdown_connections 2024-03-23T11:32:53.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.968+0000 7fc666000640 1 -- 172.21.15.62:0/2849362749 wait complete. 2024-03-23T11:32:53.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.969+0000 7fc666000640 1 Processor -- start 2024-03-23T11:32:53.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.970+0000 7fc666000640 1 -- start start 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc666000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6600addb0 0x7fc6600c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc666000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6600ce8d0 0x7fc6601f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc666000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6600cecb0 0x7fc6600c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc666000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600c96b0 con 0x7fc6600cecb0 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc666000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600b6050 con 0x7fc6600addb0 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc666000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600c8aa0 con 0x7fc6600ce8d0 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc665600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6600cecb0 0x7fc6600c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc665600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6600cecb0 0x7fc6600c49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39702/0 (socket says 172.21.15.62:39702) 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc665600640 1 -- 172.21.15.62:0/1496821973 learned_addr learned my addr 172.21.15.62:0/1496821973 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc664c00640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6600addb0 0x7fc6600c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc65fe00640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6600ce8d0 0x7fc6601f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc665600640 1 -- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6600ce8d0 msgr2=0x7fc6601f58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:53.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc665600640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6600ce8d0 0x7fc6601f58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:53.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc665600640 1 -- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6600addb0 msgr2=0x7fc6600c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:53.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc665600640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6600addb0 0x7fc6600c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:53.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.971+0000 7fc665600640 1 -- 172.21.15.62:0/1496821973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c0094d0 con 0x7fc6600cecb0 2024-03-23T11:32:53.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.972+0000 7fc665600640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6600cecb0 0x7fc6600c49d0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fc65800c6b0 tx=0x7fc65800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:53.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.972+0000 7fc65cc00640 1 -- 172.21.15.62:0/1496821973 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658003b70 con 0x7fc6600cecb0 2024-03-23T11:32:53.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.972+0000 7fc65cc00640 1 -- 172.21.15.62:0/1496821973 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc658003cf0 con 0x7fc6600cecb0 2024-03-23T11:32:53.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.972+0000 7fc666000640 1 -- 172.21.15.62:0/1496821973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6600c8d80 con 0x7fc6600cecb0 2024-03-23T11:32:53.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.972+0000 7fc65cc00640 1 -- 172.21.15.62:0/1496821973 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658003ea0 con 0x7fc6600cecb0 2024-03-23T11:32:53.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.972+0000 7fc666000640 1 -- 172.21.15.62:0/1496821973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6600c9340 con 0x7fc6600cecb0 2024-03-23T11:32:53.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.973+0000 7fc666000640 1 -- 172.21.15.62:0/1496821973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc660073540 con 0x7fc6600cecb0 2024-03-23T11:32:53.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.974+0000 7fc65cc00640 1 -- 172.21.15.62:0/1496821973 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc658005310 con 0x7fc6600cecb0 2024-03-23T11:32:53.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.976+0000 7fc65cc00640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc64006a7d0 0x7fc64006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:53.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.976+0000 7fc65cc00640 1 -- 172.21.15.62:0/1496821973 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fc658088850 con 0x7fc6600cecb0 2024-03-23T11:32:53.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.976+0000 7fc664c00640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc64006a7d0 0x7fc64006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:53.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.977+0000 7fc664c00640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc64006a7d0 0x7fc64006cc90 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc650006fd0 tx=0x7fc650008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:53.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:53.980+0000 7fc65cc00640 1 -- 172.21.15.62:0/1496821973 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc658053f50 con 0x7fc6600cecb0 2024-03-23T11:32:54.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.180+0000 7fc666000640 1 -- 172.21.15.62:0/1496821973 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6600cbc90 con 0x7fc64006a7d0 2024-03-23T11:32:54.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.181+0000 7fc65cc00640 1 -- 172.21.15.62:0/1496821973 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc65000f830 con 0x7fc64006a7d0 2024-03-23T11:32:54.170 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 0 to deep-scrub 2024-03-23T11:32:54.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 -- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc64006a7d0 msgr2=0x7fc64006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc64006a7d0 0x7fc64006cc90 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc650006fd0 tx=0x7fc650008040 comp rx=0 tx=0).stop 2024-03-23T11:32:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 -- 172.21.15.62:0/1496821973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6600cecb0 msgr2=0x7fc6600c49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6600cecb0 0x7fc6600c49d0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fc65800c6b0 tx=0x7fc65800cb70 comp rx=0 tx=0).stop 2024-03-23T11:32:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 -- 172.21.15.62:0/1496821973 shutdown_connections 2024-03-23T11:32:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc64006a7d0 0x7fc64006cc90 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6600cecb0 0x7fc6600c49d0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc6600ce8d0 0x7fc6601f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 --2- 172.21.15.62:0/1496821973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc6600addb0 0x7fc6600c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 -- 172.21.15.62:0/1496821973 >> 172.21.15.62:0/1496821973 conn(0x7fc6600b1fb0 msgr2=0x7fc66006f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 -- 172.21.15.62:0/1496821973 shutdown_connections 2024-03-23T11:32:54.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.184+0000 7fc666000640 1 -- 172.21.15.62:0/1496821973 wait complete. 2024-03-23T11:32:54.192 INFO:tasks.ceph:Scrubbing osd.1 2024-03-23T11:32:54.192 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-23T11:32:54.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.421+0000 7ff67a600640 1 Processor -- start 2024-03-23T11:32:54.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.422+0000 7ff67a600640 1 -- start start 2024-03-23T11:32:54.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.423+0000 7ff67a600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff674144350 0x7ff67406fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.423+0000 7ff67a600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6740d0220 0x7ff674070220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.423+0000 7ff67a600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff674070ad0 0x7ff67415e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.423+0000 7ff67a600640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff674144730 con 0x7ff6740d0220 2024-03-23T11:32:54.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.423+0000 7ff67a600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6740cddd0 con 0x7ff674144350 2024-03-23T11:32:54.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.423+0000 7ff67a600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6740ce260 con 0x7ff674070ad0 2024-03-23T11:32:54.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.424+0000 7ff672a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6740d0220 0x7ff674070220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.424+0000 7ff672a00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff6740d0220 0x7ff674070220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39718/0 (socket says 172.21.15.62:39718) 2024-03-23T11:32:54.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.424+0000 7ff672a00640 1 -- 172.21.15.62:0/2677506840 learned_addr learned my addr 172.21.15.62:0/2677506840 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:54.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.424+0000 7ff673400640 1 --2- 172.21.15.62:0/2677506840 >> v2:172.21.15.120:3300/0 conn(0x7ff674144350 0x7ff67406fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.424+0000 7ff672a00640 1 -- 172.21.15.62:0/2677506840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff674070ad0 msgr2=0x7ff67415e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:54.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.424+0000 7ff672a00640 1 --2- 172.21.15.62:0/2677506840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff674070ad0 0x7ff67415e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.424+0000 7ff672a00640 1 -- 172.21.15.62:0/2677506840 >> v2:172.21.15.120:3300/0 conn(0x7ff674144350 msgr2=0x7ff67406fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:54.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.424+0000 7ff672a00640 1 --2- 172.21.15.62:0/2677506840 >> v2:172.21.15.120:3300/0 conn(0x7ff674144350 0x7ff67406fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.424+0000 7ff672a00640 1 -- 172.21.15.62:0/2677506840 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6740cd1c0 con 0x7ff6740d0220 2024-03-23T11:32:54.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.424+0000 7ff673400640 1 --2- 172.21.15.62:0/2677506840 >> v2:172.21.15.120:3300/0 conn(0x7ff674144350 0x7ff67406fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:54.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.426+0000 7ff672a00640 1 --2- 172.21.15.62:0/2677506840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6740d0220 0x7ff674070220 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7ff67406f890 tx=0x7ff65c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef941fec0d45b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.426+0000 7ff672000640 1 -- 172.21.15.62:0/2677506840 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c0317d0 con 0x7ff6740d0220 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.426+0000 7ff672000640 1 -- 172.21.15.62:0/2677506840 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c031930 con 0x7ff6740d0220 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.426+0000 7ff672000640 1 -- 172.21.15.62:0/2677506840 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c031cc0 con 0x7ff6740d0220 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.427+0000 7ff67a600640 1 -- 172.21.15.62:0/2677506840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6740d0220 msgr2=0x7ff674070220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.427+0000 7ff67a600640 1 --2- 172.21.15.62:0/2677506840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6740d0220 0x7ff674070220 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7ff67406f890 tx=0x7ff65c02b870 comp rx=0 tx=0).stop 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.427+0000 7ff67a600640 1 -- 172.21.15.62:0/2677506840 shutdown_connections 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.427+0000 7ff67a600640 1 --2- 172.21.15.62:0/2677506840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff674070ad0 0x7ff67415e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.427+0000 7ff67a600640 1 --2- 172.21.15.62:0/2677506840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6740d0220 0x7ff674070220 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.427+0000 7ff67a600640 1 --2- 172.21.15.62:0/2677506840 >> v2:172.21.15.120:3300/0 conn(0x7ff674144350 0x7ff67406fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.427+0000 7ff67a600640 1 -- 172.21.15.62:0/2677506840 >> 172.21.15.62:0/2677506840 conn(0x7ff67406e560 msgr2=0x7ff67406e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.427+0000 7ff67a600640 1 -- 172.21.15.62:0/2677506840 shutdown_connections 2024-03-23T11:32:54.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.427+0000 7ff67a600640 1 -- 172.21.15.62:0/2677506840 wait complete. 2024-03-23T11:32:54.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.429+0000 7ff67a600640 1 Processor -- start 2024-03-23T11:32:54.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.429+0000 7ff67a600640 1 -- start start 2024-03-23T11:32:54.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff67a600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff674070ad0 0x7ff674158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff67a600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6740d0220 0x7ff67406ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff67a600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff674144350 0x7ff674159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff67a600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67406b750 con 0x7ff6740d0220 2024-03-23T11:32:54.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff67a600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67406ba30 con 0x7ff674144350 2024-03-23T11:32:54.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff67a600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67406bd10 con 0x7ff674070ad0 2024-03-23T11:32:54.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff672a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6740d0220 0x7ff67406ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff672a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6740d0220 0x7ff67406ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39728/0 (socket says 172.21.15.62:39728) 2024-03-23T11:32:54.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff672a00640 1 -- 172.21.15.62:0/2541601551 learned_addr learned my addr 172.21.15.62:0/2541601551 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:54.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff673400640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff674070ad0 0x7ff674158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff672a00640 1 -- 172.21.15.62:0/2541601551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff674070ad0 msgr2=0x7ff674158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff672a00640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff674070ad0 0x7ff674158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff672a00640 1 -- 172.21.15.62:0/2541601551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff674144350 msgr2=0x7ff674159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.430+0000 7ff672a00640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff674144350 0x7ff674159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.431+0000 7ff672a00640 1 -- 172.21.15.62:0/2541601551 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c0094d0 con 0x7ff6740d0220 2024-03-23T11:32:54.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.431+0000 7ff672a00640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6740d0220 0x7ff67406ad70 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7ff65c000c00 tx=0x7ff65c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:54.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.431+0000 7ff66b400640 1 -- 172.21.15.62:0/2541601551 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c0328f0 con 0x7ff6740d0220 2024-03-23T11:32:54.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.431+0000 7ff66b400640 1 -- 172.21.15.62:0/2541601551 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c032a70 con 0x7ff6740d0220 2024-03-23T11:32:54.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.431+0000 7ff67a600640 1 -- 172.21.15.62:0/2541601551 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff674136100 con 0x7ff6740d0220 2024-03-23T11:32:54.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.431+0000 7ff66b400640 1 -- 172.21.15.62:0/2541601551 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c032e20 con 0x7ff6740d0220 2024-03-23T11:32:54.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.431+0000 7ff67a600640 1 -- 172.21.15.62:0/2541601551 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff674159160 con 0x7ff6740d0220 2024-03-23T11:32:54.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.432+0000 7ff67a600640 1 -- 172.21.15.62:0/2541601551 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff6740cda60 con 0x7ff6740d0220 2024-03-23T11:32:54.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.433+0000 7ff66b400640 1 -- 172.21.15.62:0/2541601551 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7ff65c030070 con 0x7ff6740d0220 2024-03-23T11:32:54.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.434+0000 7ff66b400640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff65006a7a0 0x7ff65006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.435+0000 7ff66b400640 1 -- 172.21.15.62:0/2541601551 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7ff65c0aabc0 con 0x7ff6740d0220 2024-03-23T11:32:54.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.435+0000 7ff66b400640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x7ff650070740 0x7ff650072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.435+0000 7ff66b400640 1 -- 172.21.15.62:0/2541601551 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff650073210 con 0x7ff650070740 2024-03-23T11:32:54.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.435+0000 7ff66b400640 1 -- 172.21.15.62:0/2541601551 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff65c0af2a0 con 0x7ff6740d0220 2024-03-23T11:32:54.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.435+0000 7ff673e00640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x7ff650070740 0x7ff650072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.435+0000 7ff673400640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff65006a7a0 0x7ff65006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.436+0000 7ff673e00640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x7ff650070740 0x7ff650072b60 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.436+0000 7ff673400640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff65006a7a0 0x7ff65006cc60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff66400b840 tx=0x7ff664008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.437+0000 7ff66b400640 1 -- 172.21.15.62:0/2541601551 <== osd.1 v2:172.21.15.62:6824/37001 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff650073210 con 0x7ff650070740 2024-03-23T11:32:54.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.450+0000 7ff67a600640 1 -- 172.21.15.62:0/2541601551 --> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff6740cd780 con 0x7ff650070740 2024-03-23T11:32:54.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.451+0000 7ff66b400640 1 -- 172.21.15.62:0/2541601551 <== osd.1 v2:172.21.15.62:6824/37001 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff66c00ac70 con 0x7ff650070740 2024-03-23T11:32:54.440 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:54.441 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T11:32:54.441 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.451+0000 7ff67a600640 1 -- 172.21.15.62:0/2541601551 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x7ff650070740 msgr2=0x7ff650072b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.451+0000 7ff67a600640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.62:6824/37001,v1:172.21.15.62:6825/37001] conn(0x7ff650070740 0x7ff650072b60 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.451+0000 7ff67a600640 1 -- 172.21.15.62:0/2541601551 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff65006a7a0 msgr2=0x7ff65006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.451+0000 7ff67a600640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7ff65006a7a0 0x7ff65006cc60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff66400b840 tx=0x7ff664008040 comp rx=0 tx=0).stop 2024-03-23T11:32:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.451+0000 7ff67a600640 1 -- 172.21.15.62:0/2541601551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6740d0220 msgr2=0x7ff67406ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.451+0000 7ff67a600640 1 --2- 172.21.15.62:0/2541601551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6740d0220 0x7ff67406ad70 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7ff65c000c00 tx=0x7ff65c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.452+0000 7ff673e00640 1 -- 172.21.15.62:0/2541601551 reap_dead start 2024-03-23T11:32:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.452+0000 7ff67a600640 1 -- 172.21.15.62:0/2541601551 shutdown_connections 2024-03-23T11:32:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.452+0000 7ff67a600640 1 -- 172.21.15.62:0/2541601551 >> 172.21.15.62:0/2541601551 conn(0x7ff67406e560 msgr2=0x7ff674140710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:54.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.452+0000 7ff67a600640 1 -- 172.21.15.62:0/2541601551 shutdown_connections 2024-03-23T11:32:54.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.452+0000 7ff67a600640 1 -- 172.21.15.62:0/2541601551 wait complete. 2024-03-23T11:32:54.461 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-23T11:32:54.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.693+0000 7f8db3e00640 1 Processor -- start 2024-03-23T11:32:54.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.693+0000 7f8db3e00640 1 -- start start 2024-03-23T11:32:54.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.695+0000 7f8db3e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8db4070090 0x7f8db4070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.695+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8db40b2820 0x7f8db40b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.695+0000 7f8db3e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db40b31b0 con 0x7f8db4143dd0 2024-03-23T11:32:54.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.695+0000 7f8db3e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db4070a60 con 0x7f8db4070090 2024-03-23T11:32:54.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.695+0000 7f8db3e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40cd140 con 0x7f8db40b2820 2024-03-23T11:32:54.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.695+0000 7f8db2000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8db4070090 0x7f8db4070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.695+0000 7f8db2000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8db4070090 0x7f8db4070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39744/0 (socket says 172.21.15.62:39744) 2024-03-23T11:32:54.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.695+0000 7f8db2000640 1 -- 172.21.15.62:0/2043255200 learned_addr learned my addr 172.21.15.62:0/2043255200 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:54.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.695+0000 7f8db2a00640 1 --1- 172.21.15.62:0/2043255200 >> v1:172.21.15.62:6789/0 conn(0x7f8db4143dd0 0x7f8db415abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35558/0 (socket says 172.21.15.62:35558) 2024-03-23T11:32:54.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.696+0000 7f8db2000640 1 -- 172.21.15.62:0/2043255200 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8db40b2820 msgr2=0x7f8db40b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:54.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.696+0000 7f8db3400640 1 --2- 172.21.15.62:0/2043255200 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8db40b2820 0x7f8db40b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.696+0000 7f8db2000640 1 --2- 172.21.15.62:0/2043255200 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8db40b2820 0x7f8db40b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.696+0000 7f8db2000640 1 -- 172.21.15.62:0/2043255200 >> v1:172.21.15.62:6789/0 conn(0x7f8db4143dd0 legacy=0x7f8db415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.696+0000 7f8db2000640 1 -- 172.21.15.62:0/2043255200 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db40cd3e0 con 0x7f8db4070090 2024-03-23T11:32:54.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.696+0000 7f8db3400640 1 --2- 172.21.15.62:0/2043255200 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8db40b2820 0x7f8db40b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:54.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.696+0000 7f8db1600640 1 -- 172.21.15.62:0/2043255200 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633020130 0 0) 0x7f8db40b31b0 con 0x7f8db4143dd0 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.697+0000 7f8db2000640 1 --2- 172.21.15.62:0/2043255200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db4070090 0x7f8db4070520 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009870 tx=0x7f8da002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec56ac0eab1cec40 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.698+0000 7f8db1600640 1 -- 172.21.15.62:0/2043255200 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da0030840 con 0x7f8db4070090 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.698+0000 7f8db1600640 1 -- 172.21.15.62:0/2043255200 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da00309a0 con 0x7f8db4070090 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.698+0000 7f8db1600640 1 -- 172.21.15.62:0/2043255200 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da0030d30 con 0x7f8db4070090 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.698+0000 7f8db3e00640 1 -- 172.21.15.62:0/2043255200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db4070090 msgr2=0x7f8db4070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.698+0000 7f8db3e00640 1 --2- 172.21.15.62:0/2043255200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db4070090 0x7f8db4070520 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009870 tx=0x7f8da002b980 comp rx=0 tx=0).stop 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.698+0000 7f8db3e00640 1 -- 172.21.15.62:0/2043255200 shutdown_connections 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.698+0000 7f8db3e00640 1 --2- 172.21.15.62:0/2043255200 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8db40b2820 0x7f8db40b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.698+0000 7f8db3e00640 1 --2- 172.21.15.62:0/2043255200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db4070090 0x7f8db4070520 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.698+0000 7f8db3e00640 1 -- 172.21.15.62:0/2043255200 >> 172.21.15.62:0/2043255200 conn(0x7f8db4066050 msgr2=0x7f8db4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.698+0000 7f8db3e00640 1 -- 172.21.15.62:0/2043255200 shutdown_connections 2024-03-23T11:32:54.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.698+0000 7f8db3e00640 1 -- 172.21.15.62:0/2043255200 wait complete. 2024-03-23T11:32:54.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.700+0000 7f8db3e00640 1 Processor -- start 2024-03-23T11:32:54.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.700+0000 7f8db3e00640 1 -- start start 2024-03-23T11:32:54.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.701+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db4070090 0x7f8db4132c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.701+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db40b2820 0x7f8db40caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.701+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8db4143dd0 0x7f8db412f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.701+0000 7f8db3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40ac7f0 con 0x7f8db4070090 2024-03-23T11:32:54.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.701+0000 7f8db3e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4133160 con 0x7f8db40b2820 2024-03-23T11:32:54.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.701+0000 7f8db3e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40a8940 con 0x7f8db4143dd0 2024-03-23T11:32:54.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db4070090 0x7f8db4132c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db4070090 0x7f8db4132c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39752/0 (socket says 172.21.15.62:39752) 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db2a00640 1 -- 172.21.15.62:0/3126437896 learned_addr learned my addr 172.21.15.62:0/3126437896 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db2000640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db40b2820 0x7f8db40caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db3400640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8db4143dd0 0x7f8db412f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db2a00640 1 -- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8db4143dd0 msgr2=0x7f8db412f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db2a00640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8db4143dd0 0x7f8db412f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db2a00640 1 -- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db40b2820 msgr2=0x7f8db40caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db2a00640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db40b2820 0x7f8db40caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db2a00640 1 -- 172.21.15.62:0/3126437896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da00094d0 con 0x7f8db4070090 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db2000640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db40b2820 0x7f8db40caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.702+0000 7f8db2a00640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db4070090 0x7f8db4132c20 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f8da400c4d0 tx=0x7f8da400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.703+0000 7f8daaa00640 1 -- 172.21.15.62:0/3126437896 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da40074e0 con 0x7f8db4070090 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.703+0000 7f8daaa00640 1 -- 172.21.15.62:0/3126437896 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da4007660 con 0x7f8db4070090 2024-03-23T11:32:54.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.703+0000 7f8daaa00640 1 -- 172.21.15.62:0/3126437896 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da4004a20 con 0x7f8db4070090 2024-03-23T11:32:54.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.703+0000 7f8db3e00640 1 -- 172.21.15.62:0/3126437896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db40a8c20 con 0x7f8db4070090 2024-03-23T11:32:54.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.703+0000 7f8db3e00640 1 -- 172.21.15.62:0/3126437896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db40a91e0 con 0x7f8db4070090 2024-03-23T11:32:54.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.704+0000 7f8db3e00640 1 -- 172.21.15.62:0/3126437896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db40cd960 con 0x7f8db4070090 2024-03-23T11:32:54.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.705+0000 7f8daaa00640 1 -- 172.21.15.62:0/3126437896 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f8da4004b80 con 0x7f8db4070090 2024-03-23T11:32:54.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.706+0000 7f8daaa00640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d9406a7a0 0x7f8d9406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:54.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.706+0000 7f8daaa00640 1 -- 172.21.15.62:0/3126437896 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f8da4087df0 con 0x7f8db4070090 2024-03-23T11:32:54.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.707+0000 7f8db2000640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d9406a7a0 0x7f8d9406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:54.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.708+0000 7f8db2000640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d9406a7a0 0x7f8d9406cc60 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8da0036000 tx=0x7f8da003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:54.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.711+0000 7f8daaa00640 1 -- 172.21.15.62:0/3126437896 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8da40533c0 con 0x7f8db4070090 2024-03-23T11:32:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.911+0000 7f8db3e00640 1 -- 172.21.15.62:0/3126437896 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8db40cdc80 con 0x7f8d9406a7a0 2024-03-23T11:32:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.913+0000 7f8daaa00640 1 -- 172.21.15.62:0/3126437896 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8da0030be0 con 0x7f8d9406a7a0 2024-03-23T11:32:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 1 to deep-scrub 2024-03-23T11:32:54.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.915+0000 7f8db3e00640 1 -- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d9406a7a0 msgr2=0x7f8d9406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.915+0000 7f8db3e00640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d9406a7a0 0x7f8d9406cc60 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8da0036000 tx=0x7f8da003b040 comp rx=0 tx=0).stop 2024-03-23T11:32:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.915+0000 7f8db3e00640 1 -- 172.21.15.62:0/3126437896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db4070090 msgr2=0x7f8db4132c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.915+0000 7f8db3e00640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db4070090 0x7f8db4132c20 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f8da400c4d0 tx=0x7f8da400c990 comp rx=0 tx=0).stop 2024-03-23T11:32:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.916+0000 7f8db3e00640 1 -- 172.21.15.62:0/3126437896 shutdown_connections 2024-03-23T11:32:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.916+0000 7f8db3e00640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f8d9406a7a0 0x7f8d9406cc60 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.916+0000 7f8db3e00640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8db4143dd0 0x7f8db412f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.916+0000 7f8db3e00640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db40b2820 0x7f8db40caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.916+0000 7f8db3e00640 1 --2- 172.21.15.62:0/3126437896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db4070090 0x7f8db4132c20 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.916+0000 7f8db3e00640 1 -- 172.21.15.62:0/3126437896 >> 172.21.15.62:0/3126437896 conn(0x7f8db4066050 msgr2=0x7f8db4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.916+0000 7f8db3e00640 1 -- 172.21.15.62:0/3126437896 shutdown_connections 2024-03-23T11:32:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:54.916+0000 7f8db3e00640 1 -- 172.21.15.62:0/3126437896 wait complete. 2024-03-23T11:32:54.924 INFO:tasks.ceph:Scrubbing osd.2 2024-03-23T11:32:54.924 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-23T11:32:55.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.163+0000 7f01a7200640 1 Processor -- start 2024-03-23T11:32:55.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.164+0000 7f01a7200640 1 -- start start 2024-03-23T11:32:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.165+0000 7f01a7200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f01a815c820 0x7f01a80b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.165+0000 7f01a7200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f01a80b28e0 0x7f01a80cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.165+0000 7f01a7200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80d0200 con 0x7f01a80cf540 2024-03-23T11:32:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.165+0000 7f01a7200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a80cf9f0 con 0x7f01a815c820 2024-03-23T11:32:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.165+0000 7f01a7200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a80d07c0 con 0x7f01a80b28e0 2024-03-23T11:32:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.166+0000 7f01a5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f01a815c820 0x7f01a80b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.166+0000 7f01a5e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f01a815c820 0x7f01a80b23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39762/0 (socket says 172.21.15.62:39762) 2024-03-23T11:32:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.166+0000 7f01a5e00640 1 -- 172.21.15.62:0/292470948 learned_addr learned my addr 172.21.15.62:0/292470948 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.166+0000 7f019fe00640 1 --2- 172.21.15.62:0/292470948 >> v2:172.21.15.120:3300/0 conn(0x7f01a80b28e0 0x7f01a80cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.166+0000 7f01a5e00640 1 -- 172.21.15.62:0/292470948 >> v2:172.21.15.120:3300/0 conn(0x7f01a80b28e0 msgr2=0x7f01a80cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.166+0000 7f01a5e00640 1 --2- 172.21.15.62:0/292470948 >> v2:172.21.15.120:3300/0 conn(0x7f01a80b28e0 0x7f01a80cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.166+0000 7f01a5e00640 1 -- 172.21.15.62:0/292470948 >> v1:172.21.15.62:6789/0 conn(0x7f01a80cf540 legacy=0x7f01a80d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.166+0000 7f01a5e00640 1 -- 172.21.15.62:0/292470948 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80ce070 con 0x7f01a815c820 2024-03-23T11:32:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.166+0000 7f01a5400640 1 -- 172.21.15.62:0/292470948 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946129466 0 0) 0x7f01a80d0200 con 0x7f01a80cf540 2024-03-23T11:32:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.167+0000 7f01a5e00640 1 --2- 172.21.15.62:0/292470948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a815c820 0x7f01a80b23a0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f0198009850 tx=0x7f019802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec8e280d1fa2f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.168+0000 7f01a5400640 1 -- 172.21.15.62:0/292470948 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0198004230 con 0x7f01a815c820 2024-03-23T11:32:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.168+0000 7f01a5400640 1 -- 172.21.15.62:0/292470948 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0198004390 con 0x7f01a815c820 2024-03-23T11:32:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.168+0000 7f01a5400640 1 -- 172.21.15.62:0/292470948 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0198003810 con 0x7f01a815c820 2024-03-23T11:32:55.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.168+0000 7f01a7200640 1 -- 172.21.15.62:0/292470948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a815c820 msgr2=0x7f01a80b23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.168+0000 7f01a7200640 1 --2- 172.21.15.62:0/292470948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a815c820 0x7f01a80b23a0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f0198009850 tx=0x7f019802b8f0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.168+0000 7f01a7200640 1 -- 172.21.15.62:0/292470948 shutdown_connections 2024-03-23T11:32:55.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.168+0000 7f01a7200640 1 --2- 172.21.15.62:0/292470948 >> v2:172.21.15.120:3300/0 conn(0x7f01a80b28e0 0x7f01a80cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.168+0000 7f01a7200640 1 --2- 172.21.15.62:0/292470948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a815c820 0x7f01a80b23a0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.168+0000 7f01a7200640 1 -- 172.21.15.62:0/292470948 >> 172.21.15.62:0/292470948 conn(0x7f01a809b9d0 msgr2=0x7f01a8089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:55.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.169+0000 7f01a7200640 1 -- 172.21.15.62:0/292470948 shutdown_connections 2024-03-23T11:32:55.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.169+0000 7f01a7200640 1 -- 172.21.15.62:0/292470948 wait complete. 2024-03-23T11:32:55.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.170+0000 7f01a7200640 1 Processor -- start 2024-03-23T11:32:55.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.170+0000 7f01a7200640 1 -- start start 2024-03-23T11:32:55.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.171+0000 7f01a7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01a80b28e0 0x7f01a8151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.171+0000 7f01a7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a80cf540 0x7f01a8154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.171+0000 7f01a7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a815c820 0x7f01a80b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.171+0000 7f01a7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a8152100 con 0x7f01a815c820 2024-03-23T11:32:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.171+0000 7f01a7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a8152970 con 0x7f01a80b28e0 2024-03-23T11:32:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.171+0000 7f01a7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a8152c50 con 0x7f01a80cf540 2024-03-23T11:32:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.172+0000 7f01a6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a815c820 0x7f01a80b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.172+0000 7f01a6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a815c820 0x7f01a80b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39776/0 (socket says 172.21.15.62:39776) 2024-03-23T11:32:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.172+0000 7f01a6800640 1 -- 172.21.15.62:0/157534291 learned_addr learned my addr 172.21.15.62:0/157534291 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.172+0000 7f01a5e00640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01a80b28e0 0x7f01a8151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.172+0000 7f019fe00640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a80cf540 0x7f01a8154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.172+0000 7f01a6800640 1 -- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a80cf540 msgr2=0x7f01a8154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.172+0000 7f01a6800640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01a80cf540 0x7f01a8154400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.172+0000 7f01a6800640 1 -- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01a80b28e0 msgr2=0x7f01a8151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.172+0000 7f01a6800640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01a80b28e0 0x7f01a8151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.172+0000 7f01a6800640 1 -- 172.21.15.62:0/157534291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0198009460 con 0x7f01a815c820 2024-03-23T11:32:55.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.172+0000 7f01a5e00640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01a80b28e0 0x7f01a8151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:55.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.173+0000 7f01a6800640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a815c820 0x7f01a80b3f20 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f01a0001560 tx=0x7f01a000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:55.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.173+0000 7f019e000640 1 -- 172.21.15.62:0/157534291 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0003da0 con 0x7f01a815c820 2024-03-23T11:32:55.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.173+0000 7f019e000640 1 -- 172.21.15.62:0/157534291 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a0005070 con 0x7f01a815c820 2024-03-23T11:32:55.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.173+0000 7f019e000640 1 -- 172.21.15.62:0/157534291 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0005420 con 0x7f01a815c820 2024-03-23T11:32:55.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.173+0000 7f01a7200640 1 -- 172.21.15.62:0/157534291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a8152f30 con 0x7f01a815c820 2024-03-23T11:32:55.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.173+0000 7f01a7200640 1 -- 172.21.15.62:0/157534291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a80c6b30 con 0x7f01a815c820 2024-03-23T11:32:55.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.174+0000 7f01a7200640 1 -- 172.21.15.62:0/157534291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f01a80d0450 con 0x7f01a815c820 2024-03-23T11:32:55.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.175+0000 7f019e000640 1 -- 172.21.15.62:0/157534291 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f01a0005580 con 0x7f01a815c820 2024-03-23T11:32:55.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.176+0000 7f019e000640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f018406a7a0 0x7f018406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.176+0000 7f019e000640 1 -- 172.21.15.62:0/157534291 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f01a00884c0 con 0x7f01a815c820 2024-03-23T11:32:55.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.176+0000 7f019e000640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x7f0184070740 0x7f0184072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.176+0000 7f019e000640 1 -- 172.21.15.62:0/157534291 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0184073210 con 0x7f0184070740 2024-03-23T11:32:55.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.176+0000 7f019e000640 1 -- 172.21.15.62:0/157534291 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01a008aaa0 con 0x7f01a815c820 2024-03-23T11:32:55.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.177+0000 7f019fe00640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x7f0184070740 0x7f0184072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.177+0000 7f01a5e00640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f018406a7a0 0x7f018406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.177+0000 7f019fe00640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x7f0184070740 0x7f0184072b60 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:55.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.178+0000 7f01a5e00640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f018406a7a0 0x7f018406cc60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f0198035000 tx=0x7f0198008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:55.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.178+0000 7f019e000640 1 -- 172.21.15.62:0/157534291 <== osd.2 v2:172.21.15.62:6800/36996 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f0184073210 con 0x7f0184070740 2024-03-23T11:32:55.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.192+0000 7f01a7200640 1 -- 172.21.15.62:0/157534291 --> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f01a80c6990 con 0x7f0184070740 2024-03-23T11:32:55.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.192+0000 7f019e000640 1 -- 172.21.15.62:0/157534291 <== osd.2 v2:172.21.15.62:6800/36996 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f01a80c6990 con 0x7f0184070740 2024-03-23T11:32:55.182 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:55.182 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T11:32:55.182 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:55.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.193+0000 7f01a7200640 1 -- 172.21.15.62:0/157534291 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x7f0184070740 msgr2=0x7f0184072b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.193+0000 7f01a7200640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.62:6800/36996,v1:172.21.15.62:6801/36996] conn(0x7f0184070740 0x7f0184072b60 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.193+0000 7f01a7200640 1 -- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f018406a7a0 msgr2=0x7f018406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.193+0000 7f01a7200640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f018406a7a0 0x7f018406cc60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f0198035000 tx=0x7f0198008030 comp rx=0 tx=0).stop 2024-03-23T11:32:55.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.193+0000 7f01a7200640 1 -- 172.21.15.62:0/157534291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a815c820 msgr2=0x7f01a80b3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.193+0000 7f01a7200640 1 --2- 172.21.15.62:0/157534291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a815c820 0x7f01a80b3f20 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f01a0001560 tx=0x7f01a000cc70 comp rx=0 tx=0).stop 2024-03-23T11:32:55.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.193+0000 7f01a6800640 1 -- 172.21.15.62:0/157534291 reap_dead start 2024-03-23T11:32:55.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.193+0000 7f01a7200640 1 -- 172.21.15.62:0/157534291 shutdown_connections 2024-03-23T11:32:55.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.193+0000 7f01a7200640 1 -- 172.21.15.62:0/157534291 >> 172.21.15.62:0/157534291 conn(0x7f01a809b9d0 msgr2=0x7f01a80b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:55.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.193+0000 7f01a7200640 1 -- 172.21.15.62:0/157534291 shutdown_connections 2024-03-23T11:32:55.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.193+0000 7f01a7200640 1 -- 172.21.15.62:0/157534291 wait complete. 2024-03-23T11:32:55.201 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-23T11:32:55.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.433+0000 7fc2af800640 1 Processor -- start 2024-03-23T11:32:55.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.433+0000 7fc2af800640 1 -- start start 2024-03-23T11:32:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.435+0000 7fc2af800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc2a81571e0 0x7fc2a80ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.435+0000 7fc2af800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80ade70 0x7fc2a806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.435+0000 7fc2af800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a806cf40 con 0x7fc2a81613a0 2024-03-23T11:32:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.435+0000 7fc2af800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a81575c0 con 0x7fc2a81571e0 2024-03-23T11:32:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.435+0000 7fc2af800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a806c3a0 con 0x7fc2a80ade70 2024-03-23T11:32:55.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.435+0000 7fc2ad800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc2a81613a0 0x7fc2a815e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35568/0 (socket says 172.21.15.62:35568) 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.435+0000 7fc2ad800640 1 -- 172.21.15.62:0/1091989013 learned_addr learned my addr 172.21.15.62:0/1091989013 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.435+0000 7fc2ace00640 1 --2- 172.21.15.62:0/1091989013 >> v2:172.21.15.120:3300/0 conn(0x7fc2a81571e0 0x7fc2a80ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.435+0000 7fc2a7e00640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80ade70 0x7fc2a806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860687305 0 0) 0x7fc2a806cf40 con 0x7fc2a81613a0 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc290003520 con 0x7fc2a81613a0 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3603306417 0 0) 0x7fc290003520 con 0x7fc2a81613a0 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80ade70 msgr2=0x7fc2a806d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80ade70 0x7fc2a806d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 >> v2:172.21.15.120:3300/0 conn(0x7fc2a81571e0 msgr2=0x7fc2a80ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 --2- 172.21.15.62:0/1091989013 >> v2:172.21.15.120:3300/0 conn(0x7fc2a81571e0 0x7fc2a80ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a806c610 con 0x7fc2a81613a0 2024-03-23T11:32:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fc2a00019f0 con 0x7fc2a81613a0 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 >> v1:172.21.15.62:6789/0 conn(0x7fc2a81613a0 legacy=0x7fc2a815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc290003970 0x7fc290003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2900047c0 0x7fc290004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80ade70 0x7fc290005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a81575c0 con 0x7fc290003970 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a806c3a0 con 0x7fc2900047c0 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.436+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29000ba00 con 0x7fc2a80ade70 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.437+0000 7fc2ace00640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc290003970 0x7fc290003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.437+0000 7fc2a7e00640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2900047c0 0x7fc290004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.437+0000 7fc2ad800640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80ade70 0x7fc290005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.437+0000 7fc2ace00640 1 -- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80ade70 msgr2=0x7fc290005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.437+0000 7fc2ace00640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80ade70 0x7fc290005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.437+0000 7fc2ace00640 1 -- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2900047c0 msgr2=0x7fc290004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:55.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.437+0000 7fc2ace00640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2900047c0 0x7fc290004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.437+0000 7fc2ace00640 1 -- 172.21.15.62:0/1091989013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc290003520 con 0x7fc290003970 2024-03-23T11:32:55.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.437+0000 7fc2a7e00640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2900047c0 0x7fc290004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.439+0000 7fc2ace00640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc290003970 0x7fc290003d90 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009ac0 tx=0x7fc2900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e583ae8ebd1767 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.439+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c02c710 con 0x7fc290003970 2024-03-23T11:32:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.439+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29c02c870 con 0x7fc290003970 2024-03-23T11:32:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.439+0000 7fc2a7400640 1 -- 172.21.15.62:0/1091989013 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c02cc00 con 0x7fc290003970 2024-03-23T11:32:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.440+0000 7fc2af800640 1 -- 172.21.15.62:0/1091989013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc290003970 msgr2=0x7fc290003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.440+0000 7fc2af800640 1 --2- 172.21.15.62:0/1091989013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc290003970 0x7fc290003d90 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009ac0 tx=0x7fc2900057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.440+0000 7fc2ad800640 1 -- 172.21.15.62:0/1091989013 reap_dead start 2024-03-23T11:32:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.440+0000 7fc2af800640 1 -- 172.21.15.62:0/1091989013 shutdown_connections 2024-03-23T11:32:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.440+0000 7fc2af800640 1 -- 172.21.15.62:0/1091989013 >> 172.21.15.62:0/1091989013 conn(0x7fc2a80b02e0 msgr2=0x7fc2a80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.440+0000 7fc2af800640 1 -- 172.21.15.62:0/1091989013 shutdown_connections 2024-03-23T11:32:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.440+0000 7fc2af800640 1 -- 172.21.15.62:0/1091989013 wait complete. 2024-03-23T11:32:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.442+0000 7fc2af800640 1 Processor -- start 2024-03-23T11:32:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.442+0000 7fc2af800640 1 -- start start 2024-03-23T11:32:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.443+0000 7fc2af800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80cd0b0 0x7fc2a8157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.443+0000 7fc2af800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2a80cd490 0x7fc2a8157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.443+0000 7fc2af800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a8156110 0x7fc2a8156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.443+0000 7fc2af800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8139340 con 0x7fc2a8156110 2024-03-23T11:32:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.443+0000 7fc2af800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a813b5d0 con 0x7fc2a80cd490 2024-03-23T11:32:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.443+0000 7fc2af800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8158360 con 0x7fc2a80cd0b0 2024-03-23T11:32:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.443+0000 7fc2ad800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a8156110 0x7fc2a8156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.443+0000 7fc2ad800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a8156110 0x7fc2a8156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39788/0 (socket says 172.21.15.62:39788) 2024-03-23T11:32:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.443+0000 7fc2ad800640 1 -- 172.21.15.62:0/3199945382 learned_addr learned my addr 172.21.15.62:0/3199945382 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.444+0000 7fc2a7e00640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2a80cd490 0x7fc2a8157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.444+0000 7fc2ace00640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80cd0b0 0x7fc2a8157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.444+0000 7fc2ad800640 1 -- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80cd0b0 msgr2=0x7fc2a8157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.444+0000 7fc2ad800640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80cd0b0 0x7fc2a8157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.444+0000 7fc2ad800640 1 -- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2a80cd490 msgr2=0x7fc2a8157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.444+0000 7fc2ad800640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2a80cd490 0x7fc2a8157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.444+0000 7fc2ad800640 1 -- 172.21.15.62:0/3199945382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a8158640 con 0x7fc2a8156110 2024-03-23T11:32:55.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.444+0000 7fc2ad800640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a8156110 0x7fc2a8156590 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00013d0 tx=0x7fc2a000e220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:55.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.445+0000 7fc2a4c00640 1 -- 172.21.15.62:0/3199945382 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a000ebf0 con 0x7fc2a8156110 2024-03-23T11:32:55.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.445+0000 7fc2a4c00640 1 -- 172.21.15.62:0/3199945382 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a000ed50 con 0x7fc2a8156110 2024-03-23T11:32:55.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.445+0000 7fc2a4c00640 1 -- 172.21.15.62:0/3199945382 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0013660 con 0x7fc2a8156110 2024-03-23T11:32:55.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.445+0000 7fc2af800640 1 -- 172.21.15.62:0/3199945382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a8138cf0 con 0x7fc2a8156110 2024-03-23T11:32:55.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.445+0000 7fc2af800640 1 -- 172.21.15.62:0/3199945382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a8156ad0 con 0x7fc2a8156110 2024-03-23T11:32:55.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.447+0000 7fc2af800640 1 -- 172.21.15.62:0/3199945382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a8066c90 con 0x7fc2a8156110 2024-03-23T11:32:55.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.448+0000 7fc2a4c00640 1 -- 172.21.15.62:0/3199945382 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fc2a00137c0 con 0x7fc2a8156110 2024-03-23T11:32:55.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.448+0000 7fc2a4c00640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28806a7d0 0x7fc28806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.449+0000 7fc2a4c00640 1 -- 172.21.15.62:0/3199945382 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fc2a0088f30 con 0x7fc2a8156110 2024-03-23T11:32:55.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.449+0000 7fc2ace00640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28806a7d0 0x7fc28806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.450+0000 7fc2ace00640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28806a7d0 0x7fc28806cc90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009ac0 tx=0x7fc29c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:55.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.454+0000 7fc2a4c00640 1 -- 172.21.15.62:0/3199945382 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc2a00545a0 con 0x7fc2a8156110 2024-03-23T11:32:55.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.651+0000 7fc2af800640 1 -- 172.21.15.62:0/3199945382 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2a806cbd0 con 0x7fc28806a7d0 2024-03-23T11:32:55.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.652+0000 7fc2a4c00640 1 -- 172.21.15.62:0/3199945382 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc29c02c9e0 con 0x7fc28806a7d0 2024-03-23T11:32:55.641 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 2 to deep-scrub 2024-03-23T11:32:55.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 -- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28806a7d0 msgr2=0x7fc28806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28806a7d0 0x7fc28806cc90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009ac0 tx=0x7fc29c030000 comp rx=0 tx=0).stop 2024-03-23T11:32:55.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 -- 172.21.15.62:0/3199945382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a8156110 msgr2=0x7fc2a8156590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a8156110 0x7fc2a8156590 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00013d0 tx=0x7fc2a000e220 comp rx=0 tx=0).stop 2024-03-23T11:32:55.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 -- 172.21.15.62:0/3199945382 shutdown_connections 2024-03-23T11:32:55.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fc28806a7d0 0x7fc28806cc90 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a8156110 0x7fc2a8156590 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2a80cd490 0x7fc2a8157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 --2- 172.21.15.62:0/3199945382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc2a80cd0b0 0x7fc2a8157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 -- 172.21.15.62:0/3199945382 >> 172.21.15.62:0/3199945382 conn(0x7fc2a80b02e0 msgr2=0x7fc2a80af870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:55.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 -- 172.21.15.62:0/3199945382 shutdown_connections 2024-03-23T11:32:55.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.655+0000 7fc2af800640 1 -- 172.21.15.62:0/3199945382 wait complete. 2024-03-23T11:32:55.661 INFO:tasks.ceph:Scrubbing osd.3 2024-03-23T11:32:55.662 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-23T11:32:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.893+0000 7f3081000640 1 Processor -- start 2024-03-23T11:32:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.893+0000 7f3081000640 1 -- start start 2024-03-23T11:32:55.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.895+0000 7f3081000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f307c0b0f00 0x7f307c070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.895+0000 7f3081000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f307c0cd1a0 0x7f307c0cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.895+0000 7f3081000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c0cdb70 con 0x7f307c15a8a0 2024-03-23T11:32:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.895+0000 7f3081000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c06d610 con 0x7f307c0cd1a0 2024-03-23T11:32:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.895+0000 7f3081000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c06d8f0 con 0x7f307c0b0f00 2024-03-23T11:32:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.895+0000 7f307a800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f307c0cd1a0 0x7f307c0cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.895+0000 7f307a800640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f307c0cd1a0 0x7f307c0cd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39790/0 (socket says 172.21.15.62:39790) 2024-03-23T11:32:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.895+0000 7f307a800640 1 -- 172.21.15.62:0/803652882 learned_addr learned my addr 172.21.15.62:0/803652882 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.895+0000 7f3079400640 1 --2- 172.21.15.62:0/803652882 >> v2:172.21.15.120:3300/0 conn(0x7f307c0b0f00 0x7f307c070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.896+0000 7f307a800640 1 -- 172.21.15.62:0/803652882 >> v2:172.21.15.120:3300/0 conn(0x7f307c0b0f00 msgr2=0x7f307c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.896+0000 7f307a800640 1 --2- 172.21.15.62:0/803652882 >> v2:172.21.15.120:3300/0 conn(0x7f307c0b0f00 0x7f307c070270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.896+0000 7f307a800640 1 -- 172.21.15.62:0/803652882 >> v1:172.21.15.120:6789/0 conn(0x7f307c15a8a0 legacy=0x7f307c0b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.896+0000 7f307a800640 1 -- 172.21.15.62:0/803652882 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c06dbd0 con 0x7f307c0cd1a0 2024-03-23T11:32:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.898+0000 7f307a800640 1 --2- 172.21.15.62:0/803652882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c0cd1a0 0x7f307c0cd630 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f307c066500 tx=0x7f307402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dceca622f57aaac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.898+0000 7f3078a00640 1 -- 172.21.15.62:0/803652882 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307402ca80 con 0x7f307c0cd1a0 2024-03-23T11:32:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.898+0000 7f3078a00640 1 -- 172.21.15.62:0/803652882 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307402cbe0 con 0x7f307c0cd1a0 2024-03-23T11:32:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.898+0000 7f3078a00640 1 -- 172.21.15.62:0/803652882 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30740343f0 con 0x7f307c0cd1a0 2024-03-23T11:32:55.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.898+0000 7f3081000640 1 -- 172.21.15.62:0/803652882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c0cd1a0 msgr2=0x7f307c0cd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.898+0000 7f3081000640 1 --2- 172.21.15.62:0/803652882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c0cd1a0 0x7f307c0cd630 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f307c066500 tx=0x7f307402bee0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.899+0000 7f3081000640 1 -- 172.21.15.62:0/803652882 shutdown_connections 2024-03-23T11:32:55.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.899+0000 7f3081000640 1 --2- 172.21.15.62:0/803652882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c0cd1a0 0x7f307c0cd630 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.899+0000 7f3081000640 1 --2- 172.21.15.62:0/803652882 >> v2:172.21.15.120:3300/0 conn(0x7f307c0b0f00 0x7f307c070270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.899+0000 7f3081000640 1 -- 172.21.15.62:0/803652882 >> 172.21.15.62:0/803652882 conn(0x7f307c0b17d0 msgr2=0x7f307c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:55.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.899+0000 7f3081000640 1 -- 172.21.15.62:0/803652882 shutdown_connections 2024-03-23T11:32:55.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.899+0000 7f3081000640 1 -- 172.21.15.62:0/803652882 wait complete. 2024-03-23T11:32:55.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.901+0000 7f3081000640 1 Processor -- start 2024-03-23T11:32:55.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.901+0000 7f3081000640 1 -- start start 2024-03-23T11:32:55.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.902+0000 7f3081000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307c0b0f00 0x7f307c159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.902+0000 7f3081000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f307c0cd1a0 0x7f307c14e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.902+0000 7f3081000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c15a8a0 0x7f307c1511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.902+0000 7f3081000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c14e860 con 0x7f307c15a8a0 2024-03-23T11:32:55.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.902+0000 7f3081000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c158b70 con 0x7f307c0cd1a0 2024-03-23T11:32:55.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.902+0000 7f3081000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c158e50 con 0x7f307c0b0f00 2024-03-23T11:32:55.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.902+0000 7f307a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c15a8a0 0x7f307c1511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.902+0000 7f307a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c15a8a0 0x7f307c1511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39798/0 (socket says 172.21.15.62:39798) 2024-03-23T11:32:55.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.902+0000 7f307a800640 1 -- 172.21.15.62:0/29437355 learned_addr learned my addr 172.21.15.62:0/29437355 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:55.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.902+0000 7f3079e00640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307c0b0f00 0x7f307c159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.902+0000 7f3079400640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f307c0cd1a0 0x7f307c14e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.903+0000 7f307a800640 1 -- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307c0b0f00 msgr2=0x7f307c159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.903+0000 7f307a800640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f307c0b0f00 0x7f307c159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.903+0000 7f307a800640 1 -- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f307c0cd1a0 msgr2=0x7f307c14e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.903+0000 7f307a800640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f307c0cd1a0 0x7f307c14e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.903+0000 7f307a800640 1 -- 172.21.15.62:0/29437355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3074009b30 con 0x7f307c15a8a0 2024-03-23T11:32:55.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.903+0000 7f3079400640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f307c0cd1a0 0x7f307c14e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:32:55.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.903+0000 7f307a800640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c15a8a0 0x7f307c1511e0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f3074036040 tx=0x7f3074004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:55.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.903+0000 7f3072000640 1 -- 172.21.15.62:0/29437355 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074009510 con 0x7f307c15a8a0 2024-03-23T11:32:55.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.903+0000 7f3072000640 1 -- 172.21.15.62:0/29437355 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307403e450 con 0x7f307c15a8a0 2024-03-23T11:32:55.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.903+0000 7f3072000640 1 -- 172.21.15.62:0/29437355 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307403e800 con 0x7f307c15a8a0 2024-03-23T11:32:55.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.903+0000 7f3081000640 1 -- 172.21.15.62:0/29437355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c159130 con 0x7f307c15a8a0 2024-03-23T11:32:55.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.904+0000 7f3081000640 1 -- 172.21.15.62:0/29437355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c1596f0 con 0x7f307c15a8a0 2024-03-23T11:32:55.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.904+0000 7f3081000640 1 -- 172.21.15.62:0/29437355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f307c15ac80 con 0x7f307c15a8a0 2024-03-23T11:32:55.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.906+0000 7f3072000640 1 -- 172.21.15.62:0/29437355 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f307402cd50 con 0x7f307c15a8a0 2024-03-23T11:32:55.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.907+0000 7f3072000640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f305c06a7a0 0x7f305c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.907+0000 7f3072000640 1 -- 172.21.15.62:0/29437355 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f30740abb80 con 0x7f307c15a8a0 2024-03-23T11:32:55.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.907+0000 7f3072000640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x7f305c070740 0x7f305c072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:55.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.907+0000 7f3072000640 1 -- 172.21.15.62:0/29437355 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f305c073210 con 0x7f305c070740 2024-03-23T11:32:55.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.907+0000 7f3072000640 1 -- 172.21.15.62:0/29437355 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30740abef0 con 0x7f307c15a8a0 2024-03-23T11:32:55.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.907+0000 7f3079400640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x7f305c070740 0x7f305c072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.907+0000 7f3079e00640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f305c06a7a0 0x7f305c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:55.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.908+0000 7f3079400640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x7f305c070740 0x7f305c072b60 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:55.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.908+0000 7f3079e00640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f305c06a7a0 0x7f305c06cc60 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f306c000f60 tx=0x7f306c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:55.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.909+0000 7f3072000640 1 -- 172.21.15.62:0/29437355 <== osd.3 v2:172.21.15.62:6808/37000 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f305c073210 con 0x7f305c070740 2024-03-23T11:32:55.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.922+0000 7f3081000640 1 -- 172.21.15.62:0/29437355 --> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f307c159910 con 0x7f305c070740 2024-03-23T11:32:55.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.923+0000 7f3072000640 1 -- 172.21.15.62:0/29437355 <== osd.3 v2:172.21.15.62:6808/37000 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f307c159910 con 0x7f305c070740 2024-03-23T11:32:55.912 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:55.912 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T11:32:55.912 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:55.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.923+0000 7f3081000640 1 -- 172.21.15.62:0/29437355 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x7f305c070740 msgr2=0x7f305c072b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.923+0000 7f3081000640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.62:6808/37000,v1:172.21.15.62:6809/37000] conn(0x7f305c070740 0x7f305c072b60 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:55.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.923+0000 7f3081000640 1 -- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f305c06a7a0 msgr2=0x7f305c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.923+0000 7f3081000640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f305c06a7a0 0x7f305c06cc60 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f306c000f60 tx=0x7f306c009040 comp rx=0 tx=0).stop 2024-03-23T11:32:55.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.923+0000 7f3081000640 1 -- 172.21.15.62:0/29437355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c15a8a0 msgr2=0x7f307c1511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:55.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.923+0000 7f3081000640 1 --2- 172.21.15.62:0/29437355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f307c15a8a0 0x7f307c1511e0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f3074036040 tx=0x7f3074004a40 comp rx=0 tx=0).stop 2024-03-23T11:32:55.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.923+0000 7f307a800640 1 -- 172.21.15.62:0/29437355 reap_dead start 2024-03-23T11:32:55.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.924+0000 7f3081000640 1 -- 172.21.15.62:0/29437355 shutdown_connections 2024-03-23T11:32:55.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.924+0000 7f3081000640 1 -- 172.21.15.62:0/29437355 >> 172.21.15.62:0/29437355 conn(0x7f307c0b17d0 msgr2=0x7f307c0b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:55.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.924+0000 7f3081000640 1 -- 172.21.15.62:0/29437355 shutdown_connections 2024-03-23T11:32:55.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:55.924+0000 7f3081000640 1 -- 172.21.15.62:0/29437355 wait complete. 2024-03-23T11:32:55.931 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-23T11:32:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.152+0000 7fef89800640 1 Processor -- start 2024-03-23T11:32:56.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.152+0000 7fef89800640 1 -- start start 2024-03-23T11:32:56.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.154+0000 7fef89800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fef840b2f30 0x7fef84070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.154+0000 7fef89800640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef840cd140 con 0x7fef84070090 2024-03-23T11:32:56.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.154+0000 7fef89800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef840cd3e0 con 0x7fef84143dd0 2024-03-23T11:32:56.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.154+0000 7fef89800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef84070a60 con 0x7fef840b2f30 2024-03-23T11:32:56.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.154+0000 7fef7b400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fef84070090 0x7fef840b2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35572/0 (socket says 172.21.15.62:35572) 2024-03-23T11:32:56.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.154+0000 7fef7b400640 1 -- 172.21.15.62:0/3041573169 learned_addr learned my addr 172.21.15.62:0/3041573169 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.154+0000 7fef88e00640 1 --2- 172.21.15.62:0/3041573169 >> v2:172.21.15.120:3300/0 conn(0x7fef840b2f30 0x7fef84070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.154+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071355872 0 0) 0x7fef840cd140 con 0x7fef84070090 2024-03-23T11:32:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef68003520 con 0x7fef84070090 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3856013129 0 0) 0x7fef68003520 con 0x7fef84070090 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 >> v2:172.21.15.120:3300/0 conn(0x7fef840b2f30 msgr2=0x7fef84070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 --2- 172.21.15.62:0/3041573169 >> v2:172.21.15.120:3300/0 conn(0x7fef840b2f30 0x7fef84070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 >> v1:172.21.15.120:6789/0 conn(0x7fef84143dd0 legacy=0x7fef8415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef840cd6c0 con 0x7fef84070090 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fef6c001670 con 0x7fef84070090 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 >> v1:172.21.15.62:6789/0 conn(0x7fef84070090 legacy=0x7fef840b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 --2- 172.21.15.62:0/3041573169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef68003970 0x7fef68003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 --2- 172.21.15.62:0/3041573169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef680047c0 0x7fef68004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 --2- 172.21.15.62:0/3041573169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef68005180 0x7fef6800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef84070a60 con 0x7fef680047c0 2024-03-23T11:32:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6800c020 con 0x7fef68003970 2024-03-23T11:32:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6800c1e0 con 0x7fef68005180 2024-03-23T11:32:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.155+0000 7fef7b400640 1 --2- 172.21.15.62:0/3041573169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef680047c0 0x7fef68004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.156+0000 7fef83e00640 1 --2- 172.21.15.62:0/3041573169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef68003970 0x7fef68003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.156+0000 7fef7b400640 1 -- 172.21.15.62:0/3041573169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef68005180 msgr2=0x7fef6800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.156+0000 7fef7b400640 1 --2- 172.21.15.62:0/3041573169 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef68005180 0x7fef6800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.156+0000 7fef7b400640 1 -- 172.21.15.62:0/3041573169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef68003970 msgr2=0x7fef68003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.156+0000 7fef7b400640 1 --2- 172.21.15.62:0/3041573169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef68003970 0x7fef68003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.156+0000 7fef7b400640 1 -- 172.21.15.62:0/3041573169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef68003520 con 0x7fef680047c0 2024-03-23T11:32:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.156+0000 7fef88e00640 1 -- 172.21.15.62:0/3041573169 reap_dead start 2024-03-23T11:32:56.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.157+0000 7fef7b400640 1 --2- 172.21.15.62:0/3041573169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef680047c0 0x7fef68004c40 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00c240 tx=0x7fef680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c412522d0d5c219e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:56.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.157+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c005ad0 con 0x7fef680047c0 2024-03-23T11:32:56.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.157+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c005c30 con 0x7fef680047c0 2024-03-23T11:32:56.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.157+0000 7fef83400640 1 -- 172.21.15.62:0/3041573169 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c02cab0 con 0x7fef680047c0 2024-03-23T11:32:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.158+0000 7fef89800640 1 -- 172.21.15.62:0/3041573169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef680047c0 msgr2=0x7fef68004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.158+0000 7fef89800640 1 --2- 172.21.15.62:0/3041573169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef680047c0 0x7fef68004c40 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00c240 tx=0x7fef680057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.158+0000 7fef89800640 1 -- 172.21.15.62:0/3041573169 shutdown_connections 2024-03-23T11:32:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.158+0000 7fef89800640 1 --2- 172.21.15.62:0/3041573169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef680047c0 0x7fef68004c40 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.158+0000 7fef89800640 1 -- 172.21.15.62:0/3041573169 >> 172.21.15.62:0/3041573169 conn(0x7fef84066050 msgr2=0x7fef84069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.158+0000 7fef89800640 1 -- 172.21.15.62:0/3041573169 shutdown_connections 2024-03-23T11:32:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.158+0000 7fef89800640 1 -- 172.21.15.62:0/3041573169 wait complete. 2024-03-23T11:32:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.160+0000 7fef89800640 1 Processor -- start 2024-03-23T11:32:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.160+0000 7fef89800640 1 -- start start 2024-03-23T11:32:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.161+0000 7fef89800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef680047c0 0x7fef8413e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.161+0000 7fef89800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef8413c330 0x7fef841f12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.161+0000 7fef89800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef84141150 0x7fef841415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.161+0000 7fef89800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8413ea60 con 0x7fef84141150 2024-03-23T11:32:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.161+0000 7fef89800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef840b07e0 con 0x7fef680047c0 2024-03-23T11:32:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.161+0000 7fef89800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef840b0ac0 con 0x7fef8413c330 2024-03-23T11:32:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.161+0000 7fef88e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef84141150 0x7fef841415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.161+0000 7fef88e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef84141150 0x7fef841415d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39812/0 (socket says 172.21.15.62:39812) 2024-03-23T11:32:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.161+0000 7fef88e00640 1 -- 172.21.15.62:0/4092537412 learned_addr learned my addr 172.21.15.62:0/4092537412 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.161+0000 7fef7b400640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef8413c330 0x7fef841f12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.162+0000 7fef83e00640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef680047c0 0x7fef8413e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.162+0000 7fef88e00640 1 -- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef8413c330 msgr2=0x7fef841f12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.162+0000 7fef88e00640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef8413c330 0x7fef841f12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.162+0000 7fef88e00640 1 -- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef680047c0 msgr2=0x7fef8413e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.162+0000 7fef88e00640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef680047c0 0x7fef8413e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.162+0000 7fef88e00640 1 -- 172.21.15.62:0/4092537412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8413c790 con 0x7fef84141150 2024-03-23T11:32:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.162+0000 7fef88e00640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef84141150 0x7fef841415d0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fef8406e450 tx=0x7fef7c00ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.162+0000 7fef80c00640 1 -- 172.21.15.62:0/4092537412 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c010040 con 0x7fef84141150 2024-03-23T11:32:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.162+0000 7fef80c00640 1 -- 172.21.15.62:0/4092537412 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7c00bb70 con 0x7fef84141150 2024-03-23T11:32:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.162+0000 7fef89800640 1 -- 172.21.15.62:0/4092537412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef841f1820 con 0x7fef84141150 2024-03-23T11:32:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.163+0000 7fef80c00640 1 -- 172.21.15.62:0/4092537412 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c014400 con 0x7fef84141150 2024-03-23T11:32:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.163+0000 7fef89800640 1 -- 172.21.15.62:0/4092537412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef84141cf0 con 0x7fef84141150 2024-03-23T11:32:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.164+0000 7fef89800640 1 -- 172.21.15.62:0/4092537412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef84066bb0 con 0x7fef84141150 2024-03-23T11:32:56.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.165+0000 7fef80c00640 1 -- 172.21.15.62:0/4092537412 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fef7c014560 con 0x7fef84141150 2024-03-23T11:32:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.166+0000 7fef80c00640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef6006a7d0 0x7fef6006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.166+0000 7fef80c00640 1 -- 172.21.15.62:0/4092537412 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fef7c088da0 con 0x7fef84141150 2024-03-23T11:32:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.167+0000 7fef83e00640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef6006a7d0 0x7fef6006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.168+0000 7fef83e00640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef6006a7d0 0x7fef6006cc90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fef74005490 tx=0x7fef74009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.171+0000 7fef80c00640 1 -- 172.21.15.62:0/4092537412 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef7c0544a0 con 0x7fef84141150 2024-03-23T11:32:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.366+0000 7fef89800640 1 -- 172.21.15.62:0/4092537412 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fef840cdf60 con 0x7fef6006a7d0 2024-03-23T11:32:56.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.367+0000 7fef80c00640 1 -- 172.21.15.62:0/4092537412 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fef74001350 con 0x7fef6006a7d0 2024-03-23T11:32:56.356 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 3 to deep-scrub 2024-03-23T11:32:56.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.370+0000 7fef89800640 1 -- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef6006a7d0 msgr2=0x7fef6006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.370+0000 7fef89800640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef6006a7d0 0x7fef6006cc90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fef74005490 tx=0x7fef74009040 comp rx=0 tx=0).stop 2024-03-23T11:32:56.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.370+0000 7fef89800640 1 -- 172.21.15.62:0/4092537412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef84141150 msgr2=0x7fef841415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.370+0000 7fef89800640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef84141150 0x7fef841415d0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fef8406e450 tx=0x7fef7c00ef30 comp rx=0 tx=0).stop 2024-03-23T11:32:56.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.370+0000 7fef89800640 1 -- 172.21.15.62:0/4092537412 shutdown_connections 2024-03-23T11:32:56.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.370+0000 7fef89800640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fef6006a7d0 0x7fef6006cc90 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.370+0000 7fef89800640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef84141150 0x7fef841415d0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.370+0000 7fef89800640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef8413c330 0x7fef841f12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.370+0000 7fef89800640 1 --2- 172.21.15.62:0/4092537412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef680047c0 0x7fef8413e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.370+0000 7fef89800640 1 -- 172.21.15.62:0/4092537412 >> 172.21.15.62:0/4092537412 conn(0x7fef84066050 msgr2=0x7fef8415ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:56.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.370+0000 7fef89800640 1 -- 172.21.15.62:0/4092537412 shutdown_connections 2024-03-23T11:32:56.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.371+0000 7fef89800640 1 -- 172.21.15.62:0/4092537412 wait complete. 2024-03-23T11:32:56.379 INFO:tasks.ceph:Scrubbing osd.4 2024-03-23T11:32:56.379 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-23T11:32:56.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.614+0000 7fe172e00640 1 Processor -- start 2024-03-23T11:32:56.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.614+0000 7fe172e00640 1 -- start start 2024-03-23T11:32:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.616+0000 7fe172e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c06c890 0x7fe16c0b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.616+0000 7fe172e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe16c15cd50 0x7fe16c15d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.616+0000 7fe172e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c0b1160 con 0x7fe16c0ce690 2024-03-23T11:32:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.616+0000 7fe172e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16c0b20f0 con 0x7fe16c15cd50 2024-03-23T11:32:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.616+0000 7fe172e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c15d840 con 0x7fe16c06c890 2024-03-23T11:32:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.616+0000 7fe171000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe16c15cd50 0x7fe16c15d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.616+0000 7fe171000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe16c15cd50 0x7fe16c15d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39818/0 (socket says 172.21.15.62:39818) 2024-03-23T11:32:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.616+0000 7fe171000640 1 -- 172.21.15.62:0/700655369 learned_addr learned my addr 172.21.15.62:0/700655369 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:56.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.617+0000 7fe171a00640 1 --2- 172.21.15.62:0/700655369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c06c890 0x7fe16c0b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.617+0000 7fe171000640 1 -- 172.21.15.62:0/700655369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c06c890 msgr2=0x7fe16c0b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:56.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.617+0000 7fe171000640 1 --2- 172.21.15.62:0/700655369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c06c890 0x7fe16c0b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.617+0000 7fe171000640 1 -- 172.21.15.62:0/700655369 >> v1:172.21.15.62:6789/0 conn(0x7fe16c0ce690 legacy=0x7fe16c06ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.617+0000 7fe171000640 1 -- 172.21.15.62:0/700655369 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c15db20 con 0x7fe16c15cd50 2024-03-23T11:32:56.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.617+0000 7fe171a00640 1 --2- 172.21.15.62:0/700655369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c06c890 0x7fe16c0b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:56.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.618+0000 7fe171000640 1 --2- 172.21.15.62:0/700655369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe16c15cd50 0x7fe16c15d1b0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fe158009870 tx=0x7fe15802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c463bddfcf13d02 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:56.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.618+0000 7fe16be00640 1 -- 172.21.15.62:0/700655369 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158030840 con 0x7fe16c15cd50 2024-03-23T11:32:56.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.619+0000 7fe16be00640 1 -- 172.21.15.62:0/700655369 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1580309a0 con 0x7fe16c15cd50 2024-03-23T11:32:56.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.619+0000 7fe16be00640 1 -- 172.21.15.62:0/700655369 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158030d30 con 0x7fe16c15cd50 2024-03-23T11:32:56.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.619+0000 7fe172e00640 1 -- 172.21.15.62:0/700655369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe16c15cd50 msgr2=0x7fe16c15d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.619+0000 7fe172e00640 1 --2- 172.21.15.62:0/700655369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe16c15cd50 0x7fe16c15d1b0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fe158009870 tx=0x7fe15802b980 comp rx=0 tx=0).stop 2024-03-23T11:32:56.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.619+0000 7fe172e00640 1 -- 172.21.15.62:0/700655369 shutdown_connections 2024-03-23T11:32:56.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.619+0000 7fe172e00640 1 --2- 172.21.15.62:0/700655369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe16c15cd50 0x7fe16c15d1b0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.619+0000 7fe172e00640 1 --2- 172.21.15.62:0/700655369 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c06c890 0x7fe16c0b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.619+0000 7fe172e00640 1 -- 172.21.15.62:0/700655369 >> 172.21.15.62:0/700655369 conn(0x7fe16c0b0280 msgr2=0x7fe16c0b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:56.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.619+0000 7fe172e00640 1 -- 172.21.15.62:0/700655369 shutdown_connections 2024-03-23T11:32:56.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.619+0000 7fe172e00640 1 -- 172.21.15.62:0/700655369 wait complete. 2024-03-23T11:32:56.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.621+0000 7fe172e00640 1 Processor -- start 2024-03-23T11:32:56.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.621+0000 7fe172e00640 1 -- start start 2024-03-23T11:32:56.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.622+0000 7fe172e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe16c06c890 0x7fe16c098250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.622+0000 7fe172e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe16c0ce690 0x7fe16c160290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.622+0000 7fe172e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c15cd50 0x7fe16c15f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.622+0000 7fe172e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c1f1ee0 con 0x7fe16c06c890 2024-03-23T11:32:56.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.622+0000 7fe172e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c09a340 con 0x7fe16c0ce690 2024-03-23T11:32:56.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.622+0000 7fe172e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c151ff0 con 0x7fe16c15cd50 2024-03-23T11:32:56.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.622+0000 7fe171a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe16c06c890 0x7fe16c098250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.623+0000 7fe171a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe16c06c890 0x7fe16c098250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39820/0 (socket says 172.21.15.62:39820) 2024-03-23T11:32:56.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.623+0000 7fe171a00640 1 -- 172.21.15.62:0/1550123101 learned_addr learned my addr 172.21.15.62:0/1550123101 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:56.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.623+0000 7fe171000640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe16c0ce690 0x7fe16c160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.623+0000 7fe172400640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c15cd50 0x7fe16c15f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.623+0000 7fe171a00640 1 -- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c15cd50 msgr2=0x7fe16c15f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.623+0000 7fe171a00640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c15cd50 0x7fe16c15f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.623+0000 7fe171a00640 1 -- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe16c0ce690 msgr2=0x7fe16c160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.623+0000 7fe171a00640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe16c0ce690 0x7fe16c160290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.623+0000 7fe171a00640 1 -- 172.21.15.62:0/1550123101 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1580094d0 con 0x7fe16c06c890 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.623+0000 7fe172400640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe16c15cd50 0x7fe16c15f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.623+0000 7fe171a00640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe16c06c890 0x7fe16c098250 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fe16000c7f0 tx=0x7fe16000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.624+0000 7fe169600640 1 -- 172.21.15.62:0/1550123101 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160007a30 con 0x7fe16c06c890 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.624+0000 7fe169600640 1 -- 172.21.15.62:0/1550123101 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe160007b90 con 0x7fe16c06c890 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.624+0000 7fe169600640 1 -- 172.21.15.62:0/1550123101 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16000ce40 con 0x7fe16c06c890 2024-03-23T11:32:56.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.624+0000 7fe172e00640 1 -- 172.21.15.62:0/1550123101 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c15edb0 con 0x7fe16c06c890 2024-03-23T11:32:56.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.624+0000 7fe172e00640 1 -- 172.21.15.62:0/1550123101 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c099d80 con 0x7fe16c06c890 2024-03-23T11:32:56.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.625+0000 7fe172e00640 1 -- 172.21.15.62:0/1550123101 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe16c072190 con 0x7fe16c06c890 2024-03-23T11:32:56.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.626+0000 7fe169600640 1 -- 172.21.15.62:0/1550123101 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fe1600113f0 con 0x7fe16c06c890 2024-03-23T11:32:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.627+0000 7fe169600640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe14c06a7a0 0x7fe14c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.627+0000 7fe169600640 1 -- 172.21.15.62:0/1550123101 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fe160088c70 con 0x7fe16c06c890 2024-03-23T11:32:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.627+0000 7fe169600640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x7fe14c070740 0x7fe14c072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.627+0000 7fe169600640 1 -- 172.21.15.62:0/1550123101 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe14c073210 con 0x7fe14c070740 2024-03-23T11:32:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.627+0000 7fe169600640 1 -- 172.21.15.62:0/1550123101 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe16008b330 con 0x7fe16c06c890 2024-03-23T11:32:56.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.627+0000 7fe171000640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe14c06a7a0 0x7fe14c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.628+0000 7fe172400640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x7fe14c070740 0x7fe14c072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.629+0000 7fe171000640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe14c06a7a0 0x7fe14c06cc60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe158036000 tx=0x7fe15803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:56.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.629+0000 7fe172400640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x7fe14c070740 0x7fe14c072b60 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:56.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.631+0000 7fe169600640 1 -- 172.21.15.62:0/1550123101 <== osd.4 v2:172.21.15.120:6803/37640 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe14c073210 con 0x7fe14c070740 2024-03-23T11:32:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.646+0000 7fe172e00640 1 -- 172.21.15.62:0/1550123101 --> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe16c0b1690 con 0x7fe14c070740 2024-03-23T11:32:56.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.648+0000 7fe169600640 1 -- 172.21.15.62:0/1550123101 <== osd.4 v2:172.21.15.120:6803/37640 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe164001ea0 con 0x7fe14c070740 2024-03-23T11:32:56.637 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:56.637 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T11:32:56.637 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:56.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.648+0000 7fe172e00640 1 -- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x7fe14c070740 msgr2=0x7fe14c072b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.648+0000 7fe172e00640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:6803/37640,v1:172.21.15.120:6805/37640] conn(0x7fe14c070740 0x7fe14c072b60 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.648+0000 7fe172e00640 1 -- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe14c06a7a0 msgr2=0x7fe14c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.648+0000 7fe172e00640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fe14c06a7a0 0x7fe14c06cc60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe158036000 tx=0x7fe15803b040 comp rx=0 tx=0).stop 2024-03-23T11:32:56.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.648+0000 7fe172e00640 1 -- 172.21.15.62:0/1550123101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe16c06c890 msgr2=0x7fe16c098250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.648+0000 7fe172e00640 1 --2- 172.21.15.62:0/1550123101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe16c06c890 0x7fe16c098250 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fe16000c7f0 tx=0x7fe16000cbf0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.648+0000 7fe172400640 1 -- 172.21.15.62:0/1550123101 reap_dead start 2024-03-23T11:32:56.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.649+0000 7fe172e00640 1 -- 172.21.15.62:0/1550123101 shutdown_connections 2024-03-23T11:32:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.649+0000 7fe172e00640 1 -- 172.21.15.62:0/1550123101 >> 172.21.15.62:0/1550123101 conn(0x7fe16c0b0280 msgr2=0x7fe16c0ceae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.649+0000 7fe172e00640 1 -- 172.21.15.62:0/1550123101 shutdown_connections 2024-03-23T11:32:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.649+0000 7fe172e00640 1 -- 172.21.15.62:0/1550123101 wait complete. 2024-03-23T11:32:56.658 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-23T11:32:56.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.895+0000 7f63a3200640 1 Processor -- start 2024-03-23T11:32:56.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.896+0000 7f63a3200640 1 -- start start 2024-03-23T11:32:56.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.897+0000 7f63a3200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63a415c830 0x7f63a40ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.898+0000 7f63a3200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f63a4160310 0x7f63a40ab980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.898+0000 7f63a3200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a41608d0 con 0x7f63a40abec0 2024-03-23T11:32:56.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.898+0000 7f63a3200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a40addb0 con 0x7f63a4160310 2024-03-23T11:32:56.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.898+0000 7f63a3200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a415cc10 con 0x7f63a415c830 2024-03-23T11:32:56.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.898+0000 7f63a2800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f63a40abec0 0x7f63a40ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35598/0 (socket says 172.21.15.62:35598) 2024-03-23T11:32:56.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.898+0000 7f63a2800640 1 -- 172.21.15.62:0/1117200900 learned_addr learned my addr 172.21.15.62:0/1117200900 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:56.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.898+0000 7f63a1e00640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63a415c830 0x7f63a40ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.898+0000 7f639be00640 1 --2- 172.21.15.62:0/1117200900 >> v2:172.21.15.120:3300/0 conn(0x7f63a4160310 0x7f63a40ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.898+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735794508 0 0) 0x7f63a41608d0 con 0x7f63a40abec0 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6388003520 con 0x7f63a40abec0 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 27329874 0 0) 0x7f6388003520 con 0x7f63a40abec0 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63a415c830 msgr2=0x7f63a40ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63a415c830 0x7f63a40ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 >> v2:172.21.15.120:3300/0 conn(0x7f63a4160310 msgr2=0x7f63a40ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 --2- 172.21.15.62:0/1117200900 >> v2:172.21.15.120:3300/0 conn(0x7f63a4160310 0x7f63a40ab980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a4160bb0 con 0x7f63a40abec0 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f639c0019f0 con 0x7f63a40abec0 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 >> v1:172.21.15.62:6789/0 conn(0x7f63a40abec0 legacy=0x7f63a40ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63a415c830 0x7f63880039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388003f20 0x7f63880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6388004d00 0x7f6388005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a415cc10 con 0x7f6388003f20 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638800b980 con 0x7f6388004d00 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638800bac0 con 0x7f63a415c830 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.899+0000 7f639be00640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388003f20 0x7f63880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.900+0000 7f639be00640 1 -- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63a415c830 msgr2=0x7f63880039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.900+0000 7f63a1e00640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63a415c830 0x7f63880039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.900+0000 7f63a2800640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6388004d00 0x7f6388005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.900+0000 7f639be00640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63a415c830 0x7f63880039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.900+0000 7f639be00640 1 -- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6388004d00 msgr2=0x7f6388005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.900+0000 7f639be00640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6388004d00 0x7f6388005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.900+0000 7f639be00640 1 -- 172.21.15.62:0/1117200900 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6388003520 con 0x7f6388003f20 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.900+0000 7f63a2800640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6388004d00 0x7f6388005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:56.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.900+0000 7f63a1e00640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63a415c830 0x7f63880039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:56.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.901+0000 7f639be00640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388003f20 0x7f63880047c0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f638c009ac0 tx=0x7f63880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08b6f78225c0717 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:56.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.901+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c02c710 con 0x7f6388003f20 2024-03-23T11:32:56.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.901+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638c02c870 con 0x7f6388003f20 2024-03-23T11:32:56.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.901+0000 7f63a1400640 1 -- 172.21.15.62:0/1117200900 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c02cc00 con 0x7f6388003f20 2024-03-23T11:32:56.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.902+0000 7f63a3200640 1 -- 172.21.15.62:0/1117200900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388003f20 msgr2=0x7f63880047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.902+0000 7f63a3200640 1 --2- 172.21.15.62:0/1117200900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388003f20 0x7f63880047c0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f638c009ac0 tx=0x7f63880057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.902+0000 7f63a2800640 1 -- 172.21.15.62:0/1117200900 reap_dead start 2024-03-23T11:32:56.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.902+0000 7f63a3200640 1 -- 172.21.15.62:0/1117200900 shutdown_connections 2024-03-23T11:32:56.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.902+0000 7f63a3200640 1 -- 172.21.15.62:0/1117200900 >> 172.21.15.62:0/1117200900 conn(0x7f63a4066050 msgr2=0x7f63a4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:56.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.902+0000 7f63a3200640 1 -- 172.21.15.62:0/1117200900 shutdown_connections 2024-03-23T11:32:56.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.902+0000 7f63a3200640 1 -- 172.21.15.62:0/1117200900 wait complete. 2024-03-23T11:32:56.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.904+0000 7f63a3200640 1 Processor -- start 2024-03-23T11:32:56.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.904+0000 7f63a3200640 1 -- start start 2024-03-23T11:32:56.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.905+0000 7f63a3200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6388003f20 0x7f63a4150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.905+0000 7f63a3200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a4152900 0x7f63a40af6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.905+0000 7f63a3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63a40afc10 0x7f63a4070090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.905+0000 7f63a3200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a4152da0 con 0x7f63a40afc10 2024-03-23T11:32:56.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.905+0000 7f63a3200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a4153080 con 0x7f63a4152900 2024-03-23T11:32:56.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.905+0000 7f63a3200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a4153360 con 0x7f6388003f20 2024-03-23T11:32:56.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.905+0000 7f63a2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63a40afc10 0x7f63a4070090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.905+0000 7f63a2800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63a40afc10 0x7f63a4070090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39838/0 (socket says 172.21.15.62:39838) 2024-03-23T11:32:56.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.905+0000 7f63a2800640 1 -- 172.21.15.62:0/955690093 learned_addr learned my addr 172.21.15.62:0/955690093 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:56.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.906+0000 7f63a1e00640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6388003f20 0x7f63a4150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.906+0000 7f639be00640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a4152900 0x7f63a40af6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.906+0000 7f63a2800640 1 -- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6388003f20 msgr2=0x7f63a4150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.906+0000 7f63a2800640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6388003f20 0x7f63a4150da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.906+0000 7f63a2800640 1 -- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a4152900 msgr2=0x7f63a40af6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:56.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.906+0000 7f63a2800640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a4152900 0x7f63a40af6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:56.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.906+0000 7f63a2800640 1 -- 172.21.15.62:0/955690093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a4153640 con 0x7f63a40afc10 2024-03-23T11:32:56.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.906+0000 7f63a2800640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63a40afc10 0x7f63a4070090 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f639c000f80 tx=0x7f639c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:56.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.906+0000 7f639a000640 1 -- 172.21.15.62:0/955690093 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c004d60 con 0x7f63a40afc10 2024-03-23T11:32:56.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.907+0000 7f639a000640 1 -- 172.21.15.62:0/955690093 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c00bb80 con 0x7f63a40afc10 2024-03-23T11:32:56.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.907+0000 7f639a000640 1 -- 172.21.15.62:0/955690093 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c007070 con 0x7f63a40afc10 2024-03-23T11:32:56.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.907+0000 7f63a3200640 1 -- 172.21.15.62:0/955690093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a4153920 con 0x7f63a40afc10 2024-03-23T11:32:56.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.907+0000 7f63a3200640 1 -- 172.21.15.62:0/955690093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a40707b0 con 0x7f63a40afc10 2024-03-23T11:32:56.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.908+0000 7f63a3200640 1 -- 172.21.15.62:0/955690093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a4153b10 con 0x7f63a40afc10 2024-03-23T11:32:56.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.909+0000 7f639a000640 1 -- 172.21.15.62:0/955690093 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f639c0071d0 con 0x7f63a40afc10 2024-03-23T11:32:56.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.910+0000 7f639a000640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f638006a7a0 0x7f638006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:56.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.910+0000 7f639a000640 1 -- 172.21.15.62:0/955690093 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f639c088fd0 con 0x7f63a40afc10 2024-03-23T11:32:56.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.910+0000 7f63a1e00640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f638006a7a0 0x7f638006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:56.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.911+0000 7f63a1e00640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f638006a7a0 0x7f638006cc60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f63940099c0 tx=0x7f6394008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:56.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:56.914+0000 7f639a000640 1 -- 172.21.15.62:0/955690093 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f639c0545a0 con 0x7f63a40afc10 2024-03-23T11:32:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.120+0000 7f63a3200640 1 -- 172.21.15.62:0/955690093 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f63a41531f0 con 0x7f638006a7a0 2024-03-23T11:32:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.122+0000 7f639a000640 1 -- 172.21.15.62:0/955690093 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f63a41531f0 con 0x7f638006a7a0 2024-03-23T11:32:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 4 to deep-scrub 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 -- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f638006a7a0 msgr2=0x7f638006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f638006a7a0 0x7f638006cc60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f63940099c0 tx=0x7f6394008040 comp rx=0 tx=0).stop 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 -- 172.21.15.62:0/955690093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63a40afc10 msgr2=0x7f63a4070090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63a40afc10 0x7f63a4070090 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f639c000f80 tx=0x7f639c00b4a0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 -- 172.21.15.62:0/955690093 shutdown_connections 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f638006a7a0 0x7f638006cc60 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63a40afc10 0x7f63a4070090 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a4152900 0x7f63a40af6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 --2- 172.21.15.62:0/955690093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6388003f20 0x7f63a4150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 -- 172.21.15.62:0/955690093 >> 172.21.15.62:0/955690093 conn(0x7f63a4066050 msgr2=0x7f63a415c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 -- 172.21.15.62:0/955690093 shutdown_connections 2024-03-23T11:32:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.125+0000 7f63a3200640 1 -- 172.21.15.62:0/955690093 wait complete. 2024-03-23T11:32:57.132 INFO:tasks.ceph:Scrubbing osd.5 2024-03-23T11:32:57.132 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-23T11:32:57.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.371+0000 7f1b6b200640 1 Processor -- start 2024-03-23T11:32:57.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.371+0000 7f1b6b200640 1 -- start start 2024-03-23T11:32:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.372+0000 7f1b6b200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1b6c0b0730 0x7f1b6c15eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.373+0000 7f1b6b200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c15e700 con 0x7f1b6c0b0b10 2024-03-23T11:32:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.373+0000 7f1b6b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c15f0d0 con 0x7f1b6c06a700 2024-03-23T11:32:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.373+0000 7f1b6b200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6c15b350 con 0x7f1b6c0b0730 2024-03-23T11:32:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.373+0000 7f1b69e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1b6c0b0730 0x7f1b6c15eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.373+0000 7f1b69400640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1b6c0b0b10 0x7f1b6c15ac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54056/0 (socket says 172.21.15.62:54056) 2024-03-23T11:32:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.373+0000 7f1b69400640 1 -- 172.21.15.62:0/424836390 learned_addr learned my addr 172.21.15.62:0/424836390 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.373+0000 7f1b69e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1b6c0b0730 0x7f1b6c15eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52806/0 (socket says 172.21.15.62:52806) 2024-03-23T11:32:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.373+0000 7f1b69e00640 1 -- 172.21.15.62:0/424836390 >> v1:172.21.15.120:6789/0 conn(0x7f1b6c06a700 legacy=0x7f1b6c0caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.373+0000 7f1b69e00640 1 -- 172.21.15.62:0/424836390 >> v1:172.21.15.62:6789/0 conn(0x7f1b6c0b0b10 legacy=0x7f1b6c15ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.374+0000 7f1b69e00640 1 -- 172.21.15.62:0/424836390 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c15b610 con 0x7f1b6c0b0730 2024-03-23T11:32:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.374+0000 7f1b68a00640 1 -- 172.21.15.62:0/424836390 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998977650 0 0) 0x7f1b6c15e700 con 0x7f1b6c0b0b10 2024-03-23T11:32:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.375+0000 7f1b69e00640 1 --2- 172.21.15.62:0/424836390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0b0730 0x7f1b6c15eb90 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c009d90 tx=0x7f1b5c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f3449b4070d159 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.375+0000 7f1b68a00640 1 -- 172.21.15.62:0/424836390 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c02cbd0 con 0x7f1b6c0b0730 2024-03-23T11:32:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.375+0000 7f1b68a00640 1 -- 172.21.15.62:0/424836390 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5c02cd30 con 0x7f1b6c0b0730 2024-03-23T11:32:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.375+0000 7f1b68a00640 1 -- 172.21.15.62:0/424836390 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c02cea0 con 0x7f1b6c0b0730 2024-03-23T11:32:57.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.376+0000 7f1b6b200640 1 -- 172.21.15.62:0/424836390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0b0730 msgr2=0x7f1b6c15eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.376+0000 7f1b6b200640 1 --2- 172.21.15.62:0/424836390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0b0730 0x7f1b6c15eb90 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c009d90 tx=0x7f1b5c02be20 comp rx=0 tx=0).stop 2024-03-23T11:32:57.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.376+0000 7f1b6b200640 1 -- 172.21.15.62:0/424836390 shutdown_connections 2024-03-23T11:32:57.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.376+0000 7f1b6b200640 1 --2- 172.21.15.62:0/424836390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0b0730 0x7f1b6c15eb90 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.376+0000 7f1b6b200640 1 -- 172.21.15.62:0/424836390 >> 172.21.15.62:0/424836390 conn(0x7f1b6c0989d0 msgr2=0x7f1b6c0b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:57.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.376+0000 7f1b6b200640 1 -- 172.21.15.62:0/424836390 shutdown_connections 2024-03-23T11:32:57.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.376+0000 7f1b6b200640 1 -- 172.21.15.62:0/424836390 wait complete. 2024-03-23T11:32:57.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.378+0000 7f1b6b200640 1 Processor -- start 2024-03-23T11:32:57.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.378+0000 7f1b6b200640 1 -- start start 2024-03-23T11:32:57.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.379+0000 7f1b6b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b6c06a700 0x7f1b6c152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.379+0000 7f1b6b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0b0730 0x7f1b6c1f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.379+0000 7f1b6b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b6c0b0b10 0x7f1b6c0c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.379+0000 7f1b6b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c0c7930 con 0x7f1b6c0b0730 2024-03-23T11:32:57.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.379+0000 7f1b6b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c0b2b30 con 0x7f1b6c06a700 2024-03-23T11:32:57.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.379+0000 7f1b6b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c0b2200 con 0x7f1b6c0b0b10 2024-03-23T11:32:57.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.379+0000 7f1b69400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0b0730 0x7f1b6c1f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.379+0000 7f1b69400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0b0730 0x7f1b6c1f2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52820/0 (socket says 172.21.15.62:52820) 2024-03-23T11:32:57.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.379+0000 7f1b69400640 1 -- 172.21.15.62:0/3892298786 learned_addr learned my addr 172.21.15.62:0/3892298786 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:57.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b69400640 1 -- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b6c0b0b10 msgr2=0x7f1b6c0c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b6a800640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b6c0b0b10 0x7f1b6c0c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b69e00640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b6c06a700 0x7f1b6c152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b69400640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b6c0b0b10 0x7f1b6c0c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b69400640 1 -- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b6c06a700 msgr2=0x7f1b6c152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b69400640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b6c06a700 0x7f1b6c152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b69400640 1 -- 172.21.15.62:0/3892298786 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c009a70 con 0x7f1b6c0b0730 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b6a800640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b6c0b0b10 0x7f1b6c0c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b69e00640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b6c06a700 0x7f1b6c152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b69400640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0b0730 0x7f1b6c1f2450 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800c4b0 tx=0x7f1b5800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b62000640 1 -- 172.21.15.62:0/3892298786 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b580074e0 con 0x7f1b6c0b0730 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.380+0000 7f1b62000640 1 -- 172.21.15.62:0/3892298786 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b58007660 con 0x7f1b6c0b0730 2024-03-23T11:32:57.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.381+0000 7f1b6b200640 1 -- 172.21.15.62:0/3892298786 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c0b24e0 con 0x7f1b6c0b0730 2024-03-23T11:32:57.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.381+0000 7f1b62000640 1 -- 172.21.15.62:0/3892298786 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b58004a20 con 0x7f1b6c0b0730 2024-03-23T11:32:57.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.381+0000 7f1b6b200640 1 -- 172.21.15.62:0/3892298786 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c1f2990 con 0x7f1b6c0b0730 2024-03-23T11:32:57.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.382+0000 7f1b6b200640 1 -- 172.21.15.62:0/3892298786 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1b6c15e390 con 0x7f1b6c0b0730 2024-03-23T11:32:57.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.383+0000 7f1b62000640 1 -- 172.21.15.62:0/3892298786 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f1b58004b80 con 0x7f1b6c0b0730 2024-03-23T11:32:57.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.384+0000 7f1b62000640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1b4c06a7a0 0x7f1b4c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.384+0000 7f1b62000640 1 -- 172.21.15.62:0/3892298786 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f1b580880a0 con 0x7f1b6c0b0730 2024-03-23T11:32:57.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.385+0000 7f1b62000640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x7f1b4c070740 0x7f1b4c072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.385+0000 7f1b62000640 1 -- 172.21.15.62:0/3892298786 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b4c073210 con 0x7f1b4c070740 2024-03-23T11:32:57.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.385+0000 7f1b62000640 1 -- 172.21.15.62:0/3892298786 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b5808a680 con 0x7f1b6c0b0730 2024-03-23T11:32:57.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.385+0000 7f1b69e00640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1b4c06a7a0 0x7f1b4c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.385+0000 7f1b6a800640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x7f1b4c070740 0x7f1b4c072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.386+0000 7f1b69e00640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1b4c06a7a0 0x7f1b4c06cc60 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c036000 tx=0x7f1b5c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:57.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.386+0000 7f1b6a800640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x7f1b4c070740 0x7f1b4c072b60 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:57.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.388+0000 7f1b62000640 1 -- 172.21.15.62:0/3892298786 <== osd.5 v2:172.21.15.120:6824/37642 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1b4c073210 con 0x7f1b4c070740 2024-03-23T11:32:57.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.403+0000 7f1b6b200640 1 -- 172.21.15.62:0/3892298786 --> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1b6c0b2ca0 con 0x7f1b4c070740 2024-03-23T11:32:57.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.404+0000 7f1b62000640 1 -- 172.21.15.62:0/3892298786 <== osd.5 v2:172.21.15.120:6824/37642 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1b6c0b2ca0 con 0x7f1b4c070740 2024-03-23T11:32:57.393 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:57.394 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T11:32:57.394 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.405+0000 7f1b6b200640 1 -- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x7f1b4c070740 msgr2=0x7f1b4c072b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.405+0000 7f1b6b200640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:6824/37642,v1:172.21.15.120:6825/37642] conn(0x7f1b4c070740 0x7f1b4c072b60 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.405+0000 7f1b6b200640 1 -- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1b4c06a7a0 msgr2=0x7f1b4c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.405+0000 7f1b6b200640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f1b4c06a7a0 0x7f1b4c06cc60 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c036000 tx=0x7f1b5c030000 comp rx=0 tx=0).stop 2024-03-23T11:32:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.405+0000 7f1b6b200640 1 -- 172.21.15.62:0/3892298786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0b0730 msgr2=0x7f1b6c1f2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.405+0000 7f1b6b200640 1 --2- 172.21.15.62:0/3892298786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b6c0b0730 0x7f1b6c1f2450 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800c4b0 tx=0x7f1b5800c970 comp rx=0 tx=0).stop 2024-03-23T11:32:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.405+0000 7f1b6a800640 1 -- 172.21.15.62:0/3892298786 reap_dead start 2024-03-23T11:32:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.405+0000 7f1b6b200640 1 -- 172.21.15.62:0/3892298786 shutdown_connections 2024-03-23T11:32:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.405+0000 7f1b6b200640 1 -- 172.21.15.62:0/3892298786 >> 172.21.15.62:0/3892298786 conn(0x7f1b6c0989d0 msgr2=0x7f1b6c153c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.405+0000 7f1b6b200640 1 -- 172.21.15.62:0/3892298786 shutdown_connections 2024-03-23T11:32:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.405+0000 7f1b6b200640 1 -- 172.21.15.62:0/3892298786 wait complete. 2024-03-23T11:32:57.414 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-23T11:32:57.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.653+0000 7f4c42400640 1 Processor -- start 2024-03-23T11:32:57.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.653+0000 7f4c42400640 1 -- start start 2024-03-23T11:32:57.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.654+0000 7f4c42400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cd7c0 0x7f4c3c0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.655+0000 7f4c42400640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c0cad20 con 0x7f4c3c06c300 2024-03-23T11:32:57.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.655+0000 7f4c42400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c06c770 con 0x7f4c3c06dca0 2024-03-23T11:32:57.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.655+0000 7f4c42400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c06e130 con 0x7f4c3c0cd7c0 2024-03-23T11:32:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.655+0000 7f4c3be00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4c3c06c300 0x7f4c3c0cb490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54068/0 (socket says 172.21.15.62:54068) 2024-03-23T11:32:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.655+0000 7f4c3be00640 1 -- 172.21.15.62:0/3452904557 learned_addr learned my addr 172.21.15.62:0/3452904557 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.655+0000 7f4c3b400640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cd7c0 0x7f4c3c0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.655+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867338960 0 0) 0x7f4c3c0cad20 con 0x7f4c3c06c300 2024-03-23T11:32:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.655+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c20003520 con 0x7f4c3c06c300 2024-03-23T11:32:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1187255204 0 0) 0x7f4c20003520 con 0x7f4c3c06c300 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cd7c0 msgr2=0x7f4c3c0b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cd7c0 0x7f4c3c0b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 >> v1:172.21.15.120:6789/0 conn(0x7f4c3c06dca0 legacy=0x7f4c3c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c15a8a0 con 0x7f4c3c06c300 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f4c340019f0 con 0x7f4c3c06c300 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 >> v1:172.21.15.62:6789/0 conn(0x7f4c3c06c300 legacy=0x7f4c3c0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cd7c0 0x7f4c200039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c20003f20 0x7f4c200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c20004d00 0x7f4c20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c06e130 con 0x7f4c20004d00 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2000bb10 con 0x7f4c20003f20 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2000bcd0 con 0x7f4c3c0cd7c0 2024-03-23T11:32:57.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.656+0000 7f4c3be00640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c20004d00 0x7f4c20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.657+0000 7f4c3b400640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cd7c0 0x7f4c200039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.657+0000 7f4c33e00640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c20003f20 0x7f4c200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.657+0000 7f4c3be00640 1 -- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cd7c0 msgr2=0x7f4c200039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.657+0000 7f4c3be00640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cd7c0 0x7f4c200039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.657+0000 7f4c3be00640 1 -- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c20003f20 msgr2=0x7f4c200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.657+0000 7f4c3be00640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c20003f20 0x7f4c200047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.657+0000 7f4c3be00640 1 -- 172.21.15.62:0/3452904557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c20003520 con 0x7f4c20004d00 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.657+0000 7f4c33e00640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c20003f20 0x7f4c200047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.658+0000 7f4c3be00640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c20004d00 0x7f4c20005180 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400a410 tx=0x7f4c200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b18cbbe8589701 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.658+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c34031070 con 0x7f4c20004d00 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.658+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c340070d0 con 0x7f4c20004d00 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.658+0000 7f4c3aa00640 1 -- 172.21.15.62:0/3452904557 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c340074c0 con 0x7f4c20004d00 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.659+0000 7f4c42400640 1 -- 172.21.15.62:0/3452904557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c20004d00 msgr2=0x7f4c20005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.659+0000 7f4c42400640 1 --2- 172.21.15.62:0/3452904557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c20004d00 0x7f4c20005180 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400a410 tx=0x7f4c200057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.659+0000 7f4c3be00640 1 -- 172.21.15.62:0/3452904557 reap_dead start 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.659+0000 7f4c42400640 1 -- 172.21.15.62:0/3452904557 shutdown_connections 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.659+0000 7f4c42400640 1 -- 172.21.15.62:0/3452904557 >> 172.21.15.62:0/3452904557 conn(0x7f4c3c0b02e0 msgr2=0x7f4c3c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.659+0000 7f4c42400640 1 -- 172.21.15.62:0/3452904557 shutdown_connections 2024-03-23T11:32:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.659+0000 7f4c42400640 1 -- 172.21.15.62:0/3452904557 wait complete. 2024-03-23T11:32:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.661+0000 7f4c42400640 1 Processor -- start 2024-03-23T11:32:57.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.661+0000 7f4c42400640 1 -- start start 2024-03-23T11:32:57.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.662+0000 7f4c42400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c20004d00 0x7f4c3c140d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.662+0000 7f4c42400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c3c1edc70 0x7f4c3c14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.662+0000 7f4c42400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c153580 0x7f4c3c152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.662+0000 7f4c42400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c1f1240 con 0x7f4c3c1edc70 2024-03-23T11:32:57.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.662+0000 7f4c42400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c1ee130 con 0x7f4c20004d00 2024-03-23T11:32:57.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.662+0000 7f4c42400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c1ee410 con 0x7f4c3c153580 2024-03-23T11:32:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.662+0000 7f4c33e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c3c1edc70 0x7f4c3c14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.662+0000 7f4c33e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c3c1edc70 0x7f4c3c14ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52836/0 (socket says 172.21.15.62:52836) 2024-03-23T11:32:57.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.662+0000 7f4c33e00640 1 -- 172.21.15.62:0/2123914872 learned_addr learned my addr 172.21.15.62:0/2123914872 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.662+0000 7f4c3be00640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c153580 0x7f4c3c152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c3b400640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c20004d00 0x7f4c3c140d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c33e00640 1 -- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c153580 msgr2=0x7f4c3c152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c33e00640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c153580 0x7f4c3c152020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c33e00640 1 -- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c20004d00 msgr2=0x7f4c3c140d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c33e00640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c20004d00 0x7f4c3c140d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c33e00640 1 -- 172.21.15.62:0/2123914872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c1ee6f0 con 0x7f4c3c1edc70 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c33e00640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c3c1edc70 0x7f4c3c14ffe0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400d7a0 tx=0x7f4c2400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c33400640 1 -- 172.21.15.62:0/2123914872 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24017070 con 0x7f4c3c1edc70 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c33400640 1 -- 172.21.15.62:0/2123914872 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c24005070 con 0x7f4c3c1edc70 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c33400640 1 -- 172.21.15.62:0/2123914872 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24005420 con 0x7f4c3c1edc70 2024-03-23T11:32:57.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c42400640 1 -- 172.21.15.62:0/2123914872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3c1ee9d0 con 0x7f4c3c1edc70 2024-03-23T11:32:57.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.663+0000 7f4c42400640 1 -- 172.21.15.62:0/2123914872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3c150520 con 0x7f4c3c1edc70 2024-03-23T11:32:57.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.664+0000 7f4c42400640 1 -- 172.21.15.62:0/2123914872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3c1f13b0 con 0x7f4c3c1edc70 2024-03-23T11:32:57.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.665+0000 7f4c33400640 1 -- 172.21.15.62:0/2123914872 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f4c24007030 con 0x7f4c3c1edc70 2024-03-23T11:32:57.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.666+0000 7f4c33400640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4c1806a7d0 0x7f4c1806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:57.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.666+0000 7f4c33400640 1 -- 172.21.15.62:0/2123914872 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f4c24088370 con 0x7f4c3c1edc70 2024-03-23T11:32:57.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.667+0000 7f4c3b400640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4c1806a7d0 0x7f4c1806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:57.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.668+0000 7f4c3b400640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4c1806a7d0 0x7f4c1806cc90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009950 tx=0x7f4c2c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:57.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.671+0000 7f4c33400640 1 -- 172.21.15.62:0/2123914872 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c3c1f13b0 con 0x7f4c3c1edc70 2024-03-23T11:32:57.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.868+0000 7f4c42400640 1 -- 172.21.15.62:0/2123914872 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c3c0b1060 con 0x7f4c1806a7d0 2024-03-23T11:32:57.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.869+0000 7f4c33400640 1 -- 172.21.15.62:0/2123914872 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4c2c0095c0 con 0x7f4c1806a7d0 2024-03-23T11:32:57.859 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 5 to deep-scrub 2024-03-23T11:32:57.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.872+0000 7f4c42400640 1 -- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4c1806a7d0 msgr2=0x7f4c1806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.872+0000 7f4c42400640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4c1806a7d0 0x7f4c1806cc90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009950 tx=0x7f4c2c008040 comp rx=0 tx=0).stop 2024-03-23T11:32:57.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.872+0000 7f4c42400640 1 -- 172.21.15.62:0/2123914872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c3c1edc70 msgr2=0x7f4c3c14ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:57.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.872+0000 7f4c42400640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c3c1edc70 0x7f4c3c14ffe0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400d7a0 tx=0x7f4c2400dc60 comp rx=0 tx=0).stop 2024-03-23T11:32:57.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.872+0000 7f4c42400640 1 -- 172.21.15.62:0/2123914872 shutdown_connections 2024-03-23T11:32:57.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.872+0000 7f4c42400640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f4c1806a7d0 0x7f4c1806cc90 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.872+0000 7f4c42400640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c3c1edc70 0x7f4c3c14ffe0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.872+0000 7f4c42400640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c153580 0x7f4c3c152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.872+0000 7f4c42400640 1 --2- 172.21.15.62:0/2123914872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c20004d00 0x7f4c3c140d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:57.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.872+0000 7f4c42400640 1 -- 172.21.15.62:0/2123914872 >> 172.21.15.62:0/2123914872 conn(0x7f4c3c0b02e0 msgr2=0x7f4c3c0b2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:57.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.872+0000 7f4c42400640 1 -- 172.21.15.62:0/2123914872 shutdown_connections 2024-03-23T11:32:57.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:57.873+0000 7f4c42400640 1 -- 172.21.15.62:0/2123914872 wait complete. 2024-03-23T11:32:57.879 INFO:tasks.ceph:Scrubbing osd.6 2024-03-23T11:32:57.879 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-23T11:32:58.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.114+0000 7f9739200640 1 Processor -- start 2024-03-23T11:32:58.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.115+0000 7f9739200640 1 -- start start 2024-03-23T11:32:58.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.116+0000 7f9739200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f973406d950 0x7f973406c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.116+0000 7f9739200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973415cfd0 con 0x7f97340ce690 2024-03-23T11:32:58.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.116+0000 7f9739200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973415d220 con 0x7f973406dd30 2024-03-23T11:32:58.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.116+0000 7f9739200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973415d500 con 0x7f973406d950 2024-03-23T11:32:58.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.116+0000 7f9733e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f97340ce690 0x7f97340ceab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54074/0 (socket says 172.21.15.62:54074) 2024-03-23T11:32:58.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.116+0000 7f9733e00640 1 -- 172.21.15.62:0/4020446606 learned_addr learned my addr 172.21.15.62:0/4020446606 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9733400640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f973406d950 0x7f973406c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914732178 0 0) 0x7f973415cfd0 con 0x7f97340ce690 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9718005730 con 0x7f97340ce690 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3197494882 0 0) 0x7f9718005730 con 0x7f97340ce690 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f973406d950 msgr2=0x7f973406c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f973406d950 0x7f973406c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 >> v1:172.21.15.120:6789/0 conn(0x7f973406dd30 legacy=0x7f9734070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973415d7e0 con 0x7f97340ce690 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f972c0019f0 con 0x7f97340ce690 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 >> v1:172.21.15.62:6789/0 conn(0x7f97340ce690 legacy=0x7f97340ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9718005bb0 0x7f9718005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97180069d0 0x7f9718006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f973406d950 0x7f971800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973415d500 con 0x7f9718005bb0 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97180076f0 con 0x7f97180069d0 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.117+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971800e0d0 con 0x7f973406d950 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.118+0000 7f9733400640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9718005bb0 0x7f9718005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.118+0000 7f9733e00640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f973406d950 0x7f971800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.118+0000 7f972be00640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97180069d0 0x7f9718006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.118+0000 7f9733400640 1 -- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f973406d950 msgr2=0x7f971800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.118+0000 7f9733400640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f973406d950 0x7f971800db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.118+0000 7f9733400640 1 -- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97180069d0 msgr2=0x7f9718006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.118+0000 7f9733400640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97180069d0 0x7f9718006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.118+0000 7f9733400640 1 -- 172.21.15.62:0/4020446606 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9718005730 con 0x7f9718005bb0 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.118+0000 7f9733e00640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f973406d950 0x7f971800db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.118+0000 7f972be00640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97180069d0 0x7f9718006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.119+0000 7f9733400640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9718005bb0 0x7f9718005fd0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f972400ba30 tx=0x7f9718007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=176afff885ea8101 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.120+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9724032640 con 0x7f9718005bb0 2024-03-23T11:32:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.120+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97240327a0 con 0x7f9718005bb0 2024-03-23T11:32:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.120+0000 7f9732a00640 1 -- 172.21.15.62:0/4020446606 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9724032b30 con 0x7f9718005bb0 2024-03-23T11:32:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.120+0000 7f9739200640 1 -- 172.21.15.62:0/4020446606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9718005bb0 msgr2=0x7f9718005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.120+0000 7f9739200640 1 --2- 172.21.15.62:0/4020446606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9718005bb0 0x7f9718005fd0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f972400ba30 tx=0x7f9718007a00 comp rx=0 tx=0).stop 2024-03-23T11:32:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.120+0000 7f9733e00640 1 -- 172.21.15.62:0/4020446606 reap_dead start 2024-03-23T11:32:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.120+0000 7f9739200640 1 -- 172.21.15.62:0/4020446606 shutdown_connections 2024-03-23T11:32:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.120+0000 7f9739200640 1 -- 172.21.15.62:0/4020446606 >> 172.21.15.62:0/4020446606 conn(0x7f97340b0280 msgr2=0x7f97340b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.120+0000 7f9739200640 1 -- 172.21.15.62:0/4020446606 shutdown_connections 2024-03-23T11:32:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.120+0000 7f9739200640 1 -- 172.21.15.62:0/4020446606 wait complete. 2024-03-23T11:32:58.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.122+0000 7f9739200640 1 Processor -- start 2024-03-23T11:32:58.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.122+0000 7f9739200640 1 -- start start 2024-03-23T11:32:58.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.123+0000 7f9739200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9718005bb0 0x7f9734147610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.123+0000 7f9739200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9734069c60 0x7f973406a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.123+0000 7f9739200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9734155350 0x7f97341557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.123+0000 7f9739200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973406a600 con 0x7f9734069c60 2024-03-23T11:32:58.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.123+0000 7f9739200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9734156660 con 0x7f9718005bb0 2024-03-23T11:32:58.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.123+0000 7f9739200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97341560a0 con 0x7f9734155350 2024-03-23T11:32:58.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f972be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9734069c60 0x7f973406a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f972be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9734069c60 0x7f973406a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52856/0 (socket says 172.21.15.62:52856) 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f972be00640 1 -- 172.21.15.62:0/1529827211 learned_addr learned my addr 172.21.15.62:0/1529827211 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f9733e00640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9734155350 0x7f97341557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f9733400640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9718005bb0 0x7f9734147610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f972be00640 1 -- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9734155350 msgr2=0x7f97341557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f972be00640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9734155350 0x7f97341557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f972be00640 1 -- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9718005bb0 msgr2=0x7f9734147610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f972be00640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9718005bb0 0x7f9734147610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f972be00640 1 -- 172.21.15.62:0/1529827211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9734156380 con 0x7f9734069c60 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f9733400640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9718005bb0 0x7f9734147610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.124+0000 7f972be00640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9734069c60 0x7f973406a0c0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f971c00d7a0 tx=0x7f971c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.125+0000 7f972b400640 1 -- 172.21.15.62:0/1529827211 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c017070 con 0x7f9734069c60 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.125+0000 7f972b400640 1 -- 172.21.15.62:0/1529827211 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971c005070 con 0x7f9734069c60 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.125+0000 7f972b400640 1 -- 172.21.15.62:0/1529827211 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c005420 con 0x7f9734069c60 2024-03-23T11:32:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.125+0000 7f9739200640 1 -- 172.21.15.62:0/1529827211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97341575a0 con 0x7f9734069c60 2024-03-23T11:32:58.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.125+0000 7f9739200640 1 -- 172.21.15.62:0/1529827211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9734147b50 con 0x7f9734069c60 2024-03-23T11:32:58.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.126+0000 7f9739200640 1 -- 172.21.15.62:0/1529827211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f973415dda0 con 0x7f9734069c60 2024-03-23T11:32:58.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.127+0000 7f972b400640 1 -- 172.21.15.62:0/1529827211 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f971c007030 con 0x7f9734069c60 2024-03-23T11:32:58.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.128+0000 7f972b400640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f971006a7a0 0x7f971006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.128+0000 7f972b400640 1 -- 172.21.15.62:0/1529827211 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f971c088e30 con 0x7f9734069c60 2024-03-23T11:32:58.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.128+0000 7f972b400640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x7f9710070740 0x7f9710072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.128+0000 7f972b400640 1 -- 172.21.15.62:0/1529827211 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9710073210 con 0x7f9710070740 2024-03-23T11:32:58.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.128+0000 7f972b400640 1 -- 172.21.15.62:0/1529827211 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f971c0891a0 con 0x7f9734069c60 2024-03-23T11:32:58.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.129+0000 7f9733400640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f971006a7a0 0x7f971006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.129+0000 7f9733e00640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x7f9710070740 0x7f9710072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.130+0000 7f9733400640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f971006a7a0 0x7f971006cc60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f972400ba00 tx=0x7f972403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:58.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.130+0000 7f9733e00640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x7f9710070740 0x7f9710072b60 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:58.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.132+0000 7f972b400640 1 -- 172.21.15.62:0/1529827211 <== osd.6 v2:172.21.15.120:6800/37641 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f9710073210 con 0x7f9710070740 2024-03-23T11:32:58.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.146+0000 7f9739200640 1 -- 172.21.15.62:0/1529827211 --> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f97340713c0 con 0x7f9710070740 2024-03-23T11:32:58.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.147+0000 7f972b400640 1 -- 172.21.15.62:0/1529827211 <== osd.6 v2:172.21.15.120:6800/37641 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f972c00d040 con 0x7f9710070740 2024-03-23T11:32:58.136 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:58.136 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T11:32:58.136 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:58.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.147+0000 7f9739200640 1 -- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x7f9710070740 msgr2=0x7f9710072b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.147+0000 7f9739200640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:6800/37641,v1:172.21.15.120:6801/37641] conn(0x7f9710070740 0x7f9710072b60 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.148+0000 7f9739200640 1 -- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f971006a7a0 msgr2=0x7f971006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.148+0000 7f9739200640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f971006a7a0 0x7f971006cc60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f972400ba00 tx=0x7f972403d040 comp rx=0 tx=0).stop 2024-03-23T11:32:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.148+0000 7f9739200640 1 -- 172.21.15.62:0/1529827211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9734069c60 msgr2=0x7f973406a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.148+0000 7f9739200640 1 --2- 172.21.15.62:0/1529827211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9734069c60 0x7f973406a0c0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f971c00d7a0 tx=0x7f971c00dc60 comp rx=0 tx=0).stop 2024-03-23T11:32:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.148+0000 7f9733e00640 1 -- 172.21.15.62:0/1529827211 reap_dead start 2024-03-23T11:32:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.148+0000 7f9739200640 1 -- 172.21.15.62:0/1529827211 shutdown_connections 2024-03-23T11:32:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.148+0000 7f9739200640 1 -- 172.21.15.62:0/1529827211 >> 172.21.15.62:0/1529827211 conn(0x7f97340b0280 msgr2=0x7f97340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.148+0000 7f9739200640 1 -- 172.21.15.62:0/1529827211 shutdown_connections 2024-03-23T11:32:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.148+0000 7f9739200640 1 -- 172.21.15.62:0/1529827211 wait complete. 2024-03-23T11:32:58.156 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-23T11:32:58.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.394+0000 7f23a8c00640 1 Processor -- start 2024-03-23T11:32:58.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.394+0000 7f23a8c00640 1 -- start start 2024-03-23T11:32:58.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.395+0000 7f23a8c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a4143dd0 0x7f23a415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.396+0000 7f23a8c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a40cd140 con 0x7f23a40b2f30 2024-03-23T11:32:58.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.396+0000 7f23a8c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a40cd3e0 con 0x7f23a4070090 2024-03-23T11:32:58.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.396+0000 7f23a8c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40cd6c0 con 0x7f23a4143dd0 2024-03-23T11:32:58.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.396+0000 7f23a3e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f23a40b2f30 0x7f23a40ce480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54078/0 (socket says 172.21.15.62:54078) 2024-03-23T11:32:58.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.396+0000 7f23a3e00640 1 -- 172.21.15.62:0/2630468744 learned_addr learned my addr 172.21.15.62:0/2630468744 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:58.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.396+0000 7f23a3400640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a4143dd0 0x7f23a415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.396+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551102780 0 0) 0x7f23a40cd140 con 0x7f23a40b2f30 2024-03-23T11:32:58.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.396+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2388003520 con 0x7f23a40b2f30 2024-03-23T11:32:58.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1674271364 0 0) 0x7f2388003520 con 0x7f23a40b2f30 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a4143dd0 msgr2=0x7f23a415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a4143dd0 0x7f23a415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 >> v1:172.21.15.120:6789/0 conn(0x7f23a4070090 legacy=0x7f23a40b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40cd9a0 con 0x7f23a40b2f30 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f239c001670 con 0x7f23a40b2f30 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 >> v1:172.21.15.62:6789/0 conn(0x7f23a40b2f30 legacy=0x7f23a40ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 0x7f2388003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23880047c0 0x7f2388004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a4143dd0 0x7f2388005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40cd6c0 con 0x7f23880039a0 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238800ba30 con 0x7f23880047c0 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238800bbf0 con 0x7f23a4143dd0 2024-03-23T11:32:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.397+0000 7f23a3400640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 0x7f2388003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.398+0000 7f239aa00640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23880047c0 0x7f2388004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.398+0000 7f23a3e00640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a4143dd0 0x7f2388005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.398+0000 7f23a3400640 1 -- 172.21.15.62:0/2630468744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a4143dd0 msgr2=0x7f2388005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.398+0000 7f23a3400640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a4143dd0 0x7f2388005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.398+0000 7f23a3400640 1 -- 172.21.15.62:0/2630468744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23880047c0 msgr2=0x7f2388004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.398+0000 7f23a3400640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23880047c0 0x7f2388004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.398+0000 7f23a3400640 1 -- 172.21.15.62:0/2630468744 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2388003520 con 0x7f23880039a0 2024-03-23T11:32:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.399+0000 7f23a3400640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 0x7f2388003dc0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f2394009a90 tx=0x7f23880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4e03e0c48c6379 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.399+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239402c660 con 0x7f23880039a0 2024-03-23T11:32:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.399+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239402c7c0 con 0x7f23880039a0 2024-03-23T11:32:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.399+0000 7f23a2a00640 1 -- 172.21.15.62:0/2630468744 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239402cb50 con 0x7f23880039a0 2024-03-23T11:32:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.400+0000 7f23a8c00640 1 -- 172.21.15.62:0/2630468744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 msgr2=0x7f2388003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.400+0000 7f23a8c00640 1 --2- 172.21.15.62:0/2630468744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 0x7f2388003dc0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f2394009a90 tx=0x7f23880057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.400+0000 7f23a3e00640 1 -- 172.21.15.62:0/2630468744 reap_dead start 2024-03-23T11:32:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.400+0000 7f23a8c00640 1 -- 172.21.15.62:0/2630468744 shutdown_connections 2024-03-23T11:32:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.400+0000 7f23a8c00640 1 -- 172.21.15.62:0/2630468744 >> 172.21.15.62:0/2630468744 conn(0x7f23a4066050 msgr2=0x7f23a4069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.400+0000 7f23a8c00640 1 -- 172.21.15.62:0/2630468744 shutdown_connections 2024-03-23T11:32:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.400+0000 7f23a8c00640 1 -- 172.21.15.62:0/2630468744 wait complete. 2024-03-23T11:32:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.402+0000 7f23a8c00640 1 Processor -- start 2024-03-23T11:32:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.402+0000 7f23a8c00640 1 -- start start 2024-03-23T11:32:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.403+0000 7f23a8c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 0x7f23a413e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.403+0000 7f23a8c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23a413ea90 0x7f23a413f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.403+0000 7f23a8c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a40af470 0x7f23a413f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.403+0000 7f23a8c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40b0740 con 0x7f23880039a0 2024-03-23T11:32:58.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.403+0000 7f23a8c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a413e100 con 0x7f23a413ea90 2024-03-23T11:32:58.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.403+0000 7f23a8c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40acee0 con 0x7f23a40af470 2024-03-23T11:32:58.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.403+0000 7f23a3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 0x7f23a413e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.403+0000 7f23a3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 0x7f23a413e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52874/0 (socket says 172.21.15.62:52874) 2024-03-23T11:32:58.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.403+0000 7f23a3400640 1 -- 172.21.15.62:0/693366068 learned_addr learned my addr 172.21.15.62:0/693366068 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:58.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.404+0000 7f239aa00640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23a413ea90 0x7f23a413f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.404+0000 7f23a3400640 1 -- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a40af470 msgr2=0x7f23a413f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.404+0000 7f23a3400640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a40af470 0x7f23a413f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.404+0000 7f23a3400640 1 -- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23a413ea90 msgr2=0x7f23a413f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.404+0000 7f23a3400640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23a413ea90 0x7f23a413f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.404+0000 7f23a3400640 1 -- 172.21.15.62:0/693366068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40b0950 con 0x7f23880039a0 2024-03-23T11:32:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.404+0000 7f23a3400640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 0x7f23a413e550 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f2394009a90 tx=0x7f2394004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.404+0000 7f239be00640 1 -- 172.21.15.62:0/693366068 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394003810 con 0x7f23880039a0 2024-03-23T11:32:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.405+0000 7f239be00640 1 -- 172.21.15.62:0/693366068 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2394003970 con 0x7f23880039a0 2024-03-23T11:32:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.405+0000 7f23a8c00640 1 -- 172.21.15.62:0/693366068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a40b0c30 con 0x7f23880039a0 2024-03-23T11:32:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.405+0000 7f239be00640 1 -- 172.21.15.62:0/693366068 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394003d00 con 0x7f23880039a0 2024-03-23T11:32:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.405+0000 7f23a8c00640 1 -- 172.21.15.62:0/693366068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a4159e70 con 0x7f23880039a0 2024-03-23T11:32:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.406+0000 7f23a8c00640 1 -- 172.21.15.62:0/693366068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a40ccd20 con 0x7f23880039a0 2024-03-23T11:32:58.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.407+0000 7f239be00640 1 -- 172.21.15.62:0/693366068 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2394040450 con 0x7f23880039a0 2024-03-23T11:32:58.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.408+0000 7f239be00640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f238006a7a0 0x7f238006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.408+0000 7f239be00640 1 -- 172.21.15.62:0/693366068 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f23940aaa30 con 0x7f23880039a0 2024-03-23T11:32:58.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.409+0000 7f239aa00640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f238006a7a0 0x7f238006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.410+0000 7f239aa00640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f238006a7a0 0x7f238006cc60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f238c007fd0 tx=0x7f238c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:58.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.413+0000 7f239be00640 1 -- 172.21.15.62:0/693366068 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2394076090 con 0x7f23880039a0 2024-03-23T11:32:58.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.611+0000 7f23a8c00640 1 -- 172.21.15.62:0/693366068 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f23a40cdf60 con 0x7f238006a7a0 2024-03-23T11:32:58.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.612+0000 7f239be00640 1 -- 172.21.15.62:0/693366068 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f238c00d050 con 0x7f238006a7a0 2024-03-23T11:32:58.602 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 6 to deep-scrub 2024-03-23T11:32:58.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.615+0000 7f23a8c00640 1 -- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f238006a7a0 msgr2=0x7f238006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.615+0000 7f23a8c00640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f238006a7a0 0x7f238006cc60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f238c007fd0 tx=0x7f238c009040 comp rx=0 tx=0).stop 2024-03-23T11:32:58.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.615+0000 7f23a8c00640 1 -- 172.21.15.62:0/693366068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 msgr2=0x7f23a413e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.615+0000 7f23a8c00640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 0x7f23a413e550 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f2394009a90 tx=0x7f2394004730 comp rx=0 tx=0).stop 2024-03-23T11:32:58.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.616+0000 7f23a8c00640 1 -- 172.21.15.62:0/693366068 shutdown_connections 2024-03-23T11:32:58.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.616+0000 7f23a8c00640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f238006a7a0 0x7f238006cc60 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.616+0000 7f23a8c00640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23a40af470 0x7f23a413f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.616+0000 7f23a8c00640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23a413ea90 0x7f23a413f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.616+0000 7f23a8c00640 1 --2- 172.21.15.62:0/693366068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23880039a0 0x7f23a413e550 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.616+0000 7f23a8c00640 1 -- 172.21.15.62:0/693366068 >> 172.21.15.62:0/693366068 conn(0x7f23a4066050 msgr2=0x7f23a4142430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:58.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.616+0000 7f23a8c00640 1 -- 172.21.15.62:0/693366068 shutdown_connections 2024-03-23T11:32:58.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.616+0000 7f23a8c00640 1 -- 172.21.15.62:0/693366068 wait complete. 2024-03-23T11:32:58.624 INFO:tasks.ceph:Scrubbing osd.7 2024-03-23T11:32:58.625 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-23T11:32:58.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.858+0000 7f6256200640 1 Processor -- start 2024-03-23T11:32:58.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.859+0000 7f6256200640 1 -- start start 2024-03-23T11:32:58.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.860+0000 7f6256200640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f62500cf000 0x7f62500cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.860+0000 7f6256200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f62500d0c50 0x7f62500d10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.860+0000 7f6256200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62500d1620 con 0x7f625015c820 2024-03-23T11:32:58.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.860+0000 7f6256200640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62500d0200 con 0x7f62500cf000 2024-03-23T11:32:58.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.860+0000 7f6256200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62500d07c0 con 0x7f62500d0c50 2024-03-23T11:32:58.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.860+0000 7f624fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f62500cf000 0x7f62500cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.860+0000 7f624fe00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f62500cf000 0x7f62500cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52880/0 (socket says 172.21.15.62:52880) 2024-03-23T11:32:58.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.860+0000 7f624fe00640 1 -- 172.21.15.62:0/554688013 learned_addr learned my addr 172.21.15.62:0/554688013 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:58.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.861+0000 7f6255800640 1 --2- 172.21.15.62:0/554688013 >> v2:172.21.15.120:3300/0 conn(0x7f62500d0c50 0x7f62500d10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.861+0000 7f624fe00640 1 -- 172.21.15.62:0/554688013 >> v2:172.21.15.120:3300/0 conn(0x7f62500d0c50 msgr2=0x7f62500d10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:58.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.861+0000 7f624fe00640 1 --2- 172.21.15.62:0/554688013 >> v2:172.21.15.120:3300/0 conn(0x7f62500d0c50 0x7f62500d10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.861+0000 7f624fe00640 1 -- 172.21.15.62:0/554688013 >> v1:172.21.15.62:6789/0 conn(0x7f625015c820 legacy=0x7f62500b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.861+0000 7f624fe00640 1 -- 172.21.15.62:0/554688013 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62500cf980 con 0x7f62500cf000 2024-03-23T11:32:58.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.861+0000 7f624f400640 1 -- 172.21.15.62:0/554688013 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380410757 0 0) 0x7f62500d1620 con 0x7f625015c820 2024-03-23T11:32:58.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.862+0000 7f624fe00640 1 --2- 172.21.15.62:0/554688013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62500cf000 0x7f62500cf440 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f623c009850 tx=0x7f623c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becbd2b8f3c26560 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:58.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.862+0000 7f624f400640 1 -- 172.21.15.62:0/554688013 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c004230 con 0x7f62500cf000 2024-03-23T11:32:58.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.862+0000 7f624f400640 1 -- 172.21.15.62:0/554688013 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623c004390 con 0x7f62500cf000 2024-03-23T11:32:58.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.862+0000 7f624f400640 1 -- 172.21.15.62:0/554688013 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c003810 con 0x7f62500cf000 2024-03-23T11:32:58.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.863+0000 7f6256200640 1 -- 172.21.15.62:0/554688013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62500cf000 msgr2=0x7f62500cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.863+0000 7f6256200640 1 --2- 172.21.15.62:0/554688013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62500cf000 0x7f62500cf440 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f623c009850 tx=0x7f623c02b8f0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.863+0000 7f6256200640 1 -- 172.21.15.62:0/554688013 shutdown_connections 2024-03-23T11:32:58.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.863+0000 7f6256200640 1 --2- 172.21.15.62:0/554688013 >> v2:172.21.15.120:3300/0 conn(0x7f62500d0c50 0x7f62500d10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.863+0000 7f6256200640 1 --2- 172.21.15.62:0/554688013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62500cf000 0x7f62500cf440 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.863+0000 7f6256200640 1 -- 172.21.15.62:0/554688013 >> 172.21.15.62:0/554688013 conn(0x7f625009b9d0 msgr2=0x7f6250089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:58.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.863+0000 7f6256200640 1 -- 172.21.15.62:0/554688013 shutdown_connections 2024-03-23T11:32:58.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.863+0000 7f6256200640 1 -- 172.21.15.62:0/554688013 wait complete. 2024-03-23T11:32:58.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.865+0000 7f6256200640 1 Processor -- start 2024-03-23T11:32:58.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.865+0000 7f6256200640 1 -- start start 2024-03-23T11:32:58.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.866+0000 7f6256200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62500cf000 0x7f62501ef680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.866+0000 7f6256200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62500d0c50 0x7f62500b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.866+0000 7f6256200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f625015c820 0x7f62501f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.866+0000 7f6256200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6250152300 con 0x7f62500d0c50 2024-03-23T11:32:58.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.866+0000 7f6256200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62500c6bc0 con 0x7f625015c820 2024-03-23T11:32:58.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.866+0000 7f6256200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62500b8120 con 0x7f62500cf000 2024-03-23T11:32:58.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.866+0000 7f624fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62500d0c50 0x7f62500b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.866+0000 7f624fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62500d0c50 0x7f62500b52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52890/0 (socket says 172.21.15.62:52890) 2024-03-23T11:32:58.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.866+0000 7f624fe00640 1 -- 172.21.15.62:0/1055167225 learned_addr learned my addr 172.21.15.62:0/1055167225 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:58.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f6255800640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f625015c820 0x7f62501f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f6254e00640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62500cf000 0x7f62501ef680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f624fe00640 1 -- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62500cf000 msgr2=0x7f62501ef680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f624fe00640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62500cf000 0x7f62501ef680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f624fe00640 1 -- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f625015c820 msgr2=0x7f62501f5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f624fe00640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f625015c820 0x7f62501f5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f624fe00640 1 -- 172.21.15.62:0/1055167225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c009460 con 0x7f62500d0c50 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f6254e00640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62500cf000 0x7f62501ef680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f6255800640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f625015c820 0x7f62501f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f624fe00640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62500d0c50 0x7f62500b52b0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f623c000c00 tx=0x7f623c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f624cc00640 1 -- 172.21.15.62:0/1055167225 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c003c40 con 0x7f62500d0c50 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f624cc00640 1 -- 172.21.15.62:0/1055167225 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623c03f070 con 0x7f62500d0c50 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f624cc00640 1 -- 172.21.15.62:0/1055167225 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c004500 con 0x7f62500d0c50 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.867+0000 7f6256200640 1 -- 172.21.15.62:0/1055167225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62500c68e0 con 0x7f62500d0c50 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.868+0000 7f6256200640 1 -- 172.21.15.62:0/1055167225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62500b7db0 con 0x7f62500d0c50 2024-03-23T11:32:58.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.868+0000 7f6256200640 1 -- 172.21.15.62:0/1055167225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f62500d0450 con 0x7f62500d0c50 2024-03-23T11:32:58.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.870+0000 7f624cc00640 1 -- 172.21.15.62:0/1055167225 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f623c041030 con 0x7f62500d0c50 2024-03-23T11:32:58.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.871+0000 7f624cc00640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f623006a7a0 0x7f623006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.871+0000 7f624cc00640 1 -- 172.21.15.62:0/1055167225 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f623c0ab510 con 0x7f62500d0c50 2024-03-23T11:32:58.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.871+0000 7f624cc00640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x7f6230070740 0x7f6230072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:58.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.871+0000 7f624cc00640 1 -- 172.21.15.62:0/1055167225 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6230073210 con 0x7f6230070740 2024-03-23T11:32:58.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.871+0000 7f624cc00640 1 -- 172.21.15.62:0/1055167225 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=125) v2 ==== 24+0+0 (secure 0 0 0) 0x7f623c0ab880 con 0x7f62500d0c50 2024-03-23T11:32:58.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.871+0000 7f6254e00640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f623006a7a0 0x7f623006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.872+0000 7f6255800640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x7f6230070740 0x7f6230072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:58.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.873+0000 7f6255800640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x7f6230070740 0x7f6230072b60 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:58.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.873+0000 7f6254e00640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f623006a7a0 0x7f623006cc60 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f6240001740 tx=0x7f6240009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:58.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.875+0000 7f624cc00640 1 -- 172.21.15.62:0/1055167225 <== osd.7 v2:172.21.15.120:6816/37655 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6230073210 con 0x7f6230070740 2024-03-23T11:32:58.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.889+0000 7f6256200640 1 -- 172.21.15.62:0/1055167225 --> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f62500ce350 con 0x7f6230070740 2024-03-23T11:32:58.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.890+0000 7f624cc00640 1 -- 172.21.15.62:0/1055167225 <== osd.7 v2:172.21.15.120:6816/37655 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6248003990 con 0x7f6230070740 2024-03-23T11:32:58.880 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-23T11:32:58.880 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T11:32:58.880 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-23T11:32:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.891+0000 7f6256200640 1 -- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x7f6230070740 msgr2=0x7f6230072b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.891+0000 7f6256200640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:6816/37655,v1:172.21.15.120:6817/37655] conn(0x7f6230070740 0x7f6230072b60 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.891+0000 7f6256200640 1 -- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f623006a7a0 msgr2=0x7f623006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.891+0000 7f6256200640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f623006a7a0 0x7f623006cc60 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f6240001740 tx=0x7f6240009040 comp rx=0 tx=0).stop 2024-03-23T11:32:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.891+0000 7f6256200640 1 -- 172.21.15.62:0/1055167225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62500d0c50 msgr2=0x7f62500b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.891+0000 7f6256200640 1 --2- 172.21.15.62:0/1055167225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62500d0c50 0x7f62500b52b0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f623c000c00 tx=0x7f623c0046c0 comp rx=0 tx=0).stop 2024-03-23T11:32:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.891+0000 7f6255800640 1 -- 172.21.15.62:0/1055167225 reap_dead start 2024-03-23T11:32:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.891+0000 7f6256200640 1 -- 172.21.15.62:0/1055167225 shutdown_connections 2024-03-23T11:32:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.891+0000 7f6256200640 1 -- 172.21.15.62:0/1055167225 >> 172.21.15.62:0/1055167225 conn(0x7f625009b9d0 msgr2=0x7f625006bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:58.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.891+0000 7f6256200640 1 -- 172.21.15.62:0/1055167225 shutdown_connections 2024-03-23T11:32:58.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:58.891+0000 7f6256200640 1 -- 172.21.15.62:0/1055167225 wait complete. 2024-03-23T11:32:58.901 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-23T11:32:59.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.134+0000 7f64f7e00640 1 Processor -- start 2024-03-23T11:32:59.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.135+0000 7f64f7e00640 1 -- start start 2024-03-23T11:32:59.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.136+0000 7f64f7e00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f64f806cc60 0x7f64f80cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.137+0000 7f64f7e00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f80ced90 con 0x7f64f8159490 2024-03-23T11:32:59.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.137+0000 7f64f7e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f806e3e0 con 0x7f64f80ce420 2024-03-23T11:32:59.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.137+0000 7f64f7e00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f80ce850 con 0x7f64f806cc60 2024-03-23T11:32:59.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.137+0000 7f64f6a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f64f8159490 0x7f64f815da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54094/0 (socket says 172.21.15.62:54094) 2024-03-23T11:32:59.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.137+0000 7f64f6a00640 1 -- 172.21.15.62:0/4232187520 learned_addr learned my addr 172.21.15.62:0/4232187520 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:59.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.137+0000 7f64efe00640 1 --2- 172.21.15.62:0/4232187520 >> v2:172.21.15.62:3300/0 conn(0x7f64f806cc60 0x7f64f80cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.137+0000 7f64f6000640 1 -- 172.21.15.62:0/4232187520 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743943700 0 0) 0x7f64f80ced90 con 0x7f64f8159490 2024-03-23T11:32:59.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.137+0000 7f64f6000640 1 -- 172.21.15.62:0/4232187520 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64dc003520 con 0x7f64f8159490 2024-03-23T11:32:59.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.137+0000 7f64efe00640 1 -- 172.21.15.62:0/4232187520 >> v1:172.21.15.120:6789/0 conn(0x7f64f80ce420 legacy=0x7f64f80b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:59.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.137+0000 7f64efe00640 1 -- 172.21.15.62:0/4232187520 >> v1:172.21.15.62:6789/0 conn(0x7f64f8159490 legacy=0x7f64f815da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.137+0000 7f64efe00640 1 -- 172.21.15.62:0/4232187520 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f806c1c0 con 0x7f64f806cc60 2024-03-23T11:32:59.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.139+0000 7f64efe00640 1 --2- 172.21.15.62:0/4232187520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64f806cc60 0x7f64f80cdee0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f64e00097a0 tx=0x7f64e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68419ae4353b2517 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:59.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.139+0000 7f64f6000640 1 -- 172.21.15.62:0/4232187520 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e002e070 con 0x7f64f806cc60 2024-03-23T11:32:59.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.139+0000 7f64f6000640 1 -- 172.21.15.62:0/4232187520 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e00315a0 con 0x7f64f806cc60 2024-03-23T11:32:59.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.139+0000 7f64f6000640 1 -- 172.21.15.62:0/4232187520 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0031930 con 0x7f64f806cc60 2024-03-23T11:32:59.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.139+0000 7f64f7e00640 1 -- 172.21.15.62:0/4232187520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64f806cc60 msgr2=0x7f64f80cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.139+0000 7f64f7e00640 1 --2- 172.21.15.62:0/4232187520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64f806cc60 0x7f64f80cdee0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f64e00097a0 tx=0x7f64e002b840 comp rx=0 tx=0).stop 2024-03-23T11:32:59.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.140+0000 7f64f7e00640 1 -- 172.21.15.62:0/4232187520 shutdown_connections 2024-03-23T11:32:59.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.140+0000 7f64f7e00640 1 --2- 172.21.15.62:0/4232187520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64f806cc60 0x7f64f80cdee0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.140+0000 7f64f7e00640 1 -- 172.21.15.62:0/4232187520 >> 172.21.15.62:0/4232187520 conn(0x7f64f8066050 msgr2=0x7f64f8069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:59.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.140+0000 7f64f7e00640 1 -- 172.21.15.62:0/4232187520 shutdown_connections 2024-03-23T11:32:59.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.140+0000 7f64f7e00640 1 -- 172.21.15.62:0/4232187520 wait complete. 2024-03-23T11:32:59.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.142+0000 7f64f7e00640 1 Processor -- start 2024-03-23T11:32:59.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.142+0000 7f64f7e00640 1 -- start start 2024-03-23T11:32:59.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.143+0000 7f64f7e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64f806cc60 0x7f64f8153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.143+0000 7f64f7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64f80ce420 0x7f64f80ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.143+0000 7f64f7e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64f8159490 0x7f64f80ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.143+0000 7f64f7e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f81464b0 con 0x7f64f80ce420 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.143+0000 7f64f7e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f8146790 con 0x7f64f8159490 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64f7e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f8146a70 con 0x7f64f806cc60 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64efe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64f80ce420 0x7f64f80ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64efe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64f80ce420 0x7f64f80ae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52910/0 (socket says 172.21.15.62:52910) 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64efe00640 1 -- 172.21.15.62:0/3472432405 learned_addr learned my addr 172.21.15.62:0/3472432405 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64f6a00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64f806cc60 0x7f64f8153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64f7400640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64f8159490 0x7f64f80ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64efe00640 1 -- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64f806cc60 msgr2=0x7f64f8153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64efe00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64f806cc60 0x7f64f8153b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64efe00640 1 -- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64f8159490 msgr2=0x7f64f80ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64efe00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64f8159490 0x7f64f80ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64efe00640 1 -- 172.21.15.62:0/3472432405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e0009400 con 0x7f64f80ce420 2024-03-23T11:32:59.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.144+0000 7f64f7400640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64f8159490 0x7f64f80ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:59.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.145+0000 7f64efe00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64f80ce420 0x7f64f80ae2a0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f64e0000c00 tx=0x7f64e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:59.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.145+0000 7f64eea00640 1 -- 172.21.15.62:0/3472432405 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0032840 con 0x7f64f80ce420 2024-03-23T11:32:59.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.145+0000 7f64eea00640 1 -- 172.21.15.62:0/3472432405 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e00329a0 con 0x7f64f80ce420 2024-03-23T11:32:59.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.145+0000 7f64eea00640 1 -- 172.21.15.62:0/3472432405 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0032d50 con 0x7f64f80ce420 2024-03-23T11:32:59.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.145+0000 7f64f7e00640 1 -- 172.21.15.62:0/3472432405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f8148220 con 0x7f64f80ce420 2024-03-23T11:32:59.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.145+0000 7f64f7e00640 1 -- 172.21.15.62:0/3472432405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f8154060 con 0x7f64f80ce420 2024-03-23T11:32:59.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.147+0000 7f64f7e00640 1 -- 172.21.15.62:0/3472432405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f8148740 con 0x7f64f80ce420 2024-03-23T11:32:59.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.147+0000 7f64eea00640 1 -- 172.21.15.62:0/3472432405 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f64e004d3f0 con 0x7f64f80ce420 2024-03-23T11:32:59.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.148+0000 7f64eea00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f64d406a7a0 0x7f64d406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.148+0000 7f64eea00640 1 -- 172.21.15.62:0/3472432405 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f64e00aa690 con 0x7f64f80ce420 2024-03-23T11:32:59.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.149+0000 7f64f6a00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f64d406a7a0 0x7f64d406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.150+0000 7f64f6a00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f64d406a7a0 0x7f64d406cc60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f64e80011e0 tx=0x7f64e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:59.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.153+0000 7f64eea00640 1 -- 172.21.15.62:0/3472432405 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64f8148740 con 0x7f64f80ce420 2024-03-23T11:32:59.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.349+0000 7f64f7e00640 1 -- 172.21.15.62:0/3472432405 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f64f8146900 con 0x7f64d406a7a0 2024-03-23T11:32:59.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.350+0000 7f64eea00640 1 -- 172.21.15.62:0/3472432405 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f64f8146900 con 0x7f64d406a7a0 2024-03-23T11:32:59.339 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 7 to deep-scrub 2024-03-23T11:32:59.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.353+0000 7f64f7e00640 1 -- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f64d406a7a0 msgr2=0x7f64d406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.353+0000 7f64f7e00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f64d406a7a0 0x7f64d406cc60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f64e80011e0 tx=0x7f64e8009040 comp rx=0 tx=0).stop 2024-03-23T11:32:59.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.353+0000 7f64f7e00640 1 -- 172.21.15.62:0/3472432405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64f80ce420 msgr2=0x7f64f80ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.353+0000 7f64f7e00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64f80ce420 0x7f64f80ae2a0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f64e0000c00 tx=0x7f64e00046c0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.353+0000 7f64f7e00640 1 -- 172.21.15.62:0/3472432405 shutdown_connections 2024-03-23T11:32:59.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.353+0000 7f64f7e00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f64d406a7a0 0x7f64d406cc60 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.353+0000 7f64f7e00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64f80ce420 0x7f64f80ae2a0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.353+0000 7f64f7e00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64f8159490 0x7f64f80ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.353+0000 7f64f7e00640 1 --2- 172.21.15.62:0/3472432405 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f64f806cc60 0x7f64f8153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.353+0000 7f64f7e00640 1 -- 172.21.15.62:0/3472432405 >> 172.21.15.62:0/3472432405 conn(0x7f64f8066050 msgr2=0x7f64f80b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:59.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.353+0000 7f64f7e00640 1 -- 172.21.15.62:0/3472432405 shutdown_connections 2024-03-23T11:32:59.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.354+0000 7f64f7e00640 1 -- 172.21.15.62:0/3472432405 wait complete. 2024-03-23T11:32:59.362 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:32:59.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.589+0000 7f72bb200640 1 Processor -- start 2024-03-23T11:32:59.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.589+0000 7f72bb200640 1 -- start start 2024-03-23T11:32:59.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.591+0000 7f72bb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc159490 0x7f72bc06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.591+0000 7f72bb200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72bc0ced90 con 0x7f72bc0ce5f0 2024-03-23T11:32:59.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.591+0000 7f72bb200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72bc06e3e0 con 0x7f72bc15da90 2024-03-23T11:32:59.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.591+0000 7f72bb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc06c1c0 con 0x7f72bc159490 2024-03-23T11:32:59.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.591+0000 7f72ba800640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f72bc0ce5f0 0x7f72bc0b26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54104/0 (socket says 172.21.15.62:54104) 2024-03-23T11:32:59.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.591+0000 7f72ba800640 1 -- 172.21.15.62:0/1368727742 learned_addr learned my addr 172.21.15.62:0/1368727742 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:32:59.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b9e00640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc159490 0x7f72bc06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767591271 0 0) 0x7f72bc0ced90 con 0x7f72bc0ce5f0 2024-03-23T11:32:59.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a0003520 con 0x7f72bc0ce5f0 2024-03-23T11:32:59.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539172774 0 0) 0x7f72bc06e3e0 con 0x7f72bc15da90 2024-03-23T11:32:59.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72bc0ced90 con 0x7f72bc15da90 2024-03-23T11:32:59.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278563011 0 0) 0x7f72a0003520 con 0x7f72bc0ce5f0 2024-03-23T11:32:59.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc159490 msgr2=0x7f72bc06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:59.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc159490 0x7f72bc06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 >> v1:172.21.15.120:6789/0 conn(0x7f72bc15da90 legacy=0x7f72bc0cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc06e100 con 0x7f72bc0ce5f0 2024-03-23T11:32:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7f72b40019f0 con 0x7f72bc0ce5f0 2024-03-23T11:32:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 >> v1:172.21.15.62:6789/0 conn(0x7f72bc0ce5f0 legacy=0x7f72bc0b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.592+0000 7f72b8a00640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72a0003b30 0x7f72a0003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b8a00640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a00047c0 0x7f72a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b8a00640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc159490 0x7f72a00051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc06c1c0 con 0x7f72a0003b30 2024-03-23T11:32:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a000bb10 con 0x7f72a00047c0 2024-03-23T11:32:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a000deb0 con 0x7f72bc159490 2024-03-23T11:32:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b9e00640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72a0003b30 0x7f72a0003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b9e00640 1 -- 172.21.15.62:0/1368727742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc159490 msgr2=0x7f72a00051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:59.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b9e00640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc159490 0x7f72a00051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b9e00640 1 -- 172.21.15.62:0/1368727742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a00047c0 msgr2=0x7f72a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:32:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b9400640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a00047c0 0x7f72a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b9e00640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a00047c0 0x7f72a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b9e00640 1 -- 172.21.15.62:0/1368727742 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a0003520 con 0x7f72a0003b30 2024-03-23T11:32:59.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.593+0000 7f72b9400640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a00047c0 0x7f72a0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:32:59.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.594+0000 7f72b9e00640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72a0003b30 0x7f72a0003f30 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f72ac009af0 tx=0x7f72a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6307f477b29293d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:59.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.595+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac02c710 con 0x7f72a0003b30 2024-03-23T11:32:59.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.595+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac02c870 con 0x7f72a0003b30 2024-03-23T11:32:59.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.595+0000 7f72b8a00640 1 -- 172.21.15.62:0/1368727742 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac02cc00 con 0x7f72a0003b30 2024-03-23T11:32:59.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.595+0000 7f72bb200640 1 -- 172.21.15.62:0/1368727742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72a0003b30 msgr2=0x7f72a0003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.595+0000 7f72bb200640 1 --2- 172.21.15.62:0/1368727742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72a0003b30 0x7f72a0003f30 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f72ac009af0 tx=0x7f72a00057f0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.595+0000 7f72ba800640 1 -- 172.21.15.62:0/1368727742 reap_dead start 2024-03-23T11:32:59.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.595+0000 7f72bb200640 1 -- 172.21.15.62:0/1368727742 shutdown_connections 2024-03-23T11:32:59.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.595+0000 7f72bb200640 1 -- 172.21.15.62:0/1368727742 >> 172.21.15.62:0/1368727742 conn(0x7f72bc066050 msgr2=0x7f72bc069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:59.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.596+0000 7f72bb200640 1 -- 172.21.15.62:0/1368727742 shutdown_connections 2024-03-23T11:32:59.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.596+0000 7f72bb200640 1 -- 172.21.15.62:0/1368727742 wait complete. 2024-03-23T11:32:59.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.597+0000 7f72bb200640 1 Processor -- start 2024-03-23T11:32:59.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.597+0000 7f72bb200640 1 -- start start 2024-03-23T11:32:59.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.598+0000 7f72bb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a0003b30 0x7f72bc0ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.598+0000 7f72bb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72bc15c220 0x7f72bc154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.598+0000 7f72bb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc15c6a0 0x7f72bc1549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.598+0000 7f72bb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc1575b0 con 0x7f72bc15c220 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.598+0000 7f72bb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc1569a0 con 0x7f72a0003b30 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.598+0000 7f72bb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc156c80 con 0x7f72bc15c6a0 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.599+0000 7f72b9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72bc15c220 0x7f72bc154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.599+0000 7f72b9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72bc15c220 0x7f72bc154490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52940/0 (socket says 172.21.15.62:52940) 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.599+0000 7f72b9400640 1 -- 172.21.15.62:0/481840248 learned_addr learned my addr 172.21.15.62:0/481840248 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.599+0000 7f72b9e00640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a0003b30 0x7f72bc0ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.599+0000 7f72ba800640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc15c6a0 0x7f72bc1549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.599+0000 7f72b9400640 1 -- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc15c6a0 msgr2=0x7f72bc1549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.599+0000 7f72b9400640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc15c6a0 0x7f72bc1549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.599+0000 7f72b9400640 1 -- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a0003b30 msgr2=0x7f72bc0ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.599+0000 7f72b9400640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a0003b30 0x7f72bc0ace90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.599+0000 7f72b9400640 1 -- 172.21.15.62:0/481840248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc156f60 con 0x7f72bc15c220 2024-03-23T11:32:59.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.600+0000 7f72b9400640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72bc15c220 0x7f72bc154490 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f72a800d7a0 tx=0x7f72a800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:59.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.600+0000 7f72b2000640 1 -- 172.21.15.62:0/481840248 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a800bac0 con 0x7f72bc15c220 2024-03-23T11:32:59.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.600+0000 7f72b2000640 1 -- 172.21.15.62:0/481840248 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a800bc20 con 0x7f72bc15c220 2024-03-23T11:32:59.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.600+0000 7f72b2000640 1 -- 172.21.15.62:0/481840248 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a800de50 con 0x7f72bc15c220 2024-03-23T11:32:59.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.600+0000 7f72bb200640 1 -- 172.21.15.62:0/481840248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72bc157240 con 0x7f72bc15c220 2024-03-23T11:32:59.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.600+0000 7f72bb200640 1 -- 172.21.15.62:0/481840248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72bc157a40 con 0x7f72bc15c220 2024-03-23T11:32:59.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.601+0000 7f72bb200640 1 -- 172.21.15.62:0/481840248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72bc06d320 con 0x7f72bc15c220 2024-03-23T11:32:59.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.602+0000 7f72b2000640 1 -- 172.21.15.62:0/481840248 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f72a80067d0 con 0x7f72bc15c220 2024-03-23T11:32:59.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.603+0000 7f72b2000640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729c06a7a0 0x7f729c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:32:59.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.603+0000 7f72b2000640 1 -- 172.21.15.62:0/481840248 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f72a8088b30 con 0x7f72bc15c220 2024-03-23T11:32:59.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.604+0000 7f72b9e00640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729c06a7a0 0x7f729c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:32:59.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.605+0000 7f72b9e00640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729c06a7a0 0x7f729c06cc60 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f72ac036000 tx=0x7f72ac030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:32:59.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.607+0000 7f72b2000640 1 -- 172.21.15.62:0/481840248 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72a8054230 con 0x7f72bc15c220 2024-03-23T11:32:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.805+0000 7f72bb200640 1 -- 172.21.15.62:0/481840248 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72bc159870 con 0x7f729c06a7a0 2024-03-23T11:32:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.811+0000 7f72b2000640 1 -- 172.21.15.62:0/481840248 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47909 (secure 0 0 0) 0x7f72ac004040 con 0x7f729c06a7a0 2024-03-23T11:32:59.801 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:32:59.801 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:32:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 -- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729c06a7a0 msgr2=0x7f729c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729c06a7a0 0x7f729c06cc60 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f72ac036000 tx=0x7f72ac030000 comp rx=0 tx=0).stop 2024-03-23T11:32:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 -- 172.21.15.62:0/481840248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72bc15c220 msgr2=0x7f72bc154490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:32:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72bc15c220 0x7f72bc154490 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f72a800d7a0 tx=0x7f72a800dc60 comp rx=0 tx=0).stop 2024-03-23T11:32:59.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 -- 172.21.15.62:0/481840248 shutdown_connections 2024-03-23T11:32:59.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f729c06a7a0 0x7f729c06cc60 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72bc15c220 0x7f72bc154490 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72bc15c6a0 0x7f72bc1549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 --2- 172.21.15.62:0/481840248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72a0003b30 0x7f72bc0ace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:32:59.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 -- 172.21.15.62:0/481840248 >> 172.21.15.62:0/481840248 conn(0x7f72bc066050 msgr2=0x7f72bc0af8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:32:59.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 -- 172.21.15.62:0/481840248 shutdown_connections 2024-03-23T11:32:59.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:32:59.814+0000 7f72bb200640 1 -- 172.21.15.62:0/481840248 wait complete. 2024-03-23T11:32:59.821 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":544,"stamp":"2024-03-23T11:32:59.546953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":262264,"kb_used_data":13308,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754712456,"statfs":{"total":773094113280,"available":772825554944,"internally_reserved":0,"allocated":13627392,"data_stored":6596041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.998176"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":274,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-23T11:32:47.562424+0000","last_change":"2024-03-23T11:18:51.800909+0000","last_active":"2024-03-23T11:32:47.562424+0000","last_peered":"2024-03-23T11:32:47.562424+0000","last_clean":"2024-03-23T11:32:47.562424+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:32:47.562424+0000","last_undegraded":"2024-03-23T11:32:47.562424+0000","last_fullsized":"2024-03-23T11:32:47.562424+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T11:18:50.224193+0000","last_clean_scrub_stamp":"2024-03-23T11:18:50.224193+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:04:59.738072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542317,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33220,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338620,"statfs":{"total":96636764160,"available":96602746880,"internally_reserved":0,"allocated":2150400,"data_stored":1267532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.087,"15min":1.028},"min":{"1min":0.63600000000000001,"5min":0.623,"15min":0.5},"max":{"1min":1.456,"5min":1.631,"15min":2.8370000000000002},"last":1.109},{"interface":"front","average":{"1min":1.075,"5min":1.1220000000000001,"15min":1.0589999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.38300000000000001},"max":{"1min":1.5449999999999999,"5min":1.7230000000000001,"15min":1.8009999999999999},"last":1.069}]},{"osd":1,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.2809999999999999,"15min":1.214},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.46100000000000002},"max":{"1min":1.5720000000000001,"5min":1.708,"15min":3.569},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.2909999999999999,"15min":1.2669999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.69899999999999995},"max":{"1min":1.5309999999999999,"5min":1.8009999999999999,"15min":2.4350000000000001},"last":1.3740000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1299999999999999,"15min":1.0960000000000001},"min":{"1min":0.69899999999999995,"5min":0.65500000000000003,"15min":0.58099999999999996},"max":{"1min":1.4610000000000001,"5min":1.909,"15min":2.7440000000000002},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.23,"15min":1.1990000000000001},"min":{"1min":0.83799999999999997,"5min":0.81499999999999995,"15min":0.56999999999999995},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":3.524},"last":0.89800000000000002}]},{"osd":3,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.204,"15min":1.1930000000000001},"min":{"1min":0.86799999999999999,"5min":0.72199999999999998,"15min":0.67600000000000005},"max":{"1min":1.5169999999999999,"5min":1.7889999999999999,"15min":2.6389999999999998},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.3169999999999999,"15min":1.2809999999999999},"min":{"1min":0.94399999999999995,"5min":0.93500000000000005,"15min":0.60199999999999998},"max":{"1min":1.641,"5min":1.7050000000000001,"15min":2.9470000000000001},"last":1.228}]},{"osd":4,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1679999999999999,"15min":1.208},"min":{"1min":0.84099999999999997,"5min":0.73199999999999998,"15min":0.54300000000000004},"max":{"1min":1.534,"5min":2.1560000000000001,"15min":2.2719999999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.391,"15min":1.3979999999999999},"min":{"1min":1.071,"5min":0.81699999999999995,"15min":0.60199999999999998},"max":{"1min":1.7,"5min":2.0819999999999999,"15min":2.5569999999999999},"last":1.544}]},{"osd":5,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.369,"15min":1.351},"min":{"1min":0.745,"5min":0.745,"15min":0.53300000000000003},"max":{"1min":1.6719999999999999,"5min":1.8520000000000001,"15min":2.8999999999999999},"last":1.288},{"interface":"front","average":{"1min":1.379,"5min":1.448,"15min":1.427},"min":{"1min":1.073,"5min":0.91700000000000004,"15min":0.58199999999999996},"max":{"1min":1.6299999999999999,"5min":1.978,"15min":2.0859999999999999},"last":1.165}]},{"osd":6,"last update":"Sat Mar 23 11:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.391,"15min":1.417},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.61399999999999999},"max":{"1min":1.6379999999999999,"5min":2.0680000000000001,"15min":2.1299999999999999},"last":1.26},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.357,"15min":1.3720000000000001},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.59899999999999998},"max":{"1min":1.637,"5min":1.9970000000000001,"15min":3.218},"last":1.3140000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32636,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339204,"statfs":{"total":96636764160,"available":96603344896,"internally_reserved":0,"allocated":1552384,"data_stored":676495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.091,"15min":1.115},"min":{"1min":0.71299999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.536,"5min":1.7450000000000001,"15min":3.9140000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.133,"5min":1.1879999999999999,"15min":1.161},"min":{"1min":0.72299999999999998,"5min":0.71099999999999997,"15min":0.45200000000000001},"max":{"1min":1.53,"5min":1.8600000000000001,"15min":6.6539999999999999},"last":1.3819999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1819999999999999,"15min":1.2},"min":{"1min":0.84999999999999998,"5min":0.77300000000000002,"15min":0.69399999999999995},"max":{"1min":1.5469999999999999,"5min":1.627,"15min":2.4820000000000002},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1559999999999999,"15min":1.1970000000000001},"min":{"1min":0.79100000000000004,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.857,"5min":1.857,"15min":2.0859999999999999},"last":1.2829999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.173,"15min":1.2090000000000001},"min":{"1min":0.93000000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.6299999999999999,"5min":1.649,"15min":3.73},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.2989999999999999,"15min":1.29},"min":{"1min":0.91000000000000003,"5min":0.73499999999999999,"15min":0.68100000000000005},"max":{"1min":1.452,"5min":1.8,"15min":2.3460000000000001},"last":1.3460000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.246,"15min":1.278},"min":{"1min":0.83799999999999997,"5min":0.76700000000000002,"15min":0.749},"max":{"1min":1.6160000000000001,"5min":1.738,"15min":12.914},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.226,"5min":1.3080000000000001,"15min":1.3260000000000001},"min":{"1min":0.85099999999999998,"5min":0.76100000000000001,"15min":0.57799999999999996},"max":{"1min":1.5069999999999999,"5min":1.8919999999999999,"15min":3.8100000000000001},"last":1.476}]},{"osd":4,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.3149999999999999,"15min":1.3300000000000001},"min":{"1min":1.123,"5min":0.68000000000000005,"15min":0.51100000000000001},"max":{"1min":1.5920000000000001,"5min":1.827,"15min":2.702},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.2989999999999999,"15min":1.353},"min":{"1min":0.95499999999999996,"5min":0.752,"15min":0.752},"max":{"1min":1.875,"5min":1.875,"15min":7.2140000000000004},"last":1.129}]},{"osd":5,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.399,"15min":1.45},"min":{"1min":1.143,"5min":0.92900000000000005,"15min":0.73899999999999999},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":3.097},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.502,"5min":1.4510000000000001,"15min":1.466},"min":{"1min":1.355,"5min":0.86899999999999999,"15min":0.72499999999999998},"max":{"1min":1.6879999999999999,"5min":1.9139999999999999,"15min":3.3109999999999999},"last":1.4159999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.534,"15min":1.5860000000000001},"min":{"1min":1.2949999999999999,"5min":0.97799999999999998,"15min":0.80400000000000005},"max":{"1min":2.0150000000000001,"5min":2.0760000000000001,"15min":4.9729999999999999},"last":1.552},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4390000000000001,"15min":1.464},"min":{"1min":0.998,"5min":0.70399999999999996,"15min":0.57699999999999996},"max":{"1min":2.0640000000000001,"5min":2.0649999999999999,"15min":4.085},"last":1.2430000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.149,"15min":1.1240000000000001},"min":{"1min":0.63,"5min":0.63,"15min":0.55400000000000005},"max":{"1min":1.526,"5min":2.0179999999999998,"15min":2.2999999999999998},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.1439999999999999,"15min":1.145},"min":{"1min":0.73999999999999999,"5min":0.624,"15min":0.47799999999999998},"max":{"1min":1.518,"5min":1.7050000000000001,"15min":2.355},"last":0.67200000000000004}]},{"osd":1,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1990000000000001,"15min":1.171},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":2.2559999999999998},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.173,"5min":1.1970000000000001,"15min":1.1859999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.495},"max":{"1min":1.5,"5min":1.599,"15min":1.8580000000000001},"last":1.018}]},{"osd":2,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1799999999999999,"15min":1.212},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.60199999999999998},"max":{"1min":1.605,"5min":1.6659999999999999,"15min":2.012},"last":1.145},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.294,"15min":1.2529999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.64400000000000002},"max":{"1min":1.661,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":0.91300000000000003}]},{"osd":3,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.27,"15min":1.284},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5209999999999999,"5min":1.6499999999999999,"15min":2.3799999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.371,"15min":1.3700000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.6379999999999999,"5min":2.0750000000000002,"15min":2.3999999999999999},"last":1.1180000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.304,"15min":1.3240000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.587,"5min":2.1499999999999999,"15min":2.2090000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4730000000000001,"15min":1.48},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.6240000000000001,"5min":1.8919999999999999,"15min":2.5009999999999999},"last":0.96099999999999997}]},{"osd":6,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3759999999999999,"15min":1.393},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7,"5min":2.3170000000000002,"15min":2.4860000000000002},"last":1.196},{"interface":"front","average":{"1min":1.292,"5min":1.403,"15min":1.415},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.609,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.099}]},{"osd":7,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.4530000000000001,"15min":1.468},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.661,"5min":2.1549999999999998,"15min":2.7229999999999999},"last":1.23},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.5389999999999999,"15min":1.55},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.8640000000000001,"5min":2.2269999999999999,"15min":2.7509999999999999},"last":1.3799999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.2330000000000001,"15min":1.1830000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.8360000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.3049999999999999,"15min":1.2789999999999999},"min":{"1min":0.78900000000000003,"5min":0.74399999999999999,"15min":0.63700000000000001},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":1.339}]},{"osd":1,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.1539999999999999,"15min":1.1619999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.776,"5min":1.9350000000000001,"15min":4.327},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.129,"5min":1.1950000000000001,"15min":1.1619999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.4369999999999998},"last":0.85299999999999998}]},{"osd":2,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.177,"15min":1.202},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.624},"max":{"1min":1.6299999999999999,"5min":2.0960000000000001,"15min":2.6080000000000001},"last":1.159},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.2809999999999999,"15min":1.2689999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.71799999999999997},"max":{"1min":1.7090000000000001,"5min":1.8919999999999999,"15min":4.1310000000000002},"last":1.1890000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.92400000000000004,"5min":0.747,"15min":0.622},"max":{"1min":1.6100000000000001,"5min":1.659,"15min":2.5579999999999998},"last":1.24},{"interface":"front","average":{"1min":1.347,"5min":1.3740000000000001,"15min":1.3320000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.70199999999999996},"max":{"1min":1.835,"5min":2.024,"15min":4.2400000000000002},"last":1.3819999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.23,"15min":1.252},"min":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.74299999999999999},"max":{"1min":1.764,"5min":1.8740000000000001,"15min":2.508},"last":1.7569999999999999},{"interface":"front","average":{"1min":1.456,"5min":1.3360000000000001,"15min":1.3620000000000001},"min":{"1min":0.92800000000000005,"5min":0.67800000000000005,"15min":0.60499999999999998},"max":{"1min":1.7010000000000001,"5min":1.7849999999999999,"15min":5.859},"last":1.2909999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3759999999999999,"15min":1.401},"min":{"1min":0.94799999999999995,"5min":0.753,"15min":0.58899999999999997},"max":{"1min":2.1099999999999999,"5min":2.194,"15min":5.3339999999999996},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.5,"5min":1.4670000000000001,"15min":1.4870000000000001},"min":{"1min":1.26,"5min":0.81000000000000005,"15min":0.72599999999999998},"max":{"1min":1.885,"5min":1.885,"15min":5.3940000000000001},"last":1.514}]},{"osd":7,"last update":"Sat Mar 23 11:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.4910000000000001,"15min":1.498},"min":{"1min":1.0269999999999999,"5min":0.84699999999999998,"15min":0.754},"max":{"1min":2.0059999999999998,"5min":2.0790000000000002,"15min":2.6859999999999999},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.3839999999999999,"15min":1.3859999999999999},"min":{"1min":1.0589999999999999,"5min":0.79000000000000004,"15min":0.75600000000000001},"max":{"1min":1.978,"5min":1.978,"15min":2.641},"last":1.4259999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.88100000000000001,"15min":0.91800000000000004},"min":{"1min":0.42399999999999999,"5min":0.39300000000000002,"15min":0.32100000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.6819999999999999},"last":1.143},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.1160000000000001,"15min":1.101},"min":{"1min":0.48099999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.5589999999999999,"5min":1.8540000000000001,"15min":2.0790000000000002},"last":1.079}]},{"osd":1,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.96899999999999997,"15min":0.98299999999999998},"min":{"1min":0.51900000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.472,"5min":1.5249999999999999,"15min":1.976},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.202,"15min":1.2270000000000001},"min":{"1min":0.65200000000000002,"5min":0.47099999999999997,"15min":0.44600000000000001},"max":{"1min":1.5069999999999999,"5min":1.774,"15min":2.2309999999999999},"last":1.383}]},{"osd":2,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.107,"15min":1.1120000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5669999999999999,"5min":1.7729999999999999,"15min":3.3109999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.165,"15min":1.1879999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.48399999999999999},"max":{"1min":1.532,"5min":1.635,"15min":2.7109999999999999},"last":1.29}]},{"osd":4,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.26,"15min":1.2609999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.73199999999999998},"max":{"1min":1.6040000000000001,"5min":1.752,"15min":2.3490000000000002},"last":1.528},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.4370000000000001,"15min":1.456},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.905,"5min":2.081,"15min":4.8970000000000002},"last":1.2050000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.431,"15min":1.4670000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.7729999999999999,"5min":2.121,"15min":3.4910000000000001},"last":1.494},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.361,"15min":1.3959999999999999},"min":{"1min":0.77200000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.7909999999999999,"5min":1.8700000000000001,"15min":3.125},"last":1.2490000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.3640000000000001,"15min":1.409},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.9550000000000001,"5min":2.0630000000000002,"15min":4.5499999999999998},"last":1.052},{"interface":"front","average":{"1min":1.379,"5min":1.4379999999999999,"15min":1.4830000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.9339999999999999,"5min":2.2080000000000002,"15min":2.8450000000000002},"last":1.4399999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.4179999999999999,"15min":1.4450000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.879,"5min":1.998,"15min":2.952},"last":1.411},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.484,"15min":1.524},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8100000000000001,"5min":1.964,"15min":4.798},"last":1.345}]}]},{"osd":2,"up_from":14,"seq":60129542317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.878,"15min":0.89200000000000002},"min":{"1min":0.61099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":1.375,"5min":1.411,"15min":2.2509999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.109,"5min":1.1000000000000001,"15min":1.0660000000000001},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.35099999999999998},"max":{"1min":1.371,"5min":1.6759999999999999,"15min":1.8220000000000001},"last":1.1499999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88900000000000001,"15min":0.89600000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.32000000000000001},"max":{"1min":1.3200000000000001,"5min":1.798,"15min":1.9430000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.05,"5min":1.069,"15min":1.036},"min":{"1min":0.66700000000000004,"5min":0.45000000000000001,"15min":0.35799999999999998},"max":{"1min":1.4550000000000001,"5min":1.5349999999999999,"15min":1.7230000000000001},"last":1.2270000000000001}]},{"osd":3,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1220000000000001,"15min":1.111},"min":{"1min":0.72399999999999998,"5min":0.47399999999999998,"15min":0.38300000000000001},"max":{"1min":1.3580000000000001,"5min":1.758,"15min":1.98},"last":1.056},{"interface":"front","average":{"1min":1.008,"5min":1.0629999999999999,"15min":1.069},"min":{"1min":0.53200000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.524,"5min":1.6879999999999999,"15min":2.1560000000000001},"last":0.97499999999999998}]},{"osd":4,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3520000000000001,"15min":1.3540000000000001},"min":{"1min":0.90400000000000003,"5min":0.86699999999999999,"15min":0.69599999999999995},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":2.1280000000000001},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.4279999999999999,"15min":1.411},"min":{"1min":0.95299999999999996,"5min":0.93799999999999994,"15min":0.83499999999999996},"max":{"1min":1.774,"5min":1.9079999999999999,"15min":2.085},"last":1.319}]},{"osd":5,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.353,"15min":1.359},"min":{"1min":1.0209999999999999,"5min":0.752,"15min":0.64100000000000001},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":2.5830000000000002},"last":1.403},{"interface":"front","average":{"1min":1.395,"5min":1.341,"15min":1.359},"min":{"1min":0.93899999999999995,"5min":0.73999999999999999,"15min":0.66000000000000003},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":2.6469999999999998},"last":1.3580000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4159999999999999,"15min":1.3899999999999999},"min":{"1min":0.94899999999999995,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.758},"last":1.518},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4139999999999999,"15min":1.4350000000000001},"min":{"1min":1.0429999999999999,"5min":0.79100000000000004,"15min":0.69599999999999995},"max":{"1min":1.7869999999999999,"5min":1.883,"15min":2.8580000000000001},"last":1.645}]},{"osd":7,"last update":"Sat Mar 23 11:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.468,"15min":1.4430000000000001},"min":{"1min":0.94299999999999995,"5min":0.83099999999999996,"15min":0.71899999999999997},"max":{"1min":2.23,"5min":2.23,"15min":2.246},"last":1.381},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.476,"15min":1.5029999999999999},"min":{"1min":1.0489999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.8819999999999999,"5min":1.9550000000000001,"15min":2.7120000000000002},"last":1.5569999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542318,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32632,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1552384,"data_stored":676495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.876,"15min":0.89400000000000002},"min":{"1min":0.41599999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.6579999999999999,"5min":1.6930000000000001,"15min":3.2959999999999998},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.002,"5min":1.0620000000000001,"15min":1.0149999999999999},"min":{"1min":0.5,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.53,"5min":1.8480000000000001,"15min":1.99},"last":0.78300000000000003}]},{"osd":2,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.94699999999999995},"min":{"1min":0.51200000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2949999999999999,"5min":1.5549999999999999,"15min":1.738},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.105,"15min":1.1040000000000001},"min":{"1min":0.60499999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5109999999999999,"5min":1.702,"15min":2.2160000000000002},"last":1.105}]},{"osd":3,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2250000000000001,"15min":1.222},"min":{"1min":0.60699999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.734,"5min":1.8819999999999999,"15min":2.4430000000000001},"last":1.369},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.133,"15min":1.1220000000000001},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.434},"max":{"1min":1.637,"5min":2.0539999999999998,"15min":2.4169999999999998},"last":1.147}]},{"osd":4,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.345,"15min":1.3140000000000001},"min":{"1min":0.92100000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7190000000000001,"5min":1.8080000000000001,"15min":2.153},"last":1.026},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.286,"15min":1.3},"min":{"1min":0.747,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":2.758},"last":0.877}]},{"osd":5,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.357},"min":{"1min":0.81100000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":2.004,"5min":2.004,"15min":2.6589999999999998},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.411,"5min":1.4239999999999999,"15min":1.4259999999999999},"min":{"1min":0.97599999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.8109999999999999,"5min":1.9610000000000001,"15min":3.0329999999999999},"last":1.6160000000000001}]},{"osd":6,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4119999999999999,"15min":1.4179999999999999},"min":{"1min":1.0369999999999999,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.984,"5min":1.984,"15min":2.2730000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4359999999999999,"15min":1.4650000000000001},"min":{"1min":0.91100000000000003,"5min":0.747,"15min":0.747},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.9630000000000001},"last":1.175}]},{"osd":7,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.524,"15min":1.528},"min":{"1min":0.93500000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.1269999999999998,"5min":2.2450000000000001,"15min":3.2749999999999999},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4990000000000001,"15min":1.5249999999999999},"min":{"1min":1.0740000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.7829999999999999},"last":1.2889999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542316,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33216,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338624,"statfs":{"total":96636764160,"available":96602750976,"internally_reserved":0,"allocated":2146304,"data_stored":1266863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.85199999999999998,"15min":0.85799999999999998},"min":{"1min":0.55100000000000005,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.262,"5min":1.8660000000000001,"15min":2.8479999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":0.97599999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.30399999999999999},"max":{"1min":1.694,"5min":1.694,"15min":2.4319999999999999},"last":0.70199999999999996}]},{"osd":2,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0269999999999999,"15min":1.0349999999999999},"min":{"1min":0.55800000000000005,"5min":0.41799999999999998,"15min":0.33700000000000002},"max":{"1min":1.849,"5min":1.849,"15min":2.4980000000000002},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.024,"5min":1.1180000000000001,"15min":1.069},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.432},"max":{"1min":1.5820000000000001,"5min":1.7569999999999999,"15min":2.5760000000000001},"last":0.90000000000000002}]},{"osd":3,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1579999999999999,"15min":1.123},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.35799999999999998},"max":{"1min":1.552,"5min":1.7589999999999999,"15min":2.4580000000000002},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3460000000000001,"15min":1.3200000000000001},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.48599999999999999},"max":{"1min":1.6519999999999999,"5min":1.968,"15min":2.9390000000000001},"last":1.036}]},{"osd":4,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3320000000000001,"15min":1.3360000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.7410000000000001,"5min":1.855,"15min":2.9750000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3360000000000001,"15min":1.298},"min":{"1min":0.88500000000000001,"5min":0.81399999999999995,"15min":0.39900000000000002},"max":{"1min":1.728,"5min":1.968,"15min":3.125},"last":1.242}]},{"osd":5,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.361,"15min":1.3660000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.51200000000000001},"max":{"1min":1.595,"5min":1.925,"15min":3.0830000000000002},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.3400000000000001,"15min":1.4079999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.52700000000000002},"max":{"1min":1.772,"5min":2.0449999999999999,"15min":3.1899999999999999},"last":1.3009999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.4139999999999999,"15min":1.448},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.47999999999999998},"max":{"1min":1.77,"5min":2.1030000000000002,"15min":3.391},"last":1.506},{"interface":"front","average":{"1min":1.367,"5min":1.4259999999999999,"15min":1.4079999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.58399999999999996},"max":{"1min":1.956,"5min":2.2170000000000001,"15min":3.153},"last":1.387}]},{"osd":7,"last update":"Sat Mar 23 11:32:14 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.5349999999999999,"15min":1.5189999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.60599999999999998},"max":{"1min":1.877,"5min":2.1400000000000001,"15min":3.5350000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.516,"15min":1.494},"min":{"1min":0.92600000000000005,"5min":0.78300000000000003,"15min":0.45300000000000001},"max":{"1min":1.7090000000000001,"5min":2.2610000000000001,"15min":3.3650000000000002},"last":1.5349999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:32:59.825 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-23T11:18:50.224193+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=11, tm_min=18, tm_sec=50, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=11, tm_min=32, tm_sec=52, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T11:32:59.825 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-23T11:33:18.521 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:33:18.530 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:33:19.826 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T11:33:20.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.055+0000 7f69d1c00640 1 Processor -- start 2024-03-23T11:33:20.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.055+0000 7f69d1c00640 1 -- start start 2024-03-23T11:33:20.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.056+0000 7f69d1c00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f69cc15cb90 0x7f69cc0b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.056+0000 7f69d1c00640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc0b5ad0 con 0x7f69cc06b040 2024-03-23T11:33:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.056+0000 7f69d1c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc06b820 con 0x7f69cc0cde10 2024-03-23T11:33:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.056+0000 7f69d1c00640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69cc06c0c0 con 0x7f69cc15cb90 2024-03-23T11:33:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.056+0000 7f69caa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f69cc15cb90 0x7f69cc0b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.057+0000 7f69caa00640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f69cc15cb90 0x7f69cc0b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58082/0 (socket says 172.21.15.62:58082) 2024-03-23T11:33:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.057+0000 7f69ca000640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f69cc06b040 0x7f69cc0cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44574/0 (socket says 172.21.15.62:44574) 2024-03-23T11:33:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.057+0000 7f69caa00640 1 -- 172.21.15.62:0/4047607554 learned_addr learned my addr 172.21.15.62:0/4047607554 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:33:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.057+0000 7f69caa00640 1 -- 172.21.15.62:0/4047607554 >> v1:172.21.15.120:6789/0 conn(0x7f69cc0cde10 legacy=0x7f69cc0d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:33:20.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.057+0000 7f69caa00640 1 -- 172.21.15.62:0/4047607554 >> v1:172.21.15.62:6789/0 conn(0x7f69cc06b040 legacy=0x7f69cc0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.057+0000 7f69caa00640 1 -- 172.21.15.62:0/4047607554 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc06bb00 con 0x7f69cc15cb90 2024-03-23T11:33:20.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.057+0000 7f69c9600640 1 -- 172.21.15.62:0/4047607554 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020689286 0 0) 0x7f69cc0b5ad0 con 0x7f69cc06b040 2024-03-23T11:33:20.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.058+0000 7f69caa00640 1 --2- 172.21.15.62:0/4047607554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69cc15cb90 0x7f69cc0b84f0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009870 tx=0x7f69bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c9b4f525e98cb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:33:20.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.058+0000 7f69c9600640 1 -- 172.21.15.62:0/4047607554 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc030860 con 0x7f69cc15cb90 2024-03-23T11:33:20.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.059+0000 7f69c9600640 1 -- 172.21.15.62:0/4047607554 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69bc0309c0 con 0x7f69cc15cb90 2024-03-23T11:33:20.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.059+0000 7f69c9600640 1 -- 172.21.15.62:0/4047607554 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc030d50 con 0x7f69cc15cb90 2024-03-23T11:33:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.059+0000 7f69d1c00640 1 -- 172.21.15.62:0/4047607554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69cc15cb90 msgr2=0x7f69cc0b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.059+0000 7f69d1c00640 1 --2- 172.21.15.62:0/4047607554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69cc15cb90 0x7f69cc0b84f0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009870 tx=0x7f69bc02b9a0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.059+0000 7f69d1c00640 1 -- 172.21.15.62:0/4047607554 shutdown_connections 2024-03-23T11:33:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.059+0000 7f69d1c00640 1 --2- 172.21.15.62:0/4047607554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69cc15cb90 0x7f69cc0b84f0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.059+0000 7f69d1c00640 1 -- 172.21.15.62:0/4047607554 >> 172.21.15.62:0/4047607554 conn(0x7f69cc0cf7d0 msgr2=0x7f69cc0cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:33:20.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.059+0000 7f69d1c00640 1 -- 172.21.15.62:0/4047607554 shutdown_connections 2024-03-23T11:33:20.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.059+0000 7f69d1c00640 1 -- 172.21.15.62:0/4047607554 wait complete. 2024-03-23T11:33:20.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.061+0000 7f69d1c00640 1 Processor -- start 2024-03-23T11:33:20.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.061+0000 7f69d1c00640 1 -- start start 2024-03-23T11:33:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.062+0000 7f69d1c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69cc06b040 0x7f69cc14f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.062+0000 7f69d1c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69cc0cde10 0x7f69cc14fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.062+0000 7f69d1c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc15cb90 0x7f69cc15a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.062+0000 7f69d1c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0d1010 con 0x7f69cc06b040 2024-03-23T11:33:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.062+0000 7f69d1c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc152400 con 0x7f69cc15cb90 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.062+0000 7f69d1c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0cbaf0 con 0x7f69cc0cde10 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.062+0000 7f69caa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69cc06b040 0x7f69cc14f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.062+0000 7f69caa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69cc06b040 0x7f69cc14f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58098/0 (socket says 172.21.15.62:58098) 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.062+0000 7f69caa00640 1 -- 172.21.15.62:0/1963611323 learned_addr learned my addr 172.21.15.62:0/1963611323 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.063+0000 7f69ca000640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69cc0cde10 0x7f69cc14fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.063+0000 7f69cb400640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc15cb90 0x7f69cc15a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.063+0000 7f69caa00640 1 -- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69cc0cde10 msgr2=0x7f69cc14fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.063+0000 7f69caa00640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69cc0cde10 0x7f69cc14fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.063+0000 7f69caa00640 1 -- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc15cb90 msgr2=0x7f69cc15a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.063+0000 7f69caa00640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc15cb90 0x7f69cc15a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.063+0000 7f69caa00640 1 -- 172.21.15.62:0/1963611323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc0094d0 con 0x7f69cc06b040 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.063+0000 7f69cb400640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc15cb90 0x7f69cc15a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:33:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.063+0000 7f69caa00640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69cc06b040 0x7f69cc14f720 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009870 tx=0x7f69bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:33:20.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.064+0000 7f69c2a00640 1 -- 172.21.15.62:0/1963611323 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc030810 con 0x7f69cc06b040 2024-03-23T11:33:20.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.064+0000 7f69c2a00640 1 -- 172.21.15.62:0/1963611323 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69bc039d90 con 0x7f69cc06b040 2024-03-23T11:33:20.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.064+0000 7f69c2a00640 1 -- 172.21.15.62:0/1963611323 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc043630 con 0x7f69cc06b040 2024-03-23T11:33:20.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.064+0000 7f69d1c00640 1 -- 172.21.15.62:0/1963611323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69cc0cb1c0 con 0x7f69cc06b040 2024-03-23T11:33:20.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.064+0000 7f69d1c00640 1 -- 172.21.15.62:0/1963611323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69cc0cb780 con 0x7f69cc06b040 2024-03-23T11:33:20.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.065+0000 7f69d1c00640 1 -- 172.21.15.62:0/1963611323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69cc066c90 con 0x7f69cc06b040 2024-03-23T11:33:20.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.066+0000 7f69c2a00640 1 -- 172.21.15.62:0/1963611323 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f69bc02f3b0 con 0x7f69cc06b040 2024-03-23T11:33:20.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.067+0000 7f69c2a00640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69ac06a7d0 0x7f69ac06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.067+0000 7f69c2a00640 1 -- 172.21.15.62:0/1963611323 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f69bc0aaef0 con 0x7f69cc06b040 2024-03-23T11:33:20.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.067+0000 7f69ca000640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69ac06a7d0 0x7f69ac06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.069+0000 7f69ca000640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69ac06a7d0 0x7f69ac06cc90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0c6390 tx=0x7f69b8007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:33:20.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.072+0000 7f69c2a00640 1 -- 172.21.15.62:0/1963611323 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69bc076570 con 0x7f69cc06b040 2024-03-23T11:33:20.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.268+0000 7f69d1c00640 1 -- 172.21.15.62:0/1963611323 --> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f69cc06b4a0 con 0x7f69ac06a7d0 2024-03-23T11:33:20.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.274+0000 7f69c2a00640 1 -- 172.21.15.62:0/1963611323 <== mgr.4103 v2:172.21.15.120:6832/37071 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48104 (secure 0 0 0) 0x7f69b8004c70 con 0x7f69ac06a7d0 2024-03-23T11:33:20.263 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:33:20.264 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-03-23T11:33:20.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.276+0000 7f69d1c00640 1 -- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69ac06a7d0 msgr2=0x7f69ac06cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.276+0000 7f69d1c00640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69ac06a7d0 0x7f69ac06cc90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0c6390 tx=0x7f69b8007d50 comp rx=0 tx=0).stop 2024-03-23T11:33:20.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.277+0000 7f69d1c00640 1 -- 172.21.15.62:0/1963611323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69cc06b040 msgr2=0x7f69cc14f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.277+0000 7f69d1c00640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69cc06b040 0x7f69cc14f720 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009870 tx=0x7f69bc004730 comp rx=0 tx=0).stop 2024-03-23T11:33:20.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.277+0000 7f69d1c00640 1 -- 172.21.15.62:0/1963611323 shutdown_connections 2024-03-23T11:33:20.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.277+0000 7f69d1c00640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f69ac06a7d0 0x7f69ac06cc90 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.277+0000 7f69d1c00640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69cc15cb90 0x7f69cc15a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.277+0000 7f69d1c00640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f69cc0cde10 0x7f69cc14fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.277+0000 7f69d1c00640 1 --2- 172.21.15.62:0/1963611323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69cc06b040 0x7f69cc14f720 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.277+0000 7f69d1c00640 1 -- 172.21.15.62:0/1963611323 >> 172.21.15.62:0/1963611323 conn(0x7f69cc0cf7d0 msgr2=0x7f69cc0b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:33:20.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.277+0000 7f69d1c00640 1 -- 172.21.15.62:0/1963611323 shutdown_connections 2024-03-23T11:33:20.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.277+0000 7f69d1c00640 1 -- 172.21.15.62:0/1963611323 wait complete. 2024-03-23T11:33:20.287 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2024-03-23T11:33:19.552501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":262248,"kb_used_data":13308,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754712472,"statfs":{"total":773094113280,"available":772825571328,"internally_reserved":0,"allocated":13627392,"data_stored":6596041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003341"},"pg_stats":[{"pgid":"1.0","version":"125'119","reported_seq":288,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-23T11:33:00.334351+0000","last_change":"2024-03-23T11:33:00.334624+0000","last_active":"2024-03-23T11:33:00.334351+0000","last_peered":"2024-03-23T11:33:00.334351+0000","last_clean":"2024-03-23T11:33:00.334351+0000","last_became_active":"2024-03-23T11:18:51.798266+0000","last_became_peered":"2024-03-23T11:18:51.798266+0000","last_unstale":"2024-03-23T11:33:00.334351+0000","last_undegraded":"2024-03-23T11:33:00.334351+0000","last_fullsized":"2024-03-23T11:33:00.334351+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'119","last_scrub_stamp":"2024-03-23T11:33:00.334295+0000","last_deep_scrub":"125'119","last_deep_scrub_stamp":"2024-03-23T11:33:00.334295+0000","last_clean_scrub_stamp":"2024-03-23T11:33:00.334295+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:57:14.057216+0000","scrub_duration":0.011238819000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542321,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33212,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338628,"statfs":{"total":96636764160,"available":96602755072,"internally_reserved":0,"allocated":2150400,"data_stored":1267532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0580000000000001,"15min":1.038},"min":{"1min":0.68999999999999995,"5min":0.623,"15min":0.5},"max":{"1min":1.3400000000000001,"5min":1.599,"15min":2.8370000000000002},"last":1.492},{"interface":"front","average":{"1min":1.109,"5min":1.1080000000000001,"15min":1.0680000000000001},"min":{"1min":0.68000000000000005,"5min":0.58599999999999997,"15min":0.38300000000000001},"max":{"1min":1.5600000000000001,"5min":1.7230000000000001,"15min":1.8009999999999999},"last":1.5629999999999999}]},{"osd":1,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.2549999999999999,"15min":1.216},"min":{"1min":0.69999999999999996,"5min":0.67600000000000005,"15min":0.46100000000000002},"max":{"1min":1.526,"5min":1.708,"15min":3.569},"last":1.593},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.276,"15min":1.2669999999999999},"min":{"1min":0.83999999999999997,"5min":0.72799999999999998,"15min":0.69899999999999995},"max":{"1min":1.579,"5min":1.7969999999999999,"15min":2.4350000000000001},"last":1.4399999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1339999999999999,"15min":1.101},"min":{"1min":0.77600000000000002,"5min":0.65500000000000003,"15min":0.58099999999999996},"max":{"1min":1.4530000000000001,"5min":1.909,"15min":2.7440000000000002},"last":1.375},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.204,"15min":1.2010000000000001},"min":{"1min":0.89800000000000002,"5min":0.81499999999999995,"15min":0.56999999999999995},"max":{"1min":1.4810000000000001,"5min":1.7609999999999999,"15min":3.524},"last":1.2889999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2170000000000001,"15min":1.2030000000000001},"min":{"1min":0.89500000000000002,"5min":0.72199999999999998,"15min":0.67600000000000005},"max":{"1min":1.569,"5min":1.7889999999999999,"15min":2.6389999999999998},"last":1.639},{"interface":"front","average":{"1min":1.28,"5min":1.2949999999999999,"15min":1.2829999999999999},"min":{"1min":1.008,"5min":0.93500000000000005,"15min":0.60199999999999998},"max":{"1min":1.569,"5min":1.6850000000000001,"15min":2.9470000000000001},"last":1.6759999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.1779999999999999,"15min":1.2110000000000001},"min":{"1min":0.78900000000000003,"5min":0.73199999999999998,"15min":0.54300000000000004},"max":{"1min":1.784,"5min":2.1560000000000001,"15min":2.2719999999999998},"last":1.341},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.409,"15min":1.403},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.60199999999999998},"max":{"1min":1.704,"5min":2.0819999999999999,"15min":2.5569999999999999},"last":1.7609999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.373,"15min":1.3520000000000001},"min":{"1min":0.85799999999999998,"5min":0.745,"15min":0.53300000000000003},"max":{"1min":1.619,"5min":1.8520000000000001,"15min":2.8999999999999999},"last":1.798},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4399999999999999,"15min":1.4319999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.58199999999999996},"max":{"1min":1.7569999999999999,"5min":1.978,"15min":2.0859999999999999},"last":1.714}]},{"osd":6,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.389,"15min":1.4159999999999999},"min":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.61399999999999999},"max":{"1min":2.0510000000000002,"5min":2.0680000000000001,"15min":2.1299999999999999},"last":1.625},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.355,"15min":1.373},"min":{"1min":0.96899999999999997,"5min":0.90400000000000003,"15min":0.59899999999999998},"max":{"1min":1.6879999999999999,"5min":1.9970000000000001,"15min":3.218},"last":1.8939999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32636,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339204,"statfs":{"total":96636764160,"available":96603344896,"internally_reserved":0,"allocated":1552384,"data_stored":676495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.077,"15min":1.111},"min":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.448,"5min":1.7450000000000001,"15min":3.9140000000000001},"last":1.3919999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.1419999999999999,"15min":1.1519999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.45200000000000001},"max":{"1min":1.3819999999999999,"5min":1.8600000000000001,"15min":6.6539999999999999},"last":0.79600000000000004}]},{"osd":1,"last update":"Sat Mar 23 11:33:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1819999999999999,"15min":1.1930000000000001},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.657,"5min":1.657,"15min":2.4820000000000002},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.1619999999999999,"15min":1.1930000000000001},"min":{"1min":0.94699999999999995,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.603,"5min":1.857,"15min":2.0859999999999999},"last":1.3740000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:33:02 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.1659999999999999,"15min":1.2050000000000001},"min":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.571,"5min":1.649,"15min":3.73},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.292,"15min":1.2829999999999999},"min":{"1min":0.73899999999999999,"5min":0.73499999999999999,"15min":0.68100000000000005},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.3460000000000001},"last":1.3149999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:33:02 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2390000000000001,"15min":1.2769999999999999},"min":{"1min":0.89600000000000002,"5min":0.76700000000000002,"15min":0.749},"max":{"1min":1.7130000000000001,"5min":1.738,"15min":12.914},"last":1.272},{"interface":"front","average":{"1min":1.27,"5min":1.2949999999999999,"15min":1.3180000000000001},"min":{"1min":0.94199999999999995,"5min":0.76100000000000001,"15min":0.57799999999999996},"max":{"1min":1.6299999999999999,"5min":1.8919999999999999,"15min":3.8100000000000001},"last":1.1000000000000001}]},{"osd":4,"last update":"Sat Mar 23 11:33:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.3240000000000001,"15min":1.321},"min":{"1min":0.79000000000000004,"5min":0.68000000000000005,"15min":0.51100000000000001},"max":{"1min":2.294,"5min":2.294,"15min":2.702},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.3120000000000001,"15min":1.349},"min":{"1min":1.0029999999999999,"5min":0.752,"15min":0.752},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":7.2140000000000004},"last":1.4570000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:33:02 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4119999999999999,"15min":1.4410000000000001},"min":{"1min":1.125,"5min":0.92900000000000005,"15min":0.73899999999999999},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":3.097},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4490000000000001,"15min":1.458},"min":{"1min":0.875,"5min":0.86899999999999999,"15min":0.72499999999999998},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":3.3109999999999999},"last":1.2090000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:33:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5449999999999999,"15min":1.575},"min":{"1min":1.292,"5min":0.97799999999999998,"15min":0.80400000000000005},"max":{"1min":2.0019999999999998,"5min":2.0760000000000001,"15min":4.9729999999999999},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.363,"5min":1.4219999999999999,"15min":1.452},"min":{"1min":0.91000000000000003,"5min":0.70399999999999996,"15min":0.57699999999999996},"max":{"1min":1.694,"5min":2.0649999999999999,"15min":4.085},"last":0.91500000000000004}]}]},{"osd":5,"up_from":14,"seq":60129542321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.149,"15min":1.1240000000000001},"min":{"1min":0.63,"5min":0.63,"15min":0.55400000000000005},"max":{"1min":1.526,"5min":2.0179999999999998,"15min":2.2999999999999998},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.115,"5min":1.1439999999999999,"15min":1.145},"min":{"1min":0.73999999999999999,"5min":0.624,"15min":0.47799999999999998},"max":{"1min":1.518,"5min":1.7050000000000001,"15min":2.355},"last":0.91800000000000004}]},{"osd":1,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1990000000000001,"15min":1.171},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":2.2559999999999998},"last":1.022},{"interface":"front","average":{"1min":1.173,"5min":1.1970000000000001,"15min":1.1859999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.495},"max":{"1min":1.5,"5min":1.599,"15min":1.8580000000000001},"last":1.427}]},{"osd":2,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1799999999999999,"15min":1.212},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.60199999999999998},"max":{"1min":1.605,"5min":1.6659999999999999,"15min":2.012},"last":1.157},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.294,"15min":1.2529999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.64400000000000002},"max":{"1min":1.661,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.202}]},{"osd":3,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.27,"15min":1.284},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5209999999999999,"5min":1.6499999999999999,"15min":2.3799999999999999},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.371,"15min":1.3700000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.6379999999999999,"5min":2.0750000000000002,"15min":2.3999999999999999},"last":1.3009999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.304,"15min":1.3240000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.587,"5min":2.1499999999999999,"15min":2.2090000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4730000000000001,"15min":1.48},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.6240000000000001,"5min":1.8919999999999999,"15min":2.5009999999999999},"last":1.331}]},{"osd":6,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3759999999999999,"15min":1.393},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7,"5min":2.3170000000000002,"15min":2.4860000000000002},"last":0.877},{"interface":"front","average":{"1min":1.292,"5min":1.403,"15min":1.415},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.609,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.3600000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:32:21 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.4530000000000001,"15min":1.468},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.661,"5min":2.1549999999999998,"15min":2.7229999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.5389999999999999,"15min":1.55},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.8640000000000001,"5min":2.2269999999999999,"15min":2.7509999999999999},"last":1.1040000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.23,"15min":1.1910000000000001},"min":{"1min":0.68500000000000005,"5min":0.5,"15min":0.5},"max":{"1min":1.413,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.3080000000000001,"15min":1.2849999999999999},"min":{"1min":0.78000000000000003,"5min":0.74399999999999999,"15min":0.63700000000000001},"max":{"1min":1.5389999999999999,"5min":1.899,"15min":1.899},"last":1.504}]},{"osd":1,"last update":"Sat Mar 23 11:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.165,"15min":1.169},"min":{"1min":0.86499999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.53,"5min":1.9350000000000001,"15min":4.327},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.111,"5min":1.1830000000000001,"15min":1.1639999999999999},"min":{"1min":0.83499999999999996,"5min":0.628,"15min":0.628},"max":{"1min":1.3160000000000001,"5min":1.9099999999999999,"15min":2.4369999999999998},"last":1.48}]},{"osd":2,"last update":"Sat Mar 23 11:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1890000000000001,"15min":1.196},"min":{"1min":0.752,"5min":0.70299999999999996,"15min":0.624},"max":{"1min":1.429,"5min":2.0960000000000001,"15min":2.6080000000000001},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2649999999999999,"15min":1.2689999999999999},"min":{"1min":0.93999999999999995,"5min":0.80200000000000005,"15min":0.71799999999999997},"max":{"1min":1.514,"5min":1.8919999999999999,"15min":4.1310000000000002},"last":1.444}]},{"osd":3,"last update":"Sat Mar 23 11:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.24,"15min":1.2350000000000001},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.622},"max":{"1min":1.5029999999999999,"5min":1.659,"15min":2.5579999999999998},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.3620000000000001,"15min":1.335},"min":{"1min":0.90600000000000003,"5min":0.84599999999999997,"15min":0.70199999999999996},"max":{"1min":1.579,"5min":2.024,"15min":4.2400000000000002},"last":1.6379999999999999}]},{"osd":5,"last update":"Sat Mar 23 11:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.27,"15min":1.2490000000000001},"min":{"1min":0.92100000000000004,"5min":0.77100000000000002,"15min":0.74299999999999999},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.508},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.351,"5min":1.3480000000000001,"15min":1.3620000000000001},"min":{"1min":1.081,"5min":0.67800000000000005,"15min":0.60499999999999998},"max":{"1min":1.6599999999999999,"5min":1.7849999999999999,"15min":5.859},"last":1.5549999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.4019999999999999},"min":{"1min":1.1240000000000001,"5min":0.753,"15min":0.58899999999999997},"max":{"1min":2.1419999999999999,"5min":2.194,"15min":5.3339999999999996},"last":1.583},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.472,"15min":1.484},"min":{"1min":1.0589999999999999,"5min":0.81000000000000005,"15min":0.72599999999999998},"max":{"1min":1.8700000000000001,"5min":1.885,"15min":5.3940000000000001},"last":1.2589999999999999}]},{"osd":7,"last update":"Sat Mar 23 11:33:11 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.5269999999999999,"15min":1.5049999999999999},"min":{"1min":1.373,"5min":0.84699999999999998,"15min":0.754},"max":{"1min":2.242,"5min":2.242,"15min":2.6859999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4079999999999999,"15min":1.383},"min":{"1min":0.80300000000000005,"5min":0.79000000000000004,"15min":0.75600000000000001},"max":{"1min":1.988,"5min":1.988,"15min":2.641},"last":1.0880000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.89800000000000002,"15min":0.91700000000000004},"min":{"1min":0.64300000000000002,"5min":0.39300000000000002,"15min":0.32100000000000001},"max":{"1min":1.405,"5min":1.617,"15min":1.6819999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.1040000000000001,"15min":1.097},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.569,"5min":1.8540000000000001,"15min":2.0790000000000002},"last":1.0800000000000001}]},{"osd":1,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.94399999999999995,"15min":0.96999999999999997},"min":{"1min":0.53200000000000003,"5min":0.51900000000000002,"15min":0.42099999999999999},"max":{"1min":1.3069999999999999,"5min":1.5249999999999999,"15min":1.976},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1890000000000001,"15min":1.2210000000000001},"min":{"1min":0.73199999999999998,"5min":0.47099999999999997,"15min":0.44600000000000001},"max":{"1min":1.6100000000000001,"5min":1.774,"15min":2.2309999999999999},"last":1.4139999999999999}]},{"osd":2,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.0820000000000001,"15min":1.1060000000000001},"min":{"1min":0.69199999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.585,"5min":1.746,"15min":3.3109999999999999},"last":1.262},{"interface":"front","average":{"1min":1.232,"5min":1.1719999999999999,"15min":1.1830000000000001},"min":{"1min":0.96099999999999997,"5min":0.55200000000000005,"15min":0.48399999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":2.7109999999999999},"last":1.131}]},{"osd":4,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2450000000000001,"15min":1.258},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73199999999999998},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":2.3490000000000002},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4139999999999999,"15min":1.4430000000000001},"min":{"1min":0.92800000000000005,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.637,"5min":2.081,"15min":4.8970000000000002},"last":1.3740000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.4079999999999999,"15min":1.454},"min":{"1min":1.028,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.6879999999999999,"5min":2.121,"15min":3.4910000000000001},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.343,"15min":1.3819999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.70299999999999996},"max":{"1min":1.6830000000000001,"5min":1.8700000000000001,"15min":3.125},"last":1.494}]},{"osd":6,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.351,"15min":1.3919999999999999},"min":{"1min":1.052,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.6990000000000001,"5min":2.0630000000000002,"15min":4.5499999999999998},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.444,"15min":1.474},"min":{"1min":1.177,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.6870000000000001,"5min":2.2080000000000002,"15min":2.8450000000000002},"last":1.5940000000000001}]},{"osd":7,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3999999999999999,"15min":1.4319999999999999},"min":{"1min":0.86899999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.752,"5min":1.998,"15min":2.952},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4670000000000001,"15min":1.508},"min":{"1min":1.149,"5min":0.62,"15min":0.62},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":4.798},"last":1.46}]}]},{"osd":2,"up_from":14,"seq":60129542321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":677164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86299999999999999,"15min":0.88900000000000001},"min":{"1min":0.45600000000000002,"5min":0.246,"15min":0.246},"max":{"1min":1.2889999999999999,"5min":1.375,"15min":2.2509999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.038,"5min":1.077,"15min":1.071},"min":{"1min":0.64600000000000002,"5min":0.40300000000000002,"15min":0.35099999999999998},"max":{"1min":1.361,"5min":1.667,"15min":1.8220000000000001},"last":1.323}]},{"osd":1,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.86299999999999999,"15min":0.88600000000000001},"min":{"1min":0.52500000000000002,"5min":0.49399999999999999,"15min":0.32000000000000001},"max":{"1min":1.161,"5min":1.798,"15min":1.9430000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.0389999999999999,"15min":1.0369999999999999},"min":{"1min":0.58699999999999997,"5min":0.45000000000000001,"15min":0.35799999999999998},"max":{"1min":1.478,"5min":1.5349999999999999,"15min":1.7230000000000001},"last":1.0069999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.626,"5min":0.47399999999999998,"15min":0.38300000000000001},"max":{"1min":1.669,"5min":1.669,"15min":1.98},"last":1.272},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0609999999999999,"15min":1.069},"min":{"1min":0.71299999999999997,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.6859999999999999,"5min":1.6879999999999999,"15min":2.1560000000000001},"last":0.85599999999999998}]},{"osd":4,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.91700000000000004,"5min":0.86699999999999999,"15min":0.69599999999999995},"max":{"1min":1.853,"5min":1.8819999999999999,"15min":2.1280000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.413,"15min":1.4099999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.83499999999999996},"max":{"1min":1.667,"5min":1.9079999999999999,"15min":2.085},"last":1.2110000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3440000000000001,"15min":1.353},"min":{"1min":1.03,"5min":0.752,"15min":0.64100000000000001},"max":{"1min":1.7170000000000001,"5min":1.9299999999999999,"15min":2.5830000000000002},"last":1.359},{"interface":"front","average":{"1min":1.325,"5min":1.337,"15min":1.3560000000000001},"min":{"1min":0.83199999999999996,"5min":0.73999999999999999,"15min":0.66000000000000003},"max":{"1min":2.1030000000000002,"5min":2.1139999999999999,"15min":2.6469999999999998},"last":1.24}]},{"osd":6,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.407,"15min":1.389},"min":{"1min":0.90800000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.9390000000000001,"5min":2.0619999999999998,"15min":2.758},"last":1.46},{"interface":"front","average":{"1min":1.474,"5min":1.4139999999999999,"15min":1.4299999999999999},"min":{"1min":0.90200000000000002,"5min":0.79100000000000004,"15min":0.69599999999999995},"max":{"1min":2,"5min":2,"15min":2.8580000000000001},"last":1.417}]},{"osd":7,"last update":"Sat Mar 23 11:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.45,"15min":1.444},"min":{"1min":0.92400000000000004,"5min":0.83099999999999996,"15min":0.71899999999999997},"max":{"1min":1.754,"5min":2.23,"15min":2.246},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.464,"15min":1.494},"min":{"1min":0.85799999999999998,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.7120000000000002},"last":1.502}]}]},{"osd":1,"up_from":14,"seq":60129542322,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32632,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1552384,"data_stored":676495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.876,"15min":0.89400000000000002},"min":{"1min":0.41599999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.6579999999999999,"5min":1.6930000000000001,"15min":3.2959999999999998},"last":1.02},{"interface":"front","average":{"1min":1.002,"5min":1.0620000000000001,"15min":1.0149999999999999},"min":{"1min":0.5,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.53,"5min":1.8480000000000001,"15min":1.99},"last":0.92900000000000005}]},{"osd":2,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.94699999999999995},"min":{"1min":0.51200000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2949999999999999,"5min":1.5549999999999999,"15min":1.738},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.105,"15min":1.1040000000000001},"min":{"1min":0.60499999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5109999999999999,"5min":1.702,"15min":2.2160000000000002},"last":1.3149999999999999}]},{"osd":3,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2250000000000001,"15min":1.222},"min":{"1min":0.60699999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.734,"5min":1.8819999999999999,"15min":2.4430000000000001},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.133,"15min":1.1220000000000001},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.434},"max":{"1min":1.637,"5min":2.0539999999999998,"15min":2.4169999999999998},"last":1.2689999999999999}]},{"osd":4,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.345,"15min":1.3140000000000001},"min":{"1min":0.92100000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7190000000000001,"5min":1.8080000000000001,"15min":2.153},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.286,"15min":1.3},"min":{"1min":0.747,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":2.758},"last":1.387}]},{"osd":5,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3340000000000001,"15min":1.357},"min":{"1min":0.81100000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":2.004,"5min":2.004,"15min":2.6589999999999998},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.4239999999999999,"15min":1.4259999999999999},"min":{"1min":0.97599999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.8109999999999999,"5min":1.9610000000000001,"15min":3.0329999999999999},"last":1.712}]},{"osd":6,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4119999999999999,"15min":1.4179999999999999},"min":{"1min":1.0369999999999999,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.984,"5min":1.984,"15min":2.2730000000000001},"last":1.651},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4359999999999999,"15min":1.4650000000000001},"min":{"1min":0.91100000000000003,"5min":0.747,"15min":0.747},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.9630000000000001},"last":1.806}]},{"osd":7,"last update":"Sat Mar 23 11:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.524,"15min":1.528},"min":{"1min":0.93500000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.1269999999999998,"5min":2.2450000000000001,"15min":3.2749999999999999},"last":1.871},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4990000000000001,"15min":1.5249999999999999},"min":{"1min":1.0740000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.7829999999999999},"last":1.7629999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542320,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33208,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338632,"statfs":{"total":96636764160,"available":96602759168,"internally_reserved":0,"allocated":2146304,"data_stored":1266863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.85099999999999998,"15min":0.85599999999999998},"min":{"1min":0.46500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.294,"5min":1.8660000000000001,"15min":2.8479999999999999},"last":1.294},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.98199999999999998,"15min":0.97599999999999998},"min":{"1min":0.53900000000000003,"5min":0.40000000000000002,"15min":0.30399999999999999},"max":{"1min":1.629,"5min":1.694,"15min":2.4319999999999999},"last":0.76500000000000001}]},{"osd":2,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.028,"15min":1.0329999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.33700000000000002},"max":{"1min":1.6200000000000001,"5min":1.849,"15min":2.4980000000000002},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0980000000000001,"15min":1.0700000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.581,"5min":1.7569999999999999,"15min":2.5760000000000001},"last":0.96899999999999997}]},{"osd":3,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.113,"15min":1.1140000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.35799999999999998},"max":{"1min":1.5640000000000001,"5min":1.718,"15min":2.4580000000000002},"last":0.873},{"interface":"front","average":{"1min":1.208,"5min":1.3120000000000001,"15min":1.3129999999999999},"min":{"1min":0.68100000000000005,"5min":0.56000000000000005,"15min":0.48599999999999999},"max":{"1min":1.736,"5min":1.968,"15min":2.9390000000000001},"last":1.468}]},{"osd":4,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3280000000000001,"15min":1.333},"min":{"1min":0.78300000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.901,"5min":1.901,"15min":2.9750000000000001},"last":1.532},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.3160000000000001,"15min":1.296},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.39900000000000002},"max":{"1min":1.627,"5min":1.968,"15min":3.125},"last":1.2190000000000001}]},{"osd":5,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3380000000000001,"15min":1.3540000000000001},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.51200000000000001},"max":{"1min":1.744,"5min":1.925,"15min":3.0830000000000002},"last":1.349},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3380000000000001,"15min":1.3919999999999999},"min":{"1min":0.86699999999999999,"5min":0.66700000000000004,"15min":0.52700000000000002},"max":{"1min":1.7270000000000001,"5min":2.0449999999999999,"15min":3.1899999999999999},"last":1.6759999999999999}]},{"osd":6,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.413,"15min":1.4350000000000001},"min":{"1min":0.93500000000000005,"5min":0.73399999999999999,"15min":0.47999999999999998},"max":{"1min":1.8460000000000001,"5min":2.1030000000000002,"15min":3.391},"last":1.595},{"interface":"front","average":{"1min":1.411,"5min":1.423,"15min":1.4059999999999999},"min":{"1min":0.86899999999999999,"5min":0.623,"15min":0.58399999999999996},"max":{"1min":1.97,"5min":2.2170000000000001,"15min":3.153},"last":1.238}]},{"osd":7,"last update":"Sat Mar 23 11:33:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.5229999999999999,"15min":1.514},"min":{"1min":0.98499999999999999,"5min":0.67600000000000005,"15min":0.60599999999999998},"max":{"1min":2.0339999999999998,"5min":2.1400000000000001,"15min":3.5350000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5169999999999999,"15min":1.4930000000000001},"min":{"1min":1.161,"5min":0.78300000000000003,"15min":0.45300000000000001},"max":{"1min":1.8620000000000001,"5min":2.2610000000000001,"15min":3.3650000000000002},"last":1.415}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T11:33:20.289 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-23T11:33:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.495+0000 7f2bf7000640 1 Processor -- start 2024-03-23T11:33:20.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.495+0000 7f2bf7000640 1 -- start start 2024-03-23T11:33:20.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.497+0000 7f2bf7000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2bf015b850 0x7f2bf00b0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.497+0000 7f2bf7000640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2bf00b07a0 0x7f2bf00aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.497+0000 7f2bf7000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf015b3c0 con 0x7f2bf015a8b0 2024-03-23T11:33:20.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.497+0000 7f2bf7000640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf00b0c30 con 0x7f2bf00b07a0 2024-03-23T11:33:20.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.497+0000 7f2bf7000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf015bc90 con 0x7f2bf015b850 2024-03-23T11:33:20.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.497+0000 7f2bf6600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2bf00b07a0 0x7f2bf00aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.498+0000 7f2bf6600640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2bf00b07a0 0x7f2bf00aeef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58112/0 (socket says 172.21.15.62:58112) 2024-03-23T11:33:20.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.498+0000 7f2bf6600640 1 -- 172.21.15.62:0/2124239927 learned_addr learned my addr 172.21.15.62:0/2124239927 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:33:20.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.498+0000 7f2bf5200640 1 --2- 172.21.15.62:0/2124239927 >> v2:172.21.15.120:3300/0 conn(0x7f2bf015b850 0x7f2bf00b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.498+0000 7f2bf6600640 1 -- 172.21.15.62:0/2124239927 >> v2:172.21.15.120:3300/0 conn(0x7f2bf015b850 msgr2=0x7f2bf00b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:33:20.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.498+0000 7f2bf6600640 1 --2- 172.21.15.62:0/2124239927 >> v2:172.21.15.120:3300/0 conn(0x7f2bf015b850 0x7f2bf00b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.498+0000 7f2bf6600640 1 -- 172.21.15.62:0/2124239927 >> v1:172.21.15.120:6789/0 conn(0x7f2bf015a8b0 legacy=0x7f2bf00cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.498+0000 7f2bf6600640 1 -- 172.21.15.62:0/2124239927 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf00cca40 con 0x7f2bf00b07a0 2024-03-23T11:33:20.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.498+0000 7f2bf5200640 1 --2- 172.21.15.62:0/2124239927 >> v2:172.21.15.120:3300/0 conn(0x7f2bf015b850 0x7f2bf00b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:33:20.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.499+0000 7f2bf6600640 1 --2- 172.21.15.62:0/2124239927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bf00b07a0 0x7f2bf00aeef0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00cd070 tx=0x7f2be802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3bfdeb68479732 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:33:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.500+0000 7f2befe00640 1 -- 172.21.15.62:0/2124239927 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be802cb20 con 0x7f2bf00b07a0 2024-03-23T11:33:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.500+0000 7f2befe00640 1 -- 172.21.15.62:0/2124239927 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be802cc80 con 0x7f2bf00b07a0 2024-03-23T11:33:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.500+0000 7f2befe00640 1 -- 172.21.15.62:0/2124239927 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be80343f0 con 0x7f2bf00b07a0 2024-03-23T11:33:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.500+0000 7f2bf7000640 1 -- 172.21.15.62:0/2124239927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bf00b07a0 msgr2=0x7f2bf00aeef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.500+0000 7f2bf7000640 1 --2- 172.21.15.62:0/2124239927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bf00b07a0 0x7f2bf00aeef0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00cd070 tx=0x7f2be802bf80 comp rx=0 tx=0).stop 2024-03-23T11:33:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.501+0000 7f2bf7000640 1 -- 172.21.15.62:0/2124239927 shutdown_connections 2024-03-23T11:33:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.501+0000 7f2bf7000640 1 --2- 172.21.15.62:0/2124239927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bf00b07a0 0x7f2bf00aeef0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.501+0000 7f2bf7000640 1 --2- 172.21.15.62:0/2124239927 >> v2:172.21.15.120:3300/0 conn(0x7f2bf015b850 0x7f2bf00b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.501+0000 7f2bf7000640 1 -- 172.21.15.62:0/2124239927 >> 172.21.15.62:0/2124239927 conn(0x7f2bf00989d0 msgr2=0x7f2bf00cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:33:20.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.501+0000 7f2bf7000640 1 -- 172.21.15.62:0/2124239927 shutdown_connections 2024-03-23T11:33:20.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.501+0000 7f2bf7000640 1 -- 172.21.15.62:0/2124239927 wait complete. 2024-03-23T11:33:20.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.503+0000 7f2bf7000640 1 Processor -- start 2024-03-23T11:33:20.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.503+0000 7f2bf7000640 1 -- start start 2024-03-23T11:33:20.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.504+0000 7f2bf7000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bf00b07a0 0x7f2bf00cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.504+0000 7f2bf7000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bf015a8b0 0x7f2bf0146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.504+0000 7f2bf7000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf015b850 0x7f2bf00ce050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.504+0000 7f2bf7000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf00cda10 con 0x7f2bf015a8b0 2024-03-23T11:33:20.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.504+0000 7f2bf7000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf00b5300 con 0x7f2bf015b850 2024-03-23T11:33:20.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.504+0000 7f2bf7000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf00b2170 con 0x7f2bf00b07a0 2024-03-23T11:33:20.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.504+0000 7f2bf5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bf015a8b0 0x7f2bf0146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.504+0000 7f2bf5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bf015a8b0 0x7f2bf0146720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58124/0 (socket says 172.21.15.62:58124) 2024-03-23T11:33:20.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.504+0000 7f2bf5200640 1 -- 172.21.15.62:0/4041380299 learned_addr learned my addr 172.21.15.62:0/4041380299 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:33:20.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.504+0000 7f2bf5c00640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bf00b07a0 0x7f2bf00cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.504+0000 7f2bf6600640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf015b850 0x7f2bf00ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bf5200640 1 -- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bf00b07a0 msgr2=0x7f2bf00cd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bf5200640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bf00b07a0 0x7f2bf00cd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bf5200640 1 -- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf015b850 msgr2=0x7f2bf00ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bf5200640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf015b850 0x7f2bf00ce050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bf5200640 1 -- 172.21.15.62:0/4041380299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be8009b30 con 0x7f2bf015a8b0 2024-03-23T11:33:20.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bf6600640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf015b850 0x7f2bf00ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T11:33:20.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bf5c00640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bf00b07a0 0x7f2bf00cd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T11:33:20.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bf5200640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bf015a8b0 0x7f2bf0146720 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00e7e0 tx=0x7f2bdc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:33:20.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bed600640 1 -- 172.21.15.62:0/4041380299 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc004030 con 0x7f2bf015a8b0 2024-03-23T11:33:20.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bed600640 1 -- 172.21.15.62:0/4041380299 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc004190 con 0x7f2bf015a8b0 2024-03-23T11:33:20.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bed600640 1 -- 172.21.15.62:0/4041380299 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc004520 con 0x7f2bf015a8b0 2024-03-23T11:33:20.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.505+0000 7f2bf7000640 1 -- 172.21.15.62:0/4041380299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf00b2450 con 0x7f2bf015a8b0 2024-03-23T11:33:20.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.506+0000 7f2bf7000640 1 -- 172.21.15.62:0/4041380299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf01539f0 con 0x7f2bf015a8b0 2024-03-23T11:33:20.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.507+0000 7f2bf7000640 1 -- 172.21.15.62:0/4041380299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf00672c0 con 0x7f2bf015a8b0 2024-03-23T11:33:20.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.509+0000 7f2bed600640 1 -- 172.21.15.62:0/4041380299 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7f2bdc00c6d0 con 0x7f2bf015a8b0 2024-03-23T11:33:20.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.510+0000 7f2bed600640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bd006a7a0 0x7f2bd006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.510+0000 7f2bed600640 1 -- 172.21.15.62:0/4041380299 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f2bdc0887a0 con 0x7f2bf015a8b0 2024-03-23T11:33:20.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.510+0000 7f2bf5c00640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bd006a7a0 0x7f2bd006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.511+0000 7f2bf5c00640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bd006a7a0 0x7f2bd006cc60 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2be4007fd0 tx=0x7f2be4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:33:20.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.514+0000 7f2bed600640 1 -- 172.21.15.62:0/4041380299 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2bdc053ea0 con 0x7f2bf015a8b0 2024-03-23T11:33:20.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.713+0000 7f2bf7000640 1 -- 172.21.15.62:0/4041380299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f2bf00ca5e0 con 0x7f2bf015a8b0 2024-03-23T11:33:20.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.720+0000 7f2bed600640 1 -- 172.21.15.62:0/4041380299 <== mon.0 v2:172.21.15.62:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2bdc053ea0 con 0x7f2bf015a8b0 2024-03-23T11:33:20.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.721+0000 7f2bed600640 1 -- 172.21.15.62:0/4041380299 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v24)=0 v24) v1 ==== 123+0+0 (secure 0 0 0) 0x7f2bdc0574c0 con 0x7f2bf015a8b0 2024-03-23T11:33:20.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.724+0000 7f2bf7000640 1 -- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bd006a7a0 msgr2=0x7f2bd006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.724+0000 7f2bf7000640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bd006a7a0 0x7f2bd006cc60 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2be4007fd0 tx=0x7f2be4009040 comp rx=0 tx=0).stop 2024-03-23T11:33:20.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.724+0000 7f2bf7000640 1 -- 172.21.15.62:0/4041380299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bf015a8b0 msgr2=0x7f2bf0146720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.724+0000 7f2bf7000640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bf015a8b0 0x7f2bf0146720 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00e7e0 tx=0x7f2bdc00eca0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.724+0000 7f2bf7000640 1 -- 172.21.15.62:0/4041380299 shutdown_connections 2024-03-23T11:33:20.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.724+0000 7f2bf7000640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7f2bd006a7a0 0x7f2bd006cc60 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.724+0000 7f2bf7000640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bf015a8b0 0x7f2bf0146720 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.724+0000 7f2bf7000640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf015b850 0x7f2bf00ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.724+0000 7f2bf7000640 1 --2- 172.21.15.62:0/4041380299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bf00b07a0 0x7f2bf00cd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.724+0000 7f2bf7000640 1 -- 172.21.15.62:0/4041380299 >> 172.21.15.62:0/4041380299 conn(0x7f2bf00989d0 msgr2=0x7f2bf006eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:33:20.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.725+0000 7f2bf7000640 1 -- 172.21.15.62:0/4041380299 shutdown_connections 2024-03-23T11:33:20.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.725+0000 7f2bf7000640 1 -- 172.21.15.62:0/4041380299 wait complete. 2024-03-23T11:33:20.732 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T11:33:20.733 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T11:33:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.944+0000 7fee07200640 1 Processor -- start 2024-03-23T11:33:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.945+0000 7fee07200640 1 -- start start 2024-03-23T11:33:20.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.946+0000 7fee07200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fee08160420 0x7fee081608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.946+0000 7fee07200640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee080ce510 con 0x7fee0815fcf0 2024-03-23T11:33:20.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.946+0000 7fee07200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee080cdc70 con 0x7fee080b5db0 2024-03-23T11:33:20.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.946+0000 7fee07200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee08160df0 con 0x7fee08160420 2024-03-23T11:33:20.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.946+0000 7fee05e00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fee0815fcf0 0x7fee080d3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44586/0 (socket says 172.21.15.62:44586) 2024-03-23T11:33:20.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.946+0000 7fee05e00640 1 -- 172.21.15.62:0/2891313979 learned_addr learned my addr 172.21.15.62:0/2891313979 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-23T11:33:20.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.947+0000 7fedffe00640 1 --2- 172.21.15.62:0/2891313979 >> v2:172.21.15.120:3300/0 conn(0x7fee08160420 0x7fee081608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.947+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758936123 0 0) 0x7fee080ce510 con 0x7fee0815fcf0 2024-03-23T11:33:20.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.947+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedec003520 con 0x7fee0815fcf0 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.947+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 933918222 0 0) 0x7fedec003520 con 0x7fee0815fcf0 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.947+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 >> v2:172.21.15.120:3300/0 conn(0x7fee08160420 msgr2=0x7fee081608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.947+0000 7fee05400640 1 --2- 172.21.15.62:0/2891313979 >> v2:172.21.15.120:3300/0 conn(0x7fee08160420 0x7fee081608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.947+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 >> v1:172.21.15.120:6789/0 conn(0x7fee080b5db0 legacy=0x7fee080b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.947+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee080cdf50 con 0x7fee0815fcf0 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.947+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2676705381 0 0) 0x7fedf8001670 con 0x7fee0815fcf0 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.947+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 >> v1:172.21.15.62:6789/0 conn(0x7fee0815fcf0 legacy=0x7fee080d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05400640 1 --2- 172.21.15.62:0/2891313979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 0x7fedec003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05400640 1 --2- 172.21.15.62:0/2891313979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedec0047c0 0x7fedec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05400640 1 --2- 172.21.15.62:0/2891313979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedec0051d0 0x7fedec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee08160df0 con 0x7fedec0039c0 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec00c120 con 0x7fedec0051d0 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedec00c2e0 con 0x7fedec0047c0 2024-03-23T11:33:20.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05e00640 1 --2- 172.21.15.62:0/2891313979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 0x7fedec003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05e00640 1 -- 172.21.15.62:0/2891313979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedec0047c0 msgr2=0x7fedec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:33:20.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05e00640 1 --2- 172.21.15.62:0/2891313979 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fedec0047c0 0x7fedec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05e00640 1 -- 172.21.15.62:0/2891313979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedec0051d0 msgr2=0x7fedec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T11:33:20.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05e00640 1 --2- 172.21.15.62:0/2891313979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedec0051d0 0x7fedec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee05e00640 1 -- 172.21.15.62:0/2891313979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec003520 con 0x7fedec0039c0 2024-03-23T11:33:20.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.948+0000 7fee06800640 1 -- 172.21.15.62:0/2891313979 reap_dead start 2024-03-23T11:33:20.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.949+0000 7fee05e00640 1 --2- 172.21.15.62:0/2891313979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 0x7fedec003de0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fedf8001eb0 tx=0x7fedec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1c9f6785d8cb60 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:33:20.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.950+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8005ad0 con 0x7fedec0039c0 2024-03-23T11:33:20.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.950+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fedf8005c30 con 0x7fedec0039c0 2024-03-23T11:33:20.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.950+0000 7fee05400640 1 -- 172.21.15.62:0/2891313979 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf802ca00 con 0x7fedec0039c0 2024-03-23T11:33:20.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.950+0000 7fee07200640 1 -- 172.21.15.62:0/2891313979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 msgr2=0x7fedec003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.950+0000 7fee07200640 1 --2- 172.21.15.62:0/2891313979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 0x7fedec003de0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fedf8001eb0 tx=0x7fedec0057f0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.950+0000 7fee07200640 1 -- 172.21.15.62:0/2891313979 shutdown_connections 2024-03-23T11:33:20.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.950+0000 7fee07200640 1 --2- 172.21.15.62:0/2891313979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 0x7fedec003de0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.950+0000 7fee07200640 1 -- 172.21.15.62:0/2891313979 >> 172.21.15.62:0/2891313979 conn(0x7fee08089f60 msgr2=0x7fee080d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:33:20.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.950+0000 7fee07200640 1 -- 172.21.15.62:0/2891313979 shutdown_connections 2024-03-23T11:33:20.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.950+0000 7fee07200640 1 -- 172.21.15.62:0/2891313979 wait complete. 2024-03-23T11:33:20.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.952+0000 7fee07200640 1 Processor -- start 2024-03-23T11:33:20.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.952+0000 7fee07200640 1 -- start start 2024-03-23T11:33:20.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.953+0000 7fee07200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 0x7fee08158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.953+0000 7fee07200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee08158f20 0x7fee080c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.953+0000 7fee07200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee081589d0 0x7fee0806ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.953+0000 7fee07200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee080c9fd0 con 0x7fedec0039c0 2024-03-23T11:33:20.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.953+0000 7fee07200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee080ca270 con 0x7fee081589d0 2024-03-23T11:33:20.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.953+0000 7fee07200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee080ca550 con 0x7fee08158f20 2024-03-23T11:33:20.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.953+0000 7fee05e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 0x7fee08158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.953+0000 7fee05e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 0x7fee08158190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58142/0 (socket says 172.21.15.62:58142) 2024-03-23T11:33:20.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.953+0000 7fee05e00640 1 -- 172.21.15.62:0/864324739 learned_addr learned my addr 172.21.15.62:0/864324739 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-23T11:33:20.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.954+0000 7fee05e00640 1 -- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee08158f20 msgr2=0x7fee080c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.954+0000 7fee05e00640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee08158f20 0x7fee080c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.954+0000 7fee05e00640 1 -- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee081589d0 msgr2=0x7fee0806ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:20.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.954+0000 7fee05e00640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee081589d0 0x7fee0806ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:20.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.954+0000 7fee05e00640 1 -- 172.21.15.62:0/864324739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee080ca830 con 0x7fedec0039c0 2024-03-23T11:33:20.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.954+0000 7fee05e00640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 0x7fee08158190 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fedf802c340 tx=0x7fedf802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:33:20.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.954+0000 7fedfe000640 1 -- 172.21.15.62:0/864324739 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf803c040 con 0x7fedec0039c0 2024-03-23T11:33:20.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.954+0000 7fedfe000640 1 -- 172.21.15.62:0/864324739 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fedf802ccf0 con 0x7fedec0039c0 2024-03-23T11:33:20.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.954+0000 7fedfe000640 1 -- 172.21.15.62:0/864324739 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf80072d0 con 0x7fedec0039c0 2024-03-23T11:33:20.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.954+0000 7fee07200640 1 -- 172.21.15.62:0/864324739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee080cab10 con 0x7fedec0039c0 2024-03-23T11:33:20.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.955+0000 7fee07200640 1 -- 172.21.15.62:0/864324739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee081586d0 con 0x7fedec0039c0 2024-03-23T11:33:20.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.956+0000 7fee07200640 1 -- 172.21.15.62:0/864324739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee080ca690 con 0x7fedec0039c0 2024-03-23T11:33:20.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.957+0000 7fedfe000640 1 -- 172.21.15.62:0/864324739 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 88819+0+0 (secure 0 0 0) 0x7fedf8032070 con 0x7fedec0039c0 2024-03-23T11:33:20.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.957+0000 7fedfe000640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede406a7d0 0x7fede406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T11:33:20.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.958+0000 7fedfe000640 1 -- 172.21.15.62:0/864324739 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fedf80aa0f0 con 0x7fedec0039c0 2024-03-23T11:33:20.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.958+0000 7fedffe00640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede406a7d0 0x7fede406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T11:33:20.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.959+0000 7fedffe00640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede406a7d0 0x7fede406cc90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009390 tx=0x7fedf0009300 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T11:33:20.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:20.962+0000 7fedfe000640 1 -- 172.21.15.62:0/864324739 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fee080ca690 con 0x7fedec0039c0 2024-03-23T11:33:21.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.196+0000 7fee07200640 1 -- 172.21.15.62:0/864324739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee080ca690 con 0x7fedec0039c0 2024-03-23T11:33:21.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.198+0000 7fedfe000640 1 -- 172.21.15.62:0/864324739 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+3534 (secure 0 0 0) 0x7fee080ca690 con 0x7fedec0039c0 2024-03-23T11:33:21.187 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-23T11:33:21.187 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":109,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6997,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/2070886525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":2070886525},{"type":"v1","addr":"172.21.15.120:6836","nonce":2070886525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7213,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3481457918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3481457918},{"type":"v1","addr":"172.21.15.120:6837","nonce":3481457918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":96},{"gid":8994,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/591457103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":591457103},{"type":"v1","addr":"172.21.15.62:6833","nonce":591457103}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":9720,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3099162052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3099162052},{"type":"v1","addr":"172.21.15.62:6835","nonce":3099162052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109}],"filesystems":[]} 2024-03-23T11:33:21.187 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 109 2024-03-23T11:33:21.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.200+0000 7fee07200640 1 -- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede406a7d0 msgr2=0x7fede406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:21.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.200+0000 7fee07200640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede406a7d0 0x7fede406cc90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009390 tx=0x7fedf0009300 comp rx=0 tx=0).stop 2024-03-23T11:33:21.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.200+0000 7fee07200640 1 -- 172.21.15.62:0/864324739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 msgr2=0x7fee08158190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T11:33:21.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.200+0000 7fee07200640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 0x7fee08158190 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fedf802c340 tx=0x7fedf802e070 comp rx=0 tx=0).stop 2024-03-23T11:33:21.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.200+0000 7fee07200640 1 -- 172.21.15.62:0/864324739 shutdown_connections 2024-03-23T11:33:21.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.200+0000 7fee07200640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:6832/37071,v1:172.21.15.120:6833/37071] conn(0x7fede406a7d0 0x7fede406cc90 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:21.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.200+0000 7fee07200640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee081589d0 0x7fee0806ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:21.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.200+0000 7fee07200640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee08158f20 0x7fee080c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:21.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.200+0000 7fee07200640 1 --2- 172.21.15.62:0/864324739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedec0039c0 0x7fee08158190 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T11:33:21.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.200+0000 7fee07200640 1 -- 172.21.15.62:0/864324739 >> 172.21.15.62:0/864324739 conn(0x7fee08089f60 msgr2=0x7fee080d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T11:33:21.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.201+0000 7fee07200640 1 -- 172.21.15.62:0/864324739 shutdown_connections 2024-03-23T11:33:21.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-23T11:33:21.201+0000 7fee07200640 1 -- 172.21.15.62:0/864324739 wait complete. 2024-03-23T11:33:21.206 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-23T11:33:21.206 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-23T11:33:21.206 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T11:33:21.207 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.226 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T11:33:21.226 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T11:33:21.226 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.294 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T11:33:21.295 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T11:33:21.295 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.323 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T11:33:21.324 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T11:33:21.324 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.385 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T11:33:21.385 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-23T11:33:21.385 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-23T11:33:21.385 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.450 INFO:tasks.ceph.osd.0:Stopped 2024-03-23T11:33:21.450 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-23T11:33:21.450 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.525 INFO:tasks.ceph.osd.1:Stopped 2024-03-23T11:33:21.525 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-23T11:33:21.525 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.599 INFO:tasks.ceph.osd.2:Stopped 2024-03-23T11:33:21.600 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-23T11:33:21.600 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.644 INFO:tasks.ceph.osd.3:Stopped 2024-03-23T11:33:21.644 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-23T11:33:21.645 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.703 INFO:tasks.ceph.osd.4:Stopped 2024-03-23T11:33:21.704 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-23T11:33:21.704 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.757 INFO:tasks.ceph.osd.5:Stopped 2024-03-23T11:33:21.758 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-23T11:33:21.758 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.816 INFO:tasks.ceph.osd.6:Stopped 2024-03-23T11:33:21.817 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-23T11:33:21.817 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.879 INFO:tasks.ceph.osd.7:Stopped 2024-03-23T11:33:21.879 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-23T11:33:21.879 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-23T11:33:21.879 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:21.978 INFO:tasks.ceph.mgr.y:Stopped 2024-03-23T11:33:21.978 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-23T11:33:21.979 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:27.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-03-23T11:33:28.080 INFO:tasks.ceph.mgr.x:Stopped 2024-03-23T11:33:28.080 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-23T11:33:28.080 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-23T11:33:28.080 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:28.134 INFO:tasks.ceph.mon.a:Stopped 2024-03-23T11:33:28.134 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-23T11:33:28.134 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:28.167 INFO:tasks.ceph.mon.b:Stopped 2024-03-23T11:33:28.167 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-23T11:33:28.167 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T11:33:28.246 INFO:tasks.ceph.mon.c:Stopped 2024-03-23T11:33:28.246 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-23T11:33:28.247 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-03-23T11:33:28.357 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:24:44.132389+0000 mds.b (mds.0) 1 : cluster [WRN] Corrupt dentry 'file_to_be_damaged' in dir frag 0x10000000000: Invalid tag char 'd' pos 0: Malformed input [buffer:3](/subdir) 2024-03-23T11:33:28.358 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-03-23T11:33:28.359 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-03-23T11:33:28.453 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-03-23T11:33:28.541 INFO:teuthology.orchestra.run.smithi062.stdout:2024-03-23T11:26:57.377386+0000 mds.a (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-03-23T11:33:28.542 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:33:28.542 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-23T11:33:28.792 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:33:28.792 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-23T11:33:28.983 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:33:28.983 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-23T11:33:29.147 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:33:29.147 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-23T11:33:29.296 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:33:29.297 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-23T11:33:29.558 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:33:29.559 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-23T11:33:29.721 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:33:29.721 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-23T11:33:29.873 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:33:29.873 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-23T11:33:30.014 INFO:tasks.ceph:Archiving mon data... 2024-03-23T11:33:30.015 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234/data/mon.a.tgz 2024-03-23T11:33:30.016 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-03-23T11:33:30.076 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.Idebrm4Nka 2024-03-23T11:33:30.077 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Idebrm4Nka 2024-03-23T11:33:30.276 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.Idebrm4Nka 2024-03-23T11:33:30.432 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.Idebrm4Nka is 433KB 2024-03-23T11:33:30.547 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.Idebrm4Nka 2024-03-23T11:33:30.606 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234/data/mon.b.tgz 2024-03-23T11:33:30.606 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-03-23T11:33:30.660 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.zTiueHHERw 2024-03-23T11:33:30.660 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.zTiueHHERw 2024-03-23T11:33:30.857 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.zTiueHHERw 2024-03-23T11:33:31.007 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.zTiueHHERw is 457KB 2024-03-23T11:33:31.117 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.zTiueHHERw 2024-03-23T11:33:31.172 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234/data/mon.c.tgz 2024-03-23T11:33:31.172 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-03-23T11:33:31.223 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.OnrziX4Jga 2024-03-23T11:33:31.223 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.OnrziX4Jga 2024-03-23T11:33:31.418 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.OnrziX4Jga 2024-03-23T11:33:31.557 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.OnrziX4Jga is 457KB 2024-03-23T11:33:31.664 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.OnrziX4Jga 2024-03-23T11:33:31.721 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-23T11:33:31.721 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T11:33:31.729 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T11:33:31.827 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-03-23T11:33:31.827 DEBUG:teuthology.orchestra.run.smithi062:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-23T11:33:31.834 DEBUG:teuthology.orchestra.run.smithi120:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-23T11:33:31.932 INFO:teuthology.orchestra.run.smithi120.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-23T11:33:31.935 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-23T11:33:31.947 INFO:tasks.ceph:Archiving crash dumps... 2024-03-23T11:33:31.948 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234/remote/smithi062/crash 2024-03-23T11:33:31.950 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-23T11:33:32.037 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234/remote/smithi120/crash 2024-03-23T11:33:32.038 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-23T11:33:32.063 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-03-23T11:33:32.127 INFO:tasks.ceph:Shutting down logrotate 2024-03-23T11:33:32.128 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:33:32.136 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T11:33:32.275 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-23T11:33:32.381 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-23T11:33:32.479 INFO:tasks.ceph:Compressing logs... 2024-03-23T11:33:32.479 DEBUG:teuthology.orchestra.run.smithi062:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T11:33:32.487 DEBUG:teuthology.orchestra.run.smithi120:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T11:33:37.078 INFO:tasks.ceph:Archiving logs... 2024-03-23T11:33:37.078 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234/remote/smithi062/log 2024-03-23T11:33:37.079 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T11:33:38.224 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234/remote/smithi120/log 2024-03-23T11:33:38.225 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T11:33:38.749 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-23T11:33:38.757 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-23T11:33:38.758 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T11:33:38.765 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T11:33:38.920 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-23T11:33:38.929 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-23T11:33:38.929 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T11:33:38.937 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T11:33:38.984 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-03-23T11:33:38.989 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-03-23T11:33:38.995 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T11:33:38.995 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-03-23T11:33:38.995 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +816us[ +816us] +/- 89ms 2024-03-23T11:33:38.995 INFO:teuthology.orchestra.run.smithi062.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -1111us[-1106us] +/- 71ms 2024-03-23T11:33:38.995 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +475us[ +479us] +/- 78ms 2024-03-23T11:33:38.995 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:33:38.999 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T11:33:38.999 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-03-23T11:33:38.999 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +832us[ +829us] +/- 89ms 2024-03-23T11:33:38.999 INFO:teuthology.orchestra.run.smithi120.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -1081us[-1085us] +/- 71ms 2024-03-23T11:33:39.000 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +508us[ +504us] +/- 78ms 2024-03-23T11:33:39.000 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T11:33:39.001 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-23T11:33:39.010 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-23T11:33:39.011 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-23T11:33:39.020 DEBUG:teuthology.orchestra.run.smithi062:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T11:33:39.288 DEBUG:teuthology.orchestra.run.smithi120:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T11:33:39.494 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T11:33:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:33:39.605 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T11:33:39.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:33:39.707 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-23T11:33:39.717 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-23T11:33:39.726 INFO:teuthology.task.internal:Duration was 1445.357878 seconds 2024-03-23T11:33:39.726 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-23T11:33:39.736 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-23T11:33:39.736 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T11:33:39.744 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T11:33:39.902 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T11:33:39.918 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T11:33:40.337 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-23T11:33:40.337 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2024-03-23T11:33:40.338 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T11:33:40.403 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com 2024-03-23T11:33:40.403 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T11:33:40.477 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-23T11:33:40.477 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T11:33:40.485 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T11:33:40.575 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-23T11:33:40.575 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T11:33:40.580 DEBUG:teuthology.orchestra.run.smithi120:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T11:33:40.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-23T11:33:40.807 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-23T11:33:40.807 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T11:33:40.909 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T11:33:41.003 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-23T11:33:41.013 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T11:33:41.021 DEBUG:teuthology.orchestra.run.smithi120:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T11:33:41.127 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = core 2024-03-23T11:33:41.128 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2024-03-23T11:33:41.184 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T11:33:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:33:41.235 DEBUG:teuthology.orchestra.run.smithi120:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T11:33:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T11:33:41.287 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-23T11:33:41.298 INFO:teuthology.task.internal:Transferring archived files... 2024-03-23T11:33:41.298 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234/remote/smithi062 2024-03-23T11:33:41.299 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T11:33:41.408 DEBUG:teuthology.misc:Transferring archived files from smithi120:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234/remote/smithi120 2024-03-23T11:33:41.409 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T11:33:41.509 INFO:teuthology.task.internal:Removing archive directory... 2024-03-23T11:33:41.509 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T11:33:41.517 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T11:33:41.574 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-23T11:33:41.586 INFO:teuthology.task.internal:Not uploading archives. 2024-03-23T11:33:41.586 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-23T11:33:41.596 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-23T11:33:41.596 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T11:33:41.603 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T11:33:41.655 INFO:teuthology.orchestra.run.smithi062.stdout: 393945 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 11:33 /home/ubuntu/cephtest 2024-03-23T11:33:41.655 INFO:teuthology.orchestra.run.smithi120.stdout: 393295 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 11:33 /home/ubuntu/cephtest 2024-03-23T11:33:41.659 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-23T11:33:41.669 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-23T11:33:41.690 INFO:teuthology.nuke:Checking targets against current locks 2024-03-23T11:33:41.715 DEBUG:teuthology.nuke:shortname: smithi062 2024-03-23T11:33:41.715 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T11:33:41.743 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 11:00:41.438059', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBfVklLINf8hI4SjP+XzEWierEPkHx2H2V2ejczUNn1z4nGxBS+XfW1RinZqevtqk6b+cm2PeBcbUbgepxvTjw='} 2024-03-23T11:33:41.744 DEBUG:teuthology.nuke:shortname: smithi120 2024-03-23T11:33:41.744 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T11:33:41.771 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-23_09:38:22-fs-rishabh-yuri-RunCephCmd-base-reef-2-testing-default-smithi/7618234', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 11:00:41.435771', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMeZfkagDYOHSU0NTiM1iySUjN7prNazzwfIgxQoca6hVKsAjI6H2iOKJJ613zG69zBqLzNGYi5FLgQyo/nSRB4='} 2024-03-23T11:33:41.794 INFO:teuthology.orchestra.console.smithi062:Power off 2024-03-23T11:33:41.795 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T11:33:41.820 DEBUG:teuthology.orchestra.console.smithi062:power off output: Chassis Power Control: Down/Off 2024-03-23T11:33:41.820 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T11:33:41.833 INFO:teuthology.orchestra.console.smithi120:Power off 2024-03-23T11:33:41.834 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T11:33:41.842 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-03-23T11:33:41.854 DEBUG:teuthology.orchestra.console.smithi120:power off output: Chassis Power Control: Down/Off 2024-03-23T11:33:41.854 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T11:33:41.875 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-23T11:33:45.843 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T11:33:45.876 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T11:33:45.953 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-03-23T11:33:45.987 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-23T11:33:49.957 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T11:33:49.988 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T11:33:50.067 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-03-23T11:33:50.096 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-23T11:33:54.070 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T11:33:54.097 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T11:33:54.181 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is off 2024-03-23T11:33:54.207 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is off 2024-03-23T11:33:54.282 INFO:teuthology.orchestra.console.smithi062:Power off completed 2024-03-23T11:33:54.308 INFO:teuthology.orchestra.console.smithi120:Power off completed 2024-03-23T11:33:54.413 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} duration: 1445.3578777313232 failure_reason: '"2024-03-23T11:26:57.377386+0000 mds.a (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log' flavor: default mon.a-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 mon.b-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 owner: scheduled_rishabh@teuthology success: false 2024-03-23T11:33:54.413 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T11:33:54.492 INFO:teuthology.run:FAIL